2024-04-25T11:16:43.956 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T11:16:43.956 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673136 --owner scheduled_pdonnell@teuthology --name pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} --verbose -- /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673136/orig.config.yaml 2024-04-25T11:16:44.022 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T11:16:44.122 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673136 branch: wip-pdonnell-testing-20240425.015853-debug description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} email: pdonnell@redhat.com first_in_suite: false job_id: '7673136' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule name: pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-pdonnell-testing-20240425.015853-debug ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount dry_run: true install: ceph: flavor: default sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20240425.015853-debug sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 owner: scheduled_pdonnell@teuthology priority: 85 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 1880 sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 sleep_before_teardown: 0 subset: 178/512 suite: fs suite_branch: wip-pdonnell-testing-20240425.015853-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi090.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGyzLIALkkTm39YeVqABG79bzjJC8h1AqMGdVyJoO3A1iB96q4hkhVcOj1ZUpkjFxEnET7OUqbk8BnsSI3o/j0E= smithi138.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM2nLRDqP9X/f9DSvwry+rZWv1JVXFIzfs8m5SRfNiQD9su4TrMVlLKVnOb2WkoSmuI57nTolT2rfGOiMGqRgcw= smithi156.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH5GOPNhnM/57yML6FPj+NES+/RX79qc4PWedqM4EbgF0L7YPGDYNqYOSqaFfv0c9UOpui379W5950rYhI8UUus= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph config set mgr mgr/snap_schedule/log_level debug - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/iogen.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-25_10:00:23 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T11:16:44.122 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa; will attempt to use it 2024-04-25T11:16:44.123 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks 2024-04-25T11:16:44.123 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T11:16:44.125 INFO:teuthology.task.internal:Checking packages... 2024-04-25T11:16:44.149 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '994e2f20102a20fa9a27966c14b625cc01884df8' 2024-04-25T11:16:44.149 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T11:16:44.149 INFO:teuthology.packaging:ref: None 2024-04-25T11:16:44.150 INFO:teuthology.packaging:tag: None 2024-04-25T11:16:44.150 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240425.015853-debug 2024-04-25T11:16:44.150 INFO:teuthology.packaging:sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T11:16:44.150 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-pdonnell-testing-20240425.015853-debug 2024-04-25T11:16:44.370 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3291.g994e2f20 2024-04-25T11:16:44.371 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T11:16:44.380 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T11:16:44.380 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T11:16:44.396 INFO:teuthology.task.internal:Saving configuration 2024-04-25T11:16:44.418 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T11:16:44.426 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T11:16:44.450 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673136', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 11:10:59.738232', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGyzLIALkkTm39YeVqABG79bzjJC8h1AqMGdVyJoO3A1iB96q4hkhVcOj1ZUpkjFxEnET7OUqbk8BnsSI3o/j0E='} 2024-04-25T11:16:44.469 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi138.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673136', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 11:10:59.741095', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM2nLRDqP9X/f9DSvwry+rZWv1JVXFIzfs8m5SRfNiQD9su4TrMVlLKVnOb2WkoSmuI57nTolT2rfGOiMGqRgcw='} 2024-04-25T11:16:44.492 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673136', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 11:10:59.739976', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH5GOPNhnM/57yML6FPj+NES+/RX79qc4PWedqM4EbgF0L7YPGDYNqYOSqaFfv0c9UOpui379W5950rYhI8UUus='} 2024-04-25T11:16:44.492 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T11:16:44.503 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-25T11:16:44.503 INFO:teuthology.task.internal:roles: ubuntu@smithi138.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-25T11:16:44.503 INFO:teuthology.task.internal:roles: ubuntu@smithi156.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-25T11:16:44.503 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T11:16:44.640 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f09283e40d0>, signals=[15]) 2024-04-25T11:16:44.641 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T11:16:44.648 INFO:teuthology.task.internal:Opening connections... 2024-04-25T11:16:44.648 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T11:16:44.651 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T11:16:44.729 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi138.front.sepia.ceph.com 2024-04-25T11:16:44.730 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T11:16:44.803 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi156.front.sepia.ceph.com 2024-04-25T11:16:44.804 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T11:16:44.877 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T11:16:44.884 DEBUG:teuthology.orchestra.run.smithi090:> uname -m 2024-04-25T11:16:44.902 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64 2024-04-25T11:16:44.903 DEBUG:teuthology.orchestra.run.smithi090:> cat /etc/os-release 2024-04-25T11:16:44.959 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="CentOS Stream" 2024-04-25T11:16:44.959 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="9" 2024-04-25T11:16:44.959 INFO:teuthology.orchestra.run.smithi090.stdout:ID="centos" 2024-04-25T11:16:44.959 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE="rhel fedora" 2024-04-25T11:16:44.960 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="9" 2024-04-25T11:16:44.960 INFO:teuthology.orchestra.run.smithi090.stdout:PLATFORM_ID="platform:el9" 2024-04-25T11:16:44.960 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T11:16:44.960 INFO:teuthology.orchestra.run.smithi090.stdout:ANSI_COLOR="0;31" 2024-04-25T11:16:44.960 INFO:teuthology.orchestra.run.smithi090.stdout:LOGO="fedora-logo-icon" 2024-04-25T11:16:44.960 INFO:teuthology.orchestra.run.smithi090.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T11:16:44.960 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://centos.org/" 2024-04-25T11:16:44.960 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T11:16:44.960 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T11:16:44.960 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T11:16:44.961 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2024-04-25T11:16:44.987 DEBUG:teuthology.orchestra.run.smithi138:> uname -m 2024-04-25T11:16:45.005 INFO:teuthology.orchestra.run.smithi138.stdout:x86_64 2024-04-25T11:16:45.005 DEBUG:teuthology.orchestra.run.smithi138:> cat /etc/os-release 2024-04-25T11:16:45.062 INFO:teuthology.orchestra.run.smithi138.stdout:NAME="CentOS Stream" 2024-04-25T11:16:45.062 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION="9" 2024-04-25T11:16:45.062 INFO:teuthology.orchestra.run.smithi138.stdout:ID="centos" 2024-04-25T11:16:45.062 INFO:teuthology.orchestra.run.smithi138.stdout:ID_LIKE="rhel fedora" 2024-04-25T11:16:45.062 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION_ID="9" 2024-04-25T11:16:45.063 INFO:teuthology.orchestra.run.smithi138.stdout:PLATFORM_ID="platform:el9" 2024-04-25T11:16:45.063 INFO:teuthology.orchestra.run.smithi138.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T11:16:45.063 INFO:teuthology.orchestra.run.smithi138.stdout:ANSI_COLOR="0;31" 2024-04-25T11:16:45.063 INFO:teuthology.orchestra.run.smithi138.stdout:LOGO="fedora-logo-icon" 2024-04-25T11:16:45.063 INFO:teuthology.orchestra.run.smithi138.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T11:16:45.063 INFO:teuthology.orchestra.run.smithi138.stdout:HOME_URL="https://centos.org/" 2024-04-25T11:16:45.063 INFO:teuthology.orchestra.run.smithi138.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T11:16:45.063 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T11:16:45.063 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T11:16:45.063 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server 2024-04-25T11:16:45.087 DEBUG:teuthology.orchestra.run.smithi156:> uname -m 2024-04-25T11:16:45.105 INFO:teuthology.orchestra.run.smithi156.stdout:x86_64 2024-04-25T11:16:45.105 DEBUG:teuthology.orchestra.run.smithi156:> cat /etc/os-release 2024-04-25T11:16:45.162 INFO:teuthology.orchestra.run.smithi156.stdout:NAME="CentOS Stream" 2024-04-25T11:16:45.162 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION="9" 2024-04-25T11:16:45.163 INFO:teuthology.orchestra.run.smithi156.stdout:ID="centos" 2024-04-25T11:16:45.163 INFO:teuthology.orchestra.run.smithi156.stdout:ID_LIKE="rhel fedora" 2024-04-25T11:16:45.163 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_ID="9" 2024-04-25T11:16:45.163 INFO:teuthology.orchestra.run.smithi156.stdout:PLATFORM_ID="platform:el9" 2024-04-25T11:16:45.163 INFO:teuthology.orchestra.run.smithi156.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T11:16:45.163 INFO:teuthology.orchestra.run.smithi156.stdout:ANSI_COLOR="0;31" 2024-04-25T11:16:45.163 INFO:teuthology.orchestra.run.smithi156.stdout:LOGO="fedora-logo-icon" 2024-04-25T11:16:45.163 INFO:teuthology.orchestra.run.smithi156.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T11:16:45.163 INFO:teuthology.orchestra.run.smithi156.stdout:HOME_URL="https://centos.org/" 2024-04-25T11:16:45.163 INFO:teuthology.orchestra.run.smithi156.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T11:16:45.163 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T11:16:45.163 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T11:16:45.163 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server 2024-04-25T11:16:45.187 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T11:16:45.198 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T11:16:45.206 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T11:16:45.206 DEBUG:teuthology.orchestra.run.smithi090:> test '!' -e /home/ubuntu/cephtest 2024-04-25T11:16:45.209 DEBUG:teuthology.orchestra.run.smithi138:> test '!' -e /home/ubuntu/cephtest 2024-04-25T11:16:45.211 DEBUG:teuthology.orchestra.run.smithi156:> test '!' -e /home/ubuntu/cephtest 2024-04-25T11:16:45.228 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T11:16:45.235 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T11:16:45.235 DEBUG:teuthology.orchestra.run.smithi090:> test -z $(ls -A /var/lib/ceph) 2024-04-25T11:16:45.266 DEBUG:teuthology.orchestra.run.smithi138:> test -z $(ls -A /var/lib/ceph) 2024-04-25T11:16:45.268 DEBUG:teuthology.orchestra.run.smithi156:> test -z $(ls -A /var/lib/ceph) 2024-04-25T11:16:45.430 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T11:16:45.509 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T11:16:45.523 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-04-25T11:16:45.523 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-25T11:16:45.524 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-25T11:16:45.524 DEBUG:teuthology.orchestra.run.smithi090:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T11:16:45.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T11:16:45.541 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-04-25T11:16:45.598 INFO:teuthology.orchestra.run.smithi090.stdout:5.14.0-437.el9.x86_64 2024-04-25T11:16:45.599 INFO:teuthology.task.kernel:Running kernel on smithi090: 5.14.0-437.el9.x86_64 2024-04-25T11:16:45.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum install -y kernel 2024-04-25T11:16:48.107 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-25T11:16:50.316 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-04-25T11:16:54.309 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.2 MB 00:00 2024-04-25T11:17:01.112 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - Extras packages 2.9 kB/s | 16 kB 00:05 2024-04-25T11:17:02.347 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-04-25T11:17:07.439 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-25T11:17:09.362 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T11:17:09.363 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T11:17:09.470 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-25T11:17:09.471 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-04-25T11:17:09.472 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-25T11:17:09.573 DEBUG:teuthology.orchestra.run.smithi090:> echo no | sudo yum reinstall kernel || true 2024-04-25T11:17:10.068 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 11:17:07 AM UTC. 2024-04-25T11:17:10.196 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-25T11:17:10.214 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-25T11:17:10.214 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-04-25T11:17:10.214 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-25T11:17:10.214 INFO:teuthology.orchestra.run.smithi090.stdout:Reinstalling: 2024-04-25T11:17:10.215 INFO:teuthology.orchestra.run.smithi090.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T11:17:10.215 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:17:10.215 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-04-25T11:17:10.215 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-25T11:17:10.215 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:17:10.215 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 255 k 2024-04-25T11:17:10.215 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 0 2024-04-25T11:17:10.215 INFO:teuthology.orchestra.run.smithi090.stderr:Operation aborted. 2024-04-25T11:17:10.293 INFO:teuthology.orchestra.run.smithi090.stdout:Is this ok [y/N]: 2024-04-25T11:17:10.294 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum reinstall -y kernel || true 2024-04-25T11:17:10.790 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 11:17:07 AM UTC. 2024-04-25T11:17:10.905 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-25T11:17:10.906 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-25T11:17:10.906 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-04-25T11:17:10.906 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-25T11:17:10.906 INFO:teuthology.orchestra.run.smithi090.stdout:Reinstalling: 2024-04-25T11:17:10.906 INFO:teuthology.orchestra.run.smithi090.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T11:17:10.906 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:17:10.906 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-04-25T11:17:10.906 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-25T11:17:10.906 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:17:10.906 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 255 k 2024-04-25T11:17:10.907 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 0 2024-04-25T11:17:10.907 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-04-25T11:17:11.209 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.4 MB/s | 255 kB 00:00 2024-04-25T11:17:11.209 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-04-25T11:17:11.209 INFO:teuthology.orchestra.run.smithi090.stdout:Total 842 kB/s | 255 kB 00:00 2024-04-25T11:17:11.230 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-04-25T11:17:11.441 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-04-25T11:17:11.441 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-04-25T11:17:11.446 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-04-25T11:17:11.446 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-04-25T11:17:11.585 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-04-25T11:17:11.677 INFO:teuthology.orchestra.run.smithi090.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T11:17:11.677 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T11:17:12.520 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T11:17:12.521 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T11:17:13.286 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T11:17:13.287 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:17:13.287 INFO:teuthology.orchestra.run.smithi090.stdout:Reinstalled: 2024-04-25T11:17:13.287 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T11:17:13.287 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:17:13.287 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-25T11:17:13.499 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T11:17:13.546 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T11:17:13.546 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T11:17:13.547 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T11:17:13.547 DEBUG:teuthology.parallel:result is None 2024-04-25T11:17:13.547 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T11:17:13.555 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T11:17:13.555 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T11:17:13.557 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T11:17:13.560 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T11:17:13.577 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T11:17:13.589 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T11:17:13.602 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T11:17:13.602 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T11:17:13.612 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T11:17:13.617 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T11:17:13.719 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T11:17:13.730 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T11:17:13.730 DEBUG:teuthology.orchestra.run.smithi138:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T11:17:13.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T11:17:13.746 DEBUG:teuthology.orchestra.run.smithi156:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T11:17:13.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T11:17:13.763 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T11:17:13.765 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T11:17:13.790 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T11:17:13.793 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T11:17:13.807 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T11:17:13.823 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T11:17:13.841 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T11:17:13.841 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T11:17:13.860 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T11:17:13.862 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T11:17:13.868 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T11:17:13.868 DEBUG:teuthology.orchestra.run.smithi090:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T11:17:13.871 DEBUG:teuthology.orchestra.run.smithi138:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T11:17:13.885 DEBUG:teuthology.orchestra.run.smithi156:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T11:17:13.929 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T11:17:13.939 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T11:17:13.939 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T11:17:13.941 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T11:17:13.951 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T11:17:13.986 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T11:17:14.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T11:17:14.090 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T11:17:14.172 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T11:17:14.210 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T11:17:14.210 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T11:17:14.273 DEBUG:teuthology.orchestra.run.smithi138:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T11:17:14.318 DEBUG:teuthology.orchestra.run.smithi138:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T11:17:14.366 DEBUG:teuthology.orchestra.run.smithi138:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T11:17:14.451 DEBUG:teuthology.orchestra.run.smithi138:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T11:17:14.493 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-25T11:17:14.493 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T11:17:14.558 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T11:17:14.603 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T11:17:14.659 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T11:17:14.750 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T11:17:14.785 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-25T11:17:14.785 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T11:17:14.852 DEBUG:teuthology.orchestra.run.smithi090:> sudo service rsyslog restart 2024-04-25T11:17:14.855 DEBUG:teuthology.orchestra.run.smithi138:> sudo service rsyslog restart 2024-04-25T11:17:14.857 DEBUG:teuthology.orchestra.run.smithi156:> sudo service rsyslog restart 2024-04-25T11:17:14.889 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T11:17:14.892 INFO:teuthology.orchestra.run.smithi138.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T11:17:14.931 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T11:17:15.383 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T11:17:15.391 INFO:teuthology.task.internal:Starting timer... 2024-04-25T11:17:15.391 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T11:17:15.402 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T11:17:15.411 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-25T11:17:15.411 DEBUG:teuthology.orchestra.run.smithi090:> sudo service auditd rotate 2024-04-25T11:17:15.485 INFO:teuthology.orchestra.run.smithi090.stdout:Rotating logs: 2024-04-25T11:17:15.487 DEBUG:teuthology.orchestra.run.smithi138:> sudo service auditd rotate 2024-04-25T11:17:15.543 INFO:teuthology.orchestra.run.smithi138.stdout:Rotating logs: 2024-04-25T11:17:15.545 DEBUG:teuthology.orchestra.run.smithi156:> sudo service auditd rotate 2024-04-25T11:17:15.584 INFO:teuthology.orchestra.run.smithi156.stdout:Rotating logs: 2024-04-25T11:17:15.586 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T11:17:15.586 DEBUG:teuthology.orchestra.run.smithi090:> /usr/sbin/getenforce 2024-04-25T11:17:15.615 INFO:teuthology.orchestra.run.smithi090.stdout:Permissive 2024-04-25T11:17:15.615 DEBUG:teuthology.orchestra.run.smithi138:> /usr/sbin/getenforce 2024-04-25T11:17:15.648 INFO:teuthology.orchestra.run.smithi138.stdout:Permissive 2024-04-25T11:17:15.648 DEBUG:teuthology.orchestra.run.smithi156:> /usr/sbin/getenforce 2024-04-25T11:17:15.683 INFO:teuthology.orchestra.run.smithi156.stdout:Permissive 2024-04-25T11:17:15.683 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi090.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi138.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi156.front.sepia.ceph.com': 'permissive'} 2024-04-25T11:17:15.684 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T11:17:15.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T11:17:15.710 DEBUG:teuthology.orchestra.run.smithi138:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T11:17:15.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T11:17:15.736 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T11:17:15.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T11:17:15.763 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T11:17:15.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo /usr/sbin/setenforce permissive 2024-04-25T11:17:15.794 DEBUG:teuthology.orchestra.run.smithi138:> sudo /usr/sbin/setenforce permissive 2024-04-25T11:17:15.819 DEBUG:teuthology.orchestra.run.smithi156:> sudo /usr/sbin/setenforce permissive 2024-04-25T11:17:15.847 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T11:17:15.856 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-25T11:17:15.862 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-25T11:17:15.998 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T11:17:16.015 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T11:17:16.016 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi090.front.sepia.ceph.com,smithi138.front.sepia.ceph.com,smithi156.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T11:22:15.981 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi090.front.sepia.ceph.com'), Remote(name='ubuntu@smithi138.front.sepia.ceph.com'), Remote(name='ubuntu@smithi156.front.sepia.ceph.com')] 2024-04-25T11:22:15.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-25T11:22:15.984 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T11:22:16.062 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-04-25T11:22:16.145 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-25T11:22:16.146 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-04-25T11:22:16.146 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T11:22:16.221 DEBUG:teuthology.orchestra.run.smithi138:> true 2024-04-25T11:22:16.299 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-04-25T11:22:16.299 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-04-25T11:22:16.300 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T11:22:16.372 DEBUG:teuthology.orchestra.run.smithi156:> true 2024-04-25T11:22:16.450 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-04-25T11:22:16.450 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T11:22:16.464 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T11:22:16.464 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T11:22:16.464 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T11:22:16.467 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T11:22:16.467 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T11:22:16.469 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T11:22:16.469 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T11:22:16.493 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T11:22:16.499 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T11:22:16.507 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T11:22:16.514 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T11:22:16.519 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T11:22:16.530 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: ntpd: command not found 2024-04-25T11:22:16.533 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T11:22:16.539 INFO:teuthology.orchestra.run.smithi138.stderr:sudo: ntpd: command not found 2024-04-25T11:22:16.541 INFO:teuthology.orchestra.run.smithi090.stdout:506 Cannot talk to daemon 2024-04-25T11:22:16.550 INFO:teuthology.orchestra.run.smithi138.stdout:506 Cannot talk to daemon 2024-04-25T11:22:16.555 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T11:22:16.556 INFO:teuthology.orchestra.run.smithi156.stderr:sudo: ntpd: command not found 2024-04-25T11:22:16.564 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T11:22:16.567 INFO:teuthology.orchestra.run.smithi156.stdout:506 Cannot talk to daemon 2024-04-25T11:22:16.568 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T11:22:16.578 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T11:22:16.581 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T11:22:16.594 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T11:22:16.620 INFO:teuthology.orchestra.run.smithi090.stderr:bash: line 1: ntpq: command not found 2024-04-25T11:22:16.624 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T11:22:16.624 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2024-04-25T11:22:16.624 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T11:22:16.624 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T11:22:16.624 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T11:22:16.624 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T11:22:16.632 INFO:teuthology.orchestra.run.smithi138.stderr:bash: line 1: ntpq: command not found 2024-04-25T11:22:16.635 INFO:teuthology.orchestra.run.smithi138.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T11:22:16.635 INFO:teuthology.orchestra.run.smithi138.stdout:=============================================================================== 2024-04-25T11:22:16.636 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T11:22:16.636 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T11:22:16.636 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T11:22:16.636 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T11:22:16.648 INFO:teuthology.orchestra.run.smithi156.stderr:bash: line 1: ntpq: command not found 2024-04-25T11:22:16.651 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T11:22:16.651 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2024-04-25T11:22:16.652 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T11:22:16.652 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T11:22:16.652 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T11:22:16.652 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T11:22:16.652 INFO:teuthology.run_tasks:Running task install... 2024-04-25T11:22:16.660 DEBUG:teuthology.task.install:project ceph 2024-04-25T11:22:16.660 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}} 2024-04-25T11:22:16.661 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'} 2024-04-25T11:22:16.661 INFO:teuthology.task.install:Using flavor: default 2024-04-25T11:22:16.667 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T11:22:16.667 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-25T11:22:16.667 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'tag': None, 'wait_for_package': False} 2024-04-25T11:22:16.668 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T11:22:16.669 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'tag': None, 'wait_for_package': False} 2024-04-25T11:22:16.670 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T11:22:16.671 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'tag': None, 'wait_for_package': False} 2024-04-25T11:22:16.671 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T11:22:16.823 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/ 2024-04-25T11:22:16.823 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3291.g994e2f20 2024-04-25T11:22:16.912 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/ 2024-04-25T11:22:16.913 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3291.g994e2f20 2024-04-25T11:22:16.931 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/ 2024-04-25T11:22:16.932 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3291.g994e2f20 2024-04-25T11:22:16.936 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T11:22:16.936 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T11:22:16.936 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T11:22:16.985 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T11:22:16.985 DEBUG:teuthology.orchestra.run.smithi090:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/994e2f20102a20fa9a27966c14b625cc01884df8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T11:22:17.040 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T11:22:17.040 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-25T11:22:17.040 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T11:22:17.050 DEBUG:teuthology.orchestra.run.smithi090:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T11:22:17.064 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T11:22:17.064 DEBUG:teuthology.orchestra.run.smithi138:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/994e2f20102a20fa9a27966c14b625cc01884df8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T11:22:17.109 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T11:22:17.109 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-25T11:22:17.109 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T11:22:17.123 DEBUG:teuthology.orchestra.run.smithi090:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T11:22:17.129 DEBUG:teuthology.orchestra.run.smithi138:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T11:22:17.134 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T11:22:17.135 DEBUG:teuthology.orchestra.run.smithi156:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/994e2f20102a20fa9a27966c14b625cc01884df8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T11:22:17.186 INFO:teuthology.orchestra.run.smithi090.stdout:check_obsoletes = 1 2024-04-25T11:22:17.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum clean all 2024-04-25T11:22:17.198 DEBUG:teuthology.orchestra.run.smithi156:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T11:22:17.202 DEBUG:teuthology.orchestra.run.smithi138:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T11:22:17.266 INFO:teuthology.orchestra.run.smithi138.stdout:check_obsoletes = 1 2024-04-25T11:22:17.268 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum clean all 2024-04-25T11:22:17.271 DEBUG:teuthology.orchestra.run.smithi156:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T11:22:17.334 INFO:teuthology.orchestra.run.smithi156.stdout:check_obsoletes = 1 2024-04-25T11:22:17.335 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum clean all 2024-04-25T11:22:17.411 INFO:teuthology.orchestra.run.smithi090.stdout:48 files removed 2024-04-25T11:22:17.439 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T11:22:17.493 INFO:teuthology.orchestra.run.smithi138.stdout:48 files removed 2024-04-25T11:22:17.520 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T11:22:17.561 INFO:teuthology.orchestra.run.smithi156.stdout:48 files removed 2024-04-25T11:22:17.588 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T11:22:18.090 INFO:teuthology.orchestra.run.smithi090.stdout:ceph packages for x86_64 212 kB/s | 84 kB 00:00 2024-04-25T11:22:18.150 INFO:teuthology.orchestra.run.smithi156.stdout:ceph packages for x86_64 280 kB/s | 84 kB 00:00 2024-04-25T11:22:18.266 INFO:teuthology.orchestra.run.smithi138.stdout:ceph packages for x86_64 172 kB/s | 84 kB 00:00 2024-04-25T11:22:18.516 INFO:teuthology.orchestra.run.smithi090.stdout:ceph noarch packages 31 kB/s | 12 kB 00:00 2024-04-25T11:22:18.617 INFO:teuthology.orchestra.run.smithi156.stdout:ceph noarch packages 28 kB/s | 12 kB 00:00 2024-04-25T11:22:18.675 INFO:teuthology.orchestra.run.smithi138.stdout:ceph noarch packages 32 kB/s | 12 kB 00:00 2024-04-25T11:22:18.851 INFO:teuthology.orchestra.run.smithi090.stdout:ceph source packages 6.3 kB/s | 1.9 kB 00:00 2024-04-25T11:22:19.043 INFO:teuthology.orchestra.run.smithi156.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-04-25T11:22:19.051 INFO:teuthology.orchestra.run.smithi138.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-04-25T11:22:19.676 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-04-25T11:22:19.859 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-04-25T11:22:21.811 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-04-25T11:22:23.335 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - AppStream 7.8 MB/s | 19 MB 00:02 2024-04-25T11:22:25.059 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - BaseOS 1.3 MB/s | 8.0 MB 00:05 2024-04-25T11:22:26.058 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - CRB 8.8 MB/s | 6.2 MB 00:00 2024-04-25T11:22:27.573 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 16 kB 00:00 2024-04-25T11:22:27.653 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - CRB 7.8 MB/s | 6.2 MB 00:00 2024-04-25T11:22:27.885 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-04-25T11:22:29.338 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-04-25T11:22:30.110 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 8.7 MB/s | 21 MB 00:02 2024-04-25T11:22:30.539 INFO:teuthology.orchestra.run.smithi138.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-04-25T11:22:32.744 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 6.2 MB 00:01 2024-04-25T11:22:34.386 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-04-25T11:22:35.108 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-25T11:22:35.597 INFO:teuthology.orchestra.run.smithi138.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-04-25T11:22:35.848 INFO:teuthology.orchestra.run.smithi156.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-25T11:22:37.276 INFO:teuthology.orchestra.run.smithi090.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T11:22:37.277 INFO:teuthology.orchestra.run.smithi090.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T11:22:37.287 INFO:teuthology.orchestra.run.smithi090.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T11:22:37.288 INFO:teuthology.orchestra.run.smithi090.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T11:22:37.289 INFO:teuthology.orchestra.run.smithi090.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T11:22:37.290 INFO:teuthology.orchestra.run.smithi090.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T11:22:37.292 INFO:teuthology.orchestra.run.smithi090.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T11:22:37.294 INFO:teuthology.orchestra.run.smithi090.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T11:22:37.295 INFO:teuthology.orchestra.run.smithi090.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T11:22:37.356 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-25T11:22:37.364 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2024-04-25T11:22:37.365 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-04-25T11:22:37.365 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2024-04-25T11:22:37.365 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-04-25T11:22:37.365 INFO:teuthology.orchestra.run.smithi090.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T11:22:37.365 INFO:teuthology.orchestra.run.smithi090.stdout: ceph x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 6.6 k 2024-04-25T11:22:37.365 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.4 M 2024-04-25T11:22:37.365 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 863 k 2024-04-25T11:22:37.365 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 147 k 2024-04-25T11:22:37.365 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 1.5 M 2024-04-25T11:22:37.365 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 142 k 2024-04-25T11:22:37.365 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 3.5 M 2024-04-25T11:22:37.365 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 7.4 M 2024-04-25T11:22:37.365 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 49 k 2024-04-25T11:22:37.365 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 10 M 2024-04-25T11:22:37.366 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 43 M 2024-04-25T11:22:37.366 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-volume noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 273 k 2024-04-25T11:22:37.366 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 762 k 2024-04-25T11:22:37.366 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 224 k 2024-04-25T11:22:37.366 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-shell x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 27 k 2024-04-25T11:22:37.366 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-top noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 18 k 2024-04-25T11:22:37.366 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T11:22:37.366 INFO:teuthology.orchestra.run.smithi090.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T11:22:37.366 INFO:teuthology.orchestra.run.smithi090.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T11:22:37.366 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 32 k 2024-04-25T11:22:37.366 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 756 k 2024-04-25T11:22:37.366 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 127 k 2024-04-25T11:22:37.366 INFO:teuthology.orchestra.run.smithi090.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T11:22:37.367 INFO:teuthology.orchestra.run.smithi090.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T11:22:37.367 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T11:22:37.367 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T11:22:37.367 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T11:22:37.367 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T11:22:37.367 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 161 k 2024-04-25T11:22:37.367 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 321 k 2024-04-25T11:22:37.367 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 298 k 2024-04-25T11:22:37.367 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 100 k 2024-04-25T11:22:37.367 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 86 k 2024-04-25T11:22:37.367 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.0 M 2024-04-25T11:22:37.367 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 169 k 2024-04-25T11:22:37.367 INFO:teuthology.orchestra.run.smithi090.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T11:22:37.367 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrading: 2024-04-25T11:22:37.367 INFO:teuthology.orchestra.run.smithi090.stdout: librados2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.3 M 2024-04-25T11:22:37.368 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.1 M 2024-04-25T11:22:37.368 INFO:teuthology.orchestra.run.smithi090.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T11:22:37.368 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T11:22:37.368 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-04-25T11:22:37.368 INFO:teuthology.orchestra.run.smithi090.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T11:22:37.368 INFO:teuthology.orchestra.run.smithi090.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T11:22:37.368 INFO:teuthology.orchestra.run.smithi090.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T11:22:37.368 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 20 M 2024-04-25T11:22:37.368 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 34 k 2024-04-25T11:22:37.368 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 2.3 M 2024-04-25T11:22:37.368 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 249 k 2024-04-25T11:22:37.368 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 4.9 M 2024-04-25T11:22:37.368 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 18 M 2024-04-25T11:22:37.368 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 16 k 2024-04-25T11:22:37.369 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 25 k 2024-04-25T11:22:37.369 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T11:22:37.369 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T11:22:37.369 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T11:22:37.369 INFO:teuthology.orchestra.run.smithi090.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T11:22:37.369 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T11:22:37.369 INFO:teuthology.orchestra.run.smithi090.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T11:22:37.369 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T11:22:37.369 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T11:22:37.369 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 158 k 2024-04-25T11:22:37.369 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T11:22:37.369 INFO:teuthology.orchestra.run.smithi090.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T11:22:37.369 INFO:teuthology.orchestra.run.smithi090.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T11:22:37.369 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T11:22:37.370 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T11:22:37.370 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 497 k 2024-04-25T11:22:37.370 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T11:22:37.370 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.2 M 2024-04-25T11:22:37.370 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T11:22:37.370 INFO:teuthology.orchestra.run.smithi090.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T11:22:37.370 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T11:22:37.370 INFO:teuthology.orchestra.run.smithi090.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T11:22:37.370 INFO:teuthology.orchestra.run.smithi090.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T11:22:37.370 INFO:teuthology.orchestra.run.smithi090.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T11:22:37.370 INFO:teuthology.orchestra.run.smithi090.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T11:22:37.370 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T11:22:37.370 INFO:teuthology.orchestra.run.smithi090.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T11:22:37.371 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T11:22:37.371 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T11:22:37.371 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T11:22:37.371 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T11:22:37.371 INFO:teuthology.orchestra.run.smithi090.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T11:22:37.371 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T11:22:37.371 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T11:22:37.371 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T11:22:37.371 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 45 k 2024-04-25T11:22:37.371 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 135 k 2024-04-25T11:22:37.371 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T11:22:37.371 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T11:22:37.371 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T11:22:37.371 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T11:22:37.372 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T11:22:37.372 INFO:teuthology.orchestra.run.smithi090.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T11:22:37.372 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T11:22:37.372 INFO:teuthology.orchestra.run.smithi090.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T11:22:37.372 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T11:22:37.372 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T11:22:37.372 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T11:22:37.372 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T11:22:37.372 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T11:22:37.372 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T11:22:37.372 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T11:22:37.372 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T11:22:37.372 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T11:22:37.372 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T11:22:37.373 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T11:22:37.373 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T11:22:37.373 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T11:22:37.373 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T11:22:37.373 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T11:22:37.373 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T11:22:37.373 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T11:22:37.373 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T11:22:37.373 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T11:22:37.373 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T11:22:37.373 INFO:teuthology.orchestra.run.smithi090.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T11:22:37.373 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T11:22:37.373 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T11:22:37.373 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T11:22:37.373 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T11:22:37.374 INFO:teuthology.orchestra.run.smithi090.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T11:22:37.374 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T11:22:37.374 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T11:22:37.374 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T11:22:37.374 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T11:22:37.374 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T11:22:37.374 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T11:22:37.374 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T11:22:37.374 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T11:22:37.374 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T11:22:37.374 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T11:22:37.374 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T11:22:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T11:22:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T11:22:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T11:22:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T11:22:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T11:22:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T11:22:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T11:22:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T11:22:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T11:22:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T11:22:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T11:22:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T11:22:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T11:22:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T11:22:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T11:22:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T11:22:37.376 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T11:22:37.376 INFO:teuthology.orchestra.run.smithi090.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T11:22:37.376 INFO:teuthology.orchestra.run.smithi090.stdout:Installing weak dependencies: 2024-04-25T11:22:37.376 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T11:22:37.376 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:22:37.376 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-04-25T11:22:37.376 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2024-04-25T11:22:37.376 INFO:teuthology.orchestra.run.smithi090.stdout:Install 148 Packages 2024-04-25T11:22:37.376 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrade 4 Packages 2024-04-25T11:22:37.376 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:22:37.377 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 214 M 2024-04-25T11:22:37.377 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-04-25T11:22:37.765 INFO:teuthology.orchestra.run.smithi138.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T11:22:37.766 INFO:teuthology.orchestra.run.smithi138.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T11:22:37.777 INFO:teuthology.orchestra.run.smithi138.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T11:22:37.778 INFO:teuthology.orchestra.run.smithi138.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T11:22:37.779 INFO:teuthology.orchestra.run.smithi138.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T11:22:37.779 INFO:teuthology.orchestra.run.smithi138.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T11:22:37.781 INFO:teuthology.orchestra.run.smithi138.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T11:22:37.783 INFO:teuthology.orchestra.run.smithi138.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T11:22:37.784 INFO:teuthology.orchestra.run.smithi138.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T11:22:37.845 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-25T11:22:37.862 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================= 2024-04-25T11:22:37.862 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-04-25T11:22:37.862 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================= 2024-04-25T11:22:37.862 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-04-25T11:22:37.862 INFO:teuthology.orchestra.run.smithi138.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T11:22:37.862 INFO:teuthology.orchestra.run.smithi138.stdout: ceph x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 6.6 k 2024-04-25T11:22:37.862 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-base x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.4 M 2024-04-25T11:22:37.862 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 863 k 2024-04-25T11:22:37.863 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 147 k 2024-04-25T11:22:37.863 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 1.5 M 2024-04-25T11:22:37.863 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 142 k 2024-04-25T11:22:37.863 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 3.5 M 2024-04-25T11:22:37.863 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 7.4 M 2024-04-25T11:22:37.863 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-rook noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 49 k 2024-04-25T11:22:37.863 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-radosgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 10 M 2024-04-25T11:22:37.863 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-test x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 43 M 2024-04-25T11:22:37.863 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-volume noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 273 k 2024-04-25T11:22:37.863 INFO:teuthology.orchestra.run.smithi138.stdout: cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 762 k 2024-04-25T11:22:37.863 INFO:teuthology.orchestra.run.smithi138.stdout: cephfs-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 224 k 2024-04-25T11:22:37.863 INFO:teuthology.orchestra.run.smithi138.stdout: cephfs-shell x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 27 k 2024-04-25T11:22:37.864 INFO:teuthology.orchestra.run.smithi138.stdout: cephfs-top noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 18 k 2024-04-25T11:22:37.864 INFO:teuthology.orchestra.run.smithi138.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T11:22:37.864 INFO:teuthology.orchestra.run.smithi138.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T11:22:37.864 INFO:teuthology.orchestra.run.smithi138.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T11:22:37.864 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 32 k 2024-04-25T11:22:37.864 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 756 k 2024-04-25T11:22:37.864 INFO:teuthology.orchestra.run.smithi138.stdout: librados-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 127 k 2024-04-25T11:22:37.864 INFO:teuthology.orchestra.run.smithi138.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T11:22:37.864 INFO:teuthology.orchestra.run.smithi138.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T11:22:37.864 INFO:teuthology.orchestra.run.smithi138.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T11:22:37.864 INFO:teuthology.orchestra.run.smithi138.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T11:22:37.864 INFO:teuthology.orchestra.run.smithi138.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T11:22:37.864 INFO:teuthology.orchestra.run.smithi138.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T11:22:37.864 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cephfs x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 161 k 2024-04-25T11:22:37.865 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rados x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 321 k 2024-04-25T11:22:37.865 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 298 k 2024-04-25T11:22:37.865 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 100 k 2024-04-25T11:22:37.865 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 86 k 2024-04-25T11:22:37.865 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.0 M 2024-04-25T11:22:37.865 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-nbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 169 k 2024-04-25T11:22:37.865 INFO:teuthology.orchestra.run.smithi138.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T11:22:37.865 INFO:teuthology.orchestra.run.smithi138.stdout:Upgrading: 2024-04-25T11:22:37.865 INFO:teuthology.orchestra.run.smithi138.stdout: librados2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.3 M 2024-04-25T11:22:37.865 INFO:teuthology.orchestra.run.smithi138.stdout: librbd1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.1 M 2024-04-25T11:22:37.865 INFO:teuthology.orchestra.run.smithi138.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T11:22:37.865 INFO:teuthology.orchestra.run.smithi138.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T11:22:37.865 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-04-25T11:22:37.866 INFO:teuthology.orchestra.run.smithi138.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T11:22:37.866 INFO:teuthology.orchestra.run.smithi138.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T11:22:37.866 INFO:teuthology.orchestra.run.smithi138.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T11:22:37.866 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 20 M 2024-04-25T11:22:37.866 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 34 k 2024-04-25T11:22:37.866 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mds x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 2.3 M 2024-04-25T11:22:37.866 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 249 k 2024-04-25T11:22:37.866 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mon x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 4.9 M 2024-04-25T11:22:37.866 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-osd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 18 M 2024-04-25T11:22:37.866 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 16 k 2024-04-25T11:22:37.866 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-selinux x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 25 k 2024-04-25T11:22:37.866 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T11:22:37.866 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T11:22:37.866 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T11:22:37.867 INFO:teuthology.orchestra.run.smithi138.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T11:22:37.867 INFO:teuthology.orchestra.run.smithi138.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T11:22:37.867 INFO:teuthology.orchestra.run.smithi138.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T11:22:37.867 INFO:teuthology.orchestra.run.smithi138.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T11:22:37.867 INFO:teuthology.orchestra.run.smithi138.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T11:22:37.867 INFO:teuthology.orchestra.run.smithi138.stdout: libcephsqlite x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 158 k 2024-04-25T11:22:37.867 INFO:teuthology.orchestra.run.smithi138.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T11:22:37.867 INFO:teuthology.orchestra.run.smithi138.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T11:22:37.867 INFO:teuthology.orchestra.run.smithi138.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T11:22:37.867 INFO:teuthology.orchestra.run.smithi138.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T11:22:37.867 INFO:teuthology.orchestra.run.smithi138.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T11:22:37.867 INFO:teuthology.orchestra.run.smithi138.stdout: libradosstriper1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 497 k 2024-04-25T11:22:37.867 INFO:teuthology.orchestra.run.smithi138.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T11:22:37.868 INFO:teuthology.orchestra.run.smithi138.stdout: librgw2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.2 M 2024-04-25T11:22:37.868 INFO:teuthology.orchestra.run.smithi138.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T11:22:37.868 INFO:teuthology.orchestra.run.smithi138.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T11:22:37.868 INFO:teuthology.orchestra.run.smithi138.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T11:22:37.868 INFO:teuthology.orchestra.run.smithi138.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T11:22:37.868 INFO:teuthology.orchestra.run.smithi138.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T11:22:37.868 INFO:teuthology.orchestra.run.smithi138.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T11:22:37.868 INFO:teuthology.orchestra.run.smithi138.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T11:22:37.868 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T11:22:37.868 INFO:teuthology.orchestra.run.smithi138.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T11:22:37.868 INFO:teuthology.orchestra.run.smithi138.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T11:22:37.868 INFO:teuthology.orchestra.run.smithi138.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T11:22:37.868 INFO:teuthology.orchestra.run.smithi138.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T11:22:37.868 INFO:teuthology.orchestra.run.smithi138.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T11:22:37.868 INFO:teuthology.orchestra.run.smithi138.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T11:22:37.869 INFO:teuthology.orchestra.run.smithi138.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T11:22:37.869 INFO:teuthology.orchestra.run.smithi138.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T11:22:37.869 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T11:22:37.869 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-argparse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 45 k 2024-04-25T11:22:37.869 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 135 k 2024-04-25T11:22:37.869 INFO:teuthology.orchestra.run.smithi138.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T11:22:37.869 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T11:22:37.869 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T11:22:37.869 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T11:22:37.869 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T11:22:37.869 INFO:teuthology.orchestra.run.smithi138.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T11:22:37.869 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T11:22:37.869 INFO:teuthology.orchestra.run.smithi138.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T11:22:37.869 INFO:teuthology.orchestra.run.smithi138.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T11:22:37.870 INFO:teuthology.orchestra.run.smithi138.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T11:22:37.870 INFO:teuthology.orchestra.run.smithi138.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T11:22:37.870 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T11:22:37.870 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T11:22:37.870 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T11:22:37.870 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T11:22:37.870 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T11:22:37.870 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T11:22:37.870 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T11:22:37.870 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T11:22:37.870 INFO:teuthology.orchestra.run.smithi138.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T11:22:37.870 INFO:teuthology.orchestra.run.smithi138.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T11:22:37.870 INFO:teuthology.orchestra.run.smithi138.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T11:22:37.870 INFO:teuthology.orchestra.run.smithi138.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T11:22:37.871 INFO:teuthology.orchestra.run.smithi138.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T11:22:37.871 INFO:teuthology.orchestra.run.smithi138.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T11:22:37.871 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T11:22:37.871 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T11:22:37.871 INFO:teuthology.orchestra.run.smithi138.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T11:22:37.871 INFO:teuthology.orchestra.run.smithi138.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T11:22:37.871 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T11:22:37.871 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T11:22:37.871 INFO:teuthology.orchestra.run.smithi138.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T11:22:37.871 INFO:teuthology.orchestra.run.smithi138.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T11:22:37.871 INFO:teuthology.orchestra.run.smithi138.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T11:22:37.871 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T11:22:37.871 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T11:22:37.871 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T11:22:37.871 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T11:22:37.872 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T11:22:37.872 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T11:22:37.872 INFO:teuthology.orchestra.run.smithi138.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T11:22:37.872 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T11:22:37.872 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T11:22:37.872 INFO:teuthology.orchestra.run.smithi138.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T11:22:37.872 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T11:22:37.872 INFO:teuthology.orchestra.run.smithi138.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T11:22:37.872 INFO:teuthology.orchestra.run.smithi138.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T11:22:37.872 INFO:teuthology.orchestra.run.smithi138.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T11:22:37.872 INFO:teuthology.orchestra.run.smithi138.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T11:22:37.872 INFO:teuthology.orchestra.run.smithi138.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T11:22:37.872 INFO:teuthology.orchestra.run.smithi138.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T11:22:37.872 INFO:teuthology.orchestra.run.smithi138.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T11:22:37.873 INFO:teuthology.orchestra.run.smithi138.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T11:22:37.873 INFO:teuthology.orchestra.run.smithi138.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T11:22:37.873 INFO:teuthology.orchestra.run.smithi138.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T11:22:37.873 INFO:teuthology.orchestra.run.smithi138.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T11:22:37.873 INFO:teuthology.orchestra.run.smithi138.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T11:22:37.873 INFO:teuthology.orchestra.run.smithi138.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T11:22:37.873 INFO:teuthology.orchestra.run.smithi138.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T11:22:37.873 INFO:teuthology.orchestra.run.smithi138.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T11:22:37.873 INFO:teuthology.orchestra.run.smithi138.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T11:22:37.873 INFO:teuthology.orchestra.run.smithi138.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T11:22:37.873 INFO:teuthology.orchestra.run.smithi138.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T11:22:37.873 INFO:teuthology.orchestra.run.smithi138.stdout:Installing weak dependencies: 2024-04-25T11:22:37.873 INFO:teuthology.orchestra.run.smithi138.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T11:22:37.873 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-25T11:22:37.873 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-04-25T11:22:37.874 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================= 2024-04-25T11:22:37.874 INFO:teuthology.orchestra.run.smithi138.stdout:Install 148 Packages 2024-04-25T11:22:37.874 INFO:teuthology.orchestra.run.smithi138.stdout:Upgrade 4 Packages 2024-04-25T11:22:37.874 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-25T11:22:37.874 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 214 M 2024-04-25T11:22:37.874 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-04-25T11:22:38.193 INFO:teuthology.orchestra.run.smithi090.stdout:(1/152): ceph-19.0.0-3291.g994e2f20.el9.x86_64. 37 kB/s | 6.6 kB 00:00 2024-04-25T11:22:38.418 INFO:teuthology.orchestra.run.smithi090.stdout:(2/152): ceph-fuse-19.0.0-3291.g994e2f20.el9.x8 3.7 MB/s | 863 kB 00:00 2024-04-25T11:22:38.469 INFO:teuthology.orchestra.run.smithi090.stdout:(3/152): ceph-immutable-object-cache-19.0.0-329 2.9 MB/s | 147 kB 00:00 2024-04-25T11:22:38.778 INFO:teuthology.orchestra.run.smithi090.stdout:(4/152): ceph-base-19.0.0-3291.g994e2f20.el9.x8 7.1 MB/s | 5.4 MB 00:00 2024-04-25T11:22:38.836 INFO:teuthology.orchestra.run.smithi090.stdout:(5/152): ceph-mds-19.0.0-3291.g994e2f20.el9.x86 6.2 MB/s | 2.3 MB 00:00 2024-04-25T11:22:38.961 INFO:teuthology.orchestra.run.smithi090.stdout:(6/152): ceph-mgr-19.0.0-3291.g994e2f20.el9.x86 8.2 MB/s | 1.5 MB 00:00 2024-04-25T11:22:39.223 INFO:teuthology.orchestra.run.smithi138.stdout:(1/152): ceph-19.0.0-3291.g994e2f20.el9.x86_64. 14 kB/s | 6.6 kB 00:00 2024-04-25T11:22:39.504 INFO:teuthology.orchestra.run.smithi090.stdout:(7/152): ceph-mon-19.0.0-3291.g994e2f20.el9.x86 7.3 MB/s | 4.9 MB 00:00 2024-04-25T11:22:40.610 INFO:teuthology.orchestra.run.smithi138.stdout:(2/152): ceph-fuse-19.0.0-3291.g994e2f20.el9.x8 622 kB/s | 863 kB 00:01 2024-04-25T11:22:40.815 INFO:teuthology.orchestra.run.smithi138.stdout:(3/152): ceph-immutable-object-cache-19.0.0-329 716 kB/s | 147 kB 00:00 2024-04-25T11:22:40.887 INFO:teuthology.orchestra.run.smithi090.stdout:(8/152): ceph-osd-19.0.0-3291.g994e2f20.el9.x86 9.2 MB/s | 18 MB 00:01 2024-04-25T11:22:40.979 INFO:teuthology.orchestra.run.smithi156.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-25T11:22:41.049 INFO:teuthology.orchestra.run.smithi138.stdout:(4/152): ceph-base-19.0.0-3291.g994e2f20.el9.x8 2.3 MB/s | 5.4 MB 00:02 2024-04-25T11:22:41.122 INFO:teuthology.orchestra.run.smithi090.stdout:(9/152): ceph-common-19.0.0-3291.g994e2f20.el9. 6.3 MB/s | 20 MB 00:03 2024-04-25T11:22:41.147 INFO:teuthology.orchestra.run.smithi090.stdout:(10/152): ceph-selinux-19.0.0-3291.g994e2f20.el 97 kB/s | 25 kB 00:00 2024-04-25T11:22:41.289 INFO:teuthology.orchestra.run.smithi090.stdout:(11/152): ceph-radosgw-19.0.0-3291.g994e2f20.el 5.7 MB/s | 10 MB 00:01 2024-04-25T11:22:41.291 INFO:teuthology.orchestra.run.smithi138.stdout:(5/152): ceph-mgr-19.0.0-3291.g994e2f20.el9.x86 6.2 MB/s | 1.5 MB 00:00 2024-04-25T11:22:41.323 INFO:teuthology.orchestra.run.smithi090.stdout:(12/152): cephfs-mirror-19.0.0-3291.g994e2f20.e 1.2 MB/s | 224 kB 00:00 2024-04-25T11:22:41.348 INFO:teuthology.orchestra.run.smithi090.stdout:(13/152): cephfs-shell-19.0.0-3291.g994e2f20.el 463 kB/s | 27 kB 00:00 2024-04-25T11:22:41.382 INFO:teuthology.orchestra.run.smithi090.stdout:(14/152): libcephfs-devel-19.0.0-3291.g994e2f20 544 kB/s | 32 kB 00:00 2024-04-25T11:22:41.474 INFO:teuthology.orchestra.run.smithi090.stdout:(15/152): libcephsqlite-19.0.0-3291.g994e2f20.e 1.7 MB/s | 158 kB 00:00 2024-04-25T11:22:41.507 INFO:teuthology.orchestra.run.smithi090.stdout:(16/152): libcephfs2-19.0.0-3291.g994e2f20.el9. 4.6 MB/s | 756 kB 00:00 2024-04-25T11:22:41.541 INFO:teuthology.orchestra.run.smithi090.stdout:(17/152): librados-devel-19.0.0-3291.g994e2f20. 1.9 MB/s | 127 kB 00:00 2024-04-25T11:22:41.633 INFO:teuthology.orchestra.run.smithi090.stdout:(18/152): libradosstriper1-19.0.0-3291.g994e2f2 3.9 MB/s | 497 kB 00:00 2024-04-25T11:22:41.700 INFO:teuthology.orchestra.run.smithi138.stdout:(6/152): ceph-mds-19.0.0-3291.g994e2f20.el9.x86 2.6 MB/s | 2.3 MB 00:00 2024-04-25T11:22:41.708 INFO:teuthology.orchestra.run.smithi090.stdout:(19/152): python3-ceph-argparse-19.0.0-3291.g99 601 kB/s | 45 kB 00:00 2024-04-25T11:22:41.785 INFO:teuthology.orchestra.run.smithi090.stdout:(20/152): python3-ceph-common-19.0.0-3291.g994e 1.7 MB/s | 135 kB 00:00 2024-04-25T11:22:41.860 INFO:teuthology.orchestra.run.smithi090.stdout:(21/152): python3-cephfs-19.0.0-3291.g994e2f20. 2.1 MB/s | 161 kB 00:00 2024-04-25T11:22:41.952 INFO:teuthology.orchestra.run.smithi090.stdout:(22/152): python3-rados-19.0.0-3291.g994e2f20.e 3.4 MB/s | 321 kB 00:00 2024-04-25T11:22:42.044 INFO:teuthology.orchestra.run.smithi090.stdout:(23/152): python3-rbd-19.0.0-3291.g994e2f20.el9 3.2 MB/s | 298 kB 00:00 2024-04-25T11:22:42.127 INFO:teuthology.orchestra.run.smithi090.stdout:(24/152): python3-rgw-19.0.0-3291.g994e2f20.el9 1.2 MB/s | 100 kB 00:00 2024-04-25T11:22:42.211 INFO:teuthology.orchestra.run.smithi090.stdout:(25/152): rbd-fuse-19.0.0-3291.g994e2f20.el9.x8 1.0 MB/s | 86 kB 00:00 2024-04-25T11:22:42.536 INFO:teuthology.orchestra.run.smithi090.stdout:(26/152): librgw2-19.0.0-3291.g994e2f20.el9.x86 5.2 MB/s | 5.2 MB 00:00 2024-04-25T11:22:42.629 INFO:teuthology.orchestra.run.smithi090.stdout:(27/152): rbd-nbd-19.0.0-3291.g994e2f20.el9.x86 1.8 MB/s | 169 kB 00:00 2024-04-25T11:22:42.650 INFO:teuthology.orchestra.run.smithi138.stdout:(7/152): ceph-mon-19.0.0-3291.g994e2f20.el9.x86 3.6 MB/s | 4.9 MB 00:01 2024-04-25T11:22:42.695 INFO:teuthology.orchestra.run.smithi090.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3291.g 515 kB/s | 34 kB 00:00 2024-04-25T11:22:42.812 INFO:teuthology.orchestra.run.smithi090.stdout:(29/152): rbd-mirror-19.0.0-3291.g994e2f20.el9. 5.1 MB/s | 3.0 MB 00:00 2024-04-25T11:22:42.846 INFO:teuthology.orchestra.run.smithi090.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3291.g994e2f2 945 kB/s | 142 kB 00:00 2024-04-25T11:22:43.106 INFO:teuthology.orchestra.run.smithi156.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T11:22:43.106 INFO:teuthology.orchestra.run.smithi156.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T11:22:43.117 INFO:teuthology.orchestra.run.smithi156.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T11:22:43.118 INFO:teuthology.orchestra.run.smithi156.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T11:22:43.118 INFO:teuthology.orchestra.run.smithi156.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T11:22:43.119 INFO:teuthology.orchestra.run.smithi156.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T11:22:43.121 INFO:teuthology.orchestra.run.smithi156.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T11:22:43.123 INFO:teuthology.orchestra.run.smithi156.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T11:22:43.124 INFO:teuthology.orchestra.run.smithi156.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T11:22:43.185 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-04-25T11:22:43.199 INFO:teuthology.orchestra.run.smithi156.stdout:======================================================================================= 2024-04-25T11:22:43.199 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-04-25T11:22:43.200 INFO:teuthology.orchestra.run.smithi156.stdout:======================================================================================= 2024-04-25T11:22:43.200 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-04-25T11:22:43.200 INFO:teuthology.orchestra.run.smithi156.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T11:22:43.200 INFO:teuthology.orchestra.run.smithi156.stdout: ceph x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 6.6 k 2024-04-25T11:22:43.200 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.4 M 2024-04-25T11:22:43.200 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 863 k 2024-04-25T11:22:43.200 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 147 k 2024-04-25T11:22:43.200 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 1.5 M 2024-04-25T11:22:43.200 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 142 k 2024-04-25T11:22:43.200 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 3.5 M 2024-04-25T11:22:43.200 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 7.4 M 2024-04-25T11:22:43.200 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 49 k 2024-04-25T11:22:43.201 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 10 M 2024-04-25T11:22:43.201 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 43 M 2024-04-25T11:22:43.201 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 273 k 2024-04-25T11:22:43.201 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 762 k 2024-04-25T11:22:43.201 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 224 k 2024-04-25T11:22:43.201 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-shell x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 27 k 2024-04-25T11:22:43.201 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-top noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 18 k 2024-04-25T11:22:43.201 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T11:22:43.201 INFO:teuthology.orchestra.run.smithi156.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T11:22:43.201 INFO:teuthology.orchestra.run.smithi156.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T11:22:43.201 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 32 k 2024-04-25T11:22:43.201 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 756 k 2024-04-25T11:22:43.202 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 127 k 2024-04-25T11:22:43.202 INFO:teuthology.orchestra.run.smithi156.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T11:22:43.202 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T11:22:43.202 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T11:22:43.202 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T11:22:43.202 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T11:22:43.202 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T11:22:43.202 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 161 k 2024-04-25T11:22:43.202 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 321 k 2024-04-25T11:22:43.202 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 298 k 2024-04-25T11:22:43.202 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 100 k 2024-04-25T11:22:43.203 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 86 k 2024-04-25T11:22:43.203 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.0 M 2024-04-25T11:22:43.203 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 169 k 2024-04-25T11:22:43.203 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T11:22:43.203 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrading: 2024-04-25T11:22:43.203 INFO:teuthology.orchestra.run.smithi156.stdout: librados2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.3 M 2024-04-25T11:22:43.203 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.1 M 2024-04-25T11:22:43.203 INFO:teuthology.orchestra.run.smithi156.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T11:22:43.203 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T11:22:43.203 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-04-25T11:22:43.203 INFO:teuthology.orchestra.run.smithi156.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T11:22:43.203 INFO:teuthology.orchestra.run.smithi156.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T11:22:43.203 INFO:teuthology.orchestra.run.smithi156.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T11:22:43.204 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 20 M 2024-04-25T11:22:43.204 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 34 k 2024-04-25T11:22:43.204 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 2.3 M 2024-04-25T11:22:43.204 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 249 k 2024-04-25T11:22:43.204 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 4.9 M 2024-04-25T11:22:43.204 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 18 M 2024-04-25T11:22:43.204 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 16 k 2024-04-25T11:22:43.204 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 25 k 2024-04-25T11:22:43.204 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T11:22:43.204 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T11:22:43.204 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T11:22:43.205 INFO:teuthology.orchestra.run.smithi156.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T11:22:43.205 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T11:22:43.205 INFO:teuthology.orchestra.run.smithi156.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T11:22:43.205 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T11:22:43.205 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T11:22:43.205 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 158 k 2024-04-25T11:22:43.205 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T11:22:43.205 INFO:teuthology.orchestra.run.smithi156.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T11:22:43.205 INFO:teuthology.orchestra.run.smithi156.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T11:22:43.205 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T11:22:43.205 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T11:22:43.205 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 497 k 2024-04-25T11:22:43.206 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T11:22:43.206 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.2 M 2024-04-25T11:22:43.206 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T11:22:43.206 INFO:teuthology.orchestra.run.smithi156.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T11:22:43.206 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T11:22:43.206 INFO:teuthology.orchestra.run.smithi156.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T11:22:43.206 INFO:teuthology.orchestra.run.smithi156.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T11:22:43.206 INFO:teuthology.orchestra.run.smithi156.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T11:22:43.206 INFO:teuthology.orchestra.run.smithi156.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T11:22:43.206 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T11:22:43.206 INFO:teuthology.orchestra.run.smithi156.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T11:22:43.207 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T11:22:43.207 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T11:22:43.207 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T11:22:43.207 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T11:22:43.207 INFO:teuthology.orchestra.run.smithi156.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T11:22:43.207 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T11:22:43.207 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T11:22:43.207 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T11:22:43.207 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 45 k 2024-04-25T11:22:43.207 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 135 k 2024-04-25T11:22:43.207 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T11:22:43.207 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T11:22:43.207 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T11:22:43.208 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T11:22:43.208 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T11:22:43.208 INFO:teuthology.orchestra.run.smithi156.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T11:22:43.208 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T11:22:43.208 INFO:teuthology.orchestra.run.smithi156.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T11:22:43.208 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T11:22:43.208 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T11:22:43.208 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T11:22:43.208 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T11:22:43.208 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T11:22:43.208 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T11:22:43.208 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T11:22:43.208 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T11:22:43.208 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T11:22:43.208 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T11:22:43.209 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T11:22:43.209 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T11:22:43.209 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T11:22:43.209 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T11:22:43.209 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T11:22:43.209 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T11:22:43.209 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T11:22:43.209 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T11:22:43.209 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T11:22:43.209 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T11:22:43.209 INFO:teuthology.orchestra.run.smithi156.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T11:22:43.209 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T11:22:43.209 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T11:22:43.210 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T11:22:43.210 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T11:22:43.210 INFO:teuthology.orchestra.run.smithi156.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T11:22:43.210 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T11:22:43.210 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T11:22:43.210 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T11:22:43.210 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T11:22:43.210 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T11:22:43.210 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T11:22:43.210 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T11:22:43.210 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T11:22:43.210 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T11:22:43.211 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T11:22:43.211 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T11:22:43.211 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T11:22:43.211 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T11:22:43.211 INFO:teuthology.orchestra.run.smithi156.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T11:22:43.211 INFO:teuthology.orchestra.run.smithi156.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T11:22:43.211 INFO:teuthology.orchestra.run.smithi156.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T11:22:43.211 INFO:teuthology.orchestra.run.smithi156.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T11:22:43.211 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T11:22:43.211 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T11:22:43.211 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T11:22:43.211 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T11:22:43.211 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T11:22:43.212 INFO:teuthology.orchestra.run.smithi156.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T11:22:43.212 INFO:teuthology.orchestra.run.smithi156.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T11:22:43.212 INFO:teuthology.orchestra.run.smithi156.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T11:22:43.212 INFO:teuthology.orchestra.run.smithi156.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T11:22:43.212 INFO:teuthology.orchestra.run.smithi156.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T11:22:43.212 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T11:22:43.212 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T11:22:43.212 INFO:teuthology.orchestra.run.smithi156.stdout:Installing weak dependencies: 2024-04-25T11:22:43.212 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T11:22:43.212 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T11:22:43.212 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-04-25T11:22:43.212 INFO:teuthology.orchestra.run.smithi156.stdout:======================================================================================= 2024-04-25T11:22:43.213 INFO:teuthology.orchestra.run.smithi156.stdout:Install 148 Packages 2024-04-25T11:22:43.213 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrade 4 Packages 2024-04-25T11:22:43.213 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T11:22:43.213 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 214 M 2024-04-25T11:22:43.213 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-04-25T11:22:43.605 INFO:teuthology.orchestra.run.smithi090.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3291.g994e2 4.5 MB/s | 3.5 MB 00:00 2024-04-25T11:22:43.690 INFO:teuthology.orchestra.run.smithi090.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3291.g99 2.9 MB/s | 249 kB 00:00 2024-04-25T11:22:43.765 INFO:teuthology.orchestra.run.smithi090.stdout:(33/152): ceph-mgr-rook-19.0.0-3291.g994e2f20.e 656 kB/s | 49 kB 00:00 2024-04-25T11:22:43.840 INFO:teuthology.orchestra.run.smithi090.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3291.g9 219 kB/s | 16 kB 00:00 2024-04-25T11:22:43.924 INFO:teuthology.orchestra.run.smithi090.stdout:(35/152): ceph-volume-19.0.0-3291.g994e2f20.el9 3.2 MB/s | 273 kB 00:00 2024-04-25T11:22:44.141 INFO:teuthology.orchestra.run.smithi090.stdout:(36/152): cephadm-19.0.0-3291.g994e2f20.el9.noa 3.4 MB/s | 762 kB 00:00 2024-04-25T11:22:44.216 INFO:teuthology.orchestra.run.smithi090.stdout:(37/152): cephfs-top-19.0.0-3291.g994e2f20.el9. 242 kB/s | 18 kB 00:00 2024-04-25T11:22:44.289 INFO:teuthology.orchestra.run.smithi156.stdout:(1/152): ceph-19.0.0-3291.g994e2f20.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-04-25T11:22:44.417 INFO:teuthology.orchestra.run.smithi090.stdout:(38/152): ceph-mgr-diskprediction-local-19.0.0- 4.7 MB/s | 7.4 MB 00:01 2024-04-25T11:22:44.517 INFO:teuthology.orchestra.run.smithi090.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 267 kB/s | 80 kB 00:00 2024-04-25T11:22:44.692 INFO:teuthology.orchestra.run.smithi090.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-04-25T11:22:44.809 INFO:teuthology.orchestra.run.smithi090.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 253 kB 00:00 2024-04-25T11:22:44.843 INFO:teuthology.orchestra.run.smithi090.stdout:(42/152): libgfortran-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 803 kB 00:00 2024-04-25T11:22:44.927 INFO:teuthology.orchestra.run.smithi090.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 106 kB 00:00 2024-04-25T11:22:45.002 INFO:teuthology.orchestra.run.smithi090.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 1.8 MB/s | 135 kB 00:00 2024-04-25T11:22:45.077 INFO:teuthology.orchestra.run.smithi090.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-04-25T11:22:45.153 INFO:teuthology.orchestra.run.smithi090.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 2.8 MB/s | 215 kB 00:00 2024-04-25T11:22:45.286 INFO:teuthology.orchestra.run.smithi090.stdout:(47/152): python3-cryptography-36.0.1-4.el9.x86 2.6 MB/s | 1.2 MB 00:00 2024-04-25T11:22:45.347 INFO:teuthology.orchestra.run.smithi156.stdout:(2/152): ceph-fuse-19.0.0-3291.g994e2f20.el9.x8 815 kB/s | 863 kB 00:01 2024-04-25T11:22:45.448 INFO:teuthology.orchestra.run.smithi156.stdout:(3/152): ceph-base-19.0.0-3291.g994e2f20.el9.x8 3.4 MB/s | 5.4 MB 00:01 2024-04-25T11:22:45.506 INFO:teuthology.orchestra.run.smithi156.stdout:(4/152): ceph-immutable-object-cache-19.0.0-329 926 kB/s | 147 kB 00:00 2024-04-25T11:22:45.541 INFO:teuthology.orchestra.run.smithi138.stdout:(8/152): ceph-radosgw-19.0.0-3291.g994e2f20.el9 3.5 MB/s | 10 MB 00:02 2024-04-25T11:22:45.706 INFO:teuthology.orchestra.run.smithi156.stdout:(5/152): ceph-mds-19.0.0-3291.g994e2f20.el9.x86 8.7 MB/s | 2.3 MB 00:00 2024-04-25T11:22:45.767 INFO:teuthology.orchestra.run.smithi138.stdout:(9/152): ceph-osd-19.0.0-3291.g994e2f20.el9.x86 4.4 MB/s | 18 MB 00:04 2024-04-25T11:22:45.792 INFO:teuthology.orchestra.run.smithi138.stdout:(10/152): ceph-selinux-19.0.0-3291.g994e2f20.el 101 kB/s | 25 kB 00:00 2024-04-25T11:22:45.829 INFO:teuthology.orchestra.run.smithi090.stdout:(48/152): ceph-test-19.0.0-3291.g994e2f20.el9.x 9.2 MB/s | 43 MB 00:04 2024-04-25T11:22:45.950 INFO:teuthology.orchestra.run.smithi138.stdout:(11/152): cephfs-mirror-19.0.0-3291.g994e2f20.e 1.4 MB/s | 224 kB 00:00 2024-04-25T11:22:46.051 INFO:teuthology.orchestra.run.smithi138.stdout:(12/152): cephfs-shell-19.0.0-3291.g994e2f20.el 270 kB/s | 27 kB 00:00 2024-04-25T11:22:46.151 INFO:teuthology.orchestra.run.smithi138.stdout:(13/152): libcephfs-devel-19.0.0-3291.g994e2f20 317 kB/s | 32 kB 00:00 2024-04-25T11:22:46.254 INFO:teuthology.orchestra.run.smithi090.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 252 kB/s | 107 kB 00:00 2024-04-25T11:22:46.398 INFO:teuthology.orchestra.run.smithi156.stdout:(6/152): ceph-mon-19.0.0-3291.g994e2f20.el9.x86 7.1 MB/s | 4.9 MB 00:00 2024-04-25T11:22:46.418 INFO:teuthology.orchestra.run.smithi138.stdout:(14/152): libcephfs2-19.0.0-3291.g994e2f20.el9. 2.8 MB/s | 756 kB 00:00 2024-04-25T11:22:46.440 INFO:teuthology.orchestra.run.smithi156.stdout:(7/152): ceph-mgr-19.0.0-3291.g994e2f20.el9.x86 1.6 MB/s | 1.5 MB 00:00 2024-04-25T11:22:46.493 INFO:teuthology.orchestra.run.smithi138.stdout:(15/152): libcephsqlite-19.0.0-3291.g994e2f20.e 2.1 MB/s | 158 kB 00:00 2024-04-25T11:22:46.512 INFO:teuthology.orchestra.run.smithi090.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 83 kB/s | 21 kB 00:00 2024-04-25T11:22:46.568 INFO:teuthology.orchestra.run.smithi138.stdout:(16/152): librados-devel-19.0.0-3291.g994e2f20. 1.7 MB/s | 127 kB 00:00 2024-04-25T11:22:46.802 INFO:teuthology.orchestra.run.smithi138.stdout:(17/152): libradosstriper1-19.0.0-3291.g994e2f2 2.1 MB/s | 497 kB 00:00 2024-04-25T11:22:47.046 INFO:teuthology.orchestra.run.smithi090.stdout:(51/152): automake-1.16.2-8.el9.noarch.rpm 364 kB/s | 689 kB 00:01 2024-04-25T11:22:47.305 INFO:teuthology.orchestra.run.smithi090.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 126 kB/s | 33 kB 00:00 2024-04-25T11:22:47.447 INFO:teuthology.orchestra.run.smithi090.stdout:(53/152): bison-3.7.4-5.el9.x86_64.rpm 436 kB/s | 943 kB 00:02 2024-04-25T11:22:47.647 INFO:teuthology.orchestra.run.smithi090.stdout:(54/152): flex-2.6.4-9.el9.x86_64.rpm 276 kB/s | 313 kB 00:01 2024-04-25T11:22:47.830 INFO:teuthology.orchestra.run.smithi090.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 47 kB/s | 18 kB 00:00 2024-04-25T11:22:47.856 INFO:teuthology.orchestra.run.smithi090.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 53 kB/s | 11 kB 00:00 2024-04-25T11:22:48.191 INFO:teuthology.orchestra.run.smithi090.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 135 kB/s | 45 kB 00:00 2024-04-25T11:22:48.333 INFO:teuthology.orchestra.run.smithi090.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 319 kB/s | 160 kB 00:00 2024-04-25T11:22:48.415 INFO:teuthology.orchestra.run.smithi156.stdout:(8/152): ceph-common-19.0.0-3291.g994e2f20.el9. 4.3 MB/s | 20 MB 00:04 2024-04-25T11:22:48.524 INFO:teuthology.orchestra.run.smithi156.stdout:(9/152): ceph-selinux-19.0.0-3291.g994e2f20.el9 233 kB/s | 25 kB 00:00 2024-04-25T11:22:48.727 INFO:teuthology.orchestra.run.smithi138.stdout:(18/152): librgw2-19.0.0-3291.g994e2f20.el9.x86 2.7 MB/s | 5.2 MB 00:01 2024-04-25T11:22:48.778 INFO:teuthology.orchestra.run.smithi138.stdout:(19/152): python3-ceph-argparse-19.0.0-3291.g99 908 kB/s | 45 kB 00:00 2024-04-25T11:22:48.791 INFO:teuthology.orchestra.run.smithi156.stdout:(10/152): ceph-osd-19.0.0-3291.g994e2f20.el9.x8 7.4 MB/s | 18 MB 00:02 2024-04-25T11:22:48.828 INFO:teuthology.orchestra.run.smithi138.stdout:(20/152): python3-ceph-common-19.0.0-3291.g994e 2.6 MB/s | 135 kB 00:00 2024-04-25T11:22:48.857 INFO:teuthology.orchestra.run.smithi156.stdout:(11/152): cephfs-mirror-19.0.0-3291.g994e2f20.e 3.3 MB/s | 224 kB 00:00 2024-04-25T11:22:48.895 INFO:teuthology.orchestra.run.smithi138.stdout:(21/152): python3-cephfs-19.0.0-3291.g994e2f20. 2.4 MB/s | 161 kB 00:00 2024-04-25T11:22:48.933 INFO:teuthology.orchestra.run.smithi156.stdout:(12/152): cephfs-shell-19.0.0-3291.g994e2f20.el 361 kB/s | 27 kB 00:00 2024-04-25T11:22:49.009 INFO:teuthology.orchestra.run.smithi156.stdout:(13/152): libcephfs-devel-19.0.0-3291.g994e2f20 421 kB/s | 32 kB 00:00 2024-04-25T11:22:49.037 INFO:teuthology.orchestra.run.smithi138.stdout:(22/152): python3-rados-19.0.0-3291.g994e2f20.e 2.2 MB/s | 321 kB 00:00 2024-04-25T11:22:49.175 INFO:teuthology.orchestra.run.smithi156.stdout:(14/152): libcephfs2-19.0.0-3291.g994e2f20.el9. 4.4 MB/s | 756 kB 00:00 2024-04-25T11:22:49.195 INFO:teuthology.orchestra.run.smithi138.stdout:(23/152): python3-rbd-19.0.0-3291.g994e2f20.el9 1.8 MB/s | 298 kB 00:00 2024-04-25T11:22:49.251 INFO:teuthology.orchestra.run.smithi156.stdout:(15/152): libcephsqlite-19.0.0-3291.g994e2f20.e 2.1 MB/s | 158 kB 00:00 2024-04-25T11:22:49.279 INFO:teuthology.orchestra.run.smithi138.stdout:(24/152): python3-rgw-19.0.0-3291.g994e2f20.el9 1.2 MB/s | 100 kB 00:00 2024-04-25T11:22:49.326 INFO:teuthology.orchestra.run.smithi156.stdout:(16/152): librados-devel-19.0.0-3291.g994e2f20. 1.7 MB/s | 127 kB 00:00 2024-04-25T11:22:49.354 INFO:teuthology.orchestra.run.smithi138.stdout:(25/152): rbd-fuse-19.0.0-3291.g994e2f20.el9.x8 1.1 MB/s | 86 kB 00:00 2024-04-25T11:22:49.418 INFO:teuthology.orchestra.run.smithi156.stdout:(17/152): libradosstriper1-19.0.0-3291.g994e2f2 5.3 MB/s | 497 kB 00:00 2024-04-25T11:22:49.625 INFO:teuthology.orchestra.run.smithi090.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 462 kB/s | 662 kB 00:01 2024-04-25T11:22:49.842 INFO:teuthology.orchestra.run.smithi090.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 386 kB/s | 582 kB 00:01 2024-04-25T11:22:49.893 INFO:teuthology.orchestra.run.smithi090.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 95 kB/s | 25 kB 00:00 2024-04-25T11:22:50.184 INFO:teuthology.orchestra.run.smithi090.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 139 kB/s | 47 kB 00:00 2024-04-25T11:22:50.268 INFO:teuthology.orchestra.run.smithi156.stdout:(18/152): librgw2-19.0.0-3291.g994e2f20.el9.x86 6.1 MB/s | 5.2 MB 00:00 2024-04-25T11:22:50.335 INFO:teuthology.orchestra.run.smithi156.stdout:(19/152): python3-ceph-argparse-19.0.0-3291.g99 679 kB/s | 45 kB 00:00 2024-04-25T11:22:50.393 INFO:teuthology.orchestra.run.smithi090.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 584 kB/s | 292 kB 00:00 2024-04-25T11:22:50.402 INFO:teuthology.orchestra.run.smithi156.stdout:(20/152): python3-ceph-common-19.0.0-3291.g994e 2.0 MB/s | 135 kB 00:00 2024-04-25T11:22:50.404 INFO:teuthology.orchestra.run.smithi138.stdout:(26/152): rbd-mirror-19.0.0-3291.g994e2f20.el9. 2.9 MB/s | 3.0 MB 00:01 2024-04-25T11:22:50.461 INFO:teuthology.orchestra.run.smithi156.stdout:(21/152): python3-cephfs-19.0.0-3291.g994e2f20. 2.7 MB/s | 161 kB 00:00 2024-04-25T11:22:50.545 INFO:teuthology.orchestra.run.smithi156.stdout:(22/152): python3-rados-19.0.0-3291.g994e2f20.e 3.8 MB/s | 321 kB 00:00 2024-04-25T11:22:50.551 INFO:teuthology.orchestra.run.smithi090.stdout:(64/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 943 kB/s | 3.0 MB 00:03 2024-04-25T11:22:50.596 INFO:teuthology.orchestra.run.smithi138.stdout:(27/152): ceph-common-19.0.0-3291.g994e2f20.el9 1.7 MB/s | 20 MB 00:11 2024-04-25T11:22:50.611 INFO:teuthology.orchestra.run.smithi156.stdout:(23/152): python3-rbd-19.0.0-3291.g994e2f20.el9 4.4 MB/s | 298 kB 00:00 2024-04-25T11:22:50.622 INFO:teuthology.orchestra.run.smithi138.stdout:(28/152): rbd-nbd-19.0.0-3291.g994e2f20.el9.x86 783 kB/s | 169 kB 00:00 2024-04-25T11:22:50.672 INFO:teuthology.orchestra.run.smithi156.stdout:(24/152): python3-rgw-19.0.0-3291.g994e2f20.el9 1.6 MB/s | 100 kB 00:00 2024-04-25T11:22:50.677 INFO:teuthology.orchestra.run.smithi090.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 124 kB/s | 35 kB 00:00 2024-04-25T11:22:50.688 INFO:teuthology.orchestra.run.smithi138.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3291.g994e2f2 2.1 MB/s | 142 kB 00:00 2024-04-25T11:22:50.705 INFO:teuthology.orchestra.run.smithi138.stdout:(30/152): ceph-grafana-dashboards-19.0.0-3291.g 315 kB/s | 34 kB 00:00 2024-04-25T11:22:50.714 INFO:teuthology.orchestra.run.smithi156.stdout:(25/152): rbd-fuse-19.0.0-3291.g994e2f20.el9.x8 2.0 MB/s | 86 kB 00:00 2024-04-25T11:22:50.852 INFO:teuthology.orchestra.run.smithi090.stdout:(66/152): lua-5.4.4-4.el9.x86_64.rpm 282 kB/s | 188 kB 00:00 2024-04-25T11:22:51.006 INFO:teuthology.orchestra.run.smithi156.stdout:(26/152): ceph-radosgw-19.0.0-3291.g994e2f20.el 2.2 MB/s | 10 MB 00:04 2024-04-25T11:22:51.125 INFO:teuthology.orchestra.run.smithi156.stdout:(27/152): rbd-nbd-19.0.0-3291.g994e2f20.el9.x86 1.4 MB/s | 169 kB 00:00 2024-04-25T11:22:51.198 INFO:teuthology.orchestra.run.smithi156.stdout:(28/152): rbd-mirror-19.0.0-3291.g994e2f20.el9. 6.3 MB/s | 3.0 MB 00:00 2024-04-25T11:22:51.223 INFO:teuthology.orchestra.run.smithi156.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3291.g 341 kB/s | 34 kB 00:00 2024-04-25T11:22:51.274 INFO:teuthology.orchestra.run.smithi156.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3291.g994e2f2 1.8 MB/s | 142 kB 00:00 2024-04-25T11:22:51.698 INFO:teuthology.orchestra.run.smithi138.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3291.g994e2 3.5 MB/s | 3.5 MB 00:01 2024-04-25T11:22:51.840 INFO:teuthology.orchestra.run.smithi138.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3291.g99 1.7 MB/s | 249 kB 00:00 2024-04-25T11:22:51.915 INFO:teuthology.orchestra.run.smithi138.stdout:(33/152): ceph-mgr-rook-19.0.0-3291.g994e2f20.e 659 kB/s | 49 kB 00:00 2024-04-25T11:22:51.998 INFO:teuthology.orchestra.run.smithi138.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3291.g9 197 kB/s | 16 kB 00:00 2024-04-25T11:22:52.141 INFO:teuthology.orchestra.run.smithi138.stdout:(35/152): ceph-volume-19.0.0-3291.g994e2f20.el9 1.9 MB/s | 273 kB 00:00 2024-04-25T11:22:52.374 INFO:teuthology.orchestra.run.smithi138.stdout:(36/152): cephadm-19.0.0-3291.g994e2f20.el9.noa 3.2 MB/s | 762 kB 00:00 2024-04-25T11:22:52.386 INFO:teuthology.orchestra.run.smithi090.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 1.0 MB/s | 1.6 MB 00:01 2024-04-25T11:22:52.441 INFO:teuthology.orchestra.run.smithi138.stdout:(37/152): cephfs-top-19.0.0-3291.g994e2f20.el9. 273 kB/s | 18 kB 00:00 2024-04-25T11:22:52.666 INFO:teuthology.orchestra.run.smithi156.stdout:(31/152): ceph-mgr-diskprediction-local-19.0.0- 5.3 MB/s | 7.4 MB 00:01 2024-04-25T11:22:52.732 INFO:teuthology.orchestra.run.smithi156.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3291.g99 3.7 MB/s | 249 kB 00:00 2024-04-25T11:22:52.775 INFO:teuthology.orchestra.run.smithi138.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 241 kB/s | 80 kB 00:00 2024-04-25T11:22:52.807 INFO:teuthology.orchestra.run.smithi156.stdout:(33/152): ceph-mgr-dashboard-19.0.0-3291.g994e2 2.2 MB/s | 3.5 MB 00:01 2024-04-25T11:22:52.833 INFO:teuthology.orchestra.run.smithi156.stdout:(34/152): ceph-mgr-rook-19.0.0-3291.g994e2f20.e 492 kB/s | 49 kB 00:00 2024-04-25T11:22:52.866 INFO:teuthology.orchestra.run.smithi156.stdout:(35/152): ceph-prometheus-alerts-19.0.0-3291.g9 281 kB/s | 16 kB 00:00 2024-04-25T11:22:52.892 INFO:teuthology.orchestra.run.smithi156.stdout:(36/152): ceph-volume-19.0.0-3291.g994e2f20.el9 4.6 MB/s | 273 kB 00:00 2024-04-25T11:22:52.934 INFO:teuthology.orchestra.run.smithi156.stdout:(37/152): cephfs-top-19.0.0-3291.g994e2f20.el9. 439 kB/s | 18 kB 00:00 2024-04-25T11:22:52.950 INFO:teuthology.orchestra.run.smithi138.stdout:(39/152): ceph-mgr-diskprediction-local-19.0.0- 3.3 MB/s | 7.4 MB 00:02 2024-04-25T11:22:53.067 INFO:teuthology.orchestra.run.smithi156.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 603 kB/s | 80 kB 00:00 2024-04-25T11:22:53.094 INFO:teuthology.orchestra.run.smithi090.stdout:(68/152): openblas-openmp-0.3.21-2.el9.x86_64.r 1.9 MB/s | 4.7 MB 00:02 2024-04-25T11:22:53.101 INFO:teuthology.orchestra.run.smithi156.stdout:(39/152): cephadm-19.0.0-3291.g994e2f20.el9.noa 3.2 MB/s | 762 kB 00:00 2024-04-25T11:22:53.200 INFO:teuthology.orchestra.run.smithi138.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 782 kB/s | 195 kB 00:00 2024-04-25T11:22:53.203 INFO:teuthology.orchestra.run.smithi090.stdout:(69/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 1.8 MB/s | 4.4 MB 00:02 2024-04-25T11:22:53.225 INFO:teuthology.orchestra.run.smithi138.stdout:(41/152): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-04-25T11:22:53.243 INFO:teuthology.orchestra.run.smithi156.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-04-25T11:22:53.301 INFO:teuthology.orchestra.run.smithi156.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.3 MB/s | 253 kB 00:00 2024-04-25T11:22:53.342 INFO:teuthology.orchestra.run.smithi138.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-04-25T11:22:53.343 INFO:teuthology.orchestra.run.smithi156.stdout:(42/152): libquadmath-11.4.1-3.el9.x86_64.rpm 807 kB/s | 195 kB 00:00 2024-04-25T11:22:53.409 INFO:teuthology.orchestra.run.smithi138.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-25T11:22:53.444 INFO:teuthology.orchestra.run.smithi156.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 106 kB 00:00 2024-04-25T11:22:53.470 INFO:teuthology.orchestra.run.smithi090.stdout:(70/152): postgresql-contrib-13.14-1.el9.x86_64 841 kB/s | 911 kB 00:01 2024-04-25T11:22:53.477 INFO:teuthology.orchestra.run.smithi156.stdout:(44/152): python3-cryptography-36.0.1-4.el9.x86 7.0 MB/s | 1.2 MB 00:00 2024-04-25T11:22:53.501 INFO:teuthology.orchestra.run.smithi138.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 1.4 MB/s | 135 kB 00:00 2024-04-25T11:22:53.536 INFO:teuthology.orchestra.run.smithi156.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-25T11:22:53.561 INFO:teuthology.orchestra.run.smithi156.stdout:(46/152): python3-pycparser-2.20-6.el9.noarch.r 1.1 MB/s | 135 kB 00:00 2024-04-25T11:22:53.568 INFO:teuthology.orchestra.run.smithi138.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-04-25T11:22:53.603 INFO:teuthology.orchestra.run.smithi156.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.1 MB/s | 215 kB 00:00 2024-04-25T11:22:53.635 INFO:teuthology.orchestra.run.smithi138.stdout:(46/152): python3-cryptography-36.0.1-4.el9.x86 3.0 MB/s | 1.2 MB 00:00 2024-04-25T11:22:53.694 INFO:teuthology.orchestra.run.smithi138.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 1.7 MB/s | 215 kB 00:00 2024-04-25T11:22:53.704 INFO:teuthology.orchestra.run.smithi090.stdout:(71/152): postgresql-private-libs-13.14-1.el9.x 219 kB/s | 133 kB 00:00 2024-04-25T11:22:53.904 INFO:teuthology.orchestra.run.smithi156.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 2.0 MB/s | 689 kB 00:00 2024-04-25T11:22:53.996 INFO:teuthology.orchestra.run.smithi156.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 1.1 MB/s | 107 kB 00:00 2024-04-25T11:22:54.046 INFO:teuthology.orchestra.run.smithi090.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 256 kB/s | 87 kB 00:00 2024-04-25T11:22:54.046 INFO:teuthology.orchestra.run.smithi156.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 433 kB/s | 21 kB 00:00 2024-04-25T11:22:54.138 INFO:teuthology.orchestra.run.smithi156.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 3.3 MB/s | 313 kB 00:00 2024-04-25T11:22:54.180 INFO:teuthology.orchestra.run.smithi156.stdout:(52/152): bison-3.7.4-5.el9.x86_64.rpm 1.6 MB/s | 943 kB 00:00 2024-04-25T11:22:54.197 INFO:teuthology.orchestra.run.smithi156.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 556 kB/s | 33 kB 00:00 2024-04-25T11:22:54.256 INFO:teuthology.orchestra.run.smithi156.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 313 kB/s | 18 kB 00:00 2024-04-25T11:22:54.296 INFO:teuthology.orchestra.run.smithi090.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 1.2 MB/s | 1.0 MB 00:00 2024-04-25T11:22:54.314 INFO:teuthology.orchestra.run.smithi156.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 191 kB/s | 11 kB 00:00 2024-04-25T11:22:54.381 INFO:teuthology.orchestra.run.smithi156.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.4 MB/s | 160 kB 00:00 2024-04-25T11:22:54.440 INFO:teuthology.orchestra.run.smithi156.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 776 kB/s | 45 kB 00:00 2024-04-25T11:22:54.453 INFO:teuthology.orchestra.run.smithi138.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 843 kB/s | 689 kB 00:00 2024-04-25T11:22:54.648 INFO:teuthology.orchestra.run.smithi156.stdout:(58/152): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 662 kB 00:00 2024-04-25T11:22:54.753 INFO:teuthology.orchestra.run.smithi138.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 891 kB/s | 943 kB 00:01 2024-04-25T11:22:54.824 INFO:teuthology.orchestra.run.smithi156.stdout:(59/152): libtool-2.4.6-45.el9.x86_64.rpm 3.3 MB/s | 582 kB 00:00 2024-04-25T11:22:54.904 INFO:teuthology.orchestra.run.smithi090.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 401 kB/s | 244 kB 00:00 2024-04-25T11:22:54.907 INFO:teuthology.orchestra.run.smithi156.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 306 kB/s | 25 kB 00:00 2024-04-25T11:22:54.982 INFO:teuthology.orchestra.run.smithi156.stdout:(61/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 633 kB/s | 47 kB 00:00 2024-04-25T11:22:55.050 INFO:teuthology.orchestra.run.smithi156.stdout:(62/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 292 kB 00:00 2024-04-25T11:22:55.150 INFO:teuthology.orchestra.run.smithi156.stdout:(63/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.1 MB/s | 3.0 MB 00:00 2024-04-25T11:22:55.161 INFO:teuthology.orchestra.run.smithi138.stdout:(50/152): ceph-test-19.0.0-3291.g994e2f20.el9.x 4.6 MB/s | 43 MB 00:09 2024-04-25T11:22:55.175 INFO:teuthology.orchestra.run.smithi156.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 188 kB 00:00 2024-04-25T11:22:55.186 INFO:teuthology.orchestra.run.smithi138.stdout:(51/152): boost-program-options-1.75.0-8.el9.x8 146 kB/s | 107 kB 00:00 2024-04-25T11:22:55.225 INFO:teuthology.orchestra.run.smithi156.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 468 kB/s | 35 kB 00:00 2024-04-25T11:22:55.337 INFO:teuthology.orchestra.run.smithi138.stdout:(52/152): elfutils-libelf-devel-0.190-2.el9.x86 37 kB/s | 21 kB 00:00 2024-04-25T11:22:55.445 INFO:teuthology.orchestra.run.smithi138.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 126 kB/s | 33 kB 00:00 2024-04-25T11:22:55.696 INFO:teuthology.orchestra.run.smithi138.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 73 kB/s | 18 kB 00:00 2024-04-25T11:22:55.797 INFO:teuthology.orchestra.run.smithi090.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 278 kB/s | 248 kB 00:00 2024-04-25T11:22:55.851 INFO:teuthology.orchestra.run.smithi156.stdout:(66/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 7.1 MB/s | 4.4 MB 00:00 2024-04-25T11:22:55.879 INFO:teuthology.orchestra.run.smithi138.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 61 kB/s | 11 kB 00:00 2024-04-25T11:22:55.926 INFO:teuthology.orchestra.run.smithi156.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 6.3 MB/s | 4.7 MB 00:00 2024-04-25T11:22:56.059 INFO:teuthology.orchestra.run.smithi156.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 6.7 MB/s | 911 kB 00:00 2024-04-25T11:22:56.101 INFO:teuthology.orchestra.run.smithi156.stdout:(69/152): postgresql-13.14-1.el9.x86_64.rpm 6.3 MB/s | 1.6 MB 00:00 2024-04-25T11:22:56.122 INFO:teuthology.orchestra.run.smithi090.stdout:(76/152): postgresql-server-13.14-1.el9.x86_64. 2.0 MB/s | 5.8 MB 00:02 2024-04-25T11:22:56.126 INFO:teuthology.orchestra.run.smithi156.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 2.0 MB/s | 133 kB 00:00 2024-04-25T11:22:56.138 INFO:teuthology.orchestra.run.smithi090.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 75 kB/s | 26 kB 00:00 2024-04-25T11:22:56.221 INFO:teuthology.orchestra.run.smithi138.stdout:(56/152): flex-2.6.4-9.el9.x86_64.rpm 296 kB/s | 313 kB 00:01 2024-04-25T11:22:56.277 INFO:teuthology.orchestra.run.smithi156.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-04-25T11:22:56.344 INFO:teuthology.orchestra.run.smithi156.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-04-25T11:22:56.380 INFO:teuthology.orchestra.run.smithi138.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 320 kB/s | 160 kB 00:00 2024-04-25T11:22:56.414 INFO:teuthology.orchestra.run.smithi090.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 66 kB/s | 19 kB 00:00 2024-04-25T11:22:56.563 INFO:teuthology.orchestra.run.smithi138.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 132 kB/s | 45 kB 00:00 2024-04-25T11:22:56.594 INFO:teuthology.orchestra.run.smithi156.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 12 MB/s | 5.8 MB 00:00 2024-04-25T11:22:56.639 INFO:teuthology.orchestra.run.smithi090.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 344 kB/s | 172 kB 00:00 2024-04-25T11:22:56.661 INFO:teuthology.orchestra.run.smithi156.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 3.6 MB/s | 244 kB 00:00 2024-04-25T11:22:56.673 INFO:teuthology.orchestra.run.smithi090.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 134 kB/s | 35 kB 00:00 2024-04-25T11:22:56.736 INFO:teuthology.orchestra.run.smithi156.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 3.2 MB/s | 248 kB 00:00 2024-04-25T11:22:56.820 INFO:teuthology.orchestra.run.smithi156.stdout:(76/152): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 6.0 MB 00:00 2024-04-25T11:22:56.845 INFO:teuthology.orchestra.run.smithi156.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 236 kB/s | 26 kB 00:00 2024-04-25T11:22:56.879 INFO:teuthology.orchestra.run.smithi156.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 328 kB/s | 19 kB 00:00 2024-04-25T11:22:56.912 INFO:teuthology.orchestra.run.smithi156.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-04-25T11:22:56.938 INFO:teuthology.orchestra.run.smithi156.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 592 kB/s | 35 kB 00:00 2024-04-25T11:22:57.022 INFO:teuthology.orchestra.run.smithi156.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.9 MB/s | 250 kB 00:00 2024-04-25T11:22:57.105 INFO:teuthology.orchestra.run.smithi156.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.6 MB/s | 222 kB 00:00 2024-04-25T11:22:57.206 INFO:teuthology.orchestra.run.smithi156.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 773 kB/s | 77 kB 00:00 2024-04-25T11:22:57.222 INFO:teuthology.orchestra.run.smithi138.stdout:(59/152): libtool-2.4.6-45.el9.x86_64.rpm 884 kB/s | 582 kB 00:00 2024-04-25T11:22:57.241 INFO:teuthology.orchestra.run.smithi090.stdout:(81/152): python3-babel-2.9.1-2.el9.noarch.rpm 1.9 MB/s | 6.0 MB 00:03 2024-04-25T11:22:57.281 INFO:teuthology.orchestra.run.smithi156.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 564 kB/s | 42 kB 00:00 2024-04-25T11:22:57.376 INFO:teuthology.orchestra.run.smithi156.stdout:(85/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-04-25T11:22:57.407 INFO:teuthology.orchestra.run.smithi156.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 2.1 MB/s | 269 kB 00:00 2024-04-25T11:22:57.440 INFO:teuthology.orchestra.run.smithi156.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.4 MB/s | 159 kB 00:00 2024-04-25T11:22:57.474 INFO:teuthology.orchestra.run.smithi156.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 279 kB 00:00 2024-04-25T11:22:57.481 INFO:teuthology.orchestra.run.smithi138.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 98 kB/s | 25 kB 00:00 2024-04-25T11:22:57.516 INFO:teuthology.orchestra.run.smithi156.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 678 kB/s | 51 kB 00:00 2024-04-25T11:22:57.532 INFO:teuthology.orchestra.run.smithi156.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 916 kB/s | 54 kB 00:00 2024-04-25T11:22:57.600 INFO:teuthology.orchestra.run.smithi090.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 270 kB/s | 250 kB 00:00 2024-04-25T11:22:57.608 INFO:teuthology.orchestra.run.smithi156.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 557 kB/s | 42 kB 00:00 2024-04-25T11:22:57.658 INFO:teuthology.orchestra.run.smithi156.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 886 kB/s | 44 kB 00:00 2024-04-25T11:22:57.725 INFO:teuthology.orchestra.run.smithi156.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 221 kB 00:00 2024-04-25T11:22:57.742 INFO:teuthology.orchestra.run.smithi090.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 444 kB/s | 222 kB 00:00 2024-04-25T11:22:57.792 INFO:teuthology.orchestra.run.smithi156.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 539 kB/s | 36 kB 00:00 2024-04-25T11:22:57.831 INFO:teuthology.orchestra.run.smithi138.stdout:(61/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 135 kB/s | 47 kB 00:00 2024-04-25T11:22:57.859 INFO:teuthology.orchestra.run.smithi156.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 711 kB/s | 47 kB 00:00 2024-04-25T11:22:57.926 INFO:teuthology.orchestra.run.smithi156.stdout:(96/152): socat-1.7.4.1-5.el9.x86_64.rpm 4.5 MB/s | 305 kB 00:00 2024-04-25T11:22:57.981 INFO:teuthology.orchestra.run.smithi138.stdout:(62/152): librdkafka-1.6.1-102.el9.x86_64.rpm 414 kB/s | 662 kB 00:01 2024-04-25T11:22:57.994 INFO:teuthology.orchestra.run.smithi156.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 867 kB/s | 58 kB 00:00 2024-04-25T11:22:58.026 INFO:teuthology.orchestra.run.smithi090.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 181 kB/s | 77 kB 00:00 2024-04-25T11:22:58.052 INFO:teuthology.orchestra.run.smithi156.stdout:(98/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-04-25T11:22:58.093 INFO:teuthology.orchestra.run.smithi090.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 120 kB/s | 42 kB 00:00 2024-04-25T11:22:58.102 INFO:teuthology.orchestra.run.smithi156.stdout:(99/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-25T11:22:58.261 INFO:teuthology.orchestra.run.smithi156.stdout:(100/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-04-25T11:22:58.278 INFO:teuthology.orchestra.run.smithi156.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 255 kB/s | 45 kB 00:00 2024-04-25T11:22:58.340 INFO:teuthology.orchestra.run.smithi138.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 575 kB/s | 292 kB 00:00 2024-04-25T11:22:58.518 INFO:teuthology.orchestra.run.smithi090.stdout:(86/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 373 kB/s | 159 kB 00:00 2024-04-25T11:22:58.593 INFO:teuthology.orchestra.run.smithi090.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 475 kB/s | 269 kB 00:00 2024-04-25T11:22:58.615 INFO:teuthology.orchestra.run.smithi138.stdout:(64/152): openblas-0.3.21-2.el9.x86_64.rpm 128 kB/s | 35 kB 00:00 2024-04-25T11:22:58.640 INFO:teuthology.orchestra.run.smithi138.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 286 kB/s | 188 kB 00:00 2024-04-25T11:22:58.690 INFO:teuthology.orchestra.run.smithi138.stdout:(66/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 913 kB/s | 3.0 MB 00:03 2024-04-25T11:22:58.706 INFO:teuthology.orchestra.run.smithi156.stdout:(102/152): ceph-test-19.0.0-3291.g994e2f20.el9. 4.2 MB/s | 43 MB 00:10 2024-04-25T11:22:58.720 INFO:teuthology.orchestra.run.smithi156.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 49 kB/s | 22 kB 00:00 2024-04-25T11:22:58.878 INFO:teuthology.orchestra.run.smithi156.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 1.4 MB/s | 863 kB 00:00 2024-04-25T11:22:58.927 INFO:teuthology.orchestra.run.smithi090.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 153 kB/s | 51 kB 00:00 2024-04-25T11:22:59.012 INFO:teuthology.orchestra.run.smithi156.stdout:(105/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.0 MB/s | 309 kB 00:00 2024-04-25T11:22:59.045 INFO:teuthology.orchestra.run.smithi156.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 116 kB/s | 19 kB 00:00 2024-04-25T11:22:59.079 INFO:teuthology.orchestra.run.smithi156.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.4 MB/s | 548 kB 00:00 2024-04-25T11:22:59.114 INFO:teuthology.orchestra.run.smithi156.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 388 kB/s | 26 kB 00:00 2024-04-25T11:22:59.155 INFO:teuthology.orchestra.run.smithi156.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 643 kB/s | 49 kB 00:00 2024-04-25T11:22:59.214 INFO:teuthology.orchestra.run.smithi156.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 671 kB/s | 67 kB 00:00 2024-04-25T11:22:59.239 INFO:teuthology.orchestra.run.smithi156.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-04-25T11:22:59.269 INFO:teuthology.orchestra.run.smithi090.stdout:(89/152): python3-requests-oauthlib-1.3.0-12.el 157 kB/s | 54 kB 00:00 2024-04-25T11:22:59.364 INFO:teuthology.orchestra.run.smithi156.stdout:(112/152): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-04-25T11:22:59.398 INFO:teuthology.orchestra.run.smithi156.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 3.4 MB/s | 548 kB 00:00 2024-04-25T11:22:59.432 INFO:teuthology.orchestra.run.smithi156.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 650 kB/s | 43 kB 00:00 2024-04-25T11:22:59.465 INFO:teuthology.orchestra.run.smithi156.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 482 kB/s | 32 kB 00:00 2024-04-25T11:22:59.498 INFO:teuthology.orchestra.run.smithi156.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 224 kB/s | 15 kB 00:00 2024-04-25T11:22:59.526 INFO:teuthology.orchestra.run.smithi156.stdout:(117/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.6 MB/s | 838 kB 00:00 2024-04-25T11:22:59.549 INFO:teuthology.orchestra.run.smithi156.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-04-25T11:22:59.561 INFO:teuthology.orchestra.run.smithi090.stdout:(90/152): python3-pyasn1-modules-0.4.8-6.el9.no 267 kB/s | 279 kB 00:01 2024-04-25T11:22:59.583 INFO:teuthology.orchestra.run.smithi156.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-04-25T11:22:59.608 INFO:teuthology.orchestra.run.smithi156.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.5 MB/s | 303 kB 00:00 2024-04-25T11:22:59.633 INFO:teuthology.orchestra.run.smithi156.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 421 kB/s | 35 kB 00:00 2024-04-25T11:22:59.666 INFO:teuthology.orchestra.run.smithi156.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.4 MB/s | 206 kB 00:00 2024-04-25T11:22:59.709 INFO:teuthology.orchestra.run.smithi156.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-04-25T11:22:59.719 INFO:teuthology.orchestra.run.smithi090.stdout:(91/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 1.7 MB/s | 5.1 MB 00:03 2024-04-25T11:22:59.734 INFO:teuthology.orchestra.run.smithi156.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-04-25T11:22:59.784 INFO:teuthology.orchestra.run.smithi156.stdout:(125/152): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-04-25T11:22:59.809 INFO:teuthology.orchestra.run.smithi156.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 176 kB/s | 18 kB 00:00 2024-04-25T11:22:59.835 INFO:teuthology.orchestra.run.smithi156.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 230 kB/s | 23 kB 00:00 2024-04-25T11:22:59.860 INFO:teuthology.orchestra.run.smithi156.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-04-25T11:22:59.893 INFO:teuthology.orchestra.run.smithi156.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 337 kB/s | 20 kB 00:00 2024-04-25T11:22:59.903 INFO:teuthology.orchestra.run.smithi090.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 122 kB/s | 42 kB 00:00 2024-04-25T11:22:59.927 INFO:teuthology.orchestra.run.smithi156.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-04-25T11:22:59.944 INFO:teuthology.orchestra.run.smithi156.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 551 kB/s | 46 kB 00:00 2024-04-25T11:22:59.969 INFO:teuthology.orchestra.run.smithi156.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-04-25T11:22:59.986 INFO:teuthology.orchestra.run.smithi156.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 984 kB/s | 58 kB 00:00 2024-04-25T11:23:00.011 INFO:teuthology.orchestra.run.smithi156.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-04-25T11:23:00.029 INFO:teuthology.orchestra.run.smithi156.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 279 kB/s | 16 kB 00:00 2024-04-25T11:23:00.054 INFO:teuthology.orchestra.run.smithi156.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-04-25T11:23:00.062 INFO:teuthology.orchestra.run.smithi090.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 129 kB/s | 44 kB 00:00 2024-04-25T11:23:00.087 INFO:teuthology.orchestra.run.smithi156.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 374 kB/s | 28 kB 00:00 2024-04-25T11:23:00.112 INFO:teuthology.orchestra.run.smithi156.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 368 kB/s | 31 kB 00:00 2024-04-25T11:23:00.138 INFO:teuthology.orchestra.run.smithi156.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-04-25T11:23:00.164 INFO:teuthology.orchestra.run.smithi156.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 780 kB/s | 59 kB 00:00 2024-04-25T11:23:00.189 INFO:teuthology.orchestra.run.smithi156.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 472 kB/s | 36 kB 00:00 2024-04-25T11:23:00.214 INFO:teuthology.orchestra.run.smithi156.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 674 kB/s | 51 kB 00:00 2024-04-25T11:23:00.239 INFO:teuthology.orchestra.run.smithi156.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-25T11:23:00.265 INFO:teuthology.orchestra.run.smithi156.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-04-25T11:23:00.298 INFO:teuthology.orchestra.run.smithi156.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-04-25T11:23:00.320 INFO:teuthology.orchestra.run.smithi090.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 139 kB/s | 36 kB 00:00 2024-04-25T11:23:00.323 INFO:teuthology.orchestra.run.smithi156.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 238 kB/s | 20 kB 00:00 2024-04-25T11:23:00.349 INFO:teuthology.orchestra.run.smithi156.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-04-25T11:23:00.399 INFO:teuthology.orchestra.run.smithi156.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-04-25T11:23:00.412 INFO:teuthology.orchestra.run.smithi090.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 434 kB/s | 221 kB 00:00 2024-04-25T11:23:00.557 INFO:teuthology.orchestra.run.smithi138.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 2.4 MB/s | 4.7 MB 00:01 2024-04-25T11:23:00.624 INFO:teuthology.orchestra.run.smithi156.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 6.0 MB/s | 1.3 MB 00:00 2024-04-25T11:23:00.688 INFO:teuthology.orchestra.run.smithi090.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 129 kB/s | 47 kB 00:00 2024-04-25T11:23:00.716 INFO:teuthology.orchestra.run.smithi138.stdout:(68/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 2.1 MB/s | 4.4 MB 00:02 2024-04-25T11:23:00.791 INFO:teuthology.orchestra.run.smithi156.stdout:(150/152): librbd1-19.0.0-3291.g994e2f20.el9.x8 7.0 MB/s | 3.1 MB 00:00 2024-04-25T11:23:00.841 INFO:teuthology.orchestra.run.smithi156.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 11 MB/s | 2.4 MB 00:00 2024-04-25T11:23:01.030 INFO:teuthology.orchestra.run.smithi090.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 169 kB/s | 58 kB 00:00 2024-04-25T11:23:01.200 INFO:teuthology.orchestra.run.smithi138.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 276 kB/s | 133 kB 00:00 2024-04-25T11:23:01.250 INFO:teuthology.orchestra.run.smithi156.stdout:(152/152): librados2-19.0.0-3291.g994e2f20.el9. 3.6 MB/s | 3.3 MB 00:00 2024-04-25T11:23:01.254 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-04-25T11:23:01.254 INFO:teuthology.orchestra.run.smithi156.stdout:Total 12 MB/s | 214 MB 00:18 2024-04-25T11:23:01.313 INFO:teuthology.orchestra.run.smithi090.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 338 kB/s | 305 kB 00:00 2024-04-25T11:23:01.400 INFO:teuthology.orchestra.run.smithi138.stdout:(70/152): postgresql-13.14-1.el9.x86_64.rpm 600 kB/s | 1.6 MB 00:02 2024-04-25T11:23:01.455 INFO:teuthology.orchestra.run.smithi090.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 220 kB/s | 94 kB 00:00 2024-04-25T11:23:01.647 INFO:teuthology.orchestra.run.smithi090.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 191 kB/s | 64 kB 00:00 2024-04-25T11:23:01.739 INFO:teuthology.orchestra.run.smithi090.stdout:(101/152): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-04-25T11:23:01.806 INFO:teuthology.orchestra.run.smithi090.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 128 kB/s | 45 kB 00:00 2024-04-25T11:23:01.907 INFO:teuthology.orchestra.run.smithi090.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 5.0 MB/s | 863 kB 00:00 2024-04-25T11:23:02.074 INFO:teuthology.orchestra.run.smithi090.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 2.0 MB/s | 548 kB 00:00 2024-04-25T11:23:02.157 INFO:teuthology.orchestra.run.smithi090.stdout:(105/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.2 MB/s | 309 kB 00:00 2024-04-25T11:23:02.226 INFO:teuthology.orchestra.run.smithi090.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 128 kB/s | 19 kB 00:00 2024-04-25T11:23:02.284 INFO:teuthology.orchestra.run.smithi090.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 444 kB/s | 26 kB 00:00 2024-04-25T11:23:02.341 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-04-25T11:23:02.385 INFO:teuthology.orchestra.run.smithi090.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 486 kB/s | 49 kB 00:00 2024-04-25T11:23:02.485 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-04-25T11:23:02.485 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-04-25T11:23:02.510 INFO:teuthology.orchestra.run.smithi090.stdout:(109/152): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-04-25T11:23:02.535 INFO:teuthology.orchestra.run.smithi090.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 448 kB/s | 67 kB 00:00 2024-04-25T11:23:02.636 INFO:teuthology.orchestra.run.smithi090.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-04-25T11:23:02.686 INFO:teuthology.orchestra.run.smithi090.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.4 MB/s | 838 kB 00:00 2024-04-25T11:23:02.720 INFO:teuthology.orchestra.run.smithi090.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 6.4 MB/s | 548 kB 00:00 2024-04-25T11:23:02.780 INFO:teuthology.orchestra.run.smithi138.stdout:(71/152): postgresql-contrib-13.14-1.el9.x86_64 410 kB/s | 911 kB 00:02 2024-04-25T11:23:02.845 INFO:teuthology.orchestra.run.smithi090.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 257 kB/s | 32 kB 00:00 2024-04-25T11:23:02.870 INFO:teuthology.orchestra.run.smithi090.stdout:(115/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 236 kB/s | 43 kB 00:00 2024-04-25T11:23:03.021 INFO:teuthology.orchestra.run.smithi090.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 85 kB/s | 15 kB 00:00 2024-04-25T11:23:03.046 INFO:teuthology.orchestra.run.smithi090.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 980 kB/s | 172 kB 00:00 2024-04-25T11:23:03.146 INFO:teuthology.orchestra.run.smithi090.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2024-04-25T11:23:03.197 INFO:teuthology.orchestra.run.smithi090.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.0 MB/s | 303 kB 00:00 2024-04-25T11:23:03.208 INFO:teuthology.orchestra.run.smithi138.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 204 kB/s | 87 kB 00:00 2024-04-25T11:23:03.231 INFO:teuthology.orchestra.run.smithi090.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 422 kB/s | 35 kB 00:00 2024-04-25T11:23:03.289 INFO:teuthology.orchestra.run.smithi090.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 2.2 MB/s | 206 kB 00:00 2024-04-25T11:23:03.365 INFO:teuthology.orchestra.run.smithi090.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-04-25T11:23:03.456 INFO:teuthology.orchestra.run.smithi090.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 9.0 MB/s | 2.0 MB 00:00 2024-04-25T11:23:03.473 INFO:teuthology.orchestra.run.smithi090.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 100 kB/s | 11 kB 00:00 2024-04-25T11:23:03.515 INFO:teuthology.orchestra.run.smithi090.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 301 kB/s | 18 kB 00:00 2024-04-25T11:23:03.517 INFO:teuthology.orchestra.run.smithi138.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 488 kB/s | 1.0 MB 00:02 2024-04-25T11:23:03.566 INFO:teuthology.orchestra.run.smithi090.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 252 kB/s | 23 kB 00:00 2024-04-25T11:23:03.591 INFO:teuthology.orchestra.run.smithi090.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-04-25T11:23:03.624 INFO:teuthology.orchestra.run.smithi090.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 335 kB/s | 20 kB 00:00 2024-04-25T11:23:03.741 INFO:teuthology.orchestra.run.smithi090.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 397 kB/s | 46 kB 00:00 2024-04-25T11:23:03.802 INFO:teuthology.orchestra.run.smithi090.stdout:(130/152): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-04-25T11:23:03.843 INFO:teuthology.orchestra.run.smithi090.stdout:(131/152): python3-kubernetes-26.1.0-2.el9.noar 4.0 MB/s | 1.0 MB 00:00 2024-04-25T11:23:03.894 INFO:teuthology.orchestra.run.smithi090.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 627 kB/s | 58 kB 00:00 2024-04-25T11:23:03.944 INFO:teuthology.orchestra.run.smithi090.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-04-25T11:23:03.995 INFO:teuthology.orchestra.run.smithi090.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 163 kB/s | 16 kB 00:00 2024-04-25T11:23:04.017 INFO:teuthology.orchestra.run.smithi138.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 488 kB/s | 244 kB 00:00 2024-04-25T11:23:04.078 INFO:teuthology.orchestra.run.smithi090.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 672 kB/s | 90 kB 00:00 2024-04-25T11:23:04.104 INFO:teuthology.orchestra.run.smithi090.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 259 kB/s | 28 kB 00:00 2024-04-25T11:23:04.162 INFO:teuthology.orchestra.run.smithi090.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 368 kB/s | 31 kB 00:00 2024-04-25T11:23:04.196 INFO:teuthology.orchestra.run.smithi090.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-04-25T11:23:04.223 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-04-25T11:23:04.223 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-04-25T11:23:04.255 INFO:teuthology.orchestra.run.smithi090.stdout:(139/152): python3-tempora-5.0.0-2.el9.noarch.r 613 kB/s | 36 kB 00:00 2024-04-25T11:23:04.288 INFO:teuthology.orchestra.run.smithi090.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 471 kB/s | 59 kB 00:00 2024-04-25T11:23:04.330 INFO:teuthology.orchestra.run.smithi090.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 677 kB/s | 51 kB 00:00 2024-04-25T11:23:04.381 INFO:teuthology.orchestra.run.smithi090.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-04-25T11:23:04.406 INFO:teuthology.orchestra.run.smithi090.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-04-25T11:23:04.473 INFO:teuthology.orchestra.run.smithi090.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 4.6 MB/s | 427 kB 00:00 2024-04-25T11:23:04.506 INFO:teuthology.orchestra.run.smithi090.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 199 kB/s | 20 kB 00:00 2024-04-25T11:23:04.540 INFO:teuthology.orchestra.run.smithi090.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-04-25T11:23:04.632 INFO:teuthology.orchestra.run.smithi090.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-04-25T11:23:04.776 INFO:teuthology.orchestra.run.smithi138.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 327 kB/s | 248 kB 00:00 2024-04-25T11:23:05.026 INFO:teuthology.orchestra.run.smithi138.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 102 kB/s | 26 kB 00:00 2024-04-25T11:23:05.108 INFO:teuthology.orchestra.run.smithi090.stdout:(148/152): librados2-19.0.0-3291.g994e2f20.el9. 5.9 MB/s | 3.3 MB 00:00 2024-04-25T11:23:05.183 INFO:teuthology.orchestra.run.smithi090.stdout:(149/152): librbd1-19.0.0-3291.g994e2f20.el9.x8 5.6 MB/s | 3.1 MB 00:00 2024-04-25T11:23:05.264 INFO:teuthology.orchestra.run.smithi138.stdout:(77/152): python3-babel-2.9.1-2.el9.noarch.rpm 2.9 MB/s | 6.0 MB 00:02 2024-04-25T11:23:05.285 INFO:teuthology.orchestra.run.smithi138.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 74 kB/s | 19 kB 00:00 2024-04-25T11:23:05.508 INFO:teuthology.orchestra.run.smithi090.stdout:(150/152): openssl-3.2.1-1.el9.x86_64.rpm 3.4 MB/s | 1.3 MB 00:00 2024-04-25T11:23:05.543 INFO:teuthology.orchestra.run.smithi138.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 134 kB/s | 35 kB 00:00 2024-04-25T11:23:05.567 INFO:teuthology.orchestra.run.smithi090.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 6.2 MB/s | 2.4 MB 00:00 2024-04-25T11:23:05.619 INFO:teuthology.orchestra.run.smithi138.stdout:(80/152): postgresql-server-13.14-1.el9.x86_64. 1.3 MB/s | 5.8 MB 00:04 2024-04-25T11:23:05.694 INFO:teuthology.orchestra.run.smithi138.stdout:(81/152): python3-mako-1.1.4-6.el9.noarch.rpm 401 kB/s | 172 kB 00:00 2024-04-25T11:23:05.784 INFO:teuthology.orchestra.run.smithi090.stdout:(152/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 2.5 MB/s | 16 MB 00:06 2024-04-25T11:23:05.786 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-04-25T11:23:05.787 INFO:teuthology.orchestra.run.smithi090.stdout:Total 7.5 MB/s | 214 MB 00:28 2024-04-25T11:23:06.119 INFO:teuthology.orchestra.run.smithi138.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 501 kB/s | 250 kB 00:00 2024-04-25T11:23:06.403 INFO:teuthology.orchestra.run.smithi138.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 313 kB/s | 222 kB 00:00 2024-04-25T11:23:06.545 INFO:teuthology.orchestra.run.smithi138.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 181 kB/s | 77 kB 00:00 2024-04-25T11:23:06.745 INFO:teuthology.orchestra.run.smithi138.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 124 kB/s | 42 kB 00:00 2024-04-25T11:23:06.846 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-04-25T11:23:06.986 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-04-25T11:23:06.986 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-04-25T11:23:07.062 INFO:teuthology.orchestra.run.smithi138.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 521 kB/s | 269 kB 00:00 2024-04-25T11:23:07.237 INFO:teuthology.orchestra.run.smithi138.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 323 kB/s | 159 kB 00:00 2024-04-25T11:23:07.562 INFO:teuthology.orchestra.run.smithi138.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 557 kB/s | 279 kB 00:00 2024-04-25T11:23:07.588 INFO:teuthology.orchestra.run.smithi138.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 145 kB/s | 51 kB 00:00 2024-04-25T11:23:07.705 INFO:teuthology.orchestra.run.smithi138.stdout:(90/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.4 MB/s | 5.1 MB 00:02 2024-04-25T11:23:07.971 INFO:teuthology.orchestra.run.smithi138.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 131 kB/s | 54 kB 00:00 2024-04-25T11:23:07.997 INFO:teuthology.orchestra.run.smithi138.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 143 kB/s | 42 kB 00:00 2024-04-25T11:23:08.314 INFO:teuthology.orchestra.run.smithi138.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 129 kB/s | 44 kB 00:00 2024-04-25T11:23:08.497 INFO:teuthology.orchestra.run.smithi138.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 441 kB/s | 221 kB 00:00 2024-04-25T11:23:08.589 INFO:teuthology.orchestra.run.smithi138.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 130 kB/s | 36 kB 00:00 2024-04-25T11:23:08.732 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-04-25T11:23:08.732 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-04-25T11:23:08.848 INFO:teuthology.orchestra.run.smithi138.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 135 kB/s | 47 kB 00:00 2024-04-25T11:23:09.098 INFO:teuthology.orchestra.run.smithi138.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 599 kB/s | 305 kB 00:00 2024-04-25T11:23:09.190 INFO:teuthology.orchestra.run.smithi138.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 169 kB/s | 58 kB 00:00 2024-04-25T11:23:09.549 INFO:teuthology.orchestra.run.smithi138.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 208 kB/s | 94 kB 00:00 2024-04-25T11:23:09.574 INFO:teuthology.orchestra.run.smithi138.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 166 kB/s | 64 kB 00:00 2024-04-25T11:23:09.658 INFO:teuthology.orchestra.run.smithi138.stdout:(101/152): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-04-25T11:23:09.900 INFO:teuthology.orchestra.run.smithi138.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 128 kB/s | 45 kB 00:00 2024-04-25T11:23:09.967 INFO:teuthology.orchestra.run.smithi138.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 2.7 MB/s | 863 kB 00:00 2024-04-25T11:23:10.121 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-04-25T11:23:10.203 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T11:23:10.250 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T11:23:10.303 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T11:23:10.337 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T11:23:10.363 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T11:23:10.406 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T11:23:10.433 INFO:teuthology.orchestra.run.smithi138.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 662 kB/s | 309 kB 00:00 2024-04-25T11:23:10.459 INFO:teuthology.orchestra.run.smithi138.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 981 kB/s | 548 kB 00:00 2024-04-25T11:23:10.534 INFO:teuthology.orchestra.run.smithi138.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 194 kB/s | 19 kB 00:00 2024-04-25T11:23:10.643 INFO:teuthology.orchestra.run.smithi138.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-04-25T11:23:10.742 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T11:23:10.759 INFO:teuthology.orchestra.run.smithi138.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 416 kB/s | 49 kB 00:00 2024-04-25T11:23:10.835 INFO:teuthology.orchestra.run.smithi138.stdout:(109/152): libarrow-9.0.0-10.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-04-25T11:23:10.872 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T11:23:10.994 INFO:teuthology.orchestra.run.smithi138.stdout:(110/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 4.8 MB/s | 16 MB 00:03 2024-04-25T11:23:10.997 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T11:23:11.011 INFO:teuthology.orchestra.run.smithi138.stdout:(111/152): libunwind-1.6.2-1.el9.x86_64.rpm 269 kB/s | 67 kB 00:00 2024-04-25T11:23:11.032 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T11:23:11.036 INFO:teuthology.orchestra.run.smithi138.stdout:(112/152): luarocks-3.9.2-1.el9.noarch.rpm 753 kB/s | 151 kB 00:00 2024-04-25T11:23:11.114 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T11:23:11.153 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T11:23:11.161 INFO:teuthology.orchestra.run.smithi138.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 348 kB/s | 43 kB 00:00 2024-04-25T11:23:11.228 INFO:teuthology.orchestra.run.smithi138.stdout:(114/152): python3-asyncssh-2.13.2-5.el9.noarch 2.5 MB/s | 548 kB 00:00 2024-04-25T11:23:11.270 INFO:teuthology.orchestra.run.smithi138.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 297 kB/s | 32 kB 00:00 2024-04-25T11:23:11.302 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T11:23:11.320 INFO:teuthology.orchestra.run.smithi138.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 163 kB/s | 15 kB 00:00 2024-04-25T11:23:11.349 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T11:23:11.371 INFO:teuthology.orchestra.run.smithi138.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-04-25T11:23:11.377 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T11:23:11.475 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T11:23:11.487 INFO:teuthology.orchestra.run.smithi138.stdout:(118/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.5 MB/s | 303 kB 00:00 2024-04-25T11:23:11.504 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T11:23:11.513 INFO:teuthology.orchestra.run.smithi138.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 1.8 MB/s | 358 kB 00:00 2024-04-25T11:23:11.566 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T11:23:11.588 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T11:23:11.589 INFO:teuthology.orchestra.run.smithi138.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 352 kB/s | 35 kB 00:00 2024-04-25T11:23:11.609 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T11:23:11.613 INFO:teuthology.orchestra.run.smithi138.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 2.0 MB/s | 206 kB 00:00 2024-04-25T11:23:11.664 INFO:teuthology.orchestra.run.smithi138.stdout:(122/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.2 MB/s | 838 kB 00:00 2024-04-25T11:23:11.723 INFO:teuthology.orchestra.run.smithi138.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.3 MB/s | 144 kB 00:00 2024-04-25T11:23:11.765 INFO:teuthology.orchestra.run.smithi138.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 106 kB/s | 11 kB 00:00 2024-04-25T11:23:11.804 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T11:23:11.815 INFO:teuthology.orchestra.run.smithi138.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 192 kB/s | 18 kB 00:00 2024-04-25T11:23:11.838 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T11:23:11.857 INFO:teuthology.orchestra.run.smithi138.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 252 kB/s | 23 kB 00:00 2024-04-25T11:23:11.907 INFO:teuthology.orchestra.run.smithi138.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-04-25T11:23:11.951 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T11:23:11.957 INFO:teuthology.orchestra.run.smithi138.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-04-25T11:23:12.052 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T11:23:12.057 INFO:teuthology.orchestra.run.smithi138.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 463 kB/s | 46 kB 00:00 2024-04-25T11:23:12.087 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T11:23:12.122 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T11:23:12.175 INFO:teuthology.orchestra.run.smithi138.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 3.8 MB/s | 1.0 MB 00:00 2024-04-25T11:23:12.197 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T11:23:12.200 INFO:teuthology.orchestra.run.smithi138.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 555 kB/s | 79 kB 00:00 2024-04-25T11:23:12.238 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T11:23:12.267 INFO:teuthology.orchestra.run.smithi138.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 628 kB/s | 58 kB 00:00 2024-04-25T11:23:12.282 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T11:23:12.300 INFO:teuthology.orchestra.run.smithi138.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-04-25T11:23:12.314 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T11:23:12.342 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T11:23:12.359 INFO:teuthology.orchestra.run.smithi138.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 178 kB/s | 16 kB 00:00 2024-04-25T11:23:12.401 INFO:teuthology.orchestra.run.smithi138.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 896 kB/s | 90 kB 00:00 2024-04-25T11:23:12.412 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T11:23:12.451 INFO:teuthology.orchestra.run.smithi138.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 307 kB/s | 28 kB 00:00 2024-04-25T11:23:12.473 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T11:23:12.501 INFO:teuthology.orchestra.run.smithi138.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 307 kB/s | 31 kB 00:00 2024-04-25T11:23:12.503 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T11:23:12.547 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T11:23:12.568 INFO:teuthology.orchestra.run.smithi138.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-04-25T11:23:12.610 INFO:teuthology.orchestra.run.smithi138.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 544 kB/s | 59 kB 00:00 2024-04-25T11:23:12.624 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T11:23:12.652 INFO:teuthology.orchestra.run.smithi138.stdout:(140/152): python3-grpcio-1.46.7-10.el9.x86_64. 1.9 MB/s | 2.0 MB 00:01 2024-04-25T11:23:12.657 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 31/156 2024-04-25T11:23:12.670 INFO:teuthology.orchestra.run.smithi138.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 355 kB/s | 36 kB 00:00 2024-04-25T11:23:12.701 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 32/156 2024-04-25T11:23:12.711 INFO:teuthology.orchestra.run.smithi138.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 507 kB/s | 51 kB 00:00 2024-04-25T11:23:12.729 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 33/156 2024-04-25T11:23:12.753 INFO:teuthology.orchestra.run.smithi138.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-04-25T11:23:12.778 INFO:teuthology.orchestra.run.smithi138.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 826 kB/s | 90 kB 00:00 2024-04-25T11:23:12.800 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T11:23:12.820 INFO:teuthology.orchestra.run.smithi138.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-04-25T11:23:12.853 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T11:23:12.854 INFO:teuthology.orchestra.run.smithi138.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 199 kB/s | 20 kB 00:00 2024-04-25T11:23:12.888 INFO:teuthology.orchestra.run.smithi138.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-04-25T11:23:12.927 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T11:23:13.062 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T11:23:13.113 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T11:23:13.530 INFO:teuthology.orchestra.run.smithi138.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 2.2 MB/s | 1.6 MB 00:00 2024-04-25T11:23:13.646 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T11:23:13.725 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T11:23:13.754 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T11:23:13.797 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T11:23:13.835 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 43/156 2024-04-25T11:23:13.876 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T11:23:14.025 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T11:23:14.055 INFO:teuthology.orchestra.run.smithi138.stdout:(149/152): librbd1-19.0.0-3291.g994e2f20.el9.x8 2.6 MB/s | 3.1 MB 00:01 2024-04-25T11:23:14.080 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T11:23:14.097 INFO:teuthology.orchestra.run.smithi138.stdout:(150/152): openssl-3.2.1-1.el9.x86_64.rpm 2.4 MB/s | 1.3 MB 00:00 2024-04-25T11:23:14.135 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T11:23:14.147 INFO:teuthology.orchestra.run.smithi138.stdout:(151/152): librados2-19.0.0-3291.g994e2f20.el9. 2.6 MB/s | 3.3 MB 00:01 2024-04-25T11:23:14.215 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T11:23:14.247 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T11:23:14.283 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T11:23:14.330 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T11:23:14.366 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T11:23:14.409 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T11:23:14.451 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T11:23:14.588 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T11:23:14.649 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T11:23:14.662 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-04-25T11:23:14.679 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T11:23:14.728 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T11:23:14.773 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T11:23:14.820 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T11:23:14.823 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T11:23:14.862 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T11:23:14.873 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T11:23:14.902 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T11:23:14.907 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T11:23:14.928 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T11:23:14.933 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T11:23:14.952 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T11:23:14.972 INFO:teuthology.orchestra.run.smithi138.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 2.6 MB/s | 2.4 MB 00:00 2024-04-25T11:23:14.975 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-04-25T11:23:14.975 INFO:teuthology.orchestra.run.smithi138.stdout:Total 5.8 MB/s | 214 MB 00:37 2024-04-25T11:23:14.984 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T11:23:15.027 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T11:23:15.154 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T11:23:15.193 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T11:23:15.221 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T11:23:15.255 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T11:23:15.357 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T11:23:15.365 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T11:23:15.409 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T11:23:15.439 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T11:23:15.459 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T11:23:15.474 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T11:23:15.507 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T11:23:15.548 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T11:23:15.583 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T11:23:15.584 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T11:23:15.612 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T11:23:15.672 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T11:23:15.694 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T11:23:15.722 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T11:23:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T11:23:15.753 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T11:23:15.800 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T11:23:15.889 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T11:23:15.908 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T11:23:15.947 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T11:23:15.963 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T11:23:15.975 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T11:23:16.088 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-04-25T11:23:16.090 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T11:23:16.118 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T11:23:16.173 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T11:23:16.228 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T11:23:16.236 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-04-25T11:23:16.236 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-04-25T11:23:16.246 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T11:23:16.464 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T11:23:16.496 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T11:23:16.608 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T11:23:16.690 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T11:23:16.708 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T11:23:16.735 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T11:23:16.745 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T11:23:16.769 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T11:23:16.780 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T11:23:16.845 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T11:23:16.867 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T11:23:16.888 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T11:23:16.919 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T11:23:16.940 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T11:23:16.982 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T11:23:17.006 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T11:23:17.018 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T11:23:17.038 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T11:23:17.087 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T11:23:17.141 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T11:23:17.171 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T11:23:17.215 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T11:23:17.216 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T11:23:17.244 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T11:23:17.291 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T11:23:17.322 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T11:23:17.325 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 31/156 2024-04-25T11:23:17.369 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 32/156 2024-04-25T11:23:17.398 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 33/156 2024-04-25T11:23:17.466 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T11:23:17.521 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T11:23:17.603 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T11:23:17.737 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T11:23:17.792 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T11:23:17.966 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T11:23:18.004 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-04-25T11:23:18.005 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-04-25T11:23:18.093 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T11:23:18.322 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T11:23:18.404 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T11:23:18.434 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T11:23:18.468 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T11:23:18.506 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 43/156 2024-04-25T11:23:18.557 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T11:23:19.118 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T11:23:19.162 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T11:23:19.219 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T11:23:19.291 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T11:23:19.329 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T11:23:19.330 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T11:23:19.366 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T11:23:19.416 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T11:23:19.421 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T11:23:19.443 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T11:23:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T11:23:19.477 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T11:23:19.500 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T11:23:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T11:23:19.679 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T11:23:19.723 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T11:23:19.762 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T11:23:19.797 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T11:23:19.804 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T11:23:19.840 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T11:23:19.897 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T11:23:19.929 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T11:23:19.931 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T11:23:19.961 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 100/156 2024-04-25T11:23:19.969 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T11:23:19.997 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T11:23:20.003 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T11:23:20.037 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T11:23:20.111 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T11:23:20.240 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T11:23:20.279 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T11:23:20.306 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T11:23:20.341 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T11:23:20.389 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T11:23:20.430 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T11:23:20.468 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T11:23:20.485 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T11:23:20.503 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T11:23:20.517 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T11:23:20.536 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T11:23:20.577 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T11:23:20.589 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T11:23:20.612 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T11:23:20.619 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 104/156 2024-04-25T11:23:20.700 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T11:23:20.751 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T11:23:20.799 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T11:23:20.846 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T11:23:20.952 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T11:23:21.010 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T11:23:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T11:23:21.783 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T11:23:21.851 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T11:23:21.950 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T11:23:22.000 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T11:23:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T11:23:22.130 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T11:23:22.308 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T11:23:22.335 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T11:23:22.421 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T11:23:22.868 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T11:23:22.957 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T11:23:23.061 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T11:23:23.193 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T11:23:23.521 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T11:23:23.552 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T11:23:23.642 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T11:23:23.642 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T11:23:23.642 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T11:23:23.642 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T11:23:23.670 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T11:23:23.927 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-04-25T11:23:24.026 INFO:teuthology.orchestra.run.smithi138.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T11:23:24.064 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T11:23:24.118 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T11:23:24.160 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T11:23:24.186 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T11:23:24.228 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T11:23:24.415 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T11:23:24.520 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T11:23:24.549 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T11:23:24.562 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T11:23:24.582 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T11:23:24.686 INFO:teuthology.orchestra.run.smithi138.stdout: Upgrading : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T11:23:24.811 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T11:23:24.838 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T11:23:24.903 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T11:23:24.912 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T11:23:24.950 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T11:23:24.954 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T11:23:25.043 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T11:23:25.075 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 100/156 2024-04-25T11:23:25.101 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T11:23:25.121 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T11:23:25.156 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T11:23:25.192 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T11:23:25.297 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T11:23:25.326 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T11:23:25.389 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T11:23:25.411 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T11:23:25.435 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T11:23:25.607 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T11:23:25.632 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T11:23:25.635 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T11:23:25.669 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T11:23:25.705 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T11:23:25.726 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 104/156 2024-04-25T11:23:25.775 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T11:23:25.871 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T11:23:25.901 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T11:23:25.936 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T11:23:26.000 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T11:23:26.035 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T11:23:26.078 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T11:23:26.111 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T11:23:26.139 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T11:23:26.207 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T11:23:26.261 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T11:23:26.292 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T11:23:26.327 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T11:23:26.403 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T11:23:26.438 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 31/156 2024-04-25T11:23:26.481 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 32/156 2024-04-25T11:23:26.510 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 33/156 2024-04-25T11:23:26.580 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T11:23:26.624 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T11:23:26.700 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T11:23:26.830 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T11:23:26.894 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T11:23:27.442 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T11:23:27.531 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T11:23:27.569 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T11:23:27.611 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T11:23:27.650 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 43/156 2024-04-25T11:23:27.692 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T11:23:27.832 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T11:23:27.877 INFO:teuthology.orchestra.run.smithi138.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T11:23:27.932 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T11:23:27.954 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T11:23:27.999 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T11:23:28.006 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T11:23:28.046 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T11:23:28.082 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T11:23:28.129 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T11:23:28.164 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T11:23:28.207 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T11:23:28.250 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T11:23:28.385 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T11:23:28.430 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T11:23:28.460 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T11:23:28.502 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T11:23:28.551 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T11:23:28.596 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T11:23:28.629 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T11:23:28.635 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T11:23:28.721 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T11:23:28.721 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T11:23:28.721 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T11:23:28.722 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:23:28.723 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T11:23:28.756 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T11:23:28.905 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T11:23:29.001 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T11:23:29.093 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T11:23:29.218 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T11:23:29.258 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T11:23:29.286 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T11:23:29.320 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T11:23:29.360 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T11:23:29.392 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T11:23:29.421 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T11:23:29.456 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T11:23:29.489 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T11:23:29.531 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T11:23:29.565 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T11:23:29.651 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T11:23:29.703 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T11:23:29.735 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T11:23:29.781 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T11:23:29.889 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T11:23:29.936 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T11:23:30.666 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T11:23:30.708 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T11:23:30.751 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T11:23:30.850 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T11:23:30.900 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T11:23:30.987 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T11:23:31.036 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T11:23:31.213 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T11:23:31.251 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T11:23:31.328 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T11:23:31.969 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T11:23:32.090 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T11:23:33.310 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T11:23:33.389 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T11:23:33.417 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T11:23:33.450 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T11:23:33.778 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T11:23:33.829 INFO:teuthology.orchestra.run.smithi138.stdout: Upgrading : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T11:23:33.920 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T11:23:33.951 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 100/156 2024-04-25T11:23:33.987 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T11:23:34.478 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T11:23:34.591 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T11:23:34.670 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T11:23:34.692 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 104/156 2024-04-25T11:23:35.383 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T11:23:35.383 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /sys 2024-04-25T11:23:35.383 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /proc 2024-04-25T11:23:35.384 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /mnt 2024-04-25T11:23:35.384 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /var/tmp 2024-04-25T11:23:35.384 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /home 2024-04-25T11:23:35.384 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /root 2024-04-25T11:23:35.384 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /tmp 2024-04-25T11:23:35.384 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T11:23:35.550 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T11:23:35.566 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T11:23:35.566 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:35.566 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T11:23:35.566 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T11:23:36.931 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T11:23:36.964 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T11:23:37.137 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T11:23:37.162 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T11:23:37.162 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:37.162 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T11:23:37.162 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T11:23:37.162 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T11:23:37.162 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T11:23:37.396 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T11:23:37.418 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T11:23:37.418 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:37.418 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T11:23:37.418 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T11:23:37.421 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T11:23:37.421 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T11:23:37.525 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T11:23:37.568 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T11:23:37.660 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T11:23:37.660 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T11:23:37.660 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T11:23:37.660 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-25T11:23:37.694 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T11:23:37.880 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T11:23:37.902 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T11:23:37.902 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:37.902 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T11:23:37.902 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T11:23:37.903 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T11:23:37.903 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T11:23:38.001 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T11:23:38.050 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T11:23:38.106 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T11:23:38.301 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T11:23:38.347 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T11:23:38.425 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T11:23:38.514 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T11:23:38.654 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T11:23:38.698 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T11:23:38.730 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T11:23:38.757 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T11:23:39.184 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T11:23:39.214 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T11:23:39.246 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 124/156 2024-04-25T11:23:39.308 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 125/156 2024-04-25T11:23:39.413 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T11:23:40.451 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T11:23:40.481 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T11:23:41.448 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T11:23:41.473 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T11:23:41.574 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T11:23:41.731 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 129/156 2024-04-25T11:23:41.775 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T11:23:41.798 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T11:23:41.798 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:41.798 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T11:23:41.798 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T11:23:41.798 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T11:23:41.798 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T11:23:41.842 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T11:23:41.853 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T11:23:41.952 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 132/156 2024-04-25T11:23:42.007 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 133/156 2024-04-25T11:23:42.121 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T11:23:42.121 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /sys 2024-04-25T11:23:42.121 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /proc 2024-04-25T11:23:42.121 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /mnt 2024-04-25T11:23:42.121 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /var/tmp 2024-04-25T11:23:42.121 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /home 2024-04-25T11:23:42.121 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /root 2024-04-25T11:23:42.121 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /tmp 2024-04-25T11:23:42.121 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:23:42.298 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T11:23:42.314 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T11:23:42.314 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:42.314 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T11:23:42.314 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:23:42.946 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T11:23:42.978 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T11:23:43.001 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T11:23:43.001 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:43.001 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T11:23:43.001 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T11:23:43.001 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T11:23:43.001 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T11:23:43.045 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T11:23:43.067 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T11:23:43.067 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:43.067 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T11:23:43.067 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T11:23:43.112 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T11:23:43.135 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T11:23:43.135 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:43.135 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T11:23:43.135 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T11:23:43.135 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T11:23:43.136 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T11:23:43.447 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T11:23:43.469 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T11:23:43.469 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:43.469 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T11:23:43.469 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T11:23:43.470 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T11:23:43.470 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T11:23:43.905 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T11:23:43.928 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T11:23:43.928 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:43.929 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T11:23:43.929 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T11:23:43.929 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T11:23:43.929 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:23:44.154 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T11:23:44.175 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T11:23:44.175 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:44.175 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T11:23:44.176 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T11:23:44.176 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T11:23:44.176 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:23:44.630 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T11:23:44.652 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T11:23:44.652 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:44.652 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T11:23:44.652 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T11:23:44.652 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T11:23:44.652 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:23:44.760 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T11:23:44.816 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T11:23:44.880 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T11:23:45.065 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T11:23:45.116 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T11:23:45.194 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T11:23:45.283 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T11:23:45.423 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T11:23:45.466 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T11:23:45.498 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T11:23:45.526 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T11:23:45.996 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T11:23:46.026 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T11:23:46.067 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 124/156 2024-04-25T11:23:46.136 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 125/156 2024-04-25T11:23:46.167 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T11:23:47.194 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T11:23:47.235 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T11:23:47.751 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 139/156 2024-04-25T11:23:47.792 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 140/156 2024-04-25T11:23:47.832 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 141/156 2024-04-25T11:23:47.865 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T11:23:47.892 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T11:23:48.075 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 144/156 2024-04-25T11:23:48.130 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T11:23:48.151 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 146/156 2024-04-25T11:23:48.193 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T11:23:48.236 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T11:23:48.336 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T11:23:48.504 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 129/156 2024-04-25T11:23:48.531 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T11:23:48.554 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T11:23:48.554 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:48.554 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T11:23:48.554 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T11:23:48.554 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T11:23:48.554 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:23:48.599 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T11:23:48.609 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T11:23:48.708 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 132/156 2024-04-25T11:23:48.764 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 133/156 2024-04-25T11:23:48.801 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T11:23:48.851 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T11:23:48.869 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T11:23:49.431 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T11:23:49.431 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /sys 2024-04-25T11:23:49.431 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /proc 2024-04-25T11:23:49.431 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /mnt 2024-04-25T11:23:49.431 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /var/tmp 2024-04-25T11:23:49.431 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /home 2024-04-25T11:23:49.431 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /root 2024-04-25T11:23:49.431 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /tmp 2024-04-25T11:23:49.431 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-25T11:23:49.610 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T11:23:49.626 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T11:23:49.626 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:49.626 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T11:23:49.626 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-25T11:23:49.695 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T11:23:49.713 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 148/156 2024-04-25T11:23:49.752 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T11:23:49.775 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T11:23:49.775 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:49.775 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T11:23:49.775 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T11:23:49.775 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T11:23:49.776 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:23:49.804 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T11:23:49.819 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T11:23:49.841 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T11:23:49.841 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:49.841 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T11:23:49.841 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:23:49.845 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T11:23:49.887 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T11:23:49.909 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T11:23:49.909 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:49.909 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T11:23:49.909 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T11:23:49.909 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T11:23:49.909 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:23:49.927 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T11:23:49.967 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T11:23:49.967 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T11:23:50.051 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T11:23:50.084 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T11:23:50.084 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T11:23:50.176 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T11:23:50.229 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T11:23:50.241 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T11:23:50.252 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T11:23:50.252 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:50.252 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T11:23:50.252 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T11:23:50.252 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T11:23:50.253 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:23:51.196 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T11:23:51.221 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T11:23:51.221 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:51.221 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T11:23:51.221 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T11:23:51.221 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T11:23:51.221 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-25T11:23:51.446 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T11:23:51.469 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T11:23:51.469 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:51.469 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T11:23:51.469 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T11:23:51.469 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T11:23:51.470 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-25T11:23:51.930 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T11:23:51.953 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T11:23:51.953 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:51.953 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T11:23:51.953 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T11:23:51.953 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T11:23:51.953 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-25T11:23:52.060 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T11:23:52.109 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T11:23:52.163 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T11:23:52.350 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T11:23:52.397 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T11:23:52.475 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T11:23:52.563 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T11:23:52.719 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T11:23:52.757 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T11:23:52.789 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T11:23:52.816 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T11:23:53.269 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T11:23:53.299 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T11:23:53.331 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 124/156 2024-04-25T11:23:53.393 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 125/156 2024-04-25T11:23:53.430 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T11:23:54.470 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T11:23:54.515 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T11:23:54.569 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 139/156 2024-04-25T11:23:54.618 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 140/156 2024-04-25T11:23:54.659 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 141/156 2024-04-25T11:23:54.691 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T11:23:54.719 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T11:23:54.900 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 144/156 2024-04-25T11:23:54.950 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T11:23:54.980 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 146/156 2024-04-25T11:23:55.485 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T11:23:55.516 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T11:23:55.602 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T11:23:55.617 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T11:23:55.647 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T11:23:55.666 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T11:23:55.801 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 129/156 2024-04-25T11:23:55.835 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T11:23:55.858 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T11:23:55.858 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:55.858 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T11:23:55.858 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T11:23:55.858 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T11:23:55.858 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-25T11:23:55.901 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T11:23:55.913 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T11:23:56.010 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 132/156 2024-04-25T11:23:56.058 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 133/156 2024-04-25T11:23:56.235 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T11:23:56.235 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 1/156 2024-04-25T11:23:56.235 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2/156 2024-04-25T11:23:56.235 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 3/156 2024-04-25T11:23:56.235 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 4/156 2024-04-25T11:23:56.235 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 5/156 2024-04-25T11:23:56.235 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 6/156 2024-04-25T11:23:56.236 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 7/156 2024-04-25T11:23:56.236 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T11:23:56.236 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T11:23:56.236 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T11:23:56.236 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 11/156 2024-04-25T11:23:56.236 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 12/156 2024-04-25T11:23:56.236 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T11:23:56.236 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 14/156 2024-04-25T11:23:56.236 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 15/156 2024-04-25T11:23:56.236 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 16/156 2024-04-25T11:23:56.236 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 17/156 2024-04-25T11:23:56.236 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 18/156 2024-04-25T11:23:56.237 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 19/156 2024-04-25T11:23:56.237 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 20/156 2024-04-25T11:23:56.237 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 21/156 2024-04-25T11:23:56.237 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 22/156 2024-04-25T11:23:56.237 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 23/156 2024-04-25T11:23:56.237 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 24/156 2024-04-25T11:23:56.237 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 25/156 2024-04-25T11:23:56.237 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 26/156 2024-04-25T11:23:56.237 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 27/156 2024-04-25T11:23:56.237 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 28/156 2024-04-25T11:23:56.237 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 29/156 2024-04-25T11:23:56.237 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 30/156 2024-04-25T11:23:56.252 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 31/156 2024-04-25T11:23:56.252 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 32/156 2024-04-25T11:23:56.252 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 33/156 2024-04-25T11:23:56.253 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 34/156 2024-04-25T11:23:56.253 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 35/156 2024-04-25T11:23:56.253 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 36/156 2024-04-25T11:23:56.253 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 37/156 2024-04-25T11:23:56.253 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 38/156 2024-04-25T11:23:56.253 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 39/156 2024-04-25T11:23:56.253 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T11:23:56.253 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T11:23:56.253 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T11:23:56.253 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T11:23:56.253 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T11:23:56.253 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T11:23:56.253 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T11:23:56.253 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T11:23:56.254 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T11:23:56.254 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T11:23:56.254 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T11:23:56.254 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T11:23:56.254 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T11:23:56.254 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T11:23:56.254 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T11:23:56.254 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T11:23:56.254 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T11:23:56.254 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T11:23:56.254 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T11:23:56.254 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T11:23:56.255 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T11:23:56.255 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T11:23:56.255 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T11:23:56.255 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T11:23:56.255 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T11:23:56.255 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T11:23:56.255 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T11:23:56.255 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T11:23:56.255 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T11:23:56.255 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T11:23:56.255 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T11:23:56.255 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T11:23:56.256 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T11:23:56.256 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T11:23:56.256 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T11:23:56.256 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T11:23:56.256 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T11:23:56.256 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T11:23:56.256 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T11:23:56.256 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T11:23:56.256 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T11:23:56.256 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T11:23:56.256 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T11:23:56.256 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T11:23:56.256 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T11:23:56.257 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T11:23:56.257 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T11:23:56.257 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T11:23:56.257 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T11:23:56.257 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T11:23:56.257 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T11:23:56.257 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T11:23:56.257 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T11:23:56.257 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T11:23:56.257 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T11:23:56.257 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T11:23:56.257 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T11:23:56.257 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T11:23:56.258 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T11:23:56.258 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T11:23:56.258 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T11:23:56.258 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T11:23:56.258 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T11:23:56.258 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T11:23:56.258 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T11:23:56.258 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T11:23:56.258 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T11:23:56.258 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T11:23:56.258 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T11:23:56.258 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T11:23:56.258 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T11:23:56.259 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T11:23:56.259 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T11:23:56.259 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T11:23:56.259 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T11:23:56.259 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T11:23:56.259 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T11:23:56.259 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T11:23:56.259 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T11:23:56.259 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T11:23:56.259 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T11:23:56.259 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T11:23:56.259 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T11:23:56.259 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T11:23:56.260 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T11:23:56.260 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T11:23:56.260 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T11:23:56.260 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T11:23:56.260 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T11:23:56.260 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T11:23:56.260 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T11:23:56.260 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T11:23:56.260 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T11:23:56.260 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T11:23:56.260 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T11:23:56.260 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T11:23:56.260 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T11:23:56.260 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T11:23:56.261 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T11:23:56.261 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T11:23:56.261 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T11:23:56.261 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T11:23:56.261 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T11:23:56.261 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T11:23:56.261 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T11:23:56.261 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T11:23:56.261 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T11:23:56.261 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T11:23:56.261 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T11:23:56.261 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 149/156 2024-04-25T11:23:56.261 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T11:23:56.262 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 151/156 2024-04-25T11:23:56.262 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T11:23:56.262 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T11:23:56.262 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T11:23:56.262 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T11:23:56.488 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 148/156 2024-04-25T11:23:56.631 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T11:23:56.672 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T11:23:56.761 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T11:23:56.803 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T11:23:56.803 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T11:23:56.904 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T11:23:56.937 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T11:23:56.937 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T11:23:56.989 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T11:23:57.021 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T11:23:57.032 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T11:23:57.044 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T11:23:57.044 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:57.044 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T11:23:57.044 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T11:23:57.044 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T11:23:57.044 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-25T11:23:57.088 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T11:23:57.103 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T11:23:57.110 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T11:23:57.110 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:57.110 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T11:23:57.110 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-25T11:23:57.156 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T11:23:57.178 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T11:23:57.178 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:57.178 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T11:23:57.178 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T11:23:57.178 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T11:23:57.178 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-25T11:23:57.490 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T11:23:57.512 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T11:23:57.512 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T11:23:57.512 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T11:23:57.513 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T11:23:57.513 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T11:23:57.513 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-25T11:23:58.344 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T11:23:58.345 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T11:23:58.345 INFO:teuthology.orchestra.run.smithi156.stdout:Upgraded: 2024-04-25T11:23:58.345 INFO:teuthology.orchestra.run.smithi156.stdout: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.345 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.345 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T11:23:58.345 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T11:23:58.345 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-04-25T11:23:58.345 INFO:teuthology.orchestra.run.smithi156.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T11:23:58.345 INFO:teuthology.orchestra.run.smithi156.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T11:23:58.345 INFO:teuthology.orchestra.run.smithi156.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T11:23:58.345 INFO:teuthology.orchestra.run.smithi156.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T11:23:58.345 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.345 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.346 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.346 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.346 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:23:58.346 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.346 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.346 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.346 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:23:58.346 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:23:58.346 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:23:58.346 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:23:58.346 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:23:58.346 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.346 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.347 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:23:58.347 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.347 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.347 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.347 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:23:58.347 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:23:58.347 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.347 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.347 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:23:58.347 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T11:23:58.347 INFO:teuthology.orchestra.run.smithi156.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T11:23:58.347 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T11:23:58.347 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T11:23:58.348 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T11:23:58.348 INFO:teuthology.orchestra.run.smithi156.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T11:23:58.348 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T11:23:58.348 INFO:teuthology.orchestra.run.smithi156.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T11:23:58.348 INFO:teuthology.orchestra.run.smithi156.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T11:23:58.348 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T11:23:58.348 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T11:23:58.348 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.348 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.348 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.348 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T11:23:58.348 INFO:teuthology.orchestra.run.smithi156.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T11:23:58.348 INFO:teuthology.orchestra.run.smithi156.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T11:23:58.349 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T11:23:58.349 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T11:23:58.349 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.349 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.349 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T11:23:58.349 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.349 INFO:teuthology.orchestra.run.smithi156.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T11:23:58.349 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T11:23:58.349 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T11:23:58.349 INFO:teuthology.orchestra.run.smithi156.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T11:23:58.349 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T11:23:58.349 INFO:teuthology.orchestra.run.smithi156.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T11:23:58.349 INFO:teuthology.orchestra.run.smithi156.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T11:23:58.350 INFO:teuthology.orchestra.run.smithi156.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T11:23:58.350 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T11:23:58.350 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T11:23:58.350 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T11:23:58.350 INFO:teuthology.orchestra.run.smithi156.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T11:23:58.350 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T11:23:58.350 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T11:23:58.350 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T11:23:58.350 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T11:23:58.350 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T11:23:58.350 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T11:23:58.350 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T11:23:58.350 INFO:teuthology.orchestra.run.smithi156.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T11:23:58.351 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T11:23:58.351 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T11:23:58.351 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T11:23:58.351 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.351 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.351 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.351 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T11:23:58.351 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T11:23:58.351 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T11:23:58.351 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T11:23:58.351 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T11:23:58.351 INFO:teuthology.orchestra.run.smithi156.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T11:23:58.351 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T11:23:58.351 INFO:teuthology.orchestra.run.smithi156.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T11:23:58.352 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T11:23:58.352 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T11:23:58.352 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T11:23:58.352 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T11:23:58.352 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T11:23:58.352 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T11:23:58.352 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T11:23:58.352 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T11:23:58.352 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T11:23:58.352 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T11:23:58.352 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T11:23:58.352 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T11:23:58.352 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T11:23:58.352 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T11:23:58.353 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T11:23:58.353 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T11:23:58.353 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T11:23:58.353 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T11:23:58.353 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T11:23:58.353 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T11:23:58.353 INFO:teuthology.orchestra.run.smithi156.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T11:23:58.353 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T11:23:58.353 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T11:23:58.353 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T11:23:58.353 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T11:23:58.353 INFO:teuthology.orchestra.run.smithi156.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T11:23:58.353 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T11:23:58.353 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T11:23:58.353 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T11:23:58.354 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T11:23:58.354 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T11:23:58.354 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T11:23:58.354 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.354 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.354 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T11:23:58.354 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T11:23:58.354 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T11:23:58.354 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.354 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T11:23:58.354 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T11:23:58.354 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T11:23:58.354 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T11:23:58.355 INFO:teuthology.orchestra.run.smithi156.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T11:23:58.355 INFO:teuthology.orchestra.run.smithi156.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T11:23:58.355 INFO:teuthology.orchestra.run.smithi156.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T11:23:58.355 INFO:teuthology.orchestra.run.smithi156.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T11:23:58.355 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T11:23:58.355 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T11:23:58.355 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T11:23:58.355 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T11:23:58.355 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T11:23:58.355 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T11:23:58.355 INFO:teuthology.orchestra.run.smithi156.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T11:23:58.355 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.355 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.355 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:23:58.355 INFO:teuthology.orchestra.run.smithi156.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T11:23:58.356 INFO:teuthology.orchestra.run.smithi156.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T11:23:58.356 INFO:teuthology.orchestra.run.smithi156.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T11:23:58.356 INFO:teuthology.orchestra.run.smithi156.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T11:23:58.356 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T11:23:58.356 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T11:23:58.356 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T11:23:58.356 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T11:23:58.356 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-04-25T11:23:58.728 DEBUG:teuthology.parallel:result is None 2024-04-25T11:24:01.786 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 139/156 2024-04-25T11:24:01.826 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 140/156 2024-04-25T11:24:01.867 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 141/156 2024-04-25T11:24:01.908 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T11:24:01.935 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T11:24:02.117 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 144/156 2024-04-25T11:24:02.165 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T11:24:02.194 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 146/156 2024-04-25T11:24:02.808 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T11:24:02.851 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T11:24:02.871 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T11:24:03.694 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 148/156 2024-04-25T11:24:03.800 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T11:24:03.841 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T11:24:03.923 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T11:24:03.954 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T11:24:03.954 INFO:teuthology.orchestra.run.smithi138.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T11:24:04.038 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T11:24:04.079 INFO:teuthology.orchestra.run.smithi138.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T11:24:04.080 INFO:teuthology.orchestra.run.smithi138.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T11:24:04.167 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T11:24:04.235 INFO:teuthology.orchestra.run.smithi138.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T11:24:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T11:24:05.082 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 1/156 2024-04-25T11:24:05.082 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2/156 2024-04-25T11:24:05.082 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 3/156 2024-04-25T11:24:05.082 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 4/156 2024-04-25T11:24:05.082 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 5/156 2024-04-25T11:24:05.082 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 6/156 2024-04-25T11:24:05.082 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 7/156 2024-04-25T11:24:05.082 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T11:24:05.082 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T11:24:05.082 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T11:24:05.082 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 11/156 2024-04-25T11:24:05.083 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 12/156 2024-04-25T11:24:05.083 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T11:24:05.083 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 14/156 2024-04-25T11:24:05.083 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 15/156 2024-04-25T11:24:05.083 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 16/156 2024-04-25T11:24:05.083 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 17/156 2024-04-25T11:24:05.083 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 18/156 2024-04-25T11:24:05.083 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 19/156 2024-04-25T11:24:05.083 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 20/156 2024-04-25T11:24:05.083 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 21/156 2024-04-25T11:24:05.083 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 22/156 2024-04-25T11:24:05.083 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 23/156 2024-04-25T11:24:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 24/156 2024-04-25T11:24:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 25/156 2024-04-25T11:24:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 26/156 2024-04-25T11:24:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 27/156 2024-04-25T11:24:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 28/156 2024-04-25T11:24:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 29/156 2024-04-25T11:24:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 30/156 2024-04-25T11:24:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 31/156 2024-04-25T11:24:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 32/156 2024-04-25T11:24:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 33/156 2024-04-25T11:24:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 34/156 2024-04-25T11:24:05.084 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 35/156 2024-04-25T11:24:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 36/156 2024-04-25T11:24:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 37/156 2024-04-25T11:24:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 38/156 2024-04-25T11:24:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 39/156 2024-04-25T11:24:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T11:24:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T11:24:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T11:24:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T11:24:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T11:24:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T11:24:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T11:24:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T11:24:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T11:24:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T11:24:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T11:24:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T11:24:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T11:24:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T11:24:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T11:24:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T11:24:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T11:24:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T11:24:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T11:24:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T11:24:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T11:24:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T11:24:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T11:24:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T11:24:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T11:24:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T11:24:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T11:24:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T11:24:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T11:24:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T11:24:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T11:24:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T11:24:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T11:24:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T11:24:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T11:24:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T11:24:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T11:24:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T11:24:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T11:24:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T11:24:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T11:24:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T11:24:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T11:24:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T11:24:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T11:24:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T11:24:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T11:24:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T11:24:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T11:24:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T11:24:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T11:24:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T11:24:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T11:24:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T11:24:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T11:24:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T11:24:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T11:24:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T11:24:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T11:24:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T11:24:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T11:24:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T11:24:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T11:24:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T11:24:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T11:24:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T11:24:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T11:24:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T11:24:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T11:24:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T11:24:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T11:24:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T11:24:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T11:24:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T11:24:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T11:24:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T11:24:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T11:24:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T11:24:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T11:24:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T11:24:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T11:24:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T11:24:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T11:24:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T11:24:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T11:24:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T11:24:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T11:24:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T11:24:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T11:24:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T11:24:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T11:24:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T11:24:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T11:24:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T11:24:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T11:24:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T11:24:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T11:24:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T11:24:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T11:24:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T11:24:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T11:24:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T11:24:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T11:24:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T11:24:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T11:24:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T11:24:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T11:24:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T11:24:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T11:24:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 149/156 2024-04-25T11:24:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T11:24:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 151/156 2024-04-25T11:24:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T11:24:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T11:24:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T11:24:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T11:24:07.169 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T11:24:07.169 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:24:07.169 INFO:teuthology.orchestra.run.smithi090.stdout:Upgraded: 2024-04-25T11:24:07.170 INFO:teuthology.orchestra.run.smithi090.stdout: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.170 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.170 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T11:24:07.170 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T11:24:07.170 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-04-25T11:24:07.170 INFO:teuthology.orchestra.run.smithi090.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T11:24:07.170 INFO:teuthology.orchestra.run.smithi090.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T11:24:07.170 INFO:teuthology.orchestra.run.smithi090.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T11:24:07.170 INFO:teuthology.orchestra.run.smithi090.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T11:24:07.170 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.170 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.171 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.171 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.171 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:24:07.171 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.171 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.171 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.171 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:24:07.171 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:24:07.171 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:24:07.171 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:24:07.171 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:24:07.172 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.172 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.172 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:24:07.172 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.172 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.172 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.172 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:24:07.172 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:24:07.172 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.172 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.172 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:24:07.172 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T11:24:07.173 INFO:teuthology.orchestra.run.smithi090.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T11:24:07.173 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T11:24:07.173 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T11:24:07.173 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T11:24:07.173 INFO:teuthology.orchestra.run.smithi090.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T11:24:07.173 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T11:24:07.173 INFO:teuthology.orchestra.run.smithi090.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T11:24:07.173 INFO:teuthology.orchestra.run.smithi090.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T11:24:07.173 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T11:24:07.173 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T11:24:07.173 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.173 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.174 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.174 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T11:24:07.174 INFO:teuthology.orchestra.run.smithi090.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T11:24:07.174 INFO:teuthology.orchestra.run.smithi090.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T11:24:07.174 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T11:24:07.174 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T11:24:07.174 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.174 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.174 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T11:24:07.174 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.174 INFO:teuthology.orchestra.run.smithi090.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T11:24:07.174 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T11:24:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T11:24:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T11:24:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T11:24:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T11:24:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T11:24:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T11:24:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T11:24:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T11:24:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T11:24:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T11:24:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T11:24:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T11:24:07.175 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T11:24:07.176 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T11:24:07.176 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T11:24:07.176 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T11:24:07.176 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T11:24:07.176 INFO:teuthology.orchestra.run.smithi090.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T11:24:07.176 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T11:24:07.176 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T11:24:07.176 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T11:24:07.176 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.176 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.176 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T11:24:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T11:24:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T11:24:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T11:24:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T11:24:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T11:24:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T11:24:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T11:24:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T11:24:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T11:24:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T11:24:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T11:24:07.177 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T11:24:07.178 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T11:24:07.178 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T11:24:07.178 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T11:24:07.178 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T11:24:07.178 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T11:24:07.178 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T11:24:07.178 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T11:24:07.178 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T11:24:07.178 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T11:24:07.178 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T11:24:07.178 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T11:24:07.178 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T11:24:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T11:24:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T11:24:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T11:24:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T11:24:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T11:24:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T11:24:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T11:24:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T11:24:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T11:24:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T11:24:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T11:24:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T11:24:07.179 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T11:24:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T11:24:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T11:24:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T11:24:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T11:24:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T11:24:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T11:24:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T11:24:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T11:24:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T11:24:07.180 INFO:teuthology.orchestra.run.smithi090.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T11:24:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T11:24:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T11:24:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T11:24:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T11:24:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T11:24:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T11:24:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T11:24:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T11:24:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T11:24:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T11:24:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.181 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:07.182 INFO:teuthology.orchestra.run.smithi090.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T11:24:07.182 INFO:teuthology.orchestra.run.smithi090.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T11:24:07.182 INFO:teuthology.orchestra.run.smithi090.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T11:24:07.182 INFO:teuthology.orchestra.run.smithi090.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T11:24:07.182 INFO:teuthology.orchestra.run.smithi090.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T11:24:07.182 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T11:24:07.182 INFO:teuthology.orchestra.run.smithi090.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T11:24:07.182 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:24:07.183 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-25T11:24:07.598 DEBUG:teuthology.parallel:result is None 2024-04-25T11:24:10.393 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T11:24:10.393 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 1/156 2024-04-25T11:24:10.393 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2/156 2024-04-25T11:24:10.393 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 3/156 2024-04-25T11:24:10.393 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 4/156 2024-04-25T11:24:10.393 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 5/156 2024-04-25T11:24:10.393 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 6/156 2024-04-25T11:24:10.393 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 7/156 2024-04-25T11:24:10.393 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T11:24:10.394 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T11:24:10.394 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T11:24:10.394 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 11/156 2024-04-25T11:24:10.394 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 12/156 2024-04-25T11:24:10.394 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T11:24:10.394 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 14/156 2024-04-25T11:24:10.394 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 15/156 2024-04-25T11:24:10.394 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 16/156 2024-04-25T11:24:10.394 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 17/156 2024-04-25T11:24:10.394 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 18/156 2024-04-25T11:24:10.394 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 19/156 2024-04-25T11:24:10.394 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 20/156 2024-04-25T11:24:10.394 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 21/156 2024-04-25T11:24:10.395 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 22/156 2024-04-25T11:24:10.395 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 23/156 2024-04-25T11:24:10.395 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 24/156 2024-04-25T11:24:10.395 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 25/156 2024-04-25T11:24:10.395 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 26/156 2024-04-25T11:24:10.395 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 27/156 2024-04-25T11:24:10.395 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 28/156 2024-04-25T11:24:10.395 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 29/156 2024-04-25T11:24:10.395 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 30/156 2024-04-25T11:24:10.395 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 31/156 2024-04-25T11:24:10.395 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 32/156 2024-04-25T11:24:10.395 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 33/156 2024-04-25T11:24:10.396 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 34/156 2024-04-25T11:24:10.396 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 35/156 2024-04-25T11:24:10.396 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 36/156 2024-04-25T11:24:10.396 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 37/156 2024-04-25T11:24:10.396 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 38/156 2024-04-25T11:24:10.396 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 39/156 2024-04-25T11:24:10.396 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T11:24:10.396 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T11:24:10.396 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T11:24:10.396 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T11:24:10.396 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T11:24:10.396 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T11:24:10.396 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T11:24:10.396 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T11:24:10.397 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T11:24:10.397 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T11:24:10.397 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T11:24:10.397 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T11:24:10.397 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T11:24:10.397 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T11:24:10.397 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T11:24:10.397 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T11:24:10.397 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T11:24:10.397 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T11:24:10.397 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T11:24:10.397 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T11:24:10.397 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T11:24:10.398 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T11:24:10.398 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T11:24:10.398 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T11:24:10.398 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T11:24:10.398 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T11:24:10.398 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T11:24:10.398 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T11:24:10.398 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T11:24:10.398 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T11:24:10.398 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T11:24:10.398 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T11:24:10.398 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T11:24:10.398 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T11:24:10.399 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T11:24:10.399 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T11:24:10.399 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T11:24:10.399 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T11:24:10.399 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T11:24:10.399 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T11:24:10.399 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T11:24:10.399 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T11:24:10.399 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T11:24:10.399 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T11:24:10.399 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T11:24:10.399 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T11:24:10.399 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T11:24:10.400 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T11:24:10.400 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T11:24:10.400 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T11:24:10.400 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T11:24:10.400 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T11:24:10.400 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T11:24:10.400 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T11:24:10.400 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T11:24:10.400 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T11:24:10.400 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T11:24:10.400 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T11:24:10.400 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T11:24:10.400 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T11:24:10.400 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T11:24:10.401 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T11:24:10.401 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T11:24:10.401 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T11:24:10.401 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T11:24:10.401 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T11:24:10.401 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T11:24:10.401 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T11:24:10.401 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T11:24:10.401 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T11:24:10.401 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T11:24:10.401 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T11:24:10.401 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T11:24:10.401 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T11:24:10.402 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T11:24:10.402 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T11:24:10.402 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T11:24:10.402 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T11:24:10.402 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T11:24:10.402 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T11:24:10.402 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T11:24:10.402 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T11:24:10.402 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T11:24:10.402 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T11:24:10.402 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T11:24:10.402 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T11:24:10.402 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T11:24:10.403 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T11:24:10.403 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T11:24:10.403 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T11:24:10.403 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T11:24:10.403 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T11:24:10.403 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T11:24:10.403 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T11:24:10.403 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T11:24:10.403 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T11:24:10.403 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T11:24:10.403 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T11:24:10.403 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T11:24:10.403 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T11:24:10.404 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T11:24:10.404 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T11:24:10.404 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T11:24:10.404 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T11:24:10.404 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T11:24:10.404 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T11:24:10.404 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T11:24:10.404 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T11:24:10.404 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T11:24:10.404 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 149/156 2024-04-25T11:24:10.404 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T11:24:10.404 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 151/156 2024-04-25T11:24:10.404 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T11:24:10.405 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T11:24:10.405 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T11:24:10.405 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T11:24:12.489 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T11:24:12.490 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-25T11:24:12.490 INFO:teuthology.orchestra.run.smithi138.stdout:Upgraded: 2024-04-25T11:24:12.490 INFO:teuthology.orchestra.run.smithi138.stdout: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.490 INFO:teuthology.orchestra.run.smithi138.stdout: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.490 INFO:teuthology.orchestra.run.smithi138.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T11:24:12.490 INFO:teuthology.orchestra.run.smithi138.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T11:24:12.490 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-04-25T11:24:12.490 INFO:teuthology.orchestra.run.smithi138.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T11:24:12.490 INFO:teuthology.orchestra.run.smithi138.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T11:24:12.491 INFO:teuthology.orchestra.run.smithi138.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T11:24:12.491 INFO:teuthology.orchestra.run.smithi138.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T11:24:12.491 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.491 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.491 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.491 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.491 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:24:12.491 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.491 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.491 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.491 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:24:12.491 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:24:12.492 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:24:12.492 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:24:12.492 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:24:12.492 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.492 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.492 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:24:12.492 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.492 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.492 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.492 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:24:12.492 INFO:teuthology.orchestra.run.smithi138.stdout: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:24:12.492 INFO:teuthology.orchestra.run.smithi138.stdout: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.492 INFO:teuthology.orchestra.run.smithi138.stdout: cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.492 INFO:teuthology.orchestra.run.smithi138.stdout: cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T11:24:12.492 INFO:teuthology.orchestra.run.smithi138.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T11:24:12.493 INFO:teuthology.orchestra.run.smithi138.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T11:24:12.493 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T11:24:12.493 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T11:24:12.493 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T11:24:12.493 INFO:teuthology.orchestra.run.smithi138.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T11:24:12.493 INFO:teuthology.orchestra.run.smithi138.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T11:24:12.493 INFO:teuthology.orchestra.run.smithi138.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T11:24:12.493 INFO:teuthology.orchestra.run.smithi138.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T11:24:12.493 INFO:teuthology.orchestra.run.smithi138.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T11:24:12.493 INFO:teuthology.orchestra.run.smithi138.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T11:24:12.493 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.493 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.493 INFO:teuthology.orchestra.run.smithi138.stdout: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.493 INFO:teuthology.orchestra.run.smithi138.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T11:24:12.494 INFO:teuthology.orchestra.run.smithi138.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T11:24:12.494 INFO:teuthology.orchestra.run.smithi138.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T11:24:12.494 INFO:teuthology.orchestra.run.smithi138.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T11:24:12.494 INFO:teuthology.orchestra.run.smithi138.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T11:24:12.494 INFO:teuthology.orchestra.run.smithi138.stdout: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.494 INFO:teuthology.orchestra.run.smithi138.stdout: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.494 INFO:teuthology.orchestra.run.smithi138.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T11:24:12.494 INFO:teuthology.orchestra.run.smithi138.stdout: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.494 INFO:teuthology.orchestra.run.smithi138.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T11:24:12.494 INFO:teuthology.orchestra.run.smithi138.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T11:24:12.494 INFO:teuthology.orchestra.run.smithi138.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T11:24:12.494 INFO:teuthology.orchestra.run.smithi138.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T11:24:12.494 INFO:teuthology.orchestra.run.smithi138.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T11:24:12.495 INFO:teuthology.orchestra.run.smithi138.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T11:24:12.495 INFO:teuthology.orchestra.run.smithi138.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T11:24:12.495 INFO:teuthology.orchestra.run.smithi138.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T11:24:12.495 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T11:24:12.495 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T11:24:12.495 INFO:teuthology.orchestra.run.smithi138.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T11:24:12.495 INFO:teuthology.orchestra.run.smithi138.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T11:24:12.495 INFO:teuthology.orchestra.run.smithi138.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T11:24:12.495 INFO:teuthology.orchestra.run.smithi138.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T11:24:12.495 INFO:teuthology.orchestra.run.smithi138.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T11:24:12.495 INFO:teuthology.orchestra.run.smithi138.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T11:24:12.495 INFO:teuthology.orchestra.run.smithi138.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T11:24:12.495 INFO:teuthology.orchestra.run.smithi138.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T11:24:12.495 INFO:teuthology.orchestra.run.smithi138.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T11:24:12.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T11:24:12.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T11:24:12.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T11:24:12.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T11:24:12.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T11:24:12.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T11:24:12.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T11:24:12.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T11:24:12.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T11:24:12.496 INFO:teuthology.orchestra.run.smithi138.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T11:24:12.497 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T11:24:12.497 INFO:teuthology.orchestra.run.smithi138.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T11:24:12.497 INFO:teuthology.orchestra.run.smithi138.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T11:24:12.497 INFO:teuthology.orchestra.run.smithi138.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T11:24:12.497 INFO:teuthology.orchestra.run.smithi138.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T11:24:12.497 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T11:24:12.497 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T11:24:12.497 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T11:24:12.497 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T11:24:12.497 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T11:24:12.497 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T11:24:12.497 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T11:24:12.497 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T11:24:12.498 INFO:teuthology.orchestra.run.smithi138.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T11:24:12.498 INFO:teuthology.orchestra.run.smithi138.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T11:24:12.498 INFO:teuthology.orchestra.run.smithi138.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T11:24:12.498 INFO:teuthology.orchestra.run.smithi138.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T11:24:12.498 INFO:teuthology.orchestra.run.smithi138.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T11:24:12.498 INFO:teuthology.orchestra.run.smithi138.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T11:24:12.498 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T11:24:12.498 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T11:24:12.498 INFO:teuthology.orchestra.run.smithi138.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T11:24:12.498 INFO:teuthology.orchestra.run.smithi138.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T11:24:12.498 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T11:24:12.498 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T11:24:12.498 INFO:teuthology.orchestra.run.smithi138.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T11:24:12.499 INFO:teuthology.orchestra.run.smithi138.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T11:24:12.499 INFO:teuthology.orchestra.run.smithi138.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T11:24:12.499 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T11:24:12.499 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T11:24:12.499 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T11:24:12.499 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T11:24:12.499 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T11:24:12.499 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T11:24:12.499 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.499 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.499 INFO:teuthology.orchestra.run.smithi138.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T11:24:12.499 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T11:24:12.499 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T11:24:12.500 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.500 INFO:teuthology.orchestra.run.smithi138.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T11:24:12.500 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T11:24:12.500 INFO:teuthology.orchestra.run.smithi138.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T11:24:12.500 INFO:teuthology.orchestra.run.smithi138.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T11:24:12.500 INFO:teuthology.orchestra.run.smithi138.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T11:24:12.500 INFO:teuthology.orchestra.run.smithi138.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T11:24:12.500 INFO:teuthology.orchestra.run.smithi138.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T11:24:12.500 INFO:teuthology.orchestra.run.smithi138.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T11:24:12.500 INFO:teuthology.orchestra.run.smithi138.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T11:24:12.500 INFO:teuthology.orchestra.run.smithi138.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T11:24:12.500 INFO:teuthology.orchestra.run.smithi138.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T11:24:12.500 INFO:teuthology.orchestra.run.smithi138.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T11:24:12.500 INFO:teuthology.orchestra.run.smithi138.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T11:24:12.501 INFO:teuthology.orchestra.run.smithi138.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T11:24:12.501 INFO:teuthology.orchestra.run.smithi138.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T11:24:12.501 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.501 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.501 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T11:24:12.501 INFO:teuthology.orchestra.run.smithi138.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T11:24:12.501 INFO:teuthology.orchestra.run.smithi138.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T11:24:12.501 INFO:teuthology.orchestra.run.smithi138.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T11:24:12.501 INFO:teuthology.orchestra.run.smithi138.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T11:24:12.501 INFO:teuthology.orchestra.run.smithi138.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T11:24:12.501 INFO:teuthology.orchestra.run.smithi138.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T11:24:12.502 INFO:teuthology.orchestra.run.smithi138.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T11:24:12.502 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-25T11:24:12.502 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-25T11:24:12.878 DEBUG:teuthology.parallel:result is None 2024-04-25T11:24:12.879 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T11:24:13.028 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T11:24:13.053 INFO:teuthology.orchestra.run.smithi090.stdout:19.0.0-3291.g994e2f20.el9 2024-04-25T11:24:13.054 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291.g994e2f20.el9 2024-04-25T11:24:13.054 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291.g994e2f20 is installed. 2024-04-25T11:24:13.056 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T11:24:13.203 DEBUG:teuthology.orchestra.run.smithi138:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T11:24:13.226 INFO:teuthology.orchestra.run.smithi138.stdout:19.0.0-3291.g994e2f20.el9 2024-04-25T11:24:13.226 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291.g994e2f20.el9 2024-04-25T11:24:13.226 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291.g994e2f20 is installed. 2024-04-25T11:24:13.228 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T11:24:13.512 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T11:24:13.538 INFO:teuthology.orchestra.run.smithi156.stdout:19.0.0-3291.g994e2f20.el9 2024-04-25T11:24:13.538 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291.g994e2f20.el9 2024-04-25T11:24:13.539 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291.g994e2f20 is installed. 2024-04-25T11:24:13.540 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T11:24:13.541 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T11:24:13.541 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T11:24:13.571 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-25T11:24:13.571 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T11:24:13.599 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-25T11:24:13.599 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T11:24:13.630 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T11:24:13.630 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T11:24:13.631 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T11:24:13.659 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T11:24:13.728 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-25T11:24:13.728 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T11:24:13.752 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T11:24:13.816 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-25T11:24:13.816 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T11:24:13.844 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T11:24:13.913 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T11:24:13.913 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T11:24:13.914 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T11:24:13.941 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T11:24:14.009 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-25T11:24:14.009 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T11:24:14.033 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T11:24:14.097 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-25T11:24:14.097 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T11:24:14.122 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T11:24:14.186 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T11:24:14.186 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T11:24:14.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T11:24:14.214 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T11:24:14.281 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-25T11:24:14.282 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T11:24:14.305 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T11:24:14.370 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-25T11:24:14.370 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T11:24:14.395 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T11:24:14.458 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-25T11:24:14.577 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-25T11:24:14.578 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T11:24:14.578 INFO:tasks.cephadm:Cluster fsid is 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:24:14.578 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-25T11:24:14.579 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.90', 'mon.b': '172.21.15.138', 'mon.c': '172.21.15.156'} 2024-04-25T11:24:14.579 INFO:tasks.cephadm:First mon is mon.a on smithi090 2024-04-25T11:24:14.579 INFO:tasks.cephadm:First mgr is x 2024-04-25T11:24:14.579 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-25T11:24:14.579 DEBUG:teuthology.orchestra.run.smithi090:> sudo hostname $(hostname -s) 2024-04-25T11:24:14.607 DEBUG:teuthology.orchestra.run.smithi138:> sudo hostname $(hostname -s) 2024-04-25T11:24:14.631 DEBUG:teuthology.orchestra.run.smithi156:> sudo hostname $(hostname -s) 2024-04-25T11:24:14.655 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-25T11:24:14.655 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T11:24:14.873 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78872/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3291-g994e2f20', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3291.g994e2f20'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-25 04:02:18.990013', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20240425.015853-debug', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-04-25T11:24:15.093 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-pdonnell-testing-20240425.015853-debug, sha1 994e2f20102a20fa9a27966c14b625cc01884df8 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T11:24:15.095 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/x86_64/flavors/default/cephadm 2024-04-25T11:24:15.095 DEBUG:teuthology.orchestra.run.smithi090:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T11:24:16.402 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r--. 1 ubuntu ubuntu 781647 Apr 25 11:24 /home/ubuntu/cephtest/cephadm 2024-04-25T11:24:16.402 DEBUG:teuthology.orchestra.run.smithi138:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T11:24:17.223 INFO:teuthology.orchestra.run.smithi138.stdout:-rw-r--r--. 1 ubuntu ubuntu 781647 Apr 25 11:24 /home/ubuntu/cephtest/cephadm 2024-04-25T11:24:17.223 DEBUG:teuthology.orchestra.run.smithi156:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T11:24:17.871 INFO:teuthology.orchestra.run.smithi156.stdout:-rw-r--r--. 1 ubuntu ubuntu 781647 Apr 25 11:24 /home/ubuntu/cephtest/cephadm 2024-04-25T11:24:17.871 DEBUG:teuthology.orchestra.run.smithi090:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T11:24:17.889 DEBUG:teuthology.orchestra.run.smithi138:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T11:24:17.908 DEBUG:teuthology.orchestra.run.smithi156:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T11:24:17.943 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 on all hosts... 2024-04-25T11:24:17.944 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 pull 2024-04-25T11:24:17.946 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 pull 2024-04-25T11:24:17.950 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 pull 2024-04-25T11:24:19.495 INFO:teuthology.orchestra.run.smithi156.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8... 2024-04-25T11:24:19.665 INFO:teuthology.orchestra.run.smithi138.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8... 2024-04-25T11:24:20.461 INFO:teuthology.orchestra.run.smithi090.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8... 2024-04-25T11:24:48.051 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-04-25T11:24:48.051 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T11:24:48.051 INFO:teuthology.orchestra.run.smithi156.stdout: "image_id": "d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4", 2024-04-25T11:24:48.051 INFO:teuthology.orchestra.run.smithi156.stdout: "repo_digests": [ 2024-04-25T11:24:48.051 INFO:teuthology.orchestra.run.smithi156.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385" 2024-04-25T11:24:48.051 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2024-04-25T11:24:48.051 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-04-25T11:24:51.022 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:24:51.022 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T11:24:51.022 INFO:teuthology.orchestra.run.smithi090.stdout: "image_id": "d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4", 2024-04-25T11:24:51.023 INFO:teuthology.orchestra.run.smithi090.stdout: "repo_digests": [ 2024-04-25T11:24:51.023 INFO:teuthology.orchestra.run.smithi090.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385" 2024-04-25T11:24:51.023 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-25T11:24:51.023 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:24:57.296 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-25T11:24:57.296 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T11:24:57.296 INFO:teuthology.orchestra.run.smithi138.stdout: "image_id": "d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4", 2024-04-25T11:24:57.296 INFO:teuthology.orchestra.run.smithi138.stdout: "repo_digests": [ 2024-04-25T11:24:57.296 INFO:teuthology.orchestra.run.smithi138.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385" 2024-04-25T11:24:57.296 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-25T11:24:57.296 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-25T11:24:57.322 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /etc/ceph 2024-04-25T11:24:57.353 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /etc/ceph 2024-04-25T11:24:57.379 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /etc/ceph 2024-04-25T11:24:57.411 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 777 /etc/ceph 2024-04-25T11:24:57.438 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 777 /etc/ceph 2024-04-25T11:24:57.462 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 777 /etc/ceph 2024-04-25T11:24:57.489 INFO:tasks.cephadm:Writing seed config... 2024-04-25T11:24:57.490 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-25T11:24:57.490 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-25T11:24:57.490 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-25T11:24:57.490 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-25T11:24:57.490 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-25T11:24:57.490 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-25T11:24:57.490 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-25T11:24:57.490 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-25T11:24:57.490 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-25T11:24:57.491 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-25T11:24:57.491 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-25T11:24:57.491 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-25T11:24:57.491 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-25T11:24:57.491 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-25T11:24:57.491 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-25T11:24:57.491 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-25T11:24:57.491 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-25T11:24:57.491 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-04-25T11:24:57.491 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-25T11:24:57.491 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-25T11:24:57.491 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-25T11:24:57.491 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-25T11:24:57.492 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-25T11:24:57.492 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-25T11:24:57.492 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-25T11:24:57.492 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-25T11:24:57.492 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-25T11:24:57.492 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-25T11:24:57.492 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-25T11:24:57.492 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-25T11:24:57.492 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-25T11:24:57.492 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-25T11:24:57.492 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-25T11:24:57.492 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-25T11:24:57.492 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-25T11:24:57.492 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-25T11:24:57.493 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-25T11:24:57.493 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-25T11:24:57.493 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-25T11:24:57.493 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-25T11:24:57.493 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-25T11:24:57.493 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-25T11:24:57.493 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-25T11:24:57.493 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-25T11:24:57.493 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-25T11:24:57.493 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-25T11:24:57.493 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-25T11:24:57.493 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-25T11:24:57.493 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-25T11:24:57.494 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T11:24:57.494 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-25T11:24:57.510 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 58bd4e7a-02f6-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-25T11:24:57.511 DEBUG:teuthology.orchestra.run.smithi090:mon.a> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mon.a.service 2024-04-25T11:24:57.553 DEBUG:teuthology.orchestra.run.smithi090:mgr.x> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mgr.x.service 2024-04-25T11:24:57.596 INFO:tasks.cephadm:Bootstrapping... 2024-04-25T11:24:57.596 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 -v bootstrap --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.90 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-25T11:24:57.786 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-04-25T11:24:57.786 INFO:teuthology.orchestra.run.smithi090.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8', '-v', 'bootstrap', '--fsid', '58bd4e7a-02f6-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.90', '--skip-admin-label'] 2024-04-25T11:24:57.787 INFO:teuthology.orchestra.run.smithi090.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-25T11:24:57.787 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying podman|docker is present... 2024-04-25T11:24:57.811 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T11:24:57.811 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying lvm2 is present... 2024-04-25T11:24:57.811 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying time synchronization is in place... 2024-04-25T11:24:57.819 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T11:24:57.819 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T11:24:57.824 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T11:24:57.825 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-04-25T11:24:57.831 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout enabled 2024-04-25T11:24:57.836 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout active 2024-04-25T11:24:57.837 INFO:teuthology.orchestra.run.smithi090.stdout:Unit chronyd.service is enabled and running 2024-04-25T11:24:57.837 INFO:teuthology.orchestra.run.smithi090.stdout:Repeating the final host check... 2024-04-25T11:24:57.861 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T11:24:57.861 INFO:teuthology.orchestra.run.smithi090.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-25T11:24:57.861 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl is present 2024-04-25T11:24:57.861 INFO:teuthology.orchestra.run.smithi090.stdout:lvcreate is present 2024-04-25T11:24:57.867 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T11:24:57.867 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T11:24:57.873 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T11:24:57.873 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-04-25T11:24:57.879 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout enabled 2024-04-25T11:24:57.885 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout active 2024-04-25T11:24:57.885 INFO:teuthology.orchestra.run.smithi090.stdout:Unit chronyd.service is enabled and running 2024-04-25T11:24:57.885 INFO:teuthology.orchestra.run.smithi090.stdout:Host looks OK 2024-04-25T11:24:57.885 INFO:teuthology.orchestra.run.smithi090.stdout:Cluster fsid: 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:24:57.885 INFO:teuthology.orchestra.run.smithi090.stdout:Acquiring lock 139637454675104 on /run/cephadm/58bd4e7a-02f6-11ef-bc93-c7b262605968.lock 2024-04-25T11:24:57.885 INFO:teuthology.orchestra.run.smithi090.stdout:Lock 139637454675104 acquired on /run/cephadm/58bd4e7a-02f6-11ef-bc93-c7b262605968.lock 2024-04-25T11:24:57.886 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying IP 172.21.15.90 port 3300 ... 2024-04-25T11:24:57.886 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying IP 172.21.15.90 port 6789 ... 2024-04-25T11:24:57.886 INFO:teuthology.orchestra.run.smithi090.stdout:Base mon IP(s) is [172.21.15.90:3300, 172.21.15.90:6789], mon addrv is [v2:172.21.15.90:3300,v1:172.21.15.90:6789] 2024-04-25T11:24:57.889 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.90 metric 100 2024-04-25T11:24:57.889 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.90 metric 100 2024-04-25T11:24:57.893 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-25T11:24:57.893 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-04-25T11:24:57.893 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-04-25T11:24:57.894 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-25T11:24:57.894 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-25T11:24:57.894 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T11:24:57.894 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-25T11:24:57.894 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout inet6 fe80::cbf8:23dd:4647:b3ed/64 scope link noprefixroute 2024-04-25T11:24:57.894 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T11:24:57.895 INFO:teuthology.orchestra.run.smithi090.stdout:Mon IP `172.21.15.90` is in CIDR network `172.21.0.0/20` 2024-04-25T11:24:57.895 INFO:teuthology.orchestra.run.smithi090.stdout:Mon IP `172.21.15.90` is in CIDR network `172.21.0.0/20` 2024-04-25T11:24:57.895 INFO:teuthology.orchestra.run.smithi090.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-25T11:24:57.895 INFO:teuthology.orchestra.run.smithi090.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-25T11:24:57.896 INFO:teuthology.orchestra.run.smithi090.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8... 2024-04-25T11:24:58.374 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stdout d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 2024-04-25T11:24:58.374 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8... 2024-04-25T11:24:58.374 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Getting image source signatures 2024-04-25T11:24:58.374 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Copying blob sha256:665339d96aad80d083ba9988a62422bcef0a42210fb2b28340e33ab98589f7b8 2024-04-25T11:24:58.374 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-25T11:24:58.374 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Copying config sha256:d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 2024-04-25T11:24:58.375 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-25T11:25:08.049 INFO:teuthology.orchestra.run.smithi090.stdout:ceph: stdout ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T11:25:08.049 INFO:teuthology.orchestra.run.smithi090.stdout:Ceph version: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T11:25:08.050 INFO:teuthology.orchestra.run.smithi090.stdout:Extracting ceph user uid/gid from container image... 2024-04-25T11:25:15.311 INFO:teuthology.orchestra.run.smithi090.stdout:stat: stdout 167 167 2024-04-25T11:25:15.311 INFO:teuthology.orchestra.run.smithi090.stdout:Creating initial keys... 2024-04-25T11:25:17.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph-authtool: stdout AQCbPSpmgSlEJxAAr5WCd/7EGsj0+QgZ1QV8Iw== 2024-04-25T11:25:18.191 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph-authtool: stdout AQCdPSpm7oe0JxAAIk+yCBNo1hoqACYw70Z0cA== 2024-04-25T11:25:19.016 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph-authtool: stdout AQCePSpmL4UUHxAAzixBg2viefpjqlXkG/z3wg== 2024-04-25T11:25:19.017 INFO:teuthology.orchestra.run.smithi090.stdout:Creating initial monmap... 2024-04-25T11:25:19.943 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T11:25:19.943 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-25T11:25:19.943 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:25:19.943 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T11:25:19.943 INFO:teuthology.orchestra.run.smithi090.stdout:monmaptool for a [v2:172.21.15.90:3300,v1:172.21.15.90:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T11:25:19.943 INFO:teuthology.orchestra.run.smithi090.stdout:setting min_mon_release = quincy 2024-04-25T11:25:19.943 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: set fsid to 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:25:19.943 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T11:25:19.943 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:25:19.943 INFO:teuthology.orchestra.run.smithi090.stdout:Creating mon... 2024-04-25T11:25:22.148 INFO:teuthology.orchestra.run.smithi090.stdout:create mon.a on 2024-04-25T11:25:22.339 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-25T11:25:22.484 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-25T11:25:22.677 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968.target. 2024-04-25T11:25:22.677 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968.target. 2024-04-25T11:25:22.907 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mon.a 2024-04-25T11:25:22.907 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Failed to reset failed state of unit ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-25T11:25:23.072 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968.target.wants/ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@.service. 2024-04-25T11:25:23.610 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T11:25:23.610 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout disabled 2024-04-25T11:25:23.616 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T11:25:23.616 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-04-25T11:25:23.616 INFO:teuthology.orchestra.run.smithi090.stdout:firewalld.service is not enabled 2024-04-25T11:25:23.616 INFO:teuthology.orchestra.run.smithi090.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T11:25:23.617 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mon to start... 2024-04-25T11:25:23.617 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mon... 2024-04-25T11:25:24.666 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout cluster: 2024-04-25T11:25:24.666 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout id: 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:25:24.666 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-25T11:25:24.666 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-25T11:25:24.666 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout services: 2024-04-25T11:25:24.666 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.547006s) 2024-04-25T11:25:24.666 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-25T11:25:24.666 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-25T11:25:24.666 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-25T11:25:24.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout data: 2024-04-25T11:25:24.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-25T11:25:24.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-25T11:25:24.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-25T11:25:24.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout pgs: 2024-04-25T11:25:24.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-25T11:25:24.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.043+0000 7f6cbf78c700 1 Processor -- start 2024-04-25T11:25:24.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.043+0000 7f6cbf78c700 1 -- start start 2024-04-25T11:25:24.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.043+0000 7f6cbf78c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb8108ab0 0x7f6cb8108ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:24.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.043+0000 7f6cbf78c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb81094b0 con 0x7f6cb8108ab0 2024-04-25T11:25:24.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.043+0000 7f6cbd528700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb8108ab0 0x7f6cb8108ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:24.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.043+0000 7f6cbd528700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb8108ab0 0x7f6cb8108ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51488/0 (socket says 172.21.15.90:51488) 2024-04-25T11:25:24.667 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.043+0000 7f6cbd528700 1 -- 172.21.15.90:0/3345827884 learned_addr learned my addr 172.21.15.90:0/3345827884 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:24.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.044+0000 7f6cbd528700 1 -- 172.21.15.90:0/3345827884 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cb8109be0 con 0x7f6cb8108ab0 2024-04-25T11:25:24.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.044+0000 7f6cbd528700 1 --2- 172.21.15.90:0/3345827884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb8108ab0 0x7f6cb8108ec0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6cb40098a0 tx=0x7f6cb4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=643b6d31e20f93dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:24.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.044+0000 7f6caffff700 1 -- 172.21.15.90:0/3345827884 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6cb4004160 con 0x7f6cb8108ab0 2024-04-25T11:25:24.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.044+0000 7f6caffff700 1 -- 172.21.15.90:0/3345827884 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cb4004300 con 0x7f6cb8108ab0 2024-04-25T11:25:24.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.044+0000 7f6caffff700 1 -- 172.21.15.90:0/3345827884 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6cb40036a0 con 0x7f6cb8108ab0 2024-04-25T11:25:24.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.045+0000 7f6cbf78c700 1 -- 172.21.15.90:0/3345827884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb8108ab0 msgr2=0x7f6cb8108ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:24.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.045+0000 7f6cbf78c700 1 --2- 172.21.15.90:0/3345827884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb8108ab0 0x7f6cb8108ec0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6cb40098a0 tx=0x7f6cb4009bb0 comp rx=0 tx=0).stop 2024-04-25T11:25:24.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.045+0000 7f6cbf78c700 1 -- 172.21.15.90:0/3345827884 shutdown_connections 2024-04-25T11:25:24.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.045+0000 7f6cbf78c700 1 --2- 172.21.15.90:0/3345827884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb8108ab0 0x7f6cb8108ec0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:24.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.045+0000 7f6cbf78c700 1 -- 172.21.15.90:0/3345827884 >> 172.21.15.90:0/3345827884 conn(0x7f6cb81040f0 msgr2=0x7f6cb81065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:24.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.045+0000 7f6cbf78c700 1 -- 172.21.15.90:0/3345827884 shutdown_connections 2024-04-25T11:25:24.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.045+0000 7f6cbf78c700 1 -- 172.21.15.90:0/3345827884 wait complete. 2024-04-25T11:25:24.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.046+0000 7f6cbf78c700 1 Processor -- start 2024-04-25T11:25:24.668 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.046+0000 7f6cbf78c700 1 -- start start 2024-04-25T11:25:24.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.046+0000 7f6cbf78c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb8108ab0 0x7f6cb807d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:24.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.046+0000 7f6cbf78c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb810a110 con 0x7f6cb8108ab0 2024-04-25T11:25:24.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.046+0000 7f6cbd528700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb8108ab0 0x7f6cb807d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:24.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.046+0000 7f6cbd528700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb8108ab0 0x7f6cb807d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51496/0 (socket says 172.21.15.90:51496) 2024-04-25T11:25:24.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.046+0000 7f6cbd528700 1 -- 172.21.15.90:0/68999288 learned_addr learned my addr 172.21.15.90:0/68999288 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:24.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.046+0000 7f6cbd528700 1 -- 172.21.15.90:0/68999288 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cb807d770 con 0x7f6cb8108ab0 2024-04-25T11:25:24.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.046+0000 7f6cbd528700 1 --2- 172.21.15.90:0/68999288 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb8108ab0 0x7f6cb807d230 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6cb4000c00 tx=0x7f6cb4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:24.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.046+0000 7f6cae7fc700 1 -- 172.21.15.90:0/68999288 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6cb4003db0 con 0x7f6cb8108ab0 2024-04-25T11:25:24.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.046+0000 7f6cae7fc700 1 -- 172.21.15.90:0/68999288 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cb4011630 con 0x7f6cb8108ab0 2024-04-25T11:25:24.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.046+0000 7f6cbf78c700 1 -- 172.21.15.90:0/68999288 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cb81a0460 con 0x7f6cb8108ab0 2024-04-25T11:25:24.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.046+0000 7f6cae7fc700 1 -- 172.21.15.90:0/68999288 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6cb4011950 con 0x7f6cb8108ab0 2024-04-25T11:25:24.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.047+0000 7f6cbf78c700 1 -- 172.21.15.90:0/68999288 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cb81a08c0 con 0x7f6cb8108ab0 2024-04-25T11:25:24.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.047+0000 7f6cae7fc700 1 -- 172.21.15.90:0/68999288 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6cb401a460 con 0x7f6cb8108ab0 2024-04-25T11:25:24.669 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.047+0000 7f6cae7fc700 1 -- 172.21.15.90:0/68999288 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6cb401b780 con 0x7f6cb8108ab0 2024-04-25T11:25:24.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.047+0000 7f6cbf78c700 1 -- 172.21.15.90:0/68999288 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cb8064cf0 con 0x7f6cb8108ab0 2024-04-25T11:25:24.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.050+0000 7f6cae7fc700 1 -- 172.21.15.90:0/68999288 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f6cb401b980 con 0x7f6cb8108ab0 2024-04-25T11:25:24.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.120+0000 7f6cbf78c700 1 -- 172.21.15.90:0/68999288 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f6cb802e790 con 0x7f6cb8108ab0 2024-04-25T11:25:24.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.120+0000 7f6cae7fc700 1 -- 172.21.15.90:0/68999288 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f6cb401bb60 con 0x7f6cb8108ab0 2024-04-25T11:25:24.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.121+0000 7f6cbf78c700 1 -- 172.21.15.90:0/68999288 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb8108ab0 msgr2=0x7f6cb807d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:24.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.121+0000 7f6cbf78c700 1 --2- 172.21.15.90:0/68999288 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb8108ab0 0x7f6cb807d230 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6cb4000c00 tx=0x7f6cb4004750 comp rx=0 tx=0).stop 2024-04-25T11:25:24.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.121+0000 7f6cbf78c700 1 -- 172.21.15.90:0/68999288 shutdown_connections 2024-04-25T11:25:24.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.121+0000 7f6cbf78c700 1 --2- 172.21.15.90:0/68999288 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb8108ab0 0x7f6cb807d230 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:24.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.121+0000 7f6cbf78c700 1 -- 172.21.15.90:0/68999288 >> 172.21.15.90:0/68999288 conn(0x7f6cb81040f0 msgr2=0x7f6cb8192c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:24.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.121+0000 7f6cbf78c700 1 -- 172.21.15.90:0/68999288 shutdown_connections 2024-04-25T11:25:24.670 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:24.121+0000 7f6cbf78c700 1 -- 172.21.15.90:0/68999288 wait complete. 2024-04-25T11:25:24.670 INFO:teuthology.orchestra.run.smithi090.stdout:mon is available 2024-04-25T11:25:24.671 INFO:teuthology.orchestra.run.smithi090.stdout:Assimilating anything we can from ceph.conf... 2024-04-25T11:25:25.837 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:25 smithi090 ceph-mon[29197]: from='client.? 172.21.15.90:0/3337507578' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-25T11:25:25.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-25T11:25:25.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T11:25:25.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout fsid = 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:25:25.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-25T11:25:25.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.90:3300,v1:172.21.15.90:6789] 2024-04-25T11:25:25.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T11:25:25.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T11:25:25.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T11:25:25.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T11:25:25.846 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-25T11:25:25.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T11:25:25.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T11:25:25.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-25T11:25:25.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T11:25:25.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-04-25T11:25:25.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-25T11:25:25.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-25T11:25:25.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-25T11:25:25.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T11:25:25.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T11:25:25.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.119+0000 7f840af21700 1 Processor -- start 2024-04-25T11:25:25.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.119+0000 7f840af21700 1 -- start start 2024-04-25T11:25:25.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.119+0000 7f840af21700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8404109bf0 0x7f840410a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:25.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.119+0000 7f840af21700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f840410a5f0 con 0x7f8404109bf0 2024-04-25T11:25:25.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.119+0000 7f8408cbd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8404109bf0 0x7f840410a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:25.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.119+0000 7f8408cbd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8404109bf0 0x7f840410a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51512/0 (socket says 172.21.15.90:51512) 2024-04-25T11:25:25.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.119+0000 7f8408cbd700 1 -- 172.21.15.90:0/374618633 learned_addr learned my addr 172.21.15.90:0/374618633 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:25.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.120+0000 7f8408cbd700 1 -- 172.21.15.90:0/374618633 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f840410ad20 con 0x7f8404109bf0 2024-04-25T11:25:25.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.120+0000 7f8408cbd700 1 --2- 172.21.15.90:0/374618633 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8404109bf0 0x7f840410a000 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f83f80098a0 tx=0x7f83f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcd9d0688c9fceda server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:25.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.120+0000 7f84037fe700 1 -- 172.21.15.90:0/374618633 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83f8004160 con 0x7f8404109bf0 2024-04-25T11:25:25.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.120+0000 7f84037fe700 1 -- 172.21.15.90:0/374618633 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83f8004300 con 0x7f8404109bf0 2024-04-25T11:25:25.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.120+0000 7f84037fe700 1 -- 172.21.15.90:0/374618633 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83f80036a0 con 0x7f8404109bf0 2024-04-25T11:25:25.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.121+0000 7f840af21700 1 -- 172.21.15.90:0/374618633 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8404109bf0 msgr2=0x7f840410a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:25.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.121+0000 7f840af21700 1 --2- 172.21.15.90:0/374618633 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8404109bf0 0x7f840410a000 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f83f80098a0 tx=0x7f83f8009bb0 comp rx=0 tx=0).stop 2024-04-25T11:25:25.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.121+0000 7f840af21700 1 -- 172.21.15.90:0/374618633 shutdown_connections 2024-04-25T11:25:25.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.121+0000 7f840af21700 1 --2- 172.21.15.90:0/374618633 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8404109bf0 0x7f840410a000 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:25.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.121+0000 7f840af21700 1 -- 172.21.15.90:0/374618633 >> 172.21.15.90:0/374618633 conn(0x7f8404105250 msgr2=0x7f84041076e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:25.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.121+0000 7f840af21700 1 -- 172.21.15.90:0/374618633 shutdown_connections 2024-04-25T11:25:25.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.121+0000 7f840af21700 1 -- 172.21.15.90:0/374618633 wait complete. 2024-04-25T11:25:25.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.121+0000 7f840af21700 1 Processor -- start 2024-04-25T11:25:25.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.121+0000 7f840af21700 1 -- start start 2024-04-25T11:25:25.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.122+0000 7f840af21700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8404109bf0 0x7f840407f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:25.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.122+0000 7f840af21700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f840410b250 con 0x7f8404109bf0 2024-04-25T11:25:25.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.122+0000 7f8408cbd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8404109bf0 0x7f840407f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:25.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.122+0000 7f8408cbd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8404109bf0 0x7f840407f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51528/0 (socket says 172.21.15.90:51528) 2024-04-25T11:25:25.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.122+0000 7f8408cbd700 1 -- 172.21.15.90:0/3337507578 learned_addr learned my addr 172.21.15.90:0/3337507578 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:25.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.122+0000 7f8408cbd700 1 -- 172.21.15.90:0/3337507578 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f840407f830 con 0x7f8404109bf0 2024-04-25T11:25:25.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.122+0000 7f8408cbd700 1 --2- 172.21.15.90:0/3337507578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8404109bf0 0x7f840407f2f0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f83f8000c00 tx=0x7f83f8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:25.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.122+0000 7f8401ffb700 1 -- 172.21.15.90:0/3337507578 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83f8003db0 con 0x7f8404109bf0 2024-04-25T11:25:25.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.122+0000 7f8401ffb700 1 -- 172.21.15.90:0/3337507578 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83f8011630 con 0x7f8404109bf0 2024-04-25T11:25:25.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.122+0000 7f840af21700 1 -- 172.21.15.90:0/3337507578 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f840407da80 con 0x7f8404109bf0 2024-04-25T11:25:25.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.122+0000 7f8401ffb700 1 -- 172.21.15.90:0/3337507578 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83f8011950 con 0x7f8404109bf0 2024-04-25T11:25:25.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.123+0000 7f840af21700 1 -- 172.21.15.90:0/3337507578 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f840407dee0 con 0x7f8404109bf0 2024-04-25T11:25:25.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.123+0000 7f8401ffb700 1 -- 172.21.15.90:0/3337507578 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f83f801a460 con 0x7f8404109bf0 2024-04-25T11:25:25.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.123+0000 7f8401ffb700 1 -- 172.21.15.90:0/3337507578 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f83f801b780 con 0x7f8404109bf0 2024-04-25T11:25:25.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.124+0000 7f840af21700 1 -- 172.21.15.90:0/3337507578 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8404064cf0 con 0x7f8404109bf0 2024-04-25T11:25:25.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.126+0000 7f8401ffb700 1 -- 172.21.15.90:0/3337507578 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f83f8031020 con 0x7f8404109bf0 2024-04-25T11:25:25.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.192+0000 7f840af21700 1 -- 172.21.15.90:0/3337507578 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f8404196790 con 0x7f8404109bf0 2024-04-25T11:25:25.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.196+0000 7f8401ffb700 1 -- 172.21.15.90:0/3337507578 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+492 (secure 0 0 0) 0x7f83f801bad0 con 0x7f8404109bf0 2024-04-25T11:25:25.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.197+0000 7f8401ffb700 1 -- 172.21.15.90:0/3337507578 <== mon.0 v2:172.21.15.90:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f83f801ad70 con 0x7f8404109bf0 2024-04-25T11:25:25.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.198+0000 7f840af21700 1 -- 172.21.15.90:0/3337507578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8404109bf0 msgr2=0x7f840407f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:25.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.198+0000 7f840af21700 1 --2- 172.21.15.90:0/3337507578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8404109bf0 0x7f840407f2f0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f83f8000c00 tx=0x7f83f8004750 comp rx=0 tx=0).stop 2024-04-25T11:25:25.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.198+0000 7f840af21700 1 -- 172.21.15.90:0/3337507578 shutdown_connections 2024-04-25T11:25:25.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.198+0000 7f840af21700 1 --2- 172.21.15.90:0/3337507578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8404109bf0 0x7f840407f2f0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:25.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.198+0000 7f840af21700 1 -- 172.21.15.90:0/3337507578 >> 172.21.15.90:0/3337507578 conn(0x7f8404105250 msgr2=0x7f8404193db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:25.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.198+0000 7f840af21700 1 -- 172.21.15.90:0/3337507578 shutdown_connections 2024-04-25T11:25:25.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:25.198+0000 7f840af21700 1 -- 172.21.15.90:0/3337507578 wait complete. 2024-04-25T11:25:25.851 INFO:teuthology.orchestra.run.smithi090.stdout:Generating new minimal ceph.conf... 2024-04-25T11:25:26.820 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:26 smithi090 ceph-mon[29197]: from='client.? 172.21.15.90:0/1573467730' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:25:27.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.308+0000 7fca80a71700 1 Processor -- start 2024-04-25T11:25:27.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.308+0000 7fca80a71700 1 -- start start 2024-04-25T11:25:27.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.308+0000 7fca80a71700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca7c10a690 0x7fca7c10cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:27.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.308+0000 7fca80a71700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fca7c07d660 con 0x7fca7c10a690 2024-04-25T11:25:27.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.308+0000 7fca7a59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca7c10a690 0x7fca7c10cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:27.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.308+0000 7fca7a59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca7c10a690 0x7fca7c10cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51530/0 (socket says 172.21.15.90:51530) 2024-04-25T11:25:27.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.308+0000 7fca7a59c700 1 -- 172.21.15.90:0/1478278103 learned_addr learned my addr 172.21.15.90:0/1478278103 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:27.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.309+0000 7fca7a59c700 1 -- 172.21.15.90:0/1478278103 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca7c10cff0 con 0x7fca7c10a690 2024-04-25T11:25:27.071 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.309+0000 7fca7a59c700 1 --2- 172.21.15.90:0/1478278103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca7c10a690 0x7fca7c10cab0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0098a0 tx=0x7fca6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e01cad9c7524bb1c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:27.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.309+0000 7fca7959a700 1 -- 172.21.15.90:0/1478278103 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca6c004160 con 0x7fca7c10a690 2024-04-25T11:25:27.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.309+0000 7fca7959a700 1 -- 172.21.15.90:0/1478278103 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fca6c004300 con 0x7fca7c10a690 2024-04-25T11:25:27.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.309+0000 7fca7959a700 1 -- 172.21.15.90:0/1478278103 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca6c011af0 con 0x7fca7c10a690 2024-04-25T11:25:27.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.310+0000 7fca80a71700 1 -- 172.21.15.90:0/1478278103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca7c10a690 msgr2=0x7fca7c10cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:27.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.310+0000 7fca80a71700 1 --2- 172.21.15.90:0/1478278103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca7c10a690 0x7fca7c10cab0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0098a0 tx=0x7fca6c009bb0 comp rx=0 tx=0).stop 2024-04-25T11:25:27.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.310+0000 7fca80a71700 1 -- 172.21.15.90:0/1478278103 shutdown_connections 2024-04-25T11:25:27.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.310+0000 7fca80a71700 1 --2- 172.21.15.90:0/1478278103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca7c10a690 0x7fca7c10cab0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:27.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.310+0000 7fca80a71700 1 -- 172.21.15.90:0/1478278103 >> 172.21.15.90:0/1478278103 conn(0x7fca7c1040f0 msgr2=0x7fca7c1065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:27.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.310+0000 7fca80a71700 1 -- 172.21.15.90:0/1478278103 shutdown_connections 2024-04-25T11:25:27.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.310+0000 7fca80a71700 1 -- 172.21.15.90:0/1478278103 wait complete. 2024-04-25T11:25:27.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.311+0000 7fca80a71700 1 Processor -- start 2024-04-25T11:25:27.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.311+0000 7fca80a71700 1 -- start start 2024-04-25T11:25:27.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.311+0000 7fca80a71700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca7c10a690 0x7fca7c19df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:27.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.311+0000 7fca80a71700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fca7c10def0 con 0x7fca7c10a690 2024-04-25T11:25:27.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.311+0000 7fca7a59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca7c10a690 0x7fca7c19df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:27.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.311+0000 7fca7a59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca7c10a690 0x7fca7c19df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51544/0 (socket says 172.21.15.90:51544) 2024-04-25T11:25:27.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.311+0000 7fca7a59c700 1 -- 172.21.15.90:0/1573467730 learned_addr learned my addr 172.21.15.90:0/1573467730 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:27.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.311+0000 7fca7a59c700 1 -- 172.21.15.90:0/1573467730 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca7c19e4a0 con 0x7fca7c10a690 2024-04-25T11:25:27.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.311+0000 7fca7a59c700 1 --2- 172.21.15.90:0/1573467730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca7c10a690 0x7fca7c19df60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0117b0 tx=0x7fca6c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:27.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.311+0000 7fca6b7fe700 1 -- 172.21.15.90:0/1573467730 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca6c0039c0 con 0x7fca7c10a690 2024-04-25T11:25:27.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.311+0000 7fca80a71700 1 -- 172.21.15.90:0/1573467730 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca7c19e730 con 0x7fca7c10a690 2024-04-25T11:25:27.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.311+0000 7fca6b7fe700 1 -- 172.21.15.90:0/1573467730 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fca6c003b60 con 0x7fca7c10a690 2024-04-25T11:25:27.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.311+0000 7fca6b7fe700 1 -- 172.21.15.90:0/1573467730 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca6c0196a0 con 0x7fca7c10a690 2024-04-25T11:25:27.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.312+0000 7fca80a71700 1 -- 172.21.15.90:0/1573467730 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca7c1a1340 con 0x7fca7c10a690 2024-04-25T11:25:27.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.312+0000 7fca6b7fe700 1 -- 172.21.15.90:0/1573467730 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fca6c019cf0 con 0x7fca7c10a690 2024-04-25T11:25:27.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.312+0000 7fca6b7fe700 1 -- 172.21.15.90:0/1573467730 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fca6c024730 con 0x7fca7c10a690 2024-04-25T11:25:27.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.313+0000 7fca80a71700 1 -- 172.21.15.90:0/1573467730 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca7c064cf0 con 0x7fca7c10a690 2024-04-25T11:25:27.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.315+0000 7fca6b7fe700 1 -- 172.21.15.90:0/1573467730 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fca6c043e50 con 0x7fca7c10a690 2024-04-25T11:25:27.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.382+0000 7fca80a71700 1 -- 172.21.15.90:0/1573467730 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fca7c1a1510 con 0x7fca7c10a690 2024-04-25T11:25:27.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.383+0000 7fca6b7fe700 1 -- 172.21.15.90:0/1573467730 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7fca6c032020 con 0x7fca7c10a690 2024-04-25T11:25:27.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.383+0000 7fca80a71700 1 -- 172.21.15.90:0/1573467730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca7c10a690 msgr2=0x7fca7c19df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:27.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.383+0000 7fca80a71700 1 --2- 172.21.15.90:0/1573467730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca7c10a690 0x7fca7c19df60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0117b0 tx=0x7fca6c011b70 comp rx=0 tx=0).stop 2024-04-25T11:25:27.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.383+0000 7fca80a71700 1 -- 172.21.15.90:0/1573467730 shutdown_connections 2024-04-25T11:25:27.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.384+0000 7fca80a71700 1 --2- 172.21.15.90:0/1573467730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca7c10a690 0x7fca7c19df60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:27.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.384+0000 7fca80a71700 1 -- 172.21.15.90:0/1573467730 >> 172.21.15.90:0/1573467730 conn(0x7fca7c1040f0 msgr2=0x7fca7c104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:27.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.384+0000 7fca80a71700 1 -- 172.21.15.90:0/1573467730 shutdown_connections 2024-04-25T11:25:27.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:26.384+0000 7fca80a71700 1 -- 172.21.15.90:0/1573467730 wait complete. 2024-04-25T11:25:27.074 INFO:teuthology.orchestra.run.smithi090.stdout:Restarting the monitor... 2024-04-25T11:25:27.354 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:27 smithi090 systemd[1]: Stopping Ceph mon.a for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:25:27.354 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:27 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-a[29193]: 2024-04-25T11:25:27.341+0000 7fd81c6ea700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:25:27.354 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:27 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-a[29193]: 2024-04-25T11:25:27.341+0000 7fd81c6ea700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-25T11:25:27.355 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:27 smithi090 podman[29442]: 2024-04-25 11:25:27.361995873 +0000 UTC m=+0.118221969 container died 4a4bdf6050ac553e8084511b9147c80a2b531bd678657b9eea09a328627dac94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-25T11:25:27.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:27 smithi090 podman[29442]: 2024-04-25 11:25:27.649080022 +0000 UTC m=+0.405306116 container cleanup 4a4bdf6050ac553e8084511b9147c80a2b531bd678657b9eea09a328627dac94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-a, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T11:25:27.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:27 smithi090 bash[29442]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-a 2024-04-25T11:25:27.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:27 smithi090 podman[29456]: 2024-04-25 11:25:27.825322342 +0000 UTC m=+0.461675962 container remove 4a4bdf6050ac553e8084511b9147c80a2b531bd678657b9eea09a328627dac94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-25T11:25:28.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 systemd[1]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-25T11:25:28.230 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 systemd[1]: Stopped Ceph mon.a for 58bd4e7a-02f6-11ef-bc93-c7b262605968. 2024-04-25T11:25:28.230 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 systemd[1]: Starting Ceph mon.a for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:25:28.540 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 podman[29555]: 2024-04-25 11:25:28.277820962 +0000 UTC m=+0.107494649 container create 285c6eedf8b9106e32566c376993ff10f0e71c2f4dde2fcfead525fad39050d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-25T11:25:28.540 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 podman[29555]: 2024-04-25 11:25:28.187581647 +0000 UTC m=+0.017255318 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T11:25:28.540 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 podman[29555]: 2024-04-25 11:25:28.464148839 +0000 UTC m=+0.293822488 container init 285c6eedf8b9106e32566c376993ff10f0e71c2f4dde2fcfead525fad39050d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-a, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-25T11:25:28.540 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 podman[29555]: 2024-04-25 11:25:28.467167899 +0000 UTC m=+0.296841544 container start 285c6eedf8b9106e32566c376993ff10f0e71c2f4dde2fcfead525fad39050d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T11:25:28.540 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T11:25:28.540 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev), process ceph-mon, pid 2 2024-04-25T11:25:28.540 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: pidfile_write: ignore empty --pid-file 2024-04-25T11:25:28.540 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: load: jerasure load: lrc 2024-04-25T11:25:28.540 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: RocksDB version: 7.9.2 2024-04-25T11:25:28.540 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Git sha 0 2024-04-25T11:25:28.540 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Compile date 2024-04-25 02:57:34 2024-04-25T11:25:28.541 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: DB SUMMARY 2024-04-25T11:25:28.541 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: DB Session ID: 3PVVZSVS2P3QTNQCMO4H 2024-04-25T11:25:28.541 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: CURRENT file: CURRENT 2024-04-25T11:25:28.541 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T11:25:28.541 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-25T11:25:28.541 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-25T11:25:28.541 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105434 ; 2024-04-25T11:25:28.541 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.error_if_exists: 0 2024-04-25T11:25:28.541 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.create_if_missing: 0 2024-04-25T11:25:28.541 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.paranoid_checks: 1 2024-04-25T11:25:28.541 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T11:25:28.541 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T11:25:28.541 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T11:25:28.541 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.env: 0x55a154cc0500 2024-04-25T11:25:28.542 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T11:25:28.542 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.info_log: 0x55a156f3bfc0 2024-04-25T11:25:28.542 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T11:25:28.542 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.statistics: (nil) 2024-04-25T11:25:28.542 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.use_fsync: 0 2024-04-25T11:25:28.542 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.max_log_file_size: 0 2024-04-25T11:25:28.542 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T11:25:28.542 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T11:25:28.542 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T11:25:28.542 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T11:25:28.542 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.allow_fallocate: 1 2024-04-25T11:25:28.542 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T11:25:28.542 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T11:25:28.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.use_direct_reads: 0 2024-04-25T11:25:28.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T11:25:28.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T11:25:28.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.db_log_dir: 2024-04-25T11:25:28.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.wal_dir: 2024-04-25T11:25:28.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T11:25:28.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T11:25:28.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T11:25:28.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T11:25:28.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T11:25:28.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T11:25:28.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T11:25:28.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T11:25:28.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.write_buffer_manager: 0x55a156f0c8c0 2024-04-25T11:25:28.544 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T11:25:28.544 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T11:25:28.544 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T11:25:28.544 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T11:25:28.545 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T11:25:28.545 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T11:25:28.545 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T11:25:28.545 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T11:25:28.545 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.unordered_write: 0 2024-04-25T11:25:28.545 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T11:25:28.545 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T11:25:28.546 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T11:25:28.546 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T11:25:28.546 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.row_cache: None 2024-04-25T11:25:28.546 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.wal_filter: None 2024-04-25T11:25:28.546 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T11:25:28.546 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T11:25:28.546 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.two_write_queues: 0 2024-04-25T11:25:28.546 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T11:25:28.546 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.wal_compression: 0 2024-04-25T11:25:28.546 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.atomic_flush: 0 2024-04-25T11:25:28.546 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T11:25:28.546 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T11:25:28.546 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T11:25:28.547 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.log_readahead_size: 0 2024-04-25T11:25:28.547 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T11:25:28.547 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T11:25:28.547 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T11:25:28.547 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T11:25:28.547 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T11:25:28.547 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T11:25:28.547 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T11:25:28.547 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.max_background_jobs: 2 2024-04-25T11:25:28.547 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.max_background_compactions: -1 2024-04-25T11:25:28.547 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.max_subcompactions: 1 2024-04-25T11:25:28.547 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T11:25:28.547 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T11:25:28.547 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T11:25:28.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T11:25:28.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T11:25:28.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T11:25:28.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T11:25:28.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T11:25:28.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.max_open_files: -1 2024-04-25T11:25:28.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T11:25:28.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T11:25:28.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T11:25:28.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T11:25:28.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.max_background_flushes: -1 2024-04-25T11:25:28.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Compression algorithms supported: 2024-04-25T11:25:28.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T11:25:28.548 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: kZSTD supported: 0 2024-04-25T11:25:28.549 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: kXpressCompression supported: 0 2024-04-25T11:25:28.549 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T11:25:28.549 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: kZlibCompression supported: 1 2024-04-25T11:25:28.549 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: kSnappyCompression supported: 1 2024-04-25T11:25:28.549 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: kLZ4Compression supported: 1 2024-04-25T11:25:28.549 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: kBZip2Compression supported: 0 2024-04-25T11:25:28.549 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T11:25:28.549 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T11:25:28.549 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-25T11:25:28.549 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T11:25:28.549 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T11:25:28.549 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.merge_operator: 2024-04-25T11:25:28.549 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.compaction_filter: None 2024-04-25T11:25:28.549 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T11:25:28.549 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T11:25:28.549 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T11:25:28.550 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T11:25:28.550 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a1570421a0) 2024-04-25T11:25:28.550 INFO:journalctl@ceph.mon.a.smithi090.stdout: cache_index_and_filter_blocks: 1 2024-04-25T11:25:28.550 INFO:journalctl@ceph.mon.a.smithi090.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T11:25:28.550 INFO:journalctl@ceph.mon.a.smithi090.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T11:25:28.550 INFO:journalctl@ceph.mon.a.smithi090.stdout: pin_top_level_index_and_filter: 1 2024-04-25T11:25:28.550 INFO:journalctl@ceph.mon.a.smithi090.stdout: index_type: 0 2024-04-25T11:25:28.550 INFO:journalctl@ceph.mon.a.smithi090.stdout: data_block_index_type: 0 2024-04-25T11:25:28.550 INFO:journalctl@ceph.mon.a.smithi090.stdout: index_shortening: 1 2024-04-25T11:25:28.550 INFO:journalctl@ceph.mon.a.smithi090.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T11:25:28.550 INFO:journalctl@ceph.mon.a.smithi090.stdout: checksum: 4 2024-04-25T11:25:28.550 INFO:journalctl@ceph.mon.a.smithi090.stdout: no_block_cache: 0 2024-04-25T11:25:28.550 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_cache: 0x55a156f3f1f0 2024-04-25T11:25:28.550 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_cache_name: BinnedLRUCache 2024-04-25T11:25:28.550 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_cache_options: 2024-04-25T11:25:28.551 INFO:journalctl@ceph.mon.a.smithi090.stdout: capacity : 536870912 2024-04-25T11:25:28.551 INFO:journalctl@ceph.mon.a.smithi090.stdout: num_shard_bits : 4 2024-04-25T11:25:28.551 INFO:journalctl@ceph.mon.a.smithi090.stdout: strict_capacity_limit : 0 2024-04-25T11:25:28.551 INFO:journalctl@ceph.mon.a.smithi090.stdout: high_pri_pool_ratio: 0.000 2024-04-25T11:25:28.551 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_cache_compressed: (nil) 2024-04-25T11:25:28.551 INFO:journalctl@ceph.mon.a.smithi090.stdout: persistent_cache: (nil) 2024-04-25T11:25:28.551 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_size: 4096 2024-04-25T11:25:28.551 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_size_deviation: 10 2024-04-25T11:25:28.551 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_restart_interval: 16 2024-04-25T11:25:28.551 INFO:journalctl@ceph.mon.a.smithi090.stdout: index_block_restart_interval: 1 2024-04-25T11:25:28.551 INFO:journalctl@ceph.mon.a.smithi090.stdout: metadata_block_size: 4096 2024-04-25T11:25:28.551 INFO:journalctl@ceph.mon.a.smithi090.stdout: partition_filters: 0 2024-04-25T11:25:28.551 INFO:journalctl@ceph.mon.a.smithi090.stdout: use_delta_encoding: 1 2024-04-25T11:25:28.551 INFO:journalctl@ceph.mon.a.smithi090.stdout: filter_policy: bloomfilter 2024-04-25T11:25:28.551 INFO:journalctl@ceph.mon.a.smithi090.stdout: whole_key_filtering: 1 2024-04-25T11:25:28.552 INFO:journalctl@ceph.mon.a.smithi090.stdout: verify_compression: 0 2024-04-25T11:25:28.552 INFO:journalctl@ceph.mon.a.smithi090.stdout: read_amp_bytes_per_bit: 0 2024-04-25T11:25:28.552 INFO:journalctl@ceph.mon.a.smithi090.stdout: format_version: 5 2024-04-25T11:25:28.552 INFO:journalctl@ceph.mon.a.smithi090.stdout: enable_index_compression: 1 2024-04-25T11:25:28.552 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_align: 0 2024-04-25T11:25:28.552 INFO:journalctl@ceph.mon.a.smithi090.stdout: max_auto_readahead_size: 262144 2024-04-25T11:25:28.552 INFO:journalctl@ceph.mon.a.smithi090.stdout: prepopulate_block_cache: 0 2024-04-25T11:25:28.552 INFO:journalctl@ceph.mon.a.smithi090.stdout: initial_auto_readahead_size: 8192 2024-04-25T11:25:28.552 INFO:journalctl@ceph.mon.a.smithi090.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T11:25:28.552 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T11:25:28.552 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T11:25:28.552 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.compression: NoCompression 2024-04-25T11:25:28.552 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T11:25:28.552 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T11:25:28.552 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T11:25:28.553 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.num_levels: 7 2024-04-25T11:25:28.553 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T11:25:28.553 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T11:25:28.553 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T11:25:28.553 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T11:25:28.553 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T11:25:28.553 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T11:25:28.553 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T11:25:28.553 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T11:25:28.553 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T11:25:28.553 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T11:25:28.553 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T11:25:28.553 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T11:25:28.553 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T11:25:28.553 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T11:25:28.554 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T11:25:28.554 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T11:25:28.554 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T11:25:28.554 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T11:25:28.554 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T11:25:28.554 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T11:25:28.554 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T11:25:28.554 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T11:25:28.554 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T11:25:28.554 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T11:25:28.554 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T11:25:28.554 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T11:25:28.554 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T11:25:28.554 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T11:25:28.554 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T11:25:28.555 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T11:25:28.555 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T11:25:28.555 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T11:25:28.555 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T11:25:28.555 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T11:25:28.555 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T11:25:28.555 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T11:25:28.555 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T11:25:28.555 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T11:25:28.555 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T11:25:28.555 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T11:25:28.555 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T11:25:28.555 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T11:25:28.555 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T11:25:28.555 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T11:25:28.555 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T11:25:28.556 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T11:25:28.556 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T11:25:28.556 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T11:25:28.556 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T11:25:28.556 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T11:25:28.556 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T11:25:28.556 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T11:25:28.556 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T11:25:28.556 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-25T11:25:28.556 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.inplace_update_support: 0 2024-04-25T11:25:28.556 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T11:25:28.556 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T11:25:28.556 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T11:25:28.556 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T11:25:28.556 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.bloom_locality: 0 2024-04-25T11:25:28.556 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.max_successive_merges: 0 2024-04-25T11:25:28.557 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T11:25:28.557 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T11:25:28.557 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T11:25:28.557 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T11:25:28.557 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.ttl: 2592000 2024-04-25T11:25:28.557 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T11:25:28.557 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T11:25:28.557 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T11:25:28.557 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.enable_blob_files: false 2024-04-25T11:25:28.557 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.min_blob_size: 0 2024-04-25T11:25:28.557 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T11:25:28.557 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T11:25:28.557 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T11:25:28.557 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T11:25:28.557 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T11:25:28.558 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T11:25:28.558 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T11:25:28.558 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T11:25:28.558 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-25T11:25:28.558 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-25T11:25:28.558 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f86099e8-6a66-4232-bd7d-89c6ee836712 2024-04-25T11:25:28.558 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714044328507539, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-25T11:25:28.558 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-25T11:25:28.558 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714044328508647, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100546, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 328, "table_properties": {"data_size": 98431, "index_size": 307, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14264, "raw_average_key_size": 49, "raw_value_size": 90685, "raw_average_value_size": 315, "num_data_blocks": 12, "num_entries": 287, "num_filter_entries": 287, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714044328, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f86099e8-6a66-4232-bd7d-89c6ee836712", "db_session_id": "3PVVZSVS2P3QTNQCMO4H", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-25T11:25:28.559 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714044328508729, "job": 1, "event": "recovery_finished"} 2024-04-25T11:25:28.559 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-25T11:25:28.559 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T11:25:28.559 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a156f23c00 2024-04-25T11:25:28.559 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: DB pointer 0x55a15bd16000 2024-04-25T11:25:28.559 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T11:25:28.559 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T11:25:28.559 INFO:journalctl@ceph.mon.a.smithi090.stdout: ** DB Stats ** 2024-04-25T11:25:28.559 INFO:journalctl@ceph.mon.a.smithi090.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T11:25:28.559 INFO:journalctl@ceph.mon.a.smithi090.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T11:25:28.559 INFO:journalctl@ceph.mon.a.smithi090.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T11:25:28.559 INFO:journalctl@ceph.mon.a.smithi090.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T11:25:28.559 INFO:journalctl@ceph.mon.a.smithi090.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T11:25:28.559 INFO:journalctl@ceph.mon.a.smithi090.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T11:25:28.560 INFO:journalctl@ceph.mon.a.smithi090.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T11:25:28.560 INFO:journalctl@ceph.mon.a.smithi090.stdout: 2024-04-25T11:25:28.560 INFO:journalctl@ceph.mon.a.smithi090.stdout: ** Compaction Stats [default] ** 2024-04-25T11:25:28.560 INFO:journalctl@ceph.mon.a.smithi090.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T11:25:28.560 INFO:journalctl@ceph.mon.a.smithi090.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T11:25:28.560 INFO:journalctl@ceph.mon.a.smithi090.stdout: L0 2/0 100.05 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T11:25:28.560 INFO:journalctl@ceph.mon.a.smithi090.stdout: Sum 2/0 100.05 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T11:25:28.560 INFO:journalctl@ceph.mon.a.smithi090.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T11:25:28.560 INFO:journalctl@ceph.mon.a.smithi090.stdout: 2024-04-25T11:25:28.560 INFO:journalctl@ceph.mon.a.smithi090.stdout: ** Compaction Stats [default] ** 2024-04-25T11:25:28.560 INFO:journalctl@ceph.mon.a.smithi090.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T11:25:28.560 INFO:journalctl@ceph.mon.a.smithi090.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T11:25:28.560 INFO:journalctl@ceph.mon.a.smithi090.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 158.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T11:25:28.561 INFO:journalctl@ceph.mon.a.smithi090.stdout: 2024-04-25T11:25:28.561 INFO:journalctl@ceph.mon.a.smithi090.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T11:25:28.561 INFO:journalctl@ceph.mon.a.smithi090.stdout: 2024-04-25T11:25:28.561 INFO:journalctl@ceph.mon.a.smithi090.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T11:25:28.561 INFO:journalctl@ceph.mon.a.smithi090.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T11:25:28.561 INFO:journalctl@ceph.mon.a.smithi090.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T11:25:28.561 INFO:journalctl@ceph.mon.a.smithi090.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T11:25:28.561 INFO:journalctl@ceph.mon.a.smithi090.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T11:25:28.561 INFO:journalctl@ceph.mon.a.smithi090.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T11:25:28.561 INFO:journalctl@ceph.mon.a.smithi090.stdout: Cumulative compaction: 0.00 GB write, 30.85 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T11:25:28.561 INFO:journalctl@ceph.mon.a.smithi090.stdout: Interval compaction: 0.00 GB write, 30.85 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T11:25:28.561 INFO:journalctl@ceph.mon.a.smithi090.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T11:25:28.561 INFO:journalctl@ceph.mon.a.smithi090.stdout: Block cache BinnedLRUCache@0x55a156f3f1f0#2 capacity: 512.00 MB usage: 1.53 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-25T11:25:28.562 INFO:journalctl@ceph.mon.a.smithi090.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.03 KB,0.000196695%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-25T11:25:28.562 INFO:journalctl@ceph.mon.a.smithi090.stdout: 2024-04-25T11:25:28.562 INFO:journalctl@ceph.mon.a.smithi090.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T11:25:28.562 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: starting mon.a rank 0 at public addrs [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] at bind addrs [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:25:28.562 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: mon.a@-1(???) e1 preinit fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:25:28.562 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: mon.a@-1(???).mds e1 new map 2024-04-25T11:25:28.562 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: mon.a@-1(???).mds e1 print_map 2024-04-25T11:25:28.562 INFO:journalctl@ceph.mon.a.smithi090.stdout: e1 2024-04-25T11:25:28.562 INFO:journalctl@ceph.mon.a.smithi090.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T11:25:28.562 INFO:journalctl@ceph.mon.a.smithi090.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T11:25:28.562 INFO:journalctl@ceph.mon.a.smithi090.stdout: legacy client fscid: -1 2024-04-25T11:25:28.562 INFO:journalctl@ceph.mon.a.smithi090.stdout: 2024-04-25T11:25:28.563 INFO:journalctl@ceph.mon.a.smithi090.stdout: No filesystems configured 2024-04-25T11:25:28.563 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T11:25:28.563 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T11:25:28.563 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T11:25:28.563 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T11:25:28.563 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-25T11:25:28.563 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 bash[29555]: 285c6eedf8b9106e32566c376993ff10f0e71c2f4dde2fcfead525fad39050d7 2024-04-25T11:25:28.563 INFO:teuthology.orchestra.run.smithi090.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-25T11:25:28.876 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 systemd[1]: Started Ceph mon.a for 58bd4e7a-02f6-11ef-bc93-c7b262605968. 2024-04-25T11:25:28.876 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T11:25:28.876 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: monmap epoch 1 2024-04-25T11:25:28.876 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:25:28.876 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: last_changed 2024-04-25T11:25:19.353596+0000 2024-04-25T11:25:28.876 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: created 2024-04-25T11:25:19.353596+0000 2024-04-25T11:25:28.876 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: min_mon_release 19 (squid) 2024-04-25T11:25:28.876 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: election_strategy: 1 2024-04-25T11:25:28.876 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: 0: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.a 2024-04-25T11:25:28.877 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: fsmap 2024-04-25T11:25:28.877 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: osdmap e1: 0 total, 0 up, 0 in 2024-04-25T11:25:28.877 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:28 smithi090 ceph-mon[29575]: mgrmap e1: no daemons active 2024-04-25T11:25:29.554 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:28.996+0000 7f12425ae700 1 Processor -- start 2024-04-25T11:25:29.554 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:28.996+0000 7f12425ae700 1 -- start start 2024-04-25T11:25:29.554 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:28.997+0000 7f12425ae700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f123c108ab0 0x7f123c108ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:29.554 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:28.997+0000 7f12425ae700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f123c1094b0 con 0x7f123c108ab0 2024-04-25T11:25:29.555 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:28.997+0000 7f123bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f123c108ab0 0x7f123c108ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:29.555 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:28.997+0000 7f123bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f123c108ab0 0x7f123c108ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51554/0 (socket says 172.21.15.90:51554) 2024-04-25T11:25:29.555 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:28.997+0000 7f123bfff700 1 -- 172.21.15.90:0/3890879297 learned_addr learned my addr 172.21.15.90:0/3890879297 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:29.555 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:28.997+0000 7f123bfff700 1 -- 172.21.15.90:0/3890879297 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f123c109be0 con 0x7f123c108ab0 2024-04-25T11:25:29.555 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:28.998+0000 7f123bfff700 1 --2- 172.21.15.90:0/3890879297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f123c108ab0 0x7f123c108ec0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f122c0098a0 tx=0x7f122c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bac6576ffc166326 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:29.555 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:28.998+0000 7f123affd700 1 -- 172.21.15.90:0/3890879297 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f122c004160 con 0x7f123c108ab0 2024-04-25T11:25:29.555 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:28.998+0000 7f123affd700 1 -- 172.21.15.90:0/3890879297 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f122c004300 con 0x7f123c108ab0 2024-04-25T11:25:29.555 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:28.998+0000 7f123affd700 1 -- 172.21.15.90:0/3890879297 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f122c011af0 con 0x7f123c108ab0 2024-04-25T11:25:29.555 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:28.998+0000 7f12425ae700 1 -- 172.21.15.90:0/3890879297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f123c108ab0 msgr2=0x7f123c108ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:29.555 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:28.998+0000 7f12425ae700 1 --2- 172.21.15.90:0/3890879297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f123c108ab0 0x7f123c108ec0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f122c0098a0 tx=0x7f122c009bb0 comp rx=0 tx=0).stop 2024-04-25T11:25:29.555 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:28.999+0000 7f12425ae700 1 -- 172.21.15.90:0/3890879297 shutdown_connections 2024-04-25T11:25:29.555 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:28.999+0000 7f12425ae700 1 --2- 172.21.15.90:0/3890879297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f123c108ab0 0x7f123c108ec0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:29.555 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:28.999+0000 7f12425ae700 1 -- 172.21.15.90:0/3890879297 >> 172.21.15.90:0/3890879297 conn(0x7f123c1040f0 msgr2=0x7f123c1065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:29.556 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:28.999+0000 7f12425ae700 1 -- 172.21.15.90:0/3890879297 shutdown_connections 2024-04-25T11:25:29.556 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:28.999+0000 7f12425ae700 1 -- 172.21.15.90:0/3890879297 wait complete. 2024-04-25T11:25:29.556 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:28.999+0000 7f12425ae700 1 Processor -- start 2024-04-25T11:25:29.556 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:28.999+0000 7f12425ae700 1 -- start start 2024-04-25T11:25:29.556 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:29.000+0000 7f12425ae700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f123c108ab0 0x7f123c07c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:29.556 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:29.000+0000 7f12425ae700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f123c10cf90 con 0x7f123c108ab0 2024-04-25T11:25:29.556 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:29.000+0000 7f123bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f123c108ab0 0x7f123c07c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:29.556 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:29.000+0000 7f123bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f123c108ab0 0x7f123c07c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51556/0 (socket says 172.21.15.90:51556) 2024-04-25T11:25:29.556 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:29.000+0000 7f123bfff700 1 -- 172.21.15.90:0/3837181349 learned_addr learned my addr 172.21.15.90:0/3837181349 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:29.556 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:29.000+0000 7f123bfff700 1 -- 172.21.15.90:0/3837181349 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f123c07ce80 con 0x7f123c108ab0 2024-04-25T11:25:29.556 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:29.000+0000 7f123bfff700 1 --2- 172.21.15.90:0/3837181349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f123c108ab0 0x7f123c07c940 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f122c0090e0 tx=0x7f122c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:29.556 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:29.000+0000 7f12397fa700 1 -- 172.21.15.90:0/3837181349 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f122c0039e0 con 0x7f123c108ab0 2024-04-25T11:25:29.556 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:29.000+0000 7f12397fa700 1 -- 172.21.15.90:0/3837181349 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f122c003b80 con 0x7f123c108ab0 2024-04-25T11:25:29.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:29.000+0000 7f12425ae700 1 -- 172.21.15.90:0/3837181349 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f123c1a0d80 con 0x7f123c108ab0 2024-04-25T11:25:29.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:29.000+0000 7f12397fa700 1 -- 172.21.15.90:0/3837181349 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f122c024690 con 0x7f123c108ab0 2024-04-25T11:25:29.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:29.000+0000 7f12425ae700 1 -- 172.21.15.90:0/3837181349 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f123c1a1200 con 0x7f123c108ab0 2024-04-25T11:25:29.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:29.001+0000 7f12397fa700 1 -- 172.21.15.90:0/3837181349 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f122c022020 con 0x7f123c108ab0 2024-04-25T11:25:29.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:29.001+0000 7f12397fa700 1 -- 172.21.15.90:0/3837181349 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f122c02dc40 con 0x7f123c108ab0 2024-04-25T11:25:29.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:29.001+0000 7f12425ae700 1 -- 172.21.15.90:0/3837181349 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f123c064cf0 con 0x7f123c108ab0 2024-04-25T11:25:29.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:29.004+0000 7f12397fa700 1 -- 172.21.15.90:0/3837181349 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f122c01a9c0 con 0x7f123c108ab0 2024-04-25T11:25:29.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:29.073+0000 7f12425ae700 1 -- 172.21.15.90:0/3837181349 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f123c1a1480 con 0x7f123c108ab0 2024-04-25T11:25:29.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:29.075+0000 7f12397fa700 1 -- 172.21.15.90:0/3837181349 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f122c02d3f0 con 0x7f123c108ab0 2024-04-25T11:25:29.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:29.076+0000 7f12425ae700 1 -- 172.21.15.90:0/3837181349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f123c108ab0 msgr2=0x7f123c07c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:29.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:29.076+0000 7f12425ae700 1 --2- 172.21.15.90:0/3837181349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f123c108ab0 0x7f123c07c940 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f122c0090e0 tx=0x7f122c011940 comp rx=0 tx=0).stop 2024-04-25T11:25:29.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:29.076+0000 7f12425ae700 1 -- 172.21.15.90:0/3837181349 shutdown_connections 2024-04-25T11:25:29.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:29.076+0000 7f12425ae700 1 --2- 172.21.15.90:0/3837181349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f123c108ab0 0x7f123c07c940 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:29.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:29.076+0000 7f12425ae700 1 -- 172.21.15.90:0/3837181349 >> 172.21.15.90:0/3837181349 conn(0x7f123c1040f0 msgr2=0x7f123c104a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:29.558 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:29.076+0000 7f12425ae700 1 -- 172.21.15.90:0/3837181349 shutdown_connections 2024-04-25T11:25:29.558 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:29.076+0000 7f12425ae700 1 -- 172.21.15.90:0/3837181349 wait complete. 2024-04-25T11:25:29.558 INFO:teuthology.orchestra.run.smithi090.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-25T11:25:29.581 INFO:teuthology.orchestra.run.smithi090.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-25T11:25:29.581 INFO:teuthology.orchestra.run.smithi090.stdout:Creating mgr... 2024-04-25T11:25:29.581 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-25T11:25:29.582 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-25T11:25:29.838 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mgr.x 2024-04-25T11:25:29.838 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Failed to reset failed state of unit ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-25T11:25:29.988 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968.target.wants/ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@.service. 2024-04-25T11:25:30.067 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:30 smithi090 systemd[1]: Starting Ceph mgr.x for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:25:30.326 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:30 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3837181349' entity='client.admin' 2024-04-25T11:25:30.326 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:30 smithi090 podman[29791]: 2024-04-25 11:25:30.23352666 +0000 UTC m=+0.115014886 container create 89807c68fe179e40f7bbd2e18fdea4b8165d6ee97b0f33c86f24fc1edc5012f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T11:25:30.326 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:30 smithi090 podman[29791]: 2024-04-25 11:25:30.136237662 +0000 UTC m=+0.017725900 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T11:25:30.509 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T11:25:30.509 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout disabled 2024-04-25T11:25:30.516 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T11:25:30.516 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-04-25T11:25:30.516 INFO:teuthology.orchestra.run.smithi090.stdout:firewalld.service is not enabled 2024-04-25T11:25:30.516 INFO:teuthology.orchestra.run.smithi090.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T11:25:30.524 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T11:25:30.524 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout disabled 2024-04-25T11:25:30.531 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T11:25:30.532 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-04-25T11:25:30.532 INFO:teuthology.orchestra.run.smithi090.stdout:firewalld.service is not enabled 2024-04-25T11:25:30.532 INFO:teuthology.orchestra.run.smithi090.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-25T11:25:30.532 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mgr to start... 2024-04-25T11:25:30.532 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mgr... 2024-04-25T11:25:30.601 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:30 smithi090 podman[29791]: 2024-04-25 11:25:30.43620523 +0000 UTC m=+0.317693454 container init 89807c68fe179e40f7bbd2e18fdea4b8165d6ee97b0f33c86f24fc1edc5012f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-25T11:25:30.602 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:30 smithi090 podman[29791]: 2024-04-25 11:25:30.438790185 +0000 UTC m=+0.320278411 container start 89807c68fe179e40f7bbd2e18fdea4b8165d6ee97b0f33c86f24fc1edc5012f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T11:25:30.602 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:30 smithi090 bash[29791]: 89807c68fe179e40f7bbd2e18fdea4b8165d6ee97b0f33c86f24fc1edc5012f7 2024-04-25T11:25:30.602 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:30 smithi090 systemd[1]: Started Ceph mgr.x for 58bd4e7a-02f6-11ef-bc93-c7b262605968. 2024-04-25T11:25:30.875 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:30 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:30.607+0000 7f39f5e52200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T11:25:30.875 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:30 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:30.795+0000 7f39f5e52200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T11:25:31.215 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:31 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3162480094' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T11:25:31.215 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:30 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:30.899+0000 7f39f5e52200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T11:25:31.215 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:31 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:31.221+0000 7f39f5e52200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T11:25:31.479 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:31 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:31.307+0000 7f39f5e52200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T11:25:31.676 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-25T11:25:31.676 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-04-25T11:25:31.676 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsid": "58bd4e7a-02f6-11ef-bc93-c7b262605968", 2024-04-25T11:25:31.676 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T11:25:31.676 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T11:25:31.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T11:25:31.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T11:25:31.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:31.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T11:25:31.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T11:25:31.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 0 2024-04-25T11:25:31.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-25T11:25:31.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T11:25:31.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T11:25:31.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-25T11:25:31.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-25T11:25:31.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T11:25:31.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T11:25:31.678 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T11:25:31.678 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T11:25:31.678 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:31.678 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T11:25:31.678 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T11:25:31.678 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T11:25:31.678 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T11:25:31.678 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T11:25:31.686 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T11:25:31.686 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T11:25:31.686 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T11:25:31.686 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:31.686 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T11:25:31.686 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T11:25:31.686 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T11:25:31.686 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T11:25:31.686 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T11:25:31.686 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T11:25:31.686 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T11:25:31.686 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T11:25:31.687 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T11:25:31.687 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:31.687 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T11:25:31.687 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T11:25:31.687 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T11:25:31.687 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T11:25:31.687 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:31.687 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T11:25:31.687 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T11:25:31.687 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T11:25:31.687 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T11:25:31.687 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T11:25:31.687 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T11:25:31.687 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T11:25:31.688 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-25T11:25:31.688 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T11:25:31.688 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:31.688 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T11:25:31.688 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T11:25:31.688 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T11:25:23.575127+0000", 2024-04-25T11:25:31.688 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T11:25:31.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:31.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T11:25:31.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-04-25T11:25:31.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.992+0000 7ff1d13e2700 1 Processor -- start 2024-04-25T11:25:31.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.992+0000 7ff1d13e2700 1 -- start start 2024-04-25T11:25:31.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.992+0000 7ff1d13e2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1cc07cc20 0x7ff1cc10c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:31.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.992+0000 7ff1d13e2700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff1cc10cd50 con 0x7ff1cc07cc20 2024-04-25T11:25:31.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.992+0000 7ff1caffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1cc07cc20 0x7ff1cc10c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:31.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.992+0000 7ff1caffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1cc07cc20 0x7ff1cc10c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51576/0 (socket says 172.21.15.90:51576) 2024-04-25T11:25:31.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.992+0000 7ff1caffd700 1 -- 172.21.15.90:0/1011945760 learned_addr learned my addr 172.21.15.90:0/1011945760 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:31.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.993+0000 7ff1caffd700 1 -- 172.21.15.90:0/1011945760 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1cc10ced0 con 0x7ff1cc07cc20 2024-04-25T11:25:31.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.993+0000 7ff1caffd700 1 --2- 172.21.15.90:0/1011945760 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1cc07cc20 0x7ff1cc10c780 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc0098a0 tx=0x7ff1bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2850b878f5f6628 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:31.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.993+0000 7ff1c9ffb700 1 -- 172.21.15.90:0/1011945760 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1bc004160 con 0x7ff1cc07cc20 2024-04-25T11:25:31.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.993+0000 7ff1c9ffb700 1 -- 172.21.15.90:0/1011945760 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff1bc004300 con 0x7ff1cc07cc20 2024-04-25T11:25:31.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.993+0000 7ff1c9ffb700 1 -- 172.21.15.90:0/1011945760 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1bc011af0 con 0x7ff1cc07cc20 2024-04-25T11:25:31.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.994+0000 7ff1d13e2700 1 -- 172.21.15.90:0/1011945760 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1cc07cc20 msgr2=0x7ff1cc10c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:31.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.994+0000 7ff1d13e2700 1 --2- 172.21.15.90:0/1011945760 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1cc07cc20 0x7ff1cc10c780 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc0098a0 tx=0x7ff1bc009bb0 comp rx=0 tx=0).stop 2024-04-25T11:25:31.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.994+0000 7ff1d13e2700 1 -- 172.21.15.90:0/1011945760 shutdown_connections 2024-04-25T11:25:31.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.994+0000 7ff1d13e2700 1 --2- 172.21.15.90:0/1011945760 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1cc07cc20 0x7ff1cc10c780 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:31.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.994+0000 7ff1d13e2700 1 -- 172.21.15.90:0/1011945760 >> 172.21.15.90:0/1011945760 conn(0x7ff1cc1040f0 msgr2=0x7ff1cc1065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:31.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.994+0000 7ff1d13e2700 1 -- 172.21.15.90:0/1011945760 shutdown_connections 2024-04-25T11:25:31.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.994+0000 7ff1d13e2700 1 -- 172.21.15.90:0/1011945760 wait complete. 2024-04-25T11:25:31.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.995+0000 7ff1d13e2700 1 Processor -- start 2024-04-25T11:25:31.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.995+0000 7ff1d13e2700 1 -- start start 2024-04-25T11:25:31.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.995+0000 7ff1d13e2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1cc07cc20 0x7ff1cc109ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:31.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.995+0000 7ff1d13e2700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff1cc10def0 con 0x7ff1cc07cc20 2024-04-25T11:25:31.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.995+0000 7ff1caffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1cc07cc20 0x7ff1cc109ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:31.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.995+0000 7ff1caffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1cc07cc20 0x7ff1cc109ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51588/0 (socket says 172.21.15.90:51588) 2024-04-25T11:25:31.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.995+0000 7ff1caffd700 1 -- 172.21.15.90:0/3162480094 learned_addr learned my addr 172.21.15.90:0/3162480094 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:31.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.996+0000 7ff1caffd700 1 -- 172.21.15.90:0/3162480094 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1cc10a430 con 0x7ff1cc07cc20 2024-04-25T11:25:31.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.996+0000 7ff1caffd700 1 --2- 172.21.15.90:0/3162480094 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1cc07cc20 0x7ff1cc109ef0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc009fb0 tx=0x7ff1bc011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:31.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.996+0000 7ff1b3fff700 1 -- 172.21.15.90:0/3162480094 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1bc011bc0 con 0x7ff1cc07cc20 2024-04-25T11:25:31.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.996+0000 7ff1d13e2700 1 -- 172.21.15.90:0/3162480094 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1cc108680 con 0x7ff1cc07cc20 2024-04-25T11:25:31.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.996+0000 7ff1b3fff700 1 -- 172.21.15.90:0/3162480094 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff1bc024440 con 0x7ff1cc07cc20 2024-04-25T11:25:31.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.996+0000 7ff1b3fff700 1 -- 172.21.15.90:0/3162480094 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1bc01a5c0 con 0x7ff1cc07cc20 2024-04-25T11:25:31.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.996+0000 7ff1d13e2700 1 -- 172.21.15.90:0/3162480094 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1cc108b60 con 0x7ff1cc07cc20 2024-04-25T11:25:31.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.996+0000 7ff1b3fff700 1 -- 172.21.15.90:0/3162480094 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff1bc01ac10 con 0x7ff1cc07cc20 2024-04-25T11:25:31.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.996+0000 7ff1b3fff700 1 -- 172.21.15.90:0/3162480094 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff1bc00ba20 con 0x7ff1cc07cc20 2024-04-25T11:25:31.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.997+0000 7ff1d13e2700 1 -- 172.21.15.90:0/3162480094 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1cc064cf0 con 0x7ff1cc07cc20 2024-04-25T11:25:31.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:30.999+0000 7ff1b3fff700 1 -- 172.21.15.90:0/3162480094 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7ff1bc043e50 con 0x7ff1cc07cc20 2024-04-25T11:25:31.692 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:31.072+0000 7ff1d13e2700 1 -- 172.21.15.90:0/3162480094 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ff1cc108de0 con 0x7ff1cc07cc20 2024-04-25T11:25:31.692 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:31.072+0000 7ff1b3fff700 1 -- 172.21.15.90:0/3162480094 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7ff1bc032070 con 0x7ff1cc07cc20 2024-04-25T11:25:31.692 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:31.073+0000 7ff1d13e2700 1 -- 172.21.15.90:0/3162480094 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1cc07cc20 msgr2=0x7ff1cc109ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:31.692 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:31.073+0000 7ff1d13e2700 1 --2- 172.21.15.90:0/3162480094 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1cc07cc20 0x7ff1cc109ef0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc009fb0 tx=0x7ff1bc011890 comp rx=0 tx=0).stop 2024-04-25T11:25:31.692 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:31.073+0000 7ff1d13e2700 1 -- 172.21.15.90:0/3162480094 shutdown_connections 2024-04-25T11:25:31.692 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:31.073+0000 7ff1d13e2700 1 --2- 172.21.15.90:0/3162480094 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1cc07cc20 0x7ff1cc109ef0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:31.692 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:31.073+0000 7ff1d13e2700 1 -- 172.21.15.90:0/3162480094 >> 172.21.15.90:0/3162480094 conn(0x7ff1cc1040f0 msgr2=0x7ff1cc104ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:31.692 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:31.073+0000 7ff1d13e2700 1 -- 172.21.15.90:0/3162480094 shutdown_connections 2024-04-25T11:25:31.692 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:31.073+0000 7ff1d13e2700 1 -- 172.21.15.90:0/3162480094 wait complete. 2024-04-25T11:25:31.692 INFO:teuthology.orchestra.run.smithi090.stdout:mgr not available, waiting (1/15)... 2024-04-25T11:25:32.842 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:32 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:32.506+0000 7f39f5e52200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T11:25:32.842 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:32 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:32.677+0000 7f39f5e52200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T11:25:32.842 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:32 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:32.848+0000 7f39f5e52200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T11:25:33.420 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:33 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:33.134+0000 7f39f5e52200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T11:25:33.420 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:33 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:33.226+0000 7f39f5e52200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T11:25:33.420 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:33 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:33.427+0000 7f39f5e52200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T11:25:33.729 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:33 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:33.618+0000 7f39f5e52200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T11:25:34.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:34 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3740170078' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T11:25:34.480 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:34 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:34.237+0000 7f39f5e52200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T11:25:34.867 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-25T11:25:34.868 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-04-25T11:25:34.868 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsid": "58bd4e7a-02f6-11ef-bc93-c7b262605968", 2024-04-25T11:25:34.868 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T11:25:34.868 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T11:25:34.868 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T11:25:34.868 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T11:25:34.868 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:34.868 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T11:25:34.868 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T11:25:34.868 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 0 2024-04-25T11:25:34.868 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-25T11:25:34.868 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T11:25:34.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T11:25:34.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-25T11:25:34.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-04-25T11:25:34.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T11:25:34.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T11:25:34.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T11:25:34.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T11:25:34.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:34.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T11:25:34.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T11:25:34.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T11:25:34.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T11:25:34.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T11:25:34.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T11:25:34.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T11:25:34.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T11:25:34.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:34.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T11:25:34.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T11:25:34.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T11:25:34.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T11:25:34.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T11:25:34.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T11:25:34.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T11:25:34.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T11:25:34.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T11:25:34.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:34.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T11:25:34.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T11:25:34.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T11:25:34.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T11:25:34.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:34.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T11:25:34.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T11:25:34.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T11:25:34.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T11:25:34.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T11:25:34.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T11:25:34.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T11:25:34.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-25T11:25:34.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T11:25:34.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:34.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T11:25:34.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T11:25:34.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T11:25:23.575127+0000", 2024-04-25T11:25:34.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T11:25:34.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:34.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T11:25:34.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-04-25T11:25:34.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.127+0000 7f70de1bf700 1 Processor -- start 2024-04-25T11:25:34.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.127+0000 7f70de1bf700 1 -- start start 2024-04-25T11:25:34.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.127+0000 7f70de1bf700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70d8107f00 0x7f70d8108310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:34.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.127+0000 7f70de1bf700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f70d8108900 con 0x7f70d8107f00 2024-04-25T11:25:34.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.128+0000 7f70d77fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70d8107f00 0x7f70d8108310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:34.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.128+0000 7f70d77fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70d8107f00 0x7f70d8108310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39254/0 (socket says 172.21.15.90:39254) 2024-04-25T11:25:34.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.128+0000 7f70d77fe700 1 -- 172.21.15.90:0/876020303 learned_addr learned my addr 172.21.15.90:0/876020303 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:34.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.128+0000 7f70d77fe700 1 -- 172.21.15.90:0/876020303 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d8108a80 con 0x7f70d8107f00 2024-04-25T11:25:34.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.128+0000 7f70d77fe700 1 --2- 172.21.15.90:0/876020303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70d8107f00 0x7f70d8108310 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f70cc0098a0 tx=0x7f70cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b3f1f984d6a9998 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:34.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.128+0000 7f70d67fc700 1 -- 172.21.15.90:0/876020303 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70cc004160 con 0x7f70d8107f00 2024-04-25T11:25:34.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.128+0000 7f70d67fc700 1 -- 172.21.15.90:0/876020303 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f70cc004300 con 0x7f70d8107f00 2024-04-25T11:25:34.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.128+0000 7f70d67fc700 1 -- 172.21.15.90:0/876020303 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70cc011af0 con 0x7f70d8107f00 2024-04-25T11:25:34.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.129+0000 7f70de1bf700 1 -- 172.21.15.90:0/876020303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70d8107f00 msgr2=0x7f70d8108310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:34.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.129+0000 7f70de1bf700 1 --2- 172.21.15.90:0/876020303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70d8107f00 0x7f70d8108310 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f70cc0098a0 tx=0x7f70cc009bb0 comp rx=0 tx=0).stop 2024-04-25T11:25:34.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.129+0000 7f70de1bf700 1 -- 172.21.15.90:0/876020303 shutdown_connections 2024-04-25T11:25:34.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.129+0000 7f70de1bf700 1 --2- 172.21.15.90:0/876020303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70d8107f00 0x7f70d8108310 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:34.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.129+0000 7f70de1bf700 1 -- 172.21.15.90:0/876020303 >> 172.21.15.90:0/876020303 conn(0x7f70d8103880 msgr2=0x7f70d8105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:34.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.129+0000 7f70de1bf700 1 -- 172.21.15.90:0/876020303 shutdown_connections 2024-04-25T11:25:34.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.129+0000 7f70de1bf700 1 -- 172.21.15.90:0/876020303 wait complete. 2024-04-25T11:25:34.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.130+0000 7f70de1bf700 1 Processor -- start 2024-04-25T11:25:34.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.130+0000 7f70de1bf700 1 -- start start 2024-04-25T11:25:34.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.130+0000 7f70de1bf700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70d8107f00 0x7f70d819dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:34.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.130+0000 7f70de1bf700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f70d810df30 con 0x7f70d8107f00 2024-04-25T11:25:34.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.130+0000 7f70d77fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70d8107f00 0x7f70d819dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:34.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.130+0000 7f70d77fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70d8107f00 0x7f70d819dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39260/0 (socket says 172.21.15.90:39260) 2024-04-25T11:25:34.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.130+0000 7f70d77fe700 1 -- 172.21.15.90:0/3740170078 learned_addr learned my addr 172.21.15.90:0/3740170078 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:34.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.131+0000 7f70d77fe700 1 -- 172.21.15.90:0/3740170078 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d819e4e0 con 0x7f70d8107f00 2024-04-25T11:25:34.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.131+0000 7f70d77fe700 1 --2- 172.21.15.90:0/3740170078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70d8107f00 0x7f70d819dfa0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f70cc0117b0 tx=0x7f70cc011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:34.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.131+0000 7f70d4ff9700 1 -- 172.21.15.90:0/3740170078 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70cc0039c0 con 0x7f70d8107f00 2024-04-25T11:25:34.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.131+0000 7f70de1bf700 1 -- 172.21.15.90:0/3740170078 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70d819e770 con 0x7f70d8107f00 2024-04-25T11:25:34.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.131+0000 7f70d4ff9700 1 -- 172.21.15.90:0/3740170078 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f70cc003b60 con 0x7f70d8107f00 2024-04-25T11:25:34.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.131+0000 7f70d4ff9700 1 -- 172.21.15.90:0/3740170078 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70cc0196a0 con 0x7f70d8107f00 2024-04-25T11:25:34.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.131+0000 7f70de1bf700 1 -- 172.21.15.90:0/3740170078 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70d81a1380 con 0x7f70d8107f00 2024-04-25T11:25:34.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.131+0000 7f70d4ff9700 1 -- 172.21.15.90:0/3740170078 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f70cc019cf0 con 0x7f70d8107f00 2024-04-25T11:25:34.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.131+0000 7f70d4ff9700 1 -- 172.21.15.90:0/3740170078 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f70cc024730 con 0x7f70d8107f00 2024-04-25T11:25:34.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.132+0000 7f70de1bf700 1 -- 172.21.15.90:0/3740170078 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70d8064cf0 con 0x7f70d8107f00 2024-04-25T11:25:34.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.134+0000 7f70d4ff9700 1 -- 172.21.15.90:0/3740170078 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f70cc0042a0 con 0x7f70d8107f00 2024-04-25T11:25:34.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.207+0000 7f70de1bf700 1 -- 172.21.15.90:0/3740170078 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f70d81a1670 con 0x7f70d8107f00 2024-04-25T11:25:34.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.208+0000 7f70d4ff9700 1 -- 172.21.15.90:0/3740170078 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f70cc01abb0 con 0x7f70d8107f00 2024-04-25T11:25:34.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.209+0000 7f70de1bf700 1 -- 172.21.15.90:0/3740170078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70d8107f00 msgr2=0x7f70d819dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:34.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.209+0000 7f70de1bf700 1 --2- 172.21.15.90:0/3740170078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70d8107f00 0x7f70d819dfa0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f70cc0117b0 tx=0x7f70cc011b70 comp rx=0 tx=0).stop 2024-04-25T11:25:34.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.209+0000 7f70de1bf700 1 -- 172.21.15.90:0/3740170078 shutdown_connections 2024-04-25T11:25:34.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.209+0000 7f70de1bf700 1 --2- 172.21.15.90:0/3740170078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70d8107f00 0x7f70d819dfa0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:34.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.209+0000 7f70de1bf700 1 -- 172.21.15.90:0/3740170078 >> 172.21.15.90:0/3740170078 conn(0x7f70d8103880 msgr2=0x7f70d8104280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:34.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.209+0000 7f70de1bf700 1 -- 172.21.15.90:0/3740170078 shutdown_connections 2024-04-25T11:25:34.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:34.209+0000 7f70de1bf700 1 -- 172.21.15.90:0/3740170078 wait complete. 2024-04-25T11:25:34.879 INFO:teuthology.orchestra.run.smithi090.stdout:mgr not available, waiting (2/15)... 2024-04-25T11:25:34.958 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:34 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:34.663+0000 7f39f5e52200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T11:25:34.958 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:34 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:34.759+0000 7f39f5e52200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T11:25:34.959 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:34 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:34.965+0000 7f39f5e52200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T11:25:35.729 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:35 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:35.481+0000 7f39f5e52200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T11:25:35.729 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:35 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:35.574+0000 7f39f5e52200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T11:25:36.144 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:35 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:35.814+0000 7f39f5e52200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T11:25:36.144 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:35 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:35.945+0000 7f39f5e52200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T11:25:36.144 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:36 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:36.039+0000 7f39f5e52200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T11:25:36.144 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:36 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:36.151+0000 7f39f5e52200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T11:25:36.730 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:36 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:36.450+0000 7f39f5e52200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T11:25:37.237 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:37 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:37.055+0000 7f39f5e52200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T11:25:37.729 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:37 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:37.280+0000 7f39f5e52200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T11:25:37.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:37 smithi090 ceph-mon[29575]: Activating manager daemon x 2024-04-25T11:25:37.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:37 smithi090 ceph-mon[29575]: mgrmap e2: x(active, starting, since 0.00363221s) 2024-04-25T11:25:37.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:37 smithi090 ceph-mon[29575]: from='mgr.14100 172.21.15.90:0/394154035' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T11:25:37.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:37 smithi090 ceph-mon[29575]: from='mgr.14100 172.21.15.90:0/394154035' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T11:25:37.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:37 smithi090 ceph-mon[29575]: from='mgr.14100 172.21.15.90:0/394154035' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T11:25:37.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:37 smithi090 ceph-mon[29575]: from='mgr.14100 172.21.15.90:0/394154035' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T11:25:37.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:37 smithi090 ceph-mon[29575]: from='mgr.14100 172.21.15.90:0/394154035' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T11:25:37.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:37 smithi090 ceph-mon[29575]: Manager daemon x is now available 2024-04-25T11:25:37.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:37 smithi090 ceph-mon[29575]: from='mgr.14100 172.21.15.90:0/394154035' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T11:25:37.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:37 smithi090 ceph-mon[29575]: from='mgr.14100 172.21.15.90:0/394154035' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T11:25:37.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:37 smithi090 ceph-mon[29575]: from='mgr.14100 172.21.15.90:0/394154035' entity='mgr.x' 2024-04-25T11:25:37.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:37 smithi090 ceph-mon[29575]: from='mgr.14100 172.21.15.90:0/394154035' entity='mgr.x' 2024-04-25T11:25:37.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:37 smithi090 ceph-mon[29575]: from='mgr.14100 172.21.15.90:0/394154035' entity='mgr.x' 2024-04-25T11:25:38.062 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-25T11:25:38.062 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-04-25T11:25:38.063 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsid": "58bd4e7a-02f6-11ef-bc93-c7b262605968", 2024-04-25T11:25:38.063 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T11:25:38.063 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T11:25:38.063 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T11:25:38.063 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T11:25:38.063 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:38.063 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T11:25:38.063 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T11:25:38.063 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 0 2024-04-25T11:25:38.063 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-25T11:25:38.063 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T11:25:38.063 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T11:25:38.063 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-25T11:25:38.063 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-04-25T11:25:38.064 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T11:25:38.064 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T11:25:38.064 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T11:25:38.064 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T11:25:38.064 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:38.064 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T11:25:38.064 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T11:25:38.064 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T11:25:38.064 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T11:25:38.064 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T11:25:38.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T11:25:38.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T11:25:38.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T11:25:38.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:38.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T11:25:38.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T11:25:38.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T11:25:38.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T11:25:38.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T11:25:38.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T11:25:38.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T11:25:38.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T11:25:38.072 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T11:25:38.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:38.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T11:25:38.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T11:25:38.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T11:25:38.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T11:25:38.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:38.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T11:25:38.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T11:25:38.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T11:25:38.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T11:25:38.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T11:25:38.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T11:25:38.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T11:25:38.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-25T11:25:38.073 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T11:25:38.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:38.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T11:25:38.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T11:25:38.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T11:25:23.575127+0000", 2024-04-25T11:25:38.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T11:25:38.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:38.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T11:25:38.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-04-25T11:25:38.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.355+0000 7f77cba75700 1 Processor -- start 2024-04-25T11:25:38.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.355+0000 7f77cba75700 1 -- start start 2024-04-25T11:25:38.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.355+0000 7f77cba75700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77c410b170 0x7f77c410b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:38.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.355+0000 7f77cba75700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f77c410bb50 con 0x7f77c410b170 2024-04-25T11:25:38.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.355+0000 7f77c9811700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77c410b170 0x7f77c410b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:38.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.355+0000 7f77c9811700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77c410b170 0x7f77c410b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39322/0 (socket says 172.21.15.90:39322) 2024-04-25T11:25:38.074 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.355+0000 7f77c9811700 1 -- 172.21.15.90:0/3308808686 learned_addr learned my addr 172.21.15.90:0/3308808686 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:38.075 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.356+0000 7f77c9811700 1 -- 172.21.15.90:0/3308808686 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77c410bcd0 con 0x7f77c410b170 2024-04-25T11:25:38.075 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.356+0000 7f77c9811700 1 --2- 172.21.15.90:0/3308808686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77c410b170 0x7f77c410b580 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f77b40098a0 tx=0x7f77b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a72acc4b38ecaab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:38.075 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.356+0000 7f77c880f700 1 -- 172.21.15.90:0/3308808686 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77b4004160 con 0x7f77c410b170 2024-04-25T11:25:38.075 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.356+0000 7f77c880f700 1 -- 172.21.15.90:0/3308808686 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f77b4004300 con 0x7f77c410b170 2024-04-25T11:25:38.075 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.356+0000 7f77c880f700 1 -- 172.21.15.90:0/3308808686 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77b4011af0 con 0x7f77c410b170 2024-04-25T11:25:38.075 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.357+0000 7f77cba75700 1 -- 172.21.15.90:0/3308808686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77c410b170 msgr2=0x7f77c410b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:38.075 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.357+0000 7f77cba75700 1 --2- 172.21.15.90:0/3308808686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77c410b170 0x7f77c410b580 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f77b40098a0 tx=0x7f77b4009bb0 comp rx=0 tx=0).stop 2024-04-25T11:25:38.075 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.357+0000 7f77cba75700 1 -- 172.21.15.90:0/3308808686 shutdown_connections 2024-04-25T11:25:38.075 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.357+0000 7f77cba75700 1 --2- 172.21.15.90:0/3308808686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77c410b170 0x7f77c410b580 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:38.075 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.357+0000 7f77cba75700 1 -- 172.21.15.90:0/3308808686 >> 172.21.15.90:0/3308808686 conn(0x7f77c41038d0 msgr2=0x7f77c4105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:38.075 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.357+0000 7f77cba75700 1 -- 172.21.15.90:0/3308808686 shutdown_connections 2024-04-25T11:25:38.075 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.357+0000 7f77cba75700 1 -- 172.21.15.90:0/3308808686 wait complete. 2024-04-25T11:25:38.075 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.357+0000 7f77cba75700 1 Processor -- start 2024-04-25T11:25:38.076 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.357+0000 7f77cba75700 1 -- start start 2024-04-25T11:25:38.076 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.358+0000 7f77cba75700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77c410b170 0x7f77c419e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:38.076 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.358+0000 7f77cba75700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f77c4108a20 con 0x7f77c410b170 2024-04-25T11:25:38.076 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.358+0000 7f77c9811700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77c410b170 0x7f77c419e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:38.076 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.358+0000 7f77c9811700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77c410b170 0x7f77c419e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39326/0 (socket says 172.21.15.90:39326) 2024-04-25T11:25:38.076 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.358+0000 7f77c9811700 1 -- 172.21.15.90:0/2002903666 learned_addr learned my addr 172.21.15.90:0/2002903666 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:38.076 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.358+0000 7f77c9811700 1 -- 172.21.15.90:0/2002903666 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77c419e550 con 0x7f77c410b170 2024-04-25T11:25:38.076 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.358+0000 7f77c9811700 1 --2- 172.21.15.90:0/2002903666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77c410b170 0x7f77c419e010 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f77b4009fb0 tx=0x7f77b4011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:38.076 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.358+0000 7f77baffd700 1 -- 172.21.15.90:0/2002903666 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77b4011bc0 con 0x7f77c410b170 2024-04-25T11:25:38.076 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.358+0000 7f77baffd700 1 -- 172.21.15.90:0/2002903666 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f77b4024440 con 0x7f77c410b170 2024-04-25T11:25:38.076 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.358+0000 7f77baffd700 1 -- 172.21.15.90:0/2002903666 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77b401a430 con 0x7f77c410b170 2024-04-25T11:25:38.076 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.358+0000 7f77cba75700 1 -- 172.21.15.90:0/2002903666 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77c419e7e0 con 0x7f77c410b170 2024-04-25T11:25:38.076 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.358+0000 7f77cba75700 1 -- 172.21.15.90:0/2002903666 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77c41a13f0 con 0x7f77c410b170 2024-04-25T11:25:38.076 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.359+0000 7f77baffd700 1 -- 172.21.15.90:0/2002903666 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 2) ==== 49272+0+0 (secure 0 0 0) 0x7f77b401a6c0 con 0x7f77c410b170 2024-04-25T11:25:38.077 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.360+0000 7f77baffd700 1 -- 172.21.15.90:0/2002903666 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f77b40521c0 con 0x7f77c410b170 2024-04-25T11:25:38.077 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.359+0000 7f77cba75700 1 -- 172.21.15.90:0/2002903666 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77c4064cf0 con 0x7f77c410b170 2024-04-25T11:25:38.077 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.362+0000 7f77baffd700 1 -- 172.21.15.90:0/2002903666 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f77b4029df0 con 0x7f77c410b170 2024-04-25T11:25:38.077 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.431+0000 7f77cba75700 1 -- 172.21.15.90:0/2002903666 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f77c41a19e0 con 0x7f77c410b170 2024-04-25T11:25:38.077 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.432+0000 7f77baffd700 1 -- 172.21.15.90:0/2002903666 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f77b4021d20 con 0x7f77c410b170 2024-04-25T11:25:38.077 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.433+0000 7f77cba75700 1 -- 172.21.15.90:0/2002903666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77c410b170 msgr2=0x7f77c419e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:38.077 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.433+0000 7f77cba75700 1 --2- 172.21.15.90:0/2002903666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77c410b170 0x7f77c419e010 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f77b4009fb0 tx=0x7f77b4011890 comp rx=0 tx=0).stop 2024-04-25T11:25:38.077 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.433+0000 7f77cba75700 1 -- 172.21.15.90:0/2002903666 shutdown_connections 2024-04-25T11:25:38.077 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.433+0000 7f77cba75700 1 --2- 172.21.15.90:0/2002903666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77c410b170 0x7f77c419e010 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:38.077 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.433+0000 7f77cba75700 1 -- 172.21.15.90:0/2002903666 >> 172.21.15.90:0/2002903666 conn(0x7f77c41038d0 msgr2=0x7f77c41042a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:38.077 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.433+0000 7f77cba75700 1 -- 172.21.15.90:0/2002903666 shutdown_connections 2024-04-25T11:25:38.077 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:37.433+0000 7f77cba75700 1 -- 172.21.15.90:0/2002903666 wait complete. 2024-04-25T11:25:38.077 INFO:teuthology.orchestra.run.smithi090.stdout:mgr not available, waiting (3/15)... 2024-04-25T11:25:38.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:38 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2002903666' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T11:25:38.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:38 smithi090 ceph-mon[29575]: mgrmap e3: x(active, since 1.00788s) 2024-04-25T11:25:40.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:40 smithi090 ceph-mon[29575]: mgrmap e4: x(active, since 2s) 2024-04-25T11:25:41.387 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-25T11:25:41.387 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-04-25T11:25:41.387 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsid": "58bd4e7a-02f6-11ef-bc93-c7b262605968", 2024-04-25T11:25:41.387 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T11:25:41.387 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T11:25:41.387 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T11:25:41.387 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T11:25:41.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:41.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T11:25:41.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T11:25:41.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 0 2024-04-25T11:25:41.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-25T11:25:41.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T11:25:41.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T11:25:41.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-25T11:25:41.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-25T11:25:41.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T11:25:41.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T11:25:41.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T11:25:41.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T11:25:41.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:41.388 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T11:25:41.389 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T11:25:41.389 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T11:25:41.389 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T11:25:41.389 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T11:25:41.397 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T11:25:41.398 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T11:25:41.398 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T11:25:41.398 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:41.398 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T11:25:41.398 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T11:25:41.398 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T11:25:41.398 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T11:25:41.398 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T11:25:41.398 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T11:25:41.398 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T11:25:41.398 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T11:25:41.398 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T11:25:41.398 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:41.399 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T11:25:41.399 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T11:25:41.399 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T11:25:41.399 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T11:25:41.399 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:41.399 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T11:25:41.399 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T11:25:41.399 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T11:25:41.399 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T11:25:41.399 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T11:25:41.399 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T11:25:41.399 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T11:25:41.399 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-25T11:25:41.400 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T11:25:41.400 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:41.400 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T11:25:41.400 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T11:25:41.400 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T11:25:23.575127+0000", 2024-04-25T11:25:41.401 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T11:25:41.401 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-25T11:25:41.401 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T11:25:41.401 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-04-25T11:25:41.401 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.533+0000 7f833593d700 1 Processor -- start 2024-04-25T11:25:41.401 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.533+0000 7f833593d700 1 -- start start 2024-04-25T11:25:41.401 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.533+0000 7f833593d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8330109be0 0x7f8330109ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:41.401 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.533+0000 7f833593d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f833010a5e0 con 0x7f8330109be0 2024-04-25T11:25:41.401 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.533+0000 7f832effd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8330109be0 0x7f8330109ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:41.401 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.533+0000 7f832effd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8330109be0 0x7f8330109ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39340/0 (socket says 172.21.15.90:39340) 2024-04-25T11:25:41.401 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.533+0000 7f832effd700 1 -- 172.21.15.90:0/3436528421 learned_addr learned my addr 172.21.15.90:0/3436528421 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:41.402 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.533+0000 7f832effd700 1 -- 172.21.15.90:0/3436528421 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f833010ad10 con 0x7f8330109be0 2024-04-25T11:25:41.402 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.534+0000 7f832effd700 1 --2- 172.21.15.90:0/3436528421 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8330109be0 0x7f8330109ff0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f83200098a0 tx=0x7f8320009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c17b93bc2a7ab00 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:41.402 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.534+0000 7f832dffb700 1 -- 172.21.15.90:0/3436528421 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8320004160 con 0x7f8330109be0 2024-04-25T11:25:41.402 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.534+0000 7f832dffb700 1 -- 172.21.15.90:0/3436528421 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8320004300 con 0x7f8330109be0 2024-04-25T11:25:41.402 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.534+0000 7f832dffb700 1 -- 172.21.15.90:0/3436528421 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8320011af0 con 0x7f8330109be0 2024-04-25T11:25:41.402 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.534+0000 7f833593d700 1 -- 172.21.15.90:0/3436528421 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8330109be0 msgr2=0x7f8330109ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:41.402 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.534+0000 7f833593d700 1 --2- 172.21.15.90:0/3436528421 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8330109be0 0x7f8330109ff0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f83200098a0 tx=0x7f8320009bb0 comp rx=0 tx=0).stop 2024-04-25T11:25:41.402 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.535+0000 7f833593d700 1 -- 172.21.15.90:0/3436528421 shutdown_connections 2024-04-25T11:25:41.402 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.535+0000 7f833593d700 1 --2- 172.21.15.90:0/3436528421 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8330109be0 0x7f8330109ff0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:41.402 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.535+0000 7f833593d700 1 -- 172.21.15.90:0/3436528421 >> 172.21.15.90:0/3436528421 conn(0x7f8330105220 msgr2=0x7f83301076d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:41.402 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.535+0000 7f833593d700 1 -- 172.21.15.90:0/3436528421 shutdown_connections 2024-04-25T11:25:41.402 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.535+0000 7f833593d700 1 -- 172.21.15.90:0/3436528421 wait complete. 2024-04-25T11:25:41.403 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.535+0000 7f833593d700 1 Processor -- start 2024-04-25T11:25:41.403 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.535+0000 7f833593d700 1 -- start start 2024-04-25T11:25:41.403 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.536+0000 7f833593d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8330109be0 0x7f833019e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:41.403 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.536+0000 7f833593d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f833010e0c0 con 0x7f8330109be0 2024-04-25T11:25:41.403 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.536+0000 7f832effd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8330109be0 0x7f833019e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:41.403 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.536+0000 7f832effd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8330109be0 0x7f833019e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39342/0 (socket says 172.21.15.90:39342) 2024-04-25T11:25:41.403 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.536+0000 7f832effd700 1 -- 172.21.15.90:0/1631384277 learned_addr learned my addr 172.21.15.90:0/1631384277 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:41.403 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.536+0000 7f832effd700 1 -- 172.21.15.90:0/1631384277 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f833019e540 con 0x7f8330109be0 2024-04-25T11:25:41.403 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.536+0000 7f832effd700 1 --2- 172.21.15.90:0/1631384277 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8330109be0 0x7f833019e000 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f83200090e0 tx=0x7f8320011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:41.403 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.536+0000 7f833493b700 1 -- 172.21.15.90:0/1631384277 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83200039e0 con 0x7f8330109be0 2024-04-25T11:25:41.403 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.536+0000 7f833593d700 1 -- 172.21.15.90:0/1631384277 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f833019e7d0 con 0x7f8330109be0 2024-04-25T11:25:41.403 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.536+0000 7f833493b700 1 -- 172.21.15.90:0/1631384277 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8320003b80 con 0x7f8330109be0 2024-04-25T11:25:41.403 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.536+0000 7f833493b700 1 -- 172.21.15.90:0/1631384277 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8320024690 con 0x7f8330109be0 2024-04-25T11:25:41.403 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.536+0000 7f833593d700 1 -- 172.21.15.90:0/1631384277 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83301a13e0 con 0x7f8330109be0 2024-04-25T11:25:41.404 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.537+0000 7f833493b700 1 -- 172.21.15.90:0/1631384277 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 4) ==== 49505+0+0 (secure 0 0 0) 0x7f8320022020 con 0x7f8330109be0 2024-04-25T11:25:41.404 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.537+0000 7f833593d700 1 -- 172.21.15.90:0/1631384277 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8330064cf0 con 0x7f8330109be0 2024-04-25T11:25:41.404 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.537+0000 7f833493b700 1 --2- 172.21.15.90:0/1631384277 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f831803d520 0x7f831803fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:41.404 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.537+0000 7f833493b700 1 -- 172.21.15.90:0/1631384277 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f83200521f0 con 0x7f8330109be0 2024-04-25T11:25:41.404 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.538+0000 7f832e7fc700 1 --2- 172.21.15.90:0/1631384277 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f831803d520 0x7f831803fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:41.404 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.538+0000 7f832e7fc700 1 --2- 172.21.15.90:0/1631384277 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f831803d520 0x7f831803fa30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8324006fd0 tx=0x7f8324006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:41.404 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.543+0000 7f833493b700 1 -- 172.21.15.90:0/1631384277 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8320024a40 con 0x7f8330109be0 2024-04-25T11:25:41.404 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.819+0000 7f833593d700 1 -- 172.21.15.90:0/1631384277 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f83301a1a60 con 0x7f8330109be0 2024-04-25T11:25:41.404 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.820+0000 7f833493b700 1 -- 172.21.15.90:0/1631384277 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f83200172b0 con 0x7f8330109be0 2024-04-25T11:25:41.404 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.823+0000 7f833593d700 1 -- 172.21.15.90:0/1631384277 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f831803d520 msgr2=0x7f831803fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:41.404 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.823+0000 7f833593d700 1 --2- 172.21.15.90:0/1631384277 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f831803d520 0x7f831803fa30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8324006fd0 tx=0x7f8324006e40 comp rx=0 tx=0).stop 2024-04-25T11:25:41.404 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.823+0000 7f833593d700 1 -- 172.21.15.90:0/1631384277 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8330109be0 msgr2=0x7f833019e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:41.404 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.823+0000 7f833593d700 1 --2- 172.21.15.90:0/1631384277 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8330109be0 0x7f833019e000 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f83200090e0 tx=0x7f8320011940 comp rx=0 tx=0).stop 2024-04-25T11:25:41.404 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.823+0000 7f833593d700 1 -- 172.21.15.90:0/1631384277 shutdown_connections 2024-04-25T11:25:41.405 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.823+0000 7f833593d700 1 --2- 172.21.15.90:0/1631384277 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f831803d520 0x7f831803fa30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:41.405 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.823+0000 7f833593d700 1 --2- 172.21.15.90:0/1631384277 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8330109be0 0x7f833019e000 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:41.405 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.823+0000 7f833593d700 1 -- 172.21.15.90:0/1631384277 >> 172.21.15.90:0/1631384277 conn(0x7f8330105220 msgr2=0x7f8330105b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:41.405 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.823+0000 7f833593d700 1 -- 172.21.15.90:0/1631384277 shutdown_connections 2024-04-25T11:25:41.405 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:40.823+0000 7f833593d700 1 -- 172.21.15.90:0/1631384277 wait complete. 2024-04-25T11:25:41.405 INFO:teuthology.orchestra.run.smithi090.stdout:mgr is available 2024-04-25T11:25:41.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:41 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1631384277' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T11:25:42.649 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:42 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2075692679' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T11:25:42.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-25T11:25:42.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T11:25:42.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout fsid = 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:25:42.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-25T11:25:42.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T11:25:42.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T11:25:42.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T11:25:42.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T11:25:42.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-25T11:25:42.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T11:25:42.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T11:25:42.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-25T11:25:42.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T11:25:42.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-04-25T11:25:42.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-25T11:25:42.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-25T11:25:42.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-25T11:25:42.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T11:25:42.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T11:25:42.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.824+0000 7f1b1f275700 1 Processor -- start 2024-04-25T11:25:42.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.824+0000 7f1b1f275700 1 -- start start 2024-04-25T11:25:42.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.824+0000 7f1b1f275700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b18109bb0 0x7f1b18109fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:42.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.824+0000 7f1b1f275700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1810a5b0 con 0x7f1b18109bb0 2024-04-25T11:25:42.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.824+0000 7f1b1d011700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b18109bb0 0x7f1b18109fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:42.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.824+0000 7f1b1d011700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b18109bb0 0x7f1b18109fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59476/0 (socket says 172.21.15.90:59476) 2024-04-25T11:25:42.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.824+0000 7f1b1d011700 1 -- 172.21.15.90:0/1712249876 learned_addr learned my addr 172.21.15.90:0/1712249876 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:42.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.825+0000 7f1b1d011700 1 -- 172.21.15.90:0/1712249876 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b1810ace0 con 0x7f1b18109bb0 2024-04-25T11:25:42.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.825+0000 7f1b1d011700 1 --2- 172.21.15.90:0/1712249876 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b18109bb0 0x7f1b18109fc0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1b140098a0 tx=0x7f1b14009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=686fb525d9fe40b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:42.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.825+0000 7f1b0ffff700 1 -- 172.21.15.90:0/1712249876 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b14004160 con 0x7f1b18109bb0 2024-04-25T11:25:42.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.825+0000 7f1b0ffff700 1 -- 172.21.15.90:0/1712249876 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1b14004300 con 0x7f1b18109bb0 2024-04-25T11:25:42.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.825+0000 7f1b0ffff700 1 -- 172.21.15.90:0/1712249876 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b14011af0 con 0x7f1b18109bb0 2024-04-25T11:25:42.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.826+0000 7f1b1f275700 1 -- 172.21.15.90:0/1712249876 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b18109bb0 msgr2=0x7f1b18109fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:42.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.826+0000 7f1b1f275700 1 --2- 172.21.15.90:0/1712249876 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b18109bb0 0x7f1b18109fc0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1b140098a0 tx=0x7f1b14009bb0 comp rx=0 tx=0).stop 2024-04-25T11:25:42.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.826+0000 7f1b1f275700 1 -- 172.21.15.90:0/1712249876 shutdown_connections 2024-04-25T11:25:42.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.826+0000 7f1b1f275700 1 --2- 172.21.15.90:0/1712249876 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b18109bb0 0x7f1b18109fc0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:42.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.826+0000 7f1b1f275700 1 -- 172.21.15.90:0/1712249876 >> 172.21.15.90:0/1712249876 conn(0x7f1b181051f0 msgr2=0x7f1b181076a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:42.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.826+0000 7f1b1f275700 1 -- 172.21.15.90:0/1712249876 shutdown_connections 2024-04-25T11:25:42.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.826+0000 7f1b1f275700 1 -- 172.21.15.90:0/1712249876 wait complete. 2024-04-25T11:25:42.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.826+0000 7f1b1f275700 1 Processor -- start 2024-04-25T11:25:42.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.827+0000 7f1b1f275700 1 -- start start 2024-04-25T11:25:42.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.827+0000 7f1b1f275700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b18109bb0 0x7f1b1819dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:42.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.827+0000 7f1b1f275700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1810e090 con 0x7f1b18109bb0 2024-04-25T11:25:42.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.827+0000 7f1b1d011700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b18109bb0 0x7f1b1819dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:42.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.827+0000 7f1b1d011700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b18109bb0 0x7f1b1819dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59484/0 (socket says 172.21.15.90:59484) 2024-04-25T11:25:42.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.827+0000 7f1b1d011700 1 -- 172.21.15.90:0/2075692679 learned_addr learned my addr 172.21.15.90:0/2075692679 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:42.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.827+0000 7f1b1d011700 1 -- 172.21.15.90:0/2075692679 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b1819e510 con 0x7f1b18109bb0 2024-04-25T11:25:42.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.827+0000 7f1b1d011700 1 --2- 172.21.15.90:0/2075692679 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b18109bb0 0x7f1b1819dfd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1b14011670 tx=0x7f1b14004280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:42.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.827+0000 7f1b0e7fc700 1 -- 172.21.15.90:0/2075692679 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b140036a0 con 0x7f1b18109bb0 2024-04-25T11:25:42.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.827+0000 7f1b0e7fc700 1 -- 172.21.15.90:0/2075692679 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1b14003cd0 con 0x7f1b18109bb0 2024-04-25T11:25:42.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.827+0000 7f1b1f275700 1 -- 172.21.15.90:0/2075692679 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b1819e7a0 con 0x7f1b18109bb0 2024-04-25T11:25:42.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.827+0000 7f1b0e7fc700 1 -- 172.21.15.90:0/2075692679 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b140193c0 con 0x7f1b18109bb0 2024-04-25T11:25:42.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.827+0000 7f1b1f275700 1 -- 172.21.15.90:0/2075692679 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b181a13b0 con 0x7f1b18109bb0 2024-04-25T11:25:42.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.828+0000 7f1b0e7fc700 1 -- 172.21.15.90:0/2075692679 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 4) ==== 49505+0+0 (secure 0 0 0) 0x7f1b14019680 con 0x7f1b18109bb0 2024-04-25T11:25:42.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.828+0000 7f1b0e7fc700 1 --2- 172.21.15.90:0/2075692679 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f1b0403d4d0 0x7f1b0403f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:42.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.829+0000 7f1b0e7fc700 1 -- 172.21.15.90:0/2075692679 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1b14053490 con 0x7f1b18109bb0 2024-04-25T11:25:42.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.829+0000 7f1b1c810700 1 --2- 172.21.15.90:0/2075692679 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f1b0403d4d0 0x7f1b0403f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:42.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.829+0000 7f1b1f275700 1 -- 172.21.15.90:0/2075692679 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b18064cf0 con 0x7f1b18109bb0 2024-04-25T11:25:42.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.829+0000 7f1b1c810700 1 --2- 172.21.15.90:0/2075692679 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f1b0403d4d0 0x7f1b0403f9e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1b08006fd0 tx=0x7f1b08006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:42.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:41.834+0000 7f1b0e7fc700 1 -- 172.21.15.90:0/2075692679 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1b14017e90 con 0x7f1b18109bb0 2024-04-25T11:25:42.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:42.028+0000 7f1b1f275700 1 -- 172.21.15.90:0/2075692679 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f1b181a1580 con 0x7f1b18109bb0 2024-04-25T11:25:42.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:42.030+0000 7f1b0e7fc700 1 -- 172.21.15.90:0/2075692679 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+436 (secure 0 0 0) 0x7f1b14052eb0 con 0x7f1b18109bb0 2024-04-25T11:25:42.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:42.032+0000 7f1b1f275700 1 -- 172.21.15.90:0/2075692679 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f1b0403d4d0 msgr2=0x7f1b0403f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:42.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:42.032+0000 7f1b1f275700 1 --2- 172.21.15.90:0/2075692679 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f1b0403d4d0 0x7f1b0403f9e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1b08006fd0 tx=0x7f1b08006e40 comp rx=0 tx=0).stop 2024-04-25T11:25:42.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:42.032+0000 7f1b1f275700 1 -- 172.21.15.90:0/2075692679 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b18109bb0 msgr2=0x7f1b1819dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:42.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:42.032+0000 7f1b1f275700 1 --2- 172.21.15.90:0/2075692679 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b18109bb0 0x7f1b1819dfd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1b14011670 tx=0x7f1b14004280 comp rx=0 tx=0).stop 2024-04-25T11:25:42.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:42.032+0000 7f1b1f275700 1 -- 172.21.15.90:0/2075692679 shutdown_connections 2024-04-25T11:25:42.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:42.032+0000 7f1b1f275700 1 --2- 172.21.15.90:0/2075692679 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f1b0403d4d0 0x7f1b0403f9e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:42.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:42.032+0000 7f1b1f275700 1 --2- 172.21.15.90:0/2075692679 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b18109bb0 0x7f1b1819dfd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:42.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:42.032+0000 7f1b1f275700 1 -- 172.21.15.90:0/2075692679 >> 172.21.15.90:0/2075692679 conn(0x7f1b181051f0 msgr2=0x7f1b18105b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:42.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:42.032+0000 7f1b1f275700 1 -- 172.21.15.90:0/2075692679 shutdown_connections 2024-04-25T11:25:42.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:42.032+0000 7f1b1f275700 1 -- 172.21.15.90:0/2075692679 wait complete. 2024-04-25T11:25:42.664 INFO:teuthology.orchestra.run.smithi090.stdout:Enabling cephadm module... 2024-04-25T11:25:43.708 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:43 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1539306625' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-25T11:25:43.708 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:43 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: ignoring --setuser ceph since I am not root 2024-04-25T11:25:43.708 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:43 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: ignoring --setgroup ceph since I am not root 2024-04-25T11:25:43.708 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:43 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:43.589+0000 7f4dcf581200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T11:25:43.967 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:43 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:43.776+0000 7f4dcf581200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T11:25:43.967 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:43 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:43.878+0000 7f4dcf581200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T11:25:43.974 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.077+0000 7faa90479700 1 Processor -- start 2024-04-25T11:25:43.974 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.077+0000 7faa90479700 1 -- start start 2024-04-25T11:25:43.974 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.077+0000 7faa90479700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa88108ae0 0x7faa88108ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:43.974 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.077+0000 7faa90479700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7faa881094e0 con 0x7faa88108ae0 2024-04-25T11:25:43.974 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.077+0000 7faa8e215700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa88108ae0 0x7faa88108ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:43.974 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.077+0000 7faa8e215700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa88108ae0 0x7faa88108ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59490/0 (socket says 172.21.15.90:59490) 2024-04-25T11:25:43.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.077+0000 7faa8e215700 1 -- 172.21.15.90:0/3461475114 learned_addr learned my addr 172.21.15.90:0/3461475114 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:43.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.078+0000 7faa8e215700 1 -- 172.21.15.90:0/3461475114 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa88109c10 con 0x7faa88108ae0 2024-04-25T11:25:43.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.078+0000 7faa8e215700 1 --2- 172.21.15.90:0/3461475114 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa88108ae0 0x7faa88108ef0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7faa840098a0 tx=0x7faa84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f5fbf27d0808476 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:43.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.078+0000 7faa8d213700 1 -- 172.21.15.90:0/3461475114 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa84004160 con 0x7faa88108ae0 2024-04-25T11:25:43.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.078+0000 7faa8d213700 1 -- 172.21.15.90:0/3461475114 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7faa84004300 con 0x7faa88108ae0 2024-04-25T11:25:43.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.078+0000 7faa8d213700 1 -- 172.21.15.90:0/3461475114 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa84011af0 con 0x7faa88108ae0 2024-04-25T11:25:43.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.079+0000 7faa90479700 1 -- 172.21.15.90:0/3461475114 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa88108ae0 msgr2=0x7faa88108ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:43.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.079+0000 7faa90479700 1 --2- 172.21.15.90:0/3461475114 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa88108ae0 0x7faa88108ef0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7faa840098a0 tx=0x7faa84009bb0 comp rx=0 tx=0).stop 2024-04-25T11:25:43.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.079+0000 7faa90479700 1 -- 172.21.15.90:0/3461475114 shutdown_connections 2024-04-25T11:25:43.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.079+0000 7faa90479700 1 --2- 172.21.15.90:0/3461475114 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa88108ae0 0x7faa88108ef0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:43.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.079+0000 7faa90479700 1 -- 172.21.15.90:0/3461475114 >> 172.21.15.90:0/3461475114 conn(0x7faa88104140 msgr2=0x7faa881065d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:43.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.079+0000 7faa90479700 1 -- 172.21.15.90:0/3461475114 shutdown_connections 2024-04-25T11:25:43.975 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.079+0000 7faa90479700 1 -- 172.21.15.90:0/3461475114 wait complete. 2024-04-25T11:25:43.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.079+0000 7faa90479700 1 Processor -- start 2024-04-25T11:25:43.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.079+0000 7faa90479700 1 -- start start 2024-04-25T11:25:43.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.080+0000 7faa90479700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa88108ae0 0x7faa8819e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:43.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.080+0000 7faa90479700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7faa8810cfc0 con 0x7faa88108ae0 2024-04-25T11:25:43.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.080+0000 7faa8e215700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa88108ae0 0x7faa8819e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:43.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.080+0000 7faa8e215700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa88108ae0 0x7faa8819e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59506/0 (socket says 172.21.15.90:59506) 2024-04-25T11:25:43.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.080+0000 7faa8e215700 1 -- 172.21.15.90:0/1539306625 learned_addr learned my addr 172.21.15.90:0/1539306625 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:43.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.080+0000 7faa8e215700 1 -- 172.21.15.90:0/1539306625 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa8819e540 con 0x7faa88108ae0 2024-04-25T11:25:43.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.080+0000 7faa8e215700 1 --2- 172.21.15.90:0/1539306625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa88108ae0 0x7faa8819e000 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7faa840090e0 tx=0x7faa84011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:43.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.080+0000 7faa7f7fe700 1 -- 172.21.15.90:0/1539306625 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa840039e0 con 0x7faa88108ae0 2024-04-25T11:25:43.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.080+0000 7faa7f7fe700 1 -- 172.21.15.90:0/1539306625 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7faa84003b80 con 0x7faa88108ae0 2024-04-25T11:25:43.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.080+0000 7faa90479700 1 -- 172.21.15.90:0/1539306625 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa8819e7d0 con 0x7faa88108ae0 2024-04-25T11:25:43.976 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.080+0000 7faa7f7fe700 1 -- 172.21.15.90:0/1539306625 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa84024690 con 0x7faa88108ae0 2024-04-25T11:25:43.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.080+0000 7faa90479700 1 -- 172.21.15.90:0/1539306625 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa881a13e0 con 0x7faa88108ae0 2024-04-25T11:25:43.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.081+0000 7faa7f7fe700 1 -- 172.21.15.90:0/1539306625 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 4) ==== 49505+0+0 (secure 0 0 0) 0x7faa84022020 con 0x7faa88108ae0 2024-04-25T11:25:43.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.081+0000 7faa7f7fe700 1 --2- 172.21.15.90:0/1539306625 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7faa7403d570 0x7faa7403fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:43.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.082+0000 7faa7f7fe700 1 -- 172.21.15.90:0/1539306625 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7faa84053080 con 0x7faa88108ae0 2024-04-25T11:25:43.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.082+0000 7faa90479700 1 -- 172.21.15.90:0/1539306625 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa88064cf0 con 0x7faa88108ae0 2024-04-25T11:25:43.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.082+0000 7faa8da14700 1 --2- 172.21.15.90:0/1539306625 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7faa7403d570 0x7faa7403fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:43.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.082+0000 7faa8da14700 1 --2- 172.21.15.90:0/1539306625 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7faa7403d570 0x7faa7403fa80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7faa78006fd0 tx=0x7faa78006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:43.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.087+0000 7faa7f7fe700 1 -- 172.21.15.90:0/1539306625 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faa840172b0 con 0x7faa88108ae0 2024-04-25T11:25:43.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.343+0000 7faa90479700 1 -- 172.21.15.90:0/1539306625 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7faa8802e650 con 0x7faa88108ae0 2024-04-25T11:25:43.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.395+0000 7faa7f7fe700 1 -- 172.21.15.90:0/1539306625 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7faa8400fdd0 con 0x7faa88108ae0 2024-04-25T11:25:43.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.397+0000 7faa7f7fe700 1 -- 172.21.15.90:0/1539306625 <== mon.0 v2:172.21.15.90:3300/0 8 ==== mgrmap(e 5) ==== 49516+0+0 (secure 0 0 0) 0x7faa840248d0 con 0x7faa88108ae0 2024-04-25T11:25:43.977 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.398+0000 7faa90479700 1 -- 172.21.15.90:0/1539306625 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7faa7403d570 msgr2=0x7faa7403fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:43.978 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.398+0000 7faa90479700 1 --2- 172.21.15.90:0/1539306625 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7faa7403d570 0x7faa7403fa80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7faa78006fd0 tx=0x7faa78006e40 comp rx=0 tx=0).stop 2024-04-25T11:25:43.978 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.398+0000 7faa90479700 1 -- 172.21.15.90:0/1539306625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa88108ae0 msgr2=0x7faa8819e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:43.978 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.398+0000 7faa90479700 1 --2- 172.21.15.90:0/1539306625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa88108ae0 0x7faa8819e000 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7faa840090e0 tx=0x7faa84011940 comp rx=0 tx=0).stop 2024-04-25T11:25:43.978 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.398+0000 7faa90479700 1 -- 172.21.15.90:0/1539306625 shutdown_connections 2024-04-25T11:25:43.978 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.398+0000 7faa90479700 1 --2- 172.21.15.90:0/1539306625 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7faa7403d570 0x7faa7403fa80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:43.978 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.398+0000 7faa90479700 1 --2- 172.21.15.90:0/1539306625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa88108ae0 0x7faa8819e000 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:43.978 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.398+0000 7faa90479700 1 -- 172.21.15.90:0/1539306625 >> 172.21.15.90:0/1539306625 conn(0x7faa88104140 msgr2=0x7faa88194560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:43.978 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.398+0000 7faa90479700 1 -- 172.21.15.90:0/1539306625 shutdown_connections 2024-04-25T11:25:43.978 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:43.398+0000 7faa90479700 1 -- 172.21.15.90:0/1539306625 wait complete. 2024-04-25T11:25:44.229 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:44 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:44.199+0000 7f4dcf581200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T11:25:44.646 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:44 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1539306625' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-25T11:25:44.646 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:44 smithi090 ceph-mon[29575]: mgrmap e5: x(active, since 6s) 2024-04-25T11:25:44.647 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:44 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:44.288+0000 7f4dcf581200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T11:25:45.317 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-04-25T11:25:45.318 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-25T11:25:45.318 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T11:25:45.318 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T11:25:45.318 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T11:25:45.318 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-04-25T11:25:45.318 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.446+0000 7f4d7377a700 1 Processor -- start 2024-04-25T11:25:45.318 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.446+0000 7f4d7377a700 1 -- start start 2024-04-25T11:25:45.318 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.447+0000 7f4d7377a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c10a020 0x7f4d6c10a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:45.318 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.447+0000 7f4d7377a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4d6c10aa00 con 0x7f4d6c10a020 2024-04-25T11:25:45.318 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.447+0000 7f4d71516700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c10a020 0x7f4d6c10a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:45.318 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.447+0000 7f4d71516700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c10a020 0x7f4d6c10a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59524/0 (socket says 172.21.15.90:59524) 2024-04-25T11:25:45.319 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.447+0000 7f4d71516700 1 -- 172.21.15.90:0/884971244 learned_addr learned my addr 172.21.15.90:0/884971244 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:45.319 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.447+0000 7f4d71516700 1 -- 172.21.15.90:0/884971244 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d6c108ca0 con 0x7f4d6c10a020 2024-04-25T11:25:45.319 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.448+0000 7f4d71516700 1 --2- 172.21.15.90:0/884971244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c10a020 0x7f4d6c10a430 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4d680098a0 tx=0x7f4d68009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49b90b67805494a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:45.319 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.448+0000 7f4d63fff700 1 -- 172.21.15.90:0/884971244 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d68004160 con 0x7f4d6c10a020 2024-04-25T11:25:45.319 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.448+0000 7f4d63fff700 1 -- 172.21.15.90:0/884971244 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4d68004300 con 0x7f4d6c10a020 2024-04-25T11:25:45.319 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.448+0000 7f4d63fff700 1 -- 172.21.15.90:0/884971244 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d68011af0 con 0x7f4d6c10a020 2024-04-25T11:25:45.319 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.449+0000 7f4d7377a700 1 -- 172.21.15.90:0/884971244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c10a020 msgr2=0x7f4d6c10a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:45.319 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.449+0000 7f4d7377a700 1 --2- 172.21.15.90:0/884971244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c10a020 0x7f4d6c10a430 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4d680098a0 tx=0x7f4d68009bb0 comp rx=0 tx=0).stop 2024-04-25T11:25:45.319 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.449+0000 7f4d7377a700 1 -- 172.21.15.90:0/884971244 shutdown_connections 2024-04-25T11:25:45.319 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.449+0000 7f4d7377a700 1 --2- 172.21.15.90:0/884971244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c10a020 0x7f4d6c10a430 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:45.319 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.449+0000 7f4d7377a700 1 -- 172.21.15.90:0/884971244 >> 172.21.15.90:0/884971244 conn(0x7f4d6c103880 msgr2=0x7f4d6c105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:45.319 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.449+0000 7f4d7377a700 1 -- 172.21.15.90:0/884971244 shutdown_connections 2024-04-25T11:25:45.320 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.449+0000 7f4d7377a700 1 -- 172.21.15.90:0/884971244 wait complete. 2024-04-25T11:25:45.320 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.449+0000 7f4d7377a700 1 Processor -- start 2024-04-25T11:25:45.320 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.449+0000 7f4d7377a700 1 -- start start 2024-04-25T11:25:45.320 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.450+0000 7f4d7377a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c10a020 0x7f4d6c105a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:45.320 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.450+0000 7f4d7377a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4d6c10d060 con 0x7f4d6c10a020 2024-04-25T11:25:45.320 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.450+0000 7f4d71516700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c10a020 0x7f4d6c105a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:45.320 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.450+0000 7f4d71516700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c10a020 0x7f4d6c105a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59530/0 (socket says 172.21.15.90:59530) 2024-04-25T11:25:45.320 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.450+0000 7f4d71516700 1 -- 172.21.15.90:0/1414090747 learned_addr learned my addr 172.21.15.90:0/1414090747 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:45.320 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.450+0000 7f4d71516700 1 -- 172.21.15.90:0/1414090747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d6c105fd0 con 0x7f4d6c10a020 2024-04-25T11:25:45.320 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.450+0000 7f4d71516700 1 --2- 172.21.15.90:0/1414090747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c10a020 0x7f4d6c105a90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4d680090e0 tx=0x7f4d68011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:45.320 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.450+0000 7f4d627fc700 1 -- 172.21.15.90:0/1414090747 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d680039e0 con 0x7f4d6c10a020 2024-04-25T11:25:45.320 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.450+0000 7f4d627fc700 1 -- 172.21.15.90:0/1414090747 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4d68003b80 con 0x7f4d6c10a020 2024-04-25T11:25:45.321 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.450+0000 7f4d7377a700 1 -- 172.21.15.90:0/1414090747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d6c104220 con 0x7f4d6c10a020 2024-04-25T11:25:45.321 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.450+0000 7f4d627fc700 1 -- 172.21.15.90:0/1414090747 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d680246e0 con 0x7f4d6c10a020 2024-04-25T11:25:45.321 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.451+0000 7f4d7377a700 1 -- 172.21.15.90:0/1414090747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d6c104700 con 0x7f4d6c10a020 2024-04-25T11:25:45.321 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.451+0000 7f4d627fc700 1 -- 172.21.15.90:0/1414090747 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 49516+0+0 (secure 0 0 0) 0x7f4d68022020 con 0x7f4d6c10a020 2024-04-25T11:25:45.321 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.452+0000 7f4d627fc700 1 --2- 172.21.15.90:0/1414090747 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f4d5803d5c0 0x7f4d5803fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:45.321 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.452+0000 7f4d627fc700 1 -- 172.21.15.90:0/1414090747 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4d680531b0 con 0x7f4d6c10a020 2024-04-25T11:25:45.321 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.452+0000 7f4d70d15700 1 -- 172.21.15.90:0/1414090747 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f4d5803d5c0 msgr2=0x7f4d5803fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/3122821311 2024-04-25T11:25:45.321 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.452+0000 7f4d7377a700 1 -- 172.21.15.90:0/1414090747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d6c064cf0 con 0x7f4d6c10a020 2024-04-25T11:25:45.321 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.452+0000 7f4d70d15700 1 --2- 172.21.15.90:0/1414090747 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f4d5803d5c0 0x7f4d5803fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T11:25:45.321 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.457+0000 7f4d627fc700 1 -- 172.21.15.90:0/1414090747 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4d6802c350 con 0x7f4d6c10a020 2024-04-25T11:25:45.321 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.653+0000 7f4d70d15700 1 -- 172.21.15.90:0/1414090747 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f4d5803d5c0 msgr2=0x7f4d5803fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/3122821311 2024-04-25T11:25:45.321 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.653+0000 7f4d70d15700 1 --2- 172.21.15.90:0/1414090747 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f4d5803d5c0 0x7f4d5803fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T11:25:45.321 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.753+0000 7f4d7377a700 1 -- 172.21.15.90:0/1414090747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f4d6c1971a0 con 0x7f4d6c10a020 2024-04-25T11:25:45.322 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.754+0000 7f4d627fc700 1 -- 172.21.15.90:0/1414090747 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f4d6800e260 con 0x7f4d6c10a020 2024-04-25T11:25:45.322 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.756+0000 7f4d7377a700 1 -- 172.21.15.90:0/1414090747 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f4d5803d5c0 msgr2=0x7f4d5803fad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:25:45.322 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.756+0000 7f4d7377a700 1 --2- 172.21.15.90:0/1414090747 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f4d5803d5c0 0x7f4d5803fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:45.322 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.756+0000 7f4d7377a700 1 -- 172.21.15.90:0/1414090747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c10a020 msgr2=0x7f4d6c105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:45.322 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.756+0000 7f4d7377a700 1 --2- 172.21.15.90:0/1414090747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c10a020 0x7f4d6c105a90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4d680090e0 tx=0x7f4d68011940 comp rx=0 tx=0).stop 2024-04-25T11:25:45.322 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.756+0000 7f4d7377a700 1 -- 172.21.15.90:0/1414090747 shutdown_connections 2024-04-25T11:25:45.322 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.756+0000 7f4d7377a700 1 --2- 172.21.15.90:0/1414090747 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f4d5803d5c0 0x7f4d5803fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:45.322 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.756+0000 7f4d7377a700 1 --2- 172.21.15.90:0/1414090747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c10a020 0x7f4d6c105a90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:45.322 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.756+0000 7f4d7377a700 1 -- 172.21.15.90:0/1414090747 >> 172.21.15.90:0/1414090747 conn(0x7f4d6c103880 msgr2=0x7f4d6c194630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:45.322 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.756+0000 7f4d7377a700 1 -- 172.21.15.90:0/1414090747 shutdown_connections 2024-04-25T11:25:45.322 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:44.756+0000 7f4d7377a700 1 -- 172.21.15.90:0/1414090747 wait complete. 2024-04-25T11:25:45.322 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for the mgr to restart... 2024-04-25T11:25:45.322 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mgr epoch 5... 2024-04-25T11:25:45.687 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:45 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1414090747' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T11:25:45.687 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:45 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:45.511+0000 7f4dcf581200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T11:25:45.687 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:45 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:45.693+0000 7f4dcf581200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T11:25:45.979 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:45 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:45.867+0000 7f4dcf581200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T11:25:46.442 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:46 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:46.155+0000 7f4dcf581200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T11:25:46.442 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:46 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:46.246+0000 7f4dcf581200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T11:25:46.442 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:46 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:46.448+0000 7f4dcf581200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T11:25:46.729 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:46 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:46.641+0000 7f4dcf581200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T11:25:47.479 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:47 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:47.240+0000 7f4dcf581200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T11:25:47.960 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:47 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:47.667+0000 7f4dcf581200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T11:25:47.960 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:47 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:47.764+0000 7f4dcf581200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T11:25:47.960 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:47 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:47.966+0000 7f4dcf581200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T11:25:48.734 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:48 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:48.489+0000 7f4dcf581200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T11:25:48.734 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:48 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:48.582+0000 7f4dcf581200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T11:25:49.050 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:48 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:48.826+0000 7f4dcf581200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T11:25:49.050 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:48 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:48.960+0000 7f4dcf581200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T11:25:49.050 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:49 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:49.057+0000 7f4dcf581200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T11:25:49.479 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:49 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:49.175+0000 7f4dcf581200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T11:25:49.979 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:49 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:49.502+0000 7f4dcf581200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T11:25:50.365 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:50 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:50.100+0000 7f4dcf581200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T11:25:50.365 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:50 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:25:50.319+0000 7f4dcf581200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T11:25:50.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:50 smithi090 ceph-mon[29575]: Active manager daemon x restarted 2024-04-25T11:25:50.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:50 smithi090 ceph-mon[29575]: Activating manager daemon x 2024-04-25T11:25:50.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:50 smithi090 ceph-mon[29575]: osdmap e2: 0 total, 0 up, 0 in 2024-04-25T11:25:50.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:50 smithi090 ceph-mon[29575]: mgrmap e6: x(active, starting, since 0.00446444s) 2024-04-25T11:25:50.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:50 smithi090 ceph-mon[29575]: from='mgr.14120 172.21.15.90:0/3545268890' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T11:25:50.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:50 smithi090 ceph-mon[29575]: from='mgr.14120 172.21.15.90:0/3545268890' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T11:25:50.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:50 smithi090 ceph-mon[29575]: from='mgr.14120 172.21.15.90:0/3545268890' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T11:25:50.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:50 smithi090 ceph-mon[29575]: from='mgr.14120 172.21.15.90:0/3545268890' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T11:25:50.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:50 smithi090 ceph-mon[29575]: from='mgr.14120 172.21.15.90:0/3545268890' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T11:25:50.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:50 smithi090 ceph-mon[29575]: Manager daemon x is now available 2024-04-25T11:25:50.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:50 smithi090 ceph-mon[29575]: from='mgr.14120 172.21.15.90:0/3545268890' entity='mgr.x' 2024-04-25T11:25:50.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:50 smithi090 ceph-mon[29575]: from='mgr.14120 172.21.15.90:0/3545268890' entity='mgr.x' 2024-04-25T11:25:50.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:50 smithi090 ceph-mon[29575]: from='mgr.14120 172.21.15.90:0/3545268890' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:25:50.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:50 smithi090 ceph-mon[29575]: from='mgr.14120 172.21.15.90:0/3545268890' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:25:50.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:50 smithi090 ceph-mon[29575]: from='mgr.14120 172.21.15.90:0/3545268890' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T11:25:51.617 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:51 smithi090 ceph-mon[29575]: Found migration_current of "None". Setting to last migration. 2024-04-25T11:25:51.617 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:51 smithi090 ceph-mon[29575]: from='mgr.14120 172.21.15.90:0/3545268890' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T11:25:51.617 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:51 smithi090 ceph-mon[29575]: from='mgr.14120 172.21.15.90:0/3545268890' entity='mgr.x' 2024-04-25T11:25:51.618 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:51 smithi090 ceph-mon[29575]: from='mgr.14120 172.21.15.90:0/3545268890' entity='mgr.x' 2024-04-25T11:25:51.618 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:51 smithi090 ceph-mon[29575]: from='mgr.14120 172.21.15.90:0/3545268890' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:25:51.618 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:51 smithi090 ceph-mon[29575]: mgrmap e7: x(active, since 1.00678s) 2024-04-25T11:25:51.887 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-04-25T11:25:51.887 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-25T11:25:51.887 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T11:25:51.887 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-04-25T11:25:51.887 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.731+0000 7f4d73e52700 1 Processor -- start 2024-04-25T11:25:51.887 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.732+0000 7f4d73e52700 1 -- start start 2024-04-25T11:25:51.887 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.732+0000 7f4d73e52700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c108ab0 0x7f4d6c108ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:51.887 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.732+0000 7f4d73e52700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4d6c1094b0 con 0x7f4d6c108ab0 2024-04-25T11:25:51.887 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.732+0000 7f4d71bee700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c108ab0 0x7f4d6c108ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:51.887 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.732+0000 7f4d71bee700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c108ab0 0x7f4d6c108ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59536/0 (socket says 172.21.15.90:59536) 2024-04-25T11:25:51.888 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.732+0000 7f4d71bee700 1 -- 172.21.15.90:0/1975145048 learned_addr learned my addr 172.21.15.90:0/1975145048 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:51.888 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.732+0000 7f4d71bee700 1 -- 172.21.15.90:0/1975145048 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d6c109be0 con 0x7f4d6c108ab0 2024-04-25T11:25:51.888 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.733+0000 7f4d71bee700 1 --2- 172.21.15.90:0/1975145048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c108ab0 0x7f4d6c108ec0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4d680098a0 tx=0x7f4d68009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cad41a0faaa8d785 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:51.888 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.733+0000 7f4d70bec700 1 -- 172.21.15.90:0/1975145048 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d68004160 con 0x7f4d6c108ab0 2024-04-25T11:25:51.888 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.733+0000 7f4d70bec700 1 -- 172.21.15.90:0/1975145048 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4d68004300 con 0x7f4d6c108ab0 2024-04-25T11:25:51.888 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.733+0000 7f4d70bec700 1 -- 172.21.15.90:0/1975145048 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d68011af0 con 0x7f4d6c108ab0 2024-04-25T11:25:51.888 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.734+0000 7f4d73e52700 1 -- 172.21.15.90:0/1975145048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c108ab0 msgr2=0x7f4d6c108ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:51.888 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.734+0000 7f4d73e52700 1 --2- 172.21.15.90:0/1975145048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c108ab0 0x7f4d6c108ec0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4d680098a0 tx=0x7f4d68009bb0 comp rx=0 tx=0).stop 2024-04-25T11:25:51.888 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.734+0000 7f4d73e52700 1 -- 172.21.15.90:0/1975145048 shutdown_connections 2024-04-25T11:25:51.888 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.734+0000 7f4d73e52700 1 --2- 172.21.15.90:0/1975145048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c108ab0 0x7f4d6c108ec0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:51.888 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.734+0000 7f4d73e52700 1 -- 172.21.15.90:0/1975145048 >> 172.21.15.90:0/1975145048 conn(0x7f4d6c1040f0 msgr2=0x7f4d6c1065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:51.888 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.734+0000 7f4d73e52700 1 -- 172.21.15.90:0/1975145048 shutdown_connections 2024-04-25T11:25:51.888 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.734+0000 7f4d73e52700 1 -- 172.21.15.90:0/1975145048 wait complete. 2024-04-25T11:25:51.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.734+0000 7f4d73e52700 1 Processor -- start 2024-04-25T11:25:51.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.735+0000 7f4d73e52700 1 -- start start 2024-04-25T11:25:51.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.735+0000 7f4d73e52700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c108ab0 0x7f4d6c07c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:51.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.735+0000 7f4d73e52700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4d6c10cf90 con 0x7f4d6c108ab0 2024-04-25T11:25:51.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.735+0000 7f4d71bee700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c108ab0 0x7f4d6c07c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:51.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.735+0000 7f4d71bee700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c108ab0 0x7f4d6c07c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59546/0 (socket says 172.21.15.90:59546) 2024-04-25T11:25:51.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.735+0000 7f4d71bee700 1 -- 172.21.15.90:0/1545711078 learned_addr learned my addr 172.21.15.90:0/1545711078 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:51.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.735+0000 7f4d71bee700 1 -- 172.21.15.90:0/1545711078 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d6c07cea0 con 0x7f4d6c108ab0 2024-04-25T11:25:51.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.735+0000 7f4d71bee700 1 --2- 172.21.15.90:0/1545711078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c108ab0 0x7f4d6c07c960 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4d680090e0 tx=0x7f4d68011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:51.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.735+0000 7f4d62ffd700 1 -- 172.21.15.90:0/1545711078 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d680039e0 con 0x7f4d6c108ab0 2024-04-25T11:25:51.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.736+0000 7f4d62ffd700 1 -- 172.21.15.90:0/1545711078 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4d68003b80 con 0x7f4d6c108ab0 2024-04-25T11:25:51.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.736+0000 7f4d62ffd700 1 -- 172.21.15.90:0/1545711078 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d68024690 con 0x7f4d6c108ab0 2024-04-25T11:25:51.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.736+0000 7f4d73e52700 1 -- 172.21.15.90:0/1545711078 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d6c1a0d60 con 0x7f4d6c108ab0 2024-04-25T11:25:51.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.736+0000 7f4d73e52700 1 -- 172.21.15.90:0/1545711078 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d6c1a11e0 con 0x7f4d6c108ab0 2024-04-25T11:25:51.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.737+0000 7f4d62ffd700 1 -- 172.21.15.90:0/1545711078 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 49516+0+0 (secure 0 0 0) 0x7f4d68022020 con 0x7f4d6c108ab0 2024-04-25T11:25:51.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.737+0000 7f4d62ffd700 1 --2- 172.21.15.90:0/1545711078 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f4d5803d5c0 0x7f4d5803fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:51.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.737+0000 7f4d62ffd700 1 -- 172.21.15.90:0/1545711078 --> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4d580401e0 con 0x7f4d5803d5c0 2024-04-25T11:25:51.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.737+0000 7f4d62ffd700 1 -- 172.21.15.90:0/1545711078 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4d680522c0 con 0x7f4d6c108ab0 2024-04-25T11:25:51.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.737+0000 7f4d713ed700 1 -- 172.21.15.90:0/1545711078 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f4d5803d5c0 msgr2=0x7f4d5803fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/3122821311 2024-04-25T11:25:51.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.737+0000 7f4d713ed700 1 --2- 172.21.15.90:0/1545711078 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f4d5803d5c0 0x7f4d5803fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T11:25:51.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.938+0000 7f4d713ed700 1 -- 172.21.15.90:0/1545711078 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f4d5803d5c0 msgr2=0x7f4d5803fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/3122821311 2024-04-25T11:25:51.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:45.938+0000 7f4d713ed700 1 --2- 172.21.15.90:0/1545711078 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f4d5803d5c0 0x7f4d5803fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T11:25:51.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:46.339+0000 7f4d713ed700 1 -- 172.21.15.90:0/1545711078 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f4d5803d5c0 msgr2=0x7f4d5803fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/3122821311 2024-04-25T11:25:51.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:46.339+0000 7f4d713ed700 1 --2- 172.21.15.90:0/1545711078 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f4d5803d5c0 0x7f4d5803fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T11:25:51.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:47.140+0000 7f4d713ed700 1 -- 172.21.15.90:0/1545711078 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f4d5803d5c0 msgr2=0x7f4d5803fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/3122821311 2024-04-25T11:25:51.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:47.140+0000 7f4d713ed700 1 --2- 172.21.15.90:0/1545711078 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f4d5803d5c0 0x7f4d5803fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T11:25:51.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:48.741+0000 7f4d713ed700 1 -- 172.21.15.90:0/1545711078 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f4d5803d5c0 msgr2=0x7f4d5803fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/3122821311 2024-04-25T11:25:51.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:48.741+0000 7f4d713ed700 1 --2- 172.21.15.90:0/1545711078 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f4d5803d5c0 0x7f4d5803fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T11:25:51.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:50.323+0000 7f4d62ffd700 1 -- 172.21.15.90:0/1545711078 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mgrmap(e 6) ==== 49283+0+0 (secure 0 0 0) 0x7f4d680248d0 con 0x7f4d6c108ab0 2024-04-25T11:25:51.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:50.323+0000 7f4d62ffd700 1 -- 172.21.15.90:0/1545711078 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f4d5803d5c0 msgr2=0x7f4d5803fad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:25:51.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:50.323+0000 7f4d62ffd700 1 --2- 172.21.15.90:0/1545711078 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f4d5803d5c0 0x7f4d5803fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:51.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:51.324+0000 7f4d62ffd700 1 -- 172.21.15.90:0/1545711078 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 7) ==== 49410+0+0 (secure 0 0 0) 0x7f4d680540a0 con 0x7f4d6c108ab0 2024-04-25T11:25:51.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:51.325+0000 7f4d62ffd700 1 --2- 172.21.15.90:0/1545711078 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f4d58040f00 0x7f4d58043320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:51.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:51.325+0000 7f4d62ffd700 1 -- 172.21.15.90:0/1545711078 --> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4d68040130 con 0x7f4d58040f00 2024-04-25T11:25:51.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:51.325+0000 7f4d713ed700 1 --2- 172.21.15.90:0/1545711078 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f4d58040f00 0x7f4d58043320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:51.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:51.326+0000 7f4d713ed700 1 --2- 172.21.15.90:0/1545711078 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f4d58040f00 0x7f4d58043320 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c003a10 tx=0x7f4d5c0092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:51.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:51.326+0000 7f4d62ffd700 1 -- 172.21.15.90:0/1545711078 <== mgr.14120 v2:172.21.15.90:6800/71478430 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f4d68040130 con 0x7f4d58040f00 2024-04-25T11:25:51.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:51.334+0000 7f4d73e52700 1 -- 172.21.15.90:0/1545711078 --> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f4d6c064cf0 con 0x7f4d58040f00 2024-04-25T11:25:51.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:51.335+0000 7f4d62ffd700 1 -- 172.21.15.90:0/1545711078 <== mgr.14120 v2:172.21.15.90:6800/71478430 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f4d6c064cf0 con 0x7f4d58040f00 2024-04-25T11:25:51.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:51.335+0000 7f4d73e52700 1 -- 172.21.15.90:0/1545711078 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f4d58040f00 msgr2=0x7f4d58043320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:51.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:51.335+0000 7f4d73e52700 1 --2- 172.21.15.90:0/1545711078 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f4d58040f00 0x7f4d58043320 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c003a10 tx=0x7f4d5c0092b0 comp rx=0 tx=0).stop 2024-04-25T11:25:51.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:51.335+0000 7f4d73e52700 1 -- 172.21.15.90:0/1545711078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c108ab0 msgr2=0x7f4d6c07c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:51.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:51.335+0000 7f4d73e52700 1 --2- 172.21.15.90:0/1545711078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c108ab0 0x7f4d6c07c960 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4d680090e0 tx=0x7f4d68011940 comp rx=0 tx=0).stop 2024-04-25T11:25:51.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:51.335+0000 7f4d73e52700 1 -- 172.21.15.90:0/1545711078 shutdown_connections 2024-04-25T11:25:51.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:51.335+0000 7f4d73e52700 1 --2- 172.21.15.90:0/1545711078 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f4d58040f00 0x7f4d58043320 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:51.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:51.335+0000 7f4d73e52700 1 --2- 172.21.15.90:0/1545711078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6c108ab0 0x7f4d6c07c960 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:51.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:51.335+0000 7f4d73e52700 1 --2- 172.21.15.90:0/1545711078 >> [v2:172.21.15.90:6800/3122821311,v1:172.21.15.90:6801/3122821311] conn(0x7f4d5803d5c0 0x7f4d5803fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:51.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:51.335+0000 7f4d73e52700 1 -- 172.21.15.90:0/1545711078 >> 172.21.15.90:0/1545711078 conn(0x7f4d6c1040f0 msgr2=0x7f4d6c104a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:51.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:51.335+0000 7f4d73e52700 1 -- 172.21.15.90:0/1545711078 shutdown_connections 2024-04-25T11:25:51.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:51.335+0000 7f4d73e52700 1 -- 172.21.15.90:0/1545711078 wait complete. 2024-04-25T11:25:51.892 INFO:teuthology.orchestra.run.smithi090.stdout:mgr epoch 5 is available 2024-04-25T11:25:51.892 INFO:teuthology.orchestra.run.smithi090.stdout:Setting orchestrator backend to cephadm... 2024-04-25T11:25:52.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:52 smithi090 ceph-mon[29575]: [25/Apr/2024:11:25:51] ENGINE Bus STARTING 2024-04-25T11:25:52.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:52 smithi090 ceph-mon[29575]: [25/Apr/2024:11:25:51] ENGINE Serving on http://172.21.15.90:8765 2024-04-25T11:25:52.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:52 smithi090 ceph-mon[29575]: [25/Apr/2024:11:25:51] ENGINE Serving on https://172.21.15.90:7150 2024-04-25T11:25:52.480 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:52 smithi090 ceph-mon[29575]: [25/Apr/2024:11:25:51] ENGINE Bus STARTED 2024-04-25T11:25:52.480 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:52 smithi090 ceph-mon[29575]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T11:25:52.480 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:52 smithi090 ceph-mon[29575]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T11:25:53.151 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.343+0000 7f57c7dc1700 1 Processor -- start 2024-04-25T11:25:53.152 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.343+0000 7f57c7dc1700 1 -- start start 2024-04-25T11:25:53.152 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.343+0000 7f57c7dc1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57c007cca0 0x7f57c010c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:53.152 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.343+0000 7f57c7dc1700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f57c010ccf0 con 0x7f57c007cca0 2024-04-25T11:25:53.152 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.343+0000 7f57c5b5d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57c007cca0 0x7f57c010c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:53.152 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.343+0000 7f57c5b5d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57c007cca0 0x7f57c010c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60850/0 (socket says 172.21.15.90:60850) 2024-04-25T11:25:53.152 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.343+0000 7f57c5b5d700 1 -- 172.21.15.90:0/1796592098 learned_addr learned my addr 172.21.15.90:0/1796592098 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:53.152 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.344+0000 7f57c5b5d700 1 -- 172.21.15.90:0/1796592098 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57c010ce70 con 0x7f57c007cca0 2024-04-25T11:25:53.152 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.344+0000 7f57c5b5d700 1 --2- 172.21.15.90:0/1796592098 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57c007cca0 0x7f57c010c700 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f57bc0098a0 tx=0x7f57bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a244fc8c352f581b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:53.152 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.344+0000 7f57c4b5b700 1 -- 172.21.15.90:0/1796592098 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f57bc004160 con 0x7f57c007cca0 2024-04-25T11:25:53.152 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.344+0000 7f57c4b5b700 1 -- 172.21.15.90:0/1796592098 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f57bc004300 con 0x7f57c007cca0 2024-04-25T11:25:53.152 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.344+0000 7f57c4b5b700 1 -- 172.21.15.90:0/1796592098 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f57bc011af0 con 0x7f57c007cca0 2024-04-25T11:25:53.153 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.345+0000 7f57c7dc1700 1 -- 172.21.15.90:0/1796592098 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57c007cca0 msgr2=0x7f57c010c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:53.153 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.345+0000 7f57c7dc1700 1 --2- 172.21.15.90:0/1796592098 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57c007cca0 0x7f57c010c700 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f57bc0098a0 tx=0x7f57bc009bb0 comp rx=0 tx=0).stop 2024-04-25T11:25:53.153 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.345+0000 7f57c7dc1700 1 -- 172.21.15.90:0/1796592098 shutdown_connections 2024-04-25T11:25:53.153 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.345+0000 7f57c7dc1700 1 --2- 172.21.15.90:0/1796592098 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57c007cca0 0x7f57c010c700 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:53.153 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.345+0000 7f57c7dc1700 1 -- 172.21.15.90:0/1796592098 >> 172.21.15.90:0/1796592098 conn(0x7f57c01040f0 msgr2=0x7f57c01065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:53.153 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.345+0000 7f57c7dc1700 1 -- 172.21.15.90:0/1796592098 shutdown_connections 2024-04-25T11:25:53.153 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.345+0000 7f57c7dc1700 1 -- 172.21.15.90:0/1796592098 wait complete. 2024-04-25T11:25:53.153 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.346+0000 7f57c7dc1700 1 Processor -- start 2024-04-25T11:25:53.153 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.346+0000 7f57c7dc1700 1 -- start start 2024-04-25T11:25:53.154 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.346+0000 7f57c7dc1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57c007cca0 0x7f57c0109e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:53.154 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.346+0000 7f57c7dc1700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f57c010def0 con 0x7f57c007cca0 2024-04-25T11:25:53.154 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.346+0000 7f57c5b5d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57c007cca0 0x7f57c0109e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:53.154 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.346+0000 7f57c5b5d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57c007cca0 0x7f57c0109e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60860/0 (socket says 172.21.15.90:60860) 2024-04-25T11:25:53.154 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.346+0000 7f57c5b5d700 1 -- 172.21.15.90:0/891369224 learned_addr learned my addr 172.21.15.90:0/891369224 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:53.154 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.346+0000 7f57c5b5d700 1 -- 172.21.15.90:0/891369224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57c010a3b0 con 0x7f57c007cca0 2024-04-25T11:25:53.154 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.346+0000 7f57c5b5d700 1 --2- 172.21.15.90:0/891369224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57c007cca0 0x7f57c0109e70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f57bc009fb0 tx=0x7f57bc011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:53.154 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.347+0000 7f57b6ffd700 1 -- 172.21.15.90:0/891369224 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f57bc011bc0 con 0x7f57c007cca0 2024-04-25T11:25:53.154 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.347+0000 7f57b6ffd700 1 -- 172.21.15.90:0/891369224 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f57bc024440 con 0x7f57c007cca0 2024-04-25T11:25:53.154 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.347+0000 7f57c7dc1700 1 -- 172.21.15.90:0/891369224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57c0108600 con 0x7f57c007cca0 2024-04-25T11:25:53.154 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.347+0000 7f57b6ffd700 1 -- 172.21.15.90:0/891369224 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f57bc01a5c0 con 0x7f57c007cca0 2024-04-25T11:25:53.154 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.347+0000 7f57c7dc1700 1 -- 172.21.15.90:0/891369224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57c0108ae0 con 0x7f57c007cca0 2024-04-25T11:25:53.154 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.348+0000 7f57c7dc1700 1 -- 172.21.15.90:0/891369224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57c0064cf0 con 0x7f57c007cca0 2024-04-25T11:25:53.155 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.348+0000 7f57b6ffd700 1 -- 172.21.15.90:0/891369224 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 7) ==== 49410+0+0 (secure 0 0 0) 0x7f57bc0245f0 con 0x7f57c007cca0 2024-04-25T11:25:53.155 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.348+0000 7f57b6ffd700 1 --2- 172.21.15.90:0/891369224 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f57ac03d410 0x7f57ac03f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:53.155 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.349+0000 7f57b6ffd700 1 -- 172.21.15.90:0/891369224 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f57bc052470 con 0x7f57c007cca0 2024-04-25T11:25:53.155 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.349+0000 7f57c535c700 1 --2- 172.21.15.90:0/891369224 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f57ac03d410 0x7f57ac03f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:53.155 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.349+0000 7f57c535c700 1 --2- 172.21.15.90:0/891369224 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f57ac03d410 0x7f57ac03f920 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f57b0006fd0 tx=0x7f57b0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:53.155 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.354+0000 7f57b6ffd700 1 -- 172.21.15.90:0/891369224 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f57bc0212a0 con 0x7f57c007cca0 2024-04-25T11:25:53.155 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.380+0000 7f57b6ffd700 1 -- 172.21.15.90:0/891369224 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f57bc0038b0 con 0x7f57c007cca0 2024-04-25T11:25:53.155 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.576+0000 7f57c7dc1700 1 -- 172.21.15.90:0/891369224 --> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f57c01092d0 con 0x7f57ac03d410 2024-04-25T11:25:53.155 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.582+0000 7f57b6ffd700 1 -- 172.21.15.90:0/891369224 <== mgr.14120 v2:172.21.15.90:6800/71478430 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f57c01092d0 con 0x7f57ac03d410 2024-04-25T11:25:53.155 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.584+0000 7f57c7dc1700 1 -- 172.21.15.90:0/891369224 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f57ac03d410 msgr2=0x7f57ac03f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:53.155 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.584+0000 7f57c7dc1700 1 --2- 172.21.15.90:0/891369224 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f57ac03d410 0x7f57ac03f920 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f57b0006fd0 tx=0x7f57b0006e40 comp rx=0 tx=0).stop 2024-04-25T11:25:53.155 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.584+0000 7f57c7dc1700 1 -- 172.21.15.90:0/891369224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57c007cca0 msgr2=0x7f57c0109e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:53.155 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.584+0000 7f57c7dc1700 1 --2- 172.21.15.90:0/891369224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57c007cca0 0x7f57c0109e70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f57bc009fb0 tx=0x7f57bc011890 comp rx=0 tx=0).stop 2024-04-25T11:25:53.156 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.584+0000 7f57c7dc1700 1 -- 172.21.15.90:0/891369224 shutdown_connections 2024-04-25T11:25:53.156 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.584+0000 7f57c7dc1700 1 --2- 172.21.15.90:0/891369224 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f57ac03d410 0x7f57ac03f920 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:53.156 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.584+0000 7f57c7dc1700 1 --2- 172.21.15.90:0/891369224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57c007cca0 0x7f57c0109e70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:53.156 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.584+0000 7f57c7dc1700 1 -- 172.21.15.90:0/891369224 >> 172.21.15.90:0/891369224 conn(0x7f57c01040f0 msgr2=0x7f57c0104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:53.156 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.584+0000 7f57c7dc1700 1 -- 172.21.15.90:0/891369224 shutdown_connections 2024-04-25T11:25:53.156 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:52.584+0000 7f57c7dc1700 1 -- 172.21.15.90:0/891369224 wait complete. 2024-04-25T11:25:53.465 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:53 smithi090 ceph-mon[29575]: mgrmap e8: x(active, since 2s) 2024-04-25T11:25:53.466 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:53 smithi090 ceph-mon[29575]: from='mgr.14120 172.21.15.90:0/3545268890' entity='mgr.x' 2024-04-25T11:25:53.466 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:53 smithi090 ceph-mon[29575]: from='mgr.14120 172.21.15.90:0/3545268890' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:25:54.344 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-25T11:25:54.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.584+0000 7fa2b4f57700 1 Processor -- start 2024-04-25T11:25:54.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.584+0000 7fa2b4f57700 1 -- start start 2024-04-25T11:25:54.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.584+0000 7fa2b4f57700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2b0108ab0 0x7fa2b0108ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:54.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.584+0000 7fa2b4f57700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b01094b0 con 0x7fa2b0108ab0 2024-04-25T11:25:54.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.584+0000 7fa2ae59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2b0108ab0 0x7fa2b0108ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:54.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.585+0000 7fa2ae59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2b0108ab0 0x7fa2b0108ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60874/0 (socket says 172.21.15.90:60874) 2024-04-25T11:25:54.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.585+0000 7fa2ae59c700 1 -- 172.21.15.90:0/3283266159 learned_addr learned my addr 172.21.15.90:0/3283266159 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:54.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.585+0000 7fa2ae59c700 1 -- 172.21.15.90:0/3283266159 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2b0109be0 con 0x7fa2b0108ab0 2024-04-25T11:25:54.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.585+0000 7fa2ae59c700 1 --2- 172.21.15.90:0/3283266159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2b0108ab0 0x7fa2b0108ec0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa2a00098a0 tx=0x7fa2a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=875eb29e16c8d212 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:54.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.585+0000 7fa2ad59a700 1 -- 172.21.15.90:0/3283266159 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa2a0004160 con 0x7fa2b0108ab0 2024-04-25T11:25:54.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.585+0000 7fa2ad59a700 1 -- 172.21.15.90:0/3283266159 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa2a0004300 con 0x7fa2b0108ab0 2024-04-25T11:25:54.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.585+0000 7fa2ad59a700 1 -- 172.21.15.90:0/3283266159 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa2a0011af0 con 0x7fa2b0108ab0 2024-04-25T11:25:54.345 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.586+0000 7fa2b4f57700 1 -- 172.21.15.90:0/3283266159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2b0108ab0 msgr2=0x7fa2b0108ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:54.346 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.586+0000 7fa2b4f57700 1 --2- 172.21.15.90:0/3283266159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2b0108ab0 0x7fa2b0108ec0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa2a00098a0 tx=0x7fa2a0009bb0 comp rx=0 tx=0).stop 2024-04-25T11:25:54.346 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.586+0000 7fa2b4f57700 1 -- 172.21.15.90:0/3283266159 shutdown_connections 2024-04-25T11:25:54.346 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.586+0000 7fa2b4f57700 1 --2- 172.21.15.90:0/3283266159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2b0108ab0 0x7fa2b0108ec0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:54.346 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.586+0000 7fa2b4f57700 1 -- 172.21.15.90:0/3283266159 >> 172.21.15.90:0/3283266159 conn(0x7fa2b01040f0 msgr2=0x7fa2b01065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:54.346 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.586+0000 7fa2b4f57700 1 -- 172.21.15.90:0/3283266159 shutdown_connections 2024-04-25T11:25:54.346 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.586+0000 7fa2b4f57700 1 -- 172.21.15.90:0/3283266159 wait complete. 2024-04-25T11:25:54.346 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.587+0000 7fa2b4f57700 1 Processor -- start 2024-04-25T11:25:54.346 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.587+0000 7fa2b4f57700 1 -- start start 2024-04-25T11:25:54.346 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.587+0000 7fa2b4f57700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2b0108ab0 0x7fa2b019ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:54.346 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.587+0000 7fa2b4f57700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b010cf90 con 0x7fa2b0108ab0 2024-04-25T11:25:54.346 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.587+0000 7fa2ae59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2b0108ab0 0x7fa2b019ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:54.346 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.587+0000 7fa2ae59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2b0108ab0 0x7fa2b019ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60884/0 (socket says 172.21.15.90:60884) 2024-04-25T11:25:54.346 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.587+0000 7fa2ae59c700 1 -- 172.21.15.90:0/1838203756 learned_addr learned my addr 172.21.15.90:0/1838203756 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:54.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.587+0000 7fa2ae59c700 1 -- 172.21.15.90:0/1838203756 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2b019d410 con 0x7fa2b0108ab0 2024-04-25T11:25:54.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.588+0000 7fa2ae59c700 1 --2- 172.21.15.90:0/1838203756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2b0108ab0 0x7fa2b019ced0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa2a00090e0 tx=0x7fa2a0011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:54.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.588+0000 7fa2a77fe700 1 -- 172.21.15.90:0/1838203756 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa2a00039e0 con 0x7fa2b0108ab0 2024-04-25T11:25:54.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.588+0000 7fa2a77fe700 1 -- 172.21.15.90:0/1838203756 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa2a0003b80 con 0x7fa2b0108ab0 2024-04-25T11:25:54.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.588+0000 7fa2a77fe700 1 -- 172.21.15.90:0/1838203756 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa2a0024690 con 0x7fa2b0108ab0 2024-04-25T11:25:54.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.588+0000 7fa2b4f57700 1 -- 172.21.15.90:0/1838203756 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2b019d6a0 con 0x7fa2b0108ab0 2024-04-25T11:25:54.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.588+0000 7fa2b4f57700 1 -- 172.21.15.90:0/1838203756 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2b01a02b0 con 0x7fa2b0108ab0 2024-04-25T11:25:54.347 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.589+0000 7fa2b4f57700 1 -- 172.21.15.90:0/1838203756 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2b0064cf0 con 0x7fa2b0108ab0 2024-04-25T11:25:54.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.589+0000 7fa2a77fe700 1 -- 172.21.15.90:0/1838203756 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7fa2a0022020 con 0x7fa2b0108ab0 2024-04-25T11:25:54.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.589+0000 7fa2a77fe700 1 --2- 172.21.15.90:0/1838203756 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7fa29c03d5c0 0x7fa29c03fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:54.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.589+0000 7fa2a77fe700 1 -- 172.21.15.90:0/1838203756 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa2a0052540 con 0x7fa2b0108ab0 2024-04-25T11:25:54.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.589+0000 7fa2add9b700 1 --2- 172.21.15.90:0/1838203756 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7fa29c03d5c0 0x7fa29c03fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:54.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.590+0000 7fa2add9b700 1 --2- 172.21.15.90:0/1838203756 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7fa29c03d5c0 0x7fa29c03fad0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa298006fd0 tx=0x7fa298006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:54.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.594+0000 7fa2a77fe700 1 -- 172.21.15.90:0/1838203756 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa2a00172b0 con 0x7fa2b0108ab0 2024-04-25T11:25:54.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.797+0000 7fa2b4f57700 1 -- 172.21.15.90:0/1838203756 --> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fa2b0105190 con 0x7fa29c03d5c0 2024-04-25T11:25:54.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.798+0000 7fa2a77fe700 1 -- 172.21.15.90:0/1838203756 <== mgr.14120 v2:172.21.15.90:6800/71478430 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fa2b0105190 con 0x7fa29c03d5c0 2024-04-25T11:25:54.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.799+0000 7fa2b4f57700 1 -- 172.21.15.90:0/1838203756 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7fa29c03d5c0 msgr2=0x7fa29c03fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:54.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.799+0000 7fa2b4f57700 1 --2- 172.21.15.90:0/1838203756 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7fa29c03d5c0 0x7fa29c03fad0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa298006fd0 tx=0x7fa298006e40 comp rx=0 tx=0).stop 2024-04-25T11:25:54.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.800+0000 7fa2b4f57700 1 -- 172.21.15.90:0/1838203756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2b0108ab0 msgr2=0x7fa2b019ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:54.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.800+0000 7fa2b4f57700 1 --2- 172.21.15.90:0/1838203756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2b0108ab0 0x7fa2b019ced0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa2a00090e0 tx=0x7fa2a0011940 comp rx=0 tx=0).stop 2024-04-25T11:25:54.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.800+0000 7fa2b4f57700 1 -- 172.21.15.90:0/1838203756 shutdown_connections 2024-04-25T11:25:54.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.800+0000 7fa2b4f57700 1 --2- 172.21.15.90:0/1838203756 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7fa29c03d5c0 0x7fa29c03fad0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:54.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.800+0000 7fa2b4f57700 1 --2- 172.21.15.90:0/1838203756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2b0108ab0 0x7fa2b019ced0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:54.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.800+0000 7fa2b4f57700 1 -- 172.21.15.90:0/1838203756 >> 172.21.15.90:0/1838203756 conn(0x7fa2b01040f0 msgr2=0x7fa2b0104a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:54.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.800+0000 7fa2b4f57700 1 -- 172.21.15.90:0/1838203756 shutdown_connections 2024-04-25T11:25:54.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:53.800+0000 7fa2b4f57700 1 -- 172.21.15.90:0/1838203756 wait complete. 2024-04-25T11:25:54.349 INFO:teuthology.orchestra.run.smithi090.stdout:Generating ssh key... 2024-04-25T11:25:54.657 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:54 smithi090 ceph-mon[29575]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:25:55.611 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:55 smithi090 ceph-mon[29575]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:25:55.611 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:55 smithi090 ceph-mon[29575]: from='mgr.14120 172.21.15.90:0/3545268890' entity='mgr.x' 2024-04-25T11:25:55.611 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:55 smithi090 ceph-mon[29575]: from='mgr.14120 172.21.15.90:0/3545268890' entity='mgr.x' 2024-04-25T11:25:55.611 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:55 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: Generating public/private rsa key pair. 2024-04-25T11:25:55.612 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:55 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: Your identification has been saved in /tmp/tmpgf9gi641/key. 2024-04-25T11:25:55.612 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:55 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: Your public key has been saved in /tmp/tmpgf9gi641/key.pub. 2024-04-25T11:25:55.612 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:55 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: The key fingerprint is: 2024-04-25T11:25:55.612 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:55 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: SHA256:BCOkbD0Tu2nYNRJVhnjM8i5KzteiRMBGgLpv6+i1CCs ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:25:55.612 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:55 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: The key's randomart image is: 2024-04-25T11:25:55.612 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:55 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: +---[RSA 3072]----+ 2024-04-25T11:25:55.612 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:55 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: |+. .==+oo | 2024-04-25T11:25:55.612 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:55 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: |+. oo==+ | 2024-04-25T11:25:55.612 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:55 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: |oo+ *+o . | 2024-04-25T11:25:55.612 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:55 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: |oo o B.o | 2024-04-25T11:25:55.612 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:55 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: | .o =. S | 2024-04-25T11:25:55.612 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:55 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: |..... . | 2024-04-25T11:25:55.612 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:55 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: |.=.o o | 2024-04-25T11:25:55.613 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:55 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: |E+O + . | 2024-04-25T11:25:55.613 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:55 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: |==+= . | 2024-04-25T11:25:55.613 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:25:55 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: +----[SHA256]-----+ 2024-04-25T11:25:55.903 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.780+0000 7fa7b347b700 1 Processor -- start 2024-04-25T11:25:55.903 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.781+0000 7fa7b347b700 1 -- start start 2024-04-25T11:25:55.903 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.781+0000 7fa7b347b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7ac108ab0 0x7fa7ac108ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:55.903 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.781+0000 7fa7b347b700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ac1094b0 con 0x7fa7ac108ab0 2024-04-25T11:25:55.903 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.781+0000 7fa7b1217700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7ac108ab0 0x7fa7ac108ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:55.903 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.781+0000 7fa7b1217700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7ac108ab0 0x7fa7ac108ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60886/0 (socket says 172.21.15.90:60886) 2024-04-25T11:25:55.904 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.781+0000 7fa7b1217700 1 -- 172.21.15.90:0/2813385564 learned_addr learned my addr 172.21.15.90:0/2813385564 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:55.904 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.782+0000 7fa7b1217700 1 -- 172.21.15.90:0/2813385564 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7ac109be0 con 0x7fa7ac108ab0 2024-04-25T11:25:55.904 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.782+0000 7fa7b1217700 1 --2- 172.21.15.90:0/2813385564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7ac108ab0 0x7fa7ac108ec0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa7a80098a0 tx=0x7fa7a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e8c5bec479e5181 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:55.904 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.782+0000 7fa7a3fff700 1 -- 172.21.15.90:0/2813385564 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7a8004160 con 0x7fa7ac108ab0 2024-04-25T11:25:55.904 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.782+0000 7fa7a3fff700 1 -- 172.21.15.90:0/2813385564 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa7a8004300 con 0x7fa7ac108ab0 2024-04-25T11:25:55.904 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.782+0000 7fa7a3fff700 1 -- 172.21.15.90:0/2813385564 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7a8011af0 con 0x7fa7ac108ab0 2024-04-25T11:25:55.904 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.783+0000 7fa7b347b700 1 -- 172.21.15.90:0/2813385564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7ac108ab0 msgr2=0x7fa7ac108ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:55.904 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.783+0000 7fa7b347b700 1 --2- 172.21.15.90:0/2813385564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7ac108ab0 0x7fa7ac108ec0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa7a80098a0 tx=0x7fa7a8009bb0 comp rx=0 tx=0).stop 2024-04-25T11:25:55.904 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.783+0000 7fa7b347b700 1 -- 172.21.15.90:0/2813385564 shutdown_connections 2024-04-25T11:25:55.904 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.783+0000 7fa7b347b700 1 --2- 172.21.15.90:0/2813385564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7ac108ab0 0x7fa7ac108ec0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:55.904 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.783+0000 7fa7b347b700 1 -- 172.21.15.90:0/2813385564 >> 172.21.15.90:0/2813385564 conn(0x7fa7ac1040f0 msgr2=0x7fa7ac1065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:55.904 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.783+0000 7fa7b347b700 1 -- 172.21.15.90:0/2813385564 shutdown_connections 2024-04-25T11:25:55.904 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.783+0000 7fa7b347b700 1 -- 172.21.15.90:0/2813385564 wait complete. 2024-04-25T11:25:55.904 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.783+0000 7fa7b347b700 1 Processor -- start 2024-04-25T11:25:55.904 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.783+0000 7fa7b347b700 1 -- start start 2024-04-25T11:25:55.905 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.784+0000 7fa7b347b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7ac108ab0 0x7fa7ac07d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:55.905 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.784+0000 7fa7b347b700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ac10cf90 con 0x7fa7ac108ab0 2024-04-25T11:25:55.905 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.784+0000 7fa7b1217700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7ac108ab0 0x7fa7ac07d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:55.905 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.784+0000 7fa7b1217700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7ac108ab0 0x7fa7ac07d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60888/0 (socket says 172.21.15.90:60888) 2024-04-25T11:25:55.905 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.784+0000 7fa7b1217700 1 -- 172.21.15.90:0/4223242990 learned_addr learned my addr 172.21.15.90:0/4223242990 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:55.905 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.784+0000 7fa7b1217700 1 -- 172.21.15.90:0/4223242990 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7ac07d710 con 0x7fa7ac108ab0 2024-04-25T11:25:55.905 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.784+0000 7fa7b1217700 1 --2- 172.21.15.90:0/4223242990 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7ac108ab0 0x7fa7ac07d1d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa7a80090e0 tx=0x7fa7a8011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:55.905 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.784+0000 7fa7a27fc700 1 -- 172.21.15.90:0/4223242990 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7a80039e0 con 0x7fa7ac108ab0 2024-04-25T11:25:55.905 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.784+0000 7fa7a27fc700 1 -- 172.21.15.90:0/4223242990 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa7a8003b80 con 0x7fa7ac108ab0 2024-04-25T11:25:55.905 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.784+0000 7fa7b347b700 1 -- 172.21.15.90:0/4223242990 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7ac1a0d80 con 0x7fa7ac108ab0 2024-04-25T11:25:55.905 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.784+0000 7fa7a27fc700 1 -- 172.21.15.90:0/4223242990 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7a8024690 con 0x7fa7ac108ab0 2024-04-25T11:25:55.905 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.784+0000 7fa7b347b700 1 -- 172.21.15.90:0/4223242990 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7ac1a1200 con 0x7fa7ac108ab0 2024-04-25T11:25:55.905 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.785+0000 7fa7a27fc700 1 -- 172.21.15.90:0/4223242990 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7fa7a8022020 con 0x7fa7ac108ab0 2024-04-25T11:25:55.905 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.785+0000 7fa7b347b700 1 -- 172.21.15.90:0/4223242990 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7ac064cf0 con 0x7fa7ac108ab0 2024-04-25T11:25:55.906 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.786+0000 7fa7a27fc700 1 --2- 172.21.15.90:0/4223242990 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7fa79803d5c0 0x7fa79803fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:55.906 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.786+0000 7fa7a27fc700 1 -- 172.21.15.90:0/4223242990 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa7a8052540 con 0x7fa7ac108ab0 2024-04-25T11:25:55.906 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.786+0000 7fa7b0a16700 1 --2- 172.21.15.90:0/4223242990 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7fa79803d5c0 0x7fa79803fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:55.906 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.786+0000 7fa7b0a16700 1 --2- 172.21.15.90:0/4223242990 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7fa79803d5c0 0x7fa79803fad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa79c006fd0 tx=0x7fa79c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:55.906 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.791+0000 7fa7a27fc700 1 -- 172.21.15.90:0/4223242990 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa7a80172b0 con 0x7fa7ac108ab0 2024-04-25T11:25:55.906 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:54.987+0000 7fa7b347b700 1 -- 172.21.15.90:0/4223242990 --> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fa7ac105190 con 0x7fa79803d5c0 2024-04-25T11:25:55.906 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:55.323+0000 7fa7a27fc700 1 -- 172.21.15.90:0/4223242990 <== mgr.14120 v2:172.21.15.90:6800/71478430 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa7ac105190 con 0x7fa79803d5c0 2024-04-25T11:25:55.906 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:55.325+0000 7fa7b347b700 1 -- 172.21.15.90:0/4223242990 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7fa79803d5c0 msgr2=0x7fa79803fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:55.906 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:55.325+0000 7fa7b347b700 1 --2- 172.21.15.90:0/4223242990 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7fa79803d5c0 0x7fa79803fad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa79c006fd0 tx=0x7fa79c006e40 comp rx=0 tx=0).stop 2024-04-25T11:25:55.906 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:55.325+0000 7fa7b347b700 1 -- 172.21.15.90:0/4223242990 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7ac108ab0 msgr2=0x7fa7ac07d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:55.906 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:55.325+0000 7fa7b347b700 1 --2- 172.21.15.90:0/4223242990 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7ac108ab0 0x7fa7ac07d1d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa7a80090e0 tx=0x7fa7a8011940 comp rx=0 tx=0).stop 2024-04-25T11:25:55.906 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:55.325+0000 7fa7b347b700 1 -- 172.21.15.90:0/4223242990 shutdown_connections 2024-04-25T11:25:55.906 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:55.325+0000 7fa7b347b700 1 --2- 172.21.15.90:0/4223242990 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7fa79803d5c0 0x7fa79803fad0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:55.907 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:55.325+0000 7fa7b347b700 1 --2- 172.21.15.90:0/4223242990 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7ac108ab0 0x7fa7ac07d1d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:55.907 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:55.325+0000 7fa7b347b700 1 -- 172.21.15.90:0/4223242990 >> 172.21.15.90:0/4223242990 conn(0x7fa7ac1040f0 msgr2=0x7fa7ac104a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:55.907 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:55.325+0000 7fa7b347b700 1 -- 172.21.15.90:0/4223242990 shutdown_connections 2024-04-25T11:25:55.907 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:55.325+0000 7fa7b347b700 1 -- 172.21.15.90:0/4223242990 wait complete. 2024-04-25T11:25:56.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:56 smithi090 ceph-mon[29575]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:25:56.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:56 smithi090 ceph-mon[29575]: Generating ssh key... 2024-04-25T11:25:57.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYV47YYzGnTOax+/+SgGFXLHEkYg2jRFr7y3yEYyUuaCNtVbo/FOEE46qPqip6RQvVXFyfx0ELTPx7sx7l63drWF/qj9Jct6uSOou2aXWwv0nVU+f3k6cBGeAqj71yV7pZjdQdh/dQrB1yAKJJ2ON+liNidCIe8Eg5yu+FnUMbSs533rUHuVxSZfk4H9EVqWg9InUu0OZNWScK13phuWYJ9o6KZcvXQ3P0AlHSUC6dE6EStF4RaKfeuRgRgoIj1Le6wjeTw4v6EhvGE/JvIOUUgNlJiYMkejnv0M6h+fOQKxKZB6dokRlAYHm+Duqz4Q+STnMz6ab50DCSaqWq+6A43xrhET7xfgyge32uyjt7Q7jDm4qNI6WN3iANEQwRhXj8AhD7BO2s1dKy01DzCLqDbgdvuD3B/Do68KgnM9Xj2ygycGrbzljDzZ0rW3Lz3Vz29OwbQ+ETmkTxydlsz/gfUDzzLd52jRb8Ef7UOw+YOXPnpMP1ln2KZN0YKqDXS9k= ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:25:57.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.330+0000 7fc84c645700 1 Processor -- start 2024-04-25T11:25:57.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.330+0000 7fc84c645700 1 -- start start 2024-04-25T11:25:57.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.330+0000 7fc84c645700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc844108ae0 0x7fc844108ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:57.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.330+0000 7fc84c645700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc8441094e0 con 0x7fc844108ae0 2024-04-25T11:25:57.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.330+0000 7fc84a3e1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc844108ae0 0x7fc844108ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:57.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.330+0000 7fc84a3e1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc844108ae0 0x7fc844108ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60892/0 (socket says 172.21.15.90:60892) 2024-04-25T11:25:57.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.330+0000 7fc84a3e1700 1 -- 172.21.15.90:0/559947617 learned_addr learned my addr 172.21.15.90:0/559947617 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:57.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.330+0000 7fc84a3e1700 1 -- 172.21.15.90:0/559947617 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc844109c10 con 0x7fc844108ae0 2024-04-25T11:25:57.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.331+0000 7fc84a3e1700 1 --2- 172.21.15.90:0/559947617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc844108ae0 0x7fc844108ef0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc8400098a0 tx=0x7fc840009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2adb6787a966f2ed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:57.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.331+0000 7fc8493df700 1 -- 172.21.15.90:0/559947617 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc840004160 con 0x7fc844108ae0 2024-04-25T11:25:57.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.331+0000 7fc8493df700 1 -- 172.21.15.90:0/559947617 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc840004300 con 0x7fc844108ae0 2024-04-25T11:25:57.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.331+0000 7fc8493df700 1 -- 172.21.15.90:0/559947617 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc840011af0 con 0x7fc844108ae0 2024-04-25T11:25:57.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.332+0000 7fc84c645700 1 -- 172.21.15.90:0/559947617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc844108ae0 msgr2=0x7fc844108ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:57.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.332+0000 7fc84c645700 1 --2- 172.21.15.90:0/559947617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc844108ae0 0x7fc844108ef0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc8400098a0 tx=0x7fc840009bb0 comp rx=0 tx=0).stop 2024-04-25T11:25:57.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.332+0000 7fc84c645700 1 -- 172.21.15.90:0/559947617 shutdown_connections 2024-04-25T11:25:57.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.332+0000 7fc84c645700 1 --2- 172.21.15.90:0/559947617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc844108ae0 0x7fc844108ef0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:57.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.332+0000 7fc84c645700 1 -- 172.21.15.90:0/559947617 >> 172.21.15.90:0/559947617 conn(0x7fc844104140 msgr2=0x7fc8441065d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:57.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.332+0000 7fc84c645700 1 -- 172.21.15.90:0/559947617 shutdown_connections 2024-04-25T11:25:57.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.332+0000 7fc84c645700 1 -- 172.21.15.90:0/559947617 wait complete. 2024-04-25T11:25:57.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.332+0000 7fc84c645700 1 Processor -- start 2024-04-25T11:25:57.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.332+0000 7fc84c645700 1 -- start start 2024-04-25T11:25:57.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.333+0000 7fc84c645700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc844108ae0 0x7fc84419cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:57.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.333+0000 7fc84c645700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc84410cfc0 con 0x7fc844108ae0 2024-04-25T11:25:57.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.333+0000 7fc84a3e1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc844108ae0 0x7fc84419cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:57.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.333+0000 7fc84a3e1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc844108ae0 0x7fc84419cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60904/0 (socket says 172.21.15.90:60904) 2024-04-25T11:25:57.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.333+0000 7fc84a3e1700 1 -- 172.21.15.90:0/2326490242 learned_addr learned my addr 172.21.15.90:0/2326490242 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:25:57.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.333+0000 7fc84a3e1700 1 -- 172.21.15.90:0/2326490242 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc84419d440 con 0x7fc844108ae0 2024-04-25T11:25:57.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.333+0000 7fc84a3e1700 1 --2- 172.21.15.90:0/2326490242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc844108ae0 0x7fc84419cf00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc840011670 tx=0x7fc840004280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:57.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.333+0000 7fc83b7fe700 1 -- 172.21.15.90:0/2326490242 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc8400036a0 con 0x7fc844108ae0 2024-04-25T11:25:57.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.333+0000 7fc83b7fe700 1 -- 172.21.15.90:0/2326490242 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc840003cd0 con 0x7fc844108ae0 2024-04-25T11:25:57.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.333+0000 7fc84c645700 1 -- 172.21.15.90:0/2326490242 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc84419d6d0 con 0x7fc844108ae0 2024-04-25T11:25:57.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.333+0000 7fc83b7fe700 1 -- 172.21.15.90:0/2326490242 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc8400193c0 con 0x7fc844108ae0 2024-04-25T11:25:57.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.333+0000 7fc84c645700 1 -- 172.21.15.90:0/2326490242 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8441a02e0 con 0x7fc844108ae0 2024-04-25T11:25:57.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.334+0000 7fc83b7fe700 1 -- 172.21.15.90:0/2326490242 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7fc840019680 con 0x7fc844108ae0 2024-04-25T11:25:57.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.334+0000 7fc84c645700 1 -- 172.21.15.90:0/2326490242 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc844064cf0 con 0x7fc844108ae0 2024-04-25T11:25:57.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.335+0000 7fc83b7fe700 1 --2- 172.21.15.90:0/2326490242 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7fc83003d520 0x7fc83003fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:57.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.335+0000 7fc83b7fe700 1 -- 172.21.15.90:0/2326490242 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc8400524d0 con 0x7fc844108ae0 2024-04-25T11:25:57.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.335+0000 7fc849be0700 1 --2- 172.21.15.90:0/2326490242 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7fc83003d520 0x7fc83003fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:57.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.335+0000 7fc849be0700 1 --2- 172.21.15.90:0/2326490242 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7fc83003d520 0x7fc83003fa30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc834006fd0 tx=0x7fc834006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:57.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.340+0000 7fc83b7fe700 1 -- 172.21.15.90:0/2326490242 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc840017e90 con 0x7fc844108ae0 2024-04-25T11:25:57.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.541+0000 7fc84c645700 1 -- 172.21.15.90:0/2326490242 --> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fc8441051c0 con 0x7fc83003d520 2024-04-25T11:25:57.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.542+0000 7fc83b7fe700 1 -- 172.21.15.90:0/2326490242 <== mgr.14120 v2:172.21.15.90:6800/71478430 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fc8441051c0 con 0x7fc83003d520 2024-04-25T11:25:57.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.544+0000 7fc84c645700 1 -- 172.21.15.90:0/2326490242 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7fc83003d520 msgr2=0x7fc83003fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:57.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.544+0000 7fc84c645700 1 --2- 172.21.15.90:0/2326490242 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7fc83003d520 0x7fc83003fa30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc834006fd0 tx=0x7fc834006e40 comp rx=0 tx=0).stop 2024-04-25T11:25:57.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.544+0000 7fc84c645700 1 -- 172.21.15.90:0/2326490242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc844108ae0 msgr2=0x7fc84419cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:57.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.544+0000 7fc84c645700 1 --2- 172.21.15.90:0/2326490242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc844108ae0 0x7fc84419cf00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc840011670 tx=0x7fc840004280 comp rx=0 tx=0).stop 2024-04-25T11:25:57.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.544+0000 7fc84c645700 1 -- 172.21.15.90:0/2326490242 shutdown_connections 2024-04-25T11:25:57.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.544+0000 7fc84c645700 1 --2- 172.21.15.90:0/2326490242 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7fc83003d520 0x7fc83003fa30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:57.142 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.544+0000 7fc84c645700 1 --2- 172.21.15.90:0/2326490242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc844108ae0 0x7fc84419cf00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:57.142 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.544+0000 7fc84c645700 1 -- 172.21.15.90:0/2326490242 >> 172.21.15.90:0/2326490242 conn(0x7fc844104140 msgr2=0x7fc844104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:57.142 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.544+0000 7fc84c645700 1 -- 172.21.15.90:0/2326490242 shutdown_connections 2024-04-25T11:25:57.142 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:56.544+0000 7fc84c645700 1 -- 172.21.15.90:0/2326490242 wait complete. 2024-04-25T11:25:57.142 INFO:teuthology.orchestra.run.smithi090.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-25T11:25:57.142 INFO:teuthology.orchestra.run.smithi090.stdout:Adding key to root@localhost authorized_keys... 2024-04-25T11:25:57.142 INFO:teuthology.orchestra.run.smithi090.stdout:Adding host smithi090... 2024-04-25T11:25:57.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:57 smithi090 ceph-mon[29575]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:25:58.630 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:58 smithi090 ceph-mon[29575]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi090", "addr": "172.21.15.90", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:25:58.630 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:25:58 smithi090 ceph-mon[29575]: Deploying cephadm binary to smithi090 2024-04-25T11:26:02.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout Added host 'smithi090' with addr '172.21.15.90' 2024-04-25T11:26:02.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.555+0000 7f1cef766700 1 Processor -- start 2024-04-25T11:26:02.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.556+0000 7f1cef766700 1 -- start start 2024-04-25T11:26:02.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.556+0000 7f1cef766700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ce8109bf0 0x7f1ce810a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:02.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.556+0000 7f1cef766700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce810a5f0 con 0x7f1ce8109bf0 2024-04-25T11:26:02.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.556+0000 7f1ced502700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ce8109bf0 0x7f1ce810a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:02.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.556+0000 7f1ced502700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ce8109bf0 0x7f1ce810a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60908/0 (socket says 172.21.15.90:60908) 2024-04-25T11:26:02.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.556+0000 7f1ced502700 1 -- 172.21.15.90:0/1855094148 learned_addr learned my addr 172.21.15.90:0/1855094148 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:02.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.556+0000 7f1ced502700 1 -- 172.21.15.90:0/1855094148 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ce810ad20 con 0x7f1ce8109bf0 2024-04-25T11:26:02.350 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.556+0000 7f1ced502700 1 --2- 172.21.15.90:0/1855094148 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ce8109bf0 0x7f1ce810a000 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1ce40098a0 tx=0x7f1ce4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bbbb057cf7ec202 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:02.350 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.556+0000 7f1cdffff700 1 -- 172.21.15.90:0/1855094148 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ce4004160 con 0x7f1ce8109bf0 2024-04-25T11:26:02.350 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.557+0000 7f1cdffff700 1 -- 172.21.15.90:0/1855094148 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1ce4004300 con 0x7f1ce8109bf0 2024-04-25T11:26:02.350 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.557+0000 7f1cdffff700 1 -- 172.21.15.90:0/1855094148 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ce4011af0 con 0x7f1ce8109bf0 2024-04-25T11:26:02.350 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.557+0000 7f1cef766700 1 -- 172.21.15.90:0/1855094148 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ce8109bf0 msgr2=0x7f1ce810a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:02.350 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.557+0000 7f1cef766700 1 --2- 172.21.15.90:0/1855094148 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ce8109bf0 0x7f1ce810a000 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1ce40098a0 tx=0x7f1ce4009bb0 comp rx=0 tx=0).stop 2024-04-25T11:26:02.350 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.557+0000 7f1cef766700 1 -- 172.21.15.90:0/1855094148 shutdown_connections 2024-04-25T11:26:02.350 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.557+0000 7f1cef766700 1 --2- 172.21.15.90:0/1855094148 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ce8109bf0 0x7f1ce810a000 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:02.350 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.557+0000 7f1cef766700 1 -- 172.21.15.90:0/1855094148 >> 172.21.15.90:0/1855094148 conn(0x7f1ce8105250 msgr2=0x7f1ce81076e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:02.350 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.557+0000 7f1cef766700 1 -- 172.21.15.90:0/1855094148 shutdown_connections 2024-04-25T11:26:02.350 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.557+0000 7f1cef766700 1 -- 172.21.15.90:0/1855094148 wait complete. 2024-04-25T11:26:02.350 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.558+0000 7f1cef766700 1 Processor -- start 2024-04-25T11:26:02.350 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.558+0000 7f1cef766700 1 -- start start 2024-04-25T11:26:02.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.558+0000 7f1cef766700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ce8109bf0 0x7f1ce807f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:02.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.558+0000 7f1cef766700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce810e0d0 con 0x7f1ce8109bf0 2024-04-25T11:26:02.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.558+0000 7f1ced502700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ce8109bf0 0x7f1ce807f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:02.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.558+0000 7f1ced502700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ce8109bf0 0x7f1ce807f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60924/0 (socket says 172.21.15.90:60924) 2024-04-25T11:26:02.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.558+0000 7f1ced502700 1 -- 172.21.15.90:0/2437060415 learned_addr learned my addr 172.21.15.90:0/2437060415 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:02.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.559+0000 7f1ced502700 1 -- 172.21.15.90:0/2437060415 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ce807f830 con 0x7f1ce8109bf0 2024-04-25T11:26:02.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.559+0000 7f1ced502700 1 --2- 172.21.15.90:0/2437060415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ce8109bf0 0x7f1ce807f2f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1ce4011670 tx=0x7f1ce4004280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:02.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.559+0000 7f1cde7fc700 1 -- 172.21.15.90:0/2437060415 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ce40036a0 con 0x7f1ce8109bf0 2024-04-25T11:26:02.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.559+0000 7f1cde7fc700 1 -- 172.21.15.90:0/2437060415 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1ce4003cd0 con 0x7f1ce8109bf0 2024-04-25T11:26:02.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.559+0000 7f1cef766700 1 -- 172.21.15.90:0/2437060415 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ce807da80 con 0x7f1ce8109bf0 2024-04-25T11:26:02.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.559+0000 7f1cde7fc700 1 -- 172.21.15.90:0/2437060415 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ce40193c0 con 0x7f1ce8109bf0 2024-04-25T11:26:02.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.559+0000 7f1cef766700 1 -- 172.21.15.90:0/2437060415 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ce807df60 con 0x7f1ce8109bf0 2024-04-25T11:26:02.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.560+0000 7f1cef766700 1 -- 172.21.15.90:0/2437060415 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ce8064cf0 con 0x7f1ce8109bf0 2024-04-25T11:26:02.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.560+0000 7f1cde7fc700 1 -- 172.21.15.90:0/2437060415 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f1ce4019630 con 0x7f1ce8109bf0 2024-04-25T11:26:02.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.560+0000 7f1cde7fc700 1 --2- 172.21.15.90:0/2437060415 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f1cd403d570 0x7f1cd403fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:02.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.560+0000 7f1cde7fc700 1 -- 172.21.15.90:0/2437060415 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1ce40524d0 con 0x7f1ce8109bf0 2024-04-25T11:26:02.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.560+0000 7f1cecd01700 1 --2- 172.21.15.90:0/2437060415 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f1cd403d570 0x7f1cd403fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:02.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.561+0000 7f1cecd01700 1 --2- 172.21.15.90:0/2437060415 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f1cd403d570 0x7f1cd403fa80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8006fd0 tx=0x7f1cd8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:02.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.565+0000 7f1cde7fc700 1 -- 172.21.15.90:0/2437060415 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1ce4017e90 con 0x7f1ce8109bf0 2024-04-25T11:26:02.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:25:57.768+0000 7f1cef766700 1 -- 172.21.15.90:0/2437060415 --> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi090", "addr": "172.21.15.90", "target": ["mon-mgr", ""]}) -- 0x7f1ce807e710 con 0x7f1cd403d570 2024-04-25T11:26:02.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:01.811+0000 7f1cde7fc700 1 -- 172.21.15.90:0/2437060415 <== mgr.14120 v2:172.21.15.90:6800/71478430 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f1ce807e710 con 0x7f1cd403d570 2024-04-25T11:26:02.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:01.813+0000 7f1cef766700 1 -- 172.21.15.90:0/2437060415 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f1cd403d570 msgr2=0x7f1cd403fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:02.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:01.813+0000 7f1cef766700 1 --2- 172.21.15.90:0/2437060415 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f1cd403d570 0x7f1cd403fa80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8006fd0 tx=0x7f1cd8006e40 comp rx=0 tx=0).stop 2024-04-25T11:26:02.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:01.813+0000 7f1cef766700 1 -- 172.21.15.90:0/2437060415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ce8109bf0 msgr2=0x7f1ce807f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:02.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:01.813+0000 7f1cef766700 1 --2- 172.21.15.90:0/2437060415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ce8109bf0 0x7f1ce807f2f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1ce4011670 tx=0x7f1ce4004280 comp rx=0 tx=0).stop 2024-04-25T11:26:02.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:01.814+0000 7f1cef766700 1 -- 172.21.15.90:0/2437060415 shutdown_connections 2024-04-25T11:26:02.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:01.814+0000 7f1cef766700 1 --2- 172.21.15.90:0/2437060415 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f1cd403d570 0x7f1cd403fa80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:02.353 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:01.814+0000 7f1cef766700 1 --2- 172.21.15.90:0/2437060415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ce8109bf0 0x7f1ce807f2f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:02.353 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:01.814+0000 7f1cef766700 1 -- 172.21.15.90:0/2437060415 >> 172.21.15.90:0/2437060415 conn(0x7f1ce8105250 msgr2=0x7f1ce8105b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:02.353 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:01.814+0000 7f1cef766700 1 -- 172.21.15.90:0/2437060415 shutdown_connections 2024-04-25T11:26:02.353 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:01.814+0000 7f1cef766700 1 -- 172.21.15.90:0/2437060415 wait complete. 2024-04-25T11:26:02.353 INFO:teuthology.orchestra.run.smithi090.stdout:Deploying unmanaged mon service... 2024-04-25T11:26:02.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:02 smithi090 ceph-mon[29575]: from='mgr.14120 172.21.15.90:0/3545268890' entity='mgr.x' 2024-04-25T11:26:02.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:02 smithi090 ceph-mon[29575]: Added host smithi090 2024-04-25T11:26:02.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:02 smithi090 ceph-mon[29575]: from='mgr.14120 172.21.15.90:0/3545268890' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:26:03.949 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-25T11:26:03.949 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.865+0000 7f46d272e700 1 Processor -- start 2024-04-25T11:26:03.949 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.865+0000 7f46d272e700 1 -- start start 2024-04-25T11:26:03.949 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.865+0000 7f46d272e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46cc07d550 0x7f46cc10c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:03.949 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.865+0000 7f46d272e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc10ccf0 con 0x7f46cc07d550 2024-04-25T11:26:03.949 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.865+0000 7f46cbfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46cc07d550 0x7f46cc10c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:03.949 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.865+0000 7f46cbfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46cc07d550 0x7f46cc10c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46624/0 (socket says 172.21.15.90:46624) 2024-04-25T11:26:03.949 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.865+0000 7f46cbfff700 1 -- 172.21.15.90:0/2266008640 learned_addr learned my addr 172.21.15.90:0/2266008640 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:03.949 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.866+0000 7f46cbfff700 1 -- 172.21.15.90:0/2266008640 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46cc10ce70 con 0x7f46cc07d550 2024-04-25T11:26:03.949 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.866+0000 7f46cbfff700 1 --2- 172.21.15.90:0/2266008640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46cc07d550 0x7f46cc10c6e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0098a0 tx=0x7f46bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f48d9725256e187 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:03.949 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.866+0000 7f46caffd700 1 -- 172.21.15.90:0/2266008640 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f46bc004160 con 0x7f46cc07d550 2024-04-25T11:26:03.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.866+0000 7f46caffd700 1 -- 172.21.15.90:0/2266008640 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f46bc004300 con 0x7f46cc07d550 2024-04-25T11:26:03.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.866+0000 7f46caffd700 1 -- 172.21.15.90:0/2266008640 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f46bc011af0 con 0x7f46cc07d550 2024-04-25T11:26:03.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.867+0000 7f46d272e700 1 -- 172.21.15.90:0/2266008640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46cc07d550 msgr2=0x7f46cc10c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:03.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.867+0000 7f46d272e700 1 --2- 172.21.15.90:0/2266008640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46cc07d550 0x7f46cc10c6e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0098a0 tx=0x7f46bc009bb0 comp rx=0 tx=0).stop 2024-04-25T11:26:03.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.867+0000 7f46d272e700 1 -- 172.21.15.90:0/2266008640 shutdown_connections 2024-04-25T11:26:03.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.867+0000 7f46d272e700 1 --2- 172.21.15.90:0/2266008640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46cc07d550 0x7f46cc10c6e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:03.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.867+0000 7f46d272e700 1 -- 172.21.15.90:0/2266008640 >> 172.21.15.90:0/2266008640 conn(0x7f46cc1040f0 msgr2=0x7f46cc1065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:03.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.867+0000 7f46d272e700 1 -- 172.21.15.90:0/2266008640 shutdown_connections 2024-04-25T11:26:03.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.867+0000 7f46d272e700 1 -- 172.21.15.90:0/2266008640 wait complete. 2024-04-25T11:26:03.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.867+0000 7f46d272e700 1 Processor -- start 2024-04-25T11:26:03.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.867+0000 7f46d272e700 1 -- start start 2024-04-25T11:26:03.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.868+0000 7f46d272e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46cc07d550 0x7f46cc19df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:03.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.868+0000 7f46d272e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc10def0 con 0x7f46cc07d550 2024-04-25T11:26:03.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.868+0000 7f46cbfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46cc07d550 0x7f46cc19df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:03.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.868+0000 7f46cbfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46cc07d550 0x7f46cc19df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46630/0 (socket says 172.21.15.90:46630) 2024-04-25T11:26:03.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.868+0000 7f46cbfff700 1 -- 172.21.15.90:0/2078497541 learned_addr learned my addr 172.21.15.90:0/2078497541 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:03.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.868+0000 7f46cbfff700 1 -- 172.21.15.90:0/2078497541 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46cc19e490 con 0x7f46cc07d550 2024-04-25T11:26:03.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.868+0000 7f46cbfff700 1 --2- 172.21.15.90:0/2078497541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46cc07d550 0x7f46cc19df50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0117b0 tx=0x7f46bc011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:03.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.868+0000 7f46c97fa700 1 -- 172.21.15.90:0/2078497541 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f46bc0039c0 con 0x7f46cc07d550 2024-04-25T11:26:03.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.868+0000 7f46c97fa700 1 -- 172.21.15.90:0/2078497541 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f46bc003b60 con 0x7f46cc07d550 2024-04-25T11:26:03.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.868+0000 7f46d272e700 1 -- 172.21.15.90:0/2078497541 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46cc19e720 con 0x7f46cc07d550 2024-04-25T11:26:03.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.868+0000 7f46c97fa700 1 -- 172.21.15.90:0/2078497541 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f46bc0196a0 con 0x7f46cc07d550 2024-04-25T11:26:03.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.868+0000 7f46d272e700 1 -- 172.21.15.90:0/2078497541 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46cc1a1330 con 0x7f46cc07d550 2024-04-25T11:26:03.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.869+0000 7f46d272e700 1 -- 172.21.15.90:0/2078497541 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46cc064cf0 con 0x7f46cc07d550 2024-04-25T11:26:03.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.869+0000 7f46c97fa700 1 -- 172.21.15.90:0/2078497541 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f46bc019960 con 0x7f46cc07d550 2024-04-25T11:26:03.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.870+0000 7f46c97fa700 1 --2- 172.21.15.90:0/2078497541 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f46b403d570 0x7f46b403fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:03.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.870+0000 7f46c97fa700 1 -- 172.21.15.90:0/2078497541 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f46bc052770 con 0x7f46cc07d550 2024-04-25T11:26:03.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.870+0000 7f46cb7fe700 1 --2- 172.21.15.90:0/2078497541 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f46b403d570 0x7f46b403fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:03.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.870+0000 7f46cb7fe700 1 --2- 172.21.15.90:0/2078497541 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f46b403d570 0x7f46b403fa80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f46c0006fd0 tx=0x7f46c0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:03.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:02.875+0000 7f46c97fa700 1 -- 172.21.15.90:0/2078497541 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f46bc00b3e0 con 0x7f46cc07d550 2024-04-25T11:26:03.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:03.099+0000 7f46d272e700 1 -- 172.21.15.90:0/2078497541 --> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f46cc1051d0 con 0x7f46b403d570 2024-04-25T11:26:03.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:03.107+0000 7f46c97fa700 1 -- 172.21.15.90:0/2078497541 <== mgr.14120 v2:172.21.15.90:6800/71478430 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f46cc1051d0 con 0x7f46b403d570 2024-04-25T11:26:03.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:03.109+0000 7f46d272e700 1 -- 172.21.15.90:0/2078497541 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f46b403d570 msgr2=0x7f46b403fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:03.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:03.109+0000 7f46d272e700 1 --2- 172.21.15.90:0/2078497541 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f46b403d570 0x7f46b403fa80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f46c0006fd0 tx=0x7f46c0006e40 comp rx=0 tx=0).stop 2024-04-25T11:26:03.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:03.109+0000 7f46d272e700 1 -- 172.21.15.90:0/2078497541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46cc07d550 msgr2=0x7f46cc19df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:03.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:03.109+0000 7f46d272e700 1 --2- 172.21.15.90:0/2078497541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46cc07d550 0x7f46cc19df50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0117b0 tx=0x7f46bc011b70 comp rx=0 tx=0).stop 2024-04-25T11:26:03.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:03.109+0000 7f46d272e700 1 -- 172.21.15.90:0/2078497541 shutdown_connections 2024-04-25T11:26:03.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:03.109+0000 7f46d272e700 1 --2- 172.21.15.90:0/2078497541 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f46b403d570 0x7f46b403fa80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:03.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:03.109+0000 7f46d272e700 1 --2- 172.21.15.90:0/2078497541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46cc07d550 0x7f46cc19df50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:03.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:03.110+0000 7f46d272e700 1 -- 172.21.15.90:0/2078497541 >> 172.21.15.90:0/2078497541 conn(0x7f46cc1040f0 msgr2=0x7f46cc104a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:03.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:03.110+0000 7f46d272e700 1 -- 172.21.15.90:0/2078497541 shutdown_connections 2024-04-25T11:26:03.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:03.110+0000 7f46d272e700 1 -- 172.21.15.90:0/2078497541 wait complete. 2024-04-25T11:26:03.953 INFO:teuthology.orchestra.run.smithi090.stdout:Deploying unmanaged mgr service... 2024-04-25T11:26:04.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:04 smithi090 ceph-mon[29575]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:26:04.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:04 smithi090 ceph-mon[29575]: Saving service mon spec with placement count:5 2024-04-25T11:26:04.230 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:04 smithi090 ceph-mon[29575]: from='mgr.14120 172.21.15.90:0/3545268890' entity='mgr.x' 2024-04-25T11:26:05.528 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-25T11:26:05.528 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.574+0000 7f5b19471700 1 Processor -- start 2024-04-25T11:26:05.528 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.574+0000 7f5b19471700 1 -- start start 2024-04-25T11:26:05.528 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.574+0000 7f5b19471700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b14108ad0 0x7f5b14108ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:05.528 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.574+0000 7f5b19471700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5b141094d0 con 0x7f5b14108ad0 2024-04-25T11:26:05.528 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.575+0000 7f5b12ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b14108ad0 0x7f5b14108ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:05.529 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.575+0000 7f5b12ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b14108ad0 0x7f5b14108ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46640/0 (socket says 172.21.15.90:46640) 2024-04-25T11:26:05.529 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.575+0000 7f5b12ffd700 1 -- 172.21.15.90:0/3567576635 learned_addr learned my addr 172.21.15.90:0/3567576635 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:05.529 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.575+0000 7f5b12ffd700 1 -- 172.21.15.90:0/3567576635 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b14109c00 con 0x7f5b14108ad0 2024-04-25T11:26:05.529 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.575+0000 7f5b12ffd700 1 --2- 172.21.15.90:0/3567576635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b14108ad0 0x7f5b14108ee0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5b040098a0 tx=0x7f5b04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368e286f83c979fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:05.529 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.575+0000 7f5b127fc700 1 -- 172.21.15.90:0/3567576635 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b04004160 con 0x7f5b14108ad0 2024-04-25T11:26:05.529 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.575+0000 7f5b127fc700 1 -- 172.21.15.90:0/3567576635 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5b04004300 con 0x7f5b14108ad0 2024-04-25T11:26:05.529 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.575+0000 7f5b127fc700 1 -- 172.21.15.90:0/3567576635 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b04011af0 con 0x7f5b14108ad0 2024-04-25T11:26:05.529 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.576+0000 7f5b19471700 1 -- 172.21.15.90:0/3567576635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b14108ad0 msgr2=0x7f5b14108ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:05.529 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.576+0000 7f5b19471700 1 --2- 172.21.15.90:0/3567576635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b14108ad0 0x7f5b14108ee0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5b040098a0 tx=0x7f5b04009bb0 comp rx=0 tx=0).stop 2024-04-25T11:26:05.529 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.576+0000 7f5b19471700 1 -- 172.21.15.90:0/3567576635 shutdown_connections 2024-04-25T11:26:05.529 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.576+0000 7f5b19471700 1 --2- 172.21.15.90:0/3567576635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b14108ad0 0x7f5b14108ee0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:05.529 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.576+0000 7f5b19471700 1 -- 172.21.15.90:0/3567576635 >> 172.21.15.90:0/3567576635 conn(0x7f5b14104110 msgr2=0x7f5b141065c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:05.529 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.576+0000 7f5b19471700 1 -- 172.21.15.90:0/3567576635 shutdown_connections 2024-04-25T11:26:05.529 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.576+0000 7f5b19471700 1 -- 172.21.15.90:0/3567576635 wait complete. 2024-04-25T11:26:05.530 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.577+0000 7f5b19471700 1 Processor -- start 2024-04-25T11:26:05.530 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.577+0000 7f5b19471700 1 -- start start 2024-04-25T11:26:05.530 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.577+0000 7f5b19471700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b14108ad0 0x7f5b1419dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:05.530 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.577+0000 7f5b19471700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1410cfb0 con 0x7f5b14108ad0 2024-04-25T11:26:05.530 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.577+0000 7f5b12ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b14108ad0 0x7f5b1419dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:05.530 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.577+0000 7f5b12ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b14108ad0 0x7f5b1419dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46642/0 (socket says 172.21.15.90:46642) 2024-04-25T11:26:05.530 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.577+0000 7f5b12ffd700 1 -- 172.21.15.90:0/1026135320 learned_addr learned my addr 172.21.15.90:0/1026135320 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:05.530 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.578+0000 7f5b12ffd700 1 -- 172.21.15.90:0/1026135320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b1419e530 con 0x7f5b14108ad0 2024-04-25T11:26:05.530 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.578+0000 7f5b12ffd700 1 --2- 172.21.15.90:0/1026135320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b14108ad0 0x7f5b1419dff0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5b04011670 tx=0x7f5b04004280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:05.530 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.578+0000 7f5b10ff9700 1 -- 172.21.15.90:0/1026135320 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b040036a0 con 0x7f5b14108ad0 2024-04-25T11:26:05.530 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.578+0000 7f5b10ff9700 1 -- 172.21.15.90:0/1026135320 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5b04003cd0 con 0x7f5b14108ad0 2024-04-25T11:26:05.530 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.578+0000 7f5b10ff9700 1 -- 172.21.15.90:0/1026135320 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b040193c0 con 0x7f5b14108ad0 2024-04-25T11:26:05.530 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.578+0000 7f5b19471700 1 -- 172.21.15.90:0/1026135320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b1419e7c0 con 0x7f5b14108ad0 2024-04-25T11:26:05.531 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.578+0000 7f5b19471700 1 -- 172.21.15.90:0/1026135320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b141a13d0 con 0x7f5b14108ad0 2024-04-25T11:26:05.531 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.579+0000 7f5b10ff9700 1 -- 172.21.15.90:0/1026135320 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f5b04019680 con 0x7f5b14108ad0 2024-04-25T11:26:05.531 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.580+0000 7f5b10ff9700 1 --2- 172.21.15.90:0/1026135320 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f5b0003d520 0x7f5b0003fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:05.531 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.580+0000 7f5b10ff9700 1 -- 172.21.15.90:0/1026135320 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5b04052510 con 0x7f5b14108ad0 2024-04-25T11:26:05.531 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.580+0000 7f5b19471700 1 -- 172.21.15.90:0/1026135320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b14064cf0 con 0x7f5b14108ad0 2024-04-25T11:26:05.531 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.580+0000 7f5b0a7fc700 1 --2- 172.21.15.90:0/1026135320 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f5b0003d520 0x7f5b0003fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:05.531 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.580+0000 7f5b0a7fc700 1 --2- 172.21.15.90:0/1026135320 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f5b0003d520 0x7f5b0003fa30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5afc006fd0 tx=0x7f5afc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:05.531 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.585+0000 7f5b10ff9700 1 -- 172.21.15.90:0/1026135320 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5b04017e90 con 0x7f5b14108ad0 2024-04-25T11:26:05.531 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.798+0000 7f5b19471700 1 -- 172.21.15.90:0/1026135320 --> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f5b1402e790 con 0x7f5b0003d520 2024-04-25T11:26:05.531 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.803+0000 7f5b10ff9700 1 -- 172.21.15.90:0/1026135320 <== mgr.14120 v2:172.21.15.90:6800/71478430 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f5b1402e790 con 0x7f5b0003d520 2024-04-25T11:26:05.531 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.805+0000 7f5b19471700 1 -- 172.21.15.90:0/1026135320 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f5b0003d520 msgr2=0x7f5b0003fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:05.531 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.805+0000 7f5b19471700 1 --2- 172.21.15.90:0/1026135320 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f5b0003d520 0x7f5b0003fa30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5afc006fd0 tx=0x7f5afc006e40 comp rx=0 tx=0).stop 2024-04-25T11:26:05.531 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.805+0000 7f5b19471700 1 -- 172.21.15.90:0/1026135320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b14108ad0 msgr2=0x7f5b1419dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:05.531 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.805+0000 7f5b19471700 1 --2- 172.21.15.90:0/1026135320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b14108ad0 0x7f5b1419dff0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5b04011670 tx=0x7f5b04004280 comp rx=0 tx=0).stop 2024-04-25T11:26:05.532 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.806+0000 7f5b19471700 1 -- 172.21.15.90:0/1026135320 shutdown_connections 2024-04-25T11:26:05.532 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.806+0000 7f5b19471700 1 --2- 172.21.15.90:0/1026135320 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f5b0003d520 0x7f5b0003fa30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:05.532 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.806+0000 7f5b19471700 1 --2- 172.21.15.90:0/1026135320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b14108ad0 0x7f5b1419dff0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:05.532 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.806+0000 7f5b19471700 1 -- 172.21.15.90:0/1026135320 >> 172.21.15.90:0/1026135320 conn(0x7f5b14104110 msgr2=0x7f5b14194550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:05.532 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.806+0000 7f5b19471700 1 -- 172.21.15.90:0/1026135320 shutdown_connections 2024-04-25T11:26:05.532 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:04.806+0000 7f5b19471700 1 -- 172.21.15.90:0/1026135320 wait complete. 2024-04-25T11:26:06.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:05 smithi090 ceph-mon[29575]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:26:06.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:05 smithi090 ceph-mon[29575]: Saving service mgr spec with placement count:2 2024-04-25T11:26:06.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:05 smithi090 ceph-mon[29575]: from='mgr.14120 172.21.15.90:0/3545268890' entity='mgr.x' 2024-04-25T11:26:06.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:05 smithi090 ceph-mon[29575]: from='mgr.14120 172.21.15.90:0/3545268890' entity='mgr.x' 2024-04-25T11:26:06.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:05 smithi090 ceph-mon[29575]: from='mgr.14120 172.21.15.90:0/3545268890' entity='mgr.x' 2024-04-25T11:26:07.293 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:05.997+0000 7f1c87137700 1 Processor -- start 2024-04-25T11:26:07.293 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:05.997+0000 7f1c87137700 1 -- start start 2024-04-25T11:26:07.294 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:05.997+0000 7f1c87137700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c8007ccc0 0x7f1c8010c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:07.294 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:05.997+0000 7f1c87137700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8010ccf0 con 0x7f1c8007ccc0 2024-04-25T11:26:07.294 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:05.998+0000 7f1c84ed3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c8007ccc0 0x7f1c8010c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:07.294 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:05.998+0000 7f1c84ed3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c8007ccc0 0x7f1c8010c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46646/0 (socket says 172.21.15.90:46646) 2024-04-25T11:26:07.294 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:05.998+0000 7f1c84ed3700 1 -- 172.21.15.90:0/2798957706 learned_addr learned my addr 172.21.15.90:0/2798957706 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:07.294 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:05.998+0000 7f1c84ed3700 1 -- 172.21.15.90:0/2798957706 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c8010ce70 con 0x7f1c8007ccc0 2024-04-25T11:26:07.294 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:05.998+0000 7f1c84ed3700 1 --2- 172.21.15.90:0/2798957706 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c8007ccc0 0x7f1c8010c6e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1c740098a0 tx=0x7f1c74009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b73db84f8d2abea1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:07.294 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:05.998+0000 7f1c7f7fe700 1 -- 172.21.15.90:0/2798957706 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c74004160 con 0x7f1c8007ccc0 2024-04-25T11:26:07.294 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:05.998+0000 7f1c7f7fe700 1 -- 172.21.15.90:0/2798957706 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c74004300 con 0x7f1c8007ccc0 2024-04-25T11:26:07.294 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:05.998+0000 7f1c7f7fe700 1 -- 172.21.15.90:0/2798957706 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c74011a20 con 0x7f1c8007ccc0 2024-04-25T11:26:07.294 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:05.999+0000 7f1c87137700 1 -- 172.21.15.90:0/2798957706 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c8007ccc0 msgr2=0x7f1c8010c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:07.294 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:05.999+0000 7f1c87137700 1 --2- 172.21.15.90:0/2798957706 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c8007ccc0 0x7f1c8010c6e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1c740098a0 tx=0x7f1c74009bb0 comp rx=0 tx=0).stop 2024-04-25T11:26:07.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:05.999+0000 7f1c87137700 1 -- 172.21.15.90:0/2798957706 shutdown_connections 2024-04-25T11:26:07.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:05.999+0000 7f1c87137700 1 --2- 172.21.15.90:0/2798957706 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c8007ccc0 0x7f1c8010c6e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:07.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:05.999+0000 7f1c87137700 1 -- 172.21.15.90:0/2798957706 >> 172.21.15.90:0/2798957706 conn(0x7f1c801040f0 msgr2=0x7f1c801065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:07.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:05.999+0000 7f1c87137700 1 -- 172.21.15.90:0/2798957706 shutdown_connections 2024-04-25T11:26:07.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:05.999+0000 7f1c87137700 1 -- 172.21.15.90:0/2798957706 wait complete. 2024-04-25T11:26:07.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.000+0000 7f1c87137700 1 Processor -- start 2024-04-25T11:26:07.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.000+0000 7f1c87137700 1 -- start start 2024-04-25T11:26:07.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.000+0000 7f1c87137700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c8007ccc0 0x7f1c8019df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:07.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.000+0000 7f1c87137700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8010df30 con 0x7f1c8007ccc0 2024-04-25T11:26:07.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.000+0000 7f1c84ed3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c8007ccc0 0x7f1c8019df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:07.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.000+0000 7f1c84ed3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c8007ccc0 0x7f1c8019df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46654/0 (socket says 172.21.15.90:46654) 2024-04-25T11:26:07.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.000+0000 7f1c84ed3700 1 -- 172.21.15.90:0/2116527938 learned_addr learned my addr 172.21.15.90:0/2116527938 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:07.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.000+0000 7f1c84ed3700 1 -- 172.21.15.90:0/2116527938 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c8019e460 con 0x7f1c8007ccc0 2024-04-25T11:26:07.296 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.001+0000 7f1c84ed3700 1 --2- 172.21.15.90:0/2116527938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c8007ccc0 0x7f1c8019df20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1c74009520 tx=0x7f1c74011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:07.296 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.001+0000 7f1c7dffb700 1 -- 172.21.15.90:0/2116527938 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c740039e0 con 0x7f1c8007ccc0 2024-04-25T11:26:07.296 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.001+0000 7f1c7dffb700 1 -- 172.21.15.90:0/2116527938 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c7401b8f0 con 0x7f1c8007ccc0 2024-04-25T11:26:07.296 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.001+0000 7f1c87137700 1 -- 172.21.15.90:0/2116527938 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c8019e6f0 con 0x7f1c8007ccc0 2024-04-25T11:26:07.296 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.001+0000 7f1c7dffb700 1 -- 172.21.15.90:0/2116527938 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c740197d0 con 0x7f1c8007ccc0 2024-04-25T11:26:07.296 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.001+0000 7f1c87137700 1 -- 172.21.15.90:0/2116527938 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c801a1300 con 0x7f1c8007ccc0 2024-04-25T11:26:07.296 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.002+0000 7f1c87137700 1 -- 172.21.15.90:0/2116527938 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c80064cf0 con 0x7f1c8007ccc0 2024-04-25T11:26:07.296 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.002+0000 7f1c7dffb700 1 -- 172.21.15.90:0/2116527938 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f1c74019a40 con 0x7f1c8007ccc0 2024-04-25T11:26:07.296 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.003+0000 7f1c7dffb700 1 --2- 172.21.15.90:0/2116527938 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f1c7003d5c0 0x7f1c7003fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:07.296 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.003+0000 7f1c7dffb700 1 -- 172.21.15.90:0/2116527938 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1c74052d40 con 0x7f1c8007ccc0 2024-04-25T11:26:07.296 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.003+0000 7f1c7ffff700 1 --2- 172.21.15.90:0/2116527938 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f1c7003d5c0 0x7f1c7003fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:07.296 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.003+0000 7f1c7ffff700 1 --2- 172.21.15.90:0/2116527938 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f1c7003d5c0 0x7f1c7003fad0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c006fd0 tx=0x7f1c6c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:07.296 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.008+0000 7f1c7dffb700 1 -- 172.21.15.90:0/2116527938 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1c7401f030 con 0x7f1c8007ccc0 2024-04-25T11:26:07.297 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.230+0000 7f1c87137700 1 -- 172.21.15.90:0/2116527938 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f1c801a1610 con 0x7f1c8007ccc0 2024-04-25T11:26:07.297 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.235+0000 7f1c7dffb700 1 -- 172.21.15.90:0/2116527938 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f1c7402a350 con 0x7f1c8007ccc0 2024-04-25T11:26:07.297 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.237+0000 7f1c87137700 1 -- 172.21.15.90:0/2116527938 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f1c7003d5c0 msgr2=0x7f1c7003fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:07.297 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.237+0000 7f1c87137700 1 --2- 172.21.15.90:0/2116527938 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f1c7003d5c0 0x7f1c7003fad0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c006fd0 tx=0x7f1c6c006e40 comp rx=0 tx=0).stop 2024-04-25T11:26:07.297 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.237+0000 7f1c87137700 1 -- 172.21.15.90:0/2116527938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c8007ccc0 msgr2=0x7f1c8019df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:07.297 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.237+0000 7f1c87137700 1 --2- 172.21.15.90:0/2116527938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c8007ccc0 0x7f1c8019df20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1c74009520 tx=0x7f1c74011aa0 comp rx=0 tx=0).stop 2024-04-25T11:26:07.297 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.237+0000 7f1c87137700 1 -- 172.21.15.90:0/2116527938 shutdown_connections 2024-04-25T11:26:07.297 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.237+0000 7f1c87137700 1 --2- 172.21.15.90:0/2116527938 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f1c7003d5c0 0x7f1c7003fad0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:07.297 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.237+0000 7f1c87137700 1 --2- 172.21.15.90:0/2116527938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c8007ccc0 0x7f1c8019df20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:07.297 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.237+0000 7f1c87137700 1 -- 172.21.15.90:0/2116527938 >> 172.21.15.90:0/2116527938 conn(0x7f1c801040f0 msgr2=0x7f1c80104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:07.297 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.237+0000 7f1c87137700 1 -- 172.21.15.90:0/2116527938 shutdown_connections 2024-04-25T11:26:07.297 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:06.238+0000 7f1c87137700 1 -- 172.21.15.90:0/2116527938 wait complete. 2024-04-25T11:26:07.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:07 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2116527938' entity='client.admin' 2024-04-25T11:26:08.602 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.785+0000 7f58945fc700 1 Processor -- start 2024-04-25T11:26:08.602 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.785+0000 7f58945fc700 1 -- start start 2024-04-25T11:26:08.602 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.785+0000 7f58945fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f588c108ab0 0x7f588c108ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:08.603 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.785+0000 7f58945fc700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f588c1094b0 con 0x7f588c108ab0 2024-04-25T11:26:08.603 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.785+0000 7f5892398700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f588c108ab0 0x7f588c108ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:08.603 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.785+0000 7f5892398700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f588c108ab0 0x7f588c108ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46662/0 (socket says 172.21.15.90:46662) 2024-04-25T11:26:08.603 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.785+0000 7f5892398700 1 -- 172.21.15.90:0/1201492091 learned_addr learned my addr 172.21.15.90:0/1201492091 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:08.603 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.786+0000 7f5892398700 1 -- 172.21.15.90:0/1201492091 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f588c109be0 con 0x7f588c108ab0 2024-04-25T11:26:08.603 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.786+0000 7f5892398700 1 --2- 172.21.15.90:0/1201492091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f588c108ab0 0x7f588c108ec0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f58880098a0 tx=0x7f5888009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35ca93b63625d0bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:08.603 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.786+0000 7f5891396700 1 -- 172.21.15.90:0/1201492091 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5888004160 con 0x7f588c108ab0 2024-04-25T11:26:08.603 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.786+0000 7f5891396700 1 -- 172.21.15.90:0/1201492091 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5888004300 con 0x7f588c108ab0 2024-04-25T11:26:08.603 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.786+0000 7f5891396700 1 -- 172.21.15.90:0/1201492091 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5888011a20 con 0x7f588c108ab0 2024-04-25T11:26:08.603 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.787+0000 7f58945fc700 1 -- 172.21.15.90:0/1201492091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f588c108ab0 msgr2=0x7f588c108ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:08.603 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.787+0000 7f58945fc700 1 --2- 172.21.15.90:0/1201492091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f588c108ab0 0x7f588c108ec0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f58880098a0 tx=0x7f5888009bb0 comp rx=0 tx=0).stop 2024-04-25T11:26:08.603 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.787+0000 7f58945fc700 1 -- 172.21.15.90:0/1201492091 shutdown_connections 2024-04-25T11:26:08.604 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.787+0000 7f58945fc700 1 --2- 172.21.15.90:0/1201492091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f588c108ab0 0x7f588c108ec0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:08.604 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.787+0000 7f58945fc700 1 -- 172.21.15.90:0/1201492091 >> 172.21.15.90:0/1201492091 conn(0x7f588c1040f0 msgr2=0x7f588c1065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:08.604 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.788+0000 7f58945fc700 1 -- 172.21.15.90:0/1201492091 shutdown_connections 2024-04-25T11:26:08.604 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.788+0000 7f58945fc700 1 -- 172.21.15.90:0/1201492091 wait complete. 2024-04-25T11:26:08.604 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.788+0000 7f58945fc700 1 Processor -- start 2024-04-25T11:26:08.604 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.788+0000 7f58945fc700 1 -- start start 2024-04-25T11:26:08.604 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.789+0000 7f58945fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f588c108ab0 0x7f588c19ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:08.604 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.789+0000 7f58945fc700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f588c10cf90 con 0x7f588c108ab0 2024-04-25T11:26:08.604 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.789+0000 7f5892398700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f588c108ab0 0x7f588c19ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:08.604 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.789+0000 7f5892398700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f588c108ab0 0x7f588c19ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46674/0 (socket says 172.21.15.90:46674) 2024-04-25T11:26:08.604 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.789+0000 7f5892398700 1 -- 172.21.15.90:0/2150076205 learned_addr learned my addr 172.21.15.90:0/2150076205 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:08.604 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.789+0000 7f5892398700 1 -- 172.21.15.90:0/2150076205 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f588c19d3b0 con 0x7f588c108ab0 2024-04-25T11:26:08.604 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.789+0000 7f5892398700 1 --2- 172.21.15.90:0/2150076205 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f588c108ab0 0x7f588c19ce70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5888009130 tx=0x7f5888011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:08.604 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.789+0000 7f58837fe700 1 -- 172.21.15.90:0/2150076205 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5888003aa0 con 0x7f588c108ab0 2024-04-25T11:26:08.605 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.789+0000 7f58837fe700 1 -- 172.21.15.90:0/2150076205 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f588801a8e0 con 0x7f588c108ab0 2024-04-25T11:26:08.605 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.789+0000 7f58945fc700 1 -- 172.21.15.90:0/2150076205 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f588c19d640 con 0x7f588c108ab0 2024-04-25T11:26:08.605 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.789+0000 7f58837fe700 1 -- 172.21.15.90:0/2150076205 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58880248f0 con 0x7f588c108ab0 2024-04-25T11:26:08.605 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.789+0000 7f58945fc700 1 -- 172.21.15.90:0/2150076205 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f588c1a0250 con 0x7f588c108ab0 2024-04-25T11:26:08.605 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.790+0000 7f58837fe700 1 -- 172.21.15.90:0/2150076205 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f5888022020 con 0x7f588c108ab0 2024-04-25T11:26:08.605 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.791+0000 7f58837fe700 1 --2- 172.21.15.90:0/2150076205 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f587803d5c0 0x7f587803fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:08.605 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.791+0000 7f58837fe700 1 -- 172.21.15.90:0/2150076205 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f58880527a0 con 0x7f588c108ab0 2024-04-25T11:26:08.605 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.791+0000 7f58945fc700 1 -- 172.21.15.90:0/2150076205 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f588c064cf0 con 0x7f588c108ab0 2024-04-25T11:26:08.605 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.791+0000 7f5891b97700 1 --2- 172.21.15.90:0/2150076205 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f587803d5c0 0x7f587803fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:08.605 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.791+0000 7f5891b97700 1 --2- 172.21.15.90:0/2150076205 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f587803d5c0 0x7f587803fad0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f587c006fd0 tx=0x7f587c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:08.605 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.797+0000 7f58837fe700 1 -- 172.21.15.90:0/2150076205 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5888052130 con 0x7f588c108ab0 2024-04-25T11:26:08.605 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.996+0000 7f58945fc700 1 -- 172.21.15.90:0/2150076205 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f588c1a0420 con 0x7f588c108ab0 2024-04-25T11:26:08.605 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:07.999+0000 7f58837fe700 1 -- 172.21.15.90:0/2150076205 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f5888029df0 con 0x7f588c108ab0 2024-04-25T11:26:08.606 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:08.001+0000 7f58945fc700 1 -- 172.21.15.90:0/2150076205 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f587803d5c0 msgr2=0x7f587803fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:08.606 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:08.001+0000 7f58945fc700 1 --2- 172.21.15.90:0/2150076205 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f587803d5c0 0x7f587803fad0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f587c006fd0 tx=0x7f587c006e40 comp rx=0 tx=0).stop 2024-04-25T11:26:08.606 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:08.001+0000 7f58945fc700 1 -- 172.21.15.90:0/2150076205 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f588c108ab0 msgr2=0x7f588c19ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:08.606 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:08.001+0000 7f58945fc700 1 --2- 172.21.15.90:0/2150076205 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f588c108ab0 0x7f588c19ce70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5888009130 tx=0x7f5888011aa0 comp rx=0 tx=0).stop 2024-04-25T11:26:08.606 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:08.001+0000 7f58945fc700 1 -- 172.21.15.90:0/2150076205 shutdown_connections 2024-04-25T11:26:08.606 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:08.002+0000 7f58945fc700 1 --2- 172.21.15.90:0/2150076205 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f587803d5c0 0x7f587803fad0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:08.606 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:08.002+0000 7f58945fc700 1 --2- 172.21.15.90:0/2150076205 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f588c108ab0 0x7f588c19ce70 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:08.606 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:08.002+0000 7f58945fc700 1 -- 172.21.15.90:0/2150076205 >> 172.21.15.90:0/2150076205 conn(0x7f588c1040f0 msgr2=0x7f588c104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:08.606 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:08.002+0000 7f58945fc700 1 -- 172.21.15.90:0/2150076205 shutdown_connections 2024-04-25T11:26:08.606 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:08.002+0000 7f58945fc700 1 -- 172.21.15.90:0/2150076205 wait complete. 2024-04-25T11:26:08.606 INFO:teuthology.orchestra.run.smithi090.stdout:Enabling the dashboard module... 2024-04-25T11:26:08.722 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:08 smithi090 ceph-mon[29575]: from='mgr.14120 172.21.15.90:0/3545268890' entity='mgr.x' 2024-04-25T11:26:08.722 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:08 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2150076205' entity='client.admin' 2024-04-25T11:26:09.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:09 smithi090 ceph-mon[29575]: from='mgr.14120 172.21.15.90:0/3545268890' entity='mgr.x' 2024-04-25T11:26:09.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:09 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3929931458' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-25T11:26:09.979 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:09 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: ignoring --setuser ceph since I am not root 2024-04-25T11:26:09.979 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:09 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: ignoring --setgroup ceph since I am not root 2024-04-25T11:26:09.980 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:09 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:26:09.906+0000 7f8434bd7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T11:26:10.465 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:10 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:26:10.093+0000 7f8434bd7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T11:26:10.465 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:10 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:26:10.198+0000 7f8434bd7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T11:26:10.485 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.071+0000 7f8117e3c700 1 Processor -- start 2024-04-25T11:26:10.485 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.072+0000 7f8117e3c700 1 -- start start 2024-04-25T11:26:10.485 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.072+0000 7f8117e3c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f811007ccc0 0x7f811010c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:10.485 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.072+0000 7f8117e3c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f811010ccf0 con 0x7f811007ccc0 2024-04-25T11:26:10.485 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.072+0000 7f8115bd8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f811007ccc0 0x7f811010c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:10.485 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.072+0000 7f8115bd8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f811007ccc0 0x7f811010c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46686/0 (socket says 172.21.15.90:46686) 2024-04-25T11:26:10.485 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.072+0000 7f8115bd8700 1 -- 172.21.15.90:0/500162657 learned_addr learned my addr 172.21.15.90:0/500162657 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:10.486 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.073+0000 7f8115bd8700 1 -- 172.21.15.90:0/500162657 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f811010ce70 con 0x7f811007ccc0 2024-04-25T11:26:10.486 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.073+0000 7f8115bd8700 1 --2- 172.21.15.90:0/500162657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f811007ccc0 0x7f811010c6e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f810c0098a0 tx=0x7f810c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b00726908af37e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:10.486 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.073+0000 7f8114bd6700 1 -- 172.21.15.90:0/500162657 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f810c004160 con 0x7f811007ccc0 2024-04-25T11:26:10.486 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.073+0000 7f8114bd6700 1 -- 172.21.15.90:0/500162657 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f810c004300 con 0x7f811007ccc0 2024-04-25T11:26:10.486 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.073+0000 7f8114bd6700 1 -- 172.21.15.90:0/500162657 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f810c011a20 con 0x7f811007ccc0 2024-04-25T11:26:10.486 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.074+0000 7f8117e3c700 1 -- 172.21.15.90:0/500162657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f811007ccc0 msgr2=0x7f811010c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:10.486 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.074+0000 7f8117e3c700 1 --2- 172.21.15.90:0/500162657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f811007ccc0 0x7f811010c6e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f810c0098a0 tx=0x7f810c009bb0 comp rx=0 tx=0).stop 2024-04-25T11:26:10.486 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.074+0000 7f8117e3c700 1 -- 172.21.15.90:0/500162657 shutdown_connections 2024-04-25T11:26:10.486 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.074+0000 7f8117e3c700 1 --2- 172.21.15.90:0/500162657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f811007ccc0 0x7f811010c6e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:10.486 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.074+0000 7f8117e3c700 1 -- 172.21.15.90:0/500162657 >> 172.21.15.90:0/500162657 conn(0x7f81101040f0 msgr2=0x7f81101065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:10.486 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.074+0000 7f8117e3c700 1 -- 172.21.15.90:0/500162657 shutdown_connections 2024-04-25T11:26:10.486 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.074+0000 7f8117e3c700 1 -- 172.21.15.90:0/500162657 wait complete. 2024-04-25T11:26:10.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.075+0000 7f8117e3c700 1 Processor -- start 2024-04-25T11:26:10.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.075+0000 7f8117e3c700 1 -- start start 2024-04-25T11:26:10.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.075+0000 7f8117e3c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f811007ccc0 0x7f811019df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:10.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.075+0000 7f8117e3c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f811010df30 con 0x7f811007ccc0 2024-04-25T11:26:10.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.075+0000 7f8115bd8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f811007ccc0 0x7f811019df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:10.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.075+0000 7f8115bd8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f811007ccc0 0x7f811019df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46696/0 (socket says 172.21.15.90:46696) 2024-04-25T11:26:10.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.075+0000 7f8115bd8700 1 -- 172.21.15.90:0/3929931458 learned_addr learned my addr 172.21.15.90:0/3929931458 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:10.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.075+0000 7f8115bd8700 1 -- 172.21.15.90:0/3929931458 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f811019e450 con 0x7f811007ccc0 2024-04-25T11:26:10.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.076+0000 7f8115bd8700 1 --2- 172.21.15.90:0/3929931458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f811007ccc0 0x7f811019df10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f810c009520 tx=0x7f810c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:10.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.076+0000 7f8106ffd700 1 -- 172.21.15.90:0/3929931458 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f810c0039e0 con 0x7f811007ccc0 2024-04-25T11:26:10.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.076+0000 7f8117e3c700 1 -- 172.21.15.90:0/3929931458 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f811019e6e0 con 0x7f811007ccc0 2024-04-25T11:26:10.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.076+0000 7f8106ffd700 1 -- 172.21.15.90:0/3929931458 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f810c01b8f0 con 0x7f811007ccc0 2024-04-25T11:26:10.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.076+0000 7f8106ffd700 1 -- 172.21.15.90:0/3929931458 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f810c0197d0 con 0x7f811007ccc0 2024-04-25T11:26:10.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.076+0000 7f8117e3c700 1 -- 172.21.15.90:0/3929931458 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81101a12f0 con 0x7f811007ccc0 2024-04-25T11:26:10.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.077+0000 7f8106ffd700 1 -- 172.21.15.90:0/3929931458 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f810c019ac0 con 0x7f811007ccc0 2024-04-25T11:26:10.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.077+0000 7f8106ffd700 1 --2- 172.21.15.90:0/3929931458 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f80fc03d5c0 0x7f80fc03fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:10.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.077+0000 7f8106ffd700 1 -- 172.21.15.90:0/3929931458 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f810c052d90 con 0x7f811007ccc0 2024-04-25T11:26:10.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.077+0000 7f8117e3c700 1 -- 172.21.15.90:0/3929931458 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8110064cf0 con 0x7f811007ccc0 2024-04-25T11:26:10.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.077+0000 7f81153d7700 1 --2- 172.21.15.90:0/3929931458 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f80fc03d5c0 0x7f80fc03fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:10.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.078+0000 7f81153d7700 1 --2- 172.21.15.90:0/3929931458 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f80fc03d5c0 0x7f80fc03fad0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8100006fd0 tx=0x7f8100006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:10.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.083+0000 7f8106ffd700 1 -- 172.21.15.90:0/3929931458 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f810c01f030 con 0x7f811007ccc0 2024-04-25T11:26:10.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.410+0000 7f8117e3c700 1 -- 172.21.15.90:0/3929931458 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f81101a14c0 con 0x7f811007ccc0 2024-04-25T11:26:10.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.705+0000 7f8106ffd700 1 -- 172.21.15.90:0/3929931458 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f810c02be20 con 0x7f811007ccc0 2024-04-25T11:26:10.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.706+0000 7f8106ffd700 1 -- 172.21.15.90:0/3929931458 <== mon.0 v2:172.21.15.90:3300/0 8 ==== mgrmap(e 9) ==== 49529+0+0 (secure 0 0 0) 0x7f810c01a3d0 con 0x7f811007ccc0 2024-04-25T11:26:10.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.707+0000 7f8117e3c700 1 -- 172.21.15.90:0/3929931458 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f80fc03d5c0 msgr2=0x7f80fc03fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:10.489 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.707+0000 7f8117e3c700 1 --2- 172.21.15.90:0/3929931458 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f80fc03d5c0 0x7f80fc03fad0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8100006fd0 tx=0x7f8100006e40 comp rx=0 tx=0).stop 2024-04-25T11:26:10.489 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.707+0000 7f8117e3c700 1 -- 172.21.15.90:0/3929931458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f811007ccc0 msgr2=0x7f811019df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:10.489 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.707+0000 7f8117e3c700 1 --2- 172.21.15.90:0/3929931458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f811007ccc0 0x7f811019df10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f810c009520 tx=0x7f810c011aa0 comp rx=0 tx=0).stop 2024-04-25T11:26:10.489 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.708+0000 7f8117e3c700 1 -- 172.21.15.90:0/3929931458 shutdown_connections 2024-04-25T11:26:10.489 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.708+0000 7f8117e3c700 1 --2- 172.21.15.90:0/3929931458 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f80fc03d5c0 0x7f80fc03fad0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:10.489 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.708+0000 7f8117e3c700 1 --2- 172.21.15.90:0/3929931458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f811007ccc0 0x7f811019df10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:10.489 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.708+0000 7f8117e3c700 1 -- 172.21.15.90:0/3929931458 >> 172.21.15.90:0/3929931458 conn(0x7f81101040f0 msgr2=0x7f81101049c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:10.489 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.708+0000 7f8117e3c700 1 -- 172.21.15.90:0/3929931458 shutdown_connections 2024-04-25T11:26:10.489 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:09.708+0000 7f8117e3c700 1 -- 172.21.15.90:0/3929931458 wait complete. 2024-04-25T11:26:10.716 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:10 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3929931458' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-25T11:26:10.716 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:10 smithi090 ceph-mon[29575]: mgrmap e9: x(active, since 19s) 2024-04-25T11:26:10.716 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:10 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:26:10.516+0000 7f8434bd7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T11:26:10.716 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:10 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:26:10.604+0000 7f8434bd7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T11:26:11.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:11 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/418113031' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T11:26:11.949 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-04-25T11:26:11.949 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-25T11:26:11.949 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T11:26:11.949 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T11:26:11.949 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T11:26:11.949 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-04-25T11:26:11.949 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.960+0000 7fead9aae700 1 Processor -- start 2024-04-25T11:26:11.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.960+0000 7fead9aae700 1 -- start start 2024-04-25T11:26:11.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.960+0000 7fead9aae700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead407cc80 0x7fead410c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:11.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.960+0000 7fead9aae700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fead410cd40 con 0x7fead407cc80 2024-04-25T11:26:11.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.960+0000 7fead37fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead407cc80 0x7fead410c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:11.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.960+0000 7fead37fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead407cc80 0x7fead410c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46726/0 (socket says 172.21.15.90:46726) 2024-04-25T11:26:11.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.960+0000 7fead37fe700 1 -- 172.21.15.90:0/651197847 learned_addr learned my addr 172.21.15.90:0/651197847 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:11.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.960+0000 7fead37fe700 1 -- 172.21.15.90:0/651197847 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fead410cec0 con 0x7fead407cc80 2024-04-25T11:26:11.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.961+0000 7fead37fe700 1 --2- 172.21.15.90:0/651197847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead407cc80 0x7fead410c770 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7feacc009e80 tx=0x7feacc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f43b2240dba2f672 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:11.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.961+0000 7fead27fc700 1 -- 172.21.15.90:0/651197847 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feacc017070 con 0x7fead407cc80 2024-04-25T11:26:11.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.961+0000 7fead27fc700 1 -- 172.21.15.90:0/651197847 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feacc00b8c0 con 0x7fead407cc80 2024-04-25T11:26:11.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.961+0000 7fead27fc700 1 -- 172.21.15.90:0/651197847 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feacc004490 con 0x7fead407cc80 2024-04-25T11:26:11.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.962+0000 7fead9aae700 1 -- 172.21.15.90:0/651197847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead407cc80 msgr2=0x7fead410c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:11.950 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.962+0000 7fead9aae700 1 --2- 172.21.15.90:0/651197847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead407cc80 0x7fead410c770 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7feacc009e80 tx=0x7feacc00b330 comp rx=0 tx=0).stop 2024-04-25T11:26:11.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.962+0000 7fead9aae700 1 -- 172.21.15.90:0/651197847 shutdown_connections 2024-04-25T11:26:11.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.962+0000 7fead9aae700 1 --2- 172.21.15.90:0/651197847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead407cc80 0x7fead410c770 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:11.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.962+0000 7fead9aae700 1 -- 172.21.15.90:0/651197847 >> 172.21.15.90:0/651197847 conn(0x7fead4104140 msgr2=0x7fead41065f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:11.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.962+0000 7fead9aae700 1 -- 172.21.15.90:0/651197847 shutdown_connections 2024-04-25T11:26:11.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.962+0000 7fead9aae700 1 -- 172.21.15.90:0/651197847 wait complete. 2024-04-25T11:26:11.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.963+0000 7fead9aae700 1 Processor -- start 2024-04-25T11:26:11.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.963+0000 7fead9aae700 1 -- start start 2024-04-25T11:26:11.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.963+0000 7fead9aae700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead407cc80 0x7fead419df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:11.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.963+0000 7fead9aae700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fead410df80 con 0x7fead407cc80 2024-04-25T11:26:11.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.963+0000 7fead37fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead407cc80 0x7fead419df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:11.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.963+0000 7fead37fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead407cc80 0x7fead419df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46734/0 (socket says 172.21.15.90:46734) 2024-04-25T11:26:11.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.963+0000 7fead37fe700 1 -- 172.21.15.90:0/418113031 learned_addr learned my addr 172.21.15.90:0/418113031 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:11.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.963+0000 7fead37fe700 1 -- 172.21.15.90:0/418113031 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fead419e4a0 con 0x7fead407cc80 2024-04-25T11:26:11.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.964+0000 7fead37fe700 1 --2- 172.21.15.90:0/418113031 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead407cc80 0x7fead419df60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7feacc0072a0 tx=0x7feacc00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:11.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.964+0000 7fead0ff9700 1 -- 172.21.15.90:0/418113031 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feacc020020 con 0x7fead407cc80 2024-04-25T11:26:11.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.964+0000 7fead0ff9700 1 -- 172.21.15.90:0/418113031 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feacc012430 con 0x7fead407cc80 2024-04-25T11:26:11.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.964+0000 7fead9aae700 1 -- 172.21.15.90:0/418113031 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fead419e730 con 0x7fead407cc80 2024-04-25T11:26:11.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.964+0000 7fead0ff9700 1 -- 172.21.15.90:0/418113031 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feacc017040 con 0x7fead407cc80 2024-04-25T11:26:11.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.964+0000 7fead9aae700 1 -- 172.21.15.90:0/418113031 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fead41a1340 con 0x7fead407cc80 2024-04-25T11:26:11.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.965+0000 7fead9aae700 1 -- 172.21.15.90:0/418113031 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fead4064cf0 con 0x7fead407cc80 2024-04-25T11:26:11.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.965+0000 7fead0ff9700 1 -- 172.21.15.90:0/418113031 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 9) ==== 49529+0+0 (secure 0 0 0) 0x7feacc023020 con 0x7fead407cc80 2024-04-25T11:26:11.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.965+0000 7fead0ff9700 1 --2- 172.21.15.90:0/418113031 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7feabc03d570 0x7feabc03fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:11.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.965+0000 7fead0ff9700 1 -- 172.21.15.90:0/418113031 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7feacc00f050 con 0x7fead407cc80 2024-04-25T11:26:11.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.965+0000 7fead2ffd700 1 -- 172.21.15.90:0/418113031 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7feabc03d570 msgr2=0x7feabc03fa80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/71478430 2024-04-25T11:26:11.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.965+0000 7fead2ffd700 1 --2- 172.21.15.90:0/418113031 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7feabc03d570 0x7feabc03fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T11:26:11.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:10.975+0000 7fead0ff9700 1 -- 172.21.15.90:0/418113031 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feacc02a020 con 0x7fead407cc80 2024-04-25T11:26:11.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:11.166+0000 7fead2ffd700 1 -- 172.21.15.90:0/418113031 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7feabc03d570 msgr2=0x7feabc03fa80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/71478430 2024-04-25T11:26:11.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:11.166+0000 7fead2ffd700 1 --2- 172.21.15.90:0/418113031 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7feabc03d570 0x7feabc03fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T11:26:11.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:11.266+0000 7fead9aae700 1 -- 172.21.15.90:0/418113031 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fead41a1510 con 0x7fead407cc80 2024-04-25T11:26:11.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:11.267+0000 7fead0ff9700 1 -- 172.21.15.90:0/418113031 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fead41a1510 con 0x7fead407cc80 2024-04-25T11:26:11.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:11.269+0000 7fead9aae700 1 -- 172.21.15.90:0/418113031 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7feabc03d570 msgr2=0x7feabc03fa80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:26:11.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:11.269+0000 7fead9aae700 1 --2- 172.21.15.90:0/418113031 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7feabc03d570 0x7feabc03fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:11.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:11.269+0000 7fead9aae700 1 -- 172.21.15.90:0/418113031 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead407cc80 msgr2=0x7fead419df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:11.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:11.269+0000 7fead9aae700 1 --2- 172.21.15.90:0/418113031 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead407cc80 0x7fead419df60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7feacc0072a0 tx=0x7feacc00bd50 comp rx=0 tx=0).stop 2024-04-25T11:26:11.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:11.269+0000 7fead9aae700 1 -- 172.21.15.90:0/418113031 shutdown_connections 2024-04-25T11:26:11.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:11.269+0000 7fead9aae700 1 --2- 172.21.15.90:0/418113031 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7feabc03d570 0x7feabc03fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:11.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:11.269+0000 7fead9aae700 1 --2- 172.21.15.90:0/418113031 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead407cc80 0x7fead419df60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:11.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:11.269+0000 7fead9aae700 1 -- 172.21.15.90:0/418113031 >> 172.21.15.90:0/418113031 conn(0x7fead4104140 msgr2=0x7fead4104a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:11.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:11.270+0000 7fead9aae700 1 -- 172.21.15.90:0/418113031 shutdown_connections 2024-04-25T11:26:11.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:11.270+0000 7fead9aae700 1 -- 172.21.15.90:0/418113031 wait complete. 2024-04-25T11:26:11.953 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for the mgr to restart... 2024-04-25T11:26:11.954 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mgr epoch 9... 2024-04-25T11:26:12.167 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:11 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:26:11.844+0000 7f8434bd7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T11:26:12.167 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:12 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:26:12.030+0000 7f8434bd7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T11:26:12.479 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:12 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:26:12.205+0000 7f8434bd7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T11:26:12.798 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:12 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:26:12.510+0000 7f8434bd7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T11:26:12.798 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:12 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:26:12.604+0000 7f8434bd7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T11:26:12.799 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:12 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:26:12.806+0000 7f8434bd7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T11:26:13.229 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:13 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:26:13.001+0000 7f8434bd7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T11:26:13.979 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:13 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:26:13.598+0000 7f8434bd7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T11:26:14.323 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:14 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:26:14.029+0000 7f8434bd7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T11:26:14.323 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:14 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:26:14.126+0000 7f8434bd7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T11:26:14.323 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:14 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:26:14.331+0000 7f8434bd7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T11:26:15.180 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:14 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:26:14.848+0000 7f8434bd7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T11:26:15.180 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:14 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:26:14.942+0000 7f8434bd7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T11:26:15.479 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:15 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:26:15.188+0000 7f8434bd7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T11:26:15.479 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:15 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:26:15.322+0000 7f8434bd7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T11:26:15.479 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:15 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:26:15.419+0000 7f8434bd7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T11:26:15.826 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:15 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:26:15.537+0000 7f8434bd7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T11:26:15.826 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:15 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:26:15.834+0000 7f8434bd7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T11:26:16.699 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:16 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:26:16.432+0000 7f8434bd7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T11:26:16.700 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:26:16 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:26:16.653+0000 7f8434bd7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T11:26:16.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:16 smithi090 ceph-mon[29575]: Active manager daemon x restarted 2024-04-25T11:26:16.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:16 smithi090 ceph-mon[29575]: Activating manager daemon x 2024-04-25T11:26:16.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:16 smithi090 ceph-mon[29575]: osdmap e3: 0 total, 0 up, 0 in 2024-04-25T11:26:16.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:16 smithi090 ceph-mon[29575]: mgrmap e10: x(active, starting, since 0.00466138s) 2024-04-25T11:26:16.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:16 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T11:26:16.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:16 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T11:26:16.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:16 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T11:26:16.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:16 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T11:26:16.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:16 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T11:26:16.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:16 smithi090 ceph-mon[29575]: Manager daemon x is now available 2024-04-25T11:26:16.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:16 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:26:17.844 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T11:26:17.844 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T11:26:17.844 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:17 smithi090 ceph-mon[29575]: mgrmap e11: x(active, since 1.00672s) 2024-04-25T11:26:18.504 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-04-25T11:26:18.504 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-25T11:26:18.504 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T11:26:18.504 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-04-25T11:26:18.504 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.404+0000 7f36c2fb2700 1 Processor -- start 2024-04-25T11:26:18.504 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.404+0000 7f36c2fb2700 1 -- start start 2024-04-25T11:26:18.504 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.404+0000 7f36c2fb2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36bc107fd0 0x7f36bc108440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:18.504 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.404+0000 7f36c2fb2700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f36bc108a50 con 0x7f36bc107fd0 2024-04-25T11:26:18.505 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.404+0000 7f36c0d4e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36bc107fd0 0x7f36bc108440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:18.505 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.404+0000 7f36c0d4e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36bc107fd0 0x7f36bc108440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52330/0 (socket says 172.21.15.90:52330) 2024-04-25T11:26:18.505 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.404+0000 7f36c0d4e700 1 -- 172.21.15.90:0/2552084503 learned_addr learned my addr 172.21.15.90:0/2552084503 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:18.505 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.405+0000 7f36c0d4e700 1 -- 172.21.15.90:0/2552084503 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36bc10d070 con 0x7f36bc107fd0 2024-04-25T11:26:18.505 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.405+0000 7f36c0d4e700 1 --2- 172.21.15.90:0/2552084503 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36bc107fd0 0x7f36bc108440 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f36b00098a0 tx=0x7f36b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b0390a1620b3304 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:18.505 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.405+0000 7f36bb7fe700 1 -- 172.21.15.90:0/2552084503 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36b0004160 con 0x7f36bc107fd0 2024-04-25T11:26:18.505 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.405+0000 7f36bb7fe700 1 -- 172.21.15.90:0/2552084503 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36b0004300 con 0x7f36bc107fd0 2024-04-25T11:26:18.505 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.405+0000 7f36bb7fe700 1 -- 172.21.15.90:0/2552084503 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36b0011a20 con 0x7f36bc107fd0 2024-04-25T11:26:18.505 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.406+0000 7f36c2fb2700 1 -- 172.21.15.90:0/2552084503 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36bc107fd0 msgr2=0x7f36bc108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:18.505 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.406+0000 7f36c2fb2700 1 --2- 172.21.15.90:0/2552084503 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36bc107fd0 0x7f36bc108440 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f36b00098a0 tx=0x7f36b0009bb0 comp rx=0 tx=0).stop 2024-04-25T11:26:18.505 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.406+0000 7f36c2fb2700 1 -- 172.21.15.90:0/2552084503 shutdown_connections 2024-04-25T11:26:18.505 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.406+0000 7f36c2fb2700 1 --2- 172.21.15.90:0/2552084503 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36bc107fd0 0x7f36bc108440 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:18.505 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.406+0000 7f36c2fb2700 1 -- 172.21.15.90:0/2552084503 >> 172.21.15.90:0/2552084503 conn(0x7f36bc103880 msgr2=0x7f36bc105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:18.506 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.406+0000 7f36c2fb2700 1 -- 172.21.15.90:0/2552084503 shutdown_connections 2024-04-25T11:26:18.506 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.406+0000 7f36c2fb2700 1 -- 172.21.15.90:0/2552084503 wait complete. 2024-04-25T11:26:18.506 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.407+0000 7f36c2fb2700 1 Processor -- start 2024-04-25T11:26:18.506 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.407+0000 7f36c2fb2700 1 -- start start 2024-04-25T11:26:18.506 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.407+0000 7f36c2fb2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36bc107fd0 0x7f36bc10a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:18.506 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.407+0000 7f36c2fb2700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f36bc10e010 con 0x7f36bc107fd0 2024-04-25T11:26:18.506 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.407+0000 7f36c0d4e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36bc107fd0 0x7f36bc10a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:18.506 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.407+0000 7f36c0d4e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36bc107fd0 0x7f36bc10a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52338/0 (socket says 172.21.15.90:52338) 2024-04-25T11:26:18.506 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.407+0000 7f36c0d4e700 1 -- 172.21.15.90:0/3917410498 learned_addr learned my addr 172.21.15.90:0/3917410498 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:18.506 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.407+0000 7f36c0d4e700 1 -- 172.21.15.90:0/3917410498 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36bc10ac90 con 0x7f36bc107fd0 2024-04-25T11:26:18.506 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.407+0000 7f36c0d4e700 1 --2- 172.21.15.90:0/3917410498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36bc107fd0 0x7f36bc10a750 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f36b0004660 tx=0x7f36b0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:18.506 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.407+0000 7f36b9ffb700 1 -- 172.21.15.90:0/3917410498 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36b0003a60 con 0x7f36bc107fd0 2024-04-25T11:26:18.506 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.408+0000 7f36c2fb2700 1 -- 172.21.15.90:0/3917410498 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36bc108ee0 con 0x7f36bc107fd0 2024-04-25T11:26:18.507 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.408+0000 7f36b9ffb700 1 -- 172.21.15.90:0/3917410498 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36b001b8f0 con 0x7f36bc107fd0 2024-04-25T11:26:18.507 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.408+0000 7f36b9ffb700 1 -- 172.21.15.90:0/3917410498 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36b0023890 con 0x7f36bc107fd0 2024-04-25T11:26:18.507 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.408+0000 7f36c2fb2700 1 -- 172.21.15.90:0/3917410498 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36bc1093c0 con 0x7f36bc107fd0 2024-04-25T11:26:18.507 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.409+0000 7f36b9ffb700 1 -- 172.21.15.90:0/3917410498 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 9) ==== 49529+0+0 (secure 0 0 0) 0x7f36b002c3f0 con 0x7f36bc107fd0 2024-04-25T11:26:18.507 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.409+0000 7f36b9ffb700 1 --2- 172.21.15.90:0/3917410498 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f36ac03d5d0 0x7f36ac03fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:18.507 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.409+0000 7f36b9ffb700 1 -- 172.21.15.90:0/3917410498 --> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f36ac0401f0 con 0x7f36ac03d5d0 2024-04-25T11:26:18.507 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.409+0000 7f36bbfff700 1 -- 172.21.15.90:0/3917410498 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f36ac03d5d0 msgr2=0x7f36ac03fae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/71478430 2024-04-25T11:26:18.507 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.409+0000 7f36b9ffb700 1 -- 172.21.15.90:0/3917410498 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f36b0053d10 con 0x7f36bc107fd0 2024-04-25T11:26:18.507 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.409+0000 7f36bbfff700 1 --2- 172.21.15.90:0/3917410498 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f36ac03d5d0 0x7f36ac03fae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T11:26:18.507 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.610+0000 7f36bbfff700 1 -- 172.21.15.90:0/3917410498 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f36ac03d5d0 msgr2=0x7f36ac03fae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/71478430 2024-04-25T11:26:18.507 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:12.610+0000 7f36bbfff700 1 --2- 172.21.15.90:0/3917410498 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f36ac03d5d0 0x7f36ac03fae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T11:26:18.507 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:13.011+0000 7f36bbfff700 1 -- 172.21.15.90:0/3917410498 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f36ac03d5d0 msgr2=0x7f36ac03fae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/71478430 2024-04-25T11:26:18.507 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:13.011+0000 7f36bbfff700 1 --2- 172.21.15.90:0/3917410498 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f36ac03d5d0 0x7f36ac03fae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T11:26:18.507 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:13.812+0000 7f36bbfff700 1 -- 172.21.15.90:0/3917410498 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f36ac03d5d0 msgr2=0x7f36ac03fae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/71478430 2024-04-25T11:26:18.507 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:13.812+0000 7f36bbfff700 1 --2- 172.21.15.90:0/3917410498 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f36ac03d5d0 0x7f36ac03fae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T11:26:18.508 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:15.413+0000 7f36bbfff700 1 -- 172.21.15.90:0/3917410498 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f36ac03d5d0 msgr2=0x7f36ac03fae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/71478430 2024-04-25T11:26:18.508 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:15.413+0000 7f36bbfff700 1 --2- 172.21.15.90:0/3917410498 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f36ac03d5d0 0x7f36ac03fae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T11:26:18.508 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:16.658+0000 7f36b9ffb700 1 -- 172.21.15.90:0/3917410498 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mgrmap(e 10) ==== 49296+0+0 (secure 0 0 0) 0x7f36b002cb00 con 0x7f36bc107fd0 2024-04-25T11:26:18.508 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:16.658+0000 7f36b9ffb700 1 -- 172.21.15.90:0/3917410498 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f36ac03d5d0 msgr2=0x7f36ac03fae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:26:18.508 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:16.658+0000 7f36b9ffb700 1 --2- 172.21.15.90:0/3917410498 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f36ac03d5d0 0x7f36ac03fae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:18.508 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:17.659+0000 7f36b9ffb700 1 -- 172.21.15.90:0/3917410498 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 11) ==== 49423+0+0 (secure 0 0 0) 0x7f36b0054a40 con 0x7f36bc107fd0 2024-04-25T11:26:18.508 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:17.659+0000 7f36b9ffb700 1 --2- 172.21.15.90:0/3917410498 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f36b00538a0 0x7f36ac042fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:18.508 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:17.659+0000 7f36b9ffb700 1 -- 172.21.15.90:0/3917410498 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f36ac0401f0 con 0x7f36b00538a0 2024-04-25T11:26:18.508 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:17.660+0000 7f36bbfff700 1 --2- 172.21.15.90:0/3917410498 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f36b00538a0 0x7f36ac042fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:18.508 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:17.660+0000 7f36bbfff700 1 --2- 172.21.15.90:0/3917410498 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f36b00538a0 0x7f36ac042fc0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f36a8003a10 tx=0x7f36a8009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:18.508 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:17.661+0000 7f36b9ffb700 1 -- 172.21.15.90:0/3917410498 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f36ac0401f0 con 0x7f36b00538a0 2024-04-25T11:26:18.508 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:17.669+0000 7f36c2fb2700 1 -- 172.21.15.90:0/3917410498 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f36bc064cf0 con 0x7f36b00538a0 2024-04-25T11:26:18.508 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:17.669+0000 7f36b9ffb700 1 -- 172.21.15.90:0/3917410498 <== mgr.14152 v2:172.21.15.90:6800/219449516 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f36bc064cf0 con 0x7f36b00538a0 2024-04-25T11:26:18.508 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:17.670+0000 7f36c2fb2700 1 -- 172.21.15.90:0/3917410498 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f36b00538a0 msgr2=0x7f36ac042fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:18.509 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:17.670+0000 7f36c2fb2700 1 --2- 172.21.15.90:0/3917410498 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f36b00538a0 0x7f36ac042fc0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f36a8003a10 tx=0x7f36a8009270 comp rx=0 tx=0).stop 2024-04-25T11:26:18.509 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:17.670+0000 7f36c2fb2700 1 -- 172.21.15.90:0/3917410498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36bc107fd0 msgr2=0x7f36bc10a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:18.509 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:17.670+0000 7f36c2fb2700 1 --2- 172.21.15.90:0/3917410498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36bc107fd0 0x7f36bc10a750 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f36b0004660 tx=0x7f36b0011aa0 comp rx=0 tx=0).stop 2024-04-25T11:26:18.509 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:17.670+0000 7f36c2fb2700 1 -- 172.21.15.90:0/3917410498 shutdown_connections 2024-04-25T11:26:18.509 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:17.670+0000 7f36c2fb2700 1 --2- 172.21.15.90:0/3917410498 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f36b00538a0 0x7f36ac042fc0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:18.509 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:17.670+0000 7f36c2fb2700 1 --2- 172.21.15.90:0/3917410498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36bc107fd0 0x7f36bc10a750 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:18.509 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:17.670+0000 7f36c2fb2700 1 --2- 172.21.15.90:0/3917410498 >> [v2:172.21.15.90:6800/71478430,v1:172.21.15.90:6801/71478430] conn(0x7f36ac03d5d0 0x7f36ac03fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:18.509 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:17.670+0000 7f36c2fb2700 1 -- 172.21.15.90:0/3917410498 >> 172.21.15.90:0/3917410498 conn(0x7f36bc103880 msgr2=0x7f36bc1041b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:18.509 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:17.670+0000 7f36c2fb2700 1 -- 172.21.15.90:0/3917410498 shutdown_connections 2024-04-25T11:26:18.509 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:17.670+0000 7f36c2fb2700 1 -- 172.21.15.90:0/3917410498 wait complete. 2024-04-25T11:26:18.509 INFO:teuthology.orchestra.run.smithi090.stdout:mgr epoch 9 is available 2024-04-25T11:26:18.509 INFO:teuthology.orchestra.run.smithi090.stdout:Generating a dashboard self-signed certificate... 2024-04-25T11:26:18.850 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:18 smithi090 ceph-mon[29575]: [25/Apr/2024:11:26:17] ENGINE Bus STARTING 2024-04-25T11:26:18.850 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:18 smithi090 ceph-mon[29575]: [25/Apr/2024:11:26:17] ENGINE Serving on https://172.21.15.90:7150 2024-04-25T11:26:18.850 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:18 smithi090 ceph-mon[29575]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T11:26:18.850 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:18 smithi090 ceph-mon[29575]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T11:26:18.850 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:18 smithi090 ceph-mon[29575]: [25/Apr/2024:11:26:17] ENGINE Serving on http://172.21.15.90:8765 2024-04-25T11:26:18.850 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:18 smithi090 ceph-mon[29575]: [25/Apr/2024:11:26:17] ENGINE Bus STARTED 2024-04-25T11:26:18.850 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:18 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:19.887 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-25T11:26:19.888 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.970+0000 7f4bec1ec700 1 Processor -- start 2024-04-25T11:26:19.888 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.970+0000 7f4bec1ec700 1 -- start start 2024-04-25T11:26:19.888 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.970+0000 7f4bec1ec700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4be407ccc0 0x7f4be410c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:19.888 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.970+0000 7f4bec1ec700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4be410cd40 con 0x7f4be407ccc0 2024-04-25T11:26:19.888 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.970+0000 7f4be9f88700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4be407ccc0 0x7f4be410c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:19.888 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.970+0000 7f4be9f88700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4be407ccc0 0x7f4be410c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52398/0 (socket says 172.21.15.90:52398) 2024-04-25T11:26:19.888 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.970+0000 7f4be9f88700 1 -- 172.21.15.90:0/1362896991 learned_addr learned my addr 172.21.15.90:0/1362896991 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:19.888 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.971+0000 7f4be9f88700 1 -- 172.21.15.90:0/1362896991 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be410cec0 con 0x7f4be407ccc0 2024-04-25T11:26:19.888 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.971+0000 7f4be9f88700 1 --2- 172.21.15.90:0/1362896991 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4be407ccc0 0x7f4be410c730 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4be00098a0 tx=0x7f4be0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fecdc3c6f2d0e93 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:19.888 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.971+0000 7f4be8f86700 1 -- 172.21.15.90:0/1362896991 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4be0004160 con 0x7f4be407ccc0 2024-04-25T11:26:19.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.971+0000 7f4be8f86700 1 -- 172.21.15.90:0/1362896991 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4be0004300 con 0x7f4be407ccc0 2024-04-25T11:26:19.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.971+0000 7f4be8f86700 1 -- 172.21.15.90:0/1362896991 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4be0011a20 con 0x7f4be407ccc0 2024-04-25T11:26:19.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.972+0000 7f4bec1ec700 1 -- 172.21.15.90:0/1362896991 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4be407ccc0 msgr2=0x7f4be410c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:19.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.972+0000 7f4bec1ec700 1 --2- 172.21.15.90:0/1362896991 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4be407ccc0 0x7f4be410c730 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4be00098a0 tx=0x7f4be0009bb0 comp rx=0 tx=0).stop 2024-04-25T11:26:19.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.972+0000 7f4bec1ec700 1 -- 172.21.15.90:0/1362896991 shutdown_connections 2024-04-25T11:26:19.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.972+0000 7f4bec1ec700 1 --2- 172.21.15.90:0/1362896991 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4be407ccc0 0x7f4be410c730 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:19.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.972+0000 7f4bec1ec700 1 -- 172.21.15.90:0/1362896991 >> 172.21.15.90:0/1362896991 conn(0x7f4be4104140 msgr2=0x7f4be41065f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:19.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.972+0000 7f4bec1ec700 1 -- 172.21.15.90:0/1362896991 shutdown_connections 2024-04-25T11:26:19.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.972+0000 7f4bec1ec700 1 -- 172.21.15.90:0/1362896991 wait complete. 2024-04-25T11:26:19.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.973+0000 7f4bec1ec700 1 Processor -- start 2024-04-25T11:26:19.889 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.973+0000 7f4bec1ec700 1 -- start start 2024-04-25T11:26:19.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.973+0000 7f4bec1ec700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4be407ccc0 0x7f4be419df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:19.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.973+0000 7f4bec1ec700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4be410df80 con 0x7f4be407ccc0 2024-04-25T11:26:19.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.973+0000 7f4be9f88700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4be407ccc0 0x7f4be419df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:19.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.973+0000 7f4be9f88700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4be407ccc0 0x7f4be419df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52410/0 (socket says 172.21.15.90:52410) 2024-04-25T11:26:19.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.973+0000 7f4be9f88700 1 -- 172.21.15.90:0/3155865459 learned_addr learned my addr 172.21.15.90:0/3155865459 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:19.890 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.973+0000 7f4be9f88700 1 -- 172.21.15.90:0/3155865459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be419e4a0 con 0x7f4be407ccc0 2024-04-25T11:26:19.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.973+0000 7f4be9f88700 1 --2- 172.21.15.90:0/3155865459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4be407ccc0 0x7f4be419df60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4be0004660 tx=0x7f4be0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:19.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.974+0000 7f4bdaffd700 1 -- 172.21.15.90:0/3155865459 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4be0003a60 con 0x7f4be407ccc0 2024-04-25T11:26:19.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.974+0000 7f4bec1ec700 1 -- 172.21.15.90:0/3155865459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4be419e730 con 0x7f4be407ccc0 2024-04-25T11:26:19.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.974+0000 7f4bdaffd700 1 -- 172.21.15.90:0/3155865459 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4be001b8f0 con 0x7f4be407ccc0 2024-04-25T11:26:19.891 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.974+0000 7f4bdaffd700 1 -- 172.21.15.90:0/3155865459 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4be0023890 con 0x7f4be407ccc0 2024-04-25T11:26:19.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.974+0000 7f4bec1ec700 1 -- 172.21.15.90:0/3155865459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4be41a1340 con 0x7f4be407ccc0 2024-04-25T11:26:19.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.975+0000 7f4bec1ec700 1 -- 172.21.15.90:0/3155865459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bc8005360 con 0x7f4be407ccc0 2024-04-25T11:26:19.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.975+0000 7f4bdaffd700 1 -- 172.21.15.90:0/3155865459 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 11) ==== 49423+0+0 (secure 0 0 0) 0x7f4be002c3f0 con 0x7f4be407ccc0 2024-04-25T11:26:19.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.975+0000 7f4bdaffd700 1 --2- 172.21.15.90:0/3155865459 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f4bd003d460 0x7f4bd003f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:19.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.975+0000 7f4bdaffd700 1 -- 172.21.15.90:0/3155865459 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4be0052cf0 con 0x7f4be407ccc0 2024-04-25T11:26:19.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.975+0000 7f4be9787700 1 --2- 172.21.15.90:0/3155865459 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f4bd003d460 0x7f4bd003f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:19.892 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.976+0000 7f4be9787700 1 --2- 172.21.15.90:0/3155865459 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f4bd003d460 0x7f4bd003f970 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4006fd0 tx=0x7f4bd4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:19.893 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:18.981+0000 7f4bdaffd700 1 -- 172.21.15.90:0/3155865459 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4be001f030 con 0x7f4be407ccc0 2024-04-25T11:26:19.893 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:19.192+0000 7f4bec1ec700 1 -- 172.21.15.90:0/3155865459 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f4bc8000bf0 con 0x7f4bd003d460 2024-04-25T11:26:19.893 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:19.220+0000 7f4bdaffd700 1 -- 172.21.15.90:0/3155865459 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 12) ==== 49529+0+0 (secure 0 0 0) 0x7f4be002cb00 con 0x7f4be407ccc0 2024-04-25T11:26:19.893 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:19.362+0000 7f4bdaffd700 1 -- 172.21.15.90:0/3155865459 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f4bc8000bf0 con 0x7f4bd003d460 2024-04-25T11:26:19.893 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:19.364+0000 7f4bec1ec700 1 -- 172.21.15.90:0/3155865459 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f4bd003d460 msgr2=0x7f4bd003f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:19.893 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:19.364+0000 7f4bec1ec700 1 --2- 172.21.15.90:0/3155865459 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f4bd003d460 0x7f4bd003f970 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4006fd0 tx=0x7f4bd4006e40 comp rx=0 tx=0).stop 2024-04-25T11:26:19.894 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:19.364+0000 7f4bec1ec700 1 -- 172.21.15.90:0/3155865459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4be407ccc0 msgr2=0x7f4be419df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:19.894 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:19.364+0000 7f4bec1ec700 1 --2- 172.21.15.90:0/3155865459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4be407ccc0 0x7f4be419df60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4be0004660 tx=0x7f4be0011aa0 comp rx=0 tx=0).stop 2024-04-25T11:26:19.894 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:19.364+0000 7f4bec1ec700 1 -- 172.21.15.90:0/3155865459 shutdown_connections 2024-04-25T11:26:19.894 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:19.364+0000 7f4bec1ec700 1 --2- 172.21.15.90:0/3155865459 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f4bd003d460 0x7f4bd003f970 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:19.894 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:19.364+0000 7f4bec1ec700 1 --2- 172.21.15.90:0/3155865459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4be407ccc0 0x7f4be419df60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:19.894 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:19.364+0000 7f4bec1ec700 1 -- 172.21.15.90:0/3155865459 >> 172.21.15.90:0/3155865459 conn(0x7f4be4104140 msgr2=0x7f4be4104a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:19.895 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:19.365+0000 7f4bec1ec700 1 -- 172.21.15.90:0/3155865459 shutdown_connections 2024-04-25T11:26:19.895 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:19.365+0000 7f4bec1ec700 1 -- 172.21.15.90:0/3155865459 wait complete. 2024-04-25T11:26:19.895 INFO:teuthology.orchestra.run.smithi090.stdout:Creating initial admin user... 2024-04-25T11:26:20.309 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:20 smithi090 ceph-mon[29575]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:26:20.309 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:20 smithi090 ceph-mon[29575]: mgrmap e12: x(active, since 2s) 2024-04-25T11:26:20.309 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:20 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:20.309 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:20 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:20.309 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:20 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:21.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:21 smithi090 ceph-mon[29575]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:26:21.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:21 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:21.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$SAqMEsZ/gmXRlXyfGvxTveVpkwEh93O9hfSkj8M7q3OTjBM94Tsfe", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714044380, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-25T11:26:21.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.321+0000 7fd6eebe9700 1 Processor -- start 2024-04-25T11:26:21.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.321+0000 7fd6eebe9700 1 -- start start 2024-04-25T11:26:21.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.321+0000 7fd6eebe9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6e8108ad0 0x7fd6e8108ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:21.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.321+0000 7fd6eebe9700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e81094d0 con 0x7fd6e8108ad0 2024-04-25T11:26:21.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.321+0000 7fd6ec985700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6e8108ad0 0x7fd6e8108ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:21.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.321+0000 7fd6ec985700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6e8108ad0 0x7fd6e8108ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52418/0 (socket says 172.21.15.90:52418) 2024-04-25T11:26:21.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.321+0000 7fd6ec985700 1 -- 172.21.15.90:0/930308670 learned_addr learned my addr 172.21.15.90:0/930308670 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:21.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.322+0000 7fd6ec985700 1 -- 172.21.15.90:0/930308670 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6e8109c00 con 0x7fd6e8108ad0 2024-04-25T11:26:21.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.322+0000 7fd6ec985700 1 --2- 172.21.15.90:0/930308670 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6e8108ad0 0x7fd6e8108ee0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc0098a0 tx=0x7fd6dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba874f84a8d0d7f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:21.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.322+0000 7fd6e77fe700 1 -- 172.21.15.90:0/930308670 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6dc004160 con 0x7fd6e8108ad0 2024-04-25T11:26:21.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.322+0000 7fd6e77fe700 1 -- 172.21.15.90:0/930308670 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6dc004300 con 0x7fd6e8108ad0 2024-04-25T11:26:21.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.322+0000 7fd6e77fe700 1 -- 172.21.15.90:0/930308670 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6dc011a20 con 0x7fd6e8108ad0 2024-04-25T11:26:21.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.323+0000 7fd6eebe9700 1 -- 172.21.15.90:0/930308670 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6e8108ad0 msgr2=0x7fd6e8108ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:21.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.323+0000 7fd6eebe9700 1 --2- 172.21.15.90:0/930308670 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6e8108ad0 0x7fd6e8108ee0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc0098a0 tx=0x7fd6dc009bb0 comp rx=0 tx=0).stop 2024-04-25T11:26:21.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.323+0000 7fd6eebe9700 1 -- 172.21.15.90:0/930308670 shutdown_connections 2024-04-25T11:26:21.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.323+0000 7fd6eebe9700 1 --2- 172.21.15.90:0/930308670 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6e8108ad0 0x7fd6e8108ee0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:21.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.323+0000 7fd6eebe9700 1 -- 172.21.15.90:0/930308670 >> 172.21.15.90:0/930308670 conn(0x7fd6e8104110 msgr2=0x7fd6e81065c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:21.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.323+0000 7fd6eebe9700 1 -- 172.21.15.90:0/930308670 shutdown_connections 2024-04-25T11:26:21.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.323+0000 7fd6eebe9700 1 -- 172.21.15.90:0/930308670 wait complete. 2024-04-25T11:26:21.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.324+0000 7fd6eebe9700 1 Processor -- start 2024-04-25T11:26:21.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.324+0000 7fd6eebe9700 1 -- start start 2024-04-25T11:26:21.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.324+0000 7fd6eebe9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6e8108ad0 0x7fd6e819dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:21.488 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.324+0000 7fd6eebe9700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e810cfb0 con 0x7fd6e8108ad0 2024-04-25T11:26:21.489 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.324+0000 7fd6ec985700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6e8108ad0 0x7fd6e819dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:21.489 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.324+0000 7fd6ec985700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6e8108ad0 0x7fd6e819dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52428/0 (socket says 172.21.15.90:52428) 2024-04-25T11:26:21.489 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.324+0000 7fd6ec985700 1 -- 172.21.15.90:0/148964245 learned_addr learned my addr 172.21.15.90:0/148964245 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:21.489 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.324+0000 7fd6ec985700 1 -- 172.21.15.90:0/148964245 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6e819e4f0 con 0x7fd6e8108ad0 2024-04-25T11:26:21.489 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.325+0000 7fd6ec985700 1 --2- 172.21.15.90:0/148964245 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6e8108ad0 0x7fd6e819dfb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc009130 tx=0x7fd6dc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:21.489 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.325+0000 7fd6e5ffb700 1 -- 172.21.15.90:0/148964245 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6dc003aa0 con 0x7fd6e8108ad0 2024-04-25T11:26:21.489 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.325+0000 7fd6e5ffb700 1 -- 172.21.15.90:0/148964245 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6dc01a8e0 con 0x7fd6e8108ad0 2024-04-25T11:26:21.489 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.325+0000 7fd6eebe9700 1 -- 172.21.15.90:0/148964245 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6e819e780 con 0x7fd6e8108ad0 2024-04-25T11:26:21.489 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.325+0000 7fd6e5ffb700 1 -- 172.21.15.90:0/148964245 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6dc0248f0 con 0x7fd6e8108ad0 2024-04-25T11:26:21.489 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.325+0000 7fd6eebe9700 1 -- 172.21.15.90:0/148964245 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6e81a1390 con 0x7fd6e8108ad0 2024-04-25T11:26:21.489 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.326+0000 7fd6e5ffb700 1 -- 172.21.15.90:0/148964245 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 12) ==== 49529+0+0 (secure 0 0 0) 0x7fd6dc022020 con 0x7fd6e8108ad0 2024-04-25T11:26:21.489 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.326+0000 7fd6e5ffb700 1 --2- 172.21.15.90:0/148964245 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd6d803d620 0x7fd6d803fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:21.489 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.326+0000 7fd6eebe9700 1 -- 172.21.15.90:0/148964245 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6e8064cf0 con 0x7fd6e8108ad0 2024-04-25T11:26:21.489 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.326+0000 7fd6e5ffb700 1 -- 172.21.15.90:0/148964245 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd6dc0536e0 con 0x7fd6e8108ad0 2024-04-25T11:26:21.490 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.327+0000 7fd6e7fff700 1 --2- 172.21.15.90:0/148964245 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd6d803d620 0x7fd6d803fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:21.490 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.327+0000 7fd6e7fff700 1 --2- 172.21.15.90:0/148964245 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd6d803d620 0x7fd6d803fb30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4006fd0 tx=0x7fd6d4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:21.490 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.335+0000 7fd6e5ffb700 1 -- 172.21.15.90:0/148964245 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd6dc029df0 con 0x7fd6e8108ad0 2024-04-25T11:26:21.490 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.570+0000 7fd6eebe9700 1 -- 172.21.15.90:0/148964245 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fd6e802e760 con 0x7fd6d803d620 2024-04-25T11:26:21.490 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.794+0000 7fd6e5ffb700 1 -- 172.21.15.90:0/148964245 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fd6e802e760 con 0x7fd6d803d620 2024-04-25T11:26:21.490 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.796+0000 7fd6eebe9700 1 -- 172.21.15.90:0/148964245 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd6d803d620 msgr2=0x7fd6d803fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:21.490 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.796+0000 7fd6eebe9700 1 --2- 172.21.15.90:0/148964245 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd6d803d620 0x7fd6d803fb30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4006fd0 tx=0x7fd6d4006e40 comp rx=0 tx=0).stop 2024-04-25T11:26:21.490 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.796+0000 7fd6eebe9700 1 -- 172.21.15.90:0/148964245 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6e8108ad0 msgr2=0x7fd6e819dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:21.490 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.796+0000 7fd6eebe9700 1 --2- 172.21.15.90:0/148964245 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6e8108ad0 0x7fd6e819dfb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc009130 tx=0x7fd6dc011aa0 comp rx=0 tx=0).stop 2024-04-25T11:26:21.490 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.796+0000 7fd6eebe9700 1 -- 172.21.15.90:0/148964245 shutdown_connections 2024-04-25T11:26:21.490 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.796+0000 7fd6eebe9700 1 --2- 172.21.15.90:0/148964245 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd6d803d620 0x7fd6d803fb30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:21.490 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.796+0000 7fd6eebe9700 1 --2- 172.21.15.90:0/148964245 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6e8108ad0 0x7fd6e819dfb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:21.490 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.796+0000 7fd6eebe9700 1 -- 172.21.15.90:0/148964245 >> 172.21.15.90:0/148964245 conn(0x7fd6e8104110 msgr2=0x7fd6e8194560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:21.491 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.797+0000 7fd6eebe9700 1 -- 172.21.15.90:0/148964245 shutdown_connections 2024-04-25T11:26:21.491 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:20.797+0000 7fd6eebe9700 1 -- 172.21.15.90:0/148964245 wait complete. 2024-04-25T11:26:21.491 INFO:teuthology.orchestra.run.smithi090.stdout:Fetching dashboard port number... 2024-04-25T11:26:22.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:22 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2927039474' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-25T11:26:22.940 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 8443 2024-04-25T11:26:22.940 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.927+0000 7fb39cbea700 1 Processor -- start 2024-04-25T11:26:22.940 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.928+0000 7fb39cbea700 1 -- start start 2024-04-25T11:26:22.940 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.928+0000 7fb39cbea700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb39807ccc0 0x7fb39810c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:22.940 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.928+0000 7fb39cbea700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb39810ccf0 con 0x7fb39807ccc0 2024-04-25T11:26:22.940 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.928+0000 7fb39659c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb39807ccc0 0x7fb39810c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:22.940 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.928+0000 7fb39659c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb39807ccc0 0x7fb39810c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46070/0 (socket says 172.21.15.90:46070) 2024-04-25T11:26:22.940 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.928+0000 7fb39659c700 1 -- 172.21.15.90:0/2031446240 learned_addr learned my addr 172.21.15.90:0/2031446240 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:22.940 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.928+0000 7fb39659c700 1 -- 172.21.15.90:0/2031446240 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb39810ce70 con 0x7fb39807ccc0 2024-04-25T11:26:22.940 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.929+0000 7fb39659c700 1 --2- 172.21.15.90:0/2031446240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb39807ccc0 0x7fb39810c6e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb3880098a0 tx=0x7fb388009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e09c2cc2d98ab2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:22.940 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.929+0000 7fb39559a700 1 -- 172.21.15.90:0/2031446240 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb388004160 con 0x7fb39807ccc0 2024-04-25T11:26:22.941 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.929+0000 7fb39559a700 1 -- 172.21.15.90:0/2031446240 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb388004300 con 0x7fb39807ccc0 2024-04-25T11:26:22.941 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.929+0000 7fb39559a700 1 -- 172.21.15.90:0/2031446240 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb388011a20 con 0x7fb39807ccc0 2024-04-25T11:26:22.941 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.930+0000 7fb39cbea700 1 -- 172.21.15.90:0/2031446240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb39807ccc0 msgr2=0x7fb39810c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:22.941 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.930+0000 7fb39cbea700 1 --2- 172.21.15.90:0/2031446240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb39807ccc0 0x7fb39810c6e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb3880098a0 tx=0x7fb388009bb0 comp rx=0 tx=0).stop 2024-04-25T11:26:22.941 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.930+0000 7fb39cbea700 1 -- 172.21.15.90:0/2031446240 shutdown_connections 2024-04-25T11:26:22.941 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.930+0000 7fb39cbea700 1 --2- 172.21.15.90:0/2031446240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb39807ccc0 0x7fb39810c6e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:22.941 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.930+0000 7fb39cbea700 1 -- 172.21.15.90:0/2031446240 >> 172.21.15.90:0/2031446240 conn(0x7fb3981040f0 msgr2=0x7fb3981065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:22.941 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.930+0000 7fb39cbea700 1 -- 172.21.15.90:0/2031446240 shutdown_connections 2024-04-25T11:26:22.941 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.930+0000 7fb39cbea700 1 -- 172.21.15.90:0/2031446240 wait complete. 2024-04-25T11:26:22.941 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.931+0000 7fb39cbea700 1 Processor -- start 2024-04-25T11:26:22.941 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.931+0000 7fb39cbea700 1 -- start start 2024-04-25T11:26:22.941 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.931+0000 7fb39cbea700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb39807ccc0 0x7fb39819df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:22.941 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.931+0000 7fb39cbea700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb39810df30 con 0x7fb39807ccc0 2024-04-25T11:26:22.941 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.931+0000 7fb39659c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb39807ccc0 0x7fb39819df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:22.942 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.931+0000 7fb39659c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb39807ccc0 0x7fb39819df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46076/0 (socket says 172.21.15.90:46076) 2024-04-25T11:26:22.942 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.931+0000 7fb39659c700 1 -- 172.21.15.90:0/2927039474 learned_addr learned my addr 172.21.15.90:0/2927039474 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:22.942 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.931+0000 7fb39659c700 1 -- 172.21.15.90:0/2927039474 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb39819e450 con 0x7fb39807ccc0 2024-04-25T11:26:22.942 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.931+0000 7fb39659c700 1 --2- 172.21.15.90:0/2927039474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb39807ccc0 0x7fb39819df10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb388004660 tx=0x7fb388011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:22.942 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.932+0000 7fb3877fe700 1 -- 172.21.15.90:0/2927039474 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb388003a60 con 0x7fb39807ccc0 2024-04-25T11:26:22.942 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.932+0000 7fb3877fe700 1 -- 172.21.15.90:0/2927039474 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb38801b8f0 con 0x7fb39807ccc0 2024-04-25T11:26:22.942 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.932+0000 7fb39cbea700 1 -- 172.21.15.90:0/2927039474 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb39819e6e0 con 0x7fb39807ccc0 2024-04-25T11:26:22.942 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.932+0000 7fb3877fe700 1 -- 172.21.15.90:0/2927039474 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb388023890 con 0x7fb39807ccc0 2024-04-25T11:26:22.942 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.932+0000 7fb39cbea700 1 -- 172.21.15.90:0/2927039474 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3981a12f0 con 0x7fb39807ccc0 2024-04-25T11:26:22.942 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.933+0000 7fb39cbea700 1 -- 172.21.15.90:0/2927039474 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb398064cf0 con 0x7fb39807ccc0 2024-04-25T11:26:22.942 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.933+0000 7fb3877fe700 1 -- 172.21.15.90:0/2927039474 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 12) ==== 49529+0+0 (secure 0 0 0) 0x7fb38802c3f0 con 0x7fb39807ccc0 2024-04-25T11:26:22.942 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.933+0000 7fb3877fe700 1 --2- 172.21.15.90:0/2927039474 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb38003d5d0 0x7fb38003fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:22.942 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.933+0000 7fb3877fe700 1 -- 172.21.15.90:0/2927039474 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb388052fc0 con 0x7fb39807ccc0 2024-04-25T11:26:22.943 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.933+0000 7fb395d9b700 1 --2- 172.21.15.90:0/2927039474 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb38003d5d0 0x7fb38003fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:22.943 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.934+0000 7fb395d9b700 1 --2- 172.21.15.90:0/2927039474 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb38003d5d0 0x7fb38003fae0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb38c006fd0 tx=0x7fb38c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:22.943 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:21.939+0000 7fb3877fe700 1 -- 172.21.15.90:0/2927039474 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb38801f030 con 0x7fb39807ccc0 2024-04-25T11:26:22.943 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:22.148+0000 7fb39cbea700 1 -- 172.21.15.90:0/2927039474 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fb3981088e0 con 0x7fb39807ccc0 2024-04-25T11:26:22.943 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:22.148+0000 7fb3877fe700 1 -- 172.21.15.90:0/2927039474 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7fb38801bda0 con 0x7fb39807ccc0 2024-04-25T11:26:22.943 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:22.150+0000 7fb39cbea700 1 -- 172.21.15.90:0/2927039474 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb38003d5d0 msgr2=0x7fb38003fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:22.943 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:22.150+0000 7fb39cbea700 1 --2- 172.21.15.90:0/2927039474 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb38003d5d0 0x7fb38003fae0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb38c006fd0 tx=0x7fb38c006e40 comp rx=0 tx=0).stop 2024-04-25T11:26:22.943 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:22.150+0000 7fb39cbea700 1 -- 172.21.15.90:0/2927039474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb39807ccc0 msgr2=0x7fb39819df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:22.943 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:22.150+0000 7fb39cbea700 1 --2- 172.21.15.90:0/2927039474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb39807ccc0 0x7fb39819df10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb388004660 tx=0x7fb388011aa0 comp rx=0 tx=0).stop 2024-04-25T11:26:22.943 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:22.150+0000 7fb39cbea700 1 -- 172.21.15.90:0/2927039474 shutdown_connections 2024-04-25T11:26:22.943 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:22.150+0000 7fb39cbea700 1 --2- 172.21.15.90:0/2927039474 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb38003d5d0 0x7fb38003fae0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:22.943 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:22.150+0000 7fb39cbea700 1 --2- 172.21.15.90:0/2927039474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb39807ccc0 0x7fb39819df10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:22.944 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:22.151+0000 7fb39cbea700 1 -- 172.21.15.90:0/2927039474 >> 172.21.15.90:0/2927039474 conn(0x7fb3981040f0 msgr2=0x7fb3981049c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:22.944 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:22.151+0000 7fb39cbea700 1 -- 172.21.15.90:0/2927039474 shutdown_connections 2024-04-25T11:26:22.944 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:22.151+0000 7fb39cbea700 1 -- 172.21.15.90:0/2927039474 wait complete. 2024-04-25T11:26:22.946 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T11:26:22.946 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout disabled 2024-04-25T11:26:22.953 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T11:26:22.953 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-04-25T11:26:22.953 INFO:teuthology.orchestra.run.smithi090.stdout:firewalld.service is not enabled 2024-04-25T11:26:22.953 INFO:teuthology.orchestra.run.smithi090.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-25T11:26:22.954 INFO:teuthology.orchestra.run.smithi090.stdout:Ceph Dashboard is now available at: 2024-04-25T11:26:22.954 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:26:22.954 INFO:teuthology.orchestra.run.smithi090.stdout: URL: https://smithi090:8443/ 2024-04-25T11:26:22.954 INFO:teuthology.orchestra.run.smithi090.stdout: User: admin 2024-04-25T11:26:22.955 INFO:teuthology.orchestra.run.smithi090.stdout: Password: 30gsgk74g9 2024-04-25T11:26:22.955 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:26:22.955 INFO:teuthology.orchestra.run.smithi090.stdout:Saving cluster configuration to /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config directory 2024-04-25T11:26:22.955 INFO:teuthology.orchestra.run.smithi090.stdout:Enabling autotune for osd_memory_target 2024-04-25T11:26:24.198 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.417+0000 7f25f296c700 1 Processor -- start 2024-04-25T11:26:24.198 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.417+0000 7f25f296c700 1 -- start start 2024-04-25T11:26:24.198 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.417+0000 7f25f296c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25ec107ef0 0x7f25ec108300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:24.198 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.417+0000 7f25f296c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f25ec1088f0 con 0x7f25ec107ef0 2024-04-25T11:26:24.198 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.417+0000 7f25ebfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25ec107ef0 0x7f25ec108300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:24.199 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.417+0000 7f25ebfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25ec107ef0 0x7f25ec108300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46090/0 (socket says 172.21.15.90:46090) 2024-04-25T11:26:24.199 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.417+0000 7f25ebfff700 1 -- 172.21.15.90:0/2527461703 learned_addr learned my addr 172.21.15.90:0/2527461703 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:24.199 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.418+0000 7f25ebfff700 1 -- 172.21.15.90:0/2527461703 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25ec108a70 con 0x7f25ec107ef0 2024-04-25T11:26:24.199 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.418+0000 7f25ebfff700 1 --2- 172.21.15.90:0/2527461703 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25ec107ef0 0x7f25ec108300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f25dc0098a0 tx=0x7f25dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa1d2b56399f9d32 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:24.199 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.418+0000 7f25eaffd700 1 -- 172.21.15.90:0/2527461703 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25dc004160 con 0x7f25ec107ef0 2024-04-25T11:26:24.199 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.418+0000 7f25eaffd700 1 -- 172.21.15.90:0/2527461703 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25dc004300 con 0x7f25ec107ef0 2024-04-25T11:26:24.199 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.418+0000 7f25eaffd700 1 -- 172.21.15.90:0/2527461703 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25dc011a20 con 0x7f25ec107ef0 2024-04-25T11:26:24.199 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.419+0000 7f25f296c700 1 -- 172.21.15.90:0/2527461703 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25ec107ef0 msgr2=0x7f25ec108300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:24.199 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.419+0000 7f25f296c700 1 --2- 172.21.15.90:0/2527461703 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25ec107ef0 0x7f25ec108300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f25dc0098a0 tx=0x7f25dc009bb0 comp rx=0 tx=0).stop 2024-04-25T11:26:24.199 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.419+0000 7f25f296c700 1 -- 172.21.15.90:0/2527461703 shutdown_connections 2024-04-25T11:26:24.199 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.419+0000 7f25f296c700 1 --2- 172.21.15.90:0/2527461703 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25ec107ef0 0x7f25ec108300 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:24.199 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.419+0000 7f25f296c700 1 -- 172.21.15.90:0/2527461703 >> 172.21.15.90:0/2527461703 conn(0x7f25ec103870 msgr2=0x7f25ec105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:24.199 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.419+0000 7f25f296c700 1 -- 172.21.15.90:0/2527461703 shutdown_connections 2024-04-25T11:26:24.200 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.419+0000 7f25f296c700 1 -- 172.21.15.90:0/2527461703 wait complete. 2024-04-25T11:26:24.200 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.419+0000 7f25f296c700 1 Processor -- start 2024-04-25T11:26:24.200 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.419+0000 7f25f296c700 1 -- start start 2024-04-25T11:26:24.200 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.420+0000 7f25f296c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25ec107ef0 0x7f25ec1a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:24.200 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.420+0000 7f25f296c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f25ec10df60 con 0x7f25ec107ef0 2024-04-25T11:26:24.200 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.420+0000 7f25ebfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25ec107ef0 0x7f25ec1a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:24.200 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.420+0000 7f25ebfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25ec107ef0 0x7f25ec1a0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46104/0 (socket says 172.21.15.90:46104) 2024-04-25T11:26:24.200 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.420+0000 7f25ebfff700 1 -- 172.21.15.90:0/4026250869 learned_addr learned my addr 172.21.15.90:0/4026250869 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:24.200 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.420+0000 7f25ebfff700 1 -- 172.21.15.90:0/4026250869 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25ec1a06a0 con 0x7f25ec107ef0 2024-04-25T11:26:24.200 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.420+0000 7f25ebfff700 1 --2- 172.21.15.90:0/4026250869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25ec107ef0 0x7f25ec1a0160 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f25dc004660 tx=0x7f25dc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:24.200 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.420+0000 7f25e97fa700 1 -- 172.21.15.90:0/4026250869 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25dc003a60 con 0x7f25ec107ef0 2024-04-25T11:26:24.200 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.420+0000 7f25e97fa700 1 -- 172.21.15.90:0/4026250869 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25dc01b8f0 con 0x7f25ec107ef0 2024-04-25T11:26:24.200 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.420+0000 7f25e97fa700 1 -- 172.21.15.90:0/4026250869 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25dc023890 con 0x7f25ec107ef0 2024-04-25T11:26:24.200 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.421+0000 7f25f296c700 1 -- 172.21.15.90:0/4026250869 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25ec1a0930 con 0x7f25ec107ef0 2024-04-25T11:26:24.201 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.421+0000 7f25f296c700 1 -- 172.21.15.90:0/4026250869 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25ec1a3540 con 0x7f25ec107ef0 2024-04-25T11:26:24.201 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.421+0000 7f25e97fa700 1 -- 172.21.15.90:0/4026250869 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f25dc02c3f0 con 0x7f25ec107ef0 2024-04-25T11:26:24.201 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.422+0000 7f25e97fa700 1 --2- 172.21.15.90:0/4026250869 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f25d403d670 0x7f25d403fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:24.201 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.422+0000 7f25e97fa700 1 -- 172.21.15.90:0/4026250869 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f25dc053ef0 con 0x7f25ec107ef0 2024-04-25T11:26:24.201 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.422+0000 7f25f296c700 1 -- 172.21.15.90:0/4026250869 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25ec064cf0 con 0x7f25ec107ef0 2024-04-25T11:26:24.201 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.422+0000 7f25eb7fe700 1 --2- 172.21.15.90:0/4026250869 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f25d403d670 0x7f25d403fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:24.201 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.422+0000 7f25eb7fe700 1 --2- 172.21.15.90:0/4026250869 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f25d403d670 0x7f25d403fb80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f25e0006fd0 tx=0x7f25e0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:24.201 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.427+0000 7f25e97fa700 1 -- 172.21.15.90:0/4026250869 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f25dc01f030 con 0x7f25ec107ef0 2024-04-25T11:26:24.201 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.624+0000 7f25f296c700 1 -- 172.21.15.90:0/4026250869 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f25ec02e810 con 0x7f25ec107ef0 2024-04-25T11:26:24.201 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.625+0000 7f25e97fa700 1 -- 172.21.15.90:0/4026250869 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f25dc018e30 con 0x7f25ec107ef0 2024-04-25T11:26:24.201 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.626+0000 7f25f296c700 1 -- 172.21.15.90:0/4026250869 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f25d403d670 msgr2=0x7f25d403fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:24.201 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.626+0000 7f25f296c700 1 --2- 172.21.15.90:0/4026250869 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f25d403d670 0x7f25d403fb80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f25e0006fd0 tx=0x7f25e0006e40 comp rx=0 tx=0).stop 2024-04-25T11:26:24.201 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.626+0000 7f25f296c700 1 -- 172.21.15.90:0/4026250869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25ec107ef0 msgr2=0x7f25ec1a0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:24.202 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.626+0000 7f25f296c700 1 --2- 172.21.15.90:0/4026250869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25ec107ef0 0x7f25ec1a0160 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f25dc004660 tx=0x7f25dc011aa0 comp rx=0 tx=0).stop 2024-04-25T11:26:24.202 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.627+0000 7f25f296c700 1 -- 172.21.15.90:0/4026250869 shutdown_connections 2024-04-25T11:26:24.202 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.627+0000 7f25f296c700 1 --2- 172.21.15.90:0/4026250869 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f25d403d670 0x7f25d403fb80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:24.202 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.627+0000 7f25f296c700 1 --2- 172.21.15.90:0/4026250869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25ec107ef0 0x7f25ec1a0160 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:24.202 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.627+0000 7f25f296c700 1 -- 172.21.15.90:0/4026250869 >> 172.21.15.90:0/4026250869 conn(0x7f25ec103870 msgr2=0x7f25ec108f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:24.202 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.627+0000 7f25f296c700 1 -- 172.21.15.90:0/4026250869 shutdown_connections 2024-04-25T11:26:24.202 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:23.627+0000 7f25f296c700 1 -- 172.21.15.90:0/4026250869 wait complete. 2024-04-25T11:26:24.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:24 smithi090 ceph-mon[29575]: mgrmap e13: x(active, since 6s) 2024-04-25T11:26:25.500 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.647+0000 7f40dc1fe700 1 Processor -- start 2024-04-25T11:26:25.501 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.647+0000 7f40dc1fe700 1 -- start start 2024-04-25T11:26:25.501 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.647+0000 7f40dc1fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d410a6c0 0x7f40d410cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:25.501 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.647+0000 7f40dc1fe700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f40d407d660 con 0x7f40d410a6c0 2024-04-25T11:26:25.501 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.647+0000 7f40d9f9a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d410a6c0 0x7f40d410cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:25.501 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.648+0000 7f40d9f9a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d410a6c0 0x7f40d410cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46110/0 (socket says 172.21.15.90:46110) 2024-04-25T11:26:25.501 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.648+0000 7f40d9f9a700 1 -- 172.21.15.90:0/1098705901 learned_addr learned my addr 172.21.15.90:0/1098705901 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:25.501 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.648+0000 7f40d9f9a700 1 -- 172.21.15.90:0/1098705901 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d410d020 con 0x7f40d410a6c0 2024-04-25T11:26:25.501 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.648+0000 7f40d9f9a700 1 --2- 172.21.15.90:0/1098705901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d410a6c0 0x7f40d410cae0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f40c40098a0 tx=0x7f40c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4075a22453fa931f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:25.501 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.648+0000 7f40d8f98700 1 -- 172.21.15.90:0/1098705901 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40c4004160 con 0x7f40d410a6c0 2024-04-25T11:26:25.501 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.648+0000 7f40d8f98700 1 -- 172.21.15.90:0/1098705901 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40c4004300 con 0x7f40d410a6c0 2024-04-25T11:26:25.501 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.648+0000 7f40d8f98700 1 -- 172.21.15.90:0/1098705901 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40c4011a20 con 0x7f40d410a6c0 2024-04-25T11:26:25.502 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.649+0000 7f40dc1fe700 1 -- 172.21.15.90:0/1098705901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d410a6c0 msgr2=0x7f40d410cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:25.502 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.649+0000 7f40dc1fe700 1 --2- 172.21.15.90:0/1098705901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d410a6c0 0x7f40d410cae0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f40c40098a0 tx=0x7f40c4009bb0 comp rx=0 tx=0).stop 2024-04-25T11:26:25.502 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.649+0000 7f40dc1fe700 1 -- 172.21.15.90:0/1098705901 shutdown_connections 2024-04-25T11:26:25.502 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.649+0000 7f40dc1fe700 1 --2- 172.21.15.90:0/1098705901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d410a6c0 0x7f40d410cae0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:25.502 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.649+0000 7f40dc1fe700 1 -- 172.21.15.90:0/1098705901 >> 172.21.15.90:0/1098705901 conn(0x7f40d4104140 msgr2=0x7f40d41065d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:25.502 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.649+0000 7f40dc1fe700 1 -- 172.21.15.90:0/1098705901 shutdown_connections 2024-04-25T11:26:25.502 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.649+0000 7f40dc1fe700 1 -- 172.21.15.90:0/1098705901 wait complete. 2024-04-25T11:26:25.502 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.650+0000 7f40dc1fe700 1 Processor -- start 2024-04-25T11:26:25.502 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.650+0000 7f40dc1fe700 1 -- start start 2024-04-25T11:26:25.502 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.650+0000 7f40dc1fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d410a6c0 0x7f40d419df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:25.502 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.650+0000 7f40dc1fe700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f40d410df60 con 0x7f40d410a6c0 2024-04-25T11:26:25.503 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.650+0000 7f40d9f9a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d410a6c0 0x7f40d419df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:25.503 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.650+0000 7f40d9f9a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d410a6c0 0x7f40d419df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46112/0 (socket says 172.21.15.90:46112) 2024-04-25T11:26:25.503 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.650+0000 7f40d9f9a700 1 -- 172.21.15.90:0/1694487757 learned_addr learned my addr 172.21.15.90:0/1694487757 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:25.503 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.650+0000 7f40d9f9a700 1 -- 172.21.15.90:0/1694487757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d419e480 con 0x7f40d410a6c0 2024-04-25T11:26:25.503 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.650+0000 7f40d9f9a700 1 --2- 172.21.15.90:0/1694487757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d410a6c0 0x7f40d419df40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f40c4004660 tx=0x7f40c4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:25.503 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.651+0000 7f40caffd700 1 -- 172.21.15.90:0/1694487757 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40c4003a60 con 0x7f40d410a6c0 2024-04-25T11:26:25.503 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.651+0000 7f40caffd700 1 -- 172.21.15.90:0/1694487757 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40c401b8f0 con 0x7f40d410a6c0 2024-04-25T11:26:25.503 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.651+0000 7f40caffd700 1 -- 172.21.15.90:0/1694487757 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40c4023890 con 0x7f40d410a6c0 2024-04-25T11:26:25.503 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.651+0000 7f40dc1fe700 1 -- 172.21.15.90:0/1694487757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40d419e710 con 0x7f40d410a6c0 2024-04-25T11:26:25.503 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.651+0000 7f40dc1fe700 1 -- 172.21.15.90:0/1694487757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40d41a1320 con 0x7f40d410a6c0 2024-04-25T11:26:25.503 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.652+0000 7f40dc1fe700 1 -- 172.21.15.90:0/1694487757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40d4064cf0 con 0x7f40d410a6c0 2024-04-25T11:26:25.503 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.652+0000 7f40caffd700 1 -- 172.21.15.90:0/1694487757 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f40c402c3f0 con 0x7f40d410a6c0 2024-04-25T11:26:25.503 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.652+0000 7f40caffd700 1 --2- 172.21.15.90:0/1694487757 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f40c003d610 0x7f40c003fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:25.504 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.652+0000 7f40caffd700 1 -- 172.21.15.90:0/1694487757 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f40c4053150 con 0x7f40d410a6c0 2024-04-25T11:26:25.504 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.652+0000 7f40d9799700 1 --2- 172.21.15.90:0/1694487757 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f40c003d610 0x7f40c003fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:25.504 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.653+0000 7f40d9799700 1 --2- 172.21.15.90:0/1694487757 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f40c003d610 0x7f40c003fb20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f40d0006fd0 tx=0x7f40d0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:25.504 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.658+0000 7f40caffd700 1 -- 172.21.15.90:0/1694487757 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f40c401f030 con 0x7f40d410a6c0 2024-04-25T11:26:25.504 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.967+0000 7f40dc1fe700 1 -- 172.21.15.90:0/1694487757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f40d41a14f0 con 0x7f40d410a6c0 2024-04-25T11:26:25.504 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.971+0000 7f40caffd700 1 -- 172.21.15.90:0/1694487757 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f40c401bda0 con 0x7f40d410a6c0 2024-04-25T11:26:25.504 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.973+0000 7f40dc1fe700 1 -- 172.21.15.90:0/1694487757 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f40c003d610 msgr2=0x7f40c003fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:25.504 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.973+0000 7f40dc1fe700 1 --2- 172.21.15.90:0/1694487757 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f40c003d610 0x7f40c003fb20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f40d0006fd0 tx=0x7f40d0006e40 comp rx=0 tx=0).stop 2024-04-25T11:26:25.504 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.973+0000 7f40dc1fe700 1 -- 172.21.15.90:0/1694487757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d410a6c0 msgr2=0x7f40d419df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:25.504 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.973+0000 7f40dc1fe700 1 --2- 172.21.15.90:0/1694487757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d410a6c0 0x7f40d419df40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f40c4004660 tx=0x7f40c4011aa0 comp rx=0 tx=0).stop 2024-04-25T11:26:25.504 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.973+0000 7f40dc1fe700 1 -- 172.21.15.90:0/1694487757 shutdown_connections 2024-04-25T11:26:25.504 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.973+0000 7f40dc1fe700 1 --2- 172.21.15.90:0/1694487757 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f40c003d610 0x7f40c003fb20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:25.504 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.973+0000 7f40dc1fe700 1 --2- 172.21.15.90:0/1694487757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d410a6c0 0x7f40d419df40 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:25.505 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.973+0000 7f40dc1fe700 1 -- 172.21.15.90:0/1694487757 >> 172.21.15.90:0/1694487757 conn(0x7f40d4104140 msgr2=0x7f40d4104a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:25.505 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.973+0000 7f40dc1fe700 1 -- 172.21.15.90:0/1694487757 shutdown_connections 2024-04-25T11:26:25.505 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-25T11:26:24.973+0000 7f40dc1fe700 1 -- 172.21.15.90:0/1694487757 wait complete. 2024-04-25T11:26:25.505 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-25T11:26:25.505 INFO:teuthology.orchestra.run.smithi090.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-25T11:26:25.505 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:26:25.505 INFO:teuthology.orchestra.run.smithi090.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-25T11:26:25.505 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:26:25.505 INFO:teuthology.orchestra.run.smithi090.stdout:Or, if you are only running a single cluster on this host: 2024-04-25T11:26:25.505 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:26:25.505 INFO:teuthology.orchestra.run.smithi090.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-25T11:26:25.505 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:26:25.505 INFO:teuthology.orchestra.run.smithi090.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-25T11:26:25.505 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:26:25.506 INFO:teuthology.orchestra.run.smithi090.stdout: ceph telemetry on 2024-04-25T11:26:25.506 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:26:25.506 INFO:teuthology.orchestra.run.smithi090.stdout:For more information see: 2024-04-25T11:26:25.506 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:26:25.506 INFO:teuthology.orchestra.run.smithi090.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-25T11:26:25.506 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:26:25.506 INFO:teuthology.orchestra.run.smithi090.stdout:Bootstrap complete. 2024-04-25T11:26:25.536 INFO:tasks.cephadm:Fetching config... 2024-04-25T11:26:25.536 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T11:26:25.536 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-25T11:26:25.556 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-25T11:26:25.556 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T11:26:25.556 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-25T11:26:25.614 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-25T11:26:25.615 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T11:26:25.615 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-25T11:26:25.682 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-25T11:26:25.682 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T11:26:25.682 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-25T11:26:25.740 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-25T11:26:25.741 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYV47YYzGnTOax+/+SgGFXLHEkYg2jRFr7y3yEYyUuaCNtVbo/FOEE46qPqip6RQvVXFyfx0ELTPx7sx7l63drWF/qj9Jct6uSOou2aXWwv0nVU+f3k6cBGeAqj71yV7pZjdQdh/dQrB1yAKJJ2ON+liNidCIe8Eg5yu+FnUMbSs533rUHuVxSZfk4H9EVqWg9InUu0OZNWScK13phuWYJ9o6KZcvXQ3P0AlHSUC6dE6EStF4RaKfeuRgRgoIj1Le6wjeTw4v6EhvGE/JvIOUUgNlJiYMkejnv0M6h+fOQKxKZB6dokRlAYHm+Duqz4Q+STnMz6ab50DCSaqWq+6A43xrhET7xfgyge32uyjt7Q7jDm4qNI6WN3iANEQwRhXj8AhD7BO2s1dKy01DzCLqDbgdvuD3B/Do68KgnM9Xj2ygycGrbzljDzZ0rW3Lz3Vz29OwbQ+ETmkTxydlsz/gfUDzzLd52jRb8Ef7UOw+YOXPnpMP1ln2KZN0YKqDXS9k= ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T11:26:25.819 INFO:teuthology.orchestra.run.smithi090.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYV47YYzGnTOax+/+SgGFXLHEkYg2jRFr7y3yEYyUuaCNtVbo/FOEE46qPqip6RQvVXFyfx0ELTPx7sx7l63drWF/qj9Jct6uSOou2aXWwv0nVU+f3k6cBGeAqj71yV7pZjdQdh/dQrB1yAKJJ2ON+liNidCIe8Eg5yu+FnUMbSs533rUHuVxSZfk4H9EVqWg9InUu0OZNWScK13phuWYJ9o6KZcvXQ3P0AlHSUC6dE6EStF4RaKfeuRgRgoIj1Le6wjeTw4v6EhvGE/JvIOUUgNlJiYMkejnv0M6h+fOQKxKZB6dokRlAYHm+Duqz4Q+STnMz6ab50DCSaqWq+6A43xrhET7xfgyge32uyjt7Q7jDm4qNI6WN3iANEQwRhXj8AhD7BO2s1dKy01DzCLqDbgdvuD3B/Do68KgnM9Xj2ygycGrbzljDzZ0rW3Lz3Vz29OwbQ+ETmkTxydlsz/gfUDzzLd52jRb8Ef7UOw+YOXPnpMP1ln2KZN0YKqDXS9k= ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:26:25.829 DEBUG:teuthology.orchestra.run.smithi138:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYV47YYzGnTOax+/+SgGFXLHEkYg2jRFr7y3yEYyUuaCNtVbo/FOEE46qPqip6RQvVXFyfx0ELTPx7sx7l63drWF/qj9Jct6uSOou2aXWwv0nVU+f3k6cBGeAqj71yV7pZjdQdh/dQrB1yAKJJ2ON+liNidCIe8Eg5yu+FnUMbSs533rUHuVxSZfk4H9EVqWg9InUu0OZNWScK13phuWYJ9o6KZcvXQ3P0AlHSUC6dE6EStF4RaKfeuRgRgoIj1Le6wjeTw4v6EhvGE/JvIOUUgNlJiYMkejnv0M6h+fOQKxKZB6dokRlAYHm+Duqz4Q+STnMz6ab50DCSaqWq+6A43xrhET7xfgyge32uyjt7Q7jDm4qNI6WN3iANEQwRhXj8AhD7BO2s1dKy01DzCLqDbgdvuD3B/Do68KgnM9Xj2ygycGrbzljDzZ0rW3Lz3Vz29OwbQ+ETmkTxydlsz/gfUDzzLd52jRb8Ef7UOw+YOXPnpMP1ln2KZN0YKqDXS9k= ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T11:26:25.869 INFO:teuthology.orchestra.run.smithi138.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYV47YYzGnTOax+/+SgGFXLHEkYg2jRFr7y3yEYyUuaCNtVbo/FOEE46qPqip6RQvVXFyfx0ELTPx7sx7l63drWF/qj9Jct6uSOou2aXWwv0nVU+f3k6cBGeAqj71yV7pZjdQdh/dQrB1yAKJJ2ON+liNidCIe8Eg5yu+FnUMbSs533rUHuVxSZfk4H9EVqWg9InUu0OZNWScK13phuWYJ9o6KZcvXQ3P0AlHSUC6dE6EStF4RaKfeuRgRgoIj1Le6wjeTw4v6EhvGE/JvIOUUgNlJiYMkejnv0M6h+fOQKxKZB6dokRlAYHm+Duqz4Q+STnMz6ab50DCSaqWq+6A43xrhET7xfgyge32uyjt7Q7jDm4qNI6WN3iANEQwRhXj8AhD7BO2s1dKy01DzCLqDbgdvuD3B/Do68KgnM9Xj2ygycGrbzljDzZ0rW3Lz3Vz29OwbQ+ETmkTxydlsz/gfUDzzLd52jRb8Ef7UOw+YOXPnpMP1ln2KZN0YKqDXS9k= ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:26:25.881 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYV47YYzGnTOax+/+SgGFXLHEkYg2jRFr7y3yEYyUuaCNtVbo/FOEE46qPqip6RQvVXFyfx0ELTPx7sx7l63drWF/qj9Jct6uSOou2aXWwv0nVU+f3k6cBGeAqj71yV7pZjdQdh/dQrB1yAKJJ2ON+liNidCIe8Eg5yu+FnUMbSs533rUHuVxSZfk4H9EVqWg9InUu0OZNWScK13phuWYJ9o6KZcvXQ3P0AlHSUC6dE6EStF4RaKfeuRgRgoIj1Le6wjeTw4v6EhvGE/JvIOUUgNlJiYMkejnv0M6h+fOQKxKZB6dokRlAYHm+Duqz4Q+STnMz6ab50DCSaqWq+6A43xrhET7xfgyge32uyjt7Q7jDm4qNI6WN3iANEQwRhXj8AhD7BO2s1dKy01DzCLqDbgdvuD3B/Do68KgnM9Xj2ygycGrbzljDzZ0rW3Lz3Vz29OwbQ+ETmkTxydlsz/gfUDzzLd52jRb8Ef7UOw+YOXPnpMP1ln2KZN0YKqDXS9k= ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T11:26:25.922 INFO:teuthology.orchestra.run.smithi156.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYV47YYzGnTOax+/+SgGFXLHEkYg2jRFr7y3yEYyUuaCNtVbo/FOEE46qPqip6RQvVXFyfx0ELTPx7sx7l63drWF/qj9Jct6uSOou2aXWwv0nVU+f3k6cBGeAqj71yV7pZjdQdh/dQrB1yAKJJ2ON+liNidCIe8Eg5yu+FnUMbSs533rUHuVxSZfk4H9EVqWg9InUu0OZNWScK13phuWYJ9o6KZcvXQ3P0AlHSUC6dE6EStF4RaKfeuRgRgoIj1Le6wjeTw4v6EhvGE/JvIOUUgNlJiYMkejnv0M6h+fOQKxKZB6dokRlAYHm+Duqz4Q+STnMz6ab50DCSaqWq+6A43xrhET7xfgyge32uyjt7Q7jDm4qNI6WN3iANEQwRhXj8AhD7BO2s1dKy01DzCLqDbgdvuD3B/Do68KgnM9Xj2ygycGrbzljDzZ0rW3Lz3Vz29OwbQ+ETmkTxydlsz/gfUDzzLd52jRb8Ef7UOw+YOXPnpMP1ln2KZN0YKqDXS9k= ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:26:25.934 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-25T11:26:26.172 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:26:26.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:25 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1694487757' entity='client.admin' 2024-04-25T11:26:28.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.514+0000 7fcac2086700 1 -- 172.21.15.90:0/586535964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcabc103a90 msgr2=0x7fcabc103ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:28.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.514+0000 7fcac2086700 1 --2- 172.21.15.90:0/586535964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcabc103a90 0x7fcabc103ea0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcaac009910 tx=0x7fcaac009c20 comp rx=0 tx=0).stop 2024-04-25T11:26:28.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.514+0000 7fcac2086700 1 -- 172.21.15.90:0/586535964 shutdown_connections 2024-04-25T11:26:28.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.514+0000 7fcac2086700 1 --2- 172.21.15.90:0/586535964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcabc103a90 0x7fcabc103ea0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:28.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.514+0000 7fcac2086700 1 -- 172.21.15.90:0/586535964 >> 172.21.15.90:0/586535964 conn(0x7fcabc0ff0d0 msgr2=0x7fcabc101580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:28.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.514+0000 7fcac2086700 1 -- 172.21.15.90:0/586535964 shutdown_connections 2024-04-25T11:26:28.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.514+0000 7fcac2086700 1 -- 172.21.15.90:0/586535964 wait complete. 2024-04-25T11:26:28.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.515+0000 7fcac2086700 1 Processor -- start 2024-04-25T11:26:28.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.515+0000 7fcac2086700 1 -- start start 2024-04-25T11:26:28.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.515+0000 7fcac2086700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcabc103a90 0x7fcabc1995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:28.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.515+0000 7fcac2086700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcabc109a50 con 0x7fcabc103a90 2024-04-25T11:26:28.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.515+0000 7fcabb7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcabc103a90 0x7fcabc1995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:28.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.515+0000 7fcabb7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcabc103a90 0x7fcabc1995d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46118/0 (socket says 172.21.15.90:46118) 2024-04-25T11:26:28.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.515+0000 7fcabb7fe700 1 -- 172.21.15.90:0/1649175441 learned_addr learned my addr 172.21.15.90:0/1649175441 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:28.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.515+0000 7fcabb7fe700 1 -- 172.21.15.90:0/1649175441 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcabc199b10 con 0x7fcabc103a90 2024-04-25T11:26:28.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.516+0000 7fcabb7fe700 1 --2- 172.21.15.90:0/1649175441 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcabc103a90 0x7fcabc1995d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcaac0057d0 tx=0x7fcaac005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:28.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.516+0000 7fcab8ff9700 1 -- 172.21.15.90:0/1649175441 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcaac027070 con 0x7fcabc103a90 2024-04-25T11:26:28.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.516+0000 7fcab8ff9700 1 -- 172.21.15.90:0/1649175441 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcaac00be10 con 0x7fcabc103a90 2024-04-25T11:26:28.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.516+0000 7fcac2086700 1 -- 172.21.15.90:0/1649175441 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcabc199da0 con 0x7fcabc103a90 2024-04-25T11:26:28.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.516+0000 7fcab8ff9700 1 -- 172.21.15.90:0/1649175441 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcaac022dd0 con 0x7fcabc103a90 2024-04-25T11:26:28.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.516+0000 7fcac2086700 1 -- 172.21.15.90:0/1649175441 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcabc19c9b0 con 0x7fcabc103a90 2024-04-25T11:26:28.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.517+0000 7fcab8ff9700 1 -- 172.21.15.90:0/1649175441 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fcaac02a070 con 0x7fcabc103a90 2024-04-25T11:26:28.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.517+0000 7fcac2086700 1 -- 172.21.15.90:0/1649175441 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcabc10a1d0 con 0x7fcabc103a90 2024-04-25T11:26:28.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.517+0000 7fcab8ff9700 1 --2- 172.21.15.90:0/1649175441 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fcaa803d5c0 0x7fcaa803fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:28.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.517+0000 7fcab8ff9700 1 -- 172.21.15.90:0/1649175441 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fcaac01e030 con 0x7fcabc103a90 2024-04-25T11:26:28.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.517+0000 7fcabaffd700 1 --2- 172.21.15.90:0/1649175441 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fcaa803d5c0 0x7fcaa803fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:28.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.517+0000 7fcabaffd700 1 --2- 172.21.15.90:0/1649175441 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fcaa803d5c0 0x7fcaa803fad0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcaa4006fd0 tx=0x7fcaa4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:28.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.522+0000 7fcab8ff9700 1 -- 172.21.15.90:0/1649175441 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcaac013070 con 0x7fcabc103a90 2024-04-25T11:26:28.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.725+0000 7fcac2086700 1 -- 172.21.15.90:0/1649175441 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fcabc02e620 con 0x7fcabc103a90 2024-04-25T11:26:28.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.730+0000 7fcab8ff9700 1 -- 172.21.15.90:0/1649175441 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fcaac00c2d0 con 0x7fcabc103a90 2024-04-25T11:26:28.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.732+0000 7fcac2086700 1 -- 172.21.15.90:0/1649175441 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fcaa803d5c0 msgr2=0x7fcaa803fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:28.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.732+0000 7fcac2086700 1 --2- 172.21.15.90:0/1649175441 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fcaa803d5c0 0x7fcaa803fad0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcaa4006fd0 tx=0x7fcaa4006e40 comp rx=0 tx=0).stop 2024-04-25T11:26:28.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.732+0000 7fcac2086700 1 -- 172.21.15.90:0/1649175441 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcabc103a90 msgr2=0x7fcabc1995d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:28.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.732+0000 7fcac2086700 1 --2- 172.21.15.90:0/1649175441 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcabc103a90 0x7fcabc1995d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcaac0057d0 tx=0x7fcaac005800 comp rx=0 tx=0).stop 2024-04-25T11:26:28.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.732+0000 7fcac2086700 1 -- 172.21.15.90:0/1649175441 shutdown_connections 2024-04-25T11:26:28.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.732+0000 7fcac2086700 1 --2- 172.21.15.90:0/1649175441 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fcaa803d5c0 0x7fcaa803fad0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:28.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.732+0000 7fcac2086700 1 --2- 172.21.15.90:0/1649175441 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcabc103a90 0x7fcabc1995d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:28.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.732+0000 7fcac2086700 1 -- 172.21.15.90:0/1649175441 >> 172.21.15.90:0/1649175441 conn(0x7fcabc0ff0d0 msgr2=0x7fcabc0ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:28.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.732+0000 7fcac2086700 1 -- 172.21.15.90:0/1649175441 shutdown_connections 2024-04-25T11:26:28.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:28.732+0000 7fcac2086700 1 -- 172.21.15.90:0/1649175441 wait complete. 2024-04-25T11:26:29.367 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-25T11:26:29.368 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-25T11:26:29.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:29 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1649175441' entity='client.admin' 2024-04-25T11:26:30.120 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:26:31.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:31 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:31.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:31 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:31.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:31 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi090", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:26:31.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:31 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:31.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:31 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:26:31.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:31 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:31.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.622+0000 7fbdfe7d9700 1 -- 172.21.15.90:0/405602499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdf8103a90 msgr2=0x7fbdf8103ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:31.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.622+0000 7fbdfe7d9700 1 --2- 172.21.15.90:0/405602499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdf8103a90 0x7fbdf8103ea0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbde8009910 tx=0x7fbde8009c20 comp rx=0 tx=0).stop 2024-04-25T11:26:31.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.623+0000 7fbdfe7d9700 1 -- 172.21.15.90:0/405602499 shutdown_connections 2024-04-25T11:26:31.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.623+0000 7fbdfe7d9700 1 --2- 172.21.15.90:0/405602499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdf8103a90 0x7fbdf8103ea0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:31.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.623+0000 7fbdfe7d9700 1 -- 172.21.15.90:0/405602499 >> 172.21.15.90:0/405602499 conn(0x7fbdf80ff0d0 msgr2=0x7fbdf8101580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:31.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.623+0000 7fbdfe7d9700 1 -- 172.21.15.90:0/405602499 shutdown_connections 2024-04-25T11:26:31.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.623+0000 7fbdfe7d9700 1 -- 172.21.15.90:0/405602499 wait complete. 2024-04-25T11:26:31.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.623+0000 7fbdfe7d9700 1 Processor -- start 2024-04-25T11:26:31.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.623+0000 7fbdfe7d9700 1 -- start start 2024-04-25T11:26:31.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.624+0000 7fbdfe7d9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdf8103a90 0x7fbdf8199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:31.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.624+0000 7fbdfe7d9700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf8109a50 con 0x7fbdf8103a90 2024-04-25T11:26:31.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.624+0000 7fbdf7fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdf8103a90 0x7fbdf8199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:31.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.624+0000 7fbdf7fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdf8103a90 0x7fbdf8199600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43322/0 (socket says 172.21.15.90:43322) 2024-04-25T11:26:31.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.624+0000 7fbdf7fff700 1 -- 172.21.15.90:0/395996682 learned_addr learned my addr 172.21.15.90:0/395996682 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:31.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.624+0000 7fbdf7fff700 1 -- 172.21.15.90:0/395996682 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdf8199b40 con 0x7fbdf8103a90 2024-04-25T11:26:31.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.624+0000 7fbdf7fff700 1 --2- 172.21.15.90:0/395996682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdf8103a90 0x7fbdf8199600 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbde80057d0 tx=0x7fbde8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:31.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.624+0000 7fbdf57fa700 1 -- 172.21.15.90:0/395996682 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbde8027070 con 0x7fbdf8103a90 2024-04-25T11:26:31.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.624+0000 7fbdf57fa700 1 -- 172.21.15.90:0/395996682 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbde800be10 con 0x7fbdf8103a90 2024-04-25T11:26:31.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.624+0000 7fbdf57fa700 1 -- 172.21.15.90:0/395996682 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbde8022dd0 con 0x7fbdf8103a90 2024-04-25T11:26:31.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.624+0000 7fbdfe7d9700 1 -- 172.21.15.90:0/395996682 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdf8199dd0 con 0x7fbdf8103a90 2024-04-25T11:26:31.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.624+0000 7fbdfe7d9700 1 -- 172.21.15.90:0/395996682 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdf819c9e0 con 0x7fbdf8103a90 2024-04-25T11:26:31.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.625+0000 7fbdf57fa700 1 -- 172.21.15.90:0/395996682 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fbde802a070 con 0x7fbdf8103a90 2024-04-25T11:26:31.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.625+0000 7fbdfe7d9700 1 -- 172.21.15.90:0/395996682 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdf810a1d0 con 0x7fbdf8103a90 2024-04-25T11:26:31.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.626+0000 7fbdf57fa700 1 --2- 172.21.15.90:0/395996682 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbde003d610 0x7fbde003fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:31.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.626+0000 7fbdf57fa700 1 -- 172.21.15.90:0/395996682 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbde801e030 con 0x7fbdf8103a90 2024-04-25T11:26:31.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.626+0000 7fbdf77fe700 1 --2- 172.21.15.90:0/395996682 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbde003d610 0x7fbde003fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:31.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.626+0000 7fbdf77fe700 1 --2- 172.21.15.90:0/395996682 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbde003d610 0x7fbde003fb20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbdec006fd0 tx=0x7fbdec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:31.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.631+0000 7fbdf57fa700 1 -- 172.21.15.90:0/395996682 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbde8013070 con 0x7fbdf8103a90 2024-04-25T11:26:31.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.834+0000 7fbdfe7d9700 1 -- 172.21.15.90:0/395996682 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7fbdf802e620 con 0x7fbde003d610 2024-04-25T11:26:31.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.838+0000 7fbdf57fa700 1 -- 172.21.15.90:0/395996682 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fbdf802e620 con 0x7fbde003d610 2024-04-25T11:26:31.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.840+0000 7fbdfe7d9700 1 -- 172.21.15.90:0/395996682 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbde003d610 msgr2=0x7fbde003fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:31.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.840+0000 7fbdfe7d9700 1 --2- 172.21.15.90:0/395996682 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbde003d610 0x7fbde003fb20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbdec006fd0 tx=0x7fbdec006e40 comp rx=0 tx=0).stop 2024-04-25T11:26:31.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.840+0000 7fbdfe7d9700 1 -- 172.21.15.90:0/395996682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdf8103a90 msgr2=0x7fbdf8199600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:31.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.840+0000 7fbdfe7d9700 1 --2- 172.21.15.90:0/395996682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdf8103a90 0x7fbdf8199600 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbde80057d0 tx=0x7fbde8005800 comp rx=0 tx=0).stop 2024-04-25T11:26:31.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.840+0000 7fbdfe7d9700 1 -- 172.21.15.90:0/395996682 shutdown_connections 2024-04-25T11:26:31.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.840+0000 7fbdfe7d9700 1 --2- 172.21.15.90:0/395996682 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbde003d610 0x7fbde003fb20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:31.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.840+0000 7fbdfe7d9700 1 --2- 172.21.15.90:0/395996682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdf8103a90 0x7fbdf8199600 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:31.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.840+0000 7fbdfe7d9700 1 -- 172.21.15.90:0/395996682 >> 172.21.15.90:0/395996682 conn(0x7fbdf80ff0d0 msgr2=0x7fbdf80ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:31.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.840+0000 7fbdfe7d9700 1 -- 172.21.15.90:0/395996682 shutdown_connections 2024-04-25T11:26:31.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:31.840+0000 7fbdfe7d9700 1 -- 172.21.15.90:0/395996682 wait complete. 2024-04-25T11:26:32.522 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi138 2024-04-25T11:26:32.522 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-25T11:26:32.522 DEBUG:teuthology.orchestra.run.smithi138:> dd of=/etc/ceph/ceph.conf 2024-04-25T11:26:32.541 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-25T11:26:32.541 DEBUG:teuthology.orchestra.run.smithi138:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T11:26:32.597 INFO:tasks.cephadm:Adding host smithi138 to orchestrator... 2024-04-25T11:26:32.598 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph orch host add smithi138 2024-04-25T11:26:32.834 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:26:33.122 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:32 smithi090 ceph-mon[29575]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:26:33.122 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:32 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:33.122 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:32 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:26:33.122 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:32 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:26:33.122 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:32 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:26:33.122 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:32 smithi090 ceph-mon[29575]: Updating smithi090:/etc/ceph/ceph.conf 2024-04-25T11:26:33.122 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:32 smithi090 ceph-mon[29575]: Updating smithi090:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T11:26:33.123 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:32 smithi090 ceph-mon[29575]: Updating smithi090:/etc/ceph/ceph.client.admin.keyring 2024-04-25T11:26:33.123 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:32 smithi090 ceph-mon[29575]: Updating smithi090:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T11:26:33.123 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:32 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:33.123 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:32 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:33.123 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:32 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:34.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.309+0000 7f2d283d1700 1 -- 172.21.15.90:0/3684574875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d20109a10 msgr2=0x7f2d20109e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:34.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.309+0000 7f2d283d1700 1 --2- 172.21.15.90:0/3684574875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d20109a10 0x7f2d20109e20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c009910 tx=0x7f2d1c009c20 comp rx=0 tx=0).stop 2024-04-25T11:26:34.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.309+0000 7f2d283d1700 1 -- 172.21.15.90:0/3684574875 shutdown_connections 2024-04-25T11:26:34.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.309+0000 7f2d283d1700 1 --2- 172.21.15.90:0/3684574875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d20109a10 0x7f2d20109e20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:34.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.309+0000 7f2d283d1700 1 -- 172.21.15.90:0/3684574875 >> 172.21.15.90:0/3684574875 conn(0x7f2d200ff580 msgr2=0x7f2d201019d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:34.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.309+0000 7f2d283d1700 1 -- 172.21.15.90:0/3684574875 shutdown_connections 2024-04-25T11:26:34.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.309+0000 7f2d283d1700 1 -- 172.21.15.90:0/3684574875 wait complete. 2024-04-25T11:26:34.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.310+0000 7f2d283d1700 1 Processor -- start 2024-04-25T11:26:34.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.310+0000 7f2d283d1700 1 -- start start 2024-04-25T11:26:34.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.310+0000 7f2d283d1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d20109a10 0x7f2d201997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:34.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.310+0000 7f2d283d1700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2010df60 con 0x7f2d20109a10 2024-04-25T11:26:34.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.310+0000 7f2d2616d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d20109a10 0x7f2d201997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:34.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.310+0000 7f2d2616d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d20109a10 0x7f2d201997b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43338/0 (socket says 172.21.15.90:43338) 2024-04-25T11:26:34.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.310+0000 7f2d2616d700 1 -- 172.21.15.90:0/2667216465 learned_addr learned my addr 172.21.15.90:0/2667216465 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:34.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.310+0000 7f2d2616d700 1 -- 172.21.15.90:0/2667216465 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d20199cf0 con 0x7f2d20109a10 2024-04-25T11:26:34.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.310+0000 7f2d2616d700 1 --2- 172.21.15.90:0/2667216465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d20109a10 0x7f2d201997b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c0056d0 tx=0x7f2d1c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:34.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.311+0000 7f2d177fe700 1 -- 172.21.15.90:0/2667216465 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d1c027070 con 0x7f2d20109a10 2024-04-25T11:26:34.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.311+0000 7f2d283d1700 1 -- 172.21.15.90:0/2667216465 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d20199fe0 con 0x7f2d20109a10 2024-04-25T11:26:34.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.311+0000 7f2d177fe700 1 -- 172.21.15.90:0/2667216465 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d1c00be10 con 0x7f2d20109a10 2024-04-25T11:26:34.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.311+0000 7f2d177fe700 1 -- 172.21.15.90:0/2667216465 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d1c022d20 con 0x7f2d20109a10 2024-04-25T11:26:34.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.311+0000 7f2d283d1700 1 -- 172.21.15.90:0/2667216465 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d2019dfe0 con 0x7f2d20109a10 2024-04-25T11:26:34.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.311+0000 7f2d283d1700 1 -- 172.21.15.90:0/2667216465 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d20105120 con 0x7f2d20109a10 2024-04-25T11:26:34.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.312+0000 7f2d177fe700 1 -- 172.21.15.90:0/2667216465 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f2d1c02a070 con 0x7f2d20109a10 2024-04-25T11:26:34.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.312+0000 7f2d177fe700 1 --2- 172.21.15.90:0/2667216465 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f2d0c03d610 0x7f2d0c03fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:34.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.312+0000 7f2d177fe700 1 -- 172.21.15.90:0/2667216465 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2d1c01e030 con 0x7f2d20109a10 2024-04-25T11:26:34.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.312+0000 7f2d2596c700 1 --2- 172.21.15.90:0/2667216465 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f2d0c03d610 0x7f2d0c03fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:34.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.312+0000 7f2d2596c700 1 --2- 172.21.15.90:0/2667216465 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f2d0c03d610 0x7f2d0c03fb20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2d10006fd0 tx=0x7f2d10006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:34.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.317+0000 7f2d177fe700 1 -- 172.21.15.90:0/2667216465 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2d1c013070 con 0x7f2d20109a10 2024-04-25T11:26:34.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:34.519+0000 7f2d283d1700 1 -- 172.21.15.90:0/2667216465 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi138", "target": ["mon-mgr", ""]}) -- 0x7f2d2019ab10 con 0x7f2d0c03d610 2024-04-25T11:26:35.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:34 smithi090 ceph-mon[29575]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi138", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:26:36.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:35 smithi090 ceph-mon[29575]: Deploying cephadm binary to smithi138 2024-04-25T11:26:37.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:37 smithi090 ceph-mon[29575]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:26:38.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:38.678+0000 7f2d177fe700 1 -- 172.21.15.90:0/2667216465 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f2d2019ab10 con 0x7f2d0c03d610 2024-04-25T11:26:38.672 INFO:teuthology.orchestra.run.smithi090.stdout:Added host 'smithi138' with addr '172.21.15.138' 2024-04-25T11:26:38.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:38.681+0000 7f2d283d1700 1 -- 172.21.15.90:0/2667216465 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f2d0c03d610 msgr2=0x7f2d0c03fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:38.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:38.681+0000 7f2d283d1700 1 --2- 172.21.15.90:0/2667216465 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f2d0c03d610 0x7f2d0c03fb20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2d10006fd0 tx=0x7f2d10006e40 comp rx=0 tx=0).stop 2024-04-25T11:26:38.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:38.681+0000 7f2d283d1700 1 -- 172.21.15.90:0/2667216465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d20109a10 msgr2=0x7f2d201997b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:38.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:38.681+0000 7f2d283d1700 1 --2- 172.21.15.90:0/2667216465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d20109a10 0x7f2d201997b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c0056d0 tx=0x7f2d1c0057b0 comp rx=0 tx=0).stop 2024-04-25T11:26:38.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:38.681+0000 7f2d283d1700 1 -- 172.21.15.90:0/2667216465 shutdown_connections 2024-04-25T11:26:38.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:38.681+0000 7f2d283d1700 1 --2- 172.21.15.90:0/2667216465 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f2d0c03d610 0x7f2d0c03fb20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:38.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:38.681+0000 7f2d283d1700 1 --2- 172.21.15.90:0/2667216465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d20109a10 0x7f2d201997b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:38.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:38.681+0000 7f2d283d1700 1 -- 172.21.15.90:0/2667216465 >> 172.21.15.90:0/2667216465 conn(0x7f2d200ff580 msgr2=0x7f2d20100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:38.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:38.681+0000 7f2d283d1700 1 -- 172.21.15.90:0/2667216465 shutdown_connections 2024-04-25T11:26:38.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:38.681+0000 7f2d283d1700 1 -- 172.21.15.90:0/2667216465 wait complete. 2024-04-25T11:26:39.271 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T11:26:39.503 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:26:39.923 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:39 smithi090 ceph-mon[29575]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:26:39.923 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:39 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:39.923 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:39 smithi090 ceph-mon[29575]: Added host smithi138 2024-04-25T11:26:39.923 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:39 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:26:39.923 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:39 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:40.781 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:40 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:41.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.042+0000 7f0dc1deb700 1 -- 172.21.15.90:0/149308153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dbc108910 msgr2=0x7f0dbc108d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:41.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.042+0000 7f0dc1deb700 1 --2- 172.21.15.90:0/149308153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dbc108910 0x7f0dbc108d20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0db0009910 tx=0x7f0db0009c20 comp rx=0 tx=0).stop 2024-04-25T11:26:41.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.042+0000 7f0dc1deb700 1 -- 172.21.15.90:0/149308153 shutdown_connections 2024-04-25T11:26:41.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.042+0000 7f0dc1deb700 1 --2- 172.21.15.90:0/149308153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dbc108910 0x7f0dbc108d20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:41.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.042+0000 7f0dc1deb700 1 -- 172.21.15.90:0/149308153 >> 172.21.15.90:0/149308153 conn(0x7f0dbc0fe440 msgr2=0x7f0dbc100890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:41.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.042+0000 7f0dc1deb700 1 -- 172.21.15.90:0/149308153 shutdown_connections 2024-04-25T11:26:41.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.042+0000 7f0dc1deb700 1 -- 172.21.15.90:0/149308153 wait complete. 2024-04-25T11:26:41.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.042+0000 7f0dc1deb700 1 Processor -- start 2024-04-25T11:26:41.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.043+0000 7f0dc1deb700 1 -- start start 2024-04-25T11:26:41.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.043+0000 7f0dc1deb700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dbc108910 0x7f0dbc198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:41.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.043+0000 7f0dc1deb700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0dbc10ce60 con 0x7f0dbc108910 2024-04-25T11:26:41.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.043+0000 7f0dbb7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dbc108910 0x7f0dbc198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:41.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.043+0000 7f0dbb7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dbc108910 0x7f0dbc198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43356/0 (socket says 172.21.15.90:43356) 2024-04-25T11:26:41.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.043+0000 7f0dbb7fe700 1 -- 172.21.15.90:0/3216623629 learned_addr learned my addr 172.21.15.90:0/3216623629 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:41.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.043+0000 7f0dbb7fe700 1 -- 172.21.15.90:0/3216623629 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dbc198c40 con 0x7f0dbc108910 2024-04-25T11:26:41.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.043+0000 7f0dbb7fe700 1 --2- 172.21.15.90:0/3216623629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dbc108910 0x7f0dbc198700 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0db00056d0 tx=0x7f0db00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:41.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.044+0000 7f0db8ff9700 1 -- 172.21.15.90:0/3216623629 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0db0027070 con 0x7f0dbc108910 2024-04-25T11:26:41.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.044+0000 7f0db8ff9700 1 -- 172.21.15.90:0/3216623629 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0db000be10 con 0x7f0dbc108910 2024-04-25T11:26:41.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.044+0000 7f0dc1deb700 1 -- 172.21.15.90:0/3216623629 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dbc198f30 con 0x7f0dbc108910 2024-04-25T11:26:41.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.044+0000 7f0db8ff9700 1 -- 172.21.15.90:0/3216623629 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0db0022d20 con 0x7f0dbc108910 2024-04-25T11:26:41.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.044+0000 7f0dc1deb700 1 -- 172.21.15.90:0/3216623629 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dbc19cf30 con 0x7f0dbc108910 2024-04-25T11:26:41.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.045+0000 7f0db8ff9700 1 -- 172.21.15.90:0/3216623629 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f0db002a070 con 0x7f0dbc108910 2024-04-25T11:26:41.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.045+0000 7f0db8ff9700 1 --2- 172.21.15.90:0/3216623629 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0da403d670 0x7f0da403fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:41.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.045+0000 7f0db8ff9700 1 -- 172.21.15.90:0/3216623629 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0db001e030 con 0x7f0dbc108910 2024-04-25T11:26:41.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.045+0000 7f0dc1deb700 1 -- 172.21.15.90:0/3216623629 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dbc104020 con 0x7f0dbc108910 2024-04-25T11:26:41.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.045+0000 7f0dbaffd700 1 --2- 172.21.15.90:0/3216623629 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0da403d670 0x7f0da403fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:41.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.045+0000 7f0dbaffd700 1 --2- 172.21.15.90:0/3216623629 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0da403d670 0x7f0da403fb80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0dac006fd0 tx=0x7f0dac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:41.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.050+0000 7f0db8ff9700 1 -- 172.21.15.90:0/3216623629 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0db0013070 con 0x7f0dbc108910 2024-04-25T11:26:41.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.252+0000 7f0dc1deb700 1 -- 172.21.15.90:0/3216623629 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0dbc199a60 con 0x7f0da403d670 2024-04-25T11:26:41.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.254+0000 7f0db8ff9700 1 -- 172.21.15.90:0/3216623629 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f0dbc199a60 con 0x7f0da403d670 2024-04-25T11:26:41.246 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:26:41.246 INFO:teuthology.orchestra.run.smithi090.stdout:[{"addr": "172.21.15.90", "hostname": "smithi090", "labels": [], "status": ""}, {"addr": "172.21.15.138", "hostname": "smithi138", "labels": [], "status": ""}] 2024-04-25T11:26:41.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.256+0000 7f0dc1deb700 1 -- 172.21.15.90:0/3216623629 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0da403d670 msgr2=0x7f0da403fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:41.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.256+0000 7f0dc1deb700 1 --2- 172.21.15.90:0/3216623629 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0da403d670 0x7f0da403fb80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0dac006fd0 tx=0x7f0dac006e40 comp rx=0 tx=0).stop 2024-04-25T11:26:41.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.256+0000 7f0dc1deb700 1 -- 172.21.15.90:0/3216623629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dbc108910 msgr2=0x7f0dbc198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:41.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.256+0000 7f0dc1deb700 1 --2- 172.21.15.90:0/3216623629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dbc108910 0x7f0dbc198700 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0db00056d0 tx=0x7f0db00057b0 comp rx=0 tx=0).stop 2024-04-25T11:26:41.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.256+0000 7f0dc1deb700 1 -- 172.21.15.90:0/3216623629 shutdown_connections 2024-04-25T11:26:41.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.256+0000 7f0dc1deb700 1 --2- 172.21.15.90:0/3216623629 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0da403d670 0x7f0da403fb80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:41.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.256+0000 7f0dc1deb700 1 --2- 172.21.15.90:0/3216623629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dbc108910 0x7f0dbc198700 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:41.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.256+0000 7f0dc1deb700 1 -- 172.21.15.90:0/3216623629 >> 172.21.15.90:0/3216623629 conn(0x7f0dbc0fe440 msgr2=0x7f0dbc0ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:41.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.256+0000 7f0dc1deb700 1 -- 172.21.15.90:0/3216623629 shutdown_connections 2024-04-25T11:26:41.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:41.256+0000 7f0dc1deb700 1 -- 172.21.15.90:0/3216623629 wait complete. 2024-04-25T11:26:41.854 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:41 smithi090 ceph-mon[29575]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:26:41.854 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:41 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:41.856 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi156 2024-04-25T11:26:41.856 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-25T11:26:41.856 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/etc/ceph/ceph.conf 2024-04-25T11:26:41.874 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-25T11:26:41.874 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T11:26:41.932 INFO:tasks.cephadm:Adding host smithi156 to orchestrator... 2024-04-25T11:26:41.932 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph orch host add smithi156 2024-04-25T11:26:42.163 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:26:42.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:42 smithi090 ceph-mon[29575]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T11:26:43.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.642+0000 7fb92ac60700 1 -- 172.21.15.90:0/1984062158 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb924109a40 msgr2=0x7fb924109e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:43.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.642+0000 7fb92ac60700 1 --2- 172.21.15.90:0/1984062158 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb924109a40 0x7fb924109e50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb918009910 tx=0x7fb918009c20 comp rx=0 tx=0).stop 2024-04-25T11:26:43.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.642+0000 7fb92ac60700 1 -- 172.21.15.90:0/1984062158 shutdown_connections 2024-04-25T11:26:43.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.642+0000 7fb92ac60700 1 --2- 172.21.15.90:0/1984062158 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb924109a40 0x7fb924109e50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:43.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.642+0000 7fb92ac60700 1 -- 172.21.15.90:0/1984062158 >> 172.21.15.90:0/1984062158 conn(0x7fb9240ff530 msgr2=0x7fb924101980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:43.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.642+0000 7fb92ac60700 1 -- 172.21.15.90:0/1984062158 shutdown_connections 2024-04-25T11:26:43.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.642+0000 7fb92ac60700 1 -- 172.21.15.90:0/1984062158 wait complete. 2024-04-25T11:26:43.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.642+0000 7fb92ac60700 1 Processor -- start 2024-04-25T11:26:43.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.642+0000 7fb92ac60700 1 -- start start 2024-04-25T11:26:43.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.643+0000 7fb92ac60700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb924109a40 0x7fb924075990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:43.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.643+0000 7fb92ac60700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb92410df90 con 0x7fb924109a40 2024-04-25T11:26:43.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.643+0000 7fb9289fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb924109a40 0x7fb924075990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:43.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.643+0000 7fb9289fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb924109a40 0x7fb924075990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55854/0 (socket says 172.21.15.90:55854) 2024-04-25T11:26:43.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.643+0000 7fb9289fc700 1 -- 172.21.15.90:0/3424183516 learned_addr learned my addr 172.21.15.90:0/3424183516 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:43.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.643+0000 7fb9289fc700 1 -- 172.21.15.90:0/3424183516 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9240777e0 con 0x7fb924109a40 2024-04-25T11:26:43.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.643+0000 7fb9289fc700 1 --2- 172.21.15.90:0/3424183516 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb924109a40 0x7fb924075990 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb9180056d0 tx=0x7fb9180057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:43.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.643+0000 7fb921ffb700 1 -- 172.21.15.90:0/3424183516 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb918027070 con 0x7fb924109a40 2024-04-25T11:26:43.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.643+0000 7fb921ffb700 1 -- 172.21.15.90:0/3424183516 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb91800be10 con 0x7fb924109a40 2024-04-25T11:26:43.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.643+0000 7fb92ac60700 1 -- 172.21.15.90:0/3424183516 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb924075ed0 con 0x7fb924109a40 2024-04-25T11:26:43.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.643+0000 7fb921ffb700 1 -- 172.21.15.90:0/3424183516 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb918022d20 con 0x7fb924109a40 2024-04-25T11:26:43.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.643+0000 7fb92ac60700 1 -- 172.21.15.90:0/3424183516 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb924076390 con 0x7fb924109a40 2024-04-25T11:26:43.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.644+0000 7fb92ac60700 1 -- 172.21.15.90:0/3424183516 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb924105150 con 0x7fb924109a40 2024-04-25T11:26:43.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.644+0000 7fb921ffb700 1 -- 172.21.15.90:0/3424183516 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fb91802a070 con 0x7fb924109a40 2024-04-25T11:26:43.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.645+0000 7fb921ffb700 1 --2- 172.21.15.90:0/3424183516 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb91403d670 0x7fb91403fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:43.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.645+0000 7fb921ffb700 1 -- 172.21.15.90:0/3424183516 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb91801e030 con 0x7fb924109a40 2024-04-25T11:26:43.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.645+0000 7fb923fff700 1 --2- 172.21.15.90:0/3424183516 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb91403d670 0x7fb91403fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:43.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.645+0000 7fb923fff700 1 --2- 172.21.15.90:0/3424183516 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb91403d670 0x7fb91403fb80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb910006fd0 tx=0x7fb910006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:43.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.650+0000 7fb921ffb700 1 -- 172.21.15.90:0/3424183516 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb918013070 con 0x7fb924109a40 2024-04-25T11:26:43.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:43.852+0000 7fb92ac60700 1 -- 172.21.15.90:0/3424183516 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi156", "target": ["mon-mgr", ""]}) -- 0x7fb924076f60 con 0x7fb91403d670 2024-04-25T11:26:43.845 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:43 smithi090 ceph-mon[29575]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:26:45.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:45 smithi090 ceph-mon[29575]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi156", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:26:45.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:45 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:45.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:45 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:45.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:45 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:45.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:45 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:45.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:45 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi138", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:26:45.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:45 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:26:45.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:45 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:26:45.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:45 smithi090 ceph-mon[29575]: Updating smithi138:/etc/ceph/ceph.conf 2024-04-25T11:26:45.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:45 smithi090 ceph-mon[29575]: Deploying cephadm binary to smithi156 2024-04-25T11:26:45.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:45 smithi090 ceph-mon[29575]: Updating smithi138:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T11:26:45.730 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:45 smithi090 ceph-mon[29575]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:26:46.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:46 smithi090 ceph-mon[29575]: Updating smithi138:/etc/ceph/ceph.client.admin.keyring 2024-04-25T11:26:47.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:47 smithi090 ceph-mon[29575]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:26:47.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:47.923+0000 7fb921ffb700 1 -- 172.21.15.90:0/3424183516 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fb924076f60 con 0x7fb91403d670 2024-04-25T11:26:47.915 INFO:teuthology.orchestra.run.smithi090.stdout:Added host 'smithi156' with addr '172.21.15.156' 2024-04-25T11:26:47.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:47.925+0000 7fb92ac60700 1 -- 172.21.15.90:0/3424183516 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb91403d670 msgr2=0x7fb91403fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:47.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:47.925+0000 7fb92ac60700 1 --2- 172.21.15.90:0/3424183516 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb91403d670 0x7fb91403fb80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb910006fd0 tx=0x7fb910006e40 comp rx=0 tx=0).stop 2024-04-25T11:26:47.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:47.925+0000 7fb92ac60700 1 -- 172.21.15.90:0/3424183516 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb924109a40 msgr2=0x7fb924075990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:47.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:47.925+0000 7fb92ac60700 1 --2- 172.21.15.90:0/3424183516 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb924109a40 0x7fb924075990 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb9180056d0 tx=0x7fb9180057b0 comp rx=0 tx=0).stop 2024-04-25T11:26:47.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:47.925+0000 7fb92ac60700 1 -- 172.21.15.90:0/3424183516 shutdown_connections 2024-04-25T11:26:47.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:47.925+0000 7fb92ac60700 1 --2- 172.21.15.90:0/3424183516 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb91403d670 0x7fb91403fb80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:47.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:47.925+0000 7fb92ac60700 1 --2- 172.21.15.90:0/3424183516 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb924109a40 0x7fb924075990 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:47.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:47.925+0000 7fb92ac60700 1 -- 172.21.15.90:0/3424183516 >> 172.21.15.90:0/3424183516 conn(0x7fb9240ff530 msgr2=0x7fb924100510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:47.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:47.925+0000 7fb92ac60700 1 -- 172.21.15.90:0/3424183516 shutdown_connections 2024-04-25T11:26:47.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:47.925+0000 7fb92ac60700 1 -- 172.21.15.90:0/3424183516 wait complete. 2024-04-25T11:26:48.501 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T11:26:48.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:48 smithi090 ceph-mon[29575]: Updating smithi138:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T11:26:48.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:48 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:48.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:48 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:48.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:48 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:48.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:48 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:48.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:48 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:26:48.729 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:48 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:48.735 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:26:49.930 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:49 smithi090 ceph-mon[29575]: Added host smithi156 2024-04-25T11:26:49.930 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:49 smithi090 ceph-mon[29575]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:26:49.930 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:49 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:50.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.267+0000 7f454be5f700 1 -- 172.21.15.90:0/2363987408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4544109a60 msgr2=0x7f4544109e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:50.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.267+0000 7f454be5f700 1 --2- 172.21.15.90:0/2363987408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4544109a60 0x7f4544109e70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4540009910 tx=0x7f4540009c20 comp rx=0 tx=0).stop 2024-04-25T11:26:50.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.267+0000 7f454be5f700 1 -- 172.21.15.90:0/2363987408 shutdown_connections 2024-04-25T11:26:50.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.267+0000 7f454be5f700 1 --2- 172.21.15.90:0/2363987408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4544109a60 0x7f4544109e70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:50.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.267+0000 7f454be5f700 1 -- 172.21.15.90:0/2363987408 >> 172.21.15.90:0/2363987408 conn(0x7f45440ff570 msgr2=0x7f45441019c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:50.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.268+0000 7f454be5f700 1 -- 172.21.15.90:0/2363987408 shutdown_connections 2024-04-25T11:26:50.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.268+0000 7f454be5f700 1 -- 172.21.15.90:0/2363987408 wait complete. 2024-04-25T11:26:50.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.268+0000 7f454be5f700 1 Processor -- start 2024-04-25T11:26:50.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.268+0000 7f454be5f700 1 -- start start 2024-04-25T11:26:50.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.269+0000 7f454be5f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4544109a60 0x7f454419f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:50.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.269+0000 7f454be5f700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f454410dfb0 con 0x7f4544109a60 2024-04-25T11:26:50.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.269+0000 7f4549bfb700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4544109a60 0x7f454419f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:50.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.269+0000 7f4549bfb700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4544109a60 0x7f454419f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55870/0 (socket says 172.21.15.90:55870) 2024-04-25T11:26:50.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.269+0000 7f4549bfb700 1 -- 172.21.15.90:0/1188008438 learned_addr learned my addr 172.21.15.90:0/1188008438 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:50.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.269+0000 7f4549bfb700 1 -- 172.21.15.90:0/1188008438 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f454419fa00 con 0x7f4544109a60 2024-04-25T11:26:50.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.269+0000 7f4549bfb700 1 --2- 172.21.15.90:0/1188008438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4544109a60 0x7f454419f4c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f45400056d0 tx=0x7f45400057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:50.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.269+0000 7f453affd700 1 -- 172.21.15.90:0/1188008438 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4540027070 con 0x7f4544109a60 2024-04-25T11:26:50.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.269+0000 7f453affd700 1 -- 172.21.15.90:0/1188008438 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f454000be10 con 0x7f4544109a60 2024-04-25T11:26:50.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.269+0000 7f454be5f700 1 -- 172.21.15.90:0/1188008438 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f454419fcf0 con 0x7f4544109a60 2024-04-25T11:26:50.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.269+0000 7f453affd700 1 -- 172.21.15.90:0/1188008438 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4540022d20 con 0x7f4544109a60 2024-04-25T11:26:50.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.269+0000 7f454be5f700 1 -- 172.21.15.90:0/1188008438 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4544199820 con 0x7f4544109a60 2024-04-25T11:26:50.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.270+0000 7f453affd700 1 -- 172.21.15.90:0/1188008438 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f454002a070 con 0x7f4544109a60 2024-04-25T11:26:50.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.270+0000 7f453affd700 1 --2- 172.21.15.90:0/1188008438 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f453003d670 0x7f453003fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:50.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.271+0000 7f453affd700 1 -- 172.21.15.90:0/1188008438 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f454001e030 con 0x7f4544109a60 2024-04-25T11:26:50.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.271+0000 7f454be5f700 1 -- 172.21.15.90:0/1188008438 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4544105170 con 0x7f4544109a60 2024-04-25T11:26:50.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.271+0000 7f45493fa700 1 --2- 172.21.15.90:0/1188008438 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f453003d670 0x7f453003fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:50.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.271+0000 7f45493fa700 1 --2- 172.21.15.90:0/1188008438 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f453003d670 0x7f453003fb80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4534006fd0 tx=0x7f4534006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:50.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.276+0000 7f453affd700 1 -- 172.21.15.90:0/1188008438 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4540013070 con 0x7f4544109a60 2024-04-25T11:26:50.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.477+0000 7f454be5f700 1 -- 172.21.15.90:0/1188008438 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f454419ff20 con 0x7f453003d670 2024-04-25T11:26:50.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.478+0000 7f453affd700 1 -- 172.21.15.90:0/1188008438 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f454419ff20 con 0x7f453003d670 2024-04-25T11:26:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:26:50.471 INFO:teuthology.orchestra.run.smithi090.stdout:[{"addr": "172.21.15.90", "hostname": "smithi090", "labels": [], "status": ""}, {"addr": "172.21.15.138", "hostname": "smithi138", "labels": [], "status": ""}, {"addr": "172.21.15.156", "hostname": "smithi156", "labels": [], "status": ""}] 2024-04-25T11:26:50.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.480+0000 7f454be5f700 1 -- 172.21.15.90:0/1188008438 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f453003d670 msgr2=0x7f453003fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:50.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.480+0000 7f454be5f700 1 --2- 172.21.15.90:0/1188008438 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f453003d670 0x7f453003fb80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4534006fd0 tx=0x7f4534006e40 comp rx=0 tx=0).stop 2024-04-25T11:26:50.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.480+0000 7f454be5f700 1 -- 172.21.15.90:0/1188008438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4544109a60 msgr2=0x7f454419f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:50.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.480+0000 7f454be5f700 1 --2- 172.21.15.90:0/1188008438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4544109a60 0x7f454419f4c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f45400056d0 tx=0x7f45400057b0 comp rx=0 tx=0).stop 2024-04-25T11:26:50.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.480+0000 7f454be5f700 1 -- 172.21.15.90:0/1188008438 shutdown_connections 2024-04-25T11:26:50.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.480+0000 7f454be5f700 1 --2- 172.21.15.90:0/1188008438 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f453003d670 0x7f453003fb80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:50.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.480+0000 7f454be5f700 1 --2- 172.21.15.90:0/1188008438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4544109a60 0x7f454419f4c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:50.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.480+0000 7f454be5f700 1 -- 172.21.15.90:0/1188008438 >> 172.21.15.90:0/1188008438 conn(0x7f45440ff570 msgr2=0x7f45441004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:50.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.480+0000 7f454be5f700 1 -- 172.21.15.90:0/1188008438 shutdown_connections 2024-04-25T11:26:50.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:50.480+0000 7f454be5f700 1 -- 172.21.15.90:0/1188008438 wait complete. 2024-04-25T11:26:51.065 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-25T11:26:51.066 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-25T11:26:51.302 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:26:51.617 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:51 smithi090 ceph-mon[29575]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T11:26:51.617 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:51 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:51.617 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:51 smithi090 ceph-mon[29575]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:26:52.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.721+0000 7f213bc7a700 1 -- 172.21.15.90:0/2151633971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21341089c0 msgr2=0x7f2134108dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:52.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.721+0000 7f213bc7a700 1 --2- 172.21.15.90:0/2151633971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21341089c0 0x7f2134108dd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2130009910 tx=0x7f2130009c20 comp rx=0 tx=0).stop 2024-04-25T11:26:52.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.721+0000 7f213bc7a700 1 -- 172.21.15.90:0/2151633971 shutdown_connections 2024-04-25T11:26:52.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.721+0000 7f213bc7a700 1 --2- 172.21.15.90:0/2151633971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21341089c0 0x7f2134108dd0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:52.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.721+0000 7f213bc7a700 1 -- 172.21.15.90:0/2151633971 >> 172.21.15.90:0/2151633971 conn(0x7f21340fdd00 msgr2=0x7f2134100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:52.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.721+0000 7f213bc7a700 1 -- 172.21.15.90:0/2151633971 shutdown_connections 2024-04-25T11:26:52.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.721+0000 7f213bc7a700 1 -- 172.21.15.90:0/2151633971 wait complete. 2024-04-25T11:26:52.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.721+0000 7f213bc7a700 1 Processor -- start 2024-04-25T11:26:52.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.722+0000 7f213bc7a700 1 -- start start 2024-04-25T11:26:52.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.722+0000 7f213bc7a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21341089c0 0x7f213419fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:52.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.722+0000 7f213bc7a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f21341145f0 con 0x7f21341089c0 2024-04-25T11:26:52.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.722+0000 7f2139a16700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21341089c0 0x7f213419fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:52.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.722+0000 7f2139a16700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21341089c0 0x7f213419fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55304/0 (socket says 172.21.15.90:55304) 2024-04-25T11:26:52.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.722+0000 7f2139a16700 1 -- 172.21.15.90:0/1222248799 learned_addr learned my addr 172.21.15.90:0/1222248799 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:26:52.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.722+0000 7f2139a16700 1 -- 172.21.15.90:0/1222248799 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21341a03b0 con 0x7f21341089c0 2024-04-25T11:26:52.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.722+0000 7f2139a16700 1 --2- 172.21.15.90:0/1222248799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21341089c0 0x7f213419fe70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f21300056d0 tx=0x7f21300057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:52.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.722+0000 7f212affd700 1 -- 172.21.15.90:0/1222248799 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2130027070 con 0x7f21341089c0 2024-04-25T11:26:52.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.722+0000 7f212affd700 1 -- 172.21.15.90:0/1222248799 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f213000be10 con 0x7f21341089c0 2024-04-25T11:26:52.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.722+0000 7f212affd700 1 -- 172.21.15.90:0/1222248799 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2130022d00 con 0x7f21341089c0 2024-04-25T11:26:52.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.723+0000 7f213bc7a700 1 -- 172.21.15.90:0/1222248799 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21341a0640 con 0x7f21341089c0 2024-04-25T11:26:52.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.723+0000 7f213bc7a700 1 -- 172.21.15.90:0/1222248799 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21341a46a0 con 0x7f21341089c0 2024-04-25T11:26:52.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.723+0000 7f212affd700 1 -- 172.21.15.90:0/1222248799 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f213002a070 con 0x7f21341089c0 2024-04-25T11:26:52.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.723+0000 7f213bc7a700 1 -- 172.21.15.90:0/1222248799 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2134103080 con 0x7f21341089c0 2024-04-25T11:26:52.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.724+0000 7f212affd700 1 --2- 172.21.15.90:0/1222248799 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f212003d670 0x7f212003fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:52.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.724+0000 7f212affd700 1 -- 172.21.15.90:0/1222248799 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f213001e030 con 0x7f21341089c0 2024-04-25T11:26:52.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.724+0000 7f2139215700 1 --2- 172.21.15.90:0/1222248799 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f212003d670 0x7f212003fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:52.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.724+0000 7f2139215700 1 --2- 172.21.15.90:0/1222248799 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f212003d670 0x7f212003fb80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2124006fd0 tx=0x7f2124006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:52.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.729+0000 7f212affd700 1 -- 172.21.15.90:0/1222248799 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2130013070 con 0x7f21341089c0 2024-04-25T11:26:52.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:52.931+0000 7f213bc7a700 1 -- 172.21.15.90:0/1222248799 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f21341a11d0 con 0x7f21341089c0 2024-04-25T11:26:53.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:53.719+0000 7f212affd700 1 -- 172.21.15.90:0/1222248799 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f21300292c0 con 0x7f21341089c0 2024-04-25T11:26:53.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:53.721+0000 7f213bc7a700 1 -- 172.21.15.90:0/1222248799 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f212003d670 msgr2=0x7f212003fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:53.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:53.721+0000 7f213bc7a700 1 --2- 172.21.15.90:0/1222248799 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f212003d670 0x7f212003fb80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2124006fd0 tx=0x7f2124006e40 comp rx=0 tx=0).stop 2024-04-25T11:26:53.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:53.721+0000 7f213bc7a700 1 -- 172.21.15.90:0/1222248799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21341089c0 msgr2=0x7f213419fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:53.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:53.721+0000 7f213bc7a700 1 --2- 172.21.15.90:0/1222248799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21341089c0 0x7f213419fe70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f21300056d0 tx=0x7f21300057b0 comp rx=0 tx=0).stop 2024-04-25T11:26:53.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:53.722+0000 7f213bc7a700 1 -- 172.21.15.90:0/1222248799 shutdown_connections 2024-04-25T11:26:53.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:53.722+0000 7f213bc7a700 1 --2- 172.21.15.90:0/1222248799 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f212003d670 0x7f212003fb80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:53.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:53.722+0000 7f213bc7a700 1 --2- 172.21.15.90:0/1222248799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21341089c0 0x7f213419fe70 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:53.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:53.722+0000 7f213bc7a700 1 -- 172.21.15.90:0/1222248799 >> 172.21.15.90:0/1222248799 conn(0x7f21340fdd00 msgr2=0x7f21340feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:53.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:53.722+0000 7f213bc7a700 1 -- 172.21.15.90:0/1222248799 shutdown_connections 2024-04-25T11:26:53.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:26:53.722+0000 7f213bc7a700 1 -- 172.21.15.90:0/1222248799 wait complete. 2024-04-25T11:26:53.715 INFO:teuthology.orchestra.run.smithi090.stderr:adjusted tunables profile to default 2024-04-25T11:26:53.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:53 smithi090 ceph-mon[29575]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:26:53.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:53 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1222248799' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T11:26:53.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:53 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:53.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:53 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:53.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:53 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:53.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:53 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:53.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:53 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi156", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:26:53.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:53 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:26:53.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:53 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:26:54.359 INFO:tasks.cephadm:Adding mon.a on smithi090 2024-04-25T11:26:54.359 INFO:tasks.cephadm:Adding mon.b on smithi138 2024-04-25T11:26:54.359 INFO:tasks.cephadm:Adding mon.c on smithi156 2024-04-25T11:26:54.360 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi090:172.21.15.90=a;smithi138:172.21.15.138=b;smithi156:172.21.15.156=c' 2024-04-25T11:26:54.597 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T11:26:54.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:54 smithi090 ceph-mon[29575]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-25T11:26:54.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:54 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1222248799' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T11:26:54.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:54 smithi090 ceph-mon[29575]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T11:26:54.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:54 smithi090 ceph-mon[29575]: Updating smithi156:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T11:26:54.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:54 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:54.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:54 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:54.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:54 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:55.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.917+0000 7fdb62c76700 1 -- 172.21.15.156:0/696180195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb5c108a30 msgr2=0x7fdb5c108e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:55.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.917+0000 7fdb62c76700 1 --2- 172.21.15.156:0/696180195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb5c108a30 0x7fdb5c108e40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdb50009910 tx=0x7fdb50009c20 comp rx=0 tx=0).stop 2024-04-25T11:26:55.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.917+0000 7fdb62c76700 1 -- 172.21.15.156:0/696180195 shutdown_connections 2024-04-25T11:26:55.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.917+0000 7fdb62c76700 1 --2- 172.21.15.156:0/696180195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb5c108a30 0x7fdb5c108e40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:55.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.917+0000 7fdb62c76700 1 -- 172.21.15.156:0/696180195 >> 172.21.15.156:0/696180195 conn(0x7fdb5c078c50 msgr2=0x7fdb5c079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:55.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.917+0000 7fdb62c76700 1 -- 172.21.15.156:0/696180195 shutdown_connections 2024-04-25T11:26:55.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.917+0000 7fdb62c76700 1 -- 172.21.15.156:0/696180195 wait complete. 2024-04-25T11:26:55.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.918+0000 7fdb62c76700 1 Processor -- start 2024-04-25T11:26:55.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.918+0000 7fdb62c76700 1 -- start start 2024-04-25T11:26:55.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.918+0000 7fdb62c76700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb5c108a30 0x7fdb5c19f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:55.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.918+0000 7fdb62c76700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdb5c10e0d0 con 0x7fdb5c108a30 2024-04-25T11:26:55.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.919+0000 7fdb60a12700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb5c108a30 0x7fdb5c19f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:55.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.919+0000 7fdb60a12700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb5c108a30 0x7fdb5c19f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.156:42228/0 (socket says 172.21.15.156:42228) 2024-04-25T11:26:55.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.919+0000 7fdb60a12700 1 -- 172.21.15.156:0/4279339712 learned_addr learned my addr 172.21.15.156:0/4279339712 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-25T11:26:55.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.919+0000 7fdb60a12700 1 -- 172.21.15.156:0/4279339712 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb5c19fa30 con 0x7fdb5c108a30 2024-04-25T11:26:55.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.919+0000 7fdb60a12700 1 --2- 172.21.15.156:0/4279339712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb5c108a30 0x7fdb5c19f4f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdb500056d0 tx=0x7fdb500057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:55.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.919+0000 7fdb59ffb700 1 -- 172.21.15.156:0/4279339712 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb50027070 con 0x7fdb5c108a30 2024-04-25T11:26:55.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.919+0000 7fdb59ffb700 1 -- 172.21.15.156:0/4279339712 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb5000be10 con 0x7fdb5c108a30 2024-04-25T11:26:55.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.919+0000 7fdb62c76700 1 -- 172.21.15.156:0/4279339712 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb5c19fcc0 con 0x7fdb5c108a30 2024-04-25T11:26:55.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.919+0000 7fdb59ffb700 1 -- 172.21.15.156:0/4279339712 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb50022d20 con 0x7fdb5c108a30 2024-04-25T11:26:55.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.920+0000 7fdb62c76700 1 -- 172.21.15.156:0/4279339712 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb5c199680 con 0x7fdb5c108a30 2024-04-25T11:26:55.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.920+0000 7fdb62c76700 1 -- 172.21.15.156:0/4279339712 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb5c105250 con 0x7fdb5c108a30 2024-04-25T11:26:55.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.920+0000 7fdb59ffb700 1 -- 172.21.15.156:0/4279339712 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fdb5002a070 con 0x7fdb5c108a30 2024-04-25T11:26:55.916 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.921+0000 7fdb59ffb700 1 --2- 172.21.15.156:0/4279339712 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fdb4c03d610 0x7fdb4c03fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:55.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.921+0000 7fdb59ffb700 1 -- 172.21.15.156:0/4279339712 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fdb5001e030 con 0x7fdb5c108a30 2024-04-25T11:26:55.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.921+0000 7fdb5bfff700 1 --2- 172.21.15.156:0/4279339712 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fdb4c03d610 0x7fdb4c03fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:55.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.922+0000 7fdb5bfff700 1 --2- 172.21.15.156:0/4279339712 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fdb4c03d610 0x7fdb4c03fb20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fdb48006fd0 tx=0x7fdb48006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:55.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:55.927+0000 7fdb59ffb700 1 -- 172.21.15.156:0/4279339712 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdb50013070 con 0x7fdb5c108a30 2024-04-25T11:26:55.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:55 smithi090 ceph-mon[29575]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-25T11:26:55.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:55 smithi090 ceph-mon[29575]: Updating smithi156:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T11:26:55.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:55 smithi090 ceph-mon[29575]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:26:56.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:56.135+0000 7fdb62c76700 1 -- 172.21.15.156:0/4279339712 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi090:172.21.15.90=a;smithi138:172.21.15.138=b;smithi156:172.21.15.156=c", "target": ["mon-mgr", ""]}) -- 0x7fdb5c19ab90 con 0x7fdb4c03d610 2024-04-25T11:26:56.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:56.143+0000 7fdb59ffb700 1 -- 172.21.15.156:0/4279339712 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fdb5c19ab90 con 0x7fdb4c03d610 2024-04-25T11:26:56.135 INFO:teuthology.orchestra.run.smithi156.stdout:Scheduled mon update... 2024-04-25T11:26:56.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:56.145+0000 7fdb62c76700 1 -- 172.21.15.156:0/4279339712 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fdb4c03d610 msgr2=0x7fdb4c03fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:56.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:56.145+0000 7fdb62c76700 1 --2- 172.21.15.156:0/4279339712 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fdb4c03d610 0x7fdb4c03fb20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fdb48006fd0 tx=0x7fdb48006e40 comp rx=0 tx=0).stop 2024-04-25T11:26:56.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:56.145+0000 7fdb62c76700 1 -- 172.21.15.156:0/4279339712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb5c108a30 msgr2=0x7fdb5c19f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:56.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:56.145+0000 7fdb62c76700 1 --2- 172.21.15.156:0/4279339712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb5c108a30 0x7fdb5c19f4f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdb500056d0 tx=0x7fdb500057b0 comp rx=0 tx=0).stop 2024-04-25T11:26:56.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:56.145+0000 7fdb62c76700 1 -- 172.21.15.156:0/4279339712 shutdown_connections 2024-04-25T11:26:56.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:56.145+0000 7fdb62c76700 1 --2- 172.21.15.156:0/4279339712 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fdb4c03d610 0x7fdb4c03fb20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:56.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:56.145+0000 7fdb62c76700 1 --2- 172.21.15.156:0/4279339712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb5c108a30 0x7fdb5c19f4f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:56.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:56.145+0000 7fdb62c76700 1 -- 172.21.15.156:0/4279339712 >> 172.21.15.156:0/4279339712 conn(0x7fdb5c078c50 msgr2=0x7fdb5c0ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:56.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:56.145+0000 7fdb62c76700 1 -- 172.21.15.156:0/4279339712 shutdown_connections 2024-04-25T11:26:56.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:26:56.145+0000 7fdb62c76700 1 -- 172.21.15.156:0/4279339712 wait complete. 2024-04-25T11:26:56.675 DEBUG:teuthology.orchestra.run.smithi138:mon.b> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mon.b.service 2024-04-25T11:26:56.678 DEBUG:teuthology.orchestra.run.smithi156:mon.c> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mon.c.service 2024-04-25T11:26:56.680 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T11:26:56.681 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T11:26:57.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:57 smithi090 ceph-mon[29575]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi090:172.21.15.90=a;smithi138:172.21.15.138=b;smithi156:172.21.15.156=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:26:57.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:57 smithi090 ceph-mon[29575]: Saving service mon spec with placement smithi090:172.21.15.90=a;smithi138:172.21.15.138=b;smithi156:172.21.15.156=c;count:3 2024-04-25T11:26:57.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:57 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:57.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:57 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:26:57.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:57 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:26:57.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:57 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:26:57.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:57 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:26:57.480 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:57 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T11:26:57.480 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:57 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:26:57.480 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:57 smithi090 ceph-mon[29575]: Deploying daemon mon.c on smithi156 2024-04-25T11:26:57.480 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:57 smithi090 ceph-mon[29575]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:26:58.593 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T11:26:59.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:26:59 smithi090 ceph-mon[29575]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:00.282 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.288+0000 7f104f55b700 1 -- 172.21.15.156:0/428122903 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1048108a30 msgr2=0x7f1048108e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:00.282 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.288+0000 7f104f55b700 1 --2- 172.21.15.156:0/428122903 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1048108a30 0x7f1048108e40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1038009910 tx=0x7f1038009c20 comp rx=0 tx=0).stop 2024-04-25T11:27:00.282 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.288+0000 7f104f55b700 1 -- 172.21.15.156:0/428122903 shutdown_connections 2024-04-25T11:27:00.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.288+0000 7f104f55b700 1 --2- 172.21.15.156:0/428122903 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1048108a30 0x7f1048108e40 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:00.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.288+0000 7f104f55b700 1 -- 172.21.15.156:0/428122903 >> 172.21.15.156:0/428122903 conn(0x7f10480fe4e0 msgr2=0x7f1048100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:27:00.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.288+0000 7f104f55b700 1 -- 172.21.15.156:0/428122903 shutdown_connections 2024-04-25T11:27:00.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.288+0000 7f104f55b700 1 -- 172.21.15.156:0/428122903 wait complete. 2024-04-25T11:27:00.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.289+0000 7f104f55b700 1 Processor -- start 2024-04-25T11:27:00.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.289+0000 7f104f55b700 1 -- start start 2024-04-25T11:27:00.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.289+0000 7f104f55b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1048108a30 0x7f104819a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:27:00.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.289+0000 7f104f55b700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f104810cf80 con 0x7f1048108a30 2024-04-25T11:27:00.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.290+0000 7f104d2f7700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1048108a30 0x7f104819a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:27:00.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.290+0000 7f104d2f7700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1048108a30 0x7f104819a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.156:42254/0 (socket says 172.21.15.156:42254) 2024-04-25T11:27:00.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.290+0000 7f104d2f7700 1 -- 172.21.15.156:0/3663363113 learned_addr learned my addr 172.21.15.156:0/3663363113 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-25T11:27:00.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.290+0000 7f104d2f7700 1 -- 172.21.15.156:0/3663363113 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f104806cae0 con 0x7f1048108a30 2024-04-25T11:27:00.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.290+0000 7f104d2f7700 1 --2- 172.21.15.156:0/3663363113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1048108a30 0x7f104819a7e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f10380056d0 tx=0x7f10380057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:27:00.285 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.290+0000 7f103e7fc700 1 -- 172.21.15.156:0/3663363113 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1038027070 con 0x7f1048108a30 2024-04-25T11:27:00.285 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.290+0000 7f103e7fc700 1 -- 172.21.15.156:0/3663363113 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f103800be10 con 0x7f1048108a30 2024-04-25T11:27:00.285 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.290+0000 7f104f55b700 1 -- 172.21.15.156:0/3663363113 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f104819ade0 con 0x7f1048108a30 2024-04-25T11:27:00.285 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.290+0000 7f103e7fc700 1 -- 172.21.15.156:0/3663363113 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1038022d20 con 0x7f1048108a30 2024-04-25T11:27:00.285 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.290+0000 7f104f55b700 1 -- 172.21.15.156:0/3663363113 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f104819b2a0 con 0x7f1048108a30 2024-04-25T11:27:00.285 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.291+0000 7f103e7fc700 1 -- 172.21.15.156:0/3663363113 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f103802a070 con 0x7f1048108a30 2024-04-25T11:27:00.285 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.292+0000 7f103e7fc700 1 --2- 172.21.15.156:0/3663363113 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f103403d610 0x7f103403fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:27:00.285 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.292+0000 7f104f55b700 1 -- 172.21.15.156:0/3663363113 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1048104140 con 0x7f1048108a30 2024-04-25T11:27:00.285 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.292+0000 7f103e7fc700 1 -- 172.21.15.156:0/3663363113 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f103801e030 con 0x7f1048108a30 2024-04-25T11:27:00.285 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.292+0000 7f104caf6700 1 --2- 172.21.15.156:0/3663363113 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f103403d610 0x7f103403fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:27:00.285 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.293+0000 7f104caf6700 1 --2- 172.21.15.156:0/3663363113 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f103403d610 0x7f103403fb20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1044006fd0 tx=0x7f1044006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:27:00.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.298+0000 7f103e7fc700 1 -- 172.21.15.156:0/3663363113 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1038031020 con 0x7f1048108a30 2024-04-25T11:27:00.514 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 podman[33004]: 2024-04-25 11:27:00.522806219 +0000 UTC m=+0.433837816 container init ad026314adcac9d5e4d29022c460be023c2c49020219cd7388aa04337b6eb872 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424) 2024-04-25T11:27:00.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.588+0000 7f104f55b700 1 -- 172.21.15.156:0/3663363113 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f104819b580 con 0x7f1048108a30 2024-04-25T11:27:00.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.589+0000 7f103e7fc700 1 -- 172.21.15.156:0/3663363113 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f104806cae0 con 0x7f1048108a30 2024-04-25T11:27:00.582 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T11:27:00.582 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":1,"fsid":"58bd4e7a-02f6-11ef-bc93-c7b262605968","modified":"2024-04-25T11:25:19.353596Z","created":"2024-04-25T11:25:19.353596Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-25T11:27:00.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.591+0000 7f104f55b700 1 -- 172.21.15.156:0/3663363113 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f103403d610 msgr2=0x7f103403fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:00.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.591+0000 7f104f55b700 1 --2- 172.21.15.156:0/3663363113 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f103403d610 0x7f103403fb20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1044006fd0 tx=0x7f1044006e40 comp rx=0 tx=0).stop 2024-04-25T11:27:00.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.591+0000 7f104f55b700 1 -- 172.21.15.156:0/3663363113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1048108a30 msgr2=0x7f104819a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:00.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.591+0000 7f104f55b700 1 --2- 172.21.15.156:0/3663363113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1048108a30 0x7f104819a7e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f10380056d0 tx=0x7f10380057b0 comp rx=0 tx=0).stop 2024-04-25T11:27:00.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.591+0000 7f104f55b700 1 -- 172.21.15.156:0/3663363113 shutdown_connections 2024-04-25T11:27:00.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.591+0000 7f104f55b700 1 --2- 172.21.15.156:0/3663363113 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f103403d610 0x7f103403fb20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:00.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.591+0000 7f104f55b700 1 --2- 172.21.15.156:0/3663363113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1048108a30 0x7f104819a7e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:00.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.591+0000 7f104f55b700 1 -- 172.21.15.156:0/3663363113 >> 172.21.15.156:0/3663363113 conn(0x7f10480fe4e0 msgr2=0x7f10480fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:27:00.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.591+0000 7f104f55b700 1 -- 172.21.15.156:0/3663363113 shutdown_connections 2024-04-25T11:27:00.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:00.591+0000 7f104f55b700 1 -- 172.21.15.156:0/3663363113 wait complete. 2024-04-25T11:27:00.585 INFO:teuthology.orchestra.run.smithi156.stderr:dumped monmap epoch 1 2024-04-25T11:27:00.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 podman[33004]: 2024-04-25 11:27:00.525934627 +0000 UTC m=+0.436966223 container start ad026314adcac9d5e4d29022c460be023c2c49020219cd7388aa04337b6eb872 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-c, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T11:27:00.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T11:27:00.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev), process ceph-mon, pid 2 2024-04-25T11:27:00.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: pidfile_write: ignore empty --pid-file 2024-04-25T11:27:00.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: load: jerasure load: lrc 2024-04-25T11:27:00.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: RocksDB version: 7.9.2 2024-04-25T11:27:00.843 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Git sha 0 2024-04-25T11:27:00.843 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Compile date 2024-04-25 02:57:34 2024-04-25T11:27:00.843 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: DB SUMMARY 2024-04-25T11:27:00.843 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: DB Session ID: TT0QV93PN8DMD11R6TFK 2024-04-25T11:27:00.843 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: CURRENT file: CURRENT 2024-04-25T11:27:00.843 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T11:27:00.843 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-25T11:27:00.843 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-25T11:27:00.843 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-25T11:27:00.843 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.error_if_exists: 0 2024-04-25T11:27:00.843 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.create_if_missing: 0 2024-04-25T11:27:00.843 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.paranoid_checks: 1 2024-04-25T11:27:00.843 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T11:27:00.844 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T11:27:00.844 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T11:27:00.844 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.env: 0x55623206a500 2024-04-25T11:27:00.844 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T11:27:00.844 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.info_log: 0x556233debfc0 2024-04-25T11:27:00.844 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T11:27:00.844 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.statistics: (nil) 2024-04-25T11:27:00.844 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.use_fsync: 0 2024-04-25T11:27:00.844 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.max_log_file_size: 0 2024-04-25T11:27:00.844 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T11:27:00.844 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T11:27:00.844 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T11:27:00.844 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T11:27:00.844 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.allow_fallocate: 1 2024-04-25T11:27:00.845 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T11:27:00.845 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T11:27:00.845 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.use_direct_reads: 0 2024-04-25T11:27:00.845 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T11:27:00.845 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T11:27:00.845 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.db_log_dir: 2024-04-25T11:27:00.845 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.wal_dir: 2024-04-25T11:27:00.845 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T11:27:00.845 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T11:27:00.845 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T11:27:00.845 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T11:27:00.845 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T11:27:00.845 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T11:27:00.845 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T11:27:00.846 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T11:27:00.846 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.write_buffer_manager: 0x556233dbc8c0 2024-04-25T11:27:00.846 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T11:27:00.846 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T11:27:00.846 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T11:27:00.846 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T11:27:00.846 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T11:27:00.846 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T11:27:00.846 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T11:27:00.846 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T11:27:00.846 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.unordered_write: 0 2024-04-25T11:27:00.846 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T11:27:00.846 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T11:27:00.847 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T11:27:00.847 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T11:27:00.847 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.row_cache: None 2024-04-25T11:27:00.847 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.wal_filter: None 2024-04-25T11:27:00.847 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T11:27:00.847 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T11:27:00.847 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.two_write_queues: 0 2024-04-25T11:27:00.847 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T11:27:00.847 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.wal_compression: 0 2024-04-25T11:27:00.847 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.atomic_flush: 0 2024-04-25T11:27:00.847 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T11:27:00.847 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T11:27:00.849 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T11:27:00.849 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.log_readahead_size: 0 2024-04-25T11:27:00.849 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T11:27:00.849 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T11:27:00.849 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T11:27:00.849 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T11:27:00.849 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T11:27:00.849 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T11:27:00.850 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T11:27:00.850 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.max_background_jobs: 2 2024-04-25T11:27:00.850 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.max_background_compactions: -1 2024-04-25T11:27:00.850 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.max_subcompactions: 1 2024-04-25T11:27:00.850 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T11:27:00.850 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T11:27:00.850 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T11:27:00.850 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T11:27:00.850 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T11:27:00.850 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T11:27:00.850 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T11:27:00.850 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T11:27:00.850 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.max_open_files: -1 2024-04-25T11:27:00.851 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T11:27:00.851 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T11:27:00.851 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T11:27:00.851 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T11:27:00.851 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.max_background_flushes: -1 2024-04-25T11:27:00.851 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Compression algorithms supported: 2024-04-25T11:27:00.851 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T11:27:00.851 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: kZSTD supported: 0 2024-04-25T11:27:00.851 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: kXpressCompression supported: 0 2024-04-25T11:27:00.851 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T11:27:00.851 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: kZlibCompression supported: 1 2024-04-25T11:27:00.851 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: kSnappyCompression supported: 1 2024-04-25T11:27:00.851 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: kLZ4Compression supported: 1 2024-04-25T11:27:00.851 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: kBZip2Compression supported: 0 2024-04-25T11:27:00.852 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T11:27:00.852 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T11:27:00.852 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-25T11:27:00.852 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T11:27:00.852 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T11:27:00.852 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.merge_operator: 2024-04-25T11:27:00.852 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.compaction_filter: None 2024-04-25T11:27:00.852 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T11:27:00.852 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T11:27:00.852 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T11:27:00.852 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T11:27:00.852 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556233ef21a0) 2024-04-25T11:27:00.852 INFO:journalctl@ceph.mon.c.smithi156.stdout: cache_index_and_filter_blocks: 1 2024-04-25T11:27:00.853 INFO:journalctl@ceph.mon.c.smithi156.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T11:27:00.853 INFO:journalctl@ceph.mon.c.smithi156.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T11:27:00.853 INFO:journalctl@ceph.mon.c.smithi156.stdout: pin_top_level_index_and_filter: 1 2024-04-25T11:27:00.853 INFO:journalctl@ceph.mon.c.smithi156.stdout: index_type: 0 2024-04-25T11:27:00.853 INFO:journalctl@ceph.mon.c.smithi156.stdout: data_block_index_type: 0 2024-04-25T11:27:00.853 INFO:journalctl@ceph.mon.c.smithi156.stdout: index_shortening: 1 2024-04-25T11:27:00.853 INFO:journalctl@ceph.mon.c.smithi156.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T11:27:00.853 INFO:journalctl@ceph.mon.c.smithi156.stdout: checksum: 4 2024-04-25T11:27:00.853 INFO:journalctl@ceph.mon.c.smithi156.stdout: no_block_cache: 0 2024-04-25T11:27:00.853 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_cache: 0x556233def1f0 2024-04-25T11:27:00.853 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_cache_name: BinnedLRUCache 2024-04-25T11:27:00.853 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_cache_options: 2024-04-25T11:27:00.853 INFO:journalctl@ceph.mon.c.smithi156.stdout: capacity : 536870912 2024-04-25T11:27:00.853 INFO:journalctl@ceph.mon.c.smithi156.stdout: num_shard_bits : 4 2024-04-25T11:27:00.853 INFO:journalctl@ceph.mon.c.smithi156.stdout: strict_capacity_limit : 0 2024-04-25T11:27:00.854 INFO:journalctl@ceph.mon.c.smithi156.stdout: high_pri_pool_ratio: 0.000 2024-04-25T11:27:00.854 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_cache_compressed: (nil) 2024-04-25T11:27:00.854 INFO:journalctl@ceph.mon.c.smithi156.stdout: persistent_cache: (nil) 2024-04-25T11:27:00.854 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_size: 4096 2024-04-25T11:27:00.854 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_size_deviation: 10 2024-04-25T11:27:00.854 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_restart_interval: 16 2024-04-25T11:27:00.854 INFO:journalctl@ceph.mon.c.smithi156.stdout: index_block_restart_interval: 1 2024-04-25T11:27:00.854 INFO:journalctl@ceph.mon.c.smithi156.stdout: metadata_block_size: 4096 2024-04-25T11:27:00.854 INFO:journalctl@ceph.mon.c.smithi156.stdout: partition_filters: 0 2024-04-25T11:27:00.854 INFO:journalctl@ceph.mon.c.smithi156.stdout: use_delta_encoding: 1 2024-04-25T11:27:00.854 INFO:journalctl@ceph.mon.c.smithi156.stdout: filter_policy: bloomfilter 2024-04-25T11:27:00.854 INFO:journalctl@ceph.mon.c.smithi156.stdout: whole_key_filtering: 1 2024-04-25T11:27:00.854 INFO:journalctl@ceph.mon.c.smithi156.stdout: verify_compression: 0 2024-04-25T11:27:00.855 INFO:journalctl@ceph.mon.c.smithi156.stdout: read_amp_bytes_per_bit: 0 2024-04-25T11:27:00.855 INFO:journalctl@ceph.mon.c.smithi156.stdout: format_version: 5 2024-04-25T11:27:00.855 INFO:journalctl@ceph.mon.c.smithi156.stdout: enable_index_compression: 1 2024-04-25T11:27:00.855 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_align: 0 2024-04-25T11:27:00.855 INFO:journalctl@ceph.mon.c.smithi156.stdout: max_auto_readahead_size: 262144 2024-04-25T11:27:00.855 INFO:journalctl@ceph.mon.c.smithi156.stdout: prepopulate_block_cache: 0 2024-04-25T11:27:00.855 INFO:journalctl@ceph.mon.c.smithi156.stdout: initial_auto_readahead_size: 8192 2024-04-25T11:27:00.855 INFO:journalctl@ceph.mon.c.smithi156.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T11:27:00.855 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T11:27:00.855 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T11:27:00.855 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.compression: NoCompression 2024-04-25T11:27:00.855 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T11:27:00.855 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T11:27:00.855 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T11:27:00.855 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.num_levels: 7 2024-04-25T11:27:00.856 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T11:27:00.856 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T11:27:00.856 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T11:27:00.856 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T11:27:00.856 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T11:27:00.856 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T11:27:00.856 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T11:27:00.856 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T11:27:00.856 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T11:27:00.856 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T11:27:00.856 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T11:27:00.856 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T11:27:00.856 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T11:27:00.857 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T11:27:00.857 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T11:27:00.857 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T11:27:00.857 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T11:27:00.857 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T11:27:00.857 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T11:27:00.857 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T11:27:00.857 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T11:27:00.857 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T11:27:00.857 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T11:27:00.857 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T11:27:00.857 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T11:27:00.857 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T11:27:00.857 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T11:27:00.858 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T11:27:00.858 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T11:27:00.858 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T11:27:00.858 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T11:27:00.858 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T11:27:00.858 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T11:27:00.858 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T11:27:00.858 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T11:27:00.858 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T11:27:00.858 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T11:27:00.858 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T11:27:00.858 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T11:27:00.858 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T11:27:00.859 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T11:27:00.859 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T11:27:00.859 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T11:27:00.859 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T11:27:00.859 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T11:27:00.859 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T11:27:00.859 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T11:27:00.859 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T11:27:00.859 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T11:27:00.859 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T11:27:00.859 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T11:27:00.859 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T11:27:00.859 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T11:27:00.859 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-25T11:27:00.860 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.inplace_update_support: 0 2024-04-25T11:27:00.860 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T11:27:00.860 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T11:27:00.860 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T11:27:00.860 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T11:27:00.860 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.bloom_locality: 0 2024-04-25T11:27:00.860 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.max_successive_merges: 0 2024-04-25T11:27:00.860 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T11:27:00.860 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T11:27:00.860 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T11:27:00.860 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T11:27:00.860 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.ttl: 2592000 2024-04-25T11:27:00.860 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T11:27:00.860 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T11:27:00.861 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T11:27:00.861 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.enable_blob_files: false 2024-04-25T11:27:00.861 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.min_blob_size: 0 2024-04-25T11:27:00.861 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T11:27:00.861 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T11:27:00.861 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T11:27:00.861 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T11:27:00.861 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T11:27:00.861 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T11:27:00.861 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T11:27:00.861 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T11:27:00.861 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-25T11:27:00.861 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-25T11:27:00.861 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ba729fdd-a723-4c32-b6ab-2eed15aebcd5 2024-04-25T11:27:00.862 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714044420569560, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-25T11:27:00.862 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-25T11:27:00.862 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714044420570141, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714044420, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ba729fdd-a723-4c32-b6ab-2eed15aebcd5", "db_session_id": "TT0QV93PN8DMD11R6TFK", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-25T11:27:00.862 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714044420570248, "job": 1, "event": "recovery_finished"} 2024-04-25T11:27:00.862 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-25T11:27:00.862 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T11:27:00.862 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556233dd3c00 2024-04-25T11:27:00.862 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: DB pointer 0x556238bc6000 2024-04-25T11:27:00.862 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T11:27:00.862 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T11:27:00.862 INFO:journalctl@ceph.mon.c.smithi156.stdout: ** DB Stats ** 2024-04-25T11:27:00.862 INFO:journalctl@ceph.mon.c.smithi156.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T11:27:00.862 INFO:journalctl@ceph.mon.c.smithi156.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T11:27:00.863 INFO:journalctl@ceph.mon.c.smithi156.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T11:27:00.863 INFO:journalctl@ceph.mon.c.smithi156.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T11:27:00.863 INFO:journalctl@ceph.mon.c.smithi156.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T11:27:00.863 INFO:journalctl@ceph.mon.c.smithi156.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T11:27:00.863 INFO:journalctl@ceph.mon.c.smithi156.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T11:27:00.863 INFO:journalctl@ceph.mon.c.smithi156.stdout: 2024-04-25T11:27:00.863 INFO:journalctl@ceph.mon.c.smithi156.stdout: ** Compaction Stats [default] ** 2024-04-25T11:27:00.863 INFO:journalctl@ceph.mon.c.smithi156.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T11:27:00.863 INFO:journalctl@ceph.mon.c.smithi156.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T11:27:00.863 INFO:journalctl@ceph.mon.c.smithi156.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T11:27:00.863 INFO:journalctl@ceph.mon.c.smithi156.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T11:27:00.863 INFO:journalctl@ceph.mon.c.smithi156.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T11:27:00.863 INFO:journalctl@ceph.mon.c.smithi156.stdout: 2024-04-25T11:27:00.864 INFO:journalctl@ceph.mon.c.smithi156.stdout: ** Compaction Stats [default] ** 2024-04-25T11:27:00.864 INFO:journalctl@ceph.mon.c.smithi156.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T11:27:00.864 INFO:journalctl@ceph.mon.c.smithi156.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T11:27:00.864 INFO:journalctl@ceph.mon.c.smithi156.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T11:27:00.864 INFO:journalctl@ceph.mon.c.smithi156.stdout: 2024-04-25T11:27:00.864 INFO:journalctl@ceph.mon.c.smithi156.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T11:27:00.864 INFO:journalctl@ceph.mon.c.smithi156.stdout: 2024-04-25T11:27:00.864 INFO:journalctl@ceph.mon.c.smithi156.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T11:27:00.864 INFO:journalctl@ceph.mon.c.smithi156.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T11:27:00.864 INFO:journalctl@ceph.mon.c.smithi156.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T11:27:00.864 INFO:journalctl@ceph.mon.c.smithi156.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T11:27:00.864 INFO:journalctl@ceph.mon.c.smithi156.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T11:27:00.864 INFO:journalctl@ceph.mon.c.smithi156.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T11:27:00.864 INFO:journalctl@ceph.mon.c.smithi156.stdout: Cumulative compaction: 0.00 GB write, 0.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T11:27:00.865 INFO:journalctl@ceph.mon.c.smithi156.stdout: Interval compaction: 0.00 GB write, 0.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T11:27:00.865 INFO:journalctl@ceph.mon.c.smithi156.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T11:27:00.865 INFO:journalctl@ceph.mon.c.smithi156.stdout: Block cache BinnedLRUCache@0x556233def1f0#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-25T11:27:00.865 INFO:journalctl@ceph.mon.c.smithi156.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-25T11:27:00.865 INFO:journalctl@ceph.mon.c.smithi156.stdout: 2024-04-25T11:27:00.865 INFO:journalctl@ceph.mon.c.smithi156.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T11:27:00.865 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-25T11:27:00.865 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: using public_addr v2:172.21.15.156:0/0 -> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] 2024-04-25T11:27:00.865 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: starting mon.c rank -1 at public addrs [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] at bind addrs [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:27:00.865 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: mon.c@-1(???) e0 preinit fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:27:00.865 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 bash[33004]: ad026314adcac9d5e4d29022c460be023c2c49020219cd7388aa04337b6eb872 2024-04-25T11:27:00.865 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 systemd[1]: Started Ceph mon.c for 58bd4e7a-02f6-11ef-bc93-c7b262605968. 2024-04-25T11:27:00.865 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: mon.c@-1(synchronizing).mds e1 new map 2024-04-25T11:27:00.866 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-25T11:27:00.866 INFO:journalctl@ceph.mon.c.smithi156.stdout: e1 2024-04-25T11:27:00.866 INFO:journalctl@ceph.mon.c.smithi156.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T11:27:00.866 INFO:journalctl@ceph.mon.c.smithi156.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T11:27:00.866 INFO:journalctl@ceph.mon.c.smithi156.stdout: legacy client fscid: -1 2024-04-25T11:27:00.866 INFO:journalctl@ceph.mon.c.smithi156.stdout: 2024-04-25T11:27:00.866 INFO:journalctl@ceph.mon.c.smithi156.stdout: No filesystems configured 2024-04-25T11:27:00.866 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-25T11:27:00.866 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-25T11:27:00.866 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-25T11:27:00.866 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-25T11:27:00.866 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-25T11:27:00.866 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-25T11:27:00.867 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T11:27:00.867 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T11:27:00.867 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T11:27:00.867 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T11:27:00.867 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T11:27:00.867 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:00.867 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi156", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:27:00.867 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:00.867 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:00.867 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:00.867 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:00.867 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi138", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:27:00.867 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:00.867 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:27:00.867 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: Updating smithi138:/etc/ceph/ceph.conf 2024-04-25T11:27:00.868 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: Deploying cephadm binary to smithi156 2024-04-25T11:27:00.868 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: Updating smithi138:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T11:27:00.868 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:00.868 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: Updating smithi138:/etc/ceph/ceph.client.admin.keyring 2024-04-25T11:27:00.868 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:00.868 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: Updating smithi138:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T11:27:00.868 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:00.868 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:00.868 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:00.868 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:00.868 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:27:00.868 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:00.868 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: Added host smithi156 2024-04-25T11:27:00.869 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:00.869 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:00.869 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T11:27:00.869 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:00.869 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:00.869 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:00.869 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1222248799' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T11:27:00.869 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:00.869 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:00.869 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:00.869 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:00.869 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi156", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:27:00.869 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:00.869 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:27:00.870 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-25T11:27:00.870 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1222248799' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T11:27:00.870 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T11:27:00.870 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: Updating smithi156:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T11:27:00.870 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:00.870 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:00.870 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:00.870 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-25T11:27:00.870 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: Updating smithi156:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T11:27:00.870 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:00.870 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi090:172.21.15.90=a;smithi138:172.21.15.138=b;smithi156:172.21.15.156=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:27:00.870 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: Saving service mon spec with placement smithi090:172.21.15.90=a;smithi138:172.21.15.138=b;smithi156:172.21.15.156=c;count:3 2024-04-25T11:27:00.870 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:00.870 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:27:00.871 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:00.871 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:27:00.871 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:00.871 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T11:27:00.871 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:00.871 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: Deploying daemon mon.c on smithi156 2024-04-25T11:27:00.871 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:00.871 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:00.871 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:00 smithi156 ceph-mon[33045]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-25T11:27:02.240 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T11:27:02.240 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T11:27:02.482 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T11:27:05.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T11:27:05.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:27:05.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: mon.a calling monitor election 2024-04-25T11:27:05.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:05.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:27:05.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:27:05.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: mon.c calling monitor election 2024-04-25T11:27:05.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:05.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:27:05.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:27:05.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:05.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:27:05.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T11:27:05.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: monmap epoch 2 2024-04-25T11:27:05.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:27:05.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: last_changed 2024-04-25T11:27:00.598647+0000 2024-04-25T11:27:05.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: created 2024-04-25T11:25:19.353596+0000 2024-04-25T11:27:05.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: min_mon_release 19 (squid) 2024-04-25T11:27:05.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: election_strategy: 1 2024-04-25T11:27:05.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: 0: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.a 2024-04-25T11:27:05.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: 1: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.c 2024-04-25T11:27:05.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: fsmap 2024-04-25T11:27:05.981 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T11:27:05.981 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: mgrmap e13: x(active, since 48s) 2024-04-25T11:27:05.981 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: overall HEALTH_OK 2024-04-25T11:27:05.981 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:05.981 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:06.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T11:27:06.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:27:06.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: mon.a calling monitor election 2024-04-25T11:27:06.092 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:06.092 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:27:06.092 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:27:06.092 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: mon.c calling monitor election 2024-04-25T11:27:06.092 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:06.092 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:27:06.092 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:27:06.092 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:06.092 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:27:06.092 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T11:27:06.092 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: monmap epoch 2 2024-04-25T11:27:06.092 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:27:06.092 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: last_changed 2024-04-25T11:27:00.598647+0000 2024-04-25T11:27:06.092 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: created 2024-04-25T11:25:19.353596+0000 2024-04-25T11:27:06.093 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: min_mon_release 19 (squid) 2024-04-25T11:27:06.093 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: election_strategy: 1 2024-04-25T11:27:06.093 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: 0: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.a 2024-04-25T11:27:06.093 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: 1: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.c 2024-04-25T11:27:06.093 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: fsmap 2024-04-25T11:27:06.093 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T11:27:06.093 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: mgrmap e13: x(active, since 48s) 2024-04-25T11:27:06.093 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: overall HEALTH_OK 2024-04-25T11:27:06.093 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:06.093 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:06.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:06 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:06.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:06 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T11:27:06.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:06 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:06.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:06 smithi090 ceph-mon[29575]: Deploying daemon mon.b on smithi138 2024-04-25T11:27:06.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:06 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:27:07.086 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.094+0000 7fbfc3e00700 1 -- 172.21.15.156:0/3708042740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbfa80035c0 msgr2=0x7fbfa8005aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:07.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.094+0000 7fbfc3e00700 1 --2- 172.21.15.156:0/3708042740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbfa80035c0 0x7fbfa8005aa0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc103a20 tx=0x7fbfac00feb0 comp rx=0 tx=0).stop 2024-04-25T11:27:07.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.094+0000 7fbfc3e00700 1 -- 172.21.15.156:0/3708042740 shutdown_connections 2024-04-25T11:27:07.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.094+0000 7fbfc3e00700 1 --2- 172.21.15.156:0/3708042740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbfa80035c0 0x7fbfa8005aa0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:07.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.094+0000 7fbfc3e00700 1 -- 172.21.15.156:0/3708042740 >> 172.21.15.156:0/3708042740 conn(0x7fbfbc0ff650 msgr2=0x7fbfbc101aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:27:07.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.094+0000 7fbfc3e00700 1 -- 172.21.15.156:0/3708042740 shutdown_connections 2024-04-25T11:27:07.087 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.094+0000 7fbfc3e00700 1 -- 172.21.15.156:0/3708042740 wait complete. 2024-04-25T11:27:07.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:06 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:07.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:06 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T11:27:07.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:06 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:07.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:06 smithi156 ceph-mon[33045]: Deploying daemon mon.b on smithi138 2024-04-25T11:27:07.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:06 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:27:07.090 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.096+0000 7fbfc3e00700 1 Processor -- start 2024-04-25T11:27:07.090 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.096+0000 7fbfc3e00700 1 -- start start 2024-04-25T11:27:07.091 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.096+0000 7fbfc3e00700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbfa80035c0 0x7fbfbc19f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:27:07.091 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.096+0000 7fbfc3e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbfbc19fae0 0x7fbfbc199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:27:07.091 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.096+0000 7fbfc3e00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbfbc10bad0 con 0x7fbfbc19fae0 2024-04-25T11:27:07.091 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.096+0000 7fbfc3e00700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbfbc10b950 con 0x7fbfa80035c0 2024-04-25T11:27:07.091 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.097+0000 7fbfc1b9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbfa80035c0 0x7fbfbc19f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:27:07.091 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.097+0000 7fbfc1b9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbfa80035c0 0x7fbfbc19f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39598/0 (socket says 172.21.15.156:39598) 2024-04-25T11:27:07.091 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.097+0000 7fbfc1b9c700 1 -- 172.21.15.156:0/533075319 learned_addr learned my addr 172.21.15.156:0/533075319 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-25T11:27:07.091 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.097+0000 7fbfc139b700 1 --2- 172.21.15.156:0/533075319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbfbc19fae0 0x7fbfbc199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:27:07.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.097+0000 7fbfc1b9c700 1 -- 172.21.15.156:0/533075319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbfa80035c0 msgr2=0x7fbfbc19f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-25T11:27:07.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.097+0000 7fbfc1b9c700 1 -- 172.21.15.156:0/533075319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbfa80035c0 msgr2=0x7fbfbc19f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T11:27:07.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.097+0000 7fbfc1b9c700 1 --2- 172.21.15.156:0/533075319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbfa80035c0 0x7fbfbc19f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T11:27:07.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.097+0000 7fbfc1b9c700 1 --2- 172.21.15.156:0/533075319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbfa80035c0 0x7fbfbc19f5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T11:27:07.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.097+0000 7fbfc139b700 1 -- 172.21.15.156:0/533075319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbfa80035c0 msgr2=0x7fbfbc19f5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:27:07.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.097+0000 7fbfc139b700 1 --2- 172.21.15.156:0/533075319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbfa80035c0 0x7fbfbc19f5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:07.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.097+0000 7fbfc139b700 1 -- 172.21.15.156:0/533075319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfbc1a0080 con 0x7fbfbc19fae0 2024-04-25T11:27:07.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.097+0000 7fbfc139b700 1 --2- 172.21.15.156:0/533075319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbfbc19fae0 0x7fbfbc199730 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbfb800b4d0 tx=0x7fbfb800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:27:07.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.098+0000 7fbfb2ffd700 1 -- 172.21.15.156:0/533075319 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fbfb8011630 con 0x7fbfbc19fae0 2024-04-25T11:27:07.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.098+0000 7fbfb2ffd700 1 -- 172.21.15.156:0/533075319 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfb8011c80 con 0x7fbfbc19fae0 2024-04-25T11:27:07.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.098+0000 7fbfc3e00700 1 -- 172.21.15.156:0/533075319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfbc199d30 con 0x7fbfbc19fae0 2024-04-25T11:27:07.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.098+0000 7fbfb2ffd700 1 -- 172.21.15.156:0/533075319 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fbfb8010da0 con 0x7fbfbc19fae0 2024-04-25T11:27:07.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.098+0000 7fbfc3e00700 1 -- 172.21.15.156:0/533075319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfbc19a2c0 con 0x7fbfbc19fae0 2024-04-25T11:27:07.093 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.099+0000 7fbfb2ffd700 1 -- 172.21.15.156:0/533075319 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fbfb8018020 con 0x7fbfbc19fae0 2024-04-25T11:27:07.093 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.099+0000 7fbfc3e00700 1 -- 172.21.15.156:0/533075319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfbc19a5d0 con 0x7fbfbc19fae0 2024-04-25T11:27:07.093 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.099+0000 7fbfb2ffd700 1 --2- 172.21.15.156:0/533075319 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbfa003d400 0x7fbfa003f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:27:07.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.099+0000 7fbfb2ffd700 1 -- 172.21.15.156:0/533075319 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fbfb8050ff0 con 0x7fbfbc19fae0 2024-04-25T11:27:07.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.099+0000 7fbfc1b9c700 1 --2- 172.21.15.156:0/533075319 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbfa003d400 0x7fbfa003f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:27:07.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.100+0000 7fbfc1b9c700 1 --2- 172.21.15.156:0/533075319 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbfa003d400 0x7fbfa003f910 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbfac00fa60 tx=0x7fbfac005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:27:07.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.105+0000 7fbfb2ffd700 1 -- 172.21.15.156:0/533075319 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbfb801f020 con 0x7fbfbc19fae0 2024-04-25T11:27:07.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.386+0000 7fbfc3e00700 1 -- 172.21.15.156:0/533075319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fbfbc064d20 con 0x7fbfbc19fae0 2024-04-25T11:27:07.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.387+0000 7fbfb2ffd700 1 -- 172.21.15.156:0/533075319 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7fbfbc064d20 con 0x7fbfbc19fae0 2024-04-25T11:27:07.379 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T11:27:07.379 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":2,"fsid":"58bd4e7a-02f6-11ef-bc93-c7b262605968","modified":"2024-04-25T11:27:00.598647Z","created":"2024-04-25T11:25:19.353596Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:3300","nonce":0},{"type":"v1","addr":"172.21.15.156:6789","nonce":0}]},"addr":"172.21.15.156:6789/0","public_addr":"172.21.15.156:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-25T11:27:07.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.389+0000 7fbfc3e00700 1 -- 172.21.15.156:0/533075319 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbfa003d400 msgr2=0x7fbfa003f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:07.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.389+0000 7fbfc3e00700 1 --2- 172.21.15.156:0/533075319 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbfa003d400 0x7fbfa003f910 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbfac00fa60 tx=0x7fbfac005960 comp rx=0 tx=0).stop 2024-04-25T11:27:07.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.389+0000 7fbfc3e00700 1 -- 172.21.15.156:0/533075319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbfbc19fae0 msgr2=0x7fbfbc199730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:07.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.389+0000 7fbfc3e00700 1 --2- 172.21.15.156:0/533075319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbfbc19fae0 0x7fbfbc199730 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbfb800b4d0 tx=0x7fbfb800b890 comp rx=0 tx=0).stop 2024-04-25T11:27:07.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.389+0000 7fbfc3e00700 1 -- 172.21.15.156:0/533075319 shutdown_connections 2024-04-25T11:27:07.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.389+0000 7fbfc3e00700 1 --2- 172.21.15.156:0/533075319 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbfa003d400 0x7fbfa003f910 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:07.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.389+0000 7fbfc3e00700 1 --2- 172.21.15.156:0/533075319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbfa80035c0 0x7fbfbc19f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:07.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.389+0000 7fbfc3e00700 1 --2- 172.21.15.156:0/533075319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbfbc19fae0 0x7fbfbc199730 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:07.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.389+0000 7fbfc3e00700 1 -- 172.21.15.156:0/533075319 >> 172.21.15.156:0/533075319 conn(0x7fbfbc0ff650 msgr2=0x7fbfbc1070d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:27:07.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.389+0000 7fbfc3e00700 1 -- 172.21.15.156:0/533075319 shutdown_connections 2024-04-25T11:27:07.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:07.389+0000 7fbfc3e00700 1 -- 172.21.15.156:0/533075319 wait complete. 2024-04-25T11:27:07.383 INFO:teuthology.orchestra.run.smithi156.stderr:dumped monmap epoch 2 2024-04-25T11:27:07.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:07 smithi090 ceph-mon[29575]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:07.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:07 smithi090 ceph-mon[29575]: from='client.? 172.21.15.156:0/533075319' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T11:27:07.979 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:27:07 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:27:07.597+0000 7f83fa652700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-25T11:27:08.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:07 smithi156 ceph-mon[33045]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:08.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:07 smithi156 ceph-mon[33045]: from='client.? 172.21.15.156:0/533075319' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T11:27:08.881 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T11:27:08.882 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T11:27:09.124 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T11:27:09.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:09 smithi138 systemd[1]: Started Ceph mon.b for 58bd4e7a-02f6-11ef-bc93-c7b262605968. 2024-04-25T11:27:14.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: mon.c calling monitor election 2024-04-25T11:27:14.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T11:27:14.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: mon.a calling monitor election 2024-04-25T11:27:14.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:27:14.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:27:14.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:27:14.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:14.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:27:14.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:27:14.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:14.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:27:14.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:27:14.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T11:27:14.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: monmap epoch 3 2024-04-25T11:27:14.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:27:14.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: last_changed 2024-04-25T11:27:09.616344+0000 2024-04-25T11:27:14.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: created 2024-04-25T11:25:19.353596+0000 2024-04-25T11:27:14.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: min_mon_release 19 (squid) 2024-04-25T11:27:14.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: election_strategy: 1 2024-04-25T11:27:14.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: 0: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.a 2024-04-25T11:27:14.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: 1: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.c 2024-04-25T11:27:14.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: 2: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.b 2024-04-25T11:27:14.980 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: fsmap 2024-04-25T11:27:14.981 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T11:27:14.981 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: mgrmap e13: x(active, since 57s) 2024-04-25T11:27:14.981 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: overall HEALTH_OK 2024-04-25T11:27:14.981 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:14.981 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:14.981 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:14.981 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:14 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:15.088 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.095+0000 7f17adb57700 1 -- 172.21.15.156:0/3659455698 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17940035c0 msgr2=0x7f1794005aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:15.089 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.095+0000 7f17adb57700 1 --2- 172.21.15.156:0/3659455698 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17940035c0 0x7f1794005aa0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f17a8103a30 tx=0x7f179800fe60 comp rx=0 tx=0).stop 2024-04-25T11:27:15.089 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.096+0000 7f17adb57700 1 -- 172.21.15.156:0/3659455698 shutdown_connections 2024-04-25T11:27:15.089 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.096+0000 7f17adb57700 1 --2- 172.21.15.156:0/3659455698 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17940035c0 0x7f1794005aa0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:15.089 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.096+0000 7f17adb57700 1 -- 172.21.15.156:0/3659455698 >> 172.21.15.156:0/3659455698 conn(0x7f17a80ff640 msgr2=0x7f17a8101a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:27:15.089 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.096+0000 7f17adb57700 1 -- 172.21.15.156:0/3659455698 shutdown_connections 2024-04-25T11:27:15.089 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.096+0000 7f17adb57700 1 -- 172.21.15.156:0/3659455698 wait complete. 2024-04-25T11:27:15.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: mon.c calling monitor election 2024-04-25T11:27:15.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T11:27:15.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: mon.a calling monitor election 2024-04-25T11:27:15.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:27:15.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:27:15.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:27:15.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:15.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:27:15.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:27:15.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:15.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:27:15.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:27:15.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T11:27:15.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: monmap epoch 3 2024-04-25T11:27:15.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:27:15.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: last_changed 2024-04-25T11:27:09.616344+0000 2024-04-25T11:27:15.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: created 2024-04-25T11:25:19.353596+0000 2024-04-25T11:27:15.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: min_mon_release 19 (squid) 2024-04-25T11:27:15.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: election_strategy: 1 2024-04-25T11:27:15.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: 0: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.a 2024-04-25T11:27:15.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: 1: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.c 2024-04-25T11:27:15.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: 2: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.b 2024-04-25T11:27:15.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: fsmap 2024-04-25T11:27:15.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T11:27:15.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: mgrmap e13: x(active, since 57s) 2024-04-25T11:27:15.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: overall HEALTH_OK 2024-04-25T11:27:15.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:15.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:15.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:15.092 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:14 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:15.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.097+0000 7f17adb57700 1 Processor -- start 2024-04-25T11:27:15.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.097+0000 7f17adb57700 1 -- start start 2024-04-25T11:27:15.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.097+0000 7f17adb57700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17940035c0 0x7f17a819f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:27:15.095 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.097+0000 7f17adb57700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17a819fc50 0x7f17a8199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:27:15.095 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.097+0000 7f17adb57700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17a8199e70 0x7f17a819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:27:15.095 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.097+0000 7f17adb57700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f17a810ba90 con 0x7f17a819fc50 2024-04-25T11:27:15.095 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.097+0000 7f17adb57700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f17a810b910 con 0x7f17a8199e70 2024-04-25T11:27:15.095 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.097+0000 7f17adb57700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f17a810bc10 con 0x7f17940035c0 2024-04-25T11:27:15.095 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.097+0000 7f17a77fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17940035c0 0x7f17a819f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:27:15.095 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.097+0000 7f17a77fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17940035c0 0x7f17a819f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:58192/0 (socket says 172.21.15.156:58192) 2024-04-25T11:27:15.095 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.097+0000 7f17a77fe700 1 -- 172.21.15.156:0/932183286 learned_addr learned my addr 172.21.15.156:0/932183286 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-25T11:27:15.095 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.098+0000 7f17a77fe700 1 -- 172.21.15.156:0/932183286 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17940035c0 msgr2=0x7f17a819f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-25T11:27:15.095 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.098+0000 7f17a77fe700 1 -- 172.21.15.156:0/932183286 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17940035c0 msgr2=0x7f17a819f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T11:27:15.095 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.098+0000 7f17a77fe700 1 --2- 172.21.15.156:0/932183286 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17940035c0 0x7f17a819f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T11:27:15.095 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.098+0000 7f17a77fe700 1 --2- 172.21.15.156:0/932183286 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17940035c0 0x7f17a819f710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T11:27:15.095 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.098+0000 7f179ffff700 1 --2- 172.21.15.156:0/932183286 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17a819fc50 0x7f17a8199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:27:15.095 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.098+0000 7f17a7fff700 1 --2- 172.21.15.156:0/932183286 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17a8199e70 0x7f17a819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:27:15.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.098+0000 7f17a7fff700 1 -- 172.21.15.156:0/932183286 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17a8199e70 msgr2=0x7f17a819a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-25T11:27:15.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.098+0000 7f17a7fff700 1 -- 172.21.15.156:0/932183286 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17a8199e70 msgr2=0x7f17a819a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T11:27:15.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.098+0000 7f17a7fff700 1 --2- 172.21.15.156:0/932183286 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17a8199e70 0x7f17a819a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T11:27:15.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.098+0000 7f17a7fff700 1 --2- 172.21.15.156:0/932183286 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17a8199e70 0x7f17a819a280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T11:27:15.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.098+0000 7f179ffff700 1 -- 172.21.15.156:0/932183286 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17940035c0 msgr2=0x7f17a819f710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:27:15.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.098+0000 7f179ffff700 1 --2- 172.21.15.156:0/932183286 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17940035c0 0x7f17a819f710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:15.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.098+0000 7f179ffff700 1 -- 172.21.15.156:0/932183286 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17a8199e70 msgr2=0x7f17a819a280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:27:15.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.098+0000 7f179ffff700 1 --2- 172.21.15.156:0/932183286 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17a8199e70 0x7f17a819a280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:15.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.098+0000 7f179ffff700 1 -- 172.21.15.156:0/932183286 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a81a65a0 con 0x7f17a819fc50 2024-04-25T11:27:15.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.098+0000 7f179ffff700 1 --2- 172.21.15.156:0/932183286 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17a819fc50 0x7f17a8199930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f179000b550 tx=0x7f179000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:27:15.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.098+0000 7f17a57fa700 1 -- 172.21.15.156:0/932183286 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1790004230 con 0x7f17a819fc50 2024-04-25T11:27:15.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.099+0000 7f17adb57700 1 -- 172.21.15.156:0/932183286 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17a81a6830 con 0x7f17a819fc50 2024-04-25T11:27:15.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.098+0000 7f17a57fa700 1 -- 172.21.15.156:0/932183286 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17900043d0 con 0x7f17a819fc50 2024-04-25T11:27:15.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.099+0000 7f17a57fa700 1 -- 172.21.15.156:0/932183286 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1790011b90 con 0x7f17a819fc50 2024-04-25T11:27:15.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.099+0000 7f17adb57700 1 -- 172.21.15.156:0/932183286 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17a81a6d40 con 0x7f17a819fc50 2024-04-25T11:27:15.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.100+0000 7f17adb57700 1 -- 172.21.15.156:0/932183286 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17a8064cf0 con 0x7f17a819fc50 2024-04-25T11:27:15.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.100+0000 7f17a57fa700 1 -- 172.21.15.156:0/932183286 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f179001a400 con 0x7f17a819fc50 2024-04-25T11:27:15.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.100+0000 7f17a57fa700 1 --2- 172.21.15.156:0/932183286 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f178803daa0 0x7f178803ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:27:15.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.100+0000 7f17a57fa700 1 -- 172.21.15.156:0/932183286 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f1790051140 con 0x7f17a819fc50 2024-04-25T11:27:15.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.100+0000 7f17a77fe700 1 --2- 172.21.15.156:0/932183286 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f178803daa0 0x7f178803ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:27:15.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.101+0000 7f17a77fe700 1 --2- 172.21.15.156:0/932183286 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f178803daa0 0x7f178803ffb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f179800fa10 tx=0x7f179800f4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:27:15.098 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.106+0000 7f17a57fa700 1 -- 172.21.15.156:0/932183286 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1790020740 con 0x7f17a819fc50 2024-04-25T11:27:15.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.419+0000 7f17adb57700 1 -- 172.21.15.156:0/932183286 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f17a802e510 con 0x7f17a819fc50 2024-04-25T11:27:15.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.419+0000 7f17a57fa700 1 -- 172.21.15.156:0/932183286 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7f179001a890 con 0x7f17a819fc50 2024-04-25T11:27:15.412 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-25T11:27:15.412 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":3,"fsid":"58bd4e7a-02f6-11ef-bc93-c7b262605968","modified":"2024-04-25T11:27:09.616344Z","created":"2024-04-25T11:25:19.353596Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:3300","nonce":0},{"type":"v1","addr":"172.21.15.156:6789","nonce":0}]},"addr":"172.21.15.156:6789/0","public_addr":"172.21.15.156:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-25T11:27:15.414 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.422+0000 7f17adb57700 1 -- 172.21.15.156:0/932183286 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f178803daa0 msgr2=0x7f178803ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:15.414 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.422+0000 7f17adb57700 1 --2- 172.21.15.156:0/932183286 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f178803daa0 0x7f178803ffb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f179800fa10 tx=0x7f179800f4a0 comp rx=0 tx=0).stop 2024-04-25T11:27:15.414 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.422+0000 7f17adb57700 1 -- 172.21.15.156:0/932183286 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17a819fc50 msgr2=0x7f17a8199930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:15.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.422+0000 7f17adb57700 1 --2- 172.21.15.156:0/932183286 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17a819fc50 0x7f17a8199930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f179000b550 tx=0x7f179000b910 comp rx=0 tx=0).stop 2024-04-25T11:27:15.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.422+0000 7f17adb57700 1 -- 172.21.15.156:0/932183286 shutdown_connections 2024-04-25T11:27:15.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.422+0000 7f17adb57700 1 --2- 172.21.15.156:0/932183286 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17a8199e70 0x7f17a819a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:15.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.422+0000 7f17adb57700 1 --2- 172.21.15.156:0/932183286 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f178803daa0 0x7f178803ffb0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:15.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.422+0000 7f17adb57700 1 --2- 172.21.15.156:0/932183286 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17940035c0 0x7f17a819f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:15.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.422+0000 7f17adb57700 1 --2- 172.21.15.156:0/932183286 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17a819fc50 0x7f17a8199930 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:15.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.422+0000 7f17adb57700 1 -- 172.21.15.156:0/932183286 >> 172.21.15.156:0/932183286 conn(0x7f17a80ff640 msgr2=0x7f17a8107400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:27:15.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.422+0000 7f17adb57700 1 -- 172.21.15.156:0/932183286 shutdown_connections 2024-04-25T11:27:15.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:15.422+0000 7f17adb57700 1 -- 172.21.15.156:0/932183286 wait complete. 2024-04-25T11:27:15.416 INFO:teuthology.orchestra.run.smithi156.stderr:dumped monmap epoch 3 2024-04-25T11:27:15.966 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:15 smithi156 ceph-mon[33045]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:15.966 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:15 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:27:15.966 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:15 smithi156 ceph-mon[33045]: from='client.? 172.21.15.156:0/932183286' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T11:27:15.966 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:15 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:27:15.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:15 smithi090 ceph-mon[29575]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:15.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:15 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:27:15.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:15 smithi090 ceph-mon[29575]: from='client.? 172.21.15.156:0/932183286' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T11:27:15.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:15 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:27:15.997 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-25T11:27:15.997 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-25T11:27:16.242 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:27:17.028 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: mon.c calling monitor election 2024-04-25T11:27:17.028 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T11:27:17.028 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: mon.a calling monitor election 2024-04-25T11:27:17.028 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:27:17.028 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:27:17.028 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:27:17.028 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:17.028 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:27:17.028 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:27:17.028 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:17.028 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:27:17.029 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:27:17.029 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T11:27:17.029 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: monmap epoch 3 2024-04-25T11:27:17.029 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:27:17.029 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: last_changed 2024-04-25T11:27:09.616344+0000 2024-04-25T11:27:17.029 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: created 2024-04-25T11:25:19.353596+0000 2024-04-25T11:27:17.029 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: min_mon_release 19 (squid) 2024-04-25T11:27:17.029 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: election_strategy: 1 2024-04-25T11:27:17.029 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: 0: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.a 2024-04-25T11:27:17.029 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: 1: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.c 2024-04-25T11:27:17.029 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: 2: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.b 2024-04-25T11:27:17.029 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: fsmap 2024-04-25T11:27:17.029 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T11:27:17.030 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: mgrmap e13: x(active, since 57s) 2024-04-25T11:27:17.030 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: overall HEALTH_OK 2024-04-25T11:27:17.030 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:17.030 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:17.030 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:17.030 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:17.030 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:17.030 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:27:17.030 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: from='client.? 172.21.15.156:0/932183286' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T11:27:17.030 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:27:17.030 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: mon.b calling monitor election 2024-04-25T11:27:17.031 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: mon.b calling monitor election 2024-04-25T11:27:17.031 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: mon.a calling monitor election 2024-04-25T11:27:17.031 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: mon.c calling monitor election 2024-04-25T11:27:17.031 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T11:27:17.031 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: monmap epoch 3 2024-04-25T11:27:17.031 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:27:17.031 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: last_changed 2024-04-25T11:27:09.616344+0000 2024-04-25T11:27:17.031 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: created 2024-04-25T11:25:19.353596+0000 2024-04-25T11:27:17.031 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: min_mon_release 19 (squid) 2024-04-25T11:27:17.031 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: election_strategy: 1 2024-04-25T11:27:17.031 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: 0: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.a 2024-04-25T11:27:17.031 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: 1: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.c 2024-04-25T11:27:17.031 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: 2: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.b 2024-04-25T11:27:17.031 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: fsmap 2024-04-25T11:27:17.032 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T11:27:17.032 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: mgrmap e13: x(active, since 59s) 2024-04-25T11:27:17.032 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: overall HEALTH_OK 2024-04-25T11:27:17.032 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:17.032 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:17.032 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:17.032 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:27:17.032 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: Updating smithi090:/etc/ceph/ceph.conf 2024-04-25T11:27:17.032 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: Updating smithi138:/etc/ceph/ceph.conf 2024-04-25T11:27:17.032 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-25T11:27:17.032 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:17.032 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:17.310 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: mon.b calling monitor election 2024-04-25T11:27:17.311 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: mon.b calling monitor election 2024-04-25T11:27:17.311 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: mon.a calling monitor election 2024-04-25T11:27:17.311 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: mon.c calling monitor election 2024-04-25T11:27:17.311 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T11:27:17.311 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: monmap epoch 3 2024-04-25T11:27:17.311 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:27:17.311 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: last_changed 2024-04-25T11:27:09.616344+0000 2024-04-25T11:27:17.311 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: created 2024-04-25T11:25:19.353596+0000 2024-04-25T11:27:17.311 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: min_mon_release 19 (squid) 2024-04-25T11:27:17.311 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: election_strategy: 1 2024-04-25T11:27:17.311 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: 0: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.a 2024-04-25T11:27:17.311 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: 1: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.c 2024-04-25T11:27:17.311 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: 2: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.b 2024-04-25T11:27:17.312 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: fsmap 2024-04-25T11:27:17.312 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T11:27:17.312 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: mgrmap e13: x(active, since 59s) 2024-04-25T11:27:17.312 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: overall HEALTH_OK 2024-04-25T11:27:17.312 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:17.312 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:17.312 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:17.312 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:27:17.312 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: Updating smithi090:/etc/ceph/ceph.conf 2024-04-25T11:27:17.312 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: Updating smithi138:/etc/ceph/ceph.conf 2024-04-25T11:27:17.312 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-25T11:27:17.312 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:17.312 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:17.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: mon.b calling monitor election 2024-04-25T11:27:17.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: mon.b calling monitor election 2024-04-25T11:27:17.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: mon.a calling monitor election 2024-04-25T11:27:17.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: mon.c calling monitor election 2024-04-25T11:27:17.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T11:27:17.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: monmap epoch 3 2024-04-25T11:27:17.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:27:17.342 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: last_changed 2024-04-25T11:27:09.616344+0000 2024-04-25T11:27:17.342 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: created 2024-04-25T11:25:19.353596+0000 2024-04-25T11:27:17.342 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: min_mon_release 19 (squid) 2024-04-25T11:27:17.342 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: election_strategy: 1 2024-04-25T11:27:17.342 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: 0: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.a 2024-04-25T11:27:17.342 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: 1: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.c 2024-04-25T11:27:17.342 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: 2: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.b 2024-04-25T11:27:17.342 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: fsmap 2024-04-25T11:27:17.342 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T11:27:17.342 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: mgrmap e13: x(active, since 59s) 2024-04-25T11:27:17.342 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: overall HEALTH_OK 2024-04-25T11:27:17.342 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:17.342 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:17.343 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:17.343 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:27:17.343 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: Updating smithi090:/etc/ceph/ceph.conf 2024-04-25T11:27:17.343 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: Updating smithi138:/etc/ceph/ceph.conf 2024-04-25T11:27:17.343 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-25T11:27:17.343 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:17.343 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:17.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.838+0000 7fbe3705a700 1 -- 172.21.15.90:0/1554240571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe301022f0 msgr2=0x7fbe30102700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:17.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.838+0000 7fbe3705a700 1 --2- 172.21.15.90:0/1554240571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe301022f0 0x7fbe30102700 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbe2400aa20 tx=0x7fbe2400ad30 comp rx=0 tx=0).stop 2024-04-25T11:27:17.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.838+0000 7fbe3705a700 1 -- 172.21.15.90:0/1554240571 shutdown_connections 2024-04-25T11:27:17.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.838+0000 7fbe3705a700 1 --2- 172.21.15.90:0/1554240571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe301022f0 0x7fbe30102700 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:17.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.838+0000 7fbe3705a700 1 -- 172.21.15.90:0/1554240571 >> 172.21.15.90:0/1554240571 conn(0x7fbe300fdd00 msgr2=0x7fbe30100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:27:17.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.838+0000 7fbe3705a700 1 -- 172.21.15.90:0/1554240571 shutdown_connections 2024-04-25T11:27:17.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.838+0000 7fbe3705a700 1 -- 172.21.15.90:0/1554240571 wait complete. 2024-04-25T11:27:17.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.839+0000 7fbe3705a700 1 Processor -- start 2024-04-25T11:27:17.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.839+0000 7fbe3705a700 1 -- start start 2024-04-25T11:27:17.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.839+0000 7fbe3705a700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe301022f0 0x7fbe301a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:27:17.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.839+0000 7fbe3705a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe301a0670 0x7fbe301a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:27:17.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.839+0000 7fbe3705a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe301a4ea0 0x7fbe301a52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:27:17.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.839+0000 7fbe3705a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3010bcf0 con 0x7fbe301a0670 2024-04-25T11:27:17.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.839+0000 7fbe3705a700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3010bb70 con 0x7fbe301022f0 2024-04-25T11:27:17.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.839+0000 7fbe3705a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3010be70 con 0x7fbe301a4ea0 2024-04-25T11:27:17.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.840+0000 7fbe2ffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe301a0670 0x7fbe301a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:27:17.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.840+0000 7fbe2ffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe301a0670 0x7fbe301a4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35344/0 (socket says 172.21.15.90:35344) 2024-04-25T11:27:17.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.840+0000 7fbe2ffff700 1 -- 172.21.15.90:0/1901415278 learned_addr learned my addr 172.21.15.90:0/1901415278 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:27:17.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.840+0000 7fbe34df6700 1 --2- 172.21.15.90:0/1901415278 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe301022f0 0x7fbe301a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:27:17.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.840+0000 7fbe355f7700 1 --2- 172.21.15.90:0/1901415278 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe301a4ea0 0x7fbe301a52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:27:17.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.840+0000 7fbe2ffff700 1 -- 172.21.15.90:0/1901415278 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe301a4ea0 msgr2=0x7fbe301a52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:17.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.840+0000 7fbe2ffff700 1 --2- 172.21.15.90:0/1901415278 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe301a4ea0 0x7fbe301a52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:17.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.840+0000 7fbe2ffff700 1 -- 172.21.15.90:0/1901415278 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe301022f0 msgr2=0x7fbe301a0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:17.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.840+0000 7fbe2ffff700 1 --2- 172.21.15.90:0/1901415278 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe301022f0 0x7fbe301a0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:17.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.840+0000 7fbe2ffff700 1 -- 172.21.15.90:0/1901415278 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe301acc10 con 0x7fbe301a0670 2024-04-25T11:27:17.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.840+0000 7fbe2ffff700 1 --2- 172.21.15.90:0/1901415278 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe301a0670 0x7fbe301a4960 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c00b780 tx=0x7fbe1c00ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:27:17.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.840+0000 7fbe2dffb700 1 -- 172.21.15.90:0/1901415278 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c004230 con 0x7fbe301a0670 2024-04-25T11:27:17.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.840+0000 7fbe2dffb700 1 -- 172.21.15.90:0/1901415278 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe1c0043d0 con 0x7fbe301a0670 2024-04-25T11:27:17.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.840+0000 7fbe3705a700 1 -- 172.21.15.90:0/1901415278 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe301ace40 con 0x7fbe301a0670 2024-04-25T11:27:17.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.840+0000 7fbe2dffb700 1 -- 172.21.15.90:0/1901415278 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c011b20 con 0x7fbe301a0670 2024-04-25T11:27:17.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.841+0000 7fbe3705a700 1 -- 172.21.15.90:0/1901415278 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe301ad290 con 0x7fbe301a0670 2024-04-25T11:27:17.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.841+0000 7fbe2dffb700 1 -- 172.21.15.90:0/1901415278 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fbe1c011cc0 con 0x7fbe301a0670 2024-04-25T11:27:17.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.842+0000 7fbe2dffb700 1 --2- 172.21.15.90:0/1901415278 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbe2003daa0 0x7fbe2003ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:27:17.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.842+0000 7fbe2dffb700 1 -- 172.21.15.90:0/1901415278 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fbe1c052070 con 0x7fbe301a0670 2024-04-25T11:27:17.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.842+0000 7fbe3705a700 1 -- 172.21.15.90:0/1901415278 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe30064cf0 con 0x7fbe301a0670 2024-04-25T11:27:17.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.842+0000 7fbe34df6700 1 --2- 172.21.15.90:0/1901415278 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbe2003daa0 0x7fbe2003ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:27:17.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.842+0000 7fbe34df6700 1 --2- 172.21.15.90:0/1901415278 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbe2003daa0 0x7fbe2003ffb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbe2400c6d0 tx=0x7fbe24019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:27:17.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:17.847+0000 7fbe2dffb700 1 -- 172.21.15.90:0/1901415278 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbe1c017d70 con 0x7fbe301a0670 2024-04-25T11:27:18.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:18.062+0000 7fbe3705a700 1 -- 172.21.15.90:0/1901415278 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fbe301ad580 con 0x7fbe301a0670 2024-04-25T11:27:18.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:18.062+0000 7fbe2dffb700 1 -- 172.21.15.90:0/1901415278 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7fbe1c00f6a0 con 0x7fbe301a0670 2024-04-25T11:27:18.054 INFO:teuthology.orchestra.run.smithi090.stdout:# minimal ceph.conf for 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:27:18.054 INFO:teuthology.orchestra.run.smithi090.stdout:[global] 2024-04-25T11:27:18.054 INFO:teuthology.orchestra.run.smithi090.stdout: fsid = 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:27:18.054 INFO:teuthology.orchestra.run.smithi090.stdout: mon_host = [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] 2024-04-25T11:27:18.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:18.065+0000 7fbe3705a700 1 -- 172.21.15.90:0/1901415278 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbe2003daa0 msgr2=0x7fbe2003ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:18.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:18.065+0000 7fbe3705a700 1 --2- 172.21.15.90:0/1901415278 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbe2003daa0 0x7fbe2003ffb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbe2400c6d0 tx=0x7fbe24019040 comp rx=0 tx=0).stop 2024-04-25T11:27:18.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:18.065+0000 7fbe3705a700 1 -- 172.21.15.90:0/1901415278 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe301a0670 msgr2=0x7fbe301a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:18.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:18.065+0000 7fbe3705a700 1 --2- 172.21.15.90:0/1901415278 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe301a0670 0x7fbe301a4960 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c00b780 tx=0x7fbe1c00ba90 comp rx=0 tx=0).stop 2024-04-25T11:27:18.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:18.065+0000 7fbe3705a700 1 -- 172.21.15.90:0/1901415278 shutdown_connections 2024-04-25T11:27:18.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:18.065+0000 7fbe3705a700 1 --2- 172.21.15.90:0/1901415278 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbe2003daa0 0x7fbe2003ffb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:18.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:18.065+0000 7fbe3705a700 1 --2- 172.21.15.90:0/1901415278 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe301a4ea0 0x7fbe301a52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:18.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:18.065+0000 7fbe3705a700 1 --2- 172.21.15.90:0/1901415278 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe301022f0 0x7fbe301a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:18.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:18.065+0000 7fbe3705a700 1 --2- 172.21.15.90:0/1901415278 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe301a0670 0x7fbe301a4960 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:18.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:18.065+0000 7fbe3705a700 1 -- 172.21.15.90:0/1901415278 >> 172.21.15.90:0/1901415278 conn(0x7fbe300fdd00 msgr2=0x7fbe30107e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:27:18.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:18.065+0000 7fbe3705a700 1 -- 172.21.15.90:0/1901415278 shutdown_connections 2024-04-25T11:27:18.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:18.065+0000 7fbe3705a700 1 -- 172.21.15.90:0/1901415278 wait complete. 2024-04-25T11:27:18.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:18 smithi138 ceph-mon[32328]: Updating smithi090:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T11:27:18.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:18 smithi138 ceph-mon[32328]: Updating smithi138:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T11:27:18.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:18 smithi138 ceph-mon[32328]: Updating smithi156:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T11:27:18.473 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:18 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:18 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:18 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:18 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:18 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:18 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:18 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:18 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:18 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:18 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:18 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:18 smithi138 ceph-mon[32328]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T11:27:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:18 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T11:27:18.474 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:18 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T11:27:18.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:18 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:18.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:18 smithi138 ceph-mon[32328]: Reconfiguring daemon mon.a on smithi090 2024-04-25T11:27:18.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:18 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:27:18.475 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:18 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1901415278' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:18.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:18 smithi090 ceph-mon[29575]: Updating smithi090:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T11:27:18.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:18 smithi090 ceph-mon[29575]: Updating smithi138:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T11:27:18.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:18 smithi090 ceph-mon[29575]: Updating smithi156:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T11:27:18.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:18 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:18 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:18 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:18 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:18 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:18 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:18 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:18 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:18 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:18 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:18 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:18 smithi090 ceph-mon[29575]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T11:27:18.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:18 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T11:27:18.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:18 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T11:27:18.480 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:18 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:18.480 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:18 smithi090 ceph-mon[29575]: Reconfiguring daemon mon.a on smithi090 2024-04-25T11:27:18.480 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:18 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:27:18.480 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:18 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1901415278' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:18.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:18 smithi156 ceph-mon[33045]: Updating smithi090:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T11:27:18.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:18 smithi156 ceph-mon[33045]: Updating smithi138:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T11:27:18.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:18 smithi156 ceph-mon[33045]: Updating smithi156:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T11:27:18.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:18 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:18 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:18 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:18 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:18 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:18 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.592 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:18 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.592 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:18 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.592 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:18 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.592 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:18 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.592 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:18 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:18.592 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:18 smithi156 ceph-mon[33045]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T11:27:18.592 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:18 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T11:27:18.592 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:18 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T11:27:18.592 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:18 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:18.592 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:18 smithi156 ceph-mon[33045]: Reconfiguring daemon mon.a on smithi090 2024-04-25T11:27:18.592 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:18 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:27:18.592 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:18 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1901415278' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:19.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:19 smithi090 ceph-mon[29575]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:19.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:19 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:19.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:19 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:19.547 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-25T11:27:19.548 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T11:27:19.548 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T11:27:19.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:19 smithi138 ceph-mon[32328]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:19.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:19 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:19.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:19 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:19.573 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T11:27:19.574 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T11:27:19.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:19 smithi156 ceph-mon[33045]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:19.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:19 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:19.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:19 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:19.637 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-25T11:27:19.637 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T11:27:19.663 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-25T11:27:19.663 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T11:27:19.727 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-25T11:27:19.727 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T11:27:19.758 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-25T11:27:19.758 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T11:27:19.825 INFO:tasks.cephadm:Adding mgr.x on smithi090 2024-04-25T11:27:19.825 INFO:tasks.cephadm:Adding mgr.y on smithi138 2024-04-25T11:27:19.826 INFO:tasks.cephadm:Adding mgr.z on smithi156 2024-04-25T11:27:19.826 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi090=x;smithi138=y;smithi156=z' 2024-04-25T11:27:20.099 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T11:27:20.305 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:20 smithi138 ceph-mon[32328]: Reconfiguring mon.b (monmap changed)... 2024-04-25T11:27:20.305 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:20 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T11:27:20.305 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:20 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T11:27:20.305 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:20 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:20.305 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:20 smithi138 ceph-mon[32328]: Reconfiguring daemon mon.b on smithi138 2024-04-25T11:27:20.464 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:20 smithi156 ceph-mon[33045]: Reconfiguring mon.b (monmap changed)... 2024-04-25T11:27:20.464 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:20 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T11:27:20.465 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:20 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T11:27:20.465 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:20 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:20.465 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:20 smithi156 ceph-mon[33045]: Reconfiguring daemon mon.b on smithi138 2024-04-25T11:27:20.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:20 smithi090 ceph-mon[29575]: Reconfiguring mon.b (monmap changed)... 2024-04-25T11:27:20.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:20 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T11:27:20.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:20 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T11:27:20.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:20 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:20.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:20 smithi090 ceph-mon[29575]: Reconfiguring daemon mon.b on smithi138 2024-04-25T11:27:21.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.552+0000 7fe5e188a700 1 -- 172.21.15.156:0/4148802768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5dc102ab0 msgr2=0x7fe5dc102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:21.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.552+0000 7fe5e188a700 1 --2- 172.21.15.156:0/4148802768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5dc102ab0 0x7fe5dc102ec0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc009910 tx=0x7fe5cc009c20 comp rx=0 tx=0).stop 2024-04-25T11:27:21.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.552+0000 7fe5e188a700 1 -- 172.21.15.156:0/4148802768 shutdown_connections 2024-04-25T11:27:21.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.552+0000 7fe5e188a700 1 --2- 172.21.15.156:0/4148802768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5dc102ab0 0x7fe5dc102ec0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:21.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.552+0000 7fe5e188a700 1 -- 172.21.15.156:0/4148802768 >> 172.21.15.156:0/4148802768 conn(0x7fe5dc0fe0f0 msgr2=0x7fe5dc1005a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:27:21.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.552+0000 7fe5e188a700 1 -- 172.21.15.156:0/4148802768 shutdown_connections 2024-04-25T11:27:21.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.552+0000 7fe5e188a700 1 -- 172.21.15.156:0/4148802768 wait complete. 2024-04-25T11:27:21.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.553+0000 7fe5e188a700 1 Processor -- start 2024-04-25T11:27:21.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.553+0000 7fe5e188a700 1 -- start start 2024-04-25T11:27:21.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.553+0000 7fe5e188a700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe5dc102ab0 0x7fe5dc197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:27:21.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.553+0000 7fe5e188a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5dc197b50 0x7fe5dc19cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:27:21.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.553+0000 7fe5e188a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5dc19d020 0x7fe5dc19f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:27:21.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.553+0000 7fe5e188a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe5dc107970 con 0x7fe5dc19d020 2024-04-25T11:27:21.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.553+0000 7fe5e188a700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe5dc1077f0 con 0x7fe5dc102ab0 2024-04-25T11:27:21.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.553+0000 7fe5e188a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe5dc107af0 con 0x7fe5dc197b50 2024-04-25T11:27:21.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.554+0000 7fe5db7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5dc19d020 0x7fe5dc19f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:27:21.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.554+0000 7fe5db7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5dc19d020 0x7fe5dc19f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.156:46476/0 (socket says 172.21.15.156:46476) 2024-04-25T11:27:21.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.554+0000 7fe5db7fe700 1 -- 172.21.15.156:0/2457262185 learned_addr learned my addr 172.21.15.156:0/2457262185 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-25T11:27:21.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.554+0000 7fe5daffd700 1 --2- 172.21.15.156:0/2457262185 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe5dc102ab0 0x7fe5dc197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:27:21.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.554+0000 7fe5da7fc700 1 --2- 172.21.15.156:0/2457262185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5dc197b50 0x7fe5dc19cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:27:21.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.554+0000 7fe5db7fe700 1 -- 172.21.15.156:0/2457262185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5dc197b50 msgr2=0x7fe5dc19cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:21.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.554+0000 7fe5db7fe700 1 --2- 172.21.15.156:0/2457262185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5dc197b50 0x7fe5dc19cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:21.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.554+0000 7fe5db7fe700 1 -- 172.21.15.156:0/2457262185 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe5dc102ab0 msgr2=0x7fe5dc197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:21.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.554+0000 7fe5db7fe700 1 --2- 172.21.15.156:0/2457262185 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe5dc102ab0 0x7fe5dc197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:21.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.554+0000 7fe5db7fe700 1 -- 172.21.15.156:0/2457262185 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5dc19f9b0 con 0x7fe5dc19d020 2024-04-25T11:27:21.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.554+0000 7fe5db7fe700 1 --2- 172.21.15.156:0/2457262185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5dc19d020 0x7fe5dc19f440 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe5d400ba50 tx=0x7fe5d400bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:27:21.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.555+0000 7fe5e0888700 1 -- 172.21.15.156:0/2457262185 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d4019070 con 0x7fe5dc19d020 2024-04-25T11:27:21.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.555+0000 7fe5e188a700 1 -- 172.21.15.156:0/2457262185 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5dc19fca0 con 0x7fe5dc19d020 2024-04-25T11:27:21.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.555+0000 7fe5e0888700 1 -- 172.21.15.156:0/2457262185 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5d400cde0 con 0x7fe5dc19d020 2024-04-25T11:27:21.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.555+0000 7fe5e0888700 1 -- 172.21.15.156:0/2457262185 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d4007350 con 0x7fe5dc19d020 2024-04-25T11:27:21.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.555+0000 7fe5e188a700 1 -- 172.21.15.156:0/2457262185 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5dc1a0190 con 0x7fe5dc19d020 2024-04-25T11:27:21.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.555+0000 7fe5e188a700 1 -- 172.21.15.156:0/2457262185 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5dc064cf0 con 0x7fe5dc19d020 2024-04-25T11:27:21.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.556+0000 7fe5e0888700 1 -- 172.21.15.156:0/2457262185 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fe5d401c070 con 0x7fe5dc19d020 2024-04-25T11:27:21.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.556+0000 7fe5e0888700 1 --2- 172.21.15.156:0/2457262185 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fe5c403fcb0 0x7fe5c40421c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:27:21.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.556+0000 7fe5e0888700 1 -- 172.21.15.156:0/2457262185 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe5d4050d70 con 0x7fe5dc19d020 2024-04-25T11:27:21.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.556+0000 7fe5daffd700 1 --2- 172.21.15.156:0/2457262185 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fe5c403fcb0 0x7fe5c40421c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:27:21.550 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.557+0000 7fe5daffd700 1 --2- 172.21.15.156:0/2457262185 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fe5c403fcb0 0x7fe5c40421c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc006010 tx=0x7fe5cc01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:27:21.554 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.562+0000 7fe5e0888700 1 -- 172.21.15.156:0/2457262185 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe5d4055070 con 0x7fe5dc19d020 2024-04-25T11:27:21.775 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.783+0000 7fe5e188a700 1 -- 172.21.15.156:0/2457262185 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi090=x;smithi138=y;smithi156=z", "target": ["mon-mgr", ""]}) -- 0x7fe5dc0ffe90 con 0x7fe5c403fcb0 2024-04-25T11:27:21.781 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.790+0000 7fe5e0888700 1 -- 172.21.15.156:0/2457262185 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fe5dc0ffe90 con 0x7fe5c403fcb0 2024-04-25T11:27:21.782 INFO:teuthology.orchestra.run.smithi156.stdout:Scheduled mgr update... 2024-04-25T11:27:21.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.792+0000 7fe5e188a700 1 -- 172.21.15.156:0/2457262185 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fe5c403fcb0 msgr2=0x7fe5c40421c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:21.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.792+0000 7fe5e188a700 1 --2- 172.21.15.156:0/2457262185 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fe5c403fcb0 0x7fe5c40421c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc006010 tx=0x7fe5cc01a040 comp rx=0 tx=0).stop 2024-04-25T11:27:21.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.792+0000 7fe5e188a700 1 -- 172.21.15.156:0/2457262185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5dc19d020 msgr2=0x7fe5dc19f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:21.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.792+0000 7fe5e188a700 1 --2- 172.21.15.156:0/2457262185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5dc19d020 0x7fe5dc19f440 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe5d400ba50 tx=0x7fe5d400bd60 comp rx=0 tx=0).stop 2024-04-25T11:27:21.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.792+0000 7fe5e188a700 1 -- 172.21.15.156:0/2457262185 shutdown_connections 2024-04-25T11:27:21.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.792+0000 7fe5e188a700 1 --2- 172.21.15.156:0/2457262185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5dc19d020 0x7fe5dc19f440 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:21.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.792+0000 7fe5e188a700 1 --2- 172.21.15.156:0/2457262185 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe5dc102ab0 0x7fe5dc197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:21.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.792+0000 7fe5e188a700 1 --2- 172.21.15.156:0/2457262185 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fe5c403fcb0 0x7fe5c40421c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:21.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.792+0000 7fe5e188a700 1 --2- 172.21.15.156:0/2457262185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5dc197b50 0x7fe5dc19cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:21.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.792+0000 7fe5e188a700 1 -- 172.21.15.156:0/2457262185 >> 172.21.15.156:0/2457262185 conn(0x7fe5dc0fe0f0 msgr2=0x7fe5dc0ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:27:21.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.792+0000 7fe5e188a700 1 -- 172.21.15.156:0/2457262185 shutdown_connections 2024-04-25T11:27:21.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:27:21.792+0000 7fe5e188a700 1 -- 172.21.15.156:0/2457262185 wait complete. 2024-04-25T11:27:21.862 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:21 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:21.862 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:21 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:21.862 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:21 smithi156 ceph-mon[33045]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:21.863 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:21 smithi156 ceph-mon[33045]: Reconfiguring mon.c (monmap changed)... 2024-04-25T11:27:21.863 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:21 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T11:27:21.863 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:21 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T11:27:21.863 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:21 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:21.863 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:21 smithi156 ceph-mon[33045]: Reconfiguring daemon mon.c on smithi156 2024-04-25T11:27:21.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:21 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:21.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:21 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:21.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:21 smithi090 ceph-mon[29575]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:21.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:21 smithi090 ceph-mon[29575]: Reconfiguring mon.c (monmap changed)... 2024-04-25T11:27:21.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:21 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T11:27:21.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:21 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T11:27:21.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:21 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:21.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:21 smithi090 ceph-mon[29575]: Reconfiguring daemon mon.c on smithi156 2024-04-25T11:27:22.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:21 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:22.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:21 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:22.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:21 smithi138 ceph-mon[32328]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:22.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:21 smithi138 ceph-mon[32328]: Reconfiguring mon.c (monmap changed)... 2024-04-25T11:27:22.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:21 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T11:27:22.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:21 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T11:27:22.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:21 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:22.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:21 smithi138 ceph-mon[32328]: Reconfiguring daemon mon.c on smithi156 2024-04-25T11:27:23.014 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:22 smithi156 ceph-mon[33045]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi090=x;smithi138=y;smithi156=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:27:23.014 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:22 smithi156 ceph-mon[33045]: Saving service mgr spec with placement smithi090=x;smithi138=y;smithi156=z;count:3 2024-04-25T11:27:23.015 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:22 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:23.015 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:22 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:23.015 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:22 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:23.015 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:22 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:27:23.016 DEBUG:teuthology.orchestra.run.smithi138:mgr.y> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mgr.y.service 2024-04-25T11:27:23.019 DEBUG:teuthology.orchestra.run.smithi156:mgr.z> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mgr.z.service 2024-04-25T11:27:23.021 INFO:tasks.cephadm:Deploying OSDs... 2024-04-25T11:27:23.021 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T11:27:23.021 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T11:27:23.037 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T11:27:23.037 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_1 2024-04-25T11:27:23.042 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:22 smithi138 ceph-mon[32328]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi090=x;smithi138=y;smithi156=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:27:23.042 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:22 smithi138 ceph-mon[32328]: Saving service mgr spec with placement smithi090=x;smithi138=y;smithi156=z;count:3 2024-04-25T11:27:23.042 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:22 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:23.042 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:22 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:23.042 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:22 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:23.042 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:22 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:27:23.093 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T11:27:23.093 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T11:27:23.093 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-04-25T11:27:23.093 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T11:27:23.093 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T11:27:23.093 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-04-25 11:26:29.191130692 +0000 2024-04-25T11:27:23.093 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-04-25 11:20:25.335610534 +0000 2024-04-25T11:27:23.093 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-04-25 11:20:25.335610534 +0000 2024-04-25T11:27:23.093 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-04-25 11:20:25.335610534 +0000 2024-04-25T11:27:23.094 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T11:27:23.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:22 smithi090 ceph-mon[29575]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi090=x;smithi138=y;smithi156=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:27:23.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:22 smithi090 ceph-mon[29575]: Saving service mgr spec with placement smithi090=x;smithi138=y;smithi156=z;count:3 2024-04-25T11:27:23.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:22 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:23.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:22 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:23.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:22 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:23.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:22 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:27:23.156 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-04-25T11:27:23.156 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-04-25T11:27:23.156 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000192124 s, 2.7 MB/s 2024-04-25T11:27:23.157 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T11:27:23.212 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_2 2024-04-25T11:27:23.267 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T11:27:23.267 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T11:27:23.267 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-25T11:27:23.267 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T11:27:23.268 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T11:27:23.268 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-04-25 11:26:29.225130195 +0000 2024-04-25T11:27:23.268 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-04-25 11:20:25.662605528 +0000 2024-04-25T11:27:23.268 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-04-25 11:20:25.662605528 +0000 2024-04-25T11:27:23.268 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-04-25 11:20:25.662605528 +0000 2024-04-25T11:27:23.268 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T11:27:23.330 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-04-25T11:27:23.330 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-04-25T11:27:23.330 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000201788 s, 2.5 MB/s 2024-04-25T11:27:23.331 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T11:27:23.387 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_3 2024-04-25T11:27:23.443 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T11:27:23.443 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T11:27:23.443 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-04-25T11:27:23.443 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T11:27:23.443 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T11:27:23.443 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-04-25 11:26:29.259129698 +0000 2024-04-25T11:27:23.443 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-04-25 11:20:26.010600200 +0000 2024-04-25T11:27:23.443 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-04-25 11:20:26.010600200 +0000 2024-04-25T11:27:23.443 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-04-25 11:20:26.010600200 +0000 2024-04-25T11:27:23.444 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T11:27:23.505 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-04-25T11:27:23.505 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-04-25T11:27:23.505 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000236693 s, 2.2 MB/s 2024-04-25T11:27:23.506 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T11:27:23.562 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_4 2024-04-25T11:27:23.617 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T11:27:23.617 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T11:27:23.617 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-25T11:27:23.617 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T11:27:23.618 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T11:27:23.618 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-04-25 11:26:29.291129231 +0000 2024-04-25T11:27:23.618 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-04-25 11:20:26.337595195 +0000 2024-04-25T11:27:23.618 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-04-25 11:20:26.337595195 +0000 2024-04-25T11:27:23.618 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-04-25 11:20:26.337595195 +0000 2024-04-25T11:27:23.618 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T11:27:23.680 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-04-25T11:27:23.680 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-04-25T11:27:23.680 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.00017505 s, 2.9 MB/s 2024-04-25T11:27:23.681 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T11:27:23.736 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-25T11:27:23.736 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T11:27:23.755 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T11:27:23.755 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_1 2024-04-25T11:27:23.813 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T11:27:23.813 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T11:27:23.813 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-25T11:27:23.813 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T11:27:23.813 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T11:27:23.814 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-04-25 11:27:09.083098425 +0000 2024-04-25T11:27:23.814 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-04-25 11:21:57.825477542 +0000 2024-04-25T11:27:23.814 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-04-25 11:21:57.825477542 +0000 2024-04-25T11:27:23.814 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: 2024-04-25 11:21:57.825477542 +0000 2024-04-25T11:27:23.814 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T11:27:23.879 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-04-25T11:27:23.880 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-04-25T11:27:23.880 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000307126 s, 1.7 MB/s 2024-04-25T11:27:23.881 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T11:27:23.940 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_2 2024-04-25T11:27:23.997 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T11:27:23.997 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T11:27:23.997 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-25T11:27:23.997 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T11:27:23.997 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T11:27:23.997 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-04-25 11:27:09.083098425 +0000 2024-04-25T11:27:23.997 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-04-25 11:21:58.171469780 +0000 2024-04-25T11:27:23.997 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-04-25 11:21:58.171469780 +0000 2024-04-25T11:27:23.998 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: 2024-04-25 11:21:58.171469780 +0000 2024-04-25T11:27:23.998 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T11:27:24.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:23 smithi138 ceph-mon[32328]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:24.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:23 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:24.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:23 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:27:24.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:23 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:24.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:23 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T11:27:24.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:23 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T11:27:24.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:23 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T11:27:24.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:23 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:24.059 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:23 smithi156 ceph-mon[33045]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:24.059 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:23 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:24.059 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:23 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:27:24.059 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:23 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:24.060 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:23 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T11:27:24.060 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:23 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T11:27:24.060 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:23 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T11:27:24.060 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:23 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:24.063 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-04-25T11:27:24.063 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-04-25T11:27:24.063 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000297526 s, 1.7 MB/s 2024-04-25T11:27:24.064 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T11:27:24.122 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_3 2024-04-25T11:27:24.178 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T11:27:24.178 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T11:27:24.179 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-25T11:27:24.179 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T11:27:24.179 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T11:27:24.179 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-04-25 11:27:09.084098403 +0000 2024-04-25T11:27:24.179 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-04-25 11:21:58.504462309 +0000 2024-04-25T11:27:24.179 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-04-25 11:21:58.504462309 +0000 2024-04-25T11:27:24.179 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: 2024-04-25 11:21:58.504462309 +0000 2024-04-25T11:27:24.179 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T11:27:24.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:23 smithi090 ceph-mon[29575]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:24.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:23 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:24.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:23 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:27:24.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:23 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:24.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:23 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T11:27:24.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:23 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T11:27:24.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:23 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T11:27:24.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:23 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:24.242 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-04-25T11:27:24.242 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-04-25T11:27:24.242 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000179449 s, 2.9 MB/s 2024-04-25T11:27:24.243 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T11:27:24.300 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_4 2024-04-25T11:27:24.355 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T11:27:24.356 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T11:27:24.356 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-25T11:27:24.356 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T11:27:24.356 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T11:27:24.356 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-04-25 11:27:09.084098403 +0000 2024-04-25T11:27:24.356 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-04-25 11:21:58.851454525 +0000 2024-04-25T11:27:24.356 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-04-25 11:21:58.851454525 +0000 2024-04-25T11:27:24.356 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: 2024-04-25 11:21:58.851454525 +0000 2024-04-25T11:27:24.356 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T11:27:24.418 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-04-25T11:27:24.418 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-04-25T11:27:24.418 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000214503 s, 2.4 MB/s 2024-04-25T11:27:24.419 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T11:27:24.475 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-25T11:27:24.475 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T11:27:24.491 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T11:27:24.491 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_1 2024-04-25T11:27:24.547 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T11:27:24.547 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T11:27:24.547 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-25T11:27:24.547 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T11:27:24.547 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T11:27:24.547 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-04-25 11:26:59.573943795 +0000 2024-04-25T11:27:24.547 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-04-25 11:20:47.312539625 +0000 2024-04-25T11:27:24.547 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-04-25 11:20:47.312539625 +0000 2024-04-25T11:27:24.548 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-04-25 11:20:47.312539625 +0000 2024-04-25T11:27:24.548 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T11:27:24.610 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-04-25T11:27:24.610 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-04-25T11:27:24.610 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000117336 s, 4.4 MB/s 2024-04-25T11:27:24.611 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T11:27:24.668 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_2 2024-04-25T11:27:24.723 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T11:27:24.724 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T11:27:24.724 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-04-25T11:27:24.724 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T11:27:24.724 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T11:27:24.724 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-04-25 11:26:59.573943795 +0000 2024-04-25T11:27:24.724 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-04-25 11:20:47.648534132 +0000 2024-04-25T11:27:24.724 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-04-25 11:20:47.648534132 +0000 2024-04-25T11:27:24.725 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-04-25 11:20:47.648534132 +0000 2024-04-25T11:27:24.725 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T11:27:24.787 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-04-25T11:27:24.787 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-04-25T11:27:24.787 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 8.4148e-05 s, 6.1 MB/s 2024-04-25T11:27:24.788 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T11:27:24.847 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_3 2024-04-25T11:27:24.903 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T11:27:24.903 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T11:27:24.903 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-25T11:27:24.903 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T11:27:24.903 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T11:27:24.903 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-04-25 11:26:59.573943795 +0000 2024-04-25T11:27:24.903 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-04-25 11:20:47.988528575 +0000 2024-04-25T11:27:24.903 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-04-25 11:20:47.988528575 +0000 2024-04-25T11:27:24.903 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-04-25 11:20:47.988528575 +0000 2024-04-25T11:27:24.904 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T11:27:24.964 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:24 smithi156 ceph-mon[33045]: Deploying daemon mgr.z on smithi156 2024-04-25T11:27:24.967 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-04-25T11:27:24.967 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-04-25T11:27:24.967 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 9.1091e-05 s, 5.6 MB/s 2024-04-25T11:27:24.968 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T11:27:25.028 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_4 2024-04-25T11:27:25.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:24 smithi138 ceph-mon[32328]: Deploying daemon mgr.z on smithi156 2024-04-25T11:27:25.082 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T11:27:25.082 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T11:27:25.082 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-25T11:27:25.082 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T11:27:25.082 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T11:27:25.082 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-04-25 11:27:24.916551755 +0000 2024-04-25T11:27:25.083 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-04-25 11:20:48.321523132 +0000 2024-04-25T11:27:25.083 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-04-25 11:20:48.321523132 +0000 2024-04-25T11:27:25.083 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-04-25 11:20:48.321523132 +0000 2024-04-25T11:27:25.083 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T11:27:25.146 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-04-25T11:27:25.147 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-04-25T11:27:25.147 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 9.0789e-05 s, 5.6 MB/s 2024-04-25T11:27:25.148 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T11:27:25.204 INFO:tasks.cephadm:Deploying osd.0 on smithi090 with /dev/vg_nvme/lv_4... 2024-04-25T11:27:25.204 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T11:27:25.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:24 smithi090 ceph-mon[29575]: Deploying daemon mgr.z on smithi156 2024-04-25T11:27:25.445 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:27:25.660 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:25 smithi156 ceph-mgr[34920]: -- 172.21.15.156:0/2424082766 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55629ef965a0 con 0x5562a3d20480 2024-04-25T11:27:25.826 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:25 smithi090 ceph-mon[29575]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:25.826 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:25 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:25.826 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:25 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:25.826 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:25 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:25.827 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:25 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T11:27:25.827 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:25 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T11:27:25.827 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:25 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T11:27:25.827 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:25 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:25.827 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:25 smithi090 ceph-mon[29575]: Deploying daemon mgr.y on smithi138 2024-04-25T11:27:25.911 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:25 smithi156 ceph-mon[33045]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:25.911 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:25 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:25.911 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:25 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:25.911 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:25 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:25.911 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:25 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T11:27:25.911 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:25 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T11:27:25.911 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:25 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T11:27:25.911 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:25 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:25.911 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:25 smithi156 ceph-mon[33045]: Deploying daemon mgr.y on smithi138 2024-04-25T11:27:25.912 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:25 smithi156 bash[34900]: 6b0ef806886423b86d8c37c33e2ade481dbd2e522ba29a5148451ce5fc436c3d 2024-04-25T11:27:25.912 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:25 smithi156 systemd[1]: Started Ceph mgr.z for 58bd4e7a-02f6-11ef-bc93-c7b262605968. 2024-04-25T11:27:25.912 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:25 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:27:25.733+0000 7f9c3b470200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T11:27:25.912 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:25 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:27:25.918+0000 7f9c3b470200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T11:27:26.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:25 smithi138 ceph-mon[32328]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:26.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:25 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:26.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:25 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:26.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:25 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:26.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:25 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T11:27:26.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:25 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T11:27:26.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:25 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T11:27:26.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:25 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:26.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:25 smithi138 ceph-mon[32328]: Deploying daemon mgr.y on smithi138 2024-04-25T11:27:26.318 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:26 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:27:26.020+0000 7f9c3b470200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T11:27:26.318 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:26 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:27:26.326+0000 7f9c3b470200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T11:27:26.591 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:26 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:27:26.410+0000 7f9c3b470200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T11:27:26.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:26 smithi090 ceph-mon[29575]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:27.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:26 smithi138 ceph-mon[32328]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:27.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:26 smithi156 ceph-mon[33045]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T11:27:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T11:27:27.207 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-04-25T11:27:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-04-25T11:27:27.207 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116099 s, 903 MB/s 2024-04-25T11:27:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-04-25T11:27:27.786 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:27 smithi138 systemd[1]: Starting Ceph mgr.y for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:27:27.816 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi090:vg_nvme/lv_4 2024-04-25T11:27:27.950 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:27 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:27:27.609+0000 7f9c3b470200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T11:27:27.950 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:27 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:27:27.780+0000 7f9c3b470200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T11:27:27.950 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:27 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:27:27.957+0000 7f9c3b470200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T11:27:28.043 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:27 smithi138 podman[33688]: 2024-04-25 11:27:27.698074191 +0000 UTC m=+0.017763931 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T11:27:28.043 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:27 smithi138 podman[33688]: 2024-04-25 11:27:27.811379189 +0000 UTC m=+0.131068956 container create a7e64dbe4f67ffc5e6073d568df015f511b24a2e32b8c39b55d05693f4704f7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T11:27:28.043 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:28 smithi138 podman[33688]: 2024-04-25 11:27:28.014407963 +0000 UTC m=+0.334097689 container init a7e64dbe4f67ffc5e6073d568df015f511b24a2e32b8c39b55d05693f4704f7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-25T11:27:28.044 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:28 smithi138 podman[33688]: 2024-04-25 11:27:28.017010971 +0000 UTC m=+0.336700696 container start a7e64dbe4f67ffc5e6073d568df015f511b24a2e32b8c39b55d05693f4704f7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T11:27:28.089 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:27:28.305 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:28 smithi138 ceph-mgr[33707]: -- 172.21.15.138:0/3836556852 <== mon.2 v2:172.21.15.138:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c983c545a0 con 0x55c9889de480 2024-04-25T11:27:28.305 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:28 smithi138 bash[33688]: a7e64dbe4f67ffc5e6073d568df015f511b24a2e32b8c39b55d05693f4704f7f 2024-04-25T11:27:28.305 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:28 smithi138 systemd[1]: Started Ceph mgr.y for 58bd4e7a-02f6-11ef-bc93-c7b262605968. 2024-04-25T11:27:28.305 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:28 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:27:28.178+0000 7fc6f6456200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T11:27:28.565 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:28 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:27:28.261+0000 7f9c3b470200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T11:27:28.565 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:28 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:27:28.358+0000 7f9c3b470200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T11:27:28.565 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:28 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:27:28.572+0000 7f9c3b470200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T11:27:28.805 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:28 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:27:28.374+0000 7fc6f6456200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T11:27:28.805 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:28 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:27:28.485+0000 7fc6f6456200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T11:27:28.840 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:28 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:27:28.797+0000 7f9c3b470200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T11:27:29.179 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:28 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:27:28.818+0000 7fc6f6456200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T11:27:29.179 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:28 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:27:28.916+0000 7fc6f6456200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T11:27:29.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:29 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:29.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:29 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:29.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:29 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:29.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:29 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:29.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:29 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:27:29.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:29 smithi156 ceph-mon[33045]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:29.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:29 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:29.427 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:29 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:29.427 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:29 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:29.427 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:29 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:29.427 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:29 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:27:29.427 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:29 smithi090 ceph-mon[29575]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:29.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.465+0000 7f6a8e486700 1 -- 172.21.15.90:0/389293489 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a88102220 msgr2=0x7f6a88102630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:29.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.465+0000 7f6a8e486700 1 --2- 172.21.15.90:0/389293489 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a88102220 0x7f6a88102630 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c009990 tx=0x7f6a7c009ca0 comp rx=0 tx=0).stop 2024-04-25T11:27:29.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.465+0000 7f6a8e486700 1 -- 172.21.15.90:0/389293489 shutdown_connections 2024-04-25T11:27:29.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.465+0000 7f6a8e486700 1 --2- 172.21.15.90:0/389293489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a88102b70 0x7f6a8810c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:29.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.465+0000 7f6a8e486700 1 --2- 172.21.15.90:0/389293489 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6a881057d0 0x7f6a88105be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:29.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.465+0000 7f6a8e486700 1 --2- 172.21.15.90:0/389293489 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a88102220 0x7f6a88102630 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:29.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.465+0000 7f6a8e486700 1 -- 172.21.15.90:0/389293489 >> 172.21.15.90:0/389293489 conn(0x7f6a880fded0 msgr2=0x7f6a881003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:27:29.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.465+0000 7f6a8e486700 1 -- 172.21.15.90:0/389293489 shutdown_connections 2024-04-25T11:27:29.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.465+0000 7f6a8e486700 1 -- 172.21.15.90:0/389293489 wait complete. 2024-04-25T11:27:29.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.465+0000 7f6a8e486700 1 Processor -- start 2024-04-25T11:27:29.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.466+0000 7f6a8e486700 1 -- start start 2024-04-25T11:27:29.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.466+0000 7f6a8e486700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a88102220 0x7f6a8819a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:27:29.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.466+0000 7f6a8e486700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a88102b70 0x7f6a8819acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:27:29.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.466+0000 7f6a8e486700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6a881057d0 0x7f6a881a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:27:29.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.466+0000 7f6a8e486700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8810fc90 con 0x7f6a88102b70 2024-04-25T11:27:29.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.466+0000 7f6a8e486700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8810fb10 con 0x7f6a881057d0 2024-04-25T11:27:29.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.466+0000 7f6a8e486700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8810fe10 con 0x7f6a88102220 2024-04-25T11:27:29.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.466+0000 7f6a877fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a88102b70 0x7f6a8819acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:27:29.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.466+0000 7f6a877fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a88102b70 0x7f6a8819acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47576/0 (socket says 172.21.15.90:47576) 2024-04-25T11:27:29.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.466+0000 7f6a877fe700 1 -- 172.21.15.90:0/750335763 learned_addr learned my addr 172.21.15.90:0/750335763 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:27:29.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.466+0000 7f6a87fff700 1 --2- 172.21.15.90:0/750335763 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a88102220 0x7f6a8819a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:27:29.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.466+0000 7f6a8ca23700 1 --2- 172.21.15.90:0/750335763 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6a881057d0 0x7f6a881a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:27:29.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.466+0000 7f6a877fe700 1 -- 172.21.15.90:0/750335763 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a88102220 msgr2=0x7f6a8819a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:29.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.466+0000 7f6a877fe700 1 --2- 172.21.15.90:0/750335763 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a88102220 0x7f6a8819a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:29.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.466+0000 7f6a877fe700 1 -- 172.21.15.90:0/750335763 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6a881057d0 msgr2=0x7f6a881a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:29.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.466+0000 7f6a877fe700 1 --2- 172.21.15.90:0/750335763 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6a881057d0 0x7f6a881a1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:29.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.466+0000 7f6a877fe700 1 -- 172.21.15.90:0/750335763 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a881a2310 con 0x7f6a88102b70 2024-04-25T11:27:29.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.466+0000 7f6a8ca23700 1 --2- 172.21.15.90:0/750335763 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6a881057d0 0x7f6a881a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:27:29.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.466+0000 7f6a877fe700 1 --2- 172.21.15.90:0/750335763 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a88102b70 0x7f6a8819acf0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c00b760 tx=0x7f6a7c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:27:29.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.467+0000 7f6a857fa700 1 -- 172.21.15.90:0/750335763 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a7c026070 con 0x7f6a88102b70 2024-04-25T11:27:29.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.467+0000 7f6a857fa700 1 -- 172.21.15.90:0/750335763 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a7c021410 con 0x7f6a88102b70 2024-04-25T11:27:29.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.467+0000 7f6a8e486700 1 -- 172.21.15.90:0/750335763 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a881a25a0 con 0x7f6a88102b70 2024-04-25T11:27:29.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.467+0000 7f6a857fa700 1 -- 172.21.15.90:0/750335763 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a7c017650 con 0x7f6a88102b70 2024-04-25T11:27:29.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.467+0000 7f6a8e486700 1 -- 172.21.15.90:0/750335763 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a881a29f0 con 0x7f6a88102b70 2024-04-25T11:27:29.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.468+0000 7f6a857fa700 1 -- 172.21.15.90:0/750335763 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f6a7c029070 con 0x7f6a88102b70 2024-04-25T11:27:29.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.468+0000 7f6a857fa700 1 --2- 172.21.15.90:0/750335763 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6a7003d850 0x7f6a7003fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:27:29.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.468+0000 7f6a857fa700 1 -- 172.21.15.90:0/750335763 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6a7c0535b0 con 0x7f6a88102b70 2024-04-25T11:27:29.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.468+0000 7f6a8e486700 1 -- 172.21.15.90:0/750335763 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a88051190 con 0x7f6a88102b70 2024-04-25T11:27:29.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.468+0000 7f6a87fff700 1 --2- 172.21.15.90:0/750335763 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6a7003d850 0x7f6a7003fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:27:29.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.468+0000 7f6a87fff700 1 --2- 172.21.15.90:0/750335763 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6a7003d850 0x7f6a7003fd60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6a8806bf10 tx=0x7f6a78009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:27:29.463 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:29 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:29.463 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:29 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:29.463 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:29 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:29.463 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:29 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:29.463 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:29 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:27:29.463 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:29 smithi138 ceph-mon[32328]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:29.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.473+0000 7f6a857fa700 1 -- 172.21.15.90:0/750335763 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6a7c030020 con 0x7f6a88102b70 2024-04-25T11:27:29.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:29.682+0000 7f6a8e486700 1 -- 172.21.15.90:0/750335763 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f6a8802e970 con 0x7f6a7003d850 2024-04-25T11:27:29.841 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:29 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:27:29.470+0000 7f9c3b470200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T11:27:30.194 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:29 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:27:29.907+0000 7f9c3b470200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T11:27:30.195 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:30 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:27:30.002+0000 7f9c3b470200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T11:27:30.195 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:30 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:27:30.202+0000 7f9c3b470200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T11:27:30.509 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:30 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:27:30.180+0000 7fc6f6456200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T11:27:30.510 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:30 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:27:30.347+0000 7fc6f6456200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T11:27:30.510 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:30 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:27:30.517+0000 7fc6f6456200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T11:27:30.797 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:30 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:30.797 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:30 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:30.797 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:30 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:30.797 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:30 smithi138 ceph-mon[32328]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:27:30.797 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:30 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:30.797 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:30 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:27:30.798 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:30 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:30.798 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:30 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:27:30.798 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:30 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:27:30.798 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:30 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:30.798 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:30 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:30.798 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:30 smithi138 ceph-mon[32328]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T11:27:30.798 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:30 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T11:27:30.798 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:30 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T11:27:30.798 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:30 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:30.798 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:30 smithi138 ceph-mon[32328]: Reconfiguring daemon mgr.x on smithi090 2024-04-25T11:27:30.958 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:30 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:30.958 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:30 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:30.958 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:30 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:30.958 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:30 smithi090 ceph-mon[29575]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:27:30.959 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:30 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:30.959 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:30 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:27:30.959 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:30 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:30.959 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:30 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:27:30.959 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:30 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:27:30.959 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:30 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:30.959 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:30 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:30.959 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:30 smithi090 ceph-mon[29575]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T11:27:30.959 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:30 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T11:27:30.959 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:30 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T11:27:30.959 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:30 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:30.959 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:30 smithi090 ceph-mon[29575]: Reconfiguring daemon mgr.x on smithi090 2024-04-25T11:27:31.050 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:30 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:27:30.727+0000 7f9c3b470200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T11:27:31.050 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:30 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:27:30.819+0000 7f9c3b470200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T11:27:31.050 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:30 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:31.050 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:30 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:31.050 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:30 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:31.051 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:30 smithi156 ceph-mon[33045]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:27:31.051 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:30 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:31.051 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:30 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:27:31.051 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:30 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:31.051 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:30 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:27:31.051 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:30 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:27:31.051 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:30 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:31.051 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:30 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:31.051 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:30 smithi156 ceph-mon[33045]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T11:27:31.051 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:30 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T11:27:31.051 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:30 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T11:27:31.051 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:30 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:31.052 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:30 smithi156 ceph-mon[33045]: Reconfiguring daemon mgr.x on smithi090 2024-04-25T11:27:31.055 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:30 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:27:30.805+0000 7fc6f6456200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T11:27:31.055 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:30 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:27:30.896+0000 7fc6f6456200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T11:27:31.341 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:31 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:27:31.058+0000 7f9c3b470200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T11:27:31.341 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:31 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:27:31.191+0000 7f9c3b470200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T11:27:31.341 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:31 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:27:31.286+0000 7f9c3b470200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T11:27:31.555 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:31 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:27:31.099+0000 7fc6f6456200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T11:27:31.555 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:31 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:27:31.290+0000 7fc6f6456200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T11:27:31.645 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:31 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:27:31.401+0000 7f9c3b470200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T11:27:31.960 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:31 smithi090 ceph-mon[29575]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:32.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:31 smithi138 ceph-mon[32328]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:32.055 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:31 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:27:31.889+0000 7fc6f6456200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T11:27:32.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:31 smithi156 ceph-mon[33045]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:32.091 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:31 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:27:31.713+0000 7f9c3b470200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T11:27:32.555 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:32 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:27:32.312+0000 7fc6f6456200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T11:27:32.555 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:32 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:27:32.407+0000 7fc6f6456200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T11:27:32.590 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:32 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:27:32.317+0000 7f9c3b470200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T11:27:32.591 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:27:32 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:27:32.536+0000 7f9c3b470200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T11:27:32.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:32.928+0000 7f6a857fa700 1 -- 172.21.15.90:0/750335763 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 14) ==== 98518+0+0 (secure 0 0 0) 0x7f6a7c009d10 con 0x7f6a88102b70 2024-04-25T11:27:32.920 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:32 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:32.920 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:32 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:32.920 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:32 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:32.920 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:32 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:27:32.920 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:32 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:32.921 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:32 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:27:32.921 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:32 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:32.921 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:32 smithi090 ceph-mon[29575]: Standby manager daemon z started 2024-04-25T11:27:32.921 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:32 smithi090 ceph-mon[29575]: from='mgr.? 172.21.15.156:0/3240040190' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T11:27:32.921 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:32 smithi090 ceph-mon[29575]: from='mgr.? 172.21.15.156:0/3240040190' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T11:27:32.921 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:32 smithi090 ceph-mon[29575]: from='mgr.? 172.21.15.156:0/3240040190' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T11:27:32.921 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:32 smithi090 ceph-mon[29575]: from='mgr.? 172.21.15.156:0/3240040190' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T11:27:33.055 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:32 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:27:32.605+0000 7fc6f6456200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T11:27:33.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:32 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:33.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:32 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:33.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:32 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:33.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:32 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:27:33.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:32 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:33.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:32 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:27:33.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:32 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:33.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:32 smithi138 ceph-mon[32328]: Standby manager daemon z started 2024-04-25T11:27:33.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:32 smithi138 ceph-mon[32328]: from='mgr.? 172.21.15.156:0/3240040190' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T11:27:33.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:32 smithi138 ceph-mon[32328]: from='mgr.? 172.21.15.156:0/3240040190' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T11:27:33.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:32 smithi138 ceph-mon[32328]: from='mgr.? 172.21.15.156:0/3240040190' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T11:27:33.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:32 smithi138 ceph-mon[32328]: from='mgr.? 172.21.15.156:0/3240040190' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T11:27:33.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:32 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:33.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:32 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:33.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:32 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:33.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:32 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:27:33.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:32 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:33.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:32 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:27:33.092 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:32 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:33.092 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:32 smithi156 ceph-mon[33045]: Standby manager daemon z started 2024-04-25T11:27:33.092 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:32 smithi156 ceph-mon[33045]: from='mgr.? 172.21.15.156:0/3240040190' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T11:27:33.092 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:32 smithi156 ceph-mon[33045]: from='mgr.? 172.21.15.156:0/3240040190' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T11:27:33.092 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:32 smithi156 ceph-mon[33045]: from='mgr.? 172.21.15.156:0/3240040190' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T11:27:33.092 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:32 smithi156 ceph-mon[33045]: from='mgr.? 172.21.15.156:0/3240040190' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T11:27:33.454 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:33 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:27:33.127+0000 7fc6f6456200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T11:27:33.454 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:33 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:27:33.220+0000 7fc6f6456200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T11:27:33.717 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:33 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:27:33.462+0000 7fc6f6456200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T11:27:33.717 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:33 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:27:33.594+0000 7fc6f6456200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T11:27:33.718 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:33 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:27:33.688+0000 7fc6f6456200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T11:27:33.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:33 smithi090 ceph-mon[29575]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:33.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:33 smithi090 ceph-mon[29575]: mgrmap e14: x(active, since 76s), standbys: z 2024-04-25T11:27:33.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:33 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T11:27:33.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:33 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/4088990560' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e54d888-e67b-4951-8ae7-e355fb6b10a8"}]: dispatch 2024-04-25T11:27:33.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:33 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/4088990560' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e54d888-e67b-4951-8ae7-e355fb6b10a8"}]': finished 2024-04-25T11:27:33.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:33 smithi090 ceph-mon[29575]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T11:27:33.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:33 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T11:27:34.055 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:33 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:27:33.804+0000 7fc6f6456200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T11:27:34.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:33 smithi138 ceph-mon[32328]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:34.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:33 smithi138 ceph-mon[32328]: mgrmap e14: x(active, since 76s), standbys: z 2024-04-25T11:27:34.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:33 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T11:27:34.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:33 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/4088990560' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e54d888-e67b-4951-8ae7-e355fb6b10a8"}]: dispatch 2024-04-25T11:27:34.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:33 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/4088990560' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e54d888-e67b-4951-8ae7-e355fb6b10a8"}]': finished 2024-04-25T11:27:34.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:33 smithi138 ceph-mon[32328]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T11:27:34.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:33 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T11:27:34.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:33 smithi156 ceph-mon[33045]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:34.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:33 smithi156 ceph-mon[33045]: mgrmap e14: x(active, since 76s), standbys: z 2024-04-25T11:27:34.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:33 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T11:27:34.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:33 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/4088990560' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e54d888-e67b-4951-8ae7-e355fb6b10a8"}]: dispatch 2024-04-25T11:27:34.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:33 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/4088990560' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e54d888-e67b-4951-8ae7-e355fb6b10a8"}]': finished 2024-04-25T11:27:34.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:33 smithi156 ceph-mon[33045]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T11:27:34.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:33 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T11:27:34.555 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:34 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:27:34.108+0000 7fc6f6456200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T11:27:34.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:34 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/409921287' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:27:35.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:34 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/409921287' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:27:35.055 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:34 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:27:34.700+0000 7fc6f6456200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T11:27:35.055 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:27:34 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:27:34.915+0000 7fc6f6456200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T11:27:35.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:34 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/409921287' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:27:35.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:35.745+0000 7f6a857fa700 1 -- 172.21.15.90:0/750335763 <== mon.0 v2:172.21.15.90:3300/0 8 ==== mgrmap(e 15) ==== 147464+0+0 (secure 0 0 0) 0x7f6a7c029360 con 0x7f6a88102b70 2024-04-25T11:27:35.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:35 smithi090 ceph-mon[29575]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:35.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:35 smithi090 ceph-mon[29575]: Standby manager daemon y started 2024-04-25T11:27:35.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:35 smithi090 ceph-mon[29575]: from='mgr.? 172.21.15.138:0/2116957510' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T11:27:35.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:35 smithi090 ceph-mon[29575]: from='mgr.? 172.21.15.138:0/2116957510' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T11:27:35.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:35 smithi090 ceph-mon[29575]: from='mgr.? 172.21.15.138:0/2116957510' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T11:27:35.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:35 smithi090 ceph-mon[29575]: from='mgr.? 172.21.15.138:0/2116957510' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T11:27:36.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:35 smithi138 ceph-mon[32328]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:36.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:35 smithi138 ceph-mon[32328]: Standby manager daemon y started 2024-04-25T11:27:36.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:35 smithi138 ceph-mon[32328]: from='mgr.? 172.21.15.138:0/2116957510' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T11:27:36.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:35 smithi138 ceph-mon[32328]: from='mgr.? 172.21.15.138:0/2116957510' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T11:27:36.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:35 smithi138 ceph-mon[32328]: from='mgr.? 172.21.15.138:0/2116957510' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T11:27:36.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:35 smithi138 ceph-mon[32328]: from='mgr.? 172.21.15.138:0/2116957510' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T11:27:36.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:35 smithi156 ceph-mon[33045]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:36.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:35 smithi156 ceph-mon[33045]: Standby manager daemon y started 2024-04-25T11:27:36.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:35 smithi156 ceph-mon[33045]: from='mgr.? 172.21.15.138:0/2116957510' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T11:27:36.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:35 smithi156 ceph-mon[33045]: from='mgr.? 172.21.15.138:0/2116957510' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T11:27:36.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:35 smithi156 ceph-mon[33045]: from='mgr.? 172.21.15.138:0/2116957510' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T11:27:36.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:35 smithi156 ceph-mon[33045]: from='mgr.? 172.21.15.138:0/2116957510' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T11:27:37.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:36 smithi138 ceph-mon[32328]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-04-25T11:27:37.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:36 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T11:27:37.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:36 smithi156 ceph-mon[33045]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-04-25T11:27:37.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:36 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T11:27:37.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:36 smithi090 ceph-mon[29575]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-04-25T11:27:37.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:36 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T11:27:37.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:37 smithi090 ceph-mon[29575]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:38.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:37 smithi138 ceph-mon[32328]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:38.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:37 smithi156 ceph-mon[33045]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:39.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:39 smithi138 ceph-mon[32328]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:39.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:39 smithi090 ceph-mon[29575]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:40.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:39 smithi156 ceph-mon[33045]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:40.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:40 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T11:27:40.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:40 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:41.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:40 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T11:27:41.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:40 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:41.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:40 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T11:27:41.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:40 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:42.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:41 smithi138 ceph-mon[32328]: Deploying daemon osd.0 on smithi090 2024-04-25T11:27:42.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:41 smithi138 ceph-mon[32328]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:42.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:41 smithi156 ceph-mon[33045]: Deploying daemon osd.0 on smithi090 2024-04-25T11:27:42.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:41 smithi156 ceph-mon[33045]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:42.097 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:41 smithi090 ceph-mon[29575]: Deploying daemon osd.0 on smithi090 2024-04-25T11:27:42.098 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:41 smithi090 ceph-mon[29575]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:44.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:43 smithi138 ceph-mon[32328]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:44.073 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:43 smithi090 ceph-mon[29575]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:44.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:43 smithi156 ceph-mon[33045]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:45.826 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:45 smithi090 ceph-mon[29575]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:46.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:45 smithi138 ceph-mon[32328]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:46.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:45 smithi156 ceph-mon[33045]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:47.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:46 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:47.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:46 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:47.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:46 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:27:47.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:46 smithi090 ceph-mon[29575]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:47.305 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:46 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:47.305 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:46 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:47.305 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:46 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:27:47.305 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:46 smithi138 ceph-mon[32328]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:47.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:46 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:47.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:46 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:47.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:46 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:27:47.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:46 smithi156 ceph-mon[33045]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:49.029 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:48 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:49.029 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:48 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:49.029 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:48 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:49.030 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:48 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:27:49.030 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:48 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:49.030 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:48 smithi090 ceph-mon[29575]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:49.305 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:48 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:49.305 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:48 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:49.305 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:48 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:49.305 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:48 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:27:49.305 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:48 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:49.305 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:48 smithi138 ceph-mon[32328]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:49.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:48 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:49.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:48 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:49.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:48 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:49.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:48 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:27:49.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:48 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:49.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:48 smithi156 ceph-mon[33045]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:49.783 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 0 on host 'smithi090' 2024-04-25T11:27:49.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:49.790+0000 7f6a857fa700 1 -- 172.21.15.90:0/750335763 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6a78011ae0 con 0x7f6a7003d850 2024-04-25T11:27:49.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:49.793+0000 7f6a8e486700 1 -- 172.21.15.90:0/750335763 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6a7003d850 msgr2=0x7f6a7003fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:49.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:49.793+0000 7f6a8e486700 1 --2- 172.21.15.90:0/750335763 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6a7003d850 0x7f6a7003fd60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6a8806bf10 tx=0x7f6a78009260 comp rx=0 tx=0).stop 2024-04-25T11:27:49.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:49.793+0000 7f6a8e486700 1 -- 172.21.15.90:0/750335763 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a88102b70 msgr2=0x7f6a8819acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:49.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:49.793+0000 7f6a8e486700 1 --2- 172.21.15.90:0/750335763 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a88102b70 0x7f6a8819acf0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c00b760 tx=0x7f6a7c009580 comp rx=0 tx=0).stop 2024-04-25T11:27:49.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:49.793+0000 7f6a8e486700 1 -- 172.21.15.90:0/750335763 shutdown_connections 2024-04-25T11:27:49.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:49.793+0000 7f6a8e486700 1 --2- 172.21.15.90:0/750335763 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6a881057d0 0x7f6a881a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:49.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:49.793+0000 7f6a8e486700 1 --2- 172.21.15.90:0/750335763 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6a7003d850 0x7f6a7003fd60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:49.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:49.793+0000 7f6a8e486700 1 --2- 172.21.15.90:0/750335763 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a88102220 0x7f6a8819a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:49.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:49.793+0000 7f6a8e486700 1 --2- 172.21.15.90:0/750335763 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a88102b70 0x7f6a8819acf0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:49.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:49.793+0000 7f6a8e486700 1 -- 172.21.15.90:0/750335763 >> 172.21.15.90:0/750335763 conn(0x7f6a880fded0 msgr2=0x7f6a88108770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:27:49.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:49.794+0000 7f6a8e486700 1 -- 172.21.15.90:0/750335763 shutdown_connections 2024-04-25T11:27:49.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:27:49.794+0000 7f6a8e486700 1 -- 172.21.15.90:0/750335763 wait complete. 2024-04-25T11:27:50.381 DEBUG:teuthology.orchestra.run.smithi090:osd.0> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.0.service 2024-04-25T11:27:50.384 INFO:tasks.cephadm:Deploying osd.1 on smithi138 with /dev/vg_nvme/lv_4... 2024-04-25T11:27:50.384 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T11:27:50.629 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T11:27:50.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:50 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:50.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:50 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:50.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:50 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:27:50.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:50 smithi090 ceph-mon[29575]: from='osd.0 [v2:172.21.15.90:6802/1201629159,v1:172.21.15.90:6803/1201629159]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T11:27:51.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:50 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:51.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:50 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:51.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:50 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:27:51.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:50 smithi138 ceph-mon[32328]: from='osd.0 [v2:172.21.15.90:6802/1201629159,v1:172.21.15.90:6803/1201629159]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T11:27:51.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:50 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:51.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:50 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:51.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:50 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:27:51.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:50 smithi156 ceph-mon[33045]: from='osd.0 [v2:172.21.15.90:6802/1201629159,v1:172.21.15.90:6803/1201629159]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T11:27:51.881 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:51 smithi138 ceph-mon[32328]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:51.882 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:51 smithi138 ceph-mon[32328]: from='osd.0 [v2:172.21.15.90:6802/1201629159,v1:172.21.15.90:6803/1201629159]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T11:27:51.882 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:51 smithi138 ceph-mon[32328]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T11:27:51.882 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:51 smithi138 ceph-mon[32328]: from='osd.0 [v2:172.21.15.90:6802/1201629159,v1:172.21.15.90:6803/1201629159]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-25T11:27:51.882 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:51 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T11:27:51.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:51 smithi090 ceph-mon[29575]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:51.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:51 smithi090 ceph-mon[29575]: from='osd.0 [v2:172.21.15.90:6802/1201629159,v1:172.21.15.90:6803/1201629159]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T11:27:51.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:51 smithi090 ceph-mon[29575]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T11:27:51.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:51 smithi090 ceph-mon[29575]: from='osd.0 [v2:172.21.15.90:6802/1201629159,v1:172.21.15.90:6803/1201629159]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-25T11:27:51.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:51 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T11:27:51.979 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 25 11:27:51 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-0[44792]: 2024-04-25T11:27:51.781+0000 7f0e586a1700 -1 osd.0 0 waiting for initial osdmap 2024-04-25T11:27:51.979 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 25 11:27:51 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-0[44792]: 2024-04-25T11:27:51.786+0000 7f0e54456700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T11:27:52.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:51 smithi156 ceph-mon[33045]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:52.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:51 smithi156 ceph-mon[33045]: from='osd.0 [v2:172.21.15.90:6802/1201629159,v1:172.21.15.90:6803/1201629159]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T11:27:52.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:51 smithi156 ceph-mon[33045]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T11:27:52.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:51 smithi156 ceph-mon[33045]: from='osd.0 [v2:172.21.15.90:6802/1201629159,v1:172.21.15.90:6803/1201629159]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-25T11:27:52.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:51 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T11:27:52.466 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T11:27:52.467 INFO:teuthology.orchestra.run.smithi138.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T11:27:52.467 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10+0 records in 2024-04-25T11:27:52.467 INFO:teuthology.orchestra.run.smithi138.stderr:10+0 records out 2024-04-25T11:27:52.467 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111896 s, 937 MB/s 2024-04-25T11:27:52.467 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping successful for: 2024-04-25T11:27:53.012 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:52 smithi138 ceph-mon[32328]: from='osd.0 [v2:172.21.15.90:6802/1201629159,v1:172.21.15.90:6803/1201629159]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-25T11:27:53.012 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:52 smithi138 ceph-mon[32328]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T11:27:53.012 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:52 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T11:27:53.013 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:52 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T11:27:53.026 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:52 smithi090 ceph-mon[29575]: from='osd.0 [v2:172.21.15.90:6802/1201629159,v1:172.21.15.90:6803/1201629159]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-25T11:27:53.026 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:52 smithi090 ceph-mon[29575]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T11:27:53.026 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:52 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T11:27:53.026 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:52 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T11:27:53.039 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi138:vg_nvme/lv_4 2024-04-25T11:27:53.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:52 smithi156 ceph-mon[33045]: from='osd.0 [v2:172.21.15.90:6802/1201629159,v1:172.21.15.90:6803/1201629159]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-25T11:27:53.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:52 smithi156 ceph-mon[33045]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T11:27:53.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:52 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T11:27:53.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:52 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T11:27:53.293 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T11:27:53.995 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:53 smithi138 ceph-mon[32328]: purged_snaps scrub starts 2024-04-25T11:27:53.995 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:53 smithi138 ceph-mon[32328]: purged_snaps scrub ok 2024-04-25T11:27:53.995 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:53 smithi138 ceph-mon[32328]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:53.995 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:53 smithi138 ceph-mon[32328]: osd.0 [v2:172.21.15.90:6802/1201629159,v1:172.21.15.90:6803/1201629159] boot 2024-04-25T11:27:53.995 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:53 smithi138 ceph-mon[32328]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T11:27:53.995 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:53 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T11:27:53.995 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:53 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:53.995 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:53 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:53.995 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:53 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:27:53.995 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:53 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:53.995 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:53 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:53.996 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:53 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:27:53.996 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:53 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:54.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:53 smithi156 ceph-mon[33045]: purged_snaps scrub starts 2024-04-25T11:27:54.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:53 smithi156 ceph-mon[33045]: purged_snaps scrub ok 2024-04-25T11:27:54.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:53 smithi156 ceph-mon[33045]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:54.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:53 smithi156 ceph-mon[33045]: osd.0 [v2:172.21.15.90:6802/1201629159,v1:172.21.15.90:6803/1201629159] boot 2024-04-25T11:27:54.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:53 smithi156 ceph-mon[33045]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T11:27:54.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:53 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T11:27:54.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:53 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:54.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:53 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:54.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:53 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:27:54.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:53 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:54.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:53 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:54.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:53 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:27:54.092 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:53 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:54.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:53 smithi090 ceph-mon[29575]: purged_snaps scrub starts 2024-04-25T11:27:54.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:53 smithi090 ceph-mon[29575]: purged_snaps scrub ok 2024-04-25T11:27:54.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:53 smithi090 ceph-mon[29575]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T11:27:54.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:53 smithi090 ceph-mon[29575]: osd.0 [v2:172.21.15.90:6802/1201629159,v1:172.21.15.90:6803/1201629159] boot 2024-04-25T11:27:54.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:53 smithi090 ceph-mon[29575]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T11:27:54.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:53 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T11:27:54.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:53 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:54.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:53 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:54.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:53 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:27:54.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:53 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:54.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:53 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:54.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:53 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:27:54.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:53 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:27:54.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.803+0000 7f9d14076700 1 -- 172.21.15.138:0/2461788914 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d0c104210 msgr2=0x7f9d0c104620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:54.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.803+0000 7f9d14076700 1 --2- 172.21.15.138:0/2461788914 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d0c104210 0x7f9d0c104620 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9d08009990 tx=0x7f9d08009ca0 comp rx=0 tx=0).stop 2024-04-25T11:27:54.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.804+0000 7f9d14076700 1 -- 172.21.15.138:0/2461788914 shutdown_connections 2024-04-25T11:27:54.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.804+0000 7f9d14076700 1 --2- 172.21.15.138:0/2461788914 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d0c105ce0 0x7f9d0c10c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:54.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.804+0000 7f9d14076700 1 --2- 172.21.15.138:0/2461788914 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d0c104210 0x7f9d0c104620 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:54.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.804+0000 7f9d14076700 1 --2- 172.21.15.138:0/2461788914 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d0c105340 0x7f9d0c105750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:54.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.804+0000 7f9d14076700 1 -- 172.21.15.138:0/2461788914 >> 172.21.15.138:0/2461788914 conn(0x7f9d0c0ff870 msgr2=0x7f9d0c101d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:27:54.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.804+0000 7f9d14076700 1 -- 172.21.15.138:0/2461788914 shutdown_connections 2024-04-25T11:27:54.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.804+0000 7f9d14076700 1 -- 172.21.15.138:0/2461788914 wait complete. 2024-04-25T11:27:54.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.804+0000 7f9d14076700 1 Processor -- start 2024-04-25T11:27:54.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.804+0000 7f9d14076700 1 -- start start 2024-04-25T11:27:54.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.805+0000 7f9d14076700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d0c104210 0x7f9d0c19a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:27:54.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.805+0000 7f9d14076700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d0c105340 0x7f9d0c19ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:27:54.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.805+0000 7f9d14076700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d0c105ce0 0x7f9d0c1a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:27:54.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.805+0000 7f9d14076700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0c10fca0 con 0x7f9d0c104210 2024-04-25T11:27:54.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.805+0000 7f9d14076700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0c10fb20 con 0x7f9d0c105ce0 2024-04-25T11:27:54.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.805+0000 7f9d14076700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0c10fe20 con 0x7f9d0c105340 2024-04-25T11:27:54.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.805+0000 7f9d12613700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d0c105ce0 0x7f9d0c1a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:27:54.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.805+0000 7f9d12613700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d0c105ce0 0x7f9d0c1a1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:51886/0 (socket says 172.21.15.138:51886) 2024-04-25T11:27:54.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.805+0000 7f9d12613700 1 -- 172.21.15.138:0/11809043 learned_addr learned my addr 172.21.15.138:0/11809043 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-04-25T11:27:54.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.805+0000 7f9d11e12700 1 --2- 172.21.15.138:0/11809043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d0c104210 0x7f9d0c19a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:27:54.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.805+0000 7f9d11611700 1 --2- 172.21.15.138:0/11809043 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d0c105340 0x7f9d0c19ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:27:54.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.805+0000 7f9d12613700 1 -- 172.21.15.138:0/11809043 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d0c105340 msgr2=0x7f9d0c19ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:54.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.805+0000 7f9d12613700 1 --2- 172.21.15.138:0/11809043 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d0c105340 0x7f9d0c19ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:54.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.805+0000 7f9d12613700 1 -- 172.21.15.138:0/11809043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d0c104210 msgr2=0x7f9d0c19a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:27:54.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.805+0000 7f9d12613700 1 --2- 172.21.15.138:0/11809043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d0c104210 0x7f9d0c19a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:27:54.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.805+0000 7f9d12613700 1 -- 172.21.15.138:0/11809043 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d0c1a2340 con 0x7f9d0c105ce0 2024-04-25T11:27:54.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.805+0000 7f9d12613700 1 --2- 172.21.15.138:0/11809043 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d0c105ce0 0x7f9d0c1a1cc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9d04007ee0 tx=0x7f9d0400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:27:54.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.805+0000 7f9d02ffd700 1 -- 172.21.15.138:0/11809043 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0400dbd0 con 0x7f9d0c105ce0 2024-04-25T11:27:54.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.805+0000 7f9d14076700 1 -- 172.21.15.138:0/11809043 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d0c1a2630 con 0x7f9d0c105ce0 2024-04-25T11:27:54.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.805+0000 7f9d02ffd700 1 -- 172.21.15.138:0/11809043 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d0400dd70 con 0x7f9d0c105ce0 2024-04-25T11:27:54.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.805+0000 7f9d02ffd700 1 -- 172.21.15.138:0/11809043 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d04005990 con 0x7f9d0c105ce0 2024-04-25T11:27:54.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.805+0000 7f9d14076700 1 -- 172.21.15.138:0/11809043 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d0c1a2b70 con 0x7f9d0c105ce0 2024-04-25T11:27:54.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.806+0000 7f9d14076700 1 -- 172.21.15.138:0/11809043 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d0c051190 con 0x7f9d0c105ce0 2024-04-25T11:27:54.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.808+0000 7f9d02ffd700 1 -- 172.21.15.138:0/11809043 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 15) ==== 147464+0+0 (secure 0 0 0) 0x7f9d0401a020 con 0x7f9d0c105ce0 2024-04-25T11:27:54.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.808+0000 7f9d02ffd700 1 --2- 172.21.15.138:0/11809043 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f9cf80afaf0 0x7f9cf80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:27:54.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.808+0000 7f9d02ffd700 1 -- 172.21.15.138:0/11809043 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f9d040dcf50 con 0x7f9d0c105ce0 2024-04-25T11:27:54.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.809+0000 7f9d11e12700 1 --2- 172.21.15.138:0/11809043 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f9cf80afaf0 0x7f9cf80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:27:54.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.809+0000 7f9d11e12700 1 --2- 172.21.15.138:0/11809043 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f9cf80afaf0 0x7f9cf80b2000 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9d080038a0 tx=0x7f9d0801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:27:54.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:54.812+0000 7f9d02ffd700 1 -- 172.21.15.138:0/11809043 <== mon.2 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9d04010070 con 0x7f9d0c105ce0 2024-04-25T11:27:55.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:27:55.017+0000 7f9d14076700 1 -- 172.21.15.138:0/11809043 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f9d0c063ad0 con 0x7f9cf80afaf0 2024-04-25T11:27:55.009 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:54 smithi138 ceph-mon[32328]: Detected new or changed devices on smithi090 2024-04-25T11:27:55.009 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:54 smithi138 ceph-mon[32328]: Adjusting osd_memory_target on smithi090 to 17088M 2024-04-25T11:27:55.009 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:54 smithi138 ceph-mon[32328]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T11:27:55.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:54 smithi156 ceph-mon[33045]: Detected new or changed devices on smithi090 2024-04-25T11:27:55.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:54 smithi156 ceph-mon[33045]: Adjusting osd_memory_target on smithi090 to 17088M 2024-04-25T11:27:55.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:54 smithi156 ceph-mon[33045]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T11:27:55.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:54 smithi090 ceph-mon[29575]: Detected new or changed devices on smithi090 2024-04-25T11:27:55.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:54 smithi090 ceph-mon[29575]: Adjusting osd_memory_target on smithi090 to 17088M 2024-04-25T11:27:55.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:54 smithi090 ceph-mon[29575]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T11:27:55.923 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:55 smithi138 ceph-mon[32328]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:27:55.923 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:55 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:27:55.923 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:55 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:27:55.923 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:55 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:56.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:55 smithi156 ceph-mon[33045]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:27:56.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:55 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:27:56.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:55 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:27:56.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:55 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:56.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:55 smithi090 ceph-mon[29575]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:27:56.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:55 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:27:56.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:55 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:27:56.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:55 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:27:56.953 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:56 smithi138 ceph-mon[32328]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:27:57.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:56 smithi156 ceph-mon[33045]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:27:57.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:56 smithi090 ceph-mon[29575]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:27:58.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:57 smithi138 ceph-mon[32328]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:27:58.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:57 smithi138 ceph-mon[32328]: from='client.? 172.21.15.138:0/137631735' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91a71c78-dae9-4bc7-9f6f-9e55cfefbe28"}]: dispatch 2024-04-25T11:27:58.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:57 smithi138 ceph-mon[32328]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91a71c78-dae9-4bc7-9f6f-9e55cfefbe28"}]: dispatch 2024-04-25T11:27:58.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:57 smithi138 ceph-mon[32328]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91a71c78-dae9-4bc7-9f6f-9e55cfefbe28"}]': finished 2024-04-25T11:27:58.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:57 smithi138 ceph-mon[32328]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T11:27:58.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:57 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T11:27:58.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:57 smithi156 ceph-mon[33045]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:27:58.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:57 smithi156 ceph-mon[33045]: from='client.? 172.21.15.138:0/137631735' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91a71c78-dae9-4bc7-9f6f-9e55cfefbe28"}]: dispatch 2024-04-25T11:27:58.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:57 smithi156 ceph-mon[33045]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91a71c78-dae9-4bc7-9f6f-9e55cfefbe28"}]: dispatch 2024-04-25T11:27:58.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:57 smithi156 ceph-mon[33045]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91a71c78-dae9-4bc7-9f6f-9e55cfefbe28"}]': finished 2024-04-25T11:27:58.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:57 smithi156 ceph-mon[33045]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T11:27:58.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:57 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T11:27:58.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:57 smithi090 ceph-mon[29575]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:27:58.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:57 smithi090 ceph-mon[29575]: from='client.? 172.21.15.138:0/137631735' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91a71c78-dae9-4bc7-9f6f-9e55cfefbe28"}]: dispatch 2024-04-25T11:27:58.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:57 smithi090 ceph-mon[29575]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91a71c78-dae9-4bc7-9f6f-9e55cfefbe28"}]: dispatch 2024-04-25T11:27:58.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:57 smithi090 ceph-mon[29575]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91a71c78-dae9-4bc7-9f6f-9e55cfefbe28"}]': finished 2024-04-25T11:27:58.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:57 smithi090 ceph-mon[29575]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T11:27:58.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:57 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T11:27:59.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:58 smithi138 ceph-mon[32328]: from='client.? 172.21.15.138:0/1595437108' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:27:59.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:58 smithi156 ceph-mon[33045]: from='client.? 172.21.15.138:0/1595437108' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:27:59.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:58 smithi090 ceph-mon[29575]: from='client.? 172.21.15.138:0/1595437108' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:28:00.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:27:59 smithi138 ceph-mon[32328]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:00.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:27:59 smithi156 ceph-mon[33045]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:00.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:27:59 smithi090 ceph-mon[29575]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:01.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:00 smithi156 ceph-mon[33045]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:01.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:00 smithi090 ceph-mon[29575]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:01.305 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:00 smithi138 ceph-mon[32328]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:03.863 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:03 smithi138 ceph-mon[32328]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:03.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:03 smithi090 ceph-mon[29575]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:04.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:03 smithi156 ceph-mon[33045]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:05.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:05 smithi090 ceph-mon[29575]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:05.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T11:28:05.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:06.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:05 smithi138 ceph-mon[32328]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:06.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:05 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T11:28:06.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:05 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:06.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:05 smithi156 ceph-mon[33045]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:06.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T11:28:06.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:06.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:06 smithi138 ceph-mon[32328]: Deploying daemon osd.1 on smithi138 2024-04-25T11:28:07.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:06 smithi156 ceph-mon[33045]: Deploying daemon osd.1 on smithi138 2024-04-25T11:28:07.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:06 smithi090 ceph-mon[29575]: Deploying daemon osd.1 on smithi138 2024-04-25T11:28:08.012 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:07 smithi138 ceph-mon[32328]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:08.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:07 smithi156 ceph-mon[33045]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:08.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:07 smithi090 ceph-mon[29575]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:10.012 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:09 smithi138 ceph-mon[32328]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:10.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:09 smithi156 ceph-mon[33045]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:10.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:09 smithi090 ceph-mon[29575]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:11.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:11 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:11.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:11 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:11.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:11 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:28:11.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:11 smithi156 ceph-mon[33045]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:11.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:11 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:11.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:11 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:11.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:11 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:28:11.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:11 smithi090 ceph-mon[29575]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:11.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:11 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:11.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:11 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:11.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:11 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:28:11.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:11 smithi138 ceph-mon[32328]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:13.375 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:28:13.383+0000 7f9d02ffd700 1 -- 172.21.15.138:0/11809043 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9d0c063ad0 con 0x7f9cf80afaf0 2024-04-25T11:28:13.376 INFO:teuthology.orchestra.run.smithi138.stdout:Created osd(s) 1 on host 'smithi138' 2024-04-25T11:28:13.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:28:13.386+0000 7f9d14076700 1 -- 172.21.15.138:0/11809043 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f9cf80afaf0 msgr2=0x7f9cf80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:13.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:28:13.386+0000 7f9d14076700 1 --2- 172.21.15.138:0/11809043 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f9cf80afaf0 0x7f9cf80b2000 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9d080038a0 tx=0x7f9d0801a040 comp rx=0 tx=0).stop 2024-04-25T11:28:13.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:28:13.386+0000 7f9d14076700 1 -- 172.21.15.138:0/11809043 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d0c105ce0 msgr2=0x7f9d0c1a1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:13.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:28:13.386+0000 7f9d14076700 1 --2- 172.21.15.138:0/11809043 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d0c105ce0 0x7f9d0c1a1cc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9d04007ee0 tx=0x7f9d0400d3b0 comp rx=0 tx=0).stop 2024-04-25T11:28:13.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:28:13.386+0000 7f9d14076700 1 -- 172.21.15.138:0/11809043 shutdown_connections 2024-04-25T11:28:13.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:28:13.386+0000 7f9d14076700 1 --2- 172.21.15.138:0/11809043 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d0c105ce0 0x7f9d0c1a1cc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:13.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:28:13.386+0000 7f9d14076700 1 --2- 172.21.15.138:0/11809043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d0c104210 0x7f9d0c19a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:13.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:28:13.386+0000 7f9d14076700 1 --2- 172.21.15.138:0/11809043 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f9cf80afaf0 0x7f9cf80b2000 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:13.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:28:13.386+0000 7f9d14076700 1 --2- 172.21.15.138:0/11809043 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d0c105340 0x7f9d0c19ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:13.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:28:13.386+0000 7f9d14076700 1 -- 172.21.15.138:0/11809043 >> 172.21.15.138:0/11809043 conn(0x7f9d0c0ff870 msgr2=0x7f9d0c108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:13.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:28:13.386+0000 7f9d14076700 1 -- 172.21.15.138:0/11809043 shutdown_connections 2024-04-25T11:28:13.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:28:13.386+0000 7f9d14076700 1 -- 172.21.15.138:0/11809043 wait complete. 2024-04-25T11:28:13.961 DEBUG:teuthology.orchestra.run.smithi138:osd.1> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.1.service 2024-04-25T11:28:13.964 INFO:tasks.cephadm:Deploying osd.2 on smithi156 with /dev/vg_nvme/lv_4... 2024-04-25T11:28:13.964 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T11:28:13.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:13 smithi090 ceph-mon[29575]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:13.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:13 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:13.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:13 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:13.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:13 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:13.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:13 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:13.989 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:13 smithi138 ceph-mon[32328]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:13.989 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:13 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:13.989 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:13 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:13.989 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:13 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:13.990 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:13 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:13.991 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:13 smithi156 ceph-mon[33045]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:13.991 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:13 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:13.991 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:13 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:13.991 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:13 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:13.992 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:13 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:14.206 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T11:28:14.281 INFO:journalctl@ceph.osd.1.smithi138.stdout:Apr 25 11:28:14 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-1[36919]: 2024-04-25T11:28:14.010+0000 7fa1bfcc27c0 -1 osd.1 0 log_to_monitors true 2024-04-25T11:28:14.830 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:14 smithi156 ceph-mon[33045]: from='osd.1 [v2:172.21.15.138:6800/3976851933,v1:172.21.15.138:6801/3976851933]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T11:28:14.831 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:14 smithi156 ceph-mon[33045]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T11:28:15.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:14 smithi138 ceph-mon[32328]: from='osd.1 [v2:172.21.15.138:6800/3976851933,v1:172.21.15.138:6801/3976851933]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T11:28:15.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:14 smithi138 ceph-mon[32328]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T11:28:15.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:14 smithi090 ceph-mon[29575]: from='osd.1 [v2:172.21.15.138:6800/3976851933,v1:172.21.15.138:6801/3976851933]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T11:28:15.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:14 smithi090 ceph-mon[29575]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T11:28:15.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:15 smithi156 ceph-mon[33045]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:15.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:15 smithi156 ceph-mon[33045]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T11:28:15.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:15 smithi156 ceph-mon[33045]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T11:28:15.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:15 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T11:28:15.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:15 smithi156 ceph-mon[33045]: from='osd.1 [v2:172.21.15.138:6800/3976851933,v1:172.21.15.138:6801/3976851933]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-04-25T11:28:15.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:15 smithi156 ceph-mon[33045]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-04-25T11:28:15.904 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T11:28:15.904 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T11:28:15.904 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2024-04-25T11:28:15.904 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2024-04-25T11:28:15.904 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00832347 s, 1.3 GB/s 2024-04-25T11:28:15.905 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2024-04-25T11:28:16.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:15 smithi138 ceph-mon[32328]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:16.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:15 smithi138 ceph-mon[32328]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T11:28:16.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:15 smithi138 ceph-mon[32328]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T11:28:16.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:15 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T11:28:16.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:15 smithi138 ceph-mon[32328]: from='osd.1 [v2:172.21.15.138:6800/3976851933,v1:172.21.15.138:6801/3976851933]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-04-25T11:28:16.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:15 smithi138 ceph-mon[32328]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-04-25T11:28:16.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:15 smithi090 ceph-mon[29575]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:16.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:15 smithi090 ceph-mon[29575]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T11:28:16.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:15 smithi090 ceph-mon[29575]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T11:28:16.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:15 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T11:28:16.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:15 smithi090 ceph-mon[29575]: from='osd.1 [v2:172.21.15.138:6800/3976851933,v1:172.21.15.138:6801/3976851933]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-04-25T11:28:16.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:15 smithi090 ceph-mon[29575]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-04-25T11:28:16.473 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi156:vg_nvme/lv_4 2024-04-25T11:28:16.742 INFO:journalctl@ceph.osd.1.smithi138.stdout:Apr 25 11:28:16 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-1[36919]: 2024-04-25T11:28:16.673+0000 7fa1b5b28700 -1 osd.1 0 waiting for initial osdmap 2024-04-25T11:28:16.764 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T11:28:17.028 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:16 smithi138 ceph-mon[32328]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-04-25T11:28:17.028 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:16 smithi138 ceph-mon[32328]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T11:28:17.028 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T11:28:17.028 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T11:28:17.028 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:16 smithi138 ceph-mon[32328]: from='osd.1 ' entity='osd.1' 2024-04-25T11:28:17.028 INFO:journalctl@ceph.osd.1.smithi138.stdout:Apr 25 11:28:16 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-1[36919]: 2024-04-25T11:28:16.776+0000 7fa1ad0d4700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T11:28:17.034 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:16 smithi156 ceph-mon[33045]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-04-25T11:28:17.035 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:16 smithi156 ceph-mon[33045]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T11:28:17.035 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:16 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T11:28:17.035 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:16 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T11:28:17.035 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:16 smithi156 ceph-mon[33045]: from='osd.1 ' entity='osd.1' 2024-04-25T11:28:17.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:16 smithi090 ceph-mon[29575]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-04-25T11:28:17.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:16 smithi090 ceph-mon[29575]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T11:28:17.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:16 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T11:28:17.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:16 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T11:28:17.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:16 smithi090 ceph-mon[29575]: from='osd.1 ' entity='osd.1' 2024-04-25T11:28:18.049 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:17 smithi156 ceph-mon[33045]: purged_snaps scrub starts 2024-04-25T11:28:18.049 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:17 smithi156 ceph-mon[33045]: purged_snaps scrub ok 2024-04-25T11:28:18.049 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:17 smithi156 ceph-mon[33045]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:18.049 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T11:28:18.049 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:18.049 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:18.049 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:28:18.049 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:18.049 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:18.049 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:28:18.049 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:18.050 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:28:18.050 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:18.050 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:28:18.050 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:18.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:17 smithi138 ceph-mon[32328]: purged_snaps scrub starts 2024-04-25T11:28:18.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:17 smithi138 ceph-mon[32328]: purged_snaps scrub ok 2024-04-25T11:28:18.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:17 smithi138 ceph-mon[32328]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:18.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T11:28:18.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:18.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:18.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:28:18.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:18.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:18.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:28:18.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:18.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:28:18.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:18.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:28:18.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:18.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:17 smithi090 ceph-mon[29575]: purged_snaps scrub starts 2024-04-25T11:28:18.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:17 smithi090 ceph-mon[29575]: purged_snaps scrub ok 2024-04-25T11:28:18.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:17 smithi090 ceph-mon[29575]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:18.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T11:28:18.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:18.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:18.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:28:18.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:18.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:18.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:28:18.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:18.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:28:18.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:18.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:28:18.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:18.366 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.371+0000 7f9906a5b700 1 -- 172.21.15.156:0/695909041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9900107540 msgr2=0x7f9900107950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:18.366 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.371+0000 7f9906a5b700 1 --2- 172.21.15.156:0/695909041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9900107540 0x7f9900107950 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f98f4009990 tx=0x7f98f4009ca0 comp rx=0 tx=0).stop 2024-04-25T11:28:18.366 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.371+0000 7f9906a5b700 1 -- 172.21.15.156:0/695909041 shutdown_connections 2024-04-25T11:28:18.366 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.371+0000 7f9906a5b700 1 --2- 172.21.15.156:0/695909041 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9900107ee0 0x7f990010e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:18.366 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.371+0000 7f9906a5b700 1 --2- 172.21.15.156:0/695909041 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9900106410 0x7f9900106820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:18.366 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.371+0000 7f9906a5b700 1 --2- 172.21.15.156:0/695909041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9900107540 0x7f9900107950 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:18.367 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.371+0000 7f9906a5b700 1 -- 172.21.15.156:0/695909041 >> 172.21.15.156:0/695909041 conn(0x7f9900101a70 msgr2=0x7f9900103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:18.367 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.371+0000 7f9906a5b700 1 -- 172.21.15.156:0/695909041 shutdown_connections 2024-04-25T11:28:18.367 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.371+0000 7f9906a5b700 1 -- 172.21.15.156:0/695909041 wait complete. 2024-04-25T11:28:18.367 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.372+0000 7f9906a5b700 1 Processor -- start 2024-04-25T11:28:18.367 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.372+0000 7f9906a5b700 1 -- start start 2024-04-25T11:28:18.367 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.372+0000 7f9906a5b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9900106410 0x7f990019c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:18.367 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.372+0000 7f9906a5b700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9900107540 0x7f990019cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:18.367 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.372+0000 7f9906a5b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9900107ee0 0x7f99001a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:18.367 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.372+0000 7f9906a5b700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9900111e80 con 0x7f9900106410 2024-04-25T11:28:18.367 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.372+0000 7f9906a5b700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9900111d00 con 0x7f9900107540 2024-04-25T11:28:18.367 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.372+0000 7f9906a5b700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9900112000 con 0x7f9900107ee0 2024-04-25T11:28:18.367 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.372+0000 7f9904ff8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9900107ee0 0x7f99001a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:18.367 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.372+0000 7f9904ff8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9900107ee0 0x7f99001a3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:49678/0 (socket says 172.21.15.156:49678) 2024-04-25T11:28:18.368 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.372+0000 7f98ff7fe700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9900107540 0x7f990019cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:18.368 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.372+0000 7f9904ff8700 1 -- 172.21.15.156:0/1260977424 learned_addr learned my addr 172.21.15.156:0/1260977424 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-25T11:28:18.368 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.372+0000 7f98fffff700 1 --2- 172.21.15.156:0/1260977424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9900106410 0x7f990019c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:18.368 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.373+0000 7f9904ff8700 1 -- 172.21.15.156:0/1260977424 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9900107540 msgr2=0x7f990019cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:18.368 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.373+0000 7f9904ff8700 1 --2- 172.21.15.156:0/1260977424 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9900107540 0x7f990019cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:18.368 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.373+0000 7f9904ff8700 1 -- 172.21.15.156:0/1260977424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9900106410 msgr2=0x7f990019c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:18.368 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.373+0000 7f9904ff8700 1 --2- 172.21.15.156:0/1260977424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9900106410 0x7f990019c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:18.368 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.373+0000 7f9904ff8700 1 -- 172.21.15.156:0/1260977424 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99001a44c0 con 0x7f9900107ee0 2024-04-25T11:28:18.368 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.373+0000 7f9904ff8700 1 --2- 172.21.15.156:0/1260977424 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9900107ee0 0x7f99001a3e40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f98f8007ee0 tx=0x7f98f800d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:18.368 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.373+0000 7f98fd7fa700 1 -- 172.21.15.156:0/1260977424 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f8004bb0 con 0x7f9900107ee0 2024-04-25T11:28:18.368 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.373+0000 7f98fd7fa700 1 -- 172.21.15.156:0/1260977424 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98f8004d50 con 0x7f9900107ee0 2024-04-25T11:28:18.368 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.373+0000 7f9906a5b700 1 -- 172.21.15.156:0/1260977424 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99001a47b0 con 0x7f9900107ee0 2024-04-25T11:28:18.369 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.373+0000 7f98fd7fa700 1 -- 172.21.15.156:0/1260977424 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f8005920 con 0x7f9900107ee0 2024-04-25T11:28:18.369 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.373+0000 7f9906a5b700 1 -- 172.21.15.156:0/1260977424 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99001a4cf0 con 0x7f9900107ee0 2024-04-25T11:28:18.369 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.374+0000 7f9906a5b700 1 -- 172.21.15.156:0/1260977424 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9900051190 con 0x7f9900107ee0 2024-04-25T11:28:18.370 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.375+0000 7f98fd7fa700 1 -- 172.21.15.156:0/1260977424 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 15) ==== 147464+0+0 (secure 0 0 0) 0x7f98f801a020 con 0x7f9900107ee0 2024-04-25T11:28:18.370 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.376+0000 7f98fd7fa700 1 --2- 172.21.15.156:0/1260977424 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f98e80afb70 0x7f98e80b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:18.370 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.377+0000 7f98fd7fa700 1 -- 172.21.15.156:0/1260977424 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2309+0+0 (secure 0 0 0) 0x7f98f80dcfe0 con 0x7f9900107ee0 2024-04-25T11:28:18.370 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.377+0000 7f98fffff700 1 --2- 172.21.15.156:0/1260977424 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f98e80afb70 0x7f98e80b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:18.370 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.377+0000 7f98fffff700 1 --2- 172.21.15.156:0/1260977424 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f98e80afb70 0x7f98e80b2080 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9900107440 tx=0x7f98f0007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:18.372 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.380+0000 7f98fd7fa700 1 -- 172.21.15.156:0/1260977424 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f98f8010070 con 0x7f9900107ee0 2024-04-25T11:28:18.577 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:18.585+0000 7f9906a5b700 1 -- 172.21.15.156:0/1260977424 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f9900063ad0 con 0x7f98e80afb70 2024-04-25T11:28:18.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:18 smithi156 ceph-mon[33045]: Detected new or changed devices on smithi138 2024-04-25T11:28:18.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:18 smithi156 ceph-mon[33045]: Adjusting osd_memory_target on smithi138 to 17088M 2024-04-25T11:28:18.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:18 smithi156 ceph-mon[33045]: osd.1 [v2:172.21.15.138:6800/3976851933,v1:172.21.15.138:6801/3976851933] boot 2024-04-25T11:28:18.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:18 smithi156 ceph-mon[33045]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T11:28:18.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:18 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T11:28:18.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:18 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:28:18.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:18 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:28:18.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:18 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:19.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:18 smithi138 ceph-mon[32328]: Detected new or changed devices on smithi138 2024-04-25T11:28:19.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:18 smithi138 ceph-mon[32328]: Adjusting osd_memory_target on smithi138 to 17088M 2024-04-25T11:28:19.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:18 smithi138 ceph-mon[32328]: osd.1 [v2:172.21.15.138:6800/3976851933,v1:172.21.15.138:6801/3976851933] boot 2024-04-25T11:28:19.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:18 smithi138 ceph-mon[32328]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T11:28:19.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:18 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T11:28:19.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:18 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:28:19.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:18 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:28:19.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:18 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:19.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:18 smithi090 ceph-mon[29575]: Detected new or changed devices on smithi138 2024-04-25T11:28:19.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:18 smithi090 ceph-mon[29575]: Adjusting osd_memory_target on smithi138 to 17088M 2024-04-25T11:28:19.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:18 smithi090 ceph-mon[29575]: osd.1 [v2:172.21.15.138:6800/3976851933,v1:172.21.15.138:6801/3976851933] boot 2024-04-25T11:28:19.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:18 smithi090 ceph-mon[29575]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T11:28:19.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:18 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T11:28:19.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:18 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:28:19.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:18 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:28:19.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:18 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:19.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:19 smithi156 ceph-mon[33045]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:28:19.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:19 smithi156 ceph-mon[33045]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:19.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:19 smithi156 ceph-mon[33045]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T11:28:20.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:19 smithi138 ceph-mon[32328]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:28:20.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:19 smithi138 ceph-mon[32328]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:20.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:19 smithi138 ceph-mon[32328]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T11:28:20.129 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:19 smithi090 ceph-mon[29575]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:28:20.129 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:19 smithi090 ceph-mon[29575]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T11:28:20.129 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:19 smithi090 ceph-mon[29575]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T11:28:21.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:21 smithi156 ceph-mon[33045]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:21.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:21 smithi156 ceph-mon[33045]: from='client.? 172.21.15.156:0/2541765907' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "daff6c7f-80fa-4859-9339-ef7a5e840832"}]: dispatch 2024-04-25T11:28:21.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:21 smithi156 ceph-mon[33045]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "daff6c7f-80fa-4859-9339-ef7a5e840832"}]: dispatch 2024-04-25T11:28:21.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:21 smithi156 ceph-mon[33045]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "daff6c7f-80fa-4859-9339-ef7a5e840832"}]': finished 2024-04-25T11:28:21.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:21 smithi156 ceph-mon[33045]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T11:28:21.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:21 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T11:28:22.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:21 smithi138 ceph-mon[32328]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:22.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:21 smithi138 ceph-mon[32328]: from='client.? 172.21.15.156:0/2541765907' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "daff6c7f-80fa-4859-9339-ef7a5e840832"}]: dispatch 2024-04-25T11:28:22.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:21 smithi138 ceph-mon[32328]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "daff6c7f-80fa-4859-9339-ef7a5e840832"}]: dispatch 2024-04-25T11:28:22.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:21 smithi138 ceph-mon[32328]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "daff6c7f-80fa-4859-9339-ef7a5e840832"}]': finished 2024-04-25T11:28:22.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:21 smithi138 ceph-mon[32328]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T11:28:22.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:21 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T11:28:22.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:21 smithi090 ceph-mon[29575]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:22.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:21 smithi090 ceph-mon[29575]: from='client.? 172.21.15.156:0/2541765907' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "daff6c7f-80fa-4859-9339-ef7a5e840832"}]: dispatch 2024-04-25T11:28:22.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:21 smithi090 ceph-mon[29575]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "daff6c7f-80fa-4859-9339-ef7a5e840832"}]: dispatch 2024-04-25T11:28:22.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:21 smithi090 ceph-mon[29575]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "daff6c7f-80fa-4859-9339-ef7a5e840832"}]': finished 2024-04-25T11:28:22.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:21 smithi090 ceph-mon[29575]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T11:28:22.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:21 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T11:28:23.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:22 smithi138 ceph-mon[32328]: from='client.? 172.21.15.156:0/494715545' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:28:23.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:22 smithi156 ceph-mon[33045]: from='client.? 172.21.15.156:0/494715545' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:28:23.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:22 smithi090 ceph-mon[29575]: from='client.? 172.21.15.156:0/494715545' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:28:24.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:23 smithi138 ceph-mon[32328]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:24.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:23 smithi156 ceph-mon[33045]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:24.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:23 smithi090 ceph-mon[29575]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:26.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:25 smithi138 ceph-mon[32328]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:26.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:25 smithi156 ceph-mon[33045]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:26.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:25 smithi090 ceph-mon[29575]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:28.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:27 smithi138 ceph-mon[32328]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:28.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:27 smithi156 ceph-mon[33045]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:28.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:27 smithi090 ceph-mon[29575]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:29.046 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:28 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T11:28:29.046 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:28 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:29.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:28 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T11:28:29.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:28 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:29.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:28 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T11:28:29.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:28 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:30.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:29 smithi138 ceph-mon[32328]: Deploying daemon osd.2 on smithi156 2024-04-25T11:28:30.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:29 smithi138 ceph-mon[32328]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:30.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:29 smithi156 ceph-mon[33045]: Deploying daemon osd.2 on smithi156 2024-04-25T11:28:30.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:29 smithi156 ceph-mon[33045]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:30.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:29 smithi090 ceph-mon[29575]: Deploying daemon osd.2 on smithi156 2024-04-25T11:28:30.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:29 smithi090 ceph-mon[29575]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:32.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:31 smithi138 ceph-mon[32328]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:32.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:31 smithi156 ceph-mon[33045]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:32.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:31 smithi090 ceph-mon[29575]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:34.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:33 smithi138 ceph-mon[32328]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:34.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:33 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:34.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:33 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:34.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:33 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:28:34.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:33 smithi156 ceph-mon[33045]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:34.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:33 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:34.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:33 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:34.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:33 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:28:34.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:33 smithi090 ceph-mon[29575]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:34.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:33 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:34.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:33 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:34.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:33 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:28:36.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:35 smithi138 ceph-mon[32328]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:36.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:35 smithi156 ceph-mon[33045]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:36.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:35 smithi090 ceph-mon[29575]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:37.291 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 2 on host 'smithi156' 2024-04-25T11:28:37.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:37.300+0000 7f98fd7fa700 1 -- 172.21.15.156:0/1260977424 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9900063ad0 con 0x7f98e80afb70 2024-04-25T11:28:37.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:37.302+0000 7f9906a5b700 1 -- 172.21.15.156:0/1260977424 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f98e80afb70 msgr2=0x7f98e80b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:37.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:37.302+0000 7f9906a5b700 1 --2- 172.21.15.156:0/1260977424 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f98e80afb70 0x7f98e80b2080 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9900107440 tx=0x7f98f0007400 comp rx=0 tx=0).stop 2024-04-25T11:28:37.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:37.302+0000 7f9906a5b700 1 -- 172.21.15.156:0/1260977424 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9900107ee0 msgr2=0x7f99001a3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:37.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:37.302+0000 7f9906a5b700 1 --2- 172.21.15.156:0/1260977424 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9900107ee0 0x7f99001a3e40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f98f8007ee0 tx=0x7f98f800d3b0 comp rx=0 tx=0).stop 2024-04-25T11:28:37.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:37.302+0000 7f9906a5b700 1 -- 172.21.15.156:0/1260977424 shutdown_connections 2024-04-25T11:28:37.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:37.303+0000 7f9906a5b700 1 --2- 172.21.15.156:0/1260977424 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f98e80afb70 0x7f98e80b2080 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:37.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:37.303+0000 7f9906a5b700 1 --2- 172.21.15.156:0/1260977424 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9900107ee0 0x7f99001a3e40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:37.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:37.303+0000 7f9906a5b700 1 --2- 172.21.15.156:0/1260977424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9900106410 0x7f990019c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:37.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:37.303+0000 7f9906a5b700 1 --2- 172.21.15.156:0/1260977424 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9900107540 0x7f990019cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:37.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:37.303+0000 7f9906a5b700 1 -- 172.21.15.156:0/1260977424 >> 172.21.15.156:0/1260977424 conn(0x7f9900101a70 msgr2=0x7f990010a770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:37.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:37.303+0000 7f9906a5b700 1 -- 172.21.15.156:0/1260977424 shutdown_connections 2024-04-25T11:28:37.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:28:37.303+0000 7f9906a5b700 1 -- 172.21.15.156:0/1260977424 wait complete. 2024-04-25T11:28:37.518 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:37 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:37.518 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:37 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:37.518 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:37 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:37.518 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:37 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:28:37.518 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:37 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:37.518 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:37 smithi156 ceph-mon[33045]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:37.518 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:37 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:37.518 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:37 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:37.518 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:37 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:28:37.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:37 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:37.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:37 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:37.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:37 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:37.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:37 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:28:37.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:37 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:37.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:37 smithi090 ceph-mon[29575]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:37.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:37 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:37.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:37 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:37.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:37 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:28:37.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:37 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:37.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:37 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:37.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:37 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:37.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:37 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:28:37.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:37 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:37.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:37 smithi138 ceph-mon[32328]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:37.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:37 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:37.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:37 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:37.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:37 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:28:37.875 DEBUG:teuthology.orchestra.run.smithi156:osd.2> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.2.service 2024-04-25T11:28:37.877 INFO:tasks.cephadm:Deploying osd.3 on smithi090 with /dev/vg_nvme/lv_3... 2024-04-25T11:28:37.878 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T11:28:38.122 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:28:38.544 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:38 smithi090 ceph-mon[29575]: from='osd.2 [v2:172.21.15.156:6800/1505807670,v1:172.21.15.156:6801/1505807670]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T11:28:38.544 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:38 smithi090 ceph-mon[29575]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T11:28:38.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:38 smithi156 ceph-mon[33045]: from='osd.2 [v2:172.21.15.156:6800/1505807670,v1:172.21.15.156:6801/1505807670]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T11:28:38.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:38 smithi156 ceph-mon[33045]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T11:28:38.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:38 smithi138 ceph-mon[32328]: from='osd.2 [v2:172.21.15.156:6800/1505807670,v1:172.21.15.156:6801/1505807670]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T11:28:38.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:38 smithi138 ceph-mon[32328]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T11:28:39.578 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:39 smithi090 ceph-mon[29575]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T11:28:39.578 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:39 smithi090 ceph-mon[29575]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T11:28:39.578 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:39 smithi090 ceph-mon[29575]: from='osd.2 [v2:172.21.15.156:6800/1505807670,v1:172.21.15.156:6801/1505807670]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T11:28:39.578 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:39 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T11:28:39.578 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:39 smithi090 ceph-mon[29575]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T11:28:39.578 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:39 smithi090 ceph-mon[29575]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:39.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:39 smithi156 ceph-mon[33045]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T11:28:39.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:39 smithi156 ceph-mon[33045]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T11:28:39.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:39 smithi156 ceph-mon[33045]: from='osd.2 [v2:172.21.15.156:6800/1505807670,v1:172.21.15.156:6801/1505807670]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T11:28:39.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:39 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T11:28:39.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:39 smithi156 ceph-mon[33045]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T11:28:39.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:39 smithi156 ceph-mon[33045]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:39.653 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:39 smithi138 ceph-mon[32328]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T11:28:39.653 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:39 smithi138 ceph-mon[32328]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T11:28:39.653 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:39 smithi138 ceph-mon[32328]: from='osd.2 [v2:172.21.15.156:6800/1505807670,v1:172.21.15.156:6801/1505807670]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T11:28:39.653 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:39 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T11:28:39.653 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:39 smithi138 ceph-mon[32328]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T11:28:39.654 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:39 smithi138 ceph-mon[32328]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:39.937 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T11:28:39.937 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T11:28:39.938 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-04-25T11:28:39.938 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-04-25T11:28:39.938 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115763 s, 906 MB/s 2024-04-25T11:28:39.938 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-04-25T11:28:40.250 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 11:28:40 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-2[38187]: 2024-04-25T11:28:40.108+0000 7f1d7566f700 -1 osd.2 0 waiting for initial osdmap 2024-04-25T11:28:40.250 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 11:28:40 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-2[38187]: 2024-04-25T11:28:40.112+0000 7f1d6d41c700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T11:28:40.526 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:40 smithi156 ceph-mon[33045]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-25T11:28:40.526 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:40 smithi156 ceph-mon[33045]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T11:28:40.526 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:40 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T11:28:40.526 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:40 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T11:28:40.526 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:40 smithi156 ceph-mon[33045]: from='osd.2 ' entity='osd.2' 2024-04-25T11:28:40.527 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:40 smithi090 ceph-mon[29575]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-25T11:28:40.527 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:40 smithi090 ceph-mon[29575]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T11:28:40.527 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:40 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T11:28:40.527 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:40 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T11:28:40.527 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:40 smithi090 ceph-mon[29575]: from='osd.2 ' entity='osd.2' 2024-04-25T11:28:40.529 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi090:vg_nvme/lv_3 2024-04-25T11:28:40.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:40 smithi138 ceph-mon[32328]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-25T11:28:40.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:40 smithi138 ceph-mon[32328]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T11:28:40.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:40 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T11:28:40.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:40 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T11:28:40.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:40 smithi138 ceph-mon[32328]: from='osd.2 ' entity='osd.2' 2024-04-25T11:28:40.774 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:28:41.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:41 smithi090 ceph-mon[29575]: purged_snaps scrub starts 2024-04-25T11:28:41.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:41 smithi090 ceph-mon[29575]: purged_snaps scrub ok 2024-04-25T11:28:41.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:41 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T11:28:41.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:41 smithi090 ceph-mon[29575]: Detected new or changed devices on smithi156 2024-04-25T11:28:41.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:41 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:41.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:41 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:41.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:41 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:28:41.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:41 smithi090 ceph-mon[29575]: Adjusting osd_memory_target on smithi156 to 17088M 2024-04-25T11:28:41.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:41 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:41.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:41 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:41.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:41 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:28:41.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:41 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:41.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:41 smithi090 ceph-mon[29575]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:41.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:41 smithi090 ceph-mon[29575]: osd.2 [v2:172.21.15.156:6800/1505807670,v1:172.21.15.156:6801/1505807670] boot 2024-04-25T11:28:41.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:41 smithi090 ceph-mon[29575]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T11:28:41.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:41 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T11:28:41.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:41 smithi138 ceph-mon[32328]: purged_snaps scrub starts 2024-04-25T11:28:41.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:41 smithi138 ceph-mon[32328]: purged_snaps scrub ok 2024-04-25T11:28:41.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:41 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T11:28:41.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:41 smithi138 ceph-mon[32328]: Detected new or changed devices on smithi156 2024-04-25T11:28:41.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:41 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:41.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:41 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:41.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:41 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:28:41.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:41 smithi138 ceph-mon[32328]: Adjusting osd_memory_target on smithi156 to 17088M 2024-04-25T11:28:41.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:41 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:41.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:41 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:41.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:41 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:28:41.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:41 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:41.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:41 smithi138 ceph-mon[32328]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:41.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:41 smithi138 ceph-mon[32328]: osd.2 [v2:172.21.15.156:6800/1505807670,v1:172.21.15.156:6801/1505807670] boot 2024-04-25T11:28:41.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:41 smithi138 ceph-mon[32328]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T11:28:41.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:41 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T11:28:41.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:41 smithi156 ceph-mon[33045]: purged_snaps scrub starts 2024-04-25T11:28:41.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:41 smithi156 ceph-mon[33045]: purged_snaps scrub ok 2024-04-25T11:28:41.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:41 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T11:28:41.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:41 smithi156 ceph-mon[33045]: Detected new or changed devices on smithi156 2024-04-25T11:28:41.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:41 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:41.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:41 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:41.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:41 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:28:41.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:41 smithi156 ceph-mon[33045]: Adjusting osd_memory_target on smithi156 to 17088M 2024-04-25T11:28:41.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:41 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:41.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:41 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:41.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:41 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:28:41.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:41 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:41.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:41 smithi156 ceph-mon[33045]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T11:28:41.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:41 smithi156 ceph-mon[33045]: osd.2 [v2:172.21.15.156:6800/1505807670,v1:172.21.15.156:6801/1505807670] boot 2024-04-25T11:28:41.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:41 smithi156 ceph-mon[33045]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T11:28:41.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:41 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T11:28:42.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.244+0000 7f055d2b4700 1 -- 172.21.15.90:0/2589302121 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05581076d0 msgr2=0x7f0558109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:42.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.244+0000 7f055d2b4700 1 --2- 172.21.15.90:0/2589302121 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05581076d0 0x7f0558109af0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f054c009990 tx=0x7f054c009ca0 comp rx=0 tx=0).stop 2024-04-25T11:28:42.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.244+0000 7f055d2b4700 1 -- 172.21.15.90:0/2589302121 shutdown_connections 2024-04-25T11:28:42.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.244+0000 7f055d2b4700 1 --2- 172.21.15.90:0/2589302121 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f055810a030 0x7f055810c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:42.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.244+0000 7f055d2b4700 1 --2- 172.21.15.90:0/2589302121 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0558102550 0x7f0558102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:42.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.244+0000 7f055d2b4700 1 --2- 172.21.15.90:0/2589302121 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05581076d0 0x7f0558109af0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:42.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.244+0000 7f055d2b4700 1 -- 172.21.15.90:0/2589302121 >> 172.21.15.90:0/2589302121 conn(0x7f05580fded0 msgr2=0x7f0558100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:42.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.244+0000 7f055d2b4700 1 -- 172.21.15.90:0/2589302121 shutdown_connections 2024-04-25T11:28:42.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.244+0000 7f055d2b4700 1 -- 172.21.15.90:0/2589302121 wait complete. 2024-04-25T11:28:42.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.245+0000 7f055d2b4700 1 Processor -- start 2024-04-25T11:28:42.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.245+0000 7f055d2b4700 1 -- start start 2024-04-25T11:28:42.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.245+0000 7f055d2b4700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0558102550 0x7f055819a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:42.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.245+0000 7f055d2b4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05581076d0 0x7f055819ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:42.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.245+0000 7f055d2b4700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f055810a030 0x7f05581a1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:42.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.245+0000 7f055d2b4700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f055810fce0 con 0x7f05581076d0 2024-04-25T11:28:42.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.245+0000 7f055d2b4700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f055810fb60 con 0x7f0558102550 2024-04-25T11:28:42.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.245+0000 7f055d2b4700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f055810fe60 con 0x7f055810a030 2024-04-25T11:28:42.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.245+0000 7f05567fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05581076d0 0x7f055819ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:42.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.245+0000 7f05567fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05581076d0 0x7f055819ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52610/0 (socket says 172.21.15.90:52610) 2024-04-25T11:28:42.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.245+0000 7f05567fc700 1 -- 172.21.15.90:0/663148870 learned_addr learned my addr 172.21.15.90:0/663148870 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:28:42.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.245+0000 7f0556ffd700 1 --2- 172.21.15.90:0/663148870 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0558102550 0x7f055819a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:42.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.246+0000 7f05567fc700 1 -- 172.21.15.90:0/663148870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f055810a030 msgr2=0x7f05581a1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:42.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.246+0000 7f05567fc700 1 --2- 172.21.15.90:0/663148870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f055810a030 0x7f05581a1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:42.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.246+0000 7f05567fc700 1 -- 172.21.15.90:0/663148870 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0558102550 msgr2=0x7f055819a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:42.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.246+0000 7f05567fc700 1 --2- 172.21.15.90:0/663148870 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0558102550 0x7f055819a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:42.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.246+0000 7f05567fc700 1 -- 172.21.15.90:0/663148870 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05581a2390 con 0x7f05581076d0 2024-04-25T11:28:42.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.246+0000 7f05567fc700 1 --2- 172.21.15.90:0/663148870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05581076d0 0x7f055819ad70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f054c003800 tx=0x7f054c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:42.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.246+0000 7f053ffff700 1 -- 172.21.15.90:0/663148870 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054c027070 con 0x7f05581076d0 2024-04-25T11:28:42.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.246+0000 7f053ffff700 1 -- 172.21.15.90:0/663148870 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f054c022470 con 0x7f05581076d0 2024-04-25T11:28:42.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.246+0000 7f055d2b4700 1 -- 172.21.15.90:0/663148870 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05581a2620 con 0x7f05581076d0 2024-04-25T11:28:42.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.246+0000 7f053ffff700 1 -- 172.21.15.90:0/663148870 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054c00f6e0 con 0x7f05581076d0 2024-04-25T11:28:42.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.246+0000 7f055d2b4700 1 -- 172.21.15.90:0/663148870 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05581a2af0 con 0x7f05581076d0 2024-04-25T11:28:42.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.247+0000 7f055d2b4700 1 -- 172.21.15.90:0/663148870 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0558051190 con 0x7f05581076d0 2024-04-25T11:28:42.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.249+0000 7f053ffff700 1 -- 172.21.15.90:0/663148870 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 15) ==== 147464+0+0 (secure 0 0 0) 0x7f054c02a070 con 0x7f05581076d0 2024-04-25T11:28:42.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.250+0000 7f053ffff700 1 --2- 172.21.15.90:0/663148870 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f05400afaf0 0x7f05400b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:42.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.250+0000 7f0556ffd700 1 --2- 172.21.15.90:0/663148870 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f05400afaf0 0x7f05400b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:42.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.250+0000 7f053ffff700 1 -- 172.21.15.90:0/663148870 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 2851+0+0 (secure 0 0 0) 0x7f054c0de6b0 con 0x7f05581076d0 2024-04-25T11:28:42.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.250+0000 7f0556ffd700 1 --2- 172.21.15.90:0/663148870 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f05400afaf0 0x7f05400b2000 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0558105650 tx=0x7f0548009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:42.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.254+0000 7f053ffff700 1 -- 172.21.15.90:0/663148870 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f054c013070 con 0x7f05581076d0 2024-04-25T11:28:42.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:42.460+0000 7f055d2b4700 1 -- 172.21.15.90:0/663148870 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f0558063b90 con 0x7f05400afaf0 2024-04-25T11:28:43.696 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:43 smithi090 ceph-mon[29575]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T11:28:43.696 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:43 smithi090 ceph-mon[29575]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:28:43.696 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:43 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:28:43.696 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:43 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:28:43.696 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:43 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:43.696 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:43 smithi090 ceph-mon[29575]: pgmap v82: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:28:43.696 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:43 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T11:28:43.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:43 smithi138 ceph-mon[32328]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T11:28:43.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:43 smithi138 ceph-mon[32328]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:28:43.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:43 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:28:43.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:43 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:28:43.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:43 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:43.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:43 smithi138 ceph-mon[32328]: pgmap v82: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:28:43.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:43 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T11:28:43.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:43 smithi156 ceph-mon[33045]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T11:28:43.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:43 smithi156 ceph-mon[33045]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:28:43.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:43 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:28:43.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:43 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:28:43.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:43 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:43.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:43 smithi156 ceph-mon[33045]: pgmap v82: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:28:43.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:43 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T11:28:44.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:44 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T11:28:44.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:44 smithi090 ceph-mon[29575]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T11:28:44.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:44 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T11:28:44.482 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 25 11:28:44 smithi090 sudo[49221]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T11:28:44.482 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 25 11:28:44 smithi090 sudo[49221]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T11:28:44.482 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 25 11:28:44 smithi090 sudo[49221]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T11:28:44.482 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 25 11:28:44 smithi090 sudo[49221]: pam_unix(sudo:session): session closed for user root 2024-04-25T11:28:44.482 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 25 11:28:44 smithi090 sudo[49224]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T11:28:44.483 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 25 11:28:44 smithi090 sudo[49224]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T11:28:44.483 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 25 11:28:44 smithi090 sudo[49224]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T11:28:44.483 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 25 11:28:44 smithi090 sudo[49224]: pam_unix(sudo:session): session closed for user root 2024-04-25T11:28:44.677 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:44 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T11:28:44.678 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:44 smithi156 ceph-mon[33045]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T11:28:44.678 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:44 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T11:28:44.678 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 11:28:44 smithi156 sudo[41798]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T11:28:44.678 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 11:28:44 smithi156 sudo[41798]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T11:28:44.678 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 11:28:44 smithi156 sudo[41798]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T11:28:44.678 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 11:28:44 smithi156 sudo[41798]: pam_unix(sudo:session): session closed for user root 2024-04-25T11:28:44.678 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 11:28:44 smithi156 sudo[41801]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T11:28:44.679 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 11:28:44 smithi156 sudo[41801]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T11:28:44.679 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 11:28:44 smithi156 sudo[41801]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T11:28:44.679 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 11:28:44 smithi156 sudo[41801]: pam_unix(sudo:session): session closed for user root 2024-04-25T11:28:44.715 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:44 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T11:28:44.715 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:44 smithi138 ceph-mon[32328]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T11:28:44.716 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:44 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T11:28:44.716 INFO:journalctl@ceph.osd.1.smithi138.stdout:Apr 25 11:28:44 smithi138 sudo[40779]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T11:28:44.716 INFO:journalctl@ceph.osd.1.smithi138.stdout:Apr 25 11:28:44 smithi138 sudo[40779]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T11:28:44.716 INFO:journalctl@ceph.osd.1.smithi138.stdout:Apr 25 11:28:44 smithi138 sudo[40779]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T11:28:44.716 INFO:journalctl@ceph.osd.1.smithi138.stdout:Apr 25 11:28:44 smithi138 sudo[40779]: pam_unix(sudo:session): session closed for user root 2024-04-25T11:28:44.716 INFO:journalctl@ceph.osd.1.smithi138.stdout:Apr 25 11:28:44 smithi138 sudo[40782]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T11:28:44.716 INFO:journalctl@ceph.osd.1.smithi138.stdout:Apr 25 11:28:44 smithi138 sudo[40782]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T11:28:44.716 INFO:journalctl@ceph.osd.1.smithi138.stdout:Apr 25 11:28:44 smithi138 sudo[40782]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T11:28:44.716 INFO:journalctl@ceph.osd.1.smithi138.stdout:Apr 25 11:28:44 smithi138 sudo[40782]: pam_unix(sudo:session): session closed for user root 2024-04-25T11:28:44.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:44 smithi090 sudo[49237]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T11:28:44.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:44 smithi090 sudo[49237]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T11:28:44.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:44 smithi090 sudo[49237]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T11:28:44.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:44 smithi090 sudo[49237]: pam_unix(sudo:session): session closed for user root 2024-04-25T11:28:44.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:44 smithi090 sudo[49243]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T11:28:44.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:44 smithi090 sudo[49243]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T11:28:44.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:44 smithi090 sudo[49243]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T11:28:44.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:44 smithi090 sudo[49243]: pam_unix(sudo:session): session closed for user root 2024-04-25T11:28:45.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:44 smithi138 sudo[40785]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T11:28:45.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:44 smithi138 sudo[40785]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T11:28:45.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:44 smithi138 sudo[40785]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T11:28:45.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:44 smithi138 sudo[40785]: pam_unix(sudo:session): session closed for user root 2024-04-25T11:28:45.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:44 smithi138 sudo[40788]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T11:28:45.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:44 smithi138 sudo[40788]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T11:28:45.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:44 smithi138 sudo[40788]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T11:28:45.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:44 smithi138 sudo[40788]: pam_unix(sudo:session): session closed for user root 2024-04-25T11:28:45.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:44 smithi156 sudo[41804]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T11:28:45.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:44 smithi156 sudo[41804]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T11:28:45.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:44 smithi156 sudo[41804]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T11:28:45.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:44 smithi156 sudo[41804]: pam_unix(sudo:session): session closed for user root 2024-04-25T11:28:45.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:44 smithi156 sudo[41807]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T11:28:45.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:44 smithi156 sudo[41807]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T11:28:45.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:44 smithi156 sudo[41807]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T11:28:45.091 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:44 smithi156 sudo[41807]: pam_unix(sudo:session): session closed for user root 2024-04-25T11:28:45.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:28:45.649+0000 7f053ffff700 1 -- 172.21.15.90:0/663148870 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f054c022620 con 0x7f05581076d0 2024-04-25T11:28:45.640 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:45 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T11:28:45.640 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:45 smithi090 ceph-mon[29575]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T11:28:45.640 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:45 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:45.641 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:45 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:45.641 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:45 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:45.641 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:45 smithi090 ceph-mon[29575]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T11:28:45.641 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:45 smithi090 ceph-mon[29575]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T11:28:45.641 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:45 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T11:28:45.641 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:45 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:28:45.641 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:45 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:28:45.641 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:45 smithi090 ceph-mon[29575]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T11:28:45.641 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:45 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T11:28:45.641 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:45 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:28:45.641 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:45 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:28:45.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:45 smithi090 ceph-mon[29575]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:28:45.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:45 smithi090 ceph-mon[29575]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T11:28:45.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:45 smithi090 ceph-mon[29575]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T11:28:45.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:45 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T11:28:45.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:45 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:28:45.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:45 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:28:45.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:45 smithi090 ceph-mon[29575]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T11:28:45.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:45 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2166348240' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33fd94ca-7465-401a-bc1e-706b6b31b9b0"}]: dispatch 2024-04-25T11:28:45.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:45 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2166348240' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33fd94ca-7465-401a-bc1e-706b6b31b9b0"}]': finished 2024-04-25T11:28:45.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:45 smithi090 ceph-mon[29575]: osdmap e22: 4 total, 3 up, 4 in 2024-04-25T11:28:45.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:45 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T11:28:45.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:45 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T11:28:45.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:45 smithi138 ceph-mon[32328]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T11:28:45.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:45 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:45.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:45 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:45.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:45 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:45.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:45 smithi138 ceph-mon[32328]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T11:28:45.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:45 smithi138 ceph-mon[32328]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T11:28:45.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:45 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T11:28:45.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:45 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:28:45.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:45 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:28:45.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:45 smithi138 ceph-mon[32328]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T11:28:45.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:45 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T11:28:45.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:45 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:28:45.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:45 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:28:45.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:45 smithi138 ceph-mon[32328]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:28:45.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:45 smithi138 ceph-mon[32328]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T11:28:45.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:45 smithi138 ceph-mon[32328]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T11:28:45.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:45 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T11:28:45.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:45 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:28:45.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:45 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:28:45.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:45 smithi138 ceph-mon[32328]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T11:28:45.807 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:45 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2166348240' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33fd94ca-7465-401a-bc1e-706b6b31b9b0"}]: dispatch 2024-04-25T11:28:45.807 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:45 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2166348240' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33fd94ca-7465-401a-bc1e-706b6b31b9b0"}]': finished 2024-04-25T11:28:45.807 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:45 smithi138 ceph-mon[32328]: osdmap e22: 4 total, 3 up, 4 in 2024-04-25T11:28:45.807 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:45 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T11:28:45.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:45 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T11:28:45.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:45 smithi156 ceph-mon[33045]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T11:28:45.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:45 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:45.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:45 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:45.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:45 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:45.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:45 smithi156 ceph-mon[33045]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T11:28:45.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:45 smithi156 ceph-mon[33045]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T11:28:45.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:45 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T11:28:45.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:45 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:28:45.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:45 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:28:45.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:45 smithi156 ceph-mon[33045]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T11:28:45.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:45 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T11:28:45.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:45 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:28:45.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:45 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:28:45.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:45 smithi156 ceph-mon[33045]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:28:45.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:45 smithi156 ceph-mon[33045]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T11:28:45.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:45 smithi156 ceph-mon[33045]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T11:28:45.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:45 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T11:28:45.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:45 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:28:45.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:45 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:28:45.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:45 smithi156 ceph-mon[33045]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T11:28:45.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:45 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2166348240' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33fd94ca-7465-401a-bc1e-706b6b31b9b0"}]: dispatch 2024-04-25T11:28:45.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:45 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2166348240' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33fd94ca-7465-401a-bc1e-706b6b31b9b0"}]': finished 2024-04-25T11:28:45.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:45 smithi156 ceph-mon[33045]: osdmap e22: 4 total, 3 up, 4 in 2024-04-25T11:28:45.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:45 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T11:28:46.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:46 smithi090 ceph-mon[29575]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T11:28:46.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:46 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3414426376' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:28:47.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:46 smithi138 ceph-mon[32328]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T11:28:47.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:46 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3414426376' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:28:47.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:46 smithi156 ceph-mon[33045]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T11:28:47.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:46 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3414426376' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:28:47.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:47 smithi090 ceph-mon[29575]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 93 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:28:48.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:47 smithi138 ceph-mon[32328]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 93 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:28:48.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:47 smithi156 ceph-mon[33045]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 93 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:28:50.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:49 smithi138 ceph-mon[32328]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 93 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:28:50.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:49 smithi156 ceph-mon[33045]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 93 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:28:50.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:49 smithi090 ceph-mon[29575]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 93 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:28:51.826 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:51 smithi090 ceph-mon[29575]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:28:52.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:51 smithi138 ceph-mon[32328]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:28:52.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:51 smithi156 ceph-mon[33045]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:28:53.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:52 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T11:28:53.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:52 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:53.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:52 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T11:28:53.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:52 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:53.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:52 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T11:28:53.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:52 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:28:53.948 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:53 smithi090 ceph-mon[29575]: Deploying daemon osd.3 on smithi090 2024-04-25T11:28:53.948 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:53 smithi090 ceph-mon[29575]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:28:54.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:53 smithi138 ceph-mon[32328]: Deploying daemon osd.3 on smithi090 2024-04-25T11:28:54.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:53 smithi138 ceph-mon[32328]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:28:54.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:53 smithi156 ceph-mon[33045]: Deploying daemon osd.3 on smithi090 2024-04-25T11:28:54.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:53 smithi156 ceph-mon[33045]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:28:56.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:55 smithi138 ceph-mon[32328]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:28:56.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:55 smithi156 ceph-mon[33045]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:28:56.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:55 smithi090 ceph-mon[29575]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:28:57.931 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:57 smithi090 ceph-mon[29575]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:28:58.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:57 smithi138 ceph-mon[32328]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:28:58.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:57 smithi156 ceph-mon[33045]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:28:59.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:59 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:59.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:59 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:59.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:59 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:28:59.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:28:59 smithi090 ceph-mon[29575]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:28:59.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:59 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:59.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:59 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:59.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:59 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:28:59.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:28:59 smithi138 ceph-mon[32328]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:28:59.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:59 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:59.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:59 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:28:59.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:59 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:28:59.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:28:59 smithi156 ceph-mon[33045]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:29:01.812 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:01 smithi090 ceph-mon[29575]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:29:01.812 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:01 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:01.812 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:01 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:01.812 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:01 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:01.812 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:01 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:29:01.812 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:01 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:01.859 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 3 on host 'smithi090' 2024-04-25T11:29:01.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:01.868+0000 7f053ffff700 1 -- 172.21.15.90:0/663148870 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0558063b90 con 0x7f05400afaf0 2024-04-25T11:29:01.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:01.870+0000 7f055d2b4700 1 -- 172.21.15.90:0/663148870 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f05400afaf0 msgr2=0x7f05400b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:01.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:01.870+0000 7f055d2b4700 1 --2- 172.21.15.90:0/663148870 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f05400afaf0 0x7f05400b2000 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0558105650 tx=0x7f0548009190 comp rx=0 tx=0).stop 2024-04-25T11:29:01.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:01.870+0000 7f055d2b4700 1 -- 172.21.15.90:0/663148870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05581076d0 msgr2=0x7f055819ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:01.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:01.870+0000 7f055d2b4700 1 --2- 172.21.15.90:0/663148870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05581076d0 0x7f055819ad70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f054c003800 tx=0x7f054c0047b0 comp rx=0 tx=0).stop 2024-04-25T11:29:01.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:01.870+0000 7f055d2b4700 1 -- 172.21.15.90:0/663148870 shutdown_connections 2024-04-25T11:29:01.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:01.870+0000 7f055d2b4700 1 --2- 172.21.15.90:0/663148870 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f05400afaf0 0x7f05400b2000 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:01.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:01.870+0000 7f055d2b4700 1 --2- 172.21.15.90:0/663148870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f055810a030 0x7f05581a1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:01.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:01.870+0000 7f055d2b4700 1 --2- 172.21.15.90:0/663148870 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0558102550 0x7f055819a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:01.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:01.870+0000 7f055d2b4700 1 --2- 172.21.15.90:0/663148870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05581076d0 0x7f055819ad70 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:01.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:01.870+0000 7f055d2b4700 1 -- 172.21.15.90:0/663148870 >> 172.21.15.90:0/663148870 conn(0x7f05580fded0 msgr2=0x7f0558108630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:29:01.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:01.870+0000 7f055d2b4700 1 -- 172.21.15.90:0/663148870 shutdown_connections 2024-04-25T11:29:01.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:01.870+0000 7f055d2b4700 1 -- 172.21.15.90:0/663148870 wait complete. 2024-04-25T11:29:02.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:01 smithi138 ceph-mon[32328]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:29:02.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:01 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:02.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:01 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:02.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:01 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:02.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:01 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:29:02.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:01 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:02.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:01 smithi156 ceph-mon[33045]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:29:02.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:01 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:02.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:01 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:02.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:01 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:02.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:01 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:29:02.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:01 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:02.456 DEBUG:teuthology.orchestra.run.smithi090:osd.3> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.3.service 2024-04-25T11:29:02.458 INFO:tasks.cephadm:Deploying osd.4 on smithi138 with /dev/vg_nvme/lv_3... 2024-04-25T11:29:02.459 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T11:29:02.702 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T11:29:02.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:02 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:02.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:02 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:02.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:02 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:29:02.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:02 smithi090 ceph-mon[29575]: from='osd.3 [v2:172.21.15.90:6810/2520012383,v1:172.21.15.90:6811/2520012383]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T11:29:02.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:02 smithi090 ceph-mon[29575]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:29:03.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:02 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:03.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:02 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:03.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:02 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:29:03.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:02 smithi138 ceph-mon[32328]: from='osd.3 [v2:172.21.15.90:6810/2520012383,v1:172.21.15.90:6811/2520012383]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T11:29:03.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:02 smithi138 ceph-mon[32328]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:29:03.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:02 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:03.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:02 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:03.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:02 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:29:03.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:02 smithi156 ceph-mon[33045]: from='osd.3 [v2:172.21.15.90:6810/2520012383,v1:172.21.15.90:6811/2520012383]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T11:29:03.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:02 smithi156 ceph-mon[33045]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:29:03.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:03 smithi090 ceph-mon[29575]: from='osd.3 [v2:172.21.15.90:6810/2520012383,v1:172.21.15.90:6811/2520012383]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T11:29:03.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:03 smithi090 ceph-mon[29575]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T11:29:03.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:03 smithi090 ceph-mon[29575]: from='osd.3 [v2:172.21.15.90:6810/2520012383,v1:172.21.15.90:6811/2520012383]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-25T11:29:03.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:03 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T11:29:03.940 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 25 11:29:03 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-3[51333]: 2024-04-25T11:29:03.876+0000 7fe32d3b7700 -1 osd.3 0 waiting for initial osdmap 2024-04-25T11:29:03.940 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 25 11:29:03 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-3[51333]: 2024-04-25T11:29:03.885+0000 7fe32916c700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T11:29:04.009 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:03 smithi138 ceph-mon[32328]: from='osd.3 [v2:172.21.15.90:6810/2520012383,v1:172.21.15.90:6811/2520012383]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T11:29:04.009 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:03 smithi138 ceph-mon[32328]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T11:29:04.009 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:03 smithi138 ceph-mon[32328]: from='osd.3 [v2:172.21.15.90:6810/2520012383,v1:172.21.15.90:6811/2520012383]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-25T11:29:04.010 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:03 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T11:29:04.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:03 smithi156 ceph-mon[33045]: from='osd.3 [v2:172.21.15.90:6810/2520012383,v1:172.21.15.90:6811/2520012383]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T11:29:04.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:03 smithi156 ceph-mon[33045]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T11:29:04.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:03 smithi156 ceph-mon[33045]: from='osd.3 [v2:172.21.15.90:6810/2520012383,v1:172.21.15.90:6811/2520012383]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-25T11:29:04.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:03 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T11:29:04.555 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T11:29:04.555 INFO:teuthology.orchestra.run.smithi138.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T11:29:04.555 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10+0 records in 2024-04-25T11:29:04.555 INFO:teuthology.orchestra.run.smithi138.stderr:10+0 records out 2024-04-25T11:29:04.555 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111036 s, 944 MB/s 2024-04-25T11:29:04.555 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping successful for: 2024-04-25T11:29:05.142 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:04 smithi138 ceph-mon[32328]: from='osd.3 [v2:172.21.15.90:6810/2520012383,v1:172.21.15.90:6811/2520012383]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-25T11:29:05.142 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:04 smithi138 ceph-mon[32328]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T11:29:05.142 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:04 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T11:29:05.142 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:04 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T11:29:05.142 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:04 smithi138 ceph-mon[32328]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:29:05.143 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi138:vg_nvme/lv_3 2024-04-25T11:29:05.202 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:04 smithi090 ceph-mon[29575]: from='osd.3 [v2:172.21.15.90:6810/2520012383,v1:172.21.15.90:6811/2520012383]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-25T11:29:05.202 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:04 smithi090 ceph-mon[29575]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T11:29:05.203 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:04 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T11:29:05.203 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:04 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T11:29:05.203 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:04 smithi090 ceph-mon[29575]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:29:05.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:04 smithi156 ceph-mon[33045]: from='osd.3 [v2:172.21.15.90:6810/2520012383,v1:172.21.15.90:6811/2520012383]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-25T11:29:05.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:04 smithi156 ceph-mon[33045]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T11:29:05.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:04 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T11:29:05.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:04 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T11:29:05.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:04 smithi156 ceph-mon[33045]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T11:29:05.386 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T11:29:06.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:05 smithi138 ceph-mon[32328]: purged_snaps scrub starts 2024-04-25T11:29:06.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:05 smithi138 ceph-mon[32328]: purged_snaps scrub ok 2024-04-25T11:29:06.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:05 smithi138 ceph-mon[32328]: osd.3 [v2:172.21.15.90:6810/2520012383,v1:172.21.15.90:6811/2520012383] boot 2024-04-25T11:29:06.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:05 smithi138 ceph-mon[32328]: osdmap e25: 4 total, 4 up, 4 in 2024-04-25T11:29:06.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:05 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T11:29:06.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:05 smithi138 ceph-mon[32328]: Detected new or changed devices on smithi090 2024-04-25T11:29:06.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:05 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:06.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:05 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:06.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:05 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:29:06.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:05 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:29:06.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:05 smithi138 ceph-mon[32328]: Adjusting osd_memory_target on smithi090 to 8544M 2024-04-25T11:29:06.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:05 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:06.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:05 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:06.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:05 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:29:06.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:05 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:06.056 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:05 smithi138 ceph-mon[32328]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T11:29:06.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:05 smithi090 ceph-mon[29575]: purged_snaps scrub starts 2024-04-25T11:29:06.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:05 smithi090 ceph-mon[29575]: purged_snaps scrub ok 2024-04-25T11:29:06.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:05 smithi090 ceph-mon[29575]: osd.3 [v2:172.21.15.90:6810/2520012383,v1:172.21.15.90:6811/2520012383] boot 2024-04-25T11:29:06.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:05 smithi090 ceph-mon[29575]: osdmap e25: 4 total, 4 up, 4 in 2024-04-25T11:29:06.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T11:29:06.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:05 smithi090 ceph-mon[29575]: Detected new or changed devices on smithi090 2024-04-25T11:29:06.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:06.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:06.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:29:06.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:29:06.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:05 smithi090 ceph-mon[29575]: Adjusting osd_memory_target on smithi090 to 8544M 2024-04-25T11:29:06.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:06.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:06.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:29:06.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:06.229 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:05 smithi090 ceph-mon[29575]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T11:29:06.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:05 smithi156 ceph-mon[33045]: purged_snaps scrub starts 2024-04-25T11:29:06.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:05 smithi156 ceph-mon[33045]: purged_snaps scrub ok 2024-04-25T11:29:06.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:05 smithi156 ceph-mon[33045]: osd.3 [v2:172.21.15.90:6810/2520012383,v1:172.21.15.90:6811/2520012383] boot 2024-04-25T11:29:06.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:05 smithi156 ceph-mon[33045]: osdmap e25: 4 total, 4 up, 4 in 2024-04-25T11:29:06.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T11:29:06.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:05 smithi156 ceph-mon[33045]: Detected new or changed devices on smithi090 2024-04-25T11:29:06.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:06.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:06.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:29:06.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:29:06.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:05 smithi156 ceph-mon[33045]: Adjusting osd_memory_target on smithi090 to 8544M 2024-04-25T11:29:06.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:06.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:06.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:29:06.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:06.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:05 smithi156 ceph-mon[33045]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T11:29:06.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.934+0000 7f369c70e700 1 -- 172.21.15.138:0/3330735138 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3694104230 msgr2=0x7f3694104640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:06.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.934+0000 7f369c70e700 1 --2- 172.21.15.138:0/3330735138 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3694104230 0x7f3694104640 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3684009990 tx=0x7f3684009ca0 comp rx=0 tx=0).stop 2024-04-25T11:29:06.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.934+0000 7f369c70e700 1 -- 172.21.15.138:0/3330735138 shutdown_connections 2024-04-25T11:29:06.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.934+0000 7f369c70e700 1 --2- 172.21.15.138:0/3330735138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3694104bd0 0x7f369410b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:06.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.934+0000 7f369c70e700 1 --2- 172.21.15.138:0/3330735138 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3694103100 0x7f3694103510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:06.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.934+0000 7f369c70e700 1 --2- 172.21.15.138:0/3330735138 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3694104230 0x7f3694104640 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:06.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.934+0000 7f369c70e700 1 -- 172.21.15.138:0/3330735138 >> 172.21.15.138:0/3330735138 conn(0x7f36940fe760 msgr2=0x7f3694100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:29:06.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.934+0000 7f369c70e700 1 -- 172.21.15.138:0/3330735138 shutdown_connections 2024-04-25T11:29:06.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.934+0000 7f369c70e700 1 -- 172.21.15.138:0/3330735138 wait complete. 2024-04-25T11:29:06.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.935+0000 7f369c70e700 1 Processor -- start 2024-04-25T11:29:06.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.935+0000 7f369c70e700 1 -- start start 2024-04-25T11:29:06.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.935+0000 7f369c70e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3694103100 0x7f369419ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:06.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.935+0000 7f369c70e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3694104230 0x7f369419cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:06.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.935+0000 7f369c70e700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3694104bd0 0x7f36941a3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:06.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.935+0000 7f369c70e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f369410eb90 con 0x7f3694103100 2024-04-25T11:29:06.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.935+0000 7f369c70e700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f369410ea10 con 0x7f3694104bd0 2024-04-25T11:29:06.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.935+0000 7f369c70e700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f369410ed10 con 0x7f3694104230 2024-04-25T11:29:06.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.935+0000 7f369acab700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3694104bd0 0x7f36941a3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:06.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.935+0000 7f369acab700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3694104bd0 0x7f36941a3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:47888/0 (socket says 172.21.15.138:47888) 2024-04-25T11:29:06.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.935+0000 7f369acab700 1 -- 172.21.15.138:0/3256302364 learned_addr learned my addr 172.21.15.138:0/3256302364 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-04-25T11:29:06.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.935+0000 7f3699ca9700 1 --2- 172.21.15.138:0/3256302364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3694104230 0x7f369419cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:06.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.935+0000 7f369a4aa700 1 --2- 172.21.15.138:0/3256302364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3694103100 0x7f369419ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:06.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.935+0000 7f369acab700 1 -- 172.21.15.138:0/3256302364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3694104230 msgr2=0x7f369419cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:06.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.935+0000 7f369acab700 1 --2- 172.21.15.138:0/3256302364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3694104230 0x7f369419cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:06.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.935+0000 7f369acab700 1 -- 172.21.15.138:0/3256302364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3694103100 msgr2=0x7f369419ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:06.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.935+0000 7f369acab700 1 --2- 172.21.15.138:0/3256302364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3694103100 0x7f369419ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:06.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.935+0000 7f369acab700 1 -- 172.21.15.138:0/3256302364 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36941a45b0 con 0x7f3694104bd0 2024-04-25T11:29:06.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.935+0000 7f369a4aa700 1 --2- 172.21.15.138:0/3256302364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3694103100 0x7f369419ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:29:06.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.936+0000 7f369acab700 1 --2- 172.21.15.138:0/3256302364 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3694104bd0 0x7f36941a3f30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f368c007ee0 tx=0x7f368c00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:29:06.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.936+0000 7f368b7fe700 1 -- 172.21.15.138:0/3256302364 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f368c00ddd0 con 0x7f3694104bd0 2024-04-25T11:29:06.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.936+0000 7f368b7fe700 1 -- 172.21.15.138:0/3256302364 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f368c004020 con 0x7f3694104bd0 2024-04-25T11:29:06.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.936+0000 7f369c70e700 1 -- 172.21.15.138:0/3256302364 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36941a48a0 con 0x7f3694104bd0 2024-04-25T11:29:06.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.936+0000 7f368b7fe700 1 -- 172.21.15.138:0/3256302364 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f368c015660 con 0x7f3694104bd0 2024-04-25T11:29:06.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.936+0000 7f369c70e700 1 -- 172.21.15.138:0/3256302364 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36941a4d90 con 0x7f3694104bd0 2024-04-25T11:29:06.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.937+0000 7f369c70e700 1 -- 172.21.15.138:0/3256302364 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3694051190 con 0x7f3694104bd0 2024-04-25T11:29:06.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.939+0000 7f368b7fe700 1 -- 172.21.15.138:0/3256302364 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f368c0044f0 con 0x7f3694104bd0 2024-04-25T11:29:06.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.940+0000 7f368b7fe700 1 --2- 172.21.15.138:0/3256302364 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f36800afb80 0x7f36800b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:06.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.940+0000 7f368b7fe700 1 -- 172.21.15.138:0/3256302364 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3669+0+0 (secure 0 0 0) 0x7f368c0ddd30 con 0x7f3694104bd0 2024-04-25T11:29:06.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.940+0000 7f369a4aa700 1 --2- 172.21.15.138:0/3256302364 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f36800afb80 0x7f36800b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:06.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.941+0000 7f369a4aa700 1 --2- 172.21.15.138:0/3256302364 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f36800afb80 0x7f36800b2090 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3694104130 tx=0x7f3690009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:29:06.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:06.943+0000 7f368b7fe700 1 -- 172.21.15.138:0/3256302364 <== mon.2 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f368c010070 con 0x7f3694104bd0 2024-04-25T11:29:07.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:06 smithi138 ceph-mon[32328]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T11:29:07.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:06 smithi138 ceph-mon[32328]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T11:29:07.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:07.151+0000 7f369c70e700 1 -- 172.21.15.138:0/3256302364 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f36941007d0 con 0x7f36800afb80 2024-04-25T11:29:07.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:06 smithi090 ceph-mon[29575]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T11:29:07.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:06 smithi090 ceph-mon[29575]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T11:29:07.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:06 smithi156 ceph-mon[33045]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T11:29:07.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:06 smithi156 ceph-mon[33045]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T11:29:08.014 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:07 smithi138 ceph-mon[32328]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:29:08.014 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:07 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:29:08.015 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:07 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:29:08.015 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:07 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:08.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:07 smithi090 ceph-mon[29575]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:29:08.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:07 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:29:08.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:07 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:29:08.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:07 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:08.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:07 smithi156 ceph-mon[33045]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:29:08.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:07 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:29:08.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:07 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:29:08.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:07 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:09.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:08 smithi156 ceph-mon[33045]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T11:29:09.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:08 smithi090 ceph-mon[29575]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T11:29:09.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:08 smithi138 ceph-mon[32328]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T11:29:10.143 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:09 smithi138 ceph-mon[32328]: from='client.? 172.21.15.138:0/1265450334' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5326212-750c-49fd-96e3-d1e62bb4b7f5"}]: dispatch 2024-04-25T11:29:10.143 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:09 smithi138 ceph-mon[32328]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5326212-750c-49fd-96e3-d1e62bb4b7f5"}]: dispatch 2024-04-25T11:29:10.143 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:09 smithi138 ceph-mon[32328]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5326212-750c-49fd-96e3-d1e62bb4b7f5"}]': finished 2024-04-25T11:29:10.144 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:09 smithi138 ceph-mon[32328]: osdmap e28: 5 total, 4 up, 5 in 2024-04-25T11:29:10.144 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:09 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T11:29:10.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:09 smithi090 ceph-mon[29575]: from='client.? 172.21.15.138:0/1265450334' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5326212-750c-49fd-96e3-d1e62bb4b7f5"}]: dispatch 2024-04-25T11:29:10.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:09 smithi090 ceph-mon[29575]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5326212-750c-49fd-96e3-d1e62bb4b7f5"}]: dispatch 2024-04-25T11:29:10.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:09 smithi090 ceph-mon[29575]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5326212-750c-49fd-96e3-d1e62bb4b7f5"}]': finished 2024-04-25T11:29:10.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:09 smithi090 ceph-mon[29575]: osdmap e28: 5 total, 4 up, 5 in 2024-04-25T11:29:10.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:09 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T11:29:10.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:09 smithi156 ceph-mon[33045]: from='client.? 172.21.15.138:0/1265450334' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5326212-750c-49fd-96e3-d1e62bb4b7f5"}]: dispatch 2024-04-25T11:29:10.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:09 smithi156 ceph-mon[33045]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5326212-750c-49fd-96e3-d1e62bb4b7f5"}]: dispatch 2024-04-25T11:29:10.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:09 smithi156 ceph-mon[33045]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5326212-750c-49fd-96e3-d1e62bb4b7f5"}]': finished 2024-04-25T11:29:10.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:09 smithi156 ceph-mon[33045]: osdmap e28: 5 total, 4 up, 5 in 2024-04-25T11:29:10.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:09 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T11:29:11.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:10 smithi090 ceph-mon[29575]: from='client.? 172.21.15.138:0/2989181212' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:29:11.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:10 smithi090 ceph-mon[29575]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-25T11:29:11.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:10 smithi138 ceph-mon[32328]: from='client.? 172.21.15.138:0/2989181212' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:29:11.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:10 smithi138 ceph-mon[32328]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-25T11:29:11.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:10 smithi156 ceph-mon[33045]: from='client.? 172.21.15.138:0/2989181212' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:29:11.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:10 smithi156 ceph-mon[33045]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-25T11:29:13.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:13 smithi090 ceph-mon[29575]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-25T11:29:13.982 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:13 smithi138 ceph-mon[32328]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-25T11:29:14.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:13 smithi156 ceph-mon[33045]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-25T11:29:16.021 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:15 smithi138 ceph-mon[32328]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T11:29:16.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:15 smithi156 ceph-mon[33045]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T11:29:16.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:15 smithi090 ceph-mon[29575]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T11:29:18.007 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:17 smithi138 ceph-mon[32328]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-25T11:29:18.007 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T11:29:18.007 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:18.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:17 smithi156 ceph-mon[33045]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-25T11:29:18.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T11:29:18.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:18.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:17 smithi090 ceph-mon[29575]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-25T11:29:18.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T11:29:18.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:18.966 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:18 smithi138 ceph-mon[32328]: Deploying daemon osd.4 on smithi138 2024-04-25T11:29:19.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:18 smithi156 ceph-mon[33045]: Deploying daemon osd.4 on smithi138 2024-04-25T11:29:19.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:18 smithi090 ceph-mon[29575]: Deploying daemon osd.4 on smithi138 2024-04-25T11:29:19.917 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:19 smithi138 ceph-mon[32328]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T11:29:20.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:19 smithi156 ceph-mon[33045]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T11:29:20.129 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:19 smithi090 ceph-mon[29575]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T11:29:22.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:21 smithi138 ceph-mon[32328]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T11:29:22.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:21 smithi156 ceph-mon[33045]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T11:29:22.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:21 smithi090 ceph-mon[29575]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T11:29:23.900 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:23 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:23.900 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:23 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:23.900 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:23 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:29:23.900 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:23 smithi138 ceph-mon[32328]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T11:29:23.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:23 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:23.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:23 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:23.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:23 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:29:23.978 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:23 smithi090 ceph-mon[29575]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T11:29:24.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:23 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:24.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:23 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:24.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:23 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:29:24.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:23 smithi156 ceph-mon[33045]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T11:29:25.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:25 smithi138 ceph-mon[32328]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T11:29:26.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:25 smithi156 ceph-mon[33045]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T11:29:26.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:26.165+0000 7f368b7fe700 1 -- 172.21.15.138:0/3256302364 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f36941007d0 con 0x7f36800afb80 2024-04-25T11:29:26.156 INFO:teuthology.orchestra.run.smithi138.stdout:Created osd(s) 4 on host 'smithi138' 2024-04-25T11:29:26.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:26.167+0000 7f369c70e700 1 -- 172.21.15.138:0/3256302364 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f36800afb80 msgr2=0x7f36800b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:26.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:26.167+0000 7f369c70e700 1 --2- 172.21.15.138:0/3256302364 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f36800afb80 0x7f36800b2090 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3694104130 tx=0x7f3690009260 comp rx=0 tx=0).stop 2024-04-25T11:29:26.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:26.167+0000 7f369c70e700 1 -- 172.21.15.138:0/3256302364 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3694104bd0 msgr2=0x7f36941a3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:26.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:26.167+0000 7f369c70e700 1 --2- 172.21.15.138:0/3256302364 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3694104bd0 0x7f36941a3f30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f368c007ee0 tx=0x7f368c00d3b0 comp rx=0 tx=0).stop 2024-04-25T11:29:26.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:26.167+0000 7f369c70e700 1 -- 172.21.15.138:0/3256302364 shutdown_connections 2024-04-25T11:29:26.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:26.167+0000 7f369c70e700 1 --2- 172.21.15.138:0/3256302364 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3694104bd0 0x7f36941a3f30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:26.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:26.167+0000 7f369c70e700 1 --2- 172.21.15.138:0/3256302364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3694103100 0x7f369419ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:26.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:26.167+0000 7f369c70e700 1 --2- 172.21.15.138:0/3256302364 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f36800afb80 0x7f36800b2090 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:26.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:26.167+0000 7f369c70e700 1 --2- 172.21.15.138:0/3256302364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3694104230 0x7f369419cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:26.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:26.167+0000 7f369c70e700 1 -- 172.21.15.138:0/3256302364 >> 172.21.15.138:0/3256302364 conn(0x7f36940fe760 msgr2=0x7f36941000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:29:26.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:26.167+0000 7f369c70e700 1 -- 172.21.15.138:0/3256302364 shutdown_connections 2024-04-25T11:29:26.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:29:26.167+0000 7f369c70e700 1 -- 172.21.15.138:0/3256302364 wait complete. 2024-04-25T11:29:26.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:25 smithi090 ceph-mon[29575]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T11:29:26.773 DEBUG:teuthology.orchestra.run.smithi138:osd.4> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.4.service 2024-04-25T11:29:26.775 INFO:tasks.cephadm:Deploying osd.5 on smithi156 with /dev/vg_nvme/lv_3... 2024-04-25T11:29:26.775 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T11:29:27.021 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T11:29:27.387 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:27 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:27.387 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:27 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:27.388 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:27 smithi156 ceph-mon[33045]: from='osd.4 [v2:172.21.15.138:6808/3752678279,v1:172.21.15.138:6809/3752678279]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T11:29:27.388 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:27 smithi156 ceph-mon[33045]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T11:29:27.388 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:27 smithi156 ceph-mon[33045]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T11:29:27.388 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:27 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:27.388 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:27 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:27.456 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:27 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:27.456 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:27 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:27.456 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:27 smithi138 ceph-mon[32328]: from='osd.4 [v2:172.21.15.138:6808/3752678279,v1:172.21.15.138:6809/3752678279]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T11:29:27.456 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:27 smithi138 ceph-mon[32328]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T11:29:27.456 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:27 smithi138 ceph-mon[32328]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T11:29:27.457 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:27 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:27.457 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:27 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:27.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:27 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:27.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:27 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:27.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:27 smithi090 ceph-mon[29575]: from='osd.4 [v2:172.21.15.138:6808/3752678279,v1:172.21.15.138:6809/3752678279]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T11:29:27.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:27 smithi090 ceph-mon[29575]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T11:29:27.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:27 smithi090 ceph-mon[29575]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T11:29:27.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:27 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:27.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:27 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:28.448 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:28 smithi156 ceph-mon[33045]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T11:29:28.448 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:28 smithi156 ceph-mon[33045]: osdmap e29: 5 total, 4 up, 5 in 2024-04-25T11:29:28.448 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:28 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T11:29:28.448 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:28 smithi156 ceph-mon[33045]: from='osd.4 [v2:172.21.15.138:6808/3752678279,v1:172.21.15.138:6809/3752678279]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-04-25T11:29:28.448 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:28 smithi156 ceph-mon[33045]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-04-25T11:29:28.459 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:28 smithi138 ceph-mon[32328]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T11:29:28.459 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:28 smithi138 ceph-mon[32328]: osdmap e29: 5 total, 4 up, 5 in 2024-04-25T11:29:28.460 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:28 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T11:29:28.460 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:28 smithi138 ceph-mon[32328]: from='osd.4 [v2:172.21.15.138:6808/3752678279,v1:172.21.15.138:6809/3752678279]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-04-25T11:29:28.460 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:28 smithi138 ceph-mon[32328]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-04-25T11:29:28.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:28 smithi090 ceph-mon[29575]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T11:29:28.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:28 smithi090 ceph-mon[29575]: osdmap e29: 5 total, 4 up, 5 in 2024-04-25T11:29:28.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:28 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T11:29:28.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:28 smithi090 ceph-mon[29575]: from='osd.4 [v2:172.21.15.138:6808/3752678279,v1:172.21.15.138:6809/3752678279]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-04-25T11:29:28.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:28 smithi090 ceph-mon[29575]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-04-25T11:29:28.997 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T11:29:28.997 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T11:29:28.997 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2024-04-25T11:29:28.997 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2024-04-25T11:29:28.998 INFO:teuthology.orchestra.run.smithi156.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00873736 s, 1.2 GB/s 2024-04-25T11:29:28.998 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2024-04-25T11:29:29.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:29 smithi138 ceph-mon[32328]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-04-25T11:29:29.305 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:29 smithi138 ceph-mon[32328]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T11:29:29.305 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:29 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T11:29:29.305 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:29 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T11:29:29.305 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:29 smithi138 ceph-mon[32328]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T11:29:29.305 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:29 smithi138 ceph-mon[32328]: from='osd.4 ' entity='osd.4' 2024-04-25T11:29:29.305 INFO:journalctl@ceph.osd.4.smithi138.stdout:Apr 25 11:29:29 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-4[43600]: 2024-04-25T11:29:29.037+0000 7f6902155700 -1 osd.4 0 waiting for initial osdmap 2024-04-25T11:29:29.305 INFO:journalctl@ceph.osd.4.smithi138.stdout:Apr 25 11:29:29 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-4[43600]: 2024-04-25T11:29:29.120+0000 7f68f9f02700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T11:29:29.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:29 smithi090 ceph-mon[29575]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-04-25T11:29:29.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:29 smithi090 ceph-mon[29575]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T11:29:29.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:29 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T11:29:29.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:29 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T11:29:29.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:29 smithi090 ceph-mon[29575]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T11:29:29.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:29 smithi090 ceph-mon[29575]: from='osd.4 ' entity='osd.4' 2024-04-25T11:29:29.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:29 smithi156 ceph-mon[33045]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-04-25T11:29:29.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:29 smithi156 ceph-mon[33045]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T11:29:29.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:29 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T11:29:29.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:29 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T11:29:29.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:29 smithi156 ceph-mon[33045]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T11:29:29.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:29 smithi156 ceph-mon[33045]: from='osd.4 ' entity='osd.4' 2024-04-25T11:29:29.629 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi156:vg_nvme/lv_3 2024-04-25T11:29:29.863 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T11:29:30.305 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:30 smithi156 ceph-mon[33045]: purged_snaps scrub starts 2024-04-25T11:29:30.305 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:30 smithi156 ceph-mon[33045]: purged_snaps scrub ok 2024-04-25T11:29:30.306 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:30 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T11:29:30.306 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:30 smithi156 ceph-mon[33045]: osd.4 [v2:172.21.15.138:6808/3752678279,v1:172.21.15.138:6809/3752678279] boot 2024-04-25T11:29:30.306 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:30 smithi156 ceph-mon[33045]: osdmap e31: 5 total, 5 up, 5 in 2024-04-25T11:29:30.306 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:30 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T11:29:30.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:30 smithi090 ceph-mon[29575]: purged_snaps scrub starts 2024-04-25T11:29:30.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:30 smithi090 ceph-mon[29575]: purged_snaps scrub ok 2024-04-25T11:29:30.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:30 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T11:29:30.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:30 smithi090 ceph-mon[29575]: osd.4 [v2:172.21.15.138:6808/3752678279,v1:172.21.15.138:6809/3752678279] boot 2024-04-25T11:29:30.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:30 smithi090 ceph-mon[29575]: osdmap e31: 5 total, 5 up, 5 in 2024-04-25T11:29:30.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:30 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T11:29:30.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:30 smithi138 ceph-mon[32328]: purged_snaps scrub starts 2024-04-25T11:29:30.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:30 smithi138 ceph-mon[32328]: purged_snaps scrub ok 2024-04-25T11:29:30.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:30 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T11:29:30.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:30 smithi138 ceph-mon[32328]: osd.4 [v2:172.21.15.138:6808/3752678279,v1:172.21.15.138:6809/3752678279] boot 2024-04-25T11:29:30.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:30 smithi138 ceph-mon[32328]: osdmap e31: 5 total, 5 up, 5 in 2024-04-25T11:29:30.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:30 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T11:29:31.387 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:31 smithi156 ceph-mon[33045]: Detected new or changed devices on smithi138 2024-04-25T11:29:31.387 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:31 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:31.388 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:31 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:31.388 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:31 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:29:31.388 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:31 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:29:31.388 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:31 smithi156 ceph-mon[33045]: Adjusting osd_memory_target on smithi138 to 8544M 2024-04-25T11:29:31.388 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:31 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:31.388 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:31 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:31.388 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:31 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:29:31.388 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:31 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:31.388 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:31 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:29:31.388 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:31 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:31.388 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:31 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:29:31.388 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:31 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:31.388 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:31 smithi156 ceph-mon[33045]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:31.446 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.454+0000 7f267b70f700 1 -- 172.21.15.156:0/4019674400 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2674104bf0 msgr2=0x7f267410b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:31.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.454+0000 7f267b70f700 1 --2- 172.21.15.156:0/4019674400 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2674104bf0 0x7f267410b4a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f266c00b0a0 tx=0x7f266c00b3b0 comp rx=0 tx=0).stop 2024-04-25T11:29:31.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.454+0000 7f267b70f700 1 -- 172.21.15.156:0/4019674400 shutdown_connections 2024-04-25T11:29:31.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.454+0000 7f267b70f700 1 --2- 172.21.15.156:0/4019674400 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2674104bf0 0x7f267410b4a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:31.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.454+0000 7f267b70f700 1 --2- 172.21.15.156:0/4019674400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2674103120 0x7f2674103530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:31.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.454+0000 7f267b70f700 1 --2- 172.21.15.156:0/4019674400 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2674104250 0x7f2674104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:31.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.454+0000 7f267b70f700 1 -- 172.21.15.156:0/4019674400 >> 172.21.15.156:0/4019674400 conn(0x7f26740fe760 msgr2=0x7f2674100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:29:31.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.454+0000 7f267b70f700 1 -- 172.21.15.156:0/4019674400 shutdown_connections 2024-04-25T11:29:31.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.454+0000 7f267b70f700 1 -- 172.21.15.156:0/4019674400 wait complete. 2024-04-25T11:29:31.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.454+0000 7f267b70f700 1 Processor -- start 2024-04-25T11:29:31.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.454+0000 7f267b70f700 1 -- start start 2024-04-25T11:29:31.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.455+0000 7f267b70f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2674103120 0x7f2674076340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:31.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.455+0000 7f267b70f700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2674104250 0x7f267419f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:31.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.455+0000 7f267b70f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2674104bf0 0x7f26741a1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:31.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.455+0000 7f267b70f700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f267410eb90 con 0x7f2674104bf0 2024-04-25T11:29:31.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.455+0000 7f267b70f700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f267410ea10 con 0x7f2674104250 2024-04-25T11:29:31.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.455+0000 7f267b70f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f267410ed10 con 0x7f2674103120 2024-04-25T11:29:31.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.455+0000 7f26794ab700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2674103120 0x7f2674076340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:31.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.455+0000 7f26794ab700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2674103120 0x7f2674076340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:50940/0 (socket says 172.21.15.156:50940) 2024-04-25T11:29:31.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.455+0000 7f26794ab700 1 -- 172.21.15.156:0/2332777770 learned_addr learned my addr 172.21.15.156:0/2332777770 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-25T11:29:31.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.455+0000 7f2678caa700 1 --2- 172.21.15.156:0/2332777770 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2674104250 0x7f267419f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:31.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.455+0000 7f2679cac700 1 --2- 172.21.15.156:0/2332777770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2674104bf0 0x7f26741a1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:31.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.455+0000 7f26794ab700 1 -- 172.21.15.156:0/2332777770 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2674104250 msgr2=0x7f267419f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:31.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.455+0000 7f26794ab700 1 --2- 172.21.15.156:0/2332777770 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2674104250 0x7f267419f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:31.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.455+0000 7f26794ab700 1 -- 172.21.15.156:0/2332777770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2674104bf0 msgr2=0x7f26741a1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:31.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.455+0000 7f26794ab700 1 --2- 172.21.15.156:0/2332777770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2674104bf0 0x7f26741a1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:31.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.455+0000 7f26794ab700 1 -- 172.21.15.156:0/2332777770 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26741a22b0 con 0x7f2674103120 2024-04-25T11:29:31.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.455+0000 7f2679cac700 1 --2- 172.21.15.156:0/2332777770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2674104bf0 0x7f26741a1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:29:31.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.455+0000 7f26794ab700 1 --2- 172.21.15.156:0/2332777770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2674103120 0x7f2674076340 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f266400d950 tx=0x7f266400dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:29:31.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.455+0000 7f266a7fc700 1 -- 172.21.15.156:0/2332777770 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2664017070 con 0x7f2674103120 2024-04-25T11:29:31.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.455+0000 7f267b70f700 1 -- 172.21.15.156:0/2332777770 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26741a25a0 con 0x7f2674103120 2024-04-25T11:29:31.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.455+0000 7f266a7fc700 1 -- 172.21.15.156:0/2332777770 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2664009730 con 0x7f2674103120 2024-04-25T11:29:31.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.455+0000 7f266a7fc700 1 -- 172.21.15.156:0/2332777770 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2664003a00 con 0x7f2674103120 2024-04-25T11:29:31.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.455+0000 7f267b70f700 1 -- 172.21.15.156:0/2332777770 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26741a2b30 con 0x7f2674103120 2024-04-25T11:29:31.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.456+0000 7f267b70f700 1 -- 172.21.15.156:0/2332777770 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2674051190 con 0x7f2674103120 2024-04-25T11:29:31.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.458+0000 7f266a7fc700 1 -- 172.21.15.156:0/2332777770 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f2664003ba0 con 0x7f2674103120 2024-04-25T11:29:31.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.458+0000 7f266a7fc700 1 --2- 172.21.15.156:0/2332777770 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f26600afb80 0x7f26600b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:31.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.459+0000 7f266a7fc700 1 -- 172.21.15.156:0/2332777770 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 4101+0+0 (secure 0 0 0) 0x7f26640dde70 con 0x7f2674103120 2024-04-25T11:29:31.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.459+0000 7f2678caa700 1 --2- 172.21.15.156:0/2332777770 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f26600afb80 0x7f26600b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:31.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.459+0000 7f2678caa700 1 --2- 172.21.15.156:0/2332777770 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f26600afb80 0x7f26600b2090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2674075cf0 tx=0x7f2670009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:29:31.453 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.462+0000 7f266a7fc700 1 -- 172.21.15.156:0/2332777770 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2664020030 con 0x7f2674103120 2024-04-25T11:29:31.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:31 smithi090 ceph-mon[29575]: Detected new or changed devices on smithi138 2024-04-25T11:29:31.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:31 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:31.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:31 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:31.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:31 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:29:31.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:31 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:29:31.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:31 smithi090 ceph-mon[29575]: Adjusting osd_memory_target on smithi138 to 8544M 2024-04-25T11:29:31.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:31 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:31.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:31 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:31.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:31 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:29:31.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:31 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:31.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:31 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:29:31.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:31 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:31.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:31 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:29:31.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:31 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:31.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:31 smithi090 ceph-mon[29575]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:31.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:31 smithi138 ceph-mon[32328]: Detected new or changed devices on smithi138 2024-04-25T11:29:31.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:31 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:31.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:31 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:31.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:31 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:29:31.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:31 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:29:31.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:31 smithi138 ceph-mon[32328]: Adjusting osd_memory_target on smithi138 to 8544M 2024-04-25T11:29:31.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:31 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:31.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:31 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:31.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:31 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:29:31.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:31 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:31.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:31 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:29:31.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:31 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:31.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:31 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:29:31.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:31 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:31.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:31 smithi138 ceph-mon[32328]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:31.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:31.668+0000 7f267b70f700 1 -- 172.21.15.156:0/2332777770 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f2674063ad0 con 0x7f26600afb80 2024-04-25T11:29:32.548 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:32 smithi156 ceph-mon[33045]: osdmap e32: 5 total, 5 up, 5 in 2024-04-25T11:29:32.548 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:32 smithi156 ceph-mon[33045]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:29:32.548 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:32 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:29:32.548 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:32 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:29:32.548 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:32 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:32.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:32 smithi090 ceph-mon[29575]: osdmap e32: 5 total, 5 up, 5 in 2024-04-25T11:29:32.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:32 smithi090 ceph-mon[29575]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:29:32.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:32 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:29:32.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:32 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:29:32.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:32 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:32.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:32 smithi138 ceph-mon[32328]: osdmap e32: 5 total, 5 up, 5 in 2024-04-25T11:29:32.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:32 smithi138 ceph-mon[32328]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:29:32.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:32 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:29:32.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:32 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:29:32.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:32 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:33.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:33 smithi156 ceph-mon[33045]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:33.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:33 smithi090 ceph-mon[29575]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:33.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:33 smithi138 ceph-mon[32328]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:34.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:34 smithi090 ceph-mon[29575]: from='client.? 172.21.15.156:0/548646378' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d04a1156-820b-4b43-be89-e16a8910c3aa"}]: dispatch 2024-04-25T11:29:34.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:34 smithi090 ceph-mon[29575]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d04a1156-820b-4b43-be89-e16a8910c3aa"}]: dispatch 2024-04-25T11:29:34.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:34 smithi090 ceph-mon[29575]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d04a1156-820b-4b43-be89-e16a8910c3aa"}]': finished 2024-04-25T11:29:34.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:34 smithi090 ceph-mon[29575]: osdmap e33: 6 total, 5 up, 6 in 2024-04-25T11:29:34.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:34 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T11:29:34.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:34 smithi138 ceph-mon[32328]: from='client.? 172.21.15.156:0/548646378' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d04a1156-820b-4b43-be89-e16a8910c3aa"}]: dispatch 2024-04-25T11:29:34.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:34 smithi138 ceph-mon[32328]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d04a1156-820b-4b43-be89-e16a8910c3aa"}]: dispatch 2024-04-25T11:29:34.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:34 smithi138 ceph-mon[32328]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d04a1156-820b-4b43-be89-e16a8910c3aa"}]': finished 2024-04-25T11:29:34.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:34 smithi138 ceph-mon[32328]: osdmap e33: 6 total, 5 up, 6 in 2024-04-25T11:29:34.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:34 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T11:29:34.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:34 smithi156 ceph-mon[33045]: from='client.? 172.21.15.156:0/548646378' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d04a1156-820b-4b43-be89-e16a8910c3aa"}]: dispatch 2024-04-25T11:29:34.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:34 smithi156 ceph-mon[33045]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d04a1156-820b-4b43-be89-e16a8910c3aa"}]: dispatch 2024-04-25T11:29:34.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:34 smithi156 ceph-mon[33045]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d04a1156-820b-4b43-be89-e16a8910c3aa"}]': finished 2024-04-25T11:29:34.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:34 smithi156 ceph-mon[33045]: osdmap e33: 6 total, 5 up, 6 in 2024-04-25T11:29:34.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:34 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T11:29:35.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:35 smithi090 ceph-mon[29575]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:35.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:35 smithi090 ceph-mon[29575]: from='client.? 172.21.15.156:0/1827343819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:29:35.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:35 smithi138 ceph-mon[32328]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:35.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:35 smithi138 ceph-mon[32328]: from='client.? 172.21.15.156:0/1827343819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:29:35.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:35 smithi156 ceph-mon[33045]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:35.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:35 smithi156 ceph-mon[33045]: from='client.? 172.21.15.156:0/1827343819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:29:37.991 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:37 smithi156 ceph-mon[33045]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:38.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:37 smithi138 ceph-mon[32328]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:38.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:37 smithi090 ceph-mon[29575]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:40.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:39 smithi138 ceph-mon[32328]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:40.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:39 smithi156 ceph-mon[33045]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:40.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:39 smithi090 ceph-mon[29575]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:42.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:41 smithi138 ceph-mon[32328]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:42.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:41 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T11:29:42.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:41 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:42.068 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:41 smithi156 ceph-mon[33045]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:42.068 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:41 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T11:29:42.068 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:41 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:42.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:41 smithi090 ceph-mon[29575]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:42.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:41 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T11:29:42.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:41 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:42.968 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:42 smithi156 ceph-mon[33045]: Deploying daemon osd.5 on smithi156 2024-04-25T11:29:43.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:42 smithi138 ceph-mon[32328]: Deploying daemon osd.5 on smithi156 2024-04-25T11:29:43.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:42 smithi090 ceph-mon[29575]: Deploying daemon osd.5 on smithi156 2024-04-25T11:29:44.009 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:43 smithi156 ceph-mon[33045]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:44.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:43 smithi138 ceph-mon[32328]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:44.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:43 smithi090 ceph-mon[29575]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:45.969 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:45 smithi156 ceph-mon[33045]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:46.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:45 smithi138 ceph-mon[32328]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:46.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:45 smithi090 ceph-mon[29575]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:47.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:47 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:47.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:47 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:47.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:47 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:29:47.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:47 smithi138 ceph-mon[32328]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:47.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:47 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:47.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:47 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:47.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:47 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:29:47.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:47 smithi156 ceph-mon[33045]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:47.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:47 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:47.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:47 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:47.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:47 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:29:47.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:47 smithi090 ceph-mon[29575]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:50.046 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:49 smithi156 ceph-mon[33045]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:50.047 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:49 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:50.047 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:49 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:50.047 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:49 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:50.047 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:49 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:29:50.047 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:49 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:50.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:49 smithi138 ceph-mon[32328]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:50.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:49 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:50.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:49 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:50.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:49 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:50.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:49 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:29:50.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:49 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:50.122 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 5 on host 'smithi156' 2024-04-25T11:29:50.122 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:50.132+0000 7f266a7fc700 1 -- 172.21.15.156:0/2332777770 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2674063ad0 con 0x7f26600afb80 2024-04-25T11:29:50.124 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:50.134+0000 7f267b70f700 1 -- 172.21.15.156:0/2332777770 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f26600afb80 msgr2=0x7f26600b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:50.124 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:50.134+0000 7f267b70f700 1 --2- 172.21.15.156:0/2332777770 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f26600afb80 0x7f26600b2090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2674075cf0 tx=0x7f2670009260 comp rx=0 tx=0).stop 2024-04-25T11:29:50.124 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:50.134+0000 7f267b70f700 1 -- 172.21.15.156:0/2332777770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2674103120 msgr2=0x7f2674076340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:50.124 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:50.134+0000 7f267b70f700 1 --2- 172.21.15.156:0/2332777770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2674103120 0x7f2674076340 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f266400d950 tx=0x7f266400dc60 comp rx=0 tx=0).stop 2024-04-25T11:29:50.125 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:50.135+0000 7f267b70f700 1 -- 172.21.15.156:0/2332777770 shutdown_connections 2024-04-25T11:29:50.125 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:50.135+0000 7f267b70f700 1 --2- 172.21.15.156:0/2332777770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2674104bf0 0x7f26741a1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:50.125 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:50.135+0000 7f267b70f700 1 --2- 172.21.15.156:0/2332777770 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f26600afb80 0x7f26600b2090 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:50.125 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:50.135+0000 7f267b70f700 1 --2- 172.21.15.156:0/2332777770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2674103120 0x7f2674076340 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:50.125 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:50.135+0000 7f267b70f700 1 --2- 172.21.15.156:0/2332777770 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2674104250 0x7f267419f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:50.125 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:50.135+0000 7f267b70f700 1 -- 172.21.15.156:0/2332777770 >> 172.21.15.156:0/2332777770 conn(0x7f26740fe760 msgr2=0x7f2674107480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:29:50.126 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:50.135+0000 7f267b70f700 1 -- 172.21.15.156:0/2332777770 shutdown_connections 2024-04-25T11:29:50.126 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:29:50.135+0000 7f267b70f700 1 -- 172.21.15.156:0/2332777770 wait complete. 2024-04-25T11:29:50.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:49 smithi090 ceph-mon[29575]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:50.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:49 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:50.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:49 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:50.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:49 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:50.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:49 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:29:50.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:49 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:50.683 DEBUG:teuthology.orchestra.run.smithi156:osd.5> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.5.service 2024-04-25T11:29:50.686 INFO:tasks.cephadm:Deploying osd.6 on smithi090 with /dev/vg_nvme/lv_2... 2024-04-25T11:29:50.686 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T11:29:50.931 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:29:51.260 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:51 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:51.260 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:51 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:29:51.260 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:51 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:51.260 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:51 smithi090 ceph-mon[29575]: from='osd.5 [v2:172.21.15.156:6808/3591210191,v1:172.21.15.156:6809/3591210191]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T11:29:51.260 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:51 smithi090 ceph-mon[29575]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T11:29:51.260 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:51 smithi090 ceph-mon[29575]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:51.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:51 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:51.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:51 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:29:51.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:51 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:51.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:51 smithi156 ceph-mon[33045]: from='osd.5 [v2:172.21.15.156:6808/3591210191,v1:172.21.15.156:6809/3591210191]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T11:29:51.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:51 smithi156 ceph-mon[33045]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T11:29:51.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:51 smithi156 ceph-mon[33045]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:51.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:51 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:51.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:51 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:29:51.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:51 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:51.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:51 smithi138 ceph-mon[32328]: from='osd.5 [v2:172.21.15.156:6808/3591210191,v1:172.21.15.156:6809/3591210191]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T11:29:51.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:51 smithi138 ceph-mon[32328]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T11:29:51.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:51 smithi138 ceph-mon[32328]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:52.192 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:52 smithi156 ceph-mon[33045]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T11:29:52.192 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:52 smithi156 ceph-mon[33045]: osdmap e34: 6 total, 5 up, 6 in 2024-04-25T11:29:52.192 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:52 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T11:29:52.192 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:52 smithi156 ceph-mon[33045]: from='osd.5 [v2:172.21.15.156:6808/3591210191,v1:172.21.15.156:6809/3591210191]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T11:29:52.192 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:52 smithi156 ceph-mon[33045]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T11:29:52.193 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:29:52 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:29:52.112+0000 7f5d1e2cc700 -1 osd.5 0 waiting for initial osdmap 2024-04-25T11:29:52.193 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:29:52 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:29:52.120+0000 7f5d1a081700 -1 osd.5 35 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T11:29:52.322 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:52 smithi090 ceph-mon[29575]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T11:29:52.322 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:52 smithi090 ceph-mon[29575]: osdmap e34: 6 total, 5 up, 6 in 2024-04-25T11:29:52.322 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:52 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T11:29:52.322 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:52 smithi090 ceph-mon[29575]: from='osd.5 [v2:172.21.15.156:6808/3591210191,v1:172.21.15.156:6809/3591210191]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T11:29:52.323 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:52 smithi090 ceph-mon[29575]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T11:29:52.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:52 smithi138 ceph-mon[32328]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T11:29:52.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:52 smithi138 ceph-mon[32328]: osdmap e34: 6 total, 5 up, 6 in 2024-04-25T11:29:52.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:52 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T11:29:52.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:52 smithi138 ceph-mon[32328]: from='osd.5 [v2:172.21.15.156:6808/3591210191,v1:172.21.15.156:6809/3591210191]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T11:29:52.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:52 smithi138 ceph-mon[32328]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T11:29:52.881 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T11:29:52.881 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T11:29:52.881 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-04-25T11:29:52.881 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-04-25T11:29:52.882 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115125 s, 911 MB/s 2024-04-25T11:29:52.882 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-04-25T11:29:53.412 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:53 smithi156 ceph-mon[33045]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-25T11:29:53.412 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:53 smithi156 ceph-mon[33045]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T11:29:53.412 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:53 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T11:29:53.412 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:53 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T11:29:53.412 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:53 smithi156 ceph-mon[33045]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:53.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:53 smithi090 ceph-mon[29575]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-25T11:29:53.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:53 smithi090 ceph-mon[29575]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T11:29:53.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:53 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T11:29:53.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:53 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T11:29:53.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:53 smithi090 ceph-mon[29575]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:53.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:53 smithi138 ceph-mon[32328]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-25T11:29:53.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:53 smithi138 ceph-mon[32328]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T11:29:53.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:53 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T11:29:53.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:53 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T11:29:53.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:53 smithi138 ceph-mon[32328]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T11:29:53.565 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi090:vg_nvme/lv_2 2024-04-25T11:29:53.807 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:29:54.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:54 smithi090 ceph-mon[29575]: purged_snaps scrub starts 2024-04-25T11:29:54.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:54 smithi090 ceph-mon[29575]: purged_snaps scrub ok 2024-04-25T11:29:54.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:54 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T11:29:54.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:54 smithi090 ceph-mon[29575]: osd.5 [v2:172.21.15.156:6808/3591210191,v1:172.21.15.156:6809/3591210191] boot 2024-04-25T11:29:54.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:54 smithi090 ceph-mon[29575]: osdmap e36: 6 total, 6 up, 6 in 2024-04-25T11:29:54.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:54 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T11:29:54.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:54 smithi090 ceph-mon[29575]: Detected new or changed devices on smithi156 2024-04-25T11:29:54.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:54 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:54.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:54 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:54.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:54 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:29:54.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:54 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:29:54.480 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:54 smithi090 ceph-mon[29575]: Adjusting osd_memory_target on smithi156 to 8544M 2024-04-25T11:29:54.480 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:54 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:54.480 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:54 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:54.480 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:54 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:29:54.480 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:54 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:54.480 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:54 smithi090 ceph-mon[29575]: osdmap e37: 6 total, 6 up, 6 in 2024-04-25T11:29:54.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:54 smithi138 ceph-mon[32328]: purged_snaps scrub starts 2024-04-25T11:29:54.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:54 smithi138 ceph-mon[32328]: purged_snaps scrub ok 2024-04-25T11:29:54.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:54 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T11:29:54.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:54 smithi138 ceph-mon[32328]: osd.5 [v2:172.21.15.156:6808/3591210191,v1:172.21.15.156:6809/3591210191] boot 2024-04-25T11:29:54.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:54 smithi138 ceph-mon[32328]: osdmap e36: 6 total, 6 up, 6 in 2024-04-25T11:29:54.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:54 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T11:29:54.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:54 smithi138 ceph-mon[32328]: Detected new or changed devices on smithi156 2024-04-25T11:29:54.556 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:54 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:54.556 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:54 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:54.556 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:54 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:29:54.556 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:54 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:29:54.556 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:54 smithi138 ceph-mon[32328]: Adjusting osd_memory_target on smithi156 to 8544M 2024-04-25T11:29:54.557 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:54 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:54.557 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:54 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:54.557 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:54 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:29:54.557 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:54 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:54.557 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:54 smithi138 ceph-mon[32328]: osdmap e37: 6 total, 6 up, 6 in 2024-04-25T11:29:54.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:54 smithi156 ceph-mon[33045]: purged_snaps scrub starts 2024-04-25T11:29:54.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:54 smithi156 ceph-mon[33045]: purged_snaps scrub ok 2024-04-25T11:29:54.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:54 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T11:29:54.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:54 smithi156 ceph-mon[33045]: osd.5 [v2:172.21.15.156:6808/3591210191,v1:172.21.15.156:6809/3591210191] boot 2024-04-25T11:29:54.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:54 smithi156 ceph-mon[33045]: osdmap e36: 6 total, 6 up, 6 in 2024-04-25T11:29:54.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:54 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T11:29:54.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:54 smithi156 ceph-mon[33045]: Detected new or changed devices on smithi156 2024-04-25T11:29:54.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:54 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:54.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:54 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:54.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:54 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:29:54.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:54 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:29:54.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:54 smithi156 ceph-mon[33045]: Adjusting osd_memory_target on smithi156 to 8544M 2024-04-25T11:29:54.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:54 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:54.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:54 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:54.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:54 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:29:54.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:54 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:29:54.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:54 smithi156 ceph-mon[33045]: osdmap e37: 6 total, 6 up, 6 in 2024-04-25T11:29:55.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.288+0000 7fce0adfe700 1 -- 172.21.15.90:0/1086508602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce04104080 msgr2=0x7fce04104550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:55.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.288+0000 7fce0adfe700 1 --2- 172.21.15.90:0/1086508602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce04104080 0x7fce04104550 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0009990 tx=0x7fcdf0009ca0 comp rx=0 tx=0).stop 2024-04-25T11:29:55.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.288+0000 7fce0adfe700 1 -- 172.21.15.90:0/1086508602 shutdown_connections 2024-04-25T11:29:55.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.288+0000 7fce0adfe700 1 --2- 172.21.15.90:0/1086508602 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fce04104be0 0x7fce0410b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:55.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.288+0000 7fce0adfe700 1 --2- 172.21.15.90:0/1086508602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce041036a0 0x7fce04103ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:55.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.288+0000 7fce0adfe700 1 --2- 172.21.15.90:0/1086508602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce04104080 0x7fce04104550 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:55.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.288+0000 7fce0adfe700 1 -- 172.21.15.90:0/1086508602 >> 172.21.15.90:0/1086508602 conn(0x7fce040fdef0 msgr2=0x7fce041003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:29:55.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.288+0000 7fce0adfe700 1 -- 172.21.15.90:0/1086508602 shutdown_connections 2024-04-25T11:29:55.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.288+0000 7fce0adfe700 1 -- 172.21.15.90:0/1086508602 wait complete. 2024-04-25T11:29:55.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.288+0000 7fce0adfe700 1 Processor -- start 2024-04-25T11:29:55.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.288+0000 7fce0adfe700 1 -- start start 2024-04-25T11:29:55.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.289+0000 7fce0adfe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce041036a0 0x7fce04199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:55.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.289+0000 7fce0adfe700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fce04104080 0x7fce04199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:55.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.289+0000 7fce0adfe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce04104be0 0x7fce041a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:55.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.289+0000 7fce0adfe700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fce0410ec90 con 0x7fce04104be0 2024-04-25T11:29:55.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.289+0000 7fce0adfe700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fce0410eb10 con 0x7fce04104080 2024-04-25T11:29:55.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.289+0000 7fce0adfe700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fce0410ee10 con 0x7fce041036a0 2024-04-25T11:29:55.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.289+0000 7fce0939b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce04104be0 0x7fce041a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:55.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.289+0000 7fce0939b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce04104be0 0x7fce041a0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58128/0 (socket says 172.21.15.90:58128) 2024-04-25T11:29:55.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.289+0000 7fce0939b700 1 -- 172.21.15.90:0/3658046445 learned_addr learned my addr 172.21.15.90:0/3658046445 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:29:55.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.289+0000 7fce08b9a700 1 --2- 172.21.15.90:0/3658046445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce041036a0 0x7fce04199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:55.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.289+0000 7fce03fff700 1 --2- 172.21.15.90:0/3658046445 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fce04104080 0x7fce04199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:55.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.289+0000 7fce0939b700 1 -- 172.21.15.90:0/3658046445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce041036a0 msgr2=0x7fce04199830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:55.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.289+0000 7fce0939b700 1 --2- 172.21.15.90:0/3658046445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce041036a0 0x7fce04199830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:55.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.289+0000 7fce0939b700 1 -- 172.21.15.90:0/3658046445 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fce04104080 msgr2=0x7fce04199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:55.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.289+0000 7fce0939b700 1 --2- 172.21.15.90:0/3658046445 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fce04104080 0x7fce04199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:55.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.289+0000 7fce0939b700 1 -- 172.21.15.90:0/3658046445 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce041a1390 con 0x7fce04104be0 2024-04-25T11:29:55.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.289+0000 7fce0939b700 1 --2- 172.21.15.90:0/3658046445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce04104be0 0x7fce041a0d10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc00b970 tx=0x7fcdfc00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:29:55.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.289+0000 7fce01ffb700 1 -- 172.21.15.90:0/3658046445 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc00d860 con 0x7fce04104be0 2024-04-25T11:29:55.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.289+0000 7fce01ffb700 1 -- 172.21.15.90:0/3658046445 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdfc00e430 con 0x7fce04104be0 2024-04-25T11:29:55.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.289+0000 7fce0adfe700 1 -- 172.21.15.90:0/3658046445 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce041a1680 con 0x7fce04104be0 2024-04-25T11:29:55.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.289+0000 7fce01ffb700 1 -- 172.21.15.90:0/3658046445 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc017660 con 0x7fce04104be0 2024-04-25T11:29:55.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.290+0000 7fce0adfe700 1 -- 172.21.15.90:0/3658046445 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce041a1bf0 con 0x7fce04104be0 2024-04-25T11:29:55.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.291+0000 7fce0adfe700 1 -- 172.21.15.90:0/3658046445 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce04051190 con 0x7fce04104be0 2024-04-25T11:29:55.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.292+0000 7fce01ffb700 1 -- 172.21.15.90:0/3658046445 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fcdfc00da00 con 0x7fce04104be0 2024-04-25T11:29:55.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.292+0000 7fce01ffb700 1 --2- 172.21.15.90:0/3658046445 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fcdf40afb80 0x7fcdf40b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:55.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.293+0000 7fce08b9a700 1 --2- 172.21.15.90:0/3658046445 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fcdf40afb80 0x7fcdf40b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:55.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.293+0000 7fce01ffb700 1 -- 172.21.15.90:0/3658046445 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 4533+0+0 (secure 0 0 0) 0x7fcdfc0ddb60 con 0x7fce04104be0 2024-04-25T11:29:55.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.293+0000 7fce08b9a700 1 --2- 172.21.15.90:0/3658046445 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fcdf40afb80 0x7fcdf40b2090 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fce0406bf50 tx=0x7fcdf8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:29:55.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.296+0000 7fce01ffb700 1 -- 172.21.15.90:0/3658046445 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcdfc052ff0 con 0x7fce04104be0 2024-04-25T11:29:55.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:29:55.504+0000 7fce0adfe700 1 -- 172.21.15.90:0/3658046445 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fce04063ad0 con 0x7fcdf40afb80 2024-04-25T11:29:55.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:55 smithi156 ceph-mon[33045]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T11:29:55.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:55 smithi156 ceph-mon[33045]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-25T11:29:55.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:55 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:29:55.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:55 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:29:55.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:55 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:55.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:55 smithi090 ceph-mon[29575]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T11:29:55.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:55 smithi090 ceph-mon[29575]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-25T11:29:55.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:55 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:29:55.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:55 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:29:55.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:55 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:56.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:55 smithi138 ceph-mon[32328]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T11:29:56.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:55 smithi138 ceph-mon[32328]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-25T11:29:56.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:55 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:29:56.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:55 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:29:56.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:55 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:29:56.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:56 smithi090 ceph-mon[29575]: from='client.14376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:29:56.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:56 smithi156 ceph-mon[33045]: from='client.14376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:29:57.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:56 smithi138 ceph-mon[32328]: from='client.14376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:29:57.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:57 smithi156 ceph-mon[33045]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-25T11:29:57.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:57 smithi090 ceph-mon[29575]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-25T11:29:58.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:57 smithi138 ceph-mon[32328]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-25T11:29:58.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:58 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2257600349' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75675aa3-e978-490c-8205-13d9868a3d6d"}]: dispatch 2024-04-25T11:29:58.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:58 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2257600349' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75675aa3-e978-490c-8205-13d9868a3d6d"}]': finished 2024-04-25T11:29:58.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:58 smithi090 ceph-mon[29575]: osdmap e39: 7 total, 6 up, 7 in 2024-04-25T11:29:58.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:58 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T11:29:58.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:58 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2257600349' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75675aa3-e978-490c-8205-13d9868a3d6d"}]: dispatch 2024-04-25T11:29:58.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:58 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2257600349' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75675aa3-e978-490c-8205-13d9868a3d6d"}]': finished 2024-04-25T11:29:58.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:58 smithi156 ceph-mon[33045]: osdmap e39: 7 total, 6 up, 7 in 2024-04-25T11:29:58.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:58 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T11:29:59.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:58 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2257600349' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75675aa3-e978-490c-8205-13d9868a3d6d"}]: dispatch 2024-04-25T11:29:59.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:58 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2257600349' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75675aa3-e978-490c-8205-13d9868a3d6d"}]': finished 2024-04-25T11:29:59.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:58 smithi138 ceph-mon[32328]: osdmap e39: 7 total, 6 up, 7 in 2024-04-25T11:29:59.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:58 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T11:29:59.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:59 smithi156 ceph-mon[33045]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-25T11:29:59.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:29:59 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/327016309' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:29:59.955 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:59 smithi138 ceph-mon[32328]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-25T11:29:59.955 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:29:59 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/327016309' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:29:59.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:59 smithi090 ceph-mon[29575]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-25T11:29:59.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:29:59 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/327016309' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:30:00.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:00 smithi156 ceph-mon[33045]: overall HEALTH_OK 2024-04-25T11:30:00.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:00 smithi090 ceph-mon[29575]: overall HEALTH_OK 2024-04-25T11:30:01.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:00 smithi138 ceph-mon[32328]: overall HEALTH_OK 2024-04-25T11:30:01.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:01 smithi156 ceph-mon[33045]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-25T11:30:01.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:01 smithi090 ceph-mon[29575]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-25T11:30:02.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:01 smithi138 ceph-mon[32328]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-25T11:30:04.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:03 smithi138 ceph-mon[32328]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-25T11:30:04.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:03 smithi156 ceph-mon[33045]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-25T11:30:04.194 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:03 smithi090 ceph-mon[29575]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-25T11:30:06.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:05 smithi138 ceph-mon[32328]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T11:30:06.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:05 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T11:30:06.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:05 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:06.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:05 smithi156 ceph-mon[33045]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T11:30:06.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T11:30:06.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:06.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:05 smithi090 ceph-mon[29575]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T11:30:06.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T11:30:06.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:06.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:06 smithi090 ceph-mon[29575]: Deploying daemon osd.6 on smithi090 2024-04-25T11:30:07.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:06 smithi138 ceph-mon[32328]: Deploying daemon osd.6 on smithi090 2024-04-25T11:30:07.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:06 smithi156 ceph-mon[33045]: Deploying daemon osd.6 on smithi090 2024-04-25T11:30:07.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:07 smithi090 ceph-mon[29575]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T11:30:08.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:07 smithi138 ceph-mon[32328]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T11:30:08.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:07 smithi156 ceph-mon[33045]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T11:30:10.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:09 smithi138 ceph-mon[32328]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-25T11:30:10.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:09 smithi156 ceph-mon[33045]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-25T11:30:10.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:09 smithi090 ceph-mon[29575]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-25T11:30:11.804 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:11 smithi090 ceph-mon[29575]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T11:30:11.805 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:11 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:11.805 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:11 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:11.805 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:11 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:30:12.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:11 smithi138 ceph-mon[32328]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T11:30:12.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:11 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:12.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:11 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:12.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:11 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:30:12.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:11 smithi156 ceph-mon[33045]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T11:30:12.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:11 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:12.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:11 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:12.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:11 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:30:13.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:13 smithi090 ceph-mon[29575]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-25T11:30:14.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:13 smithi138 ceph-mon[32328]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-25T11:30:14.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:13 smithi156 ceph-mon[33045]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-25T11:30:14.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:30:14.927+0000 7fce01ffb700 1 -- 172.21.15.90:0/3658046445 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fce04063ad0 con 0x7fcdf40afb80 2024-04-25T11:30:14.918 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 6 on host 'smithi090' 2024-04-25T11:30:14.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:30:14.929+0000 7fce0adfe700 1 -- 172.21.15.90:0/3658046445 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fcdf40afb80 msgr2=0x7fcdf40b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:14.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:30:14.929+0000 7fce0adfe700 1 --2- 172.21.15.90:0/3658046445 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fcdf40afb80 0x7fcdf40b2090 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fce0406bf50 tx=0x7fcdf8009190 comp rx=0 tx=0).stop 2024-04-25T11:30:14.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:30:14.929+0000 7fce0adfe700 1 -- 172.21.15.90:0/3658046445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce04104be0 msgr2=0x7fce041a0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:14.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:30:14.929+0000 7fce0adfe700 1 --2- 172.21.15.90:0/3658046445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce04104be0 0x7fce041a0d10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc00b970 tx=0x7fcdfc00bc80 comp rx=0 tx=0).stop 2024-04-25T11:30:14.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:30:14.929+0000 7fce0adfe700 1 -- 172.21.15.90:0/3658046445 shutdown_connections 2024-04-25T11:30:14.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:30:14.929+0000 7fce0adfe700 1 --2- 172.21.15.90:0/3658046445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce04104be0 0x7fce041a0d10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:14.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:30:14.929+0000 7fce0adfe700 1 --2- 172.21.15.90:0/3658046445 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fcdf40afb80 0x7fcdf40b2090 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:14.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:30:14.929+0000 7fce0adfe700 1 --2- 172.21.15.90:0/3658046445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce041036a0 0x7fce04199830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:14.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:30:14.929+0000 7fce0adfe700 1 --2- 172.21.15.90:0/3658046445 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fce04104080 0x7fce04199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:14.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:30:14.929+0000 7fce0adfe700 1 -- 172.21.15.90:0/3658046445 >> 172.21.15.90:0/3658046445 conn(0x7fce040fdef0 msgr2=0x7fce04107650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:14.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:30:14.929+0000 7fce0adfe700 1 -- 172.21.15.90:0/3658046445 shutdown_connections 2024-04-25T11:30:14.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:30:14.929+0000 7fce0adfe700 1 -- 172.21.15.90:0/3658046445 wait complete. 2024-04-25T11:30:15.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:15 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:15.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:15 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:15.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:15 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:15.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:15 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:30:15.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:15 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:15.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:15 smithi090 ceph-mon[29575]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-25T11:30:15.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:15 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:15.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:15 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:15.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:15 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:30:15.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:15 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:15.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:15 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:15.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:15 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:15.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:15 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:30:15.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:15 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:15.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:15 smithi138 ceph-mon[32328]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-25T11:30:15.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:15 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:15.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:15 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:15.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:15 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:30:15.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:15 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:15.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:15 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:15.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:15 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:15.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:15 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:30:15.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:15 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:15.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:15 smithi156 ceph-mon[33045]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-25T11:30:15.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:15 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:15.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:15 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:15.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:15 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:30:15.629 DEBUG:teuthology.orchestra.run.smithi090:osd.6> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.6.service 2024-04-25T11:30:15.632 INFO:tasks.cephadm:Deploying osd.7 on smithi138 with /dev/vg_nvme/lv_2... 2024-04-25T11:30:15.632 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T11:30:15.877 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T11:30:16.106 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:16 smithi090 ceph-mon[29575]: from='osd.6 [v2:172.21.15.90:6818/2364955456,v1:172.21.15.90:6819/2364955456]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T11:30:16.106 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:16 smithi090 ceph-mon[29575]: from='osd.6 [v2:172.21.15.90:6818/2364955456,v1:172.21.15.90:6819/2364955456]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T11:30:16.106 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:16 smithi090 ceph-mon[29575]: osdmap e40: 7 total, 6 up, 7 in 2024-04-25T11:30:16.106 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:16 smithi090 ceph-mon[29575]: from='osd.6 [v2:172.21.15.90:6818/2364955456,v1:172.21.15.90:6819/2364955456]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-25T11:30:16.106 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:16 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T11:30:16.288 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:16 smithi138 ceph-mon[32328]: from='osd.6 [v2:172.21.15.90:6818/2364955456,v1:172.21.15.90:6819/2364955456]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T11:30:16.288 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:16 smithi138 ceph-mon[32328]: from='osd.6 [v2:172.21.15.90:6818/2364955456,v1:172.21.15.90:6819/2364955456]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T11:30:16.288 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:16 smithi138 ceph-mon[32328]: osdmap e40: 7 total, 6 up, 7 in 2024-04-25T11:30:16.288 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:16 smithi138 ceph-mon[32328]: from='osd.6 [v2:172.21.15.90:6818/2364955456,v1:172.21.15.90:6819/2364955456]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-25T11:30:16.288 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T11:30:16.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:16 smithi156 ceph-mon[33045]: from='osd.6 [v2:172.21.15.90:6818/2364955456,v1:172.21.15.90:6819/2364955456]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T11:30:16.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:16 smithi156 ceph-mon[33045]: from='osd.6 [v2:172.21.15.90:6818/2364955456,v1:172.21.15.90:6819/2364955456]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T11:30:16.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:16 smithi156 ceph-mon[33045]: osdmap e40: 7 total, 6 up, 7 in 2024-04-25T11:30:16.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:16 smithi156 ceph-mon[33045]: from='osd.6 [v2:172.21.15.90:6818/2364955456,v1:172.21.15.90:6819/2364955456]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-25T11:30:16.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:16 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T11:30:17.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:17 smithi138 ceph-mon[32328]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-25T11:30:17.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:17 smithi138 ceph-mon[32328]: from='osd.6 [v2:172.21.15.90:6818/2364955456,v1:172.21.15.90:6819/2364955456]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-25T11:30:17.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:17 smithi138 ceph-mon[32328]: osdmap e41: 7 total, 6 up, 7 in 2024-04-25T11:30:17.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T11:30:17.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T11:30:17.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:17 smithi090 ceph-mon[29575]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-25T11:30:17.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:17 smithi090 ceph-mon[29575]: from='osd.6 [v2:172.21.15.90:6818/2364955456,v1:172.21.15.90:6819/2364955456]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-25T11:30:17.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:17 smithi090 ceph-mon[29575]: osdmap e41: 7 total, 6 up, 7 in 2024-04-25T11:30:17.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T11:30:17.478 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T11:30:17.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:17 smithi156 ceph-mon[33045]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-25T11:30:17.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:17 smithi156 ceph-mon[33045]: from='osd.6 [v2:172.21.15.90:6818/2364955456,v1:172.21.15.90:6819/2364955456]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-25T11:30:17.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:17 smithi156 ceph-mon[33045]: osdmap e41: 7 total, 6 up, 7 in 2024-04-25T11:30:17.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T11:30:17.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T11:30:17.914 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T11:30:17.914 INFO:teuthology.orchestra.run.smithi138.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T11:30:17.914 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10+0 records in 2024-04-25T11:30:17.914 INFO:teuthology.orchestra.run.smithi138.stderr:10+0 records out 2024-04-25T11:30:17.914 INFO:teuthology.orchestra.run.smithi138.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111352 s, 942 MB/s 2024-04-25T11:30:17.914 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 2024-04-25T11:30:17.914 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping successful for: 2024-04-25T11:30:18.227 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:30:17 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:30:17.863+0000 7f4b741da700 -1 osd.6 0 waiting for initial osdmap 2024-04-25T11:30:18.227 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:30:17 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:30:17.963+0000 7f4b6bf87700 -1 osd.6 41 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T11:30:18.565 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi138:vg_nvme/lv_2 2024-04-25T11:30:18.809 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T11:30:19.165 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:18 smithi138 ceph-mon[32328]: purged_snaps scrub starts 2024-04-25T11:30:19.165 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:18 smithi138 ceph-mon[32328]: purged_snaps scrub ok 2024-04-25T11:30:19.165 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:18 smithi138 ceph-mon[32328]: from='osd.6 [v2:172.21.15.90:6818/2364955456,v1:172.21.15.90:6819/2364955456]' entity='osd.6' 2024-04-25T11:30:19.165 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:18 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T11:30:19.165 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:18 smithi138 ceph-mon[32328]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-25T11:30:19.165 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:18 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:19.165 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:18 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:19.166 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:18 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:30:19.166 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:18 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:30:19.166 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:18 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:30:19.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:18 smithi090 ceph-mon[29575]: purged_snaps scrub starts 2024-04-25T11:30:19.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:18 smithi090 ceph-mon[29575]: purged_snaps scrub ok 2024-04-25T11:30:19.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:18 smithi090 ceph-mon[29575]: from='osd.6 [v2:172.21.15.90:6818/2364955456,v1:172.21.15.90:6819/2364955456]' entity='osd.6' 2024-04-25T11:30:19.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:18 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T11:30:19.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:18 smithi090 ceph-mon[29575]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-25T11:30:19.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:18 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:19.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:18 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:19.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:18 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:30:19.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:18 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:30:19.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:18 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:30:19.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:18 smithi156 ceph-mon[33045]: purged_snaps scrub starts 2024-04-25T11:30:19.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:18 smithi156 ceph-mon[33045]: purged_snaps scrub ok 2024-04-25T11:30:19.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:18 smithi156 ceph-mon[33045]: from='osd.6 [v2:172.21.15.90:6818/2364955456,v1:172.21.15.90:6819/2364955456]' entity='osd.6' 2024-04-25T11:30:19.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:18 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T11:30:19.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:18 smithi156 ceph-mon[33045]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-25T11:30:19.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:18 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:19.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:18 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:19.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:18 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:30:19.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:18 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:30:19.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:18 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:30:20.129 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:19 smithi090 ceph-mon[29575]: Detected new or changed devices on smithi090 2024-04-25T11:30:20.129 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:19 smithi090 ceph-mon[29575]: Adjusting osd_memory_target on smithi090 to 5696M 2024-04-25T11:30:20.129 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:19 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:20.129 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:19 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:20.129 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:19 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:30:20.129 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:19 smithi090 ceph-mon[29575]: osd.6 [v2:172.21.15.90:6818/2364955456,v1:172.21.15.90:6819/2364955456] boot 2024-04-25T11:30:20.129 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:19 smithi090 ceph-mon[29575]: osdmap e42: 7 total, 7 up, 7 in 2024-04-25T11:30:20.129 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:19 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T11:30:20.129 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:19 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:20.162 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:19 smithi138 ceph-mon[32328]: Detected new or changed devices on smithi090 2024-04-25T11:30:20.162 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:19 smithi138 ceph-mon[32328]: Adjusting osd_memory_target on smithi090 to 5696M 2024-04-25T11:30:20.162 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:19 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:20.163 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:19 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:20.163 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:19 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:30:20.163 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:19 smithi138 ceph-mon[32328]: osd.6 [v2:172.21.15.90:6818/2364955456,v1:172.21.15.90:6819/2364955456] boot 2024-04-25T11:30:20.163 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:19 smithi138 ceph-mon[32328]: osdmap e42: 7 total, 7 up, 7 in 2024-04-25T11:30:20.163 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:19 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T11:30:20.163 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:19 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:20.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:19 smithi156 ceph-mon[33045]: Detected new or changed devices on smithi090 2024-04-25T11:30:20.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:19 smithi156 ceph-mon[33045]: Adjusting osd_memory_target on smithi090 to 5696M 2024-04-25T11:30:20.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:19 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:20.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:19 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:20.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:19 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:30:20.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:19 smithi156 ceph-mon[33045]: osd.6 [v2:172.21.15.90:6818/2364955456,v1:172.21.15.90:6819/2364955456] boot 2024-04-25T11:30:20.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:19 smithi156 ceph-mon[33045]: osdmap e42: 7 total, 7 up, 7 in 2024-04-25T11:30:20.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:19 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T11:30:20.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:19 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:20.400 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.410+0000 7f7fe8a86700 1 -- 172.21.15.138:0/2639640146 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7fe4102b70 msgr2=0x7f7fe410c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:20.401 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.410+0000 7f7fe8a86700 1 --2- 172.21.15.138:0/2639640146 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7fe4102b70 0x7f7fe410c790 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc00b0a0 tx=0x7f7fdc00b3b0 comp rx=0 tx=0).stop 2024-04-25T11:30:20.401 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.410+0000 7f7fe8a86700 1 -- 172.21.15.138:0/2639640146 shutdown_connections 2024-04-25T11:30:20.401 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.410+0000 7f7fe8a86700 1 --2- 172.21.15.138:0/2639640146 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7fe4102b70 0x7f7fe410c790 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:20.401 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.410+0000 7f7fe8a86700 1 --2- 172.21.15.138:0/2639640146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fe41057d0 0x7f7fe4105be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:20.401 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.410+0000 7f7fe8a86700 1 --2- 172.21.15.138:0/2639640146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fe4102220 0x7f7fe4102630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:20.401 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.410+0000 7f7fe8a86700 1 -- 172.21.15.138:0/2639640146 >> 172.21.15.138:0/2639640146 conn(0x7f7fe40fdef0 msgr2=0x7f7fe41003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:20.401 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.410+0000 7f7fe8a86700 1 -- 172.21.15.138:0/2639640146 shutdown_connections 2024-04-25T11:30:20.401 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.410+0000 7f7fe8a86700 1 -- 172.21.15.138:0/2639640146 wait complete. 2024-04-25T11:30:20.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.411+0000 7f7fe8a86700 1 Processor -- start 2024-04-25T11:30:20.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.411+0000 7f7fe8a86700 1 -- start start 2024-04-25T11:30:20.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.411+0000 7f7fe8a86700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fe4102220 0x7f7fe4104da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:20.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.411+0000 7f7fe8a86700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7fe4102b70 0x7f7fe4103450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:20.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.411+0000 7f7fe8a86700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fe41057d0 0x7f7fe4103990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:20.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.411+0000 7f7fe8a86700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe410fd10 con 0x7f7fe41057d0 2024-04-25T11:30:20.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.411+0000 7f7fe8a86700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe410fb90 con 0x7f7fe4102b70 2024-04-25T11:30:20.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.411+0000 7f7fe8a86700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe410fe90 con 0x7f7fe4102220 2024-04-25T11:30:20.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.411+0000 7f7fe1d9b700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7fe4102b70 0x7f7fe4103450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:20.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.411+0000 7f7fe1d9b700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7fe4102b70 0x7f7fe4103450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:54920/0 (socket says 172.21.15.138:54920) 2024-04-25T11:30:20.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.411+0000 7f7fe1d9b700 1 -- 172.21.15.138:0/4144666385 learned_addr learned my addr 172.21.15.138:0/4144666385 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-04-25T11:30:20.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.411+0000 7f7fe259c700 1 --2- 172.21.15.138:0/4144666385 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fe4102220 0x7f7fe4104da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:20.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.411+0000 7f7fe2d9d700 1 --2- 172.21.15.138:0/4144666385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fe41057d0 0x7f7fe4103990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:20.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.412+0000 7f7fe1d9b700 1 -- 172.21.15.138:0/4144666385 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fe4102220 msgr2=0x7f7fe4104da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:20.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.412+0000 7f7fe1d9b700 1 --2- 172.21.15.138:0/4144666385 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fe4102220 0x7f7fe4104da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:20.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.412+0000 7f7fe1d9b700 1 -- 172.21.15.138:0/4144666385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fe41057d0 msgr2=0x7f7fe4103990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:20.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.412+0000 7f7fe1d9b700 1 --2- 172.21.15.138:0/4144666385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fe41057d0 0x7f7fe4103990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:20.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.412+0000 7f7fe1d9b700 1 -- 172.21.15.138:0/4144666385 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe41040f0 con 0x7f7fe4102b70 2024-04-25T11:30:20.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.412+0000 7f7fe259c700 1 --2- 172.21.15.138:0/4144666385 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fe4102220 0x7f7fe4104da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:30:20.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.412+0000 7f7fe2d9d700 1 --2- 172.21.15.138:0/4144666385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fe41057d0 0x7f7fe4103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:30:20.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.412+0000 7f7fe1d9b700 1 --2- 172.21.15.138:0/4144666385 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7fe4102b70 0x7f7fe4103450 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7fd800b530 tx=0x7f7fd800b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:20.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.412+0000 7f7fd37fe700 1 -- 172.21.15.138:0/4144666385 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd8015020 con 0x7f7fe4102b70 2024-04-25T11:30:20.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.412+0000 7f7fe8a86700 1 -- 172.21.15.138:0/4144666385 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fe41a48c0 con 0x7f7fe4102b70 2024-04-25T11:30:20.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.412+0000 7f7fd37fe700 1 -- 172.21.15.138:0/4144666385 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fd8011ce0 con 0x7f7fe4102b70 2024-04-25T11:30:20.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.412+0000 7f7fd37fe700 1 -- 172.21.15.138:0/4144666385 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd8010d20 con 0x7f7fe4102b70 2024-04-25T11:30:20.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.412+0000 7f7fe8a86700 1 -- 172.21.15.138:0/4144666385 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fe41a4e70 con 0x7f7fe4102b70 2024-04-25T11:30:20.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.413+0000 7f7fe8a86700 1 -- 172.21.15.138:0/4144666385 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fe4051190 con 0x7f7fe4102b70 2024-04-25T11:30:20.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.415+0000 7f7fd37fe700 1 -- 172.21.15.138:0/4144666385 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f7fd8018020 con 0x7f7fe4102b70 2024-04-25T11:30:20.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.415+0000 7f7fd37fe700 1 --2- 172.21.15.138:0/4144666385 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f7fcc0afc00 0x7f7fcc0b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:20.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.415+0000 7f7fd37fe700 1 -- 172.21.15.138:0/4144666385 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 4965+0+0 (secure 0 0 0) 0x7f7fd80ddc10 con 0x7f7fe4102b70 2024-04-25T11:30:20.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.415+0000 7f7fe259c700 1 --2- 172.21.15.138:0/4144666385 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f7fcc0afc00 0x7f7fcc0b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:20.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.416+0000 7f7fe259c700 1 --2- 172.21.15.138:0/4144666385 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f7fcc0afc00 0x7f7fcc0b2110 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7fe406bf50 tx=0x7f7fd4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:20.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.419+0000 7f7fd37fe700 1 -- 172.21.15.138:0/4144666385 <== mon.2 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7fd801f020 con 0x7f7fe4102b70 2024-04-25T11:30:20.615 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:20.624+0000 7f7fe8a86700 1 -- 172.21.15.138:0/4144666385 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f7fe4063ad0 con 0x7f7fcc0afc00 2024-04-25T11:30:21.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:20 smithi090 ceph-mon[29575]: osdmap e43: 7 total, 7 up, 7 in 2024-04-25T11:30:21.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:20 smithi090 ceph-mon[29575]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:30:21.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:20 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:30:21.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:20 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:30:21.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:20 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:21.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:20 smithi090 ceph-mon[29575]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:21.228 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:20 smithi138 ceph-mon[32328]: osdmap e43: 7 total, 7 up, 7 in 2024-04-25T11:30:21.228 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:20 smithi138 ceph-mon[32328]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:30:21.229 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:20 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:30:21.229 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:20 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:30:21.229 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:20 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:21.229 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:20 smithi138 ceph-mon[32328]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:21.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:20 smithi156 ceph-mon[33045]: osdmap e43: 7 total, 7 up, 7 in 2024-04-25T11:30:21.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:20 smithi156 ceph-mon[33045]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:30:21.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:20 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:30:21.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:20 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:30:21.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:20 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:21.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:20 smithi156 ceph-mon[33045]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:24.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:23 smithi138 ceph-mon[32328]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:24.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:23 smithi138 ceph-mon[32328]: from='client.? 172.21.15.138:0/856113831' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a95fdf5-f548-4271-bbcd-e330f7955eaa"}]: dispatch 2024-04-25T11:30:24.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:23 smithi138 ceph-mon[32328]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a95fdf5-f548-4271-bbcd-e330f7955eaa"}]: dispatch 2024-04-25T11:30:24.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:23 smithi138 ceph-mon[32328]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a95fdf5-f548-4271-bbcd-e330f7955eaa"}]': finished 2024-04-25T11:30:24.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:23 smithi138 ceph-mon[32328]: osdmap e44: 8 total, 7 up, 8 in 2024-04-25T11:30:24.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:23 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T11:30:24.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:23 smithi156 ceph-mon[33045]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:24.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:23 smithi156 ceph-mon[33045]: from='client.? 172.21.15.138:0/856113831' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a95fdf5-f548-4271-bbcd-e330f7955eaa"}]: dispatch 2024-04-25T11:30:24.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:23 smithi156 ceph-mon[33045]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a95fdf5-f548-4271-bbcd-e330f7955eaa"}]: dispatch 2024-04-25T11:30:24.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:23 smithi156 ceph-mon[33045]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a95fdf5-f548-4271-bbcd-e330f7955eaa"}]': finished 2024-04-25T11:30:24.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:23 smithi156 ceph-mon[33045]: osdmap e44: 8 total, 7 up, 8 in 2024-04-25T11:30:24.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:23 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T11:30:24.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:23 smithi090 ceph-mon[29575]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:24.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:23 smithi090 ceph-mon[29575]: from='client.? 172.21.15.138:0/856113831' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a95fdf5-f548-4271-bbcd-e330f7955eaa"}]: dispatch 2024-04-25T11:30:24.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:23 smithi090 ceph-mon[29575]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a95fdf5-f548-4271-bbcd-e330f7955eaa"}]: dispatch 2024-04-25T11:30:24.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:23 smithi090 ceph-mon[29575]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a95fdf5-f548-4271-bbcd-e330f7955eaa"}]': finished 2024-04-25T11:30:24.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:23 smithi090 ceph-mon[29575]: osdmap e44: 8 total, 7 up, 8 in 2024-04-25T11:30:24.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:23 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T11:30:25.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:24 smithi138 ceph-mon[32328]: from='client.? 172.21.15.138:0/1016581737' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:30:25.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:24 smithi156 ceph-mon[33045]: from='client.? 172.21.15.138:0/1016581737' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:30:25.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:24 smithi090 ceph-mon[29575]: from='client.? 172.21.15.138:0/1016581737' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:30:26.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:25 smithi138 ceph-mon[32328]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:26.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:25 smithi156 ceph-mon[33045]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:26.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:25 smithi090 ceph-mon[29575]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:27.884 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:27 smithi138 ceph-mon[32328]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:28.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:27 smithi156 ceph-mon[33045]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:28.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:27 smithi090 ceph-mon[29575]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:29.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:29 smithi138 ceph-mon[32328]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:30.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:29 smithi156 ceph-mon[33045]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:30.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:29 smithi090 ceph-mon[29575]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:32.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:31 smithi138 ceph-mon[32328]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:32.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:31 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T11:30:32.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:31 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:32.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:31 smithi138 ceph-mon[32328]: Deploying daemon osd.7 on smithi138 2024-04-25T11:30:32.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:31 smithi156 ceph-mon[33045]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:32.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:31 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T11:30:32.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:31 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:32.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:31 smithi156 ceph-mon[33045]: Deploying daemon osd.7 on smithi138 2024-04-25T11:30:32.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:31 smithi090 ceph-mon[29575]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:32.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:31 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T11:30:32.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:31 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:32.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:31 smithi090 ceph-mon[29575]: Deploying daemon osd.7 on smithi138 2024-04-25T11:30:33.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:33 smithi138 ceph-mon[32328]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:34.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:33 smithi156 ceph-mon[33045]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:34.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:33 smithi090 ceph-mon[29575]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:36.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:35 smithi138 ceph-mon[32328]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:36.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:35 smithi156 ceph-mon[33045]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:36.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:35 smithi090 ceph-mon[29575]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:37.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:37 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:37.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:37 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:37.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:37 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:30:37.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:37 smithi138 ceph-mon[32328]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:37.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:37 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:37.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:37 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:37.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:37 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:30:37.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:37 smithi156 ceph-mon[33045]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:37.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:37 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:37.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:37 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:37.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:37 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:30:37.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:37 smithi090 ceph-mon[29575]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:39.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:39.540+0000 7f7fd37fe700 1 -- 172.21.15.138:0/4144666385 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7fe4063ad0 con 0x7f7fcc0afc00 2024-04-25T11:30:39.531 INFO:teuthology.orchestra.run.smithi138.stdout:Created osd(s) 7 on host 'smithi138' 2024-04-25T11:30:39.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:39.542+0000 7f7fe8a86700 1 -- 172.21.15.138:0/4144666385 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f7fcc0afc00 msgr2=0x7f7fcc0b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:39.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:39.542+0000 7f7fe8a86700 1 --2- 172.21.15.138:0/4144666385 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f7fcc0afc00 0x7f7fcc0b2110 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7fe406bf50 tx=0x7f7fd4005f90 comp rx=0 tx=0).stop 2024-04-25T11:30:39.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:39.542+0000 7f7fe8a86700 1 -- 172.21.15.138:0/4144666385 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7fe4102b70 msgr2=0x7f7fe4103450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:39.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:39.542+0000 7f7fe8a86700 1 --2- 172.21.15.138:0/4144666385 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7fe4102b70 0x7f7fe4103450 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7fd800b530 tx=0x7f7fd800b8f0 comp rx=0 tx=0).stop 2024-04-25T11:30:39.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:39.542+0000 7f7fe8a86700 1 -- 172.21.15.138:0/4144666385 shutdown_connections 2024-04-25T11:30:39.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:39.542+0000 7f7fe8a86700 1 --2- 172.21.15.138:0/4144666385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fe41057d0 0x7f7fe4103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:39.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:39.542+0000 7f7fe8a86700 1 --2- 172.21.15.138:0/4144666385 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f7fcc0afc00 0x7f7fcc0b2110 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:39.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:39.542+0000 7f7fe8a86700 1 --2- 172.21.15.138:0/4144666385 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fe4102220 0x7f7fe4104da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:39.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:39.542+0000 7f7fe8a86700 1 --2- 172.21.15.138:0/4144666385 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7fe4102b70 0x7f7fe4103450 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:39.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:39.542+0000 7f7fe8a86700 1 -- 172.21.15.138:0/4144666385 >> 172.21.15.138:0/4144666385 conn(0x7f7fe40fdef0 msgr2=0x7f7fe4108770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:39.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:39.542+0000 7f7fe8a86700 1 -- 172.21.15.138:0/4144666385 shutdown_connections 2024-04-25T11:30:39.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:30:39.542+0000 7f7fe8a86700 1 -- 172.21.15.138:0/4144666385 wait complete. 2024-04-25T11:30:40.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:39 smithi138 ceph-mon[32328]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:40.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:39 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:40.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:39 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:40.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:39 smithi156 ceph-mon[33045]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:40.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:39 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:40.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:39 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:40.202 DEBUG:teuthology.orchestra.run.smithi138:osd.7> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.7.service 2024-04-25T11:30:40.204 INFO:tasks.cephadm:Deploying osd.8 on smithi156 with /dev/vg_nvme/lv_2... 2024-04-25T11:30:40.204 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T11:30:40.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:39 smithi090 ceph-mon[29575]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:40.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:39 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:40.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:39 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:40.450 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T11:30:40.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:40 smithi156 ceph-mon[33045]: from='osd.7 [v2:172.21.15.138:6816/4012476387,v1:172.21.15.138:6817/4012476387]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T11:30:40.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:40 smithi156 ceph-mon[33045]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T11:30:40.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:40 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:40.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:40 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:41.016 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:40 smithi138 ceph-mon[32328]: from='osd.7 [v2:172.21.15.138:6816/4012476387,v1:172.21.15.138:6817/4012476387]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T11:30:41.016 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:40 smithi138 ceph-mon[32328]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T11:30:41.017 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:40 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:41.017 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:40 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:41.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:40 smithi090 ceph-mon[29575]: from='osd.7 [v2:172.21.15.138:6816/4012476387,v1:172.21.15.138:6817/4012476387]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T11:30:41.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:40 smithi090 ceph-mon[29575]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T11:30:41.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:40 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:41.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:40 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:42.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:41 smithi138 ceph-mon[32328]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:42.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:41 smithi138 ceph-mon[32328]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T11:30:42.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:41 smithi138 ceph-mon[32328]: from='osd.7 [v2:172.21.15.138:6816/4012476387,v1:172.21.15.138:6817/4012476387]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-04-25T11:30:42.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:41 smithi138 ceph-mon[32328]: osdmap e45: 8 total, 7 up, 8 in 2024-04-25T11:30:42.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:41 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T11:30:42.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:41 smithi138 ceph-mon[32328]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-04-25T11:30:42.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:41 smithi138 ceph-mon[32328]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-04-25T11:30:42.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:41 smithi156 ceph-mon[33045]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:42.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:41 smithi156 ceph-mon[33045]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T11:30:42.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:41 smithi156 ceph-mon[33045]: from='osd.7 [v2:172.21.15.138:6816/4012476387,v1:172.21.15.138:6817/4012476387]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-04-25T11:30:42.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:41 smithi156 ceph-mon[33045]: osdmap e45: 8 total, 7 up, 8 in 2024-04-25T11:30:42.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:41 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T11:30:42.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:41 smithi156 ceph-mon[33045]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-04-25T11:30:42.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:41 smithi156 ceph-mon[33045]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-04-25T11:30:42.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:41 smithi090 ceph-mon[29575]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:42.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:41 smithi090 ceph-mon[29575]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T11:30:42.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:41 smithi090 ceph-mon[29575]: from='osd.7 [v2:172.21.15.138:6816/4012476387,v1:172.21.15.138:6817/4012476387]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-04-25T11:30:42.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:41 smithi090 ceph-mon[29575]: osdmap e45: 8 total, 7 up, 8 in 2024-04-25T11:30:42.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:41 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T11:30:42.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:41 smithi090 ceph-mon[29575]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-04-25T11:30:42.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:41 smithi090 ceph-mon[29575]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-04-25T11:30:42.337 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T11:30:42.337 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T11:30:42.337 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2024-04-25T11:30:42.338 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2024-04-25T11:30:42.338 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00878365 s, 1.2 GB/s 2024-04-25T11:30:42.338 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2024-04-25T11:30:42.945 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:42 smithi156 ceph-mon[33045]: osdmap e46: 8 total, 7 up, 8 in 2024-04-25T11:30:42.945 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:42 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T11:30:42.945 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:42 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T11:30:42.946 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:42 smithi156 ceph-mon[33045]: from='osd.7 ' entity='osd.7' 2024-04-25T11:30:42.946 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:42 smithi156 ceph-mon[33045]: osdmap e47: 8 total, 7 up, 8 in 2024-04-25T11:30:42.973 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi156:vg_nvme/lv_2 2024-04-25T11:30:43.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:42 smithi138 ceph-mon[32328]: osdmap e46: 8 total, 7 up, 8 in 2024-04-25T11:30:43.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:42 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T11:30:43.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:42 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T11:30:43.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:42 smithi138 ceph-mon[32328]: from='osd.7 ' entity='osd.7' 2024-04-25T11:30:43.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:42 smithi138 ceph-mon[32328]: osdmap e47: 8 total, 7 up, 8 in 2024-04-25T11:30:43.054 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:30:42 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:30:42.657+0000 7f69a7988700 -1 osd.7 0 waiting for initial osdmap 2024-04-25T11:30:43.054 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:30:42 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:30:42.818+0000 7f699ef34700 -1 osd.7 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T11:30:43.217 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T11:30:43.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:42 smithi090 ceph-mon[29575]: osdmap e46: 8 total, 7 up, 8 in 2024-04-25T11:30:43.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:42 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T11:30:43.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:42 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T11:30:43.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:42 smithi090 ceph-mon[29575]: from='osd.7 ' entity='osd.7' 2024-04-25T11:30:43.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:42 smithi090 ceph-mon[29575]: osdmap e47: 8 total, 7 up, 8 in 2024-04-25T11:30:43.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:43 smithi156 ceph-mon[33045]: purged_snaps scrub starts 2024-04-25T11:30:43.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:43 smithi156 ceph-mon[33045]: purged_snaps scrub ok 2024-04-25T11:30:43.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:43 smithi156 ceph-mon[33045]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:43.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:43 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T11:30:43.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:43 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:43.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:43 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:43.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:43 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:30:43.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:43 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:30:43.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:43 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:30:43.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:43 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:43.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:43 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:43.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:43 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:30:43.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:43 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:43.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:43 smithi156 ceph-mon[33045]: osd.7 [v2:172.21.15.138:6816/4012476387,v1:172.21.15.138:6817/4012476387] boot 2024-04-25T11:30:43.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:43 smithi156 ceph-mon[33045]: osdmap e48: 8 total, 8 up, 8 in 2024-04-25T11:30:43.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:43 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T11:30:44.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:43 smithi138 ceph-mon[32328]: purged_snaps scrub starts 2024-04-25T11:30:44.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:43 smithi138 ceph-mon[32328]: purged_snaps scrub ok 2024-04-25T11:30:44.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:43 smithi138 ceph-mon[32328]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:44.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:43 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T11:30:44.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:43 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:44.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:43 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:44.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:43 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:30:44.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:43 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:30:44.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:43 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:30:44.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:43 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:44.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:43 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:44.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:43 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:30:44.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:43 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:44.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:43 smithi138 ceph-mon[32328]: osd.7 [v2:172.21.15.138:6816/4012476387,v1:172.21.15.138:6817/4012476387] boot 2024-04-25T11:30:44.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:43 smithi138 ceph-mon[32328]: osdmap e48: 8 total, 8 up, 8 in 2024-04-25T11:30:44.055 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:43 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T11:30:44.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:43 smithi090 ceph-mon[29575]: purged_snaps scrub starts 2024-04-25T11:30:44.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:43 smithi090 ceph-mon[29575]: purged_snaps scrub ok 2024-04-25T11:30:44.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:43 smithi090 ceph-mon[29575]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T11:30:44.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:43 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T11:30:44.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:43 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:44.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:43 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:44.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:43 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:30:44.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:43 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:30:44.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:43 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:30:44.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:43 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:44.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:43 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:44.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:43 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:30:44.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:43 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:44.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:43 smithi090 ceph-mon[29575]: osd.7 [v2:172.21.15.138:6816/4012476387,v1:172.21.15.138:6817/4012476387] boot 2024-04-25T11:30:44.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:43 smithi090 ceph-mon[29575]: osdmap e48: 8 total, 8 up, 8 in 2024-04-25T11:30:44.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:43 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T11:30:44.839 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.849+0000 7efcebc16700 1 -- 172.21.15.156:0/1208685704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efce4104380 msgr2=0x7efce410ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:44.839 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.849+0000 7efcebc16700 1 --2- 172.21.15.156:0/1208685704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efce4104380 0x7efce410ac30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efcdc00c210 tx=0x7efcdc00c520 comp rx=0 tx=0).stop 2024-04-25T11:30:44.839 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.850+0000 7efcebc16700 1 -- 172.21.15.156:0/1208685704 shutdown_connections 2024-04-25T11:30:44.840 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.850+0000 7efcebc16700 1 --2- 172.21.15.156:0/1208685704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efce4104380 0x7efce410ac30 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:44.840 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.850+0000 7efcebc16700 1 --2- 172.21.15.156:0/1208685704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efce41028b0 0x7efce4102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:44.840 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.850+0000 7efcebc16700 1 --2- 172.21.15.156:0/1208685704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efce41039e0 0x7efce4103df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:44.840 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.850+0000 7efcebc16700 1 -- 172.21.15.156:0/1208685704 >> 172.21.15.156:0/1208685704 conn(0x7efce40fdef0 msgr2=0x7efce41003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:44.840 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.850+0000 7efcebc16700 1 -- 172.21.15.156:0/1208685704 shutdown_connections 2024-04-25T11:30:44.840 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.850+0000 7efcebc16700 1 -- 172.21.15.156:0/1208685704 wait complete. 2024-04-25T11:30:44.842 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.851+0000 7efcebc16700 1 Processor -- start 2024-04-25T11:30:44.842 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.851+0000 7efcebc16700 1 -- start start 2024-04-25T11:30:44.842 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.851+0000 7efcebc16700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efce41028b0 0x7efce41984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:44.842 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.851+0000 7efcebc16700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efce41039e0 0x7efce4198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:44.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.851+0000 7efcebc16700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efce4104380 0x7efce419f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:44.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.851+0000 7efcebc16700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7efce410da90 con 0x7efce4104380 2024-04-25T11:30:44.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.851+0000 7efcebc16700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7efce410d910 con 0x7efce41028b0 2024-04-25T11:30:44.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.851+0000 7efcebc16700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7efce410dc10 con 0x7efce41039e0 2024-04-25T11:30:44.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.851+0000 7efce91b1700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efce41039e0 0x7efce4198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:44.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.851+0000 7efce91b1700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efce41039e0 0x7efce4198a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:53454/0 (socket says 172.21.15.156:53454) 2024-04-25T11:30:44.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.851+0000 7efce91b1700 1 -- 172.21.15.156:0/2775064102 learned_addr learned my addr 172.21.15.156:0/2775064102 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-25T11:30:44.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.851+0000 7efce99b2700 1 --2- 172.21.15.156:0/2775064102 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efce41028b0 0x7efce41984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:44.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.851+0000 7efcea1b3700 1 --2- 172.21.15.156:0/2775064102 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efce4104380 0x7efce419f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:44.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.851+0000 7efce91b1700 1 -- 172.21.15.156:0/2775064102 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efce41028b0 msgr2=0x7efce41984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:44.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.851+0000 7efce91b1700 1 --2- 172.21.15.156:0/2775064102 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efce41028b0 0x7efce41984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:44.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.851+0000 7efce91b1700 1 -- 172.21.15.156:0/2775064102 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efce4104380 msgr2=0x7efce419f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:44.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.851+0000 7efce91b1700 1 --2- 172.21.15.156:0/2775064102 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efce4104380 0x7efce419f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:44.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.851+0000 7efce91b1700 1 -- 172.21.15.156:0/2775064102 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efce419ff90 con 0x7efce41039e0 2024-04-25T11:30:44.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.852+0000 7efce91b1700 1 --2- 172.21.15.156:0/2775064102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efce41039e0 0x7efce4198a00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efcd400c9b0 tx=0x7efcd400ccc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:44.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.852+0000 7efcdaffd700 1 -- 172.21.15.156:0/2775064102 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd4007b00 con 0x7efce41039e0 2024-04-25T11:30:44.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.852+0000 7efcdaffd700 1 -- 172.21.15.156:0/2775064102 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcd400ce50 con 0x7efce41039e0 2024-04-25T11:30:44.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.852+0000 7efcebc16700 1 -- 172.21.15.156:0/2775064102 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efce41a0280 con 0x7efce41039e0 2024-04-25T11:30:44.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.852+0000 7efcdaffd700 1 -- 172.21.15.156:0/2775064102 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd4018660 con 0x7efce41039e0 2024-04-25T11:30:44.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.852+0000 7efcebc16700 1 -- 172.21.15.156:0/2775064102 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efce41a07c0 con 0x7efce41039e0 2024-04-25T11:30:44.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.853+0000 7efcebc16700 1 -- 172.21.15.156:0/2775064102 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efce4051190 con 0x7efce41039e0 2024-04-25T11:30:44.845 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.854+0000 7efcdaffd700 1 -- 172.21.15.156:0/2775064102 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7efcd400f450 con 0x7efce41039e0 2024-04-25T11:30:44.845 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.855+0000 7efcdaffd700 1 --2- 172.21.15.156:0/2775064102 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7efcd00afb80 0x7efcd00b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:44.845 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.855+0000 7efcdaffd700 1 -- 172.21.15.156:0/2775064102 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5397+0+0 (secure 0 0 0) 0x7efcd40de270 con 0x7efce41039e0 2024-04-25T11:30:44.845 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.855+0000 7efce99b2700 1 --2- 172.21.15.156:0/2775064102 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7efcd00afb80 0x7efcd00b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:44.846 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.856+0000 7efce99b2700 1 --2- 172.21.15.156:0/2775064102 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7efcd00afb80 0x7efcd00b2090 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efce41038e0 tx=0x7efce000a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:44.848 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:44.859+0000 7efcdaffd700 1 -- 172.21.15.156:0/2775064102 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efcd4014030 con 0x7efce41039e0 2024-04-25T11:30:45.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:44 smithi138 ceph-mon[32328]: Detected new or changed devices on smithi138 2024-04-25T11:30:45.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:44 smithi138 ceph-mon[32328]: Adjusting osd_memory_target on smithi138 to 5696M 2024-04-25T11:30:45.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:44 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:30:45.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:44 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:45.057 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:44 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:30:45.057 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:44 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:45.057 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:44 smithi138 ceph-mon[32328]: osdmap e49: 8 total, 8 up, 8 in 2024-04-25T11:30:45.058 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:30:45.067+0000 7efcebc16700 1 -- 172.21.15.156:0/2775064102 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7efce4063ad0 con 0x7efcd00afb80 2024-04-25T11:30:45.058 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:44 smithi156 ceph-mon[33045]: Detected new or changed devices on smithi138 2024-04-25T11:30:45.058 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:44 smithi156 ceph-mon[33045]: Adjusting osd_memory_target on smithi138 to 5696M 2024-04-25T11:30:45.058 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:44 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:30:45.058 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:44 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:45.058 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:44 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:30:45.059 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:44 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:45.059 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:44 smithi156 ceph-mon[33045]: osdmap e49: 8 total, 8 up, 8 in 2024-04-25T11:30:45.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:44 smithi090 ceph-mon[29575]: Detected new or changed devices on smithi138 2024-04-25T11:30:45.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:44 smithi090 ceph-mon[29575]: Adjusting osd_memory_target on smithi138 to 5696M 2024-04-25T11:30:45.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:44 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:30:45.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:44 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:45.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:44 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:30:45.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:44 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:30:45.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:44 smithi090 ceph-mon[29575]: osdmap e49: 8 total, 8 up, 8 in 2024-04-25T11:30:45.932 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:45 smithi156 ceph-mon[33045]: pgmap v172: 1 pgs: 1 remapped; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-25T11:30:45.932 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:45 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:30:45.932 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:45 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:30:45.933 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:45 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:46.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:45 smithi138 ceph-mon[32328]: pgmap v172: 1 pgs: 1 remapped; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-25T11:30:46.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:45 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:30:46.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:45 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:30:46.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:45 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:46.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:45 smithi090 ceph-mon[29575]: pgmap v172: 1 pgs: 1 remapped; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-25T11:30:46.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:45 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:30:46.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:45 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:30:46.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:45 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:46.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:46 smithi156 ceph-mon[33045]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:30:46.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:46 smithi156 ceph-mon[33045]: osdmap e50: 8 total, 8 up, 8 in 2024-04-25T11:30:47.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:46 smithi138 ceph-mon[32328]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:30:47.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:46 smithi138 ceph-mon[32328]: osdmap e50: 8 total, 8 up, 8 in 2024-04-25T11:30:47.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:46 smithi090 ceph-mon[29575]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:30:47.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:46 smithi090 ceph-mon[29575]: osdmap e50: 8 total, 8 up, 8 in 2024-04-25T11:30:48.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:47 smithi138 ceph-mon[32328]: pgmap v175: 1 pgs: 1 remapped; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-25T11:30:48.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:47 smithi138 ceph-mon[32328]: from='client.? 172.21.15.156:0/249656392' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "605e10b5-5806-4c1f-8e28-a7994d34ddce"}]: dispatch 2024-04-25T11:30:48.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:47 smithi138 ceph-mon[32328]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "605e10b5-5806-4c1f-8e28-a7994d34ddce"}]: dispatch 2024-04-25T11:30:48.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:47 smithi138 ceph-mon[32328]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "605e10b5-5806-4c1f-8e28-a7994d34ddce"}]': finished 2024-04-25T11:30:48.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:47 smithi138 ceph-mon[32328]: osdmap e51: 9 total, 8 up, 9 in 2024-04-25T11:30:48.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:47 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T11:30:48.083 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:47 smithi156 ceph-mon[33045]: pgmap v175: 1 pgs: 1 remapped; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-25T11:30:48.083 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:47 smithi156 ceph-mon[33045]: from='client.? 172.21.15.156:0/249656392' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "605e10b5-5806-4c1f-8e28-a7994d34ddce"}]: dispatch 2024-04-25T11:30:48.084 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:47 smithi156 ceph-mon[33045]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "605e10b5-5806-4c1f-8e28-a7994d34ddce"}]: dispatch 2024-04-25T11:30:48.084 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:47 smithi156 ceph-mon[33045]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "605e10b5-5806-4c1f-8e28-a7994d34ddce"}]': finished 2024-04-25T11:30:48.084 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:47 smithi156 ceph-mon[33045]: osdmap e51: 9 total, 8 up, 9 in 2024-04-25T11:30:48.084 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:47 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T11:30:48.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:47 smithi090 ceph-mon[29575]: pgmap v175: 1 pgs: 1 remapped; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-25T11:30:48.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:47 smithi090 ceph-mon[29575]: from='client.? 172.21.15.156:0/249656392' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "605e10b5-5806-4c1f-8e28-a7994d34ddce"}]: dispatch 2024-04-25T11:30:48.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:47 smithi090 ceph-mon[29575]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "605e10b5-5806-4c1f-8e28-a7994d34ddce"}]: dispatch 2024-04-25T11:30:48.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:47 smithi090 ceph-mon[29575]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "605e10b5-5806-4c1f-8e28-a7994d34ddce"}]': finished 2024-04-25T11:30:48.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:47 smithi090 ceph-mon[29575]: osdmap e51: 9 total, 8 up, 9 in 2024-04-25T11:30:48.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:47 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T11:30:49.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:48 smithi138 ceph-mon[32328]: from='client.? 172.21.15.156:0/3272654722' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:30:49.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:48 smithi156 ceph-mon[33045]: from='client.? 172.21.15.156:0/3272654722' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:30:49.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:48 smithi090 ceph-mon[29575]: from='client.? 172.21.15.156:0/3272654722' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:30:50.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:49 smithi138 ceph-mon[32328]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-25T11:30:50.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:49 smithi156 ceph-mon[33045]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-25T11:30:50.128 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:49 smithi090 ceph-mon[29575]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-25T11:30:52.028 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:51 smithi156 ceph-mon[33045]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 192 KiB/s, 0 objects/s recovering 2024-04-25T11:30:52.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:51 smithi090 ceph-mon[29575]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 192 KiB/s, 0 objects/s recovering 2024-04-25T11:30:52.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:51 smithi138 ceph-mon[32328]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 192 KiB/s, 0 objects/s recovering 2024-04-25T11:30:53.933 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:53 smithi156 ceph-mon[33045]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 145 KiB/s, 0 objects/s recovering 2024-04-25T11:30:54.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:53 smithi090 ceph-mon[29575]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 145 KiB/s, 0 objects/s recovering 2024-04-25T11:30:54.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:53 smithi138 ceph-mon[32328]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 145 KiB/s, 0 objects/s recovering 2024-04-25T11:30:56.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:55 smithi156 ceph-mon[33045]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-25T11:30:56.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:55 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T11:30:56.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:55 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:56.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:55 smithi090 ceph-mon[29575]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-25T11:30:56.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:55 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T11:30:56.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:55 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:56.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:55 smithi138 ceph-mon[32328]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-25T11:30:56.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:55 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T11:30:56.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:55 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:30:57.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:56 smithi156 ceph-mon[33045]: Deploying daemon osd.8 on smithi156 2024-04-25T11:30:57.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:56 smithi090 ceph-mon[29575]: Deploying daemon osd.8 on smithi156 2024-04-25T11:30:57.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:56 smithi138 ceph-mon[32328]: Deploying daemon osd.8 on smithi156 2024-04-25T11:30:58.080 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:57 smithi156 ceph-mon[33045]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-25T11:30:58.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:57 smithi090 ceph-mon[29575]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-25T11:30:58.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:57 smithi138 ceph-mon[32328]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-25T11:31:00.060 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:30:59 smithi156 ceph-mon[33045]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-25T11:31:00.105 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:30:59 smithi138 ceph-mon[32328]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-25T11:31:00.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:30:59 smithi090 ceph-mon[29575]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-25T11:31:01.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:01 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:01.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:01 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:01.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:01 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:31:01.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:01 smithi156 ceph-mon[33045]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T11:31:01.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:01 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:01.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:01 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:01.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:01 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:31:01.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:01 smithi090 ceph-mon[29575]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T11:31:01.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:01 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:01.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:01 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:01.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:01 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:31:01.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:01 smithi138 ceph-mon[32328]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T11:31:03.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:03 smithi156 ceph-mon[33045]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-25T11:31:03.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:03 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:03.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:03 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:03.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:03 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:03.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:03 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:31:03.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:03 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:04.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:03 smithi138 ceph-mon[32328]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-25T11:31:04.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:03 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:04.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:03 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:04.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:03 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:04.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:03 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:31:04.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:03 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:04.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:03 smithi090 ceph-mon[29575]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-25T11:31:04.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:03 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:04.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:03 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:04.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:03 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:04.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:03 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:31:04.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:03 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:04.272 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:31:04.283+0000 7efcdaffd700 1 -- 172.21.15.156:0/2775064102 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7efce4063ad0 con 0x7efcd00afb80 2024-04-25T11:31:04.273 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 8 on host 'smithi156' 2024-04-25T11:31:04.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:31:04.285+0000 7efcebc16700 1 -- 172.21.15.156:0/2775064102 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7efcd00afb80 msgr2=0x7efcd00b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:04.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:31:04.285+0000 7efcebc16700 1 --2- 172.21.15.156:0/2775064102 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7efcd00afb80 0x7efcd00b2090 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efce41038e0 tx=0x7efce000a3b0 comp rx=0 tx=0).stop 2024-04-25T11:31:04.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:31:04.285+0000 7efcebc16700 1 -- 172.21.15.156:0/2775064102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efce41039e0 msgr2=0x7efce4198a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:04.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:31:04.285+0000 7efcebc16700 1 --2- 172.21.15.156:0/2775064102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efce41039e0 0x7efce4198a00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efcd400c9b0 tx=0x7efcd400ccc0 comp rx=0 tx=0).stop 2024-04-25T11:31:04.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:31:04.285+0000 7efcebc16700 1 -- 172.21.15.156:0/2775064102 shutdown_connections 2024-04-25T11:31:04.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:31:04.285+0000 7efcebc16700 1 --2- 172.21.15.156:0/2775064102 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efce4104380 0x7efce419f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:04.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:31:04.285+0000 7efcebc16700 1 --2- 172.21.15.156:0/2775064102 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efce41028b0 0x7efce41984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:04.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:31:04.285+0000 7efcebc16700 1 --2- 172.21.15.156:0/2775064102 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7efcd00afb80 0x7efcd00b2090 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:04.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:31:04.285+0000 7efcebc16700 1 --2- 172.21.15.156:0/2775064102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efce41039e0 0x7efce4198a00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:04.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:31:04.285+0000 7efcebc16700 1 -- 172.21.15.156:0/2775064102 >> 172.21.15.156:0/2775064102 conn(0x7efce40fdef0 msgr2=0x7efce4106c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:31:04.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:31:04.285+0000 7efcebc16700 1 -- 172.21.15.156:0/2775064102 shutdown_connections 2024-04-25T11:31:04.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:31:04.285+0000 7efcebc16700 1 -- 172.21.15.156:0/2775064102 wait complete. 2024-04-25T11:31:04.900 DEBUG:teuthology.orchestra.run.smithi156:osd.8> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.8.service 2024-04-25T11:31:04.906 INFO:tasks.cephadm:Deploying osd.9 on smithi090 with /dev/vg_nvme/lv_1... 2024-04-25T11:31:04.906 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T11:31:05.153 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:31:05.494 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:05.495 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:05.495 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:31:05.495 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:05 smithi090 ceph-mon[29575]: from='osd.8 [v2:172.21.15.156:6816/101092224,v1:172.21.15.156:6817/101092224]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T11:31:05.495 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:05 smithi090 ceph-mon[29575]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T11:31:05.495 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:05 smithi090 ceph-mon[29575]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-25T11:31:05.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:05 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:05.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:05 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:05.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:05 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:31:05.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:05 smithi138 ceph-mon[32328]: from='osd.8 [v2:172.21.15.156:6816/101092224,v1:172.21.15.156:6817/101092224]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T11:31:05.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:05 smithi138 ceph-mon[32328]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T11:31:05.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:05 smithi138 ceph-mon[32328]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-25T11:31:05.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:05.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:05.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:31:05.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:05 smithi156 ceph-mon[33045]: from='osd.8 [v2:172.21.15.156:6816/101092224,v1:172.21.15.156:6817/101092224]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T11:31:05.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:05 smithi156 ceph-mon[33045]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T11:31:05.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:05 smithi156 ceph-mon[33045]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-25T11:31:06.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:06 smithi138 ceph-mon[32328]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T11:31:06.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:06 smithi138 ceph-mon[32328]: osdmap e52: 9 total, 8 up, 9 in 2024-04-25T11:31:06.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:06 smithi138 ceph-mon[32328]: from='osd.8 [v2:172.21.15.156:6816/101092224,v1:172.21.15.156:6817/101092224]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T11:31:06.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:06 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T11:31:06.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:06 smithi138 ceph-mon[32328]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T11:31:06.554 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:06 smithi090 ceph-mon[29575]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T11:31:06.554 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:06 smithi090 ceph-mon[29575]: osdmap e52: 9 total, 8 up, 9 in 2024-04-25T11:31:06.555 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:06 smithi090 ceph-mon[29575]: from='osd.8 [v2:172.21.15.156:6816/101092224,v1:172.21.15.156:6817/101092224]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T11:31:06.555 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:06 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T11:31:06.555 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:06 smithi090 ceph-mon[29575]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T11:31:06.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:06 smithi156 ceph-mon[33045]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T11:31:06.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:06 smithi156 ceph-mon[33045]: osdmap e52: 9 total, 8 up, 9 in 2024-04-25T11:31:06.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:06 smithi156 ceph-mon[33045]: from='osd.8 [v2:172.21.15.156:6816/101092224,v1:172.21.15.156:6817/101092224]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T11:31:06.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:06 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T11:31:06.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:06 smithi156 ceph-mon[33045]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T11:31:06.589 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:31:06 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:31:06.275+0000 7fe6c809d700 -1 osd.8 0 waiting for initial osdmap 2024-04-25T11:31:06.589 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:31:06 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:31:06.287+0000 7fe6c3e52700 -1 osd.8 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T11:31:07.123 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T11:31:07.123 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T11:31:07.123 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-04-25T11:31:07.123 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-04-25T11:31:07.123 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114758 s, 914 MB/s 2024-04-25T11:31:07.123 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-04-25T11:31:07.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:07 smithi156 ceph-mon[33045]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-25T11:31:07.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:07 smithi156 ceph-mon[33045]: osdmap e53: 9 total, 8 up, 9 in 2024-04-25T11:31:07.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:07 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T11:31:07.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:07 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T11:31:07.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:07 smithi156 ceph-mon[33045]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-25T11:31:07.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:07 smithi138 ceph-mon[32328]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-25T11:31:07.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:07 smithi138 ceph-mon[32328]: osdmap e53: 9 total, 8 up, 9 in 2024-04-25T11:31:07.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:07 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T11:31:07.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:07 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T11:31:07.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:07 smithi138 ceph-mon[32328]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-25T11:31:07.591 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:07 smithi090 ceph-mon[29575]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-25T11:31:07.592 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:07 smithi090 ceph-mon[29575]: osdmap e53: 9 total, 8 up, 9 in 2024-04-25T11:31:07.592 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:07 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T11:31:07.592 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:07 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T11:31:07.592 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:07 smithi090 ceph-mon[29575]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 715 GiB / 715 GiB avail 2024-04-25T11:31:07.783 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi090:vg_nvme/lv_1 2024-04-25T11:31:08.033 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:31:08.411 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:08 smithi090 ceph-mon[29575]: purged_snaps scrub starts 2024-04-25T11:31:08.411 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:08 smithi090 ceph-mon[29575]: purged_snaps scrub ok 2024-04-25T11:31:08.411 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:08 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T11:31:08.411 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:08 smithi090 ceph-mon[29575]: osd.8 [v2:172.21.15.156:6816/101092224,v1:172.21.15.156:6817/101092224] boot 2024-04-25T11:31:08.411 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:08 smithi090 ceph-mon[29575]: osdmap e54: 9 total, 9 up, 9 in 2024-04-25T11:31:08.412 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:08 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T11:31:08.412 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:08 smithi090 ceph-mon[29575]: Detected new or changed devices on smithi156 2024-04-25T11:31:08.412 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:08 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:08.412 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:08 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:08.412 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:08 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:08.412 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:08 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:08.412 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:08 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:08.412 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:08 smithi090 ceph-mon[29575]: Adjusting osd_memory_target on smithi156 to 5696M 2024-04-25T11:31:08.412 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:08 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:08.412 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:08 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:08.412 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:08 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:31:08.412 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:08 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:08.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:08 smithi138 ceph-mon[32328]: purged_snaps scrub starts 2024-04-25T11:31:08.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:08 smithi138 ceph-mon[32328]: purged_snaps scrub ok 2024-04-25T11:31:08.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:08 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T11:31:08.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:08 smithi138 ceph-mon[32328]: osd.8 [v2:172.21.15.156:6816/101092224,v1:172.21.15.156:6817/101092224] boot 2024-04-25T11:31:08.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:08 smithi138 ceph-mon[32328]: osdmap e54: 9 total, 9 up, 9 in 2024-04-25T11:31:08.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:08 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T11:31:08.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:08 smithi138 ceph-mon[32328]: Detected new or changed devices on smithi156 2024-04-25T11:31:08.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:08 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:08.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:08 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:08.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:08 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:08.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:08 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:08.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:08 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:08.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:08 smithi138 ceph-mon[32328]: Adjusting osd_memory_target on smithi156 to 5696M 2024-04-25T11:31:08.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:08 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:08.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:08 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:08.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:08 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:31:08.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:08 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:08.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:08 smithi156 ceph-mon[33045]: purged_snaps scrub starts 2024-04-25T11:31:08.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:08 smithi156 ceph-mon[33045]: purged_snaps scrub ok 2024-04-25T11:31:08.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:08 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T11:31:08.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:08 smithi156 ceph-mon[33045]: osd.8 [v2:172.21.15.156:6816/101092224,v1:172.21.15.156:6817/101092224] boot 2024-04-25T11:31:08.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:08 smithi156 ceph-mon[33045]: osdmap e54: 9 total, 9 up, 9 in 2024-04-25T11:31:08.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:08 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T11:31:08.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:08 smithi156 ceph-mon[33045]: Detected new or changed devices on smithi156 2024-04-25T11:31:08.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:08 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:08.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:08 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:08.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:08 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:08.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:08 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:08.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:08 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:08.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:08 smithi156 ceph-mon[33045]: Adjusting osd_memory_target on smithi156 to 5696M 2024-04-25T11:31:08.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:08 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:08.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:08 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:08.592 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:08 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:31:08.592 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:08 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:09.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:09 smithi090 ceph-mon[29575]: osdmap e55: 9 total, 9 up, 9 in 2024-04-25T11:31:09.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:09 smithi090 ceph-mon[29575]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-25T11:31:09.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:09 smithi138 ceph-mon[32328]: osdmap e55: 9 total, 9 up, 9 in 2024-04-25T11:31:09.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:09 smithi138 ceph-mon[32328]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-25T11:31:09.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:09 smithi156 ceph-mon[33045]: osdmap e55: 9 total, 9 up, 9 in 2024-04-25T11:31:09.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:09 smithi156 ceph-mon[33045]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-25T11:31:09.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.725+0000 7f002fc62700 1 -- 172.21.15.90:0/1871641759 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0028109fd0 msgr2=0x7f002810c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:09.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.725+0000 7f002fc62700 1 --2- 172.21.15.90:0/1871641759 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0028109fd0 0x7f002810c3f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0020009990 tx=0x7f0020009ca0 comp rx=0 tx=0).stop 2024-04-25T11:31:09.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.725+0000 7f002fc62700 1 -- 172.21.15.90:0/1871641759 shutdown_connections 2024-04-25T11:31:09.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.725+0000 7f002fc62700 1 --2- 172.21.15.90:0/1871641759 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0028109fd0 0x7f002810c3f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:09.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.725+0000 7f002fc62700 1 --2- 172.21.15.90:0/1871641759 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0028104d10 0x7f0028107130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:09.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.725+0000 7f002fc62700 1 --2- 172.21.15.90:0/1871641759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0028107670 0x7f0028109a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:09.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.725+0000 7f002fc62700 1 -- 172.21.15.90:0/1871641759 >> 172.21.15.90:0/1871641759 conn(0x7f00280fe750 msgr2=0x7f0028100c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:31:09.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.725+0000 7f002fc62700 1 -- 172.21.15.90:0/1871641759 shutdown_connections 2024-04-25T11:31:09.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.725+0000 7f002fc62700 1 -- 172.21.15.90:0/1871641759 wait complete. 2024-04-25T11:31:09.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.726+0000 7f002fc62700 1 Processor -- start 2024-04-25T11:31:09.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.726+0000 7f002fc62700 1 -- start start 2024-04-25T11:31:09.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.726+0000 7f002fc62700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0028104d10 0x7f002819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:09.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.726+0000 7f002fc62700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0028107670 0x7f002819ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:09.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.726+0000 7f002fc62700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0028109fd0 0x7f00281a1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:09.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.726+0000 7f002fc62700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f002810fc60 con 0x7f0028107670 2024-04-25T11:31:09.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.726+0000 7f002fc62700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f002810fae0 con 0x7f0028104d10 2024-04-25T11:31:09.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.726+0000 7f002fc62700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f002810fde0 con 0x7f0028109fd0 2024-04-25T11:31:09.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.726+0000 7f002d1fd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0028107670 0x7f002819ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:09.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.726+0000 7f002d1fd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0028107670 0x7f002819ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44646/0 (socket says 172.21.15.90:44646) 2024-04-25T11:31:09.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.726+0000 7f002d1fd700 1 -- 172.21.15.90:0/163332806 learned_addr learned my addr 172.21.15.90:0/163332806 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:31:09.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.726+0000 7f002d9fe700 1 --2- 172.21.15.90:0/163332806 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0028104d10 0x7f002819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:09.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.726+0000 7f002e1ff700 1 --2- 172.21.15.90:0/163332806 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0028109fd0 0x7f00281a1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:09.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.726+0000 7f002d1fd700 1 -- 172.21.15.90:0/163332806 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0028109fd0 msgr2=0x7f00281a1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:09.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.726+0000 7f002d1fd700 1 --2- 172.21.15.90:0/163332806 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0028109fd0 0x7f00281a1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:09.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.726+0000 7f002d1fd700 1 -- 172.21.15.90:0/163332806 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0028104d10 msgr2=0x7f002819a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:09.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.726+0000 7f002d1fd700 1 --2- 172.21.15.90:0/163332806 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0028104d10 0x7f002819a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:09.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.726+0000 7f002d1fd700 1 -- 172.21.15.90:0/163332806 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00281a22b0 con 0x7f0028107670 2024-04-25T11:31:09.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.727+0000 7f002d1fd700 1 --2- 172.21.15.90:0/163332806 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0028107670 0x7f002819ac90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f001800d950 tx=0x7f001800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:31:09.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.727+0000 7f001effd700 1 -- 172.21.15.90:0/163332806 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0018016070 con 0x7f0028107670 2024-04-25T11:31:09.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.727+0000 7f002fc62700 1 -- 172.21.15.90:0/163332806 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00281a25a0 con 0x7f0028107670 2024-04-25T11:31:09.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.727+0000 7f001effd700 1 -- 172.21.15.90:0/163332806 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f001800fd40 con 0x7f0028107670 2024-04-25T11:31:09.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.727+0000 7f001effd700 1 -- 172.21.15.90:0/163332806 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001800e3b0 con 0x7f0028107670 2024-04-25T11:31:09.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.727+0000 7f002fc62700 1 -- 172.21.15.90:0/163332806 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00281a2ae0 con 0x7f0028107670 2024-04-25T11:31:09.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.728+0000 7f002fc62700 1 -- 172.21.15.90:0/163332806 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0028051190 con 0x7f0028107670 2024-04-25T11:31:09.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.729+0000 7f001effd700 1 -- 172.21.15.90:0/163332806 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f0018019070 con 0x7f0028107670 2024-04-25T11:31:09.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.730+0000 7f001effd700 1 --2- 172.21.15.90:0/163332806 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f00140afb80 0x7f00140b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:09.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.730+0000 7f002d9fe700 1 --2- 172.21.15.90:0/163332806 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f00140afb80 0x7f00140b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:09.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.730+0000 7f001effd700 1 -- 172.21.15.90:0/163332806 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5829+0+0 (secure 0 0 0) 0x7f00180ddd50 con 0x7f0028107670 2024-04-25T11:31:09.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.730+0000 7f002d9fe700 1 --2- 172.21.15.90:0/163332806 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f00140afb80 0x7f00140b2090 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0024009550 tx=0x7f00240091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:31:09.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.735+0000 7f001effd700 1 -- 172.21.15.90:0/163332806 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f00180e1030 con 0x7f0028107670 2024-04-25T11:31:09.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:09.940+0000 7f002fc62700 1 -- 172.21.15.90:0/163332806 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f0028063ad0 con 0x7f00140afb80 2024-04-25T11:31:10.538 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:10 smithi090 ceph-mon[29575]: osdmap e56: 9 total, 9 up, 9 in 2024-04-25T11:31:10.538 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:10 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:31:10.539 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:10 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:31:10.539 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:10 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:10.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:10 smithi138 ceph-mon[32328]: osdmap e56: 9 total, 9 up, 9 in 2024-04-25T11:31:10.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:10 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:31:10.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:10 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:31:10.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:10 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:10.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:10 smithi156 ceph-mon[33045]: osdmap e56: 9 total, 9 up, 9 in 2024-04-25T11:31:10.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:10 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:31:10.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:10 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:31:10.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:10 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:11.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:11 smithi138 ceph-mon[32328]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:31:11.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:11 smithi138 ceph-mon[32328]: pgmap v193: 1 pgs: 1 peering; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-25T11:31:11.580 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:11 smithi090 ceph-mon[29575]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:31:11.580 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:11 smithi090 ceph-mon[29575]: pgmap v193: 1 pgs: 1 peering; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-25T11:31:11.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:11 smithi156 ceph-mon[33045]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:31:11.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:11 smithi156 ceph-mon[33045]: pgmap v193: 1 pgs: 1 peering; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-25T11:31:14.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:13 smithi138 ceph-mon[32328]: pgmap v194: 1 pgs: 1 peering; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-25T11:31:14.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:13 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/4005572719' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d70fdf5e-9d0f-483d-a7d8-505b79b8a418"}]: dispatch 2024-04-25T11:31:14.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:13 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/4005572719' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d70fdf5e-9d0f-483d-a7d8-505b79b8a418"}]': finished 2024-04-25T11:31:14.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:13 smithi138 ceph-mon[32328]: osdmap e57: 10 total, 9 up, 10 in 2024-04-25T11:31:14.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:13 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T11:31:14.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:13 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3828922626' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:31:14.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:13 smithi156 ceph-mon[33045]: pgmap v194: 1 pgs: 1 peering; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-25T11:31:14.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:13 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/4005572719' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d70fdf5e-9d0f-483d-a7d8-505b79b8a418"}]: dispatch 2024-04-25T11:31:14.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:13 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/4005572719' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d70fdf5e-9d0f-483d-a7d8-505b79b8a418"}]': finished 2024-04-25T11:31:14.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:13 smithi156 ceph-mon[33045]: osdmap e57: 10 total, 9 up, 10 in 2024-04-25T11:31:14.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:13 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T11:31:14.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:13 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3828922626' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:31:14.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:13 smithi090 ceph-mon[29575]: pgmap v194: 1 pgs: 1 peering; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-04-25T11:31:14.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:13 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/4005572719' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d70fdf5e-9d0f-483d-a7d8-505b79b8a418"}]: dispatch 2024-04-25T11:31:14.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:13 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/4005572719' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d70fdf5e-9d0f-483d-a7d8-505b79b8a418"}]': finished 2024-04-25T11:31:14.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:13 smithi090 ceph-mon[29575]: osdmap e57: 10 total, 9 up, 10 in 2024-04-25T11:31:14.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:13 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T11:31:14.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:13 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3828922626' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:31:16.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:15 smithi138 ceph-mon[32328]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-25T11:31:16.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:15 smithi156 ceph-mon[33045]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-25T11:31:16.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:15 smithi090 ceph-mon[29575]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-25T11:31:18.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:17 smithi138 ceph-mon[32328]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T11:31:18.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:17 smithi156 ceph-mon[33045]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T11:31:18.134 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:17 smithi090 ceph-mon[29575]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T11:31:20.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:19 smithi138 ceph-mon[32328]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-25T11:31:20.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:19 smithi156 ceph-mon[33045]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-25T11:31:20.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:19 smithi090 ceph-mon[29575]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-25T11:31:21.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:20 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T11:31:21.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:20 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:21.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:20 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T11:31:21.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:20 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:21.143 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:20 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T11:31:21.143 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:20 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:22.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:21 smithi138 ceph-mon[32328]: Deploying daemon osd.9 on smithi090 2024-04-25T11:31:22.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:21 smithi138 ceph-mon[32328]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T11:31:22.061 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:21 smithi090 ceph-mon[29575]: Deploying daemon osd.9 on smithi090 2024-04-25T11:31:22.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:21 smithi090 ceph-mon[29575]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T11:31:22.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:21 smithi156 ceph-mon[33045]: Deploying daemon osd.9 on smithi090 2024-04-25T11:31:22.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:21 smithi156 ceph-mon[33045]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T11:31:23.932 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:23 smithi090 ceph-mon[29575]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T11:31:24.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:23 smithi138 ceph-mon[32328]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T11:31:24.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:23 smithi156 ceph-mon[33045]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T11:31:25.898 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:25 smithi090 ceph-mon[29575]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T11:31:26.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:25 smithi138 ceph-mon[32328]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T11:31:26.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:25 smithi156 ceph-mon[33045]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T11:31:27.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:27 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:27.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:27 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:31:27.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:27 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:27.446 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:27 smithi090 ceph-mon[29575]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T11:31:27.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:27 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:27.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:27 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:31:27.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:27 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:27.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:27 smithi138 ceph-mon[32328]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T11:31:27.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:27 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:27.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:27 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:31:27.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:27 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:27.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:27 smithi156 ceph-mon[33045]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T11:31:29.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:29 smithi090 ceph-mon[29575]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T11:31:29.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:29 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:29.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:29 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:29.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:29 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:29.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:29 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:31:29.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:29 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:30.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:29 smithi138 ceph-mon[32328]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T11:31:30.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:29 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:30.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:29 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:30.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:29 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:30.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:29 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:31:30.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:29 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:30.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:29 smithi156 ceph-mon[33045]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T11:31:30.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:29 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:30.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:29 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:30.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:29 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:30.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:29 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:31:30.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:29 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:30.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:30.146+0000 7f001effd700 1 -- 172.21.15.90:0/163332806 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0028063ad0 con 0x7f00140afb80 2024-04-25T11:31:30.137 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 9 on host 'smithi090' 2024-04-25T11:31:30.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:30.149+0000 7f002fc62700 1 -- 172.21.15.90:0/163332806 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f00140afb80 msgr2=0x7f00140b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:30.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:30.149+0000 7f002fc62700 1 --2- 172.21.15.90:0/163332806 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f00140afb80 0x7f00140b2090 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0024009550 tx=0x7f00240091b0 comp rx=0 tx=0).stop 2024-04-25T11:31:30.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:30.149+0000 7f002fc62700 1 -- 172.21.15.90:0/163332806 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0028107670 msgr2=0x7f002819ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:30.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:30.149+0000 7f002fc62700 1 --2- 172.21.15.90:0/163332806 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0028107670 0x7f002819ac90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f001800d950 tx=0x7f001800dc60 comp rx=0 tx=0).stop 2024-04-25T11:31:30.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:30.149+0000 7f002fc62700 1 -- 172.21.15.90:0/163332806 shutdown_connections 2024-04-25T11:31:30.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:30.149+0000 7f002fc62700 1 --2- 172.21.15.90:0/163332806 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f00140afb80 0x7f00140b2090 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:30.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:30.149+0000 7f002fc62700 1 --2- 172.21.15.90:0/163332806 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0028109fd0 0x7f00281a1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:30.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:30.149+0000 7f002fc62700 1 --2- 172.21.15.90:0/163332806 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0028104d10 0x7f002819a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:30.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:30.149+0000 7f002fc62700 1 --2- 172.21.15.90:0/163332806 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0028107670 0x7f002819ac90 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:30.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:30.149+0000 7f002fc62700 1 -- 172.21.15.90:0/163332806 >> 172.21.15.90:0/163332806 conn(0x7f00280fe750 msgr2=0x7f0028105c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:31:30.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:30.149+0000 7f002fc62700 1 -- 172.21.15.90:0/163332806 shutdown_connections 2024-04-25T11:31:30.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:31:30.149+0000 7f002fc62700 1 -- 172.21.15.90:0/163332806 wait complete. 2024-04-25T11:31:30.871 DEBUG:teuthology.orchestra.run.smithi090:osd.9> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.9.service 2024-04-25T11:31:30.873 INFO:tasks.cephadm:Deploying osd.10 on smithi138 with /dev/vg_nvme/lv_1... 2024-04-25T11:31:30.873 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T11:31:31.035 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:30 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:31:31.035 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:30 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:31.035 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:30 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:31.035 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:30 smithi090 ceph-mon[29575]: from='osd.9 [v2:172.21.15.90:6826/1171265966,v1:172.21.15.90:6827/1171265966]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T11:31:31.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:30 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:31:31.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:30 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:31.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:30 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:31.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:30 smithi138 ceph-mon[32328]: from='osd.9 [v2:172.21.15.90:6826/1171265966,v1:172.21.15.90:6827/1171265966]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T11:31:31.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:30 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:31:31.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:30 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:31.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:30 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:31.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:30 smithi156 ceph-mon[33045]: from='osd.9 [v2:172.21.15.90:6826/1171265966,v1:172.21.15.90:6827/1171265966]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T11:31:31.120 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T11:31:32.033 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:31 smithi090 ceph-mon[29575]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T11:31:32.033 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:31 smithi090 ceph-mon[29575]: from='osd.9 [v2:172.21.15.90:6826/1171265966,v1:172.21.15.90:6827/1171265966]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T11:31:32.033 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:31 smithi090 ceph-mon[29575]: osdmap e58: 10 total, 9 up, 10 in 2024-04-25T11:31:32.034 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:31 smithi090 ceph-mon[29575]: from='osd.9 [v2:172.21.15.90:6826/1171265966,v1:172.21.15.90:6827/1171265966]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-25T11:31:32.034 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:31 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T11:31:32.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:31 smithi138 ceph-mon[32328]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T11:31:32.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:31 smithi138 ceph-mon[32328]: from='osd.9 [v2:172.21.15.90:6826/1171265966,v1:172.21.15.90:6827/1171265966]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T11:31:32.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:31 smithi138 ceph-mon[32328]: osdmap e58: 10 total, 9 up, 10 in 2024-04-25T11:31:32.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:31 smithi138 ceph-mon[32328]: from='osd.9 [v2:172.21.15.90:6826/1171265966,v1:172.21.15.90:6827/1171265966]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-25T11:31:32.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:31 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T11:31:32.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:31 smithi156 ceph-mon[33045]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T11:31:32.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:31 smithi156 ceph-mon[33045]: from='osd.9 [v2:172.21.15.90:6826/1171265966,v1:172.21.15.90:6827/1171265966]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T11:31:32.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:31 smithi156 ceph-mon[33045]: osdmap e58: 10 total, 9 up, 10 in 2024-04-25T11:31:32.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:31 smithi156 ceph-mon[33045]: from='osd.9 [v2:172.21.15.90:6826/1171265966,v1:172.21.15.90:6827/1171265966]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-25T11:31:32.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:31 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T11:31:32.330 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:31:32 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:31:32.109+0000 7fe6589eb700 -1 osd.9 0 waiting for initial osdmap 2024-04-25T11:31:32.330 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:31:32 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:31:32.121+0000 7fe6547a0700 -1 osd.9 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T11:31:33.097 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T11:31:33.098 INFO:teuthology.orchestra.run.smithi138.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T11:31:33.098 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10+0 records in 2024-04-25T11:31:33.098 INFO:teuthology.orchestra.run.smithi138.stderr:10+0 records out 2024-04-25T11:31:33.098 INFO:teuthology.orchestra.run.smithi138.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012428 s, 844 MB/s 2024-04-25T11:31:33.098 INFO:teuthology.orchestra.run.smithi138.stderr:--> Zapping successful for: 2024-04-25T11:31:33.403 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:33 smithi138 ceph-mon[32328]: purged_snaps scrub starts 2024-04-25T11:31:33.403 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:33 smithi138 ceph-mon[32328]: purged_snaps scrub ok 2024-04-25T11:31:33.403 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:33 smithi138 ceph-mon[32328]: from='osd.9 [v2:172.21.15.90:6826/1171265966,v1:172.21.15.90:6827/1171265966]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-25T11:31:33.403 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:33 smithi138 ceph-mon[32328]: osdmap e59: 10 total, 9 up, 10 in 2024-04-25T11:31:33.403 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:33 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T11:31:33.403 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:33 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T11:31:33.403 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:33 smithi138 ceph-mon[32328]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T11:31:33.468 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:33 smithi090 ceph-mon[29575]: purged_snaps scrub starts 2024-04-25T11:31:33.469 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:33 smithi090 ceph-mon[29575]: purged_snaps scrub ok 2024-04-25T11:31:33.469 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:33 smithi090 ceph-mon[29575]: from='osd.9 [v2:172.21.15.90:6826/1171265966,v1:172.21.15.90:6827/1171265966]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-25T11:31:33.469 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:33 smithi090 ceph-mon[29575]: osdmap e59: 10 total, 9 up, 10 in 2024-04-25T11:31:33.469 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:33 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T11:31:33.469 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:33 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T11:31:33.469 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:33 smithi090 ceph-mon[29575]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T11:31:33.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:33 smithi156 ceph-mon[33045]: purged_snaps scrub starts 2024-04-25T11:31:33.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:33 smithi156 ceph-mon[33045]: purged_snaps scrub ok 2024-04-25T11:31:33.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:33 smithi156 ceph-mon[33045]: from='osd.9 [v2:172.21.15.90:6826/1171265966,v1:172.21.15.90:6827/1171265966]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-25T11:31:33.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:33 smithi156 ceph-mon[33045]: osdmap e59: 10 total, 9 up, 10 in 2024-04-25T11:31:33.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:33 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T11:31:33.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:33 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T11:31:33.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:33 smithi156 ceph-mon[33045]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T11:31:33.691 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi138:vg_nvme/lv_1 2024-04-25T11:31:33.936 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T11:31:34.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:34 smithi090 ceph-mon[29575]: osd.9 [v2:172.21.15.90:6826/1171265966,v1:172.21.15.90:6827/1171265966] boot 2024-04-25T11:31:34.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:34 smithi090 ceph-mon[29575]: osdmap e60: 10 total, 10 up, 10 in 2024-04-25T11:31:34.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:34 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T11:31:34.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:34 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:34.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:34 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:34.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:34 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:34.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:34 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:34.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:34 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:34.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:34 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:34.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:34 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:34.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:34 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:34.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:34 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:31:34.228 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:34 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:34.311 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:34 smithi138 ceph-mon[32328]: osd.9 [v2:172.21.15.90:6826/1171265966,v1:172.21.15.90:6827/1171265966] boot 2024-04-25T11:31:34.311 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:34 smithi138 ceph-mon[32328]: osdmap e60: 10 total, 10 up, 10 in 2024-04-25T11:31:34.311 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:34 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T11:31:34.312 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:34 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:34.312 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:34 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:34.312 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:34 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:34.312 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:34 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:34.312 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:34 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:34.312 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:34 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:34.312 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:34 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:34.312 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:34 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:34.312 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:34 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:31:34.312 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:34 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:34.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:34 smithi156 ceph-mon[33045]: osd.9 [v2:172.21.15.90:6826/1171265966,v1:172.21.15.90:6827/1171265966] boot 2024-04-25T11:31:34.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:34 smithi156 ceph-mon[33045]: osdmap e60: 10 total, 10 up, 10 in 2024-04-25T11:31:34.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:34 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T11:31:34.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:34 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:34.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:34 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:34.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:34 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:34.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:34 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:34.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:34 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:34.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:34 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:34.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:34 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:34.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:34 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:34.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:34 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:31:34.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:34 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:35.236 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:35 smithi138 ceph-mon[32328]: Detected new or changed devices on smithi090 2024-04-25T11:31:35.236 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:35 smithi138 ceph-mon[32328]: Adjusting osd_memory_target on smithi090 to 4272M 2024-04-25T11:31:35.236 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:35 smithi138 ceph-mon[32328]: osdmap e61: 10 total, 10 up, 10 in 2024-04-25T11:31:35.236 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:35 smithi138 ceph-mon[32328]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:35.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.473+0000 7f277aa8d700 1 -- 172.21.15.138:0/934117622 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2774105d10 msgr2=0x7f277410c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:35.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.473+0000 7f277aa8d700 1 --2- 172.21.15.138:0/934117622 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2774105d10 0x7f277410c5c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f276c009990 tx=0x7f276c009ca0 comp rx=0 tx=0).stop 2024-04-25T11:31:35.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.473+0000 7f277aa8d700 1 -- 172.21.15.138:0/934117622 shutdown_connections 2024-04-25T11:31:35.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.473+0000 7f277aa8d700 1 --2- 172.21.15.138:0/934117622 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2774105d10 0x7f277410c5c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:35.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.473+0000 7f277aa8d700 1 --2- 172.21.15.138:0/934117622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2774104240 0x7f2774104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:35.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.473+0000 7f277aa8d700 1 --2- 172.21.15.138:0/934117622 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2774105370 0x7f2774105780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:35.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.473+0000 7f277aa8d700 1 -- 172.21.15.138:0/934117622 >> 172.21.15.138:0/934117622 conn(0x7f27740ff8a0 msgr2=0x7f2774101d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:31:35.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.473+0000 7f277aa8d700 1 -- 172.21.15.138:0/934117622 shutdown_connections 2024-04-25T11:31:35.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.473+0000 7f277aa8d700 1 -- 172.21.15.138:0/934117622 wait complete. 2024-04-25T11:31:35.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.474+0000 7f277aa8d700 1 Processor -- start 2024-04-25T11:31:35.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.474+0000 7f277aa8d700 1 -- start start 2024-04-25T11:31:35.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.474+0000 7f277aa8d700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2774104240 0x7f277419a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:35.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.474+0000 7f277aa8d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2774105370 0x7f277419adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:35.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.474+0000 7f277aa8d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2774105d10 0x7f27741a1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:35.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.474+0000 7f277aa8d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f277410fda0 con 0x7f2774105d10 2024-04-25T11:31:35.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.474+0000 7f277aa8d700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f277410fc20 con 0x7f2774104240 2024-04-25T11:31:35.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.474+0000 7f277aa8d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f277410ff20 con 0x7f2774105370 2024-04-25T11:31:35.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.474+0000 7f2778829700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2774104240 0x7f277419a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:35.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.474+0000 7f2778829700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2774104240 0x7f277419a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:37428/0 (socket says 172.21.15.138:37428) 2024-04-25T11:31:35.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.474+0000 7f2778829700 1 -- 172.21.15.138:0/345516701 learned_addr learned my addr 172.21.15.138:0/345516701 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-04-25T11:31:35.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.474+0000 7f2773fff700 1 --2- 172.21.15.138:0/345516701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2774105370 0x7f277419adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:35.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.474+0000 7f277902a700 1 --2- 172.21.15.138:0/345516701 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2774105d10 0x7f27741a1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:35.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.474+0000 7f2778829700 1 -- 172.21.15.138:0/345516701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2774105370 msgr2=0x7f277419adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:35.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.474+0000 7f2778829700 1 --2- 172.21.15.138:0/345516701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2774105370 0x7f277419adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:35.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.474+0000 7f2778829700 1 -- 172.21.15.138:0/345516701 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2774105d10 msgr2=0x7f27741a1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:35.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.474+0000 7f2778829700 1 --2- 172.21.15.138:0/345516701 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2774105d10 0x7f27741a1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:35.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.474+0000 7f2778829700 1 -- 172.21.15.138:0/345516701 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27741a2410 con 0x7f2774104240 2024-04-25T11:31:35.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.475+0000 7f2778829700 1 --2- 172.21.15.138:0/345516701 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2774104240 0x7f277419a8b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f276400b990 tx=0x7f276400bca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:31:35.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.475+0000 7f2771ffb700 1 -- 172.21.15.138:0/345516701 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276400d890 con 0x7f2774104240 2024-04-25T11:31:35.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.475+0000 7f2771ffb700 1 -- 172.21.15.138:0/345516701 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f276400e460 con 0x7f2774104240 2024-04-25T11:31:35.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.475+0000 7f277aa8d700 1 -- 172.21.15.138:0/345516701 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27741a2700 con 0x7f2774104240 2024-04-25T11:31:35.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.475+0000 7f2771ffb700 1 -- 172.21.15.138:0/345516701 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2764017660 con 0x7f2774104240 2024-04-25T11:31:35.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.475+0000 7f277aa8d700 1 -- 172.21.15.138:0/345516701 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27741a2cc0 con 0x7f2774104240 2024-04-25T11:31:35.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.476+0000 7f277aa8d700 1 -- 172.21.15.138:0/345516701 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2774051190 con 0x7f2774104240 2024-04-25T11:31:35.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.477+0000 7f2771ffb700 1 -- 172.21.15.138:0/345516701 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f276400e610 con 0x7f2774104240 2024-04-25T11:31:35.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.478+0000 7f2771ffb700 1 --2- 172.21.15.138:0/345516701 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f275c0afb80 0x7f275c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:35.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.478+0000 7f2771ffb700 1 -- 172.21.15.138:0/345516701 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6261+0+0 (secure 0 0 0) 0x7f27640de1d0 con 0x7f2774104240 2024-04-25T11:31:35.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.478+0000 7f2773fff700 1 --2- 172.21.15.138:0/345516701 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f275c0afb80 0x7f275c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:35.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.479+0000 7f2773fff700 1 --2- 172.21.15.138:0/345516701 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f275c0afb80 0x7f275c0b2090 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f277419b960 tx=0x7f2768009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:31:35.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.482+0000 7f2771ffb700 1 -- 172.21.15.138:0/345516701 <== mon.2 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2764052ff0 con 0x7f2774104240 2024-04-25T11:31:35.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:35 smithi090 ceph-mon[29575]: Detected new or changed devices on smithi090 2024-04-25T11:31:35.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:35 smithi090 ceph-mon[29575]: Adjusting osd_memory_target on smithi090 to 4272M 2024-04-25T11:31:35.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:35 smithi090 ceph-mon[29575]: osdmap e61: 10 total, 10 up, 10 in 2024-04-25T11:31:35.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:35 smithi090 ceph-mon[29575]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:35.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:35 smithi156 ceph-mon[33045]: Detected new or changed devices on smithi090 2024-04-25T11:31:35.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:35 smithi156 ceph-mon[33045]: Adjusting osd_memory_target on smithi090 to 4272M 2024-04-25T11:31:35.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:35 smithi156 ceph-mon[33045]: osdmap e61: 10 total, 10 up, 10 in 2024-04-25T11:31:35.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:35 smithi156 ceph-mon[33045]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:35.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:35.688+0000 7f277aa8d700 1 -- 172.21.15.138:0/345516701 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f2774063ad0 con 0x7f275c0afb80 2024-04-25T11:31:36.385 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:36 smithi138 ceph-mon[32328]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:31:36.385 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:36 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:31:36.385 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:36 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:31:36.385 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:36 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:36.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:36 smithi090 ceph-mon[29575]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:31:36.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:36 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:31:36.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:36 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:31:36.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:36 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:36.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:36 smithi156 ceph-mon[33045]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi138:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:31:36.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:36 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:31:36.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:36 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:31:36.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:36 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:37.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:37 smithi138 ceph-mon[32328]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:37.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:37 smithi090 ceph-mon[29575]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:37.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:37 smithi156 ceph-mon[33045]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:38.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:38 smithi156 ceph-mon[33045]: from='client.? 172.21.15.138:0/2951607652' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ec86c7d-c3a4-432d-9664-8e40836e1e20"}]: dispatch 2024-04-25T11:31:38.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:38 smithi156 ceph-mon[33045]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ec86c7d-c3a4-432d-9664-8e40836e1e20"}]: dispatch 2024-04-25T11:31:38.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:38 smithi156 ceph-mon[33045]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ec86c7d-c3a4-432d-9664-8e40836e1e20"}]': finished 2024-04-25T11:31:38.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:38 smithi156 ceph-mon[33045]: osdmap e62: 11 total, 10 up, 11 in 2024-04-25T11:31:38.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:38 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T11:31:38.644 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:38 smithi138 ceph-mon[32328]: from='client.? 172.21.15.138:0/2951607652' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ec86c7d-c3a4-432d-9664-8e40836e1e20"}]: dispatch 2024-04-25T11:31:38.644 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:38 smithi138 ceph-mon[32328]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ec86c7d-c3a4-432d-9664-8e40836e1e20"}]: dispatch 2024-04-25T11:31:38.644 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:38 smithi138 ceph-mon[32328]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ec86c7d-c3a4-432d-9664-8e40836e1e20"}]': finished 2024-04-25T11:31:38.644 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:38 smithi138 ceph-mon[32328]: osdmap e62: 11 total, 10 up, 11 in 2024-04-25T11:31:38.644 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:38 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T11:31:38.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:38 smithi090 ceph-mon[29575]: from='client.? 172.21.15.138:0/2951607652' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ec86c7d-c3a4-432d-9664-8e40836e1e20"}]: dispatch 2024-04-25T11:31:38.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:38 smithi090 ceph-mon[29575]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ec86c7d-c3a4-432d-9664-8e40836e1e20"}]: dispatch 2024-04-25T11:31:38.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:38 smithi090 ceph-mon[29575]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ec86c7d-c3a4-432d-9664-8e40836e1e20"}]': finished 2024-04-25T11:31:38.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:38 smithi090 ceph-mon[29575]: osdmap e62: 11 total, 10 up, 11 in 2024-04-25T11:31:38.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:38 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T11:31:39.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:39 smithi090 ceph-mon[29575]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:39.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:39 smithi090 ceph-mon[29575]: from='client.? 172.21.15.138:0/1734930088' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:31:39.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:39 smithi138 ceph-mon[32328]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:39.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:39 smithi138 ceph-mon[32328]: from='client.? 172.21.15.138:0/1734930088' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:31:39.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:39 smithi156 ceph-mon[33045]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:39.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:39 smithi156 ceph-mon[33045]: from='client.? 172.21.15.138:0/1734930088' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:31:42.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:41 smithi138 ceph-mon[32328]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:42.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:41 smithi156 ceph-mon[33045]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:42.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:41 smithi090 ceph-mon[29575]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:44.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:43 smithi138 ceph-mon[32328]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:44.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:43 smithi156 ceph-mon[33045]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:44.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:43 smithi090 ceph-mon[29575]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:45.951 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:45 smithi138 ceph-mon[32328]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:46.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:45 smithi156 ceph-mon[33045]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:46.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:45 smithi090 ceph-mon[29575]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:46.870 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:46 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T11:31:46.870 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:46 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:47.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:46 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T11:31:47.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:46 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:47.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:46 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T11:31:47.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:46 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:47.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:47 smithi138 ceph-mon[32328]: Deploying daemon osd.10 on smithi138 2024-04-25T11:31:47.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:47 smithi138 ceph-mon[32328]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:48.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:47 smithi156 ceph-mon[33045]: Deploying daemon osd.10 on smithi138 2024-04-25T11:31:48.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:47 smithi156 ceph-mon[33045]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:48.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:47 smithi090 ceph-mon[29575]: Deploying daemon osd.10 on smithi138 2024-04-25T11:31:48.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:47 smithi090 ceph-mon[29575]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:49.871 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:49 smithi138 ceph-mon[32328]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:50.043 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:49 smithi090 ceph-mon[29575]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:50.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:49 smithi156 ceph-mon[33045]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:52.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:51 smithi138 ceph-mon[32328]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:52.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:51 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:52.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:51 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:52.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:51 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:31:52.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:51 smithi156 ceph-mon[33045]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:52.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:51 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:52.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:51 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:52.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:51 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:31:52.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:51 smithi090 ceph-mon[29575]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:52.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:51 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:52.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:51 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:52.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:51 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:31:54.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:53 smithi138 ceph-mon[32328]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:54.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:53 smithi156 ceph-mon[33045]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:54.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:53 smithi090 ceph-mon[29575]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:55.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:55.659+0000 7f2771ffb700 1 -- 172.21.15.138:0/345516701 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f2774063ad0 con 0x7f275c0afb80 2024-04-25T11:31:55.650 INFO:teuthology.orchestra.run.smithi138.stdout:Created osd(s) 10 on host 'smithi138' 2024-04-25T11:31:55.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:55.662+0000 7f277aa8d700 1 -- 172.21.15.138:0/345516701 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f275c0afb80 msgr2=0x7f275c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:55.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:55.662+0000 7f277aa8d700 1 --2- 172.21.15.138:0/345516701 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f275c0afb80 0x7f275c0b2090 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f277419b960 tx=0x7f2768009190 comp rx=0 tx=0).stop 2024-04-25T11:31:55.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:55.662+0000 7f277aa8d700 1 -- 172.21.15.138:0/345516701 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2774104240 msgr2=0x7f277419a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:55.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:55.662+0000 7f277aa8d700 1 --2- 172.21.15.138:0/345516701 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2774104240 0x7f277419a8b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f276400b990 tx=0x7f276400bca0 comp rx=0 tx=0).stop 2024-04-25T11:31:55.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:55.662+0000 7f277aa8d700 1 -- 172.21.15.138:0/345516701 shutdown_connections 2024-04-25T11:31:55.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:55.662+0000 7f277aa8d700 1 --2- 172.21.15.138:0/345516701 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2774105d10 0x7f27741a1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:55.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:55.662+0000 7f277aa8d700 1 --2- 172.21.15.138:0/345516701 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2774104240 0x7f277419a8b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:55.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:55.662+0000 7f277aa8d700 1 --2- 172.21.15.138:0/345516701 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f275c0afb80 0x7f275c0b2090 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:55.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:55.662+0000 7f277aa8d700 1 --2- 172.21.15.138:0/345516701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2774105370 0x7f277419adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:55.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:55.662+0000 7f277aa8d700 1 -- 172.21.15.138:0/345516701 >> 172.21.15.138:0/345516701 conn(0x7f27740ff8a0 msgr2=0x7f27741085a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:31:55.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:55.662+0000 7f277aa8d700 1 -- 172.21.15.138:0/345516701 shutdown_connections 2024-04-25T11:31:55.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-25T11:31:55.662+0000 7f277aa8d700 1 -- 172.21.15.138:0/345516701 wait complete. 2024-04-25T11:31:55.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:55 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:55.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:55 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:55.933 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:55 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:55.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:55 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:31:55.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:55 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:55.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:55 smithi138 ceph-mon[32328]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:55.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:55 smithi138 ceph-mon[32328]: from='osd.10 [v2:172.21.15.138:6824/4264666232,v1:172.21.15.138:6825/4264666232]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T11:31:55.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:55 smithi138 ceph-mon[32328]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T11:31:55.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:55 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:55.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:55 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:55.934 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:55 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:31:55.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:55 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:55.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:55 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:55.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:55 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:55.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:55 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:31:55.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:55 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:55.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:55 smithi090 ceph-mon[29575]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:55.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:55 smithi090 ceph-mon[29575]: from='osd.10 [v2:172.21.15.138:6824/4264666232,v1:172.21.15.138:6825/4264666232]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T11:31:55.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:55 smithi090 ceph-mon[29575]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T11:31:55.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:55 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:55.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:55 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:55.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:55 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:31:56.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:55 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:56.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:55 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:56.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:55 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:56.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:55 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:31:56.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:55 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:56.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:55 smithi156 ceph-mon[33045]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:56.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:55 smithi156 ceph-mon[33045]: from='osd.10 [v2:172.21.15.138:6824/4264666232,v1:172.21.15.138:6825/4264666232]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T11:31:56.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:55 smithi156 ceph-mon[33045]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T11:31:56.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:55 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:56.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:55 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:56.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:55 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:31:56.222 DEBUG:teuthology.orchestra.run.smithi138:osd.10> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.10.service 2024-04-25T11:31:56.225 INFO:tasks.cephadm:Deploying osd.11 on smithi156 with /dev/vg_nvme/lv_1... 2024-04-25T11:31:56.225 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T11:31:56.471 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T11:31:56.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:56 smithi138 ceph-mon[32328]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T11:31:56.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:56 smithi138 ceph-mon[32328]: osdmap e63: 11 total, 10 up, 11 in 2024-04-25T11:31:56.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:56 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T11:31:56.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:56 smithi138 ceph-mon[32328]: from='osd.10 [v2:172.21.15.138:6824/4264666232,v1:172.21.15.138:6825/4264666232]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-04-25T11:31:56.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:56 smithi138 ceph-mon[32328]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-04-25T11:31:56.903 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:56 smithi156 ceph-mon[33045]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T11:31:56.903 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:56 smithi156 ceph-mon[33045]: osdmap e63: 11 total, 10 up, 11 in 2024-04-25T11:31:56.903 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:56 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T11:31:56.903 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:56 smithi156 ceph-mon[33045]: from='osd.10 [v2:172.21.15.138:6824/4264666232,v1:172.21.15.138:6825/4264666232]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-04-25T11:31:56.903 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:56 smithi156 ceph-mon[33045]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-04-25T11:31:56.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:56 smithi090 ceph-mon[29575]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T11:31:56.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:56 smithi090 ceph-mon[29575]: osdmap e63: 11 total, 10 up, 11 in 2024-04-25T11:31:56.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:56 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T11:31:56.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:56 smithi090 ceph-mon[29575]: from='osd.10 [v2:172.21.15.138:6824/4264666232,v1:172.21.15.138:6825/4264666232]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-04-25T11:31:56.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:56 smithi090 ceph-mon[29575]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]: dispatch 2024-04-25T11:31:57.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:57 smithi138 ceph-mon[32328]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-04-25T11:31:57.935 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:57 smithi138 ceph-mon[32328]: osdmap e64: 11 total, 10 up, 11 in 2024-04-25T11:31:57.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:57 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T11:31:57.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:57 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T11:31:57.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:57 smithi138 ceph-mon[32328]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:57.936 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:57 smithi138 ceph-mon[32328]: from='osd.10 ' entity='osd.10' 2024-04-25T11:31:57.936 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:31:57 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:31:57.607+0000 7f6fde95d700 -1 osd.10 0 waiting for initial osdmap 2024-04-25T11:31:57.936 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:31:57 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:31:57.776+0000 7f6fd5f09700 -1 osd.10 65 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T11:31:57.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:57 smithi090 ceph-mon[29575]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-04-25T11:31:57.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:57 smithi090 ceph-mon[29575]: osdmap e64: 11 total, 10 up, 11 in 2024-04-25T11:31:57.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:57 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T11:31:57.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:57 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T11:31:57.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:57 smithi090 ceph-mon[29575]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:57.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:57 smithi090 ceph-mon[29575]: from='osd.10 ' entity='osd.10' 2024-04-25T11:31:58.014 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:57 smithi156 ceph-mon[33045]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi138", "root=default"]}]': finished 2024-04-25T11:31:58.014 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:57 smithi156 ceph-mon[33045]: osdmap e64: 11 total, 10 up, 11 in 2024-04-25T11:31:58.014 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:57 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T11:31:58.014 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:57 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T11:31:58.015 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:57 smithi156 ceph-mon[33045]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T11:31:58.015 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:57 smithi156 ceph-mon[33045]: from='osd.10 ' entity='osd.10' 2024-04-25T11:31:58.594 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T11:31:58.594 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T11:31:58.594 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2024-04-25T11:31:58.595 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2024-04-25T11:31:58.595 INFO:teuthology.orchestra.run.smithi156.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0122462 s, 856 MB/s 2024-04-25T11:31:58.595 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 2024-04-25T11:31:58.595 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2024-04-25T11:31:58.905 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:58 smithi156 ceph-mon[33045]: purged_snaps scrub starts 2024-04-25T11:31:58.905 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:58 smithi156 ceph-mon[33045]: purged_snaps scrub ok 2024-04-25T11:31:58.905 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:58 smithi156 ceph-mon[33045]: osdmap e65: 11 total, 10 up, 11 in 2024-04-25T11:31:58.905 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:58 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T11:31:58.905 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:58 smithi156 ceph-mon[33045]: osd.10 [v2:172.21.15.138:6824/4264666232,v1:172.21.15.138:6825/4264666232] boot 2024-04-25T11:31:58.905 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:58 smithi156 ceph-mon[33045]: osdmap e66: 11 total, 11 up, 11 in 2024-04-25T11:31:58.905 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:58 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T11:31:58.906 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:58 smithi138 ceph-mon[32328]: purged_snaps scrub starts 2024-04-25T11:31:58.906 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:58 smithi138 ceph-mon[32328]: purged_snaps scrub ok 2024-04-25T11:31:58.906 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:58 smithi138 ceph-mon[32328]: osdmap e65: 11 total, 10 up, 11 in 2024-04-25T11:31:58.906 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:58 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T11:31:58.906 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:58 smithi138 ceph-mon[32328]: osd.10 [v2:172.21.15.138:6824/4264666232,v1:172.21.15.138:6825/4264666232] boot 2024-04-25T11:31:58.906 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:58 smithi138 ceph-mon[32328]: osdmap e66: 11 total, 11 up, 11 in 2024-04-25T11:31:58.906 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:58 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T11:31:58.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:58 smithi090 ceph-mon[29575]: purged_snaps scrub starts 2024-04-25T11:31:58.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:58 smithi090 ceph-mon[29575]: purged_snaps scrub ok 2024-04-25T11:31:58.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:58 smithi090 ceph-mon[29575]: osdmap e65: 11 total, 10 up, 11 in 2024-04-25T11:31:58.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:58 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T11:31:58.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:58 smithi090 ceph-mon[29575]: osd.10 [v2:172.21.15.138:6824/4264666232,v1:172.21.15.138:6825/4264666232] boot 2024-04-25T11:31:58.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:58 smithi090 ceph-mon[29575]: osdmap e66: 11 total, 11 up, 11 in 2024-04-25T11:31:58.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:58 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T11:31:59.298 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi156:vg_nvme/lv_1 2024-04-25T11:31:59.536 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T11:31:59.975 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:59 smithi156 ceph-mon[33045]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-04-25T11:31:59.975 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:59 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:59.975 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:59 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:59.975 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:59 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:59.975 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:59 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:59.975 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:59 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:59.976 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:59 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:59.976 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:59 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:59.976 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:59 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:59.976 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:59 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:31:59.976 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:59 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:59.976 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:31:59 smithi156 ceph-mon[33045]: osdmap e67: 11 total, 11 up, 11 in 2024-04-25T11:31:59.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:59 smithi090 ceph-mon[29575]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-04-25T11:31:59.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:59 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:59.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:59 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:59.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:59 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:59.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:59 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:59.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:59 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:59.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:59 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:31:59.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:59 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:59.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:59 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:31:59.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:59 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:31:59.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:59 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:31:59.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:31:59 smithi090 ceph-mon[29575]: osdmap e67: 11 total, 11 up, 11 in 2024-04-25T11:32:00.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:59 smithi138 ceph-mon[32328]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-04-25T11:32:00.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:59 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:00.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:59 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:00.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:59 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:32:00.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:59 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:32:00.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:59 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:32:00.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:59 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:32:00.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:59 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:00.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:59 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:00.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:59 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:32:00.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:59 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:00.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:31:59 smithi138 ceph-mon[32328]: osdmap e67: 11 total, 11 up, 11 in 2024-04-25T11:32:00.956 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:00 smithi156 ceph-mon[33045]: Detected new or changed devices on smithi138 2024-04-25T11:32:00.956 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:00 smithi156 ceph-mon[33045]: Adjusting osd_memory_target on smithi138 to 4272M 2024-04-25T11:32:00.957 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:00 smithi156 ceph-mon[33045]: osdmap e68: 11 total, 11 up, 11 in 2024-04-25T11:32:00.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:00 smithi090 ceph-mon[29575]: Detected new or changed devices on smithi138 2024-04-25T11:32:00.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:00 smithi090 ceph-mon[29575]: Adjusting osd_memory_target on smithi138 to 4272M 2024-04-25T11:32:00.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:00 smithi090 ceph-mon[29575]: osdmap e68: 11 total, 11 up, 11 in 2024-04-25T11:32:01.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:00 smithi138 ceph-mon[32328]: Detected new or changed devices on smithi138 2024-04-25T11:32:01.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:00 smithi138 ceph-mon[32328]: Adjusting osd_memory_target on smithi138 to 4272M 2024-04-25T11:32:01.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:00 smithi138 ceph-mon[32328]: osdmap e68: 11 total, 11 up, 11 in 2024-04-25T11:32:01.189 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.198+0000 7f5222f00700 1 -- 172.21.15.156:0/1273485200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f521c1030e0 msgr2=0x7f521c1034f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:01.189 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.198+0000 7f5222f00700 1 --2- 172.21.15.156:0/1273485200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f521c1030e0 0x7f521c1034f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5210009990 tx=0x7f5210009ca0 comp rx=0 tx=0).stop 2024-04-25T11:32:01.189 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.198+0000 7f5222f00700 1 -- 172.21.15.156:0/1273485200 shutdown_connections 2024-04-25T11:32:01.189 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.198+0000 7f5222f00700 1 --2- 172.21.15.156:0/1273485200 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f521c104bb0 0x7f521c10b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:01.189 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.198+0000 7f5222f00700 1 --2- 172.21.15.156:0/1273485200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f521c1030e0 0x7f521c1034f0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:01.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.198+0000 7f5222f00700 1 --2- 172.21.15.156:0/1273485200 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f521c104210 0x7f521c104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:01.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.198+0000 7f5222f00700 1 -- 172.21.15.156:0/1273485200 >> 172.21.15.156:0/1273485200 conn(0x7f521c0fe740 msgr2=0x7f521c100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:01.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.198+0000 7f5222f00700 1 -- 172.21.15.156:0/1273485200 shutdown_connections 2024-04-25T11:32:01.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.198+0000 7f5222f00700 1 -- 172.21.15.156:0/1273485200 wait complete. 2024-04-25T11:32:01.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.199+0000 7f5222f00700 1 Processor -- start 2024-04-25T11:32:01.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.199+0000 7f5222f00700 1 -- start start 2024-04-25T11:32:01.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.199+0000 7f5222f00700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f521c1030e0 0x7f521c19a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:01.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.199+0000 7f5222f00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f521c104210 0x7f521c19ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:01.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.199+0000 7f5222f00700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f521c104bb0 0x7f521c1a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:01.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.199+0000 7f5222f00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f521c10fcc0 con 0x7f521c104210 2024-04-25T11:32:01.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.199+0000 7f5222f00700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f521c10fb40 con 0x7f521c1030e0 2024-04-25T11:32:01.190 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.199+0000 7f5222f00700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f521c10fe40 con 0x7f521c104bb0 2024-04-25T11:32:01.191 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.199+0000 7f5220c9c700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f521c1030e0 0x7f521c19a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:01.191 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.199+0000 7f522149d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f521c104bb0 0x7f521c1a1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:01.191 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.199+0000 7f5220c9c700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f521c1030e0 0x7f521c19a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.156:47834/0 (socket says 172.21.15.156:47834) 2024-04-25T11:32:01.191 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.199+0000 7f5220c9c700 1 -- 172.21.15.156:0/2744767188 learned_addr learned my addr 172.21.15.156:0/2744767188 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-25T11:32:01.191 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.199+0000 7f521bfff700 1 --2- 172.21.15.156:0/2744767188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f521c104210 0x7f521c19ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:01.191 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.199+0000 7f5220c9c700 1 -- 172.21.15.156:0/2744767188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f521c104bb0 msgr2=0x7f521c1a1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:01.191 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.200+0000 7f5220c9c700 1 --2- 172.21.15.156:0/2744767188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f521c104bb0 0x7f521c1a1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:01.191 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.200+0000 7f5220c9c700 1 -- 172.21.15.156:0/2744767188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f521c104210 msgr2=0x7f521c19ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:01.191 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.200+0000 7f5220c9c700 1 --2- 172.21.15.156:0/2744767188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f521c104210 0x7f521c19ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:01.191 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.200+0000 7f5220c9c700 1 -- 172.21.15.156:0/2744767188 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f521c1a2330 con 0x7f521c1030e0 2024-04-25T11:32:01.191 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.200+0000 7f5220c9c700 1 --2- 172.21.15.156:0/2744767188 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f521c1030e0 0x7f521c19a7d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f521000b6c0 tx=0x7f5210009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:01.191 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.200+0000 7f5219ffb700 1 -- 172.21.15.156:0/2744767188 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5210026070 con 0x7f521c1030e0 2024-04-25T11:32:01.191 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.200+0000 7f5222f00700 1 -- 172.21.15.156:0/2744767188 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f521c1a25c0 con 0x7f521c1030e0 2024-04-25T11:32:01.192 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.200+0000 7f5219ffb700 1 -- 172.21.15.156:0/2744767188 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5210021410 con 0x7f521c1030e0 2024-04-25T11:32:01.192 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.200+0000 7f5219ffb700 1 -- 172.21.15.156:0/2744767188 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5210017650 con 0x7f521c1030e0 2024-04-25T11:32:01.192 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.200+0000 7f5222f00700 1 -- 172.21.15.156:0/2744767188 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f521c1a2a10 con 0x7f521c1030e0 2024-04-25T11:32:01.193 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.201+0000 7f5222f00700 1 -- 172.21.15.156:0/2744767188 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f521c051190 con 0x7f521c1030e0 2024-04-25T11:32:01.193 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.203+0000 7f5219ffb700 1 -- 172.21.15.156:0/2744767188 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f5210029070 con 0x7f521c1030e0 2024-04-25T11:32:01.193 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.203+0000 7f5219ffb700 1 --2- 172.21.15.156:0/2744767188 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f520c0afb80 0x7f520c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:01.193 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.203+0000 7f5219ffb700 1 -- 172.21.15.156:0/2744767188 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6694+0+0 (secure 0 0 0) 0x7f52100df8c0 con 0x7f521c1030e0 2024-04-25T11:32:01.193 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.203+0000 7f521bfff700 1 --2- 172.21.15.156:0/2744767188 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f520c0afb80 0x7f520c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:01.193 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.204+0000 7f521bfff700 1 --2- 172.21.15.156:0/2744767188 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f520c0afb80 0x7f520c0b2090 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f521c19b880 tx=0x7f5208009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:01.195 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.207+0000 7f5219ffb700 1 -- 172.21.15.156:0/2744767188 <== mon.2 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5210030080 con 0x7f521c1030e0 2024-04-25T11:32:01.406 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:01.417+0000 7f5222f00700 1 -- 172.21.15.156:0/2744767188 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f521c063b90 con 0x7f520c0afb80 2024-04-25T11:32:01.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:01 smithi090 ceph-mon[29575]: pgmap v230: 1 pgs: 1 remapped+peering; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T11:32:01.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:01 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:32:01.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:01 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:32:01.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:01 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:02.015 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:01 smithi156 ceph-mon[33045]: pgmap v230: 1 pgs: 1 remapped+peering; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T11:32:02.015 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:01 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:32:02.015 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:01 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:32:02.015 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:01 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:02.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:01 smithi138 ceph-mon[32328]: pgmap v230: 1 pgs: 1 remapped+peering; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T11:32:02.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:01 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T11:32:02.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:01 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T11:32:02.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:01 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:02.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:02 smithi090 ceph-mon[29575]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:32:03.018 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:02 smithi156 ceph-mon[33045]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:32:03.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:02 smithi138 ceph-mon[32328]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:32:04.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:03 smithi138 ceph-mon[32328]: pgmap v231: 1 pgs: 1 remapped+peering; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T11:32:04.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:03 smithi156 ceph-mon[33045]: pgmap v231: 1 pgs: 1 remapped+peering; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T11:32:04.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:03 smithi090 ceph-mon[29575]: pgmap v231: 1 pgs: 1 remapped+peering; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T11:32:04.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:04 smithi156 ceph-mon[33045]: from='client.? 172.21.15.156:0/755573051' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51cbe729-292e-4d7b-b8e9-e1e2821ffe76"}]: dispatch 2024-04-25T11:32:04.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:04 smithi156 ceph-mon[33045]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51cbe729-292e-4d7b-b8e9-e1e2821ffe76"}]: dispatch 2024-04-25T11:32:04.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:04 smithi156 ceph-mon[33045]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51cbe729-292e-4d7b-b8e9-e1e2821ffe76"}]': finished 2024-04-25T11:32:04.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:04 smithi156 ceph-mon[33045]: osdmap e69: 12 total, 11 up, 12 in 2024-04-25T11:32:04.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:04 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T11:32:05.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:04 smithi138 ceph-mon[32328]: from='client.? 172.21.15.156:0/755573051' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51cbe729-292e-4d7b-b8e9-e1e2821ffe76"}]: dispatch 2024-04-25T11:32:05.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:04 smithi138 ceph-mon[32328]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51cbe729-292e-4d7b-b8e9-e1e2821ffe76"}]: dispatch 2024-04-25T11:32:05.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:04 smithi138 ceph-mon[32328]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51cbe729-292e-4d7b-b8e9-e1e2821ffe76"}]': finished 2024-04-25T11:32:05.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:04 smithi138 ceph-mon[32328]: osdmap e69: 12 total, 11 up, 12 in 2024-04-25T11:32:05.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:04 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T11:32:05.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:04 smithi090 ceph-mon[29575]: from='client.? 172.21.15.156:0/755573051' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51cbe729-292e-4d7b-b8e9-e1e2821ffe76"}]: dispatch 2024-04-25T11:32:05.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:04 smithi090 ceph-mon[29575]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51cbe729-292e-4d7b-b8e9-e1e2821ffe76"}]: dispatch 2024-04-25T11:32:05.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:04 smithi090 ceph-mon[29575]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51cbe729-292e-4d7b-b8e9-e1e2821ffe76"}]': finished 2024-04-25T11:32:05.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:04 smithi090 ceph-mon[29575]: osdmap e69: 12 total, 11 up, 12 in 2024-04-25T11:32:05.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:04 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T11:32:06.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:05 smithi138 ceph-mon[32328]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T11:32:06.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:05 smithi138 ceph-mon[32328]: from='client.? 172.21.15.156:0/1665115982' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:32:06.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:05 smithi156 ceph-mon[33045]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T11:32:06.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:05 smithi156 ceph-mon[33045]: from='client.? 172.21.15.156:0/1665115982' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:32:06.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:05 smithi090 ceph-mon[29575]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T11:32:06.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:05 smithi090 ceph-mon[29575]: from='client.? 172.21.15.156:0/1665115982' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T11:32:08.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:07 smithi138 ceph-mon[32328]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-25T11:32:08.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:07 smithi156 ceph-mon[33045]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-25T11:32:08.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:07 smithi090 ceph-mon[29575]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-25T11:32:09.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:09 smithi156 ceph-mon[33045]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T11:32:10.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:09 smithi138 ceph-mon[32328]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T11:32:10.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:09 smithi090 ceph-mon[29575]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T11:32:12.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:11 smithi138 ceph-mon[32328]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T11:32:12.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:11 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T11:32:12.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:11 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:12.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:11 smithi156 ceph-mon[33045]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T11:32:12.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:11 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T11:32:12.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:11 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:12.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:11 smithi090 ceph-mon[29575]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T11:32:12.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:11 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T11:32:12.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:11 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:13.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:12 smithi138 ceph-mon[32328]: Deploying daemon osd.11 on smithi156 2024-04-25T11:32:13.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:12 smithi156 ceph-mon[33045]: Deploying daemon osd.11 on smithi156 2024-04-25T11:32:13.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:12 smithi090 ceph-mon[29575]: Deploying daemon osd.11 on smithi156 2024-04-25T11:32:14.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:13 smithi138 ceph-mon[32328]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T11:32:14.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:13 smithi156 ceph-mon[33045]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T11:32:14.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:13 smithi090 ceph-mon[29575]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T11:32:16.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:15 smithi138 ceph-mon[32328]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T11:32:16.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:15 smithi156 ceph-mon[33045]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T11:32:16.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:15 smithi090 ceph-mon[29575]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T11:32:17.967 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:17 smithi156 ceph-mon[33045]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T11:32:17.967 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:17.967 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:17.967 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:32:18.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:17 smithi138 ceph-mon[32328]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T11:32:18.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:18.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:18.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:32:18.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:17 smithi090 ceph-mon[29575]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T11:32:18.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:18.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:18.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:32:20.028 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:19 smithi090 ceph-mon[29575]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T11:32:20.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:19 smithi138 ceph-mon[32328]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T11:32:20.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:19 smithi156 ceph-mon[33045]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T11:32:21.174 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:21.184+0000 7f5219ffb700 1 -- 172.21.15.156:0/2744767188 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f521c063b90 con 0x7f520c0afb80 2024-04-25T11:32:21.174 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 11 on host 'smithi156' 2024-04-25T11:32:21.176 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:21.186+0000 7f5222f00700 1 -- 172.21.15.156:0/2744767188 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f520c0afb80 msgr2=0x7f520c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:21.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:21.186+0000 7f5222f00700 1 --2- 172.21.15.156:0/2744767188 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f520c0afb80 0x7f520c0b2090 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f521c19b880 tx=0x7f5208009260 comp rx=0 tx=0).stop 2024-04-25T11:32:21.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:21.186+0000 7f5222f00700 1 -- 172.21.15.156:0/2744767188 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f521c1030e0 msgr2=0x7f521c19a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:21.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:21.186+0000 7f5222f00700 1 --2- 172.21.15.156:0/2744767188 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f521c1030e0 0x7f521c19a7d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f521000b6c0 tx=0x7f5210009580 comp rx=0 tx=0).stop 2024-04-25T11:32:21.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:21.186+0000 7f5222f00700 1 -- 172.21.15.156:0/2744767188 shutdown_connections 2024-04-25T11:32:21.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:21.186+0000 7f5222f00700 1 --2- 172.21.15.156:0/2744767188 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f520c0afb80 0x7f520c0b2090 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:21.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:21.186+0000 7f5222f00700 1 --2- 172.21.15.156:0/2744767188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f521c104bb0 0x7f521c1a1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:21.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:21.186+0000 7f5222f00700 1 --2- 172.21.15.156:0/2744767188 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f521c1030e0 0x7f521c19a7d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:21.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:21.186+0000 7f5222f00700 1 --2- 172.21.15.156:0/2744767188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f521c104210 0x7f521c19ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:21.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:21.186+0000 7f5222f00700 1 -- 172.21.15.156:0/2744767188 >> 172.21.15.156:0/2744767188 conn(0x7f521c0fe740 msgr2=0x7f521c107440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:21.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:21.187+0000 7f5222f00700 1 -- 172.21.15.156:0/2744767188 shutdown_connections 2024-04-25T11:32:21.177 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:21.187+0000 7f5222f00700 1 -- 172.21.15.156:0/2744767188 wait complete. 2024-04-25T11:32:21.472 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:21 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:21.472 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:21 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:21.472 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:21 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:21.472 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:21 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:32:21.473 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:21 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:21.473 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:21 smithi156 ceph-mon[33045]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T11:32:21.473 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:21 smithi156 ceph-mon[33045]: from='osd.11 [v2:172.21.15.156:6824/1224251744,v1:172.21.15.156:6825/1224251744]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T11:32:21.473 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:21 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:21.473 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:21 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:21.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:21 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:21.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:21 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:21.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:21 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:21.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:21 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:32:21.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:21 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:21.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:21 smithi090 ceph-mon[29575]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T11:32:21.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:21 smithi090 ceph-mon[29575]: from='osd.11 [v2:172.21.15.156:6824/1224251744,v1:172.21.15.156:6825/1224251744]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T11:32:21.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:21 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:21.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:21 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:21.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:21 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:21.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:21 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:21.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:21 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:21.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:21 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:32:21.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:21 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:21.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:21 smithi138 ceph-mon[32328]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T11:32:21.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:21 smithi138 ceph-mon[32328]: from='osd.11 [v2:172.21.15.156:6824/1224251744,v1:172.21.15.156:6825/1224251744]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T11:32:21.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:21 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:21.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:21 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:21.858 DEBUG:teuthology.orchestra.run.smithi156:osd.11> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.11.service 2024-04-25T11:32:21.860 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-25T11:32:21.861 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-25T11:32:22.098 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:32:22.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:22 smithi090 ceph-mon[29575]: from='osd.11 [v2:172.21.15.156:6824/1224251744,v1:172.21.15.156:6825/1224251744]' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T11:32:22.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:22 smithi090 ceph-mon[29575]: osdmap e70: 12 total, 11 up, 12 in 2024-04-25T11:32:22.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:22 smithi090 ceph-mon[29575]: from='osd.11 [v2:172.21.15.156:6824/1224251744,v1:172.21.15.156:6825/1224251744]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T11:32:22.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:22 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T11:32:22.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:22 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:32:22.443 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:22 smithi156 ceph-mon[33045]: from='osd.11 [v2:172.21.15.156:6824/1224251744,v1:172.21.15.156:6825/1224251744]' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T11:32:22.443 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:22 smithi156 ceph-mon[33045]: osdmap e70: 12 total, 11 up, 12 in 2024-04-25T11:32:22.443 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:22 smithi156 ceph-mon[33045]: from='osd.11 [v2:172.21.15.156:6824/1224251744,v1:172.21.15.156:6825/1224251744]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T11:32:22.443 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:22 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T11:32:22.443 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:22 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:32:22.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:22 smithi138 ceph-mon[32328]: from='osd.11 [v2:172.21.15.156:6824/1224251744,v1:172.21.15.156:6825/1224251744]' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T11:32:22.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:22 smithi138 ceph-mon[32328]: osdmap e70: 12 total, 11 up, 12 in 2024-04-25T11:32:22.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:22 smithi138 ceph-mon[32328]: from='osd.11 [v2:172.21.15.156:6824/1224251744,v1:172.21.15.156:6825/1224251744]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-25T11:32:22.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:22 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T11:32:22.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:22 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:32:23.123 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:32:22 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:32:22.849+0000 7fdc7bbe4700 -1 osd.11 0 waiting for initial osdmap 2024-04-25T11:32:23.123 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:32:22 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:32:22.865+0000 7fdc73991700 -1 osd.11 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T11:32:23.390 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:23 smithi156 ceph-mon[33045]: from='osd.11 [v2:172.21.15.156:6824/1224251744,v1:172.21.15.156:6825/1224251744]' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-25T11:32:23.390 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:23 smithi156 ceph-mon[33045]: osdmap e71: 12 total, 11 up, 12 in 2024-04-25T11:32:23.390 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:23 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T11:32:23.391 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:23 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T11:32:23.391 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:23 smithi156 ceph-mon[33045]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T11:32:23.391 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:23 smithi156 ceph-mon[33045]: from='osd.11 [v2:172.21.15.156:6824/1224251744,v1:172.21.15.156:6825/1224251744]' entity='osd.11' 2024-04-25T11:32:23.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:23 smithi090 ceph-mon[29575]: from='osd.11 [v2:172.21.15.156:6824/1224251744,v1:172.21.15.156:6825/1224251744]' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-25T11:32:23.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:23 smithi090 ceph-mon[29575]: osdmap e71: 12 total, 11 up, 12 in 2024-04-25T11:32:23.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:23 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T11:32:23.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:23 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T11:32:23.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:23 smithi090 ceph-mon[29575]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T11:32:23.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:23 smithi090 ceph-mon[29575]: from='osd.11 [v2:172.21.15.156:6824/1224251744,v1:172.21.15.156:6825/1224251744]' entity='osd.11' 2024-04-25T11:32:23.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:23 smithi138 ceph-mon[32328]: from='osd.11 [v2:172.21.15.156:6824/1224251744,v1:172.21.15.156:6825/1224251744]' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-25T11:32:23.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:23 smithi138 ceph-mon[32328]: osdmap e71: 12 total, 11 up, 12 in 2024-04-25T11:32:23.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:23 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T11:32:23.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:23 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T11:32:23.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:23 smithi138 ceph-mon[32328]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T11:32:23.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:23 smithi138 ceph-mon[32328]: from='osd.11 [v2:172.21.15.156:6824/1224251744,v1:172.21.15.156:6825/1224251744]' entity='osd.11' 2024-04-25T11:32:23.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.731+0000 7f453e2a2700 1 -- 172.21.15.90:0/765107785 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4538104230 msgr2=0x7f4538104640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:23.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.731+0000 7f453e2a2700 1 --2- 172.21.15.90:0/765107785 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4538104230 0x7f4538104640 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f452c009990 tx=0x7f452c009ca0 comp rx=0 tx=0).stop 2024-04-25T11:32:23.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.731+0000 7f453e2a2700 1 -- 172.21.15.90:0/765107785 shutdown_connections 2024-04-25T11:32:23.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.731+0000 7f453e2a2700 1 --2- 172.21.15.90:0/765107785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4538104bd0 0x7f453810b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:23.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.731+0000 7f453e2a2700 1 --2- 172.21.15.90:0/765107785 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4538103100 0x7f4538103510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:23.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.731+0000 7f453e2a2700 1 --2- 172.21.15.90:0/765107785 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4538104230 0x7f4538104640 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:23.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.731+0000 7f453e2a2700 1 -- 172.21.15.90:0/765107785 >> 172.21.15.90:0/765107785 conn(0x7f45380fe740 msgr2=0x7f4538100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:23.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.731+0000 7f453e2a2700 1 -- 172.21.15.90:0/765107785 shutdown_connections 2024-04-25T11:32:23.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.731+0000 7f453e2a2700 1 -- 172.21.15.90:0/765107785 wait complete. 2024-04-25T11:32:23.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.732+0000 7f453e2a2700 1 Processor -- start 2024-04-25T11:32:23.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.732+0000 7f453e2a2700 1 -- start start 2024-04-25T11:32:23.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.732+0000 7f453e2a2700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4538103100 0x7f45381996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:23.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.732+0000 7f453e2a2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4538104230 0x7f4538199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:23.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.732+0000 7f453e2a2700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4538104bd0 0x7f45381a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:23.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.732+0000 7f453e2a2700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f453810eb70 con 0x7f4538104230 2024-04-25T11:32:23.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.732+0000 7f453e2a2700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f453810e9f0 con 0x7f4538104bd0 2024-04-25T11:32:23.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.732+0000 7f453e2a2700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f453810ecf0 con 0x7f4538103100 2024-04-25T11:32:23.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.732+0000 7f45377fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4538104230 0x7f4538199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:23.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.732+0000 7f45377fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4538104230 0x7f4538199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34296/0 (socket says 172.21.15.90:34296) 2024-04-25T11:32:23.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.732+0000 7f45377fe700 1 -- 172.21.15.90:0/23664927 learned_addr learned my addr 172.21.15.90:0/23664927 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:32:23.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.732+0000 7f4537fff700 1 --2- 172.21.15.90:0/23664927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4538103100 0x7f45381996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:23.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.732+0000 7f453c83f700 1 --2- 172.21.15.90:0/23664927 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4538104bd0 0x7f45381a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:23.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.733+0000 7f45377fe700 1 -- 172.21.15.90:0/23664927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4538103100 msgr2=0x7f45381996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:23.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.733+0000 7f45377fe700 1 --2- 172.21.15.90:0/23664927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4538103100 0x7f45381996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:23.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.733+0000 7f45377fe700 1 -- 172.21.15.90:0/23664927 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4538104bd0 msgr2=0x7f45381a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:23.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.733+0000 7f45377fe700 1 --2- 172.21.15.90:0/23664927 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4538104bd0 0x7f45381a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:23.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.733+0000 7f45377fe700 1 -- 172.21.15.90:0/23664927 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45381a1240 con 0x7f4538104230 2024-04-25T11:32:23.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.733+0000 7f45377fe700 1 --2- 172.21.15.90:0/23664927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4538104230 0x7f4538199c20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f452c006010 tx=0x7f452c0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:23.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.733+0000 7f45357fa700 1 -- 172.21.15.90:0/23664927 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f452c027070 con 0x7f4538104230 2024-04-25T11:32:23.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.733+0000 7f453e2a2700 1 -- 172.21.15.90:0/23664927 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45380758c0 con 0x7f4538104230 2024-04-25T11:32:23.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.733+0000 7f45357fa700 1 -- 172.21.15.90:0/23664927 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f452c004000 con 0x7f4538104230 2024-04-25T11:32:23.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.733+0000 7f45357fa700 1 -- 172.21.15.90:0/23664927 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f452c022d60 con 0x7f4538104230 2024-04-25T11:32:23.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.733+0000 7f453e2a2700 1 -- 172.21.15.90:0/23664927 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4538075da0 con 0x7f4538104230 2024-04-25T11:32:23.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.734+0000 7f453e2a2700 1 -- 172.21.15.90:0/23664927 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f453810f330 con 0x7f4538104230 2024-04-25T11:32:23.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.736+0000 7f45357fa700 1 -- 172.21.15.90:0/23664927 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f452c02a070 con 0x7f4538104230 2024-04-25T11:32:23.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.737+0000 7f45357fa700 1 --2- 172.21.15.90:0/23664927 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f45200afb80 0x7f45200b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:23.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.737+0000 7f4537fff700 1 --2- 172.21.15.90:0/23664927 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f45200afb80 0x7f45200b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:23.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.737+0000 7f45357fa700 1 -- 172.21.15.90:0/23664927 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6847+0+0 (secure 0 0 0) 0x7f452c0dfe00 con 0x7f4538104230 2024-04-25T11:32:23.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.737+0000 7f4537fff700 1 --2- 172.21.15.90:0/23664927 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f45200afb80 0x7f45200b2090 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4538104130 tx=0x7f452800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:23.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.741+0000 7f45357fa700 1 -- 172.21.15.90:0/23664927 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f452c031080 con 0x7f4538104230 2024-04-25T11:32:23.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.944+0000 7f453e2a2700 1 -- 172.21.15.90:0/23664927 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f4538109d00 con 0x7f4538104230 2024-04-25T11:32:23.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.944+0000 7f45357fa700 1 -- 172.21.15.90:0/23664927 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v72) ==== 74+0+133 (secure 0 0 0) 0x7f452c013070 con 0x7f4538104230 2024-04-25T11:32:23.934 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:32:23.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.947+0000 7f453e2a2700 1 -- 172.21.15.90:0/23664927 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f45200afb80 msgr2=0x7f45200b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:23.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.947+0000 7f453e2a2700 1 --2- 172.21.15.90:0/23664927 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f45200afb80 0x7f45200b2090 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4538104130 tx=0x7f452800b410 comp rx=0 tx=0).stop 2024-04-25T11:32:23.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.947+0000 7f453e2a2700 1 -- 172.21.15.90:0/23664927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4538104230 msgr2=0x7f4538199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:23.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.947+0000 7f453e2a2700 1 --2- 172.21.15.90:0/23664927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4538104230 0x7f4538199c20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f452c006010 tx=0x7f452c0057c0 comp rx=0 tx=0).stop 2024-04-25T11:32:23.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.947+0000 7f453e2a2700 1 -- 172.21.15.90:0/23664927 shutdown_connections 2024-04-25T11:32:23.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.947+0000 7f453e2a2700 1 --2- 172.21.15.90:0/23664927 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4538104bd0 0x7f45381a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:23.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.947+0000 7f453e2a2700 1 --2- 172.21.15.90:0/23664927 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f45200afb80 0x7f45200b2090 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:23.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.947+0000 7f453e2a2700 1 --2- 172.21.15.90:0/23664927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4538103100 0x7f45381996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:23.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.947+0000 7f453e2a2700 1 --2- 172.21.15.90:0/23664927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4538104230 0x7f4538199c20 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:23.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.947+0000 7f453e2a2700 1 -- 172.21.15.90:0/23664927 >> 172.21.15.90:0/23664927 conn(0x7f45380fe740 msgr2=0x7f4538107460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:23.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.947+0000 7f453e2a2700 1 -- 172.21.15.90:0/23664927 shutdown_connections 2024-04-25T11:32:23.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:23.947+0000 7f453e2a2700 1 -- 172.21.15.90:0/23664927 wait complete. 2024-04-25T11:32:24.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:24 smithi090 ceph-mon[29575]: purged_snaps scrub starts 2024-04-25T11:32:24.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:24 smithi090 ceph-mon[29575]: purged_snaps scrub ok 2024-04-25T11:32:24.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:24 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T11:32:24.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:24 smithi090 ceph-mon[29575]: osd.11 [v2:172.21.15.156:6824/1224251744,v1:172.21.15.156:6825/1224251744] boot 2024-04-25T11:32:24.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:24 smithi090 ceph-mon[29575]: osdmap e72: 12 total, 12 up, 12 in 2024-04-25T11:32:24.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:24 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T11:32:24.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:24 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/23664927' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T11:32:24.544 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":72,"num_osds":12,"num_up_osds":12,"osd_up_since":1714044743,"num_in_osds":12,"osd_in_since":1714044724,"num_remapped_pgs":0} 2024-04-25T11:32:24.545 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T11:32:24.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:24 smithi138 ceph-mon[32328]: purged_snaps scrub starts 2024-04-25T11:32:24.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:24 smithi138 ceph-mon[32328]: purged_snaps scrub ok 2024-04-25T11:32:24.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:24 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T11:32:24.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:24 smithi138 ceph-mon[32328]: osd.11 [v2:172.21.15.156:6824/1224251744,v1:172.21.15.156:6825/1224251744] boot 2024-04-25T11:32:24.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:24 smithi138 ceph-mon[32328]: osdmap e72: 12 total, 12 up, 12 in 2024-04-25T11:32:24.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:24 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T11:32:24.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:24 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/23664927' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T11:32:24.579 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:24 smithi156 ceph-mon[33045]: purged_snaps scrub starts 2024-04-25T11:32:24.579 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:24 smithi156 ceph-mon[33045]: purged_snaps scrub ok 2024-04-25T11:32:24.580 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:24 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T11:32:24.580 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:24 smithi156 ceph-mon[33045]: osd.11 [v2:172.21.15.156:6824/1224251744,v1:172.21.15.156:6825/1224251744] boot 2024-04-25T11:32:24.580 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:24 smithi156 ceph-mon[33045]: osdmap e72: 12 total, 12 up, 12 in 2024-04-25T11:32:24.580 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:24 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T11:32:24.580 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:24 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/23664927' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T11:32:24.785 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:32:25.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:25 smithi156 ceph-mon[33045]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:25.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:25 smithi156 ceph-mon[33045]: osdmap e73: 12 total, 12 up, 12 in 2024-04-25T11:32:25.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:25 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:25.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:25 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:25.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:25 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:32:25.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:25 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:32:25.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:25 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:32:25.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:25 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:32:25.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:25 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:25.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:25 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:25.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:25 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:32:25.340 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:25 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:25.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:25 smithi090 ceph-mon[29575]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:25.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:25 smithi090 ceph-mon[29575]: osdmap e73: 12 total, 12 up, 12 in 2024-04-25T11:32:25.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:25 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:25.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:25 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:25.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:25 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:32:25.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:25 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:32:25.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:25 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:32:25.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:25 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:32:25.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:25 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:25.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:25 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:25.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:25 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:32:25.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:25 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:25.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:25 smithi138 ceph-mon[32328]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:25.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:25 smithi138 ceph-mon[32328]: osdmap e73: 12 total, 12 up, 12 in 2024-04-25T11:32:25.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:25 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:25.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:25 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:25.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:25 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:32:25.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:25 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:32:25.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:25 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:32:25.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:25 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:32:25.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:25 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:25.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:25 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:25.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:25 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:32:25.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:25 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:26.201 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:26 smithi090 ceph-mon[29575]: Detected new or changed devices on smithi156 2024-04-25T11:32:26.201 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:26 smithi090 ceph-mon[29575]: Adjusting osd_memory_target on smithi156 to 4272M 2024-04-25T11:32:26.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.426+0000 7f9982307700 1 -- 172.21.15.90:0/3971376260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f997c104480 msgr2=0x7f997c104890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:26.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.426+0000 7f9982307700 1 --2- 172.21.15.90:0/3971376260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f997c104480 0x7f997c104890 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f996c0099c0 tx=0x7f996c009cd0 comp rx=0 tx=0).stop 2024-04-25T11:32:26.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.426+0000 7f9982307700 1 -- 172.21.15.90:0/3971376260 shutdown_connections 2024-04-25T11:32:26.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.426+0000 7f9982307700 1 --2- 172.21.15.90:0/3971376260 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f997c109700 0x7f997c111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:26.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.426+0000 7f9982307700 1 --2- 172.21.15.90:0/3971376260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f997c104480 0x7f997c104890 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:26.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.426+0000 7f9982307700 1 --2- 172.21.15.90:0/3971376260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f997c108db0 0x7f997c1091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:26.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.426+0000 7f9982307700 1 -- 172.21.15.90:0/3971376260 >> 172.21.15.90:0/3971376260 conn(0x7f997c0ffd70 msgr2=0x7f997c1021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:26.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.426+0000 7f9982307700 1 -- 172.21.15.90:0/3971376260 shutdown_connections 2024-04-25T11:32:26.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.426+0000 7f9982307700 1 -- 172.21.15.90:0/3971376260 wait complete. 2024-04-25T11:32:26.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.426+0000 7f9982307700 1 Processor -- start 2024-04-25T11:32:26.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.426+0000 7f9982307700 1 -- start start 2024-04-25T11:32:26.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.427+0000 7f9982307700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f997c104480 0x7f997c1a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:26.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.427+0000 7f9982307700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f997c108db0 0x7f997c1a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:26.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.427+0000 7f9982307700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f997c109700 0x7f997c19aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:26.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.427+0000 7f9982307700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f997c114330 con 0x7f997c109700 2024-04-25T11:32:26.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.427+0000 7f9982307700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f997c1141b0 con 0x7f997c104480 2024-04-25T11:32:26.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.427+0000 7f9982307700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f997c1144b0 con 0x7f997c108db0 2024-04-25T11:32:26.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.427+0000 7f99808a4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f997c109700 0x7f997c19aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:26.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.427+0000 7f99808a4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f997c109700 0x7f997c19aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34312/0 (socket says 172.21.15.90:34312) 2024-04-25T11:32:26.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.427+0000 7f99808a4700 1 -- 172.21.15.90:0/918569351 learned_addr learned my addr 172.21.15.90:0/918569351 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:32:26.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.427+0000 7f997b7fe700 1 --2- 172.21.15.90:0/918569351 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f997c108db0 0x7f997c1a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:26.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.427+0000 7f997bfff700 1 --2- 172.21.15.90:0/918569351 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f997c104480 0x7f997c1a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:26.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.427+0000 7f99808a4700 1 -- 172.21.15.90:0/918569351 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f997c108db0 msgr2=0x7f997c1a0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:26.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.427+0000 7f99808a4700 1 --2- 172.21.15.90:0/918569351 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f997c108db0 0x7f997c1a0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:26.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.427+0000 7f99808a4700 1 -- 172.21.15.90:0/918569351 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f997c104480 msgr2=0x7f997c1a0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:26.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.427+0000 7f99808a4700 1 --2- 172.21.15.90:0/918569351 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f997c104480 0x7f997c1a0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:26.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.427+0000 7f99808a4700 1 -- 172.21.15.90:0/918569351 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f997c19b140 con 0x7f997c109700 2024-04-25T11:32:26.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.427+0000 7f99808a4700 1 --2- 172.21.15.90:0/918569351 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f997c109700 0x7f997c19aa90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9974007ee0 tx=0x7f997400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:26.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.428+0000 7f99797fa700 1 -- 172.21.15.90:0/918569351 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9974004bb0 con 0x7f997c109700 2024-04-25T11:32:26.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.428+0000 7f9982307700 1 -- 172.21.15.90:0/918569351 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f997c19b430 con 0x7f997c109700 2024-04-25T11:32:26.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.428+0000 7f99797fa700 1 -- 172.21.15.90:0/918569351 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9974004d50 con 0x7f997c109700 2024-04-25T11:32:26.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.428+0000 7f99797fa700 1 -- 172.21.15.90:0/918569351 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9974005920 con 0x7f997c109700 2024-04-25T11:32:26.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.428+0000 7f9982307700 1 -- 172.21.15.90:0/918569351 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f997c1a7700 con 0x7f997c109700 2024-04-25T11:32:26.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.429+0000 7f9982307700 1 -- 172.21.15.90:0/918569351 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f997c051190 con 0x7f997c109700 2024-04-25T11:32:26.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.430+0000 7f99797fa700 1 -- 172.21.15.90:0/918569351 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f997401a020 con 0x7f997c109700 2024-04-25T11:32:26.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.431+0000 7f99797fa700 1 --2- 172.21.15.90:0/918569351 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f99640afb80 0x7f99640b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:26.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.431+0000 7f997bfff700 1 --2- 172.21.15.90:0/918569351 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f99640afb80 0x7f99640b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:26.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.431+0000 7f99797fa700 1 -- 172.21.15.90:0/918569351 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f99740de440 con 0x7f997c109700 2024-04-25T11:32:26.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.431+0000 7f997bfff700 1 --2- 172.21.15.90:0/918569351 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f99640afb80 0x7f99640b2090 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f997c19bbd0 tx=0x7f996c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:26.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.434+0000 7f99797fa700 1 -- 172.21.15.90:0/918569351 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9974010070 con 0x7f997c109700 2024-04-25T11:32:26.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:26 smithi138 ceph-mon[32328]: Detected new or changed devices on smithi156 2024-04-25T11:32:26.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:26 smithi138 ceph-mon[32328]: Adjusting osd_memory_target on smithi156 to 4272M 2024-04-25T11:32:26.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:26 smithi156 ceph-mon[33045]: Detected new or changed devices on smithi156 2024-04-25T11:32:26.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:26 smithi156 ceph-mon[33045]: Adjusting osd_memory_target on smithi156 to 4272M 2024-04-25T11:32:26.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.632+0000 7f9982307700 1 -- 172.21.15.90:0/918569351 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f997c02e510 con 0x7f997c109700 2024-04-25T11:32:26.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.633+0000 7f99797fa700 1 -- 172.21.15.90:0/918569351 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+18475 (secure 0 0 0) 0x7f99740a74c0 con 0x7f997c109700 2024-04-25T11:32:26.624 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:32:26.624 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":73,"fsid":"58bd4e7a-02f6-11ef-bc93-c7b262605968","created":"2024-04-25T11:25:23.574550+0000","modified":"2024-04-25T11:32:24.853815+0000","last_up_change":"2024-04-25T11:32:23.852211+0000","last_in_change":"2024-04-25T11:32:04.188507+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:28:42.746649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e54d888-e67b-4951-8ae7-e355fb6b10a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6803","nonce":1201629159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6805","nonce":1201629159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6809","nonce":1201629159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6807","nonce":1201629159}]},"public_addr":"172.21.15.90:6803/1201629159","cluster_addr":"172.21.15.90:6805/1201629159","heartbeat_back_addr":"172.21.15.90:6809/1201629159","heartbeat_front_addr":"172.21.15.90:6807/1201629159","state":["exists","up"]},{"osd":1,"uuid":"91a71c78-dae9-4bc7-9f6f-9e55cfefbe28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6801","nonce":3976851933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6803","nonce":3976851933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6807","nonce":3976851933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6805","nonce":3976851933}]},"public_addr":"172.21.15.138:6801/3976851933","cluster_addr":"172.21.15.138:6803/3976851933","heartbeat_back_addr":"172.21.15.138:6807/3976851933","heartbeat_front_addr":"172.21.15.138:6805/3976851933","state":["exists","up"]},{"osd":2,"uuid":"daff6c7f-80fa-4859-9339-ef7a5e840832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6801","nonce":1505807670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6803","nonce":1505807670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6807","nonce":1505807670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6805","nonce":1505807670}]},"public_addr":"172.21.15.156:6801/1505807670","cluster_addr":"172.21.15.156:6803/1505807670","heartbeat_back_addr":"172.21.15.156:6807/1505807670","heartbeat_front_addr":"172.21.15.156:6805/1505807670","state":["exists","up"]},{"osd":3,"uuid":"33fd94ca-7465-401a-bc1e-706b6b31b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6811","nonce":2520012383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6813","nonce":2520012383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6817","nonce":2520012383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6815","nonce":2520012383}]},"public_addr":"172.21.15.90:6811/2520012383","cluster_addr":"172.21.15.90:6813/2520012383","heartbeat_back_addr":"172.21.15.90:6817/2520012383","heartbeat_front_addr":"172.21.15.90:6815/2520012383","state":["exists","up"]},{"osd":4,"uuid":"c5326212-750c-49fd-96e3-d1e62bb4b7f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6809","nonce":3752678279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6811","nonce":3752678279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6815","nonce":3752678279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6813","nonce":3752678279}]},"public_addr":"172.21.15.138:6809/3752678279","cluster_addr":"172.21.15.138:6811/3752678279","heartbeat_back_addr":"172.21.15.138:6815/3752678279","heartbeat_front_addr":"172.21.15.138:6813/3752678279","state":["exists","up"]},{"osd":5,"uuid":"d04a1156-820b-4b43-be89-e16a8910c3aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6809","nonce":3591210191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6811","nonce":3591210191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6815","nonce":3591210191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6813","nonce":3591210191}]},"public_addr":"172.21.15.156:6809/3591210191","cluster_addr":"172.21.15.156:6811/3591210191","heartbeat_back_addr":"172.21.15.156:6815/3591210191","heartbeat_front_addr":"172.21.15.156:6813/3591210191","state":["exists","up"]},{"osd":6,"uuid":"75675aa3-e978-490c-8205-13d9868a3d6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6819","nonce":2364955456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6821","nonce":2364955456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6825","nonce":2364955456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6823","nonce":2364955456}]},"public_addr":"172.21.15.90:6819/2364955456","cluster_addr":"172.21.15.90:6821/2364955456","heartbeat_back_addr":"172.21.15.90:6825/2364955456","heartbeat_front_addr":"172.21.15.90:6823/2364955456","state":["exists","up"]},{"osd":7,"uuid":"1a95fdf5-f548-4271-bbcd-e330f7955eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6817","nonce":4012476387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6819","nonce":4012476387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6823","nonce":4012476387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6821","nonce":4012476387}]},"public_addr":"172.21.15.138:6817/4012476387","cluster_addr":"172.21.15.138:6819/4012476387","heartbeat_back_addr":"172.21.15.138:6823/4012476387","heartbeat_front_addr":"172.21.15.138:6821/4012476387","state":["exists","up"]},{"osd":8,"uuid":"605e10b5-5806-4c1f-8e28-a7994d34ddce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6817","nonce":101092224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6819","nonce":101092224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6823","nonce":101092224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6821","nonce":101092224}]},"public_addr":"172.21.15.156:6817/101092224","cluster_addr":"172.21.15.156:6819/101092224","heartbeat_back_addr":"172.21.15.156:6823/101092224","heartbeat_front_addr":"172.21.15.156:6821/101092224","state":["exists","up"]},{"osd":9,"uuid":"d70fdf5e-9d0f-483d-a7d8-505b79b8a418","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6827","nonce":1171265966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6829","nonce":1171265966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6833","nonce":1171265966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6831","nonce":1171265966}]},"public_addr":"172.21.15.90:6827/1171265966","cluster_addr":"172.21.15.90:6829/1171265966","heartbeat_back_addr":"172.21.15.90:6833/1171265966","heartbeat_front_addr":"172.21.15.90:6831/1171265966","state":["exists","up"]},{"osd":10,"uuid":"1ec86c7d-c3a4-432d-9664-8e40836e1e20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6825","nonce":4264666232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6827","nonce":4264666232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6831","nonce":4264666232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6829","nonce":4264666232}]},"public_addr":"172.21.15.138:6825/4264666232","cluster_addr":"172.21.15.138:6827/4264666232","heartbeat_back_addr":"172.21.15.138:6831/4264666232","heartbeat_front_addr":"172.21.15.138:6829/4264666232","state":["exists","up"]},{"osd":11,"uuid":"51cbe729-292e-4d7b-b8e9-e1e2821ffe76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6825","nonce":1224251744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6827","nonce":1224251744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6831","nonce":1224251744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6829","nonce":1224251744}]},"public_addr":"172.21.15.156:6825/1224251744","cluster_addr":"172.21.15.156:6827/1224251744","heartbeat_back_addr":"172.21.15.156:6831/1224251744","heartbeat_front_addr":"172.21.15.156:6829/1224251744","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:27:50.794226+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:15.030278+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:38.518163+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:03.107351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:27.500707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:51.463545+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:16.015076+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:41.182071+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:05.362309+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:31.048936+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:56.488509+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:32:22.129178+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/2381633953":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6800/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/2190846900":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1707792941":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6800/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6801/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/6061052":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1849633392":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6801/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/2277490702":"2024-04-26T11:26:16.655278+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:32:26.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.635+0000 7f9982307700 1 -- 172.21.15.90:0/918569351 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f99640afb80 msgr2=0x7f99640b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:26.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.635+0000 7f9982307700 1 --2- 172.21.15.90:0/918569351 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f99640afb80 0x7f99640b2090 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f997c19bbd0 tx=0x7f996c01a040 comp rx=0 tx=0).stop 2024-04-25T11:32:26.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.635+0000 7f9982307700 1 -- 172.21.15.90:0/918569351 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f997c109700 msgr2=0x7f997c19aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:26.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.635+0000 7f9982307700 1 --2- 172.21.15.90:0/918569351 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f997c109700 0x7f997c19aa90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9974007ee0 tx=0x7f997400d3b0 comp rx=0 tx=0).stop 2024-04-25T11:32:26.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.636+0000 7f9982307700 1 -- 172.21.15.90:0/918569351 shutdown_connections 2024-04-25T11:32:26.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.636+0000 7f9982307700 1 --2- 172.21.15.90:0/918569351 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f997c109700 0x7f997c19aa90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:26.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.636+0000 7f9982307700 1 --2- 172.21.15.90:0/918569351 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f997c104480 0x7f997c1a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:26.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.636+0000 7f9982307700 1 --2- 172.21.15.90:0/918569351 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f99640afb80 0x7f99640b2090 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:26.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.636+0000 7f9982307700 1 --2- 172.21.15.90:0/918569351 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f997c108db0 0x7f997c1a0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:26.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.636+0000 7f9982307700 1 -- 172.21.15.90:0/918569351 >> 172.21.15.90:0/918569351 conn(0x7f997c0ffd70 msgr2=0x7f997c102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:26.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.636+0000 7f9982307700 1 -- 172.21.15.90:0/918569351 shutdown_connections 2024-04-25T11:32:26.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:26.636+0000 7f9982307700 1 -- 172.21.15.90:0/918569351 wait complete. 2024-04-25T11:32:27.297 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T11:28:42.746649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T11:32:27.297 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-25T11:32:27.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:27 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/918569351' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:32:27.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:27 smithi090 ceph-mon[29575]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:27.490 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:27 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/918569351' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:32:27.490 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:27 smithi156 ceph-mon[33045]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:27.529 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:32:27.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:27 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/918569351' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:32:27.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:27 smithi138 ceph-mon[32328]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:29.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.194+0000 7f896c0b3700 1 -- 172.21.15.90:0/4031802979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8964103390 msgr2=0x7f89641037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:29.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.194+0000 7f896c0b3700 1 --2- 172.21.15.90:0/4031802979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8964103390 0x7f89641037a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f89600099c0 tx=0x7f8960009cd0 comp rx=0 tx=0).stop 2024-04-25T11:32:29.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.194+0000 7f896c0b3700 1 -- 172.21.15.90:0/4031802979 shutdown_connections 2024-04-25T11:32:29.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.194+0000 7f896c0b3700 1 --2- 172.21.15.90:0/4031802979 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8964108610 0x7f896410ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:29.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.194+0000 7f896c0b3700 1 --2- 172.21.15.90:0/4031802979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8964103390 0x7f89641037a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:29.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.194+0000 7f896c0b3700 1 --2- 172.21.15.90:0/4031802979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8964107cc0 0x7f89641080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:29.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.194+0000 7f896c0b3700 1 -- 172.21.15.90:0/4031802979 >> 172.21.15.90:0/4031802979 conn(0x7f89640fecc0 msgr2=0x7f89641010e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:29.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.194+0000 7f896c0b3700 1 -- 172.21.15.90:0/4031802979 shutdown_connections 2024-04-25T11:32:29.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.194+0000 7f896c0b3700 1 -- 172.21.15.90:0/4031802979 wait complete. 2024-04-25T11:32:29.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.195+0000 7f896c0b3700 1 Processor -- start 2024-04-25T11:32:29.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.195+0000 7f896c0b3700 1 -- start start 2024-04-25T11:32:29.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.195+0000 7f896c0b3700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8964103390 0x7f896419bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:29.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.195+0000 7f896c0b3700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8964107cc0 0x7f896419c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:29.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.195+0000 7f896c0b3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8964108610 0x7f896419c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:29.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.195+0000 7f896c0b3700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8964113240 con 0x7f8964108610 2024-04-25T11:32:29.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.195+0000 7f896c0b3700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f89641130c0 con 0x7f8964103390 2024-04-25T11:32:29.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.195+0000 7f896c0b3700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f89641133c0 con 0x7f8964107cc0 2024-04-25T11:32:29.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.195+0000 7f896a650700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8964108610 0x7f896419c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:29.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.195+0000 7f896964e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8964107cc0 0x7f896419c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:29.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.195+0000 7f896a650700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8964108610 0x7f896419c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34340/0 (socket says 172.21.15.90:34340) 2024-04-25T11:32:29.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.195+0000 7f896964e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8964107cc0 0x7f896419c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:56386/0 (socket says 172.21.15.90:56386) 2024-04-25T11:32:29.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.195+0000 7f896a650700 1 -- 172.21.15.90:0/1374625662 learned_addr learned my addr 172.21.15.90:0/1374625662 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:32:29.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.195+0000 7f8969e4f700 1 --2- 172.21.15.90:0/1374625662 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8964103390 0x7f896419bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:29.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.195+0000 7f896a650700 1 -- 172.21.15.90:0/1374625662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8964107cc0 msgr2=0x7f896419c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:29.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.195+0000 7f896a650700 1 --2- 172.21.15.90:0/1374625662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8964107cc0 0x7f896419c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:29.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.195+0000 7f896a650700 1 -- 172.21.15.90:0/1374625662 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8964103390 msgr2=0x7f896419bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:29.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.195+0000 7f896a650700 1 --2- 172.21.15.90:0/1374625662 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8964103390 0x7f896419bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:29.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.195+0000 7f896a650700 1 -- 172.21.15.90:0/1374625662 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89641a6e90 con 0x7f8964108610 2024-04-25T11:32:29.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.195+0000 7f8969e4f700 1 --2- 172.21.15.90:0/1374625662 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8964103390 0x7f896419bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:32:29.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.196+0000 7f896a650700 1 --2- 172.21.15.90:0/1374625662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8964108610 0x7f896419c5e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f895c00ada0 tx=0x7f895c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:29.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.196+0000 7f895affd700 1 -- 172.21.15.90:0/1374625662 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895c017020 con 0x7f8964108610 2024-04-25T11:32:29.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.196+0000 7f895affd700 1 -- 172.21.15.90:0/1374625662 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f895c00ce30 con 0x7f8964108610 2024-04-25T11:32:29.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.196+0000 7f895affd700 1 -- 172.21.15.90:0/1374625662 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895c005350 con 0x7f8964108610 2024-04-25T11:32:29.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.196+0000 7f896c0b3700 1 -- 172.21.15.90:0/1374625662 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89641a7180 con 0x7f8964108610 2024-04-25T11:32:29.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.196+0000 7f896c0b3700 1 -- 172.21.15.90:0/1374625662 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89641a7670 con 0x7f8964108610 2024-04-25T11:32:29.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.197+0000 7f896c0b3700 1 -- 172.21.15.90:0/1374625662 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8964051190 con 0x7f8964108610 2024-04-25T11:32:29.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.198+0000 7f895affd700 1 -- 172.21.15.90:0/1374625662 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f895c00c980 con 0x7f8964108610 2024-04-25T11:32:29.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.199+0000 7f895affd700 1 --2- 172.21.15.90:0/1374625662 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f89500afb80 0x7f89500b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:29.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.199+0000 7f8969e4f700 1 --2- 172.21.15.90:0/1374625662 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f89500afb80 0x7f89500b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:29.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.199+0000 7f895affd700 1 -- 172.21.15.90:0/1374625662 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f895c0de3f0 con 0x7f8964108610 2024-04-25T11:32:29.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.200+0000 7f8969e4f700 1 --2- 172.21.15.90:0/1374625662 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f89500afb80 0x7f89500b2090 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8960009fd0 tx=0x7f896001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:29.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.203+0000 7f895affd700 1 -- 172.21.15.90:0/1374625662 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f895c021020 con 0x7f8964108610 2024-04-25T11:32:29.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.399+0000 7f896c0b3700 1 -- 172.21.15.90:0/1374625662 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8964075ed0 con 0x7f8964108610 2024-04-25T11:32:29.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.400+0000 7f895affd700 1 -- 172.21.15.90:0/1374625662 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) ==== 93+0+10 (secure 0 0 0) 0x7f895c0a7500 con 0x7f8964108610 2024-04-25T11:32:29.390 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T11:32:29.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.402+0000 7f896c0b3700 1 -- 172.21.15.90:0/1374625662 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f89500afb80 msgr2=0x7f89500b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:29.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.402+0000 7f896c0b3700 1 --2- 172.21.15.90:0/1374625662 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f89500afb80 0x7f89500b2090 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8960009fd0 tx=0x7f896001a040 comp rx=0 tx=0).stop 2024-04-25T11:32:29.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.402+0000 7f896c0b3700 1 -- 172.21.15.90:0/1374625662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8964108610 msgr2=0x7f896419c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:29.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.402+0000 7f896c0b3700 1 --2- 172.21.15.90:0/1374625662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8964108610 0x7f896419c5e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f895c00ada0 tx=0x7f895c00c330 comp rx=0 tx=0).stop 2024-04-25T11:32:29.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.402+0000 7f896c0b3700 1 -- 172.21.15.90:0/1374625662 shutdown_connections 2024-04-25T11:32:29.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.402+0000 7f896c0b3700 1 --2- 172.21.15.90:0/1374625662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8964108610 0x7f896419c5e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:29.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.402+0000 7f896c0b3700 1 --2- 172.21.15.90:0/1374625662 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8964103390 0x7f896419bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:29.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.402+0000 7f896c0b3700 1 --2- 172.21.15.90:0/1374625662 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f89500afb80 0x7f89500b2090 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:29.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.402+0000 7f896c0b3700 1 --2- 172.21.15.90:0/1374625662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8964107cc0 0x7f896419c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:29.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.402+0000 7f896c0b3700 1 -- 172.21.15.90:0/1374625662 >> 172.21.15.90:0/1374625662 conn(0x7f89640fecc0 msgr2=0x7f8964101bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:29.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.402+0000 7f896c0b3700 1 -- 172.21.15.90:0/1374625662 shutdown_connections 2024-04-25T11:32:29.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:29.402+0000 7f896c0b3700 1 -- 172.21.15.90:0/1374625662 wait complete. 2024-04-25T11:32:29.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:29 smithi090 ceph-mon[29575]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:29.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:29 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1374625662' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:32:30.007 INFO:tasks.cephadm:Adding mds.a on smithi090 2024-04-25T11:32:30.007 INFO:tasks.cephadm:Adding mds.d on smithi090 2024-04-25T11:32:30.007 INFO:tasks.cephadm:Adding mds.g on smithi090 2024-04-25T11:32:30.008 INFO:tasks.cephadm:Adding mds.j on smithi090 2024-04-25T11:32:30.008 INFO:tasks.cephadm:Adding mds.b on smithi138 2024-04-25T11:32:30.008 INFO:tasks.cephadm:Adding mds.e on smithi138 2024-04-25T11:32:30.008 INFO:tasks.cephadm:Adding mds.h on smithi138 2024-04-25T11:32:30.008 INFO:tasks.cephadm:Adding mds.k on smithi138 2024-04-25T11:32:30.008 INFO:tasks.cephadm:Adding mds.c on smithi156 2024-04-25T11:32:30.008 INFO:tasks.cephadm:Adding mds.f on smithi156 2024-04-25T11:32:30.008 INFO:tasks.cephadm:Adding mds.i on smithi156 2024-04-25T11:32:30.008 INFO:tasks.cephadm:Adding mds.l on smithi156 2024-04-25T11:32:30.008 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi090=a;smithi090=d;smithi090=g;smithi090=j;smithi138=b;smithi138=e;smithi138=h;smithi138=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l' 2024-04-25T11:32:30.036 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:29 smithi156 ceph-mon[33045]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:30.036 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:29 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1374625662' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:32:30.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:29 smithi138 ceph-mon[32328]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:30.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:29 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1374625662' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:32:30.271 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T11:32:31.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.762+0000 7f54a820d700 1 -- 172.21.15.156:0/3746108345 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54a0105ce0 msgr2=0x7f54a010c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:31.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.762+0000 7f54a820d700 1 --2- 172.21.15.156:0/3746108345 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54a0105ce0 0x7f54a010c590 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f549800b0a0 tx=0x7f549800b3b0 comp rx=0 tx=0).stop 2024-04-25T11:32:31.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.762+0000 7f54a820d700 1 -- 172.21.15.156:0/3746108345 shutdown_connections 2024-04-25T11:32:31.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.762+0000 7f54a820d700 1 --2- 172.21.15.156:0/3746108345 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54a0105ce0 0x7f54a010c590 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:31.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.762+0000 7f54a820d700 1 --2- 172.21.15.156:0/3746108345 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f54a0104210 0x7f54a0104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:31.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.762+0000 7f54a820d700 1 --2- 172.21.15.156:0/3746108345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54a0105340 0x7f54a0105750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:31.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.762+0000 7f54a820d700 1 -- 172.21.15.156:0/3746108345 >> 172.21.15.156:0/3746108345 conn(0x7f54a00ff870 msgr2=0x7f54a0101d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:31.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.762+0000 7f54a820d700 1 -- 172.21.15.156:0/3746108345 shutdown_connections 2024-04-25T11:32:31.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.762+0000 7f54a820d700 1 -- 172.21.15.156:0/3746108345 wait complete. 2024-04-25T11:32:31.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.763+0000 7f54a820d700 1 Processor -- start 2024-04-25T11:32:31.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.763+0000 7f54a820d700 1 -- start start 2024-04-25T11:32:31.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.763+0000 7f54a820d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54a0104210 0x7f54a019c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:31.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.763+0000 7f54a820d700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f54a0105340 0x7f54a019cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:31.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.763+0000 7f54a820d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54a0105ce0 0x7f54a01a3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:31.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.763+0000 7f54a820d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f54a010fc90 con 0x7f54a0105ce0 2024-04-25T11:32:31.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.763+0000 7f54a820d700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f54a010fb10 con 0x7f54a0105340 2024-04-25T11:32:31.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.763+0000 7f54a820d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f54a010fe10 con 0x7f54a0104210 2024-04-25T11:32:31.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.764+0000 7f54a5fa9700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54a0104210 0x7f54a019c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:31.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.764+0000 7f54a5fa9700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54a0104210 0x7f54a019c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:37256/0 (socket says 172.21.15.156:37256) 2024-04-25T11:32:31.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.764+0000 7f54a5fa9700 1 -- 172.21.15.156:0/2101970755 learned_addr learned my addr 172.21.15.156:0/2101970755 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-25T11:32:31.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.764+0000 7f54a57a8700 1 --2- 172.21.15.156:0/2101970755 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f54a0105340 0x7f54a019cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:31.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.764+0000 7f54a67aa700 1 --2- 172.21.15.156:0/2101970755 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54a0105ce0 0x7f54a01a3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:31.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.764+0000 7f54a5fa9700 1 -- 172.21.15.156:0/2101970755 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f54a0105340 msgr2=0x7f54a019cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:31.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.764+0000 7f54a5fa9700 1 --2- 172.21.15.156:0/2101970755 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f54a0105340 0x7f54a019cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:31.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.764+0000 7f54a5fa9700 1 -- 172.21.15.156:0/2101970755 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54a0105ce0 msgr2=0x7f54a01a3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:31.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.764+0000 7f54a5fa9700 1 --2- 172.21.15.156:0/2101970755 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54a0105ce0 0x7f54a01a3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:31.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.764+0000 7f54a5fa9700 1 -- 172.21.15.156:0/2101970755 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54a01a4450 con 0x7f54a0104210 2024-04-25T11:32:31.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.764+0000 7f54a5fa9700 1 --2- 172.21.15.156:0/2101970755 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54a0104210 0x7f54a019c980 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f549c00d950 tx=0x7f549c00dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:31.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.764+0000 7f5496ffd700 1 -- 172.21.15.156:0/2101970755 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549c017070 con 0x7f54a0104210 2024-04-25T11:32:31.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.764+0000 7f54a820d700 1 -- 172.21.15.156:0/2101970755 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54a01a4740 con 0x7f54a0104210 2024-04-25T11:32:31.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.764+0000 7f5496ffd700 1 -- 172.21.15.156:0/2101970755 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f549c009de0 con 0x7f54a0104210 2024-04-25T11:32:31.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.764+0000 7f5496ffd700 1 -- 172.21.15.156:0/2101970755 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549c00f3c0 con 0x7f54a0104210 2024-04-25T11:32:31.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.764+0000 7f54a820d700 1 -- 172.21.15.156:0/2101970755 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54a01a4c80 con 0x7f54a0104210 2024-04-25T11:32:31.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.765+0000 7f54a820d700 1 -- 172.21.15.156:0/2101970755 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54a0051190 con 0x7f54a0104210 2024-04-25T11:32:31.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.767+0000 7f5496ffd700 1 -- 172.21.15.156:0/2101970755 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f549c01a070 con 0x7f54a0104210 2024-04-25T11:32:31.758 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.767+0000 7f5496ffd700 1 --2- 172.21.15.156:0/2101970755 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f548c0afb80 0x7f548c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:31.758 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.768+0000 7f5496ffd700 1 -- 172.21.15.156:0/2101970755 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f549c0de080 con 0x7f54a0104210 2024-04-25T11:32:31.758 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.768+0000 7f54a57a8700 1 --2- 172.21.15.156:0/2101970755 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f548c0afb80 0x7f548c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:31.758 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.768+0000 7f54a57a8700 1 --2- 172.21.15.156:0/2101970755 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f548c0afb80 0x7f548c0b2090 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f54a019da30 tx=0x7f549000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:31.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.771+0000 7f5496ffd700 1 -- 172.21.15.156:0/2101970755 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f549c0a7100 con 0x7f54a0104210 2024-04-25T11:32:31.967 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.978+0000 7f54a820d700 1 -- 172.21.15.156:0/2101970755 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi090=a;smithi090=d;smithi090=g;smithi090=j;smithi138=b;smithi138=e;smithi138=h;smithi138=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l", "target": ["mon-mgr", ""]}) -- 0x7f54a00781d0 con 0x7f548c0afb80 2024-04-25T11:32:31.978 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.988+0000 7f5496ffd700 1 -- 172.21.15.156:0/2101970755 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f54a00781d0 con 0x7f548c0afb80 2024-04-25T11:32:31.978 INFO:teuthology.orchestra.run.smithi156.stdout:Scheduled mds.all update... 2024-04-25T11:32:31.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.991+0000 7f54a820d700 1 -- 172.21.15.156:0/2101970755 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f548c0afb80 msgr2=0x7f548c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:31.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.991+0000 7f54a820d700 1 --2- 172.21.15.156:0/2101970755 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f548c0afb80 0x7f548c0b2090 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f54a019da30 tx=0x7f549000a380 comp rx=0 tx=0).stop 2024-04-25T11:32:31.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.991+0000 7f54a820d700 1 -- 172.21.15.156:0/2101970755 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54a0104210 msgr2=0x7f54a019c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:31.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.991+0000 7f54a820d700 1 --2- 172.21.15.156:0/2101970755 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54a0104210 0x7f54a019c980 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f549c00d950 tx=0x7f549c00dc60 comp rx=0 tx=0).stop 2024-04-25T11:32:31.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.991+0000 7f54a820d700 1 -- 172.21.15.156:0/2101970755 shutdown_connections 2024-04-25T11:32:31.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.991+0000 7f54a820d700 1 --2- 172.21.15.156:0/2101970755 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54a0105ce0 0x7f54a01a3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:31.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.991+0000 7f54a820d700 1 --2- 172.21.15.156:0/2101970755 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f548c0afb80 0x7f548c0b2090 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:31.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.991+0000 7f54a820d700 1 --2- 172.21.15.156:0/2101970755 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54a0104210 0x7f54a019c980 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:31.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.991+0000 7f54a820d700 1 --2- 172.21.15.156:0/2101970755 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f54a0105340 0x7f54a019cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:31.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.991+0000 7f54a820d700 1 -- 172.21.15.156:0/2101970755 >> 172.21.15.156:0/2101970755 conn(0x7f54a00ff870 msgr2=0x7f54a00769b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:31.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.991+0000 7f54a820d700 1 -- 172.21.15.156:0/2101970755 shutdown_connections 2024-04-25T11:32:31.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-25T11:32:31.991+0000 7f54a820d700 1 -- 172.21.15.156:0/2101970755 wait complete. 2024-04-25T11:32:32.010 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:31 smithi156 ceph-mon[33045]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:32.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:31 smithi138 ceph-mon[32328]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:32.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:31 smithi090 ceph-mon[29575]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:32.577 DEBUG:teuthology.orchestra.run.smithi090:mds.a> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.a.service 2024-04-25T11:32:32.580 DEBUG:teuthology.orchestra.run.smithi090:mds.d> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.d.service 2024-04-25T11:32:32.623 DEBUG:teuthology.orchestra.run.smithi090:mds.g> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.g.service 2024-04-25T11:32:32.666 DEBUG:teuthology.orchestra.run.smithi090:mds.j> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.j.service 2024-04-25T11:32:32.709 DEBUG:teuthology.orchestra.run.smithi138:mds.b> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.b.service 2024-04-25T11:32:32.712 DEBUG:teuthology.orchestra.run.smithi138:mds.e> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.e.service 2024-04-25T11:32:32.755 DEBUG:teuthology.orchestra.run.smithi138:mds.h> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.h.service 2024-04-25T11:32:32.798 DEBUG:teuthology.orchestra.run.smithi138:mds.k> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.k.service 2024-04-25T11:32:32.841 DEBUG:teuthology.orchestra.run.smithi156:mds.c> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.c.service 2024-04-25T11:32:32.843 DEBUG:teuthology.orchestra.run.smithi156:mds.f> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.f.service 2024-04-25T11:32:32.887 DEBUG:teuthology.orchestra.run.smithi156:mds.i> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.i.service 2024-04-25T11:32:32.929 DEBUG:teuthology.orchestra.run.smithi156:mds.l> sudo journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.l.service 2024-04-25T11:32:32.972 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-25T11:32:32.973 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:32:33.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:32 smithi138 ceph-mon[32328]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi090=a;smithi090=d;smithi090=g;smithi090=j;smithi138=b;smithi138=e;smithi138=h;smithi138=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:32:33.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:32 smithi138 ceph-mon[32328]: Saving service mds.all spec with placement smithi090=a;smithi090=d;smithi090=g;smithi090=j;smithi138=b;smithi138=e;smithi138=h;smithi138=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l;count:12 2024-04-25T11:32:33.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:32 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:33.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:32 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:32:33.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:32 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:33.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:32 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:32:33.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:32 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:33.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:32 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:33.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:32 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:33.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:32 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:33.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:32 smithi138 ceph-mon[32328]: Deploying daemon mds.l on smithi156 2024-04-25T11:32:33.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:32 smithi138 ceph-mon[32328]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:33.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.110+0000 7fc5a1851640 1 -- 172.21.15.90:0/4275061330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc59c108920 msgr2=0x7fc59c10ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:33.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.110+0000 7fc5a1851640 1 --2- 172.21.15.90:0/4275061330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc59c108920 0x7fc59c10ad40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc594009ca0 tx=0x7fc59402a8c0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.110+0000 7fc5a1851640 1 -- 172.21.15.90:0/4275061330 shutdown_connections 2024-04-25T11:32:33.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.110+0000 7fc5a1851640 1 --2- 172.21.15.90:0/4275061330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc59c108920 0x7fc59c10ad40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.110+0000 7fc5a1851640 1 --2- 172.21.15.90:0/4275061330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc59c103c20 0x7fc59c104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.110+0000 7fc5a1851640 1 --2- 172.21.15.90:0/4275061330 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc59c1029e0 0x7fc59c1036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.110+0000 7fc5a1851640 1 -- 172.21.15.90:0/4275061330 >> 172.21.15.90:0/4275061330 conn(0x7fc59c0fdbd0 msgr2=0x7fc59c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:33.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.110+0000 7fc5a1851640 1 -- 172.21.15.90:0/4275061330 shutdown_connections 2024-04-25T11:32:33.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.110+0000 7fc5a1851640 1 -- 172.21.15.90:0/4275061330 wait complete. 2024-04-25T11:32:33.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.111+0000 7fc5a1851640 1 Processor -- start 2024-04-25T11:32:33.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.111+0000 7fc5a1851640 1 -- start start 2024-04-25T11:32:33.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.111+0000 7fc5a1851640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc59c1029e0 0x7fc59c1993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:33.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.111+0000 7fc5a1851640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc59c103c20 0x7fc59c199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:33.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.111+0000 7fc5a1851640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc59c108920 0x7fc59c1a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:33.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.111+0000 7fc5a1851640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc59c10e510 con 0x7fc59c1029e0 2024-04-25T11:32:33.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.111+0000 7fc5a1851640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc59c10e390 con 0x7fc59c103c20 2024-04-25T11:32:33.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.111+0000 7fc5a1851640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc59c10e690 con 0x7fc59c108920 2024-04-25T11:32:33.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.111+0000 7fc59affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc59c1029e0 0x7fc59c1993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:33.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.112+0000 7fc59affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc59c1029e0 0x7fc59c1993c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44704/0 (socket says 172.21.15.90:44704) 2024-04-25T11:32:33.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.112+0000 7fc59affd640 1 -- 172.21.15.90:0/2987925656 learned_addr learned my addr 172.21.15.90:0/2987925656 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:32:33.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.112+0000 7fc59b7fe640 1 --2- 172.21.15.90:0/2987925656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc59c108920 0x7fc59c1a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:33.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.112+0000 7fc59a7fc640 1 --2- 172.21.15.90:0/2987925656 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc59c103c20 0x7fc59c199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:33.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.112+0000 7fc59affd640 1 -- 172.21.15.90:0/2987925656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc59c108920 msgr2=0x7fc59c1a0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:33.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.112+0000 7fc59affd640 1 --2- 172.21.15.90:0/2987925656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc59c108920 0x7fc59c1a0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.112+0000 7fc59affd640 1 -- 172.21.15.90:0/2987925656 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc59c103c20 msgr2=0x7fc59c199900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:33.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.112+0000 7fc59affd640 1 --2- 172.21.15.90:0/2987925656 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc59c103c20 0x7fc59c199900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.112+0000 7fc59affd640 1 -- 172.21.15.90:0/2987925656 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc59c1a0f00 con 0x7fc59c1029e0 2024-04-25T11:32:33.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.112+0000 7fc59affd640 1 --2- 172.21.15.90:0/2987925656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc59c1029e0 0x7fc59c1993c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc59000eb00 tx=0x7fc590009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:33.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.112+0000 7fc5a084f640 1 -- 172.21.15.90:0/2987925656 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc59000bbd0 con 0x7fc59c1029e0 2024-04-25T11:32:33.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.112+0000 7fc5a1851640 1 -- 172.21.15.90:0/2987925656 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc59c06b250 con 0x7fc59c1029e0 2024-04-25T11:32:33.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.112+0000 7fc5a084f640 1 -- 172.21.15.90:0/2987925656 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc59000bd70 con 0x7fc59c1029e0 2024-04-25T11:32:33.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.112+0000 7fc5a084f640 1 -- 172.21.15.90:0/2987925656 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc590017410 con 0x7fc59c1029e0 2024-04-25T11:32:33.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.112+0000 7fc5a1851640 1 -- 172.21.15.90:0/2987925656 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc59c06b7a0 con 0x7fc59c1029e0 2024-04-25T11:32:33.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.113+0000 7fc5a1851640 1 -- 172.21.15.90:0/2987925656 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc59c069310 con 0x7fc59c1029e0 2024-04-25T11:32:33.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.114+0000 7fc5a084f640 1 -- 172.21.15.90:0/2987925656 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fc59001e020 con 0x7fc59c1029e0 2024-04-25T11:32:33.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.115+0000 7fc5a084f640 1 --2- 172.21.15.90:0/2987925656 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fc5840afaf0 0x7fc5840b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:33.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.115+0000 7fc59a7fc640 1 --2- 172.21.15.90:0/2987925656 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fc5840afaf0 0x7fc5840b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:33.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.115+0000 7fc5a084f640 1 -- 172.21.15.90:0/2987925656 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7fc5900e0670 con 0x7fc59c1029e0 2024-04-25T11:32:33.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.115+0000 7fc59a7fc640 1 --2- 172.21.15.90:0/2987925656 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fc5840afaf0 0x7fc5840b2000 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc59c19a460 tx=0x7fc58c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:33.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.119+0000 7fc5a084f640 1 -- 172.21.15.90:0/2987925656 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc590025070 con 0x7fc59c1029e0 2024-04-25T11:32:33.210 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:32 smithi156 ceph-mon[33045]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi090=a;smithi090=d;smithi090=g;smithi090=j;smithi138=b;smithi138=e;smithi138=h;smithi138=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:32:33.210 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:32 smithi156 ceph-mon[33045]: Saving service mds.all spec with placement smithi090=a;smithi090=d;smithi090=g;smithi090=j;smithi138=b;smithi138=e;smithi138=h;smithi138=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l;count:12 2024-04-25T11:32:33.210 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:32 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:33.210 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:32 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:32:33.210 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:32 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:33.210 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:32 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:32:33.211 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:32 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:33.211 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:32 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:33.211 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:32 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:33.211 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:32 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:33.211 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:32 smithi156 ceph-mon[33045]: Deploying daemon mds.l on smithi156 2024-04-25T11:32:33.211 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:32 smithi156 ceph-mon[33045]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:33.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.298+0000 7fc5a1851640 1 -- 172.21.15.90:0/2987925656 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc59c107460 con 0x7fc59c1029e0 2024-04-25T11:32:33.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.299+0000 7fc5a084f640 1 -- 172.21.15.90:0/2987925656 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+18475 (secure 0 0 0) 0x7fc5900a96f0 con 0x7fc59c1029e0 2024-04-25T11:32:33.289 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:32:33.289 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":73,"fsid":"58bd4e7a-02f6-11ef-bc93-c7b262605968","created":"2024-04-25T11:25:23.574550+0000","modified":"2024-04-25T11:32:24.853815+0000","last_up_change":"2024-04-25T11:32:23.852211+0000","last_in_change":"2024-04-25T11:32:04.188507+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:28:42.746649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e54d888-e67b-4951-8ae7-e355fb6b10a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6803","nonce":1201629159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6805","nonce":1201629159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6809","nonce":1201629159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6807","nonce":1201629159}]},"public_addr":"172.21.15.90:6803/1201629159","cluster_addr":"172.21.15.90:6805/1201629159","heartbeat_back_addr":"172.21.15.90:6809/1201629159","heartbeat_front_addr":"172.21.15.90:6807/1201629159","state":["exists","up"]},{"osd":1,"uuid":"91a71c78-dae9-4bc7-9f6f-9e55cfefbe28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6801","nonce":3976851933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6803","nonce":3976851933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6807","nonce":3976851933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6805","nonce":3976851933}]},"public_addr":"172.21.15.138:6801/3976851933","cluster_addr":"172.21.15.138:6803/3976851933","heartbeat_back_addr":"172.21.15.138:6807/3976851933","heartbeat_front_addr":"172.21.15.138:6805/3976851933","state":["exists","up"]},{"osd":2,"uuid":"daff6c7f-80fa-4859-9339-ef7a5e840832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6801","nonce":1505807670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6803","nonce":1505807670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6807","nonce":1505807670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6805","nonce":1505807670}]},"public_addr":"172.21.15.156:6801/1505807670","cluster_addr":"172.21.15.156:6803/1505807670","heartbeat_back_addr":"172.21.15.156:6807/1505807670","heartbeat_front_addr":"172.21.15.156:6805/1505807670","state":["exists","up"]},{"osd":3,"uuid":"33fd94ca-7465-401a-bc1e-706b6b31b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6811","nonce":2520012383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6813","nonce":2520012383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6817","nonce":2520012383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6815","nonce":2520012383}]},"public_addr":"172.21.15.90:6811/2520012383","cluster_addr":"172.21.15.90:6813/2520012383","heartbeat_back_addr":"172.21.15.90:6817/2520012383","heartbeat_front_addr":"172.21.15.90:6815/2520012383","state":["exists","up"]},{"osd":4,"uuid":"c5326212-750c-49fd-96e3-d1e62bb4b7f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6809","nonce":3752678279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6811","nonce":3752678279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6815","nonce":3752678279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6813","nonce":3752678279}]},"public_addr":"172.21.15.138:6809/3752678279","cluster_addr":"172.21.15.138:6811/3752678279","heartbeat_back_addr":"172.21.15.138:6815/3752678279","heartbeat_front_addr":"172.21.15.138:6813/3752678279","state":["exists","up"]},{"osd":5,"uuid":"d04a1156-820b-4b43-be89-e16a8910c3aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6809","nonce":3591210191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6811","nonce":3591210191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6815","nonce":3591210191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6813","nonce":3591210191}]},"public_addr":"172.21.15.156:6809/3591210191","cluster_addr":"172.21.15.156:6811/3591210191","heartbeat_back_addr":"172.21.15.156:6815/3591210191","heartbeat_front_addr":"172.21.15.156:6813/3591210191","state":["exists","up"]},{"osd":6,"uuid":"75675aa3-e978-490c-8205-13d9868a3d6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6819","nonce":2364955456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6821","nonce":2364955456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6825","nonce":2364955456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6823","nonce":2364955456}]},"public_addr":"172.21.15.90:6819/2364955456","cluster_addr":"172.21.15.90:6821/2364955456","heartbeat_back_addr":"172.21.15.90:6825/2364955456","heartbeat_front_addr":"172.21.15.90:6823/2364955456","state":["exists","up"]},{"osd":7,"uuid":"1a95fdf5-f548-4271-bbcd-e330f7955eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6817","nonce":4012476387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6819","nonce":4012476387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6823","nonce":4012476387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6821","nonce":4012476387}]},"public_addr":"172.21.15.138:6817/4012476387","cluster_addr":"172.21.15.138:6819/4012476387","heartbeat_back_addr":"172.21.15.138:6823/4012476387","heartbeat_front_addr":"172.21.15.138:6821/4012476387","state":["exists","up"]},{"osd":8,"uuid":"605e10b5-5806-4c1f-8e28-a7994d34ddce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6817","nonce":101092224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6819","nonce":101092224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6823","nonce":101092224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6821","nonce":101092224}]},"public_addr":"172.21.15.156:6817/101092224","cluster_addr":"172.21.15.156:6819/101092224","heartbeat_back_addr":"172.21.15.156:6823/101092224","heartbeat_front_addr":"172.21.15.156:6821/101092224","state":["exists","up"]},{"osd":9,"uuid":"d70fdf5e-9d0f-483d-a7d8-505b79b8a418","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6827","nonce":1171265966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6829","nonce":1171265966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6833","nonce":1171265966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6831","nonce":1171265966}]},"public_addr":"172.21.15.90:6827/1171265966","cluster_addr":"172.21.15.90:6829/1171265966","heartbeat_back_addr":"172.21.15.90:6833/1171265966","heartbeat_front_addr":"172.21.15.90:6831/1171265966","state":["exists","up"]},{"osd":10,"uuid":"1ec86c7d-c3a4-432d-9664-8e40836e1e20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6825","nonce":4264666232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6827","nonce":4264666232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6831","nonce":4264666232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6829","nonce":4264666232}]},"public_addr":"172.21.15.138:6825/4264666232","cluster_addr":"172.21.15.138:6827/4264666232","heartbeat_back_addr":"172.21.15.138:6831/4264666232","heartbeat_front_addr":"172.21.15.138:6829/4264666232","state":["exists","up"]},{"osd":11,"uuid":"51cbe729-292e-4d7b-b8e9-e1e2821ffe76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6825","nonce":1224251744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6827","nonce":1224251744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6831","nonce":1224251744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6829","nonce":1224251744}]},"public_addr":"172.21.15.156:6825/1224251744","cluster_addr":"172.21.15.156:6827/1224251744","heartbeat_back_addr":"172.21.15.156:6831/1224251744","heartbeat_front_addr":"172.21.15.156:6829/1224251744","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:27:50.794226+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:15.030278+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:38.518163+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:03.107351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:27.500707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:51.463545+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:16.015076+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:41.182071+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:05.362309+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:31.048936+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:56.488509+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:32:22.129178+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/2381633953":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6800/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/2190846900":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1707792941":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6800/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6801/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/6061052":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1849633392":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6801/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/2277490702":"2024-04-26T11:26:16.655278+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:32:33.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.301+0000 7fc5a1851640 1 -- 172.21.15.90:0/2987925656 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fc5840afaf0 msgr2=0x7fc5840b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:33.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.301+0000 7fc5a1851640 1 --2- 172.21.15.90:0/2987925656 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fc5840afaf0 0x7fc5840b2000 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc59c19a460 tx=0x7fc58c005f90 comp rx=0 tx=0).stop 2024-04-25T11:32:33.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.301+0000 7fc5a1851640 1 -- 172.21.15.90:0/2987925656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc59c1029e0 msgr2=0x7fc59c1993c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:33.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.301+0000 7fc5a1851640 1 --2- 172.21.15.90:0/2987925656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc59c1029e0 0x7fc59c1993c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc59000eb00 tx=0x7fc590009ec0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.301+0000 7fc5a1851640 1 -- 172.21.15.90:0/2987925656 shutdown_connections 2024-04-25T11:32:33.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.301+0000 7fc5a1851640 1 --2- 172.21.15.90:0/2987925656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc59c108920 0x7fc59c1a0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.301+0000 7fc5a1851640 1 --2- 172.21.15.90:0/2987925656 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc59c103c20 0x7fc59c199900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.301+0000 7fc5a1851640 1 --2- 172.21.15.90:0/2987925656 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fc5840afaf0 0x7fc5840b2000 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.301+0000 7fc5a1851640 1 --2- 172.21.15.90:0/2987925656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc59c1029e0 0x7fc59c1993c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.301+0000 7fc5a1851640 1 -- 172.21.15.90:0/2987925656 >> 172.21.15.90:0/2987925656 conn(0x7fc59c0fdbd0 msgr2=0x7fc59c104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:33.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.301+0000 7fc5a1851640 1 -- 172.21.15.90:0/2987925656 shutdown_connections 2024-04-25T11:32:33.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.301+0000 7fc5a1851640 1 -- 172.21.15.90:0/2987925656 wait complete. 2024-04-25T11:32:33.301 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:32 smithi090 ceph-mon[29575]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi090=a;smithi090=d;smithi090=g;smithi090=j;smithi138=b;smithi138=e;smithi138=h;smithi138=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:32:33.301 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:32 smithi090 ceph-mon[29575]: Saving service mds.all spec with placement smithi090=a;smithi090=d;smithi090=g;smithi090=j;smithi138=b;smithi138=e;smithi138=h;smithi138=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l;count:12 2024-04-25T11:32:33.301 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:32 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:33.301 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:32 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:32:33.301 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:32 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:33.301 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:32 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:32:33.301 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:32 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:33.301 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:32 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:33.301 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:32 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:33.301 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:32 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:33.301 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:32 smithi090 ceph-mon[29575]: Deploying daemon mds.l on smithi156 2024-04-25T11:32:33.302 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:32 smithi090 ceph-mon[29575]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:33.303 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T11:28:42.746649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T11:32:33.303 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T11:32:33.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.424+0000 7f5115692640 1 -- 172.21.15.90:0/1702026968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5110101e80 msgr2=0x7f5110102290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:33.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.424+0000 7f5115692640 1 --2- 172.21.15.90:0/1702026968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5110101e80 0x7f5110102290 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5100009900 tx=0x7f510002a410 comp rx=0 tx=0).stop 2024-04-25T11:32:33.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.424+0000 7f5115692640 1 -- 172.21.15.90:0/1702026968 shutdown_connections 2024-04-25T11:32:33.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.424+0000 7f5115692640 1 --2- 172.21.15.90:0/1702026968 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5110109780 0x7f511010bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.424+0000 7f5115692640 1 --2- 172.21.15.90:0/1702026968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5110106c10 0x7f51101090f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.424+0000 7f5115692640 1 --2- 172.21.15.90:0/1702026968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5110101e80 0x7f5110102290 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.424+0000 7f5115692640 1 -- 172.21.15.90:0/1702026968 >> 172.21.15.90:0/1702026968 conn(0x7f51100fd350 msgr2=0x7f51100ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:33.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.424+0000 7f5115692640 1 -- 172.21.15.90:0/1702026968 shutdown_connections 2024-04-25T11:32:33.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.424+0000 7f5115692640 1 -- 172.21.15.90:0/1702026968 wait complete. 2024-04-25T11:32:33.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.425+0000 7f5115692640 1 Processor -- start 2024-04-25T11:32:33.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.425+0000 7f5115692640 1 -- start start 2024-04-25T11:32:33.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.425+0000 7f5115692640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5110101e80 0x7f511019a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:33.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.425+0000 7f5115692640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5110106c10 0x7f511019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:33.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.425+0000 7f5115692640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5110109780 0x7f51101a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:33.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.425+0000 7f5115692640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f511010f5e0 con 0x7f5110106c10 2024-04-25T11:32:33.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.425+0000 7f5115692640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f511010f460 con 0x7f5110101e80 2024-04-25T11:32:33.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.425+0000 7f5115692640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f511010f760 con 0x7f5110109780 2024-04-25T11:32:33.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.425+0000 7f510e7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5110106c10 0x7f511019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:33.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.425+0000 7f510e7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5110106c10 0x7f511019a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44728/0 (socket says 172.21.15.90:44728) 2024-04-25T11:32:33.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.425+0000 7f510e7fc640 1 -- 172.21.15.90:0/1612463833 learned_addr learned my addr 172.21.15.90:0/1612463833 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:32:33.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.425+0000 7f510effd640 1 --2- 172.21.15.90:0/1612463833 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5110101e80 0x7f511019a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:33.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.425+0000 7f510f7fe640 1 --2- 172.21.15.90:0/1612463833 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5110109780 0x7f51101a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:33.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.425+0000 7f510e7fc640 1 -- 172.21.15.90:0/1612463833 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5110109780 msgr2=0x7f51101a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:33.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.425+0000 7f510e7fc640 1 --2- 172.21.15.90:0/1612463833 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5110109780 0x7f51101a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.425+0000 7f510e7fc640 1 -- 172.21.15.90:0/1612463833 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5110101e80 msgr2=0x7f511019a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:33.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.425+0000 7f510e7fc640 1 --2- 172.21.15.90:0/1612463833 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5110101e80 0x7f511019a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.425+0000 7f510e7fc640 1 -- 172.21.15.90:0/1612463833 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51101a1d50 con 0x7f5110106c10 2024-04-25T11:32:33.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.426+0000 7f510e7fc640 1 --2- 172.21.15.90:0/1612463833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5110106c10 0x7f511019a750 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f510400d770 tx=0x7f510400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:33.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.426+0000 7f50f7fff640 1 -- 172.21.15.90:0/1612463833 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5104017070 con 0x7f5110106c10 2024-04-25T11:32:33.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.426+0000 7f5115692640 1 -- 172.21.15.90:0/1612463833 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51101a1fe0 con 0x7f5110106c10 2024-04-25T11:32:33.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.426+0000 7f50f7fff640 1 -- 172.21.15.90:0/1612463833 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51040054d0 con 0x7f5110106c10 2024-04-25T11:32:33.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.426+0000 7f50f7fff640 1 -- 172.21.15.90:0/1612463833 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5104004c30 con 0x7f5110106c10 2024-04-25T11:32:33.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.426+0000 7f5115692640 1 -- 172.21.15.90:0/1612463833 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51101a24a0 con 0x7f5110106c10 2024-04-25T11:32:33.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.427+0000 7f5115692640 1 -- 172.21.15.90:0/1612463833 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f511006a160 con 0x7f5110106c10 2024-04-25T11:32:33.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.428+0000 7f50f7fff640 1 -- 172.21.15.90:0/1612463833 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f5104004e10 con 0x7f5110106c10 2024-04-25T11:32:33.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.429+0000 7f50f7fff640 1 --2- 172.21.15.90:0/1612463833 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f50f80afaf0 0x7f50f80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:33.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.429+0000 7f510effd640 1 --2- 172.21.15.90:0/1612463833 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f50f80afaf0 0x7f50f80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:33.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.429+0000 7f50f7fff640 1 -- 172.21.15.90:0/1612463833 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f51040de240 con 0x7f5110106c10 2024-04-25T11:32:33.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.429+0000 7f510effd640 1 --2- 172.21.15.90:0/1612463833 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f50f80afaf0 0x7f50f80b2000 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5100005ab0 tx=0x7f5100031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:33.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.434+0000 7f50f7fff640 1 -- 172.21.15.90:0/1612463833 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f51040a72c0 con 0x7f5110106c10 2024-04-25T11:32:33.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.610+0000 7f5115692640 1 -- 172.21.15.90:0/1612463833 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f511002db50 con 0x7f5110106c10 2024-04-25T11:32:33.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.611+0000 7f50f7fff640 1 -- 172.21.15.90:0/1612463833 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) ==== 93+0+10 (secure 0 0 0) 0x7f51040a6bc0 con 0x7f5110106c10 2024-04-25T11:32:33.600 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T11:32:33.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.612+0000 7f5115692640 1 -- 172.21.15.90:0/1612463833 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f50f80afaf0 msgr2=0x7f50f80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:33.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.612+0000 7f5115692640 1 --2- 172.21.15.90:0/1612463833 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f50f80afaf0 0x7f50f80b2000 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5100005ab0 tx=0x7f5100031400 comp rx=0 tx=0).stop 2024-04-25T11:32:33.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.612+0000 7f5115692640 1 -- 172.21.15.90:0/1612463833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5110106c10 msgr2=0x7f511019a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:33.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.612+0000 7f5115692640 1 --2- 172.21.15.90:0/1612463833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5110106c10 0x7f511019a750 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f510400d770 tx=0x7f510400dc30 comp rx=0 tx=0).stop 2024-04-25T11:32:33.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.612+0000 7f5115692640 1 -- 172.21.15.90:0/1612463833 shutdown_connections 2024-04-25T11:32:33.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.612+0000 7f5115692640 1 --2- 172.21.15.90:0/1612463833 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5110109780 0x7f51101a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.613+0000 7f5115692640 1 --2- 172.21.15.90:0/1612463833 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f50f80afaf0 0x7f50f80b2000 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.613+0000 7f5115692640 1 --2- 172.21.15.90:0/1612463833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5110106c10 0x7f511019a750 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.613+0000 7f5115692640 1 --2- 172.21.15.90:0/1612463833 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5110101e80 0x7f511019a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.613+0000 7f5115692640 1 -- 172.21.15.90:0/1612463833 >> 172.21.15.90:0/1612463833 conn(0x7f51100fd350 msgr2=0x7f51101078d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:33.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.613+0000 7f5115692640 1 -- 172.21.15.90:0/1612463833 shutdown_connections 2024-04-25T11:32:33.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.613+0000 7f5115692640 1 -- 172.21.15.90:0/1612463833 wait complete. 2024-04-25T11:32:33.613 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:33.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.731+0000 7fdc999f3640 1 -- 172.21.15.90:0/3416111180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc94109a20 msgr2=0x7fdc9410be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:33.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.731+0000 7fdc999f3640 1 --2- 172.21.15.90:0/3416111180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc94109a20 0x7fdc9410be40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c009ca0 tx=0x7fdc8c02a8c0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.731+0000 7fdc999f3640 1 -- 172.21.15.90:0/3416111180 shutdown_connections 2024-04-25T11:32:33.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.731+0000 7fdc999f3640 1 --2- 172.21.15.90:0/3416111180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc94109a20 0x7fdc9410be40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.731+0000 7fdc999f3640 1 --2- 172.21.15.90:0/3416111180 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdc941070c0 0x7fdc941094e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.731+0000 7fdc999f3640 1 --2- 172.21.15.90:0/3416111180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc94101f10 0x7fdc94106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.731+0000 7fdc999f3640 1 -- 172.21.15.90:0/3416111180 >> 172.21.15.90:0/3416111180 conn(0x7fdc940fd370 msgr2=0x7fdc940ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:33.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.731+0000 7fdc999f3640 1 -- 172.21.15.90:0/3416111180 shutdown_connections 2024-04-25T11:32:33.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.731+0000 7fdc999f3640 1 -- 172.21.15.90:0/3416111180 wait complete. 2024-04-25T11:32:33.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.732+0000 7fdc999f3640 1 Processor -- start 2024-04-25T11:32:33.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.732+0000 7fdc999f3640 1 -- start start 2024-04-25T11:32:33.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.732+0000 7fdc999f3640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdc94101f10 0x7fdc9419a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:33.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.732+0000 7fdc999f3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc941070c0 0x7fdc9419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:33.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.732+0000 7fdc999f3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc94109a20 0x7fdc941a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:33.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.732+0000 7fdc999f3640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdc9410f6a0 con 0x7fdc94109a20 2024-04-25T11:32:33.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.732+0000 7fdc999f3640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fdc9410f520 con 0x7fdc94101f10 2024-04-25T11:32:33.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.732+0000 7fdc999f3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdc9410f820 con 0x7fdc941070c0 2024-04-25T11:32:33.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.733+0000 7fdc937fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc94109a20 0x7fdc941a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:33.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.733+0000 7fdc937fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc94109a20 0x7fdc941a16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44742/0 (socket says 172.21.15.90:44742) 2024-04-25T11:32:33.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.733+0000 7fdc937fe640 1 -- 172.21.15.90:0/1563947915 learned_addr learned my addr 172.21.15.90:0/1563947915 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:32:33.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.733+0000 7fdc927fc640 1 --2- 172.21.15.90:0/1563947915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc941070c0 0x7fdc9419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:33.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.733+0000 7fdc92ffd640 1 --2- 172.21.15.90:0/1563947915 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdc94101f10 0x7fdc9419a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:33.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.733+0000 7fdc937fe640 1 -- 172.21.15.90:0/1563947915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc941070c0 msgr2=0x7fdc9419a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:33.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.733+0000 7fdc937fe640 1 --2- 172.21.15.90:0/1563947915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc941070c0 0x7fdc9419a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.733+0000 7fdc937fe640 1 -- 172.21.15.90:0/1563947915 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdc94101f10 msgr2=0x7fdc9419a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:33.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.733+0000 7fdc937fe640 1 --2- 172.21.15.90:0/1563947915 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdc94101f10 0x7fdc9419a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.733+0000 7fdc937fe640 1 -- 172.21.15.90:0/1563947915 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc941a1d60 con 0x7fdc94109a20 2024-04-25T11:32:33.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.733+0000 7fdc937fe640 1 --2- 172.21.15.90:0/1563947915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc94109a20 0x7fdc941a16f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c0097b0 tx=0x7fdc8c031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:33.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.733+0000 7fdc989f1640 1 -- 172.21.15.90:0/1563947915 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c003bf0 con 0x7fdc94109a20 2024-04-25T11:32:33.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.733+0000 7fdc989f1640 1 -- 172.21.15.90:0/1563947915 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc8c003d90 con 0x7fdc94109a20 2024-04-25T11:32:33.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.733+0000 7fdc999f3640 1 -- 172.21.15.90:0/1563947915 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc941a1ff0 con 0x7fdc94109a20 2024-04-25T11:32:33.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.733+0000 7fdc989f1640 1 -- 172.21.15.90:0/1563947915 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c02fc30 con 0x7fdc94109a20 2024-04-25T11:32:33.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.733+0000 7fdc999f3640 1 -- 172.21.15.90:0/1563947915 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc941a2450 con 0x7fdc94109a20 2024-04-25T11:32:33.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.734+0000 7fdc999f3640 1 -- 172.21.15.90:0/1563947915 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc9406a1c0 con 0x7fdc94109a20 2024-04-25T11:32:33.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.736+0000 7fdc989f1640 1 -- 172.21.15.90:0/1563947915 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fdc8c004040 con 0x7fdc94109a20 2024-04-25T11:32:33.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.736+0000 7fdc989f1640 1 --2- 172.21.15.90:0/1563947915 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fdc7c0afaf0 0x7fdc7c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:33.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.736+0000 7fdc92ffd640 1 --2- 172.21.15.90:0/1563947915 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fdc7c0afaf0 0x7fdc7c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:33.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.736+0000 7fdc989f1640 1 -- 172.21.15.90:0/1563947915 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7fdc8c1000b0 con 0x7fdc94109a20 2024-04-25T11:32:33.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.737+0000 7fdc92ffd640 1 --2- 172.21.15.90:0/1563947915 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fdc7c0afaf0 0x7fdc7c0b2000 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdc84005d00 tx=0x7fdc84005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:33.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.740+0000 7fdc989f1640 1 -- 172.21.15.90:0/1563947915 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdc9406a1c0 con 0x7fdc94109a20 2024-04-25T11:32:33.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.953+0000 7fdc999f3640 1 -- 172.21.15.90:0/1563947915 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc9402dc10 con 0x7fdc94109a20 2024-04-25T11:32:33.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.954+0000 7fdc989f1640 1 -- 172.21.15.90:0/1563947915 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7fdc8c0046a0 con 0x7fdc94109a20 2024-04-25T11:32:33.944 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:32:33.944 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-25T11:32:33.944 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 1 2024-04-25T11:32:33.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.956+0000 7fdc999f3640 1 -- 172.21.15.90:0/1563947915 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fdc7c0afaf0 msgr2=0x7fdc7c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:33.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.956+0000 7fdc999f3640 1 --2- 172.21.15.90:0/1563947915 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fdc7c0afaf0 0x7fdc7c0b2000 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdc84005d00 tx=0x7fdc84005c50 comp rx=0 tx=0).stop 2024-04-25T11:32:33.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.956+0000 7fdc999f3640 1 -- 172.21.15.90:0/1563947915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc94109a20 msgr2=0x7fdc941a16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:33.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.956+0000 7fdc999f3640 1 --2- 172.21.15.90:0/1563947915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc94109a20 0x7fdc941a16f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c0097b0 tx=0x7fdc8c031840 comp rx=0 tx=0).stop 2024-04-25T11:32:33.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.956+0000 7fdc999f3640 1 -- 172.21.15.90:0/1563947915 shutdown_connections 2024-04-25T11:32:33.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.956+0000 7fdc999f3640 1 --2- 172.21.15.90:0/1563947915 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fdc7c0afaf0 0x7fdc7c0b2000 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.956+0000 7fdc999f3640 1 --2- 172.21.15.90:0/1563947915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc94109a20 0x7fdc941a16f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.956+0000 7fdc999f3640 1 --2- 172.21.15.90:0/1563947915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc941070c0 0x7fdc9419a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.956+0000 7fdc999f3640 1 --2- 172.21.15.90:0/1563947915 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdc94101f10 0x7fdc9419a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.956+0000 7fdc999f3640 1 -- 172.21.15.90:0/1563947915 >> 172.21.15.90:0/1563947915 conn(0x7fdc940fd370 msgr2=0x7fdc94105640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:33.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.956+0000 7fdc999f3640 1 -- 172.21.15.90:0/1563947915 shutdown_connections 2024-04-25T11:32:33.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:33.956+0000 7fdc999f3640 1 -- 172.21.15.90:0/1563947915 wait complete. 2024-04-25T11:32:34.173 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:33 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2987925656' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:32:34.173 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:33 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1612463833' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:32:34.173 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:33 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1563947915' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:34.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:33 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2987925656' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:32:34.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:33 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1612463833' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:32:34.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:33 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1563947915' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:34.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:33 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2987925656' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:32:34.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:33 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1612463833' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:32:34.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:33 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1563947915' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:34.839 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 11:32:34 smithi156 systemd[1]: Starting Ceph mds.l for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:32:35.113 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:35 smithi156 ceph-mon[33045]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:35.113 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 11:32:34 smithi156 podman[63141]: 2024-04-25 11:32:34.75864055 +0000 UTC m=+0.017873876 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T11:32:35.113 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 11:32:34 smithi156 podman[63141]: 2024-04-25 11:32:34.861894006 +0000 UTC m=+0.121127361 container create f5c838c4b13542b8d220ffd7c9efca1834b0adf01b81504d03fb23a136bb63d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T11:32:35.113 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 11:32:35 smithi156 podman[63141]: 2024-04-25 11:32:35.089340488 +0000 UTC m=+0.348573801 container init f5c838c4b13542b8d220ffd7c9efca1834b0adf01b81504d03fb23a136bb63d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-l, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2) 2024-04-25T11:32:35.113 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 11:32:35 smithi156 podman[63141]: 2024-04-25 11:32:35.091853895 +0000 UTC m=+0.351087222 container start f5c838c4b13542b8d220ffd7c9efca1834b0adf01b81504d03fb23a136bb63d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-l, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-25T11:32:35.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:35 smithi138 ceph-mon[32328]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:35.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:35 smithi090 ceph-mon[29575]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:35.588 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 11:32:35 smithi156 ceph-mds[63160]: -- 172.21.15.156:0/2252556126 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5625b19aba40 con 0x5625b19f3a80 2024-04-25T11:32:35.589 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 11:32:35 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-l[63156]: starting mds.l at 2024-04-25T11:32:35.589 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 11:32:35 smithi156 bash[63141]: f5c838c4b13542b8d220ffd7c9efca1834b0adf01b81504d03fb23a136bb63d9 2024-04-25T11:32:35.589 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 11:32:35 smithi156 systemd[1]: Started Ceph mds.l for 58bd4e7a-02f6-11ef-bc93-c7b262605968. 2024-04-25T11:32:35.958 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:36.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.089+0000 7fd6b359d640 1 -- 172.21.15.90:0/2835965369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6ac102800 msgr2=0x7fd6ac103500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:36.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.089+0000 7fd6b359d640 1 --2- 172.21.15.90:0/2835965369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6ac102800 0x7fd6ac103500 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd6a8009900 tx=0x7fd6a802a410 comp rx=0 tx=0).stop 2024-04-25T11:32:36.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.090+0000 7fd6b359d640 1 -- 172.21.15.90:0/2835965369 shutdown_connections 2024-04-25T11:32:36.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.090+0000 7fd6b359d640 1 --2- 172.21.15.90:0/2835965369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6ac108740 0x7fd6ac10ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:36.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.090+0000 7fd6b359d640 1 --2- 172.21.15.90:0/2835965369 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd6ac103a40 0x7fd6ac103e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:36.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.090+0000 7fd6b359d640 1 --2- 172.21.15.90:0/2835965369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6ac102800 0x7fd6ac103500 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:36.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.090+0000 7fd6b359d640 1 -- 172.21.15.90:0/2835965369 >> 172.21.15.90:0/2835965369 conn(0x7fd6ac0fd9f0 msgr2=0x7fd6ac0ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:36.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.090+0000 7fd6b359d640 1 -- 172.21.15.90:0/2835965369 shutdown_connections 2024-04-25T11:32:36.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.090+0000 7fd6b359d640 1 -- 172.21.15.90:0/2835965369 wait complete. 2024-04-25T11:32:36.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.090+0000 7fd6b359d640 1 Processor -- start 2024-04-25T11:32:36.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.090+0000 7fd6b359d640 1 -- start start 2024-04-25T11:32:36.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.090+0000 7fd6b359d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6ac102800 0x7fd6ac06a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:36.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.090+0000 7fd6b359d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6ac103a40 0x7fd6ac19ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:36.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.090+0000 7fd6b359d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd6ac108740 0x7fd6ac1a1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:36.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.090+0000 7fd6b359d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd6ac10e300 con 0x7fd6ac103a40 2024-04-25T11:32:36.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.091+0000 7fd6b359d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd6ac10e180 con 0x7fd6ac108740 2024-04-25T11:32:36.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.091+0000 7fd6b359d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd6ac10e480 con 0x7fd6ac102800 2024-04-25T11:32:36.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.091+0000 7fd6b1d9a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6ac103a40 0x7fd6ac19ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:36.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.091+0000 7fd6b1d9a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6ac103a40 0x7fd6ac19ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44758/0 (socket says 172.21.15.90:44758) 2024-04-25T11:32:36.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.091+0000 7fd6b1d9a640 1 -- 172.21.15.90:0/2288235649 learned_addr learned my addr 172.21.15.90:0/2288235649 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:32:36.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.091+0000 7fd6b259b640 1 --2- 172.21.15.90:0/2288235649 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6ac102800 0x7fd6ac06a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:36.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.091+0000 7fd6b2d9c640 1 --2- 172.21.15.90:0/2288235649 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd6ac108740 0x7fd6ac1a1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:36.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.091+0000 7fd6b1d9a640 1 -- 172.21.15.90:0/2288235649 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6ac102800 msgr2=0x7fd6ac06a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:36.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.091+0000 7fd6b1d9a640 1 --2- 172.21.15.90:0/2288235649 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6ac102800 0x7fd6ac06a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:36.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.091+0000 7fd6b1d9a640 1 -- 172.21.15.90:0/2288235649 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd6ac108740 msgr2=0x7fd6ac1a1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:36.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.091+0000 7fd6b1d9a640 1 --2- 172.21.15.90:0/2288235649 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd6ac108740 0x7fd6ac1a1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:36.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.091+0000 7fd6b1d9a640 1 -- 172.21.15.90:0/2288235649 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6ac1a1b90 con 0x7fd6ac103a40 2024-04-25T11:32:36.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.091+0000 7fd6b2d9c640 1 --2- 172.21.15.90:0/2288235649 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd6ac108740 0x7fd6ac1a1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:32:36.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.091+0000 7fd6b1d9a640 1 --2- 172.21.15.90:0/2288235649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6ac103a40 0x7fd6ac19ee00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd69c00c890 tx=0x7fd69c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:36.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.091+0000 7fd6a37fe640 1 -- 172.21.15.90:0/2288235649 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd69c004220 con 0x7fd6ac103a40 2024-04-25T11:32:36.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.091+0000 7fd6b359d640 1 -- 172.21.15.90:0/2288235649 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6ac1a1e80 con 0x7fd6ac103a40 2024-04-25T11:32:36.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.091+0000 7fd6a37fe640 1 -- 172.21.15.90:0/2288235649 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd69c0043c0 con 0x7fd6ac103a40 2024-04-25T11:32:36.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.091+0000 7fd6a37fe640 1 -- 172.21.15.90:0/2288235649 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd69c004a00 con 0x7fd6ac103a40 2024-04-25T11:32:36.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.091+0000 7fd6b359d640 1 -- 172.21.15.90:0/2288235649 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6ac1a22f0 con 0x7fd6ac103a40 2024-04-25T11:32:36.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.092+0000 7fd6b359d640 1 -- 172.21.15.90:0/2288235649 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6ac107550 con 0x7fd6ac103a40 2024-04-25T11:32:36.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.094+0000 7fd6a37fe640 1 -- 172.21.15.90:0/2288235649 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fd69c0079b0 con 0x7fd6ac103a40 2024-04-25T11:32:36.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.094+0000 7fd6a37fe640 1 --2- 172.21.15.90:0/2288235649 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd6980afae0 0x7fd6980b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:36.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.095+0000 7fd6b259b640 1 --2- 172.21.15.90:0/2288235649 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd6980afae0 0x7fd6980b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:36.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.095+0000 7fd6a37fe640 1 -- 172.21.15.90:0/2288235649 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7fd69c0de950 con 0x7fd6ac103a40 2024-04-25T11:32:36.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.095+0000 7fd6b259b640 1 --2- 172.21.15.90:0/2288235649 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd6980afae0 0x7fd6980b1ff0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd6a8009230 tx=0x7fd6a8031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:36.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.098+0000 7fd6a37fe640 1 -- 172.21.15.90:0/2288235649 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd69c014030 con 0x7fd6ac103a40 2024-04-25T11:32:36.199 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:36 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.156:6832/31237375,v1:172.21.15.156:6833/31237375] up:boot 2024-04-25T11:32:36.200 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:36 smithi138 ceph-mon[32328]: fsmap 1 up:standby 2024-04-25T11:32:36.200 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:36 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T11:32:36.200 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:36 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:36.200 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:36 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:36.200 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:36 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:36.200 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:36 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:36.200 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:36 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:36.200 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:36 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:36.200 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:36 smithi138 ceph-mon[32328]: Deploying daemon mds.k on smithi138 2024-04-25T11:32:36.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.313+0000 7fd6b359d640 1 -- 172.21.15.90:0/2288235649 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6ac02ddc0 con 0x7fd6ac103a40 2024-04-25T11:32:36.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.314+0000 7fd6a37fe640 1 -- 172.21.15.90:0/2288235649 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1390 (secure 0 0 0) 0x7fd69c0a79d0 con 0x7fd6ac103a40 2024-04-25T11:32:36.304 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:32:36.304 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T11:32:36.304 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 2 2024-04-25T11:32:36.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.316+0000 7fd6b359d640 1 -- 172.21.15.90:0/2288235649 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd6980afae0 msgr2=0x7fd6980b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:36.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.316+0000 7fd6b359d640 1 --2- 172.21.15.90:0/2288235649 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd6980afae0 0x7fd6980b1ff0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd6a8009230 tx=0x7fd6a8031400 comp rx=0 tx=0).stop 2024-04-25T11:32:36.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.316+0000 7fd6b359d640 1 -- 172.21.15.90:0/2288235649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6ac103a40 msgr2=0x7fd6ac19ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:36.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.316+0000 7fd6b359d640 1 --2- 172.21.15.90:0/2288235649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6ac103a40 0x7fd6ac19ee00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd69c00c890 tx=0x7fd69c00cd50 comp rx=0 tx=0).stop 2024-04-25T11:32:36.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.316+0000 7fd6b359d640 1 -- 172.21.15.90:0/2288235649 shutdown_connections 2024-04-25T11:32:36.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.316+0000 7fd6b359d640 1 --2- 172.21.15.90:0/2288235649 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd6ac108740 0x7fd6ac1a1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:36.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.316+0000 7fd6b359d640 1 --2- 172.21.15.90:0/2288235649 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd6980afae0 0x7fd6980b1ff0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:36.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.316+0000 7fd6b359d640 1 --2- 172.21.15.90:0/2288235649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6ac103a40 0x7fd6ac19ee00 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:36.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.316+0000 7fd6b359d640 1 --2- 172.21.15.90:0/2288235649 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6ac102800 0x7fd6ac06a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:36.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.316+0000 7fd6b359d640 1 -- 172.21.15.90:0/2288235649 >> 172.21.15.90:0/2288235649 conn(0x7fd6ac0fd9f0 msgr2=0x7fd6ac104720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:36.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.316+0000 7fd6b359d640 1 -- 172.21.15.90:0/2288235649 shutdown_connections 2024-04-25T11:32:36.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:36.316+0000 7fd6b359d640 1 -- 172.21.15.90:0/2288235649 wait complete. 2024-04-25T11:32:36.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:36 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.156:6832/31237375,v1:172.21.15.156:6833/31237375] up:boot 2024-04-25T11:32:36.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:36 smithi090 ceph-mon[29575]: fsmap 1 up:standby 2024-04-25T11:32:36.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:36 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T11:32:36.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:36 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:36.316 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:36 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:36.317 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:36 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:36.317 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:36 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:36.317 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:36 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:36.317 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:36 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:36.317 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:36 smithi090 ceph-mon[29575]: Deploying daemon mds.k on smithi138 2024-04-25T11:32:36.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:36 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.156:6832/31237375,v1:172.21.15.156:6833/31237375] up:boot 2024-04-25T11:32:36.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:36 smithi156 ceph-mon[33045]: fsmap 1 up:standby 2024-04-25T11:32:36.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:36 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T11:32:36.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:36 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:36.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:36 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:36.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:36 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:36.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:36 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:36.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:36 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:36.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:36 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:36.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:36 smithi156 ceph-mon[33045]: Deploying daemon mds.k on smithi138 2024-04-25T11:32:37.218 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:37 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2288235649' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:37.218 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:37 smithi138 ceph-mon[32328]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:37.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:37 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2288235649' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:37.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:37 smithi090 ceph-mon[29575]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:37.553 INFO:journalctl@ceph.mds.k.smithi138.stdout:Apr 25 11:32:37 smithi138 systemd[1]: Starting Ceph mds.k for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:32:37.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:37 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2288235649' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:37.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:37 smithi156 ceph-mon[33045]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:37.886 INFO:journalctl@ceph.mds.k.smithi138.stdout:Apr 25 11:32:37 smithi138 podman[61611]: 2024-04-25 11:32:37.521225782 +0000 UTC m=+0.017850043 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T11:32:37.887 INFO:journalctl@ceph.mds.k.smithi138.stdout:Apr 25 11:32:37 smithi138 podman[61611]: 2024-04-25 11:32:37.626294395 +0000 UTC m=+0.122918671 container create 745d946e79eaa9572a961b40968828843f4cff28881b93ee28a19a53e1f5c871 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-k, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-25T11:32:37.887 INFO:journalctl@ceph.mds.k.smithi138.stdout:Apr 25 11:32:37 smithi138 podman[61611]: 2024-04-25 11:32:37.862177322 +0000 UTC m=+0.358801558 container init 745d946e79eaa9572a961b40968828843f4cff28881b93ee28a19a53e1f5c871 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-04-25T11:32:37.887 INFO:journalctl@ceph.mds.k.smithi138.stdout:Apr 25 11:32:37 smithi138 podman[61611]: 2024-04-25 11:32:37.86471059 +0000 UTC m=+0.361334829 container start 745d946e79eaa9572a961b40968828843f4cff28881b93ee28a19a53e1f5c871 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-k, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-25T11:32:38.303 INFO:journalctl@ceph.mds.k.smithi138.stdout:Apr 25 11:32:37 smithi138 ceph-mds[61630]: -- 172.21.15.138:0/4115325034 <== mon.2 v2:172.21.15.138:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x565006329a40 con 0x565006371600 2024-04-25T11:32:38.303 INFO:journalctl@ceph.mds.k.smithi138.stdout:Apr 25 11:32:37 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-k[61626]: starting mds.k at 2024-04-25T11:32:38.303 INFO:journalctl@ceph.mds.k.smithi138.stdout:Apr 25 11:32:37 smithi138 bash[61611]: 745d946e79eaa9572a961b40968828843f4cff28881b93ee28a19a53e1f5c871 2024-04-25T11:32:38.303 INFO:journalctl@ceph.mds.k.smithi138.stdout:Apr 25 11:32:37 smithi138 systemd[1]: Started Ceph mds.k for 58bd4e7a-02f6-11ef-bc93-c7b262605968. 2024-04-25T11:32:38.319 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:38.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.440+0000 7f3ee039d640 1 -- 172.21.15.90:0/3647823364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ed81044d0 msgr2=0x7f3ed810ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:38.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.440+0000 7f3ee039d640 1 --2- 172.21.15.90:0/3647823364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ed81044d0 0x7f3ed810ac10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0009ca0 tx=0x7f3ed002a8c0 comp rx=0 tx=0).stop 2024-04-25T11:32:38.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.440+0000 7f3ee039d640 1 -- 172.21.15.90:0/3647823364 shutdown_connections 2024-04-25T11:32:38.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.440+0000 7f3ee039d640 1 --2- 172.21.15.90:0/3647823364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ed81044d0 0x7f3ed810ac10 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:38.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.440+0000 7f3ee039d640 1 --2- 172.21.15.90:0/3647823364 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3ed8103990 0x7f3ed8103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:38.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.440+0000 7f3ee039d640 1 --2- 172.21.15.90:0/3647823364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ed8102fb0 0x7f3ed81033c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:38.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.440+0000 7f3ee039d640 1 -- 172.21.15.90:0/3647823364 >> 172.21.15.90:0/3647823364 conn(0x7f3ed80fd3b0 msgr2=0x7f3ed80ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:38.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.440+0000 7f3ee039d640 1 -- 172.21.15.90:0/3647823364 shutdown_connections 2024-04-25T11:32:38.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.440+0000 7f3ee039d640 1 -- 172.21.15.90:0/3647823364 wait complete. 2024-04-25T11:32:38.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.440+0000 7f3ee039d640 1 Processor -- start 2024-04-25T11:32:38.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.441+0000 7f3ee039d640 1 -- start start 2024-04-25T11:32:38.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.441+0000 7f3ee039d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3ed8102fb0 0x7f3ed81065a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:38.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.441+0000 7f3ee039d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ed8103990 0x7f3ed8104c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:38.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.441+0000 7f3ee039d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ed81044d0 0x7f3ed8105180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:38.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.441+0000 7f3ee039d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed810e510 con 0x7f3ed8103990 2024-04-25T11:32:38.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.441+0000 7f3ee039d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed810e390 con 0x7f3ed8102fb0 2024-04-25T11:32:38.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.441+0000 7f3ee039d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed810e690 con 0x7f3ed81044d0 2024-04-25T11:32:38.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.441+0000 7f3edd938640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ed8103990 0x7f3ed8104c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:38.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.441+0000 7f3edd938640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ed8103990 0x7f3ed8104c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44778/0 (socket says 172.21.15.90:44778) 2024-04-25T11:32:38.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.441+0000 7f3edd938640 1 -- 172.21.15.90:0/2097131001 learned_addr learned my addr 172.21.15.90:0/2097131001 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:32:38.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.441+0000 7f3ede93a640 1 --2- 172.21.15.90:0/2097131001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ed81044d0 0x7f3ed8105180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:38.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.441+0000 7f3ede139640 1 --2- 172.21.15.90:0/2097131001 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3ed8102fb0 0x7f3ed81065a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:38.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.441+0000 7f3edd938640 1 -- 172.21.15.90:0/2097131001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ed81044d0 msgr2=0x7f3ed8105180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.441+0000 7f3edd938640 1 --2- 172.21.15.90:0/2097131001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ed81044d0 0x7f3ed8105180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.441+0000 7f3edd938640 1 -- 172.21.15.90:0/2097131001 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3ed8102fb0 msgr2=0x7f3ed81065a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.441+0000 7f3edd938640 1 --2- 172.21.15.90:0/2097131001 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3ed8102fb0 0x7f3ed81065a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.441+0000 7f3edd938640 1 -- 172.21.15.90:0/2097131001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed8105820 con 0x7f3ed8103990 2024-04-25T11:32:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.441+0000 7f3ede93a640 1 --2- 172.21.15.90:0/2097131001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ed81044d0 0x7f3ed8105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:32:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.441+0000 7f3edd938640 1 --2- 172.21.15.90:0/2097131001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ed8103990 0x7f3ed8104c40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3ed400c920 tx=0x7f3ed400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.441+0000 7f3ecf7fe640 1 -- 172.21.15.90:0/2097131001 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed4007c70 con 0x7f3ed8103990 2024-04-25T11:32:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.441+0000 7f3ecf7fe640 1 -- 172.21.15.90:0/2097131001 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ed4007e10 con 0x7f3ed8103990 2024-04-25T11:32:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.441+0000 7f3ee039d640 1 -- 172.21.15.90:0/2097131001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ed81a4250 con 0x7f3ed8103990 2024-04-25T11:32:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.441+0000 7f3ecf7fe640 1 -- 172.21.15.90:0/2097131001 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed4018820 con 0x7f3ed8103990 2024-04-25T11:32:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.442+0000 7f3ee039d640 1 -- 172.21.15.90:0/2097131001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ed81a4660 con 0x7f3ed8103990 2024-04-25T11:32:38.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.443+0000 7f3ee039d640 1 -- 172.21.15.90:0/2097131001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ed81a4920 con 0x7f3ed8103990 2024-04-25T11:32:38.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.444+0000 7f3ecf7fe640 1 -- 172.21.15.90:0/2097131001 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f3ed400f450 con 0x7f3ed8103990 2024-04-25T11:32:38.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.445+0000 7f3ecf7fe640 1 --2- 172.21.15.90:0/2097131001 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f3ec40afaf0 0x7f3ec40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:38.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.445+0000 7f3ede139640 1 --2- 172.21.15.90:0/2097131001 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f3ec40afaf0 0x7f3ec40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:38.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.445+0000 7f3ecf7fe640 1 -- 172.21.15.90:0/2097131001 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f3ed40de8f0 con 0x7f3ed8103990 2024-04-25T11:32:38.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.445+0000 7f3ede139640 1 --2- 172.21.15.90:0/2097131001 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f3ec40afaf0 0x7f3ec40b2000 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8005fd0 tx=0x7f3ec8005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:38.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.449+0000 7f3ecf7fe640 1 -- 172.21.15.90:0/2097131001 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3ed4014030 con 0x7f3ed8103990 2024-04-25T11:32:38.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.684+0000 7f3ee039d640 1 -- 172.21.15.90:0/2097131001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ed81a4af0 con 0x7f3ed8103990 2024-04-25T11:32:38.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.685+0000 7f3ecf7fe640 1 -- 172.21.15.90:0/2097131001 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2212 (secure 0 0 0) 0x7f3ed40a7970 con 0x7f3ed8103990 2024-04-25T11:32:38.675 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:32:38.675 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T11:32:38.675 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 3 2024-04-25T11:32:38.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.687+0000 7f3ee039d640 1 -- 172.21.15.90:0/2097131001 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f3ec40afaf0 msgr2=0x7f3ec40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:38.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.687+0000 7f3ee039d640 1 --2- 172.21.15.90:0/2097131001 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f3ec40afaf0 0x7f3ec40b2000 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8005fd0 tx=0x7f3ec8005c70 comp rx=0 tx=0).stop 2024-04-25T11:32:38.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.687+0000 7f3ee039d640 1 -- 172.21.15.90:0/2097131001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ed8103990 msgr2=0x7f3ed8104c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:38.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.687+0000 7f3ee039d640 1 --2- 172.21.15.90:0/2097131001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ed8103990 0x7f3ed8104c40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3ed400c920 tx=0x7f3ed400cde0 comp rx=0 tx=0).stop 2024-04-25T11:32:38.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.687+0000 7f3ee039d640 1 -- 172.21.15.90:0/2097131001 shutdown_connections 2024-04-25T11:32:38.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.687+0000 7f3ee039d640 1 --2- 172.21.15.90:0/2097131001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ed81044d0 0x7f3ed8105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:38.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.687+0000 7f3ee039d640 1 --2- 172.21.15.90:0/2097131001 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f3ec40afaf0 0x7f3ec40b2000 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:38.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.687+0000 7f3ee039d640 1 --2- 172.21.15.90:0/2097131001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ed8103990 0x7f3ed8104c40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:38.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.687+0000 7f3ee039d640 1 --2- 172.21.15.90:0/2097131001 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3ed8102fb0 0x7f3ed81065a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:38.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.687+0000 7f3ee039d640 1 -- 172.21.15.90:0/2097131001 >> 172.21.15.90:0/2097131001 conn(0x7f3ed80fd3b0 msgr2=0x7f3ed80fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:38.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.687+0000 7f3ee039d640 1 -- 172.21.15.90:0/2097131001 shutdown_connections 2024-04-25T11:32:38.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:38.687+0000 7f3ee039d640 1 -- 172.21.15.90:0/2097131001 wait complete. 2024-04-25T11:32:39.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:38 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:39.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:38 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:39.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:38 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:39.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:38 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:39.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:38 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:39.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:38 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:39.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:38 smithi138 ceph-mon[32328]: Deploying daemon mds.j on smithi090 2024-04-25T11:32:39.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:38 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] up:boot 2024-04-25T11:32:39.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:38 smithi138 ceph-mon[32328]: fsmap 2 up:standby 2024-04-25T11:32:39.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:38 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T11:32:39.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:38 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2097131001' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:39.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:38 smithi138 ceph-mon[32328]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:39.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:38 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:39.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:38 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:39.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:38 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:39.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:38 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:39.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:38 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:39.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:38 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:39.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:38 smithi156 ceph-mon[33045]: Deploying daemon mds.j on smithi090 2024-04-25T11:32:39.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:38 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] up:boot 2024-04-25T11:32:39.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:38 smithi156 ceph-mon[33045]: fsmap 2 up:standby 2024-04-25T11:32:39.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:38 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T11:32:39.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:38 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2097131001' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:39.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:38 smithi156 ceph-mon[33045]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:39.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:38 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:39.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:38 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:39.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:38 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:39.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:38 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:39.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:38 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:39.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:38 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:39.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:38 smithi090 ceph-mon[29575]: Deploying daemon mds.j on smithi090 2024-04-25T11:32:39.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:38 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] up:boot 2024-04-25T11:32:39.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:38 smithi090 ceph-mon[29575]: fsmap 2 up:standby 2024-04-25T11:32:39.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:38 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T11:32:39.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:38 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2097131001' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:39.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:38 smithi090 ceph-mon[29575]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:40.397 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 25 11:32:40 smithi090 systemd[1]: Starting Ceph mds.j for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:32:40.689 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:40.693 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 25 11:32:40 smithi090 podman[69997]: 2024-04-25 11:32:40.311096858 +0000 UTC m=+0.017779981 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T11:32:40.693 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 25 11:32:40 smithi090 podman[69997]: 2024-04-25 11:32:40.410226729 +0000 UTC m=+0.116909867 container create 4de44129c29a78115e9f372c5e1057ac9a7d8197ca9e0f2ca8bbfb5e5dcd3f35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-j, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T11:32:40.693 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 25 11:32:40 smithi090 podman[69997]: 2024-04-25 11:32:40.613155058 +0000 UTC m=+0.319838157 container init 4de44129c29a78115e9f372c5e1057ac9a7d8197ca9e0f2ca8bbfb5e5dcd3f35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-j, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-25T11:32:40.693 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 25 11:32:40 smithi090 podman[69997]: 2024-04-25 11:32:40.615630121 +0000 UTC m=+0.322313223 container start 4de44129c29a78115e9f372c5e1057ac9a7d8197ca9e0f2ca8bbfb5e5dcd3f35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-j, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-25T11:32:40.694 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 25 11:32:40 smithi090 ceph-mds[70016]: -- 172.21.15.90:0/144690473 <== mon.0 v2:172.21.15.90:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d503793a40 con 0x55d5037dba80 2024-04-25T11:32:40.694 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 25 11:32:40 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-j[70012]: starting mds.j at 2024-04-25T11:32:40.694 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 25 11:32:40 smithi090 bash[69997]: 4de44129c29a78115e9f372c5e1057ac9a7d8197ca9e0f2ca8bbfb5e5dcd3f35 2024-04-25T11:32:40.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.820+0000 7f8aea59d640 1 -- 172.21.15.90:0/4048183605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ae41095a0 msgr2=0x7f8ae410bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:40.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.820+0000 7f8aea59d640 1 --2- 172.21.15.90:0/4048183605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ae41095a0 0x7f8ae410bab0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8adc009ca0 tx=0x7f8adc02a8c0 comp rx=0 tx=0).stop 2024-04-25T11:32:40.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.820+0000 7f8aea59d640 1 -- 172.21.15.90:0/4048183605 shutdown_connections 2024-04-25T11:32:40.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.820+0000 7f8aea59d640 1 --2- 172.21.15.90:0/4048183605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ae41095a0 0x7f8ae410bab0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:40.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.820+0000 7f8aea59d640 1 --2- 172.21.15.90:0/4048183605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ae4106a30 0x7f8ae4108f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:40.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.820+0000 7f8aea59d640 1 --2- 172.21.15.90:0/4048183605 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ae4101c50 0x7f8ae4102060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:40.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.820+0000 7f8aea59d640 1 -- 172.21.15.90:0/4048183605 >> 172.21.15.90:0/4048183605 conn(0x7f8ae40fd1c0 msgr2=0x7f8ae40ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:40.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.820+0000 7f8aea59d640 1 -- 172.21.15.90:0/4048183605 shutdown_connections 2024-04-25T11:32:40.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.820+0000 7f8aea59d640 1 -- 172.21.15.90:0/4048183605 wait complete. 2024-04-25T11:32:40.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.821+0000 7f8aea59d640 1 Processor -- start 2024-04-25T11:32:40.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.821+0000 7f8aea59d640 1 -- start start 2024-04-25T11:32:40.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.821+0000 7f8aea59d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ae4101c50 0x7f8ae4104120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:40.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.821+0000 7f8aea59d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ae4106a30 0x7f8ae41027c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:40.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.821+0000 7f8aea59d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ae41095a0 0x7f8ae4102d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:40.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.821+0000 7f8aea59d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae410f410 con 0x7f8ae41095a0 2024-04-25T11:32:40.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.821+0000 7f8aea59d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae410f290 con 0x7f8ae4106a30 2024-04-25T11:32:40.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.821+0000 7f8aea59d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae410f590 con 0x7f8ae4101c50 2024-04-25T11:32:40.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.821+0000 7f8ae9d9c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ae41095a0 0x7f8ae4102d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:40.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.821+0000 7f8ae9d9c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ae41095a0 0x7f8ae4102d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44818/0 (socket says 172.21.15.90:44818) 2024-04-25T11:32:40.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.821+0000 7f8ae9d9c640 1 -- 172.21.15.90:0/1919550078 learned_addr learned my addr 172.21.15.90:0/1919550078 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:32:40.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.821+0000 7f8ae959b640 1 --2- 172.21.15.90:0/1919550078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ae4101c50 0x7f8ae4104120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:40.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.822+0000 7f8ae8d9a640 1 --2- 172.21.15.90:0/1919550078 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ae4106a30 0x7f8ae41027c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:40.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.822+0000 7f8ae9d9c640 1 -- 172.21.15.90:0/1919550078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ae4101c50 msgr2=0x7f8ae4104120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:40.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.822+0000 7f8ae9d9c640 1 --2- 172.21.15.90:0/1919550078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ae4101c50 0x7f8ae4104120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:40.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.822+0000 7f8ae9d9c640 1 -- 172.21.15.90:0/1919550078 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ae4106a30 msgr2=0x7f8ae41027c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:40.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.822+0000 7f8ae9d9c640 1 --2- 172.21.15.90:0/1919550078 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ae4106a30 0x7f8ae41027c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:40.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.822+0000 7f8ae9d9c640 1 -- 172.21.15.90:0/1919550078 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ae4103430 con 0x7f8ae41095a0 2024-04-25T11:32:40.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.822+0000 7f8ae8d9a640 1 --2- 172.21.15.90:0/1919550078 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ae4106a30 0x7f8ae41027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:32:40.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.822+0000 7f8ae9d9c640 1 --2- 172.21.15.90:0/1919550078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ae41095a0 0x7f8ae4102d00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8adc0097b0 tx=0x7f8adc003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:40.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.822+0000 7f8ada7fc640 1 -- 172.21.15.90:0/1919550078 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8adc004240 con 0x7f8ae41095a0 2024-04-25T11:32:40.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.822+0000 7f8aea59d640 1 -- 172.21.15.90:0/1919550078 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ae41a4010 con 0x7f8ae41095a0 2024-04-25T11:32:40.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.822+0000 7f8ada7fc640 1 -- 172.21.15.90:0/1919550078 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8adc02f380 con 0x7f8ae41095a0 2024-04-25T11:32:40.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.822+0000 7f8ada7fc640 1 -- 172.21.15.90:0/1919550078 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8adc02fc30 con 0x7f8ae41095a0 2024-04-25T11:32:40.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.822+0000 7f8aea59d640 1 -- 172.21.15.90:0/1919550078 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ae41a44d0 con 0x7f8ae41095a0 2024-04-25T11:32:40.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.823+0000 7f8aea59d640 1 -- 172.21.15.90:0/1919550078 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ae40691b0 con 0x7f8ae41095a0 2024-04-25T11:32:40.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.825+0000 7f8ada7fc640 1 -- 172.21.15.90:0/1919550078 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f8adc0043e0 con 0x7f8ae41095a0 2024-04-25T11:32:40.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.825+0000 7f8ada7fc640 1 --2- 172.21.15.90:0/1919550078 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f8ad00afae0 0x7f8ad00b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:40.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.825+0000 7f8ae959b640 1 --2- 172.21.15.90:0/1919550078 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f8ad00afae0 0x7f8ad00b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:40.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.825+0000 7f8ada7fc640 1 -- 172.21.15.90:0/1919550078 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f8adc1001c0 con 0x7f8ae41095a0 2024-04-25T11:32:40.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.826+0000 7f8ae959b640 1 --2- 172.21.15.90:0/1919550078 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f8ad00afae0 0x7f8ad00b1ff0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0005fd0 tx=0x7f8ae0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:40.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:40.829+0000 7f8ada7fc640 1 -- 172.21.15.90:0/1919550078 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8adc0c91a0 con 0x7f8ae41095a0 2024-04-25T11:32:40.976 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 25 11:32:40 smithi090 systemd[1]: Started Ceph mds.j for 58bd4e7a-02f6-11ef-bc93-c7b262605968. 2024-04-25T11:32:41.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:41.051+0000 7f8aea59d640 1 -- 172.21.15.90:0/1919550078 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ae40fece0 con 0x7f8ae41095a0 2024-04-25T11:32:41.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:41.051+0000 7f8ada7fc640 1 -- 172.21.15.90:0/1919550078 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3037 (secure 0 0 0) 0x7f8adc02fe90 con 0x7f8ae41095a0 2024-04-25T11:32:41.041 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:32:41.041 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T11:32:41.041 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 4 2024-04-25T11:32:41.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:41.053+0000 7f8aea59d640 1 -- 172.21.15.90:0/1919550078 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f8ad00afae0 msgr2=0x7f8ad00b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:41.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:41.053+0000 7f8aea59d640 1 --2- 172.21.15.90:0/1919550078 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f8ad00afae0 0x7f8ad00b1ff0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0005fd0 tx=0x7f8ae0005850 comp rx=0 tx=0).stop 2024-04-25T11:32:41.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:41.053+0000 7f8aea59d640 1 -- 172.21.15.90:0/1919550078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ae41095a0 msgr2=0x7f8ae4102d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:41.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:41.053+0000 7f8aea59d640 1 --2- 172.21.15.90:0/1919550078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ae41095a0 0x7f8ae4102d00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8adc0097b0 tx=0x7f8adc003c20 comp rx=0 tx=0).stop 2024-04-25T11:32:41.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:41.053+0000 7f8aea59d640 1 -- 172.21.15.90:0/1919550078 shutdown_connections 2024-04-25T11:32:41.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:41.053+0000 7f8aea59d640 1 --2- 172.21.15.90:0/1919550078 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f8ad00afae0 0x7f8ad00b1ff0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:41.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:41.053+0000 7f8aea59d640 1 --2- 172.21.15.90:0/1919550078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ae41095a0 0x7f8ae4102d00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:41.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:41.053+0000 7f8aea59d640 1 --2- 172.21.15.90:0/1919550078 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ae4106a30 0x7f8ae41027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:41.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:41.053+0000 7f8aea59d640 1 --2- 172.21.15.90:0/1919550078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ae4101c50 0x7f8ae4104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:41.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:41.053+0000 7f8aea59d640 1 -- 172.21.15.90:0/1919550078 >> 172.21.15.90:0/1919550078 conn(0x7f8ae40fd1c0 msgr2=0x7f8ae4107690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:41.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:41.053+0000 7f8aea59d640 1 -- 172.21.15.90:0/1919550078 shutdown_connections 2024-04-25T11:32:41.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:41.053+0000 7f8aea59d640 1 -- 172.21.15.90:0/1919550078 wait complete. 2024-04-25T11:32:41.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:41 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.90:6834/4061411806,v1:172.21.15.90:6835/4061411806] up:boot 2024-04-25T11:32:41.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:41 smithi090 ceph-mon[29575]: fsmap 3 up:standby 2024-04-25T11:32:41.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:41 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T11:32:41.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:41 smithi090 ceph-mon[29575]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:41.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:41 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:41.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:41 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:41.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:41 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:41.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:41 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:41.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:41 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:41.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:41 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:41.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:41 smithi090 ceph-mon[29575]: Deploying daemon mds.d on smithi090 2024-04-25T11:32:41.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:41 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1919550078' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:42.052 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:41 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.90:6834/4061411806,v1:172.21.15.90:6835/4061411806] up:boot 2024-04-25T11:32:42.052 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:41 smithi156 ceph-mon[33045]: fsmap 3 up:standby 2024-04-25T11:32:42.053 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:41 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T11:32:42.053 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:41 smithi156 ceph-mon[33045]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:42.053 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:41 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:42.053 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:41 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:42.053 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:41 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:42.053 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:41 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:42.053 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:41 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:42.053 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:41 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:42.053 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:41 smithi156 ceph-mon[33045]: Deploying daemon mds.d on smithi090 2024-04-25T11:32:42.053 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:41 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1919550078' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:42.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:41 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.90:6834/4061411806,v1:172.21.15.90:6835/4061411806] up:boot 2024-04-25T11:32:42.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:41 smithi138 ceph-mon[32328]: fsmap 3 up:standby 2024-04-25T11:32:42.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:41 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T11:32:42.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:41 smithi138 ceph-mon[32328]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:42.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:41 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:42.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:41 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:42.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:41 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:42.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:41 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:42.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:41 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:42.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:41 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:42.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:41 smithi138 ceph-mon[32328]: Deploying daemon mds.d on smithi090 2024-04-25T11:32:42.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:41 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1919550078' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:42.976 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 25 11:32:42 smithi090 systemd[1]: Starting Ceph mds.d for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:32:43.056 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:43.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.180+0000 7f79870bd640 1 -- 172.21.15.90:0/705644710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f79801029e0 msgr2=0x7f79801036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:43.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.180+0000 7f79870bd640 1 --2- 172.21.15.90:0/705644710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f79801029e0 0x7f79801036e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f797c009900 tx=0x7f797c02a410 comp rx=0 tx=0).stop 2024-04-25T11:32:43.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.180+0000 7f79870bd640 1 -- 172.21.15.90:0/705644710 shutdown_connections 2024-04-25T11:32:43.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.180+0000 7f79870bd640 1 --2- 172.21.15.90:0/705644710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7980108920 0x7f798010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:43.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.180+0000 7f79870bd640 1 --2- 172.21.15.90:0/705644710 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7980103c20 0x7f7980104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:43.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.180+0000 7f79870bd640 1 --2- 172.21.15.90:0/705644710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f79801029e0 0x7f79801036e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:43.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.180+0000 7f79870bd640 1 -- 172.21.15.90:0/705644710 >> 172.21.15.90:0/705644710 conn(0x7f79800fdbf0 msgr2=0x7f7980100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:43.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.180+0000 7f79870bd640 1 -- 172.21.15.90:0/705644710 shutdown_connections 2024-04-25T11:32:43.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.180+0000 7f79870bd640 1 -- 172.21.15.90:0/705644710 wait complete. 2024-04-25T11:32:43.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.180+0000 7f79870bd640 1 Processor -- start 2024-04-25T11:32:43.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.180+0000 7f79870bd640 1 -- start start 2024-04-25T11:32:43.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.180+0000 7f79870bd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79801029e0 0x7f798010d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:43.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.181+0000 7f79870bd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7980103c20 0x7f798010b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:43.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.181+0000 7f79870bd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7980108920 0x7f798010be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:43.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.181+0000 7f79870bd640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f798010f610 con 0x7f7980108920 2024-04-25T11:32:43.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.181+0000 7f79870bd640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f798010f490 con 0x7f7980103c20 2024-04-25T11:32:43.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.181+0000 7f79870bd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f798010f790 con 0x7f79801029e0 2024-04-25T11:32:43.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.181+0000 7f798565a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7980108920 0x7f798010be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:43.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.181+0000 7f798565a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7980108920 0x7f798010be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46780/0 (socket says 172.21.15.90:46780) 2024-04-25T11:32:43.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.181+0000 7f798565a640 1 -- 172.21.15.90:0/2754083316 learned_addr learned my addr 172.21.15.90:0/2754083316 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:32:43.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.181+0000 7f7984e59640 1 --2- 172.21.15.90:0/2754083316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79801029e0 0x7f798010d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:43.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.181+0000 7f7977fff640 1 --2- 172.21.15.90:0/2754083316 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7980103c20 0x7f798010b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:43.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.181+0000 7f798565a640 1 -- 172.21.15.90:0/2754083316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79801029e0 msgr2=0x7f798010d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:43.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.181+0000 7f798565a640 1 --2- 172.21.15.90:0/2754083316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79801029e0 0x7f798010d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:43.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.181+0000 7f798565a640 1 -- 172.21.15.90:0/2754083316 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7980103c20 msgr2=0x7f798010b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:43.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.181+0000 7f798565a640 1 --2- 172.21.15.90:0/2754083316 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7980103c20 0x7f798010b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:43.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.181+0000 7f798565a640 1 -- 172.21.15.90:0/2754083316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f798010c5c0 con 0x7f7980108920 2024-04-25T11:32:43.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.181+0000 7f7984e59640 1 --2- 172.21.15.90:0/2754083316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79801029e0 0x7f798010d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:32:43.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.181+0000 7f7977fff640 1 --2- 172.21.15.90:0/2754083316 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7980103c20 0x7f798010b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:32:43.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.181+0000 7f798565a640 1 --2- 172.21.15.90:0/2754083316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7980108920 0x7f798010be90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f797800eaa0 tx=0x7f797800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:43.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.181+0000 7f7975ffb640 1 -- 172.21.15.90:0/2754083316 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f797800cdb0 con 0x7f7980108920 2024-04-25T11:32:43.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.181+0000 7f79870bd640 1 -- 172.21.15.90:0/2754083316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79801a4340 con 0x7f7980108920 2024-04-25T11:32:43.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.181+0000 7f7975ffb640 1 -- 172.21.15.90:0/2754083316 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7978005020 con 0x7f7980108920 2024-04-25T11:32:43.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.181+0000 7f7975ffb640 1 -- 172.21.15.90:0/2754083316 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79780116d0 con 0x7f7980108920 2024-04-25T11:32:43.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.181+0000 7f79870bd640 1 -- 172.21.15.90:0/2754083316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79801a4800 con 0x7f7980108920 2024-04-25T11:32:43.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.182+0000 7f79870bd640 1 -- 172.21.15.90:0/2754083316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7980107730 con 0x7f7980108920 2024-04-25T11:32:43.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.184+0000 7f7975ffb640 1 -- 172.21.15.90:0/2754083316 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f79780051d0 con 0x7f7980108920 2024-04-25T11:32:43.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.184+0000 7f7975ffb640 1 --2- 172.21.15.90:0/2754083316 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f79680afaf0 0x7f79680b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:43.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.184+0000 7f7984e59640 1 --2- 172.21.15.90:0/2754083316 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f79680afaf0 0x7f79680b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:43.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.184+0000 7f7975ffb640 1 -- 172.21.15.90:0/2754083316 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f79780de9b0 con 0x7f7980108920 2024-04-25T11:32:43.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.185+0000 7f7984e59640 1 --2- 172.21.15.90:0/2754083316 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f79680afaf0 0x7f79680b2000 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f797c005f70 tx=0x7f797c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:43.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.189+0000 7f7975ffb640 1 -- 172.21.15.90:0/2754083316 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7978016030 con 0x7f7980108920 2024-04-25T11:32:43.328 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 25 11:32:43 smithi090 podman[70345]: 2024-04-25 11:32:42.984901279 +0000 UTC m=+0.017946434 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T11:32:43.329 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 25 11:32:43 smithi090 podman[70345]: 2024-04-25 11:32:43.110963927 +0000 UTC m=+0.144009085 container create 9b9aed298d55ebb3eb51efb57fef4f8d21fcb332ba8cc499da7b15443a2337dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-d, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True) 2024-04-25T11:32:43.329 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 25 11:32:43 smithi090 podman[70345]: 2024-04-25 11:32:43.339766587 +0000 UTC m=+0.372811745 container init 9b9aed298d55ebb3eb51efb57fef4f8d21fcb332ba8cc499da7b15443a2337dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-25T11:32:43.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.413+0000 7f79870bd640 1 -- 172.21.15.90:0/2754083316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f798002df10 con 0x7f7980108920 2024-04-25T11:32:43.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.413+0000 7f7975ffb640 1 -- 172.21.15.90:0/2754083316 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3037 (secure 0 0 0) 0x7f79780a7a30 con 0x7f7980108920 2024-04-25T11:32:43.403 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:32:43.403 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T11:32:43.404 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 4 2024-04-25T11:32:43.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.415+0000 7f79870bd640 1 -- 172.21.15.90:0/2754083316 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f79680afaf0 msgr2=0x7f79680b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:43.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.415+0000 7f79870bd640 1 --2- 172.21.15.90:0/2754083316 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f79680afaf0 0x7f79680b2000 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f797c005f70 tx=0x7f797c031400 comp rx=0 tx=0).stop 2024-04-25T11:32:43.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.415+0000 7f79870bd640 1 -- 172.21.15.90:0/2754083316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7980108920 msgr2=0x7f798010be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.415+0000 7f79870bd640 1 --2- 172.21.15.90:0/2754083316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7980108920 0x7f798010be90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f797800eaa0 tx=0x7f797800ef60 comp rx=0 tx=0).stop 2024-04-25T11:32:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.415+0000 7f79870bd640 1 -- 172.21.15.90:0/2754083316 shutdown_connections 2024-04-25T11:32:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.415+0000 7f79870bd640 1 --2- 172.21.15.90:0/2754083316 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f79680afaf0 0x7f79680b2000 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.415+0000 7f79870bd640 1 --2- 172.21.15.90:0/2754083316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7980108920 0x7f798010be90 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.415+0000 7f79870bd640 1 --2- 172.21.15.90:0/2754083316 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7980103c20 0x7f798010b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.415+0000 7f79870bd640 1 --2- 172.21.15.90:0/2754083316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79801029e0 0x7f798010d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.415+0000 7f79870bd640 1 -- 172.21.15.90:0/2754083316 >> 172.21.15.90:0/2754083316 conn(0x7f79800fdbf0 msgr2=0x7f7980104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.416+0000 7f79870bd640 1 -- 172.21.15.90:0/2754083316 shutdown_connections 2024-04-25T11:32:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:43.416+0000 7f79870bd640 1 -- 172.21.15.90:0/2754083316 wait complete. 2024-04-25T11:32:43.726 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 25 11:32:43 smithi090 podman[70345]: 2024-04-25 11:32:43.34276132 +0000 UTC m=+0.375806476 container start 9b9aed298d55ebb3eb51efb57fef4f8d21fcb332ba8cc499da7b15443a2337dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-25T11:32:43.726 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 25 11:32:43 smithi090 ceph-mds[70403]: -- 172.21.15.90:0/4269398151 <== mon.0 v2:172.21.15.90:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ab93b27a40 con 0x55ab93b6f600 2024-04-25T11:32:43.726 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 25 11:32:43 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-d[70399]: starting mds.d at 2024-04-25T11:32:43.726 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 25 11:32:43 smithi090 bash[70345]: 9b9aed298d55ebb3eb51efb57fef4f8d21fcb332ba8cc499da7b15443a2337dc 2024-04-25T11:32:43.726 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 25 11:32:43 smithi090 systemd[1]: Started Ceph mds.d for 58bd4e7a-02f6-11ef-bc93-c7b262605968. 2024-04-25T11:32:44.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:43 smithi138 ceph-mon[32328]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:44.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:43 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2754083316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:44.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:43 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:44.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:43 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:44.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:43 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:44.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:43 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:44.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:43 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:44.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:43 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:44.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:43 smithi156 ceph-mon[33045]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:44.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:43 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2754083316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:44.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:43 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:44.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:43 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:44.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:43 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:44.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:43 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:44.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:43 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:44.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:43 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:44.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:43 smithi090 ceph-mon[29575]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:44.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:43 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2754083316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:44.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:43 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:44.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:43 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:44.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:43 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:44.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:43 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:44.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:43 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:44.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:43 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:45.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:44 smithi138 ceph-mon[32328]: Deploying daemon mds.e on smithi138 2024-04-25T11:32:45.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:44 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.90:6836/3050267616,v1:172.21.15.90:6837/3050267616] up:boot 2024-04-25T11:32:45.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:44 smithi138 ceph-mon[32328]: fsmap 4 up:standby 2024-04-25T11:32:45.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:44 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T11:32:45.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:44 smithi156 ceph-mon[33045]: Deploying daemon mds.e on smithi138 2024-04-25T11:32:45.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:44 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.90:6836/3050267616,v1:172.21.15.90:6837/3050267616] up:boot 2024-04-25T11:32:45.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:44 smithi156 ceph-mon[33045]: fsmap 4 up:standby 2024-04-25T11:32:45.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:44 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T11:32:45.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:44 smithi090 ceph-mon[29575]: Deploying daemon mds.e on smithi138 2024-04-25T11:32:45.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:44 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.90:6836/3050267616,v1:172.21.15.90:6837/3050267616] up:boot 2024-04-25T11:32:45.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:44 smithi090 ceph-mon[29575]: fsmap 4 up:standby 2024-04-25T11:32:45.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:44 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T11:32:45.418 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:45.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.550+0000 7f3a6c268640 1 -- 172.21.15.90:0/1639554901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a64104d40 msgr2=0x7f3a64105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:45.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.550+0000 7f3a6c268640 1 --2- 172.21.15.90:0/1639554901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a64104d40 0x7f3a64105150 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3a60009900 tx=0x7f3a6002a410 comp rx=0 tx=0).stop 2024-04-25T11:32:45.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.550+0000 7f3a6c268640 1 -- 172.21.15.90:0/1639554901 shutdown_connections 2024-04-25T11:32:45.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.550+0000 7f3a6c268640 1 --2- 172.21.15.90:0/1639554901 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a64109a40 0x7f3a6410be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:45.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.550+0000 7f3a6c268640 1 --2- 172.21.15.90:0/1639554901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a64104d40 0x7f3a64105150 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:45.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.550+0000 7f3a6c268640 1 --2- 172.21.15.90:0/1639554901 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a64103b00 0x7f3a64104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:45.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.550+0000 7f3a6c268640 1 -- 172.21.15.90:0/1639554901 >> 172.21.15.90:0/1639554901 conn(0x7f3a640fed10 msgr2=0x7f3a641011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:45.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.551+0000 7f3a6c268640 1 -- 172.21.15.90:0/1639554901 shutdown_connections 2024-04-25T11:32:45.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.551+0000 7f3a6c268640 1 -- 172.21.15.90:0/1639554901 wait complete. 2024-04-25T11:32:45.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.551+0000 7f3a6c268640 1 Processor -- start 2024-04-25T11:32:45.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.551+0000 7f3a6c268640 1 -- start start 2024-04-25T11:32:45.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.551+0000 7f3a6c268640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a64103b00 0x7f3a6419a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:45.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.551+0000 7f3a6c268640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a64104d40 0x7f3a6419a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:45.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.551+0000 7f3a6c268640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a64109a40 0x7f3a641a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:45.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.551+0000 7f3a6c268640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3a6410f630 con 0x7f3a64103b00 2024-04-25T11:32:45.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.551+0000 7f3a6c268640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3a6410f4b0 con 0x7f3a64104d40 2024-04-25T11:32:45.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.552+0000 7f3a6c268640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3a6410f7b0 con 0x7f3a64109a40 2024-04-25T11:32:45.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.552+0000 7f3a6a004640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a64103b00 0x7f3a6419a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:45.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.552+0000 7f3a6a004640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a64103b00 0x7f3a6419a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46806/0 (socket says 172.21.15.90:46806) 2024-04-25T11:32:45.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.552+0000 7f3a6a004640 1 -- 172.21.15.90:0/2417676026 learned_addr learned my addr 172.21.15.90:0/2417676026 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:32:45.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.552+0000 7f3a69803640 1 --2- 172.21.15.90:0/2417676026 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a64104d40 0x7f3a6419a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:45.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.552+0000 7f3a6a805640 1 --2- 172.21.15.90:0/2417676026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a64109a40 0x7f3a641a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:45.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.552+0000 7f3a6a004640 1 -- 172.21.15.90:0/2417676026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a64109a40 msgr2=0x7f3a641a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:45.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.552+0000 7f3a6a004640 1 --2- 172.21.15.90:0/2417676026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a64109a40 0x7f3a641a1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:45.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.552+0000 7f3a6a004640 1 -- 172.21.15.90:0/2417676026 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a64104d40 msgr2=0x7f3a6419a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:45.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.552+0000 7f3a6a004640 1 --2- 172.21.15.90:0/2417676026 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a64104d40 0x7f3a6419a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:45.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.552+0000 7f3a6a004640 1 -- 172.21.15.90:0/2417676026 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a641a1d70 con 0x7f3a64103b00 2024-04-25T11:32:45.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.552+0000 7f3a69803640 1 --2- 172.21.15.90:0/2417676026 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a64104d40 0x7f3a6419a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:32:45.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.552+0000 7f3a6a004640 1 --2- 172.21.15.90:0/2417676026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a64103b00 0x7f3a6419a230 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3a5400e9f0 tx=0x7f3a5400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:45.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.552+0000 7f3a5b7fe640 1 -- 172.21.15.90:0/2417676026 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a5400ccc0 con 0x7f3a64103b00 2024-04-25T11:32:45.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.552+0000 7f3a5b7fe640 1 -- 172.21.15.90:0/2417676026 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a540035f0 con 0x7f3a64103b00 2024-04-25T11:32:45.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.552+0000 7f3a6c268640 1 -- 172.21.15.90:0/2417676026 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a641a2060 con 0x7f3a64103b00 2024-04-25T11:32:45.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.552+0000 7f3a5b7fe640 1 -- 172.21.15.90:0/2417676026 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a5400ce60 con 0x7f3a64103b00 2024-04-25T11:32:45.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.552+0000 7f3a6c268640 1 -- 172.21.15.90:0/2417676026 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a641a2600 con 0x7f3a64103b00 2024-04-25T11:32:45.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.553+0000 7f3a6c268640 1 -- 172.21.15.90:0/2417676026 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a64108850 con 0x7f3a64103b00 2024-04-25T11:32:45.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.555+0000 7f3a5b7fe640 1 -- 172.21.15.90:0/2417676026 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f3a54003aa0 con 0x7f3a64103b00 2024-04-25T11:32:45.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.556+0000 7f3a5b7fe640 1 --2- 172.21.15.90:0/2417676026 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f3a500afb70 0x7f3a500b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:45.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.556+0000 7f3a69803640 1 --2- 172.21.15.90:0/2417676026 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f3a500afb70 0x7f3a500b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:45.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.556+0000 7f3a5b7fe640 1 -- 172.21.15.90:0/2417676026 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f3a540decb0 con 0x7f3a64103b00 2024-04-25T11:32:45.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.557+0000 7f3a69803640 1 --2- 172.21.15.90:0/2417676026 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f3a500afb70 0x7f3a500b2080 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3a60000c00 tx=0x7f3a60005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:45.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.560+0000 7f3a5b7fe640 1 -- 172.21.15.90:0/2417676026 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3a540a7c90 con 0x7f3a64103b00 2024-04-25T11:32:45.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.777+0000 7f3a6c268640 1 -- 172.21.15.90:0/2417676026 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a6402db50 con 0x7f3a64103b00 2024-04-25T11:32:45.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.778+0000 7f3a5b7fe640 1 -- 172.21.15.90:0/2417676026 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3862 (secure 0 0 0) 0x7f3a540a7590 con 0x7f3a64103b00 2024-04-25T11:32:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:32:45.768 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T11:32:45.768 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 5 2024-04-25T11:32:45.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.780+0000 7f3a6c268640 1 -- 172.21.15.90:0/2417676026 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f3a500afb70 msgr2=0x7f3a500b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:45.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.780+0000 7f3a6c268640 1 --2- 172.21.15.90:0/2417676026 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f3a500afb70 0x7f3a500b2080 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3a60000c00 tx=0x7f3a60005d50 comp rx=0 tx=0).stop 2024-04-25T11:32:45.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.780+0000 7f3a6c268640 1 -- 172.21.15.90:0/2417676026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a64103b00 msgr2=0x7f3a6419a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:45.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.780+0000 7f3a6c268640 1 --2- 172.21.15.90:0/2417676026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a64103b00 0x7f3a6419a230 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3a5400e9f0 tx=0x7f3a5400eeb0 comp rx=0 tx=0).stop 2024-04-25T11:32:45.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.780+0000 7f3a6c268640 1 -- 172.21.15.90:0/2417676026 shutdown_connections 2024-04-25T11:32:45.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.780+0000 7f3a6c268640 1 --2- 172.21.15.90:0/2417676026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a64109a40 0x7f3a641a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:45.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.780+0000 7f3a6c268640 1 --2- 172.21.15.90:0/2417676026 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a64104d40 0x7f3a6419a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:45.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.780+0000 7f3a6c268640 1 --2- 172.21.15.90:0/2417676026 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f3a500afb70 0x7f3a500b2080 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:45.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.780+0000 7f3a6c268640 1 --2- 172.21.15.90:0/2417676026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a64103b00 0x7f3a6419a230 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:45.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.780+0000 7f3a6c268640 1 -- 172.21.15.90:0/2417676026 >> 172.21.15.90:0/2417676026 conn(0x7f3a640fed10 msgr2=0x7f3a64105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:45.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.780+0000 7f3a6c268640 1 -- 172.21.15.90:0/2417676026 shutdown_connections 2024-04-25T11:32:45.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:45.780+0000 7f3a6c268640 1 -- 172.21.15.90:0/2417676026 wait complete. 2024-04-25T11:32:45.788 INFO:journalctl@ceph.mds.e.smithi138.stdout:Apr 25 11:32:45 smithi138 systemd[1]: Starting Ceph mds.e for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:32:46.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:45 smithi138 ceph-mon[32328]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:46.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:45 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2417676026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:46.053 INFO:journalctl@ceph.mds.e.smithi138.stdout:Apr 25 11:32:45 smithi138 podman[61922]: 2024-04-25 11:32:45.725891903 +0000 UTC m=+0.018064219 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T11:32:46.053 INFO:journalctl@ceph.mds.e.smithi138.stdout:Apr 25 11:32:45 smithi138 podman[61922]: 2024-04-25 11:32:45.841511315 +0000 UTC m=+0.133683645 container create c48443ecc17aa084bf96d2c805e8d50304eec5dd1e6ec51b4728171b8a2e6888 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-e, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T11:32:46.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:45 smithi156 ceph-mon[33045]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:46.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:45 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2417676026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:46.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:45 smithi090 ceph-mon[29575]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:46.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:45 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2417676026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:46.553 INFO:journalctl@ceph.mds.e.smithi138.stdout:Apr 25 11:32:46 smithi138 podman[61922]: 2024-04-25 11:32:46.068721939 +0000 UTC m=+0.360894235 container init c48443ecc17aa084bf96d2c805e8d50304eec5dd1e6ec51b4728171b8a2e6888 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-e, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T11:32:46.553 INFO:journalctl@ceph.mds.e.smithi138.stdout:Apr 25 11:32:46 smithi138 podman[61922]: 2024-04-25 11:32:46.071359562 +0000 UTC m=+0.363531852 container start c48443ecc17aa084bf96d2c805e8d50304eec5dd1e6ec51b4728171b8a2e6888 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-e, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T11:32:46.553 INFO:journalctl@ceph.mds.e.smithi138.stdout:Apr 25 11:32:46 smithi138 ceph-mds[61941]: -- 172.21.15.138:0/268476788 <== mon.2 v2:172.21.15.138:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d0ae265a40 con 0x55d0ae2ad600 2024-04-25T11:32:46.553 INFO:journalctl@ceph.mds.e.smithi138.stdout:Apr 25 11:32:46 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-e[61937]: starting mds.e at 2024-04-25T11:32:46.553 INFO:journalctl@ceph.mds.e.smithi138.stdout:Apr 25 11:32:46 smithi138 bash[61922]: c48443ecc17aa084bf96d2c805e8d50304eec5dd1e6ec51b4728171b8a2e6888 2024-04-25T11:32:46.553 INFO:journalctl@ceph.mds.e.smithi138.stdout:Apr 25 11:32:46 smithi138 systemd[1]: Started Ceph mds.e for 58bd4e7a-02f6-11ef-bc93-c7b262605968. 2024-04-25T11:32:47.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:47 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:47.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:47 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:47.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:47 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:47.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:47 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:47.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:47 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:47.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:47 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:47.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:47 smithi090 ceph-mon[29575]: Deploying daemon mds.f on smithi156 2024-04-25T11:32:47.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:47 smithi090 ceph-mon[29575]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:47.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:47 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] up:boot 2024-04-25T11:32:47.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:47 smithi090 ceph-mon[29575]: fsmap 5 up:standby 2024-04-25T11:32:47.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:47 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T11:32:47.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:47 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:47.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:47 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:47.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:47 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:47.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:47 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:47.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:47 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:47.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:47 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:47.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:47 smithi138 ceph-mon[32328]: Deploying daemon mds.f on smithi156 2024-04-25T11:32:47.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:47 smithi138 ceph-mon[32328]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:47.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:47 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] up:boot 2024-04-25T11:32:47.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:47 smithi138 ceph-mon[32328]: fsmap 5 up:standby 2024-04-25T11:32:47.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:47 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T11:32:47.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:47 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:47.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:47 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:47.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:47 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:47.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:47 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:47.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:47 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:47.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:47 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:47.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:47 smithi156 ceph-mon[33045]: Deploying daemon mds.f on smithi156 2024-04-25T11:32:47.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:47 smithi156 ceph-mon[33045]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:47.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:47 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] up:boot 2024-04-25T11:32:47.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:47 smithi156 ceph-mon[33045]: fsmap 5 up:standby 2024-04-25T11:32:47.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:47 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T11:32:47.782 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:47.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.915+0000 7f39457b7640 1 -- 172.21.15.90:0/4247606378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39401029c0 msgr2=0x7f39401036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:47.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.915+0000 7f39457b7640 1 --2- 172.21.15.90:0/4247606378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39401029c0 0x7f39401036c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3930009900 tx=0x7f393002a410 comp rx=0 tx=0).stop 2024-04-25T11:32:47.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.915+0000 7f39457b7640 1 -- 172.21.15.90:0/4247606378 shutdown_connections 2024-04-25T11:32:47.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.915+0000 7f39457b7640 1 --2- 172.21.15.90:0/4247606378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3940108900 0x7f394010ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:47.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.915+0000 7f39457b7640 1 --2- 172.21.15.90:0/4247606378 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3940103c00 0x7f3940104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:47.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.915+0000 7f39457b7640 1 --2- 172.21.15.90:0/4247606378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39401029c0 0x7f39401036c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:47.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.915+0000 7f39457b7640 1 -- 172.21.15.90:0/4247606378 >> 172.21.15.90:0/4247606378 conn(0x7f39400fdbd0 msgr2=0x7f3940100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:47.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.915+0000 7f39457b7640 1 -- 172.21.15.90:0/4247606378 shutdown_connections 2024-04-25T11:32:47.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.915+0000 7f39457b7640 1 -- 172.21.15.90:0/4247606378 wait complete. 2024-04-25T11:32:47.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.916+0000 7f39457b7640 1 Processor -- start 2024-04-25T11:32:47.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.916+0000 7f39457b7640 1 -- start start 2024-04-25T11:32:47.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.916+0000 7f39457b7640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39401029c0 0x7f394019a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:47.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.916+0000 7f39457b7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3940103c00 0x7f394019a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:47.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.916+0000 7f39457b7640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3940108900 0x7f39401a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:47.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.916+0000 7f39457b7640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f394010f5f0 con 0x7f39401029c0 2024-04-25T11:32:47.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.916+0000 7f39457b7640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f394010f470 con 0x7f3940108900 2024-04-25T11:32:47.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.916+0000 7f39457b7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f394010f770 con 0x7f3940103c00 2024-04-25T11:32:47.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.916+0000 7f393effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39401029c0 0x7f394019a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:47.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.916+0000 7f393effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39401029c0 0x7f394019a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46824/0 (socket says 172.21.15.90:46824) 2024-04-25T11:32:47.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.916+0000 7f393effd640 1 -- 172.21.15.90:0/1110530029 learned_addr learned my addr 172.21.15.90:0/1110530029 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:32:47.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.916+0000 7f393e7fc640 1 --2- 172.21.15.90:0/1110530029 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3940103c00 0x7f394019a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:47.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.916+0000 7f393f7fe640 1 --2- 172.21.15.90:0/1110530029 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3940108900 0x7f39401a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:47.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.917+0000 7f393effd640 1 -- 172.21.15.90:0/1110530029 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3940103c00 msgr2=0x7f394019a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:47.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.917+0000 7f393effd640 1 --2- 172.21.15.90:0/1110530029 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3940103c00 0x7f394019a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:47.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.917+0000 7f393effd640 1 -- 172.21.15.90:0/1110530029 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3940108900 msgr2=0x7f39401a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:47.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.917+0000 7f393effd640 1 --2- 172.21.15.90:0/1110530029 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3940108900 0x7f39401a1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:47.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.917+0000 7f393effd640 1 -- 172.21.15.90:0/1110530029 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39401a1d70 con 0x7f39401029c0 2024-04-25T11:32:47.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.917+0000 7f393effd640 1 --2- 172.21.15.90:0/1110530029 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39401029c0 0x7f394019a230 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3930006010 tx=0x7f3930004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:47.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.917+0000 7f3937fff640 1 -- 172.21.15.90:0/1110530029 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f393003f030 con 0x7f39401029c0 2024-04-25T11:32:47.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.917+0000 7f3937fff640 1 -- 172.21.15.90:0/1110530029 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3930003aa0 con 0x7f39401029c0 2024-04-25T11:32:47.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.917+0000 7f39457b7640 1 -- 172.21.15.90:0/1110530029 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39401a2000 con 0x7f39401029c0 2024-04-25T11:32:47.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.917+0000 7f3937fff640 1 -- 172.21.15.90:0/1110530029 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3930035070 con 0x7f39401029c0 2024-04-25T11:32:47.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.917+0000 7f39457b7640 1 -- 172.21.15.90:0/1110530029 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39401a24f0 con 0x7f39401029c0 2024-04-25T11:32:47.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.918+0000 7f39457b7640 1 -- 172.21.15.90:0/1110530029 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3940107710 con 0x7f39401029c0 2024-04-25T11:32:47.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.919+0000 7f3937fff640 1 -- 172.21.15.90:0/1110530029 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f3930042030 con 0x7f39401029c0 2024-04-25T11:32:47.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.920+0000 7f3937fff640 1 --2- 172.21.15.90:0/1110530029 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f392c0afaf0 0x7f392c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:47.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.920+0000 7f393e7fc640 1 --2- 172.21.15.90:0/1110530029 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f392c0afaf0 0x7f392c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:47.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.920+0000 7f3937fff640 1 -- 172.21.15.90:0/1110530029 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f39300fed00 con 0x7f39401029c0 2024-04-25T11:32:47.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.920+0000 7f393e7fc640 1 --2- 172.21.15.90:0/1110530029 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f392c0afaf0 0x7f392c0b2000 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f394019b2d0 tx=0x7f3928005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:47.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:47.925+0000 7f3937fff640 1 -- 172.21.15.90:0/1110530029 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f39300c7730 con 0x7f39401029c0 2024-04-25T11:32:48.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:48.140+0000 7f39457b7640 1 -- 172.21.15.90:0/1110530029 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f394002db50 con 0x7f39401029c0 2024-04-25T11:32:48.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:48.140+0000 7f3937fff640 1 -- 172.21.15.90:0/1110530029 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4687 (secure 0 0 0) 0x7f39300cba30 con 0x7f39401029c0 2024-04-25T11:32:48.130 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:32:48.130 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-25T11:32:48.130 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 6 2024-04-25T11:32:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:48.142+0000 7f39457b7640 1 -- 172.21.15.90:0/1110530029 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f392c0afaf0 msgr2=0x7f392c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:48.142+0000 7f39457b7640 1 --2- 172.21.15.90:0/1110530029 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f392c0afaf0 0x7f392c0b2000 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f394019b2d0 tx=0x7f3928005f90 comp rx=0 tx=0).stop 2024-04-25T11:32:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:48.142+0000 7f39457b7640 1 -- 172.21.15.90:0/1110530029 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39401029c0 msgr2=0x7f394019a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:48.142+0000 7f39457b7640 1 --2- 172.21.15.90:0/1110530029 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39401029c0 0x7f394019a230 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3930006010 tx=0x7f3930004270 comp rx=0 tx=0).stop 2024-04-25T11:32:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:48.142+0000 7f39457b7640 1 -- 172.21.15.90:0/1110530029 shutdown_connections 2024-04-25T11:32:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:48.142+0000 7f39457b7640 1 --2- 172.21.15.90:0/1110530029 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3940108900 0x7f39401a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:48.142+0000 7f39457b7640 1 --2- 172.21.15.90:0/1110530029 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3940103c00 0x7f394019a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:48.142+0000 7f39457b7640 1 --2- 172.21.15.90:0/1110530029 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f392c0afaf0 0x7f392c0b2000 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:48.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:48.142+0000 7f39457b7640 1 --2- 172.21.15.90:0/1110530029 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39401029c0 0x7f394019a230 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:48.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:48.142+0000 7f39457b7640 1 -- 172.21.15.90:0/1110530029 >> 172.21.15.90:0/1110530029 conn(0x7f39400fdbd0 msgr2=0x7f39401048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:48.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:48.142+0000 7f39457b7640 1 -- 172.21.15.90:0/1110530029 shutdown_connections 2024-04-25T11:32:48.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:48.142+0000 7f39457b7640 1 -- 172.21.15.90:0/1110530029 wait complete. 2024-04-25T11:32:48.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:48 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1110530029' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:48.495 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:48 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1110530029' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:48.495 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 11:32:48 smithi156 systemd[1]: Starting Ceph mds.f for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:32:48.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:48 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1110530029' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:48.747 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 11:32:48 smithi156 podman[63453]: 2024-04-25 11:32:48.409618257 +0000 UTC m=+0.018000498 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T11:32:48.747 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 11:32:48 smithi156 podman[63453]: 2024-04-25 11:32:48.50621959 +0000 UTC m=+0.114601802 container create d546e8c332cf47cc7de2d3c7fbf93f81c48de9c2cb2427c4f9640baf31c1a9de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-f, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True) 2024-04-25T11:32:48.747 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 11:32:48 smithi156 podman[63453]: 2024-04-25 11:32:48.75882208 +0000 UTC m=+0.367204303 container init d546e8c332cf47cc7de2d3c7fbf93f81c48de9c2cb2427c4f9640baf31c1a9de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-f, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-25T11:32:49.088 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 11:32:48 smithi156 podman[63453]: 2024-04-25 11:32:48.761427393 +0000 UTC m=+0.369809606 container start d546e8c332cf47cc7de2d3c7fbf93f81c48de9c2cb2427c4f9640baf31c1a9de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-f, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-25T11:32:49.088 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 11:32:48 smithi156 ceph-mds[63472]: -- 172.21.15.156:0/106460173 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ca65743a40 con 0x55ca6578b600 2024-04-25T11:32:49.088 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 11:32:48 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-f[63468]: starting mds.f at 2024-04-25T11:32:49.089 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 11:32:48 smithi156 bash[63453]: d546e8c332cf47cc7de2d3c7fbf93f81c48de9c2cb2427c4f9640baf31c1a9de 2024-04-25T11:32:49.089 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 11:32:48 smithi156 systemd[1]: Started Ceph mds.f for 58bd4e7a-02f6-11ef-bc93-c7b262605968. 2024-04-25T11:32:49.540 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:49 smithi156 ceph-mon[33045]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:49.541 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:49 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:49.541 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:49 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:49.541 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:49 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:49.541 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:49 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:49.541 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:49 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:49.541 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:49 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:49.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:49 smithi138 ceph-mon[32328]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:49.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:49 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:49.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:49 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:49.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:49 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:49.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:49 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:49.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:49 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:49.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:49 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:49.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:49 smithi090 ceph-mon[29575]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:49.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:49 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:49.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:49 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:49.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:49 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:49.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:49 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:49.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:49 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:49.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:49 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:50.145 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:50.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.278+0000 7fb7daf6b640 1 -- 172.21.15.90:0/2124553880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7d4104d40 msgr2=0x7fb7d4105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:50.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.278+0000 7fb7daf6b640 1 --2- 172.21.15.90:0/2124553880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7d4104d40 0x7fb7d4105150 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4009900 tx=0x7fb7c402a410 comp rx=0 tx=0).stop 2024-04-25T11:32:50.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.278+0000 7fb7daf6b640 1 -- 172.21.15.90:0/2124553880 shutdown_connections 2024-04-25T11:32:50.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.278+0000 7fb7daf6b640 1 --2- 172.21.15.90:0/2124553880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb7d4109a40 0x7fb7d410be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:50.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.278+0000 7fb7daf6b640 1 --2- 172.21.15.90:0/2124553880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7d4104d40 0x7fb7d4105150 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:50.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.278+0000 7fb7daf6b640 1 --2- 172.21.15.90:0/2124553880 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb7d4103b00 0x7fb7d4104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:50.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.278+0000 7fb7daf6b640 1 -- 172.21.15.90:0/2124553880 >> 172.21.15.90:0/2124553880 conn(0x7fb7d40fed30 msgr2=0x7fb7d41011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:50.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.278+0000 7fb7daf6b640 1 -- 172.21.15.90:0/2124553880 shutdown_connections 2024-04-25T11:32:50.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.278+0000 7fb7daf6b640 1 -- 172.21.15.90:0/2124553880 wait complete. 2024-04-25T11:32:50.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.278+0000 7fb7daf6b640 1 Processor -- start 2024-04-25T11:32:50.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.279+0000 7fb7daf6b640 1 -- start start 2024-04-25T11:32:50.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.279+0000 7fb7daf6b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7d4103b00 0x7fb7d419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:50.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.279+0000 7fb7daf6b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb7d4104d40 0x7fb7d419a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:50.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.279+0000 7fb7daf6b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb7d4109a40 0x7fb7d41a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:50.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.279+0000 7fb7daf6b640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d410f680 con 0x7fb7d4103b00 2024-04-25T11:32:50.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.279+0000 7fb7daf6b640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d410f500 con 0x7fb7d4109a40 2024-04-25T11:32:50.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.279+0000 7fb7daf6b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d410f800 con 0x7fb7d4104d40 2024-04-25T11:32:50.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.279+0000 7fb7d8d07640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7d4103b00 0x7fb7d419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:50.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.279+0000 7fb7d8d07640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7d4103b00 0x7fb7d419a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46850/0 (socket says 172.21.15.90:46850) 2024-04-25T11:32:50.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.279+0000 7fb7d8d07640 1 -- 172.21.15.90:0/543254927 learned_addr learned my addr 172.21.15.90:0/543254927 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:32:50.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.279+0000 7fb7d3fff640 1 --2- 172.21.15.90:0/543254927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb7d4104d40 0x7fb7d419a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:50.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.279+0000 7fb7d8d07640 1 -- 172.21.15.90:0/543254927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb7d4104d40 msgr2=0x7fb7d419a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:50.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.279+0000 7fb7d8d07640 1 --2- 172.21.15.90:0/543254927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb7d4104d40 0x7fb7d419a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:50.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.279+0000 7fb7d8d07640 1 -- 172.21.15.90:0/543254927 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb7d4109a40 msgr2=0x7fb7d41a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:50.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.279+0000 7fb7d8d07640 1 --2- 172.21.15.90:0/543254927 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb7d4109a40 0x7fb7d41a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:50.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.279+0000 7fb7d8d07640 1 -- 172.21.15.90:0/543254927 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7d41a1ce0 con 0x7fb7d4103b00 2024-04-25T11:32:50.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.279+0000 7fb7d8d07640 1 --2- 172.21.15.90:0/543254927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7d4103b00 0x7fb7d419a250 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00cce0 tx=0x7fb7cc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:50.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.279+0000 7fb7d1ffb640 1 -- 172.21.15.90:0/543254927 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc00f450 con 0x7fb7d4103b00 2024-04-25T11:32:50.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.279+0000 7fb7d1ffb640 1 -- 172.21.15.90:0/543254927 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7cc00faa0 con 0x7fb7d4103b00 2024-04-25T11:32:50.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.279+0000 7fb7daf6b640 1 -- 172.21.15.90:0/543254927 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7d41a1fd0 con 0x7fb7d4103b00 2024-04-25T11:32:50.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.279+0000 7fb7d1ffb640 1 -- 172.21.15.90:0/543254927 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc018b70 con 0x7fb7d4103b00 2024-04-25T11:32:50.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.280+0000 7fb7daf6b640 1 -- 172.21.15.90:0/543254927 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7d41a24b0 con 0x7fb7d4103b00 2024-04-25T11:32:50.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.281+0000 7fb7daf6b640 1 -- 172.21.15.90:0/543254927 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7d406afd0 con 0x7fb7d4103b00 2024-04-25T11:32:50.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.283+0000 7fb7d1ffb640 1 -- 172.21.15.90:0/543254927 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fb7cc018d10 con 0x7fb7d4103b00 2024-04-25T11:32:50.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.284+0000 7fb7d1ffb640 1 --2- 172.21.15.90:0/543254927 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb7c00afaf0 0x7fb7c00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:50.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.284+0000 7fb7d3fff640 1 --2- 172.21.15.90:0/543254927 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb7c00afaf0 0x7fb7c00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:50.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.284+0000 7fb7d1ffb640 1 -- 172.21.15.90:0/543254927 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7fb7cc0ded60 con 0x7fb7d4103b00 2024-04-25T11:32:50.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.284+0000 7fb7d3fff640 1 --2- 172.21.15.90:0/543254927 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb7c00afaf0 0x7fb7c00b2000 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb7c402a930 tx=0x7fb7c4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:50.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.287+0000 7fb7d1ffb640 1 -- 172.21.15.90:0/543254927 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb7cc052ff0 con 0x7fb7d4103b00 2024-04-25T11:32:50.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:50 smithi138 ceph-mon[32328]: Deploying daemon mds.c on smithi156 2024-04-25T11:32:50.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:50 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] up:boot 2024-04-25T11:32:50.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:50 smithi138 ceph-mon[32328]: fsmap 6 up:standby 2024-04-25T11:32:50.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:50 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T11:32:50.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:50 smithi090 ceph-mon[29575]: Deploying daemon mds.c on smithi156 2024-04-25T11:32:50.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:50 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] up:boot 2024-04-25T11:32:50.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:50 smithi090 ceph-mon[29575]: fsmap 6 up:standby 2024-04-25T11:32:50.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:50 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T11:32:50.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.503+0000 7fb7daf6b640 1 -- 172.21.15.90:0/543254927 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7d402db50 con 0x7fb7d4103b00 2024-04-25T11:32:50.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.504+0000 7fb7d1ffb640 1 -- 172.21.15.90:0/543254927 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5515 (secure 0 0 0) 0x7fb7d406afd0 con 0x7fb7d4103b00 2024-04-25T11:32:50.494 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:32:50.494 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-25T11:32:50.494 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 7 2024-04-25T11:32:50.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.506+0000 7fb7daf6b640 1 -- 172.21.15.90:0/543254927 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb7c00afaf0 msgr2=0x7fb7c00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:50.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.506+0000 7fb7daf6b640 1 --2- 172.21.15.90:0/543254927 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb7c00afaf0 0x7fb7c00b2000 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb7c402a930 tx=0x7fb7c4000bc0 comp rx=0 tx=0).stop 2024-04-25T11:32:50.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.506+0000 7fb7daf6b640 1 -- 172.21.15.90:0/543254927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7d4103b00 msgr2=0x7fb7d419a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:50.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.506+0000 7fb7daf6b640 1 --2- 172.21.15.90:0/543254927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7d4103b00 0x7fb7d419a250 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00cce0 tx=0x7fb7cc00a620 comp rx=0 tx=0).stop 2024-04-25T11:32:50.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.506+0000 7fb7daf6b640 1 -- 172.21.15.90:0/543254927 shutdown_connections 2024-04-25T11:32:50.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.506+0000 7fb7daf6b640 1 --2- 172.21.15.90:0/543254927 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb7d4109a40 0x7fb7d41a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:50.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.506+0000 7fb7daf6b640 1 --2- 172.21.15.90:0/543254927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb7d4104d40 0x7fb7d419a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:50.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.506+0000 7fb7daf6b640 1 --2- 172.21.15.90:0/543254927 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb7c00afaf0 0x7fb7c00b2000 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:50.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.506+0000 7fb7daf6b640 1 --2- 172.21.15.90:0/543254927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7d4103b00 0x7fb7d419a250 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:50.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.506+0000 7fb7daf6b640 1 -- 172.21.15.90:0/543254927 >> 172.21.15.90:0/543254927 conn(0x7fb7d40fed30 msgr2=0x7fb7d4105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:50.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.506+0000 7fb7daf6b640 1 -- 172.21.15.90:0/543254927 shutdown_connections 2024-04-25T11:32:50.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:50.506+0000 7fb7daf6b640 1 -- 172.21.15.90:0/543254927 wait complete. 2024-04-25T11:32:50.543 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:50 smithi156 ceph-mon[33045]: Deploying daemon mds.c on smithi156 2024-04-25T11:32:50.543 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:50 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] up:boot 2024-04-25T11:32:50.544 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:50 smithi156 ceph-mon[33045]: fsmap 6 up:standby 2024-04-25T11:32:50.544 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:50 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T11:32:51.258 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 11:32:51 smithi156 systemd[1]: Starting Ceph mds.c for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:32:51.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:51 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/543254927' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:51.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:51 smithi138 ceph-mon[32328]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:51.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:51 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/543254927' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:51.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:51 smithi156 ceph-mon[33045]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:51.589 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 11:32:51 smithi156 podman[63764]: 2024-04-25 11:32:51.250594785 +0000 UTC m=+0.018144966 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T11:32:51.589 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 11:32:51 smithi156 podman[63764]: 2024-04-25 11:32:51.376232761 +0000 UTC m=+0.143782927 container create 23e823f56e5e6b40692550da8d7186d88f5d522144bc21f73aed549b16895d1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T11:32:51.695 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:51 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/543254927' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:51.695 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:51 smithi090 ceph-mon[29575]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:52.088 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 11:32:51 smithi156 podman[63764]: 2024-04-25 11:32:51.620598244 +0000 UTC m=+0.388148406 container init 23e823f56e5e6b40692550da8d7186d88f5d522144bc21f73aed549b16895d1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-25T11:32:52.088 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 11:32:51 smithi156 podman[63764]: 2024-04-25 11:32:51.623120588 +0000 UTC m=+0.390670751 container start 23e823f56e5e6b40692550da8d7186d88f5d522144bc21f73aed549b16895d1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-04-25T11:32:52.089 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 11:32:51 smithi156 ceph-mds[63783]: -- 172.21.15.156:0/1489908675 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560072c4da40 con 0x560072c95600 2024-04-25T11:32:52.089 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 11:32:51 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-c[63779]: starting mds.c at 2024-04-25T11:32:52.089 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 11:32:51 smithi156 bash[63764]: 23e823f56e5e6b40692550da8d7186d88f5d522144bc21f73aed549b16895d1c 2024-04-25T11:32:52.089 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 11:32:51 smithi156 systemd[1]: Started Ceph mds.c for 58bd4e7a-02f6-11ef-bc93-c7b262605968. 2024-04-25T11:32:52.509 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:52.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.634+0000 7f66cbf1f640 1 -- 172.21.15.90:0/2631850491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66c4103c20 msgr2=0x7f66c4104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:52.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.634+0000 7f66cbf1f640 1 --2- 172.21.15.90:0/2631850491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66c4103c20 0x7f66c4104030 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f66b4009900 tx=0x7f66b402a410 comp rx=0 tx=0).stop 2024-04-25T11:32:52.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.634+0000 7f66cbf1f640 1 -- 172.21.15.90:0/2631850491 shutdown_connections 2024-04-25T11:32:52.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.634+0000 7f66cbf1f640 1 --2- 172.21.15.90:0/2631850491 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66c4108920 0x7f66c410ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:52.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.634+0000 7f66cbf1f640 1 --2- 172.21.15.90:0/2631850491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66c4103c20 0x7f66c4104030 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:52.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.634+0000 7f66cbf1f640 1 --2- 172.21.15.90:0/2631850491 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f66c41029e0 0x7f66c41036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:52.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.634+0000 7f66cbf1f640 1 -- 172.21.15.90:0/2631850491 >> 172.21.15.90:0/2631850491 conn(0x7f66c40fdbd0 msgr2=0x7f66c4100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:52.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.634+0000 7f66cbf1f640 1 -- 172.21.15.90:0/2631850491 shutdown_connections 2024-04-25T11:32:52.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.634+0000 7f66cbf1f640 1 -- 172.21.15.90:0/2631850491 wait complete. 2024-04-25T11:32:52.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.634+0000 7f66cbf1f640 1 Processor -- start 2024-04-25T11:32:52.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.635+0000 7f66cbf1f640 1 -- start start 2024-04-25T11:32:52.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.635+0000 7f66cbf1f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66c41029e0 0x7f66c4075860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:52.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.635+0000 7f66cbf1f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66c4103c20 0x7f66c419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:52.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.635+0000 7f66cbf1f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f66c4108920 0x7f66c41a14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:52.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.635+0000 7f66cbf1f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f66c410e510 con 0x7f66c41029e0 2024-04-25T11:32:52.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.635+0000 7f66cbf1f640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f66c410e390 con 0x7f66c4108920 2024-04-25T11:32:52.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.635+0000 7f66cbf1f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f66c410e690 con 0x7f66c4103c20 2024-04-25T11:32:52.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.635+0000 7f66c9cbb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66c41029e0 0x7f66c4075860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:52.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.635+0000 7f66c9cbb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66c41029e0 0x7f66c4075860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58558/0 (socket says 172.21.15.90:58558) 2024-04-25T11:32:52.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.635+0000 7f66c9cbb640 1 -- 172.21.15.90:0/1953249888 learned_addr learned my addr 172.21.15.90:0/1953249888 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:32:52.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.635+0000 7f66ca4bc640 1 --2- 172.21.15.90:0/1953249888 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f66c4108920 0x7f66c41a14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:52.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.635+0000 7f66c94ba640 1 --2- 172.21.15.90:0/1953249888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66c4103c20 0x7f66c419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:52.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.635+0000 7f66c9cbb640 1 -- 172.21.15.90:0/1953249888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66c4103c20 msgr2=0x7f66c419ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:52.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.635+0000 7f66c9cbb640 1 --2- 172.21.15.90:0/1953249888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66c4103c20 0x7f66c419ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:52.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.635+0000 7f66c9cbb640 1 -- 172.21.15.90:0/1953249888 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f66c4108920 msgr2=0x7f66c41a14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:52.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.635+0000 7f66c9cbb640 1 --2- 172.21.15.90:0/1953249888 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f66c4108920 0x7f66c41a14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:52.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.635+0000 7f66c9cbb640 1 -- 172.21.15.90:0/1953249888 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c41a1cf0 con 0x7f66c41029e0 2024-04-25T11:32:52.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.635+0000 7f66c94ba640 1 --2- 172.21.15.90:0/1953249888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66c4103c20 0x7f66c419ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:32:52.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.635+0000 7f66c9cbb640 1 --2- 172.21.15.90:0/1953249888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66c41029e0 0x7f66c4075860 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f66c000e9f0 tx=0x7f66c000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:52.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.635+0000 7f66baffd640 1 -- 172.21.15.90:0/1953249888 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c000ccc0 con 0x7f66c41029e0 2024-04-25T11:32:52.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.635+0000 7f66cbf1f640 1 -- 172.21.15.90:0/1953249888 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66c41a1fe0 con 0x7f66c41029e0 2024-04-25T11:32:52.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.635+0000 7f66baffd640 1 -- 172.21.15.90:0/1953249888 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66c00035f0 con 0x7f66c41029e0 2024-04-25T11:32:52.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.635+0000 7f66baffd640 1 -- 172.21.15.90:0/1953249888 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c000ce60 con 0x7f66c41029e0 2024-04-25T11:32:52.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.636+0000 7f66cbf1f640 1 -- 172.21.15.90:0/1953249888 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66c41a2440 con 0x7f66c41029e0 2024-04-25T11:32:52.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.636+0000 7f66cbf1f640 1 -- 172.21.15.90:0/1953249888 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66c406a190 con 0x7f66c41029e0 2024-04-25T11:32:52.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.638+0000 7f66baffd640 1 -- 172.21.15.90:0/1953249888 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f66c0003aa0 con 0x7f66c41029e0 2024-04-25T11:32:52.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.638+0000 7f66baffd640 1 --2- 172.21.15.90:0/1953249888 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f66b00afaf0 0x7f66b00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:52.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.639+0000 7f66c94ba640 1 --2- 172.21.15.90:0/1953249888 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f66b00afaf0 0x7f66b00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:52.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.639+0000 7f66baffd640 1 -- 172.21.15.90:0/1953249888 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f66c00debb0 con 0x7f66c41029e0 2024-04-25T11:32:52.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.639+0000 7f66c94ba640 1 --2- 172.21.15.90:0/1953249888 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f66b00afaf0 0x7f66b00b2000 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f66b4006010 tx=0x7f66b4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:52.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.642+0000 7f66baffd640 1 -- 172.21.15.90:0/1953249888 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f66c00a7c30 con 0x7f66c41029e0 2024-04-25T11:32:52.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.859+0000 7f66cbf1f640 1 -- 172.21.15.90:0/1953249888 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66c402dc10 con 0x7f66c41029e0 2024-04-25T11:32:52.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.860+0000 7f66baffd640 1 -- 172.21.15.90:0/1953249888 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6343 (secure 0 0 0) 0x7f66c00a7530 con 0x7f66c41029e0 2024-04-25T11:32:52.850 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:32:52.851 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-25T11:32:52.851 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 8 2024-04-25T11:32:52.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.862+0000 7f66cbf1f640 1 -- 172.21.15.90:0/1953249888 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f66b00afaf0 msgr2=0x7f66b00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:52.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.862+0000 7f66cbf1f640 1 --2- 172.21.15.90:0/1953249888 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f66b00afaf0 0x7f66b00b2000 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f66b4006010 tx=0x7f66b4000bc0 comp rx=0 tx=0).stop 2024-04-25T11:32:52.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.862+0000 7f66cbf1f640 1 -- 172.21.15.90:0/1953249888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66c41029e0 msgr2=0x7f66c4075860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:52.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.862+0000 7f66cbf1f640 1 --2- 172.21.15.90:0/1953249888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66c41029e0 0x7f66c4075860 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f66c000e9f0 tx=0x7f66c000eeb0 comp rx=0 tx=0).stop 2024-04-25T11:32:52.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.862+0000 7f66cbf1f640 1 -- 172.21.15.90:0/1953249888 shutdown_connections 2024-04-25T11:32:52.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.862+0000 7f66cbf1f640 1 --2- 172.21.15.90:0/1953249888 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f66c4108920 0x7f66c41a14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:52.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.862+0000 7f66cbf1f640 1 --2- 172.21.15.90:0/1953249888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66c4103c20 0x7f66c419ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:52.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.862+0000 7f66cbf1f640 1 --2- 172.21.15.90:0/1953249888 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f66b00afaf0 0x7f66b00b2000 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:52.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.862+0000 7f66cbf1f640 1 --2- 172.21.15.90:0/1953249888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66c41029e0 0x7f66c4075860 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:52.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.862+0000 7f66cbf1f640 1 -- 172.21.15.90:0/1953249888 >> 172.21.15.90:0/1953249888 conn(0x7f66c40fdbd0 msgr2=0x7f66c4104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:52.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.862+0000 7f66cbf1f640 1 -- 172.21.15.90:0/1953249888 shutdown_connections 2024-04-25T11:32:52.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:52.862+0000 7f66cbf1f640 1 -- 172.21.15.90:0/1953249888 wait complete. 2024-04-25T11:32:52.862 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:52 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:52.862 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:52 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:52.862 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:52 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:52.862 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:52 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:52.862 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:52 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:52.862 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:52 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:52.862 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:52 smithi090 ceph-mon[29575]: Deploying daemon mds.h on smithi138 2024-04-25T11:32:52.863 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:52 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.156:6836/3466257977,v1:172.21.15.156:6837/3466257977] up:boot 2024-04-25T11:32:52.863 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:52 smithi090 ceph-mon[29575]: fsmap 7 up:standby 2024-04-25T11:32:52.863 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:52 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T11:32:53.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:52 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:53.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:52 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:53.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:52 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:53.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:52 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:53.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:52 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:53.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:52 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:53.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:52 smithi138 ceph-mon[32328]: Deploying daemon mds.h on smithi138 2024-04-25T11:32:53.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:52 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.156:6836/3466257977,v1:172.21.15.156:6837/3466257977] up:boot 2024-04-25T11:32:53.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:52 smithi138 ceph-mon[32328]: fsmap 7 up:standby 2024-04-25T11:32:53.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:52 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T11:32:53.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:52 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:53.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:52 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:53.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:52 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:53.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:52 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:53.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:52 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:53.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:52 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:53.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:52 smithi156 ceph-mon[33045]: Deploying daemon mds.h on smithi138 2024-04-25T11:32:53.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:52 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.156:6836/3466257977,v1:172.21.15.156:6837/3466257977] up:boot 2024-04-25T11:32:53.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:52 smithi156 ceph-mon[33045]: fsmap 7 up:standby 2024-04-25T11:32:53.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:52 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T11:32:54.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:53 smithi138 ceph-mon[32328]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:54.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:53 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1953249888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:54.054 INFO:journalctl@ceph.mds.h.smithi138.stdout:Apr 25 11:32:53 smithi138 systemd[1]: Starting Ceph mds.h for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:32:54.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:53 smithi156 ceph-mon[33045]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:54.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:53 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1953249888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:54.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:53 smithi090 ceph-mon[29575]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:54.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:53 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1953249888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:54.371 INFO:journalctl@ceph.mds.h.smithi138.stdout:Apr 25 11:32:54 smithi138 podman[62232]: 2024-04-25 11:32:54.020560103 +0000 UTC m=+0.018098164 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T11:32:54.372 INFO:journalctl@ceph.mds.h.smithi138.stdout:Apr 25 11:32:54 smithi138 podman[62232]: 2024-04-25 11:32:54.12048817 +0000 UTC m=+0.118026274 container create e39188a45eb272d3118ebeea3732fe11c726730d713a33f660bc5288968e342d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-25T11:32:54.372 INFO:journalctl@ceph.mds.h.smithi138.stdout:Apr 25 11:32:54 smithi138 podman[62232]: 2024-04-25 11:32:54.347753158 +0000 UTC m=+0.345291219 container init e39188a45eb272d3118ebeea3732fe11c726730d713a33f660bc5288968e342d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T11:32:54.372 INFO:journalctl@ceph.mds.h.smithi138.stdout:Apr 25 11:32:54 smithi138 podman[62232]: 2024-04-25 11:32:54.350243822 +0000 UTC m=+0.347781887 container start e39188a45eb272d3118ebeea3732fe11c726730d713a33f660bc5288968e342d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-h, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-25T11:32:54.802 INFO:journalctl@ceph.mds.h.smithi138.stdout:Apr 25 11:32:54 smithi138 ceph-mds[62251]: -- 172.21.15.138:0/1159661032 <== mon.2 v2:172.21.15.138:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ddde551a40 con 0x55ddde599a80 2024-04-25T11:32:54.803 INFO:journalctl@ceph.mds.h.smithi138.stdout:Apr 25 11:32:54 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-h[62247]: starting mds.h at 2024-04-25T11:32:54.803 INFO:journalctl@ceph.mds.h.smithi138.stdout:Apr 25 11:32:54 smithi138 bash[62232]: e39188a45eb272d3118ebeea3732fe11c726730d713a33f660bc5288968e342d 2024-04-25T11:32:54.803 INFO:journalctl@ceph.mds.h.smithi138.stdout:Apr 25 11:32:54 smithi138 systemd[1]: Started Ceph mds.h for 58bd4e7a-02f6-11ef-bc93-c7b262605968. 2024-04-25T11:32:54.864 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:54.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.996+0000 7f793acd8640 1 -- 172.21.15.90:0/2347330340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f79341029e0 msgr2=0x7f79341036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:54.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.996+0000 7f793acd8640 1 --2- 172.21.15.90:0/2347330340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f79341029e0 0x7f79341036e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7928009900 tx=0x7f792802a410 comp rx=0 tx=0).stop 2024-04-25T11:32:54.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.996+0000 7f793acd8640 1 -- 172.21.15.90:0/2347330340 shutdown_connections 2024-04-25T11:32:54.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.996+0000 7f793acd8640 1 --2- 172.21.15.90:0/2347330340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7934108920 0x7f793410ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:54.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.996+0000 7f793acd8640 1 --2- 172.21.15.90:0/2347330340 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7934103c20 0x7f7934104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:54.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.996+0000 7f793acd8640 1 --2- 172.21.15.90:0/2347330340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f79341029e0 0x7f79341036e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:54.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.996+0000 7f793acd8640 1 -- 172.21.15.90:0/2347330340 >> 172.21.15.90:0/2347330340 conn(0x7f79340fdbd0 msgr2=0x7f7934100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:54.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.996+0000 7f793acd8640 1 -- 172.21.15.90:0/2347330340 shutdown_connections 2024-04-25T11:32:54.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.996+0000 7f793acd8640 1 -- 172.21.15.90:0/2347330340 wait complete. 2024-04-25T11:32:54.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.997+0000 7f793acd8640 1 Processor -- start 2024-04-25T11:32:54.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.997+0000 7f793acd8640 1 -- start start 2024-04-25T11:32:54.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.997+0000 7f793acd8640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f79341029e0 0x7f793419a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:54.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.997+0000 7f793acd8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7934103c20 0x7f793419a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:54.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.997+0000 7f793acd8640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7934108920 0x7f79341a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:54.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.997+0000 7f793acd8640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f793410f610 con 0x7f7934108920 2024-04-25T11:32:54.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.997+0000 7f793acd8640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f793410f490 con 0x7f79341029e0 2024-04-25T11:32:54.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.997+0000 7f793acd8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f793410f790 con 0x7f7934103c20 2024-04-25T11:32:54.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.997+0000 7f7939275640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7934108920 0x7f79341a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:54.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.997+0000 7f7939275640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7934108920 0x7f79341a1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58576/0 (socket says 172.21.15.90:58576) 2024-04-25T11:32:54.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.997+0000 7f7933fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7934103c20 0x7f793419a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:54.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.997+0000 7f7939275640 1 -- 172.21.15.90:0/591422110 learned_addr learned my addr 172.21.15.90:0/591422110 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:32:54.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.997+0000 7f7938a74640 1 --2- 172.21.15.90:0/591422110 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f79341029e0 0x7f793419a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:54.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.997+0000 7f7939275640 1 -- 172.21.15.90:0/591422110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7934103c20 msgr2=0x7f793419a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:54.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.997+0000 7f7939275640 1 --2- 172.21.15.90:0/591422110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7934103c20 0x7f793419a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:54.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.997+0000 7f7939275640 1 -- 172.21.15.90:0/591422110 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f79341029e0 msgr2=0x7f793419a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:54.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.997+0000 7f7939275640 1 --2- 172.21.15.90:0/591422110 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f79341029e0 0x7f793419a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:54.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.997+0000 7f7939275640 1 -- 172.21.15.90:0/591422110 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79341a1dc0 con 0x7f7934108920 2024-04-25T11:32:54.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.997+0000 7f7933fff640 1 --2- 172.21.15.90:0/591422110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7934103c20 0x7f793419a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:32:54.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.998+0000 7f7939275640 1 --2- 172.21.15.90:0/591422110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7934108920 0x7f79341a1750 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f792c00eb10 tx=0x7f792c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:54.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.998+0000 7f7931ffb640 1 -- 172.21.15.90:0/591422110 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792c004b20 con 0x7f7934108920 2024-04-25T11:32:54.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.998+0000 7f7931ffb640 1 -- 172.21.15.90:0/591422110 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f792c004cc0 con 0x7f7934108920 2024-04-25T11:32:54.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.998+0000 7f793acd8640 1 -- 172.21.15.90:0/591422110 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79341a20b0 con 0x7f7934108920 2024-04-25T11:32:54.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.998+0000 7f7931ffb640 1 -- 172.21.15.90:0/591422110 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792c010670 con 0x7f7934108920 2024-04-25T11:32:54.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.998+0000 7f793acd8640 1 -- 172.21.15.90:0/591422110 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79341a2650 con 0x7f7934108920 2024-04-25T11:32:54.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:54.999+0000 7f793acd8640 1 -- 172.21.15.90:0/591422110 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7934107730 con 0x7f7934108920 2024-04-25T11:32:54.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:55.000+0000 7f7931ffb640 1 -- 172.21.15.90:0/591422110 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f792c005020 con 0x7f7934108920 2024-04-25T11:32:54.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:55.001+0000 7f7931ffb640 1 --2- 172.21.15.90:0/591422110 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f79240afb70 0x7f79240b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:54.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:55.001+0000 7f7938a74640 1 --2- 172.21.15.90:0/591422110 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f79240afb70 0x7f79240b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:54.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:55.001+0000 7f7931ffb640 1 -- 172.21.15.90:0/591422110 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f792c0dec70 con 0x7f7934108920 2024-04-25T11:32:54.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:55.001+0000 7f7938a74640 1 --2- 172.21.15.90:0/591422110 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f79240afb70 0x7f79240b2080 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7928005ab0 tx=0x7f7928031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:54.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:55.006+0000 7f7931ffb640 1 -- 172.21.15.90:0/591422110 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f792c0a7cf0 con 0x7f7934108920 2024-04-25T11:32:55.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:55.223+0000 7f793acd8640 1 -- 172.21.15.90:0/591422110 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f793402db50 con 0x7f7934108920 2024-04-25T11:32:55.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:55.224+0000 7f7931ffb640 1 -- 172.21.15.90:0/591422110 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7171 (secure 0 0 0) 0x7f792c0a75f0 con 0x7f7934108920 2024-04-25T11:32:55.213 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:32:55.214 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-25T11:32:55.214 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 9 2024-04-25T11:32:55.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:55.226+0000 7f793acd8640 1 -- 172.21.15.90:0/591422110 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f79240afb70 msgr2=0x7f79240b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:55.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:55.226+0000 7f793acd8640 1 --2- 172.21.15.90:0/591422110 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f79240afb70 0x7f79240b2080 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7928005ab0 tx=0x7f7928031400 comp rx=0 tx=0).stop 2024-04-25T11:32:55.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:55.226+0000 7f793acd8640 1 -- 172.21.15.90:0/591422110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7934108920 msgr2=0x7f79341a1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:55.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:55.226+0000 7f793acd8640 1 --2- 172.21.15.90:0/591422110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7934108920 0x7f79341a1750 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f792c00eb10 tx=0x7f792c00c2d0 comp rx=0 tx=0).stop 2024-04-25T11:32:55.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:55.226+0000 7f793acd8640 1 -- 172.21.15.90:0/591422110 shutdown_connections 2024-04-25T11:32:55.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:55.226+0000 7f793acd8640 1 --2- 172.21.15.90:0/591422110 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f79240afb70 0x7f79240b2080 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:55.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:55.226+0000 7f793acd8640 1 --2- 172.21.15.90:0/591422110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7934108920 0x7f79341a1750 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:55.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:55.226+0000 7f793acd8640 1 --2- 172.21.15.90:0/591422110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7934103c20 0x7f793419a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:55.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:55.226+0000 7f793acd8640 1 --2- 172.21.15.90:0/591422110 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f79341029e0 0x7f793419a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:55.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:55.226+0000 7f793acd8640 1 -- 172.21.15.90:0/591422110 >> 172.21.15.90:0/591422110 conn(0x7f79340fdbd0 msgr2=0x7f7934104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:55.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:55.226+0000 7f793acd8640 1 -- 172.21.15.90:0/591422110 shutdown_connections 2024-04-25T11:32:55.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:55.226+0000 7f793acd8640 1 -- 172.21.15.90:0/591422110 wait complete. 2024-04-25T11:32:55.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:55 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:55.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:55 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:55.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:55 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:55.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:55 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:55.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:55 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:55.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:55 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:55.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:55 smithi090 ceph-mon[29575]: Deploying daemon mds.i on smithi156 2024-04-25T11:32:55.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:55 smithi090 ceph-mon[29575]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:55.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:55 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.138:6836/2598829089,v1:172.21.15.138:6837/2598829089] up:boot 2024-04-25T11:32:55.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:55 smithi090 ceph-mon[29575]: fsmap 8 up:standby 2024-04-25T11:32:55.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:55 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T11:32:55.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:55 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/591422110' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:55.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:55 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:55.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:55 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:55.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:55 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:55.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:55 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:55.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:55 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:55.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:55 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:55.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:55 smithi138 ceph-mon[32328]: Deploying daemon mds.i on smithi156 2024-04-25T11:32:55.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:55 smithi138 ceph-mon[32328]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:55.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:55 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.138:6836/2598829089,v1:172.21.15.138:6837/2598829089] up:boot 2024-04-25T11:32:55.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:55 smithi138 ceph-mon[32328]: fsmap 8 up:standby 2024-04-25T11:32:55.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:55 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T11:32:55.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:55 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/591422110' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:55.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:55 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:55.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:55 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:55.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:55 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:55.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:55 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:55.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:55 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:55.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:55 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:55.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:55 smithi156 ceph-mon[33045]: Deploying daemon mds.i on smithi156 2024-04-25T11:32:55.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:55 smithi156 ceph-mon[33045]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:55.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:55 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.138:6836/2598829089,v1:172.21.15.138:6837/2598829089] up:boot 2024-04-25T11:32:55.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:55 smithi156 ceph-mon[33045]: fsmap 8 up:standby 2024-04-25T11:32:55.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:55 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T11:32:55.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:55 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/591422110' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:56.745 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 11:32:56 smithi156 systemd[1]: Starting Ceph mds.i for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:32:56.995 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 11:32:56 smithi156 podman[64072]: 2024-04-25 11:32:56.659451445 +0000 UTC m=+0.018310627 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T11:32:56.996 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 11:32:56 smithi156 podman[64072]: 2024-04-25 11:32:56.762893639 +0000 UTC m=+0.121752809 container create e24b5c5c6c38b0d12aec2103648a4c3385d794a2540e134784da2ce4ef43a33f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-i, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-25T11:32:56.996 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 11:32:57 smithi156 podman[64072]: 2024-04-25 11:32:57.007314842 +0000 UTC m=+0.366174009 container init e24b5c5c6c38b0d12aec2103648a4c3385d794a2540e134784da2ce4ef43a33f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T11:32:57.227 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:57.338 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 11:32:57 smithi156 podman[64072]: 2024-04-25 11:32:57.010116102 +0000 UTC m=+0.368975282 container start e24b5c5c6c38b0d12aec2103648a4c3385d794a2540e134784da2ce4ef43a33f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-i, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T11:32:57.338 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 11:32:57 smithi156 ceph-mds[64091]: -- 172.21.15.156:0/1973724268 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55655185da40 con 0x5565518a5a80 2024-04-25T11:32:57.338 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 11:32:57 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-i[64087]: starting mds.i at 2024-04-25T11:32:57.338 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 11:32:57 smithi156 bash[64072]: e24b5c5c6c38b0d12aec2103648a4c3385d794a2540e134784da2ce4ef43a33f 2024-04-25T11:32:57.339 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 11:32:57 smithi156 systemd[1]: Started Ceph mds.i for 58bd4e7a-02f6-11ef-bc93-c7b262605968. 2024-04-25T11:32:57.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.351+0000 7f1d89ce9640 1 -- 172.21.15.90:0/4039833873 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d84103c20 msgr2=0x7f1d84104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:57.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.351+0000 7f1d89ce9640 1 --2- 172.21.15.90:0/4039833873 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d84103c20 0x7f1d84104030 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1d78009900 tx=0x7f1d7802a3f0 comp rx=0 tx=0).stop 2024-04-25T11:32:57.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.351+0000 7f1d89ce9640 1 -- 172.21.15.90:0/4039833873 shutdown_connections 2024-04-25T11:32:57.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.351+0000 7f1d89ce9640 1 --2- 172.21.15.90:0/4039833873 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1d84108920 0x7f1d8410ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:57.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.351+0000 7f1d89ce9640 1 --2- 172.21.15.90:0/4039833873 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d84103c20 0x7f1d84104030 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:57.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.351+0000 7f1d89ce9640 1 --2- 172.21.15.90:0/4039833873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d841029e0 0x7f1d841036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:57.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.351+0000 7f1d89ce9640 1 -- 172.21.15.90:0/4039833873 >> 172.21.15.90:0/4039833873 conn(0x7f1d840fdbd0 msgr2=0x7f1d84100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:57.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.351+0000 7f1d89ce9640 1 -- 172.21.15.90:0/4039833873 shutdown_connections 2024-04-25T11:32:57.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.351+0000 7f1d89ce9640 1 -- 172.21.15.90:0/4039833873 wait complete. 2024-04-25T11:32:57.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.352+0000 7f1d89ce9640 1 Processor -- start 2024-04-25T11:32:57.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.352+0000 7f1d89ce9640 1 -- start start 2024-04-25T11:32:57.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.352+0000 7f1d89ce9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d841029e0 0x7f1d841991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:57.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.352+0000 7f1d89ce9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d84103c20 0x7f1d841996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:57.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.352+0000 7f1d89ce9640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1d84108920 0x7f1d841a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:57.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.352+0000 7f1d89ce9640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1d8410e500 con 0x7f1d841029e0 2024-04-25T11:32:57.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.352+0000 7f1d89ce9640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1d8410e380 con 0x7f1d84108920 2024-04-25T11:32:57.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.352+0000 7f1d89ce9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1d8410e680 con 0x7f1d84103c20 2024-04-25T11:32:57.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.352+0000 7f1d837fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d841029e0 0x7f1d841991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:57.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.352+0000 7f1d837fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d841029e0 0x7f1d841991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58596/0 (socket says 172.21.15.90:58596) 2024-04-25T11:32:57.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.352+0000 7f1d837fe640 1 -- 172.21.15.90:0/285376514 learned_addr learned my addr 172.21.15.90:0/285376514 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:32:57.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.352+0000 7f1d82ffd640 1 --2- 172.21.15.90:0/285376514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d84103c20 0x7f1d841996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:57.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.352+0000 7f1d83fff640 1 --2- 172.21.15.90:0/285376514 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1d84108920 0x7f1d841a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:57.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.352+0000 7f1d837fe640 1 -- 172.21.15.90:0/285376514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d84103c20 msgr2=0x7f1d841996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:57.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.352+0000 7f1d837fe640 1 --2- 172.21.15.90:0/285376514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d84103c20 0x7f1d841996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:57.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.352+0000 7f1d837fe640 1 -- 172.21.15.90:0/285376514 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1d84108920 msgr2=0x7f1d841a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:57.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.352+0000 7f1d837fe640 1 --2- 172.21.15.90:0/285376514 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1d84108920 0x7f1d841a0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:57.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.352+0000 7f1d837fe640 1 -- 172.21.15.90:0/285376514 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d841a0c30 con 0x7f1d841029e0 2024-04-25T11:32:57.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.353+0000 7f1d837fe640 1 --2- 172.21.15.90:0/285376514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d841029e0 0x7f1d841991a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1d7400c970 tx=0x7f1d7400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:57.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.353+0000 7f1d80ff9640 1 -- 172.21.15.90:0/285376514 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d74004220 con 0x7f1d841029e0 2024-04-25T11:32:57.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.353+0000 7f1d89ce9640 1 -- 172.21.15.90:0/285376514 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d841a0f20 con 0x7f1d841029e0 2024-04-25T11:32:57.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.353+0000 7f1d80ff9640 1 -- 172.21.15.90:0/285376514 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d740043c0 con 0x7f1d841029e0 2024-04-25T11:32:57.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.353+0000 7f1d80ff9640 1 -- 172.21.15.90:0/285376514 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d74004a00 con 0x7f1d841029e0 2024-04-25T11:32:57.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.353+0000 7f1d89ce9640 1 -- 172.21.15.90:0/285376514 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d841a1380 con 0x7f1d841029e0 2024-04-25T11:32:57.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.354+0000 7f1d89ce9640 1 -- 172.21.15.90:0/285376514 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d8406a190 con 0x7f1d841029e0 2024-04-25T11:32:57.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.356+0000 7f1d80ff9640 1 -- 172.21.15.90:0/285376514 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f1d740079b0 con 0x7f1d841029e0 2024-04-25T11:32:57.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.357+0000 7f1d80ff9640 1 --2- 172.21.15.90:0/285376514 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f1d6c0afaf0 0x7f1d6c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:57.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.357+0000 7f1d82ffd640 1 --2- 172.21.15.90:0/285376514 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f1d6c0afaf0 0x7f1d6c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:57.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.357+0000 7f1d80ff9640 1 -- 172.21.15.90:0/285376514 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f1d740de960 con 0x7f1d841029e0 2024-04-25T11:32:57.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.357+0000 7f1d82ffd640 1 --2- 172.21.15.90:0/285376514 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f1d6c0afaf0 0x7f1d6c0b2000 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1d78009230 tx=0x7f1d78031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:57.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.360+0000 7f1d80ff9640 1 -- 172.21.15.90:0/285376514 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1d74014030 con 0x7f1d841029e0 2024-04-25T11:32:57.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.585+0000 7f1d89ce9640 1 -- 172.21.15.90:0/285376514 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d84107780 con 0x7f1d841029e0 2024-04-25T11:32:57.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.585+0000 7f1d80ff9640 1 -- 172.21.15.90:0/285376514 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7171 (secure 0 0 0) 0x7f1d84107780 con 0x7f1d841029e0 2024-04-25T11:32:57.574 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:32:57.574 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-25T11:32:57.575 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 9 2024-04-25T11:32:57.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.587+0000 7f1d89ce9640 1 -- 172.21.15.90:0/285376514 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f1d6c0afaf0 msgr2=0x7f1d6c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:57.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.587+0000 7f1d89ce9640 1 --2- 172.21.15.90:0/285376514 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f1d6c0afaf0 0x7f1d6c0b2000 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1d78009230 tx=0x7f1d78031400 comp rx=0 tx=0).stop 2024-04-25T11:32:57.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.587+0000 7f1d89ce9640 1 -- 172.21.15.90:0/285376514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d841029e0 msgr2=0x7f1d841991a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:57.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.587+0000 7f1d89ce9640 1 --2- 172.21.15.90:0/285376514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d841029e0 0x7f1d841991a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1d7400c970 tx=0x7f1d7400ce30 comp rx=0 tx=0).stop 2024-04-25T11:32:57.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.587+0000 7f1d89ce9640 1 -- 172.21.15.90:0/285376514 shutdown_connections 2024-04-25T11:32:57.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.588+0000 7f1d89ce9640 1 --2- 172.21.15.90:0/285376514 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1d84108920 0x7f1d841a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:57.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.588+0000 7f1d89ce9640 1 --2- 172.21.15.90:0/285376514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d84103c20 0x7f1d841996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:57.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.588+0000 7f1d89ce9640 1 --2- 172.21.15.90:0/285376514 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f1d6c0afaf0 0x7f1d6c0b2000 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:57.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.588+0000 7f1d89ce9640 1 --2- 172.21.15.90:0/285376514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d841029e0 0x7f1d841991a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:57.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.588+0000 7f1d89ce9640 1 -- 172.21.15.90:0/285376514 >> 172.21.15.90:0/285376514 conn(0x7f1d840fdbd0 msgr2=0x7f1d84104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:57.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.588+0000 7f1d89ce9640 1 -- 172.21.15.90:0/285376514 shutdown_connections 2024-04-25T11:32:57.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:57.588+0000 7f1d89ce9640 1 -- 172.21.15.90:0/285376514 wait complete. 2024-04-25T11:32:57.840 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:57 smithi090 ceph-mon[29575]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:57.840 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:57 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:57.840 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:57 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:57.840 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:57 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:57.840 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:57 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:57.840 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:57 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:57.840 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:57 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:57.840 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:57 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/285376514' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:58.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:57 smithi138 ceph-mon[32328]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:58.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:57 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:58.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:57 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:58.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:57 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:58.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:57 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:58.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:57 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:58.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:57 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:58.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:57 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/285376514' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:58.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:57 smithi156 ceph-mon[33045]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:58.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:57 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:58.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:57 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:58.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:57 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:32:58.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:57 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:32:58.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:57 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:32:58.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:57 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:32:58.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:57 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/285376514' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:32:59.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:58 smithi138 ceph-mon[32328]: Deploying daemon mds.g on smithi090 2024-04-25T11:32:59.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:58 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] up:boot 2024-04-25T11:32:59.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:58 smithi138 ceph-mon[32328]: fsmap 9 up:standby 2024-04-25T11:32:59.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:58 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T11:32:59.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:58 smithi156 ceph-mon[33045]: Deploying daemon mds.g on smithi090 2024-04-25T11:32:59.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:58 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] up:boot 2024-04-25T11:32:59.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:58 smithi156 ceph-mon[33045]: fsmap 9 up:standby 2024-04-25T11:32:59.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:58 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T11:32:59.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:58 smithi090 ceph-mon[29575]: Deploying daemon mds.g on smithi090 2024-04-25T11:32:59.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:58 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] up:boot 2024-04-25T11:32:59.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:58 smithi090 ceph-mon[29575]: fsmap 9 up:standby 2024-04-25T11:32:59.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:58 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T11:32:59.476 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 25 11:32:59 smithi090 systemd[1]: Starting Ceph mds.g for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:32:59.588 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:59.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.711+0000 7f83bc2ee640 1 -- 172.21.15.90:0/2256246904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83b4103c00 msgr2=0x7f83b4104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:59.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.711+0000 7f83bc2ee640 1 --2- 172.21.15.90:0/2256246904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83b4103c00 0x7f83b4104010 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f83a4009900 tx=0x7f83a402a410 comp rx=0 tx=0).stop 2024-04-25T11:32:59.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.711+0000 7f83bc2ee640 1 -- 172.21.15.90:0/2256246904 shutdown_connections 2024-04-25T11:32:59.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.711+0000 7f83bc2ee640 1 --2- 172.21.15.90:0/2256246904 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f83b4108900 0x7f83b410ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:59.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.711+0000 7f83bc2ee640 1 --2- 172.21.15.90:0/2256246904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83b4103c00 0x7f83b4104010 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:59.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.711+0000 7f83bc2ee640 1 --2- 172.21.15.90:0/2256246904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83b41029c0 0x7f83b41036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:59.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.711+0000 7f83bc2ee640 1 -- 172.21.15.90:0/2256246904 >> 172.21.15.90:0/2256246904 conn(0x7f83b40fdbd0 msgr2=0x7f83b4100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:59.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.711+0000 7f83bc2ee640 1 -- 172.21.15.90:0/2256246904 shutdown_connections 2024-04-25T11:32:59.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.711+0000 7f83bc2ee640 1 -- 172.21.15.90:0/2256246904 wait complete. 2024-04-25T11:32:59.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.712+0000 7f83bc2ee640 1 Processor -- start 2024-04-25T11:32:59.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.712+0000 7f83bc2ee640 1 -- start start 2024-04-25T11:32:59.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.712+0000 7f83bc2ee640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f83b41029c0 0x7f83b419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:59.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.712+0000 7f83bc2ee640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83b4103c00 0x7f83b419a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:59.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.712+0000 7f83bc2ee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83b4108900 0x7f83b41a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:59.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.713+0000 7f83bc2ee640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f83b410e560 con 0x7f83b4103c00 2024-04-25T11:32:59.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.713+0000 7f83bc2ee640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f83b410e3e0 con 0x7f83b41029c0 2024-04-25T11:32:59.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.713+0000 7f83bc2ee640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f83b410e6e0 con 0x7f83b4108900 2024-04-25T11:32:59.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.713+0000 7f83b9889640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83b4103c00 0x7f83b419a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:59.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.713+0000 7f83b9889640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83b4103c00 0x7f83b419a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58622/0 (socket says 172.21.15.90:58622) 2024-04-25T11:32:59.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.713+0000 7f83b9889640 1 -- 172.21.15.90:0/724250798 learned_addr learned my addr 172.21.15.90:0/724250798 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:32:59.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.713+0000 7f83ba08a640 1 --2- 172.21.15.90:0/724250798 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f83b41029c0 0x7f83b419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:59.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.713+0000 7f83ba88b640 1 --2- 172.21.15.90:0/724250798 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83b4108900 0x7f83b41a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:59.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.713+0000 7f83b9889640 1 -- 172.21.15.90:0/724250798 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83b4108900 msgr2=0x7f83b41a1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:59.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.713+0000 7f83b9889640 1 --2- 172.21.15.90:0/724250798 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83b4108900 0x7f83b41a1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:59.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.713+0000 7f83b9889640 1 -- 172.21.15.90:0/724250798 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f83b41029c0 msgr2=0x7f83b419a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:59.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.713+0000 7f83b9889640 1 --2- 172.21.15.90:0/724250798 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f83b41029c0 0x7f83b419a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:59.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.713+0000 7f83b9889640 1 -- 172.21.15.90:0/724250798 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83b41a1dc0 con 0x7f83b4103c00 2024-04-25T11:32:59.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.713+0000 7f83b9889640 1 --2- 172.21.15.90:0/724250798 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83b4103c00 0x7f83b419a830 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f83a4006010 tx=0x7f83a4031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:59.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.713+0000 7f83ab7fe640 1 -- 172.21.15.90:0/724250798 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a403f030 con 0x7f83b4103c00 2024-04-25T11:32:59.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.713+0000 7f83bc2ee640 1 -- 172.21.15.90:0/724250798 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83b41a2050 con 0x7f83b4103c00 2024-04-25T11:32:59.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.713+0000 7f83ab7fe640 1 -- 172.21.15.90:0/724250798 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83a4003aa0 con 0x7f83b4103c00 2024-04-25T11:32:59.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.713+0000 7f83ab7fe640 1 -- 172.21.15.90:0/724250798 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a4035070 con 0x7f83b4103c00 2024-04-25T11:32:59.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.713+0000 7f83bc2ee640 1 -- 172.21.15.90:0/724250798 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83b41a2610 con 0x7f83b4103c00 2024-04-25T11:32:59.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.715+0000 7f83bc2ee640 1 -- 172.21.15.90:0/724250798 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83b41a2870 con 0x7f83b4103c00 2024-04-25T11:32:59.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.717+0000 7f83ab7fe640 1 -- 172.21.15.90:0/724250798 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f83a4042030 con 0x7f83b4103c00 2024-04-25T11:32:59.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.718+0000 7f83ab7fe640 1 --2- 172.21.15.90:0/724250798 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f83a00afaf0 0x7f83a00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:59.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.718+0000 7f83ba08a640 1 --2- 172.21.15.90:0/724250798 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f83a00afaf0 0x7f83a00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:59.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.718+0000 7f83ab7fe640 1 -- 172.21.15.90:0/724250798 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f83a40ff490 con 0x7f83b4103c00 2024-04-25T11:32:59.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.719+0000 7f83ba08a640 1 --2- 172.21.15.90:0/724250798 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f83a00afaf0 0x7f83a00b2000 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f83b41035c0 tx=0x7f83b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:59.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.721+0000 7f83ab7fe640 1 -- 172.21.15.90:0/724250798 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f83a40c8470 con 0x7f83b4103c00 2024-04-25T11:32:59.842 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:32:59 smithi090 ceph-mon[29575]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:32:59.842 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 25 11:32:59 smithi090 podman[70951]: 2024-04-25 11:32:59.483711682 +0000 UTC m=+0.018292269 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T11:32:59.842 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 25 11:32:59 smithi090 podman[70951]: 2024-04-25 11:32:59.599726 +0000 UTC m=+0.134306600 container create d9063c55f074c836cac7fc4c59d478c8049ff82b830f8f07d9d8bf71dc6e0ba0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-g, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-25T11:32:59.842 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 25 11:32:59 smithi090 podman[70951]: 2024-04-25 11:32:59.853885921 +0000 UTC m=+0.388466490 container init d9063c55f074c836cac7fc4c59d478c8049ff82b830f8f07d9d8bf71dc6e0ba0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-g, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True) 2024-04-25T11:32:59.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.944+0000 7f83bc2ee640 1 -- 172.21.15.90:0/724250798 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83b41a2a40 con 0x7f83b4103c00 2024-04-25T11:32:59.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.944+0000 7f83ab7fe640 1 -- 172.21.15.90:0/724250798 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8001 (secure 0 0 0) 0x7f83a40c7d70 con 0x7f83b4103c00 2024-04-25T11:32:59.933 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:32:59.934 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T11:32:59.934 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 10 2024-04-25T11:32:59.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.946+0000 7f83bc2ee640 1 -- 172.21.15.90:0/724250798 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f83a00afaf0 msgr2=0x7f83a00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:59.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.946+0000 7f83bc2ee640 1 --2- 172.21.15.90:0/724250798 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f83a00afaf0 0x7f83a00b2000 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f83b41035c0 tx=0x7f83b0005f90 comp rx=0 tx=0).stop 2024-04-25T11:32:59.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.947+0000 7f83bc2ee640 1 -- 172.21.15.90:0/724250798 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83b4103c00 msgr2=0x7f83b419a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:59.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.947+0000 7f83bc2ee640 1 --2- 172.21.15.90:0/724250798 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83b4103c00 0x7f83b419a830 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f83a4006010 tx=0x7f83a4031a50 comp rx=0 tx=0).stop 2024-04-25T11:32:59.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.947+0000 7f83bc2ee640 1 -- 172.21.15.90:0/724250798 shutdown_connections 2024-04-25T11:32:59.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.947+0000 7f83bc2ee640 1 --2- 172.21.15.90:0/724250798 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83b4108900 0x7f83b41a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:59.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.947+0000 7f83bc2ee640 1 --2- 172.21.15.90:0/724250798 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f83a00afaf0 0x7f83a00b2000 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:59.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.947+0000 7f83bc2ee640 1 --2- 172.21.15.90:0/724250798 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83b4103c00 0x7f83b419a830 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:59.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.947+0000 7f83bc2ee640 1 --2- 172.21.15.90:0/724250798 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f83b41029c0 0x7f83b419a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:59.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.947+0000 7f83bc2ee640 1 -- 172.21.15.90:0/724250798 >> 172.21.15.90:0/724250798 conn(0x7f83b40fdbd0 msgr2=0x7f83b4109dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:59.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.947+0000 7f83bc2ee640 1 -- 172.21.15.90:0/724250798 shutdown_connections 2024-04-25T11:32:59.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:32:59.947+0000 7f83bc2ee640 1 -- 172.21.15.90:0/724250798 wait complete. 2024-04-25T11:33:00.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:32:59 smithi138 ceph-mon[32328]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:33:00.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:32:59 smithi156 ceph-mon[33045]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:33:00.226 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 25 11:32:59 smithi090 podman[70951]: 2024-04-25 11:32:59.857144666 +0000 UTC m=+0.391725235 container start d9063c55f074c836cac7fc4c59d478c8049ff82b830f8f07d9d8bf71dc6e0ba0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-g, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T11:33:00.226 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 25 11:32:59 smithi090 ceph-mds[71009]: -- 172.21.15.90:0/717308267 <== mon.0 v2:172.21.15.90:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f209f1ba40 con 0x55f209f63180 2024-04-25T11:33:00.226 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 25 11:32:59 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-g[70996]: starting mds.g at 2024-04-25T11:33:00.226 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 25 11:32:59 smithi090 bash[70951]: d9063c55f074c836cac7fc4c59d478c8049ff82b830f8f07d9d8bf71dc6e0ba0 2024-04-25T11:33:00.226 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 25 11:32:59 smithi090 systemd[1]: Started Ceph mds.g for 58bd4e7a-02f6-11ef-bc93-c7b262605968. 2024-04-25T11:33:00.899 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:00 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/724250798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:33:00.899 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:00 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:00.899 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:00 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:00.899 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:00 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:00.899 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:00 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:33:00.899 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:00 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:33:00.899 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:00 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:33:01.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:00 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/724250798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:33:01.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:00 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:01.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:00 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:01.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:00 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:01.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:00 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:33:01.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:00 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:33:01.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:00 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:33:01.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:00 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/724250798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:33:01.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:01.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:01.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:01.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:33:01.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:33:01.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:00 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:33:01.882 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:01 smithi090 ceph-mon[29575]: Deploying daemon mds.a on smithi090 2024-04-25T11:33:01.882 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:01 smithi090 ceph-mon[29575]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:33:01.883 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:01 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.90:6838/3101160958,v1:172.21.15.90:6839/3101160958] up:boot 2024-04-25T11:33:01.883 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:01 smithi090 ceph-mon[29575]: fsmap 10 up:standby 2024-04-25T11:33:01.883 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:01 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T11:33:01.947 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:33:02.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:01 smithi138 ceph-mon[32328]: Deploying daemon mds.a on smithi090 2024-04-25T11:33:02.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:01 smithi138 ceph-mon[32328]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:33:02.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:01 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.90:6838/3101160958,v1:172.21.15.90:6839/3101160958] up:boot 2024-04-25T11:33:02.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:01 smithi138 ceph-mon[32328]: fsmap 10 up:standby 2024-04-25T11:33:02.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:01 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T11:33:02.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:01 smithi156 ceph-mon[33045]: Deploying daemon mds.a on smithi090 2024-04-25T11:33:02.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:01 smithi156 ceph-mon[33045]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:33:02.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:01 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.90:6838/3101160958,v1:172.21.15.90:6839/3101160958] up:boot 2024-04-25T11:33:02.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:01 smithi156 ceph-mon[33045]: fsmap 10 up:standby 2024-04-25T11:33:02.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:01 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T11:33:02.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.151+0000 7f632c184640 1 -- 172.21.15.90:0/4166327921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6324103c00 msgr2=0x7f6324104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:02.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.151+0000 7f632c184640 1 --2- 172.21.15.90:0/4166327921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6324103c00 0x7f6324104010 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6314009900 tx=0x7f631402a410 comp rx=0 tx=0).stop 2024-04-25T11:33:02.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.151+0000 7f632c184640 1 -- 172.21.15.90:0/4166327921 shutdown_connections 2024-04-25T11:33:02.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.151+0000 7f632c184640 1 --2- 172.21.15.90:0/4166327921 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6324108900 0x7f632410ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:02.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.151+0000 7f632c184640 1 --2- 172.21.15.90:0/4166327921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6324103c00 0x7f6324104010 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:02.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.151+0000 7f632c184640 1 --2- 172.21.15.90:0/4166327921 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f63241029c0 0x7f63241036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:02.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.151+0000 7f632c184640 1 -- 172.21.15.90:0/4166327921 >> 172.21.15.90:0/4166327921 conn(0x7f63240fdbd0 msgr2=0x7f6324100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:02.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.151+0000 7f632c184640 1 -- 172.21.15.90:0/4166327921 shutdown_connections 2024-04-25T11:33:02.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.151+0000 7f632c184640 1 -- 172.21.15.90:0/4166327921 wait complete. 2024-04-25T11:33:02.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.152+0000 7f632c184640 1 Processor -- start 2024-04-25T11:33:02.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.152+0000 7f632c184640 1 -- start start 2024-04-25T11:33:02.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.152+0000 7f632c184640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63241029c0 0x7f632419a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:02.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.152+0000 7f632c184640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6324103c00 0x7f632419a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:02.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.152+0000 7f632c184640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6324108900 0x7f63241a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:02.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.152+0000 7f632c184640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f632410e4f0 con 0x7f6324103c00 2024-04-25T11:33:02.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.152+0000 7f632c184640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f632410e370 con 0x7f6324108900 2024-04-25T11:33:02.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.152+0000 7f632c184640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f632410e670 con 0x7f63241029c0 2024-04-25T11:33:02.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.152+0000 7f632971f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6324103c00 0x7f632419a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:02.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.152+0000 7f632971f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6324103c00 0x7f632419a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52862/0 (socket says 172.21.15.90:52862) 2024-04-25T11:33:02.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.152+0000 7f632971f640 1 -- 172.21.15.90:0/3974030108 learned_addr learned my addr 172.21.15.90:0/3974030108 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:02.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.152+0000 7f632a721640 1 --2- 172.21.15.90:0/3974030108 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6324108900 0x7f63241a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:02.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.152+0000 7f6329f20640 1 --2- 172.21.15.90:0/3974030108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63241029c0 0x7f632419a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:02.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.152+0000 7f632971f640 1 -- 172.21.15.90:0/3974030108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63241029c0 msgr2=0x7f632419a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:02.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.152+0000 7f632971f640 1 --2- 172.21.15.90:0/3974030108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63241029c0 0x7f632419a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:02.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.152+0000 7f632971f640 1 -- 172.21.15.90:0/3974030108 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6324108900 msgr2=0x7f63241a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:02.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.152+0000 7f632971f640 1 --2- 172.21.15.90:0/3974030108 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6324108900 0x7f63241a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:02.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.153+0000 7f632971f640 1 -- 172.21.15.90:0/3974030108 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63241a1d30 con 0x7f6324103c00 2024-04-25T11:33:02.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.153+0000 7f6329f20640 1 --2- 172.21.15.90:0/3974030108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63241029c0 0x7f632419a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:33:02.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.153+0000 7f632a721640 1 --2- 172.21.15.90:0/3974030108 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6324108900 0x7f63241a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:33:02.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.153+0000 7f632971f640 1 --2- 172.21.15.90:0/3974030108 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6324103c00 0x7f632419a730 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6314006010 tx=0x7f6314031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:02.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.153+0000 7f631affd640 1 -- 172.21.15.90:0/3974030108 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f631403f030 con 0x7f6324103c00 2024-04-25T11:33:02.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.153+0000 7f632c184640 1 -- 172.21.15.90:0/3974030108 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63241a1fc0 con 0x7f6324103c00 2024-04-25T11:33:02.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.153+0000 7f631affd640 1 -- 172.21.15.90:0/3974030108 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6314003aa0 con 0x7f6324103c00 2024-04-25T11:33:02.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.153+0000 7f631affd640 1 -- 172.21.15.90:0/3974030108 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6314035070 con 0x7f6324103c00 2024-04-25T11:33:02.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.153+0000 7f632c184640 1 -- 172.21.15.90:0/3974030108 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63241a23d0 con 0x7f6324103c00 2024-04-25T11:33:02.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.154+0000 7f632c184640 1 -- 172.21.15.90:0/3974030108 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63241a2630 con 0x7f6324103c00 2024-04-25T11:33:02.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.156+0000 7f631affd640 1 -- 172.21.15.90:0/3974030108 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f6314042030 con 0x7f6324103c00 2024-04-25T11:33:02.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.156+0000 7f631affd640 1 --2- 172.21.15.90:0/3974030108 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f63100afaf0 0x7f63100b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:02.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.156+0000 7f6329f20640 1 --2- 172.21.15.90:0/3974030108 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f63100afaf0 0x7f63100b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:02.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.156+0000 7f631affd640 1 -- 172.21.15.90:0/3974030108 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f63140ff3f0 con 0x7f6324103c00 2024-04-25T11:33:02.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.157+0000 7f6329f20640 1 --2- 172.21.15.90:0/3974030108 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f63100afaf0 0x7f63100b2000 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f63241035c0 tx=0x7f6320005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:02.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.161+0000 7f631affd640 1 -- 172.21.15.90:0/3974030108 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f63140c8470 con 0x7f6324103c00 2024-04-25T11:33:02.226 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 25 11:33:02 smithi090 systemd[1]: Starting Ceph mds.a for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:33:02.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.393+0000 7f632c184640 1 -- 172.21.15.90:0/3974030108 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f63241a29c0 con 0x7f6324103c00 2024-04-25T11:33:02.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.394+0000 7f631affd640 1 -- 172.21.15.90:0/3974030108 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8827 (secure 0 0 0) 0x7f63140c7d70 con 0x7f6324103c00 2024-04-25T11:33:02.382 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:33:02.383 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T11:33:02.383 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 11 2024-04-25T11:33:02.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.396+0000 7f632c184640 1 -- 172.21.15.90:0/3974030108 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f63100afaf0 msgr2=0x7f63100b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:02.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.396+0000 7f632c184640 1 --2- 172.21.15.90:0/3974030108 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f63100afaf0 0x7f63100b2000 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f63241035c0 tx=0x7f6320005f90 comp rx=0 tx=0).stop 2024-04-25T11:33:02.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.396+0000 7f632c184640 1 -- 172.21.15.90:0/3974030108 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6324103c00 msgr2=0x7f632419a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:02.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.396+0000 7f632c184640 1 --2- 172.21.15.90:0/3974030108 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6324103c00 0x7f632419a730 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6314006010 tx=0x7f6314031a50 comp rx=0 tx=0).stop 2024-04-25T11:33:02.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.396+0000 7f632c184640 1 -- 172.21.15.90:0/3974030108 shutdown_connections 2024-04-25T11:33:02.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.396+0000 7f632c184640 1 --2- 172.21.15.90:0/3974030108 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6324108900 0x7f63241a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:02.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.396+0000 7f632c184640 1 --2- 172.21.15.90:0/3974030108 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f63100afaf0 0x7f63100b2000 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:02.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.396+0000 7f632c184640 1 --2- 172.21.15.90:0/3974030108 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6324103c00 0x7f632419a730 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:02.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.396+0000 7f632c184640 1 --2- 172.21.15.90:0/3974030108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63241029c0 0x7f632419a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:02.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.396+0000 7f632c184640 1 -- 172.21.15.90:0/3974030108 >> 172.21.15.90:0/3974030108 conn(0x7f63240fdbd0 msgr2=0x7f6324069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:02.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.396+0000 7f632c184640 1 -- 172.21.15.90:0/3974030108 shutdown_connections 2024-04-25T11:33:02.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:02.396+0000 7f632c184640 1 -- 172.21.15.90:0/3974030108 wait complete. 2024-04-25T11:33:02.564 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 25 11:33:02 smithi090 podman[71342]: 2024-04-25 11:33:02.296424136 +0000 UTC m=+0.106295120 container create eb3aab653da13720fe7d2caad8c460f4da60c58945c528353c9990db69e853b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-25T11:33:02.564 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 25 11:33:02 smithi090 podman[71342]: 2024-04-25 11:33:02.208355938 +0000 UTC m=+0.018226935 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T11:33:02.564 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 25 11:33:02 smithi090 podman[71342]: 2024-04-25 11:33:02.50057225 +0000 UTC m=+0.310443234 container init eb3aab653da13720fe7d2caad8c460f4da60c58945c528353c9990db69e853b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424) 2024-04-25T11:33:02.564 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 25 11:33:02 smithi090 podman[71342]: 2024-04-25 11:33:02.503269128 +0000 UTC m=+0.313140115 container start eb3aab653da13720fe7d2caad8c460f4da60c58945c528353c9990db69e853b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-a, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T11:33:02.565 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 25 11:33:02 smithi090 ceph-mds[71362]: -- 172.21.15.90:0/3904541041 <== mon.0 v2:172.21.15.90:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b9c9973a40 con 0x55b9c99bb600 2024-04-25T11:33:02.565 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 25 11:33:02 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-a[71356]: starting mds.a at 2024-04-25T11:33:02.565 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 25 11:33:02 smithi090 bash[71342]: eb3aab653da13720fe7d2caad8c460f4da60c58945c528353c9990db69e853b4 2024-04-25T11:33:02.976 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 25 11:33:02 smithi090 systemd[1]: Started Ceph mds.a for 58bd4e7a-02f6-11ef-bc93-c7b262605968. 2024-04-25T11:33:02.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:02 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3974030108' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:33:02.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:02 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:02.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:02 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:02.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:02 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:02.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:02 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:33:02.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:02 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:33:02.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:02 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:33:03.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:02 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3974030108' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:33:03.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:02 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:03.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:02 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:03.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:02 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:03.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:02 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:33:03.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:02 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:33:03.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:02 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:33:03.293 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:02 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3974030108' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:33:03.293 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:02 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:03.294 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:02 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:03.294 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:02 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:03.294 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:02 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T11:33:03.294 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:02 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T11:33:03.294 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:02 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:33:04.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:03 smithi138 ceph-mon[32328]: Deploying daemon mds.b on smithi138 2024-04-25T11:33:04.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:03 smithi138 ceph-mon[32328]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:33:04.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:03 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.90:6840/2617461073,v1:172.21.15.90:6841/2617461073] up:boot 2024-04-25T11:33:04.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:03 smithi138 ceph-mon[32328]: fsmap 11 up:standby 2024-04-25T11:33:04.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:03 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T11:33:04.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:03 smithi156 ceph-mon[33045]: Deploying daemon mds.b on smithi138 2024-04-25T11:33:04.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:03 smithi156 ceph-mon[33045]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:33:04.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:03 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.90:6840/2617461073,v1:172.21.15.90:6841/2617461073] up:boot 2024-04-25T11:33:04.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:03 smithi156 ceph-mon[33045]: fsmap 11 up:standby 2024-04-25T11:33:04.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:03 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T11:33:04.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:03 smithi090 ceph-mon[29575]: Deploying daemon mds.b on smithi138 2024-04-25T11:33:04.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:03 smithi090 ceph-mon[29575]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:33:04.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:03 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.90:6840/2617461073,v1:172.21.15.90:6841/2617461073] up:boot 2024-04-25T11:33:04.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:03 smithi090 ceph-mon[29575]: fsmap 11 up:standby 2024-04-25T11:33:04.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:03 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T11:33:04.397 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:33:04.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.532+0000 7efed54f0640 1 -- 172.21.15.90:0/2240457692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efed01029c0 msgr2=0x7efed01036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:04.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.532+0000 7efed54f0640 1 --2- 172.21.15.90:0/2240457692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efed01029c0 0x7efed01036c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efec0009900 tx=0x7efec002a410 comp rx=0 tx=0).stop 2024-04-25T11:33:04.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.532+0000 7efed54f0640 1 -- 172.21.15.90:0/2240457692 shutdown_connections 2024-04-25T11:33:04.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.532+0000 7efed54f0640 1 --2- 172.21.15.90:0/2240457692 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efed0108900 0x7efed010ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:04.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.532+0000 7efed54f0640 1 --2- 172.21.15.90:0/2240457692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efed0103c00 0x7efed0104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:04.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.532+0000 7efed54f0640 1 --2- 172.21.15.90:0/2240457692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efed01029c0 0x7efed01036c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:04.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.532+0000 7efed54f0640 1 -- 172.21.15.90:0/2240457692 >> 172.21.15.90:0/2240457692 conn(0x7efed00fdbd0 msgr2=0x7efed0100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:04.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.532+0000 7efed54f0640 1 -- 172.21.15.90:0/2240457692 shutdown_connections 2024-04-25T11:33:04.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.532+0000 7efed54f0640 1 -- 172.21.15.90:0/2240457692 wait complete. 2024-04-25T11:33:04.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.533+0000 7efed54f0640 1 Processor -- start 2024-04-25T11:33:04.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.533+0000 7efed54f0640 1 -- start start 2024-04-25T11:33:04.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.533+0000 7efed54f0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efed01029c0 0x7efed006a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:04.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.533+0000 7efed54f0640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efed0103c00 0x7efed019ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:04.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.533+0000 7efed54f0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efed0108900 0x7efed01a1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:04.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.533+0000 7efed54f0640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7efed010e4c0 con 0x7efed01029c0 2024-04-25T11:33:04.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.533+0000 7efed54f0640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7efed010e340 con 0x7efed0103c00 2024-04-25T11:33:04.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.533+0000 7efed54f0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7efed010e640 con 0x7efed0108900 2024-04-25T11:33:04.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.533+0000 7efeceffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efed01029c0 0x7efed006a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:04.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.533+0000 7efeceffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efed01029c0 0x7efed006a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52892/0 (socket says 172.21.15.90:52892) 2024-04-25T11:33:04.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.533+0000 7efeceffd640 1 -- 172.21.15.90:0/4067163708 learned_addr learned my addr 172.21.15.90:0/4067163708 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:04.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.533+0000 7efec7fff640 1 --2- 172.21.15.90:0/4067163708 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efed0103c00 0x7efed019ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:04.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.534+0000 7efecf7fe640 1 --2- 172.21.15.90:0/4067163708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efed0108900 0x7efed01a1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:04.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.534+0000 7efeceffd640 1 -- 172.21.15.90:0/4067163708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efed0108900 msgr2=0x7efed01a1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:04.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.534+0000 7efeceffd640 1 --2- 172.21.15.90:0/4067163708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efed0108900 0x7efed01a1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:04.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.534+0000 7efeceffd640 1 -- 172.21.15.90:0/4067163708 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efed0103c00 msgr2=0x7efed019ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:04.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.534+0000 7efeceffd640 1 --2- 172.21.15.90:0/4067163708 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efed0103c00 0x7efed019ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:04.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.534+0000 7efeceffd640 1 -- 172.21.15.90:0/4067163708 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efed01a1d20 con 0x7efed01029c0 2024-04-25T11:33:04.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.534+0000 7efec7fff640 1 --2- 172.21.15.90:0/4067163708 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efed0103c00 0x7efed019ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:33:04.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.534+0000 7efeceffd640 1 --2- 172.21.15.90:0/4067163708 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efed01029c0 0x7efed006a920 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7efec0006010 tx=0x7efec0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:04.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.534+0000 7efeccff9640 1 -- 172.21.15.90:0/4067163708 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec003f030 con 0x7efed01029c0 2024-04-25T11:33:04.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.534+0000 7efed54f0640 1 -- 172.21.15.90:0/4067163708 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efed01a1fb0 con 0x7efed01029c0 2024-04-25T11:33:04.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.534+0000 7efeccff9640 1 -- 172.21.15.90:0/4067163708 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efec0031910 con 0x7efed01029c0 2024-04-25T11:33:04.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.534+0000 7efeccff9640 1 -- 172.21.15.90:0/4067163708 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec0035070 con 0x7efed01029c0 2024-04-25T11:33:04.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.534+0000 7efed54f0640 1 -- 172.21.15.90:0/4067163708 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efed01a23b0 con 0x7efed01029c0 2024-04-25T11:33:04.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.535+0000 7efed54f0640 1 -- 172.21.15.90:0/4067163708 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efed0107710 con 0x7efed01029c0 2024-04-25T11:33:04.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.537+0000 7efeccff9640 1 -- 172.21.15.90:0/4067163708 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7efec0042030 con 0x7efed01029c0 2024-04-25T11:33:04.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.537+0000 7efeccff9640 1 --2- 172.21.15.90:0/4067163708 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7efebc0afaf0 0x7efebc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:04.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.537+0000 7efec7fff640 1 --2- 172.21.15.90:0/4067163708 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7efebc0afaf0 0x7efebc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:04.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.537+0000 7efeccff9640 1 -- 172.21.15.90:0/4067163708 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7efec00feec0 con 0x7efed01029c0 2024-04-25T11:33:04.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.538+0000 7efec7fff640 1 --2- 172.21.15.90:0/4067163708 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7efebc0afaf0 0x7efebc0b2000 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7efed006a2c0 tx=0x7efeb8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:04.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.541+0000 7efeccff9640 1 -- 172.21.15.90:0/4067163708 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efec00c7f40 con 0x7efed01029c0 2024-04-25T11:33:04.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.758+0000 7efed54f0640 1 -- 172.21.15.90:0/4067163708 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efed002db50 con 0x7efed01029c0 2024-04-25T11:33:04.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.758+0000 7efeccff9640 1 -- 172.21.15.90:0/4067163708 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9653 (secure 0 0 0) 0x7efec00c7840 con 0x7efed01029c0 2024-04-25T11:33:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:33:04.747 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T11:33:04.747 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 12 2024-04-25T11:33:04.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.760+0000 7efed54f0640 1 -- 172.21.15.90:0/4067163708 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7efebc0afaf0 msgr2=0x7efebc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:04.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.760+0000 7efed54f0640 1 --2- 172.21.15.90:0/4067163708 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7efebc0afaf0 0x7efebc0b2000 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7efed006a2c0 tx=0x7efeb8005f90 comp rx=0 tx=0).stop 2024-04-25T11:33:04.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.760+0000 7efed54f0640 1 -- 172.21.15.90:0/4067163708 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efed01029c0 msgr2=0x7efed006a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:04.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.760+0000 7efed54f0640 1 --2- 172.21.15.90:0/4067163708 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efed01029c0 0x7efed006a920 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7efec0006010 tx=0x7efec0004270 comp rx=0 tx=0).stop 2024-04-25T11:33:04.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.760+0000 7efed54f0640 1 -- 172.21.15.90:0/4067163708 shutdown_connections 2024-04-25T11:33:04.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.760+0000 7efed54f0640 1 --2- 172.21.15.90:0/4067163708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efed0108900 0x7efed01a1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:04.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.760+0000 7efed54f0640 1 --2- 172.21.15.90:0/4067163708 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efed0103c00 0x7efed019ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:04.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.760+0000 7efed54f0640 1 --2- 172.21.15.90:0/4067163708 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7efebc0afaf0 0x7efebc0b2000 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:04.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.760+0000 7efed54f0640 1 --2- 172.21.15.90:0/4067163708 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efed01029c0 0x7efed006a920 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:04.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.760+0000 7efed54f0640 1 -- 172.21.15.90:0/4067163708 >> 172.21.15.90:0/4067163708 conn(0x7efed00fdbd0 msgr2=0x7efed01048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:04.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.761+0000 7efed54f0640 1 -- 172.21.15.90:0/4067163708 shutdown_connections 2024-04-25T11:33:04.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:04.761+0000 7efed54f0640 1 -- 172.21.15.90:0/4067163708 wait complete. 2024-04-25T11:33:04.879 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:04 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/4067163708' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:33:05.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:04 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/4067163708' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:33:05.145 INFO:journalctl@ceph.mds.b.smithi138.stdout:Apr 25 11:33:05 smithi138 podman[62547]: 2024-04-25 11:33:05.03522754 +0000 UTC m=+0.024542808 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T11:33:05.145 INFO:journalctl@ceph.mds.b.smithi138.stdout:Apr 25 11:33:05 smithi138 podman[62547]: 2024-04-25 11:33:05.157375403 +0000 UTC m=+0.146690701 container create 64820dcad7edccb2506992fc68dd796f0b0255a9e3709a3794ced5cd81689477 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-b, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-25T11:33:05.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:04 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/4067163708' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:33:05.802 INFO:journalctl@ceph.mds.b.smithi138.stdout:Apr 25 11:33:05 smithi138 podman[62547]: 2024-04-25 11:33:05.410000311 +0000 UTC m=+0.399315566 container init 64820dcad7edccb2506992fc68dd796f0b0255a9e3709a3794ced5cd81689477 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-b, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True) 2024-04-25T11:33:05.803 INFO:journalctl@ceph.mds.b.smithi138.stdout:Apr 25 11:33:05 smithi138 podman[62547]: 2024-04-25 11:33:05.412594152 +0000 UTC m=+0.401909411 container start 64820dcad7edccb2506992fc68dd796f0b0255a9e3709a3794ced5cd81689477 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-b, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-25T11:33:05.803 INFO:journalctl@ceph.mds.b.smithi138.stdout:Apr 25 11:33:05 smithi138 ceph-mds[62566]: -- 172.21.15.138:0/2929423694 <== mon.2 v2:172.21.15.138:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560751b7fa40 con 0x560751bc7a80 2024-04-25T11:33:05.803 INFO:journalctl@ceph.mds.b.smithi138.stdout:Apr 25 11:33:05 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-b[62562]: starting mds.b at 2024-04-25T11:33:05.803 INFO:journalctl@ceph.mds.b.smithi138.stdout:Apr 25 11:33:05 smithi138 bash[62547]: 64820dcad7edccb2506992fc68dd796f0b0255a9e3709a3794ced5cd81689477 2024-04-25T11:33:05.803 INFO:journalctl@ceph.mds.b.smithi138.stdout:Apr 25 11:33:05 smithi138 systemd[1]: Started Ceph mds.b for 58bd4e7a-02f6-11ef-bc93-c7b262605968. 2024-04-25T11:33:06.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:05 smithi156 ceph-mon[33045]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:33:06.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:06.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:06.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:06.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:06.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:06.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:05 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:33:06.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:05 smithi090 ceph-mon[29575]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:33:06.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:06.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:06.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:06.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:06.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:06.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:05 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:33:06.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:05 smithi138 ceph-mon[32328]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:33:06.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:05 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:06.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:05 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:06.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:05 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:06.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:05 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:06.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:05 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:06.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:05 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:33:06.762 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:33:06.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.884+0000 7efe8052e640 1 -- 172.21.15.90:0/3395743955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe78108900 msgr2=0x7efe7810ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:06.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.884+0000 7efe8052e640 1 --2- 172.21.15.90:0/3395743955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe78108900 0x7efe7810ad20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7efe70009ca0 tx=0x7efe7002a8c0 comp rx=0 tx=0).stop 2024-04-25T11:33:06.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.884+0000 7efe8052e640 1 -- 172.21.15.90:0/3395743955 shutdown_connections 2024-04-25T11:33:06.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.884+0000 7efe8052e640 1 --2- 172.21.15.90:0/3395743955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe78108900 0x7efe7810ad20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:06.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.884+0000 7efe8052e640 1 --2- 172.21.15.90:0/3395743955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efe78103c00 0x7efe78104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:06.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.884+0000 7efe8052e640 1 --2- 172.21.15.90:0/3395743955 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efe781029c0 0x7efe781036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:06.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.884+0000 7efe8052e640 1 -- 172.21.15.90:0/3395743955 >> 172.21.15.90:0/3395743955 conn(0x7efe780fdbd0 msgr2=0x7efe78100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:06.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.884+0000 7efe8052e640 1 -- 172.21.15.90:0/3395743955 shutdown_connections 2024-04-25T11:33:06.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.884+0000 7efe8052e640 1 -- 172.21.15.90:0/3395743955 wait complete. 2024-04-25T11:33:06.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.884+0000 7efe8052e640 1 Processor -- start 2024-04-25T11:33:06.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.884+0000 7efe8052e640 1 -- start start 2024-04-25T11:33:06.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.885+0000 7efe8052e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efe781029c0 0x7efe781990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:06.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.885+0000 7efe8052e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe78103c00 0x7efe78199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:06.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.885+0000 7efe8052e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efe78108900 0x7efe781a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:06.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.885+0000 7efe8052e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7efe7810e4f0 con 0x7efe78103c00 2024-04-25T11:33:06.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.885+0000 7efe8052e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7efe7810e370 con 0x7efe781029c0 2024-04-25T11:33:06.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.885+0000 7efe8052e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7efe7810e670 con 0x7efe78108900 2024-04-25T11:33:06.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.885+0000 7efe7dac9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe78103c00 0x7efe78199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:06.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.885+0000 7efe7dac9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe78103c00 0x7efe78199610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52902/0 (socket says 172.21.15.90:52902) 2024-04-25T11:33:06.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.885+0000 7efe7dac9640 1 -- 172.21.15.90:0/871195786 learned_addr learned my addr 172.21.15.90:0/871195786 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:06.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.885+0000 7efe7e2ca640 1 --2- 172.21.15.90:0/871195786 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efe781029c0 0x7efe781990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:06.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.885+0000 7efe7eacb640 1 --2- 172.21.15.90:0/871195786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efe78108900 0x7efe781a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:06.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.885+0000 7efe7dac9640 1 -- 172.21.15.90:0/871195786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efe78108900 msgr2=0x7efe781a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:06.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.885+0000 7efe7dac9640 1 --2- 172.21.15.90:0/871195786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efe78108900 0x7efe781a05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:06.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.885+0000 7efe7dac9640 1 -- 172.21.15.90:0/871195786 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efe781029c0 msgr2=0x7efe781990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:06.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.885+0000 7efe7dac9640 1 --2- 172.21.15.90:0/871195786 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efe781029c0 0x7efe781990d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:06.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.885+0000 7efe7dac9640 1 -- 172.21.15.90:0/871195786 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe781a0c10 con 0x7efe78103c00 2024-04-25T11:33:06.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.885+0000 7efe7dac9640 1 --2- 172.21.15.90:0/871195786 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe78103c00 0x7efe78199610 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efe6800d770 tx=0x7efe6800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:06.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.885+0000 7efe6f7fe640 1 -- 172.21.15.90:0/871195786 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe68017070 con 0x7efe78103c00 2024-04-25T11:33:06.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.885+0000 7efe6f7fe640 1 -- 172.21.15.90:0/871195786 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe680054d0 con 0x7efe78103c00 2024-04-25T11:33:06.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.885+0000 7efe6f7fe640 1 -- 172.21.15.90:0/871195786 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe68004c30 con 0x7efe78103c00 2024-04-25T11:33:06.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.885+0000 7efe8052e640 1 -- 172.21.15.90:0/871195786 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe781a0f00 con 0x7efe78103c00 2024-04-25T11:33:06.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.885+0000 7efe8052e640 1 -- 172.21.15.90:0/871195786 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe781a1370 con 0x7efe78103c00 2024-04-25T11:33:06.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.887+0000 7efe8052e640 1 -- 172.21.15.90:0/871195786 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe78107710 con 0x7efe78103c00 2024-04-25T11:33:06.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.888+0000 7efe6f7fe640 1 -- 172.21.15.90:0/871195786 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7efe68004e10 con 0x7efe78103c00 2024-04-25T11:33:06.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.888+0000 7efe6f7fe640 1 --2- 172.21.15.90:0/871195786 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7efe640afaf0 0x7efe640b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:06.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.888+0000 7efe7e2ca640 1 --2- 172.21.15.90:0/871195786 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7efe640afaf0 0x7efe640b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:06.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.889+0000 7efe6f7fe640 1 -- 172.21.15.90:0/871195786 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7efe680de190 con 0x7efe78103c00 2024-04-25T11:33:06.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.889+0000 7efe7e2ca640 1 --2- 172.21.15.90:0/871195786 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7efe640afaf0 0x7efe640b2000 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efe740095f0 tx=0x7efe74005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:06.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:06.893+0000 7efe6f7fe640 1 -- 172.21.15.90:0/871195786 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efe680a6bc0 con 0x7efe78103c00 2024-04-25T11:33:07.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:06 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] up:boot 2024-04-25T11:33:07.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:06 smithi138 ceph-mon[32328]: fsmap 12 up:standby 2024-04-25T11:33:07.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:06 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T11:33:07.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:06 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:07.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:06 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] up:boot 2024-04-25T11:33:07.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:06 smithi156 ceph-mon[33045]: fsmap 12 up:standby 2024-04-25T11:33:07.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:06 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T11:33:07.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:06 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:07.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.144+0000 7efe8052e640 1 -- 172.21.15.90:0/871195786 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe7802dda0 con 0x7efe78103c00 2024-04-25T11:33:07.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.145+0000 7efe6f7fe640 1 -- 172.21.15.90:0/871195786 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10482 (secure 0 0 0) 0x7efe680aaec0 con 0x7efe78103c00 2024-04-25T11:33:07.134 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:33:07.134 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24524,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-25T11:33:07.134 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 13 2024-04-25T11:33:07.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.147+0000 7efe8052e640 1 -- 172.21.15.90:0/871195786 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7efe640afaf0 msgr2=0x7efe640b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:07.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.147+0000 7efe8052e640 1 --2- 172.21.15.90:0/871195786 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7efe640afaf0 0x7efe640b2000 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efe740095f0 tx=0x7efe74005f90 comp rx=0 tx=0).stop 2024-04-25T11:33:07.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.147+0000 7efe8052e640 1 -- 172.21.15.90:0/871195786 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe78103c00 msgr2=0x7efe78199610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:07.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.147+0000 7efe8052e640 1 --2- 172.21.15.90:0/871195786 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe78103c00 0x7efe78199610 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efe6800d770 tx=0x7efe6800dc30 comp rx=0 tx=0).stop 2024-04-25T11:33:07.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.147+0000 7efe8052e640 1 -- 172.21.15.90:0/871195786 shutdown_connections 2024-04-25T11:33:07.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.147+0000 7efe8052e640 1 --2- 172.21.15.90:0/871195786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efe78108900 0x7efe781a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.147+0000 7efe8052e640 1 --2- 172.21.15.90:0/871195786 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7efe640afaf0 0x7efe640b2000 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.147+0000 7efe8052e640 1 --2- 172.21.15.90:0/871195786 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe78103c00 0x7efe78199610 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.147+0000 7efe8052e640 1 --2- 172.21.15.90:0/871195786 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efe781029c0 0x7efe781990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.147+0000 7efe8052e640 1 -- 172.21.15.90:0/871195786 >> 172.21.15.90:0/871195786 conn(0x7efe780fdbd0 msgr2=0x7efe781048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:07.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.147+0000 7efe8052e640 1 -- 172.21.15.90:0/871195786 shutdown_connections 2024-04-25T11:33:07.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.147+0000 7efe8052e640 1 -- 172.21.15.90:0/871195786 wait complete. 2024-04-25T11:33:07.146 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:06 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] up:boot 2024-04-25T11:33:07.146 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:06 smithi090 ceph-mon[29575]: fsmap 12 up:standby 2024-04-25T11:33:07.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:06 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T11:33:07.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:06 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:07.148 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:33:07.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.276+0000 7fed2c6e0640 1 -- 172.21.15.90:0/3399549928 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed241050a0 msgr2=0x7fed241054b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:07.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.276+0000 7fed2c6e0640 1 --2- 172.21.15.90:0/3399549928 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed241050a0 0x7fed241054b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fed20009900 tx=0x7fed2002a410 comp rx=0 tx=0).stop 2024-04-25T11:33:07.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.276+0000 7fed2c6e0640 1 -- 172.21.15.90:0/3399549928 shutdown_connections 2024-04-25T11:33:07.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.276+0000 7fed2c6e0640 1 --2- 172.21.15.90:0/3399549928 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed24102250 0x7fed2410bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.276+0000 7fed2c6e0640 1 --2- 172.21.15.90:0/3399549928 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed24101750 0x7fed24101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.276+0000 7fed2c6e0640 1 --2- 172.21.15.90:0/3399549928 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed241050a0 0x7fed241054b0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.276+0000 7fed2c6e0640 1 -- 172.21.15.90:0/3399549928 >> 172.21.15.90:0/3399549928 conn(0x7fed240fd370 msgr2=0x7fed240ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:07.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.276+0000 7fed2c6e0640 1 -- 172.21.15.90:0/3399549928 shutdown_connections 2024-04-25T11:33:07.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.276+0000 7fed2c6e0640 1 -- 172.21.15.90:0/3399549928 wait complete. 2024-04-25T11:33:07.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.277+0000 7fed2c6e0640 1 Processor -- start 2024-04-25T11:33:07.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.277+0000 7fed2c6e0640 1 -- start start 2024-04-25T11:33:07.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.277+0000 7fed2c6e0640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed24101750 0x7fed2419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:07.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.277+0000 7fed2c6e0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed24102250 0x7fed2419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:07.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.277+0000 7fed2c6e0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed241050a0 0x7fed241a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:07.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.277+0000 7fed2c6e0640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fed2410f610 con 0x7fed24102250 2024-04-25T11:33:07.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.277+0000 7fed2c6e0640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fed2410f490 con 0x7fed24101750 2024-04-25T11:33:07.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.277+0000 7fed2c6e0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fed2410f790 con 0x7fed241050a0 2024-04-25T11:33:07.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.277+0000 7fed29c7b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed24102250 0x7fed2419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:07.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.277+0000 7fed29c7b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed24102250 0x7fed2419a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52934/0 (socket says 172.21.15.90:52934) 2024-04-25T11:33:07.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.277+0000 7fed29c7b640 1 -- 172.21.15.90:0/757811484 learned_addr learned my addr 172.21.15.90:0/757811484 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:07.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.277+0000 7fed2a47c640 1 --2- 172.21.15.90:0/757811484 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed24101750 0x7fed2419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:07.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.277+0000 7fed2ac7d640 1 --2- 172.21.15.90:0/757811484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed241050a0 0x7fed241a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:07.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.278+0000 7fed29c7b640 1 -- 172.21.15.90:0/757811484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed241050a0 msgr2=0x7fed241a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:07.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.278+0000 7fed29c7b640 1 --2- 172.21.15.90:0/757811484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed241050a0 0x7fed241a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.278+0000 7fed29c7b640 1 -- 172.21.15.90:0/757811484 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed24101750 msgr2=0x7fed2419a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:07.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.278+0000 7fed29c7b640 1 --2- 172.21.15.90:0/757811484 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed24101750 0x7fed2419a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.278+0000 7fed29c7b640 1 -- 172.21.15.90:0/757811484 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed241a1d50 con 0x7fed24102250 2024-04-25T11:33:07.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.278+0000 7fed29c7b640 1 --2- 172.21.15.90:0/757811484 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed24102250 0x7fed2419a750 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fed1400e9f0 tx=0x7fed1400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:07.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.278+0000 7fed1b7fe640 1 -- 172.21.15.90:0/757811484 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1400cd80 con 0x7fed24102250 2024-04-25T11:33:07.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.278+0000 7fed1b7fe640 1 -- 172.21.15.90:0/757811484 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed140108e0 con 0x7fed24102250 2024-04-25T11:33:07.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.278+0000 7fed2c6e0640 1 -- 172.21.15.90:0/757811484 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed241a1fe0 con 0x7fed24102250 2024-04-25T11:33:07.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.278+0000 7fed1b7fe640 1 -- 172.21.15.90:0/757811484 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed14018980 con 0x7fed24102250 2024-04-25T11:33:07.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.278+0000 7fed2c6e0640 1 -- 172.21.15.90:0/757811484 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed241a24a0 con 0x7fed24102250 2024-04-25T11:33:07.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.279+0000 7fed2c6e0640 1 -- 172.21.15.90:0/757811484 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed24108a30 con 0x7fed24102250 2024-04-25T11:33:07.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.281+0000 7fed1b7fe640 1 -- 172.21.15.90:0/757811484 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fed14010430 con 0x7fed24102250 2024-04-25T11:33:07.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.282+0000 7fed1b7fe640 1 --2- 172.21.15.90:0/757811484 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fed100afaf0 0x7fed100b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:07.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.282+0000 7fed2a47c640 1 --2- 172.21.15.90:0/757811484 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fed100afaf0 0x7fed100b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:07.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.283+0000 7fed1b7fe640 1 -- 172.21.15.90:0/757811484 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7fed140def60 con 0x7fed24102250 2024-04-25T11:33:07.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.283+0000 7fed2a47c640 1 --2- 172.21.15.90:0/757811484 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fed100afaf0 0x7fed100b2000 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fed20005ab0 tx=0x7fed20031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:07.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.285+0000 7fed1b7fe640 1 -- 172.21.15.90:0/757811484 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fed140a7990 con 0x7fed24102250 2024-04-25T11:33:07.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.484+0000 7fed2c6e0640 1 -- 172.21.15.90:0/757811484 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fed2402dd20 con 0x7fed24102250 2024-04-25T11:33:07.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.485+0000 7fed1b7fe640 1 -- 172.21.15.90:0/757811484 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+18475 (secure 0 0 0) 0x7fed140b0060 con 0x7fed24102250 2024-04-25T11:33:07.474 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:33:07.474 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":73,"fsid":"58bd4e7a-02f6-11ef-bc93-c7b262605968","created":"2024-04-25T11:25:23.574550+0000","modified":"2024-04-25T11:32:24.853815+0000","last_up_change":"2024-04-25T11:32:23.852211+0000","last_in_change":"2024-04-25T11:32:04.188507+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:28:42.746649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e54d888-e67b-4951-8ae7-e355fb6b10a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6803","nonce":1201629159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6805","nonce":1201629159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6809","nonce":1201629159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6807","nonce":1201629159}]},"public_addr":"172.21.15.90:6803/1201629159","cluster_addr":"172.21.15.90:6805/1201629159","heartbeat_back_addr":"172.21.15.90:6809/1201629159","heartbeat_front_addr":"172.21.15.90:6807/1201629159","state":["exists","up"]},{"osd":1,"uuid":"91a71c78-dae9-4bc7-9f6f-9e55cfefbe28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6801","nonce":3976851933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6803","nonce":3976851933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6807","nonce":3976851933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6805","nonce":3976851933}]},"public_addr":"172.21.15.138:6801/3976851933","cluster_addr":"172.21.15.138:6803/3976851933","heartbeat_back_addr":"172.21.15.138:6807/3976851933","heartbeat_front_addr":"172.21.15.138:6805/3976851933","state":["exists","up"]},{"osd":2,"uuid":"daff6c7f-80fa-4859-9339-ef7a5e840832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6801","nonce":1505807670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6803","nonce":1505807670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6807","nonce":1505807670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6805","nonce":1505807670}]},"public_addr":"172.21.15.156:6801/1505807670","cluster_addr":"172.21.15.156:6803/1505807670","heartbeat_back_addr":"172.21.15.156:6807/1505807670","heartbeat_front_addr":"172.21.15.156:6805/1505807670","state":["exists","up"]},{"osd":3,"uuid":"33fd94ca-7465-401a-bc1e-706b6b31b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6811","nonce":2520012383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6813","nonce":2520012383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6817","nonce":2520012383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6815","nonce":2520012383}]},"public_addr":"172.21.15.90:6811/2520012383","cluster_addr":"172.21.15.90:6813/2520012383","heartbeat_back_addr":"172.21.15.90:6817/2520012383","heartbeat_front_addr":"172.21.15.90:6815/2520012383","state":["exists","up"]},{"osd":4,"uuid":"c5326212-750c-49fd-96e3-d1e62bb4b7f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6809","nonce":3752678279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6811","nonce":3752678279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6815","nonce":3752678279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6813","nonce":3752678279}]},"public_addr":"172.21.15.138:6809/3752678279","cluster_addr":"172.21.15.138:6811/3752678279","heartbeat_back_addr":"172.21.15.138:6815/3752678279","heartbeat_front_addr":"172.21.15.138:6813/3752678279","state":["exists","up"]},{"osd":5,"uuid":"d04a1156-820b-4b43-be89-e16a8910c3aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6809","nonce":3591210191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6811","nonce":3591210191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6815","nonce":3591210191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6813","nonce":3591210191}]},"public_addr":"172.21.15.156:6809/3591210191","cluster_addr":"172.21.15.156:6811/3591210191","heartbeat_back_addr":"172.21.15.156:6815/3591210191","heartbeat_front_addr":"172.21.15.156:6813/3591210191","state":["exists","up"]},{"osd":6,"uuid":"75675aa3-e978-490c-8205-13d9868a3d6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6819","nonce":2364955456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6821","nonce":2364955456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6825","nonce":2364955456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6823","nonce":2364955456}]},"public_addr":"172.21.15.90:6819/2364955456","cluster_addr":"172.21.15.90:6821/2364955456","heartbeat_back_addr":"172.21.15.90:6825/2364955456","heartbeat_front_addr":"172.21.15.90:6823/2364955456","state":["exists","up"]},{"osd":7,"uuid":"1a95fdf5-f548-4271-bbcd-e330f7955eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6817","nonce":4012476387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6819","nonce":4012476387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6823","nonce":4012476387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6821","nonce":4012476387}]},"public_addr":"172.21.15.138:6817/4012476387","cluster_addr":"172.21.15.138:6819/4012476387","heartbeat_back_addr":"172.21.15.138:6823/4012476387","heartbeat_front_addr":"172.21.15.138:6821/4012476387","state":["exists","up"]},{"osd":8,"uuid":"605e10b5-5806-4c1f-8e28-a7994d34ddce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6817","nonce":101092224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6819","nonce":101092224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6823","nonce":101092224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6821","nonce":101092224}]},"public_addr":"172.21.15.156:6817/101092224","cluster_addr":"172.21.15.156:6819/101092224","heartbeat_back_addr":"172.21.15.156:6823/101092224","heartbeat_front_addr":"172.21.15.156:6821/101092224","state":["exists","up"]},{"osd":9,"uuid":"d70fdf5e-9d0f-483d-a7d8-505b79b8a418","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6827","nonce":1171265966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6829","nonce":1171265966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6833","nonce":1171265966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6831","nonce":1171265966}]},"public_addr":"172.21.15.90:6827/1171265966","cluster_addr":"172.21.15.90:6829/1171265966","heartbeat_back_addr":"172.21.15.90:6833/1171265966","heartbeat_front_addr":"172.21.15.90:6831/1171265966","state":["exists","up"]},{"osd":10,"uuid":"1ec86c7d-c3a4-432d-9664-8e40836e1e20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6825","nonce":4264666232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6827","nonce":4264666232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6831","nonce":4264666232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6829","nonce":4264666232}]},"public_addr":"172.21.15.138:6825/4264666232","cluster_addr":"172.21.15.138:6827/4264666232","heartbeat_back_addr":"172.21.15.138:6831/4264666232","heartbeat_front_addr":"172.21.15.138:6829/4264666232","state":["exists","up"]},{"osd":11,"uuid":"51cbe729-292e-4d7b-b8e9-e1e2821ffe76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6825","nonce":1224251744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6827","nonce":1224251744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6831","nonce":1224251744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6829","nonce":1224251744}]},"public_addr":"172.21.15.156:6825/1224251744","cluster_addr":"172.21.15.156:6827/1224251744","heartbeat_back_addr":"172.21.15.156:6831/1224251744","heartbeat_front_addr":"172.21.15.156:6829/1224251744","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:27:50.794226+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:15.030278+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:38.518163+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:03.107351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:27.500707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:51.463545+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:16.015076+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:41.182071+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:05.362309+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:31.048936+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:56.488509+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:32:22.129178+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/2381633953":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6800/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/2190846900":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1707792941":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6800/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6801/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/6061052":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1849633392":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6801/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/2277490702":"2024-04-26T11:26:16.655278+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:33:07.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.487+0000 7fed2c6e0640 1 -- 172.21.15.90:0/757811484 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fed100afaf0 msgr2=0x7fed100b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:07.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.487+0000 7fed2c6e0640 1 --2- 172.21.15.90:0/757811484 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fed100afaf0 0x7fed100b2000 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fed20005ab0 tx=0x7fed20031400 comp rx=0 tx=0).stop 2024-04-25T11:33:07.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.487+0000 7fed2c6e0640 1 -- 172.21.15.90:0/757811484 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed24102250 msgr2=0x7fed2419a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:07.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.487+0000 7fed2c6e0640 1 --2- 172.21.15.90:0/757811484 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed24102250 0x7fed2419a750 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fed1400e9f0 tx=0x7fed1400eeb0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.487+0000 7fed2c6e0640 1 -- 172.21.15.90:0/757811484 shutdown_connections 2024-04-25T11:33:07.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.487+0000 7fed2c6e0640 1 --2- 172.21.15.90:0/757811484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed241050a0 0x7fed241a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.487+0000 7fed2c6e0640 1 --2- 172.21.15.90:0/757811484 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fed100afaf0 0x7fed100b2000 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.487+0000 7fed2c6e0640 1 --2- 172.21.15.90:0/757811484 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed24102250 0x7fed2419a750 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.487+0000 7fed2c6e0640 1 --2- 172.21.15.90:0/757811484 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed24101750 0x7fed2419a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.487+0000 7fed2c6e0640 1 -- 172.21.15.90:0/757811484 >> 172.21.15.90:0/757811484 conn(0x7fed240fd370 msgr2=0x7fed24105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:07.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.488+0000 7fed2c6e0640 1 -- 172.21.15.90:0/757811484 shutdown_connections 2024-04-25T11:33:07.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.488+0000 7fed2c6e0640 1 -- 172.21.15.90:0/757811484 wait complete. 2024-04-25T11:33:07.487 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T11:28:42.746649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T11:33:07.488 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T11:33:07.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.609+0000 7eff28557640 1 -- 172.21.15.90:0/1684757502 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff201045e0 msgr2=0x7eff20106a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:07.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.609+0000 7eff28557640 1 --2- 172.21.15.90:0/1684757502 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff201045e0 0x7eff20106a00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7eff18009900 tx=0x7eff1802a410 comp rx=0 tx=0).stop 2024-04-25T11:33:07.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.610+0000 7eff28557640 1 -- 172.21.15.90:0/1684757502 shutdown_connections 2024-04-25T11:33:07.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.610+0000 7eff28557640 1 --2- 172.21.15.90:0/1684757502 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7eff201098a0 0x7eff2010bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.610+0000 7eff28557640 1 --2- 172.21.15.90:0/1684757502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff20106f40 0x7eff20109360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.610+0000 7eff28557640 1 --2- 172.21.15.90:0/1684757502 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff201045e0 0x7eff20106a00 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.610+0000 7eff28557640 1 -- 172.21.15.90:0/1684757502 >> 172.21.15.90:0/1684757502 conn(0x7eff200fdbe0 msgr2=0x7eff20100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:07.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.610+0000 7eff28557640 1 -- 172.21.15.90:0/1684757502 shutdown_connections 2024-04-25T11:33:07.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.610+0000 7eff28557640 1 -- 172.21.15.90:0/1684757502 wait complete. 2024-04-25T11:33:07.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.610+0000 7eff28557640 1 Processor -- start 2024-04-25T11:33:07.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.610+0000 7eff28557640 1 -- start start 2024-04-25T11:33:07.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.610+0000 7eff28557640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff201045e0 0x7eff2019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:07.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.610+0000 7eff28557640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff20106f40 0x7eff2019a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:07.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.610+0000 7eff28557640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7eff201098a0 0x7eff201a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:07.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.610+0000 7eff28557640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7eff2010f5f0 con 0x7eff20106f40 2024-04-25T11:33:07.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.610+0000 7eff28557640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7eff2010f470 con 0x7eff201098a0 2024-04-25T11:33:07.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.610+0000 7eff28557640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7eff2010f770 con 0x7eff201045e0 2024-04-25T11:33:07.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.611+0000 7eff25af2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff20106f40 0x7eff2019a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:07.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.611+0000 7eff25af2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff20106f40 0x7eff2019a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52964/0 (socket says 172.21.15.90:52964) 2024-04-25T11:33:07.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.611+0000 7eff25af2640 1 -- 172.21.15.90:0/1305667250 learned_addr learned my addr 172.21.15.90:0/1305667250 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:07.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.611+0000 7eff262f3640 1 --2- 172.21.15.90:0/1305667250 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff201045e0 0x7eff2019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:07.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.611+0000 7eff26af4640 1 --2- 172.21.15.90:0/1305667250 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7eff201098a0 0x7eff201a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:07.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.611+0000 7eff25af2640 1 -- 172.21.15.90:0/1305667250 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff201045e0 msgr2=0x7eff2019a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:07.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.611+0000 7eff25af2640 1 --2- 172.21.15.90:0/1305667250 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff201045e0 0x7eff2019a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.611+0000 7eff25af2640 1 -- 172.21.15.90:0/1305667250 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7eff201098a0 msgr2=0x7eff201a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:07.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.611+0000 7eff25af2640 1 --2- 172.21.15.90:0/1305667250 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7eff201098a0 0x7eff201a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.611+0000 7eff25af2640 1 -- 172.21.15.90:0/1305667250 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff201a1df0 con 0x7eff20106f40 2024-04-25T11:33:07.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.611+0000 7eff25af2640 1 --2- 172.21.15.90:0/1305667250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff20106f40 0x7eff2019a7f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7eff1000e9f0 tx=0x7eff1000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:07.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.611+0000 7eff177fe640 1 -- 172.21.15.90:0/1305667250 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff1000cd80 con 0x7eff20106f40 2024-04-25T11:33:07.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.611+0000 7eff177fe640 1 -- 172.21.15.90:0/1305667250 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff100054d0 con 0x7eff20106f40 2024-04-25T11:33:07.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.611+0000 7eff28557640 1 -- 172.21.15.90:0/1305667250 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff201a20e0 con 0x7eff20106f40 2024-04-25T11:33:07.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.611+0000 7eff177fe640 1 -- 172.21.15.90:0/1305667250 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff10010690 con 0x7eff20106f40 2024-04-25T11:33:07.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.611+0000 7eff28557640 1 -- 172.21.15.90:0/1305667250 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff201a2630 con 0x7eff20106f40 2024-04-25T11:33:07.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.612+0000 7eff28557640 1 -- 172.21.15.90:0/1305667250 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff200ff700 con 0x7eff20106f40 2024-04-25T11:33:07.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.613+0000 7eff177fe640 1 -- 172.21.15.90:0/1305667250 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7eff10004e30 con 0x7eff20106f40 2024-04-25T11:33:07.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.614+0000 7eff177fe640 1 --2- 172.21.15.90:0/1305667250 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7eff0c0afb70 0x7eff0c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:07.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.614+0000 7eff262f3640 1 --2- 172.21.15.90:0/1305667250 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7eff0c0afb70 0x7eff0c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:07.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.614+0000 7eff177fe640 1 -- 172.21.15.90:0/1305667250 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7eff100dedb0 con 0x7eff20106f40 2024-04-25T11:33:07.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.615+0000 7eff262f3640 1 --2- 172.21.15.90:0/1305667250 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7eff0c0afb70 0x7eff0c0b2080 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7eff18009230 tx=0x7eff18031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:07.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.618+0000 7eff177fe640 1 -- 172.21.15.90:0/1305667250 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7eff10053ff0 con 0x7eff20106f40 2024-04-25T11:33:07.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.796+0000 7eff28557640 1 -- 172.21.15.90:0/1305667250 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7eff200637a0 con 0x7eff20106f40 2024-04-25T11:33:07.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.797+0000 7eff177fe640 1 -- 172.21.15.90:0/1305667250 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) ==== 93+0+10 (secure 0 0 0) 0x7eff100a7e30 con 0x7eff20106f40 2024-04-25T11:33:07.786 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T11:33:07.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.799+0000 7eff28557640 1 -- 172.21.15.90:0/1305667250 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7eff0c0afb70 msgr2=0x7eff0c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:07.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.799+0000 7eff28557640 1 --2- 172.21.15.90:0/1305667250 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7eff0c0afb70 0x7eff0c0b2080 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7eff18009230 tx=0x7eff18031400 comp rx=0 tx=0).stop 2024-04-25T11:33:07.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.799+0000 7eff28557640 1 -- 172.21.15.90:0/1305667250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff20106f40 msgr2=0x7eff2019a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:07.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.799+0000 7eff28557640 1 --2- 172.21.15.90:0/1305667250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff20106f40 0x7eff2019a7f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7eff1000e9f0 tx=0x7eff1000edf0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.799+0000 7eff28557640 1 -- 172.21.15.90:0/1305667250 shutdown_connections 2024-04-25T11:33:07.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.799+0000 7eff28557640 1 --2- 172.21.15.90:0/1305667250 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7eff201098a0 0x7eff201a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.799+0000 7eff28557640 1 --2- 172.21.15.90:0/1305667250 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7eff0c0afb70 0x7eff0c0b2080 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.799+0000 7eff28557640 1 --2- 172.21.15.90:0/1305667250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff20106f40 0x7eff2019a7f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.799+0000 7eff28557640 1 --2- 172.21.15.90:0/1305667250 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff201045e0 0x7eff2019a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.799+0000 7eff28557640 1 -- 172.21.15.90:0/1305667250 >> 172.21.15.90:0/1305667250 conn(0x7eff200fdbe0 msgr2=0x7eff20106640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:07.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.799+0000 7eff28557640 1 -- 172.21.15.90:0/1305667250 shutdown_connections 2024-04-25T11:33:07.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.799+0000 7eff28557640 1 -- 172.21.15.90:0/1305667250 wait complete. 2024-04-25T11:33:07.799 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T11:33:07.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.918+0000 7f15b22d8640 1 -- 172.21.15.90:0/2290926417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15ac106c20 msgr2=0x7f15ac109100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:07.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.918+0000 7f15b22d8640 1 --2- 172.21.15.90:0/2290926417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15ac106c20 0x7f15ac109100 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f159c009900 tx=0x7f159c02a410 comp rx=0 tx=0).stop 2024-04-25T11:33:07.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.918+0000 7f15b22d8640 1 -- 172.21.15.90:0/2290926417 shutdown_connections 2024-04-25T11:33:07.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.918+0000 7f15b22d8640 1 --2- 172.21.15.90:0/2290926417 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f15ac109790 0x7f15ac10bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.918+0000 7f15b22d8640 1 --2- 172.21.15.90:0/2290926417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15ac106c20 0x7f15ac109100 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.918+0000 7f15b22d8640 1 --2- 172.21.15.90:0/2290926417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15ac101e40 0x7f15ac102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.918+0000 7f15b22d8640 1 -- 172.21.15.90:0/2290926417 >> 172.21.15.90:0/2290926417 conn(0x7f15ac0fd370 msgr2=0x7f15ac0ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:07.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.918+0000 7f15b22d8640 1 -- 172.21.15.90:0/2290926417 shutdown_connections 2024-04-25T11:33:07.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.918+0000 7f15b22d8640 1 -- 172.21.15.90:0/2290926417 wait complete. 2024-04-25T11:33:07.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.919+0000 7f15b22d8640 1 Processor -- start 2024-04-25T11:33:07.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.919+0000 7f15b22d8640 1 -- start start 2024-04-25T11:33:07.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.919+0000 7f15b22d8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15ac101e40 0x7f15ac19a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:07.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.919+0000 7f15b22d8640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15ac106c20 0x7f15ac19a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:07.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.919+0000 7f15b22d8640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f15ac109790 0x7f15ac1a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:07.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.919+0000 7f15b22d8640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac10f620 con 0x7f15ac106c20 2024-04-25T11:33:07.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.919+0000 7f15b22d8640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac10f4a0 con 0x7f15ac109790 2024-04-25T11:33:07.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.919+0000 7f15b22d8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac10f7a0 con 0x7f15ac101e40 2024-04-25T11:33:07.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.919+0000 7f15ab7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15ac106c20 0x7f15ac19a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:07.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.919+0000 7f15ab7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15ac106c20 0x7f15ac19a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52974/0 (socket says 172.21.15.90:52974) 2024-04-25T11:33:07.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.919+0000 7f15ab7fe640 1 -- 172.21.15.90:0/280760952 learned_addr learned my addr 172.21.15.90:0/280760952 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:07.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.919+0000 7f15abfff640 1 --2- 172.21.15.90:0/280760952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15ac101e40 0x7f15ac19a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:07.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.919+0000 7f15b0875640 1 --2- 172.21.15.90:0/280760952 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f15ac109790 0x7f15ac1a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:07.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.920+0000 7f15ab7fe640 1 -- 172.21.15.90:0/280760952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15ac101e40 msgr2=0x7f15ac19a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:07.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.920+0000 7f15ab7fe640 1 --2- 172.21.15.90:0/280760952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15ac101e40 0x7f15ac19a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.920+0000 7f15ab7fe640 1 -- 172.21.15.90:0/280760952 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f15ac109790 msgr2=0x7f15ac1a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:07.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.920+0000 7f15ab7fe640 1 --2- 172.21.15.90:0/280760952 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f15ac109790 0x7f15ac1a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:07.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.920+0000 7f15ab7fe640 1 -- 172.21.15.90:0/280760952 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15ac1a1d10 con 0x7f15ac106c20 2024-04-25T11:33:07.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.920+0000 7f15abfff640 1 --2- 172.21.15.90:0/280760952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15ac101e40 0x7f15ac19a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:33:07.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.920+0000 7f15ab7fe640 1 --2- 172.21.15.90:0/280760952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15ac106c20 0x7f15ac19a710 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f159c006010 tx=0x7f159c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:07.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.920+0000 7f15a97fa640 1 -- 172.21.15.90:0/280760952 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159c03f030 con 0x7f15ac106c20 2024-04-25T11:33:07.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.920+0000 7f15a97fa640 1 -- 172.21.15.90:0/280760952 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f159c031910 con 0x7f15ac106c20 2024-04-25T11:33:07.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.920+0000 7f15b22d8640 1 -- 172.21.15.90:0/280760952 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15ac1a1fa0 con 0x7f15ac106c20 2024-04-25T11:33:07.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.920+0000 7f15a97fa640 1 -- 172.21.15.90:0/280760952 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159c035070 con 0x7f15ac106c20 2024-04-25T11:33:07.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.920+0000 7f15b22d8640 1 -- 172.21.15.90:0/280760952 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15ac1a23b0 con 0x7f15ac106c20 2024-04-25T11:33:07.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.921+0000 7f15b22d8640 1 -- 172.21.15.90:0/280760952 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15ac0fee90 con 0x7f15ac106c20 2024-04-25T11:33:07.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.923+0000 7f15a97fa640 1 -- 172.21.15.90:0/280760952 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f159c042030 con 0x7f15ac106c20 2024-04-25T11:33:07.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.924+0000 7f15a97fa640 1 --2- 172.21.15.90:0/280760952 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f15940afaf0 0x7f15940b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:07.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.924+0000 7f15abfff640 1 --2- 172.21.15.90:0/280760952 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f15940afaf0 0x7f15940b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:07.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.924+0000 7f15a97fa640 1 -- 172.21.15.90:0/280760952 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f159c0ff2d0 con 0x7f15ac106c20 2024-04-25T11:33:07.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.924+0000 7f15abfff640 1 --2- 172.21.15.90:0/280760952 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f15940afaf0 0x7f15940b2000 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f15ac104b10 tx=0x7f15a0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:07.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:07.927+0000 7f15a97fa640 1 -- 172.21.15.90:0/280760952 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f159c0c7d00 con 0x7f15ac106c20 2024-04-25T11:33:08.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:07 smithi156 ceph-mon[33045]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:33:08.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:07 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/871195786' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:33:08.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:07 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:08.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:07 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:08.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:07 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:08.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:07 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:08.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:07 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:08.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:07 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:08.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:07 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/757811484' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:33:08.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:07 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1305667250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:33:08.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.112+0000 7f15b22d8640 1 -- 172.21.15.90:0/280760952 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f15ac02dd20 con 0x7f15ac106c20 2024-04-25T11:33:08.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.113+0000 7f15a97fa640 1 -- 172.21.15.90:0/280760952 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v73) ==== 84+0+65 (secure 0 0 0) 0x7f159c0cc000 con 0x7f15ac106c20 2024-04-25T11:33:08.101 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:33:08.101 INFO:teuthology.orchestra.run.smithi090.stdout:[ 2024-04-25T11:33:08.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:33:08.101 INFO:teuthology.orchestra.run.smithi090.stdout: "poolnum": 1, 2024-04-25T11:33:08.101 INFO:teuthology.orchestra.run.smithi090.stdout: "poolname": ".mgr" 2024-04-25T11:33:08.101 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:33:08.101 INFO:teuthology.orchestra.run.smithi090.stdout:] 2024-04-25T11:33:08.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.115+0000 7f15b22d8640 1 -- 172.21.15.90:0/280760952 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f15940afaf0 msgr2=0x7f15940b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:08.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.115+0000 7f15b22d8640 1 --2- 172.21.15.90:0/280760952 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f15940afaf0 0x7f15940b2000 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f15ac104b10 tx=0x7f15a0005f90 comp rx=0 tx=0).stop 2024-04-25T11:33:08.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.115+0000 7f15b22d8640 1 -- 172.21.15.90:0/280760952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15ac106c20 msgr2=0x7f15ac19a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:08.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.115+0000 7f15b22d8640 1 --2- 172.21.15.90:0/280760952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15ac106c20 0x7f15ac19a710 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f159c006010 tx=0x7f159c0042f0 comp rx=0 tx=0).stop 2024-04-25T11:33:08.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.115+0000 7f15b22d8640 1 -- 172.21.15.90:0/280760952 shutdown_connections 2024-04-25T11:33:08.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.115+0000 7f15b22d8640 1 --2- 172.21.15.90:0/280760952 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f15ac109790 0x7f15ac1a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:08.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.115+0000 7f15b22d8640 1 --2- 172.21.15.90:0/280760952 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f15940afaf0 0x7f15940b2000 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:08.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.115+0000 7f15b22d8640 1 --2- 172.21.15.90:0/280760952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15ac106c20 0x7f15ac19a710 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:08.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.115+0000 7f15b22d8640 1 --2- 172.21.15.90:0/280760952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15ac101e40 0x7f15ac19a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:08.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.115+0000 7f15b22d8640 1 -- 172.21.15.90:0/280760952 >> 172.21.15.90:0/280760952 conn(0x7f15ac0fd370 msgr2=0x7f15ac107c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:08.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.115+0000 7f15b22d8640 1 -- 172.21.15.90:0/280760952 shutdown_connections 2024-04-25T11:33:08.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.115+0000 7f15b22d8640 1 -- 172.21.15.90:0/280760952 wait complete. 2024-04-25T11:33:08.114 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:07 smithi090 ceph-mon[29575]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:33:08.114 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:07 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/871195786' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:33:08.114 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:07 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:08.114 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:07 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:08.114 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:07 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:08.114 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:07 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:08.114 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:07 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:08.114 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:07 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:08.114 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:07 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/757811484' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:33:08.114 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:07 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1305667250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:33:08.115 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T11:33:08.116 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T11:33:08.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.243+0000 7f12b5405640 1 -- 172.21.15.90:0/2187233591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12b0108930 msgr2=0x7f12b010ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:08.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.243+0000 7f12b5405640 1 --2- 172.21.15.90:0/2187233591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12b0108930 0x7f12b010ad50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f12a0009900 tx=0x7f12a002a410 comp rx=0 tx=0).stop 2024-04-25T11:33:08.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.243+0000 7f12b5405640 1 -- 172.21.15.90:0/2187233591 shutdown_connections 2024-04-25T11:33:08.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.243+0000 7f12b5405640 1 --2- 172.21.15.90:0/2187233591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12b0108930 0x7f12b010ad50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:08.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.243+0000 7f12b5405640 1 --2- 172.21.15.90:0/2187233591 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f12b0103c30 0x7f12b0104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:08.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.243+0000 7f12b5405640 1 --2- 172.21.15.90:0/2187233591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12b01029f0 0x7f12b01036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:08.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.243+0000 7f12b5405640 1 -- 172.21.15.90:0/2187233591 >> 172.21.15.90:0/2187233591 conn(0x7f12b00fdc00 msgr2=0x7f12b0100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:08.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.243+0000 7f12b5405640 1 -- 172.21.15.90:0/2187233591 shutdown_connections 2024-04-25T11:33:08.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.243+0000 7f12b5405640 1 -- 172.21.15.90:0/2187233591 wait complete. 2024-04-25T11:33:08.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.244+0000 7f12b5405640 1 Processor -- start 2024-04-25T11:33:08.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.244+0000 7f12b5405640 1 -- start start 2024-04-25T11:33:08.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.244+0000 7f12b5405640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12b01029f0 0x7f12b019a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:08.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.244+0000 7f12b5405640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12b0103c30 0x7f12b019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:08.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.244+0000 7f12b5405640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f12b0108930 0x7f12b01a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:08.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.244+0000 7f12b5405640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f12b010f620 con 0x7f12b0103c30 2024-04-25T11:33:08.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.244+0000 7f12b5405640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f12b010f4a0 con 0x7f12b0108930 2024-04-25T11:33:08.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.244+0000 7f12b5405640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f12b010f7a0 con 0x7f12b01029f0 2024-04-25T11:33:08.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.244+0000 7f12ae7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12b0103c30 0x7f12b019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:08.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.244+0000 7f12ae7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12b0103c30 0x7f12b019a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52984/0 (socket says 172.21.15.90:52984) 2024-04-25T11:33:08.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.244+0000 7f12ae7fc640 1 -- 172.21.15.90:0/4196434899 learned_addr learned my addr 172.21.15.90:0/4196434899 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:08.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.244+0000 7f12aeffd640 1 --2- 172.21.15.90:0/4196434899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12b01029f0 0x7f12b019a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:08.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.244+0000 7f12af7fe640 1 --2- 172.21.15.90:0/4196434899 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f12b0108930 0x7f12b01a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:08.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.245+0000 7f12aeffd640 1 -- 172.21.15.90:0/4196434899 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f12b0108930 msgr2=0x7f12b01a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:08.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.245+0000 7f12aeffd640 1 --2- 172.21.15.90:0/4196434899 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f12b0108930 0x7f12b01a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:08.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.245+0000 7f12aeffd640 1 -- 172.21.15.90:0/4196434899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12b0103c30 msgr2=0x7f12b019a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:08.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.245+0000 7f12aeffd640 1 --2- 172.21.15.90:0/4196434899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12b0103c30 0x7f12b019a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:08.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.245+0000 7f12aeffd640 1 -- 172.21.15.90:0/4196434899 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12b01a1d20 con 0x7f12b01029f0 2024-04-25T11:33:08.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.245+0000 7f12ae7fc640 1 --2- 172.21.15.90:0/4196434899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12b0103c30 0x7f12b019a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:33:08.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.245+0000 7f12af7fe640 1 --2- 172.21.15.90:0/4196434899 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f12b0108930 0x7f12b01a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:33:08.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.245+0000 7f12aeffd640 1 --2- 172.21.15.90:0/4196434899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12b01029f0 0x7f12b019a1e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f12b01022c0 tx=0x7f12a800ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:08.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.245+0000 7f1297fff640 1 -- 172.21.15.90:0/4196434899 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a8003970 con 0x7f12b01029f0 2024-04-25T11:33:08.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.245+0000 7f12b5405640 1 -- 172.21.15.90:0/4196434899 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12b01a1fb0 con 0x7f12b01029f0 2024-04-25T11:33:08.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.245+0000 7f1297fff640 1 -- 172.21.15.90:0/4196434899 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12a8004020 con 0x7f12b01029f0 2024-04-25T11:33:08.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.245+0000 7f12b5405640 1 -- 172.21.15.90:0/4196434899 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12b01a2470 con 0x7f12b01029f0 2024-04-25T11:33:08.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.245+0000 7f1297fff640 1 -- 172.21.15.90:0/4196434899 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a8010690 con 0x7f12b01029f0 2024-04-25T11:33:08.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.246+0000 7f12b5405640 1 -- 172.21.15.90:0/4196434899 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12b00692e0 con 0x7f12b01029f0 2024-04-25T11:33:08.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.248+0000 7f1297fff640 1 -- 172.21.15.90:0/4196434899 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f12a80041d0 con 0x7f12b01029f0 2024-04-25T11:33:08.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.248+0000 7f1297fff640 1 --2- 172.21.15.90:0/4196434899 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f12980afaf0 0x7f12980b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:08.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.249+0000 7f1297fff640 1 -- 172.21.15.90:0/4196434899 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f12a80defe0 con 0x7f12b01029f0 2024-04-25T11:33:08.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.249+0000 7f12ae7fc640 1 --2- 172.21.15.90:0/4196434899 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f12980afaf0 0x7f12980b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:08.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.249+0000 7f12ae7fc640 1 --2- 172.21.15.90:0/4196434899 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f12980afaf0 0x7f12980b2000 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f12b019b280 tx=0x7f12a4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:08.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.257+0000 7f1297fff640 1 -- 172.21.15.90:0/4196434899 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f12a80a7fc0 con 0x7f12b01029f0 2024-04-25T11:33:08.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:07 smithi138 ceph-mon[32328]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:33:08.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:07 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/871195786' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:33:08.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:07 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:08.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:07 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:08.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:07 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:08.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:07 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:08.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:07 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:08.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:07 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:08.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:07 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/757811484' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:33:08.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:07 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1305667250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:33:08.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:08.437+0000 7f12b5405640 1 -- 172.21.15.90:0/4196434899 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f12b0107740 con 0x7f12b01029f0 2024-04-25T11:33:09.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.015+0000 7f1297fff640 1 -- 172.21.15.90:0/4196434899 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v74) ==== 138+0+0 (secure 0 0 0) 0x7f12a80b0040 con 0x7f12b01029f0 2024-04-25T11:33:09.004 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_metadata' created 2024-04-25T11:33:09.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.018+0000 7f12b5405640 1 -- 172.21.15.90:0/4196434899 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f12980afaf0 msgr2=0x7f12980b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:09.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.018+0000 7f12b5405640 1 --2- 172.21.15.90:0/4196434899 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f12980afaf0 0x7f12980b2000 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f12b019b280 tx=0x7f12a4005f90 comp rx=0 tx=0).stop 2024-04-25T11:33:09.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.018+0000 7f12b5405640 1 -- 172.21.15.90:0/4196434899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12b01029f0 msgr2=0x7f12b019a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:09.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.018+0000 7f12b5405640 1 --2- 172.21.15.90:0/4196434899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12b01029f0 0x7f12b019a1e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f12b01022c0 tx=0x7f12a800ef90 comp rx=0 tx=0).stop 2024-04-25T11:33:09.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.018+0000 7f12b5405640 1 -- 172.21.15.90:0/4196434899 shutdown_connections 2024-04-25T11:33:09.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.018+0000 7f12b5405640 1 --2- 172.21.15.90:0/4196434899 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f12b0108930 0x7f12b01a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:09.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.018+0000 7f12b5405640 1 --2- 172.21.15.90:0/4196434899 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f12980afaf0 0x7f12980b2000 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:09.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.018+0000 7f12b5405640 1 --2- 172.21.15.90:0/4196434899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12b0103c30 0x7f12b019a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:09.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.018+0000 7f12b5405640 1 --2- 172.21.15.90:0/4196434899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12b01029f0 0x7f12b019a1e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:09.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.018+0000 7f12b5405640 1 -- 172.21.15.90:0/4196434899 >> 172.21.15.90:0/4196434899 conn(0x7f12b00fdc00 msgr2=0x7f12b0104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:09.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.018+0000 7f12b5405640 1 -- 172.21.15.90:0/4196434899 shutdown_connections 2024-04-25T11:33:09.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.018+0000 7f12b5405640 1 -- 172.21.15.90:0/4196434899 wait complete. 2024-04-25T11:33:09.018 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T11:33:09.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:08 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:33:09.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:08 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:33:09.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:08 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:09.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:08 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/280760952' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T11:33:09.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:08 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:33:09.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:08 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/4196434899' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T11:33:09.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:08 smithi156 ceph-mon[33045]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T11:33:09.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.144+0000 7fed3ccf2640 1 -- 172.21.15.90:0/3037486016 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed38103c20 msgr2=0x7fed38104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:09.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.144+0000 7fed3ccf2640 1 --2- 172.21.15.90:0/3037486016 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed38103c20 0x7fed38104030 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fed20009900 tx=0x7fed2002a410 comp rx=0 tx=0).stop 2024-04-25T11:33:09.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.144+0000 7fed3ccf2640 1 -- 172.21.15.90:0/3037486016 shutdown_connections 2024-04-25T11:33:09.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.144+0000 7fed3ccf2640 1 --2- 172.21.15.90:0/3037486016 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed38108920 0x7fed3810ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:09.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.144+0000 7fed3ccf2640 1 --2- 172.21.15.90:0/3037486016 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed38103c20 0x7fed38104030 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:09.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.144+0000 7fed3ccf2640 1 --2- 172.21.15.90:0/3037486016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed381029e0 0x7fed381036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:09.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.144+0000 7fed3ccf2640 1 -- 172.21.15.90:0/3037486016 >> 172.21.15.90:0/3037486016 conn(0x7fed380fdbf0 msgr2=0x7fed38100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:09.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.144+0000 7fed3ccf2640 1 -- 172.21.15.90:0/3037486016 shutdown_connections 2024-04-25T11:33:09.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.144+0000 7fed3ccf2640 1 -- 172.21.15.90:0/3037486016 wait complete. 2024-04-25T11:33:09.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.145+0000 7fed3ccf2640 1 Processor -- start 2024-04-25T11:33:09.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.145+0000 7fed3ccf2640 1 -- start start 2024-04-25T11:33:09.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.145+0000 7fed3ccf2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed381029e0 0x7fed381991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:09.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.145+0000 7fed3ccf2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed38103c20 0x7fed38199700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:09.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.145+0000 7fed3ccf2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed38108920 0x7fed381a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:09.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.145+0000 7fed3ccf2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fed3810e500 con 0x7fed38103c20 2024-04-25T11:33:09.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.145+0000 7fed3ccf2640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fed3810e380 con 0x7fed38108920 2024-04-25T11:33:09.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.145+0000 7fed3ccf2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fed3810e680 con 0x7fed381029e0 2024-04-25T11:33:09.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.145+0000 7fed2ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed38103c20 0x7fed38199700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:09.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.145+0000 7fed36d9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed38108920 0x7fed381a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:09.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.145+0000 7fed2ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed38103c20 0x7fed38199700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53000/0 (socket says 172.21.15.90:53000) 2024-04-25T11:33:09.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.145+0000 7fed2ffff640 1 -- 172.21.15.90:0/3110647287 learned_addr learned my addr 172.21.15.90:0/3110647287 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:09.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.145+0000 7fed3659c640 1 --2- 172.21.15.90:0/3110647287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed381029e0 0x7fed381991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:09.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.146+0000 7fed2ffff640 1 -- 172.21.15.90:0/3110647287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed381029e0 msgr2=0x7fed381991c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:09.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.146+0000 7fed2ffff640 1 --2- 172.21.15.90:0/3110647287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed381029e0 0x7fed381991c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:09.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.146+0000 7fed2ffff640 1 -- 172.21.15.90:0/3110647287 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed38108920 msgr2=0x7fed381a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:09.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.146+0000 7fed2ffff640 1 --2- 172.21.15.90:0/3110647287 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed38108920 0x7fed381a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:09.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.146+0000 7fed2ffff640 1 -- 172.21.15.90:0/3110647287 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed381a0cb0 con 0x7fed38103c20 2024-04-25T11:33:09.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.146+0000 7fed36d9d640 1 --2- 172.21.15.90:0/3110647287 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed38108920 0x7fed381a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:33:09.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.146+0000 7fed2ffff640 1 --2- 172.21.15.90:0/3110647287 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed38103c20 0x7fed38199700 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fed20006010 tx=0x7fed200042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:09.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.146+0000 7fed2f7fe640 1 -- 172.21.15.90:0/3110647287 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed2003f030 con 0x7fed38103c20 2024-04-25T11:33:09.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.146+0000 7fed2f7fe640 1 -- 172.21.15.90:0/3110647287 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed20031910 con 0x7fed38103c20 2024-04-25T11:33:09.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.146+0000 7fed3ccf2640 1 -- 172.21.15.90:0/3110647287 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed381a0f40 con 0x7fed38103c20 2024-04-25T11:33:09.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.146+0000 7fed2f7fe640 1 -- 172.21.15.90:0/3110647287 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed20035070 con 0x7fed38103c20 2024-04-25T11:33:09.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.146+0000 7fed3ccf2640 1 -- 172.21.15.90:0/3110647287 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed381a1350 con 0x7fed38103c20 2024-04-25T11:33:09.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.147+0000 7fed3ccf2640 1 -- 172.21.15.90:0/3110647287 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed38107730 con 0x7fed38103c20 2024-04-25T11:33:09.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.148+0000 7fed2f7fe640 1 -- 172.21.15.90:0/3110647287 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fed20042030 con 0x7fed38103c20 2024-04-25T11:33:09.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.149+0000 7fed2f7fe640 1 --2- 172.21.15.90:0/3110647287 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fed240afaf0 0x7fed240b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:09.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.149+0000 7fed3659c640 1 --2- 172.21.15.90:0/3110647287 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fed240afaf0 0x7fed240b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:09.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.149+0000 7fed2f7fe640 1 -- 172.21.15.90:0/3110647287 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7497+0+0 (secure 0 0 0) 0x7fed200fef90 con 0x7fed38103c20 2024-04-25T11:33:09.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.150+0000 7fed3659c640 1 --2- 172.21.15.90:0/3110647287 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fed240afaf0 0x7fed240b2000 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fed381035e0 tx=0x7fed28005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:09.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.153+0000 7fed2f7fe640 1 -- 172.21.15.90:0/3110647287 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fed200c7850 con 0x7fed38103c20 2024-04-25T11:33:09.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:08 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:33:09.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:08 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:33:09.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:08 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:09.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:08 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/280760952' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T11:33:09.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:08 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:33:09.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:08 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/4196434899' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T11:33:09.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:08 smithi090 ceph-mon[29575]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T11:33:09.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:08 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:33:09.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:08 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:33:09.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:08 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:09.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:08 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/280760952' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T11:33:09.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:08 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:33:09.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:08 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/4196434899' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T11:33:09.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:08 smithi138 ceph-mon[32328]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T11:33:09.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:09.348+0000 7fed3ccf2640 1 -- 172.21.15.90:0/3110647287 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fed3802db50 con 0x7fed38103c20 2024-04-25T11:33:10.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.013+0000 7fed2f7fe640 1 -- 172.21.15.90:0/3110647287 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v75) ==== 170+0+0 (secure 0 0 0) 0x7fed200cbb50 con 0x7fed38103c20 2024-04-25T11:33:10.003 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data' created 2024-04-25T11:33:10.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.015+0000 7fed3ccf2640 1 -- 172.21.15.90:0/3110647287 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fed240afaf0 msgr2=0x7fed240b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:10.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.015+0000 7fed3ccf2640 1 --2- 172.21.15.90:0/3110647287 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fed240afaf0 0x7fed240b2000 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fed381035e0 tx=0x7fed28005f90 comp rx=0 tx=0).stop 2024-04-25T11:33:10.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.015+0000 7fed3ccf2640 1 -- 172.21.15.90:0/3110647287 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed38103c20 msgr2=0x7fed38199700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:10.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.015+0000 7fed3ccf2640 1 --2- 172.21.15.90:0/3110647287 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed38103c20 0x7fed38199700 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fed20006010 tx=0x7fed200042f0 comp rx=0 tx=0).stop 2024-04-25T11:33:10.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.015+0000 7fed3ccf2640 1 -- 172.21.15.90:0/3110647287 shutdown_connections 2024-04-25T11:33:10.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.016+0000 7fed3ccf2640 1 --2- 172.21.15.90:0/3110647287 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed38108920 0x7fed381a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:10.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.016+0000 7fed3ccf2640 1 --2- 172.21.15.90:0/3110647287 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fed240afaf0 0x7fed240b2000 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:10.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.016+0000 7fed3ccf2640 1 --2- 172.21.15.90:0/3110647287 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed38103c20 0x7fed38199700 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:10.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.016+0000 7fed3ccf2640 1 --2- 172.21.15.90:0/3110647287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed381029e0 0x7fed381991c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:10.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.016+0000 7fed3ccf2640 1 -- 172.21.15.90:0/3110647287 >> 172.21.15.90:0/3110647287 conn(0x7fed380fdbf0 msgr2=0x7fed38104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:10.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.016+0000 7fed3ccf2640 1 -- 172.21.15.90:0/3110647287 shutdown_connections 2024-04-25T11:33:10.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.017+0000 7fed3ccf2640 1 -- 172.21.15.90:0/3110647287 wait complete. 2024-04-25T11:33:10.026 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T11:33:10.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:09 smithi156 ceph-mon[33045]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:33:10.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:09 smithi156 ceph-mon[33045]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T11:33:10.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:09 smithi156 ceph-mon[33045]: osdmap e74: 12 total, 12 up, 12 in 2024-04-25T11:33:10.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:09 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3110647287' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T11:33:10.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:09 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:10.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:09 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:10.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:09 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:10.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:09 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:10.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.148+0000 7fc8e8ef2640 1 -- 172.21.15.90:0/3320555471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8e4108930 msgr2=0x7fc8e410ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:10.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.148+0000 7fc8e8ef2640 1 --2- 172.21.15.90:0/3320555471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8e4108930 0x7fc8e410ad50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc009ca0 tx=0x7fc8dc02a8c0 comp rx=0 tx=0).stop 2024-04-25T11:33:10.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.148+0000 7fc8e8ef2640 1 -- 172.21.15.90:0/3320555471 shutdown_connections 2024-04-25T11:33:10.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.148+0000 7fc8e8ef2640 1 --2- 172.21.15.90:0/3320555471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8e4108930 0x7fc8e410ad50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:10.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.148+0000 7fc8e8ef2640 1 --2- 172.21.15.90:0/3320555471 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc8e4103c30 0x7fc8e4104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:10.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.148+0000 7fc8e8ef2640 1 --2- 172.21.15.90:0/3320555471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc8e41029f0 0x7fc8e41036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:10.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.148+0000 7fc8e8ef2640 1 -- 172.21.15.90:0/3320555471 >> 172.21.15.90:0/3320555471 conn(0x7fc8e40fdbe0 msgr2=0x7fc8e4100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:10.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.148+0000 7fc8e8ef2640 1 -- 172.21.15.90:0/3320555471 shutdown_connections 2024-04-25T11:33:10.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.148+0000 7fc8e8ef2640 1 -- 172.21.15.90:0/3320555471 wait complete. 2024-04-25T11:33:10.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.149+0000 7fc8e8ef2640 1 Processor -- start 2024-04-25T11:33:10.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.149+0000 7fc8e8ef2640 1 -- start start 2024-04-25T11:33:10.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.149+0000 7fc8e8ef2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc8e41029f0 0x7fc8e419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:10.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.149+0000 7fc8e8ef2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8e4103c30 0x7fc8e419a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:10.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.149+0000 7fc8e8ef2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc8e4108930 0x7fc8e41a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:10.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.149+0000 7fc8e8ef2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e410e540 con 0x7fc8e4103c30 2024-04-25T11:33:10.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.149+0000 7fc8e8ef2640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e410e3c0 con 0x7fc8e41029f0 2024-04-25T11:33:10.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.149+0000 7fc8e8ef2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e410e6c0 con 0x7fc8e4108930 2024-04-25T11:33:10.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.149+0000 7fc8dbfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8e4103c30 0x7fc8e419a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:10.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.149+0000 7fc8dbfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8e4103c30 0x7fc8e419a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53022/0 (socket says 172.21.15.90:53022) 2024-04-25T11:33:10.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.149+0000 7fc8dbfff640 1 -- 172.21.15.90:0/2190915803 learned_addr learned my addr 172.21.15.90:0/2190915803 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:10.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.149+0000 7fc8e259c640 1 --2- 172.21.15.90:0/2190915803 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc8e41029f0 0x7fc8e419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:10.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.149+0000 7fc8e2d9d640 1 --2- 172.21.15.90:0/2190915803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc8e4108930 0x7fc8e41a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:10.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.149+0000 7fc8dbfff640 1 -- 172.21.15.90:0/2190915803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc8e4108930 msgr2=0x7fc8e41a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:10.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.149+0000 7fc8dbfff640 1 --2- 172.21.15.90:0/2190915803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc8e4108930 0x7fc8e41a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:10.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.149+0000 7fc8dbfff640 1 -- 172.21.15.90:0/2190915803 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc8e41029f0 msgr2=0x7fc8e419a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:10.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.149+0000 7fc8dbfff640 1 --2- 172.21.15.90:0/2190915803 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc8e41029f0 0x7fc8e419a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:10.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.149+0000 7fc8dbfff640 1 -- 172.21.15.90:0/2190915803 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8e41a1da0 con 0x7fc8e4103c30 2024-04-25T11:33:10.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.149+0000 7fc8e259c640 1 --2- 172.21.15.90:0/2190915803 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc8e41029f0 0x7fc8e419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:33:10.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.149+0000 7fc8e2d9d640 1 --2- 172.21.15.90:0/2190915803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc8e4108930 0x7fc8e41a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:33:10.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.149+0000 7fc8dbfff640 1 --2- 172.21.15.90:0/2190915803 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8e4103c30 0x7fc8e419a7a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc00e9f0 tx=0x7fc8cc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:10.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.150+0000 7fc8db7fe640 1 -- 172.21.15.90:0/2190915803 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc00cd80 con 0x7fc8e4103c30 2024-04-25T11:33:10.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.150+0000 7fc8e8ef2640 1 -- 172.21.15.90:0/2190915803 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8e41a2090 con 0x7fc8e4103c30 2024-04-25T11:33:10.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.150+0000 7fc8db7fe640 1 -- 172.21.15.90:0/2190915803 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8cc0054d0 con 0x7fc8e4103c30 2024-04-25T11:33:10.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.150+0000 7fc8db7fe640 1 -- 172.21.15.90:0/2190915803 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc010690 con 0x7fc8e4103c30 2024-04-25T11:33:10.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.150+0000 7fc8e8ef2640 1 -- 172.21.15.90:0/2190915803 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8e41a25e0 con 0x7fc8e4103c30 2024-04-25T11:33:10.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.151+0000 7fc8e8ef2640 1 -- 172.21.15.90:0/2190915803 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8e41a28a0 con 0x7fc8e4103c30 2024-04-25T11:33:10.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.153+0000 7fc8db7fe640 1 -- 172.21.15.90:0/2190915803 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fc8cc004e30 con 0x7fc8e4103c30 2024-04-25T11:33:10.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.153+0000 7fc8db7fe640 1 --2- 172.21.15.90:0/2190915803 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fc8d00afaf0 0x7fc8d00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:10.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.153+0000 7fc8e259c640 1 --2- 172.21.15.90:0/2190915803 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fc8d00afaf0 0x7fc8d00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:10.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.153+0000 7fc8db7fe640 1 -- 172.21.15.90:0/2190915803 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7879+0+0 (secure 0 0 0) 0x7fc8cc0df110 con 0x7fc8e4103c30 2024-04-25T11:33:10.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.154+0000 7fc8e259c640 1 --2- 172.21.15.90:0/2190915803 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fc8d00afaf0 0x7fc8d00b2000 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4005fd0 tx=0x7fc8d400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:10.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.157+0000 7fc8db7fe640 1 -- 172.21.15.90:0/2190915803 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc8cc053ff0 con 0x7fc8e4103c30 2024-04-25T11:33:10.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:09 smithi090 ceph-mon[29575]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:33:10.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:09 smithi090 ceph-mon[29575]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T11:33:10.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:09 smithi090 ceph-mon[29575]: osdmap e74: 12 total, 12 up, 12 in 2024-04-25T11:33:10.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:09 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3110647287' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T11:33:10.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:09 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:10.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:09 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:10.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:09 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:10.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:09 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:10.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:09 smithi138 ceph-mon[32328]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:33:10.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:09 smithi138 ceph-mon[32328]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T11:33:10.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:09 smithi138 ceph-mon[32328]: osdmap e74: 12 total, 12 up, 12 in 2024-04-25T11:33:10.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:09 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3110647287' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T11:33:10.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:09 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:10.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:09 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:10.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:09 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:10.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:09 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:10.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:10.372+0000 7fc8e8ef2640 1 -- 172.21.15.90:0/2190915803 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fc8e41a2ea0 con 0x7fc8e4103c30 2024-04-25T11:33:11.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.014+0000 7fc8db7fe640 1 -- 172.21.15.90:0/2190915803 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fc8cc0a7ea0 con 0x7fc8e4103c30 2024-04-25T11:33:11.003 INFO:teuthology.orchestra.run.smithi090.stderr:new fs with metadata pool 2 and data pool 3 2024-04-25T11:33:11.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.017+0000 7fc8e8ef2640 1 -- 172.21.15.90:0/2190915803 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fc8d00afaf0 msgr2=0x7fc8d00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:11.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.017+0000 7fc8e8ef2640 1 --2- 172.21.15.90:0/2190915803 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fc8d00afaf0 0x7fc8d00b2000 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4005fd0 tx=0x7fc8d400b390 comp rx=0 tx=0).stop 2024-04-25T11:33:11.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.017+0000 7fc8e8ef2640 1 -- 172.21.15.90:0/2190915803 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8e4103c30 msgr2=0x7fc8e419a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:11.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.017+0000 7fc8e8ef2640 1 --2- 172.21.15.90:0/2190915803 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8e4103c30 0x7fc8e419a7a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc00e9f0 tx=0x7fc8cc00edf0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.017+0000 7fc8e8ef2640 1 -- 172.21.15.90:0/2190915803 shutdown_connections 2024-04-25T11:33:11.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.017+0000 7fc8e8ef2640 1 --2- 172.21.15.90:0/2190915803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc8e4108930 0x7fc8e41a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.017+0000 7fc8e8ef2640 1 --2- 172.21.15.90:0/2190915803 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fc8d00afaf0 0x7fc8d00b2000 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.017+0000 7fc8e8ef2640 1 --2- 172.21.15.90:0/2190915803 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8e4103c30 0x7fc8e419a7a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.017+0000 7fc8e8ef2640 1 --2- 172.21.15.90:0/2190915803 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc8e41029f0 0x7fc8e419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.017+0000 7fc8e8ef2640 1 -- 172.21.15.90:0/2190915803 >> 172.21.15.90:0/2190915803 conn(0x7fc8e40fdbe0 msgr2=0x7fc8e406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:11.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.017+0000 7fc8e8ef2640 1 -- 172.21.15.90:0/2190915803 shutdown_connections 2024-04-25T11:33:11.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.017+0000 7fc8e8ef2640 1 -- 172.21.15.90:0/2190915803 wait complete. 2024-04-25T11:33:11.020 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:33:11.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.139+0000 7f85fc9a8640 1 -- 172.21.15.90:0/3219287440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85f81029e0 msgr2=0x7f85f81036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:11.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.139+0000 7f85fc9a8640 1 --2- 172.21.15.90:0/3219287440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85f81029e0 0x7f85f81036e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f85f0009900 tx=0x7f85f002a410 comp rx=0 tx=0).stop 2024-04-25T11:33:11.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.139+0000 7f85fc9a8640 1 -- 172.21.15.90:0/3219287440 shutdown_connections 2024-04-25T11:33:11.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.139+0000 7f85fc9a8640 1 --2- 172.21.15.90:0/3219287440 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f85f8108920 0x7f85f810ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.139+0000 7f85fc9a8640 1 --2- 172.21.15.90:0/3219287440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85f8103c20 0x7f85f8104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.139+0000 7f85fc9a8640 1 --2- 172.21.15.90:0/3219287440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85f81029e0 0x7f85f81036e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.139+0000 7f85fc9a8640 1 -- 172.21.15.90:0/3219287440 >> 172.21.15.90:0/3219287440 conn(0x7f85f80fdbd0 msgr2=0x7f85f8100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:11.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.139+0000 7f85fc9a8640 1 -- 172.21.15.90:0/3219287440 shutdown_connections 2024-04-25T11:33:11.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.139+0000 7f85fc9a8640 1 -- 172.21.15.90:0/3219287440 wait complete. 2024-04-25T11:33:11.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.140+0000 7f85fc9a8640 1 Processor -- start 2024-04-25T11:33:11.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.140+0000 7f85fc9a8640 1 -- start start 2024-04-25T11:33:11.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.140+0000 7f85fc9a8640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f85f81029e0 0x7f85f819a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:11.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.140+0000 7f85fc9a8640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85f8103c20 0x7f85f819a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:11.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.140+0000 7f85fc9a8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85f8108920 0x7f85f81a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:11.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.140+0000 7f85fc9a8640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f85f810e540 con 0x7f85f8103c20 2024-04-25T11:33:11.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.140+0000 7f85fc9a8640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f85f810e3c0 con 0x7f85f81029e0 2024-04-25T11:33:11.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.140+0000 7f85fc9a8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f85f810e6c0 con 0x7f85f8108920 2024-04-25T11:33:11.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.140+0000 7f85f5d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85f8103c20 0x7f85f819a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:11.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.140+0000 7f85f5d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85f8103c20 0x7f85f819a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53044/0 (socket says 172.21.15.90:53044) 2024-04-25T11:33:11.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.140+0000 7f85f5d9b640 1 -- 172.21.15.90:0/2732371936 learned_addr learned my addr 172.21.15.90:0/2732371936 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:11.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.140+0000 7f85f659c640 1 --2- 172.21.15.90:0/2732371936 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f85f81029e0 0x7f85f819a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:11.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.141+0000 7f85f6d9d640 1 --2- 172.21.15.90:0/2732371936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85f8108920 0x7f85f81a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:11.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.141+0000 7f85f5d9b640 1 -- 172.21.15.90:0/2732371936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85f8108920 msgr2=0x7f85f81a1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:11.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.141+0000 7f85f5d9b640 1 --2- 172.21.15.90:0/2732371936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85f8108920 0x7f85f81a1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.141+0000 7f85f5d9b640 1 -- 172.21.15.90:0/2732371936 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f85f81029e0 msgr2=0x7f85f819a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:11.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.141+0000 7f85f5d9b640 1 --2- 172.21.15.90:0/2732371936 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f85f81029e0 0x7f85f819a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.141+0000 7f85f5d9b640 1 -- 172.21.15.90:0/2732371936 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85f81a1de0 con 0x7f85f8103c20 2024-04-25T11:33:11.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.141+0000 7f85f6d9d640 1 --2- 172.21.15.90:0/2732371936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85f8108920 0x7f85f81a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:33:11.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.141+0000 7f85f5d9b640 1 --2- 172.21.15.90:0/2732371936 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85f8103c20 0x7f85f819a7e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f85ec00ba50 tx=0x7f85ec00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:11.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.141+0000 7f85e77fe640 1 -- 172.21.15.90:0/2732371936 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ec019070 con 0x7f85f8103c20 2024-04-25T11:33:11.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.141+0000 7f85fc9a8640 1 -- 172.21.15.90:0/2732371936 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85f81a20d0 con 0x7f85f8103c20 2024-04-25T11:33:11.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.141+0000 7f85e77fe640 1 -- 172.21.15.90:0/2732371936 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85ec00cbe0 con 0x7f85f8103c20 2024-04-25T11:33:11.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.141+0000 7f85e77fe640 1 -- 172.21.15.90:0/2732371936 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ec0075d0 con 0x7f85f8103c20 2024-04-25T11:33:11.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.141+0000 7f85fc9a8640 1 -- 172.21.15.90:0/2732371936 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85f81a2650 con 0x7f85f8103c20 2024-04-25T11:33:11.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.142+0000 7f85fc9a8640 1 -- 172.21.15.90:0/2732371936 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85f81a2910 con 0x7f85f8103c20 2024-04-25T11:33:11.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.144+0000 7f85e77fe640 1 -- 172.21.15.90:0/2732371936 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f85ec01c070 con 0x7f85f8103c20 2024-04-25T11:33:11.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.145+0000 7f85e77fe640 1 --2- 172.21.15.90:0/2732371936 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f85e00afaf0 0x7f85e00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:11.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.145+0000 7f85f659c640 1 --2- 172.21.15.90:0/2732371936 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f85e00afaf0 0x7f85e00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:11.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.145+0000 7f85e77fe640 1 -- 172.21.15.90:0/2732371936 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f85ec0de5d0 con 0x7f85f8103c20 2024-04-25T11:33:11.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.145+0000 7f85f659c640 1 --2- 172.21.15.90:0/2732371936 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f85e00afaf0 0x7f85e00b2000 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f85f0005ab0 tx=0x7f85f0031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:11.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.148+0000 7f85e77fe640 1 -- 172.21.15.90:0/2732371936 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f85ec0a6cb0 con 0x7f85f8103c20 2024-04-25T11:33:11.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:10 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:11.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:10 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:11.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:10 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:33:11.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:10 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:33:11.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:10 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:11.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:10 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3110647287' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T11:33:11.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:10 smithi090 ceph-mon[29575]: osdmap e75: 12 total, 12 up, 12 in 2024-04-25T11:33:11.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:10 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2190915803' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T11:33:11.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:10 smithi090 ceph-mon[29575]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T11:33:11.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:10 smithi090 ceph-mon[29575]: pgmap v272: 97 pgs: 9 creating+peering, 4 active+clean, 84 unknown; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:33:11.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:10 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:11.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:10 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:11.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:10 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:33:11.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:10 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:33:11.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:10 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:11.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:10 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3110647287' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T11:33:11.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:10 smithi138 ceph-mon[32328]: osdmap e75: 12 total, 12 up, 12 in 2024-04-25T11:33:11.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:10 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2190915803' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T11:33:11.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:10 smithi138 ceph-mon[32328]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T11:33:11.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:10 smithi138 ceph-mon[32328]: pgmap v272: 97 pgs: 9 creating+peering, 4 active+clean, 84 unknown; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:33:11.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.327+0000 7f85fc9a8640 1 -- 172.21.15.90:0/2732371936 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f85f81a2ae0 con 0x7f85f8103c20 2024-04-25T11:33:11.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.329+0000 7f85e77fe640 1 -- 172.21.15.90:0/2732371936 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) ==== 74+0+22251 (secure 0 0 0) 0x7f85ec0af380 con 0x7f85f8103c20 2024-04-25T11:33:11.317 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:33:11.318 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":76,"fsid":"58bd4e7a-02f6-11ef-bc93-c7b262605968","created":"2024-04-25T11:25:23.574550+0000","modified":"2024-04-25T11:33:10.373522+0000","last_up_change":"2024-04-25T11:32:23.852211+0000","last_in_change":"2024-04-25T11:32:04.188507+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:28:42.746649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:33:08.442925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:33:09.352886+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e54d888-e67b-4951-8ae7-e355fb6b10a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6803","nonce":1201629159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6805","nonce":1201629159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6809","nonce":1201629159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6807","nonce":1201629159}]},"public_addr":"172.21.15.90:6803/1201629159","cluster_addr":"172.21.15.90:6805/1201629159","heartbeat_back_addr":"172.21.15.90:6809/1201629159","heartbeat_front_addr":"172.21.15.90:6807/1201629159","state":["exists","up"]},{"osd":1,"uuid":"91a71c78-dae9-4bc7-9f6f-9e55cfefbe28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6801","nonce":3976851933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6803","nonce":3976851933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6807","nonce":3976851933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6805","nonce":3976851933}]},"public_addr":"172.21.15.138:6801/3976851933","cluster_addr":"172.21.15.138:6803/3976851933","heartbeat_back_addr":"172.21.15.138:6807/3976851933","heartbeat_front_addr":"172.21.15.138:6805/3976851933","state":["exists","up"]},{"osd":2,"uuid":"daff6c7f-80fa-4859-9339-ef7a5e840832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6801","nonce":1505807670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6803","nonce":1505807670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6807","nonce":1505807670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6805","nonce":1505807670}]},"public_addr":"172.21.15.156:6801/1505807670","cluster_addr":"172.21.15.156:6803/1505807670","heartbeat_back_addr":"172.21.15.156:6807/1505807670","heartbeat_front_addr":"172.21.15.156:6805/1505807670","state":["exists","up"]},{"osd":3,"uuid":"33fd94ca-7465-401a-bc1e-706b6b31b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6811","nonce":2520012383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6813","nonce":2520012383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6817","nonce":2520012383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6815","nonce":2520012383}]},"public_addr":"172.21.15.90:6811/2520012383","cluster_addr":"172.21.15.90:6813/2520012383","heartbeat_back_addr":"172.21.15.90:6817/2520012383","heartbeat_front_addr":"172.21.15.90:6815/2520012383","state":["exists","up"]},{"osd":4,"uuid":"c5326212-750c-49fd-96e3-d1e62bb4b7f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6809","nonce":3752678279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6811","nonce":3752678279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6815","nonce":3752678279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6813","nonce":3752678279}]},"public_addr":"172.21.15.138:6809/3752678279","cluster_addr":"172.21.15.138:6811/3752678279","heartbeat_back_addr":"172.21.15.138:6815/3752678279","heartbeat_front_addr":"172.21.15.138:6813/3752678279","state":["exists","up"]},{"osd":5,"uuid":"d04a1156-820b-4b43-be89-e16a8910c3aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6809","nonce":3591210191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6811","nonce":3591210191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6815","nonce":3591210191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6813","nonce":3591210191}]},"public_addr":"172.21.15.156:6809/3591210191","cluster_addr":"172.21.15.156:6811/3591210191","heartbeat_back_addr":"172.21.15.156:6815/3591210191","heartbeat_front_addr":"172.21.15.156:6813/3591210191","state":["exists","up"]},{"osd":6,"uuid":"75675aa3-e978-490c-8205-13d9868a3d6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6819","nonce":2364955456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6821","nonce":2364955456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6825","nonce":2364955456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6823","nonce":2364955456}]},"public_addr":"172.21.15.90:6819/2364955456","cluster_addr":"172.21.15.90:6821/2364955456","heartbeat_back_addr":"172.21.15.90:6825/2364955456","heartbeat_front_addr":"172.21.15.90:6823/2364955456","state":["exists","up"]},{"osd":7,"uuid":"1a95fdf5-f548-4271-bbcd-e330f7955eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6817","nonce":4012476387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6819","nonce":4012476387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6823","nonce":4012476387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6821","nonce":4012476387}]},"public_addr":"172.21.15.138:6817/4012476387","cluster_addr":"172.21.15.138:6819/4012476387","heartbeat_back_addr":"172.21.15.138:6823/4012476387","heartbeat_front_addr":"172.21.15.138:6821/4012476387","state":["exists","up"]},{"osd":8,"uuid":"605e10b5-5806-4c1f-8e28-a7994d34ddce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6817","nonce":101092224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6819","nonce":101092224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6823","nonce":101092224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6821","nonce":101092224}]},"public_addr":"172.21.15.156:6817/101092224","cluster_addr":"172.21.15.156:6819/101092224","heartbeat_back_addr":"172.21.15.156:6823/101092224","heartbeat_front_addr":"172.21.15.156:6821/101092224","state":["exists","up"]},{"osd":9,"uuid":"d70fdf5e-9d0f-483d-a7d8-505b79b8a418","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6827","nonce":1171265966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6829","nonce":1171265966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6833","nonce":1171265966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6831","nonce":1171265966}]},"public_addr":"172.21.15.90:6827/1171265966","cluster_addr":"172.21.15.90:6829/1171265966","heartbeat_back_addr":"172.21.15.90:6833/1171265966","heartbeat_front_addr":"172.21.15.90:6831/1171265966","state":["exists","up"]},{"osd":10,"uuid":"1ec86c7d-c3a4-432d-9664-8e40836e1e20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6825","nonce":4264666232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6827","nonce":4264666232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6831","nonce":4264666232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6829","nonce":4264666232}]},"public_addr":"172.21.15.138:6825/4264666232","cluster_addr":"172.21.15.138:6827/4264666232","heartbeat_back_addr":"172.21.15.138:6831/4264666232","heartbeat_front_addr":"172.21.15.138:6829/4264666232","state":["exists","up"]},{"osd":11,"uuid":"51cbe729-292e-4d7b-b8e9-e1e2821ffe76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6825","nonce":1224251744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6827","nonce":1224251744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6831","nonce":1224251744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6829","nonce":1224251744}]},"public_addr":"172.21.15.156:6825/1224251744","cluster_addr":"172.21.15.156:6827/1224251744","heartbeat_back_addr":"172.21.15.156:6831/1224251744","heartbeat_front_addr":"172.21.15.156:6829/1224251744","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:27:50.794226+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:15.030278+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:38.518163+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:03.107351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:27.500707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:51.463545+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:16.015076+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:41.182071+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:05.362309+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:31.048936+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:56.488509+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:32:22.129178+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/2381633953":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6800/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/2190846900":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1707792941":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6800/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6801/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/6061052":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1849633392":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6801/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/2277490702":"2024-04-26T11:26:16.655278+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:33:11.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.330+0000 7f85fc9a8640 1 -- 172.21.15.90:0/2732371936 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f85e00afaf0 msgr2=0x7f85e00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:11.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.330+0000 7f85fc9a8640 1 --2- 172.21.15.90:0/2732371936 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f85e00afaf0 0x7f85e00b2000 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f85f0005ab0 tx=0x7f85f0031400 comp rx=0 tx=0).stop 2024-04-25T11:33:11.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.331+0000 7f85fc9a8640 1 -- 172.21.15.90:0/2732371936 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85f8103c20 msgr2=0x7f85f819a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:11.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.331+0000 7f85fc9a8640 1 --2- 172.21.15.90:0/2732371936 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85f8103c20 0x7f85f819a7e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f85ec00ba50 tx=0x7f85ec00bf10 comp rx=0 tx=0).stop 2024-04-25T11:33:11.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.331+0000 7f85fc9a8640 1 -- 172.21.15.90:0/2732371936 shutdown_connections 2024-04-25T11:33:11.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.331+0000 7f85fc9a8640 1 --2- 172.21.15.90:0/2732371936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85f8108920 0x7f85f81a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.331+0000 7f85fc9a8640 1 --2- 172.21.15.90:0/2732371936 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f85e00afaf0 0x7f85e00b2000 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.331+0000 7f85fc9a8640 1 --2- 172.21.15.90:0/2732371936 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85f8103c20 0x7f85f819a7e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.331+0000 7f85fc9a8640 1 --2- 172.21.15.90:0/2732371936 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f85f81029e0 0x7f85f819a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.331+0000 7f85fc9a8640 1 -- 172.21.15.90:0/2732371936 >> 172.21.15.90:0/2732371936 conn(0x7f85f80fdbd0 msgr2=0x7f85f8109e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:11.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.331+0000 7f85fc9a8640 1 -- 172.21.15.90:0/2732371936 shutdown_connections 2024-04-25T11:33:11.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.331+0000 7f85fc9a8640 1 -- 172.21.15.90:0/2732371936 wait complete. 2024-04-25T11:33:11.331 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:33:11.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:10 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:11.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:10 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:11.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:10 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:33:11.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:10 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:33:11.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:10 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:33:11.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:10 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3110647287' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T11:33:11.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:10 smithi156 ceph-mon[33045]: osdmap e75: 12 total, 12 up, 12 in 2024-04-25T11:33:11.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:10 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2190915803' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T11:33:11.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:10 smithi156 ceph-mon[33045]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T11:33:11.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:10 smithi156 ceph-mon[33045]: pgmap v272: 97 pgs: 9 creating+peering, 4 active+clean, 84 unknown; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:33:11.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.450+0000 7f089a492640 1 -- 172.21.15.90:0/1165320568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0894108900 msgr2=0x7f089410ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:11.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.450+0000 7f089a492640 1 --2- 172.21.15.90:0/1165320568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0894108900 0x7f089410ad20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f088c009ca0 tx=0x7f088c02a8c0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.450+0000 7f089a492640 1 -- 172.21.15.90:0/1165320568 shutdown_connections 2024-04-25T11:33:11.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.450+0000 7f089a492640 1 --2- 172.21.15.90:0/1165320568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0894108900 0x7f089410ad20 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.450+0000 7f089a492640 1 --2- 172.21.15.90:0/1165320568 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0894103c00 0x7f0894104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.450+0000 7f089a492640 1 --2- 172.21.15.90:0/1165320568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08941029c0 0x7f08941036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.450+0000 7f089a492640 1 -- 172.21.15.90:0/1165320568 >> 172.21.15.90:0/1165320568 conn(0x7f08940fdbd0 msgr2=0x7f0894100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:11.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.450+0000 7f089a492640 1 -- 172.21.15.90:0/1165320568 shutdown_connections 2024-04-25T11:33:11.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.450+0000 7f089a492640 1 -- 172.21.15.90:0/1165320568 wait complete. 2024-04-25T11:33:11.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.450+0000 7f089a492640 1 Processor -- start 2024-04-25T11:33:11.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.450+0000 7f089a492640 1 -- start start 2024-04-25T11:33:11.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.451+0000 7f089a492640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08941029c0 0x7f089419a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:11.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.451+0000 7f089a492640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0894103c00 0x7f089419a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:11.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.451+0000 7f089a492640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0894108900 0x7f08941a1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:11.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.451+0000 7f089a492640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f089410f5f0 con 0x7f0894103c00 2024-04-25T11:33:11.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.451+0000 7f089a492640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f089410f470 con 0x7f0894108900 2024-04-25T11:33:11.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.451+0000 7f089a492640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f089410f770 con 0x7f08941029c0 2024-04-25T11:33:11.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.451+0000 7f08937fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0894103c00 0x7f089419a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:11.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.451+0000 7f08937fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0894103c00 0x7f089419a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36248/0 (socket says 172.21.15.90:36248) 2024-04-25T11:33:11.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.451+0000 7f08937fe640 1 -- 172.21.15.90:0/4198119812 learned_addr learned my addr 172.21.15.90:0/4198119812 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:11.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.451+0000 7f0893fff640 1 --2- 172.21.15.90:0/4198119812 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08941029c0 0x7f089419a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:11.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.451+0000 7f0898a2f640 1 --2- 172.21.15.90:0/4198119812 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0894108900 0x7f08941a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:11.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.451+0000 7f08937fe640 1 -- 172.21.15.90:0/4198119812 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08941029c0 msgr2=0x7f089419a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:11.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.451+0000 7f08937fe640 1 --2- 172.21.15.90:0/4198119812 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08941029c0 0x7f089419a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.451+0000 7f08937fe640 1 -- 172.21.15.90:0/4198119812 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0894108900 msgr2=0x7f08941a1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:11.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.451+0000 7f08937fe640 1 --2- 172.21.15.90:0/4198119812 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0894108900 0x7f08941a1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.451+0000 7f08937fe640 1 -- 172.21.15.90:0/4198119812 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08941a1ce0 con 0x7f0894103c00 2024-04-25T11:33:11.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.451+0000 7f0893fff640 1 --2- 172.21.15.90:0/4198119812 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08941029c0 0x7f089419a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:33:11.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.451+0000 7f08937fe640 1 --2- 172.21.15.90:0/4198119812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0894103c00 0x7f089419a6e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f088800d950 tx=0x7f088800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:11.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.451+0000 7f08917fa640 1 -- 172.21.15.90:0/4198119812 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0888017070 con 0x7f0894103c00 2024-04-25T11:33:11.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.451+0000 7f08917fa640 1 -- 172.21.15.90:0/4198119812 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0888009c30 con 0x7f0894103c00 2024-04-25T11:33:11.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.451+0000 7f089a492640 1 -- 172.21.15.90:0/4198119812 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08941a1fd0 con 0x7f0894103c00 2024-04-25T11:33:11.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.451+0000 7f08917fa640 1 -- 172.21.15.90:0/4198119812 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088800f620 con 0x7f0894103c00 2024-04-25T11:33:11.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.452+0000 7f089a492640 1 -- 172.21.15.90:0/4198119812 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08941a2440 con 0x7f0894103c00 2024-04-25T11:33:11.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.452+0000 7f089a492640 1 -- 172.21.15.90:0/4198119812 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0894107710 con 0x7f0894103c00 2024-04-25T11:33:11.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.454+0000 7f08917fa640 1 -- 172.21.15.90:0/4198119812 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f0888010460 con 0x7f0894103c00 2024-04-25T11:33:11.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.454+0000 7f08917fa640 1 --2- 172.21.15.90:0/4198119812 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f087c0afaf0 0x7f087c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:11.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.454+0000 7f0893fff640 1 --2- 172.21.15.90:0/4198119812 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f087c0afaf0 0x7f087c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:11.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.455+0000 7f08917fa640 1 -- 172.21.15.90:0/4198119812 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f08880de760 con 0x7f0894103c00 2024-04-25T11:33:11.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.455+0000 7f0893fff640 1 --2- 172.21.15.90:0/4198119812 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f087c0afaf0 0x7f087c0b2000 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f08941035c0 tx=0x7f088400a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:11.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.458+0000 7f08917fa640 1 -- 172.21.15.90:0/4198119812 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f08880a7490 con 0x7f0894103c00 2024-04-25T11:33:11.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.639+0000 7f089a492640 1 -- 172.21.15.90:0/4198119812 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f089402ddc0 con 0x7f0894103c00 2024-04-25T11:33:11.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.641+0000 7f08917fa640 1 -- 172.21.15.90:0/4198119812 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) ==== 74+0+22251 (secure 0 0 0) 0x7f0888009de0 con 0x7f0894103c00 2024-04-25T11:33:11.630 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:33:11.630 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":76,"fsid":"58bd4e7a-02f6-11ef-bc93-c7b262605968","created":"2024-04-25T11:25:23.574550+0000","modified":"2024-04-25T11:33:10.373522+0000","last_up_change":"2024-04-25T11:32:23.852211+0000","last_in_change":"2024-04-25T11:32:04.188507+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:28:42.746649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:33:08.442925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:33:09.352886+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e54d888-e67b-4951-8ae7-e355fb6b10a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6803","nonce":1201629159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6805","nonce":1201629159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6809","nonce":1201629159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6807","nonce":1201629159}]},"public_addr":"172.21.15.90:6803/1201629159","cluster_addr":"172.21.15.90:6805/1201629159","heartbeat_back_addr":"172.21.15.90:6809/1201629159","heartbeat_front_addr":"172.21.15.90:6807/1201629159","state":["exists","up"]},{"osd":1,"uuid":"91a71c78-dae9-4bc7-9f6f-9e55cfefbe28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6801","nonce":3976851933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6803","nonce":3976851933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6807","nonce":3976851933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6805","nonce":3976851933}]},"public_addr":"172.21.15.138:6801/3976851933","cluster_addr":"172.21.15.138:6803/3976851933","heartbeat_back_addr":"172.21.15.138:6807/3976851933","heartbeat_front_addr":"172.21.15.138:6805/3976851933","state":["exists","up"]},{"osd":2,"uuid":"daff6c7f-80fa-4859-9339-ef7a5e840832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6801","nonce":1505807670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6803","nonce":1505807670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6807","nonce":1505807670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6805","nonce":1505807670}]},"public_addr":"172.21.15.156:6801/1505807670","cluster_addr":"172.21.15.156:6803/1505807670","heartbeat_back_addr":"172.21.15.156:6807/1505807670","heartbeat_front_addr":"172.21.15.156:6805/1505807670","state":["exists","up"]},{"osd":3,"uuid":"33fd94ca-7465-401a-bc1e-706b6b31b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6811","nonce":2520012383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6813","nonce":2520012383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6817","nonce":2520012383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6815","nonce":2520012383}]},"public_addr":"172.21.15.90:6811/2520012383","cluster_addr":"172.21.15.90:6813/2520012383","heartbeat_back_addr":"172.21.15.90:6817/2520012383","heartbeat_front_addr":"172.21.15.90:6815/2520012383","state":["exists","up"]},{"osd":4,"uuid":"c5326212-750c-49fd-96e3-d1e62bb4b7f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6809","nonce":3752678279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6811","nonce":3752678279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6815","nonce":3752678279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6813","nonce":3752678279}]},"public_addr":"172.21.15.138:6809/3752678279","cluster_addr":"172.21.15.138:6811/3752678279","heartbeat_back_addr":"172.21.15.138:6815/3752678279","heartbeat_front_addr":"172.21.15.138:6813/3752678279","state":["exists","up"]},{"osd":5,"uuid":"d04a1156-820b-4b43-be89-e16a8910c3aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6809","nonce":3591210191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6811","nonce":3591210191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6815","nonce":3591210191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6813","nonce":3591210191}]},"public_addr":"172.21.15.156:6809/3591210191","cluster_addr":"172.21.15.156:6811/3591210191","heartbeat_back_addr":"172.21.15.156:6815/3591210191","heartbeat_front_addr":"172.21.15.156:6813/3591210191","state":["exists","up"]},{"osd":6,"uuid":"75675aa3-e978-490c-8205-13d9868a3d6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6819","nonce":2364955456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6821","nonce":2364955456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6825","nonce":2364955456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6823","nonce":2364955456}]},"public_addr":"172.21.15.90:6819/2364955456","cluster_addr":"172.21.15.90:6821/2364955456","heartbeat_back_addr":"172.21.15.90:6825/2364955456","heartbeat_front_addr":"172.21.15.90:6823/2364955456","state":["exists","up"]},{"osd":7,"uuid":"1a95fdf5-f548-4271-bbcd-e330f7955eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6817","nonce":4012476387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6819","nonce":4012476387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6823","nonce":4012476387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6821","nonce":4012476387}]},"public_addr":"172.21.15.138:6817/4012476387","cluster_addr":"172.21.15.138:6819/4012476387","heartbeat_back_addr":"172.21.15.138:6823/4012476387","heartbeat_front_addr":"172.21.15.138:6821/4012476387","state":["exists","up"]},{"osd":8,"uuid":"605e10b5-5806-4c1f-8e28-a7994d34ddce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6817","nonce":101092224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6819","nonce":101092224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6823","nonce":101092224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6821","nonce":101092224}]},"public_addr":"172.21.15.156:6817/101092224","cluster_addr":"172.21.15.156:6819/101092224","heartbeat_back_addr":"172.21.15.156:6823/101092224","heartbeat_front_addr":"172.21.15.156:6821/101092224","state":["exists","up"]},{"osd":9,"uuid":"d70fdf5e-9d0f-483d-a7d8-505b79b8a418","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6827","nonce":1171265966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6829","nonce":1171265966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6833","nonce":1171265966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6831","nonce":1171265966}]},"public_addr":"172.21.15.90:6827/1171265966","cluster_addr":"172.21.15.90:6829/1171265966","heartbeat_back_addr":"172.21.15.90:6833/1171265966","heartbeat_front_addr":"172.21.15.90:6831/1171265966","state":["exists","up"]},{"osd":10,"uuid":"1ec86c7d-c3a4-432d-9664-8e40836e1e20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6825","nonce":4264666232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6827","nonce":4264666232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6831","nonce":4264666232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6829","nonce":4264666232}]},"public_addr":"172.21.15.138:6825/4264666232","cluster_addr":"172.21.15.138:6827/4264666232","heartbeat_back_addr":"172.21.15.138:6831/4264666232","heartbeat_front_addr":"172.21.15.138:6829/4264666232","state":["exists","up"]},{"osd":11,"uuid":"51cbe729-292e-4d7b-b8e9-e1e2821ffe76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6825","nonce":1224251744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6827","nonce":1224251744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6831","nonce":1224251744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6829","nonce":1224251744}]},"public_addr":"172.21.15.156:6825/1224251744","cluster_addr":"172.21.15.156:6827/1224251744","heartbeat_back_addr":"172.21.15.156:6831/1224251744","heartbeat_front_addr":"172.21.15.156:6829/1224251744","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:27:50.794226+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:15.030278+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:38.518163+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:03.107351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:27.500707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:51.463545+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:16.015076+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:41.182071+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:05.362309+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:31.048936+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:56.488509+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:32:22.129178+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/2381633953":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6800/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/2190846900":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1707792941":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6800/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6801/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/6061052":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1849633392":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6801/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/2277490702":"2024-04-26T11:26:16.655278+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:33:11.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.643+0000 7f089a492640 1 -- 172.21.15.90:0/4198119812 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f087c0afaf0 msgr2=0x7f087c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:11.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.643+0000 7f089a492640 1 --2- 172.21.15.90:0/4198119812 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f087c0afaf0 0x7f087c0b2000 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f08941035c0 tx=0x7f088400a560 comp rx=0 tx=0).stop 2024-04-25T11:33:11.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.643+0000 7f089a492640 1 -- 172.21.15.90:0/4198119812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0894103c00 msgr2=0x7f089419a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:11.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.643+0000 7f089a492640 1 --2- 172.21.15.90:0/4198119812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0894103c00 0x7f089419a6e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f088800d950 tx=0x7f088800de10 comp rx=0 tx=0).stop 2024-04-25T11:33:11.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.643+0000 7f089a492640 1 -- 172.21.15.90:0/4198119812 shutdown_connections 2024-04-25T11:33:11.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.643+0000 7f089a492640 1 --2- 172.21.15.90:0/4198119812 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0894108900 0x7f08941a1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.643+0000 7f089a492640 1 --2- 172.21.15.90:0/4198119812 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f087c0afaf0 0x7f087c0b2000 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.643+0000 7f089a492640 1 --2- 172.21.15.90:0/4198119812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0894103c00 0x7f089419a6e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.643+0000 7f089a492640 1 --2- 172.21.15.90:0/4198119812 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08941029c0 0x7f089419a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.643+0000 7f089a492640 1 -- 172.21.15.90:0/4198119812 >> 172.21.15.90:0/4198119812 conn(0x7f08940fdbd0 msgr2=0x7f08941048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:11.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.643+0000 7f089a492640 1 -- 172.21.15.90:0/4198119812 shutdown_connections 2024-04-25T11:33:11.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.643+0000 7f089a492640 1 -- 172.21.15.90:0/4198119812 wait complete. 2024-04-25T11:33:11.643 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T11:33:11.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.765+0000 7fbb2c051640 1 -- 172.21.15.90:0/3471784595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb241029e0 msgr2=0x7fbb241036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:11.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.765+0000 7fbb2c051640 1 --2- 172.21.15.90:0/3471784595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb241029e0 0x7fbb241036e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c009ca0 tx=0x7fbb1c02a8c0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.765+0000 7fbb2c051640 1 -- 172.21.15.90:0/3471784595 shutdown_connections 2024-04-25T11:33:11.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.765+0000 7fbb2c051640 1 --2- 172.21.15.90:0/3471784595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb24108920 0x7fbb2410ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.765+0000 7fbb2c051640 1 --2- 172.21.15.90:0/3471784595 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb24103c20 0x7fbb24104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.765+0000 7fbb2c051640 1 --2- 172.21.15.90:0/3471784595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb241029e0 0x7fbb241036e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.765+0000 7fbb2c051640 1 -- 172.21.15.90:0/3471784595 >> 172.21.15.90:0/3471784595 conn(0x7fbb240fdbd0 msgr2=0x7fbb24100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:11.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.765+0000 7fbb2c051640 1 -- 172.21.15.90:0/3471784595 shutdown_connections 2024-04-25T11:33:11.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.765+0000 7fbb2c051640 1 -- 172.21.15.90:0/3471784595 wait complete. 2024-04-25T11:33:11.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.766+0000 7fbb2c051640 1 Processor -- start 2024-04-25T11:33:11.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.766+0000 7fbb2c051640 1 -- start start 2024-04-25T11:33:11.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.766+0000 7fbb2c051640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb241029e0 0x7fbb2406a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:11.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.766+0000 7fbb2c051640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb24103c20 0x7fbb2419f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:11.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.766+0000 7fbb2c051640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb24108920 0x7fbb241a1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:11.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.766+0000 7fbb2c051640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2410e4e0 con 0x7fbb241029e0 2024-04-25T11:33:11.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.766+0000 7fbb2c051640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2410e360 con 0x7fbb24103c20 2024-04-25T11:33:11.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.766+0000 7fbb2c051640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2410e660 con 0x7fbb24108920 2024-04-25T11:33:11.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.766+0000 7fbb29ded640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb241029e0 0x7fbb2406a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:11.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.766+0000 7fbb29ded640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb241029e0 0x7fbb2406a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36270/0 (socket says 172.21.15.90:36270) 2024-04-25T11:33:11.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.766+0000 7fbb29ded640 1 -- 172.21.15.90:0/2593066523 learned_addr learned my addr 172.21.15.90:0/2593066523 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:11.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.766+0000 7fbb295ec640 1 --2- 172.21.15.90:0/2593066523 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb24103c20 0x7fbb2419f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:11.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.766+0000 7fbb29ded640 1 -- 172.21.15.90:0/2593066523 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb24108920 msgr2=0x7fbb241a1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:11.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.766+0000 7fbb29ded640 1 --2- 172.21.15.90:0/2593066523 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb24108920 0x7fbb241a1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.766+0000 7fbb29ded640 1 -- 172.21.15.90:0/2593066523 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb24103c20 msgr2=0x7fbb2419f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:11.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.766+0000 7fbb29ded640 1 --2- 172.21.15.90:0/2593066523 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb24103c20 0x7fbb2419f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:11.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.766+0000 7fbb29ded640 1 -- 172.21.15.90:0/2593066523 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb241a2080 con 0x7fbb241029e0 2024-04-25T11:33:11.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.766+0000 7fbb29ded640 1 --2- 172.21.15.90:0/2593066523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb241029e0 0x7fbb2406a920 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c0098d0 tx=0x7fbb1c031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:11.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.767+0000 7fbb1affd640 1 -- 172.21.15.90:0/2593066523 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c003cb0 con 0x7fbb241029e0 2024-04-25T11:33:11.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.767+0000 7fbb2c051640 1 -- 172.21.15.90:0/2593066523 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb241a2310 con 0x7fbb241029e0 2024-04-25T11:33:11.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.767+0000 7fbb1affd640 1 -- 172.21.15.90:0/2593066523 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb1c003e50 con 0x7fbb241029e0 2024-04-25T11:33:11.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.767+0000 7fbb1affd640 1 -- 172.21.15.90:0/2593066523 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c02fc60 con 0x7fbb241029e0 2024-04-25T11:33:11.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.767+0000 7fbb2c051640 1 -- 172.21.15.90:0/2593066523 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb241a28b0 con 0x7fbb241029e0 2024-04-25T11:33:11.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.768+0000 7fbb2c051640 1 -- 172.21.15.90:0/2593066523 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb24107730 con 0x7fbb241029e0 2024-04-25T11:33:11.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.769+0000 7fbb1affd640 1 -- 172.21.15.90:0/2593066523 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fbb1c004710 con 0x7fbb241029e0 2024-04-25T11:33:11.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.770+0000 7fbb1affd640 1 --2- 172.21.15.90:0/2593066523 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbb100afaf0 0x7fbb100b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:11.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.770+0000 7fbb295ec640 1 --2- 172.21.15.90:0/2593066523 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbb100afaf0 0x7fbb100b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:11.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.770+0000 7fbb1affd640 1 -- 172.21.15.90:0/2593066523 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7fbb1c1003e0 con 0x7fbb241029e0 2024-04-25T11:33:11.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.770+0000 7fbb295ec640 1 --2- 172.21.15.90:0/2593066523 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbb100afaf0 0x7fbb100b2000 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbb2406a2c0 tx=0x7fbb1400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:11.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.774+0000 7fbb1affd640 1 -- 172.21.15.90:0/2593066523 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbb1c0c9100 con 0x7fbb241029e0 2024-04-25T11:33:11.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:11 smithi090 ceph-mon[29575]: osdmap e76: 12 total, 12 up, 12 in 2024-04-25T11:33:11.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:11 smithi090 ceph-mon[29575]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T11:33:11.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:11 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2190915803' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T11:33:11.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:11 smithi090 ceph-mon[29575]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T11:33:11.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:11 smithi090 ceph-mon[29575]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T11:33:11.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:11 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2732371936' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:33:11.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:11 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/4198119812' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:33:11.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:11.996+0000 7fbb2c051640 1 -- 172.21.15.90:0/2593066523 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fbb2406c2a0 con 0x7fbb241029e0 2024-04-25T11:33:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.022+0000 7fbb1affd640 1 -- 172.21.15.90:0/2593066523 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7fbb1c0c8a00 con 0x7fbb241029e0 2024-04-25T11:33:12.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.024+0000 7fbb2c051640 1 -- 172.21.15.90:0/2593066523 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbb100afaf0 msgr2=0x7fbb100b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:12.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.024+0000 7fbb2c051640 1 --2- 172.21.15.90:0/2593066523 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbb100afaf0 0x7fbb100b2000 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbb2406a2c0 tx=0x7fbb1400b390 comp rx=0 tx=0).stop 2024-04-25T11:33:12.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.024+0000 7fbb2c051640 1 -- 172.21.15.90:0/2593066523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb241029e0 msgr2=0x7fbb2406a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:12.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.024+0000 7fbb2c051640 1 --2- 172.21.15.90:0/2593066523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb241029e0 0x7fbb2406a920 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c0098d0 tx=0x7fbb1c031840 comp rx=0 tx=0).stop 2024-04-25T11:33:12.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.024+0000 7fbb2c051640 1 -- 172.21.15.90:0/2593066523 shutdown_connections 2024-04-25T11:33:12.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.024+0000 7fbb2c051640 1 --2- 172.21.15.90:0/2593066523 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb24108920 0x7fbb241a1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:12.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.024+0000 7fbb2c051640 1 --2- 172.21.15.90:0/2593066523 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb24103c20 0x7fbb2419f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:12.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.024+0000 7fbb2c051640 1 --2- 172.21.15.90:0/2593066523 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbb100afaf0 0x7fbb100b2000 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:12.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.024+0000 7fbb2c051640 1 --2- 172.21.15.90:0/2593066523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbb241029e0 0x7fbb2406a920 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:12.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.024+0000 7fbb2c051640 1 -- 172.21.15.90:0/2593066523 >> 172.21.15.90:0/2593066523 conn(0x7fbb240fdbd0 msgr2=0x7fbb24104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:12.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.024+0000 7fbb2c051640 1 -- 172.21.15.90:0/2593066523 shutdown_connections 2024-04-25T11:33:12.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.024+0000 7fbb2c051640 1 -- 172.21.15.90:0/2593066523 wait complete. 2024-04-25T11:33:12.027 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-25T11:33:12.027 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-04-25T11:33:12.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:11 smithi138 ceph-mon[32328]: osdmap e76: 12 total, 12 up, 12 in 2024-04-25T11:33:12.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:11 smithi138 ceph-mon[32328]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T11:33:12.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:11 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2190915803' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T11:33:12.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:11 smithi138 ceph-mon[32328]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T11:33:12.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:11 smithi138 ceph-mon[32328]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T11:33:12.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:11 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2732371936' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:33:12.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:11 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/4198119812' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:33:12.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.152+0000 7fed7a98f640 1 -- 172.21.15.90:0/2639928646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed74104480 msgr2=0x7fed7410abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:12.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.152+0000 7fed7a98f640 1 --2- 172.21.15.90:0/2639928646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed74104480 0x7fed7410abc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fed6c009ca0 tx=0x7fed6c02a8e0 comp rx=0 tx=0).stop 2024-04-25T11:33:12.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.152+0000 7fed7a98f640 1 -- 172.21.15.90:0/2639928646 shutdown_connections 2024-04-25T11:33:12.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.152+0000 7fed7a98f640 1 --2- 172.21.15.90:0/2639928646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed74104480 0x7fed7410abc0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:12.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.152+0000 7fed7a98f640 1 --2- 172.21.15.90:0/2639928646 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed74103940 0x7fed74103df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:12.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.152+0000 7fed7a98f640 1 --2- 172.21.15.90:0/2639928646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed74102f60 0x7fed74103370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:12.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.152+0000 7fed7a98f640 1 -- 172.21.15.90:0/2639928646 >> 172.21.15.90:0/2639928646 conn(0x7fed740fd380 msgr2=0x7fed740ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:12.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.152+0000 7fed7a98f640 1 -- 172.21.15.90:0/2639928646 shutdown_connections 2024-04-25T11:33:12.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.152+0000 7fed7a98f640 1 -- 172.21.15.90:0/2639928646 wait complete. 2024-04-25T11:33:12.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.153+0000 7fed7a98f640 1 Processor -- start 2024-04-25T11:33:12.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.153+0000 7fed7a98f640 1 -- start start 2024-04-25T11:33:12.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.153+0000 7fed7a98f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed74102f60 0x7fed741991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:12.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.153+0000 7fed7a98f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed74103940 0x7fed741996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:12.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.153+0000 7fed7a98f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed74104480 0x7fed741a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:12.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.153+0000 7fed7a98f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fed7410e590 con 0x7fed74102f60 2024-04-25T11:33:12.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.153+0000 7fed7a98f640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fed7410e410 con 0x7fed74103940 2024-04-25T11:33:12.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.153+0000 7fed7a98f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fed7410e710 con 0x7fed74104480 2024-04-25T11:33:12.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.153+0000 7fed73fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed74102f60 0x7fed741991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:12.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.153+0000 7fed73fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed74102f60 0x7fed741991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36282/0 (socket says 172.21.15.90:36282) 2024-04-25T11:33:12.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.153+0000 7fed73fff640 1 -- 172.21.15.90:0/1715083484 learned_addr learned my addr 172.21.15.90:0/1715083484 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:12.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.154+0000 7fed737fe640 1 --2- 172.21.15.90:0/1715083484 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed74103940 0x7fed741996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:12.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.154+0000 7fed73fff640 1 -- 172.21.15.90:0/1715083484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed74104480 msgr2=0x7fed741a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:12.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.154+0000 7fed78f2c640 1 --2- 172.21.15.90:0/1715083484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed74104480 0x7fed741a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:12.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.154+0000 7fed73fff640 1 --2- 172.21.15.90:0/1715083484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed74104480 0x7fed741a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:12.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.154+0000 7fed73fff640 1 -- 172.21.15.90:0/1715083484 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed74103940 msgr2=0x7fed741996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:12.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.154+0000 7fed73fff640 1 --2- 172.21.15.90:0/1715083484 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed74103940 0x7fed741996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:12.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.154+0000 7fed73fff640 1 -- 172.21.15.90:0/1715083484 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed741a0ce0 con 0x7fed74102f60 2024-04-25T11:33:12.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.154+0000 7fed78f2c640 1 --2- 172.21.15.90:0/1715083484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed74104480 0x7fed741a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:33:12.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.154+0000 7fed737fe640 1 --2- 172.21.15.90:0/1715083484 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed74103940 0x7fed741996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:33:12.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.154+0000 7fed73fff640 1 --2- 172.21.15.90:0/1715083484 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed74102f60 0x7fed741991a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fed64006fd0 tx=0x7fed6400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:12.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.154+0000 7fed717fa640 1 -- 172.21.15.90:0/1715083484 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed64011830 con 0x7fed74102f60 2024-04-25T11:33:12.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.154+0000 7fed7a98f640 1 -- 172.21.15.90:0/1715083484 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed741a0fd0 con 0x7fed74102f60 2024-04-25T11:33:12.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.154+0000 7fed717fa640 1 -- 172.21.15.90:0/1715083484 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed64016070 con 0x7fed74102f60 2024-04-25T11:33:12.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.154+0000 7fed717fa640 1 -- 172.21.15.90:0/1715083484 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed64010e00 con 0x7fed74102f60 2024-04-25T11:33:12.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.154+0000 7fed7a98f640 1 -- 172.21.15.90:0/1715083484 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed741a1430 con 0x7fed74102f60 2024-04-25T11:33:12.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.155+0000 7fed7a98f640 1 -- 172.21.15.90:0/1715083484 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed7406a160 con 0x7fed74102f60 2024-04-25T11:33:12.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.156+0000 7fed717fa640 1 -- 172.21.15.90:0/1715083484 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fed6401b460 con 0x7fed74102f60 2024-04-25T11:33:12.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.157+0000 7fed717fa640 1 --2- 172.21.15.90:0/1715083484 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fed5c0afaf0 0x7fed5c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:12.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.157+0000 7fed717fa640 1 -- 172.21.15.90:0/1715083484 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7fed640de1a0 con 0x7fed74102f60 2024-04-25T11:33:12.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.157+0000 7fed737fe640 1 --2- 172.21.15.90:0/1715083484 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fed5c0afaf0 0x7fed5c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:12.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.158+0000 7fed737fe640 1 --2- 172.21.15.90:0/1715083484 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fed5c0afaf0 0x7fed5c0b2000 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fed68005fd0 tx=0x7fed68005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:12.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.162+0000 7fed717fa640 1 -- 172.21.15.90:0/1715083484 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fed640a6ed0 con 0x7fed74102f60 2024-04-25T11:33:12.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:11 smithi156 ceph-mon[33045]: osdmap e76: 12 total, 12 up, 12 in 2024-04-25T11:33:12.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:11 smithi156 ceph-mon[33045]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T11:33:12.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:11 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2190915803' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T11:33:12.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:11 smithi156 ceph-mon[33045]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T11:33:12.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:11 smithi156 ceph-mon[33045]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T11:33:12.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:11 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2732371936' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:33:12.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:11 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/4198119812' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:33:12.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:12.374+0000 7fed7a98f640 1 -- 172.21.15.90:0/1715083484 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7fed7402db50 con 0x7fed74102f60 2024-04-25T11:33:13.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.026+0000 7fed717fa640 1 -- 172.21.15.90:0/1715083484 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v16) ==== 105+0+0 (secure 0 0 0) 0x7fed640a67d0 con 0x7fed74102f60 2024-04-25T11:33:13.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.029+0000 7fed7a98f640 1 -- 172.21.15.90:0/1715083484 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fed5c0afaf0 msgr2=0x7fed5c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:13.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.029+0000 7fed7a98f640 1 --2- 172.21.15.90:0/1715083484 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fed5c0afaf0 0x7fed5c0b2000 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fed68005fd0 tx=0x7fed68005850 comp rx=0 tx=0).stop 2024-04-25T11:33:13.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.029+0000 7fed7a98f640 1 -- 172.21.15.90:0/1715083484 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed74102f60 msgr2=0x7fed741991a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:13.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.029+0000 7fed7a98f640 1 --2- 172.21.15.90:0/1715083484 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed74102f60 0x7fed741991a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fed64006fd0 tx=0x7fed6400b8e0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.029+0000 7fed7a98f640 1 -- 172.21.15.90:0/1715083484 shutdown_connections 2024-04-25T11:33:13.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.029+0000 7fed7a98f640 1 --2- 172.21.15.90:0/1715083484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed74104480 0x7fed741a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.029+0000 7fed7a98f640 1 --2- 172.21.15.90:0/1715083484 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed74103940 0x7fed741996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.029+0000 7fed7a98f640 1 --2- 172.21.15.90:0/1715083484 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fed5c0afaf0 0x7fed5c0b2000 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.029+0000 7fed7a98f640 1 --2- 172.21.15.90:0/1715083484 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed74102f60 0x7fed741991a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.029+0000 7fed7a98f640 1 -- 172.21.15.90:0/1715083484 >> 172.21.15.90:0/1715083484 conn(0x7fed740fd380 msgr2=0x7fed74106b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:13.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.029+0000 7fed7a98f640 1 -- 172.21.15.90:0/1715083484 shutdown_connections 2024-04-25T11:33:13.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.029+0000 7fed7a98f640 1 -- 172.21.15.90:0/1715083484 wait complete. 2024-04-25T11:33:13.033 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-25T11:33:13.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.156+0000 7f398d1b2640 1 -- 172.21.15.90:0/3553218548 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3988102f60 msgr2=0x7f3988103370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:13.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.156+0000 7f398d1b2640 1 --2- 172.21.15.90:0/3553218548 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3988102f60 0x7f3988103370 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3978009ca0 tx=0x7f397802a8c0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.156+0000 7f398d1b2640 1 -- 172.21.15.90:0/3553218548 shutdown_connections 2024-04-25T11:33:13.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.156+0000 7f398d1b2640 1 --2- 172.21.15.90:0/3553218548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3988104460 0x7f398810abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.156+0000 7f398d1b2640 1 --2- 172.21.15.90:0/3553218548 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3988103940 0x7f3988103dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.156+0000 7f398d1b2640 1 --2- 172.21.15.90:0/3553218548 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3988102f60 0x7f3988103370 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.156+0000 7f398d1b2640 1 -- 172.21.15.90:0/3553218548 >> 172.21.15.90:0/3553218548 conn(0x7f39880fd340 msgr2=0x7f39880ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:13.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.157+0000 7f398d1b2640 1 -- 172.21.15.90:0/3553218548 shutdown_connections 2024-04-25T11:33:13.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.157+0000 7f398d1b2640 1 -- 172.21.15.90:0/3553218548 wait complete. 2024-04-25T11:33:13.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.157+0000 7f398d1b2640 1 Processor -- start 2024-04-25T11:33:13.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.157+0000 7f398d1b2640 1 -- start start 2024-04-25T11:33:13.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.158+0000 7f398d1b2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3988102f60 0x7f3988102200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:13.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.158+0000 7f398d1b2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3988103940 0x7f398819f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:13.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.158+0000 7f398d1b2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3988104460 0x7f39881a1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:13.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.158+0000 7f398d1b2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f398810e5e0 con 0x7f3988102f60 2024-04-25T11:33:13.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.158+0000 7f398d1b2640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f398810e460 con 0x7f3988104460 2024-04-25T11:33:13.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.158+0000 7f398d1b2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f398810e760 con 0x7f3988103940 2024-04-25T11:33:13.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.158+0000 7f3986d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3988102f60 0x7f3988102200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:13.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.158+0000 7f3986d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3988102f60 0x7f3988102200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36306/0 (socket says 172.21.15.90:36306) 2024-04-25T11:33:13.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.158+0000 7f3986d9d640 1 -- 172.21.15.90:0/1602163597 learned_addr learned my addr 172.21.15.90:0/1602163597 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:13.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.158+0000 7f398659c640 1 --2- 172.21.15.90:0/1602163597 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3988103940 0x7f398819f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:13.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.158+0000 7f398759e640 1 --2- 172.21.15.90:0/1602163597 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3988104460 0x7f39881a1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:13.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.158+0000 7f3986d9d640 1 -- 172.21.15.90:0/1602163597 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3988103940 msgr2=0x7f398819f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:13.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.158+0000 7f3986d9d640 1 --2- 172.21.15.90:0/1602163597 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3988103940 0x7f398819f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.158+0000 7f3986d9d640 1 -- 172.21.15.90:0/1602163597 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3988104460 msgr2=0x7f39881a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:13.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.158+0000 7f3986d9d640 1 --2- 172.21.15.90:0/1602163597 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3988104460 0x7f39881a1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.158+0000 7f3986d9d640 1 -- 172.21.15.90:0/1602163597 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39881a1e70 con 0x7f3988102f60 2024-04-25T11:33:13.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.158+0000 7f3986d9d640 1 --2- 172.21.15.90:0/1602163597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3988102f60 0x7f3988102200 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3978031b50 tx=0x7f3978003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:13.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.158+0000 7f396ffff640 1 -- 172.21.15.90:0/1602163597 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3978004040 con 0x7f3988102f60 2024-04-25T11:33:13.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.159+0000 7f398d1b2640 1 -- 172.21.15.90:0/1602163597 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39881a2100 con 0x7f3988102f60 2024-04-25T11:33:13.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.159+0000 7f396ffff640 1 -- 172.21.15.90:0/1602163597 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3978004690 con 0x7f3988102f60 2024-04-25T11:33:13.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.159+0000 7f398d1b2640 1 -- 172.21.15.90:0/1602163597 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39881a2720 con 0x7f3988102f60 2024-04-25T11:33:13.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.159+0000 7f396ffff640 1 -- 172.21.15.90:0/1602163597 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f397802fde0 con 0x7f3988102f60 2024-04-25T11:33:13.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.160+0000 7f398d1b2640 1 -- 172.21.15.90:0/1602163597 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3988107900 con 0x7f3988102f60 2024-04-25T11:33:13.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.161+0000 7f396ffff640 1 -- 172.21.15.90:0/1602163597 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f39780041e0 con 0x7f3988102f60 2024-04-25T11:33:13.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.162+0000 7f396ffff640 1 --2- 172.21.15.90:0/1602163597 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f39700afaf0 0x7f39700b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:13.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.162+0000 7f398659c640 1 --2- 172.21.15.90:0/1602163597 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f39700afaf0 0x7f39700b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:13.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.162+0000 7f396ffff640 1 -- 172.21.15.90:0/1602163597 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7f3978100570 con 0x7f3988102f60 2024-04-25T11:33:13.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.162+0000 7f398659c640 1 --2- 172.21.15.90:0/1602163597 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f39700afaf0 0x7f39700b2000 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3988101ba0 tx=0x7f397c009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:13.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.165+0000 7f396ffff640 1 -- 172.21.15.90:0/1602163597 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f39780393b0 con 0x7f3988102f60 2024-04-25T11:33:13.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:12 smithi090 ceph-mon[29575]: osdmap e77: 12 total, 12 up, 12 in 2024-04-25T11:33:13.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:12 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2593066523' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T11:33:13.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:12 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2593066523' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T11:33:13.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:12 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] up:active 2024-04-25T11:33:13.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:12 smithi090 ceph-mon[29575]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T11:33:13.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:12 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1715083484' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-25T11:33:13.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:12 smithi090 ceph-mon[29575]: pgmap v275: 97 pgs: 9 creating+peering, 11 active+clean, 77 unknown; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 0 op/s 2024-04-25T11:33:13.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:12 smithi138 ceph-mon[32328]: osdmap e77: 12 total, 12 up, 12 in 2024-04-25T11:33:13.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:12 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2593066523' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T11:33:13.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:12 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2593066523' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T11:33:13.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:12 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] up:active 2024-04-25T11:33:13.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:12 smithi138 ceph-mon[32328]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T11:33:13.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:12 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1715083484' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-25T11:33:13.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:12 smithi138 ceph-mon[32328]: pgmap v275: 97 pgs: 9 creating+peering, 11 active+clean, 77 unknown; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 0 op/s 2024-04-25T11:33:13.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:12 smithi156 ceph-mon[33045]: osdmap e77: 12 total, 12 up, 12 in 2024-04-25T11:33:13.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:12 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2593066523' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T11:33:13.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:12 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2593066523' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T11:33:13.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:12 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] up:active 2024-04-25T11:33:13.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:12 smithi156 ceph-mon[33045]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T11:33:13.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:12 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1715083484' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-25T11:33:13.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:12 smithi156 ceph-mon[33045]: pgmap v275: 97 pgs: 9 creating+peering, 11 active+clean, 77 unknown; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 0 op/s 2024-04-25T11:33:13.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.381+0000 7f398d1b2640 1 -- 172.21.15.90:0/1602163597 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f398802db50 con 0x7f3988102f60 2024-04-25T11:33:13.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.606+0000 7f396ffff640 1 -- 172.21.15.90:0/1602163597 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v18) ==== 121+0+0 (secure 0 0 0) 0x7f39780c92a0 con 0x7f3988102f60 2024-04-25T11:33:13.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.608+0000 7f398d1b2640 1 -- 172.21.15.90:0/1602163597 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f39700afaf0 msgr2=0x7f39700b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:13.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.608+0000 7f398d1b2640 1 --2- 172.21.15.90:0/1602163597 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f39700afaf0 0x7f39700b2000 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3988101ba0 tx=0x7f397c009110 comp rx=0 tx=0).stop 2024-04-25T11:33:13.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.608+0000 7f398d1b2640 1 -- 172.21.15.90:0/1602163597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3988102f60 msgr2=0x7f3988102200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:13.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.608+0000 7f398d1b2640 1 --2- 172.21.15.90:0/1602163597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3988102f60 0x7f3988102200 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3978031b50 tx=0x7f3978003ba0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.609+0000 7f398d1b2640 1 -- 172.21.15.90:0/1602163597 shutdown_connections 2024-04-25T11:33:13.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.609+0000 7f398d1b2640 1 --2- 172.21.15.90:0/1602163597 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3988104460 0x7f39881a1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.609+0000 7f398d1b2640 1 --2- 172.21.15.90:0/1602163597 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3988103940 0x7f398819f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.609+0000 7f398d1b2640 1 --2- 172.21.15.90:0/1602163597 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f39700afaf0 0x7f39700b2000 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.609+0000 7f398d1b2640 1 --2- 172.21.15.90:0/1602163597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3988102f60 0x7f3988102200 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.609+0000 7f398d1b2640 1 -- 172.21.15.90:0/1602163597 >> 172.21.15.90:0/1602163597 conn(0x7f39880fd340 msgr2=0x7f3988104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:13.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.609+0000 7f398d1b2640 1 -- 172.21.15.90:0/1602163597 shutdown_connections 2024-04-25T11:33:13.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.609+0000 7f398d1b2640 1 -- 172.21.15.90:0/1602163597 wait complete. 2024-04-25T11:33:13.611 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-25T11:33:13.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.737+0000 7fbce6984640 1 -- 172.21.15.90:0/3029588340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbce0104d40 msgr2=0x7fbce0105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:13.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.737+0000 7fbce6984640 1 --2- 172.21.15.90:0/3029588340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbce0104d40 0x7fbce0105150 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbcd0009900 tx=0x7fbcd002a410 comp rx=0 tx=0).stop 2024-04-25T11:33:13.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.737+0000 7fbce6984640 1 -- 172.21.15.90:0/3029588340 shutdown_connections 2024-04-25T11:33:13.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.737+0000 7fbce6984640 1 --2- 172.21.15.90:0/3029588340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbce0109a40 0x7fbce010be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.737+0000 7fbce6984640 1 --2- 172.21.15.90:0/3029588340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbce0104d40 0x7fbce0105150 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.737+0000 7fbce6984640 1 --2- 172.21.15.90:0/3029588340 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbce0103b00 0x7fbce0104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.737+0000 7fbce6984640 1 -- 172.21.15.90:0/3029588340 >> 172.21.15.90:0/3029588340 conn(0x7fbce00fed10 msgr2=0x7fbce01011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:13.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.737+0000 7fbce6984640 1 -- 172.21.15.90:0/3029588340 shutdown_connections 2024-04-25T11:33:13.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.737+0000 7fbce6984640 1 -- 172.21.15.90:0/3029588340 wait complete. 2024-04-25T11:33:13.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.737+0000 7fbce6984640 1 Processor -- start 2024-04-25T11:33:13.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.737+0000 7fbce6984640 1 -- start start 2024-04-25T11:33:13.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.738+0000 7fbce6984640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbce0103b00 0x7fbce019a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:13.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.738+0000 7fbce6984640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbce0104d40 0x7fbce019a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:13.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.738+0000 7fbce6984640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbce0109a40 0x7fbce01a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:13.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.738+0000 7fbce6984640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbce010f600 con 0x7fbce0104d40 2024-04-25T11:33:13.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.738+0000 7fbce6984640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbce010f480 con 0x7fbce0109a40 2024-04-25T11:33:13.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.738+0000 7fbce6984640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbce010f780 con 0x7fbce0103b00 2024-04-25T11:33:13.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.738+0000 7fbcdf7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbce0104d40 0x7fbce019a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:13.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.738+0000 7fbcdf7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbce0104d40 0x7fbce019a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36318/0 (socket says 172.21.15.90:36318) 2024-04-25T11:33:13.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.738+0000 7fbcdf7fe640 1 -- 172.21.15.90:0/3070764273 learned_addr learned my addr 172.21.15.90:0/3070764273 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:13.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.738+0000 7fbce4f21640 1 --2- 172.21.15.90:0/3070764273 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbce0109a40 0x7fbce01a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:13.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.738+0000 7fbcdffff640 1 --2- 172.21.15.90:0/3070764273 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbce0103b00 0x7fbce019a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:13.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.738+0000 7fbcdf7fe640 1 -- 172.21.15.90:0/3070764273 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbce0103b00 msgr2=0x7fbce019a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:13.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.738+0000 7fbcdf7fe640 1 --2- 172.21.15.90:0/3070764273 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbce0103b00 0x7fbce019a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.738+0000 7fbcdf7fe640 1 -- 172.21.15.90:0/3070764273 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbce0109a40 msgr2=0x7fbce01a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:13.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.738+0000 7fbcdf7fe640 1 --2- 172.21.15.90:0/3070764273 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbce0109a40 0x7fbce01a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.738+0000 7fbcdf7fe640 1 -- 172.21.15.90:0/3070764273 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce01a1db0 con 0x7fbce0104d40 2024-04-25T11:33:13.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.738+0000 7fbce4f21640 1 --2- 172.21.15.90:0/3070764273 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbce0109a40 0x7fbce01a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:33:13.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.738+0000 7fbcdf7fe640 1 --2- 172.21.15.90:0/3070764273 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbce0104d40 0x7fbce019a800 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbcd0006010 tx=0x7fbcd0031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:13.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.739+0000 7fbcdd7fa640 1 -- 172.21.15.90:0/3070764273 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd003f030 con 0x7fbce0104d40 2024-04-25T11:33:13.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.739+0000 7fbcdd7fa640 1 -- 172.21.15.90:0/3070764273 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcd0003aa0 con 0x7fbce0104d40 2024-04-25T11:33:13.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.739+0000 7fbce6984640 1 -- 172.21.15.90:0/3070764273 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbce01a2040 con 0x7fbce0104d40 2024-04-25T11:33:13.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.739+0000 7fbcdd7fa640 1 -- 172.21.15.90:0/3070764273 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0035070 con 0x7fbce0104d40 2024-04-25T11:33:13.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.739+0000 7fbce6984640 1 -- 172.21.15.90:0/3070764273 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbce01a2530 con 0x7fbce0104d40 2024-04-25T11:33:13.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.740+0000 7fbce6984640 1 -- 172.21.15.90:0/3070764273 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbce0108850 con 0x7fbce0104d40 2024-04-25T11:33:13.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.741+0000 7fbcdd7fa640 1 -- 172.21.15.90:0/3070764273 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fbcd0042030 con 0x7fbce0104d40 2024-04-25T11:33:13.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.742+0000 7fbcdd7fa640 1 --2- 172.21.15.90:0/3070764273 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbccc0afaf0 0x7fbccc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:13.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.742+0000 7fbcdffff640 1 --2- 172.21.15.90:0/3070764273 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbccc0afaf0 0x7fbccc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:13.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.742+0000 7fbcdd7fa640 1 -- 172.21.15.90:0/3070764273 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7fbcd00ff260 con 0x7fbce0104d40 2024-04-25T11:33:13.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.742+0000 7fbcdffff640 1 --2- 172.21.15.90:0/3070764273 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbccc0afaf0 0x7fbccc0b2000 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbcc80095f0 tx=0x7fbcc8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:13.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.746+0000 7fbcdd7fa640 1 -- 172.21.15.90:0/3070764273 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbcd00c7940 con 0x7fbce0104d40 2024-04-25T11:33:13.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:13.963+0000 7fbce6984640 1 -- 172.21.15.90:0/3070764273 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fbce002db50 con 0x7fbce0104d40 2024-04-25T11:33:14.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:13 smithi090 ceph-mon[29575]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-25T11:33:14.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:13 smithi090 ceph-mon[29575]: Cluster is now healthy 2024-04-25T11:33:14.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:13 smithi090 ceph-mon[29575]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T11:33:14.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:13 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1715083484' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-25T11:33:14.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:13 smithi090 ceph-mon[29575]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T11:33:14.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:13 smithi090 ceph-mon[29575]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T11:33:14.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:13 smithi090 ceph-mon[29575]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T11:33:14.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:13 smithi090 ceph-mon[29575]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T11:33:14.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:13 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1602163597' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T11:33:14.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:13 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1602163597' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T11:33:14.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:13 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] up:active 2024-04-25T11:33:14.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:13 smithi090 ceph-mon[29575]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T11:33:14.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:13 smithi090 ceph-mon[29575]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2024-04-25T11:33:14.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:13 smithi090 ceph-mon[29575]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2024-04-25T11:33:14.227 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:13 smithi090 ceph-mon[29575]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-25T11:33:14.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:13 smithi138 ceph-mon[32328]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-25T11:33:14.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:13 smithi138 ceph-mon[32328]: Cluster is now healthy 2024-04-25T11:33:14.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:13 smithi138 ceph-mon[32328]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T11:33:14.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:13 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1715083484' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-25T11:33:14.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:13 smithi138 ceph-mon[32328]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T11:33:14.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:13 smithi138 ceph-mon[32328]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T11:33:14.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:13 smithi138 ceph-mon[32328]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T11:33:14.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:13 smithi138 ceph-mon[32328]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T11:33:14.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:13 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1602163597' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T11:33:14.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:13 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1602163597' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T11:33:14.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:13 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] up:active 2024-04-25T11:33:14.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:13 smithi138 ceph-mon[32328]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T11:33:14.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:13 smithi138 ceph-mon[32328]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2024-04-25T11:33:14.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:13 smithi138 ceph-mon[32328]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2024-04-25T11:33:14.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:13 smithi138 ceph-mon[32328]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-25T11:33:14.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:13 smithi156 ceph-mon[33045]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-25T11:33:14.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:13 smithi156 ceph-mon[33045]: Cluster is now healthy 2024-04-25T11:33:14.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:13 smithi156 ceph-mon[33045]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T11:33:14.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:13 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1715083484' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-25T11:33:14.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:13 smithi156 ceph-mon[33045]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T11:33:14.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:13 smithi156 ceph-mon[33045]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T11:33:14.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:13 smithi156 ceph-mon[33045]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T11:33:14.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:13 smithi156 ceph-mon[33045]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T11:33:14.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:13 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1602163597' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T11:33:14.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:13 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1602163597' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T11:33:14.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:13 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] up:active 2024-04-25T11:33:14.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:13 smithi156 ceph-mon[33045]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T11:33:14.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:13 smithi156 ceph-mon[33045]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2024-04-25T11:33:14.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:13 smithi156 ceph-mon[33045]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2024-04-25T11:33:14.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:13 smithi156 ceph-mon[33045]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-25T11:33:14.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.613+0000 7fbcdd7fa640 1 -- 172.21.15.90:0/3070764273 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v20) ==== 115+0+0 (secure 0 0 0) 0x7fbcd00cbc40 con 0x7fbce0104d40 2024-04-25T11:33:14.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.616+0000 7fbce6984640 1 -- 172.21.15.90:0/3070764273 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbccc0afaf0 msgr2=0x7fbccc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:14.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.616+0000 7fbce6984640 1 --2- 172.21.15.90:0/3070764273 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbccc0afaf0 0x7fbccc0b2000 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbcc80095f0 tx=0x7fbcc8005f90 comp rx=0 tx=0).stop 2024-04-25T11:33:14.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.616+0000 7fbce6984640 1 -- 172.21.15.90:0/3070764273 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbce0104d40 msgr2=0x7fbce019a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:14.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.616+0000 7fbce6984640 1 --2- 172.21.15.90:0/3070764273 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbce0104d40 0x7fbce019a800 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbcd0006010 tx=0x7fbcd0031a50 comp rx=0 tx=0).stop 2024-04-25T11:33:14.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.616+0000 7fbce6984640 1 -- 172.21.15.90:0/3070764273 shutdown_connections 2024-04-25T11:33:14.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.616+0000 7fbce6984640 1 --2- 172.21.15.90:0/3070764273 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbce0109a40 0x7fbce01a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:14.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.616+0000 7fbce6984640 1 --2- 172.21.15.90:0/3070764273 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbccc0afaf0 0x7fbccc0b2000 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:14.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.616+0000 7fbce6984640 1 --2- 172.21.15.90:0/3070764273 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbce0104d40 0x7fbce019a800 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:14.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.616+0000 7fbce6984640 1 --2- 172.21.15.90:0/3070764273 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbce0103b00 0x7fbce019a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:14.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.616+0000 7fbce6984640 1 -- 172.21.15.90:0/3070764273 >> 172.21.15.90:0/3070764273 conn(0x7fbce00fed10 msgr2=0x7fbce0105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:14.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.616+0000 7fbce6984640 1 -- 172.21.15.90:0/3070764273 shutdown_connections 2024-04-25T11:33:14.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.616+0000 7fbce6984640 1 -- 172.21.15.90:0/3070764273 wait complete. 2024-04-25T11:33:14.618 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-04-25T11:33:14.619 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-25T11:33:14.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.743+0000 7ff34e2b1640 1 -- 172.21.15.90:0/3695110680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3481029e0 msgr2=0x7ff3481036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:14.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.743+0000 7ff34e2b1640 1 --2- 172.21.15.90:0/3695110680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3481029e0 0x7ff3481036e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff330009900 tx=0x7ff33002a410 comp rx=0 tx=0).stop 2024-04-25T11:33:14.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.743+0000 7ff34e2b1640 1 -- 172.21.15.90:0/3695110680 shutdown_connections 2024-04-25T11:33:14.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.743+0000 7ff34e2b1640 1 --2- 172.21.15.90:0/3695110680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff348108920 0x7ff34810ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:14.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.743+0000 7ff34e2b1640 1 --2- 172.21.15.90:0/3695110680 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff348103c20 0x7ff348104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:14.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.743+0000 7ff34e2b1640 1 --2- 172.21.15.90:0/3695110680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3481029e0 0x7ff3481036e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:14.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.743+0000 7ff34e2b1640 1 -- 172.21.15.90:0/3695110680 >> 172.21.15.90:0/3695110680 conn(0x7ff3480fdbd0 msgr2=0x7ff348100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:14.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.743+0000 7ff34e2b1640 1 -- 172.21.15.90:0/3695110680 shutdown_connections 2024-04-25T11:33:14.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.743+0000 7ff34e2b1640 1 -- 172.21.15.90:0/3695110680 wait complete. 2024-04-25T11:33:14.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.743+0000 7ff34e2b1640 1 Processor -- start 2024-04-25T11:33:14.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.743+0000 7ff34e2b1640 1 -- start start 2024-04-25T11:33:14.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.744+0000 7ff34e2b1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3481029e0 0x7ff34819a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:14.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.744+0000 7ff34e2b1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff348103c20 0x7ff34819aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:14.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.744+0000 7ff34e2b1640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff348108920 0x7ff3481a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:14.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.744+0000 7ff34e2b1640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff34810e4c0 con 0x7ff3481029e0 2024-04-25T11:33:14.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.744+0000 7ff34e2b1640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff34810e340 con 0x7ff348108920 2024-04-25T11:33:14.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.744+0000 7ff34e2b1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff34810e640 con 0x7ff348103c20 2024-04-25T11:33:14.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.744+0000 7ff347fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3481029e0 0x7ff34819a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:14.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.744+0000 7ff347fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3481029e0 0x7ff34819a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36334/0 (socket says 172.21.15.90:36334) 2024-04-25T11:33:14.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.744+0000 7ff347fff640 1 -- 172.21.15.90:0/3523789508 learned_addr learned my addr 172.21.15.90:0/3523789508 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:14.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.744+0000 7ff34c84e640 1 --2- 172.21.15.90:0/3523789508 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff348108920 0x7ff3481a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:14.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.744+0000 7ff347fff640 1 -- 172.21.15.90:0/3523789508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff348103c20 msgr2=0x7ff34819aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:14.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.744+0000 7ff3477fe640 1 --2- 172.21.15.90:0/3523789508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff348103c20 0x7ff34819aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:14.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.744+0000 7ff347fff640 1 --2- 172.21.15.90:0/3523789508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff348103c20 0x7ff34819aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:14.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.744+0000 7ff347fff640 1 -- 172.21.15.90:0/3523789508 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff348108920 msgr2=0x7ff3481a1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:14.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.744+0000 7ff347fff640 1 --2- 172.21.15.90:0/3523789508 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff348108920 0x7ff3481a1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:14.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.744+0000 7ff347fff640 1 -- 172.21.15.90:0/3523789508 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3481a20c0 con 0x7ff3481029e0 2024-04-25T11:33:14.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.744+0000 7ff34c84e640 1 --2- 172.21.15.90:0/3523789508 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff348108920 0x7ff3481a1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:33:14.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.744+0000 7ff3477fe640 1 --2- 172.21.15.90:0/3523789508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff348103c20 0x7ff34819aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:33:14.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.744+0000 7ff347fff640 1 --2- 172.21.15.90:0/3523789508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3481029e0 0x7ff34819a580 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff330006010 tx=0x7ff330004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:14.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.745+0000 7ff3457fa640 1 -- 172.21.15.90:0/3523789508 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff33003f030 con 0x7ff3481029e0 2024-04-25T11:33:14.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.745+0000 7ff3457fa640 1 -- 172.21.15.90:0/3523789508 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff330003aa0 con 0x7ff3481029e0 2024-04-25T11:33:14.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.745+0000 7ff34e2b1640 1 -- 172.21.15.90:0/3523789508 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3481a2350 con 0x7ff3481029e0 2024-04-25T11:33:14.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.745+0000 7ff3457fa640 1 -- 172.21.15.90:0/3523789508 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff330035070 con 0x7ff3481029e0 2024-04-25T11:33:14.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.745+0000 7ff34e2b1640 1 -- 172.21.15.90:0/3523789508 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3481a2840 con 0x7ff3481029e0 2024-04-25T11:33:14.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.746+0000 7ff34e2b1640 1 -- 172.21.15.90:0/3523789508 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff34806c2a0 con 0x7ff3481029e0 2024-04-25T11:33:14.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.748+0000 7ff3457fa640 1 -- 172.21.15.90:0/3523789508 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7ff330042030 con 0x7ff3481029e0 2024-04-25T11:33:14.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.748+0000 7ff3457fa640 1 --2- 172.21.15.90:0/3523789508 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7ff3340afaf0 0x7ff3340b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:14.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.748+0000 7ff3477fe640 1 --2- 172.21.15.90:0/3523789508 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7ff3340afaf0 0x7ff3340b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:14.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.748+0000 7ff3457fa640 1 -- 172.21.15.90:0/3523789508 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7ff3300ff630 con 0x7ff3481029e0 2024-04-25T11:33:14.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.749+0000 7ff3477fe640 1 --2- 172.21.15.90:0/3523789508 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7ff3340afaf0 0x7ff3340b2000 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff34819b620 tx=0x7ff338005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:14.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.752+0000 7ff3457fa640 1 -- 172.21.15.90:0/3523789508 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff3300c8360 con 0x7ff3481029e0 2024-04-25T11:33:14.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:14.958+0000 7ff34e2b1640 1 -- 172.21.15.90:0/3523789508 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ff3481a2aa0 con 0x7ff3340afaf0 2024-04-25T11:33:14.952 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:14 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3070764273' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T11:33:14.952 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:14 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3070764273' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T11:33:14.952 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:14 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] up:active 2024-04-25T11:33:14.952 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:14 smithi090 ceph-mon[29575]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-25T11:33:14.953 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:14 smithi090 ceph-mon[29575]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T11:33:14.953 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:14 smithi090 ceph-mon[29575]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:creating} 2 up:standby-replay 6 up:standby 2024-04-25T11:33:14.953 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:14 smithi090 ceph-mon[29575]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-04-25T11:33:14.953 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:14 smithi090 ceph-mon[29575]: pgmap v276: 97 pgs: 4 creating+peering, 85 active+clean, 8 unknown; 580 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 3.5 KiB/s wr, 15 op/s 2024-04-25T11:33:15.226 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:33:14 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:33:14.963+0000 7f839660d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T11:33:15.226 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:33:14 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:33:14.963+0000 7f839660d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T11:33:15.226 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:33:14 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:33:14.963+0000 7f839660d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T11:33:15.226 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:33:14 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:33:14.963+0000 7f839660d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T11:33:15.226 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:33:14 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:33:14.963+0000 7f839660d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T11:33:15.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:14 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3070764273' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T11:33:15.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:14 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3070764273' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T11:33:15.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:14 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] up:active 2024-04-25T11:33:15.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:14 smithi138 ceph-mon[32328]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-25T11:33:15.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:14 smithi138 ceph-mon[32328]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T11:33:15.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:14 smithi138 ceph-mon[32328]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:creating} 2 up:standby-replay 6 up:standby 2024-04-25T11:33:15.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:14 smithi138 ceph-mon[32328]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-04-25T11:33:15.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:14 smithi138 ceph-mon[32328]: pgmap v276: 97 pgs: 4 creating+peering, 85 active+clean, 8 unknown; 580 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 3.5 KiB/s wr, 15 op/s 2024-04-25T11:33:15.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:14 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3070764273' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T11:33:15.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:14 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3070764273' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T11:33:15.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:14 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] up:active 2024-04-25T11:33:15.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:14 smithi156 ceph-mon[33045]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-25T11:33:15.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:14 smithi156 ceph-mon[33045]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T11:33:15.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:14 smithi156 ceph-mon[33045]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:creating} 2 up:standby-replay 6 up:standby 2024-04-25T11:33:15.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:14 smithi156 ceph-mon[33045]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-04-25T11:33:15.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:14 smithi156 ceph-mon[33045]: pgmap v276: 97 pgs: 4 creating+peering, 85 active+clean, 8 unknown; 580 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 3.5 KiB/s wr, 15 op/s 2024-04-25T11:33:15.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.641+0000 7ff3457fa640 1 -- 172.21.15.90:0/3523789508 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff3481a2aa0 con 0x7ff3340afaf0 2024-04-25T11:33:15.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.645+0000 7ff34e2b1640 1 -- 172.21.15.90:0/3523789508 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7ff3340afaf0 msgr2=0x7ff3340b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:15.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.645+0000 7ff34e2b1640 1 --2- 172.21.15.90:0/3523789508 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7ff3340afaf0 0x7ff3340b2000 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff34819b620 tx=0x7ff338005f90 comp rx=0 tx=0).stop 2024-04-25T11:33:15.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.645+0000 7ff34e2b1640 1 -- 172.21.15.90:0/3523789508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3481029e0 msgr2=0x7ff34819a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:15.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.645+0000 7ff34e2b1640 1 --2- 172.21.15.90:0/3523789508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3481029e0 0x7ff34819a580 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff330006010 tx=0x7ff330004270 comp rx=0 tx=0).stop 2024-04-25T11:33:15.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.646+0000 7ff34e2b1640 1 -- 172.21.15.90:0/3523789508 shutdown_connections 2024-04-25T11:33:15.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.646+0000 7ff34e2b1640 1 --2- 172.21.15.90:0/3523789508 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff348108920 0x7ff3481a1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:15.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.646+0000 7ff34e2b1640 1 --2- 172.21.15.90:0/3523789508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff348103c20 0x7ff34819aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:15.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.646+0000 7ff34e2b1640 1 --2- 172.21.15.90:0/3523789508 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7ff3340afaf0 0x7ff3340b2000 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:15.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.646+0000 7ff34e2b1640 1 --2- 172.21.15.90:0/3523789508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3481029e0 0x7ff34819a580 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:15.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.646+0000 7ff34e2b1640 1 -- 172.21.15.90:0/3523789508 >> 172.21.15.90:0/3523789508 conn(0x7ff3480fdbd0 msgr2=0x7ff348109e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:15.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.646+0000 7ff34e2b1640 1 -- 172.21.15.90:0/3523789508 shutdown_connections 2024-04-25T11:33:15.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.646+0000 7ff34e2b1640 1 -- 172.21.15.90:0/3523789508 wait complete. 2024-04-25T11:33:15.649 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-25T11:33:15.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.778+0000 7f67bd0e7640 1 -- 172.21.15.90:0/2795409180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67b81050b0 msgr2=0x7f67b81054c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:15.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.778+0000 7f67bd0e7640 1 --2- 172.21.15.90:0/2795409180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67b81050b0 0x7f67b81054c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f67a8009900 tx=0x7f67a802a410 comp rx=0 tx=0).stop 2024-04-25T11:33:15.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.778+0000 7f67bd0e7640 1 -- 172.21.15.90:0/2795409180 shutdown_connections 2024-04-25T11:33:15.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.778+0000 7f67bd0e7640 1 --2- 172.21.15.90:0/2795409180 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f67b8102260 0x7f67b810bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:15.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.778+0000 7f67bd0e7640 1 --2- 172.21.15.90:0/2795409180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67b8101760 0x7f67b8101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:15.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.778+0000 7f67bd0e7640 1 --2- 172.21.15.90:0/2795409180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67b81050b0 0x7f67b81054c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:15.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.778+0000 7f67bd0e7640 1 -- 172.21.15.90:0/2795409180 >> 172.21.15.90:0/2795409180 conn(0x7f67b80fd380 msgr2=0x7f67b80ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:15.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.778+0000 7f67bd0e7640 1 -- 172.21.15.90:0/2795409180 shutdown_connections 2024-04-25T11:33:15.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.778+0000 7f67bd0e7640 1 -- 172.21.15.90:0/2795409180 wait complete. 2024-04-25T11:33:15.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.779+0000 7f67bd0e7640 1 Processor -- start 2024-04-25T11:33:15.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.779+0000 7f67bd0e7640 1 -- start start 2024-04-25T11:33:15.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.779+0000 7f67bd0e7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67b8101760 0x7f67b819a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:15.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.779+0000 7f67bd0e7640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67b8102260 0x7f67b819a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:15.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.779+0000 7f67bd0e7640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f67b81050b0 0x7f67b81a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:15.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.779+0000 7f67bd0e7640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f67b810f600 con 0x7f67b8102260 2024-04-25T11:33:15.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.779+0000 7f67bd0e7640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f67b810f480 con 0x7f67b81050b0 2024-04-25T11:33:15.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.779+0000 7f67bd0e7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f67b810f780 con 0x7f67b8101760 2024-04-25T11:33:15.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.779+0000 7f67b659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67b8102260 0x7f67b819a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:15.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.779+0000 7f67b659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67b8102260 0x7f67b819a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36374/0 (socket says 172.21.15.90:36374) 2024-04-25T11:33:15.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.779+0000 7f67b659c640 1 -- 172.21.15.90:0/66933804 learned_addr learned my addr 172.21.15.90:0/66933804 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:15.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.779+0000 7f67b759e640 1 --2- 172.21.15.90:0/66933804 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f67b81050b0 0x7f67b81a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:15.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.779+0000 7f67b6d9d640 1 --2- 172.21.15.90:0/66933804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67b8101760 0x7f67b819a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:15.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.780+0000 7f67b659c640 1 -- 172.21.15.90:0/66933804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67b8101760 msgr2=0x7f67b819a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:15.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.780+0000 7f67b659c640 1 --2- 172.21.15.90:0/66933804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67b8101760 0x7f67b819a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:15.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.780+0000 7f67b659c640 1 -- 172.21.15.90:0/66933804 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f67b81050b0 msgr2=0x7f67b81a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:15.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.780+0000 7f67b659c640 1 --2- 172.21.15.90:0/66933804 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f67b81050b0 0x7f67b81a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:15.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.780+0000 7f67b659c640 1 -- 172.21.15.90:0/66933804 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67b81a1d90 con 0x7f67b8102260 2024-04-25T11:33:15.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.780+0000 7f67b6d9d640 1 --2- 172.21.15.90:0/66933804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67b8101760 0x7f67b819a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:33:15.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.780+0000 7f67b659c640 1 --2- 172.21.15.90:0/66933804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67b8102260 0x7f67b819a790 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f67a000c890 tx=0x7f67a000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:15.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.780+0000 7f67affff640 1 -- 172.21.15.90:0/66933804 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a0004220 con 0x7f67b8102260 2024-04-25T11:33:15.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.780+0000 7f67affff640 1 -- 172.21.15.90:0/66933804 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67a00043c0 con 0x7f67b8102260 2024-04-25T11:33:15.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.780+0000 7f67bd0e7640 1 -- 172.21.15.90:0/66933804 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67b81a2080 con 0x7f67b8102260 2024-04-25T11:33:15.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.780+0000 7f67affff640 1 -- 172.21.15.90:0/66933804 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a000f450 con 0x7f67b8102260 2024-04-25T11:33:15.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.780+0000 7f67bd0e7640 1 -- 172.21.15.90:0/66933804 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67b81a25d0 con 0x7f67b8102260 2024-04-25T11:33:15.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.781+0000 7f67bd0e7640 1 -- 172.21.15.90:0/66933804 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67b8108a40 con 0x7f67b8102260 2024-04-25T11:33:15.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.782+0000 7f67affff640 1 -- 172.21.15.90:0/66933804 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f67a00035f0 con 0x7f67b8102260 2024-04-25T11:33:15.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.783+0000 7f67affff640 1 --2- 172.21.15.90:0/66933804 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f67a40afaf0 0x7f67a40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:15.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.783+0000 7f67b6d9d640 1 --2- 172.21.15.90:0/66933804 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f67a40afaf0 0x7f67a40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:15.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.783+0000 7f67affff640 1 -- 172.21.15.90:0/66933804 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7f67a00dec00 con 0x7f67b8102260 2024-04-25T11:33:15.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.784+0000 7f67b6d9d640 1 --2- 172.21.15.90:0/66933804 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f67a40afaf0 0x7f67a40b2000 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f67a8009230 tx=0x7f67a8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:15.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.787+0000 7f67affff640 1 -- 172.21.15.90:0/66933804 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f67a0014030 con 0x7f67b8102260 2024-04-25T11:33:15.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:15.993+0000 7f67bd0e7640 1 -- 172.21.15.90:0/66933804 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f67b8062fe0 con 0x7f67a40afaf0 2024-04-25T11:33:16.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.020+0000 7f67affff640 1 -- 172.21.15.90:0/66933804 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f67b8062fe0 con 0x7f67a40afaf0 2024-04-25T11:33:16.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.022+0000 7f67bd0e7640 1 -- 172.21.15.90:0/66933804 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f67a40afaf0 msgr2=0x7f67a40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:16.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.022+0000 7f67bd0e7640 1 --2- 172.21.15.90:0/66933804 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f67a40afaf0 0x7f67a40b2000 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f67a8009230 tx=0x7f67a8000bc0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.022+0000 7f67bd0e7640 1 -- 172.21.15.90:0/66933804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67b8102260 msgr2=0x7f67b819a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:16.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.022+0000 7f67bd0e7640 1 --2- 172.21.15.90:0/66933804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67b8102260 0x7f67b819a790 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f67a000c890 tx=0x7f67a000cd50 comp rx=0 tx=0).stop 2024-04-25T11:33:16.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.022+0000 7f67bd0e7640 1 -- 172.21.15.90:0/66933804 shutdown_connections 2024-04-25T11:33:16.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.022+0000 7f67bd0e7640 1 --2- 172.21.15.90:0/66933804 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f67b81050b0 0x7f67b81a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.022+0000 7f67bd0e7640 1 --2- 172.21.15.90:0/66933804 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f67a40afaf0 0x7f67a40b2000 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.022+0000 7f67bd0e7640 1 --2- 172.21.15.90:0/66933804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67b8102260 0x7f67b819a790 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.022+0000 7f67bd0e7640 1 --2- 172.21.15.90:0/66933804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67b8101760 0x7f67b819a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.022+0000 7f67bd0e7640 1 -- 172.21.15.90:0/66933804 >> 172.21.15.90:0/66933804 conn(0x7f67b80fd380 msgr2=0x7f67b8105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:16.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.022+0000 7f67bd0e7640 1 -- 172.21.15.90:0/66933804 shutdown_connections 2024-04-25T11:33:16.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.022+0000 7f67bd0e7640 1 -- 172.21.15.90:0/66933804 wait complete. 2024-04-25T11:33:16.022 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-25T11:33:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.145+0000 7f6ca8b7d640 1 -- 172.21.15.90:0/1260435581 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ca4103c20 msgr2=0x7f6ca4104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.145+0000 7f6ca8b7d640 1 --2- 172.21.15.90:0/1260435581 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ca4103c20 0x7f6ca4104030 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c009900 tx=0x7f6c8c02a410 comp rx=0 tx=0).stop 2024-04-25T11:33:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.145+0000 7f6ca8b7d640 1 -- 172.21.15.90:0/1260435581 shutdown_connections 2024-04-25T11:33:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.145+0000 7f6ca8b7d640 1 --2- 172.21.15.90:0/1260435581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca4108920 0x7f6ca410ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.145+0000 7f6ca8b7d640 1 --2- 172.21.15.90:0/1260435581 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ca4103c20 0x7f6ca4104030 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.145+0000 7f6ca8b7d640 1 --2- 172.21.15.90:0/1260435581 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6ca41029e0 0x7f6ca41036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.145+0000 7f6ca8b7d640 1 -- 172.21.15.90:0/1260435581 >> 172.21.15.90:0/1260435581 conn(0x7f6ca40fdbd0 msgr2=0x7f6ca4100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.145+0000 7f6ca8b7d640 1 -- 172.21.15.90:0/1260435581 shutdown_connections 2024-04-25T11:33:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.145+0000 7f6ca8b7d640 1 -- 172.21.15.90:0/1260435581 wait complete. 2024-04-25T11:33:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.145+0000 7f6ca8b7d640 1 Processor -- start 2024-04-25T11:33:16.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.146+0000 7f6ca8b7d640 1 -- start start 2024-04-25T11:33:16.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.146+0000 7f6ca8b7d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ca41029e0 0x7f6ca4199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:16.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.146+0000 7f6ca8b7d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca4103c20 0x7f6ca4199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:16.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.146+0000 7f6ca8b7d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6ca4108920 0x7f6ca41a0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:16.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.146+0000 7f6ca8b7d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca410e4e0 con 0x7f6ca41029e0 2024-04-25T11:33:16.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.146+0000 7f6ca8b7d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca410e360 con 0x7f6ca4108920 2024-04-25T11:33:16.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.146+0000 7f6ca8b7d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca410e660 con 0x7f6ca4103c20 2024-04-25T11:33:16.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.146+0000 7f6ca259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ca41029e0 0x7f6ca4199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:16.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.146+0000 7f6ca259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ca41029e0 0x7f6ca4199430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36398/0 (socket says 172.21.15.90:36398) 2024-04-25T11:33:16.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.146+0000 7f6ca259c640 1 -- 172.21.15.90:0/2143943836 learned_addr learned my addr 172.21.15.90:0/2143943836 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:16.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.146+0000 7f6ca2d9d640 1 --2- 172.21.15.90:0/2143943836 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6ca4108920 0x7f6ca41a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:16.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.146+0000 7f6c9bfff640 1 --2- 172.21.15.90:0/2143943836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca4103c20 0x7f6ca4199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.146+0000 7f6ca259c640 1 -- 172.21.15.90:0/2143943836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca4103c20 msgr2=0x7f6ca4199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.146+0000 7f6ca259c640 1 --2- 172.21.15.90:0/2143943836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca4103c20 0x7f6ca4199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.146+0000 7f6ca259c640 1 -- 172.21.15.90:0/2143943836 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6ca4108920 msgr2=0x7f6ca41a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.146+0000 7f6ca259c640 1 --2- 172.21.15.90:0/2143943836 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6ca4108920 0x7f6ca41a0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.146+0000 7f6ca259c640 1 -- 172.21.15.90:0/2143943836 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ca41a0ec0 con 0x7f6ca41029e0 2024-04-25T11:33:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.146+0000 7f6ca2d9d640 1 --2- 172.21.15.90:0/2143943836 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6ca4108920 0x7f6ca41a0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:33:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.146+0000 7f6ca259c640 1 --2- 172.21.15.90:0/2143943836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ca41029e0 0x7f6ca4199430 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6c9400e9f0 tx=0x7f6c9400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.146+0000 7f6c9b7fe640 1 -- 172.21.15.90:0/2143943836 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c9400ccc0 con 0x7f6ca41029e0 2024-04-25T11:33:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.146+0000 7f6c9b7fe640 1 -- 172.21.15.90:0/2143943836 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c940035f0 con 0x7f6ca41029e0 2024-04-25T11:33:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.146+0000 7f6ca8b7d640 1 -- 172.21.15.90:0/2143943836 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ca41a11b0 con 0x7f6ca41029e0 2024-04-25T11:33:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.146+0000 7f6c9b7fe640 1 -- 172.21.15.90:0/2143943836 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c9400ce60 con 0x7f6ca41029e0 2024-04-25T11:33:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.146+0000 7f6ca8b7d640 1 -- 172.21.15.90:0/2143943836 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ca41a1700 con 0x7f6ca41029e0 2024-04-25T11:33:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.147+0000 7f6ca8b7d640 1 -- 172.21.15.90:0/2143943836 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ca4107730 con 0x7f6ca41029e0 2024-04-25T11:33:16.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.149+0000 7f6c9b7fe640 1 -- 172.21.15.90:0/2143943836 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f6c94003aa0 con 0x7f6ca41029e0 2024-04-25T11:33:16.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.149+0000 7f6c9b7fe640 1 --2- 172.21.15.90:0/2143943836 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6c900afaf0 0x7f6c900b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:16.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.150+0000 7f6c9bfff640 1 --2- 172.21.15.90:0/2143943836 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6c900afaf0 0x7f6c900b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:16.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.150+0000 7f6c9b7fe640 1 -- 172.21.15.90:0/2143943836 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7f6c940defd0 con 0x7f6ca41029e0 2024-04-25T11:33:16.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.150+0000 7f6c9bfff640 1 --2- 172.21.15.90:0/2143943836 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6c900afaf0 0x7f6c900b2000 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c000c00 tx=0x7f6c8c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:16.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.153+0000 7f6c9b7fe640 1 -- 172.21.15.90:0/2143943836 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6c940a7d00 con 0x7f6ca41029e0 2024-04-25T11:33:16.302 INFO:journalctl@ceph.mds.b.smithi138.stdout:Apr 25 11:33:15 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-b[62562]: 2024-04-25T11:33:15.836+0000 7fa9dc67e700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T11:33:16.303 INFO:journalctl@ceph.mds.b.smithi138.stdout:Apr 25 11:33:15 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-b[62562]: 2024-04-25T11:33:15.836+0000 7fa9dc67e700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-25T11:33:16.303 INFO:journalctl@ceph.mds.b.smithi138.stdout:Apr 25 11:33:15 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-b[62562]: 2024-04-25T11:33:15.836+0000 7fa9dc67e700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T11:33:16.303 INFO:journalctl@ceph.mds.b.smithi138.stdout:Apr 25 11:33:15 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-b[62562]: 2024-04-25T11:33:15.836+0000 7fa9dc67e700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-25T11:33:16.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.358+0000 7f6ca8b7d640 1 -- 172.21.15.90:0/2143943836 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f6ca402de30 con 0x7f6c900afaf0 2024-04-25T11:33:16.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.363+0000 7f6c9b7fe640 1 -- 172.21.15.90:0/2143943836 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f6ca402de30 con 0x7f6c900afaf0 2024-04-25T11:33:16.351 INFO:teuthology.orchestra.run.smithi090.stdout:/volumes/qa/sv_0/1b2771e1-84ac-4a99-a5b4-e5ad03c1e04f 2024-04-25T11:33:16.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.365+0000 7f6ca8b7d640 1 -- 172.21.15.90:0/2143943836 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6c900afaf0 msgr2=0x7f6c900b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:16.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.365+0000 7f6ca8b7d640 1 --2- 172.21.15.90:0/2143943836 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6c900afaf0 0x7f6c900b2000 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c000c00 tx=0x7f6c8c031400 comp rx=0 tx=0).stop 2024-04-25T11:33:16.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.365+0000 7f6ca8b7d640 1 -- 172.21.15.90:0/2143943836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ca41029e0 msgr2=0x7f6ca4199430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:16.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.365+0000 7f6ca8b7d640 1 --2- 172.21.15.90:0/2143943836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ca41029e0 0x7f6ca4199430 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6c9400e9f0 tx=0x7f6c9400eeb0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.365+0000 7f6ca8b7d640 1 -- 172.21.15.90:0/2143943836 shutdown_connections 2024-04-25T11:33:16.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.365+0000 7f6ca8b7d640 1 --2- 172.21.15.90:0/2143943836 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6ca4108920 0x7f6ca41a0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.365+0000 7f6ca8b7d640 1 --2- 172.21.15.90:0/2143943836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca4103c20 0x7f6ca4199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.365+0000 7f6ca8b7d640 1 --2- 172.21.15.90:0/2143943836 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6c900afaf0 0x7f6c900b2000 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.365+0000 7f6ca8b7d640 1 --2- 172.21.15.90:0/2143943836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ca41029e0 0x7f6ca4199430 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.365+0000 7f6ca8b7d640 1 -- 172.21.15.90:0/2143943836 >> 172.21.15.90:0/2143943836 conn(0x7f6ca40fdbd0 msgr2=0x7f6ca4104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:16.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.365+0000 7f6ca8b7d640 1 -- 172.21.15.90:0/2143943836 shutdown_connections 2024-04-25T11:33:16.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.365+0000 7f6ca8b7d640 1 -- 172.21.15.90:0/2143943836 wait complete. 2024-04-25T11:33:16.365 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-25T11:33:16.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.489+0000 7fd8f1ed1640 1 -- 172.21.15.90:0/2871780654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8ec103af0 msgr2=0x7fd8ec1047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:16.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.489+0000 7fd8f1ed1640 1 --2- 172.21.15.90:0/2871780654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8ec103af0 0x7fd8ec1047f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc009900 tx=0x7fd8dc02a410 comp rx=0 tx=0).stop 2024-04-25T11:33:16.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.489+0000 7fd8f1ed1640 1 -- 172.21.15.90:0/2871780654 shutdown_connections 2024-04-25T11:33:16.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.489+0000 7fd8f1ed1640 1 --2- 172.21.15.90:0/2871780654 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8ec109a30 0x7fd8ec10be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.489+0000 7fd8f1ed1640 1 --2- 172.21.15.90:0/2871780654 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd8ec104d30 0x7fd8ec105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.489+0000 7fd8f1ed1640 1 --2- 172.21.15.90:0/2871780654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8ec103af0 0x7fd8ec1047f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.489+0000 7fd8f1ed1640 1 -- 172.21.15.90:0/2871780654 >> 172.21.15.90:0/2871780654 conn(0x7fd8ec0fece0 msgr2=0x7fd8ec101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:16.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.489+0000 7fd8f1ed1640 1 -- 172.21.15.90:0/2871780654 shutdown_connections 2024-04-25T11:33:16.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.489+0000 7fd8f1ed1640 1 -- 172.21.15.90:0/2871780654 wait complete. 2024-04-25T11:33:16.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.489+0000 7fd8f1ed1640 1 Processor -- start 2024-04-25T11:33:16.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.489+0000 7fd8f1ed1640 1 -- start start 2024-04-25T11:33:16.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.490+0000 7fd8f1ed1640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd8ec103af0 0x7fd8ec19a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:16.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.490+0000 7fd8f1ed1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8ec104d30 0x7fd8ec19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:16.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.490+0000 7fd8f1ed1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8ec109a30 0x7fd8ec1a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:16.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.490+0000 7fd8f1ed1640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ec10f5f0 con 0x7fd8ec104d30 2024-04-25T11:33:16.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.490+0000 7fd8f1ed1640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ec10f470 con 0x7fd8ec103af0 2024-04-25T11:33:16.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.490+0000 7fd8f1ed1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ec10f770 con 0x7fd8ec109a30 2024-04-25T11:33:16.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.490+0000 7fd8eaffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8ec104d30 0x7fd8ec19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:16.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.490+0000 7fd8eaffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8ec104d30 0x7fd8ec19a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36412/0 (socket says 172.21.15.90:36412) 2024-04-25T11:33:16.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.490+0000 7fd8eaffd640 1 -- 172.21.15.90:0/2626704702 learned_addr learned my addr 172.21.15.90:0/2626704702 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:16.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.490+0000 7fd8eb7fe640 1 --2- 172.21.15.90:0/2626704702 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd8ec103af0 0x7fd8ec19a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:16.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.490+0000 7fd8ebfff640 1 --2- 172.21.15.90:0/2626704702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8ec109a30 0x7fd8ec1a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:16.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.490+0000 7fd8eaffd640 1 -- 172.21.15.90:0/2626704702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8ec109a30 msgr2=0x7fd8ec1a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:16.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.490+0000 7fd8eaffd640 1 --2- 172.21.15.90:0/2626704702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8ec109a30 0x7fd8ec1a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.490+0000 7fd8eaffd640 1 -- 172.21.15.90:0/2626704702 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd8ec103af0 msgr2=0x7fd8ec19a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:16.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.490+0000 7fd8eaffd640 1 --2- 172.21.15.90:0/2626704702 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd8ec103af0 0x7fd8ec19a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.490+0000 7fd8eaffd640 1 -- 172.21.15.90:0/2626704702 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8ec1a1d90 con 0x7fd8ec104d30 2024-04-25T11:33:16.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.490+0000 7fd8ebfff640 1 --2- 172.21.15.90:0/2626704702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8ec109a30 0x7fd8ec1a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:33:16.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.490+0000 7fd8eaffd640 1 --2- 172.21.15.90:0/2626704702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8ec104d30 0x7fd8ec19a790 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000e9f0 tx=0x7fd8e000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:16.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.490+0000 7fd8e8ff9640 1 -- 172.21.15.90:0/2626704702 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e000cd80 con 0x7fd8ec104d30 2024-04-25T11:33:16.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.490+0000 7fd8f1ed1640 1 -- 172.21.15.90:0/2626704702 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8ec1a2080 con 0x7fd8ec104d30 2024-04-25T11:33:16.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.490+0000 7fd8e8ff9640 1 -- 172.21.15.90:0/2626704702 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8e00108e0 con 0x7fd8ec104d30 2024-04-25T11:33:16.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.490+0000 7fd8e8ff9640 1 -- 172.21.15.90:0/2626704702 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0018980 con 0x7fd8ec104d30 2024-04-25T11:33:16.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.490+0000 7fd8f1ed1640 1 -- 172.21.15.90:0/2626704702 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8ec1a2620 con 0x7fd8ec104d30 2024-04-25T11:33:16.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.491+0000 7fd8f1ed1640 1 -- 172.21.15.90:0/2626704702 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8ec06a160 con 0x7fd8ec104d30 2024-04-25T11:33:16.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.493+0000 7fd8e8ff9640 1 -- 172.21.15.90:0/2626704702 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fd8e0010430 con 0x7fd8ec104d30 2024-04-25T11:33:16.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.494+0000 7fd8e8ff9640 1 --2- 172.21.15.90:0/2626704702 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd8d40afaf0 0x7fd8d40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:16.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.494+0000 7fd8eb7fe640 1 --2- 172.21.15.90:0/2626704702 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd8d40afaf0 0x7fd8d40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:16.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.494+0000 7fd8e8ff9640 1 -- 172.21.15.90:0/2626704702 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7fd8e00df310 con 0x7fd8ec104d30 2024-04-25T11:33:16.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.494+0000 7fd8eb7fe640 1 --2- 172.21.15.90:0/2626704702 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd8d40afaf0 0x7fd8d40b2000 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc005ab0 tx=0x7fd8dc031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:16.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.497+0000 7fd8e8ff9640 1 -- 172.21.15.90:0/2626704702 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd8e00a8040 con 0x7fd8ec104d30 2024-04-25T11:33:16.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.709+0000 7fd8f1ed1640 1 -- 172.21.15.90:0/2626704702 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fd8ec062fe0 con 0x7fd8d40afaf0 2024-04-25T11:33:16.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.742+0000 7fd8e8ff9640 1 -- 172.21.15.90:0/2626704702 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd8ec062fe0 con 0x7fd8d40afaf0 2024-04-25T11:33:16.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.744+0000 7fd8f1ed1640 1 -- 172.21.15.90:0/2626704702 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd8d40afaf0 msgr2=0x7fd8d40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:16.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.744+0000 7fd8f1ed1640 1 --2- 172.21.15.90:0/2626704702 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd8d40afaf0 0x7fd8d40b2000 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc005ab0 tx=0x7fd8dc031400 comp rx=0 tx=0).stop 2024-04-25T11:33:16.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.744+0000 7fd8f1ed1640 1 -- 172.21.15.90:0/2626704702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8ec104d30 msgr2=0x7fd8ec19a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:16.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.744+0000 7fd8f1ed1640 1 --2- 172.21.15.90:0/2626704702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8ec104d30 0x7fd8ec19a790 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000e9f0 tx=0x7fd8e000eeb0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.744+0000 7fd8f1ed1640 1 -- 172.21.15.90:0/2626704702 shutdown_connections 2024-04-25T11:33:16.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.744+0000 7fd8f1ed1640 1 --2- 172.21.15.90:0/2626704702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8ec109a30 0x7fd8ec1a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.744+0000 7fd8f1ed1640 1 --2- 172.21.15.90:0/2626704702 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd8d40afaf0 0x7fd8d40b2000 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.744+0000 7fd8f1ed1640 1 --2- 172.21.15.90:0/2626704702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8ec104d30 0x7fd8ec19a790 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.744+0000 7fd8f1ed1640 1 --2- 172.21.15.90:0/2626704702 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd8ec103af0 0x7fd8ec19a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.744+0000 7fd8f1ed1640 1 -- 172.21.15.90:0/2626704702 >> 172.21.15.90:0/2626704702 conn(0x7fd8ec0fece0 msgr2=0x7fd8ec105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.744+0000 7fd8f1ed1640 1 -- 172.21.15.90:0/2626704702 shutdown_connections 2024-04-25T11:33:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.744+0000 7fd8f1ed1640 1 -- 172.21.15.90:0/2626704702 wait complete. 2024-04-25T11:33:16.744 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-25T11:33:16.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.868+0000 7fa6c257d640 1 -- 172.21.15.90:0/812854838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6bc108940 msgr2=0x7fa6bc10ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:16.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.868+0000 7fa6c257d640 1 --2- 172.21.15.90:0/812854838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6bc108940 0x7fa6bc10ad60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4009ca0 tx=0x7fa6b402a8c0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.868+0000 7fa6c257d640 1 -- 172.21.15.90:0/812854838 shutdown_connections 2024-04-25T11:33:16.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.868+0000 7fa6c257d640 1 --2- 172.21.15.90:0/812854838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6bc108940 0x7fa6bc10ad60 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.868+0000 7fa6c257d640 1 --2- 172.21.15.90:0/812854838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6bc103c40 0x7fa6bc104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.868+0000 7fa6c257d640 1 --2- 172.21.15.90:0/812854838 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa6bc102a00 0x7fa6bc103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.868+0000 7fa6c257d640 1 -- 172.21.15.90:0/812854838 >> 172.21.15.90:0/812854838 conn(0x7fa6bc0fdbf0 msgr2=0x7fa6bc1000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:16.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.868+0000 7fa6c257d640 1 -- 172.21.15.90:0/812854838 shutdown_connections 2024-04-25T11:33:16.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.868+0000 7fa6c257d640 1 -- 172.21.15.90:0/812854838 wait complete. 2024-04-25T11:33:16.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.869+0000 7fa6c257d640 1 Processor -- start 2024-04-25T11:33:16.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.869+0000 7fa6c257d640 1 -- start start 2024-04-25T11:33:16.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.869+0000 7fa6c257d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6bc102a00 0x7fa6bc075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:16.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.869+0000 7fa6c257d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6bc103c40 0x7fa6bc19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:16.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.869+0000 7fa6c257d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa6bc108940 0x7fa6bc1a1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:16.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.869+0000 7fa6c257d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa6bc10e510 con 0x7fa6bc103c40 2024-04-25T11:33:16.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.869+0000 7fa6c257d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa6bc10e390 con 0x7fa6bc108940 2024-04-25T11:33:16.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.869+0000 7fa6c257d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa6bc10e690 con 0x7fa6bc102a00 2024-04-25T11:33:16.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.869+0000 7fa6bb7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6bc103c40 0x7fa6bc19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:16.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.869+0000 7fa6bb7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6bc103c40 0x7fa6bc19ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36426/0 (socket says 172.21.15.90:36426) 2024-04-25T11:33:16.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.869+0000 7fa6bb7fe640 1 -- 172.21.15.90:0/448240655 learned_addr learned my addr 172.21.15.90:0/448240655 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:16.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.869+0000 7fa6c0b1a640 1 --2- 172.21.15.90:0/448240655 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa6bc108940 0x7fa6bc1a1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:16.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.869+0000 7fa6bbfff640 1 --2- 172.21.15.90:0/448240655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6bc102a00 0x7fa6bc075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:16.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.869+0000 7fa6bb7fe640 1 -- 172.21.15.90:0/448240655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6bc102a00 msgr2=0x7fa6bc075840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:16.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.869+0000 7fa6bb7fe640 1 --2- 172.21.15.90:0/448240655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6bc102a00 0x7fa6bc075840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.869+0000 7fa6bb7fe640 1 -- 172.21.15.90:0/448240655 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa6bc108940 msgr2=0x7fa6bc1a1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:16.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.869+0000 7fa6bb7fe640 1 --2- 172.21.15.90:0/448240655 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa6bc108940 0x7fa6bc1a1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:16.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.869+0000 7fa6bb7fe640 1 -- 172.21.15.90:0/448240655 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6bc1a1ca0 con 0x7fa6bc103c40 2024-04-25T11:33:16.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.870+0000 7fa6bb7fe640 1 --2- 172.21.15.90:0/448240655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6bc103c40 0x7fa6bc19ef30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa6b000d950 tx=0x7fa6b000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:16.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.870+0000 7fa6b97fa640 1 -- 172.21.15.90:0/448240655 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b0017070 con 0x7fa6bc103c40 2024-04-25T11:33:16.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.870+0000 7fa6b97fa640 1 -- 172.21.15.90:0/448240655 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6b0009c30 con 0x7fa6bc103c40 2024-04-25T11:33:16.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.870+0000 7fa6b97fa640 1 -- 172.21.15.90:0/448240655 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b000f620 con 0x7fa6bc103c40 2024-04-25T11:33:16.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.870+0000 7fa6c257d640 1 -- 172.21.15.90:0/448240655 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6bc1a1f90 con 0x7fa6bc103c40 2024-04-25T11:33:16.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.870+0000 7fa6c257d640 1 -- 172.21.15.90:0/448240655 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6bc1a2400 con 0x7fa6bc103c40 2024-04-25T11:33:16.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.871+0000 7fa6c257d640 1 -- 172.21.15.90:0/448240655 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6bc107750 con 0x7fa6bc103c40 2024-04-25T11:33:16.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.873+0000 7fa6b97fa640 1 -- 172.21.15.90:0/448240655 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fa6b0010460 con 0x7fa6bc103c40 2024-04-25T11:33:16.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.874+0000 7fa6b97fa640 1 --2- 172.21.15.90:0/448240655 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fa6a40afaf0 0x7fa6a40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:16.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.874+0000 7fa6bbfff640 1 --2- 172.21.15.90:0/448240655 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fa6a40afaf0 0x7fa6a40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:16.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.874+0000 7fa6b97fa640 1 -- 172.21.15.90:0/448240655 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7fa6b00de750 con 0x7fa6bc103c40 2024-04-25T11:33:16.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.874+0000 7fa6bbfff640 1 --2- 172.21.15.90:0/448240655 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fa6a40afaf0 0x7fa6a40b2000 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc103600 tx=0x7fa6ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:16.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:16.877+0000 7fa6b97fa640 1 -- 172.21.15.90:0/448240655 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa6b00a73e0 con 0x7fa6bc103c40 2024-04-25T11:33:16.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:16 smithi090 ceph-mon[29575]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:33:16.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:16 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] up:active 2024-04-25T11:33:16.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:16 smithi090 ceph-mon[29575]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-25T11:33:16.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:16 smithi090 ceph-mon[29575]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T11:33:16.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:16 smithi090 ceph-mon[29575]: Cluster is now healthy 2024-04-25T11:33:16.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:16 smithi090 ceph-mon[29575]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:active} 2 up:standby-replay 6 up:standby 2024-04-25T11:33:16.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:16 smithi090 ceph-mon[29575]: fsmap cephfs:5 2 up:standby-replay 1 up:creating 5 up:standby 4 up:active 2024-04-25T11:33:16.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:16 smithi090 ceph-mon[29575]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-04-25T11:33:17.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:16 smithi138 ceph-mon[32328]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:33:17.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:16 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] up:active 2024-04-25T11:33:17.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:16 smithi138 ceph-mon[32328]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-25T11:33:17.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:16 smithi138 ceph-mon[32328]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T11:33:17.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:16 smithi138 ceph-mon[32328]: Cluster is now healthy 2024-04-25T11:33:17.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:16 smithi138 ceph-mon[32328]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:active} 2 up:standby-replay 6 up:standby 2024-04-25T11:33:17.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:16 smithi138 ceph-mon[32328]: fsmap cephfs:5 2 up:standby-replay 1 up:creating 5 up:standby 4 up:active 2024-04-25T11:33:17.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:16 smithi138 ceph-mon[32328]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-04-25T11:33:17.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.085+0000 7fa6c257d640 1 -- 172.21.15.90:0/448240655 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7fa6bc063070 con 0x7fa6a40afaf0 2024-04-25T11:33:17.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.090+0000 7fa6b97fa640 1 -- 172.21.15.90:0/448240655 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fa6bc063070 con 0x7fa6a40afaf0 2024-04-25T11:33:17.079 INFO:teuthology.orchestra.run.smithi090.stdout:/volumes/qa/sv_1/0dd91eff-2478-4546-a19a-4b4a80251a7f 2024-04-25T11:33:17.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.093+0000 7fa6c257d640 1 -- 172.21.15.90:0/448240655 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fa6a40afaf0 msgr2=0x7fa6a40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:17.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.093+0000 7fa6c257d640 1 --2- 172.21.15.90:0/448240655 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fa6a40afaf0 0x7fa6a40b2000 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc103600 tx=0x7fa6ac005f90 comp rx=0 tx=0).stop 2024-04-25T11:33:17.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.093+0000 7fa6c257d640 1 -- 172.21.15.90:0/448240655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6bc103c40 msgr2=0x7fa6bc19ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:17.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.093+0000 7fa6c257d640 1 --2- 172.21.15.90:0/448240655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6bc103c40 0x7fa6bc19ef30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa6b000d950 tx=0x7fa6b000de10 comp rx=0 tx=0).stop 2024-04-25T11:33:17.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.093+0000 7fa6c257d640 1 -- 172.21.15.90:0/448240655 shutdown_connections 2024-04-25T11:33:17.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.093+0000 7fa6c257d640 1 --2- 172.21.15.90:0/448240655 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa6bc108940 0x7fa6bc1a1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.093+0000 7fa6c257d640 1 --2- 172.21.15.90:0/448240655 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fa6a40afaf0 0x7fa6a40b2000 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.093+0000 7fa6c257d640 1 --2- 172.21.15.90:0/448240655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6bc103c40 0x7fa6bc19ef30 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.093+0000 7fa6c257d640 1 --2- 172.21.15.90:0/448240655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6bc102a00 0x7fa6bc075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.093+0000 7fa6c257d640 1 -- 172.21.15.90:0/448240655 >> 172.21.15.90:0/448240655 conn(0x7fa6bc0fdbf0 msgr2=0x7fa6bc104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:17.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.093+0000 7fa6c257d640 1 -- 172.21.15.90:0/448240655 shutdown_connections 2024-04-25T11:33:17.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.093+0000 7fa6c257d640 1 -- 172.21.15.90:0/448240655 wait complete. 2024-04-25T11:33:17.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:16 smithi156 ceph-mon[33045]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:33:17.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:16 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] up:active 2024-04-25T11:33:17.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:16 smithi156 ceph-mon[33045]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-25T11:33:17.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:16 smithi156 ceph-mon[33045]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T11:33:17.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:16 smithi156 ceph-mon[33045]: Cluster is now healthy 2024-04-25T11:33:17.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:16 smithi156 ceph-mon[33045]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:active} 2 up:standby-replay 6 up:standby 2024-04-25T11:33:17.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:16 smithi156 ceph-mon[33045]: fsmap cephfs:5 2 up:standby-replay 1 up:creating 5 up:standby 4 up:active 2024-04-25T11:33:17.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:16 smithi156 ceph-mon[33045]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-04-25T11:33:17.093 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:33:17.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.211+0000 7fd0c8292640 1 -- 172.21.15.90:0/2722818354 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0c0109a40 msgr2=0x7fd0c010be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:17.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.211+0000 7fd0c8292640 1 --2- 172.21.15.90:0/2722818354 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0c0109a40 0x7fd0c010be60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd0b8009ca0 tx=0x7fd0b802a8c0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.211+0000 7fd0c8292640 1 -- 172.21.15.90:0/2722818354 shutdown_connections 2024-04-25T11:33:17.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.211+0000 7fd0c8292640 1 --2- 172.21.15.90:0/2722818354 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0c0109a40 0x7fd0c010be60 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.211+0000 7fd0c8292640 1 --2- 172.21.15.90:0/2722818354 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0c0104d40 0x7fd0c0105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.211+0000 7fd0c8292640 1 --2- 172.21.15.90:0/2722818354 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd0c0103b00 0x7fd0c0104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.211+0000 7fd0c8292640 1 -- 172.21.15.90:0/2722818354 >> 172.21.15.90:0/2722818354 conn(0x7fd0c00fed10 msgr2=0x7fd0c01011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:17.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.211+0000 7fd0c8292640 1 -- 172.21.15.90:0/2722818354 shutdown_connections 2024-04-25T11:33:17.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.211+0000 7fd0c8292640 1 -- 172.21.15.90:0/2722818354 wait complete. 2024-04-25T11:33:17.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.211+0000 7fd0c8292640 1 Processor -- start 2024-04-25T11:33:17.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.211+0000 7fd0c8292640 1 -- start start 2024-04-25T11:33:17.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.212+0000 7fd0c8292640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd0c0103b00 0x7fd0c019a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:17.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.212+0000 7fd0c8292640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0c0104d40 0x7fd0c019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:17.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.212+0000 7fd0c8292640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0c0109a40 0x7fd0c01a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:17.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.212+0000 7fd0c8292640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c010f630 con 0x7fd0c0104d40 2024-04-25T11:33:17.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.212+0000 7fd0c8292640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c010f4b0 con 0x7fd0c0103b00 2024-04-25T11:33:17.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.212+0000 7fd0c8292640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c010f7b0 con 0x7fd0c0109a40 2024-04-25T11:33:17.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.212+0000 7fd0c582d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0c0104d40 0x7fd0c019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:17.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.212+0000 7fd0c582d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0c0104d40 0x7fd0c019a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36444/0 (socket says 172.21.15.90:36444) 2024-04-25T11:33:17.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.212+0000 7fd0c582d640 1 -- 172.21.15.90:0/1574051757 learned_addr learned my addr 172.21.15.90:0/1574051757 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:17.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.212+0000 7fd0c602e640 1 --2- 172.21.15.90:0/1574051757 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd0c0103b00 0x7fd0c019a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:17.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.212+0000 7fd0c582d640 1 -- 172.21.15.90:0/1574051757 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0c0109a40 msgr2=0x7fd0c01a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:17.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.212+0000 7fd0c682f640 1 --2- 172.21.15.90:0/1574051757 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0c0109a40 0x7fd0c01a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:17.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.212+0000 7fd0c582d640 1 --2- 172.21.15.90:0/1574051757 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0c0109a40 0x7fd0c01a16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.212+0000 7fd0c582d640 1 -- 172.21.15.90:0/1574051757 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd0c0103b00 msgr2=0x7fd0c019a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:17.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.212+0000 7fd0c582d640 1 --2- 172.21.15.90:0/1574051757 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd0c0103b00 0x7fd0c019a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.212+0000 7fd0c582d640 1 -- 172.21.15.90:0/1574051757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0c01a1d40 con 0x7fd0c0104d40 2024-04-25T11:33:17.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.212+0000 7fd0c602e640 1 --2- 172.21.15.90:0/1574051757 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd0c0103b00 0x7fd0c019a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:33:17.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.212+0000 7fd0c582d640 1 --2- 172.21.15.90:0/1574051757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0c0104d40 0x7fd0c019a740 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc00d770 tx=0x7fd0bc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:17.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.212+0000 7fd0b77fe640 1 -- 172.21.15.90:0/1574051757 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc017070 con 0x7fd0c0104d40 2024-04-25T11:33:17.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.212+0000 7fd0b77fe640 1 -- 172.21.15.90:0/1574051757 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0bc0054d0 con 0x7fd0c0104d40 2024-04-25T11:33:17.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.212+0000 7fd0c8292640 1 -- 172.21.15.90:0/1574051757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0c01a1fd0 con 0x7fd0c0104d40 2024-04-25T11:33:17.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.212+0000 7fd0b77fe640 1 -- 172.21.15.90:0/1574051757 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc004c30 con 0x7fd0c0104d40 2024-04-25T11:33:17.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.212+0000 7fd0c8292640 1 -- 172.21.15.90:0/1574051757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0c01a2490 con 0x7fd0c0104d40 2024-04-25T11:33:17.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.214+0000 7fd0c8292640 1 -- 172.21.15.90:0/1574051757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0c0108850 con 0x7fd0c0104d40 2024-04-25T11:33:17.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.215+0000 7fd0b77fe640 1 -- 172.21.15.90:0/1574051757 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fd0bc004e10 con 0x7fd0c0104d40 2024-04-25T11:33:17.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.215+0000 7fd0b77fe640 1 --2- 172.21.15.90:0/1574051757 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd0ac0afaf0 0x7fd0ac0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:17.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.216+0000 7fd0c602e640 1 --2- 172.21.15.90:0/1574051757 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd0ac0afaf0 0x7fd0ac0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:17.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.216+0000 7fd0b77fe640 1 -- 172.21.15.90:0/1574051757 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7fd0bc0de640 con 0x7fd0c0104d40 2024-04-25T11:33:17.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.216+0000 7fd0c602e640 1 --2- 172.21.15.90:0/1574051757 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd0ac0afaf0 0x7fd0ac0b2000 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd0c0104700 tx=0x7fd0b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:17.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.220+0000 7fd0b77fe640 1 -- 172.21.15.90:0/1574051757 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd0bc0a7370 con 0x7fd0c0104d40 2024-04-25T11:33:17.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.434+0000 7fd0c8292640 1 -- 172.21.15.90:0/1574051757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0c002db50 con 0x7fd0c0104d40 2024-04-25T11:33:17.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.435+0000 7fd0b77fe640 1 -- 172.21.15.90:0/1574051757 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11930 (secure 0 0 0) 0x7fd0bc0a6c70 con 0x7fd0c0104d40 2024-04-25T11:33:17.424 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:33:17.424 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:33:16.640084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524,24475,24499,24458,24431]},"id":1}]} 2024-04-25T11:33:17.424 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 25 2024-04-25T11:33:17.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.437+0000 7fd0c8292640 1 -- 172.21.15.90:0/1574051757 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd0ac0afaf0 msgr2=0x7fd0ac0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:17.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.437+0000 7fd0c8292640 1 --2- 172.21.15.90:0/1574051757 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd0ac0afaf0 0x7fd0ac0b2000 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd0c0104700 tx=0x7fd0b0005f90 comp rx=0 tx=0).stop 2024-04-25T11:33:17.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.437+0000 7fd0c8292640 1 -- 172.21.15.90:0/1574051757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0c0104d40 msgr2=0x7fd0c019a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:17.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.437+0000 7fd0c8292640 1 --2- 172.21.15.90:0/1574051757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0c0104d40 0x7fd0c019a740 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc00d770 tx=0x7fd0bc00dc30 comp rx=0 tx=0).stop 2024-04-25T11:33:17.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.437+0000 7fd0c8292640 1 -- 172.21.15.90:0/1574051757 shutdown_connections 2024-04-25T11:33:17.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.437+0000 7fd0c8292640 1 --2- 172.21.15.90:0/1574051757 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0c0109a40 0x7fd0c01a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.437+0000 7fd0c8292640 1 --2- 172.21.15.90:0/1574051757 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd0ac0afaf0 0x7fd0ac0b2000 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.437+0000 7fd0c8292640 1 --2- 172.21.15.90:0/1574051757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0c0104d40 0x7fd0c019a740 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.437+0000 7fd0c8292640 1 --2- 172.21.15.90:0/1574051757 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd0c0103b00 0x7fd0c019a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.437+0000 7fd0c8292640 1 -- 172.21.15.90:0/1574051757 >> 172.21.15.90:0/1574051757 conn(0x7fd0c00fed10 msgr2=0x7fd0c0105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:17.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.437+0000 7fd0c8292640 1 -- 172.21.15.90:0/1574051757 shutdown_connections 2024-04-25T11:33:17.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.437+0000 7fd0c8292640 1 -- 172.21.15.90:0/1574051757 wait complete. 2024-04-25T11:33:17.437 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:33:17.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.556+0000 7f2c74364640 1 -- 172.21.15.90:0/1767864964 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c6c108900 msgr2=0x7f2c6c10ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:17.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.556+0000 7f2c74364640 1 --2- 172.21.15.90:0/1767864964 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c6c108900 0x7f2c6c10ad20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2c64009be0 tx=0x7f2c6402a6c0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.556+0000 7f2c74364640 1 -- 172.21.15.90:0/1767864964 shutdown_connections 2024-04-25T11:33:17.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.556+0000 7f2c74364640 1 --2- 172.21.15.90:0/1767864964 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c6c108900 0x7f2c6c10ad20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.556+0000 7f2c74364640 1 --2- 172.21.15.90:0/1767864964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c6c103c00 0x7f2c6c104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.556+0000 7f2c74364640 1 --2- 172.21.15.90:0/1767864964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2c6c1029c0 0x7f2c6c1036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.556+0000 7f2c74364640 1 -- 172.21.15.90:0/1767864964 >> 172.21.15.90:0/1767864964 conn(0x7f2c6c0fdbd0 msgr2=0x7f2c6c100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:17.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.556+0000 7f2c74364640 1 -- 172.21.15.90:0/1767864964 shutdown_connections 2024-04-25T11:33:17.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.556+0000 7f2c74364640 1 -- 172.21.15.90:0/1767864964 wait complete. 2024-04-25T11:33:17.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.557+0000 7f2c74364640 1 Processor -- start 2024-04-25T11:33:17.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.557+0000 7f2c74364640 1 -- start start 2024-04-25T11:33:17.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.557+0000 7f2c74364640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2c6c1029c0 0x7f2c6c1990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:17.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.557+0000 7f2c74364640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c6c103c00 0x7f2c6c199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:17.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.557+0000 7f2c74364640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c6c108900 0x7f2c6c1a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:17.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.557+0000 7f2c74364640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c10e4f0 con 0x7f2c6c108900 2024-04-25T11:33:17.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.557+0000 7f2c74364640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c10e370 con 0x7f2c6c103c00 2024-04-25T11:33:17.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.557+0000 7f2c74364640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c10e670 con 0x7f2c6c1029c0 2024-04-25T11:33:17.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.557+0000 7f2c72901640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c6c108900 0x7f2c6c1a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:17.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.557+0000 7f2c72901640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c6c108900 0x7f2c6c1a0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36454/0 (socket says 172.21.15.90:36454) 2024-04-25T11:33:17.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.557+0000 7f2c72901640 1 -- 172.21.15.90:0/2596995302 learned_addr learned my addr 172.21.15.90:0/2596995302 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:17.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.557+0000 7f2c718ff640 1 --2- 172.21.15.90:0/2596995302 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c6c103c00 0x7f2c6c199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:17.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.557+0000 7f2c72100640 1 --2- 172.21.15.90:0/2596995302 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2c6c1029c0 0x7f2c6c1990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:17.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.558+0000 7f2c72901640 1 -- 172.21.15.90:0/2596995302 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2c6c1029c0 msgr2=0x7f2c6c1990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:17.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.558+0000 7f2c72901640 1 --2- 172.21.15.90:0/2596995302 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2c6c1029c0 0x7f2c6c1990c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.558+0000 7f2c72901640 1 -- 172.21.15.90:0/2596995302 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c6c103c00 msgr2=0x7f2c6c199600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:17.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.558+0000 7f2c72901640 1 --2- 172.21.15.90:0/2596995302 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c6c103c00 0x7f2c6c199600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.558+0000 7f2c72901640 1 -- 172.21.15.90:0/2596995302 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6c1a0c00 con 0x7f2c6c108900 2024-04-25T11:33:17.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.558+0000 7f2c718ff640 1 --2- 172.21.15.90:0/2596995302 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c6c103c00 0x7f2c6c199600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:33:17.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.558+0000 7f2c72901640 1 --2- 172.21.15.90:0/2596995302 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c6c108900 0x7f2c6c1a0590 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2c64034040 tx=0x7f2c64030d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:17.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.558+0000 7f2c637fe640 1 -- 172.21.15.90:0/2596995302 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c64041020 con 0x7f2c6c108900 2024-04-25T11:33:17.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.558+0000 7f2c74364640 1 -- 172.21.15.90:0/2596995302 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c6c1a0e90 con 0x7f2c6c108900 2024-04-25T11:33:17.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.558+0000 7f2c637fe640 1 -- 172.21.15.90:0/2596995302 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c640318f0 con 0x7f2c6c108900 2024-04-25T11:33:17.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.558+0000 7f2c637fe640 1 -- 172.21.15.90:0/2596995302 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c64036040 con 0x7f2c6c108900 2024-04-25T11:33:17.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.558+0000 7f2c74364640 1 -- 172.21.15.90:0/2596995302 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c6c1a12f0 con 0x7f2c6c108900 2024-04-25T11:33:17.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.559+0000 7f2c74364640 1 -- 172.21.15.90:0/2596995302 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c6c107710 con 0x7f2c6c108900 2024-04-25T11:33:17.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.561+0000 7f2c637fe640 1 -- 172.21.15.90:0/2596995302 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f2c64044020 con 0x7f2c6c108900 2024-04-25T11:33:17.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.561+0000 7f2c637fe640 1 --2- 172.21.15.90:0/2596995302 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f2c580afaf0 0x7f2c580b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:17.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.561+0000 7f2c72100640 1 --2- 172.21.15.90:0/2596995302 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f2c580afaf0 0x7f2c580b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:17.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.561+0000 7f2c637fe640 1 -- 172.21.15.90:0/2596995302 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7f2c640ffd00 con 0x7f2c6c108900 2024-04-25T11:33:17.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.562+0000 7f2c72100640 1 --2- 172.21.15.90:0/2596995302 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f2c580afaf0 0x7f2c580b2000 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2c68005fd0 tx=0x7f2c68005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:17.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.565+0000 7f2c637fe640 1 -- 172.21.15.90:0/2596995302 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2c6404b020 con 0x7f2c6c108900 2024-04-25T11:33:17.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.660+0000 7f2c637fe640 1 -- 172.21.15.90:0/2596995302 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f2c640c45c0 con 0x7f2c6c108900 2024-04-25T11:33:17.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:17 smithi090 ceph-mon[29575]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:33:17.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:17 smithi090 ceph-mon[29575]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:33:17.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:17 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] up:active 2024-04-25T11:33:17.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:17 smithi090 ceph-mon[29575]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-04-25T11:33:17.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:17 smithi090 ceph-mon[29575]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T11:33:17.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:17 smithi090 ceph-mon[29575]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:33:17.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:17 smithi090 ceph-mon[29575]: pgmap v277: 97 pgs: 97 active+clean; 583 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 5.1 KiB/s wr, 20 op/s 2024-04-25T11:33:17.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:17 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1574051757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:33:17.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.751+0000 7f2c74364640 1 -- 172.21.15.90:0/2596995302 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2c6c02dd50 con 0x7f2c6c108900 2024-04-25T11:33:17.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.752+0000 7f2c637fe640 1 -- 172.21.15.90:0/2596995302 <== mon.0 v2:172.21.15.90:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+22238 (secure 0 0 0) 0x7f2c640c8a30 con 0x7f2c6c108900 2024-04-25T11:33:17.741 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:33:17.741 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":77,"fsid":"58bd4e7a-02f6-11ef-bc93-c7b262605968","created":"2024-04-25T11:25:23.574550+0000","modified":"2024-04-25T11:33:11.866090+0000","last_up_change":"2024-04-25T11:32:23.852211+0000","last_in_change":"2024-04-25T11:32:04.188507+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:28:42.746649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:33:08.442925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:33:09.352886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e54d888-e67b-4951-8ae7-e355fb6b10a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6803","nonce":1201629159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6805","nonce":1201629159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6809","nonce":1201629159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6807","nonce":1201629159}]},"public_addr":"172.21.15.90:6803/1201629159","cluster_addr":"172.21.15.90:6805/1201629159","heartbeat_back_addr":"172.21.15.90:6809/1201629159","heartbeat_front_addr":"172.21.15.90:6807/1201629159","state":["exists","up"]},{"osd":1,"uuid":"91a71c78-dae9-4bc7-9f6f-9e55cfefbe28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6801","nonce":3976851933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6803","nonce":3976851933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6807","nonce":3976851933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6805","nonce":3976851933}]},"public_addr":"172.21.15.138:6801/3976851933","cluster_addr":"172.21.15.138:6803/3976851933","heartbeat_back_addr":"172.21.15.138:6807/3976851933","heartbeat_front_addr":"172.21.15.138:6805/3976851933","state":["exists","up"]},{"osd":2,"uuid":"daff6c7f-80fa-4859-9339-ef7a5e840832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6801","nonce":1505807670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6803","nonce":1505807670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6807","nonce":1505807670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6805","nonce":1505807670}]},"public_addr":"172.21.15.156:6801/1505807670","cluster_addr":"172.21.15.156:6803/1505807670","heartbeat_back_addr":"172.21.15.156:6807/1505807670","heartbeat_front_addr":"172.21.15.156:6805/1505807670","state":["exists","up"]},{"osd":3,"uuid":"33fd94ca-7465-401a-bc1e-706b6b31b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6811","nonce":2520012383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6813","nonce":2520012383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6817","nonce":2520012383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6815","nonce":2520012383}]},"public_addr":"172.21.15.90:6811/2520012383","cluster_addr":"172.21.15.90:6813/2520012383","heartbeat_back_addr":"172.21.15.90:6817/2520012383","heartbeat_front_addr":"172.21.15.90:6815/2520012383","state":["exists","up"]},{"osd":4,"uuid":"c5326212-750c-49fd-96e3-d1e62bb4b7f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6809","nonce":3752678279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6811","nonce":3752678279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6815","nonce":3752678279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6813","nonce":3752678279}]},"public_addr":"172.21.15.138:6809/3752678279","cluster_addr":"172.21.15.138:6811/3752678279","heartbeat_back_addr":"172.21.15.138:6815/3752678279","heartbeat_front_addr":"172.21.15.138:6813/3752678279","state":["exists","up"]},{"osd":5,"uuid":"d04a1156-820b-4b43-be89-e16a8910c3aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6809","nonce":3591210191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6811","nonce":3591210191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6815","nonce":3591210191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6813","nonce":3591210191}]},"public_addr":"172.21.15.156:6809/3591210191","cluster_addr":"172.21.15.156:6811/3591210191","heartbeat_back_addr":"172.21.15.156:6815/3591210191","heartbeat_front_addr":"172.21.15.156:6813/3591210191","state":["exists","up"]},{"osd":6,"uuid":"75675aa3-e978-490c-8205-13d9868a3d6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6819","nonce":2364955456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6821","nonce":2364955456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6825","nonce":2364955456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6823","nonce":2364955456}]},"public_addr":"172.21.15.90:6819/2364955456","cluster_addr":"172.21.15.90:6821/2364955456","heartbeat_back_addr":"172.21.15.90:6825/2364955456","heartbeat_front_addr":"172.21.15.90:6823/2364955456","state":["exists","up"]},{"osd":7,"uuid":"1a95fdf5-f548-4271-bbcd-e330f7955eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6817","nonce":4012476387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6819","nonce":4012476387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6823","nonce":4012476387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6821","nonce":4012476387}]},"public_addr":"172.21.15.138:6817/4012476387","cluster_addr":"172.21.15.138:6819/4012476387","heartbeat_back_addr":"172.21.15.138:6823/4012476387","heartbeat_front_addr":"172.21.15.138:6821/4012476387","state":["exists","up"]},{"osd":8,"uuid":"605e10b5-5806-4c1f-8e28-a7994d34ddce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6817","nonce":101092224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6819","nonce":101092224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6823","nonce":101092224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6821","nonce":101092224}]},"public_addr":"172.21.15.156:6817/101092224","cluster_addr":"172.21.15.156:6819/101092224","heartbeat_back_addr":"172.21.15.156:6823/101092224","heartbeat_front_addr":"172.21.15.156:6821/101092224","state":["exists","up"]},{"osd":9,"uuid":"d70fdf5e-9d0f-483d-a7d8-505b79b8a418","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6827","nonce":1171265966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6829","nonce":1171265966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6833","nonce":1171265966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6831","nonce":1171265966}]},"public_addr":"172.21.15.90:6827/1171265966","cluster_addr":"172.21.15.90:6829/1171265966","heartbeat_back_addr":"172.21.15.90:6833/1171265966","heartbeat_front_addr":"172.21.15.90:6831/1171265966","state":["exists","up"]},{"osd":10,"uuid":"1ec86c7d-c3a4-432d-9664-8e40836e1e20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6825","nonce":4264666232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6827","nonce":4264666232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6831","nonce":4264666232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6829","nonce":4264666232}]},"public_addr":"172.21.15.138:6825/4264666232","cluster_addr":"172.21.15.138:6827/4264666232","heartbeat_back_addr":"172.21.15.138:6831/4264666232","heartbeat_front_addr":"172.21.15.138:6829/4264666232","state":["exists","up"]},{"osd":11,"uuid":"51cbe729-292e-4d7b-b8e9-e1e2821ffe76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6825","nonce":1224251744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6827","nonce":1224251744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6831","nonce":1224251744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6829","nonce":1224251744}]},"public_addr":"172.21.15.156:6825/1224251744","cluster_addr":"172.21.15.156:6827/1224251744","heartbeat_back_addr":"172.21.15.156:6831/1224251744","heartbeat_front_addr":"172.21.15.156:6829/1224251744","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:27:50.794226+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:15.030278+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:38.518163+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:03.107351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:27.500707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:51.463545+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:16.015076+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:41.182071+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:05.362309+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:31.048936+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:56.488509+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:32:22.129178+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/2381633953":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6800/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/2190846900":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1707792941":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6800/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6801/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/6061052":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1849633392":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6801/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/2277490702":"2024-04-26T11:26:16.655278+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:33:17.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.754+0000 7f2c74364640 1 -- 172.21.15.90:0/2596995302 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f2c580afaf0 msgr2=0x7f2c580b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:17.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.754+0000 7f2c74364640 1 --2- 172.21.15.90:0/2596995302 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f2c580afaf0 0x7f2c580b2000 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2c68005fd0 tx=0x7f2c68005850 comp rx=0 tx=0).stop 2024-04-25T11:33:17.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.754+0000 7f2c74364640 1 -- 172.21.15.90:0/2596995302 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c6c108900 msgr2=0x7f2c6c1a0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:17.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.754+0000 7f2c74364640 1 --2- 172.21.15.90:0/2596995302 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c6c108900 0x7f2c6c1a0590 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2c64034040 tx=0x7f2c64030d20 comp rx=0 tx=0).stop 2024-04-25T11:33:17.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.754+0000 7f2c74364640 1 -- 172.21.15.90:0/2596995302 shutdown_connections 2024-04-25T11:33:17.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.754+0000 7f2c74364640 1 --2- 172.21.15.90:0/2596995302 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f2c580afaf0 0x7f2c580b2000 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.754+0000 7f2c74364640 1 --2- 172.21.15.90:0/2596995302 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c6c108900 0x7f2c6c1a0590 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.754+0000 7f2c74364640 1 --2- 172.21.15.90:0/2596995302 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c6c103c00 0x7f2c6c199600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.754+0000 7f2c74364640 1 --2- 172.21.15.90:0/2596995302 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2c6c1029c0 0x7f2c6c1990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.754+0000 7f2c74364640 1 -- 172.21.15.90:0/2596995302 >> 172.21.15.90:0/2596995302 conn(0x7f2c6c0fdbd0 msgr2=0x7f2c6c1048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:17.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.755+0000 7f2c74364640 1 -- 172.21.15.90:0/2596995302 shutdown_connections 2024-04-25T11:33:17.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.755+0000 7f2c74364640 1 -- 172.21.15.90:0/2596995302 wait complete. 2024-04-25T11:33:17.756 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T11:33:17.756 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T11:33:17.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.876+0000 7f9c4e1ee640 1 -- 172.21.15.90:0/1700605425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c48109a40 msgr2=0x7f9c4810be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:17.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.876+0000 7f9c4e1ee640 1 --2- 172.21.15.90:0/1700605425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c48109a40 0x7f9c4810be60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9c40009ca0 tx=0x7f9c4002a8c0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.876+0000 7f9c4e1ee640 1 -- 172.21.15.90:0/1700605425 shutdown_connections 2024-04-25T11:33:17.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.876+0000 7f9c4e1ee640 1 --2- 172.21.15.90:0/1700605425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c48109a40 0x7f9c4810be60 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.876+0000 7f9c4e1ee640 1 --2- 172.21.15.90:0/1700605425 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9c48104d40 0x7f9c48105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.876+0000 7f9c4e1ee640 1 --2- 172.21.15.90:0/1700605425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c48103b00 0x7f9c48104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.876+0000 7f9c4e1ee640 1 -- 172.21.15.90:0/1700605425 >> 172.21.15.90:0/1700605425 conn(0x7f9c480fed30 msgr2=0x7f9c481011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:17.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.876+0000 7f9c4e1ee640 1 -- 172.21.15.90:0/1700605425 shutdown_connections 2024-04-25T11:33:17.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.876+0000 7f9c4e1ee640 1 -- 172.21.15.90:0/1700605425 wait complete. 2024-04-25T11:33:17.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.877+0000 7f9c4e1ee640 1 Processor -- start 2024-04-25T11:33:17.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.877+0000 7f9c4e1ee640 1 -- start start 2024-04-25T11:33:17.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.877+0000 7f9c4e1ee640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c48103b00 0x7f9c4819a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:17.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.877+0000 7f9c4e1ee640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9c48104d40 0x7f9c4819a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:17.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.877+0000 7f9c4e1ee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c48109a40 0x7f9c481a1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:17.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.877+0000 7f9c4e1ee640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9c4810f630 con 0x7f9c48103b00 2024-04-25T11:33:17.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.877+0000 7f9c4e1ee640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9c4810f4b0 con 0x7f9c48104d40 2024-04-25T11:33:17.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.877+0000 7f9c4e1ee640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9c4810f7b0 con 0x7f9c48109a40 2024-04-25T11:33:17.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.877+0000 7f9c477fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c48103b00 0x7f9c4819a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:17.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.877+0000 7f9c477fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c48103b00 0x7f9c4819a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36466/0 (socket says 172.21.15.90:36466) 2024-04-25T11:33:17.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.877+0000 7f9c477fe640 1 -- 172.21.15.90:0/1022260804 learned_addr learned my addr 172.21.15.90:0/1022260804 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:17.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.877+0000 7f9c3ffff640 1 --2- 172.21.15.90:0/1022260804 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9c48104d40 0x7f9c4819a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:17.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.877+0000 7f9c477fe640 1 -- 172.21.15.90:0/1022260804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c48109a40 msgr2=0x7f9c481a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:17.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.877+0000 7f9c477fe640 1 --2- 172.21.15.90:0/1022260804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c48109a40 0x7f9c481a1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.877+0000 7f9c477fe640 1 -- 172.21.15.90:0/1022260804 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9c48104d40 msgr2=0x7f9c4819a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:17.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.877+0000 7f9c477fe640 1 --2- 172.21.15.90:0/1022260804 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9c48104d40 0x7f9c4819a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.877+0000 7f9c477fe640 1 -- 172.21.15.90:0/1022260804 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c481a1c10 con 0x7f9c48103b00 2024-04-25T11:33:17.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.878+0000 7f9c3ffff640 1 --2- 172.21.15.90:0/1022260804 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9c48104d40 0x7f9c4819a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:33:17.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.878+0000 7f9c477fe640 1 --2- 172.21.15.90:0/1022260804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c48103b00 0x7f9c4819a1d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9c38005d00 tx=0x7f9c3800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:17.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.878+0000 7f9c457fa640 1 -- 172.21.15.90:0/1022260804 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c38017070 con 0x7f9c48103b00 2024-04-25T11:33:17.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.878+0000 7f9c457fa640 1 -- 172.21.15.90:0/1022260804 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c38009c30 con 0x7f9c48103b00 2024-04-25T11:33:17.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.878+0000 7f9c4e1ee640 1 -- 172.21.15.90:0/1022260804 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c481a1f00 con 0x7f9c48103b00 2024-04-25T11:33:17.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.878+0000 7f9c457fa640 1 -- 172.21.15.90:0/1022260804 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c3800f620 con 0x7f9c48103b00 2024-04-25T11:33:17.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.878+0000 7f9c4e1ee640 1 -- 172.21.15.90:0/1022260804 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c481a2310 con 0x7f9c48103b00 2024-04-25T11:33:17.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.879+0000 7f9c4e1ee640 1 -- 172.21.15.90:0/1022260804 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c4806afa0 con 0x7f9c48103b00 2024-04-25T11:33:17.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.880+0000 7f9c457fa640 1 -- 172.21.15.90:0/1022260804 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f9c3801a070 con 0x7f9c48103b00 2024-04-25T11:33:17.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.881+0000 7f9c457fa640 1 --2- 172.21.15.90:0/1022260804 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f9c340afb80 0x7f9c340b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:17.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.881+0000 7f9c3ffff640 1 --2- 172.21.15.90:0/1022260804 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f9c340afb80 0x7f9c340b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:17.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.882+0000 7f9c457fa640 1 -- 172.21.15.90:0/1022260804 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7f9c380de6f0 con 0x7f9c48103b00 2024-04-25T11:33:17.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.882+0000 7f9c3ffff640 1 --2- 172.21.15.90:0/1022260804 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f9c340afb80 0x7f9c340b2090 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9c30005fd0 tx=0x7f9c30005f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:17.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:17.885+0000 7f9c457fa640 1 -- 172.21.15.90:0/1022260804 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9c38009de0 con 0x7f9c48103b00 2024-04-25T11:33:18.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.062+0000 7f9c4e1ee640 1 -- 172.21.15.90:0/1022260804 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9c48062fe0 con 0x7f9c340afb80 2024-04-25T11:33:18.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:17 smithi138 ceph-mon[32328]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:33:18.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:17 smithi138 ceph-mon[32328]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:33:18.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:17 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] up:active 2024-04-25T11:33:18.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:17 smithi138 ceph-mon[32328]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-04-25T11:33:18.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:17 smithi138 ceph-mon[32328]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T11:33:18.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:17 smithi138 ceph-mon[32328]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:33:18.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:17 smithi138 ceph-mon[32328]: pgmap v277: 97 pgs: 97 active+clean; 583 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 5.1 KiB/s wr, 20 op/s 2024-04-25T11:33:18.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:17 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1574051757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:33:18.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.070+0000 7f9c457fa640 1 -- 172.21.15.90:0/1022260804 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258740 (secure 0 0 0) 0x7f9c48062fe0 con 0x7f9c340afb80 2024-04-25T11:33:18.061 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:33:18.063 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-25T11:33:18.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.073+0000 7f9c4e1ee640 1 -- 172.21.15.90:0/1022260804 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f9c340afb80 msgr2=0x7f9c340b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:18.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.073+0000 7f9c4e1ee640 1 --2- 172.21.15.90:0/1022260804 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f9c340afb80 0x7f9c340b2090 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9c30005fd0 tx=0x7f9c30005f40 comp rx=0 tx=0).stop 2024-04-25T11:33:18.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.073+0000 7f9c4e1ee640 1 -- 172.21.15.90:0/1022260804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c48103b00 msgr2=0x7f9c4819a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:18.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.073+0000 7f9c4e1ee640 1 --2- 172.21.15.90:0/1022260804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c48103b00 0x7f9c4819a1d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9c38005d00 tx=0x7f9c3800dde0 comp rx=0 tx=0).stop 2024-04-25T11:33:18.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.073+0000 7f9c4e1ee640 1 -- 172.21.15.90:0/1022260804 shutdown_connections 2024-04-25T11:33:18.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.073+0000 7f9c4e1ee640 1 --2- 172.21.15.90:0/1022260804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c48109a40 0x7f9c481a1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:18.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.073+0000 7f9c4e1ee640 1 --2- 172.21.15.90:0/1022260804 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9c48104d40 0x7f9c4819a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:18.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.073+0000 7f9c4e1ee640 1 --2- 172.21.15.90:0/1022260804 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f9c340afb80 0x7f9c340b2090 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:18.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.073+0000 7f9c4e1ee640 1 --2- 172.21.15.90:0/1022260804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c48103b00 0x7f9c4819a1d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:18.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.073+0000 7f9c4e1ee640 1 -- 172.21.15.90:0/1022260804 >> 172.21.15.90:0/1022260804 conn(0x7f9c480fed30 msgr2=0x7f9c48105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:18.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.073+0000 7f9c4e1ee640 1 -- 172.21.15.90:0/1022260804 shutdown_connections 2024-04-25T11:33:18.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.073+0000 7f9c4e1ee640 1 -- 172.21.15.90:0/1022260804 wait complete. 2024-04-25T11:33:18.073 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":277,"stamp":"2024-04-25T11:33:16.741717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596513,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":94,"num_read_kb":80,"num_write":247,"num_write_kb":2184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":200,"ondisk_log_size":200,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":233,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":381592,"kb_used_data":11608,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124560232,"statfs":{"total":1151940427776,"available":1151549677568,"internally_reserved":0,"allocated":11886592,"data_stored":6530949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6145,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":13,"num_read_kb":11,"num_write":124,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.727353"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886955+0000","last_change":"2024-04-25T11:33:10.892144+0000","last_active":"2024-04-25T11:33:11.886955+0000","last_peered":"2024-04-25T11:33:11.886955+0000","last_clean":"2024-04-25T11:33:11.886955+0000","last_became_active":"2024-04-25T11:33:10.891721+0000","last_became_peered":"2024-04-25T11:33:10.891721+0000","last_unstale":"2024-04-25T11:33:11.886955+0000","last_undegraded":"2024-04-25T11:33:11.886955+0000","last_fullsized":"2024-04-25T11:33:11.886955+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:58:25.447086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.937797+0000","last_change":"2024-04-25T11:33:10.891647+0000","last_active":"2024-04-25T11:33:11.937797+0000","last_peered":"2024-04-25T11:33:11.937797+0000","last_clean":"2024-04-25T11:33:11.937797+0000","last_became_active":"2024-04-25T11:33:10.891283+0000","last_became_peered":"2024-04-25T11:33:10.891283+0000","last_unstale":"2024-04-25T11:33:11.937797+0000","last_undegraded":"2024-04-25T11:33:11.937797+0000","last_fullsized":"2024-04-25T11:33:11.937797+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:12.669438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.937801+0000","last_change":"2024-04-25T11:33:10.890928+0000","last_active":"2024-04-25T11:33:11.937801+0000","last_peered":"2024-04-25T11:33:11.937801+0000","last_clean":"2024-04-25T11:33:11.937801+0000","last_became_active":"2024-04-25T11:33:10.889093+0000","last_became_peered":"2024-04-25T11:33:10.889093+0000","last_unstale":"2024-04-25T11:33:11.937801+0000","last_undegraded":"2024-04-25T11:33:11.937801+0000","last_fullsized":"2024-04-25T11:33:11.937801+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:28:09.038650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.871223+0000","last_change":"2024-04-25T11:33:10.890184+0000","last_active":"2024-04-25T11:33:11.871223+0000","last_peered":"2024-04-25T11:33:11.871223+0000","last_clean":"2024-04-25T11:33:11.871223+0000","last_became_active":"2024-04-25T11:33:10.889367+0000","last_became_peered":"2024-04-25T11:33:10.889367+0000","last_unstale":"2024-04-25T11:33:11.871223+0000","last_undegraded":"2024-04-25T11:33:11.871223+0000","last_fullsized":"2024-04-25T11:33:11.871223+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:42.798295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886028+0000","last_change":"2024-04-25T11:33:10.889758+0000","last_active":"2024-04-25T11:33:11.886028+0000","last_peered":"2024-04-25T11:33:11.886028+0000","last_clean":"2024-04-25T11:33:11.886028+0000","last_became_active":"2024-04-25T11:33:10.889384+0000","last_became_peered":"2024-04-25T11:33:10.889384+0000","last_unstale":"2024-04-25T11:33:11.886028+0000","last_undegraded":"2024-04-25T11:33:11.886028+0000","last_fullsized":"2024-04-25T11:33:11.886028+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:54:38.794216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886638+0000","last_change":"2024-04-25T11:33:10.889348+0000","last_active":"2024-04-25T11:33:11.886638+0000","last_peered":"2024-04-25T11:33:11.886638+0000","last_clean":"2024-04-25T11:33:11.886638+0000","last_became_active":"2024-04-25T11:33:10.888908+0000","last_became_peered":"2024-04-25T11:33:10.888908+0000","last_unstale":"2024-04-25T11:33:11.886638+0000","last_undegraded":"2024-04-25T11:33:11.886638+0000","last_fullsized":"2024-04-25T11:33:11.886638+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:02:59.555921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886809+0000","last_change":"2024-04-25T11:33:10.897745+0000","last_active":"2024-04-25T11:33:11.886809+0000","last_peered":"2024-04-25T11:33:11.886809+0000","last_clean":"2024-04-25T11:33:11.886809+0000","last_became_active":"2024-04-25T11:33:10.897403+0000","last_became_peered":"2024-04-25T11:33:10.897403+0000","last_unstale":"2024-04-25T11:33:11.886809+0000","last_undegraded":"2024-04-25T11:33:11.886809+0000","last_fullsized":"2024-04-25T11:33:11.886809+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:33:34.682719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.885965+0000","last_change":"2024-04-25T11:33:10.887262+0000","last_active":"2024-04-25T11:33:11.885965+0000","last_peered":"2024-04-25T11:33:11.885965+0000","last_clean":"2024-04-25T11:33:11.885965+0000","last_became_active":"2024-04-25T11:33:10.886948+0000","last_became_peered":"2024-04-25T11:33:10.886948+0000","last_unstale":"2024-04-25T11:33:11.885965+0000","last_undegraded":"2024-04-25T11:33:11.885965+0000","last_fullsized":"2024-04-25T11:33:11.885965+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:09.721506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886224+0000","last_change":"2024-04-25T11:33:10.898392+0000","last_active":"2024-04-25T11:33:11.886224+0000","last_peered":"2024-04-25T11:33:11.886224+0000","last_clean":"2024-04-25T11:33:11.886224+0000","last_became_active":"2024-04-25T11:33:10.897935+0000","last_became_peered":"2024-04-25T11:33:10.897935+0000","last_unstale":"2024-04-25T11:33:11.886224+0000","last_undegraded":"2024-04-25T11:33:11.886224+0000","last_fullsized":"2024-04-25T11:33:11.886224+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:38:42.885463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887117+0000","last_change":"2024-04-25T11:33:10.889812+0000","last_active":"2024-04-25T11:33:11.887117+0000","last_peered":"2024-04-25T11:33:11.887117+0000","last_clean":"2024-04-25T11:33:11.887117+0000","last_became_active":"2024-04-25T11:33:10.889369+0000","last_became_peered":"2024-04-25T11:33:10.889369+0000","last_unstale":"2024-04-25T11:33:11.887117+0000","last_undegraded":"2024-04-25T11:33:11.887117+0000","last_fullsized":"2024-04-25T11:33:11.887117+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:42:14.443496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887365+0000","last_change":"2024-04-25T11:33:10.891186+0000","last_active":"2024-04-25T11:33:11.887365+0000","last_peered":"2024-04-25T11:33:11.887365+0000","last_clean":"2024-04-25T11:33:11.887365+0000","last_became_active":"2024-04-25T11:33:10.890841+0000","last_became_peered":"2024-04-25T11:33:10.890841+0000","last_unstale":"2024-04-25T11:33:11.887365+0000","last_undegraded":"2024-04-25T11:33:11.887365+0000","last_fullsized":"2024-04-25T11:33:11.887365+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:24:56.078592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.871140+0000","last_change":"2024-04-25T11:33:10.888801+0000","last_active":"2024-04-25T11:33:11.871140+0000","last_peered":"2024-04-25T11:33:11.871140+0000","last_clean":"2024-04-25T11:33:11.871140+0000","last_became_active":"2024-04-25T11:33:10.888437+0000","last_became_peered":"2024-04-25T11:33:10.888437+0000","last_unstale":"2024-04-25T11:33:11.871140+0000","last_undegraded":"2024-04-25T11:33:11.871140+0000","last_fullsized":"2024-04-25T11:33:11.871140+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:14:14.011475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.869640+0000","last_change":"2024-04-25T11:33:10.892749+0000","last_active":"2024-04-25T11:33:11.869640+0000","last_peered":"2024-04-25T11:33:11.869640+0000","last_clean":"2024-04-25T11:33:11.869640+0000","last_became_active":"2024-04-25T11:33:10.892370+0000","last_became_peered":"2024-04-25T11:33:10.892370+0000","last_unstale":"2024-04-25T11:33:11.869640+0000","last_undegraded":"2024-04-25T11:33:11.869640+0000","last_fullsized":"2024-04-25T11:33:11.869640+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:00:51.312387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.937776+0000","last_change":"2024-04-25T11:33:10.888981+0000","last_active":"2024-04-25T11:33:11.937776+0000","last_peered":"2024-04-25T11:33:11.937776+0000","last_clean":"2024-04-25T11:33:11.937776+0000","last_became_active":"2024-04-25T11:33:10.888537+0000","last_became_peered":"2024-04-25T11:33:10.888537+0000","last_unstale":"2024-04-25T11:33:11.937776+0000","last_undegraded":"2024-04-25T11:33:11.937776+0000","last_fullsized":"2024-04-25T11:33:11.937776+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:12:19.018698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.937482+0000","last_change":"2024-04-25T11:33:10.891455+0000","last_active":"2024-04-25T11:33:11.937482+0000","last_peered":"2024-04-25T11:33:11.937482+0000","last_clean":"2024-04-25T11:33:11.937482+0000","last_became_active":"2024-04-25T11:33:10.891037+0000","last_became_peered":"2024-04-25T11:33:10.891037+0000","last_unstale":"2024-04-25T11:33:11.937482+0000","last_undegraded":"2024-04-25T11:33:11.937482+0000","last_fullsized":"2024-04-25T11:33:11.937482+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:44.470674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.885677+0000","last_change":"2024-04-25T11:33:10.895309+0000","last_active":"2024-04-25T11:33:11.885677+0000","last_peered":"2024-04-25T11:33:11.885677+0000","last_clean":"2024-04-25T11:33:11.885677+0000","last_became_active":"2024-04-25T11:33:10.894952+0000","last_became_peered":"2024-04-25T11:33:10.894952+0000","last_unstale":"2024-04-25T11:33:11.885677+0000","last_undegraded":"2024-04-25T11:33:11.885677+0000","last_fullsized":"2024-04-25T11:33:11.885677+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:29:22.119017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887307+0000","last_change":"2024-04-25T11:33:10.894524+0000","last_active":"2024-04-25T11:33:11.887307+0000","last_peered":"2024-04-25T11:33:11.887307+0000","last_clean":"2024-04-25T11:33:11.887307+0000","last_became_active":"2024-04-25T11:33:10.894220+0000","last_became_peered":"2024-04-25T11:33:10.894220+0000","last_unstale":"2024-04-25T11:33:11.887307+0000","last_undegraded":"2024-04-25T11:33:11.887307+0000","last_fullsized":"2024-04-25T11:33:11.887307+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:10:29.072976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886343+0000","last_change":"2024-04-25T11:33:10.886814+0000","last_active":"2024-04-25T11:33:11.886343+0000","last_peered":"2024-04-25T11:33:11.886343+0000","last_clean":"2024-04-25T11:33:11.886343+0000","last_became_active":"2024-04-25T11:33:10.886494+0000","last_became_peered":"2024-04-25T11:33:10.886494+0000","last_unstale":"2024-04-25T11:33:11.886343+0000","last_undegraded":"2024-04-25T11:33:11.886343+0000","last_fullsized":"2024-04-25T11:33:11.886343+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:03:43.054485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886362+0000","last_change":"2024-04-25T11:33:10.886870+0000","last_active":"2024-04-25T11:33:11.886362+0000","last_peered":"2024-04-25T11:33:11.886362+0000","last_clean":"2024-04-25T11:33:11.886362+0000","last_became_active":"2024-04-25T11:33:10.886507+0000","last_became_peered":"2024-04-25T11:33:10.886507+0000","last_unstale":"2024-04-25T11:33:11.886362+0000","last_undegraded":"2024-04-25T11:33:11.886362+0000","last_fullsized":"2024-04-25T11:33:11.886362+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:50.101760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"76'4","reported_seq":19,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886345+0000","last_change":"2024-04-25T11:33:10.028970+0000","last_active":"2024-04-25T11:33:11.886345+0000","last_peered":"2024-04-25T11:33:11.886345+0000","last_clean":"2024-04-25T11:33:11.886345+0000","last_became_active":"2024-04-25T11:33:10.028636+0000","last_became_peered":"2024-04-25T11:33:10.028636+0000","last_unstale":"2024-04-25T11:33:11.886345+0000","last_undegraded":"2024-04-25T11:33:11.886345+0000","last_fullsized":"2024-04-25T11:33:11.886345+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:02.414903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886707+0000","last_change":"2024-04-25T11:33:10.882539+0000","last_active":"2024-04-25T11:33:11.886707+0000","last_peered":"2024-04-25T11:33:11.886707+0000","last_clean":"2024-04-25T11:33:11.886707+0000","last_became_active":"2024-04-25T11:33:10.882142+0000","last_became_peered":"2024-04-25T11:33:10.882142+0000","last_unstale":"2024-04-25T11:33:11.886707+0000","last_undegraded":"2024-04-25T11:33:11.886707+0000","last_fullsized":"2024-04-25T11:33:11.886707+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:55:32.959589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"77'2","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886742+0000","last_change":"2024-04-25T11:33:10.036958+0000","last_active":"2024-04-25T11:33:11.886742+0000","last_peered":"2024-04-25T11:33:11.886742+0000","last_clean":"2024-04-25T11:33:11.886742+0000","last_became_active":"2024-04-25T11:33:10.036606+0000","last_became_peered":"2024-04-25T11:33:10.036606+0000","last_unstale":"2024-04-25T11:33:11.886742+0000","last_undegraded":"2024-04-25T11:33:11.886742+0000","last_fullsized":"2024-04-25T11:33:11.886742+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:12.660208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.937901+0000","last_change":"2024-04-25T11:33:10.887501+0000","last_active":"2024-04-25T11:33:11.937901+0000","last_peered":"2024-04-25T11:33:11.937901+0000","last_clean":"2024-04-25T11:33:11.937901+0000","last_became_active":"2024-04-25T11:33:10.885985+0000","last_became_peered":"2024-04-25T11:33:10.885985+0000","last_unstale":"2024-04-25T11:33:11.937901+0000","last_undegraded":"2024-04-25T11:33:11.937901+0000","last_fullsized":"2024-04-25T11:33:11.937901+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:31:32.035646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"76'2","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887252+0000","last_change":"2024-04-25T11:33:10.031161+0000","last_active":"2024-04-25T11:33:11.887252+0000","last_peered":"2024-04-25T11:33:11.887252+0000","last_clean":"2024-04-25T11:33:11.887252+0000","last_became_active":"2024-04-25T11:33:10.028891+0000","last_became_peered":"2024-04-25T11:33:10.028891+0000","last_unstale":"2024-04-25T11:33:11.887252+0000","last_undegraded":"2024-04-25T11:33:11.887252+0000","last_fullsized":"2024-04-25T11:33:11.887252+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:25.959699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.885937+0000","last_change":"2024-04-25T11:33:10.889439+0000","last_active":"2024-04-25T11:33:11.885937+0000","last_peered":"2024-04-25T11:33:11.885937+0000","last_clean":"2024-04-25T11:33:11.885937+0000","last_became_active":"2024-04-25T11:33:10.889082+0000","last_became_peered":"2024-04-25T11:33:10.889082+0000","last_unstale":"2024-04-25T11:33:11.885937+0000","last_undegraded":"2024-04-25T11:33:11.885937+0000","last_fullsized":"2024-04-25T11:33:11.885937+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:13:48.958179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"76'2","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886707+0000","last_change":"2024-04-25T11:33:10.030793+0000","last_active":"2024-04-25T11:33:11.886707+0000","last_peered":"2024-04-25T11:33:11.886707+0000","last_clean":"2024-04-25T11:33:11.886707+0000","last_became_active":"2024-04-25T11:33:10.028805+0000","last_became_peered":"2024-04-25T11:33:10.028805+0000","last_unstale":"2024-04-25T11:33:11.886707+0000","last_undegraded":"2024-04-25T11:33:11.886707+0000","last_fullsized":"2024-04-25T11:33:11.886707+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:36.233194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.871391+0000","last_change":"2024-04-25T11:33:10.894815+0000","last_active":"2024-04-25T11:33:11.871391+0000","last_peered":"2024-04-25T11:33:11.871391+0000","last_clean":"2024-04-25T11:33:11.871391+0000","last_became_active":"2024-04-25T11:33:10.892174+0000","last_became_peered":"2024-04-25T11:33:10.892174+0000","last_unstale":"2024-04-25T11:33:11.871391+0000","last_undegraded":"2024-04-25T11:33:11.871391+0000","last_fullsized":"2024-04-25T11:33:11.871391+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:05.368441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"77'5","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.616398+0000","last_change":"2024-04-25T11:33:10.031238+0000","last_active":"2024-04-25T11:33:13.616398+0000","last_peered":"2024-04-25T11:33:13.616398+0000","last_clean":"2024-04-25T11:33:13.616398+0000","last_became_active":"2024-04-25T11:33:10.030917+0000","last_became_peered":"2024-04-25T11:33:10.030917+0000","last_unstale":"2024-04-25T11:33:13.616398+0000","last_undegraded":"2024-04-25T11:33:13.616398+0000","last_fullsized":"2024-04-25T11:33:13.616398+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:00:35.634293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887041+0000","last_change":"2024-04-25T11:33:10.896974+0000","last_active":"2024-04-25T11:33:11.887041+0000","last_peered":"2024-04-25T11:33:11.887041+0000","last_clean":"2024-04-25T11:33:11.887041+0000","last_became_active":"2024-04-25T11:33:10.896575+0000","last_became_peered":"2024-04-25T11:33:10.896575+0000","last_unstale":"2024-04-25T11:33:11.887041+0000","last_undegraded":"2024-04-25T11:33:11.887041+0000","last_fullsized":"2024-04-25T11:33:11.887041+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:32:43.533652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"77'3","reported_seq":18,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.628190+0000","last_change":"2024-04-25T11:33:10.022714+0000","last_active":"2024-04-25T11:33:13.628190+0000","last_peered":"2024-04-25T11:33:13.628190+0000","last_clean":"2024-04-25T11:33:13.628190+0000","last_became_active":"2024-04-25T11:33:10.022230+0000","last_became_peered":"2024-04-25T11:33:10.022230+0000","last_unstale":"2024-04-25T11:33:13.628190+0000","last_undegraded":"2024-04-25T11:33:13.628190+0000","last_fullsized":"2024-04-25T11:33:13.628190+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:30:00.620833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.871077+0000","last_change":"2024-04-25T11:33:10.897664+0000","last_active":"2024-04-25T11:33:11.871077+0000","last_peered":"2024-04-25T11:33:11.871077+0000","last_clean":"2024-04-25T11:33:11.871077+0000","last_became_active":"2024-04-25T11:33:10.897189+0000","last_became_peered":"2024-04-25T11:33:10.897189+0000","last_unstale":"2024-04-25T11:33:11.871077+0000","last_undegraded":"2024-04-25T11:33:11.871077+0000","last_fullsized":"2024-04-25T11:33:11.871077+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:12:06.168793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"77'2","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.626198+0000","last_change":"2024-04-25T11:33:10.031417+0000","last_active":"2024-04-25T11:33:13.626198+0000","last_peered":"2024-04-25T11:33:13.626198+0000","last_clean":"2024-04-25T11:33:13.626198+0000","last_became_active":"2024-04-25T11:33:10.030526+0000","last_became_peered":"2024-04-25T11:33:10.030526+0000","last_unstale":"2024-04-25T11:33:13.626198+0000","last_undegraded":"2024-04-25T11:33:13.626198+0000","last_fullsized":"2024-04-25T11:33:13.626198+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:05.718720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886517+0000","last_change":"2024-04-25T11:33:10.886596+0000","last_active":"2024-04-25T11:33:11.886517+0000","last_peered":"2024-04-25T11:33:11.886517+0000","last_clean":"2024-04-25T11:33:11.886517+0000","last_became_active":"2024-04-25T11:33:10.886234+0000","last_became_peered":"2024-04-25T11:33:10.886234+0000","last_unstale":"2024-04-25T11:33:11.886517+0000","last_undegraded":"2024-04-25T11:33:11.886517+0000","last_fullsized":"2024-04-25T11:33:11.886517+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:46.632814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"77'1","reported_seq":16,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.870837+0000","last_change":"2024-04-25T11:33:10.032355+0000","last_active":"2024-04-25T11:33:11.870837+0000","last_peered":"2024-04-25T11:33:11.870837+0000","last_clean":"2024-04-25T11:33:11.870837+0000","last_became_active":"2024-04-25T11:33:10.031965+0000","last_became_peered":"2024-04-25T11:33:10.031965+0000","last_unstale":"2024-04-25T11:33:11.870837+0000","last_undegraded":"2024-04-25T11:33:11.870837+0000","last_fullsized":"2024-04-25T11:33:11.870837+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:08:10.539604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.885902+0000","last_change":"2024-04-25T11:33:10.885708+0000","last_active":"2024-04-25T11:33:11.885902+0000","last_peered":"2024-04-25T11:33:11.885902+0000","last_clean":"2024-04-25T11:33:11.885902+0000","last_became_active":"2024-04-25T11:33:10.885317+0000","last_became_peered":"2024-04-25T11:33:10.885317+0000","last_unstale":"2024-04-25T11:33:11.885902+0000","last_undegraded":"2024-04-25T11:33:11.885902+0000","last_fullsized":"2024-04-25T11:33:11.885902+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:50:08.518922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"77'3","reported_seq":19,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.615808+0000","last_change":"2024-04-25T11:33:10.030995+0000","last_active":"2024-04-25T11:33:13.615808+0000","last_peered":"2024-04-25T11:33:13.615808+0000","last_clean":"2024-04-25T11:33:13.615808+0000","last_became_active":"2024-04-25T11:33:10.030670+0000","last_became_peered":"2024-04-25T11:33:10.030670+0000","last_unstale":"2024-04-25T11:33:13.615808+0000","last_undegraded":"2024-04-25T11:33:13.615808+0000","last_fullsized":"2024-04-25T11:33:13.615808+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:39.569787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887554+0000","last_change":"2024-04-25T11:33:10.892927+0000","last_active":"2024-04-25T11:33:11.887554+0000","last_peered":"2024-04-25T11:33:11.887554+0000","last_clean":"2024-04-25T11:33:11.887554+0000","last_became_active":"2024-04-25T11:33:10.892458+0000","last_became_peered":"2024-04-25T11:33:10.892458+0000","last_unstale":"2024-04-25T11:33:11.887554+0000","last_undegraded":"2024-04-25T11:33:11.887554+0000","last_fullsized":"2024-04-25T11:33:11.887554+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:53.121185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"77'2","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.045429+0000","last_change":"2024-04-25T11:33:10.029728+0000","last_active":"2024-04-25T11:33:13.045429+0000","last_peered":"2024-04-25T11:33:13.045429+0000","last_clean":"2024-04-25T11:33:13.045429+0000","last_became_active":"2024-04-25T11:33:10.029435+0000","last_became_peered":"2024-04-25T11:33:10.029435+0000","last_unstale":"2024-04-25T11:33:13.045429+0000","last_undegraded":"2024-04-25T11:33:13.045429+0000","last_fullsized":"2024-04-25T11:33:13.045429+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:31:21.585870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.937825+0000","last_change":"2024-04-25T11:33:10.895828+0000","last_active":"2024-04-25T11:33:11.937825+0000","last_peered":"2024-04-25T11:33:11.937825+0000","last_clean":"2024-04-25T11:33:11.937825+0000","last_became_active":"2024-04-25T11:33:10.895280+0000","last_became_peered":"2024-04-25T11:33:10.895280+0000","last_unstale":"2024-04-25T11:33:11.937825+0000","last_undegraded":"2024-04-25T11:33:11.937825+0000","last_fullsized":"2024-04-25T11:33:11.937825+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:03.009912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"77'4","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.043498+0000","last_change":"2024-04-25T11:33:10.036651+0000","last_active":"2024-04-25T11:33:13.043498+0000","last_peered":"2024-04-25T11:33:13.043498+0000","last_clean":"2024-04-25T11:33:13.043498+0000","last_became_active":"2024-04-25T11:33:10.036294+0000","last_became_peered":"2024-04-25T11:33:10.036294+0000","last_unstale":"2024-04-25T11:33:13.043498+0000","last_undegraded":"2024-04-25T11:33:13.043498+0000","last_fullsized":"2024-04-25T11:33:13.043498+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:12:36.566618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.885967+0000","last_change":"2024-04-25T11:33:10.890199+0000","last_active":"2024-04-25T11:33:11.885967+0000","last_peered":"2024-04-25T11:33:11.885967+0000","last_clean":"2024-04-25T11:33:11.885967+0000","last_became_active":"2024-04-25T11:33:10.889855+0000","last_became_peered":"2024-04-25T11:33:10.889855+0000","last_unstale":"2024-04-25T11:33:11.885967+0000","last_undegraded":"2024-04-25T11:33:11.885967+0000","last_fullsized":"2024-04-25T11:33:11.885967+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:54:45.476647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"77'8","reported_seq":27,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.620848+0000","last_change":"2024-04-25T11:33:10.027364+0000","last_active":"2024-04-25T11:33:13.620848+0000","last_peered":"2024-04-25T11:33:13.620848+0000","last_clean":"2024-04-25T11:33:13.620848+0000","last_became_active":"2024-04-25T11:33:10.026955+0000","last_became_peered":"2024-04-25T11:33:10.026955+0000","last_unstale":"2024-04-25T11:33:13.620848+0000","last_undegraded":"2024-04-25T11:33:13.620848+0000","last_fullsized":"2024-04-25T11:33:13.620848+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:23:46.330813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886920+0000","last_change":"2024-04-25T11:33:10.888596+0000","last_active":"2024-04-25T11:33:11.886920+0000","last_peered":"2024-04-25T11:33:11.886920+0000","last_clean":"2024-04-25T11:33:11.886920+0000","last_became_active":"2024-04-25T11:33:10.888226+0000","last_became_peered":"2024-04-25T11:33:10.888226+0000","last_unstale":"2024-04-25T11:33:11.886920+0000","last_undegraded":"2024-04-25T11:33:11.886920+0000","last_fullsized":"2024-04-25T11:33:11.886920+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:30:42.891788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"77'9","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:15.646566+0000","last_change":"2024-04-25T11:33:10.026969+0000","last_active":"2024-04-25T11:33:15.646566+0000","last_peered":"2024-04-25T11:33:15.646566+0000","last_clean":"2024-04-25T11:33:15.646566+0000","last_became_active":"2024-04-25T11:33:10.026591+0000","last_became_peered":"2024-04-25T11:33:10.026591+0000","last_unstale":"2024-04-25T11:33:15.646566+0000","last_undegraded":"2024-04-25T11:33:15.646566+0000","last_fullsized":"2024-04-25T11:33:15.646566+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:50:34.269478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.869472+0000","last_change":"2024-04-25T11:33:10.900303+0000","last_active":"2024-04-25T11:33:11.869472+0000","last_peered":"2024-04-25T11:33:11.869472+0000","last_clean":"2024-04-25T11:33:11.869472+0000","last_became_active":"2024-04-25T11:33:10.899946+0000","last_became_peered":"2024-04-25T11:33:10.899946+0000","last_unstale":"2024-04-25T11:33:11.869472+0000","last_undegraded":"2024-04-25T11:33:11.869472+0000","last_fullsized":"2024-04-25T11:33:11.869472+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:33:40.805400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"77'3","reported_seq":18,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.044412+0000","last_change":"2024-04-25T11:33:10.035668+0000","last_active":"2024-04-25T11:33:13.044412+0000","last_peered":"2024-04-25T11:33:13.044412+0000","last_clean":"2024-04-25T11:33:13.044412+0000","last_became_active":"2024-04-25T11:33:10.035241+0000","last_became_peered":"2024-04-25T11:33:10.035241+0000","last_unstale":"2024-04-25T11:33:13.044412+0000","last_undegraded":"2024-04-25T11:33:13.044412+0000","last_fullsized":"2024-04-25T11:33:13.044412+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:58:48.735577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887293+0000","last_change":"2024-04-25T11:33:10.894864+0000","last_active":"2024-04-25T11:33:11.887293+0000","last_peered":"2024-04-25T11:33:11.887293+0000","last_clean":"2024-04-25T11:33:11.887293+0000","last_became_active":"2024-04-25T11:33:10.894491+0000","last_became_peered":"2024-04-25T11:33:10.894491+0000","last_unstale":"2024-04-25T11:33:11.887293+0000","last_undegraded":"2024-04-25T11:33:11.887293+0000","last_fullsized":"2024-04-25T11:33:11.887293+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:21:28.256878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"77'6","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:14.630764+0000","last_change":"2024-04-25T11:33:10.029099+0000","last_active":"2024-04-25T11:33:14.630764+0000","last_peered":"2024-04-25T11:33:14.630764+0000","last_clean":"2024-04-25T11:33:14.630764+0000","last_became_active":"2024-04-25T11:33:10.028722+0000","last_became_peered":"2024-04-25T11:33:10.028722+0000","last_unstale":"2024-04-25T11:33:14.630764+0000","last_undegraded":"2024-04-25T11:33:14.630764+0000","last_fullsized":"2024-04-25T11:33:14.630764+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:44:12.261519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886572+0000","last_change":"2024-04-25T11:33:10.888635+0000","last_active":"2024-04-25T11:33:11.886572+0000","last_peered":"2024-04-25T11:33:11.886572+0000","last_clean":"2024-04-25T11:33:11.886572+0000","last_became_active":"2024-04-25T11:33:10.887544+0000","last_became_peered":"2024-04-25T11:33:10.887544+0000","last_unstale":"2024-04-25T11:33:11.886572+0000","last_undegraded":"2024-04-25T11:33:11.886572+0000","last_fullsized":"2024-04-25T11:33:11.886572+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:59:25.459587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"77'10","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.042745+0000","last_change":"2024-04-25T11:33:10.023566+0000","last_active":"2024-04-25T11:33:13.042745+0000","last_peered":"2024-04-25T11:33:13.042745+0000","last_clean":"2024-04-25T11:33:13.042745+0000","last_became_active":"2024-04-25T11:33:10.023234+0000","last_became_peered":"2024-04-25T11:33:10.023234+0000","last_unstale":"2024-04-25T11:33:13.042745+0000","last_undegraded":"2024-04-25T11:33:13.042745+0000","last_fullsized":"2024-04-25T11:33:13.042745+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:56:26.342148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.870905+0000","last_change":"2024-04-25T11:33:10.895866+0000","last_active":"2024-04-25T11:33:11.870905+0000","last_peered":"2024-04-25T11:33:11.870905+0000","last_clean":"2024-04-25T11:33:11.870905+0000","last_became_active":"2024-04-25T11:33:10.895455+0000","last_became_peered":"2024-04-25T11:33:10.895455+0000","last_unstale":"2024-04-25T11:33:11.870905+0000","last_undegraded":"2024-04-25T11:33:11.870905+0000","last_fullsized":"2024-04-25T11:33:11.870905+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:30:25.380439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"76'1","reported_seq":16,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887276+0000","last_change":"2024-04-25T11:33:10.028285+0000","last_active":"2024-04-25T11:33:11.887276+0000","last_peered":"2024-04-25T11:33:11.887276+0000","last_clean":"2024-04-25T11:33:11.887276+0000","last_became_active":"2024-04-25T11:33:10.027959+0000","last_became_peered":"2024-04-25T11:33:10.027959+0000","last_unstale":"2024-04-25T11:33:11.887276+0000","last_undegraded":"2024-04-25T11:33:11.887276+0000","last_fullsized":"2024-04-25T11:33:11.887276+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:44:41.094048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886892+0000","last_change":"2024-04-25T11:33:10.890123+0000","last_active":"2024-04-25T11:33:11.886892+0000","last_peered":"2024-04-25T11:33:11.886892+0000","last_clean":"2024-04-25T11:33:11.886892+0000","last_became_active":"2024-04-25T11:33:10.889746+0000","last_became_peered":"2024-04-25T11:33:10.889746+0000","last_unstale":"2024-04-25T11:33:11.886892+0000","last_undegraded":"2024-04-25T11:33:11.886892+0000","last_fullsized":"2024-04-25T11:33:11.886892+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:25.650151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.885908+0000","last_change":"2024-04-25T11:33:10.027473+0000","last_active":"2024-04-25T11:33:11.885908+0000","last_peered":"2024-04-25T11:33:11.885908+0000","last_clean":"2024-04-25T11:33:11.885908+0000","last_became_active":"2024-04-25T11:33:10.027133+0000","last_became_peered":"2024-04-25T11:33:10.027133+0000","last_unstale":"2024-04-25T11:33:11.885908+0000","last_undegraded":"2024-04-25T11:33:11.885908+0000","last_fullsized":"2024-04-25T11:33:11.885908+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:19:58.338507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886920+0000","last_change":"2024-04-25T11:33:10.892588+0000","last_active":"2024-04-25T11:33:11.886920+0000","last_peered":"2024-04-25T11:33:11.886920+0000","last_clean":"2024-04-25T11:33:11.886920+0000","last_became_active":"2024-04-25T11:33:10.892088+0000","last_became_peered":"2024-04-25T11:33:10.892088+0000","last_unstale":"2024-04-25T11:33:11.886920+0000","last_undegraded":"2024-04-25T11:33:11.886920+0000","last_fullsized":"2024-04-25T11:33:11.886920+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:10.136222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"77'1","reported_seq":16,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.625741+0000","last_change":"2024-04-25T11:33:10.030801+0000","last_active":"2024-04-25T11:33:13.625741+0000","last_peered":"2024-04-25T11:33:13.625741+0000","last_clean":"2024-04-25T11:33:13.625741+0000","last_became_active":"2024-04-25T11:33:10.030466+0000","last_became_peered":"2024-04-25T11:33:10.030466+0000","last_unstale":"2024-04-25T11:33:13.625741+0000","last_undegraded":"2024-04-25T11:33:13.625741+0000","last_fullsized":"2024-04-25T11:33:13.625741+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:22:02.922309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886714+0000","last_change":"2024-04-25T11:33:10.894846+0000","last_active":"2024-04-25T11:33:11.886714+0000","last_peered":"2024-04-25T11:33:11.886714+0000","last_clean":"2024-04-25T11:33:11.886714+0000","last_became_active":"2024-04-25T11:33:10.894456+0000","last_became_peered":"2024-04-25T11:33:10.894456+0000","last_unstale":"2024-04-25T11:33:11.886714+0000","last_undegraded":"2024-04-25T11:33:11.886714+0000","last_fullsized":"2024-04-25T11:33:11.886714+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:11:16.265126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.871114+0000","last_change":"2024-04-25T11:33:10.891813+0000","last_active":"2024-04-25T11:33:11.871114+0000","last_peered":"2024-04-25T11:33:11.871114+0000","last_clean":"2024-04-25T11:33:11.871114+0000","last_became_active":"2024-04-25T11:33:10.887249+0000","last_became_peered":"2024-04-25T11:33:10.887249+0000","last_unstale":"2024-04-25T11:33:11.871114+0000","last_undegraded":"2024-04-25T11:33:11.871114+0000","last_fullsized":"2024-04-25T11:33:11.871114+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:10:10.895726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"77'4","reported_seq":19,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.627271+0000","last_change":"2024-04-25T11:33:10.037834+0000","last_active":"2024-04-25T11:33:13.627271+0000","last_peered":"2024-04-25T11:33:13.627271+0000","last_clean":"2024-04-25T11:33:13.627271+0000","last_became_active":"2024-04-25T11:33:10.037570+0000","last_became_peered":"2024-04-25T11:33:10.037570+0000","last_unstale":"2024-04-25T11:33:13.627271+0000","last_undegraded":"2024-04-25T11:33:13.627271+0000","last_fullsized":"2024-04-25T11:33:13.627271+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:12.795190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886167+0000","last_change":"2024-04-25T11:33:10.882663+0000","last_active":"2024-04-25T11:33:11.886167+0000","last_peered":"2024-04-25T11:33:11.886167+0000","last_clean":"2024-04-25T11:33:11.886167+0000","last_became_active":"2024-04-25T11:33:10.882300+0000","last_became_peered":"2024-04-25T11:33:10.882300+0000","last_unstale":"2024-04-25T11:33:11.886167+0000","last_undegraded":"2024-04-25T11:33:11.886167+0000","last_fullsized":"2024-04-25T11:33:11.886167+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:58:11.379654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"77'5","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:14.629891+0000","last_change":"2024-04-25T11:33:10.024598+0000","last_active":"2024-04-25T11:33:14.629891+0000","last_peered":"2024-04-25T11:33:14.629891+0000","last_clean":"2024-04-25T11:33:14.629891+0000","last_became_active":"2024-04-25T11:33:10.024229+0000","last_became_peered":"2024-04-25T11:33:10.024229+0000","last_unstale":"2024-04-25T11:33:14.629891+0000","last_undegraded":"2024-04-25T11:33:14.629891+0000","last_fullsized":"2024-04-25T11:33:14.629891+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:42:14.690581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":91,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886242+0000","last_change":"2024-04-25T11:32:01.310202+0000","last_active":"2024-04-25T11:33:11.886242+0000","last_peered":"2024-04-25T11:33:11.886242+0000","last_clean":"2024-04-25T11:33:11.886242+0000","last_became_active":"2024-04-25T11:32:01.305242+0000","last_became_peered":"2024-04-25T11:32:01.305242+0000","last_unstale":"2024-04-25T11:33:11.886242+0000","last_undegraded":"2024-04-25T11:33:11.886242+0000","last_fullsized":"2024-04-25T11:33:11.886242+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:28:43.374096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:28:43.374096+0000","last_clean_scrub_stamp":"2024-04-25T11:28:43.374096+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:56:18.551747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.869641+0000","last_change":"2024-04-25T11:33:10.889124+0000","last_active":"2024-04-25T11:33:11.869641+0000","last_peered":"2024-04-25T11:33:11.869641+0000","last_clean":"2024-04-25T11:33:11.869641+0000","last_became_active":"2024-04-25T11:33:10.888790+0000","last_became_peered":"2024-04-25T11:33:10.888790+0000","last_unstale":"2024-04-25T11:33:11.869641+0000","last_undegraded":"2024-04-25T11:33:11.869641+0000","last_fullsized":"2024-04-25T11:33:11.869641+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:33:44.999949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"77'3","reported_seq":18,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.628943+0000","last_change":"2024-04-25T11:33:10.031547+0000","last_active":"2024-04-25T11:33:13.628943+0000","last_peered":"2024-04-25T11:33:13.628943+0000","last_clean":"2024-04-25T11:33:13.628943+0000","last_became_active":"2024-04-25T11:33:10.031152+0000","last_became_peered":"2024-04-25T11:33:10.031152+0000","last_unstale":"2024-04-25T11:33:13.628943+0000","last_undegraded":"2024-04-25T11:33:13.628943+0000","last_fullsized":"2024-04-25T11:33:13.628943+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:10.646771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"77'1","reported_seq":16,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.042961+0000","last_change":"2024-04-25T11:33:10.020367+0000","last_active":"2024-04-25T11:33:13.042961+0000","last_peered":"2024-04-25T11:33:13.042961+0000","last_clean":"2024-04-25T11:33:13.042961+0000","last_became_active":"2024-04-25T11:33:10.020007+0000","last_became_peered":"2024-04-25T11:33:10.020007+0000","last_unstale":"2024-04-25T11:33:13.042961+0000","last_undegraded":"2024-04-25T11:33:13.042961+0000","last_fullsized":"2024-04-25T11:33:13.042961+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:48.134907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.869849+0000","last_change":"2024-04-25T11:33:10.887741+0000","last_active":"2024-04-25T11:33:11.869849+0000","last_peered":"2024-04-25T11:33:11.869849+0000","last_clean":"2024-04-25T11:33:11.869849+0000","last_became_active":"2024-04-25T11:33:10.886455+0000","last_became_peered":"2024-04-25T11:33:10.886455+0000","last_unstale":"2024-04-25T11:33:11.869849+0000","last_undegraded":"2024-04-25T11:33:11.869849+0000","last_fullsized":"2024-04-25T11:33:11.869849+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:37:16.842817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":16,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.022519+0000","last_change":"2024-04-25T11:33:10.025672+0000","last_active":"2024-04-25T11:33:13.022519+0000","last_peered":"2024-04-25T11:33:13.022519+0000","last_clean":"2024-04-25T11:33:13.022519+0000","last_became_active":"2024-04-25T11:33:10.025232+0000","last_became_peered":"2024-04-25T11:33:10.025232+0000","last_unstale":"2024-04-25T11:33:13.022519+0000","last_undegraded":"2024-04-25T11:33:13.022519+0000","last_fullsized":"2024-04-25T11:33:13.022519+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:06:26.579126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886307+0000","last_change":"2024-04-25T11:33:10.884692+0000","last_active":"2024-04-25T11:33:11.886307+0000","last_peered":"2024-04-25T11:33:11.886307+0000","last_clean":"2024-04-25T11:33:11.886307+0000","last_became_active":"2024-04-25T11:33:10.884346+0000","last_became_peered":"2024-04-25T11:33:10.884346+0000","last_unstale":"2024-04-25T11:33:11.886307+0000","last_undegraded":"2024-04-25T11:33:11.886307+0000","last_fullsized":"2024-04-25T11:33:11.886307+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:31:00.630410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"77'5","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.043236+0000","last_change":"2024-04-25T11:33:10.033872+0000","last_active":"2024-04-25T11:33:13.043236+0000","last_peered":"2024-04-25T11:33:13.043236+0000","last_clean":"2024-04-25T11:33:13.043236+0000","last_became_active":"2024-04-25T11:33:10.033435+0000","last_became_peered":"2024-04-25T11:33:10.033435+0000","last_unstale":"2024-04-25T11:33:13.043236+0000","last_undegraded":"2024-04-25T11:33:13.043236+0000","last_fullsized":"2024-04-25T11:33:13.043236+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:12.885215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.937704+0000","last_change":"2024-04-25T11:33:10.888707+0000","last_active":"2024-04-25T11:33:11.937704+0000","last_peered":"2024-04-25T11:33:11.937704+0000","last_clean":"2024-04-25T11:33:11.937704+0000","last_became_active":"2024-04-25T11:33:10.888274+0000","last_became_peered":"2024-04-25T11:33:10.888274+0000","last_unstale":"2024-04-25T11:33:11.937704+0000","last_undegraded":"2024-04-25T11:33:11.937704+0000","last_fullsized":"2024-04-25T11:33:11.937704+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:23:36.577447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"77'2","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.043069+0000","last_change":"2024-04-25T11:33:10.032384+0000","last_active":"2024-04-25T11:33:13.043069+0000","last_peered":"2024-04-25T11:33:13.043069+0000","last_clean":"2024-04-25T11:33:13.043069+0000","last_became_active":"2024-04-25T11:33:10.032035+0000","last_became_peered":"2024-04-25T11:33:10.032035+0000","last_unstale":"2024-04-25T11:33:13.043069+0000","last_undegraded":"2024-04-25T11:33:13.043069+0000","last_fullsized":"2024-04-25T11:33:13.043069+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:21:41.723120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.885748+0000","last_change":"2024-04-25T11:33:10.893366+0000","last_active":"2024-04-25T11:33:11.885748+0000","last_peered":"2024-04-25T11:33:11.885748+0000","last_clean":"2024-04-25T11:33:11.885748+0000","last_became_active":"2024-04-25T11:33:10.892988+0000","last_became_peered":"2024-04-25T11:33:10.892988+0000","last_unstale":"2024-04-25T11:33:11.885748+0000","last_undegraded":"2024-04-25T11:33:11.885748+0000","last_fullsized":"2024-04-25T11:33:11.885748+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:06.368216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"77'5","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:14.630917+0000","last_change":"2024-04-25T11:33:10.031456+0000","last_active":"2024-04-25T11:33:14.630917+0000","last_peered":"2024-04-25T11:33:14.630917+0000","last_clean":"2024-04-25T11:33:14.630917+0000","last_became_active":"2024-04-25T11:33:10.031115+0000","last_became_peered":"2024-04-25T11:33:10.031115+0000","last_unstale":"2024-04-25T11:33:14.630917+0000","last_undegraded":"2024-04-25T11:33:14.630917+0000","last_fullsized":"2024-04-25T11:33:14.630917+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:12:56.803277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887637+0000","last_change":"2024-04-25T11:33:10.893692+0000","last_active":"2024-04-25T11:33:11.887637+0000","last_peered":"2024-04-25T11:33:11.887637+0000","last_clean":"2024-04-25T11:33:11.887637+0000","last_became_active":"2024-04-25T11:33:10.893332+0000","last_became_peered":"2024-04-25T11:33:10.893332+0000","last_unstale":"2024-04-25T11:33:11.887637+0000","last_undegraded":"2024-04-25T11:33:11.887637+0000","last_fullsized":"2024-04-25T11:33:11.887637+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:01.277465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"77'2","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:14.038897+0000","last_change":"2024-04-25T11:33:10.023804+0000","last_active":"2024-04-25T11:33:14.038897+0000","last_peered":"2024-04-25T11:33:14.038897+0000","last_clean":"2024-04-25T11:33:14.038897+0000","last_became_active":"2024-04-25T11:33:10.023503+0000","last_became_peered":"2024-04-25T11:33:10.023503+0000","last_unstale":"2024-04-25T11:33:14.038897+0000","last_undegraded":"2024-04-25T11:33:14.038897+0000","last_fullsized":"2024-04-25T11:33:14.038897+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:34:48.074176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886206+0000","last_change":"2024-04-25T11:33:10.881501+0000","last_active":"2024-04-25T11:33:11.886206+0000","last_peered":"2024-04-25T11:33:11.886206+0000","last_clean":"2024-04-25T11:33:11.886206+0000","last_became_active":"2024-04-25T11:33:10.881104+0000","last_became_peered":"2024-04-25T11:33:10.881104+0000","last_unstale":"2024-04-25T11:33:11.886206+0000","last_undegraded":"2024-04-25T11:33:11.886206+0000","last_fullsized":"2024-04-25T11:33:11.886206+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:23:01.405920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"77'4","reported_seq":19,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.045053+0000","last_change":"2024-04-25T11:33:10.030857+0000","last_active":"2024-04-25T11:33:13.045053+0000","last_peered":"2024-04-25T11:33:13.045053+0000","last_clean":"2024-04-25T11:33:13.045053+0000","last_became_active":"2024-04-25T11:33:10.028879+0000","last_became_peered":"2024-04-25T11:33:10.028879+0000","last_unstale":"2024-04-25T11:33:13.045053+0000","last_undegraded":"2024-04-25T11:33:13.045053+0000","last_fullsized":"2024-04-25T11:33:13.045053+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:28.820919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887187+0000","last_change":"2024-04-25T11:33:10.893773+0000","last_active":"2024-04-25T11:33:11.887187+0000","last_peered":"2024-04-25T11:33:11.887187+0000","last_clean":"2024-04-25T11:33:11.887187+0000","last_became_active":"2024-04-25T11:33:10.893431+0000","last_became_peered":"2024-04-25T11:33:10.893431+0000","last_unstale":"2024-04-25T11:33:11.887187+0000","last_undegraded":"2024-04-25T11:33:11.887187+0000","last_fullsized":"2024-04-25T11:33:11.887187+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:15:15.998951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"77'5","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:14.630901+0000","last_change":"2024-04-25T11:33:10.032745+0000","last_active":"2024-04-25T11:33:14.630901+0000","last_peered":"2024-04-25T11:33:14.630901+0000","last_clean":"2024-04-25T11:33:14.630901+0000","last_became_active":"2024-04-25T11:33:10.032361+0000","last_became_peered":"2024-04-25T11:33:10.032361+0000","last_unstale":"2024-04-25T11:33:14.630901+0000","last_undegraded":"2024-04-25T11:33:14.630901+0000","last_fullsized":"2024-04-25T11:33:14.630901+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:14:46.555892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887521+0000","last_change":"2024-04-25T11:33:10.888084+0000","last_active":"2024-04-25T11:33:11.887521+0000","last_peered":"2024-04-25T11:33:11.887521+0000","last_clean":"2024-04-25T11:33:11.887521+0000","last_became_active":"2024-04-25T11:33:10.887679+0000","last_became_peered":"2024-04-25T11:33:10.887679+0000","last_unstale":"2024-04-25T11:33:11.887521+0000","last_undegraded":"2024-04-25T11:33:11.887521+0000","last_fullsized":"2024-04-25T11:33:11.887521+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:10:20.073630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"76'1","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.871256+0000","last_change":"2024-04-25T11:33:10.035355+0000","last_active":"2024-04-25T11:33:11.871256+0000","last_peered":"2024-04-25T11:33:11.871256+0000","last_clean":"2024-04-25T11:33:11.871256+0000","last_became_active":"2024-04-25T11:33:10.034724+0000","last_became_peered":"2024-04-25T11:33:10.034724+0000","last_unstale":"2024-04-25T11:33:11.871256+0000","last_undegraded":"2024-04-25T11:33:11.871256+0000","last_fullsized":"2024-04-25T11:33:11.871256+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:24:26.757803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887125+0000","last_change":"2024-04-25T11:33:10.899827+0000","last_active":"2024-04-25T11:33:11.887125+0000","last_peered":"2024-04-25T11:33:11.887125+0000","last_clean":"2024-04-25T11:33:11.887125+0000","last_became_active":"2024-04-25T11:33:10.899356+0000","last_became_peered":"2024-04-25T11:33:10.899356+0000","last_unstale":"2024-04-25T11:33:11.887125+0000","last_undegraded":"2024-04-25T11:33:11.887125+0000","last_fullsized":"2024-04-25T11:33:11.887125+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:04:17.323958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"76'2","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.869597+0000","last_change":"2024-04-25T11:33:10.034981+0000","last_active":"2024-04-25T11:33:11.869597+0000","last_peered":"2024-04-25T11:33:11.869597+0000","last_clean":"2024-04-25T11:33:11.869597+0000","last_became_active":"2024-04-25T11:33:10.034601+0000","last_became_peered":"2024-04-25T11:33:10.034601+0000","last_unstale":"2024-04-25T11:33:11.869597+0000","last_undegraded":"2024-04-25T11:33:11.869597+0000","last_fullsized":"2024-04-25T11:33:11.869597+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:19:14.710731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887479+0000","last_change":"2024-04-25T11:33:10.890020+0000","last_active":"2024-04-25T11:33:11.887479+0000","last_peered":"2024-04-25T11:33:11.887479+0000","last_clean":"2024-04-25T11:33:11.887479+0000","last_became_active":"2024-04-25T11:33:10.889558+0000","last_became_peered":"2024-04-25T11:33:10.889558+0000","last_unstale":"2024-04-25T11:33:11.887479+0000","last_undegraded":"2024-04-25T11:33:11.887479+0000","last_fullsized":"2024-04-25T11:33:11.887479+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:00:49.096818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887769+0000","last_change":"2024-04-25T11:33:10.886965+0000","last_active":"2024-04-25T11:33:11.887769+0000","last_peered":"2024-04-25T11:33:11.887769+0000","last_clean":"2024-04-25T11:33:11.887769+0000","last_became_active":"2024-04-25T11:33:10.886594+0000","last_became_peered":"2024-04-25T11:33:10.886594+0000","last_unstale":"2024-04-25T11:33:11.887769+0000","last_undegraded":"2024-04-25T11:33:11.887769+0000","last_fullsized":"2024-04-25T11:33:11.887769+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:58.924857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886516+0000","last_change":"2024-04-25T11:33:10.890760+0000","last_active":"2024-04-25T11:33:11.886516+0000","last_peered":"2024-04-25T11:33:11.886516+0000","last_clean":"2024-04-25T11:33:11.886516+0000","last_became_active":"2024-04-25T11:33:10.889643+0000","last_became_peered":"2024-04-25T11:33:10.889643+0000","last_unstale":"2024-04-25T11:33:11.886516+0000","last_undegraded":"2024-04-25T11:33:11.886516+0000","last_fullsized":"2024-04-25T11:33:11.886516+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:54.644853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886800+0000","last_change":"2024-04-25T11:33:10.887850+0000","last_active":"2024-04-25T11:33:11.886800+0000","last_peered":"2024-04-25T11:33:11.886800+0000","last_clean":"2024-04-25T11:33:11.886800+0000","last_became_active":"2024-04-25T11:33:10.882323+0000","last_became_peered":"2024-04-25T11:33:10.882323+0000","last_unstale":"2024-04-25T11:33:11.886800+0000","last_undegraded":"2024-04-25T11:33:11.886800+0000","last_fullsized":"2024-04-25T11:33:11.886800+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:24:00.509704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.937820+0000","last_change":"2024-04-25T11:33:10.883842+0000","last_active":"2024-04-25T11:33:11.937820+0000","last_peered":"2024-04-25T11:33:11.937820+0000","last_clean":"2024-04-25T11:33:11.937820+0000","last_became_active":"2024-04-25T11:33:10.883513+0000","last_became_peered":"2024-04-25T11:33:10.883513+0000","last_unstale":"2024-04-25T11:33:11.937820+0000","last_undegraded":"2024-04-25T11:33:11.937820+0000","last_fullsized":"2024-04-25T11:33:11.937820+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:47:09.731875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.869816+0000","last_change":"2024-04-25T11:33:10.890503+0000","last_active":"2024-04-25T11:33:11.869816+0000","last_peered":"2024-04-25T11:33:11.869816+0000","last_clean":"2024-04-25T11:33:11.869816+0000","last_became_active":"2024-04-25T11:33:10.890162+0000","last_became_peered":"2024-04-25T11:33:10.890162+0000","last_unstale":"2024-04-25T11:33:11.869816+0000","last_undegraded":"2024-04-25T11:33:11.869816+0000","last_fullsized":"2024-04-25T11:33:11.869816+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:36.462789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886869+0000","last_change":"2024-04-25T11:33:10.897109+0000","last_active":"2024-04-25T11:33:11.886869+0000","last_peered":"2024-04-25T11:33:11.886869+0000","last_clean":"2024-04-25T11:33:11.886869+0000","last_became_active":"2024-04-25T11:33:10.896138+0000","last_became_peered":"2024-04-25T11:33:10.896138+0000","last_unstale":"2024-04-25T11:33:11.886869+0000","last_undegraded":"2024-04-25T11:33:11.886869+0000","last_fullsized":"2024-04-25T11:33:11.886869+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:47:46.202355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887556+0000","last_change":"2024-04-25T11:33:10.890397+0000","last_active":"2024-04-25T11:33:11.887556+0000","last_peered":"2024-04-25T11:33:11.887556+0000","last_clean":"2024-04-25T11:33:11.887556+0000","last_became_active":"2024-04-25T11:33:10.889580+0000","last_became_peered":"2024-04-25T11:33:10.889580+0000","last_unstale":"2024-04-25T11:33:11.887556+0000","last_undegraded":"2024-04-25T11:33:11.887556+0000","last_fullsized":"2024-04-25T11:33:11.887556+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:16:09.672131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886141+0000","last_change":"2024-04-25T11:33:10.873817+0000","last_active":"2024-04-25T11:33:11.886141+0000","last_peered":"2024-04-25T11:33:11.886141+0000","last_clean":"2024-04-25T11:33:11.886141+0000","last_became_active":"2024-04-25T11:33:10.873425+0000","last_became_peered":"2024-04-25T11:33:10.873425+0000","last_unstale":"2024-04-25T11:33:11.886141+0000","last_undegraded":"2024-04-25T11:33:11.886141+0000","last_fullsized":"2024-04-25T11:33:11.886141+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:52:26.279705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.871146+0000","last_change":"2024-04-25T11:33:10.880805+0000","last_active":"2024-04-25T11:33:11.871146+0000","last_peered":"2024-04-25T11:33:11.871146+0000","last_clean":"2024-04-25T11:33:11.871146+0000","last_became_active":"2024-04-25T11:33:10.880444+0000","last_became_peered":"2024-04-25T11:33:10.880444+0000","last_unstale":"2024-04-25T11:33:11.871146+0000","last_undegraded":"2024-04-25T11:33:11.871146+0000","last_fullsized":"2024-04-25T11:33:11.871146+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:57:03.602588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886780+0000","last_change":"2024-04-25T11:33:10.883690+0000","last_active":"2024-04-25T11:33:11.886780+0000","last_peered":"2024-04-25T11:33:11.886780+0000","last_clean":"2024-04-25T11:33:11.886780+0000","last_became_active":"2024-04-25T11:33:10.882965+0000","last_became_peered":"2024-04-25T11:33:10.882965+0000","last_unstale":"2024-04-25T11:33:11.886780+0000","last_undegraded":"2024-04-25T11:33:11.886780+0000","last_fullsized":"2024-04-25T11:33:11.886780+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:28.126631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886002+0000","last_change":"2024-04-25T11:33:10.897050+0000","last_active":"2024-04-25T11:33:11.886002+0000","last_peered":"2024-04-25T11:33:11.886002+0000","last_clean":"2024-04-25T11:33:11.886002+0000","last_became_active":"2024-04-25T11:33:10.896273+0000","last_became_peered":"2024-04-25T11:33:10.896273+0000","last_unstale":"2024-04-25T11:33:11.886002+0000","last_undegraded":"2024-04-25T11:33:11.886002+0000","last_fullsized":"2024-04-25T11:33:11.886002+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:45:53.839823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.869683+0000","last_change":"2024-04-25T11:33:10.897323+0000","last_active":"2024-04-25T11:33:11.869683+0000","last_peered":"2024-04-25T11:33:11.869683+0000","last_clean":"2024-04-25T11:33:11.869683+0000","last_became_active":"2024-04-25T11:33:10.896951+0000","last_became_peered":"2024-04-25T11:33:10.896951+0000","last_unstale":"2024-04-25T11:33:11.869683+0000","last_undegraded":"2024-04-25T11:33:11.869683+0000","last_fullsized":"2024-04-25T11:33:11.869683+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:24:31.208338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886438+0000","last_change":"2024-04-25T11:33:10.897566+0000","last_active":"2024-04-25T11:33:11.886438+0000","last_peered":"2024-04-25T11:33:11.886438+0000","last_clean":"2024-04-25T11:33:11.886438+0000","last_became_active":"2024-04-25T11:33:10.897212+0000","last_became_peered":"2024-04-25T11:33:10.897212+0000","last_unstale":"2024-04-25T11:33:11.886438+0000","last_undegraded":"2024-04-25T11:33:11.886438+0000","last_fullsized":"2024-04-25T11:33:11.886438+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:55:29.995287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6145,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":13,"num_read_kb":11,"num_write":124,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":35962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108,"ondisk_log_size":108,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645325,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36148,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709004,"statfs":{"total":95995035648,"available":95958020096,"internally_reserved":0,"allocated":827392,"data_stored":395971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841554,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32596,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712556,"statfs":{"total":95995035648,"available":95961657344,"internally_reserved":0,"allocated":1454080,"data_stored":988325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037782,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27836,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717316,"statfs":{"total":95995035648,"available":95966531584,"internally_reserved":0,"allocated":827392,"data_stored":395971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234012,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27840,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717312,"statfs":{"total":95995035648,"available":95966527488,"internally_reserved":0,"allocated":835584,"data_stored":396015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430241,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32572,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712580,"statfs":{"total":95995035648,"available":95961681920,"internally_reserved":0,"allocated":1429504,"data_stored":986961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626470,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31984,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713168,"statfs":{"total":95995035648,"available":95962284032,"internally_reserved":0,"allocated":823296,"data_stored":395389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36152,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709000,"statfs":{"total":95995035648,"available":95958016000,"internally_reserved":0,"allocated":835584,"data_stored":397001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27848,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717304,"statfs":{"total":95995035648,"available":95966519296,"internally_reserved":0,"allocated":851968,"data_stored":396867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411386,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40344,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704808,"statfs":{"total":95995035648,"available":95953723392,"internally_reserved":0,"allocated":868352,"data_stored":399364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182452,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27848,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717304,"statfs":{"total":95995035648,"available":95966519296,"internally_reserved":0,"allocated":847872,"data_stored":395797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986224,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31996,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713156,"statfs":{"total":95995035648,"available":95962271744,"internally_reserved":0,"allocated":839680,"data_stored":396055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822699,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28428,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716724,"statfs":{"total":95995035648,"available":95965925376,"internally_reserved":0,"allocated":1445888,"data_stored":987233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":22867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T11:33:18.078 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T11:33:18.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:17 smithi156 ceph-mon[33045]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:33:18.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:17 smithi156 ceph-mon[33045]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:33:18.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:17 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] up:active 2024-04-25T11:33:18.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:17 smithi156 ceph-mon[33045]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-04-25T11:33:18.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:17 smithi156 ceph-mon[33045]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T11:33:18.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:17 smithi156 ceph-mon[33045]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:33:18.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:17 smithi156 ceph-mon[33045]: pgmap v277: 97 pgs: 97 active+clean; 583 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 5.1 KiB/s wr, 20 op/s 2024-04-25T11:33:18.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:17 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1574051757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:33:18.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.231+0000 7f76f54e7640 1 -- 172.21.15.90:0/1932044934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f76f0103c20 msgr2=0x7f76f0104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:18.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.231+0000 7f76f54e7640 1 --2- 172.21.15.90:0/1932044934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f76f0103c20 0x7f76f0104030 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f76d8009900 tx=0x7f76d802a410 comp rx=0 tx=0).stop 2024-04-25T11:33:18.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.231+0000 7f76f54e7640 1 -- 172.21.15.90:0/1932044934 shutdown_connections 2024-04-25T11:33:18.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.231+0000 7f76f54e7640 1 --2- 172.21.15.90:0/1932044934 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f76f0108920 0x7f76f010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:18.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.231+0000 7f76f54e7640 1 --2- 172.21.15.90:0/1932044934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f76f0103c20 0x7f76f0104030 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:18.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.231+0000 7f76f54e7640 1 --2- 172.21.15.90:0/1932044934 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f76f01029e0 0x7f76f01036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:18.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.231+0000 7f76f54e7640 1 -- 172.21.15.90:0/1932044934 >> 172.21.15.90:0/1932044934 conn(0x7f76f00fdbd0 msgr2=0x7f76f0100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:18.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.231+0000 7f76f54e7640 1 -- 172.21.15.90:0/1932044934 shutdown_connections 2024-04-25T11:33:18.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.231+0000 7f76f54e7640 1 -- 172.21.15.90:0/1932044934 wait complete. 2024-04-25T11:33:18.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.232+0000 7f76f54e7640 1 Processor -- start 2024-04-25T11:33:18.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.232+0000 7f76f54e7640 1 -- start start 2024-04-25T11:33:18.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.232+0000 7f76f54e7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f76f01029e0 0x7f76f019a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:18.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.232+0000 7f76f54e7640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f76f0103c20 0x7f76f019aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:18.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.232+0000 7f76f54e7640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f76f0108920 0x7f76f01a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:18.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.232+0000 7f76f54e7640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f76f010f610 con 0x7f76f0108920 2024-04-25T11:33:18.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.232+0000 7f76f54e7640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f76f010f490 con 0x7f76f0103c20 2024-04-25T11:33:18.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.232+0000 7f76f54e7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f76f010f790 con 0x7f76f01029e0 2024-04-25T11:33:18.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.232+0000 7f76ef7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f76f0108920 0x7f76f01a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:18.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.232+0000 7f76ef7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f76f0108920 0x7f76f01a19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36478/0 (socket says 172.21.15.90:36478) 2024-04-25T11:33:18.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.232+0000 7f76ef7fe640 1 -- 172.21.15.90:0/3256169061 learned_addr learned my addr 172.21.15.90:0/3256169061 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:18.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.232+0000 7f76e7fff640 1 --2- 172.21.15.90:0/3256169061 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f76f0103c20 0x7f76f019aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:18.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.233+0000 7f76ef7fe640 1 -- 172.21.15.90:0/3256169061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f76f01029e0 msgr2=0x7f76f019a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:18.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.233+0000 7f76eeffd640 1 --2- 172.21.15.90:0/3256169061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f76f01029e0 0x7f76f019a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:18.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.233+0000 7f76ef7fe640 1 --2- 172.21.15.90:0/3256169061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f76f01029e0 0x7f76f019a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:18.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.233+0000 7f76ef7fe640 1 -- 172.21.15.90:0/3256169061 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f76f0103c20 msgr2=0x7f76f019aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:18.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.233+0000 7f76ef7fe640 1 --2- 172.21.15.90:0/3256169061 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f76f0103c20 0x7f76f019aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:18.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.233+0000 7f76ef7fe640 1 -- 172.21.15.90:0/3256169061 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76f01a1f80 con 0x7f76f0108920 2024-04-25T11:33:18.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.233+0000 7f76e7fff640 1 --2- 172.21.15.90:0/3256169061 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f76f0103c20 0x7f76f019aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:33:18.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.233+0000 7f76ef7fe640 1 --2- 172.21.15.90:0/3256169061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f76f0108920 0x7f76f01a19c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f76e800cce0 tx=0x7f76e800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:18.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.233+0000 7f76ecff9640 1 -- 172.21.15.90:0/3256169061 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e800f450 con 0x7f76f0108920 2024-04-25T11:33:18.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.233+0000 7f76ecff9640 1 -- 172.21.15.90:0/3256169061 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76e800faa0 con 0x7f76f0108920 2024-04-25T11:33:18.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.233+0000 7f76f54e7640 1 -- 172.21.15.90:0/3256169061 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76f01a2270 con 0x7f76f0108920 2024-04-25T11:33:18.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.233+0000 7f76ecff9640 1 -- 172.21.15.90:0/3256169061 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e8018b70 con 0x7f76f0108920 2024-04-25T11:33:18.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.233+0000 7f76f54e7640 1 -- 172.21.15.90:0/3256169061 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76f01a2770 con 0x7f76f0108920 2024-04-25T11:33:18.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.234+0000 7f76f54e7640 1 -- 172.21.15.90:0/3256169061 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76f0107730 con 0x7f76f0108920 2024-04-25T11:33:18.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.235+0000 7f76ecff9640 1 -- 172.21.15.90:0/3256169061 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f76e8018d10 con 0x7f76f0108920 2024-04-25T11:33:18.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.236+0000 7f76ecff9640 1 --2- 172.21.15.90:0/3256169061 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f76dc0afb80 0x7f76dc0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:18.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.236+0000 7f76eeffd640 1 --2- 172.21.15.90:0/3256169061 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f76dc0afb80 0x7f76dc0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:18.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.237+0000 7f76ecff9640 1 -- 172.21.15.90:0/3256169061 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7f76e80df2c0 con 0x7f76f0108920 2024-04-25T11:33:18.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.237+0000 7f76eeffd640 1 --2- 172.21.15.90:0/3256169061 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f76dc0afb80 0x7f76dc0b2090 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f76e0005fd0 tx=0x7f76e0005ce0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:18.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.240+0000 7f76ecff9640 1 -- 172.21.15.90:0/3256169061 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f76e8014030 con 0x7f76f0108920 2024-04-25T11:33:18.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.417+0000 7f76f54e7640 1 -- 172.21.15.90:0/3256169061 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f76f002de30 con 0x7f76dc0afb80 2024-04-25T11:33:18.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.424+0000 7f76ecff9640 1 -- 172.21.15.90:0/3256169061 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258740 (secure 0 0 0) 0x7f76f002de30 con 0x7f76dc0afb80 2024-04-25T11:33:18.416 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:33:18.417 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-25T11:33:18.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.427+0000 7f76f54e7640 1 -- 172.21.15.90:0/3256169061 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f76dc0afb80 msgr2=0x7f76dc0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:18.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.427+0000 7f76f54e7640 1 --2- 172.21.15.90:0/3256169061 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f76dc0afb80 0x7f76dc0b2090 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f76e0005fd0 tx=0x7f76e0005ce0 comp rx=0 tx=0).stop 2024-04-25T11:33:18.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.427+0000 7f76f54e7640 1 -- 172.21.15.90:0/3256169061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f76f0108920 msgr2=0x7f76f01a19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:18.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.427+0000 7f76f54e7640 1 --2- 172.21.15.90:0/3256169061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f76f0108920 0x7f76f01a19c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f76e800cce0 tx=0x7f76e800a620 comp rx=0 tx=0).stop 2024-04-25T11:33:18.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.427+0000 7f76f54e7640 1 -- 172.21.15.90:0/3256169061 shutdown_connections 2024-04-25T11:33:18.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.427+0000 7f76f54e7640 1 --2- 172.21.15.90:0/3256169061 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f76dc0afb80 0x7f76dc0b2090 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:18.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.427+0000 7f76f54e7640 1 --2- 172.21.15.90:0/3256169061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f76f0108920 0x7f76f01a19c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:18.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.427+0000 7f76f54e7640 1 --2- 172.21.15.90:0/3256169061 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f76f0103c20 0x7f76f019aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:18.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.427+0000 7f76f54e7640 1 --2- 172.21.15.90:0/3256169061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f76f01029e0 0x7f76f019a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:18.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.427+0000 7f76f54e7640 1 -- 172.21.15.90:0/3256169061 >> 172.21.15.90:0/3256169061 conn(0x7f76f00fdbd0 msgr2=0x7f76f0104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:18.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.428+0000 7f76f54e7640 1 -- 172.21.15.90:0/3256169061 shutdown_connections 2024-04-25T11:33:18.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:18.428+0000 7f76f54e7640 1 -- 172.21.15.90:0/3256169061 wait complete. 2024-04-25T11:33:18.427 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":277,"stamp":"2024-04-25T11:33:16.741717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596513,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":94,"num_read_kb":80,"num_write":247,"num_write_kb":2184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":200,"ondisk_log_size":200,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":233,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":381592,"kb_used_data":11608,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124560232,"statfs":{"total":1151940427776,"available":1151549677568,"internally_reserved":0,"allocated":11886592,"data_stored":6530949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6145,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":13,"num_read_kb":11,"num_write":124,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.727353"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886955+0000","last_change":"2024-04-25T11:33:10.892144+0000","last_active":"2024-04-25T11:33:11.886955+0000","last_peered":"2024-04-25T11:33:11.886955+0000","last_clean":"2024-04-25T11:33:11.886955+0000","last_became_active":"2024-04-25T11:33:10.891721+0000","last_became_peered":"2024-04-25T11:33:10.891721+0000","last_unstale":"2024-04-25T11:33:11.886955+0000","last_undegraded":"2024-04-25T11:33:11.886955+0000","last_fullsized":"2024-04-25T11:33:11.886955+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:58:25.447086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.937797+0000","last_change":"2024-04-25T11:33:10.891647+0000","last_active":"2024-04-25T11:33:11.937797+0000","last_peered":"2024-04-25T11:33:11.937797+0000","last_clean":"2024-04-25T11:33:11.937797+0000","last_became_active":"2024-04-25T11:33:10.891283+0000","last_became_peered":"2024-04-25T11:33:10.891283+0000","last_unstale":"2024-04-25T11:33:11.937797+0000","last_undegraded":"2024-04-25T11:33:11.937797+0000","last_fullsized":"2024-04-25T11:33:11.937797+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:12.669438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.937801+0000","last_change":"2024-04-25T11:33:10.890928+0000","last_active":"2024-04-25T11:33:11.937801+0000","last_peered":"2024-04-25T11:33:11.937801+0000","last_clean":"2024-04-25T11:33:11.937801+0000","last_became_active":"2024-04-25T11:33:10.889093+0000","last_became_peered":"2024-04-25T11:33:10.889093+0000","last_unstale":"2024-04-25T11:33:11.937801+0000","last_undegraded":"2024-04-25T11:33:11.937801+0000","last_fullsized":"2024-04-25T11:33:11.937801+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:28:09.038650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.871223+0000","last_change":"2024-04-25T11:33:10.890184+0000","last_active":"2024-04-25T11:33:11.871223+0000","last_peered":"2024-04-25T11:33:11.871223+0000","last_clean":"2024-04-25T11:33:11.871223+0000","last_became_active":"2024-04-25T11:33:10.889367+0000","last_became_peered":"2024-04-25T11:33:10.889367+0000","last_unstale":"2024-04-25T11:33:11.871223+0000","last_undegraded":"2024-04-25T11:33:11.871223+0000","last_fullsized":"2024-04-25T11:33:11.871223+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:42.798295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886028+0000","last_change":"2024-04-25T11:33:10.889758+0000","last_active":"2024-04-25T11:33:11.886028+0000","last_peered":"2024-04-25T11:33:11.886028+0000","last_clean":"2024-04-25T11:33:11.886028+0000","last_became_active":"2024-04-25T11:33:10.889384+0000","last_became_peered":"2024-04-25T11:33:10.889384+0000","last_unstale":"2024-04-25T11:33:11.886028+0000","last_undegraded":"2024-04-25T11:33:11.886028+0000","last_fullsized":"2024-04-25T11:33:11.886028+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:54:38.794216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886638+0000","last_change":"2024-04-25T11:33:10.889348+0000","last_active":"2024-04-25T11:33:11.886638+0000","last_peered":"2024-04-25T11:33:11.886638+0000","last_clean":"2024-04-25T11:33:11.886638+0000","last_became_active":"2024-04-25T11:33:10.888908+0000","last_became_peered":"2024-04-25T11:33:10.888908+0000","last_unstale":"2024-04-25T11:33:11.886638+0000","last_undegraded":"2024-04-25T11:33:11.886638+0000","last_fullsized":"2024-04-25T11:33:11.886638+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:02:59.555921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886809+0000","last_change":"2024-04-25T11:33:10.897745+0000","last_active":"2024-04-25T11:33:11.886809+0000","last_peered":"2024-04-25T11:33:11.886809+0000","last_clean":"2024-04-25T11:33:11.886809+0000","last_became_active":"2024-04-25T11:33:10.897403+0000","last_became_peered":"2024-04-25T11:33:10.897403+0000","last_unstale":"2024-04-25T11:33:11.886809+0000","last_undegraded":"2024-04-25T11:33:11.886809+0000","last_fullsized":"2024-04-25T11:33:11.886809+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:33:34.682719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.885965+0000","last_change":"2024-04-25T11:33:10.887262+0000","last_active":"2024-04-25T11:33:11.885965+0000","last_peered":"2024-04-25T11:33:11.885965+0000","last_clean":"2024-04-25T11:33:11.885965+0000","last_became_active":"2024-04-25T11:33:10.886948+0000","last_became_peered":"2024-04-25T11:33:10.886948+0000","last_unstale":"2024-04-25T11:33:11.885965+0000","last_undegraded":"2024-04-25T11:33:11.885965+0000","last_fullsized":"2024-04-25T11:33:11.885965+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:09.721506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886224+0000","last_change":"2024-04-25T11:33:10.898392+0000","last_active":"2024-04-25T11:33:11.886224+0000","last_peered":"2024-04-25T11:33:11.886224+0000","last_clean":"2024-04-25T11:33:11.886224+0000","last_became_active":"2024-04-25T11:33:10.897935+0000","last_became_peered":"2024-04-25T11:33:10.897935+0000","last_unstale":"2024-04-25T11:33:11.886224+0000","last_undegraded":"2024-04-25T11:33:11.886224+0000","last_fullsized":"2024-04-25T11:33:11.886224+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:38:42.885463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887117+0000","last_change":"2024-04-25T11:33:10.889812+0000","last_active":"2024-04-25T11:33:11.887117+0000","last_peered":"2024-04-25T11:33:11.887117+0000","last_clean":"2024-04-25T11:33:11.887117+0000","last_became_active":"2024-04-25T11:33:10.889369+0000","last_became_peered":"2024-04-25T11:33:10.889369+0000","last_unstale":"2024-04-25T11:33:11.887117+0000","last_undegraded":"2024-04-25T11:33:11.887117+0000","last_fullsized":"2024-04-25T11:33:11.887117+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:42:14.443496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887365+0000","last_change":"2024-04-25T11:33:10.891186+0000","last_active":"2024-04-25T11:33:11.887365+0000","last_peered":"2024-04-25T11:33:11.887365+0000","last_clean":"2024-04-25T11:33:11.887365+0000","last_became_active":"2024-04-25T11:33:10.890841+0000","last_became_peered":"2024-04-25T11:33:10.890841+0000","last_unstale":"2024-04-25T11:33:11.887365+0000","last_undegraded":"2024-04-25T11:33:11.887365+0000","last_fullsized":"2024-04-25T11:33:11.887365+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:24:56.078592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.871140+0000","last_change":"2024-04-25T11:33:10.888801+0000","last_active":"2024-04-25T11:33:11.871140+0000","last_peered":"2024-04-25T11:33:11.871140+0000","last_clean":"2024-04-25T11:33:11.871140+0000","last_became_active":"2024-04-25T11:33:10.888437+0000","last_became_peered":"2024-04-25T11:33:10.888437+0000","last_unstale":"2024-04-25T11:33:11.871140+0000","last_undegraded":"2024-04-25T11:33:11.871140+0000","last_fullsized":"2024-04-25T11:33:11.871140+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:14:14.011475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.869640+0000","last_change":"2024-04-25T11:33:10.892749+0000","last_active":"2024-04-25T11:33:11.869640+0000","last_peered":"2024-04-25T11:33:11.869640+0000","last_clean":"2024-04-25T11:33:11.869640+0000","last_became_active":"2024-04-25T11:33:10.892370+0000","last_became_peered":"2024-04-25T11:33:10.892370+0000","last_unstale":"2024-04-25T11:33:11.869640+0000","last_undegraded":"2024-04-25T11:33:11.869640+0000","last_fullsized":"2024-04-25T11:33:11.869640+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:00:51.312387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.937776+0000","last_change":"2024-04-25T11:33:10.888981+0000","last_active":"2024-04-25T11:33:11.937776+0000","last_peered":"2024-04-25T11:33:11.937776+0000","last_clean":"2024-04-25T11:33:11.937776+0000","last_became_active":"2024-04-25T11:33:10.888537+0000","last_became_peered":"2024-04-25T11:33:10.888537+0000","last_unstale":"2024-04-25T11:33:11.937776+0000","last_undegraded":"2024-04-25T11:33:11.937776+0000","last_fullsized":"2024-04-25T11:33:11.937776+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:12:19.018698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.937482+0000","last_change":"2024-04-25T11:33:10.891455+0000","last_active":"2024-04-25T11:33:11.937482+0000","last_peered":"2024-04-25T11:33:11.937482+0000","last_clean":"2024-04-25T11:33:11.937482+0000","last_became_active":"2024-04-25T11:33:10.891037+0000","last_became_peered":"2024-04-25T11:33:10.891037+0000","last_unstale":"2024-04-25T11:33:11.937482+0000","last_undegraded":"2024-04-25T11:33:11.937482+0000","last_fullsized":"2024-04-25T11:33:11.937482+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:44.470674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.885677+0000","last_change":"2024-04-25T11:33:10.895309+0000","last_active":"2024-04-25T11:33:11.885677+0000","last_peered":"2024-04-25T11:33:11.885677+0000","last_clean":"2024-04-25T11:33:11.885677+0000","last_became_active":"2024-04-25T11:33:10.894952+0000","last_became_peered":"2024-04-25T11:33:10.894952+0000","last_unstale":"2024-04-25T11:33:11.885677+0000","last_undegraded":"2024-04-25T11:33:11.885677+0000","last_fullsized":"2024-04-25T11:33:11.885677+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:29:22.119017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887307+0000","last_change":"2024-04-25T11:33:10.894524+0000","last_active":"2024-04-25T11:33:11.887307+0000","last_peered":"2024-04-25T11:33:11.887307+0000","last_clean":"2024-04-25T11:33:11.887307+0000","last_became_active":"2024-04-25T11:33:10.894220+0000","last_became_peered":"2024-04-25T11:33:10.894220+0000","last_unstale":"2024-04-25T11:33:11.887307+0000","last_undegraded":"2024-04-25T11:33:11.887307+0000","last_fullsized":"2024-04-25T11:33:11.887307+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:10:29.072976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886343+0000","last_change":"2024-04-25T11:33:10.886814+0000","last_active":"2024-04-25T11:33:11.886343+0000","last_peered":"2024-04-25T11:33:11.886343+0000","last_clean":"2024-04-25T11:33:11.886343+0000","last_became_active":"2024-04-25T11:33:10.886494+0000","last_became_peered":"2024-04-25T11:33:10.886494+0000","last_unstale":"2024-04-25T11:33:11.886343+0000","last_undegraded":"2024-04-25T11:33:11.886343+0000","last_fullsized":"2024-04-25T11:33:11.886343+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:03:43.054485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886362+0000","last_change":"2024-04-25T11:33:10.886870+0000","last_active":"2024-04-25T11:33:11.886362+0000","last_peered":"2024-04-25T11:33:11.886362+0000","last_clean":"2024-04-25T11:33:11.886362+0000","last_became_active":"2024-04-25T11:33:10.886507+0000","last_became_peered":"2024-04-25T11:33:10.886507+0000","last_unstale":"2024-04-25T11:33:11.886362+0000","last_undegraded":"2024-04-25T11:33:11.886362+0000","last_fullsized":"2024-04-25T11:33:11.886362+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:50.101760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"76'4","reported_seq":19,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886345+0000","last_change":"2024-04-25T11:33:10.028970+0000","last_active":"2024-04-25T11:33:11.886345+0000","last_peered":"2024-04-25T11:33:11.886345+0000","last_clean":"2024-04-25T11:33:11.886345+0000","last_became_active":"2024-04-25T11:33:10.028636+0000","last_became_peered":"2024-04-25T11:33:10.028636+0000","last_unstale":"2024-04-25T11:33:11.886345+0000","last_undegraded":"2024-04-25T11:33:11.886345+0000","last_fullsized":"2024-04-25T11:33:11.886345+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:02.414903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886707+0000","last_change":"2024-04-25T11:33:10.882539+0000","last_active":"2024-04-25T11:33:11.886707+0000","last_peered":"2024-04-25T11:33:11.886707+0000","last_clean":"2024-04-25T11:33:11.886707+0000","last_became_active":"2024-04-25T11:33:10.882142+0000","last_became_peered":"2024-04-25T11:33:10.882142+0000","last_unstale":"2024-04-25T11:33:11.886707+0000","last_undegraded":"2024-04-25T11:33:11.886707+0000","last_fullsized":"2024-04-25T11:33:11.886707+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:55:32.959589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"77'2","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886742+0000","last_change":"2024-04-25T11:33:10.036958+0000","last_active":"2024-04-25T11:33:11.886742+0000","last_peered":"2024-04-25T11:33:11.886742+0000","last_clean":"2024-04-25T11:33:11.886742+0000","last_became_active":"2024-04-25T11:33:10.036606+0000","last_became_peered":"2024-04-25T11:33:10.036606+0000","last_unstale":"2024-04-25T11:33:11.886742+0000","last_undegraded":"2024-04-25T11:33:11.886742+0000","last_fullsized":"2024-04-25T11:33:11.886742+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:12.660208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.937901+0000","last_change":"2024-04-25T11:33:10.887501+0000","last_active":"2024-04-25T11:33:11.937901+0000","last_peered":"2024-04-25T11:33:11.937901+0000","last_clean":"2024-04-25T11:33:11.937901+0000","last_became_active":"2024-04-25T11:33:10.885985+0000","last_became_peered":"2024-04-25T11:33:10.885985+0000","last_unstale":"2024-04-25T11:33:11.937901+0000","last_undegraded":"2024-04-25T11:33:11.937901+0000","last_fullsized":"2024-04-25T11:33:11.937901+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:31:32.035646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"76'2","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887252+0000","last_change":"2024-04-25T11:33:10.031161+0000","last_active":"2024-04-25T11:33:11.887252+0000","last_peered":"2024-04-25T11:33:11.887252+0000","last_clean":"2024-04-25T11:33:11.887252+0000","last_became_active":"2024-04-25T11:33:10.028891+0000","last_became_peered":"2024-04-25T11:33:10.028891+0000","last_unstale":"2024-04-25T11:33:11.887252+0000","last_undegraded":"2024-04-25T11:33:11.887252+0000","last_fullsized":"2024-04-25T11:33:11.887252+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:25.959699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.885937+0000","last_change":"2024-04-25T11:33:10.889439+0000","last_active":"2024-04-25T11:33:11.885937+0000","last_peered":"2024-04-25T11:33:11.885937+0000","last_clean":"2024-04-25T11:33:11.885937+0000","last_became_active":"2024-04-25T11:33:10.889082+0000","last_became_peered":"2024-04-25T11:33:10.889082+0000","last_unstale":"2024-04-25T11:33:11.885937+0000","last_undegraded":"2024-04-25T11:33:11.885937+0000","last_fullsized":"2024-04-25T11:33:11.885937+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:13:48.958179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"76'2","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886707+0000","last_change":"2024-04-25T11:33:10.030793+0000","last_active":"2024-04-25T11:33:11.886707+0000","last_peered":"2024-04-25T11:33:11.886707+0000","last_clean":"2024-04-25T11:33:11.886707+0000","last_became_active":"2024-04-25T11:33:10.028805+0000","last_became_peered":"2024-04-25T11:33:10.028805+0000","last_unstale":"2024-04-25T11:33:11.886707+0000","last_undegraded":"2024-04-25T11:33:11.886707+0000","last_fullsized":"2024-04-25T11:33:11.886707+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:36.233194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.871391+0000","last_change":"2024-04-25T11:33:10.894815+0000","last_active":"2024-04-25T11:33:11.871391+0000","last_peered":"2024-04-25T11:33:11.871391+0000","last_clean":"2024-04-25T11:33:11.871391+0000","last_became_active":"2024-04-25T11:33:10.892174+0000","last_became_peered":"2024-04-25T11:33:10.892174+0000","last_unstale":"2024-04-25T11:33:11.871391+0000","last_undegraded":"2024-04-25T11:33:11.871391+0000","last_fullsized":"2024-04-25T11:33:11.871391+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:05.368441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"77'5","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.616398+0000","last_change":"2024-04-25T11:33:10.031238+0000","last_active":"2024-04-25T11:33:13.616398+0000","last_peered":"2024-04-25T11:33:13.616398+0000","last_clean":"2024-04-25T11:33:13.616398+0000","last_became_active":"2024-04-25T11:33:10.030917+0000","last_became_peered":"2024-04-25T11:33:10.030917+0000","last_unstale":"2024-04-25T11:33:13.616398+0000","last_undegraded":"2024-04-25T11:33:13.616398+0000","last_fullsized":"2024-04-25T11:33:13.616398+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:00:35.634293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887041+0000","last_change":"2024-04-25T11:33:10.896974+0000","last_active":"2024-04-25T11:33:11.887041+0000","last_peered":"2024-04-25T11:33:11.887041+0000","last_clean":"2024-04-25T11:33:11.887041+0000","last_became_active":"2024-04-25T11:33:10.896575+0000","last_became_peered":"2024-04-25T11:33:10.896575+0000","last_unstale":"2024-04-25T11:33:11.887041+0000","last_undegraded":"2024-04-25T11:33:11.887041+0000","last_fullsized":"2024-04-25T11:33:11.887041+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:32:43.533652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"77'3","reported_seq":18,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.628190+0000","last_change":"2024-04-25T11:33:10.022714+0000","last_active":"2024-04-25T11:33:13.628190+0000","last_peered":"2024-04-25T11:33:13.628190+0000","last_clean":"2024-04-25T11:33:13.628190+0000","last_became_active":"2024-04-25T11:33:10.022230+0000","last_became_peered":"2024-04-25T11:33:10.022230+0000","last_unstale":"2024-04-25T11:33:13.628190+0000","last_undegraded":"2024-04-25T11:33:13.628190+0000","last_fullsized":"2024-04-25T11:33:13.628190+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:30:00.620833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.871077+0000","last_change":"2024-04-25T11:33:10.897664+0000","last_active":"2024-04-25T11:33:11.871077+0000","last_peered":"2024-04-25T11:33:11.871077+0000","last_clean":"2024-04-25T11:33:11.871077+0000","last_became_active":"2024-04-25T11:33:10.897189+0000","last_became_peered":"2024-04-25T11:33:10.897189+0000","last_unstale":"2024-04-25T11:33:11.871077+0000","last_undegraded":"2024-04-25T11:33:11.871077+0000","last_fullsized":"2024-04-25T11:33:11.871077+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:12:06.168793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"77'2","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.626198+0000","last_change":"2024-04-25T11:33:10.031417+0000","last_active":"2024-04-25T11:33:13.626198+0000","last_peered":"2024-04-25T11:33:13.626198+0000","last_clean":"2024-04-25T11:33:13.626198+0000","last_became_active":"2024-04-25T11:33:10.030526+0000","last_became_peered":"2024-04-25T11:33:10.030526+0000","last_unstale":"2024-04-25T11:33:13.626198+0000","last_undegraded":"2024-04-25T11:33:13.626198+0000","last_fullsized":"2024-04-25T11:33:13.626198+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:05.718720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886517+0000","last_change":"2024-04-25T11:33:10.886596+0000","last_active":"2024-04-25T11:33:11.886517+0000","last_peered":"2024-04-25T11:33:11.886517+0000","last_clean":"2024-04-25T11:33:11.886517+0000","last_became_active":"2024-04-25T11:33:10.886234+0000","last_became_peered":"2024-04-25T11:33:10.886234+0000","last_unstale":"2024-04-25T11:33:11.886517+0000","last_undegraded":"2024-04-25T11:33:11.886517+0000","last_fullsized":"2024-04-25T11:33:11.886517+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:46.632814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"77'1","reported_seq":16,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.870837+0000","last_change":"2024-04-25T11:33:10.032355+0000","last_active":"2024-04-25T11:33:11.870837+0000","last_peered":"2024-04-25T11:33:11.870837+0000","last_clean":"2024-04-25T11:33:11.870837+0000","last_became_active":"2024-04-25T11:33:10.031965+0000","last_became_peered":"2024-04-25T11:33:10.031965+0000","last_unstale":"2024-04-25T11:33:11.870837+0000","last_undegraded":"2024-04-25T11:33:11.870837+0000","last_fullsized":"2024-04-25T11:33:11.870837+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:08:10.539604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.885902+0000","last_change":"2024-04-25T11:33:10.885708+0000","last_active":"2024-04-25T11:33:11.885902+0000","last_peered":"2024-04-25T11:33:11.885902+0000","last_clean":"2024-04-25T11:33:11.885902+0000","last_became_active":"2024-04-25T11:33:10.885317+0000","last_became_peered":"2024-04-25T11:33:10.885317+0000","last_unstale":"2024-04-25T11:33:11.885902+0000","last_undegraded":"2024-04-25T11:33:11.885902+0000","last_fullsized":"2024-04-25T11:33:11.885902+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:50:08.518922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"77'3","reported_seq":19,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.615808+0000","last_change":"2024-04-25T11:33:10.030995+0000","last_active":"2024-04-25T11:33:13.615808+0000","last_peered":"2024-04-25T11:33:13.615808+0000","last_clean":"2024-04-25T11:33:13.615808+0000","last_became_active":"2024-04-25T11:33:10.030670+0000","last_became_peered":"2024-04-25T11:33:10.030670+0000","last_unstale":"2024-04-25T11:33:13.615808+0000","last_undegraded":"2024-04-25T11:33:13.615808+0000","last_fullsized":"2024-04-25T11:33:13.615808+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:39.569787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887554+0000","last_change":"2024-04-25T11:33:10.892927+0000","last_active":"2024-04-25T11:33:11.887554+0000","last_peered":"2024-04-25T11:33:11.887554+0000","last_clean":"2024-04-25T11:33:11.887554+0000","last_became_active":"2024-04-25T11:33:10.892458+0000","last_became_peered":"2024-04-25T11:33:10.892458+0000","last_unstale":"2024-04-25T11:33:11.887554+0000","last_undegraded":"2024-04-25T11:33:11.887554+0000","last_fullsized":"2024-04-25T11:33:11.887554+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:53.121185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"77'2","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.045429+0000","last_change":"2024-04-25T11:33:10.029728+0000","last_active":"2024-04-25T11:33:13.045429+0000","last_peered":"2024-04-25T11:33:13.045429+0000","last_clean":"2024-04-25T11:33:13.045429+0000","last_became_active":"2024-04-25T11:33:10.029435+0000","last_became_peered":"2024-04-25T11:33:10.029435+0000","last_unstale":"2024-04-25T11:33:13.045429+0000","last_undegraded":"2024-04-25T11:33:13.045429+0000","last_fullsized":"2024-04-25T11:33:13.045429+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:31:21.585870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.937825+0000","last_change":"2024-04-25T11:33:10.895828+0000","last_active":"2024-04-25T11:33:11.937825+0000","last_peered":"2024-04-25T11:33:11.937825+0000","last_clean":"2024-04-25T11:33:11.937825+0000","last_became_active":"2024-04-25T11:33:10.895280+0000","last_became_peered":"2024-04-25T11:33:10.895280+0000","last_unstale":"2024-04-25T11:33:11.937825+0000","last_undegraded":"2024-04-25T11:33:11.937825+0000","last_fullsized":"2024-04-25T11:33:11.937825+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:03.009912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"77'4","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.043498+0000","last_change":"2024-04-25T11:33:10.036651+0000","last_active":"2024-04-25T11:33:13.043498+0000","last_peered":"2024-04-25T11:33:13.043498+0000","last_clean":"2024-04-25T11:33:13.043498+0000","last_became_active":"2024-04-25T11:33:10.036294+0000","last_became_peered":"2024-04-25T11:33:10.036294+0000","last_unstale":"2024-04-25T11:33:13.043498+0000","last_undegraded":"2024-04-25T11:33:13.043498+0000","last_fullsized":"2024-04-25T11:33:13.043498+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:12:36.566618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.885967+0000","last_change":"2024-04-25T11:33:10.890199+0000","last_active":"2024-04-25T11:33:11.885967+0000","last_peered":"2024-04-25T11:33:11.885967+0000","last_clean":"2024-04-25T11:33:11.885967+0000","last_became_active":"2024-04-25T11:33:10.889855+0000","last_became_peered":"2024-04-25T11:33:10.889855+0000","last_unstale":"2024-04-25T11:33:11.885967+0000","last_undegraded":"2024-04-25T11:33:11.885967+0000","last_fullsized":"2024-04-25T11:33:11.885967+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:54:45.476647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"77'8","reported_seq":27,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.620848+0000","last_change":"2024-04-25T11:33:10.027364+0000","last_active":"2024-04-25T11:33:13.620848+0000","last_peered":"2024-04-25T11:33:13.620848+0000","last_clean":"2024-04-25T11:33:13.620848+0000","last_became_active":"2024-04-25T11:33:10.026955+0000","last_became_peered":"2024-04-25T11:33:10.026955+0000","last_unstale":"2024-04-25T11:33:13.620848+0000","last_undegraded":"2024-04-25T11:33:13.620848+0000","last_fullsized":"2024-04-25T11:33:13.620848+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:23:46.330813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886920+0000","last_change":"2024-04-25T11:33:10.888596+0000","last_active":"2024-04-25T11:33:11.886920+0000","last_peered":"2024-04-25T11:33:11.886920+0000","last_clean":"2024-04-25T11:33:11.886920+0000","last_became_active":"2024-04-25T11:33:10.888226+0000","last_became_peered":"2024-04-25T11:33:10.888226+0000","last_unstale":"2024-04-25T11:33:11.886920+0000","last_undegraded":"2024-04-25T11:33:11.886920+0000","last_fullsized":"2024-04-25T11:33:11.886920+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:30:42.891788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"77'9","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:15.646566+0000","last_change":"2024-04-25T11:33:10.026969+0000","last_active":"2024-04-25T11:33:15.646566+0000","last_peered":"2024-04-25T11:33:15.646566+0000","last_clean":"2024-04-25T11:33:15.646566+0000","last_became_active":"2024-04-25T11:33:10.026591+0000","last_became_peered":"2024-04-25T11:33:10.026591+0000","last_unstale":"2024-04-25T11:33:15.646566+0000","last_undegraded":"2024-04-25T11:33:15.646566+0000","last_fullsized":"2024-04-25T11:33:15.646566+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:50:34.269478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.869472+0000","last_change":"2024-04-25T11:33:10.900303+0000","last_active":"2024-04-25T11:33:11.869472+0000","last_peered":"2024-04-25T11:33:11.869472+0000","last_clean":"2024-04-25T11:33:11.869472+0000","last_became_active":"2024-04-25T11:33:10.899946+0000","last_became_peered":"2024-04-25T11:33:10.899946+0000","last_unstale":"2024-04-25T11:33:11.869472+0000","last_undegraded":"2024-04-25T11:33:11.869472+0000","last_fullsized":"2024-04-25T11:33:11.869472+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:33:40.805400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"77'3","reported_seq":18,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.044412+0000","last_change":"2024-04-25T11:33:10.035668+0000","last_active":"2024-04-25T11:33:13.044412+0000","last_peered":"2024-04-25T11:33:13.044412+0000","last_clean":"2024-04-25T11:33:13.044412+0000","last_became_active":"2024-04-25T11:33:10.035241+0000","last_became_peered":"2024-04-25T11:33:10.035241+0000","last_unstale":"2024-04-25T11:33:13.044412+0000","last_undegraded":"2024-04-25T11:33:13.044412+0000","last_fullsized":"2024-04-25T11:33:13.044412+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:58:48.735577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887293+0000","last_change":"2024-04-25T11:33:10.894864+0000","last_active":"2024-04-25T11:33:11.887293+0000","last_peered":"2024-04-25T11:33:11.887293+0000","last_clean":"2024-04-25T11:33:11.887293+0000","last_became_active":"2024-04-25T11:33:10.894491+0000","last_became_peered":"2024-04-25T11:33:10.894491+0000","last_unstale":"2024-04-25T11:33:11.887293+0000","last_undegraded":"2024-04-25T11:33:11.887293+0000","last_fullsized":"2024-04-25T11:33:11.887293+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:21:28.256878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"77'6","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:14.630764+0000","last_change":"2024-04-25T11:33:10.029099+0000","last_active":"2024-04-25T11:33:14.630764+0000","last_peered":"2024-04-25T11:33:14.630764+0000","last_clean":"2024-04-25T11:33:14.630764+0000","last_became_active":"2024-04-25T11:33:10.028722+0000","last_became_peered":"2024-04-25T11:33:10.028722+0000","last_unstale":"2024-04-25T11:33:14.630764+0000","last_undegraded":"2024-04-25T11:33:14.630764+0000","last_fullsized":"2024-04-25T11:33:14.630764+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:44:12.261519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886572+0000","last_change":"2024-04-25T11:33:10.888635+0000","last_active":"2024-04-25T11:33:11.886572+0000","last_peered":"2024-04-25T11:33:11.886572+0000","last_clean":"2024-04-25T11:33:11.886572+0000","last_became_active":"2024-04-25T11:33:10.887544+0000","last_became_peered":"2024-04-25T11:33:10.887544+0000","last_unstale":"2024-04-25T11:33:11.886572+0000","last_undegraded":"2024-04-25T11:33:11.886572+0000","last_fullsized":"2024-04-25T11:33:11.886572+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:59:25.459587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"77'10","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.042745+0000","last_change":"2024-04-25T11:33:10.023566+0000","last_active":"2024-04-25T11:33:13.042745+0000","last_peered":"2024-04-25T11:33:13.042745+0000","last_clean":"2024-04-25T11:33:13.042745+0000","last_became_active":"2024-04-25T11:33:10.023234+0000","last_became_peered":"2024-04-25T11:33:10.023234+0000","last_unstale":"2024-04-25T11:33:13.042745+0000","last_undegraded":"2024-04-25T11:33:13.042745+0000","last_fullsized":"2024-04-25T11:33:13.042745+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:56:26.342148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.870905+0000","last_change":"2024-04-25T11:33:10.895866+0000","last_active":"2024-04-25T11:33:11.870905+0000","last_peered":"2024-04-25T11:33:11.870905+0000","last_clean":"2024-04-25T11:33:11.870905+0000","last_became_active":"2024-04-25T11:33:10.895455+0000","last_became_peered":"2024-04-25T11:33:10.895455+0000","last_unstale":"2024-04-25T11:33:11.870905+0000","last_undegraded":"2024-04-25T11:33:11.870905+0000","last_fullsized":"2024-04-25T11:33:11.870905+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:30:25.380439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"76'1","reported_seq":16,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887276+0000","last_change":"2024-04-25T11:33:10.028285+0000","last_active":"2024-04-25T11:33:11.887276+0000","last_peered":"2024-04-25T11:33:11.887276+0000","last_clean":"2024-04-25T11:33:11.887276+0000","last_became_active":"2024-04-25T11:33:10.027959+0000","last_became_peered":"2024-04-25T11:33:10.027959+0000","last_unstale":"2024-04-25T11:33:11.887276+0000","last_undegraded":"2024-04-25T11:33:11.887276+0000","last_fullsized":"2024-04-25T11:33:11.887276+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:44:41.094048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886892+0000","last_change":"2024-04-25T11:33:10.890123+0000","last_active":"2024-04-25T11:33:11.886892+0000","last_peered":"2024-04-25T11:33:11.886892+0000","last_clean":"2024-04-25T11:33:11.886892+0000","last_became_active":"2024-04-25T11:33:10.889746+0000","last_became_peered":"2024-04-25T11:33:10.889746+0000","last_unstale":"2024-04-25T11:33:11.886892+0000","last_undegraded":"2024-04-25T11:33:11.886892+0000","last_fullsized":"2024-04-25T11:33:11.886892+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:25.650151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.885908+0000","last_change":"2024-04-25T11:33:10.027473+0000","last_active":"2024-04-25T11:33:11.885908+0000","last_peered":"2024-04-25T11:33:11.885908+0000","last_clean":"2024-04-25T11:33:11.885908+0000","last_became_active":"2024-04-25T11:33:10.027133+0000","last_became_peered":"2024-04-25T11:33:10.027133+0000","last_unstale":"2024-04-25T11:33:11.885908+0000","last_undegraded":"2024-04-25T11:33:11.885908+0000","last_fullsized":"2024-04-25T11:33:11.885908+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:19:58.338507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886920+0000","last_change":"2024-04-25T11:33:10.892588+0000","last_active":"2024-04-25T11:33:11.886920+0000","last_peered":"2024-04-25T11:33:11.886920+0000","last_clean":"2024-04-25T11:33:11.886920+0000","last_became_active":"2024-04-25T11:33:10.892088+0000","last_became_peered":"2024-04-25T11:33:10.892088+0000","last_unstale":"2024-04-25T11:33:11.886920+0000","last_undegraded":"2024-04-25T11:33:11.886920+0000","last_fullsized":"2024-04-25T11:33:11.886920+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:10.136222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"77'1","reported_seq":16,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.625741+0000","last_change":"2024-04-25T11:33:10.030801+0000","last_active":"2024-04-25T11:33:13.625741+0000","last_peered":"2024-04-25T11:33:13.625741+0000","last_clean":"2024-04-25T11:33:13.625741+0000","last_became_active":"2024-04-25T11:33:10.030466+0000","last_became_peered":"2024-04-25T11:33:10.030466+0000","last_unstale":"2024-04-25T11:33:13.625741+0000","last_undegraded":"2024-04-25T11:33:13.625741+0000","last_fullsized":"2024-04-25T11:33:13.625741+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:22:02.922309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886714+0000","last_change":"2024-04-25T11:33:10.894846+0000","last_active":"2024-04-25T11:33:11.886714+0000","last_peered":"2024-04-25T11:33:11.886714+0000","last_clean":"2024-04-25T11:33:11.886714+0000","last_became_active":"2024-04-25T11:33:10.894456+0000","last_became_peered":"2024-04-25T11:33:10.894456+0000","last_unstale":"2024-04-25T11:33:11.886714+0000","last_undegraded":"2024-04-25T11:33:11.886714+0000","last_fullsized":"2024-04-25T11:33:11.886714+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:11:16.265126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.871114+0000","last_change":"2024-04-25T11:33:10.891813+0000","last_active":"2024-04-25T11:33:11.871114+0000","last_peered":"2024-04-25T11:33:11.871114+0000","last_clean":"2024-04-25T11:33:11.871114+0000","last_became_active":"2024-04-25T11:33:10.887249+0000","last_became_peered":"2024-04-25T11:33:10.887249+0000","last_unstale":"2024-04-25T11:33:11.871114+0000","last_undegraded":"2024-04-25T11:33:11.871114+0000","last_fullsized":"2024-04-25T11:33:11.871114+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:10:10.895726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"77'4","reported_seq":19,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.627271+0000","last_change":"2024-04-25T11:33:10.037834+0000","last_active":"2024-04-25T11:33:13.627271+0000","last_peered":"2024-04-25T11:33:13.627271+0000","last_clean":"2024-04-25T11:33:13.627271+0000","last_became_active":"2024-04-25T11:33:10.037570+0000","last_became_peered":"2024-04-25T11:33:10.037570+0000","last_unstale":"2024-04-25T11:33:13.627271+0000","last_undegraded":"2024-04-25T11:33:13.627271+0000","last_fullsized":"2024-04-25T11:33:13.627271+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:12.795190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886167+0000","last_change":"2024-04-25T11:33:10.882663+0000","last_active":"2024-04-25T11:33:11.886167+0000","last_peered":"2024-04-25T11:33:11.886167+0000","last_clean":"2024-04-25T11:33:11.886167+0000","last_became_active":"2024-04-25T11:33:10.882300+0000","last_became_peered":"2024-04-25T11:33:10.882300+0000","last_unstale":"2024-04-25T11:33:11.886167+0000","last_undegraded":"2024-04-25T11:33:11.886167+0000","last_fullsized":"2024-04-25T11:33:11.886167+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:58:11.379654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"77'5","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:14.629891+0000","last_change":"2024-04-25T11:33:10.024598+0000","last_active":"2024-04-25T11:33:14.629891+0000","last_peered":"2024-04-25T11:33:14.629891+0000","last_clean":"2024-04-25T11:33:14.629891+0000","last_became_active":"2024-04-25T11:33:10.024229+0000","last_became_peered":"2024-04-25T11:33:10.024229+0000","last_unstale":"2024-04-25T11:33:14.629891+0000","last_undegraded":"2024-04-25T11:33:14.629891+0000","last_fullsized":"2024-04-25T11:33:14.629891+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:42:14.690581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":91,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886242+0000","last_change":"2024-04-25T11:32:01.310202+0000","last_active":"2024-04-25T11:33:11.886242+0000","last_peered":"2024-04-25T11:33:11.886242+0000","last_clean":"2024-04-25T11:33:11.886242+0000","last_became_active":"2024-04-25T11:32:01.305242+0000","last_became_peered":"2024-04-25T11:32:01.305242+0000","last_unstale":"2024-04-25T11:33:11.886242+0000","last_undegraded":"2024-04-25T11:33:11.886242+0000","last_fullsized":"2024-04-25T11:33:11.886242+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:28:43.374096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:28:43.374096+0000","last_clean_scrub_stamp":"2024-04-25T11:28:43.374096+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:56:18.551747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.869641+0000","last_change":"2024-04-25T11:33:10.889124+0000","last_active":"2024-04-25T11:33:11.869641+0000","last_peered":"2024-04-25T11:33:11.869641+0000","last_clean":"2024-04-25T11:33:11.869641+0000","last_became_active":"2024-04-25T11:33:10.888790+0000","last_became_peered":"2024-04-25T11:33:10.888790+0000","last_unstale":"2024-04-25T11:33:11.869641+0000","last_undegraded":"2024-04-25T11:33:11.869641+0000","last_fullsized":"2024-04-25T11:33:11.869641+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:33:44.999949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"77'3","reported_seq":18,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.628943+0000","last_change":"2024-04-25T11:33:10.031547+0000","last_active":"2024-04-25T11:33:13.628943+0000","last_peered":"2024-04-25T11:33:13.628943+0000","last_clean":"2024-04-25T11:33:13.628943+0000","last_became_active":"2024-04-25T11:33:10.031152+0000","last_became_peered":"2024-04-25T11:33:10.031152+0000","last_unstale":"2024-04-25T11:33:13.628943+0000","last_undegraded":"2024-04-25T11:33:13.628943+0000","last_fullsized":"2024-04-25T11:33:13.628943+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:10.646771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"77'1","reported_seq":16,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.042961+0000","last_change":"2024-04-25T11:33:10.020367+0000","last_active":"2024-04-25T11:33:13.042961+0000","last_peered":"2024-04-25T11:33:13.042961+0000","last_clean":"2024-04-25T11:33:13.042961+0000","last_became_active":"2024-04-25T11:33:10.020007+0000","last_became_peered":"2024-04-25T11:33:10.020007+0000","last_unstale":"2024-04-25T11:33:13.042961+0000","last_undegraded":"2024-04-25T11:33:13.042961+0000","last_fullsized":"2024-04-25T11:33:13.042961+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:48.134907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.869849+0000","last_change":"2024-04-25T11:33:10.887741+0000","last_active":"2024-04-25T11:33:11.869849+0000","last_peered":"2024-04-25T11:33:11.869849+0000","last_clean":"2024-04-25T11:33:11.869849+0000","last_became_active":"2024-04-25T11:33:10.886455+0000","last_became_peered":"2024-04-25T11:33:10.886455+0000","last_unstale":"2024-04-25T11:33:11.869849+0000","last_undegraded":"2024-04-25T11:33:11.869849+0000","last_fullsized":"2024-04-25T11:33:11.869849+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:37:16.842817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":16,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.022519+0000","last_change":"2024-04-25T11:33:10.025672+0000","last_active":"2024-04-25T11:33:13.022519+0000","last_peered":"2024-04-25T11:33:13.022519+0000","last_clean":"2024-04-25T11:33:13.022519+0000","last_became_active":"2024-04-25T11:33:10.025232+0000","last_became_peered":"2024-04-25T11:33:10.025232+0000","last_unstale":"2024-04-25T11:33:13.022519+0000","last_undegraded":"2024-04-25T11:33:13.022519+0000","last_fullsized":"2024-04-25T11:33:13.022519+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:06:26.579126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886307+0000","last_change":"2024-04-25T11:33:10.884692+0000","last_active":"2024-04-25T11:33:11.886307+0000","last_peered":"2024-04-25T11:33:11.886307+0000","last_clean":"2024-04-25T11:33:11.886307+0000","last_became_active":"2024-04-25T11:33:10.884346+0000","last_became_peered":"2024-04-25T11:33:10.884346+0000","last_unstale":"2024-04-25T11:33:11.886307+0000","last_undegraded":"2024-04-25T11:33:11.886307+0000","last_fullsized":"2024-04-25T11:33:11.886307+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:31:00.630410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"77'5","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.043236+0000","last_change":"2024-04-25T11:33:10.033872+0000","last_active":"2024-04-25T11:33:13.043236+0000","last_peered":"2024-04-25T11:33:13.043236+0000","last_clean":"2024-04-25T11:33:13.043236+0000","last_became_active":"2024-04-25T11:33:10.033435+0000","last_became_peered":"2024-04-25T11:33:10.033435+0000","last_unstale":"2024-04-25T11:33:13.043236+0000","last_undegraded":"2024-04-25T11:33:13.043236+0000","last_fullsized":"2024-04-25T11:33:13.043236+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:12.885215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.937704+0000","last_change":"2024-04-25T11:33:10.888707+0000","last_active":"2024-04-25T11:33:11.937704+0000","last_peered":"2024-04-25T11:33:11.937704+0000","last_clean":"2024-04-25T11:33:11.937704+0000","last_became_active":"2024-04-25T11:33:10.888274+0000","last_became_peered":"2024-04-25T11:33:10.888274+0000","last_unstale":"2024-04-25T11:33:11.937704+0000","last_undegraded":"2024-04-25T11:33:11.937704+0000","last_fullsized":"2024-04-25T11:33:11.937704+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:23:36.577447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"77'2","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.043069+0000","last_change":"2024-04-25T11:33:10.032384+0000","last_active":"2024-04-25T11:33:13.043069+0000","last_peered":"2024-04-25T11:33:13.043069+0000","last_clean":"2024-04-25T11:33:13.043069+0000","last_became_active":"2024-04-25T11:33:10.032035+0000","last_became_peered":"2024-04-25T11:33:10.032035+0000","last_unstale":"2024-04-25T11:33:13.043069+0000","last_undegraded":"2024-04-25T11:33:13.043069+0000","last_fullsized":"2024-04-25T11:33:13.043069+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:21:41.723120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.885748+0000","last_change":"2024-04-25T11:33:10.893366+0000","last_active":"2024-04-25T11:33:11.885748+0000","last_peered":"2024-04-25T11:33:11.885748+0000","last_clean":"2024-04-25T11:33:11.885748+0000","last_became_active":"2024-04-25T11:33:10.892988+0000","last_became_peered":"2024-04-25T11:33:10.892988+0000","last_unstale":"2024-04-25T11:33:11.885748+0000","last_undegraded":"2024-04-25T11:33:11.885748+0000","last_fullsized":"2024-04-25T11:33:11.885748+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:06.368216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"77'5","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:14.630917+0000","last_change":"2024-04-25T11:33:10.031456+0000","last_active":"2024-04-25T11:33:14.630917+0000","last_peered":"2024-04-25T11:33:14.630917+0000","last_clean":"2024-04-25T11:33:14.630917+0000","last_became_active":"2024-04-25T11:33:10.031115+0000","last_became_peered":"2024-04-25T11:33:10.031115+0000","last_unstale":"2024-04-25T11:33:14.630917+0000","last_undegraded":"2024-04-25T11:33:14.630917+0000","last_fullsized":"2024-04-25T11:33:14.630917+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:12:56.803277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887637+0000","last_change":"2024-04-25T11:33:10.893692+0000","last_active":"2024-04-25T11:33:11.887637+0000","last_peered":"2024-04-25T11:33:11.887637+0000","last_clean":"2024-04-25T11:33:11.887637+0000","last_became_active":"2024-04-25T11:33:10.893332+0000","last_became_peered":"2024-04-25T11:33:10.893332+0000","last_unstale":"2024-04-25T11:33:11.887637+0000","last_undegraded":"2024-04-25T11:33:11.887637+0000","last_fullsized":"2024-04-25T11:33:11.887637+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:01.277465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"77'2","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:14.038897+0000","last_change":"2024-04-25T11:33:10.023804+0000","last_active":"2024-04-25T11:33:14.038897+0000","last_peered":"2024-04-25T11:33:14.038897+0000","last_clean":"2024-04-25T11:33:14.038897+0000","last_became_active":"2024-04-25T11:33:10.023503+0000","last_became_peered":"2024-04-25T11:33:10.023503+0000","last_unstale":"2024-04-25T11:33:14.038897+0000","last_undegraded":"2024-04-25T11:33:14.038897+0000","last_fullsized":"2024-04-25T11:33:14.038897+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:34:48.074176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886206+0000","last_change":"2024-04-25T11:33:10.881501+0000","last_active":"2024-04-25T11:33:11.886206+0000","last_peered":"2024-04-25T11:33:11.886206+0000","last_clean":"2024-04-25T11:33:11.886206+0000","last_became_active":"2024-04-25T11:33:10.881104+0000","last_became_peered":"2024-04-25T11:33:10.881104+0000","last_unstale":"2024-04-25T11:33:11.886206+0000","last_undegraded":"2024-04-25T11:33:11.886206+0000","last_fullsized":"2024-04-25T11:33:11.886206+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:23:01.405920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"77'4","reported_seq":19,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:13.045053+0000","last_change":"2024-04-25T11:33:10.030857+0000","last_active":"2024-04-25T11:33:13.045053+0000","last_peered":"2024-04-25T11:33:13.045053+0000","last_clean":"2024-04-25T11:33:13.045053+0000","last_became_active":"2024-04-25T11:33:10.028879+0000","last_became_peered":"2024-04-25T11:33:10.028879+0000","last_unstale":"2024-04-25T11:33:13.045053+0000","last_undegraded":"2024-04-25T11:33:13.045053+0000","last_fullsized":"2024-04-25T11:33:13.045053+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:28.820919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887187+0000","last_change":"2024-04-25T11:33:10.893773+0000","last_active":"2024-04-25T11:33:11.887187+0000","last_peered":"2024-04-25T11:33:11.887187+0000","last_clean":"2024-04-25T11:33:11.887187+0000","last_became_active":"2024-04-25T11:33:10.893431+0000","last_became_peered":"2024-04-25T11:33:10.893431+0000","last_unstale":"2024-04-25T11:33:11.887187+0000","last_undegraded":"2024-04-25T11:33:11.887187+0000","last_fullsized":"2024-04-25T11:33:11.887187+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:15:15.998951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"77'5","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:14.630901+0000","last_change":"2024-04-25T11:33:10.032745+0000","last_active":"2024-04-25T11:33:14.630901+0000","last_peered":"2024-04-25T11:33:14.630901+0000","last_clean":"2024-04-25T11:33:14.630901+0000","last_became_active":"2024-04-25T11:33:10.032361+0000","last_became_peered":"2024-04-25T11:33:10.032361+0000","last_unstale":"2024-04-25T11:33:14.630901+0000","last_undegraded":"2024-04-25T11:33:14.630901+0000","last_fullsized":"2024-04-25T11:33:14.630901+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:14:46.555892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887521+0000","last_change":"2024-04-25T11:33:10.888084+0000","last_active":"2024-04-25T11:33:11.887521+0000","last_peered":"2024-04-25T11:33:11.887521+0000","last_clean":"2024-04-25T11:33:11.887521+0000","last_became_active":"2024-04-25T11:33:10.887679+0000","last_became_peered":"2024-04-25T11:33:10.887679+0000","last_unstale":"2024-04-25T11:33:11.887521+0000","last_undegraded":"2024-04-25T11:33:11.887521+0000","last_fullsized":"2024-04-25T11:33:11.887521+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:10:20.073630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"76'1","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.871256+0000","last_change":"2024-04-25T11:33:10.035355+0000","last_active":"2024-04-25T11:33:11.871256+0000","last_peered":"2024-04-25T11:33:11.871256+0000","last_clean":"2024-04-25T11:33:11.871256+0000","last_became_active":"2024-04-25T11:33:10.034724+0000","last_became_peered":"2024-04-25T11:33:10.034724+0000","last_unstale":"2024-04-25T11:33:11.871256+0000","last_undegraded":"2024-04-25T11:33:11.871256+0000","last_fullsized":"2024-04-25T11:33:11.871256+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:24:26.757803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887125+0000","last_change":"2024-04-25T11:33:10.899827+0000","last_active":"2024-04-25T11:33:11.887125+0000","last_peered":"2024-04-25T11:33:11.887125+0000","last_clean":"2024-04-25T11:33:11.887125+0000","last_became_active":"2024-04-25T11:33:10.899356+0000","last_became_peered":"2024-04-25T11:33:10.899356+0000","last_unstale":"2024-04-25T11:33:11.887125+0000","last_undegraded":"2024-04-25T11:33:11.887125+0000","last_fullsized":"2024-04-25T11:33:11.887125+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:04:17.323958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"76'2","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.869597+0000","last_change":"2024-04-25T11:33:10.034981+0000","last_active":"2024-04-25T11:33:11.869597+0000","last_peered":"2024-04-25T11:33:11.869597+0000","last_clean":"2024-04-25T11:33:11.869597+0000","last_became_active":"2024-04-25T11:33:10.034601+0000","last_became_peered":"2024-04-25T11:33:10.034601+0000","last_unstale":"2024-04-25T11:33:11.869597+0000","last_undegraded":"2024-04-25T11:33:11.869597+0000","last_fullsized":"2024-04-25T11:33:11.869597+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:19:14.710731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887479+0000","last_change":"2024-04-25T11:33:10.890020+0000","last_active":"2024-04-25T11:33:11.887479+0000","last_peered":"2024-04-25T11:33:11.887479+0000","last_clean":"2024-04-25T11:33:11.887479+0000","last_became_active":"2024-04-25T11:33:10.889558+0000","last_became_peered":"2024-04-25T11:33:10.889558+0000","last_unstale":"2024-04-25T11:33:11.887479+0000","last_undegraded":"2024-04-25T11:33:11.887479+0000","last_fullsized":"2024-04-25T11:33:11.887479+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:00:49.096818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887769+0000","last_change":"2024-04-25T11:33:10.886965+0000","last_active":"2024-04-25T11:33:11.887769+0000","last_peered":"2024-04-25T11:33:11.887769+0000","last_clean":"2024-04-25T11:33:11.887769+0000","last_became_active":"2024-04-25T11:33:10.886594+0000","last_became_peered":"2024-04-25T11:33:10.886594+0000","last_unstale":"2024-04-25T11:33:11.887769+0000","last_undegraded":"2024-04-25T11:33:11.887769+0000","last_fullsized":"2024-04-25T11:33:11.887769+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:58.924857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886516+0000","last_change":"2024-04-25T11:33:10.890760+0000","last_active":"2024-04-25T11:33:11.886516+0000","last_peered":"2024-04-25T11:33:11.886516+0000","last_clean":"2024-04-25T11:33:11.886516+0000","last_became_active":"2024-04-25T11:33:10.889643+0000","last_became_peered":"2024-04-25T11:33:10.889643+0000","last_unstale":"2024-04-25T11:33:11.886516+0000","last_undegraded":"2024-04-25T11:33:11.886516+0000","last_fullsized":"2024-04-25T11:33:11.886516+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:54.644853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886800+0000","last_change":"2024-04-25T11:33:10.887850+0000","last_active":"2024-04-25T11:33:11.886800+0000","last_peered":"2024-04-25T11:33:11.886800+0000","last_clean":"2024-04-25T11:33:11.886800+0000","last_became_active":"2024-04-25T11:33:10.882323+0000","last_became_peered":"2024-04-25T11:33:10.882323+0000","last_unstale":"2024-04-25T11:33:11.886800+0000","last_undegraded":"2024-04-25T11:33:11.886800+0000","last_fullsized":"2024-04-25T11:33:11.886800+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:24:00.509704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.937820+0000","last_change":"2024-04-25T11:33:10.883842+0000","last_active":"2024-04-25T11:33:11.937820+0000","last_peered":"2024-04-25T11:33:11.937820+0000","last_clean":"2024-04-25T11:33:11.937820+0000","last_became_active":"2024-04-25T11:33:10.883513+0000","last_became_peered":"2024-04-25T11:33:10.883513+0000","last_unstale":"2024-04-25T11:33:11.937820+0000","last_undegraded":"2024-04-25T11:33:11.937820+0000","last_fullsized":"2024-04-25T11:33:11.937820+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:47:09.731875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.869816+0000","last_change":"2024-04-25T11:33:10.890503+0000","last_active":"2024-04-25T11:33:11.869816+0000","last_peered":"2024-04-25T11:33:11.869816+0000","last_clean":"2024-04-25T11:33:11.869816+0000","last_became_active":"2024-04-25T11:33:10.890162+0000","last_became_peered":"2024-04-25T11:33:10.890162+0000","last_unstale":"2024-04-25T11:33:11.869816+0000","last_undegraded":"2024-04-25T11:33:11.869816+0000","last_fullsized":"2024-04-25T11:33:11.869816+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:36.462789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886869+0000","last_change":"2024-04-25T11:33:10.897109+0000","last_active":"2024-04-25T11:33:11.886869+0000","last_peered":"2024-04-25T11:33:11.886869+0000","last_clean":"2024-04-25T11:33:11.886869+0000","last_became_active":"2024-04-25T11:33:10.896138+0000","last_became_peered":"2024-04-25T11:33:10.896138+0000","last_unstale":"2024-04-25T11:33:11.886869+0000","last_undegraded":"2024-04-25T11:33:11.886869+0000","last_fullsized":"2024-04-25T11:33:11.886869+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:47:46.202355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.887556+0000","last_change":"2024-04-25T11:33:10.890397+0000","last_active":"2024-04-25T11:33:11.887556+0000","last_peered":"2024-04-25T11:33:11.887556+0000","last_clean":"2024-04-25T11:33:11.887556+0000","last_became_active":"2024-04-25T11:33:10.889580+0000","last_became_peered":"2024-04-25T11:33:10.889580+0000","last_unstale":"2024-04-25T11:33:11.887556+0000","last_undegraded":"2024-04-25T11:33:11.887556+0000","last_fullsized":"2024-04-25T11:33:11.887556+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:16:09.672131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886141+0000","last_change":"2024-04-25T11:33:10.873817+0000","last_active":"2024-04-25T11:33:11.886141+0000","last_peered":"2024-04-25T11:33:11.886141+0000","last_clean":"2024-04-25T11:33:11.886141+0000","last_became_active":"2024-04-25T11:33:10.873425+0000","last_became_peered":"2024-04-25T11:33:10.873425+0000","last_unstale":"2024-04-25T11:33:11.886141+0000","last_undegraded":"2024-04-25T11:33:11.886141+0000","last_fullsized":"2024-04-25T11:33:11.886141+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:52:26.279705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.871146+0000","last_change":"2024-04-25T11:33:10.880805+0000","last_active":"2024-04-25T11:33:11.871146+0000","last_peered":"2024-04-25T11:33:11.871146+0000","last_clean":"2024-04-25T11:33:11.871146+0000","last_became_active":"2024-04-25T11:33:10.880444+0000","last_became_peered":"2024-04-25T11:33:10.880444+0000","last_unstale":"2024-04-25T11:33:11.871146+0000","last_undegraded":"2024-04-25T11:33:11.871146+0000","last_fullsized":"2024-04-25T11:33:11.871146+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:57:03.602588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886780+0000","last_change":"2024-04-25T11:33:10.883690+0000","last_active":"2024-04-25T11:33:11.886780+0000","last_peered":"2024-04-25T11:33:11.886780+0000","last_clean":"2024-04-25T11:33:11.886780+0000","last_became_active":"2024-04-25T11:33:10.882965+0000","last_became_peered":"2024-04-25T11:33:10.882965+0000","last_unstale":"2024-04-25T11:33:11.886780+0000","last_undegraded":"2024-04-25T11:33:11.886780+0000","last_fullsized":"2024-04-25T11:33:11.886780+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:28.126631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886002+0000","last_change":"2024-04-25T11:33:10.897050+0000","last_active":"2024-04-25T11:33:11.886002+0000","last_peered":"2024-04-25T11:33:11.886002+0000","last_clean":"2024-04-25T11:33:11.886002+0000","last_became_active":"2024-04-25T11:33:10.896273+0000","last_became_peered":"2024-04-25T11:33:10.896273+0000","last_unstale":"2024-04-25T11:33:11.886002+0000","last_undegraded":"2024-04-25T11:33:11.886002+0000","last_fullsized":"2024-04-25T11:33:11.886002+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:45:53.839823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.869683+0000","last_change":"2024-04-25T11:33:10.897323+0000","last_active":"2024-04-25T11:33:11.869683+0000","last_peered":"2024-04-25T11:33:11.869683+0000","last_clean":"2024-04-25T11:33:11.869683+0000","last_became_active":"2024-04-25T11:33:10.896951+0000","last_became_peered":"2024-04-25T11:33:10.896951+0000","last_unstale":"2024-04-25T11:33:11.869683+0000","last_undegraded":"2024-04-25T11:33:11.869683+0000","last_fullsized":"2024-04-25T11:33:11.869683+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:24:31.208338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T11:33:11.886438+0000","last_change":"2024-04-25T11:33:10.897566+0000","last_active":"2024-04-25T11:33:11.886438+0000","last_peered":"2024-04-25T11:33:11.886438+0000","last_clean":"2024-04-25T11:33:11.886438+0000","last_became_active":"2024-04-25T11:33:10.897212+0000","last_became_peered":"2024-04-25T11:33:10.897212+0000","last_unstale":"2024-04-25T11:33:11.886438+0000","last_undegraded":"2024-04-25T11:33:11.886438+0000","last_fullsized":"2024-04-25T11:33:11.886438+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:55:29.995287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6145,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":13,"num_read_kb":11,"num_write":124,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":35962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108,"ondisk_log_size":108,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645325,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36148,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709004,"statfs":{"total":95995035648,"available":95958020096,"internally_reserved":0,"allocated":827392,"data_stored":395971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841554,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32596,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712556,"statfs":{"total":95995035648,"available":95961657344,"internally_reserved":0,"allocated":1454080,"data_stored":988325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037782,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27836,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717316,"statfs":{"total":95995035648,"available":95966531584,"internally_reserved":0,"allocated":827392,"data_stored":395971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234012,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27840,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717312,"statfs":{"total":95995035648,"available":95966527488,"internally_reserved":0,"allocated":835584,"data_stored":396015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430241,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32572,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712580,"statfs":{"total":95995035648,"available":95961681920,"internally_reserved":0,"allocated":1429504,"data_stored":986961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626470,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31984,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713168,"statfs":{"total":95995035648,"available":95962284032,"internally_reserved":0,"allocated":823296,"data_stored":395389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36152,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709000,"statfs":{"total":95995035648,"available":95958016000,"internally_reserved":0,"allocated":835584,"data_stored":397001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27848,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717304,"statfs":{"total":95995035648,"available":95966519296,"internally_reserved":0,"allocated":851968,"data_stored":396867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411386,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40344,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704808,"statfs":{"total":95995035648,"available":95953723392,"internally_reserved":0,"allocated":868352,"data_stored":399364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182452,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27848,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717304,"statfs":{"total":95995035648,"available":95966519296,"internally_reserved":0,"allocated":847872,"data_stored":395797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986224,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31996,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713156,"statfs":{"total":95995035648,"available":95962271744,"internally_reserved":0,"allocated":839680,"data_stored":396055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822699,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28428,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716724,"statfs":{"total":95995035648,"available":95965925376,"internally_reserved":0,"allocated":1445888,"data_stored":987233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":22867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T11:33:18.433 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T11:33:18.433 INFO:tasks.cephadm:Setting up client nodes... 2024-04-25T11:33:18.433 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-25T11:33:18.707 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:18.725 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:18 smithi090 ceph-mon[29575]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:33:18.725 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:18 smithi090 ceph-mon[29575]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T11:33:18.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:18 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2596995302' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:33:19.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:18 smithi138 ceph-mon[32328]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:33:19.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:18 smithi138 ceph-mon[32328]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T11:33:19.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:18 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2596995302' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:33:19.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:18 smithi156 ceph-mon[33045]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:33:19.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:18 smithi156 ceph-mon[33045]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T11:33:19.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:18 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2596995302' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:33:19.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:19 smithi090 ceph-mon[29575]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T11:33:19.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:19 smithi090 ceph-mon[29575]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T11:33:19.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:19 smithi090 ceph-mon[29575]: pgmap v278: 97 pgs: 97 active+clean; 619 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 11 KiB/s wr, 30 op/s 2024-04-25T11:33:20.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:19 smithi138 ceph-mon[32328]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T11:33:20.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:19 smithi138 ceph-mon[32328]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T11:33:20.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:19 smithi138 ceph-mon[32328]: pgmap v278: 97 pgs: 97 active+clean; 619 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 11 KiB/s wr, 30 op/s 2024-04-25T11:33:20.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:19 smithi156 ceph-mon[33045]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T11:33:20.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:19 smithi156 ceph-mon[33045]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T11:33:20.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:19 smithi156 ceph-mon[33045]: pgmap v278: 97 pgs: 97 active+clean; 619 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 11 KiB/s wr, 30 op/s 2024-04-25T11:33:20.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.425+0000 7f0c4a6cc700 1 -- 172.21.15.90:0/1670619055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c441048b0 msgr2=0x7f0c4410c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:20.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.425+0000 7f0c4a6cc700 1 --2- 172.21.15.90:0/1670619055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c441048b0 0x7f0c4410c5c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c00b0a0 tx=0x7f0c3c00b3b0 comp rx=0 tx=0).stop 2024-04-25T11:33:20.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.425+0000 7f0c4a6cc700 1 -- 172.21.15.90:0/1670619055 shutdown_connections 2024-04-25T11:33:20.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.425+0000 7f0c4a6cc700 1 --2- 172.21.15.90:0/1670619055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c441048b0 0x7f0c4410c5c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:20.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.425+0000 7f0c4a6cc700 1 --2- 172.21.15.90:0/1670619055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c44105880 0x7f0c44105c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:20.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.425+0000 7f0c4a6cc700 1 --2- 172.21.15.90:0/1670619055 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0c44103e30 0x7f0c44104240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:20.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.425+0000 7f0c4a6cc700 1 -- 172.21.15.90:0/1670619055 >> 172.21.15.90:0/1670619055 conn(0x7f0c440ff8a0 msgr2=0x7f0c44101d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:20.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.425+0000 7f0c4a6cc700 1 -- 172.21.15.90:0/1670619055 shutdown_connections 2024-04-25T11:33:20.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.425+0000 7f0c4a6cc700 1 -- 172.21.15.90:0/1670619055 wait complete. 2024-04-25T11:33:20.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.425+0000 7f0c4a6cc700 1 Processor -- start 2024-04-25T11:33:20.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.425+0000 7f0c4a6cc700 1 -- start start 2024-04-25T11:33:20.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.426+0000 7f0c4a6cc700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0c44103e30 0x7f0c44078440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:20.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.426+0000 7f0c4a6cc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c441048b0 0x7f0c44076af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:20.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.426+0000 7f0c4a6cc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c44105880 0x7f0c44077030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:20.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.426+0000 7f0c4a6cc700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0c4410fdc0 con 0x7f0c441048b0 2024-04-25T11:33:20.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.426+0000 7f0c4a6cc700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0c4410fc40 con 0x7f0c44103e30 2024-04-25T11:33:20.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.426+0000 7f0c4a6cc700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0c4410ff40 con 0x7f0c44105880 2024-04-25T11:33:20.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.426+0000 7f0c437fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c441048b0 0x7f0c44076af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:20.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.426+0000 7f0c437fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c441048b0 0x7f0c44076af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36490/0 (socket says 172.21.15.90:36490) 2024-04-25T11:33:20.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.426+0000 7f0c437fe700 1 -- 172.21.15.90:0/62705838 learned_addr learned my addr 172.21.15.90:0/62705838 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:20.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.426+0000 7f0c43fff700 1 --2- 172.21.15.90:0/62705838 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0c44103e30 0x7f0c44078440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:20.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.426+0000 7f0c48c69700 1 --2- 172.21.15.90:0/62705838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c44105880 0x7f0c44077030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:20.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.426+0000 7f0c437fe700 1 -- 172.21.15.90:0/62705838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c44105880 msgr2=0x7f0c44077030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:20.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.426+0000 7f0c437fe700 1 --2- 172.21.15.90:0/62705838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c44105880 0x7f0c44077030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:20.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.426+0000 7f0c437fe700 1 -- 172.21.15.90:0/62705838 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0c44103e30 msgr2=0x7f0c44078440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:20.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.426+0000 7f0c437fe700 1 --2- 172.21.15.90:0/62705838 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0c44103e30 0x7f0c44078440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:20.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.426+0000 7f0c437fe700 1 -- 172.21.15.90:0/62705838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c44077790 con 0x7f0c441048b0 2024-04-25T11:33:20.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.426+0000 7f0c437fe700 1 --2- 172.21.15.90:0/62705838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c441048b0 0x7f0c44076af0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0c3800d950 tx=0x7f0c3800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:20.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.426+0000 7f0c417fa700 1 -- 172.21.15.90:0/62705838 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c38016070 con 0x7f0c441048b0 2024-04-25T11:33:20.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.426+0000 7f0c4a6cc700 1 -- 172.21.15.90:0/62705838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c441a4940 con 0x7f0c441048b0 2024-04-25T11:33:20.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.426+0000 7f0c417fa700 1 -- 172.21.15.90:0/62705838 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c3800fd40 con 0x7f0c441048b0 2024-04-25T11:33:20.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.426+0000 7f0c417fa700 1 -- 172.21.15.90:0/62705838 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c3800e3b0 con 0x7f0c441048b0 2024-04-25T11:33:20.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.427+0000 7f0c4a6cc700 1 -- 172.21.15.90:0/62705838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c441a4e70 con 0x7f0c441048b0 2024-04-25T11:33:20.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.427+0000 7f0c4a6cc700 1 -- 172.21.15.90:0/62705838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c44051190 con 0x7f0c441048b0 2024-04-25T11:33:20.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.429+0000 7f0c417fa700 1 -- 172.21.15.90:0/62705838 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f0c38019070 con 0x7f0c441048b0 2024-04-25T11:33:20.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.430+0000 7f0c417fa700 1 --2- 172.21.15.90:0/62705838 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0c2c0afc10 0x7f0c2c0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:20.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.430+0000 7f0c43fff700 1 --2- 172.21.15.90:0/62705838 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0c2c0afc10 0x7f0c2c0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:20.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.430+0000 7f0c417fa700 1 -- 172.21.15.90:0/62705838 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7f0c380de6f0 con 0x7f0c441048b0 2024-04-25T11:33:20.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.430+0000 7f0c43fff700 1 --2- 172.21.15.90:0/62705838 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0c2c0afc10 0x7f0c2c0b2120 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0c44105270 tx=0x7f0c3400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:20.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.434+0000 7f0c417fa700 1 -- 172.21.15.90:0/62705838 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0c380a72a0 con 0x7f0c441048b0 2024-04-25T11:33:20.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.713+0000 7f0c4a6cc700 1 -- 172.21.15.90:0/62705838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f0c4402e880 con 0x7f0c441048b0 2024-04-25T11:33:20.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.716+0000 7f0c417fa700 1 -- 172.21.15.90:0/62705838 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) ==== 170+0+59 (secure 0 0 0) 0x7f0c380a6ba0 con 0x7f0c441048b0 2024-04-25T11:33:20.704 INFO:teuthology.orchestra.run.smithi090.stdout:[client.0] 2024-04-25T11:33:20.705 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQCAPypmRpKZKhAAOyR35dEvHHO2zmJM38oQ6w== 2024-04-25T11:33:20.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.718+0000 7f0c4a6cc700 1 -- 172.21.15.90:0/62705838 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0c2c0afc10 msgr2=0x7f0c2c0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:20.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.718+0000 7f0c4a6cc700 1 --2- 172.21.15.90:0/62705838 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0c2c0afc10 0x7f0c2c0b2120 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0c44105270 tx=0x7f0c3400a380 comp rx=0 tx=0).stop 2024-04-25T11:33:20.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.718+0000 7f0c4a6cc700 1 -- 172.21.15.90:0/62705838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c441048b0 msgr2=0x7f0c44076af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:20.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.718+0000 7f0c4a6cc700 1 --2- 172.21.15.90:0/62705838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c441048b0 0x7f0c44076af0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0c3800d950 tx=0x7f0c3800dc60 comp rx=0 tx=0).stop 2024-04-25T11:33:20.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.718+0000 7f0c4a6cc700 1 -- 172.21.15.90:0/62705838 shutdown_connections 2024-04-25T11:33:20.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.718+0000 7f0c4a6cc700 1 --2- 172.21.15.90:0/62705838 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0c2c0afc10 0x7f0c2c0b2120 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:20.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.718+0000 7f0c4a6cc700 1 --2- 172.21.15.90:0/62705838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c44105880 0x7f0c44077030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:20.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.718+0000 7f0c4a6cc700 1 --2- 172.21.15.90:0/62705838 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0c44103e30 0x7f0c44078440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:20.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.718+0000 7f0c4a6cc700 1 --2- 172.21.15.90:0/62705838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c441048b0 0x7f0c44076af0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:20.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.718+0000 7f0c4a6cc700 1 -- 172.21.15.90:0/62705838 >> 172.21.15.90:0/62705838 conn(0x7f0c440ff8a0 msgr2=0x7f0c441085a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:20.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.718+0000 7f0c4a6cc700 1 -- 172.21.15.90:0/62705838 shutdown_connections 2024-04-25T11:33:20.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:20.718+0000 7f0c4a6cc700 1 -- 172.21.15.90:0/62705838 wait complete. 2024-04-25T11:33:20.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:20 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/62705838' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T11:33:20.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:20 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/62705838' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T11:33:21.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:20 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/62705838' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T11:33:21.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:20 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/62705838' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T11:33:21.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:20 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/62705838' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T11:33:21.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:20 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/62705838' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T11:33:21.308 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T11:33:21.308 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-25T11:33:21.308 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T11:33:21.343 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T11:33:21.343 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-25T11:33:21.344 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-25T11:33:21.632 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:21.876 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:21 smithi090 ceph-mon[29575]: pgmap v279: 97 pgs: 97 active+clean; 621 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 11 KiB/s wr, 29 op/s 2024-04-25T11:33:22.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:21 smithi138 ceph-mon[32328]: pgmap v279: 97 pgs: 97 active+clean; 621 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 11 KiB/s wr, 29 op/s 2024-04-25T11:33:22.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:21 smithi156 ceph-mon[33045]: pgmap v279: 97 pgs: 97 active+clean; 621 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 11 KiB/s wr, 29 op/s 2024-04-25T11:33:23.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.120+0000 7fb5bfa32700 1 -- 172.21.15.90:0/2237008364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5b8104480 msgr2=0x7fb5b8104890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:23.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.120+0000 7fb5bfa32700 1 --2- 172.21.15.90:0/2237008364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5b8104480 0x7fb5b8104890 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb5b40099c0 tx=0x7fb5b4009cd0 comp rx=0 tx=0).stop 2024-04-25T11:33:23.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.120+0000 7fb5bfa32700 1 -- 172.21.15.90:0/2237008364 shutdown_connections 2024-04-25T11:33:23.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.120+0000 7fb5bfa32700 1 --2- 172.21.15.90:0/2237008364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5b8109700 0x7fb5b8111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:23.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.120+0000 7fb5bfa32700 1 --2- 172.21.15.90:0/2237008364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5b8104480 0x7fb5b8104890 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:23.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.120+0000 7fb5bfa32700 1 --2- 172.21.15.90:0/2237008364 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb5b8108db0 0x7fb5b81091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:23.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.120+0000 7fb5bfa32700 1 -- 172.21.15.90:0/2237008364 >> 172.21.15.90:0/2237008364 conn(0x7fb5b80ffd70 msgr2=0x7fb5b81021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:23.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.120+0000 7fb5bfa32700 1 -- 172.21.15.90:0/2237008364 shutdown_connections 2024-04-25T11:33:23.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.120+0000 7fb5bfa32700 1 -- 172.21.15.90:0/2237008364 wait complete. 2024-04-25T11:33:23.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.121+0000 7fb5bfa32700 1 Processor -- start 2024-04-25T11:33:23.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.121+0000 7fb5bfa32700 1 -- start start 2024-04-25T11:33:23.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.121+0000 7fb5bfa32700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5b8104480 0x7fb5b81a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:23.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.121+0000 7fb5bfa32700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5b8108db0 0x7fb5b81a0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:23.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.121+0000 7fb5bfa32700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb5b8109700 0x7fb5b819ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:23.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.121+0000 7fb5bfa32700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b81143a0 con 0x7fb5b8104480 2024-04-25T11:33:23.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.121+0000 7fb5bfa32700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b8114220 con 0x7fb5b8109700 2024-04-25T11:33:23.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.121+0000 7fb5bfa32700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b8114520 con 0x7fb5b8108db0 2024-04-25T11:33:23.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.121+0000 7fb5bd7ce700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5b8104480 0x7fb5b81a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:23.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.121+0000 7fb5bd7ce700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5b8104480 0x7fb5b81a09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33344/0 (socket says 172.21.15.90:33344) 2024-04-25T11:33:23.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.121+0000 7fb5bd7ce700 1 -- 172.21.15.90:0/2409462751 learned_addr learned my addr 172.21.15.90:0/2409462751 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:23.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.121+0000 7fb5bcfcd700 1 --2- 172.21.15.90:0/2409462751 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5b8108db0 0x7fb5b81a0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:23.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.121+0000 7fb5bdfcf700 1 --2- 172.21.15.90:0/2409462751 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb5b8109700 0x7fb5b819ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:23.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.121+0000 7fb5bd7ce700 1 -- 172.21.15.90:0/2409462751 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5b8108db0 msgr2=0x7fb5b81a0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:23.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.121+0000 7fb5bd7ce700 1 --2- 172.21.15.90:0/2409462751 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5b8108db0 0x7fb5b81a0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:23.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.121+0000 7fb5bd7ce700 1 -- 172.21.15.90:0/2409462751 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb5b8109700 msgr2=0x7fb5b819ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:23.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.121+0000 7fb5bd7ce700 1 --2- 172.21.15.90:0/2409462751 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb5b8109700 0x7fb5b819ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:23.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.121+0000 7fb5bd7ce700 1 -- 172.21.15.90:0/2409462751 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5b819b270 con 0x7fb5b8104480 2024-04-25T11:33:23.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.121+0000 7fb5bdfcf700 1 --2- 172.21.15.90:0/2409462751 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb5b8109700 0x7fb5b819ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:33:23.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.121+0000 7fb5bd7ce700 1 --2- 172.21.15.90:0/2409462751 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5b8104480 0x7fb5b81a09b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb5b400b580 tx=0x7fb5b40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:23.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.122+0000 7fb5ae7fc700 1 -- 172.21.15.90:0/2409462751 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b4027070 con 0x7fb5b8104480 2024-04-25T11:33:23.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.122+0000 7fb5ae7fc700 1 -- 172.21.15.90:0/2409462751 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5b40040a0 con 0x7fb5b8104480 2024-04-25T11:33:23.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.122+0000 7fb5ae7fc700 1 -- 172.21.15.90:0/2409462751 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b400f460 con 0x7fb5b8104480 2024-04-25T11:33:23.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.122+0000 7fb5bfa32700 1 -- 172.21.15.90:0/2409462751 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5b819b500 con 0x7fb5b8104480 2024-04-25T11:33:23.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.122+0000 7fb5bfa32700 1 -- 172.21.15.90:0/2409462751 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5b819b990 con 0x7fb5b8104480 2024-04-25T11:33:23.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.123+0000 7fb5bfa32700 1 -- 172.21.15.90:0/2409462751 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5b8105930 con 0x7fb5b8104480 2024-04-25T11:33:23.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.124+0000 7fb5ae7fc700 1 -- 172.21.15.90:0/2409462751 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fb5b402a070 con 0x7fb5b8104480 2024-04-25T11:33:23.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.125+0000 7fb5ae7fc700 1 --2- 172.21.15.90:0/2409462751 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb5a40afc10 0x7fb5a40b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:23.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.125+0000 7fb5bcfcd700 1 --2- 172.21.15.90:0/2409462751 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb5a40afc10 0x7fb5a40b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:23.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.125+0000 7fb5ae7fc700 1 -- 172.21.15.90:0/2409462751 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7fb5b40e00b0 con 0x7fb5b8104480 2024-04-25T11:33:23.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.125+0000 7fb5bcfcd700 1 --2- 172.21.15.90:0/2409462751 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb5a40afc10 0x7fb5a40b2120 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb5b819c270 tx=0x7fb5a8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:23.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.128+0000 7fb5ae7fc700 1 -- 172.21.15.90:0/2409462751 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb5b4031080 con 0x7fb5b8104480 2024-04-25T11:33:23.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.384+0000 7fb5bfa32700 1 -- 172.21.15.90:0/2409462751 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fb5b802e510 con 0x7fb5b8104480 2024-04-25T11:33:23.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.389+0000 7fb5ae7fc700 1 -- 172.21.15.90:0/2409462751 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+284474 (secure 0 0 0) 0x7fb5b4013070 con 0x7fb5b8104480 2024-04-25T11:33:23.382 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:33:23.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.392+0000 7fb5bfa32700 1 -- 172.21.15.90:0/2409462751 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb5a40afc10 msgr2=0x7fb5a40b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:23.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.392+0000 7fb5bfa32700 1 --2- 172.21.15.90:0/2409462751 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb5a40afc10 0x7fb5a40b2120 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb5b819c270 tx=0x7fb5a8009190 comp rx=0 tx=0).stop 2024-04-25T11:33:23.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.392+0000 7fb5bfa32700 1 -- 172.21.15.90:0/2409462751 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5b8104480 msgr2=0x7fb5b81a09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:23.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.392+0000 7fb5bfa32700 1 --2- 172.21.15.90:0/2409462751 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5b8104480 0x7fb5b81a09b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb5b400b580 tx=0x7fb5b40047b0 comp rx=0 tx=0).stop 2024-04-25T11:33:23.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.392+0000 7fb5bfa32700 1 -- 172.21.15.90:0/2409462751 shutdown_connections 2024-04-25T11:33:23.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.392+0000 7fb5bfa32700 1 --2- 172.21.15.90:0/2409462751 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb5b8109700 0x7fb5b819ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:23.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.392+0000 7fb5bfa32700 1 --2- 172.21.15.90:0/2409462751 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5b8104480 0x7fb5b81a09b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:23.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.392+0000 7fb5bfa32700 1 --2- 172.21.15.90:0/2409462751 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb5a40afc10 0x7fb5a40b2120 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:23.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.392+0000 7fb5bfa32700 1 --2- 172.21.15.90:0/2409462751 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5b8108db0 0x7fb5b81a0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:23.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.392+0000 7fb5bfa32700 1 -- 172.21.15.90:0/2409462751 >> 172.21.15.90:0/2409462751 conn(0x7fb5b80ffd70 msgr2=0x7fb5b8108590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:23.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.392+0000 7fb5bfa32700 1 -- 172.21.15.90:0/2409462751 shutdown_connections 2024-04-25T11:33:23.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:23.392+0000 7fb5bfa32700 1 -- 172.21.15.90:0/2409462751 wait complete. 2024-04-25T11:33:23.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:23 smithi090 ceph-mon[29575]: pgmap v280: 97 pgs: 97 active+clean; 621 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 9.0 KiB/s wr, 27 op/s 2024-04-25T11:33:23.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:23 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2409462751' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T11:33:24.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:23 smithi138 ceph-mon[32328]: pgmap v280: 97 pgs: 97 active+clean; 621 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 9.0 KiB/s wr, 27 op/s 2024-04-25T11:33:24.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:23 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2409462751' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T11:33:24.071 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":219449516},{"type":"v1","addr":"172.21.15.90:6801","nonce":219449516}]},"active_addr":"172.21.15.90:6801/219449516","active_change":"2024-04-25T11:26:16.655440+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.90:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":813599224}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":2202319376}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":888963322}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":478282854}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":2962667039}]}]} 2024-04-25T11:33:24.078 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-25T11:33:24.078 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-25T11:33:24.078 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T11:33:24.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:23 smithi156 ceph-mon[33045]: pgmap v280: 97 pgs: 97 active+clean; 621 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 9.0 KiB/s wr, 27 op/s 2024-04-25T11:33:24.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:23 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2409462751' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T11:33:24.319 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:25.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.904+0000 7fb45cd26700 1 -- 172.21.15.90:0/1554876833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb458104460 msgr2=0x7fb458104870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:25.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.904+0000 7fb45cd26700 1 --2- 172.21.15.90:0/1554876833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb458104460 0x7fb458104870 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb45000b0d0 tx=0x7fb45000b3e0 comp rx=0 tx=0).stop 2024-04-25T11:33:25.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.905+0000 7fb45cd26700 1 -- 172.21.15.90:0/1554876833 shutdown_connections 2024-04-25T11:33:25.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.905+0000 7fb45cd26700 1 --2- 172.21.15.90:0/1554876833 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb4581096e0 0x7fb458111040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:25.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.905+0000 7fb45cd26700 1 --2- 172.21.15.90:0/1554876833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb458104460 0x7fb458104870 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:25.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.905+0000 7fb45cd26700 1 --2- 172.21.15.90:0/1554876833 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb458108d90 0x7fb4581091a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:25.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.905+0000 7fb45cd26700 1 -- 172.21.15.90:0/1554876833 >> 172.21.15.90:0/1554876833 conn(0x7fb4580ffd40 msgr2=0x7fb458102160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:25.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.905+0000 7fb45cd26700 1 -- 172.21.15.90:0/1554876833 shutdown_connections 2024-04-25T11:33:25.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.905+0000 7fb45cd26700 1 -- 172.21.15.90:0/1554876833 wait complete. 2024-04-25T11:33:25.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.905+0000 7fb45cd26700 1 Processor -- start 2024-04-25T11:33:25.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.905+0000 7fb45cd26700 1 -- start start 2024-04-25T11:33:25.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.906+0000 7fb45cd26700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb458104460 0x7fb45819ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:25.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.906+0000 7fb45cd26700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb458108d90 0x7fb45819b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:25.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.906+0000 7fb45cd26700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb4581096e0 0x7fb45819f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:25.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.906+0000 7fb45cd26700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb458114400 con 0x7fb458104460 2024-04-25T11:33:25.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.906+0000 7fb45cd26700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb458114280 con 0x7fb4581096e0 2024-04-25T11:33:25.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.906+0000 7fb45cd26700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb458114580 con 0x7fb458108d90 2024-04-25T11:33:25.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.906+0000 7fb45659c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb458104460 0x7fb45819ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:25.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.906+0000 7fb45659c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb458104460 0x7fb45819ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33370/0 (socket says 172.21.15.90:33370) 2024-04-25T11:33:25.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.906+0000 7fb45659c700 1 -- 172.21.15.90:0/1753430399 learned_addr learned my addr 172.21.15.90:0/1753430399 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:25.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.906+0000 7fb455d9b700 1 --2- 172.21.15.90:0/1753430399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb458108d90 0x7fb45819b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:25.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.906+0000 7fb456d9d700 1 --2- 172.21.15.90:0/1753430399 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb4581096e0 0x7fb45819f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:25.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.906+0000 7fb45659c700 1 -- 172.21.15.90:0/1753430399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb458108d90 msgr2=0x7fb45819b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:25.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.906+0000 7fb45659c700 1 --2- 172.21.15.90:0/1753430399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb458108d90 0x7fb45819b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:25.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.906+0000 7fb45659c700 1 -- 172.21.15.90:0/1753430399 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb4581096e0 msgr2=0x7fb45819f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:25.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.906+0000 7fb45659c700 1 --2- 172.21.15.90:0/1753430399 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb4581096e0 0x7fb45819f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:25.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.906+0000 7fb45659c700 1 -- 172.21.15.90:0/1753430399 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb45819fcb0 con 0x7fb458104460 2024-04-25T11:33:25.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.906+0000 7fb456d9d700 1 --2- 172.21.15.90:0/1753430399 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb4581096e0 0x7fb45819f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:33:25.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.906+0000 7fb45659c700 1 --2- 172.21.15.90:0/1753430399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb458104460 0x7fb45819ae30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb450007a20 tx=0x7fb450007b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:25.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.906+0000 7fb4477fe700 1 -- 172.21.15.90:0/1753430399 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45001f070 con 0x7fb458104460 2024-04-25T11:33:25.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.907+0000 7fb4477fe700 1 -- 172.21.15.90:0/1753430399 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb450003c00 con 0x7fb458104460 2024-04-25T11:33:25.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.907+0000 7fb45cd26700 1 -- 172.21.15.90:0/1753430399 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb45819ff40 con 0x7fb458104460 2024-04-25T11:33:25.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.907+0000 7fb4477fe700 1 -- 172.21.15.90:0/1753430399 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45001b730 con 0x7fb458104460 2024-04-25T11:33:25.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.907+0000 7fb45cd26700 1 -- 172.21.15.90:0/1753430399 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4581a0420 con 0x7fb458104460 2024-04-25T11:33:25.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.908+0000 7fb45cd26700 1 -- 172.21.15.90:0/1753430399 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb458051190 con 0x7fb458104460 2024-04-25T11:33:25.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.909+0000 7fb4477fe700 1 -- 172.21.15.90:0/1753430399 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fb450022070 con 0x7fb458104460 2024-04-25T11:33:25.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.909+0000 7fb4477fe700 1 --2- 172.21.15.90:0/1753430399 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb4400afc10 0x7fb4400b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:25.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.910+0000 7fb455d9b700 1 --2- 172.21.15.90:0/1753430399 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb4400afc10 0x7fb4400b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:25.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.910+0000 7fb4477fe700 1 -- 172.21.15.90:0/1753430399 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7fb45000f020 con 0x7fb458104460 2024-04-25T11:33:25.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.910+0000 7fb455d9b700 1 --2- 172.21.15.90:0/1753430399 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb4400afc10 0x7fb4400b2120 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb44c006fd0 tx=0x7fb44c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:25.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:25.914+0000 7fb4477fe700 1 -- 172.21.15.90:0/1753430399 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb450029020 con 0x7fb458104460 2024-04-25T11:33:25.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:25 smithi090 ceph-mon[29575]: pgmap v281: 97 pgs: 97 active+clean; 626 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.7 KiB/s wr, 28 op/s 2024-04-25T11:33:26.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:25 smithi138 ceph-mon[32328]: pgmap v281: 97 pgs: 97 active+clean; 626 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.7 KiB/s wr, 28 op/s 2024-04-25T11:33:26.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:25 smithi156 ceph-mon[33045]: pgmap v281: 97 pgs: 97 active+clean; 626 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.7 KiB/s wr, 28 op/s 2024-04-25T11:33:26.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:26.114+0000 7fb45cd26700 1 -- 172.21.15.90:0/1753430399 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb45802e510 con 0x7fb458104460 2024-04-25T11:33:26.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:26.116+0000 7fb4477fe700 1 -- 172.21.15.90:0/1753430399 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+22238 (secure 0 0 0) 0x7fb4500a8ca0 con 0x7fb458104460 2024-04-25T11:33:26.104 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:33:26.104 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":77,"fsid":"58bd4e7a-02f6-11ef-bc93-c7b262605968","created":"2024-04-25T11:25:23.574550+0000","modified":"2024-04-25T11:33:11.866090+0000","last_up_change":"2024-04-25T11:32:23.852211+0000","last_in_change":"2024-04-25T11:32:04.188507+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:28:42.746649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:33:08.442925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:33:09.352886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e54d888-e67b-4951-8ae7-e355fb6b10a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6803","nonce":1201629159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6805","nonce":1201629159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6809","nonce":1201629159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6807","nonce":1201629159}]},"public_addr":"172.21.15.90:6803/1201629159","cluster_addr":"172.21.15.90:6805/1201629159","heartbeat_back_addr":"172.21.15.90:6809/1201629159","heartbeat_front_addr":"172.21.15.90:6807/1201629159","state":["exists","up"]},{"osd":1,"uuid":"91a71c78-dae9-4bc7-9f6f-9e55cfefbe28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6801","nonce":3976851933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6803","nonce":3976851933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6807","nonce":3976851933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6805","nonce":3976851933}]},"public_addr":"172.21.15.138:6801/3976851933","cluster_addr":"172.21.15.138:6803/3976851933","heartbeat_back_addr":"172.21.15.138:6807/3976851933","heartbeat_front_addr":"172.21.15.138:6805/3976851933","state":["exists","up"]},{"osd":2,"uuid":"daff6c7f-80fa-4859-9339-ef7a5e840832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6801","nonce":1505807670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6803","nonce":1505807670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6807","nonce":1505807670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6805","nonce":1505807670}]},"public_addr":"172.21.15.156:6801/1505807670","cluster_addr":"172.21.15.156:6803/1505807670","heartbeat_back_addr":"172.21.15.156:6807/1505807670","heartbeat_front_addr":"172.21.15.156:6805/1505807670","state":["exists","up"]},{"osd":3,"uuid":"33fd94ca-7465-401a-bc1e-706b6b31b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6811","nonce":2520012383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6813","nonce":2520012383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6817","nonce":2520012383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6815","nonce":2520012383}]},"public_addr":"172.21.15.90:6811/2520012383","cluster_addr":"172.21.15.90:6813/2520012383","heartbeat_back_addr":"172.21.15.90:6817/2520012383","heartbeat_front_addr":"172.21.15.90:6815/2520012383","state":["exists","up"]},{"osd":4,"uuid":"c5326212-750c-49fd-96e3-d1e62bb4b7f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6809","nonce":3752678279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6811","nonce":3752678279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6815","nonce":3752678279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6813","nonce":3752678279}]},"public_addr":"172.21.15.138:6809/3752678279","cluster_addr":"172.21.15.138:6811/3752678279","heartbeat_back_addr":"172.21.15.138:6815/3752678279","heartbeat_front_addr":"172.21.15.138:6813/3752678279","state":["exists","up"]},{"osd":5,"uuid":"d04a1156-820b-4b43-be89-e16a8910c3aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6809","nonce":3591210191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6811","nonce":3591210191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6815","nonce":3591210191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6813","nonce":3591210191}]},"public_addr":"172.21.15.156:6809/3591210191","cluster_addr":"172.21.15.156:6811/3591210191","heartbeat_back_addr":"172.21.15.156:6815/3591210191","heartbeat_front_addr":"172.21.15.156:6813/3591210191","state":["exists","up"]},{"osd":6,"uuid":"75675aa3-e978-490c-8205-13d9868a3d6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6819","nonce":2364955456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6821","nonce":2364955456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6825","nonce":2364955456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6823","nonce":2364955456}]},"public_addr":"172.21.15.90:6819/2364955456","cluster_addr":"172.21.15.90:6821/2364955456","heartbeat_back_addr":"172.21.15.90:6825/2364955456","heartbeat_front_addr":"172.21.15.90:6823/2364955456","state":["exists","up"]},{"osd":7,"uuid":"1a95fdf5-f548-4271-bbcd-e330f7955eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6817","nonce":4012476387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6819","nonce":4012476387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6823","nonce":4012476387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6821","nonce":4012476387}]},"public_addr":"172.21.15.138:6817/4012476387","cluster_addr":"172.21.15.138:6819/4012476387","heartbeat_back_addr":"172.21.15.138:6823/4012476387","heartbeat_front_addr":"172.21.15.138:6821/4012476387","state":["exists","up"]},{"osd":8,"uuid":"605e10b5-5806-4c1f-8e28-a7994d34ddce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6817","nonce":101092224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6819","nonce":101092224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6823","nonce":101092224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6821","nonce":101092224}]},"public_addr":"172.21.15.156:6817/101092224","cluster_addr":"172.21.15.156:6819/101092224","heartbeat_back_addr":"172.21.15.156:6823/101092224","heartbeat_front_addr":"172.21.15.156:6821/101092224","state":["exists","up"]},{"osd":9,"uuid":"d70fdf5e-9d0f-483d-a7d8-505b79b8a418","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6827","nonce":1171265966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6829","nonce":1171265966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6833","nonce":1171265966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6831","nonce":1171265966}]},"public_addr":"172.21.15.90:6827/1171265966","cluster_addr":"172.21.15.90:6829/1171265966","heartbeat_back_addr":"172.21.15.90:6833/1171265966","heartbeat_front_addr":"172.21.15.90:6831/1171265966","state":["exists","up"]},{"osd":10,"uuid":"1ec86c7d-c3a4-432d-9664-8e40836e1e20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6825","nonce":4264666232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6827","nonce":4264666232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6831","nonce":4264666232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6829","nonce":4264666232}]},"public_addr":"172.21.15.138:6825/4264666232","cluster_addr":"172.21.15.138:6827/4264666232","heartbeat_back_addr":"172.21.15.138:6831/4264666232","heartbeat_front_addr":"172.21.15.138:6829/4264666232","state":["exists","up"]},{"osd":11,"uuid":"51cbe729-292e-4d7b-b8e9-e1e2821ffe76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6825","nonce":1224251744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6827","nonce":1224251744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6831","nonce":1224251744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6829","nonce":1224251744}]},"public_addr":"172.21.15.156:6825/1224251744","cluster_addr":"172.21.15.156:6827/1224251744","heartbeat_back_addr":"172.21.15.156:6831/1224251744","heartbeat_front_addr":"172.21.15.156:6829/1224251744","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:27:50.794226+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:15.030278+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:38.518163+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:03.107351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:27.500707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:51.463545+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:16.015076+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:41.182071+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:05.362309+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:31.048936+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:56.488509+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:32:22.129178+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/2381633953":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6800/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/2190846900":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1707792941":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6800/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6801/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/6061052":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1849633392":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6801/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/2277490702":"2024-04-26T11:26:16.655278+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:33:26.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:26.118+0000 7fb45cd26700 1 -- 172.21.15.90:0/1753430399 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb4400afc10 msgr2=0x7fb4400b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:26.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:26.118+0000 7fb45cd26700 1 --2- 172.21.15.90:0/1753430399 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb4400afc10 0x7fb4400b2120 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb44c006fd0 tx=0x7fb44c008040 comp rx=0 tx=0).stop 2024-04-25T11:33:26.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:26.118+0000 7fb45cd26700 1 -- 172.21.15.90:0/1753430399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb458104460 msgr2=0x7fb45819ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:26.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:26.118+0000 7fb45cd26700 1 --2- 172.21.15.90:0/1753430399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb458104460 0x7fb45819ae30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb450007a20 tx=0x7fb450007b00 comp rx=0 tx=0).stop 2024-04-25T11:33:26.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:26.118+0000 7fb45cd26700 1 -- 172.21.15.90:0/1753430399 shutdown_connections 2024-04-25T11:33:26.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:26.118+0000 7fb45cd26700 1 --2- 172.21.15.90:0/1753430399 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb4581096e0 0x7fb45819f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:26.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:26.118+0000 7fb45cd26700 1 --2- 172.21.15.90:0/1753430399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb458104460 0x7fb45819ae30 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:26.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:26.118+0000 7fb45cd26700 1 --2- 172.21.15.90:0/1753430399 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb4400afc10 0x7fb4400b2120 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:26.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:26.118+0000 7fb45cd26700 1 --2- 172.21.15.90:0/1753430399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb458108d90 0x7fb45819b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:26.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:26.118+0000 7fb45cd26700 1 -- 172.21.15.90:0/1753430399 >> 172.21.15.90:0/1753430399 conn(0x7fb4580ffd40 msgr2=0x7fb458100590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:26.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:26.118+0000 7fb45cd26700 1 -- 172.21.15.90:0/1753430399 shutdown_connections 2024-04-25T11:33:26.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:26.118+0000 7fb45cd26700 1 -- 172.21.15.90:0/1753430399 wait complete. 2024-04-25T11:33:26.696 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-25T11:33:26.697 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T11:33:26.940 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:27.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:26 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1753430399' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:33:27.062 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:26 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1753430399' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:33:27.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:26 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1753430399' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:33:27.990 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:27 smithi090 ceph-mon[29575]: pgmap v282: 97 pgs: 97 active+clean; 626 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.7 KiB/s wr, 22 op/s 2024-04-25T11:33:28.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:27 smithi138 ceph-mon[32328]: pgmap v282: 97 pgs: 97 active+clean; 626 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.7 KiB/s wr, 22 op/s 2024-04-25T11:33:28.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:27 smithi156 ceph-mon[33045]: pgmap v282: 97 pgs: 97 active+clean; 626 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.7 KiB/s wr, 22 op/s 2024-04-25T11:33:28.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.504+0000 7f6d5ee28700 1 -- 172.21.15.90:0/40951317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d581085e0 msgr2=0x7f6d5810ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:28.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.504+0000 7f6d5ee28700 1 --2- 172.21.15.90:0/40951317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d581085e0 0x7f6d5810ff40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6d5000b0d0 tx=0x7f6d5000b3e0 comp rx=0 tx=0).stop 2024-04-25T11:33:28.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.504+0000 7f6d5ee28700 1 -- 172.21.15.90:0/40951317 shutdown_connections 2024-04-25T11:33:28.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.504+0000 7f6d5ee28700 1 --2- 172.21.15.90:0/40951317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d581085e0 0x7f6d5810ff40 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:28.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.504+0000 7f6d5ee28700 1 --2- 172.21.15.90:0/40951317 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6d58103360 0x7f6d58103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:28.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.504+0000 7f6d5ee28700 1 --2- 172.21.15.90:0/40951317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d58107c90 0x7f6d581080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:28.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.504+0000 7f6d5ee28700 1 -- 172.21.15.90:0/40951317 >> 172.21.15.90:0/40951317 conn(0x7f6d580fecb0 msgr2=0x7f6d581010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:28.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.504+0000 7f6d5ee28700 1 -- 172.21.15.90:0/40951317 shutdown_connections 2024-04-25T11:33:28.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.504+0000 7f6d5ee28700 1 -- 172.21.15.90:0/40951317 wait complete. 2024-04-25T11:33:28.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.505+0000 7f6d5ee28700 1 Processor -- start 2024-04-25T11:33:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.505+0000 7f6d5ee28700 1 -- start start 2024-04-25T11:33:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.505+0000 7f6d5ee28700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6d58103360 0x7f6d5819ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.505+0000 7f6d5ee28700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d58107c90 0x7f6d5819bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.505+0000 7f6d5ee28700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d581085e0 0x7f6d5819c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.505+0000 7f6d5ee28700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6d58113210 con 0x7f6d58107c90 2024-04-25T11:33:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.505+0000 7f6d5ee28700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6d58113090 con 0x7f6d58103360 2024-04-25T11:33:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.505+0000 7f6d5ee28700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6d58113390 con 0x7f6d581085e0 2024-04-25T11:33:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.505+0000 7f6d4ffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d58107c90 0x7f6d5819bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.505+0000 7f6d4ffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d58107c90 0x7f6d5819bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33388/0 (socket says 172.21.15.90:33388) 2024-04-25T11:33:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.505+0000 7f6d4ffff700 1 -- 172.21.15.90:0/543800320 learned_addr learned my addr 172.21.15.90:0/543800320 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.505+0000 7f6d5cbc4700 1 --2- 172.21.15.90:0/543800320 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6d58103360 0x7f6d5819ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.505+0000 7f6d5d3c5700 1 --2- 172.21.15.90:0/543800320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d581085e0 0x7f6d5819c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:28.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.506+0000 7f6d4ffff700 1 -- 172.21.15.90:0/543800320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d581085e0 msgr2=0x7f6d5819c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:28.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.506+0000 7f6d4ffff700 1 --2- 172.21.15.90:0/543800320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d581085e0 0x7f6d5819c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:28.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.506+0000 7f6d4ffff700 1 -- 172.21.15.90:0/543800320 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6d58103360 msgr2=0x7f6d5819ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:28.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.506+0000 7f6d4ffff700 1 --2- 172.21.15.90:0/543800320 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6d58103360 0x7f6d5819ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:28.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.506+0000 7f6d4ffff700 1 -- 172.21.15.90:0/543800320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d581a6de0 con 0x7f6d58107c90 2024-04-25T11:33:28.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.506+0000 7f6d4ffff700 1 --2- 172.21.15.90:0/543800320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d58107c90 0x7f6d5819bfb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6d4800d980 tx=0x7f6d4800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:28.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.506+0000 7f6d4dffb700 1 -- 172.21.15.90:0/543800320 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d48017070 con 0x7f6d58107c90 2024-04-25T11:33:28.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.506+0000 7f6d5ee28700 1 -- 172.21.15.90:0/543800320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d581a70d0 con 0x7f6d58107c90 2024-04-25T11:33:28.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.506+0000 7f6d4dffb700 1 -- 172.21.15.90:0/543800320 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d4800fd40 con 0x7f6d58107c90 2024-04-25T11:33:28.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.506+0000 7f6d4dffb700 1 -- 172.21.15.90:0/543800320 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4800e3b0 con 0x7f6d58107c90 2024-04-25T11:33:28.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.506+0000 7f6d5ee28700 1 -- 172.21.15.90:0/543800320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d581a7660 con 0x7f6d58107c90 2024-04-25T11:33:28.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.507+0000 7f6d5ee28700 1 -- 172.21.15.90:0/543800320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d58051190 con 0x7f6d58107c90 2024-04-25T11:33:28.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.508+0000 7f6d4dffb700 1 -- 172.21.15.90:0/543800320 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f6d48010610 con 0x7f6d58107c90 2024-04-25T11:33:28.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.509+0000 7f6d4dffb700 1 --2- 172.21.15.90:0/543800320 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6d400afc10 0x7f6d400b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:28.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.509+0000 7f6d5cbc4700 1 --2- 172.21.15.90:0/543800320 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6d400afc10 0x7f6d400b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:28.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.509+0000 7f6d4dffb700 1 -- 172.21.15.90:0/543800320 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7f6d480de620 con 0x7f6d58107c90 2024-04-25T11:33:28.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.510+0000 7f6d5cbc4700 1 --2- 172.21.15.90:0/543800320 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6d400afc10 0x7f6d400b2120 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6d54005950 tx=0x7f6d5400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:28.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.513+0000 7f6d4dffb700 1 -- 172.21.15.90:0/543800320 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6d480a7250 con 0x7f6d58107c90 2024-04-25T11:33:28.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.713+0000 7f6d5ee28700 1 -- 172.21.15.90:0/543800320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6d58075ed0 con 0x7f6d58107c90 2024-04-25T11:33:28.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.714+0000 7f6d4dffb700 1 -- 172.21.15.90:0/543800320 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+22238 (secure 0 0 0) 0x7f6d480a6b50 con 0x7f6d58107c90 2024-04-25T11:33:28.704 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:33:28.704 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":77,"fsid":"58bd4e7a-02f6-11ef-bc93-c7b262605968","created":"2024-04-25T11:25:23.574550+0000","modified":"2024-04-25T11:33:11.866090+0000","last_up_change":"2024-04-25T11:32:23.852211+0000","last_in_change":"2024-04-25T11:32:04.188507+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:28:42.746649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:33:08.442925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:33:09.352886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e54d888-e67b-4951-8ae7-e355fb6b10a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6803","nonce":1201629159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6805","nonce":1201629159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6809","nonce":1201629159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6807","nonce":1201629159}]},"public_addr":"172.21.15.90:6803/1201629159","cluster_addr":"172.21.15.90:6805/1201629159","heartbeat_back_addr":"172.21.15.90:6809/1201629159","heartbeat_front_addr":"172.21.15.90:6807/1201629159","state":["exists","up"]},{"osd":1,"uuid":"91a71c78-dae9-4bc7-9f6f-9e55cfefbe28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6801","nonce":3976851933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6803","nonce":3976851933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6807","nonce":3976851933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6805","nonce":3976851933}]},"public_addr":"172.21.15.138:6801/3976851933","cluster_addr":"172.21.15.138:6803/3976851933","heartbeat_back_addr":"172.21.15.138:6807/3976851933","heartbeat_front_addr":"172.21.15.138:6805/3976851933","state":["exists","up"]},{"osd":2,"uuid":"daff6c7f-80fa-4859-9339-ef7a5e840832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6801","nonce":1505807670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6803","nonce":1505807670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6807","nonce":1505807670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6805","nonce":1505807670}]},"public_addr":"172.21.15.156:6801/1505807670","cluster_addr":"172.21.15.156:6803/1505807670","heartbeat_back_addr":"172.21.15.156:6807/1505807670","heartbeat_front_addr":"172.21.15.156:6805/1505807670","state":["exists","up"]},{"osd":3,"uuid":"33fd94ca-7465-401a-bc1e-706b6b31b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6811","nonce":2520012383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6813","nonce":2520012383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6817","nonce":2520012383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6815","nonce":2520012383}]},"public_addr":"172.21.15.90:6811/2520012383","cluster_addr":"172.21.15.90:6813/2520012383","heartbeat_back_addr":"172.21.15.90:6817/2520012383","heartbeat_front_addr":"172.21.15.90:6815/2520012383","state":["exists","up"]},{"osd":4,"uuid":"c5326212-750c-49fd-96e3-d1e62bb4b7f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6809","nonce":3752678279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6811","nonce":3752678279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6815","nonce":3752678279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6813","nonce":3752678279}]},"public_addr":"172.21.15.138:6809/3752678279","cluster_addr":"172.21.15.138:6811/3752678279","heartbeat_back_addr":"172.21.15.138:6815/3752678279","heartbeat_front_addr":"172.21.15.138:6813/3752678279","state":["exists","up"]},{"osd":5,"uuid":"d04a1156-820b-4b43-be89-e16a8910c3aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6809","nonce":3591210191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6811","nonce":3591210191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6815","nonce":3591210191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6813","nonce":3591210191}]},"public_addr":"172.21.15.156:6809/3591210191","cluster_addr":"172.21.15.156:6811/3591210191","heartbeat_back_addr":"172.21.15.156:6815/3591210191","heartbeat_front_addr":"172.21.15.156:6813/3591210191","state":["exists","up"]},{"osd":6,"uuid":"75675aa3-e978-490c-8205-13d9868a3d6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6819","nonce":2364955456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6821","nonce":2364955456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6825","nonce":2364955456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6823","nonce":2364955456}]},"public_addr":"172.21.15.90:6819/2364955456","cluster_addr":"172.21.15.90:6821/2364955456","heartbeat_back_addr":"172.21.15.90:6825/2364955456","heartbeat_front_addr":"172.21.15.90:6823/2364955456","state":["exists","up"]},{"osd":7,"uuid":"1a95fdf5-f548-4271-bbcd-e330f7955eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6817","nonce":4012476387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6819","nonce":4012476387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6823","nonce":4012476387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6821","nonce":4012476387}]},"public_addr":"172.21.15.138:6817/4012476387","cluster_addr":"172.21.15.138:6819/4012476387","heartbeat_back_addr":"172.21.15.138:6823/4012476387","heartbeat_front_addr":"172.21.15.138:6821/4012476387","state":["exists","up"]},{"osd":8,"uuid":"605e10b5-5806-4c1f-8e28-a7994d34ddce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6817","nonce":101092224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6819","nonce":101092224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6823","nonce":101092224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6821","nonce":101092224}]},"public_addr":"172.21.15.156:6817/101092224","cluster_addr":"172.21.15.156:6819/101092224","heartbeat_back_addr":"172.21.15.156:6823/101092224","heartbeat_front_addr":"172.21.15.156:6821/101092224","state":["exists","up"]},{"osd":9,"uuid":"d70fdf5e-9d0f-483d-a7d8-505b79b8a418","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6827","nonce":1171265966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6829","nonce":1171265966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6833","nonce":1171265966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6831","nonce":1171265966}]},"public_addr":"172.21.15.90:6827/1171265966","cluster_addr":"172.21.15.90:6829/1171265966","heartbeat_back_addr":"172.21.15.90:6833/1171265966","heartbeat_front_addr":"172.21.15.90:6831/1171265966","state":["exists","up"]},{"osd":10,"uuid":"1ec86c7d-c3a4-432d-9664-8e40836e1e20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6825","nonce":4264666232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6827","nonce":4264666232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6831","nonce":4264666232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6829","nonce":4264666232}]},"public_addr":"172.21.15.138:6825/4264666232","cluster_addr":"172.21.15.138:6827/4264666232","heartbeat_back_addr":"172.21.15.138:6831/4264666232","heartbeat_front_addr":"172.21.15.138:6829/4264666232","state":["exists","up"]},{"osd":11,"uuid":"51cbe729-292e-4d7b-b8e9-e1e2821ffe76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6825","nonce":1224251744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6827","nonce":1224251744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6831","nonce":1224251744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6829","nonce":1224251744}]},"public_addr":"172.21.15.156:6825/1224251744","cluster_addr":"172.21.15.156:6827/1224251744","heartbeat_back_addr":"172.21.15.156:6831/1224251744","heartbeat_front_addr":"172.21.15.156:6829/1224251744","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:27:50.794226+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:15.030278+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:38.518163+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:03.107351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:27.500707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:51.463545+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:16.015076+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:41.182071+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:05.362309+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:31.048936+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:56.488509+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:32:22.129178+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/2381633953":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6800/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/2190846900":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1707792941":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6800/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6801/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/6061052":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1849633392":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6801/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/2277490702":"2024-04-26T11:26:16.655278+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:33:28.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.717+0000 7f6d5ee28700 1 -- 172.21.15.90:0/543800320 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6d400afc10 msgr2=0x7f6d400b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:28.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.717+0000 7f6d5ee28700 1 --2- 172.21.15.90:0/543800320 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6d400afc10 0x7f6d400b2120 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6d54005950 tx=0x7f6d5400a300 comp rx=0 tx=0).stop 2024-04-25T11:33:28.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.717+0000 7f6d5ee28700 1 -- 172.21.15.90:0/543800320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d58107c90 msgr2=0x7f6d5819bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:28.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.717+0000 7f6d5ee28700 1 --2- 172.21.15.90:0/543800320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d58107c90 0x7f6d5819bfb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6d4800d980 tx=0x7f6d4800dc90 comp rx=0 tx=0).stop 2024-04-25T11:33:28.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.717+0000 7f6d5ee28700 1 -- 172.21.15.90:0/543800320 shutdown_connections 2024-04-25T11:33:28.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.717+0000 7f6d5ee28700 1 --2- 172.21.15.90:0/543800320 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6d400afc10 0x7f6d400b2120 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:28.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.718+0000 7f6d5ee28700 1 --2- 172.21.15.90:0/543800320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d581085e0 0x7f6d5819c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:28.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.718+0000 7f6d5ee28700 1 --2- 172.21.15.90:0/543800320 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6d58103360 0x7f6d5819ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:28.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.718+0000 7f6d5ee28700 1 --2- 172.21.15.90:0/543800320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d58107c90 0x7f6d5819bfb0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:28.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.718+0000 7f6d5ee28700 1 -- 172.21.15.90:0/543800320 >> 172.21.15.90:0/543800320 conn(0x7f6d580fecb0 msgr2=0x7f6d58107400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:28.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.718+0000 7f6d5ee28700 1 -- 172.21.15.90:0/543800320 shutdown_connections 2024-04-25T11:33:28.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:28.718+0000 7f6d5ee28700 1 -- 172.21.15.90:0/543800320 wait complete. 2024-04-25T11:33:28.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:28 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/543800320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:33:29.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:28 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/543800320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:33:29.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:28 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/543800320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:33:29.343 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-25T11:33:29.343 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-25T11:33:29.344 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-25T11:33:29.344 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-25T11:33:29.345 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-25T11:33:29.345 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-25T11:33:29.345 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-25T11:33:29.346 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-25T11:33:29.347 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-25T11:33:29.347 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-25T11:33:29.347 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-25T11:33:29.348 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-25T11:33:29.971 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:29.978 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:30.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:29 smithi138 ceph-mon[32328]: pgmap v283: 97 pgs: 97 active+clean; 638 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.7 KiB/s wr, 21 op/s 2024-04-25T11:33:30.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:29 smithi156 ceph-mon[33045]: pgmap v283: 97 pgs: 97 active+clean; 638 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.7 KiB/s wr, 21 op/s 2024-04-25T11:33:30.129 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:29 smithi090 ceph-mon[29575]: pgmap v283: 97 pgs: 97 active+clean; 638 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.7 KiB/s wr, 21 op/s 2024-04-25T11:33:30.880 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:30.890 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:30.891 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:30.894 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:30.894 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:30.894 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:30.895 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:30.895 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:30.907 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:31.958 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:31 smithi090 ceph-mon[29575]: pgmap v284: 97 pgs: 97 active+clean; 638 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.3 KiB/s wr, 14 op/s 2024-04-25T11:33:32.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:31 smithi138 ceph-mon[32328]: pgmap v284: 97 pgs: 97 active+clean; 638 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.3 KiB/s wr, 14 op/s 2024-04-25T11:33:32.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:31 smithi156 ceph-mon[33045]: pgmap v284: 97 pgs: 97 active+clean; 638 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.3 KiB/s wr, 14 op/s 2024-04-25T11:33:33.888 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:33 smithi090 ceph-mon[29575]: pgmap v285: 97 pgs: 97 active+clean; 638 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-04-25T11:33:34.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:33 smithi138 ceph-mon[32328]: pgmap v285: 97 pgs: 97 active+clean; 638 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-04-25T11:33:34.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:33 smithi156 ceph-mon[33045]: pgmap v285: 97 pgs: 97 active+clean; 638 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-04-25T11:33:36.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:35 smithi138 ceph-mon[32328]: pgmap v286: 97 pgs: 97 active+clean; 638 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-04-25T11:33:36.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:35 smithi156 ceph-mon[33045]: pgmap v286: 97 pgs: 97 active+clean; 638 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-04-25T11:33:36.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:35 smithi090 ceph-mon[29575]: pgmap v286: 97 pgs: 97 active+clean; 638 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-04-25T11:33:38.051 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:37 smithi090 ceph-mon[29575]: pgmap v287: 97 pgs: 97 active+clean; 638 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-25T11:33:38.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:37 smithi138 ceph-mon[32328]: pgmap v287: 97 pgs: 97 active+clean; 638 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-25T11:33:38.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:37 smithi156 ceph-mon[33045]: pgmap v287: 97 pgs: 97 active+clean; 638 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-25T11:33:39.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.905+0000 7f6cf1053700 1 -- 172.21.15.90:0/4232021363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cec108da0 msgr2=0x7f6cec1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:39.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.905+0000 7f6cf1053700 1 --2- 172.21.15.90:0/4232021363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cec108da0 0x7f6cec1091b0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00099c0 tx=0x7f6ce0009cd0 comp rx=0 tx=0).stop 2024-04-25T11:33:39.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.905+0000 7f6cf1053700 1 -- 172.21.15.90:0/4232021363 shutdown_connections 2024-04-25T11:33:39.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.905+0000 7f6cf1053700 1 --2- 172.21.15.90:0/4232021363 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6cec1096f0 0x7f6cec111050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:39.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.905+0000 7f6cf1053700 1 --2- 172.21.15.90:0/4232021363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cec102980 0x7f6cec102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:39.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.905+0000 7f6cf1053700 1 --2- 172.21.15.90:0/4232021363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cec108da0 0x7f6cec1091b0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:39.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.905+0000 7f6cf1053700 1 -- 172.21.15.90:0/4232021363 >> 172.21.15.90:0/4232021363 conn(0x7f6cec0fe3f0 msgr2=0x7f6cec100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:39.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.905+0000 7f6cf1053700 1 -- 172.21.15.90:0/4232021363 shutdown_connections 2024-04-25T11:33:39.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.905+0000 7f6cf1053700 1 -- 172.21.15.90:0/4232021363 wait complete. 2024-04-25T11:33:39.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.906+0000 7f6cf1053700 1 Processor -- start 2024-04-25T11:33:39.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.906+0000 7f6cf1053700 1 -- start start 2024-04-25T11:33:39.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.906+0000 7f6cf1053700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cec102980 0x7f6cec19ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:39.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.906+0000 7f6cf1053700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cec108da0 0x7f6cec19b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:39.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.907+0000 7f6cf1053700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6cec1096f0 0x7f6cec19f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:39.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.907+0000 7f6cf1053700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6cec114300 con 0x7f6cec102980 2024-04-25T11:33:39.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.907+0000 7f6cf1053700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6cec114180 con 0x7f6cec1096f0 2024-04-25T11:33:39.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.907+0000 7f6cf1053700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6cec114480 con 0x7f6cec108da0 2024-04-25T11:33:39.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.907+0000 7f6cead9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cec102980 0x7f6cec19ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:39.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.907+0000 7f6cead9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cec102980 0x7f6cec19ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46918/0 (socket says 172.21.15.90:46918) 2024-04-25T11:33:39.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.907+0000 7f6cead9d700 1 -- 172.21.15.90:0/3228650155 learned_addr learned my addr 172.21.15.90:0/3228650155 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:39.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.907+0000 7f6cea59c700 1 --2- 172.21.15.90:0/3228650155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cec108da0 0x7f6cec19b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:39.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.907+0000 7f6ceb59e700 1 --2- 172.21.15.90:0/3228650155 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6cec1096f0 0x7f6cec19f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:39.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.907+0000 7f6cead9d700 1 -- 172.21.15.90:0/3228650155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cec108da0 msgr2=0x7f6cec19b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:39.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.907+0000 7f6cead9d700 1 --2- 172.21.15.90:0/3228650155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cec108da0 0x7f6cec19b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:39.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.907+0000 7f6cead9d700 1 -- 172.21.15.90:0/3228650155 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6cec1096f0 msgr2=0x7f6cec19f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:39.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.907+0000 7f6cead9d700 1 --2- 172.21.15.90:0/3228650155 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6cec1096f0 0x7f6cec19f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:39.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.907+0000 7f6cead9d700 1 -- 172.21.15.90:0/3228650155 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cec19fc40 con 0x7f6cec102980 2024-04-25T11:33:39.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.907+0000 7f6cead9d700 1 --2- 172.21.15.90:0/3228650155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cec102980 0x7f6cec19ad70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc00e9a0 tx=0x7f6cdc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:39.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.907+0000 7f6cd3fff700 1 -- 172.21.15.90:0/3228650155 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cdc00cbd0 con 0x7f6cec102980 2024-04-25T11:33:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.907+0000 7f6cf1053700 1 -- 172.21.15.90:0/3228650155 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cec19ff30 con 0x7f6cec102980 2024-04-25T11:33:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.907+0000 7f6cd3fff700 1 -- 172.21.15.90:0/3228650155 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6cdc00cd70 con 0x7f6cec102980 2024-04-25T11:33:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.908+0000 7f6cd3fff700 1 -- 172.21.15.90:0/3228650155 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cdc018680 con 0x7f6cec102980 2024-04-25T11:33:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.908+0000 7f6cf1053700 1 -- 172.21.15.90:0/3228650155 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cec1a7850 con 0x7f6cec102980 2024-04-25T11:33:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.908+0000 7f6cea59c700 1 --2- 172.21.15.90:0/3228650155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cec108da0 0x7f6cec19b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:33:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.908+0000 7f6cf1053700 1 -- 172.21.15.90:0/3228650155 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6cec063d60 con 0x7f6cec102980 2024-04-25T11:33:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.910+0000 7f6cd3fff700 1 -- 172.21.15.90:0/3228650155 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f6cdc018820 con 0x7f6cec102980 2024-04-25T11:33:39.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.911+0000 7f6cd3fff700 1 --2- 172.21.15.90:0/3228650155 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6cd40afc10 0x7f6cd40b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:39.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.911+0000 7f6cea59c700 1 --2- 172.21.15.90:0/3228650155 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6cd40afc10 0x7f6cd40b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:39.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.912+0000 7f6cd3fff700 1 -- 172.21.15.90:0/3228650155 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7f6cdc0df9b0 con 0x7f6cec102980 2024-04-25T11:33:39.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.912+0000 7f6cd3fff700 1 --2- 172.21.15.90:0/3228650155 >> [v2:172.21.15.156:6800/1505807670,v1:172.21.15.156:6801/1505807670] conn(0x7f6cd40b9440 0x7f6cd40bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:39.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.912+0000 7f6cd3fff700 1 -- 172.21.15.90:0/3228650155 --> [v2:172.21.15.156:6800/1505807670,v1:172.21.15.156:6801/1505807670] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6cdc019ec0 con 0x7f6cd40b9440 2024-04-25T11:33:39.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.912+0000 7f6cd3fff700 1 -- 172.21.15.90:0/3228650155 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=77) ==== 24+0+0 (secure 0 0 0) 0x7f6cdc0dfe90 con 0x7f6cec102980 2024-04-25T11:33:39.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.912+0000 7f6cea59c700 1 --2- 172.21.15.90:0/3228650155 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6cd40afc10 0x7f6cd40b2120 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0000c00 tx=0x7f6ce001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:39.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.912+0000 7f6ceb59e700 1 --2- 172.21.15.90:0/3228650155 >> [v2:172.21.15.156:6800/1505807670,v1:172.21.15.156:6801/1505807670] conn(0x7f6cd40b9440 0x7f6cd40bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:39.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.912+0000 7f6ceb59e700 1 --2- 172.21.15.90:0/3228650155 >> [v2:172.21.15.156:6800/1505807670,v1:172.21.15.156:6801/1505807670] conn(0x7f6cd40b9440 0x7f6cd40bb890 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:39.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.913+0000 7f6cd3fff700 1 -- 172.21.15.90:0/3228650155 <== osd.2 v2:172.21.15.156:6800/1505807670 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6cdc019ec0 con 0x7f6cd40b9440 2024-04-25T11:33:39.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.930+0000 7f6cf1053700 1 -- 172.21.15.90:0/3228650155 --> [v2:172.21.15.156:6800/1505807670,v1:172.21.15.156:6801/1505807670] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6cec064160 con 0x7f6cd40b9440 2024-04-25T11:33:39.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.930+0000 7f6cd3fff700 1 -- 172.21.15.90:0/3228650155 <== osd.2 v2:172.21.15.156:6800/1505807670 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6cec064160 con 0x7f6cd40b9440 2024-04-25T11:33:39.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.931+0000 7f6cf1053700 1 -- 172.21.15.90:0/3228650155 >> [v2:172.21.15.156:6800/1505807670,v1:172.21.15.156:6801/1505807670] conn(0x7f6cd40b9440 msgr2=0x7f6cd40bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:39.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.931+0000 7f6cf1053700 1 --2- 172.21.15.90:0/3228650155 >> [v2:172.21.15.156:6800/1505807670,v1:172.21.15.156:6801/1505807670] conn(0x7f6cd40b9440 0x7f6cd40bb890 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:39.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.931+0000 7f6cf1053700 1 -- 172.21.15.90:0/3228650155 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6cd40afc10 msgr2=0x7f6cd40b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:39.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.931+0000 7f6cf1053700 1 --2- 172.21.15.90:0/3228650155 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6cd40afc10 0x7f6cd40b2120 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0000c00 tx=0x7f6ce001a040 comp rx=0 tx=0).stop 2024-04-25T11:33:39.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.931+0000 7f6cf1053700 1 -- 172.21.15.90:0/3228650155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cec102980 msgr2=0x7f6cec19ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:39.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.931+0000 7f6cf1053700 1 --2- 172.21.15.90:0/3228650155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cec102980 0x7f6cec19ad70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc00e9a0 tx=0x7f6cdc00ecb0 comp rx=0 tx=0).stop 2024-04-25T11:33:39.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.931+0000 7f6ceb59e700 1 -- 172.21.15.90:0/3228650155 reap_dead start 2024-04-25T11:33:39.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.931+0000 7f6cf1053700 1 -- 172.21.15.90:0/3228650155 shutdown_connections 2024-04-25T11:33:39.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.931+0000 7f6cf1053700 1 -- 172.21.15.90:0/3228650155 >> 172.21.15.90:0/3228650155 conn(0x7f6cec0fe3f0 msgr2=0x7f6cec100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:39.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.931+0000 7f6cf1053700 1 -- 172.21.15.90:0/3228650155 shutdown_connections 2024-04-25T11:33:39.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:39.931+0000 7f6cf1053700 1 -- 172.21.15.90:0/3228650155 wait complete. 2024-04-25T11:33:40.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:39 smithi138 ceph-mon[32328]: pgmap v288: 97 pgs: 97 active+clean; 641 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-25T11:33:40.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:39 smithi156 ceph-mon[33045]: pgmap v288: 97 pgs: 97 active+clean; 641 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-25T11:33:40.161 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:39 smithi090 ceph-mon[29575]: pgmap v288: 97 pgs: 97 active+clean; 641 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-25T11:33:40.447 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:42.035 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:41 smithi090 ceph-mon[29575]: pgmap v289: 97 pgs: 97 active+clean; 641 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-25T11:33:42.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:41 smithi138 ceph-mon[32328]: pgmap v289: 97 pgs: 97 active+clean; 641 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-25T11:33:42.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:41 smithi156 ceph-mon[33045]: pgmap v289: 97 pgs: 97 active+clean; 641 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-25T11:33:42.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.795+0000 7f72d7479700 1 -- 172.21.15.90:0/2686426996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72d0103360 msgr2=0x7f72d0103770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:42.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.795+0000 7f72d7479700 1 --2- 172.21.15.90:0/2686426996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72d0103360 0x7f72d0103770 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f72cc0099c0 tx=0x7f72cc009cd0 comp rx=0 tx=0).stop 2024-04-25T11:33:42.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.795+0000 7f72d7479700 1 -- 172.21.15.90:0/2686426996 shutdown_connections 2024-04-25T11:33:42.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.795+0000 7f72d7479700 1 --2- 172.21.15.90:0/2686426996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72d01085e0 0x7f72d010ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:42.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.795+0000 7f72d7479700 1 --2- 172.21.15.90:0/2686426996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72d0103360 0x7f72d0103770 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:42.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.795+0000 7f72d7479700 1 --2- 172.21.15.90:0/2686426996 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f72d0107c90 0x7f72d01080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:42.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.795+0000 7f72d7479700 1 -- 172.21.15.90:0/2686426996 >> 172.21.15.90:0/2686426996 conn(0x7f72d00fec90 msgr2=0x7f72d01010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:42.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.795+0000 7f72d7479700 1 -- 172.21.15.90:0/2686426996 shutdown_connections 2024-04-25T11:33:42.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.795+0000 7f72d7479700 1 -- 172.21.15.90:0/2686426996 wait complete. 2024-04-25T11:33:42.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.796+0000 7f72d7479700 1 Processor -- start 2024-04-25T11:33:42.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.796+0000 7f72d7479700 1 -- start start 2024-04-25T11:33:42.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.796+0000 7f72d7479700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72d0103360 0x7f72d019bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:42.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.796+0000 7f72d7479700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72d0107c90 0x7f72d019c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:42.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.796+0000 7f72d7479700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f72d01085e0 0x7f72d019c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:42.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.796+0000 7f72d7479700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f72d0113210 con 0x7f72d0107c90 2024-04-25T11:33:42.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.796+0000 7f72d7479700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f72d0113090 con 0x7f72d01085e0 2024-04-25T11:33:42.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.796+0000 7f72d7479700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f72d0113390 con 0x7f72d0103360 2024-04-25T11:33:42.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.796+0000 7f72d4a14700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72d0107c90 0x7f72d019c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:42.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.796+0000 7f72d4a14700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72d0107c90 0x7f72d019c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50772/0 (socket says 172.21.15.90:50772) 2024-04-25T11:33:42.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.796+0000 7f72d4a14700 1 -- 172.21.15.90:0/1142322594 learned_addr learned my addr 172.21.15.90:0/1142322594 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:42.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.796+0000 7f72d5a16700 1 --2- 172.21.15.90:0/1142322594 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f72d01085e0 0x7f72d019c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:42.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.796+0000 7f72d5215700 1 --2- 172.21.15.90:0/1142322594 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72d0103360 0x7f72d019bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:42.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.796+0000 7f72d4a14700 1 -- 172.21.15.90:0/1142322594 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72d0103360 msgr2=0x7f72d019bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:42.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.796+0000 7f72d4a14700 1 --2- 172.21.15.90:0/1142322594 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72d0103360 0x7f72d019bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:42.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.796+0000 7f72d4a14700 1 -- 172.21.15.90:0/1142322594 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f72d01085e0 msgr2=0x7f72d019c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:42.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.796+0000 7f72d4a14700 1 --2- 172.21.15.90:0/1142322594 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f72d01085e0 0x7f72d019c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:42.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.796+0000 7f72d4a14700 1 -- 172.21.15.90:0/1142322594 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d01a6e50 con 0x7f72d0107c90 2024-04-25T11:33:42.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.797+0000 7f72d4a14700 1 --2- 172.21.15.90:0/1142322594 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72d0107c90 0x7f72d019c070 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f72c000d980 tx=0x7f72c000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:42.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.797+0000 7f72c67fc700 1 -- 172.21.15.90:0/1142322594 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c0017070 con 0x7f72d0107c90 2024-04-25T11:33:42.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.797+0000 7f72c67fc700 1 -- 172.21.15.90:0/1142322594 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72c000fd40 con 0x7f72d0107c90 2024-04-25T11:33:42.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.797+0000 7f72d7479700 1 -- 172.21.15.90:0/1142322594 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72d01a7140 con 0x7f72d0107c90 2024-04-25T11:33:42.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.797+0000 7f72c67fc700 1 -- 172.21.15.90:0/1142322594 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c000e3b0 con 0x7f72d0107c90 2024-04-25T11:33:42.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.797+0000 7f72d7479700 1 -- 172.21.15.90:0/1142322594 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72d01a7680 con 0x7f72d0107c90 2024-04-25T11:33:42.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.798+0000 7f72d7479700 1 -- 172.21.15.90:0/1142322594 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f72d0063da0 con 0x7f72d0107c90 2024-04-25T11:33:42.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.799+0000 7f72c67fc700 1 -- 172.21.15.90:0/1142322594 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f72c0010610 con 0x7f72d0107c90 2024-04-25T11:33:42.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.800+0000 7f72c67fc700 1 --2- 172.21.15.90:0/1142322594 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f72bc0afc10 0x7f72bc0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:42.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.800+0000 7f72d5215700 1 --2- 172.21.15.90:0/1142322594 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f72bc0afc10 0x7f72bc0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:42.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.800+0000 7f72c67fc700 1 -- 172.21.15.90:0/1142322594 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7f72c00dea40 con 0x7f72d0107c90 2024-04-25T11:33:42.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.800+0000 7f72c67fc700 1 --2- 172.21.15.90:0/1142322594 >> [v2:172.21.15.138:6816/4012476387,v1:172.21.15.138:6817/4012476387] conn(0x7f72bc0b9440 0x7f72bc0bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:42.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.800+0000 7f72c67fc700 1 -- 172.21.15.90:0/1142322594 --> [v2:172.21.15.138:6816/4012476387,v1:172.21.15.138:6817/4012476387] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f72c0016ec0 con 0x7f72bc0b9440 2024-04-25T11:33:42.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.800+0000 7f72c67fc700 1 -- 172.21.15.90:0/1142322594 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=77) ==== 24+0+0 (secure 0 0 0) 0x7f72c00def20 con 0x7f72d0107c90 2024-04-25T11:33:42.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.800+0000 7f72d5215700 1 --2- 172.21.15.90:0/1142322594 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f72bc0afc10 0x7f72bc0b2120 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f72cc009990 tx=0x7f72cc0036e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:42.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.801+0000 7f72d5a16700 1 --2- 172.21.15.90:0/1142322594 >> [v2:172.21.15.138:6816/4012476387,v1:172.21.15.138:6817/4012476387] conn(0x7f72bc0b9440 0x7f72bc0bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:42.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.801+0000 7f72d5a16700 1 --2- 172.21.15.90:0/1142322594 >> [v2:172.21.15.138:6816/4012476387,v1:172.21.15.138:6817/4012476387] conn(0x7f72bc0b9440 0x7f72bc0bb890 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:42.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.802+0000 7f72c67fc700 1 -- 172.21.15.90:0/1142322594 <== osd.7 v2:172.21.15.138:6816/4012476387 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f72c0016ec0 con 0x7f72bc0b9440 2024-04-25T11:33:42.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.818+0000 7f72d7479700 1 -- 172.21.15.90:0/1142322594 --> [v2:172.21.15.138:6816/4012476387,v1:172.21.15.138:6817/4012476387] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f72d00641d0 con 0x7f72bc0b9440 2024-04-25T11:33:42.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.819+0000 7f72c67fc700 1 -- 172.21.15.90:0/1142322594 <== osd.7 v2:172.21.15.138:6816/4012476387 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f72d00641d0 con 0x7f72bc0b9440 2024-04-25T11:33:42.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.819+0000 7f72d7479700 1 -- 172.21.15.90:0/1142322594 >> [v2:172.21.15.138:6816/4012476387,v1:172.21.15.138:6817/4012476387] conn(0x7f72bc0b9440 msgr2=0x7f72bc0bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:42.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.819+0000 7f72d7479700 1 --2- 172.21.15.90:0/1142322594 >> [v2:172.21.15.138:6816/4012476387,v1:172.21.15.138:6817/4012476387] conn(0x7f72bc0b9440 0x7f72bc0bb890 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:42.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.819+0000 7f72d7479700 1 -- 172.21.15.90:0/1142322594 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f72bc0afc10 msgr2=0x7f72bc0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:42.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.819+0000 7f72d7479700 1 --2- 172.21.15.90:0/1142322594 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f72bc0afc10 0x7f72bc0b2120 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f72cc009990 tx=0x7f72cc0036e0 comp rx=0 tx=0).stop 2024-04-25T11:33:42.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.819+0000 7f72d7479700 1 -- 172.21.15.90:0/1142322594 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72d0107c90 msgr2=0x7f72d019c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:42.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.819+0000 7f72d7479700 1 --2- 172.21.15.90:0/1142322594 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72d0107c90 0x7f72d019c070 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f72c000d980 tx=0x7f72c000dc90 comp rx=0 tx=0).stop 2024-04-25T11:33:42.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.819+0000 7f72d5a16700 1 -- 172.21.15.90:0/1142322594 reap_dead start 2024-04-25T11:33:42.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.819+0000 7f72d7479700 1 -- 172.21.15.90:0/1142322594 shutdown_connections 2024-04-25T11:33:42.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.819+0000 7f72d7479700 1 -- 172.21.15.90:0/1142322594 >> 172.21.15.90:0/1142322594 conn(0x7f72d00fec90 msgr2=0x7f72d01010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:42.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.820+0000 7f72d7479700 1 -- 172.21.15.90:0/1142322594 shutdown_connections 2024-04-25T11:33:42.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:42.820+0000 7f72d7479700 1 -- 172.21.15.90:0/1142322594 wait complete. 2024-04-25T11:33:44.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:43 smithi138 ceph-mon[32328]: pgmap v290: 97 pgs: 97 active+clean; 641 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-25T11:33:44.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:43 smithi156 ceph-mon[33045]: pgmap v290: 97 pgs: 97 active+clean; 641 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-25T11:33:44.104 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:43 smithi090 ceph-mon[29575]: pgmap v290: 97 pgs: 97 active+clean; 641 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-25T11:33:44.131 INFO:teuthology.orchestra.run.smithi090.stdout:206158430247 2024-04-25T11:33:44.131 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-25T11:33:44.575 INFO:teuthology.orchestra.run.smithi090.stdout:77309411391 2024-04-25T11:33:44.575 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-25T11:33:45.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.484+0000 7f2dbb973700 1 -- 172.21.15.90:0/394098478 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2db4109720 msgr2=0x7f2db4111080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:45.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.484+0000 7f2dbb973700 1 --2- 172.21.15.90:0/394098478 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2db4109720 0x7f2db4111080 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2dac00b0d0 tx=0x7f2dac00b3e0 comp rx=0 tx=0).stop 2024-04-25T11:33:45.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.484+0000 7f2dbb973700 1 -- 172.21.15.90:0/394098478 shutdown_connections 2024-04-25T11:33:45.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.484+0000 7f2dbb973700 1 --2- 172.21.15.90:0/394098478 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2db4109720 0x7f2db4111080 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:45.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.484+0000 7f2dbb973700 1 --2- 172.21.15.90:0/394098478 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2db41044a0 0x7f2db41048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:45.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.484+0000 7f2dbb973700 1 --2- 172.21.15.90:0/394098478 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2db4108dd0 0x7f2db41091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:45.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.484+0000 7f2dbb973700 1 -- 172.21.15.90:0/394098478 >> 172.21.15.90:0/394098478 conn(0x7f2db40ffd70 msgr2=0x7f2db41021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:45.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.484+0000 7f2dbb973700 1 -- 172.21.15.90:0/394098478 shutdown_connections 2024-04-25T11:33:45.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.484+0000 7f2dbb973700 1 -- 172.21.15.90:0/394098478 wait complete. 2024-04-25T11:33:45.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.485+0000 7f2dbb973700 1 Processor -- start 2024-04-25T11:33:45.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.485+0000 7f2dbb973700 1 -- start start 2024-04-25T11:33:45.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.485+0000 7f2dbb973700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2db41044a0 0x7f2db419ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:45.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.485+0000 7f2dbb973700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2db4108dd0 0x7f2db419d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:45.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.485+0000 7f2dbb973700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2db4109720 0x7f2db41a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:45.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.485+0000 7f2dbb973700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2db4114350 con 0x7f2db41044a0 2024-04-25T11:33:45.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.485+0000 7f2dbb973700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2db41141d0 con 0x7f2db4109720 2024-04-25T11:33:45.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.485+0000 7f2dbb973700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2db41144d0 con 0x7f2db4108dd0 2024-04-25T11:33:45.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.486+0000 7f2db970f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2db41044a0 0x7f2db419ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:45.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.486+0000 7f2db970f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2db41044a0 0x7f2db419ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50846/0 (socket says 172.21.15.90:50846) 2024-04-25T11:33:45.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.486+0000 7f2db970f700 1 -- 172.21.15.90:0/3533285586 learned_addr learned my addr 172.21.15.90:0/3533285586 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:45.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.486+0000 7f2db9f10700 1 --2- 172.21.15.90:0/3533285586 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2db4109720 0x7f2db41a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:45.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.486+0000 7f2db8f0e700 1 --2- 172.21.15.90:0/3533285586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2db4108dd0 0x7f2db419d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:45.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.486+0000 7f2db970f700 1 -- 172.21.15.90:0/3533285586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2db4108dd0 msgr2=0x7f2db419d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:45.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.486+0000 7f2db970f700 1 --2- 172.21.15.90:0/3533285586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2db4108dd0 0x7f2db419d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:45.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.486+0000 7f2db970f700 1 -- 172.21.15.90:0/3533285586 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2db4109720 msgr2=0x7f2db41a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:45.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.486+0000 7f2db970f700 1 --2- 172.21.15.90:0/3533285586 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2db4109720 0x7f2db41a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:45.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.486+0000 7f2db970f700 1 -- 172.21.15.90:0/3533285586 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db41a1ca0 con 0x7f2db41044a0 2024-04-25T11:33:45.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.486+0000 7f2db9f10700 1 --2- 172.21.15.90:0/3533285586 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2db4109720 0x7f2db41a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:33:45.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.486+0000 7f2db8f0e700 1 --2- 172.21.15.90:0/3533285586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2db4108dd0 0x7f2db419d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:33:45.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.486+0000 7f2db970f700 1 --2- 172.21.15.90:0/3533285586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2db41044a0 0x7f2db419ce80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2da400c9b0 tx=0x7f2da400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:45.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.486+0000 7f2daa7fc700 1 -- 172.21.15.90:0/3533285586 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da4007b00 con 0x7f2db41044a0 2024-04-25T11:33:45.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.486+0000 7f2dbb973700 1 -- 172.21.15.90:0/3533285586 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2db41a1f90 con 0x7f2db41044a0 2024-04-25T11:33:45.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.486+0000 7f2daa7fc700 1 -- 172.21.15.90:0/3533285586 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2da4007ca0 con 0x7f2db41044a0 2024-04-25T11:33:45.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.486+0000 7f2daa7fc700 1 -- 172.21.15.90:0/3533285586 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da4018780 con 0x7f2db41044a0 2024-04-25T11:33:45.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.486+0000 7f2dbb973700 1 -- 172.21.15.90:0/3533285586 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2db41a9960 con 0x7f2db41044a0 2024-04-25T11:33:45.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.487+0000 7f2dbb973700 1 -- 172.21.15.90:0/3533285586 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2db4077d30 con 0x7f2db41044a0 2024-04-25T11:33:45.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.489+0000 7f2daa7fc700 1 -- 172.21.15.90:0/3533285586 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f2da400fa20 con 0x7f2db41044a0 2024-04-25T11:33:45.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.489+0000 7f2daa7fc700 1 --2- 172.21.15.90:0/3533285586 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f2da00afc10 0x7f2da00b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:45.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.490+0000 7f2db8f0e700 1 --2- 172.21.15.90:0/3533285586 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f2da00afc10 0x7f2da00b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:45.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.490+0000 7f2daa7fc700 1 -- 172.21.15.90:0/3533285586 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7f2da40df270 con 0x7f2db41044a0 2024-04-25T11:33:45.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.490+0000 7f2daa7fc700 1 --2- 172.21.15.90:0/3533285586 >> [v2:172.21.15.90:6818/2364955456,v1:172.21.15.90:6819/2364955456] conn(0x7f2da00b9440 0x7f2da00bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:45.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.490+0000 7f2daa7fc700 1 -- 172.21.15.90:0/3533285586 --> [v2:172.21.15.90:6818/2364955456,v1:172.21.15.90:6819/2364955456] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2da4022e50 con 0x7f2da00b9440 2024-04-25T11:33:45.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.490+0000 7f2daa7fc700 1 -- 172.21.15.90:0/3533285586 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=77) ==== 24+0+0 (secure 0 0 0) 0x7f2da40df750 con 0x7f2db41044a0 2024-04-25T11:33:45.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.490+0000 7f2db9f10700 1 --2- 172.21.15.90:0/3533285586 >> [v2:172.21.15.90:6818/2364955456,v1:172.21.15.90:6819/2364955456] conn(0x7f2da00b9440 0x7f2da00bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:45.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.490+0000 7f2db8f0e700 1 --2- 172.21.15.90:0/3533285586 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f2da00afc10 0x7f2da00b2120 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2db0005fd0 tx=0x7f2db0005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:45.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.490+0000 7f2db9f10700 1 --2- 172.21.15.90:0/3533285586 >> [v2:172.21.15.90:6818/2364955456,v1:172.21.15.90:6819/2364955456] conn(0x7f2da00b9440 0x7f2da00bb890 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:45.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.491+0000 7f2daa7fc700 1 -- 172.21.15.90:0/3533285586 <== osd.6 v2:172.21.15.90:6818/2364955456 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2da4022e50 con 0x7f2da00b9440 2024-04-25T11:33:45.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.507+0000 7f2dbb973700 1 -- 172.21.15.90:0/3533285586 --> [v2:172.21.15.90:6818/2364955456,v1:172.21.15.90:6819/2364955456] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2db4105950 con 0x7f2da00b9440 2024-04-25T11:33:45.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.507+0000 7f2daa7fc700 1 -- 172.21.15.90:0/3533285586 <== osd.6 v2:172.21.15.90:6818/2364955456 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2db4105950 con 0x7f2da00b9440 2024-04-25T11:33:45.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.507+0000 7f2dbb973700 1 -- 172.21.15.90:0/3533285586 >> [v2:172.21.15.90:6818/2364955456,v1:172.21.15.90:6819/2364955456] conn(0x7f2da00b9440 msgr2=0x7f2da00bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:45.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.507+0000 7f2dbb973700 1 --2- 172.21.15.90:0/3533285586 >> [v2:172.21.15.90:6818/2364955456,v1:172.21.15.90:6819/2364955456] conn(0x7f2da00b9440 0x7f2da00bb890 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:45.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.508+0000 7f2dbb973700 1 -- 172.21.15.90:0/3533285586 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f2da00afc10 msgr2=0x7f2da00b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:45.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.508+0000 7f2dbb973700 1 --2- 172.21.15.90:0/3533285586 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f2da00afc10 0x7f2da00b2120 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2db0005fd0 tx=0x7f2db0005db0 comp rx=0 tx=0).stop 2024-04-25T11:33:45.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.508+0000 7f2dbb973700 1 -- 172.21.15.90:0/3533285586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2db41044a0 msgr2=0x7f2db419ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:45.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.508+0000 7f2dbb973700 1 --2- 172.21.15.90:0/3533285586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2db41044a0 0x7f2db419ce80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2da400c9b0 tx=0x7f2da400cd70 comp rx=0 tx=0).stop 2024-04-25T11:33:45.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.508+0000 7f2db9f10700 1 -- 172.21.15.90:0/3533285586 reap_dead start 2024-04-25T11:33:45.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.508+0000 7f2dbb973700 1 -- 172.21.15.90:0/3533285586 shutdown_connections 2024-04-25T11:33:45.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.508+0000 7f2dbb973700 1 -- 172.21.15.90:0/3533285586 >> 172.21.15.90:0/3533285586 conn(0x7f2db40ffd70 msgr2=0x7f2db4110d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:45.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.508+0000 7f2dbb973700 1 -- 172.21.15.90:0/3533285586 shutdown_connections 2024-04-25T11:33:45.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.508+0000 7f2dbb973700 1 -- 172.21.15.90:0/3533285586 wait complete. 2024-04-25T11:33:45.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.222+0000 7f17a87d6700 1 -- 172.21.15.90:0/2440210261 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17a0109100 msgr2=0x7f17a0109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:45.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.222+0000 7f17a87d6700 1 --2- 172.21.15.90:0/2440210261 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17a0109100 0x7f17a0109510 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f179c009990 tx=0x7f179c009ca0 comp rx=0 tx=0).stop 2024-04-25T11:33:45.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.222+0000 7f17a87d6700 1 -- 172.21.15.90:0/2440210261 shutdown_connections 2024-04-25T11:33:45.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.222+0000 7f17a87d6700 1 --2- 172.21.15.90:0/2440210261 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17a0102f90 0x7f17a0107dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:45.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.222+0000 7f17a87d6700 1 --2- 172.21.15.90:0/2440210261 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17a0109100 0x7f17a0109510 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:45.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.222+0000 7f17a87d6700 1 --2- 172.21.15.90:0/2440210261 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17a0102640 0x7f17a0102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:45.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.222+0000 7f17a87d6700 1 -- 172.21.15.90:0/2440210261 >> 172.21.15.90:0/2440210261 conn(0x7f17a00fe370 msgr2=0x7f17a01007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:45.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.222+0000 7f17a87d6700 1 -- 172.21.15.90:0/2440210261 shutdown_connections 2024-04-25T11:33:45.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.222+0000 7f17a87d6700 1 -- 172.21.15.90:0/2440210261 wait complete. 2024-04-25T11:33:45.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.223+0000 7f17a87d6700 1 Processor -- start 2024-04-25T11:33:45.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.223+0000 7f17a87d6700 1 -- start start 2024-04-25T11:33:45.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.223+0000 7f17a87d6700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17a0102640 0x7f17a01a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:45.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.223+0000 7f17a87d6700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17a0102f90 0x7f17a01a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:45.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.223+0000 7f17a87d6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17a0109100 0x7f17a01a5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:45.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.223+0000 7f17a87d6700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f17a011a910 con 0x7f17a0102640 2024-04-25T11:33:45.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.223+0000 7f17a87d6700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f17a011a790 con 0x7f17a0102f90 2024-04-25T11:33:45.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.223+0000 7f17a87d6700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f17a011aa90 con 0x7f17a0109100 2024-04-25T11:33:45.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.223+0000 7f17a6572700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17a0102640 0x7f17a01a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:45.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.223+0000 7f17a6572700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17a0102640 0x7f17a01a1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50820/0 (socket says 172.21.15.90:50820) 2024-04-25T11:33:45.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.223+0000 7f17a6572700 1 -- 172.21.15.90:0/4097520391 learned_addr learned my addr 172.21.15.90:0/4097520391 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:45.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.224+0000 7f17a6d73700 1 --2- 172.21.15.90:0/4097520391 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17a0109100 0x7f17a01a5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:45.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.224+0000 7f17a5d71700 1 --2- 172.21.15.90:0/4097520391 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17a0102f90 0x7f17a01a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:45.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.224+0000 7f17a6572700 1 -- 172.21.15.90:0/4097520391 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17a0109100 msgr2=0x7f17a01a5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:45.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.224+0000 7f17a6572700 1 --2- 172.21.15.90:0/4097520391 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17a0109100 0x7f17a01a5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:45.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.224+0000 7f17a6572700 1 -- 172.21.15.90:0/4097520391 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17a0102f90 msgr2=0x7f17a01a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:45.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.224+0000 7f17a6572700 1 --2- 172.21.15.90:0/4097520391 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17a0102f90 0x7f17a01a1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:45.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.224+0000 7f17a6572700 1 -- 172.21.15.90:0/4097520391 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a01a6130 con 0x7f17a0102640 2024-04-25T11:33:45.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.224+0000 7f17a5d71700 1 --2- 172.21.15.90:0/4097520391 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17a0102f90 0x7f17a01a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:33:45.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.224+0000 7f17a6572700 1 --2- 172.21.15.90:0/4097520391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17a0102640 0x7f17a01a1260 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f179c000c00 tx=0x7f179c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:45.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.224+0000 7f17977fe700 1 -- 172.21.15.90:0/4097520391 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179c027070 con 0x7f17a0102640 2024-04-25T11:33:45.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.224+0000 7f17a87d6700 1 -- 172.21.15.90:0/4097520391 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17a01a63c0 con 0x7f17a0102640 2024-04-25T11:33:45.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.224+0000 7f17977fe700 1 -- 172.21.15.90:0/4097520391 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f179c003fa0 con 0x7f17a0102640 2024-04-25T11:33:45.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.224+0000 7f17977fe700 1 -- 172.21.15.90:0/4097520391 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179c022dd0 con 0x7f17a0102640 2024-04-25T11:33:45.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.224+0000 7f17a87d6700 1 -- 172.21.15.90:0/4097520391 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17a01add40 con 0x7f17a0102640 2024-04-25T11:33:45.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.225+0000 7f17a87d6700 1 -- 172.21.15.90:0/4097520391 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f17a0107810 con 0x7f17a0102640 2024-04-25T11:33:45.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.226+0000 7f17977fe700 1 -- 172.21.15.90:0/4097520391 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f179c02a070 con 0x7f17a0102640 2024-04-25T11:33:45.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.227+0000 7f17977fe700 1 --2- 172.21.15.90:0/4097520391 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f178c0afc10 0x7f178c0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:45.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.227+0000 7f17a5d71700 1 --2- 172.21.15.90:0/4097520391 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f178c0afc10 0x7f178c0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:45.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.227+0000 7f17977fe700 1 -- 172.21.15.90:0/4097520391 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7f179c0e0160 con 0x7f17a0102640 2024-04-25T11:33:45.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.227+0000 7f17977fe700 1 --2- 172.21.15.90:0/4097520391 >> [v2:172.21.15.138:6824/4264666232,v1:172.21.15.138:6825/4264666232] conn(0x7f178c0b9440 0x7f178c0bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:45.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.227+0000 7f17977fe700 1 -- 172.21.15.90:0/4097520391 --> [v2:172.21.15.138:6824/4264666232,v1:172.21.15.138:6825/4264666232] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f179c0043d0 con 0x7f178c0b9440 2024-04-25T11:33:45.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.227+0000 7f17977fe700 1 -- 172.21.15.90:0/4097520391 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=77) ==== 24+0+0 (secure 0 0 0) 0x7f179c0e0640 con 0x7f17a0102640 2024-04-25T11:33:45.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.228+0000 7f17a5d71700 1 --2- 172.21.15.90:0/4097520391 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f178c0afc10 0x7f178c0b2120 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f17a00ffb70 tx=0x7f1790005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:45.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.228+0000 7f17a6d73700 1 --2- 172.21.15.90:0/4097520391 >> [v2:172.21.15.138:6824/4264666232,v1:172.21.15.138:6825/4264666232] conn(0x7f178c0b9440 0x7f178c0bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:45.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.228+0000 7f17a6d73700 1 --2- 172.21.15.90:0/4097520391 >> [v2:172.21.15.138:6824/4264666232,v1:172.21.15.138:6825/4264666232] conn(0x7f178c0b9440 0x7f178c0bb890 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:45.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.229+0000 7f17977fe700 1 -- 172.21.15.90:0/4097520391 <== osd.10 v2:172.21.15.138:6824/4264666232 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f179c0043d0 con 0x7f178c0b9440 2024-04-25T11:33:45.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.245+0000 7f17a87d6700 1 -- 172.21.15.90:0/4097520391 --> [v2:172.21.15.138:6824/4264666232,v1:172.21.15.138:6825/4264666232] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f17a0103860 con 0x7f178c0b9440 2024-04-25T11:33:45.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.246+0000 7f17977fe700 1 -- 172.21.15.90:0/4097520391 <== osd.10 v2:172.21.15.138:6824/4264666232 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f17a0103860 con 0x7f178c0b9440 2024-04-25T11:33:45.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.246+0000 7f17a87d6700 1 -- 172.21.15.90:0/4097520391 >> [v2:172.21.15.138:6824/4264666232,v1:172.21.15.138:6825/4264666232] conn(0x7f178c0b9440 msgr2=0x7f178c0bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:45.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.246+0000 7f17a87d6700 1 --2- 172.21.15.90:0/4097520391 >> [v2:172.21.15.138:6824/4264666232,v1:172.21.15.138:6825/4264666232] conn(0x7f178c0b9440 0x7f178c0bb890 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:45.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.246+0000 7f17a87d6700 1 -- 172.21.15.90:0/4097520391 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f178c0afc10 msgr2=0x7f178c0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:45.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.246+0000 7f17a87d6700 1 --2- 172.21.15.90:0/4097520391 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f178c0afc10 0x7f178c0b2120 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f17a00ffb70 tx=0x7f1790005f50 comp rx=0 tx=0).stop 2024-04-25T11:33:45.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.246+0000 7f17a87d6700 1 -- 172.21.15.90:0/4097520391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17a0102640 msgr2=0x7f17a01a1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:45.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.246+0000 7f17a87d6700 1 --2- 172.21.15.90:0/4097520391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17a0102640 0x7f17a01a1260 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f179c000c00 tx=0x7f179c0047b0 comp rx=0 tx=0).stop 2024-04-25T11:33:45.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.246+0000 7f17a6d73700 1 -- 172.21.15.90:0/4097520391 reap_dead start 2024-04-25T11:33:45.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.246+0000 7f17a87d6700 1 -- 172.21.15.90:0/4097520391 shutdown_connections 2024-04-25T11:33:45.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.246+0000 7f17a87d6700 1 -- 172.21.15.90:0/4097520391 >> 172.21.15.90:0/4097520391 conn(0x7f17a00fe370 msgr2=0x7f17a0105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:45.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.246+0000 7f17a87d6700 1 -- 172.21.15.90:0/4097520391 shutdown_connections 2024-04-25T11:33:45.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:45.246+0000 7f17a87d6700 1 -- 172.21.15.90:0/4097520391 wait complete. 2024-04-25T11:33:46.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:45 smithi138 ceph-mon[32328]: pgmap v291: 97 pgs: 97 active+clean; 641 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-25T11:33:46.085 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:45 smithi090 ceph-mon[29575]: pgmap v291: 97 pgs: 97 active+clean; 641 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-25T11:33:46.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:45 smithi156 ceph-mon[33045]: pgmap v291: 97 pgs: 97 active+clean; 641 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-25T11:33:46.395 INFO:teuthology.orchestra.run.smithi090.stdout:283467841561 2024-04-25T11:33:46.396 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-25T11:33:46.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.765+0000 7f6fc1ff7700 1 -- 172.21.15.90:0/3942933564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fbc103360 msgr2=0x7f6fbc103770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:46.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.765+0000 7f6fc1ff7700 1 --2- 172.21.15.90:0/3942933564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fbc103360 0x7f6fbc103770 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6fac0099c0 tx=0x7f6fac009cd0 comp rx=0 tx=0).stop 2024-04-25T11:33:46.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.765+0000 7f6fc1ff7700 1 -- 172.21.15.90:0/3942933564 shutdown_connections 2024-04-25T11:33:46.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.765+0000 7f6fc1ff7700 1 --2- 172.21.15.90:0/3942933564 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6fbc1085e0 0x7f6fbc10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:46.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.765+0000 7f6fc1ff7700 1 --2- 172.21.15.90:0/3942933564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fbc103360 0x7f6fbc103770 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:46.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.765+0000 7f6fc1ff7700 1 --2- 172.21.15.90:0/3942933564 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fbc107c90 0x7f6fbc1080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:46.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.765+0000 7f6fc1ff7700 1 -- 172.21.15.90:0/3942933564 >> 172.21.15.90:0/3942933564 conn(0x7f6fbc0fec90 msgr2=0x7f6fbc1010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:46.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.765+0000 7f6fc1ff7700 1 -- 172.21.15.90:0/3942933564 shutdown_connections 2024-04-25T11:33:46.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.765+0000 7f6fc1ff7700 1 -- 172.21.15.90:0/3942933564 wait complete. 2024-04-25T11:33:46.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.765+0000 7f6fc1ff7700 1 Processor -- start 2024-04-25T11:33:46.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.766+0000 7f6fc1ff7700 1 -- start start 2024-04-25T11:33:46.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.766+0000 7f6fc1ff7700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6fbc103360 0x7f6fbc19bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:46.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.766+0000 7f6fc1ff7700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fbc107c90 0x7f6fbc19c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:46.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.766+0000 7f6fc1ff7700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fbc1085e0 0x7f6fbc19c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:46.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.766+0000 7f6fc1ff7700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6fbc113210 con 0x7f6fbc107c90 2024-04-25T11:33:46.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.766+0000 7f6fc1ff7700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6fbc113090 con 0x7f6fbc103360 2024-04-25T11:33:46.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.766+0000 7f6fc1ff7700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6fbc113390 con 0x7f6fbc1085e0 2024-04-25T11:33:46.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.766+0000 7f6fbaffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fbc107c90 0x7f6fbc19c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:46.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.766+0000 7f6fbaffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fbc107c90 0x7f6fbc19c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50798/0 (socket says 172.21.15.90:50798) 2024-04-25T11:33:46.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.766+0000 7f6fbaffd700 1 -- 172.21.15.90:0/2278328917 learned_addr learned my addr 172.21.15.90:0/2278328917 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:46.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.766+0000 7f6fbbfff700 1 --2- 172.21.15.90:0/2278328917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fbc1085e0 0x7f6fbc19c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:46.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.766+0000 7f6fbb7fe700 1 --2- 172.21.15.90:0/2278328917 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6fbc103360 0x7f6fbc19bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:46.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.766+0000 7f6fbaffd700 1 -- 172.21.15.90:0/2278328917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fbc1085e0 msgr2=0x7f6fbc19c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:46.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.766+0000 7f6fbaffd700 1 --2- 172.21.15.90:0/2278328917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fbc1085e0 0x7f6fbc19c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:46.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.766+0000 7f6fbaffd700 1 -- 172.21.15.90:0/2278328917 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6fbc103360 msgr2=0x7f6fbc19bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:46.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.766+0000 7f6fbaffd700 1 --2- 172.21.15.90:0/2278328917 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6fbc103360 0x7f6fbc19bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:46.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.766+0000 7f6fbaffd700 1 -- 172.21.15.90:0/2278328917 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fbc1a6e60 con 0x7f6fbc107c90 2024-04-25T11:33:46.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.766+0000 7f6fbaffd700 1 --2- 172.21.15.90:0/2278328917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fbc107c90 0x7f6fbc19c050 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6fb000d980 tx=0x7f6fb000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:46.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.766+0000 7f6fb8ff9700 1 -- 172.21.15.90:0/2278328917 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0017070 con 0x7f6fbc107c90 2024-04-25T11:33:46.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.767+0000 7f6fb8ff9700 1 -- 172.21.15.90:0/2278328917 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6fb0009de0 con 0x7f6fbc107c90 2024-04-25T11:33:46.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.767+0000 7f6fc1ff7700 1 -- 172.21.15.90:0/2278328917 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fbc1a7150 con 0x7f6fbc107c90 2024-04-25T11:33:46.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.767+0000 7f6fb8ff9700 1 -- 172.21.15.90:0/2278328917 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb000f3c0 con 0x7f6fbc107c90 2024-04-25T11:33:46.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.767+0000 7f6fc1ff7700 1 -- 172.21.15.90:0/2278328917 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fbc1a7690 con 0x7f6fbc107c90 2024-04-25T11:33:46.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.767+0000 7f6fc1ff7700 1 -- 172.21.15.90:0/2278328917 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6fbc063da0 con 0x7f6fbc107c90 2024-04-25T11:33:46.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.769+0000 7f6fb8ff9700 1 -- 172.21.15.90:0/2278328917 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f6fb0009930 con 0x7f6fbc107c90 2024-04-25T11:33:46.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.770+0000 7f6fb8ff9700 1 --2- 172.21.15.90:0/2278328917 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6fa40afc10 0x7f6fa40b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:46.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.770+0000 7f6fbb7fe700 1 --2- 172.21.15.90:0/2278328917 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6fa40afc10 0x7f6fa40b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:46.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.770+0000 7f6fb8ff9700 1 -- 172.21.15.90:0/2278328917 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7f6fb00deb30 con 0x7f6fbc107c90 2024-04-25T11:33:46.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.770+0000 7f6fb8ff9700 1 --2- 172.21.15.90:0/2278328917 >> [v2:172.21.15.138:6800/3976851933,v1:172.21.15.138:6801/3976851933] conn(0x7f6fa40b9440 0x7f6fa40bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:46.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.770+0000 7f6fb8ff9700 1 -- 172.21.15.90:0/2278328917 --> [v2:172.21.15.138:6800/3976851933,v1:172.21.15.138:6801/3976851933] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6fb0001d30 con 0x7f6fa40b9440 2024-04-25T11:33:46.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.770+0000 7f6fb8ff9700 1 -- 172.21.15.90:0/2278328917 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=77) ==== 24+0+0 (secure 0 0 0) 0x7f6fb00df010 con 0x7f6fbc107c90 2024-04-25T11:33:46.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.770+0000 7f6fbb7fe700 1 --2- 172.21.15.90:0/2278328917 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6fa40afc10 0x7f6fa40b2120 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6fac003800 tx=0x7f6fac000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:46.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.770+0000 7f6fbbfff700 1 --2- 172.21.15.90:0/2278328917 >> [v2:172.21.15.138:6800/3976851933,v1:172.21.15.138:6801/3976851933] conn(0x7f6fa40b9440 0x7f6fa40bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:46.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.771+0000 7f6fbbfff700 1 --2- 172.21.15.90:0/2278328917 >> [v2:172.21.15.138:6800/3976851933,v1:172.21.15.138:6801/3976851933] conn(0x7f6fa40b9440 0x7f6fa40bb890 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:46.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.772+0000 7f6fb8ff9700 1 -- 172.21.15.90:0/2278328917 <== osd.1 v2:172.21.15.138:6800/3976851933 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6fb0001d30 con 0x7f6fa40b9440 2024-04-25T11:33:46.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.788+0000 7f6fc1ff7700 1 -- 172.21.15.90:0/2278328917 --> [v2:172.21.15.138:6800/3976851933,v1:172.21.15.138:6801/3976851933] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6fbc064220 con 0x7f6fa40b9440 2024-04-25T11:33:46.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.788+0000 7f6fb8ff9700 1 -- 172.21.15.90:0/2278328917 <== osd.1 v2:172.21.15.138:6800/3976851933 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6fbc064220 con 0x7f6fa40b9440 2024-04-25T11:33:46.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.788+0000 7f6fc1ff7700 1 -- 172.21.15.90:0/2278328917 >> [v2:172.21.15.138:6800/3976851933,v1:172.21.15.138:6801/3976851933] conn(0x7f6fa40b9440 msgr2=0x7f6fa40bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:46.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.788+0000 7f6fc1ff7700 1 --2- 172.21.15.90:0/2278328917 >> [v2:172.21.15.138:6800/3976851933,v1:172.21.15.138:6801/3976851933] conn(0x7f6fa40b9440 0x7f6fa40bb890 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:46.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.788+0000 7f6fc1ff7700 1 -- 172.21.15.90:0/2278328917 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6fa40afc10 msgr2=0x7f6fa40b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:46.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.788+0000 7f6fc1ff7700 1 --2- 172.21.15.90:0/2278328917 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6fa40afc10 0x7f6fa40b2120 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6fac003800 tx=0x7f6fac000bc0 comp rx=0 tx=0).stop 2024-04-25T11:33:46.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.789+0000 7f6fc1ff7700 1 -- 172.21.15.90:0/2278328917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fbc107c90 msgr2=0x7f6fbc19c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:46.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.789+0000 7f6fc1ff7700 1 --2- 172.21.15.90:0/2278328917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fbc107c90 0x7f6fbc19c050 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6fb000d980 tx=0x7f6fb000dc90 comp rx=0 tx=0).stop 2024-04-25T11:33:46.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.789+0000 7f6fbbfff700 1 -- 172.21.15.90:0/2278328917 reap_dead start 2024-04-25T11:33:46.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.789+0000 7f6fc1ff7700 1 -- 172.21.15.90:0/2278328917 shutdown_connections 2024-04-25T11:33:46.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.789+0000 7f6fc1ff7700 1 -- 172.21.15.90:0/2278328917 >> 172.21.15.90:0/2278328917 conn(0x7f6fbc0fec90 msgr2=0x7f6fbc101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:46.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.789+0000 7f6fc1ff7700 1 -- 172.21.15.90:0/2278328917 shutdown_connections 2024-04-25T11:33:46.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:43.789+0000 7f6fc1ff7700 1 -- 172.21.15.90:0/2278328917 wait complete. 2024-04-25T11:33:47.271 INFO:teuthology.orchestra.run.smithi090.stdout:55834574918 2024-04-25T11:33:47.271 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-25T11:33:47.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.508+0000 7f9841857700 1 -- 172.21.15.90:0/1640353723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f983c108d90 msgr2=0x7f983c1091a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:47.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.508+0000 7f9841857700 1 --2- 172.21.15.90:0/1640353723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f983c108d90 0x7f983c1091a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f982c0099c0 tx=0x7f982c009cd0 comp rx=0 tx=0).stop 2024-04-25T11:33:47.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.508+0000 7f9841857700 1 -- 172.21.15.90:0/1640353723 shutdown_connections 2024-04-25T11:33:47.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.508+0000 7f9841857700 1 --2- 172.21.15.90:0/1640353723 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f983c1096e0 0x7f983c111040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:47.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.508+0000 7f9841857700 1 --2- 172.21.15.90:0/1640353723 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f983c104460 0x7f983c104870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:47.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.508+0000 7f9841857700 1 --2- 172.21.15.90:0/1640353723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f983c108d90 0x7f983c1091a0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:47.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.508+0000 7f9841857700 1 -- 172.21.15.90:0/1640353723 >> 172.21.15.90:0/1640353723 conn(0x7f983c0ffd40 msgr2=0x7f983c102160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:47.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.508+0000 7f9841857700 1 -- 172.21.15.90:0/1640353723 shutdown_connections 2024-04-25T11:33:47.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.508+0000 7f9841857700 1 -- 172.21.15.90:0/1640353723 wait complete. 2024-04-25T11:33:47.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.509+0000 7f9841857700 1 Processor -- start 2024-04-25T11:33:47.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.509+0000 7f9841857700 1 -- start start 2024-04-25T11:33:47.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.509+0000 7f9841857700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f983c104460 0x7f983c19ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:47.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.509+0000 7f9841857700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f983c108d90 0x7f983c19b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:47.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.509+0000 7f9841857700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f983c1096e0 0x7f983c19f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:47.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.509+0000 7f9841857700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f983c114360 con 0x7f983c108d90 2024-04-25T11:33:47.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.509+0000 7f9841857700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f983c1141e0 con 0x7f983c1096e0 2024-04-25T11:33:47.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.509+0000 7f9841857700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f983c1144e0 con 0x7f983c104460 2024-04-25T11:33:47.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.509+0000 7f983a7fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f983c108d90 0x7f983c19b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:47.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.509+0000 7f983a7fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f983c108d90 0x7f983c19b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50864/0 (socket says 172.21.15.90:50864) 2024-04-25T11:33:47.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.509+0000 7f983a7fc700 1 -- 172.21.15.90:0/197568572 learned_addr learned my addr 172.21.15.90:0/197568572 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:47.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.509+0000 7f983b7fe700 1 --2- 172.21.15.90:0/197568572 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f983c1096e0 0x7f983c19f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:47.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.509+0000 7f983affd700 1 --2- 172.21.15.90:0/197568572 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f983c104460 0x7f983c19ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:47.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.510+0000 7f983a7fc700 1 -- 172.21.15.90:0/197568572 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f983c104460 msgr2=0x7f983c19ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:47.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.510+0000 7f983a7fc700 1 --2- 172.21.15.90:0/197568572 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f983c104460 0x7f983c19ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:47.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.510+0000 7f983a7fc700 1 -- 172.21.15.90:0/197568572 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f983c1096e0 msgr2=0x7f983c19f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:47.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.510+0000 7f983a7fc700 1 --2- 172.21.15.90:0/197568572 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f983c1096e0 0x7f983c19f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:47.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.510+0000 7f983a7fc700 1 -- 172.21.15.90:0/197568572 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f983c19fc60 con 0x7f983c108d90 2024-04-25T11:33:47.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.510+0000 7f983b7fe700 1 --2- 172.21.15.90:0/197568572 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f983c1096e0 0x7f983c19f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:33:47.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.510+0000 7f983affd700 1 --2- 172.21.15.90:0/197568572 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f983c104460 0x7f983c19ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:33:47.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.510+0000 7f983a7fc700 1 --2- 172.21.15.90:0/197568572 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f983c108d90 0x7f983c19b330 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f982c009f50 tx=0x7f982c0037c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:47.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.510+0000 7f9840855700 1 -- 172.21.15.90:0/197568572 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982c027070 con 0x7f983c108d90 2024-04-25T11:33:47.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.510+0000 7f9840855700 1 -- 172.21.15.90:0/197568572 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f982c004140 con 0x7f983c108d90 2024-04-25T11:33:47.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.510+0000 7f9840855700 1 -- 172.21.15.90:0/197568572 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982c00f460 con 0x7f983c108d90 2024-04-25T11:33:47.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.511+0000 7f9841857700 1 -- 172.21.15.90:0/197568572 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f983c19fef0 con 0x7f983c108d90 2024-04-25T11:33:47.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.511+0000 7f9841857700 1 -- 172.21.15.90:0/197568572 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f983c1a0380 con 0x7f983c108d90 2024-04-25T11:33:47.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.511+0000 7f9841857700 1 -- 172.21.15.90:0/197568572 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f983c063da0 con 0x7f983c108d90 2024-04-25T11:33:47.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.513+0000 7f9840855700 1 -- 172.21.15.90:0/197568572 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f982c02a070 con 0x7f983c108d90 2024-04-25T11:33:47.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.514+0000 7f9840855700 1 --2- 172.21.15.90:0/197568572 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f98240afc10 0x7f98240b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:47.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.514+0000 7f983affd700 1 --2- 172.21.15.90:0/197568572 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f98240afc10 0x7f98240b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:47.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.514+0000 7f9840855700 1 -- 172.21.15.90:0/197568572 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7f982c0e00d0 con 0x7f983c108d90 2024-04-25T11:33:47.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.514+0000 7f9840855700 1 --2- 172.21.15.90:0/197568572 >> [v2:172.21.15.138:6808/3752678279,v1:172.21.15.138:6809/3752678279] conn(0x7f98240b9440 0x7f98240bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:47.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.514+0000 7f9840855700 1 -- 172.21.15.90:0/197568572 --> [v2:172.21.15.138:6808/3752678279,v1:172.21.15.138:6809/3752678279] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f98240bbf60 con 0x7f98240b9440 2024-04-25T11:33:47.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.514+0000 7f9840855700 1 -- 172.21.15.90:0/197568572 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=77) ==== 24+0+0 (secure 0 0 0) 0x7f982c0e05b0 con 0x7f983c108d90 2024-04-25T11:33:47.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.514+0000 7f983affd700 1 --2- 172.21.15.90:0/197568572 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f98240afc10 0x7f98240b2120 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9830005fd0 tx=0x7f9830005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:47.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.514+0000 7f983b7fe700 1 --2- 172.21.15.90:0/197568572 >> [v2:172.21.15.138:6808/3752678279,v1:172.21.15.138:6809/3752678279] conn(0x7f98240b9440 0x7f98240bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:47.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.515+0000 7f983b7fe700 1 --2- 172.21.15.90:0/197568572 >> [v2:172.21.15.138:6808/3752678279,v1:172.21.15.138:6809/3752678279] conn(0x7f98240b9440 0x7f98240bb890 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:47.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.516+0000 7f9840855700 1 -- 172.21.15.90:0/197568572 <== osd.4 v2:172.21.15.138:6808/3752678279 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f98240bbf60 con 0x7f98240b9440 2024-04-25T11:33:47.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.531+0000 7f9841857700 1 -- 172.21.15.90:0/197568572 --> [v2:172.21.15.138:6808/3752678279,v1:172.21.15.138:6809/3752678279] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f983c0641a0 con 0x7f98240b9440 2024-04-25T11:33:47.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.532+0000 7f9840855700 1 -- 172.21.15.90:0/197568572 <== osd.4 v2:172.21.15.138:6808/3752678279 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f983c0641a0 con 0x7f98240b9440 2024-04-25T11:33:47.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.533+0000 7f9841857700 1 -- 172.21.15.90:0/197568572 >> [v2:172.21.15.138:6808/3752678279,v1:172.21.15.138:6809/3752678279] conn(0x7f98240b9440 msgr2=0x7f98240bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:47.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.533+0000 7f9841857700 1 --2- 172.21.15.90:0/197568572 >> [v2:172.21.15.138:6808/3752678279,v1:172.21.15.138:6809/3752678279] conn(0x7f98240b9440 0x7f98240bb890 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:47.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.533+0000 7f9841857700 1 -- 172.21.15.90:0/197568572 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f98240afc10 msgr2=0x7f98240b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:47.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.533+0000 7f9841857700 1 --2- 172.21.15.90:0/197568572 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f98240afc10 0x7f98240b2120 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9830005fd0 tx=0x7f9830005dc0 comp rx=0 tx=0).stop 2024-04-25T11:33:47.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.533+0000 7f9841857700 1 -- 172.21.15.90:0/197568572 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f983c108d90 msgr2=0x7f983c19b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:47.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.533+0000 7f9841857700 1 --2- 172.21.15.90:0/197568572 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f983c108d90 0x7f983c19b330 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f982c009f50 tx=0x7f982c0037c0 comp rx=0 tx=0).stop 2024-04-25T11:33:47.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.533+0000 7f983b7fe700 1 -- 172.21.15.90:0/197568572 reap_dead start 2024-04-25T11:33:47.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.533+0000 7f9841857700 1 -- 172.21.15.90:0/197568572 shutdown_connections 2024-04-25T11:33:47.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.533+0000 7f9841857700 1 -- 172.21.15.90:0/197568572 >> 172.21.15.90:0/197568572 conn(0x7f983c0ffd40 msgr2=0x7f983c10a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:47.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.533+0000 7f9841857700 1 -- 172.21.15.90:0/197568572 shutdown_connections 2024-04-25T11:33:47.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.533+0000 7f9841857700 1 -- 172.21.15.90:0/197568572 wait complete. 2024-04-25T11:33:47.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.959+0000 7f52267c8700 1 -- 172.21.15.90:0/3155803345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f52201085e0 msgr2=0x7f522010ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:47.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.959+0000 7f52267c8700 1 --2- 172.21.15.90:0/3155803345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f52201085e0 0x7f522010ff40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f521800b0d0 tx=0x7f521800b3e0 comp rx=0 tx=0).stop 2024-04-25T11:33:47.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.959+0000 7f52267c8700 1 -- 172.21.15.90:0/3155803345 shutdown_connections 2024-04-25T11:33:47.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.959+0000 7f52267c8700 1 --2- 172.21.15.90:0/3155803345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f52201085e0 0x7f522010ff40 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:47.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.959+0000 7f52267c8700 1 --2- 172.21.15.90:0/3155803345 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5220103360 0x7f5220103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:47.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.959+0000 7f52267c8700 1 --2- 172.21.15.90:0/3155803345 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5220107c90 0x7f52201080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:47.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.959+0000 7f52267c8700 1 -- 172.21.15.90:0/3155803345 >> 172.21.15.90:0/3155803345 conn(0x7f52200fec90 msgr2=0x7f52201010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:47.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.959+0000 7f52267c8700 1 -- 172.21.15.90:0/3155803345 shutdown_connections 2024-04-25T11:33:47.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.959+0000 7f52267c8700 1 -- 172.21.15.90:0/3155803345 wait complete. 2024-04-25T11:33:47.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.960+0000 7f52267c8700 1 Processor -- start 2024-04-25T11:33:47.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.960+0000 7f52267c8700 1 -- start start 2024-04-25T11:33:47.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.960+0000 7f52267c8700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5220103360 0x7f522019f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:47.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.960+0000 7f52267c8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5220107c90 0x7f522019fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:47.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.960+0000 7f52267c8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52201085e0 0x7f52201999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:47.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.960+0000 7f52267c8700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5220113210 con 0x7f5220107c90 2024-04-25T11:33:47.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.960+0000 7f52267c8700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5220113090 con 0x7f5220103360 2024-04-25T11:33:47.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.960+0000 7f52267c8700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5220113390 con 0x7f52201085e0 2024-04-25T11:33:47.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.960+0000 7f521f7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5220107c90 0x7f522019fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:47.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.960+0000 7f521f7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5220107c90 0x7f522019fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50870/0 (socket says 172.21.15.90:50870) 2024-04-25T11:33:47.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.960+0000 7f521f7fe700 1 -- 172.21.15.90:0/4233184319 learned_addr learned my addr 172.21.15.90:0/4233184319 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:47.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.960+0000 7f521ffff700 1 --2- 172.21.15.90:0/4233184319 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5220103360 0x7f522019f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:47.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.960+0000 7f521f7fe700 1 -- 172.21.15.90:0/4233184319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52201085e0 msgr2=0x7f52201999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:47.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.961+0000 7f521f7fe700 1 --2- 172.21.15.90:0/4233184319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52201085e0 0x7f52201999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:47.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.961+0000 7f521f7fe700 1 -- 172.21.15.90:0/4233184319 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5220103360 msgr2=0x7f522019f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:47.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.961+0000 7f521f7fe700 1 --2- 172.21.15.90:0/4233184319 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5220103360 0x7f522019f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:47.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.961+0000 7f521f7fe700 1 -- 172.21.15.90:0/4233184319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f522019a060 con 0x7f5220107c90 2024-04-25T11:33:47.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.961+0000 7f521ffff700 1 --2- 172.21.15.90:0/4233184319 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5220103360 0x7f522019f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:33:47.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.961+0000 7f521f7fe700 1 --2- 172.21.15.90:0/4233184319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5220107c90 0x7f522019fcd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f521000e970 tx=0x7f521000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:47.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.961+0000 7f521d7fa700 1 -- 172.21.15.90:0/4233184319 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521000cbd0 con 0x7f5220107c90 2024-04-25T11:33:47.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.961+0000 7f52267c8700 1 -- 172.21.15.90:0/4233184319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f522019a350 con 0x7f5220107c90 2024-04-25T11:33:47.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.961+0000 7f521d7fa700 1 -- 172.21.15.90:0/4233184319 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f521000cd70 con 0x7f5220107c90 2024-04-25T11:33:47.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.961+0000 7f521d7fa700 1 -- 172.21.15.90:0/4233184319 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5210010430 con 0x7f5220107c90 2024-04-25T11:33:47.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.961+0000 7f52267c8700 1 -- 172.21.15.90:0/4233184319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f522006b4e0 con 0x7f5220107c90 2024-04-25T11:33:47.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.962+0000 7f52267c8700 1 -- 172.21.15.90:0/4233184319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f522006baf0 con 0x7f5220107c90 2024-04-25T11:33:47.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.963+0000 7f521d7fa700 1 -- 172.21.15.90:0/4233184319 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f5210004d70 con 0x7f5220107c90 2024-04-25T11:33:47.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.964+0000 7f521d7fa700 1 --2- 172.21.15.90:0/4233184319 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f52080afc10 0x7f52080b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:47.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.964+0000 7f521ffff700 1 --2- 172.21.15.90:0/4233184319 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f52080afc10 0x7f52080b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:47.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.964+0000 7f521d7fa700 1 -- 172.21.15.90:0/4233184319 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7f52100df2a0 con 0x7f5220107c90 2024-04-25T11:33:47.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.964+0000 7f521d7fa700 1 --2- 172.21.15.90:0/4233184319 >> [v2:172.21.15.156:6808/3591210191,v1:172.21.15.156:6809/3591210191] conn(0x7f52080b9440 0x7f52080bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:47.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.964+0000 7f521d7fa700 1 -- 172.21.15.90:0/4233184319 --> [v2:172.21.15.156:6808/3591210191,v1:172.21.15.156:6809/3591210191] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5210001d30 con 0x7f52080b9440 2024-04-25T11:33:47.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.964+0000 7f521d7fa700 1 -- 172.21.15.90:0/4233184319 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=77) ==== 24+0+0 (secure 0 0 0) 0x7f52100df780 con 0x7f5220107c90 2024-04-25T11:33:47.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.964+0000 7f521ffff700 1 --2- 172.21.15.90:0/4233184319 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f52080afc10 0x7f52080b2120 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5214005fd0 tx=0x7f5214005f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:47.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.965+0000 7f5224d65700 1 --2- 172.21.15.90:0/4233184319 >> [v2:172.21.15.156:6808/3591210191,v1:172.21.15.156:6809/3591210191] conn(0x7f52080b9440 0x7f52080bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:47.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.965+0000 7f5224d65700 1 --2- 172.21.15.90:0/4233184319 >> [v2:172.21.15.156:6808/3591210191,v1:172.21.15.156:6809/3591210191] conn(0x7f52080b9440 0x7f52080bb890 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:47.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.966+0000 7f521d7fa700 1 -- 172.21.15.90:0/4233184319 <== osd.5 v2:172.21.15.156:6808/3591210191 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5210001d30 con 0x7f52080b9440 2024-04-25T11:33:47.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.982+0000 7f52267c8700 1 -- 172.21.15.90:0/4233184319 --> [v2:172.21.15.156:6808/3591210191,v1:172.21.15.156:6809/3591210191] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f522006bf70 con 0x7f52080b9440 2024-04-25T11:33:47.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.983+0000 7f521d7fa700 1 -- 172.21.15.90:0/4233184319 <== osd.5 v2:172.21.15.156:6808/3591210191 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f522006bf70 con 0x7f52080b9440 2024-04-25T11:33:47.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.983+0000 7f52267c8700 1 -- 172.21.15.90:0/4233184319 >> [v2:172.21.15.156:6808/3591210191,v1:172.21.15.156:6809/3591210191] conn(0x7f52080b9440 msgr2=0x7f52080bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:47.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.983+0000 7f52267c8700 1 --2- 172.21.15.90:0/4233184319 >> [v2:172.21.15.156:6808/3591210191,v1:172.21.15.156:6809/3591210191] conn(0x7f52080b9440 0x7f52080bb890 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:47.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.983+0000 7f52267c8700 1 -- 172.21.15.90:0/4233184319 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f52080afc10 msgr2=0x7f52080b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:47.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.983+0000 7f52267c8700 1 --2- 172.21.15.90:0/4233184319 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f52080afc10 0x7f52080b2120 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5214005fd0 tx=0x7f5214005f00 comp rx=0 tx=0).stop 2024-04-25T11:33:47.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.983+0000 7f52267c8700 1 -- 172.21.15.90:0/4233184319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5220107c90 msgr2=0x7f522019fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:47.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.983+0000 7f52267c8700 1 --2- 172.21.15.90:0/4233184319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5220107c90 0x7f522019fcd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f521000e970 tx=0x7f521000ed30 comp rx=0 tx=0).stop 2024-04-25T11:33:47.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.983+0000 7f5224d65700 1 -- 172.21.15.90:0/4233184319 reap_dead start 2024-04-25T11:33:47.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.983+0000 7f52267c8700 1 -- 172.21.15.90:0/4233184319 shutdown_connections 2024-04-25T11:33:47.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.983+0000 7f52267c8700 1 -- 172.21.15.90:0/4233184319 >> 172.21.15.90:0/4233184319 conn(0x7f52200fec90 msgr2=0x7f5220107400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:47.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.983+0000 7f52267c8700 1 -- 172.21.15.90:0/4233184319 shutdown_connections 2024-04-25T11:33:47.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:47.983+0000 7f52267c8700 1 -- 172.21.15.90:0/4233184319 wait complete. 2024-04-25T11:33:48.020 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:47 smithi090 ceph-mon[29575]: pgmap v292: 97 pgs: 97 active+clean; 641 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-25T11:33:48.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:47 smithi156 ceph-mon[33045]: pgmap v292: 97 pgs: 97 active+clean; 641 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-25T11:33:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.143+0000 7f72ce99f700 1 -- 172.21.15.90:0/3872926159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72c81031b0 msgr2=0x7f72c81035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.143+0000 7f72ce99f700 1 --2- 172.21.15.90:0/3872926159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72c81031b0 0x7f72c81035c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f72c40099c0 tx=0x7f72c4009cd0 comp rx=0 tx=0).stop 2024-04-25T11:33:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.143+0000 7f72ce99f700 1 -- 172.21.15.90:0/3872926159 shutdown_connections 2024-04-25T11:33:48.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.143+0000 7f72ce99f700 1 --2- 172.21.15.90:0/3872926159 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72c8108430 0x7f72c810fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:48.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.143+0000 7f72ce99f700 1 --2- 172.21.15.90:0/3872926159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72c81031b0 0x7f72c81035c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:48.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.143+0000 7f72ce99f700 1 --2- 172.21.15.90:0/3872926159 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f72c8107ae0 0x7f72c8107ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:48.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.143+0000 7f72ce99f700 1 -- 172.21.15.90:0/3872926159 >> 172.21.15.90:0/3872926159 conn(0x7f72c8078990 msgr2=0x7f72c8078df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:48.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.143+0000 7f72ce99f700 1 -- 172.21.15.90:0/3872926159 shutdown_connections 2024-04-25T11:33:48.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.143+0000 7f72ce99f700 1 -- 172.21.15.90:0/3872926159 wait complete. 2024-04-25T11:33:48.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.144+0000 7f72ce99f700 1 Processor -- start 2024-04-25T11:33:48.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.144+0000 7f72ce99f700 1 -- start start 2024-04-25T11:33:48.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.144+0000 7f72ce99f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72c81031b0 0x7f72c819b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:48.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.144+0000 7f72ce99f700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f72c8107ae0 0x7f72c819bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:48.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.144+0000 7f72ce99f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72c8108430 0x7f72c819c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:48.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.144+0000 7f72ce99f700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f72c81130e0 con 0x7f72c81031b0 2024-04-25T11:33:48.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.144+0000 7f72ce99f700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f72c8112f60 con 0x7f72c8107ae0 2024-04-25T11:33:48.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.144+0000 7f72ce99f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f72c8113260 con 0x7f72c8108430 2024-04-25T11:33:48.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.144+0000 7f72cd99d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72c81031b0 0x7f72c819b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:48.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.144+0000 7f72cd99d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72c81031b0 0x7f72c819b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50882/0 (socket says 172.21.15.90:50882) 2024-04-25T11:33:48.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.144+0000 7f72cd99d700 1 -- 172.21.15.90:0/1508355973 learned_addr learned my addr 172.21.15.90:0/1508355973 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:48.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.144+0000 7f72cd19c700 1 --2- 172.21.15.90:0/1508355973 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f72c8107ae0 0x7f72c819bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:48.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.144+0000 7f72ce19e700 1 --2- 172.21.15.90:0/1508355973 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72c8108430 0x7f72c819c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:48.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.144+0000 7f72cd99d700 1 -- 172.21.15.90:0/1508355973 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72c8108430 msgr2=0x7f72c819c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:48.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.144+0000 7f72cd99d700 1 --2- 172.21.15.90:0/1508355973 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72c8108430 0x7f72c819c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:48.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.144+0000 7f72cd99d700 1 -- 172.21.15.90:0/1508355973 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f72c8107ae0 msgr2=0x7f72c819bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:48.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.144+0000 7f72cd99d700 1 --2- 172.21.15.90:0/1508355973 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f72c8107ae0 0x7f72c819bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:48.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.144+0000 7f72cd99d700 1 -- 172.21.15.90:0/1508355973 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72c81a6c80 con 0x7f72c81031b0 2024-04-25T11:33:48.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.144+0000 7f72ce19e700 1 --2- 172.21.15.90:0/1508355973 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72c8108430 0x7f72c819c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:33:48.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.145+0000 7f72cd99d700 1 --2- 172.21.15.90:0/1508355973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72c81031b0 0x7f72c819b9c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f72c400b580 tx=0x7f72c40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:48.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.145+0000 7f72beffd700 1 -- 172.21.15.90:0/1508355973 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c4027070 con 0x7f72c81031b0 2024-04-25T11:33:48.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.145+0000 7f72beffd700 1 -- 172.21.15.90:0/1508355973 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72c40040a0 con 0x7f72c81031b0 2024-04-25T11:33:48.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.145+0000 7f72ce99f700 1 -- 172.21.15.90:0/1508355973 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72c81a6f10 con 0x7f72c81031b0 2024-04-25T11:33:48.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.145+0000 7f72beffd700 1 -- 172.21.15.90:0/1508355973 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c400f460 con 0x7f72c81031b0 2024-04-25T11:33:48.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.145+0000 7f72ce99f700 1 -- 172.21.15.90:0/1508355973 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72c81a7300 con 0x7f72c81031b0 2024-04-25T11:33:48.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.146+0000 7f72ce99f700 1 -- 172.21.15.90:0/1508355973 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f72c8063da0 con 0x7f72c81031b0 2024-04-25T11:33:48.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.148+0000 7f72beffd700 1 -- 172.21.15.90:0/1508355973 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f72c402a070 con 0x7f72c81031b0 2024-04-25T11:33:48.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.148+0000 7f72beffd700 1 --2- 172.21.15.90:0/1508355973 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f72b40afc10 0x7f72b40b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:48.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.148+0000 7f72cd19c700 1 --2- 172.21.15.90:0/1508355973 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f72b40afc10 0x7f72b40b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:48.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.148+0000 7f72beffd700 1 -- 172.21.15.90:0/1508355973 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7f72c40e03f0 con 0x7f72c81031b0 2024-04-25T11:33:48.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.148+0000 7f72beffd700 1 --2- 172.21.15.90:0/1508355973 >> [v2:172.21.15.156:6816/101092224,v1:172.21.15.156:6817/101092224] conn(0x7f72b40b9440 0x7f72b40bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:48.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.148+0000 7f72beffd700 1 -- 172.21.15.90:0/1508355973 --> [v2:172.21.15.156:6816/101092224,v1:172.21.15.156:6817/101092224] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f72b40bbf60 con 0x7f72b40b9440 2024-04-25T11:33:48.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.148+0000 7f72beffd700 1 -- 172.21.15.90:0/1508355973 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=77) ==== 24+0+0 (secure 0 0 0) 0x7f72c40e08d0 con 0x7f72c81031b0 2024-04-25T11:33:48.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.149+0000 7f72cd19c700 1 --2- 172.21.15.90:0/1508355973 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f72b40afc10 0x7f72b40b2120 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f72c80ffc90 tx=0x7f72b800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:48.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.149+0000 7f72ce19e700 1 --2- 172.21.15.90:0/1508355973 >> [v2:172.21.15.156:6816/101092224,v1:172.21.15.156:6817/101092224] conn(0x7f72b40b9440 0x7f72b40bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:48.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.149+0000 7f72ce19e700 1 --2- 172.21.15.90:0/1508355973 >> [v2:172.21.15.156:6816/101092224,v1:172.21.15.156:6817/101092224] conn(0x7f72b40b9440 0x7f72b40bb890 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:48.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.150+0000 7f72beffd700 1 -- 172.21.15.90:0/1508355973 <== osd.8 v2:172.21.15.156:6816/101092224 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f72b40bbf60 con 0x7f72b40b9440 2024-04-25T11:33:48.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.166+0000 7f72ce99f700 1 -- 172.21.15.90:0/1508355973 --> [v2:172.21.15.156:6816/101092224,v1:172.21.15.156:6817/101092224] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f72c8064220 con 0x7f72b40b9440 2024-04-25T11:33:48.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.167+0000 7f72beffd700 1 -- 172.21.15.90:0/1508355973 <== osd.8 v2:172.21.15.156:6816/101092224 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f72c8064220 con 0x7f72b40b9440 2024-04-25T11:33:48.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.167+0000 7f72ce99f700 1 -- 172.21.15.90:0/1508355973 >> [v2:172.21.15.156:6816/101092224,v1:172.21.15.156:6817/101092224] conn(0x7f72b40b9440 msgr2=0x7f72b40bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:48.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.167+0000 7f72ce99f700 1 --2- 172.21.15.90:0/1508355973 >> [v2:172.21.15.156:6816/101092224,v1:172.21.15.156:6817/101092224] conn(0x7f72b40b9440 0x7f72b40bb890 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:48.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.168+0000 7f72ce99f700 1 -- 172.21.15.90:0/1508355973 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f72b40afc10 msgr2=0x7f72b40b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:48.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.168+0000 7f72ce99f700 1 --2- 172.21.15.90:0/1508355973 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f72b40afc10 0x7f72b40b2120 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f72c80ffc90 tx=0x7f72b800a380 comp rx=0 tx=0).stop 2024-04-25T11:33:48.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.168+0000 7f72ce99f700 1 -- 172.21.15.90:0/1508355973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72c81031b0 msgr2=0x7f72c819b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:48.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.168+0000 7f72ce99f700 1 --2- 172.21.15.90:0/1508355973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72c81031b0 0x7f72c819b9c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f72c400b580 tx=0x7f72c40047b0 comp rx=0 tx=0).stop 2024-04-25T11:33:48.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.168+0000 7f72ce19e700 1 -- 172.21.15.90:0/1508355973 reap_dead start 2024-04-25T11:33:48.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.168+0000 7f72ce99f700 1 -- 172.21.15.90:0/1508355973 shutdown_connections 2024-04-25T11:33:48.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.168+0000 7f72ce99f700 1 -- 172.21.15.90:0/1508355973 >> 172.21.15.90:0/1508355973 conn(0x7f72c8078990 msgr2=0x7f72c81006a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:48.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.168+0000 7f72ce99f700 1 -- 172.21.15.90:0/1508355973 shutdown_connections 2024-04-25T11:33:48.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.168+0000 7f72ce99f700 1 -- 172.21.15.90:0/1508355973 wait complete. 2024-04-25T11:33:48.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:47 smithi138 ceph-mon[32328]: pgmap v292: 97 pgs: 97 active+clean; 641 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-25T11:33:48.951 INFO:teuthology.orchestra.run.smithi090.stdout:180388626477 2024-04-25T11:33:48.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-25T11:33:50.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:49 smithi156 ceph-mon[33045]: pgmap v293: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-25T11:33:50.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:49 smithi090 ceph-mon[29575]: pgmap v293: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-25T11:33:50.287 INFO:teuthology.orchestra.run.smithi090.stdout:133143986231 2024-04-25T11:33:50.287 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-25T11:33:50.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:49 smithi138 ceph-mon[32328]: pgmap v293: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-25T11:33:51.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.527+0000 7f4b26192700 1 -- 172.21.15.90:0/1471156337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b20103380 msgr2=0x7f4b20103790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:51.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.527+0000 7f4b26192700 1 --2- 172.21.15.90:0/1471156337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b20103380 0x7f4b20103790 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4b100099c0 tx=0x7f4b10009cd0 comp rx=0 tx=0).stop 2024-04-25T11:33:51.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.527+0000 7f4b26192700 1 -- 172.21.15.90:0/1471156337 shutdown_connections 2024-04-25T11:33:51.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.527+0000 7f4b26192700 1 --2- 172.21.15.90:0/1471156337 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4b20108600 0x7f4b2010ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:51.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.527+0000 7f4b26192700 1 --2- 172.21.15.90:0/1471156337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b20103380 0x7f4b20103790 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:51.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.527+0000 7f4b26192700 1 --2- 172.21.15.90:0/1471156337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b20107cb0 0x7f4b201080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:51.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.527+0000 7f4b26192700 1 -- 172.21.15.90:0/1471156337 >> 172.21.15.90:0/1471156337 conn(0x7f4b200fecb0 msgr2=0x7f4b201010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:51.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.527+0000 7f4b26192700 1 -- 172.21.15.90:0/1471156337 shutdown_connections 2024-04-25T11:33:51.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.527+0000 7f4b26192700 1 -- 172.21.15.90:0/1471156337 wait complete. 2024-04-25T11:33:51.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.527+0000 7f4b26192700 1 Processor -- start 2024-04-25T11:33:51.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.527+0000 7f4b26192700 1 -- start start 2024-04-25T11:33:51.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.528+0000 7f4b26192700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4b20103380 0x7f4b2019ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:51.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.528+0000 7f4b26192700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b20107cb0 0x7f4b2019d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:51.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.528+0000 7f4b26192700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b20108600 0x7f4b201a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:51.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.528+0000 7f4b26192700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4b20113230 con 0x7f4b20107cb0 2024-04-25T11:33:51.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.528+0000 7f4b26192700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4b201130b0 con 0x7f4b20103380 2024-04-25T11:33:51.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.528+0000 7f4b26192700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4b201133b0 con 0x7f4b20108600 2024-04-25T11:33:51.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.528+0000 7f4b17fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b20107cb0 0x7f4b2019d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:51.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.528+0000 7f4b17fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b20107cb0 0x7f4b2019d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35590/0 (socket says 172.21.15.90:35590) 2024-04-25T11:33:51.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.528+0000 7f4b17fff700 1 -- 172.21.15.90:0/3187978030 learned_addr learned my addr 172.21.15.90:0/3187978030 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:51.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.528+0000 7f4b1f7fe700 1 --2- 172.21.15.90:0/3187978030 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4b20103380 0x7f4b2019ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:51.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.528+0000 7f4b1ffff700 1 --2- 172.21.15.90:0/3187978030 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b20108600 0x7f4b201a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:51.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.528+0000 7f4b17fff700 1 -- 172.21.15.90:0/3187978030 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b20108600 msgr2=0x7f4b201a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:51.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.528+0000 7f4b17fff700 1 --2- 172.21.15.90:0/3187978030 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b20108600 0x7f4b201a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:51.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.528+0000 7f4b17fff700 1 -- 172.21.15.90:0/3187978030 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4b20103380 msgr2=0x7f4b2019ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:51.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.528+0000 7f4b17fff700 1 --2- 172.21.15.90:0/3187978030 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4b20103380 0x7f4b2019ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:51.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.528+0000 7f4b17fff700 1 -- 172.21.15.90:0/3187978030 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b201a1c90 con 0x7f4b20107cb0 2024-04-25T11:33:51.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.528+0000 7f4b17fff700 1 --2- 172.21.15.90:0/3187978030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b20107cb0 0x7f4b2019d410 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4b0800c8f0 tx=0x7f4b0800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:51.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.528+0000 7f4b1d7fa700 1 -- 172.21.15.90:0/3187978030 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b08007940 con 0x7f4b20107cb0 2024-04-25T11:33:51.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.528+0000 7f4b26192700 1 -- 172.21.15.90:0/3187978030 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b201a1f80 con 0x7f4b20107cb0 2024-04-25T11:33:51.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.528+0000 7f4b1d7fa700 1 -- 172.21.15.90:0/3187978030 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b0800f450 con 0x7f4b20107cb0 2024-04-25T11:33:51.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.528+0000 7f4b1d7fa700 1 -- 172.21.15.90:0/3187978030 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b08018660 con 0x7f4b20107cb0 2024-04-25T11:33:51.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.529+0000 7f4b26192700 1 -- 172.21.15.90:0/3187978030 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b201a24c0 con 0x7f4b20107cb0 2024-04-25T11:33:51.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.529+0000 7f4b26192700 1 -- 172.21.15.90:0/3187978030 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4b20063e20 con 0x7f4b20107cb0 2024-04-25T11:33:51.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.531+0000 7f4b1d7fa700 1 -- 172.21.15.90:0/3187978030 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f4b0800f890 con 0x7f4b20107cb0 2024-04-25T11:33:51.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.532+0000 7f4b1d7fa700 1 --2- 172.21.15.90:0/3187978030 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f4b0c0afc10 0x7f4b0c0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:51.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.532+0000 7f4b1f7fe700 1 --2- 172.21.15.90:0/3187978030 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f4b0c0afc10 0x7f4b0c0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:51.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.532+0000 7f4b1d7fa700 1 -- 172.21.15.90:0/3187978030 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7f4b080ded80 con 0x7f4b20107cb0 2024-04-25T11:33:51.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.532+0000 7f4b1d7fa700 1 --2- 172.21.15.90:0/3187978030 >> [v2:172.21.15.90:6802/1201629159,v1:172.21.15.90:6803/1201629159] conn(0x7f4b0c0b9440 0x7f4b0c0bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:51.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.532+0000 7f4b1d7fa700 1 -- 172.21.15.90:0/3187978030 --> [v2:172.21.15.90:6802/1201629159,v1:172.21.15.90:6803/1201629159] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4b08015e70 con 0x7f4b0c0b9440 2024-04-25T11:33:51.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.532+0000 7f4b1d7fa700 1 -- 172.21.15.90:0/3187978030 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=77) ==== 24+0+0 (secure 0 0 0) 0x7f4b080df260 con 0x7f4b20107cb0 2024-04-25T11:33:51.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.532+0000 7f4b1ffff700 1 --2- 172.21.15.90:0/3187978030 >> [v2:172.21.15.90:6802/1201629159,v1:172.21.15.90:6803/1201629159] conn(0x7f4b0c0b9440 0x7f4b0c0bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:51.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.532+0000 7f4b1f7fe700 1 --2- 172.21.15.90:0/3187978030 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f4b0c0afc10 0x7f4b0c0b2120 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4b10009990 tx=0x7f4b10000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:51.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.532+0000 7f4b1ffff700 1 --2- 172.21.15.90:0/3187978030 >> [v2:172.21.15.90:6802/1201629159,v1:172.21.15.90:6803/1201629159] conn(0x7f4b0c0b9440 0x7f4b0c0bb890 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:51.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.533+0000 7f4b1d7fa700 1 -- 172.21.15.90:0/3187978030 <== osd.0 v2:172.21.15.90:6802/1201629159 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4b08015e70 con 0x7f4b0c0b9440 2024-04-25T11:33:51.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.549+0000 7f4b26192700 1 -- 172.21.15.90:0/3187978030 --> [v2:172.21.15.90:6802/1201629159,v1:172.21.15.90:6803/1201629159] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4b200642a0 con 0x7f4b0c0b9440 2024-04-25T11:33:51.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.550+0000 7f4b1d7fa700 1 -- 172.21.15.90:0/3187978030 <== osd.0 v2:172.21.15.90:6802/1201629159 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4b200642a0 con 0x7f4b0c0b9440 2024-04-25T11:33:51.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.550+0000 7f4b26192700 1 -- 172.21.15.90:0/3187978030 >> [v2:172.21.15.90:6802/1201629159,v1:172.21.15.90:6803/1201629159] conn(0x7f4b0c0b9440 msgr2=0x7f4b0c0bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:51.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.550+0000 7f4b26192700 1 --2- 172.21.15.90:0/3187978030 >> [v2:172.21.15.90:6802/1201629159,v1:172.21.15.90:6803/1201629159] conn(0x7f4b0c0b9440 0x7f4b0c0bb890 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:51.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.550+0000 7f4b26192700 1 -- 172.21.15.90:0/3187978030 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f4b0c0afc10 msgr2=0x7f4b0c0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:51.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.550+0000 7f4b26192700 1 --2- 172.21.15.90:0/3187978030 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f4b0c0afc10 0x7f4b0c0b2120 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4b10009990 tx=0x7f4b10000bc0 comp rx=0 tx=0).stop 2024-04-25T11:33:51.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.550+0000 7f4b26192700 1 -- 172.21.15.90:0/3187978030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b20107cb0 msgr2=0x7f4b2019d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:51.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.550+0000 7f4b26192700 1 --2- 172.21.15.90:0/3187978030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b20107cb0 0x7f4b2019d410 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4b0800c8f0 tx=0x7f4b0800cc00 comp rx=0 tx=0).stop 2024-04-25T11:33:51.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.550+0000 7f4b1ffff700 1 -- 172.21.15.90:0/3187978030 reap_dead start 2024-04-25T11:33:51.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.550+0000 7f4b26192700 1 -- 172.21.15.90:0/3187978030 shutdown_connections 2024-04-25T11:33:51.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.550+0000 7f4b26192700 1 -- 172.21.15.90:0/3187978030 >> 172.21.15.90:0/3187978030 conn(0x7f4b200fecb0 msgr2=0x7f4b201063b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:51.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.550+0000 7f4b26192700 1 -- 172.21.15.90:0/3187978030 shutdown_connections 2024-04-25T11:33:51.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:51.550+0000 7f4b26192700 1 -- 172.21.15.90:0/3187978030 wait complete. 2024-04-25T11:33:51.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:51 smithi090 ceph-mon[29575]: pgmap v294: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-04-25T11:33:52.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:51 smithi156 ceph-mon[33045]: pgmap v294: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-04-25T11:33:52.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:51 smithi138 ceph-mon[32328]: pgmap v294: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-04-25T11:33:52.390 INFO:teuthology.orchestra.run.smithi090.stdout:231928234019 2024-04-25T11:33:52.391 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-25T11:33:52.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.360+0000 7f3d41324700 1 -- 172.21.15.90:0/418085126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d3c102630 msgr2=0x7f3d3c102a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:52.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.360+0000 7f3d41324700 1 --2- 172.21.15.90:0/418085126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d3c102630 0x7f3d3c102a40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c009990 tx=0x7f3d2c009ca0 comp rx=0 tx=0).stop 2024-04-25T11:33:52.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.360+0000 7f3d41324700 1 -- 172.21.15.90:0/418085126 shutdown_connections 2024-04-25T11:33:52.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.360+0000 7f3d41324700 1 --2- 172.21.15.90:0/418085126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d3c102f80 0x7f3d3c111010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:52.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.360+0000 7f3d41324700 1 --2- 172.21.15.90:0/418085126 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3d3c1091a0 0x7f3d3c1095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:52.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.360+0000 7f3d41324700 1 --2- 172.21.15.90:0/418085126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d3c102630 0x7f3d3c102a40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:52.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.360+0000 7f3d41324700 1 -- 172.21.15.90:0/418085126 >> 172.21.15.90:0/418085126 conn(0x7f3d3c0fe3f0 msgr2=0x7f3d3c100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:52.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.360+0000 7f3d41324700 1 -- 172.21.15.90:0/418085126 shutdown_connections 2024-04-25T11:33:52.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.361+0000 7f3d41324700 1 -- 172.21.15.90:0/418085126 wait complete. 2024-04-25T11:33:52.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.361+0000 7f3d41324700 1 Processor -- start 2024-04-25T11:33:52.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.361+0000 7f3d41324700 1 -- start start 2024-04-25T11:33:52.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.362+0000 7f3d41324700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d3c102630 0x7f3d3c19adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:52.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.362+0000 7f3d41324700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3d3c102f80 0x7f3d3c19b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:52.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.362+0000 7f3d41324700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d3c1091a0 0x7f3d3c19f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:52.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.362+0000 7f3d41324700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3c114440 con 0x7f3d3c102630 2024-04-25T11:33:52.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.362+0000 7f3d41324700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3c1142c0 con 0x7f3d3c102f80 2024-04-25T11:33:52.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.362+0000 7f3d41324700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3c1145c0 con 0x7f3d3c1091a0 2024-04-25T11:33:52.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.362+0000 7f3d3affd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d3c102630 0x7f3d3c19adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:52.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.362+0000 7f3d3affd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d3c102630 0x7f3d3c19adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50892/0 (socket says 172.21.15.90:50892) 2024-04-25T11:33:52.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.362+0000 7f3d3affd700 1 -- 172.21.15.90:0/1198627573 learned_addr learned my addr 172.21.15.90:0/1198627573 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:52.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.362+0000 7f3d3b7fe700 1 --2- 172.21.15.90:0/1198627573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d3c1091a0 0x7f3d3c19f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:52.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.362+0000 7f3d3a7fc700 1 --2- 172.21.15.90:0/1198627573 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3d3c102f80 0x7f3d3c19b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:52.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.362+0000 7f3d3affd700 1 -- 172.21.15.90:0/1198627573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d3c1091a0 msgr2=0x7f3d3c19f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:52.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.362+0000 7f3d3affd700 1 --2- 172.21.15.90:0/1198627573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d3c1091a0 0x7f3d3c19f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:52.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.362+0000 7f3d3affd700 1 -- 172.21.15.90:0/1198627573 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3d3c102f80 msgr2=0x7f3d3c19b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:52.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.362+0000 7f3d3affd700 1 --2- 172.21.15.90:0/1198627573 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3d3c102f80 0x7f3d3c19b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:52.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.362+0000 7f3d3affd700 1 -- 172.21.15.90:0/1198627573 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d3c19fc80 con 0x7f3d3c102630 2024-04-25T11:33:52.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.362+0000 7f3d3a7fc700 1 --2- 172.21.15.90:0/1198627573 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3d3c102f80 0x7f3d3c19b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:33:52.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.362+0000 7f3d3affd700 1 --2- 172.21.15.90:0/1198627573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d3c102630 0x7f3d3c19adb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3d3000c970 tx=0x7f3d3000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:52.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.362+0000 7f3d23fff700 1 -- 172.21.15.90:0/1198627573 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d3000ce60 con 0x7f3d3c102630 2024-04-25T11:33:52.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.362+0000 7f3d23fff700 1 -- 172.21.15.90:0/1198627573 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d30007900 con 0x7f3d3c102630 2024-04-25T11:33:52.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.362+0000 7f3d23fff700 1 -- 172.21.15.90:0/1198627573 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d30003c70 con 0x7f3d3c102630 2024-04-25T11:33:52.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.362+0000 7f3d41324700 1 -- 172.21.15.90:0/1198627573 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d3c19ff70 con 0x7f3d3c102630 2024-04-25T11:33:52.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.363+0000 7f3d41324700 1 -- 172.21.15.90:0/1198627573 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d3c1a7890 con 0x7f3d3c102630 2024-04-25T11:33:52.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.363+0000 7f3d41324700 1 -- 172.21.15.90:0/1198627573 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3d3c063da0 con 0x7f3d3c102630 2024-04-25T11:33:52.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.365+0000 7f3d23fff700 1 -- 172.21.15.90:0/1198627573 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f3d30004220 con 0x7f3d3c102630 2024-04-25T11:33:52.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.365+0000 7f3d23fff700 1 --2- 172.21.15.90:0/1198627573 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f3d240afc10 0x7f3d240b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:52.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.366+0000 7f3d3a7fc700 1 --2- 172.21.15.90:0/1198627573 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f3d240afc10 0x7f3d240b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:52.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.366+0000 7f3d23fff700 1 -- 172.21.15.90:0/1198627573 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7f3d300decf0 con 0x7f3d3c102630 2024-04-25T11:33:52.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.366+0000 7f3d23fff700 1 --2- 172.21.15.90:0/1198627573 >> [v2:172.21.15.90:6810/2520012383,v1:172.21.15.90:6811/2520012383] conn(0x7f3d240b9440 0x7f3d240bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:52.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.366+0000 7f3d23fff700 1 -- 172.21.15.90:0/1198627573 --> [v2:172.21.15.90:6810/2520012383,v1:172.21.15.90:6811/2520012383] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d30012ec0 con 0x7f3d240b9440 2024-04-25T11:33:52.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.366+0000 7f3d23fff700 1 -- 172.21.15.90:0/1198627573 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=77) ==== 24+0+0 (secure 0 0 0) 0x7f3d300df1d0 con 0x7f3d3c102630 2024-04-25T11:33:52.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.366+0000 7f3d3b7fe700 1 --2- 172.21.15.90:0/1198627573 >> [v2:172.21.15.90:6810/2520012383,v1:172.21.15.90:6811/2520012383] conn(0x7f3d240b9440 0x7f3d240bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:52.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.366+0000 7f3d3a7fc700 1 --2- 172.21.15.90:0/1198627573 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f3d240afc10 0x7f3d240b2120 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c003800 tx=0x7f3d2c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:52.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.366+0000 7f3d3b7fe700 1 --2- 172.21.15.90:0/1198627573 >> [v2:172.21.15.90:6810/2520012383,v1:172.21.15.90:6811/2520012383] conn(0x7f3d240b9440 0x7f3d240bb890 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:52.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.367+0000 7f3d23fff700 1 -- 172.21.15.90:0/1198627573 <== osd.3 v2:172.21.15.90:6810/2520012383 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3d30012ec0 con 0x7f3d240b9440 2024-04-25T11:33:52.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.383+0000 7f3d41324700 1 -- 172.21.15.90:0/1198627573 --> [v2:172.21.15.90:6810/2520012383,v1:172.21.15.90:6811/2520012383] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3d3c0641a0 con 0x7f3d240b9440 2024-04-25T11:33:52.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.384+0000 7f3d23fff700 1 -- 172.21.15.90:0/1198627573 <== osd.3 v2:172.21.15.90:6810/2520012383 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3d3c0641a0 con 0x7f3d240b9440 2024-04-25T11:33:52.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.384+0000 7f3d41324700 1 -- 172.21.15.90:0/1198627573 >> [v2:172.21.15.90:6810/2520012383,v1:172.21.15.90:6811/2520012383] conn(0x7f3d240b9440 msgr2=0x7f3d240bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:52.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.384+0000 7f3d41324700 1 --2- 172.21.15.90:0/1198627573 >> [v2:172.21.15.90:6810/2520012383,v1:172.21.15.90:6811/2520012383] conn(0x7f3d240b9440 0x7f3d240bb890 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:52.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.384+0000 7f3d41324700 1 -- 172.21.15.90:0/1198627573 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f3d240afc10 msgr2=0x7f3d240b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:52.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.384+0000 7f3d41324700 1 --2- 172.21.15.90:0/1198627573 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f3d240afc10 0x7f3d240b2120 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c003800 tx=0x7f3d2c005fb0 comp rx=0 tx=0).stop 2024-04-25T11:33:52.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.384+0000 7f3d41324700 1 -- 172.21.15.90:0/1198627573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d3c102630 msgr2=0x7f3d3c19adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:52.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.384+0000 7f3d41324700 1 --2- 172.21.15.90:0/1198627573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d3c102630 0x7f3d3c19adb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3d3000c970 tx=0x7f3d3000cc80 comp rx=0 tx=0).stop 2024-04-25T11:33:52.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.384+0000 7f3d3b7fe700 1 -- 172.21.15.90:0/1198627573 reap_dead start 2024-04-25T11:33:52.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.384+0000 7f3d41324700 1 -- 172.21.15.90:0/1198627573 shutdown_connections 2024-04-25T11:33:52.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.384+0000 7f3d41324700 1 -- 172.21.15.90:0/1198627573 >> 172.21.15.90:0/1198627573 conn(0x7f3d3c0fe3f0 msgr2=0x7f3d3c100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:52.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.384+0000 7f3d41324700 1 -- 172.21.15.90:0/1198627573 shutdown_connections 2024-04-25T11:33:52.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:48.384+0000 7f3d41324700 1 -- 172.21.15.90:0/1198627573 wait complete. 2024-04-25T11:33:53.110 INFO:teuthology.orchestra.run.smithi090.stdout:107374182459 2024-04-25T11:33:53.111 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-25T11:33:53.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.320+0000 7f19e69db700 1 -- 172.21.15.90:0/779806185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19e01096f0 msgr2=0x7f19e0111050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:53.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.320+0000 7f19e69db700 1 --2- 172.21.15.90:0/779806185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19e01096f0 0x7f19e0111050 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f19d800b0d0 tx=0x7f19d800b3e0 comp rx=0 tx=0).stop 2024-04-25T11:33:53.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.320+0000 7f19e69db700 1 -- 172.21.15.90:0/779806185 shutdown_connections 2024-04-25T11:33:53.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.320+0000 7f19e69db700 1 --2- 172.21.15.90:0/779806185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19e01096f0 0x7f19e0111050 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:53.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.320+0000 7f19e69db700 1 --2- 172.21.15.90:0/779806185 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f19e0104470 0x7f19e0104880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:53.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.320+0000 7f19e69db700 1 --2- 172.21.15.90:0/779806185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19e0108da0 0x7f19e01091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:53.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.320+0000 7f19e69db700 1 -- 172.21.15.90:0/779806185 >> 172.21.15.90:0/779806185 conn(0x7f19e00ffd00 msgr2=0x7f19e0102150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:53.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.320+0000 7f19e69db700 1 -- 172.21.15.90:0/779806185 shutdown_connections 2024-04-25T11:33:53.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.320+0000 7f19e69db700 1 -- 172.21.15.90:0/779806185 wait complete. 2024-04-25T11:33:53.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.321+0000 7f19e69db700 1 Processor -- start 2024-04-25T11:33:53.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.321+0000 7f19e69db700 1 -- start start 2024-04-25T11:33:53.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.321+0000 7f19e69db700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f19e0104470 0x7f19e01a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:53.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.321+0000 7f19e69db700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19e0108da0 0x7f19e01a0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:53.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.321+0000 7f19e69db700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19e01096f0 0x7f19e019ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:53.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.321+0000 7f19e69db700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f19e0114320 con 0x7f19e0108da0 2024-04-25T11:33:53.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.321+0000 7f19e69db700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f19e01141a0 con 0x7f19e0104470 2024-04-25T11:33:53.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.321+0000 7f19e69db700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f19e01144a0 con 0x7f19e01096f0 2024-04-25T11:33:53.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.321+0000 7f19df7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19e0108da0 0x7f19e01a0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:53.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.321+0000 7f19df7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19e0108da0 0x7f19e01a0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35604/0 (socket says 172.21.15.90:35604) 2024-04-25T11:33:53.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.322+0000 7f19df7fe700 1 -- 172.21.15.90:0/787279792 learned_addr learned my addr 172.21.15.90:0/787279792 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:53.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.322+0000 7f19e4f78700 1 --2- 172.21.15.90:0/787279792 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19e01096f0 0x7f19e019ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:53.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.322+0000 7f19dffff700 1 --2- 172.21.15.90:0/787279792 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f19e0104470 0x7f19e01a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:53.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.322+0000 7f19df7fe700 1 -- 172.21.15.90:0/787279792 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19e01096f0 msgr2=0x7f19e019ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:53.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.322+0000 7f19df7fe700 1 --2- 172.21.15.90:0/787279792 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19e01096f0 0x7f19e019ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:53.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.322+0000 7f19df7fe700 1 -- 172.21.15.90:0/787279792 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f19e0104470 msgr2=0x7f19e01a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:53.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.322+0000 7f19df7fe700 1 --2- 172.21.15.90:0/787279792 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f19e0104470 0x7f19e01a09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:53.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.322+0000 7f19df7fe700 1 -- 172.21.15.90:0/787279792 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19e019b270 con 0x7f19e0108da0 2024-04-25T11:33:53.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.322+0000 7f19df7fe700 1 --2- 172.21.15.90:0/787279792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19e0108da0 0x7f19e01a0ef0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f19d400c8f0 tx=0x7f19d400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:53.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.322+0000 7f19dd7fa700 1 -- 172.21.15.90:0/787279792 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d40078d0 con 0x7f19e0108da0 2024-04-25T11:33:53.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.322+0000 7f19e69db700 1 -- 172.21.15.90:0/787279792 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19e019b560 con 0x7f19e0108da0 2024-04-25T11:33:53.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.322+0000 7f19dd7fa700 1 -- 172.21.15.90:0/787279792 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19d400f450 con 0x7f19e0108da0 2024-04-25T11:33:53.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.322+0000 7f19dd7fa700 1 -- 172.21.15.90:0/787279792 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d4018660 con 0x7f19e0108da0 2024-04-25T11:33:53.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.322+0000 7f19e69db700 1 -- 172.21.15.90:0/787279792 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19e01a7760 con 0x7f19e0108da0 2024-04-25T11:33:53.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.323+0000 7f19e69db700 1 -- 172.21.15.90:0/787279792 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f19e0063da0 con 0x7f19e0108da0 2024-04-25T11:33:53.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.325+0000 7f19dd7fa700 1 -- 172.21.15.90:0/787279792 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f19d400f900 con 0x7f19e0108da0 2024-04-25T11:33:53.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.325+0000 7f19dd7fa700 1 --2- 172.21.15.90:0/787279792 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f19c80afc10 0x7f19c80b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:53.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.325+0000 7f19dffff700 1 --2- 172.21.15.90:0/787279792 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f19c80afc10 0x7f19c80b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:53.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.325+0000 7f19dd7fa700 1 -- 172.21.15.90:0/787279792 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7f19d40dedd0 con 0x7f19e0108da0 2024-04-25T11:33:53.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.326+0000 7f19dd7fa700 1 --2- 172.21.15.90:0/787279792 >> [v2:172.21.15.156:6824/1224251744,v1:172.21.15.156:6825/1224251744] conn(0x7f19c80b9440 0x7f19c80bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:53.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.326+0000 7f19dd7fa700 1 -- 172.21.15.90:0/787279792 --> [v2:172.21.15.156:6824/1224251744,v1:172.21.15.156:6825/1224251744] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f19d4015e70 con 0x7f19c80b9440 2024-04-25T11:33:53.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.326+0000 7f19dd7fa700 1 -- 172.21.15.90:0/787279792 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=77) ==== 24+0+0 (secure 0 0 0) 0x7f19d40df2b0 con 0x7f19e0108da0 2024-04-25T11:33:53.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.326+0000 7f19dffff700 1 --2- 172.21.15.90:0/787279792 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f19c80afc10 0x7f19c80b2120 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f19d0007be0 tx=0x7f19d00073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:53.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.326+0000 7f19e4f78700 1 --2- 172.21.15.90:0/787279792 >> [v2:172.21.15.156:6824/1224251744,v1:172.21.15.156:6825/1224251744] conn(0x7f19c80b9440 0x7f19c80bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:53.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.326+0000 7f19e4f78700 1 --2- 172.21.15.90:0/787279792 >> [v2:172.21.15.156:6824/1224251744,v1:172.21.15.156:6825/1224251744] conn(0x7f19c80b9440 0x7f19c80bb890 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:53.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.327+0000 7f19dd7fa700 1 -- 172.21.15.90:0/787279792 <== osd.11 v2:172.21.15.156:6824/1224251744 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f19d4015e70 con 0x7f19c80b9440 2024-04-25T11:33:53.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.343+0000 7f19e69db700 1 -- 172.21.15.90:0/787279792 --> [v2:172.21.15.156:6824/1224251744,v1:172.21.15.156:6825/1224251744] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f19e00641a0 con 0x7f19c80b9440 2024-04-25T11:33:53.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.344+0000 7f19dd7fa700 1 -- 172.21.15.90:0/787279792 <== osd.11 v2:172.21.15.156:6824/1224251744 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f19e00641a0 con 0x7f19c80b9440 2024-04-25T11:33:53.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.344+0000 7f19e69db700 1 -- 172.21.15.90:0/787279792 >> [v2:172.21.15.156:6824/1224251744,v1:172.21.15.156:6825/1224251744] conn(0x7f19c80b9440 msgr2=0x7f19c80bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:53.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.344+0000 7f19e69db700 1 --2- 172.21.15.90:0/787279792 >> [v2:172.21.15.156:6824/1224251744,v1:172.21.15.156:6825/1224251744] conn(0x7f19c80b9440 0x7f19c80bb890 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:53.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.344+0000 7f19e69db700 1 -- 172.21.15.90:0/787279792 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f19c80afc10 msgr2=0x7f19c80b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:53.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.344+0000 7f19e69db700 1 --2- 172.21.15.90:0/787279792 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f19c80afc10 0x7f19c80b2120 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f19d0007be0 tx=0x7f19d00073d0 comp rx=0 tx=0).stop 2024-04-25T11:33:53.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.344+0000 7f19e69db700 1 -- 172.21.15.90:0/787279792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19e0108da0 msgr2=0x7f19e01a0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:53.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.344+0000 7f19e69db700 1 --2- 172.21.15.90:0/787279792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19e0108da0 0x7f19e01a0ef0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f19d400c8f0 tx=0x7f19d400cc00 comp rx=0 tx=0).stop 2024-04-25T11:33:53.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.344+0000 7f19e4f78700 1 -- 172.21.15.90:0/787279792 reap_dead start 2024-04-25T11:33:53.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.345+0000 7f19e69db700 1 -- 172.21.15.90:0/787279792 shutdown_connections 2024-04-25T11:33:53.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.345+0000 7f19e69db700 1 -- 172.21.15.90:0/787279792 >> 172.21.15.90:0/787279792 conn(0x7f19e00ffd00 msgr2=0x7f19e0108510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:53.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.345+0000 7f19e69db700 1 -- 172.21.15.90:0/787279792 shutdown_connections 2024-04-25T11:33:53.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:53.345+0000 7f19e69db700 1 -- 172.21.15.90:0/787279792 wait complete. 2024-04-25T11:33:53.874 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:53 smithi090 ceph-mon[29575]: pgmap v295: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-25T11:33:54.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:53 smithi156 ceph-mon[33045]: pgmap v295: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-25T11:33:54.184 INFO:teuthology.orchestra.run.smithi090.stdout:34359738443 2024-04-25T11:33:54.185 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-25T11:33:54.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:53 smithi138 ceph-mon[32328]: pgmap v295: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-25T11:33:54.700 INFO:teuthology.orchestra.run.smithi090.stdout:309237645333 2024-04-25T11:33:54.701 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-25T11:33:55.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.200+0000 7fd7baef6700 1 -- 172.21.15.90:0/4248396910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7b4102660 msgr2=0x7fd7b4102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:55.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.200+0000 7fd7baef6700 1 --2- 172.21.15.90:0/4248396910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7b4102660 0x7fd7b4102a70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0009990 tx=0x7fd7a0009ca0 comp rx=0 tx=0).stop 2024-04-25T11:33:55.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.200+0000 7fd7baef6700 1 -- 172.21.15.90:0/4248396910 shutdown_connections 2024-04-25T11:33:55.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.200+0000 7fd7baef6700 1 --2- 172.21.15.90:0/4248396910 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd7b4102fb0 0x7fd7b4107df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:55.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.200+0000 7fd7baef6700 1 --2- 172.21.15.90:0/4248396910 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7b4109120 0x7fd7b4109530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:55.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.200+0000 7fd7baef6700 1 --2- 172.21.15.90:0/4248396910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7b4102660 0x7fd7b4102a70 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:55.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.200+0000 7fd7baef6700 1 -- 172.21.15.90:0/4248396910 >> 172.21.15.90:0/4248396910 conn(0x7fd7b40fe3f0 msgr2=0x7fd7b4100840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:55.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.200+0000 7fd7baef6700 1 -- 172.21.15.90:0/4248396910 shutdown_connections 2024-04-25T11:33:55.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.200+0000 7fd7baef6700 1 -- 172.21.15.90:0/4248396910 wait complete. 2024-04-25T11:33:55.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.200+0000 7fd7baef6700 1 Processor -- start 2024-04-25T11:33:55.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.200+0000 7fd7baef6700 1 -- start start 2024-04-25T11:33:55.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.201+0000 7fd7baef6700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd7b4102660 0x7fd7b41a1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:55.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.201+0000 7fd7baef6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7b4102fb0 0x7fd7b41a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:55.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.201+0000 7fd7baef6700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7b4109120 0x7fd7b41a5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:55.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.201+0000 7fd7baef6700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b411aa50 con 0x7fd7b4109120 2024-04-25T11:33:55.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.201+0000 7fd7baef6700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b411a8d0 con 0x7fd7b4102660 2024-04-25T11:33:55.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.201+0000 7fd7baef6700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b411abd0 con 0x7fd7b4102fb0 2024-04-25T11:33:55.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.201+0000 7fd7b9493700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7b4109120 0x7fd7b41a5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:55.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.201+0000 7fd7b9493700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7b4109120 0x7fd7b41a5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35624/0 (socket says 172.21.15.90:35624) 2024-04-25T11:33:55.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.201+0000 7fd7b9493700 1 -- 172.21.15.90:0/2084159276 learned_addr learned my addr 172.21.15.90:0/2084159276 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:33:55.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.201+0000 7fd7b3fff700 1 --2- 172.21.15.90:0/2084159276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7b4102fb0 0x7fd7b41a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:55.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.201+0000 7fd7b8c92700 1 --2- 172.21.15.90:0/2084159276 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd7b4102660 0x7fd7b41a1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:55.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.201+0000 7fd7b9493700 1 -- 172.21.15.90:0/2084159276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7b4102fb0 msgr2=0x7fd7b41a19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:55.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.201+0000 7fd7b9493700 1 --2- 172.21.15.90:0/2084159276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7b4102fb0 0x7fd7b41a19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:55.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.201+0000 7fd7b9493700 1 -- 172.21.15.90:0/2084159276 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd7b4102660 msgr2=0x7fd7b41a1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:55.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.201+0000 7fd7b9493700 1 --2- 172.21.15.90:0/2084159276 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd7b4102660 0x7fd7b41a1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:55.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.201+0000 7fd7b9493700 1 -- 172.21.15.90:0/2084159276 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b41a6310 con 0x7fd7b4109120 2024-04-25T11:33:55.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.201+0000 7fd7b3fff700 1 --2- 172.21.15.90:0/2084159276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7b4102fb0 0x7fd7b41a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:33:55.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.201+0000 7fd7b8c92700 1 --2- 172.21.15.90:0/2084159276 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd7b4102660 0x7fd7b41a1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:33:55.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.201+0000 7fd7b9493700 1 --2- 172.21.15.90:0/2084159276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7b4109120 0x7fd7b41a5c70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac00b970 tx=0x7fd7ac00bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:55.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.201+0000 7fd7b1ffb700 1 -- 172.21.15.90:0/2084159276 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac019070 con 0x7fd7b4109120 2024-04-25T11:33:55.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.202+0000 7fd7b1ffb700 1 -- 172.21.15.90:0/2084159276 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7ac014440 con 0x7fd7b4109120 2024-04-25T11:33:55.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.202+0000 7fd7baef6700 1 -- 172.21.15.90:0/2084159276 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7b41a6600 con 0x7fd7b4109120 2024-04-25T11:33:55.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.202+0000 7fd7b1ffb700 1 -- 172.21.15.90:0/2084159276 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac00ce60 con 0x7fd7b4109120 2024-04-25T11:33:55.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.202+0000 7fd7baef6700 1 -- 172.21.15.90:0/2084159276 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7b41adf20 con 0x7fd7b4109120 2024-04-25T11:33:55.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.202+0000 7fd7baef6700 1 -- 172.21.15.90:0/2084159276 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd7b4107830 con 0x7fd7b4109120 2024-04-25T11:33:55.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.204+0000 7fd7b1ffb700 1 -- 172.21.15.90:0/2084159276 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fd7ac01c070 con 0x7fd7b4109120 2024-04-25T11:33:55.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.205+0000 7fd7b1ffb700 1 --2- 172.21.15.90:0/2084159276 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd7a40afc10 0x7fd7a40b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:55.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.205+0000 7fd7b8c92700 1 --2- 172.21.15.90:0/2084159276 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd7a40afc10 0x7fd7a40b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:55.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.205+0000 7fd7b1ffb700 1 -- 172.21.15.90:0/2084159276 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7fd7ac0de7b0 con 0x7fd7b4109120 2024-04-25T11:33:55.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.205+0000 7fd7b1ffb700 1 --2- 172.21.15.90:0/2084159276 >> [v2:172.21.15.90:6826/1171265966,v1:172.21.15.90:6827/1171265966] conn(0x7fd7a40b94a0 0x7fd7a40bb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:55.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.205+0000 7fd7b1ffb700 1 -- 172.21.15.90:0/2084159276 --> [v2:172.21.15.90:6826/1171265966,v1:172.21.15.90:6827/1171265966] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd7ac01ce50 con 0x7fd7a40b94a0 2024-04-25T11:33:55.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.205+0000 7fd7b1ffb700 1 -- 172.21.15.90:0/2084159276 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=77) ==== 24+0+0 (secure 0 0 0) 0x7fd7ac0dec90 con 0x7fd7b4109120 2024-04-25T11:33:55.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.205+0000 7fd7b3fff700 1 --2- 172.21.15.90:0/2084159276 >> [v2:172.21.15.90:6826/1171265966,v1:172.21.15.90:6827/1171265966] conn(0x7fd7a40b94a0 0x7fd7a40bb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:55.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.205+0000 7fd7b8c92700 1 --2- 172.21.15.90:0/2084159276 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd7a40afc10 0x7fd7a40b2120 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd7b406bed0 tx=0x7fd7a8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:55.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.206+0000 7fd7b3fff700 1 --2- 172.21.15.90:0/2084159276 >> [v2:172.21.15.90:6826/1171265966,v1:172.21.15.90:6827/1171265966] conn(0x7fd7a40b94a0 0x7fd7a40bb8f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:55.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.207+0000 7fd7b1ffb700 1 -- 172.21.15.90:0/2084159276 <== osd.9 v2:172.21.15.90:6826/1171265966 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd7ac01ce50 con 0x7fd7a40b94a0 2024-04-25T11:33:55.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.224+0000 7fd7baef6700 1 -- 172.21.15.90:0/2084159276 --> [v2:172.21.15.90:6826/1171265966,v1:172.21.15.90:6827/1171265966] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd7b4103880 con 0x7fd7a40b94a0 2024-04-25T11:33:55.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.224+0000 7fd7b1ffb700 1 -- 172.21.15.90:0/2084159276 <== osd.9 v2:172.21.15.90:6826/1171265966 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd7b4103880 con 0x7fd7a40b94a0 2024-04-25T11:33:55.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.224+0000 7fd7baef6700 1 -- 172.21.15.90:0/2084159276 >> [v2:172.21.15.90:6826/1171265966,v1:172.21.15.90:6827/1171265966] conn(0x7fd7a40b94a0 msgr2=0x7fd7a40bb8f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:55.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.224+0000 7fd7baef6700 1 --2- 172.21.15.90:0/2084159276 >> [v2:172.21.15.90:6826/1171265966,v1:172.21.15.90:6827/1171265966] conn(0x7fd7a40b94a0 0x7fd7a40bb8f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:55.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.224+0000 7fd7baef6700 1 -- 172.21.15.90:0/2084159276 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd7a40afc10 msgr2=0x7fd7a40b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:55.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.224+0000 7fd7baef6700 1 --2- 172.21.15.90:0/2084159276 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd7a40afc10 0x7fd7a40b2120 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd7b406bed0 tx=0x7fd7a8008040 comp rx=0 tx=0).stop 2024-04-25T11:33:55.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.224+0000 7fd7baef6700 1 -- 172.21.15.90:0/2084159276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7b4109120 msgr2=0x7fd7b41a5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:55.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.224+0000 7fd7baef6700 1 --2- 172.21.15.90:0/2084159276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7b4109120 0x7fd7b41a5c70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac00b970 tx=0x7fd7ac00bd30 comp rx=0 tx=0).stop 2024-04-25T11:33:55.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.224+0000 7fd7b9493700 1 -- 172.21.15.90:0/2084159276 reap_dead start 2024-04-25T11:33:55.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.224+0000 7fd7baef6700 1 -- 172.21.15.90:0/2084159276 shutdown_connections 2024-04-25T11:33:55.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.225+0000 7fd7baef6700 1 -- 172.21.15.90:0/2084159276 >> 172.21.15.90:0/2084159276 conn(0x7fd7b40fe3f0 msgr2=0x7fd7b4105dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:55.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.225+0000 7fd7baef6700 1 -- 172.21.15.90:0/2084159276 shutdown_connections 2024-04-25T11:33:55.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:33:55.225+0000 7fd7baef6700 1 -- 172.21.15.90:0/2084159276 wait complete. 2024-04-25T11:33:56.009 INFO:teuthology.orchestra.run.smithi090.stdout:154618822706 2024-04-25T11:33:56.010 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-25T11:33:56.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:55 smithi156 ceph-mon[33045]: pgmap v296: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 11 op/s 2024-04-25T11:33:56.202 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:55 smithi090 ceph-mon[29575]: pgmap v296: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 11 op/s 2024-04-25T11:33:56.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:55 smithi138 ceph-mon[32328]: pgmap v296: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 11 op/s 2024-04-25T11:33:56.472 INFO:teuthology.orchestra.run.smithi090.stdout:257698037791 2024-04-25T11:33:56.472 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-25T11:33:56.795 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:56.795 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:56.800 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:56.800 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:56.805 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:56.806 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:56.808 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:56.809 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:56.811 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:56.814 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:56.821 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:56.883 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:33:58.156 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:57 smithi090 ceph-mon[29575]: pgmap v297: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-25T11:33:58.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:57 smithi138 ceph-mon[32328]: pgmap v297: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-25T11:33:58.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:57 smithi156 ceph-mon[33045]: pgmap v297: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-25T11:33:59.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:33:59 smithi090 ceph-mon[29575]: pgmap v298: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-04-25T11:34:00.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:33:59 smithi138 ceph-mon[32328]: pgmap v298: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-04-25T11:34:00.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:33:59 smithi156 ceph-mon[33045]: pgmap v298: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-04-25T11:34:02.024 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:01 smithi090 ceph-mon[29575]: pgmap v299: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-25T11:34:02.104 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:01 smithi138 ceph-mon[32328]: pgmap v299: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-25T11:34:02.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:01 smithi156 ceph-mon[33045]: pgmap v299: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-25T11:34:03.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:02 smithi090 ceph-mon[29575]: pgmap v300: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T11:34:03.293 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:02 smithi138 ceph-mon[32328]: pgmap v300: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T11:34:03.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:02 smithi156 ceph-mon[33045]: pgmap v300: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T11:34:05.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.186+0000 7f5af96fe700 1 -- 172.21.15.90:0/2154763739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5af41033b0 msgr2=0x7f5af41037c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:05.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.186+0000 7f5af96fe700 1 --2- 172.21.15.90:0/2154763739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5af41033b0 0x7f5af41037c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5ae400b0d0 tx=0x7f5ae400b3e0 comp rx=0 tx=0).stop 2024-04-25T11:34:05.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.186+0000 7f5af96fe700 1 -- 172.21.15.90:0/2154763739 shutdown_connections 2024-04-25T11:34:05.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.186+0000 7f5af96fe700 1 --2- 172.21.15.90:0/2154763739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5af4108630 0x7f5af410ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:05.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.186+0000 7f5af96fe700 1 --2- 172.21.15.90:0/2154763739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5af41033b0 0x7f5af41037c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:05.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.186+0000 7f5af96fe700 1 --2- 172.21.15.90:0/2154763739 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5af4107ce0 0x7f5af41080f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:05.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.186+0000 7f5af96fe700 1 -- 172.21.15.90:0/2154763739 >> 172.21.15.90:0/2154763739 conn(0x7f5af40fee20 msgr2=0x7f5af4101240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:05.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.186+0000 7f5af96fe700 1 -- 172.21.15.90:0/2154763739 shutdown_connections 2024-04-25T11:34:05.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.186+0000 7f5af96fe700 1 -- 172.21.15.90:0/2154763739 wait complete. 2024-04-25T11:34:05.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.187+0000 7f5af96fe700 1 Processor -- start 2024-04-25T11:34:05.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.187+0000 7f5af96fe700 1 -- start start 2024-04-25T11:34:05.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.187+0000 7f5af96fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5af41033b0 0x7f5af41a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:05.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.187+0000 7f5af96fe700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5af4107ce0 0x7f5af41a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:05.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.187+0000 7f5af96fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5af4108630 0x7f5af419aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:05.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.187+0000 7f5af96fe700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5af41143b0 con 0x7f5af4108630 2024-04-25T11:34:05.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.187+0000 7f5af96fe700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5af4114230 con 0x7f5af4107ce0 2024-04-25T11:34:05.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.187+0000 7f5af96fe700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5af4114530 con 0x7f5af41033b0 2024-04-25T11:34:05.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.187+0000 7f5af37fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5af4108630 0x7f5af419aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:05.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.187+0000 7f5af37fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5af4108630 0x7f5af419aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51792/0 (socket says 172.21.15.90:51792) 2024-04-25T11:34:05.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.187+0000 7f5af37fe700 1 -- 172.21.15.90:0/3710073403 learned_addr learned my addr 172.21.15.90:0/3710073403 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:05.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.187+0000 7f5af2ffd700 1 --2- 172.21.15.90:0/3710073403 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5af41033b0 0x7f5af41a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:05.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.187+0000 7f5af27fc700 1 --2- 172.21.15.90:0/3710073403 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5af4107ce0 0x7f5af41a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:05.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.188+0000 7f5af37fe700 1 -- 172.21.15.90:0/3710073403 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5af41033b0 msgr2=0x7f5af41a0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:05.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.188+0000 7f5af37fe700 1 --2- 172.21.15.90:0/3710073403 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5af41033b0 0x7f5af41a0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:05.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.188+0000 7f5af37fe700 1 -- 172.21.15.90:0/3710073403 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5af4107ce0 msgr2=0x7f5af41a0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:05.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.188+0000 7f5af37fe700 1 --2- 172.21.15.90:0/3710073403 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5af4107ce0 0x7f5af41a0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:05.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.188+0000 7f5af37fe700 1 -- 172.21.15.90:0/3710073403 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5af419b140 con 0x7f5af4108630 2024-04-25T11:34:05.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.188+0000 7f5af27fc700 1 --2- 172.21.15.90:0/3710073403 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5af4107ce0 0x7f5af41a0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:05.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.188+0000 7f5af37fe700 1 --2- 172.21.15.90:0/3710073403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5af4108630 0x7f5af419aad0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5aec00d980 tx=0x7f5aec00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:05.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.188+0000 7f5adbfff700 1 -- 172.21.15.90:0/3710073403 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aec017070 con 0x7f5af4108630 2024-04-25T11:34:05.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.188+0000 7f5af96fe700 1 -- 172.21.15.90:0/3710073403 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5af419b430 con 0x7f5af4108630 2024-04-25T11:34:05.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.188+0000 7f5adbfff700 1 -- 172.21.15.90:0/3710073403 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5aec005020 con 0x7f5af4108630 2024-04-25T11:34:05.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.188+0000 7f5adbfff700 1 -- 172.21.15.90:0/3710073403 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aec0099b0 con 0x7f5af4108630 2024-04-25T11:34:05.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.188+0000 7f5af96fe700 1 -- 172.21.15.90:0/3710073403 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5af41a7740 con 0x7f5af4108630 2024-04-25T11:34:05.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.189+0000 7f5af96fe700 1 -- 172.21.15.90:0/3710073403 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5af4051190 con 0x7f5af4108630 2024-04-25T11:34:05.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.190+0000 7f5adbfff700 1 -- 172.21.15.90:0/3710073403 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f5aec004a30 con 0x7f5af4108630 2024-04-25T11:34:05.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.191+0000 7f5adbfff700 1 --2- 172.21.15.90:0/3710073403 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f5adc0afc10 0x7f5adc0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:05.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.191+0000 7f5af2ffd700 1 --2- 172.21.15.90:0/3710073403 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f5adc0afc10 0x7f5adc0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:05.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.192+0000 7f5adbfff700 1 -- 172.21.15.90:0/3710073403 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7f5aec0def30 con 0x7f5af4108630 2024-04-25T11:34:05.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.192+0000 7f5af2ffd700 1 --2- 172.21.15.90:0/3710073403 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f5adc0afc10 0x7f5adc0b2120 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4009900 tx=0x7f5ae400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:05.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.195+0000 7f5adbfff700 1 -- 172.21.15.90:0/3710073403 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5aec020020 con 0x7f5af4108630 2024-04-25T11:34:05.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.395+0000 7f5af96fe700 1 -- 172.21.15.90:0/3710073403 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f5af419bfc0 con 0x7f5af4108630 2024-04-25T11:34:05.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.395+0000 7f5adbfff700 1 -- 172.21.15.90:0/3710073403 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f5aec0a7a60 con 0x7f5af4108630 2024-04-25T11:34:05.384 INFO:teuthology.orchestra.run.smithi090.stdout:133143986234 2024-04-25T11:34:05.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.397+0000 7f5af96fe700 1 -- 172.21.15.90:0/3710073403 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f5adc0afc10 msgr2=0x7f5adc0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:05.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.397+0000 7f5af96fe700 1 --2- 172.21.15.90:0/3710073403 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f5adc0afc10 0x7f5adc0b2120 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4009900 tx=0x7f5ae400b040 comp rx=0 tx=0).stop 2024-04-25T11:34:05.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.397+0000 7f5af96fe700 1 -- 172.21.15.90:0/3710073403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5af4108630 msgr2=0x7f5af419aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:05.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.397+0000 7f5af96fe700 1 --2- 172.21.15.90:0/3710073403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5af4108630 0x7f5af419aad0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5aec00d980 tx=0x7f5aec00dc90 comp rx=0 tx=0).stop 2024-04-25T11:34:05.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.397+0000 7f5af96fe700 1 -- 172.21.15.90:0/3710073403 shutdown_connections 2024-04-25T11:34:05.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.397+0000 7f5af96fe700 1 --2- 172.21.15.90:0/3710073403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5af4108630 0x7f5af419aad0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:05.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.397+0000 7f5af96fe700 1 --2- 172.21.15.90:0/3710073403 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f5adc0afc10 0x7f5adc0b2120 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:05.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.397+0000 7f5af96fe700 1 --2- 172.21.15.90:0/3710073403 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5af41033b0 0x7f5af41a0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:05.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.397+0000 7f5af96fe700 1 --2- 172.21.15.90:0/3710073403 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5af4107ce0 0x7f5af41a0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:05.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.397+0000 7f5af96fe700 1 -- 172.21.15.90:0/3710073403 >> 172.21.15.90:0/3710073403 conn(0x7f5af40fee20 msgr2=0x7f5af4107450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:05.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.397+0000 7f5af96fe700 1 -- 172.21.15.90:0/3710073403 shutdown_connections 2024-04-25T11:34:05.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.397+0000 7f5af96fe700 1 -- 172.21.15.90:0/3710073403 wait complete. 2024-04-25T11:34:05.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:05 smithi090 ceph-mon[29575]: pgmap v301: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-25T11:34:05.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:05 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3710073403' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T11:34:05.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:05 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/413876218' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T11:34:06.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:05 smithi156 ceph-mon[33045]: pgmap v301: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-25T11:34:06.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:05 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3710073403' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T11:34:06.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:05 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/413876218' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T11:34:06.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:05 smithi138 ceph-mon[32328]: pgmap v301: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-25T11:34:06.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:05 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3710073403' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T11:34:06.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:05 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/413876218' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T11:34:06.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.705+0000 7f31c9d30700 1 -- 172.21.15.90:0/2802463646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31c41031e0 msgr2=0x7f31c410f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:06.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.705+0000 7f31c9d30700 1 --2- 172.21.15.90:0/2802463646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31c41031e0 0x7f31c410f760 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f31bc00c210 tx=0x7f31bc00c520 comp rx=0 tx=0).stop 2024-04-25T11:34:06.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.706+0000 7f31c9d30700 1 -- 172.21.15.90:0/2802463646 shutdown_connections 2024-04-25T11:34:06.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.706+0000 7f31c9d30700 1 --2- 172.21.15.90:0/2802463646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31c41031e0 0x7f31c410f760 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:06.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.706+0000 7f31c9d30700 1 --2- 172.21.15.90:0/2802463646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31c41088c0 0x7f31c4108cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:06.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.706+0000 7f31c9d30700 1 --2- 172.21.15.90:0/2802463646 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31c4102890 0x7f31c4102ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:06.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.706+0000 7f31c9d30700 1 -- 172.21.15.90:0/2802463646 >> 172.21.15.90:0/2802463646 conn(0x7f31c40fe5c0 msgr2=0x7f31c41009e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:06.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.706+0000 7f31c9d30700 1 -- 172.21.15.90:0/2802463646 shutdown_connections 2024-04-25T11:34:06.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.706+0000 7f31c9d30700 1 -- 172.21.15.90:0/2802463646 wait complete. 2024-04-25T11:34:06.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.706+0000 7f31c9d30700 1 Processor -- start 2024-04-25T11:34:06.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.706+0000 7f31c9d30700 1 -- start start 2024-04-25T11:34:06.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.707+0000 7f31c9d30700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31c4102890 0x7f31c419e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:06.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.707+0000 7f31c9d30700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31c41031e0 0x7f31c419ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:06.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.707+0000 7f31c9d30700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31c41088c0 0x7f31c41988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:06.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.707+0000 7f31c9d30700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f31c41121b0 con 0x7f31c41088c0 2024-04-25T11:34:06.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.707+0000 7f31c9d30700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f31c4112030 con 0x7f31c4102890 2024-04-25T11:34:06.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.707+0000 7f31c9d30700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f31c4112330 con 0x7f31c41031e0 2024-04-25T11:34:06.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.707+0000 7f31c3fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31c41088c0 0x7f31c41988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:06.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.707+0000 7f31c3fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31c41088c0 0x7f31c41988b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51814/0 (socket says 172.21.15.90:51814) 2024-04-25T11:34:06.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.707+0000 7f31c3fff700 1 -- 172.21.15.90:0/4099291977 learned_addr learned my addr 172.21.15.90:0/4099291977 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:06.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.707+0000 7f31c2ffd700 1 --2- 172.21.15.90:0/4099291977 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31c41031e0 0x7f31c419ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:06.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.707+0000 7f31c3fff700 1 -- 172.21.15.90:0/4099291977 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31c41031e0 msgr2=0x7f31c419ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:06.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.707+0000 7f31c3fff700 1 --2- 172.21.15.90:0/4099291977 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31c41031e0 0x7f31c419ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:06.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.707+0000 7f31c3fff700 1 -- 172.21.15.90:0/4099291977 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31c4102890 msgr2=0x7f31c419e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:06.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.707+0000 7f31c3fff700 1 --2- 172.21.15.90:0/4099291977 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31c4102890 0x7f31c419e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:06.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.707+0000 7f31c3fff700 1 -- 172.21.15.90:0/4099291977 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31c4198ef0 con 0x7f31c41088c0 2024-04-25T11:34:06.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.707+0000 7f31c3fff700 1 --2- 172.21.15.90:0/4099291977 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31c41088c0 0x7f31c41988b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f31bc008ed0 tx=0x7f31bc00a8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:06.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.707+0000 7f31c0ff9700 1 -- 172.21.15.90:0/4099291977 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31bc003c00 con 0x7f31c41088c0 2024-04-25T11:34:06.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.707+0000 7f31c9d30700 1 -- 172.21.15.90:0/4099291977 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31c4199180 con 0x7f31c41088c0 2024-04-25T11:34:06.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.707+0000 7f31c0ff9700 1 -- 172.21.15.90:0/4099291977 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31bc00a5a0 con 0x7f31c41088c0 2024-04-25T11:34:06.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.707+0000 7f31c0ff9700 1 -- 172.21.15.90:0/4099291977 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31bc01ea40 con 0x7f31c41088c0 2024-04-25T11:34:06.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.707+0000 7f31c9d30700 1 -- 172.21.15.90:0/4099291977 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31c4199610 con 0x7f31c41088c0 2024-04-25T11:34:06.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.708+0000 7f31c9d30700 1 -- 172.21.15.90:0/4099291977 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31c4051190 con 0x7f31c41088c0 2024-04-25T11:34:06.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.710+0000 7f31c0ff9700 1 -- 172.21.15.90:0/4099291977 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f31bc01ebe0 con 0x7f31c41088c0 2024-04-25T11:34:06.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.711+0000 7f31c0ff9700 1 --2- 172.21.15.90:0/4099291977 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f31ac0afc10 0x7f31ac0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:06.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.711+0000 7f31c37fe700 1 --2- 172.21.15.90:0/4099291977 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f31ac0afc10 0x7f31ac0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:06.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.711+0000 7f31c0ff9700 1 -- 172.21.15.90:0/4099291977 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7f31bc0e1730 con 0x7f31c41088c0 2024-04-25T11:34:06.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.711+0000 7f31c37fe700 1 --2- 172.21.15.90:0/4099291977 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f31ac0afc10 0x7f31ac0b2120 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f31c41999f0 tx=0x7f31b4005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:06.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.714+0000 7f31c0ff9700 1 -- 172.21.15.90:0/4099291977 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f31bc0aa3f0 con 0x7f31c41088c0 2024-04-25T11:34:06.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.926+0000 7f31c9d30700 1 -- 172.21.15.90:0/4099291977 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f31c402e510 con 0x7f31c41088c0 2024-04-25T11:34:06.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.926+0000 7f31c0ff9700 1 -- 172.21.15.90:0/4099291977 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f31bc0a9cf0 con 0x7f31c41088c0 2024-04-25T11:34:06.914 INFO:teuthology.orchestra.run.smithi090.stdout:77309411397 2024-04-25T11:34:06.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.928+0000 7f31c9d30700 1 -- 172.21.15.90:0/4099291977 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f31ac0afc10 msgr2=0x7f31ac0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:06.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.928+0000 7f31c9d30700 1 --2- 172.21.15.90:0/4099291977 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f31ac0afc10 0x7f31ac0b2120 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f31c41999f0 tx=0x7f31b4005c90 comp rx=0 tx=0).stop 2024-04-25T11:34:06.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.928+0000 7f31c9d30700 1 -- 172.21.15.90:0/4099291977 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31c41088c0 msgr2=0x7f31c41988b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:06.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.928+0000 7f31c9d30700 1 --2- 172.21.15.90:0/4099291977 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31c41088c0 0x7f31c41988b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f31bc008ed0 tx=0x7f31bc00a8b0 comp rx=0 tx=0).stop 2024-04-25T11:34:06.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.928+0000 7f31c9d30700 1 -- 172.21.15.90:0/4099291977 shutdown_connections 2024-04-25T11:34:06.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.928+0000 7f31c9d30700 1 --2- 172.21.15.90:0/4099291977 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31c41088c0 0x7f31c41988b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:06.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.928+0000 7f31c9d30700 1 --2- 172.21.15.90:0/4099291977 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31c4102890 0x7f31c419e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:06.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.928+0000 7f31c9d30700 1 --2- 172.21.15.90:0/4099291977 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f31ac0afc10 0x7f31ac0b2120 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:06.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.928+0000 7f31c9d30700 1 --2- 172.21.15.90:0/4099291977 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31c41031e0 0x7f31c419ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:06.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.928+0000 7f31c9d30700 1 -- 172.21.15.90:0/4099291977 >> 172.21.15.90:0/4099291977 conn(0x7f31c40fe5c0 msgr2=0x7f31c4106b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:06.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.928+0000 7f31c9d30700 1 -- 172.21.15.90:0/4099291977 shutdown_connections 2024-04-25T11:34:06.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:06.928+0000 7f31c9d30700 1 -- 172.21.15.90:0/4099291977 wait complete. 2024-04-25T11:34:07.831 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:07 smithi090 ceph-mon[29575]: pgmap v302: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T11:34:07.831 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:07 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/4099291977' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T11:34:08.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:07 smithi156 ceph-mon[33045]: pgmap v302: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T11:34:08.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:07 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/4099291977' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T11:34:08.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:07 smithi138 ceph-mon[32328]: pgmap v302: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T11:34:08.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:07 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/4099291977' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T11:34:08.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.521+0000 7f1f8b86e700 1 -- 172.21.15.90:0/1671039845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f84105d10 msgr2=0x7f1f8410e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:08.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.521+0000 7f1f8b86e700 1 --2- 172.21.15.90:0/1671039845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f84105d10 0x7f1f8410e260 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1f74009990 tx=0x7f1f74009ca0 comp rx=0 tx=0).stop 2024-04-25T11:34:08.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.521+0000 7f1f8b86e700 1 -- 172.21.15.90:0/1671039845 shutdown_connections 2024-04-25T11:34:08.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.521+0000 7f1f8b86e700 1 --2- 172.21.15.90:0/1671039845 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1f8410e7f0 0x7f1f84110cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:08.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.521+0000 7f1f8b86e700 1 --2- 172.21.15.90:0/1671039845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f841053c0 0x7f1f841057d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:08.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.521+0000 7f1f8b86e700 1 --2- 172.21.15.90:0/1671039845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f84105d10 0x7f1f8410e260 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:08.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.521+0000 7f1f8b86e700 1 -- 172.21.15.90:0/1671039845 >> 172.21.15.90:0/1671039845 conn(0x7f1f84078cd0 msgr2=0x7f1f84079110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:08.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.521+0000 7f1f8b86e700 1 -- 172.21.15.90:0/1671039845 shutdown_connections 2024-04-25T11:34:08.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.521+0000 7f1f8b86e700 1 -- 172.21.15.90:0/1671039845 wait complete. 2024-04-25T11:34:08.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.522+0000 7f1f8b86e700 1 Processor -- start 2024-04-25T11:34:08.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.522+0000 7f1f8b86e700 1 -- start start 2024-04-25T11:34:08.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.522+0000 7f1f8b86e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f841053c0 0x7f1f8419adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:08.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.522+0000 7f1f8b86e700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1f84105d10 0x7f1f8419b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:08.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.522+0000 7f1f8b86e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f8410e7f0 0x7f1f8419f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:08.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.522+0000 7f1f8b86e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1f84114350 con 0x7f1f8410e7f0 2024-04-25T11:34:08.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.522+0000 7f1f8b86e700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1f841141d0 con 0x7f1f84105d10 2024-04-25T11:34:08.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.522+0000 7f1f8b86e700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1f841144d0 con 0x7f1f841053c0 2024-04-25T11:34:08.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.522+0000 7f1f89e0b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f8410e7f0 0x7f1f8419f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:08.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.522+0000 7f1f89e0b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f8410e7f0 0x7f1f8419f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51802/0 (socket says 172.21.15.90:51802) 2024-04-25T11:34:08.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.522+0000 7f1f89e0b700 1 -- 172.21.15.90:0/413876218 learned_addr learned my addr 172.21.15.90:0/413876218 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:08.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.523+0000 7f1f88e09700 1 --2- 172.21.15.90:0/413876218 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1f84105d10 0x7f1f8419b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:08.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.523+0000 7f1f8960a700 1 --2- 172.21.15.90:0/413876218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f841053c0 0x7f1f8419adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:08.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.523+0000 7f1f89e0b700 1 -- 172.21.15.90:0/413876218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f841053c0 msgr2=0x7f1f8419adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:08.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.523+0000 7f1f89e0b700 1 --2- 172.21.15.90:0/413876218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f841053c0 0x7f1f8419adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:08.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.523+0000 7f1f89e0b700 1 -- 172.21.15.90:0/413876218 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1f84105d10 msgr2=0x7f1f8419b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:08.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.523+0000 7f1f89e0b700 1 --2- 172.21.15.90:0/413876218 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1f84105d10 0x7f1f8419b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:08.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.523+0000 7f1f89e0b700 1 -- 172.21.15.90:0/413876218 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f8419fc90 con 0x7f1f8410e7f0 2024-04-25T11:34:08.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.523+0000 7f1f89e0b700 1 --2- 172.21.15.90:0/413876218 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f8410e7f0 0x7f1f8419f5f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c00d980 tx=0x7f1f7c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:08.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.523+0000 7f1f7a7fc700 1 -- 172.21.15.90:0/413876218 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c017070 con 0x7f1f8410e7f0 2024-04-25T11:34:08.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.523+0000 7f1f7a7fc700 1 -- 172.21.15.90:0/413876218 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f7c009df0 con 0x7f1f8410e7f0 2024-04-25T11:34:08.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.523+0000 7f1f8b86e700 1 -- 172.21.15.90:0/413876218 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f8419ff80 con 0x7f1f8410e7f0 2024-04-25T11:34:08.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.523+0000 7f1f7a7fc700 1 -- 172.21.15.90:0/413876218 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c0053a0 con 0x7f1f8410e7f0 2024-04-25T11:34:08.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.523+0000 7f1f8b86e700 1 -- 172.21.15.90:0/413876218 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f841a78a0 con 0x7f1f8410e7f0 2024-04-25T11:34:08.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.524+0000 7f1f8b86e700 1 -- 172.21.15.90:0/413876218 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f84051190 con 0x7f1f8410e7f0 2024-04-25T11:34:08.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.526+0000 7f1f7a7fc700 1 -- 172.21.15.90:0/413876218 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f1f7c004750 con 0x7f1f8410e7f0 2024-04-25T11:34:08.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.527+0000 7f1f7a7fc700 1 --2- 172.21.15.90:0/413876218 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f1f700afc10 0x7f1f700b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:08.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.527+0000 7f1f8960a700 1 --2- 172.21.15.90:0/413876218 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f1f700afc10 0x7f1f700b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:08.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.527+0000 7f1f7a7fc700 1 -- 172.21.15.90:0/413876218 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7f1f7c0defc0 con 0x7f1f8410e7f0 2024-04-25T11:34:08.536 INFO:teuthology.orchestra.run.smithi090.stdout:206158430252 2024-04-25T11:34:08.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.527+0000 7f1f8960a700 1 --2- 172.21.15.90:0/413876218 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f1f700afc10 0x7f1f700b2120 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1f80009550 tx=0x7f1f80009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:08.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.531+0000 7f1f7a7fc700 1 -- 172.21.15.90:0/413876218 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1f7c020020 con 0x7f1f8410e7f0 2024-04-25T11:34:08.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.738+0000 7f1f8b86e700 1 -- 172.21.15.90:0/413876218 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f1f8402e510 con 0x7f1f8410e7f0 2024-04-25T11:34:08.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.739+0000 7f1f7a7fc700 1 -- 172.21.15.90:0/413876218 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1f7c0a7bf0 con 0x7f1f8410e7f0 2024-04-25T11:34:08.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.741+0000 7f1f8b86e700 1 -- 172.21.15.90:0/413876218 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f1f700afc10 msgr2=0x7f1f700b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:08.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.741+0000 7f1f8b86e700 1 --2- 172.21.15.90:0/413876218 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f1f700afc10 0x7f1f700b2120 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1f80009550 tx=0x7f1f80009040 comp rx=0 tx=0).stop 2024-04-25T11:34:08.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.741+0000 7f1f8b86e700 1 -- 172.21.15.90:0/413876218 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f8410e7f0 msgr2=0x7f1f8419f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:08.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.741+0000 7f1f8b86e700 1 --2- 172.21.15.90:0/413876218 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f8410e7f0 0x7f1f8419f5f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c00d980 tx=0x7f1f7c00dd40 comp rx=0 tx=0).stop 2024-04-25T11:34:08.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.741+0000 7f1f8b86e700 1 -- 172.21.15.90:0/413876218 shutdown_connections 2024-04-25T11:34:08.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.741+0000 7f1f8b86e700 1 --2- 172.21.15.90:0/413876218 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f8410e7f0 0x7f1f8419f5f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:08.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.741+0000 7f1f8b86e700 1 --2- 172.21.15.90:0/413876218 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f1f700afc10 0x7f1f700b2120 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:08.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.741+0000 7f1f8b86e700 1 --2- 172.21.15.90:0/413876218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f841053c0 0x7f1f8419adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:08.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.741+0000 7f1f8b86e700 1 --2- 172.21.15.90:0/413876218 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1f84105d10 0x7f1f8419b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:08.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.741+0000 7f1f8b86e700 1 -- 172.21.15.90:0/413876218 >> 172.21.15.90:0/413876218 conn(0x7f1f84078cd0 msgr2=0x7f1f841010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:08.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.741+0000 7f1f8b86e700 1 -- 172.21.15.90:0/413876218 shutdown_connections 2024-04-25T11:34:08.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:05.741+0000 7f1f8b86e700 1 -- 172.21.15.90:0/413876218 wait complete. 2024-04-25T11:34:08.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.944+0000 7f18310e4700 1 -- 172.21.15.90:0/689832583 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f182c109170 msgr2=0x7f182c109580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:08.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.944+0000 7f18310e4700 1 --2- 172.21.15.90:0/689832583 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f182c109170 0x7f182c109580 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f181c009990 tx=0x7f181c009ca0 comp rx=0 tx=0).stop 2024-04-25T11:34:08.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.944+0000 7f18310e4700 1 -- 172.21.15.90:0/689832583 shutdown_connections 2024-04-25T11:34:08.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.944+0000 7f18310e4700 1 --2- 172.21.15.90:0/689832583 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f182c103060 0x7f182c111000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:08.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.944+0000 7f18310e4700 1 --2- 172.21.15.90:0/689832583 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f182c109170 0x7f182c109580 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:08.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.944+0000 7f18310e4700 1 --2- 172.21.15.90:0/689832583 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f182c102710 0x7f182c102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:08.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.944+0000 7f18310e4700 1 -- 172.21.15.90:0/689832583 >> 172.21.15.90:0/689832583 conn(0x7f182c0fe380 msgr2=0x7f182c1007d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:08.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.944+0000 7f18310e4700 1 -- 172.21.15.90:0/689832583 shutdown_connections 2024-04-25T11:34:08.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.944+0000 7f18310e4700 1 -- 172.21.15.90:0/689832583 wait complete. 2024-04-25T11:34:08.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.945+0000 7f18310e4700 1 Processor -- start 2024-04-25T11:34:08.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.945+0000 7f18310e4700 1 -- start start 2024-04-25T11:34:08.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.946+0000 7f18310e4700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f182c102710 0x7f182c19ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:08.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.946+0000 7f18310e4700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f182c103060 0x7f182c19b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:08.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.946+0000 7f18310e4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f182c109170 0x7f182c19f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:08.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.946+0000 7f18310e4700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f182c114420 con 0x7f182c109170 2024-04-25T11:34:08.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.946+0000 7f18310e4700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f182c1142a0 con 0x7f182c103060 2024-04-25T11:34:08.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.946+0000 7f18310e4700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f182c1145a0 con 0x7f182c102710 2024-04-25T11:34:08.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.946+0000 7f182b59e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f182c109170 0x7f182c19f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:08.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.946+0000 7f182ad9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f182c102710 0x7f182c19ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:08.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.946+0000 7f182ad9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f182c102710 0x7f182c19ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:50146/0 (socket says 172.21.15.90:50146) 2024-04-25T11:34:08.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.946+0000 7f182ad9d700 1 -- 172.21.15.90:0/3397950017 learned_addr learned my addr 172.21.15.90:0/3397950017 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:08.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.946+0000 7f182b59e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f182c109170 0x7f182c19f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51822/0 (socket says 172.21.15.90:51822) 2024-04-25T11:34:08.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.946+0000 7f182a59c700 1 --2- 172.21.15.90:0/3397950017 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f182c103060 0x7f182c19b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:08.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.946+0000 7f182b59e700 1 -- 172.21.15.90:0/3397950017 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f182c102710 msgr2=0x7f182c19ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:08.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.946+0000 7f182b59e700 1 --2- 172.21.15.90:0/3397950017 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f182c102710 0x7f182c19ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:08.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.946+0000 7f182b59e700 1 -- 172.21.15.90:0/3397950017 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f182c103060 msgr2=0x7f182c19b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:08.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.946+0000 7f182b59e700 1 --2- 172.21.15.90:0/3397950017 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f182c103060 0x7f182c19b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:08.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.946+0000 7f182b59e700 1 -- 172.21.15.90:0/3397950017 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f182c19fb70 con 0x7f182c109170 2024-04-25T11:34:08.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.946+0000 7f182ad9d700 1 --2- 172.21.15.90:0/3397950017 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f182c102710 0x7f182c19ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:34:08.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.946+0000 7f182a59c700 1 --2- 172.21.15.90:0/3397950017 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f182c103060 0x7f182c19b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:08.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.946+0000 7f182b59e700 1 --2- 172.21.15.90:0/3397950017 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f182c109170 0x7f182c19f580 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1824007f10 tx=0x7f182400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:08.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.946+0000 7f1813fff700 1 -- 172.21.15.90:0/3397950017 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182400ddd0 con 0x7f182c109170 2024-04-25T11:34:08.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.947+0000 7f18310e4700 1 -- 172.21.15.90:0/3397950017 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f182c19fe60 con 0x7f182c109170 2024-04-25T11:34:08.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.947+0000 7f1813fff700 1 -- 172.21.15.90:0/3397950017 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1824009e20 con 0x7f182c109170 2024-04-25T11:34:08.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.947+0000 7f1813fff700 1 -- 172.21.15.90:0/3397950017 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1824015400 con 0x7f182c109170 2024-04-25T11:34:08.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.947+0000 7f18310e4700 1 -- 172.21.15.90:0/3397950017 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f182c1a03a0 con 0x7f182c109170 2024-04-25T11:34:08.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.948+0000 7f18310e4700 1 -- 172.21.15.90:0/3397950017 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f182c051190 con 0x7f182c109170 2024-04-25T11:34:08.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.950+0000 7f1813fff700 1 -- 172.21.15.90:0/3397950017 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f1824005020 con 0x7f182c109170 2024-04-25T11:34:08.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.951+0000 7f1813fff700 1 --2- 172.21.15.90:0/3397950017 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f18140afc10 0x7f18140b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:08.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.951+0000 7f182ad9d700 1 --2- 172.21.15.90:0/3397950017 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f18140afc10 0x7f18140b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:08.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.951+0000 7f1813fff700 1 -- 172.21.15.90:0/3397950017 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7f18240de6d0 con 0x7f182c109170 2024-04-25T11:34:08.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.951+0000 7f182ad9d700 1 --2- 172.21.15.90:0/3397950017 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f18140afc10 0x7f18140b2120 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f181c003800 tx=0x7f181c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:08.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:08.955+0000 7f1813fff700 1 -- 172.21.15.90:0/3397950017 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1824010070 con 0x7f182c109170 2024-04-25T11:34:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:09.159+0000 7f18310e4700 1 -- 172.21.15.90:0/3397950017 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f182c02e510 con 0x7f182c109170 2024-04-25T11:34:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:09.160+0000 7f1813fff700 1 -- 172.21.15.90:0/3397950017 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f18240a7280 con 0x7f182c109170 2024-04-25T11:34:09.148 INFO:teuthology.orchestra.run.smithi090.stdout:34359738446 2024-04-25T11:34:09.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:09.162+0000 7f18310e4700 1 -- 172.21.15.90:0/3397950017 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f18140afc10 msgr2=0x7f18140b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:09.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:09.162+0000 7f18310e4700 1 --2- 172.21.15.90:0/3397950017 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f18140afc10 0x7f18140b2120 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f181c003800 tx=0x7f181c01a040 comp rx=0 tx=0).stop 2024-04-25T11:34:09.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:09.162+0000 7f18310e4700 1 -- 172.21.15.90:0/3397950017 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f182c109170 msgr2=0x7f182c19f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:09.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:09.162+0000 7f18310e4700 1 --2- 172.21.15.90:0/3397950017 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f182c109170 0x7f182c19f580 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1824007f10 tx=0x7f182400d3b0 comp rx=0 tx=0).stop 2024-04-25T11:34:09.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:09.162+0000 7f18310e4700 1 -- 172.21.15.90:0/3397950017 shutdown_connections 2024-04-25T11:34:09.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:09.162+0000 7f18310e4700 1 --2- 172.21.15.90:0/3397950017 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f182c109170 0x7f182c19f580 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:09.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:09.162+0000 7f18310e4700 1 --2- 172.21.15.90:0/3397950017 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f18140afc10 0x7f18140b2120 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:09.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:09.162+0000 7f18310e4700 1 --2- 172.21.15.90:0/3397950017 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f182c102710 0x7f182c19ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:09.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:09.162+0000 7f18310e4700 1 --2- 172.21.15.90:0/3397950017 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f182c103060 0x7f182c19b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:09.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:09.162+0000 7f18310e4700 1 -- 172.21.15.90:0/3397950017 >> 172.21.15.90:0/3397950017 conn(0x7f182c0fe380 msgr2=0x7f182c10a620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:09.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:09.162+0000 7f18310e4700 1 -- 172.21.15.90:0/3397950017 shutdown_connections 2024-04-25T11:34:09.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:09.162+0000 7f18310e4700 1 -- 172.21.15.90:0/3397950017 wait complete. 2024-04-25T11:34:09.803 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986231 got 133143986234 for osd.4 2024-04-25T11:34:09.803 DEBUG:teuthology.parallel:result is None 2024-04-25T11:34:10.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:09 smithi156 ceph-mon[33045]: pgmap v303: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-25T11:34:10.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:09 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3397950017' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T11:34:10.111 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:09 smithi090 ceph-mon[29575]: pgmap v303: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-25T11:34:10.111 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:09 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3397950017' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T11:34:10.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:09 smithi138 ceph-mon[32328]: pgmap v303: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-25T11:34:10.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:09 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3397950017' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T11:34:10.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:10 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:34:11.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:10 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:34:11.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:10 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:34:11.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.798+0000 7ff79f187700 1 -- 172.21.15.90:0/299402519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7981096e0 msgr2=0x7ff798111040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:11.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.798+0000 7ff79f187700 1 --2- 172.21.15.90:0/299402519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7981096e0 0x7ff798111040 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff79000b0d0 tx=0x7ff79000b3e0 comp rx=0 tx=0).stop 2024-04-25T11:34:11.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.798+0000 7ff79f187700 1 -- 172.21.15.90:0/299402519 shutdown_connections 2024-04-25T11:34:11.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.798+0000 7ff79f187700 1 --2- 172.21.15.90:0/299402519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7981096e0 0x7ff798111040 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:11.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.798+0000 7ff79f187700 1 --2- 172.21.15.90:0/299402519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff798104460 0x7ff798104870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:11.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.798+0000 7ff79f187700 1 --2- 172.21.15.90:0/299402519 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff798108d90 0x7ff7981091a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:11.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.799+0000 7ff79f187700 1 -- 172.21.15.90:0/299402519 >> 172.21.15.90:0/299402519 conn(0x7ff7980ffd40 msgr2=0x7ff798102160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:11.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.799+0000 7ff79f187700 1 -- 172.21.15.90:0/299402519 shutdown_connections 2024-04-25T11:34:11.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.799+0000 7ff79f187700 1 -- 172.21.15.90:0/299402519 wait complete. 2024-04-25T11:34:11.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.799+0000 7ff79f187700 1 Processor -- start 2024-04-25T11:34:11.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.799+0000 7ff79f187700 1 -- start start 2024-04-25T11:34:11.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.799+0000 7ff79f187700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff798104460 0x7ff7981a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:11.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.799+0000 7ff79f187700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff798108d90 0x7ff7981a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:11.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.800+0000 7ff79f187700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7981096e0 0x7ff79819aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:11.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.800+0000 7ff79f187700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff798114310 con 0x7ff7981096e0 2024-04-25T11:34:11.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.800+0000 7ff79f187700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff798114190 con 0x7ff798104460 2024-04-25T11:34:11.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.800+0000 7ff79f187700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff798114490 con 0x7ff798108d90 2024-04-25T11:34:11.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.800+0000 7ff79d724700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7981096e0 0x7ff79819aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:11.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.800+0000 7ff79d724700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7981096e0 0x7ff79819aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43804/0 (socket says 172.21.15.90:43804) 2024-04-25T11:34:11.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.800+0000 7ff79d724700 1 -- 172.21.15.90:0/2746516348 learned_addr learned my addr 172.21.15.90:0/2746516348 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:11.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.800+0000 7ff797fff700 1 --2- 172.21.15.90:0/2746516348 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff798108d90 0x7ff7981a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:11.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.800+0000 7ff79cf23700 1 --2- 172.21.15.90:0/2746516348 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff798104460 0x7ff7981a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:11.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.800+0000 7ff79d724700 1 -- 172.21.15.90:0/2746516348 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff798108d90 msgr2=0x7ff7981a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:11.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.800+0000 7ff79d724700 1 --2- 172.21.15.90:0/2746516348 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff798108d90 0x7ff7981a0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:11.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.800+0000 7ff79d724700 1 -- 172.21.15.90:0/2746516348 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff798104460 msgr2=0x7ff7981a0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:11.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.800+0000 7ff79d724700 1 --2- 172.21.15.90:0/2746516348 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff798104460 0x7ff7981a0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:11.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.800+0000 7ff79d724700 1 -- 172.21.15.90:0/2746516348 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff79819b120 con 0x7ff7981096e0 2024-04-25T11:34:11.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.800+0000 7ff79d724700 1 --2- 172.21.15.90:0/2746516348 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7981096e0 0x7ff79819aa70 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff790007a00 tx=0x7ff790007a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:11.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.800+0000 7ff795ffb700 1 -- 172.21.15.90:0/2746516348 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff79001f070 con 0x7ff7981096e0 2024-04-25T11:34:11.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.800+0000 7ff795ffb700 1 -- 172.21.15.90:0/2746516348 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff790004c50 con 0x7ff7981096e0 2024-04-25T11:34:11.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.800+0000 7ff795ffb700 1 -- 172.21.15.90:0/2746516348 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff79001b690 con 0x7ff7981096e0 2024-04-25T11:34:11.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.800+0000 7ff79f187700 1 -- 172.21.15.90:0/2746516348 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff79819b3b0 con 0x7ff7981096e0 2024-04-25T11:34:11.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.801+0000 7ff79f187700 1 -- 172.21.15.90:0/2746516348 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff79819b840 con 0x7ff7981096e0 2024-04-25T11:34:11.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.802+0000 7ff79f187700 1 -- 172.21.15.90:0/2746516348 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff798051190 con 0x7ff7981096e0 2024-04-25T11:34:11.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.804+0000 7ff795ffb700 1 -- 172.21.15.90:0/2746516348 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7ff790022070 con 0x7ff7981096e0 2024-04-25T11:34:11.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.805+0000 7ff795ffb700 1 --2- 172.21.15.90:0/2746516348 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7ff7880afc90 0x7ff7880b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:11.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.805+0000 7ff795ffb700 1 -- 172.21.15.90:0/2746516348 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7ff79000f020 con 0x7ff7981096e0 2024-04-25T11:34:11.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.805+0000 7ff79cf23700 1 --2- 172.21.15.90:0/2746516348 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7ff7880afc90 0x7ff7880b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:11.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.805+0000 7ff79cf23700 1 --2- 172.21.15.90:0/2746516348 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7ff7880afc90 0x7ff7880b21a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff79819bbb0 tx=0x7ff78c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:11.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:11.808+0000 7ff795ffb700 1 -- 172.21.15.90:0/2746516348 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff790029020 con 0x7ff7981096e0 2024-04-25T11:34:11.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:11 smithi090 ceph-mon[29575]: pgmap v304: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T11:34:11.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:11 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:34:11.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:11 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:34:11.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:11 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:34:11.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:11 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:34:12.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:12.018+0000 7ff79f187700 1 -- 172.21.15.90:0/2746516348 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7ff79802e510 con 0x7ff7981096e0 2024-04-25T11:34:12.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:12.019+0000 7ff795ffb700 1 -- 172.21.15.90:0/2746516348 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff7900a8cd0 con 0x7ff7981096e0 2024-04-25T11:34:12.008 INFO:teuthology.orchestra.run.smithi090.stdout:231928234024 2024-04-25T11:34:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:12.021+0000 7ff79f187700 1 -- 172.21.15.90:0/2746516348 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7ff7880afc90 msgr2=0x7ff7880b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:12.021+0000 7ff79f187700 1 --2- 172.21.15.90:0/2746516348 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7ff7880afc90 0x7ff7880b21a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff79819bbb0 tx=0x7ff78c009260 comp rx=0 tx=0).stop 2024-04-25T11:34:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:12.021+0000 7ff79f187700 1 -- 172.21.15.90:0/2746516348 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7981096e0 msgr2=0x7ff79819aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:12.021+0000 7ff79f187700 1 --2- 172.21.15.90:0/2746516348 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7981096e0 0x7ff79819aa70 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff790007a00 tx=0x7ff790007a30 comp rx=0 tx=0).stop 2024-04-25T11:34:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:12.021+0000 7ff79f187700 1 -- 172.21.15.90:0/2746516348 shutdown_connections 2024-04-25T11:34:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:12.021+0000 7ff79f187700 1 --2- 172.21.15.90:0/2746516348 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7981096e0 0x7ff79819aa70 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:12.021+0000 7ff79f187700 1 --2- 172.21.15.90:0/2746516348 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff798104460 0x7ff7981a0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:12.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:12.021+0000 7ff79f187700 1 --2- 172.21.15.90:0/2746516348 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7ff7880afc90 0x7ff7880b21a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:12.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:12.021+0000 7ff79f187700 1 --2- 172.21.15.90:0/2746516348 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff798108d90 0x7ff7981a0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:12.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:12.021+0000 7ff79f187700 1 -- 172.21.15.90:0/2746516348 >> 172.21.15.90:0/2746516348 conn(0x7ff7980ffd40 msgr2=0x7ff798108500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:12.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:12.021+0000 7ff79f187700 1 -- 172.21.15.90:0/2746516348 shutdown_connections 2024-04-25T11:34:12.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:12.021+0000 7ff79f187700 1 -- 172.21.15.90:0/2746516348 wait complete. 2024-04-25T11:34:12.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:11 smithi138 ceph-mon[32328]: pgmap v304: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T11:34:12.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:11 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:34:12.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:11 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:34:12.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:11 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:34:12.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:11 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:34:12.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:11 smithi156 ceph-mon[33045]: pgmap v304: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T11:34:12.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:11 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:34:12.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:11 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:34:12.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:11 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:34:12.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:11 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:34:12.624 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234019 got 231928234024 for osd.8 2024-04-25T11:34:12.624 DEBUG:teuthology.parallel:result is None 2024-04-25T11:34:13.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:12 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2746516348' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T11:34:13.149 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:12 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2746516348' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T11:34:13.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:12 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2746516348' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T11:34:13.598 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430247 got 206158430252 for osd.7 2024-04-25T11:34:13.598 DEBUG:teuthology.parallel:result is None 2024-04-25T11:34:14.165 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:13 smithi090 ceph-mon[29575]: pgmap v305: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T11:34:14.194 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411391 got 77309411397 for osd.2 2024-04-25T11:34:14.195 DEBUG:teuthology.parallel:result is None 2024-04-25T11:34:14.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:13 smithi138 ceph-mon[32328]: pgmap v305: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T11:34:14.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:13 smithi156 ceph-mon[33045]: pgmap v305: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T11:34:14.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.635+0000 7f28cca59700 1 -- 172.21.15.90:0/948257533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28c8108610 msgr2=0x7f28c810ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:14.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.635+0000 7f28cca59700 1 --2- 172.21.15.90:0/948257533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28c8108610 0x7f28c810ff70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f28c000b0d0 tx=0x7f28c000b3e0 comp rx=0 tx=0).stop 2024-04-25T11:34:14.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.635+0000 7f28cca59700 1 -- 172.21.15.90:0/948257533 shutdown_connections 2024-04-25T11:34:14.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.635+0000 7f28cca59700 1 --2- 172.21.15.90:0/948257533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28c8108610 0x7f28c810ff70 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:14.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.635+0000 7f28cca59700 1 --2- 172.21.15.90:0/948257533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28c8103390 0x7f28c81037a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:14.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.635+0000 7f28cca59700 1 --2- 172.21.15.90:0/948257533 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f28c8107cc0 0x7f28c81080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:14.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.635+0000 7f28cca59700 1 -- 172.21.15.90:0/948257533 >> 172.21.15.90:0/948257533 conn(0x7f28c80fec90 msgr2=0x7f28c81010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:14.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.635+0000 7f28cca59700 1 -- 172.21.15.90:0/948257533 shutdown_connections 2024-04-25T11:34:14.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.635+0000 7f28cca59700 1 -- 172.21.15.90:0/948257533 wait complete. 2024-04-25T11:34:14.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.636+0000 7f28cca59700 1 Processor -- start 2024-04-25T11:34:14.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.636+0000 7f28cca59700 1 -- start start 2024-04-25T11:34:14.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.636+0000 7f28cca59700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f28c8103390 0x7f28c819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:14.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.636+0000 7f28cca59700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28c8107cc0 0x7f28c819fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:14.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.636+0000 7f28cca59700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28c8108610 0x7f28c81999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:14.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.636+0000 7f28cca59700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f28c8113240 con 0x7f28c8107cc0 2024-04-25T11:34:14.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.636+0000 7f28cca59700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f28c81130c0 con 0x7f28c8103390 2024-04-25T11:34:14.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.636+0000 7f28cca59700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f28c81133c0 con 0x7f28c8108610 2024-04-25T11:34:14.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.636+0000 7f28c5d9b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28c8107cc0 0x7f28c819fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:14.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.636+0000 7f28c5d9b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28c8107cc0 0x7f28c819fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43826/0 (socket says 172.21.15.90:43826) 2024-04-25T11:34:14.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.636+0000 7f28c5d9b700 1 -- 172.21.15.90:0/2652403678 learned_addr learned my addr 172.21.15.90:0/2652403678 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:14.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.636+0000 7f28c659c700 1 --2- 172.21.15.90:0/2652403678 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f28c8103390 0x7f28c819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:14.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.637+0000 7f28c6d9d700 1 --2- 172.21.15.90:0/2652403678 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28c8108610 0x7f28c81999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:14.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.637+0000 7f28c5d9b700 1 -- 172.21.15.90:0/2652403678 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28c8108610 msgr2=0x7f28c81999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:14.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.637+0000 7f28c5d9b700 1 --2- 172.21.15.90:0/2652403678 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28c8108610 0x7f28c81999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:14.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.637+0000 7f28c5d9b700 1 -- 172.21.15.90:0/2652403678 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f28c8103390 msgr2=0x7f28c819f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:14.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.637+0000 7f28c5d9b700 1 --2- 172.21.15.90:0/2652403678 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f28c8103390 0x7f28c819f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:14.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.637+0000 7f28c5d9b700 1 -- 172.21.15.90:0/2652403678 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28c819a050 con 0x7f28c8107cc0 2024-04-25T11:34:14.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.637+0000 7f28c6d9d700 1 --2- 172.21.15.90:0/2652403678 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28c8108610 0x7f28c81999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:14.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.637+0000 7f28c5d9b700 1 --2- 172.21.15.90:0/2652403678 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28c8107cc0 0x7f28c819fcc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f28bc00e970 tx=0x7f28bc00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:14.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.637+0000 7f28b77fe700 1 -- 172.21.15.90:0/2652403678 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28bc00cbd0 con 0x7f28c8107cc0 2024-04-25T11:34:14.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.637+0000 7f28cca59700 1 -- 172.21.15.90:0/2652403678 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28c819a340 con 0x7f28c8107cc0 2024-04-25T11:34:14.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.637+0000 7f28b77fe700 1 -- 172.21.15.90:0/2652403678 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28bc00cd70 con 0x7f28c8107cc0 2024-04-25T11:34:14.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.637+0000 7f28b77fe700 1 -- 172.21.15.90:0/2652403678 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28bc005680 con 0x7f28c8107cc0 2024-04-25T11:34:14.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.637+0000 7f28cca59700 1 -- 172.21.15.90:0/2652403678 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28c81a6610 con 0x7f28c8107cc0 2024-04-25T11:34:14.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.638+0000 7f28cca59700 1 -- 172.21.15.90:0/2652403678 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28c8051190 con 0x7f28c8107cc0 2024-04-25T11:34:14.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.640+0000 7f28b77fe700 1 -- 172.21.15.90:0/2652403678 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f28bc004750 con 0x7f28c8107cc0 2024-04-25T11:34:14.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.641+0000 7f28b77fe700 1 --2- 172.21.15.90:0/2652403678 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f28b00afc10 0x7f28b00b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:14.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.641+0000 7f28c659c700 1 --2- 172.21.15.90:0/2652403678 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f28b00afc10 0x7f28b00b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:14.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.641+0000 7f28b77fe700 1 -- 172.21.15.90:0/2652403678 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7f28bc0deff0 con 0x7f28c8107cc0 2024-04-25T11:34:14.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.641+0000 7f28c659c700 1 --2- 172.21.15.90:0/2652403678 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f28b00afc10 0x7f28b00b2120 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f28c819aae0 tx=0x7f28b80073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:14.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.645+0000 7f28b77fe700 1 -- 172.21.15.90:0/2652403678 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f28bc053ff0 con 0x7f28c8107cc0 2024-04-25T11:34:14.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.851+0000 7f28cca59700 1 -- 172.21.15.90:0/2652403678 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f28c819aed0 con 0x7f28c8107cc0 2024-04-25T11:34:14.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.851+0000 7f28b77fe700 1 -- 172.21.15.90:0/2652403678 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f28bc0a7c20 con 0x7f28c8107cc0 2024-04-25T11:34:14.839 INFO:teuthology.orchestra.run.smithi090.stdout:309237645337 2024-04-25T11:34:14.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.853+0000 7f28cca59700 1 -- 172.21.15.90:0/2652403678 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f28b00afc10 msgr2=0x7f28b00b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:14.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.853+0000 7f28cca59700 1 --2- 172.21.15.90:0/2652403678 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f28b00afc10 0x7f28b00b2120 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f28c819aae0 tx=0x7f28b80073d0 comp rx=0 tx=0).stop 2024-04-25T11:34:14.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.853+0000 7f28cca59700 1 -- 172.21.15.90:0/2652403678 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28c8107cc0 msgr2=0x7f28c819fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:14.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.853+0000 7f28cca59700 1 --2- 172.21.15.90:0/2652403678 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28c8107cc0 0x7f28c819fcc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f28bc00e970 tx=0x7f28bc00ec80 comp rx=0 tx=0).stop 2024-04-25T11:34:14.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.853+0000 7f28cca59700 1 -- 172.21.15.90:0/2652403678 shutdown_connections 2024-04-25T11:34:14.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.853+0000 7f28cca59700 1 --2- 172.21.15.90:0/2652403678 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f28b00afc10 0x7f28b00b2120 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:14.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.853+0000 7f28cca59700 1 --2- 172.21.15.90:0/2652403678 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28c8108610 0x7f28c81999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:14.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.853+0000 7f28cca59700 1 --2- 172.21.15.90:0/2652403678 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f28c8103390 0x7f28c819f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:14.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.853+0000 7f28cca59700 1 --2- 172.21.15.90:0/2652403678 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28c8107cc0 0x7f28c819fcc0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:14.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.853+0000 7f28cca59700 1 -- 172.21.15.90:0/2652403678 >> 172.21.15.90:0/2652403678 conn(0x7f28c80fec90 msgr2=0x7f28c8107430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:14.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.853+0000 7f28cca59700 1 -- 172.21.15.90:0/2652403678 shutdown_connections 2024-04-25T11:34:14.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:14.853+0000 7f28cca59700 1 -- 172.21.15.90:0/2652403678 wait complete. 2024-04-25T11:34:15.166 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:14 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2652403678' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T11:34:15.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:14 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2652403678' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T11:34:15.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:14 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2652403678' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T11:34:16.066 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:15 smithi090 ceph-mon[29575]: pgmap v306: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-25T11:34:16.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:15 smithi138 ceph-mon[32328]: pgmap v306: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-25T11:34:16.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:15 smithi156 ceph-mon[33045]: pgmap v306: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-25T11:34:16.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.384+0000 7f9b63d11700 1 -- 172.21.15.90:0/3994974420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b5c1091b0 msgr2=0x7f9b5c1095c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:16.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.384+0000 7f9b63d11700 1 --2- 172.21.15.90:0/3994974420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b5c1091b0 0x7f9b5c1095c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9b58009990 tx=0x7f9b58009ca0 comp rx=0 tx=0).stop 2024-04-25T11:34:16.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.384+0000 7f9b63d11700 1 -- 172.21.15.90:0/3994974420 shutdown_connections 2024-04-25T11:34:16.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.384+0000 7f9b63d11700 1 --2- 172.21.15.90:0/3994974420 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9b5c102f90 0x7f9b5c111020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:16.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.384+0000 7f9b63d11700 1 --2- 172.21.15.90:0/3994974420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b5c1091b0 0x7f9b5c1095c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:16.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.384+0000 7f9b63d11700 1 --2- 172.21.15.90:0/3994974420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b5c102640 0x7f9b5c102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:16.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.384+0000 7f9b63d11700 1 -- 172.21.15.90:0/3994974420 >> 172.21.15.90:0/3994974420 conn(0x7f9b5c0fe370 msgr2=0x7f9b5c1007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:16.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.384+0000 7f9b63d11700 1 -- 172.21.15.90:0/3994974420 shutdown_connections 2024-04-25T11:34:16.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.384+0000 7f9b63d11700 1 -- 172.21.15.90:0/3994974420 wait complete. 2024-04-25T11:34:16.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.385+0000 7f9b63d11700 1 Processor -- start 2024-04-25T11:34:16.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.385+0000 7f9b63d11700 1 -- start start 2024-04-25T11:34:16.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.385+0000 7f9b63d11700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b5c102640 0x7f9b5c19ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:16.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.385+0000 7f9b63d11700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9b5c102f90 0x7f9b5c19b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:16.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.385+0000 7f9b63d11700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b5c1091b0 0x7f9b5c19f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:16.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.385+0000 7f9b63d11700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9b5c1144b0 con 0x7f9b5c102640 2024-04-25T11:34:16.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.385+0000 7f9b63d11700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9b5c114330 con 0x7f9b5c102f90 2024-04-25T11:34:16.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.385+0000 7f9b63d11700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9b5c114630 con 0x7f9b5c1091b0 2024-04-25T11:34:16.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.385+0000 7f9b61aad700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b5c102640 0x7f9b5c19ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:16.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.385+0000 7f9b61aad700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b5c102640 0x7f9b5c19ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43842/0 (socket says 172.21.15.90:43842) 2024-04-25T11:34:16.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.385+0000 7f9b61aad700 1 -- 172.21.15.90:0/968378037 learned_addr learned my addr 172.21.15.90:0/968378037 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:16.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.385+0000 7f9b612ac700 1 --2- 172.21.15.90:0/968378037 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9b5c102f90 0x7f9b5c19b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:16.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.386+0000 7f9b622ae700 1 --2- 172.21.15.90:0/968378037 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b5c1091b0 0x7f9b5c19f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:16.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.386+0000 7f9b61aad700 1 -- 172.21.15.90:0/968378037 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b5c1091b0 msgr2=0x7f9b5c19f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:16.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.386+0000 7f9b61aad700 1 --2- 172.21.15.90:0/968378037 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b5c1091b0 0x7f9b5c19f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:16.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.386+0000 7f9b61aad700 1 -- 172.21.15.90:0/968378037 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9b5c102f90 msgr2=0x7f9b5c19b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:16.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.386+0000 7f9b61aad700 1 --2- 172.21.15.90:0/968378037 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9b5c102f90 0x7f9b5c19b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:16.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.386+0000 7f9b61aad700 1 -- 172.21.15.90:0/968378037 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b5c19fc70 con 0x7f9b5c102640 2024-04-25T11:34:16.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.386+0000 7f9b61aad700 1 --2- 172.21.15.90:0/968378037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b5c102640 0x7f9b5c19ada0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9b58006010 tx=0x7f9b580057a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:16.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.386+0000 7f9b52ffd700 1 -- 172.21.15.90:0/968378037 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b58027070 con 0x7f9b5c102640 2024-04-25T11:34:16.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.386+0000 7f9b52ffd700 1 -- 172.21.15.90:0/968378037 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b58003fe0 con 0x7f9b5c102640 2024-04-25T11:34:16.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.386+0000 7f9b63d11700 1 -- 172.21.15.90:0/968378037 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b5c19ff00 con 0x7f9b5c102640 2024-04-25T11:34:16.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.386+0000 7f9b52ffd700 1 -- 172.21.15.90:0/968378037 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b58022af0 con 0x7f9b5c102640 2024-04-25T11:34:16.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.386+0000 7f9b63d11700 1 -- 172.21.15.90:0/968378037 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b5c1a7880 con 0x7f9b5c102640 2024-04-25T11:34:16.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.387+0000 7f9b63d11700 1 -- 172.21.15.90:0/968378037 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b5c10c750 con 0x7f9b5c102640 2024-04-25T11:34:16.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.388+0000 7f9b52ffd700 1 -- 172.21.15.90:0/968378037 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f9b5802a070 con 0x7f9b5c102640 2024-04-25T11:34:16.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.389+0000 7f9b52ffd700 1 --2- 172.21.15.90:0/968378037 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f9b480afc90 0x7f9b480b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:16.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.389+0000 7f9b612ac700 1 --2- 172.21.15.90:0/968378037 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f9b480afc90 0x7f9b480b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:16.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.389+0000 7f9b52ffd700 1 -- 172.21.15.90:0/968378037 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7979+0+0 (secure 0 0 0) 0x7f9b580e04b0 con 0x7f9b5c102640 2024-04-25T11:34:16.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.389+0000 7f9b612ac700 1 --2- 172.21.15.90:0/968378037 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f9b480afc90 0x7f9b480b21a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c100770 tx=0x7f9b4c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:16.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.393+0000 7f9b52ffd700 1 -- 172.21.15.90:0/968378037 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9b58031080 con 0x7f9b5c102640 2024-04-25T11:34:16.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.596+0000 7f9b63d11700 1 -- 172.21.15.90:0/968378037 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f9b5c19c100 con 0x7f9b5c102640 2024-04-25T11:34:16.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.596+0000 7f9b52ffd700 1 -- 172.21.15.90:0/968378037 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f9b58013070 con 0x7f9b5c102640 2024-04-25T11:34:16.584 INFO:teuthology.orchestra.run.smithi090.stdout:154618822712 2024-04-25T11:34:16.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.598+0000 7f9b63d11700 1 -- 172.21.15.90:0/968378037 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f9b480afc90 msgr2=0x7f9b480b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:16.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.598+0000 7f9b63d11700 1 --2- 172.21.15.90:0/968378037 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f9b480afc90 0x7f9b480b21a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c100770 tx=0x7f9b4c00b410 comp rx=0 tx=0).stop 2024-04-25T11:34:16.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.598+0000 7f9b63d11700 1 -- 172.21.15.90:0/968378037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b5c102640 msgr2=0x7f9b5c19ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:16.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.598+0000 7f9b63d11700 1 --2- 172.21.15.90:0/968378037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b5c102640 0x7f9b5c19ada0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9b58006010 tx=0x7f9b580057a0 comp rx=0 tx=0).stop 2024-04-25T11:34:16.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.598+0000 7f9b63d11700 1 -- 172.21.15.90:0/968378037 shutdown_connections 2024-04-25T11:34:16.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.598+0000 7f9b63d11700 1 --2- 172.21.15.90:0/968378037 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f9b480afc90 0x7f9b480b21a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:16.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.598+0000 7f9b63d11700 1 --2- 172.21.15.90:0/968378037 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b5c1091b0 0x7f9b5c19f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:16.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.598+0000 7f9b63d11700 1 --2- 172.21.15.90:0/968378037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b5c102640 0x7f9b5c19ada0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:16.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.598+0000 7f9b63d11700 1 --2- 172.21.15.90:0/968378037 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9b5c102f90 0x7f9b5c19b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:16.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.598+0000 7f9b63d11700 1 -- 172.21.15.90:0/968378037 >> 172.21.15.90:0/968378037 conn(0x7f9b5c0fe370 msgr2=0x7f9b5c0fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:16.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.598+0000 7f9b63d11700 1 -- 172.21.15.90:0/968378037 shutdown_connections 2024-04-25T11:34:16.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:16.598+0000 7f9b63d11700 1 -- 172.21.15.90:0/968378037 wait complete. 2024-04-25T11:34:16.750 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738443 got 34359738446 for osd.0 2024-04-25T11:34:16.750 DEBUG:teuthology.parallel:result is None 2024-04-25T11:34:17.118 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:16 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/968378037' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T11:34:17.118 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:16 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-04-25T11:34:17.118 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:16 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 7]}]: dispatch 2024-04-25T11:34:17.118 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:16 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2024-04-25T11:34:17.119 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:16 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]: dispatch 2024-04-25T11:34:17.119 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:16 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 7]}]: dispatch 2024-04-25T11:34:17.119 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:16 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-04-25T11:34:17.119 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:16 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-04-25T11:34:17.119 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:16 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]: dispatch 2024-04-25T11:34:17.119 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:16 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [1, 9]}]: dispatch 2024-04-25T11:34:17.119 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:16 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-04-25T11:34:17.119 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:16 smithi090 ceph-mon[29575]: pgmap v307: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-04-25T11:34:17.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.141+0000 7fb270f8f700 1 -- 172.21.15.90:0/3563937750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb26c1088a0 msgr2=0x7fb26c108cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:17.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.141+0000 7fb270f8f700 1 --2- 172.21.15.90:0/3563937750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb26c1088a0 0x7fb26c108cb0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb25c009990 tx=0x7fb25c009ca0 comp rx=0 tx=0).stop 2024-04-25T11:34:17.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.141+0000 7fb270f8f700 1 -- 172.21.15.90:0/3563937750 shutdown_connections 2024-04-25T11:34:17.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.141+0000 7fb270f8f700 1 --2- 172.21.15.90:0/3563937750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb26c1031c0 0x7fb26c10f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:17.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.141+0000 7fb270f8f700 1 --2- 172.21.15.90:0/3563937750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb26c1088a0 0x7fb26c108cb0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:17.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.141+0000 7fb270f8f700 1 --2- 172.21.15.90:0/3563937750 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb26c102870 0x7fb26c102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:17.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.141+0000 7fb270f8f700 1 -- 172.21.15.90:0/3563937750 >> 172.21.15.90:0/3563937750 conn(0x7fb26c0fe3f0 msgr2=0x7fb26c100840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:17.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.141+0000 7fb270f8f700 1 -- 172.21.15.90:0/3563937750 shutdown_connections 2024-04-25T11:34:17.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.141+0000 7fb270f8f700 1 -- 172.21.15.90:0/3563937750 wait complete. 2024-04-25T11:34:17.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.142+0000 7fb270f8f700 1 Processor -- start 2024-04-25T11:34:17.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.142+0000 7fb270f8f700 1 -- start start 2024-04-25T11:34:17.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.142+0000 7fb270f8f700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb26c102870 0x7fb26c19bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:17.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.142+0000 7fb270f8f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb26c1031c0 0x7fb26c19c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:17.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.142+0000 7fb270f8f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb26c1088a0 0x7fb26c19c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:17.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.142+0000 7fb270f8f700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb26c112a20 con 0x7fb26c1031c0 2024-04-25T11:34:17.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.142+0000 7fb270f8f700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb26c1128a0 con 0x7fb26c102870 2024-04-25T11:34:17.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.142+0000 7fb270f8f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb26c112ba0 con 0x7fb26c1088a0 2024-04-25T11:34:17.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.142+0000 7fb269d9b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb26c1031c0 0x7fb26c19c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:17.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.142+0000 7fb269d9b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb26c1031c0 0x7fb26c19c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43858/0 (socket says 172.21.15.90:43858) 2024-04-25T11:34:17.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.142+0000 7fb269d9b700 1 -- 172.21.15.90:0/3827044840 learned_addr learned my addr 172.21.15.90:0/3827044840 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:17.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.142+0000 7fb26a59c700 1 --2- 172.21.15.90:0/3827044840 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb26c102870 0x7fb26c19bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:17.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.142+0000 7fb26ad9d700 1 --2- 172.21.15.90:0/3827044840 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb26c1088a0 0x7fb26c19c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:17.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.142+0000 7fb269d9b700 1 -- 172.21.15.90:0/3827044840 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb26c1088a0 msgr2=0x7fb26c19c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:17.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.142+0000 7fb269d9b700 1 --2- 172.21.15.90:0/3827044840 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb26c1088a0 0x7fb26c19c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:17.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.142+0000 7fb269d9b700 1 -- 172.21.15.90:0/3827044840 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb26c102870 msgr2=0x7fb26c19bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:17.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.142+0000 7fb269d9b700 1 --2- 172.21.15.90:0/3827044840 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb26c102870 0x7fb26c19bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:17.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.142+0000 7fb269d9b700 1 -- 172.21.15.90:0/3827044840 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb26c1a6ed0 con 0x7fb26c1031c0 2024-04-25T11:34:17.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.142+0000 7fb26a59c700 1 --2- 172.21.15.90:0/3827044840 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb26c102870 0x7fb26c19bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:17.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.142+0000 7fb269d9b700 1 --2- 172.21.15.90:0/3827044840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb26c1031c0 0x7fb26c19c0a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb25400c8f0 tx=0x7fb25400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:17.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.143+0000 7fb2637fe700 1 -- 172.21.15.90:0/3827044840 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb254007b00 con 0x7fb26c1031c0 2024-04-25T11:34:17.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.143+0000 7fb270f8f700 1 -- 172.21.15.90:0/3827044840 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb26c1a71c0 con 0x7fb26c1031c0 2024-04-25T11:34:17.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.143+0000 7fb2637fe700 1 -- 172.21.15.90:0/3827044840 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb254007ca0 con 0x7fb26c1031c0 2024-04-25T11:34:17.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.143+0000 7fb2637fe700 1 -- 172.21.15.90:0/3827044840 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb254018660 con 0x7fb26c1031c0 2024-04-25T11:34:17.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.143+0000 7fb270f8f700 1 -- 172.21.15.90:0/3827044840 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb26c1a7750 con 0x7fb26c1031c0 2024-04-25T11:34:17.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.144+0000 7fb270f8f700 1 -- 172.21.15.90:0/3827044840 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb26c051190 con 0x7fb26c1031c0 2024-04-25T11:34:17.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.145+0000 7fb2637fe700 1 -- 172.21.15.90:0/3827044840 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fb25400f450 con 0x7fb26c1031c0 2024-04-25T11:34:17.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.146+0000 7fb2637fe700 1 --2- 172.21.15.90:0/3827044840 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb2580afc10 0x7fb2580b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:17.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.146+0000 7fb26a59c700 1 --2- 172.21.15.90:0/3827044840 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb2580afc10 0x7fb2580b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:17.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.146+0000 7fb2637fe700 1 -- 172.21.15.90:0/3827044840 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7fb2540dee90 con 0x7fb26c1031c0 2024-04-25T11:34:17.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.146+0000 7fb26a59c700 1 --2- 172.21.15.90:0/3827044840 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb2580afc10 0x7fb2580b2120 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb25c003800 tx=0x7fb25c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:17.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.150+0000 7fb2637fe700 1 -- 172.21.15.90:0/3827044840 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb254014030 con 0x7fb26c1031c0 2024-04-25T11:34:17.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:16 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/968378037' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T11:34:17.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-04-25T11:34:17.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 7]}]: dispatch 2024-04-25T11:34:17.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2024-04-25T11:34:17.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]: dispatch 2024-04-25T11:34:17.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 7]}]: dispatch 2024-04-25T11:34:17.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-04-25T11:34:17.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-04-25T11:34:17.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]: dispatch 2024-04-25T11:34:17.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [1, 9]}]: dispatch 2024-04-25T11:34:17.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:16 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-04-25T11:34:17.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:16 smithi138 ceph-mon[32328]: pgmap v307: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-04-25T11:34:17.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:16 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/968378037' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T11:34:17.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:16 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-04-25T11:34:17.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:16 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 7]}]: dispatch 2024-04-25T11:34:17.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:16 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2024-04-25T11:34:17.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:16 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]: dispatch 2024-04-25T11:34:17.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:16 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 7]}]: dispatch 2024-04-25T11:34:17.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:16 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-04-25T11:34:17.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:16 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-04-25T11:34:17.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:16 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]: dispatch 2024-04-25T11:34:17.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:16 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [1, 9]}]: dispatch 2024-04-25T11:34:17.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:16 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-04-25T11:34:17.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:16 smithi156 ceph-mon[33045]: pgmap v307: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-04-25T11:34:17.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.355+0000 7fb270f8f700 1 -- 172.21.15.90:0/3827044840 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fb26c1137f0 con 0x7fb26c1031c0 2024-04-25T11:34:17.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.355+0000 7fb2637fe700 1 -- 172.21.15.90:0/3827044840 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fb26c1137f0 con 0x7fb26c1031c0 2024-04-25T11:34:17.343 INFO:teuthology.orchestra.run.smithi090.stdout:283467841567 2024-04-25T11:34:17.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.357+0000 7fb270f8f700 1 -- 172.21.15.90:0/3827044840 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb2580afc10 msgr2=0x7fb2580b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:17.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.357+0000 7fb270f8f700 1 --2- 172.21.15.90:0/3827044840 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb2580afc10 0x7fb2580b2120 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb25c003800 tx=0x7fb25c01a040 comp rx=0 tx=0).stop 2024-04-25T11:34:17.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.357+0000 7fb270f8f700 1 -- 172.21.15.90:0/3827044840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb26c1031c0 msgr2=0x7fb26c19c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:17.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.357+0000 7fb270f8f700 1 --2- 172.21.15.90:0/3827044840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb26c1031c0 0x7fb26c19c0a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb25400c8f0 tx=0x7fb25400ccb0 comp rx=0 tx=0).stop 2024-04-25T11:34:17.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.357+0000 7fb270f8f700 1 -- 172.21.15.90:0/3827044840 shutdown_connections 2024-04-25T11:34:17.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.357+0000 7fb270f8f700 1 --2- 172.21.15.90:0/3827044840 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb2580afc10 0x7fb2580b2120 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:17.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.357+0000 7fb270f8f700 1 --2- 172.21.15.90:0/3827044840 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb26c1088a0 0x7fb26c19c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:17.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.357+0000 7fb270f8f700 1 --2- 172.21.15.90:0/3827044840 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb26c102870 0x7fb26c19bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:17.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.357+0000 7fb270f8f700 1 --2- 172.21.15.90:0/3827044840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb26c1031c0 0x7fb26c19c0a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:17.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.357+0000 7fb270f8f700 1 -- 172.21.15.90:0/3827044840 >> 172.21.15.90:0/3827044840 conn(0x7fb26c0fe3f0 msgr2=0x7fb26c106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:17.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.357+0000 7fb270f8f700 1 -- 172.21.15.90:0/3827044840 shutdown_connections 2024-04-25T11:34:17.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:17.357+0000 7fb270f8f700 1 -- 172.21.15.90:0/3827044840 wait complete. 2024-04-25T11:34:17.761 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645333 got 309237645337 for osd.11 2024-04-25T11:34:17.761 DEBUG:teuthology.parallel:result is None 2024-04-25T11:34:18.146 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-04-25T11:34:18.146 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 7]}]': finished 2024-04-25T11:34:18.146 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-04-25T11:34:18.146 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]': finished 2024-04-25T11:34:18.146 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 7]}]': finished 2024-04-25T11:34:18.146 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2024-04-25T11:34:18.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-04-25T11:34:18.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]': finished 2024-04-25T11:34:18.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [1, 9]}]': finished 2024-04-25T11:34:18.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:17 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-04-25T11:34:18.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:17 smithi090 ceph-mon[29575]: osdmap e78: 12 total, 12 up, 12 in 2024-04-25T11:34:18.147 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:17 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3827044840' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T11:34:18.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-04-25T11:34:18.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 7]}]': finished 2024-04-25T11:34:18.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-04-25T11:34:18.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]': finished 2024-04-25T11:34:18.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 7]}]': finished 2024-04-25T11:34:18.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2024-04-25T11:34:18.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-04-25T11:34:18.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]': finished 2024-04-25T11:34:18.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [1, 9]}]': finished 2024-04-25T11:34:18.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:17 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-04-25T11:34:18.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:17 smithi138 ceph-mon[32328]: osdmap e78: 12 total, 12 up, 12 in 2024-04-25T11:34:18.304 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:17 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3827044840' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T11:34:18.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-04-25T11:34:18.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 7]}]': finished 2024-04-25T11:34:18.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-04-25T11:34:18.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]': finished 2024-04-25T11:34:18.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 7]}]': finished 2024-04-25T11:34:18.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2024-04-25T11:34:18.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-04-25T11:34:18.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]': finished 2024-04-25T11:34:18.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [1, 9]}]': finished 2024-04-25T11:34:18.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:17 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-04-25T11:34:18.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:17 smithi156 ceph-mon[33045]: osdmap e78: 12 total, 12 up, 12 in 2024-04-25T11:34:18.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:17 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3827044840' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T11:34:18.465 INFO:tasks.cephadm.ceph_manager.ceph:need seq 283467841561 got 283467841567 for osd.10 2024-04-25T11:34:18.465 DEBUG:teuthology.parallel:result is None 2024-04-25T11:34:18.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.893+0000 7f284c935700 1 -- 172.21.15.90:0/3968641067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2848102a20 msgr2=0x7f2848102e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:18.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.893+0000 7f284c935700 1 --2- 172.21.15.90:0/3968641067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2848102a20 0x7f2848102e30 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f28380099c0 tx=0x7f2838009cd0 comp rx=0 tx=0).stop 2024-04-25T11:34:18.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.894+0000 7f284c935700 1 -- 172.21.15.90:0/3968641067 shutdown_connections 2024-04-25T11:34:18.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.894+0000 7f284c935700 1 --2- 172.21.15.90:0/3968641067 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2848109720 0x7f2848111080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:18.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.894+0000 7f284c935700 1 --2- 172.21.15.90:0/3968641067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2848102a20 0x7f2848102e30 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:18.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.894+0000 7f284c935700 1 --2- 172.21.15.90:0/3968641067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2848108dd0 0x7f28481091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:18.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.894+0000 7f284c935700 1 -- 172.21.15.90:0/3968641067 >> 172.21.15.90:0/3968641067 conn(0x7f28480fe5c0 msgr2=0x7f28481009e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:18.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.894+0000 7f284c935700 1 -- 172.21.15.90:0/3968641067 shutdown_connections 2024-04-25T11:34:18.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.894+0000 7f284c935700 1 -- 172.21.15.90:0/3968641067 wait complete. 2024-04-25T11:34:18.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.894+0000 7f284c935700 1 Processor -- start 2024-04-25T11:34:18.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.895+0000 7f284c935700 1 -- start start 2024-04-25T11:34:18.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.895+0000 7f284c935700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2848102a20 0x7f284819cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:18.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.895+0000 7f284c935700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2848108dd0 0x7f284819d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:18.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.895+0000 7f284c935700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2848109720 0x7f28481a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:18.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.895+0000 7f284c935700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2848114350 con 0x7f2848108dd0 2024-04-25T11:34:18.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.895+0000 7f284c935700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f28481141d0 con 0x7f2848102a20 2024-04-25T11:34:18.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.895+0000 7f284c935700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f28481144d0 con 0x7f2848109720 2024-04-25T11:34:18.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.895+0000 7f2845d9b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2848108dd0 0x7f284819d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:18.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.895+0000 7f2845d9b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2848108dd0 0x7f284819d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43878/0 (socket says 172.21.15.90:43878) 2024-04-25T11:34:18.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.895+0000 7f2845d9b700 1 -- 172.21.15.90:0/1804228752 learned_addr learned my addr 172.21.15.90:0/1804228752 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:18.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.895+0000 7f284659c700 1 --2- 172.21.15.90:0/1804228752 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2848102a20 0x7f284819cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:18.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.895+0000 7f2846d9d700 1 --2- 172.21.15.90:0/1804228752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2848109720 0x7f28481a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:18.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.895+0000 7f2845d9b700 1 -- 172.21.15.90:0/1804228752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2848109720 msgr2=0x7f28481a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:18.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.895+0000 7f2845d9b700 1 --2- 172.21.15.90:0/1804228752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2848109720 0x7f28481a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:18.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.895+0000 7f2845d9b700 1 -- 172.21.15.90:0/1804228752 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2848102a20 msgr2=0x7f284819cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:18.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.895+0000 7f2845d9b700 1 --2- 172.21.15.90:0/1804228752 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2848102a20 0x7f284819cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:18.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.895+0000 7f2845d9b700 1 -- 172.21.15.90:0/1804228752 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28481a1d90 con 0x7f2848108dd0 2024-04-25T11:34:18.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.895+0000 7f2846d9d700 1 --2- 172.21.15.90:0/1804228752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2848109720 0x7f28481a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:18.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.895+0000 7f2845d9b700 1 --2- 172.21.15.90:0/1804228752 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2848108dd0 0x7f284819d460 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f283c00d950 tx=0x7f283c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:18.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.896+0000 7f28377fe700 1 -- 172.21.15.90:0/1804228752 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f283c016070 con 0x7f2848108dd0 2024-04-25T11:34:18.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.896+0000 7f28377fe700 1 -- 172.21.15.90:0/1804228752 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f283c004830 con 0x7f2848108dd0 2024-04-25T11:34:18.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.896+0000 7f284c935700 1 -- 172.21.15.90:0/1804228752 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28481a2080 con 0x7f2848108dd0 2024-04-25T11:34:18.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.896+0000 7f28377fe700 1 -- 172.21.15.90:0/1804228752 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f283c005440 con 0x7f2848108dd0 2024-04-25T11:34:18.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.896+0000 7f284c935700 1 -- 172.21.15.90:0/1804228752 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28481a99a0 con 0x7f2848108dd0 2024-04-25T11:34:18.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.897+0000 7f284c935700 1 -- 172.21.15.90:0/1804228752 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2848051190 con 0x7f2848108dd0 2024-04-25T11:34:18.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.899+0000 7f28377fe700 1 -- 172.21.15.90:0/1804228752 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f283c0055e0 con 0x7f2848108dd0 2024-04-25T11:34:18.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.899+0000 7f28377fe700 1 --2- 172.21.15.90:0/1804228752 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f28300afc10 0x7f28300b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:18.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.899+0000 7f284659c700 1 --2- 172.21.15.90:0/1804228752 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f28300afc10 0x7f28300b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:18.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.900+0000 7f28377fe700 1 -- 172.21.15.90:0/1804228752 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f283c0defe0 con 0x7f2848108dd0 2024-04-25T11:34:18.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.900+0000 7f284659c700 1 --2- 172.21.15.90:0/1804228752 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f28300afc10 0x7f28300b2120 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2838003760 tx=0x7f283801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:18.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:18.904+0000 7f28377fe700 1 -- 172.21.15.90:0/1804228752 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f283c01f020 con 0x7f2848108dd0 2024-04-25T11:34:19.097 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:18 smithi090 ceph-mon[29575]: osdmap e79: 12 total, 12 up, 12 in 2024-04-25T11:34:19.098 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:18 smithi090 ceph-mon[29575]: pgmap v310: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s; 7.4 KiB/s, 0 objects/s recovering 2024-04-25T11:34:19.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:19.109+0000 7f284c935700 1 -- 172.21.15.90:0/1804228752 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f28481a9b70 con 0x7f2848108dd0 2024-04-25T11:34:19.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:19.109+0000 7f28377fe700 1 -- 172.21.15.90:0/1804228752 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f283c0a7ae0 con 0x7f2848108dd0 2024-04-25T11:34:19.098 INFO:teuthology.orchestra.run.smithi090.stdout:107374182465 2024-04-25T11:34:19.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:19.112+0000 7f284c935700 1 -- 172.21.15.90:0/1804228752 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f28300afc10 msgr2=0x7f28300b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:19.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:19.112+0000 7f284c935700 1 --2- 172.21.15.90:0/1804228752 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f28300afc10 0x7f28300b2120 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2838003760 tx=0x7f283801a040 comp rx=0 tx=0).stop 2024-04-25T11:34:19.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:19.112+0000 7f284c935700 1 -- 172.21.15.90:0/1804228752 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2848108dd0 msgr2=0x7f284819d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:19.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:19.112+0000 7f284c935700 1 --2- 172.21.15.90:0/1804228752 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2848108dd0 0x7f284819d460 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f283c00d950 tx=0x7f283c00dc60 comp rx=0 tx=0).stop 2024-04-25T11:34:19.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:19.112+0000 7f284c935700 1 -- 172.21.15.90:0/1804228752 shutdown_connections 2024-04-25T11:34:19.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:19.112+0000 7f284c935700 1 --2- 172.21.15.90:0/1804228752 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f28300afc10 0x7f28300b2120 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:19.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:19.112+0000 7f284c935700 1 --2- 172.21.15.90:0/1804228752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2848109720 0x7f28481a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:19.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:19.112+0000 7f284c935700 1 --2- 172.21.15.90:0/1804228752 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2848102a20 0x7f284819cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:19.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:19.112+0000 7f284c935700 1 --2- 172.21.15.90:0/1804228752 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2848108dd0 0x7f284819d460 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:19.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:19.112+0000 7f284c935700 1 -- 172.21.15.90:0/1804228752 >> 172.21.15.90:0/1804228752 conn(0x7f28480fe5c0 msgr2=0x7f2848110350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:19.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:19.112+0000 7f284c935700 1 -- 172.21.15.90:0/1804228752 shutdown_connections 2024-04-25T11:34:19.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:19.112+0000 7f284c935700 1 -- 172.21.15.90:0/1804228752 wait complete. 2024-04-25T11:34:19.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:18 smithi138 ceph-mon[32328]: osdmap e79: 12 total, 12 up, 12 in 2024-04-25T11:34:19.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:18 smithi138 ceph-mon[32328]: pgmap v310: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s; 7.4 KiB/s, 0 objects/s recovering 2024-04-25T11:34:19.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:18 smithi156 ceph-mon[33045]: osdmap e79: 12 total, 12 up, 12 in 2024-04-25T11:34:19.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:18 smithi156 ceph-mon[33045]: pgmap v310: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s; 7.4 KiB/s, 0 objects/s recovering 2024-04-25T11:34:19.888 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822706 got 154618822712 for osd.5 2024-04-25T11:34:19.889 DEBUG:teuthology.parallel:result is None 2024-04-25T11:34:20.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.094+0000 7f34744d4700 1 -- 172.21.15.90:0/113582595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f346c107cb0 msgr2=0x7f346c1080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:20.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.094+0000 7f34744d4700 1 --2- 172.21.15.90:0/113582595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f346c107cb0 0x7f346c1080c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f345c0099c0 tx=0x7f345c009cd0 comp rx=0 tx=0).stop 2024-04-25T11:34:20.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.094+0000 7f34744d4700 1 -- 172.21.15.90:0/113582595 shutdown_connections 2024-04-25T11:34:20.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.094+0000 7f34744d4700 1 --2- 172.21.15.90:0/113582595 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f346c108600 0x7f346c10ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:20.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.094+0000 7f34744d4700 1 --2- 172.21.15.90:0/113582595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f346c103380 0x7f346c103790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:20.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.094+0000 7f34744d4700 1 --2- 172.21.15.90:0/113582595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f346c107cb0 0x7f346c1080c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.094+0000 7f34744d4700 1 -- 172.21.15.90:0/113582595 >> 172.21.15.90:0/113582595 conn(0x7f346c0fecb0 msgr2=0x7f346c1010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.094+0000 7f34744d4700 1 -- 172.21.15.90:0/113582595 shutdown_connections 2024-04-25T11:34:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.094+0000 7f34744d4700 1 -- 172.21.15.90:0/113582595 wait complete. 2024-04-25T11:34:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.094+0000 7f34744d4700 1 Processor -- start 2024-04-25T11:34:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.094+0000 7f34744d4700 1 -- start start 2024-04-25T11:34:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.095+0000 7f34744d4700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f346c103380 0x7f346c19adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.095+0000 7f34744d4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f346c107cb0 0x7f346c19b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.095+0000 7f34744d4700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f346c108600 0x7f346c19f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.095+0000 7f34744d4700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f346c113280 con 0x7f346c107cb0 2024-04-25T11:34:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.095+0000 7f34744d4700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f346c113100 con 0x7f346c108600 2024-04-25T11:34:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.095+0000 7f34744d4700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f346c113400 con 0x7f346c103380 2024-04-25T11:34:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.095+0000 7f3471a6f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f346c107cb0 0x7f346c19b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.095+0000 7f3471a6f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f346c107cb0 0x7f346c19b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43898/0 (socket says 172.21.15.90:43898) 2024-04-25T11:34:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.095+0000 7f3471a6f700 1 -- 172.21.15.90:0/3677025573 learned_addr learned my addr 172.21.15.90:0/3677025573 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.095+0000 7f3472270700 1 --2- 172.21.15.90:0/3677025573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f346c103380 0x7f346c19adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.095+0000 7f3472a71700 1 --2- 172.21.15.90:0/3677025573 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f346c108600 0x7f346c19f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.095+0000 7f3471a6f700 1 -- 172.21.15.90:0/3677025573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f346c103380 msgr2=0x7f346c19adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.095+0000 7f3471a6f700 1 --2- 172.21.15.90:0/3677025573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f346c103380 0x7f346c19adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.095+0000 7f3471a6f700 1 -- 172.21.15.90:0/3677025573 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f346c108600 msgr2=0x7f346c19f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.095+0000 7f3471a6f700 1 --2- 172.21.15.90:0/3677025573 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f346c108600 0x7f346c19f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.095+0000 7f3471a6f700 1 -- 172.21.15.90:0/3677025573 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f346c19fbe0 con 0x7f346c107cb0 2024-04-25T11:34:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.095+0000 7f3472a71700 1 --2- 172.21.15.90:0/3677025573 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f346c108600 0x7f346c19f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.095+0000 7f3471a6f700 1 --2- 172.21.15.90:0/3677025573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f346c107cb0 0x7f346c19b360 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f345c0098e0 tx=0x7f345c0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.095+0000 7f34637fe700 1 -- 172.21.15.90:0/3677025573 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f345c027070 con 0x7f346c107cb0 2024-04-25T11:34:20.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.096+0000 7f34637fe700 1 -- 172.21.15.90:0/3677025573 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f345c004020 con 0x7f346c107cb0 2024-04-25T11:34:20.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.096+0000 7f34637fe700 1 -- 172.21.15.90:0/3677025573 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f345c022e30 con 0x7f346c107cb0 2024-04-25T11:34:20.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.096+0000 7f34744d4700 1 -- 172.21.15.90:0/3677025573 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f346c19fe70 con 0x7f346c107cb0 2024-04-25T11:34:20.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.096+0000 7f34744d4700 1 -- 172.21.15.90:0/3677025573 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f346c1a0350 con 0x7f346c107cb0 2024-04-25T11:34:20.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.096+0000 7f34744d4700 1 -- 172.21.15.90:0/3677025573 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f346c051190 con 0x7f346c107cb0 2024-04-25T11:34:20.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.098+0000 7f34637fe700 1 -- 172.21.15.90:0/3677025573 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f345c02a070 con 0x7f346c107cb0 2024-04-25T11:34:20.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.098+0000 7f34637fe700 1 --2- 172.21.15.90:0/3677025573 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f34580afc90 0x7f34580b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:20.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.099+0000 7f3472270700 1 --2- 172.21.15.90:0/3677025573 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f34580afc90 0x7f34580b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:20.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.099+0000 7f34637fe700 1 -- 172.21.15.90:0/3677025573 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f345c005070 con 0x7f346c107cb0 2024-04-25T11:34:20.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.099+0000 7f3472270700 1 --2- 172.21.15.90:0/3677025573 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f34580afc90 0x7f34580b21a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3468009a10 tx=0x7f3468009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:20.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.102+0000 7f34637fe700 1 -- 172.21.15.90:0/3677025573 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f345c013070 con 0x7f346c107cb0 2024-04-25T11:34:20.172 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:19 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1804228752' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T11:34:20.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.311+0000 7f34744d4700 1 -- 172.21.15.90:0/3677025573 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f346c02e510 con 0x7f346c107cb0 2024-04-25T11:34:20.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.312+0000 7f34637fe700 1 -- 172.21.15.90:0/3677025573 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f345c0a88f0 con 0x7f346c107cb0 2024-04-25T11:34:20.300 INFO:teuthology.orchestra.run.smithi090.stdout:257698037796 2024-04-25T11:34:20.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.314+0000 7f34744d4700 1 -- 172.21.15.90:0/3677025573 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f34580afc90 msgr2=0x7f34580b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:20.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.314+0000 7f34744d4700 1 --2- 172.21.15.90:0/3677025573 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f34580afc90 0x7f34580b21a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3468009a10 tx=0x7f3468009190 comp rx=0 tx=0).stop 2024-04-25T11:34:20.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.314+0000 7f34744d4700 1 -- 172.21.15.90:0/3677025573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f346c107cb0 msgr2=0x7f346c19b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:20.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.314+0000 7f34744d4700 1 --2- 172.21.15.90:0/3677025573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f346c107cb0 0x7f346c19b360 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f345c0098e0 tx=0x7f345c0057c0 comp rx=0 tx=0).stop 2024-04-25T11:34:20.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.314+0000 7f34744d4700 1 -- 172.21.15.90:0/3677025573 shutdown_connections 2024-04-25T11:34:20.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.314+0000 7f34744d4700 1 --2- 172.21.15.90:0/3677025573 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f346c108600 0x7f346c19f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:20.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.314+0000 7f34744d4700 1 --2- 172.21.15.90:0/3677025573 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f34580afc90 0x7f34580b21a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:20.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.314+0000 7f34744d4700 1 --2- 172.21.15.90:0/3677025573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f346c103380 0x7f346c19adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:20.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.314+0000 7f34744d4700 1 --2- 172.21.15.90:0/3677025573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f346c107cb0 0x7f346c19b360 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:20.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.314+0000 7f34744d4700 1 -- 172.21.15.90:0/3677025573 >> 172.21.15.90:0/3677025573 conn(0x7f346c0fecb0 msgr2=0x7f346c075c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:20.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.314+0000 7f34744d4700 1 -- 172.21.15.90:0/3677025573 shutdown_connections 2024-04-25T11:34:20.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.314+0000 7f34744d4700 1 -- 172.21.15.90:0/3677025573 wait complete. 2024-04-25T11:34:20.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:19 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1804228752' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T11:34:20.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.297+0000 7f05cb938700 1 -- 172.21.15.90:0/2345192080 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05c41038b0 msgr2=0x7f05c4103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:20.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.297+0000 7f05cb938700 1 --2- 172.21.15.90:0/2345192080 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05c41038b0 0x7f05c4103cc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f05b4009990 tx=0x7f05b4009ca0 comp rx=0 tx=0).stop 2024-04-25T11:34:20.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.297+0000 7f05cb938700 1 -- 172.21.15.90:0/2345192080 shutdown_connections 2024-04-25T11:34:20.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.297+0000 7f05cb938700 1 --2- 172.21.15.90:0/2345192080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05c4104200 0x7f05c410fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:20.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.297+0000 7f05cb938700 1 --2- 172.21.15.90:0/2345192080 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f05c41091a0 0x7f05c41095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:20.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.297+0000 7f05cb938700 1 --2- 172.21.15.90:0/2345192080 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05c41038b0 0x7f05c4103cc0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:20.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.297+0000 7f05cb938700 1 -- 172.21.15.90:0/2345192080 >> 172.21.15.90:0/2345192080 conn(0x7f05c40fe3f0 msgr2=0x7f05c4100840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:20.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.297+0000 7f05cb938700 1 -- 172.21.15.90:0/2345192080 shutdown_connections 2024-04-25T11:34:20.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.297+0000 7f05cb938700 1 -- 172.21.15.90:0/2345192080 wait complete. 2024-04-25T11:34:20.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.298+0000 7f05cb938700 1 Processor -- start 2024-04-25T11:34:20.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.298+0000 7f05cb938700 1 -- start start 2024-04-25T11:34:20.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.299+0000 7f05cb938700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05c41038b0 0x7f05c419f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:20.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.299+0000 7f05cb938700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f05c4104200 0x7f05c419fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:20.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.299+0000 7f05cb938700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05c41091a0 0x7f05c4199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:20.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.299+0000 7f05cb938700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f05c4113220 con 0x7f05c41091a0 2024-04-25T11:34:20.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.299+0000 7f05cb938700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f05c41130a0 con 0x7f05c4104200 2024-04-25T11:34:20.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.299+0000 7f05cb938700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f05c41133a0 con 0x7f05c41038b0 2024-04-25T11:34:20.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.299+0000 7f05c9ed5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05c41091a0 0x7f05c4199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:20.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.299+0000 7f05c9ed5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05c41091a0 0x7f05c4199a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43908/0 (socket says 172.21.15.90:43908) 2024-04-25T11:34:20.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.299+0000 7f05c9ed5700 1 -- 172.21.15.90:0/4189363022 learned_addr learned my addr 172.21.15.90:0/4189363022 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:20.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.299+0000 7f05c8ed3700 1 --2- 172.21.15.90:0/4189363022 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f05c4104200 0x7f05c419fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:20.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.299+0000 7f05c96d4700 1 --2- 172.21.15.90:0/4189363022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05c41038b0 0x7f05c419f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:20.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.299+0000 7f05c9ed5700 1 -- 172.21.15.90:0/4189363022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05c41038b0 msgr2=0x7f05c419f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:20.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.299+0000 7f05c9ed5700 1 --2- 172.21.15.90:0/4189363022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05c41038b0 0x7f05c419f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:20.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.299+0000 7f05c9ed5700 1 -- 172.21.15.90:0/4189363022 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f05c4104200 msgr2=0x7f05c419fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:20.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.299+0000 7f05c9ed5700 1 --2- 172.21.15.90:0/4189363022 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f05c4104200 0x7f05c419fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:20.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.299+0000 7f05c9ed5700 1 -- 172.21.15.90:0/4189363022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05c419a0b0 con 0x7f05c41091a0 2024-04-25T11:34:20.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.299+0000 7f05c96d4700 1 --2- 172.21.15.90:0/4189363022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05c41038b0 0x7f05c419f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:20.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.299+0000 7f05c8ed3700 1 --2- 172.21.15.90:0/4189363022 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f05c4104200 0x7f05c419fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:20.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.299+0000 7f05c9ed5700 1 --2- 172.21.15.90:0/4189363022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05c41091a0 0x7f05c4199a00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f05bc007ee0 tx=0x7f05bc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:20.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.300+0000 7f05ba7fc700 1 -- 172.21.15.90:0/4189363022 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05bc00dbd0 con 0x7f05c41091a0 2024-04-25T11:34:20.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.300+0000 7f05cb938700 1 -- 172.21.15.90:0/4189363022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05c419a3a0 con 0x7f05c41091a0 2024-04-25T11:34:20.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.300+0000 7f05ba7fc700 1 -- 172.21.15.90:0/4189363022 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05bc00dd70 con 0x7f05c41091a0 2024-04-25T11:34:20.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.300+0000 7f05ba7fc700 1 -- 172.21.15.90:0/4189363022 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05bc005990 con 0x7f05c41091a0 2024-04-25T11:34:20.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.300+0000 7f05cb938700 1 -- 172.21.15.90:0/4189363022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05c41a6670 con 0x7f05c41091a0 2024-04-25T11:34:20.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.301+0000 7f05cb938700 1 -- 172.21.15.90:0/4189363022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05c4051190 con 0x7f05c41091a0 2024-04-25T11:34:20.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.303+0000 7f05ba7fc700 1 -- 172.21.15.90:0/4189363022 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f05bc01a020 con 0x7f05c41091a0 2024-04-25T11:34:20.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.304+0000 7f05ba7fc700 1 --2- 172.21.15.90:0/4189363022 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f05b00afc10 0x7f05b00b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:20.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.304+0000 7f05c96d4700 1 --2- 172.21.15.90:0/4189363022 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f05b00afc10 0x7f05b00b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:20.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.304+0000 7f05ba7fc700 1 -- 172.21.15.90:0/4189363022 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f05bc0dec60 con 0x7f05c41091a0 2024-04-25T11:34:20.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.304+0000 7f05c96d4700 1 --2- 172.21.15.90:0/4189363022 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f05b00afc10 0x7f05b00b2120 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f05c0005e10 tx=0x7f05c0005d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:20.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.306+0000 7f05ba7fc700 1 -- 172.21.15.90:0/4189363022 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f05bc010070 con 0x7f05c41091a0 2024-04-25T11:34:20.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:19 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1804228752' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T11:34:20.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.511+0000 7f05cb938700 1 -- 172.21.15.90:0/4189363022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f05c419af30 con 0x7f05c41091a0 2024-04-25T11:34:20.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.512+0000 7f05ba7fc700 1 -- 172.21.15.90:0/4189363022 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f05bc0a76c0 con 0x7f05c41091a0 2024-04-25T11:34:20.500 INFO:teuthology.orchestra.run.smithi090.stdout:180388626483 2024-04-25T11:34:20.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.514+0000 7f05cb938700 1 -- 172.21.15.90:0/4189363022 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f05b00afc10 msgr2=0x7f05b00b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:20.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.514+0000 7f05cb938700 1 --2- 172.21.15.90:0/4189363022 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f05b00afc10 0x7f05b00b2120 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f05c0005e10 tx=0x7f05c0005d20 comp rx=0 tx=0).stop 2024-04-25T11:34:20.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.514+0000 7f05cb938700 1 -- 172.21.15.90:0/4189363022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05c41091a0 msgr2=0x7f05c4199a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:20.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.514+0000 7f05cb938700 1 --2- 172.21.15.90:0/4189363022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05c41091a0 0x7f05c4199a00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f05bc007ee0 tx=0x7f05bc00d3b0 comp rx=0 tx=0).stop 2024-04-25T11:34:20.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.514+0000 7f05cb938700 1 -- 172.21.15.90:0/4189363022 shutdown_connections 2024-04-25T11:34:20.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.514+0000 7f05cb938700 1 --2- 172.21.15.90:0/4189363022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05c41091a0 0x7f05c4199a00 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:20.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.514+0000 7f05cb938700 1 --2- 172.21.15.90:0/4189363022 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f05b00afc10 0x7f05b00b2120 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:20.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.514+0000 7f05cb938700 1 --2- 172.21.15.90:0/4189363022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05c41038b0 0x7f05c419f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:20.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.514+0000 7f05cb938700 1 --2- 172.21.15.90:0/4189363022 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f05c4104200 0x7f05c419fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:20.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.514+0000 7f05cb938700 1 -- 172.21.15.90:0/4189363022 >> 172.21.15.90:0/4189363022 conn(0x7f05c40fe3f0 msgr2=0x7f05c4100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:20.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.514+0000 7f05cb938700 1 -- 172.21.15.90:0/4189363022 shutdown_connections 2024-04-25T11:34:20.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:20.514+0000 7f05cb938700 1 -- 172.21.15.90:0/4189363022 wait complete. 2024-04-25T11:34:21.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:20 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3677025573' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T11:34:21.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:20 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/4189363022' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T11:34:21.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:20 smithi090 ceph-mon[29575]: pgmap v311: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s; 7.4 KiB/s, 0 objects/s recovering 2024-04-25T11:34:21.166 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626477 got 180388626483 for osd.6 2024-04-25T11:34:21.167 DEBUG:teuthology.parallel:result is None 2024-04-25T11:34:21.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:20 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3677025573' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T11:34:21.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:20 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/4189363022' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T11:34:21.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:20 smithi138 ceph-mon[32328]: pgmap v311: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s; 7.4 KiB/s, 0 objects/s recovering 2024-04-25T11:34:21.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:20 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3677025573' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T11:34:21.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:20 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/4189363022' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T11:34:21.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:20 smithi156 ceph-mon[33045]: pgmap v311: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s; 7.4 KiB/s, 0 objects/s recovering 2024-04-25T11:34:21.535 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037791 got 257698037796 for osd.9 2024-04-25T11:34:21.536 DEBUG:teuthology.parallel:result is None 2024-04-25T11:34:21.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.902+0000 7f0ec81c8700 1 -- 172.21.15.90:0/518080296 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ec0103360 msgr2=0x7f0ec0103770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:21.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.902+0000 7f0ec81c8700 1 --2- 172.21.15.90:0/518080296 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ec0103360 0x7f0ec0103770 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc0099c0 tx=0x7f0ebc009cd0 comp rx=0 tx=0).stop 2024-04-25T11:34:21.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.902+0000 7f0ec81c8700 1 -- 172.21.15.90:0/518080296 shutdown_connections 2024-04-25T11:34:21.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.902+0000 7f0ec81c8700 1 --2- 172.21.15.90:0/518080296 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ec01085e0 0x7f0ec010ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:21.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.902+0000 7f0ec81c8700 1 --2- 172.21.15.90:0/518080296 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ec0103360 0x7f0ec0103770 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:21.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.902+0000 7f0ec81c8700 1 --2- 172.21.15.90:0/518080296 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ec0107c90 0x7f0ec01080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:21.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.902+0000 7f0ec81c8700 1 -- 172.21.15.90:0/518080296 >> 172.21.15.90:0/518080296 conn(0x7f0ec00fec90 msgr2=0x7f0ec01010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:21.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.902+0000 7f0ec81c8700 1 -- 172.21.15.90:0/518080296 shutdown_connections 2024-04-25T11:34:21.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.902+0000 7f0ec81c8700 1 -- 172.21.15.90:0/518080296 wait complete. 2024-04-25T11:34:21.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.903+0000 7f0ec81c8700 1 Processor -- start 2024-04-25T11:34:21.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.903+0000 7f0ec81c8700 1 -- start start 2024-04-25T11:34:21.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.903+0000 7f0ec81c8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ec0103360 0x7f0ec019bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:21.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.903+0000 7f0ec81c8700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ec0107c90 0x7f0ec019c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:21.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.903+0000 7f0ec81c8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ec01085e0 0x7f0ec019c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:21.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.903+0000 7f0ec81c8700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec0113290 con 0x7f0ec0103360 2024-04-25T11:34:21.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.903+0000 7f0ec81c8700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec0113110 con 0x7f0ec0107c90 2024-04-25T11:34:21.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.903+0000 7f0ec81c8700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec0113410 con 0x7f0ec01085e0 2024-04-25T11:34:21.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.903+0000 7f0ec5f64700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ec0103360 0x7f0ec019bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:21.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.903+0000 7f0ec5f64700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ec0103360 0x7f0ec019bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57606/0 (socket says 172.21.15.90:57606) 2024-04-25T11:34:21.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.903+0000 7f0ec5f64700 1 -- 172.21.15.90:0/1068198061 learned_addr learned my addr 172.21.15.90:0/1068198061 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:21.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.903+0000 7f0ec5763700 1 --2- 172.21.15.90:0/1068198061 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ec0107c90 0x7f0ec019c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:21.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.904+0000 7f0ec5f64700 1 -- 172.21.15.90:0/1068198061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ec01085e0 msgr2=0x7f0ec019c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:21.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.904+0000 7f0ec5f64700 1 --2- 172.21.15.90:0/1068198061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ec01085e0 0x7f0ec019c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:21.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.904+0000 7f0ec5f64700 1 -- 172.21.15.90:0/1068198061 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ec0107c90 msgr2=0x7f0ec019c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:21.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.904+0000 7f0ec5f64700 1 --2- 172.21.15.90:0/1068198061 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ec0107c90 0x7f0ec019c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:21.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.904+0000 7f0ec5f64700 1 -- 172.21.15.90:0/1068198061 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ec01a6ee0 con 0x7f0ec0103360 2024-04-25T11:34:21.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.904+0000 7f0ec5f64700 1 --2- 172.21.15.90:0/1068198061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ec0103360 0x7f0ec019bb70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc00b580 tx=0x7f0ebc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:21.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.904+0000 7f0eb6ffd700 1 -- 172.21.15.90:0/1068198061 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc027070 con 0x7f0ec0103360 2024-04-25T11:34:21.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.904+0000 7f0ec81c8700 1 -- 172.21.15.90:0/1068198061 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ec01a7170 con 0x7f0ec0103360 2024-04-25T11:34:21.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.904+0000 7f0eb6ffd700 1 -- 172.21.15.90:0/1068198061 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ebc0040a0 con 0x7f0ec0103360 2024-04-25T11:34:21.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.904+0000 7f0eb6ffd700 1 -- 172.21.15.90:0/1068198061 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc00f460 con 0x7f0ec0103360 2024-04-25T11:34:21.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.904+0000 7f0ec81c8700 1 -- 172.21.15.90:0/1068198061 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ec01a75c0 con 0x7f0ec0103360 2024-04-25T11:34:21.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.905+0000 7f0ec81c8700 1 -- 172.21.15.90:0/1068198061 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ec0051190 con 0x7f0ec0103360 2024-04-25T11:34:21.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.907+0000 7f0eb6ffd700 1 -- 172.21.15.90:0/1068198061 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f0ebc02a070 con 0x7f0ec0103360 2024-04-25T11:34:21.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.907+0000 7f0eb6ffd700 1 --2- 172.21.15.90:0/1068198061 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0eac0afc10 0x7f0eac0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:21.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.907+0000 7f0ec5763700 1 --2- 172.21.15.90:0/1068198061 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0eac0afc10 0x7f0eac0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:21.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.907+0000 7f0eb6ffd700 1 -- 172.21.15.90:0/1068198061 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f0ebc0e0280 con 0x7f0ec0103360 2024-04-25T11:34:21.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.908+0000 7f0ec5763700 1 --2- 172.21.15.90:0/1068198061 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0eac0afc10 0x7f0eac0b2120 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0ec01064d0 tx=0x7f0eb000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:21.911+0000 7f0eb6ffd700 1 -- 172.21.15.90:0/1068198061 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0ebc031080 con 0x7f0ec0103360 2024-04-25T11:34:22.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:22.117+0000 7f0ec81c8700 1 -- 172.21.15.90:0/1068198061 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f0ec002e6d0 con 0x7f0ec0103360 2024-04-25T11:34:22.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:22.118+0000 7f0eb6ffd700 1 -- 172.21.15.90:0/1068198061 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0ebc013070 con 0x7f0ec0103360 2024-04-25T11:34:22.106 INFO:teuthology.orchestra.run.smithi090.stdout:55834574925 2024-04-25T11:34:22.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:22.120+0000 7f0ec81c8700 1 -- 172.21.15.90:0/1068198061 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0eac0afc10 msgr2=0x7f0eac0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:22.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:22.120+0000 7f0ec81c8700 1 --2- 172.21.15.90:0/1068198061 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0eac0afc10 0x7f0eac0b2120 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0ec01064d0 tx=0x7f0eb000a300 comp rx=0 tx=0).stop 2024-04-25T11:34:22.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:22.120+0000 7f0ec81c8700 1 -- 172.21.15.90:0/1068198061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ec0103360 msgr2=0x7f0ec019bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:22.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:22.120+0000 7f0ec81c8700 1 --2- 172.21.15.90:0/1068198061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ec0103360 0x7f0ec019bb70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc00b580 tx=0x7f0ebc0047b0 comp rx=0 tx=0).stop 2024-04-25T11:34:22.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:22.120+0000 7f0ec81c8700 1 -- 172.21.15.90:0/1068198061 shutdown_connections 2024-04-25T11:34:22.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:22.120+0000 7f0ec81c8700 1 --2- 172.21.15.90:0/1068198061 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0eac0afc10 0x7f0eac0b2120 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:22.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:22.120+0000 7f0ec81c8700 1 --2- 172.21.15.90:0/1068198061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ec01085e0 0x7f0ec019c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:22.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:22.120+0000 7f0ec81c8700 1 --2- 172.21.15.90:0/1068198061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ec0103360 0x7f0ec019bb70 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:22.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:22.120+0000 7f0ec81c8700 1 --2- 172.21.15.90:0/1068198061 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ec0107c90 0x7f0ec019c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:22.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:22.120+0000 7f0ec81c8700 1 -- 172.21.15.90:0/1068198061 >> 172.21.15.90:0/1068198061 conn(0x7f0ec00fec90 msgr2=0x7f0ec0101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:22.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:22.120+0000 7f0ec81c8700 1 -- 172.21.15.90:0/1068198061 shutdown_connections 2024-04-25T11:34:22.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:22.120+0000 7f0ec81c8700 1 -- 172.21.15.90:0/1068198061 wait complete. 2024-04-25T11:34:22.317 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182459 got 107374182465 for osd.3 2024-04-25T11:34:22.317 DEBUG:teuthology.parallel:result is None 2024-04-25T11:34:22.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:22 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1068198061' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T11:34:22.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:22 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1068198061' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T11:34:22.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:22 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1068198061' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T11:34:23.023 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574918 got 55834574925 for osd.1 2024-04-25T11:34:23.023 DEBUG:teuthology.parallel:result is None 2024-04-25T11:34:23.023 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-25T11:34:23.023 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T11:34:23.332 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:23 smithi090 ceph-mon[29575]: pgmap v312: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 6 op/s; 7.4 KiB/s, 0 objects/s recovering 2024-04-25T11:34:23.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:23 smithi138 ceph-mon[32328]: pgmap v312: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 6 op/s; 7.4 KiB/s, 0 objects/s recovering 2024-04-25T11:34:23.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:23 smithi156 ceph-mon[33045]: pgmap v312: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 6 op/s; 7.4 KiB/s, 0 objects/s recovering 2024-04-25T11:34:23.699 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:34:25.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.303+0000 7f6b9b80d700 1 -- 172.21.15.90:0/1494772441 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b94103390 msgr2=0x7f6b941037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:25.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.303+0000 7f6b9b80d700 1 --2- 172.21.15.90:0/1494772441 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b94103390 0x7f6b941037a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6b900099c0 tx=0x7f6b90009cd0 comp rx=0 tx=0).stop 2024-04-25T11:34:25.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.303+0000 7f6b9b80d700 1 -- 172.21.15.90:0/1494772441 shutdown_connections 2024-04-25T11:34:25.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.303+0000 7f6b9b80d700 1 --2- 172.21.15.90:0/1494772441 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b94108610 0x7f6b9410ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:25.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.303+0000 7f6b9b80d700 1 --2- 172.21.15.90:0/1494772441 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b94103390 0x7f6b941037a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:25.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.303+0000 7f6b9b80d700 1 --2- 172.21.15.90:0/1494772441 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b94107cc0 0x7f6b941080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:25.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.303+0000 7f6b9b80d700 1 -- 172.21.15.90:0/1494772441 >> 172.21.15.90:0/1494772441 conn(0x7f6b940fec90 msgr2=0x7f6b941010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:25.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.303+0000 7f6b9b80d700 1 -- 172.21.15.90:0/1494772441 shutdown_connections 2024-04-25T11:34:25.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.303+0000 7f6b9b80d700 1 -- 172.21.15.90:0/1494772441 wait complete. 2024-04-25T11:34:25.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.304+0000 7f6b9b80d700 1 Processor -- start 2024-04-25T11:34:25.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.304+0000 7f6b9b80d700 1 -- start start 2024-04-25T11:34:25.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.304+0000 7f6b9b80d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b94103390 0x7f6b940ff170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:25.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.304+0000 7f6b9b80d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b94107cc0 0x7f6b940ff6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:25.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.304+0000 7f6b9b80d700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b94108610 0x7f6b940ffbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:25.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.304+0000 7f6b9b80d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6b94113260 con 0x7f6b94103390 2024-04-25T11:34:25.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.304+0000 7f6b9b80d700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6b941130e0 con 0x7f6b94108610 2024-04-25T11:34:25.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.304+0000 7f6b9b80d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6b941133e0 con 0x7f6b94107cc0 2024-04-25T11:34:25.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.304+0000 7f6b995a9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b94103390 0x7f6b940ff170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:25.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.304+0000 7f6b995a9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b94103390 0x7f6b940ff170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57622/0 (socket says 172.21.15.90:57622) 2024-04-25T11:34:25.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.304+0000 7f6b995a9700 1 -- 172.21.15.90:0/2388076301 learned_addr learned my addr 172.21.15.90:0/2388076301 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:25.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.304+0000 7f6b99daa700 1 --2- 172.21.15.90:0/2388076301 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b94108610 0x7f6b940ffbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:25.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.304+0000 7f6b98da8700 1 --2- 172.21.15.90:0/2388076301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b94107cc0 0x7f6b940ff6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:25.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.304+0000 7f6b995a9700 1 -- 172.21.15.90:0/2388076301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b94107cc0 msgr2=0x7f6b940ff6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:25.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.304+0000 7f6b995a9700 1 --2- 172.21.15.90:0/2388076301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b94107cc0 0x7f6b940ff6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:25.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.304+0000 7f6b995a9700 1 -- 172.21.15.90:0/2388076301 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b94108610 msgr2=0x7f6b940ffbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:25.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.304+0000 7f6b995a9700 1 --2- 172.21.15.90:0/2388076301 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b94108610 0x7f6b940ffbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:25.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.304+0000 7f6b995a9700 1 -- 172.21.15.90:0/2388076301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b941a9010 con 0x7f6b94103390 2024-04-25T11:34:25.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.305+0000 7f6b995a9700 1 --2- 172.21.15.90:0/2388076301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b94103390 0x7f6b940ff170 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6b9000b580 tx=0x7f6b900037c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:25.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.305+0000 7f6b8a7fc700 1 -- 172.21.15.90:0/2388076301 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b90027070 con 0x7f6b94103390 2024-04-25T11:34:25.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.305+0000 7f6b8a7fc700 1 -- 172.21.15.90:0/2388076301 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b90004110 con 0x7f6b94103390 2024-04-25T11:34:25.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.305+0000 7f6b8a7fc700 1 -- 172.21.15.90:0/2388076301 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9000f460 con 0x7f6b94103390 2024-04-25T11:34:25.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.305+0000 7f6b9b80d700 1 -- 172.21.15.90:0/2388076301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b941a92a0 con 0x7f6b94103390 2024-04-25T11:34:25.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.306+0000 7f6b9b80d700 1 -- 172.21.15.90:0/2388076301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b941a96a0 con 0x7f6b94103390 2024-04-25T11:34:25.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.307+0000 7f6b9b80d700 1 -- 172.21.15.90:0/2388076301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b94051190 con 0x7f6b94103390 2024-04-25T11:34:25.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.308+0000 7f6b8a7fc700 1 -- 172.21.15.90:0/2388076301 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f6b9002a070 con 0x7f6b94103390 2024-04-25T11:34:25.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.308+0000 7f6b8a7fc700 1 --2- 172.21.15.90:0/2388076301 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6b800afc90 0x7f6b800b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:25.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.309+0000 7f6b98da8700 1 --2- 172.21.15.90:0/2388076301 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6b800afc90 0x7f6b800b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:25.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.309+0000 7f6b8a7fc700 1 -- 172.21.15.90:0/2388076301 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f6b900dfd20 con 0x7f6b94103390 2024-04-25T11:34:25.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.309+0000 7f6b98da8700 1 --2- 172.21.15.90:0/2388076301 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6b800afc90 0x7f6b800b21a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6b94102120 tx=0x7f6b84005cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:25.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.313+0000 7f6b8a7fc700 1 -- 172.21.15.90:0/2388076301 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6b90013070 con 0x7f6b94103390 2024-04-25T11:34:25.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.521+0000 7f6b9b80d700 1 -- 172.21.15.90:0/2388076301 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6b9402e580 con 0x7f6b800afc90 2024-04-25T11:34:25.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.529+0000 7f6b8a7fc700 1 -- 172.21.15.90:0/2388076301 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258742 (secure 0 0 0) 0x7f6b9402e580 con 0x7f6b800afc90 2024-04-25T11:34:25.521 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:34:25.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.531+0000 7f6b9b80d700 1 -- 172.21.15.90:0/2388076301 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6b800afc90 msgr2=0x7f6b800b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:25.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.531+0000 7f6b9b80d700 1 --2- 172.21.15.90:0/2388076301 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6b800afc90 0x7f6b800b21a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6b94102120 tx=0x7f6b84005cf0 comp rx=0 tx=0).stop 2024-04-25T11:34:25.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.531+0000 7f6b9b80d700 1 -- 172.21.15.90:0/2388076301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b94103390 msgr2=0x7f6b940ff170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:25.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.531+0000 7f6b9b80d700 1 --2- 172.21.15.90:0/2388076301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b94103390 0x7f6b940ff170 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6b9000b580 tx=0x7f6b900037c0 comp rx=0 tx=0).stop 2024-04-25T11:34:25.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.531+0000 7f6b9b80d700 1 -- 172.21.15.90:0/2388076301 shutdown_connections 2024-04-25T11:34:25.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.531+0000 7f6b9b80d700 1 --2- 172.21.15.90:0/2388076301 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b94108610 0x7f6b940ffbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:25.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.531+0000 7f6b9b80d700 1 --2- 172.21.15.90:0/2388076301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b94103390 0x7f6b940ff170 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:25.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.531+0000 7f6b9b80d700 1 --2- 172.21.15.90:0/2388076301 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6b800afc90 0x7f6b800b21a0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:25.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.531+0000 7f6b9b80d700 1 --2- 172.21.15.90:0/2388076301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b94107cc0 0x7f6b940ff6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:25.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.531+0000 7f6b9b80d700 1 -- 172.21.15.90:0/2388076301 >> 172.21.15.90:0/2388076301 conn(0x7f6b940fec90 msgr2=0x7f6b94106270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:25.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.532+0000 7f6b9b80d700 1 -- 172.21.15.90:0/2388076301 shutdown_connections 2024-04-25T11:34:25.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:25.532+0000 7f6b9b80d700 1 -- 172.21.15.90:0/2388076301 wait complete. 2024-04-25T11:34:25.525 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-25T11:34:26.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:25 smithi156 ceph-mon[33045]: pgmap v313: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s; 7.4 KiB/s, 0 objects/s recovering 2024-04-25T11:34:26.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:25 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:34:26.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:25 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:34:26.203 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:25 smithi090 ceph-mon[29575]: pgmap v313: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s; 7.4 KiB/s, 0 objects/s recovering 2024-04-25T11:34:26.203 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:25 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:34:26.203 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:25 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:34:26.232 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-04-25T11:34:24.761383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658461,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":795,"num_read_kb":503,"num_write":361,"num_write_kb":2284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":3012886,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":299,"ondisk_log_size":299,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":295,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":382376,"kb_used_data":12348,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124559448,"statfs":{"total":1151940427776,"available":1151548874752,"internally_reserved":0,"allocated":12644352,"data_stored":6927459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":46,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002634"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.881195+0000","last_change":"2024-04-25T11:33:10.892144+0000","last_active":"2024-04-25T11:34:17.881195+0000","last_peered":"2024-04-25T11:34:17.881195+0000","last_clean":"2024-04-25T11:34:17.881195+0000","last_became_active":"2024-04-25T11:33:10.891721+0000","last_became_peered":"2024-04-25T11:33:10.891721+0000","last_unstale":"2024-04-25T11:34:17.881195+0000","last_undegraded":"2024-04-25T11:34:17.881195+0000","last_fullsized":"2024-04-25T11:34:17.881195+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:58:25.447086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.879751+0000","last_change":"2024-04-25T11:33:10.891647+0000","last_active":"2024-04-25T11:34:17.879751+0000","last_peered":"2024-04-25T11:34:17.879751+0000","last_clean":"2024-04-25T11:34:17.879751+0000","last_became_active":"2024-04-25T11:33:10.891283+0000","last_became_peered":"2024-04-25T11:33:10.891283+0000","last_unstale":"2024-04-25T11:34:17.879751+0000","last_undegraded":"2024-04-25T11:34:17.879751+0000","last_fullsized":"2024-04-25T11:34:17.879751+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:12.669438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.879623+0000","last_change":"2024-04-25T11:33:10.890928+0000","last_active":"2024-04-25T11:34:17.879623+0000","last_peered":"2024-04-25T11:34:17.879623+0000","last_clean":"2024-04-25T11:34:17.879623+0000","last_became_active":"2024-04-25T11:33:10.889093+0000","last_became_peered":"2024-04-25T11:33:10.889093+0000","last_unstale":"2024-04-25T11:34:17.879623+0000","last_undegraded":"2024-04-25T11:34:17.879623+0000","last_fullsized":"2024-04-25T11:34:17.879623+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:28:09.038650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.298945+0000","last_change":"2024-04-25T11:33:10.890184+0000","last_active":"2024-04-25T11:34:18.298945+0000","last_peered":"2024-04-25T11:34:18.298945+0000","last_clean":"2024-04-25T11:34:18.298945+0000","last_became_active":"2024-04-25T11:33:10.889367+0000","last_became_peered":"2024-04-25T11:33:10.889367+0000","last_unstale":"2024-04-25T11:34:18.298945+0000","last_undegraded":"2024-04-25T11:34:18.298945+0000","last_fullsized":"2024-04-25T11:34:18.298945+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:42.798295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.235974+0000","last_change":"2024-04-25T11:33:10.889758+0000","last_active":"2024-04-25T11:34:18.235974+0000","last_peered":"2024-04-25T11:34:18.235974+0000","last_clean":"2024-04-25T11:34:18.235974+0000","last_became_active":"2024-04-25T11:33:10.889384+0000","last_became_peered":"2024-04-25T11:33:10.889384+0000","last_unstale":"2024-04-25T11:34:18.235974+0000","last_undegraded":"2024-04-25T11:34:18.235974+0000","last_fullsized":"2024-04-25T11:34:18.235974+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:54:38.794216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.878863+0000","last_change":"2024-04-25T11:33:10.889348+0000","last_active":"2024-04-25T11:34:17.878863+0000","last_peered":"2024-04-25T11:34:17.878863+0000","last_clean":"2024-04-25T11:34:17.878863+0000","last_became_active":"2024-04-25T11:33:10.888908+0000","last_became_peered":"2024-04-25T11:33:10.888908+0000","last_unstale":"2024-04-25T11:34:17.878863+0000","last_undegraded":"2024-04-25T11:34:17.878863+0000","last_fullsized":"2024-04-25T11:34:17.878863+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:02:59.555921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":23,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.237661+0000","last_change":"2024-04-25T11:34:18.237661+0000","last_active":"2024-04-25T11:34:18.237661+0000","last_peered":"2024-04-25T11:34:18.237661+0000","last_clean":"2024-04-25T11:34:18.237661+0000","last_became_active":"2024-04-25T11:34:18.237306+0000","last_became_peered":"2024-04-25T11:34:18.237306+0000","last_unstale":"2024-04-25T11:34:18.237661+0000","last_undegraded":"2024-04-25T11:34:18.237661+0000","last_fullsized":"2024-04-25T11:34:18.237661+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:17:34.228368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.875134+0000","last_change":"2024-04-25T11:33:10.887262+0000","last_active":"2024-04-25T11:34:17.875134+0000","last_peered":"2024-04-25T11:34:17.875134+0000","last_clean":"2024-04-25T11:34:17.875134+0000","last_became_active":"2024-04-25T11:33:10.886948+0000","last_became_peered":"2024-04-25T11:33:10.886948+0000","last_unstale":"2024-04-25T11:34:17.875134+0000","last_undegraded":"2024-04-25T11:34:17.875134+0000","last_fullsized":"2024-04-25T11:34:17.875134+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:09.721506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.883852+0000","last_change":"2024-04-25T11:34:17.883253+0000","last_active":"2024-04-25T11:34:17.883852+0000","last_peered":"2024-04-25T11:34:17.883852+0000","last_clean":"2024-04-25T11:34:17.883852+0000","last_became_active":"2024-04-25T11:34:17.882515+0000","last_became_peered":"2024-04-25T11:34:17.882515+0000","last_unstale":"2024-04-25T11:34:17.883852+0000","last_undegraded":"2024-04-25T11:34:17.883852+0000","last_fullsized":"2024-04-25T11:34:17.883852+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:52:32.813464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.879418+0000","last_change":"2024-04-25T11:33:10.889812+0000","last_active":"2024-04-25T11:34:17.879418+0000","last_peered":"2024-04-25T11:34:17.879418+0000","last_clean":"2024-04-25T11:34:17.879418+0000","last_became_active":"2024-04-25T11:33:10.889369+0000","last_became_peered":"2024-04-25T11:33:10.889369+0000","last_unstale":"2024-04-25T11:34:17.879418+0000","last_undegraded":"2024-04-25T11:34:17.879418+0000","last_fullsized":"2024-04-25T11:34:17.879418+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:42:14.443496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880749+0000","last_change":"2024-04-25T11:33:10.891186+0000","last_active":"2024-04-25T11:34:17.880749+0000","last_peered":"2024-04-25T11:34:17.880749+0000","last_clean":"2024-04-25T11:34:17.880749+0000","last_became_active":"2024-04-25T11:33:10.890841+0000","last_became_peered":"2024-04-25T11:33:10.890841+0000","last_unstale":"2024-04-25T11:34:17.880749+0000","last_undegraded":"2024-04-25T11:34:17.880749+0000","last_fullsized":"2024-04-25T11:34:17.880749+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:24:56.078592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.298870+0000","last_change":"2024-04-25T11:33:10.888801+0000","last_active":"2024-04-25T11:34:18.298870+0000","last_peered":"2024-04-25T11:34:18.298870+0000","last_clean":"2024-04-25T11:34:18.298870+0000","last_became_active":"2024-04-25T11:33:10.888437+0000","last_became_peered":"2024-04-25T11:33:10.888437+0000","last_unstale":"2024-04-25T11:34:18.298870+0000","last_undegraded":"2024-04-25T11:34:18.298870+0000","last_fullsized":"2024-04-25T11:34:18.298870+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:14:14.011475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.877702+0000","last_change":"2024-04-25T11:33:10.892749+0000","last_active":"2024-04-25T11:34:17.877702+0000","last_peered":"2024-04-25T11:34:17.877702+0000","last_clean":"2024-04-25T11:34:17.877702+0000","last_became_active":"2024-04-25T11:33:10.892370+0000","last_became_peered":"2024-04-25T11:33:10.892370+0000","last_unstale":"2024-04-25T11:34:17.877702+0000","last_undegraded":"2024-04-25T11:34:17.877702+0000","last_fullsized":"2024-04-25T11:34:17.877702+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:00:51.312387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.879027+0000","last_change":"2024-04-25T11:33:10.888981+0000","last_active":"2024-04-25T11:34:17.879027+0000","last_peered":"2024-04-25T11:34:17.879027+0000","last_clean":"2024-04-25T11:34:17.879027+0000","last_became_active":"2024-04-25T11:33:10.888537+0000","last_became_peered":"2024-04-25T11:33:10.888537+0000","last_unstale":"2024-04-25T11:34:17.879027+0000","last_undegraded":"2024-04-25T11:34:17.879027+0000","last_fullsized":"2024-04-25T11:34:17.879027+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:12:19.018698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.879026+0000","last_change":"2024-04-25T11:33:10.891455+0000","last_active":"2024-04-25T11:34:17.879026+0000","last_peered":"2024-04-25T11:34:17.879026+0000","last_clean":"2024-04-25T11:34:17.879026+0000","last_became_active":"2024-04-25T11:33:10.891037+0000","last_became_peered":"2024-04-25T11:33:10.891037+0000","last_unstale":"2024-04-25T11:34:17.879026+0000","last_undegraded":"2024-04-25T11:34:17.879026+0000","last_fullsized":"2024-04-25T11:34:17.879026+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:44.470674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876205+0000","last_change":"2024-04-25T11:33:10.895309+0000","last_active":"2024-04-25T11:34:17.876205+0000","last_peered":"2024-04-25T11:34:17.876205+0000","last_clean":"2024-04-25T11:34:17.876205+0000","last_became_active":"2024-04-25T11:33:10.894952+0000","last_became_peered":"2024-04-25T11:33:10.894952+0000","last_unstale":"2024-04-25T11:34:17.876205+0000","last_undegraded":"2024-04-25T11:34:17.876205+0000","last_fullsized":"2024-04-25T11:34:17.876205+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:29:22.119017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.881501+0000","last_change":"2024-04-25T11:33:10.894524+0000","last_active":"2024-04-25T11:34:17.881501+0000","last_peered":"2024-04-25T11:34:17.881501+0000","last_clean":"2024-04-25T11:34:17.881501+0000","last_became_active":"2024-04-25T11:33:10.894220+0000","last_became_peered":"2024-04-25T11:33:10.894220+0000","last_unstale":"2024-04-25T11:34:17.881501+0000","last_undegraded":"2024-04-25T11:34:17.881501+0000","last_fullsized":"2024-04-25T11:34:17.881501+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:10:29.072976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.878215+0000","last_change":"2024-04-25T11:33:10.886814+0000","last_active":"2024-04-25T11:34:17.878215+0000","last_peered":"2024-04-25T11:34:17.878215+0000","last_clean":"2024-04-25T11:34:17.878215+0000","last_became_active":"2024-04-25T11:33:10.886494+0000","last_became_peered":"2024-04-25T11:33:10.886494+0000","last_unstale":"2024-04-25T11:34:17.878215+0000","last_undegraded":"2024-04-25T11:34:17.878215+0000","last_fullsized":"2024-04-25T11:34:17.878215+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:03:43.054485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876873+0000","last_change":"2024-04-25T11:33:10.886870+0000","last_active":"2024-04-25T11:34:17.876873+0000","last_peered":"2024-04-25T11:34:17.876873+0000","last_clean":"2024-04-25T11:34:17.876873+0000","last_became_active":"2024-04-25T11:33:10.886507+0000","last_became_peered":"2024-04-25T11:33:10.886507+0000","last_unstale":"2024-04-25T11:34:17.876873+0000","last_undegraded":"2024-04-25T11:34:17.876873+0000","last_fullsized":"2024-04-25T11:34:17.876873+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:50.101760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"77'13","reported_seq":100,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:21.740592+0000","last_change":"2024-04-25T11:33:10.028970+0000","last_active":"2024-04-25T11:34:21.740592+0000","last_peered":"2024-04-25T11:34:21.740592+0000","last_clean":"2024-04-25T11:34:21.740592+0000","last_became_active":"2024-04-25T11:33:10.028636+0000","last_became_peered":"2024-04-25T11:33:10.028636+0000","last_unstale":"2024-04-25T11:34:21.740592+0000","last_undegraded":"2024-04-25T11:34:21.740592+0000","last_fullsized":"2024-04-25T11:34:21.740592+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:02.414903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":68,"num_read_kb":68,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.882002+0000","last_change":"2024-04-25T11:34:17.881522+0000","last_active":"2024-04-25T11:34:17.882002+0000","last_peered":"2024-04-25T11:34:17.882002+0000","last_clean":"2024-04-25T11:34:17.882002+0000","last_became_active":"2024-04-25T11:34:17.881114+0000","last_became_peered":"2024-04-25T11:34:17.881114+0000","last_unstale":"2024-04-25T11:34:17.882002+0000","last_undegraded":"2024-04-25T11:34:17.882002+0000","last_fullsized":"2024-04-25T11:34:17.882002+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:59:03.675054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"77'2","reported_seq":21,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.875563+0000","last_change":"2024-04-25T11:33:10.036958+0000","last_active":"2024-04-25T11:34:17.875563+0000","last_peered":"2024-04-25T11:34:17.875563+0000","last_clean":"2024-04-25T11:34:17.875563+0000","last_became_active":"2024-04-25T11:33:10.036606+0000","last_became_peered":"2024-04-25T11:33:10.036606+0000","last_unstale":"2024-04-25T11:34:17.875563+0000","last_undegraded":"2024-04-25T11:34:17.875563+0000","last_fullsized":"2024-04-25T11:34:17.875563+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:12.660208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.879784+0000","last_change":"2024-04-25T11:33:10.887501+0000","last_active":"2024-04-25T11:34:17.879784+0000","last_peered":"2024-04-25T11:34:17.879784+0000","last_clean":"2024-04-25T11:34:17.879784+0000","last_became_active":"2024-04-25T11:33:10.885985+0000","last_became_peered":"2024-04-25T11:33:10.885985+0000","last_unstale":"2024-04-25T11:34:17.879784+0000","last_undegraded":"2024-04-25T11:34:17.879784+0000","last_fullsized":"2024-04-25T11:34:17.879784+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:31:32.035646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"77'12","reported_seq":32,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.236511+0000","last_change":"2024-04-25T11:33:10.031161+0000","last_active":"2024-04-25T11:34:18.236511+0000","last_peered":"2024-04-25T11:34:18.236511+0000","last_clean":"2024-04-25T11:34:18.236511+0000","last_became_active":"2024-04-25T11:33:10.028891+0000","last_became_peered":"2024-04-25T11:33:10.028891+0000","last_unstale":"2024-04-25T11:34:18.236511+0000","last_undegraded":"2024-04-25T11:34:18.236511+0000","last_fullsized":"2024-04-25T11:34:18.236511+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:25.959699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876332+0000","last_change":"2024-04-25T11:33:10.889439+0000","last_active":"2024-04-25T11:34:17.876332+0000","last_peered":"2024-04-25T11:34:17.876332+0000","last_clean":"2024-04-25T11:34:17.876332+0000","last_became_active":"2024-04-25T11:33:10.889082+0000","last_became_peered":"2024-04-25T11:33:10.889082+0000","last_unstale":"2024-04-25T11:34:17.876332+0000","last_undegraded":"2024-04-25T11:34:17.876332+0000","last_fullsized":"2024-04-25T11:34:17.876332+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:13:48.958179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"77'5","reported_seq":25,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.236179+0000","last_change":"2024-04-25T11:33:10.030793+0000","last_active":"2024-04-25T11:34:18.236179+0000","last_peered":"2024-04-25T11:34:18.236179+0000","last_clean":"2024-04-25T11:34:18.236179+0000","last_became_active":"2024-04-25T11:33:10.028805+0000","last_became_peered":"2024-04-25T11:33:10.028805+0000","last_unstale":"2024-04-25T11:34:18.236179+0000","last_undegraded":"2024-04-25T11:34:18.236179+0000","last_fullsized":"2024-04-25T11:34:18.236179+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:36.233194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880042+0000","last_change":"2024-04-25T11:33:10.894815+0000","last_active":"2024-04-25T11:34:17.880042+0000","last_peered":"2024-04-25T11:34:17.880042+0000","last_clean":"2024-04-25T11:34:17.880042+0000","last_became_active":"2024-04-25T11:33:10.892174+0000","last_became_peered":"2024-04-25T11:33:10.892174+0000","last_unstale":"2024-04-25T11:34:17.880042+0000","last_undegraded":"2024-04-25T11:34:17.880042+0000","last_fullsized":"2024-04-25T11:34:17.880042+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:05.368441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"77'5","reported_seq":97,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:22.719984+0000","last_change":"2024-04-25T11:33:10.031238+0000","last_active":"2024-04-25T11:34:22.719984+0000","last_peered":"2024-04-25T11:34:22.719984+0000","last_clean":"2024-04-25T11:34:22.719984+0000","last_became_active":"2024-04-25T11:33:10.030917+0000","last_became_peered":"2024-04-25T11:33:10.030917+0000","last_unstale":"2024-04-25T11:34:22.719984+0000","last_undegraded":"2024-04-25T11:34:22.719984+0000","last_fullsized":"2024-04-25T11:34:22.719984+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:00:35.634293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.235953+0000","last_change":"2024-04-25T11:33:10.896974+0000","last_active":"2024-04-25T11:34:18.235953+0000","last_peered":"2024-04-25T11:34:18.235953+0000","last_clean":"2024-04-25T11:34:18.235953+0000","last_became_active":"2024-04-25T11:33:10.896575+0000","last_became_peered":"2024-04-25T11:33:10.896575+0000","last_unstale":"2024-04-25T11:34:18.235953+0000","last_undegraded":"2024-04-25T11:34:18.235953+0000","last_fullsized":"2024-04-25T11:34:18.235953+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:32:43.533652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"77'7","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.878710+0000","last_change":"2024-04-25T11:33:10.022714+0000","last_active":"2024-04-25T11:34:17.878710+0000","last_peered":"2024-04-25T11:34:17.878710+0000","last_clean":"2024-04-25T11:34:17.878710+0000","last_became_active":"2024-04-25T11:33:10.022230+0000","last_became_peered":"2024-04-25T11:33:10.022230+0000","last_unstale":"2024-04-25T11:34:17.878710+0000","last_undegraded":"2024-04-25T11:34:17.878710+0000","last_fullsized":"2024-04-25T11:34:17.878710+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:30:00.620833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.878993+0000","last_change":"2024-04-25T11:33:10.897664+0000","last_active":"2024-04-25T11:34:17.878993+0000","last_peered":"2024-04-25T11:34:17.878993+0000","last_clean":"2024-04-25T11:34:17.878993+0000","last_became_active":"2024-04-25T11:33:10.897189+0000","last_became_peered":"2024-04-25T11:33:10.897189+0000","last_unstale":"2024-04-25T11:34:17.878993+0000","last_undegraded":"2024-04-25T11:34:17.878993+0000","last_fullsized":"2024-04-25T11:34:17.878993+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:12:06.168793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"77'3","reported_seq":22,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.877103+0000","last_change":"2024-04-25T11:33:10.031417+0000","last_active":"2024-04-25T11:34:17.877103+0000","last_peered":"2024-04-25T11:34:17.877103+0000","last_clean":"2024-04-25T11:34:17.877103+0000","last_became_active":"2024-04-25T11:33:10.030526+0000","last_became_peered":"2024-04-25T11:33:10.030526+0000","last_unstale":"2024-04-25T11:34:17.877103+0000","last_undegraded":"2024-04-25T11:34:17.877103+0000","last_fullsized":"2024-04-25T11:34:17.877103+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:05.718720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.875014+0000","last_change":"2024-04-25T11:33:10.886596+0000","last_active":"2024-04-25T11:34:17.875014+0000","last_peered":"2024-04-25T11:34:17.875014+0000","last_clean":"2024-04-25T11:34:17.875014+0000","last_became_active":"2024-04-25T11:33:10.886234+0000","last_became_peered":"2024-04-25T11:33:10.886234+0000","last_unstale":"2024-04-25T11:34:17.875014+0000","last_undegraded":"2024-04-25T11:34:17.875014+0000","last_fullsized":"2024-04-25T11:34:17.875014+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:46.632814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"77'3","reported_seq":23,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.879958+0000","last_change":"2024-04-25T11:33:10.032355+0000","last_active":"2024-04-25T11:34:17.879958+0000","last_peered":"2024-04-25T11:34:17.879958+0000","last_clean":"2024-04-25T11:34:17.879958+0000","last_became_active":"2024-04-25T11:33:10.031965+0000","last_became_peered":"2024-04-25T11:33:10.031965+0000","last_unstale":"2024-04-25T11:34:17.879958+0000","last_undegraded":"2024-04-25T11:34:17.879958+0000","last_fullsized":"2024-04-25T11:34:17.879958+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:08:10.539604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.877282+0000","last_change":"2024-04-25T11:33:10.885708+0000","last_active":"2024-04-25T11:34:17.877282+0000","last_peered":"2024-04-25T11:34:17.877282+0000","last_clean":"2024-04-25T11:34:17.877282+0000","last_became_active":"2024-04-25T11:33:10.885317+0000","last_became_peered":"2024-04-25T11:33:10.885317+0000","last_unstale":"2024-04-25T11:34:17.877282+0000","last_undegraded":"2024-04-25T11:34:17.877282+0000","last_fullsized":"2024-04-25T11:34:17.877282+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:50:08.518922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"77'4","reported_seq":161,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:22.742809+0000","last_change":"2024-04-25T11:33:10.030995+0000","last_active":"2024-04-25T11:34:22.742809+0000","last_peered":"2024-04-25T11:34:22.742809+0000","last_clean":"2024-04-25T11:34:22.742809+0000","last_became_active":"2024-04-25T11:33:10.030670+0000","last_became_peered":"2024-04-25T11:33:10.030670+0000","last_unstale":"2024-04-25T11:34:22.742809+0000","last_undegraded":"2024-04-25T11:34:22.742809+0000","last_fullsized":"2024-04-25T11:34:22.742809+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:39.569787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876806+0000","last_change":"2024-04-25T11:33:10.892927+0000","last_active":"2024-04-25T11:34:17.876806+0000","last_peered":"2024-04-25T11:34:17.876806+0000","last_clean":"2024-04-25T11:34:17.876806+0000","last_became_active":"2024-04-25T11:33:10.892458+0000","last_became_peered":"2024-04-25T11:33:10.892458+0000","last_unstale":"2024-04-25T11:34:17.876806+0000","last_undegraded":"2024-04-25T11:34:17.876806+0000","last_fullsized":"2024-04-25T11:34:17.876806+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:53.121185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"77'4","reported_seq":23,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880660+0000","last_change":"2024-04-25T11:33:10.029728+0000","last_active":"2024-04-25T11:34:17.880660+0000","last_peered":"2024-04-25T11:34:17.880660+0000","last_clean":"2024-04-25T11:34:17.880660+0000","last_became_active":"2024-04-25T11:33:10.029435+0000","last_became_peered":"2024-04-25T11:33:10.029435+0000","last_unstale":"2024-04-25T11:34:17.880660+0000","last_undegraded":"2024-04-25T11:34:17.880660+0000","last_fullsized":"2024-04-25T11:34:17.880660+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:31:21.585870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.879781+0000","last_change":"2024-04-25T11:33:10.895828+0000","last_active":"2024-04-25T11:34:17.879781+0000","last_peered":"2024-04-25T11:34:17.879781+0000","last_clean":"2024-04-25T11:34:17.879781+0000","last_became_active":"2024-04-25T11:33:10.895280+0000","last_became_peered":"2024-04-25T11:33:10.895280+0000","last_unstale":"2024-04-25T11:34:17.879781+0000","last_undegraded":"2024-04-25T11:34:17.879781+0000","last_fullsized":"2024-04-25T11:34:17.879781+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:03.009912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"77'5","reported_seq":25,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880204+0000","last_change":"2024-04-25T11:33:10.036651+0000","last_active":"2024-04-25T11:34:17.880204+0000","last_peered":"2024-04-25T11:34:17.880204+0000","last_clean":"2024-04-25T11:34:17.880204+0000","last_became_active":"2024-04-25T11:33:10.036294+0000","last_became_peered":"2024-04-25T11:33:10.036294+0000","last_unstale":"2024-04-25T11:34:17.880204+0000","last_undegraded":"2024-04-25T11:34:17.880204+0000","last_fullsized":"2024-04-25T11:34:17.880204+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:12:36.566618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876482+0000","last_change":"2024-04-25T11:33:10.890199+0000","last_active":"2024-04-25T11:34:17.876482+0000","last_peered":"2024-04-25T11:34:17.876482+0000","last_clean":"2024-04-25T11:34:17.876482+0000","last_became_active":"2024-04-25T11:33:10.889855+0000","last_became_peered":"2024-04-25T11:33:10.889855+0000","last_unstale":"2024-04-25T11:34:17.876482+0000","last_undegraded":"2024-04-25T11:34:17.876482+0000","last_fullsized":"2024-04-25T11:34:17.876482+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:54:45.476647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"77'30","reported_seq":218,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:23.311060+0000","last_change":"2024-04-25T11:34:18.309247+0000","last_active":"2024-04-25T11:34:23.311060+0000","last_peered":"2024-04-25T11:34:23.311060+0000","last_clean":"2024-04-25T11:34:23.311060+0000","last_became_active":"2024-04-25T11:34:18.299888+0000","last_became_peered":"2024-04-25T11:34:18.299888+0000","last_unstale":"2024-04-25T11:34:23.311060+0000","last_undegraded":"2024-04-25T11:34:23.311060+0000","last_fullsized":"2024-04-25T11:34:23.311060+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:36:06.603720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61046,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":143,"num_read_kb":131,"num_write":32,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.236286+0000","last_change":"2024-04-25T11:33:10.888596+0000","last_active":"2024-04-25T11:34:18.236286+0000","last_peered":"2024-04-25T11:34:18.236286+0000","last_clean":"2024-04-25T11:34:18.236286+0000","last_became_active":"2024-04-25T11:33:10.888226+0000","last_became_peered":"2024-04-25T11:33:10.888226+0000","last_unstale":"2024-04-25T11:34:18.236286+0000","last_undegraded":"2024-04-25T11:34:18.236286+0000","last_fullsized":"2024-04-25T11:34:18.236286+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:30:42.891788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"77'9","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876570+0000","last_change":"2024-04-25T11:33:10.026969+0000","last_active":"2024-04-25T11:34:17.876570+0000","last_peered":"2024-04-25T11:34:17.876570+0000","last_clean":"2024-04-25T11:34:17.876570+0000","last_became_active":"2024-04-25T11:33:10.026591+0000","last_became_peered":"2024-04-25T11:33:10.026591+0000","last_unstale":"2024-04-25T11:34:17.876570+0000","last_undegraded":"2024-04-25T11:34:17.876570+0000","last_fullsized":"2024-04-25T11:34:17.876570+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:50:34.269478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876927+0000","last_change":"2024-04-25T11:33:10.900303+0000","last_active":"2024-04-25T11:34:17.876927+0000","last_peered":"2024-04-25T11:34:17.876927+0000","last_clean":"2024-04-25T11:34:17.876927+0000","last_became_active":"2024-04-25T11:33:10.899946+0000","last_became_peered":"2024-04-25T11:33:10.899946+0000","last_unstale":"2024-04-25T11:34:17.876927+0000","last_undegraded":"2024-04-25T11:34:17.876927+0000","last_fullsized":"2024-04-25T11:34:17.876927+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:33:40.805400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"77'5","reported_seq":25,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.879187+0000","last_change":"2024-04-25T11:33:10.035668+0000","last_active":"2024-04-25T11:34:17.879187+0000","last_peered":"2024-04-25T11:34:17.879187+0000","last_clean":"2024-04-25T11:34:17.879187+0000","last_became_active":"2024-04-25T11:33:10.035241+0000","last_became_peered":"2024-04-25T11:33:10.035241+0000","last_unstale":"2024-04-25T11:34:17.879187+0000","last_undegraded":"2024-04-25T11:34:17.879187+0000","last_fullsized":"2024-04-25T11:34:17.879187+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:58:48.735577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880602+0000","last_change":"2024-04-25T11:33:10.894864+0000","last_active":"2024-04-25T11:34:17.880602+0000","last_peered":"2024-04-25T11:34:17.880602+0000","last_clean":"2024-04-25T11:34:17.880602+0000","last_became_active":"2024-04-25T11:33:10.894491+0000","last_became_peered":"2024-04-25T11:33:10.894491+0000","last_unstale":"2024-04-25T11:34:17.880602+0000","last_undegraded":"2024-04-25T11:34:17.880602+0000","last_fullsized":"2024-04-25T11:34:17.880602+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:21:28.256878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"77'6","reported_seq":91,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.739345+0000","last_change":"2024-04-25T11:33:10.029099+0000","last_active":"2024-04-25T11:34:18.739345+0000","last_peered":"2024-04-25T11:34:18.739345+0000","last_clean":"2024-04-25T11:34:18.739345+0000","last_became_active":"2024-04-25T11:33:10.028722+0000","last_became_peered":"2024-04-25T11:33:10.028722+0000","last_unstale":"2024-04-25T11:34:18.739345+0000","last_undegraded":"2024-04-25T11:34:18.739345+0000","last_fullsized":"2024-04-25T11:34:18.739345+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:44:12.261519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":67,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880843+0000","last_change":"2024-04-25T11:33:10.888635+0000","last_active":"2024-04-25T11:34:17.880843+0000","last_peered":"2024-04-25T11:34:17.880843+0000","last_clean":"2024-04-25T11:34:17.880843+0000","last_became_active":"2024-04-25T11:33:10.887544+0000","last_became_peered":"2024-04-25T11:33:10.887544+0000","last_unstale":"2024-04-25T11:34:17.880843+0000","last_undegraded":"2024-04-25T11:34:17.880843+0000","last_fullsized":"2024-04-25T11:34:17.880843+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:59:25.459587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"77'17","reported_seq":37,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.235785+0000","last_change":"2024-04-25T11:33:10.023566+0000","last_active":"2024-04-25T11:34:18.235785+0000","last_peered":"2024-04-25T11:34:18.235785+0000","last_clean":"2024-04-25T11:34:18.235785+0000","last_became_active":"2024-04-25T11:33:10.023234+0000","last_became_peered":"2024-04-25T11:33:10.023234+0000","last_unstale":"2024-04-25T11:34:18.235785+0000","last_undegraded":"2024-04-25T11:34:18.235785+0000","last_fullsized":"2024-04-25T11:34:18.235785+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:56:26.342148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":23,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.237663+0000","last_change":"2024-04-25T11:34:18.237663+0000","last_active":"2024-04-25T11:34:18.237663+0000","last_peered":"2024-04-25T11:34:18.237663+0000","last_clean":"2024-04-25T11:34:18.237663+0000","last_became_active":"2024-04-25T11:34:18.236993+0000","last_became_peered":"2024-04-25T11:34:18.236993+0000","last_unstale":"2024-04-25T11:34:18.237663+0000","last_undegraded":"2024-04-25T11:34:18.237663+0000","last_fullsized":"2024-04-25T11:34:18.237663+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:59:47.371685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":22,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.236405+0000","last_change":"2024-04-25T11:33:10.028285+0000","last_active":"2024-04-25T11:34:18.236405+0000","last_peered":"2024-04-25T11:34:18.236405+0000","last_clean":"2024-04-25T11:34:18.236405+0000","last_became_active":"2024-04-25T11:33:10.027959+0000","last_became_peered":"2024-04-25T11:33:10.027959+0000","last_unstale":"2024-04-25T11:34:18.236405+0000","last_undegraded":"2024-04-25T11:34:18.236405+0000","last_fullsized":"2024-04-25T11:34:18.236405+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:44:41.094048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.887356+0000","last_change":"2024-04-25T11:34:17.887356+0000","last_active":"2024-04-25T11:34:17.887356+0000","last_peered":"2024-04-25T11:34:17.887356+0000","last_clean":"2024-04-25T11:34:17.887356+0000","last_became_active":"2024-04-25T11:34:17.886981+0000","last_became_peered":"2024-04-25T11:34:17.886981+0000","last_unstale":"2024-04-25T11:34:17.887356+0000","last_undegraded":"2024-04-25T11:34:17.887356+0000","last_fullsized":"2024-04-25T11:34:17.887356+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:40.573546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"77'1","reported_seq":20,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.877528+0000","last_change":"2024-04-25T11:33:10.027473+0000","last_active":"2024-04-25T11:34:17.877528+0000","last_peered":"2024-04-25T11:34:17.877528+0000","last_clean":"2024-04-25T11:34:17.877528+0000","last_became_active":"2024-04-25T11:33:10.027133+0000","last_became_peered":"2024-04-25T11:33:10.027133+0000","last_unstale":"2024-04-25T11:34:17.877528+0000","last_undegraded":"2024-04-25T11:34:17.877528+0000","last_fullsized":"2024-04-25T11:34:17.877528+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:19:58.338507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"77'1","reported_seq":18,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.879131+0000","last_change":"2024-04-25T11:33:10.892588+0000","last_active":"2024-04-25T11:34:17.879131+0000","last_peered":"2024-04-25T11:34:17.879131+0000","last_clean":"2024-04-25T11:34:17.879131+0000","last_became_active":"2024-04-25T11:33:10.892088+0000","last_became_peered":"2024-04-25T11:33:10.892088+0000","last_unstale":"2024-04-25T11:34:17.879131+0000","last_undegraded":"2024-04-25T11:34:17.879131+0000","last_fullsized":"2024-04-25T11:34:17.879131+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:10.136222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"77'5","reported_seq":93,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:22.742834+0000","last_change":"2024-04-25T11:33:10.030801+0000","last_active":"2024-04-25T11:34:22.742834+0000","last_peered":"2024-04-25T11:34:22.742834+0000","last_clean":"2024-04-25T11:34:22.742834+0000","last_became_active":"2024-04-25T11:33:10.030466+0000","last_became_peered":"2024-04-25T11:33:10.030466+0000","last_unstale":"2024-04-25T11:34:22.742834+0000","last_undegraded":"2024-04-25T11:34:22.742834+0000","last_fullsized":"2024-04-25T11:34:22.742834+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:22:02.922309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.879183+0000","last_change":"2024-04-25T11:33:10.894846+0000","last_active":"2024-04-25T11:34:17.879183+0000","last_peered":"2024-04-25T11:34:17.879183+0000","last_clean":"2024-04-25T11:34:17.879183+0000","last_became_active":"2024-04-25T11:33:10.894456+0000","last_became_peered":"2024-04-25T11:33:10.894456+0000","last_unstale":"2024-04-25T11:34:17.879183+0000","last_undegraded":"2024-04-25T11:34:17.879183+0000","last_fullsized":"2024-04-25T11:34:17.879183+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:11:16.265126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":23,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.882781+0000","last_change":"2024-04-25T11:34:17.882781+0000","last_active":"2024-04-25T11:34:17.882781+0000","last_peered":"2024-04-25T11:34:17.882781+0000","last_clean":"2024-04-25T11:34:17.882781+0000","last_became_active":"2024-04-25T11:34:17.882348+0000","last_became_peered":"2024-04-25T11:34:17.882348+0000","last_unstale":"2024-04-25T11:34:17.882781+0000","last_undegraded":"2024-04-25T11:34:17.882781+0000","last_fullsized":"2024-04-25T11:34:17.882781+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:47:34.385332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"77'5","reported_seq":93,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:22.744142+0000","last_change":"2024-04-25T11:33:10.037834+0000","last_active":"2024-04-25T11:34:22.744142+0000","last_peered":"2024-04-25T11:34:22.744142+0000","last_clean":"2024-04-25T11:34:22.744142+0000","last_became_active":"2024-04-25T11:33:10.037570+0000","last_became_peered":"2024-04-25T11:33:10.037570+0000","last_unstale":"2024-04-25T11:34:22.744142+0000","last_undegraded":"2024-04-25T11:34:22.744142+0000","last_fullsized":"2024-04-25T11:34:22.744142+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:12.795190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.875250+0000","last_change":"2024-04-25T11:33:10.882663+0000","last_active":"2024-04-25T11:34:17.875250+0000","last_peered":"2024-04-25T11:34:17.875250+0000","last_clean":"2024-04-25T11:34:17.875250+0000","last_became_active":"2024-04-25T11:33:10.882300+0000","last_became_peered":"2024-04-25T11:33:10.882300+0000","last_unstale":"2024-04-25T11:34:17.875250+0000","last_undegraded":"2024-04-25T11:34:17.875250+0000","last_fullsized":"2024-04-25T11:34:17.875250+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:58:11.379654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"77'6","reported_seq":28,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.297920+0000","last_change":"2024-04-25T11:33:10.024598+0000","last_active":"2024-04-25T11:34:18.297920+0000","last_peered":"2024-04-25T11:34:18.297920+0000","last_clean":"2024-04-25T11:34:18.297920+0000","last_became_active":"2024-04-25T11:33:10.024229+0000","last_became_peered":"2024-04-25T11:33:10.024229+0000","last_unstale":"2024-04-25T11:34:18.297920+0000","last_undegraded":"2024-04-25T11:34:18.297920+0000","last_fullsized":"2024-04-25T11:34:18.297920+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:42:14.690581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":95,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880316+0000","last_change":"2024-04-25T11:32:01.310202+0000","last_active":"2024-04-25T11:34:17.880316+0000","last_peered":"2024-04-25T11:34:17.880316+0000","last_clean":"2024-04-25T11:34:17.880316+0000","last_became_active":"2024-04-25T11:32:01.305242+0000","last_became_peered":"2024-04-25T11:32:01.305242+0000","last_unstale":"2024-04-25T11:34:17.880316+0000","last_undegraded":"2024-04-25T11:34:17.880316+0000","last_fullsized":"2024-04-25T11:34:17.880316+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:28:43.374096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:28:43.374096+0000","last_clean_scrub_stamp":"2024-04-25T11:28:43.374096+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:56:18.551747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876853+0000","last_change":"2024-04-25T11:33:10.889124+0000","last_active":"2024-04-25T11:34:17.876853+0000","last_peered":"2024-04-25T11:34:17.876853+0000","last_clean":"2024-04-25T11:34:17.876853+0000","last_became_active":"2024-04-25T11:33:10.888790+0000","last_became_peered":"2024-04-25T11:33:10.888790+0000","last_unstale":"2024-04-25T11:34:17.876853+0000","last_undegraded":"2024-04-25T11:34:17.876853+0000","last_fullsized":"2024-04-25T11:34:17.876853+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:33:44.999949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"77'4","reported_seq":23,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880130+0000","last_change":"2024-04-25T11:33:10.031547+0000","last_active":"2024-04-25T11:34:17.880130+0000","last_peered":"2024-04-25T11:34:17.880130+0000","last_clean":"2024-04-25T11:34:17.880130+0000","last_became_active":"2024-04-25T11:33:10.031152+0000","last_became_peered":"2024-04-25T11:33:10.031152+0000","last_unstale":"2024-04-25T11:34:17.880130+0000","last_undegraded":"2024-04-25T11:34:17.880130+0000","last_fullsized":"2024-04-25T11:34:17.880130+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:10.646771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"77'4","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.236346+0000","last_change":"2024-04-25T11:33:10.020367+0000","last_active":"2024-04-25T11:34:18.236346+0000","last_peered":"2024-04-25T11:34:18.236346+0000","last_clean":"2024-04-25T11:34:18.236346+0000","last_became_active":"2024-04-25T11:33:10.020007+0000","last_became_peered":"2024-04-25T11:33:10.020007+0000","last_unstale":"2024-04-25T11:34:18.236346+0000","last_undegraded":"2024-04-25T11:34:18.236346+0000","last_fullsized":"2024-04-25T11:34:18.236346+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:48.134907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.877378+0000","last_change":"2024-04-25T11:33:10.887741+0000","last_active":"2024-04-25T11:34:17.877378+0000","last_peered":"2024-04-25T11:34:17.877378+0000","last_clean":"2024-04-25T11:34:17.877378+0000","last_became_active":"2024-04-25T11:33:10.886455+0000","last_became_peered":"2024-04-25T11:33:10.886455+0000","last_unstale":"2024-04-25T11:34:17.877378+0000","last_undegraded":"2024-04-25T11:34:17.877378+0000","last_fullsized":"2024-04-25T11:34:17.877378+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:37:16.842817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"77'12","reported_seq":31,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.878184+0000","last_change":"2024-04-25T11:33:10.025672+0000","last_active":"2024-04-25T11:34:17.878184+0000","last_peered":"2024-04-25T11:34:17.878184+0000","last_clean":"2024-04-25T11:34:17.878184+0000","last_became_active":"2024-04-25T11:33:10.025232+0000","last_became_peered":"2024-04-25T11:33:10.025232+0000","last_unstale":"2024-04-25T11:34:17.878184+0000","last_undegraded":"2024-04-25T11:34:17.878184+0000","last_fullsized":"2024-04-25T11:34:17.878184+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:06:26.579126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.875733+0000","last_change":"2024-04-25T11:33:10.884692+0000","last_active":"2024-04-25T11:34:17.875733+0000","last_peered":"2024-04-25T11:34:17.875733+0000","last_clean":"2024-04-25T11:34:17.875733+0000","last_became_active":"2024-04-25T11:33:10.884346+0000","last_became_peered":"2024-04-25T11:33:10.884346+0000","last_unstale":"2024-04-25T11:34:17.875733+0000","last_undegraded":"2024-04-25T11:34:17.875733+0000","last_fullsized":"2024-04-25T11:34:17.875733+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:31:00.630410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"77'7","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.297980+0000","last_change":"2024-04-25T11:33:10.033872+0000","last_active":"2024-04-25T11:34:18.297980+0000","last_peered":"2024-04-25T11:34:18.297980+0000","last_clean":"2024-04-25T11:34:18.297980+0000","last_became_active":"2024-04-25T11:33:10.033435+0000","last_became_peered":"2024-04-25T11:33:10.033435+0000","last_unstale":"2024-04-25T11:34:18.297980+0000","last_undegraded":"2024-04-25T11:34:18.297980+0000","last_fullsized":"2024-04-25T11:34:18.297980+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:12.885215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":23,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.881545+0000","last_change":"2024-04-25T11:34:17.881545+0000","last_active":"2024-04-25T11:34:17.881545+0000","last_peered":"2024-04-25T11:34:17.881545+0000","last_clean":"2024-04-25T11:34:17.881545+0000","last_became_active":"2024-04-25T11:34:17.881125+0000","last_became_peered":"2024-04-25T11:34:17.881125+0000","last_unstale":"2024-04-25T11:34:17.881545+0000","last_undegraded":"2024-04-25T11:34:17.881545+0000","last_fullsized":"2024-04-25T11:34:17.881545+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:55:18.404023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"77'2","reported_seq":21,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876299+0000","last_change":"2024-04-25T11:33:10.032384+0000","last_active":"2024-04-25T11:34:17.876299+0000","last_peered":"2024-04-25T11:34:17.876299+0000","last_clean":"2024-04-25T11:34:17.876299+0000","last_became_active":"2024-04-25T11:33:10.032035+0000","last_became_peered":"2024-04-25T11:33:10.032035+0000","last_unstale":"2024-04-25T11:34:17.876299+0000","last_undegraded":"2024-04-25T11:34:17.876299+0000","last_fullsized":"2024-04-25T11:34:17.876299+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:21:41.723120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.236013+0000","last_change":"2024-04-25T11:33:10.893366+0000","last_active":"2024-04-25T11:34:18.236013+0000","last_peered":"2024-04-25T11:34:18.236013+0000","last_clean":"2024-04-25T11:34:18.236013+0000","last_became_active":"2024-04-25T11:33:10.892988+0000","last_became_peered":"2024-04-25T11:33:10.892988+0000","last_unstale":"2024-04-25T11:34:18.236013+0000","last_undegraded":"2024-04-25T11:34:18.236013+0000","last_fullsized":"2024-04-25T11:34:18.236013+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:06.368216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"77'7","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.875184+0000","last_change":"2024-04-25T11:33:10.031456+0000","last_active":"2024-04-25T11:34:17.875184+0000","last_peered":"2024-04-25T11:34:17.875184+0000","last_clean":"2024-04-25T11:34:17.875184+0000","last_became_active":"2024-04-25T11:33:10.031115+0000","last_became_peered":"2024-04-25T11:33:10.031115+0000","last_unstale":"2024-04-25T11:34:17.875184+0000","last_undegraded":"2024-04-25T11:34:17.875184+0000","last_fullsized":"2024-04-25T11:34:17.875184+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:12:56.803277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880640+0000","last_change":"2024-04-25T11:33:10.893692+0000","last_active":"2024-04-25T11:34:17.880640+0000","last_peered":"2024-04-25T11:34:17.880640+0000","last_clean":"2024-04-25T11:34:17.880640+0000","last_became_active":"2024-04-25T11:33:10.893332+0000","last_became_peered":"2024-04-25T11:33:10.893332+0000","last_unstale":"2024-04-25T11:34:17.880640+0000","last_undegraded":"2024-04-25T11:34:17.880640+0000","last_fullsized":"2024-04-25T11:34:17.880640+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:01.277465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"77'3","reported_seq":22,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876399+0000","last_change":"2024-04-25T11:33:10.023804+0000","last_active":"2024-04-25T11:34:17.876399+0000","last_peered":"2024-04-25T11:34:17.876399+0000","last_clean":"2024-04-25T11:34:17.876399+0000","last_became_active":"2024-04-25T11:33:10.023503+0000","last_became_peered":"2024-04-25T11:33:10.023503+0000","last_unstale":"2024-04-25T11:34:17.876399+0000","last_undegraded":"2024-04-25T11:34:17.876399+0000","last_fullsized":"2024-04-25T11:34:17.876399+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:34:48.074176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.883633+0000","last_change":"2024-04-25T11:34:17.882901+0000","last_active":"2024-04-25T11:34:17.883633+0000","last_peered":"2024-04-25T11:34:17.883633+0000","last_clean":"2024-04-25T11:34:17.883633+0000","last_became_active":"2024-04-25T11:34:17.882514+0000","last_became_peered":"2024-04-25T11:34:17.882514+0000","last_unstale":"2024-04-25T11:34:17.883633+0000","last_undegraded":"2024-04-25T11:34:17.883633+0000","last_fullsized":"2024-04-25T11:34:17.883633+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:03:30.843842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,10],"acting":[9,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.b","version":"77'4","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.236101+0000","last_change":"2024-04-25T11:33:10.030857+0000","last_active":"2024-04-25T11:34:18.236101+0000","last_peered":"2024-04-25T11:34:18.236101+0000","last_clean":"2024-04-25T11:34:18.236101+0000","last_became_active":"2024-04-25T11:33:10.028879+0000","last_became_peered":"2024-04-25T11:33:10.028879+0000","last_unstale":"2024-04-25T11:34:18.236101+0000","last_undegraded":"2024-04-25T11:34:18.236101+0000","last_fullsized":"2024-04-25T11:34:18.236101+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:28.820919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880411+0000","last_change":"2024-04-25T11:33:10.893773+0000","last_active":"2024-04-25T11:34:17.880411+0000","last_peered":"2024-04-25T11:34:17.880411+0000","last_clean":"2024-04-25T11:34:17.880411+0000","last_became_active":"2024-04-25T11:33:10.893431+0000","last_became_peered":"2024-04-25T11:33:10.893431+0000","last_unstale":"2024-04-25T11:34:17.880411+0000","last_undegraded":"2024-04-25T11:34:17.880411+0000","last_fullsized":"2024-04-25T11:34:17.880411+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:15:15.998951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"77'5","reported_seq":91,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:19.739588+0000","last_change":"2024-04-25T11:33:10.032745+0000","last_active":"2024-04-25T11:34:19.739588+0000","last_peered":"2024-04-25T11:34:19.739588+0000","last_clean":"2024-04-25T11:34:19.739588+0000","last_became_active":"2024-04-25T11:33:10.032361+0000","last_became_peered":"2024-04-25T11:33:10.032361+0000","last_unstale":"2024-04-25T11:34:19.739588+0000","last_undegraded":"2024-04-25T11:34:19.739588+0000","last_fullsized":"2024-04-25T11:34:19.739588+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:14:46.555892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":67,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880904+0000","last_change":"2024-04-25T11:33:10.888084+0000","last_active":"2024-04-25T11:34:17.880904+0000","last_peered":"2024-04-25T11:34:17.880904+0000","last_clean":"2024-04-25T11:34:17.880904+0000","last_became_active":"2024-04-25T11:33:10.887679+0000","last_became_peered":"2024-04-25T11:33:10.887679+0000","last_unstale":"2024-04-25T11:34:17.880904+0000","last_undegraded":"2024-04-25T11:34:17.880904+0000","last_fullsized":"2024-04-25T11:34:17.880904+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:10:20.073630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"77'3","reported_seq":23,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.879980+0000","last_change":"2024-04-25T11:33:10.035355+0000","last_active":"2024-04-25T11:34:17.879980+0000","last_peered":"2024-04-25T11:34:17.879980+0000","last_clean":"2024-04-25T11:34:17.879980+0000","last_became_active":"2024-04-25T11:33:10.034724+0000","last_became_peered":"2024-04-25T11:33:10.034724+0000","last_unstale":"2024-04-25T11:34:17.879980+0000","last_undegraded":"2024-04-25T11:34:17.879980+0000","last_fullsized":"2024-04-25T11:34:17.879980+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:24:26.757803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"77'1","reported_seq":18,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880517+0000","last_change":"2024-04-25T11:33:10.899827+0000","last_active":"2024-04-25T11:34:17.880517+0000","last_peered":"2024-04-25T11:34:17.880517+0000","last_clean":"2024-04-25T11:34:17.880517+0000","last_became_active":"2024-04-25T11:33:10.899356+0000","last_became_peered":"2024-04-25T11:33:10.899356+0000","last_unstale":"2024-04-25T11:34:17.880517+0000","last_undegraded":"2024-04-25T11:34:17.880517+0000","last_fullsized":"2024-04-25T11:34:17.880517+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:04:17.323958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"77'5","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.877727+0000","last_change":"2024-04-25T11:33:10.034981+0000","last_active":"2024-04-25T11:34:17.877727+0000","last_peered":"2024-04-25T11:34:17.877727+0000","last_clean":"2024-04-25T11:34:17.877727+0000","last_became_active":"2024-04-25T11:33:10.034601+0000","last_became_peered":"2024-04-25T11:33:10.034601+0000","last_unstale":"2024-04-25T11:34:17.877727+0000","last_undegraded":"2024-04-25T11:34:17.877727+0000","last_fullsized":"2024-04-25T11:34:17.877727+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:19:14.710731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880720+0000","last_change":"2024-04-25T11:33:10.890020+0000","last_active":"2024-04-25T11:34:17.880720+0000","last_peered":"2024-04-25T11:34:17.880720+0000","last_clean":"2024-04-25T11:34:17.880720+0000","last_became_active":"2024-04-25T11:33:10.889558+0000","last_became_peered":"2024-04-25T11:33:10.889558+0000","last_unstale":"2024-04-25T11:34:17.880720+0000","last_undegraded":"2024-04-25T11:34:17.880720+0000","last_fullsized":"2024-04-25T11:34:17.880720+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:00:49.096818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880777+0000","last_change":"2024-04-25T11:33:10.886965+0000","last_active":"2024-04-25T11:34:17.880777+0000","last_peered":"2024-04-25T11:34:17.880777+0000","last_clean":"2024-04-25T11:34:17.880777+0000","last_became_active":"2024-04-25T11:33:10.886594+0000","last_became_peered":"2024-04-25T11:33:10.886594+0000","last_unstale":"2024-04-25T11:34:17.880777+0000","last_undegraded":"2024-04-25T11:34:17.880777+0000","last_fullsized":"2024-04-25T11:34:17.880777+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:58.924857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876483+0000","last_change":"2024-04-25T11:33:10.890760+0000","last_active":"2024-04-25T11:34:17.876483+0000","last_peered":"2024-04-25T11:34:17.876483+0000","last_clean":"2024-04-25T11:34:17.876483+0000","last_became_active":"2024-04-25T11:33:10.889643+0000","last_became_peered":"2024-04-25T11:33:10.889643+0000","last_unstale":"2024-04-25T11:34:17.876483+0000","last_undegraded":"2024-04-25T11:34:17.876483+0000","last_fullsized":"2024-04-25T11:34:17.876483+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:54.644853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876206+0000","last_change":"2024-04-25T11:33:10.887850+0000","last_active":"2024-04-25T11:34:17.876206+0000","last_peered":"2024-04-25T11:34:17.876206+0000","last_clean":"2024-04-25T11:34:17.876206+0000","last_became_active":"2024-04-25T11:33:10.882323+0000","last_became_peered":"2024-04-25T11:33:10.882323+0000","last_unstale":"2024-04-25T11:34:17.876206+0000","last_undegraded":"2024-04-25T11:34:17.876206+0000","last_fullsized":"2024-04-25T11:34:17.876206+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:24:00.509704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.879830+0000","last_change":"2024-04-25T11:33:10.883842+0000","last_active":"2024-04-25T11:34:17.879830+0000","last_peered":"2024-04-25T11:34:17.879830+0000","last_clean":"2024-04-25T11:34:17.879830+0000","last_became_active":"2024-04-25T11:33:10.883513+0000","last_became_peered":"2024-04-25T11:33:10.883513+0000","last_unstale":"2024-04-25T11:34:17.879830+0000","last_undegraded":"2024-04-25T11:34:17.879830+0000","last_fullsized":"2024-04-25T11:34:17.879830+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:47:09.731875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.877269+0000","last_change":"2024-04-25T11:33:10.890503+0000","last_active":"2024-04-25T11:34:17.877269+0000","last_peered":"2024-04-25T11:34:17.877269+0000","last_clean":"2024-04-25T11:34:17.877269+0000","last_became_active":"2024-04-25T11:33:10.890162+0000","last_became_peered":"2024-04-25T11:33:10.890162+0000","last_unstale":"2024-04-25T11:34:17.877269+0000","last_undegraded":"2024-04-25T11:34:17.877269+0000","last_fullsized":"2024-04-25T11:34:17.877269+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:36.462789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880918+0000","last_change":"2024-04-25T11:33:10.897109+0000","last_active":"2024-04-25T11:34:17.880918+0000","last_peered":"2024-04-25T11:34:17.880918+0000","last_clean":"2024-04-25T11:34:17.880918+0000","last_became_active":"2024-04-25T11:33:10.896138+0000","last_became_peered":"2024-04-25T11:33:10.896138+0000","last_unstale":"2024-04-25T11:34:17.880918+0000","last_undegraded":"2024-04-25T11:34:17.880918+0000","last_fullsized":"2024-04-25T11:34:17.880918+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:47:46.202355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876862+0000","last_change":"2024-04-25T11:33:10.890397+0000","last_active":"2024-04-25T11:34:17.876862+0000","last_peered":"2024-04-25T11:34:17.876862+0000","last_clean":"2024-04-25T11:34:17.876862+0000","last_became_active":"2024-04-25T11:33:10.889580+0000","last_became_peered":"2024-04-25T11:33:10.889580+0000","last_unstale":"2024-04-25T11:34:17.876862+0000","last_undegraded":"2024-04-25T11:34:17.876862+0000","last_fullsized":"2024-04-25T11:34:17.876862+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:16:09.672131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.875702+0000","last_change":"2024-04-25T11:33:10.873817+0000","last_active":"2024-04-25T11:34:17.875702+0000","last_peered":"2024-04-25T11:34:17.875702+0000","last_clean":"2024-04-25T11:34:17.875702+0000","last_became_active":"2024-04-25T11:33:10.873425+0000","last_became_peered":"2024-04-25T11:33:10.873425+0000","last_unstale":"2024-04-25T11:34:17.875702+0000","last_undegraded":"2024-04-25T11:34:17.875702+0000","last_fullsized":"2024-04-25T11:34:17.875702+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:52:26.279705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880651+0000","last_change":"2024-04-25T11:34:17.880075+0000","last_active":"2024-04-25T11:34:17.880651+0000","last_peered":"2024-04-25T11:34:17.880651+0000","last_clean":"2024-04-25T11:34:17.880651+0000","last_became_active":"2024-04-25T11:34:17.879692+0000","last_became_peered":"2024-04-25T11:34:17.879692+0000","last_unstale":"2024-04-25T11:34:17.880651+0000","last_undegraded":"2024-04-25T11:34:17.880651+0000","last_fullsized":"2024-04-25T11:34:17.880651+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:54.833601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,3],"acting":[6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.875168+0000","last_change":"2024-04-25T11:33:10.883690+0000","last_active":"2024-04-25T11:34:17.875168+0000","last_peered":"2024-04-25T11:34:17.875168+0000","last_clean":"2024-04-25T11:34:17.875168+0000","last_became_active":"2024-04-25T11:33:10.882965+0000","last_became_peered":"2024-04-25T11:33:10.882965+0000","last_unstale":"2024-04-25T11:34:17.875168+0000","last_undegraded":"2024-04-25T11:34:17.875168+0000","last_fullsized":"2024-04-25T11:34:17.875168+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:28.126631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876395+0000","last_change":"2024-04-25T11:33:10.897050+0000","last_active":"2024-04-25T11:34:17.876395+0000","last_peered":"2024-04-25T11:34:17.876395+0000","last_clean":"2024-04-25T11:34:17.876395+0000","last_became_active":"2024-04-25T11:33:10.896273+0000","last_became_peered":"2024-04-25T11:33:10.896273+0000","last_unstale":"2024-04-25T11:34:17.876395+0000","last_undegraded":"2024-04-25T11:34:17.876395+0000","last_fullsized":"2024-04-25T11:34:17.876395+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:45:53.839823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.877808+0000","last_change":"2024-04-25T11:33:10.897323+0000","last_active":"2024-04-25T11:34:17.877808+0000","last_peered":"2024-04-25T11:34:17.877808+0000","last_clean":"2024-04-25T11:34:17.877808+0000","last_became_active":"2024-04-25T11:33:10.896951+0000","last_became_peered":"2024-04-25T11:33:10.896951+0000","last_unstale":"2024-04-25T11:34:17.877808+0000","last_undegraded":"2024-04-25T11:34:17.877808+0000","last_fullsized":"2024-04-25T11:34:17.877808+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:24:31.208338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880816+0000","last_change":"2024-04-25T11:33:10.897566+0000","last_active":"2024-04-25T11:34:17.880816+0000","last_peered":"2024-04-25T11:34:17.880816+0000","last_clean":"2024-04-25T11:34:17.880816+0000","last_became_active":"2024-04-25T11:33:10.897212+0000","last_became_peered":"2024-04-25T11:33:10.897212+0000","last_unstale":"2024-04-25T11:34:17.880816+0000","last_undegraded":"2024-04-25T11:34:17.880816+0000","last_fullsized":"2024-04-25T11:34:17.880816+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:55:29.995287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67859,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":714,"num_read_kb":434,"num_write":236,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":203577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645340,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36188,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708964,"statfs":{"total":95995035648,"available":95957979136,"internally_reserved":0,"allocated":868352,"data_stored":413865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841569,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32716,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712436,"statfs":{"total":95995035648,"available":95961534464,"internally_reserved":0,"allocated":1568768,"data_stored":1066331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037797,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":413795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234027,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":888832,"data_stored":414173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430256,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32636,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712516,"statfs":{"total":95995035648,"available":95961616384,"internally_reserved":0,"allocated":1486848,"data_stored":1006691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626485,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32036,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713116,"statfs":{"total":95995035648,"available":95962230784,"internally_reserved":0,"allocated":876544,"data_stored":413512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36260,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708892,"statfs":{"total":95995035648,"available":95957905408,"internally_reserved":0,"allocated":937984,"data_stored":475471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":897024,"data_stored":414851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40372,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704780,"statfs":{"total":95995035648,"available":95953694720,"internally_reserved":0,"allocated":888832,"data_stored":414645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182466,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":413337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986238,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":946176,"data_stored":474810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822714,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28480,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716672,"statfs":{"total":95995035648,"available":95965872128,"internally_reserved":0,"allocated":1499136,"data_stored":1005978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":62798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":62137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T11:34:26.237 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T11:34:26.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:25 smithi138 ceph-mon[32328]: pgmap v313: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s; 7.4 KiB/s, 0 objects/s recovering 2024-04-25T11:34:26.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:25 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:34:26.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:25 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:34:26.488 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:34:26.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:26 smithi090 ceph-mon[29575]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T11:34:26.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:26 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:34:26.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:26 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:34:26.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:26 smithi090 ceph-mon[29575]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:34:27.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:26 smithi156 ceph-mon[33045]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T11:34:27.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:26 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:34:27.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:26 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:34:27.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:26 smithi156 ceph-mon[33045]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:34:27.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:26 smithi138 ceph-mon[32328]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T11:34:27.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:26 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:34:27.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:26 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:34:27.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:26 smithi138 ceph-mon[32328]: from='mgr.14152 172.21.15.90:0/3136512634' entity='mgr.x' 2024-04-25T11:34:27.853 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:27 smithi090 ceph-mon[29575]: pgmap v314: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 6.0 KiB/s, 0 objects/s recovering 2024-04-25T11:34:28.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.096+0000 7f6f96508700 1 -- 172.21.15.90:0/132743642 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f90103810 msgr2=0x7f6f90103c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:28.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.096+0000 7f6f96508700 1 --2- 172.21.15.90:0/132743642 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f90103810 0x7f6f90103c20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6f84009990 tx=0x7f6f84009ca0 comp rx=0 tx=0).stop 2024-04-25T11:34:28.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.096+0000 7f6f96508700 1 -- 172.21.15.90:0/132743642 shutdown_connections 2024-04-25T11:34:28.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.096+0000 7f6f96508700 1 --2- 172.21.15.90:0/132743642 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6f90104160 0x7f6f9010ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:28.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.096+0000 7f6f96508700 1 --2- 172.21.15.90:0/132743642 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f901080b0 0x7f6f901084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:28.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.096+0000 7f6f96508700 1 --2- 172.21.15.90:0/132743642 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f90103810 0x7f6f90103c20 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:28.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.096+0000 7f6f96508700 1 -- 172.21.15.90:0/132743642 >> 172.21.15.90:0/132743642 conn(0x7f6f900fe370 msgr2=0x7f6f901007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:28.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.096+0000 7f6f96508700 1 -- 172.21.15.90:0/132743642 shutdown_connections 2024-04-25T11:34:28.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.096+0000 7f6f96508700 1 -- 172.21.15.90:0/132743642 wait complete. 2024-04-25T11:34:28.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.097+0000 7f6f96508700 1 Processor -- start 2024-04-25T11:34:28.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.097+0000 7f6f96508700 1 -- start start 2024-04-25T11:34:28.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.097+0000 7f6f96508700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6f90103810 0x7f6f9019bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:28.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.097+0000 7f6f96508700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f90104160 0x7f6f9019c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:28.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.097+0000 7f6f96508700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f901080b0 0x7f6f9019c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:28.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.097+0000 7f6f96508700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6f90113230 con 0x7f6f901080b0 2024-04-25T11:34:28.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.097+0000 7f6f96508700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6f901130b0 con 0x7f6f90103810 2024-04-25T11:34:28.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.097+0000 7f6f96508700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6f901133b0 con 0x7f6f90104160 2024-04-25T11:34:28.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.098+0000 7f6f94aa5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f901080b0 0x7f6f9019c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:28.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.098+0000 7f6f94aa5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f901080b0 0x7f6f9019c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57630/0 (socket says 172.21.15.90:57630) 2024-04-25T11:34:28.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.098+0000 7f6f94aa5700 1 -- 172.21.15.90:0/3446121858 learned_addr learned my addr 172.21.15.90:0/3446121858 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:28.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.098+0000 7f6f8f7fe700 1 --2- 172.21.15.90:0/3446121858 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f90104160 0x7f6f9019c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:28.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.098+0000 7f6f8ffff700 1 --2- 172.21.15.90:0/3446121858 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6f90103810 0x7f6f9019bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:28.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.098+0000 7f6f94aa5700 1 -- 172.21.15.90:0/3446121858 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f90104160 msgr2=0x7f6f9019c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:28.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.098+0000 7f6f94aa5700 1 --2- 172.21.15.90:0/3446121858 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f90104160 0x7f6f9019c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:28.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.098+0000 7f6f94aa5700 1 -- 172.21.15.90:0/3446121858 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6f90103810 msgr2=0x7f6f9019bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:28.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.098+0000 7f6f94aa5700 1 --2- 172.21.15.90:0/3446121858 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6f90103810 0x7f6f9019bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:28.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.098+0000 7f6f94aa5700 1 -- 172.21.15.90:0/3446121858 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f901a6e20 con 0x7f6f901080b0 2024-04-25T11:34:28.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.098+0000 7f6f8f7fe700 1 --2- 172.21.15.90:0/3446121858 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f90104160 0x7f6f9019c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:34:28.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.098+0000 7f6f94aa5700 1 --2- 172.21.15.90:0/3446121858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f901080b0 0x7f6f9019c570 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6f88007ee0 tx=0x7f6f8800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:28.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.098+0000 7f6f8d7fa700 1 -- 172.21.15.90:0/3446121858 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f88004bb0 con 0x7f6f901080b0 2024-04-25T11:34:28.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.098+0000 7f6f96508700 1 -- 172.21.15.90:0/3446121858 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f901a7110 con 0x7f6f901080b0 2024-04-25T11:34:28.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.098+0000 7f6f8d7fa700 1 -- 172.21.15.90:0/3446121858 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f88004d50 con 0x7f6f901080b0 2024-04-25T11:34:28.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.098+0000 7f6f96508700 1 -- 172.21.15.90:0/3446121858 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f901a7620 con 0x7f6f901080b0 2024-04-25T11:34:28.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.098+0000 7f6f8d7fa700 1 -- 172.21.15.90:0/3446121858 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f88005920 con 0x7f6f901080b0 2024-04-25T11:34:28.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.099+0000 7f6f96508700 1 -- 172.21.15.90:0/3446121858 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f9010b5a0 con 0x7f6f901080b0 2024-04-25T11:34:28.090 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:27 smithi156 ceph-mon[33045]: pgmap v314: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 6.0 KiB/s, 0 objects/s recovering 2024-04-25T11:34:28.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.101+0000 7f6f8d7fa700 1 -- 172.21.15.90:0/3446121858 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f6f8801a020 con 0x7f6f901080b0 2024-04-25T11:34:28.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.102+0000 7f6f8d7fa700 1 --2- 172.21.15.90:0/3446121858 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6f780afc10 0x7f6f780b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:28.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.102+0000 7f6f8ffff700 1 --2- 172.21.15.90:0/3446121858 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6f780afc10 0x7f6f780b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:28.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.102+0000 7f6f8d7fa700 1 -- 172.21.15.90:0/3446121858 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f6f880deb30 con 0x7f6f901080b0 2024-04-25T11:34:28.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.102+0000 7f6f8ffff700 1 --2- 172.21.15.90:0/3446121858 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6f780afc10 0x7f6f780b2120 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6f9006c1c0 tx=0x7f6f80009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:28.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.105+0000 7f6f8d7fa700 1 -- 172.21.15.90:0/3446121858 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6f88010070 con 0x7f6f901080b0 2024-04-25T11:34:28.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.298+0000 7f6f96508700 1 -- 172.21.15.90:0/3446121858 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6f90063ad0 con 0x7f6f780afc10 2024-04-25T11:34:28.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.306+0000 7f6f8d7fa700 1 -- 172.21.15.90:0/3446121858 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258742 (secure 0 0 0) 0x7f6f90063ad0 con 0x7f6f780afc10 2024-04-25T11:34:28.297 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:34:28.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.309+0000 7f6f96508700 1 -- 172.21.15.90:0/3446121858 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6f780afc10 msgr2=0x7f6f780b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:28.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.309+0000 7f6f96508700 1 --2- 172.21.15.90:0/3446121858 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6f780afc10 0x7f6f780b2120 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6f9006c1c0 tx=0x7f6f80009260 comp rx=0 tx=0).stop 2024-04-25T11:34:28.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.309+0000 7f6f96508700 1 -- 172.21.15.90:0/3446121858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f901080b0 msgr2=0x7f6f9019c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:28.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.309+0000 7f6f96508700 1 --2- 172.21.15.90:0/3446121858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f901080b0 0x7f6f9019c570 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6f88007ee0 tx=0x7f6f8800d3b0 comp rx=0 tx=0).stop 2024-04-25T11:34:28.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.309+0000 7f6f96508700 1 -- 172.21.15.90:0/3446121858 shutdown_connections 2024-04-25T11:34:28.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.309+0000 7f6f96508700 1 --2- 172.21.15.90:0/3446121858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f901080b0 0x7f6f9019c570 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:28.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.309+0000 7f6f96508700 1 --2- 172.21.15.90:0/3446121858 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6f90103810 0x7f6f9019bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:28.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.309+0000 7f6f96508700 1 --2- 172.21.15.90:0/3446121858 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6f780afc10 0x7f6f780b2120 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:28.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.309+0000 7f6f96508700 1 --2- 172.21.15.90:0/3446121858 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f90104160 0x7f6f9019c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:28.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.309+0000 7f6f96508700 1 -- 172.21.15.90:0/3446121858 >> 172.21.15.90:0/3446121858 conn(0x7f6f900fe370 msgr2=0x7f6f900fea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:28.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.309+0000 7f6f96508700 1 -- 172.21.15.90:0/3446121858 shutdown_connections 2024-04-25T11:34:28.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:28.309+0000 7f6f96508700 1 -- 172.21.15.90:0/3446121858 wait complete. 2024-04-25T11:34:28.300 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-25T11:34:28.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:27 smithi138 ceph-mon[32328]: pgmap v314: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 6.0 KiB/s, 0 objects/s recovering 2024-04-25T11:34:29.066 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":314,"stamp":"2024-04-25T11:34:26.761992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658461,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":805,"num_read_kb":503,"num_write":361,"num_write_kb":2284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":3012886,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":299,"ondisk_log_size":299,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":382376,"kb_used_data":12348,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124559448,"statfs":{"total":1151940427776,"available":1151548874752,"internally_reserved":0,"allocated":12644352,"data_stored":6927459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":92,"num_read_kb":46,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.889147"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.881195+0000","last_change":"2024-04-25T11:33:10.892144+0000","last_active":"2024-04-25T11:34:17.881195+0000","last_peered":"2024-04-25T11:34:17.881195+0000","last_clean":"2024-04-25T11:34:17.881195+0000","last_became_active":"2024-04-25T11:33:10.891721+0000","last_became_peered":"2024-04-25T11:33:10.891721+0000","last_unstale":"2024-04-25T11:34:17.881195+0000","last_undegraded":"2024-04-25T11:34:17.881195+0000","last_fullsized":"2024-04-25T11:34:17.881195+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:58:25.447086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.879751+0000","last_change":"2024-04-25T11:33:10.891647+0000","last_active":"2024-04-25T11:34:17.879751+0000","last_peered":"2024-04-25T11:34:17.879751+0000","last_clean":"2024-04-25T11:34:17.879751+0000","last_became_active":"2024-04-25T11:33:10.891283+0000","last_became_peered":"2024-04-25T11:33:10.891283+0000","last_unstale":"2024-04-25T11:34:17.879751+0000","last_undegraded":"2024-04-25T11:34:17.879751+0000","last_fullsized":"2024-04-25T11:34:17.879751+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:12.669438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.879623+0000","last_change":"2024-04-25T11:33:10.890928+0000","last_active":"2024-04-25T11:34:17.879623+0000","last_peered":"2024-04-25T11:34:17.879623+0000","last_clean":"2024-04-25T11:34:17.879623+0000","last_became_active":"2024-04-25T11:33:10.889093+0000","last_became_peered":"2024-04-25T11:33:10.889093+0000","last_unstale":"2024-04-25T11:34:17.879623+0000","last_undegraded":"2024-04-25T11:34:17.879623+0000","last_fullsized":"2024-04-25T11:34:17.879623+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:28:09.038650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.298945+0000","last_change":"2024-04-25T11:33:10.890184+0000","last_active":"2024-04-25T11:34:18.298945+0000","last_peered":"2024-04-25T11:34:18.298945+0000","last_clean":"2024-04-25T11:34:18.298945+0000","last_became_active":"2024-04-25T11:33:10.889367+0000","last_became_peered":"2024-04-25T11:33:10.889367+0000","last_unstale":"2024-04-25T11:34:18.298945+0000","last_undegraded":"2024-04-25T11:34:18.298945+0000","last_fullsized":"2024-04-25T11:34:18.298945+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:42.798295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.235974+0000","last_change":"2024-04-25T11:33:10.889758+0000","last_active":"2024-04-25T11:34:18.235974+0000","last_peered":"2024-04-25T11:34:18.235974+0000","last_clean":"2024-04-25T11:34:18.235974+0000","last_became_active":"2024-04-25T11:33:10.889384+0000","last_became_peered":"2024-04-25T11:33:10.889384+0000","last_unstale":"2024-04-25T11:34:18.235974+0000","last_undegraded":"2024-04-25T11:34:18.235974+0000","last_fullsized":"2024-04-25T11:34:18.235974+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:54:38.794216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.878863+0000","last_change":"2024-04-25T11:33:10.889348+0000","last_active":"2024-04-25T11:34:17.878863+0000","last_peered":"2024-04-25T11:34:17.878863+0000","last_clean":"2024-04-25T11:34:17.878863+0000","last_became_active":"2024-04-25T11:33:10.888908+0000","last_became_peered":"2024-04-25T11:33:10.888908+0000","last_unstale":"2024-04-25T11:34:17.878863+0000","last_undegraded":"2024-04-25T11:34:17.878863+0000","last_fullsized":"2024-04-25T11:34:17.878863+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:02:59.555921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":23,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.237661+0000","last_change":"2024-04-25T11:34:18.237661+0000","last_active":"2024-04-25T11:34:18.237661+0000","last_peered":"2024-04-25T11:34:18.237661+0000","last_clean":"2024-04-25T11:34:18.237661+0000","last_became_active":"2024-04-25T11:34:18.237306+0000","last_became_peered":"2024-04-25T11:34:18.237306+0000","last_unstale":"2024-04-25T11:34:18.237661+0000","last_undegraded":"2024-04-25T11:34:18.237661+0000","last_fullsized":"2024-04-25T11:34:18.237661+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:17:34.228368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.875134+0000","last_change":"2024-04-25T11:33:10.887262+0000","last_active":"2024-04-25T11:34:17.875134+0000","last_peered":"2024-04-25T11:34:17.875134+0000","last_clean":"2024-04-25T11:34:17.875134+0000","last_became_active":"2024-04-25T11:33:10.886948+0000","last_became_peered":"2024-04-25T11:33:10.886948+0000","last_unstale":"2024-04-25T11:34:17.875134+0000","last_undegraded":"2024-04-25T11:34:17.875134+0000","last_fullsized":"2024-04-25T11:34:17.875134+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:09.721506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.883852+0000","last_change":"2024-04-25T11:34:17.883253+0000","last_active":"2024-04-25T11:34:17.883852+0000","last_peered":"2024-04-25T11:34:17.883852+0000","last_clean":"2024-04-25T11:34:17.883852+0000","last_became_active":"2024-04-25T11:34:17.882515+0000","last_became_peered":"2024-04-25T11:34:17.882515+0000","last_unstale":"2024-04-25T11:34:17.883852+0000","last_undegraded":"2024-04-25T11:34:17.883852+0000","last_fullsized":"2024-04-25T11:34:17.883852+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:52:32.813464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.879418+0000","last_change":"2024-04-25T11:33:10.889812+0000","last_active":"2024-04-25T11:34:17.879418+0000","last_peered":"2024-04-25T11:34:17.879418+0000","last_clean":"2024-04-25T11:34:17.879418+0000","last_became_active":"2024-04-25T11:33:10.889369+0000","last_became_peered":"2024-04-25T11:33:10.889369+0000","last_unstale":"2024-04-25T11:34:17.879418+0000","last_undegraded":"2024-04-25T11:34:17.879418+0000","last_fullsized":"2024-04-25T11:34:17.879418+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:42:14.443496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880749+0000","last_change":"2024-04-25T11:33:10.891186+0000","last_active":"2024-04-25T11:34:17.880749+0000","last_peered":"2024-04-25T11:34:17.880749+0000","last_clean":"2024-04-25T11:34:17.880749+0000","last_became_active":"2024-04-25T11:33:10.890841+0000","last_became_peered":"2024-04-25T11:33:10.890841+0000","last_unstale":"2024-04-25T11:34:17.880749+0000","last_undegraded":"2024-04-25T11:34:17.880749+0000","last_fullsized":"2024-04-25T11:34:17.880749+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:24:56.078592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.298870+0000","last_change":"2024-04-25T11:33:10.888801+0000","last_active":"2024-04-25T11:34:18.298870+0000","last_peered":"2024-04-25T11:34:18.298870+0000","last_clean":"2024-04-25T11:34:18.298870+0000","last_became_active":"2024-04-25T11:33:10.888437+0000","last_became_peered":"2024-04-25T11:33:10.888437+0000","last_unstale":"2024-04-25T11:34:18.298870+0000","last_undegraded":"2024-04-25T11:34:18.298870+0000","last_fullsized":"2024-04-25T11:34:18.298870+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:14:14.011475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.877702+0000","last_change":"2024-04-25T11:33:10.892749+0000","last_active":"2024-04-25T11:34:17.877702+0000","last_peered":"2024-04-25T11:34:17.877702+0000","last_clean":"2024-04-25T11:34:17.877702+0000","last_became_active":"2024-04-25T11:33:10.892370+0000","last_became_peered":"2024-04-25T11:33:10.892370+0000","last_unstale":"2024-04-25T11:34:17.877702+0000","last_undegraded":"2024-04-25T11:34:17.877702+0000","last_fullsized":"2024-04-25T11:34:17.877702+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:00:51.312387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.879027+0000","last_change":"2024-04-25T11:33:10.888981+0000","last_active":"2024-04-25T11:34:17.879027+0000","last_peered":"2024-04-25T11:34:17.879027+0000","last_clean":"2024-04-25T11:34:17.879027+0000","last_became_active":"2024-04-25T11:33:10.888537+0000","last_became_peered":"2024-04-25T11:33:10.888537+0000","last_unstale":"2024-04-25T11:34:17.879027+0000","last_undegraded":"2024-04-25T11:34:17.879027+0000","last_fullsized":"2024-04-25T11:34:17.879027+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:12:19.018698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.879026+0000","last_change":"2024-04-25T11:33:10.891455+0000","last_active":"2024-04-25T11:34:17.879026+0000","last_peered":"2024-04-25T11:34:17.879026+0000","last_clean":"2024-04-25T11:34:17.879026+0000","last_became_active":"2024-04-25T11:33:10.891037+0000","last_became_peered":"2024-04-25T11:33:10.891037+0000","last_unstale":"2024-04-25T11:34:17.879026+0000","last_undegraded":"2024-04-25T11:34:17.879026+0000","last_fullsized":"2024-04-25T11:34:17.879026+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:44.470674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876205+0000","last_change":"2024-04-25T11:33:10.895309+0000","last_active":"2024-04-25T11:34:17.876205+0000","last_peered":"2024-04-25T11:34:17.876205+0000","last_clean":"2024-04-25T11:34:17.876205+0000","last_became_active":"2024-04-25T11:33:10.894952+0000","last_became_peered":"2024-04-25T11:33:10.894952+0000","last_unstale":"2024-04-25T11:34:17.876205+0000","last_undegraded":"2024-04-25T11:34:17.876205+0000","last_fullsized":"2024-04-25T11:34:17.876205+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:29:22.119017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.881501+0000","last_change":"2024-04-25T11:33:10.894524+0000","last_active":"2024-04-25T11:34:17.881501+0000","last_peered":"2024-04-25T11:34:17.881501+0000","last_clean":"2024-04-25T11:34:17.881501+0000","last_became_active":"2024-04-25T11:33:10.894220+0000","last_became_peered":"2024-04-25T11:33:10.894220+0000","last_unstale":"2024-04-25T11:34:17.881501+0000","last_undegraded":"2024-04-25T11:34:17.881501+0000","last_fullsized":"2024-04-25T11:34:17.881501+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:10:29.072976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.878215+0000","last_change":"2024-04-25T11:33:10.886814+0000","last_active":"2024-04-25T11:34:17.878215+0000","last_peered":"2024-04-25T11:34:17.878215+0000","last_clean":"2024-04-25T11:34:17.878215+0000","last_became_active":"2024-04-25T11:33:10.886494+0000","last_became_peered":"2024-04-25T11:33:10.886494+0000","last_unstale":"2024-04-25T11:34:17.878215+0000","last_undegraded":"2024-04-25T11:34:17.878215+0000","last_fullsized":"2024-04-25T11:34:17.878215+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:03:43.054485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876873+0000","last_change":"2024-04-25T11:33:10.886870+0000","last_active":"2024-04-25T11:34:17.876873+0000","last_peered":"2024-04-25T11:34:17.876873+0000","last_clean":"2024-04-25T11:34:17.876873+0000","last_became_active":"2024-04-25T11:33:10.886507+0000","last_became_peered":"2024-04-25T11:33:10.886507+0000","last_unstale":"2024-04-25T11:34:17.876873+0000","last_undegraded":"2024-04-25T11:34:17.876873+0000","last_fullsized":"2024-04-25T11:34:17.876873+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:50.101760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"77'13","reported_seq":100,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:21.740592+0000","last_change":"2024-04-25T11:33:10.028970+0000","last_active":"2024-04-25T11:34:21.740592+0000","last_peered":"2024-04-25T11:34:21.740592+0000","last_clean":"2024-04-25T11:34:21.740592+0000","last_became_active":"2024-04-25T11:33:10.028636+0000","last_became_peered":"2024-04-25T11:33:10.028636+0000","last_unstale":"2024-04-25T11:34:21.740592+0000","last_undegraded":"2024-04-25T11:34:21.740592+0000","last_fullsized":"2024-04-25T11:34:21.740592+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:02.414903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":68,"num_read_kb":68,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.882002+0000","last_change":"2024-04-25T11:34:17.881522+0000","last_active":"2024-04-25T11:34:17.882002+0000","last_peered":"2024-04-25T11:34:17.882002+0000","last_clean":"2024-04-25T11:34:17.882002+0000","last_became_active":"2024-04-25T11:34:17.881114+0000","last_became_peered":"2024-04-25T11:34:17.881114+0000","last_unstale":"2024-04-25T11:34:17.882002+0000","last_undegraded":"2024-04-25T11:34:17.882002+0000","last_fullsized":"2024-04-25T11:34:17.882002+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:59:03.675054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"77'2","reported_seq":21,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.875563+0000","last_change":"2024-04-25T11:33:10.036958+0000","last_active":"2024-04-25T11:34:17.875563+0000","last_peered":"2024-04-25T11:34:17.875563+0000","last_clean":"2024-04-25T11:34:17.875563+0000","last_became_active":"2024-04-25T11:33:10.036606+0000","last_became_peered":"2024-04-25T11:33:10.036606+0000","last_unstale":"2024-04-25T11:34:17.875563+0000","last_undegraded":"2024-04-25T11:34:17.875563+0000","last_fullsized":"2024-04-25T11:34:17.875563+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:12.660208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.879784+0000","last_change":"2024-04-25T11:33:10.887501+0000","last_active":"2024-04-25T11:34:17.879784+0000","last_peered":"2024-04-25T11:34:17.879784+0000","last_clean":"2024-04-25T11:34:17.879784+0000","last_became_active":"2024-04-25T11:33:10.885985+0000","last_became_peered":"2024-04-25T11:33:10.885985+0000","last_unstale":"2024-04-25T11:34:17.879784+0000","last_undegraded":"2024-04-25T11:34:17.879784+0000","last_fullsized":"2024-04-25T11:34:17.879784+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:31:32.035646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"77'12","reported_seq":32,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.236511+0000","last_change":"2024-04-25T11:33:10.031161+0000","last_active":"2024-04-25T11:34:18.236511+0000","last_peered":"2024-04-25T11:34:18.236511+0000","last_clean":"2024-04-25T11:34:18.236511+0000","last_became_active":"2024-04-25T11:33:10.028891+0000","last_became_peered":"2024-04-25T11:33:10.028891+0000","last_unstale":"2024-04-25T11:34:18.236511+0000","last_undegraded":"2024-04-25T11:34:18.236511+0000","last_fullsized":"2024-04-25T11:34:18.236511+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:25.959699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876332+0000","last_change":"2024-04-25T11:33:10.889439+0000","last_active":"2024-04-25T11:34:17.876332+0000","last_peered":"2024-04-25T11:34:17.876332+0000","last_clean":"2024-04-25T11:34:17.876332+0000","last_became_active":"2024-04-25T11:33:10.889082+0000","last_became_peered":"2024-04-25T11:33:10.889082+0000","last_unstale":"2024-04-25T11:34:17.876332+0000","last_undegraded":"2024-04-25T11:34:17.876332+0000","last_fullsized":"2024-04-25T11:34:17.876332+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:13:48.958179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"77'5","reported_seq":25,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.236179+0000","last_change":"2024-04-25T11:33:10.030793+0000","last_active":"2024-04-25T11:34:18.236179+0000","last_peered":"2024-04-25T11:34:18.236179+0000","last_clean":"2024-04-25T11:34:18.236179+0000","last_became_active":"2024-04-25T11:33:10.028805+0000","last_became_peered":"2024-04-25T11:33:10.028805+0000","last_unstale":"2024-04-25T11:34:18.236179+0000","last_undegraded":"2024-04-25T11:34:18.236179+0000","last_fullsized":"2024-04-25T11:34:18.236179+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:36.233194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880042+0000","last_change":"2024-04-25T11:33:10.894815+0000","last_active":"2024-04-25T11:34:17.880042+0000","last_peered":"2024-04-25T11:34:17.880042+0000","last_clean":"2024-04-25T11:34:17.880042+0000","last_became_active":"2024-04-25T11:33:10.892174+0000","last_became_peered":"2024-04-25T11:33:10.892174+0000","last_unstale":"2024-04-25T11:34:17.880042+0000","last_undegraded":"2024-04-25T11:34:17.880042+0000","last_fullsized":"2024-04-25T11:34:17.880042+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:05.368441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"77'5","reported_seq":97,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:22.719984+0000","last_change":"2024-04-25T11:33:10.031238+0000","last_active":"2024-04-25T11:34:22.719984+0000","last_peered":"2024-04-25T11:34:22.719984+0000","last_clean":"2024-04-25T11:34:22.719984+0000","last_became_active":"2024-04-25T11:33:10.030917+0000","last_became_peered":"2024-04-25T11:33:10.030917+0000","last_unstale":"2024-04-25T11:34:22.719984+0000","last_undegraded":"2024-04-25T11:34:22.719984+0000","last_fullsized":"2024-04-25T11:34:22.719984+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:00:35.634293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.235953+0000","last_change":"2024-04-25T11:33:10.896974+0000","last_active":"2024-04-25T11:34:18.235953+0000","last_peered":"2024-04-25T11:34:18.235953+0000","last_clean":"2024-04-25T11:34:18.235953+0000","last_became_active":"2024-04-25T11:33:10.896575+0000","last_became_peered":"2024-04-25T11:33:10.896575+0000","last_unstale":"2024-04-25T11:34:18.235953+0000","last_undegraded":"2024-04-25T11:34:18.235953+0000","last_fullsized":"2024-04-25T11:34:18.235953+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:32:43.533652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"77'7","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.878710+0000","last_change":"2024-04-25T11:33:10.022714+0000","last_active":"2024-04-25T11:34:17.878710+0000","last_peered":"2024-04-25T11:34:17.878710+0000","last_clean":"2024-04-25T11:34:17.878710+0000","last_became_active":"2024-04-25T11:33:10.022230+0000","last_became_peered":"2024-04-25T11:33:10.022230+0000","last_unstale":"2024-04-25T11:34:17.878710+0000","last_undegraded":"2024-04-25T11:34:17.878710+0000","last_fullsized":"2024-04-25T11:34:17.878710+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:30:00.620833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.878993+0000","last_change":"2024-04-25T11:33:10.897664+0000","last_active":"2024-04-25T11:34:17.878993+0000","last_peered":"2024-04-25T11:34:17.878993+0000","last_clean":"2024-04-25T11:34:17.878993+0000","last_became_active":"2024-04-25T11:33:10.897189+0000","last_became_peered":"2024-04-25T11:33:10.897189+0000","last_unstale":"2024-04-25T11:34:17.878993+0000","last_undegraded":"2024-04-25T11:34:17.878993+0000","last_fullsized":"2024-04-25T11:34:17.878993+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:12:06.168793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"77'3","reported_seq":22,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.877103+0000","last_change":"2024-04-25T11:33:10.031417+0000","last_active":"2024-04-25T11:34:17.877103+0000","last_peered":"2024-04-25T11:34:17.877103+0000","last_clean":"2024-04-25T11:34:17.877103+0000","last_became_active":"2024-04-25T11:33:10.030526+0000","last_became_peered":"2024-04-25T11:33:10.030526+0000","last_unstale":"2024-04-25T11:34:17.877103+0000","last_undegraded":"2024-04-25T11:34:17.877103+0000","last_fullsized":"2024-04-25T11:34:17.877103+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:05.718720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.875014+0000","last_change":"2024-04-25T11:33:10.886596+0000","last_active":"2024-04-25T11:34:17.875014+0000","last_peered":"2024-04-25T11:34:17.875014+0000","last_clean":"2024-04-25T11:34:17.875014+0000","last_became_active":"2024-04-25T11:33:10.886234+0000","last_became_peered":"2024-04-25T11:33:10.886234+0000","last_unstale":"2024-04-25T11:34:17.875014+0000","last_undegraded":"2024-04-25T11:34:17.875014+0000","last_fullsized":"2024-04-25T11:34:17.875014+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:46.632814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"77'3","reported_seq":23,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.879958+0000","last_change":"2024-04-25T11:33:10.032355+0000","last_active":"2024-04-25T11:34:17.879958+0000","last_peered":"2024-04-25T11:34:17.879958+0000","last_clean":"2024-04-25T11:34:17.879958+0000","last_became_active":"2024-04-25T11:33:10.031965+0000","last_became_peered":"2024-04-25T11:33:10.031965+0000","last_unstale":"2024-04-25T11:34:17.879958+0000","last_undegraded":"2024-04-25T11:34:17.879958+0000","last_fullsized":"2024-04-25T11:34:17.879958+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:08:10.539604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.877282+0000","last_change":"2024-04-25T11:33:10.885708+0000","last_active":"2024-04-25T11:34:17.877282+0000","last_peered":"2024-04-25T11:34:17.877282+0000","last_clean":"2024-04-25T11:34:17.877282+0000","last_became_active":"2024-04-25T11:33:10.885317+0000","last_became_peered":"2024-04-25T11:33:10.885317+0000","last_unstale":"2024-04-25T11:34:17.877282+0000","last_undegraded":"2024-04-25T11:34:17.877282+0000","last_fullsized":"2024-04-25T11:34:17.877282+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:50:08.518922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"77'4","reported_seq":161,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:22.742809+0000","last_change":"2024-04-25T11:33:10.030995+0000","last_active":"2024-04-25T11:34:22.742809+0000","last_peered":"2024-04-25T11:34:22.742809+0000","last_clean":"2024-04-25T11:34:22.742809+0000","last_became_active":"2024-04-25T11:33:10.030670+0000","last_became_peered":"2024-04-25T11:33:10.030670+0000","last_unstale":"2024-04-25T11:34:22.742809+0000","last_undegraded":"2024-04-25T11:34:22.742809+0000","last_fullsized":"2024-04-25T11:34:22.742809+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:39.569787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876806+0000","last_change":"2024-04-25T11:33:10.892927+0000","last_active":"2024-04-25T11:34:17.876806+0000","last_peered":"2024-04-25T11:34:17.876806+0000","last_clean":"2024-04-25T11:34:17.876806+0000","last_became_active":"2024-04-25T11:33:10.892458+0000","last_became_peered":"2024-04-25T11:33:10.892458+0000","last_unstale":"2024-04-25T11:34:17.876806+0000","last_undegraded":"2024-04-25T11:34:17.876806+0000","last_fullsized":"2024-04-25T11:34:17.876806+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:53.121185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"77'4","reported_seq":23,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880660+0000","last_change":"2024-04-25T11:33:10.029728+0000","last_active":"2024-04-25T11:34:17.880660+0000","last_peered":"2024-04-25T11:34:17.880660+0000","last_clean":"2024-04-25T11:34:17.880660+0000","last_became_active":"2024-04-25T11:33:10.029435+0000","last_became_peered":"2024-04-25T11:33:10.029435+0000","last_unstale":"2024-04-25T11:34:17.880660+0000","last_undegraded":"2024-04-25T11:34:17.880660+0000","last_fullsized":"2024-04-25T11:34:17.880660+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:31:21.585870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.879781+0000","last_change":"2024-04-25T11:33:10.895828+0000","last_active":"2024-04-25T11:34:17.879781+0000","last_peered":"2024-04-25T11:34:17.879781+0000","last_clean":"2024-04-25T11:34:17.879781+0000","last_became_active":"2024-04-25T11:33:10.895280+0000","last_became_peered":"2024-04-25T11:33:10.895280+0000","last_unstale":"2024-04-25T11:34:17.879781+0000","last_undegraded":"2024-04-25T11:34:17.879781+0000","last_fullsized":"2024-04-25T11:34:17.879781+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:03.009912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"77'5","reported_seq":25,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880204+0000","last_change":"2024-04-25T11:33:10.036651+0000","last_active":"2024-04-25T11:34:17.880204+0000","last_peered":"2024-04-25T11:34:17.880204+0000","last_clean":"2024-04-25T11:34:17.880204+0000","last_became_active":"2024-04-25T11:33:10.036294+0000","last_became_peered":"2024-04-25T11:33:10.036294+0000","last_unstale":"2024-04-25T11:34:17.880204+0000","last_undegraded":"2024-04-25T11:34:17.880204+0000","last_fullsized":"2024-04-25T11:34:17.880204+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:12:36.566618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876482+0000","last_change":"2024-04-25T11:33:10.890199+0000","last_active":"2024-04-25T11:34:17.876482+0000","last_peered":"2024-04-25T11:34:17.876482+0000","last_clean":"2024-04-25T11:34:17.876482+0000","last_became_active":"2024-04-25T11:33:10.889855+0000","last_became_peered":"2024-04-25T11:33:10.889855+0000","last_unstale":"2024-04-25T11:34:17.876482+0000","last_undegraded":"2024-04-25T11:34:17.876482+0000","last_fullsized":"2024-04-25T11:34:17.876482+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:54:45.476647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"77'30","reported_seq":218,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:23.311060+0000","last_change":"2024-04-25T11:34:18.309247+0000","last_active":"2024-04-25T11:34:23.311060+0000","last_peered":"2024-04-25T11:34:23.311060+0000","last_clean":"2024-04-25T11:34:23.311060+0000","last_became_active":"2024-04-25T11:34:18.299888+0000","last_became_peered":"2024-04-25T11:34:18.299888+0000","last_unstale":"2024-04-25T11:34:23.311060+0000","last_undegraded":"2024-04-25T11:34:23.311060+0000","last_fullsized":"2024-04-25T11:34:23.311060+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:36:06.603720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61046,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":143,"num_read_kb":131,"num_write":32,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.236286+0000","last_change":"2024-04-25T11:33:10.888596+0000","last_active":"2024-04-25T11:34:18.236286+0000","last_peered":"2024-04-25T11:34:18.236286+0000","last_clean":"2024-04-25T11:34:18.236286+0000","last_became_active":"2024-04-25T11:33:10.888226+0000","last_became_peered":"2024-04-25T11:33:10.888226+0000","last_unstale":"2024-04-25T11:34:18.236286+0000","last_undegraded":"2024-04-25T11:34:18.236286+0000","last_fullsized":"2024-04-25T11:34:18.236286+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:30:42.891788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"77'9","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876570+0000","last_change":"2024-04-25T11:33:10.026969+0000","last_active":"2024-04-25T11:34:17.876570+0000","last_peered":"2024-04-25T11:34:17.876570+0000","last_clean":"2024-04-25T11:34:17.876570+0000","last_became_active":"2024-04-25T11:33:10.026591+0000","last_became_peered":"2024-04-25T11:33:10.026591+0000","last_unstale":"2024-04-25T11:34:17.876570+0000","last_undegraded":"2024-04-25T11:34:17.876570+0000","last_fullsized":"2024-04-25T11:34:17.876570+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:50:34.269478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876927+0000","last_change":"2024-04-25T11:33:10.900303+0000","last_active":"2024-04-25T11:34:17.876927+0000","last_peered":"2024-04-25T11:34:17.876927+0000","last_clean":"2024-04-25T11:34:17.876927+0000","last_became_active":"2024-04-25T11:33:10.899946+0000","last_became_peered":"2024-04-25T11:33:10.899946+0000","last_unstale":"2024-04-25T11:34:17.876927+0000","last_undegraded":"2024-04-25T11:34:17.876927+0000","last_fullsized":"2024-04-25T11:34:17.876927+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:33:40.805400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"77'5","reported_seq":25,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.879187+0000","last_change":"2024-04-25T11:33:10.035668+0000","last_active":"2024-04-25T11:34:17.879187+0000","last_peered":"2024-04-25T11:34:17.879187+0000","last_clean":"2024-04-25T11:34:17.879187+0000","last_became_active":"2024-04-25T11:33:10.035241+0000","last_became_peered":"2024-04-25T11:33:10.035241+0000","last_unstale":"2024-04-25T11:34:17.879187+0000","last_undegraded":"2024-04-25T11:34:17.879187+0000","last_fullsized":"2024-04-25T11:34:17.879187+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:58:48.735577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880602+0000","last_change":"2024-04-25T11:33:10.894864+0000","last_active":"2024-04-25T11:34:17.880602+0000","last_peered":"2024-04-25T11:34:17.880602+0000","last_clean":"2024-04-25T11:34:17.880602+0000","last_became_active":"2024-04-25T11:33:10.894491+0000","last_became_peered":"2024-04-25T11:33:10.894491+0000","last_unstale":"2024-04-25T11:34:17.880602+0000","last_undegraded":"2024-04-25T11:34:17.880602+0000","last_fullsized":"2024-04-25T11:34:17.880602+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:21:28.256878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"77'6","reported_seq":96,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:23.746235+0000","last_change":"2024-04-25T11:33:10.029099+0000","last_active":"2024-04-25T11:34:23.746235+0000","last_peered":"2024-04-25T11:34:23.746235+0000","last_clean":"2024-04-25T11:34:23.746235+0000","last_became_active":"2024-04-25T11:33:10.028722+0000","last_became_peered":"2024-04-25T11:33:10.028722+0000","last_unstale":"2024-04-25T11:34:23.746235+0000","last_undegraded":"2024-04-25T11:34:23.746235+0000","last_fullsized":"2024-04-25T11:34:23.746235+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:44:12.261519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880843+0000","last_change":"2024-04-25T11:33:10.888635+0000","last_active":"2024-04-25T11:34:17.880843+0000","last_peered":"2024-04-25T11:34:17.880843+0000","last_clean":"2024-04-25T11:34:17.880843+0000","last_became_active":"2024-04-25T11:33:10.887544+0000","last_became_peered":"2024-04-25T11:33:10.887544+0000","last_unstale":"2024-04-25T11:34:17.880843+0000","last_undegraded":"2024-04-25T11:34:17.880843+0000","last_fullsized":"2024-04-25T11:34:17.880843+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:59:25.459587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"77'17","reported_seq":37,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.235785+0000","last_change":"2024-04-25T11:33:10.023566+0000","last_active":"2024-04-25T11:34:18.235785+0000","last_peered":"2024-04-25T11:34:18.235785+0000","last_clean":"2024-04-25T11:34:18.235785+0000","last_became_active":"2024-04-25T11:33:10.023234+0000","last_became_peered":"2024-04-25T11:33:10.023234+0000","last_unstale":"2024-04-25T11:34:18.235785+0000","last_undegraded":"2024-04-25T11:34:18.235785+0000","last_fullsized":"2024-04-25T11:34:18.235785+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:56:26.342148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":23,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.237663+0000","last_change":"2024-04-25T11:34:18.237663+0000","last_active":"2024-04-25T11:34:18.237663+0000","last_peered":"2024-04-25T11:34:18.237663+0000","last_clean":"2024-04-25T11:34:18.237663+0000","last_became_active":"2024-04-25T11:34:18.236993+0000","last_became_peered":"2024-04-25T11:34:18.236993+0000","last_unstale":"2024-04-25T11:34:18.237663+0000","last_undegraded":"2024-04-25T11:34:18.237663+0000","last_fullsized":"2024-04-25T11:34:18.237663+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:59:47.371685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":22,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.236405+0000","last_change":"2024-04-25T11:33:10.028285+0000","last_active":"2024-04-25T11:34:18.236405+0000","last_peered":"2024-04-25T11:34:18.236405+0000","last_clean":"2024-04-25T11:34:18.236405+0000","last_became_active":"2024-04-25T11:33:10.027959+0000","last_became_peered":"2024-04-25T11:33:10.027959+0000","last_unstale":"2024-04-25T11:34:18.236405+0000","last_undegraded":"2024-04-25T11:34:18.236405+0000","last_fullsized":"2024-04-25T11:34:18.236405+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:44:41.094048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.887356+0000","last_change":"2024-04-25T11:34:17.887356+0000","last_active":"2024-04-25T11:34:17.887356+0000","last_peered":"2024-04-25T11:34:17.887356+0000","last_clean":"2024-04-25T11:34:17.887356+0000","last_became_active":"2024-04-25T11:34:17.886981+0000","last_became_peered":"2024-04-25T11:34:17.886981+0000","last_unstale":"2024-04-25T11:34:17.887356+0000","last_undegraded":"2024-04-25T11:34:17.887356+0000","last_fullsized":"2024-04-25T11:34:17.887356+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:40.573546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"77'1","reported_seq":20,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.877528+0000","last_change":"2024-04-25T11:33:10.027473+0000","last_active":"2024-04-25T11:34:17.877528+0000","last_peered":"2024-04-25T11:34:17.877528+0000","last_clean":"2024-04-25T11:34:17.877528+0000","last_became_active":"2024-04-25T11:33:10.027133+0000","last_became_peered":"2024-04-25T11:33:10.027133+0000","last_unstale":"2024-04-25T11:34:17.877528+0000","last_undegraded":"2024-04-25T11:34:17.877528+0000","last_fullsized":"2024-04-25T11:34:17.877528+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:19:58.338507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"77'1","reported_seq":18,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.879131+0000","last_change":"2024-04-25T11:33:10.892588+0000","last_active":"2024-04-25T11:34:17.879131+0000","last_peered":"2024-04-25T11:34:17.879131+0000","last_clean":"2024-04-25T11:34:17.879131+0000","last_became_active":"2024-04-25T11:33:10.892088+0000","last_became_peered":"2024-04-25T11:33:10.892088+0000","last_unstale":"2024-04-25T11:34:17.879131+0000","last_undegraded":"2024-04-25T11:34:17.879131+0000","last_fullsized":"2024-04-25T11:34:17.879131+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:10.136222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"77'5","reported_seq":93,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:22.742834+0000","last_change":"2024-04-25T11:33:10.030801+0000","last_active":"2024-04-25T11:34:22.742834+0000","last_peered":"2024-04-25T11:34:22.742834+0000","last_clean":"2024-04-25T11:34:22.742834+0000","last_became_active":"2024-04-25T11:33:10.030466+0000","last_became_peered":"2024-04-25T11:33:10.030466+0000","last_unstale":"2024-04-25T11:34:22.742834+0000","last_undegraded":"2024-04-25T11:34:22.742834+0000","last_fullsized":"2024-04-25T11:34:22.742834+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:22:02.922309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.879183+0000","last_change":"2024-04-25T11:33:10.894846+0000","last_active":"2024-04-25T11:34:17.879183+0000","last_peered":"2024-04-25T11:34:17.879183+0000","last_clean":"2024-04-25T11:34:17.879183+0000","last_became_active":"2024-04-25T11:33:10.894456+0000","last_became_peered":"2024-04-25T11:33:10.894456+0000","last_unstale":"2024-04-25T11:34:17.879183+0000","last_undegraded":"2024-04-25T11:34:17.879183+0000","last_fullsized":"2024-04-25T11:34:17.879183+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:11:16.265126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":23,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.882781+0000","last_change":"2024-04-25T11:34:17.882781+0000","last_active":"2024-04-25T11:34:17.882781+0000","last_peered":"2024-04-25T11:34:17.882781+0000","last_clean":"2024-04-25T11:34:17.882781+0000","last_became_active":"2024-04-25T11:34:17.882348+0000","last_became_peered":"2024-04-25T11:34:17.882348+0000","last_unstale":"2024-04-25T11:34:17.882781+0000","last_undegraded":"2024-04-25T11:34:17.882781+0000","last_fullsized":"2024-04-25T11:34:17.882781+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:47:34.385332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"77'5","reported_seq":93,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:22.744142+0000","last_change":"2024-04-25T11:33:10.037834+0000","last_active":"2024-04-25T11:34:22.744142+0000","last_peered":"2024-04-25T11:34:22.744142+0000","last_clean":"2024-04-25T11:34:22.744142+0000","last_became_active":"2024-04-25T11:33:10.037570+0000","last_became_peered":"2024-04-25T11:33:10.037570+0000","last_unstale":"2024-04-25T11:34:22.744142+0000","last_undegraded":"2024-04-25T11:34:22.744142+0000","last_fullsized":"2024-04-25T11:34:22.744142+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:12.795190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.875250+0000","last_change":"2024-04-25T11:33:10.882663+0000","last_active":"2024-04-25T11:34:17.875250+0000","last_peered":"2024-04-25T11:34:17.875250+0000","last_clean":"2024-04-25T11:34:17.875250+0000","last_became_active":"2024-04-25T11:33:10.882300+0000","last_became_peered":"2024-04-25T11:33:10.882300+0000","last_unstale":"2024-04-25T11:34:17.875250+0000","last_undegraded":"2024-04-25T11:34:17.875250+0000","last_fullsized":"2024-04-25T11:34:17.875250+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:58:11.379654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"77'6","reported_seq":28,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.297920+0000","last_change":"2024-04-25T11:33:10.024598+0000","last_active":"2024-04-25T11:34:18.297920+0000","last_peered":"2024-04-25T11:34:18.297920+0000","last_clean":"2024-04-25T11:34:18.297920+0000","last_became_active":"2024-04-25T11:33:10.024229+0000","last_became_peered":"2024-04-25T11:33:10.024229+0000","last_unstale":"2024-04-25T11:34:18.297920+0000","last_undegraded":"2024-04-25T11:34:18.297920+0000","last_fullsized":"2024-04-25T11:34:18.297920+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:42:14.690581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":95,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880316+0000","last_change":"2024-04-25T11:32:01.310202+0000","last_active":"2024-04-25T11:34:17.880316+0000","last_peered":"2024-04-25T11:34:17.880316+0000","last_clean":"2024-04-25T11:34:17.880316+0000","last_became_active":"2024-04-25T11:32:01.305242+0000","last_became_peered":"2024-04-25T11:32:01.305242+0000","last_unstale":"2024-04-25T11:34:17.880316+0000","last_undegraded":"2024-04-25T11:34:17.880316+0000","last_fullsized":"2024-04-25T11:34:17.880316+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:28:43.374096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:28:43.374096+0000","last_clean_scrub_stamp":"2024-04-25T11:28:43.374096+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:56:18.551747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876853+0000","last_change":"2024-04-25T11:33:10.889124+0000","last_active":"2024-04-25T11:34:17.876853+0000","last_peered":"2024-04-25T11:34:17.876853+0000","last_clean":"2024-04-25T11:34:17.876853+0000","last_became_active":"2024-04-25T11:33:10.888790+0000","last_became_peered":"2024-04-25T11:33:10.888790+0000","last_unstale":"2024-04-25T11:34:17.876853+0000","last_undegraded":"2024-04-25T11:34:17.876853+0000","last_fullsized":"2024-04-25T11:34:17.876853+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:33:44.999949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"77'4","reported_seq":23,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880130+0000","last_change":"2024-04-25T11:33:10.031547+0000","last_active":"2024-04-25T11:34:17.880130+0000","last_peered":"2024-04-25T11:34:17.880130+0000","last_clean":"2024-04-25T11:34:17.880130+0000","last_became_active":"2024-04-25T11:33:10.031152+0000","last_became_peered":"2024-04-25T11:33:10.031152+0000","last_unstale":"2024-04-25T11:34:17.880130+0000","last_undegraded":"2024-04-25T11:34:17.880130+0000","last_fullsized":"2024-04-25T11:34:17.880130+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:10.646771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"77'4","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.236346+0000","last_change":"2024-04-25T11:33:10.020367+0000","last_active":"2024-04-25T11:34:18.236346+0000","last_peered":"2024-04-25T11:34:18.236346+0000","last_clean":"2024-04-25T11:34:18.236346+0000","last_became_active":"2024-04-25T11:33:10.020007+0000","last_became_peered":"2024-04-25T11:33:10.020007+0000","last_unstale":"2024-04-25T11:34:18.236346+0000","last_undegraded":"2024-04-25T11:34:18.236346+0000","last_fullsized":"2024-04-25T11:34:18.236346+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:48.134907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.877378+0000","last_change":"2024-04-25T11:33:10.887741+0000","last_active":"2024-04-25T11:34:17.877378+0000","last_peered":"2024-04-25T11:34:17.877378+0000","last_clean":"2024-04-25T11:34:17.877378+0000","last_became_active":"2024-04-25T11:33:10.886455+0000","last_became_peered":"2024-04-25T11:33:10.886455+0000","last_unstale":"2024-04-25T11:34:17.877378+0000","last_undegraded":"2024-04-25T11:34:17.877378+0000","last_fullsized":"2024-04-25T11:34:17.877378+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:37:16.842817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"77'12","reported_seq":31,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.878184+0000","last_change":"2024-04-25T11:33:10.025672+0000","last_active":"2024-04-25T11:34:17.878184+0000","last_peered":"2024-04-25T11:34:17.878184+0000","last_clean":"2024-04-25T11:34:17.878184+0000","last_became_active":"2024-04-25T11:33:10.025232+0000","last_became_peered":"2024-04-25T11:33:10.025232+0000","last_unstale":"2024-04-25T11:34:17.878184+0000","last_undegraded":"2024-04-25T11:34:17.878184+0000","last_fullsized":"2024-04-25T11:34:17.878184+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:06:26.579126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.875733+0000","last_change":"2024-04-25T11:33:10.884692+0000","last_active":"2024-04-25T11:34:17.875733+0000","last_peered":"2024-04-25T11:34:17.875733+0000","last_clean":"2024-04-25T11:34:17.875733+0000","last_became_active":"2024-04-25T11:33:10.884346+0000","last_became_peered":"2024-04-25T11:33:10.884346+0000","last_unstale":"2024-04-25T11:34:17.875733+0000","last_undegraded":"2024-04-25T11:34:17.875733+0000","last_fullsized":"2024-04-25T11:34:17.875733+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:31:00.630410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"77'7","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.297980+0000","last_change":"2024-04-25T11:33:10.033872+0000","last_active":"2024-04-25T11:34:18.297980+0000","last_peered":"2024-04-25T11:34:18.297980+0000","last_clean":"2024-04-25T11:34:18.297980+0000","last_became_active":"2024-04-25T11:33:10.033435+0000","last_became_peered":"2024-04-25T11:33:10.033435+0000","last_unstale":"2024-04-25T11:34:18.297980+0000","last_undegraded":"2024-04-25T11:34:18.297980+0000","last_fullsized":"2024-04-25T11:34:18.297980+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:12.885215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":23,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.881545+0000","last_change":"2024-04-25T11:34:17.881545+0000","last_active":"2024-04-25T11:34:17.881545+0000","last_peered":"2024-04-25T11:34:17.881545+0000","last_clean":"2024-04-25T11:34:17.881545+0000","last_became_active":"2024-04-25T11:34:17.881125+0000","last_became_peered":"2024-04-25T11:34:17.881125+0000","last_unstale":"2024-04-25T11:34:17.881545+0000","last_undegraded":"2024-04-25T11:34:17.881545+0000","last_fullsized":"2024-04-25T11:34:17.881545+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:55:18.404023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"77'2","reported_seq":21,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876299+0000","last_change":"2024-04-25T11:33:10.032384+0000","last_active":"2024-04-25T11:34:17.876299+0000","last_peered":"2024-04-25T11:34:17.876299+0000","last_clean":"2024-04-25T11:34:17.876299+0000","last_became_active":"2024-04-25T11:33:10.032035+0000","last_became_peered":"2024-04-25T11:33:10.032035+0000","last_unstale":"2024-04-25T11:34:17.876299+0000","last_undegraded":"2024-04-25T11:34:17.876299+0000","last_fullsized":"2024-04-25T11:34:17.876299+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:21:41.723120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.236013+0000","last_change":"2024-04-25T11:33:10.893366+0000","last_active":"2024-04-25T11:34:18.236013+0000","last_peered":"2024-04-25T11:34:18.236013+0000","last_clean":"2024-04-25T11:34:18.236013+0000","last_became_active":"2024-04-25T11:33:10.892988+0000","last_became_peered":"2024-04-25T11:33:10.892988+0000","last_unstale":"2024-04-25T11:34:18.236013+0000","last_undegraded":"2024-04-25T11:34:18.236013+0000","last_fullsized":"2024-04-25T11:34:18.236013+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:06.368216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"77'7","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.875184+0000","last_change":"2024-04-25T11:33:10.031456+0000","last_active":"2024-04-25T11:34:17.875184+0000","last_peered":"2024-04-25T11:34:17.875184+0000","last_clean":"2024-04-25T11:34:17.875184+0000","last_became_active":"2024-04-25T11:33:10.031115+0000","last_became_peered":"2024-04-25T11:33:10.031115+0000","last_unstale":"2024-04-25T11:34:17.875184+0000","last_undegraded":"2024-04-25T11:34:17.875184+0000","last_fullsized":"2024-04-25T11:34:17.875184+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:12:56.803277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880640+0000","last_change":"2024-04-25T11:33:10.893692+0000","last_active":"2024-04-25T11:34:17.880640+0000","last_peered":"2024-04-25T11:34:17.880640+0000","last_clean":"2024-04-25T11:34:17.880640+0000","last_became_active":"2024-04-25T11:33:10.893332+0000","last_became_peered":"2024-04-25T11:33:10.893332+0000","last_unstale":"2024-04-25T11:34:17.880640+0000","last_undegraded":"2024-04-25T11:34:17.880640+0000","last_fullsized":"2024-04-25T11:34:17.880640+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:01.277465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"77'3","reported_seq":22,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876399+0000","last_change":"2024-04-25T11:33:10.023804+0000","last_active":"2024-04-25T11:34:17.876399+0000","last_peered":"2024-04-25T11:34:17.876399+0000","last_clean":"2024-04-25T11:34:17.876399+0000","last_became_active":"2024-04-25T11:33:10.023503+0000","last_became_peered":"2024-04-25T11:33:10.023503+0000","last_unstale":"2024-04-25T11:34:17.876399+0000","last_undegraded":"2024-04-25T11:34:17.876399+0000","last_fullsized":"2024-04-25T11:34:17.876399+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:34:48.074176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.883633+0000","last_change":"2024-04-25T11:34:17.882901+0000","last_active":"2024-04-25T11:34:17.883633+0000","last_peered":"2024-04-25T11:34:17.883633+0000","last_clean":"2024-04-25T11:34:17.883633+0000","last_became_active":"2024-04-25T11:34:17.882514+0000","last_became_peered":"2024-04-25T11:34:17.882514+0000","last_unstale":"2024-04-25T11:34:17.883633+0000","last_undegraded":"2024-04-25T11:34:17.883633+0000","last_fullsized":"2024-04-25T11:34:17.883633+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:03:30.843842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,10],"acting":[9,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.b","version":"77'4","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:18.236101+0000","last_change":"2024-04-25T11:33:10.030857+0000","last_active":"2024-04-25T11:34:18.236101+0000","last_peered":"2024-04-25T11:34:18.236101+0000","last_clean":"2024-04-25T11:34:18.236101+0000","last_became_active":"2024-04-25T11:33:10.028879+0000","last_became_peered":"2024-04-25T11:33:10.028879+0000","last_unstale":"2024-04-25T11:34:18.236101+0000","last_undegraded":"2024-04-25T11:34:18.236101+0000","last_fullsized":"2024-04-25T11:34:18.236101+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:28.820919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880411+0000","last_change":"2024-04-25T11:33:10.893773+0000","last_active":"2024-04-25T11:34:17.880411+0000","last_peered":"2024-04-25T11:34:17.880411+0000","last_clean":"2024-04-25T11:34:17.880411+0000","last_became_active":"2024-04-25T11:33:10.893431+0000","last_became_peered":"2024-04-25T11:33:10.893431+0000","last_unstale":"2024-04-25T11:34:17.880411+0000","last_undegraded":"2024-04-25T11:34:17.880411+0000","last_fullsized":"2024-04-25T11:34:17.880411+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:15:15.998951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"77'5","reported_seq":96,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:24.746419+0000","last_change":"2024-04-25T11:33:10.032745+0000","last_active":"2024-04-25T11:34:24.746419+0000","last_peered":"2024-04-25T11:34:24.746419+0000","last_clean":"2024-04-25T11:34:24.746419+0000","last_became_active":"2024-04-25T11:33:10.032361+0000","last_became_peered":"2024-04-25T11:33:10.032361+0000","last_unstale":"2024-04-25T11:34:24.746419+0000","last_undegraded":"2024-04-25T11:34:24.746419+0000","last_fullsized":"2024-04-25T11:34:24.746419+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:14:46.555892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880904+0000","last_change":"2024-04-25T11:33:10.888084+0000","last_active":"2024-04-25T11:34:17.880904+0000","last_peered":"2024-04-25T11:34:17.880904+0000","last_clean":"2024-04-25T11:34:17.880904+0000","last_became_active":"2024-04-25T11:33:10.887679+0000","last_became_peered":"2024-04-25T11:33:10.887679+0000","last_unstale":"2024-04-25T11:34:17.880904+0000","last_undegraded":"2024-04-25T11:34:17.880904+0000","last_fullsized":"2024-04-25T11:34:17.880904+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:10:20.073630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"77'3","reported_seq":23,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.879980+0000","last_change":"2024-04-25T11:33:10.035355+0000","last_active":"2024-04-25T11:34:17.879980+0000","last_peered":"2024-04-25T11:34:17.879980+0000","last_clean":"2024-04-25T11:34:17.879980+0000","last_became_active":"2024-04-25T11:33:10.034724+0000","last_became_peered":"2024-04-25T11:33:10.034724+0000","last_unstale":"2024-04-25T11:34:17.879980+0000","last_undegraded":"2024-04-25T11:34:17.879980+0000","last_fullsized":"2024-04-25T11:34:17.879980+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:24:26.757803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"77'1","reported_seq":18,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880517+0000","last_change":"2024-04-25T11:33:10.899827+0000","last_active":"2024-04-25T11:34:17.880517+0000","last_peered":"2024-04-25T11:34:17.880517+0000","last_clean":"2024-04-25T11:34:17.880517+0000","last_became_active":"2024-04-25T11:33:10.899356+0000","last_became_peered":"2024-04-25T11:33:10.899356+0000","last_unstale":"2024-04-25T11:34:17.880517+0000","last_undegraded":"2024-04-25T11:34:17.880517+0000","last_fullsized":"2024-04-25T11:34:17.880517+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:04:17.323958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"77'5","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.877727+0000","last_change":"2024-04-25T11:33:10.034981+0000","last_active":"2024-04-25T11:34:17.877727+0000","last_peered":"2024-04-25T11:34:17.877727+0000","last_clean":"2024-04-25T11:34:17.877727+0000","last_became_active":"2024-04-25T11:33:10.034601+0000","last_became_peered":"2024-04-25T11:33:10.034601+0000","last_unstale":"2024-04-25T11:34:17.877727+0000","last_undegraded":"2024-04-25T11:34:17.877727+0000","last_fullsized":"2024-04-25T11:34:17.877727+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:19:14.710731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880720+0000","last_change":"2024-04-25T11:33:10.890020+0000","last_active":"2024-04-25T11:34:17.880720+0000","last_peered":"2024-04-25T11:34:17.880720+0000","last_clean":"2024-04-25T11:34:17.880720+0000","last_became_active":"2024-04-25T11:33:10.889558+0000","last_became_peered":"2024-04-25T11:33:10.889558+0000","last_unstale":"2024-04-25T11:34:17.880720+0000","last_undegraded":"2024-04-25T11:34:17.880720+0000","last_fullsized":"2024-04-25T11:34:17.880720+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:00:49.096818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880777+0000","last_change":"2024-04-25T11:33:10.886965+0000","last_active":"2024-04-25T11:34:17.880777+0000","last_peered":"2024-04-25T11:34:17.880777+0000","last_clean":"2024-04-25T11:34:17.880777+0000","last_became_active":"2024-04-25T11:33:10.886594+0000","last_became_peered":"2024-04-25T11:33:10.886594+0000","last_unstale":"2024-04-25T11:34:17.880777+0000","last_undegraded":"2024-04-25T11:34:17.880777+0000","last_fullsized":"2024-04-25T11:34:17.880777+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:58.924857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876483+0000","last_change":"2024-04-25T11:33:10.890760+0000","last_active":"2024-04-25T11:34:17.876483+0000","last_peered":"2024-04-25T11:34:17.876483+0000","last_clean":"2024-04-25T11:34:17.876483+0000","last_became_active":"2024-04-25T11:33:10.889643+0000","last_became_peered":"2024-04-25T11:33:10.889643+0000","last_unstale":"2024-04-25T11:34:17.876483+0000","last_undegraded":"2024-04-25T11:34:17.876483+0000","last_fullsized":"2024-04-25T11:34:17.876483+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:54.644853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876206+0000","last_change":"2024-04-25T11:33:10.887850+0000","last_active":"2024-04-25T11:34:17.876206+0000","last_peered":"2024-04-25T11:34:17.876206+0000","last_clean":"2024-04-25T11:34:17.876206+0000","last_became_active":"2024-04-25T11:33:10.882323+0000","last_became_peered":"2024-04-25T11:33:10.882323+0000","last_unstale":"2024-04-25T11:34:17.876206+0000","last_undegraded":"2024-04-25T11:34:17.876206+0000","last_fullsized":"2024-04-25T11:34:17.876206+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:24:00.509704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.879830+0000","last_change":"2024-04-25T11:33:10.883842+0000","last_active":"2024-04-25T11:34:17.879830+0000","last_peered":"2024-04-25T11:34:17.879830+0000","last_clean":"2024-04-25T11:34:17.879830+0000","last_became_active":"2024-04-25T11:33:10.883513+0000","last_became_peered":"2024-04-25T11:33:10.883513+0000","last_unstale":"2024-04-25T11:34:17.879830+0000","last_undegraded":"2024-04-25T11:34:17.879830+0000","last_fullsized":"2024-04-25T11:34:17.879830+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:47:09.731875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.877269+0000","last_change":"2024-04-25T11:33:10.890503+0000","last_active":"2024-04-25T11:34:17.877269+0000","last_peered":"2024-04-25T11:34:17.877269+0000","last_clean":"2024-04-25T11:34:17.877269+0000","last_became_active":"2024-04-25T11:33:10.890162+0000","last_became_peered":"2024-04-25T11:33:10.890162+0000","last_unstale":"2024-04-25T11:34:17.877269+0000","last_undegraded":"2024-04-25T11:34:17.877269+0000","last_fullsized":"2024-04-25T11:34:17.877269+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:36.462789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880918+0000","last_change":"2024-04-25T11:33:10.897109+0000","last_active":"2024-04-25T11:34:17.880918+0000","last_peered":"2024-04-25T11:34:17.880918+0000","last_clean":"2024-04-25T11:34:17.880918+0000","last_became_active":"2024-04-25T11:33:10.896138+0000","last_became_peered":"2024-04-25T11:33:10.896138+0000","last_unstale":"2024-04-25T11:34:17.880918+0000","last_undegraded":"2024-04-25T11:34:17.880918+0000","last_fullsized":"2024-04-25T11:34:17.880918+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:47:46.202355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876862+0000","last_change":"2024-04-25T11:33:10.890397+0000","last_active":"2024-04-25T11:34:17.876862+0000","last_peered":"2024-04-25T11:34:17.876862+0000","last_clean":"2024-04-25T11:34:17.876862+0000","last_became_active":"2024-04-25T11:33:10.889580+0000","last_became_peered":"2024-04-25T11:33:10.889580+0000","last_unstale":"2024-04-25T11:34:17.876862+0000","last_undegraded":"2024-04-25T11:34:17.876862+0000","last_fullsized":"2024-04-25T11:34:17.876862+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:16:09.672131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.875702+0000","last_change":"2024-04-25T11:33:10.873817+0000","last_active":"2024-04-25T11:34:17.875702+0000","last_peered":"2024-04-25T11:34:17.875702+0000","last_clean":"2024-04-25T11:34:17.875702+0000","last_became_active":"2024-04-25T11:33:10.873425+0000","last_became_peered":"2024-04-25T11:33:10.873425+0000","last_unstale":"2024-04-25T11:34:17.875702+0000","last_undegraded":"2024-04-25T11:34:17.875702+0000","last_fullsized":"2024-04-25T11:34:17.875702+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:52:26.279705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880651+0000","last_change":"2024-04-25T11:34:17.880075+0000","last_active":"2024-04-25T11:34:17.880651+0000","last_peered":"2024-04-25T11:34:17.880651+0000","last_clean":"2024-04-25T11:34:17.880651+0000","last_became_active":"2024-04-25T11:34:17.879692+0000","last_became_peered":"2024-04-25T11:34:17.879692+0000","last_unstale":"2024-04-25T11:34:17.880651+0000","last_undegraded":"2024-04-25T11:34:17.880651+0000","last_fullsized":"2024-04-25T11:34:17.880651+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:54.833601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,3],"acting":[6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.875168+0000","last_change":"2024-04-25T11:33:10.883690+0000","last_active":"2024-04-25T11:34:17.875168+0000","last_peered":"2024-04-25T11:34:17.875168+0000","last_clean":"2024-04-25T11:34:17.875168+0000","last_became_active":"2024-04-25T11:33:10.882965+0000","last_became_peered":"2024-04-25T11:33:10.882965+0000","last_unstale":"2024-04-25T11:34:17.875168+0000","last_undegraded":"2024-04-25T11:34:17.875168+0000","last_fullsized":"2024-04-25T11:34:17.875168+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:28.126631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.876395+0000","last_change":"2024-04-25T11:33:10.897050+0000","last_active":"2024-04-25T11:34:17.876395+0000","last_peered":"2024-04-25T11:34:17.876395+0000","last_clean":"2024-04-25T11:34:17.876395+0000","last_became_active":"2024-04-25T11:33:10.896273+0000","last_became_peered":"2024-04-25T11:33:10.896273+0000","last_unstale":"2024-04-25T11:34:17.876395+0000","last_undegraded":"2024-04-25T11:34:17.876395+0000","last_fullsized":"2024-04-25T11:34:17.876395+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:45:53.839823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.877808+0000","last_change":"2024-04-25T11:33:10.897323+0000","last_active":"2024-04-25T11:34:17.877808+0000","last_peered":"2024-04-25T11:34:17.877808+0000","last_clean":"2024-04-25T11:34:17.877808+0000","last_became_active":"2024-04-25T11:33:10.896951+0000","last_became_peered":"2024-04-25T11:33:10.896951+0000","last_unstale":"2024-04-25T11:34:17.877808+0000","last_undegraded":"2024-04-25T11:34:17.877808+0000","last_fullsized":"2024-04-25T11:34:17.877808+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:24:31.208338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-25T11:34:17.880816+0000","last_change":"2024-04-25T11:33:10.897566+0000","last_active":"2024-04-25T11:34:17.880816+0000","last_peered":"2024-04-25T11:34:17.880816+0000","last_clean":"2024-04-25T11:34:17.880816+0000","last_became_active":"2024-04-25T11:33:10.897212+0000","last_became_peered":"2024-04-25T11:33:10.897212+0000","last_unstale":"2024-04-25T11:34:17.880816+0000","last_undegraded":"2024-04-25T11:34:17.880816+0000","last_fullsized":"2024-04-25T11:34:17.880816+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:55:29.995287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67859,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":724,"num_read_kb":434,"num_write":236,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":203577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645340,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36188,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708964,"statfs":{"total":95995035648,"available":95957979136,"internally_reserved":0,"allocated":868352,"data_stored":413865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841569,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32716,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712436,"statfs":{"total":95995035648,"available":95961534464,"internally_reserved":0,"allocated":1568768,"data_stored":1066331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037797,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":413795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234027,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":888832,"data_stored":414173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430256,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32636,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712516,"statfs":{"total":95995035648,"available":95961616384,"internally_reserved":0,"allocated":1486848,"data_stored":1006691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626485,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32036,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713116,"statfs":{"total":95995035648,"available":95962230784,"internally_reserved":0,"allocated":876544,"data_stored":413512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36260,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708892,"statfs":{"total":95995035648,"available":95957905408,"internally_reserved":0,"allocated":937984,"data_stored":475471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":897024,"data_stored":414851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411401,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40372,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704780,"statfs":{"total":95995035648,"available":95953694720,"internally_reserved":0,"allocated":888832,"data_stored":414645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182467,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":892928,"data_stored":413337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986239,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":946176,"data_stored":474810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822714,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28480,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716672,"statfs":{"total":95995035648,"available":95965872128,"internally_reserved":0,"allocated":1499136,"data_stored":1005978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":62798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":62137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T11:34:29.070 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-25T11:34:29.071 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T11:34:29.071 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-25T11:34:29.071 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-25T11:34:29.308 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:34:30.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:29 smithi156 ceph-mon[33045]: from='client.14976 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T11:34:30.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:29 smithi156 ceph-mon[33045]: pgmap v315: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 10 op/s; 5.5 KiB/s, 0 objects/s recovering 2024-04-25T11:34:30.098 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:29 smithi090 ceph-mon[29575]: from='client.14976 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T11:34:30.098 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:29 smithi090 ceph-mon[29575]: pgmap v315: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 10 op/s; 5.5 KiB/s, 0 objects/s recovering 2024-04-25T11:34:30.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:29 smithi138 ceph-mon[32328]: from='client.14976 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T11:34:30.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:29 smithi138 ceph-mon[32328]: pgmap v315: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 10 op/s; 5.5 KiB/s, 0 objects/s recovering 2024-04-25T11:34:30.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.849+0000 7fa72a2e9700 1 -- 172.21.15.90:0/689631262 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa724109720 msgr2=0x7fa724111080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:30.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.849+0000 7fa72a2e9700 1 --2- 172.21.15.90:0/689631262 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa724109720 0x7fa724111080 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa71c00b0d0 tx=0x7fa71c00b3e0 comp rx=0 tx=0).stop 2024-04-25T11:34:30.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.849+0000 7fa72a2e9700 1 -- 172.21.15.90:0/689631262 shutdown_connections 2024-04-25T11:34:30.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.849+0000 7fa72a2e9700 1 --2- 172.21.15.90:0/689631262 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa724109720 0x7fa724111080 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:30.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.849+0000 7fa72a2e9700 1 --2- 172.21.15.90:0/689631262 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa7241029b0 0x7fa724102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:30.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.849+0000 7fa72a2e9700 1 --2- 172.21.15.90:0/689631262 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa724108dd0 0x7fa7241091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:30.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.849+0000 7fa72a2e9700 1 -- 172.21.15.90:0/689631262 >> 172.21.15.90:0/689631262 conn(0x7fa7240fe3f0 msgr2=0x7fa724100840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:30.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.850+0000 7fa72a2e9700 1 -- 172.21.15.90:0/689631262 shutdown_connections 2024-04-25T11:34:30.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.850+0000 7fa72a2e9700 1 -- 172.21.15.90:0/689631262 wait complete. 2024-04-25T11:34:30.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.850+0000 7fa72a2e9700 1 Processor -- start 2024-04-25T11:34:30.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.850+0000 7fa72a2e9700 1 -- start start 2024-04-25T11:34:30.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.851+0000 7fa72a2e9700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa7241029b0 0x7fa7241a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:30.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.851+0000 7fa72a2e9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa724108dd0 0x7fa7241a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:30.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.851+0000 7fa72a2e9700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa724109720 0x7fa72419aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:30.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.851+0000 7fa72a2e9700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa724114350 con 0x7fa724108dd0 2024-04-25T11:34:30.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.851+0000 7fa72a2e9700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa7241141d0 con 0x7fa7241029b0 2024-04-25T11:34:30.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.851+0000 7fa72a2e9700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa7241144d0 con 0x7fa724109720 2024-04-25T11:34:30.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.851+0000 7fa728886700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa724109720 0x7fa72419aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:30.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.851+0000 7fa7237fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa724108dd0 0x7fa7241a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:30.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.851+0000 7fa728886700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa724109720 0x7fa72419aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:46950/0 (socket says 172.21.15.90:46950) 2024-04-25T11:34:30.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.851+0000 7fa7237fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa724108dd0 0x7fa7241a0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57650/0 (socket says 172.21.15.90:57650) 2024-04-25T11:34:30.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.851+0000 7fa728886700 1 -- 172.21.15.90:0/207035012 learned_addr learned my addr 172.21.15.90:0/207035012 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:30.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.851+0000 7fa723fff700 1 --2- 172.21.15.90:0/207035012 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa7241029b0 0x7fa7241a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:30.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.851+0000 7fa7237fe700 1 -- 172.21.15.90:0/207035012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa724109720 msgr2=0x7fa72419aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:30.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.851+0000 7fa7237fe700 1 --2- 172.21.15.90:0/207035012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa724109720 0x7fa72419aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:30.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.851+0000 7fa7237fe700 1 -- 172.21.15.90:0/207035012 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa7241029b0 msgr2=0x7fa7241a0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:30.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.851+0000 7fa7237fe700 1 --2- 172.21.15.90:0/207035012 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa7241029b0 0x7fa7241a0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:30.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.851+0000 7fa7237fe700 1 -- 172.21.15.90:0/207035012 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa72419b0d0 con 0x7fa724108dd0 2024-04-25T11:34:30.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.851+0000 7fa723fff700 1 --2- 172.21.15.90:0/207035012 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa7241029b0 0x7fa7241a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:30.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.851+0000 7fa7237fe700 1 --2- 172.21.15.90:0/207035012 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa724108dd0 0x7fa7241a0db0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa71800c940 tx=0x7fa71800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:30.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.852+0000 7fa7217fa700 1 -- 172.21.15.90:0/207035012 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa718014070 con 0x7fa724108dd0 2024-04-25T11:34:30.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.852+0000 7fa7217fa700 1 -- 172.21.15.90:0/207035012 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa71800ce30 con 0x7fa724108dd0 2024-04-25T11:34:30.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.852+0000 7fa72a2e9700 1 -- 172.21.15.90:0/207035012 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa72419b3c0 con 0x7fa724108dd0 2024-04-25T11:34:30.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.852+0000 7fa7217fa700 1 -- 172.21.15.90:0/207035012 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa718003ab0 con 0x7fa724108dd0 2024-04-25T11:34:30.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.852+0000 7fa72a2e9700 1 -- 172.21.15.90:0/207035012 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7241a7700 con 0x7fa724108dd0 2024-04-25T11:34:30.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.853+0000 7fa72a2e9700 1 -- 172.21.15.90:0/207035012 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa724051190 con 0x7fa724108dd0 2024-04-25T11:34:30.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.855+0000 7fa7217fa700 1 -- 172.21.15.90:0/207035012 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fa718017070 con 0x7fa724108dd0 2024-04-25T11:34:30.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.856+0000 7fa7217fa700 1 --2- 172.21.15.90:0/207035012 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fa70c0afc10 0x7fa70c0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:30.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.856+0000 7fa723fff700 1 --2- 172.21.15.90:0/207035012 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fa70c0afc10 0x7fa70c0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:30.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.856+0000 7fa7217fa700 1 -- 172.21.15.90:0/207035012 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7fa7180dee30 con 0x7fa724108dd0 2024-04-25T11:34:30.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.856+0000 7fa723fff700 1 --2- 172.21.15.90:0/207035012 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fa70c0afc10 0x7fa70c0b2120 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa714005950 tx=0x7fa714005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:30.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:30.859+0000 7fa7217fa700 1 -- 172.21.15.90:0/207035012 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa71801e030 con 0x7fa724108dd0 2024-04-25T11:34:31.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.106+0000 7fa72a2e9700 1 -- 172.21.15.90:0/207035012 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fa72419bfc0 con 0x7fa724108dd0 2024-04-25T11:34:31.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.106+0000 7fa7217fa700 1 -- 172.21.15.90:0/207035012 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fa7180a7890 con 0x7fa724108dd0 2024-04-25T11:34:31.095 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:34:31.095 INFO:teuthology.orchestra.run.smithi090.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T11:34:31.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.108+0000 7fa72a2e9700 1 -- 172.21.15.90:0/207035012 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fa70c0afc10 msgr2=0x7fa70c0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:31.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.108+0000 7fa72a2e9700 1 --2- 172.21.15.90:0/207035012 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fa70c0afc10 0x7fa70c0b2120 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa714005950 tx=0x7fa714005f50 comp rx=0 tx=0).stop 2024-04-25T11:34:31.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.108+0000 7fa72a2e9700 1 -- 172.21.15.90:0/207035012 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa724108dd0 msgr2=0x7fa7241a0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:31.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.108+0000 7fa72a2e9700 1 --2- 172.21.15.90:0/207035012 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa724108dd0 0x7fa7241a0db0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa71800c940 tx=0x7fa71800cc50 comp rx=0 tx=0).stop 2024-04-25T11:34:31.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.108+0000 7fa72a2e9700 1 -- 172.21.15.90:0/207035012 shutdown_connections 2024-04-25T11:34:31.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.108+0000 7fa72a2e9700 1 --2- 172.21.15.90:0/207035012 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fa70c0afc10 0x7fa70c0b2120 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:31.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.108+0000 7fa72a2e9700 1 --2- 172.21.15.90:0/207035012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa724109720 0x7fa72419aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:31.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.108+0000 7fa72a2e9700 1 --2- 172.21.15.90:0/207035012 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa7241029b0 0x7fa7241a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:31.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.108+0000 7fa72a2e9700 1 --2- 172.21.15.90:0/207035012 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa724108dd0 0x7fa7241a0db0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:31.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.108+0000 7fa72a2e9700 1 -- 172.21.15.90:0/207035012 >> 172.21.15.90:0/207035012 conn(0x7fa7240fe3f0 msgr2=0x7fa724108540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:31.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.109+0000 7fa72a2e9700 1 -- 172.21.15.90:0/207035012 shutdown_connections 2024-04-25T11:34:31.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.109+0000 7fa72a2e9700 1 -- 172.21.15.90:0/207035012 wait complete. 2024-04-25T11:34:31.762 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-25T11:34:31.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:34:31.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.886+0000 7fbaa18fa640 1 -- 172.21.15.90:0/421535847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba9c1029e0 msgr2=0x7fba9c1036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:31.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.886+0000 7fbaa18fa640 1 --2- 172.21.15.90:0/421535847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba9c1029e0 0x7fba9c1036e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fba8c009900 tx=0x7fba8c02a410 comp rx=0 tx=0).stop 2024-04-25T11:34:31.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.886+0000 7fbaa18fa640 1 -- 172.21.15.90:0/421535847 shutdown_connections 2024-04-25T11:34:31.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.886+0000 7fbaa18fa640 1 --2- 172.21.15.90:0/421535847 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba9c108920 0x7fba9c10ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:31.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.886+0000 7fbaa18fa640 1 --2- 172.21.15.90:0/421535847 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba9c103c20 0x7fba9c104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:31.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.886+0000 7fbaa18fa640 1 --2- 172.21.15.90:0/421535847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba9c1029e0 0x7fba9c1036e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:31.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.887+0000 7fbaa18fa640 1 -- 172.21.15.90:0/421535847 >> 172.21.15.90:0/421535847 conn(0x7fba9c0fdbd0 msgr2=0x7fba9c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:31.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.887+0000 7fbaa18fa640 1 -- 172.21.15.90:0/421535847 shutdown_connections 2024-04-25T11:34:31.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.887+0000 7fbaa18fa640 1 -- 172.21.15.90:0/421535847 wait complete. 2024-04-25T11:34:31.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.887+0000 7fbaa18fa640 1 Processor -- start 2024-04-25T11:34:31.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.887+0000 7fbaa18fa640 1 -- start start 2024-04-25T11:34:31.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.887+0000 7fbaa18fa640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba9c1029e0 0x7fba9c199230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:31.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.888+0000 7fbaa18fa640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba9c103c20 0x7fba9c199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:31.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.888+0000 7fbaa18fa640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba9c108920 0x7fba9c1a0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:31.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.888+0000 7fbaa18fa640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fba9c10e540 con 0x7fba9c103c20 2024-04-25T11:34:31.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.888+0000 7fbaa18fa640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fba9c10e3c0 con 0x7fba9c1029e0 2024-04-25T11:34:31.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.888+0000 7fbaa18fa640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fba9c10e6c0 con 0x7fba9c108920 2024-04-25T11:34:31.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.888+0000 7fba9a7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba9c103c20 0x7fba9c199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:31.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.888+0000 7fba9a7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba9c103c20 0x7fba9c199770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34664/0 (socket says 172.21.15.90:34664) 2024-04-25T11:34:31.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.888+0000 7fba9a7fc640 1 -- 172.21.15.90:0/3758987068 learned_addr learned my addr 172.21.15.90:0/3758987068 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:31.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.888+0000 7fba9b7fe640 1 --2- 172.21.15.90:0/3758987068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba9c108920 0x7fba9c1a0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:31.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.888+0000 7fba9affd640 1 --2- 172.21.15.90:0/3758987068 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba9c1029e0 0x7fba9c199230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:31.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.888+0000 7fba9a7fc640 1 -- 172.21.15.90:0/3758987068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba9c108920 msgr2=0x7fba9c1a0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:31.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.888+0000 7fba9a7fc640 1 --2- 172.21.15.90:0/3758987068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba9c108920 0x7fba9c1a0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:31.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.888+0000 7fba9a7fc640 1 -- 172.21.15.90:0/3758987068 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba9c1029e0 msgr2=0x7fba9c199230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:31.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.888+0000 7fba9a7fc640 1 --2- 172.21.15.90:0/3758987068 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba9c1029e0 0x7fba9c199230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:31.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.888+0000 7fba9a7fc640 1 -- 172.21.15.90:0/3758987068 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba9c1a0d70 con 0x7fba9c103c20 2024-04-25T11:34:31.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.888+0000 7fba9a7fc640 1 --2- 172.21.15.90:0/3758987068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba9c103c20 0x7fba9c199770 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fba9000b4a0 tx=0x7fba9000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:31.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.888+0000 7fbaa08f8640 1 -- 172.21.15.90:0/3758987068 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba90014070 con 0x7fba9c103c20 2024-04-25T11:34:31.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.888+0000 7fbaa08f8640 1 -- 172.21.15.90:0/3758987068 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba90010ac0 con 0x7fba9c103c20 2024-04-25T11:34:31.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.888+0000 7fbaa18fa640 1 -- 172.21.15.90:0/3758987068 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba9c1a1060 con 0x7fba9c103c20 2024-04-25T11:34:31.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.888+0000 7fbaa08f8640 1 -- 172.21.15.90:0/3758987068 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba90017c30 con 0x7fba9c103c20 2024-04-25T11:34:31.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.888+0000 7fbaa18fa640 1 -- 172.21.15.90:0/3758987068 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba9c1a1630 con 0x7fba9c103c20 2024-04-25T11:34:31.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.889+0000 7fbaa18fa640 1 -- 172.21.15.90:0/3758987068 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba9c107730 con 0x7fba9c103c20 2024-04-25T11:34:31.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.891+0000 7fbaa08f8640 1 -- 172.21.15.90:0/3758987068 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fba90017070 con 0x7fba9c103c20 2024-04-25T11:34:31.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.891+0000 7fbaa08f8640 1 --2- 172.21.15.90:0/3758987068 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fba840afb80 0x7fba840b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:31.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.891+0000 7fba9affd640 1 --2- 172.21.15.90:0/3758987068 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fba840afb80 0x7fba840b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:31.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.891+0000 7fbaa08f8640 1 -- 172.21.15.90:0/3758987068 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7fba900de3d0 con 0x7fba9c103c20 2024-04-25T11:34:31.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.892+0000 7fba9affd640 1 --2- 172.21.15.90:0/3758987068 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fba840afb80 0x7fba840b2090 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fba8c009230 tx=0x7fba8c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:31.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:31.895+0000 7fbaa08f8640 1 -- 172.21.15.90:0/3758987068 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fba900a6f40 con 0x7fba9c103c20 2024-04-25T11:34:32.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.072+0000 7fbaa18fa640 1 -- 172.21.15.90:0/3758987068 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fba9c02dd20 con 0x7fba9c103c20 2024-04-25T11:34:32.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.074+0000 7fbaa08f8640 1 -- 172.21.15.90:0/3758987068 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+22706 (secure 0 0 0) 0x7fba900a6840 con 0x7fba9c103c20 2024-04-25T11:34:32.064 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:34:32.064 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":79,"fsid":"58bd4e7a-02f6-11ef-bc93-c7b262605968","created":"2024-04-25T11:25:23.574550+0000","modified":"2024-04-25T11:34:17.871834+0000","last_up_change":"2024-04-25T11:32:23.852211+0000","last_in_change":"2024-04-25T11:32:04.188507+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:28:42.746649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:33:08.442925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:33:09.352886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e54d888-e67b-4951-8ae7-e355fb6b10a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6803","nonce":1201629159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6805","nonce":1201629159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6809","nonce":1201629159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6807","nonce":1201629159}]},"public_addr":"172.21.15.90:6803/1201629159","cluster_addr":"172.21.15.90:6805/1201629159","heartbeat_back_addr":"172.21.15.90:6809/1201629159","heartbeat_front_addr":"172.21.15.90:6807/1201629159","state":["exists","up"]},{"osd":1,"uuid":"91a71c78-dae9-4bc7-9f6f-9e55cfefbe28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6801","nonce":3976851933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6803","nonce":3976851933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6807","nonce":3976851933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6805","nonce":3976851933}]},"public_addr":"172.21.15.138:6801/3976851933","cluster_addr":"172.21.15.138:6803/3976851933","heartbeat_back_addr":"172.21.15.138:6807/3976851933","heartbeat_front_addr":"172.21.15.138:6805/3976851933","state":["exists","up"]},{"osd":2,"uuid":"daff6c7f-80fa-4859-9339-ef7a5e840832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6801","nonce":1505807670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6803","nonce":1505807670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6807","nonce":1505807670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6805","nonce":1505807670}]},"public_addr":"172.21.15.156:6801/1505807670","cluster_addr":"172.21.15.156:6803/1505807670","heartbeat_back_addr":"172.21.15.156:6807/1505807670","heartbeat_front_addr":"172.21.15.156:6805/1505807670","state":["exists","up"]},{"osd":3,"uuid":"33fd94ca-7465-401a-bc1e-706b6b31b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6811","nonce":2520012383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6813","nonce":2520012383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6817","nonce":2520012383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6815","nonce":2520012383}]},"public_addr":"172.21.15.90:6811/2520012383","cluster_addr":"172.21.15.90:6813/2520012383","heartbeat_back_addr":"172.21.15.90:6817/2520012383","heartbeat_front_addr":"172.21.15.90:6815/2520012383","state":["exists","up"]},{"osd":4,"uuid":"c5326212-750c-49fd-96e3-d1e62bb4b7f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6809","nonce":3752678279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6811","nonce":3752678279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6815","nonce":3752678279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6813","nonce":3752678279}]},"public_addr":"172.21.15.138:6809/3752678279","cluster_addr":"172.21.15.138:6811/3752678279","heartbeat_back_addr":"172.21.15.138:6815/3752678279","heartbeat_front_addr":"172.21.15.138:6813/3752678279","state":["exists","up"]},{"osd":5,"uuid":"d04a1156-820b-4b43-be89-e16a8910c3aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6809","nonce":3591210191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6811","nonce":3591210191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6815","nonce":3591210191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6813","nonce":3591210191}]},"public_addr":"172.21.15.156:6809/3591210191","cluster_addr":"172.21.15.156:6811/3591210191","heartbeat_back_addr":"172.21.15.156:6815/3591210191","heartbeat_front_addr":"172.21.15.156:6813/3591210191","state":["exists","up"]},{"osd":6,"uuid":"75675aa3-e978-490c-8205-13d9868a3d6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6819","nonce":2364955456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6821","nonce":2364955456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6825","nonce":2364955456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6823","nonce":2364955456}]},"public_addr":"172.21.15.90:6819/2364955456","cluster_addr":"172.21.15.90:6821/2364955456","heartbeat_back_addr":"172.21.15.90:6825/2364955456","heartbeat_front_addr":"172.21.15.90:6823/2364955456","state":["exists","up"]},{"osd":7,"uuid":"1a95fdf5-f548-4271-bbcd-e330f7955eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6817","nonce":4012476387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6819","nonce":4012476387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6823","nonce":4012476387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6821","nonce":4012476387}]},"public_addr":"172.21.15.138:6817/4012476387","cluster_addr":"172.21.15.138:6819/4012476387","heartbeat_back_addr":"172.21.15.138:6823/4012476387","heartbeat_front_addr":"172.21.15.138:6821/4012476387","state":["exists","up"]},{"osd":8,"uuid":"605e10b5-5806-4c1f-8e28-a7994d34ddce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6817","nonce":101092224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6819","nonce":101092224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6823","nonce":101092224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6821","nonce":101092224}]},"public_addr":"172.21.15.156:6817/101092224","cluster_addr":"172.21.15.156:6819/101092224","heartbeat_back_addr":"172.21.15.156:6823/101092224","heartbeat_front_addr":"172.21.15.156:6821/101092224","state":["exists","up"]},{"osd":9,"uuid":"d70fdf5e-9d0f-483d-a7d8-505b79b8a418","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6827","nonce":1171265966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6829","nonce":1171265966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6833","nonce":1171265966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6831","nonce":1171265966}]},"public_addr":"172.21.15.90:6827/1171265966","cluster_addr":"172.21.15.90:6829/1171265966","heartbeat_back_addr":"172.21.15.90:6833/1171265966","heartbeat_front_addr":"172.21.15.90:6831/1171265966","state":["exists","up"]},{"osd":10,"uuid":"1ec86c7d-c3a4-432d-9664-8e40836e1e20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6825","nonce":4264666232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6827","nonce":4264666232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6831","nonce":4264666232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6829","nonce":4264666232}]},"public_addr":"172.21.15.138:6825/4264666232","cluster_addr":"172.21.15.138:6827/4264666232","heartbeat_back_addr":"172.21.15.138:6831/4264666232","heartbeat_front_addr":"172.21.15.138:6829/4264666232","state":["exists","up"]},{"osd":11,"uuid":"51cbe729-292e-4d7b-b8e9-e1e2821ffe76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6825","nonce":1224251744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6827","nonce":1224251744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6831","nonce":1224251744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6829","nonce":1224251744}]},"public_addr":"172.21.15.156:6825/1224251744","cluster_addr":"172.21.15.156:6827/1224251744","heartbeat_back_addr":"172.21.15.156:6831/1224251744","heartbeat_front_addr":"172.21.15.156:6829/1224251744","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:27:50.794226+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:15.030278+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:38.518163+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:03.107351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:27.500707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:51.463545+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:16.015076+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:41.182071+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:05.362309+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:31.048936+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:56.488509+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:32:22.129178+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":7}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/2381633953":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6800/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/2190846900":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1707792941":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6800/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6801/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/6061052":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1849633392":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6801/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/2277490702":"2024-04-26T11:26:16.655278+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:34:32.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.076+0000 7fbaa18fa640 1 -- 172.21.15.90:0/3758987068 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fba840afb80 msgr2=0x7fba840b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:32.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.076+0000 7fbaa18fa640 1 --2- 172.21.15.90:0/3758987068 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fba840afb80 0x7fba840b2090 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fba8c009230 tx=0x7fba8c031400 comp rx=0 tx=0).stop 2024-04-25T11:34:32.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.076+0000 7fbaa18fa640 1 -- 172.21.15.90:0/3758987068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba9c103c20 msgr2=0x7fba9c199770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:32.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.076+0000 7fbaa18fa640 1 --2- 172.21.15.90:0/3758987068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba9c103c20 0x7fba9c199770 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fba9000b4a0 tx=0x7fba9000b960 comp rx=0 tx=0).stop 2024-04-25T11:34:32.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.076+0000 7fbaa18fa640 1 -- 172.21.15.90:0/3758987068 shutdown_connections 2024-04-25T11:34:32.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.076+0000 7fbaa18fa640 1 --2- 172.21.15.90:0/3758987068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba9c108920 0x7fba9c1a0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.076+0000 7fbaa18fa640 1 --2- 172.21.15.90:0/3758987068 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fba840afb80 0x7fba840b2090 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.076+0000 7fbaa18fa640 1 --2- 172.21.15.90:0/3758987068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba9c103c20 0x7fba9c199770 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.076+0000 7fbaa18fa640 1 --2- 172.21.15.90:0/3758987068 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba9c1029e0 0x7fba9c199230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.076+0000 7fbaa18fa640 1 -- 172.21.15.90:0/3758987068 >> 172.21.15.90:0/3758987068 conn(0x7fba9c0fdbd0 msgr2=0x7fba9c104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:32.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.076+0000 7fbaa18fa640 1 -- 172.21.15.90:0/3758987068 shutdown_connections 2024-04-25T11:34:32.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.076+0000 7fbaa18fa640 1 -- 172.21.15.90:0/3758987068 wait complete. 2024-04-25T11:34:32.075 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:31 smithi090 ceph-mon[29575]: pgmap v316: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T11:34:32.075 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:31 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/207035012' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T11:34:32.076 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T11:28:42.746649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T11:33:08.442925+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T11:33:09.352886+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T11:34:32.077 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T11:34:32.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:31 smithi156 ceph-mon[33045]: pgmap v316: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T11:34:32.089 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:31 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/207035012' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T11:34:32.104 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:31 smithi138 ceph-mon[32328]: pgmap v316: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T11:34:32.104 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:31 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/207035012' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T11:34:32.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.199+0000 7f3101b4d640 1 -- 172.21.15.90:0/3245656062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30fc109a30 msgr2=0x7f30fc10be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:32.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.199+0000 7f3101b4d640 1 --2- 172.21.15.90:0/3245656062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30fc109a30 0x7f30fc10be50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f30f4009ca0 tx=0x7f30f402a8c0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.199+0000 7f3101b4d640 1 -- 172.21.15.90:0/3245656062 shutdown_connections 2024-04-25T11:34:32.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.199+0000 7f3101b4d640 1 --2- 172.21.15.90:0/3245656062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30fc109a30 0x7f30fc10be50 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.199+0000 7f3101b4d640 1 --2- 172.21.15.90:0/3245656062 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30fc104d30 0x7f30fc105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.199+0000 7f3101b4d640 1 --2- 172.21.15.90:0/3245656062 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f30fc103af0 0x7f30fc1047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.199+0000 7f3101b4d640 1 -- 172.21.15.90:0/3245656062 >> 172.21.15.90:0/3245656062 conn(0x7f30fc0fece0 msgr2=0x7f30fc101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:32.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.199+0000 7f3101b4d640 1 -- 172.21.15.90:0/3245656062 shutdown_connections 2024-04-25T11:34:32.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.199+0000 7f3101b4d640 1 -- 172.21.15.90:0/3245656062 wait complete. 2024-04-25T11:34:32.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.199+0000 7f3101b4d640 1 Processor -- start 2024-04-25T11:34:32.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.199+0000 7f3101b4d640 1 -- start start 2024-04-25T11:34:32.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.200+0000 7f3101b4d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f30fc103af0 0x7f30fc19a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:32.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.200+0000 7f3101b4d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30fc104d30 0x7f30fc19a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:32.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.200+0000 7f3101b4d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30fc109a30 0x7f30fc1a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:32.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.200+0000 7f3101b4d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f30fc10f620 con 0x7f30fc104d30 2024-04-25T11:34:32.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.200+0000 7f3101b4d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f30fc10f4a0 con 0x7f30fc103af0 2024-04-25T11:34:32.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.200+0000 7f3101b4d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f30fc10f7a0 con 0x7f30fc109a30 2024-04-25T11:34:32.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.200+0000 7f30f3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30fc104d30 0x7f30fc19a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:32.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.200+0000 7f30f3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30fc104d30 0x7f30fc19a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34672/0 (socket says 172.21.15.90:34672) 2024-04-25T11:34:32.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.200+0000 7f30f3fff640 1 -- 172.21.15.90:0/3177550647 learned_addr learned my addr 172.21.15.90:0/3177550647 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:32.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.200+0000 7f30fbfff640 1 --2- 172.21.15.90:0/3177550647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30fc109a30 0x7f30fc1a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:32.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.200+0000 7f30fb7fe640 1 --2- 172.21.15.90:0/3177550647 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f30fc103af0 0x7f30fc19a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:32.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.200+0000 7f30f3fff640 1 -- 172.21.15.90:0/3177550647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30fc109a30 msgr2=0x7f30fc1a1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:32.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.200+0000 7f30f3fff640 1 --2- 172.21.15.90:0/3177550647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30fc109a30 0x7f30fc1a1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.200+0000 7f30f3fff640 1 -- 172.21.15.90:0/3177550647 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f30fc103af0 msgr2=0x7f30fc19a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:32.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.200+0000 7f30f3fff640 1 --2- 172.21.15.90:0/3177550647 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f30fc103af0 0x7f30fc19a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.200+0000 7f30f3fff640 1 -- 172.21.15.90:0/3177550647 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30fc1a1c20 con 0x7f30fc104d30 2024-04-25T11:34:32.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.200+0000 7f30fbfff640 1 --2- 172.21.15.90:0/3177550647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30fc109a30 0x7f30fc1a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:32.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.200+0000 7f30fb7fe640 1 --2- 172.21.15.90:0/3177550647 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f30fc103af0 0x7f30fc19a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:32.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.200+0000 7f30f3fff640 1 --2- 172.21.15.90:0/3177550647 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30fc104d30 0x7f30fc19a6d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f30e400c970 tx=0x7f30e400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:32.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.200+0000 7f30f97fa640 1 -- 172.21.15.90:0/3177550647 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e4007bb0 con 0x7f30fc104d30 2024-04-25T11:34:32.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.200+0000 7f3101b4d640 1 -- 172.21.15.90:0/3177550647 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30fc1a1f10 con 0x7f30fc104d30 2024-04-25T11:34:32.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.200+0000 7f30f97fa640 1 -- 172.21.15.90:0/3177550647 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30e4007d50 con 0x7f30fc104d30 2024-04-25T11:34:32.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.200+0000 7f30f97fa640 1 -- 172.21.15.90:0/3177550647 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e400f450 con 0x7f30fc104d30 2024-04-25T11:34:32.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.200+0000 7f3101b4d640 1 -- 172.21.15.90:0/3177550647 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30fc1a2380 con 0x7f30fc104d30 2024-04-25T11:34:32.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.202+0000 7f3101b4d640 1 -- 172.21.15.90:0/3177550647 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30fc108840 con 0x7f30fc104d30 2024-04-25T11:34:32.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.203+0000 7f30f97fa640 1 -- 172.21.15.90:0/3177550647 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f30e4004f00 con 0x7f30fc104d30 2024-04-25T11:34:32.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.204+0000 7f30f97fa640 1 --2- 172.21.15.90:0/3177550647 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f30e80afb80 0x7f30e80b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:32.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.204+0000 7f30fb7fe640 1 --2- 172.21.15.90:0/3177550647 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f30e80afb80 0x7f30e80b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:32.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.204+0000 7f30f97fa640 1 -- 172.21.15.90:0/3177550647 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f30e40deff0 con 0x7f30fc104d30 2024-04-25T11:34:32.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.204+0000 7f30fb7fe640 1 --2- 172.21.15.90:0/3177550647 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f30e80afb80 0x7f30e80b2090 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f30fc1046f0 tx=0x7f30ec005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:32.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.207+0000 7f30f97fa640 1 -- 172.21.15.90:0/3177550647 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f30e4014030 con 0x7f30fc104d30 2024-04-25T11:34:32.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.384+0000 7f3101b4d640 1 -- 172.21.15.90:0/3177550647 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f30fc02db50 con 0x7f30fc104d30 2024-04-25T11:34:32.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.384+0000 7f30f97fa640 1 -- 172.21.15.90:0/3177550647 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) ==== 93+0+10 (secure 0 0 0) 0x7f30e40a7a50 con 0x7f30fc104d30 2024-04-25T11:34:32.373 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T11:34:32.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.386+0000 7f3101b4d640 1 -- 172.21.15.90:0/3177550647 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f30e80afb80 msgr2=0x7f30e80b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:32.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.386+0000 7f3101b4d640 1 --2- 172.21.15.90:0/3177550647 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f30e80afb80 0x7f30e80b2090 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f30fc1046f0 tx=0x7f30ec005db0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.386+0000 7f3101b4d640 1 -- 172.21.15.90:0/3177550647 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30fc104d30 msgr2=0x7f30fc19a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:32.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.386+0000 7f3101b4d640 1 --2- 172.21.15.90:0/3177550647 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30fc104d30 0x7f30fc19a6d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f30e400c970 tx=0x7f30e400ce30 comp rx=0 tx=0).stop 2024-04-25T11:34:32.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.386+0000 7f3101b4d640 1 -- 172.21.15.90:0/3177550647 shutdown_connections 2024-04-25T11:34:32.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.386+0000 7f3101b4d640 1 --2- 172.21.15.90:0/3177550647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30fc109a30 0x7f30fc1a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.386+0000 7f3101b4d640 1 --2- 172.21.15.90:0/3177550647 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f30e80afb80 0x7f30e80b2090 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.386+0000 7f3101b4d640 1 --2- 172.21.15.90:0/3177550647 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30fc104d30 0x7f30fc19a6d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.386+0000 7f3101b4d640 1 --2- 172.21.15.90:0/3177550647 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f30fc103af0 0x7f30fc19a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.386+0000 7f3101b4d640 1 -- 172.21.15.90:0/3177550647 >> 172.21.15.90:0/3177550647 conn(0x7f30fc0fece0 msgr2=0x7f30fc105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:32.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.387+0000 7f3101b4d640 1 -- 172.21.15.90:0/3177550647 shutdown_connections 2024-04-25T11:34:32.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.387+0000 7f3101b4d640 1 -- 172.21.15.90:0/3177550647 wait complete. 2024-04-25T11:34:32.387 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T11:34:32.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.509+0000 7fe517815640 1 -- 172.21.15.90:0/2996089409 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe510109a10 msgr2=0x7fe51010be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:32.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.509+0000 7fe517815640 1 --2- 172.21.15.90:0/2996089409 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe510109a10 0x7fe51010be30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe508009ca0 tx=0x7fe50802a8c0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.509+0000 7fe517815640 1 -- 172.21.15.90:0/2996089409 shutdown_connections 2024-04-25T11:34:32.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.509+0000 7fe517815640 1 --2- 172.21.15.90:0/2996089409 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe510109a10 0x7fe51010be30 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.509+0000 7fe517815640 1 --2- 172.21.15.90:0/2996089409 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5101070b0 0x7fe5101094d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.509+0000 7fe517815640 1 --2- 172.21.15.90:0/2996089409 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe510101f00 0x7fe510106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.509+0000 7fe517815640 1 -- 172.21.15.90:0/2996089409 >> 172.21.15.90:0/2996089409 conn(0x7fe5100fd340 msgr2=0x7fe5100ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:32.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.509+0000 7fe517815640 1 -- 172.21.15.90:0/2996089409 shutdown_connections 2024-04-25T11:34:32.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.509+0000 7fe517815640 1 -- 172.21.15.90:0/2996089409 wait complete. 2024-04-25T11:34:32.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.510+0000 7fe517815640 1 Processor -- start 2024-04-25T11:34:32.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.510+0000 7fe517815640 1 -- start start 2024-04-25T11:34:32.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.510+0000 7fe517815640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe510101f00 0x7fe51019a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:32.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.510+0000 7fe517815640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe5101070b0 0x7fe51019a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:32.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.510+0000 7fe517815640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe510109a10 0x7fe5101a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:32.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.510+0000 7fe517815640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe51010f6b0 con 0x7fe510101f00 2024-04-25T11:34:32.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.510+0000 7fe517815640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe51010f530 con 0x7fe5101070b0 2024-04-25T11:34:32.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.510+0000 7fe517815640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe51010f830 con 0x7fe510109a10 2024-04-25T11:34:32.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.510+0000 7fe5155b1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe510101f00 0x7fe51019a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:32.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.510+0000 7fe5155b1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe510101f00 0x7fe51019a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34698/0 (socket says 172.21.15.90:34698) 2024-04-25T11:34:32.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.510+0000 7fe5155b1640 1 -- 172.21.15.90:0/1409862672 learned_addr learned my addr 172.21.15.90:0/1409862672 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:32.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.511+0000 7fe514db0640 1 --2- 172.21.15.90:0/1409862672 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe5101070b0 0x7fe51019a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:32.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.511+0000 7fe5155b1640 1 -- 172.21.15.90:0/1409862672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe510109a10 msgr2=0x7fe5101a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:32.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.511+0000 7fe515db2640 1 --2- 172.21.15.90:0/1409862672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe510109a10 0x7fe5101a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:32.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.511+0000 7fe5155b1640 1 --2- 172.21.15.90:0/1409862672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe510109a10 0x7fe5101a17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.511+0000 7fe5155b1640 1 -- 172.21.15.90:0/1409862672 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe5101070b0 msgr2=0x7fe51019a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:32.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.511+0000 7fe5155b1640 1 --2- 172.21.15.90:0/1409862672 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe5101070b0 0x7fe51019a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.511+0000 7fe5155b1640 1 -- 172.21.15.90:0/1409862672 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5101a1e20 con 0x7fe510101f00 2024-04-25T11:34:32.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.511+0000 7fe514db0640 1 --2- 172.21.15.90:0/1409862672 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe5101070b0 0x7fe51019a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:32.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.511+0000 7fe5155b1640 1 --2- 172.21.15.90:0/1409862672 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe510101f00 0x7fe51019a2e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe50c00d770 tx=0x7fe50c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:32.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.511+0000 7fe5067fc640 1 -- 172.21.15.90:0/1409862672 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50c017070 con 0x7fe510101f00 2024-04-25T11:34:32.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.511+0000 7fe5067fc640 1 -- 172.21.15.90:0/1409862672 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe50c009c30 con 0x7fe510101f00 2024-04-25T11:34:32.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.511+0000 7fe517815640 1 -- 172.21.15.90:0/1409862672 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5101a2110 con 0x7fe510101f00 2024-04-25T11:34:32.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.511+0000 7fe5067fc640 1 -- 172.21.15.90:0/1409862672 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50c00f620 con 0x7fe510101f00 2024-04-25T11:34:32.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.511+0000 7fe517815640 1 -- 172.21.15.90:0/1409862672 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5101a2660 con 0x7fe510101f00 2024-04-25T11:34:32.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.512+0000 7fe517815640 1 -- 172.21.15.90:0/1409862672 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5100fee60 con 0x7fe510101f00 2024-04-25T11:34:32.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.514+0000 7fe5067fc640 1 -- 172.21.15.90:0/1409862672 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fe50c01a070 con 0x7fe510101f00 2024-04-25T11:34:32.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.514+0000 7fe5067fc640 1 --2- 172.21.15.90:0/1409862672 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fe4fc0afb80 0x7fe4fc0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:32.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.514+0000 7fe5067fc640 1 -- 172.21.15.90:0/1409862672 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7fe50c0de9a0 con 0x7fe510101f00 2024-04-25T11:34:32.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.514+0000 7fe514db0640 1 --2- 172.21.15.90:0/1409862672 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fe4fc0afb80 0x7fe4fc0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:32.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.515+0000 7fe514db0640 1 --2- 172.21.15.90:0/1409862672 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fe4fc0afb80 0x7fe4fc0b2090 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe5000058c0 tx=0x7fe500005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:32.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.518+0000 7fe5067fc640 1 -- 172.21.15.90:0/1409862672 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe50c009de0 con 0x7fe510101f00 2024-04-25T11:34:32.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.695+0000 7fe517815640 1 -- 172.21.15.90:0/1409862672 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe5100637a0 con 0x7fe510101f00 2024-04-25T11:34:32.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.695+0000 7fe5067fc640 1 -- 172.21.15.90:0/1409862672 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v79) ==== 104+0+11 (secure 0 0 0) 0x7fe50c0a7400 con 0x7fe510101f00 2024-04-25T11:34:32.684 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T11:34:32.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.697+0000 7fe517815640 1 -- 172.21.15.90:0/1409862672 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fe4fc0afb80 msgr2=0x7fe4fc0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:32.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.697+0000 7fe517815640 1 --2- 172.21.15.90:0/1409862672 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fe4fc0afb80 0x7fe4fc0b2090 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe5000058c0 tx=0x7fe500005d50 comp rx=0 tx=0).stop 2024-04-25T11:34:32.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.697+0000 7fe517815640 1 -- 172.21.15.90:0/1409862672 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe510101f00 msgr2=0x7fe51019a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:32.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.697+0000 7fe517815640 1 --2- 172.21.15.90:0/1409862672 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe510101f00 0x7fe51019a2e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe50c00d770 tx=0x7fe50c00dc30 comp rx=0 tx=0).stop 2024-04-25T11:34:32.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.697+0000 7fe517815640 1 -- 172.21.15.90:0/1409862672 shutdown_connections 2024-04-25T11:34:32.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.697+0000 7fe517815640 1 --2- 172.21.15.90:0/1409862672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe510109a10 0x7fe5101a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.697+0000 7fe517815640 1 --2- 172.21.15.90:0/1409862672 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe5101070b0 0x7fe51019a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.697+0000 7fe517815640 1 --2- 172.21.15.90:0/1409862672 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fe4fc0afb80 0x7fe4fc0b2090 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.697+0000 7fe517815640 1 --2- 172.21.15.90:0/1409862672 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe510101f00 0x7fe51019a2e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.698+0000 7fe517815640 1 -- 172.21.15.90:0/1409862672 >> 172.21.15.90:0/1409862672 conn(0x7fe5100fd340 msgr2=0x7fe510105200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:32.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.698+0000 7fe517815640 1 -- 172.21.15.90:0/1409862672 shutdown_connections 2024-04-25T11:34:32.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.698+0000 7fe517815640 1 -- 172.21.15.90:0/1409862672 wait complete. 2024-04-25T11:34:32.697 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T11:34:32.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.820+0000 7fb668f4d640 1 -- 172.21.15.90:0/2002675334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6641050a0 msgr2=0x7fb6641054b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:32.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.820+0000 7fb668f4d640 1 --2- 172.21.15.90:0/2002675334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6641050a0 0x7fb6641054b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb654009900 tx=0x7fb65402a410 comp rx=0 tx=0).stop 2024-04-25T11:34:32.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.820+0000 7fb668f4d640 1 -- 172.21.15.90:0/2002675334 shutdown_connections 2024-04-25T11:34:32.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.820+0000 7fb668f4d640 1 --2- 172.21.15.90:0/2002675334 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb664102250 0x7fb66410bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.820+0000 7fb668f4d640 1 --2- 172.21.15.90:0/2002675334 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb664101750 0x7fb664101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.820+0000 7fb668f4d640 1 --2- 172.21.15.90:0/2002675334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6641050a0 0x7fb6641054b0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.820+0000 7fb668f4d640 1 -- 172.21.15.90:0/2002675334 >> 172.21.15.90:0/2002675334 conn(0x7fb6640fd350 msgr2=0x7fb6640ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:32.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.820+0000 7fb668f4d640 1 -- 172.21.15.90:0/2002675334 shutdown_connections 2024-04-25T11:34:32.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.821+0000 7fb668f4d640 1 -- 172.21.15.90:0/2002675334 wait complete. 2024-04-25T11:34:32.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.821+0000 7fb668f4d640 1 Processor -- start 2024-04-25T11:34:32.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.821+0000 7fb668f4d640 1 -- start start 2024-04-25T11:34:32.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.821+0000 7fb668f4d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb664101750 0x7fb66419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:32.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.821+0000 7fb668f4d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb664102250 0x7fb66419a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:32.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.821+0000 7fb668f4d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb6641050a0 0x7fb6641a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:32.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.821+0000 7fb668f4d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb66410f610 con 0x7fb664102250 2024-04-25T11:34:32.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.821+0000 7fb668f4d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb66410f490 con 0x7fb6641050a0 2024-04-25T11:34:32.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.821+0000 7fb668f4d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb66410f790 con 0x7fb664101750 2024-04-25T11:34:32.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.822+0000 7fb661d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb664102250 0x7fb66419a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:32.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.822+0000 7fb661d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb664102250 0x7fb66419a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34712/0 (socket says 172.21.15.90:34712) 2024-04-25T11:34:32.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.822+0000 7fb661d9b640 1 -- 172.21.15.90:0/3671936185 learned_addr learned my addr 172.21.15.90:0/3671936185 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:32.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.822+0000 7fb66259c640 1 --2- 172.21.15.90:0/3671936185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb664101750 0x7fb66419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:32.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.822+0000 7fb662d9d640 1 --2- 172.21.15.90:0/3671936185 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb6641050a0 0x7fb6641a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:32.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.822+0000 7fb661d9b640 1 -- 172.21.15.90:0/3671936185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb664101750 msgr2=0x7fb66419a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.822+0000 7fb661d9b640 1 --2- 172.21.15.90:0/3671936185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb664101750 0x7fb66419a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.822+0000 7fb661d9b640 1 -- 172.21.15.90:0/3671936185 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb6641050a0 msgr2=0x7fb6641a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.822+0000 7fb661d9b640 1 --2- 172.21.15.90:0/3671936185 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb6641050a0 0x7fb6641a1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.822+0000 7fb661d9b640 1 -- 172.21.15.90:0/3671936185 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6641a1d00 con 0x7fb664102250 2024-04-25T11:34:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.822+0000 7fb661d9b640 1 --2- 172.21.15.90:0/3671936185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb664102250 0x7fb66419a7b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb65800c970 tx=0x7fb65800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.822+0000 7fb6537fe640 1 -- 172.21.15.90:0/3671936185 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb658007bb0 con 0x7fb664102250 2024-04-25T11:34:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.822+0000 7fb6537fe640 1 -- 172.21.15.90:0/3671936185 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb658007d50 con 0x7fb664102250 2024-04-25T11:34:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.822+0000 7fb668f4d640 1 -- 172.21.15.90:0/3671936185 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6641a1ff0 con 0x7fb664102250 2024-04-25T11:34:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.822+0000 7fb6537fe640 1 -- 172.21.15.90:0/3671936185 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65800f450 con 0x7fb664102250 2024-04-25T11:34:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.822+0000 7fb668f4d640 1 -- 172.21.15.90:0/3671936185 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6641a2400 con 0x7fb664102250 2024-04-25T11:34:32.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.823+0000 7fb668f4d640 1 -- 172.21.15.90:0/3671936185 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb664108a30 con 0x7fb664102250 2024-04-25T11:34:32.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.825+0000 7fb6537fe640 1 -- 172.21.15.90:0/3671936185 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fb658004f20 con 0x7fb664102250 2024-04-25T11:34:32.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.825+0000 7fb6537fe640 1 --2- 172.21.15.90:0/3671936185 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb64c0afb80 0x7fb64c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:32.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.825+0000 7fb66259c640 1 --2- 172.21.15.90:0/3671936185 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb64c0afb80 0x7fb64c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:32.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.825+0000 7fb6537fe640 1 -- 172.21.15.90:0/3671936185 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7fb6580deb90 con 0x7fb664102250 2024-04-25T11:34:32.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.826+0000 7fb66259c640 1 --2- 172.21.15.90:0/3671936185 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb64c0afb80 0x7fb64c0b2090 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb654005ab0 tx=0x7fb654031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:32.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:32.830+0000 7fb6537fe640 1 -- 172.21.15.90:0/3671936185 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb658014030 con 0x7fb664102250 2024-04-25T11:34:32.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:32 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3758987068' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:34:32.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:32 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3177550647' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:34:32.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:32 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1409862672' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:34:32.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.010+0000 7fb668f4d640 1 -- 172.21.15.90:0/3671936185 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb6640637d0 con 0x7fb664102250 2024-04-25T11:34:32.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.010+0000 7fb6537fe640 1 -- 172.21.15.90:0/3671936185 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v79) ==== 100+0+11 (secure 0 0 0) 0x7fb6580a75f0 con 0x7fb664102250 2024-04-25T11:34:32.999 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T11:34:33.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.012+0000 7fb668f4d640 1 -- 172.21.15.90:0/3671936185 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb64c0afb80 msgr2=0x7fb64c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:33.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.012+0000 7fb668f4d640 1 --2- 172.21.15.90:0/3671936185 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb64c0afb80 0x7fb64c0b2090 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb654005ab0 tx=0x7fb654031400 comp rx=0 tx=0).stop 2024-04-25T11:34:33.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.012+0000 7fb668f4d640 1 -- 172.21.15.90:0/3671936185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb664102250 msgr2=0x7fb66419a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:33.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.012+0000 7fb668f4d640 1 --2- 172.21.15.90:0/3671936185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb664102250 0x7fb66419a7b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb65800c970 tx=0x7fb65800ce30 comp rx=0 tx=0).stop 2024-04-25T11:34:33.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.012+0000 7fb668f4d640 1 -- 172.21.15.90:0/3671936185 shutdown_connections 2024-04-25T11:34:33.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.012+0000 7fb668f4d640 1 --2- 172.21.15.90:0/3671936185 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb6641050a0 0x7fb6641a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.012+0000 7fb668f4d640 1 --2- 172.21.15.90:0/3671936185 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fb64c0afb80 0x7fb64c0b2090 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.012+0000 7fb668f4d640 1 --2- 172.21.15.90:0/3671936185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb664102250 0x7fb66419a7b0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.012+0000 7fb668f4d640 1 --2- 172.21.15.90:0/3671936185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb664101750 0x7fb66419a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.012+0000 7fb668f4d640 1 -- 172.21.15.90:0/3671936185 >> 172.21.15.90:0/3671936185 conn(0x7fb6640fd350 msgr2=0x7fb664105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:33.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.012+0000 7fb668f4d640 1 -- 172.21.15.90:0/3671936185 shutdown_connections 2024-04-25T11:34:33.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.012+0000 7fb668f4d640 1 -- 172.21.15.90:0/3671936185 wait complete. 2024-04-25T11:34:33.012 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:34:33.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:32 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3758987068' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:34:33.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:32 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3177550647' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:34:33.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:32 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1409862672' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:34:33.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.130+0000 7f83d8bf3640 1 -- 172.21.15.90:0/4153148528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d4104d40 msgr2=0x7f83d4105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:33.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.130+0000 7f83d8bf3640 1 --2- 172.21.15.90:0/4153148528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d4104d40 0x7f83d4105150 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f83bc009900 tx=0x7f83bc02a410 comp rx=0 tx=0).stop 2024-04-25T11:34:33.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.130+0000 7f83d8bf3640 1 -- 172.21.15.90:0/4153148528 shutdown_connections 2024-04-25T11:34:33.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.130+0000 7f83d8bf3640 1 --2- 172.21.15.90:0/4153148528 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f83d4109a40 0x7f83d410be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.130+0000 7f83d8bf3640 1 --2- 172.21.15.90:0/4153148528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d4104d40 0x7f83d4105150 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.130+0000 7f83d8bf3640 1 --2- 172.21.15.90:0/4153148528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83d4103b00 0x7f83d4104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.130+0000 7f83d8bf3640 1 -- 172.21.15.90:0/4153148528 >> 172.21.15.90:0/4153148528 conn(0x7f83d40fed10 msgr2=0x7f83d41011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:33.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.130+0000 7f83d8bf3640 1 -- 172.21.15.90:0/4153148528 shutdown_connections 2024-04-25T11:34:33.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.130+0000 7f83d8bf3640 1 -- 172.21.15.90:0/4153148528 wait complete. 2024-04-25T11:34:33.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.131+0000 7f83d8bf3640 1 Processor -- start 2024-04-25T11:34:33.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.131+0000 7f83d8bf3640 1 -- start start 2024-04-25T11:34:33.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.131+0000 7f83d8bf3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d4103b00 0x7f83d419a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:33.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.132+0000 7f83d8bf3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83d4104d40 0x7f83d419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:33.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.132+0000 7f83d8bf3640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f83d4109a40 0x7f83d41a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:33.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.132+0000 7f83d8bf3640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f83d410f620 con 0x7f83d4103b00 2024-04-25T11:34:33.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.132+0000 7f83d8bf3640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f83d410f4a0 con 0x7f83d4109a40 2024-04-25T11:34:33.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.132+0000 7f83d8bf3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f83d410f7a0 con 0x7f83d4104d40 2024-04-25T11:34:33.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.132+0000 7f83d259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d4103b00 0x7f83d419a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:33.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.132+0000 7f83d259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d4103b00 0x7f83d419a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34726/0 (socket says 172.21.15.90:34726) 2024-04-25T11:34:33.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.132+0000 7f83d259c640 1 -- 172.21.15.90:0/1145860397 learned_addr learned my addr 172.21.15.90:0/1145860397 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:33.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.132+0000 7f83c9d9b640 1 --2- 172.21.15.90:0/1145860397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83d4104d40 0x7f83d419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:33.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.132+0000 7f83d2d9d640 1 --2- 172.21.15.90:0/1145860397 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f83d4109a40 0x7f83d41a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:33.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.132+0000 7f83d259c640 1 -- 172.21.15.90:0/1145860397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83d4104d40 msgr2=0x7f83d419a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:33.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.132+0000 7f83d259c640 1 --2- 172.21.15.90:0/1145860397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83d4104d40 0x7f83d419a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.132+0000 7f83d259c640 1 -- 172.21.15.90:0/1145860397 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f83d4109a40 msgr2=0x7f83d41a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:33.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.132+0000 7f83d259c640 1 --2- 172.21.15.90:0/1145860397 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f83d4109a40 0x7f83d41a16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.132+0000 7f83d259c640 1 -- 172.21.15.90:0/1145860397 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d41a1d60 con 0x7f83d4103b00 2024-04-25T11:34:33.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.132+0000 7f83d259c640 1 --2- 172.21.15.90:0/1145860397 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d4103b00 0x7f83d419a220 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f83c400c970 tx=0x7f83c400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:33.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.132+0000 7f83cbfff640 1 -- 172.21.15.90:0/1145860397 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c4004220 con 0x7f83d4103b00 2024-04-25T11:34:33.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.132+0000 7f83cbfff640 1 -- 172.21.15.90:0/1145860397 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83c40043c0 con 0x7f83d4103b00 2024-04-25T11:34:33.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.132+0000 7f83d8bf3640 1 -- 172.21.15.90:0/1145860397 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83d41a1ff0 con 0x7f83d4103b00 2024-04-25T11:34:33.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.132+0000 7f83cbfff640 1 -- 172.21.15.90:0/1145860397 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c4004a00 con 0x7f83d4103b00 2024-04-25T11:34:33.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.132+0000 7f83d8bf3640 1 -- 172.21.15.90:0/1145860397 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83d41a2460 con 0x7f83d4103b00 2024-04-25T11:34:33.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.133+0000 7f83d8bf3640 1 -- 172.21.15.90:0/1145860397 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83d4108850 con 0x7f83d4103b00 2024-04-25T11:34:33.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.135+0000 7f83cbfff640 1 -- 172.21.15.90:0/1145860397 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f83c40079b0 con 0x7f83d4103b00 2024-04-25T11:34:33.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.135+0000 7f83cbfff640 1 --2- 172.21.15.90:0/1145860397 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f83c00afb80 0x7f83c00b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:33.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.135+0000 7f83c9d9b640 1 --2- 172.21.15.90:0/1145860397 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f83c00afb80 0x7f83c00b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:33.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.135+0000 7f83cbfff640 1 -- 172.21.15.90:0/1145860397 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f83c40def70 con 0x7f83d4103b00 2024-04-25T11:34:33.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.136+0000 7f83c9d9b640 1 --2- 172.21.15.90:0/1145860397 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f83c00afb80 0x7f83c00b2090 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f83bc009230 tx=0x7f83bc031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:33.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.139+0000 7f83cbfff640 1 -- 172.21.15.90:0/1145860397 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f83c4014030 con 0x7f83d4103b00 2024-04-25T11:34:33.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:32 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3758987068' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:34:33.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:32 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3177550647' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:34:33.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:32 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1409862672' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:34:33.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.354+0000 7f83d8bf3640 1 -- 172.21.15.90:0/1145860397 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83d402db50 con 0x7f83d4103b00 2024-04-25T11:34:33.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.355+0000 7f83cbfff640 1 -- 172.21.15.90:0/1145860397 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11930 (secure 0 0 0) 0x7f83c40a79d0 con 0x7f83d4103b00 2024-04-25T11:34:33.343 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:34:33.343 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:33:16.640084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524,24475,24499,24458,24431]},"id":1}]} 2024-04-25T11:34:33.343 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 25 2024-04-25T11:34:33.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.357+0000 7f83d8bf3640 1 -- 172.21.15.90:0/1145860397 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f83c00afb80 msgr2=0x7f83c00b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:33.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.357+0000 7f83d8bf3640 1 --2- 172.21.15.90:0/1145860397 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f83c00afb80 0x7f83c00b2090 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f83bc009230 tx=0x7f83bc031400 comp rx=0 tx=0).stop 2024-04-25T11:34:33.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.357+0000 7f83d8bf3640 1 -- 172.21.15.90:0/1145860397 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d4103b00 msgr2=0x7f83d419a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:33.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.357+0000 7f83d8bf3640 1 --2- 172.21.15.90:0/1145860397 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d4103b00 0x7f83d419a220 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f83c400c970 tx=0x7f83c400ce30 comp rx=0 tx=0).stop 2024-04-25T11:34:33.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.357+0000 7f83d8bf3640 1 -- 172.21.15.90:0/1145860397 shutdown_connections 2024-04-25T11:34:33.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.357+0000 7f83d8bf3640 1 --2- 172.21.15.90:0/1145860397 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f83d4109a40 0x7f83d41a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.357+0000 7f83d8bf3640 1 --2- 172.21.15.90:0/1145860397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83d4104d40 0x7f83d419a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.357+0000 7f83d8bf3640 1 --2- 172.21.15.90:0/1145860397 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f83c00afb80 0x7f83c00b2090 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.357+0000 7f83d8bf3640 1 --2- 172.21.15.90:0/1145860397 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d4103b00 0x7f83d419a220 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.357+0000 7f83d8bf3640 1 -- 172.21.15.90:0/1145860397 >> 172.21.15.90:0/1145860397 conn(0x7f83d40fed10 msgr2=0x7f83d4105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:33.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.357+0000 7f83d8bf3640 1 -- 172.21.15.90:0/1145860397 shutdown_connections 2024-04-25T11:34:33.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.357+0000 7f83d8bf3640 1 -- 172.21.15.90:0/1145860397 wait complete. 2024-04-25T11:34:33.357 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:34:33.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.476+0000 7f070da9f640 1 -- 172.21.15.90:0/1987345941 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0708104d40 msgr2=0x7f0708105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:33.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.476+0000 7f070da9f640 1 --2- 172.21.15.90:0/1987345941 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0708104d40 0x7f0708105150 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f06fc009900 tx=0x7f06fc02a410 comp rx=0 tx=0).stop 2024-04-25T11:34:33.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.476+0000 7f070da9f640 1 -- 172.21.15.90:0/1987345941 shutdown_connections 2024-04-25T11:34:33.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.476+0000 7f070da9f640 1 --2- 172.21.15.90:0/1987345941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0708109a40 0x7f070810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.476+0000 7f070da9f640 1 --2- 172.21.15.90:0/1987345941 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0708104d40 0x7f0708105150 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.476+0000 7f070da9f640 1 --2- 172.21.15.90:0/1987345941 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0708103b00 0x7f0708104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.476+0000 7f070da9f640 1 -- 172.21.15.90:0/1987345941 >> 172.21.15.90:0/1987345941 conn(0x7f07080fed10 msgr2=0x7f07081011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:33.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.476+0000 7f070da9f640 1 -- 172.21.15.90:0/1987345941 shutdown_connections 2024-04-25T11:34:33.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.476+0000 7f070da9f640 1 -- 172.21.15.90:0/1987345941 wait complete. 2024-04-25T11:34:33.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.477+0000 7f070da9f640 1 Processor -- start 2024-04-25T11:34:33.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.477+0000 7f070da9f640 1 -- start start 2024-04-25T11:34:33.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.477+0000 7f070da9f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0708103b00 0x7f070819a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:33.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.477+0000 7f070da9f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0708104d40 0x7f070819a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:33.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.477+0000 7f070da9f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0708109a40 0x7f07081a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:33.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.477+0000 7f070da9f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f070810f630 con 0x7f0708104d40 2024-04-25T11:34:33.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.477+0000 7f070da9f640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f070810f4b0 con 0x7f0708109a40 2024-04-25T11:34:33.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.477+0000 7f070da9f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f070810f7b0 con 0x7f0708103b00 2024-04-25T11:34:33.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.477+0000 7f0706ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0708104d40 0x7f070819a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:33.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.477+0000 7f0706ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0708104d40 0x7f070819a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34742/0 (socket says 172.21.15.90:34742) 2024-04-25T11:34:33.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.477+0000 7f0706ffd640 1 -- 172.21.15.90:0/1087149160 learned_addr learned my addr 172.21.15.90:0/1087149160 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:33.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.477+0000 7f07077fe640 1 --2- 172.21.15.90:0/1087149160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0708103b00 0x7f070819a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:33.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.477+0000 7f0707fff640 1 --2- 172.21.15.90:0/1087149160 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0708109a40 0x7f07081a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:33.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.478+0000 7f0706ffd640 1 -- 172.21.15.90:0/1087149160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0708103b00 msgr2=0x7f070819a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:33.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.478+0000 7f0706ffd640 1 --2- 172.21.15.90:0/1087149160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0708103b00 0x7f070819a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.478+0000 7f0706ffd640 1 -- 172.21.15.90:0/1087149160 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0708109a40 msgr2=0x7f07081a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:33.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.478+0000 7f0706ffd640 1 --2- 172.21.15.90:0/1087149160 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0708109a40 0x7f07081a1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.478+0000 7f0706ffd640 1 -- 172.21.15.90:0/1087149160 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07081a1d70 con 0x7f0708104d40 2024-04-25T11:34:33.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.478+0000 7f0707fff640 1 --2- 172.21.15.90:0/1087149160 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0708109a40 0x7f07081a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:33.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.478+0000 7f0706ffd640 1 --2- 172.21.15.90:0/1087149160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0708104d40 0x7f070819a770 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f06fc006010 tx=0x7f06fc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:33.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.478+0000 7f0704ff9640 1 -- 172.21.15.90:0/1087149160 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06fc03f030 con 0x7f0708104d40 2024-04-25T11:34:33.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.478+0000 7f0704ff9640 1 -- 172.21.15.90:0/1087149160 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06fc031910 con 0x7f0708104d40 2024-04-25T11:34:33.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.478+0000 7f070da9f640 1 -- 172.21.15.90:0/1087149160 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07081a2000 con 0x7f0708104d40 2024-04-25T11:34:33.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.478+0000 7f0704ff9640 1 -- 172.21.15.90:0/1087149160 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06fc035070 con 0x7f0708104d40 2024-04-25T11:34:33.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.478+0000 7f070da9f640 1 -- 172.21.15.90:0/1087149160 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07081a24f0 con 0x7f0708104d40 2024-04-25T11:34:33.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.479+0000 7f070da9f640 1 -- 172.21.15.90:0/1087149160 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0708108850 con 0x7f0708104d40 2024-04-25T11:34:33.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.481+0000 7f0704ff9640 1 -- 172.21.15.90:0/1087149160 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f06fc042030 con 0x7f0708104d40 2024-04-25T11:34:33.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.482+0000 7f0704ff9640 1 --2- 172.21.15.90:0/1087149160 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f06f00afb80 0x7f06f00b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:33.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.482+0000 7f07077fe640 1 --2- 172.21.15.90:0/1087149160 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f06f00afb80 0x7f06f00b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:33.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.482+0000 7f0704ff9640 1 -- 172.21.15.90:0/1087149160 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f06fc0ff540 con 0x7f0708104d40 2024-04-25T11:34:33.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.482+0000 7f07077fe640 1 --2- 172.21.15.90:0/1087149160 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f06f00afb80 0x7f06f00b2090 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0708104700 tx=0x7f06f8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:33.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.485+0000 7f0704ff9640 1 -- 172.21.15.90:0/1087149160 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f06fc0c7fa0 con 0x7f0708104d40 2024-04-25T11:34:33.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.664+0000 7f070da9f640 1 -- 172.21.15.90:0/1087149160 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f07080637a0 con 0x7f0708104d40 2024-04-25T11:34:33.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.665+0000 7f0704ff9640 1 -- 172.21.15.90:0/1087149160 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+22706 (secure 0 0 0) 0x7f06fc0c78a0 con 0x7f0708104d40 2024-04-25T11:34:33.654 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:34:33.654 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":79,"fsid":"58bd4e7a-02f6-11ef-bc93-c7b262605968","created":"2024-04-25T11:25:23.574550+0000","modified":"2024-04-25T11:34:17.871834+0000","last_up_change":"2024-04-25T11:32:23.852211+0000","last_in_change":"2024-04-25T11:32:04.188507+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:28:42.746649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:33:08.442925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:33:09.352886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e54d888-e67b-4951-8ae7-e355fb6b10a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6803","nonce":1201629159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6805","nonce":1201629159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6809","nonce":1201629159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6807","nonce":1201629159}]},"public_addr":"172.21.15.90:6803/1201629159","cluster_addr":"172.21.15.90:6805/1201629159","heartbeat_back_addr":"172.21.15.90:6809/1201629159","heartbeat_front_addr":"172.21.15.90:6807/1201629159","state":["exists","up"]},{"osd":1,"uuid":"91a71c78-dae9-4bc7-9f6f-9e55cfefbe28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6801","nonce":3976851933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6803","nonce":3976851933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6807","nonce":3976851933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6805","nonce":3976851933}]},"public_addr":"172.21.15.138:6801/3976851933","cluster_addr":"172.21.15.138:6803/3976851933","heartbeat_back_addr":"172.21.15.138:6807/3976851933","heartbeat_front_addr":"172.21.15.138:6805/3976851933","state":["exists","up"]},{"osd":2,"uuid":"daff6c7f-80fa-4859-9339-ef7a5e840832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6801","nonce":1505807670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6803","nonce":1505807670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6807","nonce":1505807670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6805","nonce":1505807670}]},"public_addr":"172.21.15.156:6801/1505807670","cluster_addr":"172.21.15.156:6803/1505807670","heartbeat_back_addr":"172.21.15.156:6807/1505807670","heartbeat_front_addr":"172.21.15.156:6805/1505807670","state":["exists","up"]},{"osd":3,"uuid":"33fd94ca-7465-401a-bc1e-706b6b31b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6811","nonce":2520012383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6813","nonce":2520012383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6817","nonce":2520012383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6815","nonce":2520012383}]},"public_addr":"172.21.15.90:6811/2520012383","cluster_addr":"172.21.15.90:6813/2520012383","heartbeat_back_addr":"172.21.15.90:6817/2520012383","heartbeat_front_addr":"172.21.15.90:6815/2520012383","state":["exists","up"]},{"osd":4,"uuid":"c5326212-750c-49fd-96e3-d1e62bb4b7f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6809","nonce":3752678279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6811","nonce":3752678279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6815","nonce":3752678279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6813","nonce":3752678279}]},"public_addr":"172.21.15.138:6809/3752678279","cluster_addr":"172.21.15.138:6811/3752678279","heartbeat_back_addr":"172.21.15.138:6815/3752678279","heartbeat_front_addr":"172.21.15.138:6813/3752678279","state":["exists","up"]},{"osd":5,"uuid":"d04a1156-820b-4b43-be89-e16a8910c3aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6809","nonce":3591210191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6811","nonce":3591210191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6815","nonce":3591210191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6813","nonce":3591210191}]},"public_addr":"172.21.15.156:6809/3591210191","cluster_addr":"172.21.15.156:6811/3591210191","heartbeat_back_addr":"172.21.15.156:6815/3591210191","heartbeat_front_addr":"172.21.15.156:6813/3591210191","state":["exists","up"]},{"osd":6,"uuid":"75675aa3-e978-490c-8205-13d9868a3d6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6819","nonce":2364955456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6821","nonce":2364955456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6825","nonce":2364955456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6823","nonce":2364955456}]},"public_addr":"172.21.15.90:6819/2364955456","cluster_addr":"172.21.15.90:6821/2364955456","heartbeat_back_addr":"172.21.15.90:6825/2364955456","heartbeat_front_addr":"172.21.15.90:6823/2364955456","state":["exists","up"]},{"osd":7,"uuid":"1a95fdf5-f548-4271-bbcd-e330f7955eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6817","nonce":4012476387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6819","nonce":4012476387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6823","nonce":4012476387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6821","nonce":4012476387}]},"public_addr":"172.21.15.138:6817/4012476387","cluster_addr":"172.21.15.138:6819/4012476387","heartbeat_back_addr":"172.21.15.138:6823/4012476387","heartbeat_front_addr":"172.21.15.138:6821/4012476387","state":["exists","up"]},{"osd":8,"uuid":"605e10b5-5806-4c1f-8e28-a7994d34ddce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6817","nonce":101092224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6819","nonce":101092224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6823","nonce":101092224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6821","nonce":101092224}]},"public_addr":"172.21.15.156:6817/101092224","cluster_addr":"172.21.15.156:6819/101092224","heartbeat_back_addr":"172.21.15.156:6823/101092224","heartbeat_front_addr":"172.21.15.156:6821/101092224","state":["exists","up"]},{"osd":9,"uuid":"d70fdf5e-9d0f-483d-a7d8-505b79b8a418","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6827","nonce":1171265966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6829","nonce":1171265966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6833","nonce":1171265966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6831","nonce":1171265966}]},"public_addr":"172.21.15.90:6827/1171265966","cluster_addr":"172.21.15.90:6829/1171265966","heartbeat_back_addr":"172.21.15.90:6833/1171265966","heartbeat_front_addr":"172.21.15.90:6831/1171265966","state":["exists","up"]},{"osd":10,"uuid":"1ec86c7d-c3a4-432d-9664-8e40836e1e20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6825","nonce":4264666232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6827","nonce":4264666232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6831","nonce":4264666232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6829","nonce":4264666232}]},"public_addr":"172.21.15.138:6825/4264666232","cluster_addr":"172.21.15.138:6827/4264666232","heartbeat_back_addr":"172.21.15.138:6831/4264666232","heartbeat_front_addr":"172.21.15.138:6829/4264666232","state":["exists","up"]},{"osd":11,"uuid":"51cbe729-292e-4d7b-b8e9-e1e2821ffe76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6825","nonce":1224251744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6827","nonce":1224251744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6831","nonce":1224251744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6829","nonce":1224251744}]},"public_addr":"172.21.15.156:6825/1224251744","cluster_addr":"172.21.15.156:6827/1224251744","heartbeat_back_addr":"172.21.15.156:6831/1224251744","heartbeat_front_addr":"172.21.15.156:6829/1224251744","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:27:50.794226+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:15.030278+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:38.518163+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:03.107351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:27.500707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:51.463545+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:16.015076+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:41.182071+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:05.362309+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:31.048936+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:56.488509+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:32:22.129178+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":7}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/2381633953":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6800/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/2190846900":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1707792941":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6800/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6801/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/6061052":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1849633392":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6801/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/2277490702":"2024-04-26T11:26:16.655278+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:34:33.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.667+0000 7f070da9f640 1 -- 172.21.15.90:0/1087149160 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f06f00afb80 msgr2=0x7f06f00b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:33.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.667+0000 7f070da9f640 1 --2- 172.21.15.90:0/1087149160 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f06f00afb80 0x7f06f00b2090 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0708104700 tx=0x7f06f8005f90 comp rx=0 tx=0).stop 2024-04-25T11:34:33.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.667+0000 7f070da9f640 1 -- 172.21.15.90:0/1087149160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0708104d40 msgr2=0x7f070819a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:33.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.667+0000 7f070da9f640 1 --2- 172.21.15.90:0/1087149160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0708104d40 0x7f070819a770 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f06fc006010 tx=0x7f06fc0042f0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.668+0000 7f070da9f640 1 -- 172.21.15.90:0/1087149160 shutdown_connections 2024-04-25T11:34:33.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.668+0000 7f070da9f640 1 --2- 172.21.15.90:0/1087149160 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0708109a40 0x7f07081a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.668+0000 7f070da9f640 1 --2- 172.21.15.90:0/1087149160 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f06f00afb80 0x7f06f00b2090 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.668+0000 7f070da9f640 1 --2- 172.21.15.90:0/1087149160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0708104d40 0x7f070819a770 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.668+0000 7f070da9f640 1 --2- 172.21.15.90:0/1087149160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0708103b00 0x7f070819a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.668+0000 7f070da9f640 1 -- 172.21.15.90:0/1087149160 >> 172.21.15.90:0/1087149160 conn(0x7f07080fed10 msgr2=0x7f0708105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:33.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.668+0000 7f070da9f640 1 -- 172.21.15.90:0/1087149160 shutdown_connections 2024-04-25T11:34:33.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.668+0000 7f070da9f640 1 -- 172.21.15.90:0/1087149160 wait complete. 2024-04-25T11:34:33.668 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T11:28:42.746649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T11:33:08.442925+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T11:33:09.352886+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T11:34:33.668 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T11:34:33.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.791+0000 7f682672d640 1 -- 172.21.15.90:0/922867558 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6820103af0 msgr2=0x7f68201047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:33.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.791+0000 7f682672d640 1 --2- 172.21.15.90:0/922867558 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6820103af0 0x7f68201047f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f6810009900 tx=0x7f681002a410 comp rx=0 tx=0).stop 2024-04-25T11:34:33.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.791+0000 7f682672d640 1 -- 172.21.15.90:0/922867558 shutdown_connections 2024-04-25T11:34:33.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.791+0000 7f682672d640 1 --2- 172.21.15.90:0/922867558 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6820109a30 0x7f682010be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.791+0000 7f682672d640 1 --2- 172.21.15.90:0/922867558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6820104d30 0x7f6820105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.791+0000 7f682672d640 1 --2- 172.21.15.90:0/922867558 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6820103af0 0x7f68201047f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.791+0000 7f682672d640 1 -- 172.21.15.90:0/922867558 >> 172.21.15.90:0/922867558 conn(0x7f68200fece0 msgr2=0x7f6820101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:33.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.791+0000 7f682672d640 1 -- 172.21.15.90:0/922867558 shutdown_connections 2024-04-25T11:34:33.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.791+0000 7f682672d640 1 -- 172.21.15.90:0/922867558 wait complete. 2024-04-25T11:34:33.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.791+0000 7f682672d640 1 Processor -- start 2024-04-25T11:34:33.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.791+0000 7f682672d640 1 -- start start 2024-04-25T11:34:33.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.792+0000 7f682672d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6820103af0 0x7f682019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:33.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.792+0000 7f682672d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6820104d30 0x7f682019a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:33.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.792+0000 7f682672d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6820109a30 0x7f68201a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:33.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.792+0000 7f682672d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f682010f5d0 con 0x7f6820104d30 2024-04-25T11:34:33.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.792+0000 7f682672d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f682010f450 con 0x7f6820109a30 2024-04-25T11:34:33.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.792+0000 7f682672d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f682010f750 con 0x7f6820103af0 2024-04-25T11:34:33.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.792+0000 7f681f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6820104d30 0x7f682019a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:33.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.792+0000 7f681f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6820104d30 0x7f682019a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34772/0 (socket says 172.21.15.90:34772) 2024-04-25T11:34:33.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.792+0000 7f681f7fe640 1 -- 172.21.15.90:0/1611317524 learned_addr learned my addr 172.21.15.90:0/1611317524 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:33.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.792+0000 7f681ffff640 1 --2- 172.21.15.90:0/1611317524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6820103af0 0x7f682019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:33.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.792+0000 7f6824cca640 1 --2- 172.21.15.90:0/1611317524 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6820109a30 0x7f68201a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:33.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.792+0000 7f681f7fe640 1 -- 172.21.15.90:0/1611317524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6820103af0 msgr2=0x7f682019a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:33.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.792+0000 7f681f7fe640 1 --2- 172.21.15.90:0/1611317524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6820103af0 0x7f682019a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.792+0000 7f681f7fe640 1 -- 172.21.15.90:0/1611317524 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6820109a30 msgr2=0x7f68201a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:33.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.792+0000 7f681f7fe640 1 --2- 172.21.15.90:0/1611317524 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6820109a30 0x7f68201a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.792+0000 7f681f7fe640 1 -- 172.21.15.90:0/1611317524 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68201a1d80 con 0x7f6820104d30 2024-04-25T11:34:33.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.792+0000 7f681f7fe640 1 --2- 172.21.15.90:0/1611317524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6820104d30 0x7f682019a7d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f681400d950 tx=0x7f681400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:33.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.792+0000 7f681d7fa640 1 -- 172.21.15.90:0/1611317524 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6814017070 con 0x7f6820104d30 2024-04-25T11:34:33.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.793+0000 7f681d7fa640 1 -- 172.21.15.90:0/1611317524 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6814009c30 con 0x7f6820104d30 2024-04-25T11:34:33.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.793+0000 7f682672d640 1 -- 172.21.15.90:0/1611317524 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68201a2070 con 0x7f6820104d30 2024-04-25T11:34:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.793+0000 7f681d7fa640 1 -- 172.21.15.90:0/1611317524 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681400f620 con 0x7f6820104d30 2024-04-25T11:34:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.793+0000 7f682672d640 1 -- 172.21.15.90:0/1611317524 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68201a25c0 con 0x7f6820104d30 2024-04-25T11:34:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.794+0000 7f682672d640 1 -- 172.21.15.90:0/1611317524 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6820108840 con 0x7f6820104d30 2024-04-25T11:34:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.795+0000 7f681d7fa640 1 -- 172.21.15.90:0/1611317524 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f6814010460 con 0x7f6820104d30 2024-04-25T11:34:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.795+0000 7f681d7fa640 1 --2- 172.21.15.90:0/1611317524 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f68080afb80 0x7f68080b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.796+0000 7f681ffff640 1 --2- 172.21.15.90:0/1611317524 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f68080afb80 0x7f68080b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.796+0000 7f681d7fa640 1 -- 172.21.15.90:0/1611317524 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f68140deae0 con 0x7f6820104d30 2024-04-25T11:34:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.796+0000 7f681ffff640 1 --2- 172.21.15.90:0/1611317524 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f68080afb80 0x7f68080b2090 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6810009230 tx=0x7f6810031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:33.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.800+0000 7f681d7fa640 1 -- 172.21.15.90:0/1611317524 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f68140a7540 con 0x7f6820104d30 2024-04-25T11:34:33.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.980+0000 7f682672d640 1 -- 172.21.15.90:0/1611317524 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f68200637a0 con 0x7f6820104d30 2024-04-25T11:34:33.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.980+0000 7f681d7fa640 1 -- 172.21.15.90:0/1611317524 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) ==== 93+0+10 (secure 0 0 0) 0x7f6814009de0 con 0x7f6820104d30 2024-04-25T11:34:33.968 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T11:34:33.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.982+0000 7f682672d640 1 -- 172.21.15.90:0/1611317524 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f68080afb80 msgr2=0x7f68080b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:33.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.982+0000 7f682672d640 1 --2- 172.21.15.90:0/1611317524 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f68080afb80 0x7f68080b2090 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6810009230 tx=0x7f6810031400 comp rx=0 tx=0).stop 2024-04-25T11:34:33.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.982+0000 7f682672d640 1 -- 172.21.15.90:0/1611317524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6820104d30 msgr2=0x7f682019a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:33.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.982+0000 7f682672d640 1 --2- 172.21.15.90:0/1611317524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6820104d30 0x7f682019a7d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f681400d950 tx=0x7f681400de10 comp rx=0 tx=0).stop 2024-04-25T11:34:33.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.982+0000 7f682672d640 1 -- 172.21.15.90:0/1611317524 shutdown_connections 2024-04-25T11:34:33.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.982+0000 7f682672d640 1 --2- 172.21.15.90:0/1611317524 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6820109a30 0x7f68201a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.982+0000 7f682672d640 1 --2- 172.21.15.90:0/1611317524 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f68080afb80 0x7f68080b2090 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.982+0000 7f682672d640 1 --2- 172.21.15.90:0/1611317524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6820104d30 0x7f682019a7d0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.982+0000 7f682672d640 1 --2- 172.21.15.90:0/1611317524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6820103af0 0x7f682019a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:33.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.982+0000 7f682672d640 1 -- 172.21.15.90:0/1611317524 >> 172.21.15.90:0/1611317524 conn(0x7f68200fece0 msgr2=0x7f6820105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:33.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.982+0000 7f682672d640 1 -- 172.21.15.90:0/1611317524 shutdown_connections 2024-04-25T11:34:33.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:33.982+0000 7f682672d640 1 -- 172.21.15.90:0/1611317524 wait complete. 2024-04-25T11:34:33.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:33 smithi090 ceph-mon[29575]: pgmap v317: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T11:34:33.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:33 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3671936185' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:34:33.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:33 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1145860397' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:34:33.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:33 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1087149160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:34:33.982 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T11:34:34.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:33 smithi156 ceph-mon[33045]: pgmap v317: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T11:34:34.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:33 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3671936185' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:34:34.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:33 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1145860397' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:34:34.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:33 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1087149160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:34:34.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.105+0000 7fd393de3640 1 -- 172.21.15.90:0/3938713383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd38c109780 msgr2=0x7fd38c10bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:34.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.105+0000 7fd393de3640 1 --2- 172.21.15.90:0/3938713383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd38c109780 0x7fd38c10bc90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd384009ca0 tx=0x7fd38402a8c0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.105+0000 7fd393de3640 1 -- 172.21.15.90:0/3938713383 shutdown_connections 2024-04-25T11:34:34.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.105+0000 7fd393de3640 1 --2- 172.21.15.90:0/3938713383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd38c109780 0x7fd38c10bc90 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.105+0000 7fd393de3640 1 --2- 172.21.15.90:0/3938713383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd38c106c50 0x7fd38c1090f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.105+0000 7fd393de3640 1 --2- 172.21.15.90:0/3938713383 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd38c104260 0x7fd38c106680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.105+0000 7fd393de3640 1 -- 172.21.15.90:0/3938713383 >> 172.21.15.90:0/3938713383 conn(0x7fd38c0fdbc0 msgr2=0x7fd38c100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:34.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.105+0000 7fd393de3640 1 -- 172.21.15.90:0/3938713383 shutdown_connections 2024-04-25T11:34:34.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.105+0000 7fd393de3640 1 -- 172.21.15.90:0/3938713383 wait complete. 2024-04-25T11:34:34.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.106+0000 7fd393de3640 1 Processor -- start 2024-04-25T11:34:34.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.106+0000 7fd393de3640 1 -- start start 2024-04-25T11:34:34.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.106+0000 7fd393de3640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd38c104260 0x7fd38c19a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:34.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.106+0000 7fd393de3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd38c106c50 0x7fd38c19a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:34.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.106+0000 7fd393de3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd38c109780 0x7fd38c1a1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:34.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.106+0000 7fd393de3640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c10f5f0 con 0x7fd38c109780 2024-04-25T11:34:34.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.106+0000 7fd393de3640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c10f470 con 0x7fd38c104260 2024-04-25T11:34:34.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.106+0000 7fd393de3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c10f770 con 0x7fd38c106c50 2024-04-25T11:34:34.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.106+0000 7fd392380640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd38c109780 0x7fd38c1a1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:34.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.106+0000 7fd392380640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd38c109780 0x7fd38c1a1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34784/0 (socket says 172.21.15.90:34784) 2024-04-25T11:34:34.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.106+0000 7fd392380640 1 -- 172.21.15.90:0/1564099197 learned_addr learned my addr 172.21.15.90:0/1564099197 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:34.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.106+0000 7fd39137e640 1 --2- 172.21.15.90:0/1564099197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd38c106c50 0x7fd38c19a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:34.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.106+0000 7fd391b7f640 1 --2- 172.21.15.90:0/1564099197 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd38c104260 0x7fd38c19a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:34.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.106+0000 7fd392380640 1 -- 172.21.15.90:0/1564099197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd38c106c50 msgr2=0x7fd38c19a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:34.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.106+0000 7fd392380640 1 --2- 172.21.15.90:0/1564099197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd38c106c50 0x7fd38c19a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.106+0000 7fd392380640 1 -- 172.21.15.90:0/1564099197 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd38c104260 msgr2=0x7fd38c19a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:34.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.106+0000 7fd392380640 1 --2- 172.21.15.90:0/1564099197 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd38c104260 0x7fd38c19a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.106+0000 7fd392380640 1 -- 172.21.15.90:0/1564099197 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd38c1a1cb0 con 0x7fd38c109780 2024-04-25T11:34:34.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.106+0000 7fd39137e640 1 --2- 172.21.15.90:0/1564099197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd38c106c50 0x7fd38c19a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:34.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.107+0000 7fd392380640 1 --2- 172.21.15.90:0/1564099197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd38c109780 0x7fd38c1a1640 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd3840097b0 tx=0x7fd384003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:34.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.107+0000 7fd382ffd640 1 -- 172.21.15.90:0/1564099197 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd384004240 con 0x7fd38c109780 2024-04-25T11:34:34.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.107+0000 7fd382ffd640 1 -- 172.21.15.90:0/1564099197 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd38402f380 con 0x7fd38c109780 2024-04-25T11:34:34.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.107+0000 7fd393de3640 1 -- 172.21.15.90:0/1564099197 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd38c1a1f40 con 0x7fd38c109780 2024-04-25T11:34:34.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.107+0000 7fd382ffd640 1 -- 172.21.15.90:0/1564099197 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd38402fc30 con 0x7fd38c109780 2024-04-25T11:34:34.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.107+0000 7fd393de3640 1 -- 172.21.15.90:0/1564099197 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd38c1a23a0 con 0x7fd38c109780 2024-04-25T11:34:34.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.108+0000 7fd393de3640 1 -- 172.21.15.90:0/1564099197 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd38c06a160 con 0x7fd38c109780 2024-04-25T11:34:34.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.109+0000 7fd382ffd640 1 -- 172.21.15.90:0/1564099197 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fd3840043e0 con 0x7fd38c109780 2024-04-25T11:34:34.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.110+0000 7fd382ffd640 1 --2- 172.21.15.90:0/1564099197 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd3780afb80 0x7fd3780b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:34.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.110+0000 7fd391b7f640 1 --2- 172.21.15.90:0/1564099197 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd3780afb80 0x7fd3780b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:34.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.110+0000 7fd382ffd640 1 -- 172.21.15.90:0/1564099197 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7fd384100510 con 0x7fd38c109780 2024-04-25T11:34:34.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.110+0000 7fd391b7f640 1 --2- 172.21.15.90:0/1564099197 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd3780afb80 0x7fd3780b2090 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd388005fd0 tx=0x7fd388005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:34.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.115+0000 7fd382ffd640 1 -- 172.21.15.90:0/1564099197 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd3840c9080 con 0x7fd38c109780 2024-04-25T11:34:34.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.294+0000 7fd393de3640 1 -- 172.21.15.90:0/1564099197 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd38c02dbf0 con 0x7fd38c109780 2024-04-25T11:34:34.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.294+0000 7fd382ffd640 1 -- 172.21.15.90:0/1564099197 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v79) ==== 104+0+11 (secure 0 0 0) 0x7fd3840098b0 con 0x7fd38c109780 2024-04-25T11:34:34.282 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T11:34:34.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.296+0000 7fd393de3640 1 -- 172.21.15.90:0/1564099197 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd3780afb80 msgr2=0x7fd3780b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:34.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.296+0000 7fd393de3640 1 --2- 172.21.15.90:0/1564099197 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd3780afb80 0x7fd3780b2090 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd388005fd0 tx=0x7fd388005c70 comp rx=0 tx=0).stop 2024-04-25T11:34:34.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.296+0000 7fd393de3640 1 -- 172.21.15.90:0/1564099197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd38c109780 msgr2=0x7fd38c1a1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:34.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.296+0000 7fd393de3640 1 --2- 172.21.15.90:0/1564099197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd38c109780 0x7fd38c1a1640 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd3840097b0 tx=0x7fd384003c20 comp rx=0 tx=0).stop 2024-04-25T11:34:34.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.296+0000 7fd393de3640 1 -- 172.21.15.90:0/1564099197 shutdown_connections 2024-04-25T11:34:34.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.296+0000 7fd393de3640 1 --2- 172.21.15.90:0/1564099197 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd3780afb80 0x7fd3780b2090 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.296+0000 7fd393de3640 1 --2- 172.21.15.90:0/1564099197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd38c109780 0x7fd38c1a1640 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.296+0000 7fd393de3640 1 --2- 172.21.15.90:0/1564099197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd38c106c50 0x7fd38c19a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.296+0000 7fd393de3640 1 --2- 172.21.15.90:0/1564099197 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd38c104260 0x7fd38c19a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.296+0000 7fd393de3640 1 -- 172.21.15.90:0/1564099197 >> 172.21.15.90:0/1564099197 conn(0x7fd38c0fdbc0 msgr2=0x7fd38c104e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:34.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.296+0000 7fd393de3640 1 -- 172.21.15.90:0/1564099197 shutdown_connections 2024-04-25T11:34:34.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.296+0000 7fd393de3640 1 -- 172.21.15.90:0/1564099197 wait complete. 2024-04-25T11:34:34.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T11:34:34.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:33 smithi138 ceph-mon[32328]: pgmap v317: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T11:34:34.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:33 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3671936185' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:34:34.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:33 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1145860397' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:34:34.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:33 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1087149160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:34:34.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.418+0000 7faa207ba640 1 -- 172.21.15.90:0/2068677437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa18109a30 msgr2=0x7faa1810be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:34.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.418+0000 7faa207ba640 1 --2- 172.21.15.90:0/2068677437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa18109a30 0x7faa1810be50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7faa14009ca0 tx=0x7faa1402a8c0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.419+0000 7faa207ba640 1 -- 172.21.15.90:0/2068677437 shutdown_connections 2024-04-25T11:34:34.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.419+0000 7faa207ba640 1 --2- 172.21.15.90:0/2068677437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa18109a30 0x7faa1810be50 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.419+0000 7faa207ba640 1 --2- 172.21.15.90:0/2068677437 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa18104d30 0x7faa18105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.419+0000 7faa207ba640 1 --2- 172.21.15.90:0/2068677437 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faa18103af0 0x7faa181047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.419+0000 7faa207ba640 1 -- 172.21.15.90:0/2068677437 >> 172.21.15.90:0/2068677437 conn(0x7faa180fece0 msgr2=0x7faa18101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:34.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.419+0000 7faa207ba640 1 -- 172.21.15.90:0/2068677437 shutdown_connections 2024-04-25T11:34:34.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.419+0000 7faa207ba640 1 -- 172.21.15.90:0/2068677437 wait complete. 2024-04-25T11:34:34.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.419+0000 7faa207ba640 1 Processor -- start 2024-04-25T11:34:34.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.419+0000 7faa207ba640 1 -- start start 2024-04-25T11:34:34.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.419+0000 7faa207ba640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa18103af0 0x7faa1819a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:34.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.419+0000 7faa207ba640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa18104d30 0x7faa1819a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:34.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.419+0000 7faa207ba640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faa18109a30 0x7faa181a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:34.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.419+0000 7faa207ba640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7faa1810f670 con 0x7faa18103af0 2024-04-25T11:34:34.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.419+0000 7faa207ba640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7faa1810f4f0 con 0x7faa18109a30 2024-04-25T11:34:34.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.419+0000 7faa207ba640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7faa1810f7f0 con 0x7faa18104d30 2024-04-25T11:34:34.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.419+0000 7faa1e556640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa18103af0 0x7faa1819a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:34.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.419+0000 7faa1e556640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa18103af0 0x7faa1819a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34808/0 (socket says 172.21.15.90:34808) 2024-04-25T11:34:34.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.419+0000 7faa1e556640 1 -- 172.21.15.90:0/75515382 learned_addr learned my addr 172.21.15.90:0/75515382 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:34.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.420+0000 7faa1dd55640 1 --2- 172.21.15.90:0/75515382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa18104d30 0x7faa1819a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:34.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.420+0000 7faa1e556640 1 -- 172.21.15.90:0/75515382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa18104d30 msgr2=0x7faa1819a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:34.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.420+0000 7faa1ed57640 1 --2- 172.21.15.90:0/75515382 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faa18109a30 0x7faa181a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:34.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.420+0000 7faa1e556640 1 --2- 172.21.15.90:0/75515382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa18104d30 0x7faa1819a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.420+0000 7faa1e556640 1 -- 172.21.15.90:0/75515382 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faa18109a30 msgr2=0x7faa181a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:34.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.420+0000 7faa1e556640 1 --2- 172.21.15.90:0/75515382 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faa18109a30 0x7faa181a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.420+0000 7faa1e556640 1 -- 172.21.15.90:0/75515382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa181a1d20 con 0x7faa18103af0 2024-04-25T11:34:34.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.420+0000 7faa1dd55640 1 --2- 172.21.15.90:0/75515382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa18104d30 0x7faa1819a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:34.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.420+0000 7faa1ed57640 1 --2- 172.21.15.90:0/75515382 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faa18109a30 0x7faa181a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:34.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.420+0000 7faa1e556640 1 --2- 172.21.15.90:0/75515382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa18103af0 0x7faa1819a1e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7faa1000e9f0 tx=0x7faa1000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:34.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.420+0000 7faa0f7fe640 1 -- 172.21.15.90:0/75515382 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1000ccc0 con 0x7faa18103af0 2024-04-25T11:34:34.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.420+0000 7faa0f7fe640 1 -- 172.21.15.90:0/75515382 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa100035f0 con 0x7faa18103af0 2024-04-25T11:34:34.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.420+0000 7faa207ba640 1 -- 172.21.15.90:0/75515382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa181a1fb0 con 0x7faa18103af0 2024-04-25T11:34:34.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.420+0000 7faa0f7fe640 1 -- 172.21.15.90:0/75515382 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1000ce60 con 0x7faa18103af0 2024-04-25T11:34:34.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.420+0000 7faa207ba640 1 -- 172.21.15.90:0/75515382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa181a2470 con 0x7faa18103af0 2024-04-25T11:34:34.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.421+0000 7faa207ba640 1 -- 172.21.15.90:0/75515382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa18108840 con 0x7faa18103af0 2024-04-25T11:34:34.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.423+0000 7faa0f7fe640 1 -- 172.21.15.90:0/75515382 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7faa10003aa0 con 0x7faa18103af0 2024-04-25T11:34:34.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.423+0000 7faa0f7fe640 1 --2- 172.21.15.90:0/75515382 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7faa040afb80 0x7faa040b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:34.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.424+0000 7faa1dd55640 1 --2- 172.21.15.90:0/75515382 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7faa040afb80 0x7faa040b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:34.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.424+0000 7faa0f7fe640 1 -- 172.21.15.90:0/75515382 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7faa100df2d0 con 0x7faa18103af0 2024-04-25T11:34:34.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.424+0000 7faa1dd55640 1 --2- 172.21.15.90:0/75515382 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7faa040afb80 0x7faa040b2090 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7faa08005fd0 tx=0x7faa08005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:34.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.427+0000 7faa0f7fe640 1 -- 172.21.15.90:0/75515382 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faa100a7d30 con 0x7faa18103af0 2024-04-25T11:34:34.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.606+0000 7faa207ba640 1 -- 172.21.15.90:0/75515382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7faa1802dce0 con 0x7faa18103af0 2024-04-25T11:34:34.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.606+0000 7faa0f7fe640 1 -- 172.21.15.90:0/75515382 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v79) ==== 100+0+11 (secure 0 0 0) 0x7faa100a7630 con 0x7faa18103af0 2024-04-25T11:34:34.595 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T11:34:34.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.608+0000 7faa207ba640 1 -- 172.21.15.90:0/75515382 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7faa040afb80 msgr2=0x7faa040b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:34.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.608+0000 7faa207ba640 1 --2- 172.21.15.90:0/75515382 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7faa040afb80 0x7faa040b2090 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7faa08005fd0 tx=0x7faa08005850 comp rx=0 tx=0).stop 2024-04-25T11:34:34.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.608+0000 7faa207ba640 1 -- 172.21.15.90:0/75515382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa18103af0 msgr2=0x7faa1819a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:34.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.608+0000 7faa207ba640 1 --2- 172.21.15.90:0/75515382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa18103af0 0x7faa1819a1e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7faa1000e9f0 tx=0x7faa1000eeb0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.608+0000 7faa207ba640 1 -- 172.21.15.90:0/75515382 shutdown_connections 2024-04-25T11:34:34.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.608+0000 7faa207ba640 1 --2- 172.21.15.90:0/75515382 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faa18109a30 0x7faa181a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.608+0000 7faa207ba640 1 --2- 172.21.15.90:0/75515382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa18104d30 0x7faa1819a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.608+0000 7faa207ba640 1 --2- 172.21.15.90:0/75515382 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7faa040afb80 0x7faa040b2090 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.608+0000 7faa207ba640 1 --2- 172.21.15.90:0/75515382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa18103af0 0x7faa1819a1e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.608+0000 7faa207ba640 1 -- 172.21.15.90:0/75515382 >> 172.21.15.90:0/75515382 conn(0x7faa180fece0 msgr2=0x7faa18105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:34.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.608+0000 7faa207ba640 1 -- 172.21.15.90:0/75515382 shutdown_connections 2024-04-25T11:34:34.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.608+0000 7faa207ba640 1 -- 172.21.15.90:0/75515382 wait complete. 2024-04-25T11:34:34.608 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:34:34.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.728+0000 7f3f58957640 1 -- 172.21.15.90:0/762957150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f541050a0 msgr2=0x7f3f541054b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:34.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.728+0000 7f3f58957640 1 --2- 172.21.15.90:0/762957150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f541050a0 0x7f3f541054b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3f44009900 tx=0x7f3f4402a410 comp rx=0 tx=0).stop 2024-04-25T11:34:34.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.728+0000 7f3f58957640 1 -- 172.21.15.90:0/762957150 shutdown_connections 2024-04-25T11:34:34.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.728+0000 7f3f58957640 1 --2- 172.21.15.90:0/762957150 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f54102250 0x7f3f5410bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.728+0000 7f3f58957640 1 --2- 172.21.15.90:0/762957150 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f54101750 0x7f3f54101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.728+0000 7f3f58957640 1 --2- 172.21.15.90:0/762957150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f541050a0 0x7f3f541054b0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.728+0000 7f3f58957640 1 -- 172.21.15.90:0/762957150 >> 172.21.15.90:0/762957150 conn(0x7f3f540fd370 msgr2=0x7f3f540ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:34.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.728+0000 7f3f58957640 1 -- 172.21.15.90:0/762957150 shutdown_connections 2024-04-25T11:34:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.728+0000 7f3f58957640 1 -- 172.21.15.90:0/762957150 wait complete. 2024-04-25T11:34:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.729+0000 7f3f58957640 1 Processor -- start 2024-04-25T11:34:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.729+0000 7f3f58957640 1 -- start start 2024-04-25T11:34:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.729+0000 7f3f58957640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f54101750 0x7f3f541041e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.729+0000 7f3f58957640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f54102250 0x7f3f541028f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.729+0000 7f3f58957640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f541050a0 0x7f3f54102e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.729+0000 7f3f58957640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5410f610 con 0x7f3f54101750 2024-04-25T11:34:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.729+0000 7f3f58957640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5410f490 con 0x7f3f541050a0 2024-04-25T11:34:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.729+0000 7f3f58957640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5410f790 con 0x7f3f54102250 2024-04-25T11:34:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.729+0000 7f3f5259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f54101750 0x7f3f541041e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.729+0000 7f3f5259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f54101750 0x7f3f541041e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34834/0 (socket says 172.21.15.90:34834) 2024-04-25T11:34:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.729+0000 7f3f5259c640 1 -- 172.21.15.90:0/3512303256 learned_addr learned my addr 172.21.15.90:0/3512303256 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:34.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.729+0000 7f3f5259c640 1 -- 172.21.15.90:0/3512303256 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f54102250 msgr2=0x7f3f541028f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:34.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.729+0000 7f3f5259c640 1 --2- 172.21.15.90:0/3512303256 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f54102250 0x7f3f541028f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.729+0000 7f3f5259c640 1 -- 172.21.15.90:0/3512303256 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f541050a0 msgr2=0x7f3f54102e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:34.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.729+0000 7f3f5259c640 1 --2- 172.21.15.90:0/3512303256 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f541050a0 0x7f3f54102e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.729+0000 7f3f5259c640 1 -- 172.21.15.90:0/3512303256 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f54103560 con 0x7f3f54101750 2024-04-25T11:34:34.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.729+0000 7f3f5259c640 1 --2- 172.21.15.90:0/3512303256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f54101750 0x7f3f541041e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3f44006010 tx=0x7f3f44004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:34.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.729+0000 7f3f437fe640 1 -- 172.21.15.90:0/3512303256 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4403f030 con 0x7f3f54101750 2024-04-25T11:34:34.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.729+0000 7f3f437fe640 1 -- 172.21.15.90:0/3512303256 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f44003aa0 con 0x7f3f54101750 2024-04-25T11:34:34.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.729+0000 7f3f58957640 1 -- 172.21.15.90:0/3512303256 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f541a4370 con 0x7f3f54101750 2024-04-25T11:34:34.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.729+0000 7f3f437fe640 1 -- 172.21.15.90:0/3512303256 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f44035070 con 0x7f3f54101750 2024-04-25T11:34:34.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.730+0000 7f3f58957640 1 -- 172.21.15.90:0/3512303256 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f541a4720 con 0x7f3f54101750 2024-04-25T11:34:34.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.730+0000 7f3f58957640 1 -- 172.21.15.90:0/3512303256 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f54108a30 con 0x7f3f54101750 2024-04-25T11:34:34.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.732+0000 7f3f437fe640 1 -- 172.21.15.90:0/3512303256 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f3f44042030 con 0x7f3f54101750 2024-04-25T11:34:34.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.732+0000 7f3f437fe640 1 --2- 172.21.15.90:0/3512303256 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f3f3c0afb80 0x7f3f3c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:34.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.732+0000 7f3f51d9b640 1 --2- 172.21.15.90:0/3512303256 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f3f3c0afb80 0x7f3f3c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:34.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.733+0000 7f3f437fe640 1 -- 172.21.15.90:0/3512303256 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f3f440ff920 con 0x7f3f54101750 2024-04-25T11:34:34.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.733+0000 7f3f51d9b640 1 --2- 172.21.15.90:0/3512303256 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f3f3c0afb80 0x7f3f3c0b2090 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3f480058c0 tx=0x7f3f48005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:34.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.736+0000 7f3f437fe640 1 -- 172.21.15.90:0/3512303256 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3f440c8380 con 0x7f3f54101750 2024-04-25T11:34:34.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.951+0000 7f3f58957640 1 -- 172.21.15.90:0/3512303256 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f5402db50 con 0x7f3f54101750 2024-04-25T11:34:34.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.952+0000 7f3f437fe640 1 -- 172.21.15.90:0/3512303256 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11930 (secure 0 0 0) 0x7f3f440c7c80 con 0x7f3f54101750 2024-04-25T11:34:34.940 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:34:34.940 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:33:16.640084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524,24475,24499,24458,24431]},"id":1}]} 2024-04-25T11:34:34.941 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 25 2024-04-25T11:34:34.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.954+0000 7f3f58957640 1 -- 172.21.15.90:0/3512303256 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f3f3c0afb80 msgr2=0x7f3f3c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:34.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.954+0000 7f3f58957640 1 --2- 172.21.15.90:0/3512303256 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f3f3c0afb80 0x7f3f3c0b2090 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3f480058c0 tx=0x7f3f48005850 comp rx=0 tx=0).stop 2024-04-25T11:34:34.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.954+0000 7f3f58957640 1 -- 172.21.15.90:0/3512303256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f54101750 msgr2=0x7f3f541041e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:34.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.954+0000 7f3f58957640 1 --2- 172.21.15.90:0/3512303256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f54101750 0x7f3f541041e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3f44006010 tx=0x7f3f44004270 comp rx=0 tx=0).stop 2024-04-25T11:34:34.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.954+0000 7f3f58957640 1 -- 172.21.15.90:0/3512303256 shutdown_connections 2024-04-25T11:34:34.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.954+0000 7f3f58957640 1 --2- 172.21.15.90:0/3512303256 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f541050a0 0x7f3f54102e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.954+0000 7f3f58957640 1 --2- 172.21.15.90:0/3512303256 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f54102250 0x7f3f541028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.954+0000 7f3f58957640 1 --2- 172.21.15.90:0/3512303256 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f3f3c0afb80 0x7f3f3c0b2090 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.954+0000 7f3f58957640 1 --2- 172.21.15.90:0/3512303256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f54101750 0x7f3f541041e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:34.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.954+0000 7f3f58957640 1 -- 172.21.15.90:0/3512303256 >> 172.21.15.90:0/3512303256 conn(0x7f3f540fd370 msgr2=0x7f3f54105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:34.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.954+0000 7f3f58957640 1 -- 172.21.15.90:0/3512303256 shutdown_connections 2024-04-25T11:34:34.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:34.954+0000 7f3f58957640 1 -- 172.21.15.90:0/3512303256 wait complete. 2024-04-25T11:34:34.953 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:34 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1611317524' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:34:34.953 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:34 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1564099197' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:34:34.953 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:34 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/75515382' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:34:34.957 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:34:35.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.075+0000 7fd82c492640 1 -- 172.21.15.90:0/2786479266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd824102a00 msgr2=0x7fd824103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.075+0000 7fd82c492640 1 --2- 172.21.15.90:0/2786479266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd824102a00 0x7fd824103700 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd81c009900 tx=0x7fd81c02a410 comp rx=0 tx=0).stop 2024-04-25T11:34:35.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.075+0000 7fd82c492640 1 -- 172.21.15.90:0/2786479266 shutdown_connections 2024-04-25T11:34:35.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.075+0000 7fd82c492640 1 --2- 172.21.15.90:0/2786479266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd824108940 0x7fd82410ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.075+0000 7fd82c492640 1 --2- 172.21.15.90:0/2786479266 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd824103c40 0x7fd824104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.075+0000 7fd82c492640 1 --2- 172.21.15.90:0/2786479266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd824102a00 0x7fd824103700 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.075+0000 7fd82c492640 1 -- 172.21.15.90:0/2786479266 >> 172.21.15.90:0/2786479266 conn(0x7fd8240fdbf0 msgr2=0x7fd8241000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:35.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.075+0000 7fd82c492640 1 -- 172.21.15.90:0/2786479266 shutdown_connections 2024-04-25T11:34:35.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.075+0000 7fd82c492640 1 -- 172.21.15.90:0/2786479266 wait complete. 2024-04-25T11:34:35.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.076+0000 7fd82c492640 1 Processor -- start 2024-04-25T11:34:35.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.076+0000 7fd82c492640 1 -- start start 2024-04-25T11:34:35.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.076+0000 7fd82c492640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd824102a00 0x7fd8241992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.076+0000 7fd82c492640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd824103c40 0x7fd824199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.076+0000 7fd82c492640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd824108940 0x7fd8241a0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.076+0000 7fd82c492640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd82410e5e0 con 0x7fd824108940 2024-04-25T11:34:35.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.076+0000 7fd82c492640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd82410e460 con 0x7fd824103c40 2024-04-25T11:34:35.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.076+0000 7fd82c492640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd82410e760 con 0x7fd824102a00 2024-04-25T11:34:35.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.076+0000 7fd82aa2f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd824108940 0x7fd8241a0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.076+0000 7fd82aa2f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd824108940 0x7fd8241a0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34846/0 (socket says 172.21.15.90:34846) 2024-04-25T11:34:35.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.076+0000 7fd82aa2f640 1 -- 172.21.15.90:0/2189695618 learned_addr learned my addr 172.21.15.90:0/2189695618 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:35.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.076+0000 7fd82a22e640 1 --2- 172.21.15.90:0/2189695618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd824102a00 0x7fd8241992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.076+0000 7fd829a2d640 1 --2- 172.21.15.90:0/2189695618 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd824103c40 0x7fd824199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.076+0000 7fd82aa2f640 1 -- 172.21.15.90:0/2189695618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd824102a00 msgr2=0x7fd8241992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.076+0000 7fd82aa2f640 1 --2- 172.21.15.90:0/2189695618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd824102a00 0x7fd8241992d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.076+0000 7fd82aa2f640 1 -- 172.21.15.90:0/2189695618 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd824103c40 msgr2=0x7fd824199810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.076+0000 7fd82aa2f640 1 --2- 172.21.15.90:0/2189695618 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd824103c40 0x7fd824199810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.076+0000 7fd82aa2f640 1 -- 172.21.15.90:0/2189695618 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8241a0da0 con 0x7fd824108940 2024-04-25T11:34:35.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.076+0000 7fd829a2d640 1 --2- 172.21.15.90:0/2189695618 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd824103c40 0x7fd824199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:35.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.076+0000 7fd82aa2f640 1 --2- 172.21.15.90:0/2189695618 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd824108940 0x7fd8241a0730 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd81400b970 tx=0x7fd81400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:35.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.076+0000 7fd8237fe640 1 -- 172.21.15.90:0/2189695618 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd814019070 con 0x7fd824108940 2024-04-25T11:34:35.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.077+0000 7fd8237fe640 1 -- 172.21.15.90:0/2189695618 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd814014440 con 0x7fd824108940 2024-04-25T11:34:35.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.077+0000 7fd82c492640 1 -- 172.21.15.90:0/2189695618 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8241a1090 con 0x7fd824108940 2024-04-25T11:34:35.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.077+0000 7fd8237fe640 1 -- 172.21.15.90:0/2189695618 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8140075d0 con 0x7fd824108940 2024-04-25T11:34:35.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.077+0000 7fd82c492640 1 -- 172.21.15.90:0/2189695618 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8241a1610 con 0x7fd824108940 2024-04-25T11:34:35.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.077+0000 7fd82c492640 1 -- 172.21.15.90:0/2189695618 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd824107750 con 0x7fd824108940 2024-04-25T11:34:35.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.079+0000 7fd8237fe640 1 -- 172.21.15.90:0/2189695618 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fd81401c070 con 0x7fd824108940 2024-04-25T11:34:35.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.079+0000 7fd8237fe640 1 --2- 172.21.15.90:0/2189695618 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd80c0afb80 0x7fd80c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.080+0000 7fd82a22e640 1 --2- 172.21.15.90:0/2189695618 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd80c0afb80 0x7fd80c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.080+0000 7fd8237fe640 1 -- 172.21.15.90:0/2189695618 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7fd8140de820 con 0x7fd824108940 2024-04-25T11:34:35.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.080+0000 7fd82a22e640 1 --2- 172.21.15.90:0/2189695618 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd80c0afb80 0x7fd80c0b2090 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd81c02a930 tx=0x7fd81c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:35.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.083+0000 7fd8237fe640 1 -- 172.21.15.90:0/2189695618 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd8140a7390 con 0x7fd824108940 2024-04-25T11:34:35.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:34 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1611317524' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:34:35.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:34 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1564099197' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:34:35.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:34 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/75515382' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:34:35.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.260+0000 7fd82c492640 1 -- 172.21.15.90:0/2189695618 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd82402dd20 con 0x7fd824108940 2024-04-25T11:34:35.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.262+0000 7fd8237fe640 1 -- 172.21.15.90:0/2189695618 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+22706 (secure 0 0 0) 0x7fd8140a6c90 con 0x7fd824108940 2024-04-25T11:34:35.251 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:34:35.251 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":79,"fsid":"58bd4e7a-02f6-11ef-bc93-c7b262605968","created":"2024-04-25T11:25:23.574550+0000","modified":"2024-04-25T11:34:17.871834+0000","last_up_change":"2024-04-25T11:32:23.852211+0000","last_in_change":"2024-04-25T11:32:04.188507+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:28:42.746649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:33:08.442925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:33:09.352886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e54d888-e67b-4951-8ae7-e355fb6b10a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6803","nonce":1201629159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6805","nonce":1201629159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6809","nonce":1201629159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6807","nonce":1201629159}]},"public_addr":"172.21.15.90:6803/1201629159","cluster_addr":"172.21.15.90:6805/1201629159","heartbeat_back_addr":"172.21.15.90:6809/1201629159","heartbeat_front_addr":"172.21.15.90:6807/1201629159","state":["exists","up"]},{"osd":1,"uuid":"91a71c78-dae9-4bc7-9f6f-9e55cfefbe28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6801","nonce":3976851933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6803","nonce":3976851933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6807","nonce":3976851933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6805","nonce":3976851933}]},"public_addr":"172.21.15.138:6801/3976851933","cluster_addr":"172.21.15.138:6803/3976851933","heartbeat_back_addr":"172.21.15.138:6807/3976851933","heartbeat_front_addr":"172.21.15.138:6805/3976851933","state":["exists","up"]},{"osd":2,"uuid":"daff6c7f-80fa-4859-9339-ef7a5e840832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6801","nonce":1505807670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6803","nonce":1505807670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6807","nonce":1505807670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6805","nonce":1505807670}]},"public_addr":"172.21.15.156:6801/1505807670","cluster_addr":"172.21.15.156:6803/1505807670","heartbeat_back_addr":"172.21.15.156:6807/1505807670","heartbeat_front_addr":"172.21.15.156:6805/1505807670","state":["exists","up"]},{"osd":3,"uuid":"33fd94ca-7465-401a-bc1e-706b6b31b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6811","nonce":2520012383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6813","nonce":2520012383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6817","nonce":2520012383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6815","nonce":2520012383}]},"public_addr":"172.21.15.90:6811/2520012383","cluster_addr":"172.21.15.90:6813/2520012383","heartbeat_back_addr":"172.21.15.90:6817/2520012383","heartbeat_front_addr":"172.21.15.90:6815/2520012383","state":["exists","up"]},{"osd":4,"uuid":"c5326212-750c-49fd-96e3-d1e62bb4b7f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6809","nonce":3752678279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6811","nonce":3752678279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6815","nonce":3752678279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6813","nonce":3752678279}]},"public_addr":"172.21.15.138:6809/3752678279","cluster_addr":"172.21.15.138:6811/3752678279","heartbeat_back_addr":"172.21.15.138:6815/3752678279","heartbeat_front_addr":"172.21.15.138:6813/3752678279","state":["exists","up"]},{"osd":5,"uuid":"d04a1156-820b-4b43-be89-e16a8910c3aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6809","nonce":3591210191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6811","nonce":3591210191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6815","nonce":3591210191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6813","nonce":3591210191}]},"public_addr":"172.21.15.156:6809/3591210191","cluster_addr":"172.21.15.156:6811/3591210191","heartbeat_back_addr":"172.21.15.156:6815/3591210191","heartbeat_front_addr":"172.21.15.156:6813/3591210191","state":["exists","up"]},{"osd":6,"uuid":"75675aa3-e978-490c-8205-13d9868a3d6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6819","nonce":2364955456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6821","nonce":2364955456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6825","nonce":2364955456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6823","nonce":2364955456}]},"public_addr":"172.21.15.90:6819/2364955456","cluster_addr":"172.21.15.90:6821/2364955456","heartbeat_back_addr":"172.21.15.90:6825/2364955456","heartbeat_front_addr":"172.21.15.90:6823/2364955456","state":["exists","up"]},{"osd":7,"uuid":"1a95fdf5-f548-4271-bbcd-e330f7955eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6817","nonce":4012476387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6819","nonce":4012476387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6823","nonce":4012476387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6821","nonce":4012476387}]},"public_addr":"172.21.15.138:6817/4012476387","cluster_addr":"172.21.15.138:6819/4012476387","heartbeat_back_addr":"172.21.15.138:6823/4012476387","heartbeat_front_addr":"172.21.15.138:6821/4012476387","state":["exists","up"]},{"osd":8,"uuid":"605e10b5-5806-4c1f-8e28-a7994d34ddce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6817","nonce":101092224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6819","nonce":101092224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6823","nonce":101092224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6821","nonce":101092224}]},"public_addr":"172.21.15.156:6817/101092224","cluster_addr":"172.21.15.156:6819/101092224","heartbeat_back_addr":"172.21.15.156:6823/101092224","heartbeat_front_addr":"172.21.15.156:6821/101092224","state":["exists","up"]},{"osd":9,"uuid":"d70fdf5e-9d0f-483d-a7d8-505b79b8a418","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6827","nonce":1171265966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6829","nonce":1171265966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6833","nonce":1171265966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6831","nonce":1171265966}]},"public_addr":"172.21.15.90:6827/1171265966","cluster_addr":"172.21.15.90:6829/1171265966","heartbeat_back_addr":"172.21.15.90:6833/1171265966","heartbeat_front_addr":"172.21.15.90:6831/1171265966","state":["exists","up"]},{"osd":10,"uuid":"1ec86c7d-c3a4-432d-9664-8e40836e1e20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6825","nonce":4264666232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6827","nonce":4264666232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6831","nonce":4264666232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6829","nonce":4264666232}]},"public_addr":"172.21.15.138:6825/4264666232","cluster_addr":"172.21.15.138:6827/4264666232","heartbeat_back_addr":"172.21.15.138:6831/4264666232","heartbeat_front_addr":"172.21.15.138:6829/4264666232","state":["exists","up"]},{"osd":11,"uuid":"51cbe729-292e-4d7b-b8e9-e1e2821ffe76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6825","nonce":1224251744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6827","nonce":1224251744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6831","nonce":1224251744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6829","nonce":1224251744}]},"public_addr":"172.21.15.156:6825/1224251744","cluster_addr":"172.21.15.156:6827/1224251744","heartbeat_back_addr":"172.21.15.156:6831/1224251744","heartbeat_front_addr":"172.21.15.156:6829/1224251744","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:27:50.794226+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:15.030278+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:38.518163+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:03.107351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:27.500707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:51.463545+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:16.015076+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:41.182071+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:05.362309+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:31.048936+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:56.488509+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:32:22.129178+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":7}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/2381633953":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6800/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/2190846900":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1707792941":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6800/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6801/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/6061052":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1849633392":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6801/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/2277490702":"2024-04-26T11:26:16.655278+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:34:35.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.264+0000 7fd82c492640 1 -- 172.21.15.90:0/2189695618 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd80c0afb80 msgr2=0x7fd80c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.264+0000 7fd82c492640 1 --2- 172.21.15.90:0/2189695618 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd80c0afb80 0x7fd80c0b2090 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd81c02a930 tx=0x7fd81c005fd0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.264+0000 7fd82c492640 1 -- 172.21.15.90:0/2189695618 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd824108940 msgr2=0x7fd8241a0730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.264+0000 7fd82c492640 1 --2- 172.21.15.90:0/2189695618 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd824108940 0x7fd8241a0730 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd81400b970 tx=0x7fd81400be30 comp rx=0 tx=0).stop 2024-04-25T11:34:35.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.264+0000 7fd82c492640 1 -- 172.21.15.90:0/2189695618 shutdown_connections 2024-04-25T11:34:35.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.264+0000 7fd82c492640 1 --2- 172.21.15.90:0/2189695618 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fd80c0afb80 0x7fd80c0b2090 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.264+0000 7fd82c492640 1 --2- 172.21.15.90:0/2189695618 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd824108940 0x7fd8241a0730 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.264+0000 7fd82c492640 1 --2- 172.21.15.90:0/2189695618 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd824103c40 0x7fd824199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.264+0000 7fd82c492640 1 --2- 172.21.15.90:0/2189695618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd824102a00 0x7fd8241992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.264+0000 7fd82c492640 1 -- 172.21.15.90:0/2189695618 >> 172.21.15.90:0/2189695618 conn(0x7fd8240fdbf0 msgr2=0x7fd824104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:35.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.264+0000 7fd82c492640 1 -- 172.21.15.90:0/2189695618 shutdown_connections 2024-04-25T11:34:35.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.264+0000 7fd82c492640 1 -- 172.21.15.90:0/2189695618 wait complete. 2024-04-25T11:34:35.265 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:34:35.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:34 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1611317524' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:34:35.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:34 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1564099197' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:34:35.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:34 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/75515382' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:34:35.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.384+0000 7f4e3cd0d640 1 -- 172.21.15.90:0/2362947790 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e38109a40 msgr2=0x7f4e3810be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.384+0000 7f4e3cd0d640 1 --2- 172.21.15.90:0/2362947790 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e38109a40 0x7f4e3810be60 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4e30009ca0 tx=0x7f4e3002a8c0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.384+0000 7f4e3cd0d640 1 -- 172.21.15.90:0/2362947790 shutdown_connections 2024-04-25T11:34:35.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.384+0000 7f4e3cd0d640 1 --2- 172.21.15.90:0/2362947790 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e38109a40 0x7f4e3810be60 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.384+0000 7f4e3cd0d640 1 --2- 172.21.15.90:0/2362947790 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e38104d40 0x7f4e38105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.384+0000 7f4e3cd0d640 1 --2- 172.21.15.90:0/2362947790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4e38103b00 0x7f4e38104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.384+0000 7f4e3cd0d640 1 -- 172.21.15.90:0/2362947790 >> 172.21.15.90:0/2362947790 conn(0x7f4e380fed10 msgr2=0x7f4e381011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:35.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.384+0000 7f4e3cd0d640 1 -- 172.21.15.90:0/2362947790 shutdown_connections 2024-04-25T11:34:35.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.384+0000 7f4e3cd0d640 1 -- 172.21.15.90:0/2362947790 wait complete. 2024-04-25T11:34:35.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.385+0000 7f4e3cd0d640 1 Processor -- start 2024-04-25T11:34:35.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.385+0000 7f4e3cd0d640 1 -- start start 2024-04-25T11:34:35.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.385+0000 7f4e3cd0d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4e38103b00 0x7f4e3819a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.385+0000 7f4e3cd0d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e38104d40 0x7f4e3819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.385+0000 7f4e3cd0d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e38109a40 0x7f4e381a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.385+0000 7f4e3cd0d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3810f610 con 0x7f4e38104d40 2024-04-25T11:34:35.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.385+0000 7f4e3cd0d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3810f490 con 0x7f4e38103b00 2024-04-25T11:34:35.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.385+0000 7f4e3cd0d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3810f790 con 0x7f4e38109a40 2024-04-25T11:34:35.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.385+0000 7f4e35d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e38104d40 0x7f4e3819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.385+0000 7f4e35d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e38104d40 0x7f4e3819a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34866/0 (socket says 172.21.15.90:34866) 2024-04-25T11:34:35.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.385+0000 7f4e35d9b640 1 -- 172.21.15.90:0/3611597346 learned_addr learned my addr 172.21.15.90:0/3611597346 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:35.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.386+0000 7f4e3659c640 1 --2- 172.21.15.90:0/3611597346 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4e38103b00 0x7f4e3819a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.386+0000 7f4e36d9d640 1 --2- 172.21.15.90:0/3611597346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e38109a40 0x7f4e381a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.386+0000 7f4e35d9b640 1 -- 172.21.15.90:0/3611597346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e38109a40 msgr2=0x7f4e381a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.386+0000 7f4e35d9b640 1 --2- 172.21.15.90:0/3611597346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e38109a40 0x7f4e381a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.386+0000 7f4e35d9b640 1 -- 172.21.15.90:0/3611597346 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4e38103b00 msgr2=0x7f4e3819a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.386+0000 7f4e35d9b640 1 --2- 172.21.15.90:0/3611597346 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4e38103b00 0x7f4e3819a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.386+0000 7f4e35d9b640 1 -- 172.21.15.90:0/3611597346 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e381a1d50 con 0x7f4e38104d40 2024-04-25T11:34:35.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.386+0000 7f4e36d9d640 1 --2- 172.21.15.90:0/3611597346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e38109a40 0x7f4e381a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:35.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.386+0000 7f4e3659c640 1 --2- 172.21.15.90:0/3611597346 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4e38103b00 0x7f4e3819a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:35.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.386+0000 7f4e35d9b640 1 --2- 172.21.15.90:0/3611597346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e38104d40 0x7f4e3819a750 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c00d950 tx=0x7f4e2c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:35.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.386+0000 7f4e277fe640 1 -- 172.21.15.90:0/3611597346 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c017030 con 0x7f4e38104d40 2024-04-25T11:34:35.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.386+0000 7f4e3cd0d640 1 -- 172.21.15.90:0/3611597346 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e381a1fe0 con 0x7f4e38104d40 2024-04-25T11:34:35.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.386+0000 7f4e277fe640 1 -- 172.21.15.90:0/3611597346 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e2c004020 con 0x7f4e38104d40 2024-04-25T11:34:35.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.386+0000 7f4e277fe640 1 -- 172.21.15.90:0/3611597346 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c0047a0 con 0x7f4e38104d40 2024-04-25T11:34:35.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.386+0000 7f4e3cd0d640 1 -- 172.21.15.90:0/3611597346 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e381a24d0 con 0x7f4e38104d40 2024-04-25T11:34:35.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.387+0000 7f4e3cd0d640 1 -- 172.21.15.90:0/3611597346 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e38108850 con 0x7f4e38104d40 2024-04-25T11:34:35.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.388+0000 7f4e277fe640 1 -- 172.21.15.90:0/3611597346 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f4e2c009a30 con 0x7f4e38104d40 2024-04-25T11:34:35.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.389+0000 7f4e277fe640 1 --2- 172.21.15.90:0/3611597346 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f4e200afc00 0x7f4e200b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.389+0000 7f4e3659c640 1 --2- 172.21.15.90:0/3611597346 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f4e200afc00 0x7f4e200b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.389+0000 7f4e277fe640 1 -- 172.21.15.90:0/3611597346 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f4e2c0defa0 con 0x7f4e38104d40 2024-04-25T11:34:35.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.389+0000 7f4e3659c640 1 --2- 172.21.15.90:0/3611597346 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f4e200afc00 0x7f4e200b2110 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4e280058c0 tx=0x7f4e28005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:35.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.393+0000 7f4e277fe640 1 -- 172.21.15.90:0/3611597346 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4e2c0a79a0 con 0x7f4e38104d40 2024-04-25T11:34:35.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.609+0000 7f4e3cd0d640 1 -- 172.21.15.90:0/3611597346 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e3802db50 con 0x7f4e38104d40 2024-04-25T11:34:35.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.609+0000 7f4e277fe640 1 -- 172.21.15.90:0/3611597346 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11930 (secure 0 0 0) 0x7f4e2c0a72a0 con 0x7f4e38104d40 2024-04-25T11:34:35.599 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:34:35.599 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:33:16.640084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524,24475,24499,24458,24431]},"id":1}]} 2024-04-25T11:34:35.599 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 25 2024-04-25T11:34:35.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.612+0000 7f4e3cd0d640 1 -- 172.21.15.90:0/3611597346 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f4e200afc00 msgr2=0x7f4e200b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.612+0000 7f4e3cd0d640 1 --2- 172.21.15.90:0/3611597346 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f4e200afc00 0x7f4e200b2110 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4e280058c0 tx=0x7f4e28005850 comp rx=0 tx=0).stop 2024-04-25T11:34:35.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.612+0000 7f4e3cd0d640 1 -- 172.21.15.90:0/3611597346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e38104d40 msgr2=0x7f4e3819a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.612+0000 7f4e3cd0d640 1 --2- 172.21.15.90:0/3611597346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e38104d40 0x7f4e3819a750 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c00d950 tx=0x7f4e2c00de10 comp rx=0 tx=0).stop 2024-04-25T11:34:35.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.612+0000 7f4e3cd0d640 1 -- 172.21.15.90:0/3611597346 shutdown_connections 2024-04-25T11:34:35.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.612+0000 7f4e3cd0d640 1 --2- 172.21.15.90:0/3611597346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e38109a40 0x7f4e381a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.612+0000 7f4e3cd0d640 1 --2- 172.21.15.90:0/3611597346 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f4e200afc00 0x7f4e200b2110 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.612+0000 7f4e3cd0d640 1 --2- 172.21.15.90:0/3611597346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e38104d40 0x7f4e3819a750 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.612+0000 7f4e3cd0d640 1 --2- 172.21.15.90:0/3611597346 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4e38103b00 0x7f4e3819a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.612+0000 7f4e3cd0d640 1 -- 172.21.15.90:0/3611597346 >> 172.21.15.90:0/3611597346 conn(0x7f4e380fed10 msgr2=0x7f4e38105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:35.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.612+0000 7f4e3cd0d640 1 -- 172.21.15.90:0/3611597346 shutdown_connections 2024-04-25T11:34:35.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.612+0000 7f4e3cd0d640 1 -- 172.21.15.90:0/3611597346 wait complete. 2024-04-25T11:34:35.612 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T11:33:10.373468+0000', 'modified': '2024-04-25T11:33:16.640084+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24524, 'mds_1': 24499, 'mds_2': 24475, 'mds_3': 24458, 'mds_4': 24431}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14637': {'gid': 14637, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.90:6839/3101160958', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6838', 'nonce': 3101160958}, {'type': 'v1', 'addr': '172.21.15.90:6839', 'nonce': 3101160958}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14649': {'gid': 14649, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.90:6841/2617461073', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6840', 'nonce': 2617461073}, {'type': 'v1', 'addr': '172.21.15.90:6841', 'nonce': 2617461073}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24430': {'gid': 24430, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.156:6833/31237375', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6832', 'nonce': 31237375}, {'type': 'v1', 'addr': '172.21.15.156:6833', 'nonce': 31237375}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24431': {'gid': 24431, 'name': 'k', 'rank': 4, 'incarnation': 23, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.138:6833/51785544', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6832', 'nonce': 51785544}, {'type': 'v1', 'addr': '172.21.15.138:6833', 'nonce': 51785544}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24458': {'gid': 24458, 'name': 'e', 'rank': 3, 'incarnation': 21, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.138:6835/162690028', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6834', 'nonce': 162690028}, {'type': 'v1', 'addr': '172.21.15.138:6835', 'nonce': 162690028}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24475': {'gid': 24475, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.156:6835/1641860251', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6834', 'nonce': 1641860251}, {'type': 'v1', 'addr': '172.21.15.156:6835', 'nonce': 1641860251}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24481': {'gid': 24481, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.156:6837/3466257977', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6836', 'nonce': 3466257977}, {'type': 'v1', 'addr': '172.21.15.156:6837', 'nonce': 3466257977}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24485': {'gid': 24485, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.138:6837/2598829089', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6836', 'nonce': 2598829089}, {'type': 'v1', 'addr': '172.21.15.138:6837', 'nonce': 2598829089}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'i', 'rank': 1, 'incarnation': 17, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.156:6839/4179744597', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6838', 'nonce': 4179744597}, {'type': 'v1', 'addr': '172.21.15.156:6839', 'nonce': 4179744597}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24524': {'gid': 24524, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.138:6839/1849836338', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6838', 'nonce': 1849836338}, {'type': 'v1', 'addr': '172.21.15.138:6839', 'nonce': 1849836338}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24524, 'qdb_cluster': [24524, 24475, 24499, 24458, 24431]} 2024-04-25T11:34:35.613 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-25T11:34:35.613 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.k status 2024-04-25T11:34:35.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.732+0000 7f0b06a82640 1 -- 172.21.15.90:0/3205570249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b00106f40 msgr2=0x7f0b00109360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.732+0000 7f0b06a82640 1 --2- 172.21.15.90:0/3205570249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b00106f40 0x7f0b00109360 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0af4009900 tx=0x7f0af402a410 comp rx=0 tx=0).stop 2024-04-25T11:34:35.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.733+0000 7f0b06a82640 1 -- 172.21.15.90:0/3205570249 shutdown_connections 2024-04-25T11:34:35.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.733+0000 7f0b06a82640 1 --2- 172.21.15.90:0/3205570249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b001098a0 0x7f0b0010bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.733+0000 7f0b06a82640 1 --2- 172.21.15.90:0/3205570249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b00106f40 0x7f0b00109360 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.733+0000 7f0b06a82640 1 --2- 172.21.15.90:0/3205570249 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b001045e0 0x7f0b00106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.733+0000 7f0b06a82640 1 -- 172.21.15.90:0/3205570249 >> 172.21.15.90:0/3205570249 conn(0x7f0b000fdbe0 msgr2=0x7f0b00100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:35.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.733+0000 7f0b06a82640 1 -- 172.21.15.90:0/3205570249 shutdown_connections 2024-04-25T11:34:35.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.733+0000 7f0b06a82640 1 -- 172.21.15.90:0/3205570249 wait complete. 2024-04-25T11:34:35.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.733+0000 7f0b06a82640 1 Processor -- start 2024-04-25T11:34:35.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.733+0000 7f0b06a82640 1 -- start start 2024-04-25T11:34:35.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.733+0000 7f0b06a82640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b001045e0 0x7f0b0019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.733+0000 7f0b06a82640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b00106f40 0x7f0b0019a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.733+0000 7f0b06a82640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b001098a0 0x7f0b001a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.733+0000 7f0b06a82640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0b0010f600 con 0x7f0b00106f40 2024-04-25T11:34:35.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.733+0000 7f0b06a82640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0b0010f480 con 0x7f0b001045e0 2024-04-25T11:34:35.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.733+0000 7f0b06a82640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0b0010f780 con 0x7f0b001098a0 2024-04-25T11:34:35.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.733+0000 7f0afffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b00106f40 0x7f0b0019a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.734+0000 7f0afffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b00106f40 0x7f0b0019a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34886/0 (socket says 172.21.15.90:34886) 2024-04-25T11:34:35.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.734+0000 7f0afffff640 1 -- 172.21.15.90:0/1176052085 learned_addr learned my addr 172.21.15.90:0/1176052085 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:35.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.734+0000 7f0b0501f640 1 --2- 172.21.15.90:0/1176052085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b001098a0 0x7f0b001a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.734+0000 7f0b0481e640 1 --2- 172.21.15.90:0/1176052085 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b001045e0 0x7f0b0019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.734+0000 7f0afffff640 1 -- 172.21.15.90:0/1176052085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b001098a0 msgr2=0x7f0b001a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.734+0000 7f0afffff640 1 --2- 172.21.15.90:0/1176052085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b001098a0 0x7f0b001a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.734+0000 7f0afffff640 1 -- 172.21.15.90:0/1176052085 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b001045e0 msgr2=0x7f0b0019a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.734+0000 7f0afffff640 1 --2- 172.21.15.90:0/1176052085 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b001045e0 0x7f0b0019a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.734+0000 7f0afffff640 1 -- 172.21.15.90:0/1176052085 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b001a1d20 con 0x7f0b00106f40 2024-04-25T11:34:35.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.734+0000 7f0b0501f640 1 --2- 172.21.15.90:0/1176052085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b001098a0 0x7f0b001a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:35.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.734+0000 7f0b0481e640 1 --2- 172.21.15.90:0/1176052085 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b001045e0 0x7f0b0019a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:35.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.734+0000 7f0afffff640 1 --2- 172.21.15.90:0/1176052085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b00106f40 0x7f0b0019a7d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0af4006010 tx=0x7f0af40043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:35.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.734+0000 7f0afdffb640 1 -- 172.21.15.90:0/1176052085 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af403f030 con 0x7f0b00106f40 2024-04-25T11:34:35.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.734+0000 7f0afdffb640 1 -- 172.21.15.90:0/1176052085 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0af4003cb0 con 0x7f0b00106f40 2024-04-25T11:34:35.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.734+0000 7f0b06a82640 1 -- 172.21.15.90:0/1176052085 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b001a1fb0 con 0x7f0b00106f40 2024-04-25T11:34:35.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.734+0000 7f0afdffb640 1 -- 172.21.15.90:0/1176052085 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af4035070 con 0x7f0b00106f40 2024-04-25T11:34:35.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.734+0000 7f0b06a82640 1 -- 172.21.15.90:0/1176052085 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b001a23c0 con 0x7f0b00106f40 2024-04-25T11:34:35.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.737+0000 7f0afdffb640 1 -- 172.21.15.90:0/1176052085 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f0af4042030 con 0x7f0b00106f40 2024-04-25T11:34:35.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.737+0000 7f0b0da0d740 1 Processor -- start 2024-04-25T11:34:35.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.737+0000 7f0b0da0d740 1 -- start start 2024-04-25T11:34:35.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.737+0000 7f0b0da0d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55616913b270 0x5561694dc370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.737+0000 7f0b0da0d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561691123b0 0x5561694dc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.737+0000 7f0b0da0d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55616911a590 0x5561694e37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.737+0000 7f0b0da0d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556169441740 con 0x55616911a590 2024-04-25T11:34:35.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.737+0000 7f0b0da0d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5561693dd3a0 con 0x55616913b270 2024-04-25T11:34:35.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.737+0000 7f0b0da0d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55616943d700 con 0x5561691123b0 2024-04-25T11:34:35.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.737+0000 7f0afffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55616911a590 0x5561694e37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.737+0000 7f0afffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55616911a590 0x5561694e37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34894/0 (socket says 172.21.15.90:34894) 2024-04-25T11:34:35.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.737+0000 7f0afffff640 1 -- 172.21.15.90:0/2970088521 learned_addr learned my addr 172.21.15.90:0/2970088521 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:35.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.737+0000 7f0b0501f640 1 --2- 172.21.15.90:0/2970088521 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55616913b270 0x5561694dc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.737+0000 7f0afdffb640 1 --2- 172.21.15.90:0/1176052085 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0ae80afb80 0x7f0ae80b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.738+0000 7f0b0481e640 1 --2- 172.21.15.90:0/2970088521 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561691123b0 0x5561694dc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.738+0000 7f0afffff640 1 -- 172.21.15.90:0/2970088521 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561691123b0 msgr2=0x5561694dc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:35.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.738+0000 7f0b0501f640 1 --2- 172.21.15.90:0/1176052085 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0ae80afb80 0x7f0ae80b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.738+0000 7f0afffff640 1 --2- 172.21.15.90:0/2970088521 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561691123b0 0x5561694dc8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.738+0000 7f0afffff640 1 -- 172.21.15.90:0/2970088521 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55616913b270 msgr2=0x5561694dc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:35.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.738+0000 7f0afffff640 1 --2- 172.21.15.90:0/2970088521 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55616913b270 0x5561694dc370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.738+0000 7f0afffff640 1 -- 172.21.15.90:0/2970088521 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561693f9750 con 0x55616911a590 2024-04-25T11:34:35.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.738+0000 7f0b0481e640 1 --2- 172.21.15.90:0/2970088521 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561691123b0 0x5561694dc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:35.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.738+0000 7f0afdffb640 1 -- 172.21.15.90:0/1176052085 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f0af40ff9c0 con 0x7f0b00106f40 2024-04-25T11:34:35.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.738+0000 7f0afffff640 1 --2- 172.21.15.90:0/2970088521 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55616911a590 0x5561694e37f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0af40fd290 tx=0x7f0af4082a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2d4f12ac8f9ce1f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:35.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.738+0000 7f0b0501f640 1 --2- 172.21.15.90:0/1176052085 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0ae80afb80 0x7f0ae80b2090 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x5561694dd540 tx=0x7f0af8004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:35.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.738+0000 7f0ae6ffd640 1 -- 172.21.15.90:0/2970088521 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af40c7af0 con 0x55616911a590 2024-04-25T11:34:35.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.738+0000 7f0ae6ffd640 1 -- 172.21.15.90:0/2970088521 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0af4086790 con 0x55616911a590 2024-04-25T11:34:35.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.738+0000 7f0ae6ffd640 1 -- 172.21.15.90:0/2970088521 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af40c7490 con 0x55616911a590 2024-04-25T11:34:35.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.739+0000 7f0b0da0d740 1 -- 172.21.15.90:0/2970088521 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55616911a590 msgr2=0x5561694e37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.739+0000 7f0b0da0d740 1 --2- 172.21.15.90:0/2970088521 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55616911a590 0x5561694e37f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0af40fd290 tx=0x7f0af4082a90 comp rx=0 tx=0).stop 2024-04-25T11:34:35.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.739+0000 7f0b0da0d740 1 -- 172.21.15.90:0/2970088521 shutdown_connections 2024-04-25T11:34:35.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.739+0000 7f0b0da0d740 1 --2- 172.21.15.90:0/2970088521 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55616911a590 0x5561694e37f0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.739+0000 7f0b0da0d740 1 --2- 172.21.15.90:0/2970088521 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561691123b0 0x5561694dc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.739+0000 7f0b0da0d740 1 --2- 172.21.15.90:0/2970088521 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55616913b270 0x5561694dc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.739+0000 7f0b0da0d740 1 -- 172.21.15.90:0/2970088521 >> 172.21.15.90:0/2970088521 conn(0x55616912f3c0 msgr2=0x5561694d7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:35.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.739+0000 7f0b0da0d740 1 -- 172.21.15.90:0/2970088521 shutdown_connections 2024-04-25T11:34:35.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.739+0000 7f0b0da0d740 1 -- 172.21.15.90:0/2970088521 wait complete. 2024-04-25T11:34:35.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.739+0000 7f0b0da0d740 1 -- start start 2024-04-25T11:34:35.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.739+0000 7f0b0da0d740 1 Processor -- start 2024-04-25T11:34:35.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.740+0000 7f0b0da0d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5561691123b0 0x5561694e5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.740+0000 7f0b0da0d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55616911a590 0x5561694e62f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.740+0000 7f0b0da0d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55616913b270 0x5561694ed250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.740+0000 7f0b0da0d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556169441740 con 0x55616913b270 2024-04-25T11:34:35.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.740+0000 7f0b0da0d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5561693c8fe0 con 0x5561691123b0 2024-04-25T11:34:35.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.740+0000 7f0b0da0d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5561694e4000 con 0x55616911a590 2024-04-25T11:34:35.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.740+0000 7f0b0da0d740 10 client.0 _ll_register_callbacks cb 0x5561693df150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:34:35.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.740+0000 7f0b0501f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55616913b270 0x5561694ed250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.740+0000 7f0b0501f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55616913b270 0x5561694ed250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34908/0 (socket says 172.21.15.90:34908) 2024-04-25T11:34:35.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.740+0000 7f0b0501f640 1 -- 172.21.15.90:0/1173205042 learned_addr learned my addr 172.21.15.90:0/1173205042 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:35.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.740+0000 7f0b0481e640 1 --2- 172.21.15.90:0/1173205042 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5561691123b0 0x5561694e5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.740+0000 7f0afffff640 1 --2- 172.21.15.90:0/1173205042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55616911a590 0x5561694e62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.740+0000 7f0b0501f640 1 -- 172.21.15.90:0/1173205042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55616911a590 msgr2=0x5561694e62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:35.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.740+0000 7f0b0501f640 1 --2- 172.21.15.90:0/1173205042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55616911a590 0x5561694e62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.740+0000 7f0b0501f640 1 -- 172.21.15.90:0/1173205042 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5561691123b0 msgr2=0x5561694e5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:35.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.740+0000 7f0b0501f640 1 --2- 172.21.15.90:0/1173205042 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5561691123b0 0x5561694e5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.740+0000 7f0b0501f640 1 -- 172.21.15.90:0/1173205042 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561691532d0 con 0x55616913b270 2024-04-25T11:34:35.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.741+0000 7f0b0501f640 1 --2- 172.21.15.90:0/1173205042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55616913b270 0x5561694ed250 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0af800f6c0 tx=0x7f0af8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71e33a6c3295e448 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:35.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.741+0000 7f0adffff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:34:35.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.741+0000 7f0adffff640 1 -- 172.21.15.90:0/1173205042 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af8018070 con 0x55616913b270 2024-04-25T11:34:35.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.741+0000 7f0adffff640 1 -- 172.21.15.90:0/1173205042 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f0af8004d20 con 0x55616913b270 2024-04-25T11:34:35.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.741+0000 7f0b0da0d740 1 -- 172.21.15.90:0/1173205042 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55616944f350 con 0x55616913b270 2024-04-25T11:34:35.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.741+0000 7f0adffff640 1 -- 172.21.15.90:0/1173205042 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0af8005020 con 0x55616913b270 2024-04-25T11:34:35.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.741+0000 7f0b0da0d740 10 client.15066 fetch_fsmap learned FSMap version 25 2024-04-25T11:34:35.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.741+0000 7f0b0da0d740 1 -- 172.21.15.90:0/1173205042 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55616921d6b0 con 0x55616913b270 2024-04-25T11:34:35.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.741+0000 7f0adffff640 1 -- 172.21.15.90:0/1173205042 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55616944f350 con 0x55616913b270 2024-04-25T11:34:35.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.741+0000 7f0adffff640 1 -- 172.21.15.90:0/1173205042 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f0af8027490 con 0x55616913b270 2024-04-25T11:34:35.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.741+0000 7f0b0da0d740 10 client.15066 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T11:34:35.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.741+0000 7f0b0da0d740 10 client.15066 resolve_mds: resolved name 'k' to daemon mds.k 2024-04-25T11:34:35.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.741+0000 7f0b0da0d740 20 client.15066 populate_metadata read hostname 'smithi090' 2024-04-25T11:34:35.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.741+0000 7f0b0da0d740 1 --2- 172.21.15.90:0/1173205042 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x5561690f9e40 0x5561694f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.741+0000 7f0b0da0d740 4 client.15066 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:34:35.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.741+0000 7f0b0da0d740 1 -- 172.21.15.90:0/1173205042 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561694491c0 con 0x5561690f9e40 2024-04-25T11:34:35.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.742+0000 7f0b0481e640 1 --2- 172.21.15.90:0/1173205042 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x5561690f9e40 0x5561694f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.742+0000 7f0adffff640 10 client.15066 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:34:35.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.742+0000 7f0b0481e640 1 --2- 172.21.15.90:0/1173205042 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x5561690f9e40 0x5561694f3a80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3323a3f28533335a server_cookie=292ef9d5f16d6767 in_seq=0 out_seq=0 2024-04-25T11:34:35.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.743+0000 7f0adffff640 1 -- 172.21.15.90:0/1173205042 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5561694491c0 con 0x5561690f9e40 2024-04-25T11:34:35.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.743+0000 7f0adffff640 10 client.15066 handle_command_reply: tid=0 2024-04-25T11:34:35.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.743+0000 7f0b0da0d740 1 client.15066 shutdown 2024-04-25T11:34:35.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.743+0000 7f0b0da0d740 1 -- 172.21.15.90:0/1173205042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55616913b270 msgr2=0x5561694ed250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.743+0000 7f0b0da0d740 1 --2- 172.21.15.90:0/1173205042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55616913b270 0x5561694ed250 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0af800f6c0 tx=0x7f0af8004a40 comp rx=0 tx=0).stop 2024-04-25T11:34:35.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.743+0000 7f0b0da0d740 1 -- 172.21.15.90:0/1173205042 shutdown_connections 2024-04-25T11:34:35.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.743+0000 7f0b0da0d740 1 --2- 172.21.15.90:0/1173205042 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x5561690f9e40 0x5561694f3a80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.743+0000 7f0b0da0d740 1 --2- 172.21.15.90:0/1173205042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55616913b270 0x5561694ed250 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.743+0000 7f0b0da0d740 1 --2- 172.21.15.90:0/1173205042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55616911a590 0x5561694e62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.743+0000 7f0b0da0d740 1 --2- 172.21.15.90:0/1173205042 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5561691123b0 0x5561694e5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.743+0000 7f0adffff640 0 client.15066 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:34:35.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.743+0000 7f0b0da0d740 1 -- 172.21.15.90:0/1173205042 >> 172.21.15.90:0/1173205042 conn(0x55616912f3c0 msgr2=0x5561694d6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:35.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.743+0000 7f0b0da0d740 1 -- 172.21.15.90:0/1173205042 shutdown_connections 2024-04-25T11:34:35.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.743+0000 7f0b0da0d740 1 -- 172.21.15.90:0/1173205042 wait complete. 2024-04-25T11:34:35.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.743+0000 7f0b0da0d740 20 client.15066 trim_cache size 0 max 16384 2024-04-25T11:34:35.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.762+0000 7f0b0da0d740 1 Processor -- start 2024-04-25T11:34:35.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.762+0000 7f0b0da0d740 1 -- start start 2024-04-25T11:34:35.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.762+0000 7f0b0da0d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5561691123b0 0x5561694d76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.762+0000 7f0b0da0d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55616911a590 0x5561694d7c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.762+0000 7f0b0da0d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55616913b270 0x5561694ed170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.762+0000 7f0b0da0d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5561694f1dc0 con 0x5561691123b0 2024-04-25T11:34:35.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.762+0000 7f0b0da0d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5561694f2490 con 0x55616911a590 2024-04-25T11:34:35.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.762+0000 7f0b0da0d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5561694f2b60 con 0x55616913b270 2024-04-25T11:34:35.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.762+0000 7f0b0481e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5561691123b0 0x5561694d76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.762+0000 7f0b0481e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5561691123b0 0x5561694d76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34918/0 (socket says 172.21.15.90:34918) 2024-04-25T11:34:35.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.762+0000 7f0b0481e640 1 -- 172.21.15.90:0/1013634268 learned_addr learned my addr 172.21.15.90:0/1013634268 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:35.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.763+0000 7f0afffff640 1 --2- 172.21.15.90:0/1013634268 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55616911a590 0x5561694d7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.763+0000 7f0b0481e640 1 -- 172.21.15.90:0/1013634268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55616913b270 msgr2=0x5561694ed170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:35.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.763+0000 7f0b0481e640 1 --2- 172.21.15.90:0/1013634268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55616913b270 0x5561694ed170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.763+0000 7f0b0481e640 1 -- 172.21.15.90:0/1013634268 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55616911a590 msgr2=0x5561694d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:35.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.763+0000 7f0b0481e640 1 --2- 172.21.15.90:0/1013634268 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55616911a590 0x5561694d7c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.763+0000 7f0b0481e640 1 -- 172.21.15.90:0/1013634268 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561694e1130 con 0x5561691123b0 2024-04-25T11:34:35.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.763+0000 7f0b0481e640 1 --2- 172.21.15.90:0/1013634268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5561691123b0 0x5561694d76f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0af00058c0 tx=0x7f0af000a510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=925cfa28e2e5ac87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:35.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.763+0000 7f0ae67fc640 1 -- 172.21.15.90:0/1013634268 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af001e070 con 0x5561691123b0 2024-04-25T11:34:35.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.763+0000 7f0ae67fc640 1 -- 172.21.15.90:0/1013634268 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0af001a970 con 0x5561691123b0 2024-04-25T11:34:35.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.763+0000 7f0ae67fc640 1 -- 172.21.15.90:0/1013634268 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af0010d20 con 0x5561691123b0 2024-04-25T11:34:35.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.764+0000 7f0b0da0d740 1 -- 172.21.15.90:0/1013634268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5561691123b0 msgr2=0x5561694d76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.764+0000 7f0b0da0d740 1 --2- 172.21.15.90:0/1013634268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5561691123b0 0x5561694d76f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0af00058c0 tx=0x7f0af000a510 comp rx=0 tx=0).stop 2024-04-25T11:34:35.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.764+0000 7f0b0da0d740 1 -- 172.21.15.90:0/1013634268 shutdown_connections 2024-04-25T11:34:35.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.764+0000 7f0b0da0d740 1 --2- 172.21.15.90:0/1013634268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55616913b270 0x5561694ed170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.764+0000 7f0b0da0d740 1 --2- 172.21.15.90:0/1013634268 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55616911a590 0x5561694d7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.764+0000 7f0b0da0d740 1 --2- 172.21.15.90:0/1013634268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5561691123b0 0x5561694d76f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.764+0000 7f0b0da0d740 1 -- 172.21.15.90:0/1013634268 >> 172.21.15.90:0/1013634268 conn(0x55616912f3c0 msgr2=0x5561694e43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:35.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.764+0000 7f0b0da0d740 1 -- 172.21.15.90:0/1013634268 shutdown_connections 2024-04-25T11:34:35.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.764+0000 7f0b0da0d740 1 -- 172.21.15.90:0/1013634268 wait complete. 2024-04-25T11:34:35.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.764+0000 7f0b0da0d740 1 -- start start 2024-04-25T11:34:35.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.764+0000 7f0b0da0d740 1 Processor -- start 2024-04-25T11:34:35.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0b0da0d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5561691123b0 0x5561694e91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0b0da0d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55616911a590 0x5561694e96f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0b0da0d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55616913b270 0x5561694ec630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0b0da0d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5561694f3900 con 0x5561691123b0 2024-04-25T11:34:35.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0b0da0d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5561694f3230 con 0x55616911a590 2024-04-25T11:34:35.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0b0da0d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5561694f3fd0 con 0x55616913b270 2024-04-25T11:34:35.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0b0da0d740 10 client.0 _ll_register_callbacks cb 0x7f0af8007ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:34:35.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0b0481e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5561691123b0 0x5561694e91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0b0481e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5561691123b0 0x5561694e91b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34926/0 (socket says 172.21.15.90:34926) 2024-04-25T11:34:35.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0b0481e640 1 -- 172.21.15.90:0/3993500877 learned_addr learned my addr 172.21.15.90:0/3993500877 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:35.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0b0481e640 1 -- 172.21.15.90:0/3993500877 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55616913b270 msgr2=0x5561694ec630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:35.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0afffff640 1 --2- 172.21.15.90:0/3993500877 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55616911a590 0x5561694e96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0b0501f640 1 --2- 172.21.15.90:0/3993500877 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55616913b270 0x5561694ec630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0b0481e640 1 --2- 172.21.15.90:0/3993500877 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55616913b270 0x5561694ec630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0b0481e640 1 -- 172.21.15.90:0/3993500877 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55616911a590 msgr2=0x5561694e96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:35.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0b0481e640 1 --2- 172.21.15.90:0/3993500877 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55616911a590 0x5561694e96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0b0481e640 1 -- 172.21.15.90:0/3993500877 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561694ecbf0 con 0x5561691123b0 2024-04-25T11:34:35.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0afffff640 1 --2- 172.21.15.90:0/3993500877 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55616911a590 0x5561694e96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:35.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0b0501f640 1 --2- 172.21.15.90:0/3993500877 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55616913b270 0x5561694ec630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:35.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0b0481e640 1 --2- 172.21.15.90:0/3993500877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5561691123b0 0x5561694e91b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0af000ac10 tx=0x7f0af0010670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20c4850388dfe26c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:35.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0adffff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:34:35.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0adffff640 1 -- 172.21.15.90:0/3993500877 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af001e050 con 0x5561691123b0 2024-04-25T11:34:35.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0adffff640 1 -- 172.21.15.90:0/3993500877 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f0af0010d20 con 0x5561691123b0 2024-04-25T11:34:35.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0b0da0d740 1 -- 172.21.15.90:0/3993500877 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0af001a970 con 0x5561691123b0 2024-04-25T11:34:35.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0adffff640 1 -- 172.21.15.90:0/3993500877 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0af001a2c0 con 0x5561691123b0 2024-04-25T11:34:35.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0b0da0d740 10 client.15072 fetch_fsmap learned FSMap version 25 2024-04-25T11:34:35.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.765+0000 7f0b0da0d740 1 -- 172.21.15.90:0/3993500877 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55616955c900 con 0x5561691123b0 2024-04-25T11:34:35.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.766+0000 7f0adffff640 1 -- 172.21.15.90:0/3993500877 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0af001a970 con 0x5561691123b0 2024-04-25T11:34:35.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.766+0000 7f0adffff640 1 -- 172.21.15.90:0/3993500877 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f0af0023ca0 con 0x5561691123b0 2024-04-25T11:34:35.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.766+0000 7f0b0da0d740 10 client.15072 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T11:34:35.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.766+0000 7f0b0da0d740 10 client.15072 resolve_mds: resolved name 'k' to daemon mds.k 2024-04-25T11:34:35.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.766+0000 7f0b0da0d740 20 client.15072 populate_metadata read hostname 'smithi090' 2024-04-25T11:34:35.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.766+0000 7f0b0da0d740 1 --2- 172.21.15.90:0/3993500877 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x5561690cf280 0x55616955ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.766+0000 7f0b0da0d740 4 client.15072 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T11:34:35.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.766+0000 7f0b0da0d740 1 -- 172.21.15.90:0/3993500877 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "status"}) -- 0x5561693c49a0 con 0x5561690cf280 2024-04-25T11:34:35.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.766+0000 7f0afffff640 1 --2- 172.21.15.90:0/3993500877 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x5561690cf280 0x55616955ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.767+0000 7f0adffff640 10 client.15072 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:34:35.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.767+0000 7f0afffff640 1 --2- 172.21.15.90:0/3993500877 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x5561690cf280 0x55616955ffb0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4f84e3da7d0404d3 server_cookie=828b83eaa0d772bc in_seq=0 out_seq=0 2024-04-25T11:34:35.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.767+0000 7f0adffff640 1 -- 172.21.15.90:0/3993500877 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5561693c49a0 con 0x5561690cf280 2024-04-25T11:34:35.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.767+0000 7f0adffff640 10 client.15072 handle_command_reply: tid=0 2024-04-25T11:34:35.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.767+0000 7f0b0da0d740 1 client.15072 shutdown 2024-04-25T11:34:35.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.767+0000 7f0b0da0d740 1 -- 172.21.15.90:0/3993500877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5561691123b0 msgr2=0x5561694e91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.767+0000 7f0b0da0d740 1 --2- 172.21.15.90:0/3993500877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5561691123b0 0x5561694e91b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0af000ac10 tx=0x7f0af0010670 comp rx=0 tx=0).stop 2024-04-25T11:34:35.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.767+0000 7f0b0da0d740 1 -- 172.21.15.90:0/3993500877 shutdown_connections 2024-04-25T11:34:35.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.767+0000 7f0b0da0d740 1 --2- 172.21.15.90:0/3993500877 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x5561690cf280 0x55616955ffb0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.767+0000 7f0b0da0d740 1 --2- 172.21.15.90:0/3993500877 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55616913b270 0x5561694ec630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.767+0000 7f0adffff640 0 client.15072 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:34:35.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.767+0000 7f0b0da0d740 1 --2- 172.21.15.90:0/3993500877 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55616911a590 0x5561694e96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.767+0000 7f0b0da0d740 1 --2- 172.21.15.90:0/3993500877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5561691123b0 0x5561694e91b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.767+0000 7f0b0da0d740 1 -- 172.21.15.90:0/3993500877 >> 172.21.15.90:0/3993500877 conn(0x55616912f3c0 msgr2=0x5561694d76f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:35.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.767+0000 7f0b0da0d740 1 -- 172.21.15.90:0/3993500877 shutdown_connections 2024-04-25T11:34:35.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.767+0000 7f0b0da0d740 1 -- 172.21.15.90:0/3993500877 wait complete. 2024-04-25T11:34:35.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.767+0000 7f0b0da0d740 20 client.15072 trim_cache size 0 max 16384 2024-04-25T11:34:35.763 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:34:35.763 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "58bd4e7a-02f6-11ef-bc93-c7b262605968", 2024-04-25T11:34:35.763 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 4, 2024-04-25T11:34:35.763 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 24431, 2024-04-25T11:34:35.764 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-04-25T11:34:35.764 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-04-25T11:34:35.764 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-04-25T11:34:35.764 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 80.132890818000007, 2024-04-25T11:34:35.764 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 25, 2024-04-25T11:34:35.764 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 79, 2024-04-25T11:34:35.764 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 77, 2024-04-25T11:34:35.764 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 117.867946128 2024-04-25T11:34:35.764 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:34:35.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.768+0000 7f0b06a82640 1 -- 172.21.15.90:0/1176052085 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0ae80afb80 msgr2=0x7f0ae80b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.768+0000 7f0b06a82640 1 --2- 172.21.15.90:0/1176052085 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0ae80afb80 0x7f0ae80b2090 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x5561694dd540 tx=0x7f0af8004ab0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.768+0000 7f0b06a82640 1 -- 172.21.15.90:0/1176052085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b00106f40 msgr2=0x7f0b0019a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.768+0000 7f0b06a82640 1 --2- 172.21.15.90:0/1176052085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b00106f40 0x7f0b0019a7d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0af4006010 tx=0x7f0af40043b0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.768+0000 7f0b06a82640 1 -- 172.21.15.90:0/1176052085 shutdown_connections 2024-04-25T11:34:35.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.768+0000 7f0b06a82640 1 --2- 172.21.15.90:0/1176052085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b001098a0 0x7f0b001a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.768+0000 7f0b06a82640 1 --2- 172.21.15.90:0/1176052085 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0ae80afb80 0x7f0ae80b2090 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.768+0000 7f0b06a82640 1 --2- 172.21.15.90:0/1176052085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b00106f40 0x7f0b0019a7d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.768+0000 7f0b06a82640 1 --2- 172.21.15.90:0/1176052085 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b001045e0 0x7f0b0019a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.768+0000 7f0b06a82640 1 -- 172.21.15.90:0/1176052085 >> 172.21.15.90:0/1176052085 conn(0x7f0b000fdbe0 msgr2=0x7f0b00106610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:35.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.768+0000 7f0b06a82640 1 -- 172.21.15.90:0/1176052085 shutdown_connections 2024-04-25T11:34:35.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.768+0000 7f0b06a82640 1 -- 172.21.15.90:0/1176052085 wait complete. 2024-04-25T11:34:35.768 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-04-25T11:34:35.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.888+0000 7f775457d640 1 -- 172.21.15.90:0/1953055401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f774c109a30 msgr2=0x7f774c10be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.888+0000 7f775457d640 1 --2- 172.21.15.90:0/1953055401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f774c109a30 0x7f774c10be50 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f773c009900 tx=0x7f773c02a410 comp rx=0 tx=0).stop 2024-04-25T11:34:35.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.888+0000 7f775457d640 1 -- 172.21.15.90:0/1953055401 shutdown_connections 2024-04-25T11:34:35.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.888+0000 7f775457d640 1 --2- 172.21.15.90:0/1953055401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f774c109a30 0x7f774c10be50 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.888+0000 7f775457d640 1 --2- 172.21.15.90:0/1953055401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f774c104d30 0x7f774c105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.888+0000 7f775457d640 1 --2- 172.21.15.90:0/1953055401 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f774c103af0 0x7f774c1047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.888+0000 7f775457d640 1 -- 172.21.15.90:0/1953055401 >> 172.21.15.90:0/1953055401 conn(0x7f774c0fece0 msgr2=0x7f774c101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:35.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.888+0000 7f775457d640 1 -- 172.21.15.90:0/1953055401 shutdown_connections 2024-04-25T11:34:35.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.888+0000 7f775457d640 1 -- 172.21.15.90:0/1953055401 wait complete. 2024-04-25T11:34:35.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.888+0000 7f775457d640 1 Processor -- start 2024-04-25T11:34:35.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.888+0000 7f775457d640 1 -- start start 2024-04-25T11:34:35.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.889+0000 7f775457d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f774c103af0 0x7f774c19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.889+0000 7f775457d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f774c104d30 0x7f774c19a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.889+0000 7f775457d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f774c109a30 0x7f774c1a1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.889+0000 7f775457d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f774c10f620 con 0x7f774c104d30 2024-04-25T11:34:35.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.889+0000 7f775457d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f774c10f4a0 con 0x7f774c109a30 2024-04-25T11:34:35.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.889+0000 7f775457d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f774c10f7a0 con 0x7f774c103af0 2024-04-25T11:34:35.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.889+0000 7f7751b18640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f774c104d30 0x7f774c19a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.889+0000 7f7751b18640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f774c104d30 0x7f774c19a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34936/0 (socket says 172.21.15.90:34936) 2024-04-25T11:34:35.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.889+0000 7f7751b18640 1 -- 172.21.15.90:0/4091782382 learned_addr learned my addr 172.21.15.90:0/4091782382 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:35.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.889+0000 7f7752319640 1 --2- 172.21.15.90:0/4091782382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f774c103af0 0x7f774c19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.889+0000 7f7752b1a640 1 --2- 172.21.15.90:0/4091782382 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f774c109a30 0x7f774c1a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.889+0000 7f7751b18640 1 -- 172.21.15.90:0/4091782382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f774c103af0 msgr2=0x7f774c19a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.889+0000 7f7751b18640 1 --2- 172.21.15.90:0/4091782382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f774c103af0 0x7f774c19a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.889+0000 7f7751b18640 1 -- 172.21.15.90:0/4091782382 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f774c109a30 msgr2=0x7f774c1a1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.889+0000 7f7751b18640 1 --2- 172.21.15.90:0/4091782382 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f774c109a30 0x7f774c1a1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.889+0000 7f7751b18640 1 -- 172.21.15.90:0/4091782382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f774c1a1c30 con 0x7f774c104d30 2024-04-25T11:34:35.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.889+0000 7f7751b18640 1 --2- 172.21.15.90:0/4091782382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f774c104d30 0x7f774c19a6e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f774000c970 tx=0x7f774000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:35.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.889+0000 7f774b7fe640 1 -- 172.21.15.90:0/4091782382 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7740007bb0 con 0x7f774c104d30 2024-04-25T11:34:35.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.889+0000 7f775457d640 1 -- 172.21.15.90:0/4091782382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f774c1a1f20 con 0x7f774c104d30 2024-04-25T11:34:35.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.889+0000 7f774b7fe640 1 -- 172.21.15.90:0/4091782382 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7740007d50 con 0x7f774c104d30 2024-04-25T11:34:35.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.889+0000 7f774b7fe640 1 -- 172.21.15.90:0/4091782382 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f774000f450 con 0x7f774c104d30 2024-04-25T11:34:35.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.889+0000 7f775457d640 1 -- 172.21.15.90:0/4091782382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f774c1a2390 con 0x7f774c104d30 2024-04-25T11:34:35.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.892+0000 7f775b42f740 1 Processor -- start 2024-04-25T11:34:35.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.892+0000 7f774b7fe640 1 -- 172.21.15.90:0/4091782382 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f7740004f20 con 0x7f774c104d30 2024-04-25T11:34:35.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.892+0000 7f775b42f740 1 -- start start 2024-04-25T11:34:35.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.892+0000 7f775b42f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56260f1b4270 0x56260f555370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.892+0000 7f775b42f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f18b3b0 0x56260f5558f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.892+0000 7f775b42f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56260f193590 0x56260f55c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.892+0000 7f775b42f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56260f4ba740 con 0x56260f18b3b0 2024-04-25T11:34:35.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.892+0000 7f775b42f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56260f4563a0 con 0x56260f193590 2024-04-25T11:34:35.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.892+0000 7f775b42f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56260f4b6700 con 0x56260f1b4270 2024-04-25T11:34:35.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.892+0000 7f7752319640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f18b3b0 0x56260f5558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.892+0000 7f7752319640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f18b3b0 0x56260f5558f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34940/0 (socket says 172.21.15.90:34940) 2024-04-25T11:34:35.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.892+0000 7f7752319640 1 -- 172.21.15.90:0/499410188 learned_addr learned my addr 172.21.15.90:0/499410188 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:35.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.893+0000 7f7751b18640 1 --2- 172.21.15.90:0/499410188 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56260f193590 0x56260f55c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.893+0000 7f7752b1a640 1 --2- 172.21.15.90:0/499410188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56260f1b4270 0x56260f555370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.893+0000 7f774b7fe640 1 --2- 172.21.15.90:0/4091782382 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f77340afb80 0x7f77340b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.893+0000 7f7752319640 1 -- 172.21.15.90:0/499410188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56260f1b4270 msgr2=0x56260f555370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:35.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.893+0000 7f7752319640 1 --2- 172.21.15.90:0/499410188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56260f1b4270 0x56260f555370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.893+0000 7f7752319640 1 -- 172.21.15.90:0/499410188 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56260f193590 msgr2=0x56260f55c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:35.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.893+0000 7f7752319640 1 --2- 172.21.15.90:0/499410188 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56260f193590 0x56260f55c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.893+0000 7f7752319640 1 -- 172.21.15.90:0/499410188 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56260f472750 con 0x56260f18b3b0 2024-04-25T11:34:35.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.893+0000 7f7752b1a640 1 --2- 172.21.15.90:0/499410188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56260f1b4270 0x56260f555370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:35.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.893+0000 7f7752b1a640 1 --2- 172.21.15.90:0/4091782382 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f77340afb80 0x7f77340b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.893+0000 7f774b7fe640 1 -- 172.21.15.90:0/4091782382 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f77400dec20 con 0x7f774c104d30 2024-04-25T11:34:35.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.893+0000 7f7752319640 1 --2- 172.21.15.90:0/499410188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f18b3b0 0x56260f5558f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7744007e70 tx=0x7f774400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a113ff25cb7fd2d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:35.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.893+0000 7f7748bfe640 1 -- 172.21.15.90:0/499410188 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7744014070 con 0x56260f18b3b0 2024-04-25T11:34:35.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.893+0000 7f7748bfe640 1 -- 172.21.15.90:0/499410188 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7744003e20 con 0x56260f18b3b0 2024-04-25T11:34:35.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.893+0000 7f7748bfe640 1 -- 172.21.15.90:0/499410188 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7744004740 con 0x56260f18b3b0 2024-04-25T11:34:35.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.894+0000 7f775b42f740 1 -- 172.21.15.90:0/499410188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f18b3b0 msgr2=0x56260f5558f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.894+0000 7f7752b1a640 1 --2- 172.21.15.90:0/4091782382 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f77340afb80 0x7f77340b2090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x56260f556540 tx=0x7f773c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:35.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.894+0000 7f775b42f740 1 --2- 172.21.15.90:0/499410188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f18b3b0 0x56260f5558f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7744007e70 tx=0x7f774400e720 comp rx=0 tx=0).stop 2024-04-25T11:34:35.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.894+0000 7f775b42f740 1 -- 172.21.15.90:0/499410188 shutdown_connections 2024-04-25T11:34:35.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.894+0000 7f775b42f740 1 --2- 172.21.15.90:0/499410188 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56260f193590 0x56260f55c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.894+0000 7f775b42f740 1 --2- 172.21.15.90:0/499410188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f18b3b0 0x56260f5558f0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.894+0000 7f775b42f740 1 --2- 172.21.15.90:0/499410188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56260f1b4270 0x56260f555370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.894+0000 7f775b42f740 1 -- 172.21.15.90:0/499410188 >> 172.21.15.90:0/499410188 conn(0x56260f1a83c0 msgr2=0x56260f550d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:35.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.894+0000 7f775b42f740 1 -- 172.21.15.90:0/499410188 shutdown_connections 2024-04-25T11:34:35.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.894+0000 7f775b42f740 1 -- 172.21.15.90:0/499410188 wait complete. 2024-04-25T11:34:35.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.894+0000 7f775b42f740 1 -- start start 2024-04-25T11:34:35.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.895+0000 7f775b42f740 1 Processor -- start 2024-04-25T11:34:35.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.895+0000 7f775b42f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56260f18b3b0 0x56260f55f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.895+0000 7f775b42f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56260f193590 0x56260f55f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.895+0000 7f775b42f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f1b4270 0x56260f566720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.895+0000 7f775b42f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56260f55d240 con 0x56260f1b4270 2024-04-25T11:34:35.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.895+0000 7f775b42f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56260f441fe0 con 0x56260f193590 2024-04-25T11:34:35.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.895+0000 7f775b42f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56260f55d3c0 con 0x56260f18b3b0 2024-04-25T11:34:35.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.895+0000 7f775b42f740 10 client.0 _ll_register_callbacks cb 0x56260f458150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:34:35.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.895+0000 7f7752b1a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f1b4270 0x56260f566720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.895+0000 7f7752b1a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f1b4270 0x56260f566720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34956/0 (socket says 172.21.15.90:34956) 2024-04-25T11:34:35.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.895+0000 7f7752b1a640 1 -- 172.21.15.90:0/3124951407 learned_addr learned my addr 172.21.15.90:0/3124951407 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:35.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.895+0000 7f7752319640 1 --2- 172.21.15.90:0/3124951407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56260f18b3b0 0x56260f55f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.895+0000 7f7751b18640 1 --2- 172.21.15.90:0/3124951407 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56260f193590 0x56260f55f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.895+0000 7f7752b1a640 1 -- 172.21.15.90:0/3124951407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56260f18b3b0 msgr2=0x56260f55f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:35.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.895+0000 7f7752b1a640 1 --2- 172.21.15.90:0/3124951407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56260f18b3b0 0x56260f55f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.895+0000 7f7752b1a640 1 -- 172.21.15.90:0/3124951407 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56260f193590 msgr2=0x56260f55f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:35.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.895+0000 7f7752b1a640 1 --2- 172.21.15.90:0/3124951407 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56260f193590 0x56260f55f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.895+0000 7f7752b1a640 1 -- 172.21.15.90:0/3124951407 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56260f1cc2d0 con 0x56260f1b4270 2024-04-25T11:34:35.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.895+0000 7f7752319640 1 --2- 172.21.15.90:0/3124951407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56260f18b3b0 0x56260f55f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:35.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.895+0000 7f7751b18640 1 --2- 172.21.15.90:0/3124951407 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56260f193590 0x56260f55f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:35.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.895+0000 7f7752b1a640 1 --2- 172.21.15.90:0/3124951407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f1b4270 0x56260f566720 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f773c033000 tx=0x7f773c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9793cd7472de6833 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:35.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.895+0000 7f7731ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:34:35.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.895+0000 7f7731ffb640 1 -- 172.21.15.90:0/3124951407 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773c035040 con 0x56260f1b4270 2024-04-25T11:34:35.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.896+0000 7f7731ffb640 1 -- 172.21.15.90:0/3124951407 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f773c0035f0 con 0x56260f1b4270 2024-04-25T11:34:35.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.896+0000 7f775b42f740 1 -- 172.21.15.90:0/3124951407 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7744003e20 con 0x56260f1b4270 2024-04-25T11:34:35.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.896+0000 7f7731ffb640 1 -- 172.21.15.90:0/3124951407 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f773c058070 con 0x56260f1b4270 2024-04-25T11:34:35.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.896+0000 7f775b42f740 10 client.15084 fetch_fsmap learned FSMap version 25 2024-04-25T11:34:35.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.896+0000 7f775b42f740 1 -- 172.21.15.90:0/3124951407 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56260f2966b0 con 0x56260f1b4270 2024-04-25T11:34:35.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.896+0000 7f7731ffb640 1 -- 172.21.15.90:0/3124951407 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7744003e20 con 0x56260f1b4270 2024-04-25T11:34:35.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.896+0000 7f7731ffb640 1 -- 172.21.15.90:0/3124951407 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f773c048910 con 0x56260f1b4270 2024-04-25T11:34:35.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.896+0000 7f775b42f740 10 client.15084 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T11:34:35.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.896+0000 7f775b42f740 10 client.15084 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-25T11:34:35.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.896+0000 7f775b42f740 20 client.15084 populate_metadata read hostname 'smithi090' 2024-04-25T11:34:35.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.896+0000 7f775b42f740 1 --2- 172.21.15.90:0/3124951407 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x56260f172e40 0x56260f56d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.896+0000 7f775b42f740 4 client.15084 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:34:35.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.896+0000 7f775b42f740 1 -- 172.21.15.90:0/3124951407 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56260f4c8350 con 0x56260f172e40 2024-04-25T11:34:35.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.896+0000 7f7752319640 1 --2- 172.21.15.90:0/3124951407 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x56260f172e40 0x56260f56d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.897+0000 7f7752319640 1 --2- 172.21.15.90:0/3124951407 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x56260f172e40 0x56260f56d020 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9cdc84302f6573f7 server_cookie=a03626178527bad2 in_seq=0 out_seq=0 2024-04-25T11:34:35.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.897+0000 7f7731ffb640 10 client.15084 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:34:35.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.898+0000 7f7731ffb640 1 -- 172.21.15.90:0/3124951407 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56260f4c8350 con 0x56260f172e40 2024-04-25T11:34:35.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.898+0000 7f7731ffb640 10 client.15084 handle_command_reply: tid=0 2024-04-25T11:34:35.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.898+0000 7f775b42f740 1 client.15084 shutdown 2024-04-25T11:34:35.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.898+0000 7f775b42f740 1 -- 172.21.15.90:0/3124951407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f1b4270 msgr2=0x56260f566720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.898+0000 7f775b42f740 1 --2- 172.21.15.90:0/3124951407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f1b4270 0x56260f566720 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f773c033000 tx=0x7f773c02a410 comp rx=0 tx=0).stop 2024-04-25T11:34:35.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.898+0000 7f775b42f740 1 -- 172.21.15.90:0/3124951407 shutdown_connections 2024-04-25T11:34:35.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.898+0000 7f775b42f740 1 --2- 172.21.15.90:0/3124951407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f1b4270 0x56260f566720 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.898+0000 7f775b42f740 1 --2- 172.21.15.90:0/3124951407 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x56260f172e40 0x56260f56d020 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.898+0000 7f775b42f740 1 --2- 172.21.15.90:0/3124951407 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56260f193590 0x56260f55f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.898+0000 7f775b42f740 1 --2- 172.21.15.90:0/3124951407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56260f18b3b0 0x56260f55f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.898+0000 7f7731ffb640 0 client.15084 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:34:35.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.898+0000 7f775b42f740 1 -- 172.21.15.90:0/3124951407 >> 172.21.15.90:0/3124951407 conn(0x56260f1a83c0 msgr2=0x56260f54fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:35.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.898+0000 7f775b42f740 1 -- 172.21.15.90:0/3124951407 shutdown_connections 2024-04-25T11:34:35.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.898+0000 7f775b42f740 1 -- 172.21.15.90:0/3124951407 wait complete. 2024-04-25T11:34:35.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.898+0000 7f775b42f740 20 client.15084 trim_cache size 0 max 16384 2024-04-25T11:34:35.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.916+0000 7f775b42f740 1 Processor -- start 2024-04-25T11:34:35.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.916+0000 7f775b42f740 1 -- start start 2024-04-25T11:34:35.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.916+0000 7f775b42f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f18b3b0 0x56260f559690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.916+0000 7f775b42f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56260f193590 0x56260f559bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.916+0000 7f775b42f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56260f1b4270 0x56260f55d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.916+0000 7f775b42f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56260f56b360 con 0x56260f18b3b0 2024-04-25T11:34:35.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.916+0000 7f775b42f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56260f56ba30 con 0x56260f193590 2024-04-25T11:34:35.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.916+0000 7f775b42f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56260f56c100 con 0x56260f1b4270 2024-04-25T11:34:35.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.916+0000 7f7752319640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f18b3b0 0x56260f559690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.917+0000 7f7752319640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f18b3b0 0x56260f559690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34968/0 (socket says 172.21.15.90:34968) 2024-04-25T11:34:35.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.917+0000 7f7752319640 1 -- 172.21.15.90:0/191398290 learned_addr learned my addr 172.21.15.90:0/191398290 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:35.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.917+0000 7f7751b18640 1 --2- 172.21.15.90:0/191398290 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56260f193590 0x56260f559bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.917+0000 7f7752b1a640 1 --2- 172.21.15.90:0/191398290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56260f1b4270 0x56260f55d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.917+0000 7f7752319640 1 -- 172.21.15.90:0/191398290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56260f1b4270 msgr2=0x56260f55d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:35.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.917+0000 7f7752319640 1 --2- 172.21.15.90:0/191398290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56260f1b4270 0x56260f55d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.917+0000 7f7752319640 1 -- 172.21.15.90:0/191398290 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56260f193590 msgr2=0x56260f559bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:35.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.917+0000 7f7752319640 1 --2- 172.21.15.90:0/191398290 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56260f193590 0x56260f559bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.917+0000 7f7752319640 1 -- 172.21.15.90:0/191398290 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56260f55d560 con 0x56260f18b3b0 2024-04-25T11:34:35.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.917+0000 7f7752b1a640 1 --2- 172.21.15.90:0/191398290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56260f1b4270 0x56260f55d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:35.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.917+0000 7f7752319640 1 --2- 172.21.15.90:0/191398290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f18b3b0 0x56260f559690 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f774400ec20 tx=0x7f7744007300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d05296557ad09b64 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:35.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.917+0000 7f7733fff640 1 -- 172.21.15.90:0/191398290 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7744014070 con 0x56260f18b3b0 2024-04-25T11:34:35.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.917+0000 7f7733fff640 1 -- 172.21.15.90:0/191398290 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7744025e50 con 0x56260f18b3b0 2024-04-25T11:34:35.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.917+0000 7f7733fff640 1 -- 172.21.15.90:0/191398290 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77440108a0 con 0x56260f18b3b0 2024-04-25T11:34:35.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.918+0000 7f775b42f740 1 -- 172.21.15.90:0/191398290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f18b3b0 msgr2=0x56260f559690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.918+0000 7f775b42f740 1 --2- 172.21.15.90:0/191398290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f18b3b0 0x56260f559690 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f774400ec20 tx=0x7f7744007300 comp rx=0 tx=0).stop 2024-04-25T11:34:35.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.918+0000 7f775b42f740 1 -- 172.21.15.90:0/191398290 shutdown_connections 2024-04-25T11:34:35.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.918+0000 7f775b42f740 1 --2- 172.21.15.90:0/191398290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56260f1b4270 0x56260f55d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.918+0000 7f775b42f740 1 --2- 172.21.15.90:0/191398290 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56260f193590 0x56260f559bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.918+0000 7f775b42f740 1 --2- 172.21.15.90:0/191398290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f18b3b0 0x56260f559690 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.918+0000 7f775b42f740 1 -- 172.21.15.90:0/191398290 >> 172.21.15.90:0/191398290 conn(0x56260f1a83c0 msgr2=0x56260f54ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:35.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.918+0000 7f775b42f740 1 -- 172.21.15.90:0/191398290 shutdown_connections 2024-04-25T11:34:35.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.918+0000 7f775b42f740 1 -- 172.21.15.90:0/191398290 wait complete. 2024-04-25T11:34:35.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.918+0000 7f775b42f740 1 -- start start 2024-04-25T11:34:35.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.918+0000 7f775b42f740 1 Processor -- start 2024-04-25T11:34:35.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.918+0000 7f775b42f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f18b3b0 0x56260f561100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.918+0000 7f775b42f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56260f193590 0x56260f561640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.919+0000 7f775b42f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56260f1b4270 0x56260f564530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.919+0000 7f775b42f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56260f56cea0 con 0x56260f18b3b0 2024-04-25T11:34:35.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.919+0000 7f775b42f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56260f56c7d0 con 0x56260f193590 2024-04-25T11:34:35.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.919+0000 7f775b42f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56260f56d570 con 0x56260f1b4270 2024-04-25T11:34:35.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.919+0000 7f775b42f740 10 client.0 _ll_register_callbacks cb 0x7f773c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:34:35.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.919+0000 7f7752319640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f18b3b0 0x56260f561100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.919+0000 7f7752319640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f18b3b0 0x56260f561100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34972/0 (socket says 172.21.15.90:34972) 2024-04-25T11:34:35.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.919+0000 7f7752319640 1 -- 172.21.15.90:0/1680578850 learned_addr learned my addr 172.21.15.90:0/1680578850 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:35.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.919+0000 7f7751b18640 1 --2- 172.21.15.90:0/1680578850 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56260f193590 0x56260f561640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.919+0000 7f7752b1a640 1 --2- 172.21.15.90:0/1680578850 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56260f1b4270 0x56260f564530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.919+0000 7f7752319640 1 -- 172.21.15.90:0/1680578850 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56260f1b4270 msgr2=0x56260f564530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:35.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.919+0000 7f7752319640 1 --2- 172.21.15.90:0/1680578850 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56260f1b4270 0x56260f564530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.919+0000 7f7752319640 1 -- 172.21.15.90:0/1680578850 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56260f193590 msgr2=0x56260f561640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:35.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.919+0000 7f7752319640 1 --2- 172.21.15.90:0/1680578850 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56260f193590 0x56260f561640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.919+0000 7f7752319640 1 -- 172.21.15.90:0/1680578850 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56260f564af0 con 0x56260f18b3b0 2024-04-25T11:34:35.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.919+0000 7f7752319640 1 --2- 172.21.15.90:0/1680578850 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f18b3b0 0x56260f561100 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f774400e760 tx=0x7f7744003990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e4ac0862221f21c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:35.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.919+0000 7f7731ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:34:35.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.919+0000 7f7731ffb640 1 -- 172.21.15.90:0/1680578850 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7744033020 con 0x56260f18b3b0 2024-04-25T11:34:35.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.919+0000 7f7731ffb640 1 -- 172.21.15.90:0/1680578850 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f7744025440 con 0x56260f18b3b0 2024-04-25T11:34:35.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.919+0000 7f775b42f740 1 -- 172.21.15.90:0/1680578850 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7744025e50 con 0x56260f18b3b0 2024-04-25T11:34:35.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.919+0000 7f7731ffb640 1 -- 172.21.15.90:0/1680578850 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f774403e050 con 0x56260f18b3b0 2024-04-25T11:34:35.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.919+0000 7f775b42f740 10 client.15090 fetch_fsmap learned FSMap version 25 2024-04-25T11:34:35.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.919+0000 7f775b42f740 1 -- 172.21.15.90:0/1680578850 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56260f565ce0 con 0x56260f18b3b0 2024-04-25T11:34:35.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.920+0000 7f7731ffb640 1 -- 172.21.15.90:0/1680578850 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7744025e50 con 0x56260f18b3b0 2024-04-25T11:34:35.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.920+0000 7f7731ffb640 1 -- 172.21.15.90:0/1680578850 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f7744010740 con 0x56260f18b3b0 2024-04-25T11:34:35.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.920+0000 7f775b42f740 10 client.15090 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T11:34:35.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.920+0000 7f775b42f740 10 client.15090 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-25T11:34:35.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.920+0000 7f775b42f740 20 client.15090 populate_metadata read hostname 'smithi090' 2024-04-25T11:34:35.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.920+0000 7f775b42f740 1 --2- 172.21.15.90:0/1680578850 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x56260f148280 0x56260f5d8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:35.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.920+0000 7f775b42f740 4 client.15090 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T11:34:35.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.920+0000 7f775b42f740 1 -- 172.21.15.90:0/1680578850 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "status"}) -- 0x56260f4c21c0 con 0x56260f148280 2024-04-25T11:34:35.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.920+0000 7f7751b18640 1 --2- 172.21.15.90:0/1680578850 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x56260f148280 0x56260f5d8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:35.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.920+0000 7f7731ffb640 10 client.15090 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:34:35.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.920+0000 7f7751b18640 1 --2- 172.21.15.90:0/1680578850 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x56260f148280 0x56260f5d8c60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=231e3e6fa0b1c20d server_cookie=3f5b811f238735e5 in_seq=0 out_seq=0 2024-04-25T11:34:35.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.921+0000 7f7731ffb640 1 -- 172.21.15.90:0/1680578850 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x56260f4c21c0 con 0x56260f148280 2024-04-25T11:34:35.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.921+0000 7f7731ffb640 10 client.15090 handle_command_reply: tid=0 2024-04-25T11:34:35.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.921+0000 7f775b42f740 1 client.15090 shutdown 2024-04-25T11:34:35.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.921+0000 7f775b42f740 1 -- 172.21.15.90:0/1680578850 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f18b3b0 msgr2=0x56260f561100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.921+0000 7f775b42f740 1 --2- 172.21.15.90:0/1680578850 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f18b3b0 0x56260f561100 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f774400e760 tx=0x7f7744003990 comp rx=0 tx=0).stop 2024-04-25T11:34:35.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.921+0000 7f775b42f740 1 -- 172.21.15.90:0/1680578850 shutdown_connections 2024-04-25T11:34:35.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.921+0000 7f775b42f740 1 --2- 172.21.15.90:0/1680578850 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56260f1b4270 0x56260f564530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.921+0000 7f775b42f740 1 --2- 172.21.15.90:0/1680578850 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x56260f148280 0x56260f5d8c60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.921+0000 7f775b42f740 1 --2- 172.21.15.90:0/1680578850 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56260f193590 0x56260f561640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.921+0000 7f7731ffb640 0 client.15090 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:34:35.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.921+0000 7f775b42f740 1 --2- 172.21.15.90:0/1680578850 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56260f18b3b0 0x56260f561100 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.921+0000 7f775b42f740 1 -- 172.21.15.90:0/1680578850 >> 172.21.15.90:0/1680578850 conn(0x56260f1a83c0 msgr2=0x56260f55f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:35.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.921+0000 7f775b42f740 1 -- 172.21.15.90:0/1680578850 shutdown_connections 2024-04-25T11:34:35.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.921+0000 7f775b42f740 1 -- 172.21.15.90:0/1680578850 wait complete. 2024-04-25T11:34:35.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.921+0000 7f775b42f740 20 client.15090 trim_cache size 0 max 16384 2024-04-25T11:34:35.917 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:34:35.917 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "58bd4e7a-02f6-11ef-bc93-c7b262605968", 2024-04-25T11:34:35.917 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 3, 2024-04-25T11:34:35.917 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 24458, 2024-04-25T11:34:35.917 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-04-25T11:34:35.917 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-04-25T11:34:35.917 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-04-25T11:34:35.917 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 81.300041407999998, 2024-04-25T11:34:35.917 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 25, 2024-04-25T11:34:35.917 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 79, 2024-04-25T11:34:35.917 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 77, 2024-04-25T11:34:35.917 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 109.814586699 2024-04-25T11:34:35.918 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:34:35.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.921+0000 7f775457d640 1 -- 172.21.15.90:0/4091782382 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f77340afb80 msgr2=0x7f77340b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.921+0000 7f775457d640 1 --2- 172.21.15.90:0/4091782382 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f77340afb80 0x7f77340b2090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x56260f556540 tx=0x7f773c009260 comp rx=0 tx=0).stop 2024-04-25T11:34:35.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.921+0000 7f775457d640 1 -- 172.21.15.90:0/4091782382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f774c104d30 msgr2=0x7f774c19a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:35.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.921+0000 7f775457d640 1 --2- 172.21.15.90:0/4091782382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f774c104d30 0x7f774c19a6e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f774000c970 tx=0x7f774000ce30 comp rx=0 tx=0).stop 2024-04-25T11:34:35.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.921+0000 7f775457d640 1 -- 172.21.15.90:0/4091782382 shutdown_connections 2024-04-25T11:34:35.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.921+0000 7f775457d640 1 --2- 172.21.15.90:0/4091782382 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f774c109a30 0x7f774c1a1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.921+0000 7f775457d640 1 --2- 172.21.15.90:0/4091782382 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f77340afb80 0x7f77340b2090 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.921+0000 7f775457d640 1 --2- 172.21.15.90:0/4091782382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f774c104d30 0x7f774c19a6e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.921+0000 7f775457d640 1 --2- 172.21.15.90:0/4091782382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f774c103af0 0x7f774c19a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:35.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.921+0000 7f775457d640 1 -- 172.21.15.90:0/4091782382 >> 172.21.15.90:0/4091782382 conn(0x7f774c0fece0 msgr2=0x7f774c105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:35.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.922+0000 7f775457d640 1 -- 172.21.15.90:0/4091782382 shutdown_connections 2024-04-25T11:34:35.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:35.922+0000 7f775457d640 1 -- 172.21.15.90:0/4091782382 wait complete. 2024-04-25T11:34:35.921 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:35 smithi090 ceph-mon[29575]: pgmap v318: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-25T11:34:35.921 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:35 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3512303256' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:34:35.921 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:35 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2189695618' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:34:35.921 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:35 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3611597346' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:34:35.922 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-04-25T11:34:36.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.041+0000 7f0339a15640 1 -- 172.21.15.90:0/927815922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0334108920 msgr2=0x7f033410ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:36.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.041+0000 7f0339a15640 1 --2- 172.21.15.90:0/927815922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0334108920 0x7f033410ad40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f032c009900 tx=0x7f032c02a410 comp rx=0 tx=0).stop 2024-04-25T11:34:36.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.041+0000 7f0339a15640 1 -- 172.21.15.90:0/927815922 shutdown_connections 2024-04-25T11:34:36.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.041+0000 7f0339a15640 1 --2- 172.21.15.90:0/927815922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0334108920 0x7f033410ad40 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.041+0000 7f0339a15640 1 --2- 172.21.15.90:0/927815922 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0334103c20 0x7f0334104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.041+0000 7f0339a15640 1 --2- 172.21.15.90:0/927815922 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03341029e0 0x7f03341036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.041+0000 7f0339a15640 1 -- 172.21.15.90:0/927815922 >> 172.21.15.90:0/927815922 conn(0x7f03340fdbd0 msgr2=0x7f0334100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:36.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.041+0000 7f0339a15640 1 -- 172.21.15.90:0/927815922 shutdown_connections 2024-04-25T11:34:36.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.041+0000 7f0339a15640 1 -- 172.21.15.90:0/927815922 wait complete. 2024-04-25T11:34:36.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.041+0000 7f0339a15640 1 Processor -- start 2024-04-25T11:34:36.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.042+0000 7f0339a15640 1 -- start start 2024-04-25T11:34:36.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.042+0000 7f0339a15640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f03341029e0 0x7f03341982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.042+0000 7f0339a15640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0334103c20 0x7f03341987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.042+0000 7f0339a15640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0334108920 0x7f033419f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.042+0000 7f0339a15640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f033410d400 con 0x7f0334103c20 2024-04-25T11:34:36.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.042+0000 7f0339a15640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f033410d280 con 0x7f03341029e0 2024-04-25T11:34:36.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.042+0000 7f0339a15640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f033410d580 con 0x7f0334108920 2024-04-25T11:34:36.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.042+0000 7f03327fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0334103c20 0x7f03341987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.042+0000 7f03327fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0334103c20 0x7f03341987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34982/0 (socket says 172.21.15.90:34982) 2024-04-25T11:34:36.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.042+0000 7f03327fc640 1 -- 172.21.15.90:0/4133662274 learned_addr learned my addr 172.21.15.90:0/4133662274 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:36.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.042+0000 7f03327fc640 1 -- 172.21.15.90:0/4133662274 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0334108920 msgr2=0x7f033419f780 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:34:36.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.042+0000 7f03327fc640 1 --2- 172.21.15.90:0/4133662274 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0334108920 0x7f033419f780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.042+0000 7f03327fc640 1 -- 172.21.15.90:0/4133662274 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f03341029e0 msgr2=0x7f03341982b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:34:36.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.042+0000 7f03327fc640 1 --2- 172.21.15.90:0/4133662274 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f03341029e0 0x7f03341982b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.042+0000 7f03327fc640 1 -- 172.21.15.90:0/4133662274 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f033419fdf0 con 0x7f0334103c20 2024-04-25T11:34:36.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.042+0000 7f03327fc640 1 --2- 172.21.15.90:0/4133662274 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0334103c20 0x7f03341987f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f032800e9f0 tx=0x7f032800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:36.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.042+0000 7f0338a13640 1 -- 172.21.15.90:0/4133662274 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f032800cd80 con 0x7f0334103c20 2024-04-25T11:34:36.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.042+0000 7f0339a15640 1 -- 172.21.15.90:0/4133662274 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03341a00e0 con 0x7f0334103c20 2024-04-25T11:34:36.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.042+0000 7f0338a13640 1 -- 172.21.15.90:0/4133662274 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03280108e0 con 0x7f0334103c20 2024-04-25T11:34:36.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.042+0000 7f0338a13640 1 -- 172.21.15.90:0/4133662274 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0328018980 con 0x7f0334103c20 2024-04-25T11:34:36.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.043+0000 7f0339a15640 1 -- 172.21.15.90:0/4133662274 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03341a0680 con 0x7f0334103c20 2024-04-25T11:34:36.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.045+0000 7f03409a6740 1 Processor -- start 2024-04-25T11:34:36.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.045+0000 7f03409a6740 1 -- start start 2024-04-25T11:34:36.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.045+0000 7f0338a13640 1 -- 172.21.15.90:0/4133662274 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f0328010430 con 0x7f0334103c20 2024-04-25T11:34:36.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.045+0000 7f03409a6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f74350d270 0x55f7438ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.045+0000 7f03409a6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7434e43b0 0x55f7438ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.045+0000 7f03409a6740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7434ec590 0x55f7438b57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.045+0000 7f03409a6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f743813740 con 0x55f7434e43b0 2024-04-25T11:34:36.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.045+0000 7f03409a6740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f7437af3a0 con 0x55f7434ec590 2024-04-25T11:34:36.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.045+0000 7f03409a6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f74380f700 con 0x55f74350d270 2024-04-25T11:34:36.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.045+0000 7f0332ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7434e43b0 0x55f7438ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.045+0000 7f0332ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7434e43b0 0x55f7438ae8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34986/0 (socket says 172.21.15.90:34986) 2024-04-25T11:34:36.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.045+0000 7f0332ffd640 1 -- 172.21.15.90:0/3267619704 learned_addr learned my addr 172.21.15.90:0/3267619704 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:36.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.046+0000 7f03337fe640 1 --2- 172.21.15.90:0/3267619704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f74350d270 0x55f7438ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.046+0000 7f03327fc640 1 --2- 172.21.15.90:0/3267619704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7434ec590 0x55f7438b57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.046+0000 7f0332ffd640 1 -- 172.21.15.90:0/3267619704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f74350d270 msgr2=0x55f7438ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:36.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.046+0000 7f0332ffd640 1 --2- 172.21.15.90:0/3267619704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f74350d270 0x55f7438ae370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.046+0000 7f0332ffd640 1 -- 172.21.15.90:0/3267619704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7434ec590 msgr2=0x55f7438b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:36.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.046+0000 7f0332ffd640 1 --2- 172.21.15.90:0/3267619704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7434ec590 0x55f7438b57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.046+0000 7f0332ffd640 1 -- 172.21.15.90:0/3267619704 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7437cb750 con 0x55f7434e43b0 2024-04-25T11:34:36.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.046+0000 7f03337fe640 1 --2- 172.21.15.90:0/3267619704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f74350d270 0x55f7438ae370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:36.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.046+0000 7f0338a13640 1 --2- 172.21.15.90:0/4133662274 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f031c0b1d90 0x7f031c0b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.046+0000 7f03337fe640 1 --2- 172.21.15.90:0/4133662274 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f031c0b1d90 0x7f031c0b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.046+0000 7f0332ffd640 1 --2- 172.21.15.90:0/3267619704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7434e43b0 0x55f7438ae8f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0324009e30 tx=0x7f032400d6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a371c2a476240a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:36.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.046+0000 7f0338a13640 1 -- 172.21.15.90:0/4133662274 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f03280df1a0 con 0x7f0334103c20 2024-04-25T11:34:36.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.046+0000 7f0319ffb640 1 -- 172.21.15.90:0/3267619704 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0324016070 con 0x55f7434e43b0 2024-04-25T11:34:36.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.046+0000 7f0319ffb640 1 -- 172.21.15.90:0/3267619704 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0324010b10 con 0x55f7434e43b0 2024-04-25T11:34:36.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.046+0000 7f0319ffb640 1 -- 172.21.15.90:0/3267619704 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f032400fac0 con 0x55f7434e43b0 2024-04-25T11:34:36.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.047+0000 7f03409a6740 1 -- 172.21.15.90:0/3267619704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7434e43b0 msgr2=0x55f7438ae8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:36.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.047+0000 7f03337fe640 1 --2- 172.21.15.90:0/4133662274 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f031c0b1d90 0x7f031c0b42a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f032c031440 tx=0x7f032c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:36.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.047+0000 7f03409a6740 1 --2- 172.21.15.90:0/3267619704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7434e43b0 0x55f7438ae8f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0324009e30 tx=0x7f032400d6b0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.047+0000 7f03409a6740 1 -- 172.21.15.90:0/3267619704 shutdown_connections 2024-04-25T11:34:36.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.047+0000 7f03409a6740 1 --2- 172.21.15.90:0/3267619704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7434ec590 0x55f7438b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.047+0000 7f03409a6740 1 --2- 172.21.15.90:0/3267619704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7434e43b0 0x55f7438ae8f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.047+0000 7f03409a6740 1 --2- 172.21.15.90:0/3267619704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f74350d270 0x55f7438ae370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.047+0000 7f03409a6740 1 -- 172.21.15.90:0/3267619704 >> 172.21.15.90:0/3267619704 conn(0x55f7435013c0 msgr2=0x55f7438a9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:36.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.047+0000 7f03409a6740 1 -- 172.21.15.90:0/3267619704 shutdown_connections 2024-04-25T11:34:36.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.047+0000 7f03409a6740 1 -- 172.21.15.90:0/3267619704 wait complete. 2024-04-25T11:34:36.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.047+0000 7f03409a6740 1 -- start start 2024-04-25T11:34:36.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.048+0000 7f03409a6740 1 Processor -- start 2024-04-25T11:34:36.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.048+0000 7f03409a6740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7434e43b0 0x55f7438b8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.048+0000 7f03409a6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7434ec590 0x55f7438b8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.048+0000 7f03409a6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f74350d270 0x55f7438bf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.048+0000 7f03409a6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f7438b6240 con 0x55f74350d270 2024-04-25T11:34:36.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.048+0000 7f03409a6740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f74379afe0 con 0x55f7434e43b0 2024-04-25T11:34:36.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.048+0000 7f03409a6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f7438b63c0 con 0x55f7434ec590 2024-04-25T11:34:36.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.048+0000 7f03409a6740 10 client.0 _ll_register_callbacks cb 0x55f7437b1150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:34:36.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.048+0000 7f03337fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f74350d270 0x55f7438bf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.048+0000 7f03337fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f74350d270 0x55f7438bf6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34996/0 (socket says 172.21.15.90:34996) 2024-04-25T11:34:36.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.048+0000 7f03337fe640 1 -- 172.21.15.90:0/1336038822 learned_addr learned my addr 172.21.15.90:0/1336038822 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:36.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.048+0000 7f03327fc640 1 --2- 172.21.15.90:0/1336038822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7434ec590 0x55f7438b8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.048+0000 7f0332ffd640 1 --2- 172.21.15.90:0/1336038822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7434e43b0 0x55f7438b8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.048+0000 7f03337fe640 1 -- 172.21.15.90:0/1336038822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7434ec590 msgr2=0x55f7438b8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:36.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.048+0000 7f03337fe640 1 --2- 172.21.15.90:0/1336038822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7434ec590 0x55f7438b8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.048+0000 7f03337fe640 1 -- 172.21.15.90:0/1336038822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7434e43b0 msgr2=0x55f7438b8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:36.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.048+0000 7f03337fe640 1 --2- 172.21.15.90:0/1336038822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7434e43b0 0x55f7438b8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.048+0000 7f03337fe640 1 -- 172.21.15.90:0/1336038822 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7435252d0 con 0x55f74350d270 2024-04-25T11:34:36.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.049+0000 7f03337fe640 1 --2- 172.21.15.90:0/1336038822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f74350d270 0x55f7438bf6c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f032c033000 tx=0x7f032c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f4a4a624f8bb2c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:36.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.049+0000 7f0312ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:34:36.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.049+0000 7f0312ffd640 1 -- 172.21.15.90:0/1336038822 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f032c035040 con 0x55f74350d270 2024-04-25T11:34:36.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.049+0000 7f0312ffd640 1 -- 172.21.15.90:0/1336038822 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f032c005100 con 0x55f74350d270 2024-04-25T11:34:36.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.049+0000 7f03409a6740 1 -- 172.21.15.90:0/1336038822 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0324010b10 con 0x55f74350d270 2024-04-25T11:34:36.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.049+0000 7f0312ffd640 1 -- 172.21.15.90:0/1336038822 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f032c005300 con 0x55f74350d270 2024-04-25T11:34:36.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.049+0000 7f03409a6740 10 client.15102 fetch_fsmap learned FSMap version 25 2024-04-25T11:34:36.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.049+0000 7f03409a6740 1 -- 172.21.15.90:0/1336038822 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f7435ef6b0 con 0x55f74350d270 2024-04-25T11:34:36.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.049+0000 7f0312ffd640 1 -- 172.21.15.90:0/1336038822 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0324010b10 con 0x55f74350d270 2024-04-25T11:34:36.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.049+0000 7f0312ffd640 1 -- 172.21.15.90:0/1336038822 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f032c03a040 con 0x55f74350d270 2024-04-25T11:34:36.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.049+0000 7f03409a6740 10 client.15102 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T11:34:36.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.049+0000 7f03409a6740 10 client.15102 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-25T11:34:36.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.049+0000 7f03409a6740 20 client.15102 populate_metadata read hostname 'smithi090' 2024-04-25T11:34:36.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.049+0000 7f03409a6740 1 --2- 172.21.15.90:0/1336038822 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f7434cbe40 0x55f7438c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.049+0000 7f03409a6740 4 client.15102 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:34:36.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.049+0000 7f03409a6740 1 -- 172.21.15.90:0/1336038822 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f743821350 con 0x55f7434cbe40 2024-04-25T11:34:36.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.050+0000 7f0332ffd640 1 --2- 172.21.15.90:0/1336038822 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f7434cbe40 0x55f7438c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.050+0000 7f0332ffd640 1 --2- 172.21.15.90:0/1336038822 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f7434cbe40 0x55f7438c5fb0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9342d620c4e72b23 server_cookie=faff8fb9546f4bb8 in_seq=0 out_seq=0 2024-04-25T11:34:36.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.050+0000 7f0312ffd640 10 client.15102 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:34:36.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.051+0000 7f0312ffd640 1 -- 172.21.15.90:0/1336038822 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f743821350 con 0x55f7434cbe40 2024-04-25T11:34:36.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.051+0000 7f0312ffd640 10 client.15102 handle_command_reply: tid=0 2024-04-25T11:34:36.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.051+0000 7f03409a6740 1 client.15102 shutdown 2024-04-25T11:34:36.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.051+0000 7f03409a6740 1 -- 172.21.15.90:0/1336038822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f74350d270 msgr2=0x55f7438bf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:36.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.051+0000 7f03409a6740 1 --2- 172.21.15.90:0/1336038822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f74350d270 0x55f7438bf6c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f032c033000 tx=0x7f032c009260 comp rx=0 tx=0).stop 2024-04-25T11:34:36.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.051+0000 7f03409a6740 1 -- 172.21.15.90:0/1336038822 shutdown_connections 2024-04-25T11:34:36.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.051+0000 7f03409a6740 1 --2- 172.21.15.90:0/1336038822 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f7434cbe40 0x55f7438c5fb0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.051+0000 7f03409a6740 1 --2- 172.21.15.90:0/1336038822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f74350d270 0x55f7438bf6c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.051+0000 7f03409a6740 1 --2- 172.21.15.90:0/1336038822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7434ec590 0x55f7438b8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.051+0000 7f0312ffd640 0 client.15102 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:34:36.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.051+0000 7f03409a6740 1 --2- 172.21.15.90:0/1336038822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7434e43b0 0x55f7438b8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.051+0000 7f03409a6740 1 -- 172.21.15.90:0/1336038822 >> 172.21.15.90:0/1336038822 conn(0x55f7435013c0 msgr2=0x55f7438a8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:36.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.051+0000 7f03409a6740 1 -- 172.21.15.90:0/1336038822 shutdown_connections 2024-04-25T11:34:36.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.051+0000 7f03409a6740 1 -- 172.21.15.90:0/1336038822 wait complete. 2024-04-25T11:34:36.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.051+0000 7f03409a6740 20 client.15102 trim_cache size 0 max 16384 2024-04-25T11:34:36.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.070+0000 7f03409a6740 1 Processor -- start 2024-04-25T11:34:36.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.070+0000 7f03409a6740 1 -- start start 2024-04-25T11:34:36.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.070+0000 7f03409a6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7434e43b0 0x55f7438b6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.070+0000 7f03409a6740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7434ec590 0x55f7438bf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.070+0000 7f03409a6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f74350d270 0x55f7438b1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.070+0000 7f03409a6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f7438c42f0 con 0x55f74350d270 2024-04-25T11:34:36.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.070+0000 7f03409a6740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f7438c49c0 con 0x55f7434ec590 2024-04-25T11:34:36.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.070+0000 7f03409a6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f7438c5090 con 0x55f7434e43b0 2024-04-25T11:34:36.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.070+0000 7f03337fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f74350d270 0x55f7438b1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.070+0000 7f03337fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f74350d270 0x55f7438b1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35002/0 (socket says 172.21.15.90:35002) 2024-04-25T11:34:36.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.070+0000 7f03337fe640 1 -- 172.21.15.90:0/2161075999 learned_addr learned my addr 172.21.15.90:0/2161075999 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:36.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.070+0000 7f03327fc640 1 --2- 172.21.15.90:0/2161075999 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7434ec590 0x55f7438bf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.070+0000 7f03337fe640 1 -- 172.21.15.90:0/2161075999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7434e43b0 msgr2=0x55f7438b6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:36.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.070+0000 7f03337fe640 1 --2- 172.21.15.90:0/2161075999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7434e43b0 0x55f7438b6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.070+0000 7f03337fe640 1 -- 172.21.15.90:0/2161075999 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7434ec590 msgr2=0x55f7438bf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:36.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.070+0000 7f03337fe640 1 --2- 172.21.15.90:0/2161075999 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7434ec590 0x55f7438bf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.070+0000 7f03337fe640 1 -- 172.21.15.90:0/2161075999 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7438b30c0 con 0x55f74350d270 2024-04-25T11:34:36.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.070+0000 7f03337fe640 1 --2- 172.21.15.90:0/2161075999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f74350d270 0x55f7438b1970 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f032c009230 tx=0x7f032c0319c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=453020ffa0cd95ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:36.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.070+0000 7f03197fa640 1 -- 172.21.15.90:0/2161075999 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f032c0054a0 con 0x55f74350d270 2024-04-25T11:34:36.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.070+0000 7f03197fa640 1 -- 172.21.15.90:0/2161075999 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f032c04d070 con 0x55f74350d270 2024-04-25T11:34:36.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.070+0000 7f03197fa640 1 -- 172.21.15.90:0/2161075999 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f032c004920 con 0x55f74350d270 2024-04-25T11:34:36.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.071+0000 7f03409a6740 1 -- 172.21.15.90:0/2161075999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f74350d270 msgr2=0x55f7438b1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:36.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.071+0000 7f03409a6740 1 --2- 172.21.15.90:0/2161075999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f74350d270 0x55f7438b1970 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f032c009230 tx=0x7f032c0319c0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.071+0000 7f03409a6740 1 -- 172.21.15.90:0/2161075999 shutdown_connections 2024-04-25T11:34:36.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.071+0000 7f03409a6740 1 --2- 172.21.15.90:0/2161075999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f74350d270 0x55f7438b1970 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.071+0000 7f03409a6740 1 --2- 172.21.15.90:0/2161075999 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7434ec590 0x55f7438bf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.071+0000 7f03409a6740 1 --2- 172.21.15.90:0/2161075999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7434e43b0 0x55f7438b6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.071+0000 7f03409a6740 1 -- 172.21.15.90:0/2161075999 >> 172.21.15.90:0/2161075999 conn(0x55f7435013c0 msgr2=0x55f7438b6760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:36.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.071+0000 7f03409a6740 1 -- 172.21.15.90:0/2161075999 shutdown_connections 2024-04-25T11:34:36.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.071+0000 7f03409a6740 1 -- 172.21.15.90:0/2161075999 wait complete. 2024-04-25T11:34:36.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.071+0000 7f03409a6740 1 -- start start 2024-04-25T11:34:36.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.072+0000 7f03409a6740 1 Processor -- start 2024-04-25T11:34:36.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.072+0000 7f03409a6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7434e43b0 0x55f7438bafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.072+0000 7f03409a6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7434ec590 0x55f7438bb520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.072+0000 7f03409a6740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f74350d270 0x55f7438be460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.072+0000 7f03409a6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f7438c5e30 con 0x55f7434ec590 2024-04-25T11:34:36.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.072+0000 7f03409a6740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f7438c5760 con 0x55f74350d270 2024-04-25T11:34:36.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.072+0000 7f03409a6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f7438c6500 con 0x55f7434e43b0 2024-04-25T11:34:36.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.072+0000 7f03409a6740 10 client.0 _ll_register_callbacks cb 0x7f0328005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:34:36.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.072+0000 7f03327fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7434ec590 0x55f7438bb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.072+0000 7f03327fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7434ec590 0x55f7438bb520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35014/0 (socket says 172.21.15.90:35014) 2024-04-25T11:34:36.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.072+0000 7f03327fc640 1 -- 172.21.15.90:0/2530312281 learned_addr learned my addr 172.21.15.90:0/2530312281 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:36.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.072+0000 7f0332ffd640 1 --2- 172.21.15.90:0/2530312281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7434e43b0 0x55f7438bafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.072+0000 7f03337fe640 1 --2- 172.21.15.90:0/2530312281 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f74350d270 0x55f7438be460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.072+0000 7f03327fc640 1 -- 172.21.15.90:0/2530312281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7434e43b0 msgr2=0x55f7438bafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:36.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.072+0000 7f03327fc640 1 --2- 172.21.15.90:0/2530312281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7434e43b0 0x55f7438bafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.072+0000 7f03327fc640 1 -- 172.21.15.90:0/2530312281 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f74350d270 msgr2=0x55f7438be460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:36.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.072+0000 7f03327fc640 1 --2- 172.21.15.90:0/2530312281 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f74350d270 0x55f7438be460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.072+0000 7f03327fc640 1 -- 172.21.15.90:0/2530312281 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7438bea20 con 0x55f7434ec590 2024-04-25T11:34:36.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.073+0000 7f03327fc640 1 --2- 172.21.15.90:0/2530312281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7434ec590 0x55f7438bb520 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f032800c6a0 tx=0x7f03280dcfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=552a3730b873651f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:36.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.073+0000 7f0312ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:34:36.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.073+0000 7f0312ffd640 1 -- 172.21.15.90:0/2530312281 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03280a7c00 con 0x55f7434ec590 2024-04-25T11:34:36.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.073+0000 7f0312ffd640 1 -- 172.21.15.90:0/2530312281 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f03280b00e0 con 0x55f7434ec590 2024-04-25T11:34:36.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.073+0000 7f03409a6740 1 -- 172.21.15.90:0/2530312281 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f032c04d070 con 0x55f7434ec590 2024-04-25T11:34:36.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.073+0000 7f0312ffd640 1 -- 172.21.15.90:0/2530312281 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f03280a7500 con 0x55f7434ec590 2024-04-25T11:34:36.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.073+0000 7f03409a6740 10 client.15108 fetch_fsmap learned FSMap version 25 2024-04-25T11:34:36.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.073+0000 7f03409a6740 1 -- 172.21.15.90:0/2530312281 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f74392f020 con 0x55f7434ec590 2024-04-25T11:34:36.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.073+0000 7f0312ffd640 1 -- 172.21.15.90:0/2530312281 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f032c04d070 con 0x55f7434ec590 2024-04-25T11:34:36.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.073+0000 7f0312ffd640 1 -- 172.21.15.90:0/2530312281 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f03280a2e70 con 0x55f7434ec590 2024-04-25T11:34:36.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.073+0000 7f03409a6740 10 client.15108 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T11:34:36.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.073+0000 7f03409a6740 10 client.15108 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-25T11:34:36.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.073+0000 7f03409a6740 20 client.15108 populate_metadata read hostname 'smithi090' 2024-04-25T11:34:36.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.073+0000 7f03409a6740 1 --2- 172.21.15.90:0/2530312281 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f7434a1280 0x55f7439328e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.073+0000 7f03409a6740 4 client.15108 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T11:34:36.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.073+0000 7f03409a6740 1 -- 172.21.15.90:0/2530312281 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "status"}) -- 0x55f74381b1c0 con 0x55f7434a1280 2024-04-25T11:34:36.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.074+0000 7f0332ffd640 1 --2- 172.21.15.90:0/2530312281 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f7434a1280 0x55f7439328e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.074+0000 7f0332ffd640 1 --2- 172.21.15.90:0/2530312281 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f7434a1280 0x55f7439328e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=19abe33160f25ccb server_cookie=d66605012e10a358 in_seq=0 out_seq=0 2024-04-25T11:34:36.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.074+0000 7f0312ffd640 10 client.15108 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:34:36.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.074+0000 7f0312ffd640 1 -- 172.21.15.90:0/2530312281 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55f74381b1c0 con 0x55f7434a1280 2024-04-25T11:34:36.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.074+0000 7f0312ffd640 10 client.15108 handle_command_reply: tid=0 2024-04-25T11:34:36.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.074+0000 7f03409a6740 1 client.15108 shutdown 2024-04-25T11:34:36.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.074+0000 7f03409a6740 1 -- 172.21.15.90:0/2530312281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7434ec590 msgr2=0x55f7438bb520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:36.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.074+0000 7f03409a6740 1 --2- 172.21.15.90:0/2530312281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7434ec590 0x55f7438bb520 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f032800c6a0 tx=0x7f03280dcfb0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.074+0000 7f03409a6740 1 -- 172.21.15.90:0/2530312281 shutdown_connections 2024-04-25T11:34:36.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.074+0000 7f03409a6740 1 --2- 172.21.15.90:0/2530312281 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f7434a1280 0x55f7439328e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.074+0000 7f03409a6740 1 --2- 172.21.15.90:0/2530312281 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f74350d270 0x55f7438be460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.075+0000 7f0312ffd640 0 client.15108 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:34:36.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.075+0000 7f03409a6740 1 --2- 172.21.15.90:0/2530312281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7434ec590 0x55f7438bb520 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.075+0000 7f03409a6740 1 --2- 172.21.15.90:0/2530312281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7434e43b0 0x55f7438bafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.075+0000 7f03409a6740 1 -- 172.21.15.90:0/2530312281 >> 172.21.15.90:0/2530312281 conn(0x55f7435013c0 msgr2=0x55f7438b6760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:36.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.075+0000 7f03409a6740 1 -- 172.21.15.90:0/2530312281 shutdown_connections 2024-04-25T11:34:36.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.075+0000 7f03409a6740 1 -- 172.21.15.90:0/2530312281 wait complete. 2024-04-25T11:34:36.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.075+0000 7f03409a6740 20 client.15108 trim_cache size 0 max 16384 2024-04-25T11:34:36.070 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:34:36.070 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "58bd4e7a-02f6-11ef-bc93-c7b262605968", 2024-04-25T11:34:36.070 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 2, 2024-04-25T11:34:36.071 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 24475, 2024-04-25T11:34:36.071 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-04-25T11:34:36.071 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-04-25T11:34:36.071 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-04-25T11:34:36.071 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 82.461106242, 2024-04-25T11:34:36.071 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 25, 2024-04-25T11:34:36.071 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 79, 2024-04-25T11:34:36.071 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 77, 2024-04-25T11:34:36.071 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 107.27939357699999 2024-04-25T11:34:36.071 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:34:36.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.075+0000 7f0339a15640 1 -- 172.21.15.90:0/4133662274 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f031c0b1d90 msgr2=0x7f031c0b42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:36.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.075+0000 7f0339a15640 1 --2- 172.21.15.90:0/4133662274 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f031c0b1d90 0x7f031c0b42a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f032c031440 tx=0x7f032c0314a0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.075+0000 7f0339a15640 1 -- 172.21.15.90:0/4133662274 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0334103c20 msgr2=0x7f03341987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:36.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.075+0000 7f0339a15640 1 --2- 172.21.15.90:0/4133662274 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0334103c20 0x7f03341987f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f032800e9f0 tx=0x7f032800eeb0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.075+0000 7f0339a15640 1 -- 172.21.15.90:0/4133662274 shutdown_connections 2024-04-25T11:34:36.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.075+0000 7f0339a15640 1 --2- 172.21.15.90:0/4133662274 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0334108920 0x7f033419f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.075+0000 7f0339a15640 1 --2- 172.21.15.90:0/4133662274 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f031c0b1d90 0x7f031c0b42a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.075+0000 7f0339a15640 1 --2- 172.21.15.90:0/4133662274 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0334103c20 0x7f03341987f0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.075+0000 7f0339a15640 1 --2- 172.21.15.90:0/4133662274 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f03341029e0 0x7f03341982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.075+0000 7f0339a15640 1 -- 172.21.15.90:0/4133662274 >> 172.21.15.90:0/4133662274 conn(0x7f03340fdbd0 msgr2=0x7f0334104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:36.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.075+0000 7f0339a15640 1 -- 172.21.15.90:0/4133662274 shutdown_connections 2024-04-25T11:34:36.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.075+0000 7f0339a15640 1 -- 172.21.15.90:0/4133662274 wait complete. 2024-04-25T11:34:36.076 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-25T11:34:36.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:35 smithi156 ceph-mon[33045]: pgmap v318: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-25T11:34:36.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:35 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3512303256' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:34:36.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:35 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2189695618' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:34:36.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:35 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3611597346' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:34:36.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.195+0000 7f608cbf3640 1 -- 172.21.15.90:0/2835023533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60881029f0 msgr2=0x7f60881036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:36.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.195+0000 7f608cbf3640 1 --2- 172.21.15.90:0/2835023533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60881029f0 0x7f60881036f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f607c009900 tx=0x7f607c02a410 comp rx=0 tx=0).stop 2024-04-25T11:34:36.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.195+0000 7f608cbf3640 1 -- 172.21.15.90:0/2835023533 shutdown_connections 2024-04-25T11:34:36.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.195+0000 7f608cbf3640 1 --2- 172.21.15.90:0/2835023533 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6088108930 0x7f608810ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.195+0000 7f608cbf3640 1 --2- 172.21.15.90:0/2835023533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6088103c30 0x7f6088104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.195+0000 7f608cbf3640 1 --2- 172.21.15.90:0/2835023533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60881029f0 0x7f60881036f0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.195+0000 7f608cbf3640 1 -- 172.21.15.90:0/2835023533 >> 172.21.15.90:0/2835023533 conn(0x7f60880fdbc0 msgr2=0x7f6088100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:36.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.195+0000 7f608cbf3640 1 -- 172.21.15.90:0/2835023533 shutdown_connections 2024-04-25T11:34:36.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.195+0000 7f608cbf3640 1 -- 172.21.15.90:0/2835023533 wait complete. 2024-04-25T11:34:36.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.196+0000 7f608cbf3640 1 Processor -- start 2024-04-25T11:34:36.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.196+0000 7f608cbf3640 1 -- start start 2024-04-25T11:34:36.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.196+0000 7f608cbf3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60881029f0 0x7f6088075860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.196+0000 7f608cbf3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6088103c30 0x7f608819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.196+0000 7f608cbf3640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6088108930 0x7f60881a15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.196+0000 7f608cbf3640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f608810e550 con 0x7f60881029f0 2024-04-25T11:34:36.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.196+0000 7f608cbf3640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f608810e3d0 con 0x7f6088108930 2024-04-25T11:34:36.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.196+0000 7f608cbf3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f608810e6d0 con 0x7f6088103c30 2024-04-25T11:34:36.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.196+0000 7f608659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60881029f0 0x7f6088075860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.196+0000 7f608659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60881029f0 0x7f6088075860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35036/0 (socket says 172.21.15.90:35036) 2024-04-25T11:34:36.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.196+0000 7f608659c640 1 -- 172.21.15.90:0/1608221689 learned_addr learned my addr 172.21.15.90:0/1608221689 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:36.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.196+0000 7f6086d9d640 1 --2- 172.21.15.90:0/1608221689 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6088108930 0x7f60881a15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.197+0000 7f608659c640 1 -- 172.21.15.90:0/1608221689 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6088103c30 msgr2=0x7f608819f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:36.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.197+0000 7f608659c640 1 --2- 172.21.15.90:0/1608221689 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6088103c30 0x7f608819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.197+0000 7f608659c640 1 -- 172.21.15.90:0/1608221689 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6088108930 msgr2=0x7f60881a15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:36.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.197+0000 7f608659c640 1 --2- 172.21.15.90:0/1608221689 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6088108930 0x7f60881a15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.197+0000 7f608659c640 1 -- 172.21.15.90:0/1608221689 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60881a1d40 con 0x7f60881029f0 2024-04-25T11:34:36.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.197+0000 7f6086d9d640 1 --2- 172.21.15.90:0/1608221689 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6088108930 0x7f60881a15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:36.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.197+0000 7f608659c640 1 --2- 172.21.15.90:0/1608221689 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60881029f0 0x7f6088075860 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f607c032fd0 tx=0x7f607c0042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:36.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.197+0000 7f60777fe640 1 -- 172.21.15.90:0/1608221689 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607c03f070 con 0x7f60881029f0 2024-04-25T11:34:36.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.197+0000 7f608cbf3640 1 -- 172.21.15.90:0/1608221689 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60881a1fd0 con 0x7f60881029f0 2024-04-25T11:34:36.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.197+0000 7f60777fe640 1 -- 172.21.15.90:0/1608221689 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f607c0044d0 con 0x7f60881029f0 2024-04-25T11:34:36.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.197+0000 7f60777fe640 1 -- 172.21.15.90:0/1608221689 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607c035070 con 0x7f60881029f0 2024-04-25T11:34:36.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.197+0000 7f608cbf3640 1 -- 172.21.15.90:0/1608221689 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60881a2510 con 0x7f60881029f0 2024-04-25T11:34:36.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.200+0000 7f60777fe640 1 -- 172.21.15.90:0/1608221689 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f607c03b5f0 con 0x7f60881029f0 2024-04-25T11:34:36.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.200+0000 7f6093b7b740 1 Processor -- start 2024-04-25T11:34:36.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.200+0000 7f6093b7b740 1 -- start start 2024-04-25T11:34:36.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.200+0000 7f6093b7b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560adabd7270 0x560adaf78370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.200+0000 7f6093b7b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabae3b0 0x560adaf788f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.200+0000 7f6093b7b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560adabb6590 0x560adaf7f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.200+0000 7f6093b7b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560adaedd740 con 0x560adabae3b0 2024-04-25T11:34:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.200+0000 7f6093b7b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x560adae793a0 con 0x560adabd7270 2024-04-25T11:34:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.200+0000 7f6093b7b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560adaed9700 con 0x560adabb6590 2024-04-25T11:34:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.200+0000 7f608659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabae3b0 0x560adaf788f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.200+0000 7f608659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabae3b0 0x560adaf788f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35044/0 (socket says 172.21.15.90:35044) 2024-04-25T11:34:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.200+0000 7f608659c640 1 -- 172.21.15.90:0/3491921962 learned_addr learned my addr 172.21.15.90:0/3491921962 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.200+0000 7f6086d9d640 1 --2- 172.21.15.90:0/3491921962 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560adabd7270 0x560adaf78370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.200+0000 7f608659c640 1 -- 172.21.15.90:0/3491921962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560adabb6590 msgr2=0x560adaf7f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.200+0000 7f6085d9b640 1 --2- 172.21.15.90:0/3491921962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560adabb6590 0x560adaf7f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.200+0000 7f608659c640 1 --2- 172.21.15.90:0/3491921962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560adabb6590 0x560adaf7f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.200+0000 7f608659c640 1 -- 172.21.15.90:0/3491921962 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560adabd7270 msgr2=0x560adaf78370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.200+0000 7f608659c640 1 --2- 172.21.15.90:0/3491921962 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560adabd7270 0x560adaf78370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.201+0000 7f608659c640 1 -- 172.21.15.90:0/3491921962 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560adae95750 con 0x560adabae3b0 2024-04-25T11:34:36.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.201+0000 7f6085d9b640 1 --2- 172.21.15.90:0/3491921962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560adabb6590 0x560adaf7f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.201+0000 7f60777fe640 1 --2- 172.21.15.90:0/1608221689 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f60700afb80 0x7f60700b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.201+0000 7f6086d9d640 1 --2- 172.21.15.90:0/1608221689 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f60700afb80 0x7f60700b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.201+0000 7f608659c640 1 --2- 172.21.15.90:0/3491921962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabae3b0 0x560adaf788f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f607c0fcd80 tx=0x7f607c105750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb0dd770a743b25 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.201+0000 7f60777fe640 1 -- 172.21.15.90:0/1608221689 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f607c0ff4b0 con 0x7f60881029f0 2024-04-25T11:34:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.201+0000 7f6074ff9640 1 -- 172.21.15.90:0/3491921962 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607c048020 con 0x560adabae3b0 2024-04-25T11:34:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.201+0000 7f6074ff9640 1 -- 172.21.15.90:0/3491921962 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f607c0c7320 con 0x560adabae3b0 2024-04-25T11:34:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.201+0000 7f6074ff9640 1 -- 172.21.15.90:0/3491921962 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607c0c36d0 con 0x560adabae3b0 2024-04-25T11:34:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.202+0000 7f6093b7b740 1 -- 172.21.15.90:0/3491921962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabae3b0 msgr2=0x560adaf788f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.202+0000 7f6086d9d640 1 --2- 172.21.15.90:0/1608221689 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f60700afb80 0x7f60700b2090 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x560adaf79540 tx=0x7f6080007720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.202+0000 7f6093b7b740 1 --2- 172.21.15.90:0/3491921962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabae3b0 0x560adaf788f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f607c0fcd80 tx=0x7f607c105750 comp rx=0 tx=0).stop 2024-04-25T11:34:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.202+0000 7f6093b7b740 1 -- 172.21.15.90:0/3491921962 shutdown_connections 2024-04-25T11:34:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.202+0000 7f6093b7b740 1 --2- 172.21.15.90:0/3491921962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560adabb6590 0x560adaf7f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.202+0000 7f6093b7b740 1 --2- 172.21.15.90:0/3491921962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabae3b0 0x560adaf788f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.202+0000 7f6093b7b740 1 --2- 172.21.15.90:0/3491921962 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560adabd7270 0x560adaf78370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.202+0000 7f6093b7b740 1 -- 172.21.15.90:0/3491921962 >> 172.21.15.90:0/3491921962 conn(0x560adabcb3c0 msgr2=0x560adaf73d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:36.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.202+0000 7f6093b7b740 1 -- 172.21.15.90:0/3491921962 shutdown_connections 2024-04-25T11:34:36.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.202+0000 7f6093b7b740 1 -- 172.21.15.90:0/3491921962 wait complete. 2024-04-25T11:34:36.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.202+0000 7f6093b7b740 1 -- start start 2024-04-25T11:34:36.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.202+0000 7f6093b7b740 1 Processor -- start 2024-04-25T11:34:36.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.203+0000 7f6093b7b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560adabae3b0 0x560adaf81e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.203+0000 7f6093b7b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560adabb6590 0x560adaf823b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.203+0000 7f6093b7b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabd7270 0x560adaf89310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.203+0000 7f6093b7b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560adaedd740 con 0x560adabd7270 2024-04-25T11:34:36.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.203+0000 7f6093b7b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x560adae64fe0 con 0x560adabae3b0 2024-04-25T11:34:36.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.203+0000 7f6093b7b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560adaf80000 con 0x560adabb6590 2024-04-25T11:34:36.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.203+0000 7f6093b7b740 10 client.0 _ll_register_callbacks cb 0x560adae7b150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:34:36.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.203+0000 7f6086d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabd7270 0x560adaf89310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.203+0000 7f6086d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabd7270 0x560adaf89310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35054/0 (socket says 172.21.15.90:35054) 2024-04-25T11:34:36.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.203+0000 7f6086d9d640 1 -- 172.21.15.90:0/2431123293 learned_addr learned my addr 172.21.15.90:0/2431123293 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:36.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.203+0000 7f608659c640 1 --2- 172.21.15.90:0/2431123293 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560adabae3b0 0x560adaf81e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.203+0000 7f6085d9b640 1 --2- 172.21.15.90:0/2431123293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560adabb6590 0x560adaf823b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.203+0000 7f6086d9d640 1 -- 172.21.15.90:0/2431123293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560adabb6590 msgr2=0x560adaf823b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:36.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.203+0000 7f6086d9d640 1 --2- 172.21.15.90:0/2431123293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560adabb6590 0x560adaf823b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.203+0000 7f6086d9d640 1 -- 172.21.15.90:0/2431123293 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560adabae3b0 msgr2=0x560adaf81e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:36.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.203+0000 7f6086d9d640 1 --2- 172.21.15.90:0/2431123293 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560adabae3b0 0x560adaf81e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.203+0000 7f6086d9d640 1 -- 172.21.15.90:0/2431123293 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560adabef2d0 con 0x560adabd7270 2024-04-25T11:34:36.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.203+0000 7f6085d9b640 1 --2- 172.21.15.90:0/2431123293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560adabb6590 0x560adaf823b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:36.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.203+0000 7f6086d9d640 1 --2- 172.21.15.90:0/2431123293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabd7270 0x560adaf89310 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f60800043d0 tx=0x7f6080004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e570917aed204d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:36.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.203+0000 7f606dffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:34:36.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.203+0000 7f606dffb640 1 -- 172.21.15.90:0/2431123293 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608000fe10 con 0x560adabd7270 2024-04-25T11:34:36.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.203+0000 7f606dffb640 1 -- 172.21.15.90:0/2431123293 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f608001c420 con 0x560adabd7270 2024-04-25T11:34:36.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.203+0000 7f6093b7b740 1 -- 172.21.15.90:0/2431123293 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f607c0c7320 con 0x560adabd7270 2024-04-25T11:34:36.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.204+0000 7f606dffb640 1 -- 172.21.15.90:0/2431123293 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f608002c070 con 0x560adabd7270 2024-04-25T11:34:36.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.204+0000 7f6093b7b740 10 client.15120 fetch_fsmap learned FSMap version 25 2024-04-25T11:34:36.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.204+0000 7f6093b7b740 1 -- 172.21.15.90:0/2431123293 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560adacb96b0 con 0x560adabd7270 2024-04-25T11:34:36.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.204+0000 7f606dffb640 1 -- 172.21.15.90:0/2431123293 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f607c0c7320 con 0x560adabd7270 2024-04-25T11:34:36.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.204+0000 7f606dffb640 1 -- 172.21.15.90:0/2431123293 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f6080026750 con 0x560adabd7270 2024-04-25T11:34:36.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.204+0000 7f6093b7b740 10 client.15120 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T11:34:36.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.204+0000 7f6093b7b740 10 client.15120 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T11:34:36.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.204+0000 7f6093b7b740 20 client.15120 populate_metadata read hostname 'smithi090' 2024-04-25T11:34:36.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.204+0000 7f6093b7b740 1 --2- 172.21.15.90:0/2431123293 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x560adab95e40 0x560adaf8fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.204+0000 7f6093b7b740 4 client.15120 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:34:36.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.204+0000 7f6093b7b740 1 -- 172.21.15.90:0/2431123293 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560adaeeb350 con 0x560adab95e40 2024-04-25T11:34:36.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.204+0000 7f608659c640 1 --2- 172.21.15.90:0/2431123293 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x560adab95e40 0x560adaf8fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.205+0000 7f606dffb640 10 client.15120 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:34:36.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.205+0000 7f608659c640 1 --2- 172.21.15.90:0/2431123293 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x560adab95e40 0x560adaf8fc70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6a9f79a66d5eaa6 server_cookie=51aed9e3960bc0a5 in_seq=0 out_seq=0 2024-04-25T11:34:36.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.206+0000 7f606dffb640 1 -- 172.21.15.90:0/2431123293 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f607c0a2e50 con 0x560adab95e40 2024-04-25T11:34:36.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.206+0000 7f606dffb640 10 client.15120 handle_command_reply: tid=0 2024-04-25T11:34:36.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.206+0000 7f6093b7b740 1 client.15120 shutdown 2024-04-25T11:34:36.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.206+0000 7f6093b7b740 1 -- 172.21.15.90:0/2431123293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabd7270 msgr2=0x560adaf89310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:36.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.206+0000 7f6093b7b740 1 --2- 172.21.15.90:0/2431123293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabd7270 0x560adaf89310 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f60800043d0 tx=0x7f6080004a40 comp rx=0 tx=0).stop 2024-04-25T11:34:36.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.206+0000 7f6093b7b740 1 -- 172.21.15.90:0/2431123293 shutdown_connections 2024-04-25T11:34:36.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.206+0000 7f6093b7b740 1 --2- 172.21.15.90:0/2431123293 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x560adab95e40 0x560adaf8fc70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.206+0000 7f6093b7b740 1 --2- 172.21.15.90:0/2431123293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabd7270 0x560adaf89310 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.206+0000 7f6093b7b740 1 --2- 172.21.15.90:0/2431123293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560adabb6590 0x560adaf823b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.206+0000 7f606dffb640 0 client.15120 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:34:36.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.206+0000 7f6093b7b740 1 --2- 172.21.15.90:0/2431123293 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560adabae3b0 0x560adaf81e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.206+0000 7f6093b7b740 1 -- 172.21.15.90:0/2431123293 >> 172.21.15.90:0/2431123293 conn(0x560adabcb3c0 msgr2=0x560adaf72d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:36.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.206+0000 7f6093b7b740 1 -- 172.21.15.90:0/2431123293 shutdown_connections 2024-04-25T11:34:36.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.206+0000 7f6093b7b740 1 -- 172.21.15.90:0/2431123293 wait complete. 2024-04-25T11:34:36.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.206+0000 7f6093b7b740 20 client.15120 trim_cache size 0 max 16384 2024-04-25T11:34:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.224+0000 7f6093b7b740 1 Processor -- start 2024-04-25T11:34:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.224+0000 7f6093b7b740 1 -- start start 2024-04-25T11:34:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.224+0000 7f6093b7b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560adabae3b0 0x560adaf6ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.224+0000 7f6093b7b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabb6590 0x560adaf89230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.224+0000 7f6093b7b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560adabd7270 0x560adaf7c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.224+0000 7f6093b7b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560adaf8dfb0 con 0x560adabb6590 2024-04-25T11:34:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.224+0000 7f6093b7b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x560adaf8e680 con 0x560adabae3b0 2024-04-25T11:34:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.224+0000 7f6093b7b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560adaf8ed50 con 0x560adabd7270 2024-04-25T11:34:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.225+0000 7f6085d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabb6590 0x560adaf89230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.225+0000 7f6085d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabb6590 0x560adaf89230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35070/0 (socket says 172.21.15.90:35070) 2024-04-25T11:34:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.225+0000 7f6085d9b640 1 -- 172.21.15.90:0/1651490820 learned_addr learned my addr 172.21.15.90:0/1651490820 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.225+0000 7f608659c640 1 --2- 172.21.15.90:0/1651490820 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560adabae3b0 0x560adaf6ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.225+0000 7f6086d9d640 1 --2- 172.21.15.90:0/1651490820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560adabd7270 0x560adaf7c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.225+0000 7f6085d9b640 1 -- 172.21.15.90:0/1651490820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560adabd7270 msgr2=0x560adaf7c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:36.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.225+0000 7f6085d9b640 1 --2- 172.21.15.90:0/1651490820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560adabd7270 0x560adaf7c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.225+0000 7f6085d9b640 1 -- 172.21.15.90:0/1651490820 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560adabae3b0 msgr2=0x560adaf6ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:36.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.225+0000 7f6085d9b640 1 --2- 172.21.15.90:0/1651490820 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560adabae3b0 0x560adaf6ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.225+0000 7f6085d9b640 1 -- 172.21.15.90:0/1651490820 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560adaf77f20 con 0x560adabb6590 2024-04-25T11:34:36.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.225+0000 7f6085d9b640 1 --2- 172.21.15.90:0/1651490820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabb6590 0x560adaf89230 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f607800b860 tx=0x7f607800bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8292e4b1697effa4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:36.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.225+0000 7f606ffff640 1 -- 172.21.15.90:0/1651490820 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6078016070 con 0x560adabb6590 2024-04-25T11:34:36.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.225+0000 7f606ffff640 1 -- 172.21.15.90:0/1651490820 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f607800dbb0 con 0x560adabb6590 2024-04-25T11:34:36.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.225+0000 7f606ffff640 1 -- 172.21.15.90:0/1651490820 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6078011620 con 0x560adabb6590 2024-04-25T11:34:36.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.226+0000 7f6093b7b740 1 -- 172.21.15.90:0/1651490820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabb6590 msgr2=0x560adaf89230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:36.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.226+0000 7f6093b7b740 1 --2- 172.21.15.90:0/1651490820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabb6590 0x560adaf89230 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f607800b860 tx=0x7f607800bd20 comp rx=0 tx=0).stop 2024-04-25T11:34:36.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.226+0000 7f6093b7b740 1 -- 172.21.15.90:0/1651490820 shutdown_connections 2024-04-25T11:34:36.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.226+0000 7f6093b7b740 1 --2- 172.21.15.90:0/1651490820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560adabd7270 0x560adaf7c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.226+0000 7f6093b7b740 1 --2- 172.21.15.90:0/1651490820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabb6590 0x560adaf89230 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.226+0000 7f6093b7b740 1 --2- 172.21.15.90:0/1651490820 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560adabae3b0 0x560adaf6ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.226+0000 7f6093b7b740 1 -- 172.21.15.90:0/1651490820 >> 172.21.15.90:0/1651490820 conn(0x560adabcb3c0 msgr2=0x560adaf8d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:36.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.226+0000 7f6093b7b740 1 -- 172.21.15.90:0/1651490820 shutdown_connections 2024-04-25T11:34:36.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.226+0000 7f6093b7b740 1 -- 172.21.15.90:0/1651490820 wait complete. 2024-04-25T11:34:36.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.226+0000 7f6093b7b740 1 -- start start 2024-04-25T11:34:36.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.226+0000 7f6093b7b740 1 Processor -- start 2024-04-25T11:34:36.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.226+0000 7f6093b7b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabae3b0 0x560adaf846f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.226+0000 7f6093b7b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560adabb6590 0x560adaf84c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.226+0000 7f6093b7b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560adabd7270 0x560adaf87b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.226+0000 7f6093b7b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560adaf8faf0 con 0x560adabae3b0 2024-04-25T11:34:36.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.226+0000 7f6093b7b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x560adaf8f420 con 0x560adabb6590 2024-04-25T11:34:36.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.226+0000 7f6093b7b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560adaf901c0 con 0x560adabd7270 2024-04-25T11:34:36.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.227+0000 7f6093b7b740 10 client.0 _ll_register_callbacks cb 0x7f6080011f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:34:36.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.227+0000 7f608659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabae3b0 0x560adaf846f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.227+0000 7f608659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabae3b0 0x560adaf846f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35082/0 (socket says 172.21.15.90:35082) 2024-04-25T11:34:36.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.227+0000 7f608659c640 1 -- 172.21.15.90:0/1748394849 learned_addr learned my addr 172.21.15.90:0/1748394849 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:36.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.227+0000 7f6086d9d640 1 --2- 172.21.15.90:0/1748394849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560adabd7270 0x560adaf87b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.227+0000 7f608659c640 1 -- 172.21.15.90:0/1748394849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560adabd7270 msgr2=0x560adaf87b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:36.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.227+0000 7f608659c640 1 --2- 172.21.15.90:0/1748394849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560adabd7270 0x560adaf87b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.227+0000 7f608659c640 1 -- 172.21.15.90:0/1748394849 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560adabb6590 msgr2=0x560adaf84c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:36.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.227+0000 7f608659c640 1 --2- 172.21.15.90:0/1748394849 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560adabb6590 0x560adaf84c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.227+0000 7f608659c640 1 -- 172.21.15.90:0/1748394849 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560adaf88130 con 0x560adabae3b0 2024-04-25T11:34:36.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.227+0000 7f6086d9d640 1 --2- 172.21.15.90:0/1748394849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560adabd7270 0x560adaf87b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:36.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.227+0000 7f608659c640 1 --2- 172.21.15.90:0/1748394849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabae3b0 0x560adaf846f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f607c0c4a80 tx=0x7f607c031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa853dbaf91d4528 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:36.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.227+0000 7f606dffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:34:36.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.227+0000 7f606dffb640 1 -- 172.21.15.90:0/1748394849 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607c048040 con 0x560adabae3b0 2024-04-25T11:34:36.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.227+0000 7f606dffb640 1 -- 172.21.15.90:0/1748394849 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f607c0ff0c0 con 0x560adabae3b0 2024-04-25T11:34:36.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.227+0000 7f6093b7b740 1 -- 172.21.15.90:0/1748394849 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f607800dbb0 con 0x560adabae3b0 2024-04-25T11:34:36.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.227+0000 7f606dffb640 1 -- 172.21.15.90:0/1748394849 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f607c0c6f80 con 0x560adabae3b0 2024-04-25T11:34:36.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.227+0000 7f6093b7b740 10 client.15126 fetch_fsmap learned FSMap version 25 2024-04-25T11:34:36.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.227+0000 7f6093b7b740 1 -- 172.21.15.90:0/1748394849 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560adaf893b0 con 0x560adabae3b0 2024-04-25T11:34:36.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.227+0000 7f606dffb640 1 -- 172.21.15.90:0/1748394849 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f607800dbb0 con 0x560adabae3b0 2024-04-25T11:34:36.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.228+0000 7f606dffb640 1 -- 172.21.15.90:0/1748394849 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f607c04b060 con 0x560adabae3b0 2024-04-25T11:34:36.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.228+0000 7f6093b7b740 10 client.15126 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T11:34:36.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.228+0000 7f6093b7b740 10 client.15126 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T11:34:36.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.228+0000 7f6093b7b740 20 client.15126 populate_metadata read hostname 'smithi090' 2024-04-25T11:34:36.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.228+0000 7f6093b7b740 1 --2- 172.21.15.90:0/1748394849 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x560adab6b280 0x560adaffbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.228+0000 7f6093b7b740 4 client.15126 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T11:34:36.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.228+0000 7f6093b7b740 1 -- 172.21.15.90:0/1748394849 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "status"}) -- 0x560adaee51c0 con 0x560adab6b280 2024-04-25T11:34:36.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.228+0000 7f6085d9b640 1 --2- 172.21.15.90:0/1748394849 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x560adab6b280 0x560adaffbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.228+0000 7f606dffb640 10 client.15126 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:34:36.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.228+0000 7f6085d9b640 1 --2- 172.21.15.90:0/1748394849 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x560adab6b280 0x560adaffbf50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8e326c12e7e353e5 server_cookie=863bf3e77b28dd3 in_seq=0 out_seq=0 2024-04-25T11:34:36.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.228+0000 7f606dffb640 1 -- 172.21.15.90:0/1748394849 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x560adaee51c0 con 0x560adab6b280 2024-04-25T11:34:36.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.228+0000 7f606dffb640 10 client.15126 handle_command_reply: tid=0 2024-04-25T11:34:36.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.229+0000 7f6093b7b740 1 client.15126 shutdown 2024-04-25T11:34:36.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.229+0000 7f6093b7b740 1 -- 172.21.15.90:0/1748394849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabae3b0 msgr2=0x560adaf846f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:36.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.229+0000 7f6093b7b740 1 --2- 172.21.15.90:0/1748394849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabae3b0 0x560adaf846f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f607c0c4a80 tx=0x7f607c031400 comp rx=0 tx=0).stop 2024-04-25T11:34:36.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.229+0000 7f6093b7b740 1 -- 172.21.15.90:0/1748394849 shutdown_connections 2024-04-25T11:34:36.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.229+0000 7f6093b7b740 1 --2- 172.21.15.90:0/1748394849 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x560adab6b280 0x560adaffbf50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.229+0000 7f6093b7b740 1 --2- 172.21.15.90:0/1748394849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560adabd7270 0x560adaf87b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.229+0000 7f6093b7b740 1 --2- 172.21.15.90:0/1748394849 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560adabb6590 0x560adaf84c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.229+0000 7f6093b7b740 1 --2- 172.21.15.90:0/1748394849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560adabae3b0 0x560adaf846f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.229+0000 7f606dffb640 0 client.15126 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:34:36.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.229+0000 7f6093b7b740 1 -- 172.21.15.90:0/1748394849 >> 172.21.15.90:0/1748394849 conn(0x560adabcb3c0 msgr2=0x560adaf7c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:36.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.229+0000 7f6093b7b740 1 -- 172.21.15.90:0/1748394849 shutdown_connections 2024-04-25T11:34:36.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.229+0000 7f6093b7b740 1 -- 172.21.15.90:0/1748394849 wait complete. 2024-04-25T11:34:36.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.229+0000 7f6093b7b740 20 client.15126 trim_cache size 0 max 16384 2024-04-25T11:34:36.225 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:34:36.225 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "58bd4e7a-02f6-11ef-bc93-c7b262605968", 2024-04-25T11:34:36.225 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 1, 2024-04-25T11:34:36.225 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 24499, 2024-04-25T11:34:36.225 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-04-25T11:34:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-04-25T11:34:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-04-25T11:34:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 83.196353594000001, 2024-04-25T11:34:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 25, 2024-04-25T11:34:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 79, 2024-04-25T11:34:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 77, 2024-04-25T11:34:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 99.183396510999998 2024-04-25T11:34:36.226 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:34:36.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.229+0000 7f608cbf3640 1 -- 172.21.15.90:0/1608221689 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f60700afb80 msgr2=0x7f60700b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:36.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.229+0000 7f608cbf3640 1 --2- 172.21.15.90:0/1608221689 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f60700afb80 0x7f60700b2090 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x560adaf79540 tx=0x7f6080007720 comp rx=0 tx=0).stop 2024-04-25T11:34:36.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.229+0000 7f608cbf3640 1 -- 172.21.15.90:0/1608221689 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60881029f0 msgr2=0x7f6088075860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:36.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.229+0000 7f608cbf3640 1 --2- 172.21.15.90:0/1608221689 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60881029f0 0x7f6088075860 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f607c032fd0 tx=0x7f607c0042e0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.229+0000 7f608cbf3640 1 -- 172.21.15.90:0/1608221689 shutdown_connections 2024-04-25T11:34:36.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.229+0000 7f608cbf3640 1 --2- 172.21.15.90:0/1608221689 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6088108930 0x7f60881a15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.229+0000 7f608cbf3640 1 --2- 172.21.15.90:0/1608221689 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6088103c30 0x7f608819f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.229+0000 7f608cbf3640 1 --2- 172.21.15.90:0/1608221689 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f60700afb80 0x7f60700b2090 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.229+0000 7f608cbf3640 1 --2- 172.21.15.90:0/1608221689 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60881029f0 0x7f6088075860 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.229+0000 7f608cbf3640 1 -- 172.21.15.90:0/1608221689 >> 172.21.15.90:0/1608221689 conn(0x7f60880fdbc0 msgr2=0x7f6088104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:36.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.229+0000 7f608cbf3640 1 -- 172.21.15.90:0/1608221689 shutdown_connections 2024-04-25T11:34:36.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.229+0000 7f608cbf3640 1 -- 172.21.15.90:0/1608221689 wait complete. 2024-04-25T11:34:36.230 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T11:34:36.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:35 smithi138 ceph-mon[32328]: pgmap v318: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-25T11:34:36.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:35 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3512303256' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:34:36.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:35 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2189695618' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:34:36.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:35 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3611597346' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:34:36.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.349+0000 7f012d251640 1 -- 172.21.15.90:0/4057350416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0128102fb0 msgr2=0x7f01281033c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:36.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.349+0000 7f012d251640 1 --2- 172.21.15.90:0/4057350416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0128102fb0 0x7f01281033c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f011c009900 tx=0x7f011c02a410 comp rx=0 tx=0).stop 2024-04-25T11:34:36.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.349+0000 7f012d251640 1 -- 172.21.15.90:0/4057350416 shutdown_connections 2024-04-25T11:34:36.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.349+0000 7f012d251640 1 --2- 172.21.15.90:0/4057350416 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0128104470 0x7f012810abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.349+0000 7f012d251640 1 --2- 172.21.15.90:0/4057350416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0128103990 0x7f0128103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.349+0000 7f012d251640 1 --2- 172.21.15.90:0/4057350416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0128102fb0 0x7f01281033c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.349+0000 7f012d251640 1 -- 172.21.15.90:0/4057350416 >> 172.21.15.90:0/4057350416 conn(0x7f01280fd350 msgr2=0x7f01280ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:36.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.349+0000 7f012d251640 1 -- 172.21.15.90:0/4057350416 shutdown_connections 2024-04-25T11:34:36.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.349+0000 7f012d251640 1 -- 172.21.15.90:0/4057350416 wait complete. 2024-04-25T11:34:36.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.350+0000 7f012d251640 1 Processor -- start 2024-04-25T11:34:36.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.350+0000 7f012d251640 1 -- start start 2024-04-25T11:34:36.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.350+0000 7f012d251640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0128102fb0 0x7f0128102250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.350+0000 7f012d251640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0128103990 0x7f012819f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.350+0000 7f012d251640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0128104470 0x7f01281a1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.350+0000 7f012d251640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f012810e590 con 0x7f0128104470 2024-04-25T11:34:36.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.350+0000 7f012d251640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f012810e410 con 0x7f0128103990 2024-04-25T11:34:36.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.350+0000 7f012d251640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f012810e710 con 0x7f0128102fb0 2024-04-25T11:34:36.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.350+0000 7f012759e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0128104470 0x7f01281a1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.350+0000 7f012759e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0128104470 0x7f01281a1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35108/0 (socket says 172.21.15.90:35108) 2024-04-25T11:34:36.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.350+0000 7f012759e640 1 -- 172.21.15.90:0/3910910801 learned_addr learned my addr 172.21.15.90:0/3910910801 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:36.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.350+0000 7f012659c640 1 --2- 172.21.15.90:0/3910910801 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0128103990 0x7f012819f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.350+0000 7f0126d9d640 1 --2- 172.21.15.90:0/3910910801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0128102fb0 0x7f0128102250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.350+0000 7f012759e640 1 -- 172.21.15.90:0/3910910801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0128102fb0 msgr2=0x7f0128102250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:36.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.350+0000 7f012759e640 1 --2- 172.21.15.90:0/3910910801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0128102fb0 0x7f0128102250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.350+0000 7f012759e640 1 -- 172.21.15.90:0/3910910801 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0128103990 msgr2=0x7f012819f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:36.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.350+0000 7f012759e640 1 --2- 172.21.15.90:0/3910910801 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0128103990 0x7f012819f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.350+0000 7f012759e640 1 -- 172.21.15.90:0/3910910801 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01281a1d80 con 0x7f0128104470 2024-04-25T11:34:36.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.350+0000 7f0126d9d640 1 --2- 172.21.15.90:0/3910910801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0128102fb0 0x7f0128102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:36.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.351+0000 7f012759e640 1 --2- 172.21.15.90:0/3910910801 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0128104470 0x7f01281a1560 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f012000b970 tx=0x7f012000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:36.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.351+0000 7f010ffff640 1 -- 172.21.15.90:0/3910910801 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0120019070 con 0x7f0128104470 2024-04-25T11:34:36.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.351+0000 7f012d251640 1 -- 172.21.15.90:0/3910910801 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01281a2070 con 0x7f0128104470 2024-04-25T11:34:36.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.351+0000 7f010ffff640 1 -- 172.21.15.90:0/3910910801 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0120014440 con 0x7f0128104470 2024-04-25T11:34:36.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.351+0000 7f010ffff640 1 -- 172.21.15.90:0/3910910801 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01200075d0 con 0x7f0128104470 2024-04-25T11:34:36.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.351+0000 7f012d251640 1 -- 172.21.15.90:0/3910910801 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01281a25f0 con 0x7f0128104470 2024-04-25T11:34:36.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.353+0000 7f010ffff640 1 -- 172.21.15.90:0/3910910801 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f012001c070 con 0x7f0128104470 2024-04-25T11:34:36.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.353+0000 7f01341e8740 1 Processor -- start 2024-04-25T11:34:36.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.353+0000 7f01341e8740 1 -- start start 2024-04-25T11:34:36.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.353+0000 7f01341e8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885ea270 0x55d88898f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.353+0000 7f01341e8740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8885c13b0 0x55d88898fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.353+0000 7f01341e8740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d8885c9590 0x55d888996c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.353+0000 7f01341e8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d8888f0740 con 0x55d8885ea270 2024-04-25T11:34:36.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.353+0000 7f01341e8740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55d88888c3a0 con 0x55d8885c9590 2024-04-25T11:34:36.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.353+0000 7f01341e8740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d8888ec700 con 0x55d8885c13b0 2024-04-25T11:34:36.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.354+0000 7f012659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885ea270 0x55d88898f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.354+0000 7f012659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885ea270 0x55d88898f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35118/0 (socket says 172.21.15.90:35118) 2024-04-25T11:34:36.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.354+0000 7f012659c640 1 -- 172.21.15.90:0/110351528 learned_addr learned my addr 172.21.15.90:0/110351528 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:36.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.354+0000 7f0126d9d640 1 --2- 172.21.15.90:0/110351528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8885c13b0 0x55d88898fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.354+0000 7f012659c640 1 -- 172.21.15.90:0/110351528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8885c13b0 msgr2=0x55d88898fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:36.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.354+0000 7f012659c640 1 --2- 172.21.15.90:0/110351528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8885c13b0 0x55d88898fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.354+0000 7f010ffff640 1 --2- 172.21.15.90:0/3910910801 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f01100afb80 0x7f01100b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.354+0000 7f012659c640 1 -- 172.21.15.90:0/110351528 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d8885c9590 msgr2=0x55d888996c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:36.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.354+0000 7f012659c640 1 --2- 172.21.15.90:0/110351528 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d8885c9590 0x55d888996c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.354+0000 7f012659c640 1 -- 172.21.15.90:0/110351528 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8888a8750 con 0x55d8885ea270 2024-04-25T11:34:36.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.354+0000 7f0126d9d640 1 --2- 172.21.15.90:0/110351528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8885c13b0 0x55d88898fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:36.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.354+0000 7f012759e640 1 --2- 172.21.15.90:0/3910910801 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f01100afb80 0x7f01100b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.354+0000 7f012659c640 1 --2- 172.21.15.90:0/110351528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885ea270 0x55d88898f780 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f011800b790 tx=0x7f011800bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a88751b8f5f20489 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:36.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.354+0000 7f010ffff640 1 -- 172.21.15.90:0/3910910801 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f01200de250 con 0x7f0128104470 2024-04-25T11:34:36.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.354+0000 7f010d7fa640 1 -- 172.21.15.90:0/110351528 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f011800bea0 con 0x55d8885ea270 2024-04-25T11:34:36.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.354+0000 7f010d7fa640 1 -- 172.21.15.90:0/110351528 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0118016030 con 0x55d8885ea270 2024-04-25T11:34:36.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.354+0000 7f010d7fa640 1 -- 172.21.15.90:0/110351528 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0118004dd0 con 0x55d8885ea270 2024-04-25T11:34:36.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.355+0000 7f01341e8740 1 -- 172.21.15.90:0/110351528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885ea270 msgr2=0x55d88898f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:36.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.355+0000 7f012759e640 1 --2- 172.21.15.90:0/3910910801 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f01100afb80 0x7f01100b2090 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f01200a2070 tx=0x7f01200752c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:36.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.355+0000 7f01341e8740 1 --2- 172.21.15.90:0/110351528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885ea270 0x55d88898f780 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f011800b790 tx=0x7f011800bc50 comp rx=0 tx=0).stop 2024-04-25T11:34:36.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.355+0000 7f01341e8740 1 -- 172.21.15.90:0/110351528 shutdown_connections 2024-04-25T11:34:36.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.355+0000 7f01341e8740 1 --2- 172.21.15.90:0/110351528 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d8885c9590 0x55d888996c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.355+0000 7f01341e8740 1 --2- 172.21.15.90:0/110351528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8885c13b0 0x55d88898fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.355+0000 7f01341e8740 1 --2- 172.21.15.90:0/110351528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885ea270 0x55d88898f780 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.355+0000 7f01341e8740 1 -- 172.21.15.90:0/110351528 >> 172.21.15.90:0/110351528 conn(0x55d8885de3c0 msgr2=0x55d888986d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:36.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.355+0000 7f01341e8740 1 -- 172.21.15.90:0/110351528 shutdown_connections 2024-04-25T11:34:36.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.355+0000 7f01341e8740 1 -- 172.21.15.90:0/110351528 wait complete. 2024-04-25T11:34:36.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.356+0000 7f01341e8740 1 -- start start 2024-04-25T11:34:36.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.356+0000 7f01341e8740 1 Processor -- start 2024-04-25T11:34:36.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.356+0000 7f01341e8740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d8885c13b0 0x55d888999610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.356+0000 7f01341e8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885c9590 0x55d888999b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.356+0000 7f01341e8740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8885ea270 0x55d8889a0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.356+0000 7f01341e8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d8889975f0 con 0x55d8885c9590 2024-04-25T11:34:36.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.356+0000 7f01341e8740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55d888877fe0 con 0x55d8885c13b0 2024-04-25T11:34:36.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.356+0000 7f01341e8740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d888997770 con 0x55d8885ea270 2024-04-25T11:34:36.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.356+0000 7f01341e8740 10 client.0 _ll_register_callbacks cb 0x55d88888e150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:34:36.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.356+0000 7f0126d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885c9590 0x55d888999b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.356+0000 7f0126d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885c9590 0x55d888999b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35132/0 (socket says 172.21.15.90:35132) 2024-04-25T11:34:36.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.356+0000 7f0126d9d640 1 -- 172.21.15.90:0/2295029375 learned_addr learned my addr 172.21.15.90:0/2295029375 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:36.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.356+0000 7f012659c640 1 --2- 172.21.15.90:0/2295029375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8885ea270 0x55d8889a0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.356+0000 7f012659c640 1 --2- 172.21.15.90:0/2295029375 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d8885c13b0 0x55d888999610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.356+0000 7f0126d9d640 1 -- 172.21.15.90:0/2295029375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8885ea270 msgr2=0x55d8889a0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:36.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.356+0000 7f0126d9d640 1 --2- 172.21.15.90:0/2295029375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8885ea270 0x55d8889a0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.356+0000 7f0126d9d640 1 -- 172.21.15.90:0/2295029375 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d8885c13b0 msgr2=0x55d888999610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:36.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.356+0000 7f0126d9d640 1 --2- 172.21.15.90:0/2295029375 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d8885c13b0 0x55d888999610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.356+0000 7f0126d9d640 1 -- 172.21.15.90:0/2295029375 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8886022d0 con 0x55d8885c9590 2024-04-25T11:34:36.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.356+0000 7f0126d9d640 1 --2- 172.21.15.90:0/2295029375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885c9590 0x55d888999b50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f012806b4a0 tx=0x7f011c0317d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67216338656e49dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:36.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.356+0000 7f010aff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:34:36.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.357+0000 7f010aff5640 1 -- 172.21.15.90:0/2295029375 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f011c03f070 con 0x55d8885c9590 2024-04-25T11:34:36.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.357+0000 7f010aff5640 1 -- 172.21.15.90:0/2295029375 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f011c004450 con 0x55d8885c9590 2024-04-25T11:34:36.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.357+0000 7f01341e8740 1 -- 172.21.15.90:0/2295029375 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0118016030 con 0x55d8885c9590 2024-04-25T11:34:36.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.357+0000 7f010aff5640 1 -- 172.21.15.90:0/2295029375 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f011c049070 con 0x55d8885c9590 2024-04-25T11:34:36.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.357+0000 7f01341e8740 10 client.15138 fetch_fsmap learned FSMap version 25 2024-04-25T11:34:36.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.357+0000 7f01341e8740 1 -- 172.21.15.90:0/2295029375 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d8886cc6b0 con 0x55d8885c9590 2024-04-25T11:34:36.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.357+0000 7f010aff5640 1 -- 172.21.15.90:0/2295029375 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0118016030 con 0x55d8885c9590 2024-04-25T11:34:36.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.357+0000 7f010aff5640 1 -- 172.21.15.90:0/2295029375 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f011c004bb0 con 0x55d8885c9590 2024-04-25T11:34:36.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.357+0000 7f01341e8740 10 client.15138 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T11:34:36.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.357+0000 7f01341e8740 10 client.15138 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T11:34:36.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.357+0000 7f01341e8740 20 client.15138 populate_metadata read hostname 'smithi090' 2024-04-25T11:34:36.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.357+0000 7f01341e8740 1 --2- 172.21.15.90:0/2295029375 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d8885a8e40 0x55d8889a7360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.357+0000 7f01341e8740 4 client.15138 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:34:36.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.357+0000 7f01341e8740 1 -- 172.21.15.90:0/2295029375 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8888fe350 con 0x55d8885a8e40 2024-04-25T11:34:36.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.358+0000 7f012659c640 1 --2- 172.21.15.90:0/2295029375 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d8885a8e40 0x55d8889a7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.358+0000 7f010aff5640 10 client.15138 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:34:36.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.358+0000 7f012659c640 1 --2- 172.21.15.90:0/2295029375 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d8885a8e40 0x55d8889a7360 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20a76dee1da73597 server_cookie=6c898caf42c1ef29 in_seq=0 out_seq=0 2024-04-25T11:34:36.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.359+0000 7f010aff5640 1 -- 172.21.15.90:0/2295029375 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d8888fe350 con 0x55d8885a8e40 2024-04-25T11:34:36.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.359+0000 7f010aff5640 10 client.15138 handle_command_reply: tid=0 2024-04-25T11:34:36.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.359+0000 7f01341e8740 1 client.15138 shutdown 2024-04-25T11:34:36.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.359+0000 7f01341e8740 1 -- 172.21.15.90:0/2295029375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885c9590 msgr2=0x55d888999b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:36.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.359+0000 7f01341e8740 1 --2- 172.21.15.90:0/2295029375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885c9590 0x55d888999b50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f012806b4a0 tx=0x7f011c0317d0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.359+0000 7f01341e8740 1 -- 172.21.15.90:0/2295029375 shutdown_connections 2024-04-25T11:34:36.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.359+0000 7f01341e8740 1 --2- 172.21.15.90:0/2295029375 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d8885a8e40 0x55d8889a7360 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.359+0000 7f01341e8740 1 --2- 172.21.15.90:0/2295029375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8885ea270 0x55d8889a0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.359+0000 7f01341e8740 1 --2- 172.21.15.90:0/2295029375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885c9590 0x55d888999b50 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.359+0000 7f01341e8740 1 --2- 172.21.15.90:0/2295029375 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d8885c13b0 0x55d888999610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.359+0000 7f010aff5640 0 client.15138 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:34:36.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.359+0000 7f01341e8740 1 -- 172.21.15.90:0/2295029375 >> 172.21.15.90:0/2295029375 conn(0x55d8885de3c0 msgr2=0x55d888985d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:36.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.359+0000 7f01341e8740 1 -- 172.21.15.90:0/2295029375 shutdown_connections 2024-04-25T11:34:36.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.359+0000 7f01341e8740 1 -- 172.21.15.90:0/2295029375 wait complete. 2024-04-25T11:34:36.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.359+0000 7f01341e8740 20 client.15138 trim_cache size 0 max 16384 2024-04-25T11:34:36.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.377+0000 7f01341e8740 1 Processor -- start 2024-04-25T11:34:36.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.377+0000 7f01341e8740 1 -- start start 2024-04-25T11:34:36.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.377+0000 7f01341e8740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8885c13b0 0x55d8889866b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.377+0000 7f01341e8740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d8885c9590 0x55d8889a09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.377+0000 7f01341e8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885ea270 0x55d888986bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.377+0000 7f01341e8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d8889a56a0 con 0x55d8885ea270 2024-04-25T11:34:36.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.377+0000 7f01341e8740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55d8889a5d70 con 0x55d8885c9590 2024-04-25T11:34:36.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.377+0000 7f01341e8740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d8889a6440 con 0x55d8885c13b0 2024-04-25T11:34:36.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.377+0000 7f012659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885ea270 0x55d888986bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.378+0000 7f012659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885ea270 0x55d888986bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35140/0 (socket says 172.21.15.90:35140) 2024-04-25T11:34:36.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.378+0000 7f012659c640 1 -- 172.21.15.90:0/3887608564 learned_addr learned my addr 172.21.15.90:0/3887608564 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:36.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.378+0000 7f012659c640 1 -- 172.21.15.90:0/3887608564 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8885c13b0 msgr2=0x55d8889866b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:36.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.378+0000 7f012659c640 1 --2- 172.21.15.90:0/3887608564 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8885c13b0 0x55d8889866b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.378+0000 7f012659c640 1 -- 172.21.15.90:0/3887608564 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d8885c9590 msgr2=0x55d8889a09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:36.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.378+0000 7f012659c640 1 --2- 172.21.15.90:0/3887608564 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d8885c9590 0x55d8889a09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.378+0000 7f012659c640 1 -- 172.21.15.90:0/3887608564 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d888992b00 con 0x55d8885ea270 2024-04-25T11:34:36.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.378+0000 7f012659c640 1 --2- 172.21.15.90:0/3887608564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885ea270 0x55d888986bf0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f01180039c0 tx=0x7f011800bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=952bc86906380de8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:36.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.378+0000 7f010cff9640 1 -- 172.21.15.90:0/3887608564 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0118031070 con 0x55d8885ea270 2024-04-25T11:34:36.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.378+0000 7f010cff9640 1 -- 172.21.15.90:0/3887608564 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0118020070 con 0x55d8885ea270 2024-04-25T11:34:36.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.378+0000 7f010cff9640 1 -- 172.21.15.90:0/3887608564 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f011801bb80 con 0x55d8885ea270 2024-04-25T11:34:36.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.379+0000 7f01341e8740 1 -- 172.21.15.90:0/3887608564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885ea270 msgr2=0x55d888986bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:36.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.379+0000 7f01341e8740 1 --2- 172.21.15.90:0/3887608564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885ea270 0x55d888986bf0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f01180039c0 tx=0x7f011800bc50 comp rx=0 tx=0).stop 2024-04-25T11:34:36.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.379+0000 7f01341e8740 1 -- 172.21.15.90:0/3887608564 shutdown_connections 2024-04-25T11:34:36.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.379+0000 7f01341e8740 1 --2- 172.21.15.90:0/3887608564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885ea270 0x55d888986bf0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.379+0000 7f01341e8740 1 --2- 172.21.15.90:0/3887608564 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d8885c9590 0x55d8889a09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.379+0000 7f01341e8740 1 --2- 172.21.15.90:0/3887608564 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8885c13b0 0x55d8889866b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.379+0000 7f01341e8740 1 -- 172.21.15.90:0/3887608564 >> 172.21.15.90:0/3887608564 conn(0x55d8885de3c0 msgr2=0x55d888997c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:36.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.379+0000 7f01341e8740 1 -- 172.21.15.90:0/3887608564 shutdown_connections 2024-04-25T11:34:36.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.379+0000 7f01341e8740 1 -- 172.21.15.90:0/3887608564 wait complete. 2024-04-25T11:34:36.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.379+0000 7f01341e8740 1 -- start start 2024-04-25T11:34:36.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.379+0000 7f01341e8740 1 Processor -- start 2024-04-25T11:34:36.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.379+0000 7f01341e8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885c13b0 0x55d88899c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.379+0000 7f01341e8740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d8885c9590 0x55d88899c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.379+0000 7f01341e8740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8885ea270 0x55d88899f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.379+0000 7f01341e8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d8889a71e0 con 0x55d8885c13b0 2024-04-25T11:34:36.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.379+0000 7f01341e8740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55d8889a6b10 con 0x55d8885c9590 2024-04-25T11:34:36.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.379+0000 7f01341e8740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d8889a78b0 con 0x55d8885ea270 2024-04-25T11:34:36.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.380+0000 7f01341e8740 10 client.0 _ll_register_callbacks cb 0x7f011c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:34:36.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.380+0000 7f012659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885c13b0 0x55d88899c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.380+0000 7f012659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885c13b0 0x55d88899c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35152/0 (socket says 172.21.15.90:35152) 2024-04-25T11:34:36.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.380+0000 7f012659c640 1 -- 172.21.15.90:0/505325712 learned_addr learned my addr 172.21.15.90:0/505325712 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:36.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.380+0000 7f012659c640 1 --2- 172.21.15.90:0/505325712 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8885ea270 0x55d88899f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.380+0000 7f012659c640 1 -- 172.21.15.90:0/505325712 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8885ea270 msgr2=0x55d88899f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:36.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.380+0000 7f012659c640 1 --2- 172.21.15.90:0/505325712 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8885ea270 0x55d88899f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.380+0000 7f012659c640 1 -- 172.21.15.90:0/505325712 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d8885c9590 msgr2=0x55d88899c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:36.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.380+0000 7f012659c640 1 --2- 172.21.15.90:0/505325712 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d8885c9590 0x55d88899c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.380+0000 7f012659c640 1 -- 172.21.15.90:0/505325712 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8889944b0 con 0x55d8885c13b0 2024-04-25T11:34:36.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.380+0000 7f012659c640 1 --2- 172.21.15.90:0/505325712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885c13b0 0x55d88899c3d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f011800b520 tx=0x7f01180041e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=accf82c523e95a65 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:36.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.380+0000 7f010aff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:34:36.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.380+0000 7f010aff5640 1 -- 172.21.15.90:0/505325712 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0118031040 con 0x55d8885c13b0 2024-04-25T11:34:36.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.380+0000 7f010aff5640 1 -- 172.21.15.90:0/505325712 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f011802b7f0 con 0x55d8885c13b0 2024-04-25T11:34:36.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.380+0000 7f01341e8740 1 -- 172.21.15.90:0/505325712 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0118020070 con 0x55d8885c13b0 2024-04-25T11:34:36.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.380+0000 7f010aff5640 1 -- 172.21.15.90:0/505325712 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f011802b9f0 con 0x55d8885c13b0 2024-04-25T11:34:36.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.380+0000 7f01341e8740 10 client.15144 fetch_fsmap learned FSMap version 25 2024-04-25T11:34:36.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.380+0000 7f01341e8740 1 -- 172.21.15.90:0/505325712 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d888a101e0 con 0x55d8885c13b0 2024-04-25T11:34:36.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.380+0000 7f010aff5640 1 -- 172.21.15.90:0/505325712 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0118020070 con 0x55d8885c13b0 2024-04-25T11:34:36.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.381+0000 7f010aff5640 1 -- 172.21.15.90:0/505325712 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f011801b470 con 0x55d8885c13b0 2024-04-25T11:34:36.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.381+0000 7f01341e8740 10 client.15144 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T11:34:36.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.381+0000 7f01341e8740 10 client.15144 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T11:34:36.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.381+0000 7f01341e8740 20 client.15144 populate_metadata read hostname 'smithi090' 2024-04-25T11:34:36.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.381+0000 7f01341e8740 1 --2- 172.21.15.90:0/505325712 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d88857e280 0x55d888a13270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:36.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.381+0000 7f01341e8740 4 client.15144 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T11:34:36.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.381+0000 7f01341e8740 1 -- 172.21.15.90:0/505325712 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "status"}) -- 0x55d8888f81c0 con 0x55d88857e280 2024-04-25T11:34:36.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.381+0000 7f0126d9d640 1 --2- 172.21.15.90:0/505325712 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d88857e280 0x55d888a13270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:36.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.381+0000 7f0126d9d640 1 --2- 172.21.15.90:0/505325712 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d88857e280 0x55d888a13270 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7712c86c07fef59a server_cookie=a004ed2e46cdcaa0 in_seq=0 out_seq=0 2024-04-25T11:34:36.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.381+0000 7f010aff5640 10 client.15144 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:34:36.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.382+0000 7f010aff5640 1 -- 172.21.15.90:0/505325712 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55d8888f81c0 con 0x55d88857e280 2024-04-25T11:34:36.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.382+0000 7f010aff5640 10 client.15144 handle_command_reply: tid=0 2024-04-25T11:34:36.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.382+0000 7f01341e8740 1 client.15144 shutdown 2024-04-25T11:34:36.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.382+0000 7f01341e8740 1 -- 172.21.15.90:0/505325712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885c13b0 msgr2=0x55d88899c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:36.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.382+0000 7f01341e8740 1 --2- 172.21.15.90:0/505325712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885c13b0 0x55d88899c3d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f011800b520 tx=0x7f01180041e0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.382+0000 7f01341e8740 1 -- 172.21.15.90:0/505325712 shutdown_connections 2024-04-25T11:34:36.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.382+0000 7f01341e8740 1 --2- 172.21.15.90:0/505325712 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d88857e280 0x55d888a13270 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.382+0000 7f01341e8740 1 --2- 172.21.15.90:0/505325712 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8885ea270 0x55d88899f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.382+0000 7f010aff5640 0 client.15144 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:34:36.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.382+0000 7f01341e8740 1 --2- 172.21.15.90:0/505325712 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d8885c9590 0x55d88899c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.382+0000 7f01341e8740 1 --2- 172.21.15.90:0/505325712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8885c13b0 0x55d88899c3d0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.382+0000 7f01341e8740 1 -- 172.21.15.90:0/505325712 >> 172.21.15.90:0/505325712 conn(0x55d8885de3c0 msgr2=0x55d8889866b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:36.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.382+0000 7f01341e8740 1 -- 172.21.15.90:0/505325712 shutdown_connections 2024-04-25T11:34:36.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.382+0000 7f01341e8740 1 -- 172.21.15.90:0/505325712 wait complete. 2024-04-25T11:34:36.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.382+0000 7f01341e8740 20 client.15144 trim_cache size 0 max 16384 2024-04-25T11:34:36.378 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:34:36.378 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "58bd4e7a-02f6-11ef-bc93-c7b262605968", 2024-04-25T11:34:36.378 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 0, 2024-04-25T11:34:36.378 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 24524, 2024-04-25T11:34:36.378 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-04-25T11:34:36.378 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-04-25T11:34:36.378 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-04-25T11:34:36.378 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 85.366617043999995, 2024-04-25T11:34:36.378 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 25, 2024-04-25T11:34:36.379 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 79, 2024-04-25T11:34:36.379 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 77, 2024-04-25T11:34:36.379 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 90.935477008999996 2024-04-25T11:34:36.379 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:34:36.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.382+0000 7f012d251640 1 -- 172.21.15.90:0/3910910801 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f01100afb80 msgr2=0x7f01100b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:36.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.382+0000 7f012d251640 1 --2- 172.21.15.90:0/3910910801 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f01100afb80 0x7f01100b2090 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f01200a2070 tx=0x7f01200752c0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.382+0000 7f012d251640 1 -- 172.21.15.90:0/3910910801 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0128104470 msgr2=0x7f01281a1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:36.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.382+0000 7f012d251640 1 --2- 172.21.15.90:0/3910910801 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0128104470 0x7f01281a1560 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f012000b970 tx=0x7f012000be30 comp rx=0 tx=0).stop 2024-04-25T11:34:36.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.382+0000 7f012d251640 1 -- 172.21.15.90:0/3910910801 shutdown_connections 2024-04-25T11:34:36.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.382+0000 7f012d251640 1 --2- 172.21.15.90:0/3910910801 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f01100afb80 0x7f01100b2090 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.382+0000 7f012d251640 1 --2- 172.21.15.90:0/3910910801 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0128104470 0x7f01281a1560 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.382+0000 7f012d251640 1 --2- 172.21.15.90:0/3910910801 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0128103990 0x7f012819f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.382+0000 7f012d251640 1 --2- 172.21.15.90:0/3910910801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0128102fb0 0x7f0128102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:36.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.383+0000 7f012d251640 1 -- 172.21.15.90:0/3910910801 >> 172.21.15.90:0/3910910801 conn(0x7f01280fd350 msgr2=0x7f0128106b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:36.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.383+0000 7f012d251640 1 -- 172.21.15.90:0/3910910801 shutdown_connections 2024-04-25T11:34:36.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:36.383+0000 7f012d251640 1 -- 172.21.15.90:0/3910910801 wait complete. 2024-04-25T11:34:36.385 INFO:tasks.cephadm:Setup complete, yielding 2024-04-25T11:34:36.385 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T11:34:36.395 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T11:34:36.396 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-25T11:34:36.630 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:34:37.904 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:37 smithi090 ceph-mon[29575]: pgmap v319: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-04-25T11:34:38.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:37 smithi156 ceph-mon[33045]: pgmap v319: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-04-25T11:34:38.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.143+0000 7f943bec5700 1 -- 172.21.15.90:0/3512306536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9434104240 msgr2=0x7f9434104650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:38.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.143+0000 7f943bec5700 1 --2- 172.21.15.90:0/3512306536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9434104240 0x7f9434104650 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9430009990 tx=0x7f9430009ca0 comp rx=0 tx=0).stop 2024-04-25T11:34:38.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.143+0000 7f943bec5700 1 -- 172.21.15.90:0/3512306536 shutdown_connections 2024-04-25T11:34:38.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.143+0000 7f943bec5700 1 --2- 172.21.15.90:0/3512306536 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9434104be0 0x7f943410b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:38.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.143+0000 7f943bec5700 1 --2- 172.21.15.90:0/3512306536 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9434103110 0x7f9434103520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:38.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.143+0000 7f943bec5700 1 --2- 172.21.15.90:0/3512306536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9434104240 0x7f9434104650 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:38.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.143+0000 7f943bec5700 1 -- 172.21.15.90:0/3512306536 >> 172.21.15.90:0/3512306536 conn(0x7f94340fe750 msgr2=0x7f9434100c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:38.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.143+0000 7f943bec5700 1 -- 172.21.15.90:0/3512306536 shutdown_connections 2024-04-25T11:34:38.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.143+0000 7f943bec5700 1 -- 172.21.15.90:0/3512306536 wait complete. 2024-04-25T11:34:38.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.144+0000 7f943bec5700 1 Processor -- start 2024-04-25T11:34:38.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.144+0000 7f943bec5700 1 -- start start 2024-04-25T11:34:38.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.144+0000 7f943bec5700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9434103110 0x7f94341996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:38.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.144+0000 7f943bec5700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9434104240 0x7f9434199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:38.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.144+0000 7f943bec5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9434104be0 0x7f94341a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:38.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.144+0000 7f943bec5700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f943410eb80 con 0x7f9434104be0 2024-04-25T11:34:38.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.144+0000 7f943bec5700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f943410ea00 con 0x7f9434104240 2024-04-25T11:34:38.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.144+0000 7f943bec5700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f943410ed00 con 0x7f9434103110 2024-04-25T11:34:38.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.144+0000 7f943a462700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9434104be0 0x7f94341a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:38.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.144+0000 7f943a462700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9434104be0 0x7f94341a0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35170/0 (socket says 172.21.15.90:35170) 2024-04-25T11:34:38.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.145+0000 7f943a462700 1 -- 172.21.15.90:0/3157536535 learned_addr learned my addr 172.21.15.90:0/3157536535 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:38.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.145+0000 7f9439460700 1 --2- 172.21.15.90:0/3157536535 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9434104240 0x7f9434199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:38.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.145+0000 7f9439c61700 1 --2- 172.21.15.90:0/3157536535 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9434103110 0x7f94341996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:38.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.145+0000 7f943a462700 1 -- 172.21.15.90:0/3157536535 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9434103110 msgr2=0x7f94341996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:38.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.145+0000 7f943a462700 1 --2- 172.21.15.90:0/3157536535 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9434103110 0x7f94341996f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:38.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.145+0000 7f943a462700 1 -- 172.21.15.90:0/3157536535 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9434104240 msgr2=0x7f9434199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:38.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.145+0000 7f943a462700 1 --2- 172.21.15.90:0/3157536535 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9434104240 0x7f9434199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:38.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.145+0000 7f943a462700 1 -- 172.21.15.90:0/3157536535 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94341a11c0 con 0x7f9434104be0 2024-04-25T11:34:38.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.145+0000 7f943a462700 1 --2- 172.21.15.90:0/3157536535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9434104be0 0x7f94341a0bd0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f942c007d70 tx=0x7f942c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:38.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.145+0000 7f942affd700 1 -- 172.21.15.90:0/3157536535 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942c016020 con 0x7f9434104be0 2024-04-25T11:34:38.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.145+0000 7f942affd700 1 -- 172.21.15.90:0/3157536535 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f942c00ad40 con 0x7f9434104be0 2024-04-25T11:34:38.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.145+0000 7f943bec5700 1 -- 172.21.15.90:0/3157536535 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94341a14b0 con 0x7f9434104be0 2024-04-25T11:34:38.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.145+0000 7f942affd700 1 -- 172.21.15.90:0/3157536535 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942c0115e0 con 0x7f9434104be0 2024-04-25T11:34:38.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.145+0000 7f943bec5700 1 -- 172.21.15.90:0/3157536535 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94341a1a40 con 0x7f9434104be0 2024-04-25T11:34:38.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.146+0000 7f943bec5700 1 -- 172.21.15.90:0/3157536535 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9434051190 con 0x7f9434104be0 2024-04-25T11:34:38.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.149+0000 7f942affd700 1 -- 172.21.15.90:0/3157536535 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f942c019020 con 0x7f9434104be0 2024-04-25T11:34:38.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.149+0000 7f942affd700 1 --2- 172.21.15.90:0/3157536535 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f94200afc90 0x7f94200b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:38.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.149+0000 7f9439c61700 1 --2- 172.21.15.90:0/3157536535 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f94200afc90 0x7f94200b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:38.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.149+0000 7f942affd700 1 -- 172.21.15.90:0/3157536535 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f942c0def80 con 0x7f9434104be0 2024-04-25T11:34:38.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.150+0000 7f9439c61700 1 --2- 172.21.15.90:0/3157536535 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f94200afc90 0x7f94200b21a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9424005fd0 tx=0x7f9424005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:38.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.153+0000 7f942affd700 1 -- 172.21.15.90:0/3157536535 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f942c020020 con 0x7f9434104be0 2024-04-25T11:34:38.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:37 smithi138 ceph-mon[32328]: pgmap v319: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-04-25T11:34:38.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.368+0000 7f943bec5700 1 -- 172.21.15.90:0/3157536535 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f9434063ad0 con 0x7f94200afc90 2024-04-25T11:34:38.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.369+0000 7f942affd700 1 -- 172.21.15.90:0/3157536535 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f9434063ad0 con 0x7f94200afc90 2024-04-25T11:34:38.358 INFO:teuthology.orchestra.run.smithi090.stdout:Backend: cephadm 2024-04-25T11:34:38.358 INFO:teuthology.orchestra.run.smithi090.stdout:Available: Yes 2024-04-25T11:34:38.358 INFO:teuthology.orchestra.run.smithi090.stdout:Paused: No 2024-04-25T11:34:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.371+0000 7f943bec5700 1 -- 172.21.15.90:0/3157536535 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f94200afc90 msgr2=0x7f94200b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.371+0000 7f943bec5700 1 --2- 172.21.15.90:0/3157536535 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f94200afc90 0x7f94200b21a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9424005fd0 tx=0x7f9424005dd0 comp rx=0 tx=0).stop 2024-04-25T11:34:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.371+0000 7f943bec5700 1 -- 172.21.15.90:0/3157536535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9434104be0 msgr2=0x7f94341a0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.371+0000 7f943bec5700 1 --2- 172.21.15.90:0/3157536535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9434104be0 0x7f94341a0bd0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f942c007d70 tx=0x7f942c00a340 comp rx=0 tx=0).stop 2024-04-25T11:34:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.371+0000 7f943bec5700 1 -- 172.21.15.90:0/3157536535 shutdown_connections 2024-04-25T11:34:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.371+0000 7f943bec5700 1 --2- 172.21.15.90:0/3157536535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9434104be0 0x7f94341a0bd0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.371+0000 7f943bec5700 1 --2- 172.21.15.90:0/3157536535 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f94200afc90 0x7f94200b21a0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:38.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.371+0000 7f943bec5700 1 --2- 172.21.15.90:0/3157536535 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9434103110 0x7f94341996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:38.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.371+0000 7f943bec5700 1 --2- 172.21.15.90:0/3157536535 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9434104240 0x7f9434199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:38.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.371+0000 7f943bec5700 1 -- 172.21.15.90:0/3157536535 >> 172.21.15.90:0/3157536535 conn(0x7f94340fe750 msgr2=0x7f9434107470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:38.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.371+0000 7f943bec5700 1 -- 172.21.15.90:0/3157536535 shutdown_connections 2024-04-25T11:34:38.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:38.371+0000 7f943bec5700 1 -- 172.21.15.90:0/3157536535 wait complete. 2024-04-25T11:34:38.989 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-25T11:34:39.226 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:34:39.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:39 smithi090 ceph-mon[29575]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:34:39.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:39 smithi090 ceph-mon[29575]: pgmap v320: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-25T11:34:40.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:39 smithi138 ceph-mon[32328]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:34:40.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:39 smithi138 ceph-mon[32328]: pgmap v320: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-25T11:34:40.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:39 smithi156 ceph-mon[33045]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:34:40.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:39 smithi156 ceph-mon[33045]: pgmap v320: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-25T11:34:40.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.789+0000 7f101459c700 1 -- 172.21.15.90:0/3514342317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f100c104bd0 msgr2=0x7f100c10b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:40.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.789+0000 7f101459c700 1 --2- 172.21.15.90:0/3514342317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f100c104bd0 0x7f100c10b480 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f100400b0a0 tx=0x7f100400b3b0 comp rx=0 tx=0).stop 2024-04-25T11:34:40.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.789+0000 7f101459c700 1 -- 172.21.15.90:0/3514342317 shutdown_connections 2024-04-25T11:34:40.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.789+0000 7f101459c700 1 --2- 172.21.15.90:0/3514342317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f100c104bd0 0x7f100c10b480 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:40.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.789+0000 7f101459c700 1 --2- 172.21.15.90:0/3514342317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f100c103100 0x7f100c103510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:40.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.789+0000 7f101459c700 1 --2- 172.21.15.90:0/3514342317 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f100c104230 0x7f100c104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:40.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.789+0000 7f101459c700 1 -- 172.21.15.90:0/3514342317 >> 172.21.15.90:0/3514342317 conn(0x7f100c0fe740 msgr2=0x7f100c100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:40.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.789+0000 7f101459c700 1 -- 172.21.15.90:0/3514342317 shutdown_connections 2024-04-25T11:34:40.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.789+0000 7f101459c700 1 -- 172.21.15.90:0/3514342317 wait complete. 2024-04-25T11:34:40.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.790+0000 7f101459c700 1 Processor -- start 2024-04-25T11:34:40.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.790+0000 7f101459c700 1 -- start start 2024-04-25T11:34:40.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.790+0000 7f101459c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f100c103100 0x7f100c076380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:40.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.790+0000 7f101459c700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f100c104230 0x7f100c19f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:40.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.790+0000 7f101459c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f100c104bd0 0x7f100c1a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:40.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.790+0000 7f101459c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f100c10eb70 con 0x7f100c103100 2024-04-25T11:34:40.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.790+0000 7f101459c700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f100c10e9f0 con 0x7f100c104230 2024-04-25T11:34:40.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.790+0000 7f101459c700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f100c10ecf0 con 0x7f100c104bd0 2024-04-25T11:34:40.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.790+0000 7f1012338700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f100c103100 0x7f100c076380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:40.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.790+0000 7f1012338700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f100c103100 0x7f100c076380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35202/0 (socket says 172.21.15.90:35202) 2024-04-25T11:34:40.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.790+0000 7f1012338700 1 -- 172.21.15.90:0/3164550448 learned_addr learned my addr 172.21.15.90:0/3164550448 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:40.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.790+0000 7f1011b37700 1 --2- 172.21.15.90:0/3164550448 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f100c104230 0x7f100c19f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:40.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.790+0000 7f1012338700 1 -- 172.21.15.90:0/3164550448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f100c104bd0 msgr2=0x7f100c1a1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:40.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.790+0000 7f1012338700 1 --2- 172.21.15.90:0/3164550448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f100c104bd0 0x7f100c1a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:40.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.790+0000 7f1012338700 1 -- 172.21.15.90:0/3164550448 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f100c104230 msgr2=0x7f100c19f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:40.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.790+0000 7f1012338700 1 --2- 172.21.15.90:0/3164550448 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f100c104230 0x7f100c19f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:40.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.790+0000 7f1012338700 1 -- 172.21.15.90:0/3164550448 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f100c1a2200 con 0x7f100c103100 2024-04-25T11:34:40.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.790+0000 7f1012338700 1 --2- 172.21.15.90:0/3164550448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f100c103100 0x7f100c076380 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f100800c970 tx=0x7f100800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:40.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.791+0000 7f10037fe700 1 -- 172.21.15.90:0/3164550448 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1008007b00 con 0x7f100c103100 2024-04-25T11:34:40.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.791+0000 7f10037fe700 1 -- 172.21.15.90:0/3164550448 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1008007ca0 con 0x7f100c103100 2024-04-25T11:34:40.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.791+0000 7f101459c700 1 -- 172.21.15.90:0/3164550448 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f100c1a24f0 con 0x7f100c103100 2024-04-25T11:34:40.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.791+0000 7f10037fe700 1 -- 172.21.15.90:0/3164550448 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1008018660 con 0x7f100c103100 2024-04-25T11:34:40.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.791+0000 7f101459c700 1 -- 172.21.15.90:0/3164550448 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f100c1a2a80 con 0x7f100c103100 2024-04-25T11:34:40.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.792+0000 7f101459c700 1 -- 172.21.15.90:0/3164550448 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f100c051190 con 0x7f100c103100 2024-04-25T11:34:40.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.793+0000 7f10037fe700 1 -- 172.21.15.90:0/3164550448 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f100800f450 con 0x7f100c103100 2024-04-25T11:34:40.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.794+0000 7f10037fe700 1 --2- 172.21.15.90:0/3164550448 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0ff80afc90 0x7f0ff80b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:40.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.794+0000 7f1011b37700 1 --2- 172.21.15.90:0/3164550448 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0ff80afc90 0x7f0ff80b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:40.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.794+0000 7f10037fe700 1 -- 172.21.15.90:0/3164550448 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f10080dede0 con 0x7f100c103100 2024-04-25T11:34:40.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.794+0000 7f1011b37700 1 --2- 172.21.15.90:0/3164550448 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0ff80afc90 0x7f0ff80b21a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc005950 tx=0x7f0ffc005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:40.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:40.797+0000 7f10037fe700 1 -- 172.21.15.90:0/3164550448 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1008014030 con 0x7f100c103100 2024-04-25T11:34:41.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:41.012+0000 7f101459c700 1 -- 172.21.15.90:0/3164550448 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f100c063ad0 con 0x7f0ff80afc90 2024-04-25T11:34:41.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:41.025+0000 7f10037fe700 1 -- 172.21.15.90:0/3164550448 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7f100c063ad0 con 0x7f0ff80afc90 2024-04-25T11:34:41.014 INFO:teuthology.orchestra.run.smithi090.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-25T11:34:41.015 INFO:teuthology.orchestra.run.smithi090.stdout:mds.a smithi090 running (98s) 15s ago 98s 84.7M - 19.0.0-3291-g994e2f20 d53c96779a29 eb3aab653da1 2024-04-25T11:34:41.015 INFO:teuthology.orchestra.run.smithi090.stdout:mds.b smithi138 running (95s) 29s ago 95s 94.2M - 19.0.0-3291-g994e2f20 d53c96779a29 64820dcad7ed 2024-04-25T11:34:41.015 INFO:teuthology.orchestra.run.smithi090.stdout:mds.c smithi156 running (109s) 29s ago 109s 84.2M - 19.0.0-3291-g994e2f20 d53c96779a29 23e823f56e5e 2024-04-25T11:34:41.015 INFO:teuthology.orchestra.run.smithi090.stdout:mds.d smithi090 running (118s) 15s ago 117s 80.4M - 19.0.0-3291-g994e2f20 d53c96779a29 9b9aed298d55 2024-04-25T11:34:41.015 INFO:teuthology.orchestra.run.smithi090.stdout:mds.e smithi138 running (115s) 29s ago 114s 85.1M - 19.0.0-3291-g994e2f20 d53c96779a29 c48443ecc17a 2024-04-25T11:34:41.015 INFO:teuthology.orchestra.run.smithi090.stdout:mds.f smithi156 running (112s) 29s ago 112s 85.6M - 19.0.0-3291-g994e2f20 d53c96779a29 d546e8c332cf 2024-04-25T11:34:41.015 INFO:teuthology.orchestra.run.smithi090.stdout:mds.g smithi090 running (101s) 15s ago 101s 83.6M - 19.0.0-3291-g994e2f20 d53c96779a29 d9063c55f074 2024-04-25T11:34:41.015 INFO:teuthology.orchestra.run.smithi090.stdout:mds.h smithi138 running (106s) 29s ago 106s 85.5M - 19.0.0-3291-g994e2f20 d53c96779a29 e39188a45eb2 2024-04-25T11:34:41.015 INFO:teuthology.orchestra.run.smithi090.stdout:mds.i smithi156 running (104s) 29s ago 103s 85.5M - 19.0.0-3291-g994e2f20 d53c96779a29 e24b5c5c6c38 2024-04-25T11:34:41.015 INFO:teuthology.orchestra.run.smithi090.stdout:mds.j smithi090 running (2m) 15s ago 2m 81.7M - 19.0.0-3291-g994e2f20 d53c96779a29 4de44129c29a 2024-04-25T11:34:41.015 INFO:teuthology.orchestra.run.smithi090.stdout:mds.k smithi138 running (2m) 29s ago 2m 84.0M - 19.0.0-3291-g994e2f20 d53c96779a29 745d946e79ea 2024-04-25T11:34:41.016 INFO:teuthology.orchestra.run.smithi090.stdout:mds.l smithi156 running (2m) 29s ago 2m 84.6M - 19.0.0-3291-g994e2f20 d53c96779a29 f5c838c4b135 2024-04-25T11:34:41.016 INFO:teuthology.orchestra.run.smithi090.stdout:mgr.x smithi090 *:9283,8765 running (9m) 15s ago 9m 585M - 19.0.0-3291-g994e2f20 d53c96779a29 89807c68fe17 2024-04-25T11:34:41.016 INFO:teuthology.orchestra.run.smithi090.stdout:mgr.y smithi138 *:8443,8765 running (7m) 29s ago 7m 520M - 19.0.0-3291-g994e2f20 d53c96779a29 a7e64dbe4f67 2024-04-25T11:34:41.016 INFO:teuthology.orchestra.run.smithi090.stdout:mgr.z smithi156 *:8443,8765 running (7m) 29s ago 7m 521M - 19.0.0-3291-g994e2f20 d53c96779a29 6b0ef8068864 2024-04-25T11:34:41.016 INFO:teuthology.orchestra.run.smithi090.stdout:mon.a smithi090 running (9m) 15s ago 9m 117M 2048M 19.0.0-3291-g994e2f20 d53c96779a29 285c6eedf8b9 2024-04-25T11:34:41.016 INFO:teuthology.orchestra.run.smithi090.stdout:mon.b smithi138 running (7m) 29s ago 7m 107M 2048M 19.0.0-3291-g994e2f20 d53c96779a29 f9ca4b15f41e 2024-04-25T11:34:41.016 INFO:teuthology.orchestra.run.smithi090.stdout:mon.c smithi156 running (7m) 29s ago 7m 106M 2048M 19.0.0-3291-g994e2f20 d53c96779a29 ad026314adca 2024-04-25T11:34:41.016 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0 smithi090 running (6m) 15s ago 6m 116M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 94273764a144 2024-04-25T11:34:41.016 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1 smithi138 running (6m) 29s ago 6m 124M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 5bb785777d3e 2024-04-25T11:34:41.016 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2 smithi156 running (6m) 29s ago 6m 123M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 85c7fb478862 2024-04-25T11:34:41.016 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3 smithi090 running (5m) 15s ago 5m 113M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 c38a020443bf 2024-04-25T11:34:41.016 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4 smithi138 running (5m) 29s ago 5m 122M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 50984d573bf4 2024-04-25T11:34:41.016 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5 smithi156 running (4m) 29s ago 4m 115M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 5ea6488d4794 2024-04-25T11:34:41.017 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6 smithi090 running (4m) 15s ago 4m 122M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 970baa2826bf 2024-04-25T11:34:41.017 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7 smithi138 running (4m) 29s ago 4m 123M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 1df0a9ab48ad 2024-04-25T11:34:41.017 INFO:teuthology.orchestra.run.smithi090.stdout:osd.8 smithi156 running (3m) 29s ago 3m 114M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 0ffaee681c9d 2024-04-25T11:34:41.017 INFO:teuthology.orchestra.run.smithi090.stdout:osd.9 smithi090 running (3m) 15s ago 3m 114M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 1954684f948a 2024-04-25T11:34:41.017 INFO:teuthology.orchestra.run.smithi090.stdout:osd.10 smithi138 running (2m) 29s ago 2m 124M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 5c8c90aefd2c 2024-04-25T11:34:41.017 INFO:teuthology.orchestra.run.smithi090.stdout:osd.11 smithi156 running (2m) 29s ago 2m 119M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 2cedbc52dd3f 2024-04-25T11:34:41.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:41.028+0000 7f101459c700 1 -- 172.21.15.90:0/3164550448 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0ff80afc90 msgr2=0x7f0ff80b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:41.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:41.028+0000 7f101459c700 1 --2- 172.21.15.90:0/3164550448 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0ff80afc90 0x7f0ff80b21a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc005950 tx=0x7f0ffc005ed0 comp rx=0 tx=0).stop 2024-04-25T11:34:41.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:41.028+0000 7f101459c700 1 -- 172.21.15.90:0/3164550448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f100c103100 msgr2=0x7f100c076380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:41.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:41.028+0000 7f101459c700 1 --2- 172.21.15.90:0/3164550448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f100c103100 0x7f100c076380 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f100800c970 tx=0x7f100800cd30 comp rx=0 tx=0).stop 2024-04-25T11:34:41.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:41.028+0000 7f101459c700 1 -- 172.21.15.90:0/3164550448 shutdown_connections 2024-04-25T11:34:41.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:41.028+0000 7f101459c700 1 --2- 172.21.15.90:0/3164550448 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0ff80afc90 0x7f0ff80b21a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:41.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:41.028+0000 7f101459c700 1 --2- 172.21.15.90:0/3164550448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f100c104bd0 0x7f100c1a1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:41.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:41.028+0000 7f101459c700 1 --2- 172.21.15.90:0/3164550448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f100c103100 0x7f100c076380 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:41.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:41.028+0000 7f101459c700 1 --2- 172.21.15.90:0/3164550448 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f100c104230 0x7f100c19f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:41.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:41.028+0000 7f101459c700 1 -- 172.21.15.90:0/3164550448 >> 172.21.15.90:0/3164550448 conn(0x7f100c0fe740 msgr2=0x7f100c107460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:41.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:41.028+0000 7f101459c700 1 -- 172.21.15.90:0/3164550448 shutdown_connections 2024-04-25T11:34:41.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:41.028+0000 7f101459c700 1 -- 172.21.15.90:0/3164550448 wait complete. 2024-04-25T11:34:41.667 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-25T11:34:41.921 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:34:41.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:41 smithi090 ceph-mon[29575]: pgmap v321: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-25T11:34:42.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:41 smithi138 ceph-mon[32328]: pgmap v321: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-25T11:34:42.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:41 smithi156 ceph-mon[33045]: pgmap v321: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-25T11:34:43.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:42 smithi090 ceph-mon[29575]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:34:43.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:42 smithi138 ceph-mon[32328]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:34:43.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:42 smithi156 ceph-mon[33045]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:34:43.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.633+0000 7fa8d2d28700 1 -- 172.21.15.90:0/2595360149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8cc104210 msgr2=0x7fa8cc104620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:43.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.633+0000 7fa8d2d28700 1 --2- 172.21.15.90:0/2595360149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8cc104210 0x7fa8cc104620 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc009990 tx=0x7fa8bc009ca0 comp rx=0 tx=0).stop 2024-04-25T11:34:43.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.633+0000 7fa8d2d28700 1 -- 172.21.15.90:0/2595360149 shutdown_connections 2024-04-25T11:34:43.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.633+0000 7fa8d2d28700 1 --2- 172.21.15.90:0/2595360149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8cc104bb0 0x7fa8cc10b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:43.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.633+0000 7fa8d2d28700 1 --2- 172.21.15.90:0/2595360149 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa8cc1030e0 0x7fa8cc1034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:43.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.633+0000 7fa8d2d28700 1 --2- 172.21.15.90:0/2595360149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8cc104210 0x7fa8cc104620 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:43.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.633+0000 7fa8d2d28700 1 -- 172.21.15.90:0/2595360149 >> 172.21.15.90:0/2595360149 conn(0x7fa8cc0fe740 msgr2=0x7fa8cc100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:43.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.633+0000 7fa8d2d28700 1 -- 172.21.15.90:0/2595360149 shutdown_connections 2024-04-25T11:34:43.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.633+0000 7fa8d2d28700 1 -- 172.21.15.90:0/2595360149 wait complete. 2024-04-25T11:34:43.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.634+0000 7fa8d2d28700 1 Processor -- start 2024-04-25T11:34:43.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.634+0000 7fa8d2d28700 1 -- start start 2024-04-25T11:34:43.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.634+0000 7fa8d2d28700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa8cc1030e0 0x7fa8cc199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:43.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.634+0000 7fa8d2d28700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8cc104210 0x7fa8cc199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:43.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.634+0000 7fa8d2d28700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8cc104bb0 0x7fa8cc1a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:43.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.634+0000 7fa8d2d28700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa8cc10ebd0 con 0x7fa8cc104bb0 2024-04-25T11:34:43.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.634+0000 7fa8d2d28700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa8cc10ea50 con 0x7fa8cc1030e0 2024-04-25T11:34:43.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.634+0000 7fa8d2d28700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa8cc10ed50 con 0x7fa8cc104210 2024-04-25T11:34:43.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.634+0000 7fa8d12c5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8cc104bb0 0x7fa8cc1a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:43.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.634+0000 7fa8d12c5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8cc104bb0 0x7fa8cc1a0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41430/0 (socket says 172.21.15.90:41430) 2024-04-25T11:34:43.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.634+0000 7fa8d12c5700 1 -- 172.21.15.90:0/709697126 learned_addr learned my addr 172.21.15.90:0/709697126 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:43.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.634+0000 7fa8d0ac4700 1 --2- 172.21.15.90:0/709697126 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa8cc1030e0 0x7fa8cc199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:43.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.634+0000 7fa8cbfff700 1 --2- 172.21.15.90:0/709697126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8cc104210 0x7fa8cc199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.634+0000 7fa8d12c5700 1 -- 172.21.15.90:0/709697126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8cc104210 msgr2=0x7fa8cc199c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.634+0000 7fa8d12c5700 1 --2- 172.21.15.90:0/709697126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8cc104210 0x7fa8cc199c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.634+0000 7fa8d12c5700 1 -- 172.21.15.90:0/709697126 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa8cc1030e0 msgr2=0x7fa8cc199710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.634+0000 7fa8d12c5700 1 --2- 172.21.15.90:0/709697126 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa8cc1030e0 0x7fa8cc199710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.634+0000 7fa8d12c5700 1 -- 172.21.15.90:0/709697126 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8cc06b590 con 0x7fa8cc104bb0 2024-04-25T11:34:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.635+0000 7fa8cbfff700 1 --2- 172.21.15.90:0/709697126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8cc104210 0x7fa8cc199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.635+0000 7fa8d0ac4700 1 --2- 172.21.15.90:0/709697126 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa8cc1030e0 0x7fa8cc199710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.635+0000 7fa8d12c5700 1 --2- 172.21.15.90:0/709697126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8cc104bb0 0x7fa8cc1a0bf0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa8b400b550 tx=0x7fa8b400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.635+0000 7fa8c9ffb700 1 -- 172.21.15.90:0/709697126 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b4015020 con 0x7fa8cc104bb0 2024-04-25T11:34:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.635+0000 7fa8d2d28700 1 -- 172.21.15.90:0/709697126 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8cc06b880 con 0x7fa8cc104bb0 2024-04-25T11:34:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.635+0000 7fa8c9ffb700 1 -- 172.21.15.90:0/709697126 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8b4011ce0 con 0x7fa8cc104bb0 2024-04-25T11:34:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.635+0000 7fa8c9ffb700 1 -- 172.21.15.90:0/709697126 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b4010d40 con 0x7fa8cc104bb0 2024-04-25T11:34:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.635+0000 7fa8d2d28700 1 -- 172.21.15.90:0/709697126 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8cc06be40 con 0x7fa8cc104bb0 2024-04-25T11:34:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.636+0000 7fa8d2d28700 1 -- 172.21.15.90:0/709697126 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8cc051190 con 0x7fa8cc104bb0 2024-04-25T11:34:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.637+0000 7fa8c9ffb700 1 -- 172.21.15.90:0/709697126 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fa8b4018020 con 0x7fa8cc104bb0 2024-04-25T11:34:43.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.638+0000 7fa8c9ffb700 1 --2- 172.21.15.90:0/709697126 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fa8b80afc10 0x7fa8b80b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:43.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.638+0000 7fa8d0ac4700 1 --2- 172.21.15.90:0/709697126 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fa8b80afc10 0x7fa8b80b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:43.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.638+0000 7fa8c9ffb700 1 -- 172.21.15.90:0/709697126 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7fa8b40dea50 con 0x7fa8cc104bb0 2024-04-25T11:34:43.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.638+0000 7fa8d0ac4700 1 --2- 172.21.15.90:0/709697126 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fa8b80afc10 0x7fa8b80b2120 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc104110 tx=0x7fa8c4009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:43.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.642+0000 7fa8c9ffb700 1 -- 172.21.15.90:0/709697126 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa8b401f020 con 0x7fa8cc104bb0 2024-04-25T11:34:43.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.851+0000 7fa8d2d28700 1 -- 172.21.15.90:0/709697126 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fa8cc063ad0 con 0x7fa8b80afc10 2024-04-25T11:34:43.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.859+0000 7fa8c9ffb700 1 -- 172.21.15.90:0/709697126 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fa8cc063ad0 con 0x7fa8b80afc10 2024-04-25T11:34:43.848 INFO:teuthology.orchestra.run.smithi090.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-25T11:34:43.848 INFO:teuthology.orchestra.run.smithi090.stdout:mds.all 12/12 32s ago 2m smithi090=a;smithi090=d;smithi090=g;smithi090=j;smithi138=b;smithi138=e;smithi138=h;smithi138=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l;count:12 2024-04-25T11:34:43.848 INFO:teuthology.orchestra.run.smithi090.stdout:mgr 3/3 32s ago 7m smithi090=x;smithi138=y;smithi156=z;count:3 2024-04-25T11:34:43.848 INFO:teuthology.orchestra.run.smithi090.stdout:mon 3/3 32s ago 7m smithi090:172.21.15.90=a;smithi138:172.21.15.138=b;smithi156:172.21.15.156=c;count:3 2024-04-25T11:34:43.848 INFO:teuthology.orchestra.run.smithi090.stdout:osd 12 32s ago - 2024-04-25T11:34:43.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.861+0000 7fa8d2d28700 1 -- 172.21.15.90:0/709697126 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fa8b80afc10 msgr2=0x7fa8b80b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:43.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.861+0000 7fa8d2d28700 1 --2- 172.21.15.90:0/709697126 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fa8b80afc10 0x7fa8b80b2120 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc104110 tx=0x7fa8c4009040 comp rx=0 tx=0).stop 2024-04-25T11:34:43.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.861+0000 7fa8d2d28700 1 -- 172.21.15.90:0/709697126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8cc104bb0 msgr2=0x7fa8cc1a0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:43.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.861+0000 7fa8d2d28700 1 --2- 172.21.15.90:0/709697126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8cc104bb0 0x7fa8cc1a0bf0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa8b400b550 tx=0x7fa8b400b910 comp rx=0 tx=0).stop 2024-04-25T11:34:43.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.862+0000 7fa8d2d28700 1 -- 172.21.15.90:0/709697126 shutdown_connections 2024-04-25T11:34:43.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.862+0000 7fa8d2d28700 1 --2- 172.21.15.90:0/709697126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8cc104bb0 0x7fa8cc1a0bf0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:43.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.862+0000 7fa8d2d28700 1 --2- 172.21.15.90:0/709697126 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa8cc1030e0 0x7fa8cc199710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:43.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.862+0000 7fa8d2d28700 1 --2- 172.21.15.90:0/709697126 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fa8b80afc10 0x7fa8b80b2120 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:43.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.862+0000 7fa8d2d28700 1 --2- 172.21.15.90:0/709697126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8cc104210 0x7fa8cc199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:43.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.862+0000 7fa8d2d28700 1 -- 172.21.15.90:0/709697126 >> 172.21.15.90:0/709697126 conn(0x7fa8cc0fe740 msgr2=0x7fa8cc107440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:43.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.862+0000 7fa8d2d28700 1 -- 172.21.15.90:0/709697126 shutdown_connections 2024-04-25T11:34:43.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:43.862+0000 7fa8d2d28700 1 -- 172.21.15.90:0/709697126 wait complete. 2024-04-25T11:34:44.155 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:43 smithi090 ceph-mon[29575]: pgmap v322: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-25T11:34:44.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:43 smithi138 ceph-mon[32328]: pgmap v322: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-25T11:34:44.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:43 smithi156 ceph-mon[33045]: pgmap v322: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-25T11:34:44.460 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-25T11:34:44.699 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:34:45.154 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:44 smithi090 ceph-mon[29575]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:34:45.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:44 smithi138 ceph-mon[32328]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:34:45.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:44 smithi156 ceph-mon[33045]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:34:46.135 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:45 smithi090 ceph-mon[29575]: pgmap v323: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-25T11:34:46.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.259+0000 7f77a5e87700 1 -- 172.21.15.90:0/767084349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77a0104bd0 msgr2=0x7f77a010b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:46.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.259+0000 7f77a5e87700 1 --2- 172.21.15.90:0/767084349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77a0104bd0 0x7f77a010b480 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f779800b0a0 tx=0x7f779800b3b0 comp rx=0 tx=0).stop 2024-04-25T11:34:46.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.259+0000 7f77a5e87700 1 -- 172.21.15.90:0/767084349 shutdown_connections 2024-04-25T11:34:46.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.259+0000 7f77a5e87700 1 --2- 172.21.15.90:0/767084349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77a0104bd0 0x7f77a010b480 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:46.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.259+0000 7f77a5e87700 1 --2- 172.21.15.90:0/767084349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77a0103100 0x7f77a0103510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:46.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.259+0000 7f77a5e87700 1 --2- 172.21.15.90:0/767084349 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f77a0104230 0x7f77a0104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:46.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.259+0000 7f77a5e87700 1 -- 172.21.15.90:0/767084349 >> 172.21.15.90:0/767084349 conn(0x7f77a00fe740 msgr2=0x7f77a0100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:46.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.259+0000 7f77a5e87700 1 -- 172.21.15.90:0/767084349 shutdown_connections 2024-04-25T11:34:46.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.259+0000 7f77a5e87700 1 -- 172.21.15.90:0/767084349 wait complete. 2024-04-25T11:34:46.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.259+0000 7f77a5e87700 1 Processor -- start 2024-04-25T11:34:46.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.260+0000 7f77a5e87700 1 -- start start 2024-04-25T11:34:46.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.260+0000 7f77a5e87700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77a0103100 0x7f77a0199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:46.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.260+0000 7f77a5e87700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77a0104230 0x7f77a0199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:46.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.260+0000 7f77a5e87700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f77a0104bd0 0x7f77a01a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:46.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.260+0000 7f77a5e87700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f77a010eb70 con 0x7f77a0103100 2024-04-25T11:34:46.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.260+0000 7f77a5e87700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f77a010e9f0 con 0x7f77a0104bd0 2024-04-25T11:34:46.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.260+0000 7f77a5e87700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f77a010ecf0 con 0x7f77a0104230 2024-04-25T11:34:46.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.260+0000 7f779f7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77a0103100 0x7f77a0199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:46.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.260+0000 7f779f7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77a0103100 0x7f77a0199620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41440/0 (socket says 172.21.15.90:41440) 2024-04-25T11:34:46.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.260+0000 7f779f7fe700 1 -- 172.21.15.90:0/2503325276 learned_addr learned my addr 172.21.15.90:0/2503325276 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:46.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.260+0000 7f779ffff700 1 --2- 172.21.15.90:0/2503325276 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f77a0104bd0 0x7f77a01a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:46.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.260+0000 7f779effd700 1 --2- 172.21.15.90:0/2503325276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77a0104230 0x7f77a0199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:46.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.260+0000 7f779f7fe700 1 -- 172.21.15.90:0/2503325276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77a0104230 msgr2=0x7f77a0199b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:46.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.260+0000 7f779f7fe700 1 --2- 172.21.15.90:0/2503325276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77a0104230 0x7f77a0199b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:46.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.260+0000 7f779f7fe700 1 -- 172.21.15.90:0/2503325276 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f77a0104bd0 msgr2=0x7f77a01a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:46.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.260+0000 7f779f7fe700 1 --2- 172.21.15.90:0/2503325276 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f77a0104bd0 0x7f77a01a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:46.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.260+0000 7f779f7fe700 1 -- 172.21.15.90:0/2503325276 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77a01a1180 con 0x7f77a0103100 2024-04-25T11:34:46.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.260+0000 7f779f7fe700 1 --2- 172.21.15.90:0/2503325276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77a0103100 0x7f77a0199620 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f779000d950 tx=0x7f779000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:46.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.261+0000 7f779cff9700 1 -- 172.21.15.90:0/2503325276 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7790017070 con 0x7f77a0103100 2024-04-25T11:34:46.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.261+0000 7f779cff9700 1 -- 172.21.15.90:0/2503325276 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7790009730 con 0x7f77a0103100 2024-04-25T11:34:46.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.261+0000 7f77a5e87700 1 -- 172.21.15.90:0/2503325276 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77a01a1470 con 0x7f77a0103100 2024-04-25T11:34:46.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.261+0000 7f779cff9700 1 -- 172.21.15.90:0/2503325276 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7790003a50 con 0x7f77a0103100 2024-04-25T11:34:46.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.261+0000 7f77a5e87700 1 -- 172.21.15.90:0/2503325276 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77a01a1a00 con 0x7f77a0103100 2024-04-25T11:34:46.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.262+0000 7f77a5e87700 1 -- 172.21.15.90:0/2503325276 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77a0051190 con 0x7f77a0103100 2024-04-25T11:34:46.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.263+0000 7f779cff9700 1 -- 172.21.15.90:0/2503325276 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f7790003bf0 con 0x7f77a0103100 2024-04-25T11:34:46.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.264+0000 7f779cff9700 1 --2- 172.21.15.90:0/2503325276 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f77880afc90 0x7f77880b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:46.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.264+0000 7f779effd700 1 --2- 172.21.15.90:0/2503325276 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f77880afc90 0x7f77880b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:46.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.264+0000 7f779cff9700 1 -- 172.21.15.90:0/2503325276 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f77900defd0 con 0x7f77a0103100 2024-04-25T11:34:46.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.265+0000 7f779effd700 1 --2- 172.21.15.90:0/2503325276 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f77880afc90 0x7f77880b21a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7794005950 tx=0x7f77940058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:46.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.268+0000 7f779cff9700 1 -- 172.21.15.90:0/2503325276 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7790020030 con 0x7f77a0103100 2024-04-25T11:34:46.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:45 smithi138 ceph-mon[32328]: pgmap v323: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-25T11:34:46.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:45 smithi156 ceph-mon[33045]: pgmap v323: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-25T11:34:46.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.471+0000 7f77a5e87700 1 -- 172.21.15.90:0/2503325276 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f77a0063ad0 con 0x7f77880afc90 2024-04-25T11:34:46.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.472+0000 7f779cff9700 1 -- 172.21.15.90:0/2503325276 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f77a0063ad0 con 0x7f77880afc90 2024-04-25T11:34:46.461 INFO:teuthology.orchestra.run.smithi090.stdout:HOST ADDR LABELS STATUS 2024-04-25T11:34:46.462 INFO:teuthology.orchestra.run.smithi090.stdout:smithi090 172.21.15.90 2024-04-25T11:34:46.462 INFO:teuthology.orchestra.run.smithi090.stdout:smithi138 172.21.15.138 2024-04-25T11:34:46.462 INFO:teuthology.orchestra.run.smithi090.stdout:smithi156 172.21.15.156 2024-04-25T11:34:46.462 INFO:teuthology.orchestra.run.smithi090.stdout:3 hosts in cluster 2024-04-25T11:34:46.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.474+0000 7f77a5e87700 1 -- 172.21.15.90:0/2503325276 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f77880afc90 msgr2=0x7f77880b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:46.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.474+0000 7f77a5e87700 1 --2- 172.21.15.90:0/2503325276 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f77880afc90 0x7f77880b21a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7794005950 tx=0x7f77940058e0 comp rx=0 tx=0).stop 2024-04-25T11:34:46.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.474+0000 7f77a5e87700 1 -- 172.21.15.90:0/2503325276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77a0103100 msgr2=0x7f77a0199620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:46.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.474+0000 7f77a5e87700 1 --2- 172.21.15.90:0/2503325276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77a0103100 0x7f77a0199620 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f779000d950 tx=0x7f779000dc60 comp rx=0 tx=0).stop 2024-04-25T11:34:46.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.474+0000 7f77a5e87700 1 -- 172.21.15.90:0/2503325276 shutdown_connections 2024-04-25T11:34:46.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.475+0000 7f77a5e87700 1 --2- 172.21.15.90:0/2503325276 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f77a0104bd0 0x7f77a01a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:46.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.475+0000 7f77a5e87700 1 --2- 172.21.15.90:0/2503325276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77a0103100 0x7f77a0199620 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:46.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.475+0000 7f77a5e87700 1 --2- 172.21.15.90:0/2503325276 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f77880afc90 0x7f77880b21a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:46.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.475+0000 7f77a5e87700 1 --2- 172.21.15.90:0/2503325276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77a0104230 0x7f77a0199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:46.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.475+0000 7f77a5e87700 1 -- 172.21.15.90:0/2503325276 >> 172.21.15.90:0/2503325276 conn(0x7f77a00fe740 msgr2=0x7f77a0107460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:46.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.475+0000 7f77a5e87700 1 -- 172.21.15.90:0/2503325276 shutdown_connections 2024-04-25T11:34:46.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:46.475+0000 7f77a5e87700 1 -- 172.21.15.90:0/2503325276 wait complete. 2024-04-25T11:34:47.125 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:46 smithi090 ceph-mon[29575]: from='client.15168 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:34:47.125 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:46 smithi090 ceph-mon[29575]: pgmap v324: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T11:34:47.126 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-25T11:34:47.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:46 smithi138 ceph-mon[32328]: from='client.15168 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:34:47.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:46 smithi138 ceph-mon[32328]: pgmap v324: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T11:34:47.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:46 smithi156 ceph-mon[33045]: from='client.15168 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:34:47.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:46 smithi156 ceph-mon[33045]: pgmap v324: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-25T11:34:47.364 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:34:48.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:47 smithi090 ceph-mon[29575]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-25T11:34:48.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:47 smithi138 ceph-mon[32328]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-25T11:34:48.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:47 smithi156 ceph-mon[33045]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-25T11:34:48.964 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:48 smithi090 ceph-mon[29575]: pgmap v325: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T11:34:48.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.970+0000 7f976e3ef700 1 -- 172.21.15.90:0/2504652146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9768104230 msgr2=0x7f9768104640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:49.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.970+0000 7f976e3ef700 1 --2- 172.21.15.90:0/2504652146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9768104230 0x7f9768104640 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f975c009990 tx=0x7f975c009ca0 comp rx=0 tx=0).stop 2024-04-25T11:34:49.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.970+0000 7f976e3ef700 1 -- 172.21.15.90:0/2504652146 shutdown_connections 2024-04-25T11:34:49.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.970+0000 7f976e3ef700 1 --2- 172.21.15.90:0/2504652146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9768104bd0 0x7f976810b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:49.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.970+0000 7f976e3ef700 1 --2- 172.21.15.90:0/2504652146 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9768103100 0x7f9768103510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:49.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.970+0000 7f976e3ef700 1 --2- 172.21.15.90:0/2504652146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9768104230 0x7f9768104640 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:49.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.970+0000 7f976e3ef700 1 -- 172.21.15.90:0/2504652146 >> 172.21.15.90:0/2504652146 conn(0x7f97680fe760 msgr2=0x7f9768100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:49.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.970+0000 7f976e3ef700 1 -- 172.21.15.90:0/2504652146 shutdown_connections 2024-04-25T11:34:49.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.970+0000 7f976e3ef700 1 -- 172.21.15.90:0/2504652146 wait complete. 2024-04-25T11:34:49.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.970+0000 7f976e3ef700 1 Processor -- start 2024-04-25T11:34:49.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.970+0000 7f976e3ef700 1 -- start start 2024-04-25T11:34:49.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.971+0000 7f976e3ef700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9768103100 0x7f9768076320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:49.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.971+0000 7f976e3ef700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9768104230 0x7f976819f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:49.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.971+0000 7f976e3ef700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9768104bd0 0x7f97681a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:49.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.971+0000 7f976e3ef700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f976810eb70 con 0x7f9768104230 2024-04-25T11:34:49.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.971+0000 7f976e3ef700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f976810e9f0 con 0x7f9768103100 2024-04-25T11:34:49.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.971+0000 7f976e3ef700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f976810ecf0 con 0x7f9768104bd0 2024-04-25T11:34:49.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.971+0000 7f97677fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9768104230 0x7f976819f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:49.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.971+0000 7f97677fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9768104230 0x7f976819f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41454/0 (socket says 172.21.15.90:41454) 2024-04-25T11:34:49.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.971+0000 7f97677fe700 1 -- 172.21.15.90:0/3216920718 learned_addr learned my addr 172.21.15.90:0/3216920718 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:49.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.971+0000 7f9767fff700 1 --2- 172.21.15.90:0/3216920718 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9768103100 0x7f9768076320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:49.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.971+0000 7f976c98c700 1 --2- 172.21.15.90:0/3216920718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9768104bd0 0x7f97681a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:49.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.971+0000 7f97677fe700 1 -- 172.21.15.90:0/3216920718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9768104bd0 msgr2=0x7f97681a1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:49.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.971+0000 7f97677fe700 1 --2- 172.21.15.90:0/3216920718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9768104bd0 0x7f97681a1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:49.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.971+0000 7f97677fe700 1 -- 172.21.15.90:0/3216920718 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9768103100 msgr2=0x7f9768076320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.971+0000 7f97677fe700 1 --2- 172.21.15.90:0/3216920718 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9768103100 0x7f9768076320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.971+0000 7f97677fe700 1 -- 172.21.15.90:0/3216920718 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97681a2320 con 0x7f9768104230 2024-04-25T11:34:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.971+0000 7f976c98c700 1 --2- 172.21.15.90:0/3216920718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9768104bd0 0x7f97681a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.971+0000 7f97677fe700 1 --2- 172.21.15.90:0/3216920718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9768104230 0x7f976819f580 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f975c006010 tx=0x7f975c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.971+0000 7f97657fa700 1 -- 172.21.15.90:0/3216920718 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975c027070 con 0x7f9768104230 2024-04-25T11:34:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.971+0000 7f976e3ef700 1 -- 172.21.15.90:0/3216920718 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97681a25b0 con 0x7f9768104230 2024-04-25T11:34:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.971+0000 7f97657fa700 1 -- 172.21.15.90:0/3216920718 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f975c0040c0 con 0x7f9768104230 2024-04-25T11:34:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.971+0000 7f97657fa700 1 -- 172.21.15.90:0/3216920718 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975c022d90 con 0x7f9768104230 2024-04-25T11:34:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.971+0000 7f976e3ef700 1 -- 172.21.15.90:0/3216920718 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97681a2a60 con 0x7f9768104230 2024-04-25T11:34:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.973+0000 7f976e3ef700 1 -- 172.21.15.90:0/3216920718 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9768051190 con 0x7f9768104230 2024-04-25T11:34:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.974+0000 7f97657fa700 1 -- 172.21.15.90:0/3216920718 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f975c02a070 con 0x7f9768104230 2024-04-25T11:34:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.974+0000 7f97657fa700 1 --2- 172.21.15.90:0/3216920718 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f97500afb80 0x7f97500b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.974+0000 7f9767fff700 1 --2- 172.21.15.90:0/3216920718 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f97500afb80 0x7f97500b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:49.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.975+0000 7f97657fa700 1 -- 172.21.15.90:0/3216920718 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f975c0dfa20 con 0x7f9768104230 2024-04-25T11:34:49.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.975+0000 7f9767fff700 1 --2- 172.21.15.90:0/3216920718 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f97500afb80 0x7f97500b2090 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9768104130 tx=0x7f975800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:49.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:48.979+0000 7f97657fa700 1 -- 172.21.15.90:0/3216920718 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f975c013070 con 0x7f9768104230 2024-04-25T11:34:49.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:49.189+0000 7f976e3ef700 1 -- 172.21.15.90:0/3216920718 --> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f9768063ad0 con 0x7f97500afb80 2024-04-25T11:34:49.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:49.191+0000 7f97657fa700 1 -- 172.21.15.90:0/3216920718 <== mgr.14152 v2:172.21.15.90:6800/219449516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f9768063ad0 con 0x7f97500afb80 2024-04-25T11:34:49.180 INFO:teuthology.orchestra.run.smithi090.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-25T11:34:49.180 INFO:teuthology.orchestra.run.smithi090.stdout:smithi090 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400UK400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-25T11:34:49.180 INFO:teuthology.orchestra.run.smithi090.stdout:smithi138 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6161003C400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-25T11:34:49.180 INFO:teuthology.orchestra.run.smithi090.stdout:smithi156 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB74760040480DGN 447G No 2m ago Has a FileSystem, LVM detected 2024-04-25T11:34:49.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:49.193+0000 7f976e3ef700 1 -- 172.21.15.90:0/3216920718 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f97500afb80 msgr2=0x7f97500b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:49.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:49.193+0000 7f976e3ef700 1 --2- 172.21.15.90:0/3216920718 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f97500afb80 0x7f97500b2090 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9768104130 tx=0x7f975800b410 comp rx=0 tx=0).stop 2024-04-25T11:34:49.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:49.193+0000 7f976e3ef700 1 -- 172.21.15.90:0/3216920718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9768104230 msgr2=0x7f976819f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:49.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:49.193+0000 7f976e3ef700 1 --2- 172.21.15.90:0/3216920718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9768104230 0x7f976819f580 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f975c006010 tx=0x7f975c0047b0 comp rx=0 tx=0).stop 2024-04-25T11:34:49.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:49.193+0000 7f976e3ef700 1 -- 172.21.15.90:0/3216920718 shutdown_connections 2024-04-25T11:34:49.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:49.193+0000 7f976e3ef700 1 --2- 172.21.15.90:0/3216920718 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f97500afb80 0x7f97500b2090 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:49.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:49.193+0000 7f976e3ef700 1 --2- 172.21.15.90:0/3216920718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9768104bd0 0x7f97681a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:49.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:49.193+0000 7f976e3ef700 1 --2- 172.21.15.90:0/3216920718 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9768103100 0x7f9768076320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:49.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:49.193+0000 7f976e3ef700 1 --2- 172.21.15.90:0/3216920718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9768104230 0x7f976819f580 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:49.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:49.193+0000 7f976e3ef700 1 -- 172.21.15.90:0/3216920718 >> 172.21.15.90:0/3216920718 conn(0x7f97680fe760 msgr2=0x7f9768107460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:49.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:49.193+0000 7f976e3ef700 1 -- 172.21.15.90:0/3216920718 shutdown_connections 2024-04-25T11:34:49.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:49.193+0000 7f976e3ef700 1 -- 172.21.15.90:0/3216920718 wait complete. 2024-04-25T11:34:49.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:48 smithi138 ceph-mon[32328]: pgmap v325: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T11:34:49.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:48 smithi156 ceph-mon[33045]: pgmap v325: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T11:34:49.814 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T11:34:49.823 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T11:34:49.824 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-25T11:34:50.065 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:34:50.187 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:49 smithi090 ceph-mon[29575]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:34:50.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:49 smithi138 ceph-mon[32328]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:34:50.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:49 smithi156 ceph-mon[33045]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:34:51.178 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:50 smithi090 ceph-mon[29575]: pgmap v326: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-25T11:34:51.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:50 smithi138 ceph-mon[32328]: pgmap v326: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-25T11:34:51.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:50 smithi156 ceph-mon[33045]: pgmap v326: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-25T11:34:51.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.688+0000 7f0a416a1700 1 -- 172.21.15.90:0/911463857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a3c1057b0 msgr2=0x7f0a3c105bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:51.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.688+0000 7f0a416a1700 1 --2- 172.21.15.90:0/911463857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a3c1057b0 0x7f0a3c105bc0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c009990 tx=0x7f0a2c009ca0 comp rx=0 tx=0).stop 2024-04-25T11:34:51.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.688+0000 7f0a416a1700 1 -- 172.21.15.90:0/911463857 shutdown_connections 2024-04-25T11:34:51.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.688+0000 7f0a416a1700 1 --2- 172.21.15.90:0/911463857 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a3c102b50 0x7f0a3c10c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:51.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.688+0000 7f0a416a1700 1 --2- 172.21.15.90:0/911463857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a3c1057b0 0x7f0a3c105bc0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:51.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.688+0000 7f0a416a1700 1 --2- 172.21.15.90:0/911463857 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0a3c102200 0x7f0a3c102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:51.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.688+0000 7f0a416a1700 1 -- 172.21.15.90:0/911463857 >> 172.21.15.90:0/911463857 conn(0x7f0a3c0fded0 msgr2=0x7f0a3c100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:51.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.688+0000 7f0a416a1700 1 -- 172.21.15.90:0/911463857 shutdown_connections 2024-04-25T11:34:51.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.688+0000 7f0a416a1700 1 -- 172.21.15.90:0/911463857 wait complete. 2024-04-25T11:34:51.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.689+0000 7f0a416a1700 1 Processor -- start 2024-04-25T11:34:51.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.689+0000 7f0a416a1700 1 -- start start 2024-04-25T11:34:51.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.689+0000 7f0a416a1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a3c102200 0x7f0a3c19a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:51.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.689+0000 7f0a416a1700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0a3c102b50 0x7f0a3c19acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:51.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.689+0000 7f0a416a1700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a3c1057b0 0x7f0a3c1a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:51.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.689+0000 7f0a416a1700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3c10fc60 con 0x7f0a3c102200 2024-04-25T11:34:51.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.689+0000 7f0a416a1700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3c10fae0 con 0x7f0a3c102b50 2024-04-25T11:34:51.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.689+0000 7f0a416a1700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3c10fde0 con 0x7f0a3c1057b0 2024-04-25T11:34:51.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.689+0000 7f0a32ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a3c102200 0x7f0a3c19a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:51.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.689+0000 7f0a32ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a3c102200 0x7f0a3c19a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57594/0 (socket says 172.21.15.90:57594) 2024-04-25T11:34:51.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.689+0000 7f0a32ffd700 1 -- 172.21.15.90:0/2284219452 learned_addr learned my addr 172.21.15.90:0/2284219452 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:51.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.689+0000 7f0a3affd700 1 --2- 172.21.15.90:0/2284219452 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0a3c102b50 0x7f0a3c19acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:51.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.689+0000 7f0a3b7fe700 1 --2- 172.21.15.90:0/2284219452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a3c1057b0 0x7f0a3c1a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:51.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.689+0000 7f0a32ffd700 1 -- 172.21.15.90:0/2284219452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a3c1057b0 msgr2=0x7f0a3c1a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:51.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.689+0000 7f0a32ffd700 1 --2- 172.21.15.90:0/2284219452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a3c1057b0 0x7f0a3c1a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:51.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.689+0000 7f0a32ffd700 1 -- 172.21.15.90:0/2284219452 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0a3c102b50 msgr2=0x7f0a3c19acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:51.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.690+0000 7f0a32ffd700 1 --2- 172.21.15.90:0/2284219452 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0a3c102b50 0x7f0a3c19acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:51.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.690+0000 7f0a32ffd700 1 -- 172.21.15.90:0/2284219452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a3c1a22d0 con 0x7f0a3c102200 2024-04-25T11:34:51.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.690+0000 7f0a32ffd700 1 --2- 172.21.15.90:0/2284219452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a3c102200 0x7f0a3c19a770 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c000c00 tx=0x7f0a2c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:51.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.690+0000 7f0a38ff9700 1 -- 172.21.15.90:0/2284219452 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c004830 con 0x7f0a3c102200 2024-04-25T11:34:51.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.690+0000 7f0a38ff9700 1 -- 172.21.15.90:0/2284219452 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a2c005020 con 0x7f0a3c102200 2024-04-25T11:34:51.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.690+0000 7f0a38ff9700 1 -- 172.21.15.90:0/2284219452 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c003d90 con 0x7f0a3c102200 2024-04-25T11:34:51.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.690+0000 7f0a416a1700 1 -- 172.21.15.90:0/2284219452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a3c1a2560 con 0x7f0a3c102200 2024-04-25T11:34:51.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.690+0000 7f0a416a1700 1 -- 172.21.15.90:0/2284219452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a3c1a2950 con 0x7f0a3c102200 2024-04-25T11:34:51.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.691+0000 7f0a416a1700 1 -- 172.21.15.90:0/2284219452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a3c051190 con 0x7f0a3c102200 2024-04-25T11:34:51.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.692+0000 7f0a38ff9700 1 -- 172.21.15.90:0/2284219452 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0a2c0051d0 con 0x7f0a3c102200 2024-04-25T11:34:51.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.693+0000 7f0a38ff9700 1 --2- 172.21.15.90:0/2284219452 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0a280afb80 0x7f0a280b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:51.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.693+0000 7f0a3affd700 1 --2- 172.21.15.90:0/2284219452 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0a280afb80 0x7f0a280b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:51.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.693+0000 7f0a38ff9700 1 -- 172.21.15.90:0/2284219452 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f0a2c0dffc0 con 0x7f0a3c102200 2024-04-25T11:34:51.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.693+0000 7f0a3affd700 1 --2- 172.21.15.90:0/2284219452 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0a280afb80 0x7f0a280b2090 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c19b820 tx=0x7f0a2400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:51.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.696+0000 7f0a38ff9700 1 -- 172.21.15.90:0/2284219452 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0a2c013070 con 0x7f0a3c102200 2024-04-25T11:34:51.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.941+0000 7f0a416a1700 1 -- 172.21.15.90:0/2284219452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f0a3c02e510 con 0x7f0a3c102200 2024-04-25T11:34:51.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.941+0000 7f0a38ff9700 1 -- 172.21.15.90:0/2284219452 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) ==== 76+0+3089 (secure 0 0 0) 0x7f0a2c0a8c10 con 0x7f0a3c102200 2024-04-25T11:34:51.930 INFO:teuthology.orchestra.run.smithi090.stdout:e25 2024-04-25T11:34:51.931 INFO:teuthology.orchestra.run.smithi090.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T11:34:51.931 INFO:teuthology.orchestra.run.smithi090.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T11:34:51.931 INFO:teuthology.orchestra.run.smithi090.stdout:legacy client fscid: 1 2024-04-25T11:34:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:34:51.931 INFO:teuthology.orchestra.run.smithi090.stdout:Filesystem 'cephfs' (1) 2024-04-25T11:34:51.931 INFO:teuthology.orchestra.run.smithi090.stdout:fs_name cephfs 2024-04-25T11:34:51.931 INFO:teuthology.orchestra.run.smithi090.stdout:epoch 25 2024-04-25T11:34:51.931 INFO:teuthology.orchestra.run.smithi090.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-25T11:34:51.931 INFO:teuthology.orchestra.run.smithi090.stdout:created 2024-04-25T11:33:10.373468+0000 2024-04-25T11:34:51.931 INFO:teuthology.orchestra.run.smithi090.stdout:modified 2024-04-25T11:33:16.640084+0000 2024-04-25T11:34:51.931 INFO:teuthology.orchestra.run.smithi090.stdout:tableserver 0 2024-04-25T11:34:51.931 INFO:teuthology.orchestra.run.smithi090.stdout:root 0 2024-04-25T11:34:51.931 INFO:teuthology.orchestra.run.smithi090.stdout:session_timeout 300 2024-04-25T11:34:51.932 INFO:teuthology.orchestra.run.smithi090.stdout:session_autoclose 300 2024-04-25T11:34:51.932 INFO:teuthology.orchestra.run.smithi090.stdout:max_file_size 1099511627776 2024-04-25T11:34:51.932 INFO:teuthology.orchestra.run.smithi090.stdout:max_xattr_size 65536 2024-04-25T11:34:51.932 INFO:teuthology.orchestra.run.smithi090.stdout:required_client_features {} 2024-04-25T11:34:51.932 INFO:teuthology.orchestra.run.smithi090.stdout:last_failure 0 2024-04-25T11:34:51.932 INFO:teuthology.orchestra.run.smithi090.stdout:last_failure_osd_epoch 0 2024-04-25T11:34:51.932 INFO:teuthology.orchestra.run.smithi090.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T11:34:51.932 INFO:teuthology.orchestra.run.smithi090.stdout:max_mds 5 2024-04-25T11:34:51.932 INFO:teuthology.orchestra.run.smithi090.stdout:in 0,1,2,3,4 2024-04-25T11:34:51.932 INFO:teuthology.orchestra.run.smithi090.stdout:up {0=24524,1=24499,2=24475,3=24458,4=24431} 2024-04-25T11:34:51.932 INFO:teuthology.orchestra.run.smithi090.stdout:failed 2024-04-25T11:34:51.932 INFO:teuthology.orchestra.run.smithi090.stdout:damaged 2024-04-25T11:34:51.932 INFO:teuthology.orchestra.run.smithi090.stdout:stopped 2024-04-25T11:34:51.932 INFO:teuthology.orchestra.run.smithi090.stdout:data_pools [3] 2024-04-25T11:34:51.932 INFO:teuthology.orchestra.run.smithi090.stdout:metadata_pool 2 2024-04-25T11:34:51.933 INFO:teuthology.orchestra.run.smithi090.stdout:inline_data disabled 2024-04-25T11:34:51.933 INFO:teuthology.orchestra.run.smithi090.stdout:balancer 2024-04-25T11:34:51.933 INFO:teuthology.orchestra.run.smithi090.stdout:bal_rank_mask -1 2024-04-25T11:34:51.933 INFO:teuthology.orchestra.run.smithi090.stdout:standby_count_wanted 0 2024-04-25T11:34:51.933 INFO:teuthology.orchestra.run.smithi090.stdout:qdb_cluster leader: 24524 members: 24524,24475,24499,24458,24431 2024-04-25T11:34:51.933 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.b{0:24524} state up:active seq 3 addr [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T11:34:51.933 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.h{0:24485} state up:standby-replay seq 1 addr [v2:172.21.15.138:6836/2598829089,v1:172.21.15.138:6837/2598829089] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T11:34:51.933 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.i{1:24499} state up:active seq 5 addr [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T11:34:51.933 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.c{1:24481} state up:standby-replay seq 1 addr [v2:172.21.15.156:6836/3466257977,v1:172.21.15.156:6837/3466257977] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T11:34:51.933 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.f{2:24475} state up:active seq 8 addr [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T11:34:51.933 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.l{2:24430} state up:standby-replay seq 1 addr [v2:172.21.15.156:6832/31237375,v1:172.21.15.156:6833/31237375] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T11:34:51.933 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.e{3:24458} state up:active seq 9 addr [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T11:34:51.933 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.a{3:14649} state up:standby-replay seq 1 addr [v2:172.21.15.90:6840/2617461073,v1:172.21.15.90:6841/2617461073] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T11:34:51.933 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.k{4:24431} state up:active seq 11 addr [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T11:34:51.934 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.g{4:14637} state up:standby-replay seq 1 addr [v2:172.21.15.90:6838/3101160958,v1:172.21.15.90:6839/3101160958] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T11:34:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:34:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:34:51.934 INFO:teuthology.orchestra.run.smithi090.stdout:Standby daemons: 2024-04-25T11:34:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:34:51.934 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.j{-1:14556} state up:standby seq 1 addr [v2:172.21.15.90:6834/4061411806,v1:172.21.15.90:6835/4061411806] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T11:34:51.934 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.d{-1:14574} state up:standby seq 1 addr [v2:172.21.15.90:6836/3050267616,v1:172.21.15.90:6837/3050267616] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T11:34:51.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.943+0000 7f0a416a1700 1 -- 172.21.15.90:0/2284219452 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0a280afb80 msgr2=0x7f0a280b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:51.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.943+0000 7f0a416a1700 1 --2- 172.21.15.90:0/2284219452 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0a280afb80 0x7f0a280b2090 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c19b820 tx=0x7f0a2400b410 comp rx=0 tx=0).stop 2024-04-25T11:34:51.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.943+0000 7f0a416a1700 1 -- 172.21.15.90:0/2284219452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a3c102200 msgr2=0x7f0a3c19a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:51.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.943+0000 7f0a416a1700 1 --2- 172.21.15.90:0/2284219452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a3c102200 0x7f0a3c19a770 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c000c00 tx=0x7f0a2c0047b0 comp rx=0 tx=0).stop 2024-04-25T11:34:51.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.943+0000 7f0a416a1700 1 -- 172.21.15.90:0/2284219452 shutdown_connections 2024-04-25T11:34:51.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.943+0000 7f0a416a1700 1 --2- 172.21.15.90:0/2284219452 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f0a280afb80 0x7f0a280b2090 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:51.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.943+0000 7f0a416a1700 1 --2- 172.21.15.90:0/2284219452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a3c1057b0 0x7f0a3c1a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:51.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.943+0000 7f0a416a1700 1 --2- 172.21.15.90:0/2284219452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a3c102200 0x7f0a3c19a770 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:51.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.943+0000 7f0a416a1700 1 --2- 172.21.15.90:0/2284219452 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0a3c102b50 0x7f0a3c19acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:51.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.943+0000 7f0a416a1700 1 -- 172.21.15.90:0/2284219452 >> 172.21.15.90:0/2284219452 conn(0x7f0a3c0fded0 msgr2=0x7f0a3c108750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:51.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.944+0000 7f0a416a1700 1 -- 172.21.15.90:0/2284219452 shutdown_connections 2024-04-25T11:34:51.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:51.944+0000 7f0a416a1700 1 -- 172.21.15.90:0/2284219452 wait complete. 2024-04-25T11:34:51.936 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 25 2024-04-25T11:34:52.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:51 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2284219452' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T11:34:52.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:51 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2284219452' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T11:34:52.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:51 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2284219452' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T11:34:52.736 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-25T11:34:52.981 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:34:53.292 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:53 smithi090 ceph-mon[29575]: pgmap v327: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-25T11:34:53.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:53 smithi138 ceph-mon[32328]: pgmap v327: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-25T11:34:53.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:53 smithi156 ceph-mon[33045]: pgmap v327: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-25T11:34:54.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.602+0000 7f19f0cde700 1 -- 172.21.15.90:0/3168692331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19ec1036e0 msgr2=0x7f19ec103af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:54.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.602+0000 7f19f0cde700 1 --2- 172.21.15.90:0/3168692331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19ec1036e0 0x7f19ec103af0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f19dc0099c0 tx=0x7f19dc009cd0 comp rx=0 tx=0).stop 2024-04-25T11:34:54.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.602+0000 7f19f0cde700 1 -- 172.21.15.90:0/3168692331 shutdown_connections 2024-04-25T11:34:54.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.602+0000 7f19f0cde700 1 --2- 172.21.15.90:0/3168692331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19ec104f30 0x7f19ec10b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:54.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.602+0000 7f19f0cde700 1 --2- 172.21.15.90:0/3168692331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19ec1036e0 0x7f19ec103af0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:54.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.602+0000 7f19f0cde700 1 --2- 172.21.15.90:0/3168692331 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f19ec1045e0 0x7f19ec1049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:54.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.602+0000 7f19f0cde700 1 -- 172.21.15.90:0/3168692331 >> 172.21.15.90:0/3168692331 conn(0x7f19ec0fded0 msgr2=0x7f19ec100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:54.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.602+0000 7f19f0cde700 1 -- 172.21.15.90:0/3168692331 shutdown_connections 2024-04-25T11:34:54.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.602+0000 7f19f0cde700 1 -- 172.21.15.90:0/3168692331 wait complete. 2024-04-25T11:34:54.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.603+0000 7f19f0cde700 1 Processor -- start 2024-04-25T11:34:54.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.603+0000 7f19f0cde700 1 -- start start 2024-04-25T11:34:54.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.603+0000 7f19f0cde700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19ec1036e0 0x7f19ec102da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:54.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.603+0000 7f19f0cde700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19ec1045e0 0x7f19ec19f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:54.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.603+0000 7f19f0cde700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f19ec104f30 0x7f19ec1a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:54.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.603+0000 7f19f0cde700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f19ec10edb0 con 0x7f19ec1045e0 2024-04-25T11:34:54.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.603+0000 7f19f0cde700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f19ec10ec30 con 0x7f19ec104f30 2024-04-25T11:34:54.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.603+0000 7f19f0cde700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f19ec10ef30 con 0x7f19ec1036e0 2024-04-25T11:34:54.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.603+0000 7f19e9d9b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19ec1045e0 0x7f19ec19f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:54.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.603+0000 7f19e9d9b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19ec1045e0 0x7f19ec19f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57616/0 (socket says 172.21.15.90:57616) 2024-04-25T11:34:54.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.603+0000 7f19e9d9b700 1 -- 172.21.15.90:0/934949764 learned_addr learned my addr 172.21.15.90:0/934949764 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:54.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.603+0000 7f19ea59c700 1 --2- 172.21.15.90:0/934949764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19ec1036e0 0x7f19ec102da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:54.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.603+0000 7f19ead9d700 1 --2- 172.21.15.90:0/934949764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f19ec104f30 0x7f19ec1a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:54.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.603+0000 7f19e9d9b700 1 -- 172.21.15.90:0/934949764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19ec1036e0 msgr2=0x7f19ec102da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:54.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.603+0000 7f19e9d9b700 1 --2- 172.21.15.90:0/934949764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19ec1036e0 0x7f19ec102da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:54.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.603+0000 7f19e9d9b700 1 -- 172.21.15.90:0/934949764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f19ec104f30 msgr2=0x7f19ec1a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:54.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.603+0000 7f19e9d9b700 1 --2- 172.21.15.90:0/934949764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f19ec104f30 0x7f19ec1a1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:54.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.603+0000 7f19e9d9b700 1 -- 172.21.15.90:0/934949764 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19ec1a2520 con 0x7f19ec1045e0 2024-04-25T11:34:54.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.604+0000 7f19e9d9b700 1 --2- 172.21.15.90:0/934949764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19ec1045e0 0x7f19ec19f780 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f19e000e9a0 tx=0x7f19e000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:54.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.604+0000 7f19db7fe700 1 -- 172.21.15.90:0/934949764 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e000ca10 con 0x7f19ec1045e0 2024-04-25T11:34:54.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.604+0000 7f19db7fe700 1 -- 172.21.15.90:0/934949764 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19e000cbb0 con 0x7f19ec1045e0 2024-04-25T11:34:54.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.604+0000 7f19f0cde700 1 -- 172.21.15.90:0/934949764 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19ec1a2810 con 0x7f19ec1045e0 2024-04-25T11:34:54.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.604+0000 7f19db7fe700 1 -- 172.21.15.90:0/934949764 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e0018680 con 0x7f19ec1045e0 2024-04-25T11:34:54.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.604+0000 7f19f0cde700 1 -- 172.21.15.90:0/934949764 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19ec1a2da0 con 0x7f19ec1045e0 2024-04-25T11:34:54.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.605+0000 7f19f0cde700 1 -- 172.21.15.90:0/934949764 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19ec051190 con 0x7f19ec1045e0 2024-04-25T11:34:54.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.606+0000 7f19db7fe700 1 -- 172.21.15.90:0/934949764 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f19e0010430 con 0x7f19ec1045e0 2024-04-25T11:34:54.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.607+0000 7f19db7fe700 1 --2- 172.21.15.90:0/934949764 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f19d40afc00 0x7f19d40b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:54.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.607+0000 7f19ea59c700 1 --2- 172.21.15.90:0/934949764 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f19d40afc00 0x7f19d40b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:54.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.607+0000 7f19db7fe700 1 -- 172.21.15.90:0/934949764 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f19e00defc0 con 0x7f19ec1045e0 2024-04-25T11:34:54.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.607+0000 7f19ea59c700 1 --2- 172.21.15.90:0/934949764 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f19d40afc00 0x7f19d40b2110 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f19dc000c00 tx=0x7f19dc01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:54.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.611+0000 7f19db7fe700 1 -- 172.21.15.90:0/934949764 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f19e00a7c20 con 0x7f19ec1045e0 2024-04-25T11:34:54.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.811+0000 7f19f0cde700 1 -- 172.21.15.90:0/934949764 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f19ec02e510 con 0x7f19ec1045e0 2024-04-25T11:34:54.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.811+0000 7f19db7fe700 1 -- 172.21.15.90:0/934949764 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v79) ==== 56+0+5335 (secure 0 0 0) 0x7f19e00a7520 con 0x7f19ec1045e0 2024-04-25T11:34:54.801 INFO:teuthology.orchestra.run.smithi090.stdout:epoch 79 2024-04-25T11:34:54.801 INFO:teuthology.orchestra.run.smithi090.stdout:fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:34:54.801 INFO:teuthology.orchestra.run.smithi090.stdout:created 2024-04-25T11:25:23.574550+0000 2024-04-25T11:34:54.801 INFO:teuthology.orchestra.run.smithi090.stdout:modified 2024-04-25T11:34:17.871834+0000 2024-04-25T11:34:54.801 INFO:teuthology.orchestra.run.smithi090.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-25T11:34:54.801 INFO:teuthology.orchestra.run.smithi090.stdout:crush_version 26 2024-04-25T11:34:54.801 INFO:teuthology.orchestra.run.smithi090.stdout:full_ratio 0.95 2024-04-25T11:34:54.801 INFO:teuthology.orchestra.run.smithi090.stdout:backfillfull_ratio 0.9 2024-04-25T11:34:54.801 INFO:teuthology.orchestra.run.smithi090.stdout:nearfull_ratio 0.85 2024-04-25T11:34:54.801 INFO:teuthology.orchestra.run.smithi090.stdout:require_min_compat_client luminous 2024-04-25T11:34:54.801 INFO:teuthology.orchestra.run.smithi090.stdout:min_compat_client luminous 2024-04-25T11:34:54.801 INFO:teuthology.orchestra.run.smithi090.stdout:require_osd_release squid 2024-04-25T11:34:54.801 INFO:teuthology.orchestra.run.smithi090.stdout:stretch_mode_enabled false 2024-04-25T11:34:54.801 INFO:teuthology.orchestra.run.smithi090.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-25T11:34:54.802 INFO:teuthology.orchestra.run.smithi090.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 76 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-04-25T11:34:54.802 INFO:teuthology.orchestra.run.smithi090.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 77 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-25T11:34:54.802 INFO:teuthology.orchestra.run.smithi090.stdout:max_osd 12 2024-04-25T11:34:54.802 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0 up in weight 1 up_from 8 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6802/1201629159,v1:172.21.15.90:6803/1201629159] [v2:172.21.15.90:6804/1201629159,v1:172.21.15.90:6805/1201629159] exists,up 5e54d888-e67b-4951-8ae7-e355fb6b10a8 2024-04-25T11:34:54.802 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1 up in weight 1 up_from 13 up_thru 75 down_at 0 last_clean_interval [0,0) [v2:172.21.15.138:6800/3976851933,v1:172.21.15.138:6801/3976851933] [v2:172.21.15.138:6802/3976851933,v1:172.21.15.138:6803/3976851933] exists,up 91a71c78-dae9-4bc7-9f6f-9e55cfefbe28 2024-04-25T11:34:54.802 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2 up in weight 1 up_from 18 up_thru 75 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6800/1505807670,v1:172.21.15.156:6801/1505807670] [v2:172.21.15.156:6802/1505807670,v1:172.21.15.156:6803/1505807670] exists,up daff6c7f-80fa-4859-9339-ef7a5e840832 2024-04-25T11:34:54.802 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3 up in weight 1 up_from 25 up_thru 75 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6810/2520012383,v1:172.21.15.90:6811/2520012383] [v2:172.21.15.90:6812/2520012383,v1:172.21.15.90:6813/2520012383] exists,up 33fd94ca-7465-401a-bc1e-706b6b31b9b0 2024-04-25T11:34:54.802 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4 up in weight 1 up_from 31 up_thru 75 down_at 0 last_clean_interval [0,0) [v2:172.21.15.138:6808/3752678279,v1:172.21.15.138:6809/3752678279] [v2:172.21.15.138:6810/3752678279,v1:172.21.15.138:6811/3752678279] exists,up c5326212-750c-49fd-96e3-d1e62bb4b7f5 2024-04-25T11:34:54.802 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5 up in weight 1 up_from 36 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6808/3591210191,v1:172.21.15.156:6809/3591210191] [v2:172.21.15.156:6810/3591210191,v1:172.21.15.156:6811/3591210191] exists,up d04a1156-820b-4b43-be89-e16a8910c3aa 2024-04-25T11:34:54.802 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6 up in weight 1 up_from 42 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6818/2364955456,v1:172.21.15.90:6819/2364955456] [v2:172.21.15.90:6820/2364955456,v1:172.21.15.90:6821/2364955456] exists,up 75675aa3-e978-490c-8205-13d9868a3d6d 2024-04-25T11:34:54.802 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7 up in weight 1 up_from 48 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.138:6816/4012476387,v1:172.21.15.138:6817/4012476387] [v2:172.21.15.138:6818/4012476387,v1:172.21.15.138:6819/4012476387] exists,up 1a95fdf5-f548-4271-bbcd-e330f7955eaa 2024-04-25T11:34:54.802 INFO:teuthology.orchestra.run.smithi090.stdout:osd.8 up in weight 1 up_from 54 up_thru 75 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6816/101092224,v1:172.21.15.156:6817/101092224] [v2:172.21.15.156:6818/101092224,v1:172.21.15.156:6819/101092224] exists,up 605e10b5-5806-4c1f-8e28-a7994d34ddce 2024-04-25T11:34:54.802 INFO:teuthology.orchestra.run.smithi090.stdout:osd.9 up in weight 1 up_from 60 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6826/1171265966,v1:172.21.15.90:6827/1171265966] [v2:172.21.15.90:6828/1171265966,v1:172.21.15.90:6829/1171265966] exists,up d70fdf5e-9d0f-483d-a7d8-505b79b8a418 2024-04-25T11:34:54.803 INFO:teuthology.orchestra.run.smithi090.stdout:osd.10 up in weight 1 up_from 66 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.138:6824/4264666232,v1:172.21.15.138:6825/4264666232] [v2:172.21.15.138:6826/4264666232,v1:172.21.15.138:6827/4264666232] exists,up 1ec86c7d-c3a4-432d-9664-8e40836e1e20 2024-04-25T11:34:54.803 INFO:teuthology.orchestra.run.smithi090.stdout:osd.11 up in weight 1 up_from 72 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6824/1224251744,v1:172.21.15.156:6825/1224251744] [v2:172.21.15.156:6826/1224251744,v1:172.21.15.156:6827/1224251744] exists,up 51cbe729-292e-4d7b-b8e9-e1e2821ffe76 2024-04-25T11:34:54.803 INFO:teuthology.orchestra.run.smithi090.stdout:pg_upmap_items 2.14 [2,4] 2024-04-25T11:34:54.803 INFO:teuthology.orchestra.run.smithi090.stdout:pg_upmap_items 3.0 [4,7] 2024-04-25T11:34:54.803 INFO:teuthology.orchestra.run.smithi090.stdout:pg_upmap_items 3.6 [4,6] 2024-04-25T11:34:54.803 INFO:teuthology.orchestra.run.smithi090.stdout:pg_upmap_items 3.b [3,9] 2024-04-25T11:34:54.803 INFO:teuthology.orchestra.run.smithi090.stdout:pg_upmap_items 3.e [4,7] 2024-04-25T11:34:54.803 INFO:teuthology.orchestra.run.smithi090.stdout:pg_upmap_items 3.11 [4,8] 2024-04-25T11:34:54.803 INFO:teuthology.orchestra.run.smithi090.stdout:pg_upmap_items 3.1e [3,9] 2024-04-25T11:34:54.803 INFO:teuthology.orchestra.run.smithi090.stdout:pg_upmap_items 3.28 [5,6] 2024-04-25T11:34:54.803 INFO:teuthology.orchestra.run.smithi090.stdout:pg_upmap_items 3.37 [1,9] 2024-04-25T11:34:54.803 INFO:teuthology.orchestra.run.smithi090.stdout:pg_upmap_items 3.39 [5,8] 2024-04-25T11:34:54.803 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:0/2381633953 expires 2024-04-26T11:26:16.655278+0000 2024-04-25T11:34:54.803 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:6800/71478430 expires 2024-04-26T11:26:16.655278+0000 2024-04-25T11:34:54.804 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:0/2190846900 expires 2024-04-26T11:25:50.322011+0000 2024-04-25T11:34:54.804 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:0/1707792941 expires 2024-04-26T11:25:50.322011+0000 2024-04-25T11:34:54.804 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:6800/3122821311 expires 2024-04-26T11:25:50.322011+0000 2024-04-25T11:34:54.804 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:6801/71478430 expires 2024-04-26T11:26:16.655278+0000 2024-04-25T11:34:54.804 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:0/6061052 expires 2024-04-26T11:25:50.322011+0000 2024-04-25T11:34:54.804 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:0/1849633392 expires 2024-04-26T11:26:16.655278+0000 2024-04-25T11:34:54.804 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:6801/3122821311 expires 2024-04-26T11:25:50.322011+0000 2024-04-25T11:34:54.804 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:0/2277490702 expires 2024-04-26T11:26:16.655278+0000 2024-04-25T11:34:54.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.814+0000 7f19f0cde700 1 -- 172.21.15.90:0/934949764 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f19d40afc00 msgr2=0x7f19d40b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:54.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.814+0000 7f19f0cde700 1 --2- 172.21.15.90:0/934949764 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f19d40afc00 0x7f19d40b2110 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f19dc000c00 tx=0x7f19dc01a040 comp rx=0 tx=0).stop 2024-04-25T11:34:54.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.814+0000 7f19f0cde700 1 -- 172.21.15.90:0/934949764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19ec1045e0 msgr2=0x7f19ec19f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:54.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.814+0000 7f19f0cde700 1 --2- 172.21.15.90:0/934949764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19ec1045e0 0x7f19ec19f780 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f19e000e9a0 tx=0x7f19e000ecb0 comp rx=0 tx=0).stop 2024-04-25T11:34:54.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.814+0000 7f19f0cde700 1 -- 172.21.15.90:0/934949764 shutdown_connections 2024-04-25T11:34:54.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.814+0000 7f19f0cde700 1 --2- 172.21.15.90:0/934949764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f19ec104f30 0x7f19ec1a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:54.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.814+0000 7f19f0cde700 1 --2- 172.21.15.90:0/934949764 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f19d40afc00 0x7f19d40b2110 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:54.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.814+0000 7f19f0cde700 1 --2- 172.21.15.90:0/934949764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19ec1036e0 0x7f19ec102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:54.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.814+0000 7f19f0cde700 1 --2- 172.21.15.90:0/934949764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19ec1045e0 0x7f19ec19f780 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:54.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.814+0000 7f19f0cde700 1 -- 172.21.15.90:0/934949764 >> 172.21.15.90:0/934949764 conn(0x7f19ec0fded0 msgr2=0x7f19ec107810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:54.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.814+0000 7f19f0cde700 1 -- 172.21.15.90:0/934949764 shutdown_connections 2024-04-25T11:34:54.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:54.814+0000 7f19f0cde700 1 -- 172.21.15.90:0/934949764 wait complete. 2024-04-25T11:34:55.211 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:54 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/934949764' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T11:34:55.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:54 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/934949764' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T11:34:55.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:54 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/934949764' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T11:34:55.510 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-25T11:34:55.522 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:34:55.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.644+0000 7f6b24d5f640 1 -- 172.21.15.90:0/383151829 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b20103c20 msgr2=0x7f6b20104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:55.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.644+0000 7f6b24d5f640 1 --2- 172.21.15.90:0/383151829 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b20103c20 0x7f6b20104030 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6b14009900 tx=0x7f6b1402a410 comp rx=0 tx=0).stop 2024-04-25T11:34:55.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.645+0000 7f6b24d5f640 1 -- 172.21.15.90:0/383151829 shutdown_connections 2024-04-25T11:34:55.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.645+0000 7f6b24d5f640 1 --2- 172.21.15.90:0/383151829 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b20108920 0x7f6b2010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:55.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.645+0000 7f6b24d5f640 1 --2- 172.21.15.90:0/383151829 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b20103c20 0x7f6b20104030 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:55.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.645+0000 7f6b24d5f640 1 --2- 172.21.15.90:0/383151829 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b201029e0 0x7f6b201036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:55.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.645+0000 7f6b24d5f640 1 -- 172.21.15.90:0/383151829 >> 172.21.15.90:0/383151829 conn(0x7f6b200fdbd0 msgr2=0x7f6b20100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:55.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.645+0000 7f6b24d5f640 1 -- 172.21.15.90:0/383151829 shutdown_connections 2024-04-25T11:34:55.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.645+0000 7f6b24d5f640 1 -- 172.21.15.90:0/383151829 wait complete. 2024-04-25T11:34:55.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.645+0000 7f6b24d5f640 1 Processor -- start 2024-04-25T11:34:55.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.645+0000 7f6b24d5f640 1 -- start start 2024-04-25T11:34:55.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.646+0000 7f6b24d5f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b201029e0 0x7f6b20075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:55.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.646+0000 7f6b24d5f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b20103c20 0x7f6b2019f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:55.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.646+0000 7f6b24d5f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b20108920 0x7f6b201a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:55.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.646+0000 7f6b24d5f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6b2010e630 con 0x7f6b201029e0 2024-04-25T11:34:55.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.646+0000 7f6b24d5f640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6b2010e4b0 con 0x7f6b20103c20 2024-04-25T11:34:55.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.646+0000 7f6b24d5f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6b2010e7b0 con 0x7f6b20108920 2024-04-25T11:34:55.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.646+0000 7f6b1e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b201029e0 0x7f6b20075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:55.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.646+0000 7f6b1e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b201029e0 0x7f6b20075840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57632/0 (socket says 172.21.15.90:57632) 2024-04-25T11:34:55.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.646+0000 7f6b1e59c640 1 -- 172.21.15.90:0/1385358998 learned_addr learned my addr 172.21.15.90:0/1385358998 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:55.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.646+0000 7f6b1ed9d640 1 --2- 172.21.15.90:0/1385358998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b20108920 0x7f6b201a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:55.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.646+0000 7f6b1e59c640 1 -- 172.21.15.90:0/1385358998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b20108920 msgr2=0x7f6b201a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:55.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.646+0000 7f6b1e59c640 1 --2- 172.21.15.90:0/1385358998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b20108920 0x7f6b201a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:55.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.646+0000 7f6b1e59c640 1 -- 172.21.15.90:0/1385358998 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b20103c20 msgr2=0x7f6b2019f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:55.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.646+0000 7f6b1dd9b640 1 --2- 172.21.15.90:0/1385358998 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b20103c20 0x7f6b2019f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:55.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.646+0000 7f6b1e59c640 1 --2- 172.21.15.90:0/1385358998 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b20103c20 0x7f6b2019f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:55.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.646+0000 7f6b1e59c640 1 -- 172.21.15.90:0/1385358998 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b201a1f10 con 0x7f6b201029e0 2024-04-25T11:34:55.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.646+0000 7f6b1e59c640 1 --2- 172.21.15.90:0/1385358998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b201029e0 0x7f6b20075840 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6b10006fd0 tx=0x7f6b1000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:55.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.646+0000 7f6b0f7fe640 1 -- 172.21.15.90:0/1385358998 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b10011830 con 0x7f6b201029e0 2024-04-25T11:34:55.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.646+0000 7f6b0f7fe640 1 -- 172.21.15.90:0/1385358998 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b10016070 con 0x7f6b201029e0 2024-04-25T11:34:55.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.646+0000 7f6b0f7fe640 1 -- 172.21.15.90:0/1385358998 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b10010e00 con 0x7f6b201029e0 2024-04-25T11:34:55.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.646+0000 7f6b24d5f640 1 -- 172.21.15.90:0/1385358998 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b201a2200 con 0x7f6b201029e0 2024-04-25T11:34:55.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.646+0000 7f6b24d5f640 1 -- 172.21.15.90:0/1385358998 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b201a2780 con 0x7f6b201029e0 2024-04-25T11:34:55.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.647+0000 7f6b24d5f640 1 -- 172.21.15.90:0/1385358998 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b20107730 con 0x7f6b201029e0 2024-04-25T11:34:55.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.649+0000 7f6b0f7fe640 1 -- 172.21.15.90:0/1385358998 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6b100119d0 con 0x7f6b201029e0 2024-04-25T11:34:55.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.649+0000 7f6b0f7fe640 1 --2- 172.21.15.90:0/1385358998 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6b080afaf0 0x7f6b080b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:55.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.649+0000 7f6b1dd9b640 1 --2- 172.21.15.90:0/1385358998 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6b080afaf0 0x7f6b080b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:55.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.649+0000 7f6b0f7fe640 1 -- 172.21.15.90:0/1385358998 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f6b100de940 con 0x7f6b201029e0 2024-04-25T11:34:55.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.650+0000 7f6b1dd9b640 1 --2- 172.21.15.90:0/1385358998 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6b080afaf0 0x7f6b080b2000 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6b140096c0 tx=0x7f6b14005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:55.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.653+0000 7f6b0f7fe640 1 -- 172.21.15.90:0/1385358998 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6b100a7500 con 0x7f6b201029e0 2024-04-25T11:34:55.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.834+0000 7f6b24d5f640 1 -- 172.21.15.90:0/1385358998 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6b2002dd20 con 0x7f6b201029e0 2024-04-25T11:34:55.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.835+0000 7f6b0f7fe640 1 -- 172.21.15.90:0/1385358998 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+22706 (secure 0 0 0) 0x7f6b100a6e00 con 0x7f6b201029e0 2024-04-25T11:34:55.825 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:34:55.825 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":79,"fsid":"58bd4e7a-02f6-11ef-bc93-c7b262605968","created":"2024-04-25T11:25:23.574550+0000","modified":"2024-04-25T11:34:17.871834+0000","last_up_change":"2024-04-25T11:32:23.852211+0000","last_in_change":"2024-04-25T11:32:04.188507+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:28:42.746649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:33:08.442925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:33:09.352886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e54d888-e67b-4951-8ae7-e355fb6b10a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6803","nonce":1201629159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6805","nonce":1201629159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6809","nonce":1201629159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6807","nonce":1201629159}]},"public_addr":"172.21.15.90:6803/1201629159","cluster_addr":"172.21.15.90:6805/1201629159","heartbeat_back_addr":"172.21.15.90:6809/1201629159","heartbeat_front_addr":"172.21.15.90:6807/1201629159","state":["exists","up"]},{"osd":1,"uuid":"91a71c78-dae9-4bc7-9f6f-9e55cfefbe28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6801","nonce":3976851933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6803","nonce":3976851933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6807","nonce":3976851933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6805","nonce":3976851933}]},"public_addr":"172.21.15.138:6801/3976851933","cluster_addr":"172.21.15.138:6803/3976851933","heartbeat_back_addr":"172.21.15.138:6807/3976851933","heartbeat_front_addr":"172.21.15.138:6805/3976851933","state":["exists","up"]},{"osd":2,"uuid":"daff6c7f-80fa-4859-9339-ef7a5e840832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6801","nonce":1505807670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6803","nonce":1505807670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6807","nonce":1505807670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6805","nonce":1505807670}]},"public_addr":"172.21.15.156:6801/1505807670","cluster_addr":"172.21.15.156:6803/1505807670","heartbeat_back_addr":"172.21.15.156:6807/1505807670","heartbeat_front_addr":"172.21.15.156:6805/1505807670","state":["exists","up"]},{"osd":3,"uuid":"33fd94ca-7465-401a-bc1e-706b6b31b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6811","nonce":2520012383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6813","nonce":2520012383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6817","nonce":2520012383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6815","nonce":2520012383}]},"public_addr":"172.21.15.90:6811/2520012383","cluster_addr":"172.21.15.90:6813/2520012383","heartbeat_back_addr":"172.21.15.90:6817/2520012383","heartbeat_front_addr":"172.21.15.90:6815/2520012383","state":["exists","up"]},{"osd":4,"uuid":"c5326212-750c-49fd-96e3-d1e62bb4b7f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6809","nonce":3752678279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6811","nonce":3752678279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6815","nonce":3752678279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6813","nonce":3752678279}]},"public_addr":"172.21.15.138:6809/3752678279","cluster_addr":"172.21.15.138:6811/3752678279","heartbeat_back_addr":"172.21.15.138:6815/3752678279","heartbeat_front_addr":"172.21.15.138:6813/3752678279","state":["exists","up"]},{"osd":5,"uuid":"d04a1156-820b-4b43-be89-e16a8910c3aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6809","nonce":3591210191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6811","nonce":3591210191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6815","nonce":3591210191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6813","nonce":3591210191}]},"public_addr":"172.21.15.156:6809/3591210191","cluster_addr":"172.21.15.156:6811/3591210191","heartbeat_back_addr":"172.21.15.156:6815/3591210191","heartbeat_front_addr":"172.21.15.156:6813/3591210191","state":["exists","up"]},{"osd":6,"uuid":"75675aa3-e978-490c-8205-13d9868a3d6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6819","nonce":2364955456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6821","nonce":2364955456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6825","nonce":2364955456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6823","nonce":2364955456}]},"public_addr":"172.21.15.90:6819/2364955456","cluster_addr":"172.21.15.90:6821/2364955456","heartbeat_back_addr":"172.21.15.90:6825/2364955456","heartbeat_front_addr":"172.21.15.90:6823/2364955456","state":["exists","up"]},{"osd":7,"uuid":"1a95fdf5-f548-4271-bbcd-e330f7955eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6817","nonce":4012476387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6819","nonce":4012476387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6823","nonce":4012476387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6821","nonce":4012476387}]},"public_addr":"172.21.15.138:6817/4012476387","cluster_addr":"172.21.15.138:6819/4012476387","heartbeat_back_addr":"172.21.15.138:6823/4012476387","heartbeat_front_addr":"172.21.15.138:6821/4012476387","state":["exists","up"]},{"osd":8,"uuid":"605e10b5-5806-4c1f-8e28-a7994d34ddce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6817","nonce":101092224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6819","nonce":101092224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6823","nonce":101092224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6821","nonce":101092224}]},"public_addr":"172.21.15.156:6817/101092224","cluster_addr":"172.21.15.156:6819/101092224","heartbeat_back_addr":"172.21.15.156:6823/101092224","heartbeat_front_addr":"172.21.15.156:6821/101092224","state":["exists","up"]},{"osd":9,"uuid":"d70fdf5e-9d0f-483d-a7d8-505b79b8a418","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6827","nonce":1171265966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6829","nonce":1171265966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6833","nonce":1171265966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6831","nonce":1171265966}]},"public_addr":"172.21.15.90:6827/1171265966","cluster_addr":"172.21.15.90:6829/1171265966","heartbeat_back_addr":"172.21.15.90:6833/1171265966","heartbeat_front_addr":"172.21.15.90:6831/1171265966","state":["exists","up"]},{"osd":10,"uuid":"1ec86c7d-c3a4-432d-9664-8e40836e1e20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6825","nonce":4264666232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6827","nonce":4264666232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6831","nonce":4264666232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6829","nonce":4264666232}]},"public_addr":"172.21.15.138:6825/4264666232","cluster_addr":"172.21.15.138:6827/4264666232","heartbeat_back_addr":"172.21.15.138:6831/4264666232","heartbeat_front_addr":"172.21.15.138:6829/4264666232","state":["exists","up"]},{"osd":11,"uuid":"51cbe729-292e-4d7b-b8e9-e1e2821ffe76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6825","nonce":1224251744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6827","nonce":1224251744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6831","nonce":1224251744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6829","nonce":1224251744}]},"public_addr":"172.21.15.156:6825/1224251744","cluster_addr":"172.21.15.156:6827/1224251744","heartbeat_back_addr":"172.21.15.156:6831/1224251744","heartbeat_front_addr":"172.21.15.156:6829/1224251744","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:27:50.794226+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:15.030278+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:38.518163+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:03.107351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:27.500707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:51.463545+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:16.015076+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:41.182071+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:05.362309+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:31.048936+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:56.488509+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:32:22.129178+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":7}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/2381633953":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6800/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/2190846900":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1707792941":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6800/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6801/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/6061052":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1849633392":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6801/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/2277490702":"2024-04-26T11:26:16.655278+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:34:55.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.837+0000 7f6b24d5f640 1 -- 172.21.15.90:0/1385358998 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6b080afaf0 msgr2=0x7f6b080b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:55.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.837+0000 7f6b24d5f640 1 --2- 172.21.15.90:0/1385358998 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6b080afaf0 0x7f6b080b2000 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6b140096c0 tx=0x7f6b14005fd0 comp rx=0 tx=0).stop 2024-04-25T11:34:55.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.837+0000 7f6b24d5f640 1 -- 172.21.15.90:0/1385358998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b201029e0 msgr2=0x7f6b20075840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:55.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.837+0000 7f6b24d5f640 1 --2- 172.21.15.90:0/1385358998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b201029e0 0x7f6b20075840 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6b10006fd0 tx=0x7f6b1000b8e0 comp rx=0 tx=0).stop 2024-04-25T11:34:55.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.837+0000 7f6b24d5f640 1 -- 172.21.15.90:0/1385358998 shutdown_connections 2024-04-25T11:34:55.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.837+0000 7f6b24d5f640 1 --2- 172.21.15.90:0/1385358998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b20108920 0x7f6b201a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:55.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.837+0000 7f6b24d5f640 1 --2- 172.21.15.90:0/1385358998 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b20103c20 0x7f6b2019f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:55.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.837+0000 7f6b24d5f640 1 --2- 172.21.15.90:0/1385358998 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6b080afaf0 0x7f6b080b2000 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:55.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.837+0000 7f6b24d5f640 1 --2- 172.21.15.90:0/1385358998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b201029e0 0x7f6b20075840 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:55.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.837+0000 7f6b24d5f640 1 -- 172.21.15.90:0/1385358998 >> 172.21.15.90:0/1385358998 conn(0x7f6b200fdbd0 msgr2=0x7f6b20104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:55.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.838+0000 7f6b24d5f640 1 -- 172.21.15.90:0/1385358998 shutdown_connections 2024-04-25T11:34:55.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.838+0000 7f6b24d5f640 1 -- 172.21.15.90:0/1385358998 wait complete. 2024-04-25T11:34:55.839 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T11:28:42.746649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T11:33:08.442925+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T11:33:09.352886+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T11:34:55.839 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T11:34:55.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.961+0000 7f39c414d640 1 -- 172.21.15.90:0/3437716802 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39bc109a30 msgr2=0x7f39bc10be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:55.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.961+0000 7f39c414d640 1 --2- 172.21.15.90:0/3437716802 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39bc109a30 0x7f39bc10be50 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f39b4009ca0 tx=0x7f39b402a8e0 comp rx=0 tx=0).stop 2024-04-25T11:34:55.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.961+0000 7f39c414d640 1 -- 172.21.15.90:0/3437716802 shutdown_connections 2024-04-25T11:34:55.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.961+0000 7f39c414d640 1 --2- 172.21.15.90:0/3437716802 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39bc109a30 0x7f39bc10be50 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:55.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.961+0000 7f39c414d640 1 --2- 172.21.15.90:0/3437716802 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39bc104d30 0x7f39bc105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:55.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.961+0000 7f39c414d640 1 --2- 172.21.15.90:0/3437716802 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39bc103af0 0x7f39bc1047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:55.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.961+0000 7f39c414d640 1 -- 172.21.15.90:0/3437716802 >> 172.21.15.90:0/3437716802 conn(0x7f39bc0fece0 msgr2=0x7f39bc101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:55.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.961+0000 7f39c414d640 1 -- 172.21.15.90:0/3437716802 shutdown_connections 2024-04-25T11:34:55.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.961+0000 7f39c414d640 1 -- 172.21.15.90:0/3437716802 wait complete. 2024-04-25T11:34:55.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.962+0000 7f39c414d640 1 Processor -- start 2024-04-25T11:34:55.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.962+0000 7f39c414d640 1 -- start start 2024-04-25T11:34:55.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.962+0000 7f39c414d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39bc103af0 0x7f39bc19a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:55.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.962+0000 7f39c414d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39bc104d30 0x7f39bc19a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:55.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.962+0000 7f39c414d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39bc109a30 0x7f39bc1a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:55.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.962+0000 7f39c414d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f39bc10f670 con 0x7f39bc103af0 2024-04-25T11:34:55.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.962+0000 7f39c414d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f39bc10f4f0 con 0x7f39bc104d30 2024-04-25T11:34:55.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.962+0000 7f39c414d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f39bc10f7f0 con 0x7f39bc109a30 2024-04-25T11:34:55.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.962+0000 7f39c1ee9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39bc103af0 0x7f39bc19a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:55.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.962+0000 7f39c1ee9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39bc103af0 0x7f39bc19a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57652/0 (socket says 172.21.15.90:57652) 2024-04-25T11:34:55.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.962+0000 7f39c1ee9640 1 -- 172.21.15.90:0/2602781838 learned_addr learned my addr 172.21.15.90:0/2602781838 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:55.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.962+0000 7f39c16e8640 1 --2- 172.21.15.90:0/2602781838 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39bc104d30 0x7f39bc19a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:55.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.962+0000 7f39c26ea640 1 --2- 172.21.15.90:0/2602781838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39bc109a30 0x7f39bc1a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:55.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.963+0000 7f39c1ee9640 1 -- 172.21.15.90:0/2602781838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39bc109a30 msgr2=0x7f39bc1a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:55.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.963+0000 7f39c1ee9640 1 --2- 172.21.15.90:0/2602781838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39bc109a30 0x7f39bc1a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:55.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.963+0000 7f39c1ee9640 1 -- 172.21.15.90:0/2602781838 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39bc104d30 msgr2=0x7f39bc19a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:55.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.963+0000 7f39c1ee9640 1 --2- 172.21.15.90:0/2602781838 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39bc104d30 0x7f39bc19a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:55.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.963+0000 7f39c1ee9640 1 -- 172.21.15.90:0/2602781838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39bc1a1d60 con 0x7f39bc103af0 2024-04-25T11:34:55.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.963+0000 7f39c16e8640 1 --2- 172.21.15.90:0/2602781838 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39bc104d30 0x7f39bc19a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:55.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.963+0000 7f39c26ea640 1 --2- 172.21.15.90:0/2602781838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39bc109a30 0x7f39bc1a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:55.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.963+0000 7f39c1ee9640 1 --2- 172.21.15.90:0/2602781838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39bc103af0 0x7f39bc19a220 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f39b8006fd0 tx=0x7f39b800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:55.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.963+0000 7f39b2ffd640 1 -- 172.21.15.90:0/2602781838 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b8011830 con 0x7f39bc103af0 2024-04-25T11:34:55.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.963+0000 7f39b2ffd640 1 -- 172.21.15.90:0/2602781838 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39b8016070 con 0x7f39bc103af0 2024-04-25T11:34:55.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.963+0000 7f39b2ffd640 1 -- 172.21.15.90:0/2602781838 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b8010e00 con 0x7f39bc103af0 2024-04-25T11:34:55.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.963+0000 7f39c414d640 1 -- 172.21.15.90:0/2602781838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39bc1a1ff0 con 0x7f39bc103af0 2024-04-25T11:34:55.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.963+0000 7f39c414d640 1 -- 172.21.15.90:0/2602781838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39bc1a2480 con 0x7f39bc103af0 2024-04-25T11:34:55.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.964+0000 7f39c414d640 1 -- 172.21.15.90:0/2602781838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39bc06a1a0 con 0x7f39bc103af0 2024-04-25T11:34:55.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.965+0000 7f39b2ffd640 1 -- 172.21.15.90:0/2602781838 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f39b80119d0 con 0x7f39bc103af0 2024-04-25T11:34:55.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.966+0000 7f39b2ffd640 1 --2- 172.21.15.90:0/2602781838 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f39a80afaf0 0x7f39a80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:55.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.966+0000 7f39c16e8640 1 --2- 172.21.15.90:0/2602781838 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f39a80afaf0 0x7f39a80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:55.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.966+0000 7f39b2ffd640 1 -- 172.21.15.90:0/2602781838 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f39b80de990 con 0x7f39bc103af0 2024-04-25T11:34:55.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.966+0000 7f39c16e8640 1 --2- 172.21.15.90:0/2602781838 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f39a80afaf0 0x7f39a80b2000 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f39ac005fd0 tx=0x7f39ac005ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:55.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:55.970+0000 7f39b2ffd640 1 -- 172.21.15.90:0/2602781838 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f39b80a7550 con 0x7f39bc103af0 2024-04-25T11:34:56.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.147+0000 7f39c414d640 1 -- 172.21.15.90:0/2602781838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f39bc02dc10 con 0x7f39bc103af0 2024-04-25T11:34:56.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.148+0000 7f39b2ffd640 1 -- 172.21.15.90:0/2602781838 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) ==== 93+0+10 (secure 0 0 0) 0x7f39b80a6e50 con 0x7f39bc103af0 2024-04-25T11:34:56.137 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T11:34:56.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.150+0000 7f39c414d640 1 -- 172.21.15.90:0/2602781838 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f39a80afaf0 msgr2=0x7f39a80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:56.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.150+0000 7f39c414d640 1 --2- 172.21.15.90:0/2602781838 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f39a80afaf0 0x7f39a80b2000 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f39ac005fd0 tx=0x7f39ac005ec0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.150+0000 7f39c414d640 1 -- 172.21.15.90:0/2602781838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39bc103af0 msgr2=0x7f39bc19a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:56.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.150+0000 7f39c414d640 1 --2- 172.21.15.90:0/2602781838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39bc103af0 0x7f39bc19a220 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f39b8006fd0 tx=0x7f39b800b8e0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.150+0000 7f39c414d640 1 -- 172.21.15.90:0/2602781838 shutdown_connections 2024-04-25T11:34:56.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.150+0000 7f39c414d640 1 --2- 172.21.15.90:0/2602781838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39bc109a30 0x7f39bc1a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.150+0000 7f39c414d640 1 --2- 172.21.15.90:0/2602781838 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39bc104d30 0x7f39bc19a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.150+0000 7f39c414d640 1 --2- 172.21.15.90:0/2602781838 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f39a80afaf0 0x7f39a80b2000 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.150+0000 7f39c414d640 1 --2- 172.21.15.90:0/2602781838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39bc103af0 0x7f39bc19a220 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.150+0000 7f39c414d640 1 -- 172.21.15.90:0/2602781838 >> 172.21.15.90:0/2602781838 conn(0x7f39bc0fece0 msgr2=0x7f39bc105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:56.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.150+0000 7f39c414d640 1 -- 172.21.15.90:0/2602781838 shutdown_connections 2024-04-25T11:34:56.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.150+0000 7f39c414d640 1 -- 172.21.15.90:0/2602781838 wait complete. 2024-04-25T11:34:56.149 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:55 smithi090 ceph-mon[29575]: pgmap v328: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 426 B/s wr, 12 op/s 2024-04-25T11:34:56.150 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:55 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1385358998' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:34:56.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T11:34:56.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.272+0000 7f000dc13640 1 -- 172.21.15.90:0/4275026316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0008109790 msgr2=0x7f000810bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:56.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.272+0000 7f000dc13640 1 --2- 172.21.15.90:0/4275026316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0008109790 0x7f000810bca0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0000009ca0 tx=0x7f000002a8c0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.272+0000 7f000dc13640 1 -- 172.21.15.90:0/4275026316 shutdown_connections 2024-04-25T11:34:56.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.272+0000 7f000dc13640 1 --2- 172.21.15.90:0/4275026316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0008109790 0x7f000810bca0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.272+0000 7f000dc13640 1 --2- 172.21.15.90:0/4275026316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0008102370 0x7f0008109100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.272+0000 7f000dc13640 1 --2- 172.21.15.90:0/4275026316 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0008101e70 0x7f0008106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.272+0000 7f000dc13640 1 -- 172.21.15.90:0/4275026316 >> 172.21.15.90:0/4275026316 conn(0x7f00080fd380 msgr2=0x7f00080ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:56.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.272+0000 7f000dc13640 1 -- 172.21.15.90:0/4275026316 shutdown_connections 2024-04-25T11:34:56.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.272+0000 7f000dc13640 1 -- 172.21.15.90:0/4275026316 wait complete. 2024-04-25T11:34:56.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.273+0000 7f000dc13640 1 Processor -- start 2024-04-25T11:34:56.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.273+0000 7f000dc13640 1 -- start start 2024-04-25T11:34:56.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.273+0000 7f000dc13640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0008101e70 0x7f000819a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:56.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.273+0000 7f000dc13640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0008102370 0x7f000819a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:56.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.273+0000 7f000dc13640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0008109790 0x7f00081a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:56.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.273+0000 7f000dc13640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f000810f600 con 0x7f0008109790 2024-04-25T11:34:56.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.273+0000 7f000dc13640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f000810f480 con 0x7f0008101e70 2024-04-25T11:34:56.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.273+0000 7f000dc13640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f000810f780 con 0x7f0008102370 2024-04-25T11:34:56.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.273+0000 7f0007fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0008109790 0x7f00081a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:56.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.273+0000 7f0007fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0008109790 0x7f00081a1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57674/0 (socket says 172.21.15.90:57674) 2024-04-25T11:34:56.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.273+0000 7f0007fff640 1 -- 172.21.15.90:0/1249335952 learned_addr learned my addr 172.21.15.90:0/1249335952 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:56.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.273+0000 7f00077fe640 1 --2- 172.21.15.90:0/1249335952 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0008101e70 0x7f000819a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:56.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.273+0000 7f0006ffd640 1 --2- 172.21.15.90:0/1249335952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0008102370 0x7f000819a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:56.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.273+0000 7f0007fff640 1 -- 172.21.15.90:0/1249335952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0008102370 msgr2=0x7f000819a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:56.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.273+0000 7f0007fff640 1 --2- 172.21.15.90:0/1249335952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0008102370 0x7f000819a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.274+0000 7f0007fff640 1 -- 172.21.15.90:0/1249335952 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0008101e70 msgr2=0x7f000819a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:56.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.274+0000 7f0007fff640 1 --2- 172.21.15.90:0/1249335952 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0008101e70 0x7f000819a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.274+0000 7f0007fff640 1 -- 172.21.15.90:0/1249335952 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00081a1c90 con 0x7f0008109790 2024-04-25T11:34:56.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.274+0000 7f0006ffd640 1 --2- 172.21.15.90:0/1249335952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0008102370 0x7f000819a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:56.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.274+0000 7f0007fff640 1 --2- 172.21.15.90:0/1249335952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0008109790 0x7f00081a1620 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f00000097b0 tx=0x7f0000031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:56.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.274+0000 7f0004ff9640 1 -- 172.21.15.90:0/1249335952 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0000003bf0 con 0x7f0008109790 2024-04-25T11:34:56.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.274+0000 7f000dc13640 1 -- 172.21.15.90:0/1249335952 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00081a1f20 con 0x7f0008109790 2024-04-25T11:34:56.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.274+0000 7f0004ff9640 1 -- 172.21.15.90:0/1249335952 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0000003d90 con 0x7f0008109790 2024-04-25T11:34:56.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.274+0000 7f0004ff9640 1 -- 172.21.15.90:0/1249335952 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f000002fc30 con 0x7f0008109790 2024-04-25T11:34:56.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.274+0000 7f000dc13640 1 -- 172.21.15.90:0/1249335952 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00081a2380 con 0x7f0008109790 2024-04-25T11:34:56.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.275+0000 7f000dc13640 1 -- 172.21.15.90:0/1249335952 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f000806a190 con 0x7f0008109790 2024-04-25T11:34:56.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.276+0000 7f0004ff9640 1 -- 172.21.15.90:0/1249335952 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0000004040 con 0x7f0008109790 2024-04-25T11:34:56.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.277+0000 7f0004ff9640 1 --2- 172.21.15.90:0/1249335952 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7efff00afaf0 0x7efff00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:56.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.277+0000 7f00077fe640 1 --2- 172.21.15.90:0/1249335952 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7efff00afaf0 0x7efff00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:56.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.277+0000 7f0004ff9640 1 -- 172.21.15.90:0/1249335952 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f00001004b0 con 0x7f0008109790 2024-04-25T11:34:56.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.277+0000 7f00077fe640 1 --2- 172.21.15.90:0/1249335952 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7efff00afaf0 0x7efff00b2000 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7efff8005d00 tx=0x7efff800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:56.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.280+0000 7f0004ff9640 1 -- 172.21.15.90:0/1249335952 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f000806a190 con 0x7f0008109790 2024-04-25T11:34:56.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:55 smithi138 ceph-mon[32328]: pgmap v328: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 426 B/s wr, 12 op/s 2024-04-25T11:34:56.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:55 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1385358998' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:34:56.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:55 smithi156 ceph-mon[33045]: pgmap v328: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 426 B/s wr, 12 op/s 2024-04-25T11:34:56.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:55 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1385358998' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:34:56.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.458+0000 7f000dc13640 1 -- 172.21.15.90:0/1249335952 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f000802dc10 con 0x7f0008109790 2024-04-25T11:34:56.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.459+0000 7f0004ff9640 1 -- 172.21.15.90:0/1249335952 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v79) ==== 104+0+11 (secure 0 0 0) 0x7f00000c9070 con 0x7f0008109790 2024-04-25T11:34:56.447 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T11:34:56.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.461+0000 7f000dc13640 1 -- 172.21.15.90:0/1249335952 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7efff00afaf0 msgr2=0x7efff00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:56.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.461+0000 7f000dc13640 1 --2- 172.21.15.90:0/1249335952 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7efff00afaf0 0x7efff00b2000 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7efff8005d00 tx=0x7efff800b410 comp rx=0 tx=0).stop 2024-04-25T11:34:56.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.461+0000 7f000dc13640 1 -- 172.21.15.90:0/1249335952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0008109790 msgr2=0x7f00081a1620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:56.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.461+0000 7f000dc13640 1 --2- 172.21.15.90:0/1249335952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0008109790 0x7f00081a1620 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f00000097b0 tx=0x7f0000031840 comp rx=0 tx=0).stop 2024-04-25T11:34:56.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.461+0000 7f000dc13640 1 -- 172.21.15.90:0/1249335952 shutdown_connections 2024-04-25T11:34:56.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.461+0000 7f000dc13640 1 --2- 172.21.15.90:0/1249335952 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7efff00afaf0 0x7efff00b2000 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.461+0000 7f000dc13640 1 --2- 172.21.15.90:0/1249335952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0008109790 0x7f00081a1620 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.461+0000 7f000dc13640 1 --2- 172.21.15.90:0/1249335952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0008102370 0x7f000819a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.461+0000 7f000dc13640 1 --2- 172.21.15.90:0/1249335952 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0008101e70 0x7f000819a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.461+0000 7f000dc13640 1 -- 172.21.15.90:0/1249335952 >> 172.21.15.90:0/1249335952 conn(0x7f00080fd380 msgr2=0x7f00081053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:56.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.461+0000 7f000dc13640 1 -- 172.21.15.90:0/1249335952 shutdown_connections 2024-04-25T11:34:56.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.461+0000 7f000dc13640 1 -- 172.21.15.90:0/1249335952 wait complete. 2024-04-25T11:34:56.461 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T11:34:56.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.582+0000 7f56d8b3f640 1 -- 172.21.15.90:0/520024748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56d4104d30 msgr2=0x7f56d4105140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:56.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.582+0000 7f56d8b3f640 1 --2- 172.21.15.90:0/520024748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56d4104d30 0x7f56d4105140 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f56c4009900 tx=0x7f56c402a410 comp rx=0 tx=0).stop 2024-04-25T11:34:56.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.582+0000 7f56d8b3f640 1 -- 172.21.15.90:0/520024748 shutdown_connections 2024-04-25T11:34:56.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.582+0000 7f56d8b3f640 1 --2- 172.21.15.90:0/520024748 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f56d4109a30 0x7f56d410be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.582+0000 7f56d8b3f640 1 --2- 172.21.15.90:0/520024748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56d4104d30 0x7f56d4105140 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.582+0000 7f56d8b3f640 1 --2- 172.21.15.90:0/520024748 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56d4103af0 0x7f56d41047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.582+0000 7f56d8b3f640 1 -- 172.21.15.90:0/520024748 >> 172.21.15.90:0/520024748 conn(0x7f56d40fece0 msgr2=0x7f56d4101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:56.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.582+0000 7f56d8b3f640 1 -- 172.21.15.90:0/520024748 shutdown_connections 2024-04-25T11:34:56.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.582+0000 7f56d8b3f640 1 -- 172.21.15.90:0/520024748 wait complete. 2024-04-25T11:34:56.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.583+0000 7f56d8b3f640 1 Processor -- start 2024-04-25T11:34:56.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.583+0000 7f56d8b3f640 1 -- start start 2024-04-25T11:34:56.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.583+0000 7f56d8b3f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56d4103af0 0x7f56d419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:56.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.583+0000 7f56d8b3f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f56d4104d30 0x7f56d419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:56.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.583+0000 7f56d8b3f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56d4109a30 0x7f56d41a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:56.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.583+0000 7f56d8b3f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f56d410f610 con 0x7f56d4109a30 2024-04-25T11:34:56.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.583+0000 7f56d8b3f640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f56d410f490 con 0x7f56d4104d30 2024-04-25T11:34:56.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.583+0000 7f56d8b3f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f56d410f790 con 0x7f56d4103af0 2024-04-25T11:34:56.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.583+0000 7f56d2d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56d4109a30 0x7f56d41a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:56.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.583+0000 7f56d2d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56d4109a30 0x7f56d41a16e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57694/0 (socket says 172.21.15.90:57694) 2024-04-25T11:34:56.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.583+0000 7f56d2d9d640 1 -- 172.21.15.90:0/3186660341 learned_addr learned my addr 172.21.15.90:0/3186660341 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:56.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.583+0000 7f56d259c640 1 --2- 172.21.15.90:0/3186660341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56d4103af0 0x7f56d419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:56.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.583+0000 7f56d1d9b640 1 --2- 172.21.15.90:0/3186660341 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f56d4104d30 0x7f56d419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:56.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.583+0000 7f56d2d9d640 1 -- 172.21.15.90:0/3186660341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56d4103af0 msgr2=0x7f56d419a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:56.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.583+0000 7f56d2d9d640 1 --2- 172.21.15.90:0/3186660341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56d4103af0 0x7f56d419a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.583+0000 7f56d2d9d640 1 -- 172.21.15.90:0/3186660341 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f56d4104d30 msgr2=0x7f56d419a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:56.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.583+0000 7f56d2d9d640 1 --2- 172.21.15.90:0/3186660341 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f56d4104d30 0x7f56d419a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.583+0000 7f56d2d9d640 1 -- 172.21.15.90:0/3186660341 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56d41a1d50 con 0x7f56d4109a30 2024-04-25T11:34:56.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.583+0000 7f56d259c640 1 --2- 172.21.15.90:0/3186660341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56d4103af0 0x7f56d419a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:56.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.584+0000 7f56d2d9d640 1 --2- 172.21.15.90:0/3186660341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56d4109a30 0x7f56d41a16e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f56cc00cad0 tx=0x7f56cc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:56.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.584+0000 7f56c37fe640 1 -- 172.21.15.90:0/3186660341 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56cc00ae20 con 0x7f56d4109a30 2024-04-25T11:34:56.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.584+0000 7f56c37fe640 1 -- 172.21.15.90:0/3186660341 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56cc005020 con 0x7f56d4109a30 2024-04-25T11:34:56.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.584+0000 7f56d8b3f640 1 -- 172.21.15.90:0/3186660341 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56d41a1fe0 con 0x7f56d4109a30 2024-04-25T11:34:56.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.584+0000 7f56c37fe640 1 -- 172.21.15.90:0/3186660341 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56cc00f6b0 con 0x7f56d4109a30 2024-04-25T11:34:56.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.584+0000 7f56d8b3f640 1 -- 172.21.15.90:0/3186660341 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56d41a24a0 con 0x7f56d4109a30 2024-04-25T11:34:56.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.585+0000 7f56d8b3f640 1 -- 172.21.15.90:0/3186660341 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56d4108840 con 0x7f56d4109a30 2024-04-25T11:34:56.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.586+0000 7f56c37fe640 1 -- 172.21.15.90:0/3186660341 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f56cc0051d0 con 0x7f56d4109a30 2024-04-25T11:34:56.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.587+0000 7f56c37fe640 1 --2- 172.21.15.90:0/3186660341 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f56bc0afb70 0x7f56bc0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:56.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.587+0000 7f56d259c640 1 --2- 172.21.15.90:0/3186660341 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f56bc0afb70 0x7f56bc0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:56.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.587+0000 7f56c37fe640 1 -- 172.21.15.90:0/3186660341 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f56cc0decb0 con 0x7f56d4109a30 2024-04-25T11:34:56.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.587+0000 7f56d259c640 1 --2- 172.21.15.90:0/3186660341 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f56bc0afb70 0x7f56bc0b2080 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f56d41046f0 tx=0x7f56c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:56.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.591+0000 7f56c37fe640 1 -- 172.21.15.90:0/3186660341 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f56cc014030 con 0x7f56d4109a30 2024-04-25T11:34:56.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.771+0000 7f56d8b3f640 1 -- 172.21.15.90:0/3186660341 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f56d40637a0 con 0x7f56d4109a30 2024-04-25T11:34:56.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.771+0000 7f56c37fe640 1 -- 172.21.15.90:0/3186660341 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v79) ==== 100+0+11 (secure 0 0 0) 0x7f56cc0a7870 con 0x7f56d4109a30 2024-04-25T11:34:56.760 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T11:34:56.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.773+0000 7f56d8b3f640 1 -- 172.21.15.90:0/3186660341 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f56bc0afb70 msgr2=0x7f56bc0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.773+0000 7f56d8b3f640 1 --2- 172.21.15.90:0/3186660341 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f56bc0afb70 0x7f56bc0b2080 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f56d41046f0 tx=0x7f56c8005f90 comp rx=0 tx=0).stop 2024-04-25T11:34:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.773+0000 7f56d8b3f640 1 -- 172.21.15.90:0/3186660341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56d4109a30 msgr2=0x7f56d41a16e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.773+0000 7f56d8b3f640 1 --2- 172.21.15.90:0/3186660341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56d4109a30 0x7f56d41a16e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f56cc00cad0 tx=0x7f56cc00cf90 comp rx=0 tx=0).stop 2024-04-25T11:34:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.773+0000 7f56d8b3f640 1 -- 172.21.15.90:0/3186660341 shutdown_connections 2024-04-25T11:34:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.773+0000 7f56d8b3f640 1 --2- 172.21.15.90:0/3186660341 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f56bc0afb70 0x7f56bc0b2080 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.773+0000 7f56d8b3f640 1 --2- 172.21.15.90:0/3186660341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56d4109a30 0x7f56d41a16e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.773+0000 7f56d8b3f640 1 --2- 172.21.15.90:0/3186660341 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f56d4104d30 0x7f56d419a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.773+0000 7f56d8b3f640 1 --2- 172.21.15.90:0/3186660341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56d4103af0 0x7f56d419a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.773+0000 7f56d8b3f640 1 -- 172.21.15.90:0/3186660341 >> 172.21.15.90:0/3186660341 conn(0x7f56d40fece0 msgr2=0x7f56d4105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.773+0000 7f56d8b3f640 1 -- 172.21.15.90:0/3186660341 shutdown_connections 2024-04-25T11:34:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.774+0000 7f56d8b3f640 1 -- 172.21.15.90:0/3186660341 wait complete. 2024-04-25T11:34:56.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:34:56.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.894+0000 7f1902242640 1 -- 172.21.15.90:0/3351217135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18fc106f50 msgr2=0x7f18fc109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:56.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.894+0000 7f1902242640 1 --2- 172.21.15.90:0/3351217135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18fc106f50 0x7f18fc109370 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f18ec009900 tx=0x7f18ec02a410 comp rx=0 tx=0).stop 2024-04-25T11:34:56.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.894+0000 7f1902242640 1 -- 172.21.15.90:0/3351217135 shutdown_connections 2024-04-25T11:34:56.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.894+0000 7f1902242640 1 --2- 172.21.15.90:0/3351217135 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18fc1098b0 0x7f18fc10bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.894+0000 7f1902242640 1 --2- 172.21.15.90:0/3351217135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18fc106f50 0x7f18fc109370 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.894+0000 7f1902242640 1 --2- 172.21.15.90:0/3351217135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18fc1045f0 0x7f18fc106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.894+0000 7f1902242640 1 -- 172.21.15.90:0/3351217135 >> 172.21.15.90:0/3351217135 conn(0x7f18fc0fdbf0 msgr2=0x7f18fc1000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:56.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.894+0000 7f1902242640 1 -- 172.21.15.90:0/3351217135 shutdown_connections 2024-04-25T11:34:56.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.894+0000 7f1902242640 1 -- 172.21.15.90:0/3351217135 wait complete. 2024-04-25T11:34:56.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.895+0000 7f1902242640 1 Processor -- start 2024-04-25T11:34:56.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.895+0000 7f1902242640 1 -- start start 2024-04-25T11:34:56.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.895+0000 7f1902242640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18fc1045f0 0x7f18fc19c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:56.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.895+0000 7f1902242640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18fc106f50 0x7f18fc19c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:56.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.895+0000 7f1902242640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18fc1098b0 0x7f18fc1a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:56.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.895+0000 7f1902242640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f18fc102ab0 con 0x7f18fc1045f0 2024-04-25T11:34:56.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.895+0000 7f1902242640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f18fc102930 con 0x7f18fc1098b0 2024-04-25T11:34:56.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.895+0000 7f1902242640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f18fc102c30 con 0x7f18fc106f50 2024-04-25T11:34:56.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.895+0000 7f18fb7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18fc1045f0 0x7f18fc19c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:56.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.895+0000 7f18fb7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18fc1045f0 0x7f18fc19c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57702/0 (socket says 172.21.15.90:57702) 2024-04-25T11:34:56.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.895+0000 7f18fb7fe640 1 -- 172.21.15.90:0/3922058685 learned_addr learned my addr 172.21.15.90:0/3922058685 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:56.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.896+0000 7f18fbfff640 1 --2- 172.21.15.90:0/3922058685 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18fc1098b0 0x7f18fc1a38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:56.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.896+0000 7f18faffd640 1 --2- 172.21.15.90:0/3922058685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18fc106f50 0x7f18fc19c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:56.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.896+0000 7f18fb7fe640 1 -- 172.21.15.90:0/3922058685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18fc106f50 msgr2=0x7f18fc19c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:56.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.896+0000 7f18fb7fe640 1 --2- 172.21.15.90:0/3922058685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18fc106f50 0x7f18fc19c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.896+0000 7f18fb7fe640 1 -- 172.21.15.90:0/3922058685 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18fc1098b0 msgr2=0x7f18fc1a38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:56.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.896+0000 7f18fb7fe640 1 --2- 172.21.15.90:0/3922058685 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18fc1098b0 0x7f18fc1a38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:56.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.896+0000 7f18fb7fe640 1 -- 172.21.15.90:0/3922058685 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18fc1a3f30 con 0x7f18fc1045f0 2024-04-25T11:34:56.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.896+0000 7f18fbfff640 1 --2- 172.21.15.90:0/3922058685 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18fc1098b0 0x7f18fc1a38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:56.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.896+0000 7f18fb7fe640 1 --2- 172.21.15.90:0/3922058685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18fc1045f0 0x7f18fc19c3f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f18f0005fd0 tx=0x7f18f000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:56.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.896+0000 7f18f8ff9640 1 -- 172.21.15.90:0/3922058685 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f0017070 con 0x7f18fc1045f0 2024-04-25T11:34:56.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.896+0000 7f18f8ff9640 1 -- 172.21.15.90:0/3922058685 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18f0009c30 con 0x7f18fc1045f0 2024-04-25T11:34:56.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.896+0000 7f18f8ff9640 1 -- 172.21.15.90:0/3922058685 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f0003d30 con 0x7f18fc1045f0 2024-04-25T11:34:56.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.896+0000 7f1902242640 1 -- 172.21.15.90:0/3922058685 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18fc1a4220 con 0x7f18fc1045f0 2024-04-25T11:34:56.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.896+0000 7f1902242640 1 -- 172.21.15.90:0/3922058685 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18fc1a4770 con 0x7f18fc1045f0 2024-04-25T11:34:56.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.897+0000 7f1902242640 1 -- 172.21.15.90:0/3922058685 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18fc1a49d0 con 0x7f18fc1045f0 2024-04-25T11:34:56.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.898+0000 7f18f8ff9640 1 -- 172.21.15.90:0/3922058685 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f18f0003ed0 con 0x7f18fc1045f0 2024-04-25T11:34:56.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.899+0000 7f18f8ff9640 1 --2- 172.21.15.90:0/3922058685 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f18e40afaf0 0x7f18e40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:56.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.899+0000 7f18faffd640 1 --2- 172.21.15.90:0/3922058685 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f18e40afaf0 0x7f18e40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:56.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.899+0000 7f18f8ff9640 1 -- 172.21.15.90:0/3922058685 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f18f00df1a0 con 0x7f18fc1045f0 2024-04-25T11:34:56.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.899+0000 7f18faffd640 1 --2- 172.21.15.90:0/3922058685 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f18e40afaf0 0x7f18e40b2000 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f18ec02a930 tx=0x7f18ec000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:56.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:56.902+0000 7f18f8ff9640 1 -- 172.21.15.90:0/3922058685 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f18f0020030 con 0x7f18fc1045f0 2024-04-25T11:34:57.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.116+0000 7f1902242640 1 -- 172.21.15.90:0/3922058685 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18fc1a4ba0 con 0x7f18fc1045f0 2024-04-25T11:34:57.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.117+0000 7f18f8ff9640 1 -- 172.21.15.90:0/3922058685 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11930 (secure 0 0 0) 0x7f18f0009de0 con 0x7f18fc1045f0 2024-04-25T11:34:57.107 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:34:57.107 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:33:16.640084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524,24475,24499,24458,24431]},"id":1}]} 2024-04-25T11:34:57.107 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 25 2024-04-25T11:34:57.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.119+0000 7f1902242640 1 -- 172.21.15.90:0/3922058685 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f18e40afaf0 msgr2=0x7f18e40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:57.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.119+0000 7f1902242640 1 --2- 172.21.15.90:0/3922058685 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f18e40afaf0 0x7f18e40b2000 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f18ec02a930 tx=0x7f18ec000bc0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.119+0000 7f1902242640 1 -- 172.21.15.90:0/3922058685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18fc1045f0 msgr2=0x7f18fc19c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:57.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.119+0000 7f1902242640 1 --2- 172.21.15.90:0/3922058685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18fc1045f0 0x7f18fc19c3f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f18f0005fd0 tx=0x7f18f000dde0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.119+0000 7f1902242640 1 -- 172.21.15.90:0/3922058685 shutdown_connections 2024-04-25T11:34:57.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.119+0000 7f1902242640 1 --2- 172.21.15.90:0/3922058685 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18fc1098b0 0x7f18fc1a38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.119+0000 7f1902242640 1 --2- 172.21.15.90:0/3922058685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18fc106f50 0x7f18fc19c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.119+0000 7f1902242640 1 --2- 172.21.15.90:0/3922058685 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f18e40afaf0 0x7f18e40b2000 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.119+0000 7f1902242640 1 --2- 172.21.15.90:0/3922058685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18fc1045f0 0x7f18fc19c3f0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.119+0000 7f1902242640 1 -- 172.21.15.90:0/3922058685 >> 172.21.15.90:0/3922058685 conn(0x7f18fc0fdbf0 msgr2=0x7f18fc109cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:57.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.119+0000 7f1902242640 1 -- 172.21.15.90:0/3922058685 shutdown_connections 2024-04-25T11:34:57.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.119+0000 7f1902242640 1 -- 172.21.15.90:0/3922058685 wait complete. 2024-04-25T11:34:57.118 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:56 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2602781838' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:34:57.118 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:56 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1249335952' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:34:57.119 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:56 smithi090 ceph-mon[29575]: pgmap v329: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-25T11:34:57.119 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:56 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3186660341' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:34:57.120 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:34:57.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.238+0000 7f1d05315640 1 -- 172.21.15.90:0/3421475174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d001029e0 msgr2=0x7f1d001036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:57.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.238+0000 7f1d05315640 1 --2- 172.21.15.90:0/3421475174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d001029e0 0x7f1d001036e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1cf8009900 tx=0x7f1cf802a410 comp rx=0 tx=0).stop 2024-04-25T11:34:57.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.238+0000 7f1d05315640 1 -- 172.21.15.90:0/3421475174 shutdown_connections 2024-04-25T11:34:57.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.238+0000 7f1d05315640 1 --2- 172.21.15.90:0/3421475174 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1d00108920 0x7f1d0010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.238+0000 7f1d05315640 1 --2- 172.21.15.90:0/3421475174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d00103c20 0x7f1d00104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.238+0000 7f1d05315640 1 --2- 172.21.15.90:0/3421475174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d001029e0 0x7f1d001036e0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.238+0000 7f1d05315640 1 -- 172.21.15.90:0/3421475174 >> 172.21.15.90:0/3421475174 conn(0x7f1d000fdbd0 msgr2=0x7f1d00100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:57.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.238+0000 7f1d05315640 1 -- 172.21.15.90:0/3421475174 shutdown_connections 2024-04-25T11:34:57.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.238+0000 7f1d05315640 1 -- 172.21.15.90:0/3421475174 wait complete. 2024-04-25T11:34:57.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.239+0000 7f1d05315640 1 Processor -- start 2024-04-25T11:34:57.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.239+0000 7f1d05315640 1 -- start start 2024-04-25T11:34:57.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.239+0000 7f1d05315640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1d001029e0 0x7f1d0006bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:57.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.239+0000 7f1d05315640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d00103c20 0x7f1d0019f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:57.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.239+0000 7f1d05315640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d00108920 0x7f1d001a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:57.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.239+0000 7f1d05315640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0010e4e0 con 0x7f1d00108920 2024-04-25T11:34:57.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.239+0000 7f1d05315640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0010e360 con 0x7f1d001029e0 2024-04-25T11:34:57.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.239+0000 7f1d05315640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0010e660 con 0x7f1d00103c20 2024-04-25T11:34:57.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.239+0000 7f1cff7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d00108920 0x7f1d001a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:57.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.239+0000 7f1cff7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d00108920 0x7f1d001a1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57722/0 (socket says 172.21.15.90:57722) 2024-04-25T11:34:57.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.239+0000 7f1cff7fe640 1 -- 172.21.15.90:0/3429381777 learned_addr learned my addr 172.21.15.90:0/3429381777 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:57.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.239+0000 7f1cfeffd640 1 --2- 172.21.15.90:0/3429381777 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1d001029e0 0x7f1d0006bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:57.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.239+0000 7f1cfe7fc640 1 --2- 172.21.15.90:0/3429381777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d00103c20 0x7f1d0019f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:57.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.239+0000 7f1cff7fe640 1 -- 172.21.15.90:0/3429381777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d00103c20 msgr2=0x7f1d0019f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:57.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.239+0000 7f1cff7fe640 1 --2- 172.21.15.90:0/3429381777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d00103c20 0x7f1d0019f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.239+0000 7f1cff7fe640 1 -- 172.21.15.90:0/3429381777 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1d001029e0 msgr2=0x7f1d0006bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:57.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.239+0000 7f1cff7fe640 1 --2- 172.21.15.90:0/3429381777 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1d001029e0 0x7f1d0006bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.239+0000 7f1cff7fe640 1 -- 172.21.15.90:0/3429381777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d001a2060 con 0x7f1d00108920 2024-04-25T11:34:57.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.239+0000 7f1cfeffd640 1 --2- 172.21.15.90:0/3429381777 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1d001029e0 0x7f1d0006bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:57.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.240+0000 7f1cff7fe640 1 --2- 172.21.15.90:0/3429381777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d00108920 0x7f1d001a1840 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1cf000c970 tx=0x7f1cf000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:57.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.240+0000 7f1cf7fff640 1 -- 172.21.15.90:0/3429381777 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf0007bb0 con 0x7f1d00108920 2024-04-25T11:34:57.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.240+0000 7f1d05315640 1 -- 172.21.15.90:0/3429381777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d001a2350 con 0x7f1d00108920 2024-04-25T11:34:57.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.240+0000 7f1cf7fff640 1 -- 172.21.15.90:0/3429381777 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1cf0007d50 con 0x7f1d00108920 2024-04-25T11:34:57.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.240+0000 7f1cf7fff640 1 -- 172.21.15.90:0/3429381777 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf0004e60 con 0x7f1d00108920 2024-04-25T11:34:57.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.240+0000 7f1d05315640 1 -- 172.21.15.90:0/3429381777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d001a28a0 con 0x7f1d00108920 2024-04-25T11:34:57.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.241+0000 7f1d05315640 1 -- 172.21.15.90:0/3429381777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d00107730 con 0x7f1d00108920 2024-04-25T11:34:57.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.242+0000 7f1cf7fff640 1 -- 172.21.15.90:0/3429381777 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1cf0005000 con 0x7f1d00108920 2024-04-25T11:34:57.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.243+0000 7f1cf7fff640 1 --2- 172.21.15.90:0/3429381777 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f1cec0afaf0 0x7f1cec0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:57.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.243+0000 7f1cfeffd640 1 --2- 172.21.15.90:0/3429381777 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f1cec0afaf0 0x7f1cec0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:57.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.243+0000 7f1cf7fff640 1 -- 172.21.15.90:0/3429381777 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f1cf00deda0 con 0x7f1d00108920 2024-04-25T11:34:57.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.243+0000 7f1cfeffd640 1 --2- 172.21.15.90:0/3429381777 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f1cec0afaf0 0x7f1cec0b2000 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1cf8009230 tx=0x7f1cf8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:57.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.246+0000 7f1cf7fff640 1 -- 172.21.15.90:0/3429381777 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1cf0014030 con 0x7f1d00108920 2024-04-25T11:34:57.239 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:56 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2602781838' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:34:57.239 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:56 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1249335952' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:34:57.239 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:56 smithi156 ceph-mon[33045]: pgmap v329: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-25T11:34:57.239 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:56 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3186660341' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:34:57.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:56 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2602781838' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:34:57.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:56 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1249335952' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:34:57.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:56 smithi138 ceph-mon[32328]: pgmap v329: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-25T11:34:57.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:56 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3186660341' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:34:57.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.426+0000 7f1d05315640 1 -- 172.21.15.90:0/3429381777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1d0002dcd0 con 0x7f1d00108920 2024-04-25T11:34:57.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.428+0000 7f1cf7fff640 1 -- 172.21.15.90:0/3429381777 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+22706 (secure 0 0 0) 0x7f1cf00a7960 con 0x7f1d00108920 2024-04-25T11:34:57.417 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:34:57.417 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":79,"fsid":"58bd4e7a-02f6-11ef-bc93-c7b262605968","created":"2024-04-25T11:25:23.574550+0000","modified":"2024-04-25T11:34:17.871834+0000","last_up_change":"2024-04-25T11:32:23.852211+0000","last_in_change":"2024-04-25T11:32:04.188507+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:28:42.746649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:33:08.442925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:33:09.352886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e54d888-e67b-4951-8ae7-e355fb6b10a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6803","nonce":1201629159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6805","nonce":1201629159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6809","nonce":1201629159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6807","nonce":1201629159}]},"public_addr":"172.21.15.90:6803/1201629159","cluster_addr":"172.21.15.90:6805/1201629159","heartbeat_back_addr":"172.21.15.90:6809/1201629159","heartbeat_front_addr":"172.21.15.90:6807/1201629159","state":["exists","up"]},{"osd":1,"uuid":"91a71c78-dae9-4bc7-9f6f-9e55cfefbe28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6801","nonce":3976851933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6803","nonce":3976851933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6807","nonce":3976851933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6805","nonce":3976851933}]},"public_addr":"172.21.15.138:6801/3976851933","cluster_addr":"172.21.15.138:6803/3976851933","heartbeat_back_addr":"172.21.15.138:6807/3976851933","heartbeat_front_addr":"172.21.15.138:6805/3976851933","state":["exists","up"]},{"osd":2,"uuid":"daff6c7f-80fa-4859-9339-ef7a5e840832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6801","nonce":1505807670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6803","nonce":1505807670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6807","nonce":1505807670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6805","nonce":1505807670}]},"public_addr":"172.21.15.156:6801/1505807670","cluster_addr":"172.21.15.156:6803/1505807670","heartbeat_back_addr":"172.21.15.156:6807/1505807670","heartbeat_front_addr":"172.21.15.156:6805/1505807670","state":["exists","up"]},{"osd":3,"uuid":"33fd94ca-7465-401a-bc1e-706b6b31b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6811","nonce":2520012383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6813","nonce":2520012383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6817","nonce":2520012383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6815","nonce":2520012383}]},"public_addr":"172.21.15.90:6811/2520012383","cluster_addr":"172.21.15.90:6813/2520012383","heartbeat_back_addr":"172.21.15.90:6817/2520012383","heartbeat_front_addr":"172.21.15.90:6815/2520012383","state":["exists","up"]},{"osd":4,"uuid":"c5326212-750c-49fd-96e3-d1e62bb4b7f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6809","nonce":3752678279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6811","nonce":3752678279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6815","nonce":3752678279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6813","nonce":3752678279}]},"public_addr":"172.21.15.138:6809/3752678279","cluster_addr":"172.21.15.138:6811/3752678279","heartbeat_back_addr":"172.21.15.138:6815/3752678279","heartbeat_front_addr":"172.21.15.138:6813/3752678279","state":["exists","up"]},{"osd":5,"uuid":"d04a1156-820b-4b43-be89-e16a8910c3aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6809","nonce":3591210191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6811","nonce":3591210191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6815","nonce":3591210191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6813","nonce":3591210191}]},"public_addr":"172.21.15.156:6809/3591210191","cluster_addr":"172.21.15.156:6811/3591210191","heartbeat_back_addr":"172.21.15.156:6815/3591210191","heartbeat_front_addr":"172.21.15.156:6813/3591210191","state":["exists","up"]},{"osd":6,"uuid":"75675aa3-e978-490c-8205-13d9868a3d6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6819","nonce":2364955456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6821","nonce":2364955456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6825","nonce":2364955456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6823","nonce":2364955456}]},"public_addr":"172.21.15.90:6819/2364955456","cluster_addr":"172.21.15.90:6821/2364955456","heartbeat_back_addr":"172.21.15.90:6825/2364955456","heartbeat_front_addr":"172.21.15.90:6823/2364955456","state":["exists","up"]},{"osd":7,"uuid":"1a95fdf5-f548-4271-bbcd-e330f7955eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6817","nonce":4012476387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6819","nonce":4012476387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6823","nonce":4012476387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6821","nonce":4012476387}]},"public_addr":"172.21.15.138:6817/4012476387","cluster_addr":"172.21.15.138:6819/4012476387","heartbeat_back_addr":"172.21.15.138:6823/4012476387","heartbeat_front_addr":"172.21.15.138:6821/4012476387","state":["exists","up"]},{"osd":8,"uuid":"605e10b5-5806-4c1f-8e28-a7994d34ddce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6817","nonce":101092224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6819","nonce":101092224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6823","nonce":101092224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6821","nonce":101092224}]},"public_addr":"172.21.15.156:6817/101092224","cluster_addr":"172.21.15.156:6819/101092224","heartbeat_back_addr":"172.21.15.156:6823/101092224","heartbeat_front_addr":"172.21.15.156:6821/101092224","state":["exists","up"]},{"osd":9,"uuid":"d70fdf5e-9d0f-483d-a7d8-505b79b8a418","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6827","nonce":1171265966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6829","nonce":1171265966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6833","nonce":1171265966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6831","nonce":1171265966}]},"public_addr":"172.21.15.90:6827/1171265966","cluster_addr":"172.21.15.90:6829/1171265966","heartbeat_back_addr":"172.21.15.90:6833/1171265966","heartbeat_front_addr":"172.21.15.90:6831/1171265966","state":["exists","up"]},{"osd":10,"uuid":"1ec86c7d-c3a4-432d-9664-8e40836e1e20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6825","nonce":4264666232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6827","nonce":4264666232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6831","nonce":4264666232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6829","nonce":4264666232}]},"public_addr":"172.21.15.138:6825/4264666232","cluster_addr":"172.21.15.138:6827/4264666232","heartbeat_back_addr":"172.21.15.138:6831/4264666232","heartbeat_front_addr":"172.21.15.138:6829/4264666232","state":["exists","up"]},{"osd":11,"uuid":"51cbe729-292e-4d7b-b8e9-e1e2821ffe76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6825","nonce":1224251744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6827","nonce":1224251744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6831","nonce":1224251744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6829","nonce":1224251744}]},"public_addr":"172.21.15.156:6825/1224251744","cluster_addr":"172.21.15.156:6827/1224251744","heartbeat_back_addr":"172.21.15.156:6831/1224251744","heartbeat_front_addr":"172.21.15.156:6829/1224251744","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:27:50.794226+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:15.030278+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:38.518163+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:03.107351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:27.500707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:51.463545+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:16.015076+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:41.182071+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:05.362309+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:31.048936+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:56.488509+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:32:22.129178+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":7}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/2381633953":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6800/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/2190846900":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1707792941":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6800/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6801/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/6061052":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1849633392":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6801/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/2277490702":"2024-04-26T11:26:16.655278+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:34:57.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.430+0000 7f1d05315640 1 -- 172.21.15.90:0/3429381777 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f1cec0afaf0 msgr2=0x7f1cec0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:57.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.430+0000 7f1d05315640 1 --2- 172.21.15.90:0/3429381777 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f1cec0afaf0 0x7f1cec0b2000 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1cf8009230 tx=0x7f1cf8000bc0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.430+0000 7f1d05315640 1 -- 172.21.15.90:0/3429381777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d00108920 msgr2=0x7f1d001a1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:57.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.430+0000 7f1d05315640 1 --2- 172.21.15.90:0/3429381777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d00108920 0x7f1d001a1840 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1cf000c970 tx=0x7f1cf000ce30 comp rx=0 tx=0).stop 2024-04-25T11:34:57.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.430+0000 7f1d05315640 1 -- 172.21.15.90:0/3429381777 shutdown_connections 2024-04-25T11:34:57.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.430+0000 7f1d05315640 1 --2- 172.21.15.90:0/3429381777 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f1cec0afaf0 0x7f1cec0b2000 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.430+0000 7f1d05315640 1 --2- 172.21.15.90:0/3429381777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d00108920 0x7f1d001a1840 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.430+0000 7f1d05315640 1 --2- 172.21.15.90:0/3429381777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d00103c20 0x7f1d0019f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.430+0000 7f1d05315640 1 --2- 172.21.15.90:0/3429381777 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1d001029e0 0x7f1d0006bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.430+0000 7f1d05315640 1 -- 172.21.15.90:0/3429381777 >> 172.21.15.90:0/3429381777 conn(0x7f1d000fdbd0 msgr2=0x7f1d00104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:57.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.430+0000 7f1d05315640 1 -- 172.21.15.90:0/3429381777 shutdown_connections 2024-04-25T11:34:57.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.430+0000 7f1d05315640 1 -- 172.21.15.90:0/3429381777 wait complete. 2024-04-25T11:34:57.431 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T11:28:42.746649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T11:33:08.442925+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T11:33:09.352886+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T11:34:57.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T11:34:57.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.552+0000 7fbf1b4a7640 1 -- 172.21.15.90:0/2252610899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf141050b0 msgr2=0x7fbf141054c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.552+0000 7fbf1b4a7640 1 --2- 172.21.15.90:0/2252610899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf141050b0 0x7fbf141054c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fbf10009900 tx=0x7fbf1002a410 comp rx=0 tx=0).stop 2024-04-25T11:34:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.552+0000 7fbf1b4a7640 1 -- 172.21.15.90:0/2252610899 shutdown_connections 2024-04-25T11:34:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.552+0000 7fbf1b4a7640 1 --2- 172.21.15.90:0/2252610899 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbf14102260 0x7fbf1410bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.552+0000 7fbf1b4a7640 1 --2- 172.21.15.90:0/2252610899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf14101760 0x7fbf14101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.552+0000 7fbf1b4a7640 1 --2- 172.21.15.90:0/2252610899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf141050b0 0x7fbf141054c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.552+0000 7fbf1b4a7640 1 -- 172.21.15.90:0/2252610899 >> 172.21.15.90:0/2252610899 conn(0x7fbf140fd380 msgr2=0x7fbf140ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.552+0000 7fbf1b4a7640 1 -- 172.21.15.90:0/2252610899 shutdown_connections 2024-04-25T11:34:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.552+0000 7fbf1b4a7640 1 -- 172.21.15.90:0/2252610899 wait complete. 2024-04-25T11:34:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.553+0000 7fbf1b4a7640 1 Processor -- start 2024-04-25T11:34:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.553+0000 7fbf1b4a7640 1 -- start start 2024-04-25T11:34:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.553+0000 7fbf1b4a7640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbf14101760 0x7fbf1419c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.553+0000 7fbf1b4a7640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf14102260 0x7fbf1419c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.553+0000 7fbf1b4a7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf141050b0 0x7fbf141a3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:57.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.553+0000 7fbf1b4a7640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1410f600 con 0x7fbf14102260 2024-04-25T11:34:57.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.553+0000 7fbf1b4a7640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1410f480 con 0x7fbf14101760 2024-04-25T11:34:57.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.553+0000 7fbf1b4a7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1410f780 con 0x7fbf141050b0 2024-04-25T11:34:57.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.553+0000 7fbf18a42640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf14102260 0x7fbf1419c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:57.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.553+0000 7fbf18a42640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf14102260 0x7fbf1419c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57746/0 (socket says 172.21.15.90:57746) 2024-04-25T11:34:57.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.553+0000 7fbf18a42640 1 -- 172.21.15.90:0/3404419826 learned_addr learned my addr 172.21.15.90:0/3404419826 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:57.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.553+0000 7fbf19243640 1 --2- 172.21.15.90:0/3404419826 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbf14101760 0x7fbf1419c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:57.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.553+0000 7fbf19a44640 1 --2- 172.21.15.90:0/3404419826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf141050b0 0x7fbf141a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:57.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.554+0000 7fbf18a42640 1 -- 172.21.15.90:0/3404419826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf141050b0 msgr2=0x7fbf141a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:57.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.554+0000 7fbf18a42640 1 --2- 172.21.15.90:0/3404419826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf141050b0 0x7fbf141a3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.554+0000 7fbf18a42640 1 -- 172.21.15.90:0/3404419826 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbf14101760 msgr2=0x7fbf1419c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:57.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.554+0000 7fbf18a42640 1 --2- 172.21.15.90:0/3404419826 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbf14101760 0x7fbf1419c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.554+0000 7fbf18a42640 1 -- 172.21.15.90:0/3404419826 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf141a3fb0 con 0x7fbf14102260 2024-04-25T11:34:57.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.554+0000 7fbf19a44640 1 --2- 172.21.15.90:0/3404419826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf141050b0 0x7fbf141a3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:57.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.554+0000 7fbf19243640 1 --2- 172.21.15.90:0/3404419826 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbf14101760 0x7fbf1419c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:57.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.554+0000 7fbf18a42640 1 --2- 172.21.15.90:0/3404419826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf14102260 0x7fbf1419c9b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fbf0400d950 tx=0x7fbf0400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:57.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.554+0000 7fbf0a7fc640 1 -- 172.21.15.90:0/3404419826 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf04017070 con 0x7fbf14102260 2024-04-25T11:34:57.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.554+0000 7fbf0a7fc640 1 -- 172.21.15.90:0/3404419826 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf04009c30 con 0x7fbf14102260 2024-04-25T11:34:57.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.554+0000 7fbf0a7fc640 1 -- 172.21.15.90:0/3404419826 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0400f620 con 0x7fbf14102260 2024-04-25T11:34:57.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.554+0000 7fbf1b4a7640 1 -- 172.21.15.90:0/3404419826 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf141a42a0 con 0x7fbf14102260 2024-04-25T11:34:57.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.554+0000 7fbf1b4a7640 1 -- 172.21.15.90:0/3404419826 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf141a47f0 con 0x7fbf14102260 2024-04-25T11:34:57.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.555+0000 7fbf1b4a7640 1 -- 172.21.15.90:0/3404419826 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf141a4a50 con 0x7fbf14102260 2024-04-25T11:34:57.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.556+0000 7fbf0a7fc640 1 -- 172.21.15.90:0/3404419826 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fbf04010460 con 0x7fbf14102260 2024-04-25T11:34:57.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.557+0000 7fbf0a7fc640 1 --2- 172.21.15.90:0/3404419826 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbf000afaf0 0x7fbf000b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:57.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.557+0000 7fbf19243640 1 --2- 172.21.15.90:0/3404419826 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbf000afaf0 0x7fbf000b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:57.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.557+0000 7fbf0a7fc640 1 -- 172.21.15.90:0/3404419826 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7fbf040de960 con 0x7fbf14102260 2024-04-25T11:34:57.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.558+0000 7fbf19243640 1 --2- 172.21.15.90:0/3404419826 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbf000afaf0 0x7fbf000b2000 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbf10005f70 tx=0x7fbf10000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:57.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.562+0000 7fbf0a7fc640 1 -- 172.21.15.90:0/3404419826 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbf040a7520 con 0x7fbf14102260 2024-04-25T11:34:57.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.740+0000 7fbf1b4a7640 1 -- 172.21.15.90:0/3404419826 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbf141a4c20 con 0x7fbf14102260 2024-04-25T11:34:57.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.740+0000 7fbf0a7fc640 1 -- 172.21.15.90:0/3404419826 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) ==== 93+0+10 (secure 0 0 0) 0x7fbf04009de0 con 0x7fbf14102260 2024-04-25T11:34:57.729 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T11:34:57.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.742+0000 7fbf1b4a7640 1 -- 172.21.15.90:0/3404419826 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbf000afaf0 msgr2=0x7fbf000b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:57.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.742+0000 7fbf1b4a7640 1 --2- 172.21.15.90:0/3404419826 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbf000afaf0 0x7fbf000b2000 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbf10005f70 tx=0x7fbf10000bc0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.742+0000 7fbf1b4a7640 1 -- 172.21.15.90:0/3404419826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf14102260 msgr2=0x7fbf1419c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:57.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.742+0000 7fbf1b4a7640 1 --2- 172.21.15.90:0/3404419826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf14102260 0x7fbf1419c9b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fbf0400d950 tx=0x7fbf0400de10 comp rx=0 tx=0).stop 2024-04-25T11:34:57.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.742+0000 7fbf1b4a7640 1 -- 172.21.15.90:0/3404419826 shutdown_connections 2024-04-25T11:34:57.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.742+0000 7fbf1b4a7640 1 --2- 172.21.15.90:0/3404419826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf141050b0 0x7fbf141a3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.742+0000 7fbf1b4a7640 1 --2- 172.21.15.90:0/3404419826 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fbf000afaf0 0x7fbf000b2000 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.742+0000 7fbf1b4a7640 1 --2- 172.21.15.90:0/3404419826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf14102260 0x7fbf1419c9b0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.742+0000 7fbf1b4a7640 1 --2- 172.21.15.90:0/3404419826 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbf14101760 0x7fbf1419c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.742+0000 7fbf1b4a7640 1 -- 172.21.15.90:0/3404419826 >> 172.21.15.90:0/3404419826 conn(0x7fbf140fd380 msgr2=0x7fbf140ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:57.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.743+0000 7fbf1b4a7640 1 -- 172.21.15.90:0/3404419826 shutdown_connections 2024-04-25T11:34:57.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.743+0000 7fbf1b4a7640 1 -- 172.21.15.90:0/3404419826 wait complete. 2024-04-25T11:34:57.743 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T11:34:57.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.865+0000 7f6c07d00640 1 -- 172.21.15.90:0/395041625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c00102190 msgr2=0x7f6c00102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:57.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.865+0000 7f6c07d00640 1 --2- 172.21.15.90:0/395041625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c00102190 0x7f6c00102e90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc009900 tx=0x7f6bfc02a410 comp rx=0 tx=0).stop 2024-04-25T11:34:57.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.865+0000 7f6c07d00640 1 -- 172.21.15.90:0/395041625 shutdown_connections 2024-04-25T11:34:57.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.865+0000 7f6c07d00640 1 --2- 172.21.15.90:0/395041625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6c001080d0 0x7f6c0010a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.865+0000 7f6c07d00640 1 --2- 172.21.15.90:0/395041625 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6c001033d0 0x7f6c001037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.865+0000 7f6c07d00640 1 --2- 172.21.15.90:0/395041625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c00102190 0x7f6c00102e90 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.865+0000 7f6c07d00640 1 -- 172.21.15.90:0/395041625 >> 172.21.15.90:0/395041625 conn(0x7f6c000fd380 msgr2=0x7f6c000ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:57.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.865+0000 7f6c07d00640 1 -- 172.21.15.90:0/395041625 shutdown_connections 2024-04-25T11:34:57.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.865+0000 7f6c07d00640 1 -- 172.21.15.90:0/395041625 wait complete. 2024-04-25T11:34:57.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.866+0000 7f6c07d00640 1 Processor -- start 2024-04-25T11:34:57.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.866+0000 7f6c07d00640 1 -- start start 2024-04-25T11:34:57.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.866+0000 7f6c07d00640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6c00102190 0x7f6c0019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:57.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.866+0000 7f6c07d00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c001033d0 0x7f6c0019a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:57.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.866+0000 7f6c07d00640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6c001080d0 0x7f6c001a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:57.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.866+0000 7f6c07d00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0010f680 con 0x7f6c001033d0 2024-04-25T11:34:57.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.866+0000 7f6c07d00640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0010f500 con 0x7f6c00102190 2024-04-25T11:34:57.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.866+0000 7f6c07d00640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0010f800 con 0x7f6c001080d0 2024-04-25T11:34:57.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.866+0000 7f6c0529b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c001033d0 0x7f6c0019a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:57.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.866+0000 7f6c0529b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c001033d0 0x7f6c0019a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57764/0 (socket says 172.21.15.90:57764) 2024-04-25T11:34:57.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.866+0000 7f6c0529b640 1 -- 172.21.15.90:0/3924233990 learned_addr learned my addr 172.21.15.90:0/3924233990 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:57.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.866+0000 7f6c0629d640 1 --2- 172.21.15.90:0/3924233990 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6c001080d0 0x7f6c001a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:57.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.866+0000 7f6c05a9c640 1 --2- 172.21.15.90:0/3924233990 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6c00102190 0x7f6c0019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:57.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.866+0000 7f6c0529b640 1 -- 172.21.15.90:0/3924233990 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6c001080d0 msgr2=0x7f6c001a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:57.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.866+0000 7f6c0529b640 1 --2- 172.21.15.90:0/3924233990 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6c001080d0 0x7f6c001a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.866+0000 7f6c0529b640 1 -- 172.21.15.90:0/3924233990 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6c00102190 msgr2=0x7f6c0019a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:57.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.866+0000 7f6c0529b640 1 --2- 172.21.15.90:0/3924233990 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6c00102190 0x7f6c0019a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:57.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.866+0000 7f6c0529b640 1 -- 172.21.15.90:0/3924233990 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c001a1cd0 con 0x7f6c001033d0 2024-04-25T11:34:57.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.866+0000 7f6c05a9c640 1 --2- 172.21.15.90:0/3924233990 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6c00102190 0x7f6c0019a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:57.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.867+0000 7f6c0529b640 1 --2- 172.21.15.90:0/3924233990 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c001033d0 0x7f6c0019a780 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6bf000c920 tx=0x7f6bf000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:57.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.867+0000 7f6bf6ffd640 1 -- 172.21.15.90:0/3924233990 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf0007c70 con 0x7f6c001033d0 2024-04-25T11:34:57.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.867+0000 7f6bf6ffd640 1 -- 172.21.15.90:0/3924233990 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bf0007e10 con 0x7f6c001033d0 2024-04-25T11:34:57.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.867+0000 7f6c07d00640 1 -- 172.21.15.90:0/3924233990 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c001a1fc0 con 0x7f6c001033d0 2024-04-25T11:34:57.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.867+0000 7f6bf6ffd640 1 -- 172.21.15.90:0/3924233990 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf0018820 con 0x7f6c001033d0 2024-04-25T11:34:57.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.867+0000 7f6c07d00640 1 -- 172.21.15.90:0/3924233990 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c001a2420 con 0x7f6c001033d0 2024-04-25T11:34:57.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.868+0000 7f6c07d00640 1 -- 172.21.15.90:0/3924233990 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c00106ee0 con 0x7f6c001033d0 2024-04-25T11:34:57.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.870+0000 7f6bf6ffd640 1 -- 172.21.15.90:0/3924233990 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6bf000f450 con 0x7f6c001033d0 2024-04-25T11:34:57.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.870+0000 7f6bf6ffd640 1 --2- 172.21.15.90:0/3924233990 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6bec0afaf0 0x7f6bec0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:57.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.871+0000 7f6c05a9c640 1 --2- 172.21.15.90:0/3924233990 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6bec0afaf0 0x7f6bec0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:57.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.871+0000 7f6bf6ffd640 1 -- 172.21.15.90:0/3924233990 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f6bf00df000 con 0x7f6c001033d0 2024-04-25T11:34:57.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.871+0000 7f6c05a9c640 1 --2- 172.21.15.90:0/3924233990 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6bec0afaf0 0x7f6bec0b2000 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc02aa00 tx=0x7f6bfc031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:57.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:57.874+0000 7f6bf6ffd640 1 -- 172.21.15.90:0/3924233990 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6bf0014030 con 0x7f6c001033d0 2024-04-25T11:34:58.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.053+0000 7f6c07d00640 1 -- 172.21.15.90:0/3924233990 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6c000637d0 con 0x7f6c001033d0 2024-04-25T11:34:58.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.053+0000 7f6bf6ffd640 1 -- 172.21.15.90:0/3924233990 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v79) ==== 104+0+11 (secure 0 0 0) 0x7f6bf00a7c50 con 0x7f6c001033d0 2024-04-25T11:34:58.042 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T11:34:58.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.055+0000 7f6c07d00640 1 -- 172.21.15.90:0/3924233990 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6bec0afaf0 msgr2=0x7f6bec0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:58.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.055+0000 7f6c07d00640 1 --2- 172.21.15.90:0/3924233990 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6bec0afaf0 0x7f6bec0b2000 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc02aa00 tx=0x7f6bfc031400 comp rx=0 tx=0).stop 2024-04-25T11:34:58.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.055+0000 7f6c07d00640 1 -- 172.21.15.90:0/3924233990 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c001033d0 msgr2=0x7f6c0019a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:58.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.055+0000 7f6c07d00640 1 --2- 172.21.15.90:0/3924233990 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c001033d0 0x7f6c0019a780 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6bf000c920 tx=0x7f6bf000cde0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.055+0000 7f6c07d00640 1 -- 172.21.15.90:0/3924233990 shutdown_connections 2024-04-25T11:34:58.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.055+0000 7f6c07d00640 1 --2- 172.21.15.90:0/3924233990 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6c001080d0 0x7f6c001a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.055+0000 7f6c07d00640 1 --2- 172.21.15.90:0/3924233990 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f6bec0afaf0 0x7f6bec0b2000 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.055+0000 7f6c07d00640 1 --2- 172.21.15.90:0/3924233990 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c001033d0 0x7f6c0019a780 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.055+0000 7f6c07d00640 1 --2- 172.21.15.90:0/3924233990 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6c00102190 0x7f6c0019a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.055+0000 7f6c07d00640 1 -- 172.21.15.90:0/3924233990 >> 172.21.15.90:0/3924233990 conn(0x7f6c000fd380 msgr2=0x7f6c001040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:58.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.055+0000 7f6c07d00640 1 -- 172.21.15.90:0/3924233990 shutdown_connections 2024-04-25T11:34:58.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.055+0000 7f6c07d00640 1 -- 172.21.15.90:0/3924233990 wait complete. 2024-04-25T11:34:58.054 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:57 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3922058685' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:34:58.054 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:57 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3429381777' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:34:58.054 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:57 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3404419826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:34:58.055 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T11:34:58.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.177+0000 7f916e934640 1 -- 172.21.15.90:0/1239226545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9168102180 msgr2=0x7f9168102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:58.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.177+0000 7f916e934640 1 --2- 172.21.15.90:0/1239226545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9168102180 0x7f9168102e80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f915800aa10 tx=0x7f915802b720 comp rx=0 tx=0).stop 2024-04-25T11:34:58.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.177+0000 7f916e934640 1 -- 172.21.15.90:0/1239226545 shutdown_connections 2024-04-25T11:34:58.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.177+0000 7f916e934640 1 --2- 172.21.15.90:0/1239226545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91681080c0 0x7f916810a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.177+0000 7f916e934640 1 --2- 172.21.15.90:0/1239226545 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f91681033c0 0x7f91681037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.177+0000 7f916e934640 1 --2- 172.21.15.90:0/1239226545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9168102180 0x7f9168102e80 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.177+0000 7f916e934640 1 -- 172.21.15.90:0/1239226545 >> 172.21.15.90:0/1239226545 conn(0x7f91680fd350 msgr2=0x7f91680ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:58.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.177+0000 7f916e934640 1 -- 172.21.15.90:0/1239226545 shutdown_connections 2024-04-25T11:34:58.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.177+0000 7f916e934640 1 -- 172.21.15.90:0/1239226545 wait complete. 2024-04-25T11:34:58.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.177+0000 7f916e934640 1 Processor -- start 2024-04-25T11:34:58.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.177+0000 7f916e934640 1 -- start start 2024-04-25T11:34:58.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.178+0000 7f916e934640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9168102180 0x7f916819a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:58.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.178+0000 7f916e934640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f91681033c0 0x7f916819a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:58.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.178+0000 7f916e934640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91681080c0 0x7f91681a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:58.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.178+0000 7f916e934640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f916810d450 con 0x7f9168102180 2024-04-25T11:34:58.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.178+0000 7f916e934640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f916810d2d0 con 0x7f91681033c0 2024-04-25T11:34:58.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.178+0000 7f916e934640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f916810d5d0 con 0x7f91681080c0 2024-04-25T11:34:58.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.178+0000 7f9167fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9168102180 0x7f916819a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:58.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.178+0000 7f9167fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9168102180 0x7f916819a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57790/0 (socket says 172.21.15.90:57790) 2024-04-25T11:34:58.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.178+0000 7f9167fff640 1 -- 172.21.15.90:0/1064516884 learned_addr learned my addr 172.21.15.90:0/1064516884 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:58.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.178+0000 7f91677fe640 1 --2- 172.21.15.90:0/1064516884 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f91681033c0 0x7f916819a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:58.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.178+0000 7f9167fff640 1 -- 172.21.15.90:0/1064516884 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91681080c0 msgr2=0x7f91681a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:58.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.178+0000 7f916ced1640 1 --2- 172.21.15.90:0/1064516884 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91681080c0 0x7f91681a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:58.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.178+0000 7f9167fff640 1 --2- 172.21.15.90:0/1064516884 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91681080c0 0x7f91681a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.178+0000 7f9167fff640 1 -- 172.21.15.90:0/1064516884 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f91681033c0 msgr2=0x7f916819a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:58.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.178+0000 7f9167fff640 1 --2- 172.21.15.90:0/1064516884 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f91681033c0 0x7f916819a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.178+0000 7f9167fff640 1 -- 172.21.15.90:0/1064516884 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91681a1e10 con 0x7f9168102180 2024-04-25T11:34:58.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.178+0000 7f9167fff640 1 --2- 172.21.15.90:0/1064516884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9168102180 0x7f916819a2d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9158006010 tx=0x7f91580043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:58.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.178+0000 7f91657fa640 1 -- 172.21.15.90:0/1064516884 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f915803f070 con 0x7f9168102180 2024-04-25T11:34:58.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.179+0000 7f91657fa640 1 -- 172.21.15.90:0/1064516884 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9158032400 con 0x7f9168102180 2024-04-25T11:34:58.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.179+0000 7f916e934640 1 -- 172.21.15.90:0/1064516884 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91681a20a0 con 0x7f9168102180 2024-04-25T11:34:58.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.179+0000 7f91657fa640 1 -- 172.21.15.90:0/1064516884 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9158036040 con 0x7f9168102180 2024-04-25T11:34:58.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.179+0000 7f916e934640 1 -- 172.21.15.90:0/1064516884 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91681a2610 con 0x7f9168102180 2024-04-25T11:34:58.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.179+0000 7f916e934640 1 -- 172.21.15.90:0/1064516884 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9168106ed0 con 0x7f9168102180 2024-04-25T11:34:58.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.182+0000 7f91657fa640 1 -- 172.21.15.90:0/1064516884 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9158042070 con 0x7f9168102180 2024-04-25T11:34:58.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.182+0000 7f91657fa640 1 --2- 172.21.15.90:0/1064516884 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f91500afaf0 0x7f91500b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:58.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.182+0000 7f91677fe640 1 --2- 172.21.15.90:0/1064516884 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f91500afaf0 0x7f91500b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:58.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.182+0000 7f91657fa640 1 -- 172.21.15.90:0/1064516884 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f91580ff8e0 con 0x7f9168102180 2024-04-25T11:34:58.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.183+0000 7f91677fe640 1 --2- 172.21.15.90:0/1064516884 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f91500afaf0 0x7f91500b2000 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f915c005fd0 tx=0x7f915c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:58.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.186+0000 7f91657fa640 1 -- 172.21.15.90:0/1064516884 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f91580c84a0 con 0x7f9168102180 2024-04-25T11:34:58.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:57 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3922058685' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:34:58.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:57 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3429381777' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:34:58.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:57 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3404419826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:34:58.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:57 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3922058685' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:34:58.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:57 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3429381777' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:34:58.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:57 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3404419826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:34:58.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.362+0000 7f916e934640 1 -- 172.21.15.90:0/1064516884 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f91680637a0 con 0x7f9168102180 2024-04-25T11:34:58.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.362+0000 7f91657fa640 1 -- 172.21.15.90:0/1064516884 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v79) ==== 100+0+11 (secure 0 0 0) 0x7f91580c7da0 con 0x7f9168102180 2024-04-25T11:34:58.351 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T11:34:58.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.364+0000 7f916e934640 1 -- 172.21.15.90:0/1064516884 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f91500afaf0 msgr2=0x7f91500b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:58.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.364+0000 7f916e934640 1 --2- 172.21.15.90:0/1064516884 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f91500afaf0 0x7f91500b2000 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f915c005fd0 tx=0x7f915c005850 comp rx=0 tx=0).stop 2024-04-25T11:34:58.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.364+0000 7f916e934640 1 -- 172.21.15.90:0/1064516884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9168102180 msgr2=0x7f916819a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:58.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.364+0000 7f916e934640 1 --2- 172.21.15.90:0/1064516884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9168102180 0x7f916819a2d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9158006010 tx=0x7f91580043b0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.364+0000 7f916e934640 1 -- 172.21.15.90:0/1064516884 shutdown_connections 2024-04-25T11:34:58.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.365+0000 7f916e934640 1 --2- 172.21.15.90:0/1064516884 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91681080c0 0x7f91681a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.365+0000 7f916e934640 1 --2- 172.21.15.90:0/1064516884 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f91681033c0 0x7f916819a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.365+0000 7f916e934640 1 --2- 172.21.15.90:0/1064516884 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f91500afaf0 0x7f91500b2000 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.365+0000 7f916e934640 1 --2- 172.21.15.90:0/1064516884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9168102180 0x7f916819a2d0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.365+0000 7f916e934640 1 -- 172.21.15.90:0/1064516884 >> 172.21.15.90:0/1064516884 conn(0x7f91680fd350 msgr2=0x7f91681040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:58.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.365+0000 7f916e934640 1 -- 172.21.15.90:0/1064516884 shutdown_connections 2024-04-25T11:34:58.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.365+0000 7f916e934640 1 -- 172.21.15.90:0/1064516884 wait complete. 2024-04-25T11:34:58.365 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:34:58.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.483+0000 7fe1a5c51640 1 -- 172.21.15.90:0/1766042394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1a0104d40 msgr2=0x7fe1a0105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:58.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.483+0000 7fe1a5c51640 1 --2- 172.21.15.90:0/1766042394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1a0104d40 0x7fe1a0105150 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fe194009900 tx=0x7fe19402a410 comp rx=0 tx=0).stop 2024-04-25T11:34:58.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.483+0000 7fe1a5c51640 1 -- 172.21.15.90:0/1766042394 shutdown_connections 2024-04-25T11:34:58.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.483+0000 7fe1a5c51640 1 --2- 172.21.15.90:0/1766042394 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe1a0109a40 0x7fe1a010be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.483+0000 7fe1a5c51640 1 --2- 172.21.15.90:0/1766042394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1a0104d40 0x7fe1a0105150 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.483+0000 7fe1a5c51640 1 --2- 172.21.15.90:0/1766042394 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1a0103b00 0x7fe1a0104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.483+0000 7fe1a5c51640 1 -- 172.21.15.90:0/1766042394 >> 172.21.15.90:0/1766042394 conn(0x7fe1a00fed10 msgr2=0x7fe1a01011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:58.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.483+0000 7fe1a5c51640 1 -- 172.21.15.90:0/1766042394 shutdown_connections 2024-04-25T11:34:58.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.483+0000 7fe1a5c51640 1 -- 172.21.15.90:0/1766042394 wait complete. 2024-04-25T11:34:58.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.483+0000 7fe1a5c51640 1 Processor -- start 2024-04-25T11:34:58.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.484+0000 7fe1a5c51640 1 -- start start 2024-04-25T11:34:58.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.484+0000 7fe1a5c51640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe1a0103b00 0x7fe1a019a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:58.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.484+0000 7fe1a5c51640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1a0104d40 0x7fe1a019a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:58.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.484+0000 7fe1a5c51640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1a0109a40 0x7fe1a01a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:58.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.484+0000 7fe1a5c51640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a010f620 con 0x7fe1a0109a40 2024-04-25T11:34:58.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.484+0000 7fe1a5c51640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a010f4a0 con 0x7fe1a0103b00 2024-04-25T11:34:58.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.484+0000 7fe1a5c51640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a010f7a0 con 0x7fe1a0104d40 2024-04-25T11:34:58.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.484+0000 7fe19ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1a0109a40 0x7fe1a01a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:58.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.484+0000 7fe19f7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe1a0103b00 0x7fe1a019a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:58.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.484+0000 7fe19ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1a0109a40 0x7fe1a01a16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57806/0 (socket says 172.21.15.90:57806) 2024-04-25T11:34:58.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.484+0000 7fe19ffff640 1 -- 172.21.15.90:0/3997845434 learned_addr learned my addr 172.21.15.90:0/3997845434 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:58.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.484+0000 7fe19effd640 1 --2- 172.21.15.90:0/3997845434 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1a0104d40 0x7fe1a019a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:58.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.484+0000 7fe19ffff640 1 -- 172.21.15.90:0/3997845434 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1a0104d40 msgr2=0x7fe1a019a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:58.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.484+0000 7fe19ffff640 1 --2- 172.21.15.90:0/3997845434 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1a0104d40 0x7fe1a019a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.484+0000 7fe19ffff640 1 -- 172.21.15.90:0/3997845434 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe1a0103b00 msgr2=0x7fe1a019a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:58.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.484+0000 7fe19ffff640 1 --2- 172.21.15.90:0/3997845434 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe1a0103b00 0x7fe1a019a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.484+0000 7fe19ffff640 1 -- 172.21.15.90:0/3997845434 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1a01a1d60 con 0x7fe1a0109a40 2024-04-25T11:34:58.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.484+0000 7fe19effd640 1 --2- 172.21.15.90:0/3997845434 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1a0104d40 0x7fe1a019a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:58.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.484+0000 7fe19ffff640 1 --2- 172.21.15.90:0/3997845434 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1a0109a40 0x7fe1a01a16f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fe19800eb10 tx=0x7fe19800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:58.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.485+0000 7fe19cff9640 1 -- 172.21.15.90:0/3997845434 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe198004b20 con 0x7fe1a0109a40 2024-04-25T11:34:58.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.485+0000 7fe19cff9640 1 -- 172.21.15.90:0/3997845434 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe198004cc0 con 0x7fe1a0109a40 2024-04-25T11:34:58.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.485+0000 7fe1a5c51640 1 -- 172.21.15.90:0/3997845434 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1a01a1ff0 con 0x7fe1a0109a40 2024-04-25T11:34:58.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.485+0000 7fe19cff9640 1 -- 172.21.15.90:0/3997845434 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe198010670 con 0x7fe1a0109a40 2024-04-25T11:34:58.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.485+0000 7fe1a5c51640 1 -- 172.21.15.90:0/3997845434 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1a01a2460 con 0x7fe1a0109a40 2024-04-25T11:34:58.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.486+0000 7fe1a5c51640 1 -- 172.21.15.90:0/3997845434 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1a0108850 con 0x7fe1a0109a40 2024-04-25T11:34:58.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.487+0000 7fe19cff9640 1 -- 172.21.15.90:0/3997845434 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe198005000 con 0x7fe1a0109a40 2024-04-25T11:34:58.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.487+0000 7fe19cff9640 1 --2- 172.21.15.90:0/3997845434 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fe1880afaf0 0x7fe1880b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:58.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.488+0000 7fe19f7fe640 1 --2- 172.21.15.90:0/3997845434 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fe1880afaf0 0x7fe1880b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:58.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.488+0000 7fe19cff9640 1 -- 172.21.15.90:0/3997845434 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7fe1980df430 con 0x7fe1a0109a40 2024-04-25T11:34:58.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.488+0000 7fe19f7fe640 1 --2- 172.21.15.90:0/3997845434 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fe1880afaf0 0x7fe1880b2000 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe190005fd0 tx=0x7fe19000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:58.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.491+0000 7fe19cff9640 1 -- 172.21.15.90:0/3997845434 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe198053ff0 con 0x7fe1a0109a40 2024-04-25T11:34:58.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.710+0000 7fe1a5c51640 1 -- 172.21.15.90:0/3997845434 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1a002db50 con 0x7fe1a0109a40 2024-04-25T11:34:58.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.711+0000 7fe19cff9640 1 -- 172.21.15.90:0/3997845434 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11930 (secure 0 0 0) 0x7fe1980a7ff0 con 0x7fe1a0109a40 2024-04-25T11:34:58.700 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:34:58.700 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:33:16.640084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524,24475,24499,24458,24431]},"id":1}]} 2024-04-25T11:34:58.700 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 25 2024-04-25T11:34:58.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.713+0000 7fe1a5c51640 1 -- 172.21.15.90:0/3997845434 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fe1880afaf0 msgr2=0x7fe1880b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:58.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.713+0000 7fe1a5c51640 1 --2- 172.21.15.90:0/3997845434 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fe1880afaf0 0x7fe1880b2000 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe190005fd0 tx=0x7fe19000b390 comp rx=0 tx=0).stop 2024-04-25T11:34:58.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.713+0000 7fe1a5c51640 1 -- 172.21.15.90:0/3997845434 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1a0109a40 msgr2=0x7fe1a01a16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:58.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.713+0000 7fe1a5c51640 1 --2- 172.21.15.90:0/3997845434 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1a0109a40 0x7fe1a01a16f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fe19800eb10 tx=0x7fe19800c2d0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.713+0000 7fe1a5c51640 1 -- 172.21.15.90:0/3997845434 shutdown_connections 2024-04-25T11:34:58.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.713+0000 7fe1a5c51640 1 --2- 172.21.15.90:0/3997845434 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fe1880afaf0 0x7fe1880b2000 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.713+0000 7fe1a5c51640 1 --2- 172.21.15.90:0/3997845434 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1a0109a40 0x7fe1a01a16f0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.713+0000 7fe1a5c51640 1 --2- 172.21.15.90:0/3997845434 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1a0104d40 0x7fe1a019a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.713+0000 7fe1a5c51640 1 --2- 172.21.15.90:0/3997845434 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe1a0103b00 0x7fe1a019a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.713+0000 7fe1a5c51640 1 -- 172.21.15.90:0/3997845434 >> 172.21.15.90:0/3997845434 conn(0x7fe1a00fed10 msgr2=0x7fe1a0105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:58.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.713+0000 7fe1a5c51640 1 -- 172.21.15.90:0/3997845434 shutdown_connections 2024-04-25T11:34:58.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.713+0000 7fe1a5c51640 1 -- 172.21.15.90:0/3997845434 wait complete. 2024-04-25T11:34:58.714 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:34:58.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.834+0000 7fa79a75f640 1 -- 172.21.15.90:0/1140845137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa794104d40 msgr2=0x7fa794105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:58.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.834+0000 7fa79a75f640 1 --2- 172.21.15.90:0/1140845137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa794104d40 0x7fa794105150 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa788009900 tx=0x7fa78802a410 comp rx=0 tx=0).stop 2024-04-25T11:34:58.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.834+0000 7fa79a75f640 1 -- 172.21.15.90:0/1140845137 shutdown_connections 2024-04-25T11:34:58.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.834+0000 7fa79a75f640 1 --2- 172.21.15.90:0/1140845137 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa794109a40 0x7fa79410be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.834+0000 7fa79a75f640 1 --2- 172.21.15.90:0/1140845137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa794104d40 0x7fa794105150 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.834+0000 7fa79a75f640 1 --2- 172.21.15.90:0/1140845137 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa794103b00 0x7fa794104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.834+0000 7fa79a75f640 1 -- 172.21.15.90:0/1140845137 >> 172.21.15.90:0/1140845137 conn(0x7fa7940fed10 msgr2=0x7fa7941011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:58.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.834+0000 7fa79a75f640 1 -- 172.21.15.90:0/1140845137 shutdown_connections 2024-04-25T11:34:58.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.834+0000 7fa79a75f640 1 -- 172.21.15.90:0/1140845137 wait complete. 2024-04-25T11:34:58.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.835+0000 7fa79a75f640 1 Processor -- start 2024-04-25T11:34:58.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.835+0000 7fa79a75f640 1 -- start start 2024-04-25T11:34:58.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.835+0000 7fa79a75f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa794103b00 0x7fa79419a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:58.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.835+0000 7fa79a75f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa794104d40 0x7fa79419a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:58.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.835+0000 7fa79a75f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa794109a40 0x7fa7941a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:58.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.835+0000 7fa79a75f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa79410f6a0 con 0x7fa794103b00 2024-04-25T11:34:58.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.835+0000 7fa79a75f640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa79410f520 con 0x7fa794109a40 2024-04-25T11:34:58.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.835+0000 7fa79a75f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa79410f820 con 0x7fa794104d40 2024-04-25T11:34:58.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.835+0000 7fa793fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa794103b00 0x7fa79419a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:58.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.835+0000 7fa793fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa794103b00 0x7fa79419a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57824/0 (socket says 172.21.15.90:57824) 2024-04-25T11:34:58.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.835+0000 7fa793fff640 1 -- 172.21.15.90:0/342937216 learned_addr learned my addr 172.21.15.90:0/342937216 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:58.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.835+0000 7fa7937fe640 1 --2- 172.21.15.90:0/342937216 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa794104d40 0x7fa79419a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:58.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.835+0000 7fa798cfc640 1 --2- 172.21.15.90:0/342937216 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa794109a40 0x7fa7941a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:58.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.835+0000 7fa793fff640 1 -- 172.21.15.90:0/342937216 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa794104d40 msgr2=0x7fa79419a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:58.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.835+0000 7fa793fff640 1 --2- 172.21.15.90:0/342937216 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa794104d40 0x7fa79419a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.835+0000 7fa793fff640 1 -- 172.21.15.90:0/342937216 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa794109a40 msgr2=0x7fa7941a1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:58.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.835+0000 7fa793fff640 1 --2- 172.21.15.90:0/342937216 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa794109a40 0x7fa7941a1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:58.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.835+0000 7fa793fff640 1 -- 172.21.15.90:0/342937216 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7941a1de0 con 0x7fa794103b00 2024-04-25T11:34:58.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.836+0000 7fa793fff640 1 --2- 172.21.15.90:0/342937216 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa794103b00 0x7fa79419a2a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa784005d00 tx=0x7fa78400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:58.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.836+0000 7fa7917fa640 1 -- 172.21.15.90:0/342937216 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa784017070 con 0x7fa794103b00 2024-04-25T11:34:58.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.836+0000 7fa7917fa640 1 -- 172.21.15.90:0/342937216 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa784009c30 con 0x7fa794103b00 2024-04-25T11:34:58.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.836+0000 7fa79a75f640 1 -- 172.21.15.90:0/342937216 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7941a20d0 con 0x7fa794103b00 2024-04-25T11:34:58.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.836+0000 7fa7917fa640 1 -- 172.21.15.90:0/342937216 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa78400f620 con 0x7fa794103b00 2024-04-25T11:34:58.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.836+0000 7fa79a75f640 1 -- 172.21.15.90:0/342937216 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7941a26a0 con 0x7fa794103b00 2024-04-25T11:34:58.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.837+0000 7fa79a75f640 1 -- 172.21.15.90:0/342937216 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa794108850 con 0x7fa794103b00 2024-04-25T11:34:58.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.839+0000 7fa7917fa640 1 -- 172.21.15.90:0/342937216 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa78401a070 con 0x7fa794103b00 2024-04-25T11:34:58.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.839+0000 7fa7917fa640 1 --2- 172.21.15.90:0/342937216 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fa77c0afb70 0x7fa77c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:58.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.839+0000 7fa7937fe640 1 --2- 172.21.15.90:0/342937216 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fa77c0afb70 0x7fa77c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:58.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.839+0000 7fa7917fa640 1 -- 172.21.15.90:0/342937216 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7fa7840de6b0 con 0x7fa794103b00 2024-04-25T11:34:58.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.840+0000 7fa7937fe640 1 --2- 172.21.15.90:0/342937216 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fa77c0afb70 0x7fa77c0b2080 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa788000c00 tx=0x7fa788031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:58.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:58.843+0000 7fa7917fa640 1 -- 172.21.15.90:0/342937216 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa784009de0 con 0x7fa794103b00 2024-04-25T11:34:58.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:58 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3924233990' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:34:58.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:58 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1064516884' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:34:58.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:58 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3997845434' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:34:58.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:58 smithi090 ceph-mon[29575]: pgmap v330: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-25T11:34:59.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.023+0000 7fa79a75f640 1 -- 172.21.15.90:0/342937216 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa7940637a0 con 0x7fa794103b00 2024-04-25T11:34:59.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.024+0000 7fa7917fa640 1 -- 172.21.15.90:0/342937216 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+22706 (secure 0 0 0) 0x7fa7840a72f0 con 0x7fa794103b00 2024-04-25T11:34:59.014 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:34:59.014 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":79,"fsid":"58bd4e7a-02f6-11ef-bc93-c7b262605968","created":"2024-04-25T11:25:23.574550+0000","modified":"2024-04-25T11:34:17.871834+0000","last_up_change":"2024-04-25T11:32:23.852211+0000","last_in_change":"2024-04-25T11:32:04.188507+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:28:42.746649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:33:08.442925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:33:09.352886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e54d888-e67b-4951-8ae7-e355fb6b10a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6803","nonce":1201629159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6805","nonce":1201629159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6809","nonce":1201629159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6807","nonce":1201629159}]},"public_addr":"172.21.15.90:6803/1201629159","cluster_addr":"172.21.15.90:6805/1201629159","heartbeat_back_addr":"172.21.15.90:6809/1201629159","heartbeat_front_addr":"172.21.15.90:6807/1201629159","state":["exists","up"]},{"osd":1,"uuid":"91a71c78-dae9-4bc7-9f6f-9e55cfefbe28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6801","nonce":3976851933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6803","nonce":3976851933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6807","nonce":3976851933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6805","nonce":3976851933}]},"public_addr":"172.21.15.138:6801/3976851933","cluster_addr":"172.21.15.138:6803/3976851933","heartbeat_back_addr":"172.21.15.138:6807/3976851933","heartbeat_front_addr":"172.21.15.138:6805/3976851933","state":["exists","up"]},{"osd":2,"uuid":"daff6c7f-80fa-4859-9339-ef7a5e840832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6801","nonce":1505807670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6803","nonce":1505807670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6807","nonce":1505807670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6805","nonce":1505807670}]},"public_addr":"172.21.15.156:6801/1505807670","cluster_addr":"172.21.15.156:6803/1505807670","heartbeat_back_addr":"172.21.15.156:6807/1505807670","heartbeat_front_addr":"172.21.15.156:6805/1505807670","state":["exists","up"]},{"osd":3,"uuid":"33fd94ca-7465-401a-bc1e-706b6b31b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6811","nonce":2520012383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6813","nonce":2520012383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6817","nonce":2520012383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6815","nonce":2520012383}]},"public_addr":"172.21.15.90:6811/2520012383","cluster_addr":"172.21.15.90:6813/2520012383","heartbeat_back_addr":"172.21.15.90:6817/2520012383","heartbeat_front_addr":"172.21.15.90:6815/2520012383","state":["exists","up"]},{"osd":4,"uuid":"c5326212-750c-49fd-96e3-d1e62bb4b7f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6809","nonce":3752678279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6811","nonce":3752678279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6815","nonce":3752678279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6813","nonce":3752678279}]},"public_addr":"172.21.15.138:6809/3752678279","cluster_addr":"172.21.15.138:6811/3752678279","heartbeat_back_addr":"172.21.15.138:6815/3752678279","heartbeat_front_addr":"172.21.15.138:6813/3752678279","state":["exists","up"]},{"osd":5,"uuid":"d04a1156-820b-4b43-be89-e16a8910c3aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6809","nonce":3591210191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6811","nonce":3591210191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6815","nonce":3591210191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6813","nonce":3591210191}]},"public_addr":"172.21.15.156:6809/3591210191","cluster_addr":"172.21.15.156:6811/3591210191","heartbeat_back_addr":"172.21.15.156:6815/3591210191","heartbeat_front_addr":"172.21.15.156:6813/3591210191","state":["exists","up"]},{"osd":6,"uuid":"75675aa3-e978-490c-8205-13d9868a3d6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6819","nonce":2364955456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6821","nonce":2364955456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6825","nonce":2364955456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6823","nonce":2364955456}]},"public_addr":"172.21.15.90:6819/2364955456","cluster_addr":"172.21.15.90:6821/2364955456","heartbeat_back_addr":"172.21.15.90:6825/2364955456","heartbeat_front_addr":"172.21.15.90:6823/2364955456","state":["exists","up"]},{"osd":7,"uuid":"1a95fdf5-f548-4271-bbcd-e330f7955eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6817","nonce":4012476387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6819","nonce":4012476387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6823","nonce":4012476387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6821","nonce":4012476387}]},"public_addr":"172.21.15.138:6817/4012476387","cluster_addr":"172.21.15.138:6819/4012476387","heartbeat_back_addr":"172.21.15.138:6823/4012476387","heartbeat_front_addr":"172.21.15.138:6821/4012476387","state":["exists","up"]},{"osd":8,"uuid":"605e10b5-5806-4c1f-8e28-a7994d34ddce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6817","nonce":101092224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6819","nonce":101092224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6823","nonce":101092224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6821","nonce":101092224}]},"public_addr":"172.21.15.156:6817/101092224","cluster_addr":"172.21.15.156:6819/101092224","heartbeat_back_addr":"172.21.15.156:6823/101092224","heartbeat_front_addr":"172.21.15.156:6821/101092224","state":["exists","up"]},{"osd":9,"uuid":"d70fdf5e-9d0f-483d-a7d8-505b79b8a418","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6827","nonce":1171265966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6829","nonce":1171265966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6833","nonce":1171265966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6831","nonce":1171265966}]},"public_addr":"172.21.15.90:6827/1171265966","cluster_addr":"172.21.15.90:6829/1171265966","heartbeat_back_addr":"172.21.15.90:6833/1171265966","heartbeat_front_addr":"172.21.15.90:6831/1171265966","state":["exists","up"]},{"osd":10,"uuid":"1ec86c7d-c3a4-432d-9664-8e40836e1e20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6825","nonce":4264666232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6827","nonce":4264666232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6831","nonce":4264666232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6829","nonce":4264666232}]},"public_addr":"172.21.15.138:6825/4264666232","cluster_addr":"172.21.15.138:6827/4264666232","heartbeat_back_addr":"172.21.15.138:6831/4264666232","heartbeat_front_addr":"172.21.15.138:6829/4264666232","state":["exists","up"]},{"osd":11,"uuid":"51cbe729-292e-4d7b-b8e9-e1e2821ffe76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6825","nonce":1224251744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6827","nonce":1224251744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6831","nonce":1224251744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6829","nonce":1224251744}]},"public_addr":"172.21.15.156:6825/1224251744","cluster_addr":"172.21.15.156:6827/1224251744","heartbeat_back_addr":"172.21.15.156:6831/1224251744","heartbeat_front_addr":"172.21.15.156:6829/1224251744","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:27:50.794226+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:15.030278+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:38.518163+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:03.107351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:27.500707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:51.463545+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:16.015076+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:41.182071+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:05.362309+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:31.048936+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:56.488509+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:32:22.129178+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":7}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/2381633953":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6800/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/2190846900":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1707792941":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6800/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6801/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/6061052":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1849633392":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6801/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/2277490702":"2024-04-26T11:26:16.655278+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:34:59.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.026+0000 7fa79a75f640 1 -- 172.21.15.90:0/342937216 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fa77c0afb70 msgr2=0x7fa77c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.026+0000 7fa79a75f640 1 --2- 172.21.15.90:0/342937216 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fa77c0afb70 0x7fa77c0b2080 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa788000c00 tx=0x7fa788031400 comp rx=0 tx=0).stop 2024-04-25T11:34:59.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.026+0000 7fa79a75f640 1 -- 172.21.15.90:0/342937216 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa794103b00 msgr2=0x7fa79419a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.026+0000 7fa79a75f640 1 --2- 172.21.15.90:0/342937216 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa794103b00 0x7fa79419a2a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa784005d00 tx=0x7fa78400dde0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.026+0000 7fa79a75f640 1 -- 172.21.15.90:0/342937216 shutdown_connections 2024-04-25T11:34:59.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.026+0000 7fa79a75f640 1 --2- 172.21.15.90:0/342937216 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa794109a40 0x7fa7941a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.026+0000 7fa79a75f640 1 --2- 172.21.15.90:0/342937216 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa794104d40 0x7fa79419a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.026+0000 7fa79a75f640 1 --2- 172.21.15.90:0/342937216 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fa77c0afb70 0x7fa77c0b2080 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.026+0000 7fa79a75f640 1 --2- 172.21.15.90:0/342937216 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa794103b00 0x7fa79419a2a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.026+0000 7fa79a75f640 1 -- 172.21.15.90:0/342937216 >> 172.21.15.90:0/342937216 conn(0x7fa7940fed10 msgr2=0x7fa794105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.027+0000 7fa79a75f640 1 -- 172.21.15.90:0/342937216 shutdown_connections 2024-04-25T11:34:59.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.027+0000 7fa79a75f640 1 -- 172.21.15.90:0/342937216 wait complete. 2024-04-25T11:34:59.027 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T11:33:10.373468+0000', 'modified': '2024-04-25T11:33:16.640084+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24524, 'mds_1': 24499, 'mds_2': 24475, 'mds_3': 24458, 'mds_4': 24431}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14637': {'gid': 14637, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.90:6839/3101160958', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6838', 'nonce': 3101160958}, {'type': 'v1', 'addr': '172.21.15.90:6839', 'nonce': 3101160958}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14649': {'gid': 14649, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.90:6841/2617461073', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6840', 'nonce': 2617461073}, {'type': 'v1', 'addr': '172.21.15.90:6841', 'nonce': 2617461073}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24430': {'gid': 24430, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.156:6833/31237375', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6832', 'nonce': 31237375}, {'type': 'v1', 'addr': '172.21.15.156:6833', 'nonce': 31237375}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24431': {'gid': 24431, 'name': 'k', 'rank': 4, 'incarnation': 23, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.138:6833/51785544', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6832', 'nonce': 51785544}, {'type': 'v1', 'addr': '172.21.15.138:6833', 'nonce': 51785544}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24458': {'gid': 24458, 'name': 'e', 'rank': 3, 'incarnation': 21, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.138:6835/162690028', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6834', 'nonce': 162690028}, {'type': 'v1', 'addr': '172.21.15.138:6835', 'nonce': 162690028}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24475': {'gid': 24475, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.156:6835/1641860251', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6834', 'nonce': 1641860251}, {'type': 'v1', 'addr': '172.21.15.156:6835', 'nonce': 1641860251}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24481': {'gid': 24481, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.156:6837/3466257977', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6836', 'nonce': 3466257977}, {'type': 'v1', 'addr': '172.21.15.156:6837', 'nonce': 3466257977}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24485': {'gid': 24485, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.138:6837/2598829089', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6836', 'nonce': 2598829089}, {'type': 'v1', 'addr': '172.21.15.138:6837', 'nonce': 2598829089}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'i', 'rank': 1, 'incarnation': 17, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.156:6839/4179744597', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6838', 'nonce': 4179744597}, {'type': 'v1', 'addr': '172.21.15.156:6839', 'nonce': 4179744597}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24524': {'gid': 24524, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.138:6839/1849836338', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6838', 'nonce': 1849836338}, {'type': 'v1', 'addr': '172.21.15.138:6839', 'nonce': 1849836338}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24524, 'qdb_cluster': [24524, 24475, 24499, 24458, 24431]} 2024-04-25T11:34:59.027 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-25T11:34:59.028 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.k status 2024-04-25T11:34:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.148+0000 7f5607ab6640 1 -- 172.21.15.90:0/3513053778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5600106c60 msgr2=0x7f56001090e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.148+0000 7f5607ab6640 1 --2- 172.21.15.90:0/3513053778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5600106c60 0x7f56001090e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f55f0009900 tx=0x7f55f002a410 comp rx=0 tx=0).stop 2024-04-25T11:34:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.148+0000 7f5607ab6640 1 -- 172.21.15.90:0/3513053778 shutdown_connections 2024-04-25T11:34:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.148+0000 7f5607ab6640 1 --2- 172.21.15.90:0/3513053778 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5600109770 0x7f560010bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.148+0000 7f5607ab6640 1 --2- 172.21.15.90:0/3513053778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5600106c60 0x7f56001090e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.148+0000 7f5607ab6640 1 --2- 172.21.15.90:0/3513053778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5600104270 0x7f5600106690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.148+0000 7f5607ab6640 1 -- 172.21.15.90:0/3513053778 >> 172.21.15.90:0/3513053778 conn(0x7f56000fdbd0 msgr2=0x7f5600100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.148+0000 7f5607ab6640 1 -- 172.21.15.90:0/3513053778 shutdown_connections 2024-04-25T11:34:59.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.148+0000 7f5607ab6640 1 -- 172.21.15.90:0/3513053778 wait complete. 2024-04-25T11:34:59.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.148+0000 7f5607ab6640 1 Processor -- start 2024-04-25T11:34:59.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.148+0000 7f5607ab6640 1 -- start start 2024-04-25T11:34:59.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.148+0000 7f5607ab6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5600104270 0x7f560019a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.148+0000 7f5607ab6640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5600106c60 0x7f560019aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.148+0000 7f5607ab6640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5600109770 0x7f56001a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.148+0000 7f5607ab6640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f560010f670 con 0x7f5600109770 2024-04-25T11:34:59.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.148+0000 7f5607ab6640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f560010f4f0 con 0x7f5600106c60 2024-04-25T11:34:59.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.148+0000 7f5607ab6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f560010f7f0 con 0x7f5600104270 2024-04-25T11:34:59.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.149+0000 7f5606053640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5600109770 0x7f56001a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.149+0000 7f5606053640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5600109770 0x7f56001a1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57838/0 (socket says 172.21.15.90:57838) 2024-04-25T11:34:59.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.149+0000 7f5606053640 1 -- 172.21.15.90:0/3322914767 learned_addr learned my addr 172.21.15.90:0/3322914767 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:59.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.149+0000 7f5605852640 1 --2- 172.21.15.90:0/3322914767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5600104270 0x7f560019a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.149+0000 7f5606053640 1 -- 172.21.15.90:0/3322914767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5600104270 msgr2=0x7f560019a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.149+0000 7f5606053640 1 --2- 172.21.15.90:0/3322914767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5600104270 0x7f560019a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.149+0000 7f5606053640 1 -- 172.21.15.90:0/3322914767 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5600106c60 msgr2=0x7f560019aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.149+0000 7f5605051640 1 --2- 172.21.15.90:0/3322914767 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5600106c60 0x7f560019aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.149+0000 7f5606053640 1 --2- 172.21.15.90:0/3322914767 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5600106c60 0x7f560019aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.149+0000 7f5606053640 1 -- 172.21.15.90:0/3322914767 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56001a20d0 con 0x7f5600109770 2024-04-25T11:34:59.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.149+0000 7f5606053640 1 --2- 172.21.15.90:0/3322914767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5600109770 0x7f56001a1a60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f55fc009fd0 tx=0x7f55fc00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.149+0000 7f55f6ffd640 1 -- 172.21.15.90:0/3322914767 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55fc011830 con 0x7f5600109770 2024-04-25T11:34:59.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.149+0000 7f5607ab6640 1 -- 172.21.15.90:0/3322914767 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56001a23c0 con 0x7f5600109770 2024-04-25T11:34:59.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.149+0000 7f55f6ffd640 1 -- 172.21.15.90:0/3322914767 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55fc016070 con 0x7f5600109770 2024-04-25T11:34:59.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.149+0000 7f55f6ffd640 1 -- 172.21.15.90:0/3322914767 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55fc010de0 con 0x7f5600109770 2024-04-25T11:34:59.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.149+0000 7f5607ab6640 1 -- 172.21.15.90:0/3322914767 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56001a2940 con 0x7f5600109770 2024-04-25T11:34:59.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.152+0000 7f560e96f740 1 Processor -- start 2024-04-25T11:34:59.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.152+0000 7f55f6ffd640 1 -- 172.21.15.90:0/3322914767 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f55fc01b460 con 0x7f5600109770 2024-04-25T11:34:59.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.152+0000 7f560e96f740 1 -- start start 2024-04-25T11:34:59.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.152+0000 7f560e96f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f91124270 0x559f914c5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.152+0000 7f560e96f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559f910fb3b0 0x559f914c58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.152+0000 7f560e96f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f91103590 0x559f914cc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.152+0000 7f560e96f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559f9142a740 con 0x559f91103590 2024-04-25T11:34:59.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.152+0000 7f560e96f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x559f913c63a0 con 0x559f910fb3b0 2024-04-25T11:34:59.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.152+0000 7f560e96f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559f91426700 con 0x559f91124270 2024-04-25T11:34:59.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.153+0000 7f5605051640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f91103590 0x559f914cc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.153+0000 7f5605051640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f91103590 0x559f914cc7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57844/0 (socket says 172.21.15.90:57844) 2024-04-25T11:34:59.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.153+0000 7f5605051640 1 -- 172.21.15.90:0/3826591599 learned_addr learned my addr 172.21.15.90:0/3826591599 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:59.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.153+0000 7f5605852640 1 --2- 172.21.15.90:0/3826591599 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559f910fb3b0 0x559f914c58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.153+0000 7f55f6ffd640 1 --2- 172.21.15.90:0/3322914767 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f55ec0afaf0 0x7f55ec0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.153+0000 7f5605051640 1 --2- 172.21.15.90:0/3826591599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f91124270 0x559f914c5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.153+0000 7f5606053640 1 --2- 172.21.15.90:0/3322914767 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f55ec0afaf0 0x7f55ec0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.153+0000 7f55f6ffd640 1 -- 172.21.15.90:0/3322914767 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f55fc0de350 con 0x7f5600109770 2024-04-25T11:34:59.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.153+0000 7f5605051640 1 -- 172.21.15.90:0/3826591599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f91124270 msgr2=0x559f914c5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.153+0000 7f5605051640 1 --2- 172.21.15.90:0/3826591599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f91124270 0x559f914c5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.153+0000 7f5605051640 1 -- 172.21.15.90:0/3826591599 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559f910fb3b0 msgr2=0x559f914c58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.153+0000 7f5605051640 1 --2- 172.21.15.90:0/3826591599 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559f910fb3b0 0x559f914c58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.153+0000 7f5605051640 1 -- 172.21.15.90:0/3826591599 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f913e2750 con 0x559f91103590 2024-04-25T11:34:59.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.153+0000 7f5605051640 1 --2- 172.21.15.90:0/3826591599 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f91103590 0x559f914cc7f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f55f0005710 tx=0x7f55f0031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4c954cfe89c0cd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.153+0000 7f5606053640 1 --2- 172.21.15.90:0/3322914767 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f55ec0afaf0 0x7f55ec0b2000 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f55fc0dbe80 tx=0x7f55fc00e000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.153+0000 7f55ebfff640 1 -- 172.21.15.90:0/3826591599 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f0035040 con 0x559f91103590 2024-04-25T11:34:59.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.153+0000 7f55ebfff640 1 -- 172.21.15.90:0/3826591599 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55f0004030 con 0x559f91103590 2024-04-25T11:34:59.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.153+0000 7f55ebfff640 1 -- 172.21.15.90:0/3826591599 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f00046c0 con 0x559f91103590 2024-04-25T11:34:59.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.154+0000 7f560e96f740 1 -- 172.21.15.90:0/3826591599 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f91103590 msgr2=0x559f914cc7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.154+0000 7f560e96f740 1 --2- 172.21.15.90:0/3826591599 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f91103590 0x559f914cc7f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f55f0005710 tx=0x7f55f0031400 comp rx=0 tx=0).stop 2024-04-25T11:34:59.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.154+0000 7f560e96f740 1 -- 172.21.15.90:0/3826591599 shutdown_connections 2024-04-25T11:34:59.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.154+0000 7f560e96f740 1 --2- 172.21.15.90:0/3826591599 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f91103590 0x559f914cc7f0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.154+0000 7f560e96f740 1 --2- 172.21.15.90:0/3826591599 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559f910fb3b0 0x559f914c58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.154+0000 7f560e96f740 1 --2- 172.21.15.90:0/3826591599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f91124270 0x559f914c5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.154+0000 7f560e96f740 1 -- 172.21.15.90:0/3826591599 >> 172.21.15.90:0/3826591599 conn(0x559f911183c0 msgr2=0x559f914c0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.154+0000 7f560e96f740 1 -- 172.21.15.90:0/3826591599 shutdown_connections 2024-04-25T11:34:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.154+0000 7f560e96f740 1 -- 172.21.15.90:0/3826591599 wait complete. 2024-04-25T11:34:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.155+0000 7f560e96f740 1 -- start start 2024-04-25T11:34:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.155+0000 7f560e96f740 1 Processor -- start 2024-04-25T11:34:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.155+0000 7f560e96f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f910fb3b0 0x559f914cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.155+0000 7f560e96f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559f91103590 0x559f914cf6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.155+0000 7f560e96f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f91124270 0x559f914d6640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.155+0000 7f560e96f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559f914cd180 con 0x559f91124270 2024-04-25T11:34:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.155+0000 7f560e96f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x559f913b1fe0 con 0x559f91103590 2024-04-25T11:34:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.155+0000 7f560e96f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559f914cd300 con 0x559f910fb3b0 2024-04-25T11:34:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.155+0000 7f560e96f740 10 client.0 _ll_register_callbacks cb 0x559f913c8150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:34:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.155+0000 7f5605051640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f91124270 0x559f914d6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.155+0000 7f5605051640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f91124270 0x559f914d6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57848/0 (socket says 172.21.15.90:57848) 2024-04-25T11:34:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.155+0000 7f5605051640 1 -- 172.21.15.90:0/2204090920 learned_addr learned my addr 172.21.15.90:0/2204090920 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.155+0000 7f5605852640 1 --2- 172.21.15.90:0/2204090920 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559f91103590 0x559f914cf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.155+0000 7f5605051640 1 --2- 172.21.15.90:0/2204090920 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f910fb3b0 0x559f914cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.156+0000 7f5605051640 1 -- 172.21.15.90:0/2204090920 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f910fb3b0 msgr2=0x559f914cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.156+0000 7f5605051640 1 --2- 172.21.15.90:0/2204090920 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f910fb3b0 0x559f914cf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.156+0000 7f5605051640 1 -- 172.21.15.90:0/2204090920 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559f91103590 msgr2=0x559f914cf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.156+0000 7f5605051640 1 --2- 172.21.15.90:0/2204090920 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559f91103590 0x559f914cf6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.156+0000 7f5605051640 1 -- 172.21.15.90:0/2204090920 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f9113c2d0 con 0x559f91124270 2024-04-25T11:34:59.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.156+0000 7f5605051640 1 --2- 172.21.15.90:0/2204090920 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f91124270 0x559f914d6640 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f55f0009900 tx=0x7f55f0041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d598cf5670489e17 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.156+0000 7f55e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:34:59.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.156+0000 7f55e97fa640 1 -- 172.21.15.90:0/2204090920 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f0035070 con 0x559f91124270 2024-04-25T11:34:59.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.156+0000 7f55e97fa640 1 -- 172.21.15.90:0/2204090920 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f55f00043f0 con 0x559f91124270 2024-04-25T11:34:59.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.156+0000 7f560e96f740 1 -- 172.21.15.90:0/2204090920 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55f0004030 con 0x559f91124270 2024-04-25T11:34:59.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.156+0000 7f55e97fa640 1 -- 172.21.15.90:0/2204090920 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f55f0031b10 con 0x559f91124270 2024-04-25T11:34:59.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.156+0000 7f560e96f740 10 client.15264 fetch_fsmap learned FSMap version 25 2024-04-25T11:34:59.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.156+0000 7f560e96f740 1 -- 172.21.15.90:0/2204090920 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559f912066b0 con 0x559f91124270 2024-04-25T11:34:59.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.156+0000 7f55e97fa640 1 -- 172.21.15.90:0/2204090920 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55f0004030 con 0x559f91124270 2024-04-25T11:34:59.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.156+0000 7f55e97fa640 1 -- 172.21.15.90:0/2204090920 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f55f003a050 con 0x559f91124270 2024-04-25T11:34:59.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.156+0000 7f560e96f740 10 client.15264 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T11:34:59.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.157+0000 7f560e96f740 10 client.15264 resolve_mds: resolved name 'k' to daemon mds.k 2024-04-25T11:34:59.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.157+0000 7f560e96f740 20 client.15264 populate_metadata read hostname 'smithi090' 2024-04-25T11:34:59.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.157+0000 7f560e96f740 1 --2- 172.21.15.90:0/2204090920 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x559f910e2e40 0x559f914dc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.157+0000 7f560e96f740 4 client.15264 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:34:59.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.157+0000 7f560e96f740 1 -- 172.21.15.90:0/2204090920 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f91438350 con 0x559f910e2e40 2024-04-25T11:34:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.157+0000 7f5605852640 1 --2- 172.21.15.90:0/2204090920 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x559f910e2e40 0x559f914dc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.157+0000 7f5605852640 1 --2- 172.21.15.90:0/2204090920 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x559f910e2e40 0x559f914dc8c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=faf5c2f14dcfc1de server_cookie=8442f37efdf6b2ec in_seq=0 out_seq=0 2024-04-25T11:34:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.157+0000 7f55e97fa640 10 client.15264 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:34:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.158+0000 7f55e97fa640 1 -- 172.21.15.90:0/2204090920 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559f91438350 con 0x559f910e2e40 2024-04-25T11:34:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.158+0000 7f55e97fa640 10 client.15264 handle_command_reply: tid=0 2024-04-25T11:34:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.158+0000 7f560e96f740 1 client.15264 shutdown 2024-04-25T11:34:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.158+0000 7f560e96f740 1 -- 172.21.15.90:0/2204090920 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f91124270 msgr2=0x559f914d6640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.158+0000 7f560e96f740 1 --2- 172.21.15.90:0/2204090920 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f91124270 0x559f914d6640 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f55f0009900 tx=0x7f55f0041040 comp rx=0 tx=0).stop 2024-04-25T11:34:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.158+0000 7f560e96f740 1 -- 172.21.15.90:0/2204090920 shutdown_connections 2024-04-25T11:34:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.158+0000 7f560e96f740 1 --2- 172.21.15.90:0/2204090920 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x559f910e2e40 0x559f914dc8c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.158+0000 7f560e96f740 1 --2- 172.21.15.90:0/2204090920 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f91124270 0x559f914d6640 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.158+0000 7f560e96f740 1 --2- 172.21.15.90:0/2204090920 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559f91103590 0x559f914cf6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.158+0000 7f55e97fa640 0 client.15264 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:34:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.158+0000 7f560e96f740 1 --2- 172.21.15.90:0/2204090920 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f910fb3b0 0x559f914cf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.158+0000 7f560e96f740 1 -- 172.21.15.90:0/2204090920 >> 172.21.15.90:0/2204090920 conn(0x559f911183c0 msgr2=0x559f914bfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.158+0000 7f560e96f740 1 -- 172.21.15.90:0/2204090920 shutdown_connections 2024-04-25T11:34:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.158+0000 7f560e96f740 1 -- 172.21.15.90:0/2204090920 wait complete. 2024-04-25T11:34:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.158+0000 7f560e96f740 20 client.15264 trim_cache size 0 max 16384 2024-04-25T11:34:59.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.176+0000 7f560e96f740 1 Processor -- start 2024-04-25T11:34:59.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.176+0000 7f560e96f740 1 -- start start 2024-04-25T11:34:59.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.176+0000 7f560e96f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559f910e2e40 0x559f914cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.176+0000 7f560e96f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f910fb3b0 0x559f914d6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.176+0000 7f560e96f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f91103590 0x559f914cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.176+0000 7f560e96f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559f914dac00 con 0x559f910fb3b0 2024-04-25T11:34:59.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.176+0000 7f560e96f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x559f914db2d0 con 0x559f910e2e40 2024-04-25T11:34:59.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.176+0000 7f560e96f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559f914db9a0 con 0x559f91103590 2024-04-25T11:34:59.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.177+0000 7f5605852640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f910fb3b0 0x559f914d6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.177+0000 7f5605852640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f910fb3b0 0x559f914d6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57854/0 (socket says 172.21.15.90:57854) 2024-04-25T11:34:59.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.177+0000 7f5605852640 1 -- 172.21.15.90:0/2523878903 learned_addr learned my addr 172.21.15.90:0/2523878903 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:59.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.177+0000 7f5605852640 1 -- 172.21.15.90:0/2523878903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f91103590 msgr2=0x559f914cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.177+0000 7f5605852640 1 --2- 172.21.15.90:0/2523878903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f91103590 0x559f914cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.177+0000 7f5605852640 1 -- 172.21.15.90:0/2523878903 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559f910e2e40 msgr2=0x559f914cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.177+0000 7f5605852640 1 --2- 172.21.15.90:0/2523878903 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559f910e2e40 0x559f914cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.177+0000 7f5605852640 1 -- 172.21.15.90:0/2523878903 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f914c5d30 con 0x559f910fb3b0 2024-04-25T11:34:59.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.177+0000 7f5605852640 1 --2- 172.21.15.90:0/2523878903 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f910fb3b0 0x559f914d6560 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x559f914c6750 tx=0x7f55f8015b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34c58edd59375df5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.177+0000 7f55eb7fe640 1 -- 172.21.15.90:0/2523878903 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f8015ea0 con 0x559f910fb3b0 2024-04-25T11:34:59.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.177+0000 7f55eb7fe640 1 -- 172.21.15.90:0/2523878903 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55f8023070 con 0x559f910fb3b0 2024-04-25T11:34:59.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.177+0000 7f55eb7fe640 1 -- 172.21.15.90:0/2523878903 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f8003bd0 con 0x559f910fb3b0 2024-04-25T11:34:59.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.178+0000 7f560e96f740 1 -- 172.21.15.90:0/2523878903 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f910fb3b0 msgr2=0x559f914d6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.178+0000 7f560e96f740 1 --2- 172.21.15.90:0/2523878903 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f910fb3b0 0x559f914d6560 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x559f914c6750 tx=0x7f55f8015b70 comp rx=0 tx=0).stop 2024-04-25T11:34:59.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.178+0000 7f560e96f740 1 -- 172.21.15.90:0/2523878903 shutdown_connections 2024-04-25T11:34:59.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.178+0000 7f560e96f740 1 --2- 172.21.15.90:0/2523878903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f91103590 0x559f914cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.178+0000 7f560e96f740 1 --2- 172.21.15.90:0/2523878903 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f910fb3b0 0x559f914d6560 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.178+0000 7f560e96f740 1 --2- 172.21.15.90:0/2523878903 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559f910e2e40 0x559f914cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.178+0000 7f560e96f740 1 -- 172.21.15.90:0/2523878903 >> 172.21.15.90:0/2523878903 conn(0x559f911183c0 msgr2=0x559f914c96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.178+0000 7f560e96f740 1 -- 172.21.15.90:0/2523878903 shutdown_connections 2024-04-25T11:34:59.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.178+0000 7f560e96f740 1 -- 172.21.15.90:0/2523878903 wait complete. 2024-04-25T11:34:59.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.178+0000 7f560e96f740 1 -- start start 2024-04-25T11:34:59.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.178+0000 7f560e96f740 1 Processor -- start 2024-04-25T11:34:59.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.179+0000 7f560e96f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f910e2e40 0x559f914d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.179+0000 7f560e96f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559f910fb3b0 0x559f914d20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.179+0000 7f560e96f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f91103590 0x559f914d5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.179+0000 7f560e96f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559f914dc740 con 0x559f91103590 2024-04-25T11:34:59.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.179+0000 7f560e96f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x559f914dc070 con 0x559f910fb3b0 2024-04-25T11:34:59.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.179+0000 7f560e96f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559f914dce10 con 0x559f910e2e40 2024-04-25T11:34:59.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.179+0000 7f560e96f740 10 client.0 _ll_register_callbacks cb 0x7f55f8009a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:34:59.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.179+0000 7f5605051640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f91103590 0x559f914d5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.179+0000 7f5605051640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f91103590 0x559f914d5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57856/0 (socket says 172.21.15.90:57856) 2024-04-25T11:34:59.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.179+0000 7f5605051640 1 -- 172.21.15.90:0/2943992789 learned_addr learned my addr 172.21.15.90:0/2943992789 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:59.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.179+0000 7f5605051640 1 --2- 172.21.15.90:0/2943992789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f910e2e40 0x559f914d1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.179+0000 7f5605852640 1 --2- 172.21.15.90:0/2943992789 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559f910fb3b0 0x559f914d20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.179+0000 7f5605051640 1 -- 172.21.15.90:0/2943992789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f910e2e40 msgr2=0x559f914d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.179+0000 7f5605051640 1 --2- 172.21.15.90:0/2943992789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f910e2e40 0x559f914d1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.179+0000 7f5605051640 1 -- 172.21.15.90:0/2943992789 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559f910fb3b0 msgr2=0x559f914d20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.179+0000 7f5605051640 1 --2- 172.21.15.90:0/2943992789 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559f910fb3b0 0x559f914d20d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.179+0000 7f5605051640 1 -- 172.21.15.90:0/2943992789 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f914d55d0 con 0x559f91103590 2024-04-25T11:34:59.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.179+0000 7f5605051640 1 --2- 172.21.15.90:0/2943992789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f91103590 0x559f914d5010 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f55f0005120 tx=0x7f55f0041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cdfc813bc142c2e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.179+0000 7f55e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:34:59.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.179+0000 7f55e97fa640 1 -- 172.21.15.90:0/2943992789 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f0035070 con 0x559f91103590 2024-04-25T11:34:59.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.179+0000 7f55e97fa640 1 -- 172.21.15.90:0/2943992789 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f55f0003d10 con 0x559f91103590 2024-04-25T11:34:59.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.179+0000 7f560e96f740 1 -- 172.21.15.90:0/2943992789 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55f8023070 con 0x559f91103590 2024-04-25T11:34:59.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.180+0000 7f55e97fa640 1 -- 172.21.15.90:0/2943992789 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f55f006c050 con 0x559f91103590 2024-04-25T11:34:59.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.180+0000 7f560e96f740 10 client.15270 fetch_fsmap learned FSMap version 25 2024-04-25T11:34:59.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.180+0000 7f560e96f740 1 -- 172.21.15.90:0/2943992789 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559f914d6880 con 0x559f91103590 2024-04-25T11:34:59.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.180+0000 7f55e97fa640 1 -- 172.21.15.90:0/2943992789 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55f8023070 con 0x559f91103590 2024-04-25T11:34:59.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.180+0000 7f55e97fa640 1 -- 172.21.15.90:0/2943992789 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f55f003a040 con 0x559f91103590 2024-04-25T11:34:59.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.180+0000 7f560e96f740 10 client.15270 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T11:34:59.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.180+0000 7f560e96f740 10 client.15270 resolve_mds: resolved name 'k' to daemon mds.k 2024-04-25T11:34:59.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.180+0000 7f560e96f740 20 client.15270 populate_metadata read hostname 'smithi090' 2024-04-25T11:34:59.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.180+0000 7f560e96f740 1 --2- 172.21.15.90:0/2943992789 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x559f91124270 0x559f91548e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.180+0000 7f560e96f740 4 client.15270 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T11:34:59.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.180+0000 7f560e96f740 1 -- 172.21.15.90:0/2943992789 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "status"}) -- 0x559f914321c0 con 0x559f91124270 2024-04-25T11:34:59.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.180+0000 7f5605852640 1 --2- 172.21.15.90:0/2943992789 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x559f91124270 0x559f91548e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.181+0000 7f55e97fa640 10 client.15270 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:34:59.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.181+0000 7f5605852640 1 --2- 172.21.15.90:0/2943992789 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x559f91124270 0x559f91548e00 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7060fd739dbd1967 server_cookie=5283625f9552a957 in_seq=0 out_seq=0 2024-04-25T11:34:59.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.181+0000 7f55e97fa640 1 -- 172.21.15.90:0/2943992789 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x559f914321c0 con 0x559f91124270 2024-04-25T11:34:59.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.181+0000 7f55e97fa640 10 client.15270 handle_command_reply: tid=0 2024-04-25T11:34:59.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.181+0000 7f560e96f740 1 client.15270 shutdown 2024-04-25T11:34:59.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.181+0000 7f560e96f740 1 -- 172.21.15.90:0/2943992789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f91103590 msgr2=0x559f914d5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.181+0000 7f560e96f740 1 --2- 172.21.15.90:0/2943992789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f91103590 0x559f914d5010 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f55f0005120 tx=0x7f55f0041040 comp rx=0 tx=0).stop 2024-04-25T11:34:59.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.181+0000 7f560e96f740 1 -- 172.21.15.90:0/2943992789 shutdown_connections 2024-04-25T11:34:59.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.181+0000 7f560e96f740 1 --2- 172.21.15.90:0/2943992789 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x559f91124270 0x559f91548e00 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.181+0000 7f560e96f740 1 --2- 172.21.15.90:0/2943992789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559f91103590 0x559f914d5010 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.181+0000 7f560e96f740 1 --2- 172.21.15.90:0/2943992789 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559f910fb3b0 0x559f914d20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.181+0000 7f55e97fa640 0 client.15270 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:34:59.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.181+0000 7f560e96f740 1 --2- 172.21.15.90:0/2943992789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559f910e2e40 0x559f914d1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.181+0000 7f560e96f740 1 -- 172.21.15.90:0/2943992789 >> 172.21.15.90:0/2943992789 conn(0x559f911183c0 msgr2=0x559f914cd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.181+0000 7f560e96f740 1 -- 172.21.15.90:0/2943992789 shutdown_connections 2024-04-25T11:34:59.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.181+0000 7f560e96f740 1 -- 172.21.15.90:0/2943992789 wait complete. 2024-04-25T11:34:59.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.182+0000 7f560e96f740 20 client.15270 trim_cache size 0 max 16384 2024-04-25T11:34:59.178 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:34:59.179 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "58bd4e7a-02f6-11ef-bc93-c7b262605968", 2024-04-25T11:34:59.179 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 4, 2024-04-25T11:34:59.179 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 24431, 2024-04-25T11:34:59.179 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-04-25T11:34:59.179 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-04-25T11:34:59.179 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-04-25T11:34:59.179 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 103.546971695, 2024-04-25T11:34:59.179 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 25, 2024-04-25T11:34:59.179 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 79, 2024-04-25T11:34:59.179 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 77, 2024-04-25T11:34:59.179 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 141.28203057499999 2024-04-25T11:34:59.179 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:34:59.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.182+0000 7f5607ab6640 1 -- 172.21.15.90:0/3322914767 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f55ec0afaf0 msgr2=0x7f55ec0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.182+0000 7f5607ab6640 1 --2- 172.21.15.90:0/3322914767 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f55ec0afaf0 0x7f55ec0b2000 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f55fc0dbe80 tx=0x7f55fc00e000 comp rx=0 tx=0).stop 2024-04-25T11:34:59.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.182+0000 7f5607ab6640 1 -- 172.21.15.90:0/3322914767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5600109770 msgr2=0x7f56001a1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.182+0000 7f5607ab6640 1 --2- 172.21.15.90:0/3322914767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5600109770 0x7f56001a1a60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f55fc009fd0 tx=0x7f55fc00b8e0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.182+0000 7f5607ab6640 1 -- 172.21.15.90:0/3322914767 shutdown_connections 2024-04-25T11:34:59.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.182+0000 7f5607ab6640 1 --2- 172.21.15.90:0/3322914767 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f55ec0afaf0 0x7f55ec0b2000 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.182+0000 7f5607ab6640 1 --2- 172.21.15.90:0/3322914767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5600109770 0x7f56001a1a60 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.182+0000 7f5607ab6640 1 --2- 172.21.15.90:0/3322914767 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5600106c60 0x7f560019aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.182+0000 7f5607ab6640 1 --2- 172.21.15.90:0/3322914767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5600104270 0x7f560019a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.182+0000 7f5607ab6640 1 -- 172.21.15.90:0/3322914767 >> 172.21.15.90:0/3322914767 conn(0x7f56000fdbd0 msgr2=0x7f5600105b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.182+0000 7f5607ab6640 1 -- 172.21.15.90:0/3322914767 shutdown_connections 2024-04-25T11:34:59.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.182+0000 7f5607ab6640 1 -- 172.21.15.90:0/3322914767 wait complete. 2024-04-25T11:34:59.182 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-04-25T11:34:59.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.301+0000 7f25c9042640 1 -- 172.21.15.90:0/1365875346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25c41098a0 msgr2=0x7f25c410bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.301+0000 7f25c9042640 1 --2- 172.21.15.90:0/1365875346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25c41098a0 0x7f25c410bcc0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f25bc009900 tx=0x7f25bc02a410 comp rx=0 tx=0).stop 2024-04-25T11:34:59.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.301+0000 7f25c9042640 1 -- 172.21.15.90:0/1365875346 shutdown_connections 2024-04-25T11:34:59.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.301+0000 7f25c9042640 1 --2- 172.21.15.90:0/1365875346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25c41098a0 0x7f25c410bcc0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.301+0000 7f25c9042640 1 --2- 172.21.15.90:0/1365875346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25c4106f40 0x7f25c4109360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.301+0000 7f25c9042640 1 --2- 172.21.15.90:0/1365875346 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f25c41045e0 0x7f25c4106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.301+0000 7f25c9042640 1 -- 172.21.15.90:0/1365875346 >> 172.21.15.90:0/1365875346 conn(0x7f25c40fdbe0 msgr2=0x7f25c4100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.301+0000 7f25c9042640 1 -- 172.21.15.90:0/1365875346 shutdown_connections 2024-04-25T11:34:59.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.301+0000 7f25c9042640 1 -- 172.21.15.90:0/1365875346 wait complete. 2024-04-25T11:34:59.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.302+0000 7f25c9042640 1 Processor -- start 2024-04-25T11:34:59.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.302+0000 7f25c9042640 1 -- start start 2024-04-25T11:34:59.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.303+0000 7f25c9042640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f25c41045e0 0x7f25c419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.303+0000 7f25c9042640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25c4106f40 0x7f25c419a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.303+0000 7f25c9042640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25c41098a0 0x7f25c41a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.303+0000 7f25c9042640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f25c410f620 con 0x7f25c4106f40 2024-04-25T11:34:59.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.303+0000 7f25c9042640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f25c410f4a0 con 0x7f25c41045e0 2024-04-25T11:34:59.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.303+0000 7f25c9042640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f25c410f7a0 con 0x7f25c41098a0 2024-04-25T11:34:59.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.303+0000 7f25c259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25c4106f40 0x7f25c419a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.303+0000 7f25c259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25c4106f40 0x7f25c419a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57870/0 (socket says 172.21.15.90:57870) 2024-04-25T11:34:59.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.303+0000 7f25c259c640 1 -- 172.21.15.90:0/2918324105 learned_addr learned my addr 172.21.15.90:0/2918324105 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:59.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.303+0000 7f25c359e640 1 --2- 172.21.15.90:0/2918324105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25c41098a0 0x7f25c41a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.303+0000 7f25c2d9d640 1 --2- 172.21.15.90:0/2918324105 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f25c41045e0 0x7f25c419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.303+0000 7f25c259c640 1 -- 172.21.15.90:0/2918324105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25c41098a0 msgr2=0x7f25c41a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.303+0000 7f25c259c640 1 --2- 172.21.15.90:0/2918324105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25c41098a0 0x7f25c41a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.303+0000 7f25c259c640 1 -- 172.21.15.90:0/2918324105 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f25c41045e0 msgr2=0x7f25c419a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.303+0000 7f25c259c640 1 --2- 172.21.15.90:0/2918324105 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f25c41045e0 0x7f25c419a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.303+0000 7f25c259c640 1 -- 172.21.15.90:0/2918324105 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25c41a1da0 con 0x7f25c4106f40 2024-04-25T11:34:59.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.303+0000 7f25c2d9d640 1 --2- 172.21.15.90:0/2918324105 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f25c41045e0 0x7f25c419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:59.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.303+0000 7f25c259c640 1 --2- 172.21.15.90:0/2918324105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25c4106f40 0x7f25c419a7a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f25b800d770 tx=0x7f25b800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.303+0000 7f25abfff640 1 -- 172.21.15.90:0/2918324105 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b8017070 con 0x7f25c4106f40 2024-04-25T11:34:59.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.303+0000 7f25c9042640 1 -- 172.21.15.90:0/2918324105 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25c41a2090 con 0x7f25c4106f40 2024-04-25T11:34:59.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.303+0000 7f25abfff640 1 -- 172.21.15.90:0/2918324105 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25b80054d0 con 0x7f25c4106f40 2024-04-25T11:34:59.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.303+0000 7f25abfff640 1 -- 172.21.15.90:0/2918324105 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b8004c30 con 0x7f25c4106f40 2024-04-25T11:34:59.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.303+0000 7f25c9042640 1 -- 172.21.15.90:0/2918324105 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25c41a2590 con 0x7f25c4106f40 2024-04-25T11:34:59.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.306+0000 7f25cffcb740 1 Processor -- start 2024-04-25T11:34:59.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.306+0000 7f25cffcb740 1 -- start start 2024-04-25T11:34:59.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.307+0000 7f25abfff640 1 -- 172.21.15.90:0/2918324105 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f25b8010460 con 0x7f25c4106f40 2024-04-25T11:34:59.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.307+0000 7f25cffcb740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec2eede270 0x55ec2f27f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.307+0000 7f25cffcb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec2eeb53b0 0x55ec2f27f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.307+0000 7f25cffcb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eebd590 0x55ec2f2867f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.307+0000 7f25cffcb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ec2f1e4740 con 0x55ec2eebd590 2024-04-25T11:34:59.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.307+0000 7f25cffcb740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ec2f1803a0 con 0x55ec2eede270 2024-04-25T11:34:59.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.307+0000 7f25cffcb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ec2f1e0700 con 0x55ec2eeb53b0 2024-04-25T11:34:59.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.307+0000 7f25c259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eebd590 0x55ec2f2867f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.307+0000 7f25c259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eebd590 0x55ec2f2867f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57874/0 (socket says 172.21.15.90:57874) 2024-04-25T11:34:59.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.307+0000 7f25c259c640 1 -- 172.21.15.90:0/165471062 learned_addr learned my addr 172.21.15.90:0/165471062 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:59.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.307+0000 7f25c2d9d640 1 --2- 172.21.15.90:0/165471062 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec2eeb53b0 0x55ec2f27f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.307+0000 7f25c259c640 1 -- 172.21.15.90:0/165471062 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec2eeb53b0 msgr2=0x55ec2f27f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.307+0000 7f25c259c640 1 --2- 172.21.15.90:0/165471062 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec2eeb53b0 0x55ec2f27f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.307+0000 7f25c259c640 1 -- 172.21.15.90:0/165471062 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec2eede270 msgr2=0x55ec2f27f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.307+0000 7f25c259c640 1 --2- 172.21.15.90:0/165471062 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec2eede270 0x55ec2f27f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.307+0000 7f25c259c640 1 -- 172.21.15.90:0/165471062 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec2f19c750 con 0x55ec2eebd590 2024-04-25T11:34:59.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.307+0000 7f25c259c640 1 --2- 172.21.15.90:0/165471062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eebd590 0x55ec2f2867f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f25b80e4c10 tx=0x7f25b80e6690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdf147b67f7481fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.307+0000 7f25abfff640 1 --2- 172.21.15.90:0/2918324105 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f25ac0afaf0 0x7f25ac0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.307+0000 7f25a97fa640 1 -- 172.21.15.90:0/165471062 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b80ed070 con 0x55ec2eebd590 2024-04-25T11:34:59.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.307+0000 7f25c359e640 1 --2- 172.21.15.90:0/2918324105 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f25ac0afaf0 0x7f25ac0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.307+0000 7f25a97fa640 1 -- 172.21.15.90:0/165471062 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25b80a6720 con 0x55ec2eebd590 2024-04-25T11:34:59.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.307+0000 7f25a97fa640 1 -- 172.21.15.90:0/165471062 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b80a6380 con 0x55ec2eebd590 2024-04-25T11:34:59.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.308+0000 7f25abfff640 1 -- 172.21.15.90:0/2918324105 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f25b80de750 con 0x7f25c4106f40 2024-04-25T11:34:59.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.308+0000 7f25cffcb740 1 -- 172.21.15.90:0/165471062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eebd590 msgr2=0x55ec2f2867f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.308+0000 7f25c359e640 1 --2- 172.21.15.90:0/2918324105 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f25ac0afaf0 0x7f25ac0b2000 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f25bc006010 tx=0x7f25bc031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.308+0000 7f25cffcb740 1 --2- 172.21.15.90:0/165471062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eebd590 0x55ec2f2867f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f25b80e4c10 tx=0x7f25b80e6690 comp rx=0 tx=0).stop 2024-04-25T11:34:59.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.308+0000 7f25cffcb740 1 -- 172.21.15.90:0/165471062 shutdown_connections 2024-04-25T11:34:59.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.308+0000 7f25cffcb740 1 --2- 172.21.15.90:0/165471062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eebd590 0x55ec2f2867f0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.308+0000 7f25cffcb740 1 --2- 172.21.15.90:0/165471062 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec2eeb53b0 0x55ec2f27f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.308+0000 7f25cffcb740 1 --2- 172.21.15.90:0/165471062 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec2eede270 0x55ec2f27f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.308+0000 7f25cffcb740 1 -- 172.21.15.90:0/165471062 >> 172.21.15.90:0/165471062 conn(0x55ec2eed23c0 msgr2=0x55ec2f27ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.308+0000 7f25cffcb740 1 -- 172.21.15.90:0/165471062 shutdown_connections 2024-04-25T11:34:59.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.308+0000 7f25cffcb740 1 -- 172.21.15.90:0/165471062 wait complete. 2024-04-25T11:34:59.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.309+0000 7f25cffcb740 1 -- start start 2024-04-25T11:34:59.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.309+0000 7f25cffcb740 1 Processor -- start 2024-04-25T11:34:59.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.309+0000 7f25cffcb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec2eeb53b0 0x55ec2f2892c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.309+0000 7f25cffcb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eebd590 0x55ec2f289800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.309+0000 7f25cffcb740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec2eede270 0x55ec2f290760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.309+0000 7f25cffcb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ec2f287240 con 0x55ec2eebd590 2024-04-25T11:34:59.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.309+0000 7f25cffcb740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ec2f16bfe0 con 0x55ec2eede270 2024-04-25T11:34:59.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.309+0000 7f25cffcb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ec2f2873c0 con 0x55ec2eeb53b0 2024-04-25T11:34:59.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.309+0000 7f25cffcb740 10 client.0 _ll_register_callbacks cb 0x55ec2f182150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:34:59.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.309+0000 7f25c259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eebd590 0x55ec2f289800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.309+0000 7f25c259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eebd590 0x55ec2f289800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57882/0 (socket says 172.21.15.90:57882) 2024-04-25T11:34:59.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.309+0000 7f25c259c640 1 -- 172.21.15.90:0/3302807969 learned_addr learned my addr 172.21.15.90:0/3302807969 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:59.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.309+0000 7f25c359e640 1 --2- 172.21.15.90:0/3302807969 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec2eede270 0x55ec2f290760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.309+0000 7f25c259c640 1 -- 172.21.15.90:0/3302807969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec2eeb53b0 msgr2=0x55ec2f2892c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.309+0000 7f25c259c640 1 --2- 172.21.15.90:0/3302807969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec2eeb53b0 0x55ec2f2892c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.309+0000 7f25c259c640 1 -- 172.21.15.90:0/3302807969 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec2eede270 msgr2=0x55ec2f290760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.309+0000 7f25c259c640 1 --2- 172.21.15.90:0/3302807969 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec2eede270 0x55ec2f290760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.309+0000 7f25c259c640 1 -- 172.21.15.90:0/3302807969 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec2eef62d0 con 0x55ec2eebd590 2024-04-25T11:34:59.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.309+0000 7f25c359e640 1 --2- 172.21.15.90:0/3302807969 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec2eede270 0x55ec2f290760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:59.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.309+0000 7f25c259c640 1 --2- 172.21.15.90:0/3302807969 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eebd590 0x55ec2f289800 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f25b80a6ba0 tx=0x7f25b8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3912847bf9267daf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.309+0000 7f25a6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:34:59.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.310+0000 7f25a6ff5640 1 -- 172.21.15.90:0/3302807969 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b807f1a0 con 0x55ec2eebd590 2024-04-25T11:34:59.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.310+0000 7f25a6ff5640 1 -- 172.21.15.90:0/3302807969 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f25b8009c30 con 0x55ec2eebd590 2024-04-25T11:34:59.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.310+0000 7f25cffcb740 1 -- 172.21.15.90:0/3302807969 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25b80a6720 con 0x55ec2eebd590 2024-04-25T11:34:59.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.310+0000 7f25a6ff5640 1 -- 172.21.15.90:0/3302807969 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f25b8020070 con 0x55ec2eebd590 2024-04-25T11:34:59.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.310+0000 7f25cffcb740 10 client.15282 fetch_fsmap learned FSMap version 25 2024-04-25T11:34:59.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.310+0000 7f25cffcb740 1 -- 172.21.15.90:0/3302807969 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ec2efc06b0 con 0x55ec2eebd590 2024-04-25T11:34:59.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.310+0000 7f25a6ff5640 1 -- 172.21.15.90:0/3302807969 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25b80a6720 con 0x55ec2eebd590 2024-04-25T11:34:59.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.310+0000 7f25a6ff5640 1 -- 172.21.15.90:0/3302807969 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f25b80a2ea0 con 0x55ec2eebd590 2024-04-25T11:34:59.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.310+0000 7f25cffcb740 10 client.15282 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T11:34:59.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.310+0000 7f25cffcb740 10 client.15282 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-25T11:34:59.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.310+0000 7f25cffcb740 20 client.15282 populate_metadata read hostname 'smithi090' 2024-04-25T11:34:59.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.310+0000 7f25cffcb740 1 --2- 172.21.15.90:0/3302807969 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55ec2ee9ce40 0x55ec2f296a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.310+0000 7f25cffcb740 4 client.15282 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:34:59.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.310+0000 7f25cffcb740 1 -- 172.21.15.90:0/3302807969 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec2f1f2350 con 0x55ec2ee9ce40 2024-04-25T11:34:59.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.311+0000 7f25c2d9d640 1 --2- 172.21.15.90:0/3302807969 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55ec2ee9ce40 0x55ec2f296a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.311+0000 7f25a6ff5640 10 client.15282 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:34:59.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.311+0000 7f25c2d9d640 1 --2- 172.21.15.90:0/3302807969 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55ec2ee9ce40 0x55ec2f296a30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=34a7f641ab916cbe server_cookie=9d6d55c1747f3b75 in_seq=0 out_seq=0 2024-04-25T11:34:59.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.312+0000 7f25a6ff5640 1 -- 172.21.15.90:0/3302807969 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ec2f1f2350 con 0x55ec2ee9ce40 2024-04-25T11:34:59.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.312+0000 7f25a6ff5640 10 client.15282 handle_command_reply: tid=0 2024-04-25T11:34:59.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.312+0000 7f25cffcb740 1 client.15282 shutdown 2024-04-25T11:34:59.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.312+0000 7f25cffcb740 1 -- 172.21.15.90:0/3302807969 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eebd590 msgr2=0x55ec2f289800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.312+0000 7f25cffcb740 1 --2- 172.21.15.90:0/3302807969 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eebd590 0x55ec2f289800 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f25b80a6ba0 tx=0x7f25b8009f90 comp rx=0 tx=0).stop 2024-04-25T11:34:59.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.312+0000 7f25cffcb740 1 -- 172.21.15.90:0/3302807969 shutdown_connections 2024-04-25T11:34:59.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.312+0000 7f25cffcb740 1 --2- 172.21.15.90:0/3302807969 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55ec2ee9ce40 0x55ec2f296a30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.312+0000 7f25cffcb740 1 --2- 172.21.15.90:0/3302807969 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec2eede270 0x55ec2f290760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.312+0000 7f25cffcb740 1 --2- 172.21.15.90:0/3302807969 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eebd590 0x55ec2f289800 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.312+0000 7f25a6ff5640 0 client.15282 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:34:59.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.312+0000 7f25cffcb740 1 --2- 172.21.15.90:0/3302807969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec2eeb53b0 0x55ec2f2892c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.312+0000 7f25cffcb740 1 -- 172.21.15.90:0/3302807969 >> 172.21.15.90:0/3302807969 conn(0x55ec2eed23c0 msgr2=0x55ec2f279d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.312+0000 7f25cffcb740 1 -- 172.21.15.90:0/3302807969 shutdown_connections 2024-04-25T11:34:59.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.312+0000 7f25cffcb740 1 -- 172.21.15.90:0/3302807969 wait complete. 2024-04-25T11:34:59.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.312+0000 7f25cffcb740 20 client.15282 trim_cache size 0 max 16384 2024-04-25T11:34:59.309 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:58 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3924233990' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:34:59.309 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:58 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1064516884' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:34:59.309 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:58 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3997845434' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:34:59.309 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:58 smithi138 ceph-mon[32328]: pgmap v330: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-25T11:34:59.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.330+0000 7f25cffcb740 1 Processor -- start 2024-04-25T11:34:59.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.330+0000 7f25cffcb740 1 -- start start 2024-04-25T11:34:59.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.330+0000 7f25cffcb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec2ee9ce40 0x55ec2f294810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.330+0000 7f25cffcb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eeb53b0 0x55ec2f2829b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.330+0000 7f25cffcb740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec2eebd590 0x55ec2f283700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.330+0000 7f25cffcb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ec2f294d70 con 0x55ec2eeb53b0 2024-04-25T11:34:59.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.330+0000 7f25cffcb740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ec2f295440 con 0x55ec2eebd590 2024-04-25T11:34:59.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.330+0000 7f25cffcb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ec2f295b10 con 0x55ec2ee9ce40 2024-04-25T11:34:59.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.331+0000 7f25c259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eeb53b0 0x55ec2f2829b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.331+0000 7f25c259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eeb53b0 0x55ec2f2829b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57898/0 (socket says 172.21.15.90:57898) 2024-04-25T11:34:59.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.331+0000 7f25c259c640 1 -- 172.21.15.90:0/2470516761 learned_addr learned my addr 172.21.15.90:0/2470516761 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:59.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.331+0000 7f25c259c640 1 -- 172.21.15.90:0/2470516761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec2ee9ce40 msgr2=0x55ec2f294810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.331+0000 7f25c259c640 1 --2- 172.21.15.90:0/2470516761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec2ee9ce40 0x55ec2f294810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.331+0000 7f25c259c640 1 -- 172.21.15.90:0/2470516761 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec2eebd590 msgr2=0x55ec2f283700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.331+0000 7f25c359e640 1 --2- 172.21.15.90:0/2470516761 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec2eebd590 0x55ec2f283700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.331+0000 7f25c259c640 1 --2- 172.21.15.90:0/2470516761 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec2eebd590 0x55ec2f283700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.331+0000 7f25c259c640 1 -- 172.21.15.90:0/2470516761 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec2f289ec0 con 0x55ec2eeb53b0 2024-04-25T11:34:59.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.331+0000 7f25c259c640 1 --2- 172.21.15.90:0/2470516761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eeb53b0 0x55ec2f2829b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f25b80a3e80 tx=0x7f25b80801f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeb70f072d9eb405 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.331+0000 7f25a8ff9640 1 -- 172.21.15.90:0/2470516761 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b80ed070 con 0x55ec2eeb53b0 2024-04-25T11:34:59.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.331+0000 7f25a8ff9640 1 -- 172.21.15.90:0/2470516761 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25b80e6b90 con 0x55ec2eeb53b0 2024-04-25T11:34:59.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.331+0000 7f25a8ff9640 1 -- 172.21.15.90:0/2470516761 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b807fe20 con 0x55ec2eeb53b0 2024-04-25T11:34:59.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.332+0000 7f25cffcb740 1 -- 172.21.15.90:0/2470516761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eeb53b0 msgr2=0x55ec2f2829b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.332+0000 7f25cffcb740 1 --2- 172.21.15.90:0/2470516761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eeb53b0 0x55ec2f2829b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f25b80a3e80 tx=0x7f25b80801f0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.332+0000 7f25cffcb740 1 -- 172.21.15.90:0/2470516761 shutdown_connections 2024-04-25T11:34:59.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.332+0000 7f25cffcb740 1 --2- 172.21.15.90:0/2470516761 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec2eebd590 0x55ec2f283700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.332+0000 7f25cffcb740 1 --2- 172.21.15.90:0/2470516761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eeb53b0 0x55ec2f2829b0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.332+0000 7f25cffcb740 1 --2- 172.21.15.90:0/2470516761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec2ee9ce40 0x55ec2f294810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.332+0000 7f25cffcb740 1 -- 172.21.15.90:0/2470516761 >> 172.21.15.90:0/2470516761 conn(0x55ec2eed23c0 msgr2=0x55ec2f275a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.332+0000 7f25cffcb740 1 -- 172.21.15.90:0/2470516761 shutdown_connections 2024-04-25T11:34:59.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.332+0000 7f25cffcb740 1 -- 172.21.15.90:0/2470516761 wait complete. 2024-04-25T11:34:59.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.332+0000 7f25cffcb740 1 -- start start 2024-04-25T11:34:59.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.332+0000 7f25cffcb740 1 Processor -- start 2024-04-25T11:34:59.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.332+0000 7f25cffcb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec2ee9ce40 0x55ec2f28b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.332+0000 7f25cffcb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eebd590 0x55ec2f28b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.332+0000 7f25cffcb740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec2eede270 0x55ec2f28e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.332+0000 7f25cffcb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ec2f2968b0 con 0x55ec2eebd590 2024-04-25T11:34:59.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.332+0000 7f25cffcb740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ec2f2961e0 con 0x55ec2eede270 2024-04-25T11:34:59.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.332+0000 7f25cffcb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ec2f296f80 con 0x55ec2ee9ce40 2024-04-25T11:34:59.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.332+0000 7f25cffcb740 10 client.0 _ll_register_callbacks cb 0x7f25b80e4a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:34:59.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.333+0000 7f25c259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eebd590 0x55ec2f28b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.333+0000 7f25c259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eebd590 0x55ec2f28b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57908/0 (socket says 172.21.15.90:57908) 2024-04-25T11:34:59.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.333+0000 7f25c259c640 1 -- 172.21.15.90:0/3591503747 learned_addr learned my addr 172.21.15.90:0/3591503747 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:59.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.333+0000 7f25c259c640 1 -- 172.21.15.90:0/3591503747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec2ee9ce40 msgr2=0x55ec2f28b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.333+0000 7f25c259c640 1 --2- 172.21.15.90:0/3591503747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec2ee9ce40 0x55ec2f28b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.333+0000 7f25c259c640 1 -- 172.21.15.90:0/3591503747 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec2eede270 msgr2=0x55ec2f28e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.333+0000 7f25c259c640 1 --2- 172.21.15.90:0/3591503747 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec2eede270 0x55ec2f28e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.333+0000 7f25c259c640 1 -- 172.21.15.90:0/3591503747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec2f28ebe0 con 0x55ec2eebd590 2024-04-25T11:34:59.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.333+0000 7f25c259c640 1 --2- 172.21.15.90:0/3591503747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eebd590 0x55ec2f28b6e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f25b80dc5a0 tx=0x7f25b80e4960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e643aa7f2fd3cfa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.333+0000 7f25a6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:34:59.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.333+0000 7f25a6ff5640 1 -- 172.21.15.90:0/3591503747 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b80c1060 con 0x55ec2eebd590 2024-04-25T11:34:59.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.333+0000 7f25a6ff5640 1 -- 172.21.15.90:0/3591503747 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f25b80a3200 con 0x55ec2eebd590 2024-04-25T11:34:59.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.333+0000 7f25cffcb740 1 -- 172.21.15.90:0/3591503747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25b80e6b90 con 0x55ec2eebd590 2024-04-25T11:34:59.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.333+0000 7f25a6ff5640 1 -- 172.21.15.90:0/3591503747 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f25b80ed040 con 0x55ec2eebd590 2024-04-25T11:34:59.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.333+0000 7f25cffcb740 10 client.15288 fetch_fsmap learned FSMap version 25 2024-04-25T11:34:59.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.333+0000 7f25cffcb740 1 -- 172.21.15.90:0/3591503747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ec2f28ebe0 con 0x55ec2eebd590 2024-04-25T11:34:59.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.333+0000 7f25a6ff5640 1 -- 172.21.15.90:0/3591503747 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25b80e6b90 con 0x55ec2eebd590 2024-04-25T11:34:59.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.333+0000 7f25a6ff5640 1 -- 172.21.15.90:0/3591503747 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f25b80af5e0 con 0x55ec2eebd590 2024-04-25T11:34:59.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.333+0000 7f25cffcb740 10 client.15288 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T11:34:59.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.334+0000 7f25cffcb740 10 client.15288 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-25T11:34:59.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.334+0000 7f25cffcb740 20 client.15288 populate_metadata read hostname 'smithi090' 2024-04-25T11:34:59.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.334+0000 7f25cffcb740 1 --2- 172.21.15.90:0/3591503747 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55ec2ee72280 0x55ec2f2ffe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.334+0000 7f25cffcb740 4 client.15288 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T11:34:59.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.334+0000 7f25cffcb740 1 -- 172.21.15.90:0/3591503747 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "status"}) -- 0x55ec2f1ec1c0 con 0x55ec2ee72280 2024-04-25T11:34:59.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.334+0000 7f25c2d9d640 1 --2- 172.21.15.90:0/3591503747 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55ec2ee72280 0x55ec2f2ffe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.334+0000 7f25a6ff5640 10 client.15288 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:34:59.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.334+0000 7f25c2d9d640 1 --2- 172.21.15.90:0/3591503747 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55ec2ee72280 0x55ec2f2ffe80 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=dd2774469deb64dd server_cookie=4e14f7354c912ca8 in_seq=0 out_seq=0 2024-04-25T11:34:59.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25a6ff5640 1 -- 172.21.15.90:0/3591503747 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55ec2f1ec1c0 con 0x55ec2ee72280 2024-04-25T11:34:59.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25a6ff5640 10 client.15288 handle_command_reply: tid=0 2024-04-25T11:34:59.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25cffcb740 1 client.15288 shutdown 2024-04-25T11:34:59.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25cffcb740 1 -- 172.21.15.90:0/3591503747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eebd590 msgr2=0x55ec2f28b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25cffcb740 1 --2- 172.21.15.90:0/3591503747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eebd590 0x55ec2f28b6e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f25b80dc5a0 tx=0x7f25b80e4960 comp rx=0 tx=0).stop 2024-04-25T11:34:59.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25cffcb740 1 -- 172.21.15.90:0/3591503747 shutdown_connections 2024-04-25T11:34:59.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25cffcb740 1 --2- 172.21.15.90:0/3591503747 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55ec2ee72280 0x55ec2f2ffe80 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25cffcb740 1 --2- 172.21.15.90:0/3591503747 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec2eede270 0x55ec2f28e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25a6ff5640 0 client.15288 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:34:59.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25cffcb740 1 --2- 172.21.15.90:0/3591503747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec2eebd590 0x55ec2f28b6e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25cffcb740 1 --2- 172.21.15.90:0/3591503747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec2ee9ce40 0x55ec2f28b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25cffcb740 1 -- 172.21.15.90:0/3591503747 >> 172.21.15.90:0/3591503747 conn(0x55ec2eed23c0 msgr2=0x55ec2f283700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25cffcb740 1 -- 172.21.15.90:0/3591503747 shutdown_connections 2024-04-25T11:34:59.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25cffcb740 1 -- 172.21.15.90:0/3591503747 wait complete. 2024-04-25T11:34:59.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25cffcb740 20 client.15288 trim_cache size 0 max 16384 2024-04-25T11:34:59.331 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:34:59.331 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "58bd4e7a-02f6-11ef-bc93-c7b262605968", 2024-04-25T11:34:59.331 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 3, 2024-04-25T11:34:59.331 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 24458, 2024-04-25T11:34:59.331 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-04-25T11:34:59.332 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-04-25T11:34:59.332 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-04-25T11:34:59.332 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 104.71393573899999, 2024-04-25T11:34:59.332 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 25, 2024-04-25T11:34:59.332 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 79, 2024-04-25T11:34:59.332 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 77, 2024-04-25T11:34:59.332 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 133.22847806199999 2024-04-25T11:34:59.332 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:34:59.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25c9042640 1 -- 172.21.15.90:0/2918324105 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f25ac0afaf0 msgr2=0x7f25ac0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25c9042640 1 --2- 172.21.15.90:0/2918324105 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f25ac0afaf0 0x7f25ac0b2000 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f25bc006010 tx=0x7f25bc031400 comp rx=0 tx=0).stop 2024-04-25T11:34:59.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25c9042640 1 -- 172.21.15.90:0/2918324105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25c4106f40 msgr2=0x7f25c419a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25c9042640 1 --2- 172.21.15.90:0/2918324105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25c4106f40 0x7f25c419a7a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f25b800d770 tx=0x7f25b800dc30 comp rx=0 tx=0).stop 2024-04-25T11:34:59.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25c9042640 1 -- 172.21.15.90:0/2918324105 shutdown_connections 2024-04-25T11:34:59.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25c9042640 1 --2- 172.21.15.90:0/2918324105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25c41098a0 0x7f25c41a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25c9042640 1 --2- 172.21.15.90:0/2918324105 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f25ac0afaf0 0x7f25ac0b2000 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25c9042640 1 --2- 172.21.15.90:0/2918324105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25c4106f40 0x7f25c419a7a0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25c9042640 1 --2- 172.21.15.90:0/2918324105 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f25c41045e0 0x7f25c419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25c9042640 1 -- 172.21.15.90:0/2918324105 >> 172.21.15.90:0/2918324105 conn(0x7f25c40fdbe0 msgr2=0x7f25c4105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25c9042640 1 -- 172.21.15.90:0/2918324105 shutdown_connections 2024-04-25T11:34:59.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.335+0000 7f25c9042640 1 -- 172.21.15.90:0/2918324105 wait complete. 2024-04-25T11:34:59.336 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-04-25T11:34:59.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:58 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3924233990' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:34:59.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:58 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1064516884' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:34:59.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:58 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3997845434' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:34:59.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:58 smithi156 ceph-mon[33045]: pgmap v330: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-25T11:34:59.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.455+0000 7fde3a096640 1 -- 172.21.15.90:0/4267741884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde34102a00 msgr2=0x7fde34103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.455+0000 7fde3a096640 1 --2- 172.21.15.90:0/4267741884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde34102a00 0x7fde34103700 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fde24009900 tx=0x7fde2402a410 comp rx=0 tx=0).stop 2024-04-25T11:34:59.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.455+0000 7fde3a096640 1 -- 172.21.15.90:0/4267741884 shutdown_connections 2024-04-25T11:34:59.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.455+0000 7fde3a096640 1 --2- 172.21.15.90:0/4267741884 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde34108940 0x7fde3410ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.455+0000 7fde3a096640 1 --2- 172.21.15.90:0/4267741884 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fde34103c40 0x7fde34104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.455+0000 7fde3a096640 1 --2- 172.21.15.90:0/4267741884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde34102a00 0x7fde34103700 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.455+0000 7fde3a096640 1 -- 172.21.15.90:0/4267741884 >> 172.21.15.90:0/4267741884 conn(0x7fde340fdbf0 msgr2=0x7fde341000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.455+0000 7fde3a096640 1 -- 172.21.15.90:0/4267741884 shutdown_connections 2024-04-25T11:34:59.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.455+0000 7fde3a096640 1 -- 172.21.15.90:0/4267741884 wait complete. 2024-04-25T11:34:59.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.456+0000 7fde3a096640 1 Processor -- start 2024-04-25T11:34:59.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.456+0000 7fde3a096640 1 -- start start 2024-04-25T11:34:59.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.456+0000 7fde3a096640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fde34102a00 0x7fde34075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.456+0000 7fde3a096640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde34103c40 0x7fde3419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.456+0000 7fde3a096640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde34108940 0x7fde341a15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.456+0000 7fde3a096640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fde3410e4e0 con 0x7fde34103c40 2024-04-25T11:34:59.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.456+0000 7fde3a096640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fde3410e360 con 0x7fde34102a00 2024-04-25T11:34:59.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.456+0000 7fde3a096640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fde3410e660 con 0x7fde34108940 2024-04-25T11:34:59.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.456+0000 7fde32ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde34103c40 0x7fde3419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.456+0000 7fde32ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde34103c40 0x7fde3419f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57920/0 (socket says 172.21.15.90:57920) 2024-04-25T11:34:59.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.456+0000 7fde32ffd640 1 -- 172.21.15.90:0/33975062 learned_addr learned my addr 172.21.15.90:0/33975062 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:59.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.457+0000 7fde337fe640 1 --2- 172.21.15.90:0/33975062 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fde34102a00 0x7fde34075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.457+0000 7fde33fff640 1 --2- 172.21.15.90:0/33975062 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde34108940 0x7fde341a15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.457+0000 7fde32ffd640 1 -- 172.21.15.90:0/33975062 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde34108940 msgr2=0x7fde341a15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.457+0000 7fde32ffd640 1 --2- 172.21.15.90:0/33975062 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde34108940 0x7fde341a15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.457+0000 7fde32ffd640 1 -- 172.21.15.90:0/33975062 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fde34102a00 msgr2=0x7fde34075840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.457+0000 7fde32ffd640 1 --2- 172.21.15.90:0/33975062 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fde34102a00 0x7fde34075840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.457+0000 7fde32ffd640 1 -- 172.21.15.90:0/33975062 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde341a1d30 con 0x7fde34103c40 2024-04-25T11:34:59.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.457+0000 7fde337fe640 1 --2- 172.21.15.90:0/33975062 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fde34102a00 0x7fde34075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:59.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.457+0000 7fde33fff640 1 --2- 172.21.15.90:0/33975062 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde34108940 0x7fde341a15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:59.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.457+0000 7fde32ffd640 1 --2- 172.21.15.90:0/33975062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde34103c40 0x7fde3419f050 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fde2800c920 tx=0x7fde2800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.457+0000 7fde30ff9640 1 -- 172.21.15.90:0/33975062 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde28007c70 con 0x7fde34103c40 2024-04-25T11:34:59.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.457+0000 7fde30ff9640 1 -- 172.21.15.90:0/33975062 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde28007e10 con 0x7fde34103c40 2024-04-25T11:34:59.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.457+0000 7fde3a096640 1 -- 172.21.15.90:0/33975062 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde341a1fc0 con 0x7fde34103c40 2024-04-25T11:34:59.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.457+0000 7fde30ff9640 1 -- 172.21.15.90:0/33975062 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde28018820 con 0x7fde34103c40 2024-04-25T11:34:59.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.457+0000 7fde3a096640 1 -- 172.21.15.90:0/33975062 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde341a24b0 con 0x7fde34103c40 2024-04-25T11:34:59.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.459+0000 7fde41026740 1 Processor -- start 2024-04-25T11:34:59.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.459+0000 7fde30ff9640 1 -- 172.21.15.90:0/33975062 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fde2800f450 con 0x7fde34103c40 2024-04-25T11:34:59.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.459+0000 7fde41026740 1 -- start start 2024-04-25T11:34:59.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.460+0000 7fde41026740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5d1ba270 0x557f5d55b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.460+0000 7fde41026740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d1913b0 0x557f5d55b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.460+0000 7fde41026740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5d199590 0x557f5d5627f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.460+0000 7fde41026740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557f5d4c0740 con 0x557f5d1913b0 2024-04-25T11:34:59.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.460+0000 7fde41026740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x557f5d45c3a0 con 0x557f5d199590 2024-04-25T11:34:59.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.460+0000 7fde41026740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x557f5d4bc700 con 0x557f5d1ba270 2024-04-25T11:34:59.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.460+0000 7fde337fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d1913b0 0x557f5d55b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.460+0000 7fde337fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d1913b0 0x557f5d55b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57924/0 (socket says 172.21.15.90:57924) 2024-04-25T11:34:59.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.460+0000 7fde337fe640 1 -- 172.21.15.90:0/2176455489 learned_addr learned my addr 172.21.15.90:0/2176455489 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:59.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.460+0000 7fde33fff640 1 --2- 172.21.15.90:0/2176455489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5d1ba270 0x557f5d55b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.460+0000 7fde32ffd640 1 --2- 172.21.15.90:0/2176455489 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5d199590 0x557f5d5627f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.460+0000 7fde337fe640 1 -- 172.21.15.90:0/2176455489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5d1ba270 msgr2=0x557f5d55b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.460+0000 7fde337fe640 1 --2- 172.21.15.90:0/2176455489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5d1ba270 0x557f5d55b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.460+0000 7fde337fe640 1 -- 172.21.15.90:0/2176455489 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5d199590 msgr2=0x557f5d5627f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.460+0000 7fde337fe640 1 --2- 172.21.15.90:0/2176455489 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5d199590 0x557f5d5627f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.460+0000 7fde337fe640 1 -- 172.21.15.90:0/2176455489 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f5d478750 con 0x557f5d1913b0 2024-04-25T11:34:59.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.460+0000 7fde30ff9640 1 --2- 172.21.15.90:0/33975062 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fde1c0afaf0 0x7fde1c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.460+0000 7fde337fe640 1 --2- 172.21.15.90:0/2176455489 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d1913b0 0x557f5d55b8f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fde24005ab0 tx=0x7fde24032e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd09c163c2cb3411 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.460+0000 7fde33fff640 1 --2- 172.21.15.90:0/33975062 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fde1c0afaf0 0x7fde1c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.460+0000 7fde30ff9640 1 -- 172.21.15.90:0/33975062 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7fde280dee40 con 0x7fde34103c40 2024-04-25T11:34:59.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.460+0000 7fde19ffb640 1 -- 172.21.15.90:0/2176455489 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2403f070 con 0x557f5d1913b0 2024-04-25T11:34:59.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.461+0000 7fde19ffb640 1 -- 172.21.15.90:0/2176455489 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde240317a0 con 0x557f5d1913b0 2024-04-25T11:34:59.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.461+0000 7fde19ffb640 1 -- 172.21.15.90:0/2176455489 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde24035070 con 0x557f5d1913b0 2024-04-25T11:34:59.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.461+0000 7fde41026740 1 -- 172.21.15.90:0/2176455489 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d1913b0 msgr2=0x557f5d55b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.461+0000 7fde41026740 1 --2- 172.21.15.90:0/2176455489 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d1913b0 0x557f5d55b8f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fde24005ab0 tx=0x7fde24032e90 comp rx=0 tx=0).stop 2024-04-25T11:34:59.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.461+0000 7fde33fff640 1 --2- 172.21.15.90:0/33975062 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fde1c0afaf0 0x7fde1c0b2000 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x557f5d55c540 tx=0x7fde2c008630 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.461+0000 7fde41026740 1 -- 172.21.15.90:0/2176455489 shutdown_connections 2024-04-25T11:34:59.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.461+0000 7fde41026740 1 --2- 172.21.15.90:0/2176455489 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5d199590 0x557f5d5627f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.461+0000 7fde41026740 1 --2- 172.21.15.90:0/2176455489 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d1913b0 0x557f5d55b8f0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.461+0000 7fde41026740 1 --2- 172.21.15.90:0/2176455489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5d1ba270 0x557f5d55b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.461+0000 7fde41026740 1 -- 172.21.15.90:0/2176455489 >> 172.21.15.90:0/2176455489 conn(0x557f5d1ae3c0 msgr2=0x557f5d556d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.461+0000 7fde41026740 1 -- 172.21.15.90:0/2176455489 shutdown_connections 2024-04-25T11:34:59.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.461+0000 7fde41026740 1 -- 172.21.15.90:0/2176455489 wait complete. 2024-04-25T11:34:59.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.462+0000 7fde41026740 1 -- start start 2024-04-25T11:34:59.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.462+0000 7fde41026740 1 Processor -- start 2024-04-25T11:34:59.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.462+0000 7fde41026740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5d1913b0 0x557f5d565220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.462+0000 7fde41026740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5d199590 0x557f5d565760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.462+0000 7fde41026740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d1ba270 0x557f5d56c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.462+0000 7fde41026740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557f5d5631e0 con 0x557f5d1ba270 2024-04-25T11:34:59.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.462+0000 7fde41026740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x557f5d447fe0 con 0x557f5d1913b0 2024-04-25T11:34:59.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.462+0000 7fde41026740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x557f5d563360 con 0x557f5d199590 2024-04-25T11:34:59.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.462+0000 7fde41026740 10 client.0 _ll_register_callbacks cb 0x557f5d45e150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:34:59.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.462+0000 7fde33fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d1ba270 0x557f5d56c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.462+0000 7fde33fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d1ba270 0x557f5d56c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57936/0 (socket says 172.21.15.90:57936) 2024-04-25T11:34:59.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.462+0000 7fde33fff640 1 -- 172.21.15.90:0/1449366280 learned_addr learned my addr 172.21.15.90:0/1449366280 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:59.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.463+0000 7fde337fe640 1 --2- 172.21.15.90:0/1449366280 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5d1913b0 0x557f5d565220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.463+0000 7fde32ffd640 1 --2- 172.21.15.90:0/1449366280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5d199590 0x557f5d565760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.463+0000 7fde33fff640 1 -- 172.21.15.90:0/1449366280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5d199590 msgr2=0x557f5d565760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.463+0000 7fde33fff640 1 --2- 172.21.15.90:0/1449366280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5d199590 0x557f5d565760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.463+0000 7fde33fff640 1 -- 172.21.15.90:0/1449366280 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5d1913b0 msgr2=0x557f5d565220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.463+0000 7fde33fff640 1 --2- 172.21.15.90:0/1449366280 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5d1913b0 0x557f5d565220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.463+0000 7fde33fff640 1 -- 172.21.15.90:0/1449366280 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f5d1d22d0 con 0x557f5d1ba270 2024-04-25T11:34:59.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.463+0000 7fde32ffd640 1 --2- 172.21.15.90:0/1449366280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5d199590 0x557f5d565760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:59.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.463+0000 7fde337fe640 1 --2- 172.21.15.90:0/1449366280 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5d1913b0 0x557f5d565220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:59.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.463+0000 7fde33fff640 1 --2- 172.21.15.90:0/1449366280 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d1ba270 0x557f5d56c6c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fde2c01bd80 tx=0x7fde2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a18e5182871e76 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.463+0000 7fde12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:34:59.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.463+0000 7fde12ffd640 1 -- 172.21.15.90:0/1449366280 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c028060 con 0x557f5d1ba270 2024-04-25T11:34:59.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.463+0000 7fde12ffd640 1 -- 172.21.15.90:0/1449366280 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7fde2c023410 con 0x557f5d1ba270 2024-04-25T11:34:59.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.463+0000 7fde41026740 1 -- 172.21.15.90:0/1449366280 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde240317a0 con 0x557f5d1ba270 2024-04-25T11:34:59.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.463+0000 7fde12ffd640 1 -- 172.21.15.90:0/1449366280 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fde2c023610 con 0x557f5d1ba270 2024-04-25T11:34:59.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.463+0000 7fde41026740 10 client.15300 fetch_fsmap learned FSMap version 25 2024-04-25T11:34:59.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.463+0000 7fde41026740 1 -- 172.21.15.90:0/1449366280 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557f5d29c6b0 con 0x557f5d1ba270 2024-04-25T11:34:59.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.463+0000 7fde12ffd640 1 -- 172.21.15.90:0/1449366280 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde240317a0 con 0x557f5d1ba270 2024-04-25T11:34:59.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.464+0000 7fde12ffd640 1 -- 172.21.15.90:0/1449366280 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fde2c017040 con 0x557f5d1ba270 2024-04-25T11:34:59.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.464+0000 7fde41026740 10 client.15300 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T11:34:59.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.464+0000 7fde41026740 10 client.15300 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-25T11:34:59.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.464+0000 7fde41026740 20 client.15300 populate_metadata read hostname 'smithi090' 2024-04-25T11:34:59.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.464+0000 7fde41026740 1 --2- 172.21.15.90:0/1449366280 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x557f5d178e40 0x557f5d573010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.464+0000 7fde41026740 4 client.15300 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:34:59.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.464+0000 7fde41026740 1 -- 172.21.15.90:0/1449366280 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f5d4ce350 con 0x557f5d178e40 2024-04-25T11:34:59.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.464+0000 7fde337fe640 1 --2- 172.21.15.90:0/1449366280 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x557f5d178e40 0x557f5d573010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.464+0000 7fde337fe640 1 --2- 172.21.15.90:0/1449366280 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x557f5d178e40 0x557f5d573010 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=27d8317625ce5a47 server_cookie=79ff4d4e685d4993 in_seq=0 out_seq=0 2024-04-25T11:34:59.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.464+0000 7fde12ffd640 10 client.15300 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:34:59.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.465+0000 7fde12ffd640 1 -- 172.21.15.90:0/1449366280 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557f5d4ce350 con 0x557f5d178e40 2024-04-25T11:34:59.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.465+0000 7fde12ffd640 10 client.15300 handle_command_reply: tid=0 2024-04-25T11:34:59.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.465+0000 7fde41026740 1 client.15300 shutdown 2024-04-25T11:34:59.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.465+0000 7fde41026740 1 -- 172.21.15.90:0/1449366280 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d1ba270 msgr2=0x557f5d56c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.465+0000 7fde41026740 1 --2- 172.21.15.90:0/1449366280 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d1ba270 0x557f5d56c6c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fde2c01bd80 tx=0x7fde2c004a40 comp rx=0 tx=0).stop 2024-04-25T11:34:59.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.466+0000 7fde41026740 1 -- 172.21.15.90:0/1449366280 shutdown_connections 2024-04-25T11:34:59.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.466+0000 7fde41026740 1 --2- 172.21.15.90:0/1449366280 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x557f5d178e40 0x557f5d573010 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.466+0000 7fde41026740 1 --2- 172.21.15.90:0/1449366280 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d1ba270 0x557f5d56c6c0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.466+0000 7fde41026740 1 --2- 172.21.15.90:0/1449366280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5d199590 0x557f5d565760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.466+0000 7fde12ffd640 0 client.15300 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:34:59.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.466+0000 7fde41026740 1 --2- 172.21.15.90:0/1449366280 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5d1913b0 0x557f5d565220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.466+0000 7fde41026740 1 -- 172.21.15.90:0/1449366280 >> 172.21.15.90:0/1449366280 conn(0x557f5d1ae3c0 msgr2=0x557f5d555d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.466+0000 7fde41026740 1 -- 172.21.15.90:0/1449366280 shutdown_connections 2024-04-25T11:34:59.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.466+0000 7fde41026740 1 -- 172.21.15.90:0/1449366280 wait complete. 2024-04-25T11:34:59.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.466+0000 7fde41026740 20 client.15300 trim_cache size 0 max 16384 2024-04-25T11:34:59.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.484+0000 7fde41026740 1 Processor -- start 2024-04-25T11:34:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.484+0000 7fde41026740 1 -- start start 2024-04-25T11:34:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.484+0000 7fde41026740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5d178e40 0x557f5d55f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.484+0000 7fde41026740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5d1913b0 0x557f5d56c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.484+0000 7fde41026740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d199590 0x557f5d55fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.484+0000 7fde41026740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557f5d571350 con 0x557f5d199590 2024-04-25T11:34:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.484+0000 7fde41026740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x557f5d571a20 con 0x557f5d178e40 2024-04-25T11:34:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.484+0000 7fde41026740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x557f5d5720f0 con 0x557f5d1913b0 2024-04-25T11:34:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.484+0000 7fde33fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d199590 0x557f5d55fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.484+0000 7fde33fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d199590 0x557f5d55fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57944/0 (socket says 172.21.15.90:57944) 2024-04-25T11:34:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.484+0000 7fde33fff640 1 -- 172.21.15.90:0/3578476997 learned_addr learned my addr 172.21.15.90:0/3578476997 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.484+0000 7fde33fff640 1 -- 172.21.15.90:0/3578476997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5d1913b0 msgr2=0x557f5d56c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.484+0000 7fde33fff640 1 --2- 172.21.15.90:0/3578476997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5d1913b0 0x557f5d56c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.484+0000 7fde33fff640 1 -- 172.21.15.90:0/3578476997 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5d178e40 msgr2=0x557f5d55f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.484+0000 7fde33fff640 1 --2- 172.21.15.90:0/3578476997 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5d178e40 0x557f5d55f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.484+0000 7fde33fff640 1 -- 172.21.15.90:0/3578476997 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f5d55c570 con 0x557f5d199590 2024-04-25T11:34:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.484+0000 7fde33fff640 1 --2- 172.21.15.90:0/3578476997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d199590 0x557f5d55fbd0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fde2c0224b0 tx=0x7fde2c01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80589aaaf0c443fa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.485+0000 7fde197fa640 1 -- 172.21.15.90:0/3578476997 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c005e60 con 0x557f5d199590 2024-04-25T11:34:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.485+0000 7fde197fa640 1 -- 172.21.15.90:0/3578476997 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde2c023c60 con 0x557f5d199590 2024-04-25T11:34:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.485+0000 7fde197fa640 1 -- 172.21.15.90:0/3578476997 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c02e400 con 0x557f5d199590 2024-04-25T11:34:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.485+0000 7fde41026740 1 -- 172.21.15.90:0/3578476997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d199590 msgr2=0x557f5d55fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.485+0000 7fde41026740 1 --2- 172.21.15.90:0/3578476997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d199590 0x557f5d55fbd0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fde2c0224b0 tx=0x7fde2c01ba90 comp rx=0 tx=0).stop 2024-04-25T11:34:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.485+0000 7fde41026740 1 -- 172.21.15.90:0/3578476997 shutdown_connections 2024-04-25T11:34:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.485+0000 7fde41026740 1 --2- 172.21.15.90:0/3578476997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d199590 0x557f5d55fbd0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.485+0000 7fde41026740 1 --2- 172.21.15.90:0/3578476997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5d1913b0 0x557f5d56c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.485+0000 7fde41026740 1 --2- 172.21.15.90:0/3578476997 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5d178e40 0x557f5d55f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.485+0000 7fde41026740 1 -- 172.21.15.90:0/3578476997 >> 172.21.15.90:0/3578476997 conn(0x557f5d1ae3c0 msgr2=0x557f5d551a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.485+0000 7fde41026740 1 -- 172.21.15.90:0/3578476997 shutdown_connections 2024-04-25T11:34:59.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.485+0000 7fde41026740 1 -- 172.21.15.90:0/3578476997 wait complete. 2024-04-25T11:34:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.486+0000 7fde41026740 1 -- start start 2024-04-25T11:34:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.486+0000 7fde41026740 1 Processor -- start 2024-04-25T11:34:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.486+0000 7fde41026740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5d178e40 0x557f5d567b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.486+0000 7fde41026740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d1913b0 0x557f5d568080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.486+0000 7fde41026740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5d199590 0x557f5d56afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.486+0000 7fde41026740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557f5d572e90 con 0x557f5d1913b0 2024-04-25T11:34:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.486+0000 7fde41026740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x557f5d5727c0 con 0x557f5d178e40 2024-04-25T11:34:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.486+0000 7fde41026740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x557f5d573560 con 0x557f5d199590 2024-04-25T11:34:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.486+0000 7fde41026740 10 client.0 _ll_register_callbacks cb 0x7fde2c008b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:34:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.486+0000 7fde32ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d1913b0 0x557f5d568080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.486+0000 7fde32ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d1913b0 0x557f5d568080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57958/0 (socket says 172.21.15.90:57958) 2024-04-25T11:34:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.486+0000 7fde32ffd640 1 -- 172.21.15.90:0/4191072305 learned_addr learned my addr 172.21.15.90:0/4191072305 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.486+0000 7fde337fe640 1 --2- 172.21.15.90:0/4191072305 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5d178e40 0x557f5d567b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.486+0000 7fde32ffd640 1 -- 172.21.15.90:0/4191072305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5d199590 msgr2=0x557f5d56afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.486+0000 7fde32ffd640 1 --2- 172.21.15.90:0/4191072305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5d199590 0x557f5d56afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.487+0000 7fde32ffd640 1 -- 172.21.15.90:0/4191072305 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5d178e40 msgr2=0x557f5d567b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.487+0000 7fde32ffd640 1 --2- 172.21.15.90:0/4191072305 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5d178e40 0x557f5d567b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.487+0000 7fde32ffd640 1 -- 172.21.15.90:0/4191072305 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f5d56b580 con 0x557f5d1913b0 2024-04-25T11:34:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.487+0000 7fde337fe640 1 --2- 172.21.15.90:0/4191072305 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5d178e40 0x557f5d567b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.487+0000 7fde32ffd640 1 --2- 172.21.15.90:0/4191072305 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d1913b0 0x557f5d568080 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fde2800c8f0 tx=0x7fde2800fcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c42b0f3f3d42feeb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.487+0000 7fde12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:34:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.487+0000 7fde12ffd640 1 -- 172.21.15.90:0/4191072305 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde280a7a00 con 0x557f5d1913b0 2024-04-25T11:34:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.487+0000 7fde12ffd640 1 -- 172.21.15.90:0/4191072305 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7fde280a7560 con 0x557f5d1913b0 2024-04-25T11:34:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.487+0000 7fde41026740 1 -- 172.21.15.90:0/4191072305 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde2c023c60 con 0x557f5d1913b0 2024-04-25T11:34:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.487+0000 7fde12ffd640 1 -- 172.21.15.90:0/4191072305 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fde280a7300 con 0x557f5d1913b0 2024-04-25T11:34:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.487+0000 7fde41026740 10 client.15306 fetch_fsmap learned FSMap version 25 2024-04-25T11:34:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.487+0000 7fde41026740 1 -- 172.21.15.90:0/4191072305 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557f5d56c8d0 con 0x557f5d1913b0 2024-04-25T11:34:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.487+0000 7fde12ffd640 1 -- 172.21.15.90:0/4191072305 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde2c023c60 con 0x557f5d1913b0 2024-04-25T11:34:59.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.487+0000 7fde12ffd640 1 -- 172.21.15.90:0/4191072305 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fde280e0070 con 0x557f5d1913b0 2024-04-25T11:34:59.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.487+0000 7fde41026740 10 client.15306 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T11:34:59.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.487+0000 7fde41026740 10 client.15306 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-25T11:34:59.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.487+0000 7fde41026740 20 client.15306 populate_metadata read hostname 'smithi090' 2024-04-25T11:34:59.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.487+0000 7fde41026740 1 --2- 172.21.15.90:0/4191072305 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x557f5d1ba270 0x557f5d5df520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.487+0000 7fde41026740 4 client.15306 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T11:34:59.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.487+0000 7fde41026740 1 -- 172.21.15.90:0/4191072305 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "status"}) -- 0x557f5d4c81c0 con 0x557f5d1ba270 2024-04-25T11:34:59.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.488+0000 7fde337fe640 1 --2- 172.21.15.90:0/4191072305 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x557f5d1ba270 0x557f5d5df520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.488+0000 7fde12ffd640 10 client.15306 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:34:59.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.488+0000 7fde337fe640 1 --2- 172.21.15.90:0/4191072305 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x557f5d1ba270 0x557f5d5df520 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=21aa33dc53bd376c server_cookie=f2707f5da5b57458 in_seq=0 out_seq=0 2024-04-25T11:34:59.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.488+0000 7fde12ffd640 1 -- 172.21.15.90:0/4191072305 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x557f5d4c81c0 con 0x557f5d1ba270 2024-04-25T11:34:59.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.488+0000 7fde12ffd640 10 client.15306 handle_command_reply: tid=0 2024-04-25T11:34:59.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.488+0000 7fde41026740 1 client.15306 shutdown 2024-04-25T11:34:59.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.488+0000 7fde41026740 1 -- 172.21.15.90:0/4191072305 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d1913b0 msgr2=0x557f5d568080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.488+0000 7fde41026740 1 --2- 172.21.15.90:0/4191072305 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d1913b0 0x557f5d568080 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fde2800c8f0 tx=0x7fde2800fcc0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.488+0000 7fde41026740 1 -- 172.21.15.90:0/4191072305 shutdown_connections 2024-04-25T11:34:59.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.488+0000 7fde41026740 1 --2- 172.21.15.90:0/4191072305 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x557f5d1ba270 0x557f5d5df520 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.488+0000 7fde41026740 1 --2- 172.21.15.90:0/4191072305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5d199590 0x557f5d56afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.488+0000 7fde12ffd640 0 client.15306 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:34:59.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.488+0000 7fde41026740 1 --2- 172.21.15.90:0/4191072305 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5d1913b0 0x557f5d568080 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.488+0000 7fde41026740 1 --2- 172.21.15.90:0/4191072305 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5d178e40 0x557f5d567b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.488+0000 7fde41026740 1 -- 172.21.15.90:0/4191072305 >> 172.21.15.90:0/4191072305 conn(0x557f5d1ae3c0 msgr2=0x557f5d55f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.489+0000 7fde41026740 1 -- 172.21.15.90:0/4191072305 shutdown_connections 2024-04-25T11:34:59.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.489+0000 7fde41026740 1 -- 172.21.15.90:0/4191072305 wait complete. 2024-04-25T11:34:59.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.489+0000 7fde41026740 20 client.15306 trim_cache size 0 max 16384 2024-04-25T11:34:59.485 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:34:59.485 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "58bd4e7a-02f6-11ef-bc93-c7b262605968", 2024-04-25T11:34:59.485 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 2, 2024-04-25T11:34:59.485 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 24475, 2024-04-25T11:34:59.485 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-04-25T11:34:59.485 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-04-25T11:34:59.485 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-04-25T11:34:59.485 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 105.875073818, 2024-04-25T11:34:59.485 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 25, 2024-04-25T11:34:59.485 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 79, 2024-04-25T11:34:59.485 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 77, 2024-04-25T11:34:59.485 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 130.69336028000001 2024-04-25T11:34:59.485 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:34:59.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.489+0000 7fde3a096640 1 -- 172.21.15.90:0/33975062 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fde1c0afaf0 msgr2=0x7fde1c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.489+0000 7fde3a096640 1 --2- 172.21.15.90:0/33975062 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fde1c0afaf0 0x7fde1c0b2000 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x557f5d55c540 tx=0x7fde2c008630 comp rx=0 tx=0).stop 2024-04-25T11:34:59.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.489+0000 7fde3a096640 1 -- 172.21.15.90:0/33975062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde34103c40 msgr2=0x7fde3419f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.489+0000 7fde3a096640 1 --2- 172.21.15.90:0/33975062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde34103c40 0x7fde3419f050 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fde2800c920 tx=0x7fde2800cde0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.489+0000 7fde3a096640 1 -- 172.21.15.90:0/33975062 shutdown_connections 2024-04-25T11:34:59.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.489+0000 7fde3a096640 1 --2- 172.21.15.90:0/33975062 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde34108940 0x7fde341a15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.489+0000 7fde3a096640 1 --2- 172.21.15.90:0/33975062 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fde1c0afaf0 0x7fde1c0b2000 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.489+0000 7fde3a096640 1 --2- 172.21.15.90:0/33975062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde34103c40 0x7fde3419f050 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.489+0000 7fde3a096640 1 --2- 172.21.15.90:0/33975062 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fde34102a00 0x7fde34075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.489+0000 7fde3a096640 1 -- 172.21.15.90:0/33975062 >> 172.21.15.90:0/33975062 conn(0x7fde340fdbf0 msgr2=0x7fde34104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.489+0000 7fde3a096640 1 -- 172.21.15.90:0/33975062 shutdown_connections 2024-04-25T11:34:59.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.489+0000 7fde3a096640 1 -- 172.21.15.90:0/33975062 wait complete. 2024-04-25T11:34:59.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-25T11:34:59.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.608+0000 7f09aad9d640 1 -- 172.21.15.90:0/2949241839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09a4109870 msgr2=0x7f09a410bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.608+0000 7f09aad9d640 1 --2- 172.21.15.90:0/2949241839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09a4109870 0x7f09a410bc90 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f099c009900 tx=0x7f099c02a410 comp rx=0 tx=0).stop 2024-04-25T11:34:59.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.609+0000 7f09aad9d640 1 -- 172.21.15.90:0/2949241839 shutdown_connections 2024-04-25T11:34:59.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.609+0000 7f09aad9d640 1 --2- 172.21.15.90:0/2949241839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09a4109870 0x7f09a410bc90 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.609+0000 7f09aad9d640 1 --2- 172.21.15.90:0/2949241839 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f09a4104b70 0x7f09a4104f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.609+0000 7f09aad9d640 1 --2- 172.21.15.90:0/2949241839 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09a4103930 0x7f09a4104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.609+0000 7f09aad9d640 1 -- 172.21.15.90:0/2949241839 >> 172.21.15.90:0/2949241839 conn(0x7f09a40feb20 msgr2=0x7f09a4100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.609+0000 7f09aad9d640 1 -- 172.21.15.90:0/2949241839 shutdown_connections 2024-04-25T11:34:59.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.609+0000 7f09aad9d640 1 -- 172.21.15.90:0/2949241839 wait complete. 2024-04-25T11:34:59.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.609+0000 7f09aad9d640 1 Processor -- start 2024-04-25T11:34:59.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.609+0000 7f09aad9d640 1 -- start start 2024-04-25T11:34:59.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.609+0000 7f09aad9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f09a4103930 0x7f09a419a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.609+0000 7f09aad9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09a4104b70 0x7f09a419a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.609+0000 7f09aad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09a4109870 0x7f09a41a14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.609+0000 7f09aad9d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f09a410f460 con 0x7f09a4109870 2024-04-25T11:34:59.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.609+0000 7f09aad9d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f09a410f2e0 con 0x7f09a4103930 2024-04-25T11:34:59.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.609+0000 7f09aad9d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f09a410f5e0 con 0x7f09a4104b70 2024-04-25T11:34:59.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.610+0000 7f09aa59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09a4109870 0x7f09a41a14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.610+0000 7f09aa59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09a4109870 0x7f09a41a14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57972/0 (socket says 172.21.15.90:57972) 2024-04-25T11:34:59.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.610+0000 7f09aa59c640 1 -- 172.21.15.90:0/4174331840 learned_addr learned my addr 172.21.15.90:0/4174331840 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:59.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.610+0000 7f09a959a640 1 --2- 172.21.15.90:0/4174331840 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09a4104b70 0x7f09a419a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.610+0000 7f09a9d9b640 1 --2- 172.21.15.90:0/4174331840 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f09a4103930 0x7f09a419a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.610+0000 7f09aa59c640 1 -- 172.21.15.90:0/4174331840 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09a4104b70 msgr2=0x7f09a419a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.610+0000 7f09aa59c640 1 --2- 172.21.15.90:0/4174331840 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09a4104b70 0x7f09a419a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.610+0000 7f09aa59c640 1 -- 172.21.15.90:0/4174331840 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f09a4103930 msgr2=0x7f09a419a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.610+0000 7f09aa59c640 1 --2- 172.21.15.90:0/4174331840 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f09a4103930 0x7f09a419a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.610+0000 7f09aa59c640 1 -- 172.21.15.90:0/4174331840 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a41a1a90 con 0x7f09a4109870 2024-04-25T11:34:59.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.610+0000 7f09aa59c640 1 --2- 172.21.15.90:0/4174331840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09a4109870 0x7f09a41a14d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f099c009230 tx=0x7f099c031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.610+0000 7f099affd640 1 -- 172.21.15.90:0/4174331840 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099c03f030 con 0x7f09a4109870 2024-04-25T11:34:59.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.610+0000 7f099affd640 1 -- 172.21.15.90:0/4174331840 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f099c003aa0 con 0x7f09a4109870 2024-04-25T11:34:59.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.610+0000 7f09aad9d640 1 -- 172.21.15.90:0/4174331840 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09a41a1d20 con 0x7f09a4109870 2024-04-25T11:34:59.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.610+0000 7f099affd640 1 -- 172.21.15.90:0/4174331840 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099c035070 con 0x7f09a4109870 2024-04-25T11:34:59.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.610+0000 7f09aad9d640 1 -- 172.21.15.90:0/4174331840 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09a41a21c0 con 0x7f09a4109870 2024-04-25T11:34:59.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.613+0000 7f09b2f85740 1 Processor -- start 2024-04-25T11:34:59.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.613+0000 7f099affd640 1 -- 172.21.15.90:0/4174331840 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f099c042030 con 0x7f09a4109870 2024-04-25T11:34:59.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.614+0000 7f09b2f85740 1 -- start start 2024-04-25T11:34:59.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.614+0000 7f09b2f85740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564327311270 0x5643276b2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.614+0000 7f09b2f85740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643272e83b0 0x5643276b2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.614+0000 7f09b2f85740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643272f0590 0x5643276b99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.614+0000 7f09b2f85740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564327617740 con 0x5643272e83b0 2024-04-25T11:34:59.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.614+0000 7f09b2f85740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5643275b33a0 con 0x564327311270 2024-04-25T11:34:59.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.614+0000 7f09b2f85740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x564327613700 con 0x5643272f0590 2024-04-25T11:34:59.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.614+0000 7f09a9d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643272e83b0 0x5643276b2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.614+0000 7f09a9d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643272e83b0 0x5643276b2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57980/0 (socket says 172.21.15.90:57980) 2024-04-25T11:34:59.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.614+0000 7f09a9d9b640 1 -- 172.21.15.90:0/3788775910 learned_addr learned my addr 172.21.15.90:0/3788775910 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:59.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.614+0000 7f09a959a640 1 --2- 172.21.15.90:0/3788775910 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564327311270 0x5643276b2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.614+0000 7f099affd640 1 --2- 172.21.15.90:0/4174331840 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f09900afae0 0x7f09900b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.614+0000 7f09a9d9b640 1 -- 172.21.15.90:0/3788775910 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643272f0590 msgr2=0x5643276b99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.614+0000 7f09a959a640 1 --2- 172.21.15.90:0/3788775910 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643272f0590 0x5643276b99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.614+0000 7f09a9d9b640 1 --2- 172.21.15.90:0/3788775910 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643272f0590 0x5643276b99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.614+0000 7f09a9d9b640 1 -- 172.21.15.90:0/3788775910 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564327311270 msgr2=0x5643276b2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.614+0000 7f09a9d9b640 1 --2- 172.21.15.90:0/3788775910 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564327311270 0x5643276b2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.614+0000 7f09a9d9b640 1 -- 172.21.15.90:0/3788775910 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643275cf750 con 0x5643272e83b0 2024-04-25T11:34:59.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.614+0000 7f09aa59c640 1 --2- 172.21.15.90:0/4174331840 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f09900afae0 0x7f09900b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.614+0000 7f099affd640 1 -- 172.21.15.90:0/4174331840 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f099c0ff4c0 con 0x7f09a4109870 2024-04-25T11:34:59.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.614+0000 7f09a9d9b640 1 --2- 172.21.15.90:0/3788775910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643272e83b0 0x5643276b2ae0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f09a00103a0 tx=0x7f09a0010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7ffa23fffef7705 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.615+0000 7f098ffff640 1 -- 172.21.15.90:0/3788775910 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0016070 con 0x5643272e83b0 2024-04-25T11:34:59.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.615+0000 7f098ffff640 1 -- 172.21.15.90:0/3788775910 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09a0003b70 con 0x5643272e83b0 2024-04-25T11:34:59.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.615+0000 7f098ffff640 1 -- 172.21.15.90:0/3788775910 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a00128b0 con 0x5643272e83b0 2024-04-25T11:34:59.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.615+0000 7f09b2f85740 1 -- 172.21.15.90:0/3788775910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643272e83b0 msgr2=0x5643276b2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.615+0000 7f09b2f85740 1 --2- 172.21.15.90:0/3788775910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643272e83b0 0x5643276b2ae0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f09a00103a0 tx=0x7f09a0010860 comp rx=0 tx=0).stop 2024-04-25T11:34:59.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.615+0000 7f09aa59c640 1 --2- 172.21.15.90:0/4174331840 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f09900afae0 0x7f09900b1ff0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f099c0fd310 tx=0x7f099c0b0040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.615+0000 7f09b2f85740 1 -- 172.21.15.90:0/3788775910 shutdown_connections 2024-04-25T11:34:59.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.615+0000 7f09b2f85740 1 --2- 172.21.15.90:0/3788775910 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643272f0590 0x5643276b99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.615+0000 7f09b2f85740 1 --2- 172.21.15.90:0/3788775910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643272e83b0 0x5643276b2ae0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.615+0000 7f09b2f85740 1 --2- 172.21.15.90:0/3788775910 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564327311270 0x5643276b2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.616+0000 7f09b2f85740 1 -- 172.21.15.90:0/3788775910 >> 172.21.15.90:0/3788775910 conn(0x5643273053c0 msgr2=0x5643276adef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.616+0000 7f09b2f85740 1 -- 172.21.15.90:0/3788775910 shutdown_connections 2024-04-25T11:34:59.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.616+0000 7f09b2f85740 1 -- 172.21.15.90:0/3788775910 wait complete. 2024-04-25T11:34:59.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.616+0000 7f09b2f85740 1 -- start start 2024-04-25T11:34:59.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.616+0000 7f09b2f85740 1 Processor -- start 2024-04-25T11:34:59.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.616+0000 7f09b2f85740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643272e83b0 0x5643276bc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.616+0000 7f09b2f85740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643272f0590 0x5643276bc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.616+0000 7f09b2f85740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564327311270 0x5643276c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.616+0000 7f09b2f85740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5643276ba430 con 0x5643272e83b0 2024-04-25T11:34:59.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.616+0000 7f09b2f85740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56432759efe0 con 0x564327311270 2024-04-25T11:34:59.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.616+0000 7f09b2f85740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5643276ba5b0 con 0x5643272f0590 2024-04-25T11:34:59.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.616+0000 7f09b2f85740 10 client.0 _ll_register_callbacks cb 0x5643275b5150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:34:59.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.617+0000 7f09a959a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643272e83b0 0x5643276bc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.617+0000 7f09a959a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643272e83b0 0x5643276bc460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57982/0 (socket says 172.21.15.90:57982) 2024-04-25T11:34:59.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.617+0000 7f09a959a640 1 -- 172.21.15.90:0/382056507 learned_addr learned my addr 172.21.15.90:0/382056507 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:59.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.617+0000 7f09a9d9b640 1 --2- 172.21.15.90:0/382056507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643272f0590 0x5643276bc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.617+0000 7f09a959a640 1 -- 172.21.15.90:0/382056507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643272f0590 msgr2=0x5643276bc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.617+0000 7f09a959a640 1 --2- 172.21.15.90:0/382056507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643272f0590 0x5643276bc9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.617+0000 7f09a959a640 1 -- 172.21.15.90:0/382056507 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564327311270 msgr2=0x5643276c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.617+0000 7f09a959a640 1 --2- 172.21.15.90:0/382056507 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564327311270 0x5643276c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.617+0000 7f09a959a640 1 -- 172.21.15.90:0/382056507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643273292d0 con 0x5643272e83b0 2024-04-25T11:34:59.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.617+0000 7f09a959a640 1 --2- 172.21.15.90:0/382056507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643272e83b0 0x5643276bc460 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0994010a50 tx=0x7f0994010f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d561305869dafd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.617+0000 7f098d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:34:59.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.617+0000 7f098d7fa640 1 -- 172.21.15.90:0/382056507 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099401f060 con 0x5643272e83b0 2024-04-25T11:34:59.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.617+0000 7f098d7fa640 1 -- 172.21.15.90:0/382056507 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f099400fbd0 con 0x5643272e83b0 2024-04-25T11:34:59.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.617+0000 7f09b2f85740 1 -- 172.21.15.90:0/382056507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09a0003b70 con 0x5643272e83b0 2024-04-25T11:34:59.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.617+0000 7f098d7fa640 1 -- 172.21.15.90:0/382056507 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f099400fe10 con 0x5643272e83b0 2024-04-25T11:34:59.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.617+0000 7f09b2f85740 10 client.15318 fetch_fsmap learned FSMap version 25 2024-04-25T11:34:59.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.617+0000 7f09b2f85740 1 -- 172.21.15.90:0/382056507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5643273f36b0 con 0x5643272e83b0 2024-04-25T11:34:59.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.617+0000 7f098d7fa640 1 -- 172.21.15.90:0/382056507 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09a0003b70 con 0x5643272e83b0 2024-04-25T11:34:59.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.618+0000 7f098d7fa640 1 -- 172.21.15.90:0/382056507 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f09940244d0 con 0x5643272e83b0 2024-04-25T11:34:59.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.618+0000 7f09b2f85740 10 client.15318 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T11:34:59.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.618+0000 7f09b2f85740 10 client.15318 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T11:34:59.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.618+0000 7f09b2f85740 20 client.15318 populate_metadata read hostname 'smithi090' 2024-04-25T11:34:59.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.618+0000 7f09b2f85740 1 --2- 172.21.15.90:0/382056507 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5643272cfe40 0x5643276ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.618+0000 7f09b2f85740 4 client.15318 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:34:59.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.618+0000 7f09b2f85740 1 -- 172.21.15.90:0/382056507 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564327625350 con 0x5643272cfe40 2024-04-25T11:34:59.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.618+0000 7f09a9d9b640 1 --2- 172.21.15.90:0/382056507 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5643272cfe40 0x5643276ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.618+0000 7f098d7fa640 10 client.15318 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:34:59.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.618+0000 7f09a9d9b640 1 --2- 172.21.15.90:0/382056507 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5643272cfe40 0x5643276ca1a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b5de762be8c97476 server_cookie=7d6b49c227c3a29e in_seq=0 out_seq=0 2024-04-25T11:34:59.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.619+0000 7f098d7fa640 1 -- 172.21.15.90:0/382056507 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564327625350 con 0x5643272cfe40 2024-04-25T11:34:59.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.619+0000 7f098d7fa640 10 client.15318 handle_command_reply: tid=0 2024-04-25T11:34:59.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.619+0000 7f09b2f85740 1 client.15318 shutdown 2024-04-25T11:34:59.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.620+0000 7f09b2f85740 1 -- 172.21.15.90:0/382056507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643272e83b0 msgr2=0x5643276bc460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.620+0000 7f09b2f85740 1 --2- 172.21.15.90:0/382056507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643272e83b0 0x5643276bc460 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0994010a50 tx=0x7f0994010f10 comp rx=0 tx=0).stop 2024-04-25T11:34:59.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.620+0000 7f09b2f85740 1 -- 172.21.15.90:0/382056507 shutdown_connections 2024-04-25T11:34:59.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.620+0000 7f09b2f85740 1 --2- 172.21.15.90:0/382056507 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5643272cfe40 0x5643276ca1a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.620+0000 7f09b2f85740 1 --2- 172.21.15.90:0/382056507 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564327311270 0x5643276c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.620+0000 7f09b2f85740 1 --2- 172.21.15.90:0/382056507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643272f0590 0x5643276bc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.620+0000 7f09b2f85740 1 --2- 172.21.15.90:0/382056507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643272e83b0 0x5643276bc460 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.620+0000 7f098d7fa640 0 client.15318 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:34:59.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.620+0000 7f09b2f85740 1 -- 172.21.15.90:0/382056507 >> 172.21.15.90:0/382056507 conn(0x5643273053c0 msgr2=0x5643276acf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.620+0000 7f09b2f85740 1 -- 172.21.15.90:0/382056507 shutdown_connections 2024-04-25T11:34:59.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.620+0000 7f09b2f85740 1 -- 172.21.15.90:0/382056507 wait complete. 2024-04-25T11:34:59.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.620+0000 7f09b2f85740 20 client.15318 trim_cache size 0 max 16384 2024-04-25T11:34:59.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.638+0000 7f09b2f85740 1 Processor -- start 2024-04-25T11:34:59.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.638+0000 7f09b2f85740 1 -- start start 2024-04-25T11:34:59.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.638+0000 7f09b2f85740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643272e83b0 0x5643276bb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.638+0000 7f09b2f85740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643272f0590 0x5643276c3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.638+0000 7f09b2f85740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564327311270 0x5643276baaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.638+0000 7f09b2f85740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5643276c84e0 con 0x5643272f0590 2024-04-25T11:34:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.638+0000 7f09b2f85740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5643276c8bb0 con 0x5643272e83b0 2024-04-25T11:34:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.638+0000 7f09b2f85740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5643276c9280 con 0x564327311270 2024-04-25T11:34:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.638+0000 7f09a9d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643272f0590 0x5643276c3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.638+0000 7f09a9d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643272f0590 0x5643276c3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57996/0 (socket says 172.21.15.90:57996) 2024-04-25T11:34:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.638+0000 7f09a9d9b640 1 -- 172.21.15.90:0/3344133110 learned_addr learned my addr 172.21.15.90:0/3344133110 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.638+0000 7f09a959a640 1 --2- 172.21.15.90:0/3344133110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564327311270 0x5643276baaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.638+0000 7f09a9d9b640 1 -- 172.21.15.90:0/3344133110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564327311270 msgr2=0x5643276baaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.638+0000 7f09a9d9b640 1 --2- 172.21.15.90:0/3344133110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564327311270 0x5643276baaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.638+0000 7f09a9d9b640 1 -- 172.21.15.90:0/3344133110 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643272e83b0 msgr2=0x5643276bb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.638+0000 7f09a9d9b640 1 --2- 172.21.15.90:0/3344133110 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643272e83b0 0x5643276bb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.638+0000 7f09a9d9b640 1 -- 172.21.15.90:0/3344133110 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643276b72e0 con 0x5643272f0590 2024-04-25T11:34:59.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.638+0000 7f09a959a640 1 --2- 172.21.15.90:0/3344133110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564327311270 0x5643276baaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:59.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.638+0000 7f09a9d9b640 1 --2- 172.21.15.90:0/3344133110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643272f0590 0x5643276c3890 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x5643276b3940 tx=0x7f09a0009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3639c9dd3f2e4df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.638+0000 7f098f7fe640 1 -- 172.21.15.90:0/3344133110 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0016040 con 0x5643272f0590 2024-04-25T11:34:59.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.639+0000 7f098f7fe640 1 -- 172.21.15.90:0/3344133110 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09a0003d70 con 0x5643272f0590 2024-04-25T11:34:59.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.639+0000 7f098f7fe640 1 -- 172.21.15.90:0/3344133110 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0012e00 con 0x5643272f0590 2024-04-25T11:34:59.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.639+0000 7f09b2f85740 1 -- 172.21.15.90:0/3344133110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643272f0590 msgr2=0x5643276c3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.639+0000 7f09b2f85740 1 --2- 172.21.15.90:0/3344133110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643272f0590 0x5643276c3890 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x5643276b3940 tx=0x7f09a0009680 comp rx=0 tx=0).stop 2024-04-25T11:34:59.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.639+0000 7f09b2f85740 1 -- 172.21.15.90:0/3344133110 shutdown_connections 2024-04-25T11:34:59.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.639+0000 7f09b2f85740 1 --2- 172.21.15.90:0/3344133110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564327311270 0x5643276baaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.639+0000 7f09b2f85740 1 --2- 172.21.15.90:0/3344133110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643272f0590 0x5643276c3890 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.639+0000 7f09b2f85740 1 --2- 172.21.15.90:0/3344133110 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643272e83b0 0x5643276bb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.639+0000 7f09b2f85740 1 -- 172.21.15.90:0/3344133110 >> 172.21.15.90:0/3344133110 conn(0x5643273053c0 msgr2=0x5643276b5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.639+0000 7f09b2f85740 1 -- 172.21.15.90:0/3344133110 shutdown_connections 2024-04-25T11:34:59.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.639+0000 7f09b2f85740 1 -- 172.21.15.90:0/3344133110 wait complete. 2024-04-25T11:34:59.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.640+0000 7f09b2f85740 1 -- start start 2024-04-25T11:34:59.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.640+0000 7f09b2f85740 1 Processor -- start 2024-04-25T11:34:59.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.640+0000 7f09b2f85740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643272e83b0 0x5643276bf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.640+0000 7f09b2f85740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643272f0590 0x5643276bf570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.640+0000 7f09b2f85740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564327311270 0x5643276c24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.640+0000 7f09b2f85740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5643276ca020 con 0x564327311270 2024-04-25T11:34:59.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.640+0000 7f09b2f85740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5643276c9950 con 0x5643272e83b0 2024-04-25T11:34:59.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.640+0000 7f09b2f85740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5643276ca6f0 con 0x5643272f0590 2024-04-25T11:34:59.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.640+0000 7f09b2f85740 10 client.0 _ll_register_callbacks cb 0x7f099400a7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:34:59.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.640+0000 7f09a959a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564327311270 0x5643276c24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.640+0000 7f09a959a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564327311270 0x5643276c24b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58010/0 (socket says 172.21.15.90:58010) 2024-04-25T11:34:59.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.640+0000 7f09a959a640 1 -- 172.21.15.90:0/2350473436 learned_addr learned my addr 172.21.15.90:0/2350473436 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:59.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.640+0000 7f09a9d9b640 1 --2- 172.21.15.90:0/2350473436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643272f0590 0x5643276bf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.640+0000 7f09a959a640 1 --2- 172.21.15.90:0/2350473436 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643272e83b0 0x5643276bf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.641+0000 7f09a959a640 1 -- 172.21.15.90:0/2350473436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643272f0590 msgr2=0x5643276bf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.641+0000 7f09a959a640 1 --2- 172.21.15.90:0/2350473436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643272f0590 0x5643276bf570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.641+0000 7f09a959a640 1 -- 172.21.15.90:0/2350473436 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643272e83b0 msgr2=0x5643276bf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.641+0000 7f09a959a640 1 --2- 172.21.15.90:0/2350473436 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643272e83b0 0x5643276bf030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.641+0000 7f09a959a640 1 -- 172.21.15.90:0/2350473436 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643276c2a70 con 0x564327311270 2024-04-25T11:34:59.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.641+0000 7f09a959a640 1 --2- 172.21.15.90:0/2350473436 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564327311270 0x5643276c24b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x5643275bc980 tx=0x7f0994027040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e5a6902c9dfa720 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.641+0000 7f098d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:34:59.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.641+0000 7f098d7fa640 1 -- 172.21.15.90:0/2350473436 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099400a2d0 con 0x564327311270 2024-04-25T11:34:59.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.641+0000 7f098d7fa640 1 -- 172.21.15.90:0/2350473436 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f099400fbd0 con 0x564327311270 2024-04-25T11:34:59.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.641+0000 7f09b2f85740 1 -- 172.21.15.90:0/2350473436 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09a0003d70 con 0x564327311270 2024-04-25T11:34:59.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.641+0000 7f098d7fa640 1 -- 172.21.15.90:0/2350473436 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0994036070 con 0x564327311270 2024-04-25T11:34:59.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.641+0000 7f09b2f85740 10 client.15324 fetch_fsmap learned FSMap version 25 2024-04-25T11:34:59.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.641+0000 7f09b2f85740 1 -- 172.21.15.90:0/2350473436 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564327733020 con 0x564327311270 2024-04-25T11:34:59.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.641+0000 7f098d7fa640 1 -- 172.21.15.90:0/2350473436 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09a0003d70 con 0x564327311270 2024-04-25T11:34:59.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.641+0000 7f098d7fa640 1 -- 172.21.15.90:0/2350473436 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f099402e020 con 0x564327311270 2024-04-25T11:34:59.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.641+0000 7f09b2f85740 10 client.15324 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T11:34:59.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.641+0000 7f09b2f85740 10 client.15324 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T11:34:59.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.641+0000 7f09b2f85740 20 client.15324 populate_metadata read hostname 'smithi090' 2024-04-25T11:34:59.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.641+0000 7f09b2f85740 1 --2- 172.21.15.90:0/2350473436 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5643272a5280 0x564327736810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.641+0000 7f09b2f85740 4 client.15324 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T11:34:59.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.641+0000 7f09b2f85740 1 -- 172.21.15.90:0/2350473436 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "status"}) -- 0x56432761f1c0 con 0x5643272a5280 2024-04-25T11:34:59.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.642+0000 7f09a9d9b640 1 --2- 172.21.15.90:0/2350473436 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5643272a5280 0x564327736810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.642+0000 7f098d7fa640 10 client.15324 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:34:59.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.642+0000 7f09a9d9b640 1 --2- 172.21.15.90:0/2350473436 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5643272a5280 0x564327736810 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c447b8166ff213d4 server_cookie=e019540abd57628a in_seq=0 out_seq=0 2024-04-25T11:34:59.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.642+0000 7f098d7fa640 1 -- 172.21.15.90:0/2350473436 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x56432761f1c0 con 0x5643272a5280 2024-04-25T11:34:59.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.642+0000 7f098d7fa640 10 client.15324 handle_command_reply: tid=0 2024-04-25T11:34:59.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.642+0000 7f09b2f85740 1 client.15324 shutdown 2024-04-25T11:34:59.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.643+0000 7f09b2f85740 1 -- 172.21.15.90:0/2350473436 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564327311270 msgr2=0x5643276c24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.643+0000 7f09b2f85740 1 --2- 172.21.15.90:0/2350473436 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564327311270 0x5643276c24b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x5643275bc980 tx=0x7f0994027040 comp rx=0 tx=0).stop 2024-04-25T11:34:59.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.643+0000 7f09b2f85740 1 -- 172.21.15.90:0/2350473436 shutdown_connections 2024-04-25T11:34:59.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.643+0000 7f09b2f85740 1 --2- 172.21.15.90:0/2350473436 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5643272a5280 0x564327736810 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.643+0000 7f09b2f85740 1 --2- 172.21.15.90:0/2350473436 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564327311270 0x5643276c24b0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.643+0000 7f09b2f85740 1 --2- 172.21.15.90:0/2350473436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643272f0590 0x5643276bf570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.643+0000 7f09b2f85740 1 --2- 172.21.15.90:0/2350473436 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643272e83b0 0x5643276bf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.643+0000 7f098d7fa640 0 client.15324 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:34:59.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.643+0000 7f09b2f85740 1 -- 172.21.15.90:0/2350473436 >> 172.21.15.90:0/2350473436 conn(0x5643273053c0 msgr2=0x5643276ba9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.643+0000 7f09b2f85740 1 -- 172.21.15.90:0/2350473436 shutdown_connections 2024-04-25T11:34:59.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.643+0000 7f09b2f85740 1 -- 172.21.15.90:0/2350473436 wait complete. 2024-04-25T11:34:59.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.643+0000 7f09b2f85740 20 client.15324 trim_cache size 0 max 16384 2024-04-25T11:34:59.639 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:34:59.639 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "58bd4e7a-02f6-11ef-bc93-c7b262605968", 2024-04-25T11:34:59.639 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 1, 2024-04-25T11:34:59.639 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 24499, 2024-04-25T11:34:59.639 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-04-25T11:34:59.639 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-04-25T11:34:59.639 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-04-25T11:34:59.639 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 106.610304462, 2024-04-25T11:34:59.639 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 25, 2024-04-25T11:34:59.639 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 79, 2024-04-25T11:34:59.639 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 77, 2024-04-25T11:34:59.639 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 122.597345887 2024-04-25T11:34:59.640 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:34:59.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.643+0000 7f09aad9d640 1 -- 172.21.15.90:0/4174331840 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f09900afae0 msgr2=0x7f09900b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.643+0000 7f09aad9d640 1 --2- 172.21.15.90:0/4174331840 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f09900afae0 0x7f09900b1ff0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f099c0fd310 tx=0x7f099c0b0040 comp rx=0 tx=0).stop 2024-04-25T11:34:59.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.643+0000 7f09aad9d640 1 -- 172.21.15.90:0/4174331840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09a4109870 msgr2=0x7f09a41a14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.643+0000 7f09aad9d640 1 --2- 172.21.15.90:0/4174331840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09a4109870 0x7f09a41a14d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f099c009230 tx=0x7f099c031a50 comp rx=0 tx=0).stop 2024-04-25T11:34:59.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.643+0000 7f09aad9d640 1 -- 172.21.15.90:0/4174331840 shutdown_connections 2024-04-25T11:34:59.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.643+0000 7f09aad9d640 1 --2- 172.21.15.90:0/4174331840 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f09900afae0 0x7f09900b1ff0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.643+0000 7f09aad9d640 1 --2- 172.21.15.90:0/4174331840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09a4109870 0x7f09a41a14d0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.643+0000 7f09aad9d640 1 --2- 172.21.15.90:0/4174331840 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09a4104b70 0x7f09a419a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.643+0000 7f09aad9d640 1 --2- 172.21.15.90:0/4174331840 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f09a4103930 0x7f09a419a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.643+0000 7f09aad9d640 1 -- 172.21.15.90:0/4174331840 >> 172.21.15.90:0/4174331840 conn(0x7f09a40feb20 msgr2=0x7f09a4105850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.643+0000 7f09aad9d640 1 -- 172.21.15.90:0/4174331840 shutdown_connections 2024-04-25T11:34:59.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.643+0000 7f09aad9d640 1 -- 172.21.15.90:0/4174331840 wait complete. 2024-04-25T11:34:59.643 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T11:34:59.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.763+0000 7f08d4c40640 1 -- 172.21.15.90:0/500612069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08d01029f0 msgr2=0x7f08d01036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.763+0000 7f08d4c40640 1 --2- 172.21.15.90:0/500612069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08d01029f0 0x7f08d01036f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f08c8009900 tx=0x7f08c802a410 comp rx=0 tx=0).stop 2024-04-25T11:34:59.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.763+0000 7f08d4c40640 1 -- 172.21.15.90:0/500612069 shutdown_connections 2024-04-25T11:34:59.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.763+0000 7f08d4c40640 1 --2- 172.21.15.90:0/500612069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08d0108930 0x7f08d010ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.763+0000 7f08d4c40640 1 --2- 172.21.15.90:0/500612069 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f08d0103c30 0x7f08d0104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.763+0000 7f08d4c40640 1 --2- 172.21.15.90:0/500612069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08d01029f0 0x7f08d01036f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.763+0000 7f08d4c40640 1 -- 172.21.15.90:0/500612069 >> 172.21.15.90:0/500612069 conn(0x7f08d00fdbc0 msgr2=0x7f08d0100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.763+0000 7f08d4c40640 1 -- 172.21.15.90:0/500612069 shutdown_connections 2024-04-25T11:34:59.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.763+0000 7f08d4c40640 1 -- 172.21.15.90:0/500612069 wait complete. 2024-04-25T11:34:59.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.764+0000 7f08d4c40640 1 Processor -- start 2024-04-25T11:34:59.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.764+0000 7f08d4c40640 1 -- start start 2024-04-25T11:34:59.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.764+0000 7f08d4c40640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08d01029f0 0x7f08d01991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.764+0000 7f08d4c40640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f08d0103c30 0x7f08d0199700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.764+0000 7f08d4c40640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08d0108930 0x7f08d01a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.764+0000 7f08d4c40640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f08d010e4d0 con 0x7f08d0108930 2024-04-25T11:34:59.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.764+0000 7f08d4c40640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f08d010e350 con 0x7f08d0103c30 2024-04-25T11:34:59.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.764+0000 7f08d4c40640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f08d010e650 con 0x7f08d01029f0 2024-04-25T11:34:59.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.764+0000 7f08ced9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08d0108930 0x7f08d01a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.764+0000 7f08ced9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08d0108930 0x7f08d01a0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58022/0 (socket says 172.21.15.90:58022) 2024-04-25T11:34:59.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.764+0000 7f08ced9d640 1 -- 172.21.15.90:0/4109082398 learned_addr learned my addr 172.21.15.90:0/4109082398 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:59.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.765+0000 7f08cdd9b640 1 --2- 172.21.15.90:0/4109082398 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f08d0103c30 0x7f08d0199700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.765+0000 7f08ce59c640 1 --2- 172.21.15.90:0/4109082398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08d01029f0 0x7f08d01991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.765+0000 7f08ced9d640 1 -- 172.21.15.90:0/4109082398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08d01029f0 msgr2=0x7f08d01991c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.765+0000 7f08ced9d640 1 --2- 172.21.15.90:0/4109082398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08d01029f0 0x7f08d01991c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.765+0000 7f08ced9d640 1 -- 172.21.15.90:0/4109082398 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f08d0103c30 msgr2=0x7f08d0199700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.765+0000 7f08ced9d640 1 --2- 172.21.15.90:0/4109082398 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f08d0103c30 0x7f08d0199700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.765+0000 7f08ced9d640 1 -- 172.21.15.90:0/4109082398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08d01a0be0 con 0x7f08d0108930 2024-04-25T11:34:59.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.765+0000 7f08cdd9b640 1 --2- 172.21.15.90:0/4109082398 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f08d0103c30 0x7f08d0199700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:59.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.765+0000 7f08ced9d640 1 --2- 172.21.15.90:0/4109082398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08d0108930 0x7f08d01a0620 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f08c000cce0 tx=0x7f08c000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.765+0000 7f08bf7fe640 1 -- 172.21.15.90:0/4109082398 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c000f450 con 0x7f08d0108930 2024-04-25T11:34:59.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.765+0000 7f08bf7fe640 1 -- 172.21.15.90:0/4109082398 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08c000faa0 con 0x7f08d0108930 2024-04-25T11:34:59.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.765+0000 7f08d4c40640 1 -- 172.21.15.90:0/4109082398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08d01a0ed0 con 0x7f08d0108930 2024-04-25T11:34:59.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.765+0000 7f08bf7fe640 1 -- 172.21.15.90:0/4109082398 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c0018b70 con 0x7f08d0108930 2024-04-25T11:34:59.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.765+0000 7f08d4c40640 1 -- 172.21.15.90:0/4109082398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08d01a1330 con 0x7f08d0108930 2024-04-25T11:34:59.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.767+0000 7f08bf7fe640 1 -- 172.21.15.90:0/4109082398 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f08c000f5f0 con 0x7f08d0108930 2024-04-25T11:34:59.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.768+0000 7f08dbbcd740 1 Processor -- start 2024-04-25T11:34:59.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.768+0000 7f08dbbcd740 1 -- start start 2024-04-25T11:34:59.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.768+0000 7f08dbbcd740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5650ed1cb270 0x5650ed56e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.768+0000 7f08dbbcd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5650ed1a23b0 0x5650ed56eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.768+0000 7f08dbbcd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1aa590 0x5650ed575a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.768+0000 7f08dbbcd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5650ed4d1740 con 0x5650ed1aa590 2024-04-25T11:34:59.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.768+0000 7f08dbbcd740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5650ed46d3a0 con 0x5650ed1cb270 2024-04-25T11:34:59.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.768+0000 7f08dbbcd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5650ed4cd700 con 0x5650ed1a23b0 2024-04-25T11:34:59.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.768+0000 7f08cdd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1aa590 0x5650ed575a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.768+0000 7f08cdd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1aa590 0x5650ed575a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58036/0 (socket says 172.21.15.90:58036) 2024-04-25T11:34:59.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.768+0000 7f08cdd9b640 1 -- 172.21.15.90:0/2067763589 learned_addr learned my addr 172.21.15.90:0/2067763589 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:59.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.768+0000 7f08cdd9b640 1 --2- 172.21.15.90:0/2067763589 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5650ed1cb270 0x5650ed56e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.768+0000 7f08ce59c640 1 --2- 172.21.15.90:0/2067763589 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5650ed1a23b0 0x5650ed56eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.768+0000 7f08bf7fe640 1 --2- 172.21.15.90:0/4109082398 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f08b80afaf0 0x7f08b80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.768+0000 7f08cdd9b640 1 -- 172.21.15.90:0/2067763589 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5650ed1a23b0 msgr2=0x5650ed56eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.768+0000 7f08cdd9b640 1 --2- 172.21.15.90:0/2067763589 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5650ed1a23b0 0x5650ed56eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.768+0000 7f08cdd9b640 1 -- 172.21.15.90:0/2067763589 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5650ed1cb270 msgr2=0x5650ed56e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.768+0000 7f08cdd9b640 1 --2- 172.21.15.90:0/2067763589 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5650ed1cb270 0x5650ed56e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.768+0000 7f08ced9d640 1 --2- 172.21.15.90:0/4109082398 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f08b80afaf0 0x7f08b80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.768+0000 7f08cdd9b640 1 -- 172.21.15.90:0/2067763589 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650ed489750 con 0x5650ed1aa590 2024-04-25T11:34:59.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.769+0000 7f08cdd9b640 1 --2- 172.21.15.90:0/2067763589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1aa590 0x5650ed575a00 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f08c4013a30 tx=0x7f08c400c5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fa1284cbc357fc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.769+0000 7f08bf7fe640 1 -- 172.21.15.90:0/4109082398 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f08c00df050 con 0x7f08d0108930 2024-04-25T11:34:59.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.769+0000 7f08bcff9640 1 -- 172.21.15.90:0/2067763589 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c4004220 con 0x5650ed1aa590 2024-04-25T11:34:59.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.769+0000 7f08bcff9640 1 -- 172.21.15.90:0/2067763589 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08c400ca60 con 0x5650ed1aa590 2024-04-25T11:34:59.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.769+0000 7f08bcff9640 1 -- 172.21.15.90:0/2067763589 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c4003a10 con 0x5650ed1aa590 2024-04-25T11:34:59.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.770+0000 7f08dbbcd740 1 -- 172.21.15.90:0/2067763589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1aa590 msgr2=0x5650ed575a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.770+0000 7f08ced9d640 1 --2- 172.21.15.90:0/4109082398 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f08b80afaf0 0x7f08b80b2000 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f08c00a7210 tx=0x7f08c00d5c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.770+0000 7f08dbbcd740 1 --2- 172.21.15.90:0/2067763589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1aa590 0x5650ed575a00 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f08c4013a30 tx=0x7f08c400c5a0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.770+0000 7f08dbbcd740 1 -- 172.21.15.90:0/2067763589 shutdown_connections 2024-04-25T11:34:59.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.770+0000 7f08dbbcd740 1 --2- 172.21.15.90:0/2067763589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1aa590 0x5650ed575a00 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.770+0000 7f08dbbcd740 1 --2- 172.21.15.90:0/2067763589 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5650ed1a23b0 0x5650ed56eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.770+0000 7f08dbbcd740 1 --2- 172.21.15.90:0/2067763589 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5650ed1cb270 0x5650ed56e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.770+0000 7f08dbbcd740 1 -- 172.21.15.90:0/2067763589 >> 172.21.15.90:0/2067763589 conn(0x5650ed1bf3c0 msgr2=0x5650ed567d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.770+0000 7f08dbbcd740 1 -- 172.21.15.90:0/2067763589 shutdown_connections 2024-04-25T11:34:59.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.770+0000 7f08dbbcd740 1 -- 172.21.15.90:0/2067763589 wait complete. 2024-04-25T11:34:59.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.770+0000 7f08dbbcd740 1 -- start start 2024-04-25T11:34:59.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.770+0000 7f08dbbcd740 1 Processor -- start 2024-04-25T11:34:59.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.770+0000 7f08dbbcd740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5650ed1a23b0 0x5650ed578450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.771+0000 7f08dbbcd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1aa590 0x5650ed578990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.771+0000 7f08dbbcd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5650ed1cb270 0x5650ed57f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.771+0000 7f08dbbcd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5650ed576450 con 0x5650ed1aa590 2024-04-25T11:34:59.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.771+0000 7f08dbbcd740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5650ed458fe0 con 0x5650ed1a23b0 2024-04-25T11:34:59.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.771+0000 7f08dbbcd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5650ed5765d0 con 0x5650ed1cb270 2024-04-25T11:34:59.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.771+0000 7f08dbbcd740 10 client.0 _ll_register_callbacks cb 0x5650ed46f150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:34:59.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.771+0000 7f08ce59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1aa590 0x5650ed578990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.771+0000 7f08ce59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1aa590 0x5650ed578990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58052/0 (socket says 172.21.15.90:58052) 2024-04-25T11:34:59.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.771+0000 7f08ce59c640 1 -- 172.21.15.90:0/812377224 learned_addr learned my addr 172.21.15.90:0/812377224 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:59.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.771+0000 7f08cdd9b640 1 --2- 172.21.15.90:0/812377224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5650ed1cb270 0x5650ed57f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.771+0000 7f08cdd9b640 1 --2- 172.21.15.90:0/812377224 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5650ed1a23b0 0x5650ed578450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.771+0000 7f08ce59c640 1 -- 172.21.15.90:0/812377224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5650ed1cb270 msgr2=0x5650ed57f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.771+0000 7f08ce59c640 1 --2- 172.21.15.90:0/812377224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5650ed1cb270 0x5650ed57f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.771+0000 7f08ce59c640 1 -- 172.21.15.90:0/812377224 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5650ed1a23b0 msgr2=0x5650ed578450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.771+0000 7f08ce59c640 1 --2- 172.21.15.90:0/812377224 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5650ed1a23b0 0x5650ed578450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.771+0000 7f08ce59c640 1 -- 172.21.15.90:0/812377224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650ed1e32d0 con 0x5650ed1aa590 2024-04-25T11:34:59.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.771+0000 7f08ce59c640 1 --2- 172.21.15.90:0/812377224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1aa590 0x5650ed578990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f08c8005ab0 tx=0x7f08c80317d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a470f68756f5780 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.771+0000 7f08b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:34:59.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.771+0000 7f08b5ffb640 1 -- 172.21.15.90:0/812377224 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c8035040 con 0x5650ed1aa590 2024-04-25T11:34:59.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.771+0000 7f08b5ffb640 1 -- 172.21.15.90:0/812377224 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f08c8004240 con 0x5650ed1aa590 2024-04-25T11:34:59.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.771+0000 7f08dbbcd740 1 -- 172.21.15.90:0/812377224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08c400ca60 con 0x5650ed1aa590 2024-04-25T11:34:59.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.771+0000 7f08b5ffb640 1 -- 172.21.15.90:0/812377224 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f08c804b070 con 0x5650ed1aa590 2024-04-25T11:34:59.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.771+0000 7f08dbbcd740 10 client.15336 fetch_fsmap learned FSMap version 25 2024-04-25T11:34:59.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.771+0000 7f08dbbcd740 1 -- 172.21.15.90:0/812377224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5650ed2ad6b0 con 0x5650ed1aa590 2024-04-25T11:34:59.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.772+0000 7f08b5ffb640 1 -- 172.21.15.90:0/812377224 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08c400ca60 con 0x5650ed1aa590 2024-04-25T11:34:59.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.772+0000 7f08b5ffb640 1 -- 172.21.15.90:0/812377224 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f08c8004b70 con 0x5650ed1aa590 2024-04-25T11:34:59.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.772+0000 7f08dbbcd740 10 client.15336 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T11:34:59.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.772+0000 7f08dbbcd740 10 client.15336 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T11:34:59.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.772+0000 7f08dbbcd740 20 client.15336 populate_metadata read hostname 'smithi090' 2024-04-25T11:34:59.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.772+0000 7f08dbbcd740 1 --2- 172.21.15.90:0/812377224 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5650ed189e40 0x5650ed586230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.772+0000 7f08dbbcd740 4 client.15336 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:34:59.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.772+0000 7f08dbbcd740 1 -- 172.21.15.90:0/812377224 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5650ed4df350 con 0x5650ed189e40 2024-04-25T11:34:59.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.772+0000 7f08cdd9b640 1 --2- 172.21.15.90:0/812377224 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5650ed189e40 0x5650ed586230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.773+0000 7f08cdd9b640 1 --2- 172.21.15.90:0/812377224 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5650ed189e40 0x5650ed586230 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28784952c04b1628 server_cookie=4be1f6b1108f3699 in_seq=0 out_seq=0 2024-04-25T11:34:59.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.773+0000 7f08b5ffb640 10 client.15336 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:34:59.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.773+0000 7f08b5ffb640 1 -- 172.21.15.90:0/812377224 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5650ed4df350 con 0x5650ed189e40 2024-04-25T11:34:59.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.773+0000 7f08b5ffb640 10 client.15336 handle_command_reply: tid=0 2024-04-25T11:34:59.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.774+0000 7f08dbbcd740 1 client.15336 shutdown 2024-04-25T11:34:59.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.774+0000 7f08dbbcd740 1 -- 172.21.15.90:0/812377224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1aa590 msgr2=0x5650ed578990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.774+0000 7f08dbbcd740 1 --2- 172.21.15.90:0/812377224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1aa590 0x5650ed578990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f08c8005ab0 tx=0x7f08c80317d0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.774+0000 7f08dbbcd740 1 -- 172.21.15.90:0/812377224 shutdown_connections 2024-04-25T11:34:59.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.774+0000 7f08dbbcd740 1 --2- 172.21.15.90:0/812377224 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5650ed189e40 0x5650ed586230 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.774+0000 7f08dbbcd740 1 --2- 172.21.15.90:0/812377224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5650ed1cb270 0x5650ed57f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.774+0000 7f08dbbcd740 1 --2- 172.21.15.90:0/812377224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1aa590 0x5650ed578990 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.774+0000 7f08b5ffb640 0 client.15336 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:34:59.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.774+0000 7f08dbbcd740 1 --2- 172.21.15.90:0/812377224 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5650ed1a23b0 0x5650ed578450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.774+0000 7f08dbbcd740 1 -- 172.21.15.90:0/812377224 >> 172.21.15.90:0/812377224 conn(0x5650ed1bf3c0 msgr2=0x5650ed566d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.774+0000 7f08dbbcd740 1 -- 172.21.15.90:0/812377224 shutdown_connections 2024-04-25T11:34:59.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.774+0000 7f08dbbcd740 1 -- 172.21.15.90:0/812377224 wait complete. 2024-04-25T11:34:59.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.774+0000 7f08dbbcd740 20 client.15336 trim_cache size 0 max 16384 2024-04-25T11:34:59.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.793+0000 7f08dbbcd740 1 Processor -- start 2024-04-25T11:34:59.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.793+0000 7f08dbbcd740 1 -- start start 2024-04-25T11:34:59.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.793+0000 7f08dbbcd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5650ed1a23b0 0x5650ed571b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.793+0000 7f08dbbcd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1aa590 0x5650ed57f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.793+0000 7f08dbbcd740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5650ed1cb270 0x5650ed577700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.793+0000 7f08dbbcd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5650ed584570 con 0x5650ed1aa590 2024-04-25T11:34:59.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.793+0000 7f08dbbcd740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5650ed584c40 con 0x5650ed1cb270 2024-04-25T11:34:59.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.793+0000 7f08dbbcd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5650ed585310 con 0x5650ed1a23b0 2024-04-25T11:34:59.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.793+0000 7f08ce59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1aa590 0x5650ed57f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.793+0000 7f08ce59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1aa590 0x5650ed57f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58054/0 (socket says 172.21.15.90:58054) 2024-04-25T11:34:59.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.793+0000 7f08ce59c640 1 -- 172.21.15.90:0/3282157022 learned_addr learned my addr 172.21.15.90:0/3282157022 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:59.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.793+0000 7f08cdd9b640 1 --2- 172.21.15.90:0/3282157022 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5650ed1cb270 0x5650ed577700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.793+0000 7f08cdd9b640 1 --2- 172.21.15.90:0/3282157022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5650ed1a23b0 0x5650ed571b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.793+0000 7f08ce59c640 1 -- 172.21.15.90:0/3282157022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5650ed1a23b0 msgr2=0x5650ed571b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.793+0000 7f08ce59c640 1 --2- 172.21.15.90:0/3282157022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5650ed1a23b0 0x5650ed571b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.793+0000 7f08ce59c640 1 -- 172.21.15.90:0/3282157022 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5650ed1cb270 msgr2=0x5650ed577700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.793+0000 7f08ce59c640 1 --2- 172.21.15.90:0/3282157022 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5650ed1cb270 0x5650ed577700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.793+0000 7f08ce59c640 1 -- 172.21.15.90:0/3282157022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650ed5732a0 con 0x5650ed1aa590 2024-04-25T11:34:59.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.793+0000 7f08ce59c640 1 --2- 172.21.15.90:0/3282157022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1aa590 0x5650ed57f810 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f08c80058b0 tx=0x7f08c8001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98f5d2c201ea9a85 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.794+0000 7f08b7fff640 1 -- 172.21.15.90:0/3282157022 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c8035070 con 0x5650ed1aa590 2024-04-25T11:34:59.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.794+0000 7f08b7fff640 1 -- 172.21.15.90:0/3282157022 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08c8054020 con 0x5650ed1aa590 2024-04-25T11:34:59.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.794+0000 7f08b7fff640 1 -- 172.21.15.90:0/3282157022 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c8004c40 con 0x5650ed1aa590 2024-04-25T11:34:59.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.794+0000 7f08dbbcd740 1 -- 172.21.15.90:0/3282157022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1aa590 msgr2=0x5650ed57f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.794+0000 7f08dbbcd740 1 --2- 172.21.15.90:0/3282157022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1aa590 0x5650ed57f810 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f08c80058b0 tx=0x7f08c8001e50 comp rx=0 tx=0).stop 2024-04-25T11:34:59.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.794+0000 7f08dbbcd740 1 -- 172.21.15.90:0/3282157022 shutdown_connections 2024-04-25T11:34:59.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.794+0000 7f08dbbcd740 1 --2- 172.21.15.90:0/3282157022 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5650ed1cb270 0x5650ed577700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.794+0000 7f08dbbcd740 1 --2- 172.21.15.90:0/3282157022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1aa590 0x5650ed57f810 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.794+0000 7f08dbbcd740 1 --2- 172.21.15.90:0/3282157022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5650ed1a23b0 0x5650ed571b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.794+0000 7f08dbbcd740 1 -- 172.21.15.90:0/3282157022 >> 172.21.15.90:0/3282157022 conn(0x5650ed1bf3c0 msgr2=0x5650ed576a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.794+0000 7f08dbbcd740 1 -- 172.21.15.90:0/3282157022 shutdown_connections 2024-04-25T11:34:59.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.795+0000 7f08dbbcd740 1 -- 172.21.15.90:0/3282157022 wait complete. 2024-04-25T11:34:59.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.795+0000 7f08dbbcd740 1 -- start start 2024-04-25T11:34:59.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.795+0000 7f08dbbcd740 1 Processor -- start 2024-04-25T11:34:59.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.795+0000 7f08dbbcd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1a23b0 0x5650ed57ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.795+0000 7f08dbbcd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5650ed1aa590 0x5650ed57b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.795+0000 7f08dbbcd740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5650ed1cb270 0x5650ed57e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.795+0000 7f08dbbcd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5650ed5860b0 con 0x5650ed1a23b0 2024-04-25T11:34:59.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.795+0000 7f08dbbcd740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5650ed5859e0 con 0x5650ed1cb270 2024-04-25T11:34:59.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.795+0000 7f08dbbcd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5650ed586780 con 0x5650ed1aa590 2024-04-25T11:34:59.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.795+0000 7f08dbbcd740 10 client.0 _ll_register_callbacks cb 0x7f08c8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:34:59.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.795+0000 7f08cdd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1a23b0 0x5650ed57ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.795+0000 7f08cdd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1a23b0 0x5650ed57ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58062/0 (socket says 172.21.15.90:58062) 2024-04-25T11:34:59.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.795+0000 7f08cdd9b640 1 -- 172.21.15.90:0/799688032 learned_addr learned my addr 172.21.15.90:0/799688032 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:59.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.795+0000 7f08ce59c640 1 --2- 172.21.15.90:0/799688032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5650ed1aa590 0x5650ed57b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.795+0000 7f08cdd9b640 1 --2- 172.21.15.90:0/799688032 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5650ed1cb270 0x5650ed57e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.795+0000 7f08cdd9b640 1 -- 172.21.15.90:0/799688032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5650ed1aa590 msgr2=0x5650ed57b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.795+0000 7f08cdd9b640 1 --2- 172.21.15.90:0/799688032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5650ed1aa590 0x5650ed57b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.795+0000 7f08cdd9b640 1 -- 172.21.15.90:0/799688032 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5650ed1cb270 msgr2=0x5650ed57e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:59.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.795+0000 7f08cdd9b640 1 --2- 172.21.15.90:0/799688032 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5650ed1cb270 0x5650ed57e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.795+0000 7f08cdd9b640 1 -- 172.21.15.90:0/799688032 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650ed57e6d0 con 0x5650ed1a23b0 2024-04-25T11:34:59.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.796+0000 7f08cdd9b640 1 --2- 172.21.15.90:0/799688032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1a23b0 0x5650ed57ac90 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f08c400f730 tx=0x7f08c8032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f3bdf124a76ad31 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.796+0000 7f08b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:34:59.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.796+0000 7f08b5ffb640 1 -- 172.21.15.90:0/799688032 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c40040f0 con 0x5650ed1a23b0 2024-04-25T11:34:59.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.796+0000 7f08b5ffb640 1 -- 172.21.15.90:0/799688032 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f08c4004290 con 0x5650ed1a23b0 2024-04-25T11:34:59.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.796+0000 7f08dbbcd740 1 -- 172.21.15.90:0/799688032 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08c8054020 con 0x5650ed1a23b0 2024-04-25T11:34:59.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.796+0000 7f08b5ffb640 1 -- 172.21.15.90:0/799688032 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f08c40044b0 con 0x5650ed1a23b0 2024-04-25T11:34:59.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.796+0000 7f08dbbcd740 10 client.15342 fetch_fsmap learned FSMap version 25 2024-04-25T11:34:59.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.796+0000 7f08dbbcd740 1 -- 172.21.15.90:0/799688032 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5650ed57f950 con 0x5650ed1a23b0 2024-04-25T11:34:59.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.796+0000 7f08b5ffb640 1 -- 172.21.15.90:0/799688032 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08c8054020 con 0x5650ed1a23b0 2024-04-25T11:34:59.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.796+0000 7f08b5ffb640 1 -- 172.21.15.90:0/799688032 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f08c401d020 con 0x5650ed1a23b0 2024-04-25T11:34:59.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.796+0000 7f08dbbcd740 10 client.15342 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T11:34:59.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.796+0000 7f08dbbcd740 10 client.15342 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T11:34:59.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.796+0000 7f08dbbcd740 20 client.15342 populate_metadata read hostname 'smithi090' 2024-04-25T11:34:59.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.796+0000 7f08dbbcd740 1 --2- 172.21.15.90:0/799688032 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5650ed15f280 0x5650ed5f22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.796+0000 7f08dbbcd740 4 client.15342 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T11:34:59.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.796+0000 7f08dbbcd740 1 -- 172.21.15.90:0/799688032 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "status"}) -- 0x5650ed4d91c0 con 0x5650ed15f280 2024-04-25T11:34:59.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.797+0000 7f08ce59c640 1 --2- 172.21.15.90:0/799688032 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5650ed15f280 0x5650ed5f22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.797+0000 7f08b5ffb640 10 client.15342 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:34:59.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.797+0000 7f08ce59c640 1 --2- 172.21.15.90:0/799688032 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5650ed15f280 0x5650ed5f22f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a7d05eda22da2a server_cookie=5795be45f1a832ac in_seq=0 out_seq=0 2024-04-25T11:34:59.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.797+0000 7f08b5ffb640 1 -- 172.21.15.90:0/799688032 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5650ed4d91c0 con 0x5650ed15f280 2024-04-25T11:34:59.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.797+0000 7f08b5ffb640 10 client.15342 handle_command_reply: tid=0 2024-04-25T11:34:59.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.797+0000 7f08dbbcd740 1 client.15342 shutdown 2024-04-25T11:34:59.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.797+0000 7f08dbbcd740 1 -- 172.21.15.90:0/799688032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1a23b0 msgr2=0x5650ed57ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.797+0000 7f08dbbcd740 1 --2- 172.21.15.90:0/799688032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1a23b0 0x5650ed57ac90 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f08c400f730 tx=0x7f08c8032f90 comp rx=0 tx=0).stop 2024-04-25T11:34:59.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.797+0000 7f08dbbcd740 1 -- 172.21.15.90:0/799688032 shutdown_connections 2024-04-25T11:34:59.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.797+0000 7f08dbbcd740 1 --2- 172.21.15.90:0/799688032 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5650ed15f280 0x5650ed5f22f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.797+0000 7f08dbbcd740 1 --2- 172.21.15.90:0/799688032 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5650ed1cb270 0x5650ed57e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.797+0000 7f08dbbcd740 1 --2- 172.21.15.90:0/799688032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5650ed1aa590 0x5650ed57b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.797+0000 7f08b5ffb640 0 client.15342 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:34:59.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.797+0000 7f08dbbcd740 1 --2- 172.21.15.90:0/799688032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650ed1a23b0 0x5650ed57ac90 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.797+0000 7f08dbbcd740 1 -- 172.21.15.90:0/799688032 >> 172.21.15.90:0/799688032 conn(0x5650ed1bf3c0 msgr2=0x5650ed571970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.797+0000 7f08dbbcd740 1 -- 172.21.15.90:0/799688032 shutdown_connections 2024-04-25T11:34:59.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.797+0000 7f08dbbcd740 1 -- 172.21.15.90:0/799688032 wait complete. 2024-04-25T11:34:59.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.798+0000 7f08dbbcd740 20 client.15342 trim_cache size 0 max 16384 2024-04-25T11:34:59.796 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:34:59.796 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "58bd4e7a-02f6-11ef-bc93-c7b262605968", 2024-04-25T11:34:59.796 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 0, 2024-04-25T11:34:59.796 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 24524, 2024-04-25T11:34:59.796 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-04-25T11:34:59.796 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-04-25T11:34:59.796 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-04-25T11:34:59.797 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 108.782187605, 2024-04-25T11:34:59.797 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 25, 2024-04-25T11:34:59.797 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 79, 2024-04-25T11:34:59.797 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 77, 2024-04-25T11:34:59.797 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 114.35104703499999 2024-04-25T11:34:59.797 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:34:59.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.798+0000 7f08d4c40640 1 -- 172.21.15.90:0/4109082398 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f08b80afaf0 msgr2=0x7f08b80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.798+0000 7f08d4c40640 1 --2- 172.21.15.90:0/4109082398 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f08b80afaf0 0x7f08b80b2000 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f08c00a7210 tx=0x7f08c00d5c60 comp rx=0 tx=0).stop 2024-04-25T11:34:59.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.798+0000 7f08d4c40640 1 -- 172.21.15.90:0/4109082398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08d0108930 msgr2=0x7f08d01a0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.798+0000 7f08d4c40640 1 --2- 172.21.15.90:0/4109082398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08d0108930 0x7f08d01a0620 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f08c000cce0 tx=0x7f08c000a620 comp rx=0 tx=0).stop 2024-04-25T11:34:59.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.798+0000 7f08d4c40640 1 -- 172.21.15.90:0/4109082398 shutdown_connections 2024-04-25T11:34:59.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.798+0000 7f08d4c40640 1 --2- 172.21.15.90:0/4109082398 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f08b80afaf0 0x7f08b80b2000 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.798+0000 7f08d4c40640 1 --2- 172.21.15.90:0/4109082398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08d0108930 0x7f08d01a0620 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.798+0000 7f08d4c40640 1 --2- 172.21.15.90:0/4109082398 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f08d0103c30 0x7f08d0199700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.798+0000 7f08d4c40640 1 --2- 172.21.15.90:0/4109082398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08d01029f0 0x7f08d01991c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.798+0000 7f08d4c40640 1 -- 172.21.15.90:0/4109082398 >> 172.21.15.90:0/4109082398 conn(0x7f08d00fdbc0 msgr2=0x7f08d0104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.798+0000 7f08d4c40640 1 -- 172.21.15.90:0/4109082398 shutdown_connections 2024-04-25T11:34:59.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.798+0000 7f08d4c40640 1 -- 172.21.15.90:0/4109082398 wait complete. 2024-04-25T11:34:59.800 INFO:teuthology.run_tasks:Running task sequential... 2024-04-25T11:34:59.813 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-25T11:34:59.813 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-25T11:34:59.813 INFO:teuthology.task.print:Enabling mgr modules 2024-04-25T11:34:59.813 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-25T11:34:59.814 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T11:34:59.814 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T11:34:59.814 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-25T11:34:59.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.933+0000 7f8add5eb640 1 -- 172.21.15.90:0/2309153711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ad8103c80 msgr2=0x7f8ad8104090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.933+0000 7f8add5eb640 1 --2- 172.21.15.90:0/2309153711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ad8103c80 0x7f8ad8104090 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f8acc009900 tx=0x7f8acc02a410 comp rx=0 tx=0).stop 2024-04-25T11:34:59.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.933+0000 7f8add5eb640 1 -- 172.21.15.90:0/2309153711 shutdown_connections 2024-04-25T11:34:59.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.933+0000 7f8add5eb640 1 --2- 172.21.15.90:0/2309153711 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ad8108980 0x7f8ad810ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.933+0000 7f8add5eb640 1 --2- 172.21.15.90:0/2309153711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ad8103c80 0x7f8ad8104090 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.933+0000 7f8add5eb640 1 --2- 172.21.15.90:0/2309153711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ad8102a40 0x7f8ad8103740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.933+0000 7f8add5eb640 1 -- 172.21.15.90:0/2309153711 >> 172.21.15.90:0/2309153711 conn(0x7f8ad80fdc30 msgr2=0x7f8ad81000e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:59.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.933+0000 7f8add5eb640 1 -- 172.21.15.90:0/2309153711 shutdown_connections 2024-04-25T11:34:59.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.934+0000 7f8add5eb640 1 -- 172.21.15.90:0/2309153711 wait complete. 2024-04-25T11:34:59.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.934+0000 7f8add5eb640 1 Processor -- start 2024-04-25T11:34:59.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.934+0000 7f8add5eb640 1 -- start start 2024-04-25T11:34:59.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.934+0000 7f8add5eb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ad8102a40 0x7f8ad806bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.934+0000 7f8add5eb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ad8103c80 0x7f8ad819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.934+0000 7f8add5eb640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ad8108980 0x7f8ad81a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.934+0000 7f8add5eb640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad810e5c0 con 0x7f8ad8102a40 2024-04-25T11:34:59.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.934+0000 7f8add5eb640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad810e440 con 0x7f8ad8108980 2024-04-25T11:34:59.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.934+0000 7f8add5eb640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad810e740 con 0x7f8ad8103c80 2024-04-25T11:34:59.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.935+0000 7f8ad6ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ad8102a40 0x7f8ad806bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.935+0000 7f8ad6ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ad8102a40 0x7f8ad806bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58076/0 (socket says 172.21.15.90:58076) 2024-04-25T11:34:59.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.935+0000 7f8ad6ffd640 1 -- 172.21.15.90:0/567751108 learned_addr learned my addr 172.21.15.90:0/567751108 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:34:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.935+0000 7f8ad67fc640 1 --2- 172.21.15.90:0/567751108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ad8103c80 0x7f8ad819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.935+0000 7f8ad77fe640 1 --2- 172.21.15.90:0/567751108 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ad8108980 0x7f8ad81a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.935+0000 7f8ad6ffd640 1 -- 172.21.15.90:0/567751108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ad8103c80 msgr2=0x7f8ad819f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.935+0000 7f8ad6ffd640 1 --2- 172.21.15.90:0/567751108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ad8103c80 0x7f8ad819f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.935+0000 7f8ad6ffd640 1 -- 172.21.15.90:0/567751108 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ad8108980 msgr2=0x7f8ad81a15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.935+0000 7f8ad6ffd640 1 --2- 172.21.15.90:0/567751108 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ad8108980 0x7f8ad81a15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.935+0000 7f8ad6ffd640 1 -- 172.21.15.90:0/567751108 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad81a1d60 con 0x7f8ad8102a40 2024-04-25T11:34:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.935+0000 7f8ad67fc640 1 --2- 172.21.15.90:0/567751108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ad8103c80 0x7f8ad819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:34:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.935+0000 7f8ad77fe640 1 --2- 172.21.15.90:0/567751108 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ad8108980 0x7f8ad81a15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:34:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.935+0000 7f8ad6ffd640 1 --2- 172.21.15.90:0/567751108 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ad8102a40 0x7f8ad806bd40 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8ac800b990 tx=0x7f8ac800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.935+0000 7f8abffff640 1 -- 172.21.15.90:0/567751108 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac8019070 con 0x7f8ad8102a40 2024-04-25T11:34:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.935+0000 7f8add5eb640 1 -- 172.21.15.90:0/567751108 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ad81a1ff0 con 0x7f8ad8102a40 2024-04-25T11:34:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.935+0000 7f8abffff640 1 -- 172.21.15.90:0/567751108 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ac800cba0 con 0x7f8ad8102a40 2024-04-25T11:34:59.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.935+0000 7f8abffff640 1 -- 172.21.15.90:0/567751108 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac80125a0 con 0x7f8ad8102a40 2024-04-25T11:34:59.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.935+0000 7f8add5eb640 1 -- 172.21.15.90:0/567751108 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ad81a2530 con 0x7f8ad8102a40 2024-04-25T11:34:59.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.936+0000 7f8add5eb640 1 -- 172.21.15.90:0/567751108 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ad8107790 con 0x7f8ad8102a40 2024-04-25T11:34:59.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.938+0000 7f8abffff640 1 -- 172.21.15.90:0/567751108 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f8ac801c070 con 0x7f8ad8102a40 2024-04-25T11:34:59.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.939+0000 7f8abffff640 1 --2- 172.21.15.90:0/567751108 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f8ac00afae0 0x7f8ac00b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:59.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.939+0000 7f8ad67fc640 1 --2- 172.21.15.90:0/567751108 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f8ac00afae0 0x7f8ac00b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:59.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.939+0000 7f8abffff640 1 -- 172.21.15.90:0/567751108 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f8ac80de6d0 con 0x7f8ad8102a40 2024-04-25T11:34:59.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.940+0000 7f8ad67fc640 1 --2- 172.21.15.90:0/567751108 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f8ac00afae0 0x7f8ac00b1ff0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8acc000c00 tx=0x7f8acc031d10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:59.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:34:59.943+0000 7f8abffff640 1 -- 172.21.15.90:0/567751108 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8ac80a7290 con 0x7f8ad8102a40 2024-04-25T11:34:59.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:34:59 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/342937216' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:35:00.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:00.168+0000 7f8add5eb640 1 -- 172.21.15.90:0/567751108 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f8ad802dda0 con 0x7f8ad8102a40 2024-04-25T11:35:00.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:34:59 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/342937216' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:35:00.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:34:59 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/342937216' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:35:00.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:00.892+0000 7f8abffff640 1 -- 172.21.15.90:0/567751108 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f8ac80af310 con 0x7f8ad8102a40 2024-04-25T11:35:00.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:00.895+0000 7f8add5eb640 1 -- 172.21.15.90:0/567751108 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f8ac00afae0 msgr2=0x7f8ac00b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:00.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:00.895+0000 7f8add5eb640 1 --2- 172.21.15.90:0/567751108 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f8ac00afae0 0x7f8ac00b1ff0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8acc000c00 tx=0x7f8acc031d10 comp rx=0 tx=0).stop 2024-04-25T11:35:00.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:00.895+0000 7f8add5eb640 1 -- 172.21.15.90:0/567751108 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ad8102a40 msgr2=0x7f8ad806bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:00.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:00.895+0000 7f8add5eb640 1 --2- 172.21.15.90:0/567751108 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ad8102a40 0x7f8ad806bd40 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8ac800b990 tx=0x7f8ac800be50 comp rx=0 tx=0).stop 2024-04-25T11:35:00.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:00.895+0000 7f8add5eb640 1 -- 172.21.15.90:0/567751108 shutdown_connections 2024-04-25T11:35:00.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:00.895+0000 7f8add5eb640 1 --2- 172.21.15.90:0/567751108 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ad8108980 0x7f8ad81a15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:00.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:00.895+0000 7f8add5eb640 1 --2- 172.21.15.90:0/567751108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ad8103c80 0x7f8ad819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:00.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:00.895+0000 7f8add5eb640 1 --2- 172.21.15.90:0/567751108 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f8ac00afae0 0x7f8ac00b1ff0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:00.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:00.895+0000 7f8add5eb640 1 --2- 172.21.15.90:0/567751108 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ad8102a40 0x7f8ad806bd40 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:00.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:00.895+0000 7f8add5eb640 1 -- 172.21.15.90:0/567751108 >> 172.21.15.90:0/567751108 conn(0x7f8ad80fdc30 msgr2=0x7f8ad8104960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:00.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:00.895+0000 7f8add5eb640 1 -- 172.21.15.90:0/567751108 shutdown_connections 2024-04-25T11:35:00.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:00.895+0000 7f8add5eb640 1 -- 172.21.15.90:0/567751108 wait complete. 2024-04-25T11:35:00.905 INFO:teuthology.run_tasks:Running task kclient... 2024-04-25T11:35:00.928 INFO:tasks.kclient:Mounting kernel clients... 2024-04-25T11:35:00.929 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-25T11:35:00.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:35:00.929 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-04-25T11:35:00.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:35:00.946 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link delete ceph-brx 2024-04-25T11:35:01.012 INFO:teuthology.orchestra.run.smithi090.stderr:Cannot find device "ceph-brx" 2024-04-25T11:35:01.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T11:35:01.014 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-04-25T11:35:01.014 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T11:35:01.014 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T11:35:01.014 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-25T11:35:01.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:00 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/567751108' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T11:35:01.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:00 smithi090 ceph-mon[29575]: pgmap v331: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-04-25T11:35:01.226 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:00 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: ignoring --setuser ceph since I am not root 2024-04-25T11:35:01.226 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:00 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: ignoring --setgroup ceph since I am not root 2024-04-25T11:35:01.226 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:01 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:01.111+0000 7f440e686200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T11:35:01.270 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:00 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/567751108' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T11:35:01.270 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:00 smithi138 ceph-mon[32328]: pgmap v331: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-04-25T11:35:01.271 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:00 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: ignoring --setuser ceph since I am not root 2024-04-25T11:35:01.271 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:00 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: ignoring --setgroup ceph since I am not root 2024-04-25T11:35:01.271 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:00 smithi138 ceph-mgr[33707]: -- 172.21.15.138:0/3483415485 <== mon.2 v2:172.21.15.138:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5627926765a0 con 0x562797400480 2024-04-25T11:35:01.271 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:01 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:35:01.094+0000 7ff3b55c3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T11:35:01.271 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:01 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:35:01.282+0000 7ff3b55c3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T11:35:01.275 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:00 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/567751108' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T11:35:01.276 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:00 smithi156 ceph-mon[33045]: pgmap v331: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-04-25T11:35:01.276 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:00 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: ignoring --setuser ceph since I am not root 2024-04-25T11:35:01.276 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:00 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: ignoring --setgroup ceph since I am not root 2024-04-25T11:35:01.276 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:00 smithi156 ceph-mgr[34920]: -- 172.21.15.156:0/2078487533 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x565426aa85a0 con 0x56542b832000 2024-04-25T11:35:01.276 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:01 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:35:01.100+0000 7f8a53947200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T11:35:01.276 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:01 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:35:01.287+0000 7f8a53947200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T11:35:01.305 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:35:01.552 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:01 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:35:01.384+0000 7ff3b55c3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T11:35:01.568 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:01 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:01.320+0000 7f440e686200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T11:35:01.568 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:01 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:01.434+0000 7f440e686200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T11:35:01.587 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:01 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:35:01.391+0000 7f8a53947200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T11:35:01.842 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:01 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:01.766+0000 7f440e686200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T11:35:01.842 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:01 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:01.852+0000 7f440e686200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T11:35:02.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:01 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/567751108' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T11:35:02.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:01 smithi138 ceph-mon[32328]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-04-25T11:35:02.052 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:01 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:35:01.692+0000 7ff3b55c3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T11:35:02.052 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:01 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:35:01.777+0000 7ff3b55c3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T11:35:02.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:01 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/567751108' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T11:35:02.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:01 smithi156 ceph-mon[33045]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-04-25T11:35:02.088 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:01 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:35:01.702+0000 7f8a53947200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T11:35:02.088 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:01 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:35:01.788+0000 7f8a53947200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T11:35:02.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:01 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/567751108' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T11:35:02.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:01 smithi090 ceph-mon[29575]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-04-25T11:35:03.132 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:03 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:03.070+0000 7f440e686200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T11:35:03.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.263+0000 7f653dad1700 1 -- 172.21.15.90:0/90442560 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6538108db0 msgr2=0x7f65381091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:03.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.263+0000 7f653dad1700 1 --2- 172.21.15.90:0/90442560 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6538108db0 0x7f65381091c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f65280099c0 tx=0x7f6528009cd0 comp rx=0 tx=0).stop 2024-04-25T11:35:03.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.263+0000 7f653dad1700 1 -- 172.21.15.90:0/90442560 shutdown_connections 2024-04-25T11:35:03.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.263+0000 7f653dad1700 1 --2- 172.21.15.90:0/90442560 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6538109700 0x7f6538111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:03.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.263+0000 7f653dad1700 1 --2- 172.21.15.90:0/90442560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6538104480 0x7f6538104890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:03.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.263+0000 7f653dad1700 1 --2- 172.21.15.90:0/90442560 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6538108db0 0x7f65381091c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:03.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.263+0000 7f653dad1700 1 -- 172.21.15.90:0/90442560 >> 172.21.15.90:0/90442560 conn(0x7f65380ffd70 msgr2=0x7f65381021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:03.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.263+0000 7f653dad1700 1 -- 172.21.15.90:0/90442560 shutdown_connections 2024-04-25T11:35:03.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.263+0000 7f653dad1700 1 -- 172.21.15.90:0/90442560 wait complete. 2024-04-25T11:35:03.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.264+0000 7f653dad1700 1 Processor -- start 2024-04-25T11:35:03.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.264+0000 7f653dad1700 1 -- start start 2024-04-25T11:35:03.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.265+0000 7f653dad1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6538104480 0x7f65381a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:03.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.265+0000 7f653dad1700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6538108db0 0x7f65381a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:03.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.265+0000 7f653dad1700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6538109700 0x7f653819aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:03.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.265+0000 7f653dad1700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6538114310 con 0x7f6538104480 2024-04-25T11:35:03.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.265+0000 7f653dad1700 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6538114190 con 0x7f6538108db0 2024-04-25T11:35:03.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.265+0000 7f653dad1700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6538114490 con 0x7f6538109700 2024-04-25T11:35:03.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.265+0000 7f65377fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6538104480 0x7f65381a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:03.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.265+0000 7f6536ffd700 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6538108db0 0x7f65381a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:03.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.265+0000 7f65377fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6538104480 0x7f65381a0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34052/0 (socket says 172.21.15.90:34052) 2024-04-25T11:35:03.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.265+0000 7f65377fe700 1 -- 172.21.15.90:0/4009102249 learned_addr learned my addr 172.21.15.90:0/4009102249 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:03.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.265+0000 7f6537fff700 1 --2- 172.21.15.90:0/4009102249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6538109700 0x7f653819aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:03.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.265+0000 7f65377fe700 1 -- 172.21.15.90:0/4009102249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6538109700 msgr2=0x7f653819aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:03.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.265+0000 7f65377fe700 1 --2- 172.21.15.90:0/4009102249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6538109700 0x7f653819aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:03.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.265+0000 7f65377fe700 1 -- 172.21.15.90:0/4009102249 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6538108db0 msgr2=0x7f65381a0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:03.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.265+0000 7f65377fe700 1 --2- 172.21.15.90:0/4009102249 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6538108db0 0x7f65381a0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:03.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.265+0000 7f65377fe700 1 -- 172.21.15.90:0/4009102249 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f653819b1b0 con 0x7f6538104480 2024-04-25T11:35:03.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.265+0000 7f6536ffd700 1 --2- 172.21.15.90:0/4009102249 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6538108db0 0x7f65381a0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:35:03.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.265+0000 7f6537fff700 1 --2- 172.21.15.90:0/4009102249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6538109700 0x7f653819aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:35:03.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.266+0000 7f65377fe700 1 --2- 172.21.15.90:0/4009102249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6538104480 0x7f65381a0890 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f652c00e9a0 tx=0x7f652c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:03.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.266+0000 7f6534ff9700 1 -- 172.21.15.90:0/4009102249 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652c00ca10 con 0x7f6538104480 2024-04-25T11:35:03.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.266+0000 7f6534ff9700 1 -- 172.21.15.90:0/4009102249 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f652c00cbb0 con 0x7f6538104480 2024-04-25T11:35:03.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.266+0000 7f6534ff9700 1 -- 172.21.15.90:0/4009102249 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652c018680 con 0x7f6538104480 2024-04-25T11:35:03.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.266+0000 7f653dad1700 1 -- 172.21.15.90:0/4009102249 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f653819b4a0 con 0x7f6538104480 2024-04-25T11:35:03.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.266+0000 7f653dad1700 1 -- 172.21.15.90:0/4009102249 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65381a7740 con 0x7f6538104480 2024-04-25T11:35:03.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.267+0000 7f653dad1700 1 -- 172.21.15.90:0/4009102249 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6538105930 con 0x7f6538104480 2024-04-25T11:35:03.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.268+0000 7f6534ff9700 1 -- 172.21.15.90:0/4009102249 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) ==== 147537+0+0 (secure 0 0 0) 0x7f652c010430 con 0x7f6538104480 2024-04-25T11:35:03.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.269+0000 7f6534ff9700 1 --2- 172.21.15.90:0/4009102249 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f65200afbd0 0x7f65200b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:03.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.269+0000 7f6536ffd700 1 -- 172.21.15.90:0/4009102249 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f65200afbd0 msgr2=0x7f65200b20e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/219449516 2024-04-25T11:35:03.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.270+0000 7f6534ff9700 1 -- 172.21.15.90:0/4009102249 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f652c0df1e0 con 0x7f6538104480 2024-04-25T11:35:03.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.270+0000 7f6536ffd700 1 --2- 172.21.15.90:0/4009102249 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f65200afbd0 0x7f65200b20e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T11:35:03.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.273+0000 7f6534ff9700 1 -- 172.21.15.90:0/4009102249 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f652c0a7d40 con 0x7f6538104480 2024-04-25T11:35:03.301 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:02 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:35:02.969+0000 7ff3b55c3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T11:35:03.302 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:03 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:35:03.142+0000 7ff3b55c3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T11:35:03.323 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:02 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:35:02.987+0000 7f8a53947200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T11:35:03.323 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:03 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:35:03.160+0000 7f8a53947200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T11:35:03.323 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:03 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:35:03.335+0000 7f8a53947200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T11:35:03.426 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:03 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:03.260+0000 7f440e686200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T11:35:03.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.471+0000 7f6536ffd700 1 -- 172.21.15.90:0/4009102249 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f65200afbd0 msgr2=0x7f65200b20e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/219449516 2024-04-25T11:35:03.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.471+0000 7f6536ffd700 1 --2- 172.21.15.90:0/4009102249 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f65200afbd0 0x7f65200b20e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T11:35:03.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.522+0000 7f653dad1700 1 -- 172.21.15.90:0/4009102249 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f653819bfc0 con 0x7f6538104480 2024-04-25T11:35:03.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.523+0000 7f6534ff9700 1 -- 172.21.15.90:0/4009102249 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7f652c0a7640 con 0x7f6538104480 2024-04-25T11:35:03.512 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T11:35:03.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.526+0000 7f653dad1700 1 -- 172.21.15.90:0/4009102249 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f65200afbd0 msgr2=0x7f65200b20e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:35:03.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.526+0000 7f653dad1700 1 --2- 172.21.15.90:0/4009102249 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f65200afbd0 0x7f65200b20e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:03.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.526+0000 7f653dad1700 1 -- 172.21.15.90:0/4009102249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6538104480 msgr2=0x7f65381a0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:03.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.526+0000 7f653dad1700 1 --2- 172.21.15.90:0/4009102249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6538104480 0x7f65381a0890 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f652c00e9a0 tx=0x7f652c00ecb0 comp rx=0 tx=0).stop 2024-04-25T11:35:03.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.526+0000 7f653dad1700 1 -- 172.21.15.90:0/4009102249 shutdown_connections 2024-04-25T11:35:03.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.526+0000 7f653dad1700 1 --2- 172.21.15.90:0/4009102249 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f65200afbd0 0x7f65200b20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:03.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.526+0000 7f653dad1700 1 --2- 172.21.15.90:0/4009102249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6538109700 0x7f653819aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:03.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.526+0000 7f653dad1700 1 --2- 172.21.15.90:0/4009102249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6538104480 0x7f65381a0890 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:03.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.526+0000 7f653dad1700 1 --2- 172.21.15.90:0/4009102249 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6538108db0 0x7f65381a0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:03.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.526+0000 7f653dad1700 1 -- 172.21.15.90:0/4009102249 >> 172.21.15.90:0/4009102249 conn(0x7f65380ffd70 msgr2=0x7f6538108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:03.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.527+0000 7f653dad1700 1 -- 172.21.15.90:0/4009102249 shutdown_connections 2024-04-25T11:35:03.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:03.527+0000 7f653dad1700 1 -- 172.21.15.90:0/4009102249 wait complete. 2024-04-25T11:35:03.563 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:03 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:35:03.314+0000 7ff3b55c3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T11:35:03.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:03 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/4009102249' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T11:35:03.725 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:03 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/4009102249' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T11:35:03.725 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:03 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:03.437+0000 7f440e686200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T11:35:03.897 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:03 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/4009102249' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T11:35:03.897 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:03 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:35:03.609+0000 7ff3b55c3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T11:35:03.897 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:03 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:35:03.704+0000 7ff3b55c3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T11:35:03.897 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:03 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:35:03.908+0000 7ff3b55c3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T11:35:03.918 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:03 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:35:03.629+0000 7f8a53947200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T11:35:03.918 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:03 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:35:03.725+0000 7f8a53947200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T11:35:03.918 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:03 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:35:03.930+0000 7f8a53947200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T11:35:04.044 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:03 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:03.750+0000 7f440e686200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T11:35:04.044 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:03 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:03.849+0000 7f440e686200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T11:35:04.044 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:04 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:04.055+0000 7f440e686200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T11:35:04.281 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T11:35:04.281 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T11:35:04.281 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-04-25T11:35:04.282 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T11:35:04.282 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T11:35:04.282 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T11:35:04.282 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T11:35:04.282 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T11:35:04.282 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T11:35:04.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:35:04.282 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-04-25T11:35:04.299 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T11:35:04.299 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T11:35:04.299 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T11:35:04.299 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T11:35:04.299 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-04-25T11:35:04.300 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T11:35:04.300 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T11:35:04.300 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-04-25T11:35:04.300 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-04-25T11:35:04.300 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T11:35:04.300 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-04-25T11:35:04.300 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-04-25T11:35:04.300 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T11:35:04.300 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-04-25T11:35:04.300 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T11:35:04.300 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-04-25T11:35:04.300 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T11:35:04.300 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42073sec preferred_lft 42073sec 2024-04-25T11:35:04.301 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::cbf8:23dd:4647:b3ed/64 scope link noprefixroute 2024-04-25T11:35:04.301 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-25T11:35:04.302 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T11:35:04.302 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T11:35:04.302 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-04-25T11:35:04.302 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add name ceph-brx type bridge 2024-04-25T11:35:04.302 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr flush dev ceph-brx 2024-04-25T11:35:04.302 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set ceph-brx up 2024-04-25T11:35:04.302 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T11:35:04.302 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-04-25T11:35:04.303 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:04 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:35:04.104+0000 7ff3b55c3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T11:35:04.337 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:04 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:35:04.126+0000 7f8a53947200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T11:35:04.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T11:35:04.394 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:04 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:04.255+0000 7f440e686200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T11:35:04.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T11:35:04.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:35:04.484 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T11:35:04.507 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-04-25T11:35:04.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:35:04.509 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-04-25T11:35:04.583 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-04-25T11:35:04.583 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T11:35:04.583 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T11:35:04.584 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T11:35:04.584 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T11:35:04.584 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T11:35:04.585 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-04-25T11:35:04.585 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T11:35:04.585 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T11:35:04.585 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T11:35:04.585 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-04-25T11:35:04.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T11:35:04.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T11:35:04.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:35:04.864 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-04-25T11:35:04.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:35:04.879 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list-id 2024-04-25T11:35:04.934 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T11:35:04.934 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-04-25T11:35:04.934 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T11:35:04.934 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-25T11:35:04.934 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-04-25T11:35:04.975 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:04 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:04.899+0000 7f440e686200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T11:35:05.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T11:35:05.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T11:35:05.051 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-25T11:35:05.051 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T11:35:05.051 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-04-25T11:35:05.051 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-25T11:35:05.051 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T11:35:05.051 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-25T11:35:05.051 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-25T11:35:05.052 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-25T11:35:05.052 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-04-25T11:35:05.052 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:04 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:35:04.709+0000 7ff3b55c3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T11:35:05.088 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:04 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:35:04.731+0000 7f8a53947200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T11:35:05.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T11:35:05.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T11:35:05.175 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T11:35:05.175 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-04-25T11:35:05.175 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-04-25T11:35:05.176 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T11:35:05.176 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-04-25T11:35:05.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T11:35:05.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T11:35:05.251 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T11:35:05.251 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T11:35:05.266 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T11:35:05.266 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T11:35:05.266 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T11:35:05.321 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-25T11:35:05.322 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-25T11:35:05.322 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/0dd91eff-2478-4546-a19a-4b4a80251a7f 2024-04-25T11:35:05.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:35:05.323 DEBUG:teuthology.orchestra.run.smithi090:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/0dd91eff-2478-4546-a19a-4b4a80251a7f /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-04-25T11:35:05.423 INFO:teuthology.orchestra.run.smithi090.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-04-25T11:35:05.423 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync". 2024-04-25T11:35:05.423 INFO:teuthology.orchestra.run.smithi090.stdout:invalid new device string format 2024-04-25T11:35:05.425 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:05 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:35:05.138+0000 7ff3b55c3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T11:35:05.425 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:05 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:35:05.234+0000 7ff3b55c3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T11:35:05.452 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:05 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:35:05.161+0000 7f8a53947200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T11:35:05.452 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:05 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:35:05.260+0000 7f8a53947200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T11:35:05.452 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:05 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:35:05.464+0000 7f8a53947200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T11:35:05.537 INFO:teuthology.orchestra.run.smithi090.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-04-25T11:35:05.537 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync". 2024-04-25T11:35:05.538 INFO:teuthology.orchestra.run.smithi090.stdout:invalid new device string format 2024-04-25T11:35:05.538 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: resolved to: "172.21.15.90:3300,172.21.15.138:3300,172.21.15.156:3300" 2024-04-25T11:35:05.538 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.90:3300,172.21.15.138:3300,172.21.15.156:3300:/volumes/qa/sv_1/0dd91eff-2478-4546-a19a-4b4a80251a7f 2024-04-25T11:35:05.538 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync,key=0,fsid=58bd4e7a-02f6-11ef-bc93-c7b262605968" will pass to kernel 2024-04-25T11:35:05.539 INFO:teuthology.orchestra.run.smithi090.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T11:35:05.539 INFO:teuthology.orchestra.run.smithi090.stdout: You just mounted a file system that supports labels which does not 2024-04-25T11:35:05.539 INFO:teuthology.orchestra.run.smithi090.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T11:35:05.539 INFO:teuthology.orchestra.run.smithi090.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T11:35:05.539 INFO:teuthology.orchestra.run.smithi090.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T11:35:05.540 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T11:35:05.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:35:05.541 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T11:35:05.565 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T11:35:05.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T11:35:05.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T11:35:05.662 INFO:teuthology.orchestra.run.smithi090.stdout:client24977 2024-04-25T11:35:05.663 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T11:35:05.663 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T11:35:05.678 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T11:35:05.684 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:05 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:05.364+0000 7f440e686200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T11:35:05.684 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:05 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:05.474+0000 7f440e686200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T11:35:05.684 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:05 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:05.695+0000 7f440e686200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T11:35:05.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T11:35:05.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T11:35:05.740 INFO:teuthology.orchestra.run.smithi090.stdout:client24977 2024-04-25T11:35:05.740 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T11:35:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T11:35:05.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T11:35:05.796 INFO:teuthology.orchestra.run.smithi090.stdout:client24977 2024-04-25T11:35:05.797 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968.client24977/status') 2024-04-25T11:35:05.801 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:05 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:35:05.436+0000 7ff3b55c3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T11:35:05.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T11:35:05.856 INFO:teuthology.orchestra.run.smithi090.stdout:instance: client.24977 (3)172.21.15.90:0/2672494381 2024-04-25T11:35:05.856 INFO:teuthology.orchestra.run.smithi090.stdout:blocklisted: false 2024-04-25T11:35:05.857 INFO:teuthology.orchestra.run.smithi090.stderr:0+1 records in 2024-04-25T11:35:05.857 INFO:teuthology.orchestra.run.smithi090.stderr:0+1 records out 2024-04-25T11:35:05.857 INFO:teuthology.orchestra.run.smithi090.stderr:71 bytes copied, 2.7381e-05 s, 2.6 MB/s 2024-04-25T11:35:05.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T11:35:05.864 INFO:teuthology.run_tasks:Running task exec... 2024-04-25T11:35:05.874 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T11:35:05.874 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T11:35:05.874 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-04-25T11:35:05.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.998+0000 7f72865be640 1 -- 172.21.15.90:0/3750258248 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7280108100 msgr2=0x7f728010a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:05.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.998+0000 7f72865be640 1 --2- 172.21.15.90:0/3750258248 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7280108100 0x7f728010a520 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7278009ca0 tx=0x7f727802a8c0 comp rx=0 tx=0).stop 2024-04-25T11:35:05.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.998+0000 7f72865be640 1 -- 172.21.15.90:0/3750258248 shutdown_connections 2024-04-25T11:35:05.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.998+0000 7f72865be640 1 --2- 172.21.15.90:0/3750258248 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7280108100 0x7f728010a520 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:05.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.998+0000 7f72865be640 1 --2- 172.21.15.90:0/3750258248 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7280103400 0x7f7280103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:05.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.998+0000 7f72865be640 1 --2- 172.21.15.90:0/3750258248 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72801021c0 0x7f7280102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:05.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.998+0000 7f72865be640 1 -- 172.21.15.90:0/3750258248 >> 172.21.15.90:0/3750258248 conn(0x7f72800fd390 msgr2=0x7f72800ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:05.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.998+0000 7f72865be640 1 -- 172.21.15.90:0/3750258248 shutdown_connections 2024-04-25T11:35:05.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.998+0000 7f72865be640 1 -- 172.21.15.90:0/3750258248 wait complete. 2024-04-25T11:35:05.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.999+0000 7f72865be640 1 Processor -- start 2024-04-25T11:35:05.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.999+0000 7f72865be640 1 -- start start 2024-04-25T11:35:05.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.999+0000 7f72865be640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72801021c0 0x7f728010f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:05.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.999+0000 7f72865be640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7280103400 0x7f728019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:05.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.999+0000 7f72865be640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7280108100 0x7f72801a14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:05.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.999+0000 7f72865be640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f728010dce0 con 0x7f72801021c0 2024-04-25T11:35:05.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.999+0000 7f72865be640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f728010db60 con 0x7f7280108100 2024-04-25T11:35:05.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.999+0000 7f72865be640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f728010de60 con 0x7f7280103400 2024-04-25T11:35:05.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.999+0000 7f727ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72801021c0 0x7f728010f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:05.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.999+0000 7f727ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72801021c0 0x7f728010f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34074/0 (socket says 172.21.15.90:34074) 2024-04-25T11:35:05.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.999+0000 7f727ffff640 1 -- 172.21.15.90:0/229832068 learned_addr learned my addr 172.21.15.90:0/229832068 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:05.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.999+0000 7f7284b5b640 1 --2- 172.21.15.90:0/229832068 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7280108100 0x7f72801a14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:05.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.999+0000 7f727f7fe640 1 --2- 172.21.15.90:0/229832068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7280103400 0x7f728019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:05.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.999+0000 7f727ffff640 1 -- 172.21.15.90:0/229832068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7280103400 msgr2=0x7f728019ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:05.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.999+0000 7f727ffff640 1 --2- 172.21.15.90:0/229832068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7280103400 0x7f728019ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:05.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.999+0000 7f727ffff640 1 -- 172.21.15.90:0/229832068 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7280108100 msgr2=0x7f72801a14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:05.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.999+0000 7f727ffff640 1 --2- 172.21.15.90:0/229832068 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7280108100 0x7f72801a14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:05.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:05.999+0000 7f727ffff640 1 -- 172.21.15.90:0/229832068 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72801a1cf0 con 0x7f72801021c0 2024-04-25T11:35:05.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.000+0000 7f727ffff640 1 --2- 172.21.15.90:0/229832068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72801021c0 0x7f728010f100 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f727000e980 tx=0x7f727000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:05.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.000+0000 7f727d7fa640 1 -- 172.21.15.90:0/229832068 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727000cd80 con 0x7f72801021c0 2024-04-25T11:35:05.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.000+0000 7f72865be640 1 -- 172.21.15.90:0/229832068 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72801a1fe0 con 0x7f72801021c0 2024-04-25T11:35:05.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.000+0000 7f727d7fa640 1 -- 172.21.15.90:0/229832068 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72700108e0 con 0x7f72801021c0 2024-04-25T11:35:05.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.000+0000 7f727d7fa640 1 -- 172.21.15.90:0/229832068 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7270018890 con 0x7f72801021c0 2024-04-25T11:35:05.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.000+0000 7f72865be640 1 -- 172.21.15.90:0/229832068 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72801a2440 con 0x7f72801021c0 2024-04-25T11:35:05.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.001+0000 7f72865be640 1 -- 172.21.15.90:0/229832068 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7280050740 con 0x7f72801021c0 2024-04-25T11:35:05.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.002+0000 7f727d7fa640 1 -- 172.21.15.90:0/229832068 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) ==== 147537+0+0 (secure 0 0 0) 0x7f727001f020 con 0x7f72801021c0 2024-04-25T11:35:05.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.003+0000 7f727d7fa640 1 --2- 172.21.15.90:0/229832068 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f72680afb30 0x7f72680b2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:05.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.003+0000 7f727f7fe640 1 -- 172.21.15.90:0/229832068 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f72680afb30 msgr2=0x7f72680b2040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/219449516 2024-04-25T11:35:05.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.003+0000 7f727d7fa640 1 -- 172.21.15.90:0/229832068 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f72700ded30 con 0x7f72801021c0 2024-04-25T11:35:05.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.003+0000 7f727f7fe640 1 --2- 172.21.15.90:0/229832068 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f72680afb30 0x7f72680b2040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T11:35:05.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.008+0000 7f727d7fa640 1 -- 172.21.15.90:0/229832068 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f72700a78f0 con 0x7f72801021c0 2024-04-25T11:35:06.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.204+0000 7f727f7fe640 1 -- 172.21.15.90:0/229832068 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f72680afb30 msgr2=0x7f72680b2040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/219449516 2024-04-25T11:35:06.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.204+0000 7f727f7fe640 1 --2- 172.21.15.90:0/229832068 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f72680afb30 0x7f72680b2040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T11:35:06.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.227+0000 7f72865be640 1 -- 172.21.15.90:0/229832068 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f72800637a0 con 0x7f72801021c0 2024-04-25T11:35:06.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.279+0000 7f727d7fa640 1 -- 172.21.15.90:0/229832068 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v26) ==== 117+0+0 (secure 0 0 0) 0x7f72700a71f0 con 0x7f72801021c0 2024-04-25T11:35:06.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.281+0000 7f72865be640 1 -- 172.21.15.90:0/229832068 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f72680afb30 msgr2=0x7f72680b2040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:35:06.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.281+0000 7f72865be640 1 --2- 172.21.15.90:0/229832068 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f72680afb30 0x7f72680b2040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:06.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.281+0000 7f72865be640 1 -- 172.21.15.90:0/229832068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72801021c0 msgr2=0x7f728010f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:06.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.281+0000 7f72865be640 1 --2- 172.21.15.90:0/229832068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72801021c0 0x7f728010f100 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f727000e980 tx=0x7f727000ee40 comp rx=0 tx=0).stop 2024-04-25T11:35:06.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.281+0000 7f72865be640 1 -- 172.21.15.90:0/229832068 shutdown_connections 2024-04-25T11:35:06.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.281+0000 7f72865be640 1 --2- 172.21.15.90:0/229832068 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7280108100 0x7f72801a14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:06.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.281+0000 7f72865be640 1 --2- 172.21.15.90:0/229832068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7280103400 0x7f728019ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:06.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.281+0000 7f72865be640 1 --2- 172.21.15.90:0/229832068 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f72680afb30 0x7f72680b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:06.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.281+0000 7f72865be640 1 --2- 172.21.15.90:0/229832068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72801021c0 0x7f728010f100 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:06.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.281+0000 7f72865be640 1 -- 172.21.15.90:0/229832068 >> 172.21.15.90:0/229832068 conn(0x7f72800fd390 msgr2=0x7f72800ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:06.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.281+0000 7f72865be640 1 -- 172.21.15.90:0/229832068 shutdown_connections 2024-04-25T11:35:06.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.281+0000 7f72865be640 1 -- 172.21.15.90:0/229832068 wait complete. 2024-04-25T11:35:06.273 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:05 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:35:05.998+0000 7f8a53947200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T11:35:06.273 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:06 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:35:06.094+0000 7f8a53947200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T11:35:06.275 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:05 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:35:05.968+0000 7ff3b55c3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T11:35:06.275 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:06 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:35:06.062+0000 7ff3b55c3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T11:35:06.283 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-25T11:35:06.295 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-25T11:35:06.354 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:35:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.476+0000 7f3861267640 1 -- 172.21.15.90:0/1334335907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f385c102f30 msgr2=0x7f385c103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.476+0000 7f3861267640 1 --2- 172.21.15.90:0/1334335907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f385c102f30 0x7f385c103340 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f3850009900 tx=0x7f385002a410 comp rx=0 tx=0).stop 2024-04-25T11:35:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.476+0000 7f3861267640 1 -- 172.21.15.90:0/1334335907 shutdown_connections 2024-04-25T11:35:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.476+0000 7f3861267640 1 --2- 172.21.15.90:0/1334335907 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f385c104450 0x7f385c10ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.476+0000 7f3861267640 1 --2- 172.21.15.90:0/1334335907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f385c103910 0x7f385c103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.476+0000 7f3861267640 1 --2- 172.21.15.90:0/1334335907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f385c102f30 0x7f385c103340 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:06.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.476+0000 7f3861267640 1 -- 172.21.15.90:0/1334335907 >> 172.21.15.90:0/1334335907 conn(0x7f385c0fd330 msgr2=0x7f385c0ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:06.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.476+0000 7f3861267640 1 -- 172.21.15.90:0/1334335907 shutdown_connections 2024-04-25T11:35:06.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.476+0000 7f3861267640 1 -- 172.21.15.90:0/1334335907 wait complete. 2024-04-25T11:35:06.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.476+0000 7f3861267640 1 Processor -- start 2024-04-25T11:35:06.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.477+0000 7f3861267640 1 -- start start 2024-04-25T11:35:06.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.477+0000 7f3861267640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f385c102f30 0x7f385c19a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:06.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.477+0000 7f3861267640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f385c103910 0x7f385c19a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:06.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.477+0000 7f3861267640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f385c104450 0x7f385c1a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:06.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.477+0000 7f3861267640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f385c10e4c0 con 0x7f385c102f30 2024-04-25T11:35:06.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.477+0000 7f3861267640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f385c10e340 con 0x7f385c103910 2024-04-25T11:35:06.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.477+0000 7f3861267640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f385c10e640 con 0x7f385c104450 2024-04-25T11:35:06.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.477+0000 7f385affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f385c102f30 0x7f385c19a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:06.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.477+0000 7f385affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f385c102f30 0x7f385c19a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34102/0 (socket says 172.21.15.90:34102) 2024-04-25T11:35:06.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.477+0000 7f385affd640 1 -- 172.21.15.90:0/1269285371 learned_addr learned my addr 172.21.15.90:0/1269285371 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:06.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.477+0000 7f385b7fe640 1 --2- 172.21.15.90:0/1269285371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f385c104450 0x7f385c1a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:06.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.477+0000 7f385a7fc640 1 --2- 172.21.15.90:0/1269285371 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f385c103910 0x7f385c19a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:06.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.477+0000 7f385affd640 1 -- 172.21.15.90:0/1269285371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f385c104450 msgr2=0x7f385c1a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:06.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.477+0000 7f385affd640 1 --2- 172.21.15.90:0/1269285371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f385c104450 0x7f385c1a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:06.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.477+0000 7f385affd640 1 -- 172.21.15.90:0/1269285371 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f385c103910 msgr2=0x7f385c19a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:06.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.477+0000 7f385affd640 1 --2- 172.21.15.90:0/1269285371 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f385c103910 0x7f385c19a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:06.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.477+0000 7f385affd640 1 -- 172.21.15.90:0/1269285371 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f385c1a1d50 con 0x7f385c102f30 2024-04-25T11:35:06.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.477+0000 7f385affd640 1 --2- 172.21.15.90:0/1269285371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f385c102f30 0x7f385c19a210 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f3850006010 tx=0x7f3850004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:06.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.477+0000 7f3843fff640 1 -- 172.21.15.90:0/1269285371 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385003f030 con 0x7f385c102f30 2024-04-25T11:35:06.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.477+0000 7f3843fff640 1 -- 172.21.15.90:0/1269285371 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3850003aa0 con 0x7f385c102f30 2024-04-25T11:35:06.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.477+0000 7f3861267640 1 -- 172.21.15.90:0/1269285371 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f385c1a1fe0 con 0x7f385c102f30 2024-04-25T11:35:06.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.478+0000 7f3843fff640 1 -- 172.21.15.90:0/1269285371 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3850035070 con 0x7f385c102f30 2024-04-25T11:35:06.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.478+0000 7f3861267640 1 -- 172.21.15.90:0/1269285371 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f385c1a23f0 con 0x7f385c102f30 2024-04-25T11:35:06.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.478+0000 7f3861267640 1 -- 172.21.15.90:0/1269285371 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f385c1a2620 con 0x7f385c102f30 2024-04-25T11:35:06.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.480+0000 7f3843fff640 1 -- 172.21.15.90:0/1269285371 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) ==== 147537+0+0 (secure 0 0 0) 0x7f3850042030 con 0x7f385c102f30 2024-04-25T11:35:06.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.481+0000 7f3843fff640 1 --2- 172.21.15.90:0/1269285371 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f38440afb40 0x7f38440b2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:06.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.481+0000 7f385a7fc640 1 -- 172.21.15.90:0/1269285371 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f38440afb40 msgr2=0x7f38440b2050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/219449516 2024-04-25T11:35:06.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.481+0000 7f3843fff640 1 -- 172.21.15.90:0/1269285371 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f38500ff690 con 0x7f385c102f30 2024-04-25T11:35:06.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.481+0000 7f385a7fc640 1 --2- 172.21.15.90:0/1269285371 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f38440afb40 0x7f38440b2050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T11:35:06.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.485+0000 7f3843fff640 1 -- 172.21.15.90:0/1269285371 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f38500c7c00 con 0x7f385c102f30 2024-04-25T11:35:06.527 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:06 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:35:06.309+0000 7ff3b55c3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T11:35:06.527 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:06 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:35:06.442+0000 7ff3b55c3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T11:35:06.527 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:06 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:35:06.538+0000 7ff3b55c3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T11:35:06.528 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:06 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/229832068' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T11:35:06.563 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:06 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/229832068' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T11:35:06.563 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:06 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:35:06.342+0000 7f8a53947200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T11:35:06.563 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:06 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:35:06.478+0000 7f8a53947200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T11:35:06.601 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:06 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/229832068' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T11:35:06.602 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:06 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:06.246+0000 7f440e686200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T11:35:06.602 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:06 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:06.345+0000 7f440e686200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T11:35:06.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.667+0000 7f3861267640 1 -- 172.21.15.90:0/1269285371 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f385c1a2960 con 0x7f385c102f30 2024-04-25T11:35:06.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.669+0000 7f3843fff640 1 -- 172.21.15.90:0/1269285371 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+22706 (secure 0 0 0) 0x7f38500cbf00 con 0x7f385c102f30 2024-04-25T11:35:06.658 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:35:06.658 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":79,"fsid":"58bd4e7a-02f6-11ef-bc93-c7b262605968","created":"2024-04-25T11:25:23.574550+0000","modified":"2024-04-25T11:34:17.871834+0000","last_up_change":"2024-04-25T11:32:23.852211+0000","last_in_change":"2024-04-25T11:32:04.188507+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:28:42.746649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:33:08.442925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:33:09.352886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e54d888-e67b-4951-8ae7-e355fb6b10a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6803","nonce":1201629159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6805","nonce":1201629159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6809","nonce":1201629159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6807","nonce":1201629159}]},"public_addr":"172.21.15.90:6803/1201629159","cluster_addr":"172.21.15.90:6805/1201629159","heartbeat_back_addr":"172.21.15.90:6809/1201629159","heartbeat_front_addr":"172.21.15.90:6807/1201629159","state":["exists","up"]},{"osd":1,"uuid":"91a71c78-dae9-4bc7-9f6f-9e55cfefbe28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6801","nonce":3976851933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6803","nonce":3976851933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6807","nonce":3976851933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6805","nonce":3976851933}]},"public_addr":"172.21.15.138:6801/3976851933","cluster_addr":"172.21.15.138:6803/3976851933","heartbeat_back_addr":"172.21.15.138:6807/3976851933","heartbeat_front_addr":"172.21.15.138:6805/3976851933","state":["exists","up"]},{"osd":2,"uuid":"daff6c7f-80fa-4859-9339-ef7a5e840832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6801","nonce":1505807670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6803","nonce":1505807670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6807","nonce":1505807670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6805","nonce":1505807670}]},"public_addr":"172.21.15.156:6801/1505807670","cluster_addr":"172.21.15.156:6803/1505807670","heartbeat_back_addr":"172.21.15.156:6807/1505807670","heartbeat_front_addr":"172.21.15.156:6805/1505807670","state":["exists","up"]},{"osd":3,"uuid":"33fd94ca-7465-401a-bc1e-706b6b31b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6811","nonce":2520012383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6813","nonce":2520012383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6817","nonce":2520012383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6815","nonce":2520012383}]},"public_addr":"172.21.15.90:6811/2520012383","cluster_addr":"172.21.15.90:6813/2520012383","heartbeat_back_addr":"172.21.15.90:6817/2520012383","heartbeat_front_addr":"172.21.15.90:6815/2520012383","state":["exists","up"]},{"osd":4,"uuid":"c5326212-750c-49fd-96e3-d1e62bb4b7f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6809","nonce":3752678279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6811","nonce":3752678279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6815","nonce":3752678279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6813","nonce":3752678279}]},"public_addr":"172.21.15.138:6809/3752678279","cluster_addr":"172.21.15.138:6811/3752678279","heartbeat_back_addr":"172.21.15.138:6815/3752678279","heartbeat_front_addr":"172.21.15.138:6813/3752678279","state":["exists","up"]},{"osd":5,"uuid":"d04a1156-820b-4b43-be89-e16a8910c3aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6809","nonce":3591210191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6811","nonce":3591210191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6815","nonce":3591210191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6813","nonce":3591210191}]},"public_addr":"172.21.15.156:6809/3591210191","cluster_addr":"172.21.15.156:6811/3591210191","heartbeat_back_addr":"172.21.15.156:6815/3591210191","heartbeat_front_addr":"172.21.15.156:6813/3591210191","state":["exists","up"]},{"osd":6,"uuid":"75675aa3-e978-490c-8205-13d9868a3d6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6819","nonce":2364955456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6821","nonce":2364955456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6825","nonce":2364955456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6823","nonce":2364955456}]},"public_addr":"172.21.15.90:6819/2364955456","cluster_addr":"172.21.15.90:6821/2364955456","heartbeat_back_addr":"172.21.15.90:6825/2364955456","heartbeat_front_addr":"172.21.15.90:6823/2364955456","state":["exists","up"]},{"osd":7,"uuid":"1a95fdf5-f548-4271-bbcd-e330f7955eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6817","nonce":4012476387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6819","nonce":4012476387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6823","nonce":4012476387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6821","nonce":4012476387}]},"public_addr":"172.21.15.138:6817/4012476387","cluster_addr":"172.21.15.138:6819/4012476387","heartbeat_back_addr":"172.21.15.138:6823/4012476387","heartbeat_front_addr":"172.21.15.138:6821/4012476387","state":["exists","up"]},{"osd":8,"uuid":"605e10b5-5806-4c1f-8e28-a7994d34ddce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6817","nonce":101092224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6819","nonce":101092224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6823","nonce":101092224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6821","nonce":101092224}]},"public_addr":"172.21.15.156:6817/101092224","cluster_addr":"172.21.15.156:6819/101092224","heartbeat_back_addr":"172.21.15.156:6823/101092224","heartbeat_front_addr":"172.21.15.156:6821/101092224","state":["exists","up"]},{"osd":9,"uuid":"d70fdf5e-9d0f-483d-a7d8-505b79b8a418","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6827","nonce":1171265966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6829","nonce":1171265966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6833","nonce":1171265966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6831","nonce":1171265966}]},"public_addr":"172.21.15.90:6827/1171265966","cluster_addr":"172.21.15.90:6829/1171265966","heartbeat_back_addr":"172.21.15.90:6833/1171265966","heartbeat_front_addr":"172.21.15.90:6831/1171265966","state":["exists","up"]},{"osd":10,"uuid":"1ec86c7d-c3a4-432d-9664-8e40836e1e20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6825","nonce":4264666232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6827","nonce":4264666232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6831","nonce":4264666232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6829","nonce":4264666232}]},"public_addr":"172.21.15.138:6825/4264666232","cluster_addr":"172.21.15.138:6827/4264666232","heartbeat_back_addr":"172.21.15.138:6831/4264666232","heartbeat_front_addr":"172.21.15.138:6829/4264666232","state":["exists","up"]},{"osd":11,"uuid":"51cbe729-292e-4d7b-b8e9-e1e2821ffe76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6825","nonce":1224251744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6827","nonce":1224251744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6831","nonce":1224251744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6829","nonce":1224251744}]},"public_addr":"172.21.15.156:6825/1224251744","cluster_addr":"172.21.15.156:6827/1224251744","heartbeat_back_addr":"172.21.15.156:6831/1224251744","heartbeat_front_addr":"172.21.15.156:6829/1224251744","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:27:50.794226+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:15.030278+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:38.518163+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:03.107351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:27.500707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:51.463545+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:16.015076+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:41.182071+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:05.362309+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:31.048936+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:56.488509+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:32:22.129178+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":7}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/2381633953":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6800/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/2190846900":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1707792941":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6800/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6801/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/6061052":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1849633392":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6801/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/2277490702":"2024-04-26T11:26:16.655278+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:35:06.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.671+0000 7f3861267640 1 -- 172.21.15.90:0/1269285371 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f38440afb40 msgr2=0x7f38440b2050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:35:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.671+0000 7f3861267640 1 --2- 172.21.15.90:0/1269285371 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f38440afb40 0x7f38440b2050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.671+0000 7f3861267640 1 -- 172.21.15.90:0/1269285371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f385c102f30 msgr2=0x7f385c19a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.671+0000 7f3861267640 1 --2- 172.21.15.90:0/1269285371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f385c102f30 0x7f385c19a210 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f3850006010 tx=0x7f3850004270 comp rx=0 tx=0).stop 2024-04-25T11:35:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.672+0000 7f3861267640 1 -- 172.21.15.90:0/1269285371 shutdown_connections 2024-04-25T11:35:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.672+0000 7f3861267640 1 --2- 172.21.15.90:0/1269285371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f385c104450 0x7f385c1a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.672+0000 7f3861267640 1 --2- 172.21.15.90:0/1269285371 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f385c103910 0x7f385c19a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.672+0000 7f3861267640 1 --2- 172.21.15.90:0/1269285371 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f38440afb40 0x7f38440b2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.672+0000 7f3861267640 1 --2- 172.21.15.90:0/1269285371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f385c102f30 0x7f385c19a210 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.672+0000 7f3861267640 1 -- 172.21.15.90:0/1269285371 >> 172.21.15.90:0/1269285371 conn(0x7f385c0fd330 msgr2=0x7f385c101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.672+0000 7f3861267640 1 -- 172.21.15.90:0/1269285371 shutdown_connections 2024-04-25T11:35:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.672+0000 7f3861267640 1 -- 172.21.15.90:0/1269285371 wait complete. 2024-04-25T11:35:06.673 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T11:28:42.746649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T11:33:08.442925+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T11:33:09.352886+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T11:35:06.673 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T11:35:06.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.796+0000 7f7f5339d640 1 -- 172.21.15.90:0/4219895613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f4c102800 msgr2=0x7f7f4c103500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:06.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.796+0000 7f7f5339d640 1 --2- 172.21.15.90:0/4219895613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f4c102800 0x7f7f4c103500 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c009900 tx=0x7f7f3c02a410 comp rx=0 tx=0).stop 2024-04-25T11:35:06.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.796+0000 7f7f5339d640 1 -- 172.21.15.90:0/4219895613 shutdown_connections 2024-04-25T11:35:06.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.796+0000 7f7f5339d640 1 --2- 172.21.15.90:0/4219895613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f4c108740 0x7f7f4c10ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:06.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.796+0000 7f7f5339d640 1 --2- 172.21.15.90:0/4219895613 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f4c103a40 0x7f7f4c103e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:06.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.796+0000 7f7f5339d640 1 --2- 172.21.15.90:0/4219895613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f4c102800 0x7f7f4c103500 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:06.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.796+0000 7f7f5339d640 1 -- 172.21.15.90:0/4219895613 >> 172.21.15.90:0/4219895613 conn(0x7f7f4c0fd9f0 msgr2=0x7f7f4c0ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:06.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.797+0000 7f7f5339d640 1 -- 172.21.15.90:0/4219895613 shutdown_connections 2024-04-25T11:35:06.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.797+0000 7f7f5339d640 1 -- 172.21.15.90:0/4219895613 wait complete. 2024-04-25T11:35:06.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.797+0000 7f7f5339d640 1 Processor -- start 2024-04-25T11:35:06.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.797+0000 7f7f5339d640 1 -- start start 2024-04-25T11:35:06.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.797+0000 7f7f5339d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f4c102800 0x7f7f4c198f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:06.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.797+0000 7f7f5339d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f4c103a40 0x7f7f4c199490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:06.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.797+0000 7f7f5339d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f4c108740 0x7f7f4c1a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:06.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.797+0000 7f7f5339d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7f4c10e2e0 con 0x7f7f4c102800 2024-04-25T11:35:06.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.797+0000 7f7f5339d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7f4c10e160 con 0x7f7f4c108740 2024-04-25T11:35:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.797+0000 7f7f5339d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7f4c10e460 con 0x7f7f4c103a40 2024-04-25T11:35:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.797+0000 7f7f5239b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f4c102800 0x7f7f4c198f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.798+0000 7f7f5239b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f4c102800 0x7f7f4c198f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34124/0 (socket says 172.21.15.90:34124) 2024-04-25T11:35:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.798+0000 7f7f5239b640 1 -- 172.21.15.90:0/2053407577 learned_addr learned my addr 172.21.15.90:0/2053407577 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.798+0000 7f7f52b9c640 1 --2- 172.21.15.90:0/2053407577 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f4c108740 0x7f7f4c1a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.798+0000 7f7f51b9a640 1 --2- 172.21.15.90:0/2053407577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f4c103a40 0x7f7f4c199490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.798+0000 7f7f5239b640 1 -- 172.21.15.90:0/2053407577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f4c103a40 msgr2=0x7f7f4c199490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.798+0000 7f7f5239b640 1 --2- 172.21.15.90:0/2053407577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f4c103a40 0x7f7f4c199490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.798+0000 7f7f5239b640 1 -- 172.21.15.90:0/2053407577 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f4c108740 msgr2=0x7f7f4c1a0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.798+0000 7f7f5239b640 1 --2- 172.21.15.90:0/2053407577 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f4c108740 0x7f7f4c1a0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.798+0000 7f7f5239b640 1 -- 172.21.15.90:0/2053407577 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f4c1a0a90 con 0x7f7f4c102800 2024-04-25T11:35:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.798+0000 7f7f5239b640 1 --2- 172.21.15.90:0/2053407577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f4c102800 0x7f7f4c198f50 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c006010 tx=0x7f7f3c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:06.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.798+0000 7f7f437fe640 1 -- 172.21.15.90:0/2053407577 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c03f030 con 0x7f7f4c102800 2024-04-25T11:35:06.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.798+0000 7f7f5339d640 1 -- 172.21.15.90:0/2053407577 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f4c1a0d20 con 0x7f7f4c102800 2024-04-25T11:35:06.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.798+0000 7f7f437fe640 1 -- 172.21.15.90:0/2053407577 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f3c003aa0 con 0x7f7f4c102800 2024-04-25T11:35:06.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.798+0000 7f7f437fe640 1 -- 172.21.15.90:0/2053407577 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c035070 con 0x7f7f4c102800 2024-04-25T11:35:06.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.798+0000 7f7f5339d640 1 -- 172.21.15.90:0/2053407577 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f4c1a11b0 con 0x7f7f4c102800 2024-04-25T11:35:06.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.800+0000 7f7f5339d640 1 -- 172.21.15.90:0/2053407577 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f4c107550 con 0x7f7f4c102800 2024-04-25T11:35:06.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.801+0000 7f7f437fe640 1 -- 172.21.15.90:0/2053407577 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) ==== 147537+0+0 (secure 0 0 0) 0x7f7f3c042030 con 0x7f7f4c102800 2024-04-25T11:35:06.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.801+0000 7f7f437fe640 1 --2- 172.21.15.90:0/2053407577 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f7f380afb30 0x7f7f380b2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:06.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.801+0000 7f7f51b9a640 1 -- 172.21.15.90:0/2053407577 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f7f380afb30 msgr2=0x7f7f380b2040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/219449516 2024-04-25T11:35:06.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.801+0000 7f7f437fe640 1 -- 172.21.15.90:0/2053407577 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f7f3c0ff280 con 0x7f7f4c102800 2024-04-25T11:35:06.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.801+0000 7f7f51b9a640 1 --2- 172.21.15.90:0/2053407577 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f7f380afb30 0x7f7f380b2040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T11:35:06.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.806+0000 7f7f437fe640 1 -- 172.21.15.90:0/2053407577 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7f3c0c77f0 con 0x7f7f4c102800 2024-04-25T11:35:06.801 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:06 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:35:06.656+0000 7ff3b55c3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T11:35:06.837 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:06 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:35:06.575+0000 7f8a53947200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T11:35:06.838 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:06 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:35:06.694+0000 7f8a53947200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T11:35:06.862 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:06 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:06.612+0000 7f440e686200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T11:35:06.862 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:06 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:06.757+0000 7f440e686200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T11:35:06.863 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:06 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:06.873+0000 7f440e686200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T11:35:06.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.998+0000 7f7f5339d640 1 -- 172.21.15.90:0/2053407577 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7f4c0637a0 con 0x7f7f4c102800 2024-04-25T11:35:06.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:06.999+0000 7f7f437fe640 1 -- 172.21.15.90:0/2053407577 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) ==== 93+0+10 (secure 0 0 0) 0x7f7f3c0cbaf0 con 0x7f7f4c102800 2024-04-25T11:35:06.988 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T11:35:06.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.001+0000 7f7f5339d640 1 -- 172.21.15.90:0/2053407577 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f7f380afb30 msgr2=0x7f7f380b2040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:35:06.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.001+0000 7f7f5339d640 1 --2- 172.21.15.90:0/2053407577 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f7f380afb30 0x7f7f380b2040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:06.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.001+0000 7f7f5339d640 1 -- 172.21.15.90:0/2053407577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f4c102800 msgr2=0x7f7f4c198f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:06.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.001+0000 7f7f5339d640 1 --2- 172.21.15.90:0/2053407577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f4c102800 0x7f7f4c198f50 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c006010 tx=0x7f7f3c004270 comp rx=0 tx=0).stop 2024-04-25T11:35:06.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.001+0000 7f7f5339d640 1 -- 172.21.15.90:0/2053407577 shutdown_connections 2024-04-25T11:35:06.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.001+0000 7f7f5339d640 1 --2- 172.21.15.90:0/2053407577 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f4c108740 0x7f7f4c1a0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:06.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.001+0000 7f7f5339d640 1 --2- 172.21.15.90:0/2053407577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f4c103a40 0x7f7f4c199490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:06.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.001+0000 7f7f5339d640 1 --2- 172.21.15.90:0/2053407577 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f7f380afb30 0x7f7f380b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:06.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.001+0000 7f7f5339d640 1 --2- 172.21.15.90:0/2053407577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f4c102800 0x7f7f4c198f50 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:06.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.001+0000 7f7f5339d640 1 -- 172.21.15.90:0/2053407577 >> 172.21.15.90:0/2053407577 conn(0x7f7f4c0fd9f0 msgr2=0x7f7f4c104720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:06.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.001+0000 7f7f5339d640 1 -- 172.21.15.90:0/2053407577 shutdown_connections 2024-04-25T11:35:06.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.001+0000 7f7f5339d640 1 -- 172.21.15.90:0/2053407577 wait complete. 2024-04-25T11:35:07.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T11:35:07.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.125+0000 7f7a886d1640 1 -- 172.21.15.90:0/73607804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a801045e0 msgr2=0x7f7a80106a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:07.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.125+0000 7f7a886d1640 1 --2- 172.21.15.90:0/73607804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a801045e0 0x7f7a80106a00 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f7a70009900 tx=0x7f7a7002a410 comp rx=0 tx=0).stop 2024-04-25T11:35:07.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.125+0000 7f7a886d1640 1 -- 172.21.15.90:0/73607804 shutdown_connections 2024-04-25T11:35:07.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.125+0000 7f7a886d1640 1 --2- 172.21.15.90:0/73607804 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7a801098a0 0x7f7a8010bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.125+0000 7f7a886d1640 1 --2- 172.21.15.90:0/73607804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a80106f40 0x7f7a80109360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.125+0000 7f7a886d1640 1 --2- 172.21.15.90:0/73607804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a801045e0 0x7f7a80106a00 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.125+0000 7f7a886d1640 1 -- 172.21.15.90:0/73607804 >> 172.21.15.90:0/73607804 conn(0x7f7a800fdbe0 msgr2=0x7f7a80100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:07.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.125+0000 7f7a886d1640 1 -- 172.21.15.90:0/73607804 shutdown_connections 2024-04-25T11:35:07.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.125+0000 7f7a886d1640 1 -- 172.21.15.90:0/73607804 wait complete. 2024-04-25T11:35:07.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.126+0000 7f7a886d1640 1 Processor -- start 2024-04-25T11:35:07.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.126+0000 7f7a886d1640 1 -- start start 2024-04-25T11:35:07.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.126+0000 7f7a886d1640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7a801045e0 0x7f7a8019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:07.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.126+0000 7f7a886d1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a80106f40 0x7f7a8019a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:07.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.126+0000 7f7a886d1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a801098a0 0x7f7a801a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:07.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.126+0000 7f7a886d1640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8010f5f0 con 0x7f7a80106f40 2024-04-25T11:35:07.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.126+0000 7f7a886d1640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8010f470 con 0x7f7a801045e0 2024-04-25T11:35:07.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.126+0000 7f7a886d1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8010f770 con 0x7f7a801098a0 2024-04-25T11:35:07.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.126+0000 7f7a85c6c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a80106f40 0x7f7a8019a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:07.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.126+0000 7f7a85c6c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a80106f40 0x7f7a8019a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34146/0 (socket says 172.21.15.90:34146) 2024-04-25T11:35:07.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.126+0000 7f7a85c6c640 1 -- 172.21.15.90:0/1459134065 learned_addr learned my addr 172.21.15.90:0/1459134065 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:07.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.126+0000 7f7a8646d640 1 --2- 172.21.15.90:0/1459134065 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7a801045e0 0x7f7a8019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:07.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.126+0000 7f7a86c6e640 1 --2- 172.21.15.90:0/1459134065 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a801098a0 0x7f7a801a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:07.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.126+0000 7f7a85c6c640 1 -- 172.21.15.90:0/1459134065 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a801098a0 msgr2=0x7f7a801a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:07.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.126+0000 7f7a85c6c640 1 --2- 172.21.15.90:0/1459134065 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a801098a0 0x7f7a801a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.126+0000 7f7a85c6c640 1 -- 172.21.15.90:0/1459134065 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7a801045e0 msgr2=0x7f7a8019a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:07.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.126+0000 7f7a85c6c640 1 --2- 172.21.15.90:0/1459134065 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7a801045e0 0x7f7a8019a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.126+0000 7f7a85c6c640 1 -- 172.21.15.90:0/1459134065 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a801a1df0 con 0x7f7a80106f40 2024-04-25T11:35:07.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.127+0000 7f7a85c6c640 1 --2- 172.21.15.90:0/1459134065 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a80106f40 0x7f7a8019a7f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c00e9f0 tx=0x7f7a7c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:07.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.127+0000 7f7a777fe640 1 -- 172.21.15.90:0/1459134065 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c00cd80 con 0x7f7a80106f40 2024-04-25T11:35:07.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.127+0000 7f7a886d1640 1 -- 172.21.15.90:0/1459134065 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a801a20e0 con 0x7f7a80106f40 2024-04-25T11:35:07.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.127+0000 7f7a777fe640 1 -- 172.21.15.90:0/1459134065 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a7c0054d0 con 0x7f7a80106f40 2024-04-25T11:35:07.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.127+0000 7f7a777fe640 1 -- 172.21.15.90:0/1459134065 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c010690 con 0x7f7a80106f40 2024-04-25T11:35:07.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.127+0000 7f7a886d1640 1 -- 172.21.15.90:0/1459134065 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a801a2630 con 0x7f7a80106f40 2024-04-25T11:35:07.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.128+0000 7f7a886d1640 1 -- 172.21.15.90:0/1459134065 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a800ff700 con 0x7f7a80106f40 2024-04-25T11:35:07.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.129+0000 7f7a777fe640 1 -- 172.21.15.90:0/1459134065 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) ==== 147537+0+0 (secure 0 0 0) 0x7f7a7c004e90 con 0x7f7a80106f40 2024-04-25T11:35:07.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.130+0000 7f7a777fe640 1 --2- 172.21.15.90:0/1459134065 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f7a6c0afb40 0x7f7a6c0b2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:07.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.130+0000 7f7a8646d640 1 -- 172.21.15.90:0/1459134065 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f7a6c0afb40 msgr2=0x7f7a6c0b2050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/219449516 2024-04-25T11:35:07.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.130+0000 7f7a777fe640 1 -- 172.21.15.90:0/1459134065 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f7a7c0df0c0 con 0x7f7a80106f40 2024-04-25T11:35:07.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.130+0000 7f7a8646d640 1 --2- 172.21.15.90:0/1459134065 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f7a6c0afb40 0x7f7a6c0b2050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T11:35:07.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.134+0000 7f7a777fe640 1 -- 172.21.15.90:0/1459134065 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7a7c0a7c80 con 0x7f7a80106f40 2024-04-25T11:35:07.225 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:06 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:06.997+0000 7f440e686200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T11:35:07.274 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:06 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:35:06.961+0000 7ff3b55c3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T11:35:07.274 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:07 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:35:07.002+0000 7f8a53947200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T11:35:07.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.314+0000 7f7a886d1640 1 -- 172.21.15.90:0/1459134065 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7a800637a0 con 0x7f7a80106f40 2024-04-25T11:35:07.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.314+0000 7f7a777fe640 1 -- 172.21.15.90:0/1459134065 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v79) ==== 104+0+11 (secure 0 0 0) 0x7f7a7c0a7580 con 0x7f7a80106f40 2024-04-25T11:35:07.303 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T11:35:07.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.316+0000 7f7a886d1640 1 -- 172.21.15.90:0/1459134065 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f7a6c0afb40 msgr2=0x7f7a6c0b2050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:35:07.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.316+0000 7f7a886d1640 1 --2- 172.21.15.90:0/1459134065 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f7a6c0afb40 0x7f7a6c0b2050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.316+0000 7f7a886d1640 1 -- 172.21.15.90:0/1459134065 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a80106f40 msgr2=0x7f7a8019a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:07.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.316+0000 7f7a886d1640 1 --2- 172.21.15.90:0/1459134065 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a80106f40 0x7f7a8019a7f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c00e9f0 tx=0x7f7a7c00edf0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.316+0000 7f7a886d1640 1 -- 172.21.15.90:0/1459134065 shutdown_connections 2024-04-25T11:35:07.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.316+0000 7f7a886d1640 1 --2- 172.21.15.90:0/1459134065 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a801098a0 0x7f7a801a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.316+0000 7f7a886d1640 1 --2- 172.21.15.90:0/1459134065 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f7a6c0afb40 0x7f7a6c0b2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.316+0000 7f7a886d1640 1 --2- 172.21.15.90:0/1459134065 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a80106f40 0x7f7a8019a7f0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.316+0000 7f7a886d1640 1 --2- 172.21.15.90:0/1459134065 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7a801045e0 0x7f7a8019a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.316+0000 7f7a886d1640 1 -- 172.21.15.90:0/1459134065 >> 172.21.15.90:0/1459134065 conn(0x7f7a800fdbe0 msgr2=0x7f7a80106640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:07.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.317+0000 7f7a886d1640 1 -- 172.21.15.90:0/1459134065 shutdown_connections 2024-04-25T11:35:07.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.317+0000 7f7a886d1640 1 -- 172.21.15.90:0/1459134065 wait complete. 2024-04-25T11:35:07.317 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T11:35:07.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.441+0000 7f43a6735640 1 -- 172.21.15.90:0/1463456684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43a01029e0 msgr2=0x7f43a01036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:07.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.441+0000 7f43a6735640 1 --2- 172.21.15.90:0/1463456684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43a01029e0 0x7f43a01036e0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4388009900 tx=0x7f438802a410 comp rx=0 tx=0).stop 2024-04-25T11:35:07.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.441+0000 7f43a6735640 1 -- 172.21.15.90:0/1463456684 shutdown_connections 2024-04-25T11:35:07.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.441+0000 7f43a6735640 1 --2- 172.21.15.90:0/1463456684 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f43a0108920 0x7f43a010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.441+0000 7f43a6735640 1 --2- 172.21.15.90:0/1463456684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43a0103c20 0x7f43a0104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.441+0000 7f43a6735640 1 --2- 172.21.15.90:0/1463456684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43a01029e0 0x7f43a01036e0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.441+0000 7f43a6735640 1 -- 172.21.15.90:0/1463456684 >> 172.21.15.90:0/1463456684 conn(0x7f43a00fdbd0 msgr2=0x7f43a0100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:07.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.441+0000 7f43a6735640 1 -- 172.21.15.90:0/1463456684 shutdown_connections 2024-04-25T11:35:07.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.441+0000 7f43a6735640 1 -- 172.21.15.90:0/1463456684 wait complete. 2024-04-25T11:35:07.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.441+0000 7f43a6735640 1 Processor -- start 2024-04-25T11:35:07.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.441+0000 7f43a6735640 1 -- start start 2024-04-25T11:35:07.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.442+0000 7f43a6735640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f43a01029e0 0x7f43a006a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:07.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.442+0000 7f43a6735640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43a0103c20 0x7f43a019f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:07.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.442+0000 7f43a6735640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43a0108920 0x7f43a01a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:07.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.442+0000 7f43a6735640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f43a010e4c0 con 0x7f43a0108920 2024-04-25T11:35:07.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.442+0000 7f43a6735640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f43a010e340 con 0x7f43a01029e0 2024-04-25T11:35:07.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.442+0000 7f43a6735640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f43a010e640 con 0x7f43a0103c20 2024-04-25T11:35:07.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.442+0000 7f43a4cd2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43a0108920 0x7f43a01a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:07.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.442+0000 7f439ffff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f43a01029e0 0x7f43a006a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:07.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.442+0000 7f439ffff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f43a01029e0 0x7f43a006a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:51814/0 (socket says 172.21.15.90:51814) 2024-04-25T11:35:07.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.442+0000 7f43a4cd2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43a0108920 0x7f43a01a1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34166/0 (socket says 172.21.15.90:34166) 2024-04-25T11:35:07.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.442+0000 7f439ffff640 1 -- 172.21.15.90:0/1548516717 learned_addr learned my addr 172.21.15.90:0/1548516717 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:07.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.442+0000 7f439f7fe640 1 --2- 172.21.15.90:0/1548516717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43a0103c20 0x7f43a019f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:07.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.442+0000 7f439ffff640 1 -- 172.21.15.90:0/1548516717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43a0103c20 msgr2=0x7f43a019f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:07.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.442+0000 7f439ffff640 1 --2- 172.21.15.90:0/1548516717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43a0103c20 0x7f43a019f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.442+0000 7f439ffff640 1 -- 172.21.15.90:0/1548516717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43a0108920 msgr2=0x7f43a01a1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:07.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.442+0000 7f439ffff640 1 --2- 172.21.15.90:0/1548516717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43a0108920 0x7f43a01a1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.442+0000 7f439ffff640 1 -- 172.21.15.90:0/1548516717 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43a01a2040 con 0x7f43a01029e0 2024-04-25T11:35:07.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.442+0000 7f43a4cd2640 1 --2- 172.21.15.90:0/1548516717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43a0108920 0x7f43a01a1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:35:07.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.443+0000 7f439ffff640 1 --2- 172.21.15.90:0/1548516717 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f43a01029e0 0x7f43a006a960 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4388006010 tx=0x7f4388004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:07.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.443+0000 7f439d7fa640 1 -- 172.21.15.90:0/1548516717 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438803f030 con 0x7f43a01029e0 2024-04-25T11:35:07.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.443+0000 7f43a6735640 1 -- 172.21.15.90:0/1548516717 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43a01a22d0 con 0x7f43a01029e0 2024-04-25T11:35:07.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.443+0000 7f439d7fa640 1 -- 172.21.15.90:0/1548516717 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4388031910 con 0x7f43a01029e0 2024-04-25T11:35:07.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.443+0000 7f439d7fa640 1 -- 172.21.15.90:0/1548516717 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4388035070 con 0x7f43a01029e0 2024-04-25T11:35:07.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.443+0000 7f43a6735640 1 -- 172.21.15.90:0/1548516717 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43a01a27c0 con 0x7f43a01029e0 2024-04-25T11:35:07.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.444+0000 7f43a6735640 1 -- 172.21.15.90:0/1548516717 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43a0107730 con 0x7f43a01029e0 2024-04-25T11:35:07.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.445+0000 7f439d7fa640 1 -- 172.21.15.90:0/1548516717 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 19) ==== 147537+0+0 (secure 0 0 0) 0x7f4388042030 con 0x7f43a01029e0 2024-04-25T11:35:07.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.446+0000 7f439d7fa640 1 --2- 172.21.15.90:0/1548516717 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f438c0afb40 0x7f438c0b2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:07.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.446+0000 7f439f7fe640 1 -- 172.21.15.90:0/1548516717 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f438c0afb40 msgr2=0x7f438c0b2050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/219449516 2024-04-25T11:35:07.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.446+0000 7f439d7fa640 1 -- 172.21.15.90:0/1548516717 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7f43880ff3c0 con 0x7f43a01029e0 2024-04-25T11:35:07.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.446+0000 7f439f7fe640 1 --2- 172.21.15.90:0/1548516717 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f438c0afb40 0x7f438c0b2050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T11:35:07.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.449+0000 7f439d7fa640 1 -- 172.21.15.90:0/1548516717 <== mon.2 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f43880c7930 con 0x7f43a01029e0 2024-04-25T11:35:07.548 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:07 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/229832068' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-25T11:35:07.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:07 smithi138 ceph-mon[32328]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T11:35:07.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:07 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1269285371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:35:07.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:07 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2053407577' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:35:07.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:07 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/229832068' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-25T11:35:07.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:07 smithi156 ceph-mon[33045]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T11:35:07.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:07 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1269285371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:35:07.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:07 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2053407577' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:35:07.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.628+0000 7f43a6735640 1 -- 172.21.15.90:0/1548516717 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f43a002dda0 con 0x7f43a01029e0 2024-04-25T11:35:07.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.629+0000 7f439d7fa640 1 -- 172.21.15.90:0/1548516717 <== mon.2 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v79) ==== 100+0+11 (secure 0 0 0) 0x7f43880cbc30 con 0x7f43a01029e0 2024-04-25T11:35:07.618 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T11:35:07.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.631+0000 7f43a6735640 1 -- 172.21.15.90:0/1548516717 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f438c0afb40 msgr2=0x7f438c0b2050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:35:07.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.631+0000 7f43a6735640 1 --2- 172.21.15.90:0/1548516717 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f438c0afb40 0x7f438c0b2050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.631+0000 7f43a6735640 1 -- 172.21.15.90:0/1548516717 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f43a01029e0 msgr2=0x7f43a006a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:07.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.631+0000 7f43a6735640 1 --2- 172.21.15.90:0/1548516717 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f43a01029e0 0x7f43a006a960 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4388006010 tx=0x7f4388004270 comp rx=0 tx=0).stop 2024-04-25T11:35:07.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.632+0000 7f43a6735640 1 -- 172.21.15.90:0/1548516717 shutdown_connections 2024-04-25T11:35:07.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.632+0000 7f43a6735640 1 --2- 172.21.15.90:0/1548516717 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7f438c0afb40 0x7f438c0b2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.632+0000 7f43a6735640 1 --2- 172.21.15.90:0/1548516717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43a0108920 0x7f43a01a1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.632+0000 7f43a6735640 1 --2- 172.21.15.90:0/1548516717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43a0103c20 0x7f43a019f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.632+0000 7f43a6735640 1 --2- 172.21.15.90:0/1548516717 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f43a01029e0 0x7f43a006a960 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.632+0000 7f43a6735640 1 -- 172.21.15.90:0/1548516717 >> 172.21.15.90:0/1548516717 conn(0x7f43a00fdbd0 msgr2=0x7f43a0104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:07.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.632+0000 7f43a6735640 1 -- 172.21.15.90:0/1548516717 shutdown_connections 2024-04-25T11:35:07.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.632+0000 7f43a6735640 1 -- 172.21.15.90:0/1548516717 wait complete. 2024-04-25T11:35:07.631 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:07 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:07.315+0000 7f440e686200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T11:35:07.631 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:07 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/229832068' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-25T11:35:07.631 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:07 smithi090 ceph-mon[29575]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T11:35:07.631 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:07 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1269285371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:35:07.631 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:07 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2053407577' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:35:07.632 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:35:07.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.753+0000 7fcbb9eaa640 1 -- 172.21.15.90:0/844684385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbb4103910 msgr2=0x7fcbb4103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:07.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.753+0000 7fcbb9eaa640 1 --2- 172.21.15.90:0/844684385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbb4103910 0x7fcbb4103dc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fcba8009900 tx=0x7fcba802a410 comp rx=0 tx=0).stop 2024-04-25T11:35:07.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.753+0000 7fcbb9eaa640 1 -- 172.21.15.90:0/844684385 shutdown_connections 2024-04-25T11:35:07.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.753+0000 7fcbb9eaa640 1 --2- 172.21.15.90:0/844684385 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbb4104450 0x7fcbb410ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.753+0000 7fcbb9eaa640 1 --2- 172.21.15.90:0/844684385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbb4103910 0x7fcbb4103dc0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.753+0000 7fcbb9eaa640 1 --2- 172.21.15.90:0/844684385 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcbb4102f30 0x7fcbb4103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.753+0000 7fcbb9eaa640 1 -- 172.21.15.90:0/844684385 >> 172.21.15.90:0/844684385 conn(0x7fcbb40fd330 msgr2=0x7fcbb40ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:07.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.753+0000 7fcbb9eaa640 1 -- 172.21.15.90:0/844684385 shutdown_connections 2024-04-25T11:35:07.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.753+0000 7fcbb9eaa640 1 -- 172.21.15.90:0/844684385 wait complete. 2024-04-25T11:35:07.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.754+0000 7fcbb9eaa640 1 Processor -- start 2024-04-25T11:35:07.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.754+0000 7fcbb9eaa640 1 -- start start 2024-04-25T11:35:07.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.754+0000 7fcbb9eaa640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbb4102f30 0x7fcbb419a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:07.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.754+0000 7fcbb9eaa640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcbb4103910 0x7fcbb419a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:07.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.754+0000 7fcbb9eaa640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbb4104450 0x7fcbb41a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:07.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.754+0000 7fcbb9eaa640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb410e500 con 0x7fcbb4104450 2024-04-25T11:35:07.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.754+0000 7fcbb9eaa640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb410e380 con 0x7fcbb4103910 2024-04-25T11:35:07.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.754+0000 7fcbb9eaa640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb410e680 con 0x7fcbb4102f30 2024-04-25T11:35:07.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.754+0000 7fcbb3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbb4104450 0x7fcbb41a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:07.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.754+0000 7fcbb3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbb4104450 0x7fcbb41a16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34178/0 (socket says 172.21.15.90:34178) 2024-04-25T11:35:07.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.755+0000 7fcbb3fff640 1 -- 172.21.15.90:0/1174584781 learned_addr learned my addr 172.21.15.90:0/1174584781 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:07.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.755+0000 7fcbb37fe640 1 --2- 172.21.15.90:0/1174584781 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbb4102f30 0x7fcbb419a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:07.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.755+0000 7fcbb2ffd640 1 --2- 172.21.15.90:0/1174584781 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcbb4103910 0x7fcbb419a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:07.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.755+0000 7fcbb3fff640 1 -- 172.21.15.90:0/1174584781 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbb4102f30 msgr2=0x7fcbb419a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:07.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.755+0000 7fcbb3fff640 1 --2- 172.21.15.90:0/1174584781 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbb4102f30 0x7fcbb419a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.755+0000 7fcbb3fff640 1 -- 172.21.15.90:0/1174584781 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcbb4103910 msgr2=0x7fcbb419a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:07.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.755+0000 7fcbb3fff640 1 --2- 172.21.15.90:0/1174584781 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcbb4103910 0x7fcbb419a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.755+0000 7fcbb3fff640 1 -- 172.21.15.90:0/1174584781 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbb41a1d30 con 0x7fcbb4104450 2024-04-25T11:35:07.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.755+0000 7fcbb3fff640 1 --2- 172.21.15.90:0/1174584781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbb4104450 0x7fcbb41a16c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fcbac00eb10 tx=0x7fcbac00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:07.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.755+0000 7fcbb0ff9640 1 -- 172.21.15.90:0/1174584781 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbac004b20 con 0x7fcbb4104450 2024-04-25T11:35:07.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.755+0000 7fcbb0ff9640 1 -- 172.21.15.90:0/1174584781 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbac004cc0 con 0x7fcbb4104450 2024-04-25T11:35:07.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.755+0000 7fcbb0ff9640 1 -- 172.21.15.90:0/1174584781 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbac010670 con 0x7fcbb4104450 2024-04-25T11:35:07.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.755+0000 7fcbb9eaa640 1 -- 172.21.15.90:0/1174584781 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbb41a1fc0 con 0x7fcbb4104450 2024-04-25T11:35:07.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.755+0000 7fcbb9eaa640 1 -- 172.21.15.90:0/1174584781 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbb41a2430 con 0x7fcbb4104450 2024-04-25T11:35:07.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.756+0000 7fcbb9eaa640 1 -- 172.21.15.90:0/1174584781 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbb41a26c0 con 0x7fcbb4104450 2024-04-25T11:35:07.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.758+0000 7fcbb0ff9640 1 -- 172.21.15.90:0/1174584781 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) ==== 147537+0+0 (secure 0 0 0) 0x7fcbac005000 con 0x7fcbb4104450 2024-04-25T11:35:07.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.758+0000 7fcbb0ff9640 1 --2- 172.21.15.90:0/1174584781 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fcb9c0afb40 0x7fcb9c0b2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:07.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.758+0000 7fcbb37fe640 1 -- 172.21.15.90:0/1174584781 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fcb9c0afb40 msgr2=0x7fcb9c0b2050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/219449516 2024-04-25T11:35:07.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.758+0000 7fcbb0ff9640 1 -- 172.21.15.90:0/1174584781 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7fcbac0df430 con 0x7fcbb4104450 2024-04-25T11:35:07.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.759+0000 7fcbb37fe640 1 --2- 172.21.15.90:0/1174584781 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fcb9c0afb40 0x7fcb9c0b2050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T11:35:07.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.762+0000 7fcbb0ff9640 1 -- 172.21.15.90:0/1174584781 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcbac053ff0 con 0x7fcbb4104450 2024-04-25T11:35:07.802 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:07 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:35:07.560+0000 7ff3b55c3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T11:35:07.802 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:35:07 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y[33703]: 2024-04-25T11:35:07.779+0000 7ff3b55c3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T11:35:07.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.943+0000 7fcbb9eaa640 1 -- 172.21.15.90:0/1174584781 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcbb41a2890 con 0x7fcbb4104450 2024-04-25T11:35:07.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.945+0000 7fcbb0ff9640 1 -- 172.21.15.90:0/1174584781 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+22706 (secure 0 0 0) 0x7fcbac0a7ff0 con 0x7fcbb4104450 2024-04-25T11:35:07.934 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:35:07.934 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":79,"fsid":"58bd4e7a-02f6-11ef-bc93-c7b262605968","created":"2024-04-25T11:25:23.574550+0000","modified":"2024-04-25T11:34:17.871834+0000","last_up_change":"2024-04-25T11:32:23.852211+0000","last_in_change":"2024-04-25T11:32:04.188507+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:28:42.746649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:33:08.442925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:33:09.352886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e54d888-e67b-4951-8ae7-e355fb6b10a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6803","nonce":1201629159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6805","nonce":1201629159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6809","nonce":1201629159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6807","nonce":1201629159}]},"public_addr":"172.21.15.90:6803/1201629159","cluster_addr":"172.21.15.90:6805/1201629159","heartbeat_back_addr":"172.21.15.90:6809/1201629159","heartbeat_front_addr":"172.21.15.90:6807/1201629159","state":["exists","up"]},{"osd":1,"uuid":"91a71c78-dae9-4bc7-9f6f-9e55cfefbe28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6801","nonce":3976851933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6803","nonce":3976851933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6807","nonce":3976851933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6805","nonce":3976851933}]},"public_addr":"172.21.15.138:6801/3976851933","cluster_addr":"172.21.15.138:6803/3976851933","heartbeat_back_addr":"172.21.15.138:6807/3976851933","heartbeat_front_addr":"172.21.15.138:6805/3976851933","state":["exists","up"]},{"osd":2,"uuid":"daff6c7f-80fa-4859-9339-ef7a5e840832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6801","nonce":1505807670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6803","nonce":1505807670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6807","nonce":1505807670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6805","nonce":1505807670}]},"public_addr":"172.21.15.156:6801/1505807670","cluster_addr":"172.21.15.156:6803/1505807670","heartbeat_back_addr":"172.21.15.156:6807/1505807670","heartbeat_front_addr":"172.21.15.156:6805/1505807670","state":["exists","up"]},{"osd":3,"uuid":"33fd94ca-7465-401a-bc1e-706b6b31b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6811","nonce":2520012383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6813","nonce":2520012383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6817","nonce":2520012383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6815","nonce":2520012383}]},"public_addr":"172.21.15.90:6811/2520012383","cluster_addr":"172.21.15.90:6813/2520012383","heartbeat_back_addr":"172.21.15.90:6817/2520012383","heartbeat_front_addr":"172.21.15.90:6815/2520012383","state":["exists","up"]},{"osd":4,"uuid":"c5326212-750c-49fd-96e3-d1e62bb4b7f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6809","nonce":3752678279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6811","nonce":3752678279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6815","nonce":3752678279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6813","nonce":3752678279}]},"public_addr":"172.21.15.138:6809/3752678279","cluster_addr":"172.21.15.138:6811/3752678279","heartbeat_back_addr":"172.21.15.138:6815/3752678279","heartbeat_front_addr":"172.21.15.138:6813/3752678279","state":["exists","up"]},{"osd":5,"uuid":"d04a1156-820b-4b43-be89-e16a8910c3aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6809","nonce":3591210191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6811","nonce":3591210191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6815","nonce":3591210191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6813","nonce":3591210191}]},"public_addr":"172.21.15.156:6809/3591210191","cluster_addr":"172.21.15.156:6811/3591210191","heartbeat_back_addr":"172.21.15.156:6815/3591210191","heartbeat_front_addr":"172.21.15.156:6813/3591210191","state":["exists","up"]},{"osd":6,"uuid":"75675aa3-e978-490c-8205-13d9868a3d6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6819","nonce":2364955456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6821","nonce":2364955456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6825","nonce":2364955456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6823","nonce":2364955456}]},"public_addr":"172.21.15.90:6819/2364955456","cluster_addr":"172.21.15.90:6821/2364955456","heartbeat_back_addr":"172.21.15.90:6825/2364955456","heartbeat_front_addr":"172.21.15.90:6823/2364955456","state":["exists","up"]},{"osd":7,"uuid":"1a95fdf5-f548-4271-bbcd-e330f7955eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6817","nonce":4012476387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6819","nonce":4012476387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6823","nonce":4012476387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6821","nonce":4012476387}]},"public_addr":"172.21.15.138:6817/4012476387","cluster_addr":"172.21.15.138:6819/4012476387","heartbeat_back_addr":"172.21.15.138:6823/4012476387","heartbeat_front_addr":"172.21.15.138:6821/4012476387","state":["exists","up"]},{"osd":8,"uuid":"605e10b5-5806-4c1f-8e28-a7994d34ddce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6817","nonce":101092224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6819","nonce":101092224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6823","nonce":101092224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6821","nonce":101092224}]},"public_addr":"172.21.15.156:6817/101092224","cluster_addr":"172.21.15.156:6819/101092224","heartbeat_back_addr":"172.21.15.156:6823/101092224","heartbeat_front_addr":"172.21.15.156:6821/101092224","state":["exists","up"]},{"osd":9,"uuid":"d70fdf5e-9d0f-483d-a7d8-505b79b8a418","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6827","nonce":1171265966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6829","nonce":1171265966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6833","nonce":1171265966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6831","nonce":1171265966}]},"public_addr":"172.21.15.90:6827/1171265966","cluster_addr":"172.21.15.90:6829/1171265966","heartbeat_back_addr":"172.21.15.90:6833/1171265966","heartbeat_front_addr":"172.21.15.90:6831/1171265966","state":["exists","up"]},{"osd":10,"uuid":"1ec86c7d-c3a4-432d-9664-8e40836e1e20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6825","nonce":4264666232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6827","nonce":4264666232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6831","nonce":4264666232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6829","nonce":4264666232}]},"public_addr":"172.21.15.138:6825/4264666232","cluster_addr":"172.21.15.138:6827/4264666232","heartbeat_back_addr":"172.21.15.138:6831/4264666232","heartbeat_front_addr":"172.21.15.138:6829/4264666232","state":["exists","up"]},{"osd":11,"uuid":"51cbe729-292e-4d7b-b8e9-e1e2821ffe76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6825","nonce":1224251744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6827","nonce":1224251744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6831","nonce":1224251744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6829","nonce":1224251744}]},"public_addr":"172.21.15.156:6825/1224251744","cluster_addr":"172.21.15.156:6827/1224251744","heartbeat_back_addr":"172.21.15.156:6831/1224251744","heartbeat_front_addr":"172.21.15.156:6829/1224251744","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:27:50.794226+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:15.030278+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:38.518163+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:03.107351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:27.500707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:51.463545+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:16.015076+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:41.182071+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:05.362309+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:31.048936+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:56.488509+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:32:22.129178+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":7}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/2381633953":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6800/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/2190846900":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1707792941":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6800/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6801/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/6061052":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1849633392":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6801/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/2277490702":"2024-04-26T11:26:16.655278+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:35:07.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.947+0000 7fcbb9eaa640 1 -- 172.21.15.90:0/1174584781 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fcb9c0afb40 msgr2=0x7fcb9c0b2050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:35:07.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.947+0000 7fcbb9eaa640 1 --2- 172.21.15.90:0/1174584781 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fcb9c0afb40 0x7fcb9c0b2050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.947+0000 7fcbb9eaa640 1 -- 172.21.15.90:0/1174584781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbb4104450 msgr2=0x7fcbb41a16c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:07.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.947+0000 7fcbb9eaa640 1 --2- 172.21.15.90:0/1174584781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbb4104450 0x7fcbb41a16c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fcbac00eb10 tx=0x7fcbac00c2d0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.947+0000 7fcbb9eaa640 1 -- 172.21.15.90:0/1174584781 shutdown_connections 2024-04-25T11:35:07.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.947+0000 7fcbb9eaa640 1 --2- 172.21.15.90:0/1174584781 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fcb9c0afb40 0x7fcb9c0b2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.947+0000 7fcbb9eaa640 1 --2- 172.21.15.90:0/1174584781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbb4104450 0x7fcbb41a16c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.947+0000 7fcbb9eaa640 1 --2- 172.21.15.90:0/1174584781 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcbb4103910 0x7fcbb419a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.947+0000 7fcbb9eaa640 1 --2- 172.21.15.90:0/1174584781 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbb4102f30 0x7fcbb419a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:07.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.947+0000 7fcbb9eaa640 1 -- 172.21.15.90:0/1174584781 >> 172.21.15.90:0/1174584781 conn(0x7fcbb40fd330 msgr2=0x7fcbb4109d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:07.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.947+0000 7fcbb9eaa640 1 -- 172.21.15.90:0/1174584781 shutdown_connections 2024-04-25T11:35:07.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:07.947+0000 7fcbb9eaa640 1 -- 172.21.15.90:0/1174584781 wait complete. 2024-04-25T11:35:07.947 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T11:28:42.746649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T11:33:08.442925+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T11:33:09.352886+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T11:35:07.948 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T11:35:08.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.071+0000 7fdfeff06640 1 -- 172.21.15.90:0/1534658416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfe8104d20 msgr2=0x7fdfe8105130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:08.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.071+0000 7fdfeff06640 1 --2- 172.21.15.90:0/1534658416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfe8104d20 0x7fdfe8105130 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8009900 tx=0x7fdfd802a410 comp rx=0 tx=0).stop 2024-04-25T11:35:08.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.071+0000 7fdfeff06640 1 -- 172.21.15.90:0/1534658416 shutdown_connections 2024-04-25T11:35:08.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.071+0000 7fdfeff06640 1 --2- 172.21.15.90:0/1534658416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfe8109a20 0x7fdfe810be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.071+0000 7fdfeff06640 1 --2- 172.21.15.90:0/1534658416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfe8104d20 0x7fdfe8105130 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.071+0000 7fdfeff06640 1 --2- 172.21.15.90:0/1534658416 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdfe8103ae0 0x7fdfe81047e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.071+0000 7fdfeff06640 1 -- 172.21.15.90:0/1534658416 >> 172.21.15.90:0/1534658416 conn(0x7fdfe80fed10 msgr2=0x7fdfe8101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:08.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.071+0000 7fdfeff06640 1 -- 172.21.15.90:0/1534658416 shutdown_connections 2024-04-25T11:35:08.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.071+0000 7fdfeff06640 1 -- 172.21.15.90:0/1534658416 wait complete. 2024-04-25T11:35:08.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.072+0000 7fdfeff06640 1 Processor -- start 2024-04-25T11:35:08.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.072+0000 7fdfeff06640 1 -- start start 2024-04-25T11:35:08.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.072+0000 7fdfeff06640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdfe8103ae0 0x7fdfe819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:08.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.072+0000 7fdfeff06640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfe8104d20 0x7fdfe819a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:08.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.072+0000 7fdfeff06640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfe8109a20 0x7fdfe81a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:08.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.072+0000 7fdfeff06640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe810f5e0 con 0x7fdfe8109a20 2024-04-25T11:35:08.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.072+0000 7fdfeff06640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe810f460 con 0x7fdfe8103ae0 2024-04-25T11:35:08.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.072+0000 7fdfeff06640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe810f760 con 0x7fdfe8104d20 2024-04-25T11:35:08.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.072+0000 7fdfee4a3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfe8109a20 0x7fdfe81a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:08.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.072+0000 7fdfee4a3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfe8109a20 0x7fdfe81a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34184/0 (socket says 172.21.15.90:34184) 2024-04-25T11:35:08.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.072+0000 7fdfee4a3640 1 -- 172.21.15.90:0/3872376766 learned_addr learned my addr 172.21.15.90:0/3872376766 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:08.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.072+0000 7fdfedca2640 1 --2- 172.21.15.90:0/3872376766 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdfe8103ae0 0x7fdfe819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:08.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.072+0000 7fdfed4a1640 1 --2- 172.21.15.90:0/3872376766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfe8104d20 0x7fdfe819a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:08.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.073+0000 7fdfee4a3640 1 -- 172.21.15.90:0/3872376766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfe8104d20 msgr2=0x7fdfe819a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:08.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.073+0000 7fdfee4a3640 1 --2- 172.21.15.90:0/3872376766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfe8104d20 0x7fdfe819a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.073+0000 7fdfee4a3640 1 -- 172.21.15.90:0/3872376766 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdfe8103ae0 msgr2=0x7fdfe819a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:08.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.073+0000 7fdfee4a3640 1 --2- 172.21.15.90:0/3872376766 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdfe8103ae0 0x7fdfe819a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.073+0000 7fdfee4a3640 1 -- 172.21.15.90:0/3872376766 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfe81a1d80 con 0x7fdfe8109a20 2024-04-25T11:35:08.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.073+0000 7fdfee4a3640 1 --2- 172.21.15.90:0/3872376766 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfe8109a20 0x7fdfe81a1710 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fdfe0009fd0 tx=0x7fdfe000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:08.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.073+0000 7fdfdeffd640 1 -- 172.21.15.90:0/3872376766 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe0017070 con 0x7fdfe8109a20 2024-04-25T11:35:08.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.073+0000 7fdfeff06640 1 -- 172.21.15.90:0/3872376766 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfe81a2070 con 0x7fdfe8109a20 2024-04-25T11:35:08.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.073+0000 7fdfdeffd640 1 -- 172.21.15.90:0/3872376766 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfe0003e20 con 0x7fdfe8109a20 2024-04-25T11:35:08.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.073+0000 7fdfdeffd640 1 -- 172.21.15.90:0/3872376766 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe0010460 con 0x7fdfe8109a20 2024-04-25T11:35:08.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.073+0000 7fdfeff06640 1 -- 172.21.15.90:0/3872376766 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfe81a2570 con 0x7fdfe8109a20 2024-04-25T11:35:08.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.074+0000 7fdfeff06640 1 -- 172.21.15.90:0/3872376766 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfe806a160 con 0x7fdfe8109a20 2024-04-25T11:35:08.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.076+0000 7fdfdeffd640 1 -- 172.21.15.90:0/3872376766 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) ==== 147537+0+0 (secure 0 0 0) 0x7fdfe0003970 con 0x7fdfe8109a20 2024-04-25T11:35:08.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.076+0000 7fdfdeffd640 1 --2- 172.21.15.90:0/3872376766 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fdfd40afb40 0x7fdfd40b2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:08.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.076+0000 7fdfedca2640 1 -- 172.21.15.90:0/3872376766 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fdfd40afb40 msgr2=0x7fdfd40b2050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/219449516 2024-04-25T11:35:08.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.076+0000 7fdfdeffd640 1 -- 172.21.15.90:0/3872376766 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8269+0+0 (secure 0 0 0) 0x7fdfe00dea40 con 0x7fdfe8109a20 2024-04-25T11:35:08.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.076+0000 7fdfedca2640 1 --2- 172.21.15.90:0/3872376766 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fdfd40afb40 0x7fdfd40b2050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T11:35:08.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.080+0000 7fdfdeffd640 1 -- 172.21.15.90:0/3872376766 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdfe00a7600 con 0x7fdfe8109a20 2024-04-25T11:35:08.088 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:07 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:35:07.608+0000 7f8a53947200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T11:35:08.088 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:35:07 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z[34916]: 2024-04-25T11:35:07.829+0000 7f8a53947200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T11:35:08.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.215+0000 7fdfdeffd640 1 -- 172.21.15.90:0/3872376766 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 20) ==== 147205+0+0 (secure 0 0 0) 0x7fdfe00a3190 con 0x7fdfe8109a20 2024-04-25T11:35:08.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.215+0000 7fdfdeffd640 1 -- 172.21.15.90:0/3872376766 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fdfd40afb40 msgr2=0x7fdfd40b2050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:35:08.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.215+0000 7fdfdeffd640 1 --2- 172.21.15.90:0/3872376766 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fdfd40afb40 0x7fdfd40b2050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.225 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:07 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:07.960+0000 7f440e686200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T11:35:08.225 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:08.198+0000 7f440e686200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T11:35:08.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.275+0000 7fdfeff06640 1 -- 172.21.15.90:0/3872376766 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdfe802db50 con 0x7fdfe8109a20 2024-04-25T11:35:08.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.276+0000 7fdfdeffd640 1 -- 172.21.15.90:0/3872376766 <== mon.0 v2:172.21.15.90:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) ==== 93+0+10 (secure 0 0 0) 0x7fdfe00c90a0 con 0x7fdfe8109a20 2024-04-25T11:35:08.265 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T11:35:08.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.278+0000 7fdfeff06640 1 -- 172.21.15.90:0/3872376766 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfe8109a20 msgr2=0x7fdfe81a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:08.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.278+0000 7fdfeff06640 1 --2- 172.21.15.90:0/3872376766 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfe8109a20 0x7fdfe81a1710 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fdfe0009fd0 tx=0x7fdfe000b260 comp rx=0 tx=0).stop 2024-04-25T11:35:08.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.278+0000 7fdfeff06640 1 -- 172.21.15.90:0/3872376766 shutdown_connections 2024-04-25T11:35:08.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.278+0000 7fdfeff06640 1 --2- 172.21.15.90:0/3872376766 >> [v2:172.21.15.90:6800/219449516,v1:172.21.15.90:6801/219449516] conn(0x7fdfd40afb40 0x7fdfd40b2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.278+0000 7fdfeff06640 1 --2- 172.21.15.90:0/3872376766 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfe8109a20 0x7fdfe81a1710 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.278+0000 7fdfeff06640 1 --2- 172.21.15.90:0/3872376766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfe8104d20 0x7fdfe819a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.278+0000 7fdfeff06640 1 --2- 172.21.15.90:0/3872376766 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdfe8103ae0 0x7fdfe819a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.278+0000 7fdfeff06640 1 -- 172.21.15.90:0/3872376766 >> 172.21.15.90:0/3872376766 conn(0x7fdfe80fed10 msgr2=0x7fdfe8105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:08.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.278+0000 7fdfeff06640 1 -- 172.21.15.90:0/3872376766 shutdown_connections 2024-04-25T11:35:08.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.278+0000 7fdfeff06640 1 -- 172.21.15.90:0/3872376766 wait complete. 2024-04-25T11:35:08.281 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T11:35:08.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.434+0000 7f87139c5640 1 -- 172.21.15.90:0/1397266440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f870c103c20 msgr2=0x7f870c104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:08.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.434+0000 7f87139c5640 1 --2- 172.21.15.90:0/1397266440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f870c103c20 0x7f870c104030 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f86fc009900 tx=0x7f86fc02a410 comp rx=0 tx=0).stop 2024-04-25T11:35:08.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.434+0000 7f87139c5640 1 -- 172.21.15.90:0/1397266440 shutdown_connections 2024-04-25T11:35:08.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.434+0000 7f87139c5640 1 --2- 172.21.15.90:0/1397266440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f870c108920 0x7f870c10ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.434+0000 7f87139c5640 1 --2- 172.21.15.90:0/1397266440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f870c103c20 0x7f870c104030 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.434+0000 7f87139c5640 1 --2- 172.21.15.90:0/1397266440 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f870c1029e0 0x7f870c1036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.434+0000 7f87139c5640 1 -- 172.21.15.90:0/1397266440 >> 172.21.15.90:0/1397266440 conn(0x7f870c0fdbd0 msgr2=0x7f870c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:08.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.434+0000 7f87139c5640 1 -- 172.21.15.90:0/1397266440 shutdown_connections 2024-04-25T11:35:08.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.434+0000 7f87139c5640 1 -- 172.21.15.90:0/1397266440 wait complete. 2024-04-25T11:35:08.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.435+0000 7f87139c5640 1 Processor -- start 2024-04-25T11:35:08.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.435+0000 7f87139c5640 1 -- start start 2024-04-25T11:35:08.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.435+0000 7f87139c5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f870c1029e0 0x7f870c198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:08.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.435+0000 7f87139c5640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f870c103c20 0x7f870c198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:08.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.435+0000 7f87139c5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f870c108920 0x7f870c19f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:08.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.435+0000 7f87139c5640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f870c10d400 con 0x7f870c108920 2024-04-25T11:35:08.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.435+0000 7f87139c5640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f870c10d280 con 0x7f870c103c20 2024-04-25T11:35:08.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.435+0000 7f87139c5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f870c10d580 con 0x7f870c1029e0 2024-04-25T11:35:08.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.435+0000 7f8711f62640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f870c108920 0x7f870c19f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:08.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.435+0000 7f8711f62640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f870c108920 0x7f870c19f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34260/0 (socket says 172.21.15.90:34260) 2024-04-25T11:35:08.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.435+0000 7f8711f62640 1 -- 172.21.15.90:0/690826379 learned_addr learned my addr 172.21.15.90:0/690826379 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:08.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.435+0000 7f8711761640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f870c1029e0 0x7f870c198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:08.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.435+0000 7f8710f60640 1 --2- 172.21.15.90:0/690826379 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f870c103c20 0x7f870c198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:08.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.436+0000 7f8711761640 1 -- 172.21.15.90:0/690826379 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f870c103c20 msgr2=0x7f870c198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:08.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.436+0000 7f8711761640 1 --2- 172.21.15.90:0/690826379 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f870c103c20 0x7f870c198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.436+0000 7f8711761640 1 -- 172.21.15.90:0/690826379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f870c108920 msgr2=0x7f870c19f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:08.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.436+0000 7f8711761640 1 --2- 172.21.15.90:0/690826379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f870c108920 0x7f870c19f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.436+0000 7f8711761640 1 -- 172.21.15.90:0/690826379 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f870c19fe70 con 0x7f870c1029e0 2024-04-25T11:35:08.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.436+0000 7f8711761640 1 --2- 172.21.15.90:0/690826379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f870c1029e0 0x7f870c198330 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f870c1035e0 tx=0x7f870800ad60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:08.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.436+0000 7f87027fc640 1 -- 172.21.15.90:0/690826379 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8708019070 con 0x7f870c1029e0 2024-04-25T11:35:08.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.436+0000 7f87139c5640 1 -- 172.21.15.90:0/690826379 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f870c1a0160 con 0x7f870c1029e0 2024-04-25T11:35:08.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.436+0000 7f87139c5640 1 -- 172.21.15.90:0/690826379 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f870c1a0700 con 0x7f870c1029e0 2024-04-25T11:35:08.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.436+0000 7f87027fc640 1 -- 172.21.15.90:0/690826379 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8708007e30 con 0x7f870c1029e0 2024-04-25T11:35:08.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.436+0000 7f87027fc640 1 -- 172.21.15.90:0/690826379 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87080133b0 con 0x7f870c1029e0 2024-04-25T11:35:08.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.437+0000 7f87139c5640 1 -- 172.21.15.90:0/690826379 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86f00052c0 con 0x7f870c1029e0 2024-04-25T11:35:08.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.439+0000 7f87027fc640 1 -- 172.21.15.90:0/690826379 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 20) ==== 147205+0+0 (secure 0 0 0) 0x7f870801c070 con 0x7f870c1029e0 2024-04-25T11:35:08.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.440+0000 7f87027fc640 1 -- 172.21.15.90:0/690826379 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f87080dda30 con 0x7f870c1029e0 2024-04-25T11:35:08.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.445+0000 7f87027fc640 1 -- 172.21.15.90:0/690826379 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f87080a6240 con 0x7f870c1029e0 2024-04-25T11:35:08.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1459134065' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:35:08.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1548516717' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:35:08.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: Standby manager daemon y restarted 2024-04-25T11:35:08.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: Standby manager daemon y started 2024-04-25T11:35:08.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.? 172.21.15.138:0/3325406563' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T11:35:08.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.? 172.21.15.138:0/3325406563' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T11:35:08.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.? 172.21.15.138:0/3325406563' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T11:35:08.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.? 172.21.15.138:0/3325406563' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T11:35:08.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: Standby manager daemon z restarted 2024-04-25T11:35:08.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: Standby manager daemon z started 2024-04-25T11:35:08.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.? 172.21.15.156:0/1587347362' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T11:35:08.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.? 172.21.15.156:0/1587347362' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T11:35:08.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.? 172.21.15.156:0/1587347362' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T11:35:08.554 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.? 172.21.15.156:0/1587347362' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T11:35:08.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1174584781' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:35:08.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: Active manager daemon x restarted 2024-04-25T11:35:08.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: Activating manager daemon x 2024-04-25T11:35:08.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: osdmap e80: 12 total, 12 up, 12 in 2024-04-25T11:35:08.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: mgrmap e20: x(active, starting, since 0.0170728s), standbys: z, y 2024-04-25T11:35:08.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T11:35:08.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:35:08.555 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:35:08.556 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T11:35:08.556 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T11:35:08.556 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T11:35:08.556 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T11:35:08.556 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T11:35:08.556 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T11:35:08.556 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T11:35:08.556 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T11:35:08.556 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T11:35:08.556 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T11:35:08.557 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T11:35:08.557 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T11:35:08.557 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T11:35:08.557 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T11:35:08.557 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T11:35:08.557 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T11:35:08.557 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T11:35:08.557 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T11:35:08.557 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T11:35:08.557 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T11:35:08.558 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T11:35:08.558 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T11:35:08.558 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T11:35:08.558 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T11:35:08.558 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T11:35:08.558 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T11:35:08.558 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T11:35:08.558 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T11:35:08.559 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T11:35:08.559 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T11:35:08.559 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: Manager daemon x is now available 2024-04-25T11:35:08.559 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3872376766' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:35:08.559 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:35:08.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1459134065' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:35:08.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1548516717' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:35:08.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: Standby manager daemon y restarted 2024-04-25T11:35:08.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: Standby manager daemon y started 2024-04-25T11:35:08.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.? 172.21.15.138:0/3325406563' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T11:35:08.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.? 172.21.15.138:0/3325406563' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T11:35:08.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.? 172.21.15.138:0/3325406563' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T11:35:08.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.? 172.21.15.138:0/3325406563' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T11:35:08.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: Standby manager daemon z restarted 2024-04-25T11:35:08.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: Standby manager daemon z started 2024-04-25T11:35:08.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.? 172.21.15.156:0/1587347362' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T11:35:08.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.? 172.21.15.156:0/1587347362' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T11:35:08.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.? 172.21.15.156:0/1587347362' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T11:35:08.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.? 172.21.15.156:0/1587347362' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T11:35:08.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1174584781' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:35:08.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: Active manager daemon x restarted 2024-04-25T11:35:08.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: Activating manager daemon x 2024-04-25T11:35:08.590 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: osdmap e80: 12 total, 12 up, 12 in 2024-04-25T11:35:08.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: mgrmap e20: x(active, starting, since 0.0170728s), standbys: z, y 2024-04-25T11:35:08.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T11:35:08.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:35:08.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:35:08.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T11:35:08.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T11:35:08.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T11:35:08.591 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T11:35:08.592 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T11:35:08.592 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T11:35:08.592 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T11:35:08.592 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T11:35:08.592 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T11:35:08.592 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T11:35:08.592 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T11:35:08.592 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T11:35:08.593 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T11:35:08.593 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T11:35:08.593 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T11:35:08.593 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T11:35:08.593 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T11:35:08.593 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T11:35:08.593 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T11:35:08.593 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T11:35:08.593 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T11:35:08.593 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T11:35:08.593 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T11:35:08.593 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T11:35:08.593 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T11:35:08.593 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T11:35:08.593 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T11:35:08.594 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T11:35:08.594 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T11:35:08.594 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T11:35:08.594 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: Manager daemon x is now available 2024-04-25T11:35:08.594 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3872376766' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:35:08.594 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:35:08.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.655+0000 7f87139c5640 1 -- 172.21.15.90:0/690826379 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f86f0005880 con 0x7f870c1029e0 2024-04-25T11:35:08.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.655+0000 7f87027fc640 1 -- 172.21.15.90:0/690826379 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v80) ==== 104+0+11 (secure 0 0 0) 0x7f87080aa540 con 0x7f870c1029e0 2024-04-25T11:35:08.644 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T11:35:08.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.658+0000 7f87139c5640 1 -- 172.21.15.90:0/690826379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f870c1029e0 msgr2=0x7f870c198330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:08.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.658+0000 7f87139c5640 1 --2- 172.21.15.90:0/690826379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f870c1029e0 0x7f870c198330 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f870c1035e0 tx=0x7f870800ad60 comp rx=0 tx=0).stop 2024-04-25T11:35:08.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.658+0000 7f87139c5640 1 -- 172.21.15.90:0/690826379 shutdown_connections 2024-04-25T11:35:08.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.658+0000 7f87139c5640 1 --2- 172.21.15.90:0/690826379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f870c108920 0x7f870c19f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.658+0000 7f87139c5640 1 --2- 172.21.15.90:0/690826379 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f870c103c20 0x7f870c198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.658+0000 7f87139c5640 1 --2- 172.21.15.90:0/690826379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f870c1029e0 0x7f870c198330 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.658+0000 7f87139c5640 1 -- 172.21.15.90:0/690826379 >> 172.21.15.90:0/690826379 conn(0x7f870c0fdbd0 msgr2=0x7f870c104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:08.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.658+0000 7f87139c5640 1 -- 172.21.15.90:0/690826379 shutdown_connections 2024-04-25T11:35:08.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.658+0000 7f87139c5640 1 -- 172.21.15.90:0/690826379 wait complete. 2024-04-25T11:35:08.658 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:08.492+0000 7f436d58d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T11:35:08.658 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:08.492+0000 7f436d58d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T11:35:08.659 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:08.492+0000 7f436d58d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T11:35:08.659 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:08.492+0000 7f436d58d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T11:35:08.659 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:08.492+0000 7f436d58d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T11:35:08.659 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:08.492+0000 7f436fd92700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T11:35:08.659 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:08.492+0000 7f436fd92700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T11:35:08.659 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:08.492+0000 7f436fd92700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T11:35:08.659 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:08.492+0000 7f436fd92700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T11:35:08.659 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:35:08.492+0000 7f436fd92700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T11:35:08.660 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1459134065' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:35:08.660 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1548516717' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:35:08.660 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: Standby manager daemon y restarted 2024-04-25T11:35:08.660 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: Standby manager daemon y started 2024-04-25T11:35:08.660 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.? 172.21.15.138:0/3325406563' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T11:35:08.660 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.? 172.21.15.138:0/3325406563' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T11:35:08.660 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.? 172.21.15.138:0/3325406563' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T11:35:08.660 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.? 172.21.15.138:0/3325406563' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T11:35:08.660 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: Standby manager daemon z restarted 2024-04-25T11:35:08.660 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: Standby manager daemon z started 2024-04-25T11:35:08.660 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.? 172.21.15.156:0/1587347362' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T11:35:08.660 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.? 172.21.15.156:0/1587347362' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T11:35:08.660 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.? 172.21.15.156:0/1587347362' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T11:35:08.660 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.? 172.21.15.156:0/1587347362' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T11:35:08.661 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1174584781' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:35:08.661 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: Active manager daemon x restarted 2024-04-25T11:35:08.661 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: Activating manager daemon x 2024-04-25T11:35:08.661 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: osdmap e80: 12 total, 12 up, 12 in 2024-04-25T11:35:08.661 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: mgrmap e20: x(active, starting, since 0.0170728s), standbys: z, y 2024-04-25T11:35:08.661 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T11:35:08.661 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T11:35:08.661 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T11:35:08.662 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T11:35:08.662 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T11:35:08.662 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T11:35:08.662 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T11:35:08.662 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T11:35:08.662 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T11:35:08.662 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T11:35:08.662 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T11:35:08.662 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T11:35:08.662 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T11:35:08.662 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T11:35:08.662 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T11:35:08.663 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T11:35:08.663 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T11:35:08.663 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T11:35:08.663 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T11:35:08.663 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T11:35:08.663 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T11:35:08.663 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T11:35:08.663 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T11:35:08.663 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T11:35:08.663 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T11:35:08.663 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T11:35:08.663 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T11:35:08.664 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T11:35:08.664 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T11:35:08.664 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T11:35:08.664 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T11:35:08.664 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T11:35:08.664 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T11:35:08.664 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: Manager daemon x is now available 2024-04-25T11:35:08.664 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3872376766' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:35:08.664 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:35:08.665 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T11:35:08.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.792+0000 7f98ba35e640 1 -- 172.21.15.90:0/455923326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98b41080b0 msgr2=0x7f98b410a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:08.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.792+0000 7f98ba35e640 1 --2- 172.21.15.90:0/455923326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98b41080b0 0x7f98b410a4d0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f98ac00adb0 tx=0x7f98ac02b7a0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.792+0000 7f98ba35e640 1 -- 172.21.15.90:0/455923326 shutdown_connections 2024-04-25T11:35:08.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.792+0000 7f98ba35e640 1 --2- 172.21.15.90:0/455923326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98b41080b0 0x7f98b410a4d0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.792+0000 7f98ba35e640 1 --2- 172.21.15.90:0/455923326 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98b41033b0 0x7f98b41037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.792+0000 7f98ba35e640 1 --2- 172.21.15.90:0/455923326 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f98b4102170 0x7f98b4102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.792+0000 7f98ba35e640 1 -- 172.21.15.90:0/455923326 >> 172.21.15.90:0/455923326 conn(0x7f98b40fd340 msgr2=0x7f98b40ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:08.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.792+0000 7f98ba35e640 1 -- 172.21.15.90:0/455923326 shutdown_connections 2024-04-25T11:35:08.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.792+0000 7f98ba35e640 1 -- 172.21.15.90:0/455923326 wait complete. 2024-04-25T11:35:08.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.793+0000 7f98ba35e640 1 Processor -- start 2024-04-25T11:35:08.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.793+0000 7f98ba35e640 1 -- start start 2024-04-25T11:35:08.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.793+0000 7f98ba35e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98b4102170 0x7f98b40ff1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:08.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.793+0000 7f98ba35e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f98b41033b0 0x7f98b40fd870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:08.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.793+0000 7f98ba35e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98b41080b0 0x7f98b40fddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:08.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.793+0000 7f98ba35e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f98b410d400 con 0x7f98b4102170 2024-04-25T11:35:08.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.793+0000 7f98ba35e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f98b410d280 con 0x7f98b41033b0 2024-04-25T11:35:08.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.793+0000 7f98ba35e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f98b410d580 con 0x7f98b41080b0 2024-04-25T11:35:08.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.794+0000 7f98b3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98b4102170 0x7f98b40ff1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:08.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.794+0000 7f98b3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98b4102170 0x7f98b40ff1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34320/0 (socket says 172.21.15.90:34320) 2024-04-25T11:35:08.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.794+0000 7f98b3fff640 1 -- 172.21.15.90:0/2324453382 learned_addr learned my addr 172.21.15.90:0/2324453382 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:08.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.794+0000 7f98b37fe640 1 --2- 172.21.15.90:0/2324453382 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f98b41033b0 0x7f98b40fd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:08.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.794+0000 7f98b88fb640 1 --2- 172.21.15.90:0/2324453382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98b41080b0 0x7f98b40fddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:08.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.794+0000 7f98b3fff640 1 -- 172.21.15.90:0/2324453382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98b41080b0 msgr2=0x7f98b40fddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:08.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.794+0000 7f98b3fff640 1 --2- 172.21.15.90:0/2324453382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98b41080b0 0x7f98b40fddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.794+0000 7f98b3fff640 1 -- 172.21.15.90:0/2324453382 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f98b41033b0 msgr2=0x7f98b40fd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:08.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.794+0000 7f98b3fff640 1 --2- 172.21.15.90:0/2324453382 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f98b41033b0 0x7f98b40fd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.794+0000 7f98b3fff640 1 -- 172.21.15.90:0/2324453382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b40fe4e0 con 0x7f98b4102170 2024-04-25T11:35:08.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.794+0000 7f98b3fff640 1 --2- 172.21.15.90:0/2324453382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98b4102170 0x7f98b40ff1d0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f98a400e9f0 tx=0x7f98a400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:08.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.794+0000 7f98b17fa640 1 -- 172.21.15.90:0/2324453382 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a400ccc0 con 0x7f98b4102170 2024-04-25T11:35:08.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.794+0000 7f98ba35e640 1 -- 172.21.15.90:0/2324453382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98b41a42c0 con 0x7f98b4102170 2024-04-25T11:35:08.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.794+0000 7f98b17fa640 1 -- 172.21.15.90:0/2324453382 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98a40035f0 con 0x7f98b4102170 2024-04-25T11:35:08.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.794+0000 7f98b17fa640 1 -- 172.21.15.90:0/2324453382 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a400ce60 con 0x7f98b4102170 2024-04-25T11:35:08.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.794+0000 7f98ba35e640 1 -- 172.21.15.90:0/2324453382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98b41a47b0 con 0x7f98b4102170 2024-04-25T11:35:08.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.796+0000 7f98ba35e640 1 -- 172.21.15.90:0/2324453382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98b40692e0 con 0x7f98b4102170 2024-04-25T11:35:08.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.797+0000 7f98b17fa640 1 -- 172.21.15.90:0/2324453382 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 20) ==== 147205+0+0 (secure 0 0 0) 0x7f98a4004020 con 0x7f98b4102170 2024-04-25T11:35:08.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.797+0000 7f98b17fa640 1 -- 172.21.15.90:0/2324453382 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f98a40dea10 con 0x7f98b4102170 2024-04-25T11:35:08.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:08.802+0000 7f98b17fa640 1 -- 172.21.15.90:0/2324453382 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f98a40a78d0 con 0x7f98b4102170 2024-04-25T11:35:08.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.006+0000 7f98ba35e640 1 -- 172.21.15.90:0/2324453382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f98b41a4a70 con 0x7f98b4102170 2024-04-25T11:35:08.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.007+0000 7f98b17fa640 1 -- 172.21.15.90:0/2324453382 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v80) ==== 100+0+11 (secure 0 0 0) 0x7f98a40a71d0 con 0x7f98b4102170 2024-04-25T11:35:08.995 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T11:35:08.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.009+0000 7f98ba35e640 1 -- 172.21.15.90:0/2324453382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98b4102170 msgr2=0x7f98b40ff1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:08.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.009+0000 7f98ba35e640 1 --2- 172.21.15.90:0/2324453382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98b4102170 0x7f98b40ff1d0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f98a400e9f0 tx=0x7f98a400eeb0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.010+0000 7f98ba35e640 1 -- 172.21.15.90:0/2324453382 shutdown_connections 2024-04-25T11:35:08.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.010+0000 7f98ba35e640 1 --2- 172.21.15.90:0/2324453382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98b41080b0 0x7f98b40fddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.010+0000 7f98ba35e640 1 --2- 172.21.15.90:0/2324453382 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f98b41033b0 0x7f98b40fd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.010+0000 7f98ba35e640 1 --2- 172.21.15.90:0/2324453382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98b4102170 0x7f98b40ff1d0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:08.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.010+0000 7f98ba35e640 1 -- 172.21.15.90:0/2324453382 >> 172.21.15.90:0/2324453382 conn(0x7f98b40fd340 msgr2=0x7f98b4109bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:08.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.010+0000 7f98ba35e640 1 -- 172.21.15.90:0/2324453382 shutdown_connections 2024-04-25T11:35:08.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.010+0000 7f98ba35e640 1 -- 172.21.15.90:0/2324453382 wait complete. 2024-04-25T11:35:09.011 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-25T11:35:09.011 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:35:09.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.131+0000 7f32bbe27640 1 -- 172.21.15.90:0/48402604 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32b4102f30 msgr2=0x7f32b4103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:09.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.131+0000 7f32bbe27640 1 --2- 172.21.15.90:0/48402604 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32b4102f30 0x7f32b4103340 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f32b0009900 tx=0x7f32b002a410 comp rx=0 tx=0).stop 2024-04-25T11:35:09.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.131+0000 7f32bbe27640 1 -- 172.21.15.90:0/48402604 shutdown_connections 2024-04-25T11:35:09.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.131+0000 7f32bbe27640 1 --2- 172.21.15.90:0/48402604 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f32b4104450 0x7f32b410ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:09.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.131+0000 7f32bbe27640 1 --2- 172.21.15.90:0/48402604 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32b4103910 0x7f32b4103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:09.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.131+0000 7f32bbe27640 1 --2- 172.21.15.90:0/48402604 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32b4102f30 0x7f32b4103340 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:09.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.131+0000 7f32bbe27640 1 -- 172.21.15.90:0/48402604 >> 172.21.15.90:0/48402604 conn(0x7f32b40fd330 msgr2=0x7f32b40ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:09.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.131+0000 7f32bbe27640 1 -- 172.21.15.90:0/48402604 shutdown_connections 2024-04-25T11:35:09.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.131+0000 7f32bbe27640 1 -- 172.21.15.90:0/48402604 wait complete. 2024-04-25T11:35:09.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.132+0000 7f32bbe27640 1 Processor -- start 2024-04-25T11:35:09.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.132+0000 7f32bbe27640 1 -- start start 2024-04-25T11:35:09.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.132+0000 7f32bbe27640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f32b4102f30 0x7f32b419a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:09.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.132+0000 7f32bbe27640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32b4103910 0x7f32b419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:09.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.132+0000 7f32bbe27640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32b4104450 0x7f32b41a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:09.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.132+0000 7f32bbe27640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f32b410f610 con 0x7f32b4103910 2024-04-25T11:35:09.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.132+0000 7f32bbe27640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f32b410f490 con 0x7f32b4102f30 2024-04-25T11:35:09.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.132+0000 7f32bbe27640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f32b410f790 con 0x7f32b4104450 2024-04-25T11:35:09.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.132+0000 7f32b93c2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32b4103910 0x7f32b419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:09.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.132+0000 7f32b93c2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32b4103910 0x7f32b419a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34342/0 (socket says 172.21.15.90:34342) 2024-04-25T11:35:09.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.132+0000 7f32b93c2640 1 -- 172.21.15.90:0/3509944741 learned_addr learned my addr 172.21.15.90:0/3509944741 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:09.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.132+0000 7f32b9bc3640 1 --2- 172.21.15.90:0/3509944741 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f32b4102f30 0x7f32b419a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:09.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.132+0000 7f32ba3c4640 1 --2- 172.21.15.90:0/3509944741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32b4104450 0x7f32b41a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:09.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.132+0000 7f32b93c2640 1 -- 172.21.15.90:0/3509944741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32b4104450 msgr2=0x7f32b41a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:09.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.132+0000 7f32b93c2640 1 --2- 172.21.15.90:0/3509944741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32b4104450 0x7f32b41a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:09.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.132+0000 7f32b93c2640 1 -- 172.21.15.90:0/3509944741 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f32b4102f30 msgr2=0x7f32b419a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:09.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.132+0000 7f32b93c2640 1 --2- 172.21.15.90:0/3509944741 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f32b4102f30 0x7f32b419a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:09.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.132+0000 7f32b93c2640 1 -- 172.21.15.90:0/3509944741 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32b41a1d60 con 0x7f32b4103910 2024-04-25T11:35:09.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.133+0000 7f32b93c2640 1 --2- 172.21.15.90:0/3509944741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32b4103910 0x7f32b419a760 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f32a400d770 tx=0x7f32a400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:09.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.133+0000 7f32aaffd640 1 -- 172.21.15.90:0/3509944741 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a4017070 con 0x7f32b4103910 2024-04-25T11:35:09.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.133+0000 7f32bbe27640 1 -- 172.21.15.90:0/3509944741 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32b41a1ff0 con 0x7f32b4103910 2024-04-25T11:35:09.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.133+0000 7f32aaffd640 1 -- 172.21.15.90:0/3509944741 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32a40054d0 con 0x7f32b4103910 2024-04-25T11:35:09.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.133+0000 7f32aaffd640 1 -- 172.21.15.90:0/3509944741 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a4004c30 con 0x7f32b4103910 2024-04-25T11:35:09.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.133+0000 7f32bbe27640 1 -- 172.21.15.90:0/3509944741 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32b41a24e0 con 0x7f32b4103910 2024-04-25T11:35:09.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.134+0000 7f32bbe27640 1 -- 172.21.15.90:0/3509944741 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32b41078d0 con 0x7f32b4103910 2024-04-25T11:35:09.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.136+0000 7f32aaffd640 1 -- 172.21.15.90:0/3509944741 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 20) ==== 147205+0+0 (secure 0 0 0) 0x7f32a4004e10 con 0x7f32b4103910 2024-04-25T11:35:09.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.137+0000 7f32aaffd640 1 -- 172.21.15.90:0/3509944741 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f32a40de200 con 0x7f32b4103910 2024-04-25T11:35:09.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.143+0000 7f32aaffd640 1 -- 172.21.15.90:0/3509944741 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f32a40a6a70 con 0x7f32b4103910 2024-04-25T11:35:09.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.222+0000 7f32aaffd640 1 -- 172.21.15.90:0/3509944741 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 21) ==== 147332+0+0 (secure 0 0 0) 0x7f32a40a23e0 con 0x7f32b4103910 2024-04-25T11:35:09.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.223+0000 7f32aaffd640 1 --2- 172.21.15.90:0/3509944741 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f32a00b68e0 0x7f32a00b8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:09.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.226+0000 7f32b9bc3640 1 --2- 172.21.15.90:0/3509944741 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f32a00b68e0 0x7f32a00b8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:09.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.232+0000 7f32b9bc3640 1 --2- 172.21.15.90:0/3509944741 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f32a00b68e0 0x7f32a00b8d00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f32b0005ab0 tx=0x7f32b0031400 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:09.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.407+0000 7f32bbe27640 1 -- 172.21.15.90:0/3509944741 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32b402db50 con 0x7f32b4103910 2024-04-25T11:35:09.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.407+0000 7f32aaffd640 1 -- 172.21.15.90:0/3509944741 <== mon.0 v2:172.21.15.90:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11930 (secure 0 0 0) 0x7f32a4004c30 con 0x7f32b4103910 2024-04-25T11:35:09.397 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:35:09.397 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:35:06.229136+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24431,24458,24499,24524,24475]},"id":1}]} 2024-04-25T11:35:09.397 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 26 2024-04-25T11:35:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.409+0000 7f32bbe27640 1 -- 172.21.15.90:0/3509944741 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f32a00b68e0 msgr2=0x7f32a00b8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.409+0000 7f32bbe27640 1 --2- 172.21.15.90:0/3509944741 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f32a00b68e0 0x7f32a00b8d00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f32b0005ab0 tx=0x7f32b0031400 comp rx=0 tx=0).stop 2024-04-25T11:35:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.409+0000 7f32bbe27640 1 -- 172.21.15.90:0/3509944741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32b4103910 msgr2=0x7f32b419a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.409+0000 7f32bbe27640 1 --2- 172.21.15.90:0/3509944741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32b4103910 0x7f32b419a760 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f32a400d770 tx=0x7f32a400dc30 comp rx=0 tx=0).stop 2024-04-25T11:35:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.409+0000 7f32bbe27640 1 -- 172.21.15.90:0/3509944741 shutdown_connections 2024-04-25T11:35:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.409+0000 7f32bbe27640 1 --2- 172.21.15.90:0/3509944741 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f32a00b68e0 0x7f32a00b8d00 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.409+0000 7f32bbe27640 1 --2- 172.21.15.90:0/3509944741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32b4104450 0x7f32b41a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.410+0000 7f32bbe27640 1 --2- 172.21.15.90:0/3509944741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32b4103910 0x7f32b419a760 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.410+0000 7f32bbe27640 1 --2- 172.21.15.90:0/3509944741 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f32b4102f30 0x7f32b419a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.410+0000 7f32bbe27640 1 -- 172.21.15.90:0/3509944741 >> 172.21.15.90:0/3509944741 conn(0x7f32b40fd330 msgr2=0x7f32b4104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.410+0000 7f32bbe27640 1 -- 172.21.15.90:0/3509944741 shutdown_connections 2024-04-25T11:35:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.410+0000 7f32bbe27640 1 -- 172.21.15.90:0/3509944741 wait complete. 2024-04-25T11:35:09.410 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-25T11:35:09.410 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-25T11:35:09.411 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T11:35:09.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.535+0000 7fe75d030640 1 -- 172.21.15.90:0/2529250024 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe758109870 msgr2=0x7fe75810bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:09.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.535+0000 7fe75d030640 1 --2- 172.21.15.90:0/2529250024 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe758109870 0x7fe75810bc90 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe750009be0 tx=0x7fe75002a6c0 comp rx=0 tx=0).stop 2024-04-25T11:35:09.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.535+0000 7fe75d030640 1 -- 172.21.15.90:0/2529250024 shutdown_connections 2024-04-25T11:35:09.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.535+0000 7fe75d030640 1 --2- 172.21.15.90:0/2529250024 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe758109870 0x7fe75810bc90 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:09.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.535+0000 7fe75d030640 1 --2- 172.21.15.90:0/2529250024 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe758106f10 0x7fe758109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:09.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.535+0000 7fe75d030640 1 --2- 172.21.15.90:0/2529250024 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe7581045b0 0x7fe7581069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:09.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.535+0000 7fe75d030640 1 -- 172.21.15.90:0/2529250024 >> 172.21.15.90:0/2529250024 conn(0x7fe7580fdbd0 msgr2=0x7fe758100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:09.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.536+0000 7fe75d030640 1 -- 172.21.15.90:0/2529250024 shutdown_connections 2024-04-25T11:35:09.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.536+0000 7fe75d030640 1 -- 172.21.15.90:0/2529250024 wait complete. 2024-04-25T11:35:09.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.537+0000 7fe75d030640 1 Processor -- start 2024-04-25T11:35:09.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.537+0000 7fe75d030640 1 -- start start 2024-04-25T11:35:09.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.537+0000 7fe75d030640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7581045b0 0x7fe75819a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:09.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.537+0000 7fe75d030640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe758106f10 0x7fe75819a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:09.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.537+0000 7fe75d030640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe758109870 0x7fe7581a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:09.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.537+0000 7fe75d030640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe75810f5d0 con 0x7fe758109870 2024-04-25T11:35:09.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.537+0000 7fe75d030640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe75810f450 con 0x7fe758106f10 2024-04-25T11:35:09.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.537+0000 7fe75d030640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe75810f750 con 0x7fe7581045b0 2024-04-25T11:35:09.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.537+0000 7fe75759e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe758109870 0x7fe7581a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:09.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.537+0000 7fe75659c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe758106f10 0x7fe75819a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:09.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.537+0000 7fe75759e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe758109870 0x7fe7581a1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34366/0 (socket says 172.21.15.90:34366) 2024-04-25T11:35:09.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.537+0000 7fe75759e640 1 -- 172.21.15.90:0/1685999897 learned_addr learned my addr 172.21.15.90:0/1685999897 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:09.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.537+0000 7fe75659c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe758106f10 0x7fe75819a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:52010/0 (socket says 172.21.15.90:52010) 2024-04-25T11:35:09.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.537+0000 7fe756d9d640 1 --2- 172.21.15.90:0/1685999897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7581045b0 0x7fe75819a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:09.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.538+0000 7fe75659c640 1 -- 172.21.15.90:0/1685999897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7581045b0 msgr2=0x7fe75819a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:09.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.538+0000 7fe75659c640 1 --2- 172.21.15.90:0/1685999897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7581045b0 0x7fe75819a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:09.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.538+0000 7fe75659c640 1 -- 172.21.15.90:0/1685999897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe758109870 msgr2=0x7fe7581a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:09.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.538+0000 7fe75659c640 1 --2- 172.21.15.90:0/1685999897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe758109870 0x7fe7581a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:09.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.538+0000 7fe75659c640 1 -- 172.21.15.90:0/1685999897 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7581a1cf0 con 0x7fe758106f10 2024-04-25T11:35:09.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.538+0000 7fe75759e640 1 --2- 172.21.15.90:0/1685999897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe758109870 0x7fe7581a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:35:09.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.538+0000 7fe75659c640 1 --2- 172.21.15.90:0/1685999897 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe758106f10 0x7fe75819a6f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe74c00e9f0 tx=0x7fe74c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:09.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.538+0000 7fe73ffff640 1 -- 172.21.15.90:0/1685999897 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe74c00ccc0 con 0x7fe758106f10 2024-04-25T11:35:09.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.538+0000 7fe75d030640 1 -- 172.21.15.90:0/1685999897 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7581a1fe0 con 0x7fe758106f10 2024-04-25T11:35:09.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.538+0000 7fe73ffff640 1 -- 172.21.15.90:0/1685999897 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe74c0108e0 con 0x7fe758106f10 2024-04-25T11:35:09.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.538+0000 7fe73ffff640 1 -- 172.21.15.90:0/1685999897 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe74c018880 con 0x7fe758106f10 2024-04-25T11:35:09.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.538+0000 7fe75d030640 1 -- 172.21.15.90:0/1685999897 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7581a24a0 con 0x7fe758106f10 2024-04-25T11:35:09.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.539+0000 7fe75d030640 1 -- 172.21.15.90:0/1685999897 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7580ff6f0 con 0x7fe758106f10 2024-04-25T11:35:09.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.541+0000 7fe73ffff640 1 -- 172.21.15.90:0/1685999897 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 21) ==== 147332+0+0 (secure 0 0 0) 0x7fe74c010430 con 0x7fe758106f10 2024-04-25T11:35:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.542+0000 7fe73ffff640 1 --2- 172.21.15.90:0/1685999897 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe7400af8e0 0x7fe7400b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.542+0000 7fe756d9d640 1 --2- 172.21.15.90:0/1685999897 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe7400af8e0 0x7fe7400b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.542+0000 7fe73ffff640 1 -- 172.21.15.90:0/1685999897 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7fe74c0df0f0 con 0x7fe758106f10 2024-04-25T11:35:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.542+0000 7fe756d9d640 1 --2- 172.21.15.90:0/1685999897 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe7400af8e0 0x7fe7400b1df0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe748009530 tx=0x7fe748009260 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:09.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.550+0000 7fe73ffff640 1 -- 172.21.15.90:0/1685999897 <== mon.2 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe74c0a7810 con 0x7fe758106f10 2024-04-25T11:35:09.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:09 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T11:35:09.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:09 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T11:35:09.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:09 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/690826379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:35:09.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:09 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2324453382' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:35:09.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:09 smithi138 ceph-mon[32328]: mgrmap e21: x(active, since 1.02433s), standbys: z, y 2024-04-25T11:35:09.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:09 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T11:35:09.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:09 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T11:35:09.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:09 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/690826379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:35:09.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:09 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2324453382' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:35:09.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:09 smithi156 ceph-mon[33045]: mgrmap e21: x(active, since 1.02433s), standbys: z, y 2024-04-25T11:35:09.685 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:09 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T11:35:09.686 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:09 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T11:35:09.686 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:09 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/690826379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:35:09.686 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:09 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2324453382' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:35:09.686 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:09 smithi090 ceph-mon[29575]: mgrmap e21: x(active, since 1.02433s), standbys: z, y 2024-04-25T11:35:09.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.733+0000 7fe75d030640 1 -- 172.21.15.90:0/1685999897 --> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe758062fe0 con 0x7fe7400af8e0 2024-04-25T11:35:09.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.741+0000 7fe73ffff640 1 -- 172.21.15.90:0/1685999897 <== mgr.15366 v2:172.21.15.90:6800/4078377640 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257767 (secure 0 0 0) 0x7fe758062fe0 con 0x7fe7400af8e0 2024-04-25T11:35:09.733 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:35:09.734 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-25T11:35:09.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.743+0000 7fe75d030640 1 -- 172.21.15.90:0/1685999897 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe7400af8e0 msgr2=0x7fe7400b1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:09.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.743+0000 7fe75d030640 1 --2- 172.21.15.90:0/1685999897 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe7400af8e0 0x7fe7400b1df0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe748009530 tx=0x7fe748009260 comp rx=0 tx=0).stop 2024-04-25T11:35:09.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.743+0000 7fe75d030640 1 -- 172.21.15.90:0/1685999897 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe758106f10 msgr2=0x7fe75819a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:09.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.743+0000 7fe75d030640 1 --2- 172.21.15.90:0/1685999897 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe758106f10 0x7fe75819a6f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe74c00e9f0 tx=0x7fe74c00eeb0 comp rx=0 tx=0).stop 2024-04-25T11:35:09.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.743+0000 7fe75d030640 1 -- 172.21.15.90:0/1685999897 shutdown_connections 2024-04-25T11:35:09.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.743+0000 7fe75d030640 1 --2- 172.21.15.90:0/1685999897 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe7400af8e0 0x7fe7400b1df0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:09.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.743+0000 7fe75d030640 1 --2- 172.21.15.90:0/1685999897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe758109870 0x7fe7581a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:09.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.743+0000 7fe75d030640 1 --2- 172.21.15.90:0/1685999897 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe758106f10 0x7fe75819a6f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:09.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.743+0000 7fe75d030640 1 --2- 172.21.15.90:0/1685999897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7581045b0 0x7fe75819a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:09.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.743+0000 7fe75d030640 1 -- 172.21.15.90:0/1685999897 >> 172.21.15.90:0/1685999897 conn(0x7fe7580fdbd0 msgr2=0x7fe7581054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:09.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.744+0000 7fe75d030640 1 -- 172.21.15.90:0/1685999897 shutdown_connections 2024-04-25T11:35:09.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.744+0000 7fe75d030640 1 -- 172.21.15.90:0/1685999897 wait complete. 2024-04-25T11:35:09.744 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":3,"stamp":"2024-04-25T11:35:09.241136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1192393,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1353,"num_read_kb":816,"num_write":495,"num_write_kb":3616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":3012886,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":893,"ondisk_log_size":893,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":392232,"kb_used_data":13932,"kb_used_omap":18,"kb_used_meta":378157,"kb_avail":1124549592,"statfs":{"total":1151940427776,"available":1151538782208,"internally_reserved":0,"allocated":14266368,"data_stored":8475001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":387233081},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"3.3f","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.505112+0000","last_change":"2024-04-25T11:33:10.892144+0000","last_active":"2024-04-25T11:35:08.505112+0000","last_peered":"2024-04-25T11:35:08.505112+0000","last_clean":"2024-04-25T11:35:08.505112+0000","last_became_active":"2024-04-25T11:33:10.891721+0000","last_became_peered":"2024-04-25T11:33:10.891721+0000","last_unstale":"2024-04-25T11:35:08.505112+0000","last_undegraded":"2024-04-25T11:35:08.505112+0000","last_fullsized":"2024-04-25T11:35:08.505112+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:58:25.447086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"79'4","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.502017+0000","last_change":"2024-04-25T11:33:10.891647+0000","last_active":"2024-04-25T11:35:08.502017+0000","last_peered":"2024-04-25T11:35:08.502017+0000","last_clean":"2024-04-25T11:35:08.502017+0000","last_became_active":"2024-04-25T11:33:10.891283+0000","last_became_peered":"2024-04-25T11:33:10.891283+0000","last_unstale":"2024-04-25T11:35:08.502017+0000","last_undegraded":"2024-04-25T11:35:08.502017+0000","last_fullsized":"2024-04-25T11:35:08.502017+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:12.669438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501829+0000","last_change":"2024-04-25T11:33:10.890928+0000","last_active":"2024-04-25T11:35:08.501829+0000","last_peered":"2024-04-25T11:35:08.501829+0000","last_clean":"2024-04-25T11:35:08.501829+0000","last_became_active":"2024-04-25T11:33:10.889093+0000","last_became_peered":"2024-04-25T11:33:10.889093+0000","last_unstale":"2024-04-25T11:35:08.501829+0000","last_undegraded":"2024-04-25T11:35:08.501829+0000","last_fullsized":"2024-04-25T11:35:08.501829+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:28:09.038650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501058+0000","last_change":"2024-04-25T11:33:10.890184+0000","last_active":"2024-04-25T11:35:08.501058+0000","last_peered":"2024-04-25T11:35:08.501058+0000","last_clean":"2024-04-25T11:35:08.501058+0000","last_became_active":"2024-04-25T11:33:10.889367+0000","last_became_peered":"2024-04-25T11:33:10.889367+0000","last_unstale":"2024-04-25T11:35:08.501058+0000","last_undegraded":"2024-04-25T11:35:08.501058+0000","last_fullsized":"2024-04-25T11:35:08.501058+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:42.798295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"79'4","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.208757+0000","last_change":"2024-04-25T11:33:10.889758+0000","last_active":"2024-04-25T11:35:08.208757+0000","last_peered":"2024-04-25T11:35:08.208757+0000","last_clean":"2024-04-25T11:35:08.208757+0000","last_became_active":"2024-04-25T11:33:10.889384+0000","last_became_peered":"2024-04-25T11:33:10.889384+0000","last_unstale":"2024-04-25T11:35:08.208757+0000","last_undegraded":"2024-04-25T11:35:08.208757+0000","last_fullsized":"2024-04-25T11:35:08.208757+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:54:38.794216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"79'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.210810+0000","last_change":"2024-04-25T11:33:10.889348+0000","last_active":"2024-04-25T11:35:08.210810+0000","last_peered":"2024-04-25T11:35:08.210810+0000","last_clean":"2024-04-25T11:35:08.210810+0000","last_became_active":"2024-04-25T11:33:10.888908+0000","last_became_peered":"2024-04-25T11:33:10.888908+0000","last_unstale":"2024-04-25T11:35:08.210810+0000","last_undegraded":"2024-04-25T11:35:08.210810+0000","last_fullsized":"2024-04-25T11:35:08.210810+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:02:59.555921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"79'5","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500248+0000","last_change":"2024-04-25T11:34:18.237661+0000","last_active":"2024-04-25T11:35:08.500248+0000","last_peered":"2024-04-25T11:35:08.500248+0000","last_clean":"2024-04-25T11:35:08.500248+0000","last_became_active":"2024-04-25T11:34:18.237306+0000","last_became_peered":"2024-04-25T11:34:18.237306+0000","last_unstale":"2024-04-25T11:35:08.500248+0000","last_undegraded":"2024-04-25T11:35:08.500248+0000","last_fullsized":"2024-04-25T11:35:08.500248+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:17:34.228368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.317192+0000","last_change":"2024-04-25T11:33:10.887262+0000","last_active":"2024-04-25T11:35:08.317192+0000","last_peered":"2024-04-25T11:35:08.317192+0000","last_clean":"2024-04-25T11:35:08.317192+0000","last_became_active":"2024-04-25T11:33:10.886948+0000","last_became_peered":"2024-04-25T11:33:10.886948+0000","last_unstale":"2024-04-25T11:35:08.317192+0000","last_undegraded":"2024-04-25T11:35:08.317192+0000","last_fullsized":"2024-04-25T11:35:08.317192+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:09.721506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"79'11","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.314351+0000","last_change":"2024-04-25T11:34:17.883253+0000","last_active":"2024-04-25T11:35:08.314351+0000","last_peered":"2024-04-25T11:35:08.314351+0000","last_clean":"2024-04-25T11:35:08.314351+0000","last_became_active":"2024-04-25T11:34:17.882515+0000","last_became_peered":"2024-04-25T11:34:17.882515+0000","last_unstale":"2024-04-25T11:35:08.314351+0000","last_undegraded":"2024-04-25T11:35:08.314351+0000","last_fullsized":"2024-04-25T11:35:08.314351+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:52:32.813464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.36","version":"79'16","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.211225+0000","last_change":"2024-04-25T11:33:10.889812+0000","last_active":"2024-04-25T11:35:08.211225+0000","last_peered":"2024-04-25T11:35:08.211225+0000","last_clean":"2024-04-25T11:35:08.211225+0000","last_became_active":"2024-04-25T11:33:10.889369+0000","last_became_peered":"2024-04-25T11:33:10.889369+0000","last_unstale":"2024-04-25T11:35:08.211225+0000","last_undegraded":"2024-04-25T11:35:08.211225+0000","last_fullsized":"2024-04-25T11:35:08.211225+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:42:14.443496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"79'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501487+0000","last_change":"2024-04-25T11:33:10.891186+0000","last_active":"2024-04-25T11:35:08.501487+0000","last_peered":"2024-04-25T11:35:08.501487+0000","last_clean":"2024-04-25T11:35:08.501487+0000","last_became_active":"2024-04-25T11:33:10.890841+0000","last_became_peered":"2024-04-25T11:33:10.890841+0000","last_unstale":"2024-04-25T11:35:08.501487+0000","last_undegraded":"2024-04-25T11:35:08.501487+0000","last_fullsized":"2024-04-25T11:35:08.501487+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:24:56.078592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501001+0000","last_change":"2024-04-25T11:33:10.888801+0000","last_active":"2024-04-25T11:35:08.501001+0000","last_peered":"2024-04-25T11:35:08.501001+0000","last_clean":"2024-04-25T11:35:08.501001+0000","last_became_active":"2024-04-25T11:33:10.888437+0000","last_became_peered":"2024-04-25T11:33:10.888437+0000","last_unstale":"2024-04-25T11:35:08.501001+0000","last_undegraded":"2024-04-25T11:35:08.501001+0000","last_fullsized":"2024-04-25T11:35:08.501001+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:14:14.011475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"79'10","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207334+0000","last_change":"2024-04-25T11:33:10.892749+0000","last_active":"2024-04-25T11:35:08.207334+0000","last_peered":"2024-04-25T11:35:08.207334+0000","last_clean":"2024-04-25T11:35:08.207334+0000","last_became_active":"2024-04-25T11:33:10.892370+0000","last_became_peered":"2024-04-25T11:33:10.892370+0000","last_unstale":"2024-04-25T11:35:08.207334+0000","last_undegraded":"2024-04-25T11:35:08.207334+0000","last_fullsized":"2024-04-25T11:35:08.207334+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:00:51.312387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500624+0000","last_change":"2024-04-25T11:33:10.888981+0000","last_active":"2024-04-25T11:35:08.500624+0000","last_peered":"2024-04-25T11:35:08.500624+0000","last_clean":"2024-04-25T11:35:08.500624+0000","last_became_active":"2024-04-25T11:33:10.888537+0000","last_became_peered":"2024-04-25T11:33:10.888537+0000","last_unstale":"2024-04-25T11:35:08.500624+0000","last_undegraded":"2024-04-25T11:35:08.500624+0000","last_fullsized":"2024-04-25T11:35:08.500624+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:12:19.018698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501399+0000","last_change":"2024-04-25T11:33:10.891455+0000","last_active":"2024-04-25T11:35:08.501399+0000","last_peered":"2024-04-25T11:35:08.501399+0000","last_clean":"2024-04-25T11:35:08.501399+0000","last_became_active":"2024-04-25T11:33:10.891037+0000","last_became_peered":"2024-04-25T11:33:10.891037+0000","last_unstale":"2024-04-25T11:35:08.501399+0000","last_undegraded":"2024-04-25T11:35:08.501399+0000","last_fullsized":"2024-04-25T11:35:08.501399+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:44.470674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.380415+0000","last_change":"2024-04-25T11:33:10.895309+0000","last_active":"2024-04-25T11:35:08.380415+0000","last_peered":"2024-04-25T11:35:08.380415+0000","last_clean":"2024-04-25T11:35:08.380415+0000","last_became_active":"2024-04-25T11:33:10.894952+0000","last_became_peered":"2024-04-25T11:33:10.894952+0000","last_unstale":"2024-04-25T11:35:08.380415+0000","last_undegraded":"2024-04-25T11:35:08.380415+0000","last_fullsized":"2024-04-25T11:35:08.380415+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:29:22.119017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.505207+0000","last_change":"2024-04-25T11:33:10.894524+0000","last_active":"2024-04-25T11:35:08.505207+0000","last_peered":"2024-04-25T11:35:08.505207+0000","last_clean":"2024-04-25T11:35:08.505207+0000","last_became_active":"2024-04-25T11:33:10.894220+0000","last_became_peered":"2024-04-25T11:33:10.894220+0000","last_unstale":"2024-04-25T11:35:08.505207+0000","last_undegraded":"2024-04-25T11:35:08.505207+0000","last_fullsized":"2024-04-25T11:35:08.505207+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:10:29.072976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"79'10","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.315017+0000","last_change":"2024-04-25T11:33:10.886814+0000","last_active":"2024-04-25T11:35:08.315017+0000","last_peered":"2024-04-25T11:35:08.315017+0000","last_clean":"2024-04-25T11:35:08.315017+0000","last_became_active":"2024-04-25T11:33:10.886494+0000","last_became_peered":"2024-04-25T11:33:10.886494+0000","last_unstale":"2024-04-25T11:35:08.315017+0000","last_undegraded":"2024-04-25T11:35:08.315017+0000","last_fullsized":"2024-04-25T11:35:08.315017+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:03:43.054485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500081+0000","last_change":"2024-04-25T11:33:10.886870+0000","last_active":"2024-04-25T11:35:08.500081+0000","last_peered":"2024-04-25T11:35:08.500081+0000","last_clean":"2024-04-25T11:35:08.500081+0000","last_became_active":"2024-04-25T11:33:10.886507+0000","last_became_peered":"2024-04-25T11:33:10.886507+0000","last_unstale":"2024-04-25T11:35:08.500081+0000","last_undegraded":"2024-04-25T11:35:08.500081+0000","last_fullsized":"2024-04-25T11:35:08.500081+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:50.101760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"77'13","reported_seq":148,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.314751+0000","last_change":"2024-04-25T11:33:10.028970+0000","last_active":"2024-04-25T11:35:08.314751+0000","last_peered":"2024-04-25T11:35:08.314751+0000","last_clean":"2024-04-25T11:35:08.314751+0000","last_became_active":"2024-04-25T11:33:10.028636+0000","last_became_peered":"2024-04-25T11:33:10.028636+0000","last_unstale":"2024-04-25T11:35:08.314751+0000","last_undegraded":"2024-04-25T11:35:08.314751+0000","last_fullsized":"2024-04-25T11:35:08.314751+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:02.414903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"79'5","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.314684+0000","last_change":"2024-04-25T11:34:17.881522+0000","last_active":"2024-04-25T11:35:08.314684+0000","last_peered":"2024-04-25T11:35:08.314684+0000","last_clean":"2024-04-25T11:35:08.314684+0000","last_became_active":"2024-04-25T11:34:17.881114+0000","last_became_peered":"2024-04-25T11:34:17.881114+0000","last_unstale":"2024-04-25T11:35:08.314684+0000","last_undegraded":"2024-04-25T11:35:08.314684+0000","last_fullsized":"2024-04-25T11:35:08.314684+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:59:03.675054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"77'2","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.317914+0000","last_change":"2024-04-25T11:33:10.036958+0000","last_active":"2024-04-25T11:35:08.317914+0000","last_peered":"2024-04-25T11:35:08.317914+0000","last_clean":"2024-04-25T11:35:08.317914+0000","last_became_active":"2024-04-25T11:33:10.036606+0000","last_became_peered":"2024-04-25T11:33:10.036606+0000","last_unstale":"2024-04-25T11:35:08.317914+0000","last_undegraded":"2024-04-25T11:35:08.317914+0000","last_fullsized":"2024-04-25T11:35:08.317914+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:12.660208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"79'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.502299+0000","last_change":"2024-04-25T11:33:10.887501+0000","last_active":"2024-04-25T11:35:08.502299+0000","last_peered":"2024-04-25T11:35:08.502299+0000","last_clean":"2024-04-25T11:35:08.502299+0000","last_became_active":"2024-04-25T11:33:10.885985+0000","last_became_peered":"2024-04-25T11:33:10.885985+0000","last_unstale":"2024-04-25T11:35:08.502299+0000","last_undegraded":"2024-04-25T11:35:08.502299+0000","last_fullsized":"2024-04-25T11:35:08.502299+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:31:32.035646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"77'12","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.209236+0000","last_change":"2024-04-25T11:33:10.031161+0000","last_active":"2024-04-25T11:35:08.209236+0000","last_peered":"2024-04-25T11:35:08.209236+0000","last_clean":"2024-04-25T11:35:08.209236+0000","last_became_active":"2024-04-25T11:33:10.028891+0000","last_became_peered":"2024-04-25T11:33:10.028891+0000","last_unstale":"2024-04-25T11:35:08.209236+0000","last_undegraded":"2024-04-25T11:35:08.209236+0000","last_fullsized":"2024-04-25T11:35:08.209236+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:25.959699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.380513+0000","last_change":"2024-04-25T11:33:10.889439+0000","last_active":"2024-04-25T11:35:08.380513+0000","last_peered":"2024-04-25T11:35:08.380513+0000","last_clean":"2024-04-25T11:35:08.380513+0000","last_became_active":"2024-04-25T11:33:10.889082+0000","last_became_peered":"2024-04-25T11:33:10.889082+0000","last_unstale":"2024-04-25T11:35:08.380513+0000","last_undegraded":"2024-04-25T11:35:08.380513+0000","last_fullsized":"2024-04-25T11:35:08.380513+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:13:48.958179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"77'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.208951+0000","last_change":"2024-04-25T11:33:10.030793+0000","last_active":"2024-04-25T11:35:08.208951+0000","last_peered":"2024-04-25T11:35:08.208951+0000","last_clean":"2024-04-25T11:35:08.208951+0000","last_became_active":"2024-04-25T11:33:10.028805+0000","last_became_peered":"2024-04-25T11:33:10.028805+0000","last_unstale":"2024-04-25T11:35:08.208951+0000","last_undegraded":"2024-04-25T11:35:08.208951+0000","last_fullsized":"2024-04-25T11:35:08.208951+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:36.233194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"79'15","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.324396+0000","last_change":"2024-04-25T11:33:10.894815+0000","last_active":"2024-04-25T11:35:08.324396+0000","last_peered":"2024-04-25T11:35:08.324396+0000","last_clean":"2024-04-25T11:35:08.324396+0000","last_became_active":"2024-04-25T11:33:10.892174+0000","last_became_peered":"2024-04-25T11:33:10.892174+0000","last_unstale":"2024-04-25T11:35:08.324396+0000","last_undegraded":"2024-04-25T11:35:08.324396+0000","last_fullsized":"2024-04-25T11:35:08.324396+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:05.368441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"80'8","reported_seq":149,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.802370+0000","last_change":"2024-04-25T11:33:10.031238+0000","last_active":"2024-04-25T11:35:08.802370+0000","last_peered":"2024-04-25T11:35:08.802370+0000","last_clean":"2024-04-25T11:35:08.802370+0000","last_became_active":"2024-04-25T11:33:10.030917+0000","last_became_peered":"2024-04-25T11:33:10.030917+0000","last_unstale":"2024-04-25T11:35:08.802370+0000","last_undegraded":"2024-04-25T11:35:08.802370+0000","last_fullsized":"2024-04-25T11:35:08.802370+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:00:35.634293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1712,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":120,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.208813+0000","last_change":"2024-04-25T11:33:10.896974+0000","last_active":"2024-04-25T11:35:08.208813+0000","last_peered":"2024-04-25T11:35:08.208813+0000","last_clean":"2024-04-25T11:35:08.208813+0000","last_became_active":"2024-04-25T11:33:10.896575+0000","last_became_peered":"2024-04-25T11:33:10.896575+0000","last_unstale":"2024-04-25T11:35:08.208813+0000","last_undegraded":"2024-04-25T11:35:08.208813+0000","last_fullsized":"2024-04-25T11:35:08.208813+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:32:43.533652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"77'7","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.210783+0000","last_change":"2024-04-25T11:33:10.022714+0000","last_active":"2024-04-25T11:35:08.210783+0000","last_peered":"2024-04-25T11:35:08.210783+0000","last_clean":"2024-04-25T11:35:08.210783+0000","last_became_active":"2024-04-25T11:33:10.022230+0000","last_became_peered":"2024-04-25T11:33:10.022230+0000","last_unstale":"2024-04-25T11:35:08.210783+0000","last_undegraded":"2024-04-25T11:35:08.210783+0000","last_fullsized":"2024-04-25T11:35:08.210783+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:30:00.620833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.324208+0000","last_change":"2024-04-25T11:33:10.897664+0000","last_active":"2024-04-25T11:35:08.324208+0000","last_peered":"2024-04-25T11:35:08.324208+0000","last_clean":"2024-04-25T11:35:08.324208+0000","last_became_active":"2024-04-25T11:33:10.897189+0000","last_became_peered":"2024-04-25T11:33:10.897189+0000","last_unstale":"2024-04-25T11:35:08.324208+0000","last_undegraded":"2024-04-25T11:35:08.324208+0000","last_fullsized":"2024-04-25T11:35:08.324208+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:12:06.168793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"77'3","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207308+0000","last_change":"2024-04-25T11:33:10.031417+0000","last_active":"2024-04-25T11:35:08.207308+0000","last_peered":"2024-04-25T11:35:08.207308+0000","last_clean":"2024-04-25T11:35:08.207308+0000","last_became_active":"2024-04-25T11:33:10.030526+0000","last_became_peered":"2024-04-25T11:33:10.030526+0000","last_unstale":"2024-04-25T11:35:08.207308+0000","last_undegraded":"2024-04-25T11:35:08.207308+0000","last_fullsized":"2024-04-25T11:35:08.207308+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:05.718720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.317017+0000","last_change":"2024-04-25T11:33:10.886596+0000","last_active":"2024-04-25T11:35:08.317017+0000","last_peered":"2024-04-25T11:35:08.317017+0000","last_clean":"2024-04-25T11:35:08.317017+0000","last_became_active":"2024-04-25T11:33:10.886234+0000","last_became_peered":"2024-04-25T11:33:10.886234+0000","last_unstale":"2024-04-25T11:35:08.317017+0000","last_undegraded":"2024-04-25T11:35:08.317017+0000","last_fullsized":"2024-04-25T11:35:08.317017+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:46.632814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"77'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.324040+0000","last_change":"2024-04-25T11:33:10.032355+0000","last_active":"2024-04-25T11:35:08.324040+0000","last_peered":"2024-04-25T11:35:08.324040+0000","last_clean":"2024-04-25T11:35:08.324040+0000","last_became_active":"2024-04-25T11:33:10.031965+0000","last_became_peered":"2024-04-25T11:33:10.031965+0000","last_unstale":"2024-04-25T11:35:08.324040+0000","last_undegraded":"2024-04-25T11:35:08.324040+0000","last_fullsized":"2024-04-25T11:35:08.324040+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:08:10.539604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"79'11","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.314255+0000","last_change":"2024-04-25T11:33:10.885708+0000","last_active":"2024-04-25T11:35:08.314255+0000","last_peered":"2024-04-25T11:35:08.314255+0000","last_clean":"2024-04-25T11:35:08.314255+0000","last_became_active":"2024-04-25T11:33:10.885317+0000","last_became_peered":"2024-04-25T11:33:10.885317+0000","last_unstale":"2024-04-25T11:35:08.314255+0000","last_undegraded":"2024-04-25T11:35:08.314255+0000","last_fullsized":"2024-04-25T11:35:08.314255+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:50:08.518922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"80'6","reported_seq":256,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.801798+0000","last_change":"2024-04-25T11:33:10.030995+0000","last_active":"2024-04-25T11:35:08.801798+0000","last_peered":"2024-04-25T11:35:08.801798+0000","last_clean":"2024-04-25T11:35:08.801798+0000","last_became_active":"2024-04-25T11:33:10.030670+0000","last_became_peered":"2024-04-25T11:33:10.030670+0000","last_unstale":"2024-04-25T11:35:08.801798+0000","last_undegraded":"2024-04-25T11:35:08.801798+0000","last_fullsized":"2024-04-25T11:35:08.801798+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:39.569787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":229,"num_read_kb":229,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"79'10","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.380745+0000","last_change":"2024-04-25T11:33:10.892927+0000","last_active":"2024-04-25T11:35:08.380745+0000","last_peered":"2024-04-25T11:35:08.380745+0000","last_clean":"2024-04-25T11:35:08.380745+0000","last_became_active":"2024-04-25T11:33:10.892458+0000","last_became_peered":"2024-04-25T11:33:10.892458+0000","last_unstale":"2024-04-25T11:35:08.380745+0000","last_undegraded":"2024-04-25T11:35:08.380745+0000","last_fullsized":"2024-04-25T11:35:08.380745+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:53.121185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"77'4","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501417+0000","last_change":"2024-04-25T11:33:10.029728+0000","last_active":"2024-04-25T11:35:08.501417+0000","last_peered":"2024-04-25T11:35:08.501417+0000","last_clean":"2024-04-25T11:35:08.501417+0000","last_became_active":"2024-04-25T11:33:10.029435+0000","last_became_peered":"2024-04-25T11:33:10.029435+0000","last_unstale":"2024-04-25T11:35:08.501417+0000","last_undegraded":"2024-04-25T11:35:08.501417+0000","last_fullsized":"2024-04-25T11:35:08.501417+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:31:21.585870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.502258+0000","last_change":"2024-04-25T11:33:10.895828+0000","last_active":"2024-04-25T11:35:08.502258+0000","last_peered":"2024-04-25T11:35:08.502258+0000","last_clean":"2024-04-25T11:35:08.502258+0000","last_became_active":"2024-04-25T11:33:10.895280+0000","last_became_peered":"2024-04-25T11:33:10.895280+0000","last_unstale":"2024-04-25T11:35:08.502258+0000","last_undegraded":"2024-04-25T11:35:08.502258+0000","last_fullsized":"2024-04-25T11:35:08.502258+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:03.009912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"77'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.324544+0000","last_change":"2024-04-25T11:33:10.036651+0000","last_active":"2024-04-25T11:35:08.324544+0000","last_peered":"2024-04-25T11:35:08.324544+0000","last_clean":"2024-04-25T11:35:08.324544+0000","last_became_active":"2024-04-25T11:33:10.036294+0000","last_became_peered":"2024-04-25T11:33:10.036294+0000","last_unstale":"2024-04-25T11:35:08.324544+0000","last_undegraded":"2024-04-25T11:35:08.324544+0000","last_fullsized":"2024-04-25T11:35:08.324544+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:12:36.566618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"79'11","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.380605+0000","last_change":"2024-04-25T11:33:10.890199+0000","last_active":"2024-04-25T11:35:08.380605+0000","last_peered":"2024-04-25T11:35:08.380605+0000","last_clean":"2024-04-25T11:35:08.380605+0000","last_became_active":"2024-04-25T11:33:10.889855+0000","last_became_peered":"2024-04-25T11:33:10.889855+0000","last_unstale":"2024-04-25T11:35:08.380605+0000","last_undegraded":"2024-04-25T11:35:08.380605+0000","last_fullsized":"2024-04-25T11:35:08.380605+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:54:45.476647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'37","reported_seq":319,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.511117+0000","last_change":"2024-04-25T11:34:18.309247+0000","last_active":"2024-04-25T11:35:08.511117+0000","last_peered":"2024-04-25T11:35:08.511117+0000","last_clean":"2024-04-25T11:35:08.511117+0000","last_became_active":"2024-04-25T11:34:18.299888+0000","last_became_peered":"2024-04-25T11:34:18.299888+0000","last_unstale":"2024-04-25T11:35:08.511117+0000","last_undegraded":"2024-04-25T11:35:08.511117+0000","last_fullsized":"2024-04-25T11:35:08.511117+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:36:06.603720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67758,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":235,"num_read_kb":178,"num_write":39,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"79'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.208993+0000","last_change":"2024-04-25T11:33:10.888596+0000","last_active":"2024-04-25T11:35:08.208993+0000","last_peered":"2024-04-25T11:35:08.208993+0000","last_clean":"2024-04-25T11:35:08.208993+0000","last_became_active":"2024-04-25T11:33:10.888226+0000","last_became_peered":"2024-04-25T11:33:10.888226+0000","last_unstale":"2024-04-25T11:35:08.208993+0000","last_undegraded":"2024-04-25T11:35:08.208993+0000","last_fullsized":"2024-04-25T11:35:08.208993+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:30:42.891788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"77'9","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500117+0000","last_change":"2024-04-25T11:33:10.026969+0000","last_active":"2024-04-25T11:35:08.500117+0000","last_peered":"2024-04-25T11:35:08.500117+0000","last_clean":"2024-04-25T11:35:08.500117+0000","last_became_active":"2024-04-25T11:33:10.026591+0000","last_became_peered":"2024-04-25T11:33:10.026591+0000","last_unstale":"2024-04-25T11:35:08.500117+0000","last_undegraded":"2024-04-25T11:35:08.500117+0000","last_fullsized":"2024-04-25T11:35:08.500117+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:50:34.269478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207191+0000","last_change":"2024-04-25T11:33:10.900303+0000","last_active":"2024-04-25T11:35:08.207191+0000","last_peered":"2024-04-25T11:35:08.207191+0000","last_clean":"2024-04-25T11:35:08.207191+0000","last_became_active":"2024-04-25T11:33:10.899946+0000","last_became_peered":"2024-04-25T11:33:10.899946+0000","last_unstale":"2024-04-25T11:35:08.207191+0000","last_undegraded":"2024-04-25T11:35:08.207191+0000","last_fullsized":"2024-04-25T11:35:08.207191+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:33:40.805400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"77'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.324114+0000","last_change":"2024-04-25T11:33:10.035668+0000","last_active":"2024-04-25T11:35:08.324114+0000","last_peered":"2024-04-25T11:35:08.324114+0000","last_clean":"2024-04-25T11:35:08.324114+0000","last_became_active":"2024-04-25T11:33:10.035241+0000","last_became_peered":"2024-04-25T11:33:10.035241+0000","last_unstale":"2024-04-25T11:35:08.324114+0000","last_undegraded":"2024-04-25T11:35:08.324114+0000","last_fullsized":"2024-04-25T11:35:08.324114+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:58:48.735577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"79'15","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500819+0000","last_change":"2024-04-25T11:33:10.894864+0000","last_active":"2024-04-25T11:35:08.500819+0000","last_peered":"2024-04-25T11:35:08.500819+0000","last_clean":"2024-04-25T11:35:08.500819+0000","last_became_active":"2024-04-25T11:33:10.894491+0000","last_became_peered":"2024-04-25T11:33:10.894491+0000","last_unstale":"2024-04-25T11:35:08.500819+0000","last_undegraded":"2024-04-25T11:35:08.500819+0000","last_fullsized":"2024-04-25T11:35:08.500819+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:21:28.256878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"80'8","reported_seq":146,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.816304+0000","last_change":"2024-04-25T11:33:10.029099+0000","last_active":"2024-04-25T11:35:08.816304+0000","last_peered":"2024-04-25T11:35:08.816304+0000","last_clean":"2024-04-25T11:35:08.816304+0000","last_became_active":"2024-04-25T11:33:10.028722+0000","last_became_peered":"2024-04-25T11:33:10.028722+0000","last_unstale":"2024-04-25T11:35:08.816304+0000","last_undegraded":"2024-04-25T11:35:08.816304+0000","last_fullsized":"2024-04-25T11:35:08.816304+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:44:12.261519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1511,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"79'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500425+0000","last_change":"2024-04-25T11:33:10.888635+0000","last_active":"2024-04-25T11:35:08.500425+0000","last_peered":"2024-04-25T11:35:08.500425+0000","last_clean":"2024-04-25T11:35:08.500425+0000","last_became_active":"2024-04-25T11:33:10.887544+0000","last_became_peered":"2024-04-25T11:33:10.887544+0000","last_unstale":"2024-04-25T11:35:08.500425+0000","last_undegraded":"2024-04-25T11:35:08.500425+0000","last_fullsized":"2024-04-25T11:35:08.500425+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:59:25.459587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"77'17","reported_seq":39,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.208662+0000","last_change":"2024-04-25T11:33:10.023566+0000","last_active":"2024-04-25T11:35:08.208662+0000","last_peered":"2024-04-25T11:35:08.208662+0000","last_clean":"2024-04-25T11:35:08.208662+0000","last_became_active":"2024-04-25T11:33:10.023234+0000","last_became_peered":"2024-04-25T11:33:10.023234+0000","last_unstale":"2024-04-25T11:35:08.208662+0000","last_undegraded":"2024-04-25T11:35:08.208662+0000","last_fullsized":"2024-04-25T11:35:08.208662+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:56:26.342148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"79'6","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.324020+0000","last_change":"2024-04-25T11:34:18.237663+0000","last_active":"2024-04-25T11:35:08.324020+0000","last_peered":"2024-04-25T11:35:08.324020+0000","last_clean":"2024-04-25T11:35:08.324020+0000","last_became_active":"2024-04-25T11:34:18.236993+0000","last_became_peered":"2024-04-25T11:34:18.236993+0000","last_unstale":"2024-04-25T11:35:08.324020+0000","last_undegraded":"2024-04-25T11:35:08.324020+0000","last_fullsized":"2024-04-25T11:35:08.324020+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:59:47.371685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.209203+0000","last_change":"2024-04-25T11:33:10.028285+0000","last_active":"2024-04-25T11:35:08.209203+0000","last_peered":"2024-04-25T11:35:08.209203+0000","last_clean":"2024-04-25T11:35:08.209203+0000","last_became_active":"2024-04-25T11:33:10.027959+0000","last_became_peered":"2024-04-25T11:33:10.027959+0000","last_unstale":"2024-04-25T11:35:08.209203+0000","last_undegraded":"2024-04-25T11:35:08.209203+0000","last_fullsized":"2024-04-25T11:35:08.209203+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:44:41.094048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"79'5","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.505018+0000","last_change":"2024-04-25T11:34:17.887356+0000","last_active":"2024-04-25T11:35:08.505018+0000","last_peered":"2024-04-25T11:35:08.505018+0000","last_clean":"2024-04-25T11:35:08.505018+0000","last_became_active":"2024-04-25T11:34:17.886981+0000","last_became_peered":"2024-04-25T11:34:17.886981+0000","last_unstale":"2024-04-25T11:35:08.505018+0000","last_undegraded":"2024-04-25T11:35:08.505018+0000","last_fullsized":"2024-04-25T11:35:08.505018+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:40.573546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"77'1","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.314543+0000","last_change":"2024-04-25T11:33:10.027473+0000","last_active":"2024-04-25T11:35:08.314543+0000","last_peered":"2024-04-25T11:35:08.314543+0000","last_clean":"2024-04-25T11:35:08.314543+0000","last_became_active":"2024-04-25T11:33:10.027133+0000","last_became_peered":"2024-04-25T11:33:10.027133+0000","last_unstale":"2024-04-25T11:35:08.314543+0000","last_undegraded":"2024-04-25T11:35:08.314543+0000","last_fullsized":"2024-04-25T11:35:08.314543+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:19:58.338507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"79'5","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.211036+0000","last_change":"2024-04-25T11:33:10.892588+0000","last_active":"2024-04-25T11:35:08.211036+0000","last_peered":"2024-04-25T11:35:08.211036+0000","last_clean":"2024-04-25T11:35:08.211036+0000","last_became_active":"2024-04-25T11:33:10.892088+0000","last_became_peered":"2024-04-25T11:33:10.892088+0000","last_unstale":"2024-04-25T11:35:08.211036+0000","last_undegraded":"2024-04-25T11:35:08.211036+0000","last_fullsized":"2024-04-25T11:35:08.211036+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:10.136222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"77'5","reported_seq":140,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500939+0000","last_change":"2024-04-25T11:33:10.030801+0000","last_active":"2024-04-25T11:35:08.500939+0000","last_peered":"2024-04-25T11:35:08.500939+0000","last_clean":"2024-04-25T11:35:08.500939+0000","last_became_active":"2024-04-25T11:33:10.030466+0000","last_became_peered":"2024-04-25T11:33:10.030466+0000","last_unstale":"2024-04-25T11:35:08.500939+0000","last_undegraded":"2024-04-25T11:35:08.500939+0000","last_fullsized":"2024-04-25T11:35:08.500939+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:22:02.922309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.211021+0000","last_change":"2024-04-25T11:33:10.894846+0000","last_active":"2024-04-25T11:35:08.211021+0000","last_peered":"2024-04-25T11:35:08.211021+0000","last_clean":"2024-04-25T11:35:08.211021+0000","last_became_active":"2024-04-25T11:33:10.894456+0000","last_became_peered":"2024-04-25T11:33:10.894456+0000","last_unstale":"2024-04-25T11:35:08.211021+0000","last_undegraded":"2024-04-25T11:35:08.211021+0000","last_fullsized":"2024-04-25T11:35:08.211021+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:11:16.265126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"79'8","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.324115+0000","last_change":"2024-04-25T11:34:17.882781+0000","last_active":"2024-04-25T11:35:08.324115+0000","last_peered":"2024-04-25T11:35:08.324115+0000","last_clean":"2024-04-25T11:35:08.324115+0000","last_became_active":"2024-04-25T11:34:17.882348+0000","last_became_peered":"2024-04-25T11:34:17.882348+0000","last_unstale":"2024-04-25T11:35:08.324115+0000","last_undegraded":"2024-04-25T11:35:08.324115+0000","last_fullsized":"2024-04-25T11:35:08.324115+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:47:34.385332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"80'7","reported_seq":142,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500831+0000","last_change":"2024-04-25T11:33:10.037834+0000","last_active":"2024-04-25T11:35:08.500831+0000","last_peered":"2024-04-25T11:35:08.500831+0000","last_clean":"2024-04-25T11:35:08.500831+0000","last_became_active":"2024-04-25T11:33:10.037570+0000","last_became_peered":"2024-04-25T11:33:10.037570+0000","last_unstale":"2024-04-25T11:35:08.500831+0000","last_undegraded":"2024-04-25T11:35:08.500831+0000","last_fullsized":"2024-04-25T11:35:08.500831+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:12.795190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.317600+0000","last_change":"2024-04-25T11:33:10.882663+0000","last_active":"2024-04-25T11:35:08.317600+0000","last_peered":"2024-04-25T11:35:08.317600+0000","last_clean":"2024-04-25T11:35:08.317600+0000","last_became_active":"2024-04-25T11:33:10.882300+0000","last_became_peered":"2024-04-25T11:33:10.882300+0000","last_unstale":"2024-04-25T11:35:08.317600+0000","last_undegraded":"2024-04-25T11:35:08.317600+0000","last_fullsized":"2024-04-25T11:35:08.317600+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:58:11.379654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"77'6","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500572+0000","last_change":"2024-04-25T11:33:10.024598+0000","last_active":"2024-04-25T11:35:08.500572+0000","last_peered":"2024-04-25T11:35:08.500572+0000","last_clean":"2024-04-25T11:35:08.500572+0000","last_became_active":"2024-04-25T11:33:10.024229+0000","last_became_peered":"2024-04-25T11:33:10.024229+0000","last_unstale":"2024-04-25T11:35:08.500572+0000","last_undegraded":"2024-04-25T11:35:08.500572+0000","last_fullsized":"2024-04-25T11:35:08.500572+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:42:14.690581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":98,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500206+0000","last_change":"2024-04-25T11:32:01.310202+0000","last_active":"2024-04-25T11:35:08.500206+0000","last_peered":"2024-04-25T11:35:08.500206+0000","last_clean":"2024-04-25T11:35:08.500206+0000","last_became_active":"2024-04-25T11:32:01.305242+0000","last_became_peered":"2024-04-25T11:32:01.305242+0000","last_unstale":"2024-04-25T11:35:08.500206+0000","last_undegraded":"2024-04-25T11:35:08.500206+0000","last_fullsized":"2024-04-25T11:35:08.500206+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:28:43.374096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:28:43.374096+0000","last_clean_scrub_stamp":"2024-04-25T11:28:43.374096+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:56:18.551747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"79'5","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207126+0000","last_change":"2024-04-25T11:33:10.889124+0000","last_active":"2024-04-25T11:35:08.207126+0000","last_peered":"2024-04-25T11:35:08.207126+0000","last_clean":"2024-04-25T11:35:08.207126+0000","last_became_active":"2024-04-25T11:33:10.888790+0000","last_became_peered":"2024-04-25T11:33:10.888790+0000","last_unstale":"2024-04-25T11:35:08.207126+0000","last_undegraded":"2024-04-25T11:35:08.207126+0000","last_fullsized":"2024-04-25T11:35:08.207126+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:33:44.999949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"77'4","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500562+0000","last_change":"2024-04-25T11:33:10.031547+0000","last_active":"2024-04-25T11:35:08.500562+0000","last_peered":"2024-04-25T11:35:08.500562+0000","last_clean":"2024-04-25T11:35:08.500562+0000","last_became_active":"2024-04-25T11:33:10.031152+0000","last_became_peered":"2024-04-25T11:33:10.031152+0000","last_unstale":"2024-04-25T11:35:08.500562+0000","last_undegraded":"2024-04-25T11:35:08.500562+0000","last_fullsized":"2024-04-25T11:35:08.500562+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:10.646771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"77'4","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.209106+0000","last_change":"2024-04-25T11:33:10.020367+0000","last_active":"2024-04-25T11:35:08.209106+0000","last_peered":"2024-04-25T11:35:08.209106+0000","last_clean":"2024-04-25T11:35:08.209106+0000","last_became_active":"2024-04-25T11:33:10.020007+0000","last_became_peered":"2024-04-25T11:33:10.020007+0000","last_unstale":"2024-04-25T11:35:08.209106+0000","last_undegraded":"2024-04-25T11:35:08.209106+0000","last_fullsized":"2024-04-25T11:35:08.209106+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:48.134907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207404+0000","last_change":"2024-04-25T11:33:10.887741+0000","last_active":"2024-04-25T11:35:08.207404+0000","last_peered":"2024-04-25T11:35:08.207404+0000","last_clean":"2024-04-25T11:35:08.207404+0000","last_became_active":"2024-04-25T11:33:10.886455+0000","last_became_peered":"2024-04-25T11:33:10.886455+0000","last_unstale":"2024-04-25T11:35:08.207404+0000","last_undegraded":"2024-04-25T11:35:08.207404+0000","last_fullsized":"2024-04-25T11:35:08.207404+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:37:16.842817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'48","reported_seq":92,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.448574+0000","last_change":"2024-04-25T11:33:10.025672+0000","last_active":"2024-04-25T11:35:08.448574+0000","last_peered":"2024-04-25T11:35:08.448574+0000","last_clean":"2024-04-25T11:35:08.448574+0000","last_became_active":"2024-04-25T11:33:10.025232+0000","last_became_peered":"2024-04-25T11:33:10.025232+0000","last_unstale":"2024-04-25T11:35:08.448574+0000","last_undegraded":"2024-04-25T11:35:08.448574+0000","last_fullsized":"2024-04-25T11:35:08.448574+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:06:26.579126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":84,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"79'5","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.317930+0000","last_change":"2024-04-25T11:33:10.884692+0000","last_active":"2024-04-25T11:35:08.317930+0000","last_peered":"2024-04-25T11:35:08.317930+0000","last_clean":"2024-04-25T11:35:08.317930+0000","last_became_active":"2024-04-25T11:33:10.884346+0000","last_became_peered":"2024-04-25T11:33:10.884346+0000","last_unstale":"2024-04-25T11:35:08.317930+0000","last_undegraded":"2024-04-25T11:35:08.317930+0000","last_fullsized":"2024-04-25T11:35:08.317930+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:31:00.630410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"77'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500890+0000","last_change":"2024-04-25T11:33:10.033872+0000","last_active":"2024-04-25T11:35:08.500890+0000","last_peered":"2024-04-25T11:35:08.500890+0000","last_clean":"2024-04-25T11:35:08.500890+0000","last_became_active":"2024-04-25T11:33:10.033435+0000","last_became_peered":"2024-04-25T11:33:10.033435+0000","last_unstale":"2024-04-25T11:35:08.500890+0000","last_undegraded":"2024-04-25T11:35:08.500890+0000","last_fullsized":"2024-04-25T11:35:08.500890+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:12.885215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"79'7","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501866+0000","last_change":"2024-04-25T11:34:17.881545+0000","last_active":"2024-04-25T11:35:08.501866+0000","last_peered":"2024-04-25T11:35:08.501866+0000","last_clean":"2024-04-25T11:35:08.501866+0000","last_became_active":"2024-04-25T11:34:17.881125+0000","last_became_peered":"2024-04-25T11:34:17.881125+0000","last_unstale":"2024-04-25T11:35:08.501866+0000","last_undegraded":"2024-04-25T11:35:08.501866+0000","last_fullsized":"2024-04-25T11:35:08.501866+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:55:18.404023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"77'2","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.380513+0000","last_change":"2024-04-25T11:33:10.032384+0000","last_active":"2024-04-25T11:35:08.380513+0000","last_peered":"2024-04-25T11:35:08.380513+0000","last_clean":"2024-04-25T11:35:08.380513+0000","last_became_active":"2024-04-25T11:33:10.032035+0000","last_became_peered":"2024-04-25T11:33:10.032035+0000","last_unstale":"2024-04-25T11:35:08.380513+0000","last_undegraded":"2024-04-25T11:35:08.380513+0000","last_fullsized":"2024-04-25T11:35:08.380513+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:21:41.723120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"79'2","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.208679+0000","last_change":"2024-04-25T11:33:10.893366+0000","last_active":"2024-04-25T11:35:08.208679+0000","last_peered":"2024-04-25T11:35:08.208679+0000","last_clean":"2024-04-25T11:35:08.208679+0000","last_became_active":"2024-04-25T11:33:10.892988+0000","last_became_peered":"2024-04-25T11:33:10.892988+0000","last_unstale":"2024-04-25T11:35:08.208679+0000","last_undegraded":"2024-04-25T11:35:08.208679+0000","last_fullsized":"2024-04-25T11:35:08.208679+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:06.368216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"77'7","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.317231+0000","last_change":"2024-04-25T11:33:10.031456+0000","last_active":"2024-04-25T11:35:08.317231+0000","last_peered":"2024-04-25T11:35:08.317231+0000","last_clean":"2024-04-25T11:35:08.317231+0000","last_became_active":"2024-04-25T11:33:10.031115+0000","last_became_peered":"2024-04-25T11:33:10.031115+0000","last_unstale":"2024-04-25T11:35:08.317231+0000","last_undegraded":"2024-04-25T11:35:08.317231+0000","last_fullsized":"2024-04-25T11:35:08.317231+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:12:56.803277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.504698+0000","last_change":"2024-04-25T11:33:10.893692+0000","last_active":"2024-04-25T11:35:08.504698+0000","last_peered":"2024-04-25T11:35:08.504698+0000","last_clean":"2024-04-25T11:35:08.504698+0000","last_became_active":"2024-04-25T11:33:10.893332+0000","last_became_peered":"2024-04-25T11:33:10.893332+0000","last_unstale":"2024-04-25T11:35:08.504698+0000","last_undegraded":"2024-04-25T11:35:08.504698+0000","last_fullsized":"2024-04-25T11:35:08.504698+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:01.277465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"80'5","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.641604+0000","last_change":"2024-04-25T11:33:10.023804+0000","last_active":"2024-04-25T11:35:08.641604+0000","last_peered":"2024-04-25T11:35:08.641604+0000","last_clean":"2024-04-25T11:35:08.641604+0000","last_became_active":"2024-04-25T11:33:10.023503+0000","last_became_peered":"2024-04-25T11:33:10.023503+0000","last_unstale":"2024-04-25T11:35:08.641604+0000","last_undegraded":"2024-04-25T11:35:08.641604+0000","last_fullsized":"2024-04-25T11:35:08.641604+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:34:48.074176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"79'8","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.314248+0000","last_change":"2024-04-25T11:34:17.882901+0000","last_active":"2024-04-25T11:35:08.314248+0000","last_peered":"2024-04-25T11:35:08.314248+0000","last_clean":"2024-04-25T11:35:08.314248+0000","last_became_active":"2024-04-25T11:34:17.882514+0000","last_became_peered":"2024-04-25T11:34:17.882514+0000","last_unstale":"2024-04-25T11:35:08.314248+0000","last_undegraded":"2024-04-25T11:35:08.314248+0000","last_fullsized":"2024-04-25T11:35:08.314248+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:03:30.843842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,10],"acting":[9,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.b","version":"77'4","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.208848+0000","last_change":"2024-04-25T11:33:10.030857+0000","last_active":"2024-04-25T11:35:08.208848+0000","last_peered":"2024-04-25T11:35:08.208848+0000","last_clean":"2024-04-25T11:35:08.208848+0000","last_became_active":"2024-04-25T11:33:10.028879+0000","last_became_peered":"2024-04-25T11:33:10.028879+0000","last_unstale":"2024-04-25T11:35:08.208848+0000","last_undegraded":"2024-04-25T11:35:08.208848+0000","last_fullsized":"2024-04-25T11:35:08.208848+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:28.820919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"79'5","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500748+0000","last_change":"2024-04-25T11:33:10.893773+0000","last_active":"2024-04-25T11:35:08.500748+0000","last_peered":"2024-04-25T11:35:08.500748+0000","last_clean":"2024-04-25T11:35:08.500748+0000","last_became_active":"2024-04-25T11:33:10.893431+0000","last_became_peered":"2024-04-25T11:33:10.893431+0000","last_unstale":"2024-04-25T11:35:08.500748+0000","last_undegraded":"2024-04-25T11:35:08.500748+0000","last_fullsized":"2024-04-25T11:35:08.500748+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:15:15.998951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"80'7","reported_seq":144,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.807310+0000","last_change":"2024-04-25T11:33:10.032745+0000","last_active":"2024-04-25T11:35:08.807310+0000","last_peered":"2024-04-25T11:35:08.807310+0000","last_clean":"2024-04-25T11:35:08.807310+0000","last_became_active":"2024-04-25T11:33:10.032361+0000","last_became_peered":"2024-04-25T11:33:10.032361+0000","last_unstale":"2024-04-25T11:35:08.807310+0000","last_undegraded":"2024-04-25T11:35:08.807310+0000","last_fullsized":"2024-04-25T11:35:08.807310+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:14:46.555892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1952,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":116,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501571+0000","last_change":"2024-04-25T11:33:10.888084+0000","last_active":"2024-04-25T11:35:08.501571+0000","last_peered":"2024-04-25T11:35:08.501571+0000","last_clean":"2024-04-25T11:35:08.501571+0000","last_became_active":"2024-04-25T11:33:10.887679+0000","last_became_peered":"2024-04-25T11:33:10.887679+0000","last_unstale":"2024-04-25T11:35:08.501571+0000","last_undegraded":"2024-04-25T11:35:08.501571+0000","last_fullsized":"2024-04-25T11:35:08.501571+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:10:20.073630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"80'41","reported_seq":70,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.445891+0000","last_change":"2024-04-25T11:33:10.035355+0000","last_active":"2024-04-25T11:35:08.445891+0000","last_peered":"2024-04-25T11:35:08.445891+0000","last_clean":"2024-04-25T11:35:08.445891+0000","last_became_active":"2024-04-25T11:33:10.034724+0000","last_became_peered":"2024-04-25T11:33:10.034724+0000","last_unstale":"2024-04-25T11:35:08.445891+0000","last_undegraded":"2024-04-25T11:35:08.445891+0000","last_fullsized":"2024-04-25T11:35:08.445891+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:24:26.757803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500628+0000","last_change":"2024-04-25T11:33:10.899827+0000","last_active":"2024-04-25T11:35:08.500628+0000","last_peered":"2024-04-25T11:35:08.500628+0000","last_clean":"2024-04-25T11:35:08.500628+0000","last_became_active":"2024-04-25T11:33:10.899356+0000","last_became_peered":"2024-04-25T11:33:10.899356+0000","last_unstale":"2024-04-25T11:35:08.500628+0000","last_undegraded":"2024-04-25T11:35:08.500628+0000","last_fullsized":"2024-04-25T11:35:08.500628+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:04:17.323958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"77'5","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207700+0000","last_change":"2024-04-25T11:33:10.034981+0000","last_active":"2024-04-25T11:35:08.207700+0000","last_peered":"2024-04-25T11:35:08.207700+0000","last_clean":"2024-04-25T11:35:08.207700+0000","last_became_active":"2024-04-25T11:33:10.034601+0000","last_became_peered":"2024-04-25T11:33:10.034601+0000","last_unstale":"2024-04-25T11:35:08.207700+0000","last_undegraded":"2024-04-25T11:35:08.207700+0000","last_fullsized":"2024-04-25T11:35:08.207700+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:19:14.710731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501693+0000","last_change":"2024-04-25T11:33:10.890020+0000","last_active":"2024-04-25T11:35:08.501693+0000","last_peered":"2024-04-25T11:35:08.501693+0000","last_clean":"2024-04-25T11:35:08.501693+0000","last_became_active":"2024-04-25T11:33:10.889558+0000","last_became_peered":"2024-04-25T11:33:10.889558+0000","last_unstale":"2024-04-25T11:35:08.501693+0000","last_undegraded":"2024-04-25T11:35:08.501693+0000","last_fullsized":"2024-04-25T11:35:08.501693+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:00:49.096818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.504776+0000","last_change":"2024-04-25T11:33:10.886965+0000","last_active":"2024-04-25T11:35:08.504776+0000","last_peered":"2024-04-25T11:35:08.504776+0000","last_clean":"2024-04-25T11:35:08.504776+0000","last_became_active":"2024-04-25T11:33:10.886594+0000","last_became_peered":"2024-04-25T11:33:10.886594+0000","last_unstale":"2024-04-25T11:35:08.504776+0000","last_undegraded":"2024-04-25T11:35:08.504776+0000","last_fullsized":"2024-04-25T11:35:08.504776+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:58.924857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"79'12","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.499974+0000","last_change":"2024-04-25T11:33:10.890760+0000","last_active":"2024-04-25T11:35:08.499974+0000","last_peered":"2024-04-25T11:35:08.499974+0000","last_clean":"2024-04-25T11:35:08.499974+0000","last_became_active":"2024-04-25T11:33:10.889643+0000","last_became_peered":"2024-04-25T11:33:10.889643+0000","last_unstale":"2024-04-25T11:35:08.499974+0000","last_undegraded":"2024-04-25T11:35:08.499974+0000","last_fullsized":"2024-04-25T11:35:08.499974+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:54.644853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.499871+0000","last_change":"2024-04-25T11:33:10.887850+0000","last_active":"2024-04-25T11:35:08.499871+0000","last_peered":"2024-04-25T11:35:08.499871+0000","last_clean":"2024-04-25T11:35:08.499871+0000","last_became_active":"2024-04-25T11:33:10.882323+0000","last_became_peered":"2024-04-25T11:33:10.882323+0000","last_unstale":"2024-04-25T11:35:08.499871+0000","last_undegraded":"2024-04-25T11:35:08.499871+0000","last_fullsized":"2024-04-25T11:35:08.499871+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:24:00.509704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501134+0000","last_change":"2024-04-25T11:33:10.883842+0000","last_active":"2024-04-25T11:35:08.501134+0000","last_peered":"2024-04-25T11:35:08.501134+0000","last_clean":"2024-04-25T11:35:08.501134+0000","last_became_active":"2024-04-25T11:33:10.883513+0000","last_became_peered":"2024-04-25T11:33:10.883513+0000","last_unstale":"2024-04-25T11:35:08.501134+0000","last_undegraded":"2024-04-25T11:35:08.501134+0000","last_fullsized":"2024-04-25T11:35:08.501134+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:47:09.731875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"79'4","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207357+0000","last_change":"2024-04-25T11:33:10.890503+0000","last_active":"2024-04-25T11:35:08.207357+0000","last_peered":"2024-04-25T11:35:08.207357+0000","last_clean":"2024-04-25T11:35:08.207357+0000","last_became_active":"2024-04-25T11:33:10.890162+0000","last_became_peered":"2024-04-25T11:33:10.890162+0000","last_unstale":"2024-04-25T11:35:08.207357+0000","last_undegraded":"2024-04-25T11:35:08.207357+0000","last_fullsized":"2024-04-25T11:35:08.207357+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:36.462789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"79'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.504912+0000","last_change":"2024-04-25T11:33:10.897109+0000","last_active":"2024-04-25T11:35:08.504912+0000","last_peered":"2024-04-25T11:35:08.504912+0000","last_clean":"2024-04-25T11:35:08.504912+0000","last_became_active":"2024-04-25T11:33:10.896138+0000","last_became_peered":"2024-04-25T11:33:10.896138+0000","last_unstale":"2024-04-25T11:35:08.504912+0000","last_undegraded":"2024-04-25T11:35:08.504912+0000","last_fullsized":"2024-04-25T11:35:08.504912+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:47:46.202355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"79'15","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500379+0000","last_change":"2024-04-25T11:33:10.890397+0000","last_active":"2024-04-25T11:35:08.500379+0000","last_peered":"2024-04-25T11:35:08.500379+0000","last_clean":"2024-04-25T11:35:08.500379+0000","last_became_active":"2024-04-25T11:33:10.889580+0000","last_became_peered":"2024-04-25T11:33:10.889580+0000","last_unstale":"2024-04-25T11:35:08.500379+0000","last_undegraded":"2024-04-25T11:35:08.500379+0000","last_fullsized":"2024-04-25T11:35:08.500379+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:16:09.672131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"79'2","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.317841+0000","last_change":"2024-04-25T11:33:10.873817+0000","last_active":"2024-04-25T11:35:08.317841+0000","last_peered":"2024-04-25T11:35:08.317841+0000","last_clean":"2024-04-25T11:35:08.317841+0000","last_became_active":"2024-04-25T11:33:10.873425+0000","last_became_peered":"2024-04-25T11:33:10.873425+0000","last_unstale":"2024-04-25T11:35:08.317841+0000","last_undegraded":"2024-04-25T11:35:08.317841+0000","last_fullsized":"2024-04-25T11:35:08.317841+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:52:26.279705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"79'6","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207126+0000","last_change":"2024-04-25T11:34:17.880075+0000","last_active":"2024-04-25T11:35:08.207126+0000","last_peered":"2024-04-25T11:35:08.207126+0000","last_clean":"2024-04-25T11:35:08.207126+0000","last_became_active":"2024-04-25T11:34:17.879692+0000","last_became_peered":"2024-04-25T11:34:17.879692+0000","last_unstale":"2024-04-25T11:35:08.207126+0000","last_undegraded":"2024-04-25T11:35:08.207126+0000","last_fullsized":"2024-04-25T11:35:08.207126+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:54.833601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,3],"acting":[6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.29","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.317356+0000","last_change":"2024-04-25T11:33:10.883690+0000","last_active":"2024-04-25T11:35:08.317356+0000","last_peered":"2024-04-25T11:35:08.317356+0000","last_clean":"2024-04-25T11:35:08.317356+0000","last_became_active":"2024-04-25T11:33:10.882965+0000","last_became_peered":"2024-04-25T11:33:10.882965+0000","last_unstale":"2024-04-25T11:35:08.317356+0000","last_undegraded":"2024-04-25T11:35:08.317356+0000","last_fullsized":"2024-04-25T11:35:08.317356+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:28.126631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"79'10","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.380742+0000","last_change":"2024-04-25T11:33:10.897050+0000","last_active":"2024-04-25T11:35:08.380742+0000","last_peered":"2024-04-25T11:35:08.380742+0000","last_clean":"2024-04-25T11:35:08.380742+0000","last_became_active":"2024-04-25T11:33:10.896273+0000","last_became_peered":"2024-04-25T11:33:10.896273+0000","last_unstale":"2024-04-25T11:35:08.380742+0000","last_undegraded":"2024-04-25T11:35:08.380742+0000","last_fullsized":"2024-04-25T11:35:08.380742+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:45:53.839823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"79'11","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207505+0000","last_change":"2024-04-25T11:33:10.897323+0000","last_active":"2024-04-25T11:35:08.207505+0000","last_peered":"2024-04-25T11:35:08.207505+0000","last_clean":"2024-04-25T11:35:08.207505+0000","last_became_active":"2024-04-25T11:33:10.896951+0000","last_became_peered":"2024-04-25T11:33:10.896951+0000","last_unstale":"2024-04-25T11:35:08.207505+0000","last_undegraded":"2024-04-25T11:35:08.207505+0000","last_fullsized":"2024-04-25T11:35:08.207505+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:24:31.208338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.504797+0000","last_change":"2024-04-25T11:33:10.897566+0000","last_active":"2024-04-25T11:35:08.504797+0000","last_peered":"2024-04-25T11:35:08.504797+0000","last_clean":"2024-04-25T11:35:08.504797+0000","last_became_active":"2024-04-25T11:33:10.897212+0000","last_became_peered":"2024-04-25T11:33:10.897212+0000","last_unstale":"2024-04-25T11:35:08.504797+0000","last_undegraded":"2024-04-25T11:35:08.504797+0000","last_fullsized":"2024-04-25T11:35:08.504797+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:55:29.995287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":601791,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1272,"num_read_kb":747,"num_write":370,"num_write_kb":1464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1805373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":299,"ondisk_log_size":299,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645348,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36204,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708948,"statfs":{"total":95995035648,"available":95957962752,"internally_reserved":0,"allocated":884736,"data_stored":423632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841577,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37024,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708128,"statfs":{"total":95995035648,"available":95957123072,"internally_reserved":0,"allocated":1728512,"data_stored":1214160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037805,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1302528,"data_stored":815996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234035,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28288,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716864,"statfs":{"total":95995035648,"available":95966068736,"internally_reserved":0,"allocated":1298432,"data_stored":816374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430264,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36788,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708364,"statfs":{"total":95995035648,"available":95957364736,"internally_reserved":0,"allocated":1486848,"data_stored":1007140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626493,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32052,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713100,"statfs":{"total":95995035648,"available":95962214400,"internally_reserved":0,"allocated":892928,"data_stored":422497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":962560,"data_stored":491168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":424618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40364,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704788,"statfs":{"total":95995035648,"available":95953702912,"internally_reserved":0,"allocated":888832,"data_stored":414953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28420,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716732,"statfs":{"total":95995035648,"available":95965933568,"internally_reserved":0,"allocated":1437696,"data_stored":947138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986247,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":970752,"data_stored":491039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822722,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28480,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716672,"statfs":{"total":95995035648,"available":95965872128,"internally_reserved":0,"allocated":1499136,"data_stored":1006286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":69510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":69381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":135510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":394716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":202017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T11:35:09.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T11:35:09.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.904+0000 7f265ef40640 1 -- 172.21.15.90:0/145755649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2658109750 msgr2=0x7f265810bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:09.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.904+0000 7f265ef40640 1 --2- 172.21.15.90:0/145755649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2658109750 0x7f265810bc60 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f2650009900 tx=0x7f265002a410 comp rx=0 tx=0).stop 2024-04-25T11:35:09.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.905+0000 7f265ef40640 1 -- 172.21.15.90:0/145755649 shutdown_connections 2024-04-25T11:35:09.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.905+0000 7f265ef40640 1 --2- 172.21.15.90:0/145755649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2658109750 0x7f265810bc60 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:09.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.905+0000 7f265ef40640 1 --2- 172.21.15.90:0/145755649 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2658106be0 0x7f26581090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:09.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.905+0000 7f265ef40640 1 --2- 172.21.15.90:0/145755649 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26581041f0 0x7f2658106610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:09.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.905+0000 7f265ef40640 1 -- 172.21.15.90:0/145755649 >> 172.21.15.90:0/145755649 conn(0x7f26580fdbd0 msgr2=0x7f2658100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:09.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.905+0000 7f265ef40640 1 -- 172.21.15.90:0/145755649 shutdown_connections 2024-04-25T11:35:09.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.905+0000 7f265ef40640 1 -- 172.21.15.90:0/145755649 wait complete. 2024-04-25T11:35:09.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.905+0000 7f265ef40640 1 Processor -- start 2024-04-25T11:35:09.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.905+0000 7f265ef40640 1 -- start start 2024-04-25T11:35:09.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.905+0000 7f265ef40640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26581041f0 0x7f265819a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:09.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.905+0000 7f265ef40640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2658106be0 0x7f265819a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:09.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.905+0000 7f265ef40640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2658109750 0x7f26581a1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:09.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.905+0000 7f265ef40640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f265810f5a0 con 0x7f2658109750 2024-04-25T11:35:09.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.905+0000 7f265ef40640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f265810f420 con 0x7f2658106be0 2024-04-25T11:35:09.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.905+0000 7f265ef40640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f265810f720 con 0x7f26581041f0 2024-04-25T11:35:09.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.905+0000 7f265d4dd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2658109750 0x7f26581a1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:09.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.906+0000 7f265d4dd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2658109750 0x7f26581a1610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34388/0 (socket says 172.21.15.90:34388) 2024-04-25T11:35:09.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.906+0000 7f265d4dd640 1 -- 172.21.15.90:0/791961593 learned_addr learned my addr 172.21.15.90:0/791961593 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:09.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.906+0000 7f2657fff640 1 --2- 172.21.15.90:0/791961593 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2658106be0 0x7f265819a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:09.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.906+0000 7f265ccdc640 1 --2- 172.21.15.90:0/791961593 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26581041f0 0x7f265819a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:09.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.906+0000 7f265d4dd640 1 -- 172.21.15.90:0/791961593 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26581041f0 msgr2=0x7f265819a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:09.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.906+0000 7f265d4dd640 1 --2- 172.21.15.90:0/791961593 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26581041f0 0x7f265819a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:09.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.906+0000 7f265d4dd640 1 -- 172.21.15.90:0/791961593 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2658106be0 msgr2=0x7f265819a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:09.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.906+0000 7f265d4dd640 1 --2- 172.21.15.90:0/791961593 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2658106be0 0x7f265819a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:09.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.906+0000 7f265d4dd640 1 -- 172.21.15.90:0/791961593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26581a1c80 con 0x7f2658109750 2024-04-25T11:35:09.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.906+0000 7f265d4dd640 1 --2- 172.21.15.90:0/791961593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2658109750 0x7f26581a1610 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f2650009230 tx=0x7f2650031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:09.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.906+0000 7f2655ffb640 1 -- 172.21.15.90:0/791961593 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f265003f030 con 0x7f2658109750 2024-04-25T11:35:09.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.906+0000 7f2655ffb640 1 -- 172.21.15.90:0/791961593 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2650003aa0 con 0x7f2658109750 2024-04-25T11:35:09.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.906+0000 7f265ef40640 1 -- 172.21.15.90:0/791961593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26581a1f10 con 0x7f2658109750 2024-04-25T11:35:09.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.906+0000 7f2655ffb640 1 -- 172.21.15.90:0/791961593 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2650035070 con 0x7f2658109750 2024-04-25T11:35:09.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.906+0000 7f265ef40640 1 -- 172.21.15.90:0/791961593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26581a2320 con 0x7f2658109750 2024-04-25T11:35:09.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.907+0000 7f265ef40640 1 -- 172.21.15.90:0/791961593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26580ff6f0 con 0x7f2658109750 2024-04-25T11:35:09.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.909+0000 7f2655ffb640 1 -- 172.21.15.90:0/791961593 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 21) ==== 147332+0+0 (secure 0 0 0) 0x7f2650042030 con 0x7f2658109750 2024-04-25T11:35:09.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.909+0000 7f2655ffb640 1 --2- 172.21.15.90:0/791961593 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f26400af8e0 0x7f26400b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:09.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.910+0000 7f265ccdc640 1 --2- 172.21.15.90:0/791961593 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f26400af8e0 0x7f26400b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:09.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.910+0000 7f2655ffb640 1 -- 172.21.15.90:0/791961593 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f26500ff1f0 con 0x7f2658109750 2024-04-25T11:35:09.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.910+0000 7f265ccdc640 1 --2- 172.21.15.90:0/791961593 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f26400af8e0 0x7f26400b1df0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f265806a850 tx=0x7f2648009670 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:09.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:09.913+0000 7f2655ffb640 1 -- 172.21.15.90:0/791961593 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f26500c7950 con 0x7f2658109750 2024-04-25T11:35:10.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.119+0000 7f265ef40640 1 -- 172.21.15.90:0/791961593 --> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2658062fe0 con 0x7f26400af8e0 2024-04-25T11:35:10.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.128+0000 7f2655ffb640 1 -- 172.21.15.90:0/791961593 <== mgr.15366 v2:172.21.15.90:6800/4078377640 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257767 (secure 0 0 0) 0x7f2658062fe0 con 0x7f26400af8e0 2024-04-25T11:35:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:35:10.124 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-25T11:35:10.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.131+0000 7f265ef40640 1 -- 172.21.15.90:0/791961593 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f26400af8e0 msgr2=0x7f26400b1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:10.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.131+0000 7f265ef40640 1 --2- 172.21.15.90:0/791961593 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f26400af8e0 0x7f26400b1df0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f265806a850 tx=0x7f2648009670 comp rx=0 tx=0).stop 2024-04-25T11:35:10.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.131+0000 7f265ef40640 1 -- 172.21.15.90:0/791961593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2658109750 msgr2=0x7f26581a1610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:10.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.131+0000 7f265ef40640 1 --2- 172.21.15.90:0/791961593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2658109750 0x7f26581a1610 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f2650009230 tx=0x7f2650031a50 comp rx=0 tx=0).stop 2024-04-25T11:35:10.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.131+0000 7f265ef40640 1 -- 172.21.15.90:0/791961593 shutdown_connections 2024-04-25T11:35:10.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.131+0000 7f265ef40640 1 --2- 172.21.15.90:0/791961593 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f26400af8e0 0x7f26400b1df0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:10.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.131+0000 7f265ef40640 1 --2- 172.21.15.90:0/791961593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2658109750 0x7f26581a1610 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:10.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.131+0000 7f265ef40640 1 --2- 172.21.15.90:0/791961593 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2658106be0 0x7f265819a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:10.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.131+0000 7f265ef40640 1 --2- 172.21.15.90:0/791961593 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26581041f0 0x7f265819a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:10.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.131+0000 7f265ef40640 1 -- 172.21.15.90:0/791961593 >> 172.21.15.90:0/791961593 conn(0x7f26580fdbd0 msgr2=0x7f2658104ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:10.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.131+0000 7f265ef40640 1 -- 172.21.15.90:0/791961593 shutdown_connections 2024-04-25T11:35:10.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.131+0000 7f265ef40640 1 -- 172.21.15.90:0/791961593 wait complete. 2024-04-25T11:35:10.133 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":3,"stamp":"2024-04-25T11:35:09.241136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1192393,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1353,"num_read_kb":816,"num_write":495,"num_write_kb":3616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":3012886,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":893,"ondisk_log_size":893,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":392232,"kb_used_data":13932,"kb_used_omap":18,"kb_used_meta":378157,"kb_avail":1124549592,"statfs":{"total":1151940427776,"available":1151538782208,"internally_reserved":0,"allocated":14266368,"data_stored":8475001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":387233081},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"3.3f","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.505112+0000","last_change":"2024-04-25T11:33:10.892144+0000","last_active":"2024-04-25T11:35:08.505112+0000","last_peered":"2024-04-25T11:35:08.505112+0000","last_clean":"2024-04-25T11:35:08.505112+0000","last_became_active":"2024-04-25T11:33:10.891721+0000","last_became_peered":"2024-04-25T11:33:10.891721+0000","last_unstale":"2024-04-25T11:35:08.505112+0000","last_undegraded":"2024-04-25T11:35:08.505112+0000","last_fullsized":"2024-04-25T11:35:08.505112+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:58:25.447086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"79'4","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.502017+0000","last_change":"2024-04-25T11:33:10.891647+0000","last_active":"2024-04-25T11:35:08.502017+0000","last_peered":"2024-04-25T11:35:08.502017+0000","last_clean":"2024-04-25T11:35:08.502017+0000","last_became_active":"2024-04-25T11:33:10.891283+0000","last_became_peered":"2024-04-25T11:33:10.891283+0000","last_unstale":"2024-04-25T11:35:08.502017+0000","last_undegraded":"2024-04-25T11:35:08.502017+0000","last_fullsized":"2024-04-25T11:35:08.502017+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:12.669438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501829+0000","last_change":"2024-04-25T11:33:10.890928+0000","last_active":"2024-04-25T11:35:08.501829+0000","last_peered":"2024-04-25T11:35:08.501829+0000","last_clean":"2024-04-25T11:35:08.501829+0000","last_became_active":"2024-04-25T11:33:10.889093+0000","last_became_peered":"2024-04-25T11:33:10.889093+0000","last_unstale":"2024-04-25T11:35:08.501829+0000","last_undegraded":"2024-04-25T11:35:08.501829+0000","last_fullsized":"2024-04-25T11:35:08.501829+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:28:09.038650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501058+0000","last_change":"2024-04-25T11:33:10.890184+0000","last_active":"2024-04-25T11:35:08.501058+0000","last_peered":"2024-04-25T11:35:08.501058+0000","last_clean":"2024-04-25T11:35:08.501058+0000","last_became_active":"2024-04-25T11:33:10.889367+0000","last_became_peered":"2024-04-25T11:33:10.889367+0000","last_unstale":"2024-04-25T11:35:08.501058+0000","last_undegraded":"2024-04-25T11:35:08.501058+0000","last_fullsized":"2024-04-25T11:35:08.501058+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:42.798295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"79'4","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.208757+0000","last_change":"2024-04-25T11:33:10.889758+0000","last_active":"2024-04-25T11:35:08.208757+0000","last_peered":"2024-04-25T11:35:08.208757+0000","last_clean":"2024-04-25T11:35:08.208757+0000","last_became_active":"2024-04-25T11:33:10.889384+0000","last_became_peered":"2024-04-25T11:33:10.889384+0000","last_unstale":"2024-04-25T11:35:08.208757+0000","last_undegraded":"2024-04-25T11:35:08.208757+0000","last_fullsized":"2024-04-25T11:35:08.208757+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:54:38.794216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"79'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.210810+0000","last_change":"2024-04-25T11:33:10.889348+0000","last_active":"2024-04-25T11:35:08.210810+0000","last_peered":"2024-04-25T11:35:08.210810+0000","last_clean":"2024-04-25T11:35:08.210810+0000","last_became_active":"2024-04-25T11:33:10.888908+0000","last_became_peered":"2024-04-25T11:33:10.888908+0000","last_unstale":"2024-04-25T11:35:08.210810+0000","last_undegraded":"2024-04-25T11:35:08.210810+0000","last_fullsized":"2024-04-25T11:35:08.210810+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:02:59.555921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"79'5","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500248+0000","last_change":"2024-04-25T11:34:18.237661+0000","last_active":"2024-04-25T11:35:08.500248+0000","last_peered":"2024-04-25T11:35:08.500248+0000","last_clean":"2024-04-25T11:35:08.500248+0000","last_became_active":"2024-04-25T11:34:18.237306+0000","last_became_peered":"2024-04-25T11:34:18.237306+0000","last_unstale":"2024-04-25T11:35:08.500248+0000","last_undegraded":"2024-04-25T11:35:08.500248+0000","last_fullsized":"2024-04-25T11:35:08.500248+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:17:34.228368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.317192+0000","last_change":"2024-04-25T11:33:10.887262+0000","last_active":"2024-04-25T11:35:08.317192+0000","last_peered":"2024-04-25T11:35:08.317192+0000","last_clean":"2024-04-25T11:35:08.317192+0000","last_became_active":"2024-04-25T11:33:10.886948+0000","last_became_peered":"2024-04-25T11:33:10.886948+0000","last_unstale":"2024-04-25T11:35:08.317192+0000","last_undegraded":"2024-04-25T11:35:08.317192+0000","last_fullsized":"2024-04-25T11:35:08.317192+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:09.721506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"79'11","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.314351+0000","last_change":"2024-04-25T11:34:17.883253+0000","last_active":"2024-04-25T11:35:08.314351+0000","last_peered":"2024-04-25T11:35:08.314351+0000","last_clean":"2024-04-25T11:35:08.314351+0000","last_became_active":"2024-04-25T11:34:17.882515+0000","last_became_peered":"2024-04-25T11:34:17.882515+0000","last_unstale":"2024-04-25T11:35:08.314351+0000","last_undegraded":"2024-04-25T11:35:08.314351+0000","last_fullsized":"2024-04-25T11:35:08.314351+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:52:32.813464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.36","version":"79'16","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.211225+0000","last_change":"2024-04-25T11:33:10.889812+0000","last_active":"2024-04-25T11:35:08.211225+0000","last_peered":"2024-04-25T11:35:08.211225+0000","last_clean":"2024-04-25T11:35:08.211225+0000","last_became_active":"2024-04-25T11:33:10.889369+0000","last_became_peered":"2024-04-25T11:33:10.889369+0000","last_unstale":"2024-04-25T11:35:08.211225+0000","last_undegraded":"2024-04-25T11:35:08.211225+0000","last_fullsized":"2024-04-25T11:35:08.211225+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:42:14.443496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"79'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501487+0000","last_change":"2024-04-25T11:33:10.891186+0000","last_active":"2024-04-25T11:35:08.501487+0000","last_peered":"2024-04-25T11:35:08.501487+0000","last_clean":"2024-04-25T11:35:08.501487+0000","last_became_active":"2024-04-25T11:33:10.890841+0000","last_became_peered":"2024-04-25T11:33:10.890841+0000","last_unstale":"2024-04-25T11:35:08.501487+0000","last_undegraded":"2024-04-25T11:35:08.501487+0000","last_fullsized":"2024-04-25T11:35:08.501487+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:24:56.078592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501001+0000","last_change":"2024-04-25T11:33:10.888801+0000","last_active":"2024-04-25T11:35:08.501001+0000","last_peered":"2024-04-25T11:35:08.501001+0000","last_clean":"2024-04-25T11:35:08.501001+0000","last_became_active":"2024-04-25T11:33:10.888437+0000","last_became_peered":"2024-04-25T11:33:10.888437+0000","last_unstale":"2024-04-25T11:35:08.501001+0000","last_undegraded":"2024-04-25T11:35:08.501001+0000","last_fullsized":"2024-04-25T11:35:08.501001+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:14:14.011475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"79'10","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207334+0000","last_change":"2024-04-25T11:33:10.892749+0000","last_active":"2024-04-25T11:35:08.207334+0000","last_peered":"2024-04-25T11:35:08.207334+0000","last_clean":"2024-04-25T11:35:08.207334+0000","last_became_active":"2024-04-25T11:33:10.892370+0000","last_became_peered":"2024-04-25T11:33:10.892370+0000","last_unstale":"2024-04-25T11:35:08.207334+0000","last_undegraded":"2024-04-25T11:35:08.207334+0000","last_fullsized":"2024-04-25T11:35:08.207334+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:00:51.312387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500624+0000","last_change":"2024-04-25T11:33:10.888981+0000","last_active":"2024-04-25T11:35:08.500624+0000","last_peered":"2024-04-25T11:35:08.500624+0000","last_clean":"2024-04-25T11:35:08.500624+0000","last_became_active":"2024-04-25T11:33:10.888537+0000","last_became_peered":"2024-04-25T11:33:10.888537+0000","last_unstale":"2024-04-25T11:35:08.500624+0000","last_undegraded":"2024-04-25T11:35:08.500624+0000","last_fullsized":"2024-04-25T11:35:08.500624+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:12:19.018698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501399+0000","last_change":"2024-04-25T11:33:10.891455+0000","last_active":"2024-04-25T11:35:08.501399+0000","last_peered":"2024-04-25T11:35:08.501399+0000","last_clean":"2024-04-25T11:35:08.501399+0000","last_became_active":"2024-04-25T11:33:10.891037+0000","last_became_peered":"2024-04-25T11:33:10.891037+0000","last_unstale":"2024-04-25T11:35:08.501399+0000","last_undegraded":"2024-04-25T11:35:08.501399+0000","last_fullsized":"2024-04-25T11:35:08.501399+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:44.470674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.380415+0000","last_change":"2024-04-25T11:33:10.895309+0000","last_active":"2024-04-25T11:35:08.380415+0000","last_peered":"2024-04-25T11:35:08.380415+0000","last_clean":"2024-04-25T11:35:08.380415+0000","last_became_active":"2024-04-25T11:33:10.894952+0000","last_became_peered":"2024-04-25T11:33:10.894952+0000","last_unstale":"2024-04-25T11:35:08.380415+0000","last_undegraded":"2024-04-25T11:35:08.380415+0000","last_fullsized":"2024-04-25T11:35:08.380415+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:29:22.119017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.505207+0000","last_change":"2024-04-25T11:33:10.894524+0000","last_active":"2024-04-25T11:35:08.505207+0000","last_peered":"2024-04-25T11:35:08.505207+0000","last_clean":"2024-04-25T11:35:08.505207+0000","last_became_active":"2024-04-25T11:33:10.894220+0000","last_became_peered":"2024-04-25T11:33:10.894220+0000","last_unstale":"2024-04-25T11:35:08.505207+0000","last_undegraded":"2024-04-25T11:35:08.505207+0000","last_fullsized":"2024-04-25T11:35:08.505207+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:10:29.072976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"79'10","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.315017+0000","last_change":"2024-04-25T11:33:10.886814+0000","last_active":"2024-04-25T11:35:08.315017+0000","last_peered":"2024-04-25T11:35:08.315017+0000","last_clean":"2024-04-25T11:35:08.315017+0000","last_became_active":"2024-04-25T11:33:10.886494+0000","last_became_peered":"2024-04-25T11:33:10.886494+0000","last_unstale":"2024-04-25T11:35:08.315017+0000","last_undegraded":"2024-04-25T11:35:08.315017+0000","last_fullsized":"2024-04-25T11:35:08.315017+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:03:43.054485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500081+0000","last_change":"2024-04-25T11:33:10.886870+0000","last_active":"2024-04-25T11:35:08.500081+0000","last_peered":"2024-04-25T11:35:08.500081+0000","last_clean":"2024-04-25T11:35:08.500081+0000","last_became_active":"2024-04-25T11:33:10.886507+0000","last_became_peered":"2024-04-25T11:33:10.886507+0000","last_unstale":"2024-04-25T11:35:08.500081+0000","last_undegraded":"2024-04-25T11:35:08.500081+0000","last_fullsized":"2024-04-25T11:35:08.500081+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:50.101760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"77'13","reported_seq":148,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.314751+0000","last_change":"2024-04-25T11:33:10.028970+0000","last_active":"2024-04-25T11:35:08.314751+0000","last_peered":"2024-04-25T11:35:08.314751+0000","last_clean":"2024-04-25T11:35:08.314751+0000","last_became_active":"2024-04-25T11:33:10.028636+0000","last_became_peered":"2024-04-25T11:33:10.028636+0000","last_unstale":"2024-04-25T11:35:08.314751+0000","last_undegraded":"2024-04-25T11:35:08.314751+0000","last_fullsized":"2024-04-25T11:35:08.314751+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:02.414903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"79'5","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.314684+0000","last_change":"2024-04-25T11:34:17.881522+0000","last_active":"2024-04-25T11:35:08.314684+0000","last_peered":"2024-04-25T11:35:08.314684+0000","last_clean":"2024-04-25T11:35:08.314684+0000","last_became_active":"2024-04-25T11:34:17.881114+0000","last_became_peered":"2024-04-25T11:34:17.881114+0000","last_unstale":"2024-04-25T11:35:08.314684+0000","last_undegraded":"2024-04-25T11:35:08.314684+0000","last_fullsized":"2024-04-25T11:35:08.314684+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:59:03.675054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"77'2","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.317914+0000","last_change":"2024-04-25T11:33:10.036958+0000","last_active":"2024-04-25T11:35:08.317914+0000","last_peered":"2024-04-25T11:35:08.317914+0000","last_clean":"2024-04-25T11:35:08.317914+0000","last_became_active":"2024-04-25T11:33:10.036606+0000","last_became_peered":"2024-04-25T11:33:10.036606+0000","last_unstale":"2024-04-25T11:35:08.317914+0000","last_undegraded":"2024-04-25T11:35:08.317914+0000","last_fullsized":"2024-04-25T11:35:08.317914+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:12.660208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"79'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.502299+0000","last_change":"2024-04-25T11:33:10.887501+0000","last_active":"2024-04-25T11:35:08.502299+0000","last_peered":"2024-04-25T11:35:08.502299+0000","last_clean":"2024-04-25T11:35:08.502299+0000","last_became_active":"2024-04-25T11:33:10.885985+0000","last_became_peered":"2024-04-25T11:33:10.885985+0000","last_unstale":"2024-04-25T11:35:08.502299+0000","last_undegraded":"2024-04-25T11:35:08.502299+0000","last_fullsized":"2024-04-25T11:35:08.502299+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:31:32.035646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"77'12","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.209236+0000","last_change":"2024-04-25T11:33:10.031161+0000","last_active":"2024-04-25T11:35:08.209236+0000","last_peered":"2024-04-25T11:35:08.209236+0000","last_clean":"2024-04-25T11:35:08.209236+0000","last_became_active":"2024-04-25T11:33:10.028891+0000","last_became_peered":"2024-04-25T11:33:10.028891+0000","last_unstale":"2024-04-25T11:35:08.209236+0000","last_undegraded":"2024-04-25T11:35:08.209236+0000","last_fullsized":"2024-04-25T11:35:08.209236+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:25.959699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.380513+0000","last_change":"2024-04-25T11:33:10.889439+0000","last_active":"2024-04-25T11:35:08.380513+0000","last_peered":"2024-04-25T11:35:08.380513+0000","last_clean":"2024-04-25T11:35:08.380513+0000","last_became_active":"2024-04-25T11:33:10.889082+0000","last_became_peered":"2024-04-25T11:33:10.889082+0000","last_unstale":"2024-04-25T11:35:08.380513+0000","last_undegraded":"2024-04-25T11:35:08.380513+0000","last_fullsized":"2024-04-25T11:35:08.380513+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:13:48.958179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"77'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.208951+0000","last_change":"2024-04-25T11:33:10.030793+0000","last_active":"2024-04-25T11:35:08.208951+0000","last_peered":"2024-04-25T11:35:08.208951+0000","last_clean":"2024-04-25T11:35:08.208951+0000","last_became_active":"2024-04-25T11:33:10.028805+0000","last_became_peered":"2024-04-25T11:33:10.028805+0000","last_unstale":"2024-04-25T11:35:08.208951+0000","last_undegraded":"2024-04-25T11:35:08.208951+0000","last_fullsized":"2024-04-25T11:35:08.208951+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:36.233194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"79'15","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.324396+0000","last_change":"2024-04-25T11:33:10.894815+0000","last_active":"2024-04-25T11:35:08.324396+0000","last_peered":"2024-04-25T11:35:08.324396+0000","last_clean":"2024-04-25T11:35:08.324396+0000","last_became_active":"2024-04-25T11:33:10.892174+0000","last_became_peered":"2024-04-25T11:33:10.892174+0000","last_unstale":"2024-04-25T11:35:08.324396+0000","last_undegraded":"2024-04-25T11:35:08.324396+0000","last_fullsized":"2024-04-25T11:35:08.324396+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:05.368441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"80'8","reported_seq":149,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.802370+0000","last_change":"2024-04-25T11:33:10.031238+0000","last_active":"2024-04-25T11:35:08.802370+0000","last_peered":"2024-04-25T11:35:08.802370+0000","last_clean":"2024-04-25T11:35:08.802370+0000","last_became_active":"2024-04-25T11:33:10.030917+0000","last_became_peered":"2024-04-25T11:33:10.030917+0000","last_unstale":"2024-04-25T11:35:08.802370+0000","last_undegraded":"2024-04-25T11:35:08.802370+0000","last_fullsized":"2024-04-25T11:35:08.802370+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:00:35.634293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1712,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":120,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.208813+0000","last_change":"2024-04-25T11:33:10.896974+0000","last_active":"2024-04-25T11:35:08.208813+0000","last_peered":"2024-04-25T11:35:08.208813+0000","last_clean":"2024-04-25T11:35:08.208813+0000","last_became_active":"2024-04-25T11:33:10.896575+0000","last_became_peered":"2024-04-25T11:33:10.896575+0000","last_unstale":"2024-04-25T11:35:08.208813+0000","last_undegraded":"2024-04-25T11:35:08.208813+0000","last_fullsized":"2024-04-25T11:35:08.208813+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:32:43.533652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"77'7","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.210783+0000","last_change":"2024-04-25T11:33:10.022714+0000","last_active":"2024-04-25T11:35:08.210783+0000","last_peered":"2024-04-25T11:35:08.210783+0000","last_clean":"2024-04-25T11:35:08.210783+0000","last_became_active":"2024-04-25T11:33:10.022230+0000","last_became_peered":"2024-04-25T11:33:10.022230+0000","last_unstale":"2024-04-25T11:35:08.210783+0000","last_undegraded":"2024-04-25T11:35:08.210783+0000","last_fullsized":"2024-04-25T11:35:08.210783+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:30:00.620833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.324208+0000","last_change":"2024-04-25T11:33:10.897664+0000","last_active":"2024-04-25T11:35:08.324208+0000","last_peered":"2024-04-25T11:35:08.324208+0000","last_clean":"2024-04-25T11:35:08.324208+0000","last_became_active":"2024-04-25T11:33:10.897189+0000","last_became_peered":"2024-04-25T11:33:10.897189+0000","last_unstale":"2024-04-25T11:35:08.324208+0000","last_undegraded":"2024-04-25T11:35:08.324208+0000","last_fullsized":"2024-04-25T11:35:08.324208+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:12:06.168793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"77'3","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207308+0000","last_change":"2024-04-25T11:33:10.031417+0000","last_active":"2024-04-25T11:35:08.207308+0000","last_peered":"2024-04-25T11:35:08.207308+0000","last_clean":"2024-04-25T11:35:08.207308+0000","last_became_active":"2024-04-25T11:33:10.030526+0000","last_became_peered":"2024-04-25T11:33:10.030526+0000","last_unstale":"2024-04-25T11:35:08.207308+0000","last_undegraded":"2024-04-25T11:35:08.207308+0000","last_fullsized":"2024-04-25T11:35:08.207308+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:05.718720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.317017+0000","last_change":"2024-04-25T11:33:10.886596+0000","last_active":"2024-04-25T11:35:08.317017+0000","last_peered":"2024-04-25T11:35:08.317017+0000","last_clean":"2024-04-25T11:35:08.317017+0000","last_became_active":"2024-04-25T11:33:10.886234+0000","last_became_peered":"2024-04-25T11:33:10.886234+0000","last_unstale":"2024-04-25T11:35:08.317017+0000","last_undegraded":"2024-04-25T11:35:08.317017+0000","last_fullsized":"2024-04-25T11:35:08.317017+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:46.632814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"77'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.324040+0000","last_change":"2024-04-25T11:33:10.032355+0000","last_active":"2024-04-25T11:35:08.324040+0000","last_peered":"2024-04-25T11:35:08.324040+0000","last_clean":"2024-04-25T11:35:08.324040+0000","last_became_active":"2024-04-25T11:33:10.031965+0000","last_became_peered":"2024-04-25T11:33:10.031965+0000","last_unstale":"2024-04-25T11:35:08.324040+0000","last_undegraded":"2024-04-25T11:35:08.324040+0000","last_fullsized":"2024-04-25T11:35:08.324040+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:08:10.539604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"79'11","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.314255+0000","last_change":"2024-04-25T11:33:10.885708+0000","last_active":"2024-04-25T11:35:08.314255+0000","last_peered":"2024-04-25T11:35:08.314255+0000","last_clean":"2024-04-25T11:35:08.314255+0000","last_became_active":"2024-04-25T11:33:10.885317+0000","last_became_peered":"2024-04-25T11:33:10.885317+0000","last_unstale":"2024-04-25T11:35:08.314255+0000","last_undegraded":"2024-04-25T11:35:08.314255+0000","last_fullsized":"2024-04-25T11:35:08.314255+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:50:08.518922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"80'6","reported_seq":256,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.801798+0000","last_change":"2024-04-25T11:33:10.030995+0000","last_active":"2024-04-25T11:35:08.801798+0000","last_peered":"2024-04-25T11:35:08.801798+0000","last_clean":"2024-04-25T11:35:08.801798+0000","last_became_active":"2024-04-25T11:33:10.030670+0000","last_became_peered":"2024-04-25T11:33:10.030670+0000","last_unstale":"2024-04-25T11:35:08.801798+0000","last_undegraded":"2024-04-25T11:35:08.801798+0000","last_fullsized":"2024-04-25T11:35:08.801798+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:39.569787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":229,"num_read_kb":229,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"79'10","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.380745+0000","last_change":"2024-04-25T11:33:10.892927+0000","last_active":"2024-04-25T11:35:08.380745+0000","last_peered":"2024-04-25T11:35:08.380745+0000","last_clean":"2024-04-25T11:35:08.380745+0000","last_became_active":"2024-04-25T11:33:10.892458+0000","last_became_peered":"2024-04-25T11:33:10.892458+0000","last_unstale":"2024-04-25T11:35:08.380745+0000","last_undegraded":"2024-04-25T11:35:08.380745+0000","last_fullsized":"2024-04-25T11:35:08.380745+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:53.121185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"77'4","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501417+0000","last_change":"2024-04-25T11:33:10.029728+0000","last_active":"2024-04-25T11:35:08.501417+0000","last_peered":"2024-04-25T11:35:08.501417+0000","last_clean":"2024-04-25T11:35:08.501417+0000","last_became_active":"2024-04-25T11:33:10.029435+0000","last_became_peered":"2024-04-25T11:33:10.029435+0000","last_unstale":"2024-04-25T11:35:08.501417+0000","last_undegraded":"2024-04-25T11:35:08.501417+0000","last_fullsized":"2024-04-25T11:35:08.501417+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:31:21.585870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.502258+0000","last_change":"2024-04-25T11:33:10.895828+0000","last_active":"2024-04-25T11:35:08.502258+0000","last_peered":"2024-04-25T11:35:08.502258+0000","last_clean":"2024-04-25T11:35:08.502258+0000","last_became_active":"2024-04-25T11:33:10.895280+0000","last_became_peered":"2024-04-25T11:33:10.895280+0000","last_unstale":"2024-04-25T11:35:08.502258+0000","last_undegraded":"2024-04-25T11:35:08.502258+0000","last_fullsized":"2024-04-25T11:35:08.502258+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:03.009912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"77'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.324544+0000","last_change":"2024-04-25T11:33:10.036651+0000","last_active":"2024-04-25T11:35:08.324544+0000","last_peered":"2024-04-25T11:35:08.324544+0000","last_clean":"2024-04-25T11:35:08.324544+0000","last_became_active":"2024-04-25T11:33:10.036294+0000","last_became_peered":"2024-04-25T11:33:10.036294+0000","last_unstale":"2024-04-25T11:35:08.324544+0000","last_undegraded":"2024-04-25T11:35:08.324544+0000","last_fullsized":"2024-04-25T11:35:08.324544+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:12:36.566618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"79'11","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.380605+0000","last_change":"2024-04-25T11:33:10.890199+0000","last_active":"2024-04-25T11:35:08.380605+0000","last_peered":"2024-04-25T11:35:08.380605+0000","last_clean":"2024-04-25T11:35:08.380605+0000","last_became_active":"2024-04-25T11:33:10.889855+0000","last_became_peered":"2024-04-25T11:33:10.889855+0000","last_unstale":"2024-04-25T11:35:08.380605+0000","last_undegraded":"2024-04-25T11:35:08.380605+0000","last_fullsized":"2024-04-25T11:35:08.380605+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:54:45.476647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'37","reported_seq":319,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.511117+0000","last_change":"2024-04-25T11:34:18.309247+0000","last_active":"2024-04-25T11:35:08.511117+0000","last_peered":"2024-04-25T11:35:08.511117+0000","last_clean":"2024-04-25T11:35:08.511117+0000","last_became_active":"2024-04-25T11:34:18.299888+0000","last_became_peered":"2024-04-25T11:34:18.299888+0000","last_unstale":"2024-04-25T11:35:08.511117+0000","last_undegraded":"2024-04-25T11:35:08.511117+0000","last_fullsized":"2024-04-25T11:35:08.511117+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:36:06.603720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67758,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":235,"num_read_kb":178,"num_write":39,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"79'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.208993+0000","last_change":"2024-04-25T11:33:10.888596+0000","last_active":"2024-04-25T11:35:08.208993+0000","last_peered":"2024-04-25T11:35:08.208993+0000","last_clean":"2024-04-25T11:35:08.208993+0000","last_became_active":"2024-04-25T11:33:10.888226+0000","last_became_peered":"2024-04-25T11:33:10.888226+0000","last_unstale":"2024-04-25T11:35:08.208993+0000","last_undegraded":"2024-04-25T11:35:08.208993+0000","last_fullsized":"2024-04-25T11:35:08.208993+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:30:42.891788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"77'9","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500117+0000","last_change":"2024-04-25T11:33:10.026969+0000","last_active":"2024-04-25T11:35:08.500117+0000","last_peered":"2024-04-25T11:35:08.500117+0000","last_clean":"2024-04-25T11:35:08.500117+0000","last_became_active":"2024-04-25T11:33:10.026591+0000","last_became_peered":"2024-04-25T11:33:10.026591+0000","last_unstale":"2024-04-25T11:35:08.500117+0000","last_undegraded":"2024-04-25T11:35:08.500117+0000","last_fullsized":"2024-04-25T11:35:08.500117+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:50:34.269478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207191+0000","last_change":"2024-04-25T11:33:10.900303+0000","last_active":"2024-04-25T11:35:08.207191+0000","last_peered":"2024-04-25T11:35:08.207191+0000","last_clean":"2024-04-25T11:35:08.207191+0000","last_became_active":"2024-04-25T11:33:10.899946+0000","last_became_peered":"2024-04-25T11:33:10.899946+0000","last_unstale":"2024-04-25T11:35:08.207191+0000","last_undegraded":"2024-04-25T11:35:08.207191+0000","last_fullsized":"2024-04-25T11:35:08.207191+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:33:40.805400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"77'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.324114+0000","last_change":"2024-04-25T11:33:10.035668+0000","last_active":"2024-04-25T11:35:08.324114+0000","last_peered":"2024-04-25T11:35:08.324114+0000","last_clean":"2024-04-25T11:35:08.324114+0000","last_became_active":"2024-04-25T11:33:10.035241+0000","last_became_peered":"2024-04-25T11:33:10.035241+0000","last_unstale":"2024-04-25T11:35:08.324114+0000","last_undegraded":"2024-04-25T11:35:08.324114+0000","last_fullsized":"2024-04-25T11:35:08.324114+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:58:48.735577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"79'15","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500819+0000","last_change":"2024-04-25T11:33:10.894864+0000","last_active":"2024-04-25T11:35:08.500819+0000","last_peered":"2024-04-25T11:35:08.500819+0000","last_clean":"2024-04-25T11:35:08.500819+0000","last_became_active":"2024-04-25T11:33:10.894491+0000","last_became_peered":"2024-04-25T11:33:10.894491+0000","last_unstale":"2024-04-25T11:35:08.500819+0000","last_undegraded":"2024-04-25T11:35:08.500819+0000","last_fullsized":"2024-04-25T11:35:08.500819+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:21:28.256878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"80'8","reported_seq":146,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.816304+0000","last_change":"2024-04-25T11:33:10.029099+0000","last_active":"2024-04-25T11:35:08.816304+0000","last_peered":"2024-04-25T11:35:08.816304+0000","last_clean":"2024-04-25T11:35:08.816304+0000","last_became_active":"2024-04-25T11:33:10.028722+0000","last_became_peered":"2024-04-25T11:33:10.028722+0000","last_unstale":"2024-04-25T11:35:08.816304+0000","last_undegraded":"2024-04-25T11:35:08.816304+0000","last_fullsized":"2024-04-25T11:35:08.816304+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:44:12.261519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1511,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"79'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500425+0000","last_change":"2024-04-25T11:33:10.888635+0000","last_active":"2024-04-25T11:35:08.500425+0000","last_peered":"2024-04-25T11:35:08.500425+0000","last_clean":"2024-04-25T11:35:08.500425+0000","last_became_active":"2024-04-25T11:33:10.887544+0000","last_became_peered":"2024-04-25T11:33:10.887544+0000","last_unstale":"2024-04-25T11:35:08.500425+0000","last_undegraded":"2024-04-25T11:35:08.500425+0000","last_fullsized":"2024-04-25T11:35:08.500425+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:59:25.459587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"77'17","reported_seq":39,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.208662+0000","last_change":"2024-04-25T11:33:10.023566+0000","last_active":"2024-04-25T11:35:08.208662+0000","last_peered":"2024-04-25T11:35:08.208662+0000","last_clean":"2024-04-25T11:35:08.208662+0000","last_became_active":"2024-04-25T11:33:10.023234+0000","last_became_peered":"2024-04-25T11:33:10.023234+0000","last_unstale":"2024-04-25T11:35:08.208662+0000","last_undegraded":"2024-04-25T11:35:08.208662+0000","last_fullsized":"2024-04-25T11:35:08.208662+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:56:26.342148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"79'6","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.324020+0000","last_change":"2024-04-25T11:34:18.237663+0000","last_active":"2024-04-25T11:35:08.324020+0000","last_peered":"2024-04-25T11:35:08.324020+0000","last_clean":"2024-04-25T11:35:08.324020+0000","last_became_active":"2024-04-25T11:34:18.236993+0000","last_became_peered":"2024-04-25T11:34:18.236993+0000","last_unstale":"2024-04-25T11:35:08.324020+0000","last_undegraded":"2024-04-25T11:35:08.324020+0000","last_fullsized":"2024-04-25T11:35:08.324020+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:59:47.371685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.209203+0000","last_change":"2024-04-25T11:33:10.028285+0000","last_active":"2024-04-25T11:35:08.209203+0000","last_peered":"2024-04-25T11:35:08.209203+0000","last_clean":"2024-04-25T11:35:08.209203+0000","last_became_active":"2024-04-25T11:33:10.027959+0000","last_became_peered":"2024-04-25T11:33:10.027959+0000","last_unstale":"2024-04-25T11:35:08.209203+0000","last_undegraded":"2024-04-25T11:35:08.209203+0000","last_fullsized":"2024-04-25T11:35:08.209203+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:44:41.094048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"79'5","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.505018+0000","last_change":"2024-04-25T11:34:17.887356+0000","last_active":"2024-04-25T11:35:08.505018+0000","last_peered":"2024-04-25T11:35:08.505018+0000","last_clean":"2024-04-25T11:35:08.505018+0000","last_became_active":"2024-04-25T11:34:17.886981+0000","last_became_peered":"2024-04-25T11:34:17.886981+0000","last_unstale":"2024-04-25T11:35:08.505018+0000","last_undegraded":"2024-04-25T11:35:08.505018+0000","last_fullsized":"2024-04-25T11:35:08.505018+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:40.573546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"77'1","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.314543+0000","last_change":"2024-04-25T11:33:10.027473+0000","last_active":"2024-04-25T11:35:08.314543+0000","last_peered":"2024-04-25T11:35:08.314543+0000","last_clean":"2024-04-25T11:35:08.314543+0000","last_became_active":"2024-04-25T11:33:10.027133+0000","last_became_peered":"2024-04-25T11:33:10.027133+0000","last_unstale":"2024-04-25T11:35:08.314543+0000","last_undegraded":"2024-04-25T11:35:08.314543+0000","last_fullsized":"2024-04-25T11:35:08.314543+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:19:58.338507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"79'5","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.211036+0000","last_change":"2024-04-25T11:33:10.892588+0000","last_active":"2024-04-25T11:35:08.211036+0000","last_peered":"2024-04-25T11:35:08.211036+0000","last_clean":"2024-04-25T11:35:08.211036+0000","last_became_active":"2024-04-25T11:33:10.892088+0000","last_became_peered":"2024-04-25T11:33:10.892088+0000","last_unstale":"2024-04-25T11:35:08.211036+0000","last_undegraded":"2024-04-25T11:35:08.211036+0000","last_fullsized":"2024-04-25T11:35:08.211036+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:10.136222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"77'5","reported_seq":140,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500939+0000","last_change":"2024-04-25T11:33:10.030801+0000","last_active":"2024-04-25T11:35:08.500939+0000","last_peered":"2024-04-25T11:35:08.500939+0000","last_clean":"2024-04-25T11:35:08.500939+0000","last_became_active":"2024-04-25T11:33:10.030466+0000","last_became_peered":"2024-04-25T11:33:10.030466+0000","last_unstale":"2024-04-25T11:35:08.500939+0000","last_undegraded":"2024-04-25T11:35:08.500939+0000","last_fullsized":"2024-04-25T11:35:08.500939+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:22:02.922309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.211021+0000","last_change":"2024-04-25T11:33:10.894846+0000","last_active":"2024-04-25T11:35:08.211021+0000","last_peered":"2024-04-25T11:35:08.211021+0000","last_clean":"2024-04-25T11:35:08.211021+0000","last_became_active":"2024-04-25T11:33:10.894456+0000","last_became_peered":"2024-04-25T11:33:10.894456+0000","last_unstale":"2024-04-25T11:35:08.211021+0000","last_undegraded":"2024-04-25T11:35:08.211021+0000","last_fullsized":"2024-04-25T11:35:08.211021+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:11:16.265126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"79'8","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.324115+0000","last_change":"2024-04-25T11:34:17.882781+0000","last_active":"2024-04-25T11:35:08.324115+0000","last_peered":"2024-04-25T11:35:08.324115+0000","last_clean":"2024-04-25T11:35:08.324115+0000","last_became_active":"2024-04-25T11:34:17.882348+0000","last_became_peered":"2024-04-25T11:34:17.882348+0000","last_unstale":"2024-04-25T11:35:08.324115+0000","last_undegraded":"2024-04-25T11:35:08.324115+0000","last_fullsized":"2024-04-25T11:35:08.324115+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:47:34.385332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"80'7","reported_seq":142,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500831+0000","last_change":"2024-04-25T11:33:10.037834+0000","last_active":"2024-04-25T11:35:08.500831+0000","last_peered":"2024-04-25T11:35:08.500831+0000","last_clean":"2024-04-25T11:35:08.500831+0000","last_became_active":"2024-04-25T11:33:10.037570+0000","last_became_peered":"2024-04-25T11:33:10.037570+0000","last_unstale":"2024-04-25T11:35:08.500831+0000","last_undegraded":"2024-04-25T11:35:08.500831+0000","last_fullsized":"2024-04-25T11:35:08.500831+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:12.795190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.317600+0000","last_change":"2024-04-25T11:33:10.882663+0000","last_active":"2024-04-25T11:35:08.317600+0000","last_peered":"2024-04-25T11:35:08.317600+0000","last_clean":"2024-04-25T11:35:08.317600+0000","last_became_active":"2024-04-25T11:33:10.882300+0000","last_became_peered":"2024-04-25T11:33:10.882300+0000","last_unstale":"2024-04-25T11:35:08.317600+0000","last_undegraded":"2024-04-25T11:35:08.317600+0000","last_fullsized":"2024-04-25T11:35:08.317600+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:58:11.379654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"77'6","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500572+0000","last_change":"2024-04-25T11:33:10.024598+0000","last_active":"2024-04-25T11:35:08.500572+0000","last_peered":"2024-04-25T11:35:08.500572+0000","last_clean":"2024-04-25T11:35:08.500572+0000","last_became_active":"2024-04-25T11:33:10.024229+0000","last_became_peered":"2024-04-25T11:33:10.024229+0000","last_unstale":"2024-04-25T11:35:08.500572+0000","last_undegraded":"2024-04-25T11:35:08.500572+0000","last_fullsized":"2024-04-25T11:35:08.500572+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:42:14.690581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":98,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500206+0000","last_change":"2024-04-25T11:32:01.310202+0000","last_active":"2024-04-25T11:35:08.500206+0000","last_peered":"2024-04-25T11:35:08.500206+0000","last_clean":"2024-04-25T11:35:08.500206+0000","last_became_active":"2024-04-25T11:32:01.305242+0000","last_became_peered":"2024-04-25T11:32:01.305242+0000","last_unstale":"2024-04-25T11:35:08.500206+0000","last_undegraded":"2024-04-25T11:35:08.500206+0000","last_fullsized":"2024-04-25T11:35:08.500206+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:28:43.374096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:28:43.374096+0000","last_clean_scrub_stamp":"2024-04-25T11:28:43.374096+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:56:18.551747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"79'5","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207126+0000","last_change":"2024-04-25T11:33:10.889124+0000","last_active":"2024-04-25T11:35:08.207126+0000","last_peered":"2024-04-25T11:35:08.207126+0000","last_clean":"2024-04-25T11:35:08.207126+0000","last_became_active":"2024-04-25T11:33:10.888790+0000","last_became_peered":"2024-04-25T11:33:10.888790+0000","last_unstale":"2024-04-25T11:35:08.207126+0000","last_undegraded":"2024-04-25T11:35:08.207126+0000","last_fullsized":"2024-04-25T11:35:08.207126+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:33:44.999949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"77'4","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500562+0000","last_change":"2024-04-25T11:33:10.031547+0000","last_active":"2024-04-25T11:35:08.500562+0000","last_peered":"2024-04-25T11:35:08.500562+0000","last_clean":"2024-04-25T11:35:08.500562+0000","last_became_active":"2024-04-25T11:33:10.031152+0000","last_became_peered":"2024-04-25T11:33:10.031152+0000","last_unstale":"2024-04-25T11:35:08.500562+0000","last_undegraded":"2024-04-25T11:35:08.500562+0000","last_fullsized":"2024-04-25T11:35:08.500562+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:10.646771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"77'4","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.209106+0000","last_change":"2024-04-25T11:33:10.020367+0000","last_active":"2024-04-25T11:35:08.209106+0000","last_peered":"2024-04-25T11:35:08.209106+0000","last_clean":"2024-04-25T11:35:08.209106+0000","last_became_active":"2024-04-25T11:33:10.020007+0000","last_became_peered":"2024-04-25T11:33:10.020007+0000","last_unstale":"2024-04-25T11:35:08.209106+0000","last_undegraded":"2024-04-25T11:35:08.209106+0000","last_fullsized":"2024-04-25T11:35:08.209106+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:48.134907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207404+0000","last_change":"2024-04-25T11:33:10.887741+0000","last_active":"2024-04-25T11:35:08.207404+0000","last_peered":"2024-04-25T11:35:08.207404+0000","last_clean":"2024-04-25T11:35:08.207404+0000","last_became_active":"2024-04-25T11:33:10.886455+0000","last_became_peered":"2024-04-25T11:33:10.886455+0000","last_unstale":"2024-04-25T11:35:08.207404+0000","last_undegraded":"2024-04-25T11:35:08.207404+0000","last_fullsized":"2024-04-25T11:35:08.207404+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:37:16.842817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'48","reported_seq":92,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.448574+0000","last_change":"2024-04-25T11:33:10.025672+0000","last_active":"2024-04-25T11:35:08.448574+0000","last_peered":"2024-04-25T11:35:08.448574+0000","last_clean":"2024-04-25T11:35:08.448574+0000","last_became_active":"2024-04-25T11:33:10.025232+0000","last_became_peered":"2024-04-25T11:33:10.025232+0000","last_unstale":"2024-04-25T11:35:08.448574+0000","last_undegraded":"2024-04-25T11:35:08.448574+0000","last_fullsized":"2024-04-25T11:35:08.448574+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:06:26.579126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":84,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"79'5","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.317930+0000","last_change":"2024-04-25T11:33:10.884692+0000","last_active":"2024-04-25T11:35:08.317930+0000","last_peered":"2024-04-25T11:35:08.317930+0000","last_clean":"2024-04-25T11:35:08.317930+0000","last_became_active":"2024-04-25T11:33:10.884346+0000","last_became_peered":"2024-04-25T11:33:10.884346+0000","last_unstale":"2024-04-25T11:35:08.317930+0000","last_undegraded":"2024-04-25T11:35:08.317930+0000","last_fullsized":"2024-04-25T11:35:08.317930+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:31:00.630410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"77'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500890+0000","last_change":"2024-04-25T11:33:10.033872+0000","last_active":"2024-04-25T11:35:08.500890+0000","last_peered":"2024-04-25T11:35:08.500890+0000","last_clean":"2024-04-25T11:35:08.500890+0000","last_became_active":"2024-04-25T11:33:10.033435+0000","last_became_peered":"2024-04-25T11:33:10.033435+0000","last_unstale":"2024-04-25T11:35:08.500890+0000","last_undegraded":"2024-04-25T11:35:08.500890+0000","last_fullsized":"2024-04-25T11:35:08.500890+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:12.885215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"79'7","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501866+0000","last_change":"2024-04-25T11:34:17.881545+0000","last_active":"2024-04-25T11:35:08.501866+0000","last_peered":"2024-04-25T11:35:08.501866+0000","last_clean":"2024-04-25T11:35:08.501866+0000","last_became_active":"2024-04-25T11:34:17.881125+0000","last_became_peered":"2024-04-25T11:34:17.881125+0000","last_unstale":"2024-04-25T11:35:08.501866+0000","last_undegraded":"2024-04-25T11:35:08.501866+0000","last_fullsized":"2024-04-25T11:35:08.501866+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:55:18.404023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"77'2","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.380513+0000","last_change":"2024-04-25T11:33:10.032384+0000","last_active":"2024-04-25T11:35:08.380513+0000","last_peered":"2024-04-25T11:35:08.380513+0000","last_clean":"2024-04-25T11:35:08.380513+0000","last_became_active":"2024-04-25T11:33:10.032035+0000","last_became_peered":"2024-04-25T11:33:10.032035+0000","last_unstale":"2024-04-25T11:35:08.380513+0000","last_undegraded":"2024-04-25T11:35:08.380513+0000","last_fullsized":"2024-04-25T11:35:08.380513+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:21:41.723120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"79'2","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.208679+0000","last_change":"2024-04-25T11:33:10.893366+0000","last_active":"2024-04-25T11:35:08.208679+0000","last_peered":"2024-04-25T11:35:08.208679+0000","last_clean":"2024-04-25T11:35:08.208679+0000","last_became_active":"2024-04-25T11:33:10.892988+0000","last_became_peered":"2024-04-25T11:33:10.892988+0000","last_unstale":"2024-04-25T11:35:08.208679+0000","last_undegraded":"2024-04-25T11:35:08.208679+0000","last_fullsized":"2024-04-25T11:35:08.208679+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:06.368216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"77'7","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.317231+0000","last_change":"2024-04-25T11:33:10.031456+0000","last_active":"2024-04-25T11:35:08.317231+0000","last_peered":"2024-04-25T11:35:08.317231+0000","last_clean":"2024-04-25T11:35:08.317231+0000","last_became_active":"2024-04-25T11:33:10.031115+0000","last_became_peered":"2024-04-25T11:33:10.031115+0000","last_unstale":"2024-04-25T11:35:08.317231+0000","last_undegraded":"2024-04-25T11:35:08.317231+0000","last_fullsized":"2024-04-25T11:35:08.317231+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:12:56.803277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.504698+0000","last_change":"2024-04-25T11:33:10.893692+0000","last_active":"2024-04-25T11:35:08.504698+0000","last_peered":"2024-04-25T11:35:08.504698+0000","last_clean":"2024-04-25T11:35:08.504698+0000","last_became_active":"2024-04-25T11:33:10.893332+0000","last_became_peered":"2024-04-25T11:33:10.893332+0000","last_unstale":"2024-04-25T11:35:08.504698+0000","last_undegraded":"2024-04-25T11:35:08.504698+0000","last_fullsized":"2024-04-25T11:35:08.504698+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:01.277465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"80'5","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.641604+0000","last_change":"2024-04-25T11:33:10.023804+0000","last_active":"2024-04-25T11:35:08.641604+0000","last_peered":"2024-04-25T11:35:08.641604+0000","last_clean":"2024-04-25T11:35:08.641604+0000","last_became_active":"2024-04-25T11:33:10.023503+0000","last_became_peered":"2024-04-25T11:33:10.023503+0000","last_unstale":"2024-04-25T11:35:08.641604+0000","last_undegraded":"2024-04-25T11:35:08.641604+0000","last_fullsized":"2024-04-25T11:35:08.641604+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:34:48.074176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"79'8","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.314248+0000","last_change":"2024-04-25T11:34:17.882901+0000","last_active":"2024-04-25T11:35:08.314248+0000","last_peered":"2024-04-25T11:35:08.314248+0000","last_clean":"2024-04-25T11:35:08.314248+0000","last_became_active":"2024-04-25T11:34:17.882514+0000","last_became_peered":"2024-04-25T11:34:17.882514+0000","last_unstale":"2024-04-25T11:35:08.314248+0000","last_undegraded":"2024-04-25T11:35:08.314248+0000","last_fullsized":"2024-04-25T11:35:08.314248+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:03:30.843842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,10],"acting":[9,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.b","version":"77'4","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.208848+0000","last_change":"2024-04-25T11:33:10.030857+0000","last_active":"2024-04-25T11:35:08.208848+0000","last_peered":"2024-04-25T11:35:08.208848+0000","last_clean":"2024-04-25T11:35:08.208848+0000","last_became_active":"2024-04-25T11:33:10.028879+0000","last_became_peered":"2024-04-25T11:33:10.028879+0000","last_unstale":"2024-04-25T11:35:08.208848+0000","last_undegraded":"2024-04-25T11:35:08.208848+0000","last_fullsized":"2024-04-25T11:35:08.208848+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:28.820919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"79'5","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500748+0000","last_change":"2024-04-25T11:33:10.893773+0000","last_active":"2024-04-25T11:35:08.500748+0000","last_peered":"2024-04-25T11:35:08.500748+0000","last_clean":"2024-04-25T11:35:08.500748+0000","last_became_active":"2024-04-25T11:33:10.893431+0000","last_became_peered":"2024-04-25T11:33:10.893431+0000","last_unstale":"2024-04-25T11:35:08.500748+0000","last_undegraded":"2024-04-25T11:35:08.500748+0000","last_fullsized":"2024-04-25T11:35:08.500748+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:15:15.998951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"80'7","reported_seq":144,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.807310+0000","last_change":"2024-04-25T11:33:10.032745+0000","last_active":"2024-04-25T11:35:08.807310+0000","last_peered":"2024-04-25T11:35:08.807310+0000","last_clean":"2024-04-25T11:35:08.807310+0000","last_became_active":"2024-04-25T11:33:10.032361+0000","last_became_peered":"2024-04-25T11:33:10.032361+0000","last_unstale":"2024-04-25T11:35:08.807310+0000","last_undegraded":"2024-04-25T11:35:08.807310+0000","last_fullsized":"2024-04-25T11:35:08.807310+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:14:46.555892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1952,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":116,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501571+0000","last_change":"2024-04-25T11:33:10.888084+0000","last_active":"2024-04-25T11:35:08.501571+0000","last_peered":"2024-04-25T11:35:08.501571+0000","last_clean":"2024-04-25T11:35:08.501571+0000","last_became_active":"2024-04-25T11:33:10.887679+0000","last_became_peered":"2024-04-25T11:33:10.887679+0000","last_unstale":"2024-04-25T11:35:08.501571+0000","last_undegraded":"2024-04-25T11:35:08.501571+0000","last_fullsized":"2024-04-25T11:35:08.501571+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:10:20.073630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"80'41","reported_seq":70,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.445891+0000","last_change":"2024-04-25T11:33:10.035355+0000","last_active":"2024-04-25T11:35:08.445891+0000","last_peered":"2024-04-25T11:35:08.445891+0000","last_clean":"2024-04-25T11:35:08.445891+0000","last_became_active":"2024-04-25T11:33:10.034724+0000","last_became_peered":"2024-04-25T11:33:10.034724+0000","last_unstale":"2024-04-25T11:35:08.445891+0000","last_undegraded":"2024-04-25T11:35:08.445891+0000","last_fullsized":"2024-04-25T11:35:08.445891+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:24:26.757803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500628+0000","last_change":"2024-04-25T11:33:10.899827+0000","last_active":"2024-04-25T11:35:08.500628+0000","last_peered":"2024-04-25T11:35:08.500628+0000","last_clean":"2024-04-25T11:35:08.500628+0000","last_became_active":"2024-04-25T11:33:10.899356+0000","last_became_peered":"2024-04-25T11:33:10.899356+0000","last_unstale":"2024-04-25T11:35:08.500628+0000","last_undegraded":"2024-04-25T11:35:08.500628+0000","last_fullsized":"2024-04-25T11:35:08.500628+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:04:17.323958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"77'5","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207700+0000","last_change":"2024-04-25T11:33:10.034981+0000","last_active":"2024-04-25T11:35:08.207700+0000","last_peered":"2024-04-25T11:35:08.207700+0000","last_clean":"2024-04-25T11:35:08.207700+0000","last_became_active":"2024-04-25T11:33:10.034601+0000","last_became_peered":"2024-04-25T11:33:10.034601+0000","last_unstale":"2024-04-25T11:35:08.207700+0000","last_undegraded":"2024-04-25T11:35:08.207700+0000","last_fullsized":"2024-04-25T11:35:08.207700+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:19:14.710731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501693+0000","last_change":"2024-04-25T11:33:10.890020+0000","last_active":"2024-04-25T11:35:08.501693+0000","last_peered":"2024-04-25T11:35:08.501693+0000","last_clean":"2024-04-25T11:35:08.501693+0000","last_became_active":"2024-04-25T11:33:10.889558+0000","last_became_peered":"2024-04-25T11:33:10.889558+0000","last_unstale":"2024-04-25T11:35:08.501693+0000","last_undegraded":"2024-04-25T11:35:08.501693+0000","last_fullsized":"2024-04-25T11:35:08.501693+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:00:49.096818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.504776+0000","last_change":"2024-04-25T11:33:10.886965+0000","last_active":"2024-04-25T11:35:08.504776+0000","last_peered":"2024-04-25T11:35:08.504776+0000","last_clean":"2024-04-25T11:35:08.504776+0000","last_became_active":"2024-04-25T11:33:10.886594+0000","last_became_peered":"2024-04-25T11:33:10.886594+0000","last_unstale":"2024-04-25T11:35:08.504776+0000","last_undegraded":"2024-04-25T11:35:08.504776+0000","last_fullsized":"2024-04-25T11:35:08.504776+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:58.924857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"79'12","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.499974+0000","last_change":"2024-04-25T11:33:10.890760+0000","last_active":"2024-04-25T11:35:08.499974+0000","last_peered":"2024-04-25T11:35:08.499974+0000","last_clean":"2024-04-25T11:35:08.499974+0000","last_became_active":"2024-04-25T11:33:10.889643+0000","last_became_peered":"2024-04-25T11:33:10.889643+0000","last_unstale":"2024-04-25T11:35:08.499974+0000","last_undegraded":"2024-04-25T11:35:08.499974+0000","last_fullsized":"2024-04-25T11:35:08.499974+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:54.644853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.499871+0000","last_change":"2024-04-25T11:33:10.887850+0000","last_active":"2024-04-25T11:35:08.499871+0000","last_peered":"2024-04-25T11:35:08.499871+0000","last_clean":"2024-04-25T11:35:08.499871+0000","last_became_active":"2024-04-25T11:33:10.882323+0000","last_became_peered":"2024-04-25T11:33:10.882323+0000","last_unstale":"2024-04-25T11:35:08.499871+0000","last_undegraded":"2024-04-25T11:35:08.499871+0000","last_fullsized":"2024-04-25T11:35:08.499871+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:24:00.509704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501134+0000","last_change":"2024-04-25T11:33:10.883842+0000","last_active":"2024-04-25T11:35:08.501134+0000","last_peered":"2024-04-25T11:35:08.501134+0000","last_clean":"2024-04-25T11:35:08.501134+0000","last_became_active":"2024-04-25T11:33:10.883513+0000","last_became_peered":"2024-04-25T11:33:10.883513+0000","last_unstale":"2024-04-25T11:35:08.501134+0000","last_undegraded":"2024-04-25T11:35:08.501134+0000","last_fullsized":"2024-04-25T11:35:08.501134+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:47:09.731875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"79'4","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207357+0000","last_change":"2024-04-25T11:33:10.890503+0000","last_active":"2024-04-25T11:35:08.207357+0000","last_peered":"2024-04-25T11:35:08.207357+0000","last_clean":"2024-04-25T11:35:08.207357+0000","last_became_active":"2024-04-25T11:33:10.890162+0000","last_became_peered":"2024-04-25T11:33:10.890162+0000","last_unstale":"2024-04-25T11:35:08.207357+0000","last_undegraded":"2024-04-25T11:35:08.207357+0000","last_fullsized":"2024-04-25T11:35:08.207357+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:36.462789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"79'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.504912+0000","last_change":"2024-04-25T11:33:10.897109+0000","last_active":"2024-04-25T11:35:08.504912+0000","last_peered":"2024-04-25T11:35:08.504912+0000","last_clean":"2024-04-25T11:35:08.504912+0000","last_became_active":"2024-04-25T11:33:10.896138+0000","last_became_peered":"2024-04-25T11:33:10.896138+0000","last_unstale":"2024-04-25T11:35:08.504912+0000","last_undegraded":"2024-04-25T11:35:08.504912+0000","last_fullsized":"2024-04-25T11:35:08.504912+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:47:46.202355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"79'15","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500379+0000","last_change":"2024-04-25T11:33:10.890397+0000","last_active":"2024-04-25T11:35:08.500379+0000","last_peered":"2024-04-25T11:35:08.500379+0000","last_clean":"2024-04-25T11:35:08.500379+0000","last_became_active":"2024-04-25T11:33:10.889580+0000","last_became_peered":"2024-04-25T11:33:10.889580+0000","last_unstale":"2024-04-25T11:35:08.500379+0000","last_undegraded":"2024-04-25T11:35:08.500379+0000","last_fullsized":"2024-04-25T11:35:08.500379+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:16:09.672131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"79'2","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.317841+0000","last_change":"2024-04-25T11:33:10.873817+0000","last_active":"2024-04-25T11:35:08.317841+0000","last_peered":"2024-04-25T11:35:08.317841+0000","last_clean":"2024-04-25T11:35:08.317841+0000","last_became_active":"2024-04-25T11:33:10.873425+0000","last_became_peered":"2024-04-25T11:33:10.873425+0000","last_unstale":"2024-04-25T11:35:08.317841+0000","last_undegraded":"2024-04-25T11:35:08.317841+0000","last_fullsized":"2024-04-25T11:35:08.317841+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:52:26.279705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"79'6","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207126+0000","last_change":"2024-04-25T11:34:17.880075+0000","last_active":"2024-04-25T11:35:08.207126+0000","last_peered":"2024-04-25T11:35:08.207126+0000","last_clean":"2024-04-25T11:35:08.207126+0000","last_became_active":"2024-04-25T11:34:17.879692+0000","last_became_peered":"2024-04-25T11:34:17.879692+0000","last_unstale":"2024-04-25T11:35:08.207126+0000","last_undegraded":"2024-04-25T11:35:08.207126+0000","last_fullsized":"2024-04-25T11:35:08.207126+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:54.833601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,3],"acting":[6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.29","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.317356+0000","last_change":"2024-04-25T11:33:10.883690+0000","last_active":"2024-04-25T11:35:08.317356+0000","last_peered":"2024-04-25T11:35:08.317356+0000","last_clean":"2024-04-25T11:35:08.317356+0000","last_became_active":"2024-04-25T11:33:10.882965+0000","last_became_peered":"2024-04-25T11:33:10.882965+0000","last_unstale":"2024-04-25T11:35:08.317356+0000","last_undegraded":"2024-04-25T11:35:08.317356+0000","last_fullsized":"2024-04-25T11:35:08.317356+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:28.126631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"79'10","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.380742+0000","last_change":"2024-04-25T11:33:10.897050+0000","last_active":"2024-04-25T11:35:08.380742+0000","last_peered":"2024-04-25T11:35:08.380742+0000","last_clean":"2024-04-25T11:35:08.380742+0000","last_became_active":"2024-04-25T11:33:10.896273+0000","last_became_peered":"2024-04-25T11:33:10.896273+0000","last_unstale":"2024-04-25T11:35:08.380742+0000","last_undegraded":"2024-04-25T11:35:08.380742+0000","last_fullsized":"2024-04-25T11:35:08.380742+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:45:53.839823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"79'11","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207505+0000","last_change":"2024-04-25T11:33:10.897323+0000","last_active":"2024-04-25T11:35:08.207505+0000","last_peered":"2024-04-25T11:35:08.207505+0000","last_clean":"2024-04-25T11:35:08.207505+0000","last_became_active":"2024-04-25T11:33:10.896951+0000","last_became_peered":"2024-04-25T11:33:10.896951+0000","last_unstale":"2024-04-25T11:35:08.207505+0000","last_undegraded":"2024-04-25T11:35:08.207505+0000","last_fullsized":"2024-04-25T11:35:08.207505+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:24:31.208338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.504797+0000","last_change":"2024-04-25T11:33:10.897566+0000","last_active":"2024-04-25T11:35:08.504797+0000","last_peered":"2024-04-25T11:35:08.504797+0000","last_clean":"2024-04-25T11:35:08.504797+0000","last_became_active":"2024-04-25T11:33:10.897212+0000","last_became_peered":"2024-04-25T11:33:10.897212+0000","last_unstale":"2024-04-25T11:35:08.504797+0000","last_undegraded":"2024-04-25T11:35:08.504797+0000","last_fullsized":"2024-04-25T11:35:08.504797+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:55:29.995287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":601791,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1272,"num_read_kb":747,"num_write":370,"num_write_kb":1464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1805373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":299,"ondisk_log_size":299,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645348,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36204,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708948,"statfs":{"total":95995035648,"available":95957962752,"internally_reserved":0,"allocated":884736,"data_stored":423632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841577,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37024,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708128,"statfs":{"total":95995035648,"available":95957123072,"internally_reserved":0,"allocated":1728512,"data_stored":1214160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037805,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1302528,"data_stored":815996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234035,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28288,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716864,"statfs":{"total":95995035648,"available":95966068736,"internally_reserved":0,"allocated":1298432,"data_stored":816374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430264,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36788,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708364,"statfs":{"total":95995035648,"available":95957364736,"internally_reserved":0,"allocated":1486848,"data_stored":1007140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626493,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32052,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713100,"statfs":{"total":95995035648,"available":95962214400,"internally_reserved":0,"allocated":892928,"data_stored":422497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":962560,"data_stored":491168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":424618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40364,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704788,"statfs":{"total":95995035648,"available":95953702912,"internally_reserved":0,"allocated":888832,"data_stored":414953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28420,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716732,"statfs":{"total":95995035648,"available":95965933568,"internally_reserved":0,"allocated":1437696,"data_stored":947138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986247,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":970752,"data_stored":491039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822722,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28480,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716672,"statfs":{"total":95995035648,"available":95965872128,"internally_reserved":0,"allocated":1499136,"data_stored":1006286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":69510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":69381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":135510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":394716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":202017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T11:35:10.138 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-25T11:35:10.138 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T11:35:10.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.312+0000 7f9311ffb640 1 -- 172.21.15.90:0/3381659685 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c030d70 con 0x7f9314110440 2024-04-25T11:35:10.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.313+0000 7f9319a42640 1 -- 172.21.15.90:0/3381659685 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9314110440 msgr2=0x7f9314112860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:10.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.313+0000 7f9319a42640 1 --2- 172.21.15.90:0/3381659685 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9314110440 0x7f9314112860 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f930c009be0 tx=0x7f930c02a6f0 comp rx=0 tx=0).stop 2024-04-25T11:35:10.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.313+0000 7f9319a42640 1 -- 172.21.15.90:0/3381659685 shutdown_connections 2024-04-25T11:35:10.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.313+0000 7f9319a42640 1 --2- 172.21.15.90:0/3381659685 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9314110440 0x7f9314112860 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:10.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.313+0000 7f9319a42640 1 --2- 172.21.15.90:0/3381659685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f931410b740 0x7f931410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:10.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.313+0000 7f9319a42640 1 --2- 172.21.15.90:0/3381659685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f931410a500 0x7f931410b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:10.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.313+0000 7f9319a42640 1 -- 172.21.15.90:0/3381659685 >> 172.21.15.90:0/3381659685 conn(0x7f9314070920 msgr2=0x7f9314072dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:10.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.313+0000 7f9319a42640 1 -- 172.21.15.90:0/3381659685 shutdown_connections 2024-04-25T11:35:10.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.313+0000 7f9319a42640 1 -- 172.21.15.90:0/3381659685 wait complete. 2024-04-25T11:35:10.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.314+0000 7f9319a42640 1 Processor -- start 2024-04-25T11:35:10.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.314+0000 7f9319a42640 1 -- start start 2024-04-25T11:35:10.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.314+0000 7f9319a42640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f931410a500 0x7f93141a90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:10.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.314+0000 7f9319a42640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f931410b740 0x7f93141a9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:10.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.314+0000 7f9319a42640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9314110440 0x7f93141b05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:10.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.315+0000 7f9319a42640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9314115c90 con 0x7f931410a500 2024-04-25T11:35:10.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.315+0000 7f9319a42640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9314115b10 con 0x7f9314110440 2024-04-25T11:35:10.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.315+0000 7f9319a42640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9314115e10 con 0x7f931410b740 2024-04-25T11:35:10.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.315+0000 7f93127fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f931410b740 0x7f93141a9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:10.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.315+0000 7f93127fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f931410b740 0x7f93141a9610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:48542/0 (socket says 172.21.15.90:48542) 2024-04-25T11:35:10.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.315+0000 7f93127fc640 1 -- 172.21.15.90:0/105125684 learned_addr learned my addr 172.21.15.90:0/105125684 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:10.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.315+0000 7f9312ffd640 1 --2- 172.21.15.90:0/105125684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f931410a500 0x7f93141a90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:10.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.315+0000 7f93137fe640 1 --2- 172.21.15.90:0/105125684 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9314110440 0x7f93141b05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:10.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.315+0000 7f93127fc640 1 -- 172.21.15.90:0/105125684 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9314110440 msgr2=0x7f93141b05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:10.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.315+0000 7f93127fc640 1 --2- 172.21.15.90:0/105125684 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9314110440 0x7f93141b05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:10.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.315+0000 7f93127fc640 1 -- 172.21.15.90:0/105125684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f931410a500 msgr2=0x7f93141a90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:10.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.315+0000 7f93127fc640 1 --2- 172.21.15.90:0/105125684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f931410a500 0x7f93141a90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:10.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.315+0000 7f93127fc640 1 -- 172.21.15.90:0/105125684 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93141b0c10 con 0x7f931410b740 2024-04-25T11:35:10.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.315+0000 7f93127fc640 1 --2- 172.21.15.90:0/105125684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f931410b740 0x7f93141a9610 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9304007820 tx=0x7f9304007c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:10.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.315+0000 7f9318a40640 1 -- 172.21.15.90:0/105125684 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9304017020 con 0x7f931410b740 2024-04-25T11:35:10.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.316+0000 7f9318a40640 1 -- 172.21.15.90:0/105125684 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9304013400 con 0x7f931410b740 2024-04-25T11:35:10.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.316+0000 7f9318a40640 1 -- 172.21.15.90:0/105125684 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93040125c0 con 0x7f931410b740 2024-04-25T11:35:10.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.316+0000 7f9319a42640 1 -- 172.21.15.90:0/105125684 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93141b0f00 con 0x7f931410b740 2024-04-25T11:35:10.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.316+0000 7f9319a42640 1 -- 172.21.15.90:0/105125684 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93141b1440 con 0x7f931410b740 2024-04-25T11:35:10.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.317+0000 7f9319a42640 1 -- 172.21.15.90:0/105125684 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9314069d40 con 0x7f931410b740 2024-04-25T11:35:10.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.318+0000 7f9318a40640 1 -- 172.21.15.90:0/105125684 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 22) ==== 147640+0+0 (secure 0 0 0) 0x7f930401a020 con 0x7f931410b740 2024-04-25T11:35:10.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.319+0000 7f9318a40640 1 --2- 172.21.15.90:0/105125684 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f92fc0afd10 0x7f92fc0b2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:10.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.319+0000 7f9318a40640 1 -- 172.21.15.90:0/105125684 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f93040dec80 con 0x7f931410b740 2024-04-25T11:35:10.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.319+0000 7f9312ffd640 1 --2- 172.21.15.90:0/105125684 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f92fc0afd10 0x7f92fc0b2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:10.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.320+0000 7f9312ffd640 1 --2- 172.21.15.90:0/105125684 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f92fc0afd10 0x7f92fc0b2220 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f931410b100 tx=0x7f9308009110 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:10.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.324+0000 7f9318a40640 1 -- 172.21.15.90:0/105125684 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9304021020 con 0x7f931410b740 2024-04-25T11:35:10.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:10 smithi090 ceph-mon[29575]: [25/Apr/2024:11:35:09] ENGINE Bus STARTING 2024-04-25T11:35:10.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:10 smithi090 ceph-mon[29575]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:35:10.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:10 smithi090 ceph-mon[29575]: [25/Apr/2024:11:35:09] ENGINE Serving on https://172.21.15.90:7150 2024-04-25T11:35:10.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:10 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3509944741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:35:10.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:10 smithi090 ceph-mon[29575]: [25/Apr/2024:11:35:09] ENGINE Serving on http://172.21.15.90:8765 2024-04-25T11:35:10.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:10 smithi090 ceph-mon[29575]: [25/Apr/2024:11:35:09] ENGINE Bus STARTED 2024-04-25T11:35:10.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:10 smithi090 ceph-mon[29575]: from='client.25070 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T11:35:10.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.545+0000 7f9319a42640 1 -- 172.21.15.90:0/105125684 --> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9314062fe0 con 0x7f92fc0afd10 2024-04-25T11:35:10.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.552+0000 7f9318a40640 1 -- 172.21.15.90:0/105125684 <== mgr.15366 v2:172.21.15.90:6800/4078377640 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257767 (secure 0 0 0) 0x7f9314062fe0 con 0x7f92fc0afd10 2024-04-25T11:35:10.543 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:35:10.545 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-25T11:35:10.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.555+0000 7f9319a42640 1 -- 172.21.15.90:0/105125684 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f92fc0afd10 msgr2=0x7f92fc0b2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:10.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.555+0000 7f9319a42640 1 --2- 172.21.15.90:0/105125684 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f92fc0afd10 0x7f92fc0b2220 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f931410b100 tx=0x7f9308009110 comp rx=0 tx=0).stop 2024-04-25T11:35:10.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.555+0000 7f9319a42640 1 -- 172.21.15.90:0/105125684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f931410b740 msgr2=0x7f93141a9610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:10.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.555+0000 7f9319a42640 1 --2- 172.21.15.90:0/105125684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f931410b740 0x7f93141a9610 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9304007820 tx=0x7f9304007c20 comp rx=0 tx=0).stop 2024-04-25T11:35:10.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.555+0000 7f9319a42640 1 -- 172.21.15.90:0/105125684 shutdown_connections 2024-04-25T11:35:10.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.555+0000 7f9319a42640 1 --2- 172.21.15.90:0/105125684 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f92fc0afd10 0x7f92fc0b2220 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:10.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.555+0000 7f9319a42640 1 --2- 172.21.15.90:0/105125684 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9314110440 0x7f93141b05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:10.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.555+0000 7f9319a42640 1 --2- 172.21.15.90:0/105125684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f931410b740 0x7f93141a9610 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:10.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.555+0000 7f9319a42640 1 --2- 172.21.15.90:0/105125684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f931410a500 0x7f93141a90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:10.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.555+0000 7f9319a42640 1 -- 172.21.15.90:0/105125684 >> 172.21.15.90:0/105125684 conn(0x7f9314070920 msgr2=0x7f931410c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:10.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.555+0000 7f9319a42640 1 -- 172.21.15.90:0/105125684 shutdown_connections 2024-04-25T11:35:10.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.555+0000 7f9319a42640 1 -- 172.21.15.90:0/105125684 wait complete. 2024-04-25T11:35:10.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:10 smithi138 ceph-mon[32328]: [25/Apr/2024:11:35:09] ENGINE Bus STARTING 2024-04-25T11:35:10.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:10 smithi138 ceph-mon[32328]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:35:10.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:10 smithi138 ceph-mon[32328]: [25/Apr/2024:11:35:09] ENGINE Serving on https://172.21.15.90:7150 2024-04-25T11:35:10.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:10 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3509944741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:35:10.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:10 smithi138 ceph-mon[32328]: [25/Apr/2024:11:35:09] ENGINE Serving on http://172.21.15.90:8765 2024-04-25T11:35:10.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:10 smithi138 ceph-mon[32328]: [25/Apr/2024:11:35:09] ENGINE Bus STARTED 2024-04-25T11:35:10.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:10 smithi138 ceph-mon[32328]: from='client.25070 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T11:35:10.555 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-04-25T11:35:10.225098+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1192393,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1353,"num_read_kb":816,"num_write":495,"num_write_kb":3616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":3012886,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":893,"ondisk_log_size":893,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":392232,"kb_used_data":13932,"kb_used_omap":18,"kb_used_meta":378157,"kb_avail":1124549592,"statfs":{"total":1151940427776,"available":1151538782208,"internally_reserved":0,"allocated":14266368,"data_stored":8475001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":387233081},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.983962"},"pg_stats":[{"pgid":"3.3f","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.505112+0000","last_change":"2024-04-25T11:33:10.892144+0000","last_active":"2024-04-25T11:35:08.505112+0000","last_peered":"2024-04-25T11:35:08.505112+0000","last_clean":"2024-04-25T11:35:08.505112+0000","last_became_active":"2024-04-25T11:33:10.891721+0000","last_became_peered":"2024-04-25T11:33:10.891721+0000","last_unstale":"2024-04-25T11:35:08.505112+0000","last_undegraded":"2024-04-25T11:35:08.505112+0000","last_fullsized":"2024-04-25T11:35:08.505112+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:58:25.447086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"79'4","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.502017+0000","last_change":"2024-04-25T11:33:10.891647+0000","last_active":"2024-04-25T11:35:08.502017+0000","last_peered":"2024-04-25T11:35:08.502017+0000","last_clean":"2024-04-25T11:35:08.502017+0000","last_became_active":"2024-04-25T11:33:10.891283+0000","last_became_peered":"2024-04-25T11:33:10.891283+0000","last_unstale":"2024-04-25T11:35:08.502017+0000","last_undegraded":"2024-04-25T11:35:08.502017+0000","last_fullsized":"2024-04-25T11:35:08.502017+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:12.669438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501829+0000","last_change":"2024-04-25T11:33:10.890928+0000","last_active":"2024-04-25T11:35:08.501829+0000","last_peered":"2024-04-25T11:35:08.501829+0000","last_clean":"2024-04-25T11:35:08.501829+0000","last_became_active":"2024-04-25T11:33:10.889093+0000","last_became_peered":"2024-04-25T11:33:10.889093+0000","last_unstale":"2024-04-25T11:35:08.501829+0000","last_undegraded":"2024-04-25T11:35:08.501829+0000","last_fullsized":"2024-04-25T11:35:08.501829+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:28:09.038650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501058+0000","last_change":"2024-04-25T11:33:10.890184+0000","last_active":"2024-04-25T11:35:08.501058+0000","last_peered":"2024-04-25T11:35:08.501058+0000","last_clean":"2024-04-25T11:35:08.501058+0000","last_became_active":"2024-04-25T11:33:10.889367+0000","last_became_peered":"2024-04-25T11:33:10.889367+0000","last_unstale":"2024-04-25T11:35:08.501058+0000","last_undegraded":"2024-04-25T11:35:08.501058+0000","last_fullsized":"2024-04-25T11:35:08.501058+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:42.798295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"79'4","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.208757+0000","last_change":"2024-04-25T11:33:10.889758+0000","last_active":"2024-04-25T11:35:08.208757+0000","last_peered":"2024-04-25T11:35:08.208757+0000","last_clean":"2024-04-25T11:35:08.208757+0000","last_became_active":"2024-04-25T11:33:10.889384+0000","last_became_peered":"2024-04-25T11:33:10.889384+0000","last_unstale":"2024-04-25T11:35:08.208757+0000","last_undegraded":"2024-04-25T11:35:08.208757+0000","last_fullsized":"2024-04-25T11:35:08.208757+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:54:38.794216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"79'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.210810+0000","last_change":"2024-04-25T11:33:10.889348+0000","last_active":"2024-04-25T11:35:08.210810+0000","last_peered":"2024-04-25T11:35:08.210810+0000","last_clean":"2024-04-25T11:35:08.210810+0000","last_became_active":"2024-04-25T11:33:10.888908+0000","last_became_peered":"2024-04-25T11:33:10.888908+0000","last_unstale":"2024-04-25T11:35:08.210810+0000","last_undegraded":"2024-04-25T11:35:08.210810+0000","last_fullsized":"2024-04-25T11:35:08.210810+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:02:59.555921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"79'5","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500248+0000","last_change":"2024-04-25T11:34:18.237661+0000","last_active":"2024-04-25T11:35:08.500248+0000","last_peered":"2024-04-25T11:35:08.500248+0000","last_clean":"2024-04-25T11:35:08.500248+0000","last_became_active":"2024-04-25T11:34:18.237306+0000","last_became_peered":"2024-04-25T11:34:18.237306+0000","last_unstale":"2024-04-25T11:35:08.500248+0000","last_undegraded":"2024-04-25T11:35:08.500248+0000","last_fullsized":"2024-04-25T11:35:08.500248+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:17:34.228368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.317192+0000","last_change":"2024-04-25T11:33:10.887262+0000","last_active":"2024-04-25T11:35:08.317192+0000","last_peered":"2024-04-25T11:35:08.317192+0000","last_clean":"2024-04-25T11:35:08.317192+0000","last_became_active":"2024-04-25T11:33:10.886948+0000","last_became_peered":"2024-04-25T11:33:10.886948+0000","last_unstale":"2024-04-25T11:35:08.317192+0000","last_undegraded":"2024-04-25T11:35:08.317192+0000","last_fullsized":"2024-04-25T11:35:08.317192+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:09.721506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"79'11","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.314351+0000","last_change":"2024-04-25T11:34:17.883253+0000","last_active":"2024-04-25T11:35:08.314351+0000","last_peered":"2024-04-25T11:35:08.314351+0000","last_clean":"2024-04-25T11:35:08.314351+0000","last_became_active":"2024-04-25T11:34:17.882515+0000","last_became_peered":"2024-04-25T11:34:17.882515+0000","last_unstale":"2024-04-25T11:35:08.314351+0000","last_undegraded":"2024-04-25T11:35:08.314351+0000","last_fullsized":"2024-04-25T11:35:08.314351+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:52:32.813464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.36","version":"79'16","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.211225+0000","last_change":"2024-04-25T11:33:10.889812+0000","last_active":"2024-04-25T11:35:08.211225+0000","last_peered":"2024-04-25T11:35:08.211225+0000","last_clean":"2024-04-25T11:35:08.211225+0000","last_became_active":"2024-04-25T11:33:10.889369+0000","last_became_peered":"2024-04-25T11:33:10.889369+0000","last_unstale":"2024-04-25T11:35:08.211225+0000","last_undegraded":"2024-04-25T11:35:08.211225+0000","last_fullsized":"2024-04-25T11:35:08.211225+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:42:14.443496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"79'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501487+0000","last_change":"2024-04-25T11:33:10.891186+0000","last_active":"2024-04-25T11:35:08.501487+0000","last_peered":"2024-04-25T11:35:08.501487+0000","last_clean":"2024-04-25T11:35:08.501487+0000","last_became_active":"2024-04-25T11:33:10.890841+0000","last_became_peered":"2024-04-25T11:33:10.890841+0000","last_unstale":"2024-04-25T11:35:08.501487+0000","last_undegraded":"2024-04-25T11:35:08.501487+0000","last_fullsized":"2024-04-25T11:35:08.501487+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:24:56.078592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501001+0000","last_change":"2024-04-25T11:33:10.888801+0000","last_active":"2024-04-25T11:35:08.501001+0000","last_peered":"2024-04-25T11:35:08.501001+0000","last_clean":"2024-04-25T11:35:08.501001+0000","last_became_active":"2024-04-25T11:33:10.888437+0000","last_became_peered":"2024-04-25T11:33:10.888437+0000","last_unstale":"2024-04-25T11:35:08.501001+0000","last_undegraded":"2024-04-25T11:35:08.501001+0000","last_fullsized":"2024-04-25T11:35:08.501001+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:14:14.011475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"79'10","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207334+0000","last_change":"2024-04-25T11:33:10.892749+0000","last_active":"2024-04-25T11:35:08.207334+0000","last_peered":"2024-04-25T11:35:08.207334+0000","last_clean":"2024-04-25T11:35:08.207334+0000","last_became_active":"2024-04-25T11:33:10.892370+0000","last_became_peered":"2024-04-25T11:33:10.892370+0000","last_unstale":"2024-04-25T11:35:08.207334+0000","last_undegraded":"2024-04-25T11:35:08.207334+0000","last_fullsized":"2024-04-25T11:35:08.207334+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:00:51.312387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500624+0000","last_change":"2024-04-25T11:33:10.888981+0000","last_active":"2024-04-25T11:35:08.500624+0000","last_peered":"2024-04-25T11:35:08.500624+0000","last_clean":"2024-04-25T11:35:08.500624+0000","last_became_active":"2024-04-25T11:33:10.888537+0000","last_became_peered":"2024-04-25T11:33:10.888537+0000","last_unstale":"2024-04-25T11:35:08.500624+0000","last_undegraded":"2024-04-25T11:35:08.500624+0000","last_fullsized":"2024-04-25T11:35:08.500624+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:12:19.018698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501399+0000","last_change":"2024-04-25T11:33:10.891455+0000","last_active":"2024-04-25T11:35:08.501399+0000","last_peered":"2024-04-25T11:35:08.501399+0000","last_clean":"2024-04-25T11:35:08.501399+0000","last_became_active":"2024-04-25T11:33:10.891037+0000","last_became_peered":"2024-04-25T11:33:10.891037+0000","last_unstale":"2024-04-25T11:35:08.501399+0000","last_undegraded":"2024-04-25T11:35:08.501399+0000","last_fullsized":"2024-04-25T11:35:08.501399+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:44.470674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.380415+0000","last_change":"2024-04-25T11:33:10.895309+0000","last_active":"2024-04-25T11:35:08.380415+0000","last_peered":"2024-04-25T11:35:08.380415+0000","last_clean":"2024-04-25T11:35:08.380415+0000","last_became_active":"2024-04-25T11:33:10.894952+0000","last_became_peered":"2024-04-25T11:33:10.894952+0000","last_unstale":"2024-04-25T11:35:08.380415+0000","last_undegraded":"2024-04-25T11:35:08.380415+0000","last_fullsized":"2024-04-25T11:35:08.380415+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:29:22.119017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.505207+0000","last_change":"2024-04-25T11:33:10.894524+0000","last_active":"2024-04-25T11:35:08.505207+0000","last_peered":"2024-04-25T11:35:08.505207+0000","last_clean":"2024-04-25T11:35:08.505207+0000","last_became_active":"2024-04-25T11:33:10.894220+0000","last_became_peered":"2024-04-25T11:33:10.894220+0000","last_unstale":"2024-04-25T11:35:08.505207+0000","last_undegraded":"2024-04-25T11:35:08.505207+0000","last_fullsized":"2024-04-25T11:35:08.505207+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:10:29.072976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"79'10","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.315017+0000","last_change":"2024-04-25T11:33:10.886814+0000","last_active":"2024-04-25T11:35:08.315017+0000","last_peered":"2024-04-25T11:35:08.315017+0000","last_clean":"2024-04-25T11:35:08.315017+0000","last_became_active":"2024-04-25T11:33:10.886494+0000","last_became_peered":"2024-04-25T11:33:10.886494+0000","last_unstale":"2024-04-25T11:35:08.315017+0000","last_undegraded":"2024-04-25T11:35:08.315017+0000","last_fullsized":"2024-04-25T11:35:08.315017+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:03:43.054485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500081+0000","last_change":"2024-04-25T11:33:10.886870+0000","last_active":"2024-04-25T11:35:08.500081+0000","last_peered":"2024-04-25T11:35:08.500081+0000","last_clean":"2024-04-25T11:35:08.500081+0000","last_became_active":"2024-04-25T11:33:10.886507+0000","last_became_peered":"2024-04-25T11:33:10.886507+0000","last_unstale":"2024-04-25T11:35:08.500081+0000","last_undegraded":"2024-04-25T11:35:08.500081+0000","last_fullsized":"2024-04-25T11:35:08.500081+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:50.101760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"77'13","reported_seq":148,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.314751+0000","last_change":"2024-04-25T11:33:10.028970+0000","last_active":"2024-04-25T11:35:08.314751+0000","last_peered":"2024-04-25T11:35:08.314751+0000","last_clean":"2024-04-25T11:35:08.314751+0000","last_became_active":"2024-04-25T11:33:10.028636+0000","last_became_peered":"2024-04-25T11:33:10.028636+0000","last_unstale":"2024-04-25T11:35:08.314751+0000","last_undegraded":"2024-04-25T11:35:08.314751+0000","last_fullsized":"2024-04-25T11:35:08.314751+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:02.414903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"79'5","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.314684+0000","last_change":"2024-04-25T11:34:17.881522+0000","last_active":"2024-04-25T11:35:08.314684+0000","last_peered":"2024-04-25T11:35:08.314684+0000","last_clean":"2024-04-25T11:35:08.314684+0000","last_became_active":"2024-04-25T11:34:17.881114+0000","last_became_peered":"2024-04-25T11:34:17.881114+0000","last_unstale":"2024-04-25T11:35:08.314684+0000","last_undegraded":"2024-04-25T11:35:08.314684+0000","last_fullsized":"2024-04-25T11:35:08.314684+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:59:03.675054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"77'2","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.317914+0000","last_change":"2024-04-25T11:33:10.036958+0000","last_active":"2024-04-25T11:35:08.317914+0000","last_peered":"2024-04-25T11:35:08.317914+0000","last_clean":"2024-04-25T11:35:08.317914+0000","last_became_active":"2024-04-25T11:33:10.036606+0000","last_became_peered":"2024-04-25T11:33:10.036606+0000","last_unstale":"2024-04-25T11:35:08.317914+0000","last_undegraded":"2024-04-25T11:35:08.317914+0000","last_fullsized":"2024-04-25T11:35:08.317914+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:12.660208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"79'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.502299+0000","last_change":"2024-04-25T11:33:10.887501+0000","last_active":"2024-04-25T11:35:08.502299+0000","last_peered":"2024-04-25T11:35:08.502299+0000","last_clean":"2024-04-25T11:35:08.502299+0000","last_became_active":"2024-04-25T11:33:10.885985+0000","last_became_peered":"2024-04-25T11:33:10.885985+0000","last_unstale":"2024-04-25T11:35:08.502299+0000","last_undegraded":"2024-04-25T11:35:08.502299+0000","last_fullsized":"2024-04-25T11:35:08.502299+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:31:32.035646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"77'12","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.209236+0000","last_change":"2024-04-25T11:33:10.031161+0000","last_active":"2024-04-25T11:35:08.209236+0000","last_peered":"2024-04-25T11:35:08.209236+0000","last_clean":"2024-04-25T11:35:08.209236+0000","last_became_active":"2024-04-25T11:33:10.028891+0000","last_became_peered":"2024-04-25T11:33:10.028891+0000","last_unstale":"2024-04-25T11:35:08.209236+0000","last_undegraded":"2024-04-25T11:35:08.209236+0000","last_fullsized":"2024-04-25T11:35:08.209236+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:25.959699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.380513+0000","last_change":"2024-04-25T11:33:10.889439+0000","last_active":"2024-04-25T11:35:08.380513+0000","last_peered":"2024-04-25T11:35:08.380513+0000","last_clean":"2024-04-25T11:35:08.380513+0000","last_became_active":"2024-04-25T11:33:10.889082+0000","last_became_peered":"2024-04-25T11:33:10.889082+0000","last_unstale":"2024-04-25T11:35:08.380513+0000","last_undegraded":"2024-04-25T11:35:08.380513+0000","last_fullsized":"2024-04-25T11:35:08.380513+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:13:48.958179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"77'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.208951+0000","last_change":"2024-04-25T11:33:10.030793+0000","last_active":"2024-04-25T11:35:08.208951+0000","last_peered":"2024-04-25T11:35:08.208951+0000","last_clean":"2024-04-25T11:35:08.208951+0000","last_became_active":"2024-04-25T11:33:10.028805+0000","last_became_peered":"2024-04-25T11:33:10.028805+0000","last_unstale":"2024-04-25T11:35:08.208951+0000","last_undegraded":"2024-04-25T11:35:08.208951+0000","last_fullsized":"2024-04-25T11:35:08.208951+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:36.233194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"79'15","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.324396+0000","last_change":"2024-04-25T11:33:10.894815+0000","last_active":"2024-04-25T11:35:08.324396+0000","last_peered":"2024-04-25T11:35:08.324396+0000","last_clean":"2024-04-25T11:35:08.324396+0000","last_became_active":"2024-04-25T11:33:10.892174+0000","last_became_peered":"2024-04-25T11:33:10.892174+0000","last_unstale":"2024-04-25T11:35:08.324396+0000","last_undegraded":"2024-04-25T11:35:08.324396+0000","last_fullsized":"2024-04-25T11:35:08.324396+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:05.368441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"80'8","reported_seq":149,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.802370+0000","last_change":"2024-04-25T11:33:10.031238+0000","last_active":"2024-04-25T11:35:08.802370+0000","last_peered":"2024-04-25T11:35:08.802370+0000","last_clean":"2024-04-25T11:35:08.802370+0000","last_became_active":"2024-04-25T11:33:10.030917+0000","last_became_peered":"2024-04-25T11:33:10.030917+0000","last_unstale":"2024-04-25T11:35:08.802370+0000","last_undegraded":"2024-04-25T11:35:08.802370+0000","last_fullsized":"2024-04-25T11:35:08.802370+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:00:35.634293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1712,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":120,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.208813+0000","last_change":"2024-04-25T11:33:10.896974+0000","last_active":"2024-04-25T11:35:08.208813+0000","last_peered":"2024-04-25T11:35:08.208813+0000","last_clean":"2024-04-25T11:35:08.208813+0000","last_became_active":"2024-04-25T11:33:10.896575+0000","last_became_peered":"2024-04-25T11:33:10.896575+0000","last_unstale":"2024-04-25T11:35:08.208813+0000","last_undegraded":"2024-04-25T11:35:08.208813+0000","last_fullsized":"2024-04-25T11:35:08.208813+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:32:43.533652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"77'7","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.210783+0000","last_change":"2024-04-25T11:33:10.022714+0000","last_active":"2024-04-25T11:35:08.210783+0000","last_peered":"2024-04-25T11:35:08.210783+0000","last_clean":"2024-04-25T11:35:08.210783+0000","last_became_active":"2024-04-25T11:33:10.022230+0000","last_became_peered":"2024-04-25T11:33:10.022230+0000","last_unstale":"2024-04-25T11:35:08.210783+0000","last_undegraded":"2024-04-25T11:35:08.210783+0000","last_fullsized":"2024-04-25T11:35:08.210783+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:30:00.620833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.324208+0000","last_change":"2024-04-25T11:33:10.897664+0000","last_active":"2024-04-25T11:35:08.324208+0000","last_peered":"2024-04-25T11:35:08.324208+0000","last_clean":"2024-04-25T11:35:08.324208+0000","last_became_active":"2024-04-25T11:33:10.897189+0000","last_became_peered":"2024-04-25T11:33:10.897189+0000","last_unstale":"2024-04-25T11:35:08.324208+0000","last_undegraded":"2024-04-25T11:35:08.324208+0000","last_fullsized":"2024-04-25T11:35:08.324208+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:12:06.168793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"77'3","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207308+0000","last_change":"2024-04-25T11:33:10.031417+0000","last_active":"2024-04-25T11:35:08.207308+0000","last_peered":"2024-04-25T11:35:08.207308+0000","last_clean":"2024-04-25T11:35:08.207308+0000","last_became_active":"2024-04-25T11:33:10.030526+0000","last_became_peered":"2024-04-25T11:33:10.030526+0000","last_unstale":"2024-04-25T11:35:08.207308+0000","last_undegraded":"2024-04-25T11:35:08.207308+0000","last_fullsized":"2024-04-25T11:35:08.207308+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:05.718720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.317017+0000","last_change":"2024-04-25T11:33:10.886596+0000","last_active":"2024-04-25T11:35:08.317017+0000","last_peered":"2024-04-25T11:35:08.317017+0000","last_clean":"2024-04-25T11:35:08.317017+0000","last_became_active":"2024-04-25T11:33:10.886234+0000","last_became_peered":"2024-04-25T11:33:10.886234+0000","last_unstale":"2024-04-25T11:35:08.317017+0000","last_undegraded":"2024-04-25T11:35:08.317017+0000","last_fullsized":"2024-04-25T11:35:08.317017+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:46.632814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"77'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.324040+0000","last_change":"2024-04-25T11:33:10.032355+0000","last_active":"2024-04-25T11:35:08.324040+0000","last_peered":"2024-04-25T11:35:08.324040+0000","last_clean":"2024-04-25T11:35:08.324040+0000","last_became_active":"2024-04-25T11:33:10.031965+0000","last_became_peered":"2024-04-25T11:33:10.031965+0000","last_unstale":"2024-04-25T11:35:08.324040+0000","last_undegraded":"2024-04-25T11:35:08.324040+0000","last_fullsized":"2024-04-25T11:35:08.324040+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:08:10.539604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"79'11","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.314255+0000","last_change":"2024-04-25T11:33:10.885708+0000","last_active":"2024-04-25T11:35:08.314255+0000","last_peered":"2024-04-25T11:35:08.314255+0000","last_clean":"2024-04-25T11:35:08.314255+0000","last_became_active":"2024-04-25T11:33:10.885317+0000","last_became_peered":"2024-04-25T11:33:10.885317+0000","last_unstale":"2024-04-25T11:35:08.314255+0000","last_undegraded":"2024-04-25T11:35:08.314255+0000","last_fullsized":"2024-04-25T11:35:08.314255+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:50:08.518922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"80'6","reported_seq":256,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.801798+0000","last_change":"2024-04-25T11:33:10.030995+0000","last_active":"2024-04-25T11:35:08.801798+0000","last_peered":"2024-04-25T11:35:08.801798+0000","last_clean":"2024-04-25T11:35:08.801798+0000","last_became_active":"2024-04-25T11:33:10.030670+0000","last_became_peered":"2024-04-25T11:33:10.030670+0000","last_unstale":"2024-04-25T11:35:08.801798+0000","last_undegraded":"2024-04-25T11:35:08.801798+0000","last_fullsized":"2024-04-25T11:35:08.801798+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:39.569787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":229,"num_read_kb":229,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"79'10","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.380745+0000","last_change":"2024-04-25T11:33:10.892927+0000","last_active":"2024-04-25T11:35:08.380745+0000","last_peered":"2024-04-25T11:35:08.380745+0000","last_clean":"2024-04-25T11:35:08.380745+0000","last_became_active":"2024-04-25T11:33:10.892458+0000","last_became_peered":"2024-04-25T11:33:10.892458+0000","last_unstale":"2024-04-25T11:35:08.380745+0000","last_undegraded":"2024-04-25T11:35:08.380745+0000","last_fullsized":"2024-04-25T11:35:08.380745+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:53.121185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"77'4","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501417+0000","last_change":"2024-04-25T11:33:10.029728+0000","last_active":"2024-04-25T11:35:08.501417+0000","last_peered":"2024-04-25T11:35:08.501417+0000","last_clean":"2024-04-25T11:35:08.501417+0000","last_became_active":"2024-04-25T11:33:10.029435+0000","last_became_peered":"2024-04-25T11:33:10.029435+0000","last_unstale":"2024-04-25T11:35:08.501417+0000","last_undegraded":"2024-04-25T11:35:08.501417+0000","last_fullsized":"2024-04-25T11:35:08.501417+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:31:21.585870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.502258+0000","last_change":"2024-04-25T11:33:10.895828+0000","last_active":"2024-04-25T11:35:08.502258+0000","last_peered":"2024-04-25T11:35:08.502258+0000","last_clean":"2024-04-25T11:35:08.502258+0000","last_became_active":"2024-04-25T11:33:10.895280+0000","last_became_peered":"2024-04-25T11:33:10.895280+0000","last_unstale":"2024-04-25T11:35:08.502258+0000","last_undegraded":"2024-04-25T11:35:08.502258+0000","last_fullsized":"2024-04-25T11:35:08.502258+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:03.009912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"77'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.324544+0000","last_change":"2024-04-25T11:33:10.036651+0000","last_active":"2024-04-25T11:35:08.324544+0000","last_peered":"2024-04-25T11:35:08.324544+0000","last_clean":"2024-04-25T11:35:08.324544+0000","last_became_active":"2024-04-25T11:33:10.036294+0000","last_became_peered":"2024-04-25T11:33:10.036294+0000","last_unstale":"2024-04-25T11:35:08.324544+0000","last_undegraded":"2024-04-25T11:35:08.324544+0000","last_fullsized":"2024-04-25T11:35:08.324544+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:12:36.566618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"79'11","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.380605+0000","last_change":"2024-04-25T11:33:10.890199+0000","last_active":"2024-04-25T11:35:08.380605+0000","last_peered":"2024-04-25T11:35:08.380605+0000","last_clean":"2024-04-25T11:35:08.380605+0000","last_became_active":"2024-04-25T11:33:10.889855+0000","last_became_peered":"2024-04-25T11:33:10.889855+0000","last_unstale":"2024-04-25T11:35:08.380605+0000","last_undegraded":"2024-04-25T11:35:08.380605+0000","last_fullsized":"2024-04-25T11:35:08.380605+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:54:45.476647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'37","reported_seq":319,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.511117+0000","last_change":"2024-04-25T11:34:18.309247+0000","last_active":"2024-04-25T11:35:08.511117+0000","last_peered":"2024-04-25T11:35:08.511117+0000","last_clean":"2024-04-25T11:35:08.511117+0000","last_became_active":"2024-04-25T11:34:18.299888+0000","last_became_peered":"2024-04-25T11:34:18.299888+0000","last_unstale":"2024-04-25T11:35:08.511117+0000","last_undegraded":"2024-04-25T11:35:08.511117+0000","last_fullsized":"2024-04-25T11:35:08.511117+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:36:06.603720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67758,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":235,"num_read_kb":178,"num_write":39,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"79'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.208993+0000","last_change":"2024-04-25T11:33:10.888596+0000","last_active":"2024-04-25T11:35:08.208993+0000","last_peered":"2024-04-25T11:35:08.208993+0000","last_clean":"2024-04-25T11:35:08.208993+0000","last_became_active":"2024-04-25T11:33:10.888226+0000","last_became_peered":"2024-04-25T11:33:10.888226+0000","last_unstale":"2024-04-25T11:35:08.208993+0000","last_undegraded":"2024-04-25T11:35:08.208993+0000","last_fullsized":"2024-04-25T11:35:08.208993+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:30:42.891788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"77'9","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500117+0000","last_change":"2024-04-25T11:33:10.026969+0000","last_active":"2024-04-25T11:35:08.500117+0000","last_peered":"2024-04-25T11:35:08.500117+0000","last_clean":"2024-04-25T11:35:08.500117+0000","last_became_active":"2024-04-25T11:33:10.026591+0000","last_became_peered":"2024-04-25T11:33:10.026591+0000","last_unstale":"2024-04-25T11:35:08.500117+0000","last_undegraded":"2024-04-25T11:35:08.500117+0000","last_fullsized":"2024-04-25T11:35:08.500117+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:50:34.269478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207191+0000","last_change":"2024-04-25T11:33:10.900303+0000","last_active":"2024-04-25T11:35:08.207191+0000","last_peered":"2024-04-25T11:35:08.207191+0000","last_clean":"2024-04-25T11:35:08.207191+0000","last_became_active":"2024-04-25T11:33:10.899946+0000","last_became_peered":"2024-04-25T11:33:10.899946+0000","last_unstale":"2024-04-25T11:35:08.207191+0000","last_undegraded":"2024-04-25T11:35:08.207191+0000","last_fullsized":"2024-04-25T11:35:08.207191+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:33:40.805400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"77'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.324114+0000","last_change":"2024-04-25T11:33:10.035668+0000","last_active":"2024-04-25T11:35:08.324114+0000","last_peered":"2024-04-25T11:35:08.324114+0000","last_clean":"2024-04-25T11:35:08.324114+0000","last_became_active":"2024-04-25T11:33:10.035241+0000","last_became_peered":"2024-04-25T11:33:10.035241+0000","last_unstale":"2024-04-25T11:35:08.324114+0000","last_undegraded":"2024-04-25T11:35:08.324114+0000","last_fullsized":"2024-04-25T11:35:08.324114+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:58:48.735577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"79'15","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500819+0000","last_change":"2024-04-25T11:33:10.894864+0000","last_active":"2024-04-25T11:35:08.500819+0000","last_peered":"2024-04-25T11:35:08.500819+0000","last_clean":"2024-04-25T11:35:08.500819+0000","last_became_active":"2024-04-25T11:33:10.894491+0000","last_became_peered":"2024-04-25T11:33:10.894491+0000","last_unstale":"2024-04-25T11:35:08.500819+0000","last_undegraded":"2024-04-25T11:35:08.500819+0000","last_fullsized":"2024-04-25T11:35:08.500819+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:21:28.256878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"80'8","reported_seq":146,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.816304+0000","last_change":"2024-04-25T11:33:10.029099+0000","last_active":"2024-04-25T11:35:08.816304+0000","last_peered":"2024-04-25T11:35:08.816304+0000","last_clean":"2024-04-25T11:35:08.816304+0000","last_became_active":"2024-04-25T11:33:10.028722+0000","last_became_peered":"2024-04-25T11:33:10.028722+0000","last_unstale":"2024-04-25T11:35:08.816304+0000","last_undegraded":"2024-04-25T11:35:08.816304+0000","last_fullsized":"2024-04-25T11:35:08.816304+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:44:12.261519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1511,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"79'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500425+0000","last_change":"2024-04-25T11:33:10.888635+0000","last_active":"2024-04-25T11:35:08.500425+0000","last_peered":"2024-04-25T11:35:08.500425+0000","last_clean":"2024-04-25T11:35:08.500425+0000","last_became_active":"2024-04-25T11:33:10.887544+0000","last_became_peered":"2024-04-25T11:33:10.887544+0000","last_unstale":"2024-04-25T11:35:08.500425+0000","last_undegraded":"2024-04-25T11:35:08.500425+0000","last_fullsized":"2024-04-25T11:35:08.500425+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:59:25.459587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"77'17","reported_seq":39,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.208662+0000","last_change":"2024-04-25T11:33:10.023566+0000","last_active":"2024-04-25T11:35:08.208662+0000","last_peered":"2024-04-25T11:35:08.208662+0000","last_clean":"2024-04-25T11:35:08.208662+0000","last_became_active":"2024-04-25T11:33:10.023234+0000","last_became_peered":"2024-04-25T11:33:10.023234+0000","last_unstale":"2024-04-25T11:35:08.208662+0000","last_undegraded":"2024-04-25T11:35:08.208662+0000","last_fullsized":"2024-04-25T11:35:08.208662+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:56:26.342148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"79'6","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.324020+0000","last_change":"2024-04-25T11:34:18.237663+0000","last_active":"2024-04-25T11:35:08.324020+0000","last_peered":"2024-04-25T11:35:08.324020+0000","last_clean":"2024-04-25T11:35:08.324020+0000","last_became_active":"2024-04-25T11:34:18.236993+0000","last_became_peered":"2024-04-25T11:34:18.236993+0000","last_unstale":"2024-04-25T11:35:08.324020+0000","last_undegraded":"2024-04-25T11:35:08.324020+0000","last_fullsized":"2024-04-25T11:35:08.324020+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:59:47.371685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"77'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.209203+0000","last_change":"2024-04-25T11:33:10.028285+0000","last_active":"2024-04-25T11:35:08.209203+0000","last_peered":"2024-04-25T11:35:08.209203+0000","last_clean":"2024-04-25T11:35:08.209203+0000","last_became_active":"2024-04-25T11:33:10.027959+0000","last_became_peered":"2024-04-25T11:33:10.027959+0000","last_unstale":"2024-04-25T11:35:08.209203+0000","last_undegraded":"2024-04-25T11:35:08.209203+0000","last_fullsized":"2024-04-25T11:35:08.209203+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:44:41.094048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"79'5","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.505018+0000","last_change":"2024-04-25T11:34:17.887356+0000","last_active":"2024-04-25T11:35:08.505018+0000","last_peered":"2024-04-25T11:35:08.505018+0000","last_clean":"2024-04-25T11:35:08.505018+0000","last_became_active":"2024-04-25T11:34:17.886981+0000","last_became_peered":"2024-04-25T11:34:17.886981+0000","last_unstale":"2024-04-25T11:35:08.505018+0000","last_undegraded":"2024-04-25T11:35:08.505018+0000","last_fullsized":"2024-04-25T11:35:08.505018+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:40.573546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"77'1","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.314543+0000","last_change":"2024-04-25T11:33:10.027473+0000","last_active":"2024-04-25T11:35:08.314543+0000","last_peered":"2024-04-25T11:35:08.314543+0000","last_clean":"2024-04-25T11:35:08.314543+0000","last_became_active":"2024-04-25T11:33:10.027133+0000","last_became_peered":"2024-04-25T11:33:10.027133+0000","last_unstale":"2024-04-25T11:35:08.314543+0000","last_undegraded":"2024-04-25T11:35:08.314543+0000","last_fullsized":"2024-04-25T11:35:08.314543+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:19:58.338507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"79'5","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.211036+0000","last_change":"2024-04-25T11:33:10.892588+0000","last_active":"2024-04-25T11:35:08.211036+0000","last_peered":"2024-04-25T11:35:08.211036+0000","last_clean":"2024-04-25T11:35:08.211036+0000","last_became_active":"2024-04-25T11:33:10.892088+0000","last_became_peered":"2024-04-25T11:33:10.892088+0000","last_unstale":"2024-04-25T11:35:08.211036+0000","last_undegraded":"2024-04-25T11:35:08.211036+0000","last_fullsized":"2024-04-25T11:35:08.211036+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:10.136222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"77'5","reported_seq":140,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500939+0000","last_change":"2024-04-25T11:33:10.030801+0000","last_active":"2024-04-25T11:35:08.500939+0000","last_peered":"2024-04-25T11:35:08.500939+0000","last_clean":"2024-04-25T11:35:08.500939+0000","last_became_active":"2024-04-25T11:33:10.030466+0000","last_became_peered":"2024-04-25T11:33:10.030466+0000","last_unstale":"2024-04-25T11:35:08.500939+0000","last_undegraded":"2024-04-25T11:35:08.500939+0000","last_fullsized":"2024-04-25T11:35:08.500939+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:22:02.922309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.211021+0000","last_change":"2024-04-25T11:33:10.894846+0000","last_active":"2024-04-25T11:35:08.211021+0000","last_peered":"2024-04-25T11:35:08.211021+0000","last_clean":"2024-04-25T11:35:08.211021+0000","last_became_active":"2024-04-25T11:33:10.894456+0000","last_became_peered":"2024-04-25T11:33:10.894456+0000","last_unstale":"2024-04-25T11:35:08.211021+0000","last_undegraded":"2024-04-25T11:35:08.211021+0000","last_fullsized":"2024-04-25T11:35:08.211021+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:11:16.265126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"79'8","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.324115+0000","last_change":"2024-04-25T11:34:17.882781+0000","last_active":"2024-04-25T11:35:08.324115+0000","last_peered":"2024-04-25T11:35:08.324115+0000","last_clean":"2024-04-25T11:35:08.324115+0000","last_became_active":"2024-04-25T11:34:17.882348+0000","last_became_peered":"2024-04-25T11:34:17.882348+0000","last_unstale":"2024-04-25T11:35:08.324115+0000","last_undegraded":"2024-04-25T11:35:08.324115+0000","last_fullsized":"2024-04-25T11:35:08.324115+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:47:34.385332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"80'7","reported_seq":142,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500831+0000","last_change":"2024-04-25T11:33:10.037834+0000","last_active":"2024-04-25T11:35:08.500831+0000","last_peered":"2024-04-25T11:35:08.500831+0000","last_clean":"2024-04-25T11:35:08.500831+0000","last_became_active":"2024-04-25T11:33:10.037570+0000","last_became_peered":"2024-04-25T11:33:10.037570+0000","last_unstale":"2024-04-25T11:35:08.500831+0000","last_undegraded":"2024-04-25T11:35:08.500831+0000","last_fullsized":"2024-04-25T11:35:08.500831+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:12.795190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.317600+0000","last_change":"2024-04-25T11:33:10.882663+0000","last_active":"2024-04-25T11:35:08.317600+0000","last_peered":"2024-04-25T11:35:08.317600+0000","last_clean":"2024-04-25T11:35:08.317600+0000","last_became_active":"2024-04-25T11:33:10.882300+0000","last_became_peered":"2024-04-25T11:33:10.882300+0000","last_unstale":"2024-04-25T11:35:08.317600+0000","last_undegraded":"2024-04-25T11:35:08.317600+0000","last_fullsized":"2024-04-25T11:35:08.317600+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:58:11.379654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"77'6","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500572+0000","last_change":"2024-04-25T11:33:10.024598+0000","last_active":"2024-04-25T11:35:08.500572+0000","last_peered":"2024-04-25T11:35:08.500572+0000","last_clean":"2024-04-25T11:35:08.500572+0000","last_became_active":"2024-04-25T11:33:10.024229+0000","last_became_peered":"2024-04-25T11:33:10.024229+0000","last_unstale":"2024-04-25T11:35:08.500572+0000","last_undegraded":"2024-04-25T11:35:08.500572+0000","last_fullsized":"2024-04-25T11:35:08.500572+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:42:14.690581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":98,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500206+0000","last_change":"2024-04-25T11:32:01.310202+0000","last_active":"2024-04-25T11:35:08.500206+0000","last_peered":"2024-04-25T11:35:08.500206+0000","last_clean":"2024-04-25T11:35:08.500206+0000","last_became_active":"2024-04-25T11:32:01.305242+0000","last_became_peered":"2024-04-25T11:32:01.305242+0000","last_unstale":"2024-04-25T11:35:08.500206+0000","last_undegraded":"2024-04-25T11:35:08.500206+0000","last_fullsized":"2024-04-25T11:35:08.500206+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:28:43.374096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:28:43.374096+0000","last_clean_scrub_stamp":"2024-04-25T11:28:43.374096+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:56:18.551747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"79'5","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207126+0000","last_change":"2024-04-25T11:33:10.889124+0000","last_active":"2024-04-25T11:35:08.207126+0000","last_peered":"2024-04-25T11:35:08.207126+0000","last_clean":"2024-04-25T11:35:08.207126+0000","last_became_active":"2024-04-25T11:33:10.888790+0000","last_became_peered":"2024-04-25T11:33:10.888790+0000","last_unstale":"2024-04-25T11:35:08.207126+0000","last_undegraded":"2024-04-25T11:35:08.207126+0000","last_fullsized":"2024-04-25T11:35:08.207126+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:33:44.999949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"77'4","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500562+0000","last_change":"2024-04-25T11:33:10.031547+0000","last_active":"2024-04-25T11:35:08.500562+0000","last_peered":"2024-04-25T11:35:08.500562+0000","last_clean":"2024-04-25T11:35:08.500562+0000","last_became_active":"2024-04-25T11:33:10.031152+0000","last_became_peered":"2024-04-25T11:33:10.031152+0000","last_unstale":"2024-04-25T11:35:08.500562+0000","last_undegraded":"2024-04-25T11:35:08.500562+0000","last_fullsized":"2024-04-25T11:35:08.500562+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:10.646771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"77'4","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.209106+0000","last_change":"2024-04-25T11:33:10.020367+0000","last_active":"2024-04-25T11:35:08.209106+0000","last_peered":"2024-04-25T11:35:08.209106+0000","last_clean":"2024-04-25T11:35:08.209106+0000","last_became_active":"2024-04-25T11:33:10.020007+0000","last_became_peered":"2024-04-25T11:33:10.020007+0000","last_unstale":"2024-04-25T11:35:08.209106+0000","last_undegraded":"2024-04-25T11:35:08.209106+0000","last_fullsized":"2024-04-25T11:35:08.209106+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:48.134907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207404+0000","last_change":"2024-04-25T11:33:10.887741+0000","last_active":"2024-04-25T11:35:08.207404+0000","last_peered":"2024-04-25T11:35:08.207404+0000","last_clean":"2024-04-25T11:35:08.207404+0000","last_became_active":"2024-04-25T11:33:10.886455+0000","last_became_peered":"2024-04-25T11:33:10.886455+0000","last_unstale":"2024-04-25T11:35:08.207404+0000","last_undegraded":"2024-04-25T11:35:08.207404+0000","last_fullsized":"2024-04-25T11:35:08.207404+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:37:16.842817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'48","reported_seq":92,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.448574+0000","last_change":"2024-04-25T11:33:10.025672+0000","last_active":"2024-04-25T11:35:08.448574+0000","last_peered":"2024-04-25T11:35:08.448574+0000","last_clean":"2024-04-25T11:35:08.448574+0000","last_became_active":"2024-04-25T11:33:10.025232+0000","last_became_peered":"2024-04-25T11:33:10.025232+0000","last_unstale":"2024-04-25T11:35:08.448574+0000","last_undegraded":"2024-04-25T11:35:08.448574+0000","last_fullsized":"2024-04-25T11:35:08.448574+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:06:26.579126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":84,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"79'5","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.317930+0000","last_change":"2024-04-25T11:33:10.884692+0000","last_active":"2024-04-25T11:35:08.317930+0000","last_peered":"2024-04-25T11:35:08.317930+0000","last_clean":"2024-04-25T11:35:08.317930+0000","last_became_active":"2024-04-25T11:33:10.884346+0000","last_became_peered":"2024-04-25T11:33:10.884346+0000","last_unstale":"2024-04-25T11:35:08.317930+0000","last_undegraded":"2024-04-25T11:35:08.317930+0000","last_fullsized":"2024-04-25T11:35:08.317930+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:31:00.630410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"77'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500890+0000","last_change":"2024-04-25T11:33:10.033872+0000","last_active":"2024-04-25T11:35:08.500890+0000","last_peered":"2024-04-25T11:35:08.500890+0000","last_clean":"2024-04-25T11:35:08.500890+0000","last_became_active":"2024-04-25T11:33:10.033435+0000","last_became_peered":"2024-04-25T11:33:10.033435+0000","last_unstale":"2024-04-25T11:35:08.500890+0000","last_undegraded":"2024-04-25T11:35:08.500890+0000","last_fullsized":"2024-04-25T11:35:08.500890+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:12.885215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"79'7","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501866+0000","last_change":"2024-04-25T11:34:17.881545+0000","last_active":"2024-04-25T11:35:08.501866+0000","last_peered":"2024-04-25T11:35:08.501866+0000","last_clean":"2024-04-25T11:35:08.501866+0000","last_became_active":"2024-04-25T11:34:17.881125+0000","last_became_peered":"2024-04-25T11:34:17.881125+0000","last_unstale":"2024-04-25T11:35:08.501866+0000","last_undegraded":"2024-04-25T11:35:08.501866+0000","last_fullsized":"2024-04-25T11:35:08.501866+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:55:18.404023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"77'2","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.380513+0000","last_change":"2024-04-25T11:33:10.032384+0000","last_active":"2024-04-25T11:35:08.380513+0000","last_peered":"2024-04-25T11:35:08.380513+0000","last_clean":"2024-04-25T11:35:08.380513+0000","last_became_active":"2024-04-25T11:33:10.032035+0000","last_became_peered":"2024-04-25T11:33:10.032035+0000","last_unstale":"2024-04-25T11:35:08.380513+0000","last_undegraded":"2024-04-25T11:35:08.380513+0000","last_fullsized":"2024-04-25T11:35:08.380513+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:21:41.723120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"79'2","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.208679+0000","last_change":"2024-04-25T11:33:10.893366+0000","last_active":"2024-04-25T11:35:08.208679+0000","last_peered":"2024-04-25T11:35:08.208679+0000","last_clean":"2024-04-25T11:35:08.208679+0000","last_became_active":"2024-04-25T11:33:10.892988+0000","last_became_peered":"2024-04-25T11:33:10.892988+0000","last_unstale":"2024-04-25T11:35:08.208679+0000","last_undegraded":"2024-04-25T11:35:08.208679+0000","last_fullsized":"2024-04-25T11:35:08.208679+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:06.368216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"77'7","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.317231+0000","last_change":"2024-04-25T11:33:10.031456+0000","last_active":"2024-04-25T11:35:08.317231+0000","last_peered":"2024-04-25T11:35:08.317231+0000","last_clean":"2024-04-25T11:35:08.317231+0000","last_became_active":"2024-04-25T11:33:10.031115+0000","last_became_peered":"2024-04-25T11:33:10.031115+0000","last_unstale":"2024-04-25T11:35:08.317231+0000","last_undegraded":"2024-04-25T11:35:08.317231+0000","last_fullsized":"2024-04-25T11:35:08.317231+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:12:56.803277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.504698+0000","last_change":"2024-04-25T11:33:10.893692+0000","last_active":"2024-04-25T11:35:08.504698+0000","last_peered":"2024-04-25T11:35:08.504698+0000","last_clean":"2024-04-25T11:35:08.504698+0000","last_became_active":"2024-04-25T11:33:10.893332+0000","last_became_peered":"2024-04-25T11:33:10.893332+0000","last_unstale":"2024-04-25T11:35:08.504698+0000","last_undegraded":"2024-04-25T11:35:08.504698+0000","last_fullsized":"2024-04-25T11:35:08.504698+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:01.277465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"80'5","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.641604+0000","last_change":"2024-04-25T11:33:10.023804+0000","last_active":"2024-04-25T11:35:08.641604+0000","last_peered":"2024-04-25T11:35:08.641604+0000","last_clean":"2024-04-25T11:35:08.641604+0000","last_became_active":"2024-04-25T11:33:10.023503+0000","last_became_peered":"2024-04-25T11:33:10.023503+0000","last_unstale":"2024-04-25T11:35:08.641604+0000","last_undegraded":"2024-04-25T11:35:08.641604+0000","last_fullsized":"2024-04-25T11:35:08.641604+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:34:48.074176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"79'8","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.314248+0000","last_change":"2024-04-25T11:34:17.882901+0000","last_active":"2024-04-25T11:35:08.314248+0000","last_peered":"2024-04-25T11:35:08.314248+0000","last_clean":"2024-04-25T11:35:08.314248+0000","last_became_active":"2024-04-25T11:34:17.882514+0000","last_became_peered":"2024-04-25T11:34:17.882514+0000","last_unstale":"2024-04-25T11:35:08.314248+0000","last_undegraded":"2024-04-25T11:35:08.314248+0000","last_fullsized":"2024-04-25T11:35:08.314248+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:03:30.843842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,10],"acting":[9,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.b","version":"77'4","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.208848+0000","last_change":"2024-04-25T11:33:10.030857+0000","last_active":"2024-04-25T11:35:08.208848+0000","last_peered":"2024-04-25T11:35:08.208848+0000","last_clean":"2024-04-25T11:35:08.208848+0000","last_became_active":"2024-04-25T11:33:10.028879+0000","last_became_peered":"2024-04-25T11:33:10.028879+0000","last_unstale":"2024-04-25T11:35:08.208848+0000","last_undegraded":"2024-04-25T11:35:08.208848+0000","last_fullsized":"2024-04-25T11:35:08.208848+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:28.820919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"79'5","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500748+0000","last_change":"2024-04-25T11:33:10.893773+0000","last_active":"2024-04-25T11:35:08.500748+0000","last_peered":"2024-04-25T11:35:08.500748+0000","last_clean":"2024-04-25T11:35:08.500748+0000","last_became_active":"2024-04-25T11:33:10.893431+0000","last_became_peered":"2024-04-25T11:33:10.893431+0000","last_unstale":"2024-04-25T11:35:08.500748+0000","last_undegraded":"2024-04-25T11:35:08.500748+0000","last_fullsized":"2024-04-25T11:35:08.500748+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:15:15.998951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"80'7","reported_seq":144,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.807310+0000","last_change":"2024-04-25T11:33:10.032745+0000","last_active":"2024-04-25T11:35:08.807310+0000","last_peered":"2024-04-25T11:35:08.807310+0000","last_clean":"2024-04-25T11:35:08.807310+0000","last_became_active":"2024-04-25T11:33:10.032361+0000","last_became_peered":"2024-04-25T11:33:10.032361+0000","last_unstale":"2024-04-25T11:35:08.807310+0000","last_undegraded":"2024-04-25T11:35:08.807310+0000","last_fullsized":"2024-04-25T11:35:08.807310+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:14:46.555892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1952,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":116,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501571+0000","last_change":"2024-04-25T11:33:10.888084+0000","last_active":"2024-04-25T11:35:08.501571+0000","last_peered":"2024-04-25T11:35:08.501571+0000","last_clean":"2024-04-25T11:35:08.501571+0000","last_became_active":"2024-04-25T11:33:10.887679+0000","last_became_peered":"2024-04-25T11:33:10.887679+0000","last_unstale":"2024-04-25T11:35:08.501571+0000","last_undegraded":"2024-04-25T11:35:08.501571+0000","last_fullsized":"2024-04-25T11:35:08.501571+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:10:20.073630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"80'41","reported_seq":70,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.445891+0000","last_change":"2024-04-25T11:33:10.035355+0000","last_active":"2024-04-25T11:35:08.445891+0000","last_peered":"2024-04-25T11:35:08.445891+0000","last_clean":"2024-04-25T11:35:08.445891+0000","last_became_active":"2024-04-25T11:33:10.034724+0000","last_became_peered":"2024-04-25T11:33:10.034724+0000","last_unstale":"2024-04-25T11:35:08.445891+0000","last_undegraded":"2024-04-25T11:35:08.445891+0000","last_fullsized":"2024-04-25T11:35:08.445891+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:24:26.757803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500628+0000","last_change":"2024-04-25T11:33:10.899827+0000","last_active":"2024-04-25T11:35:08.500628+0000","last_peered":"2024-04-25T11:35:08.500628+0000","last_clean":"2024-04-25T11:35:08.500628+0000","last_became_active":"2024-04-25T11:33:10.899356+0000","last_became_peered":"2024-04-25T11:33:10.899356+0000","last_unstale":"2024-04-25T11:35:08.500628+0000","last_undegraded":"2024-04-25T11:35:08.500628+0000","last_fullsized":"2024-04-25T11:35:08.500628+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:04:17.323958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"77'5","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207700+0000","last_change":"2024-04-25T11:33:10.034981+0000","last_active":"2024-04-25T11:35:08.207700+0000","last_peered":"2024-04-25T11:35:08.207700+0000","last_clean":"2024-04-25T11:35:08.207700+0000","last_became_active":"2024-04-25T11:33:10.034601+0000","last_became_peered":"2024-04-25T11:33:10.034601+0000","last_unstale":"2024-04-25T11:35:08.207700+0000","last_undegraded":"2024-04-25T11:35:08.207700+0000","last_fullsized":"2024-04-25T11:35:08.207700+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:09.004630+0000","last_clean_scrub_stamp":"2024-04-25T11:33:09.004630+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:19:14.710731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501693+0000","last_change":"2024-04-25T11:33:10.890020+0000","last_active":"2024-04-25T11:35:08.501693+0000","last_peered":"2024-04-25T11:35:08.501693+0000","last_clean":"2024-04-25T11:35:08.501693+0000","last_became_active":"2024-04-25T11:33:10.889558+0000","last_became_peered":"2024-04-25T11:33:10.889558+0000","last_unstale":"2024-04-25T11:35:08.501693+0000","last_undegraded":"2024-04-25T11:35:08.501693+0000","last_fullsized":"2024-04-25T11:35:08.501693+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:00:49.096818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"79'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.504776+0000","last_change":"2024-04-25T11:33:10.886965+0000","last_active":"2024-04-25T11:35:08.504776+0000","last_peered":"2024-04-25T11:35:08.504776+0000","last_clean":"2024-04-25T11:35:08.504776+0000","last_became_active":"2024-04-25T11:33:10.886594+0000","last_became_peered":"2024-04-25T11:33:10.886594+0000","last_unstale":"2024-04-25T11:35:08.504776+0000","last_undegraded":"2024-04-25T11:35:08.504776+0000","last_fullsized":"2024-04-25T11:35:08.504776+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:58.924857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"79'12","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.499974+0000","last_change":"2024-04-25T11:33:10.890760+0000","last_active":"2024-04-25T11:35:08.499974+0000","last_peered":"2024-04-25T11:35:08.499974+0000","last_clean":"2024-04-25T11:35:08.499974+0000","last_became_active":"2024-04-25T11:33:10.889643+0000","last_became_peered":"2024-04-25T11:33:10.889643+0000","last_unstale":"2024-04-25T11:35:08.499974+0000","last_undegraded":"2024-04-25T11:35:08.499974+0000","last_fullsized":"2024-04-25T11:35:08.499974+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:54.644853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.499871+0000","last_change":"2024-04-25T11:33:10.887850+0000","last_active":"2024-04-25T11:35:08.499871+0000","last_peered":"2024-04-25T11:35:08.499871+0000","last_clean":"2024-04-25T11:35:08.499871+0000","last_became_active":"2024-04-25T11:33:10.882323+0000","last_became_peered":"2024-04-25T11:33:10.882323+0000","last_unstale":"2024-04-25T11:35:08.499871+0000","last_undegraded":"2024-04-25T11:35:08.499871+0000","last_fullsized":"2024-04-25T11:35:08.499871+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:24:00.509704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"79'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.501134+0000","last_change":"2024-04-25T11:33:10.883842+0000","last_active":"2024-04-25T11:35:08.501134+0000","last_peered":"2024-04-25T11:35:08.501134+0000","last_clean":"2024-04-25T11:35:08.501134+0000","last_became_active":"2024-04-25T11:33:10.883513+0000","last_became_peered":"2024-04-25T11:33:10.883513+0000","last_unstale":"2024-04-25T11:35:08.501134+0000","last_undegraded":"2024-04-25T11:35:08.501134+0000","last_fullsized":"2024-04-25T11:35:08.501134+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:47:09.731875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"79'4","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207357+0000","last_change":"2024-04-25T11:33:10.890503+0000","last_active":"2024-04-25T11:35:08.207357+0000","last_peered":"2024-04-25T11:35:08.207357+0000","last_clean":"2024-04-25T11:35:08.207357+0000","last_became_active":"2024-04-25T11:33:10.890162+0000","last_became_peered":"2024-04-25T11:33:10.890162+0000","last_unstale":"2024-04-25T11:35:08.207357+0000","last_undegraded":"2024-04-25T11:35:08.207357+0000","last_fullsized":"2024-04-25T11:35:08.207357+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:36.462789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"79'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.504912+0000","last_change":"2024-04-25T11:33:10.897109+0000","last_active":"2024-04-25T11:35:08.504912+0000","last_peered":"2024-04-25T11:35:08.504912+0000","last_clean":"2024-04-25T11:35:08.504912+0000","last_became_active":"2024-04-25T11:33:10.896138+0000","last_became_peered":"2024-04-25T11:33:10.896138+0000","last_unstale":"2024-04-25T11:35:08.504912+0000","last_undegraded":"2024-04-25T11:35:08.504912+0000","last_fullsized":"2024-04-25T11:35:08.504912+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:47:46.202355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"79'15","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.500379+0000","last_change":"2024-04-25T11:33:10.890397+0000","last_active":"2024-04-25T11:35:08.500379+0000","last_peered":"2024-04-25T11:35:08.500379+0000","last_clean":"2024-04-25T11:35:08.500379+0000","last_became_active":"2024-04-25T11:33:10.889580+0000","last_became_peered":"2024-04-25T11:33:10.889580+0000","last_unstale":"2024-04-25T11:35:08.500379+0000","last_undegraded":"2024-04-25T11:35:08.500379+0000","last_fullsized":"2024-04-25T11:35:08.500379+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:16:09.672131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"79'2","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.317841+0000","last_change":"2024-04-25T11:33:10.873817+0000","last_active":"2024-04-25T11:35:08.317841+0000","last_peered":"2024-04-25T11:35:08.317841+0000","last_clean":"2024-04-25T11:35:08.317841+0000","last_became_active":"2024-04-25T11:33:10.873425+0000","last_became_peered":"2024-04-25T11:33:10.873425+0000","last_unstale":"2024-04-25T11:35:08.317841+0000","last_undegraded":"2024-04-25T11:35:08.317841+0000","last_fullsized":"2024-04-25T11:35:08.317841+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:52:26.279705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"79'6","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207126+0000","last_change":"2024-04-25T11:34:17.880075+0000","last_active":"2024-04-25T11:35:08.207126+0000","last_peered":"2024-04-25T11:35:08.207126+0000","last_clean":"2024-04-25T11:35:08.207126+0000","last_became_active":"2024-04-25T11:34:17.879692+0000","last_became_peered":"2024-04-25T11:34:17.879692+0000","last_unstale":"2024-04-25T11:35:08.207126+0000","last_undegraded":"2024-04-25T11:35:08.207126+0000","last_fullsized":"2024-04-25T11:35:08.207126+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:54.833601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,3],"acting":[6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.29","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.317356+0000","last_change":"2024-04-25T11:33:10.883690+0000","last_active":"2024-04-25T11:35:08.317356+0000","last_peered":"2024-04-25T11:35:08.317356+0000","last_clean":"2024-04-25T11:35:08.317356+0000","last_became_active":"2024-04-25T11:33:10.882965+0000","last_became_peered":"2024-04-25T11:33:10.882965+0000","last_unstale":"2024-04-25T11:35:08.317356+0000","last_undegraded":"2024-04-25T11:35:08.317356+0000","last_fullsized":"2024-04-25T11:35:08.317356+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:28.126631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"79'10","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.380742+0000","last_change":"2024-04-25T11:33:10.897050+0000","last_active":"2024-04-25T11:35:08.380742+0000","last_peered":"2024-04-25T11:35:08.380742+0000","last_clean":"2024-04-25T11:35:08.380742+0000","last_became_active":"2024-04-25T11:33:10.896273+0000","last_became_peered":"2024-04-25T11:33:10.896273+0000","last_unstale":"2024-04-25T11:35:08.380742+0000","last_undegraded":"2024-04-25T11:35:08.380742+0000","last_fullsized":"2024-04-25T11:35:08.380742+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:45:53.839823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"79'11","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.207505+0000","last_change":"2024-04-25T11:33:10.897323+0000","last_active":"2024-04-25T11:35:08.207505+0000","last_peered":"2024-04-25T11:35:08.207505+0000","last_clean":"2024-04-25T11:35:08.207505+0000","last_became_active":"2024-04-25T11:33:10.896951+0000","last_became_peered":"2024-04-25T11:33:10.896951+0000","last_unstale":"2024-04-25T11:35:08.207505+0000","last_undegraded":"2024-04-25T11:35:08.207505+0000","last_fullsized":"2024-04-25T11:35:08.207505+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:24:31.208338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"79'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T11:35:08.504797+0000","last_change":"2024-04-25T11:33:10.897566+0000","last_active":"2024-04-25T11:35:08.504797+0000","last_peered":"2024-04-25T11:35:08.504797+0000","last_clean":"2024-04-25T11:35:08.504797+0000","last_became_active":"2024-04-25T11:33:10.897212+0000","last_became_peered":"2024-04-25T11:33:10.897212+0000","last_unstale":"2024-04-25T11:35:08.504797+0000","last_undegraded":"2024-04-25T11:35:08.504797+0000","last_fullsized":"2024-04-25T11:35:08.504797+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:33:10.011283+0000","last_clean_scrub_stamp":"2024-04-25T11:33:10.011283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:55:29.995287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":601791,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1272,"num_read_kb":747,"num_write":370,"num_write_kb":1464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1805373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":299,"ondisk_log_size":299,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645348,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36204,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708948,"statfs":{"total":95995035648,"available":95957962752,"internally_reserved":0,"allocated":884736,"data_stored":423632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841577,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37024,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708128,"statfs":{"total":95995035648,"available":95957123072,"internally_reserved":0,"allocated":1728512,"data_stored":1214160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037805,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1302528,"data_stored":815996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234035,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28288,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716864,"statfs":{"total":95995035648,"available":95966068736,"internally_reserved":0,"allocated":1298432,"data_stored":816374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430264,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36788,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708364,"statfs":{"total":95995035648,"available":95957364736,"internally_reserved":0,"allocated":1486848,"data_stored":1007140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626493,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32052,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713100,"statfs":{"total":95995035648,"available":95962214400,"internally_reserved":0,"allocated":892928,"data_stored":422497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":962560,"data_stored":491168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":424618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40364,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704788,"statfs":{"total":95995035648,"available":95953702912,"internally_reserved":0,"allocated":888832,"data_stored":414953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28420,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716732,"statfs":{"total":95995035648,"available":95965933568,"internally_reserved":0,"allocated":1437696,"data_stored":947138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986247,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":970752,"data_stored":491039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822722,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28480,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716672,"statfs":{"total":95995035648,"available":95965872128,"internally_reserved":0,"allocated":1499136,"data_stored":1006286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":69510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":69381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":135510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":394716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":202017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T11:35:10.560 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:35:10.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:10 smithi156 ceph-mon[33045]: [25/Apr/2024:11:35:09] ENGINE Bus STARTING 2024-04-25T11:35:10.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:10 smithi156 ceph-mon[33045]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:35:10.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:10 smithi156 ceph-mon[33045]: [25/Apr/2024:11:35:09] ENGINE Serving on https://172.21.15.90:7150 2024-04-25T11:35:10.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:10 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3509944741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:35:10.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:10 smithi156 ceph-mon[33045]: [25/Apr/2024:11:35:09] ENGINE Serving on http://172.21.15.90:8765 2024-04-25T11:35:10.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:10 smithi156 ceph-mon[33045]: [25/Apr/2024:11:35:09] ENGINE Bus STARTED 2024-04-25T11:35:10.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:10 smithi156 ceph-mon[33045]: from='client.25070 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T11:35:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.717+0000 7fb64d884640 1 -- 172.21.15.90:0/433721149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb648103b00 msgr2=0x7fb648104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.717+0000 7fb64d884640 1 --2- 172.21.15.90:0/433721149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb648103b00 0x7fb648104800 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb63c009900 tx=0x7fb63c02a410 comp rx=0 tx=0).stop 2024-04-25T11:35:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.717+0000 7fb64d884640 1 -- 172.21.15.90:0/433721149 shutdown_connections 2024-04-25T11:35:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.717+0000 7fb64d884640 1 --2- 172.21.15.90:0/433721149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb648109a40 0x7fb64810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.717+0000 7fb64d884640 1 --2- 172.21.15.90:0/433721149 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb648104d40 0x7fb648105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:10.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.717+0000 7fb64d884640 1 --2- 172.21.15.90:0/433721149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb648103b00 0x7fb648104800 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:10.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.717+0000 7fb64d884640 1 -- 172.21.15.90:0/433721149 >> 172.21.15.90:0/433721149 conn(0x7fb6480fed10 msgr2=0x7fb6481011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:10.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.717+0000 7fb64d884640 1 -- 172.21.15.90:0/433721149 shutdown_connections 2024-04-25T11:35:10.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.717+0000 7fb64d884640 1 -- 172.21.15.90:0/433721149 wait complete. 2024-04-25T11:35:10.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.717+0000 7fb64d884640 1 Processor -- start 2024-04-25T11:35:10.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.718+0000 7fb64d884640 1 -- start start 2024-04-25T11:35:10.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.718+0000 7fb64d884640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb648103b00 0x7fb64819a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:10.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.718+0000 7fb64d884640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb648104d40 0x7fb64819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:10.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.718+0000 7fb64d884640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb648109a40 0x7fb6481a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:10.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.718+0000 7fb64d884640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb64810f600 con 0x7fb648109a40 2024-04-25T11:35:10.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.718+0000 7fb64d884640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb64810f480 con 0x7fb648103b00 2024-04-25T11:35:10.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.718+0000 7fb64d884640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb64810f780 con 0x7fb648104d40 2024-04-25T11:35:10.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.718+0000 7fb6477fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb648109a40 0x7fb6481a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:10.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.718+0000 7fb646ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb648103b00 0x7fb64819a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:10.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.718+0000 7fb6477fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb648109a40 0x7fb6481a16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34422/0 (socket says 172.21.15.90:34422) 2024-04-25T11:35:10.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.718+0000 7fb646ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb648103b00 0x7fb64819a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:52086/0 (socket says 172.21.15.90:52086) 2024-04-25T11:35:10.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.718+0000 7fb6477fe640 1 -- 172.21.15.90:0/2216824709 learned_addr learned my addr 172.21.15.90:0/2216824709 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:10.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.718+0000 7fb6467fc640 1 --2- 172.21.15.90:0/2216824709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb648104d40 0x7fb64819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:10.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.718+0000 7fb6477fe640 1 -- 172.21.15.90:0/2216824709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb648104d40 msgr2=0x7fb64819a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:10.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.718+0000 7fb6477fe640 1 --2- 172.21.15.90:0/2216824709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb648104d40 0x7fb64819a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:10.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.718+0000 7fb6477fe640 1 -- 172.21.15.90:0/2216824709 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb648103b00 msgr2=0x7fb64819a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:10.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.718+0000 7fb6477fe640 1 --2- 172.21.15.90:0/2216824709 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb648103b00 0x7fb64819a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:10.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.718+0000 7fb6477fe640 1 -- 172.21.15.90:0/2216824709 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6481a1d60 con 0x7fb648109a40 2024-04-25T11:35:10.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.718+0000 7fb646ffd640 1 --2- 172.21.15.90:0/2216824709 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb648103b00 0x7fb64819a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:35:10.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.718+0000 7fb6477fe640 1 --2- 172.21.15.90:0/2216824709 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb648109a40 0x7fb6481a16f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb64000eb10 tx=0x7fb64000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:10.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.719+0000 7fb64c882640 1 -- 172.21.15.90:0/2216824709 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb640004b20 con 0x7fb648109a40 2024-04-25T11:35:10.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.719+0000 7fb64c882640 1 -- 172.21.15.90:0/2216824709 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb640004cc0 con 0x7fb648109a40 2024-04-25T11:35:10.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.719+0000 7fb64d884640 1 -- 172.21.15.90:0/2216824709 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6481a1ff0 con 0x7fb648109a40 2024-04-25T11:35:10.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.719+0000 7fb64c882640 1 -- 172.21.15.90:0/2216824709 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb640010670 con 0x7fb648109a40 2024-04-25T11:35:10.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.719+0000 7fb64d884640 1 -- 172.21.15.90:0/2216824709 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6481a24b0 con 0x7fb648109a40 2024-04-25T11:35:10.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.720+0000 7fb64d884640 1 -- 172.21.15.90:0/2216824709 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb648108850 con 0x7fb648109a40 2024-04-25T11:35:10.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.721+0000 7fb64c882640 1 -- 172.21.15.90:0/2216824709 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 22) ==== 147640+0+0 (secure 0 0 0) 0x7fb640005020 con 0x7fb648109a40 2024-04-25T11:35:10.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.722+0000 7fb64c882640 1 --2- 172.21.15.90:0/2216824709 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb6300afc60 0x7fb6300b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:10.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.722+0000 7fb64c882640 1 -- 172.21.15.90:0/2216824709 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7fb6400df580 con 0x7fb648109a40 2024-04-25T11:35:10.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.722+0000 7fb646ffd640 1 --2- 172.21.15.90:0/2216824709 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb6300afc60 0x7fb6300b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:10.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.722+0000 7fb646ffd640 1 --2- 172.21.15.90:0/2216824709 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb6300afc60 0x7fb6300b2170 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb63c005ab0 tx=0x7fb63c031400 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:10.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.727+0000 7fb64c882640 1 -- 172.21.15.90:0/2216824709 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb6400a7e20 con 0x7fb648109a40 2024-04-25T11:35:10.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.915+0000 7fb64d884640 1 -- 172.21.15.90:0/2216824709 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb6480637a0 con 0x7fb648109a40 2024-04-25T11:35:10.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.916+0000 7fb64c882640 1 -- 172.21.15.90:0/2216824709 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+23079 (secure 0 0 0) 0x7fb6400a7720 con 0x7fb648109a40 2024-04-25T11:35:10.905 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:35:10.906 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":80,"fsid":"58bd4e7a-02f6-11ef-bc93-c7b262605968","created":"2024-04-25T11:25:23.574550+0000","modified":"2024-04-25T11:35:08.201635+0000","last_up_change":"2024-04-25T11:32:23.852211+0000","last_in_change":"2024-04-25T11:32:04.188507+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:28:42.746649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:33:08.442925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:33:09.352886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e54d888-e67b-4951-8ae7-e355fb6b10a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6803","nonce":1201629159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6805","nonce":1201629159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6809","nonce":1201629159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6807","nonce":1201629159}]},"public_addr":"172.21.15.90:6803/1201629159","cluster_addr":"172.21.15.90:6805/1201629159","heartbeat_back_addr":"172.21.15.90:6809/1201629159","heartbeat_front_addr":"172.21.15.90:6807/1201629159","state":["exists","up"]},{"osd":1,"uuid":"91a71c78-dae9-4bc7-9f6f-9e55cfefbe28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6801","nonce":3976851933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6803","nonce":3976851933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6807","nonce":3976851933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6805","nonce":3976851933}]},"public_addr":"172.21.15.138:6801/3976851933","cluster_addr":"172.21.15.138:6803/3976851933","heartbeat_back_addr":"172.21.15.138:6807/3976851933","heartbeat_front_addr":"172.21.15.138:6805/3976851933","state":["exists","up"]},{"osd":2,"uuid":"daff6c7f-80fa-4859-9339-ef7a5e840832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6801","nonce":1505807670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6803","nonce":1505807670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6807","nonce":1505807670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6805","nonce":1505807670}]},"public_addr":"172.21.15.156:6801/1505807670","cluster_addr":"172.21.15.156:6803/1505807670","heartbeat_back_addr":"172.21.15.156:6807/1505807670","heartbeat_front_addr":"172.21.15.156:6805/1505807670","state":["exists","up"]},{"osd":3,"uuid":"33fd94ca-7465-401a-bc1e-706b6b31b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6811","nonce":2520012383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6813","nonce":2520012383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6817","nonce":2520012383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6815","nonce":2520012383}]},"public_addr":"172.21.15.90:6811/2520012383","cluster_addr":"172.21.15.90:6813/2520012383","heartbeat_back_addr":"172.21.15.90:6817/2520012383","heartbeat_front_addr":"172.21.15.90:6815/2520012383","state":["exists","up"]},{"osd":4,"uuid":"c5326212-750c-49fd-96e3-d1e62bb4b7f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6809","nonce":3752678279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6811","nonce":3752678279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6815","nonce":3752678279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6813","nonce":3752678279}]},"public_addr":"172.21.15.138:6809/3752678279","cluster_addr":"172.21.15.138:6811/3752678279","heartbeat_back_addr":"172.21.15.138:6815/3752678279","heartbeat_front_addr":"172.21.15.138:6813/3752678279","state":["exists","up"]},{"osd":5,"uuid":"d04a1156-820b-4b43-be89-e16a8910c3aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6809","nonce":3591210191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6811","nonce":3591210191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6815","nonce":3591210191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6813","nonce":3591210191}]},"public_addr":"172.21.15.156:6809/3591210191","cluster_addr":"172.21.15.156:6811/3591210191","heartbeat_back_addr":"172.21.15.156:6815/3591210191","heartbeat_front_addr":"172.21.15.156:6813/3591210191","state":["exists","up"]},{"osd":6,"uuid":"75675aa3-e978-490c-8205-13d9868a3d6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6819","nonce":2364955456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6821","nonce":2364955456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6825","nonce":2364955456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6823","nonce":2364955456}]},"public_addr":"172.21.15.90:6819/2364955456","cluster_addr":"172.21.15.90:6821/2364955456","heartbeat_back_addr":"172.21.15.90:6825/2364955456","heartbeat_front_addr":"172.21.15.90:6823/2364955456","state":["exists","up"]},{"osd":7,"uuid":"1a95fdf5-f548-4271-bbcd-e330f7955eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6817","nonce":4012476387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6819","nonce":4012476387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6823","nonce":4012476387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6821","nonce":4012476387}]},"public_addr":"172.21.15.138:6817/4012476387","cluster_addr":"172.21.15.138:6819/4012476387","heartbeat_back_addr":"172.21.15.138:6823/4012476387","heartbeat_front_addr":"172.21.15.138:6821/4012476387","state":["exists","up"]},{"osd":8,"uuid":"605e10b5-5806-4c1f-8e28-a7994d34ddce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6817","nonce":101092224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6819","nonce":101092224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6823","nonce":101092224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6821","nonce":101092224}]},"public_addr":"172.21.15.156:6817/101092224","cluster_addr":"172.21.15.156:6819/101092224","heartbeat_back_addr":"172.21.15.156:6823/101092224","heartbeat_front_addr":"172.21.15.156:6821/101092224","state":["exists","up"]},{"osd":9,"uuid":"d70fdf5e-9d0f-483d-a7d8-505b79b8a418","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6827","nonce":1171265966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6829","nonce":1171265966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6833","nonce":1171265966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6831","nonce":1171265966}]},"public_addr":"172.21.15.90:6827/1171265966","cluster_addr":"172.21.15.90:6829/1171265966","heartbeat_back_addr":"172.21.15.90:6833/1171265966","heartbeat_front_addr":"172.21.15.90:6831/1171265966","state":["exists","up"]},{"osd":10,"uuid":"1ec86c7d-c3a4-432d-9664-8e40836e1e20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6825","nonce":4264666232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6827","nonce":4264666232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6831","nonce":4264666232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6829","nonce":4264666232}]},"public_addr":"172.21.15.138:6825/4264666232","cluster_addr":"172.21.15.138:6827/4264666232","heartbeat_back_addr":"172.21.15.138:6831/4264666232","heartbeat_front_addr":"172.21.15.138:6829/4264666232","state":["exists","up"]},{"osd":11,"uuid":"51cbe729-292e-4d7b-b8e9-e1e2821ffe76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6825","nonce":1224251744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6827","nonce":1224251744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6831","nonce":1224251744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6829","nonce":1224251744}]},"public_addr":"172.21.15.156:6825/1224251744","cluster_addr":"172.21.15.156:6827/1224251744","heartbeat_back_addr":"172.21.15.156:6831/1224251744","heartbeat_front_addr":"172.21.15.156:6829/1224251744","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:27:50.794226+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:15.030278+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:38.518163+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:03.107351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:27.500707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:51.463545+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:16.015076+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:41.182071+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:05.362309+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:31.048936+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:56.488509+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:32:22.129178+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":7}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/813599224":"2024-04-26T11:35:08.201598+0000","172.21.15.90:6800/219449516":"2024-04-26T11:35:08.201598+0000","172.21.15.90:0/478282854":"2024-04-26T11:35:08.201598+0000","172.21.15.90:0/2202319376":"2024-04-26T11:35:08.201598+0000","172.21.15.90:0/2381633953":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6800/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/2190846900":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/888963322":"2024-04-26T11:35:08.201598+0000","172.21.15.90:6801/219449516":"2024-04-26T11:35:08.201598+0000","172.21.15.90:0/1707792941":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6800/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6801/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/6061052":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1849633392":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6801/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/2277490702":"2024-04-26T11:26:16.655278+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:35:10.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.919+0000 7fb64d884640 1 -- 172.21.15.90:0/2216824709 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb6300afc60 msgr2=0x7fb6300b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:10.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.919+0000 7fb64d884640 1 --2- 172.21.15.90:0/2216824709 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb6300afc60 0x7fb6300b2170 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb63c005ab0 tx=0x7fb63c031400 comp rx=0 tx=0).stop 2024-04-25T11:35:10.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.919+0000 7fb64d884640 1 -- 172.21.15.90:0/2216824709 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb648109a40 msgr2=0x7fb6481a16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:10.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.919+0000 7fb64d884640 1 --2- 172.21.15.90:0/2216824709 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb648109a40 0x7fb6481a16f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb64000eb10 tx=0x7fb64000c2d0 comp rx=0 tx=0).stop 2024-04-25T11:35:10.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.919+0000 7fb64d884640 1 -- 172.21.15.90:0/2216824709 shutdown_connections 2024-04-25T11:35:10.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.919+0000 7fb64d884640 1 --2- 172.21.15.90:0/2216824709 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb6300afc60 0x7fb6300b2170 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:10.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.919+0000 7fb64d884640 1 --2- 172.21.15.90:0/2216824709 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb648109a40 0x7fb6481a16f0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:10.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.919+0000 7fb64d884640 1 --2- 172.21.15.90:0/2216824709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb648104d40 0x7fb64819a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:10.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.919+0000 7fb64d884640 1 --2- 172.21.15.90:0/2216824709 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb648103b00 0x7fb64819a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:10.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.919+0000 7fb64d884640 1 -- 172.21.15.90:0/2216824709 >> 172.21.15.90:0/2216824709 conn(0x7fb6480fed10 msgr2=0x7fb648105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:10.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.919+0000 7fb64d884640 1 -- 172.21.15.90:0/2216824709 shutdown_connections 2024-04-25T11:35:10.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:10.919+0000 7fb64d884640 1 -- 172.21.15.90:0/2216824709 wait complete. 2024-04-25T11:35:10.924 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T11:28:42.746649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T11:33:08.442925+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T11:33:09.352886+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T11:35:10.925 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T11:35:11.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.050+0000 7fe58bb9e640 1 -- 172.21.15.90:0/3924047522 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe584104680 msgr2=0x7fe58410add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:11.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.050+0000 7fe58bb9e640 1 --2- 172.21.15.90:0/3924047522 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe584104680 0x7fe58410add0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe57c009ca0 tx=0x7fe57c02a8c0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.050+0000 7fe58bb9e640 1 -- 172.21.15.90:0/3924047522 shutdown_connections 2024-04-25T11:35:11.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.050+0000 7fe58bb9e640 1 --2- 172.21.15.90:0/3924047522 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe584104680 0x7fe58410add0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.050+0000 7fe58bb9e640 1 --2- 172.21.15.90:0/3924047522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe584103b20 0x7fe584103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.050+0000 7fe58bb9e640 1 --2- 172.21.15.90:0/3924047522 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe584102fc0 0x7fe5841033d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.050+0000 7fe58bb9e640 1 -- 172.21.15.90:0/3924047522 >> 172.21.15.90:0/3924047522 conn(0x7fe5840fd340 msgr2=0x7fe5840ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:11.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.050+0000 7fe58bb9e640 1 -- 172.21.15.90:0/3924047522 shutdown_connections 2024-04-25T11:35:11.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.050+0000 7fe58bb9e640 1 -- 172.21.15.90:0/3924047522 wait complete. 2024-04-25T11:35:11.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.050+0000 7fe58bb9e640 1 Processor -- start 2024-04-25T11:35:11.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.051+0000 7fe58bb9e640 1 -- start start 2024-04-25T11:35:11.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.051+0000 7fe58bb9e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe584102fc0 0x7fe584102280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:11.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.051+0000 7fe58bb9e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe584103b20 0x7fe58419ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:11.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.051+0000 7fe58bb9e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe58419f280 0x7fe5841a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:11.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.051+0000 7fe58bb9e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe58410e750 con 0x7fe584103b20 2024-04-25T11:35:11.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.051+0000 7fe58bb9e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe58410e5d0 con 0x7fe58419f280 2024-04-25T11:35:11.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.051+0000 7fe58bb9e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe58410e8d0 con 0x7fe584102fc0 2024-04-25T11:35:11.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.051+0000 7fe589139640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe584103b20 0x7fe58419ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:11.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.051+0000 7fe589139640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe584103b20 0x7fe58419ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34440/0 (socket says 172.21.15.90:34440) 2024-04-25T11:35:11.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.051+0000 7fe589139640 1 -- 172.21.15.90:0/1953283948 learned_addr learned my addr 172.21.15.90:0/1953283948 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:11.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.051+0000 7fe58993a640 1 --2- 172.21.15.90:0/1953283948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe584102fc0 0x7fe584102280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:11.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.051+0000 7fe58a13b640 1 --2- 172.21.15.90:0/1953283948 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe58419f280 0x7fe5841a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:11.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.051+0000 7fe589139640 1 -- 172.21.15.90:0/1953283948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe584102fc0 msgr2=0x7fe584102280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:11.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.051+0000 7fe589139640 1 --2- 172.21.15.90:0/1953283948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe584102fc0 0x7fe584102280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.051+0000 7fe589139640 1 -- 172.21.15.90:0/1953283948 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe58419f280 msgr2=0x7fe5841a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:11.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.051+0000 7fe589139640 1 --2- 172.21.15.90:0/1953283948 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe58419f280 0x7fe5841a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.051+0000 7fe589139640 1 -- 172.21.15.90:0/1953283948 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5841a1ee0 con 0x7fe584103b20 2024-04-25T11:35:11.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.051+0000 7fe58a13b640 1 --2- 172.21.15.90:0/1953283948 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe58419f280 0x7fe5841a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:35:11.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.052+0000 7fe589139640 1 --2- 172.21.15.90:0/1953283948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe584103b20 0x7fe58419ed40 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fe57400d770 tx=0x7fe57400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:11.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.052+0000 7fe57affd640 1 -- 172.21.15.90:0/1953283948 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe574017070 con 0x7fe584103b20 2024-04-25T11:35:11.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.052+0000 7fe58bb9e640 1 -- 172.21.15.90:0/1953283948 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5841a21d0 con 0x7fe584103b20 2024-04-25T11:35:11.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.052+0000 7fe57affd640 1 -- 172.21.15.90:0/1953283948 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5740054d0 con 0x7fe584103b20 2024-04-25T11:35:11.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.052+0000 7fe58bb9e640 1 -- 172.21.15.90:0/1953283948 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5841a2720 con 0x7fe584103b20 2024-04-25T11:35:11.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.052+0000 7fe57affd640 1 -- 172.21.15.90:0/1953283948 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe574004c30 con 0x7fe584103b20 2024-04-25T11:35:11.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.053+0000 7fe58bb9e640 1 -- 172.21.15.90:0/1953283948 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe584069c70 con 0x7fe584103b20 2024-04-25T11:35:11.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.056+0000 7fe57affd640 1 -- 172.21.15.90:0/1953283948 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 22) ==== 147640+0+0 (secure 0 0 0) 0x7fe574004dd0 con 0x7fe584103b20 2024-04-25T11:35:11.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.056+0000 7fe57affd640 1 --2- 172.21.15.90:0/1953283948 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe5700afc60 0x7fe5700b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:11.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.057+0000 7fe57affd640 1 -- 172.21.15.90:0/1953283948 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7fe5740dee80 con 0x7fe584103b20 2024-04-25T11:35:11.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.057+0000 7fe58993a640 1 --2- 172.21.15.90:0/1953283948 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe5700afc60 0x7fe5700b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:11.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.057+0000 7fe58993a640 1 --2- 172.21.15.90:0/1953283948 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe5700afc60 0x7fe5700b2170 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe5800095f0 tx=0x7fe580005f90 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:11.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.060+0000 7fe57affd640 1 -- 172.21.15.90:0/1953283948 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe5740a70d0 con 0x7fe584103b20 2024-04-25T11:35:11.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.259+0000 7fe58bb9e640 1 -- 172.21.15.90:0/1953283948 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe58402dda0 con 0x7fe584103b20 2024-04-25T11:35:11.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.259+0000 7fe57affd640 1 -- 172.21.15.90:0/1953283948 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) ==== 93+0+10 (secure 0 0 0) 0x7fe5740ab3d0 con 0x7fe584103b20 2024-04-25T11:35:11.251 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T11:35:11.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.261+0000 7fe58bb9e640 1 -- 172.21.15.90:0/1953283948 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe5700afc60 msgr2=0x7fe5700b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:11.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.261+0000 7fe58bb9e640 1 --2- 172.21.15.90:0/1953283948 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe5700afc60 0x7fe5700b2170 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe5800095f0 tx=0x7fe580005f90 comp rx=0 tx=0).stop 2024-04-25T11:35:11.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.261+0000 7fe58bb9e640 1 -- 172.21.15.90:0/1953283948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe584103b20 msgr2=0x7fe58419ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:11.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.261+0000 7fe58bb9e640 1 --2- 172.21.15.90:0/1953283948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe584103b20 0x7fe58419ed40 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fe57400d770 tx=0x7fe57400dc30 comp rx=0 tx=0).stop 2024-04-25T11:35:11.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.261+0000 7fe58bb9e640 1 -- 172.21.15.90:0/1953283948 shutdown_connections 2024-04-25T11:35:11.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.261+0000 7fe58bb9e640 1 --2- 172.21.15.90:0/1953283948 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe5700afc60 0x7fe5700b2170 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.261+0000 7fe58bb9e640 1 --2- 172.21.15.90:0/1953283948 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe58419f280 0x7fe5841a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.261+0000 7fe58bb9e640 1 --2- 172.21.15.90:0/1953283948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe584103b20 0x7fe58419ed40 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.261+0000 7fe58bb9e640 1 --2- 172.21.15.90:0/1953283948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe584102fc0 0x7fe584102280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.261+0000 7fe58bb9e640 1 -- 172.21.15.90:0/1953283948 >> 172.21.15.90:0/1953283948 conn(0x7fe5840fd340 msgr2=0x7fe5840fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:11.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.261+0000 7fe58bb9e640 1 -- 172.21.15.90:0/1953283948 shutdown_connections 2024-04-25T11:35:11.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.261+0000 7fe58bb9e640 1 -- 172.21.15.90:0/1953283948 wait complete. 2024-04-25T11:35:11.262 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T11:35:11.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.387+0000 7fa2bfcf2640 1 -- 172.21.15.90:0/2083630850 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2b8103ee0 msgr2=0x7fa2b81027c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:11.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.387+0000 7fa2bfcf2640 1 --2- 172.21.15.90:0/2083630850 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2b8103ee0 0x7fa2b81027c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4009900 tx=0x7fa2b402a410 comp rx=0 tx=0).stop 2024-04-25T11:35:11.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.387+0000 7fa2bfcf2640 1 -- 172.21.15.90:0/2083630850 shutdown_connections 2024-04-25T11:35:11.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.387+0000 7fa2bfcf2640 1 --2- 172.21.15.90:0/2083630850 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa2b8108a10 0x7fa2b810ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.387+0000 7fa2bfcf2640 1 --2- 172.21.15.90:0/2083630850 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa2b81043c0 0x7fa2b8102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.387+0000 7fa2bfcf2640 1 --2- 172.21.15.90:0/2083630850 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2b8103ee0 0x7fa2b81027c0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.387+0000 7fa2bfcf2640 1 -- 172.21.15.90:0/2083630850 >> 172.21.15.90:0/2083630850 conn(0x7fa2b80fd340 msgr2=0x7fa2b80ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:11.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.387+0000 7fa2bfcf2640 1 -- 172.21.15.90:0/2083630850 shutdown_connections 2024-04-25T11:35:11.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.387+0000 7fa2bfcf2640 1 -- 172.21.15.90:0/2083630850 wait complete. 2024-04-25T11:35:11.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.387+0000 7fa2bfcf2640 1 Processor -- start 2024-04-25T11:35:11.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.388+0000 7fa2bfcf2640 1 -- start start 2024-04-25T11:35:11.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.388+0000 7fa2bfcf2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa2b8103ee0 0x7fa2b8198d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:11.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.388+0000 7fa2bfcf2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa2b8108a10 0x7fa2b8199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:11.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.388+0000 7fa2bfcf2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2b819e1d0 0x7fa2b81a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:11.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.388+0000 7fa2bfcf2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b810e4d0 con 0x7fa2b819e1d0 2024-04-25T11:35:11.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.388+0000 7fa2bfcf2640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b810e350 con 0x7fa2b8108a10 2024-04-25T11:35:11.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.388+0000 7fa2bfcf2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b810e650 con 0x7fa2b8103ee0 2024-04-25T11:35:11.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.388+0000 7fa2be28f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2b819e1d0 0x7fa2b81a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:11.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.388+0000 7fa2be28f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2b819e1d0 0x7fa2b81a05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34464/0 (socket says 172.21.15.90:34464) 2024-04-25T11:35:11.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.388+0000 7fa2be28f640 1 -- 172.21.15.90:0/3184600101 learned_addr learned my addr 172.21.15.90:0/3184600101 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:11.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.388+0000 7fa2bda8e640 1 --2- 172.21.15.90:0/3184600101 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa2b8103ee0 0x7fa2b8198d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:11.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.388+0000 7fa2bd28d640 1 --2- 172.21.15.90:0/3184600101 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa2b8108a10 0x7fa2b8199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:11.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.388+0000 7fa2be28f640 1 -- 172.21.15.90:0/3184600101 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa2b8103ee0 msgr2=0x7fa2b8198d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:11.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.388+0000 7fa2be28f640 1 --2- 172.21.15.90:0/3184600101 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa2b8103ee0 0x7fa2b8198d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.388+0000 7fa2be28f640 1 -- 172.21.15.90:0/3184600101 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa2b8108a10 msgr2=0x7fa2b8199290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:11.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.388+0000 7fa2be28f640 1 --2- 172.21.15.90:0/3184600101 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa2b8108a10 0x7fa2b8199290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.388+0000 7fa2be28f640 1 -- 172.21.15.90:0/3184600101 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2b81a0c60 con 0x7fa2b819e1d0 2024-04-25T11:35:11.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.388+0000 7fa2be28f640 1 --2- 172.21.15.90:0/3184600101 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2b819e1d0 0x7fa2b81a05f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa2b000eca0 tx=0x7fa2b000b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:11.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.389+0000 7fa2aeffd640 1 -- 172.21.15.90:0/3184600101 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b0004b20 con 0x7fa2b819e1d0 2024-04-25T11:35:11.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.389+0000 7fa2aeffd640 1 -- 172.21.15.90:0/3184600101 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2b0004cc0 con 0x7fa2b819e1d0 2024-04-25T11:35:11.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.389+0000 7fa2bfcf2640 1 -- 172.21.15.90:0/3184600101 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2b8101710 con 0x7fa2b819e1d0 2024-04-25T11:35:11.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.389+0000 7fa2aeffd640 1 -- 172.21.15.90:0/3184600101 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b0017670 con 0x7fa2b819e1d0 2024-04-25T11:35:11.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.389+0000 7fa2bfcf2640 1 -- 172.21.15.90:0/3184600101 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2b8101c10 con 0x7fa2b819e1d0 2024-04-25T11:35:11.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.390+0000 7fa2bfcf2640 1 -- 172.21.15.90:0/3184600101 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2b80692e0 con 0x7fa2b819e1d0 2024-04-25T11:35:11.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.391+0000 7fa2aeffd640 1 -- 172.21.15.90:0/3184600101 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 22) ==== 147640+0+0 (secure 0 0 0) 0x7fa2b0005020 con 0x7fa2b819e1d0 2024-04-25T11:35:11.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.392+0000 7fa2aeffd640 1 --2- 172.21.15.90:0/3184600101 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa2a40afc90 0x7fa2a40b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:11.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.392+0000 7fa2bda8e640 1 --2- 172.21.15.90:0/3184600101 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa2a40afc90 0x7fa2a40b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:11.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.392+0000 7fa2aeffd640 1 -- 172.21.15.90:0/3184600101 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7fa2b00e0ab0 con 0x7fa2b819e1d0 2024-04-25T11:35:11.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.392+0000 7fa2bda8e640 1 --2- 172.21.15.90:0/3184600101 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa2a40afc90 0x7fa2a40b21a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa2b819a140 tx=0x7fa2b4031400 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:11.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.396+0000 7fa2aeffd640 1 -- 172.21.15.90:0/3184600101 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa2b00a8d00 con 0x7fa2b819e1d0 2024-04-25T11:35:11.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:11 smithi138 ceph-mon[32328]: from='client.15480 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T11:35:11.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:11 smithi138 ceph-mon[32328]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:35:11.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:11 smithi138 ceph-mon[32328]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-04-25T11:35:11.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:11 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:11.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:11 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:11.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:11 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:11.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:11 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:11.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:11 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:11.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:11 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:11.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:11 smithi138 ceph-mon[32328]: from='client.25069 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T11:35:11.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:11 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2216824709' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:35:11.553 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:11 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1953283948' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:35:11.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.587+0000 7fa2bfcf2640 1 -- 172.21.15.90:0/3184600101 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa2b81a0ef0 con 0x7fa2b819e1d0 2024-04-25T11:35:11.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.588+0000 7fa2aeffd640 1 -- 172.21.15.90:0/3184600101 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v80) ==== 104+0+11 (secure 0 0 0) 0x7fa2b00ad000 con 0x7fa2b819e1d0 2024-04-25T11:35:11.577 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T11:35:11.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.590+0000 7fa2bfcf2640 1 -- 172.21.15.90:0/3184600101 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa2a40afc90 msgr2=0x7fa2a40b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:11.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.590+0000 7fa2bfcf2640 1 --2- 172.21.15.90:0/3184600101 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa2a40afc90 0x7fa2a40b21a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa2b819a140 tx=0x7fa2b4031400 comp rx=0 tx=0).stop 2024-04-25T11:35:11.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.590+0000 7fa2bfcf2640 1 -- 172.21.15.90:0/3184600101 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2b819e1d0 msgr2=0x7fa2b81a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:11.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.590+0000 7fa2bfcf2640 1 --2- 172.21.15.90:0/3184600101 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2b819e1d0 0x7fa2b81a05f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa2b000eca0 tx=0x7fa2b000b690 comp rx=0 tx=0).stop 2024-04-25T11:35:11.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.590+0000 7fa2bfcf2640 1 -- 172.21.15.90:0/3184600101 shutdown_connections 2024-04-25T11:35:11.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.590+0000 7fa2bfcf2640 1 --2- 172.21.15.90:0/3184600101 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa2a40afc90 0x7fa2a40b21a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.590+0000 7fa2bfcf2640 1 --2- 172.21.15.90:0/3184600101 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2b819e1d0 0x7fa2b81a05f0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.590+0000 7fa2bfcf2640 1 --2- 172.21.15.90:0/3184600101 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa2b8108a10 0x7fa2b8199290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.590+0000 7fa2bfcf2640 1 --2- 172.21.15.90:0/3184600101 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa2b8103ee0 0x7fa2b8198d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.590+0000 7fa2bfcf2640 1 -- 172.21.15.90:0/3184600101 >> 172.21.15.90:0/3184600101 conn(0x7fa2b80fd340 msgr2=0x7fa2b80ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:11.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.590+0000 7fa2bfcf2640 1 -- 172.21.15.90:0/3184600101 shutdown_connections 2024-04-25T11:35:11.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.590+0000 7fa2bfcf2640 1 -- 172.21.15.90:0/3184600101 wait complete. 2024-04-25T11:35:11.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:11 smithi156 ceph-mon[33045]: from='client.15480 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T11:35:11.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:11 smithi156 ceph-mon[33045]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:35:11.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:11 smithi156 ceph-mon[33045]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-04-25T11:35:11.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:11 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:11.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:11 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:11.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:11 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:11.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:11 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:11.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:11 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:11.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:11 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:11.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:11 smithi156 ceph-mon[33045]: from='client.25069 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T11:35:11.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:11 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2216824709' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:35:11.589 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:11 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1953283948' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:35:11.590 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:11 smithi090 ceph-mon[29575]: from='client.15480 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T11:35:11.590 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:11 smithi090 ceph-mon[29575]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:35:11.590 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:11 smithi090 ceph-mon[29575]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-04-25T11:35:11.590 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:11 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:11.590 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:11 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:11.590 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:11 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:11.590 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:11 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:11.590 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:11 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:11.591 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:11 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:11.591 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:11 smithi090 ceph-mon[29575]: from='client.25069 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T11:35:11.591 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:11 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2216824709' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:35:11.591 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:11 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1953283948' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:35:11.592 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T11:35:11.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.720+0000 7f12ca157640 1 -- 172.21.15.90:0/1215912970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12c41029f0 msgr2=0x7f12c41036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:11.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.720+0000 7f12ca157640 1 --2- 172.21.15.90:0/1215912970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12c41029f0 0x7f12c41036f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009860 tx=0x7f12b402a340 comp rx=0 tx=0).stop 2024-04-25T11:35:11.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.720+0000 7f12ca157640 1 -- 172.21.15.90:0/1215912970 shutdown_connections 2024-04-25T11:35:11.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.720+0000 7f12ca157640 1 --2- 172.21.15.90:0/1215912970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12c4108930 0x7f12c410ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.720+0000 7f12ca157640 1 --2- 172.21.15.90:0/1215912970 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f12c4103c30 0x7f12c4104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.720+0000 7f12ca157640 1 --2- 172.21.15.90:0/1215912970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12c41029f0 0x7f12c41036f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.720+0000 7f12ca157640 1 -- 172.21.15.90:0/1215912970 >> 172.21.15.90:0/1215912970 conn(0x7f12c40fdc00 msgr2=0x7f12c4100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:11.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.720+0000 7f12ca157640 1 -- 172.21.15.90:0/1215912970 shutdown_connections 2024-04-25T11:35:11.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.720+0000 7f12ca157640 1 -- 172.21.15.90:0/1215912970 wait complete. 2024-04-25T11:35:11.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.721+0000 7f12ca157640 1 Processor -- start 2024-04-25T11:35:11.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.721+0000 7f12ca157640 1 -- start start 2024-04-25T11:35:11.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.721+0000 7f12ca157640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12c41029f0 0x7f12c41991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.721+0000 7f12ca157640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12c4103c30 0x7f12c41996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.721+0000 7f12ca157640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f12c4108930 0x7f12c41a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.721+0000 7f12ca157640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f12c410e4f0 con 0x7f12c41029f0 2024-04-25T11:35:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.721+0000 7f12ca157640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f12c410e370 con 0x7f12c4108930 2024-04-25T11:35:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.721+0000 7f12ca157640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f12c410e670 con 0x7f12c4103c30 2024-04-25T11:35:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.721+0000 7f12c37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12c41029f0 0x7f12c41991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.721+0000 7f12c2ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12c4103c30 0x7f12c41996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.721+0000 7f12c2ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12c4103c30 0x7f12c41996f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:48058/0 (socket says 172.21.15.90:48058) 2024-04-25T11:35:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.721+0000 7f12c2ffd640 1 -- 172.21.15.90:0/1936851046 learned_addr learned my addr 172.21.15.90:0/1936851046 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.721+0000 7f12c37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12c41029f0 0x7f12c41991b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33662/0 (socket says 172.21.15.90:33662) 2024-04-25T11:35:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.721+0000 7f12c3fff640 1 --2- 172.21.15.90:0/1936851046 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f12c4108930 0x7f12c41a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.722+0000 7f12c2ffd640 1 -- 172.21.15.90:0/1936851046 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f12c4108930 msgr2=0x7f12c41a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.722+0000 7f12c2ffd640 1 --2- 172.21.15.90:0/1936851046 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f12c4108930 0x7f12c41a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.722+0000 7f12c2ffd640 1 -- 172.21.15.90:0/1936851046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12c41029f0 msgr2=0x7f12c41991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:11.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.722+0000 7f12c2ffd640 1 --2- 172.21.15.90:0/1936851046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12c41029f0 0x7f12c41991b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.722+0000 7f12c2ffd640 1 -- 172.21.15.90:0/1936851046 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12c41a0ca0 con 0x7f12c4103c30 2024-04-25T11:35:11.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.722+0000 7f12c2ffd640 1 --2- 172.21.15.90:0/1936851046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12c4103c30 0x7f12c41996f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f12b800e9f0 tx=0x7f12b800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:11.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.722+0000 7f12c0ff9640 1 -- 172.21.15.90:0/1936851046 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b800cd80 con 0x7f12c4103c30 2024-04-25T11:35:11.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.722+0000 7f12c0ff9640 1 -- 172.21.15.90:0/1936851046 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12b80054d0 con 0x7f12c4103c30 2024-04-25T11:35:11.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.722+0000 7f12ca157640 1 -- 172.21.15.90:0/1936851046 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12c41a0f90 con 0x7f12c4103c30 2024-04-25T11:35:11.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.722+0000 7f12c0ff9640 1 -- 172.21.15.90:0/1936851046 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b8010690 con 0x7f12c4103c30 2024-04-25T11:35:11.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.722+0000 7f12ca157640 1 -- 172.21.15.90:0/1936851046 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12c41a13f0 con 0x7f12c4103c30 2024-04-25T11:35:11.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.723+0000 7f12ca157640 1 -- 172.21.15.90:0/1936851046 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12c4107740 con 0x7f12c4103c30 2024-04-25T11:35:11.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.724+0000 7f12c0ff9640 1 -- 172.21.15.90:0/1936851046 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 22) ==== 147640+0+0 (secure 0 0 0) 0x7f12b8004e90 con 0x7f12c4103c30 2024-04-25T11:35:11.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.725+0000 7f12c0ff9640 1 --2- 172.21.15.90:0/1936851046 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f12ac0afce0 0x7f12ac0b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:11.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.725+0000 7f12c37fe640 1 --2- 172.21.15.90:0/1936851046 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f12ac0afce0 0x7f12ac0b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:11.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.725+0000 7f12c0ff9640 1 -- 172.21.15.90:0/1936851046 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f12b80df3f0 con 0x7f12c4103c30 2024-04-25T11:35:11.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.726+0000 7f12c37fe640 1 --2- 172.21.15.90:0/1936851046 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f12ac0afce0 0x7f12ac0b21f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009830 tx=0x7f12b4030450 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:11.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.730+0000 7f12c0ff9640 1 -- 172.21.15.90:0/1936851046 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f12b80a7c30 con 0x7f12c4103c30 2024-04-25T11:35:11.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.936+0000 7f12ca157640 1 -- 172.21.15.90:0/1936851046 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f12c40637d0 con 0x7f12c4103c30 2024-04-25T11:35:11.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.936+0000 7f12c0ff9640 1 -- 172.21.15.90:0/1936851046 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v80) ==== 100+0+11 (secure 0 0 0) 0x7f12b80a7530 con 0x7f12c4103c30 2024-04-25T11:35:11.925 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T11:35:11.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.938+0000 7f12ca157640 1 -- 172.21.15.90:0/1936851046 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f12ac0afce0 msgr2=0x7f12ac0b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:11.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.938+0000 7f12ca157640 1 --2- 172.21.15.90:0/1936851046 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f12ac0afce0 0x7f12ac0b21f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009830 tx=0x7f12b4030450 comp rx=0 tx=0).stop 2024-04-25T11:35:11.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.938+0000 7f12ca157640 1 -- 172.21.15.90:0/1936851046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12c4103c30 msgr2=0x7f12c41996f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:11.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.938+0000 7f12ca157640 1 --2- 172.21.15.90:0/1936851046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12c4103c30 0x7f12c41996f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f12b800e9f0 tx=0x7f12b800edf0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.938+0000 7f12ca157640 1 -- 172.21.15.90:0/1936851046 shutdown_connections 2024-04-25T11:35:11.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.938+0000 7f12ca157640 1 --2- 172.21.15.90:0/1936851046 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f12ac0afce0 0x7f12ac0b21f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.938+0000 7f12ca157640 1 --2- 172.21.15.90:0/1936851046 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f12c4108930 0x7f12c41a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.938+0000 7f12ca157640 1 --2- 172.21.15.90:0/1936851046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12c4103c30 0x7f12c41996f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.938+0000 7f12ca157640 1 --2- 172.21.15.90:0/1936851046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12c41029f0 0x7f12c41991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:11.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.938+0000 7f12ca157640 1 -- 172.21.15.90:0/1936851046 >> 172.21.15.90:0/1936851046 conn(0x7f12c40fdc00 msgr2=0x7f12c4104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:11.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.939+0000 7f12ca157640 1 -- 172.21.15.90:0/1936851046 shutdown_connections 2024-04-25T11:35:11.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:11.939+0000 7f12ca157640 1 -- 172.21.15.90:0/1936851046 wait complete. 2024-04-25T11:35:11.939 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:35:12.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.062+0000 7f31d74e7640 1 -- 172.21.15.90:0/1880768541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31d0106f10 msgr2=0x7f31d0109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:12.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.062+0000 7f31d74e7640 1 --2- 172.21.15.90:0/1880768541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31d0106f10 0x7f31d0109330 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f31cc009900 tx=0x7f31cc02a410 comp rx=0 tx=0).stop 2024-04-25T11:35:12.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.062+0000 7f31d74e7640 1 -- 172.21.15.90:0/1880768541 shutdown_connections 2024-04-25T11:35:12.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.062+0000 7f31d74e7640 1 --2- 172.21.15.90:0/1880768541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31d0109870 0x7f31d010bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.062+0000 7f31d74e7640 1 --2- 172.21.15.90:0/1880768541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31d0106f10 0x7f31d0109330 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.062+0000 7f31d74e7640 1 --2- 172.21.15.90:0/1880768541 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31d01045b0 0x7f31d01069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.062+0000 7f31d74e7640 1 -- 172.21.15.90:0/1880768541 >> 172.21.15.90:0/1880768541 conn(0x7f31d00fdbd0 msgr2=0x7f31d0100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:12.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.062+0000 7f31d74e7640 1 -- 172.21.15.90:0/1880768541 shutdown_connections 2024-04-25T11:35:12.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.062+0000 7f31d74e7640 1 -- 172.21.15.90:0/1880768541 wait complete. 2024-04-25T11:35:12.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.062+0000 7f31d74e7640 1 Processor -- start 2024-04-25T11:35:12.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.063+0000 7f31d74e7640 1 -- start start 2024-04-25T11:35:12.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.063+0000 7f31d74e7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31d01045b0 0x7f31d019a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.063+0000 7f31d74e7640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31d0106f10 0x7f31d019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.063+0000 7f31d74e7640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31d0109870 0x7f31d01a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.063+0000 7f31d74e7640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f31d010f5d0 con 0x7f31d0106f10 2024-04-25T11:35:12.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.063+0000 7f31d74e7640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f31d010f450 con 0x7f31d0109870 2024-04-25T11:35:12.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.063+0000 7f31d74e7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f31d010f750 con 0x7f31d01045b0 2024-04-25T11:35:12.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.063+0000 7f31d4a82640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31d0106f10 0x7f31d019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.063+0000 7f31d4a82640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31d0106f10 0x7f31d019a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33682/0 (socket says 172.21.15.90:33682) 2024-04-25T11:35:12.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.063+0000 7f31d4a82640 1 -- 172.21.15.90:0/2581241863 learned_addr learned my addr 172.21.15.90:0/2581241863 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:12.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.063+0000 7f31d5a84640 1 --2- 172.21.15.90:0/2581241863 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31d0109870 0x7f31d01a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.063+0000 7f31d5283640 1 --2- 172.21.15.90:0/2581241863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31d01045b0 0x7f31d019a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.063+0000 7f31d4a82640 1 -- 172.21.15.90:0/2581241863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31d01045b0 msgr2=0x7f31d019a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:12.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.063+0000 7f31d4a82640 1 --2- 172.21.15.90:0/2581241863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31d01045b0 0x7f31d019a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.063+0000 7f31d4a82640 1 -- 172.21.15.90:0/2581241863 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31d0109870 msgr2=0x7f31d01a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:12.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.063+0000 7f31d4a82640 1 --2- 172.21.15.90:0/2581241863 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31d0109870 0x7f31d01a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.063+0000 7f31d4a82640 1 -- 172.21.15.90:0/2581241863 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31d01a1d40 con 0x7f31d0106f10 2024-04-25T11:35:12.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.063+0000 7f31d5a84640 1 --2- 172.21.15.90:0/2581241863 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31d0109870 0x7f31d01a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:35:12.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.063+0000 7f31d4a82640 1 --2- 172.21.15.90:0/2581241863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31d0106f10 0x7f31d019a740 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f31cc006010 tx=0x7f31cc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:12.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.064+0000 7f31c67fc640 1 -- 172.21.15.90:0/2581241863 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc03f030 con 0x7f31d0106f10 2024-04-25T11:35:12.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.064+0000 7f31c67fc640 1 -- 172.21.15.90:0/2581241863 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31cc031910 con 0x7f31d0106f10 2024-04-25T11:35:12.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.064+0000 7f31d74e7640 1 -- 172.21.15.90:0/2581241863 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31d01a1fd0 con 0x7f31d0106f10 2024-04-25T11:35:12.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.064+0000 7f31c67fc640 1 -- 172.21.15.90:0/2581241863 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc035070 con 0x7f31d0106f10 2024-04-25T11:35:12.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.064+0000 7f31d74e7640 1 -- 172.21.15.90:0/2581241863 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31d01a23e0 con 0x7f31d0106f10 2024-04-25T11:35:12.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.065+0000 7f31d74e7640 1 -- 172.21.15.90:0/2581241863 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31d00ff6f0 con 0x7f31d0106f10 2024-04-25T11:35:12.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.066+0000 7f31c67fc640 1 -- 172.21.15.90:0/2581241863 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 22) ==== 147640+0+0 (secure 0 0 0) 0x7f31cc042030 con 0x7f31d0106f10 2024-04-25T11:35:12.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.067+0000 7f31c67fc640 1 --2- 172.21.15.90:0/2581241863 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f31bc0afc60 0x7f31bc0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.067+0000 7f31d5283640 1 --2- 172.21.15.90:0/2581241863 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f31bc0afc60 0x7f31bc0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.067+0000 7f31c67fc640 1 -- 172.21.15.90:0/2581241863 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f31cc0ff570 con 0x7f31d0106f10 2024-04-25T11:35:12.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.067+0000 7f31d5283640 1 --2- 172.21.15.90:0/2581241863 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f31bc0afc60 0x7f31bc0b2170 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f31d006a8b0 tx=0x7f31c0005f90 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:12.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.072+0000 7f31c67fc640 1 -- 172.21.15.90:0/2581241863 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f31cc0c7760 con 0x7f31d0106f10 2024-04-25T11:35:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.298+0000 7f31d74e7640 1 -- 172.21.15.90:0/2581241863 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31d002db50 con 0x7f31d0106f10 2024-04-25T11:35:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.299+0000 7f31c67fc640 1 -- 172.21.15.90:0/2581241863 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11930 (secure 0 0 0) 0x7f31cc0cba60 con 0x7f31d0106f10 2024-04-25T11:35:12.288 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:35:12.288 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:35:06.229136+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24431,24458,24499,24524,24475]},"id":1}]} 2024-04-25T11:35:12.288 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 26 2024-04-25T11:35:12.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.301+0000 7f31d74e7640 1 -- 172.21.15.90:0/2581241863 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f31bc0afc60 msgr2=0x7f31bc0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:12.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.301+0000 7f31d74e7640 1 --2- 172.21.15.90:0/2581241863 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f31bc0afc60 0x7f31bc0b2170 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f31d006a8b0 tx=0x7f31c0005f90 comp rx=0 tx=0).stop 2024-04-25T11:35:12.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.301+0000 7f31d74e7640 1 -- 172.21.15.90:0/2581241863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31d0106f10 msgr2=0x7f31d019a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:12.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.301+0000 7f31d74e7640 1 --2- 172.21.15.90:0/2581241863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31d0106f10 0x7f31d019a740 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f31cc006010 tx=0x7f31cc0042f0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.301+0000 7f31d74e7640 1 -- 172.21.15.90:0/2581241863 shutdown_connections 2024-04-25T11:35:12.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.301+0000 7f31d74e7640 1 --2- 172.21.15.90:0/2581241863 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f31bc0afc60 0x7f31bc0b2170 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.301+0000 7f31d74e7640 1 --2- 172.21.15.90:0/2581241863 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31d0109870 0x7f31d01a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.301+0000 7f31d74e7640 1 --2- 172.21.15.90:0/2581241863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31d0106f10 0x7f31d019a740 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.301+0000 7f31d74e7640 1 --2- 172.21.15.90:0/2581241863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31d01045b0 0x7f31d019a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.301+0000 7f31d74e7640 1 -- 172.21.15.90:0/2581241863 >> 172.21.15.90:0/2581241863 conn(0x7f31d00fdbd0 msgr2=0x7f31d01051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:12.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.301+0000 7f31d74e7640 1 -- 172.21.15.90:0/2581241863 shutdown_connections 2024-04-25T11:35:12.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.301+0000 7f31d74e7640 1 -- 172.21.15.90:0/2581241863 wait complete. 2024-04-25T11:35:12.302 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:35:12.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.423+0000 7ff5d6425640 1 -- 172.21.15.90:0/3003919680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5d0104280 msgr2=0x7ff5d0110650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:12.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.423+0000 7ff5d6425640 1 --2- 172.21.15.90:0/3003919680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5d0104280 0x7ff5d0110650 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff5c8009ca0 tx=0x7ff5c802a8c0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.423+0000 7ff5d6425640 1 -- 172.21.15.90:0/3003919680 shutdown_connections 2024-04-25T11:35:12.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.423+0000 7ff5d6425640 1 --2- 172.21.15.90:0/3003919680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5d0104280 0x7ff5d0110650 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.423+0000 7ff5d6425640 1 --2- 172.21.15.90:0/3003919680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5d0109fd0 0x7ff5d0103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.423+0000 7ff5d6425640 1 --2- 172.21.15.90:0/3003919680 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff5d0109b30 0x7ff5d01036d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.423+0000 7ff5d6425640 1 -- 172.21.15.90:0/3003919680 >> 172.21.15.90:0/3003919680 conn(0x7ff5d00ff210 msgr2=0x7ff5d0101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:12.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.423+0000 7ff5d6425640 1 -- 172.21.15.90:0/3003919680 shutdown_connections 2024-04-25T11:35:12.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.423+0000 7ff5d6425640 1 -- 172.21.15.90:0/3003919680 wait complete. 2024-04-25T11:35:12.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.424+0000 7ff5d6425640 1 Processor -- start 2024-04-25T11:35:12.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.424+0000 7ff5d6425640 1 -- start start 2024-04-25T11:35:12.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.424+0000 7ff5d6425640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5d0104280 0x7ff5d019a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.424+0000 7ff5d6425640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5d0109b30 0x7ff5d019ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.424+0000 7ff5d6425640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff5d0109fd0 0x7ff5d019eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.424+0000 7ff5d6425640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d0113d20 con 0x7ff5d0104280 2024-04-25T11:35:12.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.424+0000 7ff5d6425640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d0113ba0 con 0x7ff5d0109fd0 2024-04-25T11:35:12.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.424+0000 7ff5d6425640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d0113ea0 con 0x7ff5d0109b30 2024-04-25T11:35:12.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.424+0000 7ff5cffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5d0104280 0x7ff5d019a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.424+0000 7ff5cffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5d0104280 0x7ff5d019a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33692/0 (socket says 172.21.15.90:33692) 2024-04-25T11:35:12.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.424+0000 7ff5cffff640 1 -- 172.21.15.90:0/2835368353 learned_addr learned my addr 172.21.15.90:0/2835368353 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:12.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.425+0000 7ff5cf7fe640 1 --2- 172.21.15.90:0/2835368353 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5d0109b30 0x7ff5d019ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.425+0000 7ff5d49c2640 1 --2- 172.21.15.90:0/2835368353 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff5d0109fd0 0x7ff5d019eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.425+0000 7ff5cffff640 1 -- 172.21.15.90:0/2835368353 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5d0109b30 msgr2=0x7ff5d019ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:12.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.425+0000 7ff5cffff640 1 --2- 172.21.15.90:0/2835368353 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5d0109b30 0x7ff5d019ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.425+0000 7ff5cffff640 1 -- 172.21.15.90:0/2835368353 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff5d0109fd0 msgr2=0x7ff5d019eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:12.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.425+0000 7ff5cffff640 1 --2- 172.21.15.90:0/2835368353 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff5d0109fd0 0x7ff5d019eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.425+0000 7ff5cffff640 1 -- 172.21.15.90:0/2835368353 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5d019f520 con 0x7ff5d0104280 2024-04-25T11:35:12.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.425+0000 7ff5cffff640 1 --2- 172.21.15.90:0/2835368353 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5d0104280 0x7ff5d019a6c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff5c000e9c0 tx=0x7ff5c000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:12.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.425+0000 7ff5cd7fa640 1 -- 172.21.15.90:0/2835368353 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c000cd80 con 0x7ff5d0104280 2024-04-25T11:35:12.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.425+0000 7ff5cd7fa640 1 -- 172.21.15.90:0/2835368353 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5c00054d0 con 0x7ff5d0104280 2024-04-25T11:35:12.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.425+0000 7ff5d6425640 1 -- 172.21.15.90:0/2835368353 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5d019f810 con 0x7ff5d0104280 2024-04-25T11:35:12.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.425+0000 7ff5cd7fa640 1 -- 172.21.15.90:0/2835368353 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c0010430 con 0x7ff5d0104280 2024-04-25T11:35:12.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.425+0000 7ff5d6425640 1 -- 172.21.15.90:0/2835368353 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5d01a7190 con 0x7ff5d0104280 2024-04-25T11:35:12.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.426+0000 7ff5d6425640 1 -- 172.21.15.90:0/2835368353 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5d006a190 con 0x7ff5d0104280 2024-04-25T11:35:12.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.428+0000 7ff5cd7fa640 1 -- 172.21.15.90:0/2835368353 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 22) ==== 147640+0+0 (secure 0 0 0) 0x7ff5c0004fb0 con 0x7ff5d0104280 2024-04-25T11:35:12.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.429+0000 7ff5cd7fa640 1 --2- 172.21.15.90:0/2835368353 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff5b80afc60 0x7ff5b80b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.429+0000 7ff5cf7fe640 1 --2- 172.21.15.90:0/2835368353 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff5b80afc60 0x7ff5b80b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.429+0000 7ff5cd7fa640 1 -- 172.21.15.90:0/2835368353 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7ff5c00df650 con 0x7ff5d0104280 2024-04-25T11:35:12.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.429+0000 7ff5cf7fe640 1 --2- 172.21.15.90:0/2835368353 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff5b80afc60 0x7ff5b80b2170 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff5d00ff740 tx=0x7ff5c400b390 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:12.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.433+0000 7ff5cd7fa640 1 -- 172.21.15.90:0/2835368353 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff5c0053ff0 con 0x7ff5d0104280 2024-04-25T11:35:12.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.623+0000 7ff5d6425640 1 -- 172.21.15.90:0/2835368353 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff5d010bc70 con 0x7ff5d0104280 2024-04-25T11:35:12.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.625+0000 7ff5cd7fa640 1 -- 172.21.15.90:0/2835368353 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+23079 (secure 0 0 0) 0x7ff5c00a7ef0 con 0x7ff5d0104280 2024-04-25T11:35:12.614 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:35:12.614 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":80,"fsid":"58bd4e7a-02f6-11ef-bc93-c7b262605968","created":"2024-04-25T11:25:23.574550+0000","modified":"2024-04-25T11:35:08.201635+0000","last_up_change":"2024-04-25T11:32:23.852211+0000","last_in_change":"2024-04-25T11:32:04.188507+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:28:42.746649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:33:08.442925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:33:09.352886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e54d888-e67b-4951-8ae7-e355fb6b10a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6803","nonce":1201629159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6805","nonce":1201629159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6809","nonce":1201629159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6807","nonce":1201629159}]},"public_addr":"172.21.15.90:6803/1201629159","cluster_addr":"172.21.15.90:6805/1201629159","heartbeat_back_addr":"172.21.15.90:6809/1201629159","heartbeat_front_addr":"172.21.15.90:6807/1201629159","state":["exists","up"]},{"osd":1,"uuid":"91a71c78-dae9-4bc7-9f6f-9e55cfefbe28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6801","nonce":3976851933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6803","nonce":3976851933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6807","nonce":3976851933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6805","nonce":3976851933}]},"public_addr":"172.21.15.138:6801/3976851933","cluster_addr":"172.21.15.138:6803/3976851933","heartbeat_back_addr":"172.21.15.138:6807/3976851933","heartbeat_front_addr":"172.21.15.138:6805/3976851933","state":["exists","up"]},{"osd":2,"uuid":"daff6c7f-80fa-4859-9339-ef7a5e840832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6801","nonce":1505807670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6803","nonce":1505807670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6807","nonce":1505807670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6805","nonce":1505807670}]},"public_addr":"172.21.15.156:6801/1505807670","cluster_addr":"172.21.15.156:6803/1505807670","heartbeat_back_addr":"172.21.15.156:6807/1505807670","heartbeat_front_addr":"172.21.15.156:6805/1505807670","state":["exists","up"]},{"osd":3,"uuid":"33fd94ca-7465-401a-bc1e-706b6b31b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6811","nonce":2520012383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6813","nonce":2520012383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6817","nonce":2520012383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6815","nonce":2520012383}]},"public_addr":"172.21.15.90:6811/2520012383","cluster_addr":"172.21.15.90:6813/2520012383","heartbeat_back_addr":"172.21.15.90:6817/2520012383","heartbeat_front_addr":"172.21.15.90:6815/2520012383","state":["exists","up"]},{"osd":4,"uuid":"c5326212-750c-49fd-96e3-d1e62bb4b7f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6809","nonce":3752678279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6811","nonce":3752678279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6815","nonce":3752678279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6813","nonce":3752678279}]},"public_addr":"172.21.15.138:6809/3752678279","cluster_addr":"172.21.15.138:6811/3752678279","heartbeat_back_addr":"172.21.15.138:6815/3752678279","heartbeat_front_addr":"172.21.15.138:6813/3752678279","state":["exists","up"]},{"osd":5,"uuid":"d04a1156-820b-4b43-be89-e16a8910c3aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6809","nonce":3591210191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6811","nonce":3591210191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6815","nonce":3591210191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6813","nonce":3591210191}]},"public_addr":"172.21.15.156:6809/3591210191","cluster_addr":"172.21.15.156:6811/3591210191","heartbeat_back_addr":"172.21.15.156:6815/3591210191","heartbeat_front_addr":"172.21.15.156:6813/3591210191","state":["exists","up"]},{"osd":6,"uuid":"75675aa3-e978-490c-8205-13d9868a3d6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6819","nonce":2364955456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6821","nonce":2364955456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6825","nonce":2364955456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6823","nonce":2364955456}]},"public_addr":"172.21.15.90:6819/2364955456","cluster_addr":"172.21.15.90:6821/2364955456","heartbeat_back_addr":"172.21.15.90:6825/2364955456","heartbeat_front_addr":"172.21.15.90:6823/2364955456","state":["exists","up"]},{"osd":7,"uuid":"1a95fdf5-f548-4271-bbcd-e330f7955eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6817","nonce":4012476387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6819","nonce":4012476387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6823","nonce":4012476387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6821","nonce":4012476387}]},"public_addr":"172.21.15.138:6817/4012476387","cluster_addr":"172.21.15.138:6819/4012476387","heartbeat_back_addr":"172.21.15.138:6823/4012476387","heartbeat_front_addr":"172.21.15.138:6821/4012476387","state":["exists","up"]},{"osd":8,"uuid":"605e10b5-5806-4c1f-8e28-a7994d34ddce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6817","nonce":101092224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6819","nonce":101092224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6823","nonce":101092224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6821","nonce":101092224}]},"public_addr":"172.21.15.156:6817/101092224","cluster_addr":"172.21.15.156:6819/101092224","heartbeat_back_addr":"172.21.15.156:6823/101092224","heartbeat_front_addr":"172.21.15.156:6821/101092224","state":["exists","up"]},{"osd":9,"uuid":"d70fdf5e-9d0f-483d-a7d8-505b79b8a418","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6827","nonce":1171265966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6829","nonce":1171265966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6833","nonce":1171265966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6831","nonce":1171265966}]},"public_addr":"172.21.15.90:6827/1171265966","cluster_addr":"172.21.15.90:6829/1171265966","heartbeat_back_addr":"172.21.15.90:6833/1171265966","heartbeat_front_addr":"172.21.15.90:6831/1171265966","state":["exists","up"]},{"osd":10,"uuid":"1ec86c7d-c3a4-432d-9664-8e40836e1e20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6825","nonce":4264666232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6827","nonce":4264666232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6831","nonce":4264666232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6829","nonce":4264666232}]},"public_addr":"172.21.15.138:6825/4264666232","cluster_addr":"172.21.15.138:6827/4264666232","heartbeat_back_addr":"172.21.15.138:6831/4264666232","heartbeat_front_addr":"172.21.15.138:6829/4264666232","state":["exists","up"]},{"osd":11,"uuid":"51cbe729-292e-4d7b-b8e9-e1e2821ffe76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6825","nonce":1224251744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6827","nonce":1224251744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6831","nonce":1224251744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6829","nonce":1224251744}]},"public_addr":"172.21.15.156:6825/1224251744","cluster_addr":"172.21.15.156:6827/1224251744","heartbeat_back_addr":"172.21.15.156:6831/1224251744","heartbeat_front_addr":"172.21.15.156:6829/1224251744","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:27:50.794226+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:15.030278+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:38.518163+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:03.107351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:27.500707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:51.463545+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:16.015076+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:41.182071+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:05.362309+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:31.048936+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:56.488509+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:32:22.129178+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":7}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/813599224":"2024-04-26T11:35:08.201598+0000","172.21.15.90:6800/219449516":"2024-04-26T11:35:08.201598+0000","172.21.15.90:0/478282854":"2024-04-26T11:35:08.201598+0000","172.21.15.90:0/2202319376":"2024-04-26T11:35:08.201598+0000","172.21.15.90:0/2381633953":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6800/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/2190846900":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/888963322":"2024-04-26T11:35:08.201598+0000","172.21.15.90:6801/219449516":"2024-04-26T11:35:08.201598+0000","172.21.15.90:0/1707792941":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6800/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6801/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/6061052":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1849633392":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6801/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/2277490702":"2024-04-26T11:26:16.655278+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:35:12.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.627+0000 7ff5d6425640 1 -- 172.21.15.90:0/2835368353 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff5b80afc60 msgr2=0x7ff5b80b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:12.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.627+0000 7ff5d6425640 1 --2- 172.21.15.90:0/2835368353 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff5b80afc60 0x7ff5b80b2170 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff5d00ff740 tx=0x7ff5c400b390 comp rx=0 tx=0).stop 2024-04-25T11:35:12.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.627+0000 7ff5d6425640 1 -- 172.21.15.90:0/2835368353 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5d0104280 msgr2=0x7ff5d019a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:12.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.627+0000 7ff5d6425640 1 --2- 172.21.15.90:0/2835368353 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5d0104280 0x7ff5d019a6c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff5c000e9c0 tx=0x7ff5c000ee80 comp rx=0 tx=0).stop 2024-04-25T11:35:12.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.627+0000 7ff5d6425640 1 -- 172.21.15.90:0/2835368353 shutdown_connections 2024-04-25T11:35:12.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.627+0000 7ff5d6425640 1 --2- 172.21.15.90:0/2835368353 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff5b80afc60 0x7ff5b80b2170 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.627+0000 7ff5d6425640 1 --2- 172.21.15.90:0/2835368353 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff5d0109fd0 0x7ff5d019eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.627+0000 7ff5d6425640 1 --2- 172.21.15.90:0/2835368353 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5d0109b30 0x7ff5d019ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.627+0000 7ff5d6425640 1 --2- 172.21.15.90:0/2835368353 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5d0104280 0x7ff5d019a6c0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.627+0000 7ff5d6425640 1 -- 172.21.15.90:0/2835368353 >> 172.21.15.90:0/2835368353 conn(0x7ff5d00ff210 msgr2=0x7ff5d01064a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:12.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.627+0000 7ff5d6425640 1 -- 172.21.15.90:0/2835368353 shutdown_connections 2024-04-25T11:35:12.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.627+0000 7ff5d6425640 1 -- 172.21.15.90:0/2835368353 wait complete. 2024-04-25T11:35:12.635 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:12.635 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:12.635 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.635 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.635 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.635 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.635 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: Adjusting osd_memory_target on smithi138 to 176.2M 2024-04-25T11:35:12.635 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: Unable to set osd_memory_target on smithi138 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-04-25T11:35:12.635 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:12.636 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:12.636 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.636 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.637 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.637 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.637 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: Adjusting osd_memory_target on smithi156 to 176.2M 2024-04-25T11:35:12.637 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: Unable to set osd_memory_target on smithi156 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-25T11:35:12.637 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3184600101' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:35:12.637 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:12.637 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:12.637 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.637 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.637 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.637 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.637 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: Adjusting osd_memory_target on smithi090 to 176.2M 2024-04-25T11:35:12.637 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: Unable to set osd_memory_target on smithi090 to 184765952: error parsing value: Value '184765952' is below minimum 939524096 2024-04-25T11:35:12.638 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:35:12.638 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:35:12.638 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: Updating smithi090:/etc/ceph/ceph.conf 2024-04-25T11:35:12.638 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: Updating smithi138:/etc/ceph/ceph.conf 2024-04-25T11:35:12.638 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-25T11:35:12.638 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: Updating smithi090:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T11:35:12.638 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: Updating smithi138:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T11:35:12.638 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: Updating smithi156:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T11:35:12.638 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1936851046' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:35:12.638 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:12 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2581241863' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:35:12.640 DEBUG:tasks.fwd_scrub:Yielding 2024-04-25T11:35:12.640 INFO:teuthology.run_tasks:Running task exec... 2024-04-25T11:35:12.651 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T11:35:12.651 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T11:35:12.651 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/log_level debug' 2024-04-25T11:35:12.652 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-25T11:35:12.652 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T11:35:12.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:35:12.652 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T11:35:12.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.783+0000 7f5dc7eb1640 1 -- 172.21.15.90:0/4181684437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5dc01042e0 msgr2=0x7f5dc01106b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:12.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.783+0000 7f5dc7eb1640 1 --2- 172.21.15.90:0/4181684437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5dc01042e0 0x7f5dc01106b0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f5db8009ca0 tx=0x7f5db802a8c0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.783+0000 7f5dc7eb1640 1 -- 172.21.15.90:0/4181684437 shutdown_connections 2024-04-25T11:35:12.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.783+0000 7f5dc7eb1640 1 --2- 172.21.15.90:0/4181684437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5dc01042e0 0x7f5dc01106b0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.783+0000 7f5dc7eb1640 1 --2- 172.21.15.90:0/4181684437 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5dc010a030 0x7f5dc0103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.783+0000 7f5dc7eb1640 1 --2- 172.21.15.90:0/4181684437 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5dc0109b90 0x7f5dc0103730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.783+0000 7f5dc7eb1640 1 -- 172.21.15.90:0/4181684437 >> 172.21.15.90:0/4181684437 conn(0x7f5dc00ff360 msgr2=0x7f5dc0101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.783+0000 7f5dc7eb1640 1 -- 172.21.15.90:0/4181684437 shutdown_connections 2024-04-25T11:35:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.783+0000 7f5dc7eb1640 1 -- 172.21.15.90:0/4181684437 wait complete. 2024-04-25T11:35:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.784+0000 7f5dc7eb1640 1 Processor -- start 2024-04-25T11:35:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.784+0000 7f5dc7eb1640 1 -- start start 2024-04-25T11:35:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.784+0000 7f5dc7eb1640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5dc01042e0 0x7f5dc019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.784+0000 7f5dc7eb1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5dc0109b90 0x7f5dc019ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.784+0000 7f5dc7eb1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5dc010a030 0x7f5dc019ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.784+0000 7f5dc7eb1640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc0113da0 con 0x7f5dc0109b90 2024-04-25T11:35:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.784+0000 7f5dc7eb1640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc0113c20 con 0x7f5dc01042e0 2024-04-25T11:35:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.784+0000 7f5dc7eb1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc0113f20 con 0x7f5dc010a030 2024-04-25T11:35:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.784+0000 7f5dc544c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5dc0109b90 0x7f5dc019ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.784+0000 7f5dc544c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5dc0109b90 0x7f5dc019ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33722/0 (socket says 172.21.15.90:33722) 2024-04-25T11:35:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.784+0000 7f5dc544c640 1 -- 172.21.15.90:0/512053787 learned_addr learned my addr 172.21.15.90:0/512053787 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.784+0000 7f5dc644e640 1 --2- 172.21.15.90:0/512053787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5dc010a030 0x7f5dc019ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.784+0000 7f5dc5c4d640 1 --2- 172.21.15.90:0/512053787 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5dc01042e0 0x7f5dc019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.784+0000 7f5dc544c640 1 -- 172.21.15.90:0/512053787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5dc010a030 msgr2=0x7f5dc019ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.784+0000 7f5dc544c640 1 --2- 172.21.15.90:0/512053787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5dc010a030 0x7f5dc019ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.784+0000 7f5dc544c640 1 -- 172.21.15.90:0/512053787 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5dc01042e0 msgr2=0x7f5dc019a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.784+0000 7f5dc544c640 1 --2- 172.21.15.90:0/512053787 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5dc01042e0 0x7f5dc019a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.784+0000 7f5dc544c640 1 -- 172.21.15.90:0/512053787 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dc019f580 con 0x7f5dc0109b90 2024-04-25T11:35:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.785+0000 7f5dc544c640 1 --2- 172.21.15.90:0/512053787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5dc0109b90 0x7f5dc019ac80 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f5db0005d00 tx=0x7f5db000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.785+0000 7f5db6ffd640 1 -- 172.21.15.90:0/512053787 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db0017070 con 0x7f5dc0109b90 2024-04-25T11:35:12.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.785+0000 7f5db6ffd640 1 -- 172.21.15.90:0/512053787 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5db0009c30 con 0x7f5dc0109b90 2024-04-25T11:35:12.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.785+0000 7f5dc7eb1640 1 -- 172.21.15.90:0/512053787 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dc019f870 con 0x7f5dc0109b90 2024-04-25T11:35:12.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.785+0000 7f5db6ffd640 1 -- 172.21.15.90:0/512053787 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db000f620 con 0x7f5dc0109b90 2024-04-25T11:35:12.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.785+0000 7f5dc7eb1640 1 -- 172.21.15.90:0/512053787 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dc019fd70 con 0x7f5dc0109b90 2024-04-25T11:35:12.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.786+0000 7f5dc7eb1640 1 -- 172.21.15.90:0/512053787 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dc006a110 con 0x7f5dc0109b90 2024-04-25T11:35:12.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.787+0000 7f5db6ffd640 1 -- 172.21.15.90:0/512053787 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 23) ==== 147696+0+0 (secure 0 0 0) 0x7f5db001a070 con 0x7f5dc0109b90 2024-04-25T11:35:12.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.788+0000 7f5db6ffd640 1 --2- 172.21.15.90:0/512053787 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5dac0afce0 0x7f5dac0b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.788+0000 7f5dc5c4d640 1 --2- 172.21.15.90:0/512053787 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5dac0afce0 0x7f5dac0b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.788+0000 7f5db6ffd640 1 -- 172.21.15.90:0/512053787 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f5db00deb10 con 0x7f5dc0109b90 2024-04-25T11:35:12.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.789+0000 7f5dc5c4d640 1 --2- 172.21.15.90:0/512053787 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5dac0afce0 0x7f5dac0b21f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0104870 tx=0x7f5dbc005f90 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:12.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.792+0000 7f5db6ffd640 1 -- 172.21.15.90:0/512053787 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5dc019fd70 con 0x7f5dc0109b90 2024-04-25T11:35:12.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.795+0000 7f291c984640 1 -- 172.21.15.90:0/3291963989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2918102b10 msgr2=0x7f2918102f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:12.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.795+0000 7f291c984640 1 --2- 172.21.15.90:0/3291963989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2918102b10 0x7f2918102f20 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f290c009900 tx=0x7f290c02a410 comp rx=0 tx=0).stop 2024-04-25T11:35:12.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.795+0000 7f291c984640 1 -- 172.21.15.90:0/3291963989 shutdown_connections 2024-04-25T11:35:12.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.795+0000 7f291c984640 1 --2- 172.21.15.90:0/3291963989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f291810d3a0 0x7f291810f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.795+0000 7f291c984640 1 --2- 172.21.15.90:0/3291963989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2918102b10 0x7f2918102f20 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.795+0000 7f291c984640 1 --2- 172.21.15.90:0/3291963989 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2918108a40 0x7f29181025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.795+0000 7f291c984640 1 -- 172.21.15.90:0/3291963989 >> 172.21.15.90:0/3291963989 conn(0x7f2918078180 msgr2=0x7f29180785c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:12.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.796+0000 7f291c984640 1 -- 172.21.15.90:0/3291963989 shutdown_connections 2024-04-25T11:35:12.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.796+0000 7f291c984640 1 -- 172.21.15.90:0/3291963989 wait complete. 2024-04-25T11:35:12.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.796+0000 7f291c984640 1 Processor -- start 2024-04-25T11:35:12.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.796+0000 7f291c984640 1 -- start start 2024-04-25T11:35:12.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.797+0000 7f291c984640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2918102b10 0x7f29181993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.797+0000 7f291c984640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2918108a40 0x7f2918199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.797+0000 7f291c984640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f291810d3a0 0x7f2918199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.797+0000 7f291c984640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2918112c50 con 0x7f2918108a40 2024-04-25T11:35:12.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.797+0000 7f291c984640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2918112ad0 con 0x7f291810d3a0 2024-04-25T11:35:12.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.797+0000 7f291c984640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2918112dd0 con 0x7f2918102b10 2024-04-25T11:35:12.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.797+0000 7f2915d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2918108a40 0x7f2918199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.797+0000 7f2915d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2918108a40 0x7f2918199900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33752/0 (socket says 172.21.15.90:33752) 2024-04-25T11:35:12.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.797+0000 7f2915d9b640 1 -- 172.21.15.90:0/1160436878 learned_addr learned my addr 172.21.15.90:0/1160436878 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:12.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.797+0000 7f291659c640 1 --2- 172.21.15.90:0/1160436878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2918102b10 0x7f29181993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.797+0000 7f2916d9d640 1 --2- 172.21.15.90:0/1160436878 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f291810d3a0 0x7f2918199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.797+0000 7f2915d9b640 1 -- 172.21.15.90:0/1160436878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2918102b10 msgr2=0x7f29181993c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:12.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.797+0000 7f2915d9b640 1 --2- 172.21.15.90:0/1160436878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2918102b10 0x7f29181993c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.797+0000 7f2915d9b640 1 -- 172.21.15.90:0/1160436878 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f291810d3a0 msgr2=0x7f2918199e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:12.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.797+0000 7f2915d9b640 1 --2- 172.21.15.90:0/1160436878 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f291810d3a0 0x7f2918199e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.797+0000 7f2915d9b640 1 -- 172.21.15.90:0/1160436878 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29181a8a70 con 0x7f2918108a40 2024-04-25T11:35:12.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.797+0000 7f2915d9b640 1 --2- 172.21.15.90:0/1160436878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2918108a40 0x7f2918199900 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f290c005ab0 tx=0x7f290c003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:12.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.797+0000 7f29077fe640 1 -- 172.21.15.90:0/1160436878 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f290c03f070 con 0x7f2918108a40 2024-04-25T11:35:12.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.798+0000 7f29077fe640 1 -- 172.21.15.90:0/1160436878 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f290c031b50 con 0x7f2918108a40 2024-04-25T11:35:12.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.798+0000 7f291c984640 1 -- 172.21.15.90:0/1160436878 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29181a8d00 con 0x7f2918108a40 2024-04-25T11:35:12.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.798+0000 7f29077fe640 1 -- 172.21.15.90:0/1160436878 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f290c035030 con 0x7f2918108a40 2024-04-25T11:35:12.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.798+0000 7f291c984640 1 -- 172.21.15.90:0/1160436878 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29181a91a0 con 0x7f2918108a40 2024-04-25T11:35:12.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.800+0000 7f2923839740 1 Processor -- start 2024-04-25T11:35:12.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.800+0000 7f2923839740 1 -- start start 2024-04-25T11:35:12.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.800+0000 7f29077fe640 1 -- 172.21.15.90:0/1160436878 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 23) ==== 147696+0+0 (secure 0 0 0) 0x7f290c042070 con 0x7f2918108a40 2024-04-25T11:35:12.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.800+0000 7f2923839740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73aae8410 0x55b73ae88620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.800+0000 7f2923839740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aabf550 0x55b73ae88b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.800+0000 7f2923839740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b73aac7730 0x55b73ae890a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.800+0000 7f2923839740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b73adeeb00 con 0x55b73aabf550 2024-04-25T11:35:12.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.800+0000 7f2923839740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55b73ad8a920 con 0x55b73aac7730 2024-04-25T11:35:12.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.800+0000 7f2923839740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b73adeaac0 con 0x55b73aae8410 2024-04-25T11:35:12.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.801+0000 7f291659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aabf550 0x55b73ae88b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.801+0000 7f2916d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73aae8410 0x55b73ae88620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.801+0000 7f291659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aabf550 0x55b73ae88b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33768/0 (socket says 172.21.15.90:33768) 2024-04-25T11:35:12.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.801+0000 7f2916d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73aae8410 0x55b73ae88620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:48124/0 (socket says 172.21.15.90:48124) 2024-04-25T11:35:12.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.801+0000 7f291659c640 1 -- 172.21.15.90:0/4074255239 learned_addr learned my addr 172.21.15.90:0/4074255239 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:12.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.801+0000 7f2915d9b640 1 --2- 172.21.15.90:0/4074255239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b73aac7730 0x55b73ae890a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.801+0000 7f291659c640 1 -- 172.21.15.90:0/4074255239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73aae8410 msgr2=0x55b73ae88620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:12.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.801+0000 7f291659c640 1 --2- 172.21.15.90:0/4074255239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73aae8410 0x55b73ae88620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.801+0000 7f291659c640 1 -- 172.21.15.90:0/4074255239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b73aac7730 msgr2=0x55b73ae890a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.801+0000 7f291659c640 1 --2- 172.21.15.90:0/4074255239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b73aac7730 0x55b73ae890a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.801+0000 7f291659c640 1 -- 172.21.15.90:0/4074255239 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b73abc87e0 con 0x55b73aabf550 2024-04-25T11:35:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.801+0000 7f2916d9d640 1 --2- 172.21.15.90:0/4074255239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73aae8410 0x55b73ae88620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:35:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.801+0000 7f2915d9b640 1 --2- 172.21.15.90:0/4074255239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b73aac7730 0x55b73ae890a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:35:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.801+0000 7f29077fe640 1 --2- 172.21.15.90:0/1160436878 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f29000afcf0 0x7f29000b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.801+0000 7f291659c640 1 --2- 172.21.15.90:0/4074255239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aabf550 0x55b73ae88b60 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f290800bc90 tx=0x7f29080107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=802caa5fe81a6d1f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.801+0000 7f2916d9d640 1 --2- 172.21.15.90:0/1160436878 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f29000afcf0 0x7f29000b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.801+0000 7f29077fe640 1 -- 172.21.15.90:0/1160436878 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f290c0ff860 con 0x7f2918108a40 2024-04-25T11:35:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.802+0000 7f2904ff9640 1 -- 172.21.15.90:0/4074255239 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2908005220 con 0x55b73aabf550 2024-04-25T11:35:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.802+0000 7f2904ff9640 1 -- 172.21.15.90:0/4074255239 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f290800be40 con 0x55b73aabf550 2024-04-25T11:35:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.802+0000 7f2904ff9640 1 -- 172.21.15.90:0/4074255239 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29080117b0 con 0x55b73aabf550 2024-04-25T11:35:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.802+0000 7f2923839740 1 -- 172.21.15.90:0/4074255239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aabf550 msgr2=0x55b73ae88b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.802+0000 7f2923839740 1 --2- 172.21.15.90:0/4074255239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aabf550 0x55b73ae88b60 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f290800bc90 tx=0x7f29080107a0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.802+0000 7f2916d9d640 1 --2- 172.21.15.90:0/1160436878 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f29000afcf0 0x7f29000b2200 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x55b73ae85790 tx=0x7f2910009500 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.803+0000 7f2923839740 1 -- 172.21.15.90:0/4074255239 shutdown_connections 2024-04-25T11:35:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.803+0000 7f2923839740 1 --2- 172.21.15.90:0/4074255239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b73aac7730 0x55b73ae890a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.803+0000 7f2923839740 1 --2- 172.21.15.90:0/4074255239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aabf550 0x55b73ae88b60 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.803+0000 7f2923839740 1 --2- 172.21.15.90:0/4074255239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73aae8410 0x55b73ae88620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.803+0000 7f2923839740 1 -- 172.21.15.90:0/4074255239 >> 172.21.15.90:0/4074255239 conn(0x55b73aadc560 msgr2=0x55b73ae84dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.803+0000 7f2923839740 1 -- 172.21.15.90:0/4074255239 shutdown_connections 2024-04-25T11:35:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.803+0000 7f2923839740 1 -- 172.21.15.90:0/4074255239 wait complete. 2024-04-25T11:35:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.803+0000 7f2923839740 1 -- start start 2024-04-25T11:35:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.803+0000 7f2923839740 1 Processor -- start 2024-04-25T11:35:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.803+0000 7f2923839740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aabf550 0x55b73ae92c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.803+0000 7f2923839740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b73aac7730 0x55b73ae93170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.803+0000 7f2923839740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73aae8410 0x55b73ae9ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.803+0000 7f2923839740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b73ae95970 con 0x55b73aabf550 2024-04-25T11:35:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.803+0000 7f2923839740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55b73ad885d0 con 0x55b73aac7730 2024-04-25T11:35:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.803+0000 7f2923839740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b73ae95af0 con 0x55b73aae8410 2024-04-25T11:35:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.803+0000 7f2923839740 10 client.0 _ll_register_callbacks cb 0x55b73ad76080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:35:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.804+0000 7f291659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aabf550 0x55b73ae92c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.804+0000 7f291659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aabf550 0x55b73ae92c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33772/0 (socket says 172.21.15.90:33772) 2024-04-25T11:35:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.804+0000 7f291659c640 1 -- 172.21.15.90:0/3756597731 learned_addr learned my addr 172.21.15.90:0/3756597731 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.804+0000 7f2915d9b640 1 --2- 172.21.15.90:0/3756597731 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b73aac7730 0x55b73ae93170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.804+0000 7f2916d9d640 1 --2- 172.21.15.90:0/3756597731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73aae8410 0x55b73ae9ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.804+0000 7f291659c640 1 -- 172.21.15.90:0/3756597731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73aae8410 msgr2=0x55b73ae9ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.804+0000 7f291659c640 1 --2- 172.21.15.90:0/3756597731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73aae8410 0x55b73ae9ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.804+0000 7f291659c640 1 -- 172.21.15.90:0/3756597731 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b73aac7730 msgr2=0x55b73ae93170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.804+0000 7f291659c640 1 --2- 172.21.15.90:0/3756597731 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b73aac7730 0x55b73ae93170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.804+0000 7f291659c640 1 -- 172.21.15.90:0/3756597731 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b73ac28f70 con 0x55b73aabf550 2024-04-25T11:35:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.804+0000 7f291659c640 1 --2- 172.21.15.90:0/3756597731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aabf550 0x55b73ae92c30 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f2908000c00 tx=0x7f29080116e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e0e59d2083008e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.804+0000 7f28fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:35:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.804+0000 7f28fdffb640 1 -- 172.21.15.90:0/3756597731 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2908011ce0 con 0x55b73aabf550 2024-04-25T11:35:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.804+0000 7f28fdffb640 1 -- 172.21.15.90:0/3756597731 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f2908004020 con 0x55b73aabf550 2024-04-25T11:35:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.804+0000 7f2923839740 1 -- 172.21.15.90:0/3756597731 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b73adfc710 con 0x55b73aabf550 2024-04-25T11:35:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.804+0000 7f28fdffb640 1 -- 172.21.15.90:0/3756597731 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2908004220 con 0x55b73aabf550 2024-04-25T11:35:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.804+0000 7f2923839740 10 client.15540 fetch_fsmap learned FSMap version 26 2024-04-25T11:35:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.804+0000 7f2923839740 1 -- 172.21.15.90:0/3756597731 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b73aac0170 con 0x55b73aabf550 2024-04-25T11:35:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.805+0000 7f28fdffb640 1 -- 172.21.15.90:0/3756597731 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55b73adfc710 con 0x55b73aabf550 2024-04-25T11:35:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.805+0000 7f28fdffb640 1 -- 172.21.15.90:0/3756597731 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f2908025460 con 0x55b73aabf550 2024-04-25T11:35:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.805+0000 7f2923839740 10 client.15540 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:35:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.805+0000 7f2923839740 10 client.15540 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:35:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.805+0000 7f2923839740 20 client.15540 populate_metadata read hostname 'smithi090' 2024-04-25T11:35:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.805+0000 7f2923839740 1 --2- 172.21.15.90:0/3756597731 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b73aaa71f0 0x55b73aea4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.805+0000 7f2923839740 4 client.15540 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:35:12.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.805+0000 7f2923839740 1 -- 172.21.15.90:0/3756597731 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b73adf6580 con 0x55b73aaa71f0 2024-04-25T11:35:12.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.805+0000 7f2915d9b640 1 --2- 172.21.15.90:0/3756597731 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b73aaa71f0 0x55b73aea4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.805+0000 7f2915d9b640 1 --2- 172.21.15.90:0/3756597731 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b73aaa71f0 0x55b73aea4ec0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12ea8b51fd1790c2 server_cookie=adaa83b8f4b47dce in_seq=0 out_seq=0 2024-04-25T11:35:12.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.805+0000 7f28fdffb640 10 client.15540 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:12.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.806+0000 7f28fdffb640 1 -- 172.21.15.90:0/3756597731 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b73adf6580 con 0x55b73aaa71f0 2024-04-25T11:35:12.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.806+0000 7f28fdffb640 10 client.15540 handle_command_reply: tid=0 2024-04-25T11:35:12.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.806+0000 7f2923839740 1 client.15540 shutdown 2024-04-25T11:35:12.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.806+0000 7f2923839740 1 -- 172.21.15.90:0/3756597731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aabf550 msgr2=0x55b73ae92c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:12.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.806+0000 7f2923839740 1 --2- 172.21.15.90:0/3756597731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aabf550 0x55b73ae92c30 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f2908000c00 tx=0x7f29080116e0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.806+0000 7f2923839740 1 -- 172.21.15.90:0/3756597731 shutdown_connections 2024-04-25T11:35:12.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.806+0000 7f2923839740 1 --2- 172.21.15.90:0/3756597731 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b73aaa71f0 0x55b73aea4ec0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.806+0000 7f2923839740 1 --2- 172.21.15.90:0/3756597731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73aae8410 0x55b73ae9ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.806+0000 7f2923839740 1 --2- 172.21.15.90:0/3756597731 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b73aac7730 0x55b73ae93170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.806+0000 7f28fdffb640 0 client.15540 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:12.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.806+0000 7f2923839740 1 --2- 172.21.15.90:0/3756597731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aabf550 0x55b73ae92c30 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.806+0000 7f2923839740 1 -- 172.21.15.90:0/3756597731 >> 172.21.15.90:0/3756597731 conn(0x55b73aadc560 msgr2=0x55b73ae88450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:12.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.806+0000 7f2923839740 1 -- 172.21.15.90:0/3756597731 shutdown_connections 2024-04-25T11:35:12.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.806+0000 7f2923839740 1 -- 172.21.15.90:0/3756597731 wait complete. 2024-04-25T11:35:12.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.806+0000 7f2923839740 20 client.15540 trim_cache size 0 max 16384 2024-04-25T11:35:12.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:12.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:12.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: Adjusting osd_memory_target on smithi138 to 176.2M 2024-04-25T11:35:12.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: Unable to set osd_memory_target on smithi138 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-04-25T11:35:12.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:12.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:12.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: Adjusting osd_memory_target on smithi156 to 176.2M 2024-04-25T11:35:12.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: Unable to set osd_memory_target on smithi156 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-25T11:35:12.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3184600101' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:35:12.805 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:12.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:12.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: Adjusting osd_memory_target on smithi090 to 176.2M 2024-04-25T11:35:12.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: Unable to set osd_memory_target on smithi090 to 184765952: error parsing value: Value '184765952' is below minimum 939524096 2024-04-25T11:35:12.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:35:12.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:35:12.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: Updating smithi090:/etc/ceph/ceph.conf 2024-04-25T11:35:12.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: Updating smithi138:/etc/ceph/ceph.conf 2024-04-25T11:35:12.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-25T11:35:12.806 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: Updating smithi090:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T11:35:12.807 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: Updating smithi138:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T11:35:12.807 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: Updating smithi156:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T11:35:12.807 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1936851046' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:35:12.807 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:12 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2581241863' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:35:12.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.827+0000 7f2923839740 1 Processor -- start 2024-04-25T11:35:12.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.827+0000 7f2923839740 1 -- start start 2024-04-25T11:35:12.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.827+0000 7f2923839740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b73aabf550 0x55b73ae8d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.827+0000 7f2923839740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73aac7730 0x55b73ae83b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.827+0000 7f2923839740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aae8410 0x55b73aea7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.827+0000 7f2923839740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f29181b1cb0 con 0x55b73aae8410 2024-04-25T11:35:12.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.827+0000 7f2923839740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f29181b37f0 con 0x55b73aabf550 2024-04-25T11:35:12.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.827+0000 7f2923839740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f29181b2380 con 0x55b73aac7730 2024-04-25T11:35:12.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.827+0000 7f2916d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aae8410 0x55b73aea7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.827+0000 7f2916d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aae8410 0x55b73aea7ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33774/0 (socket says 172.21.15.90:33774) 2024-04-25T11:35:12.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.827+0000 7f2916d9d640 1 -- 172.21.15.90:0/3436292764 learned_addr learned my addr 172.21.15.90:0/3436292764 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:12.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.827+0000 7f291659c640 1 --2- 172.21.15.90:0/3436292764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b73aabf550 0x55b73ae8d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.827+0000 7f2915d9b640 1 --2- 172.21.15.90:0/3436292764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73aac7730 0x55b73ae83b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.828+0000 7f2916d9d640 1 -- 172.21.15.90:0/3436292764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73aac7730 msgr2=0x55b73ae83b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:12.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.828+0000 7f2916d9d640 1 --2- 172.21.15.90:0/3436292764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73aac7730 0x55b73ae83b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.828+0000 7f2916d9d640 1 -- 172.21.15.90:0/3436292764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b73aabf550 msgr2=0x55b73ae8d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:12.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.828+0000 7f2916d9d640 1 --2- 172.21.15.90:0/3436292764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b73aabf550 0x55b73ae8d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.828+0000 7f2916d9d640 1 -- 172.21.15.90:0/3436292764 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b73ae95850 con 0x55b73aae8410 2024-04-25T11:35:12.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.828+0000 7f291659c640 1 --2- 172.21.15.90:0/3436292764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b73aabf550 0x55b73ae8d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:35:12.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.828+0000 7f2916d9d640 1 --2- 172.21.15.90:0/3436292764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aae8410 0x55b73aea7ee0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f29100108e0 tx=0x7f2910016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64ebb39c490c6edb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:12.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.828+0000 7f28fffff640 1 -- 172.21.15.90:0/3436292764 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f291002d070 con 0x55b73aae8410 2024-04-25T11:35:12.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.828+0000 7f28fffff640 1 -- 172.21.15.90:0/3436292764 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f291001fde0 con 0x55b73aae8410 2024-04-25T11:35:12.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.828+0000 7f28fffff640 1 -- 172.21.15.90:0/3436292764 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2910028ae0 con 0x55b73aae8410 2024-04-25T11:35:12.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.829+0000 7f2923839740 1 -- 172.21.15.90:0/3436292764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aae8410 msgr2=0x55b73aea7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:12.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.829+0000 7f2923839740 1 --2- 172.21.15.90:0/3436292764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aae8410 0x55b73aea7ee0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f29100108e0 tx=0x7f2910016000 comp rx=0 tx=0).stop 2024-04-25T11:35:12.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.829+0000 7f2923839740 1 -- 172.21.15.90:0/3436292764 shutdown_connections 2024-04-25T11:35:12.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.829+0000 7f2923839740 1 --2- 172.21.15.90:0/3436292764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aae8410 0x55b73aea7ee0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.829+0000 7f2923839740 1 --2- 172.21.15.90:0/3436292764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73aac7730 0x55b73ae83b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.829+0000 7f2923839740 1 --2- 172.21.15.90:0/3436292764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b73aabf550 0x55b73ae8d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.829+0000 7f2923839740 1 -- 172.21.15.90:0/3436292764 >> 172.21.15.90:0/3436292764 conn(0x55b73aadc560 msgr2=0x55b73ae88de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:12.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.829+0000 7f2923839740 1 -- 172.21.15.90:0/3436292764 shutdown_connections 2024-04-25T11:35:12.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.829+0000 7f2923839740 1 -- 172.21.15.90:0/3436292764 wait complete. 2024-04-25T11:35:12.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.829+0000 7f2923839740 1 -- start start 2024-04-25T11:35:12.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.829+0000 7f2923839740 1 Processor -- start 2024-04-25T11:35:12.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.830+0000 7f2923839740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aabf550 0x55b73aea7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.830+0000 7f2923839740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b73aac7730 0x55b73aea8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.830+0000 7f2923839740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73aae8410 0x55b73aea52d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.830+0000 7f2923839740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f29181b3120 con 0x55b73aabf550 2024-04-25T11:35:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.830+0000 7f2923839740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f29181b2a50 con 0x55b73aac7730 2024-04-25T11:35:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.830+0000 7f2923839740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f29181afc90 con 0x55b73aae8410 2024-04-25T11:35:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.830+0000 7f2923839740 10 client.0 _ll_register_callbacks cb 0x7f2908005720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:35:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.830+0000 7f291659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aabf550 0x55b73aea7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.830+0000 7f291659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aabf550 0x55b73aea7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33782/0 (socket says 172.21.15.90:33782) 2024-04-25T11:35:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.830+0000 7f291659c640 1 -- 172.21.15.90:0/625844131 learned_addr learned my addr 172.21.15.90:0/625844131 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.830+0000 7f2915d9b640 1 --2- 172.21.15.90:0/625844131 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b73aac7730 0x55b73aea8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.830+0000 7f2916d9d640 1 --2- 172.21.15.90:0/625844131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73aae8410 0x55b73aea52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.830+0000 7f291659c640 1 -- 172.21.15.90:0/625844131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73aae8410 msgr2=0x55b73aea52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.830+0000 7f291659c640 1 --2- 172.21.15.90:0/625844131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73aae8410 0x55b73aea52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.830+0000 7f291659c640 1 -- 172.21.15.90:0/625844131 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b73aac7730 msgr2=0x55b73aea8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:12.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.830+0000 7f291659c640 1 --2- 172.21.15.90:0/625844131 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b73aac7730 0x55b73aea8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.830+0000 7f291659c640 1 -- 172.21.15.90:0/625844131 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b73aea59c0 con 0x55b73aabf550 2024-04-25T11:35:12.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.830+0000 7f2916d9d640 1 --2- 172.21.15.90:0/625844131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73aae8410 0x55b73aea52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:35:12.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.830+0000 7f291659c640 1 --2- 172.21.15.90:0/625844131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aabf550 0x55b73aea7e50 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f2908005d00 tx=0x7f2908025770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14b18d2a568fe4aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:12.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.830+0000 7f28fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:35:12.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.831+0000 7f28fdffb640 1 -- 172.21.15.90:0/625844131 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2908025ca0 con 0x55b73aabf550 2024-04-25T11:35:12.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.831+0000 7f28fdffb640 1 -- 172.21.15.90:0/625844131 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f2908011ae0 con 0x55b73aabf550 2024-04-25T11:35:12.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.831+0000 7f2923839740 1 -- 172.21.15.90:0/625844131 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f291001fde0 con 0x55b73aabf550 2024-04-25T11:35:12.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.831+0000 7f28fdffb640 1 -- 172.21.15.90:0/625844131 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2908043050 con 0x55b73aabf550 2024-04-25T11:35:12.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.831+0000 7f2923839740 10 client.15546 fetch_fsmap learned FSMap version 26 2024-04-25T11:35:12.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.831+0000 7f2923839740 1 -- 172.21.15.90:0/625844131 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b73af0c970 con 0x55b73aabf550 2024-04-25T11:35:12.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.831+0000 7f28fdffb640 1 -- 172.21.15.90:0/625844131 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f291001fde0 con 0x55b73aabf550 2024-04-25T11:35:12.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.831+0000 7f28fdffb640 1 -- 172.21.15.90:0/625844131 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f290801b430 con 0x55b73aabf550 2024-04-25T11:35:12.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.831+0000 7f2923839740 10 client.15546 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:35:12.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.831+0000 7f2923839740 10 client.15546 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:35:12.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.831+0000 7f2923839740 20 client.15546 populate_metadata read hostname 'smithi090' 2024-04-25T11:35:12.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.831+0000 7f2923839740 1 --2- 172.21.15.90:0/625844131 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b73aa7c630 0x55b73af0f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:12.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.831+0000 7f2923839740 4 client.15546 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T11:35:12.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.831+0000 7f2923839740 1 -- 172.21.15.90:0/625844131 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b73ad71ac0 con 0x55b73aa7c630 2024-04-25T11:35:12.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.831+0000 7f2915d9b640 1 --2- 172.21.15.90:0/625844131 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b73aa7c630 0x55b73af0f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:12.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.832+0000 7f2915d9b640 1 --2- 172.21.15.90:0/625844131 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b73aa7c630 0x55b73af0f120 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c61fa1ecf5047139 server_cookie=9560be4aaf59c773 in_seq=0 out_seq=0 2024-04-25T11:35:12.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.832+0000 7f28fdffb640 10 client.15546 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:12.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.833+0000 7f28fdffb640 1 -- 172.21.15.90:0/625844131 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b73ad71ac0 con 0x55b73aa7c630 2024-04-25T11:35:12.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.833+0000 7f28fdffb640 10 client.15546 handle_command_reply: tid=0 2024-04-25T11:35:12.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.833+0000 7f2923839740 1 client.15546 shutdown 2024-04-25T11:35:12.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.833+0000 7f2923839740 1 -- 172.21.15.90:0/625844131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aabf550 msgr2=0x55b73aea7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:12.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.833+0000 7f2923839740 1 --2- 172.21.15.90:0/625844131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aabf550 0x55b73aea7e50 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f2908005d00 tx=0x7f2908025770 comp rx=0 tx=0).stop 2024-04-25T11:35:12.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.833+0000 7f2923839740 1 -- 172.21.15.90:0/625844131 shutdown_connections 2024-04-25T11:35:12.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.833+0000 7f2923839740 1 --2- 172.21.15.90:0/625844131 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b73aa7c630 0x55b73af0f120 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.833+0000 7f28fdffb640 0 client.15546 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:12.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.833+0000 7f2923839740 1 --2- 172.21.15.90:0/625844131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73aae8410 0x55b73aea52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.833+0000 7f2923839740 1 --2- 172.21.15.90:0/625844131 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b73aac7730 0x55b73aea8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.833+0000 7f2923839740 1 --2- 172.21.15.90:0/625844131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b73aabf550 0x55b73aea7e50 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.833+0000 7f2923839740 1 -- 172.21.15.90:0/625844131 >> 172.21.15.90:0/625844131 conn(0x55b73aadc560 msgr2=0x55b73ae88de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:12.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.833+0000 7f2923839740 1 -- 172.21.15.90:0/625844131 shutdown_connections 2024-04-25T11:35:12.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.833+0000 7f2923839740 1 -- 172.21.15.90:0/625844131 wait complete. 2024-04-25T11:35:12.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.834+0000 7f2923839740 20 client.15546 trim_cache size 0 max 16384 2024-04-25T11:35:12.829 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:35:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-25T11:35:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "bc6d0341-dd38-4d17-94da-e472ae3d9656", 2024-04-25T11:35:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-25T11:35:12.829 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:35:12.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.834+0000 7f291c984640 1 -- 172.21.15.90:0/1160436878 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f29000afcf0 msgr2=0x7f29000b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:12.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.834+0000 7f291c984640 1 --2- 172.21.15.90:0/1160436878 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f29000afcf0 0x7f29000b2200 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x55b73ae85790 tx=0x7f2910009500 comp rx=0 tx=0).stop 2024-04-25T11:35:12.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.834+0000 7f291c984640 1 -- 172.21.15.90:0/1160436878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2918108a40 msgr2=0x7f2918199900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:12.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.834+0000 7f291c984640 1 --2- 172.21.15.90:0/1160436878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2918108a40 0x7f2918199900 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f290c005ab0 tx=0x7f290c003b90 comp rx=0 tx=0).stop 2024-04-25T11:35:12.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.834+0000 7f291c984640 1 -- 172.21.15.90:0/1160436878 shutdown_connections 2024-04-25T11:35:12.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.834+0000 7f291c984640 1 --2- 172.21.15.90:0/1160436878 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f29000afcf0 0x7f29000b2200 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.834+0000 7f291c984640 1 --2- 172.21.15.90:0/1160436878 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f291810d3a0 0x7f2918199e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.834+0000 7f291c984640 1 --2- 172.21.15.90:0/1160436878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2918108a40 0x7f2918199900 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.834+0000 7f291c984640 1 --2- 172.21.15.90:0/1160436878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2918102b10 0x7f29181993c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:12.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.834+0000 7f291c984640 1 -- 172.21.15.90:0/1160436878 >> 172.21.15.90:0/1160436878 conn(0x7f2918078180 msgr2=0x7f29181040d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:12.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.834+0000 7f291c984640 1 -- 172.21.15.90:0/1160436878 shutdown_connections 2024-04-25T11:35:12.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.834+0000 7f291c984640 1 -- 172.21.15.90:0/1160436878 wait complete. 2024-04-25T11:35:12.835 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:35:12.835 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:35:12.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:12.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:12.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: Adjusting osd_memory_target on smithi138 to 176.2M 2024-04-25T11:35:12.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: Unable to set osd_memory_target on smithi138 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-04-25T11:35:12.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:12.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:12.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: Adjusting osd_memory_target on smithi156 to 176.2M 2024-04-25T11:35:12.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: Unable to set osd_memory_target on smithi156 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-25T11:35:12.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3184600101' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:35:12.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:12.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:12.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:35:12.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: Adjusting osd_memory_target on smithi090 to 176.2M 2024-04-25T11:35:12.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: Unable to set osd_memory_target on smithi090 to 184765952: error parsing value: Value '184765952' is below minimum 939524096 2024-04-25T11:35:12.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:35:12.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:35:12.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: Updating smithi090:/etc/ceph/ceph.conf 2024-04-25T11:35:12.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: Updating smithi138:/etc/ceph/ceph.conf 2024-04-25T11:35:12.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-25T11:35:12.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: Updating smithi090:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T11:35:12.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: Updating smithi138:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T11:35:12.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: Updating smithi156:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T11:35:12.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1936851046' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:35:12.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:12 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2581241863' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:35:12.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.959+0000 7f27ef7a8640 1 -- 172.21.15.90:0/2839324783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27e8108ec0 msgr2=0x7f27e8103160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:13.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.959+0000 7f27ef7a8640 1 --2- 172.21.15.90:0/2839324783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27e8108ec0 0x7f27e8103160 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f27d8009900 tx=0x7f27d802a410 comp rx=0 tx=0).stop 2024-04-25T11:35:13.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.959+0000 7f27ef7a8640 1 -- 172.21.15.90:0/2839324783 shutdown_connections 2024-04-25T11:35:13.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.959+0000 7f27ef7a8640 1 --2- 172.21.15.90:0/2839324783 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27e81037d0 0x7f27e810f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.959+0000 7f27ef7a8640 1 --2- 172.21.15.90:0/2839324783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27e8108ec0 0x7f27e8103160 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.959+0000 7f27ef7a8640 1 --2- 172.21.15.90:0/2839324783 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f27e8108a20 0x7f27e8102c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.959+0000 7f27ef7a8640 1 -- 172.21.15.90:0/2839324783 >> 172.21.15.90:0/2839324783 conn(0x7f27e8077880 msgr2=0x7f27e8077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:13.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.959+0000 7f27ef7a8640 1 -- 172.21.15.90:0/2839324783 shutdown_connections 2024-04-25T11:35:13.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.959+0000 7f27ef7a8640 1 -- 172.21.15.90:0/2839324783 wait complete. 2024-04-25T11:35:13.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.960+0000 7f27ef7a8640 1 Processor -- start 2024-04-25T11:35:13.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.960+0000 7f27ef7a8640 1 -- start start 2024-04-25T11:35:13.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.960+0000 7f27ef7a8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27e81037d0 0x7f27e819b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.960+0000 7f27ef7a8640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27e8108a20 0x7f27e819bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.960+0000 7f27ef7a8640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f27e8108ec0 0x7f27e819c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.960+0000 7f27ef7a8640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f27e8112c10 con 0x7f27e8108a20 2024-04-25T11:35:13.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.960+0000 7f27ef7a8640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f27e8112a90 con 0x7f27e8108ec0 2024-04-25T11:35:13.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.960+0000 7f27ef7a8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f27e8112d90 con 0x7f27e81037d0 2024-04-25T11:35:13.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.960+0000 7f27ecd43640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27e8108a20 0x7f27e819bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.960+0000 7f27ecd43640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27e8108a20 0x7f27e819bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33808/0 (socket says 172.21.15.90:33808) 2024-04-25T11:35:13.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.960+0000 7f27ecd43640 1 -- 172.21.15.90:0/302747888 learned_addr learned my addr 172.21.15.90:0/302747888 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:13.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.960+0000 7f27edd45640 1 --2- 172.21.15.90:0/302747888 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f27e8108ec0 0x7f27e819c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.961+0000 7f27ed544640 1 --2- 172.21.15.90:0/302747888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27e81037d0 0x7f27e819b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.961+0000 7f27edd45640 1 -- 172.21.15.90:0/302747888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27e81037d0 msgr2=0x7f27e819b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:13.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.961+0000 7f27edd45640 1 --2- 172.21.15.90:0/302747888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27e81037d0 0x7f27e819b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.961+0000 7f27edd45640 1 -- 172.21.15.90:0/302747888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27e8108a20 msgr2=0x7f27e819bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:13.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.961+0000 7f27edd45640 1 --2- 172.21.15.90:0/302747888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27e8108a20 0x7f27e819bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.961+0000 7f27edd45640 1 -- 172.21.15.90:0/302747888 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27e81a6900 con 0x7f27e8108ec0 2024-04-25T11:35:13.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.961+0000 7f27ecd43640 1 --2- 172.21.15.90:0/302747888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27e8108a20 0x7f27e819bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:35:13.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.961+0000 7f27edd45640 1 --2- 172.21.15.90:0/302747888 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f27e8108ec0 0x7f27e819c070 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f27e0009fd0 tx=0x7f27e000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:13.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.961+0000 7f27de7fc640 1 -- 172.21.15.90:0/302747888 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e000ce20 con 0x7f27e8108ec0 2024-04-25T11:35:13.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.961+0000 7f27ef7a8640 1 -- 172.21.15.90:0/302747888 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27e81a6bf0 con 0x7f27e8108ec0 2024-04-25T11:35:13.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.961+0000 7f27de7fc640 1 -- 172.21.15.90:0/302747888 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27e0005020 con 0x7f27e8108ec0 2024-04-25T11:35:13.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.961+0000 7f27de7fc640 1 -- 172.21.15.90:0/302747888 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e0010690 con 0x7f27e8108ec0 2024-04-25T11:35:13.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.961+0000 7f27ef7a8640 1 -- 172.21.15.90:0/302747888 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27e81a7190 con 0x7f27e8108ec0 2024-04-25T11:35:13.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.963+0000 7f27de7fc640 1 -- 172.21.15.90:0/302747888 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 23) ==== 147696+0+0 (secure 0 0 0) 0x7f27e00054d0 con 0x7f27e8108ec0 2024-04-25T11:35:13.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.964+0000 7f27f6655740 1 Processor -- start 2024-04-25T11:35:13.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.964+0000 7f27f6655740 1 -- start start 2024-04-25T11:35:13.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.964+0000 7f27f6655740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5557b2148390 0x5557b24e8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.964+0000 7f27f6655740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557b211f4d0 0x5557b24e8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.964+0000 7f27f6655740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5557b21276b0 0x5557b24f58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.964+0000 7f27f6655740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5557b244ed80 con 0x5557b21276b0 2024-04-25T11:35:13.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.964+0000 7f27f6655740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5557b24088b0 con 0x5557b2148390 2024-04-25T11:35:13.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.964+0000 7f27f6655740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5557b244ad40 con 0x5557b211f4d0 2024-04-25T11:35:13.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.964+0000 7f27de7fc640 1 --2- 172.21.15.90:0/302747888 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f27d40afcf0 0x7f27d40b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.965+0000 7f27ecd43640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5557b2148390 0x5557b24e8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.965+0000 7f27ecd43640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5557b2148390 0x5557b24e8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:32982/0 (socket says 172.21.15.90:32982) 2024-04-25T11:35:13.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.965+0000 7f27edd45640 1 --2- 172.21.15.90:0/302747888 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f27d40afcf0 0x7f27d40b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.965+0000 7f27ecd43640 1 -- 172.21.15.90:0/3829186590 learned_addr learned my addr 172.21.15.90:0/3829186590 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:13.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.965+0000 7f27ecd43640 1 --2- 172.21.15.90:0/3829186590 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5557b21276b0 0x5557b24f58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.965+0000 7f27ed544640 1 --2- 172.21.15.90:0/3829186590 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557b211f4d0 0x5557b24e8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.965+0000 7f27de7fc640 1 -- 172.21.15.90:0/302747888 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f27e00df8e0 con 0x7f27e8108ec0 2024-04-25T11:35:13.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.965+0000 7f27ecd43640 1 -- 172.21.15.90:0/3829186590 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557b211f4d0 msgr2=0x5557b24e8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:13.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.965+0000 7f27ecd43640 1 --2- 172.21.15.90:0/3829186590 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557b211f4d0 0x5557b24e8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.965+0000 7f27ecd43640 1 -- 172.21.15.90:0/3829186590 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5557b21276b0 msgr2=0x5557b24f58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:13.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.965+0000 7f27ecd43640 1 --2- 172.21.15.90:0/3829186590 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5557b21276b0 0x5557b24f58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.965+0000 7f27ecd43640 1 -- 172.21.15.90:0/3829186590 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557b23d9a40 con 0x5557b2148390 2024-04-25T11:35:13.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.965+0000 7f27edd45640 1 --2- 172.21.15.90:0/302747888 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f27d40afcf0 0x7f27d40b2200 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f27e00dcf70 tx=0x7f27e00a3040 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:13.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.965+0000 7f27ecd43640 1 --2- 172.21.15.90:0/3829186590 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5557b2148390 0x5557b24e8a60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f27d8005f70 tx=0x7f27d8032f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=849475d34b375b87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:13.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.965+0000 7f27d37fe640 1 -- 172.21.15.90:0/3829186590 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d8035070 con 0x5557b2148390 2024-04-25T11:35:13.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.965+0000 7f27d37fe640 1 -- 172.21.15.90:0/3829186590 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27d80054d0 con 0x5557b2148390 2024-04-25T11:35:13.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.965+0000 7f27d37fe640 1 -- 172.21.15.90:0/3829186590 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d8004b20 con 0x5557b2148390 2024-04-25T11:35:13.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.966+0000 7f27f6655740 1 -- 172.21.15.90:0/3829186590 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5557b2148390 msgr2=0x5557b24e8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:13.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.966+0000 7f27f6655740 1 --2- 172.21.15.90:0/3829186590 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5557b2148390 0x5557b24e8a60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f27d8005f70 tx=0x7f27d8032f40 comp rx=0 tx=0).stop 2024-04-25T11:35:13.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.966+0000 7f27f6655740 1 -- 172.21.15.90:0/3829186590 shutdown_connections 2024-04-25T11:35:13.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.966+0000 7f27f6655740 1 --2- 172.21.15.90:0/3829186590 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5557b21276b0 0x5557b24f58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.966+0000 7f27f6655740 1 --2- 172.21.15.90:0/3829186590 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557b211f4d0 0x5557b24e8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.966+0000 7f27f6655740 1 --2- 172.21.15.90:0/3829186590 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5557b2148390 0x5557b24e8a60 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.966+0000 7f27f6655740 1 -- 172.21.15.90:0/3829186590 >> 172.21.15.90:0/3829186590 conn(0x5557b213c4e0 msgr2=0x5557b24e5210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:13.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.966+0000 7f27f6655740 1 -- 172.21.15.90:0/3829186590 shutdown_connections 2024-04-25T11:35:13.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.966+0000 7f27f6655740 1 -- 172.21.15.90:0/3829186590 wait complete. 2024-04-25T11:35:13.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.967+0000 7f27f6655740 1 -- start start 2024-04-25T11:35:13.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.967+0000 7f27f6655740 1 Processor -- start 2024-04-25T11:35:13.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.967+0000 7f27f6655740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5557b211f4d0 0x5557b24f5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.967+0000 7f27f6655740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557b21276b0 0x5557b24f55d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.967+0000 7f27f6655740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5557b2148390 0x5557b24f3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.967+0000 7f27f6655740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5557b24f5f80 con 0x5557b211f4d0 2024-04-25T11:35:13.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.967+0000 7f27f6655740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5557b23d5d10 con 0x5557b2148390 2024-04-25T11:35:13.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.967+0000 7f27f6655740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5557b24f6100 con 0x5557b21276b0 2024-04-25T11:35:13.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.967+0000 7f27ecd43640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5557b211f4d0 0x5557b24f5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.967+0000 7f27f6655740 10 client.0 _ll_register_callbacks cb 0x5557b2402210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:35:13.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.967+0000 7f27ecd43640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5557b211f4d0 0x5557b24f5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33832/0 (socket says 172.21.15.90:33832) 2024-04-25T11:35:13.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.967+0000 7f27ecd43640 1 -- 172.21.15.90:0/4278142715 learned_addr learned my addr 172.21.15.90:0/4278142715 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:13.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.967+0000 7f27ed544640 1 --2- 172.21.15.90:0/4278142715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557b21276b0 0x5557b24f55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.967+0000 7f27ecd43640 1 --2- 172.21.15.90:0/4278142715 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5557b2148390 0x5557b24f3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.968+0000 7f27ed544640 1 -- 172.21.15.90:0/4278142715 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5557b2148390 msgr2=0x5557b24f3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:13.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.968+0000 7f27ed544640 1 --2- 172.21.15.90:0/4278142715 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5557b2148390 0x5557b24f3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.968+0000 7f27ed544640 1 -- 172.21.15.90:0/4278142715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5557b211f4d0 msgr2=0x5557b24f5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:13.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.968+0000 7f27ed544640 1 --2- 172.21.15.90:0/4278142715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5557b211f4d0 0x5557b24f5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.968+0000 7f27ed544640 1 -- 172.21.15.90:0/4278142715 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557b222a7d0 con 0x5557b21276b0 2024-04-25T11:35:13.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.968+0000 7f27ecd43640 1 --2- 172.21.15.90:0/4278142715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5557b211f4d0 0x5557b24f5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:35:13.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.968+0000 7f27ed544640 1 --2- 172.21.15.90:0/4278142715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557b21276b0 0x5557b24f55d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x5557b24e61a0 tx=0x7f27e4012b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21babecfcf9ffc60 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:13.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.968+0000 7f27d0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:35:13.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.968+0000 7f27d0ff9640 1 -- 172.21.15.90:0/4278142715 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e4004030 con 0x5557b21276b0 2024-04-25T11:35:13.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.968+0000 7f27f6655740 1 -- 172.21.15.90:0/4278142715 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27d80054d0 con 0x5557b21276b0 2024-04-25T11:35:13.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.968+0000 7f27d0ff9640 1 -- 172.21.15.90:0/4278142715 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f27e40041d0 con 0x5557b21276b0 2024-04-25T11:35:13.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.968+0000 7f27d0ff9640 1 -- 172.21.15.90:0/4278142715 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f27e40043d0 con 0x5557b21276b0 2024-04-25T11:35:13.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.970+0000 7f27f6655740 10 client.25141 fetch_fsmap learned FSMap version 26 2024-04-25T11:35:13.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.970+0000 7f27f6655740 1 -- 172.21.15.90:0/4278142715 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5557b21603f0 con 0x5557b21276b0 2024-04-25T11:35:13.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.970+0000 7f27d0ff9640 1 -- 172.21.15.90:0/4278142715 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27d80054d0 con 0x5557b21276b0 2024-04-25T11:35:13.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.970+0000 7f27d0ff9640 1 -- 172.21.15.90:0/4278142715 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f27e4003ab0 con 0x5557b21276b0 2024-04-25T11:35:13.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.970+0000 7f27f6655740 10 client.25141 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:35:13.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.970+0000 7f27f6655740 10 client.25141 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:35:13.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.970+0000 7f27f6655740 20 client.25141 populate_metadata read hostname 'smithi090' 2024-04-25T11:35:13.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.970+0000 7f27f6655740 1 --2- 172.21.15.90:0/4278142715 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5557b2106f60 0x5557b2505840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.970+0000 7f27f6655740 4 client.25141 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:35:13.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.970+0000 7f27f6655740 1 -- 172.21.15.90:0/4278142715 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5557b245c990 con 0x5557b2106f60 2024-04-25T11:35:13.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.970+0000 7f27ecd43640 1 --2- 172.21.15.90:0/4278142715 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5557b2106f60 0x5557b2505840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.970+0000 7f27d0ff9640 10 client.25141 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:13.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.970+0000 7f27ecd43640 1 --2- 172.21.15.90:0/4278142715 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5557b2106f60 0x5557b2505840 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78285a986207fd14 server_cookie=89da774924332f0c in_seq=0 out_seq=0 2024-04-25T11:35:13.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.970+0000 7f27d0ff9640 1 -- 172.21.15.90:0/4278142715 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5557b245c990 con 0x5557b2106f60 2024-04-25T11:35:13.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.970+0000 7f27d0ff9640 10 client.25141 handle_command_reply: tid=0 2024-04-25T11:35:13.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.971+0000 7f27f6655740 1 client.25141 shutdown 2024-04-25T11:35:13.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.971+0000 7f27f6655740 1 -- 172.21.15.90:0/4278142715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557b21276b0 msgr2=0x5557b24f55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:13.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.971+0000 7f27f6655740 1 --2- 172.21.15.90:0/4278142715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557b21276b0 0x5557b24f55d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x5557b24e61a0 tx=0x7f27e4012b60 comp rx=0 tx=0).stop 2024-04-25T11:35:13.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.971+0000 7f27f6655740 1 -- 172.21.15.90:0/4278142715 shutdown_connections 2024-04-25T11:35:13.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.971+0000 7f27f6655740 1 --2- 172.21.15.90:0/4278142715 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5557b2106f60 0x5557b2505840 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.971+0000 7f27f6655740 1 --2- 172.21.15.90:0/4278142715 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5557b2148390 0x5557b24f3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.971+0000 7f27d0ff9640 0 client.25141 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:13.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.971+0000 7f27f6655740 1 --2- 172.21.15.90:0/4278142715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557b21276b0 0x5557b24f55d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.971+0000 7f27f6655740 1 --2- 172.21.15.90:0/4278142715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5557b211f4d0 0x5557b24f5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.971+0000 7f27f6655740 1 -- 172.21.15.90:0/4278142715 >> 172.21.15.90:0/4278142715 conn(0x5557b213c4e0 msgr2=0x5557b24e39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:13.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.971+0000 7f27f6655740 1 -- 172.21.15.90:0/4278142715 shutdown_connections 2024-04-25T11:35:13.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.971+0000 7f27f6655740 1 -- 172.21.15.90:0/4278142715 wait complete. 2024-04-25T11:35:13.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.971+0000 7f27f6655740 20 client.25141 trim_cache size 0 max 16384 2024-04-25T11:35:13.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.990+0000 7f5dc7eb1640 1 -- 172.21.15.90:0/512053787 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/log_level}] v 0) -- 0x7f5dc019ba70 con 0x7f5dc0109b90 2024-04-25T11:35:13.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.993+0000 7f27f6655740 1 Processor -- start 2024-04-25T11:35:13.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.994+0000 7f27f6655740 1 -- start start 2024-04-25T11:35:13.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.994+0000 7f27f6655740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5557b211f4d0 0x5557b24e8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.994+0000 7f27f6655740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5557b21276b0 0x5557b24fb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.994+0000 7f27f6655740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557b2148390 0x5557b24fb920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.994+0000 7f27f6655740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f27e81afac0 con 0x5557b21276b0 2024-04-25T11:35:13.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.994+0000 7f27f6655740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f27e81b1600 con 0x5557b211f4d0 2024-04-25T11:35:13.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.994+0000 7f27f6655740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f27e81b0190 con 0x5557b2148390 2024-04-25T11:35:13.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.994+0000 7f27ed544640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5557b21276b0 0x5557b24fb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.994+0000 7f27ed544640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5557b21276b0 0x5557b24fb3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33840/0 (socket says 172.21.15.90:33840) 2024-04-25T11:35:13.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.994+0000 7f27ed544640 1 -- 172.21.15.90:0/847135600 learned_addr learned my addr 172.21.15.90:0/847135600 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:13.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.994+0000 7f27ecd43640 1 --2- 172.21.15.90:0/847135600 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5557b211f4d0 0x5557b24e8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.994+0000 7f27ecd43640 1 --2- 172.21.15.90:0/847135600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557b2148390 0x5557b24fb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.994+0000 7f27ed544640 1 -- 172.21.15.90:0/847135600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557b2148390 msgr2=0x5557b24fb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:13.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.994+0000 7f27ed544640 1 --2- 172.21.15.90:0/847135600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557b2148390 0x5557b24fb920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.994+0000 7f27ed544640 1 -- 172.21.15.90:0/847135600 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5557b211f4d0 msgr2=0x5557b24e8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:13.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.994+0000 7f27ed544640 1 --2- 172.21.15.90:0/847135600 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5557b211f4d0 0x5557b24e8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.994+0000 7f27ed544640 1 -- 172.21.15.90:0/847135600 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557b21200f0 con 0x5557b21276b0 2024-04-25T11:35:13.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.994+0000 7f27ed544640 1 --2- 172.21.15.90:0/847135600 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5557b21276b0 0x5557b24fb3e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f27e4003f50 tx=0x7f27e4009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ea8480619819d7c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:13.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.996+0000 7f5db6ffd640 1 -- 172.21.15.90:0/512053787 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/log_level}]=0 v30)=0 v30) ==== 129+0+0 (secure 0 0 0) 0x7f5dc019ba70 con 0x7f5dc0109b90 2024-04-25T11:35:13.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.998+0000 7f27d2ffd640 1 -- 172.21.15.90:0/847135600 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e4003fd0 con 0x5557b21276b0 2024-04-25T11:35:13.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.998+0000 7f27d2ffd640 1 -- 172.21.15.90:0/847135600 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27e4010dd0 con 0x5557b21276b0 2024-04-25T11:35:13.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.998+0000 7f27d2ffd640 1 -- 172.21.15.90:0/847135600 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e400fe70 con 0x5557b21276b0 2024-04-25T11:35:13.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.998+0000 7f5dc7eb1640 1 -- 172.21.15.90:0/512053787 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5dac0afce0 msgr2=0x7f5dac0b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:13.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.998+0000 7f5dc7eb1640 1 --2- 172.21.15.90:0/512053787 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5dac0afce0 0x7f5dac0b21f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0104870 tx=0x7f5dbc005f90 comp rx=0 tx=0).stop 2024-04-25T11:35:13.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.998+0000 7f5dc7eb1640 1 -- 172.21.15.90:0/512053787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5dc0109b90 msgr2=0x7f5dc019ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:13.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.998+0000 7f5dc7eb1640 1 --2- 172.21.15.90:0/512053787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5dc0109b90 0x7f5dc019ac80 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f5db0005d00 tx=0x7f5db000dde0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.998+0000 7f5dc7eb1640 1 -- 172.21.15.90:0/512053787 shutdown_connections 2024-04-25T11:35:13.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.998+0000 7f5dc7eb1640 1 --2- 172.21.15.90:0/512053787 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5dac0afce0 0x7f5dac0b21f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.998+0000 7f5dc7eb1640 1 --2- 172.21.15.90:0/512053787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5dc010a030 0x7f5dc019ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.998+0000 7f5dc7eb1640 1 --2- 172.21.15.90:0/512053787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5dc0109b90 0x7f5dc019ac80 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.998+0000 7f5dc7eb1640 1 --2- 172.21.15.90:0/512053787 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5dc01042e0 0x7f5dc019a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.998+0000 7f5dc7eb1640 1 -- 172.21.15.90:0/512053787 >> 172.21.15.90:0/512053787 conn(0x7f5dc00ff360 msgr2=0x7f5dc0106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:13.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.998+0000 7f5dc7eb1640 1 -- 172.21.15.90:0/512053787 shutdown_connections 2024-04-25T11:35:13.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.998+0000 7f5dc7eb1640 1 -- 172.21.15.90:0/512053787 wait complete. 2024-04-25T11:35:13.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.998+0000 7f27f6655740 1 -- 172.21.15.90:0/847135600 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5557b21276b0 msgr2=0x5557b24fb3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:13.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.998+0000 7f27f6655740 1 --2- 172.21.15.90:0/847135600 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5557b21276b0 0x5557b24fb3e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f27e4003f50 tx=0x7f27e4009f80 comp rx=0 tx=0).stop 2024-04-25T11:35:13.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.998+0000 7f27f6655740 1 -- 172.21.15.90:0/847135600 shutdown_connections 2024-04-25T11:35:13.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.998+0000 7f27f6655740 1 --2- 172.21.15.90:0/847135600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557b2148390 0x5557b24fb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.998+0000 7f27f6655740 1 --2- 172.21.15.90:0/847135600 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5557b21276b0 0x5557b24fb3e0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.998+0000 7f27f6655740 1 --2- 172.21.15.90:0/847135600 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5557b211f4d0 0x5557b24e8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.998+0000 7f27f6655740 1 -- 172.21.15.90:0/847135600 >> 172.21.15.90:0/847135600 conn(0x5557b213c4e0 msgr2=0x5557b24e87f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:13.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.998+0000 7f27f6655740 1 -- 172.21.15.90:0/847135600 shutdown_connections 2024-04-25T11:35:13.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.998+0000 7f27f6655740 1 -- 172.21.15.90:0/847135600 wait complete. 2024-04-25T11:35:13.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.998+0000 7f27f6655740 1 -- start start 2024-04-25T11:35:13.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:12.998+0000 7f27f6655740 1 Processor -- start 2024-04-25T11:35:13.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.000+0000 7f27f6655740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5557b211f4d0 0x5557b2564f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.000+0000 7f27f6655740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5557b21276b0 0x5557b2565460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.000+0000 7f27f6655740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557b2148390 0x5557b24f87a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.000+0000 7f27f6655740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f27e81b0f30 con 0x5557b211f4d0 2024-04-25T11:35:13.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.000+0000 7f27f6655740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f27e81b0860 con 0x5557b21276b0 2024-04-25T11:35:13.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.000+0000 7f27f6655740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f27e81adaa0 con 0x5557b2148390 2024-04-25T11:35:13.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.000+0000 7f27f6655740 10 client.0 _ll_register_callbacks cb 0x7f27e4009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:35:13.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.000+0000 7f27ecd43640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5557b211f4d0 0x5557b2564f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.000+0000 7f27ecd43640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5557b211f4d0 0x5557b2564f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33856/0 (socket says 172.21.15.90:33856) 2024-04-25T11:35:13.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.000+0000 7f27ecd43640 1 -- 172.21.15.90:0/483779684 learned_addr learned my addr 172.21.15.90:0/483779684 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:13.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.000+0000 7f27ed544640 1 --2- 172.21.15.90:0/483779684 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5557b21276b0 0x5557b2565460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.000+0000 7f27ecd43640 1 --2- 172.21.15.90:0/483779684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557b2148390 0x5557b24f87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.000+0000 7f27ecd43640 1 -- 172.21.15.90:0/483779684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557b2148390 msgr2=0x5557b24f87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:13.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.000+0000 7f27ecd43640 1 --2- 172.21.15.90:0/483779684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557b2148390 0x5557b24f87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.000+0000 7f27ecd43640 1 -- 172.21.15.90:0/483779684 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5557b21276b0 msgr2=0x5557b2565460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:13.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.000+0000 7f27ecd43640 1 --2- 172.21.15.90:0/483779684 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5557b21276b0 0x5557b2565460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.000+0000 7f27ecd43640 1 -- 172.21.15.90:0/483779684 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557b240f4b0 con 0x5557b211f4d0 2024-04-25T11:35:13.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.000+0000 7f27ed544640 1 --2- 172.21.15.90:0/483779684 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5557b21276b0 0x5557b2565460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:35:13.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.001+0000 7f27ecd43640 1 --2- 172.21.15.90:0/483779684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5557b211f4d0 0x5557b2564f20 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f27d8006010 tx=0x7f27d8005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb302bcaa2590b6a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:13.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.001+0000 7f27d0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:35:13.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.001+0000 7f27d0ff9640 1 -- 172.21.15.90:0/483779684 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d805e070 con 0x5557b211f4d0 2024-04-25T11:35:13.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.001+0000 7f27d0ff9640 1 -- 172.21.15.90:0/483779684 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f27d8004400 con 0x5557b211f4d0 2024-04-25T11:35:13.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.001+0000 7f27f6655740 1 -- 172.21.15.90:0/483779684 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27e4010dd0 con 0x5557b211f4d0 2024-04-25T11:35:13.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.001+0000 7f27d0ff9640 1 -- 172.21.15.90:0/483779684 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f27d8035040 con 0x5557b211f4d0 2024-04-25T11:35:13.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.001+0000 7f27f6655740 10 client.15564 fetch_fsmap learned FSMap version 26 2024-04-25T11:35:13.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.001+0000 7f27f6655740 1 -- 172.21.15.90:0/483779684 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5557b256d3a0 con 0x5557b211f4d0 2024-04-25T11:35:13.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.001+0000 7f27d0ff9640 1 -- 172.21.15.90:0/483779684 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27e4010dd0 con 0x5557b211f4d0 2024-04-25T11:35:13.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.001+0000 7f27d0ff9640 1 -- 172.21.15.90:0/483779684 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f27d8003710 con 0x5557b211f4d0 2024-04-25T11:35:13.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.001+0000 7f27f6655740 10 client.15564 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:35:13.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.001+0000 7f27f6655740 10 client.15564 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:35:13.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.001+0000 7f27f6655740 20 client.15564 populate_metadata read hostname 'smithi090' 2024-04-25T11:35:13.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.001+0000 7f27f6655740 1 --2- 172.21.15.90:0/483779684 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5557b20dc3a0 0x5557b256f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.001+0000 7f27f6655740 4 client.15564 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:35:13.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.001+0000 7f27f6655740 1 -- 172.21.15.90:0/483779684 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5557b2456800 con 0x5557b20dc3a0 2024-04-25T11:35:13.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.002+0000 7f27ed544640 1 --2- 172.21.15.90:0/483779684 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5557b20dc3a0 0x5557b256f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.002+0000 7f27d0ff9640 10 client.15564 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:13.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.002+0000 7f27ed544640 1 --2- 172.21.15.90:0/483779684 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5557b20dc3a0 0x5557b256f5d0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=373d406b4fac5d89 server_cookie=822fafcc08c0860d in_seq=0 out_seq=0 2024-04-25T11:35:13.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.002+0000 7f27d0ff9640 1 -- 172.21.15.90:0/483779684 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5557b2456800 con 0x5557b20dc3a0 2024-04-25T11:35:13.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.002+0000 7f27d0ff9640 10 client.15564 handle_command_reply: tid=0 2024-04-25T11:35:13.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.003+0000 7f27f6655740 1 client.15564 shutdown 2024-04-25T11:35:13.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.003+0000 7f27f6655740 1 -- 172.21.15.90:0/483779684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5557b211f4d0 msgr2=0x5557b2564f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:13.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.003+0000 7f27f6655740 1 --2- 172.21.15.90:0/483779684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5557b211f4d0 0x5557b2564f20 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f27d8006010 tx=0x7f27d8005850 comp rx=0 tx=0).stop 2024-04-25T11:35:13.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.003+0000 7f27f6655740 1 -- 172.21.15.90:0/483779684 shutdown_connections 2024-04-25T11:35:13.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.003+0000 7f27f6655740 1 --2- 172.21.15.90:0/483779684 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5557b20dc3a0 0x5557b256f5d0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.003+0000 7f27f6655740 1 --2- 172.21.15.90:0/483779684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5557b2148390 0x5557b24f87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.003+0000 7f27f6655740 1 --2- 172.21.15.90:0/483779684 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5557b21276b0 0x5557b2565460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.003+0000 7f27f6655740 1 --2- 172.21.15.90:0/483779684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5557b211f4d0 0x5557b2564f20 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.003+0000 7f27f6655740 1 -- 172.21.15.90:0/483779684 >> 172.21.15.90:0/483779684 conn(0x5557b213c4e0 msgr2=0x5557b24e39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:13.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.003+0000 7f27d0ff9640 0 client.15564 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:13.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.003+0000 7f27f6655740 1 -- 172.21.15.90:0/483779684 shutdown_connections 2024-04-25T11:35:13.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.003+0000 7f27f6655740 1 -- 172.21.15.90:0/483779684 wait complete. 2024-04-25T11:35:13.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.003+0000 7f27f6655740 20 client.15564 trim_cache size 0 max 16384 2024-04-25T11:35:13.090 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:35:13.090 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T11:35:13.090 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-25T11:35:13.090 INFO:teuthology.orchestra.run.smithi090.stdout: "bc6d0341-dd38-4d17-94da-e472ae3d9656": { 2024-04-25T11:35:13.091 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-25T11:35:13.091 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "bc6d0341-dd38-4d17-94da-e472ae3d9656", 2024-04-25T11:35:13.091 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-25T11:35:13.091 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:35:13.091 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:35:13.091 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:35:13.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.004+0000 7f27ef7a8640 1 -- 172.21.15.90:0/302747888 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f27d40afcf0 msgr2=0x7f27d40b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:13.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.004+0000 7f27ef7a8640 1 --2- 172.21.15.90:0/302747888 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f27d40afcf0 0x7f27d40b2200 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f27e00dcf70 tx=0x7f27e00a3040 comp rx=0 tx=0).stop 2024-04-25T11:35:13.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.004+0000 7f27ef7a8640 1 -- 172.21.15.90:0/302747888 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f27e8108ec0 msgr2=0x7f27e819c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:13.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.004+0000 7f27ef7a8640 1 --2- 172.21.15.90:0/302747888 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f27e8108ec0 0x7f27e819c070 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f27e0009fd0 tx=0x7f27e000ef10 comp rx=0 tx=0).stop 2024-04-25T11:35:13.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.004+0000 7f27ef7a8640 1 -- 172.21.15.90:0/302747888 shutdown_connections 2024-04-25T11:35:13.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.004+0000 7f27ef7a8640 1 --2- 172.21.15.90:0/302747888 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f27d40afcf0 0x7f27d40b2200 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.004+0000 7f27ef7a8640 1 --2- 172.21.15.90:0/302747888 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f27e8108ec0 0x7f27e819c070 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.004+0000 7f27ef7a8640 1 --2- 172.21.15.90:0/302747888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27e8108a20 0x7f27e819bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.004+0000 7f27ef7a8640 1 --2- 172.21.15.90:0/302747888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27e81037d0 0x7f27e819b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.004+0000 7f27ef7a8640 1 -- 172.21.15.90:0/302747888 >> 172.21.15.90:0/302747888 conn(0x7f27e8077880 msgr2=0x7f27e81008c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:13.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.004+0000 7f27ef7a8640 1 -- 172.21.15.90:0/302747888 shutdown_connections 2024-04-25T11:35:13.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.004+0000 7f27ef7a8640 1 -- 172.21.15.90:0/302747888 wait complete. 2024-04-25T11:35:13.094 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-25T11:35:13.095 INFO:tasks.cephfs.filesystem:scrub status for tag:bc6d0341-dd38-4d17-94da-e472ae3d9656 - {'path': '/', 'tag': 'bc6d0341-dd38-4d17-94da-e472ae3d9656', 'options': 'recursive,force'} 2024-04-25T11:35:13.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.254+0000 7fd461a67640 1 -- 172.21.15.90:0/1609521759 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd45c10e0f0 msgr2=0x7fd45c1105e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:13.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.254+0000 7fd461a67640 1 --2- 172.21.15.90:0/1609521759 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd45c10e0f0 0x7fd45c1105e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd454009ca0 tx=0x7fd45402a8c0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.254+0000 7fd461a67640 1 -- 172.21.15.90:0/1609521759 shutdown_connections 2024-04-25T11:35:13.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.254+0000 7fd461a67640 1 --2- 172.21.15.90:0/1609521759 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd45c10e0f0 0x7fd45c1105e0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.254+0000 7fd461a67640 1 --2- 172.21.15.90:0/1609521759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd45c10b700 0x7fd45c10db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.254+0000 7fd461a67640 1 --2- 172.21.15.90:0/1609521759 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd45c104ca0 0x7fd45c1050b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.254+0000 7fd461a67640 1 -- 172.21.15.90:0/1609521759 >> 172.21.15.90:0/1609521759 conn(0x7fd45c0781a0 msgr2=0x7fd45c0785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.254+0000 7fd461a67640 1 -- 172.21.15.90:0/1609521759 shutdown_connections 2024-04-25T11:35:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.254+0000 7fd461a67640 1 -- 172.21.15.90:0/1609521759 wait complete. 2024-04-25T11:35:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.255+0000 7fd461a67640 1 Processor -- start 2024-04-25T11:35:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.255+0000 7fd461a67640 1 -- start start 2024-04-25T11:35:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.255+0000 7fd461a67640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd45c104ca0 0x7fd45c19a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.255+0000 7fd461a67640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd45c10b700 0x7fd45c19ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.255+0000 7fd461a67640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd45c10e0f0 0x7fd45c19ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.255+0000 7fd461a67640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd45c113d50 con 0x7fd45c104ca0 2024-04-25T11:35:13.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.255+0000 7fd461a67640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd45c113bd0 con 0x7fd45c10e0f0 2024-04-25T11:35:13.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.255+0000 7fd461a67640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd45c113ed0 con 0x7fd45c10b700 2024-04-25T11:35:13.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.255+0000 7fd45b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd45c104ca0 0x7fd45c19a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.255+0000 7fd45b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd45c104ca0 0x7fd45c19a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33874/0 (socket says 172.21.15.90:33874) 2024-04-25T11:35:13.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.255+0000 7fd45b7fe640 1 -- 172.21.15.90:0/2655933904 learned_addr learned my addr 172.21.15.90:0/2655933904 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:13.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.255+0000 7fd45bfff640 1 --2- 172.21.15.90:0/2655933904 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd45c10e0f0 0x7fd45c19ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.255+0000 7fd45affd640 1 --2- 172.21.15.90:0/2655933904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd45c10b700 0x7fd45c19ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.256+0000 7fd45b7fe640 1 -- 172.21.15.90:0/2655933904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd45c10b700 msgr2=0x7fd45c19ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:13.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.256+0000 7fd45b7fe640 1 --2- 172.21.15.90:0/2655933904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd45c10b700 0x7fd45c19ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.256+0000 7fd45b7fe640 1 -- 172.21.15.90:0/2655933904 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd45c10e0f0 msgr2=0x7fd45c19ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:13.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.256+0000 7fd45b7fe640 1 --2- 172.21.15.90:0/2655933904 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd45c10e0f0 0x7fd45c19ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.256+0000 7fd45b7fe640 1 -- 172.21.15.90:0/2655933904 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd45c19f4d0 con 0x7fd45c104ca0 2024-04-25T11:35:13.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.256+0000 7fd45bfff640 1 --2- 172.21.15.90:0/2655933904 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd45c10e0f0 0x7fd45c19ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:35:13.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.256+0000 7fd45b7fe640 1 --2- 172.21.15.90:0/2655933904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd45c104ca0 0x7fd45c19a6f0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fd44c00d980 tx=0x7fd44c00dd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:13.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.256+0000 7fd458ff9640 1 -- 172.21.15.90:0/2655933904 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd44c017070 con 0x7fd45c104ca0 2024-04-25T11:35:13.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.256+0000 7fd461a67640 1 -- 172.21.15.90:0/2655933904 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd45c19f7c0 con 0x7fd45c104ca0 2024-04-25T11:35:13.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.256+0000 7fd458ff9640 1 -- 172.21.15.90:0/2655933904 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd44c009c30 con 0x7fd45c104ca0 2024-04-25T11:35:13.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.256+0000 7fd458ff9640 1 -- 172.21.15.90:0/2655933904 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd44c00f620 con 0x7fd45c104ca0 2024-04-25T11:35:13.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.256+0000 7fd461a67640 1 -- 172.21.15.90:0/2655933904 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd45c19fd10 con 0x7fd45c104ca0 2024-04-25T11:35:13.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.257+0000 7fd461a67640 1 -- 172.21.15.90:0/2655933904 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd45c050740 con 0x7fd45c104ca0 2024-04-25T11:35:13.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.259+0000 7fd458ff9640 1 -- 172.21.15.90:0/2655933904 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 23) ==== 147696+0+0 (secure 0 0 0) 0x7fd44c010460 con 0x7fd45c104ca0 2024-04-25T11:35:13.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.259+0000 7fd458ff9640 1 --2- 172.21.15.90:0/2655933904 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd4440afce0 0x7fd4440b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.259+0000 7fd45affd640 1 --2- 172.21.15.90:0/2655933904 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd4440afce0 0x7fd4440b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.259+0000 7fd458ff9640 1 -- 172.21.15.90:0/2655933904 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7fd44c0decb0 con 0x7fd45c104ca0 2024-04-25T11:35:13.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.260+0000 7fd45affd640 1 --2- 172.21.15.90:0/2655933904 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd4440afce0 0x7fd4440b21f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd45c110500 tx=0x7fd45000a2d0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:13.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.263+0000 7fd458ff9640 1 -- 172.21.15.90:0/2655933904 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd44c0a73e0 con 0x7fd45c104ca0 2024-04-25T11:35:13.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.439+0000 7fd461a67640 1 -- 172.21.15.90:0/2655933904 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fd45c1a7280 con 0x7fd45c104ca0 2024-04-25T11:35:13.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.443+0000 7fd458ff9640 1 -- 172.21.15.90:0/2655933904 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v31)=0 v31) ==== 138+0+0 (secure 0 0 0) 0x7fd44c0a6ce0 con 0x7fd45c104ca0 2024-04-25T11:35:13.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.445+0000 7fd461a67640 1 -- 172.21.15.90:0/2655933904 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd4440afce0 msgr2=0x7fd4440b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:13.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.445+0000 7fd461a67640 1 --2- 172.21.15.90:0/2655933904 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd4440afce0 0x7fd4440b21f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd45c110500 tx=0x7fd45000a2d0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.445+0000 7fd461a67640 1 -- 172.21.15.90:0/2655933904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd45c104ca0 msgr2=0x7fd45c19a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:13.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.445+0000 7fd461a67640 1 --2- 172.21.15.90:0/2655933904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd45c104ca0 0x7fd45c19a6f0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fd44c00d980 tx=0x7fd44c00dd80 comp rx=0 tx=0).stop 2024-04-25T11:35:13.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.445+0000 7fd461a67640 1 -- 172.21.15.90:0/2655933904 shutdown_connections 2024-04-25T11:35:13.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.445+0000 7fd461a67640 1 --2- 172.21.15.90:0/2655933904 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd4440afce0 0x7fd4440b21f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.445+0000 7fd461a67640 1 --2- 172.21.15.90:0/2655933904 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd45c10e0f0 0x7fd45c19ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.445+0000 7fd461a67640 1 --2- 172.21.15.90:0/2655933904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd45c10b700 0x7fd45c19ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.445+0000 7fd461a67640 1 --2- 172.21.15.90:0/2655933904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd45c104ca0 0x7fd45c19a6f0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.445+0000 7fd461a67640 1 -- 172.21.15.90:0/2655933904 >> 172.21.15.90:0/2655933904 conn(0x7fd45c0781a0 msgr2=0x7fd45c1011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:13.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.445+0000 7fd461a67640 1 -- 172.21.15.90:0/2655933904 shutdown_connections 2024-04-25T11:35:13.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.445+0000 7fd461a67640 1 -- 172.21.15.90:0/2655933904 wait complete. 2024-04-25T11:35:13.446 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-25T11:35:13.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.578+0000 7fe1973d5640 1 -- 172.21.15.90:0/3328493047 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe190108a60 msgr2=0x7fe190102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:13.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.578+0000 7fe1973d5640 1 --2- 172.21.15.90:0/3328493047 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe190108a60 0x7fe190102610 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe188009ca0 tx=0x7fe18802c790 comp rx=0 tx=0).stop 2024-04-25T11:35:13.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.578+0000 7fe1973d5640 1 -- 172.21.15.90:0/3328493047 shutdown_connections 2024-04-25T11:35:13.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.578+0000 7fe1973d5640 1 --2- 172.21.15.90:0/3328493047 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1901031c0 0x7fe19010f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.578+0000 7fe1973d5640 1 --2- 172.21.15.90:0/3328493047 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe190108f00 0x7fe190102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.578+0000 7fe1973d5640 1 --2- 172.21.15.90:0/3328493047 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe190108a60 0x7fe190102610 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.578+0000 7fe1973d5640 1 -- 172.21.15.90:0/3328493047 >> 172.21.15.90:0/3328493047 conn(0x7fe1900781b0 msgr2=0x7fe1900785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:13.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.578+0000 7fe1973d5640 1 -- 172.21.15.90:0/3328493047 shutdown_connections 2024-04-25T11:35:13.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.578+0000 7fe1973d5640 1 -- 172.21.15.90:0/3328493047 wait complete. 2024-04-25T11:35:13.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.579+0000 7fe1973d5640 1 Processor -- start 2024-04-25T11:35:13.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.579+0000 7fe1973d5640 1 -- start start 2024-04-25T11:35:13.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.579+0000 7fe1973d5640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe1901031c0 0x7fe190198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.579+0000 7fe1973d5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe190108a60 0x7fe190198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.579+0000 7fe1973d5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe190108f00 0x7fe19019cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.579+0000 7fe1973d5640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe190111b60 con 0x7fe190108f00 2024-04-25T11:35:13.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.579+0000 7fe1973d5640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe1901119e0 con 0x7fe1901031c0 2024-04-25T11:35:13.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.579+0000 7fe1973d5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe190111ce0 con 0x7fe190108a60 2024-04-25T11:35:13.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.579+0000 7fe195972640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe190108f00 0x7fe19019cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.579+0000 7fe195972640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe190108f00 0x7fe19019cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33902/0 (socket says 172.21.15.90:33902) 2024-04-25T11:35:13.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.579+0000 7fe195171640 1 --2- 172.21.15.90:0/5576330 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe1901031c0 0x7fe190198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.579+0000 7fe195972640 1 -- 172.21.15.90:0/5576330 learned_addr learned my addr 172.21.15.90:0/5576330 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:13.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.579+0000 7fe194970640 1 --2- 172.21.15.90:0/5576330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe190108a60 0x7fe190198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.579+0000 7fe195972640 1 -- 172.21.15.90:0/5576330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe190108a60 msgr2=0x7fe190198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:13.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.579+0000 7fe195972640 1 --2- 172.21.15.90:0/5576330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe190108a60 0x7fe190198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.579+0000 7fe195972640 1 -- 172.21.15.90:0/5576330 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe1901031c0 msgr2=0x7fe190198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:13.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.579+0000 7fe195972640 1 --2- 172.21.15.90:0/5576330 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe1901031c0 0x7fe190198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.579+0000 7fe195972640 1 -- 172.21.15.90:0/5576330 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe19019d3e0 con 0x7fe190108f00 2024-04-25T11:35:13.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.580+0000 7fe195972640 1 --2- 172.21.15.90:0/5576330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe190108f00 0x7fe19019cda0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fe18000e970 tx=0x7fe18000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:13.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.580+0000 7fe18e7fc640 1 -- 172.21.15.90:0/5576330 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18000ccc0 con 0x7fe190108f00 2024-04-25T11:35:13.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.580+0000 7fe18e7fc640 1 -- 172.21.15.90:0/5576330 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe18000ce60 con 0x7fe190108f00 2024-04-25T11:35:13.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.580+0000 7fe1973d5640 1 -- 172.21.15.90:0/5576330 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe19019d6d0 con 0x7fe190108f00 2024-04-25T11:35:13.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.580+0000 7fe18e7fc640 1 -- 172.21.15.90:0/5576330 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe180010430 con 0x7fe190108f00 2024-04-25T11:35:13.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.580+0000 7fe1973d5640 1 -- 172.21.15.90:0/5576330 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1901a5050 con 0x7fe190108f00 2024-04-25T11:35:13.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.581+0000 7fe1973d5640 1 -- 172.21.15.90:0/5576330 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe19006a110 con 0x7fe190108f00 2024-04-25T11:35:13.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.582+0000 7fe18e7fc640 1 -- 172.21.15.90:0/5576330 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 23) ==== 147696+0+0 (secure 0 0 0) 0x7fe180004f70 con 0x7fe190108f00 2024-04-25T11:35:13.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.583+0000 7fe18e7fc640 1 --2- 172.21.15.90:0/5576330 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe1780afce0 0x7fe1780b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.583+0000 7fe195171640 1 --2- 172.21.15.90:0/5576330 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe1780afce0 0x7fe1780b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.583+0000 7fe18e7fc640 1 -- 172.21.15.90:0/5576330 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7fe1800df600 con 0x7fe190108f00 2024-04-25T11:35:13.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.583+0000 7fe195171640 1 --2- 172.21.15.90:0/5576330 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe1780afce0 0x7fe1780b21f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe18802cd80 tx=0x7fe188031f10 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:13.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.588+0000 7fe18e7fc640 1 -- 172.21.15.90:0/5576330 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe1800a7d30 con 0x7fe190108f00 2024-04-25T11:35:13.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: Updating smithi090:/etc/ceph/ceph.client.admin.keyring 2024-04-25T11:35:13.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: Updating smithi138:/etc/ceph/ceph.client.admin.keyring 2024-04-25T11:35:13.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-25T11:35:13.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:35:13.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: Updating smithi090:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T11:35:13.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: Updating smithi138:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T11:35:13.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: Updating smithi156:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T11:35:13.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2835368353' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:35:13.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-04-25T11:35:13.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: scrub queued for path: / 2024-04-25T11:35:13.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:35:13.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:35:13.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:35:13.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:35:13.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:35:13.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:35:13.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/512053787' entity='client.admin' 2024-04-25T11:35:13.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:35:13.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:35:13.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:35:13.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2655933904' entity='client.admin' 2024-04-25T11:35:13.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:35:13.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:35:13.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:35:13.728 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:13 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.791+0000 7fe1973d5640 1 -- 172.21.15.90:0/5576330 --> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fe190106fe0 con 0x7fe1780afce0 2024-04-25T11:35:13.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: Updating smithi090:/etc/ceph/ceph.client.admin.keyring 2024-04-25T11:35:13.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: Updating smithi138:/etc/ceph/ceph.client.admin.keyring 2024-04-25T11:35:13.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-25T11:35:13.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:35:13.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: Updating smithi090:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T11:35:13.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: Updating smithi138:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T11:35:13.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: Updating smithi156:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T11:35:13.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2835368353' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:35:13.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-04-25T11:35:13.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: scrub queued for path: / 2024-04-25T11:35:13.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:35:13.803 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:35:13.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:35:13.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:35:13.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:35:13.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:35:13.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/512053787' entity='client.admin' 2024-04-25T11:35:13.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:35:13.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:35:13.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:35:13.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2655933904' entity='client.admin' 2024-04-25T11:35:13.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:35:13.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:35:13.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:35:13.804 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:13 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.822+0000 7fe18e7fc640 1 -- 172.21.15.90:0/5576330 <== mgr.15366 v2:172.21.15.90:6800/4078377640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fe188033b50 con 0x7fe1780afce0 2024-04-25T11:35:13.809 INFO:teuthology.orchestra.run.smithi090.stdout:Schedule set for path / 2024-04-25T11:35:13.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.824+0000 7fe1973d5640 1 -- 172.21.15.90:0/5576330 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe1780afce0 msgr2=0x7fe1780b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:13.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.824+0000 7fe1973d5640 1 --2- 172.21.15.90:0/5576330 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe1780afce0 0x7fe1780b21f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe18802cd80 tx=0x7fe188031f10 comp rx=0 tx=0).stop 2024-04-25T11:35:13.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.824+0000 7fe1973d5640 1 -- 172.21.15.90:0/5576330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe190108f00 msgr2=0x7fe19019cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:13.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.824+0000 7fe1973d5640 1 --2- 172.21.15.90:0/5576330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe190108f00 0x7fe19019cda0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fe18000e970 tx=0x7fe18000ed70 comp rx=0 tx=0).stop 2024-04-25T11:35:13.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.824+0000 7fe1973d5640 1 -- 172.21.15.90:0/5576330 shutdown_connections 2024-04-25T11:35:13.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.824+0000 7fe1973d5640 1 --2- 172.21.15.90:0/5576330 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe1780afce0 0x7fe1780b21f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.824+0000 7fe1973d5640 1 --2- 172.21.15.90:0/5576330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe190108f00 0x7fe19019cda0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.824+0000 7fe1973d5640 1 --2- 172.21.15.90:0/5576330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe190108a60 0x7fe190198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.824+0000 7fe1973d5640 1 --2- 172.21.15.90:0/5576330 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe1901031c0 0x7fe190198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.824+0000 7fe1973d5640 1 -- 172.21.15.90:0/5576330 >> 172.21.15.90:0/5576330 conn(0x7fe1900781b0 msgr2=0x7fe190104140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:13.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.824+0000 7fe1973d5640 1 -- 172.21.15.90:0/5576330 shutdown_connections 2024-04-25T11:35:13.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.824+0000 7fe1973d5640 1 -- 172.21.15.90:0/5576330 wait complete. 2024-04-25T11:35:13.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-25T11:35:13.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: Updating smithi090:/etc/ceph/ceph.client.admin.keyring 2024-04-25T11:35:13.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: Updating smithi138:/etc/ceph/ceph.client.admin.keyring 2024-04-25T11:35:13.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-25T11:35:13.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:35:13.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: Updating smithi090:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T11:35:13.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: Updating smithi138:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T11:35:13.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: Updating smithi156:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T11:35:13.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2835368353' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:35:13.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-04-25T11:35:13.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: scrub queued for path: / 2024-04-25T11:35:13.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:35:13.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:35:13.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:35:13.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:35:13.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:35:13.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:35:13.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/512053787' entity='client.admin' 2024-04-25T11:35:13.839 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:35:13.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:35:13.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:35:13.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2655933904' entity='client.admin' 2024-04-25T11:35:13.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:35:13.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:35:13.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:35:13.840 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:13 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:35:13.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.946+0000 7f5d8e3a8640 1 -- 172.21.15.90:0/3874836934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d88109b60 msgr2=0x7f5d88103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:13.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.946+0000 7f5d8e3a8640 1 --2- 172.21.15.90:0/3874836934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d88109b60 0x7f5d88103700 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f5d78009900 tx=0x7f5d7802a410 comp rx=0 tx=0).stop 2024-04-25T11:35:13.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.946+0000 7f5d8e3a8640 1 -- 172.21.15.90:0/3874836934 shutdown_connections 2024-04-25T11:35:13.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.946+0000 7f5d8e3a8640 1 --2- 172.21.15.90:0/3874836934 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d881042b0 0x7f5d88110680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.946+0000 7f5d8e3a8640 1 --2- 172.21.15.90:0/3874836934 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d8810a000 0x7f5d88103c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.946+0000 7f5d8e3a8640 1 --2- 172.21.15.90:0/3874836934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d88109b60 0x7f5d88103700 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.946+0000 7f5d8e3a8640 1 -- 172.21.15.90:0/3874836934 >> 172.21.15.90:0/3874836934 conn(0x7f5d88079170 msgr2=0x7f5d880795b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:13.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.946+0000 7f5d8e3a8640 1 -- 172.21.15.90:0/3874836934 shutdown_connections 2024-04-25T11:35:13.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.946+0000 7f5d8e3a8640 1 -- 172.21.15.90:0/3874836934 wait complete. 2024-04-25T11:35:13.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.946+0000 7f5d8e3a8640 1 Processor -- start 2024-04-25T11:35:13.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.946+0000 7f5d8e3a8640 1 -- start start 2024-04-25T11:35:13.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.947+0000 7f5d8e3a8640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d881042b0 0x7f5d8819a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.947+0000 7f5d8e3a8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d88109b60 0x7f5d8819ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.947+0000 7f5d8e3a8640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d8810a000 0x7f5d8819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.947+0000 7f5d8e3a8640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5d88113d50 con 0x7f5d881042b0 2024-04-25T11:35:13.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.947+0000 7f5d8e3a8640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5d88113bd0 con 0x7f5d8810a000 2024-04-25T11:35:13.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.947+0000 7f5d8e3a8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5d88113ed0 con 0x7f5d88109b60 2024-04-25T11:35:13.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.947+0000 7f5d87fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d881042b0 0x7f5d8819a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.947+0000 7f5d87fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d881042b0 0x7f5d8819a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33916/0 (socket says 172.21.15.90:33916) 2024-04-25T11:35:13.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.947+0000 7f5d87fff640 1 -- 172.21.15.90:0/2031226218 learned_addr learned my addr 172.21.15.90:0/2031226218 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:13.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.947+0000 7f5d8c945640 1 --2- 172.21.15.90:0/2031226218 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d8810a000 0x7f5d8819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.947+0000 7f5d877fe640 1 --2- 172.21.15.90:0/2031226218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d88109b60 0x7f5d8819ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.947+0000 7f5d87fff640 1 -- 172.21.15.90:0/2031226218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d88109b60 msgr2=0x7f5d8819ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:13.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.947+0000 7f5d87fff640 1 --2- 172.21.15.90:0/2031226218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d88109b60 0x7f5d8819ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.947+0000 7f5d87fff640 1 -- 172.21.15.90:0/2031226218 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d8810a000 msgr2=0x7f5d8819eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:13.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.947+0000 7f5d87fff640 1 --2- 172.21.15.90:0/2031226218 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d8810a000 0x7f5d8819eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:13.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.947+0000 7f5d87fff640 1 -- 172.21.15.90:0/2031226218 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d8819f630 con 0x7f5d881042b0 2024-04-25T11:35:13.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.947+0000 7f5d87fff640 1 --2- 172.21.15.90:0/2031226218 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d881042b0 0x7f5d8819a7d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f5d78006010 tx=0x7f5d78004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:13.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.947+0000 7f5d857fa640 1 -- 172.21.15.90:0/2031226218 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d7803f030 con 0x7f5d881042b0 2024-04-25T11:35:13.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.947+0000 7f5d857fa640 1 -- 172.21.15.90:0/2031226218 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d78003aa0 con 0x7f5d881042b0 2024-04-25T11:35:13.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.947+0000 7f5d8e3a8640 1 -- 172.21.15.90:0/2031226218 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d8819f8c0 con 0x7f5d881042b0 2024-04-25T11:35:13.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.948+0000 7f5d857fa640 1 -- 172.21.15.90:0/2031226218 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d78035070 con 0x7f5d881042b0 2024-04-25T11:35:13.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.948+0000 7f5d8e3a8640 1 -- 172.21.15.90:0/2031226218 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d8819fdb0 con 0x7f5d881042b0 2024-04-25T11:35:13.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.948+0000 7f5d8e3a8640 1 -- 172.21.15.90:0/2031226218 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d88050740 con 0x7f5d881042b0 2024-04-25T11:35:13.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.950+0000 7f5d857fa640 1 -- 172.21.15.90:0/2031226218 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 23) ==== 147696+0+0 (secure 0 0 0) 0x7f5d78042030 con 0x7f5d881042b0 2024-04-25T11:35:13.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.950+0000 7f5d857fa640 1 --2- 172.21.15.90:0/2031226218 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5d700afce0 0x7f5d700b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:13.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.951+0000 7f5d877fe640 1 --2- 172.21.15.90:0/2031226218 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5d700afce0 0x7f5d700b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:13.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.951+0000 7f5d857fa640 1 -- 172.21.15.90:0/2031226218 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f5d780ffa20 con 0x7f5d881042b0 2024-04-25T11:35:13.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.951+0000 7f5d877fe640 1 --2- 172.21.15.90:0/2031226218 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5d700afce0 0x7f5d700b21f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5d88101800 tx=0x7f5d7c005f90 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:13.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:13.954+0000 7f5d857fa640 1 -- 172.21.15.90:0/2031226218 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d780c8150 con 0x7f5d881042b0 2024-04-25T11:35:14.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.153+0000 7f5d8e3a8640 1 -- 172.21.15.90:0/2031226218 --> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f5d88101fb0 con 0x7f5d700afce0 2024-04-25T11:35:14.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.180+0000 7f5d857fa640 1 -- 172.21.15.90:0/2031226218 <== mgr.15366 v2:172.21.15.90:6800/4078377640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f5d88101fb0 con 0x7f5d700afce0 2024-04-25T11:35:14.168 INFO:teuthology.orchestra.run.smithi090.stdout:Retention added to path / 2024-04-25T11:35:14.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.182+0000 7f5d8e3a8640 1 -- 172.21.15.90:0/2031226218 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5d700afce0 msgr2=0x7f5d700b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:14.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.182+0000 7f5d8e3a8640 1 --2- 172.21.15.90:0/2031226218 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5d700afce0 0x7f5d700b21f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5d88101800 tx=0x7f5d7c005f90 comp rx=0 tx=0).stop 2024-04-25T11:35:14.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.182+0000 7f5d8e3a8640 1 -- 172.21.15.90:0/2031226218 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d881042b0 msgr2=0x7f5d8819a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:14.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.182+0000 7f5d8e3a8640 1 --2- 172.21.15.90:0/2031226218 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d881042b0 0x7f5d8819a7d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f5d78006010 tx=0x7f5d78004060 comp rx=0 tx=0).stop 2024-04-25T11:35:14.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.182+0000 7f5d8e3a8640 1 -- 172.21.15.90:0/2031226218 shutdown_connections 2024-04-25T11:35:14.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.182+0000 7f5d8e3a8640 1 --2- 172.21.15.90:0/2031226218 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5d700afce0 0x7f5d700b21f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:14.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.182+0000 7f5d8e3a8640 1 --2- 172.21.15.90:0/2031226218 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d8810a000 0x7f5d8819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:14.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.182+0000 7f5d8e3a8640 1 --2- 172.21.15.90:0/2031226218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d88109b60 0x7f5d8819ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:14.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.182+0000 7f5d8e3a8640 1 --2- 172.21.15.90:0/2031226218 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d881042b0 0x7f5d8819a7d0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:14.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.182+0000 7f5d8e3a8640 1 -- 172.21.15.90:0/2031226218 >> 172.21.15.90:0/2031226218 conn(0x7f5d88079170 msgr2=0x7f5d881018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:14.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.183+0000 7f5d8e3a8640 1 -- 172.21.15.90:0/2031226218 shutdown_connections 2024-04-25T11:35:14.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.183+0000 7f5d8e3a8640 1 -- 172.21.15.90:0/2031226218 wait complete. 2024-04-25T11:35:14.181 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-25T11:35:14.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.303+0000 7f9cc54d8640 1 -- 172.21.15.90:0/2632675378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9cc0108f00 msgr2=0x7f9cc0102b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:14.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.303+0000 7f9cc54d8640 1 --2- 172.21.15.90:0/2632675378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9cc0108f00 0x7f9cc0102b50 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8009900 tx=0x7f9ca802a410 comp rx=0 tx=0).stop 2024-04-25T11:35:14.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.303+0000 7f9cc54d8640 1 -- 172.21.15.90:0/2632675378 shutdown_connections 2024-04-25T11:35:14.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.303+0000 7f9cc54d8640 1 --2- 172.21.15.90:0/2632675378 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9cc01031c0 0x7f9cc010f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:14.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.303+0000 7f9cc54d8640 1 --2- 172.21.15.90:0/2632675378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9cc0108f00 0x7f9cc0102b50 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:14.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.303+0000 7f9cc54d8640 1 --2- 172.21.15.90:0/2632675378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cc0108a60 0x7f9cc0102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:14.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.303+0000 7f9cc54d8640 1 -- 172.21.15.90:0/2632675378 >> 172.21.15.90:0/2632675378 conn(0x7f9cc00781b0 msgr2=0x7f9cc00785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:14.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.303+0000 7f9cc54d8640 1 -- 172.21.15.90:0/2632675378 shutdown_connections 2024-04-25T11:35:14.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.303+0000 7f9cc54d8640 1 -- 172.21.15.90:0/2632675378 wait complete. 2024-04-25T11:35:14.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.304+0000 7f9cc54d8640 1 Processor -- start 2024-04-25T11:35:14.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.304+0000 7f9cc54d8640 1 -- start start 2024-04-25T11:35:14.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.304+0000 7f9cc54d8640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9cc01031c0 0x7f9cc0199700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:14.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.304+0000 7f9cc54d8640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9cc0108a60 0x7f9cc0199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:14.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.304+0000 7f9cc54d8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cc0108f00 0x7f9cc019df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:14.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.304+0000 7f9cc54d8640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc0112c80 con 0x7f9cc01031c0 2024-04-25T11:35:14.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.304+0000 7f9cc54d8640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc0112b00 con 0x7f9cc0108a60 2024-04-25T11:35:14.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.304+0000 7f9cc54d8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc0112e00 con 0x7f9cc0108f00 2024-04-25T11:35:14.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.304+0000 7f9cbeffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9cc01031c0 0x7f9cc0199700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:14.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.304+0000 7f9cbeffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9cc01031c0 0x7f9cc0199700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33940/0 (socket says 172.21.15.90:33940) 2024-04-25T11:35:14.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.304+0000 7f9cbeffd640 1 -- 172.21.15.90:0/4116645304 learned_addr learned my addr 172.21.15.90:0/4116645304 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:14.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.304+0000 7f9cb7fff640 1 --2- 172.21.15.90:0/4116645304 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9cc0108a60 0x7f9cc0199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:14.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.304+0000 7f9cbf7fe640 1 --2- 172.21.15.90:0/4116645304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cc0108f00 0x7f9cc019df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:14.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.304+0000 7f9cbeffd640 1 -- 172.21.15.90:0/4116645304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cc0108f00 msgr2=0x7f9cc019df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:14.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.304+0000 7f9cbeffd640 1 --2- 172.21.15.90:0/4116645304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cc0108f00 0x7f9cc019df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:14.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.304+0000 7f9cbeffd640 1 -- 172.21.15.90:0/4116645304 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9cc0108a60 msgr2=0x7f9cc0199c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:14.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.304+0000 7f9cbeffd640 1 --2- 172.21.15.90:0/4116645304 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9cc0108a60 0x7f9cc0199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:14.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.304+0000 7f9cbeffd640 1 -- 172.21.15.90:0/4116645304 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc019e560 con 0x7f9cc01031c0 2024-04-25T11:35:14.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.304+0000 7f9cbeffd640 1 --2- 172.21.15.90:0/4116645304 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9cc01031c0 0x7f9cc0199700 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f9cb000e9c0 tx=0x7f9cb000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:14.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.305+0000 7f9cbcff9640 1 -- 172.21.15.90:0/4116645304 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb000ccc0 con 0x7f9cc01031c0 2024-04-25T11:35:14.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.305+0000 7f9cc54d8640 1 -- 172.21.15.90:0/4116645304 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cc019e850 con 0x7f9cc01031c0 2024-04-25T11:35:14.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.305+0000 7f9cbcff9640 1 -- 172.21.15.90:0/4116645304 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9cb00054d0 con 0x7f9cc01031c0 2024-04-25T11:35:14.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.305+0000 7f9cc54d8640 1 -- 172.21.15.90:0/4116645304 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cc019ed20 con 0x7f9cc01031c0 2024-04-25T11:35:14.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.305+0000 7f9cbcff9640 1 -- 172.21.15.90:0/4116645304 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb0018430 con 0x7f9cc01031c0 2024-04-25T11:35:14.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.306+0000 7f9cc54d8640 1 -- 172.21.15.90:0/4116645304 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cc006a110 con 0x7f9cc01031c0 2024-04-25T11:35:14.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.307+0000 7f9cbcff9640 1 -- 172.21.15.90:0/4116645304 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 23) ==== 147696+0+0 (secure 0 0 0) 0x7f9cb0004fa0 con 0x7f9cc01031c0 2024-04-25T11:35:14.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.308+0000 7f9cbcff9640 1 --2- 172.21.15.90:0/4116645304 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9cac0afce0 0x7f9cac0b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:14.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.308+0000 7f9cb7fff640 1 --2- 172.21.15.90:0/4116645304 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9cac0afce0 0x7f9cac0b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:14.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.308+0000 7f9cbcff9640 1 -- 172.21.15.90:0/4116645304 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f9cb00dfc20 con 0x7f9cc01031c0 2024-04-25T11:35:14.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.308+0000 7f9cb7fff640 1 --2- 172.21.15.90:0/4116645304 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9cac0afce0 0x7f9cac0b21f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8009230 tx=0x7f9ca8031400 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:14.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.312+0000 7f9cbcff9640 1 -- 172.21.15.90:0/4116645304 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9cb00a8330 con 0x7f9cc01031c0 2024-04-25T11:35:14.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.510+0000 7f9cc54d8640 1 -- 172.21.15.90:0/4116645304 --> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f9cc0106fe0 con 0x7f9cac0afce0 2024-04-25T11:35:14.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.518+0000 7f9cbcff9640 1 -- 172.21.15.90:0/4116645304 <== mgr.15366 v2:172.21.15.90:6800/4078377640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f9ca8032d50 con 0x7f9cac0afce0 2024-04-25T11:35:14.506 INFO:teuthology.orchestra.run.smithi090.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-25T00:00:00", "created": "2024-04-25T11:35:13", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-25T11:35:14.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.520+0000 7f9cc54d8640 1 -- 172.21.15.90:0/4116645304 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9cac0afce0 msgr2=0x7f9cac0b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:14.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.520+0000 7f9cc54d8640 1 --2- 172.21.15.90:0/4116645304 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9cac0afce0 0x7f9cac0b21f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8009230 tx=0x7f9ca8031400 comp rx=0 tx=0).stop 2024-04-25T11:35:14.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.520+0000 7f9cc54d8640 1 -- 172.21.15.90:0/4116645304 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9cc01031c0 msgr2=0x7f9cc0199700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:14.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.520+0000 7f9cc54d8640 1 --2- 172.21.15.90:0/4116645304 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9cc01031c0 0x7f9cc0199700 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f9cb000e9c0 tx=0x7f9cb000ee80 comp rx=0 tx=0).stop 2024-04-25T11:35:14.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.520+0000 7f9cc54d8640 1 -- 172.21.15.90:0/4116645304 shutdown_connections 2024-04-25T11:35:14.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.520+0000 7f9cc54d8640 1 --2- 172.21.15.90:0/4116645304 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9cac0afce0 0x7f9cac0b21f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:14.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.520+0000 7f9cc54d8640 1 --2- 172.21.15.90:0/4116645304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cc0108f00 0x7f9cc019df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:14.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.520+0000 7f9cc54d8640 1 --2- 172.21.15.90:0/4116645304 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9cc0108a60 0x7f9cc0199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:14.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.520+0000 7f9cc54d8640 1 --2- 172.21.15.90:0/4116645304 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9cc01031c0 0x7f9cc0199700 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:14.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.520+0000 7f9cc54d8640 1 -- 172.21.15.90:0/4116645304 >> 172.21.15.90:0/4116645304 conn(0x7f9cc00781b0 msgr2=0x7f9cc0104140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:14.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.521+0000 7f9cc54d8640 1 -- 172.21.15.90:0/4116645304 shutdown_connections 2024-04-25T11:35:14.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.521+0000 7f9cc54d8640 1 -- 172.21.15.90:0/4116645304 wait complete. 2024-04-25T11:35:14.519 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-25T11:35:14.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.640+0000 7f9901627640 1 -- 172.21.15.90:0/1963274924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98fc1020a0 msgr2=0x7f98fc1024b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:14.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.640+0000 7f9901627640 1 --2- 172.21.15.90:0/1963274924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98fc1020a0 0x7f98fc1024b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f98f0009900 tx=0x7f98f002a410 comp rx=0 tx=0).stop 2024-04-25T11:35:14.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.641+0000 7f9901627640 1 -- 172.21.15.90:0/1963274924 shutdown_connections 2024-04-25T11:35:14.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.641+0000 7f9901627640 1 --2- 172.21.15.90:0/1963274924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98fc10e5a0 0x7f98fc1109c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:14.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.641+0000 7f9901627640 1 --2- 172.21.15.90:0/1963274924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98fc1020a0 0x7f98fc1024b0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:14.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.641+0000 7f9901627640 1 --2- 172.21.15.90:0/1963274924 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f98fc109bf0 0x7f98fc101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:14.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.641+0000 7f9901627640 1 -- 172.21.15.90:0/1963274924 >> 172.21.15.90:0/1963274924 conn(0x7f98fc0778b0 msgr2=0x7f98fc077cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:14.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.641+0000 7f9901627640 1 -- 172.21.15.90:0/1963274924 shutdown_connections 2024-04-25T11:35:14.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.641+0000 7f9901627640 1 -- 172.21.15.90:0/1963274924 wait complete. 2024-04-25T11:35:14.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.641+0000 7f9901627640 1 Processor -- start 2024-04-25T11:35:14.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.641+0000 7f9901627640 1 -- start start 2024-04-25T11:35:14.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.642+0000 7f9901627640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98fc1020a0 0x7f98fc19a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:14.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.642+0000 7f9901627640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f98fc109bf0 0x7f98fc19acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:14.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.642+0000 7f9901627640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98fc10e5a0 0x7f98fc19ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:14.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.642+0000 7f9901627640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f98fc113d60 con 0x7f98fc1020a0 2024-04-25T11:35:14.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.642+0000 7f9901627640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f98fc113be0 con 0x7f98fc109bf0 2024-04-25T11:35:14.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.642+0000 7f9901627640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f98fc113ee0 con 0x7f98fc10e5a0 2024-04-25T11:35:14.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.642+0000 7f98faffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98fc1020a0 0x7f98fc19a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:14.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.642+0000 7f98faffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98fc1020a0 0x7f98fc19a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33964/0 (socket says 172.21.15.90:33964) 2024-04-25T11:35:14.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.642+0000 7f98faffd640 1 -- 172.21.15.90:0/2040960197 learned_addr learned my addr 172.21.15.90:0/2040960197 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:14.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.642+0000 7f98fa7fc640 1 --2- 172.21.15.90:0/2040960197 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f98fc109bf0 0x7f98fc19acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:14.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.642+0000 7f98fb7fe640 1 --2- 172.21.15.90:0/2040960197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98fc10e5a0 0x7f98fc19ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:14.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.642+0000 7f98faffd640 1 -- 172.21.15.90:0/2040960197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98fc10e5a0 msgr2=0x7f98fc19ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:14.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.642+0000 7f98faffd640 1 --2- 172.21.15.90:0/2040960197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98fc10e5a0 0x7f98fc19ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:14.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.642+0000 7f98faffd640 1 -- 172.21.15.90:0/2040960197 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f98fc109bf0 msgr2=0x7f98fc19acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:14.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.642+0000 7f98faffd640 1 --2- 172.21.15.90:0/2040960197 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f98fc109bf0 0x7f98fc19acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:14.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.642+0000 7f98faffd640 1 -- 172.21.15.90:0/2040960197 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98fc19f5c0 con 0x7f98fc1020a0 2024-04-25T11:35:14.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.642+0000 7f98fb7fe640 1 --2- 172.21.15.90:0/2040960197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98fc10e5a0 0x7f98fc19ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:35:14.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.642+0000 7f98faffd640 1 --2- 172.21.15.90:0/2040960197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98fc1020a0 0x7f98fc19a760 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f98ec00ead0 tx=0x7f98ec00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:14.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.642+0000 7f98e3fff640 1 -- 172.21.15.90:0/2040960197 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec00ce40 con 0x7f98fc1020a0 2024-04-25T11:35:14.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.642+0000 7f98e3fff640 1 -- 172.21.15.90:0/2040960197 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98ec011920 con 0x7f98fc1020a0 2024-04-25T11:35:14.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.642+0000 7f9901627640 1 -- 172.21.15.90:0/2040960197 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98fc19f8b0 con 0x7f98fc1020a0 2024-04-25T11:35:14.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.642+0000 7f98e3fff640 1 -- 172.21.15.90:0/2040960197 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec01aae0 con 0x7f98fc1020a0 2024-04-25T11:35:14.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.643+0000 7f9901627640 1 -- 172.21.15.90:0/2040960197 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98fc1a7230 con 0x7f98fc1020a0 2024-04-25T11:35:14.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.643+0000 7f9901627640 1 -- 172.21.15.90:0/2040960197 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98fc050740 con 0x7f98fc1020a0 2024-04-25T11:35:14.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.645+0000 7f98e3fff640 1 -- 172.21.15.90:0/2040960197 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 23) ==== 147696+0+0 (secure 0 0 0) 0x7f98ec01ac80 con 0x7f98fc1020a0 2024-04-25T11:35:14.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.646+0000 7f98e3fff640 1 --2- 172.21.15.90:0/2040960197 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f98e40afce0 0x7f98e40b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:14.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.646+0000 7f98fa7fc640 1 --2- 172.21.15.90:0/2040960197 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f98e40afce0 0x7f98e40b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:14.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.646+0000 7f98e3fff640 1 -- 172.21.15.90:0/2040960197 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f98ec0df9b0 con 0x7f98fc1020a0 2024-04-25T11:35:14.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.646+0000 7f98fa7fc640 1 --2- 172.21.15.90:0/2040960197 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f98e40afce0 0x7f98e40b21f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f98f0000c00 tx=0x7f98f0031400 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:14.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.650+0000 7f98e3fff640 1 -- 172.21.15.90:0/2040960197 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f98ec055ff0 con 0x7f98fc1020a0 2024-04-25T11:35:14.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:14 smithi138 ceph-mon[32328]: from='client.15576 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:35:14.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:14 smithi156 ceph-mon[33045]: from='client.15576 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:35:14.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.851+0000 7f9901627640 1 -- 172.21.15.90:0/2040960197 --> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f98fc062fe0 con 0x7f98e40afce0 2024-04-25T11:35:14.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.859+0000 7f98e3fff640 1 -- 172.21.15.90:0/2040960197 <== mgr.15366 v2:172.21.15.90:6800/4078377640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f98fc062fe0 con 0x7f98e40afce0 2024-04-25T11:35:14.846 INFO:teuthology.orchestra.run.smithi090.stdout:/ 1m 6m3h 2024-04-25T11:35:14.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.861+0000 7f9901627640 1 -- 172.21.15.90:0/2040960197 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f98e40afce0 msgr2=0x7f98e40b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:14.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.861+0000 7f9901627640 1 --2- 172.21.15.90:0/2040960197 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f98e40afce0 0x7f98e40b21f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f98f0000c00 tx=0x7f98f0031400 comp rx=0 tx=0).stop 2024-04-25T11:35:14.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.861+0000 7f9901627640 1 -- 172.21.15.90:0/2040960197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98fc1020a0 msgr2=0x7f98fc19a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:14.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.861+0000 7f9901627640 1 --2- 172.21.15.90:0/2040960197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98fc1020a0 0x7f98fc19a760 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f98ec00ead0 tx=0x7f98ec00ef90 comp rx=0 tx=0).stop 2024-04-25T11:35:14.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.861+0000 7f9901627640 1 -- 172.21.15.90:0/2040960197 shutdown_connections 2024-04-25T11:35:14.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.861+0000 7f9901627640 1 --2- 172.21.15.90:0/2040960197 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f98e40afce0 0x7f98e40b21f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:14.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.861+0000 7f9901627640 1 --2- 172.21.15.90:0/2040960197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98fc10e5a0 0x7f98fc19ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:14.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.861+0000 7f9901627640 1 --2- 172.21.15.90:0/2040960197 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f98fc109bf0 0x7f98fc19acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:14.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.861+0000 7f9901627640 1 --2- 172.21.15.90:0/2040960197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98fc1020a0 0x7f98fc19a760 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:14.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.861+0000 7f9901627640 1 -- 172.21.15.90:0/2040960197 >> 172.21.15.90:0/2040960197 conn(0x7f98fc0778b0 msgr2=0x7f98fc10a560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:14.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.861+0000 7f9901627640 1 -- 172.21.15.90:0/2040960197 shutdown_connections 2024-04-25T11:35:14.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:14.861+0000 7f9901627640 1 -- 172.21.15.90:0/2040960197 wait complete. 2024-04-25T11:35:14.858 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:14 smithi090 ceph-mon[29575]: from='client.15576 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:35:14.860 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-25T11:35:14.884 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-25T11:35:14.895 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-25T11:35:14.915 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T11:35:14.915 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-25T11:35:14.927 INFO:teuthology.run_tasks:Running task workunit... 2024-04-25T11:35:15.014 INFO:tasks.workunit:Pulling workunits from ref 994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T11:35:15.015 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-25T11:35:15.015 INFO:tasks.workunit:timeout=3h 2024-04-25T11:35:15.015 INFO:tasks.workunit:cleanup=True 2024-04-25T11:35:15.015 DEBUG:teuthology.orchestra.run.smithi090:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-25T11:35:15.031 INFO:teuthology.orchestra.run.smithi090.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-25T11:35:15.031 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-25T11:35:15.031 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 73h/115d Inode: 1099511628283 Links: 2 2024-04-25T11:35:15.031 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T11:35:15.031 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-25T11:35:15.031 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-04-25 11:33:16.714874637 +0000 2024-04-25T11:35:15.031 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-04-25 11:33:16.714874637 +0000 2024-04-25T11:35:15.031 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-04-25 11:35:05.574550407 +0000 2024-04-25T11:35:15.031 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-04-25 11:33:16.714874637 +0000 2024-04-25T11:35:15.031 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-25T11:35:15.032 DEBUG:teuthology.orchestra.run.smithi090:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-25T11:35:15.103 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T11:35:15.158 INFO:tasks.workunit.client.0.smithi090.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-25T11:35:15.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:15 smithi138 ceph-mon[32328]: from='client.15582 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:35:15.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:15 smithi138 ceph-mon[32328]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:35:15.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:15 smithi138 ceph-mon[32328]: from='client.15588 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:35:15.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:15 smithi138 ceph-mon[32328]: from='client.15594 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:35:15.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:15 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:35:15.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:15 smithi156 ceph-mon[33045]: from='client.15582 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:35:15.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:15 smithi156 ceph-mon[33045]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:35:15.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:15 smithi156 ceph-mon[33045]: from='client.15588 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:35:15.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:15 smithi156 ceph-mon[33045]: from='client.15594 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:35:15.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:15 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:35:15.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:15 smithi090 ceph-mon[29575]: from='client.15582 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:35:15.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:15 smithi090 ceph-mon[29575]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:35:15.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:15 smithi090 ceph-mon[29575]: from='client.15588 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:35:15.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:15 smithi090 ceph-mon[29575]: from='client.15594 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:35:15.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:15 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:35:17.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:17 smithi138 ceph-mon[32328]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 131 KiB/s wr, 33 op/s 2024-04-25T11:35:17.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:17 smithi156 ceph-mon[33045]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 131 KiB/s wr, 33 op/s 2024-04-25T11:35:17.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:17 smithi090 ceph-mon[29575]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 131 KiB/s wr, 33 op/s 2024-04-25T11:35:19.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:19 smithi090 ceph-mon[29575]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 102 KiB/s wr, 26 op/s 2024-04-25T11:35:20.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:19 smithi138 ceph-mon[32328]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 102 KiB/s wr, 26 op/s 2024-04-25T11:35:20.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:19 smithi156 ceph-mon[33045]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 102 KiB/s wr, 26 op/s 2024-04-25T11:35:21.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:21 smithi090 ceph-mon[29575]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 84 KiB/s wr, 29 op/s 2024-04-25T11:35:22.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:21 smithi138 ceph-mon[32328]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 84 KiB/s wr, 29 op/s 2024-04-25T11:35:22.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:21 smithi156 ceph-mon[33045]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 84 KiB/s wr, 29 op/s 2024-04-25T11:35:23.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:23 smithi090 ceph-mon[29575]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 27 op/s 2024-04-25T11:35:24.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:23 smithi138 ceph-mon[32328]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 27 op/s 2024-04-25T11:35:24.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:23 smithi156 ceph-mon[33045]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 27 op/s 2024-04-25T11:35:25.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:25 smithi090 ceph-mon[29575]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 77 KiB/s wr, 27 op/s 2024-04-25T11:35:26.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:25 smithi138 ceph-mon[32328]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 77 KiB/s wr, 27 op/s 2024-04-25T11:35:26.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:25 smithi156 ceph-mon[33045]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 77 KiB/s wr, 27 op/s 2024-04-25T11:35:27.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:27 smithi090 ceph-mon[29575]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 77 KiB/s wr, 31 op/s 2024-04-25T11:35:28.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:27 smithi138 ceph-mon[32328]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 77 KiB/s wr, 31 op/s 2024-04-25T11:35:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.085+0000 7f6130a4a640 1 -- 172.21.15.90:0/1181063536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f612c10e5a0 msgr2=0x7f612c1109c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.085+0000 7f6130a4a640 1 --2- 172.21.15.90:0/1181063536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f612c10e5a0 0x7f612c1109c0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f6124009900 tx=0x7f612402a410 comp rx=0 tx=0).stop 2024-04-25T11:35:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.085+0000 7f6130a4a640 1 -- 172.21.15.90:0/1181063536 shutdown_connections 2024-04-25T11:35:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.085+0000 7f6130a4a640 1 --2- 172.21.15.90:0/1181063536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f612c10e5a0 0x7f612c1109c0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.085+0000 7f6130a4a640 1 --2- 172.21.15.90:0/1181063536 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f612c1020a0 0x7f612c1024b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.085+0000 7f6130a4a640 1 --2- 172.21.15.90:0/1181063536 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f612c109bf0 0x7f612c101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.085+0000 7f6130a4a640 1 -- 172.21.15.90:0/1181063536 >> 172.21.15.90:0/1181063536 conn(0x7f612c0778b0 msgr2=0x7f612c077cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.085+0000 7f6130a4a640 1 -- 172.21.15.90:0/1181063536 shutdown_connections 2024-04-25T11:35:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.085+0000 7f6130a4a640 1 -- 172.21.15.90:0/1181063536 wait complete. 2024-04-25T11:35:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.086+0000 7f6130a4a640 1 Processor -- start 2024-04-25T11:35:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.086+0000 7f6130a4a640 1 -- start start 2024-04-25T11:35:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.086+0000 7f6130a4a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f612c1020a0 0x7f612c19a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.086+0000 7f6130a4a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f612c109bf0 0x7f612c19ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.086+0000 7f6130a4a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f612c10e5a0 0x7f612c19ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.086+0000 7f6130a4a640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f612c113d60 con 0x7f612c109bf0 2024-04-25T11:35:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.086+0000 7f6130a4a640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f612c113be0 con 0x7f612c10e5a0 2024-04-25T11:35:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.086+0000 7f6130a4a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f612c113ee0 con 0x7f612c1020a0 2024-04-25T11:35:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.086+0000 7f6129d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f612c109bf0 0x7f612c19ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.086+0000 7f6129d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f612c109bf0 0x7f612c19ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56580/0 (socket says 172.21.15.90:56580) 2024-04-25T11:35:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.086+0000 7f6129d9b640 1 -- 172.21.15.90:0/254376509 learned_addr learned my addr 172.21.15.90:0/254376509 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.086+0000 7f612ad9d640 1 --2- 172.21.15.90:0/254376509 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f612c10e5a0 0x7f612c19ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.087+0000 7f612a59c640 1 --2- 172.21.15.90:0/254376509 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f612c1020a0 0x7f612c19a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.087+0000 7f6129d9b640 1 -- 172.21.15.90:0/254376509 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f612c1020a0 msgr2=0x7f612c19a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.087+0000 7f6129d9b640 1 --2- 172.21.15.90:0/254376509 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f612c1020a0 0x7f612c19a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.087+0000 7f6129d9b640 1 -- 172.21.15.90:0/254376509 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f612c10e5a0 msgr2=0x7f612c19ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.087+0000 7f6129d9b640 1 --2- 172.21.15.90:0/254376509 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f612c10e5a0 0x7f612c19ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.087+0000 7f6129d9b640 1 -- 172.21.15.90:0/254376509 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f612c19f580 con 0x7f612c109bf0 2024-04-25T11:35:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.087+0000 7f6129d9b640 1 --2- 172.21.15.90:0/254376509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f612c109bf0 0x7f612c19ac80 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f611400d770 tx=0x7f611400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.087+0000 7f61237fe640 1 -- 172.21.15.90:0/254376509 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6114017070 con 0x7f612c109bf0 2024-04-25T11:35:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.087+0000 7f61237fe640 1 -- 172.21.15.90:0/254376509 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6114009c30 con 0x7f612c109bf0 2024-04-25T11:35:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.087+0000 7f6130a4a640 1 -- 172.21.15.90:0/254376509 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f612c19f870 con 0x7f612c109bf0 2024-04-25T11:35:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.087+0000 7f61237fe640 1 -- 172.21.15.90:0/254376509 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611400f620 con 0x7f612c109bf0 2024-04-25T11:35:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.087+0000 7f6130a4a640 1 -- 172.21.15.90:0/254376509 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f612c1a71f0 con 0x7f612c109bf0 2024-04-25T11:35:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.090+0000 7f61237fe640 1 -- 172.21.15.90:0/254376509 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 23) ==== 147696+0+0 (secure 0 0 0) 0x7f611401a070 con 0x7f612c109bf0 2024-04-25T11:35:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.090+0000 7f61379de740 1 Processor -- start 2024-04-25T11:35:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.090+0000 7f61379de740 1 -- start start 2024-04-25T11:35:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.090+0000 7f61379de740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9769b91b0 0x55f976d58af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.090+0000 7f61379de740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f9769902f0 0x55f976d59030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.090+0000 7f61379de740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769984d0 0x55f976d59570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.090+0000 7f61379de740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f976cbf460 con 0x55f9769984d0 2024-04-25T11:35:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.090+0000 7f61379de740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f976c79590 con 0x55f9769902f0 2024-04-25T11:35:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.090+0000 7f61379de740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f976cbb420 con 0x55f9769b91b0 2024-04-25T11:35:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.091+0000 7f6129d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769984d0 0x55f976d59570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.091+0000 7f6129d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769984d0 0x55f976d59570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56590/0 (socket says 172.21.15.90:56590) 2024-04-25T11:35:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.091+0000 7f6129d9b640 1 -- 172.21.15.90:0/2544933926 learned_addr learned my addr 172.21.15.90:0/2544933926 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.091+0000 7f612a59c640 1 --2- 172.21.15.90:0/2544933926 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f9769902f0 0x55f976d59030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.091+0000 7f612ad9d640 1 --2- 172.21.15.90:0/2544933926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9769b91b0 0x55f976d58af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.091+0000 7f61237fe640 1 --2- 172.21.15.90:0/254376509 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f61180afce0 0x7f61180b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.091+0000 7f6129d9b640 1 -- 172.21.15.90:0/2544933926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9769b91b0 msgr2=0x55f976d58af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.091+0000 7f6129d9b640 1 --2- 172.21.15.90:0/2544933926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9769b91b0 0x55f976d58af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.091+0000 7f6129d9b640 1 -- 172.21.15.90:0/2544933926 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f9769902f0 msgr2=0x55f976d59030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.091+0000 7f6129d9b640 1 --2- 172.21.15.90:0/2544933926 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f9769902f0 0x55f976d59030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.091+0000 7f6129d9b640 1 -- 172.21.15.90:0/2544933926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f976c860d0 con 0x55f9769984d0 2024-04-25T11:35:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.091+0000 7f612ad9d640 1 --2- 172.21.15.90:0/254376509 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f61180afce0 0x7f61180b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.091+0000 7f61237fe640 1 -- 172.21.15.90:0/254376509 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f61140dedd0 con 0x7f612c109bf0 2024-04-25T11:35:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.091+0000 7f6129d9b640 1 --2- 172.21.15.90:0/2544933926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769984d0 0x55f976d59570 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f61140dc400 tx=0x7f611406d220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f178752670c85a74 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.091+0000 7f6120ff9640 1 -- 172.21.15.90:0/2544933926 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6114021020 con 0x55f9769984d0 2024-04-25T11:35:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.091+0000 7f612ad9d640 1 --2- 172.21.15.90:0/254376509 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f61180afce0 0x7f61180b21f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6124031440 tx=0x7f6124009260 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.092+0000 7f6120ff9640 1 -- 172.21.15.90:0/2544933926 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61140a6910 con 0x55f9769984d0 2024-04-25T11:35:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.092+0000 7f6120ff9640 1 -- 172.21.15.90:0/2544933926 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6114073b40 con 0x55f9769984d0 2024-04-25T11:35:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.092+0000 7f61379de740 1 -- 172.21.15.90:0/2544933926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769984d0 msgr2=0x55f976d59570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.092+0000 7f61379de740 1 --2- 172.21.15.90:0/2544933926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769984d0 0x55f976d59570 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f61140dc400 tx=0x7f611406d220 comp rx=0 tx=0).stop 2024-04-25T11:35:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.092+0000 7f61379de740 1 -- 172.21.15.90:0/2544933926 shutdown_connections 2024-04-25T11:35:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.092+0000 7f61379de740 1 --2- 172.21.15.90:0/2544933926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769984d0 0x55f976d59570 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.092+0000 7f61379de740 1 --2- 172.21.15.90:0/2544933926 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f9769902f0 0x55f976d59030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.092+0000 7f61379de740 1 --2- 172.21.15.90:0/2544933926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9769b91b0 0x55f976d58af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.093+0000 7f61379de740 1 -- 172.21.15.90:0/2544933926 >> 172.21.15.90:0/2544933926 conn(0x55f9769ad300 msgr2=0x55f976d552a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.093+0000 7f61379de740 1 -- 172.21.15.90:0/2544933926 shutdown_connections 2024-04-25T11:35:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.093+0000 7f61379de740 1 -- 172.21.15.90:0/2544933926 wait complete. 2024-04-25T11:35:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.093+0000 7f61379de740 1 -- start start 2024-04-25T11:35:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.093+0000 7f61379de740 1 Processor -- start 2024-04-25T11:35:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.093+0000 7f61379de740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769902f0 0x55f976d63190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.093+0000 7f61379de740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9769984d0 0x55f976d636d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.093+0000 7f61379de740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f9769b91b0 0x55f976d690b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.093+0000 7f61379de740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f976d65bc0 con 0x55f9769902f0 2024-04-25T11:35:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.093+0000 7f61379de740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f976c46b60 con 0x55f9769b91b0 2024-04-25T11:35:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.093+0000 7f61379de740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f976d65d40 con 0x55f9769984d0 2024-04-25T11:35:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.093+0000 7f61379de740 10 client.0 _ll_register_callbacks cb 0x55f976c6b2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:35:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.093+0000 7f612a59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769902f0 0x55f976d63190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.094+0000 7f612a59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769902f0 0x55f976d63190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56592/0 (socket says 172.21.15.90:56592) 2024-04-25T11:35:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.094+0000 7f612a59c640 1 -- 172.21.15.90:0/3916695621 learned_addr learned my addr 172.21.15.90:0/3916695621 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.094+0000 7f6129d9b640 1 --2- 172.21.15.90:0/3916695621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9769984d0 0x55f976d636d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.094+0000 7f612a59c640 1 -- 172.21.15.90:0/3916695621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9769984d0 msgr2=0x55f976d636d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.094+0000 7f612ad9d640 1 --2- 172.21.15.90:0/3916695621 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f9769b91b0 0x55f976d690b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.094+0000 7f612a59c640 1 --2- 172.21.15.90:0/3916695621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9769984d0 0x55f976d636d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.094+0000 7f612a59c640 1 -- 172.21.15.90:0/3916695621 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f9769b91b0 msgr2=0x55f976d690b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.094+0000 7f612a59c640 1 --2- 172.21.15.90:0/3916695621 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f9769b91b0 0x55f976d690b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.094+0000 7f612a59c640 1 -- 172.21.15.90:0/3916695621 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9769d1210 con 0x55f9769902f0 2024-04-25T11:35:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.094+0000 7f6129d9b640 1 --2- 172.21.15.90:0/3916695621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9769984d0 0x55f976d636d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:35:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.094+0000 7f612a59c640 1 --2- 172.21.15.90:0/3916695621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769902f0 0x55f976d63190 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f611c005fd0 tx=0x7f611c012d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62deba0f0d9922ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.094+0000 7f6111ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:35:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.094+0000 7f6111ffb640 1 -- 172.21.15.90:0/3916695621 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c00cde0 con 0x55f9769902f0 2024-04-25T11:35:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.094+0000 7f6111ffb640 1 -- 172.21.15.90:0/3916695621 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f611c01a3e0 con 0x55f9769902f0 2024-04-25T11:35:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.094+0000 7f61379de740 1 -- 172.21.15.90:0/3916695621 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61140a6910 con 0x55f9769902f0 2024-04-25T11:35:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.094+0000 7f6111ffb640 1 -- 172.21.15.90:0/3916695621 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f611c004b20 con 0x55f9769902f0 2024-04-25T11:35:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.094+0000 7f61379de740 10 client.15606 fetch_fsmap learned FSMap version 26 2024-04-25T11:35:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.094+0000 7f61379de740 1 -- 172.21.15.90:0/3916695621 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f976a9d620 con 0x55f9769902f0 2024-04-25T11:35:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.094+0000 7f6111ffb640 1 -- 172.21.15.90:0/3916695621 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61140a6910 con 0x55f9769902f0 2024-04-25T11:35:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.095+0000 7f6111ffb640 1 -- 172.21.15.90:0/3916695621 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f611c019070 con 0x55f9769902f0 2024-04-25T11:35:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.095+0000 7f61379de740 10 client.15606 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:35:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.095+0000 7f61379de740 10 client.15606 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:35:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.095+0000 7f61379de740 20 client.15606 populate_metadata read hostname 'smithi090' 2024-04-25T11:35:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.095+0000 7f61379de740 1 --2- 172.21.15.90:0/3916695621 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f976977d80 0x55f976d757e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.095+0000 7f61379de740 4 client.15606 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:35:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.095+0000 7f61379de740 1 -- 172.21.15.90:0/3916695621 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f976ccd070 con 0x55f976977d80 2024-04-25T11:35:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.095+0000 7f6129d9b640 1 --2- 172.21.15.90:0/3916695621 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f976977d80 0x55f976d757e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.096+0000 7f6111ffb640 10 client.15606 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.096+0000 7f6129d9b640 1 --2- 172.21.15.90:0/3916695621 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f976977d80 0x55f976d757e0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6bdba722943e95c server_cookie=843e84b5b7fd9028 in_seq=0 out_seq=0 2024-04-25T11:35:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.096+0000 7f6111ffb640 1 -- 172.21.15.90:0/3916695621 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f976ccd070 con 0x55f976977d80 2024-04-25T11:35:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.096+0000 7f6111ffb640 10 client.15606 handle_command_reply: tid=0 2024-04-25T11:35:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.096+0000 7f61379de740 1 client.15606 shutdown 2024-04-25T11:35:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.096+0000 7f61379de740 1 -- 172.21.15.90:0/3916695621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769902f0 msgr2=0x55f976d63190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.096+0000 7f61379de740 1 --2- 172.21.15.90:0/3916695621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769902f0 0x55f976d63190 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f611c005fd0 tx=0x7f611c012d30 comp rx=0 tx=0).stop 2024-04-25T11:35:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.097+0000 7f61379de740 1 -- 172.21.15.90:0/3916695621 shutdown_connections 2024-04-25T11:35:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.097+0000 7f61379de740 1 --2- 172.21.15.90:0/3916695621 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f976977d80 0x55f976d757e0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.097+0000 7f61379de740 1 --2- 172.21.15.90:0/3916695621 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f9769b91b0 0x55f976d690b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.097+0000 7f6111ffb640 0 client.15606 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.097+0000 7f61379de740 1 --2- 172.21.15.90:0/3916695621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9769984d0 0x55f976d636d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.097+0000 7f61379de740 1 --2- 172.21.15.90:0/3916695621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769902f0 0x55f976d63190 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.097+0000 7f61379de740 1 -- 172.21.15.90:0/3916695621 >> 172.21.15.90:0/3916695621 conn(0x55f9769ad300 msgr2=0x55f976d5a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.097+0000 7f61379de740 1 -- 172.21.15.90:0/3916695621 shutdown_connections 2024-04-25T11:35:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.097+0000 7f61379de740 1 -- 172.21.15.90:0/3916695621 wait complete. 2024-04-25T11:35:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.097+0000 7f61379de740 20 client.15606 trim_cache size 0 max 16384 2024-04-25T11:35:28.099 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:27 smithi156 ceph-mon[33045]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 77 KiB/s wr, 31 op/s 2024-04-25T11:35:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.112+0000 7f61379de740 1 Processor -- start 2024-04-25T11:35:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.112+0000 7f61379de740 1 -- start start 2024-04-25T11:35:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.112+0000 7f61379de740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f976977d80 0x55f976d59890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.112+0000 7f61379de740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9769902f0 0x55f976d64c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.112+0000 7f61379de740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769984d0 0x55f976d651a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.112+0000 7f61379de740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f612c1afa50 con 0x55f9769984d0 2024-04-25T11:35:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.112+0000 7f61379de740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f612c1b1590 con 0x55f976977d80 2024-04-25T11:35:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.112+0000 7f61379de740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f612c1b0120 con 0x55f9769902f0 2024-04-25T11:35:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.112+0000 7f612ad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769984d0 0x55f976d651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.112+0000 7f612ad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769984d0 0x55f976d651a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56608/0 (socket says 172.21.15.90:56608) 2024-04-25T11:35:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.112+0000 7f612ad9d640 1 -- 172.21.15.90:0/2864576124 learned_addr learned my addr 172.21.15.90:0/2864576124 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.112+0000 7f6129d9b640 1 --2- 172.21.15.90:0/2864576124 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9769902f0 0x55f976d64c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.112+0000 7f612a59c640 1 --2- 172.21.15.90:0/2864576124 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f976977d80 0x55f976d59890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.112+0000 7f612ad9d640 1 -- 172.21.15.90:0/2864576124 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9769902f0 msgr2=0x55f976d64c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.112+0000 7f612ad9d640 1 --2- 172.21.15.90:0/2864576124 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9769902f0 0x55f976d64c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.112+0000 7f612ad9d640 1 -- 172.21.15.90:0/2864576124 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f976977d80 msgr2=0x55f976d59890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.112+0000 7f612ad9d640 1 --2- 172.21.15.90:0/2864576124 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f976977d80 0x55f976d59890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.112+0000 7f612ad9d640 1 -- 172.21.15.90:0/2864576124 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f976d59ed0 con 0x55f9769984d0 2024-04-25T11:35:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.112+0000 7f612ad9d640 1 --2- 172.21.15.90:0/2864576124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769984d0 0x55f976d651a0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f6124009900 tx=0x7f612403e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=608b587f36923cfc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.113+0000 7f6113fff640 1 -- 172.21.15.90:0/2864576124 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6124035040 con 0x55f9769984d0 2024-04-25T11:35:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.113+0000 7f6113fff640 1 -- 172.21.15.90:0/2864576124 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f612404c030 con 0x55f9769984d0 2024-04-25T11:35:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.113+0000 7f6113fff640 1 -- 172.21.15.90:0/2864576124 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6124004990 con 0x55f9769984d0 2024-04-25T11:35:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.113+0000 7f61379de740 1 -- 172.21.15.90:0/2864576124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769984d0 msgr2=0x55f976d651a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.113+0000 7f61379de740 1 --2- 172.21.15.90:0/2864576124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769984d0 0x55f976d651a0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f6124009900 tx=0x7f612403e000 comp rx=0 tx=0).stop 2024-04-25T11:35:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.113+0000 7f61379de740 1 -- 172.21.15.90:0/2864576124 shutdown_connections 2024-04-25T11:35:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.113+0000 7f61379de740 1 --2- 172.21.15.90:0/2864576124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769984d0 0x55f976d651a0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.113+0000 7f61379de740 1 --2- 172.21.15.90:0/2864576124 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9769902f0 0x55f976d64c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.113+0000 7f61379de740 1 --2- 172.21.15.90:0/2864576124 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f976977d80 0x55f976d59890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.113+0000 7f61379de740 1 -- 172.21.15.90:0/2864576124 >> 172.21.15.90:0/2864576124 conn(0x55f9769ad300 msgr2=0x55f976d64550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.113+0000 7f61379de740 1 -- 172.21.15.90:0/2864576124 shutdown_connections 2024-04-25T11:35:28.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.113+0000 7f61379de740 1 -- 172.21.15.90:0/2864576124 wait complete. 2024-04-25T11:35:28.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.114+0000 7f61379de740 1 -- start start 2024-04-25T11:35:28.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.114+0000 7f61379de740 1 Processor -- start 2024-04-25T11:35:28.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.114+0000 7f61379de740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f976977d80 0x55f976d6f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:28.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.114+0000 7f61379de740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f9769902f0 0x55f976d6a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:28.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.114+0000 7f61379de740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769984d0 0x55f976d6abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:28.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.114+0000 7f61379de740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f612c1b0ec0 con 0x55f9769984d0 2024-04-25T11:35:28.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.114+0000 7f61379de740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f612c1b07f0 con 0x55f9769902f0 2024-04-25T11:35:28.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.114+0000 7f61379de740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f612c1ada30 con 0x55f976977d80 2024-04-25T11:35:28.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.114+0000 7f61379de740 10 client.0 _ll_register_callbacks cb 0x7f611c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:35:28.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.114+0000 7f612ad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769984d0 0x55f976d6abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:28.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.114+0000 7f612ad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769984d0 0x55f976d6abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56622/0 (socket says 172.21.15.90:56622) 2024-04-25T11:35:28.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.114+0000 7f612ad9d640 1 -- 172.21.15.90:0/2882004198 learned_addr learned my addr 172.21.15.90:0/2882004198 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:28.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.114+0000 7f6129d9b640 1 --2- 172.21.15.90:0/2882004198 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f9769902f0 0x55f976d6a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:28.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.114+0000 7f612a59c640 1 --2- 172.21.15.90:0/2882004198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f976977d80 0x55f976d6f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:28.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.114+0000 7f612ad9d640 1 -- 172.21.15.90:0/2882004198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f976977d80 msgr2=0x55f976d6f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:28.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.114+0000 7f612ad9d640 1 --2- 172.21.15.90:0/2882004198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f976977d80 0x55f976d6f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.114+0000 7f612ad9d640 1 -- 172.21.15.90:0/2882004198 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f9769902f0 msgr2=0x55f976d6a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:28.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.114+0000 7f612ad9d640 1 --2- 172.21.15.90:0/2882004198 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f9769902f0 0x55f976d6a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.114+0000 7f612ad9d640 1 -- 172.21.15.90:0/2882004198 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f976d6b100 con 0x55f9769984d0 2024-04-25T11:35:28.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.114+0000 7f6129d9b640 1 --2- 172.21.15.90:0/2882004198 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f9769902f0 0x55f976d6a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:35:28.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.115+0000 7f612ad9d640 1 --2- 172.21.15.90:0/2882004198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769984d0 0x55f976d6abc0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f6124004570 tx=0x7f6124005610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e868130d6fd92aee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:28.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.115+0000 7f6111ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:35:28.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.115+0000 7f6111ffb640 1 -- 172.21.15.90:0/2882004198 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612405f060 con 0x55f9769984d0 2024-04-25T11:35:28.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.115+0000 7f6111ffb640 1 -- 172.21.15.90:0/2882004198 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f6124004730 con 0x55f9769984d0 2024-04-25T11:35:28.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.115+0000 7f61379de740 1 -- 172.21.15.90:0/2882004198 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f612404c030 con 0x55f9769984d0 2024-04-25T11:35:28.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.115+0000 7f6111ffb640 1 -- 172.21.15.90:0/2882004198 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6124035040 con 0x55f9769984d0 2024-04-25T11:35:28.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.115+0000 7f61379de740 10 client.15612 fetch_fsmap learned FSMap version 26 2024-04-25T11:35:28.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.115+0000 7f61379de740 1 -- 172.21.15.90:0/2882004198 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f976ddd7a0 con 0x55f9769984d0 2024-04-25T11:35:28.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.115+0000 7f6111ffb640 1 -- 172.21.15.90:0/2882004198 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f612404c030 con 0x55f9769984d0 2024-04-25T11:35:28.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.115+0000 7f6111ffb640 1 -- 172.21.15.90:0/2882004198 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f6124003940 con 0x55f9769984d0 2024-04-25T11:35:28.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.115+0000 7f61379de740 10 client.15612 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:35:28.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.115+0000 7f61379de740 10 client.15612 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:35:28.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.115+0000 7f61379de740 20 client.15612 populate_metadata read hostname 'smithi090' 2024-04-25T11:35:28.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.115+0000 7f61379de740 1 --2- 172.21.15.90:0/2882004198 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f9769b91b0 0x55f976ddff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:28.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.115+0000 7f61379de740 4 client.15612 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:35:28.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.115+0000 7f61379de740 1 -- 172.21.15.90:0/2882004198 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f976cc6ee0 con 0x55f9769b91b0 2024-04-25T11:35:28.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.116+0000 7f612a59c640 1 --2- 172.21.15.90:0/2882004198 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f9769b91b0 0x55f976ddff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:28.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.116+0000 7f6111ffb640 10 client.15612 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:28.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.116+0000 7f612a59c640 1 --2- 172.21.15.90:0/2882004198 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f9769b91b0 0x55f976ddff00 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a42339397879acad server_cookie=aa7d38f5dc5b152e in_seq=0 out_seq=0 2024-04-25T11:35:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.129+0000 7f6111ffb640 1 -- 172.21.15.90:0/2882004198 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f976cc6ee0 con 0x55f9769b91b0 2024-04-25T11:35:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.129+0000 7f6111ffb640 10 client.15612 handle_command_reply: tid=0 2024-04-25T11:35:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.129+0000 7f61379de740 1 client.15612 shutdown 2024-04-25T11:35:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.129+0000 7f61379de740 1 -- 172.21.15.90:0/2882004198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769984d0 msgr2=0x55f976d6abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.129+0000 7f61379de740 1 --2- 172.21.15.90:0/2882004198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769984d0 0x55f976d6abc0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f6124004570 tx=0x7f6124005610 comp rx=0 tx=0).stop 2024-04-25T11:35:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.129+0000 7f61379de740 1 -- 172.21.15.90:0/2882004198 shutdown_connections 2024-04-25T11:35:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.130+0000 7f61379de740 1 --2- 172.21.15.90:0/2882004198 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f9769b91b0 0x55f976ddff00 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.130+0000 7f61379de740 1 --2- 172.21.15.90:0/2882004198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f9769984d0 0x55f976d6abc0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.130+0000 7f6111ffb640 0 client.15612 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.130+0000 7f61379de740 1 --2- 172.21.15.90:0/2882004198 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f9769902f0 0x55f976d6a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.130+0000 7f61379de740 1 --2- 172.21.15.90:0/2882004198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f976977d80 0x55f976d6f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.130+0000 7f61379de740 1 -- 172.21.15.90:0/2882004198 >> 172.21.15.90:0/2882004198 conn(0x55f9769ad300 msgr2=0x55f976d64f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.130+0000 7f61379de740 1 -- 172.21.15.90:0/2882004198 shutdown_connections 2024-04-25T11:35:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.130+0000 7f61379de740 1 -- 172.21.15.90:0/2882004198 wait complete. 2024-04-25T11:35:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.130+0000 7f61379de740 20 client.15612 trim_cache size 0 max 16384 2024-04-25T11:35:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:35:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:35:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:35:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:35:28.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.130+0000 7f6130a4a640 1 -- 172.21.15.90:0/254376509 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f61180afce0 msgr2=0x7f61180b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:28.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.130+0000 7f6130a4a640 1 --2- 172.21.15.90:0/254376509 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f61180afce0 0x7f61180b21f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6124031440 tx=0x7f6124009260 comp rx=0 tx=0).stop 2024-04-25T11:35:28.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.130+0000 7f6130a4a640 1 -- 172.21.15.90:0/254376509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f612c109bf0 msgr2=0x7f612c19ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:28.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.130+0000 7f6130a4a640 1 --2- 172.21.15.90:0/254376509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f612c109bf0 0x7f612c19ac80 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f611400d770 tx=0x7f611400dc30 comp rx=0 tx=0).stop 2024-04-25T11:35:28.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.130+0000 7f6130a4a640 1 -- 172.21.15.90:0/254376509 shutdown_connections 2024-04-25T11:35:28.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.130+0000 7f6130a4a640 1 --2- 172.21.15.90:0/254376509 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f61180afce0 0x7f61180b21f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.130+0000 7f6130a4a640 1 --2- 172.21.15.90:0/254376509 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f612c10e5a0 0x7f612c19ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.130+0000 7f6130a4a640 1 --2- 172.21.15.90:0/254376509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f612c109bf0 0x7f612c19ac80 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.130+0000 7f6130a4a640 1 --2- 172.21.15.90:0/254376509 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f612c1020a0 0x7f612c19a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:28.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.130+0000 7f6130a4a640 1 -- 172.21.15.90:0/254376509 >> 172.21.15.90:0/254376509 conn(0x7f612c0778b0 msgr2=0x7f612c10a560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:28.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.130+0000 7f6130a4a640 1 -- 172.21.15.90:0/254376509 shutdown_connections 2024-04-25T11:35:28.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:28.130+0000 7f6130a4a640 1 -- 172.21.15.90:0/254376509 wait complete. 2024-04-25T11:35:29.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:29 smithi090 ceph-mon[29575]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 767 B/s wr, 11 op/s 2024-04-25T11:35:30.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:29 smithi138 ceph-mon[32328]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 767 B/s wr, 11 op/s 2024-04-25T11:35:30.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:29 smithi156 ceph-mon[33045]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 767 B/s wr, 11 op/s 2024-04-25T11:35:31.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:31 smithi090 ceph-mon[29575]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 2.3 KiB/s wr, 20 op/s 2024-04-25T11:35:32.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:31 smithi138 ceph-mon[32328]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 2.3 KiB/s wr, 20 op/s 2024-04-25T11:35:32.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:31 smithi156 ceph-mon[33045]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 2.3 KiB/s wr, 20 op/s 2024-04-25T11:35:33.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:33 smithi090 ceph-mon[29575]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-04-25T11:35:34.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:33 smithi138 ceph-mon[32328]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-04-25T11:35:34.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:33 smithi156 ceph-mon[33045]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-04-25T11:35:35.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:35 smithi090 ceph-mon[29575]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-04-25T11:35:36.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:35 smithi138 ceph-mon[32328]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-04-25T11:35:36.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:35 smithi156 ceph-mon[33045]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-04-25T11:35:37.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:37 smithi090 ceph-mon[29575]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-04-25T11:35:38.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:37 smithi138 ceph-mon[32328]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-04-25T11:35:38.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:37 smithi156 ceph-mon[33045]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-04-25T11:35:39.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:39 smithi090 ceph-mon[29575]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-04-25T11:35:40.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:39 smithi138 ceph-mon[32328]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-04-25T11:35:40.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:39 smithi156 ceph-mon[33045]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-04-25T11:35:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.243+0000 7fbfa18d9640 1 -- 172.21.15.90:0/666859576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf9c109ff0 msgr2=0x7fbf9c103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.243+0000 7fbfa18d9640 1 --2- 172.21.15.90:0/666859576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf9c109ff0 0x7fbf9c103c30 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fbf84009900 tx=0x7fbf8402a410 comp rx=0 tx=0).stop 2024-04-25T11:35:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.243+0000 7fbfa18d9640 1 -- 172.21.15.90:0/666859576 shutdown_connections 2024-04-25T11:35:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.243+0000 7fbfa18d9640 1 --2- 172.21.15.90:0/666859576 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbf9c1042a0 0x7fbf9c110670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.243+0000 7fbfa18d9640 1 --2- 172.21.15.90:0/666859576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf9c109ff0 0x7fbf9c103c30 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.243+0000 7fbfa18d9640 1 --2- 172.21.15.90:0/666859576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf9c109b50 0x7fbf9c1036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.243+0000 7fbfa18d9640 1 -- 172.21.15.90:0/666859576 >> 172.21.15.90:0/666859576 conn(0x7fbf9c0ff230 msgr2=0x7fbf9c101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.243+0000 7fbfa18d9640 1 -- 172.21.15.90:0/666859576 shutdown_connections 2024-04-25T11:35:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.243+0000 7fbfa18d9640 1 -- 172.21.15.90:0/666859576 wait complete. 2024-04-25T11:35:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.243+0000 7fbfa18d9640 1 Processor -- start 2024-04-25T11:35:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.243+0000 7fbfa18d9640 1 -- start start 2024-04-25T11:35:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.244+0000 7fbfa18d9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf9c1042a0 0x7fbf9c19a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.244+0000 7fbfa18d9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf9c109b50 0x7fbf9c19ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.244+0000 7fbfa18d9640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbf9c109ff0 0x7fbf9c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.244+0000 7fbfa18d9640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9c113d40 con 0x7fbf9c1042a0 2024-04-25T11:35:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.244+0000 7fbfa18d9640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9c113bc0 con 0x7fbf9c109ff0 2024-04-25T11:35:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.244+0000 7fbfa18d9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9c113ec0 con 0x7fbf9c109b50 2024-04-25T11:35:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.244+0000 7fbf9affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf9c1042a0 0x7fbf9c19a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.244+0000 7fbf9affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf9c1042a0 0x7fbf9c19a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50092/0 (socket says 172.21.15.90:50092) 2024-04-25T11:35:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.244+0000 7fbf9affd640 1 -- 172.21.15.90:0/4207895551 learned_addr learned my addr 172.21.15.90:0/4207895551 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.244+0000 7fbf927fc640 1 --2- 172.21.15.90:0/4207895551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf9c109b50 0x7fbf9c19ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.244+0000 7fbf9b7fe640 1 --2- 172.21.15.90:0/4207895551 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbf9c109ff0 0x7fbf9c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.244+0000 7fbf9affd640 1 -- 172.21.15.90:0/4207895551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf9c109b50 msgr2=0x7fbf9c19ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.244+0000 7fbf9affd640 1 --2- 172.21.15.90:0/4207895551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf9c109b50 0x7fbf9c19ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.244+0000 7fbf9affd640 1 -- 172.21.15.90:0/4207895551 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbf9c109ff0 msgr2=0x7fbf9c19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.244+0000 7fbf9affd640 1 --2- 172.21.15.90:0/4207895551 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbf9c109ff0 0x7fbf9c19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.244+0000 7fbf9affd640 1 -- 172.21.15.90:0/4207895551 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf9c19f620 con 0x7fbf9c1042a0 2024-04-25T11:35:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.245+0000 7fbf9affd640 1 --2- 172.21.15.90:0/4207895551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf9c1042a0 0x7fbf9c19a7c0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c005fd0 tx=0x7fbf8c00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.245+0000 7fbfa08d7640 1 -- 172.21.15.90:0/4207895551 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c017070 con 0x7fbf9c1042a0 2024-04-25T11:35:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.245+0000 7fbfa08d7640 1 -- 172.21.15.90:0/4207895551 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf8c009c30 con 0x7fbf9c1042a0 2024-04-25T11:35:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.245+0000 7fbfa18d9640 1 -- 172.21.15.90:0/4207895551 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf9c19f910 con 0x7fbf9c1042a0 2024-04-25T11:35:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.245+0000 7fbfa08d7640 1 -- 172.21.15.90:0/4207895551 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c003d30 con 0x7fbf9c1042a0 2024-04-25T11:35:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.245+0000 7fbfa18d9640 1 -- 172.21.15.90:0/4207895551 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf9c1a7290 con 0x7fbf9c1042a0 2024-04-25T11:35:41.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.247+0000 7fbfa08d7640 1 -- 172.21.15.90:0/4207895551 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 23) ==== 147696+0+0 (secure 0 0 0) 0x7fbf8c003ed0 con 0x7fbf9c1042a0 2024-04-25T11:35:41.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.247+0000 7fbfa8782740 1 Processor -- start 2024-04-25T11:35:41.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.248+0000 7fbfa8782740 1 -- start start 2024-04-25T11:35:41.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.248+0000 7fbfa8782740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621f5ad21b0 0x5621f5e71980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:41.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.248+0000 7fbfa8782740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5aa92f0 0x5621f5e71ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:41.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.248+0000 7fbfa8782740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5621f5ab14d0 0x5621f5e72400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:41.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.248+0000 7fbfa8782740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5621f5dd8460 con 0x5621f5aa92f0 2024-04-25T11:35:41.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.248+0000 7fbfa8782740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5621f5d92590 con 0x5621f5ab14d0 2024-04-25T11:35:41.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.248+0000 7fbfa8782740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5621f5dd4420 con 0x5621f5ad21b0 2024-04-25T11:35:41.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.248+0000 7fbf9affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5aa92f0 0x5621f5e71ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:41.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.248+0000 7fbf9affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5aa92f0 0x5621f5e71ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50102/0 (socket says 172.21.15.90:50102) 2024-04-25T11:35:41.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.248+0000 7fbf9affd640 1 -- 172.21.15.90:0/4214060822 learned_addr learned my addr 172.21.15.90:0/4214060822 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:41.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.248+0000 7fbf9b7fe640 1 --2- 172.21.15.90:0/4214060822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621f5ad21b0 0x5621f5e71980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:41.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.248+0000 7fbfa08d7640 1 --2- 172.21.15.90:0/4207895551 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbf880afce0 0x7fbf880b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:41.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.248+0000 7fbf927fc640 1 --2- 172.21.15.90:0/4214060822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5621f5ab14d0 0x5621f5e72400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:41.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.248+0000 7fbf9b7fe640 1 --2- 172.21.15.90:0/4207895551 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbf880afce0 0x7fbf880b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:41.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.248+0000 7fbf9affd640 1 -- 172.21.15.90:0/4214060822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621f5ad21b0 msgr2=0x5621f5e71980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:41.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.248+0000 7fbf9affd640 1 --2- 172.21.15.90:0/4214060822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621f5ad21b0 0x5621f5e71980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.248+0000 7fbf9affd640 1 -- 172.21.15.90:0/4214060822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5621f5ab14d0 msgr2=0x5621f5e72400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:41.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.248+0000 7fbf9affd640 1 --2- 172.21.15.90:0/4214060822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5621f5ab14d0 0x5621f5e72400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.248+0000 7fbf9affd640 1 -- 172.21.15.90:0/4214060822 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621f5d9f0d0 con 0x5621f5aa92f0 2024-04-25T11:35:41.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.248+0000 7fbf9b7fe640 1 --2- 172.21.15.90:0/4214060822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621f5ad21b0 0x5621f5e71980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:35:41.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.249+0000 7fbfa08d7640 1 -- 172.21.15.90:0/4207895551 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7fbf8c0ded20 con 0x7fbf9c1042a0 2024-04-25T11:35:41.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.249+0000 7fbf9affd640 1 --2- 172.21.15.90:0/4214060822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5aa92f0 0x5621f5e71ec0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0dc490 tx=0x7fbf8c012f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=717442ab49bac4b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:41.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.249+0000 7fbf9b7fe640 1 --2- 172.21.15.90:0/4207895551 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbf880afce0 0x7fbf880b21f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x5621f5e6ec90 tx=0x7fbf94004ab0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:41.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.249+0000 7fbf91ffb640 1 -- 172.21.15.90:0/4214060822 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c0a6860 con 0x5621f5aa92f0 2024-04-25T11:35:41.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.249+0000 7fbf91ffb640 1 -- 172.21.15.90:0/4214060822 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf8c0a7450 con 0x5621f5aa92f0 2024-04-25T11:35:41.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.249+0000 7fbf91ffb640 1 -- 172.21.15.90:0/4214060822 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c0dc660 con 0x5621f5aa92f0 2024-04-25T11:35:41.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.250+0000 7fbfa8782740 1 -- 172.21.15.90:0/4214060822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5aa92f0 msgr2=0x5621f5e71ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:41.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.250+0000 7fbfa8782740 1 --2- 172.21.15.90:0/4214060822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5aa92f0 0x5621f5e71ec0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0dc490 tx=0x7fbf8c012f90 comp rx=0 tx=0).stop 2024-04-25T11:35:41.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.250+0000 7fbfa8782740 1 -- 172.21.15.90:0/4214060822 shutdown_connections 2024-04-25T11:35:41.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.250+0000 7fbfa8782740 1 --2- 172.21.15.90:0/4214060822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5621f5ab14d0 0x5621f5e72400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.250+0000 7fbfa8782740 1 --2- 172.21.15.90:0/4214060822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5aa92f0 0x5621f5e71ec0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.250+0000 7fbfa8782740 1 --2- 172.21.15.90:0/4214060822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621f5ad21b0 0x5621f5e71980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.250+0000 7fbfa8782740 1 -- 172.21.15.90:0/4214060822 >> 172.21.15.90:0/4214060822 conn(0x5621f5ac6300 msgr2=0x5621f5e6e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:41.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.250+0000 7fbfa8782740 1 -- 172.21.15.90:0/4214060822 shutdown_connections 2024-04-25T11:35:41.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.250+0000 7fbfa8782740 1 -- 172.21.15.90:0/4214060822 wait complete. 2024-04-25T11:35:41.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.250+0000 7fbfa8782740 1 -- start start 2024-04-25T11:35:41.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.251+0000 7fbfa8782740 1 Processor -- start 2024-04-25T11:35:41.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.251+0000 7fbfa8782740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5621f5aa92f0 0x5621f5e80540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:41.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.251+0000 7fbfa8782740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5ab14d0 0x5621f5e80a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:41.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.251+0000 7fbfa8782740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621f5ad21b0 0x5621f5e864b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:41.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.251+0000 7fbfa8782740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5621f5e83030 con 0x5621f5ab14d0 2024-04-25T11:35:41.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.251+0000 7fbfa8782740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5621f5d5fb60 con 0x5621f5aa92f0 2024-04-25T11:35:41.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.251+0000 7fbfa8782740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5621f5e831b0 con 0x5621f5ad21b0 2024-04-25T11:35:41.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.251+0000 7fbfa8782740 10 client.0 _ll_register_callbacks cb 0x5621f5d842b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:35:41.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.251+0000 7fbf927fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5ab14d0 0x5621f5e80a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:41.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.251+0000 7fbf927fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5ab14d0 0x5621f5e80a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50112/0 (socket says 172.21.15.90:50112) 2024-04-25T11:35:41.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.251+0000 7fbf927fc640 1 -- 172.21.15.90:0/144077717 learned_addr learned my addr 172.21.15.90:0/144077717 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:41.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.251+0000 7fbf9b7fe640 1 --2- 172.21.15.90:0/144077717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621f5ad21b0 0x5621f5e864b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:41.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.251+0000 7fbf9affd640 1 --2- 172.21.15.90:0/144077717 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5621f5aa92f0 0x5621f5e80540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:41.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.251+0000 7fbf927fc640 1 -- 172.21.15.90:0/144077717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621f5ad21b0 msgr2=0x5621f5e864b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:41.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.251+0000 7fbf927fc640 1 --2- 172.21.15.90:0/144077717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621f5ad21b0 0x5621f5e864b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.251+0000 7fbf927fc640 1 -- 172.21.15.90:0/144077717 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5621f5aa92f0 msgr2=0x5621f5e80540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:41.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.251+0000 7fbf927fc640 1 --2- 172.21.15.90:0/144077717 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5621f5aa92f0 0x5621f5e80540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.251+0000 7fbf927fc640 1 -- 172.21.15.90:0/144077717 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621f5aea210 con 0x5621f5ab14d0 2024-04-25T11:35:41.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.251+0000 7fbf9affd640 1 --2- 172.21.15.90:0/144077717 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5621f5aa92f0 0x5621f5e80540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:35:41.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.252+0000 7fbf927fc640 1 --2- 172.21.15.90:0/144077717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5ab14d0 0x5621f5e80a80 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c06b330 tx=0x7fbf84032f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d403e1e145c5272 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:41.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.252+0000 7fbf7affd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:35:41.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.252+0000 7fbf7affd640 1 -- 172.21.15.90:0/144077717 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf84035040 con 0x5621f5ab14d0 2024-04-25T11:35:41.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.252+0000 7fbf7affd640 1 -- 172.21.15.90:0/144077717 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7fbf840317a0 con 0x5621f5ab14d0 2024-04-25T11:35:41.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.252+0000 7fbfa8782740 1 -- 172.21.15.90:0/144077717 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf8c0a7450 con 0x5621f5ab14d0 2024-04-25T11:35:41.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.252+0000 7fbf7affd640 1 -- 172.21.15.90:0/144077717 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbf84004040 con 0x5621f5ab14d0 2024-04-25T11:35:41.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.252+0000 7fbfa8782740 10 client.15624 fetch_fsmap learned FSMap version 26 2024-04-25T11:35:41.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.252+0000 7fbfa8782740 1 -- 172.21.15.90:0/144077717 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5621f5bb6620 con 0x5621f5ab14d0 2024-04-25T11:35:41.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.252+0000 7fbf7affd640 1 -- 172.21.15.90:0/144077717 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf8c0a7450 con 0x5621f5ab14d0 2024-04-25T11:35:41.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.252+0000 7fbf7affd640 1 -- 172.21.15.90:0/144077717 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fbf84004750 con 0x5621f5ab14d0 2024-04-25T11:35:41.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.252+0000 7fbfa8782740 10 client.15624 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:35:41.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.252+0000 7fbfa8782740 10 client.15624 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:35:41.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.252+0000 7fbfa8782740 20 client.15624 populate_metadata read hostname 'smithi090' 2024-04-25T11:35:41.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.253+0000 7fbfa8782740 1 --2- 172.21.15.90:0/144077717 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5621f5a90d80 0x5621f5e92ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:41.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.253+0000 7fbfa8782740 4 client.15624 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:35:41.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.253+0000 7fbfa8782740 1 -- 172.21.15.90:0/144077717 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621f5de6070 con 0x5621f5a90d80 2024-04-25T11:35:41.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.253+0000 7fbf9affd640 1 --2- 172.21.15.90:0/144077717 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5621f5a90d80 0x5621f5e92ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:41.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.253+0000 7fbf7affd640 10 client.15624 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:41.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.253+0000 7fbf9affd640 1 --2- 172.21.15.90:0/144077717 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5621f5a90d80 0x5621f5e92ab0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48756cea88aecc35 server_cookie=4200639347124bca in_seq=0 out_seq=0 2024-04-25T11:35:41.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.254+0000 7fbf7affd640 1 -- 172.21.15.90:0/144077717 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5621f5de6070 con 0x5621f5a90d80 2024-04-25T11:35:41.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.254+0000 7fbf7affd640 10 client.15624 handle_command_reply: tid=0 2024-04-25T11:35:41.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.254+0000 7fbfa8782740 1 client.15624 shutdown 2024-04-25T11:35:41.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.254+0000 7fbfa8782740 1 -- 172.21.15.90:0/144077717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5ab14d0 msgr2=0x5621f5e80a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:41.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.254+0000 7fbfa8782740 1 --2- 172.21.15.90:0/144077717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5ab14d0 0x5621f5e80a80 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c06b330 tx=0x7fbf84032f60 comp rx=0 tx=0).stop 2024-04-25T11:35:41.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.254+0000 7fbfa8782740 1 -- 172.21.15.90:0/144077717 shutdown_connections 2024-04-25T11:35:41.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.254+0000 7fbfa8782740 1 --2- 172.21.15.90:0/144077717 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5621f5a90d80 0x5621f5e92ab0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.254+0000 7fbfa8782740 1 --2- 172.21.15.90:0/144077717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621f5ad21b0 0x5621f5e864b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.254+0000 7fbf7affd640 0 client.15624 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:41.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.254+0000 7fbfa8782740 1 --2- 172.21.15.90:0/144077717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5ab14d0 0x5621f5e80a80 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.254+0000 7fbfa8782740 1 --2- 172.21.15.90:0/144077717 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5621f5aa92f0 0x5621f5e80540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.255+0000 7fbfa8782740 1 -- 172.21.15.90:0/144077717 >> 172.21.15.90:0/144077717 conn(0x5621f5ac6300 msgr2=0x5621f5e72330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:41.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.255+0000 7fbfa8782740 1 -- 172.21.15.90:0/144077717 shutdown_connections 2024-04-25T11:35:41.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.255+0000 7fbfa8782740 1 -- 172.21.15.90:0/144077717 wait complete. 2024-04-25T11:35:41.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.255+0000 7fbfa8782740 20 client.15624 trim_cache size 0 max 16384 2024-04-25T11:35:41.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.270+0000 7fbfa8782740 1 Processor -- start 2024-04-25T11:35:41.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.270+0000 7fbfa8782740 1 -- start start 2024-04-25T11:35:41.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.270+0000 7fbfa8782740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621f5aa92f0 0x5621f5e863e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:41.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.270+0000 7fbfa8782740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5621f5ab14d0 0x5621f5e82480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:41.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.270+0000 7fbfa8782740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5ad21b0 0x5621f5e7fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:41.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.270+0000 7fbfa8782740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9c1ad830 con 0x5621f5ad21b0 2024-04-25T11:35:41.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.270+0000 7fbfa8782740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9c1af370 con 0x5621f5ab14d0 2024-04-25T11:35:41.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.270+0000 7fbfa8782740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9c1adf00 con 0x5621f5aa92f0 2024-04-25T11:35:41.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.270+0000 7fbf9b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5ad21b0 0x5621f5e7fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:41.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.270+0000 7fbf9b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5ad21b0 0x5621f5e7fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50118/0 (socket says 172.21.15.90:50118) 2024-04-25T11:35:41.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.270+0000 7fbf9b7fe640 1 -- 172.21.15.90:0/469990963 learned_addr learned my addr 172.21.15.90:0/469990963 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:41.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.270+0000 7fbf927fc640 1 --2- 172.21.15.90:0/469990963 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5621f5ab14d0 0x5621f5e82480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:41.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.270+0000 7fbf9affd640 1 --2- 172.21.15.90:0/469990963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621f5aa92f0 0x5621f5e863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:41.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.271+0000 7fbf9b7fe640 1 -- 172.21.15.90:0/469990963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621f5aa92f0 msgr2=0x5621f5e863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:41.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.271+0000 7fbf9b7fe640 1 --2- 172.21.15.90:0/469990963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621f5aa92f0 0x5621f5e863e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.271+0000 7fbf9b7fe640 1 -- 172.21.15.90:0/469990963 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5621f5ab14d0 msgr2=0x5621f5e82480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:41.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.271+0000 7fbf9b7fe640 1 --2- 172.21.15.90:0/469990963 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5621f5ab14d0 0x5621f5e82480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.271+0000 7fbf9b7fe640 1 -- 172.21.15.90:0/469990963 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621f5e72ad0 con 0x5621f5ad21b0 2024-04-25T11:35:41.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.271+0000 7fbf9b7fe640 1 --2- 172.21.15.90:0/469990963 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5ad21b0 0x5621f5e7fc10 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fbf94013440 tx=0x7fbf94005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c5b16f34f14a238 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:41.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.271+0000 7fbf917fa640 1 -- 172.21.15.90:0/469990963 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf94004250 con 0x5621f5ad21b0 2024-04-25T11:35:41.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.271+0000 7fbf917fa640 1 -- 172.21.15.90:0/469990963 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf94026070 con 0x5621f5ad21b0 2024-04-25T11:35:41.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.271+0000 7fbf917fa640 1 -- 172.21.15.90:0/469990963 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf94022d10 con 0x5621f5ad21b0 2024-04-25T11:35:41.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.272+0000 7fbfa8782740 1 -- 172.21.15.90:0/469990963 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5ad21b0 msgr2=0x5621f5e7fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:41.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.272+0000 7fbfa8782740 1 --2- 172.21.15.90:0/469990963 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5ad21b0 0x5621f5e7fc10 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fbf94013440 tx=0x7fbf94005bd0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.272+0000 7fbfa8782740 1 -- 172.21.15.90:0/469990963 shutdown_connections 2024-04-25T11:35:41.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.272+0000 7fbfa8782740 1 --2- 172.21.15.90:0/469990963 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5ad21b0 0x5621f5e7fc10 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.272+0000 7fbfa8782740 1 --2- 172.21.15.90:0/469990963 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5621f5ab14d0 0x5621f5e82480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.272+0000 7fbfa8782740 1 --2- 172.21.15.90:0/469990963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621f5aa92f0 0x5621f5e863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.272+0000 7fbfa8782740 1 -- 172.21.15.90:0/469990963 >> 172.21.15.90:0/469990963 conn(0x5621f5ac6300 msgr2=0x5621f5e71880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:41.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.272+0000 7fbfa8782740 1 -- 172.21.15.90:0/469990963 shutdown_connections 2024-04-25T11:35:41.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.272+0000 7fbfa8782740 1 -- 172.21.15.90:0/469990963 wait complete. 2024-04-25T11:35:41.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.272+0000 7fbfa8782740 1 -- start start 2024-04-25T11:35:41.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.272+0000 7fbfa8782740 1 Processor -- start 2024-04-25T11:35:41.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.273+0000 7fbfa8782740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5aa92f0 0x5621f5e8cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:41.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.273+0000 7fbfa8782740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5621f5ab14d0 0x5621f5e8d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:41.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.273+0000 7fbfa8782740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621f5ad21b0 0x5621f5e97bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:41.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.273+0000 7fbfa8782740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9c1aeca0 con 0x5621f5aa92f0 2024-04-25T11:35:41.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.273+0000 7fbfa8782740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9c1ae5d0 con 0x5621f5ab14d0 2024-04-25T11:35:41.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.273+0000 7fbfa8782740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9c1ab830 con 0x5621f5ad21b0 2024-04-25T11:35:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.273+0000 7fbfa8782740 10 client.0 _ll_register_callbacks cb 0x7fbf84009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:35:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.273+0000 7fbf9affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5aa92f0 0x5621f5e8cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.273+0000 7fbf9affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5aa92f0 0x5621f5e8cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50130/0 (socket says 172.21.15.90:50130) 2024-04-25T11:35:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.273+0000 7fbf9affd640 1 -- 172.21.15.90:0/1058144671 learned_addr learned my addr 172.21.15.90:0/1058144671 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.273+0000 7fbf927fc640 1 --2- 172.21.15.90:0/1058144671 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5621f5ab14d0 0x5621f5e8d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.273+0000 7fbf9b7fe640 1 --2- 172.21.15.90:0/1058144671 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621f5ad21b0 0x5621f5e97bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.273+0000 7fbf9affd640 1 -- 172.21.15.90:0/1058144671 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621f5ad21b0 msgr2=0x5621f5e97bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.273+0000 7fbf9affd640 1 --2- 172.21.15.90:0/1058144671 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621f5ad21b0 0x5621f5e97bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.273+0000 7fbf9affd640 1 -- 172.21.15.90:0/1058144671 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5621f5ab14d0 msgr2=0x5621f5e8d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.273+0000 7fbf9affd640 1 --2- 172.21.15.90:0/1058144671 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5621f5ab14d0 0x5621f5e8d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.273+0000 7fbf9affd640 1 -- 172.21.15.90:0/1058144671 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621f5e98310 con 0x5621f5aa92f0 2024-04-25T11:35:41.266 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.273+0000 7fbf927fc640 1 --2- 172.21.15.90:0/1058144671 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5621f5ab14d0 0x5621f5e8d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:35:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.273+0000 7fbf9b7fe640 1 --2- 172.21.15.90:0/1058144671 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621f5ad21b0 0x5621f5e97bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:35:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.273+0000 7fbf9affd640 1 --2- 172.21.15.90:0/1058144671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5aa92f0 0x5621f5e8cd60 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0dc490 tx=0x7fbf8c0def20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e70eef91703e71 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.273+0000 7fbf7affd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:35:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.273+0000 7fbf7affd640 1 -- 172.21.15.90:0/1058144671 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c009de0 con 0x5621f5aa92f0 2024-04-25T11:35:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.273+0000 7fbf7affd640 1 -- 172.21.15.90:0/1058144671 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7fbf8c0dc660 con 0x5621f5aa92f0 2024-04-25T11:35:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.273+0000 7fbfa8782740 1 -- 172.21.15.90:0/1058144671 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf94026070 con 0x5621f5aa92f0 2024-04-25T11:35:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.274+0000 7fbf7affd640 1 -- 172.21.15.90:0/1058144671 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbf8c0dc860 con 0x5621f5aa92f0 2024-04-25T11:35:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.274+0000 7fbfa8782740 10 client.15630 fetch_fsmap learned FSMap version 26 2024-04-25T11:35:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.274+0000 7fbfa8782740 1 -- 172.21.15.90:0/1058144671 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5621f5e8b1d0 con 0x5621f5aa92f0 2024-04-25T11:35:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.274+0000 7fbf7affd640 1 -- 172.21.15.90:0/1058144671 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf94026070 con 0x5621f5aa92f0 2024-04-25T11:35:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.274+0000 7fbf7affd640 1 -- 172.21.15.90:0/1058144671 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fbf8c0dcbc0 con 0x5621f5aa92f0 2024-04-25T11:35:41.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.274+0000 7fbfa8782740 10 client.15630 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:35:41.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.274+0000 7fbfa8782740 10 client.15630 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:35:41.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.274+0000 7fbfa8782740 20 client.15630 populate_metadata read hostname 'smithi090' 2024-04-25T11:35:41.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.274+0000 7fbfa8782740 1 --2- 172.21.15.90:0/1058144671 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5621f5a661c0 0x5621f5efce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:41.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.274+0000 7fbfa8782740 4 client.15630 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:35:41.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.274+0000 7fbfa8782740 1 -- 172.21.15.90:0/1058144671 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5621f5ddfee0 con 0x5621f5a661c0 2024-04-25T11:35:41.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.274+0000 7fbf927fc640 1 --2- 172.21.15.90:0/1058144671 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5621f5a661c0 0x5621f5efce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:41.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.275+0000 7fbf7affd640 10 client.15630 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:41.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.275+0000 7fbf927fc640 1 --2- 172.21.15.90:0/1058144671 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5621f5a661c0 0x5621f5efce50 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbfe65d505a83c98 server_cookie=769e85a868399e20 in_seq=0 out_seq=0 2024-04-25T11:35:41.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.279+0000 7fbf7affd640 1 -- 172.21.15.90:0/1058144671 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5621f5ddfee0 con 0x5621f5a661c0 2024-04-25T11:35:41.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.279+0000 7fbf7affd640 10 client.15630 handle_command_reply: tid=0 2024-04-25T11:35:41.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.279+0000 7fbfa8782740 1 client.15630 shutdown 2024-04-25T11:35:41.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.279+0000 7fbfa8782740 1 -- 172.21.15.90:0/1058144671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5aa92f0 msgr2=0x5621f5e8cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:41.270 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.279+0000 7fbfa8782740 1 --2- 172.21.15.90:0/1058144671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5aa92f0 0x5621f5e8cd60 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0dc490 tx=0x7fbf8c0def20 comp rx=0 tx=0).stop 2024-04-25T11:35:41.270 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.279+0000 7fbfa8782740 1 -- 172.21.15.90:0/1058144671 shutdown_connections 2024-04-25T11:35:41.270 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.279+0000 7fbfa8782740 1 --2- 172.21.15.90:0/1058144671 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5621f5a661c0 0x5621f5efce50 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.270 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.279+0000 7fbfa8782740 1 --2- 172.21.15.90:0/1058144671 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621f5ad21b0 0x5621f5e97bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.270 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.279+0000 7fbfa8782740 1 --2- 172.21.15.90:0/1058144671 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5621f5ab14d0 0x5621f5e8d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.270 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.279+0000 7fbf7affd640 0 client.15630 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:41.270 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.279+0000 7fbfa8782740 1 --2- 172.21.15.90:0/1058144671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5621f5aa92f0 0x5621f5e8cd60 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.270 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.279+0000 7fbfa8782740 1 -- 172.21.15.90:0/1058144671 >> 172.21.15.90:0/1058144671 conn(0x5621f5ac6300 msgr2=0x5621f5e735f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:41.270 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.279+0000 7fbfa8782740 1 -- 172.21.15.90:0/1058144671 shutdown_connections 2024-04-25T11:35:41.270 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.279+0000 7fbfa8782740 1 -- 172.21.15.90:0/1058144671 wait complete. 2024-04-25T11:35:41.270 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.279+0000 7fbfa8782740 20 client.15630 trim_cache size 0 max 16384 2024-04-25T11:35:41.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:35:41.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:35:41.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:35:41.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:35:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.280+0000 7fbfa18d9640 1 -- 172.21.15.90:0/4207895551 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbf880afce0 msgr2=0x7fbf880b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.280+0000 7fbfa18d9640 1 --2- 172.21.15.90:0/4207895551 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbf880afce0 0x7fbf880b21f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x5621f5e6ec90 tx=0x7fbf94004ab0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.280+0000 7fbfa18d9640 1 -- 172.21.15.90:0/4207895551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf9c1042a0 msgr2=0x7fbf9c19a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.280+0000 7fbfa18d9640 1 --2- 172.21.15.90:0/4207895551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf9c1042a0 0x7fbf9c19a7c0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c005fd0 tx=0x7fbf8c00dde0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.280+0000 7fbfa18d9640 1 -- 172.21.15.90:0/4207895551 shutdown_connections 2024-04-25T11:35:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.280+0000 7fbfa18d9640 1 --2- 172.21.15.90:0/4207895551 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbf880afce0 0x7fbf880b21f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.280+0000 7fbfa18d9640 1 --2- 172.21.15.90:0/4207895551 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbf9c109ff0 0x7fbf9c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.280+0000 7fbfa18d9640 1 --2- 172.21.15.90:0/4207895551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf9c109b50 0x7fbf9c19ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.280+0000 7fbfa18d9640 1 --2- 172.21.15.90:0/4207895551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf9c1042a0 0x7fbf9c19a7c0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.280+0000 7fbfa18d9640 1 -- 172.21.15.90:0/4207895551 >> 172.21.15.90:0/4207895551 conn(0x7fbf9c0ff230 msgr2=0x7fbf9c10f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.280+0000 7fbfa18d9640 1 -- 172.21.15.90:0/4207895551 shutdown_connections 2024-04-25T11:35:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:41.280+0000 7fbfa18d9640 1 -- 172.21.15.90:0/4207895551 wait complete. 2024-04-25T11:35:41.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:41 smithi090 ceph-mon[29575]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-04-25T11:35:42.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:41 smithi138 ceph-mon[32328]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-04-25T11:35:42.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:41 smithi156 ceph-mon[33045]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-04-25T11:35:43.096 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:35:43.097 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:35:43.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.244+0000 7fe6ae8bd640 1 -- 172.21.15.90:0/2336994457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6a8104260 msgr2=0x7fe6a8110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.244+0000 7fe6ae8bd640 1 --2- 172.21.15.90:0/2336994457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6a8104260 0x7fe6a8110630 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0009ca0 tx=0x7fe6a002a8c0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.244+0000 7fe6ae8bd640 1 -- 172.21.15.90:0/2336994457 shutdown_connections 2024-04-25T11:35:43.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.244+0000 7fe6ae8bd640 1 --2- 172.21.15.90:0/2336994457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6a8104260 0x7fe6a8110630 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.244+0000 7fe6ae8bd640 1 --2- 172.21.15.90:0/2336994457 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe6a8109fb0 0x7fe6a8103bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.244+0000 7fe6ae8bd640 1 --2- 172.21.15.90:0/2336994457 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6a8109b10 0x7fe6a81036b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.244+0000 7fe6ae8bd640 1 -- 172.21.15.90:0/2336994457 >> 172.21.15.90:0/2336994457 conn(0x7fe6a80ff230 msgr2=0x7fe6a8101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:43.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.244+0000 7fe6ae8bd640 1 -- 172.21.15.90:0/2336994457 shutdown_connections 2024-04-25T11:35:43.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.244+0000 7fe6ae8bd640 1 -- 172.21.15.90:0/2336994457 wait complete. 2024-04-25T11:35:43.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.244+0000 7fe6ae8bd640 1 Processor -- start 2024-04-25T11:35:43.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.244+0000 7fe6ae8bd640 1 -- start start 2024-04-25T11:35:43.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.245+0000 7fe6ae8bd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6a8104260 0x7fe6a819a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.245+0000 7fe6ae8bd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6a8109b10 0x7fe6a819ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.245+0000 7fe6ae8bd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe6a8109fb0 0x7fe6a819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.245+0000 7fe6ae8bd640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a8113d20 con 0x7fe6a8104260 2024-04-25T11:35:43.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.245+0000 7fe6ae8bd640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a8113ba0 con 0x7fe6a8109fb0 2024-04-25T11:35:43.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.245+0000 7fe6ae8bd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a8113ea0 con 0x7fe6a8109b10 2024-04-25T11:35:43.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.245+0000 7fe6a7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6a8104260 0x7fe6a819a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.245+0000 7fe6a7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6a8104260 0x7fe6a819a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43354/0 (socket says 172.21.15.90:43354) 2024-04-25T11:35:43.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.245+0000 7fe6a7fff640 1 -- 172.21.15.90:0/2396290657 learned_addr learned my addr 172.21.15.90:0/2396290657 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:43.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.245+0000 7fe6a77fe640 1 --2- 172.21.15.90:0/2396290657 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6a8109b10 0x7fe6a819ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.245+0000 7fe6ace5a640 1 --2- 172.21.15.90:0/2396290657 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe6a8109fb0 0x7fe6a819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.245+0000 7fe6a7fff640 1 -- 172.21.15.90:0/2396290657 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6a8109b10 msgr2=0x7fe6a819ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.245+0000 7fe6a7fff640 1 --2- 172.21.15.90:0/2396290657 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6a8109b10 0x7fe6a819ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.245+0000 7fe6a7fff640 1 -- 172.21.15.90:0/2396290657 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe6a8109fb0 msgr2=0x7fe6a819ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.245+0000 7fe6a7fff640 1 --2- 172.21.15.90:0/2396290657 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe6a8109fb0 0x7fe6a819ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.245+0000 7fe6a7fff640 1 -- 172.21.15.90:0/2396290657 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6a819f560 con 0x7fe6a8104260 2024-04-25T11:35:43.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.245+0000 7fe6a77fe640 1 --2- 172.21.15.90:0/2396290657 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6a8109b10 0x7fe6a819ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:35:43.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.246+0000 7fe6a7fff640 1 --2- 172.21.15.90:0/2396290657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6a8104260 0x7fe6a819a700 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe69800d740 tx=0x7fe69800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:43.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.246+0000 7fe6a57fa640 1 -- 172.21.15.90:0/2396290657 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe698017070 con 0x7fe6a8104260 2024-04-25T11:35:43.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.246+0000 7fe6a57fa640 1 -- 172.21.15.90:0/2396290657 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe69800de50 con 0x7fe6a8104260 2024-04-25T11:35:43.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.246+0000 7fe6ae8bd640 1 -- 172.21.15.90:0/2396290657 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6a819f850 con 0x7fe6a8104260 2024-04-25T11:35:43.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.246+0000 7fe6a57fa640 1 -- 172.21.15.90:0/2396290657 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe698004ce0 con 0x7fe6a8104260 2024-04-25T11:35:43.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.246+0000 7fe6ae8bd640 1 -- 172.21.15.90:0/2396290657 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6a819fd50 con 0x7fe6a8104260 2024-04-25T11:35:43.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.249+0000 7fe6a57fa640 1 -- 172.21.15.90:0/2396290657 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 23) ==== 147696+0+0 (secure 0 0 0) 0x7fe698004e80 con 0x7fe6a8104260 2024-04-25T11:35:43.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.249+0000 7fe6b5773740 1 Processor -- start 2024-04-25T11:35:43.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.249+0000 7fe6b5773740 1 -- start start 2024-04-25T11:35:43.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.249+0000 7fe6b5773740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9f7495390 0x55e9f7835a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.249+0000 7fe6b5773740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9f746c4d0 0x55e9f7835fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.249+0000 7fe6b5773740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 0x55e9f78428f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.249+0000 7fe6b5773740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e9f779bd80 con 0x55e9f74746b0 2024-04-25T11:35:43.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.249+0000 7fe6b5773740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55e9f77558b0 con 0x55e9f7495390 2024-04-25T11:35:43.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.249+0000 7fe6b5773740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e9f7797d40 con 0x55e9f746c4d0 2024-04-25T11:35:43.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.249+0000 7fe6a77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 0x55e9f78428f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.249+0000 7fe6a77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 0x55e9f78428f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43366/0 (socket says 172.21.15.90:43366) 2024-04-25T11:35:43.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.249+0000 7fe6a77fe640 1 -- 172.21.15.90:0/1686133135 learned_addr learned my addr 172.21.15.90:0/1686133135 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:43.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.249+0000 7fe6a57fa640 1 --2- 172.21.15.90:0/2396290657 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe6900afcf0 0x7fe6900b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.249+0000 7fe6a7fff640 1 --2- 172.21.15.90:0/1686133135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9f746c4d0 0x55e9f7835fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.249+0000 7fe6ace5a640 1 --2- 172.21.15.90:0/1686133135 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9f7495390 0x55e9f7835a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.250+0000 7fe6ace5a640 1 --2- 172.21.15.90:0/2396290657 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe6900afcf0 0x7fe6900b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.250+0000 7fe6a77fe640 1 -- 172.21.15.90:0/1686133135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9f746c4d0 msgr2=0x55e9f7835fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:43.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.250+0000 7fe6a77fe640 1 --2- 172.21.15.90:0/1686133135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9f746c4d0 0x55e9f7835fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.250+0000 7fe6a77fe640 1 -- 172.21.15.90:0/1686133135 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9f7495390 msgr2=0x55e9f7835a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:43.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.250+0000 7fe6a77fe640 1 --2- 172.21.15.90:0/1686133135 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9f7495390 0x55e9f7835a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.250+0000 7fe6a77fe640 1 -- 172.21.15.90:0/1686133135 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9f7726a40 con 0x55e9f74746b0 2024-04-25T11:35:43.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.250+0000 7fe6a57fa640 1 -- 172.21.15.90:0/2396290657 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7fe6980dec30 con 0x7fe6a8104260 2024-04-25T11:35:43.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.250+0000 7fe6a77fe640 1 --2- 172.21.15.90:0/1686133135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 0x55e9f78428f0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe69c009bf0 tx=0x7fe69c0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b74c3aec7e32d712 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:43.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.250+0000 7fe6ace5a640 1 --2- 172.21.15.90:0/2396290657 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe6900afcf0 0x7fe6900b2200 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x55e9f7836b60 tx=0x7fe6a00095c0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:43.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.250+0000 7fe68e7fc640 1 -- 172.21.15.90:0/1686133135 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe69c012450 con 0x55e9f74746b0 2024-04-25T11:35:43.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.250+0000 7fe68e7fc640 1 -- 172.21.15.90:0/1686133135 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe69c012aa0 con 0x55e9f74746b0 2024-04-25T11:35:43.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.250+0000 7fe68e7fc640 1 -- 172.21.15.90:0/1686133135 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe69c00fb90 con 0x55e9f74746b0 2024-04-25T11:35:43.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.251+0000 7fe6b5773740 1 -- 172.21.15.90:0/1686133135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 msgr2=0x55e9f78428f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.251+0000 7fe6b5773740 1 --2- 172.21.15.90:0/1686133135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 0x55e9f78428f0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe69c009bf0 tx=0x7fe69c0117b0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.251+0000 7fe6b5773740 1 -- 172.21.15.90:0/1686133135 shutdown_connections 2024-04-25T11:35:43.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.251+0000 7fe6b5773740 1 --2- 172.21.15.90:0/1686133135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 0x55e9f78428f0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.251+0000 7fe6b5773740 1 --2- 172.21.15.90:0/1686133135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9f746c4d0 0x55e9f7835fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.251+0000 7fe6b5773740 1 --2- 172.21.15.90:0/1686133135 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9f7495390 0x55e9f7835a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.251+0000 7fe6b5773740 1 -- 172.21.15.90:0/1686133135 >> 172.21.15.90:0/1686133135 conn(0x55e9f74894e0 msgr2=0x55e9f7832210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:43.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.251+0000 7fe6b5773740 1 -- 172.21.15.90:0/1686133135 shutdown_connections 2024-04-25T11:35:43.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.251+0000 7fe6b5773740 1 -- 172.21.15.90:0/1686133135 wait complete. 2024-04-25T11:35:43.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.251+0000 7fe6b5773740 1 -- start start 2024-04-25T11:35:43.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.252+0000 7fe6b5773740 1 Processor -- start 2024-04-25T11:35:43.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.252+0000 7fe6b5773740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9f746c4d0 0x55e9f7840580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.252+0000 7fe6b5773740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 0x55e9f7840ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.252+0000 7fe6b5773740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9f7495390 0x55e9f784c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.252+0000 7fe6b5773740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e9f7842fe0 con 0x55e9f74746b0 2024-04-25T11:35:43.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.252+0000 7fe6b5773740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55e9f7722d10 con 0x55e9f746c4d0 2024-04-25T11:35:43.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.252+0000 7fe6b5773740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e9f7843160 con 0x55e9f7495390 2024-04-25T11:35:43.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.252+0000 7fe6b5773740 10 client.0 _ll_register_callbacks cb 0x55e9f774f210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:35:43.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.252+0000 7fe6a77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 0x55e9f7840ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.252+0000 7fe6a77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 0x55e9f7840ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43374/0 (socket says 172.21.15.90:43374) 2024-04-25T11:35:43.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.252+0000 7fe6a77fe640 1 -- 172.21.15.90:0/385136665 learned_addr learned my addr 172.21.15.90:0/385136665 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:43.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.252+0000 7fe6a7fff640 1 --2- 172.21.15.90:0/385136665 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9f746c4d0 0x55e9f7840580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.253+0000 7fe6ace5a640 1 --2- 172.21.15.90:0/385136665 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9f7495390 0x55e9f784c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.253+0000 7fe6a77fe640 1 -- 172.21.15.90:0/385136665 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9f7495390 msgr2=0x55e9f784c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:43.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.253+0000 7fe6a77fe640 1 --2- 172.21.15.90:0/385136665 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9f7495390 0x55e9f784c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.253+0000 7fe6a77fe640 1 -- 172.21.15.90:0/385136665 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9f746c4d0 msgr2=0x55e9f7840580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:43.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.253+0000 7fe6a77fe640 1 --2- 172.21.15.90:0/385136665 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9f746c4d0 0x55e9f7840580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.253+0000 7fe6a77fe640 1 -- 172.21.15.90:0/385136665 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9f75777d0 con 0x55e9f74746b0 2024-04-25T11:35:43.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.253+0000 7fe6a77fe640 1 --2- 172.21.15.90:0/385136665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 0x55e9f7840ac0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe69c01c040 tx=0x7fe69c012f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c46a369b20a873d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:43.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.253+0000 7fe6877fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:35:43.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.253+0000 7fe6877fe640 1 -- 172.21.15.90:0/385136665 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe69c012660 con 0x55e9f74746b0 2024-04-25T11:35:43.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.253+0000 7fe6877fe640 1 -- 172.21.15.90:0/385136665 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7fe69c012800 con 0x55e9f74746b0 2024-04-25T11:35:43.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.253+0000 7fe6b5773740 1 -- 172.21.15.90:0/385136665 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe69c012aa0 con 0x55e9f74746b0 2024-04-25T11:35:43.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.253+0000 7fe6877fe640 1 -- 172.21.15.90:0/385136665 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe69c033050 con 0x55e9f74746b0 2024-04-25T11:35:43.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.253+0000 7fe6b5773740 10 client.15642 fetch_fsmap learned FSMap version 26 2024-04-25T11:35:43.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.253+0000 7fe6b5773740 1 -- 172.21.15.90:0/385136665 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e9f74ad3f0 con 0x55e9f74746b0 2024-04-25T11:35:43.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.253+0000 7fe6877fe640 1 -- 172.21.15.90:0/385136665 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe69c012aa0 con 0x55e9f74746b0 2024-04-25T11:35:43.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.254+0000 7fe6877fe640 1 -- 172.21.15.90:0/385136665 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fe69c005040 con 0x55e9f74746b0 2024-04-25T11:35:43.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.254+0000 7fe6b5773740 10 client.15642 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:35:43.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.254+0000 7fe6b5773740 10 client.15642 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:35:43.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.254+0000 7fe6b5773740 20 client.15642 populate_metadata read hostname 'smithi090' 2024-04-25T11:35:43.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.254+0000 7fe6b5773740 1 --2- 172.21.15.90:0/385136665 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e9f7453f60 0x55e9f78525e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.254+0000 7fe6b5773740 4 client.15642 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:35:43.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.254+0000 7fe6b5773740 1 -- 172.21.15.90:0/385136665 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9f77a9990 con 0x55e9f7453f60 2024-04-25T11:35:43.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.254+0000 7fe6a7fff640 1 --2- 172.21.15.90:0/385136665 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e9f7453f60 0x55e9f78525e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.254+0000 7fe6877fe640 10 client.15642 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:43.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.254+0000 7fe6a7fff640 1 --2- 172.21.15.90:0/385136665 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e9f7453f60 0x55e9f78525e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=658f50b631bc2707 server_cookie=550c4d0260f7e80f in_seq=0 out_seq=0 2024-04-25T11:35:43.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.255+0000 7fe6877fe640 1 -- 172.21.15.90:0/385136665 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e9f77a9990 con 0x55e9f7453f60 2024-04-25T11:35:43.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.255+0000 7fe6877fe640 10 client.15642 handle_command_reply: tid=0 2024-04-25T11:35:43.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.255+0000 7fe6b5773740 1 client.15642 shutdown 2024-04-25T11:35:43.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.255+0000 7fe6b5773740 1 -- 172.21.15.90:0/385136665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 msgr2=0x55e9f7840ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.255+0000 7fe6b5773740 1 --2- 172.21.15.90:0/385136665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 0x55e9f7840ac0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe69c01c040 tx=0x7fe69c012f00 comp rx=0 tx=0).stop 2024-04-25T11:35:43.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.255+0000 7fe6b5773740 1 -- 172.21.15.90:0/385136665 shutdown_connections 2024-04-25T11:35:43.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.255+0000 7fe6b5773740 1 --2- 172.21.15.90:0/385136665 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e9f7453f60 0x55e9f78525e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.255+0000 7fe6b5773740 1 --2- 172.21.15.90:0/385136665 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9f7495390 0x55e9f784c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.255+0000 7fe6b5773740 1 --2- 172.21.15.90:0/385136665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 0x55e9f7840ac0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.255+0000 7fe6b5773740 1 --2- 172.21.15.90:0/385136665 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9f746c4d0 0x55e9f7840580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.255+0000 7fe6877fe640 0 client.15642 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:43.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.255+0000 7fe6b5773740 1 -- 172.21.15.90:0/385136665 >> 172.21.15.90:0/385136665 conn(0x55e9f74894e0 msgr2=0x55e9f7837780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:43.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.255+0000 7fe6b5773740 1 -- 172.21.15.90:0/385136665 shutdown_connections 2024-04-25T11:35:43.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.256+0000 7fe6b5773740 1 -- 172.21.15.90:0/385136665 wait complete. 2024-04-25T11:35:43.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.256+0000 7fe6b5773740 20 client.15642 trim_cache size 0 max 16384 2024-04-25T11:35:43.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.274+0000 7fe6b5773740 1 Processor -- start 2024-04-25T11:35:43.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.274+0000 7fe6b5773740 1 -- start start 2024-04-25T11:35:43.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.274+0000 7fe6b5773740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9f7453f60 0x55e9f7837d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.274+0000 7fe6b5773740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9f746c4d0 0x55e9f784c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.274+0000 7fe6b5773740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 0x55e9f784c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.274+0000 7fe6b5773740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a81af9c0 con 0x55e9f74746b0 2024-04-25T11:35:43.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.274+0000 7fe6b5773740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a81b1500 con 0x55e9f746c4d0 2024-04-25T11:35:43.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.274+0000 7fe6b5773740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a81b0090 con 0x55e9f7453f60 2024-04-25T11:35:43.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.274+0000 7fe6ace5a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 0x55e9f784c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.274+0000 7fe6ace5a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 0x55e9f784c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43386/0 (socket says 172.21.15.90:43386) 2024-04-25T11:35:43.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.274+0000 7fe6ace5a640 1 -- 172.21.15.90:0/195077962 learned_addr learned my addr 172.21.15.90:0/195077962 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:43.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.274+0000 7fe6a7fff640 1 --2- 172.21.15.90:0/195077962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9f7453f60 0x55e9f7837d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.274+0000 7fe6ace5a640 1 -- 172.21.15.90:0/195077962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9f7453f60 msgr2=0x55e9f7837d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:43.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.274+0000 7fe6ace5a640 1 --2- 172.21.15.90:0/195077962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9f7453f60 0x55e9f7837d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.274+0000 7fe6a77fe640 1 --2- 172.21.15.90:0/195077962 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9f746c4d0 0x55e9f784c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.274+0000 7fe6ace5a640 1 -- 172.21.15.90:0/195077962 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9f746c4d0 msgr2=0x55e9f784c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:43.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.274+0000 7fe6ace5a640 1 --2- 172.21.15.90:0/195077962 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9f746c4d0 0x55e9f784c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.274+0000 7fe6ace5a640 1 -- 172.21.15.90:0/195077962 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9f746d0f0 con 0x55e9f74746b0 2024-04-25T11:35:43.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.274+0000 7fe6a77fe640 1 --2- 172.21.15.90:0/195077962 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9f746c4d0 0x55e9f784c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:35:43.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.275+0000 7fe6ace5a640 1 --2- 172.21.15.90:0/195077962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 0x55e9f784c990 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0035000 tx=0x7fe6a0043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d130734f43066a8c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:43.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.275+0000 7fe68dffb640 1 -- 172.21.15.90:0/195077962 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a00318e0 con 0x55e9f74746b0 2024-04-25T11:35:43.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.275+0000 7fe68dffb640 1 -- 172.21.15.90:0/195077962 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6a0036040 con 0x55e9f74746b0 2024-04-25T11:35:43.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.275+0000 7fe68dffb640 1 -- 172.21.15.90:0/195077962 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a002fa20 con 0x55e9f74746b0 2024-04-25T11:35:43.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.275+0000 7fe6b5773740 1 -- 172.21.15.90:0/195077962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 msgr2=0x55e9f784c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.275+0000 7fe6b5773740 1 --2- 172.21.15.90:0/195077962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 0x55e9f784c990 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0035000 tx=0x7fe6a0043000 comp rx=0 tx=0).stop 2024-04-25T11:35:43.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.275+0000 7fe6b5773740 1 -- 172.21.15.90:0/195077962 shutdown_connections 2024-04-25T11:35:43.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.275+0000 7fe6b5773740 1 --2- 172.21.15.90:0/195077962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 0x55e9f784c990 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.275+0000 7fe6b5773740 1 --2- 172.21.15.90:0/195077962 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9f746c4d0 0x55e9f784c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.275+0000 7fe6b5773740 1 --2- 172.21.15.90:0/195077962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9f7453f60 0x55e9f7837d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.276+0000 7fe6b5773740 1 -- 172.21.15.90:0/195077962 >> 172.21.15.90:0/195077962 conn(0x55e9f74894e0 msgr2=0x55e9f7837660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:43.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.276+0000 7fe6b5773740 1 -- 172.21.15.90:0/195077962 shutdown_connections 2024-04-25T11:35:43.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.276+0000 7fe6b5773740 1 -- 172.21.15.90:0/195077962 wait complete. 2024-04-25T11:35:43.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.276+0000 7fe6b5773740 1 -- start start 2024-04-25T11:35:43.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.276+0000 7fe6b5773740 1 Processor -- start 2024-04-25T11:35:43.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.276+0000 7fe6b5773740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9f7453f60 0x55e9f78528b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.276+0000 7fe6b5773740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9f746c4d0 0x55e9f7852df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.276+0000 7fe6b5773740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 0x55e9f78b8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.276+0000 7fe6b5773740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a81b0e30 con 0x55e9f74746b0 2024-04-25T11:35:43.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.276+0000 7fe6b5773740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a81b0760 con 0x55e9f7453f60 2024-04-25T11:35:43.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.276+0000 7fe6b5773740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a81ad9a0 con 0x55e9f746c4d0 2024-04-25T11:35:43.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.276+0000 7fe6b5773740 10 client.0 _ll_register_callbacks cb 0x7fe69c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:35:43.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.277+0000 7fe6ace5a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 0x55e9f78b8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.277+0000 7fe6ace5a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 0x55e9f78b8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43388/0 (socket says 172.21.15.90:43388) 2024-04-25T11:35:43.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.277+0000 7fe6ace5a640 1 -- 172.21.15.90:0/1233847743 learned_addr learned my addr 172.21.15.90:0/1233847743 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:43.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.277+0000 7fe6a7fff640 1 --2- 172.21.15.90:0/1233847743 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9f7453f60 0x55e9f78528b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.277+0000 7fe6a77fe640 1 --2- 172.21.15.90:0/1233847743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9f746c4d0 0x55e9f7852df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.277+0000 7fe6ace5a640 1 -- 172.21.15.90:0/1233847743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9f746c4d0 msgr2=0x55e9f7852df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:43.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.277+0000 7fe6ace5a640 1 --2- 172.21.15.90:0/1233847743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9f746c4d0 0x55e9f7852df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.277+0000 7fe6ace5a640 1 -- 172.21.15.90:0/1233847743 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9f7453f60 msgr2=0x55e9f78528b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:43.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.277+0000 7fe6ace5a640 1 --2- 172.21.15.90:0/1233847743 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9f7453f60 0x55e9f78528b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.277+0000 7fe6ace5a640 1 -- 172.21.15.90:0/1233847743 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9f775c4b0 con 0x55e9f74746b0 2024-04-25T11:35:43.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.277+0000 7fe6ace5a640 1 --2- 172.21.15.90:0/1233847743 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 0x55e9f78b8700 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0006010 tx=0x7fe6a0009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9db9c4d2a76d4b53 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:43.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.277+0000 7fe6877fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:35:43.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.277+0000 7fe6877fe640 1 -- 172.21.15.90:0/1233847743 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a005d070 con 0x55e9f74746b0 2024-04-25T11:35:43.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.277+0000 7fe6877fe640 1 -- 172.21.15.90:0/1233847743 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7fe6a0004310 con 0x55e9f74746b0 2024-04-25T11:35:43.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.277+0000 7fe6b5773740 1 -- 172.21.15.90:0/1233847743 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6a0036040 con 0x55e9f74746b0 2024-04-25T11:35:43.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.277+0000 7fe6877fe640 1 -- 172.21.15.90:0/1233847743 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe6a0069050 con 0x55e9f74746b0 2024-04-25T11:35:43.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.277+0000 7fe6b5773740 10 client.15648 fetch_fsmap learned FSMap version 26 2024-04-25T11:35:43.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.277+0000 7fe6b5773740 1 -- 172.21.15.90:0/1233847743 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e9f78b9b40 con 0x55e9f74746b0 2024-04-25T11:35:43.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.278+0000 7fe6877fe640 1 -- 172.21.15.90:0/1233847743 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6a0036040 con 0x55e9f74746b0 2024-04-25T11:35:43.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.278+0000 7fe6877fe640 1 -- 172.21.15.90:0/1233847743 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fe6a004a040 con 0x55e9f74746b0 2024-04-25T11:35:43.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.278+0000 7fe6b5773740 10 client.15648 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:35:43.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.278+0000 7fe6b5773740 10 client.15648 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:35:43.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.278+0000 7fe6b5773740 20 client.15648 populate_metadata read hostname 'smithi090' 2024-04-25T11:35:43.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.278+0000 7fe6b5773740 1 --2- 172.21.15.90:0/1233847743 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e9f7495390 0x55e9f78bc220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.278+0000 7fe6b5773740 4 client.15648 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:35:43.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.278+0000 7fe6b5773740 1 -- 172.21.15.90:0/1233847743 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e9f77a3800 con 0x55e9f7495390 2024-04-25T11:35:43.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.278+0000 7fe6a7fff640 1 --2- 172.21.15.90:0/1233847743 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e9f7495390 0x55e9f78bc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.279+0000 7fe6877fe640 10 client.15648 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:43.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.279+0000 7fe6a7fff640 1 --2- 172.21.15.90:0/1233847743 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e9f7495390 0x55e9f78bc220 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=372bd98b6301cc5 server_cookie=59a7130069062234 in_seq=0 out_seq=0 2024-04-25T11:35:43.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.279+0000 7fe6877fe640 1 -- 172.21.15.90:0/1233847743 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e9f77a3800 con 0x55e9f7495390 2024-04-25T11:35:43.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.279+0000 7fe6877fe640 10 client.15648 handle_command_reply: tid=0 2024-04-25T11:35:43.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.279+0000 7fe6b5773740 1 client.15648 shutdown 2024-04-25T11:35:43.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.279+0000 7fe6b5773740 1 -- 172.21.15.90:0/1233847743 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 msgr2=0x55e9f78b8700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.279+0000 7fe6b5773740 1 --2- 172.21.15.90:0/1233847743 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 0x55e9f78b8700 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0006010 tx=0x7fe6a0009360 comp rx=0 tx=0).stop 2024-04-25T11:35:43.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.279+0000 7fe6b5773740 1 -- 172.21.15.90:0/1233847743 shutdown_connections 2024-04-25T11:35:43.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.279+0000 7fe6b5773740 1 --2- 172.21.15.90:0/1233847743 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e9f7495390 0x55e9f78bc220 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.279+0000 7fe6b5773740 1 --2- 172.21.15.90:0/1233847743 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f74746b0 0x55e9f78b8700 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.279+0000 7fe6b5773740 1 --2- 172.21.15.90:0/1233847743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9f746c4d0 0x55e9f7852df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.279+0000 7fe6b5773740 1 --2- 172.21.15.90:0/1233847743 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9f7453f60 0x55e9f78528b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.279+0000 7fe6877fe640 0 client.15648 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:43.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.279+0000 7fe6b5773740 1 -- 172.21.15.90:0/1233847743 >> 172.21.15.90:0/1233847743 conn(0x55e9f74894e0 msgr2=0x55e9f78419b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:43.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.279+0000 7fe6b5773740 1 -- 172.21.15.90:0/1233847743 shutdown_connections 2024-04-25T11:35:43.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.279+0000 7fe6b5773740 1 -- 172.21.15.90:0/1233847743 wait complete. 2024-04-25T11:35:43.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.279+0000 7fe6b5773740 20 client.15648 trim_cache size 0 max 16384 2024-04-25T11:35:43.287 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:35:43.287 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-25T11:35:43.287 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-25T11:35:43.287 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:35:43.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.280+0000 7fe6ae8bd640 1 -- 172.21.15.90:0/2396290657 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe6900afcf0 msgr2=0x7fe6900b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.280+0000 7fe6ae8bd640 1 --2- 172.21.15.90:0/2396290657 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe6900afcf0 0x7fe6900b2200 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x55e9f7836b60 tx=0x7fe6a00095c0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.280+0000 7fe6ae8bd640 1 -- 172.21.15.90:0/2396290657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6a8104260 msgr2=0x7fe6a819a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.280+0000 7fe6ae8bd640 1 --2- 172.21.15.90:0/2396290657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6a8104260 0x7fe6a819a700 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe69800d740 tx=0x7fe69800dc00 comp rx=0 tx=0).stop 2024-04-25T11:35:43.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.280+0000 7fe6ae8bd640 1 -- 172.21.15.90:0/2396290657 shutdown_connections 2024-04-25T11:35:43.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.280+0000 7fe6ae8bd640 1 --2- 172.21.15.90:0/2396290657 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe6900afcf0 0x7fe6900b2200 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.280+0000 7fe6ae8bd640 1 --2- 172.21.15.90:0/2396290657 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe6a8109fb0 0x7fe6a819ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.280+0000 7fe6ae8bd640 1 --2- 172.21.15.90:0/2396290657 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6a8109b10 0x7fe6a819ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.280+0000 7fe6ae8bd640 1 --2- 172.21.15.90:0/2396290657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6a8104260 0x7fe6a819a700 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.280+0000 7fe6ae8bd640 1 -- 172.21.15.90:0/2396290657 >> 172.21.15.90:0/2396290657 conn(0x7fe6a80ff230 msgr2=0x7fe6a81019a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:43.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.280+0000 7fe6ae8bd640 1 -- 172.21.15.90:0/2396290657 shutdown_connections 2024-04-25T11:35:43.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.280+0000 7fe6ae8bd640 1 -- 172.21.15.90:0/2396290657 wait complete. 2024-04-25T11:35:43.290 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T11:35:43.290 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:35:43.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.410+0000 7f58f14a7640 1 -- 172.21.15.90:0/487657861 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58ec108a20 msgr2=0x7f58ec1025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.410+0000 7f58f14a7640 1 --2- 172.21.15.90:0/487657861 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58ec108a20 0x7f58ec1025b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f58dc009900 tx=0x7f58dc02a410 comp rx=0 tx=0).stop 2024-04-25T11:35:43.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.410+0000 7f58f14a7640 1 -- 172.21.15.90:0/487657861 shutdown_connections 2024-04-25T11:35:43.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.410+0000 7f58f14a7640 1 --2- 172.21.15.90:0/487657861 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f58ec10d380 0x7f58ec10f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.410+0000 7f58f14a7640 1 --2- 172.21.15.90:0/487657861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58ec102af0 0x7f58ec102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.410+0000 7f58f14a7640 1 --2- 172.21.15.90:0/487657861 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58ec108a20 0x7f58ec1025b0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.410+0000 7f58f14a7640 1 -- 172.21.15.90:0/487657861 >> 172.21.15.90:0/487657861 conn(0x7f58ec0781c0 msgr2=0x7f58ec078600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:43.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.410+0000 7f58f14a7640 1 -- 172.21.15.90:0/487657861 shutdown_connections 2024-04-25T11:35:43.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.410+0000 7f58f14a7640 1 -- 172.21.15.90:0/487657861 wait complete. 2024-04-25T11:35:43.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.411+0000 7f58f14a7640 1 Processor -- start 2024-04-25T11:35:43.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.411+0000 7f58f14a7640 1 -- start start 2024-04-25T11:35:43.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.411+0000 7f58f14a7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58ec102af0 0x7f58ec19b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.411+0000 7f58f14a7640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58ec108a20 0x7f58ec19baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.411+0000 7f58f14a7640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f58ec10d380 0x7f58ec19bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.411+0000 7f58f14a7640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec112be0 con 0x7f58ec108a20 2024-04-25T11:35:43.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.411+0000 7f58f14a7640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec112a60 con 0x7f58ec10d380 2024-04-25T11:35:43.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.411+0000 7f58f14a7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec112d60 con 0x7f58ec102af0 2024-04-25T11:35:43.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.411+0000 7f58e3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58ec108a20 0x7f58ec19baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.411+0000 7f58e3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58ec108a20 0x7f58ec19baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43396/0 (socket says 172.21.15.90:43396) 2024-04-25T11:35:43.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.411+0000 7f58e3fff640 1 -- 172.21.15.90:0/1440177886 learned_addr learned my addr 172.21.15.90:0/1440177886 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:43.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.412+0000 7f58eb7fe640 1 --2- 172.21.15.90:0/1440177886 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f58ec10d380 0x7f58ec19bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.412+0000 7f58eaffd640 1 --2- 172.21.15.90:0/1440177886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58ec102af0 0x7f58ec19b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.412+0000 7f58e3fff640 1 -- 172.21.15.90:0/1440177886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58ec102af0 msgr2=0x7f58ec19b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.412+0000 7f58e3fff640 1 --2- 172.21.15.90:0/1440177886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58ec102af0 0x7f58ec19b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.412+0000 7f58e3fff640 1 -- 172.21.15.90:0/1440177886 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f58ec10d380 msgr2=0x7f58ec19bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.412+0000 7f58e3fff640 1 --2- 172.21.15.90:0/1440177886 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f58ec10d380 0x7f58ec19bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.412+0000 7f58e3fff640 1 -- 172.21.15.90:0/1440177886 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58ec1a6870 con 0x7f58ec108a20 2024-04-25T11:35:43.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.412+0000 7f58eb7fe640 1 --2- 172.21.15.90:0/1440177886 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f58ec10d380 0x7f58ec19bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:35:43.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.412+0000 7f58eaffd640 1 --2- 172.21.15.90:0/1440177886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58ec102af0 0x7f58ec19b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:35:43.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.412+0000 7f58e3fff640 1 --2- 172.21.15.90:0/1440177886 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58ec108a20 0x7f58ec19baa0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f58d4005d00 tx=0x7f58d400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:43.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.412+0000 7f58e8ff9640 1 -- 172.21.15.90:0/1440177886 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d4017070 con 0x7f58ec108a20 2024-04-25T11:35:43.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.412+0000 7f58e8ff9640 1 -- 172.21.15.90:0/1440177886 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58d4009c30 con 0x7f58ec108a20 2024-04-25T11:35:43.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.412+0000 7f58f14a7640 1 -- 172.21.15.90:0/1440177886 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58ec1a6b60 con 0x7f58ec108a20 2024-04-25T11:35:43.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.412+0000 7f58e8ff9640 1 -- 172.21.15.90:0/1440177886 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d400f620 con 0x7f58ec108a20 2024-04-25T11:35:43.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.412+0000 7f58f14a7640 1 -- 172.21.15.90:0/1440177886 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58ec1a70b0 con 0x7f58ec108a20 2024-04-25T11:35:43.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.413+0000 7f58f14a7640 1 -- 172.21.15.90:0/1440177886 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58ec069310 con 0x7f58ec108a20 2024-04-25T11:35:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.414+0000 7f58e8ff9640 1 -- 172.21.15.90:0/1440177886 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 23) ==== 147696+0+0 (secure 0 0 0) 0x7f58d401a070 con 0x7f58ec108a20 2024-04-25T11:35:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.415+0000 7f58e8ff9640 1 --2- 172.21.15.90:0/1440177886 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f58d80afd70 0x7f58d80b2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.415+0000 7f58eaffd640 1 --2- 172.21.15.90:0/1440177886 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f58d80afd70 0x7f58d80b2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.415+0000 7f58e8ff9640 1 -- 172.21.15.90:0/1440177886 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f58d40dec40 con 0x7f58ec108a20 2024-04-25T11:35:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.416+0000 7f58eaffd640 1 --2- 172.21.15.90:0/1440177886 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f58d80afd70 0x7f58d80b2280 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f58dc0096c0 tx=0x7f58dc031400 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:43.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.419+0000 7f58e8ff9640 1 -- 172.21.15.90:0/1440177886 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f58d4009de0 con 0x7f58ec108a20 2024-04-25T11:35:43.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.635+0000 7f58f14a7640 1 -- 172.21.15.90:0/1440177886 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f58ec02db50 con 0x7f58ec108a20 2024-04-25T11:35:43.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.636+0000 7f58e8ff9640 1 -- 172.21.15.90:0/1440177886 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11930 (secure 0 0 0) 0x7f58d40a73f0 con 0x7f58ec108a20 2024-04-25T11:35:43.625 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:35:43.625 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:35:06.229136+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24431,24458,24499,24524,24475]},"id":1}]} 2024-04-25T11:35:43.626 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 26 2024-04-25T11:35:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.638+0000 7f58f14a7640 1 -- 172.21.15.90:0/1440177886 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f58d80afd70 msgr2=0x7f58d80b2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.638+0000 7f58f14a7640 1 --2- 172.21.15.90:0/1440177886 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f58d80afd70 0x7f58d80b2280 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f58dc0096c0 tx=0x7f58dc031400 comp rx=0 tx=0).stop 2024-04-25T11:35:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.638+0000 7f58f14a7640 1 -- 172.21.15.90:0/1440177886 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58ec108a20 msgr2=0x7f58ec19baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.638+0000 7f58f14a7640 1 --2- 172.21.15.90:0/1440177886 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58ec108a20 0x7f58ec19baa0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f58d4005d00 tx=0x7f58d400dde0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.638+0000 7f58f14a7640 1 -- 172.21.15.90:0/1440177886 shutdown_connections 2024-04-25T11:35:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.638+0000 7f58f14a7640 1 --2- 172.21.15.90:0/1440177886 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f58d80afd70 0x7f58d80b2280 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.638+0000 7f58f14a7640 1 --2- 172.21.15.90:0/1440177886 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f58ec10d380 0x7f58ec19bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.638+0000 7f58f14a7640 1 --2- 172.21.15.90:0/1440177886 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58ec108a20 0x7f58ec19baa0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.638+0000 7f58f14a7640 1 --2- 172.21.15.90:0/1440177886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58ec102af0 0x7f58ec19b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.638+0000 7f58f14a7640 1 -- 172.21.15.90:0/1440177886 >> 172.21.15.90:0/1440177886 conn(0x7f58ec0781c0 msgr2=0x7f58ec1040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.638+0000 7f58f14a7640 1 -- 172.21.15.90:0/1440177886 shutdown_connections 2024-04-25T11:35:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.638+0000 7f58f14a7640 1 -- 172.21.15.90:0/1440177886 wait complete. 2024-04-25T11:35:43.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T11:35:43.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.761+0000 7fba01027640 1 -- 172.21.15.90:0/409838419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9fc108a30 msgr2=0x7fb9fc1025e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.761+0000 7fba01027640 1 --2- 172.21.15.90:0/409838419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9fc108a30 0x7fb9fc1025e0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec009900 tx=0x7fb9ec02a410 comp rx=0 tx=0).stop 2024-04-25T11:35:43.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.761+0000 7fba01027640 1 -- 172.21.15.90:0/409838419 shutdown_connections 2024-04-25T11:35:43.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.761+0000 7fba01027640 1 --2- 172.21.15.90:0/409838419 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb9fc103190 0x7fb9fc10f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.761+0000 7fba01027640 1 --2- 172.21.15.90:0/409838419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb9fc108ed0 0x7fb9fc102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.761+0000 7fba01027640 1 --2- 172.21.15.90:0/409838419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9fc108a30 0x7fb9fc1025e0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.761+0000 7fba01027640 1 -- 172.21.15.90:0/409838419 >> 172.21.15.90:0/409838419 conn(0x7fb9fc0fe260 msgr2=0x7fb9fc100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:43.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.761+0000 7fba01027640 1 -- 172.21.15.90:0/409838419 shutdown_connections 2024-04-25T11:35:43.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.761+0000 7fba01027640 1 -- 172.21.15.90:0/409838419 wait complete. 2024-04-25T11:35:43.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.762+0000 7fba01027640 1 Processor -- start 2024-04-25T11:35:43.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.762+0000 7fba01027640 1 -- start start 2024-04-25T11:35:43.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.762+0000 7fba01027640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb9fc103190 0x7fb9fc19a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.762+0000 7fba01027640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb9fc108a30 0x7fb9fc19aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.762+0000 7fba01027640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9fc108ed0 0x7fb9fc19ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.762+0000 7fba01027640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb9fc112c30 con 0x7fb9fc108ed0 2024-04-25T11:35:43.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.762+0000 7fba01027640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb9fc112ab0 con 0x7fb9fc108a30 2024-04-25T11:35:43.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.762+0000 7fba01027640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb9fc112db0 con 0x7fb9fc103190 2024-04-25T11:35:43.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.763+0000 7fb9fb59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9fc108ed0 0x7fb9fc19ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.763+0000 7fb9fb59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9fc108ed0 0x7fb9fc19ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43414/0 (socket says 172.21.15.90:43414) 2024-04-25T11:35:43.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.763+0000 7fb9fb59e640 1 -- 172.21.15.90:0/1795189235 learned_addr learned my addr 172.21.15.90:0/1795189235 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:43.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.763+0000 7fb9fa59c640 1 --2- 172.21.15.90:0/1795189235 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb9fc108a30 0x7fb9fc19aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.763+0000 7fb9fad9d640 1 --2- 172.21.15.90:0/1795189235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb9fc103190 0x7fb9fc19a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.763+0000 7fb9fb59e640 1 -- 172.21.15.90:0/1795189235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb9fc103190 msgr2=0x7fb9fc19a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.763+0000 7fb9fb59e640 1 --2- 172.21.15.90:0/1795189235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb9fc103190 0x7fb9fc19a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.763+0000 7fb9fb59e640 1 -- 172.21.15.90:0/1795189235 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb9fc108a30 msgr2=0x7fb9fc19aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.763+0000 7fb9fb59e640 1 --2- 172.21.15.90:0/1795189235 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb9fc108a30 0x7fb9fc19aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.763+0000 7fb9fb59e640 1 -- 172.21.15.90:0/1795189235 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9fc19f5a0 con 0x7fb9fc108ed0 2024-04-25T11:35:43.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.763+0000 7fb9fb59e640 1 --2- 172.21.15.90:0/1795189235 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9fc108ed0 0x7fb9fc19ef60 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb9f400eaa0 tx=0x7fb9f400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:43.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.763+0000 7fb9e3fff640 1 -- 172.21.15.90:0/1795189235 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f400cdb0 con 0x7fb9fc108ed0 2024-04-25T11:35:43.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.763+0000 7fb9e3fff640 1 -- 172.21.15.90:0/1795189235 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9f4005020 con 0x7fb9fc108ed0 2024-04-25T11:35:43.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.763+0000 7fba01027640 1 -- 172.21.15.90:0/1795189235 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9fc19f890 con 0x7fb9fc108ed0 2024-04-25T11:35:43.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.763+0000 7fb9e3fff640 1 -- 172.21.15.90:0/1795189235 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f40116d0 con 0x7fb9fc108ed0 2024-04-25T11:35:43.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.763+0000 7fba01027640 1 -- 172.21.15.90:0/1795189235 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9fc1a7210 con 0x7fb9fc108ed0 2024-04-25T11:35:43.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.766+0000 7fb9e3fff640 1 -- 172.21.15.90:0/1795189235 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 23) ==== 147696+0+0 (secure 0 0 0) 0x7fb9f4011870 con 0x7fb9fc108ed0 2024-04-25T11:35:43.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.766+0000 7fba07fbb740 1 Processor -- start 2024-04-25T11:35:43.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.766+0000 7fba07fbb740 1 -- start start 2024-04-25T11:35:43.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.766+0000 7fb9e3fff640 1 --2- 172.21.15.90:0/1795189235 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb9e40afd70 0x7fb9e40b2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.766+0000 7fba07fbb740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559e823ce390 0x559e8276f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.766+0000 7fba07fbb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 0x559e8276fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.766+0000 7fba07fbb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e823ad6b0 0x559e8277c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.766+0000 7fba07fbb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559e826d4d80 con 0x559e823a54d0 2024-04-25T11:35:43.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.766+0000 7fba07fbb740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x559e8268ea20 con 0x559e823ce390 2024-04-25T11:35:43.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.766+0000 7fba07fbb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559e826d0d40 con 0x559e823ad6b0 2024-04-25T11:35:43.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.766+0000 7fb9fa59c640 1 --2- 172.21.15.90:0/1795189235 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb9e40afd70 0x7fb9e40b2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.766+0000 7fb9e3fff640 1 -- 172.21.15.90:0/1795189235 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7fb9f40df350 con 0x7fb9fc108ed0 2024-04-25T11:35:43.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.766+0000 7fb9fa59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 0x559e8276fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.766+0000 7fb9fad9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559e823ce390 0x559e8276f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.767+0000 7fb9fad9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559e823ce390 0x559e8276f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:32918/0 (socket says 172.21.15.90:32918) 2024-04-25T11:35:43.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.767+0000 7fb9fa59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 0x559e8276fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43416/0 (socket says 172.21.15.90:43416) 2024-04-25T11:35:43.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.767+0000 7fb9fad9d640 1 -- 172.21.15.90:0/1223998930 learned_addr learned my addr 172.21.15.90:0/1223998930 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:43.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.767+0000 7fb9fb59e640 1 --2- 172.21.15.90:0/1223998930 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e823ad6b0 0x559e8277c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.767+0000 7fb9fad9d640 1 -- 172.21.15.90:0/1223998930 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e823ad6b0 msgr2=0x559e8277c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:43.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.767+0000 7fb9fad9d640 1 --2- 172.21.15.90:0/1223998930 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e823ad6b0 0x559e8277c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.767+0000 7fb9fad9d640 1 -- 172.21.15.90:0/1223998930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 msgr2=0x559e8276fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:43.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.767+0000 7fb9fad9d640 1 --2- 172.21.15.90:0/1223998930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 0x559e8276fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.767+0000 7fb9fad9d640 1 -- 172.21.15.90:0/1223998930 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e82699c30 con 0x559e823ce390 2024-04-25T11:35:43.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.767+0000 7fb9fa59c640 1 --2- 172.21.15.90:0/1223998930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 0x559e8276fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:35:43.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.767+0000 7fb9fb59e640 1 --2- 172.21.15.90:0/1223998930 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e823ad6b0 0x559e8277c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:35:43.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.767+0000 7fb9fa59c640 1 --2- 172.21.15.90:0/1795189235 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb9e40afd70 0x7fb9e40b2280 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x559e82770fe0 tx=0x7fb9f000b5b0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:43.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.767+0000 7fb9fad9d640 1 --2- 172.21.15.90:0/1223998930 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559e823ce390 0x559e8276f810 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec000c00 tx=0x7fb9ec032c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ed331193b2d019d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:43.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.767+0000 7fb9e17fa640 1 -- 172.21.15.90:0/1223998930 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec03f070 con 0x559e823ce390 2024-04-25T11:35:43.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.767+0000 7fb9e17fa640 1 -- 172.21.15.90:0/1223998930 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9ec032e40 con 0x559e823ce390 2024-04-25T11:35:43.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.767+0000 7fb9e17fa640 1 -- 172.21.15.90:0/1223998930 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec035070 con 0x559e823ce390 2024-04-25T11:35:43.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.768+0000 7fba07fbb740 1 -- 172.21.15.90:0/1223998930 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559e823ce390 msgr2=0x559e8276f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.768+0000 7fba07fbb740 1 --2- 172.21.15.90:0/1223998930 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559e823ce390 0x559e8276f810 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec000c00 tx=0x7fb9ec032c50 comp rx=0 tx=0).stop 2024-04-25T11:35:43.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.768+0000 7fba07fbb740 1 -- 172.21.15.90:0/1223998930 shutdown_connections 2024-04-25T11:35:43.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.768+0000 7fba07fbb740 1 --2- 172.21.15.90:0/1223998930 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e823ad6b0 0x559e8277c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.768+0000 7fba07fbb740 1 --2- 172.21.15.90:0/1223998930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 0x559e8276fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.768+0000 7fba07fbb740 1 --2- 172.21.15.90:0/1223998930 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559e823ce390 0x559e8276f810 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.768+0000 7fba07fbb740 1 -- 172.21.15.90:0/1223998930 >> 172.21.15.90:0/1223998930 conn(0x559e823c24e0 msgr2=0x559e8276bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:43.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.768+0000 7fba07fbb740 1 -- 172.21.15.90:0/1223998930 shutdown_connections 2024-04-25T11:35:43.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.768+0000 7fba07fbb740 1 -- 172.21.15.90:0/1223998930 wait complete. 2024-04-25T11:35:43.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.769+0000 7fba07fbb740 1 -- start start 2024-04-25T11:35:43.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.769+0000 7fba07fbb740 1 Processor -- start 2024-04-25T11:35:43.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.769+0000 7fba07fbb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 0x559e8277a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.769+0000 7fba07fbb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e823ad6b0 0x559e8277a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.769+0000 7fba07fbb740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559e823ce390 0x559e827861b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.769+0000 7fba07fbb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559e8277cd30 con 0x559e823a54d0 2024-04-25T11:35:43.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.769+0000 7fba07fbb740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x559e8265bdc0 con 0x559e823ce390 2024-04-25T11:35:43.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.769+0000 7fba07fbb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559e8277ceb0 con 0x559e823ad6b0 2024-04-25T11:35:43.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.769+0000 7fba07fbb740 10 client.0 _ll_register_callbacks cb 0x559e82677600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:35:43.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.769+0000 7fb9fad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 0x559e8277a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.769+0000 7fb9fad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 0x559e8277a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43420/0 (socket says 172.21.15.90:43420) 2024-04-25T11:35:43.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.769+0000 7fb9fad9d640 1 -- 172.21.15.90:0/99120843 learned_addr learned my addr 172.21.15.90:0/99120843 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:43.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.769+0000 7fb9fb59e640 1 --2- 172.21.15.90:0/99120843 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559e823ce390 0x559e827861b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.769+0000 7fb9fa59c640 1 --2- 172.21.15.90:0/99120843 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e823ad6b0 0x559e8277a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.769+0000 7fb9fad9d640 1 -- 172.21.15.90:0/99120843 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e823ad6b0 msgr2=0x559e8277a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:43.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.769+0000 7fb9fad9d640 1 --2- 172.21.15.90:0/99120843 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e823ad6b0 0x559e8277a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.769+0000 7fb9fad9d640 1 -- 172.21.15.90:0/99120843 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559e823ce390 msgr2=0x559e827861b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:43.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.769+0000 7fb9fad9d640 1 --2- 172.21.15.90:0/99120843 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559e823ce390 0x559e827861b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.769+0000 7fb9fad9d640 1 -- 172.21.15.90:0/99120843 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e823e63f0 con 0x559e823a54d0 2024-04-25T11:35:43.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.769+0000 7fb9fb59e640 1 --2- 172.21.15.90:0/99120843 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559e823ce390 0x559e827861b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:35:43.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.770+0000 7fb9fad9d640 1 --2- 172.21.15.90:0/99120843 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 0x559e8277a290 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec031440 tx=0x7fb9ec031750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b37d074d5ce38a34 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:43.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.770+0000 7fb9deff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:35:43.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.770+0000 7fb9deff5640 1 -- 172.21.15.90:0/99120843 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec04b060 con 0x559e823a54d0 2024-04-25T11:35:43.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.770+0000 7fba07fbb740 1 -- 172.21.15.90:0/99120843 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559e826e2990 con 0x559e823a54d0 2024-04-25T11:35:43.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.770+0000 7fb9deff5640 1 -- 172.21.15.90:0/99120843 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7fb9ec004440 con 0x559e823a54d0 2024-04-25T11:35:43.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.770+0000 7fb9deff5640 1 -- 172.21.15.90:0/99120843 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb9ec03f030 con 0x559e823a54d0 2024-04-25T11:35:43.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.770+0000 7fba07fbb740 10 client.15666 fetch_fsmap learned FSMap version 26 2024-04-25T11:35:43.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.770+0000 7fba07fbb740 1 -- 172.21.15.90:0/99120843 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559e824b07d0 con 0x559e823a54d0 2024-04-25T11:35:43.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.770+0000 7fb9deff5640 1 -- 172.21.15.90:0/99120843 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x559e826e2990 con 0x559e823a54d0 2024-04-25T11:35:43.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.770+0000 7fb9deff5640 1 -- 172.21.15.90:0/99120843 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fb9ec03b5c0 con 0x559e823a54d0 2024-04-25T11:35:43.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.770+0000 7fba07fbb740 10 client.15666 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:35:43.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.770+0000 7fba07fbb740 10 client.15666 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:35:43.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.770+0000 7fba07fbb740 20 client.15666 populate_metadata read hostname 'smithi090' 2024-04-25T11:35:43.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.770+0000 7fba07fbb740 1 --2- 172.21.15.90:0/99120843 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x559e8238cf60 0x559e8278c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.770+0000 7fba07fbb740 4 client.15666 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:35:43.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.770+0000 7fba07fbb740 1 -- 172.21.15.90:0/99120843 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e826dc800 con 0x559e8238cf60 2024-04-25T11:35:43.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.771+0000 7fb9fa59c640 1 --2- 172.21.15.90:0/99120843 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x559e8238cf60 0x559e8278c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.771+0000 7fb9deff5640 10 client.15666 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:35:43.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.771+0000 7fb9fa59c640 1 --2- 172.21.15.90:0/99120843 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x559e8238cf60 0x559e8278c4a0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bc55850833ce9c55 server_cookie=589d6bb22ba0a63 in_seq=0 out_seq=0 2024-04-25T11:35:43.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.772+0000 7fb9deff5640 1 -- 172.21.15.90:0/99120843 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559e826dc800 con 0x559e8238cf60 2024-04-25T11:35:43.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.772+0000 7fb9deff5640 10 client.15666 handle_command_reply: tid=0 2024-04-25T11:35:43.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.772+0000 7fba07fbb740 1 client.15666 shutdown 2024-04-25T11:35:43.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.772+0000 7fba07fbb740 1 -- 172.21.15.90:0/99120843 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 msgr2=0x559e8277a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.772+0000 7fba07fbb740 1 --2- 172.21.15.90:0/99120843 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 0x559e8277a290 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec031440 tx=0x7fb9ec031750 comp rx=0 tx=0).stop 2024-04-25T11:35:43.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.772+0000 7fba07fbb740 1 -- 172.21.15.90:0/99120843 shutdown_connections 2024-04-25T11:35:43.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.772+0000 7fba07fbb740 1 --2- 172.21.15.90:0/99120843 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x559e8238cf60 0x559e8278c4a0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.772+0000 7fba07fbb740 1 --2- 172.21.15.90:0/99120843 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559e823ce390 0x559e827861b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.772+0000 7fba07fbb740 1 --2- 172.21.15.90:0/99120843 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e823ad6b0 0x559e8277a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.772+0000 7fba07fbb740 1 --2- 172.21.15.90:0/99120843 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 0x559e8277a290 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.772+0000 7fb9deff5640 0 client.15666 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:35:43.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.772+0000 7fba07fbb740 1 -- 172.21.15.90:0/99120843 >> 172.21.15.90:0/99120843 conn(0x559e823c24e0 msgr2=0x559e82771530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:43.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.772+0000 7fba07fbb740 1 -- 172.21.15.90:0/99120843 shutdown_connections 2024-04-25T11:35:43.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.772+0000 7fba07fbb740 1 -- 172.21.15.90:0/99120843 wait complete. 2024-04-25T11:35:43.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.772+0000 7fba07fbb740 20 client.15666 trim_cache size 0 max 16384 2024-04-25T11:35:43.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.786+0000 7fba07fbb740 1 Processor -- start 2024-04-25T11:35:43.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.786+0000 7fba07fbb740 1 -- start start 2024-04-25T11:35:43.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.786+0000 7fba07fbb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 0x559e827861a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.786+0000 7fba07fbb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e823ad6b0 0x559e82779380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.786+0000 7fba07fbb740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559e823ce390 0x559e8277beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.786+0000 7fba07fbb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb9fc1af9e0 con 0x559e823a54d0 2024-04-25T11:35:43.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.786+0000 7fba07fbb740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb9fc1b1520 con 0x559e823ce390 2024-04-25T11:35:43.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.786+0000 7fba07fbb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb9fc1b00b0 con 0x559e823ad6b0 2024-04-25T11:35:43.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.786+0000 7fb9fad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 0x559e827861a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.786+0000 7fb9fad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 0x559e827861a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43426/0 (socket says 172.21.15.90:43426) 2024-04-25T11:35:43.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.786+0000 7fb9fad9d640 1 -- 172.21.15.90:0/2003137948 learned_addr learned my addr 172.21.15.90:0/2003137948 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:43.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.786+0000 7fb9fa59c640 1 --2- 172.21.15.90:0/2003137948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e823ad6b0 0x559e82779380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.786+0000 7fb9fad9d640 1 -- 172.21.15.90:0/2003137948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e823ad6b0 msgr2=0x559e82779380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:43.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.786+0000 7fb9fad9d640 1 --2- 172.21.15.90:0/2003137948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e823ad6b0 0x559e82779380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.786+0000 7fb9fad9d640 1 -- 172.21.15.90:0/2003137948 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559e823ce390 msgr2=0x559e8277beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:43.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.786+0000 7fb9fad9d640 1 --2- 172.21.15.90:0/2003137948 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559e823ce390 0x559e8277beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.786+0000 7fb9fad9d640 1 -- 172.21.15.90:0/2003137948 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e82770120 con 0x559e823a54d0 2024-04-25T11:35:43.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.786+0000 7fb9fa59c640 1 --2- 172.21.15.90:0/2003137948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e823ad6b0 0x559e82779380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:35:43.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.786+0000 7fb9fad9d640 1 --2- 172.21.15.90:0/2003137948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 0x559e827861a0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec03ba10 tx=0x7fb9ec0097c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef1f14e486b0bef7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:43.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.787+0000 7fb9e0ff9640 1 -- 172.21.15.90:0/2003137948 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec04b070 con 0x559e823a54d0 2024-04-25T11:35:43.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.787+0000 7fb9e0ff9640 1 -- 172.21.15.90:0/2003137948 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9ec03be00 con 0x559e823a54d0 2024-04-25T11:35:43.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.787+0000 7fb9e0ff9640 1 -- 172.21.15.90:0/2003137948 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec035040 con 0x559e823a54d0 2024-04-25T11:35:43.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.787+0000 7fba07fbb740 1 -- 172.21.15.90:0/2003137948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 msgr2=0x559e827861a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.787+0000 7fba07fbb740 1 --2- 172.21.15.90:0/2003137948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 0x559e827861a0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec03ba10 tx=0x7fb9ec0097c0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.787+0000 7fba07fbb740 1 -- 172.21.15.90:0/2003137948 shutdown_connections 2024-04-25T11:35:43.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.787+0000 7fba07fbb740 1 --2- 172.21.15.90:0/2003137948 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559e823ce390 0x559e8277beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.787+0000 7fba07fbb740 1 --2- 172.21.15.90:0/2003137948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e823ad6b0 0x559e82779380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.787+0000 7fba07fbb740 1 --2- 172.21.15.90:0/2003137948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 0x559e827861a0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.787+0000 7fba07fbb740 1 -- 172.21.15.90:0/2003137948 >> 172.21.15.90:0/2003137948 conn(0x559e823c24e0 msgr2=0x559e8277b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:43.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.787+0000 7fba07fbb740 1 -- 172.21.15.90:0/2003137948 shutdown_connections 2024-04-25T11:35:43.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.787+0000 7fba07fbb740 1 -- 172.21.15.90:0/2003137948 wait complete. 2024-04-25T11:35:43.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.788+0000 7fba07fbb740 1 -- start start 2024-04-25T11:35:43.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.788+0000 7fba07fbb740 1 Processor -- start 2024-04-25T11:35:43.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.788+0000 7fba07fbb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 0x559e8278f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.788+0000 7fba07fbb740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559e823ad6b0 0x559e8278f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.788+0000 7fba07fbb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e823ce390 0x559e8278c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.788+0000 7fba07fbb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb9fc1b0e50 con 0x559e823a54d0 2024-04-25T11:35:43.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.788+0000 7fba07fbb740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb9fc1b0780 con 0x559e823ad6b0 2024-04-25T11:35:43.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.788+0000 7fba07fbb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb9fc1ad9c0 con 0x559e823ce390 2024-04-25T11:35:43.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.788+0000 7fba07fbb740 10 client.0 _ll_register_callbacks cb 0x7fb9f000bc80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:35:43.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.788+0000 7fb9fad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 0x559e8278f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.788+0000 7fb9fad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 0x559e8278f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43436/0 (socket says 172.21.15.90:43436) 2024-04-25T11:35:43.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.788+0000 7fb9fad9d640 1 -- 172.21.15.90:0/141746349 learned_addr learned my addr 172.21.15.90:0/141746349 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:43.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.788+0000 7fb9fad9d640 1 -- 172.21.15.90:0/141746349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e823ce390 msgr2=0x559e8278c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:43.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.788+0000 7fb9fa59c640 1 --2- 172.21.15.90:0/141746349 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559e823ad6b0 0x559e8278f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.788+0000 7fb9fad9d640 1 --2- 172.21.15.90:0/141746349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e823ce390 0x559e8278c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.788+0000 7fb9fad9d640 1 -- 172.21.15.90:0/141746349 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559e823ad6b0 msgr2=0x559e8278f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:43.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.788+0000 7fb9fad9d640 1 --2- 172.21.15.90:0/141746349 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559e823ad6b0 0x559e8278f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.788+0000 7fb9fad9d640 1 -- 172.21.15.90:0/141746349 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e8278cf30 con 0x559e823a54d0 2024-04-25T11:35:43.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.789+0000 7fb9fad9d640 1 --2- 172.21.15.90:0/141746349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 0x559e8278f390 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec03b990 tx=0x7fb9ec0092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51ef358c978dbac1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:43.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.789+0000 7fb9deff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:35:43.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.789+0000 7fb9deff5640 1 -- 172.21.15.90:0/141746349 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec04b070 con 0x559e823a54d0 2024-04-25T11:35:43.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.789+0000 7fb9deff5640 1 -- 172.21.15.90:0/141746349 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7fb9ec03b630 con 0x559e823a54d0 2024-04-25T11:35:43.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.789+0000 7fba07fbb740 1 -- 172.21.15.90:0/141746349 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9ec03be00 con 0x559e823a54d0 2024-04-25T11:35:43.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.789+0000 7fb9deff5640 1 -- 172.21.15.90:0/141746349 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb9ec069050 con 0x559e823a54d0 2024-04-25T11:35:43.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.789+0000 7fba07fbb740 10 client.15672 fetch_fsmap learned FSMap version 26 2024-04-25T11:35:43.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.789+0000 7fba07fbb740 1 -- 172.21.15.90:0/141746349 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559e827f3e80 con 0x559e823a54d0 2024-04-25T11:35:43.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.789+0000 7fb9deff5640 1 -- 172.21.15.90:0/141746349 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9ec03be00 con 0x559e823a54d0 2024-04-25T11:35:43.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.789+0000 7fb9deff5640 1 -- 172.21.15.90:0/141746349 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fb9ec060070 con 0x559e823a54d0 2024-04-25T11:35:43.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.789+0000 7fba07fbb740 10 client.15672 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:35:43.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.789+0000 7fba07fbb740 10 client.15672 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:35:43.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.789+0000 7fba07fbb740 20 client.15672 populate_metadata read hostname 'smithi090' 2024-04-25T11:35:43.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.789+0000 7fba07fbb740 1 --2- 172.21.15.90:0/141746349 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x559e823623a0 0x559e827f64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.789+0000 7fba07fbb740 4 client.15672 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:35:43.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.789+0000 7fba07fbb740 1 -- 172.21.15.90:0/141746349 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559e82657610 con 0x559e823623a0 2024-04-25T11:35:43.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.790+0000 7fb9fa59c640 1 --2- 172.21.15.90:0/141746349 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x559e823623a0 0x559e827f64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.790+0000 7fb9deff5640 10 client.15672 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:35:43.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.790+0000 7fb9fa59c640 1 --2- 172.21.15.90:0/141746349 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x559e823623a0 0x559e827f64a0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=342011f7fa77d274 server_cookie=e254d280d44054f0 in_seq=0 out_seq=0 2024-04-25T11:35:43.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.790+0000 7fb9deff5640 1 -- 172.21.15.90:0/141746349 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fb9f002be50 con 0x559e823623a0 2024-04-25T11:35:43.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.790+0000 7fb9deff5640 10 client.15672 handle_command_reply: tid=0 2024-04-25T11:35:43.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.790+0000 7fba07fbb740 1 client.15672 shutdown 2024-04-25T11:35:43.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.790+0000 7fba07fbb740 1 -- 172.21.15.90:0/141746349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 msgr2=0x559e8278f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.790+0000 7fba07fbb740 1 --2- 172.21.15.90:0/141746349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 0x559e8278f390 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec03b990 tx=0x7fb9ec0092d0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.790+0000 7fba07fbb740 1 -- 172.21.15.90:0/141746349 shutdown_connections 2024-04-25T11:35:43.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.790+0000 7fba07fbb740 1 --2- 172.21.15.90:0/141746349 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x559e823623a0 0x559e827f64a0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.790+0000 7fba07fbb740 1 --2- 172.21.15.90:0/141746349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e823ce390 0x559e8278c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.790+0000 7fb9deff5640 0 client.15672 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:35:43.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.790+0000 7fba07fbb740 1 --2- 172.21.15.90:0/141746349 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559e823ad6b0 0x559e8278f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.790+0000 7fba07fbb740 1 --2- 172.21.15.90:0/141746349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e823a54d0 0x559e8278f390 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.790+0000 7fba07fbb740 1 -- 172.21.15.90:0/141746349 >> 172.21.15.90:0/141746349 conn(0x559e823c24e0 msgr2=0x559e827790e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:43.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.790+0000 7fba07fbb740 1 -- 172.21.15.90:0/141746349 shutdown_connections 2024-04-25T11:35:43.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.790+0000 7fba07fbb740 1 -- 172.21.15.90:0/141746349 wait complete. 2024-04-25T11:35:43.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.791+0000 7fba07fbb740 20 client.15672 trim_cache size 0 max 16384 2024-04-25T11:35:43.792 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:35:43.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.791+0000 7fba01027640 1 -- 172.21.15.90:0/1795189235 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb9e40afd70 msgr2=0x7fb9e40b2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.791+0000 7fba01027640 1 --2- 172.21.15.90:0/1795189235 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb9e40afd70 0x7fb9e40b2280 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x559e82770fe0 tx=0x7fb9f000b5b0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.791+0000 7fba01027640 1 -- 172.21.15.90:0/1795189235 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9fc108ed0 msgr2=0x7fb9fc19ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.791+0000 7fba01027640 1 --2- 172.21.15.90:0/1795189235 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9fc108ed0 0x7fb9fc19ef60 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb9f400eaa0 tx=0x7fb9f400ef60 comp rx=0 tx=0).stop 2024-04-25T11:35:43.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.791+0000 7fba01027640 1 -- 172.21.15.90:0/1795189235 shutdown_connections 2024-04-25T11:35:43.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.791+0000 7fba01027640 1 --2- 172.21.15.90:0/1795189235 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb9e40afd70 0x7fb9e40b2280 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.791+0000 7fba01027640 1 --2- 172.21.15.90:0/1795189235 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9fc108ed0 0x7fb9fc19ef60 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.791+0000 7fba01027640 1 --2- 172.21.15.90:0/1795189235 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb9fc108a30 0x7fb9fc19aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.791+0000 7fba01027640 1 --2- 172.21.15.90:0/1795189235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb9fc103190 0x7fb9fc19a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.791+0000 7fba01027640 1 -- 172.21.15.90:0/1795189235 >> 172.21.15.90:0/1795189235 conn(0x7fb9fc0fe260 msgr2=0x7fb9fc100a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:43.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.791+0000 7fba01027640 1 -- 172.21.15.90:0/1795189235 shutdown_connections 2024-04-25T11:35:43.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.791+0000 7fba01027640 1 -- 172.21.15.90:0/1795189235 wait complete. 2024-04-25T11:35:43.794 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T11:35:43.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.916+0000 7f608f5c5640 1 -- 172.21.15.90:0/3473182360 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f608810e040 msgr2=0x7f6088110530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.916+0000 7f608f5c5640 1 --2- 172.21.15.90:0/3473182360 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f608810e040 0x7f6088110530 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f6080009ca0 tx=0x7f608002a8c0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.916+0000 7f608f5c5640 1 -- 172.21.15.90:0/3473182360 shutdown_connections 2024-04-25T11:35:43.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.916+0000 7f608f5c5640 1 --2- 172.21.15.90:0/3473182360 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f608810e040 0x7f6088110530 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.916+0000 7f608f5c5640 1 --2- 172.21.15.90:0/3473182360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f608810b650 0x7f608810da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.916+0000 7f608f5c5640 1 --2- 172.21.15.90:0/3473182360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6088104bf0 0x7f6088105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.916+0000 7f608f5c5640 1 -- 172.21.15.90:0/3473182360 >> 172.21.15.90:0/3473182360 conn(0x7f6088078100 msgr2=0x7f6088078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:43.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.916+0000 7f608f5c5640 1 -- 172.21.15.90:0/3473182360 shutdown_connections 2024-04-25T11:35:43.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.916+0000 7f608f5c5640 1 -- 172.21.15.90:0/3473182360 wait complete. 2024-04-25T11:35:43.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.917+0000 7f608f5c5640 1 Processor -- start 2024-04-25T11:35:43.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.917+0000 7f608f5c5640 1 -- start start 2024-04-25T11:35:43.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.917+0000 7f608f5c5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6088104bf0 0x7f608819a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.917+0000 7f608f5c5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f608810b650 0x7f608819ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.917+0000 7f608f5c5640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f608810e040 0x7f608819eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.917+0000 7f608f5c5640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6088113ca0 con 0x7f608810b650 2024-04-25T11:35:43.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.917+0000 7f608f5c5640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6088113b20 con 0x7f608810e040 2024-04-25T11:35:43.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.917+0000 7f608f5c5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6088113e20 con 0x7f6088104bf0 2024-04-25T11:35:43.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.917+0000 7f608cb60640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f608810b650 0x7f608819ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.917+0000 7f608cb60640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f608810b650 0x7f608819ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43456/0 (socket says 172.21.15.90:43456) 2024-04-25T11:35:43.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.917+0000 7f608cb60640 1 -- 172.21.15.90:0/60106656 learned_addr learned my addr 172.21.15.90:0/60106656 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:43.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.917+0000 7f608db62640 1 --2- 172.21.15.90:0/60106656 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f608810e040 0x7f608819eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.917+0000 7f608d361640 1 --2- 172.21.15.90:0/60106656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6088104bf0 0x7f608819a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.917+0000 7f608cb60640 1 -- 172.21.15.90:0/60106656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6088104bf0 msgr2=0x7f608819a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.917+0000 7f608cb60640 1 --2- 172.21.15.90:0/60106656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6088104bf0 0x7f608819a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.917+0000 7f608cb60640 1 -- 172.21.15.90:0/60106656 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f608810e040 msgr2=0x7f608819eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.917+0000 7f608cb60640 1 --2- 172.21.15.90:0/60106656 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f608810e040 0x7f608819eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.917+0000 7f608cb60640 1 -- 172.21.15.90:0/60106656 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f608819f520 con 0x7f608810b650 2024-04-25T11:35:43.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.918+0000 7f608cb60640 1 --2- 172.21.15.90:0/60106656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f608810b650 0x7f608819ac20 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f607800d950 tx=0x7f607800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:43.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.918+0000 7f607e7fc640 1 -- 172.21.15.90:0/60106656 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6078017070 con 0x7f608810b650 2024-04-25T11:35:43.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.918+0000 7f607e7fc640 1 -- 172.21.15.90:0/60106656 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6078009c30 con 0x7f608810b650 2024-04-25T11:35:43.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.918+0000 7f608f5c5640 1 -- 172.21.15.90:0/60106656 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f608819f810 con 0x7f608810b650 2024-04-25T11:35:43.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.918+0000 7f607e7fc640 1 -- 172.21.15.90:0/60106656 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607800f620 con 0x7f608810b650 2024-04-25T11:35:43.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.918+0000 7f608f5c5640 1 -- 172.21.15.90:0/60106656 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60881a7190 con 0x7f608810b650 2024-04-25T11:35:43.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.920+0000 7f6096476740 1 Processor -- start 2024-04-25T11:35:43.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.920+0000 7f607e7fc640 1 -- 172.21.15.90:0/60106656 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 23) ==== 147696+0+0 (secure 0 0 0) 0x7f6078010460 con 0x7f608810b650 2024-04-25T11:35:43.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.921+0000 7f6096476740 1 -- start start 2024-04-25T11:35:43.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.921+0000 7f6096476740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557b0d57e390 0x557b0d91f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.921+0000 7f6096476740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d5554d0 0x557b0d91fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.921+0000 7f6096476740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b0d55d6b0 0x557b0d92c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.921+0000 7f6096476740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557b0d884d80 con 0x557b0d5554d0 2024-04-25T11:35:43.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.921+0000 7f6096476740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x557b0d83ea20 con 0x557b0d57e390 2024-04-25T11:35:43.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.921+0000 7f6096476740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x557b0d880d40 con 0x557b0d55d6b0 2024-04-25T11:35:43.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.921+0000 7f608d361640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d5554d0 0x557b0d91fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.921+0000 7f608d361640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d5554d0 0x557b0d91fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43464/0 (socket says 172.21.15.90:43464) 2024-04-25T11:35:43.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.921+0000 7f608d361640 1 -- 172.21.15.90:0/2952890836 learned_addr learned my addr 172.21.15.90:0/2952890836 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:43.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.921+0000 7f608cb60640 1 --2- 172.21.15.90:0/2952890836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b0d55d6b0 0x557b0d92c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.921+0000 7f608db62640 1 --2- 172.21.15.90:0/2952890836 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557b0d57e390 0x557b0d91f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.921+0000 7f607e7fc640 1 --2- 172.21.15.90:0/60106656 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f60740afcf0 0x7f60740b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.921+0000 7f608d361640 1 -- 172.21.15.90:0/2952890836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b0d55d6b0 msgr2=0x557b0d92c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:43.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.921+0000 7f608d361640 1 --2- 172.21.15.90:0/2952890836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b0d55d6b0 0x557b0d92c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.921+0000 7f608d361640 1 -- 172.21.15.90:0/2952890836 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557b0d57e390 msgr2=0x557b0d91f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:43.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.921+0000 7f608d361640 1 --2- 172.21.15.90:0/2952890836 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557b0d57e390 0x557b0d91f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.921+0000 7f608d361640 1 -- 172.21.15.90:0/2952890836 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b0d849c30 con 0x557b0d5554d0 2024-04-25T11:35:43.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.921+0000 7f608db62640 1 --2- 172.21.15.90:0/60106656 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f60740afcf0 0x7f60740b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.921+0000 7f607e7fc640 1 -- 172.21.15.90:0/60106656 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f60780dee80 con 0x7f608810b650 2024-04-25T11:35:43.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.922+0000 7f608d361640 1 --2- 172.21.15.90:0/2952890836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d5554d0 0x557b0d91fd50 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f6084009bc0 tx=0x7f608400e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=425ab376180e8cc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:43.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.922+0000 7f60737fe640 1 -- 172.21.15.90:0/2952890836 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6084016070 con 0x557b0d5554d0 2024-04-25T11:35:43.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.922+0000 7f60737fe640 1 -- 172.21.15.90:0/2952890836 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6084011920 con 0x557b0d5554d0 2024-04-25T11:35:43.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.922+0000 7f60737fe640 1 -- 172.21.15.90:0/2952890836 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60840107f0 con 0x557b0d5554d0 2024-04-25T11:35:43.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.923+0000 7f6096476740 1 -- 172.21.15.90:0/2952890836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d5554d0 msgr2=0x557b0d91fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.923+0000 7f6096476740 1 --2- 172.21.15.90:0/2952890836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d5554d0 0x557b0d91fd50 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f6084009bc0 tx=0x7f608400e6c0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.923+0000 7f608db62640 1 --2- 172.21.15.90:0/60106656 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f60740afcf0 0x7f60740b2200 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6080005cc0 tx=0x7f60800095c0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:43.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.923+0000 7f6096476740 1 -- 172.21.15.90:0/2952890836 shutdown_connections 2024-04-25T11:35:43.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.923+0000 7f6096476740 1 --2- 172.21.15.90:0/2952890836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b0d55d6b0 0x557b0d92c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.923+0000 7f6096476740 1 --2- 172.21.15.90:0/2952890836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d5554d0 0x557b0d91fd50 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.923+0000 7f6096476740 1 --2- 172.21.15.90:0/2952890836 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557b0d57e390 0x557b0d91f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.923+0000 7f6096476740 1 -- 172.21.15.90:0/2952890836 >> 172.21.15.90:0/2952890836 conn(0x557b0d5724e0 msgr2=0x557b0d91bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:43.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.923+0000 7f6096476740 1 -- 172.21.15.90:0/2952890836 shutdown_connections 2024-04-25T11:35:43.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.923+0000 7f6096476740 1 -- 172.21.15.90:0/2952890836 wait complete. 2024-04-25T11:35:43.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.923+0000 7f6096476740 1 -- start start 2024-04-25T11:35:43.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.923+0000 7f6096476740 1 Processor -- start 2024-04-25T11:35:43.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.924+0000 7f6096476740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b0d5554d0 0x557b0d92a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.924+0000 7f6096476740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557b0d55d6b0 0x557b0d92a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.924+0000 7f6096476740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d57e390 0x557b0d936210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.924+0000 7f6096476740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557b0d92cd90 con 0x557b0d57e390 2024-04-25T11:35:43.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.924+0000 7f6096476740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x557b0d80bdc0 con 0x557b0d55d6b0 2024-04-25T11:35:43.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.924+0000 7f6096476740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x557b0d92cf10 con 0x557b0d5554d0 2024-04-25T11:35:43.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.924+0000 7f6096476740 10 client.0 _ll_register_callbacks cb 0x557b0d827600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:35:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.924+0000 7f608db62640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d57e390 0x557b0d936210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.924+0000 7f608db62640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d57e390 0x557b0d936210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43468/0 (socket says 172.21.15.90:43468) 2024-04-25T11:35:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.924+0000 7f608db62640 1 -- 172.21.15.90:0/1746533310 learned_addr learned my addr 172.21.15.90:0/1746533310 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.924+0000 7f608d361640 1 --2- 172.21.15.90:0/1746533310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b0d5554d0 0x557b0d92a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.924+0000 7f608cb60640 1 --2- 172.21.15.90:0/1746533310 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557b0d55d6b0 0x557b0d92a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.924+0000 7f608db62640 1 -- 172.21.15.90:0/1746533310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b0d5554d0 msgr2=0x557b0d92a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.924+0000 7f608db62640 1 --2- 172.21.15.90:0/1746533310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b0d5554d0 0x557b0d92a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.924+0000 7f608db62640 1 -- 172.21.15.90:0/1746533310 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557b0d55d6b0 msgr2=0x557b0d92a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.924+0000 7f608db62640 1 --2- 172.21.15.90:0/1746533310 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557b0d55d6b0 0x557b0d92a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.924+0000 7f608db62640 1 -- 172.21.15.90:0/1746533310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b0d5963f0 con 0x557b0d57e390 2024-04-25T11:35:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.924+0000 7f608d361640 1 --2- 172.21.15.90:0/1746533310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b0d5554d0 0x557b0d92a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:35:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.924+0000 7f608db62640 1 --2- 172.21.15.90:0/1746533310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d57e390 0x557b0d936210 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f6080035000 tx=0x7f608002a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e197b8a7d975f20 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.924+0000 7f6070ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:35:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.924+0000 7f6070ff9640 1 -- 172.21.15.90:0/1746533310 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6080004450 con 0x557b0d57e390 2024-04-25T11:35:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.924+0000 7f6070ff9640 1 -- 172.21.15.90:0/1746533310 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f60800045f0 con 0x557b0d57e390 2024-04-25T11:35:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.924+0000 7f6096476740 1 -- 172.21.15.90:0/1746533310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6084011920 con 0x557b0d57e390 2024-04-25T11:35:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.925+0000 7f6070ff9640 1 -- 172.21.15.90:0/1746533310 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f60800047f0 con 0x557b0d57e390 2024-04-25T11:35:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.925+0000 7f6096476740 10 client.15684 fetch_fsmap learned FSMap version 26 2024-04-25T11:35:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.925+0000 7f6096476740 1 -- 172.21.15.90:0/1746533310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557b0d6607d0 con 0x557b0d57e390 2024-04-25T11:35:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.925+0000 7f6070ff9640 1 -- 172.21.15.90:0/1746533310 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6084011920 con 0x557b0d57e390 2024-04-25T11:35:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.925+0000 7f6070ff9640 1 -- 172.21.15.90:0/1746533310 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f6080030860 con 0x557b0d57e390 2024-04-25T11:35:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.925+0000 7f6096476740 10 client.15684 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:35:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.925+0000 7f6096476740 10 client.15684 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:35:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.925+0000 7f6096476740 20 client.15684 populate_metadata read hostname 'smithi090' 2024-04-25T11:35:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.925+0000 7f6096476740 1 --2- 172.21.15.90:0/1746533310 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x557b0d53cf60 0x557b0d93c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.925+0000 7f6096476740 4 client.15684 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:35:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.925+0000 7f6096476740 1 -- 172.21.15.90:0/1746533310 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b0d892990 con 0x557b0d53cf60 2024-04-25T11:35:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.925+0000 7f608d361640 1 --2- 172.21.15.90:0/1746533310 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x557b0d53cf60 0x557b0d93c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.926+0000 7f6070ff9640 10 client.15684 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:35:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.926+0000 7f608d361640 1 --2- 172.21.15.90:0/1746533310 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x557b0d53cf60 0x557b0d93c6e0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6c61fe95f99c10c7 server_cookie=903b25937555b6a0 in_seq=0 out_seq=0 2024-04-25T11:35:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.927+0000 7f6070ff9640 1 -- 172.21.15.90:0/1746533310 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557b0d892990 con 0x557b0d53cf60 2024-04-25T11:35:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.927+0000 7f6070ff9640 10 client.15684 handle_command_reply: tid=0 2024-04-25T11:35:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.927+0000 7f6096476740 1 client.15684 shutdown 2024-04-25T11:35:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.927+0000 7f6096476740 1 -- 172.21.15.90:0/1746533310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d57e390 msgr2=0x557b0d936210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.927+0000 7f6096476740 1 --2- 172.21.15.90:0/1746533310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d57e390 0x557b0d936210 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f6080035000 tx=0x7f608002a8c0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.927+0000 7f6096476740 1 -- 172.21.15.90:0/1746533310 shutdown_connections 2024-04-25T11:35:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.927+0000 7f6096476740 1 --2- 172.21.15.90:0/1746533310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d57e390 0x557b0d936210 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.927+0000 7f6096476740 1 --2- 172.21.15.90:0/1746533310 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x557b0d53cf60 0x557b0d93c6e0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.927+0000 7f6096476740 1 --2- 172.21.15.90:0/1746533310 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557b0d55d6b0 0x557b0d92a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.927+0000 7f6096476740 1 --2- 172.21.15.90:0/1746533310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b0d5554d0 0x557b0d92a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.927+0000 7f6096476740 1 -- 172.21.15.90:0/1746533310 >> 172.21.15.90:0/1746533310 conn(0x557b0d5724e0 msgr2=0x557b0d921530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:43.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.927+0000 7f6070ff9640 0 client.15684 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:35:43.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.927+0000 7f6096476740 1 -- 172.21.15.90:0/1746533310 shutdown_connections 2024-04-25T11:35:43.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.927+0000 7f6096476740 1 -- 172.21.15.90:0/1746533310 wait complete. 2024-04-25T11:35:43.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.927+0000 7f6096476740 20 client.15684 trim_cache size 0 max 16384 2024-04-25T11:35:43.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.940+0000 7f6096476740 1 Processor -- start 2024-04-25T11:35:43.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.941+0000 7f6096476740 1 -- start start 2024-04-25T11:35:43.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.941+0000 7f6096476740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b0d5554d0 0x557b0d935e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.941+0000 7f6096476740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d55d6b0 0x557b0d941ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.941+0000 7f6096476740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557b0d57e390 0x557b0d942420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.941+0000 7f6096476740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f60881afa80 con 0x557b0d55d6b0 2024-04-25T11:35:43.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.941+0000 7f6096476740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f60881b15c0 con 0x557b0d57e390 2024-04-25T11:35:43.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.941+0000 7f6096476740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f60881b0150 con 0x557b0d5554d0 2024-04-25T11:35:43.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.941+0000 7f608cb60640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d55d6b0 0x557b0d941ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.941+0000 7f608cb60640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d55d6b0 0x557b0d941ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43484/0 (socket says 172.21.15.90:43484) 2024-04-25T11:35:43.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.941+0000 7f608cb60640 1 -- 172.21.15.90:0/4065779081 learned_addr learned my addr 172.21.15.90:0/4065779081 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:43.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.941+0000 7f608db62640 1 --2- 172.21.15.90:0/4065779081 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557b0d57e390 0x557b0d942420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.941+0000 7f608d361640 1 --2- 172.21.15.90:0/4065779081 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b0d5554d0 0x557b0d935e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.941+0000 7f608cb60640 1 -- 172.21.15.90:0/4065779081 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b0d5554d0 msgr2=0x557b0d935e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:43.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.941+0000 7f608cb60640 1 --2- 172.21.15.90:0/4065779081 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b0d5554d0 0x557b0d935e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.941+0000 7f608cb60640 1 -- 172.21.15.90:0/4065779081 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557b0d57e390 msgr2=0x557b0d942420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:43.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.941+0000 7f608cb60640 1 --2- 172.21.15.90:0/4065779081 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557b0d57e390 0x557b0d942420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.941+0000 7f608cb60640 1 -- 172.21.15.90:0/4065779081 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b0d92ae20 con 0x557b0d55d6b0 2024-04-25T11:35:43.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.941+0000 7f608db62640 1 --2- 172.21.15.90:0/4065779081 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557b0d57e390 0x557b0d942420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:35:43.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.941+0000 7f608d361640 1 --2- 172.21.15.90:0/4065779081 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b0d5554d0 0x557b0d935e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:35:43.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.941+0000 7f608cb60640 1 --2- 172.21.15.90:0/4065779081 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d55d6b0 0x557b0d941ee0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f60780a4120 tx=0x7f6078000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22b5b19a0267cb6b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:43.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.941+0000 7f6072ffd640 1 -- 172.21.15.90:0/4065779081 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6078020040 con 0x557b0d55d6b0 2024-04-25T11:35:43.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.942+0000 7f6072ffd640 1 -- 172.21.15.90:0/4065779081 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60780a69c0 con 0x557b0d55d6b0 2024-04-25T11:35:43.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.942+0000 7f6072ffd640 1 -- 172.21.15.90:0/4065779081 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60780afd10 con 0x557b0d55d6b0 2024-04-25T11:35:43.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.942+0000 7f6096476740 1 -- 172.21.15.90:0/4065779081 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d55d6b0 msgr2=0x557b0d941ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.942+0000 7f6096476740 1 --2- 172.21.15.90:0/4065779081 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d55d6b0 0x557b0d941ee0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f60780a4120 tx=0x7f6078000bc0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.942+0000 7f6096476740 1 -- 172.21.15.90:0/4065779081 shutdown_connections 2024-04-25T11:35:43.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.942+0000 7f6096476740 1 --2- 172.21.15.90:0/4065779081 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557b0d57e390 0x557b0d942420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.942+0000 7f6096476740 1 --2- 172.21.15.90:0/4065779081 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d55d6b0 0x557b0d941ee0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.942+0000 7f6096476740 1 --2- 172.21.15.90:0/4065779081 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b0d5554d0 0x557b0d935e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.942+0000 7f6096476740 1 -- 172.21.15.90:0/4065779081 >> 172.21.15.90:0/4065779081 conn(0x557b0d5724e0 msgr2=0x557b0d9289b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:43.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.942+0000 7f6096476740 1 -- 172.21.15.90:0/4065779081 shutdown_connections 2024-04-25T11:35:43.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.942+0000 7f6096476740 1 -- 172.21.15.90:0/4065779081 wait complete. 2024-04-25T11:35:43.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.943+0000 7f6096476740 1 -- start start 2024-04-25T11:35:43.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.943+0000 7f6096476740 1 Processor -- start 2024-04-25T11:35:43.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.943+0000 7f6096476740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557b0d5554d0 0x557b0d92dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.943+0000 7f6096476740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b0d55d6b0 0x557b0d92e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.943+0000 7f6096476740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d57e390 0x557b0d92e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.943+0000 7f6096476740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f60881b0ef0 con 0x557b0d57e390 2024-04-25T11:35:43.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.943+0000 7f6096476740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f60881b0820 con 0x557b0d5554d0 2024-04-25T11:35:43.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.943+0000 7f6096476740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f60881ada60 con 0x557b0d55d6b0 2024-04-25T11:35:43.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.943+0000 7f6096476740 10 client.0 _ll_register_callbacks cb 0x7f6080009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:35:43.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.943+0000 7f608db62640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d57e390 0x557b0d92e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.943+0000 7f608db62640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d57e390 0x557b0d92e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43488/0 (socket says 172.21.15.90:43488) 2024-04-25T11:35:43.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.943+0000 7f608db62640 1 -- 172.21.15.90:0/297966085 learned_addr learned my addr 172.21.15.90:0/297966085 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:43.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.943+0000 7f608d361640 1 --2- 172.21.15.90:0/297966085 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557b0d5554d0 0x557b0d92dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.943+0000 7f608cb60640 1 --2- 172.21.15.90:0/297966085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b0d55d6b0 0x557b0d92e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.943+0000 7f608db62640 1 -- 172.21.15.90:0/297966085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b0d55d6b0 msgr2=0x557b0d92e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:43.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.943+0000 7f608db62640 1 --2- 172.21.15.90:0/297966085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b0d55d6b0 0x557b0d92e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.943+0000 7f608db62640 1 -- 172.21.15.90:0/297966085 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557b0d5554d0 msgr2=0x557b0d92dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:43.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.944+0000 7f608db62640 1 --2- 172.21.15.90:0/297966085 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557b0d5554d0 0x557b0d92dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.944+0000 7f608db62640 1 -- 172.21.15.90:0/297966085 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b0d932c80 con 0x557b0d57e390 2024-04-25T11:35:43.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.944+0000 7f608db62640 1 --2- 172.21.15.90:0/297966085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d57e390 0x557b0d92e790 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f60800094e0 tx=0x7f6080005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a408ebe3fb4c915b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:43.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.944+0000 7f6070ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:35:43.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.944+0000 7f6070ff9640 1 -- 172.21.15.90:0/297966085 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60800307c0 con 0x557b0d57e390 2024-04-25T11:35:43.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.944+0000 7f6070ff9640 1 -- 172.21.15.90:0/297966085 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f6080030960 con 0x557b0d57e390 2024-04-25T11:35:43.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.944+0000 7f6096476740 1 -- 172.21.15.90:0/297966085 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60780a69c0 con 0x557b0d57e390 2024-04-25T11:35:43.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.944+0000 7f6070ff9640 1 -- 172.21.15.90:0/297966085 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f608005a640 con 0x557b0d57e390 2024-04-25T11:35:43.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.944+0000 7f6096476740 10 client.15690 fetch_fsmap learned FSMap version 26 2024-04-25T11:35:43.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.944+0000 7f6096476740 1 -- 172.21.15.90:0/297966085 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557b0d9a4b20 con 0x557b0d57e390 2024-04-25T11:35:43.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.944+0000 7f6070ff9640 1 -- 172.21.15.90:0/297966085 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60780a69c0 con 0x557b0d57e390 2024-04-25T11:35:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.944+0000 7f6070ff9640 1 -- 172.21.15.90:0/297966085 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f608002f780 con 0x557b0d57e390 2024-04-25T11:35:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.944+0000 7f6096476740 10 client.15690 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:35:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.944+0000 7f6096476740 10 client.15690 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:35:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.944+0000 7f6096476740 20 client.15690 populate_metadata read hostname 'smithi090' 2024-04-25T11:35:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.944+0000 7f6096476740 1 --2- 172.21.15.90:0/297966085 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x557b0d5123a0 0x557b0d9a72b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.944+0000 7f6096476740 4 client.15690 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:35:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.944+0000 7f6096476740 1 -- 172.21.15.90:0/297966085 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557b0d88c800 con 0x557b0d5123a0 2024-04-25T11:35:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.945+0000 7f608d361640 1 --2- 172.21.15.90:0/297966085 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x557b0d5123a0 0x557b0d9a72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.945+0000 7f608d361640 1 --2- 172.21.15.90:0/297966085 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x557b0d5123a0 0x557b0d9a72b0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b707caa764f713da server_cookie=ab50a47fc97f8845 in_seq=0 out_seq=0 2024-04-25T11:35:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.945+0000 7f6070ff9640 10 client.15690 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:35:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.945+0000 7f6070ff9640 1 -- 172.21.15.90:0/297966085 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557b0d88c800 con 0x557b0d5123a0 2024-04-25T11:35:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.945+0000 7f6070ff9640 10 client.15690 handle_command_reply: tid=0 2024-04-25T11:35:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.945+0000 7f6096476740 1 client.15690 shutdown 2024-04-25T11:35:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.945+0000 7f6096476740 1 -- 172.21.15.90:0/297966085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d57e390 msgr2=0x557b0d92e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.945+0000 7f6096476740 1 --2- 172.21.15.90:0/297966085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d57e390 0x557b0d92e790 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f60800094e0 tx=0x7f6080005b50 comp rx=0 tx=0).stop 2024-04-25T11:35:43.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.946+0000 7f6096476740 1 -- 172.21.15.90:0/297966085 shutdown_connections 2024-04-25T11:35:43.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.946+0000 7f6096476740 1 --2- 172.21.15.90:0/297966085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b0d57e390 0x557b0d92e790 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.946+0000 7f6096476740 1 --2- 172.21.15.90:0/297966085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b0d55d6b0 0x557b0d92e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.946+0000 7f6096476740 1 --2- 172.21.15.90:0/297966085 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x557b0d5123a0 0x557b0d9a72b0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.946+0000 7f6096476740 1 --2- 172.21.15.90:0/297966085 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557b0d5554d0 0x557b0d92dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.946+0000 7f6070ff9640 0 client.15690 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:35:43.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.946+0000 7f6096476740 1 -- 172.21.15.90:0/297966085 >> 172.21.15.90:0/297966085 conn(0x557b0d5724e0 msgr2=0x557b0d928970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:43.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.946+0000 7f6096476740 1 -- 172.21.15.90:0/297966085 shutdown_connections 2024-04-25T11:35:43.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.946+0000 7f6096476740 1 -- 172.21.15.90:0/297966085 wait complete. 2024-04-25T11:35:43.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.946+0000 7f6096476740 20 client.15690 trim_cache size 0 max 16384 2024-04-25T11:35:43.941 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:35:43.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.946+0000 7f608f5c5640 1 -- 172.21.15.90:0/60106656 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f60740afcf0 msgr2=0x7f60740b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.946+0000 7f608f5c5640 1 --2- 172.21.15.90:0/60106656 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f60740afcf0 0x7f60740b2200 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6080005cc0 tx=0x7f60800095c0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.946+0000 7f608f5c5640 1 -- 172.21.15.90:0/60106656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f608810b650 msgr2=0x7f608819ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.946+0000 7f608f5c5640 1 --2- 172.21.15.90:0/60106656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f608810b650 0x7f608819ac20 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f607800d950 tx=0x7f607800de10 comp rx=0 tx=0).stop 2024-04-25T11:35:43.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.946+0000 7f608f5c5640 1 -- 172.21.15.90:0/60106656 shutdown_connections 2024-04-25T11:35:43.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.946+0000 7f608f5c5640 1 --2- 172.21.15.90:0/60106656 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f60740afcf0 0x7f60740b2200 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.946+0000 7f608f5c5640 1 --2- 172.21.15.90:0/60106656 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f608810e040 0x7f608819eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.946+0000 7f608f5c5640 1 --2- 172.21.15.90:0/60106656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f608810b650 0x7f608819ac20 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.946+0000 7f608f5c5640 1 --2- 172.21.15.90:0/60106656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6088104bf0 0x7f608819a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.946+0000 7f608f5c5640 1 -- 172.21.15.90:0/60106656 >> 172.21.15.90:0/60106656 conn(0x7f6088078100 msgr2=0x7f6088101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:43.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.946+0000 7f608f5c5640 1 -- 172.21.15.90:0/60106656 shutdown_connections 2024-04-25T11:35:43.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:43.946+0000 7f608f5c5640 1 -- 172.21.15.90:0/60106656 wait complete. 2024-04-25T11:35:43.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:43 smithi090 ceph-mon[29575]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T11:35:43.945 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:43 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1440177886' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:35:43.946 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T11:35:44.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:43 smithi138 ceph-mon[32328]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T11:35:44.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:43 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1440177886' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:35:44.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.067+0000 7f454f453640 1 -- 172.21.15.90:0/1977079471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4548102440 msgr2=0x7f4548102850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.067+0000 7f454f453640 1 --2- 172.21.15.90:0/1977079471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4548102440 0x7f4548102850 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f4538009900 tx=0x7f453802a410 comp rx=0 tx=0).stop 2024-04-25T11:35:44.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.067+0000 7f454f453640 1 -- 172.21.15.90:0/1977079471 shutdown_connections 2024-04-25T11:35:44.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.067+0000 7f454f453640 1 --2- 172.21.15.90:0/1977079471 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4548107440 0x7f454810ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.067+0000 7f454f453640 1 --2- 172.21.15.90:0/1977079471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4548106980 0x7f4548106db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.067+0000 7f454f453640 1 --2- 172.21.15.90:0/1977079471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4548102440 0x7f4548102850 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.067+0000 7f454f453640 1 -- 172.21.15.90:0/1977079471 >> 172.21.15.90:0/1977079471 conn(0x7f4548077880 msgr2=0x7f4548077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:44.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.067+0000 7f454f453640 1 -- 172.21.15.90:0/1977079471 shutdown_connections 2024-04-25T11:35:44.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.067+0000 7f454f453640 1 -- 172.21.15.90:0/1977079471 wait complete. 2024-04-25T11:35:44.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.068+0000 7f454f453640 1 Processor -- start 2024-04-25T11:35:44.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.068+0000 7f454f453640 1 -- start start 2024-04-25T11:35:44.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.068+0000 7f454f453640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4548102440 0x7f454819b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.068+0000 7f454f453640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4548106980 0x7f454819baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.068+0000 7f454f453640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4548107440 0x7f454819c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.068+0000 7f454f453640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4548112bd0 con 0x7f4548107440 2024-04-25T11:35:44.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.068+0000 7f454f453640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4548112a50 con 0x7f4548106980 2024-04-25T11:35:44.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.068+0000 7f454f453640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4548112d50 con 0x7f4548102440 2024-04-25T11:35:44.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.068+0000 7f454d9f0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4548107440 0x7f454819c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.068+0000 7f454d9f0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4548107440 0x7f454819c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43506/0 (socket says 172.21.15.90:43506) 2024-04-25T11:35:44.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.068+0000 7f454d9f0640 1 -- 172.21.15.90:0/812784576 learned_addr learned my addr 172.21.15.90:0/812784576 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:44.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.068+0000 7f454d1ef640 1 --2- 172.21.15.90:0/812784576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4548102440 0x7f454819b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.068+0000 7f454c9ee640 1 --2- 172.21.15.90:0/812784576 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4548106980 0x7f454819baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.068+0000 7f454d9f0640 1 -- 172.21.15.90:0/812784576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4548102440 msgr2=0x7f454819b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.068+0000 7f454d9f0640 1 --2- 172.21.15.90:0/812784576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4548102440 0x7f454819b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.068+0000 7f454d9f0640 1 -- 172.21.15.90:0/812784576 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4548106980 msgr2=0x7f454819baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.068+0000 7f454d9f0640 1 --2- 172.21.15.90:0/812784576 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4548106980 0x7f454819baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.068+0000 7f454d9f0640 1 -- 172.21.15.90:0/812784576 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45481a68f0 con 0x7f4548107440 2024-04-25T11:35:44.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.068+0000 7f454d1ef640 1 --2- 172.21.15.90:0/812784576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4548102440 0x7f454819b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:35:44.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.068+0000 7f454c9ee640 1 --2- 172.21.15.90:0/812784576 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4548106980 0x7f454819baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:35:44.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.069+0000 7f454d9f0640 1 --2- 172.21.15.90:0/812784576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4548107440 0x7f454819c050 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f454000d750 tx=0x7f454000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:44.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.069+0000 7f453e7fc640 1 -- 172.21.15.90:0/812784576 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4540017070 con 0x7f4548107440 2024-04-25T11:35:44.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.069+0000 7f453e7fc640 1 -- 172.21.15.90:0/812784576 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45400054d0 con 0x7f4548107440 2024-04-25T11:35:44.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.069+0000 7f454f453640 1 -- 172.21.15.90:0/812784576 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45481a6be0 con 0x7f4548107440 2024-04-25T11:35:44.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.069+0000 7f453e7fc640 1 -- 172.21.15.90:0/812784576 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4540004c60 con 0x7f4548107440 2024-04-25T11:35:44.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.069+0000 7f454f453640 1 -- 172.21.15.90:0/812784576 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45481a70e0 con 0x7f4548107440 2024-04-25T11:35:44.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.072+0000 7f45563e0740 1 Processor -- start 2024-04-25T11:35:44.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.072+0000 7f453e7fc640 1 -- 172.21.15.90:0/812784576 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 23) ==== 147696+0+0 (secure 0 0 0) 0x7f4540009bd0 con 0x7f4548107440 2024-04-25T11:35:44.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.072+0000 7f45563e0740 1 -- start start 2024-04-25T11:35:44.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.072+0000 7f45563e0740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562bbc50b390 0x562bbc8ac810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.072+0000 7f45563e0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc4e24d0 0x562bbc8acd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.072+0000 7f45563e0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bbc4ea6b0 0x562bbc8b96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.072+0000 7f45563e0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562bbc811d80 con 0x562bbc4e24d0 2024-04-25T11:35:44.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.072+0000 7f45563e0740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x562bbc7cba20 con 0x562bbc50b390 2024-04-25T11:35:44.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.072+0000 7f45563e0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562bbc80dd40 con 0x562bbc4ea6b0 2024-04-25T11:35:44.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.072+0000 7f454d1ef640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc4e24d0 0x562bbc8acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.072+0000 7f454d1ef640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc4e24d0 0x562bbc8acd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43520/0 (socket says 172.21.15.90:43520) 2024-04-25T11:35:44.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.072+0000 7f454d1ef640 1 -- 172.21.15.90:0/845357967 learned_addr learned my addr 172.21.15.90:0/845357967 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:44.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.072+0000 7f453e7fc640 1 --2- 172.21.15.90:0/812784576 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f45340afcf0 0x7f45340b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.072+0000 7f454c9ee640 1 --2- 172.21.15.90:0/845357967 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562bbc50b390 0x562bbc8ac810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.072+0000 7f454c9ee640 1 --2- 172.21.15.90:0/845357967 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bbc4ea6b0 0x562bbc8b96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.072+0000 7f454d9f0640 1 --2- 172.21.15.90:0/812784576 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f45340afcf0 0x7f45340b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.073+0000 7f454d1ef640 1 -- 172.21.15.90:0/845357967 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bbc4ea6b0 msgr2=0x562bbc8b96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:44.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.073+0000 7f454d1ef640 1 --2- 172.21.15.90:0/845357967 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bbc4ea6b0 0x562bbc8b96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.073+0000 7f454d1ef640 1 -- 172.21.15.90:0/845357967 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562bbc50b390 msgr2=0x562bbc8ac810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:44.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.073+0000 7f454d1ef640 1 --2- 172.21.15.90:0/845357967 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562bbc50b390 0x562bbc8ac810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.073+0000 7f454d1ef640 1 -- 172.21.15.90:0/845357967 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bbc7d6c30 con 0x562bbc4e24d0 2024-04-25T11:35:44.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.073+0000 7f454c9ee640 1 --2- 172.21.15.90:0/845357967 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bbc4ea6b0 0x562bbc8b96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:35:44.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.073+0000 7f453e7fc640 1 -- 172.21.15.90:0/812784576 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f45400dec00 con 0x7f4548107440 2024-04-25T11:35:44.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.073+0000 7f454d1ef640 1 --2- 172.21.15.90:0/845357967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc4e24d0 0x562bbc8acd50 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f453802a930 tx=0x7f4538032c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ac9205409afa797 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:44.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.073+0000 7f454d9f0640 1 --2- 172.21.15.90:0/812784576 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f45340afcf0 0x7f45340b2200 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f45400dc2f0 tx=0x7f4540072330 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:44.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.073+0000 7f45337fe640 1 -- 172.21.15.90:0/845357967 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f453803f070 con 0x562bbc4e24d0 2024-04-25T11:35:44.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.073+0000 7f45337fe640 1 -- 172.21.15.90:0/845357967 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45380316a0 con 0x562bbc4e24d0 2024-04-25T11:35:44.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.073+0000 7f45337fe640 1 -- 172.21.15.90:0/845357967 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4538035070 con 0x562bbc4e24d0 2024-04-25T11:35:44.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.074+0000 7f45563e0740 1 -- 172.21.15.90:0/845357967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc4e24d0 msgr2=0x562bbc8acd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.074+0000 7f45563e0740 1 --2- 172.21.15.90:0/845357967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc4e24d0 0x562bbc8acd50 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f453802a930 tx=0x7f4538032c30 comp rx=0 tx=0).stop 2024-04-25T11:35:44.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.074+0000 7f45563e0740 1 -- 172.21.15.90:0/845357967 shutdown_connections 2024-04-25T11:35:44.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.074+0000 7f45563e0740 1 --2- 172.21.15.90:0/845357967 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bbc4ea6b0 0x562bbc8b96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.074+0000 7f45563e0740 1 --2- 172.21.15.90:0/845357967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc4e24d0 0x562bbc8acd50 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.074+0000 7f45563e0740 1 --2- 172.21.15.90:0/845357967 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562bbc50b390 0x562bbc8ac810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.074+0000 7f45563e0740 1 -- 172.21.15.90:0/845357967 >> 172.21.15.90:0/845357967 conn(0x562bbc4ff4e0 msgr2=0x562bbc8a8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:44.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.074+0000 7f45563e0740 1 -- 172.21.15.90:0/845357967 shutdown_connections 2024-04-25T11:35:44.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.074+0000 7f45563e0740 1 -- 172.21.15.90:0/845357967 wait complete. 2024-04-25T11:35:44.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.074+0000 7f45563e0740 1 -- start start 2024-04-25T11:35:44.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.074+0000 7f45563e0740 1 Processor -- start 2024-04-25T11:35:44.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.075+0000 7f45563e0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc4e24d0 0x562bbc8b7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.075+0000 7f45563e0740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562bbc4ea6b0 0x562bbc8b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.075+0000 7f45563e0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bbc50b390 0x562bbc8c31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.075+0000 7f45563e0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562bbc8b9d30 con 0x562bbc4e24d0 2024-04-25T11:35:44.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.075+0000 7f45563e0740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x562bbc798dc0 con 0x562bbc4ea6b0 2024-04-25T11:35:44.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.075+0000 7f45563e0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562bbc8b9eb0 con 0x562bbc50b390 2024-04-25T11:35:44.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.075+0000 7f45563e0740 10 client.0 _ll_register_callbacks cb 0x562bbc7b4600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:35:44.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.075+0000 7f454c9ee640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc4e24d0 0x562bbc8b7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.075+0000 7f454c9ee640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc4e24d0 0x562bbc8b7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43530/0 (socket says 172.21.15.90:43530) 2024-04-25T11:35:44.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.075+0000 7f454c9ee640 1 -- 172.21.15.90:0/1628559911 learned_addr learned my addr 172.21.15.90:0/1628559911 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:44.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.075+0000 7f454c9ee640 1 -- 172.21.15.90:0/1628559911 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bbc50b390 msgr2=0x562bbc8c31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:44.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.075+0000 7f454c9ee640 1 --2- 172.21.15.90:0/1628559911 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bbc50b390 0x562bbc8c31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.075+0000 7f454c9ee640 1 -- 172.21.15.90:0/1628559911 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562bbc4ea6b0 msgr2=0x562bbc8b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:44.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.075+0000 7f454c9ee640 1 --2- 172.21.15.90:0/1628559911 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562bbc4ea6b0 0x562bbc8b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.075+0000 7f454c9ee640 1 -- 172.21.15.90:0/1628559911 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bbc5233f0 con 0x562bbc4e24d0 2024-04-25T11:35:44.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.075+0000 7f454c9ee640 1 --2- 172.21.15.90:0/1628559911 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc4e24d0 0x562bbc8b7280 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f45480754b0 tx=0x7f4544012da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4479a9e40d59a06 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:44.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.075+0000 7f4530ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:35:44.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.075+0000 7f4530ff9640 1 -- 172.21.15.90:0/1628559911 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4544011de0 con 0x562bbc4e24d0 2024-04-25T11:35:44.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.076+0000 7f4530ff9640 1 -- 172.21.15.90:0/1628559911 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f454400f420 con 0x562bbc4e24d0 2024-04-25T11:35:44.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.076+0000 7f45563e0740 1 -- 172.21.15.90:0/1628559911 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45380316a0 con 0x562bbc4e24d0 2024-04-25T11:35:44.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.076+0000 7f4530ff9640 1 -- 172.21.15.90:0/1628559911 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f454400f620 con 0x562bbc4e24d0 2024-04-25T11:35:44.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.076+0000 7f45563e0740 10 client.15702 fetch_fsmap learned FSMap version 26 2024-04-25T11:35:44.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.076+0000 7f45563e0740 1 -- 172.21.15.90:0/1628559911 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562bbc5ed7d0 con 0x562bbc4e24d0 2024-04-25T11:35:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.076+0000 7f4530ff9640 1 -- 172.21.15.90:0/1628559911 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45380316a0 con 0x562bbc4e24d0 2024-04-25T11:35:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.076+0000 7f4530ff9640 1 -- 172.21.15.90:0/1628559911 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f454401c070 con 0x562bbc4e24d0 2024-04-25T11:35:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.076+0000 7f45563e0740 10 client.15702 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:35:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.076+0000 7f45563e0740 10 client.15702 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:35:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.076+0000 7f45563e0740 20 client.15702 populate_metadata read hostname 'smithi090' 2024-04-25T11:35:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.076+0000 7f45563e0740 1 --2- 172.21.15.90:0/1628559911 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x562bbc4c9f60 0x562bbc8c9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.076+0000 7f45563e0740 4 client.15702 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:35:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.076+0000 7f45563e0740 1 -- 172.21.15.90:0/1628559911 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562bbc81f990 con 0x562bbc4c9f60 2024-04-25T11:35:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.076+0000 7f454d1ef640 1 --2- 172.21.15.90:0/1628559911 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x562bbc4c9f60 0x562bbc8c9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.077+0000 7f454d1ef640 1 --2- 172.21.15.90:0/1628559911 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x562bbc4c9f60 0x562bbc8c9860 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dc3259a96a7319da server_cookie=a48fe39eb3149d4e in_seq=0 out_seq=0 2024-04-25T11:35:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.077+0000 7f4530ff9640 10 client.15702 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:35:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.078+0000 7f4530ff9640 1 -- 172.21.15.90:0/1628559911 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562bbc81f990 con 0x562bbc4c9f60 2024-04-25T11:35:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.078+0000 7f4530ff9640 10 client.15702 handle_command_reply: tid=0 2024-04-25T11:35:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.078+0000 7f45563e0740 1 client.15702 shutdown 2024-04-25T11:35:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.078+0000 7f45563e0740 1 -- 172.21.15.90:0/1628559911 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc4e24d0 msgr2=0x562bbc8b7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.078+0000 7f45563e0740 1 --2- 172.21.15.90:0/1628559911 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc4e24d0 0x562bbc8b7280 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f45480754b0 tx=0x7f4544012da0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.078+0000 7f45563e0740 1 -- 172.21.15.90:0/1628559911 shutdown_connections 2024-04-25T11:35:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.078+0000 7f45563e0740 1 --2- 172.21.15.90:0/1628559911 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x562bbc4c9f60 0x562bbc8c9860 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.078+0000 7f45563e0740 1 --2- 172.21.15.90:0/1628559911 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bbc50b390 0x562bbc8c31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.078+0000 7f45563e0740 1 --2- 172.21.15.90:0/1628559911 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562bbc4ea6b0 0x562bbc8b77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.078+0000 7f45563e0740 1 --2- 172.21.15.90:0/1628559911 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc4e24d0 0x562bbc8b7280 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.078+0000 7f4530ff9640 0 client.15702 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:35:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.078+0000 7f45563e0740 1 -- 172.21.15.90:0/1628559911 >> 172.21.15.90:0/1628559911 conn(0x562bbc4ff4e0 msgr2=0x562bbc8ae530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.078+0000 7f45563e0740 1 -- 172.21.15.90:0/1628559911 shutdown_connections 2024-04-25T11:35:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.078+0000 7f45563e0740 1 -- 172.21.15.90:0/1628559911 wait complete. 2024-04-25T11:35:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.078+0000 7f45563e0740 20 client.15702 trim_cache size 0 max 16384 2024-04-25T11:35:44.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.091+0000 7f45563e0740 1 Processor -- start 2024-04-25T11:35:44.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.091+0000 7f45563e0740 1 -- start start 2024-04-25T11:35:44.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.092+0000 7f45563e0740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562bbc4e24d0 0x562bbc8bf570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.092+0000 7f45563e0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc4ea6b0 0x562bbc8c2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.092+0000 7f45563e0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bbc50b390 0x562bbc8b1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.092+0000 7f45563e0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f45481af9a0 con 0x562bbc4ea6b0 2024-04-25T11:35:44.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.092+0000 7f45563e0740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f45481b14e0 con 0x562bbc4e24d0 2024-04-25T11:35:44.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.092+0000 7f45563e0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f45481b0070 con 0x562bbc50b390 2024-04-25T11:35:44.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.092+0000 7f454d1ef640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc4ea6b0 0x562bbc8c2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.092+0000 7f454d1ef640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc4ea6b0 0x562bbc8c2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43534/0 (socket says 172.21.15.90:43534) 2024-04-25T11:35:44.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.092+0000 7f454d1ef640 1 -- 172.21.15.90:0/4194918388 learned_addr learned my addr 172.21.15.90:0/4194918388 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:44.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.092+0000 7f454c9ee640 1 --2- 172.21.15.90:0/4194918388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bbc50b390 0x562bbc8b1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.092+0000 7f454c9ee640 1 --2- 172.21.15.90:0/4194918388 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562bbc4e24d0 0x562bbc8bf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.092+0000 7f454d1ef640 1 -- 172.21.15.90:0/4194918388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bbc50b390 msgr2=0x562bbc8b1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:44.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.092+0000 7f454d1ef640 1 --2- 172.21.15.90:0/4194918388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bbc50b390 0x562bbc8b1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.092+0000 7f454d1ef640 1 -- 172.21.15.90:0/4194918388 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562bbc4e24d0 msgr2=0x562bbc8bf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:44.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.092+0000 7f454d1ef640 1 --2- 172.21.15.90:0/4194918388 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562bbc4e24d0 0x562bbc8bf570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.092+0000 7f454d1ef640 1 -- 172.21.15.90:0/4194918388 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bbc8ac5a0 con 0x562bbc4ea6b0 2024-04-25T11:35:44.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.092+0000 7f454c9ee640 1 --2- 172.21.15.90:0/4194918388 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562bbc4e24d0 0x562bbc8bf570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:35:44.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.092+0000 7f454d1ef640 1 --2- 172.21.15.90:0/4194918388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc4ea6b0 0x562bbc8c2e10 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f45380052a0 tx=0x7f4538005230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80277ac13f3e596 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:44.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.092+0000 7f4532ffd640 1 -- 172.21.15.90:0/4194918388 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f453803f030 con 0x562bbc4ea6b0 2024-04-25T11:35:44.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.092+0000 7f4532ffd640 1 -- 172.21.15.90:0/4194918388 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4538050070 con 0x562bbc4ea6b0 2024-04-25T11:35:44.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.093+0000 7f4532ffd640 1 -- 172.21.15.90:0/4194918388 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4538035040 con 0x562bbc4ea6b0 2024-04-25T11:35:44.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.093+0000 7f45563e0740 1 -- 172.21.15.90:0/4194918388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc4ea6b0 msgr2=0x562bbc8c2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.093+0000 7f45563e0740 1 --2- 172.21.15.90:0/4194918388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc4ea6b0 0x562bbc8c2e10 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f45380052a0 tx=0x7f4538005230 comp rx=0 tx=0).stop 2024-04-25T11:35:44.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.093+0000 7f45563e0740 1 -- 172.21.15.90:0/4194918388 shutdown_connections 2024-04-25T11:35:44.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.093+0000 7f45563e0740 1 --2- 172.21.15.90:0/4194918388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bbc50b390 0x562bbc8b1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.093+0000 7f45563e0740 1 --2- 172.21.15.90:0/4194918388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc4ea6b0 0x562bbc8c2e10 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.093+0000 7f45563e0740 1 --2- 172.21.15.90:0/4194918388 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562bbc4e24d0 0x562bbc8bf570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.093+0000 7f45563e0740 1 -- 172.21.15.90:0/4194918388 >> 172.21.15.90:0/4194918388 conn(0x562bbc4ff4e0 msgr2=0x562bbc8a7610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:44.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.093+0000 7f45563e0740 1 -- 172.21.15.90:0/4194918388 shutdown_connections 2024-04-25T11:35:44.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.093+0000 7f45563e0740 1 -- 172.21.15.90:0/4194918388 wait complete. 2024-04-25T11:35:44.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.094+0000 7f45563e0740 1 -- start start 2024-04-25T11:35:44.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.094+0000 7f45563e0740 1 Processor -- start 2024-04-25T11:35:44.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.094+0000 7f45563e0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bbc4e24d0 0x562bbc8a7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.094+0000 7f45563e0740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562bbc4ea6b0 0x562bbc8a8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.094+0000 7f45563e0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc50b390 0x562bbc92dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.094+0000 7f45563e0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f45481b0e10 con 0x562bbc50b390 2024-04-25T11:35:44.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.094+0000 7f45563e0740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f45481b0740 con 0x562bbc4ea6b0 2024-04-25T11:35:44.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.094+0000 7f45563e0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f45481ad980 con 0x562bbc4e24d0 2024-04-25T11:35:44.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.094+0000 7f45563e0740 10 client.0 _ll_register_callbacks cb 0x7f4544012550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:35:44.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.094+0000 7f454c9ee640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc50b390 0x562bbc92dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.094+0000 7f454c9ee640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc50b390 0x562bbc92dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43538/0 (socket says 172.21.15.90:43538) 2024-04-25T11:35:44.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.094+0000 7f454c9ee640 1 -- 172.21.15.90:0/309462362 learned_addr learned my addr 172.21.15.90:0/309462362 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:44.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.094+0000 7f454d1ef640 1 --2- 172.21.15.90:0/309462362 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562bbc4ea6b0 0x562bbc8a8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.094+0000 7f454c9ee640 1 --2- 172.21.15.90:0/309462362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bbc4e24d0 0x562bbc8a7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.094+0000 7f454c9ee640 1 -- 172.21.15.90:0/309462362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bbc4e24d0 msgr2=0x562bbc8a7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:44.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.094+0000 7f454c9ee640 1 --2- 172.21.15.90:0/309462362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bbc4e24d0 0x562bbc8a7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.094+0000 7f454c9ee640 1 -- 172.21.15.90:0/309462362 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562bbc4ea6b0 msgr2=0x562bbc8a8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:44.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.094+0000 7f454c9ee640 1 --2- 172.21.15.90:0/309462362 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562bbc4ea6b0 0x562bbc8a8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.094+0000 7f454c9ee640 1 -- 172.21.15.90:0/309462362 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bbc92e1c0 con 0x562bbc50b390 2024-04-25T11:35:44.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.095+0000 7f454c9ee640 1 --2- 172.21.15.90:0/309462362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc50b390 0x562bbc92dc80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4544005ee0 tx=0x7f4544004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6f536d09df1f6c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:44.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.095+0000 7f4530ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:35:44.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.095+0000 7f4530ff9640 1 -- 172.21.15.90:0/309462362 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4544004d20 con 0x562bbc50b390 2024-04-25T11:35:44.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.095+0000 7f4530ff9640 1 -- 172.21.15.90:0/309462362 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f454400f7c0 con 0x562bbc50b390 2024-04-25T11:35:44.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.095+0000 7f45563e0740 1 -- 172.21.15.90:0/309462362 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4538050070 con 0x562bbc50b390 2024-04-25T11:35:44.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.095+0000 7f4530ff9640 1 -- 172.21.15.90:0/309462362 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f454400f9c0 con 0x562bbc50b390 2024-04-25T11:35:44.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.095+0000 7f45563e0740 10 client.15708 fetch_fsmap learned FSMap version 26 2024-04-25T11:35:44.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.095+0000 7f45563e0740 1 -- 172.21.15.90:0/309462362 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562bbc935400 con 0x562bbc50b390 2024-04-25T11:35:44.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.095+0000 7f4530ff9640 1 -- 172.21.15.90:0/309462362 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4538050070 con 0x562bbc50b390 2024-04-25T11:35:44.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.095+0000 7f4530ff9640 1 -- 172.21.15.90:0/309462362 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f454401c040 con 0x562bbc50b390 2024-04-25T11:35:44.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.095+0000 7f45563e0740 10 client.15708 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:35:44.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.095+0000 7f45563e0740 10 client.15708 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:35:44.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.095+0000 7f45563e0740 20 client.15708 populate_metadata read hostname 'smithi090' 2024-04-25T11:35:44.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.095+0000 7f45563e0740 1 --2- 172.21.15.90:0/309462362 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x562bbc49f3a0 0x562bbc937b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.095+0000 7f45563e0740 4 client.15708 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:35:44.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.095+0000 7f45563e0740 1 -- 172.21.15.90:0/309462362 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562bbc819800 con 0x562bbc49f3a0 2024-04-25T11:35:44.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.096+0000 7f454d1ef640 1 --2- 172.21.15.90:0/309462362 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x562bbc49f3a0 0x562bbc937b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.096+0000 7f4530ff9640 10 client.15708 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:35:44.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.096+0000 7f454d1ef640 1 --2- 172.21.15.90:0/309462362 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x562bbc49f3a0 0x562bbc937b70 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f4dc2f4a73dc9856 server_cookie=106ab16857a01433 in_seq=0 out_seq=0 2024-04-25T11:35:44.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.096+0000 7f4530ff9640 1 -- 172.21.15.90:0/309462362 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562bbc819800 con 0x562bbc49f3a0 2024-04-25T11:35:44.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.096+0000 7f4530ff9640 10 client.15708 handle_command_reply: tid=0 2024-04-25T11:35:44.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.096+0000 7f45563e0740 1 client.15708 shutdown 2024-04-25T11:35:44.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.097+0000 7f45563e0740 1 -- 172.21.15.90:0/309462362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc50b390 msgr2=0x562bbc92dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.097+0000 7f45563e0740 1 --2- 172.21.15.90:0/309462362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc50b390 0x562bbc92dc80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4544005ee0 tx=0x7f4544004ab0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.097+0000 7f45563e0740 1 -- 172.21.15.90:0/309462362 shutdown_connections 2024-04-25T11:35:44.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.097+0000 7f45563e0740 1 --2- 172.21.15.90:0/309462362 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x562bbc49f3a0 0x562bbc937b70 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.097+0000 7f45563e0740 1 --2- 172.21.15.90:0/309462362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562bbc50b390 0x562bbc92dc80 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.097+0000 7f4530ff9640 0 client.15708 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:35:44.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.097+0000 7f45563e0740 1 --2- 172.21.15.90:0/309462362 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562bbc4ea6b0 0x562bbc8a8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.097+0000 7f45563e0740 1 --2- 172.21.15.90:0/309462362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bbc4e24d0 0x562bbc8a7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.097+0000 7f45563e0740 1 -- 172.21.15.90:0/309462362 >> 172.21.15.90:0/309462362 conn(0x562bbc4ff4e0 msgr2=0x562bbc8b8670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:44.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.097+0000 7f45563e0740 1 -- 172.21.15.90:0/309462362 shutdown_connections 2024-04-25T11:35:44.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.097+0000 7f45563e0740 1 -- 172.21.15.90:0/309462362 wait complete. 2024-04-25T11:35:44.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.097+0000 7f45563e0740 20 client.15708 trim_cache size 0 max 16384 2024-04-25T11:35:44.093 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:35:44.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.097+0000 7f454f453640 1 -- 172.21.15.90:0/812784576 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f45340afcf0 msgr2=0x7f45340b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.097+0000 7f454f453640 1 --2- 172.21.15.90:0/812784576 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f45340afcf0 0x7f45340b2200 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f45400dc2f0 tx=0x7f4540072330 comp rx=0 tx=0).stop 2024-04-25T11:35:44.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.097+0000 7f454f453640 1 -- 172.21.15.90:0/812784576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4548107440 msgr2=0x7f454819c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.097+0000 7f454f453640 1 --2- 172.21.15.90:0/812784576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4548107440 0x7f454819c050 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f454000d750 tx=0x7f454000dc10 comp rx=0 tx=0).stop 2024-04-25T11:35:44.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.097+0000 7f454f453640 1 -- 172.21.15.90:0/812784576 shutdown_connections 2024-04-25T11:35:44.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.097+0000 7f454f453640 1 --2- 172.21.15.90:0/812784576 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f45340afcf0 0x7f45340b2200 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.097+0000 7f454f453640 1 --2- 172.21.15.90:0/812784576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4548107440 0x7f454819c050 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.097+0000 7f454f453640 1 --2- 172.21.15.90:0/812784576 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4548106980 0x7f454819baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.097+0000 7f454f453640 1 --2- 172.21.15.90:0/812784576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4548102440 0x7f454819b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.097+0000 7f454f453640 1 -- 172.21.15.90:0/812784576 >> 172.21.15.90:0/812784576 conn(0x7f4548077880 msgr2=0x7f4548100980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:44.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.097+0000 7f454f453640 1 -- 172.21.15.90:0/812784576 shutdown_connections 2024-04-25T11:35:44.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.097+0000 7f454f453640 1 -- 172.21.15.90:0/812784576 wait complete. 2024-04-25T11:35:44.095 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:43 smithi156 ceph-mon[33045]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T11:35:44.095 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:43 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1440177886' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:35:44.097 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T11:35:44.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.218+0000 7f5bb3aee640 1 -- 172.21.15.90:0/2440919471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bac102720 msgr2=0x7f5bac107340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.218+0000 7f5bb3aee640 1 --2- 172.21.15.90:0/2440919471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bac102720 0x7f5bac107340 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4009900 tx=0x7f5ba402a480 comp rx=0 tx=0).stop 2024-04-25T11:35:44.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.218+0000 7f5bb3aee640 1 -- 172.21.15.90:0/2440919471 shutdown_connections 2024-04-25T11:35:44.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.218+0000 7f5bb3aee640 1 --2- 172.21.15.90:0/2440919471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bac102720 0x7f5bac107340 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.218+0000 7f5bb3aee640 1 --2- 172.21.15.90:0/2440919471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bac108de0 0x7f5bac1020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.218+0000 7f5bb3aee640 1 --2- 172.21.15.90:0/2440919471 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5bac1089d0 0x7f5bac101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.218+0000 7f5bb3aee640 1 -- 172.21.15.90:0/2440919471 >> 172.21.15.90:0/2440919471 conn(0x7f5bac077880 msgr2=0x7f5bac077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:44.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.218+0000 7f5bb3aee640 1 -- 172.21.15.90:0/2440919471 shutdown_connections 2024-04-25T11:35:44.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.218+0000 7f5bb3aee640 1 -- 172.21.15.90:0/2440919471 wait complete. 2024-04-25T11:35:44.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.219+0000 7f5bb3aee640 1 Processor -- start 2024-04-25T11:35:44.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.219+0000 7f5bb3aee640 1 -- start start 2024-04-25T11:35:44.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.219+0000 7f5bb3aee640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bac102720 0x7f5bac1a69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.219+0000 7f5bb3aee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bac1089d0 0x7f5bac1a6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.219+0000 7f5bb3aee640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5bac108de0 0x7f5bac1a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.219+0000 7f5bb3aee640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac11a2b0 con 0x7f5bac102720 2024-04-25T11:35:44.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.219+0000 7f5bb3aee640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac11a130 con 0x7f5bac108de0 2024-04-25T11:35:44.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.219+0000 7f5bb3aee640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac11a430 con 0x7f5bac1089d0 2024-04-25T11:35:44.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.219+0000 7f5bb188a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bac102720 0x7f5bac1a69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.219+0000 7f5bb188a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bac102720 0x7f5bac1a69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43558/0 (socket says 172.21.15.90:43558) 2024-04-25T11:35:44.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.219+0000 7f5bb188a640 1 -- 172.21.15.90:0/2823218972 learned_addr learned my addr 172.21.15.90:0/2823218972 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:44.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.219+0000 7f5bb208b640 1 --2- 172.21.15.90:0/2823218972 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5bac108de0 0x7f5bac1a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.219+0000 7f5bb1089640 1 --2- 172.21.15.90:0/2823218972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bac1089d0 0x7f5bac1a6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.220+0000 7f5bb188a640 1 -- 172.21.15.90:0/2823218972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bac1089d0 msgr2=0x7f5bac1a6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.220+0000 7f5bb188a640 1 --2- 172.21.15.90:0/2823218972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bac1089d0 0x7f5bac1a6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.220+0000 7f5bb188a640 1 -- 172.21.15.90:0/2823218972 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5bac108de0 msgr2=0x7f5bac1a0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.220+0000 7f5bb188a640 1 --2- 172.21.15.90:0/2823218972 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5bac108de0 0x7f5bac1a0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.220+0000 7f5bb188a640 1 -- 172.21.15.90:0/2823218972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bac1a1260 con 0x7f5bac102720 2024-04-25T11:35:44.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.220+0000 7f5bb208b640 1 --2- 172.21.15.90:0/2823218972 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5bac108de0 0x7f5bac1a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:35:44.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.220+0000 7f5bb188a640 1 --2- 172.21.15.90:0/2823218972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bac102720 0x7f5bac1a69b0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5ba800db80 tx=0x7f5ba800b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:44.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.220+0000 7f5ba2ffd640 1 -- 172.21.15.90:0/2823218972 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba8017070 con 0x7f5bac102720 2024-04-25T11:35:44.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.220+0000 7f5bb3aee640 1 -- 172.21.15.90:0/2823218972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bac1a14f0 con 0x7f5bac102720 2024-04-25T11:35:44.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.220+0000 7f5ba2ffd640 1 -- 172.21.15.90:0/2823218972 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ba8003b70 con 0x7f5bac102720 2024-04-25T11:35:44.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.220+0000 7f5ba2ffd640 1 -- 172.21.15.90:0/2823218972 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba80106c0 con 0x7f5bac102720 2024-04-25T11:35:44.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.220+0000 7f5bb3aee640 1 -- 172.21.15.90:0/2823218972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bac1ad750 con 0x7f5bac102720 2024-04-25T11:35:44.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.223+0000 7f5ba2ffd640 1 -- 172.21.15.90:0/2823218972 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 23) ==== 147696+0+0 (secure 0 0 0) 0x7f5ba8004020 con 0x7f5bac102720 2024-04-25T11:35:44.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.223+0000 7f5bba99f740 1 Processor -- start 2024-04-25T11:35:44.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.223+0000 7f5bba99f740 1 -- start start 2024-04-25T11:35:44.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.223+0000 7f5bba99f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f79278390 0x561f79619810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.223+0000 7f5bba99f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f7924f4d0 0x561f79619d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.223+0000 7f5bba99f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561f792576b0 0x561f796266a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.223+0000 7f5bba99f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561f7957ed80 con 0x561f7924f4d0 2024-04-25T11:35:44.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.223+0000 7f5bba99f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x561f79538a20 con 0x561f792576b0 2024-04-25T11:35:44.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.223+0000 7f5bba99f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561f7957ad40 con 0x561f79278390 2024-04-25T11:35:44.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.224+0000 7f5bb188a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f7924f4d0 0x561f79619d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.224+0000 7f5bb188a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f7924f4d0 0x561f79619d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43564/0 (socket says 172.21.15.90:43564) 2024-04-25T11:35:44.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.224+0000 7f5bb188a640 1 -- 172.21.15.90:0/51696330 learned_addr learned my addr 172.21.15.90:0/51696330 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:44.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.224+0000 7f5bb1089640 1 --2- 172.21.15.90:0/51696330 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561f792576b0 0x561f796266a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.224+0000 7f5bb208b640 1 --2- 172.21.15.90:0/51696330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f79278390 0x561f79619810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.224+0000 7f5ba2ffd640 1 --2- 172.21.15.90:0/2823218972 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5b980afcf0 0x7f5b980b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.224+0000 7f5bb208b640 1 --2- 172.21.15.90:0/2823218972 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5b980afcf0 0x7f5b980b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.224+0000 7f5bb188a640 1 -- 172.21.15.90:0/51696330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f79278390 msgr2=0x561f79619810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:44.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.224+0000 7f5bb188a640 1 --2- 172.21.15.90:0/51696330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f79278390 0x561f79619810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.224+0000 7f5bb188a640 1 -- 172.21.15.90:0/51696330 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561f792576b0 msgr2=0x561f796266a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:44.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.224+0000 7f5bb188a640 1 --2- 172.21.15.90:0/51696330 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561f792576b0 0x561f796266a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.224+0000 7f5bb188a640 1 -- 172.21.15.90:0/51696330 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f79543c30 con 0x561f7924f4d0 2024-04-25T11:35:44.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.224+0000 7f5ba2ffd640 1 -- 172.21.15.90:0/2823218972 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f5ba80df310 con 0x7f5bac102720 2024-04-25T11:35:44.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.224+0000 7f5bb188a640 1 --2- 172.21.15.90:0/51696330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f7924f4d0 0x561f79619d50 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f5ba80dc8b0 tx=0x7f5ba8021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e689b00418e2abc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:44.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.224+0000 7f5bb208b640 1 --2- 172.21.15.90:0/2823218972 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5b980afcf0 0x7f5b980b2200 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4005940 tx=0x7f5ba402a480 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:44.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.224+0000 7f5b97fff640 1 -- 172.21.15.90:0/51696330 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba80a70b0 con 0x561f7924f4d0 2024-04-25T11:35:44.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.225+0000 7f5b97fff640 1 -- 172.21.15.90:0/51696330 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ba80a6df0 con 0x561f7924f4d0 2024-04-25T11:35:44.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.225+0000 7f5b97fff640 1 -- 172.21.15.90:0/51696330 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba80a6a50 con 0x561f7924f4d0 2024-04-25T11:35:44.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.225+0000 7f5bba99f740 1 -- 172.21.15.90:0/51696330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f7924f4d0 msgr2=0x561f79619d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.225+0000 7f5bba99f740 1 --2- 172.21.15.90:0/51696330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f7924f4d0 0x561f79619d50 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f5ba80dc8b0 tx=0x7f5ba8021000 comp rx=0 tx=0).stop 2024-04-25T11:35:44.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.225+0000 7f5bba99f740 1 -- 172.21.15.90:0/51696330 shutdown_connections 2024-04-25T11:35:44.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.225+0000 7f5bba99f740 1 --2- 172.21.15.90:0/51696330 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561f792576b0 0x561f796266a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.225+0000 7f5bba99f740 1 --2- 172.21.15.90:0/51696330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f7924f4d0 0x561f79619d50 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.225+0000 7f5bba99f740 1 --2- 172.21.15.90:0/51696330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f79278390 0x561f79619810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.225+0000 7f5bba99f740 1 -- 172.21.15.90:0/51696330 >> 172.21.15.90:0/51696330 conn(0x561f7926c4e0 msgr2=0x561f79615fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:44.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.226+0000 7f5bba99f740 1 -- 172.21.15.90:0/51696330 shutdown_connections 2024-04-25T11:35:44.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.226+0000 7f5bba99f740 1 -- 172.21.15.90:0/51696330 wait complete. 2024-04-25T11:35:44.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.226+0000 7f5bba99f740 1 -- start start 2024-04-25T11:35:44.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.226+0000 7f5bba99f740 1 Processor -- start 2024-04-25T11:35:44.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.226+0000 7f5bba99f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f7924f4d0 0x561f79623f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.226+0000 7f5bba99f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561f792576b0 0x561f79624490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.226+0000 7f5bba99f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f79278390 0x561f796301b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.226+0000 7f5bba99f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561f79626d30 con 0x561f79278390 2024-04-25T11:35:44.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.226+0000 7f5bba99f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x561f79505dc0 con 0x561f792576b0 2024-04-25T11:35:44.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.226+0000 7f5bba99f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561f79626eb0 con 0x561f7924f4d0 2024-04-25T11:35:44.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.226+0000 7f5bba99f740 10 client.0 _ll_register_callbacks cb 0x561f79521600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:35:44.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.227+0000 7f5bb208b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f79278390 0x561f796301b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.227+0000 7f5bb1089640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561f792576b0 0x561f79624490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.227+0000 7f5bb208b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f79278390 0x561f796301b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43580/0 (socket says 172.21.15.90:43580) 2024-04-25T11:35:44.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.227+0000 7f5bb1089640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561f792576b0 0x561f79624490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:33086/0 (socket says 172.21.15.90:33086) 2024-04-25T11:35:44.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.227+0000 7f5bb208b640 1 -- 172.21.15.90:0/2808745736 learned_addr learned my addr 172.21.15.90:0/2808745736 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:44.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.227+0000 7f5bb188a640 1 --2- 172.21.15.90:0/2808745736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f7924f4d0 0x561f79623f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.227+0000 7f5bb208b640 1 -- 172.21.15.90:0/2808745736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f7924f4d0 msgr2=0x561f79623f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:44.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.227+0000 7f5bb208b640 1 --2- 172.21.15.90:0/2808745736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f7924f4d0 0x561f79623f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.227+0000 7f5bb208b640 1 -- 172.21.15.90:0/2808745736 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561f792576b0 msgr2=0x561f79624490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:44.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.227+0000 7f5bb208b640 1 --2- 172.21.15.90:0/2808745736 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561f792576b0 0x561f79624490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.227+0000 7f5bb208b640 1 -- 172.21.15.90:0/2808745736 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f792903f0 con 0x561f79278390 2024-04-25T11:35:44.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.227+0000 7f5bb208b640 1 --2- 172.21.15.90:0/2808745736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f79278390 0x561f796301b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4009900 tx=0x7f5ba4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa535b13e3dc3fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:44.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.227+0000 7f5b957fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:35:44.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.227+0000 7f5b957fa640 1 -- 172.21.15.90:0/2808745736 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba404e020 con 0x561f79278390 2024-04-25T11:35:44.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.227+0000 7f5b957fa640 1 -- 172.21.15.90:0/2808745736 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f5ba40046c0 con 0x561f79278390 2024-04-25T11:35:44.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.227+0000 7f5bba99f740 1 -- 172.21.15.90:0/2808745736 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ba80a6df0 con 0x561f79278390 2024-04-25T11:35:44.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.227+0000 7f5b957fa640 1 -- 172.21.15.90:0/2808745736 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5ba4059050 con 0x561f79278390 2024-04-25T11:35:44.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.227+0000 7f5bba99f740 10 client.15720 fetch_fsmap learned FSMap version 26 2024-04-25T11:35:44.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.227+0000 7f5bba99f740 1 -- 172.21.15.90:0/2808745736 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561f7935a7d0 con 0x561f79278390 2024-04-25T11:35:44.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.227+0000 7f5b957fa640 1 -- 172.21.15.90:0/2808745736 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ba80a6df0 con 0x561f79278390 2024-04-25T11:35:44.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.228+0000 7f5b957fa640 1 -- 172.21.15.90:0/2808745736 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f5ba40039d0 con 0x561f79278390 2024-04-25T11:35:44.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.228+0000 7f5bba99f740 10 client.15720 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:35:44.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.228+0000 7f5bba99f740 10 client.15720 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:35:44.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.228+0000 7f5bba99f740 20 client.15720 populate_metadata read hostname 'smithi090' 2024-04-25T11:35:44.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.228+0000 7f5bba99f740 1 --2- 172.21.15.90:0/2808745736 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x561f79236f60 0x561f79636580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.228+0000 7f5bba99f740 4 client.15720 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:35:44.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.228+0000 7f5bba99f740 1 -- 172.21.15.90:0/2808745736 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f7958c990 con 0x561f79236f60 2024-04-25T11:35:44.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.228+0000 7f5bb188a640 1 --2- 172.21.15.90:0/2808745736 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x561f79236f60 0x561f79636580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.228+0000 7f5b957fa640 10 client.15720 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:35:44.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.228+0000 7f5bb188a640 1 --2- 172.21.15.90:0/2808745736 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x561f79236f60 0x561f79636580 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f82d1565ceff3230 server_cookie=772ad5e2ddd6bf19 in_seq=0 out_seq=0 2024-04-25T11:35:44.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.229+0000 7f5b957fa640 1 -- 172.21.15.90:0/2808745736 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561f7958c990 con 0x561f79236f60 2024-04-25T11:35:44.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.229+0000 7f5b957fa640 10 client.15720 handle_command_reply: tid=0 2024-04-25T11:35:44.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.229+0000 7f5bba99f740 1 client.15720 shutdown 2024-04-25T11:35:44.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.229+0000 7f5bba99f740 1 -- 172.21.15.90:0/2808745736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f79278390 msgr2=0x561f796301b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.229+0000 7f5bba99f740 1 --2- 172.21.15.90:0/2808745736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f79278390 0x561f796301b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4009900 tx=0x7f5ba4009260 comp rx=0 tx=0).stop 2024-04-25T11:35:44.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.229+0000 7f5bba99f740 1 -- 172.21.15.90:0/2808745736 shutdown_connections 2024-04-25T11:35:44.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.229+0000 7f5bba99f740 1 --2- 172.21.15.90:0/2808745736 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x561f79236f60 0x561f79636580 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.229+0000 7f5bba99f740 1 --2- 172.21.15.90:0/2808745736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f79278390 0x561f796301b0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.229+0000 7f5bba99f740 1 --2- 172.21.15.90:0/2808745736 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561f792576b0 0x561f79624490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.229+0000 7f5bba99f740 1 --2- 172.21.15.90:0/2808745736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f7924f4d0 0x561f79623f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.229+0000 7f5b957fa640 0 client.15720 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:35:44.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.229+0000 7f5bba99f740 1 -- 172.21.15.90:0/2808745736 >> 172.21.15.90:0/2808745736 conn(0x561f7926c4e0 msgr2=0x561f7961b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:44.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.229+0000 7f5bba99f740 1 -- 172.21.15.90:0/2808745736 shutdown_connections 2024-04-25T11:35:44.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.229+0000 7f5bba99f740 1 -- 172.21.15.90:0/2808745736 wait complete. 2024-04-25T11:35:44.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.230+0000 7f5bba99f740 20 client.15720 trim_cache size 0 max 16384 2024-04-25T11:35:44.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.243+0000 7f5bba99f740 1 Processor -- start 2024-04-25T11:35:44.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.243+0000 7f5bba99f740 1 -- start start 2024-04-25T11:35:44.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.244+0000 7f5bba99f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561f7924f4d0 0x561f79623250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.244+0000 7f5bba99f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f792576b0 0x561f7962fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.244+0000 7f5bba99f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f79278390 0x561f7961e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.244+0000 7f5bba99f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac1b6150 con 0x561f792576b0 2024-04-25T11:35:44.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.244+0000 7f5bba99f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac1b7c90 con 0x561f7924f4d0 2024-04-25T11:35:44.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.244+0000 7f5bba99f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac1b6820 con 0x561f79278390 2024-04-25T11:35:44.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.244+0000 7f5bb1089640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f792576b0 0x561f7962fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.244+0000 7f5bb1089640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f792576b0 0x561f7962fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43586/0 (socket says 172.21.15.90:43586) 2024-04-25T11:35:44.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.244+0000 7f5bb1089640 1 -- 172.21.15.90:0/554378311 learned_addr learned my addr 172.21.15.90:0/554378311 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:44.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.244+0000 7f5bb188a640 1 --2- 172.21.15.90:0/554378311 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561f7924f4d0 0x561f79623250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.244+0000 7f5bb208b640 1 --2- 172.21.15.90:0/554378311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f79278390 0x561f7961e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.244+0000 7f5bb1089640 1 -- 172.21.15.90:0/554378311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f79278390 msgr2=0x561f7961e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:44.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.244+0000 7f5bb1089640 1 --2- 172.21.15.90:0/554378311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f79278390 0x561f7961e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.244+0000 7f5bb1089640 1 -- 172.21.15.90:0/554378311 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561f7924f4d0 msgr2=0x561f79623250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:44.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.244+0000 7f5bb1089640 1 --2- 172.21.15.90:0/554378311 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561f7924f4d0 0x561f79623250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.244+0000 7f5bb1089640 1 -- 172.21.15.90:0/554378311 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f79615760 con 0x561f792576b0 2024-04-25T11:35:44.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.244+0000 7f5bb1089640 1 --2- 172.21.15.90:0/554378311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f792576b0 0x561f7962fe10 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c00bce0 tx=0x7f5b9c012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bc6280b2dc73d08 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:44.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.245+0000 7f5b977fe640 1 -- 172.21.15.90:0/554378311 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c004020 con 0x561f792576b0 2024-04-25T11:35:44.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.245+0000 7f5b977fe640 1 -- 172.21.15.90:0/554378311 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b9c0054d0 con 0x561f792576b0 2024-04-25T11:35:44.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.245+0000 7f5b977fe640 1 -- 172.21.15.90:0/554378311 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c004b20 con 0x561f792576b0 2024-04-25T11:35:44.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.245+0000 7f5bba99f740 1 -- 172.21.15.90:0/554378311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f792576b0 msgr2=0x561f7962fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.245+0000 7f5bba99f740 1 --2- 172.21.15.90:0/554378311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f792576b0 0x561f7962fe10 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c00bce0 tx=0x7f5b9c012820 comp rx=0 tx=0).stop 2024-04-25T11:35:44.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.245+0000 7f5bba99f740 1 -- 172.21.15.90:0/554378311 shutdown_connections 2024-04-25T11:35:44.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.245+0000 7f5bba99f740 1 --2- 172.21.15.90:0/554378311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f79278390 0x561f7961e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.245+0000 7f5bba99f740 1 --2- 172.21.15.90:0/554378311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f792576b0 0x561f7962fe10 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.245+0000 7f5bba99f740 1 --2- 172.21.15.90:0/554378311 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561f7924f4d0 0x561f79623250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.246+0000 7f5bba99f740 1 -- 172.21.15.90:0/554378311 >> 172.21.15.90:0/554378311 conn(0x561f7926c4e0 msgr2=0x561f79625670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:44.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.246+0000 7f5bba99f740 1 -- 172.21.15.90:0/554378311 shutdown_connections 2024-04-25T11:35:44.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.246+0000 7f5bba99f740 1 -- 172.21.15.90:0/554378311 wait complete. 2024-04-25T11:35:44.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.246+0000 7f5bba99f740 1 -- start start 2024-04-25T11:35:44.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.246+0000 7f5bba99f740 1 Processor -- start 2024-04-25T11:35:44.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.246+0000 7f5bba99f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f7924f4d0 0x561f796274a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.246+0000 7f5bba99f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f792576b0 0x561f796279e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.246+0000 7f5bba99f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561f79278390 0x561f7969c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.246+0000 7f5bba99f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac1b75c0 con 0x561f792576b0 2024-04-25T11:35:44.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.246+0000 7f5bba99f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac1b6ef0 con 0x561f79278390 2024-04-25T11:35:44.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.246+0000 7f5bba99f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac1b4130 con 0x561f7924f4d0 2024-04-25T11:35:44.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.246+0000 7f5bba99f740 10 client.0 _ll_register_callbacks cb 0x7f5ba4031640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:35:44.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.246+0000 7f5bb1089640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f792576b0 0x561f796279e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.247+0000 7f5bb1089640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f792576b0 0x561f796279e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43588/0 (socket says 172.21.15.90:43588) 2024-04-25T11:35:44.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.247+0000 7f5bb1089640 1 -- 172.21.15.90:0/1648110816 learned_addr learned my addr 172.21.15.90:0/1648110816 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:44.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.247+0000 7f5bb208b640 1 --2- 172.21.15.90:0/1648110816 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561f79278390 0x561f7969c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.247+0000 7f5bb188a640 1 --2- 172.21.15.90:0/1648110816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f7924f4d0 0x561f796274a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.247+0000 7f5bb1089640 1 -- 172.21.15.90:0/1648110816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f7924f4d0 msgr2=0x561f796274a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:44.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.247+0000 7f5bb1089640 1 --2- 172.21.15.90:0/1648110816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f7924f4d0 0x561f796274a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.247+0000 7f5bb1089640 1 -- 172.21.15.90:0/1648110816 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561f79278390 msgr2=0x561f7969c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:44.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.247+0000 7f5bb1089640 1 --2- 172.21.15.90:0/1648110816 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561f79278390 0x561f7969c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.247+0000 7f5bb1089640 1 -- 172.21.15.90:0/1648110816 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f7969ce20 con 0x561f792576b0 2024-04-25T11:35:44.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.247+0000 7f5bb1089640 1 --2- 172.21.15.90:0/1648110816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f792576b0 0x561f796279e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c000c00 tx=0x7f5b9c005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84593087e59fb0f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:44.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.247+0000 7f5b957fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:35:44.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.247+0000 7f5b957fa640 1 -- 172.21.15.90:0/1648110816 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c028020 con 0x561f792576b0 2024-04-25T11:35:44.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.247+0000 7f5b957fa640 1 -- 172.21.15.90:0/1648110816 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f5b9c0046c0 con 0x561f792576b0 2024-04-25T11:35:44.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.247+0000 7f5bba99f740 1 -- 172.21.15.90:0/1648110816 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b9c0054d0 con 0x561f792576b0 2024-04-25T11:35:44.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.247+0000 7f5b957fa640 1 -- 172.21.15.90:0/1648110816 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5b9c033050 con 0x561f792576b0 2024-04-25T11:35:44.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.247+0000 7f5bba99f740 10 client.15726 fetch_fsmap learned FSMap version 26 2024-04-25T11:35:44.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.247+0000 7f5bba99f740 1 -- 172.21.15.90:0/1648110816 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561f7969e070 con 0x561f792576b0 2024-04-25T11:35:44.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.247+0000 7f5b957fa640 1 -- 172.21.15.90:0/1648110816 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b9c0054d0 con 0x561f792576b0 2024-04-25T11:35:44.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.248+0000 7f5b957fa640 1 -- 172.21.15.90:0/1648110816 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f5b9c01ba10 con 0x561f792576b0 2024-04-25T11:35:44.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.248+0000 7f5bba99f740 10 client.15726 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:35:44.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.248+0000 7f5bba99f740 10 client.15726 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:35:44.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.248+0000 7f5bba99f740 20 client.15726 populate_metadata read hostname 'smithi090' 2024-04-25T11:35:44.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.248+0000 7f5bba99f740 1 --2- 172.21.15.90:0/1648110816 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x561f7920c3a0 0x561f796a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.248+0000 7f5bba99f740 4 client.15726 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:35:44.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.248+0000 7f5bba99f740 1 -- 172.21.15.90:0/1648110816 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561f79586800 con 0x561f7920c3a0 2024-04-25T11:35:44.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.248+0000 7f5bb188a640 1 --2- 172.21.15.90:0/1648110816 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x561f7920c3a0 0x561f796a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.248+0000 7f5bb188a640 1 --2- 172.21.15.90:0/1648110816 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x561f7920c3a0 0x561f796a07d0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c0526df3461a7701 server_cookie=5c1f221d0f761879 in_seq=0 out_seq=0 2024-04-25T11:35:44.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.248+0000 7f5b957fa640 10 client.15726 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:35:44.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.249+0000 7f5b957fa640 1 -- 172.21.15.90:0/1648110816 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561f79586800 con 0x561f7920c3a0 2024-04-25T11:35:44.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.249+0000 7f5b957fa640 10 client.15726 handle_command_reply: tid=0 2024-04-25T11:35:44.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.249+0000 7f5bba99f740 1 client.15726 shutdown 2024-04-25T11:35:44.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.249+0000 7f5bba99f740 1 -- 172.21.15.90:0/1648110816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f792576b0 msgr2=0x561f796279e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.249+0000 7f5bba99f740 1 --2- 172.21.15.90:0/1648110816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f792576b0 0x561f796279e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c000c00 tx=0x7f5b9c005020 comp rx=0 tx=0).stop 2024-04-25T11:35:44.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.249+0000 7f5bba99f740 1 -- 172.21.15.90:0/1648110816 shutdown_connections 2024-04-25T11:35:44.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.249+0000 7f5bba99f740 1 --2- 172.21.15.90:0/1648110816 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x561f7920c3a0 0x561f796a07d0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.249+0000 7f5bba99f740 1 --2- 172.21.15.90:0/1648110816 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561f79278390 0x561f7969c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.249+0000 7f5bba99f740 1 --2- 172.21.15.90:0/1648110816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561f792576b0 0x561f796279e0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.249+0000 7f5bba99f740 1 --2- 172.21.15.90:0/1648110816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f7924f4d0 0x561f796274a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.249+0000 7f5b957fa640 0 client.15726 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:35:44.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.249+0000 7f5bba99f740 1 -- 172.21.15.90:0/1648110816 >> 172.21.15.90:0/1648110816 conn(0x561f7926c4e0 msgr2=0x561f79622fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:44.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.249+0000 7f5bba99f740 1 -- 172.21.15.90:0/1648110816 shutdown_connections 2024-04-25T11:35:44.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.249+0000 7f5bba99f740 1 -- 172.21.15.90:0/1648110816 wait complete. 2024-04-25T11:35:44.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.249+0000 7f5bba99f740 20 client.15726 trim_cache size 0 max 16384 2024-04-25T11:35:44.247 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:35:44.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.250+0000 7f5bb3aee640 1 -- 172.21.15.90:0/2823218972 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5b980afcf0 msgr2=0x7f5b980b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.250+0000 7f5bb3aee640 1 --2- 172.21.15.90:0/2823218972 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5b980afcf0 0x7f5b980b2200 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4005940 tx=0x7f5ba402a480 comp rx=0 tx=0).stop 2024-04-25T11:35:44.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.250+0000 7f5bb3aee640 1 -- 172.21.15.90:0/2823218972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bac102720 msgr2=0x7f5bac1a69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.250+0000 7f5bb3aee640 1 --2- 172.21.15.90:0/2823218972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bac102720 0x7f5bac1a69b0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5ba800db80 tx=0x7f5ba800b630 comp rx=0 tx=0).stop 2024-04-25T11:35:44.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.250+0000 7f5bb3aee640 1 -- 172.21.15.90:0/2823218972 shutdown_connections 2024-04-25T11:35:44.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.250+0000 7f5bb3aee640 1 --2- 172.21.15.90:0/2823218972 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5b980afcf0 0x7f5b980b2200 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.250+0000 7f5bb3aee640 1 --2- 172.21.15.90:0/2823218972 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5bac108de0 0x7f5bac1a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.250+0000 7f5bb3aee640 1 --2- 172.21.15.90:0/2823218972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bac1089d0 0x7f5bac1a6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.250+0000 7f5bb3aee640 1 --2- 172.21.15.90:0/2823218972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bac102720 0x7f5bac1a69b0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.250+0000 7f5bb3aee640 1 -- 172.21.15.90:0/2823218972 >> 172.21.15.90:0/2823218972 conn(0x7f5bac077880 msgr2=0x7f5bac105260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:44.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.250+0000 7f5bb3aee640 1 -- 172.21.15.90:0/2823218972 shutdown_connections 2024-04-25T11:35:44.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.250+0000 7f5bb3aee640 1 -- 172.21.15.90:0/2823218972 wait complete. 2024-04-25T11:35:44.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T11:35:44.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.370+0000 7f364f55f640 1 -- 172.21.15.90:0/4098316692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f364810e570 msgr2=0x7f3648110990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.370+0000 7f364f55f640 1 --2- 172.21.15.90:0/4098316692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f364810e570 0x7f3648110990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3640009ca0 tx=0x7f364002a8c0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.370+0000 7f364f55f640 1 -- 172.21.15.90:0/4098316692 shutdown_connections 2024-04-25T11:35:44.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.370+0000 7f364f55f640 1 --2- 172.21.15.90:0/4098316692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f364810e570 0x7f3648110990 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.370+0000 7f364f55f640 1 --2- 172.21.15.90:0/4098316692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3648102070 0x7f3648102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.370+0000 7f364f55f640 1 --2- 172.21.15.90:0/4098316692 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3648109bc0 0x7f3648101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.370+0000 7f364f55f640 1 -- 172.21.15.90:0/4098316692 >> 172.21.15.90:0/4098316692 conn(0x7f3648077920 msgr2=0x7f3648077d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:44.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.370+0000 7f364f55f640 1 -- 172.21.15.90:0/4098316692 shutdown_connections 2024-04-25T11:35:44.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.370+0000 7f364f55f640 1 -- 172.21.15.90:0/4098316692 wait complete. 2024-04-25T11:35:44.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.371+0000 7f364f55f640 1 Processor -- start 2024-04-25T11:35:44.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.371+0000 7f364f55f640 1 -- start start 2024-04-25T11:35:44.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.371+0000 7f364f55f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3648102070 0x7f364819a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.371+0000 7f364f55f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3648109bc0 0x7f364819ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.371+0000 7f364f55f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f364810e570 0x7f364819ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.371+0000 7f364f55f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3648113d30 con 0x7f3648109bc0 2024-04-25T11:35:44.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.371+0000 7f364f55f640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3648113bb0 con 0x7f364810e570 2024-04-25T11:35:44.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.371+0000 7f364f55f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3648113eb0 con 0x7f3648102070 2024-04-25T11:35:44.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.371+0000 7f364cafa640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3648109bc0 0x7f364819ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.371+0000 7f364cafa640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3648109bc0 0x7f364819ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43602/0 (socket says 172.21.15.90:43602) 2024-04-25T11:35:44.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.371+0000 7f364cafa640 1 -- 172.21.15.90:0/97735074 learned_addr learned my addr 172.21.15.90:0/97735074 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:44.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.371+0000 7f364d2fb640 1 --2- 172.21.15.90:0/97735074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3648102070 0x7f364819a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.371+0000 7f364dafc640 1 --2- 172.21.15.90:0/97735074 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f364810e570 0x7f364819ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.372+0000 7f364cafa640 1 -- 172.21.15.90:0/97735074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3648102070 msgr2=0x7f364819a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.372+0000 7f364cafa640 1 --2- 172.21.15.90:0/97735074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3648102070 0x7f364819a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.372+0000 7f364cafa640 1 -- 172.21.15.90:0/97735074 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f364810e570 msgr2=0x7f364819ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.372+0000 7f364cafa640 1 --2- 172.21.15.90:0/97735074 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f364810e570 0x7f364819ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.372+0000 7f364cafa640 1 -- 172.21.15.90:0/97735074 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f364819f4d0 con 0x7f3648109bc0 2024-04-25T11:35:44.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.372+0000 7f364dafc640 1 --2- 172.21.15.90:0/97735074 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f364810e570 0x7f364819ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:35:44.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.372+0000 7f364cafa640 1 --2- 172.21.15.90:0/97735074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3648109bc0 0x7f364819ac50 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f364400c9a0 tx=0x7f364400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:44.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.372+0000 7f363e7fc640 1 -- 172.21.15.90:0/97735074 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3644014070 con 0x7f3648109bc0 2024-04-25T11:35:44.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.372+0000 7f363e7fc640 1 -- 172.21.15.90:0/97735074 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3644004440 con 0x7f3648109bc0 2024-04-25T11:35:44.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.372+0000 7f364f55f640 1 -- 172.21.15.90:0/97735074 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f364819f7c0 con 0x7f3648109bc0 2024-04-25T11:35:44.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.372+0000 7f363e7fc640 1 -- 172.21.15.90:0/97735074 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f364400f450 con 0x7f3648109bc0 2024-04-25T11:35:44.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.372+0000 7f364f55f640 1 -- 172.21.15.90:0/97735074 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f364819fd10 con 0x7f3648109bc0 2024-04-25T11:35:44.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.374+0000 7f363e7fc640 1 -- 172.21.15.90:0/97735074 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 23) ==== 147696+0+0 (secure 0 0 0) 0x7f3644017070 con 0x7f3648109bc0 2024-04-25T11:35:44.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.375+0000 7f365640b740 1 Processor -- start 2024-04-25T11:35:44.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.375+0000 7f365640b740 1 -- start start 2024-04-25T11:35:44.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.375+0000 7f365640b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c257b8390 0x563c25b59810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.375+0000 7f365640b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c2578f4d0 0x563c25b59d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.375+0000 7f363e7fc640 1 --2- 172.21.15.90:0/97735074 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f36340afcf0 0x7f36340b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.375+0000 7f365640b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c257976b0 0x563c25b666a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.375+0000 7f365640b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563c25abed80 con 0x563c257976b0 2024-04-25T11:35:44.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.375+0000 7f365640b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x563c25a78a20 con 0x563c257b8390 2024-04-25T11:35:44.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.375+0000 7f365640b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563c25abad40 con 0x563c2578f4d0 2024-04-25T11:35:44.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.375+0000 7f364dafc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c257976b0 0x563c25b666a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.375+0000 7f364cafa640 1 --2- 172.21.15.90:0/97735074 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f36340afcf0 0x7f36340b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.375+0000 7f364dafc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c257976b0 0x563c25b666a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43612/0 (socket says 172.21.15.90:43612) 2024-04-25T11:35:44.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.375+0000 7f364dafc640 1 -- 172.21.15.90:0/1662992427 learned_addr learned my addr 172.21.15.90:0/1662992427 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:44.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.375+0000 7f363e7fc640 1 -- 172.21.15.90:0/97735074 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f36440dece0 con 0x7f3648109bc0 2024-04-25T11:35:44.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.375+0000 7f364d2fb640 1 --2- 172.21.15.90:0/1662992427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c2578f4d0 0x563c25b59d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.375+0000 7f364dafc640 1 --2- 172.21.15.90:0/1662992427 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c257b8390 0x563c25b59810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.376+0000 7f364dafc640 1 -- 172.21.15.90:0/1662992427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c2578f4d0 msgr2=0x563c25b59d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:44.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.376+0000 7f364dafc640 1 --2- 172.21.15.90:0/1662992427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c2578f4d0 0x563c25b59d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.376+0000 7f364dafc640 1 -- 172.21.15.90:0/1662992427 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c257b8390 msgr2=0x563c25b59810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:44.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.376+0000 7f364dafc640 1 --2- 172.21.15.90:0/1662992427 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c257b8390 0x563c25b59810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.376+0000 7f364dafc640 1 -- 172.21.15.90:0/1662992427 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c25a83c30 con 0x563c257976b0 2024-04-25T11:35:44.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.376+0000 7f364cafa640 1 --2- 172.21.15.90:0/97735074 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f36340afcf0 0x7f36340b2200 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f36440dc3d0 tx=0x7f364400cf90 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:44.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.376+0000 7f364dafc640 1 --2- 172.21.15.90:0/1662992427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c257976b0 0x563c25b666a0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f3640035000 tx=0x7f36400093a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=572f95cc8d6a52a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:44.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.376+0000 7f36337fe640 1 -- 172.21.15.90:0/1662992427 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3640032e80 con 0x563c257976b0 2024-04-25T11:35:44.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.376+0000 7f36337fe640 1 -- 172.21.15.90:0/1662992427 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3640036040 con 0x563c257976b0 2024-04-25T11:35:44.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.376+0000 7f36337fe640 1 -- 172.21.15.90:0/1662992427 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3640004040 con 0x563c257976b0 2024-04-25T11:35:44.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.377+0000 7f365640b740 1 -- 172.21.15.90:0/1662992427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c257976b0 msgr2=0x563c25b666a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.377+0000 7f365640b740 1 --2- 172.21.15.90:0/1662992427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c257976b0 0x563c25b666a0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f3640035000 tx=0x7f36400093a0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.377+0000 7f365640b740 1 -- 172.21.15.90:0/1662992427 shutdown_connections 2024-04-25T11:35:44.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.377+0000 7f365640b740 1 --2- 172.21.15.90:0/1662992427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c257976b0 0x563c25b666a0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.377+0000 7f365640b740 1 --2- 172.21.15.90:0/1662992427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c2578f4d0 0x563c25b59d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.377+0000 7f365640b740 1 --2- 172.21.15.90:0/1662992427 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c257b8390 0x563c25b59810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.377+0000 7f365640b740 1 -- 172.21.15.90:0/1662992427 >> 172.21.15.90:0/1662992427 conn(0x563c257ac4e0 msgr2=0x563c25b55fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:44.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.377+0000 7f365640b740 1 -- 172.21.15.90:0/1662992427 shutdown_connections 2024-04-25T11:35:44.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.377+0000 7f365640b740 1 -- 172.21.15.90:0/1662992427 wait complete. 2024-04-25T11:35:44.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.377+0000 7f365640b740 1 -- start start 2024-04-25T11:35:44.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.377+0000 7f365640b740 1 Processor -- start 2024-04-25T11:35:44.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.378+0000 7f365640b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c2578f4d0 0x563c25b64260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.378+0000 7f365640b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c257976b0 0x563c25b647a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.378+0000 7f365640b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c257b8390 0x563c25b700d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.378+0000 7f365640b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563c25b66be0 con 0x563c2578f4d0 2024-04-25T11:35:44.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.378+0000 7f365640b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x563c25a45dc0 con 0x563c257b8390 2024-04-25T11:35:44.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.378+0000 7f365640b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563c25b66d60 con 0x563c257976b0 2024-04-25T11:35:44.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.378+0000 7f365640b740 10 client.0 _ll_register_callbacks cb 0x563c25a61600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:35:44.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.378+0000 7f364dafc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c2578f4d0 0x563c25b64260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.378+0000 7f364dafc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c2578f4d0 0x563c25b64260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43626/0 (socket says 172.21.15.90:43626) 2024-04-25T11:35:44.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.378+0000 7f364dafc640 1 -- 172.21.15.90:0/2838969133 learned_addr learned my addr 172.21.15.90:0/2838969133 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:44.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.378+0000 7f364d2fb640 1 --2- 172.21.15.90:0/2838969133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c257976b0 0x563c25b647a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.378+0000 7f364dafc640 1 -- 172.21.15.90:0/2838969133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c257976b0 msgr2=0x563c25b647a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:44.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.378+0000 7f364dafc640 1 --2- 172.21.15.90:0/2838969133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c257976b0 0x563c25b647a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.378+0000 7f364dafc640 1 -- 172.21.15.90:0/2838969133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c257b8390 msgr2=0x563c25b700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:44.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.378+0000 7f364dafc640 1 --2- 172.21.15.90:0/2838969133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c257b8390 0x563c25b700d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.378+0000 7f364dafc640 1 -- 172.21.15.90:0/2838969133 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c257d03f0 con 0x563c2578f4d0 2024-04-25T11:35:44.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.378+0000 7f364dafc640 1 --2- 172.21.15.90:0/2838969133 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c2578f4d0 0x563c25b64260 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f3640006010 tx=0x7f3640003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beef3e431e075b0e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:44.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.378+0000 7f3630ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:35:44.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.378+0000 7f3630ff9640 1 -- 172.21.15.90:0/2838969133 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3640005bd0 con 0x563c2578f4d0 2024-04-25T11:35:44.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.379+0000 7f3630ff9640 1 -- 172.21.15.90:0/2838969133 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f3640004000 con 0x563c2578f4d0 2024-04-25T11:35:44.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.379+0000 7f365640b740 1 -- 172.21.15.90:0/2838969133 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3640036040 con 0x563c2578f4d0 2024-04-25T11:35:44.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.379+0000 7f3630ff9640 1 -- 172.21.15.90:0/2838969133 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3640032e80 con 0x563c2578f4d0 2024-04-25T11:35:44.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.379+0000 7f365640b740 10 client.15738 fetch_fsmap learned FSMap version 26 2024-04-25T11:35:44.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.379+0000 7f365640b740 1 -- 172.21.15.90:0/2838969133 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563c2589a7d0 con 0x563c2578f4d0 2024-04-25T11:35:44.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.379+0000 7f3630ff9640 1 -- 172.21.15.90:0/2838969133 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3640036040 con 0x563c2578f4d0 2024-04-25T11:35:44.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.379+0000 7f3630ff9640 1 -- 172.21.15.90:0/2838969133 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f364002fa10 con 0x563c2578f4d0 2024-04-25T11:35:44.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.379+0000 7f365640b740 10 client.15738 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:35:44.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.379+0000 7f365640b740 10 client.15738 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:35:44.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.379+0000 7f365640b740 20 client.15738 populate_metadata read hostname 'smithi090' 2024-04-25T11:35:44.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.379+0000 7f365640b740 1 --2- 172.21.15.90:0/2838969133 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563c25776f60 0x563c25b763b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.379+0000 7f365640b740 4 client.15738 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:35:44.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.379+0000 7f365640b740 1 -- 172.21.15.90:0/2838969133 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c25acc990 con 0x563c25776f60 2024-04-25T11:35:44.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.379+0000 7f364d2fb640 1 --2- 172.21.15.90:0/2838969133 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563c25776f60 0x563c25b763b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.380+0000 7f3630ff9640 10 client.15738 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:44.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.380+0000 7f364d2fb640 1 --2- 172.21.15.90:0/2838969133 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563c25776f60 0x563c25b763b0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7152697dedcf735 server_cookie=49ec3f4c42af8264 in_seq=0 out_seq=0 2024-04-25T11:35:44.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.381+0000 7f3630ff9640 1 -- 172.21.15.90:0/2838969133 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563c25acc990 con 0x563c25776f60 2024-04-25T11:35:44.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.381+0000 7f3630ff9640 10 client.15738 handle_command_reply: tid=0 2024-04-25T11:35:44.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.381+0000 7f365640b740 1 client.15738 shutdown 2024-04-25T11:35:44.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.381+0000 7f365640b740 1 -- 172.21.15.90:0/2838969133 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c2578f4d0 msgr2=0x563c25b64260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.381+0000 7f365640b740 1 --2- 172.21.15.90:0/2838969133 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c2578f4d0 0x563c25b64260 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f3640006010 tx=0x7f3640003ba0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.381+0000 7f365640b740 1 -- 172.21.15.90:0/2838969133 shutdown_connections 2024-04-25T11:35:44.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.381+0000 7f365640b740 1 --2- 172.21.15.90:0/2838969133 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563c25776f60 0x563c25b763b0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.381+0000 7f365640b740 1 --2- 172.21.15.90:0/2838969133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c257b8390 0x563c25b700d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.381+0000 7f365640b740 1 --2- 172.21.15.90:0/2838969133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c257976b0 0x563c25b647a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.381+0000 7f365640b740 1 --2- 172.21.15.90:0/2838969133 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c2578f4d0 0x563c25b64260 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.381+0000 7f3630ff9640 0 client.15738 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:44.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.381+0000 7f365640b740 1 -- 172.21.15.90:0/2838969133 >> 172.21.15.90:0/2838969133 conn(0x563c257ac4e0 msgr2=0x563c25b5b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:44.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.381+0000 7f365640b740 1 -- 172.21.15.90:0/2838969133 shutdown_connections 2024-04-25T11:35:44.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.381+0000 7f365640b740 1 -- 172.21.15.90:0/2838969133 wait complete. 2024-04-25T11:35:44.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.381+0000 7f365640b740 20 client.15738 trim_cache size 0 max 16384 2024-04-25T11:35:44.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.394+0000 7f365640b740 1 Processor -- start 2024-04-25T11:35:44.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.394+0000 7f365640b740 1 -- start start 2024-04-25T11:35:44.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.395+0000 7f365640b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c2578f4d0 0x563c25b63030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.395+0000 7f365640b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c257976b0 0x563c25b6fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.395+0000 7f365640b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c257b8390 0x563c25b7b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.395+0000 7f365640b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f36481afa00 con 0x563c257b8390 2024-04-25T11:35:44.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.395+0000 7f365640b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f36481b1540 con 0x563c257976b0 2024-04-25T11:35:44.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.395+0000 7f365640b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f36481b00d0 con 0x563c2578f4d0 2024-04-25T11:35:44.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.395+0000 7f364dafc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c257b8390 0x563c25b7b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.395+0000 7f364dafc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c257b8390 0x563c25b7b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43636/0 (socket says 172.21.15.90:43636) 2024-04-25T11:35:44.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.395+0000 7f364dafc640 1 -- 172.21.15.90:0/2476009741 learned_addr learned my addr 172.21.15.90:0/2476009741 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:44.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.395+0000 7f364dafc640 1 --2- 172.21.15.90:0/2476009741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c2578f4d0 0x563c25b63030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.395+0000 7f364d2fb640 1 --2- 172.21.15.90:0/2476009741 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c257976b0 0x563c25b6fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.395+0000 7f364dafc640 1 -- 172.21.15.90:0/2476009741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c2578f4d0 msgr2=0x563c25b63030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:44.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.395+0000 7f364dafc640 1 --2- 172.21.15.90:0/2476009741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c2578f4d0 0x563c25b63030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.395+0000 7f364dafc640 1 -- 172.21.15.90:0/2476009741 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c257976b0 msgr2=0x563c25b6fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:44.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.395+0000 7f364dafc640 1 --2- 172.21.15.90:0/2476009741 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c257976b0 0x563c25b6fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.395+0000 7f364dafc640 1 -- 172.21.15.90:0/2476009741 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c25b64f40 con 0x563c257b8390 2024-04-25T11:35:44.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.395+0000 7f364d2fb640 1 --2- 172.21.15.90:0/2476009741 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c257976b0 0x563c25b6fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:35:44.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.395+0000 7f364dafc640 1 --2- 172.21.15.90:0/2476009741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c257b8390 0x563c25b7b950 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f364002ade0 tx=0x7f36400093a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=176d1a0c19de0526 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:44.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.395+0000 7f3632ffd640 1 -- 172.21.15.90:0/2476009741 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36400098c0 con 0x563c257b8390 2024-04-25T11:35:44.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.395+0000 7f3632ffd640 1 -- 172.21.15.90:0/2476009741 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3640036040 con 0x563c257b8390 2024-04-25T11:35:44.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.395+0000 7f3632ffd640 1 -- 172.21.15.90:0/2476009741 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f364002fbc0 con 0x563c257b8390 2024-04-25T11:35:44.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.396+0000 7f365640b740 1 -- 172.21.15.90:0/2476009741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c257b8390 msgr2=0x563c25b7b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.396+0000 7f365640b740 1 --2- 172.21.15.90:0/2476009741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c257b8390 0x563c25b7b950 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f364002ade0 tx=0x7f36400093a0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.396+0000 7f365640b740 1 -- 172.21.15.90:0/2476009741 shutdown_connections 2024-04-25T11:35:44.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.396+0000 7f365640b740 1 --2- 172.21.15.90:0/2476009741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c257b8390 0x563c25b7b950 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.396+0000 7f365640b740 1 --2- 172.21.15.90:0/2476009741 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c257976b0 0x563c25b6fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.396+0000 7f365640b740 1 --2- 172.21.15.90:0/2476009741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c2578f4d0 0x563c25b63030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.396+0000 7f365640b740 1 -- 172.21.15.90:0/2476009741 >> 172.21.15.90:0/2476009741 conn(0x563c257ac4e0 msgr2=0x563c25b62920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:44.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.396+0000 7f365640b740 1 -- 172.21.15.90:0/2476009741 shutdown_connections 2024-04-25T11:35:44.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.396+0000 7f365640b740 1 -- 172.21.15.90:0/2476009741 wait complete. 2024-04-25T11:35:44.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.397+0000 7f365640b740 1 -- start start 2024-04-25T11:35:44.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.397+0000 7f365640b740 1 Processor -- start 2024-04-25T11:35:44.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.397+0000 7f365640b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c2578f4d0 0x563c25b674a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.397+0000 7f365640b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c257976b0 0x563c25b679e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.397+0000 7f365640b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c257b8390 0x563c25bd64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.397+0000 7f365640b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f36481b0e70 con 0x563c257976b0 2024-04-25T11:35:44.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.397+0000 7f365640b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f36481b07a0 con 0x563c257b8390 2024-04-25T11:35:44.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.397+0000 7f365640b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f36481ad9e0 con 0x563c2578f4d0 2024-04-25T11:35:44.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.397+0000 7f365640b740 10 client.0 _ll_register_callbacks cb 0x7f3640009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:35:44.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.397+0000 7f364d2fb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c257976b0 0x563c25b679e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.397+0000 7f364d2fb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c257976b0 0x563c25b679e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43642/0 (socket says 172.21.15.90:43642) 2024-04-25T11:35:44.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.397+0000 7f364d2fb640 1 -- 172.21.15.90:0/1802876534 learned_addr learned my addr 172.21.15.90:0/1802876534 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:44.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.397+0000 7f364dafc640 1 --2- 172.21.15.90:0/1802876534 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c257b8390 0x563c25bd64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.397+0000 7f364dafc640 1 --2- 172.21.15.90:0/1802876534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c2578f4d0 0x563c25b674a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.397+0000 7f364d2fb640 1 -- 172.21.15.90:0/1802876534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c2578f4d0 msgr2=0x563c25b674a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:44.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.397+0000 7f364d2fb640 1 --2- 172.21.15.90:0/1802876534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c2578f4d0 0x563c25b674a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.397+0000 7f364d2fb640 1 -- 172.21.15.90:0/1802876534 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c257b8390 msgr2=0x563c25bd64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:44.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.397+0000 7f364d2fb640 1 --2- 172.21.15.90:0/1802876534 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c257b8390 0x563c25bd64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.397+0000 7f364d2fb640 1 -- 172.21.15.90:0/1802876534 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c25bd6a30 con 0x563c257976b0 2024-04-25T11:35:44.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.397+0000 7f364dafc640 1 --2- 172.21.15.90:0/1802876534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c2578f4d0 0x563c25b674a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:35:44.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.398+0000 7f3630ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:35:44.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.398+0000 7f364d2fb640 1 --2- 172.21.15.90:0/1802876534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c257976b0 0x563c25b679e0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x563c25b56f50 tx=0x7f3638012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=201e8521e6f3446d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:44.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.398+0000 7f3630ff9640 1 -- 172.21.15.90:0/1802876534 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3638004250 con 0x563c257976b0 2024-04-25T11:35:44.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.398+0000 7f3630ff9640 1 -- 172.21.15.90:0/1802876534 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f36380043f0 con 0x563c257976b0 2024-04-25T11:35:44.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.398+0000 7f365640b740 1 -- 172.21.15.90:0/1802876534 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3640036040 con 0x563c257976b0 2024-04-25T11:35:44.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.398+0000 7f3630ff9640 1 -- 172.21.15.90:0/1802876534 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f363802d050 con 0x563c257976b0 2024-04-25T11:35:44.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.398+0000 7f365640b740 10 client.15744 fetch_fsmap learned FSMap version 26 2024-04-25T11:35:44.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.398+0000 7f365640b740 1 -- 172.21.15.90:0/1802876534 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563c25bddd50 con 0x563c257976b0 2024-04-25T11:35:44.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.398+0000 7f3630ff9640 1 -- 172.21.15.90:0/1802876534 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3640036040 con 0x563c257976b0 2024-04-25T11:35:44.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.398+0000 7f3630ff9640 1 -- 172.21.15.90:0/1802876534 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f3638003b60 con 0x563c257976b0 2024-04-25T11:35:44.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.398+0000 7f365640b740 10 client.15744 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:35:44.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.398+0000 7f365640b740 10 client.15744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:35:44.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.398+0000 7f365640b740 20 client.15744 populate_metadata read hostname 'smithi090' 2024-04-25T11:35:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.398+0000 7f365640b740 1 --2- 172.21.15.90:0/1802876534 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563c2574c3a0 0x563c25be0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.398+0000 7f365640b740 4 client.15744 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:35:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.398+0000 7f365640b740 1 -- 172.21.15.90:0/1802876534 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563c25ac6800 con 0x563c2574c3a0 2024-04-25T11:35:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.399+0000 7f364dafc640 1 --2- 172.21.15.90:0/1802876534 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563c2574c3a0 0x563c25be0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.399+0000 7f3630ff9640 10 client.15744 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.399+0000 7f364dafc640 1 --2- 172.21.15.90:0/1802876534 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563c2574c3a0 0x563c25be0540 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=499e6bbd58ee2dbe server_cookie=dc67542715f1fc7d in_seq=0 out_seq=0 2024-04-25T11:35:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.399+0000 7f3630ff9640 1 -- 172.21.15.90:0/1802876534 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563c25ac6800 con 0x563c2574c3a0 2024-04-25T11:35:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.399+0000 7f3630ff9640 10 client.15744 handle_command_reply: tid=0 2024-04-25T11:35:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.399+0000 7f365640b740 1 client.15744 shutdown 2024-04-25T11:35:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.399+0000 7f365640b740 1 -- 172.21.15.90:0/1802876534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c257976b0 msgr2=0x563c25b679e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.399+0000 7f365640b740 1 --2- 172.21.15.90:0/1802876534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c257976b0 0x563c25b679e0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x563c25b56f50 tx=0x7f3638012820 comp rx=0 tx=0).stop 2024-04-25T11:35:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.399+0000 7f365640b740 1 -- 172.21.15.90:0/1802876534 shutdown_connections 2024-04-25T11:35:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.399+0000 7f365640b740 1 --2- 172.21.15.90:0/1802876534 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563c2574c3a0 0x563c25be0540 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.399+0000 7f365640b740 1 --2- 172.21.15.90:0/1802876534 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c257b8390 0x563c25bd64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.399+0000 7f365640b740 1 --2- 172.21.15.90:0/1802876534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c257976b0 0x563c25b679e0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.399+0000 7f365640b740 1 --2- 172.21.15.90:0/1802876534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c2578f4d0 0x563c25b674a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.399+0000 7f3630ff9640 0 client.15744 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:44.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.399+0000 7f365640b740 1 -- 172.21.15.90:0/1802876534 >> 172.21.15.90:0/1802876534 conn(0x563c257ac4e0 msgr2=0x563c25b5be40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:44.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.400+0000 7f365640b740 1 -- 172.21.15.90:0/1802876534 shutdown_connections 2024-04-25T11:35:44.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.400+0000 7f365640b740 1 -- 172.21.15.90:0/1802876534 wait complete. 2024-04-25T11:35:44.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.400+0000 7f365640b740 20 client.15744 trim_cache size 0 max 16384 2024-04-25T11:35:44.395 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:35:44.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.400+0000 7f364f55f640 1 -- 172.21.15.90:0/97735074 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f36340afcf0 msgr2=0x7f36340b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.400+0000 7f364f55f640 1 --2- 172.21.15.90:0/97735074 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f36340afcf0 0x7f36340b2200 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f36440dc3d0 tx=0x7f364400cf90 comp rx=0 tx=0).stop 2024-04-25T11:35:44.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.400+0000 7f364f55f640 1 -- 172.21.15.90:0/97735074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3648109bc0 msgr2=0x7f364819ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:44.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.400+0000 7f364f55f640 1 --2- 172.21.15.90:0/97735074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3648109bc0 0x7f364819ac50 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f364400c9a0 tx=0x7f364400ce60 comp rx=0 tx=0).stop 2024-04-25T11:35:44.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.400+0000 7f364f55f640 1 -- 172.21.15.90:0/97735074 shutdown_connections 2024-04-25T11:35:44.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.400+0000 7f364f55f640 1 --2- 172.21.15.90:0/97735074 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f36340afcf0 0x7f36340b2200 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.400+0000 7f364f55f640 1 --2- 172.21.15.90:0/97735074 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f364810e570 0x7f364819ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.400+0000 7f364f55f640 1 --2- 172.21.15.90:0/97735074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3648109bc0 0x7f364819ac50 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.400+0000 7f364f55f640 1 --2- 172.21.15.90:0/97735074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3648102070 0x7f364819a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:44.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.400+0000 7f364f55f640 1 -- 172.21.15.90:0/97735074 >> 172.21.15.90:0/97735074 conn(0x7f3648077920 msgr2=0x7f364810a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:44.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.400+0000 7f364f55f640 1 -- 172.21.15.90:0/97735074 shutdown_connections 2024-04-25T11:35:44.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:44.400+0000 7f364f55f640 1 -- 172.21.15.90:0/97735074 wait complete. 2024-04-25T11:35:45.400 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T11:35:45.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:35:45.401 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T11:35:45.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.540+0000 7f68b2dce640 1 -- 172.21.15.90:0/2403074266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68ac107470 msgr2=0x7f68ac10ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:45.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.540+0000 7f68b2dce640 1 --2- 172.21.15.90:0/2403074266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68ac107470 0x7f68ac10ec10 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f68a4009ca0 tx=0x7f68a402a8c0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.540+0000 7f68b2dce640 1 -- 172.21.15.90:0/2403074266 shutdown_connections 2024-04-25T11:35:45.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.540+0000 7f68b2dce640 1 --2- 172.21.15.90:0/2403074266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68ac107470 0x7f68ac10ec10 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.540+0000 7f68b2dce640 1 --2- 172.21.15.90:0/2403074266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68ac1069b0 0x7f68ac106de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.540+0000 7f68b2dce640 1 --2- 172.21.15.90:0/2403074266 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f68ac102420 0x7f68ac102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.540+0000 7f68b2dce640 1 -- 172.21.15.90:0/2403074266 >> 172.21.15.90:0/2403074266 conn(0x7f68ac077920 msgr2=0x7f68ac077d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:45.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.540+0000 7f68b2dce640 1 -- 172.21.15.90:0/2403074266 shutdown_connections 2024-04-25T11:35:45.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.541+0000 7f68b2dce640 1 -- 172.21.15.90:0/2403074266 wait complete. 2024-04-25T11:35:45.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.541+0000 7f68b2dce640 1 Processor -- start 2024-04-25T11:35:45.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.541+0000 7f68b2dce640 1 -- start start 2024-04-25T11:35:45.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.541+0000 7f68b2dce640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68ac102420 0x7f68ac19a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.541+0000 7f68b2dce640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f68ac1069b0 0x7f68ac19ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.541+0000 7f68b2dce640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68ac107470 0x7f68ac19ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.541+0000 7f68b2dce640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f68ac112360 con 0x7f68ac107470 2024-04-25T11:35:45.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.541+0000 7f68b2dce640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f68ac1121e0 con 0x7f68ac1069b0 2024-04-25T11:35:45.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.541+0000 7f68b2dce640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f68ac1124e0 con 0x7f68ac102420 2024-04-25T11:35:45.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.542+0000 7f68b136b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68ac107470 0x7f68ac19ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.542+0000 7f68b136b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68ac107470 0x7f68ac19ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43658/0 (socket says 172.21.15.90:43658) 2024-04-25T11:35:45.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.542+0000 7f68b136b640 1 -- 172.21.15.90:0/1568276667 learned_addr learned my addr 172.21.15.90:0/1568276667 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:45.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.542+0000 7f68abfff640 1 --2- 172.21.15.90:0/1568276667 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f68ac1069b0 0x7f68ac19ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.542+0000 7f68b0b6a640 1 --2- 172.21.15.90:0/1568276667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68ac102420 0x7f68ac19a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.542+0000 7f68b136b640 1 -- 172.21.15.90:0/1568276667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68ac102420 msgr2=0x7f68ac19a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:45.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.542+0000 7f68b136b640 1 --2- 172.21.15.90:0/1568276667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68ac102420 0x7f68ac19a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.542+0000 7f68b136b640 1 -- 172.21.15.90:0/1568276667 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f68ac1069b0 msgr2=0x7f68ac19ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:45.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.542+0000 7f68b136b640 1 --2- 172.21.15.90:0/1568276667 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f68ac1069b0 0x7f68ac19ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.542+0000 7f68b136b640 1 -- 172.21.15.90:0/1568276667 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68ac19f580 con 0x7f68ac107470 2024-04-25T11:35:45.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.542+0000 7f68abfff640 1 --2- 172.21.15.90:0/1568276667 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f68ac1069b0 0x7f68ac19ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:35:45.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.542+0000 7f68b136b640 1 --2- 172.21.15.90:0/1568276667 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68ac107470 0x7f68ac19ef40 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f68a40098d0 tx=0x7f68a40317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:45.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.542+0000 7f68a9ffb640 1 -- 172.21.15.90:0/1568276667 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a4003b70 con 0x7f68ac107470 2024-04-25T11:35:45.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.542+0000 7f68a9ffb640 1 -- 172.21.15.90:0/1568276667 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68a4003d10 con 0x7f68ac107470 2024-04-25T11:35:45.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.542+0000 7f68b2dce640 1 -- 172.21.15.90:0/1568276667 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68ac19f810 con 0x7f68ac107470 2024-04-25T11:35:45.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.542+0000 7f68a9ffb640 1 -- 172.21.15.90:0/1568276667 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a402fa20 con 0x7f68ac107470 2024-04-25T11:35:45.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.542+0000 7f68b2dce640 1 -- 172.21.15.90:0/1568276667 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68ac19fd60 con 0x7f68ac107470 2024-04-25T11:35:45.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.545+0000 7f68b9c84740 1 Processor -- start 2024-04-25T11:35:45.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.546+0000 7f68a9ffb640 1 -- 172.21.15.90:0/1568276667 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 23) ==== 147696+0+0 (secure 0 0 0) 0x7f68a402fc40 con 0x7f68ac107470 2024-04-25T11:35:45.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.546+0000 7f68b9c84740 1 -- start start 2024-04-25T11:35:45.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.546+0000 7f68b9c84740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5f81e410 0x557f5fbbe4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.546+0000 7f68b9c84740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f7f5550 0x557f5fbbea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.546+0000 7f68b9c84740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5f7fd730 0x557f5fbbef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.546+0000 7f68b9c84740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557f5fb24b00 con 0x557f5f7f5550 2024-04-25T11:35:45.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.546+0000 7f68b9c84740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x557f5fac0920 con 0x557f5f7fd730 2024-04-25T11:35:45.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.546+0000 7f68b9c84740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x557f5fb20ac0 con 0x557f5f81e410 2024-04-25T11:35:45.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.546+0000 7f68b0b6a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f7f5550 0x557f5fbbea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.546+0000 7f68b0b6a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f7f5550 0x557f5fbbea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43668/0 (socket says 172.21.15.90:43668) 2024-04-25T11:35:45.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.546+0000 7f68b0b6a640 1 -- 172.21.15.90:0/1900117705 learned_addr learned my addr 172.21.15.90:0/1900117705 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:45.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.546+0000 7f68abfff640 1 --2- 172.21.15.90:0/1900117705 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5f81e410 0x557f5fbbe4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.546+0000 7f68a9ffb640 1 --2- 172.21.15.90:0/1568276667 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f689c0afcf0 0x7f689c0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.546+0000 7f68abfff640 1 --2- 172.21.15.90:0/1900117705 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5f7fd730 0x557f5fbbef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.546+0000 7f68b0b6a640 1 -- 172.21.15.90:0/1900117705 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5f81e410 msgr2=0x557f5fbbe4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:45.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.546+0000 7f68b0b6a640 1 --2- 172.21.15.90:0/1900117705 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5f81e410 0x557f5fbbe4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.546+0000 7f68b0b6a640 1 -- 172.21.15.90:0/1900117705 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5f7fd730 msgr2=0x557f5fbbef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:45.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.546+0000 7f68b0b6a640 1 --2- 172.21.15.90:0/1900117705 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5f7fd730 0x557f5fbbef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.546+0000 7f68b0b6a640 1 -- 172.21.15.90:0/1900117705 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f5f8fe7e0 con 0x557f5f7f5550 2024-04-25T11:35:45.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.546+0000 7f68abfff640 1 --2- 172.21.15.90:0/1900117705 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5f7fd730 0x557f5fbbef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:35:45.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.546+0000 7f68b136b640 1 --2- 172.21.15.90:0/1568276667 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f689c0afcf0 0x7f689c0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.546+0000 7f68a9ffb640 1 -- 172.21.15.90:0/1568276667 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f68a41006b0 con 0x7f68ac107470 2024-04-25T11:35:45.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.547+0000 7f68b0b6a640 1 --2- 172.21.15.90:0/1900117705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f7f5550 0x557f5fbbea10 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f68a0009c70 tx=0x7f68a00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e62bfdb78b8ffbc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:45.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.547+0000 7f6896ffd640 1 -- 172.21.15.90:0/1900117705 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a0012450 con 0x557f5f7f5550 2024-04-25T11:35:45.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.547+0000 7f6896ffd640 1 -- 172.21.15.90:0/1900117705 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68a0012aa0 con 0x557f5f7f5550 2024-04-25T11:35:45.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.547+0000 7f6896ffd640 1 -- 172.21.15.90:0/1900117705 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a000faf0 con 0x557f5f7f5550 2024-04-25T11:35:45.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.548+0000 7f68b9c84740 1 -- 172.21.15.90:0/1900117705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f7f5550 msgr2=0x557f5fbbea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:45.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.548+0000 7f68b136b640 1 --2- 172.21.15.90:0/1568276667 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f689c0afcf0 0x7f689c0b2200 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f68a40fdeb0 tx=0x7f68a40095c0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:45.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.548+0000 7f68b9c84740 1 --2- 172.21.15.90:0/1900117705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f7f5550 0x557f5fbbea10 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f68a0009c70 tx=0x7f68a00117b0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.548+0000 7f68b9c84740 1 -- 172.21.15.90:0/1900117705 shutdown_connections 2024-04-25T11:35:45.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.548+0000 7f68b9c84740 1 --2- 172.21.15.90:0/1900117705 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5f7fd730 0x557f5fbbef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.548+0000 7f68b9c84740 1 --2- 172.21.15.90:0/1900117705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f7f5550 0x557f5fbbea10 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.548+0000 7f68b9c84740 1 --2- 172.21.15.90:0/1900117705 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5f81e410 0x557f5fbbe4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.548+0000 7f68b9c84740 1 -- 172.21.15.90:0/1900117705 >> 172.21.15.90:0/1900117705 conn(0x557f5f812560 msgr2=0x557f5fbbadd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:45.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.548+0000 7f68b9c84740 1 -- 172.21.15.90:0/1900117705 shutdown_connections 2024-04-25T11:35:45.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.548+0000 7f68b9c84740 1 -- 172.21.15.90:0/1900117705 wait complete. 2024-04-25T11:35:45.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.548+0000 7f68b9c84740 1 -- start start 2024-04-25T11:35:45.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.548+0000 7f68b9c84740 1 Processor -- start 2024-04-25T11:35:45.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.549+0000 7f68b9c84740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f7f5550 0x557f5fbcd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.549+0000 7f68b9c84740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5f7fd730 0x557f5fbcd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.549+0000 7f68b9c84740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5f81e410 0x557f5fbd9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.549+0000 7f68b9c84740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557f5fbcfd80 con 0x557f5f7f5550 2024-04-25T11:35:45.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.549+0000 7f68b9c84740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x557f5fabe5d0 con 0x557f5f7fd730 2024-04-25T11:35:45.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.549+0000 7f68b9c84740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x557f5fbcff00 con 0x557f5f81e410 2024-04-25T11:35:45.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.549+0000 7f68abfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f7f5550 0x557f5fbcd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.549+0000 7f68b9c84740 10 client.0 _ll_register_callbacks cb 0x557f5faac080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:35:45.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.549+0000 7f68abfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f7f5550 0x557f5fbcd020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43682/0 (socket says 172.21.15.90:43682) 2024-04-25T11:35:45.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.549+0000 7f68abfff640 1 -- 172.21.15.90:0/2397746220 learned_addr learned my addr 172.21.15.90:0/2397746220 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:45.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.549+0000 7f68b0b6a640 1 --2- 172.21.15.90:0/2397746220 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5f7fd730 0x557f5fbcd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.549+0000 7f68abfff640 1 -- 172.21.15.90:0/2397746220 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5f81e410 msgr2=0x557f5fbd9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:45.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.549+0000 7f68abfff640 1 --2- 172.21.15.90:0/2397746220 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5f81e410 0x557f5fbd9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.549+0000 7f68abfff640 1 -- 172.21.15.90:0/2397746220 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5f7fd730 msgr2=0x557f5fbcd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:45.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.549+0000 7f68abfff640 1 --2- 172.21.15.90:0/2397746220 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5f7fd730 0x557f5fbcd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.549+0000 7f68abfff640 1 -- 172.21.15.90:0/2397746220 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f5f95ef70 con 0x557f5f7f5550 2024-04-25T11:35:45.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.549+0000 7f68abfff640 1 --2- 172.21.15.90:0/2397746220 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f7f5550 0x557f5fbcd020 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f6898000e40 tx=0x7f6898011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3023878d98a34c14 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:45.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.549+0000 7f688bfff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:35:45.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.549+0000 7f688bfff640 1 -- 172.21.15.90:0/2397746220 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6898005020 con 0x557f5f7f5550 2024-04-25T11:35:45.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.549+0000 7f688bfff640 1 -- 172.21.15.90:0/2397746220 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f68980051c0 con 0x557f5f7f5550 2024-04-25T11:35:45.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.549+0000 7f68b9c84740 1 -- 172.21.15.90:0/2397746220 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f68a0012aa0 con 0x557f5f7f5550 2024-04-25T11:35:45.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.550+0000 7f688bfff640 1 -- 172.21.15.90:0/2397746220 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f68980053c0 con 0x557f5f7f5550 2024-04-25T11:35:45.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.550+0000 7f68b9c84740 10 client.15756 fetch_fsmap learned FSMap version 26 2024-04-25T11:35:45.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.550+0000 7f68b9c84740 1 -- 172.21.15.90:0/2397746220 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557f5f7f6170 con 0x557f5f7f5550 2024-04-25T11:35:45.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.550+0000 7f688bfff640 1 -- 172.21.15.90:0/2397746220 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68a0012aa0 con 0x557f5f7f5550 2024-04-25T11:35:45.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.550+0000 7f688bfff640 1 -- 172.21.15.90:0/2397746220 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f689801d470 con 0x557f5f7f5550 2024-04-25T11:35:45.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.550+0000 7f68b9c84740 10 client.15756 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:35:45.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.550+0000 7f68b9c84740 10 client.15756 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:35:45.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.550+0000 7f68b9c84740 20 client.15756 populate_metadata read hostname 'smithi090' 2024-04-25T11:35:45.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.550+0000 7f68b9c84740 1 --2- 172.21.15.90:0/2397746220 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x557f5f7dd1f0 0x557f5fbdf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.550+0000 7f68b9c84740 4 client.15756 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:35:45.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.550+0000 7f68b9c84740 1 -- 172.21.15.90:0/2397746220 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f5fb32710 con 0x557f5f7dd1f0 2024-04-25T11:35:45.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.550+0000 7f68b0b6a640 1 --2- 172.21.15.90:0/2397746220 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x557f5f7dd1f0 0x557f5fbdf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.551+0000 7f68b0b6a640 1 --2- 172.21.15.90:0/2397746220 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x557f5f7dd1f0 0x557f5fbdf440 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9cb593383709347 server_cookie=c27d842ff2efdc37 in_seq=0 out_seq=0 2024-04-25T11:35:45.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.551+0000 7f688bfff640 10 client.15756 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:45.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.552+0000 7f688bfff640 1 -- 172.21.15.90:0/2397746220 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557f5fb32710 con 0x557f5f7dd1f0 2024-04-25T11:35:45.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.552+0000 7f688bfff640 10 client.15756 handle_command_reply: tid=0 2024-04-25T11:35:45.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.552+0000 7f68b9c84740 1 client.15756 shutdown 2024-04-25T11:35:45.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.552+0000 7f68b9c84740 1 -- 172.21.15.90:0/2397746220 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f7f5550 msgr2=0x557f5fbcd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:45.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.552+0000 7f68b9c84740 1 --2- 172.21.15.90:0/2397746220 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f7f5550 0x557f5fbcd020 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f6898000e40 tx=0x7f6898011810 comp rx=0 tx=0).stop 2024-04-25T11:35:45.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.552+0000 7f68b9c84740 1 -- 172.21.15.90:0/2397746220 shutdown_connections 2024-04-25T11:35:45.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.552+0000 7f68b9c84740 1 --2- 172.21.15.90:0/2397746220 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x557f5f7dd1f0 0x557f5fbdf440 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.552+0000 7f68b9c84740 1 --2- 172.21.15.90:0/2397746220 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5f81e410 0x557f5fbd9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.552+0000 7f68b9c84740 1 --2- 172.21.15.90:0/2397746220 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5f7fd730 0x557f5fbcd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.552+0000 7f68b9c84740 1 --2- 172.21.15.90:0/2397746220 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f7f5550 0x557f5fbcd020 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.552+0000 7f688bfff640 0 client.15756 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:45.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.552+0000 7f68b9c84740 1 -- 172.21.15.90:0/2397746220 >> 172.21.15.90:0/2397746220 conn(0x557f5f812560 msgr2=0x557f5fbc02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:45.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.552+0000 7f68b9c84740 1 -- 172.21.15.90:0/2397746220 shutdown_connections 2024-04-25T11:35:45.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.552+0000 7f68b9c84740 1 -- 172.21.15.90:0/2397746220 wait complete. 2024-04-25T11:35:45.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.552+0000 7f68b9c84740 20 client.15756 trim_cache size 0 max 16384 2024-04-25T11:35:45.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.570+0000 7f68b9c84740 1 Processor -- start 2024-04-25T11:35:45.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.570+0000 7f68b9c84740 1 -- start start 2024-04-25T11:35:45.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.570+0000 7f68b9c84740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5f7f5550 0x557f5fbb9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.570+0000 7f68b9c84740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5f7fd730 0x557f5fbc0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.570+0000 7f68b9c84740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f81e410 0x557f5fbe4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.570+0000 7f68b9c84740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f68ac1ad820 con 0x557f5f81e410 2024-04-25T11:35:45.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.570+0000 7f68b9c84740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f68ac1af360 con 0x557f5f7f5550 2024-04-25T11:35:45.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.570+0000 7f68b9c84740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f68ac1adef0 con 0x557f5f7fd730 2024-04-25T11:35:45.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.570+0000 7f68abfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f81e410 0x557f5fbe4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.570+0000 7f68abfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f81e410 0x557f5fbe4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43686/0 (socket says 172.21.15.90:43686) 2024-04-25T11:35:45.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.570+0000 7f68abfff640 1 -- 172.21.15.90:0/3326387325 learned_addr learned my addr 172.21.15.90:0/3326387325 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:45.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.570+0000 7f68b0b6a640 1 --2- 172.21.15.90:0/3326387325 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5f7fd730 0x557f5fbc0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.570+0000 7f68abfff640 1 -- 172.21.15.90:0/3326387325 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5f7fd730 msgr2=0x557f5fbc0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:45.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.570+0000 7f68abfff640 1 --2- 172.21.15.90:0/3326387325 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5f7fd730 0x557f5fbc0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.570+0000 7f68abfff640 1 -- 172.21.15.90:0/3326387325 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5f7f5550 msgr2=0x557f5fbb9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:45.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.570+0000 7f68abfff640 1 --2- 172.21.15.90:0/3326387325 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5f7f5550 0x557f5fbb9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.570+0000 7f68abfff640 1 -- 172.21.15.90:0/3326387325 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f5fbcc1b0 con 0x557f5f81e410 2024-04-25T11:35:45.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.570+0000 7f68b0b6a640 1 --2- 172.21.15.90:0/3326387325 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5f7fd730 0x557f5fbc0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:35:45.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.571+0000 7f68abfff640 1 --2- 172.21.15.90:0/3326387325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f81e410 0x557f5fbe4c00 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f689800b460 tx=0x7f6898012890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=452b044701a0df12 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:45.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.571+0000 7f68967fc640 1 -- 172.21.15.90:0/3326387325 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689802a070 con 0x557f5f81e410 2024-04-25T11:35:45.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.571+0000 7f68967fc640 1 -- 172.21.15.90:0/3326387325 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f689802b040 con 0x557f5f81e410 2024-04-25T11:35:45.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.571+0000 7f68967fc640 1 -- 172.21.15.90:0/3326387325 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6898004410 con 0x557f5f81e410 2024-04-25T11:35:45.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.571+0000 7f68b9c84740 1 -- 172.21.15.90:0/3326387325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f81e410 msgr2=0x557f5fbe4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:45.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.571+0000 7f68b9c84740 1 --2- 172.21.15.90:0/3326387325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f81e410 0x557f5fbe4c00 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f689800b460 tx=0x7f6898012890 comp rx=0 tx=0).stop 2024-04-25T11:35:45.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.572+0000 7f68b9c84740 1 -- 172.21.15.90:0/3326387325 shutdown_connections 2024-04-25T11:35:45.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.572+0000 7f68b9c84740 1 --2- 172.21.15.90:0/3326387325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f81e410 0x557f5fbe4c00 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.572+0000 7f68b9c84740 1 --2- 172.21.15.90:0/3326387325 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5f7fd730 0x557f5fbc0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.572+0000 7f68b9c84740 1 --2- 172.21.15.90:0/3326387325 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5f7f5550 0x557f5fbb9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.572+0000 7f68b9c84740 1 -- 172.21.15.90:0/3326387325 >> 172.21.15.90:0/3326387325 conn(0x557f5f812560 msgr2=0x557f5fbc0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:45.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.572+0000 7f68b9c84740 1 -- 172.21.15.90:0/3326387325 shutdown_connections 2024-04-25T11:35:45.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.572+0000 7f68b9c84740 1 -- 172.21.15.90:0/3326387325 wait complete. 2024-04-25T11:35:45.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.572+0000 7f68b9c84740 1 -- start start 2024-04-25T11:35:45.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.572+0000 7f68b9c84740 1 Processor -- start 2024-04-25T11:35:45.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.572+0000 7f68b9c84740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f7f5550 0x557f5fc3fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.572+0000 7f68b9c84740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5f7fd730 0x557f5fc3ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.572+0000 7f68b9c84740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5f81e410 0x557f5fbd4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.572+0000 7f68b9c84740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f68ac1aec90 con 0x557f5f7f5550 2024-04-25T11:35:45.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.572+0000 7f68b9c84740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f68ac1ae5c0 con 0x557f5f81e410 2024-04-25T11:35:45.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.572+0000 7f68b9c84740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f68ac1ab800 con 0x557f5f7fd730 2024-04-25T11:35:45.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.572+0000 7f68b9c84740 10 client.0 _ll_register_callbacks cb 0x7f6898012910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:35:45.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.573+0000 7f68abfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f7f5550 0x557f5fc3fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.573+0000 7f68abfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f7f5550 0x557f5fc3fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43692/0 (socket says 172.21.15.90:43692) 2024-04-25T11:35:45.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.573+0000 7f68abfff640 1 -- 172.21.15.90:0/2497228109 learned_addr learned my addr 172.21.15.90:0/2497228109 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:45.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.573+0000 7f68b0b6a640 1 --2- 172.21.15.90:0/2497228109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5f7fd730 0x557f5fc3ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.573+0000 7f68abfff640 1 -- 172.21.15.90:0/2497228109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5f7fd730 msgr2=0x557f5fc3ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:45.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.573+0000 7f68abfff640 1 --2- 172.21.15.90:0/2497228109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5f7fd730 0x557f5fc3ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.573+0000 7f68abfff640 1 -- 172.21.15.90:0/2497228109 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5f81e410 msgr2=0x557f5fbd4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:45.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.573+0000 7f68abfff640 1 --2- 172.21.15.90:0/2497228109 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5f81e410 0x557f5fbd4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.573+0000 7f68abfff640 1 -- 172.21.15.90:0/2497228109 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f5fbd5330 con 0x557f5f7f5550 2024-04-25T11:35:45.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.573+0000 7f68abfff640 1 --2- 172.21.15.90:0/2497228109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f7f5550 0x557f5fc3fa30 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f6898005790 tx=0x7f689801d720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e167ca9d44bb9b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:45.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.573+0000 7f688bfff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:35:45.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.573+0000 7f688bfff640 1 -- 172.21.15.90:0/2497228109 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689802a050 con 0x557f5f7f5550 2024-04-25T11:35:45.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.573+0000 7f688bfff640 1 -- 172.21.15.90:0/2497228109 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f68980035f0 con 0x557f5f7f5550 2024-04-25T11:35:45.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.573+0000 7f68b9c84740 1 -- 172.21.15.90:0/2497228109 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f689802b040 con 0x557f5f7f5550 2024-04-25T11:35:45.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.573+0000 7f688bfff640 1 -- 172.21.15.90:0/2497228109 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f68980037f0 con 0x557f5f7f5550 2024-04-25T11:35:45.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.573+0000 7f68b9c84740 10 client.15762 fetch_fsmap learned FSMap version 26 2024-04-25T11:35:45.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.573+0000 7f68b9c84740 1 -- 172.21.15.90:0/2497228109 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557f5fc47140 con 0x557f5f7f5550 2024-04-25T11:35:45.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.573+0000 7f688bfff640 1 -- 172.21.15.90:0/2497228109 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f689802b040 con 0x557f5f7f5550 2024-04-25T11:35:45.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.574+0000 7f688bfff640 1 -- 172.21.15.90:0/2497228109 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f689802e070 con 0x557f5f7f5550 2024-04-25T11:35:45.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.574+0000 7f68b9c84740 10 client.15762 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:35:45.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.574+0000 7f68b9c84740 10 client.15762 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:35:45.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.574+0000 7f68b9c84740 20 client.15762 populate_metadata read hostname 'smithi090' 2024-04-25T11:35:45.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.574+0000 7f68b9c84740 1 --2- 172.21.15.90:0/2497228109 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x557f5f7b2630 0x557f5fc497b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.574+0000 7f68b9c84740 4 client.15762 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T11:35:45.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.574+0000 7f68b9c84740 1 -- 172.21.15.90:0/2497228109 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557f5fb2c580 con 0x557f5f7b2630 2024-04-25T11:35:45.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.574+0000 7f68b0b6a640 1 --2- 172.21.15.90:0/2497228109 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x557f5f7b2630 0x557f5fc497b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.575+0000 7f688bfff640 10 client.15762 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:45.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.575+0000 7f68b0b6a640 1 --2- 172.21.15.90:0/2497228109 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x557f5f7b2630 0x557f5fc497b0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=594520a12daf755a server_cookie=f5b50fb6ad64fe12 in_seq=0 out_seq=0 2024-04-25T11:35:45.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.576+0000 7f688bfff640 1 -- 172.21.15.90:0/2497228109 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557f5fb2c580 con 0x557f5f7b2630 2024-04-25T11:35:45.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.576+0000 7f688bfff640 10 client.15762 handle_command_reply: tid=0 2024-04-25T11:35:45.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.576+0000 7f68b9c84740 1 client.15762 shutdown 2024-04-25T11:35:45.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.576+0000 7f68b9c84740 1 -- 172.21.15.90:0/2497228109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f7f5550 msgr2=0x557f5fc3fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:45.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.576+0000 7f68b9c84740 1 --2- 172.21.15.90:0/2497228109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f7f5550 0x557f5fc3fa30 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f6898005790 tx=0x7f689801d720 comp rx=0 tx=0).stop 2024-04-25T11:35:45.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.576+0000 7f68b9c84740 1 -- 172.21.15.90:0/2497228109 shutdown_connections 2024-04-25T11:35:45.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.576+0000 7f68b9c84740 1 --2- 172.21.15.90:0/2497228109 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x557f5f7b2630 0x557f5fc497b0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.576+0000 7f68b9c84740 1 --2- 172.21.15.90:0/2497228109 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557f5f81e410 0x557f5fbd4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.576+0000 7f688bfff640 0 client.15762 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:45.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.576+0000 7f68b9c84740 1 --2- 172.21.15.90:0/2497228109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557f5f7fd730 0x557f5fc3ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.576+0000 7f68b9c84740 1 --2- 172.21.15.90:0/2497228109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f5f7f5550 0x557f5fc3fa30 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.576+0000 7f68b9c84740 1 -- 172.21.15.90:0/2497228109 >> 172.21.15.90:0/2497228109 conn(0x557f5f812560 msgr2=0x557f5fbb9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:45.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.576+0000 7f68b9c84740 1 -- 172.21.15.90:0/2497228109 shutdown_connections 2024-04-25T11:35:45.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.576+0000 7f68b9c84740 1 -- 172.21.15.90:0/2497228109 wait complete. 2024-04-25T11:35:45.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.576+0000 7f68b9c84740 20 client.15762 trim_cache size 0 max 16384 2024-04-25T11:35:45.570 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:35:45.570 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-25T11:35:45.571 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "15bb81e3-6259-44bc-9c8f-47e671c2966a", 2024-04-25T11:35:45.571 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-25T11:35:45.571 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:35:45.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.577+0000 7f68b2dce640 1 -- 172.21.15.90:0/1568276667 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f689c0afcf0 msgr2=0x7f689c0b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:45.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.577+0000 7f68b2dce640 1 --2- 172.21.15.90:0/1568276667 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f689c0afcf0 0x7f689c0b2200 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f68a40fdeb0 tx=0x7f68a40095c0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.577+0000 7f68b2dce640 1 -- 172.21.15.90:0/1568276667 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68ac107470 msgr2=0x7f68ac19ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:45.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.577+0000 7f68b2dce640 1 --2- 172.21.15.90:0/1568276667 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68ac107470 0x7f68ac19ef40 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f68a40098d0 tx=0x7f68a40317b0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.577+0000 7f68b2dce640 1 -- 172.21.15.90:0/1568276667 shutdown_connections 2024-04-25T11:35:45.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.577+0000 7f68b2dce640 1 --2- 172.21.15.90:0/1568276667 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f689c0afcf0 0x7f689c0b2200 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.577+0000 7f68b2dce640 1 --2- 172.21.15.90:0/1568276667 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68ac107470 0x7f68ac19ef40 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.577+0000 7f68b2dce640 1 --2- 172.21.15.90:0/1568276667 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f68ac1069b0 0x7f68ac19ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.577+0000 7f68b2dce640 1 --2- 172.21.15.90:0/1568276667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68ac102420 0x7f68ac19a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.577+0000 7f68b2dce640 1 -- 172.21.15.90:0/1568276667 >> 172.21.15.90:0/1568276667 conn(0x7f68ac077920 msgr2=0x7f68ac108b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:45.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.577+0000 7f68b2dce640 1 -- 172.21.15.90:0/1568276667 shutdown_connections 2024-04-25T11:35:45.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.577+0000 7f68b2dce640 1 -- 172.21.15.90:0/1568276667 wait complete. 2024-04-25T11:35:45.577 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:35:45.577 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:35:45.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.699+0000 7f4ed3c61640 1 -- 172.21.15.90:0/3859727262 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ecc108a40 msgr2=0x7f4ecc102c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:45.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.699+0000 7f4ed3c61640 1 --2- 172.21.15.90:0/3859727262 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ecc108a40 0x7f4ecc102c40 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8009900 tx=0x7f4ec802a410 comp rx=0 tx=0).stop 2024-04-25T11:35:45.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.699+0000 7f4ed3c61640 1 -- 172.21.15.90:0/3859727262 shutdown_connections 2024-04-25T11:35:45.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.699+0000 7f4ed3c61640 1 --2- 172.21.15.90:0/3859727262 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ecc1037f0 0x7f4ecc10f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.699+0000 7f4ed3c61640 1 --2- 172.21.15.90:0/3859727262 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4ecc108ee0 0x7f4ecc103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.699+0000 7f4ed3c61640 1 --2- 172.21.15.90:0/3859727262 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ecc108a40 0x7f4ecc102c40 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.699+0000 7f4ed3c61640 1 -- 172.21.15.90:0/3859727262 >> 172.21.15.90:0/3859727262 conn(0x7f4ecc077920 msgr2=0x7f4ecc077d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:45.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.700+0000 7f4ed3c61640 1 -- 172.21.15.90:0/3859727262 shutdown_connections 2024-04-25T11:35:45.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.700+0000 7f4ed3c61640 1 -- 172.21.15.90:0/3859727262 wait complete. 2024-04-25T11:35:45.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.700+0000 7f4ed3c61640 1 Processor -- start 2024-04-25T11:35:45.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.700+0000 7f4ed3c61640 1 -- start start 2024-04-25T11:35:45.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.700+0000 7f4ed3c61640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ecc1037f0 0x7f4ecc19b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.700+0000 7f4ed3c61640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ecc108a40 0x7f4ecc19bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.700+0000 7f4ed3c61640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4ecc108ee0 0x7f4ecc19c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.700+0000 7f4ed3c61640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4ecc112c30 con 0x7f4ecc108a40 2024-04-25T11:35:45.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.700+0000 7f4ed3c61640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4ecc112ab0 con 0x7f4ecc108ee0 2024-04-25T11:35:45.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.700+0000 7f4ed3c61640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4ecc112db0 con 0x7f4ecc1037f0 2024-04-25T11:35:45.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.701+0000 7f4ed11fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ecc108a40 0x7f4ecc19bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.701+0000 7f4ed11fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ecc108a40 0x7f4ecc19bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43712/0 (socket says 172.21.15.90:43712) 2024-04-25T11:35:45.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.701+0000 7f4ed11fc640 1 -- 172.21.15.90:0/786069510 learned_addr learned my addr 172.21.15.90:0/786069510 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:45.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.701+0000 7f4ed19fd640 1 --2- 172.21.15.90:0/786069510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ecc1037f0 0x7f4ecc19b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.701+0000 7f4ed21fe640 1 --2- 172.21.15.90:0/786069510 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4ecc108ee0 0x7f4ecc19c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.701+0000 7f4ed11fc640 1 -- 172.21.15.90:0/786069510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ecc1037f0 msgr2=0x7f4ecc19b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:45.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.701+0000 7f4ed11fc640 1 --2- 172.21.15.90:0/786069510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ecc1037f0 0x7f4ecc19b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.701+0000 7f4ed11fc640 1 -- 172.21.15.90:0/786069510 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4ecc108ee0 msgr2=0x7f4ecc19c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:45.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.701+0000 7f4ed11fc640 1 --2- 172.21.15.90:0/786069510 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4ecc108ee0 0x7f4ecc19c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.701+0000 7f4ed11fc640 1 -- 172.21.15.90:0/786069510 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ecc1a6930 con 0x7f4ecc108a40 2024-04-25T11:35:45.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.701+0000 7f4ed21fe640 1 --2- 172.21.15.90:0/786069510 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4ecc108ee0 0x7f4ecc19c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:35:45.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.701+0000 7f4ed11fc640 1 --2- 172.21.15.90:0/786069510 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ecc108a40 0x7f4ecc19bb30 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc00e9f0 tx=0x7f4ebc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:45.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.701+0000 7f4ec2ffd640 1 -- 172.21.15.90:0/786069510 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc00ccc0 con 0x7f4ecc108a40 2024-04-25T11:35:45.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.701+0000 7f4ec2ffd640 1 -- 172.21.15.90:0/786069510 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ebc0035f0 con 0x7f4ecc108a40 2024-04-25T11:35:45.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.701+0000 7f4ed3c61640 1 -- 172.21.15.90:0/786069510 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ecc1a6c20 con 0x7f4ecc108a40 2024-04-25T11:35:45.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.701+0000 7f4ec2ffd640 1 -- 172.21.15.90:0/786069510 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc00ce60 con 0x7f4ecc108a40 2024-04-25T11:35:45.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.701+0000 7f4ed3c61640 1 -- 172.21.15.90:0/786069510 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ecc1a71c0 con 0x7f4ecc108a40 2024-04-25T11:35:45.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.704+0000 7f4edabed740 1 Processor -- start 2024-04-25T11:35:45.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.704+0000 7f4edabed740 1 -- start start 2024-04-25T11:35:45.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.704+0000 7f4ec2ffd640 1 -- 172.21.15.90:0/786069510 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 23) ==== 147696+0+0 (secure 0 0 0) 0x7f4ebc010690 con 0x7f4ecc108a40 2024-04-25T11:35:45.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.705+0000 7f4edabed740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56338b83e390 0x56338bbdea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.705+0000 7f4edabed740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b8154d0 0x56338bbdefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.705+0000 7f4edabed740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338b81d6b0 0x56338bbeb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.705+0000 7f4edabed740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56338bb44d80 con 0x56338b8154d0 2024-04-25T11:35:45.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.705+0000 7f4edabed740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56338bafe8b0 con 0x56338b83e390 2024-04-25T11:35:45.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.705+0000 7f4edabed740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56338bb40d40 con 0x56338b81d6b0 2024-04-25T11:35:45.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.705+0000 7f4ed19fd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b8154d0 0x56338bbdefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.705+0000 7f4ed19fd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b8154d0 0x56338bbdefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43716/0 (socket says 172.21.15.90:43716) 2024-04-25T11:35:45.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.705+0000 7f4ed19fd640 1 -- 172.21.15.90:0/2158441234 learned_addr learned my addr 172.21.15.90:0/2158441234 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:45.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.705+0000 7f4ed11fc640 1 --2- 172.21.15.90:0/2158441234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338b81d6b0 0x56338bbeb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.705+0000 7f4ed21fe640 1 --2- 172.21.15.90:0/2158441234 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56338b83e390 0x56338bbdea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.705+0000 7f4ec2ffd640 1 --2- 172.21.15.90:0/786069510 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4eb80afd70 0x7f4eb80b2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.705+0000 7f4ed19fd640 1 -- 172.21.15.90:0/2158441234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338b81d6b0 msgr2=0x56338bbeb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:45.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.705+0000 7f4ed19fd640 1 --2- 172.21.15.90:0/2158441234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338b81d6b0 0x56338bbeb8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.705+0000 7f4ed19fd640 1 -- 172.21.15.90:0/2158441234 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56338b83e390 msgr2=0x56338bbdea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:45.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.705+0000 7f4ed19fd640 1 --2- 172.21.15.90:0/2158441234 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56338b83e390 0x56338bbdea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.705+0000 7f4ed19fd640 1 -- 172.21.15.90:0/2158441234 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56338bacfa40 con 0x56338b8154d0 2024-04-25T11:35:45.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.705+0000 7f4ed21fe640 1 --2- 172.21.15.90:0/786069510 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4eb80afd70 0x7f4eb80b2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.705+0000 7f4ec2ffd640 1 -- 172.21.15.90:0/786069510 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f4ebc0df430 con 0x7f4ecc108a40 2024-04-25T11:35:45.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.705+0000 7f4ed19fd640 1 --2- 172.21.15.90:0/2158441234 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b8154d0 0x56338bbdefa0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8009900 tx=0x7f4ec8032bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44b3a2e14847851b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:45.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.706+0000 7f4eb7fff640 1 -- 172.21.15.90:0/2158441234 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8040070 con 0x56338b8154d0 2024-04-25T11:35:45.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.706+0000 7f4eb7fff640 1 -- 172.21.15.90:0/2158441234 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ec8004030 con 0x56338b8154d0 2024-04-25T11:35:45.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.706+0000 7f4eb7fff640 1 -- 172.21.15.90:0/2158441234 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8035040 con 0x56338b8154d0 2024-04-25T11:35:45.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.706+0000 7f4edabed740 1 -- 172.21.15.90:0/2158441234 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b8154d0 msgr2=0x56338bbdefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:45.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.706+0000 7f4ed21fe640 1 --2- 172.21.15.90:0/786069510 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4eb80afd70 0x7f4eb80b2280 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x56338bbdbbd0 tx=0x7f4ec4004a40 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:45.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.706+0000 7f4edabed740 1 --2- 172.21.15.90:0/2158441234 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b8154d0 0x56338bbdefa0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8009900 tx=0x7f4ec8032bd0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.707+0000 7f4edabed740 1 -- 172.21.15.90:0/2158441234 shutdown_connections 2024-04-25T11:35:45.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.707+0000 7f4edabed740 1 --2- 172.21.15.90:0/2158441234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338b81d6b0 0x56338bbeb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.707+0000 7f4edabed740 1 --2- 172.21.15.90:0/2158441234 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b8154d0 0x56338bbdefa0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.707+0000 7f4edabed740 1 --2- 172.21.15.90:0/2158441234 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56338b83e390 0x56338bbdea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.707+0000 7f4edabed740 1 -- 172.21.15.90:0/2158441234 >> 172.21.15.90:0/2158441234 conn(0x56338b8324e0 msgr2=0x56338bbdb210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:45.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.707+0000 7f4edabed740 1 -- 172.21.15.90:0/2158441234 shutdown_connections 2024-04-25T11:35:45.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.707+0000 7f4edabed740 1 -- 172.21.15.90:0/2158441234 wait complete. 2024-04-25T11:35:45.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.707+0000 7f4edabed740 1 -- start start 2024-04-25T11:35:45.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.707+0000 7f4edabed740 1 Processor -- start 2024-04-25T11:35:45.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.707+0000 7f4edabed740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56338b8154d0 0x56338bbe95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.708+0000 7f4edabed740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b81d6b0 0x56338bbe9b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.708+0000 7f4edabed740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338b83e390 0x56338bbef410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.708+0000 7f4edabed740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56338bbebf80 con 0x56338b81d6b0 2024-04-25T11:35:45.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.708+0000 7f4edabed740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56338bacbd10 con 0x56338b8154d0 2024-04-25T11:35:45.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.708+0000 7f4edabed740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56338bbec100 con 0x56338b83e390 2024-04-25T11:35:45.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.708+0000 7f4edabed740 10 client.0 _ll_register_callbacks cb 0x56338baf8210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:35:45.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.708+0000 7f4ed11fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b81d6b0 0x56338bbe9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.708+0000 7f4ed11fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b81d6b0 0x56338bbe9b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43732/0 (socket says 172.21.15.90:43732) 2024-04-25T11:35:45.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.708+0000 7f4ed11fc640 1 -- 172.21.15.90:0/3268433988 learned_addr learned my addr 172.21.15.90:0/3268433988 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:45.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.708+0000 7f4ed19fd640 1 --2- 172.21.15.90:0/3268433988 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56338b8154d0 0x56338bbe95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.708+0000 7f4ed21fe640 1 --2- 172.21.15.90:0/3268433988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338b83e390 0x56338bbef410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.708+0000 7f4ed11fc640 1 -- 172.21.15.90:0/3268433988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338b83e390 msgr2=0x56338bbef410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:45.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.708+0000 7f4ed11fc640 1 --2- 172.21.15.90:0/3268433988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338b83e390 0x56338bbef410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.708+0000 7f4ed11fc640 1 -- 172.21.15.90:0/3268433988 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56338b8154d0 msgr2=0x56338bbe95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:45.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.708+0000 7f4ed11fc640 1 --2- 172.21.15.90:0/3268433988 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56338b8154d0 0x56338bbe95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.708+0000 7f4ed11fc640 1 -- 172.21.15.90:0/3268433988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56338b9207d0 con 0x56338b81d6b0 2024-04-25T11:35:45.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.708+0000 7f4ed21fe640 1 --2- 172.21.15.90:0/3268433988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338b83e390 0x56338bbef410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:35:45.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.708+0000 7f4ed11fc640 1 --2- 172.21.15.90:0/3268433988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b81d6b0 0x56338bbe9b30 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc0a46d0 tx=0x7f4ebc003d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fa0a5d9394f5400 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:45.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.708+0000 7f4eb57fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:35:45.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.708+0000 7f4eb57fa640 1 -- 172.21.15.90:0/3268433988 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc0a7230 con 0x56338b81d6b0 2024-04-25T11:35:45.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.708+0000 7f4eb57fa640 1 -- 172.21.15.90:0/3268433988 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f4ebc0afe30 con 0x56338b81d6b0 2024-04-25T11:35:45.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.708+0000 7f4edabed740 1 -- 172.21.15.90:0/3268433988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ec8004030 con 0x56338b81d6b0 2024-04-25T11:35:45.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.708+0000 7f4eb57fa640 1 -- 172.21.15.90:0/3268433988 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4ebc0a6bd0 con 0x56338b81d6b0 2024-04-25T11:35:45.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.708+0000 7f4edabed740 10 client.15774 fetch_fsmap learned FSMap version 26 2024-04-25T11:35:45.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.708+0000 7f4edabed740 1 -- 172.21.15.90:0/3268433988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56338b8563f0 con 0x56338b81d6b0 2024-04-25T11:35:45.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.709+0000 7f4eb57fa640 1 -- 172.21.15.90:0/3268433988 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ec8004030 con 0x56338b81d6b0 2024-04-25T11:35:45.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.709+0000 7f4eb57fa640 1 -- 172.21.15.90:0/3268433988 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f4ebc0aeb90 con 0x56338b81d6b0 2024-04-25T11:35:45.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.709+0000 7f4edabed740 10 client.15774 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:35:45.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.709+0000 7f4edabed740 10 client.15774 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:35:45.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.709+0000 7f4edabed740 20 client.15774 populate_metadata read hostname 'smithi090' 2024-04-25T11:35:45.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.709+0000 7f4edabed740 1 --2- 172.21.15.90:0/3268433988 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56338b7fcf60 0x56338bbfb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.709+0000 7f4edabed740 4 client.15774 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:35:45.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.709+0000 7f4edabed740 1 -- 172.21.15.90:0/3268433988 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56338bb52990 con 0x56338b7fcf60 2024-04-25T11:35:45.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.709+0000 7f4ed19fd640 1 --2- 172.21.15.90:0/3268433988 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56338b7fcf60 0x56338bbfb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.710+0000 7f4eb57fa640 10 client.15774 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:45.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.710+0000 7f4ed19fd640 1 --2- 172.21.15.90:0/3268433988 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56338b7fcf60 0x56338bbfb7b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d96988cbd9855e78 server_cookie=9674dd3acaff89ee in_seq=0 out_seq=0 2024-04-25T11:35:45.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.710+0000 7f4eb57fa640 1 -- 172.21.15.90:0/3268433988 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56338bb52990 con 0x56338b7fcf60 2024-04-25T11:35:45.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.710+0000 7f4eb57fa640 10 client.15774 handle_command_reply: tid=0 2024-04-25T11:35:45.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.710+0000 7f4edabed740 1 client.15774 shutdown 2024-04-25T11:35:45.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.711+0000 7f4edabed740 1 -- 172.21.15.90:0/3268433988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b81d6b0 msgr2=0x56338bbe9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:45.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.711+0000 7f4edabed740 1 --2- 172.21.15.90:0/3268433988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b81d6b0 0x56338bbe9b30 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc0a46d0 tx=0x7f4ebc003d90 comp rx=0 tx=0).stop 2024-04-25T11:35:45.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.711+0000 7f4edabed740 1 -- 172.21.15.90:0/3268433988 shutdown_connections 2024-04-25T11:35:45.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.711+0000 7f4edabed740 1 --2- 172.21.15.90:0/3268433988 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56338b7fcf60 0x56338bbfb7b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.711+0000 7f4edabed740 1 --2- 172.21.15.90:0/3268433988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338b83e390 0x56338bbef410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.711+0000 7f4edabed740 1 --2- 172.21.15.90:0/3268433988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b81d6b0 0x56338bbe9b30 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.711+0000 7f4eb57fa640 0 client.15774 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:45.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.711+0000 7f4edabed740 1 --2- 172.21.15.90:0/3268433988 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56338b8154d0 0x56338bbe95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.711+0000 7f4edabed740 1 -- 172.21.15.90:0/3268433988 >> 172.21.15.90:0/3268433988 conn(0x56338b8324e0 msgr2=0x56338bbe0780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:45.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.711+0000 7f4edabed740 1 -- 172.21.15.90:0/3268433988 shutdown_connections 2024-04-25T11:35:45.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.711+0000 7f4edabed740 1 -- 172.21.15.90:0/3268433988 wait complete. 2024-04-25T11:35:45.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.711+0000 7f4edabed740 20 client.15774 trim_cache size 0 max 16384 2024-04-25T11:35:45.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.729+0000 7f4edabed740 1 Processor -- start 2024-04-25T11:35:45.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.729+0000 7f4edabed740 1 -- start start 2024-04-25T11:35:45.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.729+0000 7f4edabed740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56338b8154d0 0x56338bbeb230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.729+0000 7f4edabed740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338b81d6b0 0x56338bc00f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.729+0000 7f4edabed740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b83e390 0x56338bc014b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.729+0000 7f4edabed740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4ecc1afb00 con 0x56338b83e390 2024-04-25T11:35:45.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.729+0000 7f4edabed740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4ecc1b1640 con 0x56338b8154d0 2024-04-25T11:35:45.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.729+0000 7f4edabed740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4ecc1b01d0 con 0x56338b81d6b0 2024-04-25T11:35:45.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.729+0000 7f4ed21fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b83e390 0x56338bc014b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.729+0000 7f4ed21fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b83e390 0x56338bc014b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43746/0 (socket says 172.21.15.90:43746) 2024-04-25T11:35:45.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.729+0000 7f4ed21fe640 1 -- 172.21.15.90:0/42700880 learned_addr learned my addr 172.21.15.90:0/42700880 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:45.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.729+0000 7f4ed19fd640 1 --2- 172.21.15.90:0/42700880 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56338b8154d0 0x56338bbeb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.729+0000 7f4ed11fc640 1 --2- 172.21.15.90:0/42700880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338b81d6b0 0x56338bc00f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.729+0000 7f4ed21fe640 1 -- 172.21.15.90:0/42700880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338b81d6b0 msgr2=0x56338bc00f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:45.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.729+0000 7f4ed21fe640 1 --2- 172.21.15.90:0/42700880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338b81d6b0 0x56338bc00f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.729+0000 7f4ed21fe640 1 -- 172.21.15.90:0/42700880 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56338b8154d0 msgr2=0x56338bbeb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:45.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.729+0000 7f4ed21fe640 1 --2- 172.21.15.90:0/42700880 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56338b8154d0 0x56338bbeb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.729+0000 7f4ed21fe640 1 -- 172.21.15.90:0/42700880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56338b8160f0 con 0x56338b83e390 2024-04-25T11:35:45.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.729+0000 7f4ed19fd640 1 --2- 172.21.15.90:0/42700880 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56338b8154d0 0x56338bbeb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:35:45.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.729+0000 7f4ed11fc640 1 --2- 172.21.15.90:0/42700880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338b81d6b0 0x56338bc00f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:35:45.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.729+0000 7f4ed21fe640 1 --2- 172.21.15.90:0/42700880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b83e390 0x56338bc014b0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4013840 tx=0x7f4ec4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcc59e3fb11f5a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:45.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.730+0000 7f4eb77fe640 1 -- 172.21.15.90:0/42700880 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec401fbd0 con 0x56338b83e390 2024-04-25T11:35:45.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.730+0000 7f4eb77fe640 1 -- 172.21.15.90:0/42700880 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ec4017070 con 0x56338b83e390 2024-04-25T11:35:45.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.730+0000 7f4eb77fe640 1 -- 172.21.15.90:0/42700880 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec40273e0 con 0x56338b83e390 2024-04-25T11:35:45.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.730+0000 7f4edabed740 1 -- 172.21.15.90:0/42700880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b83e390 msgr2=0x56338bc014b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:45.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.730+0000 7f4edabed740 1 --2- 172.21.15.90:0/42700880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b83e390 0x56338bc014b0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4013840 tx=0x7f4ec4015000 comp rx=0 tx=0).stop 2024-04-25T11:35:45.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.730+0000 7f4edabed740 1 -- 172.21.15.90:0/42700880 shutdown_connections 2024-04-25T11:35:45.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.730+0000 7f4edabed740 1 --2- 172.21.15.90:0/42700880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b83e390 0x56338bc014b0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.730+0000 7f4edabed740 1 --2- 172.21.15.90:0/42700880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338b81d6b0 0x56338bc00f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.730+0000 7f4edabed740 1 --2- 172.21.15.90:0/42700880 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56338b8154d0 0x56338bbeb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.730+0000 7f4edabed740 1 -- 172.21.15.90:0/42700880 >> 172.21.15.90:0/42700880 conn(0x56338b8324e0 msgr2=0x56338bbeab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:45.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.730+0000 7f4edabed740 1 -- 172.21.15.90:0/42700880 shutdown_connections 2024-04-25T11:35:45.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.730+0000 7f4edabed740 1 -- 172.21.15.90:0/42700880 wait complete. 2024-04-25T11:35:45.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.731+0000 7f4edabed740 1 -- start start 2024-04-25T11:35:45.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.731+0000 7f4edabed740 1 Processor -- start 2024-04-25T11:35:45.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.731+0000 7f4edabed740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56338b8154d0 0x56338bbf0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.731+0000 7f4edabed740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b81d6b0 0x56338bbf1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.731+0000 7f4edabed740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338b83e390 0x56338bc5bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.731+0000 7f4edabed740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4ecc1b0f70 con 0x56338b81d6b0 2024-04-25T11:35:45.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.731+0000 7f4edabed740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4ecc1b08a0 con 0x56338b8154d0 2024-04-25T11:35:45.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.731+0000 7f4edabed740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4ecc1adae0 con 0x56338b83e390 2024-04-25T11:35:45.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.731+0000 7f4edabed740 10 client.0 _ll_register_callbacks cb 0x7f4ebc0e2c50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:35:45.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.731+0000 7f4ed11fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b81d6b0 0x56338bbf1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.731+0000 7f4ed11fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b81d6b0 0x56338bbf1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43760/0 (socket says 172.21.15.90:43760) 2024-04-25T11:35:45.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.731+0000 7f4ed11fc640 1 -- 172.21.15.90:0/837440362 learned_addr learned my addr 172.21.15.90:0/837440362 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:45.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.731+0000 7f4ed19fd640 1 --2- 172.21.15.90:0/837440362 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56338b8154d0 0x56338bbf0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.732+0000 7f4ed11fc640 1 -- 172.21.15.90:0/837440362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338b83e390 msgr2=0x56338bc5bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:45.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.732+0000 7f4ed11fc640 1 --2- 172.21.15.90:0/837440362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338b83e390 0x56338bc5bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.732+0000 7f4ed11fc640 1 -- 172.21.15.90:0/837440362 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56338b8154d0 msgr2=0x56338bbf0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:45.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.732+0000 7f4ed11fc640 1 --2- 172.21.15.90:0/837440362 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56338b8154d0 0x56338bbf0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.732+0000 7f4ed11fc640 1 -- 172.21.15.90:0/837440362 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56338bb054b0 con 0x56338b81d6b0 2024-04-25T11:35:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.732+0000 7f4ed19fd640 1 --2- 172.21.15.90:0/837440362 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56338b8154d0 0x56338bbf0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:35:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.732+0000 7f4ed11fc640 1 --2- 172.21.15.90:0/837440362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b81d6b0 0x56338bbf1340 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc0a34e0 tx=0x7f4ebc0ab810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc2ead52e832f4ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.732+0000 7f4eb57fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:35:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.732+0000 7f4eb57fa640 1 -- 172.21.15.90:0/837440362 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc0a6db0 con 0x56338b81d6b0 2024-04-25T11:35:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.732+0000 7f4eb57fa640 1 -- 172.21.15.90:0/837440362 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7f4ebc0a2e50 con 0x56338b81d6b0 2024-04-25T11:35:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.732+0000 7f4edabed740 1 -- 172.21.15.90:0/837440362 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ec4017070 con 0x56338b81d6b0 2024-04-25T11:35:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.732+0000 7f4eb57fa640 1 -- 172.21.15.90:0/837440362 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4ebc0a7230 con 0x56338b81d6b0 2024-04-25T11:35:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.732+0000 7f4edabed740 10 client.15780 fetch_fsmap learned FSMap version 26 2024-04-25T11:35:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.732+0000 7f4edabed740 1 -- 172.21.15.90:0/837440362 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56338bc630a0 con 0x56338b81d6b0 2024-04-25T11:35:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.732+0000 7f4eb57fa640 1 -- 172.21.15.90:0/837440362 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ec4017070 con 0x56338b81d6b0 2024-04-25T11:35:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.732+0000 7f4eb57fa640 1 -- 172.21.15.90:0/837440362 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f4ebc003fb0 con 0x56338b81d6b0 2024-04-25T11:35:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.732+0000 7f4edabed740 10 client.15780 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:35:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.732+0000 7f4edabed740 10 client.15780 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:35:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.732+0000 7f4edabed740 20 client.15780 populate_metadata read hostname 'smithi090' 2024-04-25T11:35:45.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.732+0000 7f4edabed740 1 --2- 172.21.15.90:0/837440362 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56338b7d23a0 0x56338bc65720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.732+0000 7f4edabed740 4 client.15780 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:35:45.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.732+0000 7f4edabed740 1 -- 172.21.15.90:0/837440362 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56338bbe8ab0 con 0x56338b7d23a0 2024-04-25T11:35:45.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.733+0000 7f4ed19fd640 1 --2- 172.21.15.90:0/837440362 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56338b7d23a0 0x56338bc65720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.733+0000 7f4eb57fa640 10 client.15780 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:45.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.733+0000 7f4ed19fd640 1 --2- 172.21.15.90:0/837440362 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56338b7d23a0 0x56338bc65720 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=840f863a9e2f0f6a server_cookie=17129429ad30ab76 in_seq=0 out_seq=0 2024-04-25T11:35:45.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.733+0000 7f4eb57fa640 1 -- 172.21.15.90:0/837440362 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56338bbe8ab0 con 0x56338b7d23a0 2024-04-25T11:35:45.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.733+0000 7f4eb57fa640 10 client.15780 handle_command_reply: tid=0 2024-04-25T11:35:45.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.733+0000 7f4edabed740 1 client.15780 shutdown 2024-04-25T11:35:45.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.734+0000 7f4edabed740 1 -- 172.21.15.90:0/837440362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b81d6b0 msgr2=0x56338bbf1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:45.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.734+0000 7f4edabed740 1 --2- 172.21.15.90:0/837440362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b81d6b0 0x56338bbf1340 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc0a34e0 tx=0x7f4ebc0ab810 comp rx=0 tx=0).stop 2024-04-25T11:35:45.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.734+0000 7f4edabed740 1 -- 172.21.15.90:0/837440362 shutdown_connections 2024-04-25T11:35:45.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.734+0000 7f4edabed740 1 --2- 172.21.15.90:0/837440362 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56338b7d23a0 0x56338bc65720 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.734+0000 7f4edabed740 1 --2- 172.21.15.90:0/837440362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338b83e390 0x56338bc5bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.734+0000 7f4edabed740 1 --2- 172.21.15.90:0/837440362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56338b81d6b0 0x56338bbf1340 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.734+0000 7f4edabed740 1 --2- 172.21.15.90:0/837440362 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56338b8154d0 0x56338bbf0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.734+0000 7f4eb57fa640 0 client.15780 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:45.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.734+0000 7f4edabed740 1 -- 172.21.15.90:0/837440362 >> 172.21.15.90:0/837440362 conn(0x56338b8324e0 msgr2=0x56338bbe8330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:45.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.734+0000 7f4edabed740 1 -- 172.21.15.90:0/837440362 shutdown_connections 2024-04-25T11:35:45.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.734+0000 7f4edabed740 1 -- 172.21.15.90:0/837440362 wait complete. 2024-04-25T11:35:45.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.734+0000 7f4edabed740 20 client.15780 trim_cache size 0 max 16384 2024-04-25T11:35:45.729 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:35:45.729 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T11:35:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-25T11:35:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: "15bb81e3-6259-44bc-9c8f-47e671c2966a": { 2024-04-25T11:35:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-25T11:35:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "15bb81e3-6259-44bc-9c8f-47e671c2966a", 2024-04-25T11:35:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-25T11:35:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:35:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:35:45.730 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:35:45.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.734+0000 7f4ed3c61640 1 -- 172.21.15.90:0/786069510 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4eb80afd70 msgr2=0x7f4eb80b2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:45.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.734+0000 7f4ed3c61640 1 --2- 172.21.15.90:0/786069510 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4eb80afd70 0x7f4eb80b2280 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x56338bbdbbd0 tx=0x7f4ec4004a40 comp rx=0 tx=0).stop 2024-04-25T11:35:45.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.734+0000 7f4ed3c61640 1 -- 172.21.15.90:0/786069510 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ecc108a40 msgr2=0x7f4ecc19bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:45.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.734+0000 7f4ed3c61640 1 --2- 172.21.15.90:0/786069510 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ecc108a40 0x7f4ecc19bb30 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc00e9f0 tx=0x7f4ebc00eeb0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.734+0000 7f4ed3c61640 1 -- 172.21.15.90:0/786069510 shutdown_connections 2024-04-25T11:35:45.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.734+0000 7f4ed3c61640 1 --2- 172.21.15.90:0/786069510 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4eb80afd70 0x7f4eb80b2280 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.734+0000 7f4ed3c61640 1 --2- 172.21.15.90:0/786069510 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4ecc108ee0 0x7f4ecc19c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.734+0000 7f4ed3c61640 1 --2- 172.21.15.90:0/786069510 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ecc108a40 0x7f4ecc19bb30 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.734+0000 7f4ed3c61640 1 --2- 172.21.15.90:0/786069510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ecc1037f0 0x7f4ecc19b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.734+0000 7f4ed3c61640 1 -- 172.21.15.90:0/786069510 >> 172.21.15.90:0/786069510 conn(0x7f4ecc077920 msgr2=0x7f4ecc1040e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:45.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.735+0000 7f4ed3c61640 1 -- 172.21.15.90:0/786069510 shutdown_connections 2024-04-25T11:35:45.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:35:45.735+0000 7f4ed3c61640 1 -- 172.21.15.90:0/786069510 wait complete. 2024-04-25T11:35:45.732 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:45 smithi090 ceph-mon[29575]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T11:35:45.734 INFO:tasks.cephfs.filesystem:scrub status for tag:15bb81e3-6259-44bc-9c8f-47e671c2966a - {'path': '/', 'tag': '15bb81e3-6259-44bc-9c8f-47e671c2966a', 'options': 'recursive,force'} 2024-04-25T11:35:46.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:45 smithi138 ceph-mon[32328]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T11:35:46.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:45 smithi156 ceph-mon[33045]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T11:35:46.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:46 smithi090 ceph-mon[29575]: scrub queued for path: / 2024-04-25T11:35:46.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:46 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:35:46.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:46 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:35:46.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:46 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:35:47.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:46 smithi138 ceph-mon[32328]: scrub queued for path: / 2024-04-25T11:35:47.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:46 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:35:47.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:46 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:35:47.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:46 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:35:47.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:46 smithi156 ceph-mon[33045]: scrub queued for path: / 2024-04-25T11:35:47.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:46 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:35:47.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:46 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:35:47.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:46 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:35:47.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:47 smithi090 ceph-mon[29575]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-25T11:35:48.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:47 smithi138 ceph-mon[32328]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-25T11:35:48.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:47 smithi156 ceph-mon[33045]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-25T11:35:49.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:49 smithi090 ceph-mon[29575]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T11:35:50.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:49 smithi138 ceph-mon[32328]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T11:35:50.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:49 smithi156 ceph-mon[33045]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T11:35:50.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:50 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:35:51.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:50 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:35:51.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:50 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:35:51.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:51 smithi090 ceph-mon[29575]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-04-25T11:35:52.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:51 smithi138 ceph-mon[32328]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-04-25T11:35:52.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:51 smithi156 ceph-mon[33045]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-04-25T11:35:53.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:53 smithi090 ceph-mon[29575]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-25T11:35:54.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:53 smithi138 ceph-mon[32328]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-25T11:35:54.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:53 smithi156 ceph-mon[33045]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-25T11:35:54.378 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.387+0000 7fc11dfa8640 1 -- 172.21.15.90:0/2148506571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc118109b70 msgr2=0x7fc118103710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:54.378 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.387+0000 7fc11dfa8640 1 --2- 172.21.15.90:0/2148506571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc118109b70 0x7fc118103710 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fc108009ca0 tx=0x7fc10802a8c0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.379 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.387+0000 7fc11dfa8640 1 -- 172.21.15.90:0/2148506571 shutdown_connections 2024-04-25T11:35:54.379 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.387+0000 7fc11dfa8640 1 --2- 172.21.15.90:0/2148506571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc1181042c0 0x7fc118110690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.379 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.387+0000 7fc11dfa8640 1 --2- 172.21.15.90:0/2148506571 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc11810a010 0x7fc118103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.379 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.387+0000 7fc11dfa8640 1 --2- 172.21.15.90:0/2148506571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc118109b70 0x7fc118103710 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.379 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.387+0000 7fc11dfa8640 1 -- 172.21.15.90:0/2148506571 >> 172.21.15.90:0/2148506571 conn(0x7fc1180ff210 msgr2=0x7fc118101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:54.379 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.387+0000 7fc11dfa8640 1 -- 172.21.15.90:0/2148506571 shutdown_connections 2024-04-25T11:35:54.379 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.387+0000 7fc11dfa8640 1 -- 172.21.15.90:0/2148506571 wait complete. 2024-04-25T11:35:54.379 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.388+0000 7fc11dfa8640 1 Processor -- start 2024-04-25T11:35:54.379 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.388+0000 7fc11dfa8640 1 -- start start 2024-04-25T11:35:54.379 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.388+0000 7fc11dfa8640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1181042c0 0x7fc11819a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:54.379 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.388+0000 7fc11dfa8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc118109b70 0x7fc11819ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:54.379 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.388+0000 7fc11dfa8640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc11810a010 0x7fc11819f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:54.379 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.388+0000 7fc11dfa8640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc118113d60 con 0x7fc1181042c0 2024-04-25T11:35:54.379 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.388+0000 7fc11dfa8640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc118113be0 con 0x7fc11810a010 2024-04-25T11:35:54.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.388+0000 7fc11dfa8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc118113ee0 con 0x7fc118109b70 2024-04-25T11:35:54.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.388+0000 7fc1177fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1181042c0 0x7fc11819a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:54.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.388+0000 7fc1177fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1181042c0 0x7fc11819a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39748/0 (socket says 172.21.15.90:39748) 2024-04-25T11:35:54.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.388+0000 7fc1177fe640 1 -- 172.21.15.90:0/2759450142 learned_addr learned my addr 172.21.15.90:0/2759450142 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:54.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.389+0000 7fc116ffd640 1 --2- 172.21.15.90:0/2759450142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc118109b70 0x7fc11819ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:54.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.389+0000 7fc117fff640 1 --2- 172.21.15.90:0/2759450142 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc11810a010 0x7fc11819f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:54.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.389+0000 7fc1177fe640 1 -- 172.21.15.90:0/2759450142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc118109b70 msgr2=0x7fc11819ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:54.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.389+0000 7fc1177fe640 1 --2- 172.21.15.90:0/2759450142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc118109b70 0x7fc11819ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.389+0000 7fc1177fe640 1 -- 172.21.15.90:0/2759450142 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc11810a010 msgr2=0x7fc11819f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:54.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.389+0000 7fc1177fe640 1 --2- 172.21.15.90:0/2759450142 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc11810a010 0x7fc11819f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.389+0000 7fc1177fe640 1 -- 172.21.15.90:0/2759450142 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc11819f5c0 con 0x7fc1181042c0 2024-04-25T11:35:54.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.389+0000 7fc117fff640 1 --2- 172.21.15.90:0/2759450142 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc11810a010 0x7fc11819f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:35:54.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.389+0000 7fc1177fe640 1 --2- 172.21.15.90:0/2759450142 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1181042c0 0x7fc11819a7e0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc1080097b0 tx=0x7fc1080319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:54.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.389+0000 7fc114ff9640 1 -- 172.21.15.90:0/2759450142 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc108003bf0 con 0x7fc1181042c0 2024-04-25T11:35:54.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.389+0000 7fc114ff9640 1 -- 172.21.15.90:0/2759450142 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc108003d90 con 0x7fc1181042c0 2024-04-25T11:35:54.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.389+0000 7fc11dfa8640 1 -- 172.21.15.90:0/2759450142 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc11819f850 con 0x7fc1181042c0 2024-04-25T11:35:54.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.389+0000 7fc114ff9640 1 -- 172.21.15.90:0/2759450142 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc10802fc20 con 0x7fc1181042c0 2024-04-25T11:35:54.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.389+0000 7fc11dfa8640 1 -- 172.21.15.90:0/2759450142 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc11819fda0 con 0x7fc1181042c0 2024-04-25T11:35:54.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.392+0000 7fc114ff9640 1 -- 172.21.15.90:0/2759450142 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 23) ==== 147696+0+0 (secure 0 0 0) 0x7fc1080046a0 con 0x7fc1181042c0 2024-04-25T11:35:54.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.392+0000 7fc124e55740 1 Processor -- start 2024-04-25T11:35:54.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.392+0000 7fc124e55740 1 -- start start 2024-04-25T11:35:54.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.392+0000 7fc124e55740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165bc51b0 0x55a165f64af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:54.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.392+0000 7fc124e55740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a165b9c2f0 0x55a165f65030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:54.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.392+0000 7fc124e55740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a165ba44d0 0x55a165f65570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:54.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.392+0000 7fc124e55740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a165ecb460 con 0x55a165bc51b0 2024-04-25T11:35:54.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.392+0000 7fc124e55740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55a165e85590 con 0x55a165ba44d0 2024-04-25T11:35:54.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.392+0000 7fc124e55740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55a165ec7420 con 0x55a165b9c2f0 2024-04-25T11:35:54.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.392+0000 7fc117fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165bc51b0 0x55a165f64af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:54.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.392+0000 7fc117fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165bc51b0 0x55a165f64af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39754/0 (socket says 172.21.15.90:39754) 2024-04-25T11:35:54.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.392+0000 7fc117fff640 1 -- 172.21.15.90:0/4192944510 learned_addr learned my addr 172.21.15.90:0/4192944510 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:54.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.393+0000 7fc1177fe640 1 --2- 172.21.15.90:0/4192944510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a165b9c2f0 0x55a165f65030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:54.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.393+0000 7fc116ffd640 1 --2- 172.21.15.90:0/4192944510 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a165ba44d0 0x55a165f65570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:54.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.393+0000 7fc114ff9640 1 --2- 172.21.15.90:0/2759450142 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc1000afce0 0x7fc1000b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:54.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.393+0000 7fc117fff640 1 -- 172.21.15.90:0/4192944510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a165b9c2f0 msgr2=0x55a165f65030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:54.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.393+0000 7fc117fff640 1 --2- 172.21.15.90:0/4192944510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a165b9c2f0 0x55a165f65030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.393+0000 7fc117fff640 1 -- 172.21.15.90:0/4192944510 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a165ba44d0 msgr2=0x55a165f65570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:54.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.393+0000 7fc117fff640 1 --2- 172.21.15.90:0/4192944510 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a165ba44d0 0x55a165f65570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.393+0000 7fc117fff640 1 -- 172.21.15.90:0/4192944510 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a165e920d0 con 0x55a165bc51b0 2024-04-25T11:35:54.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.393+0000 7fc116ffd640 1 --2- 172.21.15.90:0/4192944510 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a165ba44d0 0x55a165f65570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:35:54.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.393+0000 7fc114ff9640 1 -- 172.21.15.90:0/2759450142 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7fc108100670 con 0x7fc1181042c0 2024-04-25T11:35:54.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.393+0000 7fc117fff640 1 --2- 172.21.15.90:0/4192944510 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165bc51b0 0x55a165f64af0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fc11000bb60 tx=0x7fc11000d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a3cfae86a062f55 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:54.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.393+0000 7fc117fff640 1 --2- 172.21.15.90:0/2759450142 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc1000afce0 0x7fc1000b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:54.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.393+0000 7fc0fdffb640 1 -- 172.21.15.90:0/4192944510 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc110017620 con 0x55a165bc51b0 2024-04-25T11:35:54.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.393+0000 7fc0fdffb640 1 -- 172.21.15.90:0/4192944510 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc110017e30 con 0x55a165bc51b0 2024-04-25T11:35:54.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.393+0000 7fc0fdffb640 1 -- 172.21.15.90:0/4192944510 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc110016630 con 0x55a165bc51b0 2024-04-25T11:35:54.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.394+0000 7fc124e55740 1 -- 172.21.15.90:0/4192944510 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165bc51b0 msgr2=0x55a165f64af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:54.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.394+0000 7fc117fff640 1 --2- 172.21.15.90:0/2759450142 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc1000afce0 0x7fc1000b21f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc110018000 tx=0x7fc110016da0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:54.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.394+0000 7fc124e55740 1 --2- 172.21.15.90:0/4192944510 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165bc51b0 0x55a165f64af0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fc11000bb60 tx=0x7fc11000d710 comp rx=0 tx=0).stop 2024-04-25T11:35:54.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.394+0000 7fc124e55740 1 -- 172.21.15.90:0/4192944510 shutdown_connections 2024-04-25T11:35:54.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.394+0000 7fc124e55740 1 --2- 172.21.15.90:0/4192944510 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a165ba44d0 0x55a165f65570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.394+0000 7fc124e55740 1 --2- 172.21.15.90:0/4192944510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a165b9c2f0 0x55a165f65030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.394+0000 7fc124e55740 1 --2- 172.21.15.90:0/4192944510 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165bc51b0 0x55a165f64af0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.394+0000 7fc124e55740 1 -- 172.21.15.90:0/4192944510 >> 172.21.15.90:0/4192944510 conn(0x55a165bb9300 msgr2=0x55a165f612a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:54.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.394+0000 7fc124e55740 1 -- 172.21.15.90:0/4192944510 shutdown_connections 2024-04-25T11:35:54.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.394+0000 7fc124e55740 1 -- 172.21.15.90:0/4192944510 wait complete. 2024-04-25T11:35:54.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.395+0000 7fc124e55740 1 -- start start 2024-04-25T11:35:54.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.395+0000 7fc124e55740 1 Processor -- start 2024-04-25T11:35:54.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.395+0000 7fc124e55740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165b9c2f0 0x55a165f6f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:54.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.395+0000 7fc124e55740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a165ba44d0 0x55a165f6f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:54.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.395+0000 7fc124e55740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a165bc51b0 0x55a165f750a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:54.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.395+0000 7fc124e55740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a165f71c20 con 0x55a165b9c2f0 2024-04-25T11:35:54.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.395+0000 7fc124e55740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55a165e52b60 con 0x55a165bc51b0 2024-04-25T11:35:54.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.395+0000 7fc124e55740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55a165f71da0 con 0x55a165ba44d0 2024-04-25T11:35:54.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.395+0000 7fc124e55740 10 client.0 _ll_register_callbacks cb 0x55a165e772b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:35:54.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.395+0000 7fc1177fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165b9c2f0 0x55a165f6f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:54.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.395+0000 7fc1177fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165b9c2f0 0x55a165f6f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39770/0 (socket says 172.21.15.90:39770) 2024-04-25T11:35:54.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.395+0000 7fc1177fe640 1 -- 172.21.15.90:0/2485990966 learned_addr learned my addr 172.21.15.90:0/2485990966 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:54.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.395+0000 7fc117fff640 1 --2- 172.21.15.90:0/2485990966 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a165bc51b0 0x55a165f750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:54.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.395+0000 7fc116ffd640 1 --2- 172.21.15.90:0/2485990966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a165ba44d0 0x55a165f6f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:54.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.396+0000 7fc1177fe640 1 -- 172.21.15.90:0/2485990966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a165ba44d0 msgr2=0x55a165f6f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:54.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.396+0000 7fc1177fe640 1 --2- 172.21.15.90:0/2485990966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a165ba44d0 0x55a165f6f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.396+0000 7fc1177fe640 1 -- 172.21.15.90:0/2485990966 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a165bc51b0 msgr2=0x55a165f750a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:54.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.396+0000 7fc1177fe640 1 --2- 172.21.15.90:0/2485990966 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a165bc51b0 0x55a165f750a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.396+0000 7fc1177fe640 1 -- 172.21.15.90:0/2485990966 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a165bdd210 con 0x55a165b9c2f0 2024-04-25T11:35:54.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.396+0000 7fc1177fe640 1 --2- 172.21.15.90:0/2485990966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165b9c2f0 0x55a165f6f140 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc108031a90 tx=0x7fc1080b12d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f277cc01e409c96d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:54.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.396+0000 7fc0f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:35:54.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.396+0000 7fc0f6ffd640 1 -- 172.21.15.90:0/2485990966 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1080c8580 con 0x55a165b9c2f0 2024-04-25T11:35:54.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.396+0000 7fc0f6ffd640 1 -- 172.21.15.90:0/2485990966 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7fc1080c8c10 con 0x55a165b9c2f0 2024-04-25T11:35:54.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.396+0000 7fc124e55740 1 -- 172.21.15.90:0/2485990966 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc110017e30 con 0x55a165b9c2f0 2024-04-25T11:35:54.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.396+0000 7fc0f6ffd640 1 -- 172.21.15.90:0/2485990966 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc1080c7f20 con 0x55a165b9c2f0 2024-04-25T11:35:54.393 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.396+0000 7fc124e55740 10 client.15792 fetch_fsmap learned FSMap version 26 2024-04-25T11:35:54.393 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.396+0000 7fc124e55740 1 -- 172.21.15.90:0/2485990966 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a165ca9620 con 0x55a165b9c2f0 2024-04-25T11:35:54.393 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.396+0000 7fc0f6ffd640 1 -- 172.21.15.90:0/2485990966 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc110017e30 con 0x55a165b9c2f0 2024-04-25T11:35:54.393 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.396+0000 7fc0f6ffd640 1 -- 172.21.15.90:0/2485990966 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fc1080d1180 con 0x55a165b9c2f0 2024-04-25T11:35:54.393 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.396+0000 7fc124e55740 10 client.15792 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:35:54.393 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.396+0000 7fc124e55740 10 client.15792 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:35:54.393 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.396+0000 7fc124e55740 20 client.15792 populate_metadata read hostname 'smithi090' 2024-04-25T11:35:54.393 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.396+0000 7fc124e55740 1 --2- 172.21.15.90:0/2485990966 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55a165b83d80 0x55a165f81660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:54.393 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.396+0000 7fc124e55740 4 client.15792 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:35:54.393 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.396+0000 7fc124e55740 1 -- 172.21.15.90:0/2485990966 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a165ed9070 con 0x55a165b83d80 2024-04-25T11:35:54.393 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.397+0000 7fc116ffd640 1 --2- 172.21.15.90:0/2485990966 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55a165b83d80 0x55a165f81660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:54.393 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.397+0000 7fc0f6ffd640 10 client.15792 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:54.393 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.397+0000 7fc116ffd640 1 --2- 172.21.15.90:0/2485990966 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55a165b83d80 0x55a165f81660 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5cbf9080d686fb1 server_cookie=71c6f3f9325caded in_seq=0 out_seq=0 2024-04-25T11:35:54.393 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.398+0000 7fc0f6ffd640 1 -- 172.21.15.90:0/2485990966 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a165ed9070 con 0x55a165b83d80 2024-04-25T11:35:54.394 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.398+0000 7fc0f6ffd640 10 client.15792 handle_command_reply: tid=0 2024-04-25T11:35:54.394 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.398+0000 7fc124e55740 1 client.15792 shutdown 2024-04-25T11:35:54.394 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.398+0000 7fc124e55740 1 -- 172.21.15.90:0/2485990966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165b9c2f0 msgr2=0x55a165f6f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:54.394 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.398+0000 7fc124e55740 1 --2- 172.21.15.90:0/2485990966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165b9c2f0 0x55a165f6f140 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc108031a90 tx=0x7fc1080b12d0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.394 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.398+0000 7fc124e55740 1 -- 172.21.15.90:0/2485990966 shutdown_connections 2024-04-25T11:35:54.394 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.398+0000 7fc124e55740 1 --2- 172.21.15.90:0/2485990966 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55a165b83d80 0x55a165f81660 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.394 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.398+0000 7fc124e55740 1 --2- 172.21.15.90:0/2485990966 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a165bc51b0 0x55a165f750a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.394 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.398+0000 7fc0f6ffd640 0 client.15792 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:54.394 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.398+0000 7fc124e55740 1 --2- 172.21.15.90:0/2485990966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a165ba44d0 0x55a165f6f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.394 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.398+0000 7fc124e55740 1 --2- 172.21.15.90:0/2485990966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165b9c2f0 0x55a165f6f140 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.394 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.398+0000 7fc124e55740 1 -- 172.21.15.90:0/2485990966 >> 172.21.15.90:0/2485990966 conn(0x55a165bb9300 msgr2=0x55a165f64920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:54.394 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.398+0000 7fc124e55740 1 -- 172.21.15.90:0/2485990966 shutdown_connections 2024-04-25T11:35:54.394 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.398+0000 7fc124e55740 1 -- 172.21.15.90:0/2485990966 wait complete. 2024-04-25T11:35:54.395 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.398+0000 7fc124e55740 20 client.15792 trim_cache size 0 max 16384 2024-04-25T11:35:54.403 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.413+0000 7fc124e55740 1 Processor -- start 2024-04-25T11:35:54.403 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.413+0000 7fc124e55740 1 -- start start 2024-04-25T11:35:54.403 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.414+0000 7fc124e55740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a165b83d80 0x55a165f6e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:54.403 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.414+0000 7fc124e55740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a165b9c2f0 0x55a165f70d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:54.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.414+0000 7fc124e55740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165ba44d0 0x55a165f712a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:54.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.414+0000 7fc124e55740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc1181afab0 con 0x55a165ba44d0 2024-04-25T11:35:54.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.414+0000 7fc124e55740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc1181b15f0 con 0x55a165b9c2f0 2024-04-25T11:35:54.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.414+0000 7fc124e55740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc1181b0180 con 0x55a165b83d80 2024-04-25T11:35:54.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.414+0000 7fc117fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165ba44d0 0x55a165f712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:54.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.414+0000 7fc117fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165ba44d0 0x55a165f712a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39776/0 (socket says 172.21.15.90:39776) 2024-04-25T11:35:54.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.414+0000 7fc117fff640 1 -- 172.21.15.90:0/3374890655 learned_addr learned my addr 172.21.15.90:0/3374890655 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:54.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.414+0000 7fc116ffd640 1 --2- 172.21.15.90:0/3374890655 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a165b9c2f0 0x55a165f70d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:54.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.414+0000 7fc1177fe640 1 --2- 172.21.15.90:0/3374890655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a165b83d80 0x55a165f6e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:54.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.414+0000 7fc117fff640 1 -- 172.21.15.90:0/3374890655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a165b83d80 msgr2=0x55a165f6e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:54.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.414+0000 7fc117fff640 1 --2- 172.21.15.90:0/3374890655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a165b83d80 0x55a165f6e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.414+0000 7fc117fff640 1 -- 172.21.15.90:0/3374890655 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a165b9c2f0 msgr2=0x55a165f70d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:54.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.414+0000 7fc117fff640 1 --2- 172.21.15.90:0/3374890655 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a165b9c2f0 0x55a165f70d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.414+0000 7fc117fff640 1 -- 172.21.15.90:0/3374890655 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a165f65490 con 0x55a165ba44d0 2024-04-25T11:35:54.405 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.414+0000 7fc117fff640 1 --2- 172.21.15.90:0/3374890655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165ba44d0 0x55a165f712a0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc11000ba80 tx=0x7fc110025000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2077755a83cf933 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:54.405 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.414+0000 7fc0fd7fa640 1 -- 172.21.15.90:0/3374890655 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc110017e30 con 0x55a165ba44d0 2024-04-25T11:35:54.405 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.414+0000 7fc0fd7fa640 1 -- 172.21.15.90:0/3374890655 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc110028040 con 0x55a165ba44d0 2024-04-25T11:35:54.405 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.414+0000 7fc0fd7fa640 1 -- 172.21.15.90:0/3374890655 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11000e950 con 0x55a165ba44d0 2024-04-25T11:35:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.415+0000 7fc124e55740 1 -- 172.21.15.90:0/3374890655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165ba44d0 msgr2=0x55a165f712a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.415+0000 7fc124e55740 1 --2- 172.21.15.90:0/3374890655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165ba44d0 0x55a165f712a0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc11000ba80 tx=0x7fc110025000 comp rx=0 tx=0).stop 2024-04-25T11:35:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.415+0000 7fc124e55740 1 -- 172.21.15.90:0/3374890655 shutdown_connections 2024-04-25T11:35:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.415+0000 7fc124e55740 1 --2- 172.21.15.90:0/3374890655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165ba44d0 0x55a165f712a0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.415+0000 7fc124e55740 1 --2- 172.21.15.90:0/3374890655 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a165b9c2f0 0x55a165f70d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.415+0000 7fc124e55740 1 --2- 172.21.15.90:0/3374890655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a165b83d80 0x55a165f6e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.415+0000 7fc124e55740 1 -- 172.21.15.90:0/3374890655 >> 172.21.15.90:0/3374890655 conn(0x55a165bb9300 msgr2=0x55a165f70650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.415+0000 7fc124e55740 1 -- 172.21.15.90:0/3374890655 shutdown_connections 2024-04-25T11:35:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.415+0000 7fc124e55740 1 -- 172.21.15.90:0/3374890655 wait complete. 2024-04-25T11:35:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.416+0000 7fc124e55740 1 -- start start 2024-04-25T11:35:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.416+0000 7fc124e55740 1 Processor -- start 2024-04-25T11:35:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.416+0000 7fc124e55740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a165b83d80 0x55a165f7b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.416+0000 7fc124e55740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165b9c2f0 0x55a165f7be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.416+0000 7fc124e55740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a165ba44d0 0x55a165f78d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.416+0000 7fc124e55740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc1181b0f20 con 0x55a165b9c2f0 2024-04-25T11:35:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.416+0000 7fc124e55740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc1181b0850 con 0x55a165ba44d0 2024-04-25T11:35:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.416+0000 7fc124e55740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc1181ada90 con 0x55a165b83d80 2024-04-25T11:35:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.416+0000 7fc124e55740 10 client.0 _ll_register_callbacks cb 0x7fc1080c4ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:35:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.416+0000 7fc116ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165b9c2f0 0x55a165f7be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.416+0000 7fc116ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165b9c2f0 0x55a165f7be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39790/0 (socket says 172.21.15.90:39790) 2024-04-25T11:35:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.416+0000 7fc116ffd640 1 -- 172.21.15.90:0/3172833560 learned_addr learned my addr 172.21.15.90:0/3172833560 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:35:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.416+0000 7fc1177fe640 1 --2- 172.21.15.90:0/3172833560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a165b83d80 0x55a165f7b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.416+0000 7fc117fff640 1 --2- 172.21.15.90:0/3172833560 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a165ba44d0 0x55a165f78d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.416+0000 7fc116ffd640 1 -- 172.21.15.90:0/3172833560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a165b83d80 msgr2=0x55a165f7b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.416+0000 7fc116ffd640 1 --2- 172.21.15.90:0/3172833560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a165b83d80 0x55a165f7b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.416+0000 7fc116ffd640 1 -- 172.21.15.90:0/3172833560 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a165ba44d0 msgr2=0x55a165f78d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.416+0000 7fc116ffd640 1 --2- 172.21.15.90:0/3172833560 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a165ba44d0 0x55a165f78d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.417+0000 7fc116ffd640 1 -- 172.21.15.90:0/3172833560 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a165f79410 con 0x55a165b9c2f0 2024-04-25T11:35:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.417+0000 7fc1177fe640 1 --2- 172.21.15.90:0/3172833560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a165b83d80 0x55a165f7b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:35:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.417+0000 7fc117fff640 1 --2- 172.21.15.90:0/3172833560 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a165ba44d0 0x55a165f78d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:35:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.417+0000 7fc116ffd640 1 --2- 172.21.15.90:0/3172833560 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165b9c2f0 0x55a165f7be00 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc118100fa0 tx=0x7fc10c00bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=270cda465d699596 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.417+0000 7fc0f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:35:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.417+0000 7fc0f6ffd640 1 -- 172.21.15.90:0/3172833560 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc10c01b760 con 0x55a165b9c2f0 2024-04-25T11:35:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.417+0000 7fc0f6ffd640 1 -- 172.21.15.90:0/3172833560 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7fc10c01b900 con 0x55a165b9c2f0 2024-04-25T11:35:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.417+0000 7fc124e55740 1 -- 172.21.15.90:0/3172833560 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc110028040 con 0x55a165b9c2f0 2024-04-25T11:35:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.417+0000 7fc0f6ffd640 1 -- 172.21.15.90:0/3172833560 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc10c024470 con 0x55a165b9c2f0 2024-04-25T11:35:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.417+0000 7fc124e55740 10 client.15798 fetch_fsmap learned FSMap version 26 2024-04-25T11:35:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.417+0000 7fc124e55740 1 -- 172.21.15.90:0/3172833560 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a165fe93d0 con 0x55a165b9c2f0 2024-04-25T11:35:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.417+0000 7fc0f6ffd640 1 -- 172.21.15.90:0/3172833560 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc110028040 con 0x55a165b9c2f0 2024-04-25T11:35:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.417+0000 7fc0f6ffd640 1 -- 172.21.15.90:0/3172833560 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fc10c0115d0 con 0x55a165b9c2f0 2024-04-25T11:35:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.417+0000 7fc124e55740 10 client.15798 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:35:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.417+0000 7fc124e55740 10 client.15798 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:35:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.417+0000 7fc124e55740 20 client.15798 populate_metadata read hostname 'smithi090' 2024-04-25T11:35:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.417+0000 7fc124e55740 1 --2- 172.21.15.90:0/3172833560 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55a165bc51b0 0x55a165febbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.417+0000 7fc124e55740 4 client.15798 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:35:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.417+0000 7fc124e55740 1 -- 172.21.15.90:0/3172833560 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a165ed2ee0 con 0x55a165bc51b0 2024-04-25T11:35:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.418+0000 7fc1177fe640 1 --2- 172.21.15.90:0/3172833560 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55a165bc51b0 0x55a165febbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.418+0000 7fc0f6ffd640 10 client.15798 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.418+0000 7fc1177fe640 1 --2- 172.21.15.90:0/3172833560 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55a165bc51b0 0x55a165febbc0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65334410fca80ff7 server_cookie=ec52c53aaaddb6f4 in_seq=0 out_seq=0 2024-04-25T11:35:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.426+0000 7fc0f6ffd640 1 -- 172.21.15.90:0/3172833560 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a165ed2ee0 con 0x55a165bc51b0 2024-04-25T11:35:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.426+0000 7fc0f6ffd640 10 client.15798 handle_command_reply: tid=0 2024-04-25T11:35:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.426+0000 7fc124e55740 1 client.15798 shutdown 2024-04-25T11:35:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.426+0000 7fc124e55740 1 -- 172.21.15.90:0/3172833560 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165b9c2f0 msgr2=0x55a165f7be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.426+0000 7fc124e55740 1 --2- 172.21.15.90:0/3172833560 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165b9c2f0 0x55a165f7be00 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc118100fa0 tx=0x7fc10c00bf90 comp rx=0 tx=0).stop 2024-04-25T11:35:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.426+0000 7fc124e55740 1 -- 172.21.15.90:0/3172833560 shutdown_connections 2024-04-25T11:35:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.426+0000 7fc124e55740 1 --2- 172.21.15.90:0/3172833560 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55a165bc51b0 0x55a165febbc0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.426+0000 7fc124e55740 1 --2- 172.21.15.90:0/3172833560 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a165ba44d0 0x55a165f78d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.426+0000 7fc0f6ffd640 0 client.15798 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:35:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.426+0000 7fc124e55740 1 --2- 172.21.15.90:0/3172833560 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a165b9c2f0 0x55a165f7be00 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.426+0000 7fc124e55740 1 --2- 172.21.15.90:0/3172833560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a165b83d80 0x55a165f7b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.426+0000 7fc124e55740 1 -- 172.21.15.90:0/3172833560 >> 172.21.15.90:0/3172833560 conn(0x55a165bb9300 msgr2=0x55a165f6df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.426+0000 7fc124e55740 1 -- 172.21.15.90:0/3172833560 shutdown_connections 2024-04-25T11:35:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.426+0000 7fc124e55740 1 -- 172.21.15.90:0/3172833560 wait complete. 2024-04-25T11:35:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.426+0000 7fc124e55740 20 client.15798 trim_cache size 0 max 16384 2024-04-25T11:35:54.417 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:35:54.417 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:35:54.417 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:35:54.417 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:35:54.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.427+0000 7fc11dfa8640 1 -- 172.21.15.90:0/2759450142 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc1000afce0 msgr2=0x7fc1000b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:54.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.427+0000 7fc11dfa8640 1 --2- 172.21.15.90:0/2759450142 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc1000afce0 0x7fc1000b21f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc110018000 tx=0x7fc110016da0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.427+0000 7fc11dfa8640 1 -- 172.21.15.90:0/2759450142 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1181042c0 msgr2=0x7fc11819a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:54.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.427+0000 7fc11dfa8640 1 --2- 172.21.15.90:0/2759450142 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1181042c0 0x7fc11819a7e0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc1080097b0 tx=0x7fc1080319f0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.427+0000 7fc11dfa8640 1 -- 172.21.15.90:0/2759450142 shutdown_connections 2024-04-25T11:35:54.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.427+0000 7fc11dfa8640 1 --2- 172.21.15.90:0/2759450142 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc1000afce0 0x7fc1000b21f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.427+0000 7fc11dfa8640 1 --2- 172.21.15.90:0/2759450142 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc11810a010 0x7fc11819f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.427+0000 7fc11dfa8640 1 --2- 172.21.15.90:0/2759450142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc118109b70 0x7fc11819ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.427+0000 7fc11dfa8640 1 --2- 172.21.15.90:0/2759450142 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1181042c0 0x7fc11819a7e0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:54.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.427+0000 7fc11dfa8640 1 -- 172.21.15.90:0/2759450142 >> 172.21.15.90:0/2759450142 conn(0x7fc1180ff210 msgr2=0x7fc1180ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:54.419 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.427+0000 7fc11dfa8640 1 -- 172.21.15.90:0/2759450142 shutdown_connections 2024-04-25T11:35:54.419 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:35:54.427+0000 7fc11dfa8640 1 -- 172.21.15.90:0/2759450142 wait complete. 2024-04-25T11:35:55.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:55 smithi090 ceph-mon[29575]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-04-25T11:35:56.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:55 smithi138 ceph-mon[32328]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-04-25T11:35:56.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:55 smithi156 ceph-mon[33045]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-04-25T11:35:57.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:57 smithi090 ceph-mon[29575]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-04-25T11:35:58.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:57 smithi138 ceph-mon[32328]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-04-25T11:35:58.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:57 smithi156 ceph-mon[33045]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-04-25T11:35:59.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:35:59 smithi090 ceph-mon[29575]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-25T11:36:00.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:35:59 smithi138 ceph-mon[32328]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-25T11:36:00.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:35:59 smithi156 ceph-mon[33045]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-25T11:36:00.475 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:36:00 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:36:00.192+0000 7f434a087700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T11:36:00.475 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:36:00 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:36:00.192+0000 7f434a087700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T11:36:00.475 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:36:00 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:36:00.192+0000 7f434a087700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T11:36:00.475 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:36:00 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:36:00.192+0000 7f434a087700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T11:36:00.475 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:36:00 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:36:00.192+0000 7f434a087700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T11:36:01.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:01 smithi090 ceph-mon[29575]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.7 KiB/s wr, 15 op/s 2024-04-25T11:36:01.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:01 smithi090 ceph-mon[29575]: mgrmap e24: x(active, since 52s), standbys: z, y 2024-04-25T11:36:02.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:01 smithi138 ceph-mon[32328]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.7 KiB/s wr, 15 op/s 2024-04-25T11:36:02.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:01 smithi138 ceph-mon[32328]: mgrmap e24: x(active, since 52s), standbys: z, y 2024-04-25T11:36:02.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:01 smithi156 ceph-mon[33045]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.7 KiB/s wr, 15 op/s 2024-04-25T11:36:02.088 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:01 smithi156 ceph-mon[33045]: mgrmap e24: x(active, since 52s), standbys: z, y 2024-04-25T11:36:03.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:03 smithi090 ceph-mon[29575]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 767 B/s wr, 10 op/s 2024-04-25T11:36:04.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:03 smithi138 ceph-mon[32328]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 767 B/s wr, 10 op/s 2024-04-25T11:36:04.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:03 smithi156 ceph-mon[33045]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 767 B/s wr, 10 op/s 2024-04-25T11:36:05.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:05 smithi090 ceph-mon[29575]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-04-25T11:36:06.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:05 smithi138 ceph-mon[32328]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-04-25T11:36:06.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:05 smithi156 ceph-mon[33045]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-04-25T11:36:07.516 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.525+0000 7fd533814640 1 -- 172.21.15.90:0/4274082893 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd52c108f00 msgr2=0x7fd52c102b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.525+0000 7fd533814640 1 --2- 172.21.15.90:0/4274082893 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd52c108f00 0x7fd52c102b50 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fd51c009900 tx=0x7fd51c02a410 comp rx=0 tx=0).stop 2024-04-25T11:36:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.526+0000 7fd533814640 1 -- 172.21.15.90:0/4274082893 shutdown_connections 2024-04-25T11:36:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.526+0000 7fd533814640 1 --2- 172.21.15.90:0/4274082893 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd52c1031c0 0x7fd52c10f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.526+0000 7fd533814640 1 --2- 172.21.15.90:0/4274082893 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd52c108f00 0x7fd52c102b50 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.526+0000 7fd533814640 1 --2- 172.21.15.90:0/4274082893 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd52c108a60 0x7fd52c102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.526+0000 7fd533814640 1 -- 172.21.15.90:0/4274082893 >> 172.21.15.90:0/4274082893 conn(0x7fd52c0781b0 msgr2=0x7fd52c0785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.526+0000 7fd533814640 1 -- 172.21.15.90:0/4274082893 shutdown_connections 2024-04-25T11:36:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.526+0000 7fd533814640 1 -- 172.21.15.90:0/4274082893 wait complete. 2024-04-25T11:36:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.526+0000 7fd533814640 1 Processor -- start 2024-04-25T11:36:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.526+0000 7fd533814640 1 -- start start 2024-04-25T11:36:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.527+0000 7fd533814640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd52c1031c0 0x7fd52c1a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.527+0000 7fd533814640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd52c108a60 0x7fd52c1a0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.527+0000 7fd533814640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd52c108f00 0x7fd52c19a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.527+0000 7fd533814640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c112c80 con 0x7fd52c108a60 2024-04-25T11:36:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.527+0000 7fd533814640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c112b00 con 0x7fd52c108f00 2024-04-25T11:36:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.527+0000 7fd533814640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c112e00 con 0x7fd52c1031c0 2024-04-25T11:36:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.527+0000 7fd530daf640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd52c108a60 0x7fd52c1a0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.527+0000 7fd530daf640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd52c108a60 0x7fd52c1a0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55726/0 (socket says 172.21.15.90:55726) 2024-04-25T11:36:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.527+0000 7fd530daf640 1 -- 172.21.15.90:0/400996967 learned_addr learned my addr 172.21.15.90:0/400996967 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.527+0000 7fd5315b0640 1 --2- 172.21.15.90:0/400996967 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd52c1031c0 0x7fd52c1a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.527+0000 7fd531db1640 1 --2- 172.21.15.90:0/400996967 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd52c108f00 0x7fd52c19a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.527+0000 7fd530daf640 1 -- 172.21.15.90:0/400996967 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd52c1031c0 msgr2=0x7fd52c1a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.527+0000 7fd530daf640 1 --2- 172.21.15.90:0/400996967 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd52c1031c0 0x7fd52c1a04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.527+0000 7fd530daf640 1 -- 172.21.15.90:0/400996967 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd52c108f00 msgr2=0x7fd52c19a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.527+0000 7fd530daf640 1 --2- 172.21.15.90:0/400996967 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd52c108f00 0x7fd52c19a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.527+0000 7fd530daf640 1 -- 172.21.15.90:0/400996967 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd52c19ad80 con 0x7fd52c108a60 2024-04-25T11:36:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.527+0000 7fd531db1640 1 --2- 172.21.15.90:0/400996967 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd52c108f00 0x7fd52c19a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:36:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.527+0000 7fd530daf640 1 --2- 172.21.15.90:0/400996967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd52c108a60 0x7fd52c1a0a10 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fd51c02a930 tx=0x7fd51c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.527+0000 7fd5227fc640 1 -- 172.21.15.90:0/400996967 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51c03f030 con 0x7fd52c108a60 2024-04-25T11:36:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.527+0000 7fd533814640 1 -- 172.21.15.90:0/400996967 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd52c19afb0 con 0x7fd52c108a60 2024-04-25T11:36:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.527+0000 7fd5227fc640 1 -- 172.21.15.90:0/400996967 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd51c003aa0 con 0x7fd52c108a60 2024-04-25T11:36:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.528+0000 7fd5227fc640 1 -- 172.21.15.90:0/400996967 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51c035070 con 0x7fd52c108a60 2024-04-25T11:36:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.528+0000 7fd533814640 1 -- 172.21.15.90:0/400996967 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd52c19b4a0 con 0x7fd52c108a60 2024-04-25T11:36:07.527 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.530+0000 7fd5227fc640 1 -- 172.21.15.90:0/400996967 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 24) ==== 147753+0+0 (secure 0 0 0) 0x7fd51c042030 con 0x7fd52c108a60 2024-04-25T11:36:07.527 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.530+0000 7fd5227fc640 1 --2- 172.21.15.90:0/400996967 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd5180afd70 0x7fd5180b2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:07.527 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.531+0000 7fd5315b0640 1 --2- 172.21.15.90:0/400996967 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd5180afd70 0x7fd5180b2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:07.527 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.531+0000 7fd5227fc640 1 -- 172.21.15.90:0/400996967 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7fd51c0ff5b0 con 0x7fd52c108a60 2024-04-25T11:36:07.527 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.531+0000 7fd53a79c740 1 Processor -- start 2024-04-25T11:36:07.527 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.531+0000 7fd5315b0640 1 --2- 172.21.15.90:0/400996967 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd5180afd70 0x7fd5180b2280 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd528005fd0 tx=0x7fd52800b390 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:07.527 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.531+0000 7fd53a79c740 1 -- start start 2024-04-25T11:36:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.532+0000 7fd53a79c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a4e1b0 0x55c8a4dedaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.532+0000 7fd53a79c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8a4a252f0 0x55c8a4dee030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.532+0000 7fd53a79c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8a4a2d4d0 0x55c8a4dee570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.532+0000 7fd53a79c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c8a4d54460 con 0x55c8a4a4e1b0 2024-04-25T11:36:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.532+0000 7fd53a79c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c8a4d0e590 con 0x55c8a4a252f0 2024-04-25T11:36:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.532+0000 7fd53a79c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c8a4d50420 con 0x55c8a4a2d4d0 2024-04-25T11:36:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.532+0000 7fd5315b0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a4e1b0 0x55c8a4dedaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.532+0000 7fd5315b0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a4e1b0 0x55c8a4dedaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55734/0 (socket says 172.21.15.90:55734) 2024-04-25T11:36:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.532+0000 7fd5315b0640 1 -- 172.21.15.90:0/1194872907 learned_addr learned my addr 172.21.15.90:0/1194872907 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.532+0000 7fd531db1640 1 --2- 172.21.15.90:0/1194872907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8a4a2d4d0 0x55c8a4dee570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.532+0000 7fd530daf640 1 --2- 172.21.15.90:0/1194872907 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8a4a252f0 0x55c8a4dee030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.532+0000 7fd5315b0640 1 -- 172.21.15.90:0/1194872907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8a4a2d4d0 msgr2=0x55c8a4dee570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.532+0000 7fd5315b0640 1 --2- 172.21.15.90:0/1194872907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8a4a2d4d0 0x55c8a4dee570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.532+0000 7fd5315b0640 1 -- 172.21.15.90:0/1194872907 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8a4a252f0 msgr2=0x55c8a4dee030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.532+0000 7fd5315b0640 1 --2- 172.21.15.90:0/1194872907 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8a4a252f0 0x55c8a4dee030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.532+0000 7fd5315b0640 1 -- 172.21.15.90:0/1194872907 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8a4d1b0d0 con 0x55c8a4a4e1b0 2024-04-25T11:36:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.532+0000 7fd530daf640 1 --2- 172.21.15.90:0/1194872907 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8a4a252f0 0x55c8a4dee030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:36:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.532+0000 7fd5315b0640 1 --2- 172.21.15.90:0/1194872907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a4e1b0 0x55c8a4dedaf0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fd528003bc0 tx=0x7fd528013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a905c09ccd47843 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.532+0000 7fd5177fe640 1 -- 172.21.15.90:0/1194872907 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd528011d20 con 0x55c8a4a4e1b0 2024-04-25T11:36:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.532+0000 7fd5177fe640 1 -- 172.21.15.90:0/1194872907 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5280054d0 con 0x55c8a4a4e1b0 2024-04-25T11:36:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.532+0000 7fd5177fe640 1 -- 172.21.15.90:0/1194872907 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52801e680 con 0x55c8a4a4e1b0 2024-04-25T11:36:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.533+0000 7fd53a79c740 1 -- 172.21.15.90:0/1194872907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a4e1b0 msgr2=0x55c8a4dedaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.533+0000 7fd53a79c740 1 --2- 172.21.15.90:0/1194872907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a4e1b0 0x55c8a4dedaf0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fd528003bc0 tx=0x7fd528013000 comp rx=0 tx=0).stop 2024-04-25T11:36:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.533+0000 7fd53a79c740 1 -- 172.21.15.90:0/1194872907 shutdown_connections 2024-04-25T11:36:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.533+0000 7fd53a79c740 1 --2- 172.21.15.90:0/1194872907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8a4a2d4d0 0x55c8a4dee570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.533+0000 7fd53a79c740 1 --2- 172.21.15.90:0/1194872907 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8a4a252f0 0x55c8a4dee030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.533+0000 7fd53a79c740 1 --2- 172.21.15.90:0/1194872907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a4e1b0 0x55c8a4dedaf0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.533+0000 7fd53a79c740 1 -- 172.21.15.90:0/1194872907 >> 172.21.15.90:0/1194872907 conn(0x55c8a4a42300 msgr2=0x55c8a4dea2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.533+0000 7fd53a79c740 1 -- 172.21.15.90:0/1194872907 shutdown_connections 2024-04-25T11:36:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.533+0000 7fd53a79c740 1 -- 172.21.15.90:0/1194872907 wait complete. 2024-04-25T11:36:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.534+0000 7fd53a79c740 1 -- start start 2024-04-25T11:36:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.534+0000 7fd53a79c740 1 Processor -- start 2024-04-25T11:36:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.534+0000 7fd53a79c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a252f0 0x55c8a4df9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.534+0000 7fd53a79c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8a4a2d4d0 0x55c8a4dfa040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.534+0000 7fd53a79c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8a4a4e1b0 0x55c8a4df7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.534+0000 7fd53a79c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c8a4dfac20 con 0x55c8a4a252f0 2024-04-25T11:36:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.534+0000 7fd53a79c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c8a4cdbb60 con 0x55c8a4a2d4d0 2024-04-25T11:36:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.534+0000 7fd53a79c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c8a4dfada0 con 0x55c8a4a4e1b0 2024-04-25T11:36:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.534+0000 7fd53a79c740 10 client.0 _ll_register_callbacks cb 0x55c8a4d002b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.534+0000 7fd5315b0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a252f0 0x55c8a4df9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.534+0000 7fd5315b0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a252f0 0x55c8a4df9b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55748/0 (socket says 172.21.15.90:55748) 2024-04-25T11:36:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.534+0000 7fd5315b0640 1 -- 172.21.15.90:0/2882393442 learned_addr learned my addr 172.21.15.90:0/2882393442 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.534+0000 7fd531db1640 1 --2- 172.21.15.90:0/2882393442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8a4a4e1b0 0x55c8a4df7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.535+0000 7fd530daf640 1 --2- 172.21.15.90:0/2882393442 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8a4a2d4d0 0x55c8a4dfa040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.535+0000 7fd5315b0640 1 -- 172.21.15.90:0/2882393442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8a4a4e1b0 msgr2=0x55c8a4df7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.535+0000 7fd5315b0640 1 --2- 172.21.15.90:0/2882393442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8a4a4e1b0 0x55c8a4df7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.535+0000 7fd5315b0640 1 -- 172.21.15.90:0/2882393442 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8a4a2d4d0 msgr2=0x55c8a4dfa040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.535+0000 7fd5315b0640 1 --2- 172.21.15.90:0/2882393442 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8a4a2d4d0 0x55c8a4dfa040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.535+0000 7fd5315b0640 1 -- 172.21.15.90:0/2882393442 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8a4a66210 con 0x55c8a4a252f0 2024-04-25T11:36:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.535+0000 7fd5315b0640 1 --2- 172.21.15.90:0/2882393442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a252f0 0x55c8a4df9b00 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fd528005d50 tx=0x7fd528011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e107e95a18cc072 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.535+0000 7fd514ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.535+0000 7fd514ff9640 1 -- 172.21.15.90:0/2882393442 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd528036060 con 0x55c8a4a252f0 2024-04-25T11:36:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.535+0000 7fd514ff9640 1 -- 172.21.15.90:0/2882393442 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7fd528004d80 con 0x55c8a4a252f0 2024-04-25T11:36:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.535+0000 7fd53a79c740 1 -- 172.21.15.90:0/2882393442 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5280054d0 con 0x55c8a4a252f0 2024-04-25T11:36:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.535+0000 7fd514ff9640 1 -- 172.21.15.90:0/2882393442 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd528004f80 con 0x55c8a4a252f0 2024-04-25T11:36:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.535+0000 7fd53a79c740 10 client.15816 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.535+0000 7fd53a79c740 1 -- 172.21.15.90:0/2882393442 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c8a4b32620 con 0x55c8a4a252f0 2024-04-25T11:36:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.536+0000 7fd514ff9640 1 -- 172.21.15.90:0/2882393442 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5280054d0 con 0x55c8a4a252f0 2024-04-25T11:36:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.536+0000 7fd514ff9640 1 -- 172.21.15.90:0/2882393442 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fd52801eca0 con 0x55c8a4a252f0 2024-04-25T11:36:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.536+0000 7fd53a79c740 10 client.15816 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.536+0000 7fd53a79c740 10 client.15816 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:36:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.536+0000 7fd53a79c740 20 client.15816 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.536+0000 7fd53a79c740 1 --2- 172.21.15.90:0/2882393442 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c8a4a0cd80 0x55c8a4e0a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.536+0000 7fd53a79c740 4 client.15816 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:36:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.536+0000 7fd53a79c740 1 -- 172.21.15.90:0/2882393442 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8a4d62070 con 0x55c8a4a0cd80 2024-04-25T11:36:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.536+0000 7fd530daf640 1 --2- 172.21.15.90:0/2882393442 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c8a4a0cd80 0x55c8a4e0a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.537+0000 7fd514ff9640 10 client.15816 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.537+0000 7fd530daf640 1 --2- 172.21.15.90:0/2882393442 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c8a4a0cd80 0x55c8a4e0a630 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0c82e05e96cd033 server_cookie=d0e9b09a78ef82f1 in_seq=0 out_seq=0 2024-04-25T11:36:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.537+0000 7fd514ff9640 1 -- 172.21.15.90:0/2882393442 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c8a4d62070 con 0x55c8a4a0cd80 2024-04-25T11:36:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.537+0000 7fd514ff9640 10 client.15816 handle_command_reply: tid=0 2024-04-25T11:36:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.537+0000 7fd53a79c740 1 client.15816 shutdown 2024-04-25T11:36:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.537+0000 7fd53a79c740 1 -- 172.21.15.90:0/2882393442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a252f0 msgr2=0x55c8a4df9b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.537+0000 7fd53a79c740 1 --2- 172.21.15.90:0/2882393442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a252f0 0x55c8a4df9b00 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fd528005d50 tx=0x7fd528011e90 comp rx=0 tx=0).stop 2024-04-25T11:36:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.538+0000 7fd53a79c740 1 -- 172.21.15.90:0/2882393442 shutdown_connections 2024-04-25T11:36:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.538+0000 7fd53a79c740 1 --2- 172.21.15.90:0/2882393442 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c8a4a0cd80 0x55c8a4e0a630 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.538+0000 7fd53a79c740 1 --2- 172.21.15.90:0/2882393442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8a4a4e1b0 0x55c8a4df7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.538+0000 7fd53a79c740 1 --2- 172.21.15.90:0/2882393442 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8a4a2d4d0 0x55c8a4dfa040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.538+0000 7fd53a79c740 1 --2- 172.21.15.90:0/2882393442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a252f0 0x55c8a4df9b00 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.538+0000 7fd53a79c740 1 -- 172.21.15.90:0/2882393442 >> 172.21.15.90:0/2882393442 conn(0x55c8a4a42300 msgr2=0x55c8a4ded920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.538+0000 7fd514ff9640 0 client.15816 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:07.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.538+0000 7fd53a79c740 1 -- 172.21.15.90:0/2882393442 shutdown_connections 2024-04-25T11:36:07.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.538+0000 7fd53a79c740 1 -- 172.21.15.90:0/2882393442 wait complete. 2024-04-25T11:36:07.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.538+0000 7fd53a79c740 20 client.15816 trim_cache size 0 max 16384 2024-04-25T11:36:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.553+0000 7fd53a79c740 1 Processor -- start 2024-04-25T11:36:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.553+0000 7fd53a79c740 1 -- start start 2024-04-25T11:36:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.553+0000 7fd53a79c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a252f0 0x55c8a4df6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.553+0000 7fd53a79c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8a4a2d4d0 0x55c8a4df7f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.553+0000 7fd53a79c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8a4a4e1b0 0x55c8a4e0fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.553+0000 7fd53a79c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1afaf0 con 0x55c8a4a252f0 2024-04-25T11:36:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.553+0000 7fd53a79c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1b1630 con 0x55c8a4a2d4d0 2024-04-25T11:36:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.553+0000 7fd53a79c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1b01c0 con 0x55c8a4a4e1b0 2024-04-25T11:36:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.553+0000 7fd5315b0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a252f0 0x55c8a4df6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.553+0000 7fd5315b0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a252f0 0x55c8a4df6f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55762/0 (socket says 172.21.15.90:55762) 2024-04-25T11:36:07.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.553+0000 7fd5315b0640 1 -- 172.21.15.90:0/361732568 learned_addr learned my addr 172.21.15.90:0/361732568 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:07.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.553+0000 7fd530daf640 1 --2- 172.21.15.90:0/361732568 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8a4a2d4d0 0x55c8a4df7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:07.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.554+0000 7fd531db1640 1 --2- 172.21.15.90:0/361732568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8a4a4e1b0 0x55c8a4e0fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:07.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.554+0000 7fd5315b0640 1 -- 172.21.15.90:0/361732568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8a4a4e1b0 msgr2=0x55c8a4e0fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:07.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.554+0000 7fd5315b0640 1 --2- 172.21.15.90:0/361732568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8a4a4e1b0 0x55c8a4e0fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.554+0000 7fd5315b0640 1 -- 172.21.15.90:0/361732568 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8a4a2d4d0 msgr2=0x55c8a4df7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:07.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.554+0000 7fd5315b0640 1 --2- 172.21.15.90:0/361732568 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8a4a2d4d0 0x55c8a4df7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.554+0000 7fd5315b0640 1 -- 172.21.15.90:0/361732568 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8a4a0cfb0 con 0x55c8a4a252f0 2024-04-25T11:36:07.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.554+0000 7fd531db1640 1 --2- 172.21.15.90:0/361732568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8a4a4e1b0 0x55c8a4e0fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:36:07.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.554+0000 7fd5315b0640 1 --2- 172.21.15.90:0/361732568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a252f0 0x55c8a4df6f30 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fd528026550 tx=0x7fd528003b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d8fbf8cfac07ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:07.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.554+0000 7fd516ffd640 1 -- 172.21.15.90:0/361732568 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd528036070 con 0x55c8a4a252f0 2024-04-25T11:36:07.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.554+0000 7fd516ffd640 1 -- 172.21.15.90:0/361732568 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd528026be0 con 0x55c8a4a252f0 2024-04-25T11:36:07.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.554+0000 7fd516ffd640 1 -- 172.21.15.90:0/361732568 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd528026960 con 0x55c8a4a252f0 2024-04-25T11:36:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.555+0000 7fd53a79c740 1 -- 172.21.15.90:0/361732568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a252f0 msgr2=0x55c8a4df6f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.555+0000 7fd53a79c740 1 --2- 172.21.15.90:0/361732568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a252f0 0x55c8a4df6f30 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fd528026550 tx=0x7fd528003b40 comp rx=0 tx=0).stop 2024-04-25T11:36:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.555+0000 7fd53a79c740 1 -- 172.21.15.90:0/361732568 shutdown_connections 2024-04-25T11:36:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.555+0000 7fd53a79c740 1 --2- 172.21.15.90:0/361732568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8a4a4e1b0 0x55c8a4e0fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.555+0000 7fd53a79c740 1 --2- 172.21.15.90:0/361732568 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8a4a2d4d0 0x55c8a4df7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.555+0000 7fd53a79c740 1 --2- 172.21.15.90:0/361732568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a252f0 0x55c8a4df6f30 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.555+0000 7fd53a79c740 1 -- 172.21.15.90:0/361732568 >> 172.21.15.90:0/361732568 conn(0x55c8a4a42300 msgr2=0x55c8a4df7880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.555+0000 7fd53a79c740 1 -- 172.21.15.90:0/361732568 shutdown_connections 2024-04-25T11:36:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.555+0000 7fd53a79c740 1 -- 172.21.15.90:0/361732568 wait complete. 2024-04-25T11:36:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.555+0000 7fd53a79c740 1 -- start start 2024-04-25T11:36:07.546 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.555+0000 7fd53a79c740 1 Processor -- start 2024-04-25T11:36:07.549 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.556+0000 7fd53a79c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a252f0 0x55c8a4dfece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:07.549 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.556+0000 7fd53a79c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8a4a2d4d0 0x55c8a4dff220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:07.549 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.556+0000 7fd53a79c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8a4a4e1b0 0x55c8a4dff760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:07.549 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.556+0000 7fd53a79c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1b0f60 con 0x55c8a4a252f0 2024-04-25T11:36:07.549 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.556+0000 7fd53a79c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1b0890 con 0x55c8a4a2d4d0 2024-04-25T11:36:07.549 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.556+0000 7fd53a79c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1adad0 con 0x55c8a4a4e1b0 2024-04-25T11:36:07.549 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.556+0000 7fd53a79c740 10 client.0 _ll_register_callbacks cb 0x7fd528003a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:07.550 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.556+0000 7fd5315b0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a252f0 0x55c8a4dfece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:07.550 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.556+0000 7fd5315b0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a252f0 0x55c8a4dfece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55770/0 (socket says 172.21.15.90:55770) 2024-04-25T11:36:07.550 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.556+0000 7fd5315b0640 1 -- 172.21.15.90:0/2321142327 learned_addr learned my addr 172.21.15.90:0/2321142327 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:07.550 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.556+0000 7fd530daf640 1 --2- 172.21.15.90:0/2321142327 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8a4a2d4d0 0x55c8a4dff220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:07.550 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.556+0000 7fd5315b0640 1 -- 172.21.15.90:0/2321142327 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8a4a4e1b0 msgr2=0x55c8a4dff760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:07.550 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.556+0000 7fd5315b0640 1 --2- 172.21.15.90:0/2321142327 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8a4a4e1b0 0x55c8a4dff760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.550 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.556+0000 7fd5315b0640 1 -- 172.21.15.90:0/2321142327 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8a4a2d4d0 msgr2=0x55c8a4dff220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:07.550 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.556+0000 7fd5315b0640 1 --2- 172.21.15.90:0/2321142327 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8a4a2d4d0 0x55c8a4dff220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.550 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.556+0000 7fd5315b0640 1 -- 172.21.15.90:0/2321142327 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8a4e708d0 con 0x55c8a4a252f0 2024-04-25T11:36:07.550 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.556+0000 7fd530daf640 1 --2- 172.21.15.90:0/2321142327 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8a4a2d4d0 0x55c8a4dff220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:36:07.550 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.556+0000 7fd5315b0640 1 --2- 172.21.15.90:0/2321142327 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a252f0 0x55c8a4dfece0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fd528011910 tx=0x7fd528004be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=723c4da2612a55e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:07.550 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.556+0000 7fd514ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:07.550 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.557+0000 7fd514ff9640 1 -- 172.21.15.90:0/2321142327 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd528036040 con 0x55c8a4a252f0 2024-04-25T11:36:07.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.557+0000 7fd514ff9640 1 -- 172.21.15.90:0/2321142327 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8527+0+0 (secure 0 0 0) 0x7fd528044480 con 0x55c8a4a252f0 2024-04-25T11:36:07.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.557+0000 7fd53a79c740 1 -- 172.21.15.90:0/2321142327 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd528026be0 con 0x55c8a4a252f0 2024-04-25T11:36:07.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.557+0000 7fd514ff9640 1 -- 172.21.15.90:0/2321142327 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd528053050 con 0x55c8a4a252f0 2024-04-25T11:36:07.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.557+0000 7fd53a79c740 10 client.15822 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:07.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.557+0000 7fd53a79c740 1 -- 172.21.15.90:0/2321142327 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c8a4e71ad0 con 0x55c8a4a252f0 2024-04-25T11:36:07.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.557+0000 7fd514ff9640 1 -- 172.21.15.90:0/2321142327 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd528026be0 con 0x55c8a4a252f0 2024-04-25T11:36:07.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.557+0000 7fd514ff9640 1 -- 172.21.15.90:0/2321142327 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fd528042070 con 0x55c8a4a252f0 2024-04-25T11:36:07.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.557+0000 7fd53a79c740 10 client.15822 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:07.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.557+0000 7fd53a79c740 10 client.15822 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:36:07.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.557+0000 7fd53a79c740 20 client.15822 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:07.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.557+0000 7fd53a79c740 1 --2- 172.21.15.90:0/2321142327 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c8a49e21c0 0x55c8a4e74220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:07.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.557+0000 7fd53a79c740 4 client.15822 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:36:07.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.557+0000 7fd53a79c740 1 -- 172.21.15.90:0/2321142327 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c8a4d5bee0 con 0x55c8a49e21c0 2024-04-25T11:36:07.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.558+0000 7fd530daf640 1 --2- 172.21.15.90:0/2321142327 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c8a49e21c0 0x55c8a4e74220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:07.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.558+0000 7fd514ff9640 10 client.15822 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:07.552 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.558+0000 7fd530daf640 1 --2- 172.21.15.90:0/2321142327 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c8a49e21c0 0x55c8a4e74220 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d50574a9de4a2223 server_cookie=48a16de099e4f57e in_seq=0 out_seq=0 2024-04-25T11:36:07.554 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.565+0000 7fd514ff9640 1 -- 172.21.15.90:0/2321142327 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c8a4d5bee0 con 0x55c8a49e21c0 2024-04-25T11:36:07.554 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.565+0000 7fd514ff9640 10 client.15822 handle_command_reply: tid=0 2024-04-25T11:36:07.554 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.565+0000 7fd53a79c740 1 client.15822 shutdown 2024-04-25T11:36:07.554 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.565+0000 7fd53a79c740 1 -- 172.21.15.90:0/2321142327 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a252f0 msgr2=0x55c8a4dfece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:07.554 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.565+0000 7fd53a79c740 1 --2- 172.21.15.90:0/2321142327 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a252f0 0x55c8a4dfece0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fd528011910 tx=0x7fd528004be0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.554 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.565+0000 7fd53a79c740 1 -- 172.21.15.90:0/2321142327 shutdown_connections 2024-04-25T11:36:07.554 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.565+0000 7fd53a79c740 1 --2- 172.21.15.90:0/2321142327 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c8a49e21c0 0x55c8a4e74220 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.554 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.565+0000 7fd53a79c740 1 --2- 172.21.15.90:0/2321142327 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8a4a4e1b0 0x55c8a4dff760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.554 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.565+0000 7fd53a79c740 1 --2- 172.21.15.90:0/2321142327 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8a4a2d4d0 0x55c8a4dff220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.565+0000 7fd514ff9640 0 client.15822 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:07.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.565+0000 7fd53a79c740 1 --2- 172.21.15.90:0/2321142327 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8a4a252f0 0x55c8a4dfece0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.565+0000 7fd53a79c740 1 -- 172.21.15.90:0/2321142327 >> 172.21.15.90:0/2321142327 conn(0x55c8a4a42300 msgr2=0x55c8a4ded880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:07.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.565+0000 7fd53a79c740 1 -- 172.21.15.90:0/2321142327 shutdown_connections 2024-04-25T11:36:07.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.565+0000 7fd53a79c740 1 -- 172.21.15.90:0/2321142327 wait complete. 2024-04-25T11:36:07.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.565+0000 7fd53a79c740 20 client.15822 trim_cache size 0 max 16384 2024-04-25T11:36:07.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:36:07.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:36:07.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:36:07.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:36:07.556 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.566+0000 7fd533814640 1 -- 172.21.15.90:0/400996967 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd5180afd70 msgr2=0x7fd5180b2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:07.556 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.566+0000 7fd533814640 1 --2- 172.21.15.90:0/400996967 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd5180afd70 0x7fd5180b2280 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd528005fd0 tx=0x7fd52800b390 comp rx=0 tx=0).stop 2024-04-25T11:36:07.556 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.566+0000 7fd533814640 1 -- 172.21.15.90:0/400996967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd52c108a60 msgr2=0x7fd52c1a0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:07.556 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.566+0000 7fd533814640 1 --2- 172.21.15.90:0/400996967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd52c108a60 0x7fd52c1a0a10 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fd51c02a930 tx=0x7fd51c004270 comp rx=0 tx=0).stop 2024-04-25T11:36:07.556 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.566+0000 7fd533814640 1 -- 172.21.15.90:0/400996967 shutdown_connections 2024-04-25T11:36:07.557 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.566+0000 7fd533814640 1 --2- 172.21.15.90:0/400996967 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd5180afd70 0x7fd5180b2280 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.557 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.566+0000 7fd533814640 1 --2- 172.21.15.90:0/400996967 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd52c108f00 0x7fd52c19a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.557 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.566+0000 7fd533814640 1 --2- 172.21.15.90:0/400996967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd52c108a60 0x7fd52c1a0a10 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.557 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.566+0000 7fd533814640 1 --2- 172.21.15.90:0/400996967 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd52c1031c0 0x7fd52c1a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.557 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.566+0000 7fd533814640 1 -- 172.21.15.90:0/400996967 >> 172.21.15.90:0/400996967 conn(0x7fd52c0781b0 msgr2=0x7fd52c06a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:07.557 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.566+0000 7fd533814640 1 -- 172.21.15.90:0/400996967 shutdown_connections 2024-04-25T11:36:07.557 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:07.566+0000 7fd533814640 1 -- 172.21.15.90:0/400996967 wait complete. 2024-04-25T11:36:07.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:07 smithi090 ceph-mon[29575]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-04-25T11:36:08.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:07 smithi138 ceph-mon[32328]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-04-25T11:36:08.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:07 smithi156 ceph-mon[33045]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-04-25T11:36:08.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2024-04-25T11:36:08.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2024-04-25T11:36:08.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:08 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-25T11:36:09.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2024-04-25T11:36:09.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2024-04-25T11:36:09.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:08 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-25T11:36:09.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2024-04-25T11:36:09.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2024-04-25T11:36:09.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:08 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-25T11:36:10.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:09 smithi138 ceph-mon[32328]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 23 KiB/s wr, 15 op/s 2024-04-25T11:36:10.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:09 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2024-04-25T11:36:10.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:09 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2024-04-25T11:36:10.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:09 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-25T11:36:10.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:09 smithi138 ceph-mon[32328]: osdmap e81: 12 total, 12 up, 12 in 2024-04-25T11:36:10.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:09 smithi156 ceph-mon[33045]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 23 KiB/s wr, 15 op/s 2024-04-25T11:36:10.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:09 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2024-04-25T11:36:10.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:09 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2024-04-25T11:36:10.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:09 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-25T11:36:10.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:09 smithi156 ceph-mon[33045]: osdmap e81: 12 total, 12 up, 12 in 2024-04-25T11:36:10.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:09 smithi090 ceph-mon[29575]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 23 KiB/s wr, 15 op/s 2024-04-25T11:36:10.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:09 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2024-04-25T11:36:10.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:09 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2024-04-25T11:36:10.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:09 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-25T11:36:10.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:09 smithi090 ceph-mon[29575]: osdmap e81: 12 total, 12 up, 12 in 2024-04-25T11:36:11.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:10 smithi138 ceph-mon[32328]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T11:36:11.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:10 smithi156 ceph-mon[33045]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T11:36:11.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:10 smithi090 ceph-mon[29575]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T11:36:12.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:11 smithi138 ceph-mon[32328]: pgmap v36: 97 pgs: 3 peering, 94 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 35 KiB/s wr, 14 op/s 2024-04-25T11:36:12.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:11 smithi138 ceph-mon[32328]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-04-25T11:36:12.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:11 smithi156 ceph-mon[33045]: pgmap v36: 97 pgs: 3 peering, 94 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 35 KiB/s wr, 14 op/s 2024-04-25T11:36:12.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:11 smithi156 ceph-mon[33045]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-04-25T11:36:12.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:11 smithi090 ceph-mon[29575]: pgmap v36: 97 pgs: 3 peering, 94 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 35 KiB/s wr, 14 op/s 2024-04-25T11:36:12.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:11 smithi090 ceph-mon[29575]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-04-25T11:36:14.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:13 smithi138 ceph-mon[32328]: pgmap v37: 97 pgs: 3 peering, 94 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 1 op/s 2024-04-25T11:36:14.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:13 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:36:14.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:13 smithi090 ceph-mon[29575]: pgmap v37: 97 pgs: 3 peering, 94 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 1 op/s 2024-04-25T11:36:14.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:13 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:36:14.289 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:13 smithi156 ceph-mon[33045]: pgmap v37: 97 pgs: 3 peering, 94 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 1 op/s 2024-04-25T11:36:14.289 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:13 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:36:15.725 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:15 smithi090 ceph-mon[29575]: pgmap v38: 97 pgs: 2 peering, 95 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 895 B/s wr, 5 op/s; 243 B/s, 0 objects/s recovering 2024-04-25T11:36:15.725 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:15 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:36:15.725 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:15 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:36:15.725 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:15 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:36:15.725 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:15 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:36:15.725 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:15 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:36:15.736 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:36:15.736 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:36:15.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:15 smithi138 ceph-mon[32328]: pgmap v38: 97 pgs: 2 peering, 95 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 895 B/s wr, 5 op/s; 243 B/s, 0 objects/s recovering 2024-04-25T11:36:15.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:15 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:36:15.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:15 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:36:15.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:15 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:36:15.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:15 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:36:15.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:15 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:36:15.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:15 smithi156 ceph-mon[33045]: pgmap v38: 97 pgs: 2 peering, 95 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 895 B/s wr, 5 op/s; 243 B/s, 0 objects/s recovering 2024-04-25T11:36:15.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:15 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:36:15.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:15 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:36:15.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:15 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:36:15.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:15 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:36:15.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:15 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:36:15.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.897+0000 7f2697506640 1 -- 172.21.15.90:0/1783438962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26901069b0 msgr2=0x7f2690106de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:15.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.897+0000 7f2697506640 1 --2- 172.21.15.90:0/1783438962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26901069b0 0x7f2690106de0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f2680009900 tx=0x7f268002a410 comp rx=0 tx=0).stop 2024-04-25T11:36:15.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.898+0000 7f2697506640 1 -- 172.21.15.90:0/1783438962 shutdown_connections 2024-04-25T11:36:15.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.898+0000 7f2697506640 1 --2- 172.21.15.90:0/1783438962 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2690107470 0x7f269010ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.898+0000 7f2697506640 1 --2- 172.21.15.90:0/1783438962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26901069b0 0x7f2690106de0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.898+0000 7f2697506640 1 --2- 172.21.15.90:0/1783438962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2690102420 0x7f2690102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.898+0000 7f2697506640 1 -- 172.21.15.90:0/1783438962 >> 172.21.15.90:0/1783438962 conn(0x7f2690077920 msgr2=0x7f2690077d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:15.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.898+0000 7f2697506640 1 -- 172.21.15.90:0/1783438962 shutdown_connections 2024-04-25T11:36:15.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.898+0000 7f2697506640 1 -- 172.21.15.90:0/1783438962 wait complete. 2024-04-25T11:36:15.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.899+0000 7f2697506640 1 Processor -- start 2024-04-25T11:36:15.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.899+0000 7f2697506640 1 -- start start 2024-04-25T11:36:15.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.899+0000 7f2697506640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2690102420 0x7f269019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:15.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.899+0000 7f2697506640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f26901069b0 0x7f269019acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:15.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.899+0000 7f2697506640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2690107470 0x7f269019ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:15.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.899+0000 7f2695aa3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2690107470 0x7f269019ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:15.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.899+0000 7f2695aa3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2690107470 0x7f269019ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39868/0 (socket says 172.21.15.90:39868) 2024-04-25T11:36:15.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.899+0000 7f2695aa3640 1 -- 172.21.15.90:0/2185122668 learned_addr learned my addr 172.21.15.90:0/2185122668 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:15.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.899+0000 7f26952a2640 1 --2- 172.21.15.90:0/2185122668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2690102420 0x7f269019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:15.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.899+0000 7f2697506640 1 -- 172.21.15.90:0/2185122668 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2690111b00 con 0x7f2690107470 2024-04-25T11:36:15.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.899+0000 7f2697506640 1 -- 172.21.15.90:0/2185122668 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2690111980 con 0x7f26901069b0 2024-04-25T11:36:15.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.899+0000 7f2697506640 1 -- 172.21.15.90:0/2185122668 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2690111c80 con 0x7f2690102420 2024-04-25T11:36:15.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.899+0000 7f2694aa1640 1 --2- 172.21.15.90:0/2185122668 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f26901069b0 0x7f269019acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:15.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.900+0000 7f2695aa3640 1 -- 172.21.15.90:0/2185122668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2690102420 msgr2=0x7f269019a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:15.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.900+0000 7f2695aa3640 1 --2- 172.21.15.90:0/2185122668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2690102420 0x7f269019a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.900+0000 7f2695aa3640 1 -- 172.21.15.90:0/2185122668 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f26901069b0 msgr2=0x7f269019acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:15.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.900+0000 7f2695aa3640 1 --2- 172.21.15.90:0/2185122668 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f26901069b0 0x7f269019acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.900+0000 7f2695aa3640 1 -- 172.21.15.90:0/2185122668 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f269019f5b0 con 0x7f2690107470 2024-04-25T11:36:15.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.900+0000 7f2695aa3640 1 --2- 172.21.15.90:0/2185122668 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2690107470 0x7f269019ef70 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f268800ec60 tx=0x7f268800c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:15.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.900+0000 7f26867fc640 1 -- 172.21.15.90:0/2185122668 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2688004d20 con 0x7f2690107470 2024-04-25T11:36:15.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.900+0000 7f26867fc640 1 -- 172.21.15.90:0/2185122668 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f268800ba20 con 0x7f2690107470 2024-04-25T11:36:15.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.900+0000 7f26867fc640 1 -- 172.21.15.90:0/2185122668 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26880056c0 con 0x7f2690107470 2024-04-25T11:36:15.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.900+0000 7f2697506640 1 -- 172.21.15.90:0/2185122668 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f269019f8a0 con 0x7f2690107470 2024-04-25T11:36:15.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.900+0000 7f2697506640 1 -- 172.21.15.90:0/2185122668 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26901a7220 con 0x7f2690107470 2024-04-25T11:36:15.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.903+0000 7f26867fc640 1 -- 172.21.15.90:0/2185122668 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 24) ==== 147753+0+0 (secure 0 0 0) 0x7f2688005860 con 0x7f2690107470 2024-04-25T11:36:15.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.904+0000 7f269e3ba740 1 Processor -- start 2024-04-25T11:36:15.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.904+0000 7f269e3ba740 1 -- start start 2024-04-25T11:36:15.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.904+0000 7f269e3ba740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85809d390 0x55f85843da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:15.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.904+0000 7f269e3ba740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8580744d0 0x55f85843dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:15.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.904+0000 7f269e3ba740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85807c6b0 0x55f85844a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:15.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.904+0000 7f26867fc640 1 --2- 172.21.15.90:0/2185122668 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f267c0afe00 0x7f267c0b2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:15.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.904+0000 7f269e3ba740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f8583a3d80 con 0x55f8580744d0 2024-04-25T11:36:15.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.904+0000 7f269e3ba740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f85835d8b0 con 0x55f85807c6b0 2024-04-25T11:36:15.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.904+0000 7f269e3ba740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f85839fd40 con 0x55f85809d390 2024-04-25T11:36:15.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.904+0000 7f26952a2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8580744d0 0x55f85843dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:15.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.904+0000 7f26952a2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8580744d0 0x55f85843dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39878/0 (socket says 172.21.15.90:39878) 2024-04-25T11:36:15.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.904+0000 7f26952a2640 1 -- 172.21.15.90:0/2530609096 learned_addr learned my addr 172.21.15.90:0/2530609096 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:15.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.904+0000 7f2694aa1640 1 --2- 172.21.15.90:0/2185122668 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f267c0afe00 0x7f267c0b2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:15.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.904+0000 7f26867fc640 1 -- 172.21.15.90:0/2185122668 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f26880df780 con 0x7f2690107470 2024-04-25T11:36:15.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.905+0000 7f2694aa1640 1 --2- 172.21.15.90:0/2530609096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85809d390 0x55f85843da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:15.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.905+0000 7f2695aa3640 1 --2- 172.21.15.90:0/2530609096 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85807c6b0 0x55f85844a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:15.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.905+0000 7f26952a2640 1 -- 172.21.15.90:0/2530609096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85809d390 msgr2=0x55f85843da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:15.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.905+0000 7f26952a2640 1 --2- 172.21.15.90:0/2530609096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85809d390 0x55f85843da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.905+0000 7f26952a2640 1 -- 172.21.15.90:0/2530609096 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85807c6b0 msgr2=0x55f85844a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:15.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.905+0000 7f26952a2640 1 --2- 172.21.15.90:0/2530609096 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85807c6b0 0x55f85844a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.905+0000 7f26952a2640 1 -- 172.21.15.90:0/2530609096 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f85832ea40 con 0x55f8580744d0 2024-04-25T11:36:15.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.905+0000 7f2695aa3640 1 --2- 172.21.15.90:0/2530609096 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85807c6b0 0x55f85844a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:36:15.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.905+0000 7f26952a2640 1 --2- 172.21.15.90:0/2530609096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8580744d0 0x55f85843dfa0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f268c00bcd0 tx=0x7f268c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44a41a4dd0e68fe8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:15.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.905+0000 7f2694aa1640 1 --2- 172.21.15.90:0/2185122668 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f267c0afe00 0x7f267c0b2310 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f269010dfe0 tx=0x7f2680000bc0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:15.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.905+0000 7f267b7fe640 1 -- 172.21.15.90:0/2530609096 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268c01c680 con 0x55f8580744d0 2024-04-25T11:36:15.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.905+0000 7f267b7fe640 1 -- 172.21.15.90:0/2530609096 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f268c01ccd0 con 0x55f8580744d0 2024-04-25T11:36:15.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.905+0000 7f267b7fe640 1 -- 172.21.15.90:0/2530609096 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268c011aa0 con 0x55f8580744d0 2024-04-25T11:36:15.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.906+0000 7f269e3ba740 1 -- 172.21.15.90:0/2530609096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8580744d0 msgr2=0x55f85843dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:15.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.906+0000 7f269e3ba740 1 --2- 172.21.15.90:0/2530609096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8580744d0 0x55f85843dfa0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f268c00bcd0 tx=0x7f268c0107a0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.906+0000 7f269e3ba740 1 -- 172.21.15.90:0/2530609096 shutdown_connections 2024-04-25T11:36:15.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.906+0000 7f269e3ba740 1 --2- 172.21.15.90:0/2530609096 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85807c6b0 0x55f85844a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.906+0000 7f269e3ba740 1 --2- 172.21.15.90:0/2530609096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8580744d0 0x55f85843dfa0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.906+0000 7f269e3ba740 1 --2- 172.21.15.90:0/2530609096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85809d390 0x55f85843da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.906+0000 7f269e3ba740 1 -- 172.21.15.90:0/2530609096 >> 172.21.15.90:0/2530609096 conn(0x55f8580914e0 msgr2=0x55f85843a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:15.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.906+0000 7f269e3ba740 1 -- 172.21.15.90:0/2530609096 shutdown_connections 2024-04-25T11:36:15.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.906+0000 7f269e3ba740 1 -- 172.21.15.90:0/2530609096 wait complete. 2024-04-25T11:36:15.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.907+0000 7f269e3ba740 1 -- start start 2024-04-25T11:36:15.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.907+0000 7f269e3ba740 1 Processor -- start 2024-04-25T11:36:15.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.907+0000 7f269e3ba740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f8580744d0 0x55f8584485c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:15.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.907+0000 7f269e3ba740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85807c6b0 0x55f858448b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:15.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.907+0000 7f269e3ba740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85809d390 0x55f858458930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:15.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.907+0000 7f269e3ba740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f85844f3f0 con 0x55f85807c6b0 2024-04-25T11:36:15.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.907+0000 7f269e3ba740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f85832ad10 con 0x55f8580744d0 2024-04-25T11:36:15.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.907+0000 7f269e3ba740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f85844f570 con 0x55f85809d390 2024-04-25T11:36:15.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.908+0000 7f269e3ba740 10 client.0 _ll_register_callbacks cb 0x55f858357210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:15.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.908+0000 7f2694aa1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85807c6b0 0x55f858448b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:15.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.908+0000 7f2694aa1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85807c6b0 0x55f858448b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39882/0 (socket says 172.21.15.90:39882) 2024-04-25T11:36:15.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.908+0000 7f2694aa1640 1 -- 172.21.15.90:0/2546029394 learned_addr learned my addr 172.21.15.90:0/2546029394 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:15.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.908+0000 7f26952a2640 1 --2- 172.21.15.90:0/2546029394 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f8580744d0 0x55f8584485c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:15.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.908+0000 7f2695aa3640 1 --2- 172.21.15.90:0/2546029394 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85809d390 0x55f858458930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:15.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.908+0000 7f2694aa1640 1 -- 172.21.15.90:0/2546029394 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85809d390 msgr2=0x55f858458930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:15.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.908+0000 7f2694aa1640 1 --2- 172.21.15.90:0/2546029394 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85809d390 0x55f858458930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.908+0000 7f2694aa1640 1 -- 172.21.15.90:0/2546029394 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f8580744d0 msgr2=0x55f8584485c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:15.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.908+0000 7f2694aa1640 1 --2- 172.21.15.90:0/2546029394 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f8580744d0 0x55f8584485c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.908+0000 7f2694aa1640 1 -- 172.21.15.90:0/2546029394 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f85817f7d0 con 0x55f85807c6b0 2024-04-25T11:36:15.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.908+0000 7f26952a2640 1 --2- 172.21.15.90:0/2546029394 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f8580744d0 0x55f8584485c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:36:15.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.908+0000 7f2694aa1640 1 --2- 172.21.15.90:0/2546029394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85807c6b0 0x55f858448b00 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f2680033000 tx=0x7f2680005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acc6e694b97f62d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:15.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.908+0000 7f2678ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:15.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.908+0000 7f2678ff9640 1 -- 172.21.15.90:0/2546029394 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2680050060 con 0x55f85807c6b0 2024-04-25T11:36:15.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.908+0000 7f2678ff9640 1 -- 172.21.15.90:0/2546029394 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f26800035f0 con 0x55f85807c6b0 2024-04-25T11:36:15.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.908+0000 7f269e3ba740 1 -- 172.21.15.90:0/2546029394 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f268c01ccd0 con 0x55f85807c6b0 2024-04-25T11:36:15.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.909+0000 7f2678ff9640 1 -- 172.21.15.90:0/2546029394 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2680035040 con 0x55f85807c6b0 2024-04-25T11:36:15.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.909+0000 7f269e3ba740 10 client.15834 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:15.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.909+0000 7f269e3ba740 1 -- 172.21.15.90:0/2546029394 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f8580b53f0 con 0x55f85807c6b0 2024-04-25T11:36:15.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.909+0000 7f2678ff9640 1 -- 172.21.15.90:0/2546029394 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2680003ca0 con 0x55f85807c6b0 2024-04-25T11:36:15.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.909+0000 7f2678ff9640 1 -- 172.21.15.90:0/2546029394 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f268004bb10 con 0x55f85807c6b0 2024-04-25T11:36:15.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.909+0000 7f269e3ba740 10 client.15834 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:15.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.909+0000 7f269e3ba740 10 client.15834 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:36:15.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.909+0000 7f269e3ba740 20 client.15834 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:15.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.909+0000 7f269e3ba740 1 --2- 172.21.15.90:0/2546029394 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f85805bf60 0x55f85845ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:15.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.909+0000 7f269e3ba740 4 client.15834 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:36:15.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.909+0000 7f269e3ba740 1 -- 172.21.15.90:0/2546029394 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f268c01ccd0 con 0x55f85805bf60 2024-04-25T11:36:15.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.910+0000 7f26952a2640 1 --2- 172.21.15.90:0/2546029394 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f85805bf60 0x55f85845ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:15.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.910+0000 7f2678ff9640 10 client.15834 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:15.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.910+0000 7f26952a2640 1 --2- 172.21.15.90:0/2546029394 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f85805bf60 0x55f85845ee50 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c757aff37e75fbc server_cookie=a0932ecd3b812028 in_seq=0 out_seq=0 2024-04-25T11:36:15.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.911+0000 7f2678ff9640 1 -- 172.21.15.90:0/2546029394 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f268c01ccd0 con 0x55f85805bf60 2024-04-25T11:36:15.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.911+0000 7f2678ff9640 10 client.15834 handle_command_reply: tid=0 2024-04-25T11:36:15.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.911+0000 7f269e3ba740 1 client.15834 shutdown 2024-04-25T11:36:15.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.911+0000 7f269e3ba740 1 -- 172.21.15.90:0/2546029394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85807c6b0 msgr2=0x55f858448b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:15.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.911+0000 7f269e3ba740 1 --2- 172.21.15.90:0/2546029394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85807c6b0 0x55f858448b00 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f2680033000 tx=0x7f2680005fd0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.911+0000 7f269e3ba740 1 -- 172.21.15.90:0/2546029394 shutdown_connections 2024-04-25T11:36:15.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.911+0000 7f269e3ba740 1 --2- 172.21.15.90:0/2546029394 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f85805bf60 0x55f85845ee50 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.911+0000 7f269e3ba740 1 --2- 172.21.15.90:0/2546029394 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85809d390 0x55f858458930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.911+0000 7f269e3ba740 1 --2- 172.21.15.90:0/2546029394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85807c6b0 0x55f858448b00 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.911+0000 7f269e3ba740 1 --2- 172.21.15.90:0/2546029394 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f8580744d0 0x55f8584485c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.911+0000 7f269e3ba740 1 -- 172.21.15.90:0/2546029394 >> 172.21.15.90:0/2546029394 conn(0x55f8580914e0 msgr2=0x55f85843f780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:15.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.911+0000 7f2678ff9640 0 client.15834 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:15.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.911+0000 7f269e3ba740 1 -- 172.21.15.90:0/2546029394 shutdown_connections 2024-04-25T11:36:15.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.911+0000 7f269e3ba740 1 -- 172.21.15.90:0/2546029394 wait complete. 2024-04-25T11:36:15.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.911+0000 7f269e3ba740 20 client.15834 trim_cache size 0 max 16384 2024-04-25T11:36:15.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.929+0000 7f269e3ba740 1 Processor -- start 2024-04-25T11:36:15.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.929+0000 7f269e3ba740 1 -- start start 2024-04-25T11:36:15.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.929+0000 7f269e3ba740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f8580744d0 0x55f858446c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:15.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.929+0000 7f269e3ba740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85807c6b0 0x55f8584471c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:15.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.929+0000 7f269e3ba740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85809d390 0x55f8584399d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:15.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.929+0000 7f269e3ba740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f26901ad9f0 con 0x55f85807c6b0 2024-04-25T11:36:15.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.929+0000 7f269e3ba740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f26901af530 con 0x55f8580744d0 2024-04-25T11:36:15.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.929+0000 7f269e3ba740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f26901ae0c0 con 0x55f85809d390 2024-04-25T11:36:15.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.929+0000 7f2694aa1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85807c6b0 0x55f8584471c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:15.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.929+0000 7f2694aa1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85807c6b0 0x55f8584471c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39886/0 (socket says 172.21.15.90:39886) 2024-04-25T11:36:15.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.929+0000 7f2694aa1640 1 -- 172.21.15.90:0/328174168 learned_addr learned my addr 172.21.15.90:0/328174168 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:15.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.930+0000 7f26952a2640 1 --2- 172.21.15.90:0/328174168 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f8580744d0 0x55f858446c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:15.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.930+0000 7f2695aa3640 1 --2- 172.21.15.90:0/328174168 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85809d390 0x55f8584399d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:15.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.930+0000 7f2694aa1640 1 -- 172.21.15.90:0/328174168 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85809d390 msgr2=0x55f8584399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:15.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.930+0000 7f2694aa1640 1 --2- 172.21.15.90:0/328174168 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85809d390 0x55f8584399d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.930+0000 7f2694aa1640 1 -- 172.21.15.90:0/328174168 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f8580744d0 msgr2=0x55f858446c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:15.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.930+0000 7f2694aa1640 1 --2- 172.21.15.90:0/328174168 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f8580744d0 0x55f858446c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.930+0000 7f2694aa1640 1 -- 172.21.15.90:0/328174168 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8580750f0 con 0x55f85807c6b0 2024-04-25T11:36:15.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.930+0000 7f2695aa3640 1 --2- 172.21.15.90:0/328174168 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85809d390 0x55f8584399d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:36:15.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.930+0000 7f2694aa1640 1 --2- 172.21.15.90:0/328174168 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85807c6b0 0x55f8584471c0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f268002a930 tx=0x7f2680031830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d729c7329305d94a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:15.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.930+0000 7f267affd640 1 -- 172.21.15.90:0/328174168 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2680050040 con 0x55f85807c6b0 2024-04-25T11:36:15.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.930+0000 7f267affd640 1 -- 172.21.15.90:0/328174168 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f268003d690 con 0x55f85807c6b0 2024-04-25T11:36:15.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.930+0000 7f267affd640 1 -- 172.21.15.90:0/328174168 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26800035f0 con 0x55f85807c6b0 2024-04-25T11:36:15.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.931+0000 7f269e3ba740 1 -- 172.21.15.90:0/328174168 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85807c6b0 msgr2=0x55f8584471c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:15.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.931+0000 7f269e3ba740 1 --2- 172.21.15.90:0/328174168 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85807c6b0 0x55f8584471c0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f268002a930 tx=0x7f2680031830 comp rx=0 tx=0).stop 2024-04-25T11:36:15.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.931+0000 7f269e3ba740 1 -- 172.21.15.90:0/328174168 shutdown_connections 2024-04-25T11:36:15.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.931+0000 7f269e3ba740 1 --2- 172.21.15.90:0/328174168 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85809d390 0x55f8584399d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.931+0000 7f269e3ba740 1 --2- 172.21.15.90:0/328174168 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85807c6b0 0x55f8584471c0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.931+0000 7f269e3ba740 1 --2- 172.21.15.90:0/328174168 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f8580744d0 0x55f858446c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.931+0000 7f269e3ba740 1 -- 172.21.15.90:0/328174168 >> 172.21.15.90:0/328174168 conn(0x55f8580914e0 msgr2=0x55f858443540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:15.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.931+0000 7f269e3ba740 1 -- 172.21.15.90:0/328174168 shutdown_connections 2024-04-25T11:36:15.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.931+0000 7f269e3ba740 1 -- 172.21.15.90:0/328174168 wait complete. 2024-04-25T11:36:15.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.932+0000 7f269e3ba740 1 -- start start 2024-04-25T11:36:15.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.932+0000 7f269e3ba740 1 Processor -- start 2024-04-25T11:36:15.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.932+0000 7f269e3ba740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8580744d0 0x55f85845ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:15.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.932+0000 7f269e3ba740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85807c6b0 0x55f85845f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:15.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.932+0000 7f269e3ba740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85809d390 0x55f85845f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:15.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.932+0000 7f269e3ba740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f26901aee60 con 0x55f85807c6b0 2024-04-25T11:36:15.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.932+0000 7f269e3ba740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f26901ae790 con 0x55f85809d390 2024-04-25T11:36:15.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.932+0000 7f269e3ba740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f26901ab9d0 con 0x55f8580744d0 2024-04-25T11:36:15.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.932+0000 7f269e3ba740 10 client.0 _ll_register_callbacks cb 0x7f268800a4b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:15.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.932+0000 7f2694aa1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85807c6b0 0x55f85845f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:15.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.932+0000 7f2694aa1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85807c6b0 0x55f85845f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39902/0 (socket says 172.21.15.90:39902) 2024-04-25T11:36:15.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.932+0000 7f2694aa1640 1 -- 172.21.15.90:0/2883275699 learned_addr learned my addr 172.21.15.90:0/2883275699 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:15.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.932+0000 7f26952a2640 1 --2- 172.21.15.90:0/2883275699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8580744d0 0x55f85845ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:15.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.933+0000 7f2695aa3640 1 --2- 172.21.15.90:0/2883275699 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85809d390 0x55f85845f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:15.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.933+0000 7f2694aa1640 1 -- 172.21.15.90:0/2883275699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8580744d0 msgr2=0x55f85845ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:15.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.933+0000 7f2694aa1640 1 --2- 172.21.15.90:0/2883275699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8580744d0 0x55f85845ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.933+0000 7f2694aa1640 1 -- 172.21.15.90:0/2883275699 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85809d390 msgr2=0x55f85845f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:15.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.933+0000 7f2694aa1640 1 --2- 172.21.15.90:0/2883275699 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85809d390 0x55f85845f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.933+0000 7f2694aa1640 1 -- 172.21.15.90:0/2883275699 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8583644b0 con 0x55f85807c6b0 2024-04-25T11:36:15.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.933+0000 7f2694aa1640 1 --2- 172.21.15.90:0/2883275699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85807c6b0 0x55f85845f100 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f2680031440 tx=0x7f2680032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8b28fa283211b3d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:15.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.933+0000 7f2678ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:15.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.933+0000 7f2678ff9640 1 -- 172.21.15.90:0/2883275699 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2680050040 con 0x55f85807c6b0 2024-04-25T11:36:15.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.933+0000 7f269e3ba740 1 -- 172.21.15.90:0/2883275699 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f268003d690 con 0x55f85807c6b0 2024-04-25T11:36:15.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.933+0000 7f2678ff9640 1 -- 172.21.15.90:0/2883275699 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f26800035f0 con 0x55f85807c6b0 2024-04-25T11:36:15.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.933+0000 7f2678ff9640 1 -- 172.21.15.90:0/2883275699 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2680032cd0 con 0x55f85807c6b0 2024-04-25T11:36:15.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.933+0000 7f269e3ba740 10 client.15840 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:15.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.933+0000 7f269e3ba740 1 -- 172.21.15.90:0/2883275699 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f8584c62e0 con 0x55f85807c6b0 2024-04-25T11:36:15.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.934+0000 7f2678ff9640 1 -- 172.21.15.90:0/2883275699 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f268003d690 con 0x55f85807c6b0 2024-04-25T11:36:15.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.934+0000 7f2678ff9640 1 -- 172.21.15.90:0/2883275699 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f268004b7f0 con 0x55f85807c6b0 2024-04-25T11:36:15.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.934+0000 7f269e3ba740 10 client.15840 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:15.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.934+0000 7f269e3ba740 10 client.15840 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:36:15.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.934+0000 7f269e3ba740 20 client.15840 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:15.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.934+0000 7f269e3ba740 1 --2- 172.21.15.90:0/2883275699 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f8580313a0 0x55f8584c8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:15.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.934+0000 7f269e3ba740 4 client.15840 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:36:15.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.934+0000 7f269e3ba740 1 -- 172.21.15.90:0/2883275699 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f8583b1990 con 0x55f8580313a0 2024-04-25T11:36:15.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.934+0000 7f26952a2640 1 --2- 172.21.15.90:0/2883275699 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f8580313a0 0x55f8584c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:15.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.935+0000 7f2678ff9640 10 client.15840 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:15.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.935+0000 7f26952a2640 1 --2- 172.21.15.90:0/2883275699 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f8580313a0 0x55f8584c8980 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60379227bfdc20db server_cookie=10f2e98933b8efa3 in_seq=0 out_seq=0 2024-04-25T11:36:15.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.935+0000 7f2678ff9640 1 -- 172.21.15.90:0/2883275699 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f8583b1990 con 0x55f8580313a0 2024-04-25T11:36:15.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.935+0000 7f2678ff9640 10 client.15840 handle_command_reply: tid=0 2024-04-25T11:36:15.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.935+0000 7f269e3ba740 1 client.15840 shutdown 2024-04-25T11:36:15.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.935+0000 7f269e3ba740 1 -- 172.21.15.90:0/2883275699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85807c6b0 msgr2=0x55f85845f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:15.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.935+0000 7f269e3ba740 1 --2- 172.21.15.90:0/2883275699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85807c6b0 0x55f85845f100 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f2680031440 tx=0x7f2680032f90 comp rx=0 tx=0).stop 2024-04-25T11:36:15.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.935+0000 7f269e3ba740 1 -- 172.21.15.90:0/2883275699 shutdown_connections 2024-04-25T11:36:15.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.935+0000 7f269e3ba740 1 --2- 172.21.15.90:0/2883275699 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f8580313a0 0x55f8584c8980 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.935+0000 7f2678ff9640 0 client.15840 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:15.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.935+0000 7f269e3ba740 1 --2- 172.21.15.90:0/2883275699 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85809d390 0x55f85845f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.935+0000 7f269e3ba740 1 --2- 172.21.15.90:0/2883275699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85807c6b0 0x55f85845f100 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.935+0000 7f269e3ba740 1 --2- 172.21.15.90:0/2883275699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8580744d0 0x55f85845ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.935+0000 7f269e3ba740 1 -- 172.21.15.90:0/2883275699 >> 172.21.15.90:0/2883275699 conn(0x55f8580914e0 msgr2=0x55f858446b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:15.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.935+0000 7f269e3ba740 1 -- 172.21.15.90:0/2883275699 shutdown_connections 2024-04-25T11:36:15.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.935+0000 7f269e3ba740 1 -- 172.21.15.90:0/2883275699 wait complete. 2024-04-25T11:36:15.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.935+0000 7f269e3ba740 20 client.15840 trim_cache size 0 max 16384 2024-04-25T11:36:15.929 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:36:15.929 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-25T11:36:15.929 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-25T11:36:15.929 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:36:15.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.936+0000 7f2697506640 1 -- 172.21.15.90:0/2185122668 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f267c0afe00 msgr2=0x7f267c0b2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:15.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.936+0000 7f2697506640 1 --2- 172.21.15.90:0/2185122668 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f267c0afe00 0x7f267c0b2310 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f269010dfe0 tx=0x7f2680000bc0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.936+0000 7f2697506640 1 -- 172.21.15.90:0/2185122668 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2690107470 msgr2=0x7f269019ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:15.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.936+0000 7f2697506640 1 --2- 172.21.15.90:0/2185122668 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2690107470 0x7f269019ef70 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f268800ec60 tx=0x7f268800c7a0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.936+0000 7f2697506640 1 -- 172.21.15.90:0/2185122668 shutdown_connections 2024-04-25T11:36:15.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.936+0000 7f2697506640 1 --2- 172.21.15.90:0/2185122668 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f267c0afe00 0x7f267c0b2310 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.936+0000 7f2697506640 1 --2- 172.21.15.90:0/2185122668 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2690107470 0x7f269019ef70 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.936+0000 7f2697506640 1 --2- 172.21.15.90:0/2185122668 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f26901069b0 0x7f269019acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.936+0000 7f2697506640 1 --2- 172.21.15.90:0/2185122668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2690102420 0x7f269019a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.936+0000 7f2697506640 1 -- 172.21.15.90:0/2185122668 >> 172.21.15.90:0/2185122668 conn(0x7f2690077920 msgr2=0x7f2690100900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:15.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.936+0000 7f2697506640 1 -- 172.21.15.90:0/2185122668 shutdown_connections 2024-04-25T11:36:15.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:15.936+0000 7f2697506640 1 -- 172.21.15.90:0/2185122668 wait complete. 2024-04-25T11:36:15.936 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T11:36:15.937 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:36:16.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.073+0000 7fba5478f640 1 -- 172.21.15.90:0/870595470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba4c10e040 msgr2=0x7fba4c110530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.073+0000 7fba5478f640 1 --2- 172.21.15.90:0/870595470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba4c10e040 0x7fba4c110530 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fba44009ca0 tx=0x7fba4402a8c0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.073+0000 7fba5478f640 1 -- 172.21.15.90:0/870595470 shutdown_connections 2024-04-25T11:36:16.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.073+0000 7fba5478f640 1 --2- 172.21.15.90:0/870595470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba4c10e040 0x7fba4c110530 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.073+0000 7fba5478f640 1 --2- 172.21.15.90:0/870595470 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba4c10b650 0x7fba4c10da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.073+0000 7fba5478f640 1 --2- 172.21.15.90:0/870595470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba4c104bf0 0x7fba4c105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.073+0000 7fba5478f640 1 -- 172.21.15.90:0/870595470 >> 172.21.15.90:0/870595470 conn(0x7fba4c078110 msgr2=0x7fba4c078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:16.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.073+0000 7fba5478f640 1 -- 172.21.15.90:0/870595470 shutdown_connections 2024-04-25T11:36:16.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.073+0000 7fba5478f640 1 -- 172.21.15.90:0/870595470 wait complete. 2024-04-25T11:36:16.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.074+0000 7fba5478f640 1 Processor -- start 2024-04-25T11:36:16.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.074+0000 7fba5478f640 1 -- start start 2024-04-25T11:36:16.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.074+0000 7fba5478f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba4c104bf0 0x7fba4c1024b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.074+0000 7fba5478f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba4c10b650 0x7fba4c1029f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.074+0000 7fba5478f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba4c10e040 0x7fba4c102f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.074+0000 7fba5252b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba4c104bf0 0x7fba4c1024b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.074+0000 7fba5478f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c113ca0 con 0x7fba4c104bf0 2024-04-25T11:36:16.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.074+0000 7fba5478f640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c113b20 con 0x7fba4c10e040 2024-04-25T11:36:16.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.074+0000 7fba5478f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c113e20 con 0x7fba4c10b650 2024-04-25T11:36:16.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.074+0000 7fba5252b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba4c104bf0 0x7fba4c1024b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39914/0 (socket says 172.21.15.90:39914) 2024-04-25T11:36:16.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.074+0000 7fba5252b640 1 -- 172.21.15.90:0/117753333 learned_addr learned my addr 172.21.15.90:0/117753333 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:16.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.074+0000 7fba52d2c640 1 --2- 172.21.15.90:0/117753333 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba4c10e040 0x7fba4c102f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.074+0000 7fba51d2a640 1 --2- 172.21.15.90:0/117753333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba4c10b650 0x7fba4c1029f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.075+0000 7fba5252b640 1 -- 172.21.15.90:0/117753333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba4c10b650 msgr2=0x7fba4c1029f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.075+0000 7fba5252b640 1 --2- 172.21.15.90:0/117753333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba4c10b650 0x7fba4c1029f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.075+0000 7fba5252b640 1 -- 172.21.15.90:0/117753333 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba4c10e040 msgr2=0x7fba4c102f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.075+0000 7fba5252b640 1 --2- 172.21.15.90:0/117753333 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba4c10e040 0x7fba4c102f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.075+0000 7fba5252b640 1 -- 172.21.15.90:0/117753333 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba4c1a89e0 con 0x7fba4c104bf0 2024-04-25T11:36:16.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.075+0000 7fba5252b640 1 --2- 172.21.15.90:0/117753333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba4c104bf0 0x7fba4c1024b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fba4800d950 tx=0x7fba4800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:16.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.075+0000 7fba437fe640 1 -- 172.21.15.90:0/117753333 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba48017030 con 0x7fba4c104bf0 2024-04-25T11:36:16.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.075+0000 7fba437fe640 1 -- 172.21.15.90:0/117753333 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba48004020 con 0x7fba4c104bf0 2024-04-25T11:36:16.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.075+0000 7fba5478f640 1 -- 172.21.15.90:0/117753333 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba4c1a8cd0 con 0x7fba4c104bf0 2024-04-25T11:36:16.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.075+0000 7fba437fe640 1 -- 172.21.15.90:0/117753333 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba480047a0 con 0x7fba4c104bf0 2024-04-25T11:36:16.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.075+0000 7fba5478f640 1 -- 172.21.15.90:0/117753333 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba4c1a91d0 con 0x7fba4c104bf0 2024-04-25T11:36:16.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.076+0000 7fba5478f640 1 -- 172.21.15.90:0/117753333 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba4c06a770 con 0x7fba4c104bf0 2024-04-25T11:36:16.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.079+0000 7fba437fe640 1 -- 172.21.15.90:0/117753333 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 24) ==== 147753+0+0 (secure 0 0 0) 0x7fba48009a30 con 0x7fba4c104bf0 2024-04-25T11:36:16.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.079+0000 7fba437fe640 1 --2- 172.21.15.90:0/117753333 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fba380afd80 0x7fba380b2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.079+0000 7fba51d2a640 1 --2- 172.21.15.90:0/117753333 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fba380afd80 0x7fba380b2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.079+0000 7fba437fe640 1 -- 172.21.15.90:0/117753333 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fba480df340 con 0x7fba4c104bf0 2024-04-25T11:36:16.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.080+0000 7fba51d2a640 1 --2- 172.21.15.90:0/117753333 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fba380afd80 0x7fba380b2290 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fba4c10e9d0 tx=0x7fba3c005f90 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:16.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.086+0000 7fba437fe640 1 -- 172.21.15.90:0/117753333 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fba480a7930 con 0x7fba4c104bf0 2024-04-25T11:36:16.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.366+0000 7fba5478f640 1 -- 172.21.15.90:0/117753333 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba4c06afa0 con 0x7fba4c104bf0 2024-04-25T11:36:16.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.367+0000 7fba437fe640 1 -- 172.21.15.90:0/117753333 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11930 (secure 0 0 0) 0x7fba480a7230 con 0x7fba4c104bf0 2024-04-25T11:36:16.355 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:36:16.356 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:35:06.229136+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24431,24458,24499,24524,24475]},"id":1}]} 2024-04-25T11:36:16.356 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 26 2024-04-25T11:36:16.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.369+0000 7fba5478f640 1 -- 172.21.15.90:0/117753333 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fba380afd80 msgr2=0x7fba380b2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.369+0000 7fba5478f640 1 --2- 172.21.15.90:0/117753333 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fba380afd80 0x7fba380b2290 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fba4c10e9d0 tx=0x7fba3c005f90 comp rx=0 tx=0).stop 2024-04-25T11:36:16.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.369+0000 7fba5478f640 1 -- 172.21.15.90:0/117753333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba4c104bf0 msgr2=0x7fba4c1024b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.369+0000 7fba5478f640 1 --2- 172.21.15.90:0/117753333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba4c104bf0 0x7fba4c1024b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fba4800d950 tx=0x7fba4800de10 comp rx=0 tx=0).stop 2024-04-25T11:36:16.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.369+0000 7fba5478f640 1 -- 172.21.15.90:0/117753333 shutdown_connections 2024-04-25T11:36:16.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.369+0000 7fba5478f640 1 --2- 172.21.15.90:0/117753333 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fba380afd80 0x7fba380b2290 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.369+0000 7fba5478f640 1 --2- 172.21.15.90:0/117753333 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba4c10e040 0x7fba4c102f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.369+0000 7fba5478f640 1 --2- 172.21.15.90:0/117753333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba4c10b650 0x7fba4c1029f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.369+0000 7fba5478f640 1 --2- 172.21.15.90:0/117753333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba4c104bf0 0x7fba4c1024b0 secure :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fba4800d950 tx=0x7fba4800de10 comp rx=0 tx=0).stop 2024-04-25T11:36:16.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.369+0000 7fba5478f640 1 -- 172.21.15.90:0/117753333 >> 172.21.15.90:0/117753333 conn(0x7fba4c078110 msgr2=0x7fba4c101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:16.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.370+0000 7fba5478f640 1 -- 172.21.15.90:0/117753333 shutdown_connections 2024-04-25T11:36:16.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.370+0000 7fba5478f640 1 -- 172.21.15.90:0/117753333 wait complete. 2024-04-25T11:36:16.374 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T11:36:16.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.524+0000 7fed0275f640 1 -- 172.21.15.90:0/2370598410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecfc102af0 msgr2=0x7fecfc102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.524+0000 7fed0275f640 1 --2- 172.21.15.90:0/2370598410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecfc102af0 0x7fecfc102f00 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fecf0009900 tx=0x7fecf002a410 comp rx=0 tx=0).stop 2024-04-25T11:36:16.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.524+0000 7fed0275f640 1 -- 172.21.15.90:0/2370598410 shutdown_connections 2024-04-25T11:36:16.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.524+0000 7fed0275f640 1 --2- 172.21.15.90:0/2370598410 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fecfc10d380 0x7fecfc10f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.524+0000 7fed0275f640 1 --2- 172.21.15.90:0/2370598410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecfc102af0 0x7fecfc102f00 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.524+0000 7fed0275f640 1 --2- 172.21.15.90:0/2370598410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fecfc108a20 0x7fecfc1025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.524+0000 7fed0275f640 1 -- 172.21.15.90:0/2370598410 >> 172.21.15.90:0/2370598410 conn(0x7fecfc078100 msgr2=0x7fecfc078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:16.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.524+0000 7fed0275f640 1 -- 172.21.15.90:0/2370598410 shutdown_connections 2024-04-25T11:36:16.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.525+0000 7fed0275f640 1 -- 172.21.15.90:0/2370598410 wait complete. 2024-04-25T11:36:16.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.525+0000 7fed0275f640 1 Processor -- start 2024-04-25T11:36:16.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.525+0000 7fed0275f640 1 -- start start 2024-04-25T11:36:16.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.526+0000 7fed0275f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecfc102af0 0x7fecfc199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.526+0000 7fed0275f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fecfc108a20 0x7fecfc199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.526+0000 7fed0275f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fecfc10d380 0x7fecfc19de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.526+0000 7fed0275f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fecfc112be0 con 0x7fecfc102af0 2024-04-25T11:36:16.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.526+0000 7fed0275f640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fecfc112a60 con 0x7fecfc108a20 2024-04-25T11:36:16.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.526+0000 7fed0275f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fecfc112d60 con 0x7fecfc10d380 2024-04-25T11:36:16.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.526+0000 7fecfbfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecfc102af0 0x7fecfc199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.526+0000 7fecfbfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecfc102af0 0x7fecfc199660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39920/0 (socket says 172.21.15.90:39920) 2024-04-25T11:36:16.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.526+0000 7fecfbfff640 1 -- 172.21.15.90:0/2349016882 learned_addr learned my addr 172.21.15.90:0/2349016882 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:16.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.526+0000 7fed00cfc640 1 --2- 172.21.15.90:0/2349016882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fecfc10d380 0x7fecfc19de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.526+0000 7fecfb7fe640 1 --2- 172.21.15.90:0/2349016882 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fecfc108a20 0x7fecfc199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.527+0000 7fecfbfff640 1 -- 172.21.15.90:0/2349016882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fecfc10d380 msgr2=0x7fecfc19de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.527+0000 7fecfbfff640 1 --2- 172.21.15.90:0/2349016882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fecfc10d380 0x7fecfc19de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.527+0000 7fecfbfff640 1 -- 172.21.15.90:0/2349016882 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fecfc108a20 msgr2=0x7fecfc199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.527+0000 7fecfbfff640 1 --2- 172.21.15.90:0/2349016882 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fecfc108a20 0x7fecfc199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.527+0000 7fecfbfff640 1 -- 172.21.15.90:0/2349016882 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecfc19e4c0 con 0x7fecfc102af0 2024-04-25T11:36:16.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.527+0000 7fecfb7fe640 1 --2- 172.21.15.90:0/2349016882 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fecfc108a20 0x7fecfc199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:36:16.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.527+0000 7fecfbfff640 1 --2- 172.21.15.90:0/2349016882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecfc102af0 0x7fecfc199660 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fecec00d740 tx=0x7fecec00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:16.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.527+0000 7fecf97fa640 1 -- 172.21.15.90:0/2349016882 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecec017070 con 0x7fecfc102af0 2024-04-25T11:36:16.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.527+0000 7fecf97fa640 1 -- 172.21.15.90:0/2349016882 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecec00de50 con 0x7fecfc102af0 2024-04-25T11:36:16.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.527+0000 7fed0275f640 1 -- 172.21.15.90:0/2349016882 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecfc19e7b0 con 0x7fecfc102af0 2024-04-25T11:36:16.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.527+0000 7fecf97fa640 1 -- 172.21.15.90:0/2349016882 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecec004ce0 con 0x7fecfc102af0 2024-04-25T11:36:16.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.527+0000 7fed0275f640 1 -- 172.21.15.90:0/2349016882 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecfc1a6130 con 0x7fecfc102af0 2024-04-25T11:36:16.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.530+0000 7fecf97fa640 1 -- 172.21.15.90:0/2349016882 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 24) ==== 147753+0+0 (secure 0 0 0) 0x7fecec004e80 con 0x7fecfc102af0 2024-04-25T11:36:16.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.531+0000 7fed0960c740 1 Processor -- start 2024-04-25T11:36:16.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.531+0000 7fed0960c740 1 -- start start 2024-04-25T11:36:16.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.531+0000 7fecf97fa640 1 --2- 172.21.15.90:0/2349016882 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fece40afed0 0x7fece40b23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.531+0000 7fed0960c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca4c390 0x562d3cded660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.531+0000 7fed0960c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d3ca234d0 0x562d3cdedba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.531+0000 7fed0960c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562d3ca2b6b0 0x562d3cdfc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.531+0000 7fed0960c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562d3cd52d80 con 0x562d3ca4c390 2024-04-25T11:36:16.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.531+0000 7fed0960c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x562d3cd0ca20 con 0x562d3ca2b6b0 2024-04-25T11:36:16.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.531+0000 7fed0960c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562d3cd4ed40 con 0x562d3ca234d0 2024-04-25T11:36:16.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.531+0000 7fed00cfc640 1 --2- 172.21.15.90:0/2349016882 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fece40afed0 0x7fece40b23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.531+0000 7fecf97fa640 1 -- 172.21.15.90:0/2349016882 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fecec0dec10 con 0x7fecfc102af0 2024-04-25T11:36:16.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.531+0000 7fecfbfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca4c390 0x562d3cded660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.531+0000 7fecfbfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca4c390 0x562d3cded660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39936/0 (socket says 172.21.15.90:39936) 2024-04-25T11:36:16.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.531+0000 7fecfbfff640 1 -- 172.21.15.90:0/3722294011 learned_addr learned my addr 172.21.15.90:0/3722294011 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:16.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.531+0000 7fed00cfc640 1 --2- 172.21.15.90:0/3722294011 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562d3ca2b6b0 0x562d3cdfc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.532+0000 7fecfb7fe640 1 --2- 172.21.15.90:0/3722294011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d3ca234d0 0x562d3cdedba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.532+0000 7fecfbfff640 1 -- 172.21.15.90:0/3722294011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d3ca234d0 msgr2=0x562d3cdedba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:16.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.532+0000 7fecfbfff640 1 --2- 172.21.15.90:0/3722294011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d3ca234d0 0x562d3cdedba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.532+0000 7fecfbfff640 1 -- 172.21.15.90:0/3722294011 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562d3ca2b6b0 msgr2=0x562d3cdfc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:16.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.532+0000 7fecfbfff640 1 --2- 172.21.15.90:0/3722294011 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562d3ca2b6b0 0x562d3cdfc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.532+0000 7fecfbfff640 1 -- 172.21.15.90:0/3722294011 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d3cd17c30 con 0x562d3ca4c390 2024-04-25T11:36:16.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.532+0000 7fed00cfc640 1 --2- 172.21.15.90:0/2349016882 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fece40afed0 0x7fece40b23e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fecf4009fd0 tx=0x7fecf4013b80 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:16.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.532+0000 7fecfbfff640 1 --2- 172.21.15.90:0/3722294011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca4c390 0x562d3cded660 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fecec009610 tx=0x7fecec0a63a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ba845203a0511ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:16.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.532+0000 7fece27fc640 1 -- 172.21.15.90:0/3722294011 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecec020070 con 0x562d3ca4c390 2024-04-25T11:36:16.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.532+0000 7fece27fc640 1 -- 172.21.15.90:0/3722294011 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecec0a6b00 con 0x562d3ca4c390 2024-04-25T11:36:16.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.532+0000 7fece27fc640 1 -- 172.21.15.90:0/3722294011 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecec0dc660 con 0x562d3ca4c390 2024-04-25T11:36:16.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.533+0000 7fed0960c740 1 -- 172.21.15.90:0/3722294011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca4c390 msgr2=0x562d3cded660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.533+0000 7fed0960c740 1 --2- 172.21.15.90:0/3722294011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca4c390 0x562d3cded660 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fecec009610 tx=0x7fecec0a63a0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.533+0000 7fed0960c740 1 -- 172.21.15.90:0/3722294011 shutdown_connections 2024-04-25T11:36:16.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.533+0000 7fed0960c740 1 --2- 172.21.15.90:0/3722294011 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562d3ca2b6b0 0x562d3cdfc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.533+0000 7fed0960c740 1 --2- 172.21.15.90:0/3722294011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d3ca234d0 0x562d3cdedba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.533+0000 7fed0960c740 1 --2- 172.21.15.90:0/3722294011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca4c390 0x562d3cded660 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.533+0000 7fed0960c740 1 -- 172.21.15.90:0/3722294011 >> 172.21.15.90:0/3722294011 conn(0x562d3ca404e0 msgr2=0x562d3cde9fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:16.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.533+0000 7fed0960c740 1 -- 172.21.15.90:0/3722294011 shutdown_connections 2024-04-25T11:36:16.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.533+0000 7fed0960c740 1 -- 172.21.15.90:0/3722294011 wait complete. 2024-04-25T11:36:16.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.534+0000 7fed0960c740 1 -- start start 2024-04-25T11:36:16.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.534+0000 7fed0960c740 1 Processor -- start 2024-04-25T11:36:16.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.535+0000 7fed0960c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d3ca234d0 0x562d3cdfa450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.535+0000 7fed0960c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562d3ca2b6b0 0x562d3cdfa990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.535+0000 7fed0960c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca4c390 0x562d3ce063f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.535+0000 7fed0960c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562d3cdfcf40 con 0x562d3ca4c390 2024-04-25T11:36:16.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.535+0000 7fed0960c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x562d3ccd9dc0 con 0x562d3ca2b6b0 2024-04-25T11:36:16.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.535+0000 7fed0960c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562d3cdfd0c0 con 0x562d3ca234d0 2024-04-25T11:36:16.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.535+0000 7fed0960c740 10 client.0 _ll_register_callbacks cb 0x562d3ccf5600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:16.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.535+0000 7fed00cfc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca4c390 0x562d3ce063f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.535+0000 7fed00cfc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca4c390 0x562d3ce063f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39946/0 (socket says 172.21.15.90:39946) 2024-04-25T11:36:16.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.535+0000 7fed00cfc640 1 -- 172.21.15.90:0/2618816781 learned_addr learned my addr 172.21.15.90:0/2618816781 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:16.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.535+0000 7fecfbfff640 1 --2- 172.21.15.90:0/2618816781 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d3ca234d0 0x562d3cdfa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.535+0000 7fecfb7fe640 1 --2- 172.21.15.90:0/2618816781 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562d3ca2b6b0 0x562d3cdfa990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.535+0000 7fed00cfc640 1 -- 172.21.15.90:0/2618816781 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d3ca234d0 msgr2=0x562d3cdfa450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:16.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.535+0000 7fed00cfc640 1 --2- 172.21.15.90:0/2618816781 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d3ca234d0 0x562d3cdfa450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.535+0000 7fed00cfc640 1 -- 172.21.15.90:0/2618816781 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562d3ca2b6b0 msgr2=0x562d3cdfa990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:16.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.535+0000 7fed00cfc640 1 --2- 172.21.15.90:0/2618816781 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562d3ca2b6b0 0x562d3cdfa990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.535+0000 7fed00cfc640 1 -- 172.21.15.90:0/2618816781 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d3ca643f0 con 0x562d3ca4c390 2024-04-25T11:36:16.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.535+0000 7fed00cfc640 1 --2- 172.21.15.90:0/2618816781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca4c390 0x562d3ce063f0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fecf40206b0 tx=0x7fecf4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecbcf0319d8a87c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:16.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.535+0000 7fecdb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:16.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.536+0000 7fecdb7fe640 1 -- 172.21.15.90:0/2618816781 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf4003970 con 0x562d3ca4c390 2024-04-25T11:36:16.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.536+0000 7fed0960c740 1 -- 172.21.15.90:0/2618816781 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecec0a6b00 con 0x562d3ca4c390 2024-04-25T11:36:16.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.536+0000 7fecdb7fe640 1 -- 172.21.15.90:0/2618816781 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fecf4003b10 con 0x562d3ca4c390 2024-04-25T11:36:16.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.536+0000 7fecdb7fe640 1 -- 172.21.15.90:0/2618816781 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fecf4003d10 con 0x562d3ca4c390 2024-04-25T11:36:16.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.536+0000 7fed0960c740 10 client.15858 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:16.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.536+0000 7fed0960c740 1 -- 172.21.15.90:0/2618816781 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562d3cb2e7d0 con 0x562d3ca4c390 2024-04-25T11:36:16.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.536+0000 7fecdb7fe640 1 -- 172.21.15.90:0/2618816781 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecec0a6b00 con 0x562d3ca4c390 2024-04-25T11:36:16.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.537+0000 7fecdb7fe640 1 -- 172.21.15.90:0/2618816781 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fecf401f070 con 0x562d3ca4c390 2024-04-25T11:36:16.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.537+0000 7fed0960c740 10 client.15858 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:16.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.537+0000 7fed0960c740 10 client.15858 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:36:16.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.537+0000 7fed0960c740 20 client.15858 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:16.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.537+0000 7fed0960c740 1 --2- 172.21.15.90:0/2618816781 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x562d3ca0af60 0x562d3ce0cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.537+0000 7fed0960c740 4 client.15858 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:36:16.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.537+0000 7fed0960c740 1 -- 172.21.15.90:0/2618816781 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d3cd60990 con 0x562d3ca0af60 2024-04-25T11:36:16.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.538+0000 7fecfbfff640 1 --2- 172.21.15.90:0/2618816781 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x562d3ca0af60 0x562d3ce0cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.539+0000 7fecdb7fe640 10 client.15858 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:36:16.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.539+0000 7fecfbfff640 1 --2- 172.21.15.90:0/2618816781 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x562d3ca0af60 0x562d3ce0cc10 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b355009c55e507e1 server_cookie=294d655d8b3ed331 in_seq=0 out_seq=0 2024-04-25T11:36:16.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.540+0000 7fecdb7fe640 1 -- 172.21.15.90:0/2618816781 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562d3cd60990 con 0x562d3ca0af60 2024-04-25T11:36:16.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.540+0000 7fecdb7fe640 10 client.15858 handle_command_reply: tid=0 2024-04-25T11:36:16.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.540+0000 7fed0960c740 1 client.15858 shutdown 2024-04-25T11:36:16.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.540+0000 7fed0960c740 1 -- 172.21.15.90:0/2618816781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca4c390 msgr2=0x562d3ce063f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.540+0000 7fed0960c740 1 --2- 172.21.15.90:0/2618816781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca4c390 0x562d3ce063f0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fecf40206b0 tx=0x7fecf4004ab0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.540+0000 7fed0960c740 1 -- 172.21.15.90:0/2618816781 shutdown_connections 2024-04-25T11:36:16.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.540+0000 7fed0960c740 1 --2- 172.21.15.90:0/2618816781 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x562d3ca0af60 0x562d3ce0cc10 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.540+0000 7fed0960c740 1 --2- 172.21.15.90:0/2618816781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca4c390 0x562d3ce063f0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.540+0000 7fed0960c740 1 --2- 172.21.15.90:0/2618816781 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562d3ca2b6b0 0x562d3cdfa990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.540+0000 7fed0960c740 1 --2- 172.21.15.90:0/2618816781 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d3ca234d0 0x562d3cdfa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.540+0000 7fed0960c740 1 -- 172.21.15.90:0/2618816781 >> 172.21.15.90:0/2618816781 conn(0x562d3ca404e0 msgr2=0x562d3cdef4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:16.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.540+0000 7fecdb7fe640 0 client.15858 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:36:16.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.540+0000 7fed0960c740 1 -- 172.21.15.90:0/2618816781 shutdown_connections 2024-04-25T11:36:16.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.540+0000 7fed0960c740 1 -- 172.21.15.90:0/2618816781 wait complete. 2024-04-25T11:36:16.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.540+0000 7fed0960c740 20 client.15858 trim_cache size 0 max 16384 2024-04-25T11:36:16.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.566+0000 7fed0960c740 1 Processor -- start 2024-04-25T11:36:16.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.566+0000 7fed0960c740 1 -- start start 2024-04-25T11:36:16.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.566+0000 7fed0960c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d3ca234d0 0x562d3ce05fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.566+0000 7fed0960c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca2b6b0 0x562d3ce12410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.566+0000 7fed0960c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562d3ca4c390 0x562d3ce12950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.567+0000 7fed0960c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fecfc1ad970 con 0x562d3ca2b6b0 2024-04-25T11:36:16.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.567+0000 7fed0960c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fecfc1af4b0 con 0x562d3ca4c390 2024-04-25T11:36:16.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.567+0000 7fed0960c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fecfc1ae040 con 0x562d3ca234d0 2024-04-25T11:36:16.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.567+0000 7fecfb7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca2b6b0 0x562d3ce12410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.567+0000 7fecfb7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca2b6b0 0x562d3ce12410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39962/0 (socket says 172.21.15.90:39962) 2024-04-25T11:36:16.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.567+0000 7fecfb7fe640 1 -- 172.21.15.90:0/2717868677 learned_addr learned my addr 172.21.15.90:0/2717868677 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:16.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.567+0000 7fecfbfff640 1 --2- 172.21.15.90:0/2717868677 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d3ca234d0 0x562d3ce05fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.567+0000 7fed00cfc640 1 --2- 172.21.15.90:0/2717868677 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562d3ca4c390 0x562d3ce12950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.567+0000 7fecfb7fe640 1 -- 172.21.15.90:0/2717868677 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d3ca234d0 msgr2=0x562d3ce05fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:16.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.567+0000 7fecfb7fe640 1 --2- 172.21.15.90:0/2717868677 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d3ca234d0 0x562d3ce05fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.567+0000 7fecfb7fe640 1 -- 172.21.15.90:0/2717868677 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562d3ca4c390 msgr2=0x562d3ce12950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:16.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.567+0000 7fecfb7fe640 1 --2- 172.21.15.90:0/2717868677 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562d3ca4c390 0x562d3ce12950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.567+0000 7fecfb7fe640 1 -- 172.21.15.90:0/2717868677 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d3ca0b190 con 0x562d3ca2b6b0 2024-04-25T11:36:16.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.567+0000 7fecfb7fe640 1 --2- 172.21.15.90:0/2717868677 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca2b6b0 0x562d3ce12410 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fecf0000c00 tx=0x7fecf0005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6adf003301ba207 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:16.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.567+0000 7fece1ffb640 1 -- 172.21.15.90:0/2717868677 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf0035050 con 0x562d3ca2b6b0 2024-04-25T11:36:16.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.568+0000 7fece1ffb640 1 -- 172.21.15.90:0/2717868677 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecf0004030 con 0x562d3ca2b6b0 2024-04-25T11:36:16.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.568+0000 7fece1ffb640 1 -- 172.21.15.90:0/2717868677 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf0005370 con 0x562d3ca2b6b0 2024-04-25T11:36:16.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.568+0000 7fed0960c740 1 -- 172.21.15.90:0/2717868677 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca2b6b0 msgr2=0x562d3ce12410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.568+0000 7fed0960c740 1 --2- 172.21.15.90:0/2717868677 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca2b6b0 0x562d3ce12410 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fecf0000c00 tx=0x7fecf0005f30 comp rx=0 tx=0).stop 2024-04-25T11:36:16.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.568+0000 7fed0960c740 1 -- 172.21.15.90:0/2717868677 shutdown_connections 2024-04-25T11:36:16.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.568+0000 7fed0960c740 1 --2- 172.21.15.90:0/2717868677 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562d3ca4c390 0x562d3ce12950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.568+0000 7fed0960c740 1 --2- 172.21.15.90:0/2717868677 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca2b6b0 0x562d3ce12410 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.568+0000 7fed0960c740 1 --2- 172.21.15.90:0/2717868677 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d3ca234d0 0x562d3ce05fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.568+0000 7fed0960c740 1 -- 172.21.15.90:0/2717868677 >> 172.21.15.90:0/2717868677 conn(0x562d3ca404e0 msgr2=0x562d3cdf8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:16.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.568+0000 7fed0960c740 1 -- 172.21.15.90:0/2717868677 shutdown_connections 2024-04-25T11:36:16.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.569+0000 7fed0960c740 1 -- 172.21.15.90:0/2717868677 wait complete. 2024-04-25T11:36:16.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.569+0000 7fed0960c740 1 -- start start 2024-04-25T11:36:16.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.569+0000 7fed0960c740 1 Processor -- start 2024-04-25T11:36:16.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.569+0000 7fed0960c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562d3ca234d0 0x562d3cdfd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.569+0000 7fed0960c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d3ca2b6b0 0x562d3cdfd850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.569+0000 7fed0960c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca4c390 0x562d3cdfdd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.569+0000 7fed0960c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fecfc1aede0 con 0x562d3ca4c390 2024-04-25T11:36:16.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.569+0000 7fed0960c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fecfc1ae710 con 0x562d3ca234d0 2024-04-25T11:36:16.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.569+0000 7fed0960c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fecfc1ab950 con 0x562d3ca2b6b0 2024-04-25T11:36:16.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.569+0000 7fed0960c740 10 client.0 _ll_register_callbacks cb 0x7fecf400fa10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:16.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.569+0000 7fed00cfc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca4c390 0x562d3cdfdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.569+0000 7fed00cfc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca4c390 0x562d3cdfdd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39972/0 (socket says 172.21.15.90:39972) 2024-04-25T11:36:16.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.570+0000 7fed00cfc640 1 -- 172.21.15.90:0/2365577207 learned_addr learned my addr 172.21.15.90:0/2365577207 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:16.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.570+0000 7fecfbfff640 1 --2- 172.21.15.90:0/2365577207 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562d3ca234d0 0x562d3cdfd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.570+0000 7fecfb7fe640 1 --2- 172.21.15.90:0/2365577207 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d3ca2b6b0 0x562d3cdfd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.570+0000 7fed00cfc640 1 -- 172.21.15.90:0/2365577207 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d3ca2b6b0 msgr2=0x562d3cdfd850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:16.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.570+0000 7fed00cfc640 1 --2- 172.21.15.90:0/2365577207 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d3ca2b6b0 0x562d3cdfd850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.570+0000 7fed00cfc640 1 -- 172.21.15.90:0/2365577207 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562d3ca234d0 msgr2=0x562d3cdfd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:16.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.570+0000 7fed00cfc640 1 --2- 172.21.15.90:0/2365577207 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562d3ca234d0 0x562d3cdfd310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.570+0000 7fed00cfc640 1 -- 172.21.15.90:0/2365577207 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d3ce72fb0 con 0x562d3ca4c390 2024-04-25T11:36:16.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.570+0000 7fed00cfc640 1 --2- 172.21.15.90:0/2365577207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca4c390 0x562d3cdfdd90 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x562d3cdfcdc0 tx=0x7fecf401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=492aee461e5803f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:16.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.570+0000 7fecdb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:16.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.570+0000 7fecdb7fe640 1 -- 172.21.15.90:0/2365577207 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf400fdd0 con 0x562d3ca4c390 2024-04-25T11:36:16.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.570+0000 7fed0960c740 1 -- 172.21.15.90:0/2365577207 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecf0004030 con 0x562d3ca4c390 2024-04-25T11:36:16.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.570+0000 7fecdb7fe640 1 -- 172.21.15.90:0/2365577207 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fecf4004b20 con 0x562d3ca4c390 2024-04-25T11:36:16.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.570+0000 7fecdb7fe640 1 -- 172.21.15.90:0/2365577207 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fecf4004d20 con 0x562d3ca4c390 2024-04-25T11:36:16.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.570+0000 7fed0960c740 10 client.15864 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:16.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.570+0000 7fed0960c740 1 -- 172.21.15.90:0/2365577207 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562d3ce74080 con 0x562d3ca4c390 2024-04-25T11:36:16.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.571+0000 7fecdb7fe640 1 -- 172.21.15.90:0/2365577207 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecf0004030 con 0x562d3ca4c390 2024-04-25T11:36:16.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.571+0000 7fecdb7fe640 1 -- 172.21.15.90:0/2365577207 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fecf4005850 con 0x562d3ca4c390 2024-04-25T11:36:16.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.571+0000 7fed0960c740 10 client.15864 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:16.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.571+0000 7fed0960c740 10 client.15864 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:36:16.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.571+0000 7fed0960c740 20 client.15864 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:16.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.571+0000 7fed0960c740 1 --2- 172.21.15.90:0/2365577207 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x562d3c9e03a0 0x562d3ce76830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.571+0000 7fed0960c740 4 client.15864 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:36:16.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.571+0000 7fed0960c740 1 -- 172.21.15.90:0/2365577207 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562d3cd5a800 con 0x562d3c9e03a0 2024-04-25T11:36:16.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.571+0000 7fecfbfff640 1 --2- 172.21.15.90:0/2365577207 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x562d3c9e03a0 0x562d3ce76830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.572+0000 7fecfbfff640 1 --2- 172.21.15.90:0/2365577207 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x562d3c9e03a0 0x562d3ce76830 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fde8dde80ee1f54b server_cookie=48793b8bfc852b68 in_seq=0 out_seq=0 2024-04-25T11:36:16.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.572+0000 7fecdb7fe640 10 client.15864 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:36:16.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.572+0000 7fecdb7fe640 1 -- 172.21.15.90:0/2365577207 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562d3cd5a800 con 0x562d3c9e03a0 2024-04-25T11:36:16.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.572+0000 7fecdb7fe640 10 client.15864 handle_command_reply: tid=0 2024-04-25T11:36:16.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.572+0000 7fed0960c740 1 client.15864 shutdown 2024-04-25T11:36:16.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.572+0000 7fed0960c740 1 -- 172.21.15.90:0/2365577207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca4c390 msgr2=0x562d3cdfdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.572+0000 7fed0960c740 1 --2- 172.21.15.90:0/2365577207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca4c390 0x562d3cdfdd90 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x562d3cdfcdc0 tx=0x7fecf401f000 comp rx=0 tx=0).stop 2024-04-25T11:36:16.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.572+0000 7fed0960c740 1 -- 172.21.15.90:0/2365577207 shutdown_connections 2024-04-25T11:36:16.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.572+0000 7fed0960c740 1 --2- 172.21.15.90:0/2365577207 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x562d3c9e03a0 0x562d3ce76830 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.572+0000 7fed0960c740 1 --2- 172.21.15.90:0/2365577207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562d3ca4c390 0x562d3cdfdd90 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.572+0000 7fed0960c740 1 --2- 172.21.15.90:0/2365577207 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d3ca2b6b0 0x562d3cdfd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.572+0000 7fecdb7fe640 0 client.15864 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:36:16.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.572+0000 7fed0960c740 1 --2- 172.21.15.90:0/2365577207 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562d3ca234d0 0x562d3cdfd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.572+0000 7fed0960c740 1 -- 172.21.15.90:0/2365577207 >> 172.21.15.90:0/2365577207 conn(0x562d3ca404e0 msgr2=0x562d3cdf8a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:16.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.572+0000 7fed0960c740 1 -- 172.21.15.90:0/2365577207 shutdown_connections 2024-04-25T11:36:16.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.572+0000 7fed0960c740 1 -- 172.21.15.90:0/2365577207 wait complete. 2024-04-25T11:36:16.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.572+0000 7fed0960c740 20 client.15864 trim_cache size 0 max 16384 2024-04-25T11:36:16.566 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:36:16.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.573+0000 7fed0275f640 1 -- 172.21.15.90:0/2349016882 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fece40afed0 msgr2=0x7fece40b23e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.573+0000 7fed0275f640 1 --2- 172.21.15.90:0/2349016882 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fece40afed0 0x7fece40b23e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fecf4009fd0 tx=0x7fecf4013b80 comp rx=0 tx=0).stop 2024-04-25T11:36:16.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.573+0000 7fed0275f640 1 -- 172.21.15.90:0/2349016882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecfc102af0 msgr2=0x7fecfc199660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.573+0000 7fed0275f640 1 --2- 172.21.15.90:0/2349016882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecfc102af0 0x7fecfc199660 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fecec00d740 tx=0x7fecec00dc00 comp rx=0 tx=0).stop 2024-04-25T11:36:16.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.573+0000 7fed0275f640 1 -- 172.21.15.90:0/2349016882 shutdown_connections 2024-04-25T11:36:16.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.573+0000 7fed0275f640 1 --2- 172.21.15.90:0/2349016882 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fece40afed0 0x7fece40b23e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.573+0000 7fed0275f640 1 --2- 172.21.15.90:0/2349016882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fecfc10d380 0x7fecfc19de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.573+0000 7fed0275f640 1 --2- 172.21.15.90:0/2349016882 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fecfc108a20 0x7fecfc199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.573+0000 7fed0275f640 1 --2- 172.21.15.90:0/2349016882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecfc102af0 0x7fecfc199660 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.573+0000 7fed0275f640 1 -- 172.21.15.90:0/2349016882 >> 172.21.15.90:0/2349016882 conn(0x7fecfc078100 msgr2=0x7fecfc1040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:16.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.573+0000 7fed0275f640 1 -- 172.21.15.90:0/2349016882 shutdown_connections 2024-04-25T11:36:16.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.573+0000 7fed0275f640 1 -- 172.21.15.90:0/2349016882 wait complete. 2024-04-25T11:36:16.573 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T11:36:16.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.708+0000 7fb8a4696640 1 -- 172.21.15.90:0/3695617575 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb89c103180 msgr2=0x7fb89c10f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.708+0000 7fb8a4696640 1 --2- 172.21.15.90:0/3695617575 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb89c103180 0x7fb89c10f4b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb894009be0 tx=0x7fb89402a6c0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.708+0000 7fb8a4696640 1 -- 172.21.15.90:0/3695617575 shutdown_connections 2024-04-25T11:36:16.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.708+0000 7fb8a4696640 1 --2- 172.21.15.90:0/3695617575 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb89c103180 0x7fb89c10f4b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.708+0000 7fb8a4696640 1 --2- 172.21.15.90:0/3695617575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb89c108ec0 0x7fb89c102b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.708+0000 7fb8a4696640 1 --2- 172.21.15.90:0/3695617575 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb89c108a20 0x7fb89c1025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.708+0000 7fb8a4696640 1 -- 172.21.15.90:0/3695617575 >> 172.21.15.90:0/3695617575 conn(0x7fb89c078110 msgr2=0x7fb89c078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:16.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.708+0000 7fb8a4696640 1 -- 172.21.15.90:0/3695617575 shutdown_connections 2024-04-25T11:36:16.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.708+0000 7fb8a4696640 1 -- 172.21.15.90:0/3695617575 wait complete. 2024-04-25T11:36:16.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.709+0000 7fb8a4696640 1 Processor -- start 2024-04-25T11:36:16.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.709+0000 7fb8a4696640 1 -- start start 2024-04-25T11:36:16.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.709+0000 7fb8a4696640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb89c103180 0x7fb89c19b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.709+0000 7fb8a4696640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb89c108a20 0x7fb89c19baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.709+0000 7fb8a4696640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb89c108ec0 0x7fb89c19c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.709+0000 7fb8a4696640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb89c112c60 con 0x7fb89c108a20 2024-04-25T11:36:16.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.709+0000 7fb8a4696640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb89c112ae0 con 0x7fb89c108ec0 2024-04-25T11:36:16.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.709+0000 7fb8a4696640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb89c112de0 con 0x7fb89c103180 2024-04-25T11:36:16.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.709+0000 7fb8a1c31640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb89c108a20 0x7fb89c19baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.709+0000 7fb8a1c31640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb89c108a20 0x7fb89c19baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39988/0 (socket says 172.21.15.90:39988) 2024-04-25T11:36:16.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.709+0000 7fb8a1c31640 1 -- 172.21.15.90:0/955804357 learned_addr learned my addr 172.21.15.90:0/955804357 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:16.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.710+0000 7fb8a2432640 1 --2- 172.21.15.90:0/955804357 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb89c103180 0x7fb89c19b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.710+0000 7fb8a2c33640 1 --2- 172.21.15.90:0/955804357 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb89c108ec0 0x7fb89c19c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.710+0000 7fb8a1c31640 1 -- 172.21.15.90:0/955804357 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb89c103180 msgr2=0x7fb89c19b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.710+0000 7fb8a1c31640 1 --2- 172.21.15.90:0/955804357 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb89c103180 0x7fb89c19b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.710+0000 7fb8a1c31640 1 -- 172.21.15.90:0/955804357 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb89c108ec0 msgr2=0x7fb89c19c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.710+0000 7fb8a1c31640 1 --2- 172.21.15.90:0/955804357 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb89c108ec0 0x7fb89c19c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.710+0000 7fb8a1c31640 1 -- 172.21.15.90:0/955804357 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb89c1a68a0 con 0x7fb89c108a20 2024-04-25T11:36:16.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.710+0000 7fb8a1c31640 1 --2- 172.21.15.90:0/955804357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb89c108a20 0x7fb89c19baa0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fb88c00e980 tx=0x7fb88c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:16.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.710+0000 7fb8937fe640 1 -- 172.21.15.90:0/955804357 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c00cd80 con 0x7fb89c108a20 2024-04-25T11:36:16.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.710+0000 7fb8a4696640 1 -- 172.21.15.90:0/955804357 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb89c1a6b90 con 0x7fb89c108a20 2024-04-25T11:36:16.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.710+0000 7fb8a4696640 1 -- 172.21.15.90:0/955804357 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb89c1a7100 con 0x7fb89c108a20 2024-04-25T11:36:16.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.710+0000 7fb8937fe640 1 -- 172.21.15.90:0/955804357 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb88c0108e0 con 0x7fb89c108a20 2024-04-25T11:36:16.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.710+0000 7fb8937fe640 1 -- 172.21.15.90:0/955804357 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c0188f0 con 0x7fb89c108a20 2024-04-25T11:36:16.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.714+0000 7fb8ab625740 1 Processor -- start 2024-04-25T11:36:16.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.714+0000 7fb8ab625740 1 -- start start 2024-04-25T11:36:16.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.714+0000 7fb8937fe640 1 -- 172.21.15.90:0/955804357 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 24) ==== 147753+0+0 (secure 0 0 0) 0x7fb88c01f020 con 0x7fb89c108a20 2024-04-25T11:36:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.715+0000 7fb8ab625740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5606e251c390 0x5606e28bd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.715+0000 7fb8ab625740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e24f34d0 0x5606e28bdd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.715+0000 7fb8ab625740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606e24fb6b0 0x5606e28ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.715+0000 7fb8ab625740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5606e2822d80 con 0x5606e24f34d0 2024-04-25T11:36:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.715+0000 7fb8ab625740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5606e27dca20 con 0x5606e251c390 2024-04-25T11:36:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.715+0000 7fb8ab625740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5606e281ed40 con 0x5606e24fb6b0 2024-04-25T11:36:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.715+0000 7fb8a2432640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e24f34d0 0x5606e28bdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.715+0000 7fb8a2432640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e24f34d0 0x5606e28bdd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39994/0 (socket says 172.21.15.90:39994) 2024-04-25T11:36:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.715+0000 7fb8a2432640 1 -- 172.21.15.90:0/2664379704 learned_addr learned my addr 172.21.15.90:0/2664379704 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.715+0000 7fb8a2c33640 1 --2- 172.21.15.90:0/2664379704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5606e251c390 0x5606e28bd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.715+0000 7fb8a1c31640 1 --2- 172.21.15.90:0/2664379704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606e24fb6b0 0x5606e28ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.715+0000 7fb8a2432640 1 -- 172.21.15.90:0/2664379704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606e24fb6b0 msgr2=0x5606e28ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:16.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.715+0000 7fb8a2432640 1 --2- 172.21.15.90:0/2664379704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606e24fb6b0 0x5606e28ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.715+0000 7fb8a2432640 1 -- 172.21.15.90:0/2664379704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5606e251c390 msgr2=0x5606e28bd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.715+0000 7fb8a2432640 1 --2- 172.21.15.90:0/2664379704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5606e251c390 0x5606e28bd810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.715+0000 7fb8a2432640 1 -- 172.21.15.90:0/2664379704 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606e27e7c30 con 0x5606e24f34d0 2024-04-25T11:36:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.715+0000 7fb8937fe640 1 --2- 172.21.15.90:0/955804357 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8880afd80 0x7fb8880b2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.715+0000 7fb8a2c33640 1 --2- 172.21.15.90:0/2664379704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5606e251c390 0x5606e28bd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:36:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.716+0000 7fb8a2c33640 1 --2- 172.21.15.90:0/955804357 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8880afd80 0x7fb8880b2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.716+0000 7fb8a2432640 1 --2- 172.21.15.90:0/2664379704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e24f34d0 0x5606e28bdd50 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fb898009c60 tx=0x7fb898011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d366b7b32bb51c13 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.716+0000 7fb8937fe640 1 -- 172.21.15.90:0/955804357 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb88c0df2a0 con 0x7fb89c108a20 2024-04-25T11:36:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.716+0000 7fb890bfe640 1 -- 172.21.15.90:0/2664379704 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb898004230 con 0x5606e24f34d0 2024-04-25T11:36:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.716+0000 7fb890bfe640 1 -- 172.21.15.90:0/2664379704 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8980043d0 con 0x5606e24f34d0 2024-04-25T11:36:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.716+0000 7fb890bfe640 1 -- 172.21.15.90:0/2664379704 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8980196d0 con 0x5606e24f34d0 2024-04-25T11:36:16.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.717+0000 7fb8a2c33640 1 --2- 172.21.15.90:0/955804357 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8880afd80 0x7fb8880b2290 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb894009370 tx=0x7fb894004a40 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:16.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.717+0000 7fb8ab625740 1 -- 172.21.15.90:0/2664379704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e24f34d0 msgr2=0x5606e28bdd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.717+0000 7fb8ab625740 1 --2- 172.21.15.90:0/2664379704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e24f34d0 0x5606e28bdd50 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fb898009c60 tx=0x7fb898011810 comp rx=0 tx=0).stop 2024-04-25T11:36:16.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.717+0000 7fb8ab625740 1 -- 172.21.15.90:0/2664379704 shutdown_connections 2024-04-25T11:36:16.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.717+0000 7fb8ab625740 1 --2- 172.21.15.90:0/2664379704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606e24fb6b0 0x5606e28ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.717+0000 7fb8ab625740 1 --2- 172.21.15.90:0/2664379704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e24f34d0 0x5606e28bdd50 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.717+0000 7fb8ab625740 1 --2- 172.21.15.90:0/2664379704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5606e251c390 0x5606e28bd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.717+0000 7fb8ab625740 1 -- 172.21.15.90:0/2664379704 >> 172.21.15.90:0/2664379704 conn(0x5606e25104e0 msgr2=0x5606e28b9fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:16.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.717+0000 7fb8ab625740 1 -- 172.21.15.90:0/2664379704 shutdown_connections 2024-04-25T11:36:16.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.717+0000 7fb8ab625740 1 -- 172.21.15.90:0/2664379704 wait complete. 2024-04-25T11:36:16.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.718+0000 7fb8ab625740 1 -- start start 2024-04-25T11:36:16.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.718+0000 7fb8ab625740 1 Processor -- start 2024-04-25T11:36:16.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.718+0000 7fb8ab625740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606e24f34d0 0x5606e28c7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.718+0000 7fb8ab625740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5606e24fb6b0 0x5606e28c8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.718+0000 7fb8ab625740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e251c390 0x5606e28d42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.718+0000 7fb8ab625740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5606e28cad90 con 0x5606e251c390 2024-04-25T11:36:16.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.718+0000 7fb8ab625740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5606e27a9dc0 con 0x5606e24fb6b0 2024-04-25T11:36:16.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.718+0000 7fb8ab625740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5606e28caf10 con 0x5606e24f34d0 2024-04-25T11:36:16.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.718+0000 7fb8ab625740 10 client.0 _ll_register_callbacks cb 0x5606e27c5600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:16.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.718+0000 7fb8a2c33640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e251c390 0x5606e28d42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.718+0000 7fb8a2c33640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e251c390 0x5606e28d42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39998/0 (socket says 172.21.15.90:39998) 2024-04-25T11:36:16.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.718+0000 7fb8a2c33640 1 -- 172.21.15.90:0/3058040978 learned_addr learned my addr 172.21.15.90:0/3058040978 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:16.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.718+0000 7fb8a2432640 1 --2- 172.21.15.90:0/3058040978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606e24f34d0 0x5606e28c7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.719+0000 7fb8a2c33640 1 -- 172.21.15.90:0/3058040978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606e24f34d0 msgr2=0x5606e28c7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:16.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.719+0000 7fb8a2c33640 1 --2- 172.21.15.90:0/3058040978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606e24f34d0 0x5606e28c7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.719+0000 7fb8a2c33640 1 -- 172.21.15.90:0/3058040978 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5606e24fb6b0 msgr2=0x5606e28c8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:16.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.719+0000 7fb8a2c33640 1 --2- 172.21.15.90:0/3058040978 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5606e24fb6b0 0x5606e28c8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.719+0000 7fb8a2c33640 1 -- 172.21.15.90:0/3058040978 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606e25343f0 con 0x5606e251c390 2024-04-25T11:36:16.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.719+0000 7fb8a2432640 1 --2- 172.21.15.90:0/3058040978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606e24f34d0 0x5606e28c7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:36:16.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.719+0000 7fb8a2c33640 1 --2- 172.21.15.90:0/3058040978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e251c390 0x5606e28d42d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fb894009420 tx=0x7fb89402a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd0b16d451e62607 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:16.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.719+0000 7fb885ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:16.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.719+0000 7fb885ffb640 1 -- 172.21.15.90:0/3058040978 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89404f020 con 0x5606e251c390 2024-04-25T11:36:16.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.719+0000 7fb885ffb640 1 -- 172.21.15.90:0/3058040978 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb894030b70 con 0x5606e251c390 2024-04-25T11:36:16.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.719+0000 7fb8ab625740 1 -- 172.21.15.90:0/3058040978 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8980043d0 con 0x5606e251c390 2024-04-25T11:36:16.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.719+0000 7fb885ffb640 1 -- 172.21.15.90:0/3058040978 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb89405a050 con 0x5606e251c390 2024-04-25T11:36:16.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.719+0000 7fb8ab625740 10 client.15876 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:16.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.719+0000 7fb8ab625740 1 -- 172.21.15.90:0/3058040978 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5606e25fe7d0 con 0x5606e251c390 2024-04-25T11:36:16.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.720+0000 7fb885ffb640 1 -- 172.21.15.90:0/3058040978 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8980043d0 con 0x5606e251c390 2024-04-25T11:36:16.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.720+0000 7fb885ffb640 1 -- 172.21.15.90:0/3058040978 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fb89404b480 con 0x5606e251c390 2024-04-25T11:36:16.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.720+0000 7fb8ab625740 10 client.15876 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:16.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.720+0000 7fb8ab625740 10 client.15876 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:36:16.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.720+0000 7fb8ab625740 20 client.15876 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:16.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.720+0000 7fb8ab625740 1 --2- 172.21.15.90:0/3058040978 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5606e24daf60 0x5606e28deb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.720+0000 7fb8ab625740 4 client.15876 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:36:16.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.720+0000 7fb8ab625740 1 -- 172.21.15.90:0/3058040978 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5606e2830990 con 0x5606e24daf60 2024-04-25T11:36:16.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.720+0000 7fb8a2432640 1 --2- 172.21.15.90:0/3058040978 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5606e24daf60 0x5606e28deb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.722+0000 7fb885ffb640 10 client.15876 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:36:16.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.722+0000 7fb8a2432640 1 --2- 172.21.15.90:0/3058040978 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5606e24daf60 0x5606e28deb60 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5f1845f768d3482f server_cookie=91569d7855a08ef5 in_seq=0 out_seq=0 2024-04-25T11:36:16.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.723+0000 7fb885ffb640 1 -- 172.21.15.90:0/3058040978 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5606e2830990 con 0x5606e24daf60 2024-04-25T11:36:16.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.723+0000 7fb885ffb640 10 client.15876 handle_command_reply: tid=0 2024-04-25T11:36:16.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.723+0000 7fb8ab625740 1 client.15876 shutdown 2024-04-25T11:36:16.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.723+0000 7fb8ab625740 1 -- 172.21.15.90:0/3058040978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e251c390 msgr2=0x5606e28d42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.723+0000 7fb8ab625740 1 --2- 172.21.15.90:0/3058040978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e251c390 0x5606e28d42d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fb894009420 tx=0x7fb89402a6c0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.723+0000 7fb8ab625740 1 -- 172.21.15.90:0/3058040978 shutdown_connections 2024-04-25T11:36:16.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.723+0000 7fb8ab625740 1 --2- 172.21.15.90:0/3058040978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e251c390 0x5606e28d42d0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.723+0000 7fb8ab625740 1 --2- 172.21.15.90:0/3058040978 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5606e24daf60 0x5606e28deb60 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.723+0000 7fb8ab625740 1 --2- 172.21.15.90:0/3058040978 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5606e24fb6b0 0x5606e28c8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.723+0000 7fb8ab625740 1 --2- 172.21.15.90:0/3058040978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606e24f34d0 0x5606e28c7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.723+0000 7fb8ab625740 1 -- 172.21.15.90:0/3058040978 >> 172.21.15.90:0/3058040978 conn(0x5606e25104e0 msgr2=0x5606e28bf530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:16.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.723+0000 7fb885ffb640 0 client.15876 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:36:16.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.723+0000 7fb8ab625740 1 -- 172.21.15.90:0/3058040978 shutdown_connections 2024-04-25T11:36:16.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.723+0000 7fb8ab625740 1 -- 172.21.15.90:0/3058040978 wait complete. 2024-04-25T11:36:16.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.723+0000 7fb8ab625740 20 client.15876 trim_cache size 0 max 16384 2024-04-25T11:36:16.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:16 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/117753333' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:36:16.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.737+0000 7fb8ab625740 1 Processor -- start 2024-04-25T11:36:16.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.737+0000 7fb8ab625740 1 -- start start 2024-04-25T11:36:16.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.738+0000 7fb8ab625740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e24f34d0 0x5606e28c3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.738+0000 7fb8ab625740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606e24fb6b0 0x5606e28c6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.738+0000 7fb8ab625740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5606e251c390 0x5606e28c2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.738+0000 7fb8ab625740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb88000e8a0 con 0x5606e24f34d0 2024-04-25T11:36:16.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.738+0000 7fb8ab625740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb8800103e0 con 0x5606e251c390 2024-04-25T11:36:16.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.738+0000 7fb8ab625740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb88000ef70 con 0x5606e24fb6b0 2024-04-25T11:36:16.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.738+0000 7fb8a2432640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e24f34d0 0x5606e28c3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.738+0000 7fb8a2432640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e24f34d0 0x5606e28c3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40012/0 (socket says 172.21.15.90:40012) 2024-04-25T11:36:16.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.738+0000 7fb8a2432640 1 -- 172.21.15.90:0/3006564413 learned_addr learned my addr 172.21.15.90:0/3006564413 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:16.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.738+0000 7fb8a2c33640 1 --2- 172.21.15.90:0/3006564413 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5606e251c390 0x5606e28c2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.738+0000 7fb8a1c31640 1 --2- 172.21.15.90:0/3006564413 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606e24fb6b0 0x5606e28c6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.738+0000 7fb8a2432640 1 -- 172.21.15.90:0/3006564413 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606e24fb6b0 msgr2=0x5606e28c6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:16.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.738+0000 7fb8a2432640 1 --2- 172.21.15.90:0/3006564413 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606e24fb6b0 0x5606e28c6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.738+0000 7fb8a2432640 1 -- 172.21.15.90:0/3006564413 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5606e251c390 msgr2=0x5606e28c2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:16.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.738+0000 7fb8a2432640 1 --2- 172.21.15.90:0/3006564413 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5606e251c390 0x5606e28c2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.738+0000 7fb8a2432640 1 -- 172.21.15.90:0/3006564413 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606e24db190 con 0x5606e24f34d0 2024-04-25T11:36:16.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.738+0000 7fb8a2432640 1 --2- 172.21.15.90:0/3006564413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e24f34d0 0x5606e28c3760 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fb898009c30 tx=0x7fb898005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=393888e67866441f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:16.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.739+0000 7fb887fff640 1 -- 172.21.15.90:0/3006564413 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb898031070 con 0x5606e24f34d0 2024-04-25T11:36:16.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.739+0000 7fb887fff640 1 -- 172.21.15.90:0/3006564413 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb898005d30 con 0x5606e24f34d0 2024-04-25T11:36:16.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.739+0000 7fb887fff640 1 -- 172.21.15.90:0/3006564413 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89801f040 con 0x5606e24f34d0 2024-04-25T11:36:16.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.740+0000 7fb8ab625740 1 -- 172.21.15.90:0/3006564413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e24f34d0 msgr2=0x5606e28c3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.740+0000 7fb8ab625740 1 --2- 172.21.15.90:0/3006564413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e24f34d0 0x5606e28c3760 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fb898009c30 tx=0x7fb898005850 comp rx=0 tx=0).stop 2024-04-25T11:36:16.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.740+0000 7fb8ab625740 1 -- 172.21.15.90:0/3006564413 shutdown_connections 2024-04-25T11:36:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.740+0000 7fb8ab625740 1 --2- 172.21.15.90:0/3006564413 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5606e251c390 0x5606e28c2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.740+0000 7fb8ab625740 1 --2- 172.21.15.90:0/3006564413 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606e24fb6b0 0x5606e28c6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.740+0000 7fb8ab625740 1 --2- 172.21.15.90:0/3006564413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e24f34d0 0x5606e28c3760 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.740+0000 7fb8ab625740 1 -- 172.21.15.90:0/3006564413 >> 172.21.15.90:0/3006564413 conn(0x5606e25104e0 msgr2=0x5606e28c77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.740+0000 7fb8ab625740 1 -- 172.21.15.90:0/3006564413 shutdown_connections 2024-04-25T11:36:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.740+0000 7fb8ab625740 1 -- 172.21.15.90:0/3006564413 wait complete. 2024-04-25T11:36:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.740+0000 7fb8ab625740 1 -- start start 2024-04-25T11:36:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.741+0000 7fb8ab625740 1 Processor -- start 2024-04-25T11:36:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.741+0000 7fb8ab625740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e24f34d0 0x5606e28d0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.741+0000 7fb8ab625740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5606e24fb6b0 0x5606e28cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.741+0000 7fb8ab625740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606e251c390 0x5606e28cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.741+0000 7fb8ab625740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb88000fd10 con 0x5606e24f34d0 2024-04-25T11:36:16.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.741+0000 7fb8ab625740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb88000f640 con 0x5606e24fb6b0 2024-04-25T11:36:16.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.741+0000 7fb8ab625740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb88000c880 con 0x5606e251c390 2024-04-25T11:36:16.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.741+0000 7fb8a2432640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e24f34d0 0x5606e28d0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.741+0000 7fb8a2432640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e24f34d0 0x5606e28d0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40026/0 (socket says 172.21.15.90:40026) 2024-04-25T11:36:16.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.741+0000 7fb8a2432640 1 -- 172.21.15.90:0/3593443248 learned_addr learned my addr 172.21.15.90:0/3593443248 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:16.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.741+0000 7fb8ab625740 10 client.0 _ll_register_callbacks cb 0x7fb894042070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:16.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.741+0000 7fb8a1c31640 1 --2- 172.21.15.90:0/3593443248 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5606e24fb6b0 0x5606e28cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.741+0000 7fb8a2c33640 1 --2- 172.21.15.90:0/3593443248 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606e251c390 0x5606e28cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.741+0000 7fb8a2432640 1 -- 172.21.15.90:0/3593443248 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606e251c390 msgr2=0x5606e28cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:16.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.741+0000 7fb8a2432640 1 --2- 172.21.15.90:0/3593443248 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606e251c390 0x5606e28cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.741+0000 7fb8a2432640 1 -- 172.21.15.90:0/3593443248 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5606e24fb6b0 msgr2=0x5606e28cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:16.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.741+0000 7fb8a2432640 1 --2- 172.21.15.90:0/3593443248 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5606e24fb6b0 0x5606e28cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.741+0000 7fb8a2432640 1 -- 172.21.15.90:0/3593443248 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606e28cbe30 con 0x5606e24f34d0 2024-04-25T11:36:16.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.742+0000 7fb8a2432640 1 --2- 172.21.15.90:0/3593443248 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e24f34d0 0x5606e28d0800 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fb898005c80 tx=0x7fb898005cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cb48641cce30496 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.742+0000 7fb885ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.742+0000 7fb885ffb640 1 -- 172.21.15.90:0/3593443248 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb898031070 con 0x5606e24f34d0 2024-04-25T11:36:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.742+0000 7fb885ffb640 1 -- 172.21.15.90:0/3593443248 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb898003b20 con 0x5606e24f34d0 2024-04-25T11:36:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.742+0000 7fb8ab625740 1 -- 172.21.15.90:0/3593443248 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb898005d30 con 0x5606e24f34d0 2024-04-25T11:36:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.742+0000 7fb885ffb640 1 -- 172.21.15.90:0/3593443248 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb898003d40 con 0x5606e24f34d0 2024-04-25T11:36:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.742+0000 7fb8ab625740 10 client.15882 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.742+0000 7fb8ab625740 1 -- 172.21.15.90:0/3593443248 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5606e2945d90 con 0x5606e24f34d0 2024-04-25T11:36:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.742+0000 7fb885ffb640 1 -- 172.21.15.90:0/3593443248 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb898005d30 con 0x5606e24f34d0 2024-04-25T11:36:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.742+0000 7fb885ffb640 1 -- 172.21.15.90:0/3593443248 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fb898018070 con 0x5606e24f34d0 2024-04-25T11:36:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.743+0000 7fb8ab625740 10 client.15882 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.743+0000 7fb8ab625740 10 client.15882 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:36:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.743+0000 7fb8ab625740 20 client.15882 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.743+0000 7fb8ab625740 1 --2- 172.21.15.90:0/3593443248 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5606e24b03a0 0x5606e2948210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.743+0000 7fb8ab625740 4 client.15882 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:36:16.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.743+0000 7fb8ab625740 1 -- 172.21.15.90:0/3593443248 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5606e282a800 con 0x5606e24b03a0 2024-04-25T11:36:16.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.743+0000 7fb8a1c31640 1 --2- 172.21.15.90:0/3593443248 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5606e24b03a0 0x5606e2948210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.743+0000 7fb885ffb640 10 client.15882 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:36:16.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.743+0000 7fb8a1c31640 1 --2- 172.21.15.90:0/3593443248 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5606e24b03a0 0x5606e2948210 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3bb0d66a3ed73ad0 server_cookie=e5b0524a9f17a6c6 in_seq=0 out_seq=0 2024-04-25T11:36:16.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.743+0000 7fb885ffb640 1 -- 172.21.15.90:0/3593443248 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5606e282a800 con 0x5606e24b03a0 2024-04-25T11:36:16.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.743+0000 7fb885ffb640 10 client.15882 handle_command_reply: tid=0 2024-04-25T11:36:16.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.743+0000 7fb8ab625740 1 client.15882 shutdown 2024-04-25T11:36:16.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.744+0000 7fb8ab625740 1 -- 172.21.15.90:0/3593443248 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e24f34d0 msgr2=0x5606e28d0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.744+0000 7fb8ab625740 1 --2- 172.21.15.90:0/3593443248 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e24f34d0 0x5606e28d0800 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fb898005c80 tx=0x7fb898005cb0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.744+0000 7fb8ab625740 1 -- 172.21.15.90:0/3593443248 shutdown_connections 2024-04-25T11:36:16.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.744+0000 7fb8ab625740 1 --2- 172.21.15.90:0/3593443248 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606e251c390 0x5606e28cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.744+0000 7fb8ab625740 1 --2- 172.21.15.90:0/3593443248 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5606e24b03a0 0x5606e2948210 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.744+0000 7fb8ab625740 1 --2- 172.21.15.90:0/3593443248 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5606e24fb6b0 0x5606e28cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.744+0000 7fb8ab625740 1 --2- 172.21.15.90:0/3593443248 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606e24f34d0 0x5606e28d0800 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.744+0000 7fb885ffb640 0 client.15882 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:36:16.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.744+0000 7fb8ab625740 1 -- 172.21.15.90:0/3593443248 >> 172.21.15.90:0/3593443248 conn(0x5606e25104e0 msgr2=0x5606e28b9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:16.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.744+0000 7fb8ab625740 1 -- 172.21.15.90:0/3593443248 shutdown_connections 2024-04-25T11:36:16.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.744+0000 7fb8ab625740 1 -- 172.21.15.90:0/3593443248 wait complete. 2024-04-25T11:36:16.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.744+0000 7fb8ab625740 20 client.15882 trim_cache size 0 max 16384 2024-04-25T11:36:16.738 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:36:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.744+0000 7fb8a4696640 1 -- 172.21.15.90:0/955804357 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8880afd80 msgr2=0x7fb8880b2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.744+0000 7fb8a4696640 1 --2- 172.21.15.90:0/955804357 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8880afd80 0x7fb8880b2290 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb894009370 tx=0x7fb894004a40 comp rx=0 tx=0).stop 2024-04-25T11:36:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.744+0000 7fb8a4696640 1 -- 172.21.15.90:0/955804357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb89c108a20 msgr2=0x7fb89c19baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.744+0000 7fb8a4696640 1 --2- 172.21.15.90:0/955804357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb89c108a20 0x7fb89c19baa0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fb88c00e980 tx=0x7fb88c00ee40 comp rx=0 tx=0).stop 2024-04-25T11:36:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.745+0000 7fb8a4696640 1 -- 172.21.15.90:0/955804357 shutdown_connections 2024-04-25T11:36:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.745+0000 7fb8a4696640 1 --2- 172.21.15.90:0/955804357 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8880afd80 0x7fb8880b2290 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.745+0000 7fb8a4696640 1 --2- 172.21.15.90:0/955804357 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb89c108ec0 0x7fb89c19c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.745+0000 7fb8a4696640 1 --2- 172.21.15.90:0/955804357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb89c108a20 0x7fb89c19baa0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.745+0000 7fb8a4696640 1 --2- 172.21.15.90:0/955804357 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb89c103180 0x7fb89c19b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.745+0000 7fb8a4696640 1 -- 172.21.15.90:0/955804357 >> 172.21.15.90:0/955804357 conn(0x7fb89c078110 msgr2=0x7fb89c100d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:16.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.745+0000 7fb8a4696640 1 -- 172.21.15.90:0/955804357 shutdown_connections 2024-04-25T11:36:16.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.745+0000 7fb8a4696640 1 -- 172.21.15.90:0/955804357 wait complete. 2024-04-25T11:36:16.746 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T11:36:16.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:16 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/117753333' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:36:16.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:16 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/117753333' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:36:16.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.900+0000 7fcfa7e34640 1 -- 172.21.15.90:0/16364545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa0108a40 msgr2=0x7fcfa01025d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.901+0000 7fcfa7e34640 1 --2- 172.21.15.90:0/16364545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa0108a40 0x7fcfa01025d0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c009900 tx=0x7fcf9c02a410 comp rx=0 tx=0).stop 2024-04-25T11:36:16.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.901+0000 7fcfa7e34640 1 -- 172.21.15.90:0/16364545 shutdown_connections 2024-04-25T11:36:16.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.901+0000 7fcfa7e34640 1 --2- 172.21.15.90:0/16364545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfa010d3a0 0x7fcfa010f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.901+0000 7fcfa7e34640 1 --2- 172.21.15.90:0/16364545 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcfa0102b10 0x7fcfa0102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.901+0000 7fcfa7e34640 1 --2- 172.21.15.90:0/16364545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa0108a40 0x7fcfa01025d0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.901+0000 7fcfa7e34640 1 -- 172.21.15.90:0/16364545 >> 172.21.15.90:0/16364545 conn(0x7fcfa0078180 msgr2=0x7fcfa00785c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:16.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.901+0000 7fcfa7e34640 1 -- 172.21.15.90:0/16364545 shutdown_connections 2024-04-25T11:36:16.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.901+0000 7fcfa7e34640 1 -- 172.21.15.90:0/16364545 wait complete. 2024-04-25T11:36:16.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.902+0000 7fcfa7e34640 1 Processor -- start 2024-04-25T11:36:16.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.902+0000 7fcfa7e34640 1 -- start start 2024-04-25T11:36:16.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.902+0000 7fcfa7e34640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa0102b10 0x7fcfa019b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.902+0000 7fcfa7e34640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfa0108a40 0x7fcfa019baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.902+0000 7fcfa7e34640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcfa010d3a0 0x7fcfa019bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.902+0000 7fcfa7e34640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa0112c00 con 0x7fcfa0102b10 2024-04-25T11:36:16.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.902+0000 7fcfa7e34640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa0112a80 con 0x7fcfa010d3a0 2024-04-25T11:36:16.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.902+0000 7fcfa7e34640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa0112d80 con 0x7fcfa0108a40 2024-04-25T11:36:16.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.902+0000 7fcfa5bd0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa0102b10 0x7fcfa019b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.902+0000 7fcfa5bd0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa0102b10 0x7fcfa019b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40048/0 (socket says 172.21.15.90:40048) 2024-04-25T11:36:16.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.902+0000 7fcfa5bd0640 1 -- 172.21.15.90:0/3691149891 learned_addr learned my addr 172.21.15.90:0/3691149891 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:16.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.902+0000 7fcfa53cf640 1 --2- 172.21.15.90:0/3691149891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfa0108a40 0x7fcfa019baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.902+0000 7fcfa63d1640 1 --2- 172.21.15.90:0/3691149891 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcfa010d3a0 0x7fcfa019bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.903+0000 7fcfa5bd0640 1 -- 172.21.15.90:0/3691149891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfa0108a40 msgr2=0x7fcfa019baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.903+0000 7fcfa5bd0640 1 --2- 172.21.15.90:0/3691149891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfa0108a40 0x7fcfa019baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.903+0000 7fcfa5bd0640 1 -- 172.21.15.90:0/3691149891 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcfa010d3a0 msgr2=0x7fcfa019bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.903+0000 7fcfa5bd0640 1 --2- 172.21.15.90:0/3691149891 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcfa010d3a0 0x7fcfa019bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.903+0000 7fcfa5bd0640 1 -- 172.21.15.90:0/3691149891 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa01a6870 con 0x7fcfa0102b10 2024-04-25T11:36:16.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.903+0000 7fcfa63d1640 1 --2- 172.21.15.90:0/3691149891 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcfa010d3a0 0x7fcfa019bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:36:16.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.903+0000 7fcfa5bd0640 1 --2- 172.21.15.90:0/3691149891 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa0102b10 0x7fcfa019b560 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c032fd0 tx=0x7fcf9c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:16.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.903+0000 7fcf96ffd640 1 -- 172.21.15.90:0/3691149891 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c03f030 con 0x7fcfa0102b10 2024-04-25T11:36:16.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.903+0000 7fcfa7e34640 1 -- 172.21.15.90:0/3691149891 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfa01a6b00 con 0x7fcfa0102b10 2024-04-25T11:36:16.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.903+0000 7fcf96ffd640 1 -- 172.21.15.90:0/3691149891 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf9c004490 con 0x7fcfa0102b10 2024-04-25T11:36:16.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.903+0000 7fcf96ffd640 1 -- 172.21.15.90:0/3691149891 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c035070 con 0x7fcfa0102b10 2024-04-25T11:36:16.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.903+0000 7fcfa7e34640 1 -- 172.21.15.90:0/3691149891 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfa01a7040 con 0x7fcfa0102b10 2024-04-25T11:36:16.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.906+0000 7fcfaedbf740 1 Processor -- start 2024-04-25T11:36:16.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.906+0000 7fcfaedbf740 1 -- start start 2024-04-25T11:36:16.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.907+0000 7fcfaedbf740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a79993390 0x560a79d34810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.907+0000 7fcfaedbf740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a7996a4d0 0x560a79d34d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.907+0000 7fcfaedbf740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a799726b0 0x560a79d416a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.907+0000 7fcfaedbf740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560a79c99d80 con 0x560a799726b0 2024-04-25T11:36:16.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.907+0000 7fcfaedbf740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x560a79c53a20 con 0x560a7996a4d0 2024-04-25T11:36:16.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.907+0000 7fcfaedbf740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560a79c95d40 con 0x560a79993390 2024-04-25T11:36:16.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.907+0000 7fcfa53cf640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a799726b0 0x560a79d416a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.907+0000 7fcfa63d1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a79993390 0x560a79d34810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.907+0000 7fcfa63d1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a79993390 0x560a79d34810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:41570/0 (socket says 172.21.15.90:41570) 2024-04-25T11:36:16.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.907+0000 7fcfa63d1640 1 -- 172.21.15.90:0/2865343531 learned_addr learned my addr 172.21.15.90:0/2865343531 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:16.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.907+0000 7fcfa53cf640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a799726b0 0x560a79d416a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40054/0 (socket says 172.21.15.90:40054) 2024-04-25T11:36:16.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.907+0000 7fcf96ffd640 1 -- 172.21.15.90:0/3691149891 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 24) ==== 147753+0+0 (secure 0 0 0) 0x7fcf9c042030 con 0x7fcfa0102b10 2024-04-25T11:36:16.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.907+0000 7fcfa53cf640 1 -- 172.21.15.90:0/2865343531 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a79993390 msgr2=0x560a79d34810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:16.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.907+0000 7fcfa53cf640 1 --2- 172.21.15.90:0/2865343531 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a79993390 0x560a79d34810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.907+0000 7fcfa53cf640 1 -- 172.21.15.90:0/2865343531 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a7996a4d0 msgr2=0x560a79d34d50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:36:16.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.907+0000 7fcfa53cf640 1 --2- 172.21.15.90:0/2865343531 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a7996a4d0 0x560a79d34d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.907+0000 7fcfa53cf640 1 -- 172.21.15.90:0/2865343531 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a79c5ec30 con 0x560a799726b0 2024-04-25T11:36:16.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.908+0000 7fcfa53cf640 1 --2- 172.21.15.90:0/2865343531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a799726b0 0x560a79d416a0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0075fd0 tx=0x7fcf900107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf9d72a91533054d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:16.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.908+0000 7fcf8bfff640 1 -- 172.21.15.90:0/2865343531 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf90004030 con 0x560a799726b0 2024-04-25T11:36:16.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.908+0000 7fcf8bfff640 1 -- 172.21.15.90:0/2865343531 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf90005020 con 0x560a799726b0 2024-04-25T11:36:16.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.908+0000 7fcf8bfff640 1 -- 172.21.15.90:0/2865343531 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf90004b80 con 0x560a799726b0 2024-04-25T11:36:16.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.908+0000 7fcf96ffd640 1 --2- 172.21.15.90:0/3691149891 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcf8c0afd80 0x7fcf8c0b2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.908+0000 7fcfa63d1640 1 --2- 172.21.15.90:0/3691149891 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcf8c0afd80 0x7fcf8c0b2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.908+0000 7fcf96ffd640 1 -- 172.21.15.90:0/3691149891 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fcf9c0ff7a0 con 0x7fcfa0102b10 2024-04-25T11:36:16.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.909+0000 7fcfaedbf740 1 -- 172.21.15.90:0/2865343531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a799726b0 msgr2=0x560a79d416a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.909+0000 7fcfa63d1640 1 --2- 172.21.15.90:0/3691149891 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcf8c0afd80 0x7fcf8c0b2290 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x560a79d31980 tx=0x7fcf980097e0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:16.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.909+0000 7fcfaedbf740 1 --2- 172.21.15.90:0/2865343531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a799726b0 0x560a79d416a0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0075fd0 tx=0x7fcf900107a0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.909+0000 7fcfaedbf740 1 -- 172.21.15.90:0/2865343531 shutdown_connections 2024-04-25T11:36:16.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.909+0000 7fcfaedbf740 1 --2- 172.21.15.90:0/2865343531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a799726b0 0x560a79d416a0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.909+0000 7fcfaedbf740 1 --2- 172.21.15.90:0/2865343531 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a7996a4d0 0x560a79d34d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.909+0000 7fcfaedbf740 1 --2- 172.21.15.90:0/2865343531 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a79993390 0x560a79d34810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.909+0000 7fcfaedbf740 1 -- 172.21.15.90:0/2865343531 >> 172.21.15.90:0/2865343531 conn(0x560a799874e0 msgr2=0x560a79d30fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:16.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.909+0000 7fcfaedbf740 1 -- 172.21.15.90:0/2865343531 shutdown_connections 2024-04-25T11:36:16.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.909+0000 7fcfaedbf740 1 -- 172.21.15.90:0/2865343531 wait complete. 2024-04-25T11:36:16.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.910+0000 7fcfaedbf740 1 -- start start 2024-04-25T11:36:16.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.910+0000 7fcfaedbf740 1 Processor -- start 2024-04-25T11:36:16.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.910+0000 7fcfaedbf740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a7996a4d0 0x560a79d3f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.910+0000 7fcfaedbf740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a799726b0 0x560a79d3f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.910+0000 7fcfaedbf740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a79993390 0x560a79d4f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.910+0000 7fcfaedbf740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560a79d461a0 con 0x560a7996a4d0 2024-04-25T11:36:16.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.910+0000 7fcfaedbf740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x560a79c20dc0 con 0x560a799726b0 2024-04-25T11:36:16.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.910+0000 7fcfaedbf740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560a79d46320 con 0x560a79993390 2024-04-25T11:36:16.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.910+0000 7fcfa5bd0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a7996a4d0 0x560a79d3f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.910+0000 7fcfaedbf740 10 client.0 _ll_register_callbacks cb 0x560a79c3c600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:16.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.911+0000 7fcfa5bd0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a7996a4d0 0x560a79d3f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40068/0 (socket says 172.21.15.90:40068) 2024-04-25T11:36:16.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.911+0000 7fcfa5bd0640 1 -- 172.21.15.90:0/1972008232 learned_addr learned my addr 172.21.15.90:0/1972008232 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:16.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.911+0000 7fcfa63d1640 1 --2- 172.21.15.90:0/1972008232 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a79993390 0x560a79d4f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.911+0000 7fcfa53cf640 1 --2- 172.21.15.90:0/1972008232 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a799726b0 0x560a79d3f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.911+0000 7fcfa5bd0640 1 -- 172.21.15.90:0/1972008232 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a79993390 msgr2=0x560a79d4f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:16.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.911+0000 7fcfa5bd0640 1 --2- 172.21.15.90:0/1972008232 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a79993390 0x560a79d4f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.911+0000 7fcfa5bd0640 1 -- 172.21.15.90:0/1972008232 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a799726b0 msgr2=0x560a79d3f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:16.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.911+0000 7fcfa5bd0640 1 --2- 172.21.15.90:0/1972008232 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a799726b0 0x560a79d3f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.911+0000 7fcfa5bd0640 1 -- 172.21.15.90:0/1972008232 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a799ab3f0 con 0x560a7996a4d0 2024-04-25T11:36:16.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.911+0000 7fcfa5bd0640 1 --2- 172.21.15.90:0/1972008232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a7996a4d0 0x560a79d3f1c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c0fd490 tx=0x7fcf9c0c75f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1edcd70cede23211 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:16.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.911+0000 7fcf897fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:16.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.911+0000 7fcf897fa640 1 -- 172.21.15.90:0/1972008232 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c049070 con 0x560a7996a4d0 2024-04-25T11:36:16.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.911+0000 7fcf897fa640 1 -- 172.21.15.90:0/1972008232 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fcf9c03b460 con 0x560a7996a4d0 2024-04-25T11:36:16.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.911+0000 7fcfaedbf740 1 -- 172.21.15.90:0/1972008232 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf90005020 con 0x560a7996a4d0 2024-04-25T11:36:16.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.911+0000 7fcf897fa640 1 -- 172.21.15.90:0/1972008232 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcf9c0c7e60 con 0x560a7996a4d0 2024-04-25T11:36:16.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.911+0000 7fcfaedbf740 10 client.15894 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:16.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.911+0000 7fcfaedbf740 1 -- 172.21.15.90:0/1972008232 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560a79a757d0 con 0x560a7996a4d0 2024-04-25T11:36:16.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.912+0000 7fcf897fa640 1 -- 172.21.15.90:0/1972008232 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf90005020 con 0x560a7996a4d0 2024-04-25T11:36:16.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.912+0000 7fcf897fa640 1 -- 172.21.15.90:0/1972008232 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fcf9c0b00b0 con 0x560a7996a4d0 2024-04-25T11:36:16.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.912+0000 7fcfaedbf740 10 client.15894 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:16.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.912+0000 7fcfaedbf740 10 client.15894 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:36:16.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.912+0000 7fcfaedbf740 20 client.15894 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:16.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.912+0000 7fcfaedbf740 1 --2- 172.21.15.90:0/1972008232 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x560a79951f60 0x560a79d55a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.912+0000 7fcfaedbf740 4 client.15894 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:36:16.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.912+0000 7fcfaedbf740 1 -- 172.21.15.90:0/1972008232 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a79ca7990 con 0x560a79951f60 2024-04-25T11:36:16.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.912+0000 7fcfa53cf640 1 --2- 172.21.15.90:0/1972008232 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x560a79951f60 0x560a79d55a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.913+0000 7fcf897fa640 10 client.15894 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:36:16.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.913+0000 7fcfa53cf640 1 --2- 172.21.15.90:0/1972008232 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x560a79951f60 0x560a79d55a20 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9dd72f913d8da202 server_cookie=181981c404c72907 in_seq=0 out_seq=0 2024-04-25T11:36:16.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.914+0000 7fcf897fa640 1 -- 172.21.15.90:0/1972008232 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560a79ca7990 con 0x560a79951f60 2024-04-25T11:36:16.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.914+0000 7fcf897fa640 10 client.15894 handle_command_reply: tid=0 2024-04-25T11:36:16.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.914+0000 7fcfaedbf740 1 client.15894 shutdown 2024-04-25T11:36:16.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.914+0000 7fcfaedbf740 1 -- 172.21.15.90:0/1972008232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a7996a4d0 msgr2=0x560a79d3f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.914+0000 7fcfaedbf740 1 --2- 172.21.15.90:0/1972008232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a7996a4d0 0x560a79d3f1c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c0fd490 tx=0x7fcf9c0c75f0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.914+0000 7fcfaedbf740 1 -- 172.21.15.90:0/1972008232 shutdown_connections 2024-04-25T11:36:16.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.914+0000 7fcfaedbf740 1 --2- 172.21.15.90:0/1972008232 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x560a79951f60 0x560a79d55a20 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.914+0000 7fcfaedbf740 1 --2- 172.21.15.90:0/1972008232 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a79993390 0x560a79d4f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.914+0000 7fcfaedbf740 1 --2- 172.21.15.90:0/1972008232 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a799726b0 0x560a79d3f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.914+0000 7fcfaedbf740 1 --2- 172.21.15.90:0/1972008232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a7996a4d0 0x560a79d3f1c0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.914+0000 7fcfaedbf740 1 -- 172.21.15.90:0/1972008232 >> 172.21.15.90:0/1972008232 conn(0x560a799874e0 msgr2=0x560a79d36530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:16.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.914+0000 7fcf897fa640 0 client.15894 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:36:16.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.914+0000 7fcfaedbf740 1 -- 172.21.15.90:0/1972008232 shutdown_connections 2024-04-25T11:36:16.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.914+0000 7fcfaedbf740 1 -- 172.21.15.90:0/1972008232 wait complete. 2024-04-25T11:36:16.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.914+0000 7fcfaedbf740 20 client.15894 trim_cache size 0 max 16384 2024-04-25T11:36:16.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.928+0000 7fcfaedbf740 1 Processor -- start 2024-04-25T11:36:16.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.928+0000 7fcfaedbf740 1 -- start start 2024-04-25T11:36:16.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.928+0000 7fcfaedbf740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a7996a4d0 0x560a79d3df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.928+0000 7fcfaedbf740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a799726b0 0x560a79d4f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.929+0000 7fcfaedbf740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a79993390 0x560a79d5b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.929+0000 7fcfaedbf740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa01ada00 con 0x560a79993390 2024-04-25T11:36:16.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.929+0000 7fcfaedbf740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa01af540 con 0x560a7996a4d0 2024-04-25T11:36:16.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.929+0000 7fcfaedbf740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa01ae0d0 con 0x560a799726b0 2024-04-25T11:36:16.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.929+0000 7fcfa63d1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a79993390 0x560a79d5b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.929+0000 7fcfa63d1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a79993390 0x560a79d5b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40080/0 (socket says 172.21.15.90:40080) 2024-04-25T11:36:16.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.929+0000 7fcfa63d1640 1 -- 172.21.15.90:0/2549761933 learned_addr learned my addr 172.21.15.90:0/2549761933 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:16.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.929+0000 7fcfa5bd0640 1 --2- 172.21.15.90:0/2549761933 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a7996a4d0 0x560a79d3df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.929+0000 7fcfa53cf640 1 --2- 172.21.15.90:0/2549761933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a799726b0 0x560a79d4f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.929+0000 7fcfa63d1640 1 -- 172.21.15.90:0/2549761933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a799726b0 msgr2=0x560a79d4f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:16.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.929+0000 7fcfa63d1640 1 --2- 172.21.15.90:0/2549761933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a799726b0 0x560a79d4f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.929+0000 7fcfa63d1640 1 -- 172.21.15.90:0/2549761933 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a7996a4d0 msgr2=0x560a79d3df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:16.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.929+0000 7fcfa63d1640 1 --2- 172.21.15.90:0/2549761933 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a7996a4d0 0x560a79d3df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.929+0000 7fcfa63d1640 1 -- 172.21.15.90:0/2549761933 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a79d30760 con 0x560a79993390 2024-04-25T11:36:16.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.929+0000 7fcfa53cf640 1 --2- 172.21.15.90:0/2549761933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a799726b0 0x560a79d4f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:36:16.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.929+0000 7fcfa63d1640 1 --2- 172.21.15.90:0/2549761933 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a79993390 0x560a79d5b1e0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fcf980116b0 tx=0x7fcf98016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea53af590eeefd39 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:16.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.929+0000 7fcf8b7fe640 1 -- 172.21.15.90:0/2549761933 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf98011e80 con 0x560a79993390 2024-04-25T11:36:16.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.930+0000 7fcf8b7fe640 1 -- 172.21.15.90:0/2549761933 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf98005020 con 0x560a79993390 2024-04-25T11:36:16.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.930+0000 7fcf8b7fe640 1 -- 172.21.15.90:0/2549761933 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9802a670 con 0x560a79993390 2024-04-25T11:36:16.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.931+0000 7fcfaedbf740 1 -- 172.21.15.90:0/2549761933 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a79993390 msgr2=0x560a79d5b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.931+0000 7fcfaedbf740 1 --2- 172.21.15.90:0/2549761933 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a79993390 0x560a79d5b1e0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fcf980116b0 tx=0x7fcf98016000 comp rx=0 tx=0).stop 2024-04-25T11:36:16.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.931+0000 7fcfaedbf740 1 -- 172.21.15.90:0/2549761933 shutdown_connections 2024-04-25T11:36:16.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.931+0000 7fcfaedbf740 1 --2- 172.21.15.90:0/2549761933 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a79993390 0x560a79d5b1e0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.931+0000 7fcfaedbf740 1 --2- 172.21.15.90:0/2549761933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a799726b0 0x560a79d4f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.931+0000 7fcfaedbf740 1 --2- 172.21.15.90:0/2549761933 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a7996a4d0 0x560a79d3df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.931+0000 7fcfaedbf740 1 -- 172.21.15.90:0/2549761933 >> 172.21.15.90:0/2549761933 conn(0x560a799874e0 msgr2=0x560a79d3d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:16.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.931+0000 7fcfaedbf740 1 -- 172.21.15.90:0/2549761933 shutdown_connections 2024-04-25T11:36:16.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.931+0000 7fcfaedbf740 1 -- 172.21.15.90:0/2549761933 wait complete. 2024-04-25T11:36:16.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.931+0000 7fcfaedbf740 1 -- start start 2024-04-25T11:36:16.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.931+0000 7fcfaedbf740 1 Processor -- start 2024-04-25T11:36:16.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.932+0000 7fcfaedbf740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a7996a4d0 0x560a79d46600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.932+0000 7fcfaedbf740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a799726b0 0x560a79d46b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.932+0000 7fcfaedbf740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a79993390 0x560a79dbbd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.932+0000 7fcfaedbf740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa01aee70 con 0x560a79993390 2024-04-25T11:36:16.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.932+0000 7fcfaedbf740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa01ae7a0 con 0x560a7996a4d0 2024-04-25T11:36:16.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.932+0000 7fcfaedbf740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa01ab9e0 con 0x560a799726b0 2024-04-25T11:36:16.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.932+0000 7fcfaedbf740 10 client.0 _ll_register_callbacks cb 0x7fcf9c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:16.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.932+0000 7fcfa63d1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a79993390 0x560a79dbbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.932+0000 7fcfa63d1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a79993390 0x560a79dbbd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40086/0 (socket says 172.21.15.90:40086) 2024-04-25T11:36:16.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.932+0000 7fcfa63d1640 1 -- 172.21.15.90:0/701413717 learned_addr learned my addr 172.21.15.90:0/701413717 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:16.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.932+0000 7fcfa53cf640 1 --2- 172.21.15.90:0/701413717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a799726b0 0x560a79d46b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.932+0000 7fcfa5bd0640 1 --2- 172.21.15.90:0/701413717 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a7996a4d0 0x560a79d46600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.932+0000 7fcfa63d1640 1 -- 172.21.15.90:0/701413717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a799726b0 msgr2=0x560a79d46b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:16.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.932+0000 7fcfa63d1640 1 --2- 172.21.15.90:0/701413717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a799726b0 0x560a79d46b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.932+0000 7fcfa63d1640 1 -- 172.21.15.90:0/701413717 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a7996a4d0 msgr2=0x560a79d46600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:16.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.932+0000 7fcfa63d1640 1 --2- 172.21.15.90:0/701413717 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a7996a4d0 0x560a79d46600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.932+0000 7fcfa63d1640 1 -- 172.21.15.90:0/701413717 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a79d473c0 con 0x560a79993390 2024-04-25T11:36:16.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.932+0000 7fcfa63d1640 1 --2- 172.21.15.90:0/701413717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a79993390 0x560a79dbbd80 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fcf98004ca0 tx=0x7fcf98004cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dce5816d4baf487 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:16.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.932+0000 7fcf897fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:16.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.933+0000 7fcf897fa640 1 -- 172.21.15.90:0/701413717 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf98011e80 con 0x560a79993390 2024-04-25T11:36:16.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.933+0000 7fcfaedbf740 1 -- 172.21.15.90:0/701413717 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf98005020 con 0x560a79993390 2024-04-25T11:36:16.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.933+0000 7fcf897fa640 1 -- 172.21.15.90:0/701413717 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fcf98026070 con 0x560a79993390 2024-04-25T11:36:16.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.933+0000 7fcf897fa640 1 -- 172.21.15.90:0/701413717 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcf980056e0 con 0x560a79993390 2024-04-25T11:36:16.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.933+0000 7fcfaedbf740 10 client.15900 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:16.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.933+0000 7fcfaedbf740 1 -- 172.21.15.90:0/701413717 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560a79dbd300 con 0x560a79993390 2024-04-25T11:36:16.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.933+0000 7fcf897fa640 1 -- 172.21.15.90:0/701413717 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf98005020 con 0x560a79993390 2024-04-25T11:36:16.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.933+0000 7fcf897fa640 1 -- 172.21.15.90:0/701413717 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fcf9800f040 con 0x560a79993390 2024-04-25T11:36:16.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.933+0000 7fcfaedbf740 10 client.15900 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:16.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.933+0000 7fcfaedbf740 10 client.15900 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:36:16.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.933+0000 7fcfaedbf740 20 client.15900 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:16.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.933+0000 7fcfaedbf740 1 --2- 172.21.15.90:0/701413717 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x560a799273a0 0x560a79dbfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:16.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.933+0000 7fcfaedbf740 4 client.15900 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:36:16.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.933+0000 7fcfaedbf740 1 -- 172.21.15.90:0/701413717 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560a79ca1800 con 0x560a799273a0 2024-04-25T11:36:16.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.934+0000 7fcfa5bd0640 1 --2- 172.21.15.90:0/701413717 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x560a799273a0 0x560a79dbfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:16.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.934+0000 7fcf897fa640 10 client.15900 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:36:16.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.934+0000 7fcfa5bd0640 1 --2- 172.21.15.90:0/701413717 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x560a799273a0 0x560a79dbfa40 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8f4eb112b15e0563 server_cookie=a0793ac06c1ded4b in_seq=0 out_seq=0 2024-04-25T11:36:16.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.935+0000 7fcf897fa640 1 -- 172.21.15.90:0/701413717 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560a79ca1800 con 0x560a799273a0 2024-04-25T11:36:16.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.935+0000 7fcf897fa640 10 client.15900 handle_command_reply: tid=0 2024-04-25T11:36:16.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.935+0000 7fcfaedbf740 1 client.15900 shutdown 2024-04-25T11:36:16.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.935+0000 7fcfaedbf740 1 -- 172.21.15.90:0/701413717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a79993390 msgr2=0x560a79dbbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.935+0000 7fcfaedbf740 1 --2- 172.21.15.90:0/701413717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a79993390 0x560a79dbbd80 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fcf98004ca0 tx=0x7fcf98004cd0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.935+0000 7fcfaedbf740 1 -- 172.21.15.90:0/701413717 shutdown_connections 2024-04-25T11:36:16.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.935+0000 7fcfaedbf740 1 --2- 172.21.15.90:0/701413717 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x560a799273a0 0x560a79dbfa40 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.935+0000 7fcfaedbf740 1 --2- 172.21.15.90:0/701413717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a79993390 0x560a79dbbd80 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.935+0000 7fcfaedbf740 1 --2- 172.21.15.90:0/701413717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a799726b0 0x560a79d46b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.935+0000 7fcfaedbf740 1 --2- 172.21.15.90:0/701413717 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a7996a4d0 0x560a79d46600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.935+0000 7fcf897fa640 0 client.15900 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:36:16.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.935+0000 7fcfaedbf740 1 -- 172.21.15.90:0/701413717 >> 172.21.15.90:0/701413717 conn(0x560a799874e0 msgr2=0x560a79d2f610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:16.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.935+0000 7fcfaedbf740 1 -- 172.21.15.90:0/701413717 shutdown_connections 2024-04-25T11:36:16.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.935+0000 7fcfaedbf740 1 -- 172.21.15.90:0/701413717 wait complete. 2024-04-25T11:36:16.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.935+0000 7fcfaedbf740 20 client.15900 trim_cache size 0 max 16384 2024-04-25T11:36:16.929 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:36:16.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.936+0000 7fcfa7e34640 1 -- 172.21.15.90:0/3691149891 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcf8c0afd80 msgr2=0x7fcf8c0b2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.936+0000 7fcfa7e34640 1 --2- 172.21.15.90:0/3691149891 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcf8c0afd80 0x7fcf8c0b2290 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x560a79d31980 tx=0x7fcf980097e0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.936+0000 7fcfa7e34640 1 -- 172.21.15.90:0/3691149891 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa0102b10 msgr2=0x7fcfa019b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:16.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.936+0000 7fcfa7e34640 1 --2- 172.21.15.90:0/3691149891 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa0102b10 0x7fcfa019b560 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c032fd0 tx=0x7fcf9c004270 comp rx=0 tx=0).stop 2024-04-25T11:36:16.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.936+0000 7fcfa7e34640 1 -- 172.21.15.90:0/3691149891 shutdown_connections 2024-04-25T11:36:16.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.936+0000 7fcfa7e34640 1 --2- 172.21.15.90:0/3691149891 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcf8c0afd80 0x7fcf8c0b2290 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.936+0000 7fcfa7e34640 1 --2- 172.21.15.90:0/3691149891 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcfa010d3a0 0x7fcfa019bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.936+0000 7fcfa7e34640 1 --2- 172.21.15.90:0/3691149891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfa0108a40 0x7fcfa019baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.936+0000 7fcfa7e34640 1 --2- 172.21.15.90:0/3691149891 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfa0102b10 0x7fcfa019b560 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:16.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.936+0000 7fcfa7e34640 1 -- 172.21.15.90:0/3691149891 >> 172.21.15.90:0/3691149891 conn(0x7fcfa0078180 msgr2=0x7fcfa01040d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:16.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.936+0000 7fcfa7e34640 1 -- 172.21.15.90:0/3691149891 shutdown_connections 2024-04-25T11:36:16.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:16.936+0000 7fcfa7e34640 1 -- 172.21.15.90:0/3691149891 wait complete. 2024-04-25T11:36:16.939 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T11:36:17.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.092+0000 7f3fac468640 1 -- 172.21.15.90:0/1585142120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fa410e040 msgr2=0x7f3fa4110530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:17.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.092+0000 7f3fac468640 1 --2- 172.21.15.90:0/1585142120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fa410e040 0x7f3fa4110530 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c009ca0 tx=0x7f3f9c02a8c0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.092+0000 7f3fac468640 1 -- 172.21.15.90:0/1585142120 shutdown_connections 2024-04-25T11:36:17.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.092+0000 7f3fac468640 1 --2- 172.21.15.90:0/1585142120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fa410e040 0x7f3fa4110530 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.092+0000 7f3fac468640 1 --2- 172.21.15.90:0/1585142120 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3fa410b650 0x7f3fa410da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.092+0000 7f3fac468640 1 --2- 172.21.15.90:0/1585142120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3fa4104bf0 0x7f3fa4105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.092+0000 7f3fac468640 1 -- 172.21.15.90:0/1585142120 >> 172.21.15.90:0/1585142120 conn(0x7f3fa4078100 msgr2=0x7f3fa4078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:17.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.092+0000 7f3fac468640 1 -- 172.21.15.90:0/1585142120 shutdown_connections 2024-04-25T11:36:17.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.092+0000 7f3fac468640 1 -- 172.21.15.90:0/1585142120 wait complete. 2024-04-25T11:36:17.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.093+0000 7f3fac468640 1 Processor -- start 2024-04-25T11:36:17.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.093+0000 7f3fac468640 1 -- start start 2024-04-25T11:36:17.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.094+0000 7f3fac468640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fa4104bf0 0x7f3fa419a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.094+0000 7f3fac468640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3fa410b650 0x7f3fa419ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.094+0000 7f3fac468640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3fa410e040 0x7f3fa419eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.094+0000 7f3fac468640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa4113ca0 con 0x7f3fa4104bf0 2024-04-25T11:36:17.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.094+0000 7f3fac468640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa4113b20 con 0x7f3fa410b650 2024-04-25T11:36:17.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.094+0000 7f3fac468640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa4113e20 con 0x7f3fa410e040 2024-04-25T11:36:17.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.094+0000 7f3faa204640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fa4104bf0 0x7f3fa419a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.094+0000 7f3faa204640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fa4104bf0 0x7f3fa419a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40102/0 (socket says 172.21.15.90:40102) 2024-04-25T11:36:17.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.094+0000 7f3faa204640 1 -- 172.21.15.90:0/400636003 learned_addr learned my addr 172.21.15.90:0/400636003 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:17.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.094+0000 7f3fa9a03640 1 --2- 172.21.15.90:0/400636003 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3fa410b650 0x7f3fa419ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.094+0000 7f3faaa05640 1 --2- 172.21.15.90:0/400636003 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3fa410e040 0x7f3fa419eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.094+0000 7f3faa204640 1 -- 172.21.15.90:0/400636003 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3fa410e040 msgr2=0x7f3fa419eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:17.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.094+0000 7f3faa204640 1 --2- 172.21.15.90:0/400636003 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3fa410e040 0x7f3fa419eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.094+0000 7f3faa204640 1 -- 172.21.15.90:0/400636003 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3fa410b650 msgr2=0x7f3fa419ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:17.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.094+0000 7f3faa204640 1 --2- 172.21.15.90:0/400636003 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3fa410b650 0x7f3fa419ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.094+0000 7f3faa204640 1 -- 172.21.15.90:0/400636003 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fa419f520 con 0x7f3fa4104bf0 2024-04-25T11:36:17.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.094+0000 7f3faaa05640 1 --2- 172.21.15.90:0/400636003 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3fa410e040 0x7f3fa419eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:36:17.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.094+0000 7f3faa204640 1 --2- 172.21.15.90:0/400636003 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fa4104bf0 0x7f3fa419a6c0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f3fa000e9f0 tx=0x7f3fa000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:17.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.095+0000 7f3f9b7fe640 1 -- 172.21.15.90:0/400636003 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa000cd80 con 0x7f3fa4104bf0 2024-04-25T11:36:17.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.095+0000 7f3f9b7fe640 1 -- 172.21.15.90:0/400636003 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fa00054d0 con 0x7f3fa4104bf0 2024-04-25T11:36:17.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.095+0000 7f3fac468640 1 -- 172.21.15.90:0/400636003 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fa419f810 con 0x7f3fa4104bf0 2024-04-25T11:36:17.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.095+0000 7f3f9b7fe640 1 -- 172.21.15.90:0/400636003 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa0010690 con 0x7f3fa4104bf0 2024-04-25T11:36:17.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.095+0000 7f3fac468640 1 -- 172.21.15.90:0/400636003 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fa419fd10 con 0x7f3fa4104bf0 2024-04-25T11:36:17.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.098+0000 7f3fb33f2740 1 Processor -- start 2024-04-25T11:36:17.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.098+0000 7f3fb33f2740 1 -- start start 2024-04-25T11:36:17.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.099+0000 7f3f9b7fe640 1 -- 172.21.15.90:0/400636003 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 24) ==== 147753+0+0 (secure 0 0 0) 0x7f3fa0010830 con 0x7f3fa4104bf0 2024-04-25T11:36:17.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.099+0000 7f3fb33f2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 0x55db413c8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.099+0000 7f3fb33f2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db40ffe4d0 0x55db413c8d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.099+0000 7f3fb33f2740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55db410066b0 0x55db413d56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.099+0000 7f3fb33f2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55db4132dd80 con 0x55db41027390 2024-04-25T11:36:17.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.099+0000 7f3fb33f2740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55db412e7a20 con 0x55db410066b0 2024-04-25T11:36:17.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.099+0000 7f3fb33f2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55db41329d40 con 0x55db40ffe4d0 2024-04-25T11:36:17.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.099+0000 7f3faaa05640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 0x55db413c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.099+0000 7f3faaa05640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 0x55db413c8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40116/0 (socket says 172.21.15.90:40116) 2024-04-25T11:36:17.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.099+0000 7f3faaa05640 1 -- 172.21.15.90:0/296501213 learned_addr learned my addr 172.21.15.90:0/296501213 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:17.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.099+0000 7f3faa204640 1 --2- 172.21.15.90:0/296501213 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db40ffe4d0 0x55db413c8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.099+0000 7f3fa9a03640 1 --2- 172.21.15.90:0/296501213 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55db410066b0 0x55db413d56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.099+0000 7f3faaa05640 1 -- 172.21.15.90:0/296501213 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db40ffe4d0 msgr2=0x55db413c8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:17.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.099+0000 7f3faaa05640 1 --2- 172.21.15.90:0/296501213 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db40ffe4d0 0x55db413c8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.099+0000 7f3faaa05640 1 -- 172.21.15.90:0/296501213 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55db410066b0 msgr2=0x55db413d56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:17.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.099+0000 7f3faaa05640 1 --2- 172.21.15.90:0/296501213 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55db410066b0 0x55db413d56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.099+0000 7f3faaa05640 1 -- 172.21.15.90:0/296501213 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db412f2c30 con 0x55db41027390 2024-04-25T11:36:17.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.099+0000 7f3fa9a03640 1 --2- 172.21.15.90:0/296501213 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55db410066b0 0x55db413d56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:36:17.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.100+0000 7f3faaa05640 1 --2- 172.21.15.90:0/296501213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 0x55db413c8810 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c006010 tx=0x7f3f9c009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5dfcaa877688c23 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:17.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.100+0000 7f3f9b7fe640 1 --2- 172.21.15.90:0/400636003 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3f900afd80 0x7f3f900b2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.100+0000 7f3f98bfe640 1 -- 172.21.15.90:0/296501213 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c031cf0 con 0x55db41027390 2024-04-25T11:36:17.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.100+0000 7f3f98bfe640 1 -- 172.21.15.90:0/296501213 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f9c036070 con 0x55db41027390 2024-04-25T11:36:17.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.100+0000 7f3f98bfe640 1 -- 172.21.15.90:0/296501213 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c03fa70 con 0x55db41027390 2024-04-25T11:36:17.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.100+0000 7f3f9b7fe640 1 -- 172.21.15.90:0/400636003 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f3fa00df840 con 0x7f3fa4104bf0 2024-04-25T11:36:17.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.100+0000 7f3faa204640 1 --2- 172.21.15.90:0/400636003 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3f900afd80 0x7f3f900b2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.101+0000 7f3fb33f2740 1 -- 172.21.15.90:0/296501213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 msgr2=0x55db413c8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:17.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.101+0000 7f3faa204640 1 --2- 172.21.15.90:0/400636003 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3f900afd80 0x7f3f900b2290 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3fa00dce10 tx=0x7f3fa00e5510 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:17.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.101+0000 7f3fb33f2740 1 --2- 172.21.15.90:0/296501213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 0x55db413c8810 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c006010 tx=0x7f3f9c009800 comp rx=0 tx=0).stop 2024-04-25T11:36:17.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.101+0000 7f3fb33f2740 1 -- 172.21.15.90:0/296501213 shutdown_connections 2024-04-25T11:36:17.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.101+0000 7f3fb33f2740 1 --2- 172.21.15.90:0/296501213 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55db410066b0 0x55db413d56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.101+0000 7f3fb33f2740 1 --2- 172.21.15.90:0/296501213 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db40ffe4d0 0x55db413c8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.101+0000 7f3fb33f2740 1 --2- 172.21.15.90:0/296501213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 0x55db413c8810 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.101+0000 7f3fb33f2740 1 -- 172.21.15.90:0/296501213 >> 172.21.15.90:0/296501213 conn(0x55db4101b4e0 msgr2=0x55db413c4fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:17.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.101+0000 7f3fb33f2740 1 -- 172.21.15.90:0/296501213 shutdown_connections 2024-04-25T11:36:17.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.102+0000 7f3fb33f2740 1 -- 172.21.15.90:0/296501213 wait complete. 2024-04-25T11:36:17.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.102+0000 7f3fb33f2740 1 -- start start 2024-04-25T11:36:17.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.102+0000 7f3fb33f2740 1 Processor -- start 2024-04-25T11:36:17.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.102+0000 7f3fb33f2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db40ffe4d0 0x55db413d3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.102+0000 7f3fb33f2740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55db410066b0 0x55db413d37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.102+0000 7f3fb33f2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 0x55db413df1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.102+0000 7f3fb33f2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55db413d5d30 con 0x55db41027390 2024-04-25T11:36:17.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.102+0000 7f3fb33f2740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55db412b4dc0 con 0x55db410066b0 2024-04-25T11:36:17.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.102+0000 7f3fb33f2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55db413d5eb0 con 0x55db40ffe4d0 2024-04-25T11:36:17.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.102+0000 7f3fb33f2740 10 client.0 _ll_register_callbacks cb 0x55db412d0600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:17.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.102+0000 7f3faaa05640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 0x55db413df1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.102+0000 7f3faaa05640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 0x55db413df1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40118/0 (socket says 172.21.15.90:40118) 2024-04-25T11:36:17.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.103+0000 7f3faaa05640 1 -- 172.21.15.90:0/3301737911 learned_addr learned my addr 172.21.15.90:0/3301737911 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:17.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.103+0000 7f3fa9a03640 1 --2- 172.21.15.90:0/3301737911 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55db410066b0 0x55db413d37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.103+0000 7f3faaa05640 1 --2- 172.21.15.90:0/3301737911 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db40ffe4d0 0x55db413d3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.103+0000 7f3faaa05640 1 -- 172.21.15.90:0/3301737911 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db40ffe4d0 msgr2=0x55db413d3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:17.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.103+0000 7f3faaa05640 1 --2- 172.21.15.90:0/3301737911 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db40ffe4d0 0x55db413d3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.103+0000 7f3faaa05640 1 -- 172.21.15.90:0/3301737911 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55db410066b0 msgr2=0x55db413d37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:17.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.103+0000 7f3faaa05640 1 --2- 172.21.15.90:0/3301737911 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55db410066b0 0x55db413d37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.103+0000 7f3faaa05640 1 -- 172.21.15.90:0/3301737911 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db4103f3f0 con 0x55db41027390 2024-04-25T11:36:17.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.103+0000 7f3fa9a03640 1 --2- 172.21.15.90:0/3301737911 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55db410066b0 0x55db413d37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:36:17.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.103+0000 7f3faaa05640 1 --2- 172.21.15.90:0/3301737911 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 0x55db413df1b0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c003fd0 tx=0x7f3f9c003d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1edd8ce8cb2657aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:17.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.103+0000 7f3f8dffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:17.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.103+0000 7f3f8dffb640 1 -- 172.21.15.90:0/3301737911 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c04d040 con 0x55db41027390 2024-04-25T11:36:17.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.103+0000 7f3f8dffb640 1 -- 172.21.15.90:0/3301737911 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f3f9c03fa10 con 0x55db41027390 2024-04-25T11:36:17.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.103+0000 7f3fb33f2740 1 -- 172.21.15.90:0/3301737911 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f9c036070 con 0x55db41027390 2024-04-25T11:36:17.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.104+0000 7f3f8dffb640 1 -- 172.21.15.90:0/3301737911 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3f9c004b40 con 0x55db41027390 2024-04-25T11:36:17.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.104+0000 7f3fb33f2740 10 client.15912 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:17.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.104+0000 7f3fb33f2740 1 -- 172.21.15.90:0/3301737911 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55db411097d0 con 0x55db41027390 2024-04-25T11:36:17.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.104+0000 7f3f8dffb640 1 -- 172.21.15.90:0/3301737911 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f9c036070 con 0x55db41027390 2024-04-25T11:36:17.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.104+0000 7f3f8dffb640 1 -- 172.21.15.90:0/3301737911 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f3f9c053070 con 0x55db41027390 2024-04-25T11:36:17.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.104+0000 7f3fb33f2740 10 client.15912 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:17.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.104+0000 7f3fb33f2740 10 client.15912 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:36:17.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.104+0000 7f3fb33f2740 20 client.15912 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:17.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.104+0000 7f3fb33f2740 1 --2- 172.21.15.90:0/3301737911 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55db40fe5f60 0x55db413e54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.104+0000 7f3fb33f2740 4 client.15912 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:36:17.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.104+0000 7f3fb33f2740 1 -- 172.21.15.90:0/3301737911 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db4133b990 con 0x55db40fe5f60 2024-04-25T11:36:17.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.105+0000 7f3fa9a03640 1 --2- 172.21.15.90:0/3301737911 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55db40fe5f60 0x55db413e54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.105+0000 7f3f8dffb640 10 client.15912 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:36:17.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.105+0000 7f3fa9a03640 1 --2- 172.21.15.90:0/3301737911 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55db40fe5f60 0x55db413e54a0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c4c52f88c855f148 server_cookie=35c1580179efc19 in_seq=0 out_seq=0 2024-04-25T11:36:17.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.106+0000 7f3f8dffb640 1 -- 172.21.15.90:0/3301737911 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55db4133b990 con 0x55db40fe5f60 2024-04-25T11:36:17.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.106+0000 7f3f8dffb640 10 client.15912 handle_command_reply: tid=0 2024-04-25T11:36:17.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.106+0000 7f3fb33f2740 1 client.15912 shutdown 2024-04-25T11:36:17.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.106+0000 7f3fb33f2740 1 -- 172.21.15.90:0/3301737911 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 msgr2=0x55db413df1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:17.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.106+0000 7f3fb33f2740 1 --2- 172.21.15.90:0/3301737911 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 0x55db413df1b0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c003fd0 tx=0x7f3f9c003d70 comp rx=0 tx=0).stop 2024-04-25T11:36:17.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.106+0000 7f3fb33f2740 1 -- 172.21.15.90:0/3301737911 shutdown_connections 2024-04-25T11:36:17.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.106+0000 7f3fb33f2740 1 --2- 172.21.15.90:0/3301737911 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55db40fe5f60 0x55db413e54a0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.106+0000 7f3fb33f2740 1 --2- 172.21.15.90:0/3301737911 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 0x55db413df1b0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.106+0000 7f3fb33f2740 1 --2- 172.21.15.90:0/3301737911 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55db410066b0 0x55db413d37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.106+0000 7f3fb33f2740 1 --2- 172.21.15.90:0/3301737911 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db40ffe4d0 0x55db413d3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.106+0000 7f3fb33f2740 1 -- 172.21.15.90:0/3301737911 >> 172.21.15.90:0/3301737911 conn(0x55db4101b4e0 msgr2=0x55db413ca530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:17.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.106+0000 7f3f8dffb640 0 client.15912 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:36:17.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.106+0000 7f3fb33f2740 1 -- 172.21.15.90:0/3301737911 shutdown_connections 2024-04-25T11:36:17.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.106+0000 7f3fb33f2740 1 -- 172.21.15.90:0/3301737911 wait complete. 2024-04-25T11:36:17.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.106+0000 7f3fb33f2740 20 client.15912 trim_cache size 0 max 16384 2024-04-25T11:36:17.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.120+0000 7f3fb33f2740 1 Processor -- start 2024-04-25T11:36:17.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.120+0000 7f3fb33f2740 1 -- start start 2024-04-25T11:36:17.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.120+0000 7f3fb33f2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db40ffe4d0 0x55db413df1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.120+0000 7f3fb33f2740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55db410066b0 0x55db413c4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.120+0000 7f3fb33f2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 0x55db413c86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.120+0000 7f3fb33f2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa41ada20 con 0x55db41027390 2024-04-25T11:36:17.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.120+0000 7f3fb33f2740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa41af560 con 0x55db410066b0 2024-04-25T11:36:17.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.120+0000 7f3fb33f2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa41ae0f0 con 0x55db40ffe4d0 2024-04-25T11:36:17.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.120+0000 7f3faaa05640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 0x55db413c86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.120+0000 7f3faaa05640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 0x55db413c86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40130/0 (socket says 172.21.15.90:40130) 2024-04-25T11:36:17.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.120+0000 7f3faaa05640 1 -- 172.21.15.90:0/3769851042 learned_addr learned my addr 172.21.15.90:0/3769851042 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:17.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.120+0000 7f3fa9a03640 1 --2- 172.21.15.90:0/3769851042 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55db410066b0 0x55db413c4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.120+0000 7f3faaa05640 1 -- 172.21.15.90:0/3769851042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db40ffe4d0 msgr2=0x55db413df1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:36:17.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.120+0000 7f3faaa05640 1 --2- 172.21.15.90:0/3769851042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db40ffe4d0 0x55db413df1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.120+0000 7f3faaa05640 1 -- 172.21.15.90:0/3769851042 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55db410066b0 msgr2=0x55db413c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:17.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.120+0000 7f3faaa05640 1 --2- 172.21.15.90:0/3769851042 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55db410066b0 0x55db413c4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.120+0000 7f3faaa05640 1 -- 172.21.15.90:0/3769851042 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db40fe6190 con 0x55db41027390 2024-04-25T11:36:17.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.121+0000 7f3faaa05640 1 --2- 172.21.15.90:0/3769851042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 0x55db413c86b0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c003ca0 tx=0x7f3f9c02bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5500b5e3ef719ee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:17.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.121+0000 7f3f8ffff640 1 -- 172.21.15.90:0/3769851042 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c004ce0 con 0x55db41027390 2024-04-25T11:36:17.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.121+0000 7f3f8ffff640 1 -- 172.21.15.90:0/3769851042 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f9c036070 con 0x55db41027390 2024-04-25T11:36:17.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.121+0000 7f3f8ffff640 1 -- 172.21.15.90:0/3769851042 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c0046d0 con 0x55db41027390 2024-04-25T11:36:17.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.122+0000 7f3fb33f2740 1 -- 172.21.15.90:0/3769851042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 msgr2=0x55db413c86b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:17.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.122+0000 7f3fb33f2740 1 --2- 172.21.15.90:0/3769851042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 0x55db413c86b0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c003ca0 tx=0x7f3f9c02bfd0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.122+0000 7f3fb33f2740 1 -- 172.21.15.90:0/3769851042 shutdown_connections 2024-04-25T11:36:17.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.122+0000 7f3fb33f2740 1 --2- 172.21.15.90:0/3769851042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 0x55db413c86b0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.122+0000 7f3fb33f2740 1 --2- 172.21.15.90:0/3769851042 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55db410066b0 0x55db413c4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.122+0000 7f3fb33f2740 1 --2- 172.21.15.90:0/3769851042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db40ffe4d0 0x55db413df1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.122+0000 7f3fb33f2740 1 -- 172.21.15.90:0/3769851042 >> 172.21.15.90:0/3769851042 conn(0x55db4101b4e0 msgr2=0x55db413c48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:17.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.122+0000 7f3fb33f2740 1 -- 172.21.15.90:0/3769851042 shutdown_connections 2024-04-25T11:36:17.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.122+0000 7f3fb33f2740 1 -- 172.21.15.90:0/3769851042 wait complete. 2024-04-25T11:36:17.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.122+0000 7f3fb33f2740 1 -- start start 2024-04-25T11:36:17.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.122+0000 7f3fb33f2740 1 Processor -- start 2024-04-25T11:36:17.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.122+0000 7f3fb33f2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db40ffe4d0 0x55db413e89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.123+0000 7f3fb33f2740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55db410066b0 0x55db413e4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.123+0000 7f3fb33f2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 0x55db413e5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.123+0000 7f3fb33f2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa41aee90 con 0x55db41027390 2024-04-25T11:36:17.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.123+0000 7f3fb33f2740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa41ae7c0 con 0x55db410066b0 2024-04-25T11:36:17.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.123+0000 7f3fb33f2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa41aba00 con 0x55db40ffe4d0 2024-04-25T11:36:17.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.123+0000 7f3fb33f2740 10 client.0 _ll_register_callbacks cb 0x7f3fa00e53e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:17.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.123+0000 7f3faaa05640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 0x55db413e5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.123+0000 7f3faaa05640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 0x55db413e5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40144/0 (socket says 172.21.15.90:40144) 2024-04-25T11:36:17.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.123+0000 7f3faaa05640 1 -- 172.21.15.90:0/799630278 learned_addr learned my addr 172.21.15.90:0/799630278 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:17.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.123+0000 7f3fa9a03640 1 --2- 172.21.15.90:0/799630278 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55db410066b0 0x55db413e4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.123+0000 7f3faaa05640 1 -- 172.21.15.90:0/799630278 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db40ffe4d0 msgr2=0x55db413e89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:17.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.123+0000 7f3faaa05640 1 --2- 172.21.15.90:0/799630278 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db40ffe4d0 0x55db413e89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.123+0000 7f3faaa05640 1 -- 172.21.15.90:0/799630278 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55db410066b0 msgr2=0x55db413e4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:17.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.123+0000 7f3faaa05640 1 --2- 172.21.15.90:0/799630278 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55db410066b0 0x55db413e4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.123+0000 7f3faaa05640 1 -- 172.21.15.90:0/799630278 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db413e5980 con 0x55db41027390 2024-04-25T11:36:17.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.123+0000 7f3fa9a03640 1 --2- 172.21.15.90:0/799630278 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55db410066b0 0x55db413e4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:36:17.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.123+0000 7f3faaa05640 1 --2- 172.21.15.90:0/799630278 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 0x55db413e5290 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c009bf0 tx=0x7f3f9c02f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0b552f7bde25aac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:17.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.123+0000 7f3f8dffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:17.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.123+0000 7f3f8dffb640 1 -- 172.21.15.90:0/799630278 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c062030 con 0x55db41027390 2024-04-25T11:36:17.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.124+0000 7f3f8dffb640 1 -- 172.21.15.90:0/799630278 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f3f9c031510 con 0x55db41027390 2024-04-25T11:36:17.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.124+0000 7f3fb33f2740 1 -- 172.21.15.90:0/799630278 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f9c036070 con 0x55db41027390 2024-04-25T11:36:17.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.124+0000 7f3f8dffb640 1 -- 172.21.15.90:0/799630278 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3f9c004040 con 0x55db41027390 2024-04-25T11:36:17.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.124+0000 7f3fb33f2740 10 client.15918 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:17.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.124+0000 7f3fb33f2740 1 -- 172.21.15.90:0/799630278 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55db41454c60 con 0x55db41027390 2024-04-25T11:36:17.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.124+0000 7f3f8dffb640 1 -- 172.21.15.90:0/799630278 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f9c036070 con 0x55db41027390 2024-04-25T11:36:17.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.125+0000 7f3f8dffb640 1 -- 172.21.15.90:0/799630278 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f3f9c053040 con 0x55db41027390 2024-04-25T11:36:17.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.125+0000 7f3fb33f2740 10 client.15918 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:17.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.125+0000 7f3fb33f2740 10 client.15918 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:36:17.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.125+0000 7f3fb33f2740 20 client.15918 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:17.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.125+0000 7f3fb33f2740 1 --2- 172.21.15.90:0/799630278 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55db40fbb3a0 0x55db414572a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.125+0000 7f3fb33f2740 4 client.15918 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:36:17.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.125+0000 7f3fb33f2740 1 -- 172.21.15.90:0/799630278 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55db41335800 con 0x55db40fbb3a0 2024-04-25T11:36:17.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.125+0000 7f3fa9a03640 1 --2- 172.21.15.90:0/799630278 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55db40fbb3a0 0x55db414572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.125+0000 7f3f8dffb640 10 client.15918 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:36:17.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.125+0000 7f3fa9a03640 1 --2- 172.21.15.90:0/799630278 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55db40fbb3a0 0x55db414572a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a9ed2037f60ab3ae server_cookie=8e5f23cbd6d45c4 in_seq=0 out_seq=0 2024-04-25T11:36:17.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.126+0000 7f3f8dffb640 1 -- 172.21.15.90:0/799630278 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55db41335800 con 0x55db40fbb3a0 2024-04-25T11:36:17.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.126+0000 7f3f8dffb640 10 client.15918 handle_command_reply: tid=0 2024-04-25T11:36:17.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.126+0000 7f3fb33f2740 1 client.15918 shutdown 2024-04-25T11:36:17.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.126+0000 7f3fb33f2740 1 -- 172.21.15.90:0/799630278 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 msgr2=0x55db413e5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:17.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.126+0000 7f3fb33f2740 1 --2- 172.21.15.90:0/799630278 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 0x55db413e5290 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c009bf0 tx=0x7f3f9c02f3c0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.126+0000 7f3fb33f2740 1 -- 172.21.15.90:0/799630278 shutdown_connections 2024-04-25T11:36:17.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.126+0000 7f3fb33f2740 1 --2- 172.21.15.90:0/799630278 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55db40fbb3a0 0x55db414572a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.126+0000 7f3fb33f2740 1 --2- 172.21.15.90:0/799630278 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db41027390 0x55db413e5290 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.126+0000 7f3fb33f2740 1 --2- 172.21.15.90:0/799630278 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55db410066b0 0x55db413e4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.126+0000 7f3fb33f2740 1 --2- 172.21.15.90:0/799630278 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db40ffe4d0 0x55db413e89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.126+0000 7f3fb33f2740 1 -- 172.21.15.90:0/799630278 >> 172.21.15.90:0/799630278 conn(0x55db4101b4e0 msgr2=0x55db413d1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:17.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.126+0000 7f3f8dffb640 0 client.15918 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:36:17.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.126+0000 7f3fb33f2740 1 -- 172.21.15.90:0/799630278 shutdown_connections 2024-04-25T11:36:17.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.126+0000 7f3fb33f2740 1 -- 172.21.15.90:0/799630278 wait complete. 2024-04-25T11:36:17.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.126+0000 7f3fb33f2740 20 client.15918 trim_cache size 0 max 16384 2024-04-25T11:36:17.120 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:36:17.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.127+0000 7f3fac468640 1 -- 172.21.15.90:0/400636003 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3f900afd80 msgr2=0x7f3f900b2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:17.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.127+0000 7f3fac468640 1 --2- 172.21.15.90:0/400636003 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3f900afd80 0x7f3f900b2290 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3fa00dce10 tx=0x7f3fa00e5510 comp rx=0 tx=0).stop 2024-04-25T11:36:17.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.127+0000 7f3fac468640 1 -- 172.21.15.90:0/400636003 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fa4104bf0 msgr2=0x7f3fa419a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:17.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.127+0000 7f3fac468640 1 --2- 172.21.15.90:0/400636003 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fa4104bf0 0x7f3fa419a6c0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f3fa000e9f0 tx=0x7f3fa000edf0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.127+0000 7f3fac468640 1 -- 172.21.15.90:0/400636003 shutdown_connections 2024-04-25T11:36:17.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.127+0000 7f3fac468640 1 --2- 172.21.15.90:0/400636003 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3f900afd80 0x7f3f900b2290 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.127+0000 7f3fac468640 1 --2- 172.21.15.90:0/400636003 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3fa410e040 0x7f3fa419eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.127+0000 7f3fac468640 1 --2- 172.21.15.90:0/400636003 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3fa410b650 0x7f3fa419ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.127+0000 7f3fac468640 1 --2- 172.21.15.90:0/400636003 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fa4104bf0 0x7f3fa419a6c0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.127+0000 7f3fac468640 1 -- 172.21.15.90:0/400636003 >> 172.21.15.90:0/400636003 conn(0x7f3fa4078100 msgr2=0x7f3fa4101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:17.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.127+0000 7f3fac468640 1 -- 172.21.15.90:0/400636003 shutdown_connections 2024-04-25T11:36:17.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.127+0000 7f3fac468640 1 -- 172.21.15.90:0/400636003 wait complete. 2024-04-25T11:36:17.127 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T11:36:17.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.249+0000 7fa1769c5640 1 -- 172.21.15.90:0/607557071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa170104bf0 msgr2=0x7fa170105000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:17.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.249+0000 7fa1769c5640 1 --2- 172.21.15.90:0/607557071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa170104bf0 0x7fa170105000 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fa160009900 tx=0x7fa16002a410 comp rx=0 tx=0).stop 2024-04-25T11:36:17.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.249+0000 7fa1769c5640 1 -- 172.21.15.90:0/607557071 shutdown_connections 2024-04-25T11:36:17.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.249+0000 7fa1769c5640 1 --2- 172.21.15.90:0/607557071 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa17010e040 0x7fa170110530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.249+0000 7fa1769c5640 1 --2- 172.21.15.90:0/607557071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa17010b650 0x7fa17010da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.249+0000 7fa1769c5640 1 --2- 172.21.15.90:0/607557071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa170104bf0 0x7fa170105000 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.249+0000 7fa1769c5640 1 -- 172.21.15.90:0/607557071 >> 172.21.15.90:0/607557071 conn(0x7fa170078100 msgr2=0x7fa170078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:17.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.249+0000 7fa1769c5640 1 -- 172.21.15.90:0/607557071 shutdown_connections 2024-04-25T11:36:17.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.249+0000 7fa1769c5640 1 -- 172.21.15.90:0/607557071 wait complete. 2024-04-25T11:36:17.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.250+0000 7fa1769c5640 1 Processor -- start 2024-04-25T11:36:17.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.250+0000 7fa1769c5640 1 -- start start 2024-04-25T11:36:17.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.250+0000 7fa1769c5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa170104bf0 0x7fa17019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.250+0000 7fa1769c5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa17010b650 0x7fa17019ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.251+0000 7fa1769c5640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa17010e040 0x7fa17019ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.251+0000 7fa1769c5640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa170113ca0 con 0x7fa17010b650 2024-04-25T11:36:17.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.251+0000 7fa1769c5640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa170113b20 con 0x7fa17010e040 2024-04-25T11:36:17.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.251+0000 7fa1769c5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa170113e20 con 0x7fa170104bf0 2024-04-25T11:36:17.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.251+0000 7fa16f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa17010b650 0x7fa17019ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.251+0000 7fa16ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa170104bf0 0x7fa17019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.251+0000 7fa16f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa17010b650 0x7fa17019ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40174/0 (socket says 172.21.15.90:40174) 2024-04-25T11:36:17.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.251+0000 7fa16f7fe640 1 -- 172.21.15.90:0/1254941732 learned_addr learned my addr 172.21.15.90:0/1254941732 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:17.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.251+0000 7fa174f62640 1 --2- 172.21.15.90:0/1254941732 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa17010e040 0x7fa17019ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.251+0000 7fa16f7fe640 1 -- 172.21.15.90:0/1254941732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa170104bf0 msgr2=0x7fa17019a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:17.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.251+0000 7fa16f7fe640 1 --2- 172.21.15.90:0/1254941732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa170104bf0 0x7fa17019a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.251+0000 7fa16f7fe640 1 -- 172.21.15.90:0/1254941732 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa17010e040 msgr2=0x7fa17019ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:17.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.251+0000 7fa16f7fe640 1 --2- 172.21.15.90:0/1254941732 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa17010e040 0x7fa17019ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.251+0000 7fa16f7fe640 1 -- 172.21.15.90:0/1254941732 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa17019f580 con 0x7fa17010b650 2024-04-25T11:36:17.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.251+0000 7fa16f7fe640 1 --2- 172.21.15.90:0/1254941732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa17010b650 0x7fa17019ac80 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa16400d770 tx=0x7fa16400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:17.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.252+0000 7fa16d7fa640 1 -- 172.21.15.90:0/1254941732 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa164017070 con 0x7fa17010b650 2024-04-25T11:36:17.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.252+0000 7fa1769c5640 1 -- 172.21.15.90:0/1254941732 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa17019f870 con 0x7fa17010b650 2024-04-25T11:36:17.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.252+0000 7fa16d7fa640 1 -- 172.21.15.90:0/1254941732 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1640054d0 con 0x7fa17010b650 2024-04-25T11:36:17.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.252+0000 7fa16d7fa640 1 -- 172.21.15.90:0/1254941732 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa164004c30 con 0x7fa17010b650 2024-04-25T11:36:17.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.252+0000 7fa1769c5640 1 -- 172.21.15.90:0/1254941732 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1701023e0 con 0x7fa17010b650 2024-04-25T11:36:17.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.255+0000 7fa17d87b740 1 Processor -- start 2024-04-25T11:36:17.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.255+0000 7fa17d87b740 1 -- start start 2024-04-25T11:36:17.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.255+0000 7fa16d7fa640 1 -- 172.21.15.90:0/1254941732 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 24) ==== 147753+0+0 (secure 0 0 0) 0x7fa164004e10 con 0x7fa17010b650 2024-04-25T11:36:17.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.255+0000 7fa17d87b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c77641b390 0x55c7767bc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.255+0000 7fa17d87b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7763f24d0 0x55c7767bcd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.255+0000 7fa17d87b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763fa6b0 0x55c7767c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.255+0000 7fa17d87b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c776721d80 con 0x55c7763fa6b0 2024-04-25T11:36:17.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.255+0000 7fa17d87b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c7766dba20 con 0x55c7763f24d0 2024-04-25T11:36:17.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.255+0000 7fa17d87b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c77671dd40 con 0x55c77641b390 2024-04-25T11:36:17.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.255+0000 7fa16f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763fa6b0 0x55c7767c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.255+0000 7fa16f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763fa6b0 0x55c7767c96a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40176/0 (socket says 172.21.15.90:40176) 2024-04-25T11:36:17.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.255+0000 7fa16f7fe640 1 -- 172.21.15.90:0/3121121760 learned_addr learned my addr 172.21.15.90:0/3121121760 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:17.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.255+0000 7fa174f62640 1 --2- 172.21.15.90:0/3121121760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c77641b390 0x55c7767bc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.255+0000 7fa16ffff640 1 --2- 172.21.15.90:0/3121121760 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7763f24d0 0x55c7767bcd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.256+0000 7fa16f7fe640 1 -- 172.21.15.90:0/3121121760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c77641b390 msgr2=0x55c7767bc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:17.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.256+0000 7fa16f7fe640 1 --2- 172.21.15.90:0/3121121760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c77641b390 0x55c7767bc810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.256+0000 7fa16f7fe640 1 -- 172.21.15.90:0/3121121760 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7763f24d0 msgr2=0x55c7767bcd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:17.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.256+0000 7fa16f7fe640 1 --2- 172.21.15.90:0/3121121760 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7763f24d0 0x55c7767bcd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.256+0000 7fa16f7fe640 1 -- 172.21.15.90:0/3121121760 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7766e6c30 con 0x55c7763fa6b0 2024-04-25T11:36:17.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.256+0000 7fa16d7fa640 1 --2- 172.21.15.90:0/1254941732 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa1580afd80 0x7fa1580b2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.256+0000 7fa174f62640 1 --2- 172.21.15.90:0/1254941732 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa1580afd80 0x7fa1580b2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.256+0000 7fa16f7fe640 1 --2- 172.21.15.90:0/3121121760 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763fa6b0 0x55c7767c96a0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa1640dc410 tx=0x7fa1640a3380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97cb53276c55a72a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:17.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.256+0000 7fa16d7fa640 1 -- 172.21.15.90:0/1254941732 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fa1640ded90 con 0x7fa17010b650 2024-04-25T11:36:17.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.256+0000 7fa1567fc640 1 -- 172.21.15.90:0/3121121760 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa164020070 con 0x55c7763fa6b0 2024-04-25T11:36:17.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.256+0000 7fa1567fc640 1 -- 172.21.15.90:0/3121121760 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa16408f030 con 0x55c7763fa6b0 2024-04-25T11:36:17.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.256+0000 7fa1567fc640 1 -- 172.21.15.90:0/3121121760 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1640a6680 con 0x55c7763fa6b0 2024-04-25T11:36:17.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.257+0000 7fa17d87b740 1 -- 172.21.15.90:0/3121121760 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763fa6b0 msgr2=0x55c7767c96a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:17.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.257+0000 7fa17d87b740 1 --2- 172.21.15.90:0/3121121760 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763fa6b0 0x55c7767c96a0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa1640dc410 tx=0x7fa1640a3380 comp rx=0 tx=0).stop 2024-04-25T11:36:17.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.257+0000 7fa174f62640 1 --2- 172.21.15.90:0/1254941732 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa1580afd80 0x7fa1580b2290 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x55c7767b9980 tx=0x7fa168009500 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:17.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.257+0000 7fa17d87b740 1 -- 172.21.15.90:0/3121121760 shutdown_connections 2024-04-25T11:36:17.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.257+0000 7fa17d87b740 1 --2- 172.21.15.90:0/3121121760 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763fa6b0 0x55c7767c96a0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.257+0000 7fa17d87b740 1 --2- 172.21.15.90:0/3121121760 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7763f24d0 0x55c7767bcd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.257+0000 7fa17d87b740 1 --2- 172.21.15.90:0/3121121760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c77641b390 0x55c7767bc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.257+0000 7fa17d87b740 1 -- 172.21.15.90:0/3121121760 >> 172.21.15.90:0/3121121760 conn(0x55c77640f4e0 msgr2=0x55c7767b8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:17.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.258+0000 7fa17d87b740 1 -- 172.21.15.90:0/3121121760 shutdown_connections 2024-04-25T11:36:17.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.258+0000 7fa17d87b740 1 -- 172.21.15.90:0/3121121760 wait complete. 2024-04-25T11:36:17.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.258+0000 7fa17d87b740 1 -- start start 2024-04-25T11:36:17.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.258+0000 7fa17d87b740 1 Processor -- start 2024-04-25T11:36:17.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.259+0000 7fa17d87b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763f24d0 0x55c7767c72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.259+0000 7fa16ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763f24d0 0x55c7767c72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.259+0000 7fa16ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763f24d0 0x55c7767c72d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40180/0 (socket says 172.21.15.90:40180) 2024-04-25T11:36:17.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.259+0000 7fa16ffff640 1 -- 172.21.15.90:0/1734980398 learned_addr learned my addr 172.21.15.90:0/1734980398 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:17.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.259+0000 7fa17d87b740 1 --2- 172.21.15.90:0/1734980398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7763fa6b0 0x55c7767c7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.259+0000 7fa17d87b740 1 --2- 172.21.15.90:0/1734980398 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c77641b390 0x55c7767d3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.259+0000 7fa17d87b740 1 -- 172.21.15.90:0/1734980398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c7767c9d90 con 0x55c7763f24d0 2024-04-25T11:36:17.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.259+0000 7fa17d87b740 1 -- 172.21.15.90:0/1734980398 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c7766a8dc0 con 0x55c77641b390 2024-04-25T11:36:17.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.259+0000 7fa17d87b740 1 -- 172.21.15.90:0/1734980398 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c7767c9f10 con 0x55c7763fa6b0 2024-04-25T11:36:17.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.259+0000 7fa17d87b740 10 client.0 _ll_register_callbacks cb 0x55c7766c4600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:17.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.259+0000 7fa16f7fe640 1 --2- 172.21.15.90:0/1734980398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7763fa6b0 0x55c7767c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.259+0000 7fa174f62640 1 --2- 172.21.15.90:0/1734980398 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c77641b390 0x55c7767d3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.259+0000 7fa16ffff640 1 -- 172.21.15.90:0/1734980398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7763fa6b0 msgr2=0x55c7767c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:17.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.259+0000 7fa16ffff640 1 --2- 172.21.15.90:0/1734980398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7763fa6b0 0x55c7767c7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.259+0000 7fa16ffff640 1 -- 172.21.15.90:0/1734980398 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c77641b390 msgr2=0x55c7767d3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:17.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.259+0000 7fa16ffff640 1 --2- 172.21.15.90:0/1734980398 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c77641b390 0x55c7767d3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.259+0000 7fa16ffff640 1 -- 172.21.15.90:0/1734980398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7764333f0 con 0x55c7763f24d0 2024-04-25T11:36:17.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.259+0000 7fa174f62640 1 --2- 172.21.15.90:0/1734980398 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c77641b390 0x55c7767d3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:36:17.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.259+0000 7fa16ffff640 1 --2- 172.21.15.90:0/1734980398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763f24d0 0x55c7767c72d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fa160009900 tx=0x7fa1600316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd97a40626c8312b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:17.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.259+0000 7fa14f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:17.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.259+0000 7fa14f7fe640 1 -- 172.21.15.90:0/1734980398 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa160035040 con 0x55c7763f24d0 2024-04-25T11:36:17.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.260+0000 7fa14f7fe640 1 -- 172.21.15.90:0/1734980398 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fa160004240 con 0x55c7763f24d0 2024-04-25T11:36:17.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.260+0000 7fa17d87b740 1 -- 172.21.15.90:0/1734980398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa16408f030 con 0x55c7763f24d0 2024-04-25T11:36:17.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.260+0000 7fa14f7fe640 1 -- 172.21.15.90:0/1734980398 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa160052050 con 0x55c7763f24d0 2024-04-25T11:36:17.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.260+0000 7fa17d87b740 10 client.15930 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:17.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.260+0000 7fa17d87b740 1 -- 172.21.15.90:0/1734980398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c7764fd7d0 con 0x55c7763f24d0 2024-04-25T11:36:17.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.260+0000 7fa14f7fe640 1 -- 172.21.15.90:0/1734980398 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa16408f030 con 0x55c7763f24d0 2024-04-25T11:36:17.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.260+0000 7fa14f7fe640 1 -- 172.21.15.90:0/1734980398 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fa16004d030 con 0x55c7763f24d0 2024-04-25T11:36:17.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.260+0000 7fa17d87b740 10 client.15930 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:17.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.260+0000 7fa17d87b740 10 client.15930 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:36:17.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.260+0000 7fa17d87b740 20 client.15930 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:17.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.260+0000 7fa17d87b740 1 --2- 172.21.15.90:0/1734980398 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c7763d9f60 0x55c7767d97c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.260+0000 7fa17d87b740 4 client.15930 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:36:17.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.260+0000 7fa17d87b740 1 -- 172.21.15.90:0/1734980398 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c77672f990 con 0x55c7763d9f60 2024-04-25T11:36:17.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.261+0000 7fa16f7fe640 1 --2- 172.21.15.90:0/1734980398 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c7763d9f60 0x55c7767d97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.261+0000 7fa16f7fe640 1 --2- 172.21.15.90:0/1734980398 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c7763d9f60 0x55c7767d97c0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da9b37ec9e82f61 server_cookie=60e958deebef3777 in_seq=0 out_seq=0 2024-04-25T11:36:17.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.261+0000 7fa14f7fe640 10 client.15930 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:17.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.262+0000 7fa14f7fe640 1 -- 172.21.15.90:0/1734980398 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c77672f990 con 0x55c7763d9f60 2024-04-25T11:36:17.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.262+0000 7fa14f7fe640 10 client.15930 handle_command_reply: tid=0 2024-04-25T11:36:17.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.262+0000 7fa17d87b740 1 client.15930 shutdown 2024-04-25T11:36:17.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.262+0000 7fa17d87b740 1 -- 172.21.15.90:0/1734980398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763f24d0 msgr2=0x55c7767c72d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:17.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.262+0000 7fa17d87b740 1 --2- 172.21.15.90:0/1734980398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763f24d0 0x55c7767c72d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fa160009900 tx=0x7fa1600316d0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.262+0000 7fa17d87b740 1 -- 172.21.15.90:0/1734980398 shutdown_connections 2024-04-25T11:36:17.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.262+0000 7fa17d87b740 1 --2- 172.21.15.90:0/1734980398 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c7763d9f60 0x55c7767d97c0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.262+0000 7fa17d87b740 1 --2- 172.21.15.90:0/1734980398 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c77641b390 0x55c7767d3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.262+0000 7fa14f7fe640 0 client.15930 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:17.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.262+0000 7fa17d87b740 1 --2- 172.21.15.90:0/1734980398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7763fa6b0 0x55c7767c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.262+0000 7fa17d87b740 1 --2- 172.21.15.90:0/1734980398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763f24d0 0x55c7767c72d0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.262+0000 7fa17d87b740 1 -- 172.21.15.90:0/1734980398 >> 172.21.15.90:0/1734980398 conn(0x55c77640f4e0 msgr2=0x55c7767be530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:17.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.262+0000 7fa17d87b740 1 -- 172.21.15.90:0/1734980398 shutdown_connections 2024-04-25T11:36:17.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.262+0000 7fa17d87b740 1 -- 172.21.15.90:0/1734980398 wait complete. 2024-04-25T11:36:17.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.262+0000 7fa17d87b740 20 client.15930 trim_cache size 0 max 16384 2024-04-25T11:36:17.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.288+0000 7fa17d87b740 1 Processor -- start 2024-04-25T11:36:17.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.288+0000 7fa17d87b740 1 -- start start 2024-04-25T11:36:17.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.288+0000 7fa17d87b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763d9f60 0x55c7767d2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.288+0000 7fa17d87b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7763f24d0 0x55c7767c8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.288+0000 7fa17d87b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7763fa6b0 0x55c7767c14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.288+0000 7fa17d87b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa1701ad8e0 con 0x55c7763d9f60 2024-04-25T11:36:17.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.288+0000 7fa17d87b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa1701af420 con 0x55c7763fa6b0 2024-04-25T11:36:17.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.288+0000 7fa17d87b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa1701adfb0 con 0x55c7763f24d0 2024-04-25T11:36:17.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.288+0000 7fa16ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763d9f60 0x55c7767d2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.288+0000 7fa16ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763d9f60 0x55c7767d2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40192/0 (socket says 172.21.15.90:40192) 2024-04-25T11:36:17.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.288+0000 7fa16ffff640 1 -- 172.21.15.90:0/737917746 learned_addr learned my addr 172.21.15.90:0/737917746 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:17.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.288+0000 7fa16f7fe640 1 --2- 172.21.15.90:0/737917746 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7763f24d0 0x55c7767c8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.288+0000 7fa174f62640 1 --2- 172.21.15.90:0/737917746 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7763fa6b0 0x55c7767c14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.288+0000 7fa16ffff640 1 -- 172.21.15.90:0/737917746 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7763f24d0 msgr2=0x55c7767c8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:17.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.288+0000 7fa16ffff640 1 --2- 172.21.15.90:0/737917746 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7763f24d0 0x55c7767c8ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.289+0000 7fa16ffff640 1 -- 172.21.15.90:0/737917746 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7763fa6b0 msgr2=0x55c7767c14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:17.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.289+0000 7fa16ffff640 1 --2- 172.21.15.90:0/737917746 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7763fa6b0 0x55c7767c14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.289+0000 7fa16ffff640 1 -- 172.21.15.90:0/737917746 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7767c8000 con 0x55c7763d9f60 2024-04-25T11:36:17.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.289+0000 7fa16f7fe640 1 --2- 172.21.15.90:0/737917746 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7763f24d0 0x55c7767c8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:36:17.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.289+0000 7fa174f62640 1 --2- 172.21.15.90:0/737917746 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7763fa6b0 0x55c7767c14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:36:17.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.289+0000 7fa16ffff640 1 --2- 172.21.15.90:0/737917746 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763d9f60 0x55c7767d2e70 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa160032f20 tx=0x7fa16002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4af3fdb8855e1450 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:17.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.289+0000 7fa155ffb640 1 -- 172.21.15.90:0/737917746 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16005b070 con 0x55c7763d9f60 2024-04-25T11:36:17.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.289+0000 7fa155ffb640 1 -- 172.21.15.90:0/737917746 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa160004dd0 con 0x55c7763d9f60 2024-04-25T11:36:17.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.289+0000 7fa155ffb640 1 -- 172.21.15.90:0/737917746 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa160035070 con 0x55c7763d9f60 2024-04-25T11:36:17.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.290+0000 7fa17d87b740 1 -- 172.21.15.90:0/737917746 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763d9f60 msgr2=0x55c7767d2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:17.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.290+0000 7fa17d87b740 1 --2- 172.21.15.90:0/737917746 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763d9f60 0x55c7767d2e70 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa160032f20 tx=0x7fa16002a410 comp rx=0 tx=0).stop 2024-04-25T11:36:17.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.290+0000 7fa17d87b740 1 -- 172.21.15.90:0/737917746 shutdown_connections 2024-04-25T11:36:17.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.290+0000 7fa17d87b740 1 --2- 172.21.15.90:0/737917746 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7763fa6b0 0x55c7767c14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.290+0000 7fa17d87b740 1 --2- 172.21.15.90:0/737917746 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7763f24d0 0x55c7767c8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.290+0000 7fa17d87b740 1 --2- 172.21.15.90:0/737917746 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763d9f60 0x55c7767d2e70 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.290+0000 7fa17d87b740 1 -- 172.21.15.90:0/737917746 >> 172.21.15.90:0/737917746 conn(0x55c77640f4e0 msgr2=0x55c7767c87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:17.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.290+0000 7fa17d87b740 1 -- 172.21.15.90:0/737917746 shutdown_connections 2024-04-25T11:36:17.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.290+0000 7fa17d87b740 1 -- 172.21.15.90:0/737917746 wait complete. 2024-04-25T11:36:17.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.291+0000 7fa17d87b740 1 -- start start 2024-04-25T11:36:17.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.291+0000 7fa17d87b740 1 Processor -- start 2024-04-25T11:36:17.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.291+0000 7fa17d87b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7763d9f60 0x55c7767ca830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.291+0000 7fa17d87b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7763f24d0 0x55c7767cad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.291+0000 7fa17d87b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763fa6b0 0x55c776839af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.292+0000 7fa17d87b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa1701aed50 con 0x55c7763fa6b0 2024-04-25T11:36:17.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.292+0000 7fa17d87b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa1701ae680 con 0x55c7763f24d0 2024-04-25T11:36:17.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.292+0000 7fa17d87b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa1701ab8c0 con 0x55c7763d9f60 2024-04-25T11:36:17.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.292+0000 7fa174f62640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763fa6b0 0x55c776839af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.292+0000 7fa17d87b740 10 client.0 _ll_register_callbacks cb 0x7fa160009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:17.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.292+0000 7fa174f62640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763fa6b0 0x55c776839af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40198/0 (socket says 172.21.15.90:40198) 2024-04-25T11:36:17.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.292+0000 7fa174f62640 1 -- 172.21.15.90:0/3669087596 learned_addr learned my addr 172.21.15.90:0/3669087596 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:17.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.292+0000 7fa16ffff640 1 --2- 172.21.15.90:0/3669087596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7763d9f60 0x55c7767ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.292+0000 7fa16f7fe640 1 --2- 172.21.15.90:0/3669087596 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7763f24d0 0x55c7767cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.292+0000 7fa174f62640 1 -- 172.21.15.90:0/3669087596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7763d9f60 msgr2=0x55c7767ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:17.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.292+0000 7fa174f62640 1 --2- 172.21.15.90:0/3669087596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7763d9f60 0x55c7767ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.292+0000 7fa174f62640 1 -- 172.21.15.90:0/3669087596 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7763f24d0 msgr2=0x55c7767cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:17.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.292+0000 7fa174f62640 1 --2- 172.21.15.90:0/3669087596 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7763f24d0 0x55c7767cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.292+0000 7fa174f62640 1 -- 172.21.15.90:0/3669087596 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c77683a0b0 con 0x55c7763fa6b0 2024-04-25T11:36:17.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.292+0000 7fa16f7fe640 1 --2- 172.21.15.90:0/3669087596 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7763f24d0 0x55c7767cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:36:17.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.292+0000 7fa174f62640 1 --2- 172.21.15.90:0/3669087596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763fa6b0 0x55c776839af0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa168009810 tx=0x7fa168004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbc5078359cfe86e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:17.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.292+0000 7fa14f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:17.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.292+0000 7fa14f7fe640 1 -- 172.21.15.90:0/3669087596 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16802c060 con 0x55c7763fa6b0 2024-04-25T11:36:17.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.292+0000 7fa14f7fe640 1 -- 172.21.15.90:0/3669087596 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fa16800f480 con 0x55c7763fa6b0 2024-04-25T11:36:17.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.292+0000 7fa17d87b740 1 -- 172.21.15.90:0/3669087596 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa160004dd0 con 0x55c7763fa6b0 2024-04-25T11:36:17.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.293+0000 7fa14f7fe640 1 -- 172.21.15.90:0/3669087596 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa16800f680 con 0x55c7763fa6b0 2024-04-25T11:36:17.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.293+0000 7fa17d87b740 10 client.15936 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:17.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.293+0000 7fa17d87b740 1 -- 172.21.15.90:0/3669087596 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c7768412a0 con 0x55c7763fa6b0 2024-04-25T11:36:17.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.293+0000 7fa14f7fe640 1 -- 172.21.15.90:0/3669087596 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa160004dd0 con 0x55c7763fa6b0 2024-04-25T11:36:17.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.293+0000 7fa14f7fe640 1 -- 172.21.15.90:0/3669087596 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fa16802f9e0 con 0x55c7763fa6b0 2024-04-25T11:36:17.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.293+0000 7fa17d87b740 10 client.15936 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:17.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.293+0000 7fa17d87b740 10 client.15936 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:36:17.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.293+0000 7fa17d87b740 20 client.15936 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:17.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.293+0000 7fa17d87b740 1 --2- 172.21.15.90:0/3669087596 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c77641b390 0x55c776843a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.293+0000 7fa17d87b740 4 client.15936 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:36:17.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.293+0000 7fa17d87b740 1 -- 172.21.15.90:0/3669087596 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c776729800 con 0x55c77641b390 2024-04-25T11:36:17.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.293+0000 7fa16ffff640 1 --2- 172.21.15.90:0/3669087596 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c77641b390 0x55c776843a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.294+0000 7fa16ffff640 1 --2- 172.21.15.90:0/3669087596 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c77641b390 0x55c776843a60 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3b7da5c27e58bd7 server_cookie=1f9495dc29a38949 in_seq=0 out_seq=0 2024-04-25T11:36:17.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.294+0000 7fa14f7fe640 10 client.15936 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:17.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.294+0000 7fa14f7fe640 1 -- 172.21.15.90:0/3669087596 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c776729800 con 0x55c77641b390 2024-04-25T11:36:17.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.294+0000 7fa14f7fe640 10 client.15936 handle_command_reply: tid=0 2024-04-25T11:36:17.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.294+0000 7fa17d87b740 1 client.15936 shutdown 2024-04-25T11:36:17.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.294+0000 7fa17d87b740 1 -- 172.21.15.90:0/3669087596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763fa6b0 msgr2=0x55c776839af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:17.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.294+0000 7fa17d87b740 1 --2- 172.21.15.90:0/3669087596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763fa6b0 0x55c776839af0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa168009810 tx=0x7fa168004ab0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.294+0000 7fa17d87b740 1 -- 172.21.15.90:0/3669087596 shutdown_connections 2024-04-25T11:36:17.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.294+0000 7fa17d87b740 1 --2- 172.21.15.90:0/3669087596 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c77641b390 0x55c776843a60 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.294+0000 7fa14f7fe640 0 client.15936 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:17.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.294+0000 7fa17d87b740 1 --2- 172.21.15.90:0/3669087596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7763fa6b0 0x55c776839af0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.294+0000 7fa17d87b740 1 --2- 172.21.15.90:0/3669087596 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7763f24d0 0x55c7767cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.294+0000 7fa17d87b740 1 --2- 172.21.15.90:0/3669087596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7763d9f60 0x55c7767ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.294+0000 7fa17d87b740 1 -- 172.21.15.90:0/3669087596 >> 172.21.15.90:0/3669087596 conn(0x55c77640f4e0 msgr2=0x55c7767c8640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:17.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.294+0000 7fa17d87b740 1 -- 172.21.15.90:0/3669087596 shutdown_connections 2024-04-25T11:36:17.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.294+0000 7fa17d87b740 1 -- 172.21.15.90:0/3669087596 wait complete. 2024-04-25T11:36:17.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.294+0000 7fa17d87b740 20 client.15936 trim_cache size 0 max 16384 2024-04-25T11:36:17.288 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:36:17.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.295+0000 7fa1769c5640 1 -- 172.21.15.90:0/1254941732 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa1580afd80 msgr2=0x7fa1580b2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:17.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.295+0000 7fa1769c5640 1 --2- 172.21.15.90:0/1254941732 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa1580afd80 0x7fa1580b2290 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x55c7767b9980 tx=0x7fa168009500 comp rx=0 tx=0).stop 2024-04-25T11:36:17.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.295+0000 7fa1769c5640 1 -- 172.21.15.90:0/1254941732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa17010b650 msgr2=0x7fa17019ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:17.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.295+0000 7fa1769c5640 1 --2- 172.21.15.90:0/1254941732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa17010b650 0x7fa17019ac80 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa16400d770 tx=0x7fa16400dc30 comp rx=0 tx=0).stop 2024-04-25T11:36:17.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.295+0000 7fa1769c5640 1 -- 172.21.15.90:0/1254941732 shutdown_connections 2024-04-25T11:36:17.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.295+0000 7fa1769c5640 1 --2- 172.21.15.90:0/1254941732 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa1580afd80 0x7fa1580b2290 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.295+0000 7fa1769c5640 1 --2- 172.21.15.90:0/1254941732 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa17010e040 0x7fa17019ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.295+0000 7fa1769c5640 1 --2- 172.21.15.90:0/1254941732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa17010b650 0x7fa17019ac80 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.295+0000 7fa1769c5640 1 --2- 172.21.15.90:0/1254941732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa170104bf0 0x7fa17019a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.295+0000 7fa1769c5640 1 -- 172.21.15.90:0/1254941732 >> 172.21.15.90:0/1254941732 conn(0x7fa170078100 msgr2=0x7fa170101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:17.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.295+0000 7fa1769c5640 1 -- 172.21.15.90:0/1254941732 shutdown_connections 2024-04-25T11:36:17.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:17.295+0000 7fa1769c5640 1 -- 172.21.15.90:0/1254941732 wait complete. 2024-04-25T11:36:17.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:17 smithi090 ceph-mon[29575]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 895 B/s wr, 5 op/s; 243 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T11:36:17.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:17 smithi090 ceph-mon[29575]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-25T11:36:17.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:17 smithi090 ceph-mon[29575]: Cluster is now healthy 2024-04-25T11:36:17.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:17 smithi138 ceph-mon[32328]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 895 B/s wr, 5 op/s; 243 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T11:36:17.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:17 smithi138 ceph-mon[32328]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-25T11:36:17.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:17 smithi138 ceph-mon[32328]: Cluster is now healthy 2024-04-25T11:36:17.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:17 smithi156 ceph-mon[33045]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 895 B/s wr, 5 op/s; 243 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T11:36:17.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:17 smithi156 ceph-mon[33045]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-25T11:36:17.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:17 smithi156 ceph-mon[33045]: Cluster is now healthy 2024-04-25T11:36:18.296 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T11:36:18.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:36:18.296 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T11:36:18.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.461+0000 7fcc1a19d640 1 -- 172.21.15.90:0/3954308979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc14108ec0 msgr2=0x7fcc14102b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:18.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.461+0000 7fcc1a19d640 1 --2- 172.21.15.90:0/3954308979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc14108ec0 0x7fcc14102b10 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fcc08009900 tx=0x7fcc0802a410 comp rx=0 tx=0).stop 2024-04-25T11:36:18.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.462+0000 7fcc1a19d640 1 -- 172.21.15.90:0/3954308979 shutdown_connections 2024-04-25T11:36:18.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.462+0000 7fcc1a19d640 1 --2- 172.21.15.90:0/3954308979 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcc14103180 0x7fcc1410f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.462+0000 7fcc1a19d640 1 --2- 172.21.15.90:0/3954308979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc14108ec0 0x7fcc14102b10 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.462+0000 7fcc1a19d640 1 --2- 172.21.15.90:0/3954308979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc14108a20 0x7fcc141025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.462+0000 7fcc1a19d640 1 -- 172.21.15.90:0/3954308979 >> 172.21.15.90:0/3954308979 conn(0x7fcc14078110 msgr2=0x7fcc14078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:18.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.462+0000 7fcc1a19d640 1 -- 172.21.15.90:0/3954308979 shutdown_connections 2024-04-25T11:36:18.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.462+0000 7fcc1a19d640 1 -- 172.21.15.90:0/3954308979 wait complete. 2024-04-25T11:36:18.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.462+0000 7fcc1a19d640 1 Processor -- start 2024-04-25T11:36:18.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.462+0000 7fcc1a19d640 1 -- start start 2024-04-25T11:36:18.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.463+0000 7fcc1a19d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcc14103180 0x7fcc1419a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.463+0000 7fcc1a19d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc14108a20 0x7fcc1419ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.463+0000 7fcc1a19d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc14108ec0 0x7fcc1419ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.463+0000 7fcc1a19d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcc14113d40 con 0x7fcc14108a20 2024-04-25T11:36:18.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.463+0000 7fcc1a19d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fcc14113bc0 con 0x7fcc14103180 2024-04-25T11:36:18.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.463+0000 7fcc1a19d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcc14113ec0 con 0x7fcc14108ec0 2024-04-25T11:36:18.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.463+0000 7fcc12ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc14108a20 0x7fcc1419ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.463+0000 7fcc12ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc14108a20 0x7fcc1419ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40212/0 (socket says 172.21.15.90:40212) 2024-04-25T11:36:18.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.463+0000 7fcc12ffd640 1 -- 172.21.15.90:0/147626646 learned_addr learned my addr 172.21.15.90:0/147626646 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:18.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.463+0000 7fcc13fff640 1 --2- 172.21.15.90:0/147626646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc14108ec0 0x7fcc1419ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.463+0000 7fcc137fe640 1 --2- 172.21.15.90:0/147626646 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcc14103180 0x7fcc1419a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.463+0000 7fcc12ffd640 1 -- 172.21.15.90:0/147626646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc14108ec0 msgr2=0x7fcc1419ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:18.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.463+0000 7fcc12ffd640 1 --2- 172.21.15.90:0/147626646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc14108ec0 0x7fcc1419ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.463+0000 7fcc12ffd640 1 -- 172.21.15.90:0/147626646 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcc14103180 msgr2=0x7fcc1419a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:18.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.463+0000 7fcc12ffd640 1 --2- 172.21.15.90:0/147626646 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcc14103180 0x7fcc1419a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.463+0000 7fcc12ffd640 1 -- 172.21.15.90:0/147626646 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc1419f580 con 0x7fcc14108a20 2024-04-25T11:36:18.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.463+0000 7fcc13fff640 1 --2- 172.21.15.90:0/147626646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc14108ec0 0x7fcc1419ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:36:18.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.464+0000 7fcc12ffd640 1 --2- 172.21.15.90:0/147626646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc14108a20 0x7fcc1419ac80 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fcc0802a930 tx=0x7fcc08004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:18.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.464+0000 7fcc10ff9640 1 -- 172.21.15.90:0/147626646 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0803f030 con 0x7fcc14108a20 2024-04-25T11:36:18.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.464+0000 7fcc10ff9640 1 -- 172.21.15.90:0/147626646 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc08003aa0 con 0x7fcc14108a20 2024-04-25T11:36:18.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.464+0000 7fcc10ff9640 1 -- 172.21.15.90:0/147626646 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc08035070 con 0x7fcc14108a20 2024-04-25T11:36:18.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.464+0000 7fcc1a19d640 1 -- 172.21.15.90:0/147626646 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc1419f810 con 0x7fcc14108a20 2024-04-25T11:36:18.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.464+0000 7fcc1a19d640 1 -- 172.21.15.90:0/147626646 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc1419fd50 con 0x7fcc14108a20 2024-04-25T11:36:18.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.467+0000 7fcc2104f740 1 Processor -- start 2024-04-25T11:36:18.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.467+0000 7fcc2104f740 1 -- start start 2024-04-25T11:36:18.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.467+0000 7fcc2104f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c85d0e8410 0x55c85d488620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.467+0000 7fcc2104f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c85d0bf550 0x55c85d488b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.467+0000 7fcc2104f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0c7730 0x55c85d4890a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.467+0000 7fcc2104f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c85d3eeb00 con 0x55c85d0c7730 2024-04-25T11:36:18.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.467+0000 7fcc2104f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c85d38a920 con 0x55c85d0e8410 2024-04-25T11:36:18.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.467+0000 7fcc2104f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c85d3eaac0 con 0x55c85d0bf550 2024-04-25T11:36:18.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.467+0000 7fcc10ff9640 1 -- 172.21.15.90:0/147626646 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 24) ==== 147753+0+0 (secure 0 0 0) 0x7fcc08042030 con 0x7fcc14108a20 2024-04-25T11:36:18.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.468+0000 7fcc13fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c85d0e8410 0x55c85d488620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.468+0000 7fcc12ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0c7730 0x55c85d4890a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.468+0000 7fcc13fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c85d0e8410 0x55c85d488620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:37582/0 (socket says 172.21.15.90:37582) 2024-04-25T11:36:18.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.468+0000 7fcc12ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0c7730 0x55c85d4890a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40226/0 (socket says 172.21.15.90:40226) 2024-04-25T11:36:18.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.468+0000 7fcc13fff640 1 -- 172.21.15.90:0/2786471123 learned_addr learned my addr 172.21.15.90:0/2786471123 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:18.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.468+0000 7fcc137fe640 1 --2- 172.21.15.90:0/2786471123 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c85d0bf550 0x55c85d488b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.468+0000 7fcc12ffd640 1 -- 172.21.15.90:0/2786471123 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c85d0bf550 msgr2=0x55c85d488b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:18.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.468+0000 7fcc12ffd640 1 --2- 172.21.15.90:0/2786471123 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c85d0bf550 0x55c85d488b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.468+0000 7fcc12ffd640 1 -- 172.21.15.90:0/2786471123 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c85d0e8410 msgr2=0x55c85d488620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:18.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.468+0000 7fcc12ffd640 1 --2- 172.21.15.90:0/2786471123 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c85d0e8410 0x55c85d488620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.468+0000 7fcc12ffd640 1 -- 172.21.15.90:0/2786471123 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c85d1c87e0 con 0x55c85d0c7730 2024-04-25T11:36:18.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.468+0000 7fcc10ff9640 1 --2- 172.21.15.90:0/147626646 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcbfc0afe00 0x7fcbfc0b2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.468+0000 7fcc12ffd640 1 --2- 172.21.15.90:0/2786471123 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0c7730 0x55c85d4890a0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fcc080fcf40 tx=0x7fcc0810af00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33e24c734c89ec68 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:18.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.468+0000 7fcc13fff640 1 --2- 172.21.15.90:0/147626646 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcbfc0afe00 0x7fcbfc0b2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.469+0000 7fcc10ff9640 1 -- 172.21.15.90:0/147626646 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fcc080ff9d0 con 0x7fcc14108a20 2024-04-25T11:36:18.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.469+0000 7fcbf9ffb640 1 -- 172.21.15.90:0/2786471123 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc080c3850 con 0x55c85d0c7730 2024-04-25T11:36:18.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.469+0000 7fcbf9ffb640 1 -- 172.21.15.90:0/2786471123 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc080c74a0 con 0x55c85d0c7730 2024-04-25T11:36:18.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.469+0000 7fcbf9ffb640 1 -- 172.21.15.90:0/2786471123 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc080c7100 con 0x55c85d0c7730 2024-04-25T11:36:18.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.470+0000 7fcc2104f740 1 -- 172.21.15.90:0/2786471123 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0c7730 msgr2=0x55c85d4890a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:18.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.470+0000 7fcc13fff640 1 --2- 172.21.15.90:0/147626646 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcbfc0afe00 0x7fcbfc0b2310 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x55c85d485790 tx=0x7fcc0c0098a0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:18.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.470+0000 7fcc2104f740 1 --2- 172.21.15.90:0/2786471123 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0c7730 0x55c85d4890a0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fcc080fcf40 tx=0x7fcc0810af00 comp rx=0 tx=0).stop 2024-04-25T11:36:18.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.470+0000 7fcc2104f740 1 -- 172.21.15.90:0/2786471123 shutdown_connections 2024-04-25T11:36:18.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.470+0000 7fcc2104f740 1 --2- 172.21.15.90:0/2786471123 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0c7730 0x55c85d4890a0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.470+0000 7fcc2104f740 1 --2- 172.21.15.90:0/2786471123 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c85d0bf550 0x55c85d488b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.470+0000 7fcc2104f740 1 --2- 172.21.15.90:0/2786471123 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c85d0e8410 0x55c85d488620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.470+0000 7fcc2104f740 1 -- 172.21.15.90:0/2786471123 >> 172.21.15.90:0/2786471123 conn(0x55c85d0dc560 msgr2=0x55c85d484dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:18.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.470+0000 7fcc2104f740 1 -- 172.21.15.90:0/2786471123 shutdown_connections 2024-04-25T11:36:18.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.470+0000 7fcc2104f740 1 -- 172.21.15.90:0/2786471123 wait complete. 2024-04-25T11:36:18.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.470+0000 7fcc2104f740 1 -- start start 2024-04-25T11:36:18.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.470+0000 7fcc2104f740 1 Processor -- start 2024-04-25T11:36:18.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.471+0000 7fcc2104f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c85d0bf550 0x55c85d492cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.471+0000 7fcc2104f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0c7730 0x55c85d493210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.471+0000 7fcc2104f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c85d0e8410 0x55c85d498b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.471+0000 7fcc2104f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c85d495730 con 0x55c85d0c7730 2024-04-25T11:36:18.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.471+0000 7fcc2104f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c85d3885d0 con 0x55c85d0bf550 2024-04-25T11:36:18.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.471+0000 7fcc2104f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c85d4958b0 con 0x55c85d0e8410 2024-04-25T11:36:18.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.471+0000 7fcc2104f740 10 client.0 _ll_register_callbacks cb 0x55c85d376080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:18.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.471+0000 7fcc12ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0c7730 0x55c85d493210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.471+0000 7fcc12ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0c7730 0x55c85d493210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40230/0 (socket says 172.21.15.90:40230) 2024-04-25T11:36:18.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.471+0000 7fcc12ffd640 1 -- 172.21.15.90:0/794381096 learned_addr learned my addr 172.21.15.90:0/794381096 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:18.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.471+0000 7fcc137fe640 1 --2- 172.21.15.90:0/794381096 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c85d0bf550 0x55c85d492cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.471+0000 7fcc12ffd640 1 -- 172.21.15.90:0/794381096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c85d0e8410 msgr2=0x55c85d498b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:18.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.471+0000 7fcc12ffd640 1 --2- 172.21.15.90:0/794381096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c85d0e8410 0x55c85d498b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.471+0000 7fcc12ffd640 1 -- 172.21.15.90:0/794381096 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c85d0bf550 msgr2=0x55c85d492cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:18.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.471+0000 7fcc12ffd640 1 --2- 172.21.15.90:0/794381096 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c85d0bf550 0x55c85d492cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.471+0000 7fcc12ffd640 1 -- 172.21.15.90:0/794381096 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c85d228f70 con 0x55c85d0c7730 2024-04-25T11:36:18.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.471+0000 7fcc12ffd640 1 --2- 172.21.15.90:0/794381096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0c7730 0x55c85d493210 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fcc080c7690 tx=0x7fcc0802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51fe3804ca281b26 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:18.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.471+0000 7fcbf2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:18.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.472+0000 7fcbf2ffd640 1 -- 172.21.15.90:0/794381096 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc08049070 con 0x55c85d0c7730 2024-04-25T11:36:18.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.472+0000 7fcbf2ffd640 1 -- 172.21.15.90:0/794381096 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fcc08003c50 con 0x55c85d0c7730 2024-04-25T11:36:18.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.472+0000 7fcc2104f740 1 -- 172.21.15.90:0/794381096 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc080c74a0 con 0x55c85d0c7730 2024-04-25T11:36:18.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.472+0000 7fcbf2ffd640 1 -- 172.21.15.90:0/794381096 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcc080c72e0 con 0x55c85d0c7730 2024-04-25T11:36:18.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.472+0000 7fcc2104f740 10 client.15948 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:18.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.472+0000 7fcc2104f740 1 -- 172.21.15.90:0/794381096 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c85d0c0170 con 0x55c85d0c7730 2024-04-25T11:36:18.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.472+0000 7fcbf2ffd640 1 -- 172.21.15.90:0/794381096 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc080c74a0 con 0x55c85d0c7730 2024-04-25T11:36:18.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.472+0000 7fcbf2ffd640 1 -- 172.21.15.90:0/794381096 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fcc080ea080 con 0x55c85d0c7730 2024-04-25T11:36:18.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.472+0000 7fcc2104f740 10 client.15948 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:18.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.472+0000 7fcc2104f740 10 client.15948 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:36:18.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.472+0000 7fcc2104f740 20 client.15948 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:18.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.472+0000 7fcc2104f740 1 --2- 172.21.15.90:0/794381096 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c85d0a71f0 0x55c85d4a4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.472+0000 7fcc2104f740 4 client.15948 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:36:18.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.472+0000 7fcc2104f740 1 -- 172.21.15.90:0/794381096 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c85d3fc710 con 0x55c85d0a71f0 2024-04-25T11:36:18.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.473+0000 7fcc137fe640 1 --2- 172.21.15.90:0/794381096 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c85d0a71f0 0x55c85d4a4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.473+0000 7fcbf2ffd640 10 client.15948 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:18.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.473+0000 7fcc137fe640 1 --2- 172.21.15.90:0/794381096 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c85d0a71f0 0x55c85d4a4c80 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a6f4d4fd8be0ef5 server_cookie=ed4c5e832c742069 in_seq=0 out_seq=0 2024-04-25T11:36:18.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.474+0000 7fcbf2ffd640 1 -- 172.21.15.90:0/794381096 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c85d3fc710 con 0x55c85d0a71f0 2024-04-25T11:36:18.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.474+0000 7fcbf2ffd640 10 client.15948 handle_command_reply: tid=0 2024-04-25T11:36:18.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.474+0000 7fcc2104f740 1 client.15948 shutdown 2024-04-25T11:36:18.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.474+0000 7fcc2104f740 1 -- 172.21.15.90:0/794381096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0c7730 msgr2=0x55c85d493210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:18.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.474+0000 7fcc2104f740 1 --2- 172.21.15.90:0/794381096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0c7730 0x55c85d493210 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fcc080c7690 tx=0x7fcc0802a410 comp rx=0 tx=0).stop 2024-04-25T11:36:18.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.474+0000 7fcc2104f740 1 -- 172.21.15.90:0/794381096 shutdown_connections 2024-04-25T11:36:18.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.474+0000 7fcc2104f740 1 --2- 172.21.15.90:0/794381096 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c85d0a71f0 0x55c85d4a4c80 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.474+0000 7fcc2104f740 1 --2- 172.21.15.90:0/794381096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c85d0e8410 0x55c85d498b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.474+0000 7fcc2104f740 1 --2- 172.21.15.90:0/794381096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0c7730 0x55c85d493210 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.474+0000 7fcc2104f740 1 --2- 172.21.15.90:0/794381096 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c85d0bf550 0x55c85d492cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.474+0000 7fcc2104f740 1 -- 172.21.15.90:0/794381096 >> 172.21.15.90:0/794381096 conn(0x55c85d0dc560 msgr2=0x55c85d48a340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:18.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.474+0000 7fcbf2ffd640 0 client.15948 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:18.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.474+0000 7fcc2104f740 1 -- 172.21.15.90:0/794381096 shutdown_connections 2024-04-25T11:36:18.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.474+0000 7fcc2104f740 1 -- 172.21.15.90:0/794381096 wait complete. 2024-04-25T11:36:18.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.474+0000 7fcc2104f740 20 client.15948 trim_cache size 0 max 16384 2024-04-25T11:36:18.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.492+0000 7fcc2104f740 1 Processor -- start 2024-04-25T11:36:18.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.492+0000 7fcc2104f740 1 -- start start 2024-04-25T11:36:18.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.492+0000 7fcc2104f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c85d0bf550 0x55c85d492710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.492+0000 7fcc2104f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0c7730 0x55c85d49f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.492+0000 7fcc2104f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c85d0e8410 0x55c85d498ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.492+0000 7fcc2104f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcc141ad9b0 con 0x55c85d0c7730 2024-04-25T11:36:18.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.493+0000 7fcc2104f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fcc141af4f0 con 0x55c85d0e8410 2024-04-25T11:36:18.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.493+0000 7fcc2104f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcc141ae080 con 0x55c85d0bf550 2024-04-25T11:36:18.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.493+0000 7fcc12ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0c7730 0x55c85d49f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.493+0000 7fcc12ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0c7730 0x55c85d49f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40242/0 (socket says 172.21.15.90:40242) 2024-04-25T11:36:18.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.493+0000 7fcc12ffd640 1 -- 172.21.15.90:0/3028486910 learned_addr learned my addr 172.21.15.90:0/3028486910 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:18.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.493+0000 7fcc13fff640 1 --2- 172.21.15.90:0/3028486910 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c85d0e8410 0x55c85d498ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.493+0000 7fcc137fe640 1 --2- 172.21.15.90:0/3028486910 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c85d0bf550 0x55c85d492710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.493+0000 7fcc12ffd640 1 -- 172.21.15.90:0/3028486910 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c85d0bf550 msgr2=0x55c85d492710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:18.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.493+0000 7fcc12ffd640 1 --2- 172.21.15.90:0/3028486910 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c85d0bf550 0x55c85d492710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.493+0000 7fcc12ffd640 1 -- 172.21.15.90:0/3028486910 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c85d0e8410 msgr2=0x55c85d498ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:18.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.493+0000 7fcc12ffd640 1 --2- 172.21.15.90:0/3028486910 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c85d0e8410 0x55c85d498ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.493+0000 7fcc12ffd640 1 -- 172.21.15.90:0/3028486910 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c85d491010 con 0x55c85d0c7730 2024-04-25T11:36:18.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.493+0000 7fcc12ffd640 1 --2- 172.21.15.90:0/3028486910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0c7730 0x55c85d49f850 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fcc080ffe50 tx=0x7fcc0810add0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b06fbbc42d6454d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:18.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.493+0000 7fcbf97fa640 1 -- 172.21.15.90:0/3028486910 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc08049070 con 0x55c85d0c7730 2024-04-25T11:36:18.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.493+0000 7fcbf97fa640 1 -- 172.21.15.90:0/3028486910 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc0810aae0 con 0x55c85d0c7730 2024-04-25T11:36:18.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.493+0000 7fcbf97fa640 1 -- 172.21.15.90:0/3028486910 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc080c7760 con 0x55c85d0c7730 2024-04-25T11:36:18.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.494+0000 7fcc2104f740 1 -- 172.21.15.90:0/3028486910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0c7730 msgr2=0x55c85d49f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:18.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.494+0000 7fcc2104f740 1 --2- 172.21.15.90:0/3028486910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0c7730 0x55c85d49f850 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fcc080ffe50 tx=0x7fcc0810add0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.494+0000 7fcc2104f740 1 -- 172.21.15.90:0/3028486910 shutdown_connections 2024-04-25T11:36:18.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.494+0000 7fcc2104f740 1 --2- 172.21.15.90:0/3028486910 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c85d0e8410 0x55c85d498ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.494+0000 7fcc2104f740 1 --2- 172.21.15.90:0/3028486910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0c7730 0x55c85d49f850 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.494+0000 7fcc2104f740 1 --2- 172.21.15.90:0/3028486910 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c85d0bf550 0x55c85d492710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.494+0000 7fcc2104f740 1 -- 172.21.15.90:0/3028486910 >> 172.21.15.90:0/3028486910 conn(0x55c85d0dc560 msgr2=0x55c85d492000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:18.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.494+0000 7fcc2104f740 1 -- 172.21.15.90:0/3028486910 shutdown_connections 2024-04-25T11:36:18.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.494+0000 7fcc2104f740 1 -- 172.21.15.90:0/3028486910 wait complete. 2024-04-25T11:36:18.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.495+0000 7fcc2104f740 1 -- start start 2024-04-25T11:36:18.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.495+0000 7fcc2104f740 1 Processor -- start 2024-04-25T11:36:18.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.495+0000 7fcc2104f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0bf550 0x55c85d499c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.495+0000 7fcc2104f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c85d0c7730 0x55c85d49a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.495+0000 7fcc2104f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c85d0e8410 0x55c85d49a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.496+0000 7fcc137fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0bf550 0x55c85d499c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.496+0000 7fcc2104f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcc141aee20 con 0x55c85d0bf550 2024-04-25T11:36:18.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.496+0000 7fcc2104f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fcc141ae750 con 0x55c85d0e8410 2024-04-25T11:36:18.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.496+0000 7fcc2104f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcc141ab990 con 0x55c85d0c7730 2024-04-25T11:36:18.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.496+0000 7fcc137fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0bf550 0x55c85d499c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40254/0 (socket says 172.21.15.90:40254) 2024-04-25T11:36:18.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.496+0000 7fcc137fe640 1 -- 172.21.15.90:0/2897273317 learned_addr learned my addr 172.21.15.90:0/2897273317 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:18.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.496+0000 7fcc2104f740 10 client.0 _ll_register_callbacks cb 0x7fcc0810ad20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:18.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.496+0000 7fcc12ffd640 1 --2- 172.21.15.90:0/2897273317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c85d0c7730 0x55c85d49a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.496+0000 7fcc13fff640 1 --2- 172.21.15.90:0/2897273317 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c85d0e8410 0x55c85d49a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.496+0000 7fcc137fe640 1 -- 172.21.15.90:0/2897273317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c85d0c7730 msgr2=0x55c85d49a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:18.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.496+0000 7fcc137fe640 1 --2- 172.21.15.90:0/2897273317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c85d0c7730 0x55c85d49a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.496+0000 7fcc137fe640 1 -- 172.21.15.90:0/2897273317 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c85d0e8410 msgr2=0x55c85d49a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:18.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.496+0000 7fcc137fe640 1 --2- 172.21.15.90:0/2897273317 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c85d0e8410 0x55c85d49a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.496+0000 7fcc137fe640 1 -- 172.21.15.90:0/2897273317 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c85d50aef0 con 0x55c85d0bf550 2024-04-25T11:36:18.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.496+0000 7fcc137fe640 1 --2- 172.21.15.90:0/2897273317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0bf550 0x55c85d499c30 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fcc04005fd0 tx=0x7fcc0400dd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a68a1da7d11c00ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:18.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.496+0000 7fcbf2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:18.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.496+0000 7fcbf2ffd640 1 -- 172.21.15.90:0/2897273317 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0401d380 con 0x55c85d0bf550 2024-04-25T11:36:18.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.497+0000 7fcbf2ffd640 1 -- 172.21.15.90:0/2897273317 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fcc0401d520 con 0x55c85d0bf550 2024-04-25T11:36:18.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.497+0000 7fcc2104f740 1 -- 172.21.15.90:0/2897273317 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc0810aae0 con 0x55c85d0bf550 2024-04-25T11:36:18.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.497+0000 7fcbf2ffd640 1 -- 172.21.15.90:0/2897273317 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcc0401d720 con 0x55c85d0bf550 2024-04-25T11:36:18.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.497+0000 7fcc2104f740 10 client.15954 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:18.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.497+0000 7fcc2104f740 1 -- 172.21.15.90:0/2897273317 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c85d50c090 con 0x55c85d0bf550 2024-04-25T11:36:18.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.497+0000 7fcbf2ffd640 1 -- 172.21.15.90:0/2897273317 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc0810aae0 con 0x55c85d0bf550 2024-04-25T11:36:18.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.497+0000 7fcbf2ffd640 1 -- 172.21.15.90:0/2897273317 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fcc0402d410 con 0x55c85d0bf550 2024-04-25T11:36:18.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.497+0000 7fcc2104f740 10 client.15954 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:18.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.497+0000 7fcc2104f740 10 client.15954 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:36:18.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.497+0000 7fcc2104f740 20 client.15954 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:18.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.497+0000 7fcc2104f740 1 --2- 172.21.15.90:0/2897273317 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c85d07c630 0x55c85d50e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.497+0000 7fcc2104f740 4 client.15954 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T11:36:18.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.497+0000 7fcc2104f740 1 -- 172.21.15.90:0/2897273317 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c85d3f6580 con 0x55c85d07c630 2024-04-25T11:36:18.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.498+0000 7fcc12ffd640 1 --2- 172.21.15.90:0/2897273317 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c85d07c630 0x55c85d50e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.498+0000 7fcbf2ffd640 10 client.15954 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:18.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.498+0000 7fcc12ffd640 1 --2- 172.21.15.90:0/2897273317 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c85d07c630 0x55c85d50e830 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=799490cfab0e1115 server_cookie=ac2122373618c6a6 in_seq=0 out_seq=0 2024-04-25T11:36:18.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.500+0000 7fcbf2ffd640 1 -- 172.21.15.90:0/2897273317 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c85d3f6580 con 0x55c85d07c630 2024-04-25T11:36:18.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.500+0000 7fcbf2ffd640 10 client.15954 handle_command_reply: tid=0 2024-04-25T11:36:18.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.500+0000 7fcc2104f740 1 client.15954 shutdown 2024-04-25T11:36:18.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.500+0000 7fcc2104f740 1 -- 172.21.15.90:0/2897273317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0bf550 msgr2=0x55c85d499c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:18.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.500+0000 7fcc2104f740 1 --2- 172.21.15.90:0/2897273317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0bf550 0x55c85d499c30 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fcc04005fd0 tx=0x7fcc0400dd80 comp rx=0 tx=0).stop 2024-04-25T11:36:18.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.500+0000 7fcc2104f740 1 -- 172.21.15.90:0/2897273317 shutdown_connections 2024-04-25T11:36:18.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.500+0000 7fcc2104f740 1 --2- 172.21.15.90:0/2897273317 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c85d07c630 0x55c85d50e830 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.500+0000 7fcc2104f740 1 --2- 172.21.15.90:0/2897273317 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c85d0e8410 0x55c85d49a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.500+0000 7fcc2104f740 1 --2- 172.21.15.90:0/2897273317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c85d0c7730 0x55c85d49a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.500+0000 7fcc2104f740 1 --2- 172.21.15.90:0/2897273317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c85d0bf550 0x55c85d499c30 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.500+0000 7fcc2104f740 1 -- 172.21.15.90:0/2897273317 >> 172.21.15.90:0/2897273317 conn(0x55c85d0dc560 msgr2=0x55c85d484710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:18.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.500+0000 7fcbf2ffd640 0 client.15954 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:18.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.500+0000 7fcc2104f740 1 -- 172.21.15.90:0/2897273317 shutdown_connections 2024-04-25T11:36:18.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.501+0000 7fcc2104f740 1 -- 172.21.15.90:0/2897273317 wait complete. 2024-04-25T11:36:18.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.501+0000 7fcc2104f740 20 client.15954 trim_cache size 0 max 16384 2024-04-25T11:36:18.493 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:36:18.493 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-25T11:36:18.494 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "155f9be0-4590-4bbd-899f-01969b4ff704", 2024-04-25T11:36:18.494 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-25T11:36:18.494 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:36:18.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.501+0000 7fcc1a19d640 1 -- 172.21.15.90:0/147626646 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcbfc0afe00 msgr2=0x7fcbfc0b2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:18.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.501+0000 7fcc1a19d640 1 --2- 172.21.15.90:0/147626646 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcbfc0afe00 0x7fcbfc0b2310 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x55c85d485790 tx=0x7fcc0c0098a0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.501+0000 7fcc1a19d640 1 -- 172.21.15.90:0/147626646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc14108a20 msgr2=0x7fcc1419ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:18.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.501+0000 7fcc1a19d640 1 --2- 172.21.15.90:0/147626646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc14108a20 0x7fcc1419ac80 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fcc0802a930 tx=0x7fcc08004270 comp rx=0 tx=0).stop 2024-04-25T11:36:18.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.501+0000 7fcc1a19d640 1 -- 172.21.15.90:0/147626646 shutdown_connections 2024-04-25T11:36:18.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.501+0000 7fcc1a19d640 1 --2- 172.21.15.90:0/147626646 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcbfc0afe00 0x7fcbfc0b2310 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.501+0000 7fcc1a19d640 1 --2- 172.21.15.90:0/147626646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc14108ec0 0x7fcc1419ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.501+0000 7fcc1a19d640 1 --2- 172.21.15.90:0/147626646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc14108a20 0x7fcc1419ac80 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.501+0000 7fcc1a19d640 1 --2- 172.21.15.90:0/147626646 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcc14103180 0x7fcc1419a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.501+0000 7fcc1a19d640 1 -- 172.21.15.90:0/147626646 >> 172.21.15.90:0/147626646 conn(0x7fcc14078110 msgr2=0x7fcc14104100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:18.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.502+0000 7fcc1a19d640 1 -- 172.21.15.90:0/147626646 shutdown_connections 2024-04-25T11:36:18.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.502+0000 7fcc1a19d640 1 -- 172.21.15.90:0/147626646 wait complete. 2024-04-25T11:36:18.504 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:36:18.505 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:36:18.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.635+0000 7ff22239d640 1 -- 172.21.15.90:0/2299538313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c103a40 msgr2=0x7ff21c103e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:18.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.635+0000 7ff22239d640 1 --2- 172.21.15.90:0/2299538313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c103a40 0x7ff21c103e50 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7ff20c009900 tx=0x7ff20c02a410 comp rx=0 tx=0).stop 2024-04-25T11:36:18.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.635+0000 7ff22239d640 1 -- 172.21.15.90:0/2299538313 shutdown_connections 2024-04-25T11:36:18.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.635+0000 7ff22239d640 1 --2- 172.21.15.90:0/2299538313 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff21c10e2d0 0x7ff21c1106f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.635+0000 7ff22239d640 1 --2- 172.21.15.90:0/2299538313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c103a40 0x7ff21c103e50 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.635+0000 7ff22239d640 1 --2- 172.21.15.90:0/2299538313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff21c109970 0x7ff21c103500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.635+0000 7ff22239d640 1 -- 172.21.15.90:0/2299538313 >> 172.21.15.90:0/2299538313 conn(0x7ff21c0fefc0 msgr2=0x7ff21c101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:18.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.635+0000 7ff22239d640 1 -- 172.21.15.90:0/2299538313 shutdown_connections 2024-04-25T11:36:18.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.635+0000 7ff22239d640 1 -- 172.21.15.90:0/2299538313 wait complete. 2024-04-25T11:36:18.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.636+0000 7ff22239d640 1 Processor -- start 2024-04-25T11:36:18.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.636+0000 7ff22239d640 1 -- start start 2024-04-25T11:36:18.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.636+0000 7ff22239d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff21c103a40 0x7ff21c19a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.636+0000 7ff22239d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c109970 0x7ff21c19ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.636+0000 7ff22239d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff21c10e2d0 0x7ff21c19edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.636+0000 7ff22239d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff21c113b90 con 0x7ff21c109970 2024-04-25T11:36:18.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.636+0000 7ff22239d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff21c113a10 con 0x7ff21c103a40 2024-04-25T11:36:18.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.636+0000 7ff220b9a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c109970 0x7ff21c19ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.636+0000 7ff22239d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff21c113d10 con 0x7ff21c10e2d0 2024-04-25T11:36:18.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.636+0000 7ff220b9a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c109970 0x7ff21c19ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40282/0 (socket says 172.21.15.90:40282) 2024-04-25T11:36:18.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.636+0000 7ff220b9a640 1 -- 172.21.15.90:0/1823955761 learned_addr learned my addr 172.21.15.90:0/1823955761 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:18.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.637+0000 7ff220b9a640 1 -- 172.21.15.90:0/1823955761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff21c10e2d0 msgr2=0x7ff21c19edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:18.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.637+0000 7ff220b9a640 1 --2- 172.21.15.90:0/1823955761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff21c10e2d0 0x7ff21c19edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.637+0000 7ff220b9a640 1 -- 172.21.15.90:0/1823955761 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff21c103a40 msgr2=0x7ff21c19a5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:36:18.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.637+0000 7ff220b9a640 1 --2- 172.21.15.90:0/1823955761 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff21c103a40 0x7ff21c19a5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.637+0000 7ff220b9a640 1 -- 172.21.15.90:0/1823955761 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff21c19f410 con 0x7ff21c109970 2024-04-25T11:36:18.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.637+0000 7ff220b9a640 1 --2- 172.21.15.90:0/1823955761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c109970 0x7ff21c19ab10 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7ff20c02a930 tx=0x7ff20c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:18.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.637+0000 7ff2127fc640 1 -- 172.21.15.90:0/1823955761 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20c03f030 con 0x7ff21c109970 2024-04-25T11:36:18.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.637+0000 7ff2127fc640 1 -- 172.21.15.90:0/1823955761 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff20c003aa0 con 0x7ff21c109970 2024-04-25T11:36:18.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.637+0000 7ff2127fc640 1 -- 172.21.15.90:0/1823955761 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20c035070 con 0x7ff21c109970 2024-04-25T11:36:18.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.637+0000 7ff22239d640 1 -- 172.21.15.90:0/1823955761 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff21c19f6a0 con 0x7ff21c109970 2024-04-25T11:36:18.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.637+0000 7ff22239d640 1 -- 172.21.15.90:0/1823955761 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff21c19fc10 con 0x7ff21c109970 2024-04-25T11:36:18.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.640+0000 7ff2127fc640 1 -- 172.21.15.90:0/1823955761 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 24) ==== 147753+0+0 (secure 0 0 0) 0x7ff20c042030 con 0x7ff21c109970 2024-04-25T11:36:18.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.640+0000 7ff22a58d740 1 Processor -- start 2024-04-25T11:36:18.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.641+0000 7ff22a58d740 1 -- start start 2024-04-25T11:36:18.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.641+0000 7ff22a58d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bf6bd9c390 0x55bf6c13cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.641+0000 7ff22a58d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd734d0 0x55bf6c13d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.641+0000 7ff22a58d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bf6bd7b6b0 0x55bf6c149ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.641+0000 7ff22a58d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55bf6c0a2d80 con 0x55bf6bd734d0 2024-04-25T11:36:18.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.641+0000 7ff22a58d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55bf6c05c8b0 con 0x55bf6bd9c390 2024-04-25T11:36:18.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.641+0000 7ff22a58d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55bf6c09ed40 con 0x55bf6bd7b6b0 2024-04-25T11:36:18.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.641+0000 7ff22139b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd734d0 0x55bf6c13d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.641+0000 7ff22139b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd734d0 0x55bf6c13d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40296/0 (socket says 172.21.15.90:40296) 2024-04-25T11:36:18.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.641+0000 7ff22139b640 1 -- 172.21.15.90:0/249142052 learned_addr learned my addr 172.21.15.90:0/249142052 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:18.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.641+0000 7ff2127fc640 1 --2- 172.21.15.90:0/1823955761 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff2080afd70 0x7ff2080b2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.641+0000 7ff221b9c640 1 --2- 172.21.15.90:0/249142052 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bf6bd9c390 0x55bf6c13cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.641+0000 7ff220b9a640 1 --2- 172.21.15.90:0/249142052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bf6bd7b6b0 0x55bf6c149ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.641+0000 7ff221b9c640 1 --2- 172.21.15.90:0/1823955761 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff2080afd70 0x7ff2080b2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.641+0000 7ff22139b640 1 -- 172.21.15.90:0/249142052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bf6bd7b6b0 msgr2=0x55bf6c149ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:18.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.641+0000 7ff22139b640 1 --2- 172.21.15.90:0/249142052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bf6bd7b6b0 0x55bf6c149ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.641+0000 7ff22139b640 1 -- 172.21.15.90:0/249142052 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bf6bd9c390 msgr2=0x55bf6c13cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:18.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.642+0000 7ff22139b640 1 --2- 172.21.15.90:0/249142052 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bf6bd9c390 0x55bf6c13cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.642+0000 7ff22139b640 1 -- 172.21.15.90:0/249142052 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf6c02da40 con 0x55bf6bd734d0 2024-04-25T11:36:18.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.642+0000 7ff2127fc640 1 -- 172.21.15.90:0/1823955761 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7ff20c0ffab0 con 0x7ff21c109970 2024-04-25T11:36:18.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.642+0000 7ff22139b640 1 --2- 172.21.15.90:0/249142052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd734d0 0x55bf6c13d190 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff218009740 tx=0x7ff218009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b4066625503642c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:18.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.642+0000 7ff221b9c640 1 --2- 172.21.15.90:0/1823955761 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff2080afd70 0x7ff2080b2280 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x55bf6c139dc0 tx=0x7ff214009200 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:18.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.642+0000 7ff2077fe640 1 -- 172.21.15.90:0/249142052 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff218015020 con 0x55bf6bd734d0 2024-04-25T11:36:18.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.642+0000 7ff2077fe640 1 -- 172.21.15.90:0/249142052 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff218009e50 con 0x55bf6bd734d0 2024-04-25T11:36:18.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.642+0000 7ff2077fe640 1 -- 172.21.15.90:0/249142052 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff218010380 con 0x55bf6bd734d0 2024-04-25T11:36:18.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.643+0000 7ff22a58d740 1 -- 172.21.15.90:0/249142052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd734d0 msgr2=0x55bf6c13d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:18.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.643+0000 7ff22a58d740 1 --2- 172.21.15.90:0/249142052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd734d0 0x55bf6c13d190 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff218009740 tx=0x7ff218009c00 comp rx=0 tx=0).stop 2024-04-25T11:36:18.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.644+0000 7ff22a58d740 1 -- 172.21.15.90:0/249142052 shutdown_connections 2024-04-25T11:36:18.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.644+0000 7ff22a58d740 1 --2- 172.21.15.90:0/249142052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bf6bd7b6b0 0x55bf6c149ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.644+0000 7ff22a58d740 1 --2- 172.21.15.90:0/249142052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd734d0 0x55bf6c13d190 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.644+0000 7ff22a58d740 1 --2- 172.21.15.90:0/249142052 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bf6bd9c390 0x55bf6c13cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.644+0000 7ff22a58d740 1 -- 172.21.15.90:0/249142052 >> 172.21.15.90:0/249142052 conn(0x55bf6bd904e0 msgr2=0x55bf6c139400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:18.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.644+0000 7ff22a58d740 1 -- 172.21.15.90:0/249142052 shutdown_connections 2024-04-25T11:36:18.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.644+0000 7ff22a58d740 1 -- 172.21.15.90:0/249142052 wait complete. 2024-04-25T11:36:18.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.644+0000 7ff22a58d740 1 -- start start 2024-04-25T11:36:18.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.645+0000 7ff22a58d740 1 Processor -- start 2024-04-25T11:36:18.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.645+0000 7ff22a58d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bf6bd734d0 0x55bf6c1476e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.645+0000 7ff22a58d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd7b6b0 0x55bf6c147c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.645+0000 7ff22a58d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bf6bd9c390 0x55bf6c153740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.645+0000 7ff220b9a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd7b6b0 0x55bf6c147c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.645+0000 7ff22a58d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55bf6c14a1d0 con 0x55bf6bd7b6b0 2024-04-25T11:36:18.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.645+0000 7ff220b9a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd7b6b0 0x55bf6c147c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40310/0 (socket says 172.21.15.90:40310) 2024-04-25T11:36:18.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.645+0000 7ff220b9a640 1 -- 172.21.15.90:0/503524298 learned_addr learned my addr 172.21.15.90:0/503524298 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:18.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.645+0000 7ff22a58d740 1 -- 172.21.15.90:0/503524298 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55bf6c029d10 con 0x55bf6bd734d0 2024-04-25T11:36:18.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.645+0000 7ff22139b640 1 --2- 172.21.15.90:0/503524298 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bf6bd734d0 0x55bf6c1476e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.646+0000 7ff22a58d740 1 -- 172.21.15.90:0/503524298 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55bf6c14a350 con 0x55bf6bd9c390 2024-04-25T11:36:18.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.646+0000 7ff22a58d740 10 client.0 _ll_register_callbacks cb 0x55bf6c056210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:18.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.646+0000 7ff220b9a640 1 -- 172.21.15.90:0/503524298 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bf6bd9c390 msgr2=0x55bf6c153740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:18.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.646+0000 7ff221b9c640 1 --2- 172.21.15.90:0/503524298 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bf6bd9c390 0x55bf6c153740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.646+0000 7ff220b9a640 1 --2- 172.21.15.90:0/503524298 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bf6bd9c390 0x55bf6c153740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.646+0000 7ff220b9a640 1 -- 172.21.15.90:0/503524298 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bf6bd734d0 msgr2=0x55bf6c1476e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:18.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.646+0000 7ff220b9a640 1 --2- 172.21.15.90:0/503524298 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bf6bd734d0 0x55bf6c1476e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.646+0000 7ff220b9a640 1 -- 172.21.15.90:0/503524298 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf6be7e7d0 con 0x55bf6bd7b6b0 2024-04-25T11:36:18.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.646+0000 7ff22139b640 1 --2- 172.21.15.90:0/503524298 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bf6bd734d0 0x55bf6c1476e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:36:18.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.646+0000 7ff220b9a640 1 --2- 172.21.15.90:0/503524298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd7b6b0 0x55bf6c147c20 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0c7710 tx=0x7ff20c0921e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbfef7bc907cc5c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:18.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.646+0000 7ff221b9c640 1 --2- 172.21.15.90:0/503524298 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bf6bd9c390 0x55bf6c153740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:36:18.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.646+0000 7ff204ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:18.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.646+0000 7ff22a58d740 1 -- 172.21.15.90:0/503524298 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff218009e50 con 0x55bf6bd7b6b0 2024-04-25T11:36:18.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.646+0000 7ff204ff9640 1 -- 172.21.15.90:0/503524298 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20c0aee60 con 0x55bf6bd7b6b0 2024-04-25T11:36:18.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.646+0000 7ff204ff9640 1 -- 172.21.15.90:0/503524298 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7ff20c003c50 con 0x55bf6bd7b6b0 2024-04-25T11:36:18.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.647+0000 7ff204ff9640 1 -- 172.21.15.90:0/503524298 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff20c049040 con 0x55bf6bd7b6b0 2024-04-25T11:36:18.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.647+0000 7ff22a58d740 10 client.15966 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:18.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.647+0000 7ff22a58d740 1 -- 172.21.15.90:0/503524298 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=26}) -- 0x55bf6bdb43f0 con 0x55bf6bd7b6b0 2024-04-25T11:36:18.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.647+0000 7ff204ff9640 1 -- 172.21.15.90:0/503524298 <== mon.0 v2:172.21.15.90:3300/0 4 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7ff20c0c3ca0 con 0x55bf6bd7b6b0 2024-04-25T11:36:18.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.647+0000 7ff22a58d740 10 client.15966 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:18.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.647+0000 7ff22a58d740 10 client.15966 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:36:18.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.647+0000 7ff22a58d740 20 client.15966 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:18.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.647+0000 7ff22a58d740 1 --2- 172.21.15.90:0/503524298 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55bf6bd5af60 0x55bf6c1595c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.647+0000 7ff22a58d740 4 client.15966 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:36:18.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.647+0000 7ff22a58d740 1 -- 172.21.15.90:0/503524298 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf6c0b0990 con 0x55bf6bd5af60 2024-04-25T11:36:18.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.647+0000 7ff22139b640 1 --2- 172.21.15.90:0/503524298 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55bf6bd5af60 0x55bf6c1595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.648+0000 7ff204ff9640 10 client.15966 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:18.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.648+0000 7ff22139b640 1 --2- 172.21.15.90:0/503524298 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55bf6bd5af60 0x55bf6c1595c0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99d78ce245690a52 server_cookie=c5528179421db2de in_seq=0 out_seq=0 2024-04-25T11:36:18.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.649+0000 7ff204ff9640 1 -- 172.21.15.90:0/503524298 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55bf6c0b0990 con 0x55bf6bd5af60 2024-04-25T11:36:18.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.649+0000 7ff204ff9640 10 client.15966 handle_command_reply: tid=0 2024-04-25T11:36:18.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.649+0000 7ff22a58d740 1 client.15966 shutdown 2024-04-25T11:36:18.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.649+0000 7ff22a58d740 1 -- 172.21.15.90:0/503524298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd7b6b0 msgr2=0x55bf6c147c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:18.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.649+0000 7ff22a58d740 1 --2- 172.21.15.90:0/503524298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd7b6b0 0x55bf6c147c20 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0c7710 tx=0x7ff20c0921e0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.649+0000 7ff22a58d740 1 -- 172.21.15.90:0/503524298 shutdown_connections 2024-04-25T11:36:18.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.650+0000 7ff22a58d740 1 --2- 172.21.15.90:0/503524298 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55bf6bd5af60 0x55bf6c1595c0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.650+0000 7ff22a58d740 1 --2- 172.21.15.90:0/503524298 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bf6bd9c390 0x55bf6c153740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.650+0000 7ff204ff9640 0 client.15966 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:18.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.650+0000 7ff22a58d740 1 --2- 172.21.15.90:0/503524298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd7b6b0 0x55bf6c147c20 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.650+0000 7ff22a58d740 1 --2- 172.21.15.90:0/503524298 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bf6bd734d0 0x55bf6c1476e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.650+0000 7ff22a58d740 1 -- 172.21.15.90:0/503524298 >> 172.21.15.90:0/503524298 conn(0x55bf6bd904e0 msgr2=0x55bf6c13e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:18.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.650+0000 7ff22a58d740 1 -- 172.21.15.90:0/503524298 shutdown_connections 2024-04-25T11:36:18.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.650+0000 7ff22a58d740 1 -- 172.21.15.90:0/503524298 wait complete. 2024-04-25T11:36:18.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.650+0000 7ff22a58d740 20 client.15966 trim_cache size 0 max 16384 2024-04-25T11:36:18.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.679+0000 7ff22a58d740 1 Processor -- start 2024-04-25T11:36:18.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.679+0000 7ff22a58d740 1 -- start start 2024-04-25T11:36:18.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.680+0000 7ff22a58d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd734d0 0x55bf6c141900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.680+0000 7ff22139b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd734d0 0x55bf6c141900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.680+0000 7ff22a58d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bf6bd7b6b0 0x55bf6c15ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.680+0000 7ff22139b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd734d0 0x55bf6c141900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40320/0 (socket says 172.21.15.90:40320) 2024-04-25T11:36:18.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.680+0000 7ff22139b640 1 -- 172.21.15.90:0/4106219127 learned_addr learned my addr 172.21.15.90:0/4106219127 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:18.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.680+0000 7ff22a58d740 1 --2- 172.21.15.90:0/4106219127 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bf6bd9c390 0x55bf6c15f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.680+0000 7ff22a58d740 1 -- 172.21.15.90:0/4106219127 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff21c1afa00 con 0x55bf6bd734d0 2024-04-25T11:36:18.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.680+0000 7ff22a58d740 1 -- 172.21.15.90:0/4106219127 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff21c1b1540 con 0x55bf6bd9c390 2024-04-25T11:36:18.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.680+0000 7ff22a58d740 1 -- 172.21.15.90:0/4106219127 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff21c1b00d0 con 0x55bf6bd7b6b0 2024-04-25T11:36:18.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.680+0000 7ff220b9a640 1 --2- 172.21.15.90:0/4106219127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bf6bd7b6b0 0x55bf6c15ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.680+0000 7ff22139b640 1 -- 172.21.15.90:0/4106219127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bf6bd7b6b0 msgr2=0x55bf6c15ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:18.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.680+0000 7ff22139b640 1 --2- 172.21.15.90:0/4106219127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bf6bd7b6b0 0x55bf6c15ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.680+0000 7ff221b9c640 1 --2- 172.21.15.90:0/4106219127 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bf6bd9c390 0x55bf6c15f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.680+0000 7ff22139b640 1 -- 172.21.15.90:0/4106219127 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bf6bd9c390 msgr2=0x55bf6c15f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:18.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.680+0000 7ff22139b640 1 --2- 172.21.15.90:0/4106219127 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bf6bd9c390 0x55bf6c15f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.680+0000 7ff22139b640 1 -- 172.21.15.90:0/4106219127 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf6c146480 con 0x55bf6bd734d0 2024-04-25T11:36:18.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.681+0000 7ff22139b640 1 --2- 172.21.15.90:0/4106219127 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd734d0 0x55bf6c141900 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7ff218011ab0 tx=0x7ff218009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5575538dbdad2f4e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:18.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.681+0000 7ff206ffd640 1 -- 172.21.15.90:0/4106219127 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff218015040 con 0x55bf6bd734d0 2024-04-25T11:36:18.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.681+0000 7ff206ffd640 1 -- 172.21.15.90:0/4106219127 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff218027e50 con 0x55bf6bd734d0 2024-04-25T11:36:18.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.681+0000 7ff206ffd640 1 -- 172.21.15.90:0/4106219127 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff218003690 con 0x55bf6bd734d0 2024-04-25T11:36:18.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.682+0000 7ff22a58d740 1 -- 172.21.15.90:0/4106219127 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd734d0 msgr2=0x55bf6c141900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:18.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.682+0000 7ff22a58d740 1 --2- 172.21.15.90:0/4106219127 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd734d0 0x55bf6c141900 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7ff218011ab0 tx=0x7ff218009250 comp rx=0 tx=0).stop 2024-04-25T11:36:18.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.682+0000 7ff22a58d740 1 -- 172.21.15.90:0/4106219127 shutdown_connections 2024-04-25T11:36:18.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.682+0000 7ff22a58d740 1 --2- 172.21.15.90:0/4106219127 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bf6bd9c390 0x55bf6c15f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.682+0000 7ff22a58d740 1 --2- 172.21.15.90:0/4106219127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bf6bd7b6b0 0x55bf6c15ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.682+0000 7ff22a58d740 1 --2- 172.21.15.90:0/4106219127 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd734d0 0x55bf6c141900 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.682+0000 7ff22a58d740 1 -- 172.21.15.90:0/4106219127 >> 172.21.15.90:0/4106219127 conn(0x55bf6bd904e0 msgr2=0x55bf6c142740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:18.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.682+0000 7ff22a58d740 1 -- 172.21.15.90:0/4106219127 shutdown_connections 2024-04-25T11:36:18.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.682+0000 7ff22a58d740 1 -- 172.21.15.90:0/4106219127 wait complete. 2024-04-25T11:36:18.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.682+0000 7ff22a58d740 1 -- start start 2024-04-25T11:36:18.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.682+0000 7ff22a58d740 1 Processor -- start 2024-04-25T11:36:18.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.683+0000 7ff22a58d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd734d0 0x55bf6c1b9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.683+0000 7ff22a58d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bf6bd7b6b0 0x55bf6c1ba1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.683+0000 7ff22a58d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bf6bd9c390 0x55bf6c14aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.683+0000 7ff22a58d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff21c1b0e70 con 0x55bf6bd734d0 2024-04-25T11:36:18.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.683+0000 7ff22a58d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff21c1b07a0 con 0x55bf6bd7b6b0 2024-04-25T11:36:18.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.683+0000 7ff22a58d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff21c1ad9e0 con 0x55bf6bd9c390 2024-04-25T11:36:18.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.683+0000 7ff22a58d740 10 client.0 _ll_register_callbacks cb 0x7ff20c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:18.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.683+0000 7ff22139b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd734d0 0x55bf6c1b9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.683+0000 7ff22139b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd734d0 0x55bf6c1b9c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40322/0 (socket says 172.21.15.90:40322) 2024-04-25T11:36:18.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.683+0000 7ff22139b640 1 -- 172.21.15.90:0/3522645485 learned_addr learned my addr 172.21.15.90:0/3522645485 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:18.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.683+0000 7ff220b9a640 1 --2- 172.21.15.90:0/3522645485 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bf6bd7b6b0 0x55bf6c1ba1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.683+0000 7ff221b9c640 1 --2- 172.21.15.90:0/3522645485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bf6bd9c390 0x55bf6c14aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.683+0000 7ff22139b640 1 -- 172.21.15.90:0/3522645485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bf6bd9c390 msgr2=0x55bf6c14aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:18.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.683+0000 7ff22139b640 1 --2- 172.21.15.90:0/3522645485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bf6bd9c390 0x55bf6c14aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.683+0000 7ff22139b640 1 -- 172.21.15.90:0/3522645485 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bf6bd7b6b0 msgr2=0x55bf6c1ba1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:18.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.683+0000 7ff22139b640 1 --2- 172.21.15.90:0/3522645485 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bf6bd7b6b0 0x55bf6c1ba1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.683+0000 7ff22139b640 1 -- 172.21.15.90:0/3522645485 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf6bd740f0 con 0x55bf6bd734d0 2024-04-25T11:36:18.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.683+0000 7ff221b9c640 1 --2- 172.21.15.90:0/3522645485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bf6bd9c390 0x55bf6c14aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:36:18.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.684+0000 7ff22139b640 1 --2- 172.21.15.90:0/3522645485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd734d0 0x55bf6c1b9c70 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7ff218000c00 tx=0x7ff218004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4e8265196ba9568 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:18.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.684+0000 7ff204ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:18.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.684+0000 7ff204ff9640 1 -- 172.21.15.90:0/3522645485 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff218037060 con 0x55bf6bd734d0 2024-04-25T11:36:18.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.684+0000 7ff204ff9640 1 -- 172.21.15.90:0/3522645485 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7ff21800b8d0 con 0x55bf6bd734d0 2024-04-25T11:36:18.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.684+0000 7ff22a58d740 1 -- 172.21.15.90:0/3522645485 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff218027e50 con 0x55bf6bd734d0 2024-04-25T11:36:18.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.684+0000 7ff204ff9640 1 -- 172.21.15.90:0/3522645485 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff218015040 con 0x55bf6bd734d0 2024-04-25T11:36:18.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.684+0000 7ff22a58d740 10 client.15972 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:18.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.684+0000 7ff22a58d740 1 -- 172.21.15.90:0/3522645485 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55bf6c0634b0 con 0x55bf6bd734d0 2024-04-25T11:36:18.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.684+0000 7ff204ff9640 1 -- 172.21.15.90:0/3522645485 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff218003690 con 0x55bf6bd734d0 2024-04-25T11:36:18.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.684+0000 7ff204ff9640 1 -- 172.21.15.90:0/3522645485 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7ff218024070 con 0x55bf6bd734d0 2024-04-25T11:36:18.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.685+0000 7ff22a58d740 10 client.15972 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:18.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.685+0000 7ff22a58d740 10 client.15972 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:36:18.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.685+0000 7ff22a58d740 20 client.15972 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:18.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.685+0000 7ff22a58d740 1 --2- 172.21.15.90:0/3522645485 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55bf6bd303a0 0x55bf6c1c7820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:18.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.685+0000 7ff22a58d740 4 client.15972 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:36:18.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.685+0000 7ff22a58d740 1 -- 172.21.15.90:0/3522645485 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7ff218027e50 con 0x55bf6bd303a0 2024-04-25T11:36:18.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.685+0000 7ff220b9a640 1 --2- 172.21.15.90:0/3522645485 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55bf6bd303a0 0x55bf6c1c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:18.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.685+0000 7ff204ff9640 10 client.15972 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:18.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.685+0000 7ff220b9a640 1 --2- 172.21.15.90:0/3522645485 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55bf6bd303a0 0x55bf6c1c7820 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b787dc0992f8137 server_cookie=d81b735183f57838 in_seq=0 out_seq=0 2024-04-25T11:36:18.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.686+0000 7ff204ff9640 1 -- 172.21.15.90:0/3522645485 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7ff218027e50 con 0x55bf6bd303a0 2024-04-25T11:36:18.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.686+0000 7ff204ff9640 10 client.15972 handle_command_reply: tid=0 2024-04-25T11:36:18.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.686+0000 7ff22a58d740 1 client.15972 shutdown 2024-04-25T11:36:18.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.686+0000 7ff22a58d740 1 -- 172.21.15.90:0/3522645485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd734d0 msgr2=0x55bf6c1b9c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.686+0000 7ff22a58d740 1 --2- 172.21.15.90:0/3522645485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd734d0 0x55bf6c1b9c70 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7ff218000c00 tx=0x7ff218004270 comp rx=0 tx=0).stop 2024-04-25T11:36:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.686+0000 7ff22a58d740 1 -- 172.21.15.90:0/3522645485 shutdown_connections 2024-04-25T11:36:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.686+0000 7ff22a58d740 1 --2- 172.21.15.90:0/3522645485 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55bf6bd303a0 0x55bf6c1c7820 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.686+0000 7ff22a58d740 1 --2- 172.21.15.90:0/3522645485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bf6bd9c390 0x55bf6c14aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.686+0000 7ff204ff9640 0 client.15972 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.686+0000 7ff22a58d740 1 --2- 172.21.15.90:0/3522645485 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bf6bd7b6b0 0x55bf6c1ba1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.686+0000 7ff22a58d740 1 --2- 172.21.15.90:0/3522645485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf6bd734d0 0x55bf6c1b9c70 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.686+0000 7ff22a58d740 1 -- 172.21.15.90:0/3522645485 >> 172.21.15.90:0/3522645485 conn(0x55bf6bd904e0 msgr2=0x55bf6c13e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.686+0000 7ff22a58d740 1 -- 172.21.15.90:0/3522645485 shutdown_connections 2024-04-25T11:36:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.686+0000 7ff22a58d740 1 -- 172.21.15.90:0/3522645485 wait complete. 2024-04-25T11:36:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.686+0000 7ff22a58d740 20 client.15972 trim_cache size 0 max 16384 2024-04-25T11:36:18.682 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:36:18.682 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T11:36:18.682 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-25T11:36:18.682 INFO:teuthology.orchestra.run.smithi090.stdout: "155f9be0-4590-4bbd-899f-01969b4ff704": { 2024-04-25T11:36:18.682 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-25T11:36:18.682 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "155f9be0-4590-4bbd-899f-01969b4ff704", 2024-04-25T11:36:18.682 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-25T11:36:18.682 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:36:18.682 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:36:18.682 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:36:18.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.687+0000 7ff22239d640 1 -- 172.21.15.90:0/1823955761 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff2080afd70 msgr2=0x7ff2080b2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:18.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.687+0000 7ff22239d640 1 --2- 172.21.15.90:0/1823955761 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff2080afd70 0x7ff2080b2280 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x55bf6c139dc0 tx=0x7ff214009200 comp rx=0 tx=0).stop 2024-04-25T11:36:18.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.687+0000 7ff22239d640 1 -- 172.21.15.90:0/1823955761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c109970 msgr2=0x7ff21c19ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:18.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.687+0000 7ff22239d640 1 --2- 172.21.15.90:0/1823955761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c109970 0x7ff21c19ab10 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7ff20c02a930 tx=0x7ff20c004270 comp rx=0 tx=0).stop 2024-04-25T11:36:18.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.687+0000 7ff22239d640 1 -- 172.21.15.90:0/1823955761 shutdown_connections 2024-04-25T11:36:18.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.687+0000 7ff22239d640 1 --2- 172.21.15.90:0/1823955761 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff2080afd70 0x7ff2080b2280 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.687+0000 7ff22239d640 1 --2- 172.21.15.90:0/1823955761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff21c10e2d0 0x7ff21c19edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.687+0000 7ff22239d640 1 --2- 172.21.15.90:0/1823955761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c109970 0x7ff21c19ab10 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.687+0000 7ff22239d640 1 --2- 172.21.15.90:0/1823955761 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff21c103a40 0x7ff21c19a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:18.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.687+0000 7ff22239d640 1 -- 172.21.15.90:0/1823955761 >> 172.21.15.90:0/1823955761 conn(0x7ff21c0fefc0 msgr2=0x7ff21c10f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:18.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.687+0000 7ff22239d640 1 -- 172.21.15.90:0/1823955761 shutdown_connections 2024-04-25T11:36:18.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:18.687+0000 7ff22239d640 1 -- 172.21.15.90:0/1823955761 wait complete. 2024-04-25T11:36:18.689 INFO:tasks.cephfs.filesystem:scrub status for tag:155f9be0-4590-4bbd-899f-01969b4ff704 - {'path': '/', 'tag': '155f9be0-4590-4bbd-899f-01969b4ff704', 'options': 'recursive,force'} 2024-04-25T11:36:19.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:19 smithi090 ceph-mon[29575]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 754 B/s wr, 4 op/s; 205 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T11:36:19.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:19 smithi138 ceph-mon[32328]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 754 B/s wr, 4 op/s; 205 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T11:36:19.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:19 smithi156 ceph-mon[33045]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 754 B/s wr, 4 op/s; 205 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T11:36:20.694 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.705+0000 7fb3cc334640 1 -- 172.21.15.90:0/1576833979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3c4102450 msgr2=0x7fb3c4102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:20.694 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.705+0000 7fb3cc334640 1 --2- 172.21.15.90:0/1576833979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3c4102450 0x7fb3c4102860 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0009900 tx=0x7fb3c002a410 comp rx=0 tx=0).stop 2024-04-25T11:36:20.694 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.705+0000 7fb3cc334640 1 -- 172.21.15.90:0/1576833979 shutdown_connections 2024-04-25T11:36:20.695 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.705+0000 7fb3cc334640 1 --2- 172.21.15.90:0/1576833979 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb3c41074a0 0x7fb3c410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.695 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.705+0000 7fb3cc334640 1 --2- 172.21.15.90:0/1576833979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3c41069e0 0x7fb3c4106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.695 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.705+0000 7fb3cc334640 1 --2- 172.21.15.90:0/1576833979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3c4102450 0x7fb3c4102860 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.695 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.705+0000 7fb3cc334640 1 -- 172.21.15.90:0/1576833979 >> 172.21.15.90:0/1576833979 conn(0x7fb3c40fd8b0 msgr2=0x7fb3c40ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:20.695 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.705+0000 7fb3cc334640 1 -- 172.21.15.90:0/1576833979 shutdown_connections 2024-04-25T11:36:20.695 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.705+0000 7fb3cc334640 1 -- 172.21.15.90:0/1576833979 wait complete. 2024-04-25T11:36:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.706+0000 7fb3cc334640 1 Processor -- start 2024-04-25T11:36:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.706+0000 7fb3cc334640 1 -- start start 2024-04-25T11:36:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.706+0000 7fb3cc334640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3c4102450 0x7fb3c419a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.706+0000 7fb3cc334640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3c41069e0 0x7fb3c419acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.706+0000 7fb3cc334640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb3c41074a0 0x7fb3c419ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.707+0000 7fb3cc334640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c4113d70 con 0x7fb3c4102450 2024-04-25T11:36:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.707+0000 7fb3cc334640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c4113bf0 con 0x7fb3c41074a0 2024-04-25T11:36:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.707+0000 7fb3cc334640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c4113ef0 con 0x7fb3c41069e0 2024-04-25T11:36:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.707+0000 7fb3ca0d0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3c4102450 0x7fb3c419a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.707+0000 7fb3ca0d0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3c4102450 0x7fb3c419a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40342/0 (socket says 172.21.15.90:40342) 2024-04-25T11:36:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.707+0000 7fb3ca0d0640 1 -- 172.21.15.90:0/1842454845 learned_addr learned my addr 172.21.15.90:0/1842454845 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.707+0000 7fb3c98cf640 1 --2- 172.21.15.90:0/1842454845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3c41069e0 0x7fb3c419acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.707+0000 7fb3ca8d1640 1 --2- 172.21.15.90:0/1842454845 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb3c41074a0 0x7fb3c419ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.707+0000 7fb3ca0d0640 1 -- 172.21.15.90:0/1842454845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3c41069e0 msgr2=0x7fb3c419acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.707+0000 7fb3ca0d0640 1 --2- 172.21.15.90:0/1842454845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3c41069e0 0x7fb3c419acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.707+0000 7fb3ca0d0640 1 -- 172.21.15.90:0/1842454845 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb3c41074a0 msgr2=0x7fb3c419ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.707+0000 7fb3ca0d0640 1 --2- 172.21.15.90:0/1842454845 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb3c41074a0 0x7fb3c419ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.707+0000 7fb3ca0d0640 1 -- 172.21.15.90:0/1842454845 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3c419f5d0 con 0x7fb3c4102450 2024-04-25T11:36:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.707+0000 7fb3ca0d0640 1 --2- 172.21.15.90:0/1842454845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3c4102450 0x7fb3c419a770 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0005940 tx=0x7fb3c00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.707+0000 7fb3bb7fe640 1 -- 172.21.15.90:0/1842454845 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c0035070 con 0x7fb3c4102450 2024-04-25T11:36:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.707+0000 7fb3bb7fe640 1 -- 172.21.15.90:0/1842454845 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3c0041d60 con 0x7fb3c4102450 2024-04-25T11:36:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.708+0000 7fb3cc334640 1 -- 172.21.15.90:0/1842454845 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3c419f860 con 0x7fb3c4102450 2024-04-25T11:36:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.708+0000 7fb3bb7fe640 1 -- 172.21.15.90:0/1842454845 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c0049630 con 0x7fb3c4102450 2024-04-25T11:36:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.708+0000 7fb3cc334640 1 -- 172.21.15.90:0/1842454845 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3c419fda0 con 0x7fb3c4102450 2024-04-25T11:36:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.710+0000 7fb3bb7fe640 1 -- 172.21.15.90:0/1842454845 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 24) ==== 147753+0+0 (secure 0 0 0) 0x7fb3c00035f0 con 0x7fb3c4102450 2024-04-25T11:36:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.711+0000 7fb3bb7fe640 1 --2- 172.21.15.90:0/1842454845 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb3b00afdf0 0x7fb3b00b2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.711+0000 7fb3d31e7740 1 Processor -- start 2024-04-25T11:36:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.711+0000 7fb3bb7fe640 1 -- 172.21.15.90:0/1842454845 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb3c01082b0 con 0x7fb3c4102450 2024-04-25T11:36:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.711+0000 7fb3c98cf640 1 --2- 172.21.15.90:0/1842454845 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb3b00afdf0 0x7fb3b00b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.712+0000 7fb3d31e7740 1 -- start start 2024-04-25T11:36:20.702 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.712+0000 7fb3c98cf640 1 --2- 172.21.15.90:0/1842454845 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb3b00afdf0 0x7fb3b00b2300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4101250 tx=0x7fb3b400b390 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:20.702 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.712+0000 7fb3d31e7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa8481b0 0x5615fabe7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:20.702 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.712+0000 7fb3d31e7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5615fa81f2f0 0x5615fabe8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:20.702 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.712+0000 7fb3d31e7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5615fa8274d0 0x5615fabe8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:20.703 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.712+0000 7fb3d31e7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5615fab4e460 con 0x5615fa8481b0 2024-04-25T11:36:20.703 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.712+0000 7fb3d31e7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5615fab08590 con 0x5615fa8274d0 2024-04-25T11:36:20.703 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.712+0000 7fb3d31e7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5615fab4a420 con 0x5615fa81f2f0 2024-04-25T11:36:20.703 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.713+0000 7fb3ca0d0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa8481b0 0x5615fabe7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:20.703 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.713+0000 7fb3ca8d1640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5615fa8274d0 0x5615fabe8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:20.703 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.713+0000 7fb3ca0d0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa8481b0 0x5615fabe7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40354/0 (socket says 172.21.15.90:40354) 2024-04-25T11:36:20.703 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.713+0000 7fb3ca0d0640 1 -- 172.21.15.90:0/1659411745 learned_addr learned my addr 172.21.15.90:0/1659411745 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:20.703 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.713+0000 7fb3c98cf640 1 --2- 172.21.15.90:0/1659411745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5615fa81f2f0 0x5615fabe8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:20.703 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.713+0000 7fb3ca0d0640 1 -- 172.21.15.90:0/1659411745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5615fa81f2f0 msgr2=0x5615fabe8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:20.703 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.713+0000 7fb3ca0d0640 1 --2- 172.21.15.90:0/1659411745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5615fa81f2f0 0x5615fabe8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.703 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.713+0000 7fb3ca0d0640 1 -- 172.21.15.90:0/1659411745 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5615fa8274d0 msgr2=0x5615fabe8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:20.703 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.713+0000 7fb3ca0d0640 1 --2- 172.21.15.90:0/1659411745 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5615fa8274d0 0x5615fabe8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.713+0000 7fb3ca0d0640 1 -- 172.21.15.90:0/1659411745 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615fab150d0 con 0x5615fa8481b0 2024-04-25T11:36:20.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.713+0000 7fb3ca0d0640 1 --2- 172.21.15.90:0/1659411745 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa8481b0 0x5615fabe7af0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fb3c00d0080 tx=0x7fb3c00cfaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66c6fa13877df5ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:20.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.713+0000 7fb3b8bfe640 1 -- 172.21.15.90:0/1659411745 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c00cfe90 con 0x5615fa8481b0 2024-04-25T11:36:20.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.714+0000 7fb3b8bfe640 1 -- 172.21.15.90:0/1659411745 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3c00d0a80 con 0x5615fa8481b0 2024-04-25T11:36:20.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.714+0000 7fb3b8bfe640 1 -- 172.21.15.90:0/1659411745 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c00d8d50 con 0x5615fa8481b0 2024-04-25T11:36:20.710 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.715+0000 7fb3d31e7740 1 -- 172.21.15.90:0/1659411745 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa8481b0 msgr2=0x5615fabe7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:20.710 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.715+0000 7fb3d31e7740 1 --2- 172.21.15.90:0/1659411745 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa8481b0 0x5615fabe7af0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fb3c00d0080 tx=0x7fb3c00cfaf0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.710 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.715+0000 7fb3d31e7740 1 -- 172.21.15.90:0/1659411745 shutdown_connections 2024-04-25T11:36:20.710 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.715+0000 7fb3d31e7740 1 --2- 172.21.15.90:0/1659411745 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5615fa8274d0 0x5615fabe8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.710 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.715+0000 7fb3d31e7740 1 --2- 172.21.15.90:0/1659411745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5615fa81f2f0 0x5615fabe8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.710 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.715+0000 7fb3d31e7740 1 --2- 172.21.15.90:0/1659411745 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa8481b0 0x5615fabe7af0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.710 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.715+0000 7fb3d31e7740 1 -- 172.21.15.90:0/1659411745 >> 172.21.15.90:0/1659411745 conn(0x5615fa83c300 msgr2=0x5615fabe42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:20.710 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.715+0000 7fb3d31e7740 1 -- 172.21.15.90:0/1659411745 shutdown_connections 2024-04-25T11:36:20.710 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.715+0000 7fb3d31e7740 1 -- 172.21.15.90:0/1659411745 wait complete. 2024-04-25T11:36:20.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.715+0000 7fb3d31e7740 1 -- start start 2024-04-25T11:36:20.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.716+0000 7fb3d31e7740 1 Processor -- start 2024-04-25T11:36:20.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.716+0000 7fb3d31e7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5615fa81f2f0 0x5615fabf3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:20.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.716+0000 7fb3d31e7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa8274d0 0x5615fabf41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:20.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.716+0000 7fb3c98cf640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa8274d0 0x5615fabf41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:20.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.716+0000 7fb3c98cf640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa8274d0 0x5615fabf41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40368/0 (socket says 172.21.15.90:40368) 2024-04-25T11:36:20.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.716+0000 7fb3c98cf640 1 -- 172.21.15.90:0/1934598207 learned_addr learned my addr 172.21.15.90:0/1934598207 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:20.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.716+0000 7fb3d31e7740 1 --2- 172.21.15.90:0/1934598207 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5615fa8481b0 0x5615fabf20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:20.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.716+0000 7fb3d31e7740 1 -- 172.21.15.90:0/1934598207 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5615fabf49e0 con 0x5615fa8274d0 2024-04-25T11:36:20.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.716+0000 7fb3ca0d0640 1 --2- 172.21.15.90:0/1934598207 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5615fa81f2f0 0x5615fabf3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:20.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.716+0000 7fb3d31e7740 1 -- 172.21.15.90:0/1934598207 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5615faad5b60 con 0x5615fa8481b0 2024-04-25T11:36:20.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.716+0000 7fb3d31e7740 1 -- 172.21.15.90:0/1934598207 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5615fabf4b60 con 0x5615fa81f2f0 2024-04-25T11:36:20.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.717+0000 7fb3d31e7740 10 client.0 _ll_register_callbacks cb 0x5615faafa2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.717+0000 7fb3ca8d1640 1 --2- 172.21.15.90:0/1934598207 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5615fa8481b0 0x5615fabf20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.717+0000 7fb3c98cf640 1 -- 172.21.15.90:0/1934598207 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5615fa81f2f0 msgr2=0x5615fabf3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.717+0000 7fb3c98cf640 1 --2- 172.21.15.90:0/1934598207 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5615fa81f2f0 0x5615fabf3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.717+0000 7fb3c98cf640 1 -- 172.21.15.90:0/1934598207 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5615fa8481b0 msgr2=0x5615fabf20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.717+0000 7fb3c98cf640 1 --2- 172.21.15.90:0/1934598207 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5615fa8481b0 0x5615fabf20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.717+0000 7fb3c98cf640 1 -- 172.21.15.90:0/1934598207 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615fa860210 con 0x5615fa8274d0 2024-04-25T11:36:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.717+0000 7fb3ca8d1640 1 --2- 172.21.15.90:0/1934598207 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5615fa8481b0 0x5615fabf20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:36:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.717+0000 7fb3ca0d0640 1 --2- 172.21.15.90:0/1934598207 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5615fa81f2f0 0x5615fabf3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:36:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.717+0000 7fb3d31e7740 1 -- 172.21.15.90:0/1934598207 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3c00d0a80 con 0x5615fa8274d0 2024-04-25T11:36:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.717+0000 7fb3c98cf640 1 --2- 172.21.15.90:0/1934598207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa8274d0 0x5615fabf41d0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4005fd0 tx=0x7fb3b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e32d0705b59f9340 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.717+0000 7fb3adffb640 10 client.15984 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.717+0000 7fb3adffb640 1 -- 172.21.15.90:0/1934598207 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b401e830 con 0x5615fa8274d0 2024-04-25T11:36:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.717+0000 7fb3adffb640 1 -- 172.21.15.90:0/1934598207 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb3b401e9d0 con 0x5615fa8274d0 2024-04-25T11:36:20.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.717+0000 7fb3adffb640 1 -- 172.21.15.90:0/1934598207 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb3b401ebf0 con 0x5615fa8274d0 2024-04-25T11:36:20.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.717+0000 7fb3d31e7740 10 client.15984 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:20.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.717+0000 7fb3d31e7740 1 -- 172.21.15.90:0/1934598207 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=26}) -- 0x5615fa92c620 con 0x5615fa8274d0 2024-04-25T11:36:20.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.719+0000 7fb3adffb640 1 -- 172.21.15.90:0/1934598207 <== mon.0 v2:172.21.15.90:3300/0 4 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fb3b4026020 con 0x5615fa8274d0 2024-04-25T11:36:20.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.719+0000 7fb3d31e7740 10 client.15984 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:20.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.719+0000 7fb3d31e7740 10 client.15984 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:36:20.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.719+0000 7fb3d31e7740 20 client.15984 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:20.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.719+0000 7fb3d31e7740 1 --2- 172.21.15.90:0/1934598207 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5615fa806d80 0x5615fac04350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:20.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.719+0000 7fb3d31e7740 4 client.15984 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:36:20.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.719+0000 7fb3d31e7740 1 -- 172.21.15.90:0/1934598207 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5615fab5c070 con 0x5615fa806d80 2024-04-25T11:36:20.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.719+0000 7fb3ca0d0640 1 --2- 172.21.15.90:0/1934598207 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5615fa806d80 0x5615fac04350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:20.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.719+0000 7fb3ca0d0640 1 --2- 172.21.15.90:0/1934598207 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5615fa806d80 0x5615fac04350 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=976f24df26016dfc server_cookie=66e313e370d2dc0b in_seq=0 out_seq=0 2024-04-25T11:36:20.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.719+0000 7fb3adffb640 10 client.15984 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:20.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.719+0000 7fb3adffb640 1 -- 172.21.15.90:0/1934598207 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5615fab5c070 con 0x5615fa806d80 2024-04-25T11:36:20.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.719+0000 7fb3adffb640 10 client.15984 handle_command_reply: tid=0 2024-04-25T11:36:20.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.719+0000 7fb3d31e7740 1 client.15984 shutdown 2024-04-25T11:36:20.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.720+0000 7fb3d31e7740 1 -- 172.21.15.90:0/1934598207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa8274d0 msgr2=0x5615fabf41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:20.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.720+0000 7fb3d31e7740 1 --2- 172.21.15.90:0/1934598207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa8274d0 0x5615fabf41d0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4005fd0 tx=0x7fb3b4004730 comp rx=0 tx=0).stop 2024-04-25T11:36:20.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.720+0000 7fb3d31e7740 1 -- 172.21.15.90:0/1934598207 shutdown_connections 2024-04-25T11:36:20.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.720+0000 7fb3d31e7740 1 --2- 172.21.15.90:0/1934598207 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5615fa806d80 0x5615fac04350 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.720+0000 7fb3d31e7740 1 --2- 172.21.15.90:0/1934598207 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5615fa8481b0 0x5615fabf20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.720+0000 7fb3d31e7740 1 --2- 172.21.15.90:0/1934598207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa8274d0 0x5615fabf41d0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.720+0000 7fb3adffb640 0 client.15984 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:20.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.720+0000 7fb3d31e7740 1 --2- 172.21.15.90:0/1934598207 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5615fa81f2f0 0x5615fabf3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.720+0000 7fb3d31e7740 1 -- 172.21.15.90:0/1934598207 >> 172.21.15.90:0/1934598207 conn(0x5615fa83c300 msgr2=0x5615fabe9810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:20.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.720+0000 7fb3d31e7740 1 -- 172.21.15.90:0/1934598207 shutdown_connections 2024-04-25T11:36:20.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.720+0000 7fb3d31e7740 1 -- 172.21.15.90:0/1934598207 wait complete. 2024-04-25T11:36:20.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.720+0000 7fb3d31e7740 20 client.15984 trim_cache size 0 max 16384 2024-04-25T11:36:20.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:20 smithi090 ceph-mon[29575]: scrub queued for path: / 2024-04-25T11:36:20.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:20 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:36:20.725 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:20 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:36:20.725 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:20 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:36:20.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.749+0000 7fb3d31e7740 1 Processor -- start 2024-04-25T11:36:20.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.749+0000 7fb3d31e7740 1 -- start start 2024-04-25T11:36:20.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.749+0000 7fb3d31e7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5615fa806d80 0x5615fabf2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:20.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.749+0000 7fb3d31e7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa81f2f0 0x5615fabeb850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:20.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.749+0000 7fb3d31e7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5615fa8274d0 0x5615fabebd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:20.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.749+0000 7fb3d31e7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c41adae0 con 0x5615fa81f2f0 2024-04-25T11:36:20.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.749+0000 7fb3d31e7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c41af620 con 0x5615fa806d80 2024-04-25T11:36:20.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.749+0000 7fb3d31e7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c41ae1b0 con 0x5615fa8274d0 2024-04-25T11:36:20.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.749+0000 7fb3c98cf640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa81f2f0 0x5615fabeb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:20.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.749+0000 7fb3c98cf640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa81f2f0 0x5615fabeb850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40376/0 (socket says 172.21.15.90:40376) 2024-04-25T11:36:20.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.749+0000 7fb3c98cf640 1 -- 172.21.15.90:0/2542811236 learned_addr learned my addr 172.21.15.90:0/2542811236 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:20.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.749+0000 7fb3ca0d0640 1 --2- 172.21.15.90:0/2542811236 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5615fa806d80 0x5615fabf2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:20.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.749+0000 7fb3ca8d1640 1 --2- 172.21.15.90:0/2542811236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5615fa8274d0 0x5615fabebd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:20.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.750+0000 7fb3c98cf640 1 -- 172.21.15.90:0/2542811236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5615fa8274d0 msgr2=0x5615fabebd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:20.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.750+0000 7fb3c98cf640 1 --2- 172.21.15.90:0/2542811236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5615fa8274d0 0x5615fabebd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.750+0000 7fb3c98cf640 1 -- 172.21.15.90:0/2542811236 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5615fa806d80 msgr2=0x5615fabf2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:20.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.750+0000 7fb3c98cf640 1 --2- 172.21.15.90:0/2542811236 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5615fa806d80 0x5615fabf2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.750+0000 7fb3c98cf640 1 -- 172.21.15.90:0/2542811236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615fabfec60 con 0x5615fa81f2f0 2024-04-25T11:36:20.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.750+0000 7fb3c98cf640 1 --2- 172.21.15.90:0/2542811236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa81f2f0 0x5615fabeb850 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4004690 tx=0x7fb3b4021040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=232ce289a2e67fa0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:20.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.750+0000 7fb3affff640 1 -- 172.21.15.90:0/2542811236 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b4032470 con 0x5615fa81f2f0 2024-04-25T11:36:20.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.750+0000 7fb3affff640 1 -- 172.21.15.90:0/2542811236 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3b4038070 con 0x5615fa81f2f0 2024-04-25T11:36:20.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.750+0000 7fb3affff640 1 -- 172.21.15.90:0/2542811236 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b40287c0 con 0x5615fa81f2f0 2024-04-25T11:36:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.751+0000 7fb3d31e7740 1 -- 172.21.15.90:0/2542811236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa81f2f0 msgr2=0x5615fabeb850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.751+0000 7fb3d31e7740 1 --2- 172.21.15.90:0/2542811236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa81f2f0 0x5615fabeb850 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4004690 tx=0x7fb3b4021040 comp rx=0 tx=0).stop 2024-04-25T11:36:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.751+0000 7fb3d31e7740 1 -- 172.21.15.90:0/2542811236 shutdown_connections 2024-04-25T11:36:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.751+0000 7fb3d31e7740 1 --2- 172.21.15.90:0/2542811236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5615fa8274d0 0x5615fabebd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.751+0000 7fb3d31e7740 1 --2- 172.21.15.90:0/2542811236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa81f2f0 0x5615fabeb850 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.751+0000 7fb3d31e7740 1 --2- 172.21.15.90:0/2542811236 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5615fa806d80 0x5615fabf2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.751+0000 7fb3d31e7740 1 -- 172.21.15.90:0/2542811236 >> 172.21.15.90:0/2542811236 conn(0x5615fa83c300 msgr2=0x5615fabf1930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.751+0000 7fb3d31e7740 1 -- 172.21.15.90:0/2542811236 shutdown_connections 2024-04-25T11:36:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.751+0000 7fb3d31e7740 1 -- 172.21.15.90:0/2542811236 wait complete. 2024-04-25T11:36:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.752+0000 7fb3d31e7740 1 -- start start 2024-04-25T11:36:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.752+0000 7fb3d31e7740 1 Processor -- start 2024-04-25T11:36:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.752+0000 7fb3d31e7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5615fa806d80 0x5615fabfa300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.752+0000 7fb3d31e7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5615fa81f2f0 0x5615fabf7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.752+0000 7fb3d31e7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa8274d0 0x5615fabf7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.752+0000 7fb3d31e7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c41aef50 con 0x5615fa8274d0 2024-04-25T11:36:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.752+0000 7fb3d31e7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c41ae880 con 0x5615fa81f2f0 2024-04-25T11:36:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.752+0000 7fb3d31e7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c41abac0 con 0x5615fa806d80 2024-04-25T11:36:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.753+0000 7fb3ca8d1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa8274d0 0x5615fabf7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.753+0000 7fb3d31e7740 10 client.0 _ll_register_callbacks cb 0x7fb3b4003c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.753+0000 7fb3ca8d1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa8274d0 0x5615fabf7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40386/0 (socket says 172.21.15.90:40386) 2024-04-25T11:36:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.753+0000 7fb3ca8d1640 1 -- 172.21.15.90:0/4278033764 learned_addr learned my addr 172.21.15.90:0/4278033764 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.753+0000 7fb3c98cf640 1 --2- 172.21.15.90:0/4278033764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5615fa81f2f0 0x5615fabf7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.753+0000 7fb3ca0d0640 1 --2- 172.21.15.90:0/4278033764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5615fa806d80 0x5615fabfa300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.753+0000 7fb3ca8d1640 1 -- 172.21.15.90:0/4278033764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5615fa806d80 msgr2=0x5615fabfa300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.753+0000 7fb3ca8d1640 1 --2- 172.21.15.90:0/4278033764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5615fa806d80 0x5615fabfa300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.753+0000 7fb3ca8d1640 1 -- 172.21.15.90:0/4278033764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5615fa81f2f0 msgr2=0x5615fabf7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.753+0000 7fb3ca8d1640 1 --2- 172.21.15.90:0/4278033764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5615fa81f2f0 0x5615fabf7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.753+0000 7fb3ca8d1640 1 -- 172.21.15.90:0/4278033764 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615fabf82f0 con 0x5615fa8274d0 2024-04-25T11:36:20.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.753+0000 7fb3ca8d1640 1 --2- 172.21.15.90:0/4278033764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa8274d0 0x5615fabf7bb0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc003fa0 tx=0x7fb3bc0044d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34eb8283cfddffbd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:20.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.753+0000 7fb3adffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:20.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.753+0000 7fb3adffb640 1 -- 172.21.15.90:0/4278033764 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc011990 con 0x5615fa8274d0 2024-04-25T11:36:20.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.753+0000 7fb3adffb640 1 -- 172.21.15.90:0/4278033764 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb3bc011b30 con 0x5615fa8274d0 2024-04-25T11:36:20.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.753+0000 7fb3d31e7740 1 -- 172.21.15.90:0/4278033764 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3b4038070 con 0x5615fa8274d0 2024-04-25T11:36:20.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.754+0000 7fb3adffb640 1 -- 172.21.15.90:0/4278033764 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb3bc027050 con 0x5615fa8274d0 2024-04-25T11:36:20.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.754+0000 7fb3d31e7740 10 client.15990 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:20.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.754+0000 7fb3d31e7740 1 -- 172.21.15.90:0/4278033764 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5615fac6c5b0 con 0x5615fa8274d0 2024-04-25T11:36:20.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.754+0000 7fb3adffb640 1 -- 172.21.15.90:0/4278033764 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3b4038070 con 0x5615fa8274d0 2024-04-25T11:36:20.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.754+0000 7fb3adffb640 1 -- 172.21.15.90:0/4278033764 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fb3bc01a400 con 0x5615fa8274d0 2024-04-25T11:36:20.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.754+0000 7fb3d31e7740 10 client.15990 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:20.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.754+0000 7fb3d31e7740 10 client.15990 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:36:20.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.754+0000 7fb3d31e7740 20 client.15990 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:20.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.754+0000 7fb3d31e7740 1 --2- 172.21.15.90:0/4278033764 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5615fa8481b0 0x5615fac6ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:20.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.754+0000 7fb3d31e7740 4 client.15990 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:36:20.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.754+0000 7fb3d31e7740 1 -- 172.21.15.90:0/4278033764 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5615fab55ee0 con 0x5615fa8481b0 2024-04-25T11:36:20.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.755+0000 7fb3ca0d0640 1 --2- 172.21.15.90:0/4278033764 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5615fa8481b0 0x5615fac6ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:20.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.755+0000 7fb3adffb640 10 client.15990 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:20.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.755+0000 7fb3ca0d0640 1 --2- 172.21.15.90:0/4278033764 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5615fa8481b0 0x5615fac6ed60 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b88b7560b5775d server_cookie=8f02587489f38d69 in_seq=0 out_seq=0 2024-04-25T11:36:20.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.759+0000 7fb3adffb640 1 -- 172.21.15.90:0/4278033764 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5615fab55ee0 con 0x5615fa8481b0 2024-04-25T11:36:20.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.759+0000 7fb3adffb640 10 client.15990 handle_command_reply: tid=0 2024-04-25T11:36:20.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.759+0000 7fb3d31e7740 1 client.15990 shutdown 2024-04-25T11:36:20.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.759+0000 7fb3d31e7740 1 -- 172.21.15.90:0/4278033764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa8274d0 msgr2=0x5615fabf7bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:20.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.759+0000 7fb3d31e7740 1 --2- 172.21.15.90:0/4278033764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa8274d0 0x5615fabf7bb0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc003fa0 tx=0x7fb3bc0044d0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.759+0000 7fb3d31e7740 1 -- 172.21.15.90:0/4278033764 shutdown_connections 2024-04-25T11:36:20.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.759+0000 7fb3d31e7740 1 --2- 172.21.15.90:0/4278033764 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5615fa8481b0 0x5615fac6ed60 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.760+0000 7fb3d31e7740 1 --2- 172.21.15.90:0/4278033764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615fa8274d0 0x5615fabf7bb0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.760+0000 7fb3adffb640 0 client.15990 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:20.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.760+0000 7fb3d31e7740 1 --2- 172.21.15.90:0/4278033764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5615fa81f2f0 0x5615fabf7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.760+0000 7fb3d31e7740 1 --2- 172.21.15.90:0/4278033764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5615fa806d80 0x5615fabfa300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.760+0000 7fb3d31e7740 1 -- 172.21.15.90:0/4278033764 >> 172.21.15.90:0/4278033764 conn(0x5615fa83c300 msgr2=0x5615fabf0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:20.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.760+0000 7fb3d31e7740 1 -- 172.21.15.90:0/4278033764 shutdown_connections 2024-04-25T11:36:20.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.760+0000 7fb3d31e7740 1 -- 172.21.15.90:0/4278033764 wait complete. 2024-04-25T11:36:20.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.760+0000 7fb3d31e7740 20 client.15990 trim_cache size 0 max 16384 2024-04-25T11:36:20.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:36:20.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:36:20.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:36:20.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:36:20.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.761+0000 7fb3cc334640 1 -- 172.21.15.90:0/1842454845 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb3b00afdf0 msgr2=0x7fb3b00b2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:20.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.761+0000 7fb3cc334640 1 --2- 172.21.15.90:0/1842454845 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb3b00afdf0 0x7fb3b00b2300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4101250 tx=0x7fb3b400b390 comp rx=0 tx=0).stop 2024-04-25T11:36:20.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.761+0000 7fb3cc334640 1 -- 172.21.15.90:0/1842454845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3c4102450 msgr2=0x7fb3c419a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:20.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.761+0000 7fb3cc334640 1 --2- 172.21.15.90:0/1842454845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3c4102450 0x7fb3c419a770 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0005940 tx=0x7fb3c00042f0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.761+0000 7fb3cc334640 1 -- 172.21.15.90:0/1842454845 shutdown_connections 2024-04-25T11:36:20.753 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.761+0000 7fb3cc334640 1 --2- 172.21.15.90:0/1842454845 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb3b00afdf0 0x7fb3b00b2300 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.753 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.761+0000 7fb3cc334640 1 --2- 172.21.15.90:0/1842454845 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb3c41074a0 0x7fb3c419ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.753 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.761+0000 7fb3cc334640 1 --2- 172.21.15.90:0/1842454845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3c41069e0 0x7fb3c419acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.753 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.761+0000 7fb3cc334640 1 --2- 172.21.15.90:0/1842454845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3c4102450 0x7fb3c419a770 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.753 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.761+0000 7fb3cc334640 1 -- 172.21.15.90:0/1842454845 >> 172.21.15.90:0/1842454845 conn(0x7fb3c40fd8b0 msgr2=0x7fb3c410ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:20.753 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.761+0000 7fb3cc334640 1 -- 172.21.15.90:0/1842454845 shutdown_connections 2024-04-25T11:36:20.753 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:20.761+0000 7fb3cc334640 1 -- 172.21.15.90:0/1842454845 wait complete. 2024-04-25T11:36:20.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:20 smithi138 ceph-mon[32328]: scrub queued for path: / 2024-04-25T11:36:20.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:20 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:36:20.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:20 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:36:20.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:20 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:36:20.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:20 smithi156 ceph-mon[33045]: scrub queued for path: / 2024-04-25T11:36:20.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:20 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:36:20.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:20 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:36:20.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:20 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:36:21.627 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:21 smithi090 ceph-mon[29575]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 10 op/s; 185 B/s, 0 keys/s, 1 objects/s recovering 2024-04-25T11:36:21.627 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:21 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:36:21.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:21 smithi138 ceph-mon[32328]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 10 op/s; 185 B/s, 0 keys/s, 1 objects/s recovering 2024-04-25T11:36:21.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:21 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:36:21.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:21 smithi156 ceph-mon[33045]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 10 op/s; 185 B/s, 0 keys/s, 1 objects/s recovering 2024-04-25T11:36:21.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:21 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:36:23.725 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:23 smithi090 ceph-mon[29575]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s; 162 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T11:36:23.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:23 smithi138 ceph-mon[32328]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s; 162 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T11:36:23.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:23 smithi156 ceph-mon[33045]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s; 162 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T11:36:25.725 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:25 smithi090 ceph-mon[29575]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 255 B/s wr, 13 op/s; 162 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T11:36:25.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:25 smithi138 ceph-mon[32328]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 255 B/s wr, 13 op/s; 162 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T11:36:25.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:25 smithi156 ceph-mon[33045]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 255 B/s wr, 13 op/s; 162 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T11:36:27.638 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:27 smithi156 ceph-mon[33045]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 10 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T11:36:27.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:27 smithi090 ceph-mon[29575]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 10 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T11:36:27.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:27 smithi138 ceph-mon[32328]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 10 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T11:36:29.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:29 smithi090 ceph-mon[29575]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T11:36:29.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:29 smithi138 ceph-mon[32328]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T11:36:29.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:29 smithi156 ceph-mon[33045]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T11:36:31.725 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:31 smithi090 ceph-mon[29575]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-04-25T11:36:31.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:31 smithi138 ceph-mon[32328]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-04-25T11:36:31.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:31 smithi156 ceph-mon[33045]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-04-25T11:36:33.697 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:33 smithi090 ceph-mon[29575]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T11:36:33.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:33 smithi138 ceph-mon[32328]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T11:36:33.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:33 smithi156 ceph-mon[33045]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T11:36:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.862+0000 7f2aee4f2640 1 -- 172.21.15.90:0/541732848 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ae8102810 msgr2=0x7f2ae8110690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.862+0000 7f2aee4f2640 1 --2- 172.21.15.90:0/541732848 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ae8102810 0x7f2ae8110690 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0009ca0 tx=0x7f2ae002a8c0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.862+0000 7f2aee4f2640 1 -- 172.21.15.90:0/541732848 shutdown_connections 2024-04-25T11:36:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.862+0000 7f2aee4f2640 1 --2- 172.21.15.90:0/541732848 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ae8102810 0x7f2ae8110690 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.862+0000 7f2aee4f2640 1 --2- 172.21.15.90:0/541732848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ae810a010 0x7f2ae81021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.862+0000 7f2aee4f2640 1 --2- 172.21.15.90:0/541732848 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2ae8109b70 0x7f2ae8101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.862+0000 7f2aee4f2640 1 -- 172.21.15.90:0/541732848 >> 172.21.15.90:0/541732848 conn(0x7f2ae80778b0 msgr2=0x7f2ae8077cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.863+0000 7f2aee4f2640 1 -- 172.21.15.90:0/541732848 shutdown_connections 2024-04-25T11:36:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.863+0000 7f2aee4f2640 1 -- 172.21.15.90:0/541732848 wait complete. 2024-04-25T11:36:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.863+0000 7f2aee4f2640 1 Processor -- start 2024-04-25T11:36:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.863+0000 7f2aee4f2640 1 -- start start 2024-04-25T11:36:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.864+0000 7f2aee4f2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2ae8102810 0x7f2ae81a25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.864+0000 7f2aee4f2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ae8109b70 0x7f2ae81a2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.864+0000 7f2aee4f2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ae810a010 0x7f2ae819c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.864+0000 7f2aee4f2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae8115fc0 con 0x7f2ae810a010 2024-04-25T11:36:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.864+0000 7f2aee4f2640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae8115e40 con 0x7f2ae8102810 2024-04-25T11:36:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.864+0000 7f2aee4f2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae8116140 con 0x7f2ae8109b70 2024-04-25T11:36:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.864+0000 7f2aeca8f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ae810a010 0x7f2ae819c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.864+0000 7f2aeca8f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ae810a010 0x7f2ae819c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57720/0 (socket says 172.21.15.90:57720) 2024-04-25T11:36:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.864+0000 7f2aeca8f640 1 -- 172.21.15.90:0/3389155379 learned_addr learned my addr 172.21.15.90:0/3389155379 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.864+0000 7f2ae77fe640 1 --2- 172.21.15.90:0/3389155379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ae8109b70 0x7f2ae81a2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.864+0000 7f2ae7fff640 1 --2- 172.21.15.90:0/3389155379 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2ae8102810 0x7f2ae81a25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.864+0000 7f2aeca8f640 1 -- 172.21.15.90:0/3389155379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ae8109b70 msgr2=0x7f2ae81a2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.864+0000 7f2aeca8f640 1 --2- 172.21.15.90:0/3389155379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ae8109b70 0x7f2ae81a2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.864+0000 7f2aeca8f640 1 -- 172.21.15.90:0/3389155379 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2ae8102810 msgr2=0x7f2ae81a25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.864+0000 7f2aeca8f640 1 --2- 172.21.15.90:0/3389155379 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2ae8102810 0x7f2ae81a25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.864+0000 7f2aeca8f640 1 -- 172.21.15.90:0/3389155379 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae819ce90 con 0x7f2ae810a010 2024-04-25T11:36:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.865+0000 7f2aeca8f640 1 --2- 172.21.15.90:0/3389155379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ae810a010 0x7f2ae819c760 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0009430 tx=0x7f2ae00317c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.865+0000 7f2ae57fa640 1 -- 172.21.15.90:0/3389155379 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae0003b70 con 0x7f2ae810a010 2024-04-25T11:36:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.865+0000 7f2aee4f2640 1 -- 172.21.15.90:0/3389155379 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ae819d0c0 con 0x7f2ae810a010 2024-04-25T11:36:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.865+0000 7f2aee4f2640 1 -- 172.21.15.90:0/3389155379 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ae81a9380 con 0x7f2ae810a010 2024-04-25T11:36:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.865+0000 7f2ae57fa640 1 -- 172.21.15.90:0/3389155379 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ae0003d10 con 0x7f2ae810a010 2024-04-25T11:36:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.865+0000 7f2ae57fa640 1 -- 172.21.15.90:0/3389155379 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae002f9e0 con 0x7f2ae810a010 2024-04-25T11:36:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.867+0000 7f2af53a7740 1 Processor -- start 2024-04-25T11:36:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.869+0000 7f2ae57fa640 1 -- 172.21.15.90:0/3389155379 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 24) ==== 147753+0+0 (secure 0 0 0) 0x7f2ae002fc60 con 0x7f2ae810a010 2024-04-25T11:36:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.869+0000 7f2af53a7740 1 -- start start 2024-04-25T11:36:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.869+0000 7f2af53a7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643e4ede1b0 0x5643e527daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.869+0000 7f2af53a7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643e4eb52f0 0x5643e527e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.869+0000 7f2af53a7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4ebd4d0 0x5643e527e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.869+0000 7f2af53a7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5643e51e4460 con 0x5643e4ebd4d0 2024-04-25T11:36:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.869+0000 7f2af53a7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5643e519e590 con 0x5643e4ede1b0 2024-04-25T11:36:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.869+0000 7f2af53a7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5643e51e0420 con 0x5643e4eb52f0 2024-04-25T11:36:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.869+0000 7f2ae57fa640 1 --2- 172.21.15.90:0/3389155379 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f2ad00afdf0 0x7f2ad00b2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.870+0000 7f2ae77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4ebd4d0 0x5643e527e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.870+0000 7f2ae77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4ebd4d0 0x5643e527e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57734/0 (socket says 172.21.15.90:57734) 2024-04-25T11:36:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.870+0000 7f2ae77fe640 1 -- 172.21.15.90:0/3999877807 learned_addr learned my addr 172.21.15.90:0/3999877807 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.870+0000 7f2ae7fff640 1 --2- 172.21.15.90:0/3999877807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643e4eb52f0 0x5643e527e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.870+0000 7f2aeca8f640 1 --2- 172.21.15.90:0/3389155379 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f2ad00afdf0 0x7f2ad00b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.870+0000 7f2ae77fe640 1 --2- 172.21.15.90:0/3999877807 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643e4ede1b0 0x5643e527daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.870+0000 7f2ae57fa640 1 -- 172.21.15.90:0/3389155379 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f2ae0100510 con 0x7f2ae810a010 2024-04-25T11:36:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.870+0000 7f2ae77fe640 1 -- 172.21.15.90:0/3999877807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643e4eb52f0 msgr2=0x5643e527e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.870+0000 7f2ae77fe640 1 --2- 172.21.15.90:0/3999877807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643e4eb52f0 0x5643e527e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.870+0000 7f2ae77fe640 1 -- 172.21.15.90:0/3999877807 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643e4ede1b0 msgr2=0x5643e527daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.870+0000 7f2ae77fe640 1 --2- 172.21.15.90:0/3999877807 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643e4ede1b0 0x5643e527daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.870+0000 7f2ae77fe640 1 -- 172.21.15.90:0/3999877807 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643e51ab0d0 con 0x5643e4ebd4d0 2024-04-25T11:36:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.870+0000 7f2aeca8f640 1 --2- 172.21.15.90:0/3389155379 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f2ad00afdf0 0x7f2ad00b2300 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2ae00fdbf0 tx=0x7f2ae00095c0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.870+0000 7f2ae77fe640 1 --2- 172.21.15.90:0/3999877807 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4ebd4d0 0x5643e527e570 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f2ae80754a0 tx=0x7f2adc00b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69d8d2a18c773c00 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.870+0000 7f2ace7fc640 1 -- 172.21.15.90:0/3999877807 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2adc005220 con 0x5643e4ebd4d0 2024-04-25T11:36:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.871+0000 7f2ace7fc640 1 -- 172.21.15.90:0/3999877807 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2adc004c50 con 0x5643e4ebd4d0 2024-04-25T11:36:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.871+0000 7f2ace7fc640 1 -- 172.21.15.90:0/3999877807 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2adc00f7e0 con 0x5643e4ebd4d0 2024-04-25T11:36:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.872+0000 7f2af53a7740 1 -- 172.21.15.90:0/3999877807 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4ebd4d0 msgr2=0x5643e527e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.872+0000 7f2af53a7740 1 --2- 172.21.15.90:0/3999877807 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4ebd4d0 0x5643e527e570 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f2ae80754a0 tx=0x7f2adc00b790 comp rx=0 tx=0).stop 2024-04-25T11:36:33.865 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.872+0000 7f2af53a7740 1 -- 172.21.15.90:0/3999877807 shutdown_connections 2024-04-25T11:36:33.865 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.872+0000 7f2af53a7740 1 --2- 172.21.15.90:0/3999877807 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4ebd4d0 0x5643e527e570 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.865 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.872+0000 7f2af53a7740 1 --2- 172.21.15.90:0/3999877807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643e4eb52f0 0x5643e527e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.865 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.872+0000 7f2af53a7740 1 --2- 172.21.15.90:0/3999877807 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643e4ede1b0 0x5643e527daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.865 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.872+0000 7f2af53a7740 1 -- 172.21.15.90:0/3999877807 >> 172.21.15.90:0/3999877807 conn(0x5643e4ed2300 msgr2=0x5643e527a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:33.865 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.872+0000 7f2af53a7740 1 -- 172.21.15.90:0/3999877807 shutdown_connections 2024-04-25T11:36:33.865 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.872+0000 7f2af53a7740 1 -- 172.21.15.90:0/3999877807 wait complete. 2024-04-25T11:36:33.865 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.872+0000 7f2af53a7740 1 -- start start 2024-04-25T11:36:33.865 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.872+0000 7f2af53a7740 1 Processor -- start 2024-04-25T11:36:33.865 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.873+0000 7f2af53a7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643e4eb52f0 0x5643e52881e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:33.865 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.873+0000 7f2af53a7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643e4ebd4d0 0x5643e5288720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:33.865 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.873+0000 7f2af53a7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4ede1b0 0x5643e528e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:33.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.873+0000 7f2af53a7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5643e528ac20 con 0x5643e4ede1b0 2024-04-25T11:36:33.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.873+0000 7f2af53a7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5643e516bb60 con 0x5643e4eb52f0 2024-04-25T11:36:33.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.873+0000 7f2af53a7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5643e528ada0 con 0x5643e4ebd4d0 2024-04-25T11:36:33.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.873+0000 7f2af53a7740 10 client.0 _ll_register_callbacks cb 0x5643e51902b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:33.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.873+0000 7f2ae77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4ede1b0 0x5643e528e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:33.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.873+0000 7f2ae77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4ede1b0 0x5643e528e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57750/0 (socket says 172.21.15.90:57750) 2024-04-25T11:36:33.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.873+0000 7f2ae77fe640 1 -- 172.21.15.90:0/68148399 learned_addr learned my addr 172.21.15.90:0/68148399 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:33.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.873+0000 7f2ae7fff640 1 --2- 172.21.15.90:0/68148399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643e4ebd4d0 0x5643e5288720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:33.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.873+0000 7f2ae77fe640 1 --2- 172.21.15.90:0/68148399 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643e4eb52f0 0x5643e52881e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:33.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.873+0000 7f2ae77fe640 1 -- 172.21.15.90:0/68148399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643e4ebd4d0 msgr2=0x5643e5288720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:33.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.873+0000 7f2ae77fe640 1 --2- 172.21.15.90:0/68148399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643e4ebd4d0 0x5643e5288720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.873+0000 7f2ae77fe640 1 -- 172.21.15.90:0/68148399 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643e4eb52f0 msgr2=0x5643e52881e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:33.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.873+0000 7f2ae77fe640 1 --2- 172.21.15.90:0/68148399 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643e4eb52f0 0x5643e52881e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.873+0000 7f2ae77fe640 1 -- 172.21.15.90:0/68148399 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643e4ef6210 con 0x5643e4ede1b0 2024-04-25T11:36:33.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.874+0000 7f2ae77fe640 1 --2- 172.21.15.90:0/68148399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4ede1b0 0x5643e528e1a0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f2adc00b3f0 tx=0x7f2adc005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bc95145ecc9a492 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:33.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.874+0000 7f2ac77fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:33.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.874+0000 7f2ac77fe640 1 -- 172.21.15.90:0/68148399 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2adc005000 con 0x5643e4ede1b0 2024-04-25T11:36:33.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.874+0000 7f2ac77fe640 1 -- 172.21.15.90:0/68148399 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f2adc0051a0 con 0x5643e4ede1b0 2024-04-25T11:36:33.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.874+0000 7f2af53a7740 1 -- 172.21.15.90:0/68148399 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2adc004c50 con 0x5643e4ede1b0 2024-04-25T11:36:33.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.874+0000 7f2ac77fe640 1 -- 172.21.15.90:0/68148399 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2adc0053a0 con 0x5643e4ede1b0 2024-04-25T11:36:33.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.874+0000 7f2af53a7740 10 client.16002 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:33.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.874+0000 7f2af53a7740 1 -- 172.21.15.90:0/68148399 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5643e4fc2620 con 0x5643e4ede1b0 2024-04-25T11:36:33.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.874+0000 7f2ac77fe640 1 -- 172.21.15.90:0/68148399 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2adc004c50 con 0x5643e4ede1b0 2024-04-25T11:36:33.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.874+0000 7f2ac77fe640 1 -- 172.21.15.90:0/68148399 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f2adc01c020 con 0x5643e4ede1b0 2024-04-25T11:36:33.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.875+0000 7f2af53a7740 10 client.16002 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:33.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.875+0000 7f2af53a7740 10 client.16002 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:36:33.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.875+0000 7f2af53a7740 20 client.16002 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:33.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.875+0000 7f2af53a7740 1 --2- 172.21.15.90:0/68148399 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5643e4e9cd80 0x5643e529a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:33.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.875+0000 7f2af53a7740 4 client.16002 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:36:33.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.875+0000 7f2af53a7740 1 -- 172.21.15.90:0/68148399 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5643e51f2070 con 0x5643e4e9cd80 2024-04-25T11:36:33.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.875+0000 7f2ae7fff640 1 --2- 172.21.15.90:0/68148399 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5643e4e9cd80 0x5643e529a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:33.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.876+0000 7f2ae7fff640 1 --2- 172.21.15.90:0/68148399 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5643e4e9cd80 0x5643e529a6c0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62871df59f64f6c6 server_cookie=f644968348dc9b42 in_seq=0 out_seq=0 2024-04-25T11:36:33.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.876+0000 7f2ac77fe640 10 client.16002 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:33.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.876+0000 7f2ac77fe640 1 -- 172.21.15.90:0/68148399 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5643e51f2070 con 0x5643e4e9cd80 2024-04-25T11:36:33.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.876+0000 7f2ac77fe640 10 client.16002 handle_command_reply: tid=0 2024-04-25T11:36:33.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.876+0000 7f2af53a7740 1 client.16002 shutdown 2024-04-25T11:36:33.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.877+0000 7f2af53a7740 1 -- 172.21.15.90:0/68148399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4ede1b0 msgr2=0x5643e528e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:33.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.877+0000 7f2af53a7740 1 --2- 172.21.15.90:0/68148399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4ede1b0 0x5643e528e1a0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f2adc00b3f0 tx=0x7f2adc005f30 comp rx=0 tx=0).stop 2024-04-25T11:36:33.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.877+0000 7f2af53a7740 1 -- 172.21.15.90:0/68148399 shutdown_connections 2024-04-25T11:36:33.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.877+0000 7f2af53a7740 1 --2- 172.21.15.90:0/68148399 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5643e4e9cd80 0x5643e529a6c0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.869 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.877+0000 7f2af53a7740 1 --2- 172.21.15.90:0/68148399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4ede1b0 0x5643e528e1a0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.869 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.877+0000 7f2af53a7740 1 --2- 172.21.15.90:0/68148399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643e4ebd4d0 0x5643e5288720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.869 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.877+0000 7f2af53a7740 1 --2- 172.21.15.90:0/68148399 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643e4eb52f0 0x5643e52881e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.869 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.877+0000 7f2af53a7740 1 -- 172.21.15.90:0/68148399 >> 172.21.15.90:0/68148399 conn(0x5643e4ed2300 msgr2=0x5643e527f810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:33.869 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.877+0000 7f2ac77fe640 0 client.16002 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:33.869 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.877+0000 7f2af53a7740 1 -- 172.21.15.90:0/68148399 shutdown_connections 2024-04-25T11:36:33.869 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.877+0000 7f2af53a7740 1 -- 172.21.15.90:0/68148399 wait complete. 2024-04-25T11:36:33.869 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.877+0000 7f2af53a7740 20 client.16002 trim_cache size 0 max 16384 2024-04-25T11:36:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.901+0000 7f2af53a7740 1 Processor -- start 2024-04-25T11:36:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.901+0000 7f2af53a7740 1 -- start start 2024-04-25T11:36:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.901+0000 7f2af53a7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4eb52f0 0x5643e528e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.901+0000 7f2af53a7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643e4ebd4d0 0x5643e529fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.901+0000 7f2af53a7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643e4ede1b0 0x5643e52a0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.901+0000 7f2af53a7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad400e900 con 0x5643e4eb52f0 2024-04-25T11:36:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.901+0000 7f2af53a7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad4010440 con 0x5643e4ede1b0 2024-04-25T11:36:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.901+0000 7f2af53a7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad400efd0 con 0x5643e4ebd4d0 2024-04-25T11:36:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.901+0000 7f2ae77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4eb52f0 0x5643e528e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.902+0000 7f2ae77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4eb52f0 0x5643e528e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57758/0 (socket says 172.21.15.90:57758) 2024-04-25T11:36:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.902+0000 7f2ae77fe640 1 -- 172.21.15.90:0/3017206312 learned_addr learned my addr 172.21.15.90:0/3017206312 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.902+0000 7f2ae77fe640 1 --2- 172.21.15.90:0/3017206312 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643e4ede1b0 0x5643e52a0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.902+0000 7f2ae7fff640 1 --2- 172.21.15.90:0/3017206312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643e4ebd4d0 0x5643e529fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.902+0000 7f2ae77fe640 1 -- 172.21.15.90:0/3017206312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643e4ebd4d0 msgr2=0x5643e529fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.902+0000 7f2ae77fe640 1 --2- 172.21.15.90:0/3017206312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643e4ebd4d0 0x5643e529fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.902+0000 7f2ae77fe640 1 -- 172.21.15.90:0/3017206312 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643e4ede1b0 msgr2=0x5643e52a0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.902+0000 7f2ae77fe640 1 --2- 172.21.15.90:0/3017206312 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643e4ede1b0 0x5643e52a0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.902+0000 7f2ae77fe640 1 -- 172.21.15.90:0/3017206312 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643e5283b70 con 0x5643e4eb52f0 2024-04-25T11:36:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.902+0000 7f2ae7fff640 1 --2- 172.21.15.90:0/3017206312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643e4ebd4d0 0x5643e529fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:36:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.902+0000 7f2ae77fe640 1 --2- 172.21.15.90:0/3017206312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4eb52f0 0x5643e528e0a0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f2adc004dd0 tx=0x7f2adc00b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=290b74cfba169fe1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.902+0000 7f2acdffb640 1 -- 172.21.15.90:0/3017206312 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2adc03c070 con 0x5643e4eb52f0 2024-04-25T11:36:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.902+0000 7f2acdffb640 1 -- 172.21.15.90:0/3017206312 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2adc01ee20 con 0x5643e4eb52f0 2024-04-25T11:36:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.903+0000 7f2acdffb640 1 -- 172.21.15.90:0/3017206312 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2adc004940 con 0x5643e4eb52f0 2024-04-25T11:36:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.904+0000 7f2af53a7740 1 -- 172.21.15.90:0/3017206312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4eb52f0 msgr2=0x5643e528e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.904+0000 7f2af53a7740 1 --2- 172.21.15.90:0/3017206312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4eb52f0 0x5643e528e0a0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f2adc004dd0 tx=0x7f2adc00b5b0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.904+0000 7f2af53a7740 1 -- 172.21.15.90:0/3017206312 shutdown_connections 2024-04-25T11:36:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.904+0000 7f2af53a7740 1 --2- 172.21.15.90:0/3017206312 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643e4ede1b0 0x5643e52a0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.904+0000 7f2af53a7740 1 --2- 172.21.15.90:0/3017206312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643e4ebd4d0 0x5643e529fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.904+0000 7f2af53a7740 1 --2- 172.21.15.90:0/3017206312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4eb52f0 0x5643e528e0a0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.904+0000 7f2af53a7740 1 -- 172.21.15.90:0/3017206312 >> 172.21.15.90:0/3017206312 conn(0x5643e4ed2300 msgr2=0x5643e52895a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.904+0000 7f2af53a7740 1 -- 172.21.15.90:0/3017206312 shutdown_connections 2024-04-25T11:36:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.904+0000 7f2af53a7740 1 -- 172.21.15.90:0/3017206312 wait complete. 2024-04-25T11:36:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.904+0000 7f2af53a7740 1 -- start start 2024-04-25T11:36:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.905+0000 7f2af53a7740 1 Processor -- start 2024-04-25T11:36:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.905+0000 7f2af53a7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643e4ebd4d0 0x5643e5294d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.905+0000 7f2af53a7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4ede1b0 0x5643e52900b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.905+0000 7f2af53a7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643e4e721c0 0x5643e52905f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.905+0000 7f2ae7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4ede1b0 0x5643e52900b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.905+0000 7f2ae7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4ede1b0 0x5643e52900b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57768/0 (socket says 172.21.15.90:57768) 2024-04-25T11:36:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.905+0000 7f2ae7fff640 1 -- 172.21.15.90:0/4142868416 learned_addr learned my addr 172.21.15.90:0/4142868416 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.905+0000 7f2af53a7740 1 -- 172.21.15.90:0/4142868416 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad400fd70 con 0x5643e4ede1b0 2024-04-25T11:36:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.905+0000 7f2af53a7740 1 -- 172.21.15.90:0/4142868416 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad400f6a0 con 0x5643e4e721c0 2024-04-25T11:36:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.905+0000 7f2af53a7740 1 -- 172.21.15.90:0/4142868416 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad400c8e0 con 0x5643e4ebd4d0 2024-04-25T11:36:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.905+0000 7f2ae77fe640 1 --2- 172.21.15.90:0/4142868416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643e4ebd4d0 0x5643e5294d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.906+0000 7f2ae77fe640 1 --2- 172.21.15.90:0/4142868416 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643e4e721c0 0x5643e52905f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.906+0000 7f2ae77fe640 1 -- 172.21.15.90:0/4142868416 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643e4e721c0 msgr2=0x5643e52905f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.906+0000 7f2ae77fe640 1 --2- 172.21.15.90:0/4142868416 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643e4e721c0 0x5643e52905f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.906+0000 7f2ae77fe640 1 -- 172.21.15.90:0/4142868416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4ede1b0 msgr2=0x5643e52900b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.906+0000 7f2ae77fe640 1 --2- 172.21.15.90:0/4142868416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4ede1b0 0x5643e52900b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.906+0000 7f2ae77fe640 1 -- 172.21.15.90:0/4142868416 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643e5290b30 con 0x5643e4ebd4d0 2024-04-25T11:36:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.906+0000 7f2ae7fff640 1 --2- 172.21.15.90:0/4142868416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4ede1b0 0x5643e52900b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:36:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.906+0000 7f2af53a7740 10 client.0 _ll_register_callbacks cb 0x7f2ad80097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.906+0000 7f2af53a7740 1 -- 172.21.15.90:0/4142868416 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2adc01ee20 con 0x5643e4ebd4d0 2024-04-25T11:36:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.906+0000 7f2ae77fe640 1 --2- 172.21.15.90:0/4142868416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643e4ebd4d0 0x5643e5294d60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2adc014000 tx=0x7f2adc0056b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=213d0774567528ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.906+0000 7f2ac77fe640 10 client.25501 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:36:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.906+0000 7f2ac77fe640 1 -- 172.21.15.90:0/4142868416 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2adc03c070 con 0x5643e4ebd4d0 2024-04-25T11:36:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.906+0000 7f2ac77fe640 1 -- 172.21.15.90:0/4142868416 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f2adc004b40 con 0x5643e4ebd4d0 2024-04-25T11:36:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.907+0000 7f2ac77fe640 1 -- 172.21.15.90:0/4142868416 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2adc01e970 con 0x5643e4ebd4d0 2024-04-25T11:36:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.907+0000 7f2af53a7740 10 client.25501 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.907+0000 7f2af53a7740 1 -- 172.21.15.90:0/4142868416 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({fsmap=26}) -- 0x5643e53022e0 con 0x5643e4ebd4d0 2024-04-25T11:36:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.907+0000 7f2ac77fe640 1 -- 172.21.15.90:0/4142868416 <== mon.1 v2:172.21.15.156:3300/0 4 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f2adc028040 con 0x5643e4ebd4d0 2024-04-25T11:36:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.907+0000 7f2af53a7740 10 client.25501 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.907+0000 7f2af53a7740 10 client.25501 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:36:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.907+0000 7f2af53a7740 20 client.25501 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.907+0000 7f2af53a7740 1 --2- 172.21.15.90:0/4142868416 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5643e53028f0 0x5643e5304dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.907+0000 7f2af53a7740 4 client.25501 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:36:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.907+0000 7f2af53a7740 1 -- 172.21.15.90:0/4142868416 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5643e51ebee0 con 0x5643e53028f0 2024-04-25T11:36:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.908+0000 7f2ae7fff640 1 --2- 172.21.15.90:0/4142868416 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5643e53028f0 0x5643e5304dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.908+0000 7f2ae7fff640 1 --2- 172.21.15.90:0/4142868416 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5643e53028f0 0x5643e5304dc0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f75ce41809ceb4 server_cookie=cb4b799f631127a0 in_seq=0 out_seq=0 2024-04-25T11:36:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.908+0000 7f2ac77fe640 10 client.25501 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.912+0000 7f2ac77fe640 1 -- 172.21.15.90:0/4142868416 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5643e51ebee0 con 0x5643e53028f0 2024-04-25T11:36:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.912+0000 7f2ac77fe640 10 client.25501 handle_command_reply: tid=0 2024-04-25T11:36:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.912+0000 7f2af53a7740 1 client.25501 shutdown 2024-04-25T11:36:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.912+0000 7f2af53a7740 1 -- 172.21.15.90:0/4142868416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643e4ebd4d0 msgr2=0x5643e5294d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.913+0000 7f2af53a7740 1 --2- 172.21.15.90:0/4142868416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643e4ebd4d0 0x5643e5294d60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2adc014000 tx=0x7f2adc0056b0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.913+0000 7f2af53a7740 1 -- 172.21.15.90:0/4142868416 shutdown_connections 2024-04-25T11:36:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.913+0000 7f2af53a7740 1 --2- 172.21.15.90:0/4142868416 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5643e53028f0 0x5643e5304dc0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.913+0000 7f2af53a7740 1 --2- 172.21.15.90:0/4142868416 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5643e4e721c0 0x5643e52905f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.913+0000 7f2ac77fe640 0 client.25501 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.913+0000 7f2af53a7740 1 --2- 172.21.15.90:0/4142868416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5643e4ede1b0 0x5643e52900b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.913+0000 7f2af53a7740 1 --2- 172.21.15.90:0/4142868416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5643e4ebd4d0 0x5643e5294d60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.913+0000 7f2af53a7740 1 -- 172.21.15.90:0/4142868416 >> 172.21.15.90:0/4142868416 conn(0x5643e4ed2300 msgr2=0x5643e5289f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.913+0000 7f2af53a7740 1 -- 172.21.15.90:0/4142868416 shutdown_connections 2024-04-25T11:36:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.913+0000 7f2af53a7740 1 -- 172.21.15.90:0/4142868416 wait complete. 2024-04-25T11:36:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.913+0000 7f2af53a7740 20 client.25501 trim_cache size 0 max 16384 2024-04-25T11:36:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:36:33.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:36:33.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:36:33.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:36:33.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.914+0000 7f2aee4f2640 1 -- 172.21.15.90:0/3389155379 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f2ad00afdf0 msgr2=0x7f2ad00b2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:33.904 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.914+0000 7f2aee4f2640 1 --2- 172.21.15.90:0/3389155379 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f2ad00afdf0 0x7f2ad00b2300 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2ae00fdbf0 tx=0x7f2ae00095c0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.904 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.914+0000 7f2aee4f2640 1 -- 172.21.15.90:0/3389155379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ae810a010 msgr2=0x7f2ae819c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:33.904 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.914+0000 7f2aee4f2640 1 --2- 172.21.15.90:0/3389155379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ae810a010 0x7f2ae819c760 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0009430 tx=0x7f2ae00317c0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.904 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.914+0000 7f2aee4f2640 1 -- 172.21.15.90:0/3389155379 shutdown_connections 2024-04-25T11:36:33.904 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.914+0000 7f2aee4f2640 1 --2- 172.21.15.90:0/3389155379 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f2ad00afdf0 0x7f2ad00b2300 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.904 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.914+0000 7f2aee4f2640 1 --2- 172.21.15.90:0/3389155379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ae810a010 0x7f2ae819c760 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.904 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.914+0000 7f2aee4f2640 1 --2- 172.21.15.90:0/3389155379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ae8109b70 0x7f2ae81a2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.904 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.914+0000 7f2aee4f2640 1 --2- 172.21.15.90:0/3389155379 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2ae8102810 0x7f2ae81a25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:33.904 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.914+0000 7f2aee4f2640 1 -- 172.21.15.90:0/3389155379 >> 172.21.15.90:0/3389155379 conn(0x7f2ae80778b0 msgr2=0x7f2ae8100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:33.904 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.914+0000 7f2aee4f2640 1 -- 172.21.15.90:0/3389155379 shutdown_connections 2024-04-25T11:36:33.904 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:33.914+0000 7f2aee4f2640 1 -- 172.21.15.90:0/3389155379 wait complete. 2024-04-25T11:36:35.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:35 smithi090 ceph-mon[29575]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 511 B/s wr, 12 op/s 2024-04-25T11:36:35.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:35 smithi138 ceph-mon[32328]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 511 B/s wr, 12 op/s 2024-04-25T11:36:35.838 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:35 smithi156 ceph-mon[33045]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 511 B/s wr, 12 op/s 2024-04-25T11:36:37.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:37 smithi090 ceph-mon[29575]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T11:36:37.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:37 smithi138 ceph-mon[32328]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T11:36:37.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:37 smithi156 ceph-mon[33045]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T11:36:39.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:39 smithi090 ceph-mon[29575]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T11:36:39.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:39 smithi138 ceph-mon[32328]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T11:36:39.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:39 smithi156 ceph-mon[33045]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T11:36:41.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:41 smithi090 ceph-mon[29575]: pgmap v51: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.1 KiB/s wr, 14 op/s 2024-04-25T11:36:41.725 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:41 smithi090 ceph-mon[29575]: mgrmap e25: x(active, since 92s), standbys: z, y 2024-04-25T11:36:41.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:41 smithi138 ceph-mon[32328]: pgmap v51: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.1 KiB/s wr, 14 op/s 2024-04-25T11:36:41.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:41 smithi138 ceph-mon[32328]: mgrmap e25: x(active, since 92s), standbys: z, y 2024-04-25T11:36:41.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:41 smithi156 ceph-mon[33045]: pgmap v51: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.1 KiB/s wr, 14 op/s 2024-04-25T11:36:41.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:41 smithi156 ceph-mon[33045]: mgrmap e25: x(active, since 92s), standbys: z, y 2024-04-25T11:36:43.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:43 smithi090 ceph-mon[29575]: pgmap v52: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-25T11:36:43.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:43 smithi138 ceph-mon[32328]: pgmap v52: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-25T11:36:43.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:43 smithi156 ceph-mon[33045]: pgmap v52: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-25T11:36:45.458 INFO:tasks.workunit.client.0.smithi090.stderr:Note: switching to '994e2f20102a20fa9a27966c14b625cc01884df8'. 2024-04-25T11:36:45.458 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-04-25T11:36:45.458 INFO:tasks.workunit.client.0.smithi090.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-25T11:36:45.458 INFO:tasks.workunit.client.0.smithi090.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-25T11:36:45.459 INFO:tasks.workunit.client.0.smithi090.stderr:state without impacting any branches by switching back to a branch. 2024-04-25T11:36:45.459 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-04-25T11:36:45.459 INFO:tasks.workunit.client.0.smithi090.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-25T11:36:45.459 INFO:tasks.workunit.client.0.smithi090.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-25T11:36:45.459 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-04-25T11:36:45.459 INFO:tasks.workunit.client.0.smithi090.stderr: git switch -c 2024-04-25T11:36:45.459 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-04-25T11:36:45.459 INFO:tasks.workunit.client.0.smithi090.stderr:Or undo this operation with: 2024-04-25T11:36:45.459 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-04-25T11:36:45.459 INFO:tasks.workunit.client.0.smithi090.stderr: git switch - 2024-04-25T11:36:45.459 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-04-25T11:36:45.459 INFO:tasks.workunit.client.0.smithi090.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-25T11:36:45.459 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-04-25T11:36:45.459 INFO:tasks.workunit.client.0.smithi090.stderr:HEAD is now at 994e2f20102 Merge PR #56810 into wip-pdonnell-testing-20240425.015853-debug 2024-04-25T11:36:45.465 DEBUG:teuthology.orchestra.run.smithi090:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-25T11:36:45.635 INFO:tasks.workunit.client.0.smithi090.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-25T11:36:45.637 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T11:36:45.637 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-25T11:36:45.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:45 smithi090 ceph-mon[29575]: pgmap v53: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.1 KiB/s wr, 14 op/s 2024-04-25T11:36:45.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:45 smithi138 ceph-mon[32328]: pgmap v53: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.1 KiB/s wr, 14 op/s 2024-04-25T11:36:45.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:45 smithi156 ceph-mon[33045]: pgmap v53: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.1 KiB/s wr, 14 op/s 2024-04-25T11:36:46.384 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-25T11:36:46.461 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-25T11:36:46.502 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T11:36:46.503 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T11:36:46.503 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-25T11:36:46.542 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T11:36:46.545 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T11:36:46.545 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-25T11:36:46.560 INFO:tasks.workunit:Running workunits matching suites/iogen.sh on client.0... 2024-04-25T11:36:46.562 INFO:tasks.workunit:Running workunit suites/iogen.sh... 2024-04-25T11:36:46.562 DEBUG:teuthology.orchestra.run.smithi090:workunit test suites/iogen.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=994e2f20102a20fa9a27966c14b625cc01884df8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iogen.sh 2024-04-25T11:36:46.622 INFO:tasks.workunit.client.0.smithi090.stdout:getting iogen 2024-04-25T11:36:46.623 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'getting iogen' 2024-04-25T11:36:46.623 INFO:tasks.workunit.client.0.smithi090.stderr:+ wget http://download.ceph.com/qa/iogen_3.1p0.tar 2024-04-25T11:36:46.644 INFO:tasks.workunit.client.0.smithi090.stderr:--2024-04-25 11:36:46-- http://download.ceph.com/qa/iogen_3.1p0.tar 2024-04-25T11:36:46.645 INFO:tasks.workunit.client.0.smithi090.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-04-25T11:36:46.666 INFO:tasks.workunit.client.0.smithi090.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-04-25T11:36:46.687 INFO:tasks.workunit.client.0.smithi090.stderr:HTTP request sent, awaiting response... 200 OK 2024-04-25T11:36:46.687 INFO:tasks.workunit.client.0.smithi090.stderr:Length: 8071 (7.9K) [application/octet-stream] 2024-04-25T11:36:46.689 INFO:tasks.workunit.client.0.smithi090.stderr:Saving to: ‘iogen_3.1p0.tar’ 2024-04-25T11:36:46.692 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-04-25T11:36:46.692 INFO:tasks.workunit.client.0.smithi090.stderr: 0K ....... 100% 1.17G=0s 2024-04-25T11:36:46.692 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-04-25T11:36:46.692 INFO:tasks.workunit.client.0.smithi090.stderr:2024-04-25 11:36:46 (1.17 GB/s) - ‘iogen_3.1p0.tar’ saved [8071/8071] 2024-04-25T11:36:46.692 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-04-25T11:36:46.692 INFO:tasks.workunit.client.0.smithi090.stderr:+ tar -xvzf iogen_3.1p0.tar 2024-04-25T11:36:46.705 INFO:tasks.workunit.client.0.smithi090.stdout:iogen_3.1p0 2024-04-25T11:36:46.706 INFO:tasks.workunit.client.0.smithi090.stdout:iogen_3.1p0/src 2024-04-25T11:36:46.708 INFO:tasks.workunit.client.0.smithi090.stdout:iogen_3.1p0/src/iogen.8 2024-04-25T11:36:46.709 INFO:tasks.workunit.client.0.smithi090.stdout:iogen_3.1p0/src/iogen.c 2024-04-25T11:36:46.711 INFO:tasks.workunit.client.0.smithi090.stdout:iogen_3.1p0/README 2024-04-25T11:36:46.712 INFO:tasks.workunit.client.0.smithi090.stdout:iogen_3.1p0/Makefile.bsd 2024-04-25T11:36:46.714 INFO:tasks.workunit.client.0.smithi090.stdout:iogen_3.1p0/Makefile 2024-04-25T11:36:46.715 INFO:tasks.workunit.client.0.smithi090.stderr:+ cd iogen_3.1p0 2024-04-25T11:36:46.716 INFO:tasks.workunit.client.0.smithi090.stdout:making iogen 2024-04-25T11:36:46.716 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'making iogen' 2024-04-25T11:36:46.716 INFO:tasks.workunit.client.0.smithi090.stderr:+ make 2024-04-25T11:36:46.724 INFO:tasks.workunit.client.0.smithi090.stdout:gcc -D_FILE_OFFSET_BITS=64 -DVERSION=\"3.1p0\" -Dstrlcpy=strncpy -Darc4random=rand -O2 src/iogen.c -o iogen 2024-04-25T11:36:47.001 INFO:tasks.workunit.client.0.smithi090.stdout:nroff -mandoc -Tascii src/iogen.8 > iogen.cat8 2024-04-25T11:36:47.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.017+0000 7fb83fbb7640 1 -- 172.21.15.90:0/4292553905 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb838108f00 msgr2=0x7fb838102b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:47.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.017+0000 7fb83fbb7640 1 --2- 172.21.15.90:0/4292553905 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb838108f00 0x7fb838102b50 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fb834009900 tx=0x7fb83402a410 comp rx=0 tx=0).stop 2024-04-25T11:36:47.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.017+0000 7fb83fbb7640 1 -- 172.21.15.90:0/4292553905 shutdown_connections 2024-04-25T11:36:47.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.017+0000 7fb83fbb7640 1 --2- 172.21.15.90:0/4292553905 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb8381031c0 0x7fb83810f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.017+0000 7fb83fbb7640 1 --2- 172.21.15.90:0/4292553905 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb838108f00 0x7fb838102b50 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.017+0000 7fb83fbb7640 1 --2- 172.21.15.90:0/4292553905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb838108a60 0x7fb838102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.017+0000 7fb83fbb7640 1 -- 172.21.15.90:0/4292553905 >> 172.21.15.90:0/4292553905 conn(0x7fb8380781b0 msgr2=0x7fb8380785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:47.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.017+0000 7fb83fbb7640 1 -- 172.21.15.90:0/4292553905 shutdown_connections 2024-04-25T11:36:47.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.017+0000 7fb83fbb7640 1 -- 172.21.15.90:0/4292553905 wait complete. 2024-04-25T11:36:47.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.018+0000 7fb83fbb7640 1 Processor -- start 2024-04-25T11:36:47.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.018+0000 7fb83fbb7640 1 -- start start 2024-04-25T11:36:47.007 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.018+0000 7fb83fbb7640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb8381031c0 0x7fb838199700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.008 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.018+0000 7fb83fbb7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb838108a60 0x7fb838199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.008 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.018+0000 7fb83fbb7640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb838108f00 0x7fb83819df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.008 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.018+0000 7fb83fbb7640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb838112c80 con 0x7fb838108f00 2024-04-25T11:36:47.008 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.018+0000 7fb83fbb7640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb838112b00 con 0x7fb8381031c0 2024-04-25T11:36:47.008 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.018+0000 7fb83fbb7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb838112e00 con 0x7fb838108a60 2024-04-25T11:36:47.008 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.018+0000 7fb83e154640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb838108f00 0x7fb83819df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.008 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.018+0000 7fb83e154640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb838108f00 0x7fb83819df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52298/0 (socket says 172.21.15.90:52298) 2024-04-25T11:36:47.008 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.018+0000 7fb83e154640 1 -- 172.21.15.90:0/2347185064 learned_addr learned my addr 172.21.15.90:0/2347185064 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:47.008 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.018+0000 7fb83d953640 1 --2- 172.21.15.90:0/2347185064 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb8381031c0 0x7fb838199700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.008 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.018+0000 7fb83d152640 1 --2- 172.21.15.90:0/2347185064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb838108a60 0x7fb838199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.008 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.018+0000 7fb83e154640 1 -- 172.21.15.90:0/2347185064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb838108a60 msgr2=0x7fb838199c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:47.008 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.018+0000 7fb83e154640 1 --2- 172.21.15.90:0/2347185064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb838108a60 0x7fb838199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.018+0000 7fb83e154640 1 -- 172.21.15.90:0/2347185064 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb8381031c0 msgr2=0x7fb838199700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:47.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.018+0000 7fb83e154640 1 --2- 172.21.15.90:0/2347185064 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb8381031c0 0x7fb838199700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.019+0000 7fb83e154640 1 -- 172.21.15.90:0/2347185064 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb83819e560 con 0x7fb838108f00 2024-04-25T11:36:47.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.019+0000 7fb83d152640 1 --2- 172.21.15.90:0/2347185064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb838108a60 0x7fb838199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:36:47.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.019+0000 7fb83e154640 1 --2- 172.21.15.90:0/2347185064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb838108f00 0x7fb83819df20 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb83000eaa0 tx=0x7fb83000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:47.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.019+0000 7fb82effd640 1 -- 172.21.15.90:0/2347185064 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb83000cdb0 con 0x7fb838108f00 2024-04-25T11:36:47.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.019+0000 7fb83fbb7640 1 -- 172.21.15.90:0/2347185064 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb83819e850 con 0x7fb838108f00 2024-04-25T11:36:47.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.019+0000 7fb82effd640 1 -- 172.21.15.90:0/2347185064 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb830005020 con 0x7fb838108f00 2024-04-25T11:36:47.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.019+0000 7fb83fbb7640 1 -- 172.21.15.90:0/2347185064 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8381a61d0 con 0x7fb838108f00 2024-04-25T11:36:47.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.019+0000 7fb82effd640 1 -- 172.21.15.90:0/2347185064 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8300116d0 con 0x7fb838108f00 2024-04-25T11:36:47.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.021+0000 7fb846a6d740 1 Processor -- start 2024-04-25T11:36:47.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.022+0000 7fb82effd640 1 -- 172.21.15.90:0/2347185064 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fb830011870 con 0x7fb838108f00 2024-04-25T11:36:47.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.022+0000 7fb846a6d740 1 -- start start 2024-04-25T11:36:47.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.022+0000 7fb846a6d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3801b0 0x55c9cc71f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.022+0000 7fb846a6d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3572f0 0x55c9cc71fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.022+0000 7fb846a6d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc35f4d0 0x55c9cc720380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.022+0000 7fb846a6d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c9cc686460 con 0x55c9cc3801b0 2024-04-25T11:36:47.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.022+0000 7fb846a6d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c9cc640590 con 0x55c9cc3572f0 2024-04-25T11:36:47.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.022+0000 7fb846a6d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c9cc682420 con 0x55c9cc35f4d0 2024-04-25T11:36:47.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.022+0000 7fb83d152640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3801b0 0x55c9cc71f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.022+0000 7fb83d152640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3801b0 0x55c9cc71f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52312/0 (socket says 172.21.15.90:52312) 2024-04-25T11:36:47.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.022+0000 7fb83d152640 1 -- 172.21.15.90:0/101664132 learned_addr learned my addr 172.21.15.90:0/101664132 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:47.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.022+0000 7fb83d953640 1 --2- 172.21.15.90:0/101664132 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3572f0 0x55c9cc71fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.022+0000 7fb82effd640 1 --2- 172.21.15.90:0/2347185064 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8240afc50 0x7fb8240b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.022+0000 7fb83d152640 1 --2- 172.21.15.90:0/101664132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc35f4d0 0x55c9cc720380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.023+0000 7fb83d152640 1 -- 172.21.15.90:0/101664132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc35f4d0 msgr2=0x55c9cc720380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:47.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.023+0000 7fb83d152640 1 --2- 172.21.15.90:0/101664132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc35f4d0 0x55c9cc720380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.023+0000 7fb83d152640 1 -- 172.21.15.90:0/101664132 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3572f0 msgr2=0x55c9cc71fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:47.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.023+0000 7fb83d152640 1 --2- 172.21.15.90:0/101664132 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3572f0 0x55c9cc71fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.023+0000 7fb83d152640 1 -- 172.21.15.90:0/101664132 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9cc64d0d0 con 0x55c9cc3801b0 2024-04-25T11:36:47.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.023+0000 7fb83e154640 1 --2- 172.21.15.90:0/2347185064 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8240afc50 0x7fb8240b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.023+0000 7fb83d152640 1 --2- 172.21.15.90:0/101664132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3801b0 0x55c9cc71f900 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb834005ab0 tx=0x7fb834009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8b53c1d222a4a47 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:47.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.023+0000 7fb82effd640 1 -- 172.21.15.90:0/2347185064 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb8300df270 con 0x7fb838108f00 2024-04-25T11:36:47.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.023+0000 7fb823fff640 1 -- 172.21.15.90:0/101664132 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb834035050 con 0x55c9cc3801b0 2024-04-25T11:36:47.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.023+0000 7fb823fff640 1 -- 172.21.15.90:0/101664132 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8340316f0 con 0x55c9cc3801b0 2024-04-25T11:36:47.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.023+0000 7fb823fff640 1 -- 172.21.15.90:0/101664132 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8340046c0 con 0x55c9cc3801b0 2024-04-25T11:36:47.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.024+0000 7fb846a6d740 1 -- 172.21.15.90:0/101664132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3801b0 msgr2=0x55c9cc71f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:47.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.024+0000 7fb83e154640 1 --2- 172.21.15.90:0/2347185064 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8240afc50 0x7fb8240b2160 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb8300a3430 tx=0x7fb830075eb0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:47.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.024+0000 7fb846a6d740 1 --2- 172.21.15.90:0/101664132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3801b0 0x55c9cc71f900 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb834005ab0 tx=0x7fb834009560 comp rx=0 tx=0).stop 2024-04-25T11:36:47.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.024+0000 7fb846a6d740 1 -- 172.21.15.90:0/101664132 shutdown_connections 2024-04-25T11:36:47.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.024+0000 7fb846a6d740 1 --2- 172.21.15.90:0/101664132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc35f4d0 0x55c9cc720380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.024+0000 7fb846a6d740 1 --2- 172.21.15.90:0/101664132 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3572f0 0x55c9cc71fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.024+0000 7fb846a6d740 1 --2- 172.21.15.90:0/101664132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3801b0 0x55c9cc71f900 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.024+0000 7fb846a6d740 1 -- 172.21.15.90:0/101664132 >> 172.21.15.90:0/101664132 conn(0x55c9cc374300 msgr2=0x55c9cc71c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:47.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.024+0000 7fb846a6d740 1 -- 172.21.15.90:0/101664132 shutdown_connections 2024-04-25T11:36:47.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.024+0000 7fb846a6d740 1 -- 172.21.15.90:0/101664132 wait complete. 2024-04-25T11:36:47.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.024+0000 7fb846a6d740 1 -- start start 2024-04-25T11:36:47.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.025+0000 7fb846a6d740 1 Processor -- start 2024-04-25T11:36:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.025+0000 7fb846a6d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc3572f0 0x55c9cc72c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.025+0000 7fb846a6d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc35f4d0 0x55c9cc72c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.025+0000 7fb846a6d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3801b0 0x55c9cc7322b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.025+0000 7fb846a6d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c9cc72ee30 con 0x55c9cc35f4d0 2024-04-25T11:36:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.025+0000 7fb846a6d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c9cc60db60 con 0x55c9cc3801b0 2024-04-25T11:36:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.025+0000 7fb846a6d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c9cc72efb0 con 0x55c9cc3572f0 2024-04-25T11:36:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.025+0000 7fb846a6d740 10 client.0 _ll_register_callbacks cb 0x55c9cc6322b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.025+0000 7fb83d953640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc35f4d0 0x55c9cc72c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.025+0000 7fb83d953640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc35f4d0 0x55c9cc72c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52326/0 (socket says 172.21.15.90:52326) 2024-04-25T11:36:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.025+0000 7fb83d953640 1 -- 172.21.15.90:0/987698750 learned_addr learned my addr 172.21.15.90:0/987698750 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.025+0000 7fb83d152640 1 --2- 172.21.15.90:0/987698750 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3801b0 0x55c9cc7322b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.025+0000 7fb83d152640 1 --2- 172.21.15.90:0/987698750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc3572f0 0x55c9cc72c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.025+0000 7fb83d953640 1 -- 172.21.15.90:0/987698750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc3572f0 msgr2=0x55c9cc72c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:47.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.025+0000 7fb83d953640 1 --2- 172.21.15.90:0/987698750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc3572f0 0x55c9cc72c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.025+0000 7fb83d953640 1 -- 172.21.15.90:0/987698750 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3801b0 msgr2=0x55c9cc7322b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:47.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.025+0000 7fb83d953640 1 --2- 172.21.15.90:0/987698750 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3801b0 0x55c9cc7322b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.025+0000 7fb83d953640 1 -- 172.21.15.90:0/987698750 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9cc398210 con 0x55c9cc35f4d0 2024-04-25T11:36:47.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.025+0000 7fb83d152640 1 --2- 172.21.15.90:0/987698750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc3572f0 0x55c9cc72c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:36:47.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.026+0000 7fb83d953640 1 --2- 172.21.15.90:0/987698750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc35f4d0 0x55c9cc72c880 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fb82800b990 tx=0x7fb82800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3912c7431b8fae9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:47.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.026+0000 7fb8217fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:47.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.026+0000 7fb8217fa640 1 -- 172.21.15.90:0/987698750 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb828005020 con 0x55c9cc35f4d0 2024-04-25T11:36:47.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.026+0000 7fb8217fa640 1 -- 172.21.15.90:0/987698750 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb8280051c0 con 0x55c9cc35f4d0 2024-04-25T11:36:47.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.026+0000 7fb846a6d740 1 -- 172.21.15.90:0/987698750 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8340316f0 con 0x55c9cc35f4d0 2024-04-25T11:36:47.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.026+0000 7fb8217fa640 1 -- 172.21.15.90:0/987698750 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb8280053c0 con 0x55c9cc35f4d0 2024-04-25T11:36:47.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.026+0000 7fb846a6d740 10 client.16020 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:47.020 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.026+0000 7fb846a6d740 1 -- 172.21.15.90:0/987698750 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c9cc464620 con 0x55c9cc35f4d0 2024-04-25T11:36:47.020 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.026+0000 7fb8217fa640 1 -- 172.21.15.90:0/987698750 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8340316f0 con 0x55c9cc35f4d0 2024-04-25T11:36:47.020 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.026+0000 7fb8217fa640 1 -- 172.21.15.90:0/987698750 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fb82801b590 con 0x55c9cc35f4d0 2024-04-25T11:36:47.020 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.026+0000 7fb846a6d740 10 client.16020 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:47.020 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.026+0000 7fb846a6d740 10 client.16020 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:36:47.020 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.026+0000 7fb846a6d740 20 client.16020 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:47.020 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.026+0000 7fb846a6d740 1 --2- 172.21.15.90:0/987698750 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c9cc33ed80 0x55c9cc73e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.020 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.026+0000 7fb846a6d740 4 client.16020 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:36:47.020 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.026+0000 7fb846a6d740 1 -- 172.21.15.90:0/987698750 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c9cc694070 con 0x55c9cc33ed80 2024-04-25T11:36:47.020 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.027+0000 7fb83d152640 1 --2- 172.21.15.90:0/987698750 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c9cc33ed80 0x55c9cc73e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.020 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.027+0000 7fb8217fa640 10 client.16020 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:47.020 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.027+0000 7fb83d152640 1 --2- 172.21.15.90:0/987698750 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c9cc33ed80 0x55c9cc73e930 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ef94dbaa6d79ae4 server_cookie=46e8761873a1fbea in_seq=0 out_seq=0 2024-04-25T11:36:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.028+0000 7fb8217fa640 1 -- 172.21.15.90:0/987698750 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c9cc694070 con 0x55c9cc33ed80 2024-04-25T11:36:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.028+0000 7fb8217fa640 10 client.16020 handle_command_reply: tid=0 2024-04-25T11:36:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.028+0000 7fb846a6d740 1 client.16020 shutdown 2024-04-25T11:36:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.028+0000 7fb846a6d740 1 -- 172.21.15.90:0/987698750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc35f4d0 msgr2=0x55c9cc72c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.028+0000 7fb846a6d740 1 --2- 172.21.15.90:0/987698750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc35f4d0 0x55c9cc72c880 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fb82800b990 tx=0x7fb82800be50 comp rx=0 tx=0).stop 2024-04-25T11:36:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.028+0000 7fb846a6d740 1 -- 172.21.15.90:0/987698750 shutdown_connections 2024-04-25T11:36:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.028+0000 7fb846a6d740 1 --2- 172.21.15.90:0/987698750 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c9cc33ed80 0x55c9cc73e930 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.028+0000 7fb846a6d740 1 --2- 172.21.15.90:0/987698750 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3801b0 0x55c9cc7322b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.028+0000 7fb8217fa640 0 client.16020 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.028+0000 7fb846a6d740 1 --2- 172.21.15.90:0/987698750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc35f4d0 0x55c9cc72c880 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.028+0000 7fb846a6d740 1 --2- 172.21.15.90:0/987698750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc3572f0 0x55c9cc72c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.028+0000 7fb846a6d740 1 -- 172.21.15.90:0/987698750 >> 172.21.15.90:0/987698750 conn(0x55c9cc374300 msgr2=0x55c9cc7217a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:47.023 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.028+0000 7fb846a6d740 1 -- 172.21.15.90:0/987698750 shutdown_connections 2024-04-25T11:36:47.023 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.028+0000 7fb846a6d740 1 -- 172.21.15.90:0/987698750 wait complete. 2024-04-25T11:36:47.023 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.028+0000 7fb846a6d740 20 client.16020 trim_cache size 0 max 16384 2024-04-25T11:36:47.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.043+0000 7fb846a6d740 1 Processor -- start 2024-04-25T11:36:47.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.043+0000 7fb846a6d740 1 -- start start 2024-04-25T11:36:47.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.044+0000 7fb846a6d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3572f0 0x55c9cc7321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.044+0000 7fb846a6d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc35f4d0 0x55c9cc72b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.044+0000 7fb846a6d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3801b0 0x55c9cc7440f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.044+0000 7fb846a6d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb8381ad9d0 con 0x55c9cc3801b0 2024-04-25T11:36:47.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.044+0000 7fb846a6d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb8381af510 con 0x55c9cc3572f0 2024-04-25T11:36:47.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.044+0000 7fb846a6d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb8381ae0a0 con 0x55c9cc35f4d0 2024-04-25T11:36:47.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.044+0000 7fb83d152640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3801b0 0x55c9cc7440f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.044+0000 7fb83d152640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3801b0 0x55c9cc7440f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52330/0 (socket says 172.21.15.90:52330) 2024-04-25T11:36:47.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.044+0000 7fb83d152640 1 -- 172.21.15.90:0/864388776 learned_addr learned my addr 172.21.15.90:0/864388776 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:47.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.044+0000 7fb83d152640 1 --2- 172.21.15.90:0/864388776 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3572f0 0x55c9cc7321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.044+0000 7fb83d152640 1 -- 172.21.15.90:0/864388776 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc35f4d0 msgr2=0x55c9cc72b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:47.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.044+0000 7fb83d953640 1 --2- 172.21.15.90:0/864388776 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc35f4d0 0x55c9cc72b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.044+0000 7fb83d152640 1 --2- 172.21.15.90:0/864388776 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc35f4d0 0x55c9cc72b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.044+0000 7fb83d152640 1 -- 172.21.15.90:0/864388776 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3572f0 msgr2=0x55c9cc7321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:47.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.044+0000 7fb83d152640 1 --2- 172.21.15.90:0/864388776 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3572f0 0x55c9cc7321e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.044+0000 7fb83d152640 1 -- 172.21.15.90:0/864388776 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9cc717a60 con 0x55c9cc3801b0 2024-04-25T11:36:47.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.044+0000 7fb83d152640 1 --2- 172.21.15.90:0/864388776 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3801b0 0x55c9cc7440f0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fb834033000 tx=0x7fb83404b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5da47273b81bd82f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:47.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.044+0000 7fb8237fe640 1 -- 172.21.15.90:0/864388776 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb834035040 con 0x55c9cc3801b0 2024-04-25T11:36:47.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.044+0000 7fb8237fe640 1 -- 172.21.15.90:0/864388776 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb83404e070 con 0x55c9cc3801b0 2024-04-25T11:36:47.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.044+0000 7fb8237fe640 1 -- 172.21.15.90:0/864388776 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8340043b0 con 0x55c9cc3801b0 2024-04-25T11:36:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.045+0000 7fb846a6d740 1 -- 172.21.15.90:0/864388776 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3801b0 msgr2=0x55c9cc7440f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.045+0000 7fb846a6d740 1 --2- 172.21.15.90:0/864388776 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3801b0 0x55c9cc7440f0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fb834033000 tx=0x7fb83404b040 comp rx=0 tx=0).stop 2024-04-25T11:36:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.045+0000 7fb846a6d740 1 -- 172.21.15.90:0/864388776 shutdown_connections 2024-04-25T11:36:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.045+0000 7fb846a6d740 1 --2- 172.21.15.90:0/864388776 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3801b0 0x55c9cc7440f0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.045+0000 7fb846a6d740 1 --2- 172.21.15.90:0/864388776 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc35f4d0 0x55c9cc72b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.045+0000 7fb846a6d740 1 --2- 172.21.15.90:0/864388776 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3572f0 0x55c9cc7321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.045+0000 7fb846a6d740 1 -- 172.21.15.90:0/864388776 >> 172.21.15.90:0/864388776 conn(0x55c9cc374300 msgr2=0x55c9cc72aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.045+0000 7fb846a6d740 1 -- 172.21.15.90:0/864388776 shutdown_connections 2024-04-25T11:36:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.045+0000 7fb846a6d740 1 -- 172.21.15.90:0/864388776 wait complete. 2024-04-25T11:36:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.046+0000 7fb846a6d740 1 -- start start 2024-04-25T11:36:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.046+0000 7fb846a6d740 1 Processor -- start 2024-04-25T11:36:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.046+0000 7fb846a6d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3572f0 0x55c9cc738ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.046+0000 7fb846a6d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc35f4d0 0x55c9cc739000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.046+0000 7fb846a6d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc3801b0 0x55c9cc725300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.046+0000 7fb846a6d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb8381aee40 con 0x55c9cc35f4d0 2024-04-25T11:36:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.046+0000 7fb846a6d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb8381ae770 con 0x55c9cc3572f0 2024-04-25T11:36:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.046+0000 7fb846a6d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb8381ab9b0 con 0x55c9cc3801b0 2024-04-25T11:36:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.046+0000 7fb846a6d740 10 client.0 _ll_register_callbacks cb 0x7fb8280042b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.046+0000 7fb83d953640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc35f4d0 0x55c9cc739000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.046+0000 7fb83d953640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc35f4d0 0x55c9cc739000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52332/0 (socket says 172.21.15.90:52332) 2024-04-25T11:36:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.046+0000 7fb83d953640 1 -- 172.21.15.90:0/954988611 learned_addr learned my addr 172.21.15.90:0/954988611 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.046+0000 7fb83d152640 1 --2- 172.21.15.90:0/954988611 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3572f0 0x55c9cc738ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.046+0000 7fb83d953640 1 -- 172.21.15.90:0/954988611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc3801b0 msgr2=0x55c9cc725300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.046+0000 7fb83d953640 1 --2- 172.21.15.90:0/954988611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc3801b0 0x55c9cc725300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.046+0000 7fb83d953640 1 -- 172.21.15.90:0/954988611 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3572f0 msgr2=0x55c9cc738ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.046+0000 7fb83d953640 1 --2- 172.21.15.90:0/954988611 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3572f0 0x55c9cc738ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.047+0000 7fb83d953640 1 -- 172.21.15.90:0/954988611 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9cc725a10 con 0x55c9cc35f4d0 2024-04-25T11:36:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.047+0000 7fb83d152640 1 --2- 172.21.15.90:0/954988611 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3572f0 0x55c9cc738ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:36:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.047+0000 7fb83d953640 1 --2- 172.21.15.90:0/954988611 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc35f4d0 0x55c9cc739000 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fb828000c00 tx=0x7fb828003fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd0508073692f7bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.047+0000 7fb8217fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.047+0000 7fb8217fa640 1 -- 172.21.15.90:0/954988611 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82802f020 con 0x55c9cc35f4d0 2024-04-25T11:36:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.047+0000 7fb8217fa640 1 -- 172.21.15.90:0/954988611 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb82801b470 con 0x55c9cc35f4d0 2024-04-25T11:36:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.047+0000 7fb846a6d740 1 -- 172.21.15.90:0/954988611 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb83404e070 con 0x55c9cc35f4d0 2024-04-25T11:36:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.047+0000 7fb8217fa640 1 -- 172.21.15.90:0/954988611 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb828027040 con 0x55c9cc35f4d0 2024-04-25T11:36:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.047+0000 7fb846a6d740 10 client.16026 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.047+0000 7fb846a6d740 1 -- 172.21.15.90:0/954988611 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c9cc736f20 con 0x55c9cc35f4d0 2024-04-25T11:36:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.047+0000 7fb8217fa640 1 -- 172.21.15.90:0/954988611 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb83404e070 con 0x55c9cc35f4d0 2024-04-25T11:36:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.047+0000 7fb8217fa640 1 -- 172.21.15.90:0/954988611 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fb82802a030 con 0x55c9cc35f4d0 2024-04-25T11:36:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.047+0000 7fb846a6d740 10 client.16026 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.047+0000 7fb846a6d740 10 client.16026 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:36:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.047+0000 7fb846a6d740 20 client.16026 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.047+0000 7fb846a6d740 1 --2- 172.21.15.90:0/954988611 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c9cc3141c0 0x55c9cc7ab220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.047+0000 7fb846a6d740 4 client.16026 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:36:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.047+0000 7fb846a6d740 1 -- 172.21.15.90:0/954988611 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c9cc68dee0 con 0x55c9cc3141c0 2024-04-25T11:36:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.048+0000 7fb83d152640 1 --2- 172.21.15.90:0/954988611 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c9cc3141c0 0x55c9cc7ab220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.048+0000 7fb8217fa640 10 client.16026 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.048+0000 7fb83d152640 1 --2- 172.21.15.90:0/954988611 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c9cc3141c0 0x55c9cc7ab220 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3668036cf3d27f9d server_cookie=5ceb8321602f7218 in_seq=0 out_seq=0 2024-04-25T11:36:47.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.066+0000 7fb8217fa640 1 -- 172.21.15.90:0/954988611 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c9cc68dee0 con 0x55c9cc3141c0 2024-04-25T11:36:47.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.066+0000 7fb8217fa640 10 client.16026 handle_command_reply: tid=0 2024-04-25T11:36:47.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.066+0000 7fb846a6d740 1 client.16026 shutdown 2024-04-25T11:36:47.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.066+0000 7fb846a6d740 1 -- 172.21.15.90:0/954988611 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc35f4d0 msgr2=0x55c9cc739000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:47.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.066+0000 7fb846a6d740 1 --2- 172.21.15.90:0/954988611 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc35f4d0 0x55c9cc739000 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fb828000c00 tx=0x7fb828003fb0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.066+0000 7fb846a6d740 1 -- 172.21.15.90:0/954988611 shutdown_connections 2024-04-25T11:36:47.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.066+0000 7fb846a6d740 1 --2- 172.21.15.90:0/954988611 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c9cc3141c0 0x55c9cc7ab220 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.066+0000 7fb846a6d740 1 --2- 172.21.15.90:0/954988611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc3801b0 0x55c9cc725300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.066+0000 7fb846a6d740 1 --2- 172.21.15.90:0/954988611 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc35f4d0 0x55c9cc739000 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.066+0000 7fb8217fa640 0 client.16026 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.066+0000 7fb846a6d740 1 --2- 172.21.15.90:0/954988611 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3572f0 0x55c9cc738ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.066+0000 7fb846a6d740 1 -- 172.21.15.90:0/954988611 >> 172.21.15.90:0/954988611 conn(0x55c9cc374300 msgr2=0x55c9cc721d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.066+0000 7fb846a6d740 1 -- 172.21.15.90:0/954988611 shutdown_connections 2024-04-25T11:36:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.066+0000 7fb846a6d740 1 -- 172.21.15.90:0/954988611 wait complete. 2024-04-25T11:36:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.066+0000 7fb846a6d740 20 client.16026 trim_cache size 0 max 16384 2024-04-25T11:36:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:36:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:36:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:36:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:36:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.067+0000 7fb83fbb7640 1 -- 172.21.15.90:0/2347185064 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8240afc50 msgr2=0x7fb8240b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.067+0000 7fb83fbb7640 1 --2- 172.21.15.90:0/2347185064 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8240afc50 0x7fb8240b2160 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb8300a3430 tx=0x7fb830075eb0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.067+0000 7fb83fbb7640 1 -- 172.21.15.90:0/2347185064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb838108f00 msgr2=0x7fb83819df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.067+0000 7fb83fbb7640 1 --2- 172.21.15.90:0/2347185064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb838108f00 0x7fb83819df20 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb83000eaa0 tx=0x7fb83000ef60 comp rx=0 tx=0).stop 2024-04-25T11:36:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.067+0000 7fb83fbb7640 1 -- 172.21.15.90:0/2347185064 shutdown_connections 2024-04-25T11:36:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.067+0000 7fb83fbb7640 1 --2- 172.21.15.90:0/2347185064 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8240afc50 0x7fb8240b2160 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.067+0000 7fb83fbb7640 1 --2- 172.21.15.90:0/2347185064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb838108f00 0x7fb83819df20 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.067+0000 7fb83fbb7640 1 --2- 172.21.15.90:0/2347185064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb838108a60 0x7fb838199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.067+0000 7fb83fbb7640 1 --2- 172.21.15.90:0/2347185064 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb8381031c0 0x7fb838199700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.067+0000 7fb83fbb7640 1 -- 172.21.15.90:0/2347185064 >> 172.21.15.90:0/2347185064 conn(0x7fb8380781b0 msgr2=0x7fb838104140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.067+0000 7fb83fbb7640 1 -- 172.21.15.90:0/2347185064 shutdown_connections 2024-04-25T11:36:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:36:47.067+0000 7fb83fbb7640 1 -- 172.21.15.90:0/2347185064 wait complete. 2024-04-25T11:36:47.094 INFO:tasks.workunit.client.0.smithi090.stdout:running iogen 2024-04-25T11:36:47.095 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'running iogen' 2024-04-25T11:36:47.095 INFO:tasks.workunit.client.0.smithi090.stderr:+ ./iogen -n 5 -s 2g 2024-04-25T11:36:47.095 INFO:tasks.workunit.client.0.smithi090.stderr:child spawned 86696 2024-04-25T11:36:47.095 INFO:tasks.workunit.client.0.smithi090.stderr:child spawned 86697 2024-04-25T11:36:47.095 INFO:tasks.workunit.client.0.smithi090.stderr:child spawned 86698 2024-04-25T11:36:47.095 INFO:tasks.workunit.client.0.smithi090.stderr:child spawned 86699 2024-04-25T11:36:47.095 INFO:tasks.workunit.client.0.smithi090.stderr:child spawned 86700 2024-04-25T11:36:47.096 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'sleep for 10 min' 2024-04-25T11:36:47.096 INFO:tasks.workunit.client.0.smithi090.stdout:sleep for 10 min 2024-04-25T11:36:47.096 INFO:tasks.workunit.client.0.smithi090.stderr:+ sleep 600 2024-04-25T11:36:47.419 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:47 smithi090 ceph-mon[29575]: pgmap v54: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 853 B/s wr, 10 op/s 2024-04-25T11:36:47.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:47 smithi138 ceph-mon[32328]: pgmap v54: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 853 B/s wr, 10 op/s 2024-04-25T11:36:47.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:47 smithi156 ceph-mon[33045]: pgmap v54: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 853 B/s wr, 10 op/s 2024-04-25T11:36:48.691 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:36:48.692 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:36:48.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.878+0000 7f60beffd640 1 -- 172.21.15.90:0/231174953 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b802f340 con 0x7f60c01064a0 2024-04-25T11:36:48.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.879+0000 7f60c66d8640 1 -- 172.21.15.90:0/231174953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60c01064a0 msgr2=0x7f60c0112870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:48.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.879+0000 7f60c66d8640 1 --2- 172.21.15.90:0/231174953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60c01064a0 0x7f60c0112870 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f60b8009ca0 tx=0x7f60b802a8f0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.879+0000 7f60c66d8640 1 -- 172.21.15.90:0/231174953 shutdown_connections 2024-04-25T11:36:48.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.879+0000 7f60c66d8640 1 --2- 172.21.15.90:0/231174953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60c01064a0 0x7f60c0112870 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.879+0000 7f60c66d8640 1 --2- 172.21.15.90:0/231174953 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f60c010c1f0 0x7f60c0105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.879+0000 7f60c66d8640 1 --2- 172.21.15.90:0/231174953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60c010bd50 0x7f60c01058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.879+0000 7f60c66d8640 1 -- 172.21.15.90:0/231174953 >> 172.21.15.90:0/231174953 conn(0x7f60c01013f0 msgr2=0x7f60c0103840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:48.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.879+0000 7f60c66d8640 1 -- 172.21.15.90:0/231174953 shutdown_connections 2024-04-25T11:36:48.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.879+0000 7f60c66d8640 1 -- 172.21.15.90:0/231174953 wait complete. 2024-04-25T11:36:48.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.880+0000 7f60c66d8640 1 Processor -- start 2024-04-25T11:36:48.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.880+0000 7f60c66d8640 1 -- start start 2024-04-25T11:36:48.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.881+0000 7f60c66d8640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60c01064a0 0x7f60c01a26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:48.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.881+0000 7f60c66d8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60c010bd50 0x7f60c01a2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:48.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.881+0000 7f60c66d8640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f60c010c1f0 0x7f60c019c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:48.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.881+0000 7f60c66d8640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f60c0116090 con 0x7f60c01064a0 2024-04-25T11:36:48.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.881+0000 7f60c66d8640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f60c0115f10 con 0x7f60c010c1f0 2024-04-25T11:36:48.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.881+0000 7f60bffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60c01064a0 0x7f60c01a26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:48.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.881+0000 7f60c66d8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f60c0116210 con 0x7f60c010bd50 2024-04-25T11:36:48.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.881+0000 7f60bffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60c01064a0 0x7f60c01a26a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52356/0 (socket says 172.21.15.90:52356) 2024-04-25T11:36:48.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.881+0000 7f60bffff640 1 -- 172.21.15.90:0/2291279274 learned_addr learned my addr 172.21.15.90:0/2291279274 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:48.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.881+0000 7f60c4c75640 1 --2- 172.21.15.90:0/2291279274 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f60c010c1f0 0x7f60c019c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:48.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.882+0000 7f60bffff640 1 -- 172.21.15.90:0/2291279274 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60c010bd50 msgr2=0x7f60c01a2be0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:36:48.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.882+0000 7f60bffff640 1 --2- 172.21.15.90:0/2291279274 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60c010bd50 0x7f60c01a2be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.882+0000 7f60bffff640 1 -- 172.21.15.90:0/2291279274 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f60c010c1f0 msgr2=0x7f60c019c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:48.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.882+0000 7f60bffff640 1 --2- 172.21.15.90:0/2291279274 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f60c010c1f0 0x7f60c019c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.882+0000 7f60bffff640 1 -- 172.21.15.90:0/2291279274 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60c019cf50 con 0x7f60c01064a0 2024-04-25T11:36:48.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.882+0000 7f60c4c75640 1 --2- 172.21.15.90:0/2291279274 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f60c010c1f0 0x7f60c019c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:36:48.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.882+0000 7f60bffff640 1 --2- 172.21.15.90:0/2291279274 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60c01064a0 0x7f60c01a26a0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f60b000b470 tx=0x7f60b000b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:48.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.882+0000 7f60bd7fa640 1 -- 172.21.15.90:0/2291279274 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b0015020 con 0x7f60c01064a0 2024-04-25T11:36:48.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.882+0000 7f60c66d8640 1 -- 172.21.15.90:0/2291279274 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60c019d240 con 0x7f60c01064a0 2024-04-25T11:36:48.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.882+0000 7f60c66d8640 1 -- 172.21.15.90:0/2291279274 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60c01a9520 con 0x7f60c01064a0 2024-04-25T11:36:48.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.882+0000 7f60bd7fa640 1 -- 172.21.15.90:0/2291279274 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60b0011d40 con 0x7f60c01064a0 2024-04-25T11:36:48.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.883+0000 7f60bd7fa640 1 -- 172.21.15.90:0/2291279274 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b000f340 con 0x7f60c01064a0 2024-04-25T11:36:48.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.887+0000 7f60cd583740 1 Processor -- start 2024-04-25T11:36:48.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.887+0000 7f60cd583740 1 -- start start 2024-04-25T11:36:48.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.887+0000 7f60cd583740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a82fe4390 0x564a83384a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:48.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.887+0000 7f60bd7fa640 1 -- 172.21.15.90:0/2291279274 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f60b0018020 con 0x7f60c01064a0 2024-04-25T11:36:48.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.887+0000 7f60cd583740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564a82fbb4d0 0x564a83384fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:48.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.888+0000 7f60cd583740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fc36b0 0x564a833918f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:48.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.888+0000 7f60cd583740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564a832ead80 con 0x564a82fc36b0 2024-04-25T11:36:48.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.888+0000 7f60cd583740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x564a832a48b0 con 0x564a82fbb4d0 2024-04-25T11:36:48.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.888+0000 7f60cd583740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x564a832e6d40 con 0x564a82fe4390 2024-04-25T11:36:48.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.888+0000 7f60bf7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fc36b0 0x564a833918f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:48.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.888+0000 7f60bf7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fc36b0 0x564a833918f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52358/0 (socket says 172.21.15.90:52358) 2024-04-25T11:36:48.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.888+0000 7f60bf7fe640 1 -- 172.21.15.90:0/1063088593 learned_addr learned my addr 172.21.15.90:0/1063088593 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:48.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.888+0000 7f60bf7fe640 1 -- 172.21.15.90:0/1063088593 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a82fe4390 msgr2=0x564a83384a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:48.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.888+0000 7f60bf7fe640 1 --2- 172.21.15.90:0/1063088593 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a82fe4390 0x564a83384a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.888+0000 7f60bf7fe640 1 -- 172.21.15.90:0/1063088593 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564a82fbb4d0 msgr2=0x564a83384fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:36:48.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.888+0000 7f60bf7fe640 1 --2- 172.21.15.90:0/1063088593 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564a82fbb4d0 0x564a83384fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.888+0000 7f60bf7fe640 1 -- 172.21.15.90:0/1063088593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a83275a40 con 0x564a82fc36b0 2024-04-25T11:36:48.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.889+0000 7f60bf7fe640 1 --2- 172.21.15.90:0/1063088593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fc36b0 0x564a833918f0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f60b4009940 tx=0x7f60b4009e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=504d1e8a2ceccadb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:48.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.889+0000 7f60bd7fa640 1 --2- 172.21.15.90:0/2291279274 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f60a80afce0 0x7f60a80b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:48.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.889+0000 7f60c4c75640 1 --2- 172.21.15.90:0/2291279274 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f60a80afce0 0x7f60a80b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:48.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.889+0000 7f60a67fc640 1 -- 172.21.15.90:0/1063088593 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b4015020 con 0x564a82fc36b0 2024-04-25T11:36:48.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.889+0000 7f60a67fc640 1 -- 172.21.15.90:0/1063088593 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60b400be30 con 0x564a82fc36b0 2024-04-25T11:36:48.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.889+0000 7f60a67fc640 1 -- 172.21.15.90:0/1063088593 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b4010380 con 0x564a82fc36b0 2024-04-25T11:36:48.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.889+0000 7f60bd7fa640 1 -- 172.21.15.90:0/2291279274 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f60b00dea20 con 0x7f60c01064a0 2024-04-25T11:36:48.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.890+0000 7f60cd583740 1 -- 172.21.15.90:0/1063088593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fc36b0 msgr2=0x564a833918f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:48.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.890+0000 7f60cd583740 1 --2- 172.21.15.90:0/1063088593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fc36b0 0x564a833918f0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f60b4009940 tx=0x7f60b4009e00 comp rx=0 tx=0).stop 2024-04-25T11:36:48.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.890+0000 7f60c4c75640 1 --2- 172.21.15.90:0/2291279274 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f60a80afce0 0x7f60a80b21f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x564a83381bd0 tx=0x7f60b802f3f0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:48.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.890+0000 7f60cd583740 1 -- 172.21.15.90:0/1063088593 shutdown_connections 2024-04-25T11:36:48.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.890+0000 7f60cd583740 1 --2- 172.21.15.90:0/1063088593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fc36b0 0x564a833918f0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.891+0000 7f60cd583740 1 --2- 172.21.15.90:0/1063088593 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564a82fbb4d0 0x564a83384fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.891+0000 7f60cd583740 1 --2- 172.21.15.90:0/1063088593 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a82fe4390 0x564a83384a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.891+0000 7f60cd583740 1 -- 172.21.15.90:0/1063088593 >> 172.21.15.90:0/1063088593 conn(0x564a82fd84e0 msgr2=0x564a83381210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:48.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.891+0000 7f60cd583740 1 -- 172.21.15.90:0/1063088593 shutdown_connections 2024-04-25T11:36:48.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.891+0000 7f60cd583740 1 -- 172.21.15.90:0/1063088593 wait complete. 2024-04-25T11:36:48.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.892+0000 7f60cd583740 1 -- start start 2024-04-25T11:36:48.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.892+0000 7f60cd583740 1 Processor -- start 2024-04-25T11:36:48.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.893+0000 7f60cd583740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564a82fbb4d0 0x564a8338f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:48.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.893+0000 7f60cd583740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fc36b0 0x564a8338fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:48.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.893+0000 7f60cd583740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a82fe4390 0x564a8339b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:48.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.893+0000 7f60cd583740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564a83391fe0 con 0x564a82fc36b0 2024-04-25T11:36:48.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.893+0000 7f60cd583740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x564a83271d10 con 0x564a82fbb4d0 2024-04-25T11:36:48.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.893+0000 7f60cd583740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x564a83392160 con 0x564a82fe4390 2024-04-25T11:36:48.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.893+0000 7f60bf7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fc36b0 0x564a8338fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:48.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.893+0000 7f60cd583740 10 client.0 _ll_register_callbacks cb 0x564a8329e210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:48.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.893+0000 7f60bf7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fc36b0 0x564a8338fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52364/0 (socket says 172.21.15.90:52364) 2024-04-25T11:36:48.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.893+0000 7f60bf7fe640 1 -- 172.21.15.90:0/75552022 learned_addr learned my addr 172.21.15.90:0/75552022 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:48.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.894+0000 7f60bf7fe640 1 -- 172.21.15.90:0/75552022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a82fe4390 msgr2=0x564a8339b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:48.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.894+0000 7f60bf7fe640 1 --2- 172.21.15.90:0/75552022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a82fe4390 0x564a8339b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.894+0000 7f60bf7fe640 1 -- 172.21.15.90:0/75552022 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564a82fbb4d0 msgr2=0x564a8338f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:48.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.894+0000 7f60bf7fe640 1 --2- 172.21.15.90:0/75552022 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564a82fbb4d0 0x564a8338f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.894+0000 7f60bf7fe640 1 -- 172.21.15.90:0/75552022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a830c67d0 con 0x564a82fc36b0 2024-04-25T11:36:48.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.895+0000 7f60bf7fe640 1 --2- 172.21.15.90:0/75552022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fc36b0 0x564a8338fb00 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f60b4011b20 tx=0x7f60b4011650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=812040e8d48c0cb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:48.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.895+0000 7f609f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:48.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.895+0000 7f609f7fe640 1 -- 172.21.15.90:0/75552022 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b400bb90 con 0x564a82fc36b0 2024-04-25T11:36:48.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.895+0000 7f609f7fe640 1 -- 172.21.15.90:0/75552022 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f60b4024470 con 0x564a82fc36b0 2024-04-25T11:36:48.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.895+0000 7f60cd583740 1 -- 172.21.15.90:0/75552022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60b400be30 con 0x564a82fc36b0 2024-04-25T11:36:48.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.896+0000 7f609f7fe640 1 -- 172.21.15.90:0/75552022 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f60b4024690 con 0x564a82fc36b0 2024-04-25T11:36:48.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.896+0000 7f60cd583740 10 client.16038 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:48.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.896+0000 7f60cd583740 1 -- 172.21.15.90:0/75552022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564a82ffc3f0 con 0x564a82fc36b0 2024-04-25T11:36:48.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.896+0000 7f609f7fe640 1 -- 172.21.15.90:0/75552022 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60b401f070 con 0x564a82fc36b0 2024-04-25T11:36:48.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.896+0000 7f609f7fe640 1 -- 172.21.15.90:0/75552022 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f60b401a880 con 0x564a82fc36b0 2024-04-25T11:36:48.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.896+0000 7f60cd583740 10 client.16038 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:48.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.896+0000 7f60cd583740 10 client.16038 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:36:48.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.896+0000 7f60cd583740 20 client.16038 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:48.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.896+0000 7f60cd583740 1 --2- 172.21.15.90:0/75552022 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564a82fa2f60 0x564a833a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:48.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.897+0000 7f60cd583740 4 client.16038 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:36:48.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.897+0000 7f60cd583740 1 -- 172.21.15.90:0/75552022 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f60b400be30 con 0x564a82fa2f60 2024-04-25T11:36:48.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.900+0000 7f60bffff640 1 --2- 172.21.15.90:0/75552022 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564a82fa2f60 0x564a833a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:48.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.909+0000 7f60bffff640 1 --2- 172.21.15.90:0/75552022 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564a82fa2f60 0x564a833a15d0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=662ff03811c9c940 server_cookie=f284e9f79e2912f1 in_seq=0 out_seq=0 2024-04-25T11:36:48.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.909+0000 7f609f7fe640 10 client.16038 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:48.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.921+0000 7f609f7fe640 1 -- 172.21.15.90:0/75552022 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f60b400be30 con 0x564a82fa2f60 2024-04-25T11:36:48.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.922+0000 7f609f7fe640 10 client.16038 handle_command_reply: tid=0 2024-04-25T11:36:48.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.931+0000 7f60cd583740 1 client.16038 shutdown 2024-04-25T11:36:48.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.933+0000 7f60cd583740 1 -- 172.21.15.90:0/75552022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fc36b0 msgr2=0x564a8338fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:48.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.933+0000 7f60cd583740 1 --2- 172.21.15.90:0/75552022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fc36b0 0x564a8338fb00 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f60b4011b20 tx=0x7f60b4011650 comp rx=0 tx=0).stop 2024-04-25T11:36:48.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.934+0000 7f60cd583740 1 -- 172.21.15.90:0/75552022 shutdown_connections 2024-04-25T11:36:48.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.934+0000 7f60cd583740 1 --2- 172.21.15.90:0/75552022 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564a82fa2f60 0x564a833a15d0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.934+0000 7f60cd583740 1 --2- 172.21.15.90:0/75552022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a82fe4390 0x564a8339b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.934+0000 7f60cd583740 1 --2- 172.21.15.90:0/75552022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fc36b0 0x564a8338fb00 secure :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f60b4011b20 tx=0x7f60b4011650 comp rx=0 tx=0).stop 2024-04-25T11:36:48.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.934+0000 7f60cd583740 1 --2- 172.21.15.90:0/75552022 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564a82fbb4d0 0x564a8338f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.934+0000 7f60cd583740 1 -- 172.21.15.90:0/75552022 >> 172.21.15.90:0/75552022 conn(0x564a82fd84e0 msgr2=0x564a83386780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:48.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.934+0000 7f609f7fe640 0 client.16038 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:48.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.935+0000 7f60cd583740 1 -- 172.21.15.90:0/75552022 shutdown_connections 2024-04-25T11:36:48.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.935+0000 7f60cd583740 1 -- 172.21.15.90:0/75552022 wait complete. 2024-04-25T11:36:48.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.936+0000 7f60cd583740 20 client.16038 trim_cache size 0 max 16384 2024-04-25T11:36:48.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.965+0000 7f60cd583740 1 Processor -- start 2024-04-25T11:36:48.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.965+0000 7f60cd583740 1 -- start start 2024-04-25T11:36:48.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.966+0000 7f60cd583740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a82fbb4d0 0x564a833974d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:48.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.966+0000 7f60cd583740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fe4390 0x564a83397a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:48.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.966+0000 7f60cd583740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564a82f783a0 0x564a83386660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:48.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.966+0000 7f60cd583740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac00e950 con 0x564a82fe4390 2024-04-25T11:36:48.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.966+0000 7f60cd583740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac010490 con 0x564a82f783a0 2024-04-25T11:36:48.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.966+0000 7f60cd583740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac00f020 con 0x564a82fbb4d0 2024-04-25T11:36:48.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.966+0000 7f60bf7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fe4390 0x564a83397a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:48.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.966+0000 7f60bf7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fe4390 0x564a83397a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52380/0 (socket says 172.21.15.90:52380) 2024-04-25T11:36:48.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.966+0000 7f60bf7fe640 1 -- 172.21.15.90:0/2893351390 learned_addr learned my addr 172.21.15.90:0/2893351390 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:48.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.966+0000 7f60bf7fe640 1 -- 172.21.15.90:0/2893351390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a82fbb4d0 msgr2=0x564a833974d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:36:48.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.966+0000 7f60bf7fe640 1 --2- 172.21.15.90:0/2893351390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a82fbb4d0 0x564a833974d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.966+0000 7f60bf7fe640 1 -- 172.21.15.90:0/2893351390 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564a82f783a0 msgr2=0x564a83386660 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:36:48.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.966+0000 7f60bf7fe640 1 --2- 172.21.15.90:0/2893351390 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564a82f783a0 0x564a83386660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.966+0000 7f60bf7fe640 1 -- 172.21.15.90:0/2893351390 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a82fbc0f0 con 0x564a82fe4390 2024-04-25T11:36:48.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.966+0000 7f60bf7fe640 1 --2- 172.21.15.90:0/2893351390 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fe4390 0x564a83397a10 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f60b4009860 tx=0x7f60b4009e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f30d6f88a1f8e267 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:48.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.967+0000 7f60a5ffb640 1 -- 172.21.15.90:0/2893351390 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b4015040 con 0x564a82fe4390 2024-04-25T11:36:48.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.967+0000 7f60a5ffb640 1 -- 172.21.15.90:0/2893351390 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60b401f070 con 0x564a82fe4390 2024-04-25T11:36:48.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.967+0000 7f60a5ffb640 1 -- 172.21.15.90:0/2893351390 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b4024b60 con 0x564a82fe4390 2024-04-25T11:36:48.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.968+0000 7f60cd583740 1 -- 172.21.15.90:0/2893351390 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fe4390 msgr2=0x564a83397a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:48.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.968+0000 7f60cd583740 1 --2- 172.21.15.90:0/2893351390 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fe4390 0x564a83397a10 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f60b4009860 tx=0x7f60b4009e00 comp rx=0 tx=0).stop 2024-04-25T11:36:48.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.968+0000 7f60cd583740 1 -- 172.21.15.90:0/2893351390 shutdown_connections 2024-04-25T11:36:48.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.968+0000 7f60cd583740 1 --2- 172.21.15.90:0/2893351390 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564a82f783a0 0x564a83386660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.968+0000 7f60cd583740 1 --2- 172.21.15.90:0/2893351390 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fe4390 0x564a83397a10 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.968+0000 7f60cd583740 1 --2- 172.21.15.90:0/2893351390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a82fbb4d0 0x564a833974d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.968+0000 7f60cd583740 1 -- 172.21.15.90:0/2893351390 >> 172.21.15.90:0/2893351390 conn(0x564a82fd84e0 msgr2=0x564a8338dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:48.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.968+0000 7f60cd583740 1 -- 172.21.15.90:0/2893351390 shutdown_connections 2024-04-25T11:36:48.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.968+0000 7f60cd583740 1 -- 172.21.15.90:0/2893351390 wait complete. 2024-04-25T11:36:48.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.968+0000 7f60cd583740 1 -- start start 2024-04-25T11:36:48.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.969+0000 7f60cd583740 1 Processor -- start 2024-04-25T11:36:48.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.969+0000 7f60cd583740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564a82f783a0 0x564a83392ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:48.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.969+0000 7f60cd583740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fbb4d0 0x564a83393020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:48.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.969+0000 7f60cd583740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a82fe4390 0x564a83401880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:48.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.969+0000 7f60cd583740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac00fdc0 con 0x564a82fbb4d0 2024-04-25T11:36:48.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.969+0000 7f60cd583740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac00f6f0 con 0x564a82f783a0 2024-04-25T11:36:48.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.969+0000 7f60cd583740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac00c930 con 0x564a82fe4390 2024-04-25T11:36:48.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.969+0000 7f60cd583740 10 client.0 _ll_register_callbacks cb 0x7f60b4009740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:48.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.969+0000 7f60bf7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fbb4d0 0x564a83393020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:48.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.969+0000 7f60bf7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fbb4d0 0x564a83393020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52382/0 (socket says 172.21.15.90:52382) 2024-04-25T11:36:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.969+0000 7f60bf7fe640 1 -- 172.21.15.90:0/1116161007 learned_addr learned my addr 172.21.15.90:0/1116161007 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.969+0000 7f60bf7fe640 1 -- 172.21.15.90:0/1116161007 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a82fe4390 msgr2=0x564a83401880 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:36:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.969+0000 7f60bf7fe640 1 --2- 172.21.15.90:0/1116161007 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a82fe4390 0x564a83401880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.969+0000 7f60bf7fe640 1 -- 172.21.15.90:0/1116161007 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564a82f783a0 msgr2=0x564a83392ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.969+0000 7f60bf7fe640 1 --2- 172.21.15.90:0/1116161007 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564a82f783a0 0x564a83392ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.969+0000 7f60bf7fe640 1 -- 172.21.15.90:0/1116161007 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a832ab4b0 con 0x564a82fbb4d0 2024-04-25T11:36:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.970+0000 7f60bf7fe640 1 --2- 172.21.15.90:0/1116161007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fbb4d0 0x564a83393020 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f60b4011690 tx=0x7f60b400bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae0a0761f0b0080a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.970+0000 7f609f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.970+0000 7f609f7fe640 1 -- 172.21.15.90:0/1116161007 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b4015040 con 0x564a82fbb4d0 2024-04-25T11:36:48.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.970+0000 7f609f7fe640 1 -- 172.21.15.90:0/1116161007 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f60b4024a40 con 0x564a82fbb4d0 2024-04-25T11:36:48.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.970+0000 7f60cd583740 1 -- 172.21.15.90:0/1116161007 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60b401f070 con 0x564a82fbb4d0 2024-04-25T11:36:48.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.971+0000 7f609f7fe640 1 -- 172.21.15.90:0/1116161007 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f60b4043050 con 0x564a82fbb4d0 2024-04-25T11:36:48.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.971+0000 7f60cd583740 10 client.16044 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:48.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.971+0000 7f60cd583740 1 -- 172.21.15.90:0/1116161007 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564a83408f00 con 0x564a82fbb4d0 2024-04-25T11:36:48.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.971+0000 7f609f7fe640 1 -- 172.21.15.90:0/1116161007 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60b401f070 con 0x564a82fbb4d0 2024-04-25T11:36:48.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.972+0000 7f609f7fe640 1 -- 172.21.15.90:0/1116161007 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f60b4033020 con 0x564a82fbb4d0 2024-04-25T11:36:48.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.972+0000 7f60cd583740 10 client.16044 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:48.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.972+0000 7f60cd583740 10 client.16044 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:36:48.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.972+0000 7f60cd583740 20 client.16044 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:48.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.972+0000 7f60cd583740 1 --2- 172.21.15.90:0/1116161007 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564a83409560 0x564a8340b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:48.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.972+0000 7f60cd583740 4 client.16044 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:36:48.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.972+0000 7f60cd583740 1 -- 172.21.15.90:0/1116161007 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564a832f8990 con 0x564a83409560 2024-04-25T11:36:48.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.974+0000 7f60bffff640 1 --2- 172.21.15.90:0/1116161007 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564a83409560 0x564a8340b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:48.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.979+0000 7f609f7fe640 10 client.16044 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:48.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.979+0000 7f60bffff640 1 --2- 172.21.15.90:0/1116161007 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564a83409560 0x564a8340b980 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b1206476c6dbac5 server_cookie=25498644e6f686d8 in_seq=0 out_seq=0 2024-04-25T11:36:48.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.984+0000 7f609f7fe640 1 -- 172.21.15.90:0/1116161007 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564a832f8990 con 0x564a83409560 2024-04-25T11:36:48.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.984+0000 7f609f7fe640 10 client.16044 handle_command_reply: tid=0 2024-04-25T11:36:48.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.984+0000 7f60cd583740 1 client.16044 shutdown 2024-04-25T11:36:48.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.984+0000 7f60cd583740 1 -- 172.21.15.90:0/1116161007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fbb4d0 msgr2=0x564a83393020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:48.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.984+0000 7f60cd583740 1 --2- 172.21.15.90:0/1116161007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fbb4d0 0x564a83393020 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f60b4011690 tx=0x7f60b400bfd0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.984+0000 7f60cd583740 1 -- 172.21.15.90:0/1116161007 shutdown_connections 2024-04-25T11:36:48.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.984+0000 7f60cd583740 1 --2- 172.21.15.90:0/1116161007 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564a83409560 0x564a8340b980 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.984+0000 7f60cd583740 1 --2- 172.21.15.90:0/1116161007 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a82fe4390 0x564a83401880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.984+0000 7f60cd583740 1 --2- 172.21.15.90:0/1116161007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a82fbb4d0 0x564a83393020 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.984+0000 7f609f7fe640 0 client.16044 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:48.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.984+0000 7f60cd583740 1 --2- 172.21.15.90:0/1116161007 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564a82f783a0 0x564a83392ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.984+0000 7f60cd583740 1 -- 172.21.15.90:0/1116161007 >> 172.21.15.90:0/1116161007 conn(0x564a82fd84e0 msgr2=0x564a83386660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:48.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.984+0000 7f60cd583740 1 -- 172.21.15.90:0/1116161007 shutdown_connections 2024-04-25T11:36:48.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.984+0000 7f60cd583740 1 -- 172.21.15.90:0/1116161007 wait complete. 2024-04-25T11:36:48.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.984+0000 7f60cd583740 20 client.16044 trim_cache size 0 max 16384 2024-04-25T11:36:48.974 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:36:48.974 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-25T11:36:48.974 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-25T11:36:48.974 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:36:48.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.986+0000 7f60c66d8640 1 -- 172.21.15.90:0/2291279274 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f60a80afce0 msgr2=0x7f60a80b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:48.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.986+0000 7f60c66d8640 1 --2- 172.21.15.90:0/2291279274 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f60a80afce0 0x7f60a80b21f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x564a83381bd0 tx=0x7f60b802f3f0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.986+0000 7f60c66d8640 1 -- 172.21.15.90:0/2291279274 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60c01064a0 msgr2=0x7f60c01a26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:48.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.986+0000 7f60c66d8640 1 --2- 172.21.15.90:0/2291279274 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60c01064a0 0x7f60c01a26a0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f60b000b470 tx=0x7f60b000b930 comp rx=0 tx=0).stop 2024-04-25T11:36:48.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.986+0000 7f60c66d8640 1 -- 172.21.15.90:0/2291279274 shutdown_connections 2024-04-25T11:36:48.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.986+0000 7f60c66d8640 1 --2- 172.21.15.90:0/2291279274 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f60a80afce0 0x7f60a80b21f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.986+0000 7f60c66d8640 1 --2- 172.21.15.90:0/2291279274 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f60c010c1f0 0x7f60c019c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.986+0000 7f60c66d8640 1 --2- 172.21.15.90:0/2291279274 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60c010bd50 0x7f60c01a2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.986+0000 7f60c66d8640 1 --2- 172.21.15.90:0/2291279274 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60c01064a0 0x7f60c01a26a0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:48.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.986+0000 7f60c66d8640 1 -- 172.21.15.90:0/2291279274 >> 172.21.15.90:0/2291279274 conn(0x7f60c01013f0 msgr2=0x7f60c01087c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:48.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.987+0000 7f60c66d8640 1 -- 172.21.15.90:0/2291279274 shutdown_connections 2024-04-25T11:36:48.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:48.987+0000 7f60c66d8640 1 -- 172.21.15.90:0/2291279274 wait complete. 2024-04-25T11:36:48.994 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T11:36:48.994 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:36:49.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.236+0000 7f71afba8640 1 -- 172.21.15.90:0/1566242239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71a810a0b0 msgr2=0x7f71a810a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:49.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.236+0000 7f71afba8640 1 --2- 172.21.15.90:0/1566242239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71a810a0b0 0x7f71a810a4c0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f71a4009900 tx=0x7f71a402a480 comp rx=0 tx=0).stop 2024-04-25T11:36:49.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.236+0000 7f71afba8640 1 -- 172.21.15.90:0/1566242239 shutdown_connections 2024-04-25T11:36:49.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.236+0000 7f71afba8640 1 --2- 172.21.15.90:0/1566242239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f71a811d120 0x7f71a811f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.236+0000 7f71afba8640 1 --2- 172.21.15.90:0/1566242239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71a811a7c0 0x7f71a811cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.236+0000 7f71afba8640 1 --2- 172.21.15.90:0/1566242239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71a810a0b0 0x7f71a810a4c0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.236+0000 7f71afba8640 1 -- 172.21.15.90:0/1566242239 >> 172.21.15.90:0/1566242239 conn(0x7f71a806e7b0 msgr2=0x7f71a806ec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:49.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.237+0000 7f71afba8640 1 -- 172.21.15.90:0/1566242239 shutdown_connections 2024-04-25T11:36:49.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.237+0000 7f71afba8640 1 -- 172.21.15.90:0/1566242239 wait complete. 2024-04-25T11:36:49.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.238+0000 7f71afba8640 1 Processor -- start 2024-04-25T11:36:49.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.239+0000 7f71afba8640 1 -- start start 2024-04-25T11:36:49.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.239+0000 7f71afba8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71a811a7c0 0x7f71a8113420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.239+0000 7f71afba8640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71a811d120 0x7f71a8113960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.239+0000 7f71afba8640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f71a8113ea0 0x7f71a8117e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.239+0000 7f71afba8640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f71a8122b20 con 0x7f71a811d120 2024-04-25T11:36:49.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.239+0000 7f71afba8640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f71a81229a0 con 0x7f71a8113ea0 2024-04-25T11:36:49.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.239+0000 7f71afba8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f71a8122ca0 con 0x7f71a811a7c0 2024-04-25T11:36:49.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.240+0000 7f71ad143640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71a811d120 0x7f71a8113960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:49.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.240+0000 7f71ad143640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71a811d120 0x7f71a8113960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52404/0 (socket says 172.21.15.90:52404) 2024-04-25T11:36:49.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.240+0000 7f71ad944640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71a811a7c0 0x7f71a8113420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:49.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.240+0000 7f71ad944640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71a811a7c0 0x7f71a8113420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:39992/0 (socket says 172.21.15.90:39992) 2024-04-25T11:36:49.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.240+0000 7f71ad143640 1 -- 172.21.15.90:0/566066618 learned_addr learned my addr 172.21.15.90:0/566066618 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:49.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.240+0000 7f71ae145640 1 --2- 172.21.15.90:0/566066618 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f71a8113ea0 0x7f71a8117e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:49.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.241+0000 7f71ad143640 1 -- 172.21.15.90:0/566066618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71a811a7c0 msgr2=0x7f71a8113420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:49.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.241+0000 7f71ad143640 1 --2- 172.21.15.90:0/566066618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71a811a7c0 0x7f71a8113420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.241+0000 7f71ad143640 1 -- 172.21.15.90:0/566066618 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f71a8113ea0 msgr2=0x7f71a8117e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:49.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.242+0000 7f71ad143640 1 --2- 172.21.15.90:0/566066618 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f71a8113ea0 0x7f71a8117e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.242+0000 7f71ad143640 1 -- 172.21.15.90:0/566066618 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71a8118360 con 0x7f71a811d120 2024-04-25T11:36:49.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.244+0000 7f71ad143640 1 --2- 172.21.15.90:0/566066618 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71a811d120 0x7f71a8113960 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f719800c890 tx=0x7f719800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:49.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.246+0000 7f719effd640 1 -- 172.21.15.90:0/566066618 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7198004220 con 0x7f71a811d120 2024-04-25T11:36:49.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.246+0000 7f719effd640 1 -- 172.21.15.90:0/566066618 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71980043c0 con 0x7f71a811d120 2024-04-25T11:36:49.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.246+0000 7f719effd640 1 -- 172.21.15.90:0/566066618 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7198004a00 con 0x7f71a811d120 2024-04-25T11:36:49.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.246+0000 7f71afba8640 1 -- 172.21.15.90:0/566066618 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71a8118650 con 0x7f71a811d120 2024-04-25T11:36:49.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.247+0000 7f71afba8640 1 -- 172.21.15.90:0/566066618 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71a8118b20 con 0x7f71a811d120 2024-04-25T11:36:49.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.250+0000 7f71afba8640 1 -- 172.21.15.90:0/566066618 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71a810b3d0 con 0x7f71a811d120 2024-04-25T11:36:49.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.253+0000 7f719effd640 1 -- 172.21.15.90:0/566066618 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f71980079b0 con 0x7f71a811d120 2024-04-25T11:36:49.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.254+0000 7f719effd640 1 --2- 172.21.15.90:0/566066618 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f71940afdb0 0x7f71940b22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.255+0000 7f719effd640 1 -- 172.21.15.90:0/566066618 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f71980df110 con 0x7f71a811d120 2024-04-25T11:36:49.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.255+0000 7f71ad944640 1 --2- 172.21.15.90:0/566066618 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f71940afdb0 0x7f71940b22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:49.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.257+0000 7f71ad944640 1 --2- 172.21.15.90:0/566066618 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f71940afdb0 0x7f71940b22c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f71a4006010 tx=0x7f71a4005a20 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:49.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.263+0000 7f719effd640 1 -- 172.21.15.90:0/566066618 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7198014030 con 0x7f71a811d120 2024-04-25T11:36:49.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.669+0000 7f71afba8640 1 -- 172.21.15.90:0/566066618 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f71a81bf010 con 0x7f71a811d120 2024-04-25T11:36:49.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.671+0000 7f719effd640 1 -- 172.21.15.90:0/566066618 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11930 (secure 0 0 0) 0x7f71980a7a50 con 0x7f71a811d120 2024-04-25T11:36:49.659 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:36:49.659 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:35:06.229136+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24431,24458,24499,24524,24475]},"id":1}]} 2024-04-25T11:36:49.660 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 26 2024-04-25T11:36:49.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.675+0000 7f71afba8640 1 -- 172.21.15.90:0/566066618 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f71940afdb0 msgr2=0x7f71940b22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:49.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.675+0000 7f71afba8640 1 --2- 172.21.15.90:0/566066618 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f71940afdb0 0x7f71940b22c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f71a4006010 tx=0x7f71a4005a20 comp rx=0 tx=0).stop 2024-04-25T11:36:49.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.675+0000 7f71afba8640 1 -- 172.21.15.90:0/566066618 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71a811d120 msgr2=0x7f71a8113960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:49.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.675+0000 7f71afba8640 1 --2- 172.21.15.90:0/566066618 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71a811d120 0x7f71a8113960 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f719800c890 tx=0x7f719800cd50 comp rx=0 tx=0).stop 2024-04-25T11:36:49.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.675+0000 7f71afba8640 1 -- 172.21.15.90:0/566066618 shutdown_connections 2024-04-25T11:36:49.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.675+0000 7f71afba8640 1 --2- 172.21.15.90:0/566066618 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f71940afdb0 0x7f71940b22c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.675+0000 7f71afba8640 1 --2- 172.21.15.90:0/566066618 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f71a8113ea0 0x7f71a8117e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.675+0000 7f71afba8640 1 --2- 172.21.15.90:0/566066618 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71a811d120 0x7f71a8113960 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.675+0000 7f71afba8640 1 --2- 172.21.15.90:0/566066618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71a811a7c0 0x7f71a8113420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.675+0000 7f71afba8640 1 -- 172.21.15.90:0/566066618 >> 172.21.15.90:0/566066618 conn(0x7f71a806e7b0 msgr2=0x7f71a811abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:49.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.675+0000 7f71afba8640 1 -- 172.21.15.90:0/566066618 shutdown_connections 2024-04-25T11:36:49.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.675+0000 7f71afba8640 1 -- 172.21.15.90:0/566066618 wait complete. 2024-04-25T11:36:49.682 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T11:36:49.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:49 smithi138 ceph-mon[32328]: pgmap v55: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 853 B/s wr, 9 op/s 2024-04-25T11:36:49.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:49 smithi156 ceph-mon[33045]: pgmap v55: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 853 B/s wr, 9 op/s 2024-04-25T11:36:49.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.915+0000 7f2e7b5f3640 1 -- 172.21.15.90:0/548308496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e741031c0 msgr2=0x7f2e7410f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:49.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.915+0000 7f2e7b5f3640 1 --2- 172.21.15.90:0/548308496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e741031c0 0x7f2e7410f4a0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c009ca0 tx=0x7f2e6c02c7c0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.915+0000 7f2e7b5f3640 1 -- 172.21.15.90:0/548308496 shutdown_connections 2024-04-25T11:36:49.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.915+0000 7f2e7b5f3640 1 --2- 172.21.15.90:0/548308496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e741031c0 0x7f2e7410f4a0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.915+0000 7f2e7b5f3640 1 --2- 172.21.15.90:0/548308496 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2e74108eb0 0x7f2e74102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.915+0000 7f2e7b5f3640 1 --2- 172.21.15.90:0/548308496 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e74108a10 0x7f2e74102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.915+0000 7f2e7b5f3640 1 -- 172.21.15.90:0/548308496 >> 172.21.15.90:0/548308496 conn(0x7f2e74078110 msgr2=0x7f2e74078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:49.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.915+0000 7f2e7b5f3640 1 -- 172.21.15.90:0/548308496 shutdown_connections 2024-04-25T11:36:49.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.916+0000 7f2e7b5f3640 1 -- 172.21.15.90:0/548308496 wait complete. 2024-04-25T11:36:49.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.917+0000 7f2e7b5f3640 1 Processor -- start 2024-04-25T11:36:49.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.917+0000 7f2e7b5f3640 1 -- start start 2024-04-25T11:36:49.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.918+0000 7f2e7b5f3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e741031c0 0x7f2e74198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.918+0000 7f2e7b5f3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e74108a10 0x7f2e74198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.918+0000 7f2e7b5f3640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2e74108eb0 0x7f2e7419cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.918+0000 7f2e7b5f3640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2e74111b90 con 0x7f2e741031c0 2024-04-25T11:36:49.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.918+0000 7f2e7b5f3640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2e74111a10 con 0x7f2e74108eb0 2024-04-25T11:36:49.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.918+0000 7f2e7b5f3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2e74111d10 con 0x7f2e74108a10 2024-04-25T11:36:49.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.918+0000 7f2e7938f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e741031c0 0x7f2e74198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:49.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.918+0000 7f2e7938f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e741031c0 0x7f2e74198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52426/0 (socket says 172.21.15.90:52426) 2024-04-25T11:36:49.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.918+0000 7f2e7938f640 1 -- 172.21.15.90:0/4207961788 learned_addr learned my addr 172.21.15.90:0/4207961788 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:49.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.918+0000 7f2e79b90640 1 --2- 172.21.15.90:0/4207961788 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2e74108eb0 0x7f2e7419cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:49.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.918+0000 7f2e7938f640 1 -- 172.21.15.90:0/4207961788 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e74108a10 msgr2=0x7f2e74198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:49.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.918+0000 7f2e7938f640 1 --2- 172.21.15.90:0/4207961788 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e74108a10 0x7f2e74198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.918+0000 7f2e7938f640 1 -- 172.21.15.90:0/4207961788 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2e74108eb0 msgr2=0x7f2e7419cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:49.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.918+0000 7f2e7938f640 1 --2- 172.21.15.90:0/4207961788 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2e74108eb0 0x7f2e7419cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.918+0000 7f2e7938f640 1 -- 172.21.15.90:0/4207961788 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e7419d390 con 0x7f2e741031c0 2024-04-25T11:36:49.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.918+0000 7f2e79b90640 1 --2- 172.21.15.90:0/4207961788 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2e74108eb0 0x7f2e7419cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:36:49.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.919+0000 7f2e7938f640 1 --2- 172.21.15.90:0/4207961788 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e741031c0 0x7f2e74198530 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2e70006fd0 tx=0x7f2e7000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:49.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.919+0000 7f2e6a7fc640 1 -- 172.21.15.90:0/4207961788 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e70011830 con 0x7f2e741031c0 2024-04-25T11:36:49.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.919+0000 7f2e6a7fc640 1 -- 172.21.15.90:0/4207961788 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e70016070 con 0x7f2e741031c0 2024-04-25T11:36:49.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.919+0000 7f2e7b5f3640 1 -- 172.21.15.90:0/4207961788 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e7419d680 con 0x7f2e741031c0 2024-04-25T11:36:49.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.919+0000 7f2e6a7fc640 1 -- 172.21.15.90:0/4207961788 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e70010e00 con 0x7f2e741031c0 2024-04-25T11:36:49.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.919+0000 7f2e7b5f3640 1 -- 172.21.15.90:0/4207961788 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e7419db80 con 0x7f2e741031c0 2024-04-25T11:36:49.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.922+0000 7f2e82581740 1 Processor -- start 2024-04-25T11:36:49.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.922+0000 7f2e82581740 1 -- start start 2024-04-25T11:36:49.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.923+0000 7f2e82581740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609eeff390 0x55609f2a0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.923+0000 7f2e82581740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55609eed64d0 0x55609f2a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.923+0000 7f2e82581740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eede6b0 0x55609f2a1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.923+0000 7f2e82581740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55609f205d80 con 0x55609eede6b0 2024-04-25T11:36:49.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.923+0000 7f2e82581740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55609f1bfa20 con 0x55609eed64d0 2024-04-25T11:36:49.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.923+0000 7f2e82581740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55609f201d40 con 0x55609eeff390 2024-04-25T11:36:49.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.923+0000 7f2e6a7fc640 1 -- 172.21.15.90:0/4207961788 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f2e700119d0 con 0x7f2e741031c0 2024-04-25T11:36:49.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.923+0000 7f2e78b8e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eede6b0 0x55609f2a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:49.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.923+0000 7f2e78b8e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eede6b0 0x55609f2a1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52432/0 (socket says 172.21.15.90:52432) 2024-04-25T11:36:49.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.923+0000 7f2e78b8e640 1 -- 172.21.15.90:0/186921443 learned_addr learned my addr 172.21.15.90:0/186921443 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:49.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.924+0000 7f2e78b8e640 1 -- 172.21.15.90:0/186921443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609eeff390 msgr2=0x55609f2a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:49.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.924+0000 7f2e78b8e640 1 --2- 172.21.15.90:0/186921443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609eeff390 0x55609f2a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.924+0000 7f2e78b8e640 1 -- 172.21.15.90:0/186921443 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55609eed64d0 msgr2=0x55609f2a0bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:36:49.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.924+0000 7f2e78b8e640 1 --2- 172.21.15.90:0/186921443 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55609eed64d0 0x55609f2a0bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.924+0000 7f2e78b8e640 1 -- 172.21.15.90:0/186921443 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55609f1cac30 con 0x55609eede6b0 2024-04-25T11:36:49.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.924+0000 7f2e78b8e640 1 --2- 172.21.15.90:0/186921443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eede6b0 0x55609f2a1100 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f2e6400a980 tx=0x7f2e6400ae40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a46852d8e8a1a70 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:49.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.924+0000 7f2e5f7fe640 1 -- 172.21.15.90:0/186921443 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e6400cd80 con 0x55609eede6b0 2024-04-25T11:36:49.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.925+0000 7f2e5f7fe640 1 -- 172.21.15.90:0/186921443 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e640054d0 con 0x55609eede6b0 2024-04-25T11:36:49.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.925+0000 7f2e5f7fe640 1 -- 172.21.15.90:0/186921443 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e6400f6b0 con 0x55609eede6b0 2024-04-25T11:36:49.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.925+0000 7f2e6a7fc640 1 --2- 172.21.15.90:0/4207961788 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f2e600afc60 0x7f2e600b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.925+0000 7f2e6a7fc640 1 -- 172.21.15.90:0/4207961788 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f2e700de6a0 con 0x7f2e741031c0 2024-04-25T11:36:49.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.925+0000 7f2e79b90640 1 --2- 172.21.15.90:0/4207961788 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f2e600afc60 0x7f2e600b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:49.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.926+0000 7f2e79b90640 1 --2- 172.21.15.90:0/4207961788 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f2e600afc60 0x7f2e600b2170 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c0094e0 tx=0x7f2e6c032000 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:49.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.926+0000 7f2e82581740 1 -- 172.21.15.90:0/186921443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eede6b0 msgr2=0x55609f2a1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:49.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.926+0000 7f2e82581740 1 --2- 172.21.15.90:0/186921443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eede6b0 0x55609f2a1100 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f2e6400a980 tx=0x7f2e6400ae40 comp rx=0 tx=0).stop 2024-04-25T11:36:49.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.926+0000 7f2e82581740 1 -- 172.21.15.90:0/186921443 shutdown_connections 2024-04-25T11:36:49.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.926+0000 7f2e82581740 1 --2- 172.21.15.90:0/186921443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eede6b0 0x55609f2a1100 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.926+0000 7f2e82581740 1 --2- 172.21.15.90:0/186921443 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55609eed64d0 0x55609f2a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.926+0000 7f2e82581740 1 --2- 172.21.15.90:0/186921443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609eeff390 0x55609f2a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.926+0000 7f2e82581740 1 -- 172.21.15.90:0/186921443 >> 172.21.15.90:0/186921443 conn(0x55609eef34e0 msgr2=0x55609f29cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:49.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.927+0000 7f2e82581740 1 -- 172.21.15.90:0/186921443 shutdown_connections 2024-04-25T11:36:49.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.927+0000 7f2e82581740 1 -- 172.21.15.90:0/186921443 wait complete. 2024-04-25T11:36:49.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.927+0000 7f2e82581740 1 -- start start 2024-04-25T11:36:49.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.928+0000 7f2e82581740 1 Processor -- start 2024-04-25T11:36:49.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.929+0000 7f2e82581740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eed64d0 0x55609f2b38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.929+0000 7f2e82581740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609eede6b0 0x55609f2b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.929+0000 7f2e7938f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eed64d0 0x55609f2b38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:49.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.929+0000 7f2e82581740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55609eeff390 0x55609f2bfa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.929+0000 7f2e7938f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eed64d0 0x55609f2b38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52434/0 (socket says 172.21.15.90:52434) 2024-04-25T11:36:49.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.929+0000 7f2e7938f640 1 -- 172.21.15.90:0/778951682 learned_addr learned my addr 172.21.15.90:0/778951682 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:49.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.929+0000 7f2e82581740 1 -- 172.21.15.90:0/778951682 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55609f2b6580 con 0x55609eed64d0 2024-04-25T11:36:49.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.929+0000 7f2e82581740 1 -- 172.21.15.90:0/778951682 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55609f18cdc0 con 0x55609eeff390 2024-04-25T11:36:49.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.929+0000 7f2e82581740 1 -- 172.21.15.90:0/778951682 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55609f2b6700 con 0x55609eede6b0 2024-04-25T11:36:49.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.929+0000 7f2e82581740 10 client.0 _ll_register_callbacks cb 0x55609f1a8600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:49.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.930+0000 7f2e7938f640 1 -- 172.21.15.90:0/778951682 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609eede6b0 msgr2=0x55609f2b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:49.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.930+0000 7f2e7938f640 1 --2- 172.21.15.90:0/778951682 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609eede6b0 0x55609f2b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.930+0000 7f2e7938f640 1 -- 172.21.15.90:0/778951682 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55609eeff390 msgr2=0x55609f2bfa30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:36:49.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.930+0000 7f2e7938f640 1 --2- 172.21.15.90:0/778951682 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55609eeff390 0x55609f2bfa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.930+0000 7f2e7938f640 1 -- 172.21.15.90:0/778951682 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55609ef173f0 con 0x55609eed64d0 2024-04-25T11:36:49.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.930+0000 7f2e7938f640 1 --2- 172.21.15.90:0/778951682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eed64d0 0x55609f2b38f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2e700dbf00 tx=0x7f2e700dc2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f14718d220446c54 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:49.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.930+0000 7f2e5cff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:49.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.930+0000 7f2e5cff9640 1 -- 172.21.15.90:0/778951682 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e700a6400 con 0x55609eed64d0 2024-04-25T11:36:49.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.930+0000 7f2e5cff9640 1 -- 172.21.15.90:0/778951682 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f2e700a2b80 con 0x55609eed64d0 2024-04-25T11:36:49.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.931+0000 7f2e82581740 1 -- 172.21.15.90:0/778951682 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e640054d0 con 0x55609eed64d0 2024-04-25T11:36:49.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.931+0000 7f2e5cff9640 1 -- 172.21.15.90:0/778951682 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2e700a6ff0 con 0x55609eed64d0 2024-04-25T11:36:49.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.931+0000 7f2e82581740 10 client.16062 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:49.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.931+0000 7f2e82581740 1 -- 172.21.15.90:0/778951682 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55609efe17d0 con 0x55609eed64d0 2024-04-25T11:36:49.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.932+0000 7f2e5cff9640 1 -- 172.21.15.90:0/778951682 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e640054d0 con 0x55609eed64d0 2024-04-25T11:36:49.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.932+0000 7f2e5cff9640 1 -- 172.21.15.90:0/778951682 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f2e700c7ba0 con 0x55609eed64d0 2024-04-25T11:36:49.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.932+0000 7f2e82581740 10 client.16062 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:49.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.932+0000 7f2e82581740 10 client.16062 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:36:49.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.932+0000 7f2e82581740 20 client.16062 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:49.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.932+0000 7f2e82581740 1 --2- 172.21.15.90:0/778951682 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55609eebdf60 0x55609f2c5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.932+0000 7f2e82581740 4 client.16062 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:36:49.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.932+0000 7f2e82581740 1 -- 172.21.15.90:0/778951682 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55609f213990 con 0x55609eebdf60 2024-04-25T11:36:49.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.933+0000 7f2e78b8e640 1 --2- 172.21.15.90:0/778951682 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55609eebdf60 0x55609f2c5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:49.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.936+0000 7f2e5cff9640 10 client.16062 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:36:49.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.936+0000 7f2e78b8e640 1 --2- 172.21.15.90:0/778951682 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55609eebdf60 0x55609f2c5e40 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=64ae66d8d90c3399 server_cookie=ac0bcf34c6748057 in_seq=0 out_seq=0 2024-04-25T11:36:49.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.938+0000 7f2e5cff9640 1 -- 172.21.15.90:0/778951682 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55609f213990 con 0x55609eebdf60 2024-04-25T11:36:49.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.938+0000 7f2e5cff9640 10 client.16062 handle_command_reply: tid=0 2024-04-25T11:36:49.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.938+0000 7f2e82581740 1 client.16062 shutdown 2024-04-25T11:36:49.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.939+0000 7f2e82581740 1 -- 172.21.15.90:0/778951682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eed64d0 msgr2=0x55609f2b38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:49.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.939+0000 7f2e82581740 1 --2- 172.21.15.90:0/778951682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eed64d0 0x55609f2b38f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2e700dbf00 tx=0x7f2e700dc2c0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.939+0000 7f2e82581740 1 -- 172.21.15.90:0/778951682 shutdown_connections 2024-04-25T11:36:49.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.939+0000 7f2e82581740 1 --2- 172.21.15.90:0/778951682 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55609eebdf60 0x55609f2c5e40 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.939+0000 7f2e82581740 1 --2- 172.21.15.90:0/778951682 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55609eeff390 0x55609f2bfa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.939+0000 7f2e5cff9640 0 client.16062 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:36:49.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.939+0000 7f2e82581740 1 --2- 172.21.15.90:0/778951682 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609eede6b0 0x55609f2b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.939+0000 7f2e82581740 1 --2- 172.21.15.90:0/778951682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eed64d0 0x55609f2b38f0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.939+0000 7f2e82581740 1 -- 172.21.15.90:0/778951682 >> 172.21.15.90:0/778951682 conn(0x55609eef34e0 msgr2=0x55609f29b610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:49.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.939+0000 7f2e82581740 1 -- 172.21.15.90:0/778951682 shutdown_connections 2024-04-25T11:36:49.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.939+0000 7f2e82581740 1 -- 172.21.15.90:0/778951682 wait complete. 2024-04-25T11:36:49.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.939+0000 7f2e82581740 20 client.16062 trim_cache size 0 max 16384 2024-04-25T11:36:49.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.958+0000 7f2e82581740 1 Processor -- start 2024-04-25T11:36:49.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.958+0000 7f2e82581740 1 -- start start 2024-04-25T11:36:49.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.959+0000 7f2e82581740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55609eed64d0 0x55609f2b29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.959+0000 7f2e82581740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eede6b0 0x55609f2b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.959+0000 7f2e82581740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609eeff390 0x55609f2bf610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.959+0000 7f2e82581740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2e741ad9c0 con 0x55609eede6b0 2024-04-25T11:36:49.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.959+0000 7f2e82581740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2e741af500 con 0x55609eed64d0 2024-04-25T11:36:49.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.959+0000 7f2e82581740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2e741ae090 con 0x55609eeff390 2024-04-25T11:36:49.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.959+0000 7f2e78b8e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eede6b0 0x55609f2b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:49.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.959+0000 7f2e78b8e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eede6b0 0x55609f2b1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52440/0 (socket says 172.21.15.90:52440) 2024-04-25T11:36:49.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.959+0000 7f2e78b8e640 1 -- 172.21.15.90:0/205986299 learned_addr learned my addr 172.21.15.90:0/205986299 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:49.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.959+0000 7f2e78b8e640 1 -- 172.21.15.90:0/205986299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609eeff390 msgr2=0x55609f2bf610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:49.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.959+0000 7f2e78b8e640 1 --2- 172.21.15.90:0/205986299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609eeff390 0x55609f2bf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.959+0000 7f2e78b8e640 1 -- 172.21.15.90:0/205986299 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55609eed64d0 msgr2=0x55609f2b29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:49.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.959+0000 7f2e78b8e640 1 --2- 172.21.15.90:0/205986299 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55609eed64d0 0x55609f2b29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.959+0000 7f2e78b8e640 1 -- 172.21.15.90:0/205986299 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55609f2a7820 con 0x55609eede6b0 2024-04-25T11:36:49.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.960+0000 7f2e78b8e640 1 --2- 172.21.15.90:0/205986299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eede6b0 0x55609f2b1260 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x55609f297460 tx=0x7f2e64004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f0463d6365e0e29 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:49.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.960+0000 7f2e5effd640 1 -- 172.21.15.90:0/205986299 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e64029070 con 0x55609eede6b0 2024-04-25T11:36:49.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.960+0000 7f2e5effd640 1 -- 172.21.15.90:0/205986299 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e64025e50 con 0x55609eede6b0 2024-04-25T11:36:49.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.960+0000 7f2e5effd640 1 -- 172.21.15.90:0/205986299 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e64025730 con 0x55609eede6b0 2024-04-25T11:36:49.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.962+0000 7f2e82581740 1 -- 172.21.15.90:0/205986299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eede6b0 msgr2=0x55609f2b1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:49.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.962+0000 7f2e82581740 1 --2- 172.21.15.90:0/205986299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eede6b0 0x55609f2b1260 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x55609f297460 tx=0x7f2e64004d10 comp rx=0 tx=0).stop 2024-04-25T11:36:49.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.963+0000 7f2e82581740 1 -- 172.21.15.90:0/205986299 shutdown_connections 2024-04-25T11:36:49.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.963+0000 7f2e82581740 1 --2- 172.21.15.90:0/205986299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609eeff390 0x55609f2bf610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.963+0000 7f2e82581740 1 --2- 172.21.15.90:0/205986299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eede6b0 0x55609f2b1260 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.963+0000 7f2e82581740 1 --2- 172.21.15.90:0/205986299 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55609eed64d0 0x55609f2b29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.963+0000 7f2e82581740 1 -- 172.21.15.90:0/205986299 >> 172.21.15.90:0/205986299 conn(0x55609eef34e0 msgr2=0x55609f2a05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:49.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.963+0000 7f2e82581740 1 -- 172.21.15.90:0/205986299 shutdown_connections 2024-04-25T11:36:49.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.963+0000 7f2e82581740 1 -- 172.21.15.90:0/205986299 wait complete. 2024-04-25T11:36:49.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.964+0000 7f2e82581740 1 -- start start 2024-04-25T11:36:49.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.965+0000 7f2e82581740 1 Processor -- start 2024-04-25T11:36:49.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.965+0000 7f2e82581740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eed64d0 0x55609f2bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.965+0000 7f2e82581740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55609eede6b0 0x55609f2b90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.966+0000 7f2e82581740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609eeff390 0x55609f2b95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.966+0000 7f2e82581740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2e741aee30 con 0x55609eed64d0 2024-04-25T11:36:49.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.966+0000 7f2e82581740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2e741ae760 con 0x55609eede6b0 2024-04-25T11:36:49.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.966+0000 7f2e82581740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2e741ab9a0 con 0x55609eeff390 2024-04-25T11:36:49.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.966+0000 7f2e82581740 10 client.0 _ll_register_callbacks cb 0x7f2e70010720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:49.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.966+0000 7f2e7938f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eed64d0 0x55609f2bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:49.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.966+0000 7f2e7938f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eed64d0 0x55609f2bbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52454/0 (socket says 172.21.15.90:52454) 2024-04-25T11:36:49.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.966+0000 7f2e7938f640 1 -- 172.21.15.90:0/2719538189 learned_addr learned my addr 172.21.15.90:0/2719538189 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:49.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.966+0000 7f2e78b8e640 1 --2- 172.21.15.90:0/2719538189 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55609eede6b0 0x55609f2b90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:49.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.967+0000 7f2e7938f640 1 -- 172.21.15.90:0/2719538189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609eeff390 msgr2=0x55609f2b95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:49.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.967+0000 7f2e7938f640 1 --2- 172.21.15.90:0/2719538189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609eeff390 0x55609f2b95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.967+0000 7f2e7938f640 1 -- 172.21.15.90:0/2719538189 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55609eede6b0 msgr2=0x55609f2b90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:49.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.967+0000 7f2e7938f640 1 --2- 172.21.15.90:0/2719538189 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55609eede6b0 0x55609f2b90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.967+0000 7f2e7938f640 1 -- 172.21.15.90:0/2719538189 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55609f2b9b20 con 0x55609eed64d0 2024-04-25T11:36:49.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.967+0000 7f2e7938f640 1 --2- 172.21.15.90:0/2719538189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eed64d0 0x55609f2bbce0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f2e700a2970 tx=0x7f2e70010380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66946483965609fa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:49.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.967+0000 7f2e5cff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:49.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.967+0000 7f2e5cff9640 1 -- 172.21.15.90:0/2719538189 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e700a6060 con 0x55609eed64d0 2024-04-25T11:36:49.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.967+0000 7f2e5cff9640 1 -- 172.21.15.90:0/2719538189 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f2e700a2b80 con 0x55609eed64d0 2024-04-25T11:36:49.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.967+0000 7f2e82581740 1 -- 172.21.15.90:0/2719538189 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e64025e50 con 0x55609eed64d0 2024-04-25T11:36:49.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.968+0000 7f2e5cff9640 1 -- 172.21.15.90:0/2719538189 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2e700a6400 con 0x55609eed64d0 2024-04-25T11:36:49.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.968+0000 7f2e82581740 10 client.16068 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:49.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.968+0000 7f2e82581740 1 -- 172.21.15.90:0/2719538189 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55609f32de70 con 0x55609eed64d0 2024-04-25T11:36:49.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.968+0000 7f2e5cff9640 1 -- 172.21.15.90:0/2719538189 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e700a68f0 con 0x55609eed64d0 2024-04-25T11:36:49.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.968+0000 7f2e5cff9640 1 -- 172.21.15.90:0/2719538189 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f2e7006d080 con 0x55609eed64d0 2024-04-25T11:36:49.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.969+0000 7f2e82581740 10 client.16068 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:49.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.969+0000 7f2e82581740 10 client.16068 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:36:49.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.969+0000 7f2e82581740 20 client.16068 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:49.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.969+0000 7f2e82581740 1 --2- 172.21.15.90:0/2719538189 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55609ee933a0 0x55609f330550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.969+0000 7f2e82581740 4 client.16068 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:36:49.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.969+0000 7f2e82581740 1 -- 172.21.15.90:0/2719538189 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f2e64025e50 con 0x55609ee933a0 2024-04-25T11:36:49.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.970+0000 7f2e78b8e640 1 --2- 172.21.15.90:0/2719538189 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55609ee933a0 0x55609f330550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:49.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.975+0000 7f2e78b8e640 1 --2- 172.21.15.90:0/2719538189 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55609ee933a0 0x55609f330550 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=96cc95abc7bcadf2 server_cookie=d69a376df2f4bc6a in_seq=0 out_seq=0 2024-04-25T11:36:49.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.975+0000 7f2e5cff9640 10 client.16068 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:36:49.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.977+0000 7f2e5cff9640 1 -- 172.21.15.90:0/2719538189 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f2e64025e50 con 0x55609ee933a0 2024-04-25T11:36:49.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.977+0000 7f2e5cff9640 10 client.16068 handle_command_reply: tid=0 2024-04-25T11:36:49.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.977+0000 7f2e82581740 1 client.16068 shutdown 2024-04-25T11:36:49.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.978+0000 7f2e82581740 1 -- 172.21.15.90:0/2719538189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eed64d0 msgr2=0x55609f2bbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:49.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.978+0000 7f2e82581740 1 --2- 172.21.15.90:0/2719538189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eed64d0 0x55609f2bbce0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f2e700a2970 tx=0x7f2e70010380 comp rx=0 tx=0).stop 2024-04-25T11:36:49.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.978+0000 7f2e82581740 1 -- 172.21.15.90:0/2719538189 shutdown_connections 2024-04-25T11:36:49.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.978+0000 7f2e82581740 1 --2- 172.21.15.90:0/2719538189 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55609ee933a0 0x55609f330550 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.978+0000 7f2e82581740 1 --2- 172.21.15.90:0/2719538189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55609eeff390 0x55609f2b95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.978+0000 7f2e82581740 1 --2- 172.21.15.90:0/2719538189 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55609eede6b0 0x55609f2b90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.978+0000 7f2e82581740 1 --2- 172.21.15.90:0/2719538189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55609eed64d0 0x55609f2bbce0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.978+0000 7f2e5cff9640 0 client.16068 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:36:49.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.978+0000 7f2e82581740 1 -- 172.21.15.90:0/2719538189 >> 172.21.15.90:0/2719538189 conn(0x55609eef34e0 msgr2=0x55609f2a0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:49.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.978+0000 7f2e82581740 1 -- 172.21.15.90:0/2719538189 shutdown_connections 2024-04-25T11:36:49.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.978+0000 7f2e82581740 1 -- 172.21.15.90:0/2719538189 wait complete. 2024-04-25T11:36:49.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.979+0000 7f2e82581740 20 client.16068 trim_cache size 0 max 16384 2024-04-25T11:36:49.967 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:36:49.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.980+0000 7f2e7b5f3640 1 -- 172.21.15.90:0/4207961788 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f2e600afc60 msgr2=0x7f2e600b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:49.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.980+0000 7f2e7b5f3640 1 --2- 172.21.15.90:0/4207961788 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f2e600afc60 0x7f2e600b2170 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c0094e0 tx=0x7f2e6c032000 comp rx=0 tx=0).stop 2024-04-25T11:36:49.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.980+0000 7f2e7b5f3640 1 -- 172.21.15.90:0/4207961788 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e741031c0 msgr2=0x7f2e74198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:49.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.980+0000 7f2e7b5f3640 1 --2- 172.21.15.90:0/4207961788 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e741031c0 0x7f2e74198530 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2e70006fd0 tx=0x7f2e7000b8e0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.980+0000 7f2e7b5f3640 1 -- 172.21.15.90:0/4207961788 shutdown_connections 2024-04-25T11:36:49.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.980+0000 7f2e7b5f3640 1 --2- 172.21.15.90:0/4207961788 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f2e600afc60 0x7f2e600b2170 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.980+0000 7f2e7b5f3640 1 --2- 172.21.15.90:0/4207961788 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2e74108eb0 0x7f2e7419cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.980+0000 7f2e7b5f3640 1 --2- 172.21.15.90:0/4207961788 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e74108a10 0x7f2e74198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.980+0000 7f2e7b5f3640 1 --2- 172.21.15.90:0/4207961788 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e741031c0 0x7f2e74198530 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.980+0000 7f2e7b5f3640 1 -- 172.21.15.90:0/4207961788 >> 172.21.15.90:0/4207961788 conn(0x7f2e74078110 msgr2=0x7f2e74100ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:49.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.980+0000 7f2e7b5f3640 1 -- 172.21.15.90:0/4207961788 shutdown_connections 2024-04-25T11:36:49.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:49.980+0000 7f2e7b5f3640 1 -- 172.21.15.90:0/4207961788 wait complete. 2024-04-25T11:36:49.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:49 smithi090 ceph-mon[29575]: pgmap v55: 97 pgs: 97 active+clean; 1.3 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 853 B/s wr, 9 op/s 2024-04-25T11:36:50.009 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T11:36:50.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.241+0000 7fdd05230640 1 -- 172.21.15.90:0/781687547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd00114b40 msgr2=0x7fdd00116f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.241+0000 7fdd05230640 1 --2- 172.21.15.90:0/781687547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd00114b40 0x7fdd00116f60 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8009900 tx=0x7fdcf802a4a0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.241+0000 7fdd05230640 1 -- 172.21.15.90:0/781687547 shutdown_connections 2024-04-25T11:36:50.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.241+0000 7fdd05230640 1 --2- 172.21.15.90:0/781687547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd00114b40 0x7fdd00116f60 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.241+0000 7fdd05230640 1 --2- 172.21.15.90:0/781687547 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd001121e0 0x7fdd00114600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.241+0000 7fdd05230640 1 --2- 172.21.15.90:0/781687547 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd00075360 0x7fdd00075770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.241+0000 7fdd05230640 1 -- 172.21.15.90:0/781687547 >> 172.21.15.90:0/781687547 conn(0x7fdd00079280 msgr2=0x7fdd000796c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:50.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.241+0000 7fdd05230640 1 -- 172.21.15.90:0/781687547 shutdown_connections 2024-04-25T11:36:50.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.242+0000 7fdd05230640 1 -- 172.21.15.90:0/781687547 wait complete. 2024-04-25T11:36:50.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.243+0000 7fdd05230640 1 Processor -- start 2024-04-25T11:36:50.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.243+0000 7fdd05230640 1 -- start start 2024-04-25T11:36:50.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.244+0000 7fdd05230640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd00075360 0x7fdd001a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.244+0000 7fdd05230640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd001121e0 0x7fdd001a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.244+0000 7fdd05230640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd00114b40 0x7fdd001a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.244+0000 7fdd05230640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0011a360 con 0x7fdd001121e0 2024-04-25T11:36:50.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.244+0000 7fdd05230640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0011a1e0 con 0x7fdd00114b40 2024-04-25T11:36:50.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.244+0000 7fdd05230640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0011a4e0 con 0x7fdd00075360 2024-04-25T11:36:50.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.245+0000 7fdcfed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd00075360 0x7fdd001a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.245+0000 7fdcf7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd001121e0 0x7fdd001a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.245+0000 7fdcf7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd001121e0 0x7fdd001a1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52472/0 (socket says 172.21.15.90:52472) 2024-04-25T11:36:50.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.245+0000 7fdcf7fff640 1 -- 172.21.15.90:0/3169565323 learned_addr learned my addr 172.21.15.90:0/3169565323 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:50.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.246+0000 7fdcff59e640 1 --2- 172.21.15.90:0/3169565323 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd00114b40 0x7fdd001a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.246+0000 7fdcf7fff640 1 -- 172.21.15.90:0/3169565323 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd00075360 msgr2=0x7fdd001a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.246+0000 7fdcf7fff640 1 --2- 172.21.15.90:0/3169565323 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd00075360 0x7fdd001a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.246+0000 7fdcf7fff640 1 -- 172.21.15.90:0/3169565323 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd00114b40 msgr2=0x7fdd001a5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.246+0000 7fdcf7fff640 1 --2- 172.21.15.90:0/3169565323 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd00114b40 0x7fdd001a5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.246+0000 7fdcf7fff640 1 -- 172.21.15.90:0/3169565323 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd001a5b80 con 0x7fdd001121e0 2024-04-25T11:36:50.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.246+0000 7fdcfed9d640 1 --2- 172.21.15.90:0/3169565323 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd00075360 0x7fdd001a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:36:50.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.247+0000 7fdcf7fff640 1 --2- 172.21.15.90:0/3169565323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd001121e0 0x7fdd001a1280 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fdce800b450 tx=0x7fdce800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:50.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.247+0000 7fdcfcd99640 1 -- 172.21.15.90:0/3169565323 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce8015020 con 0x7fdd001121e0 2024-04-25T11:36:50.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.248+0000 7fdd05230640 1 -- 172.21.15.90:0/3169565323 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd001a5e70 con 0x7fdd001121e0 2024-04-25T11:36:50.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.248+0000 7fdcfcd99640 1 -- 172.21.15.90:0/3169565323 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdce8011d40 con 0x7fdd001121e0 2024-04-25T11:36:50.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.248+0000 7fdd05230640 1 -- 172.21.15.90:0/3169565323 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd001ad7f0 con 0x7fdd001121e0 2024-04-25T11:36:50.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.248+0000 7fdcfcd99640 1 -- 172.21.15.90:0/3169565323 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce8015020 con 0x7fdd001121e0 2024-04-25T11:36:50.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.252+0000 7fdd0c1b6740 1 Processor -- start 2024-04-25T11:36:50.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.252+0000 7fdd0c1b6740 1 -- start start 2024-04-25T11:36:50.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.252+0000 7fdd0c1b6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595b88a3390 0x5595b8c446a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.252+0000 7fdd0c1b6740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595b887a4d0 0x5595b8c44be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.252+0000 7fdd0c1b6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595b88826b0 0x5595b8c626b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.252+0000 7fdd0c1b6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5595b8ba9d80 con 0x5595b88a3390 2024-04-25T11:36:50.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.252+0000 7fdd0c1b6740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5595b8b63a20 con 0x5595b887a4d0 2024-04-25T11:36:50.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.252+0000 7fdd0c1b6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5595b8ba5d40 con 0x5595b88826b0 2024-04-25T11:36:50.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.253+0000 7fdcff59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595b88a3390 0x5595b8c446a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.253+0000 7fdcff59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595b88a3390 0x5595b8c446a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52480/0 (socket says 172.21.15.90:52480) 2024-04-25T11:36:50.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.253+0000 7fdcff59e640 1 -- 172.21.15.90:0/3523819463 learned_addr learned my addr 172.21.15.90:0/3523819463 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:50.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.253+0000 7fdcfed9d640 1 --2- 172.21.15.90:0/3523819463 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595b887a4d0 0x5595b8c44be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.254+0000 7fdcff59e640 1 -- 172.21.15.90:0/3523819463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595b88826b0 msgr2=0x5595b8c626b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:36:50.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.254+0000 7fdcff59e640 1 --2- 172.21.15.90:0/3523819463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595b88826b0 0x5595b8c626b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.254+0000 7fdcff59e640 1 -- 172.21.15.90:0/3523819463 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595b887a4d0 msgr2=0x5595b8c44be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:50.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.254+0000 7fdcff59e640 1 --2- 172.21.15.90:0/3523819463 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595b887a4d0 0x5595b8c44be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.254+0000 7fdcff59e640 1 -- 172.21.15.90:0/3523819463 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595b8b6ec30 con 0x5595b88a3390 2024-04-25T11:36:50.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.254+0000 7fdcff59e640 1 --2- 172.21.15.90:0/3523819463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595b88a3390 0x5595b8c446a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8009900 tx=0x7fdcf8032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b709699abd116 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:50.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.254+0000 7fdcfcd99640 1 -- 172.21.15.90:0/3169565323 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fdce8018020 con 0x7fdd001121e0 2024-04-25T11:36:50.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.255+0000 7fdcf57fa640 1 -- 172.21.15.90:0/3523819463 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8061070 con 0x5595b88a3390 2024-04-25T11:36:50.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.255+0000 7fdcf57fa640 1 -- 172.21.15.90:0/3523819463 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdcf80316a0 con 0x5595b88a3390 2024-04-25T11:36:50.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.255+0000 7fdcf57fa640 1 -- 172.21.15.90:0/3523819463 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8035070 con 0x5595b88a3390 2024-04-25T11:36:50.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.256+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/3523819463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595b88a3390 msgr2=0x5595b8c446a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.256+0000 7fdd0c1b6740 1 --2- 172.21.15.90:0/3523819463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595b88a3390 0x5595b8c446a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8009900 tx=0x7fdcf8032ce0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.257+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/3523819463 shutdown_connections 2024-04-25T11:36:50.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.257+0000 7fdd0c1b6740 1 --2- 172.21.15.90:0/3523819463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595b88826b0 0x5595b8c626b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.257+0000 7fdd0c1b6740 1 --2- 172.21.15.90:0/3523819463 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595b887a4d0 0x5595b8c44be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.257+0000 7fdd0c1b6740 1 --2- 172.21.15.90:0/3523819463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595b88a3390 0x5595b8c446a0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.257+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/3523819463 >> 172.21.15.90:0/3523819463 conn(0x5595b88974e0 msgr2=0x5595b8c40fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:50.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.257+0000 7fdcfcd99640 1 --2- 172.21.15.90:0/3169565323 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fdcec0afee0 0x7fdcec0b23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.258+0000 7fdcfcd99640 1 -- 172.21.15.90:0/3169565323 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fdce80de6d0 con 0x7fdd001121e0 2024-04-25T11:36:50.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.258+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/3523819463 shutdown_connections 2024-04-25T11:36:50.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.258+0000 7fdcfed9d640 1 --2- 172.21.15.90:0/3169565323 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fdcec0afee0 0x7fdcec0b23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.258+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/3523819463 wait complete. 2024-04-25T11:36:50.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.258+0000 7fdd0c1b6740 1 -- start start 2024-04-25T11:36:50.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.259+0000 7fdd0c1b6740 1 Processor -- start 2024-04-25T11:36:50.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.260+0000 7fdcfed9d640 1 --2- 172.21.15.90:0/3169565323 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fdcec0afee0 0x7fdcec0b23f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x5595b8c42070 tx=0x7fdcf00098f0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:50.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.260+0000 7fdd0c1b6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595b887a4d0 0x5595b8c55310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.260+0000 7fdd0c1b6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595b88826b0 0x5595b8c55850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.260+0000 7fdd0c1b6740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595b88a3390 0x5595b8c4f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.260+0000 7fdd0c1b6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5595b8c47780 con 0x5595b88826b0 2024-04-25T11:36:50.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.260+0000 7fdd0c1b6740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5595b8b30dc0 con 0x5595b88a3390 2024-04-25T11:36:50.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.260+0000 7fdd0c1b6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5595b8c47900 con 0x5595b887a4d0 2024-04-25T11:36:50.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.260+0000 7fdd0c1b6740 10 client.0 _ll_register_callbacks cb 0x5595b8b4c600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:50.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.261+0000 7fdcff59e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595b88a3390 0x5595b8c4f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.261+0000 7fdcff59e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595b88a3390 0x5595b8c4f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:53476/0 (socket says 172.21.15.90:53476) 2024-04-25T11:36:50.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.261+0000 7fdcff59e640 1 -- 172.21.15.90:0/924337104 learned_addr learned my addr 172.21.15.90:0/924337104 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:50.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.261+0000 7fdcf7fff640 1 --2- 172.21.15.90:0/924337104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595b88826b0 0x5595b8c55850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.261+0000 7fdcff59e640 1 -- 172.21.15.90:0/924337104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595b887a4d0 msgr2=0x5595b8c55310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:50.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.261+0000 7fdcff59e640 1 --2- 172.21.15.90:0/924337104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595b887a4d0 0x5595b8c55310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.261+0000 7fdcff59e640 1 -- 172.21.15.90:0/924337104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595b88826b0 msgr2=0x5595b8c55850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:50.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.261+0000 7fdcff59e640 1 --2- 172.21.15.90:0/924337104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595b88826b0 0x5595b8c55850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.261+0000 7fdcff59e640 1 -- 172.21.15.90:0/924337104 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595b88bb3f0 con 0x5595b88a3390 2024-04-25T11:36:50.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.262+0000 7fdcff59e640 1 --2- 172.21.15.90:0/924337104 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595b88a3390 0x5595b8c4f700 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdcf802a4e0 tx=0x7fdcf8031570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f97212527a5d555a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:50.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.262+0000 7fdce67fc640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:36:50.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.262+0000 7fdce67fc640 1 -- 172.21.15.90:0/924337104 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8061040 con 0x5595b88a3390 2024-04-25T11:36:50.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.262+0000 7fdce67fc640 1 -- 172.21.15.90:0/924337104 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fdcf8004240 con 0x5595b88a3390 2024-04-25T11:36:50.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.263+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/924337104 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcf80316a0 con 0x5595b88a3390 2024-04-25T11:36:50.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.268+0000 7fdce67fc640 1 -- 172.21.15.90:0/924337104 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdcf8035040 con 0x5595b88a3390 2024-04-25T11:36:50.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.268+0000 7fdd0c1b6740 10 client.25553 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:50.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.268+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/924337104 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5595b89857d0 con 0x5595b88a3390 2024-04-25T11:36:50.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.270+0000 7fdce67fc640 1 -- 172.21.15.90:0/924337104 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdcf80316a0 con 0x5595b88a3390 2024-04-25T11:36:50.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.270+0000 7fdce67fc640 1 -- 172.21.15.90:0/924337104 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fdcf80046c0 con 0x5595b88a3390 2024-04-25T11:36:50.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.270+0000 7fdd0c1b6740 10 client.25553 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:50.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.270+0000 7fdd0c1b6740 10 client.25553 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:36:50.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.270+0000 7fdd0c1b6740 20 client.25553 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:50.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.270+0000 7fdd0c1b6740 1 --2- 172.21.15.90:0/924337104 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5595b8861f60 0x5595b8c617b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.270+0000 7fdd0c1b6740 4 client.25553 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:36:50.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.270+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/924337104 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5595b8bb7990 con 0x5595b8861f60 2024-04-25T11:36:50.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.271+0000 7fdcfed9d640 1 --2- 172.21.15.90:0/924337104 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5595b8861f60 0x5595b8c617b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.272+0000 7fdcfed9d640 1 --2- 172.21.15.90:0/924337104 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5595b8861f60 0x5595b8c617b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=12a48652144b1e7a server_cookie=a350df2e979803a7 in_seq=0 out_seq=0 2024-04-25T11:36:50.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.272+0000 7fdce67fc640 10 client.25553 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:36:50.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.273+0000 7fdce67fc640 1 -- 172.21.15.90:0/924337104 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5595b8bb7990 con 0x5595b8861f60 2024-04-25T11:36:50.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.274+0000 7fdce67fc640 10 client.25553 handle_command_reply: tid=0 2024-04-25T11:36:50.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.274+0000 7fdd0c1b6740 1 client.25553 shutdown 2024-04-25T11:36:50.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.275+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/924337104 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595b88a3390 msgr2=0x5595b8c4f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.275+0000 7fdd0c1b6740 1 --2- 172.21.15.90:0/924337104 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595b88a3390 0x5595b8c4f700 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdcf802a4e0 tx=0x7fdcf8031570 comp rx=0 tx=0).stop 2024-04-25T11:36:50.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.275+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/924337104 shutdown_connections 2024-04-25T11:36:50.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.275+0000 7fdd0c1b6740 1 --2- 172.21.15.90:0/924337104 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5595b8861f60 0x5595b8c617b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.275+0000 7fdd0c1b6740 1 --2- 172.21.15.90:0/924337104 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595b88a3390 0x5595b8c4f700 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.275+0000 7fdd0c1b6740 1 --2- 172.21.15.90:0/924337104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595b88826b0 0x5595b8c55850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.275+0000 7fdd0c1b6740 1 --2- 172.21.15.90:0/924337104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595b887a4d0 0x5595b8c55310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.275+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/924337104 >> 172.21.15.90:0/924337104 conn(0x5595b88974e0 msgr2=0x5595b8c46480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:50.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.275+0000 7fdce67fc640 0 client.25553 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:36:50.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.276+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/924337104 shutdown_connections 2024-04-25T11:36:50.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.276+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/924337104 wait complete. 2024-04-25T11:36:50.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.276+0000 7fdd0c1b6740 20 client.25553 trim_cache size 0 max 16384 2024-04-25T11:36:50.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.338+0000 7fdd0c1b6740 1 Processor -- start 2024-04-25T11:36:50.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.338+0000 7fdd0c1b6740 1 -- start start 2024-04-25T11:36:50.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.340+0000 7fdd0c1b6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595b887a4d0 0x5595b8c4f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.340+0000 7fdd0c1b6740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595b88826b0 0x5595b8c66f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.341+0000 7fdd0c1b6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595b88a3390 0x5595b8c674c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.341+0000 7fdcff59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595b88a3390 0x5595b8c674c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.342+0000 7fdd0c1b6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0006c5c0 con 0x5595b88a3390 2024-04-25T11:36:50.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.342+0000 7fdcff59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595b88a3390 0x5595b8c674c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52510/0 (socket says 172.21.15.90:52510) 2024-04-25T11:36:50.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.342+0000 7fdcff59e640 1 -- 172.21.15.90:0/3712335044 learned_addr learned my addr 172.21.15.90:0/3712335044 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:50.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.342+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/3712335044 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0006e100 con 0x5595b88826b0 2024-04-25T11:36:50.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.342+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/3712335044 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0006cc90 con 0x5595b887a4d0 2024-04-25T11:36:50.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.343+0000 7fdcfed9d640 1 --2- 172.21.15.90:0/3712335044 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595b887a4d0 0x5595b8c4f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.343+0000 7fdcff59e640 1 -- 172.21.15.90:0/3712335044 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595b887a4d0 msgr2=0x5595b8c4f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:50.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.343+0000 7fdcff59e640 1 --2- 172.21.15.90:0/3712335044 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595b887a4d0 0x5595b8c4f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.343+0000 7fdcff59e640 1 -- 172.21.15.90:0/3712335044 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595b88826b0 msgr2=0x5595b8c66f80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:36:50.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.343+0000 7fdcff59e640 1 --2- 172.21.15.90:0/3712335044 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595b88826b0 0x5595b8c66f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.343+0000 7fdcff59e640 1 -- 172.21.15.90:0/3712335044 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595b8c40620 con 0x5595b88a3390 2024-04-25T11:36:50.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.344+0000 7fdcff59e640 1 --2- 172.21.15.90:0/3712335044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595b88a3390 0x5595b8c674c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8033000 tx=0x7fdcf8032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=896b947446129e08 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:50.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.345+0000 7fdcf4ff9640 1 -- 172.21.15.90:0/3712335044 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf807c070 con 0x5595b88a3390 2024-04-25T11:36:50.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.346+0000 7fdcf4ff9640 1 -- 172.21.15.90:0/3712335044 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdcf80046c0 con 0x5595b88a3390 2024-04-25T11:36:50.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.346+0000 7fdcf4ff9640 1 -- 172.21.15.90:0/3712335044 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf80043a0 con 0x5595b88a3390 2024-04-25T11:36:50.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.347+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/3712335044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595b88a3390 msgr2=0x5595b8c674c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.347+0000 7fdd0c1b6740 1 --2- 172.21.15.90:0/3712335044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595b88a3390 0x5595b8c674c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8033000 tx=0x7fdcf8032ce0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.347+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/3712335044 shutdown_connections 2024-04-25T11:36:50.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.347+0000 7fdd0c1b6740 1 --2- 172.21.15.90:0/3712335044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595b88a3390 0x5595b8c674c0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.348+0000 7fdd0c1b6740 1 --2- 172.21.15.90:0/3712335044 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595b88826b0 0x5595b8c66f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.348+0000 7fdd0c1b6740 1 --2- 172.21.15.90:0/3712335044 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595b887a4d0 0x5595b8c4f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.348+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/3712335044 >> 172.21.15.90:0/3712335044 conn(0x5595b88974e0 msgr2=0x5595b8c4ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:50.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.348+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/3712335044 shutdown_connections 2024-04-25T11:36:50.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.348+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/3712335044 wait complete. 2024-04-25T11:36:50.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.348+0000 7fdd0c1b6740 1 -- start start 2024-04-25T11:36:50.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.348+0000 7fdd0c1b6740 1 Processor -- start 2024-04-25T11:36:50.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.349+0000 7fdd0c1b6740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595b887a4d0 0x5595b8c582d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.349+0000 7fdd0c1b6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595b88826b0 0x5595b8c58810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.349+0000 7fdd0c1b6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595b88a3390 0x5595b8c58d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.349+0000 7fdd0c1b6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0006da30 con 0x5595b88a3390 2024-04-25T11:36:50.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.349+0000 7fdd0c1b6740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0006d360 con 0x5595b887a4d0 2024-04-25T11:36:50.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.349+0000 7fdd0c1b6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0006afa0 con 0x5595b88826b0 2024-04-25T11:36:50.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.349+0000 7fdd0c1b6740 10 client.0 _ll_register_callbacks cb 0x7fdcf8032bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:50.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.349+0000 7fdcff59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595b88a3390 0x5595b8c58d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.349+0000 7fdcff59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595b88a3390 0x5595b8c58d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52518/0 (socket says 172.21.15.90:52518) 2024-04-25T11:36:50.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.349+0000 7fdcff59e640 1 -- 172.21.15.90:0/4284469468 learned_addr learned my addr 172.21.15.90:0/4284469468 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:50.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.350+0000 7fdcff59e640 1 -- 172.21.15.90:0/4284469468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595b88826b0 msgr2=0x5595b8c58810 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:36:50.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.350+0000 7fdcff59e640 1 --2- 172.21.15.90:0/4284469468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595b88826b0 0x5595b8c58810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.350+0000 7fdcff59e640 1 -- 172.21.15.90:0/4284469468 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595b887a4d0 msgr2=0x5595b8c582d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:36:50.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.350+0000 7fdcff59e640 1 --2- 172.21.15.90:0/4284469468 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595b887a4d0 0x5595b8c582d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.350+0000 7fdcff59e640 1 -- 172.21.15.90:0/4284469468 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595b8cc7b30 con 0x5595b88a3390 2024-04-25T11:36:50.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.351+0000 7fdcff59e640 1 --2- 172.21.15.90:0/4284469468 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595b88a3390 0x5595b8c58d50 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8004e60 tx=0x7fdcf8005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a35da4be24618daf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:50.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.351+0000 7fdce67fc640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:50.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.351+0000 7fdce67fc640 1 -- 172.21.15.90:0/4284469468 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf807c030 con 0x5595b88a3390 2024-04-25T11:36:50.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.351+0000 7fdce67fc640 1 -- 172.21.15.90:0/4284469468 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fdcf8004040 con 0x5595b88a3390 2024-04-25T11:36:50.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.351+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/4284469468 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcf80046c0 con 0x5595b88a3390 2024-04-25T11:36:50.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.351+0000 7fdce67fc640 1 -- 172.21.15.90:0/4284469468 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdcf8085050 con 0x5595b88a3390 2024-04-25T11:36:50.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.351+0000 7fdd0c1b6740 10 client.16086 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:50.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.351+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/4284469468 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5595b8cc8d20 con 0x5595b88a3390 2024-04-25T11:36:50.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.351+0000 7fdce67fc640 1 -- 172.21.15.90:0/4284469468 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdcf80046c0 con 0x5595b88a3390 2024-04-25T11:36:50.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.352+0000 7fdce67fc640 1 -- 172.21.15.90:0/4284469468 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fdcf8077480 con 0x5595b88a3390 2024-04-25T11:36:50.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.352+0000 7fdd0c1b6740 10 client.16086 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:50.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.352+0000 7fdd0c1b6740 10 client.16086 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:36:50.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.352+0000 7fdd0c1b6740 20 client.16086 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:50.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.352+0000 7fdd0c1b6740 1 --2- 172.21.15.90:0/4284469468 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5595b88373a0 0x5595b8ccb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.352+0000 7fdd0c1b6740 4 client.16086 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:36:50.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.352+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/4284469468 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5595b8bb1800 con 0x5595b88373a0 2024-04-25T11:36:50.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.355+0000 7fdcfed9d640 1 --2- 172.21.15.90:0/4284469468 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5595b88373a0 0x5595b8ccb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.367+0000 7fdcfed9d640 1 --2- 172.21.15.90:0/4284469468 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5595b88373a0 0x5595b8ccb410 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1ac4ef108342e961 server_cookie=be03a158489fbd2e in_seq=0 out_seq=0 2024-04-25T11:36:50.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.367+0000 7fdce67fc640 10 client.16086 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:36:50.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.368+0000 7fdce67fc640 1 -- 172.21.15.90:0/4284469468 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5595b8bb1800 con 0x5595b88373a0 2024-04-25T11:36:50.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.368+0000 7fdce67fc640 10 client.16086 handle_command_reply: tid=0 2024-04-25T11:36:50.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.368+0000 7fdd0c1b6740 1 client.16086 shutdown 2024-04-25T11:36:50.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.368+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/4284469468 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595b88a3390 msgr2=0x5595b8c58d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.368+0000 7fdd0c1b6740 1 --2- 172.21.15.90:0/4284469468 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595b88a3390 0x5595b8c58d50 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8004e60 tx=0x7fdcf8005fd0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.368+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/4284469468 shutdown_connections 2024-04-25T11:36:50.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.368+0000 7fdd0c1b6740 1 --2- 172.21.15.90:0/4284469468 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595b88a3390 0x5595b8c58d50 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.368+0000 7fdd0c1b6740 1 --2- 172.21.15.90:0/4284469468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595b88826b0 0x5595b8c58810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.368+0000 7fdd0c1b6740 1 --2- 172.21.15.90:0/4284469468 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5595b88373a0 0x5595b8ccb410 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.368+0000 7fdce67fc640 0 client.16086 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:36:50.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.368+0000 7fdd0c1b6740 1 --2- 172.21.15.90:0/4284469468 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595b887a4d0 0x5595b8c582d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.369+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/4284469468 >> 172.21.15.90:0/4284469468 conn(0x5595b88974e0 msgr2=0x5595b8c3f610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:50.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.369+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/4284469468 shutdown_connections 2024-04-25T11:36:50.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.369+0000 7fdd0c1b6740 1 -- 172.21.15.90:0/4284469468 wait complete. 2024-04-25T11:36:50.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.369+0000 7fdd0c1b6740 20 client.16086 trim_cache size 0 max 16384 2024-04-25T11:36:50.359 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:36:50.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.370+0000 7fdd05230640 1 -- 172.21.15.90:0/3169565323 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fdcec0afee0 msgr2=0x7fdcec0b23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.370+0000 7fdd05230640 1 --2- 172.21.15.90:0/3169565323 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fdcec0afee0 0x7fdcec0b23f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x5595b8c42070 tx=0x7fdcf00098f0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.370+0000 7fdd05230640 1 -- 172.21.15.90:0/3169565323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd001121e0 msgr2=0x7fdd001a1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.370+0000 7fdd05230640 1 --2- 172.21.15.90:0/3169565323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd001121e0 0x7fdd001a1280 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fdce800b450 tx=0x7fdce800b910 comp rx=0 tx=0).stop 2024-04-25T11:36:50.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.370+0000 7fdd05230640 1 -- 172.21.15.90:0/3169565323 shutdown_connections 2024-04-25T11:36:50.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.370+0000 7fdd05230640 1 --2- 172.21.15.90:0/3169565323 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fdcec0afee0 0x7fdcec0b23f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.370+0000 7fdd05230640 1 --2- 172.21.15.90:0/3169565323 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd00114b40 0x7fdd001a5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.370+0000 7fdd05230640 1 --2- 172.21.15.90:0/3169565323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd001121e0 0x7fdd001a1280 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.370+0000 7fdd05230640 1 --2- 172.21.15.90:0/3169565323 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd00075360 0x7fdd001a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.370+0000 7fdd05230640 1 -- 172.21.15.90:0/3169565323 >> 172.21.15.90:0/3169565323 conn(0x7fdd00079280 msgr2=0x7fdd001125f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:50.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.370+0000 7fdd05230640 1 -- 172.21.15.90:0/3169565323 shutdown_connections 2024-04-25T11:36:50.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.370+0000 7fdd05230640 1 -- 172.21.15.90:0/3169565323 wait complete. 2024-04-25T11:36:50.377 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T11:36:50.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.563+0000 7f40b670d640 1 -- 172.21.15.90:0/4043724279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40b010e910 msgr2=0x7f40b010ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.563+0000 7f40b670d640 1 --2- 172.21.15.90:0/4043724279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40b010e910 0x7f40b010ed40 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f40a4009900 tx=0x7f40a402a410 comp rx=0 tx=0).stop 2024-04-25T11:36:50.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.563+0000 7f40b670d640 1 -- 172.21.15.90:0/4043724279 shutdown_connections 2024-04-25T11:36:50.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.563+0000 7f40b670d640 1 --2- 172.21.15.90:0/4043724279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40b010f3d0 0x7f40b0116b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.563+0000 7f40b670d640 1 --2- 172.21.15.90:0/4043724279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40b010e910 0x7f40b010ed40 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.563+0000 7f40b670d640 1 --2- 172.21.15.90:0/4043724279 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40b010a380 0x7f40b010a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.563+0000 7f40b670d640 1 -- 172.21.15.90:0/4043724279 >> 172.21.15.90:0/4043724279 conn(0x7f40b00760a0 msgr2=0x7f40b00764b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:50.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.566+0000 7f40b670d640 1 -- 172.21.15.90:0/4043724279 shutdown_connections 2024-04-25T11:36:50.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.566+0000 7f40b670d640 1 -- 172.21.15.90:0/4043724279 wait complete. 2024-04-25T11:36:50.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.567+0000 7f40b670d640 1 Processor -- start 2024-04-25T11:36:50.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.567+0000 7f40b670d640 1 -- start start 2024-04-25T11:36:50.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.568+0000 7f40b670d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40b010a380 0x7f40b01a67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.568+0000 7f40b670d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40b010e910 0x7f40b01a6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.568+0000 7f40affff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40b010a380 0x7f40b01a67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.568+0000 7f40affff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40b010a380 0x7f40b01a67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52530/0 (socket says 172.21.15.90:52530) 2024-04-25T11:36:50.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.568+0000 7f40b670d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40b010f3d0 0x7f40b01a09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.568+0000 7f40affff640 1 -- 172.21.15.90:0/1362384173 learned_addr learned my addr 172.21.15.90:0/1362384173 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:50.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.569+0000 7f40b670d640 1 -- 172.21.15.90:0/1362384173 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f40b011a370 con 0x7f40b010a380 2024-04-25T11:36:50.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.569+0000 7f40b670d640 1 -- 172.21.15.90:0/1362384173 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f40b011a1f0 con 0x7f40b010f3d0 2024-04-25T11:36:50.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.569+0000 7f40b670d640 1 -- 172.21.15.90:0/1362384173 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f40b011a4f0 con 0x7f40b010e910 2024-04-25T11:36:50.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.569+0000 7f40affff640 1 -- 172.21.15.90:0/1362384173 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40b010e910 msgr2=0x7f40b01a6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.569+0000 7f40affff640 1 --2- 172.21.15.90:0/1362384173 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40b010e910 0x7f40b01a6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.569+0000 7f40affff640 1 -- 172.21.15.90:0/1362384173 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40b010f3d0 msgr2=0x7f40b01a09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.569+0000 7f40b4caa640 1 --2- 172.21.15.90:0/1362384173 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40b010f3d0 0x7f40b01a09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.569+0000 7f40affff640 1 --2- 172.21.15.90:0/1362384173 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40b010f3d0 0x7f40b01a09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.569+0000 7f40affff640 1 -- 172.21.15.90:0/1362384173 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b01a1060 con 0x7f40b010a380 2024-04-25T11:36:50.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.569+0000 7f40b4caa640 1 --2- 172.21.15.90:0/1362384173 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40b010f3d0 0x7f40b01a09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:36:50.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.569+0000 7f40affff640 1 --2- 172.21.15.90:0/1362384173 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40b010a380 0x7f40b01a67b0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f40a000e9f0 tx=0x7f40a000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:50.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.570+0000 7f40ad7fa640 1 -- 172.21.15.90:0/1362384173 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a000cd80 con 0x7f40b010a380 2024-04-25T11:36:50.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.570+0000 7f40ad7fa640 1 -- 172.21.15.90:0/1362384173 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40a00108e0 con 0x7f40b010a380 2024-04-25T11:36:50.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.570+0000 7f40ad7fa640 1 -- 172.21.15.90:0/1362384173 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a0018980 con 0x7f40b010a380 2024-04-25T11:36:50.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.570+0000 7f40b670d640 1 -- 172.21.15.90:0/1362384173 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40b01a1350 con 0x7f40b010a380 2024-04-25T11:36:50.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.570+0000 7f40b670d640 1 -- 172.21.15.90:0/1362384173 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40b01a1790 con 0x7f40b010a380 2024-04-25T11:36:50.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.574+0000 7f40bd5be740 1 Processor -- start 2024-04-25T11:36:50.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.574+0000 7f40bd5be740 1 -- start start 2024-04-25T11:36:50.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.575+0000 7f40bd5be740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561fc35f6390 0x561fc3997810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.575+0000 7f40bd5be740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561fc35cd4d0 0x561fc3997d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.575+0000 7f40bd5be740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35d56b0 0x561fc39a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.575+0000 7f40bd5be740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561fc38fcd80 con 0x561fc35d56b0 2024-04-25T11:36:50.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.575+0000 7f40bd5be740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x561fc38b6a20 con 0x561fc35cd4d0 2024-04-25T11:36:50.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.575+0000 7f40bd5be740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561fc38f8d40 con 0x561fc35f6390 2024-04-25T11:36:50.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.575+0000 7f40af7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35d56b0 0x561fc39a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.575+0000 7f40ad7fa640 1 -- 172.21.15.90:0/1362384173 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f40a0010430 con 0x7f40b010a380 2024-04-25T11:36:50.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.575+0000 7f40af7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35d56b0 0x561fc39a46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52546/0 (socket says 172.21.15.90:52546) 2024-04-25T11:36:50.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.575+0000 7f40af7fe640 1 -- 172.21.15.90:0/762491890 learned_addr learned my addr 172.21.15.90:0/762491890 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:50.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.575+0000 7f40b4caa640 1 --2- 172.21.15.90:0/762491890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561fc35f6390 0x561fc3997810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.575+0000 7f40af7fe640 1 -- 172.21.15.90:0/762491890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561fc35f6390 msgr2=0x561fc3997810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:50.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.575+0000 7f40af7fe640 1 --2- 172.21.15.90:0/762491890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561fc35f6390 0x561fc3997810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.575+0000 7f40af7fe640 1 -- 172.21.15.90:0/762491890 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561fc35cd4d0 msgr2=0x561fc3997d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:50.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.575+0000 7f40af7fe640 1 --2- 172.21.15.90:0/762491890 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561fc35cd4d0 0x561fc3997d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.575+0000 7f40af7fe640 1 -- 172.21.15.90:0/762491890 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fc38c1c30 con 0x561fc35d56b0 2024-04-25T11:36:50.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.575+0000 7f40b4caa640 1 --2- 172.21.15.90:0/762491890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561fc35f6390 0x561fc3997810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:36:50.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.576+0000 7f40af7fe640 1 --2- 172.21.15.90:0/762491890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35d56b0 0x561fc39a46a0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f40a4009850 tx=0x7f40a4005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a38b6c709ecd9ef7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:50.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.576+0000 7f40ad7fa640 1 --2- 172.21.15.90:0/1362384173 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f40980afd10 0x7f40980b2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.576+0000 7f40967fc640 1 -- 172.21.15.90:0/762491890 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a403e070 con 0x561fc35d56b0 2024-04-25T11:36:50.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.576+0000 7f40967fc640 1 -- 172.21.15.90:0/762491890 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40a4005520 con 0x561fc35d56b0 2024-04-25T11:36:50.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.576+0000 7f40967fc640 1 -- 172.21.15.90:0/762491890 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a4035040 con 0x561fc35d56b0 2024-04-25T11:36:50.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.576+0000 7f40ad7fa640 1 -- 172.21.15.90:0/1362384173 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f40a00df500 con 0x7f40b010a380 2024-04-25T11:36:50.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.576+0000 7f40b4caa640 1 --2- 172.21.15.90:0/1362384173 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f40980afd10 0x7f40980b2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.577+0000 7f40bd5be740 1 -- 172.21.15.90:0/762491890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35d56b0 msgr2=0x561fc39a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.577+0000 7f40bd5be740 1 --2- 172.21.15.90:0/762491890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35d56b0 0x561fc39a46a0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f40a4009850 tx=0x7f40a4005f00 comp rx=0 tx=0).stop 2024-04-25T11:36:50.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.578+0000 7f40b4caa640 1 --2- 172.21.15.90:0/1362384173 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f40980afd10 0x7f40980b2220 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x561fc3994980 tx=0x7f40a8007450 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:50.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.578+0000 7f40bd5be740 1 -- 172.21.15.90:0/762491890 shutdown_connections 2024-04-25T11:36:50.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.578+0000 7f40bd5be740 1 --2- 172.21.15.90:0/762491890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35d56b0 0x561fc39a46a0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.578+0000 7f40bd5be740 1 --2- 172.21.15.90:0/762491890 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561fc35cd4d0 0x561fc3997d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.578+0000 7f40bd5be740 1 --2- 172.21.15.90:0/762491890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561fc35f6390 0x561fc3997810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.578+0000 7f40bd5be740 1 -- 172.21.15.90:0/762491890 >> 172.21.15.90:0/762491890 conn(0x561fc35ea4e0 msgr2=0x561fc3993fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:50.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.578+0000 7f40bd5be740 1 -- 172.21.15.90:0/762491890 shutdown_connections 2024-04-25T11:36:50.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.579+0000 7f40bd5be740 1 -- 172.21.15.90:0/762491890 wait complete. 2024-04-25T11:36:50.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.579+0000 7f40bd5be740 1 -- start start 2024-04-25T11:36:50.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.580+0000 7f40bd5be740 1 Processor -- start 2024-04-25T11:36:50.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.580+0000 7f40bd5be740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35cd4d0 0x561fc39a2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.581+0000 7f40bd5be740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561fc35d56b0 0x561fc39a27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.581+0000 7f40bd5be740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561fc35f6390 0x561fc39ae1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.581+0000 7f40bd5be740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561fc39a4d30 con 0x561fc35cd4d0 2024-04-25T11:36:50.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.581+0000 7f40bd5be740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x561fc3883dc0 con 0x561fc35d56b0 2024-04-25T11:36:50.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.581+0000 7f40bd5be740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561fc39a4eb0 con 0x561fc35f6390 2024-04-25T11:36:50.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.581+0000 7f40bd5be740 10 client.0 _ll_register_callbacks cb 0x561fc389f600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:50.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.581+0000 7f40affff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35cd4d0 0x561fc39a2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.581+0000 7f40affff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35cd4d0 0x561fc39a2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52560/0 (socket says 172.21.15.90:52560) 2024-04-25T11:36:50.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.581+0000 7f40affff640 1 -- 172.21.15.90:0/3657893066 learned_addr learned my addr 172.21.15.90:0/3657893066 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:50.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.582+0000 7f40affff640 1 -- 172.21.15.90:0/3657893066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561fc35f6390 msgr2=0x561fc39ae1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:50.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.582+0000 7f40affff640 1 --2- 172.21.15.90:0/3657893066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561fc35f6390 0x561fc39ae1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.582+0000 7f40affff640 1 -- 172.21.15.90:0/3657893066 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561fc35d56b0 msgr2=0x561fc39a27c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:36:50.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.582+0000 7f40affff640 1 --2- 172.21.15.90:0/3657893066 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561fc35d56b0 0x561fc39a27c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.582+0000 7f40affff640 1 -- 172.21.15.90:0/3657893066 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fc360e3f0 con 0x561fc35cd4d0 2024-04-25T11:36:50.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.582+0000 7f40affff640 1 --2- 172.21.15.90:0/3657893066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35cd4d0 0x561fc39a2280 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f40a000e9c0 tx=0x7f40a00a6e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f09746c4f4f50cde server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:50.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.582+0000 7f408f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:50.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.583+0000 7f408f7fe640 1 -- 172.21.15.90:0/3657893066 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a00dd020 con 0x561fc35cd4d0 2024-04-25T11:36:50.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.583+0000 7f40bd5be740 1 -- 172.21.15.90:0/3657893066 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40a4005520 con 0x561fc35cd4d0 2024-04-25T11:36:50.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.583+0000 7f408f7fe640 1 -- 172.21.15.90:0/3657893066 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f40a00a6840 con 0x561fc35cd4d0 2024-04-25T11:36:50.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.583+0000 7f408f7fe640 1 -- 172.21.15.90:0/3657893066 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f40a00a7ce0 con 0x561fc35cd4d0 2024-04-25T11:36:50.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.583+0000 7f40bd5be740 10 client.16098 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:50.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.584+0000 7f40bd5be740 1 -- 172.21.15.90:0/3657893066 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561fc36d87d0 con 0x561fc35cd4d0 2024-04-25T11:36:50.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.585+0000 7f408f7fe640 1 -- 172.21.15.90:0/3657893066 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40a4005520 con 0x561fc35cd4d0 2024-04-25T11:36:50.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.586+0000 7f408f7fe640 1 -- 172.21.15.90:0/3657893066 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f40a00a2f50 con 0x561fc35cd4d0 2024-04-25T11:36:50.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.586+0000 7f40bd5be740 10 client.16098 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:50.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.586+0000 7f40bd5be740 10 client.16098 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:36:50.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.586+0000 7f40bd5be740 20 client.16098 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:50.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.586+0000 7f40bd5be740 1 --2- 172.21.15.90:0/3657893066 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x561fc35b4f60 0x561fc39b4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.586+0000 7f40bd5be740 4 client.16098 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:36:50.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.586+0000 7f40bd5be740 1 -- 172.21.15.90:0/3657893066 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561fc390a990 con 0x561fc35b4f60 2024-04-25T11:36:50.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.588+0000 7f40af7fe640 1 --2- 172.21.15.90:0/3657893066 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x561fc35b4f60 0x561fc39b4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.592+0000 7f40af7fe640 1 --2- 172.21.15.90:0/3657893066 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x561fc35b4f60 0x561fc39b4a80 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=40e7f444750cb0d4 server_cookie=b94af38eea51f02 in_seq=0 out_seq=0 2024-04-25T11:36:50.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.592+0000 7f408f7fe640 10 client.16098 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:36:50.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.594+0000 7f408f7fe640 1 -- 172.21.15.90:0/3657893066 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561fc390a990 con 0x561fc35b4f60 2024-04-25T11:36:50.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.594+0000 7f408f7fe640 10 client.16098 handle_command_reply: tid=0 2024-04-25T11:36:50.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.594+0000 7f40bd5be740 1 client.16098 shutdown 2024-04-25T11:36:50.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.594+0000 7f40bd5be740 1 -- 172.21.15.90:0/3657893066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35cd4d0 msgr2=0x561fc39a2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.594+0000 7f40bd5be740 1 --2- 172.21.15.90:0/3657893066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35cd4d0 0x561fc39a2280 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f40a000e9c0 tx=0x7f40a00a6e40 comp rx=0 tx=0).stop 2024-04-25T11:36:50.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.594+0000 7f40bd5be740 1 -- 172.21.15.90:0/3657893066 shutdown_connections 2024-04-25T11:36:50.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.594+0000 7f40bd5be740 1 --2- 172.21.15.90:0/3657893066 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x561fc35b4f60 0x561fc39b4a80 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.594+0000 7f40bd5be740 1 --2- 172.21.15.90:0/3657893066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561fc35f6390 0x561fc39ae1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.594+0000 7f40bd5be740 1 --2- 172.21.15.90:0/3657893066 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561fc35d56b0 0x561fc39a27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.594+0000 7f40bd5be740 1 --2- 172.21.15.90:0/3657893066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35cd4d0 0x561fc39a2280 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.594+0000 7f40bd5be740 1 -- 172.21.15.90:0/3657893066 >> 172.21.15.90:0/3657893066 conn(0x561fc35ea4e0 msgr2=0x561fc3999530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:50.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.594+0000 7f408f7fe640 0 client.16098 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:36:50.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.595+0000 7f40bd5be740 1 -- 172.21.15.90:0/3657893066 shutdown_connections 2024-04-25T11:36:50.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.595+0000 7f40bd5be740 1 -- 172.21.15.90:0/3657893066 wait complete. 2024-04-25T11:36:50.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.595+0000 7f40bd5be740 20 client.16098 trim_cache size 0 max 16384 2024-04-25T11:36:50.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.619+0000 7f40bd5be740 1 Processor -- start 2024-04-25T11:36:50.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.619+0000 7f40bd5be740 1 -- start start 2024-04-25T11:36:50.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.619+0000 7f40bd5be740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35cd4d0 0x561fc39a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.619+0000 7f40bd5be740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561fc35d56b0 0x561fc3993ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.619+0000 7f40bd5be740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561fc35f6390 0x561fc39ade10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.619+0000 7f40bd5be740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f40b01b6090 con 0x561fc35cd4d0 2024-04-25T11:36:50.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.619+0000 7f40bd5be740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f40b01b7bd0 con 0x561fc35f6390 2024-04-25T11:36:50.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.619+0000 7f40bd5be740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f40b01b6760 con 0x561fc35d56b0 2024-04-25T11:36:50.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.619+0000 7f40affff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35cd4d0 0x561fc39a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.619+0000 7f40affff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35cd4d0 0x561fc39a0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52566/0 (socket says 172.21.15.90:52566) 2024-04-25T11:36:50.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.619+0000 7f40affff640 1 -- 172.21.15.90:0/1420581833 learned_addr learned my addr 172.21.15.90:0/1420581833 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:50.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.620+0000 7f40b4caa640 1 --2- 172.21.15.90:0/1420581833 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561fc35f6390 0x561fc39ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.620+0000 7f40affff640 1 -- 172.21.15.90:0/1420581833 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561fc35d56b0 msgr2=0x561fc3993ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:36:50.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.620+0000 7f40affff640 1 --2- 172.21.15.90:0/1420581833 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561fc35d56b0 0x561fc3993ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.620+0000 7f40affff640 1 -- 172.21.15.90:0/1420581833 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561fc35f6390 msgr2=0x561fc39ade10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:50.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.620+0000 7f40affff640 1 --2- 172.21.15.90:0/1420581833 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561fc35f6390 0x561fc39ade10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.620+0000 7f40affff640 1 -- 172.21.15.90:0/1420581833 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fc39a05c0 con 0x561fc35cd4d0 2024-04-25T11:36:50.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.620+0000 7f40affff640 1 --2- 172.21.15.90:0/1420581833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35cd4d0 0x561fc39a0fb0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f40a00a72e0 tx=0x7f40a0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=450e0066a041de8f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:50.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.648+0000 7f4095ffb640 1 -- 172.21.15.90:0/1420581833 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a00ab990 con 0x561fc35cd4d0 2024-04-25T11:36:50.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.648+0000 7f4095ffb640 1 -- 172.21.15.90:0/1420581833 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40a00b0440 con 0x561fc35cd4d0 2024-04-25T11:36:50.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.648+0000 7f4095ffb640 1 -- 172.21.15.90:0/1420581833 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a00a6630 con 0x561fc35cd4d0 2024-04-25T11:36:50.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.649+0000 7f40bd5be740 1 -- 172.21.15.90:0/1420581833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35cd4d0 msgr2=0x561fc39a0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.649+0000 7f40bd5be740 1 --2- 172.21.15.90:0/1420581833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35cd4d0 0x561fc39a0fb0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f40a00a72e0 tx=0x7f40a0000bc0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.649+0000 7f40bd5be740 1 -- 172.21.15.90:0/1420581833 shutdown_connections 2024-04-25T11:36:50.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.649+0000 7f40bd5be740 1 --2- 172.21.15.90:0/1420581833 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561fc35f6390 0x561fc39ade10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.649+0000 7f40bd5be740 1 --2- 172.21.15.90:0/1420581833 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561fc35d56b0 0x561fc3993ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.649+0000 7f40bd5be740 1 --2- 172.21.15.90:0/1420581833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35cd4d0 0x561fc39a0fb0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.649+0000 7f40bd5be740 1 -- 172.21.15.90:0/1420581833 >> 172.21.15.90:0/1420581833 conn(0x561fc35ea4e0 msgr2=0x561fc39937c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:50.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.649+0000 7f40bd5be740 1 -- 172.21.15.90:0/1420581833 shutdown_connections 2024-04-25T11:36:50.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.649+0000 7f40bd5be740 1 -- 172.21.15.90:0/1420581833 wait complete. 2024-04-25T11:36:50.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.650+0000 7f40bd5be740 1 -- start start 2024-04-25T11:36:50.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.650+0000 7f40bd5be740 1 Processor -- start 2024-04-25T11:36:50.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.650+0000 7f40bd5be740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561fc35cd4d0 0x561fc39a5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.650+0000 7f40bd5be740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35d56b0 0x561fc39a5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.650+0000 7f40bd5be740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561fc35f6390 0x561fc3a1ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.650+0000 7f40bd5be740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f40b01b7500 con 0x561fc35d56b0 2024-04-25T11:36:50.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.650+0000 7f40bd5be740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f40b01b6e30 con 0x561fc35f6390 2024-04-25T11:36:50.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.650+0000 7f40bd5be740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f40b01b4070 con 0x561fc35cd4d0 2024-04-25T11:36:50.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.650+0000 7f40bd5be740 10 client.0 _ll_register_callbacks cb 0x7f40a0009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:50.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.650+0000 7f40af7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35d56b0 0x561fc39a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.650+0000 7f40af7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35d56b0 0x561fc39a5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52568/0 (socket says 172.21.15.90:52568) 2024-04-25T11:36:50.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.650+0000 7f40af7fe640 1 -- 172.21.15.90:0/2399094096 learned_addr learned my addr 172.21.15.90:0/2399094096 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:50.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.651+0000 7f40affff640 1 --2- 172.21.15.90:0/2399094096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561fc35cd4d0 0x561fc39a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.651+0000 7f40af7fe640 1 -- 172.21.15.90:0/2399094096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561fc35cd4d0 msgr2=0x561fc39a5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:50.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.651+0000 7f40af7fe640 1 --2- 172.21.15.90:0/2399094096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561fc35cd4d0 0x561fc39a5150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.651+0000 7f40af7fe640 1 -- 172.21.15.90:0/2399094096 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561fc35f6390 msgr2=0x561fc3a1ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:50.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.651+0000 7f40af7fe640 1 --2- 172.21.15.90:0/2399094096 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561fc35f6390 0x561fc3a1ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.651+0000 7f40af7fe640 1 -- 172.21.15.90:0/2399094096 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fc39a5f60 con 0x561fc35d56b0 2024-04-25T11:36:50.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.651+0000 7f40af7fe640 1 --2- 172.21.15.90:0/2399094096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35d56b0 0x561fc39a5690 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f40a4005f70 tx=0x7f40a402a910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b14a9ee06ee9849 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:50.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.651+0000 7f408f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:50.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.651+0000 7f408f7fe640 1 -- 172.21.15.90:0/2399094096 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a4050070 con 0x561fc35d56b0 2024-04-25T11:36:50.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.652+0000 7f408f7fe640 1 -- 172.21.15.90:0/2399094096 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f40a4004f30 con 0x561fc35d56b0 2024-04-25T11:36:50.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.652+0000 7f40bd5be740 1 -- 172.21.15.90:0/2399094096 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40a00b0440 con 0x561fc35d56b0 2024-04-25T11:36:50.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.652+0000 7f408f7fe640 1 -- 172.21.15.90:0/2399094096 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f40a405c050 con 0x561fc35d56b0 2024-04-25T11:36:50.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.652+0000 7f40bd5be740 10 client.16104 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:50.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.652+0000 7f40bd5be740 1 -- 172.21.15.90:0/2399094096 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561fc3a1c420 con 0x561fc35d56b0 2024-04-25T11:36:50.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.652+0000 7f408f7fe640 1 -- 172.21.15.90:0/2399094096 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40a00b0440 con 0x561fc35d56b0 2024-04-25T11:36:50.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.652+0000 7f408f7fe640 1 -- 172.21.15.90:0/2399094096 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f40a404c7f0 con 0x561fc35d56b0 2024-04-25T11:36:50.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.652+0000 7f40bd5be740 10 client.16104 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:50.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.652+0000 7f40bd5be740 10 client.16104 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:36:50.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.652+0000 7f40bd5be740 20 client.16104 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:50.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.652+0000 7f40bd5be740 1 --2- 172.21.15.90:0/2399094096 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x561fc358a3a0 0x561fc3a1e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.653+0000 7f40bd5be740 4 client.16104 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:36:50.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.653+0000 7f40bd5be740 1 -- 172.21.15.90:0/2399094096 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561fc3904800 con 0x561fc358a3a0 2024-04-25T11:36:50.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.653+0000 7f40affff640 1 --2- 172.21.15.90:0/2399094096 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x561fc358a3a0 0x561fc3a1e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.655+0000 7f408f7fe640 10 client.16104 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:36:50.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.655+0000 7f40affff640 1 --2- 172.21.15.90:0/2399094096 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x561fc358a3a0 0x561fc3a1e770 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6005a624a5a85eca server_cookie=d19dcb57d6ae36f1 in_seq=0 out_seq=0 2024-04-25T11:36:50.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.659+0000 7f408f7fe640 1 -- 172.21.15.90:0/2399094096 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561fc3904800 con 0x561fc358a3a0 2024-04-25T11:36:50.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.659+0000 7f408f7fe640 10 client.16104 handle_command_reply: tid=0 2024-04-25T11:36:50.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.659+0000 7f40bd5be740 1 client.16104 shutdown 2024-04-25T11:36:50.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.660+0000 7f40bd5be740 1 -- 172.21.15.90:0/2399094096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35d56b0 msgr2=0x561fc39a5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.660+0000 7f40bd5be740 1 --2- 172.21.15.90:0/2399094096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35d56b0 0x561fc39a5690 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f40a4005f70 tx=0x7f40a402a910 comp rx=0 tx=0).stop 2024-04-25T11:36:50.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.660+0000 7f40bd5be740 1 -- 172.21.15.90:0/2399094096 shutdown_connections 2024-04-25T11:36:50.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.660+0000 7f40bd5be740 1 --2- 172.21.15.90:0/2399094096 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x561fc358a3a0 0x561fc3a1e770 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.660+0000 7f40bd5be740 1 --2- 172.21.15.90:0/2399094096 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561fc35f6390 0x561fc3a1ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.660+0000 7f40bd5be740 1 --2- 172.21.15.90:0/2399094096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fc35d56b0 0x561fc39a5690 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.660+0000 7f40bd5be740 1 --2- 172.21.15.90:0/2399094096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561fc35cd4d0 0x561fc39a5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.660+0000 7f40bd5be740 1 -- 172.21.15.90:0/2399094096 >> 172.21.15.90:0/2399094096 conn(0x561fc35ea4e0 msgr2=0x561fc3992610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:50.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.660+0000 7f408f7fe640 0 client.16104 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:36:50.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.660+0000 7f40bd5be740 1 -- 172.21.15.90:0/2399094096 shutdown_connections 2024-04-25T11:36:50.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.661+0000 7f40bd5be740 1 -- 172.21.15.90:0/2399094096 wait complete. 2024-04-25T11:36:50.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.661+0000 7f40bd5be740 20 client.16104 trim_cache size 0 max 16384 2024-04-25T11:36:50.651 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:36:50.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.662+0000 7f40b670d640 1 -- 172.21.15.90:0/1362384173 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f40980afd10 msgr2=0x7f40980b2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.662+0000 7f40b670d640 1 --2- 172.21.15.90:0/1362384173 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f40980afd10 0x7f40980b2220 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x561fc3994980 tx=0x7f40a8007450 comp rx=0 tx=0).stop 2024-04-25T11:36:50.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.662+0000 7f40b670d640 1 -- 172.21.15.90:0/1362384173 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40b010a380 msgr2=0x7f40b01a67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.662+0000 7f40b670d640 1 --2- 172.21.15.90:0/1362384173 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40b010a380 0x7f40b01a67b0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f40a000e9f0 tx=0x7f40a000eeb0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.662+0000 7f40b670d640 1 -- 172.21.15.90:0/1362384173 shutdown_connections 2024-04-25T11:36:50.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.662+0000 7f40b670d640 1 --2- 172.21.15.90:0/1362384173 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f40980afd10 0x7f40980b2220 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.662+0000 7f40b670d640 1 --2- 172.21.15.90:0/1362384173 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40b010f3d0 0x7f40b01a09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.662+0000 7f40b670d640 1 --2- 172.21.15.90:0/1362384173 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40b010e910 0x7f40b01a6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.662+0000 7f40b670d640 1 --2- 172.21.15.90:0/1362384173 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40b010a380 0x7f40b01a67b0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.662+0000 7f40b670d640 1 -- 172.21.15.90:0/1362384173 >> 172.21.15.90:0/1362384173 conn(0x7f40b00760a0 msgr2=0x7f40b01083f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:50.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.662+0000 7f40b670d640 1 -- 172.21.15.90:0/1362384173 shutdown_connections 2024-04-25T11:36:50.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.662+0000 7f40b670d640 1 -- 172.21.15.90:0/1362384173 wait complete. 2024-04-25T11:36:50.672 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:50 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/566066618' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:36:50.673 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T11:36:50.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:50 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/566066618' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:36:50.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.880+0000 7effe4d14640 1 -- 172.21.15.90:0/1500784199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effe010d380 msgr2=0x7effe010f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.880+0000 7effe4d14640 1 --2- 172.21.15.90:0/1500784199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effe010d380 0x7effe010f7a0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7effd8009ca0 tx=0x7effd802a8e0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.880+0000 7effe4d14640 1 -- 172.21.15.90:0/1500784199 shutdown_connections 2024-04-25T11:36:50.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.880+0000 7effe4d14640 1 --2- 172.21.15.90:0/1500784199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effe010d380 0x7effe010f7a0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.880+0000 7effe4d14640 1 --2- 172.21.15.90:0/1500784199 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7effe0102af0 0x7effe0102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.880+0000 7effe4d14640 1 --2- 172.21.15.90:0/1500784199 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7effe0108a20 0x7effe01025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.880+0000 7effe4d14640 1 -- 172.21.15.90:0/1500784199 >> 172.21.15.90:0/1500784199 conn(0x7effe0078100 msgr2=0x7effe0078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:50.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.880+0000 7effe4d14640 1 -- 172.21.15.90:0/1500784199 shutdown_connections 2024-04-25T11:36:50.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.880+0000 7effe4d14640 1 -- 172.21.15.90:0/1500784199 wait complete. 2024-04-25T11:36:50.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.881+0000 7effe4d14640 1 Processor -- start 2024-04-25T11:36:50.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.881+0000 7effe4d14640 1 -- start start 2024-04-25T11:36:50.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.881+0000 7effe4d14640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7effe0102af0 0x7effe019a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.881+0000 7effe4d14640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effe0108a20 0x7effe019acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.881+0000 7effe4d14640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7effe010d380 0x7effe019ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.881+0000 7effe4d14640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7effe0112c80 con 0x7effe0108a20 2024-04-25T11:36:50.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.881+0000 7effe4d14640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7effe0112b00 con 0x7effe010d380 2024-04-25T11:36:50.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.881+0000 7effe4d14640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7effe0112e00 con 0x7effe0102af0 2024-04-25T11:36:50.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.881+0000 7effd7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effe0108a20 0x7effe019acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.881+0000 7effd7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effe0108a20 0x7effe019acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52594/0 (socket says 172.21.15.90:52594) 2024-04-25T11:36:50.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.881+0000 7effd7fff640 1 -- 172.21.15.90:0/1020064491 learned_addr learned my addr 172.21.15.90:0/1020064491 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:50.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.882+0000 7effd7fff640 1 -- 172.21.15.90:0/1020064491 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7effe0102af0 msgr2=0x7effe019a770 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:36:50.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.882+0000 7effd7fff640 1 --2- 172.21.15.90:0/1020064491 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7effe0102af0 0x7effe019a770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.882+0000 7effd7fff640 1 -- 172.21.15.90:0/1020064491 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7effe010d380 msgr2=0x7effe019ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.882+0000 7effd7fff640 1 --2- 172.21.15.90:0/1020064491 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7effe010d380 0x7effe019ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.882+0000 7effd7fff640 1 -- 172.21.15.90:0/1020064491 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effe019f5d0 con 0x7effe0108a20 2024-04-25T11:36:50.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.882+0000 7effd7fff640 1 --2- 172.21.15.90:0/1020064491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effe0108a20 0x7effe019acd0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7effc8006fd0 tx=0x7effc800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:50.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.883+0000 7effd77fe640 1 -- 172.21.15.90:0/1020064491 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc8011830 con 0x7effe0108a20 2024-04-25T11:36:50.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.883+0000 7effe4d14640 1 -- 172.21.15.90:0/1020064491 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effe019f8c0 con 0x7effe0108a20 2024-04-25T11:36:50.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.883+0000 7effd77fe640 1 -- 172.21.15.90:0/1020064491 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effc8016070 con 0x7effe0108a20 2024-04-25T11:36:50.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.883+0000 7effd77fe640 1 -- 172.21.15.90:0/1020064491 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc8010e00 con 0x7effe0108a20 2024-04-25T11:36:50.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.883+0000 7effe4d14640 1 -- 172.21.15.90:0/1020064491 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effe01a7240 con 0x7effe0108a20 2024-04-25T11:36:50.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.887+0000 7effebbc9740 1 Processor -- start 2024-04-25T11:36:50.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.887+0000 7effebbc9740 1 -- start start 2024-04-25T11:36:50.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.887+0000 7effd77fe640 1 -- 172.21.15.90:0/1020064491 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7effc801b460 con 0x7effe0108a20 2024-04-25T11:36:50.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.887+0000 7effebbc9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563e62c86390 0x563e63027810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.887+0000 7effebbc9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c5d4d0 0x563e63027d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.887+0000 7effebbc9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e62c656b0 0x563e630346a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.887+0000 7effebbc9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563e62f8cd80 con 0x563e62c5d4d0 2024-04-25T11:36:50.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.888+0000 7effebbc9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x563e62f46a20 con 0x563e62c86390 2024-04-25T11:36:50.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.888+0000 7effebbc9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563e62f88d40 con 0x563e62c656b0 2024-04-25T11:36:50.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.888+0000 7effde59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c5d4d0 0x563e63027d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.888+0000 7effde59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c5d4d0 0x563e63027d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52608/0 (socket says 172.21.15.90:52608) 2024-04-25T11:36:50.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.888+0000 7effde59c640 1 -- 172.21.15.90:0/3834974144 learned_addr learned my addr 172.21.15.90:0/3834974144 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:50.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.888+0000 7effde59c640 1 -- 172.21.15.90:0/3834974144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e62c656b0 msgr2=0x563e630346a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:50.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.888+0000 7effde59c640 1 --2- 172.21.15.90:0/3834974144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e62c656b0 0x563e630346a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.888+0000 7effde59c640 1 -- 172.21.15.90:0/3834974144 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563e62c86390 msgr2=0x563e63027810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:50.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.888+0000 7effde59c640 1 --2- 172.21.15.90:0/3834974144 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563e62c86390 0x563e63027810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.888+0000 7effde59c640 1 -- 172.21.15.90:0/3834974144 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e62f51c30 con 0x563e62c5d4d0 2024-04-25T11:36:50.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.889+0000 7effde59c640 1 --2- 172.21.15.90:0/3834974144 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c5d4d0 0x563e63027d50 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7effd0009b70 tx=0x7effd000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89eb33e0b98b3968 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:50.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.890+0000 7effd77fe640 1 --2- 172.21.15.90:0/1020064491 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7effcc0afce0 0x7effcc0b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.890+0000 7effded9d640 1 --2- 172.21.15.90:0/1020064491 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7effcc0afce0 0x7effcc0b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.890+0000 7effd4ff9640 1 -- 172.21.15.90:0/3834974144 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd0009e60 con 0x563e62c5d4d0 2024-04-25T11:36:50.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.891+0000 7effd4ff9640 1 -- 172.21.15.90:0/3834974144 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effd00054d0 con 0x563e62c5d4d0 2024-04-25T11:36:50.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.891+0000 7effd4ff9640 1 -- 172.21.15.90:0/3834974144 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd0010690 con 0x563e62c5d4d0 2024-04-25T11:36:50.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.891+0000 7effd77fe640 1 -- 172.21.15.90:0/1020064491 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7effc80de6f0 con 0x7effe0108a20 2024-04-25T11:36:50.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.892+0000 7effebbc9740 1 -- 172.21.15.90:0/3834974144 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c5d4d0 msgr2=0x563e63027d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.892+0000 7effded9d640 1 --2- 172.21.15.90:0/1020064491 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7effcc0afce0 0x7effcc0b21f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7effd802ae00 tx=0x7effd8005c50 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:50.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.892+0000 7effebbc9740 1 --2- 172.21.15.90:0/3834974144 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c5d4d0 0x563e63027d50 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7effd0009b70 tx=0x7effd000e720 comp rx=0 tx=0).stop 2024-04-25T11:36:50.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.892+0000 7effebbc9740 1 -- 172.21.15.90:0/3834974144 shutdown_connections 2024-04-25T11:36:50.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.892+0000 7effebbc9740 1 --2- 172.21.15.90:0/3834974144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e62c656b0 0x563e630346a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.892+0000 7effebbc9740 1 --2- 172.21.15.90:0/3834974144 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c5d4d0 0x563e63027d50 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.892+0000 7effebbc9740 1 --2- 172.21.15.90:0/3834974144 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563e62c86390 0x563e63027810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.892+0000 7effebbc9740 1 -- 172.21.15.90:0/3834974144 >> 172.21.15.90:0/3834974144 conn(0x563e62c7a4e0 msgr2=0x563e63023fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:50.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.892+0000 7effebbc9740 1 -- 172.21.15.90:0/3834974144 shutdown_connections 2024-04-25T11:36:50.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.892+0000 7effebbc9740 1 -- 172.21.15.90:0/3834974144 wait complete. 2024-04-25T11:36:50.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.893+0000 7effebbc9740 1 -- start start 2024-04-25T11:36:50.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.894+0000 7effebbc9740 1 Processor -- start 2024-04-25T11:36:50.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.894+0000 7effebbc9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c5d4d0 0x563e630322e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.894+0000 7effebbc9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563e62c656b0 0x563e63032820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.894+0000 7effebbc9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e62c86390 0x563e63042700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.894+0000 7effde59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c5d4d0 0x563e630322e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.894+0000 7effebbc9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563e63039140 con 0x563e62c5d4d0 2024-04-25T11:36:50.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.894+0000 7effebbc9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x563e62f13dc0 con 0x563e62c656b0 2024-04-25T11:36:50.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.894+0000 7effebbc9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563e630392c0 con 0x563e62c86390 2024-04-25T11:36:50.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.894+0000 7effde59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c5d4d0 0x563e630322e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52620/0 (socket says 172.21.15.90:52620) 2024-04-25T11:36:50.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.894+0000 7effde59c640 1 -- 172.21.15.90:0/1426617332 learned_addr learned my addr 172.21.15.90:0/1426617332 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:50.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.894+0000 7effebbc9740 10 client.0 _ll_register_callbacks cb 0x563e62f2f600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:50.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.895+0000 7effde59c640 1 -- 172.21.15.90:0/1426617332 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e62c86390 msgr2=0x563e63042700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:50.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.895+0000 7effde59c640 1 --2- 172.21.15.90:0/1426617332 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e62c86390 0x563e63042700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.895+0000 7effde59c640 1 -- 172.21.15.90:0/1426617332 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563e62c656b0 msgr2=0x563e63032820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:50.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.895+0000 7effde59c640 1 --2- 172.21.15.90:0/1426617332 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563e62c656b0 0x563e63032820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.895+0000 7effde59c640 1 -- 172.21.15.90:0/1426617332 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e62c9e3f0 con 0x563e62c5d4d0 2024-04-25T11:36:50.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.895+0000 7effde59c640 1 --2- 172.21.15.90:0/1426617332 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c5d4d0 0x563e630322e0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7effd0004c50 tx=0x7effd00098f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d54ce4f6017b7d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:50.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.895+0000 7effc5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:50.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.896+0000 7effc5ffb640 1 -- 172.21.15.90:0/1426617332 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd001d040 con 0x563e62c5d4d0 2024-04-25T11:36:50.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.896+0000 7effc5ffb640 1 -- 172.21.15.90:0/1426617332 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7effd0005020 con 0x563e62c5d4d0 2024-04-25T11:36:50.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.896+0000 7effebbc9740 1 -- 172.21.15.90:0/1426617332 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7effd00054d0 con 0x563e62c5d4d0 2024-04-25T11:36:50.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.896+0000 7effc5ffb640 1 -- 172.21.15.90:0/1426617332 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7effd0030050 con 0x563e62c5d4d0 2024-04-25T11:36:50.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.896+0000 7effebbc9740 10 client.16116 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:50.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.896+0000 7effebbc9740 1 -- 172.21.15.90:0/1426617332 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563e62d687d0 con 0x563e62c5d4d0 2024-04-25T11:36:50.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.897+0000 7effc5ffb640 1 -- 172.21.15.90:0/1426617332 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effd00054d0 con 0x563e62c5d4d0 2024-04-25T11:36:50.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.897+0000 7effc5ffb640 1 -- 172.21.15.90:0/1426617332 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7effd0018c80 con 0x563e62c5d4d0 2024-04-25T11:36:50.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.897+0000 7effebbc9740 10 client.16116 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:50.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.897+0000 7effebbc9740 10 client.16116 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:36:50.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.897+0000 7effebbc9740 20 client.16116 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:50.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.897+0000 7effebbc9740 1 --2- 172.21.15.90:0/1426617332 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x563e62c44f60 0x563e63048b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.897+0000 7effebbc9740 4 client.16116 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:36:50.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.897+0000 7effebbc9740 1 -- 172.21.15.90:0/1426617332 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e62f9a990 con 0x563e62c44f60 2024-04-25T11:36:50.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.898+0000 7effd7fff640 1 --2- 172.21.15.90:0/1426617332 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x563e62c44f60 0x563e63048b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.900+0000 7effd7fff640 1 --2- 172.21.15.90:0/1426617332 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x563e62c44f60 0x563e63048b10 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dd61c7d08786e757 server_cookie=19ff8c26489692f3 in_seq=0 out_seq=0 2024-04-25T11:36:50.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.900+0000 7effc5ffb640 10 client.16116 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:36:50.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.903+0000 7effc5ffb640 1 -- 172.21.15.90:0/1426617332 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563e62f9a990 con 0x563e62c44f60 2024-04-25T11:36:50.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.903+0000 7effc5ffb640 10 client.16116 handle_command_reply: tid=0 2024-04-25T11:36:50.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.903+0000 7effebbc9740 1 client.16116 shutdown 2024-04-25T11:36:50.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.903+0000 7effebbc9740 1 -- 172.21.15.90:0/1426617332 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c5d4d0 msgr2=0x563e630322e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.903+0000 7effebbc9740 1 --2- 172.21.15.90:0/1426617332 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c5d4d0 0x563e630322e0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7effd0004c50 tx=0x7effd00098f0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.903+0000 7effebbc9740 1 -- 172.21.15.90:0/1426617332 shutdown_connections 2024-04-25T11:36:50.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.903+0000 7effebbc9740 1 --2- 172.21.15.90:0/1426617332 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x563e62c44f60 0x563e63048b10 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.903+0000 7effebbc9740 1 --2- 172.21.15.90:0/1426617332 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e62c86390 0x563e63042700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.903+0000 7effebbc9740 1 --2- 172.21.15.90:0/1426617332 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563e62c656b0 0x563e63032820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.903+0000 7effebbc9740 1 --2- 172.21.15.90:0/1426617332 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c5d4d0 0x563e630322e0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.903+0000 7effc5ffb640 0 client.16116 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:36:50.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.903+0000 7effebbc9740 1 -- 172.21.15.90:0/1426617332 >> 172.21.15.90:0/1426617332 conn(0x563e62c7a4e0 msgr2=0x563e63029530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:50.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.903+0000 7effebbc9740 1 -- 172.21.15.90:0/1426617332 shutdown_connections 2024-04-25T11:36:50.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.903+0000 7effebbc9740 1 -- 172.21.15.90:0/1426617332 wait complete. 2024-04-25T11:36:50.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.903+0000 7effebbc9740 20 client.16116 trim_cache size 0 max 16384 2024-04-25T11:36:50.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.933+0000 7effebbc9740 1 Processor -- start 2024-04-25T11:36:50.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.933+0000 7effebbc9740 1 -- start start 2024-04-25T11:36:50.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.934+0000 7effebbc9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c5d4d0 0x563e630312a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.935+0000 7effebbc9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e62c656b0 0x563e63038820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.935+0000 7effebbc9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563e62c86390 0x563e6302c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.935+0000 7effebbc9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7effe01adaf0 con 0x563e62c5d4d0 2024-04-25T11:36:50.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.935+0000 7effde59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c5d4d0 0x563e630312a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.935+0000 7effebbc9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7effe01af630 con 0x563e62c86390 2024-04-25T11:36:50.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.935+0000 7effebbc9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7effe01ae1c0 con 0x563e62c656b0 2024-04-25T11:36:50.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.935+0000 7effde59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c5d4d0 0x563e630312a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52624/0 (socket says 172.21.15.90:52624) 2024-04-25T11:36:50.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.935+0000 7effde59c640 1 -- 172.21.15.90:0/2756656136 learned_addr learned my addr 172.21.15.90:0/2756656136 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:50.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.936+0000 7effde59c640 1 -- 172.21.15.90:0/2756656136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e62c656b0 msgr2=0x563e63038820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:50.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.936+0000 7effde59c640 1 --2- 172.21.15.90:0/2756656136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e62c656b0 0x563e63038820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.936+0000 7effde59c640 1 -- 172.21.15.90:0/2756656136 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563e62c86390 msgr2=0x563e6302c430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:36:50.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.936+0000 7effde59c640 1 --2- 172.21.15.90:0/2756656136 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563e62c86390 0x563e6302c430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.936+0000 7effde59c640 1 -- 172.21.15.90:0/2756656136 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e630275a0 con 0x563e62c5d4d0 2024-04-25T11:36:50.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.937+0000 7effde59c640 1 --2- 172.21.15.90:0/2756656136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c5d4d0 0x563e630312a0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7effd0009690 tx=0x7effd00106a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b64b2348a4c914a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:50.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.937+0000 7effc7fff640 1 -- 172.21.15.90:0/2756656136 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd0038030 con 0x563e62c5d4d0 2024-04-25T11:36:50.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.937+0000 7effc7fff640 1 -- 172.21.15.90:0/2756656136 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effd0027070 con 0x563e62c5d4d0 2024-04-25T11:36:50.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.937+0000 7effc7fff640 1 -- 172.21.15.90:0/2756656136 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd0004ee0 con 0x563e62c5d4d0 2024-04-25T11:36:50.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.939+0000 7effebbc9740 1 -- 172.21.15.90:0/2756656136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c5d4d0 msgr2=0x563e630312a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.939+0000 7effebbc9740 1 --2- 172.21.15.90:0/2756656136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c5d4d0 0x563e630312a0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7effd0009690 tx=0x7effd00106a0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.941+0000 7effebbc9740 1 -- 172.21.15.90:0/2756656136 shutdown_connections 2024-04-25T11:36:50.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.941+0000 7effebbc9740 1 --2- 172.21.15.90:0/2756656136 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563e62c86390 0x563e6302c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.941+0000 7effebbc9740 1 --2- 172.21.15.90:0/2756656136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e62c656b0 0x563e63038820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.941+0000 7effebbc9740 1 --2- 172.21.15.90:0/2756656136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c5d4d0 0x563e630312a0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.941+0000 7effebbc9740 1 -- 172.21.15.90:0/2756656136 >> 172.21.15.90:0/2756656136 conn(0x563e62c7a4e0 msgr2=0x563e63022610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:50.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.943+0000 7effebbc9740 1 -- 172.21.15.90:0/2756656136 shutdown_connections 2024-04-25T11:36:50.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.944+0000 7effebbc9740 1 -- 172.21.15.90:0/2756656136 wait complete. 2024-04-25T11:36:50.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.944+0000 7effebbc9740 1 -- start start 2024-04-25T11:36:50.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.945+0000 7effebbc9740 1 Processor -- start 2024-04-25T11:36:50.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.945+0000 7effebbc9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e62c5d4d0 0x563e63038830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.945+0000 7effebbc9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c656b0 0x563e63034be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.946+0000 7effebbc9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563e62c86390 0x563e63035120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.946+0000 7effebbc9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7effe01aef60 con 0x563e62c656b0 2024-04-25T11:36:50.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.946+0000 7effebbc9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7effe01ae890 con 0x563e62c86390 2024-04-25T11:36:50.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.946+0000 7effebbc9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7effe01abad0 con 0x563e62c5d4d0 2024-04-25T11:36:50.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.946+0000 7effebbc9740 10 client.0 _ll_register_callbacks cb 0x7effd0009a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:50.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.946+0000 7effd7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c656b0 0x563e63034be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.946+0000 7effd7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c656b0 0x563e63034be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52640/0 (socket says 172.21.15.90:52640) 2024-04-25T11:36:50.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.946+0000 7effd7fff640 1 -- 172.21.15.90:0/1739219408 learned_addr learned my addr 172.21.15.90:0/1739219408 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:50.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.949+0000 7effde59c640 1 --2- 172.21.15.90:0/1739219408 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e62c5d4d0 0x563e63038830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.949+0000 7effded9d640 1 --2- 172.21.15.90:0/1739219408 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563e62c86390 0x563e63035120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.955+0000 7effde59c640 1 -- 172.21.15.90:0/1739219408 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563e62c86390 msgr2=0x563e63035120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:50.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.955+0000 7effde59c640 1 --2- 172.21.15.90:0/1739219408 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563e62c86390 0x563e63035120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.956+0000 7effde59c640 1 -- 172.21.15.90:0/1739219408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c656b0 msgr2=0x563e63034be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:50.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.956+0000 7effde59c640 1 --2- 172.21.15.90:0/1739219408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c656b0 0x563e63034be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.956+0000 7effde59c640 1 -- 172.21.15.90:0/1739219408 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e63035660 con 0x563e62c5d4d0 2024-04-25T11:36:50.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.956+0000 7effded9d640 1 --2- 172.21.15.90:0/1739219408 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563e62c86390 0x563e63035120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:36:50.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.959+0000 7effde59c640 1 --2- 172.21.15.90:0/1739219408 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e62c5d4d0 0x563e63038830 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7effd0000c00 tx=0x7effd0010920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f9e2c74f47a947 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:50.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.959+0000 7effc5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:36:50.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.966+0000 7effc5ffb640 1 -- 172.21.15.90:0/1739219408 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd0041070 con 0x563e62c5d4d0 2024-04-25T11:36:50.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.966+0000 7effc5ffb640 1 -- 172.21.15.90:0/1739219408 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7effd0010cc0 con 0x563e62c5d4d0 2024-04-25T11:36:50.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.971+0000 7effebbc9740 1 -- 172.21.15.90:0/1739219408 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7effd0027070 con 0x563e62c5d4d0 2024-04-25T11:36:50.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.975+0000 7effc5ffb640 1 -- 172.21.15.90:0/1739219408 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7effd0038040 con 0x563e62c5d4d0 2024-04-25T11:36:50.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.976+0000 7effebbc9740 10 client.25519 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:50.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.976+0000 7effebbc9740 1 -- 172.21.15.90:0/1739219408 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563e6303e300 con 0x563e62c5d4d0 2024-04-25T11:36:50.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.978+0000 7effc5ffb640 1 -- 172.21.15.90:0/1739219408 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effd0027070 con 0x563e62c5d4d0 2024-04-25T11:36:50.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.978+0000 7effc5ffb640 1 -- 172.21.15.90:0/1739219408 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7effd0044020 con 0x563e62c5d4d0 2024-04-25T11:36:50.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.980+0000 7effebbc9740 10 client.25519 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:50.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.981+0000 7effebbc9740 10 client.25519 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:36:50.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.981+0000 7effebbc9740 20 client.25519 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:50.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.981+0000 7effebbc9740 1 --2- 172.21.15.90:0/1739219408 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x563e62c1a3a0 0x563e6303ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:50.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.981+0000 7effebbc9740 4 client.25519 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:36:50.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.981+0000 7effebbc9740 1 -- 172.21.15.90:0/1739219408 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563e62f94800 con 0x563e62c1a3a0 2024-04-25T11:36:50.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.985+0000 7effd7fff640 1 --2- 172.21.15.90:0/1739219408 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x563e62c1a3a0 0x563e6303ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:50.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.990+0000 7effd7fff640 1 --2- 172.21.15.90:0/1739219408 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x563e62c1a3a0 0x563e6303ea20 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=afbb05b47ffdc51f server_cookie=6d4aee196f819839 in_seq=0 out_seq=0 2024-04-25T11:36:50.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.990+0000 7effc5ffb640 10 client.25519 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:36:50.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.992+0000 7effc5ffb640 1 -- 172.21.15.90:0/1739219408 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563e62f94800 con 0x563e62c1a3a0 2024-04-25T11:36:50.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.992+0000 7effc5ffb640 10 client.25519 handle_command_reply: tid=0 2024-04-25T11:36:50.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.992+0000 7effebbc9740 1 client.25519 shutdown 2024-04-25T11:36:50.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.993+0000 7effebbc9740 1 -- 172.21.15.90:0/1739219408 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e62c5d4d0 msgr2=0x563e63038830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.993+0000 7effebbc9740 1 --2- 172.21.15.90:0/1739219408 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e62c5d4d0 0x563e63038830 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7effd0000c00 tx=0x7effd0010920 comp rx=0 tx=0).stop 2024-04-25T11:36:50.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.993+0000 7effebbc9740 1 -- 172.21.15.90:0/1739219408 shutdown_connections 2024-04-25T11:36:50.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.993+0000 7effebbc9740 1 --2- 172.21.15.90:0/1739219408 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x563e62c1a3a0 0x563e6303ea20 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.993+0000 7effebbc9740 1 --2- 172.21.15.90:0/1739219408 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563e62c86390 0x563e63035120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.993+0000 7effc5ffb640 0 client.25519 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:36:50.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.993+0000 7effebbc9740 1 --2- 172.21.15.90:0/1739219408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563e62c656b0 0x563e63034be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.993+0000 7effebbc9740 1 --2- 172.21.15.90:0/1739219408 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e62c5d4d0 0x563e63038830 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.993+0000 7effebbc9740 1 -- 172.21.15.90:0/1739219408 >> 172.21.15.90:0/1739219408 conn(0x563e62c7a4e0 msgr2=0x563e63031000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:50.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.993+0000 7effebbc9740 1 -- 172.21.15.90:0/1739219408 shutdown_connections 2024-04-25T11:36:50.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.994+0000 7effebbc9740 1 -- 172.21.15.90:0/1739219408 wait complete. 2024-04-25T11:36:50.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.994+0000 7effebbc9740 20 client.25519 trim_cache size 0 max 16384 2024-04-25T11:36:50.985 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:36:50.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.995+0000 7effe4d14640 1 -- 172.21.15.90:0/1020064491 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7effcc0afce0 msgr2=0x7effcc0b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.995+0000 7effe4d14640 1 --2- 172.21.15.90:0/1020064491 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7effcc0afce0 0x7effcc0b21f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7effd802ae00 tx=0x7effd8005c50 comp rx=0 tx=0).stop 2024-04-25T11:36:50.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.995+0000 7effe4d14640 1 -- 172.21.15.90:0/1020064491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effe0108a20 msgr2=0x7effe019acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.995+0000 7effe4d14640 1 --2- 172.21.15.90:0/1020064491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effe0108a20 0x7effe019acd0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7effc8006fd0 tx=0x7effc800b8e0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.995+0000 7effe4d14640 1 -- 172.21.15.90:0/1020064491 shutdown_connections 2024-04-25T11:36:50.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.995+0000 7effe4d14640 1 --2- 172.21.15.90:0/1020064491 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7effcc0afce0 0x7effcc0b21f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.995+0000 7effe4d14640 1 --2- 172.21.15.90:0/1020064491 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7effe010d380 0x7effe019ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.995+0000 7effe4d14640 1 --2- 172.21.15.90:0/1020064491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effe0108a20 0x7effe019acd0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.995+0000 7effe4d14640 1 --2- 172.21.15.90:0/1020064491 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7effe0102af0 0x7effe019a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.995+0000 7effe4d14640 1 -- 172.21.15.90:0/1020064491 >> 172.21.15.90:0/1020064491 conn(0x7effe0078100 msgr2=0x7effe0069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:50.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.995+0000 7effe4d14640 1 -- 172.21.15.90:0/1020064491 shutdown_connections 2024-04-25T11:36:50.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:50.995+0000 7effe4d14640 1 -- 172.21.15.90:0/1020064491 wait complete. 2024-04-25T11:36:51.000 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T11:36:51.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:50 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/566066618' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:36:51.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.212+0000 7ff61359e640 1 -- 172.21.15.90:0/2580700992 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff6141070e0 msgr2=0x7ff6141074f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:51.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.212+0000 7ff61359e640 1 --2- 172.21.15.90:0/2580700992 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff6141070e0 0x7ff6141074f0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7ff60c009be0 tx=0x7ff60c02a750 comp rx=0 tx=0).stop 2024-04-25T11:36:51.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.212+0000 7ff61359e640 1 -- 172.21.15.90:0/2580700992 shutdown_connections 2024-04-25T11:36:51.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.212+0000 7ff61359e640 1 --2- 172.21.15.90:0/2580700992 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff6141070e0 0x7ff6141074f0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.212+0000 7ff61359e640 1 --2- 172.21.15.90:0/2580700992 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff614108c50 0x7ff614101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.212+0000 7ff61359e640 1 --2- 172.21.15.90:0/2580700992 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff614108840 0x7ff614101950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.212+0000 7ff61359e640 1 -- 172.21.15.90:0/2580700992 >> 172.21.15.90:0/2580700992 conn(0x7ff6140fd800 msgr2=0x7ff6140ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:51.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.213+0000 7ff61359e640 1 -- 172.21.15.90:0/2580700992 shutdown_connections 2024-04-25T11:36:51.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.213+0000 7ff61359e640 1 -- 172.21.15.90:0/2580700992 wait complete. 2024-04-25T11:36:51.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.215+0000 7ff61359e640 1 Processor -- start 2024-04-25T11:36:51.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.216+0000 7ff61359e640 1 -- start start 2024-04-25T11:36:51.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.217+0000 7ff61359e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff6141070e0 0x7ff6141a67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:51.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.217+0000 7ff61359e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff614108840 0x7ff6141a6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:51.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.217+0000 7ff61359e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff614108c50 0x7ff6141a09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:51.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.217+0000 7ff61359e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff61411a140 con 0x7ff614108840 2024-04-25T11:36:51.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.217+0000 7ff61359e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff614119fc0 con 0x7ff6141070e0 2024-04-25T11:36:51.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.217+0000 7ff61359e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff61411a2c0 con 0x7ff614108c50 2024-04-25T11:36:51.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.217+0000 7ff611d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff614108840 0x7ff6141a6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:51.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.217+0000 7ff611d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff614108840 0x7ff6141a6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52670/0 (socket says 172.21.15.90:52670) 2024-04-25T11:36:51.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.217+0000 7ff611d9b640 1 -- 172.21.15.90:0/2609454700 learned_addr learned my addr 172.21.15.90:0/2609454700 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:51.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.218+0000 7ff611d9b640 1 -- 172.21.15.90:0/2609454700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff614108c50 msgr2=0x7ff6141a09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:51.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.218+0000 7ff611d9b640 1 --2- 172.21.15.90:0/2609454700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff614108c50 0x7ff6141a09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.218+0000 7ff611d9b640 1 -- 172.21.15.90:0/2609454700 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff6141070e0 msgr2=0x7ff6141a67b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:36:51.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.218+0000 7ff611d9b640 1 --2- 172.21.15.90:0/2609454700 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff6141070e0 0x7ff6141a67b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.218+0000 7ff611d9b640 1 -- 172.21.15.90:0/2609454700 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6141a1060 con 0x7ff614108840 2024-04-25T11:36:51.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.218+0000 7ff611d9b640 1 --2- 172.21.15.90:0/2609454700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff614108840 0x7ff6141a6cf0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff60800e940 tx=0x7ff60800ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:51.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.219+0000 7ff6037fe640 1 -- 172.21.15.90:0/2609454700 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60800ccc0 con 0x7ff614108840 2024-04-25T11:36:51.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.219+0000 7ff6037fe640 1 -- 172.21.15.90:0/2609454700 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6080054d0 con 0x7ff614108840 2024-04-25T11:36:51.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.219+0000 7ff6037fe640 1 -- 172.21.15.90:0/2609454700 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff608010430 con 0x7ff614108840 2024-04-25T11:36:51.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.219+0000 7ff61359e640 1 -- 172.21.15.90:0/2609454700 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6141a1350 con 0x7ff614108840 2024-04-25T11:36:51.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.219+0000 7ff61359e640 1 -- 172.21.15.90:0/2609454700 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6141ad630 con 0x7ff614108840 2024-04-25T11:36:51.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.223+0000 7ff61ff7b740 1 Processor -- start 2024-04-25T11:36:51.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.223+0000 7ff61ff7b740 1 -- start start 2024-04-25T11:36:51.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.223+0000 7ff61ff7b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55626380b390 0x556263bac9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:51.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.223+0000 7ff61ff7b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562637e24d0 0x556263bacf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:51.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.223+0000 7ff61ff7b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5562637ea6b0 0x556263bb9870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:51.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.224+0000 7ff61ff7b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556263b11d80 con 0x5562637ea6b0 2024-04-25T11:36:51.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.224+0000 7ff61ff7b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x556263acba20 con 0x55626380b390 2024-04-25T11:36:51.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.224+0000 7ff61ff7b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x556263b0dd40 con 0x5562637e24d0 2024-04-25T11:36:51.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.224+0000 7ff6037fe640 1 -- 172.21.15.90:0/2609454700 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7ff6080105d0 con 0x7ff614108840 2024-04-25T11:36:51.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.224+0000 7ff611d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5562637ea6b0 0x556263bb9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:51.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.224+0000 7ff611d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5562637ea6b0 0x556263bb9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52678/0 (socket says 172.21.15.90:52678) 2024-04-25T11:36:51.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.224+0000 7ff611d9b640 1 -- 172.21.15.90:0/1752303153 learned_addr learned my addr 172.21.15.90:0/1752303153 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:51.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.225+0000 7ff611d9b640 1 -- 172.21.15.90:0/1752303153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562637e24d0 msgr2=0x556263bacf20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:36:51.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.225+0000 7ff611d9b640 1 --2- 172.21.15.90:0/1752303153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562637e24d0 0x556263bacf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.225+0000 7ff611d9b640 1 -- 172.21.15.90:0/1752303153 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55626380b390 msgr2=0x556263bac9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:36:51.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.225+0000 7ff611d9b640 1 --2- 172.21.15.90:0/1752303153 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55626380b390 0x556263bac9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.225+0000 7ff611d9b640 1 -- 172.21.15.90:0/1752303153 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556263ad6c30 con 0x5562637ea6b0 2024-04-25T11:36:51.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.225+0000 7ff6037fe640 1 --2- 172.21.15.90:0/2609454700 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff5fc0afde0 0x7ff5fc0b22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:51.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.225+0000 7ff6037fe640 1 -- 172.21.15.90:0/2609454700 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7ff6080df4e0 con 0x7ff614108840 2024-04-25T11:36:51.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.225+0000 7ff611d9b640 1 --2- 172.21.15.90:0/1752303153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5562637ea6b0 0x556263bb9870 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ff6080dcb40 tx=0x7ff6080e5aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab4e192afb770456 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:51.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.226+0000 7ff612d9d640 1 --2- 172.21.15.90:0/2609454700 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff5fc0afde0 0x7ff5fc0b22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:51.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.226+0000 7ff600ff9640 1 -- 172.21.15.90:0/1752303153 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6080a34e0 con 0x5562637ea6b0 2024-04-25T11:36:51.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.226+0000 7ff600ff9640 1 -- 172.21.15.90:0/1752303153 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6080046c0 con 0x5562637ea6b0 2024-04-25T11:36:51.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.226+0000 7ff600ff9640 1 -- 172.21.15.90:0/1752303153 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6080a6d90 con 0x5562637ea6b0 2024-04-25T11:36:51.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.227+0000 7ff612d9d640 1 --2- 172.21.15.90:0/2609454700 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff5fc0afde0 0x7ff5fc0b22f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff60c0099a0 tx=0x7ff60c0093a0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:51.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.227+0000 7ff61ff7b740 1 -- 172.21.15.90:0/1752303153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5562637ea6b0 msgr2=0x556263bb9870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:51.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.227+0000 7ff61ff7b740 1 --2- 172.21.15.90:0/1752303153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5562637ea6b0 0x556263bb9870 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ff6080dcb40 tx=0x7ff6080e5aa0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.227+0000 7ff61ff7b740 1 -- 172.21.15.90:0/1752303153 shutdown_connections 2024-04-25T11:36:51.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.227+0000 7ff61ff7b740 1 --2- 172.21.15.90:0/1752303153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5562637ea6b0 0x556263bb9870 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.227+0000 7ff61ff7b740 1 --2- 172.21.15.90:0/1752303153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562637e24d0 0x556263bacf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.227+0000 7ff61ff7b740 1 --2- 172.21.15.90:0/1752303153 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55626380b390 0x556263bac9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.227+0000 7ff61ff7b740 1 -- 172.21.15.90:0/1752303153 >> 172.21.15.90:0/1752303153 conn(0x5562637ff4e0 msgr2=0x556263ba9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:51.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.228+0000 7ff61ff7b740 1 -- 172.21.15.90:0/1752303153 shutdown_connections 2024-04-25T11:36:51.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.228+0000 7ff61ff7b740 1 -- 172.21.15.90:0/1752303153 wait complete. 2024-04-25T11:36:51.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.228+0000 7ff61ff7b740 1 -- start start 2024-04-25T11:36:51.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.229+0000 7ff61ff7b740 1 Processor -- start 2024-04-25T11:36:51.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.229+0000 7ff61ff7b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562637e24d0 0x556263bb73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:51.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.229+0000 7ff61ff7b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5562637ea6b0 0x556263bb7920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:51.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.229+0000 7ff61ff7b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55626380b390 0x556263bd44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:51.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.229+0000 7ff61ff7b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556263bcaf70 con 0x55626380b390 2024-04-25T11:36:51.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.229+0000 7ff61ff7b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x556263a98dc0 con 0x5562637ea6b0 2024-04-25T11:36:51.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.229+0000 7ff61ff7b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x556263bcb0f0 con 0x5562637e24d0 2024-04-25T11:36:51.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.229+0000 7ff61ff7b740 10 client.0 _ll_register_callbacks cb 0x556263ab4600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:51.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.229+0000 7ff612d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55626380b390 0x556263bd44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:51.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.229+0000 7ff612d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55626380b390 0x556263bd44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52684/0 (socket says 172.21.15.90:52684) 2024-04-25T11:36:51.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.229+0000 7ff612d9d640 1 -- 172.21.15.90:0/2039861194 learned_addr learned my addr 172.21.15.90:0/2039861194 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:51.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.229+0000 7ff612d9d640 1 -- 172.21.15.90:0/2039861194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562637e24d0 msgr2=0x556263bb73e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:36:51.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.229+0000 7ff612d9d640 1 --2- 172.21.15.90:0/2039861194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562637e24d0 0x556263bb73e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.229+0000 7ff612d9d640 1 -- 172.21.15.90:0/2039861194 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5562637ea6b0 msgr2=0x556263bb7920 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:36:51.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.229+0000 7ff612d9d640 1 --2- 172.21.15.90:0/2039861194 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5562637ea6b0 0x556263bb7920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.230+0000 7ff612d9d640 1 -- 172.21.15.90:0/2039861194 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562638233f0 con 0x55626380b390 2024-04-25T11:36:51.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.230+0000 7ff612d9d640 1 --2- 172.21.15.90:0/2039861194 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55626380b390 0x556263bd44b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7ff60c009be0 tx=0x7ff60c009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a85d165a4d82388 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:51.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.230+0000 7ff5f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:51.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.231+0000 7ff5f9ffb640 1 -- 172.21.15.90:0/2039861194 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60c043050 con 0x55626380b390 2024-04-25T11:36:51.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.231+0000 7ff5f9ffb640 1 -- 172.21.15.90:0/2039861194 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7ff60c004240 con 0x55626380b390 2024-04-25T11:36:51.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.231+0000 7ff61ff7b740 1 -- 172.21.15.90:0/2039861194 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556263b1f990 con 0x55626380b390 2024-04-25T11:36:51.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.232+0000 7ff5f9ffb640 1 -- 172.21.15.90:0/2039861194 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff60c036040 con 0x55626380b390 2024-04-25T11:36:51.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.232+0000 7ff61ff7b740 10 client.16134 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:51.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.232+0000 7ff61ff7b740 1 -- 172.21.15.90:0/2039861194 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5562638ed7d0 con 0x55626380b390 2024-04-25T11:36:51.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.232+0000 7ff5f9ffb640 1 -- 172.21.15.90:0/2039861194 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x556263b1f990 con 0x55626380b390 2024-04-25T11:36:51.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.232+0000 7ff5f9ffb640 1 -- 172.21.15.90:0/2039861194 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7ff60c03c9f0 con 0x55626380b390 2024-04-25T11:36:51.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.232+0000 7ff61ff7b740 10 client.16134 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:51.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.232+0000 7ff61ff7b740 10 client.16134 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:36:51.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.232+0000 7ff61ff7b740 20 client.16134 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:51.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.232+0000 7ff61ff7b740 1 --2- 172.21.15.90:0/2039861194 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5562637c9f60 0x556263bdaa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:51.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.232+0000 7ff61ff7b740 4 client.16134 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:36:51.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.232+0000 7ff61ff7b740 1 -- 172.21.15.90:0/2039861194 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556263b19800 con 0x5562637c9f60 2024-04-25T11:36:51.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.235+0000 7ff61259c640 1 --2- 172.21.15.90:0/2039861194 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5562637c9f60 0x556263bdaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:51.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.243+0000 7ff61259c640 1 --2- 172.21.15.90:0/2039861194 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5562637c9f60 0x556263bdaa10 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e43ac14409e565e6 server_cookie=a964a910799f0b37 in_seq=0 out_seq=0 2024-04-25T11:36:51.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.243+0000 7ff5f9ffb640 10 client.16134 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:51.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.252+0000 7ff5f9ffb640 1 -- 172.21.15.90:0/2039861194 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556263b19800 con 0x5562637c9f60 2024-04-25T11:36:51.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.252+0000 7ff5f9ffb640 10 client.16134 handle_command_reply: tid=0 2024-04-25T11:36:51.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.252+0000 7ff61ff7b740 1 client.16134 shutdown 2024-04-25T11:36:51.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.252+0000 7ff61ff7b740 1 -- 172.21.15.90:0/2039861194 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55626380b390 msgr2=0x556263bd44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:51.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.252+0000 7ff61ff7b740 1 --2- 172.21.15.90:0/2039861194 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55626380b390 0x556263bd44b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7ff60c009be0 tx=0x7ff60c009500 comp rx=0 tx=0).stop 2024-04-25T11:36:51.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.252+0000 7ff61ff7b740 1 -- 172.21.15.90:0/2039861194 shutdown_connections 2024-04-25T11:36:51.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.252+0000 7ff61ff7b740 1 --2- 172.21.15.90:0/2039861194 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5562637c9f60 0x556263bdaa10 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.252+0000 7ff61ff7b740 1 --2- 172.21.15.90:0/2039861194 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55626380b390 0x556263bd44b0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.252+0000 7ff61ff7b740 1 --2- 172.21.15.90:0/2039861194 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5562637ea6b0 0x556263bb7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.252+0000 7ff61ff7b740 1 --2- 172.21.15.90:0/2039861194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562637e24d0 0x556263bb73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.252+0000 7ff61ff7b740 1 -- 172.21.15.90:0/2039861194 >> 172.21.15.90:0/2039861194 conn(0x5562637ff4e0 msgr2=0x556263ba79c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:51.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.252+0000 7ff5f9ffb640 0 client.16134 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:51.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.253+0000 7ff61ff7b740 1 -- 172.21.15.90:0/2039861194 shutdown_connections 2024-04-25T11:36:51.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.253+0000 7ff61ff7b740 1 -- 172.21.15.90:0/2039861194 wait complete. 2024-04-25T11:36:51.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.253+0000 7ff61ff7b740 20 client.16134 trim_cache size 0 max 16384 2024-04-25T11:36:51.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.272+0000 7ff61ff7b740 1 Processor -- start 2024-04-25T11:36:51.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.272+0000 7ff61ff7b740 1 -- start start 2024-04-25T11:36:51.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.272+0000 7ff61ff7b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5562637e24d0 0x556263bd4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:51.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.272+0000 7ff61ff7b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5562637ea6b0 0x556263bb1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:51.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.273+0000 7ff61ff7b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55626380b390 0x556263be0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:51.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.273+0000 7ff61ff7b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff6141ad880 con 0x5562637e24d0 2024-04-25T11:36:51.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.273+0000 7ff61ff7b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff6141af3a0 con 0x5562637ea6b0 2024-04-25T11:36:51.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.273+0000 7ff61ff7b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff6141adf30 con 0x55626380b390 2024-04-25T11:36:51.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.273+0000 7ff61259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5562637e24d0 0x556263bd4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:51.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.273+0000 7ff61259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5562637e24d0 0x556263bd4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52698/0 (socket says 172.21.15.90:52698) 2024-04-25T11:36:51.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.273+0000 7ff61259c640 1 -- 172.21.15.90:0/2988084664 learned_addr learned my addr 172.21.15.90:0/2988084664 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:51.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.273+0000 7ff61259c640 1 -- 172.21.15.90:0/2988084664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55626380b390 msgr2=0x556263be0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:51.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.273+0000 7ff61259c640 1 --2- 172.21.15.90:0/2988084664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55626380b390 0x556263be0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.273+0000 7ff61259c640 1 -- 172.21.15.90:0/2988084664 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5562637ea6b0 msgr2=0x556263bb1600 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:36:51.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.273+0000 7ff61259c640 1 --2- 172.21.15.90:0/2988084664 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5562637ea6b0 0x556263bb1600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.273+0000 7ff61259c640 1 -- 172.21.15.90:0/2988084664 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556263bad080 con 0x5562637e24d0 2024-04-25T11:36:51.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.274+0000 7ff61259c640 1 --2- 172.21.15.90:0/2988084664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5562637e24d0 0x556263bd4090 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7ff6040058c0 tx=0x7ff604005ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c6b932bc3a6ffc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:51.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.274+0000 7ff5fbfff640 1 -- 172.21.15.90:0/2988084664 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff604004230 con 0x5562637e24d0 2024-04-25T11:36:51.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.274+0000 7ff5fbfff640 1 -- 172.21.15.90:0/2988084664 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff604011e10 con 0x5562637e24d0 2024-04-25T11:36:51.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.274+0000 7ff5fbfff640 1 -- 172.21.15.90:0/2988084664 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6040035f0 con 0x5562637e24d0 2024-04-25T11:36:51.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.275+0000 7ff61ff7b740 1 -- 172.21.15.90:0/2988084664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5562637e24d0 msgr2=0x556263bd4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:51.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.275+0000 7ff61ff7b740 1 --2- 172.21.15.90:0/2988084664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5562637e24d0 0x556263bd4090 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7ff6040058c0 tx=0x7ff604005ba0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.276+0000 7ff61ff7b740 1 -- 172.21.15.90:0/2988084664 shutdown_connections 2024-04-25T11:36:51.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.276+0000 7ff61ff7b740 1 --2- 172.21.15.90:0/2988084664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55626380b390 0x556263be0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.276+0000 7ff61ff7b740 1 --2- 172.21.15.90:0/2988084664 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5562637ea6b0 0x556263bb1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.276+0000 7ff61ff7b740 1 --2- 172.21.15.90:0/2988084664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5562637e24d0 0x556263bd4090 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.276+0000 7ff61ff7b740 1 -- 172.21.15.90:0/2988084664 >> 172.21.15.90:0/2988084664 conn(0x5562637ff4e0 msgr2=0x556263bb5aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:51.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.276+0000 7ff61ff7b740 1 -- 172.21.15.90:0/2988084664 shutdown_connections 2024-04-25T11:36:51.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.276+0000 7ff61ff7b740 1 -- 172.21.15.90:0/2988084664 wait complete. 2024-04-25T11:36:51.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.276+0000 7ff61ff7b740 1 -- start start 2024-04-25T11:36:51.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.277+0000 7ff61ff7b740 1 Processor -- start 2024-04-25T11:36:51.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.277+0000 7ff61ff7b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5562637e24d0 0x556263c3b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:51.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.277+0000 7ff61ff7b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5562637ea6b0 0x556263c3b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:51.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.277+0000 7ff61ff7b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55626380b390 0x556263bcb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:51.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.277+0000 7ff61ff7b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff6141aecd0 con 0x5562637ea6b0 2024-04-25T11:36:51.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.277+0000 7ff61ff7b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff6141ae600 con 0x5562637e24d0 2024-04-25T11:36:51.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.277+0000 7ff61ff7b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff6140691b0 con 0x55626380b390 2024-04-25T11:36:51.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.277+0000 7ff611d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5562637ea6b0 0x556263c3b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:51.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.277+0000 7ff611d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5562637ea6b0 0x556263c3b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52706/0 (socket says 172.21.15.90:52706) 2024-04-25T11:36:51.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.277+0000 7ff611d9b640 1 -- 172.21.15.90:0/2096156777 learned_addr learned my addr 172.21.15.90:0/2096156777 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:51.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.278+0000 7ff61ff7b740 10 client.0 _ll_register_callbacks cb 0x7ff60c034070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:51.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.278+0000 7ff611d9b640 1 -- 172.21.15.90:0/2096156777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55626380b390 msgr2=0x556263bcb9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:36:51.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.278+0000 7ff611d9b640 1 --2- 172.21.15.90:0/2096156777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55626380b390 0x556263bcb9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.278+0000 7ff611d9b640 1 -- 172.21.15.90:0/2096156777 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5562637e24d0 msgr2=0x556263c3b1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:36:51.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.278+0000 7ff611d9b640 1 --2- 172.21.15.90:0/2096156777 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5562637e24d0 0x556263c3b1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.278+0000 7ff611d9b640 1 -- 172.21.15.90:0/2096156777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556263bcbfc0 con 0x5562637ea6b0 2024-04-25T11:36:51.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.278+0000 7ff611d9b640 1 --2- 172.21.15.90:0/2096156777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5562637ea6b0 0x556263c3b710 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7ff60800cfd0 tx=0x7ff6080a7ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a1c6c5c97ac4cea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:51.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.278+0000 7ff5f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:51.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.278+0000 7ff5f9ffb640 1 -- 172.21.15.90:0/2096156777 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6080a6f70 con 0x5562637ea6b0 2024-04-25T11:36:51.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.278+0000 7ff5f9ffb640 1 -- 172.21.15.90:0/2096156777 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7ff6080b0200 con 0x5562637ea6b0 2024-04-25T11:36:51.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.279+0000 7ff61ff7b740 1 -- 172.21.15.90:0/2096156777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff604011e10 con 0x5562637ea6b0 2024-04-25T11:36:51.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.279+0000 7ff5f9ffb640 1 -- 172.21.15.90:0/2096156777 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff6080a73f0 con 0x5562637ea6b0 2024-04-25T11:36:51.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.279+0000 7ff61ff7b740 10 client.16140 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:51.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.279+0000 7ff61ff7b740 1 -- 172.21.15.90:0/2096156777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x556263c427f0 con 0x5562637ea6b0 2024-04-25T11:36:51.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.279+0000 7ff5f9ffb640 1 -- 172.21.15.90:0/2096156777 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff604011e10 con 0x5562637ea6b0 2024-04-25T11:36:51.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.279+0000 7ff5f9ffb640 1 -- 172.21.15.90:0/2096156777 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7ff6080108c0 con 0x5562637ea6b0 2024-04-25T11:36:51.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.279+0000 7ff61ff7b740 10 client.16140 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:51.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.279+0000 7ff61ff7b740 10 client.16140 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:36:51.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.279+0000 7ff61ff7b740 20 client.16140 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:51.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.279+0000 7ff61ff7b740 1 --2- 172.21.15.90:0/2096156777 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55626379f3a0 0x556263c44ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:51.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.279+0000 7ff61ff7b740 4 client.16140 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:36:51.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.279+0000 7ff61ff7b740 1 -- 172.21.15.90:0/2096156777 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556263a94610 con 0x55626379f3a0 2024-04-25T11:36:51.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.283+0000 7ff61259c640 1 --2- 172.21.15.90:0/2096156777 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55626379f3a0 0x556263c44ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:51.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.288+0000 7ff5f9ffb640 10 client.16140 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:51.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.288+0000 7ff61259c640 1 --2- 172.21.15.90:0/2096156777 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55626379f3a0 0x556263c44ad0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aafe2fcb7ff1afe9 server_cookie=2327491e43c62407 in_seq=0 out_seq=0 2024-04-25T11:36:51.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.293+0000 7ff5f9ffb640 1 -- 172.21.15.90:0/2096156777 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556263a94610 con 0x55626379f3a0 2024-04-25T11:36:51.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.293+0000 7ff5f9ffb640 10 client.16140 handle_command_reply: tid=0 2024-04-25T11:36:51.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.293+0000 7ff61ff7b740 1 client.16140 shutdown 2024-04-25T11:36:51.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.294+0000 7ff61ff7b740 1 -- 172.21.15.90:0/2096156777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5562637ea6b0 msgr2=0x556263c3b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:51.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.294+0000 7ff61ff7b740 1 --2- 172.21.15.90:0/2096156777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5562637ea6b0 0x556263c3b710 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7ff60800cfd0 tx=0x7ff6080a7ab0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.294+0000 7ff61ff7b740 1 -- 172.21.15.90:0/2096156777 shutdown_connections 2024-04-25T11:36:51.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.294+0000 7ff61ff7b740 1 --2- 172.21.15.90:0/2096156777 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55626379f3a0 0x556263c44ad0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.294+0000 7ff61ff7b740 1 --2- 172.21.15.90:0/2096156777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55626380b390 0x556263bcb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.294+0000 7ff61ff7b740 1 --2- 172.21.15.90:0/2096156777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5562637ea6b0 0x556263c3b710 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.294+0000 7ff61ff7b740 1 --2- 172.21.15.90:0/2096156777 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5562637e24d0 0x556263c3b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.294+0000 7ff61ff7b740 1 -- 172.21.15.90:0/2096156777 >> 172.21.15.90:0/2096156777 conn(0x5562637ff4e0 msgr2=0x556263ba8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:51.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.294+0000 7ff5f9ffb640 0 client.16140 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:51.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.294+0000 7ff61ff7b740 1 -- 172.21.15.90:0/2096156777 shutdown_connections 2024-04-25T11:36:51.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.294+0000 7ff61ff7b740 1 -- 172.21.15.90:0/2096156777 wait complete. 2024-04-25T11:36:51.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.294+0000 7ff61ff7b740 20 client.16140 trim_cache size 0 max 16384 2024-04-25T11:36:51.284 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:36:51.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.295+0000 7ff61359e640 1 -- 172.21.15.90:0/2609454700 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff5fc0afde0 msgr2=0x7ff5fc0b22f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:51.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.295+0000 7ff61359e640 1 --2- 172.21.15.90:0/2609454700 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff5fc0afde0 0x7ff5fc0b22f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff60c0099a0 tx=0x7ff60c0093a0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.295+0000 7ff61359e640 1 -- 172.21.15.90:0/2609454700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff614108840 msgr2=0x7ff6141a6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:51.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.295+0000 7ff61359e640 1 --2- 172.21.15.90:0/2609454700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff614108840 0x7ff6141a6cf0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff60800e940 tx=0x7ff60800ee00 comp rx=0 tx=0).stop 2024-04-25T11:36:51.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.295+0000 7ff61359e640 1 -- 172.21.15.90:0/2609454700 shutdown_connections 2024-04-25T11:36:51.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.295+0000 7ff61359e640 1 --2- 172.21.15.90:0/2609454700 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff5fc0afde0 0x7ff5fc0b22f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.295+0000 7ff61359e640 1 --2- 172.21.15.90:0/2609454700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff614108c50 0x7ff6141a09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.295+0000 7ff61359e640 1 --2- 172.21.15.90:0/2609454700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff614108840 0x7ff6141a6cf0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.295+0000 7ff61359e640 1 --2- 172.21.15.90:0/2609454700 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff6141070e0 0x7ff6141a67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:51.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.295+0000 7ff61359e640 1 -- 172.21.15.90:0/2609454700 >> 172.21.15.90:0/2609454700 conn(0x7ff6140fd800 msgr2=0x7ff6141050b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:51.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.296+0000 7ff61359e640 1 -- 172.21.15.90:0/2609454700 shutdown_connections 2024-04-25T11:36:51.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:51.296+0000 7ff61359e640 1 -- 172.21.15.90:0/2609454700 wait complete. 2024-04-25T11:36:51.979 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:51 smithi090 ceph-mon[29575]: pgmap v56: 97 pgs: 97 active+clean; 1.4 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 121 MiB/s wr, 101 op/s 2024-04-25T11:36:52.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:51 smithi138 ceph-mon[32328]: pgmap v56: 97 pgs: 97 active+clean; 1.4 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 121 MiB/s wr, 101 op/s 2024-04-25T11:36:52.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:51 smithi156 ceph-mon[33045]: pgmap v56: 97 pgs: 97 active+clean; 1.4 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 121 MiB/s wr, 101 op/s 2024-04-25T11:36:52.301 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T11:36:52.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:36:52.301 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T11:36:52.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.546+0000 7f9efc34e640 1 -- 172.21.15.90:0/1773442631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ef4108a20 msgr2=0x7f9ef41025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:52.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.546+0000 7f9efc34e640 1 --2- 172.21.15.90:0/1773442631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ef4108a20 0x7f9ef41025b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f9eec009900 tx=0x7f9eec02a440 comp rx=0 tx=0).stop 2024-04-25T11:36:52.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.546+0000 7f9efc34e640 1 -- 172.21.15.90:0/1773442631 shutdown_connections 2024-04-25T11:36:52.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.546+0000 7f9efc34e640 1 --2- 172.21.15.90:0/1773442631 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9ef410d380 0x7f9ef410f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.546+0000 7f9efc34e640 1 --2- 172.21.15.90:0/1773442631 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ef4102af0 0x7f9ef4102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.546+0000 7f9efc34e640 1 --2- 172.21.15.90:0/1773442631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ef4108a20 0x7f9ef41025b0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.546+0000 7f9efc34e640 1 -- 172.21.15.90:0/1773442631 >> 172.21.15.90:0/1773442631 conn(0x7f9ef4078100 msgr2=0x7f9ef4078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:52.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.546+0000 7f9efc34e640 1 -- 172.21.15.90:0/1773442631 shutdown_connections 2024-04-25T11:36:52.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.546+0000 7f9efc34e640 1 -- 172.21.15.90:0/1773442631 wait complete. 2024-04-25T11:36:52.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.547+0000 7f9efc34e640 1 Processor -- start 2024-04-25T11:36:52.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.547+0000 7f9efc34e640 1 -- start start 2024-04-25T11:36:52.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.547+0000 7f9efc34e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ef4102af0 0x7f9ef41a26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:52.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.547+0000 7f9efc34e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ef4108a20 0x7f9ef41a2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:52.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.547+0000 7f9efc34e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9ef410d380 0x7f9ef419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:52.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.547+0000 7f9efc34e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef4112ce0 con 0x7f9ef4102af0 2024-04-25T11:36:52.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.547+0000 7f9efc34e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef4112b60 con 0x7f9ef410d380 2024-04-25T11:36:52.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.547+0000 7f9efc34e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef4112e60 con 0x7f9ef4108a20 2024-04-25T11:36:52.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.547+0000 7f9efa0ea640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ef4102af0 0x7f9ef41a26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:52.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.547+0000 7f9efa0ea640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ef4102af0 0x7f9ef41a26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56958/0 (socket says 172.21.15.90:56958) 2024-04-25T11:36:52.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.548+0000 7f9efa0ea640 1 -- 172.21.15.90:0/75396447 learned_addr learned my addr 172.21.15.90:0/75396447 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:52.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.548+0000 7f9efa0ea640 1 -- 172.21.15.90:0/75396447 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ef4108a20 msgr2=0x7f9ef41a2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:52.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.548+0000 7f9efa0ea640 1 --2- 172.21.15.90:0/75396447 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ef4108a20 0x7f9ef41a2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.548+0000 7f9efa0ea640 1 -- 172.21.15.90:0/75396447 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9ef410d380 msgr2=0x7f9ef419c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:52.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.548+0000 7f9efa0ea640 1 --2- 172.21.15.90:0/75396447 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9ef410d380 0x7f9ef419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.548+0000 7f9efa0ea640 1 -- 172.21.15.90:0/75396447 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ef419cf70 con 0x7f9ef4102af0 2024-04-25T11:36:52.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.548+0000 7f9efa0ea640 1 --2- 172.21.15.90:0/75396447 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ef4102af0 0x7f9ef41a26c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f9eec005940 tx=0x7f9eec004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:52.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.549+0000 7f9eeb7fe640 1 -- 172.21.15.90:0/75396447 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eec03f030 con 0x7f9ef4102af0 2024-04-25T11:36:52.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.549+0000 7f9eeb7fe640 1 -- 172.21.15.90:0/75396447 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9eec003aa0 con 0x7f9ef4102af0 2024-04-25T11:36:52.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.549+0000 7f9eeb7fe640 1 -- 172.21.15.90:0/75396447 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eec035070 con 0x7f9ef4102af0 2024-04-25T11:36:52.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.549+0000 7f9efc34e640 1 -- 172.21.15.90:0/75396447 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ef419d200 con 0x7f9ef4102af0 2024-04-25T11:36:52.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.550+0000 7f9efc34e640 1 -- 172.21.15.90:0/75396447 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ef41a9540 con 0x7f9ef4102af0 2024-04-25T11:36:52.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.553+0000 7f9f031fd740 1 Processor -- start 2024-04-25T11:36:52.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.554+0000 7f9f031fd740 1 -- start start 2024-04-25T11:36:52.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.554+0000 7f9eeb7fe640 1 -- 172.21.15.90:0/75396447 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f9eec042030 con 0x7f9ef4102af0 2024-04-25T11:36:52.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.554+0000 7f9f031fd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589c8296410 0x5589c8636620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:52.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.554+0000 7f9f031fd740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5589c826d550 0x5589c8636b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:52.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.554+0000 7f9f031fd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c8275730 0x5589c86370a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:52.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.554+0000 7f9f031fd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5589c859cb00 con 0x5589c8275730 2024-04-25T11:36:52.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.554+0000 7f9f031fd740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5589c8538920 con 0x5589c826d550 2024-04-25T11:36:52.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.554+0000 7f9f031fd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5589c8598ac0 con 0x5589c8296410 2024-04-25T11:36:52.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.554+0000 7f9ef98e9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c8275730 0x5589c86370a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:52.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.554+0000 7f9ef98e9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c8275730 0x5589c86370a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56968/0 (socket says 172.21.15.90:56968) 2024-04-25T11:36:52.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.554+0000 7f9ef98e9640 1 -- 172.21.15.90:0/4136939202 learned_addr learned my addr 172.21.15.90:0/4136939202 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:52.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.555+0000 7f9efa8eb640 1 --2- 172.21.15.90:0/4136939202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589c8296410 0x5589c8636620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:52.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.555+0000 7f9eeb7fe640 1 --2- 172.21.15.90:0/75396447 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9ee00afdb0 0x7f9ee00b22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:52.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.555+0000 7f9ef98e9640 1 -- 172.21.15.90:0/4136939202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589c8296410 msgr2=0x5589c8636620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:52.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.555+0000 7f9eeb7fe640 1 -- 172.21.15.90:0/75396447 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f9eec0ff7f0 con 0x7f9ef4102af0 2024-04-25T11:36:52.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.555+0000 7f9ef98e9640 1 --2- 172.21.15.90:0/4136939202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589c8296410 0x5589c8636620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.555+0000 7f9efa0ea640 1 --2- 172.21.15.90:0/4136939202 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5589c826d550 0x5589c8636b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:52.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.555+0000 7f9ef98e9640 1 -- 172.21.15.90:0/4136939202 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5589c826d550 msgr2=0x5589c8636b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:52.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.555+0000 7f9ef98e9640 1 --2- 172.21.15.90:0/4136939202 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5589c826d550 0x5589c8636b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.555+0000 7f9ef98e9640 1 -- 172.21.15.90:0/4136939202 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589c83767e0 con 0x5589c8275730 2024-04-25T11:36:52.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.555+0000 7f9efa8eb640 1 --2- 172.21.15.90:0/4136939202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589c8296410 0x5589c8636620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:36:52.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.555+0000 7f9efa0ea640 1 --2- 172.21.15.90:0/4136939202 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5589c826d550 0x5589c8636b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:36:52.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.555+0000 7f9efa8eb640 1 --2- 172.21.15.90:0/75396447 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9ee00afdb0 0x7f9ee00b22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:52.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.556+0000 7f9ef98e9640 1 --2- 172.21.15.90:0/4136939202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c8275730 0x5589c86370a0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f9ee4009750 tx=0x7f9ee4009c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3e41e441b71478d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:52.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.556+0000 7f9efa8eb640 1 --2- 172.21.15.90:0/75396447 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9ee00afdb0 0x7f9ee00b22c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x5589c8633790 tx=0x7f9ef0004ad0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:52.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.556+0000 7f9ee8bfe640 1 -- 172.21.15.90:0/4136939202 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee4015020 con 0x5589c8275730 2024-04-25T11:36:52.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.556+0000 7f9ee8bfe640 1 -- 172.21.15.90:0/4136939202 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ee4011420 con 0x5589c8275730 2024-04-25T11:36:52.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.556+0000 7f9ee8bfe640 1 -- 172.21.15.90:0/4136939202 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee4009e60 con 0x5589c8275730 2024-04-25T11:36:52.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.557+0000 7f9f031fd740 1 -- 172.21.15.90:0/4136939202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c8275730 msgr2=0x5589c86370a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:52.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.557+0000 7f9f031fd740 1 --2- 172.21.15.90:0/4136939202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c8275730 0x5589c86370a0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f9ee4009750 tx=0x7f9ee4009c10 comp rx=0 tx=0).stop 2024-04-25T11:36:52.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.558+0000 7f9f031fd740 1 -- 172.21.15.90:0/4136939202 shutdown_connections 2024-04-25T11:36:52.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.558+0000 7f9f031fd740 1 --2- 172.21.15.90:0/4136939202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c8275730 0x5589c86370a0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.558+0000 7f9f031fd740 1 --2- 172.21.15.90:0/4136939202 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5589c826d550 0x5589c8636b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.558+0000 7f9f031fd740 1 --2- 172.21.15.90:0/4136939202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589c8296410 0x5589c8636620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.558+0000 7f9f031fd740 1 -- 172.21.15.90:0/4136939202 >> 172.21.15.90:0/4136939202 conn(0x5589c828a560 msgr2=0x5589c8632dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:52.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.558+0000 7f9f031fd740 1 -- 172.21.15.90:0/4136939202 shutdown_connections 2024-04-25T11:36:52.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.558+0000 7f9f031fd740 1 -- 172.21.15.90:0/4136939202 wait complete. 2024-04-25T11:36:52.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.559+0000 7f9f031fd740 1 -- start start 2024-04-25T11:36:52.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.559+0000 7f9f031fd740 1 Processor -- start 2024-04-25T11:36:52.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.560+0000 7f9f031fd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c826d550 0x5589c864f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:52.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.560+0000 7f9f031fd740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5589c8275730 0x5589c864fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:52.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.560+0000 7f9f031fd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589c8296410 0x5589c8649b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:52.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.560+0000 7f9f031fd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5589c8643970 con 0x5589c826d550 2024-04-25T11:36:52.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.560+0000 7f9f031fd740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5589c85365d0 con 0x5589c8275730 2024-04-25T11:36:52.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.560+0000 7f9f031fd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5589c8643af0 con 0x5589c8296410 2024-04-25T11:36:52.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.560+0000 7f9efa0ea640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c826d550 0x5589c864f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:52.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.560+0000 7f9efa0ea640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c826d550 0x5589c864f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56970/0 (socket says 172.21.15.90:56970) 2024-04-25T11:36:52.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.560+0000 7f9efa0ea640 1 -- 172.21.15.90:0/3352672485 learned_addr learned my addr 172.21.15.90:0/3352672485 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:52.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.560+0000 7f9f031fd740 10 client.0 _ll_register_callbacks cb 0x5589c8524080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:52.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.561+0000 7f9efa0ea640 1 -- 172.21.15.90:0/3352672485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589c8296410 msgr2=0x5589c8649b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:52.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.561+0000 7f9efa0ea640 1 --2- 172.21.15.90:0/3352672485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589c8296410 0x5589c8649b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.561+0000 7f9efa0ea640 1 -- 172.21.15.90:0/3352672485 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5589c8275730 msgr2=0x5589c864fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:52.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.561+0000 7f9efa0ea640 1 --2- 172.21.15.90:0/3352672485 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5589c8275730 0x5589c864fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.561+0000 7f9efa0ea640 1 -- 172.21.15.90:0/3352672485 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589c83d6f70 con 0x5589c826d550 2024-04-25T11:36:52.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.561+0000 7f9efa0ea640 1 --2- 172.21.15.90:0/3352672485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c826d550 0x5589c864f750 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f9eec0c4ba0 tx=0x7f9eec0c7440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91865a1050eedb16 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:52.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.561+0000 7f9eddffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:52.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.561+0000 7f9eddffb640 1 -- 172.21.15.90:0/3352672485 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eec049040 con 0x5589c826d550 2024-04-25T11:36:52.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.561+0000 7f9eddffb640 1 -- 172.21.15.90:0/3352672485 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f9eec003c50 con 0x5589c826d550 2024-04-25T11:36:52.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.561+0000 7f9f031fd740 1 -- 172.21.15.90:0/3352672485 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ee4011420 con 0x5589c826d550 2024-04-25T11:36:52.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.562+0000 7f9eddffb640 1 -- 172.21.15.90:0/3352672485 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9eec0cbce0 con 0x5589c826d550 2024-04-25T11:36:52.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.562+0000 7f9f031fd740 10 client.16152 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:52.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.562+0000 7f9f031fd740 1 -- 172.21.15.90:0/3352672485 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5589c826e170 con 0x5589c826d550 2024-04-25T11:36:52.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.562+0000 7f9eddffb640 1 -- 172.21.15.90:0/3352672485 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ee4011420 con 0x5589c826d550 2024-04-25T11:36:52.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.562+0000 7f9eddffb640 1 -- 172.21.15.90:0/3352672485 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f9eec0ea070 con 0x5589c826d550 2024-04-25T11:36:52.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.562+0000 7f9f031fd740 10 client.16152 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:52.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.562+0000 7f9f031fd740 10 client.16152 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:36:52.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.562+0000 7f9f031fd740 20 client.16152 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:52.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.563+0000 7f9f031fd740 1 --2- 172.21.15.90:0/3352672485 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5589c82551f0 0x5589c865bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:52.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.563+0000 7f9f031fd740 4 client.16152 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:36:52.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.563+0000 7f9f031fd740 1 -- 172.21.15.90:0/3352672485 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5589c85aa710 con 0x5589c82551f0 2024-04-25T11:36:52.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.563+0000 7f9ef98e9640 1 --2- 172.21.15.90:0/3352672485 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5589c82551f0 0x5589c865bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:52.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.566+0000 7f9eddffb640 10 client.16152 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:52.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.566+0000 7f9ef98e9640 1 --2- 172.21.15.90:0/3352672485 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5589c82551f0 0x5589c865bbe0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1373a8e54edc9872 server_cookie=7126b7be0539133b in_seq=0 out_seq=0 2024-04-25T11:36:52.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.571+0000 7f9eddffb640 1 -- 172.21.15.90:0/3352672485 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5589c85aa710 con 0x5589c82551f0 2024-04-25T11:36:52.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.571+0000 7f9eddffb640 10 client.16152 handle_command_reply: tid=0 2024-04-25T11:36:52.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.571+0000 7f9f031fd740 1 client.16152 shutdown 2024-04-25T11:36:52.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.571+0000 7f9f031fd740 1 -- 172.21.15.90:0/3352672485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c826d550 msgr2=0x5589c864f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:52.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.571+0000 7f9f031fd740 1 --2- 172.21.15.90:0/3352672485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c826d550 0x5589c864f750 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f9eec0c4ba0 tx=0x7f9eec0c7440 comp rx=0 tx=0).stop 2024-04-25T11:36:52.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.571+0000 7f9f031fd740 1 -- 172.21.15.90:0/3352672485 shutdown_connections 2024-04-25T11:36:52.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.571+0000 7f9f031fd740 1 --2- 172.21.15.90:0/3352672485 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5589c82551f0 0x5589c865bbe0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.571+0000 7f9f031fd740 1 --2- 172.21.15.90:0/3352672485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589c8296410 0x5589c8649b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.572+0000 7f9f031fd740 1 --2- 172.21.15.90:0/3352672485 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5589c8275730 0x5589c864fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.572+0000 7f9f031fd740 1 --2- 172.21.15.90:0/3352672485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c826d550 0x5589c864f750 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.572+0000 7f9f031fd740 1 -- 172.21.15.90:0/3352672485 >> 172.21.15.90:0/3352672485 conn(0x5589c828a560 msgr2=0x5589c8638340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:52.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.572+0000 7f9eddffb640 0 client.16152 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:52.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.572+0000 7f9f031fd740 1 -- 172.21.15.90:0/3352672485 shutdown_connections 2024-04-25T11:36:52.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.572+0000 7f9f031fd740 1 -- 172.21.15.90:0/3352672485 wait complete. 2024-04-25T11:36:52.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.572+0000 7f9f031fd740 20 client.16152 trim_cache size 0 max 16384 2024-04-25T11:36:52.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.606+0000 7f9f031fd740 1 Processor -- start 2024-04-25T11:36:52.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.606+0000 7f9f031fd740 1 -- start start 2024-04-25T11:36:52.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.606+0000 7f9f031fd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589c826d550 0x5589c8649ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:52.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.606+0000 7f9f031fd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c8275730 0x5589c8642c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:52.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.607+0000 7f9f031fd740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5589c8296410 0x5589c86613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:52.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.607+0000 7f9f031fd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef41adad0 con 0x5589c8275730 2024-04-25T11:36:52.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.607+0000 7f9f031fd740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef41af610 con 0x5589c8296410 2024-04-25T11:36:52.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.607+0000 7f9ef98e9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c8275730 0x5589c8642c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:52.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.607+0000 7f9ef98e9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c8275730 0x5589c8642c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56980/0 (socket says 172.21.15.90:56980) 2024-04-25T11:36:52.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.607+0000 7f9ef98e9640 1 -- 172.21.15.90:0/1999713500 learned_addr learned my addr 172.21.15.90:0/1999713500 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:52.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.607+0000 7f9f031fd740 1 -- 172.21.15.90:0/1999713500 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef41ae1a0 con 0x5589c826d550 2024-04-25T11:36:52.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.607+0000 7f9efa0ea640 1 --2- 172.21.15.90:0/1999713500 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589c826d550 0x5589c8649ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:52.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.607+0000 7f9ef98e9640 1 -- 172.21.15.90:0/1999713500 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589c826d550 msgr2=0x5589c8649ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:52.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.607+0000 7f9ef98e9640 1 --2- 172.21.15.90:0/1999713500 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589c826d550 0x5589c8649ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.607+0000 7f9ef98e9640 1 -- 172.21.15.90:0/1999713500 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5589c8296410 msgr2=0x5589c86613e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:36:52.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.608+0000 7f9ef98e9640 1 --2- 172.21.15.90:0/1999713500 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5589c8296410 0x5589c86613e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.608+0000 7f9ef98e9640 1 -- 172.21.15.90:0/1999713500 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589c8643850 con 0x5589c8275730 2024-04-25T11:36:52.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.608+0000 7f9ef98e9640 1 --2- 172.21.15.90:0/1999713500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c8275730 0x5589c8642c20 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x5589c8633b30 tx=0x7f9ee4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11d6274cbe041735 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:52.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.608+0000 7f9edffff640 1 -- 172.21.15.90:0/1999713500 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee4015040 con 0x5589c8275730 2024-04-25T11:36:52.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.608+0000 7f9edffff640 1 -- 172.21.15.90:0/1999713500 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ee4028070 con 0x5589c8275730 2024-04-25T11:36:52.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.608+0000 7f9edffff640 1 -- 172.21.15.90:0/1999713500 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee4011ba0 con 0x5589c8275730 2024-04-25T11:36:52.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.609+0000 7f9f031fd740 1 -- 172.21.15.90:0/1999713500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c8275730 msgr2=0x5589c8642c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:52.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.609+0000 7f9f031fd740 1 --2- 172.21.15.90:0/1999713500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c8275730 0x5589c8642c20 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x5589c8633b30 tx=0x7f9ee4000bc0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.610+0000 7f9f031fd740 1 -- 172.21.15.90:0/1999713500 shutdown_connections 2024-04-25T11:36:52.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.610+0000 7f9f031fd740 1 --2- 172.21.15.90:0/1999713500 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5589c8296410 0x5589c86613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.610+0000 7f9f031fd740 1 --2- 172.21.15.90:0/1999713500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c8275730 0x5589c8642c20 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.610+0000 7f9f031fd740 1 --2- 172.21.15.90:0/1999713500 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589c826d550 0x5589c8649ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.610+0000 7f9f031fd740 1 -- 172.21.15.90:0/1999713500 >> 172.21.15.90:0/1999713500 conn(0x5589c828a560 msgr2=0x5589c86493d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:52.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.611+0000 7f9f031fd740 1 -- 172.21.15.90:0/1999713500 shutdown_connections 2024-04-25T11:36:52.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.611+0000 7f9f031fd740 1 -- 172.21.15.90:0/1999713500 wait complete. 2024-04-25T11:36:52.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.611+0000 7f9f031fd740 1 -- start start 2024-04-25T11:36:52.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.612+0000 7f9f031fd740 1 Processor -- start 2024-04-25T11:36:52.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.612+0000 7f9f031fd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c826d550 0x5589c8650c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:52.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.612+0000 7f9f031fd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589c8275730 0x5589c8651150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:52.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.613+0000 7f9f031fd740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5589c8296410 0x5589c86c1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:52.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.613+0000 7f9f031fd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef41aef40 con 0x5589c826d550 2024-04-25T11:36:52.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.613+0000 7f9f031fd740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef41ae870 con 0x5589c8296410 2024-04-25T11:36:52.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.613+0000 7f9f031fd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef41abaf0 con 0x5589c8275730 2024-04-25T11:36:52.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.613+0000 7f9f031fd740 10 client.0 _ll_register_callbacks cb 0x7f9eec0c7280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:52.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.613+0000 7f9efa0ea640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c826d550 0x5589c8650c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:52.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.613+0000 7f9efa0ea640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c826d550 0x5589c8650c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56992/0 (socket says 172.21.15.90:56992) 2024-04-25T11:36:52.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.613+0000 7f9efa0ea640 1 -- 172.21.15.90:0/4148134498 learned_addr learned my addr 172.21.15.90:0/4148134498 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:52.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.613+0000 7f9ef98e9640 1 --2- 172.21.15.90:0/4148134498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589c8275730 0x5589c8651150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:52.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.613+0000 7f9efa8eb640 1 --2- 172.21.15.90:0/4148134498 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5589c8296410 0x5589c86c1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:52.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.614+0000 7f9efa0ea640 1 -- 172.21.15.90:0/4148134498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589c8275730 msgr2=0x5589c8651150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:52.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.614+0000 7f9efa0ea640 1 --2- 172.21.15.90:0/4148134498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589c8275730 0x5589c8651150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.614+0000 7f9efa0ea640 1 -- 172.21.15.90:0/4148134498 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5589c8296410 msgr2=0x5589c86c1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:52.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.614+0000 7f9efa0ea640 1 --2- 172.21.15.90:0/4148134498 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5589c8296410 0x5589c86c1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.614+0000 7f9efa0ea640 1 -- 172.21.15.90:0/4148134498 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589c86c24c0 con 0x5589c826d550 2024-04-25T11:36:52.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.614+0000 7f9ef98e9640 1 --2- 172.21.15.90:0/4148134498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589c8275730 0x5589c8651150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:36:52.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.614+0000 7f9efa0ea640 1 --2- 172.21.15.90:0/4148134498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c826d550 0x5589c8650c10 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f9eec0fcf90 tx=0x7f9eec03b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=600965c891d140c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:52.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.614+0000 7f9eddffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:52.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.614+0000 7f9eddffb640 1 -- 172.21.15.90:0/4148134498 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eec049040 con 0x5589c826d550 2024-04-25T11:36:52.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.614+0000 7f9eddffb640 1 -- 172.21.15.90:0/4148134498 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f9eec0d0300 con 0x5589c826d550 2024-04-25T11:36:52.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.614+0000 7f9f031fd740 1 -- 172.21.15.90:0/4148134498 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ee4028070 con 0x5589c826d550 2024-04-25T11:36:52.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.615+0000 7f9eddffb640 1 -- 172.21.15.90:0/4148134498 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9eec0d0790 con 0x5589c826d550 2024-04-25T11:36:52.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.615+0000 7f9f031fd740 10 client.16158 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:52.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.615+0000 7f9f031fd740 1 -- 172.21.15.90:0/4148134498 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5589c86c35f0 con 0x5589c826d550 2024-04-25T11:36:52.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.615+0000 7f9eddffb640 1 -- 172.21.15.90:0/4148134498 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ee4028070 con 0x5589c826d550 2024-04-25T11:36:52.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.615+0000 7f9eddffb640 1 -- 172.21.15.90:0/4148134498 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f9eec0c3330 con 0x5589c826d550 2024-04-25T11:36:52.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.615+0000 7f9f031fd740 10 client.16158 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:52.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.615+0000 7f9f031fd740 10 client.16158 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:36:52.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.615+0000 7f9f031fd740 20 client.16158 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:52.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.615+0000 7f9f031fd740 1 --2- 172.21.15.90:0/4148134498 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5589c822a630 0x5589c86c5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:52.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.615+0000 7f9f031fd740 4 client.16158 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T11:36:52.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.615+0000 7f9f031fd740 1 -- 172.21.15.90:0/4148134498 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5589c85a4580 con 0x5589c822a630 2024-04-25T11:36:52.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.617+0000 7f9ef98e9640 1 --2- 172.21.15.90:0/4148134498 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5589c822a630 0x5589c86c5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:52.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.620+0000 7f9eddffb640 10 client.16158 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:52.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.620+0000 7f9ef98e9640 1 --2- 172.21.15.90:0/4148134498 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5589c822a630 0x5589c86c5be0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36d0186c54a75063 server_cookie=ad24efe4f37245fd in_seq=0 out_seq=0 2024-04-25T11:36:52.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.814+0000 7f9eddffb640 1 -- 172.21.15.90:0/4148134498 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5589c85a4580 con 0x5589c822a630 2024-04-25T11:36:52.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.814+0000 7f9eddffb640 10 client.16158 handle_command_reply: tid=0 2024-04-25T11:36:52.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.814+0000 7f9f031fd740 1 client.16158 shutdown 2024-04-25T11:36:52.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.814+0000 7f9f031fd740 1 -- 172.21.15.90:0/4148134498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c826d550 msgr2=0x5589c8650c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:52.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.814+0000 7f9f031fd740 1 --2- 172.21.15.90:0/4148134498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c826d550 0x5589c8650c10 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f9eec0fcf90 tx=0x7f9eec03b670 comp rx=0 tx=0).stop 2024-04-25T11:36:52.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.814+0000 7f9f031fd740 1 -- 172.21.15.90:0/4148134498 shutdown_connections 2024-04-25T11:36:52.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.814+0000 7f9f031fd740 1 --2- 172.21.15.90:0/4148134498 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5589c822a630 0x5589c86c5be0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.814+0000 7f9eddffb640 0 client.16158 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:52.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.815+0000 7f9f031fd740 1 --2- 172.21.15.90:0/4148134498 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5589c8296410 0x5589c86c1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.815+0000 7f9f031fd740 1 --2- 172.21.15.90:0/4148134498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589c8275730 0x5589c8651150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.815+0000 7f9f031fd740 1 --2- 172.21.15.90:0/4148134498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589c826d550 0x5589c8650c10 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.815+0000 7f9f031fd740 1 -- 172.21.15.90:0/4148134498 >> 172.21.15.90:0/4148134498 conn(0x5589c828a560 msgr2=0x5589c864f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:52.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.815+0000 7f9f031fd740 1 -- 172.21.15.90:0/4148134498 shutdown_connections 2024-04-25T11:36:52.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.815+0000 7f9f031fd740 1 -- 172.21.15.90:0/4148134498 wait complete. 2024-04-25T11:36:52.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.815+0000 7f9f031fd740 20 client.16158 trim_cache size 0 max 16384 2024-04-25T11:36:52.806 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:36:52.806 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-25T11:36:52.806 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "4a0f7cc7-46ac-49f6-889b-92a443418dcb", 2024-04-25T11:36:52.806 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-25T11:36:52.806 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:36:52.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.815+0000 7f9efc34e640 1 -- 172.21.15.90:0/75396447 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9ee00afdb0 msgr2=0x7f9ee00b22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:52.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.815+0000 7f9efc34e640 1 --2- 172.21.15.90:0/75396447 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9ee00afdb0 0x7f9ee00b22c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x5589c8633790 tx=0x7f9ef0004ad0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.815+0000 7f9efc34e640 1 -- 172.21.15.90:0/75396447 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ef4102af0 msgr2=0x7f9ef41a26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:52.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.815+0000 7f9efc34e640 1 --2- 172.21.15.90:0/75396447 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ef4102af0 0x7f9ef41a26c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f9eec005940 tx=0x7f9eec004270 comp rx=0 tx=0).stop 2024-04-25T11:36:52.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.816+0000 7f9efc34e640 1 -- 172.21.15.90:0/75396447 shutdown_connections 2024-04-25T11:36:52.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.816+0000 7f9efc34e640 1 --2- 172.21.15.90:0/75396447 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9ee00afdb0 0x7f9ee00b22c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.816+0000 7f9efc34e640 1 --2- 172.21.15.90:0/75396447 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9ef410d380 0x7f9ef419c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.816+0000 7f9efc34e640 1 --2- 172.21.15.90:0/75396447 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ef4108a20 0x7f9ef41a2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.816+0000 7f9efc34e640 1 --2- 172.21.15.90:0/75396447 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ef4102af0 0x7f9ef41a26c0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.816+0000 7f9efc34e640 1 -- 172.21.15.90:0/75396447 >> 172.21.15.90:0/75396447 conn(0x7f9ef4078100 msgr2=0x7f9ef4101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:52.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.816+0000 7f9efc34e640 1 -- 172.21.15.90:0/75396447 shutdown_connections 2024-04-25T11:36:52.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:52.816+0000 7f9efc34e640 1 -- 172.21.15.90:0/75396447 wait complete. 2024-04-25T11:36:52.822 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:36:52.822 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:36:53.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.066+0000 7f941a746640 1 -- 172.21.15.90:0/3593560941 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94141042b0 msgr2=0x7f9414110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:53.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.066+0000 7f941a746640 1 --2- 172.21.15.90:0/3593560941 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94141042b0 0x7f9414110630 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f940c009900 tx=0x7f940c02a410 comp rx=0 tx=0).stop 2024-04-25T11:36:53.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.066+0000 7f941a746640 1 -- 172.21.15.90:0/3593560941 shutdown_connections 2024-04-25T11:36:53.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.066+0000 7f941a746640 1 --2- 172.21.15.90:0/3593560941 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94141042b0 0x7f9414110630 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.066+0000 7f941a746640 1 --2- 172.21.15.90:0/3593560941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9414109fb0 0x7f9414103c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.066+0000 7f941a746640 1 --2- 172.21.15.90:0/3593560941 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9414109b10 0x7f9414103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.066+0000 7f941a746640 1 -- 172.21.15.90:0/3593560941 >> 172.21.15.90:0/3593560941 conn(0x7f94140ff360 msgr2=0x7f9414101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:53.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.066+0000 7f941a746640 1 -- 172.21.15.90:0/3593560941 shutdown_connections 2024-04-25T11:36:53.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.067+0000 7f941a746640 1 -- 172.21.15.90:0/3593560941 wait complete. 2024-04-25T11:36:53.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.068+0000 7f941a746640 1 Processor -- start 2024-04-25T11:36:53.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.068+0000 7f941a746640 1 -- start start 2024-04-25T11:36:53.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.068+0000 7f941a746640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94141042b0 0x7f941419a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:53.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.068+0000 7f941a746640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9414109b10 0x7f941419acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:53.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.068+0000 7f941a746640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9414109fb0 0x7f941419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:53.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.068+0000 7f941a746640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9414113dd0 con 0x7f94141042b0 2024-04-25T11:36:53.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.068+0000 7f941a746640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9414113c50 con 0x7f9414109b10 2024-04-25T11:36:53.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.068+0000 7f941a746640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9414113f50 con 0x7f9414109fb0 2024-04-25T11:36:53.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.068+0000 7f9413fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94141042b0 0x7f941419a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:53.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.068+0000 7f9413fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94141042b0 0x7f941419a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57010/0 (socket says 172.21.15.90:57010) 2024-04-25T11:36:53.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.068+0000 7f9413fff640 1 -- 172.21.15.90:0/3450897170 learned_addr learned my addr 172.21.15.90:0/3450897170 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:53.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.069+0000 7f9413fff640 1 -- 172.21.15.90:0/3450897170 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9414109fb0 msgr2=0x7f941419efb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:36:53.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.069+0000 7f9413fff640 1 --2- 172.21.15.90:0/3450897170 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9414109fb0 0x7f941419efb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.069+0000 7f9413fff640 1 -- 172.21.15.90:0/3450897170 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9414109b10 msgr2=0x7f941419acf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:36:53.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.069+0000 7f9413fff640 1 --2- 172.21.15.90:0/3450897170 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9414109b10 0x7f941419acf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.069+0000 7f9413fff640 1 -- 172.21.15.90:0/3450897170 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f941419f5f0 con 0x7f94141042b0 2024-04-25T11:36:53.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.069+0000 7f9413fff640 1 --2- 172.21.15.90:0/3450897170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94141042b0 0x7f941419a790 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f940400ea80 tx=0x7f940400ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:53.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.069+0000 7f9411ffb640 1 -- 172.21.15.90:0/3450897170 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940400ce40 con 0x7f94141042b0 2024-04-25T11:36:53.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.069+0000 7f9411ffb640 1 -- 172.21.15.90:0/3450897170 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9404011920 con 0x7f94141042b0 2024-04-25T11:36:53.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.069+0000 7f9411ffb640 1 -- 172.21.15.90:0/3450897170 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9404010a30 con 0x7f94141042b0 2024-04-25T11:36:53.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.070+0000 7f941a746640 1 -- 172.21.15.90:0/3450897170 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f941419f8e0 con 0x7f94141042b0 2024-04-25T11:36:53.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.070+0000 7f941a746640 1 -- 172.21.15.90:0/3450897170 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f941419fde0 con 0x7f94141042b0 2024-04-25T11:36:53.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.073+0000 7f9411ffb640 1 -- 172.21.15.90:0/3450897170 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f9404010bd0 con 0x7f94141042b0 2024-04-25T11:36:53.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.073+0000 7f9411ffb640 1 --2- 172.21.15.90:0/3450897170 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f94000afd60 0x7f94000b2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:53.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.074+0000 7f9411ffb640 1 -- 172.21.15.90:0/3450897170 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f94040df7f0 con 0x7f94141042b0 2024-04-25T11:36:53.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.074+0000 7f94215fb740 1 Processor -- start 2024-04-25T11:36:53.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.074+0000 7f940b7fe640 1 --2- 172.21.15.90:0/3450897170 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f94000afd60 0x7f94000b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:53.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.074+0000 7f94215fb740 1 -- start start 2024-04-25T11:36:53.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.074+0000 7f94215fb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55760e9b3390 0x55760ed538d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:53.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.074+0000 7f94215fb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e98a4d0 0x55760ed53e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:53.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.074+0000 7f94215fb740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55760e9926b0 0x55760ed54350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:53.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.074+0000 7f94215fb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55760ecb9d80 con 0x55760e98a4d0 2024-04-25T11:36:53.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.074+0000 7f94215fb740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55760ec738b0 con 0x55760e9926b0 2024-04-25T11:36:53.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.074+0000 7f94215fb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55760ecb5d40 con 0x55760e9b3390 2024-04-25T11:36:53.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.075+0000 7f940b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e98a4d0 0x55760ed53e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:53.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.075+0000 7f940b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e98a4d0 0x55760ed53e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57020/0 (socket says 172.21.15.90:57020) 2024-04-25T11:36:53.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.075+0000 7f940b7fe640 1 -- 172.21.15.90:0/4106017535 learned_addr learned my addr 172.21.15.90:0/4106017535 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:53.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.075+0000 7f940b7fe640 1 --2- 172.21.15.90:0/3450897170 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f94000afd60 0x7f94000b2270 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9414108bf0 tx=0x7f93fc009b10 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:53.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.075+0000 7f940b7fe640 1 -- 172.21.15.90:0/4106017535 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55760e9b3390 msgr2=0x55760ed538d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:53.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.075+0000 7f940b7fe640 1 --2- 172.21.15.90:0/4106017535 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55760e9b3390 0x55760ed538d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.075+0000 7f940b7fe640 1 -- 172.21.15.90:0/4106017535 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55760e9926b0 msgr2=0x55760ed54350 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:36:53.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.075+0000 7f940b7fe640 1 --2- 172.21.15.90:0/4106017535 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55760e9926b0 0x55760ed54350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.075+0000 7f940b7fe640 1 -- 172.21.15.90:0/4106017535 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55760ec44a40 con 0x55760e98a4d0 2024-04-25T11:36:53.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.076+0000 7f940b7fe640 1 --2- 172.21.15.90:0/4106017535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e98a4d0 0x55760ed53e10 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f93fc00ba30 tx=0x7f93fc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b397a3ea82b9ebb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:53.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.076+0000 7f940a7fc640 1 -- 172.21.15.90:0/4106017535 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93fc00f040 con 0x55760e98a4d0 2024-04-25T11:36:53.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.076+0000 7f940a7fc640 1 -- 172.21.15.90:0/4106017535 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93fc022070 con 0x55760e98a4d0 2024-04-25T11:36:53.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.076+0000 7f940a7fc640 1 -- 172.21.15.90:0/4106017535 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93fc01ddc0 con 0x55760e98a4d0 2024-04-25T11:36:53.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.078+0000 7f94215fb740 1 -- 172.21.15.90:0/4106017535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e98a4d0 msgr2=0x55760ed53e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:53.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.078+0000 7f94215fb740 1 --2- 172.21.15.90:0/4106017535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e98a4d0 0x55760ed53e10 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f93fc00ba30 tx=0x7f93fc016000 comp rx=0 tx=0).stop 2024-04-25T11:36:53.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.078+0000 7f94215fb740 1 -- 172.21.15.90:0/4106017535 shutdown_connections 2024-04-25T11:36:53.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.078+0000 7f94215fb740 1 --2- 172.21.15.90:0/4106017535 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55760e9926b0 0x55760ed54350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.078+0000 7f94215fb740 1 --2- 172.21.15.90:0/4106017535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e98a4d0 0x55760ed53e10 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.078+0000 7f94215fb740 1 --2- 172.21.15.90:0/4106017535 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55760e9b3390 0x55760ed538d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.078+0000 7f94215fb740 1 -- 172.21.15.90:0/4106017535 >> 172.21.15.90:0/4106017535 conn(0x55760e9a74e0 msgr2=0x55760ed50210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:53.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.078+0000 7f94215fb740 1 -- 172.21.15.90:0/4106017535 shutdown_connections 2024-04-25T11:36:53.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.078+0000 7f94215fb740 1 -- 172.21.15.90:0/4106017535 wait complete. 2024-04-25T11:36:53.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.079+0000 7f94215fb740 1 -- start start 2024-04-25T11:36:53.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.080+0000 7f94215fb740 1 Processor -- start 2024-04-25T11:36:53.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.080+0000 7f94215fb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55760e98a4d0 0x55760ed635b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:53.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.081+0000 7f94215fb740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55760e9926b0 0x55760ed63b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:53.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.081+0000 7f94215fb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e9b3390 0x55760ed5e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:53.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.081+0000 7f94215fb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55760ed5af90 con 0x55760e9b3390 2024-04-25T11:36:53.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.081+0000 7f94215fb740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55760ec40d10 con 0x55760e9926b0 2024-04-25T11:36:53.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.081+0000 7f94215fb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55760ed5b110 con 0x55760e98a4d0 2024-04-25T11:36:53.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.081+0000 7f94215fb740 10 client.0 _ll_register_callbacks cb 0x55760ec6d210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:53.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.081+0000 7f9418ce3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e9b3390 0x55760ed5e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:53.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.081+0000 7f9418ce3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e9b3390 0x55760ed5e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57024/0 (socket says 172.21.15.90:57024) 2024-04-25T11:36:53.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.081+0000 7f9418ce3640 1 -- 172.21.15.90:0/821296471 learned_addr learned my addr 172.21.15.90:0/821296471 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:53.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.082+0000 7f9413fff640 1 --2- 172.21.15.90:0/821296471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55760e98a4d0 0x55760ed635b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:53.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.082+0000 7f9418ce3640 1 -- 172.21.15.90:0/821296471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55760e98a4d0 msgr2=0x55760ed635b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:53.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.082+0000 7f940b7fe640 1 --2- 172.21.15.90:0/821296471 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55760e9926b0 0x55760ed63b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:53.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.082+0000 7f9418ce3640 1 --2- 172.21.15.90:0/821296471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55760e98a4d0 0x55760ed635b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.082+0000 7f9418ce3640 1 -- 172.21.15.90:0/821296471 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55760e9926b0 msgr2=0x55760ed63b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:53.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.082+0000 7f9418ce3640 1 --2- 172.21.15.90:0/821296471 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55760e9926b0 0x55760ed63b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.082+0000 7f9418ce3640 1 -- 172.21.15.90:0/821296471 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55760ea957d0 con 0x55760e9b3390 2024-04-25T11:36:53.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.082+0000 7f940b7fe640 1 --2- 172.21.15.90:0/821296471 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55760e9926b0 0x55760ed63b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:36:53.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.083+0000 7f9418ce3640 1 --2- 172.21.15.90:0/821296471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e9b3390 0x55760ed5e990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f940c005ab0 tx=0x7f940c009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87908391a31bd328 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:53.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.083+0000 7f93f37fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:53.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.083+0000 7f93f37fe640 1 -- 172.21.15.90:0/821296471 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940c03e030 con 0x55760e9b3390 2024-04-25T11:36:53.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.083+0000 7f94215fb740 1 -- 172.21.15.90:0/821296471 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93fc022070 con 0x55760e9b3390 2024-04-25T11:36:53.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.084+0000 7f93f37fe640 1 -- 172.21.15.90:0/821296471 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f940c003880 con 0x55760e9b3390 2024-04-25T11:36:53.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.084+0000 7f93f37fe640 1 -- 172.21.15.90:0/821296471 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f940c035040 con 0x55760e9b3390 2024-04-25T11:36:53.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.084+0000 7f94215fb740 10 client.16170 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:53.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.084+0000 7f94215fb740 1 -- 172.21.15.90:0/821296471 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55760e9cb3f0 con 0x55760e9b3390 2024-04-25T11:36:53.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.084+0000 7f93f37fe640 1 -- 172.21.15.90:0/821296471 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93fc022070 con 0x55760e9b3390 2024-04-25T11:36:53.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.085+0000 7f93f37fe640 1 -- 172.21.15.90:0/821296471 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f940c03a3f0 con 0x55760e9b3390 2024-04-25T11:36:53.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.085+0000 7f94215fb740 10 client.16170 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:53.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.085+0000 7f94215fb740 10 client.16170 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:36:53.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.085+0000 7f94215fb740 20 client.16170 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:53.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.085+0000 7f94215fb740 1 --2- 172.21.15.90:0/821296471 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55760e971f60 0x55760ed81770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:53.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.085+0000 7f94215fb740 4 client.16170 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:36:53.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.085+0000 7f94215fb740 1 -- 172.21.15.90:0/821296471 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55760ecc7990 con 0x55760e971f60 2024-04-25T11:36:53.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.086+0000 7f9413fff640 1 --2- 172.21.15.90:0/821296471 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55760e971f60 0x55760ed81770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:53.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.090+0000 7f93f37fe640 10 client.16170 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:53.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.090+0000 7f9413fff640 1 --2- 172.21.15.90:0/821296471 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55760e971f60 0x55760ed81770 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f9a64c08430f965 server_cookie=77e3110b595d930b in_seq=0 out_seq=0 2024-04-25T11:36:53.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.096+0000 7f93f37fe640 1 -- 172.21.15.90:0/821296471 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55760ecc7990 con 0x55760e971f60 2024-04-25T11:36:53.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.096+0000 7f93f37fe640 10 client.16170 handle_command_reply: tid=0 2024-04-25T11:36:53.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.096+0000 7f94215fb740 1 client.16170 shutdown 2024-04-25T11:36:53.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.096+0000 7f94215fb740 1 -- 172.21.15.90:0/821296471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e9b3390 msgr2=0x55760ed5e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:53.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.096+0000 7f94215fb740 1 --2- 172.21.15.90:0/821296471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e9b3390 0x55760ed5e990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f940c005ab0 tx=0x7f940c009600 comp rx=0 tx=0).stop 2024-04-25T11:36:53.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.096+0000 7f94215fb740 1 -- 172.21.15.90:0/821296471 shutdown_connections 2024-04-25T11:36:53.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.096+0000 7f94215fb740 1 --2- 172.21.15.90:0/821296471 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55760e971f60 0x55760ed81770 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.096+0000 7f94215fb740 1 --2- 172.21.15.90:0/821296471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e9b3390 0x55760ed5e990 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.096+0000 7f94215fb740 1 --2- 172.21.15.90:0/821296471 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55760e9926b0 0x55760ed63b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.096+0000 7f93f37fe640 0 client.16170 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:53.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.096+0000 7f94215fb740 1 --2- 172.21.15.90:0/821296471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55760e98a4d0 0x55760ed635b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.096+0000 7f94215fb740 1 -- 172.21.15.90:0/821296471 >> 172.21.15.90:0/821296471 conn(0x55760e9a74e0 msgr2=0x55760ed4e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:53.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.096+0000 7f94215fb740 1 -- 172.21.15.90:0/821296471 shutdown_connections 2024-04-25T11:36:53.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.097+0000 7f94215fb740 1 -- 172.21.15.90:0/821296471 wait complete. 2024-04-25T11:36:53.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.097+0000 7f94215fb740 20 client.16170 trim_cache size 0 max 16384 2024-04-25T11:36:53.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.124+0000 7f94215fb740 1 Processor -- start 2024-04-25T11:36:53.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.124+0000 7f94215fb740 1 -- start start 2024-04-25T11:36:53.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.125+0000 7f94215fb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55760e98a4d0 0x55760ed4f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:53.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.125+0000 7f94215fb740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55760e9926b0 0x55760ed62f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:53.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.125+0000 7f94215fb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e9b3390 0x55760ed57d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:53.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.125+0000 7f94215fb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f94141ada80 con 0x55760e9b3390 2024-04-25T11:36:53.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.125+0000 7f94215fb740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f94141af5c0 con 0x55760e9926b0 2024-04-25T11:36:53.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.125+0000 7f94215fb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f94141ae150 con 0x55760e98a4d0 2024-04-25T11:36:53.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.125+0000 7f9418ce3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e9b3390 0x55760ed57d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:53.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.125+0000 7f9418ce3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e9b3390 0x55760ed57d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57032/0 (socket says 172.21.15.90:57032) 2024-04-25T11:36:53.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.125+0000 7f9418ce3640 1 -- 172.21.15.90:0/19594192 learned_addr learned my addr 172.21.15.90:0/19594192 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:53.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.126+0000 7f9418ce3640 1 -- 172.21.15.90:0/19594192 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55760e98a4d0 msgr2=0x55760ed4f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:53.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.126+0000 7f9418ce3640 1 --2- 172.21.15.90:0/19594192 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55760e98a4d0 0x55760ed4f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.126+0000 7f9418ce3640 1 -- 172.21.15.90:0/19594192 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55760e9926b0 msgr2=0x55760ed62f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:53.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.126+0000 7f9418ce3640 1 --2- 172.21.15.90:0/19594192 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55760e9926b0 0x55760ed62f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.126+0000 7f9418ce3640 1 -- 172.21.15.90:0/19594192 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55760ed63e80 con 0x55760e9b3390 2024-04-25T11:36:53.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.126+0000 7f9418ce3640 1 --2- 172.21.15.90:0/19594192 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e9b3390 0x55760ed57d60 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f940c03a630 tx=0x7f940c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4537659d5b30a2d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:53.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.126+0000 7f9409ffb640 1 -- 172.21.15.90:0/19594192 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940c04f070 con 0x55760e9b3390 2024-04-25T11:36:53.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.126+0000 7f9409ffb640 1 -- 172.21.15.90:0/19594192 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f940c003d30 con 0x55760e9b3390 2024-04-25T11:36:53.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.126+0000 7f9409ffb640 1 -- 172.21.15.90:0/19594192 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940c03e070 con 0x55760e9b3390 2024-04-25T11:36:53.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.128+0000 7f94215fb740 1 -- 172.21.15.90:0/19594192 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e9b3390 msgr2=0x55760ed57d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:53.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.128+0000 7f94215fb740 1 --2- 172.21.15.90:0/19594192 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e9b3390 0x55760ed57d60 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f940c03a630 tx=0x7f940c02a410 comp rx=0 tx=0).stop 2024-04-25T11:36:53.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.129+0000 7f94215fb740 1 -- 172.21.15.90:0/19594192 shutdown_connections 2024-04-25T11:36:53.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.129+0000 7f94215fb740 1 --2- 172.21.15.90:0/19594192 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e9b3390 0x55760ed57d60 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.129+0000 7f94215fb740 1 --2- 172.21.15.90:0/19594192 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55760e9926b0 0x55760ed62f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.129+0000 7f94215fb740 1 --2- 172.21.15.90:0/19594192 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55760e98a4d0 0x55760ed4f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.129+0000 7f94215fb740 1 -- 172.21.15.90:0/19594192 >> 172.21.15.90:0/19594192 conn(0x55760e9a74e0 msgr2=0x55760ed5d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:53.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.129+0000 7f94215fb740 1 -- 172.21.15.90:0/19594192 shutdown_connections 2024-04-25T11:36:53.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.129+0000 7f94215fb740 1 -- 172.21.15.90:0/19594192 wait complete. 2024-04-25T11:36:53.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.130+0000 7f94215fb740 1 -- start start 2024-04-25T11:36:53.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.130+0000 7f94215fb740 1 Processor -- start 2024-04-25T11:36:53.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.130+0000 7f94215fb740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55760e98a4d0 0x55760ed86ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:53.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.130+0000 7f94215fb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55760e9926b0 0x55760ed86ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:53.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.130+0000 7f94215fb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e9b3390 0x55760ede8750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:53.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.130+0000 7f94215fb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f94141aeef0 con 0x55760e9b3390 2024-04-25T11:36:53.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.130+0000 7f94215fb740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f94141ae820 con 0x55760e98a4d0 2024-04-25T11:36:53.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.130+0000 7f94215fb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f94141aba60 con 0x55760e9926b0 2024-04-25T11:36:53.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.130+0000 7f94215fb740 10 client.0 _ll_register_callbacks cb 0x7f940c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:36:53.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.131+0000 7f9418ce3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e9b3390 0x55760ede8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:53.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.131+0000 7f9418ce3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e9b3390 0x55760ede8750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57034/0 (socket says 172.21.15.90:57034) 2024-04-25T11:36:53.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.131+0000 7f9418ce3640 1 -- 172.21.15.90:0/4206627217 learned_addr learned my addr 172.21.15.90:0/4206627217 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:36:53.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.131+0000 7f9418ce3640 1 -- 172.21.15.90:0/4206627217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55760e9926b0 msgr2=0x55760ed86ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:36:53.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.131+0000 7f9418ce3640 1 --2- 172.21.15.90:0/4206627217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55760e9926b0 0x55760ed86ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.131+0000 7f9418ce3640 1 -- 172.21.15.90:0/4206627217 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55760e98a4d0 msgr2=0x55760ed86ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:53.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.131+0000 7f9418ce3640 1 --2- 172.21.15.90:0/4206627217 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55760e98a4d0 0x55760ed86ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.131+0000 7f9418ce3640 1 -- 172.21.15.90:0/4206627217 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55760ed63920 con 0x55760e9b3390 2024-04-25T11:36:53.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.131+0000 7f9418ce3640 1 --2- 172.21.15.90:0/4206627217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e9b3390 0x55760ede8750 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f940c009900 tx=0x7f940c003ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9b8bb49b740d3ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:53.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.131+0000 7f93f37fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:36:53.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.132+0000 7f93f37fe640 1 -- 172.21.15.90:0/4206627217 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940c03e070 con 0x55760e9b3390 2024-04-25T11:36:53.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.132+0000 7f93f37fe640 1 -- 172.21.15.90:0/4206627217 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f940c03abb0 con 0x55760e9b3390 2024-04-25T11:36:53.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.132+0000 7f94215fb740 1 -- 172.21.15.90:0/4206627217 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f940c003d30 con 0x55760e9b3390 2024-04-25T11:36:53.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.132+0000 7f93f37fe640 1 -- 172.21.15.90:0/4206627217 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f940c04f040 con 0x55760e9b3390 2024-04-25T11:36:53.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.132+0000 7f94215fb740 10 client.16176 fetch_fsmap learned FSMap version 26 2024-04-25T11:36:53.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.132+0000 7f94215fb740 1 -- 172.21.15.90:0/4206627217 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55760e98b0f0 con 0x55760e9b3390 2024-04-25T11:36:53.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.132+0000 7f93f37fe640 1 -- 172.21.15.90:0/4206627217 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f940c003d30 con 0x55760e9b3390 2024-04-25T11:36:53.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.133+0000 7f93f37fe640 1 -- 172.21.15.90:0/4206627217 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f940c004730 con 0x55760e9b3390 2024-04-25T11:36:53.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.133+0000 7f94215fb740 10 client.16176 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:36:53.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.133+0000 7f94215fb740 10 client.16176 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:36:53.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.133+0000 7f94215fb740 20 client.16176 populate_metadata read hostname 'smithi090' 2024-04-25T11:36:53.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.133+0000 7f94215fb740 1 --2- 172.21.15.90:0/4206627217 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55760e9473a0 0x55760edec090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:53.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.133+0000 7f94215fb740 4 client.16176 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:36:53.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.133+0000 7f94215fb740 1 -- 172.21.15.90:0/4206627217 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55760ecc1800 con 0x55760e9473a0 2024-04-25T11:36:53.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.135+0000 7f9413fff640 1 --2- 172.21.15.90:0/4206627217 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55760e9473a0 0x55760edec090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:53.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.137+0000 7f93f37fe640 10 client.16176 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:53.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.137+0000 7f9413fff640 1 --2- 172.21.15.90:0/4206627217 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55760e9473a0 0x55760edec090 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68c19ca85b22f442 server_cookie=3a87322e19c30c47 in_seq=0 out_seq=0 2024-04-25T11:36:53.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.141+0000 7f93f37fe640 1 -- 172.21.15.90:0/4206627217 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x55760ecc1800 con 0x55760e9473a0 2024-04-25T11:36:53.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.141+0000 7f93f37fe640 10 client.16176 handle_command_reply: tid=0 2024-04-25T11:36:53.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.141+0000 7f94215fb740 1 client.16176 shutdown 2024-04-25T11:36:53.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.142+0000 7f94215fb740 1 -- 172.21.15.90:0/4206627217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e9b3390 msgr2=0x55760ede8750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:53.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.142+0000 7f94215fb740 1 --2- 172.21.15.90:0/4206627217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e9b3390 0x55760ede8750 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f940c009900 tx=0x7f940c003ab0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.142+0000 7f94215fb740 1 -- 172.21.15.90:0/4206627217 shutdown_connections 2024-04-25T11:36:53.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.142+0000 7f94215fb740 1 --2- 172.21.15.90:0/4206627217 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55760e9473a0 0x55760edec090 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.142+0000 7f94215fb740 1 --2- 172.21.15.90:0/4206627217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55760e9b3390 0x55760ede8750 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.142+0000 7f94215fb740 1 --2- 172.21.15.90:0/4206627217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55760e9926b0 0x55760ed86ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.142+0000 7f94215fb740 1 --2- 172.21.15.90:0/4206627217 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55760e98a4d0 0x55760ed86ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.142+0000 7f94215fb740 1 -- 172.21.15.90:0/4206627217 >> 172.21.15.90:0/4206627217 conn(0x55760e9a74e0 msgr2=0x55760ed4f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:53.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.142+0000 7f93f37fe640 0 client.16176 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:36:53.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.142+0000 7f94215fb740 1 -- 172.21.15.90:0/4206627217 shutdown_connections 2024-04-25T11:36:53.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.142+0000 7f94215fb740 1 -- 172.21.15.90:0/4206627217 wait complete. 2024-04-25T11:36:53.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.142+0000 7f94215fb740 20 client.16176 trim_cache size 0 max 16384 2024-04-25T11:36:53.132 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:36:53.132 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "scrub active (15 inodes in the stack)", 2024-04-25T11:36:53.132 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-25T11:36:53.132 INFO:teuthology.orchestra.run.smithi090.stdout: "4a0f7cc7-46ac-49f6-889b-92a443418dcb": { 2024-04-25T11:36:53.132 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-25T11:36:53.132 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "4a0f7cc7-46ac-49f6-889b-92a443418dcb", 2024-04-25T11:36:53.132 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-25T11:36:53.132 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:36:53.133 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:36:53.133 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:36:53.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.143+0000 7f941a746640 1 -- 172.21.15.90:0/3450897170 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f94000afd60 msgr2=0x7f94000b2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:53.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.143+0000 7f941a746640 1 --2- 172.21.15.90:0/3450897170 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f94000afd60 0x7f94000b2270 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9414108bf0 tx=0x7f93fc009b10 comp rx=0 tx=0).stop 2024-04-25T11:36:53.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.143+0000 7f941a746640 1 -- 172.21.15.90:0/3450897170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94141042b0 msgr2=0x7f941419a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:53.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.143+0000 7f941a746640 1 --2- 172.21.15.90:0/3450897170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94141042b0 0x7f941419a790 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f940400ea80 tx=0x7f940400ef40 comp rx=0 tx=0).stop 2024-04-25T11:36:53.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.144+0000 7f941a746640 1 -- 172.21.15.90:0/3450897170 shutdown_connections 2024-04-25T11:36:53.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.144+0000 7f941a746640 1 --2- 172.21.15.90:0/3450897170 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f94000afd60 0x7f94000b2270 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.144+0000 7f941a746640 1 --2- 172.21.15.90:0/3450897170 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9414109fb0 0x7f941419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.144+0000 7f941a746640 1 --2- 172.21.15.90:0/3450897170 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9414109b10 0x7f941419acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.144+0000 7f941a746640 1 --2- 172.21.15.90:0/3450897170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94141042b0 0x7f941419a790 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:53.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.144+0000 7f941a746640 1 -- 172.21.15.90:0/3450897170 >> 172.21.15.90:0/3450897170 conn(0x7f94140ff360 msgr2=0x7f9414101ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:53.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.144+0000 7f941a746640 1 -- 172.21.15.90:0/3450897170 shutdown_connections 2024-04-25T11:36:53.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:36:53.144+0000 7f941a746640 1 -- 172.21.15.90:0/3450897170 wait complete. 2024-04-25T11:36:53.155 INFO:tasks.cephfs.filesystem:scrub status for tag:4a0f7cc7-46ac-49f6-889b-92a443418dcb - {'path': '/', 'tag': '4a0f7cc7-46ac-49f6-889b-92a443418dcb', 'options': 'recursive,force'} 2024-04-25T11:36:53.818 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:53 smithi090 ceph-mon[29575]: pgmap v57: 97 pgs: 97 active+clean; 1.4 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 121 MiB/s wr, 95 op/s 2024-04-25T11:36:53.818 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:53 smithi090 ceph-mon[29575]: scrub queued for path: / 2024-04-25T11:36:53.818 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:53 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:36:53.818 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:53 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:36:54.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:53 smithi138 ceph-mon[32328]: pgmap v57: 97 pgs: 97 active+clean; 1.4 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 121 MiB/s wr, 95 op/s 2024-04-25T11:36:54.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:53 smithi138 ceph-mon[32328]: scrub queued for path: / 2024-04-25T11:36:54.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:53 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:36:54.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:53 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:36:54.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:53 smithi156 ceph-mon[33045]: pgmap v57: 97 pgs: 97 active+clean; 1.4 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 121 MiB/s wr, 95 op/s 2024-04-25T11:36:54.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:53 smithi156 ceph-mon[33045]: scrub queued for path: / 2024-04-25T11:36:54.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:53 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:36:54.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:53 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:36:54.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:54 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:36:55.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:54 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:36:55.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:54 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:36:56.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:55 smithi090 ceph-mon[29575]: pgmap v58: 97 pgs: 97 active+clean; 5.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 487 MiB/s wr, 379 op/s 2024-04-25T11:36:56.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:55 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:36:56.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:55 smithi138 ceph-mon[32328]: pgmap v58: 97 pgs: 97 active+clean; 5.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 487 MiB/s wr, 379 op/s 2024-04-25T11:36:56.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:55 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:36:56.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:55 smithi156 ceph-mon[33045]: pgmap v58: 97 pgs: 97 active+clean; 5.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 487 MiB/s wr, 379 op/s 2024-04-25T11:36:56.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:55 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:36:58.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:58 smithi138 ceph-mon[32328]: pgmap v59: 97 pgs: 97 active+clean; 5.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 487 MiB/s wr, 375 op/s 2024-04-25T11:36:58.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:58 smithi156 ceph-mon[33045]: pgmap v59: 97 pgs: 97 active+clean; 5.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 487 MiB/s wr, 375 op/s 2024-04-25T11:36:58.485 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:58 smithi090 ceph-mon[29575]: pgmap v59: 97 pgs: 97 active+clean; 5.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 487 MiB/s wr, 375 op/s 2024-04-25T11:36:59.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:36:59 smithi090 ceph-mon[29575]: pgmap v60: 97 pgs: 97 active+clean; 5.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 487 MiB/s wr, 375 op/s 2024-04-25T11:36:59.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:36:59 smithi138 ceph-mon[32328]: pgmap v60: 97 pgs: 97 active+clean; 5.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 487 MiB/s wr, 375 op/s 2024-04-25T11:36:59.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:36:59 smithi156 ceph-mon[33045]: pgmap v60: 97 pgs: 97 active+clean; 5.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 487 MiB/s wr, 375 op/s 2024-04-25T11:37:00.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.325+0000 7f91e1100640 1 -- 172.21.15.90:0/1700012001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91dc108a80 msgr2=0x7f91dc102620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:00.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.325+0000 7f91e1100640 1 --2- 172.21.15.90:0/1700012001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91dc108a80 0x7f91dc102620 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f91cc009900 tx=0x7f91cc02a410 comp rx=0 tx=0).stop 2024-04-25T11:37:00.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.325+0000 7f91e1100640 1 -- 172.21.15.90:0/1700012001 shutdown_connections 2024-04-25T11:37:00.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.325+0000 7f91e1100640 1 --2- 172.21.15.90:0/1700012001 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f91dc1031d0 0x7f91dc10f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.325+0000 7f91e1100640 1 --2- 172.21.15.90:0/1700012001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91dc108f20 0x7f91dc102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.325+0000 7f91e1100640 1 --2- 172.21.15.90:0/1700012001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91dc108a80 0x7f91dc102620 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.326+0000 7f91e1100640 1 -- 172.21.15.90:0/1700012001 >> 172.21.15.90:0/1700012001 conn(0x7f91dc078150 msgr2=0x7f91dc078590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:00.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.326+0000 7f91e1100640 1 -- 172.21.15.90:0/1700012001 shutdown_connections 2024-04-25T11:37:00.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.326+0000 7f91e1100640 1 -- 172.21.15.90:0/1700012001 wait complete. 2024-04-25T11:37:00.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.327+0000 7f91e1100640 1 Processor -- start 2024-04-25T11:37:00.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.328+0000 7f91e1100640 1 -- start start 2024-04-25T11:37:00.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.328+0000 7f91e1100640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91dc1031d0 0x7f91dc198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:00.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.328+0000 7f91e1100640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91dc108a80 0x7f91dc198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:00.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.328+0000 7f91e1100640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f91dc108f20 0x7f91dc19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:00.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.328+0000 7f91e1100640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f91dc111ca0 con 0x7f91dc1031d0 2024-04-25T11:37:00.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.328+0000 7f91e1100640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f91dc111b20 con 0x7f91dc108f20 2024-04-25T11:37:00.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.328+0000 7f91e1100640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f91dc111e20 con 0x7f91dc108a80 2024-04-25T11:37:00.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.328+0000 7f91dad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91dc1031d0 0x7f91dc198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:00.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.328+0000 7f91dad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91dc1031d0 0x7f91dc198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57064/0 (socket says 172.21.15.90:57064) 2024-04-25T11:37:00.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.328+0000 7f91dad9d640 1 -- 172.21.15.90:0/4188839203 learned_addr learned my addr 172.21.15.90:0/4188839203 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:00.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.329+0000 7f91da59c640 1 --2- 172.21.15.90:0/4188839203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91dc108a80 0x7f91dc198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:00.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.330+0000 7f91dad9d640 1 -- 172.21.15.90:0/4188839203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91dc108a80 msgr2=0x7f91dc198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:00.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.330+0000 7f91dad9d640 1 --2- 172.21.15.90:0/4188839203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91dc108a80 0x7f91dc198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.330+0000 7f91dad9d640 1 -- 172.21.15.90:0/4188839203 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f91dc108f20 msgr2=0x7f91dc19cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:00.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.330+0000 7f91dad9d640 1 --2- 172.21.15.90:0/4188839203 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f91dc108f20 0x7f91dc19cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.330+0000 7f91dad9d640 1 -- 172.21.15.90:0/4188839203 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91dc19d360 con 0x7f91dc1031d0 2024-04-25T11:37:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.330+0000 7f91dad9d640 1 --2- 172.21.15.90:0/4188839203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91dc1031d0 0x7f91dc198500 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f91cc030710 tx=0x7f91cc030740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.330+0000 7f91c3fff640 1 -- 172.21.15.90:0/4188839203 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91cc03f070 con 0x7f91dc1031d0 2024-04-25T11:37:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.330+0000 7f91e1100640 1 -- 172.21.15.90:0/4188839203 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91dc19d5f0 con 0x7f91dc1031d0 2024-04-25T11:37:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.330+0000 7f91c3fff640 1 -- 172.21.15.90:0/4188839203 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91cc030e10 con 0x7f91dc1031d0 2024-04-25T11:37:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.330+0000 7f91c3fff640 1 -- 172.21.15.90:0/4188839203 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91cc035070 con 0x7f91dc1031d0 2024-04-25T11:37:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.331+0000 7f91e1100640 1 -- 172.21.15.90:0/4188839203 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91dc19db30 con 0x7f91dc1031d0 2024-04-25T11:37:00.324 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.335+0000 7f91c3fff640 1 -- 172.21.15.90:0/4188839203 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f91cc042070 con 0x7f91dc1031d0 2024-04-25T11:37:00.324 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.336+0000 7f91e7fb5740 1 Processor -- start 2024-04-25T11:37:00.325 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.336+0000 7f91e7fb5740 1 -- start start 2024-04-25T11:37:00.326 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.336+0000 7f91e7fb5740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc9e711b0 0x555cca210af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:00.326 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.337+0000 7f91c3fff640 1 --2- 172.21.15.90:0/4188839203 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f91c40b1f30 0x7f91c40b4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:00.327 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.337+0000 7f91e7fb5740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc9e482f0 0x555cca211030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:00.327 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.337+0000 7f91e7fb5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e504d0 0x555cca211570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:00.327 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.337+0000 7f91dad9d640 1 --2- 172.21.15.90:0/4188839203 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f91c40b1f30 0x7f91c40b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:00.327 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.337+0000 7f91db59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e504d0 0x555cca211570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:00.327 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.337+0000 7f91db59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e504d0 0x555cca211570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57066/0 (socket says 172.21.15.90:57066) 2024-04-25T11:37:00.327 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.337+0000 7f91c3fff640 1 -- 172.21.15.90:0/4188839203 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f91cc0ffee0 con 0x7f91dc1031d0 2024-04-25T11:37:00.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.337+0000 7f91db59e640 1 -- 172.21.15.90:0/1679837591 learned_addr learned my addr 172.21.15.90:0/1679837591 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:00.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.337+0000 7f91e7fb5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555cca177460 con 0x555cc9e504d0 2024-04-25T11:37:00.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.338+0000 7f91dad9d640 1 --2- 172.21.15.90:0/4188839203 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f91c40b1f30 0x7f91c40b4440 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f91cc0fdbd0 tx=0x7f91cc005fd0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:00.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.338+0000 7f91da59c640 1 --2- 172.21.15.90:0/1679837591 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc9e482f0 0x555cca211030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:00.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.338+0000 7f91e7fb5740 1 -- 172.21.15.90:0/1679837591 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x555cca131590 con 0x555cc9e482f0 2024-04-25T11:37:00.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.338+0000 7f91e7fb5740 1 -- 172.21.15.90:0/1679837591 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x555cca173420 con 0x555cc9e711b0 2024-04-25T11:37:00.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.338+0000 7f91db59e640 1 -- 172.21.15.90:0/1679837591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc9e711b0 msgr2=0x555cca210af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:00.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.338+0000 7f91db59e640 1 --2- 172.21.15.90:0/1679837591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc9e711b0 0x555cca210af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.338+0000 7f91db59e640 1 -- 172.21.15.90:0/1679837591 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc9e482f0 msgr2=0x555cca211030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:00.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.338+0000 7f91db59e640 1 --2- 172.21.15.90:0/1679837591 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc9e482f0 0x555cca211030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.338+0000 7f91db59e640 1 -- 172.21.15.90:0/1679837591 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555cca13e0d0 con 0x555cc9e504d0 2024-04-25T11:37:00.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.339+0000 7f91db59e640 1 --2- 172.21.15.90:0/1679837591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e504d0 0x555cca211570 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f91d4002100 tx=0x7f91d40021f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=324d1aea7f107e2e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:00.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.339+0000 7f91c17fa640 1 -- 172.21.15.90:0/1679837591 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d4019070 con 0x555cc9e504d0 2024-04-25T11:37:00.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.339+0000 7f91c17fa640 1 -- 172.21.15.90:0/1679837591 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91d4014b00 con 0x555cc9e504d0 2024-04-25T11:37:00.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.339+0000 7f91c17fa640 1 -- 172.21.15.90:0/1679837591 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d4010070 con 0x555cc9e504d0 2024-04-25T11:37:00.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.341+0000 7f91e7fb5740 1 -- 172.21.15.90:0/1679837591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e504d0 msgr2=0x555cca211570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:00.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.341+0000 7f91e7fb5740 1 --2- 172.21.15.90:0/1679837591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e504d0 0x555cca211570 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f91d4002100 tx=0x7f91d40021f0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.341+0000 7f91e7fb5740 1 -- 172.21.15.90:0/1679837591 shutdown_connections 2024-04-25T11:37:00.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.341+0000 7f91e7fb5740 1 --2- 172.21.15.90:0/1679837591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e504d0 0x555cca211570 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.341+0000 7f91e7fb5740 1 --2- 172.21.15.90:0/1679837591 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc9e482f0 0x555cca211030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.341+0000 7f91e7fb5740 1 --2- 172.21.15.90:0/1679837591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc9e711b0 0x555cca210af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.341+0000 7f91e7fb5740 1 -- 172.21.15.90:0/1679837591 >> 172.21.15.90:0/1679837591 conn(0x555cc9e65300 msgr2=0x555cca20d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:00.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.341+0000 7f91e7fb5740 1 -- 172.21.15.90:0/1679837591 shutdown_connections 2024-04-25T11:37:00.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.341+0000 7f91e7fb5740 1 -- 172.21.15.90:0/1679837591 wait complete. 2024-04-25T11:37:00.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.342+0000 7f91e7fb5740 1 -- start start 2024-04-25T11:37:00.332 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.342+0000 7f91e7fb5740 1 Processor -- start 2024-04-25T11:37:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.344+0000 7f91e7fb5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e482f0 0x555cca21cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.344+0000 7f91e7fb5740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc9e504d0 0x555cca21d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.344+0000 7f91e7fb5740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc9e711b0 0x555cca21b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.344+0000 7f91e7fb5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555cca21dc20 con 0x555cc9e482f0 2024-04-25T11:37:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.344+0000 7f91e7fb5740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x555cca0feb60 con 0x555cc9e711b0 2024-04-25T11:37:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.344+0000 7f91e7fb5740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x555cca21dda0 con 0x555cc9e504d0 2024-04-25T11:37:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.344+0000 7f91e7fb5740 10 client.0 _ll_register_callbacks cb 0x555cca1232b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:37:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.344+0000 7f91db59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e482f0 0x555cca21cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.344+0000 7f91db59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e482f0 0x555cca21cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57076/0 (socket says 172.21.15.90:57076) 2024-04-25T11:37:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.345+0000 7f91db59e640 1 -- 172.21.15.90:0/2920608386 learned_addr learned my addr 172.21.15.90:0/2920608386 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.345+0000 7f91db59e640 1 -- 172.21.15.90:0/2920608386 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc9e504d0 msgr2=0x555cca21d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.345+0000 7f91db59e640 1 --2- 172.21.15.90:0/2920608386 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc9e504d0 0x555cca21d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.346+0000 7f91db59e640 1 -- 172.21.15.90:0/2920608386 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc9e711b0 msgr2=0x555cca21b050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:37:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.346+0000 7f91db59e640 1 --2- 172.21.15.90:0/2920608386 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc9e711b0 0x555cca21b050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.346+0000 7f91db59e640 1 -- 172.21.15.90:0/2920608386 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555cc9e89210 con 0x555cc9e482f0 2024-04-25T11:37:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.346+0000 7f91db59e640 1 --2- 172.21.15.90:0/2920608386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e482f0 0x555cca21cb40 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f91d400a040 tx=0x7f91d4014680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32424d1d74a119e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.347+0000 7f91beff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:37:00.339 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.350+0000 7f91beff5640 1 -- 172.21.15.90:0/2920608386 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d4019040 con 0x555cc9e482f0 2024-04-25T11:37:00.339 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.350+0000 7f91beff5640 1 -- 172.21.15.90:0/2920608386 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f91d4012610 con 0x555cc9e482f0 2024-04-25T11:37:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.350+0000 7f91e7fb5740 1 -- 172.21.15.90:0/2920608386 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91d4014b00 con 0x555cc9e482f0 2024-04-25T11:37:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.351+0000 7f91beff5640 1 -- 172.21.15.90:0/2920608386 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f91d4010040 con 0x555cc9e482f0 2024-04-25T11:37:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.351+0000 7f91e7fb5740 10 client.16188 fetch_fsmap learned FSMap version 26 2024-04-25T11:37:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.351+0000 7f91e7fb5740 1 -- 172.21.15.90:0/2920608386 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555cc9f55620 con 0x555cc9e482f0 2024-04-25T11:37:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.351+0000 7f91beff5640 1 -- 172.21.15.90:0/2920608386 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91d4012cc0 con 0x555cc9e482f0 2024-04-25T11:37:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.352+0000 7f91beff5640 1 -- 172.21.15.90:0/2920608386 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f91d401e950 con 0x555cc9e482f0 2024-04-25T11:37:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.352+0000 7f91e7fb5740 10 client.16188 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:37:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.352+0000 7f91e7fb5740 10 client.16188 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:37:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.352+0000 7f91e7fb5740 20 client.16188 populate_metadata read hostname 'smithi090' 2024-04-25T11:37:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.352+0000 7f91e7fb5740 1 --2- 172.21.15.90:0/2920608386 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x555cc9e2fd80 0x555cca231860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.352+0000 7f91e7fb5740 4 client.16188 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:37:00.342 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.352+0000 7f91e7fb5740 1 -- 172.21.15.90:0/2920608386 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f91d4014b00 con 0x555cc9e2fd80 2024-04-25T11:37:00.342 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.354+0000 7f91da59c640 1 --2- 172.21.15.90:0/2920608386 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x555cc9e2fd80 0x555cca231860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:00.344 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.356+0000 7f91beff5640 10 client.16188 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:00.345 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.356+0000 7f91da59c640 1 --2- 172.21.15.90:0/2920608386 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x555cc9e2fd80 0x555cca231860 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=761e6acdb5740af4 server_cookie=ce7f563359eea9b1 in_seq=0 out_seq=0 2024-04-25T11:37:00.351 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.362+0000 7f91beff5640 1 -- 172.21.15.90:0/2920608386 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f91d4014b00 con 0x555cc9e2fd80 2024-04-25T11:37:00.351 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.363+0000 7f91beff5640 10 client.16188 handle_command_reply: tid=0 2024-04-25T11:37:00.351 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.363+0000 7f91e7fb5740 1 client.16188 shutdown 2024-04-25T11:37:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.363+0000 7f91e7fb5740 1 -- 172.21.15.90:0/2920608386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e482f0 msgr2=0x555cca21cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.363+0000 7f91e7fb5740 1 --2- 172.21.15.90:0/2920608386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e482f0 0x555cca21cb40 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f91d400a040 tx=0x7f91d4014680 comp rx=0 tx=0).stop 2024-04-25T11:37:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.363+0000 7f91e7fb5740 1 -- 172.21.15.90:0/2920608386 shutdown_connections 2024-04-25T11:37:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.363+0000 7f91e7fb5740 1 --2- 172.21.15.90:0/2920608386 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x555cc9e2fd80 0x555cca231860 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.363+0000 7f91e7fb5740 1 --2- 172.21.15.90:0/2920608386 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc9e711b0 0x555cca21b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.363+0000 7f91e7fb5740 1 --2- 172.21.15.90:0/2920608386 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc9e504d0 0x555cca21d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.363+0000 7f91beff5640 0 client.16188 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.363+0000 7f91e7fb5740 1 --2- 172.21.15.90:0/2920608386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e482f0 0x555cca21cb40 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.363+0000 7f91e7fb5740 1 -- 172.21.15.90:0/2920608386 >> 172.21.15.90:0/2920608386 conn(0x555cc9e65300 msgr2=0x555cca212810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:00.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.363+0000 7f91e7fb5740 1 -- 172.21.15.90:0/2920608386 shutdown_connections 2024-04-25T11:37:00.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.363+0000 7f91e7fb5740 1 -- 172.21.15.90:0/2920608386 wait complete. 2024-04-25T11:37:00.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.364+0000 7f91e7fb5740 20 client.16188 trim_cache size 0 max 16384 2024-04-25T11:37:00.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.396+0000 7f91e7fb5740 1 Processor -- start 2024-04-25T11:37:00.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.396+0000 7f91e7fb5740 1 -- start start 2024-04-25T11:37:00.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.397+0000 7f91e7fb5740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc9e2fd80 0x555cca21ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:00.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.397+0000 7f91e7fb5740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc9e482f0 0x555cca234890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:00.386 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.397+0000 7f91e7fb5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e504d0 0x555cca234dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:00.386 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.397+0000 7f91e7fb5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f91dc1ad970 con 0x555cc9e504d0 2024-04-25T11:37:00.386 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.397+0000 7f91e7fb5740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f91dc1af4b0 con 0x555cc9e2fd80 2024-04-25T11:37:00.386 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.397+0000 7f91e7fb5740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f91dc1ae040 con 0x555cc9e482f0 2024-04-25T11:37:00.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.397+0000 7f91db59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e504d0 0x555cca234dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:00.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.397+0000 7f91db59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e504d0 0x555cca234dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57088/0 (socket says 172.21.15.90:57088) 2024-04-25T11:37:00.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.397+0000 7f91db59e640 1 -- 172.21.15.90:0/520184983 learned_addr learned my addr 172.21.15.90:0/520184983 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:00.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.397+0000 7f91db59e640 1 --2- 172.21.15.90:0/520184983 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc9e2fd80 0x555cca21ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:00.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.398+0000 7f91db59e640 1 -- 172.21.15.90:0/520184983 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc9e482f0 msgr2=0x555cca234890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:00.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.398+0000 7f91db59e640 1 --2- 172.21.15.90:0/520184983 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc9e482f0 0x555cca234890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.398+0000 7f91db59e640 1 -- 172.21.15.90:0/520184983 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc9e2fd80 msgr2=0x555cca21ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:00.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.398+0000 7f91db59e640 1 --2- 172.21.15.90:0/520184983 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc9e2fd80 0x555cca21ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.398+0000 7f91db59e640 1 -- 172.21.15.90:0/520184983 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555cca235310 con 0x555cc9e504d0 2024-04-25T11:37:00.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.398+0000 7f91db59e640 1 --2- 172.21.15.90:0/520184983 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e504d0 0x555cca234dd0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f91d401ed20 tx=0x7f91d401ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfe59117919e5d08 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:00.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.398+0000 7f91c0ff9640 1 -- 172.21.15.90:0/520184983 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d4001450 con 0x555cc9e504d0 2024-04-25T11:37:00.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.398+0000 7f91c0ff9640 1 -- 172.21.15.90:0/520184983 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91d4003e20 con 0x555cc9e504d0 2024-04-25T11:37:00.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.398+0000 7f91c0ff9640 1 -- 172.21.15.90:0/520184983 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d401e6b0 con 0x555cc9e504d0 2024-04-25T11:37:00.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.400+0000 7f91e7fb5740 1 -- 172.21.15.90:0/520184983 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e504d0 msgr2=0x555cca234dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:00.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.400+0000 7f91e7fb5740 1 --2- 172.21.15.90:0/520184983 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e504d0 0x555cca234dd0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f91d401ed20 tx=0x7f91d401ef90 comp rx=0 tx=0).stop 2024-04-25T11:37:00.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.400+0000 7f91e7fb5740 1 -- 172.21.15.90:0/520184983 shutdown_connections 2024-04-25T11:37:00.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.400+0000 7f91e7fb5740 1 --2- 172.21.15.90:0/520184983 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e504d0 0x555cca234dd0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.400+0000 7f91e7fb5740 1 --2- 172.21.15.90:0/520184983 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc9e482f0 0x555cca234890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.400+0000 7f91e7fb5740 1 --2- 172.21.15.90:0/520184983 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc9e2fd80 0x555cca21ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.400+0000 7f91e7fb5740 1 -- 172.21.15.90:0/520184983 >> 172.21.15.90:0/520184983 conn(0x555cc9e65300 msgr2=0x555cca21c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:00.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.400+0000 7f91e7fb5740 1 -- 172.21.15.90:0/520184983 shutdown_connections 2024-04-25T11:37:00.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.400+0000 7f91e7fb5740 1 -- 172.21.15.90:0/520184983 wait complete. 2024-04-25T11:37:00.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.401+0000 7f91e7fb5740 1 -- start start 2024-04-25T11:37:00.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.401+0000 7f91e7fb5740 1 Processor -- start 2024-04-25T11:37:00.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.402+0000 7f91e7fb5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e2fd80 0x555cca22b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:00.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.402+0000 7f91e7fb5740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc9e482f0 0x555cca22bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:00.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.402+0000 7f91e7fb5740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc9e504d0 0x555cca296130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:00.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.402+0000 7f91e7fb5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f91dc1aede0 con 0x555cc9e2fd80 2024-04-25T11:37:00.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.402+0000 7f91e7fb5740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f91dc1ae710 con 0x555cc9e504d0 2024-04-25T11:37:00.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.402+0000 7f91e7fb5740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f91dc1ab950 con 0x555cc9e482f0 2024-04-25T11:37:00.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.402+0000 7f91e7fb5740 10 client.0 _ll_register_callbacks cb 0x7f91d4017070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:37:00.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.403+0000 7f91db59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e2fd80 0x555cca22b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:00.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.403+0000 7f91db59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e2fd80 0x555cca22b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57104/0 (socket says 172.21.15.90:57104) 2024-04-25T11:37:00.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.403+0000 7f91db59e640 1 -- 172.21.15.90:0/18326282 learned_addr learned my addr 172.21.15.90:0/18326282 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.403+0000 7f91db59e640 1 --2- 172.21.15.90:0/18326282 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc9e504d0 0x555cca296130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.404+0000 7f91db59e640 1 -- 172.21.15.90:0/18326282 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc9e482f0 msgr2=0x555cca22bce0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:37:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.404+0000 7f91db59e640 1 --2- 172.21.15.90:0/18326282 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc9e482f0 0x555cca22bce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.404+0000 7f91db59e640 1 -- 172.21.15.90:0/18326282 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc9e504d0 msgr2=0x555cca296130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.404+0000 7f91db59e640 1 --2- 172.21.15.90:0/18326282 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc9e504d0 0x555cca296130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.404+0000 7f91db59e640 1 -- 172.21.15.90:0/18326282 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555cca296720 con 0x555cc9e2fd80 2024-04-25T11:37:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.404+0000 7f91db59e640 1 --2- 172.21.15.90:0/18326282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e2fd80 0x555cca22b7a0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f91d4012510 tx=0x7f91d4014660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=680b9373ffcf738 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.405+0000 7f91beff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:37:00.394 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.406+0000 7f91beff5640 1 -- 172.21.15.90:0/18326282 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d4019040 con 0x555cc9e2fd80 2024-04-25T11:37:00.394 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.406+0000 7f91beff5640 1 -- 172.21.15.90:0/18326282 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f91d4003970 con 0x555cc9e2fd80 2024-04-25T11:37:00.394 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.406+0000 7f91e7fb5740 1 -- 172.21.15.90:0/18326282 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91d4003e20 con 0x555cc9e2fd80 2024-04-25T11:37:00.395 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.407+0000 7f91beff5640 1 -- 172.21.15.90:0/18326282 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f91d4003b70 con 0x555cc9e2fd80 2024-04-25T11:37:00.398 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.410+0000 7f91e7fb5740 10 client.16194 fetch_fsmap learned FSMap version 26 2024-04-25T11:37:00.399 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.410+0000 7f91e7fb5740 1 -- 172.21.15.90:0/18326282 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555cca29da90 con 0x555cc9e2fd80 2024-04-25T11:37:00.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.411+0000 7f91beff5640 1 -- 172.21.15.90:0/18326282 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91d4003e20 con 0x555cc9e2fd80 2024-04-25T11:37:00.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.411+0000 7f91beff5640 1 -- 172.21.15.90:0/18326282 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f91d4012680 con 0x555cc9e2fd80 2024-04-25T11:37:00.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.411+0000 7f91e7fb5740 10 client.16194 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T11:37:00.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.411+0000 7f91e7fb5740 10 client.16194 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:37:00.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.412+0000 7f91e7fb5740 20 client.16194 populate_metadata read hostname 'smithi090' 2024-04-25T11:37:00.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.412+0000 7f91e7fb5740 1 --2- 172.21.15.90:0/18326282 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x555cc9e711b0 0x555cca2a0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:00.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.412+0000 7f91e7fb5740 4 client.16194 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:37:00.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.412+0000 7f91e7fb5740 1 -- 172.21.15.90:0/18326282 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555cca185070 con 0x555cc9e711b0 2024-04-25T11:37:00.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.416+0000 7f91da59c640 1 --2- 172.21.15.90:0/18326282 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x555cc9e711b0 0x555cca2a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:00.406 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.418+0000 7f91da59c640 1 --2- 172.21.15.90:0/18326282 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x555cc9e711b0 0x555cca2a0160 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca3d31bfc62a8b6 server_cookie=a915861c63dc145e in_seq=0 out_seq=0 2024-04-25T11:37:00.406 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:00.418+0000 7f91beff5640 10 client.16194 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:01.725 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:01 smithi090 ceph-mon[29575]: pgmap v61: 97 pgs: 97 active+clean; 8.7 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 780 MiB/s wr, 929 op/s 2024-04-25T11:37:01.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:01 smithi138 ceph-mon[32328]: pgmap v61: 97 pgs: 97 active+clean; 8.7 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 780 MiB/s wr, 929 op/s 2024-04-25T11:37:01.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:01 smithi156 ceph-mon[33045]: pgmap v61: 97 pgs: 97 active+clean; 8.7 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 780 MiB/s wr, 929 op/s 2024-04-25T11:37:03.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.487+0000 7f91beff5640 1 -- 172.21.15.90:0/18326282 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555cca185070 con 0x555cc9e711b0 2024-04-25T11:37:03.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.487+0000 7f91beff5640 10 client.16194 handle_command_reply: tid=0 2024-04-25T11:37:03.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.487+0000 7f91e7fb5740 1 client.16194 shutdown 2024-04-25T11:37:03.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.487+0000 7f91e7fb5740 1 -- 172.21.15.90:0/18326282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e2fd80 msgr2=0x555cca22b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:03.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.487+0000 7f91e7fb5740 1 --2- 172.21.15.90:0/18326282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e2fd80 0x555cca22b7a0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f91d4012510 tx=0x7f91d4014660 comp rx=0 tx=0).stop 2024-04-25T11:37:03.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.487+0000 7f91e7fb5740 1 -- 172.21.15.90:0/18326282 shutdown_connections 2024-04-25T11:37:03.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.487+0000 7f91e7fb5740 1 --2- 172.21.15.90:0/18326282 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x555cc9e711b0 0x555cca2a0160 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:03.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.487+0000 7f91e7fb5740 1 --2- 172.21.15.90:0/18326282 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc9e504d0 0x555cca296130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:03.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.487+0000 7f91e7fb5740 1 --2- 172.21.15.90:0/18326282 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc9e482f0 0x555cca22bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:03.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.487+0000 7f91e7fb5740 1 --2- 172.21.15.90:0/18326282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc9e2fd80 0x555cca22b7a0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:03.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.487+0000 7f91e7fb5740 1 -- 172.21.15.90:0/18326282 >> 172.21.15.90:0/18326282 conn(0x555cc9e65300 msgr2=0x555cca21cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:03.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.487+0000 7f91beff5640 0 client.16194 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:03.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.488+0000 7f91e7fb5740 1 -- 172.21.15.90:0/18326282 shutdown_connections 2024-04-25T11:37:03.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.488+0000 7f91e7fb5740 1 -- 172.21.15.90:0/18326282 wait complete. 2024-04-25T11:37:03.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.488+0000 7f91e7fb5740 20 client.16194 trim_cache size 0 max 16384 2024-04-25T11:37:03.478 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:37:03.478 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:37:03.478 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:37:03.478 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:37:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.488+0000 7f91e1100640 1 -- 172.21.15.90:0/4188839203 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f91c40b1f30 msgr2=0x7f91c40b4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.488+0000 7f91e1100640 1 --2- 172.21.15.90:0/4188839203 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f91c40b1f30 0x7f91c40b4440 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f91cc0fdbd0 tx=0x7f91cc005fd0 comp rx=0 tx=0).stop 2024-04-25T11:37:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.488+0000 7f91e1100640 1 -- 172.21.15.90:0/4188839203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91dc1031d0 msgr2=0x7f91dc198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.488+0000 7f91e1100640 1 --2- 172.21.15.90:0/4188839203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91dc1031d0 0x7f91dc198500 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f91cc030710 tx=0x7f91cc030740 comp rx=0 tx=0).stop 2024-04-25T11:37:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.488+0000 7f91e1100640 1 -- 172.21.15.90:0/4188839203 shutdown_connections 2024-04-25T11:37:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.488+0000 7f91e1100640 1 --2- 172.21.15.90:0/4188839203 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f91c40b1f30 0x7f91c40b4440 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.488+0000 7f91e1100640 1 --2- 172.21.15.90:0/4188839203 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f91dc108f20 0x7f91dc19cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.488+0000 7f91e1100640 1 --2- 172.21.15.90:0/4188839203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91dc108a80 0x7f91dc198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.488+0000 7f91e1100640 1 --2- 172.21.15.90:0/4188839203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91dc1031d0 0x7f91dc198500 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.488+0000 7f91e1100640 1 -- 172.21.15.90:0/4188839203 >> 172.21.15.90:0/4188839203 conn(0x7f91dc078150 msgr2=0x7f91dc104120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.488+0000 7f91e1100640 1 -- 172.21.15.90:0/4188839203 shutdown_connections 2024-04-25T11:37:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:03.488+0000 7f91e1100640 1 -- 172.21.15.90:0/4188839203 wait complete. 2024-04-25T11:37:03.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:03 smithi090 ceph-mon[29575]: pgmap v62: 97 pgs: 97 active+clean; 8.7 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 658 MiB/s wr, 837 op/s 2024-04-25T11:37:03.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:03 smithi138 ceph-mon[32328]: pgmap v62: 97 pgs: 97 active+clean; 8.7 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 658 MiB/s wr, 837 op/s 2024-04-25T11:37:03.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:03 smithi156 ceph-mon[33045]: pgmap v62: 97 pgs: 97 active+clean; 8.7 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 658 MiB/s wr, 837 op/s 2024-04-25T11:37:05.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:05 smithi138 ceph-mon[32328]: pgmap v63: 97 pgs: 97 active+clean; 9.6 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 856 MiB/s wr, 1.14k op/s 2024-04-25T11:37:05.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:05 smithi156 ceph-mon[33045]: pgmap v63: 97 pgs: 97 active+clean; 9.6 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 856 MiB/s wr, 1.14k op/s 2024-04-25T11:37:05.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:05 smithi090 ceph-mon[29575]: pgmap v63: 97 pgs: 97 active+clean; 9.6 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 856 MiB/s wr, 1.14k op/s 2024-04-25T11:37:07.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:07 smithi156 ceph-mon[33045]: pgmap v64: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 568 MiB/s wr, 967 op/s 2024-04-25T11:37:07.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:07 smithi090 ceph-mon[29575]: pgmap v64: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 568 MiB/s wr, 967 op/s 2024-04-25T11:37:08.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:07 smithi138 ceph-mon[32328]: pgmap v64: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 568 MiB/s wr, 967 op/s 2024-04-25T11:37:09.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:09 smithi156 ceph-mon[33045]: pgmap v65: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 568 MiB/s wr, 967 op/s 2024-04-25T11:37:09.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:09 smithi090 ceph-mon[29575]: pgmap v65: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 568 MiB/s wr, 967 op/s 2024-04-25T11:37:10.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:09 smithi138 ceph-mon[32328]: pgmap v65: 97 pgs: 97 active+clean; 10 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 568 MiB/s wr, 967 op/s 2024-04-25T11:37:11.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:11 smithi090 ceph-mon[29575]: pgmap v66: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 796 MiB/s wr, 2.54k op/s 2024-04-25T11:37:12.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:11 smithi138 ceph-mon[32328]: pgmap v66: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 796 MiB/s wr, 2.54k op/s 2024-04-25T11:37:12.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:11 smithi156 ceph-mon[33045]: pgmap v66: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 796 MiB/s wr, 2.54k op/s 2024-04-25T11:37:13.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:13 smithi090 ceph-mon[29575]: pgmap v67: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 504 MiB/s wr, 1.99k op/s 2024-04-25T11:37:14.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:13 smithi138 ceph-mon[32328]: pgmap v67: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 504 MiB/s wr, 1.99k op/s 2024-04-25T11:37:14.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:13 smithi156 ceph-mon[33045]: pgmap v67: 97 pgs: 97 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 504 MiB/s wr, 1.99k op/s 2024-04-25T11:37:16.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:15 smithi138 ceph-mon[32328]: pgmap v68: 97 pgs: 97 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 644 MiB/s wr, 3.39k op/s 2024-04-25T11:37:16.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:15 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:37:16.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:15 smithi156 ceph-mon[33045]: pgmap v68: 97 pgs: 97 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 644 MiB/s wr, 3.39k op/s 2024-04-25T11:37:16.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:15 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:37:16.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:15 smithi090 ceph-mon[29575]: pgmap v68: 97 pgs: 97 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 644 MiB/s wr, 3.39k op/s 2024-04-25T11:37:16.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:15 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:37:16.629 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.640+0000 7f15c3db2640 1 -- 172.21.15.90:0/3366116303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15bc1042f0 msgr2=0x7f15bc1106c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.640+0000 7f15c3db2640 1 --2- 172.21.15.90:0/3366116303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15bc1042f0 0x7f15bc1106c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009ca0 tx=0x7f15b402a8c0 comp rx=0 tx=0).stop 2024-04-25T11:37:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.641+0000 7f15c3db2640 1 -- 172.21.15.90:0/3366116303 shutdown_connections 2024-04-25T11:37:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.641+0000 7f15c3db2640 1 --2- 172.21.15.90:0/3366116303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15bc1042f0 0x7f15bc1106c0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.641+0000 7f15c3db2640 1 --2- 172.21.15.90:0/3366116303 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f15bc10a040 0x7f15bc103c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.641+0000 7f15c3db2640 1 --2- 172.21.15.90:0/3366116303 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15bc109ba0 0x7f15bc103740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.641+0000 7f15c3db2640 1 -- 172.21.15.90:0/3366116303 >> 172.21.15.90:0/3366116303 conn(0x7f15bc0792a0 msgr2=0x7f15bc0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.641+0000 7f15c3db2640 1 -- 172.21.15.90:0/3366116303 shutdown_connections 2024-04-25T11:37:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.641+0000 7f15c3db2640 1 -- 172.21.15.90:0/3366116303 wait complete. 2024-04-25T11:37:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.641+0000 7f15c3db2640 1 Processor -- start 2024-04-25T11:37:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.641+0000 7f15c3db2640 1 -- start start 2024-04-25T11:37:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.642+0000 7f15c3db2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15bc1042f0 0x7f15bc19a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.642+0000 7f15c3db2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15bc109ba0 0x7f15bc19ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.642+0000 7f15c3db2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f15bc10a040 0x7f15bc19ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.642+0000 7f15c3db2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f15bc113e00 con 0x7f15bc1042f0 2024-04-25T11:37:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.642+0000 7f15c3db2640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f15bc113c80 con 0x7f15bc10a040 2024-04-25T11:37:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.642+0000 7f15c3db2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f15bc113f80 con 0x7f15bc109ba0 2024-04-25T11:37:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.642+0000 7f15c1b4e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15bc1042f0 0x7f15bc19a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.642+0000 7f15c1b4e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15bc1042f0 0x7f15bc19a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48834/0 (socket says 172.21.15.90:48834) 2024-04-25T11:37:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.642+0000 7f15c1b4e640 1 -- 172.21.15.90:0/778700224 learned_addr learned my addr 172.21.15.90:0/778700224 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.642+0000 7f15c134d640 1 --2- 172.21.15.90:0/778700224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15bc109ba0 0x7f15bc19ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.642+0000 7f15c234f640 1 --2- 172.21.15.90:0/778700224 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f15bc10a040 0x7f15bc19ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.642+0000 7f15c1b4e640 1 -- 172.21.15.90:0/778700224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15bc109ba0 msgr2=0x7f15bc19ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.642+0000 7f15c1b4e640 1 --2- 172.21.15.90:0/778700224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15bc109ba0 0x7f15bc19ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.642+0000 7f15c1b4e640 1 -- 172.21.15.90:0/778700224 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f15bc10a040 msgr2=0x7f15bc19ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.642+0000 7f15c1b4e640 1 --2- 172.21.15.90:0/778700224 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f15bc10a040 0x7f15bc19ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.642+0000 7f15c1b4e640 1 -- 172.21.15.90:0/778700224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15bc19f560 con 0x7f15bc1042f0 2024-04-25T11:37:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.642+0000 7f15c1b4e640 1 --2- 172.21.15.90:0/778700224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15bc1042f0 0x7f15bc19a700 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f15b800e9f0 tx=0x7f15b800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.643+0000 7f15b2ffd640 1 -- 172.21.15.90:0/778700224 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b800ccc0 con 0x7f15bc1042f0 2024-04-25T11:37:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.643+0000 7f15c3db2640 1 -- 172.21.15.90:0/778700224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15bc19f850 con 0x7f15bc1042f0 2024-04-25T11:37:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.643+0000 7f15b2ffd640 1 -- 172.21.15.90:0/778700224 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15b80054d0 con 0x7f15bc1042f0 2024-04-25T11:37:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.643+0000 7f15b2ffd640 1 -- 172.21.15.90:0/778700224 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b8010430 con 0x7f15bc1042f0 2024-04-25T11:37:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.643+0000 7f15c3db2640 1 -- 172.21.15.90:0/778700224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15bc19fd50 con 0x7f15bc1042f0 2024-04-25T11:37:16.635 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.646+0000 7f15cac66740 1 Processor -- start 2024-04-25T11:37:16.635 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.646+0000 7f15cac66740 1 -- start start 2024-04-25T11:37:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.646+0000 7f15cac66740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56482f9fb1b0 0x56482fd9aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.646+0000 7f15cac66740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56482f9d22f0 0x56482fd9b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.646+0000 7f15cac66740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56482f9da4d0 0x56482fd9b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.646+0000 7f15cac66740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56482fd01460 con 0x56482f9da4d0 2024-04-25T11:37:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.646+0000 7f15cac66740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56482fcbb590 con 0x56482f9d22f0 2024-04-25T11:37:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.646+0000 7f15cac66740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56482fcfd420 con 0x56482f9fb1b0 2024-04-25T11:37:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.647+0000 7f15b2ffd640 1 -- 172.21.15.90:0/778700224 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f15b80046c0 con 0x7f15bc1042f0 2024-04-25T11:37:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.647+0000 7f15c234f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56482f9fb1b0 0x56482fd9aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.647+0000 7f15c234f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56482f9fb1b0 0x56482fd9aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:40618/0 (socket says 172.21.15.90:40618) 2024-04-25T11:37:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.647+0000 7f15c234f640 1 -- 172.21.15.90:0/839295344 learned_addr learned my addr 172.21.15.90:0/839295344 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.647+0000 7f15c134d640 1 --2- 172.21.15.90:0/839295344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56482f9da4d0 0x56482fd9b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.647+0000 7f15c1b4e640 1 --2- 172.21.15.90:0/839295344 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56482f9d22f0 0x56482fd9b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.647+0000 7f15c234f640 1 -- 172.21.15.90:0/839295344 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56482f9d22f0 msgr2=0x56482fd9b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.647+0000 7f15c234f640 1 --2- 172.21.15.90:0/839295344 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56482f9d22f0 0x56482fd9b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.647+0000 7f15c234f640 1 -- 172.21.15.90:0/839295344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56482f9da4d0 msgr2=0x56482fd9b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.647+0000 7f15c234f640 1 --2- 172.21.15.90:0/839295344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56482f9da4d0 0x56482fd9b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.647+0000 7f15c234f640 1 -- 172.21.15.90:0/839295344 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56482fcc80d0 con 0x56482f9fb1b0 2024-04-25T11:37:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.647+0000 7f15c234f640 1 --2- 172.21.15.90:0/839295344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56482f9fb1b0 0x56482fd9aaf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009ca0 tx=0x7f15b4009430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5b27d739bcdc993 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.647+0000 7f15a7fff640 1 -- 172.21.15.90:0/839295344 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b4031cf0 con 0x56482f9fb1b0 2024-04-25T11:37:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.648+0000 7f15a7fff640 1 -- 172.21.15.90:0/839295344 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15b4036030 con 0x56482f9fb1b0 2024-04-25T11:37:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.648+0000 7f15a7fff640 1 -- 172.21.15.90:0/839295344 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b4004280 con 0x56482f9fb1b0 2024-04-25T11:37:16.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.648+0000 7f15b2ffd640 1 --2- 172.21.15.90:0/778700224 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f15a80afc50 0x7f15a80b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:16.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.648+0000 7f15c1b4e640 1 --2- 172.21.15.90:0/778700224 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f15a80afc50 0x7f15a80b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:16.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.648+0000 7f15b2ffd640 1 -- 172.21.15.90:0/778700224 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f15b80df420 con 0x7f15bc1042f0 2024-04-25T11:37:16.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.649+0000 7f15cac66740 1 -- 172.21.15.90:0/839295344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56482f9fb1b0 msgr2=0x56482fd9aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:16.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.649+0000 7f15cac66740 1 --2- 172.21.15.90:0/839295344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56482f9fb1b0 0x56482fd9aaf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009ca0 tx=0x7f15b4009430 comp rx=0 tx=0).stop 2024-04-25T11:37:16.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.649+0000 7f15c1b4e640 1 --2- 172.21.15.90:0/778700224 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f15a80afc50 0x7f15a80b2160 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f15b80dcbc0 tx=0x7f15b80e1040 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:16.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.649+0000 7f15cac66740 1 -- 172.21.15.90:0/839295344 shutdown_connections 2024-04-25T11:37:16.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.649+0000 7f15cac66740 1 --2- 172.21.15.90:0/839295344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56482f9da4d0 0x56482fd9b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:16.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.649+0000 7f15cac66740 1 --2- 172.21.15.90:0/839295344 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56482f9d22f0 0x56482fd9b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:16.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.649+0000 7f15cac66740 1 --2- 172.21.15.90:0/839295344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56482f9fb1b0 0x56482fd9aaf0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:16.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.649+0000 7f15cac66740 1 -- 172.21.15.90:0/839295344 >> 172.21.15.90:0/839295344 conn(0x56482f9ef300 msgr2=0x56482fd972a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:16.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.649+0000 7f15cac66740 1 -- 172.21.15.90:0/839295344 shutdown_connections 2024-04-25T11:37:16.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.649+0000 7f15cac66740 1 -- 172.21.15.90:0/839295344 wait complete. 2024-04-25T11:37:16.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.649+0000 7f15cac66740 1 -- start start 2024-04-25T11:37:16.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.650+0000 7f15cac66740 1 Processor -- start 2024-04-25T11:37:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.650+0000 7f15cac66740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56482f9d22f0 0x56482fda51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.650+0000 7f15cac66740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56482f9da4d0 0x56482fda5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.650+0000 7f15cac66740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56482f9fb1b0 0x56482fdaaf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.650+0000 7f15cac66740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56482fda7a40 con 0x56482f9d22f0 2024-04-25T11:37:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.650+0000 7f15cac66740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56482fc88b60 con 0x56482f9fb1b0 2024-04-25T11:37:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.650+0000 7f15cac66740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56482fda7bc0 con 0x56482f9da4d0 2024-04-25T11:37:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.651+0000 7f15cac66740 10 client.0 _ll_register_callbacks cb 0x56482fcad2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:37:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.651+0000 7f15c134d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56482f9da4d0 0x56482fda5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.651+0000 7f15c134d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56482f9da4d0 0x56482fda5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:40630/0 (socket says 172.21.15.90:40630) 2024-04-25T11:37:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.651+0000 7f15c134d640 1 -- 172.21.15.90:0/566630801 learned_addr learned my addr 172.21.15.90:0/566630801 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.651+0000 7f15c234f640 1 --2- 172.21.15.90:0/566630801 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56482f9d22f0 0x56482fda51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.651+0000 7f15c134d640 1 -- 172.21.15.90:0/566630801 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56482f9fb1b0 msgr2=0x56482fdaaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.651+0000 7f15c134d640 1 --2- 172.21.15.90:0/566630801 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56482f9fb1b0 0x56482fdaaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.651+0000 7f15c134d640 1 -- 172.21.15.90:0/566630801 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56482f9d22f0 msgr2=0x56482fda51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.651+0000 7f15c134d640 1 --2- 172.21.15.90:0/566630801 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56482f9d22f0 0x56482fda51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:16.642 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.651+0000 7f15c134d640 1 -- 172.21.15.90:0/566630801 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56482fa13210 con 0x56482f9da4d0 2024-04-25T11:37:16.642 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.651+0000 7f15c234f640 1 --2- 172.21.15.90:0/566630801 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56482f9d22f0 0x56482fda51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:37:16.642 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.651+0000 7f15c134d640 1 --2- 172.21.15.90:0/566630801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56482f9da4d0 0x56482fda5720 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f15ac012700 tx=0x7f15ac012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=553eaaccb585a44c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:16.642 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.651+0000 7f15a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:37:16.643 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.652+0000 7f15a57fa640 1 -- 172.21.15.90:0/566630801 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ac004050 con 0x56482f9da4d0 2024-04-25T11:37:16.643 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.652+0000 7f15a57fa640 1 -- 172.21.15.90:0/566630801 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f15ac0041f0 con 0x56482f9da4d0 2024-04-25T11:37:16.643 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.652+0000 7f15cac66740 1 -- 172.21.15.90:0/566630801 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15b4036030 con 0x56482f9da4d0 2024-04-25T11:37:16.643 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.652+0000 7f15a57fa640 1 -- 172.21.15.90:0/566630801 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f15ac0043f0 con 0x56482f9da4d0 2024-04-25T11:37:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.652+0000 7f15cac66740 10 client.25549 fetch_fsmap learned FSMap version 27 2024-04-25T11:37:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.652+0000 7f15cac66740 1 -- 172.21.15.90:0/566630801 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56482fadf620 con 0x56482f9da4d0 2024-04-25T11:37:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.652+0000 7f15a57fa640 1 -- 172.21.15.90:0/566630801 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15b4036030 con 0x56482f9da4d0 2024-04-25T11:37:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.653+0000 7f15a57fa640 1 -- 172.21.15.90:0/566630801 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f15ac003b50 con 0x56482f9da4d0 2024-04-25T11:37:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.653+0000 7f15cac66740 10 client.25549 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:37:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.653+0000 7f15cac66740 10 client.25549 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:37:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.653+0000 7f15cac66740 20 client.25549 populate_metadata read hostname 'smithi090' 2024-04-25T11:37:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.653+0000 7f15cac66740 1 --2- 172.21.15.90:0/566630801 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56482f9b9d80 0x56482fdb7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.653+0000 7f15cac66740 4 client.25549 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:37:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.653+0000 7f15cac66740 1 -- 172.21.15.90:0/566630801 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56482fd0f070 con 0x56482f9b9d80 2024-04-25T11:37:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.653+0000 7f15c234f640 1 --2- 172.21.15.90:0/566630801 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56482f9b9d80 0x56482fdb7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.653+0000 7f15a57fa640 10 client.25549 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.654+0000 7f15c234f640 1 --2- 172.21.15.90:0/566630801 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56482f9b9d80 0x56482fdb7630 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=933554fcd7ea6aad server_cookie=1a5acd3e9546ac40 in_seq=0 out_seq=0 2024-04-25T11:37:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.656+0000 7f15a57fa640 1 -- 172.21.15.90:0/566630801 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56482fd0f070 con 0x56482f9b9d80 2024-04-25T11:37:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.656+0000 7f15a57fa640 10 client.25549 handle_command_reply: tid=0 2024-04-25T11:37:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.656+0000 7f15cac66740 1 client.25549 shutdown 2024-04-25T11:37:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.656+0000 7f15cac66740 1 -- 172.21.15.90:0/566630801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56482f9da4d0 msgr2=0x56482fda5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.656+0000 7f15cac66740 1 --2- 172.21.15.90:0/566630801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56482f9da4d0 0x56482fda5720 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f15ac012700 tx=0x7f15ac012bc0 comp rx=0 tx=0).stop 2024-04-25T11:37:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.656+0000 7f15cac66740 1 -- 172.21.15.90:0/566630801 shutdown_connections 2024-04-25T11:37:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.656+0000 7f15cac66740 1 --2- 172.21.15.90:0/566630801 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56482f9b9d80 0x56482fdb7630 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.656+0000 7f15cac66740 1 --2- 172.21.15.90:0/566630801 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56482f9fb1b0 0x56482fdaaf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.656+0000 7f15a57fa640 0 client.25549 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.656+0000 7f15cac66740 1 --2- 172.21.15.90:0/566630801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56482f9da4d0 0x56482fda5720 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.656+0000 7f15cac66740 1 --2- 172.21.15.90:0/566630801 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56482f9d22f0 0x56482fda51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.656+0000 7f15cac66740 1 -- 172.21.15.90:0/566630801 >> 172.21.15.90:0/566630801 conn(0x56482f9ef300 msgr2=0x56482fd9a920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.656+0000 7f15cac66740 1 -- 172.21.15.90:0/566630801 shutdown_connections 2024-04-25T11:37:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.656+0000 7f15cac66740 1 -- 172.21.15.90:0/566630801 wait complete. 2024-04-25T11:37:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.657+0000 7f15cac66740 20 client.25549 trim_cache size 0 max 16384 2024-04-25T11:37:16.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.673+0000 7f15cac66740 1 Processor -- start 2024-04-25T11:37:16.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.673+0000 7f15cac66740 1 -- start start 2024-04-25T11:37:16.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.673+0000 7f15cac66740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56482f9d22f0 0x56482fda7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:16.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.673+0000 7f15cac66740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56482f9da4d0 0x56482fd9ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:16.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.673+0000 7f15cac66740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56482f9fb1b0 0x56482fdbce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:16.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.673+0000 7f15cac66740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f15bc1afbb0 con 0x56482f9da4d0 2024-04-25T11:37:16.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.673+0000 7f15cac66740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f15bc1b1720 con 0x56482f9d22f0 2024-04-25T11:37:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.673+0000 7f15cac66740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f15bc1b0280 con 0x56482f9fb1b0 2024-04-25T11:37:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.673+0000 7f15c234f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56482f9d22f0 0x56482fda7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.673+0000 7f15c134d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56482f9da4d0 0x56482fd9ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.674+0000 7f15c234f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56482f9d22f0 0x56482fda7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:58998/0 (socket says 172.21.15.90:58998) 2024-04-25T11:37:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.674+0000 7f15c134d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56482f9da4d0 0x56482fd9ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48854/0 (socket says 172.21.15.90:48854) 2024-04-25T11:37:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.674+0000 7f15c234f640 1 -- 172.21.15.90:0/3052868829 learned_addr learned my addr 172.21.15.90:0/3052868829 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.674+0000 7f15c234f640 1 --2- 172.21.15.90:0/3052868829 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56482f9fb1b0 0x56482fdbce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.674+0000 7f15c134d640 1 -- 172.21.15.90:0/3052868829 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56482f9fb1b0 msgr2=0x56482fdbce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.674+0000 7f15c134d640 1 --2- 172.21.15.90:0/3052868829 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56482f9fb1b0 0x56482fdbce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.674+0000 7f15c134d640 1 -- 172.21.15.90:0/3052868829 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56482f9d22f0 msgr2=0x56482fda7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.674+0000 7f15c134d640 1 --2- 172.21.15.90:0/3052868829 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56482f9d22f0 0x56482fda7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.674+0000 7f15c134d640 1 -- 172.21.15.90:0/3052868829 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56482fdb79c0 con 0x56482f9da4d0 2024-04-25T11:37:16.664 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.674+0000 7f15c134d640 1 --2- 172.21.15.90:0/3052868829 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56482f9da4d0 0x56482fd9ee50 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f15ac010520 tx=0x7f15ac009e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46247b10f189250 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:16.665 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.674+0000 7f15a77fe640 1 -- 172.21.15.90:0/3052868829 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ac003fc0 con 0x56482f9da4d0 2024-04-25T11:37:16.665 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.674+0000 7f15a77fe640 1 -- 172.21.15.90:0/3052868829 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15ac004160 con 0x56482f9da4d0 2024-04-25T11:37:16.665 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.674+0000 7f15a77fe640 1 -- 172.21.15.90:0/3052868829 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ac0035f0 con 0x56482f9da4d0 2024-04-25T11:37:16.665 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.675+0000 7f15cac66740 1 -- 172.21.15.90:0/3052868829 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56482f9da4d0 msgr2=0x56482fd9ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:16.665 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.675+0000 7f15cac66740 1 --2- 172.21.15.90:0/3052868829 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56482f9da4d0 0x56482fd9ee50 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f15ac010520 tx=0x7f15ac009e80 comp rx=0 tx=0).stop 2024-04-25T11:37:16.665 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.675+0000 7f15cac66740 1 -- 172.21.15.90:0/3052868829 shutdown_connections 2024-04-25T11:37:16.665 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.675+0000 7f15cac66740 1 --2- 172.21.15.90:0/3052868829 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56482f9fb1b0 0x56482fdbce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:16.665 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.675+0000 7f15cac66740 1 --2- 172.21.15.90:0/3052868829 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56482f9da4d0 0x56482fd9ee50 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:16.665 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.675+0000 7f15cac66740 1 --2- 172.21.15.90:0/3052868829 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56482f9d22f0 0x56482fda7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:16.665 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.675+0000 7f15cac66740 1 -- 172.21.15.90:0/3052868829 >> 172.21.15.90:0/3052868829 conn(0x56482f9ef300 msgr2=0x56482fd9e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:16.665 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.675+0000 7f15cac66740 1 -- 172.21.15.90:0/3052868829 shutdown_connections 2024-04-25T11:37:16.665 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.675+0000 7f15cac66740 1 -- 172.21.15.90:0/3052868829 wait complete. 2024-04-25T11:37:16.665 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.676+0000 7f15cac66740 1 -- start start 2024-04-25T11:37:16.665 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.676+0000 7f15cac66740 1 Processor -- start 2024-04-25T11:37:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.677+0000 7f15cac66740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56482f9d22f0 0x56482fdb1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.677+0000 7f15cac66740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56482f9da4d0 0x56482fdb1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.677+0000 7f15cac66740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56482f9fb1b0 0x56482fdae5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.677+0000 7f15cac66740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f15bc1b1020 con 0x56482f9fb1b0 2024-04-25T11:37:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.677+0000 7f15cac66740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f15bc1b0950 con 0x56482f9d22f0 2024-04-25T11:37:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.677+0000 7f15cac66740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f15bc1adb90 con 0x56482f9da4d0 2024-04-25T11:37:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.678+0000 7f15c134d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56482f9da4d0 0x56482fdb1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.678+0000 7f15cac66740 10 client.0 _ll_register_callbacks cb 0x7f15ac009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:37:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.678+0000 7f15c134d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56482f9da4d0 0x56482fdb1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:40640/0 (socket says 172.21.15.90:40640) 2024-04-25T11:37:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.678+0000 7f15c134d640 1 -- 172.21.15.90:0/3911939904 learned_addr learned my addr 172.21.15.90:0/3911939904 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.678+0000 7f15c234f640 1 --2- 172.21.15.90:0/3911939904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56482f9fb1b0 0x56482fdae5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.678+0000 7f15c234f640 1 --2- 172.21.15.90:0/3911939904 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56482f9d22f0 0x56482fdb1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.678+0000 7f15c134d640 1 -- 172.21.15.90:0/3911939904 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56482f9d22f0 msgr2=0x56482fdb1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.678+0000 7f15c134d640 1 --2- 172.21.15.90:0/3911939904 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56482f9d22f0 0x56482fdb1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.678+0000 7f15c134d640 1 -- 172.21.15.90:0/3911939904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56482f9fb1b0 msgr2=0x56482fdae5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.678+0000 7f15c134d640 1 --2- 172.21.15.90:0/3911939904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56482f9fb1b0 0x56482fdae5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.678+0000 7f15c134d640 1 -- 172.21.15.90:0/3911939904 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56482fdaed50 con 0x56482f9da4d0 2024-04-25T11:37:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.678+0000 7f15c134d640 1 --2- 172.21.15.90:0/3911939904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56482f9da4d0 0x56482fdb1680 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f15ac02d000 tx=0x7f15ac00fc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6f3e56eb0eae7f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.678+0000 7f15a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:37:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.678+0000 7f15a57fa640 1 -- 172.21.15.90:0/3911939904 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ac003fc0 con 0x56482f9da4d0 2024-04-25T11:37:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.679+0000 7f15a57fa640 1 -- 172.21.15.90:0/3911939904 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f15ac005050 con 0x56482f9da4d0 2024-04-25T11:37:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.679+0000 7f15cac66740 1 -- 172.21.15.90:0/3911939904 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15ac004160 con 0x56482f9da4d0 2024-04-25T11:37:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.679+0000 7f15a57fa640 1 -- 172.21.15.90:0/3911939904 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f15ac005250 con 0x56482f9da4d0 2024-04-25T11:37:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.679+0000 7f15cac66740 10 client.25555 fetch_fsmap learned FSMap version 27 2024-04-25T11:37:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.679+0000 7f15cac66740 1 -- 172.21.15.90:0/3911939904 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56482fe1f570 con 0x56482f9da4d0 2024-04-25T11:37:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.679+0000 7f15a57fa640 1 -- 172.21.15.90:0/3911939904 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15ac004160 con 0x56482f9da4d0 2024-04-25T11:37:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.679+0000 7f15a57fa640 1 -- 172.21.15.90:0/3911939904 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f15ac02b070 con 0x56482f9da4d0 2024-04-25T11:37:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.679+0000 7f15cac66740 10 client.25555 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:37:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.679+0000 7f15cac66740 10 client.25555 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:37:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.679+0000 7f15cac66740 20 client.25555 populate_metadata read hostname 'smithi090' 2024-04-25T11:37:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.679+0000 7f15cac66740 1 --2- 172.21.15.90:0/3911939904 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56482f98f1c0 0x56482fe21be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.680+0000 7f15cac66740 4 client.25555 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:37:16.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.680+0000 7f15cac66740 1 -- 172.21.15.90:0/3911939904 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56482fd08ee0 con 0x56482f98f1c0 2024-04-25T11:37:16.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.680+0000 7f15c234f640 1 --2- 172.21.15.90:0/3911939904 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56482f98f1c0 0x56482fe21be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:16.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.681+0000 7f15a57fa640 10 client.25555 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:16.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:16.681+0000 7f15c234f640 1 --2- 172.21.15.90:0/3911939904 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56482f98f1c0 0x56482fe21be0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f783c4fbf9ed178 server_cookie=c4d139b7f5073190 in_seq=0 out_seq=0 2024-04-25T11:37:17.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:17 smithi090 ceph-mon[29575]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T11:37:17.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:17 smithi090 ceph-mon[29575]: pgmap v69: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 476 MiB/s wr, 3.37k op/s 2024-04-25T11:37:17.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:17 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:37:17.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:17 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:37:17.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:17 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:37:17.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:17 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:37:17.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:17 smithi138 ceph-mon[32328]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T11:37:17.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:17 smithi138 ceph-mon[32328]: pgmap v69: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 476 MiB/s wr, 3.37k op/s 2024-04-25T11:37:17.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:17 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:37:17.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:17 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:37:17.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:17 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:37:17.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:17 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:37:17.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:17 smithi156 ceph-mon[33045]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T11:37:17.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:17 smithi156 ceph-mon[33045]: pgmap v69: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 476 MiB/s wr, 3.37k op/s 2024-04-25T11:37:17.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:17 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:37:17.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:17 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:37:17.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:17 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:37:17.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:17 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:37:18.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:18 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:37:18.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:18 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:37:18.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:18 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:37:18.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:18 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:37:18.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:18 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:37:18.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:18 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:37:18.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:18 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:37:18.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:18 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:37:18.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:18 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:37:19.479 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:19 smithi090 ceph-mon[29575]: pgmap v70: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 398 MiB/s wr, 3.26k op/s 2024-04-25T11:37:19.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:19 smithi138 ceph-mon[32328]: pgmap v70: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 398 MiB/s wr, 3.26k op/s 2024-04-25T11:37:19.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:19 smithi156 ceph-mon[33045]: pgmap v70: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 398 MiB/s wr, 3.26k op/s 2024-04-25T11:37:21.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:21 smithi156 ceph-mon[33045]: pgmap v71: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 528 MiB/s wr, 4.43k op/s 2024-04-25T11:37:21.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:21 smithi090 ceph-mon[29575]: pgmap v71: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 528 MiB/s wr, 4.43k op/s 2024-04-25T11:37:21.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:21 smithi138 ceph-mon[32328]: pgmap v71: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 528 MiB/s wr, 4.43k op/s 2024-04-25T11:37:21.859 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.870+0000 7f15a57fa640 1 -- 172.21.15.90:0/3911939904 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56482fd08ee0 con 0x56482f98f1c0 2024-04-25T11:37:21.859 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.870+0000 7f15a57fa640 10 client.25555 handle_command_reply: tid=0 2024-04-25T11:37:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.870+0000 7f15cac66740 1 client.25555 shutdown 2024-04-25T11:37:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.871+0000 7f15cac66740 1 -- 172.21.15.90:0/3911939904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56482f9da4d0 msgr2=0x56482fdb1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.871+0000 7f15cac66740 1 --2- 172.21.15.90:0/3911939904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56482f9da4d0 0x56482fdb1680 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f15ac02d000 tx=0x7f15ac00fc50 comp rx=0 tx=0).stop 2024-04-25T11:37:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.871+0000 7f15cac66740 1 -- 172.21.15.90:0/3911939904 shutdown_connections 2024-04-25T11:37:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.871+0000 7f15cac66740 1 --2- 172.21.15.90:0/3911939904 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56482f98f1c0 0x56482fe21be0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.871+0000 7f15cac66740 1 --2- 172.21.15.90:0/3911939904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56482f9fb1b0 0x56482fdae5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.871+0000 7f15cac66740 1 --2- 172.21.15.90:0/3911939904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56482f9da4d0 0x56482fdb1680 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.871+0000 7f15cac66740 1 --2- 172.21.15.90:0/3911939904 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56482f9d22f0 0x56482fdb1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.871+0000 7f15cac66740 1 -- 172.21.15.90:0/3911939904 >> 172.21.15.90:0/3911939904 conn(0x56482f9ef300 msgr2=0x56482fd9e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.871+0000 7f15a57fa640 0 client.25555 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.871+0000 7f15cac66740 1 -- 172.21.15.90:0/3911939904 shutdown_connections 2024-04-25T11:37:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.871+0000 7f15cac66740 1 -- 172.21.15.90:0/3911939904 wait complete. 2024-04-25T11:37:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.871+0000 7f15cac66740 20 client.25555 trim_cache size 0 max 16384 2024-04-25T11:37:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:37:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:37:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:37:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:37:21.862 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.872+0000 7f15c3db2640 1 -- 172.21.15.90:0/778700224 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f15a80afc50 msgr2=0x7f15a80b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:21.862 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.872+0000 7f15c3db2640 1 --2- 172.21.15.90:0/778700224 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f15a80afc50 0x7f15a80b2160 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f15b80dcbc0 tx=0x7f15b80e1040 comp rx=0 tx=0).stop 2024-04-25T11:37:21.862 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.872+0000 7f15c3db2640 1 -- 172.21.15.90:0/778700224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15bc1042f0 msgr2=0x7f15bc19a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:21.862 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.872+0000 7f15c3db2640 1 --2- 172.21.15.90:0/778700224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15bc1042f0 0x7f15bc19a700 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f15b800e9f0 tx=0x7f15b800eeb0 comp rx=0 tx=0).stop 2024-04-25T11:37:21.862 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.872+0000 7f15c3db2640 1 -- 172.21.15.90:0/778700224 shutdown_connections 2024-04-25T11:37:21.862 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.872+0000 7f15c3db2640 1 --2- 172.21.15.90:0/778700224 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f15a80afc50 0x7f15a80b2160 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:21.862 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.872+0000 7f15c3db2640 1 --2- 172.21.15.90:0/778700224 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f15bc10a040 0x7f15bc19ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:21.863 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.872+0000 7f15c3db2640 1 --2- 172.21.15.90:0/778700224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15bc109ba0 0x7f15bc19ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:21.863 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.872+0000 7f15c3db2640 1 --2- 172.21.15.90:0/778700224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15bc1042f0 0x7f15bc19a700 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:21.863 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.872+0000 7f15c3db2640 1 -- 172.21.15.90:0/778700224 >> 172.21.15.90:0/778700224 conn(0x7f15bc0792a0 msgr2=0x7f15bc101d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:21.863 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.872+0000 7f15c3db2640 1 -- 172.21.15.90:0/778700224 shutdown_connections 2024-04-25T11:37:21.863 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:21.872+0000 7f15c3db2640 1 -- 172.21.15.90:0/778700224 wait complete. 2024-04-25T11:37:23.156 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:37:23.156 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:37:23.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.318+0000 7f8535c96640 1 -- 172.21.15.90:0/357803932 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8530106990 msgr2=0x7f8530106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:23.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.318+0000 7f8535c96640 1 --2- 172.21.15.90:0/357803932 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8530106990 0x7f8530106dc0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f8524009900 tx=0x7f852402a410 comp rx=0 tx=0).stop 2024-04-25T11:37:23.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.318+0000 7f8535c96640 1 -- 172.21.15.90:0/357803932 shutdown_connections 2024-04-25T11:37:23.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.318+0000 7f8535c96640 1 --2- 172.21.15.90:0/357803932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8530107450 0x7f853010ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.318+0000 7f8535c96640 1 --2- 172.21.15.90:0/357803932 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8530106990 0x7f8530106dc0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.318+0000 7f8535c96640 1 --2- 172.21.15.90:0/357803932 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8530102400 0x7f8530102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.318+0000 7f8535c96640 1 -- 172.21.15.90:0/357803932 >> 172.21.15.90:0/357803932 conn(0x7f8530077880 msgr2=0x7f8530077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:23.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.318+0000 7f8535c96640 1 -- 172.21.15.90:0/357803932 shutdown_connections 2024-04-25T11:37:23.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.318+0000 7f8535c96640 1 -- 172.21.15.90:0/357803932 wait complete. 2024-04-25T11:37:23.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.319+0000 7f8535c96640 1 Processor -- start 2024-04-25T11:37:23.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.319+0000 7f8535c96640 1 -- start start 2024-04-25T11:37:23.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.319+0000 7f8535c96640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8530102400 0x7f853019a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:23.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.319+0000 7f8535c96640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8530106990 0x7f853019ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:23.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.319+0000 7f8535c96640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8530107450 0x7f853019efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:23.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.319+0000 7f8535c96640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8530113d00 con 0x7f8530106990 2024-04-25T11:37:23.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.319+0000 7f8535c96640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8530113b80 con 0x7f8530102400 2024-04-25T11:37:23.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.319+0000 7f8535c96640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8530113e80 con 0x7f8530107450 2024-04-25T11:37:23.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.320+0000 7f852ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8530107450 0x7f853019efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:23.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.320+0000 7f852effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8530106990 0x7f853019ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:23.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.320+0000 7f852ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8530107450 0x7f853019efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:52082/0 (socket says 172.21.15.90:52082) 2024-04-25T11:37:23.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.320+0000 7f852ffff640 1 -- 172.21.15.90:0/3791289048 learned_addr learned my addr 172.21.15.90:0/3791289048 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:23.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.320+0000 7f852f7fe640 1 --2- 172.21.15.90:0/3791289048 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8530102400 0x7f853019a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:23.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.320+0000 7f852ffff640 1 -- 172.21.15.90:0/3791289048 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8530102400 msgr2=0x7f853019a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:23.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.320+0000 7f852ffff640 1 --2- 172.21.15.90:0/3791289048 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8530102400 0x7f853019a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.320+0000 7f852ffff640 1 -- 172.21.15.90:0/3791289048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8530106990 msgr2=0x7f853019ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:23.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.320+0000 7f852ffff640 1 --2- 172.21.15.90:0/3791289048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8530106990 0x7f853019ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.320+0000 7f852ffff640 1 -- 172.21.15.90:0/3791289048 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f853019f5e0 con 0x7f8530107450 2024-04-25T11:37:23.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.320+0000 7f852f7fe640 1 --2- 172.21.15.90:0/3791289048 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8530102400 0x7f853019a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:37:23.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.321+0000 7f852ffff640 1 --2- 172.21.15.90:0/3791289048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8530107450 0x7f853019efa0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f852800eb10 tx=0x7f852800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:23.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.321+0000 7f852cff9640 1 -- 172.21.15.90:0/3791289048 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8528004b20 con 0x7f8530107450 2024-04-25T11:37:23.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.321+0000 7f852cff9640 1 -- 172.21.15.90:0/3791289048 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8528004cc0 con 0x7f8530107450 2024-04-25T11:37:23.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.321+0000 7f852cff9640 1 -- 172.21.15.90:0/3791289048 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8528010670 con 0x7f8530107450 2024-04-25T11:37:23.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.321+0000 7f8535c96640 1 -- 172.21.15.90:0/3791289048 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f853019f8d0 con 0x7f8530107450 2024-04-25T11:37:23.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.321+0000 7f8535c96640 1 -- 172.21.15.90:0/3791289048 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f853019fda0 con 0x7f8530107450 2024-04-25T11:37:23.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.324+0000 7f852cff9640 1 -- 172.21.15.90:0/3791289048 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f8528005020 con 0x7f8530107450 2024-04-25T11:37:23.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.324+0000 7f852cff9640 1 --2- 172.21.15.90:0/3791289048 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f85180afce0 0x7f85180b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:23.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.325+0000 7f852cff9640 1 -- 172.21.15.90:0/3791289048 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f85280df5e0 con 0x7f8530107450 2024-04-25T11:37:23.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.325+0000 7f852f7fe640 1 --2- 172.21.15.90:0/3791289048 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f85180afce0 0x7f85180b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:23.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.325+0000 7f853cc29740 1 Processor -- start 2024-04-25T11:37:23.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.325+0000 7f852f7fe640 1 --2- 172.21.15.90:0/3791289048 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f85180afce0 0x7f85180b21f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8520009c20 tx=0x7f8520005f90 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:23.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.325+0000 7f853cc29740 1 -- start start 2024-04-25T11:37:23.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.326+0000 7f853cc29740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ced6824390 0x55ced6bc4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:23.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.326+0000 7f853cc29740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ced67fb4d0 0x55ced6bc4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:23.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.326+0000 7f853cc29740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced68036b0 0x55ced6bd18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:23.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.326+0000 7f853cc29740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ced6b2ad80 con 0x55ced67fb4d0 2024-04-25T11:37:23.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.326+0000 7f853cc29740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ced6ae48b0 con 0x55ced6824390 2024-04-25T11:37:23.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.326+0000 7f853cc29740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ced6b26d40 con 0x55ced68036b0 2024-04-25T11:37:23.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.326+0000 7f852f7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ced6824390 0x55ced6bc4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:23.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.326+0000 7f852f7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ced6824390 0x55ced6bc4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:45986/0 (socket says 172.21.15.90:45986) 2024-04-25T11:37:23.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.326+0000 7f852f7fe640 1 -- 172.21.15.90:0/2136579731 learned_addr learned my addr 172.21.15.90:0/2136579731 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:23.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.326+0000 7f852effd640 1 --2- 172.21.15.90:0/2136579731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ced67fb4d0 0x55ced6bc4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:23.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.326+0000 7f852ffff640 1 --2- 172.21.15.90:0/2136579731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced68036b0 0x55ced6bd18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:23.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.326+0000 7f852f7fe640 1 -- 172.21.15.90:0/2136579731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced68036b0 msgr2=0x55ced6bd18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:23.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.326+0000 7f852f7fe640 1 --2- 172.21.15.90:0/2136579731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced68036b0 0x55ced6bd18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.326+0000 7f852f7fe640 1 -- 172.21.15.90:0/2136579731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ced67fb4d0 msgr2=0x55ced6bc4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:23.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.326+0000 7f852f7fe640 1 --2- 172.21.15.90:0/2136579731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ced67fb4d0 0x55ced6bc4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.326+0000 7f852f7fe640 1 -- 172.21.15.90:0/2136579731 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ced6ab5a40 con 0x55ced6824390 2024-04-25T11:37:23.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.326+0000 7f852ffff640 1 --2- 172.21.15.90:0/2136579731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced68036b0 0x55ced6bd18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:37:23.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.326+0000 7f852effd640 1 --2- 172.21.15.90:0/2136579731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ced67fb4d0 0x55ced6bc4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:37:23.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.327+0000 7f852f7fe640 1 --2- 172.21.15.90:0/2136579731 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ced6824390 0x55ced6bc4a60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8520004160 tx=0x7f8520014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b73c9699f8971fa9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:23.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.327+0000 7f8515ffb640 1 -- 172.21.15.90:0/2136579731 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8520003e30 con 0x55ced6824390 2024-04-25T11:37:23.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.327+0000 7f8515ffb640 1 -- 172.21.15.90:0/2136579731 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8520025070 con 0x55ced6824390 2024-04-25T11:37:23.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.327+0000 7f8515ffb640 1 -- 172.21.15.90:0/2136579731 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8520020760 con 0x55ced6824390 2024-04-25T11:37:23.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.328+0000 7f853cc29740 1 -- 172.21.15.90:0/2136579731 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ced6824390 msgr2=0x55ced6bc4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:23.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.328+0000 7f853cc29740 1 --2- 172.21.15.90:0/2136579731 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ced6824390 0x55ced6bc4a60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8520004160 tx=0x7f8520014000 comp rx=0 tx=0).stop 2024-04-25T11:37:23.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.328+0000 7f853cc29740 1 -- 172.21.15.90:0/2136579731 shutdown_connections 2024-04-25T11:37:23.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.328+0000 7f853cc29740 1 --2- 172.21.15.90:0/2136579731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced68036b0 0x55ced6bd18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.328+0000 7f853cc29740 1 --2- 172.21.15.90:0/2136579731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ced67fb4d0 0x55ced6bc4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.328+0000 7f853cc29740 1 --2- 172.21.15.90:0/2136579731 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ced6824390 0x55ced6bc4a60 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.328+0000 7f853cc29740 1 -- 172.21.15.90:0/2136579731 >> 172.21.15.90:0/2136579731 conn(0x55ced68184e0 msgr2=0x55ced6bc1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:23.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.328+0000 7f853cc29740 1 -- 172.21.15.90:0/2136579731 shutdown_connections 2024-04-25T11:37:23.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.328+0000 7f853cc29740 1 -- 172.21.15.90:0/2136579731 wait complete. 2024-04-25T11:37:23.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.329+0000 7f853cc29740 1 -- start start 2024-04-25T11:37:23.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.329+0000 7f853cc29740 1 Processor -- start 2024-04-25T11:37:23.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.329+0000 7f853cc29740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ced67fb4d0 0x55ced6bcf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:23.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.329+0000 7f853cc29740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ced68036b0 0x55ced6bcfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:23.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.329+0000 7f853cc29740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced6824390 0x55ced6bdb460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:23.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.329+0000 7f853cc29740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ced6bd1fe0 con 0x55ced67fb4d0 2024-04-25T11:37:23.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.329+0000 7f853cc29740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ced6ab1d10 con 0x55ced68036b0 2024-04-25T11:37:23.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.329+0000 7f853cc29740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ced6bd2160 con 0x55ced6824390 2024-04-25T11:37:23.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.330+0000 7f853cc29740 10 client.0 _ll_register_callbacks cb 0x55ced6ade210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:37:23.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.330+0000 7f852f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ced67fb4d0 0x55ced6bcf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:23.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.330+0000 7f852f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ced67fb4d0 0x55ced6bcf5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55470/0 (socket says 172.21.15.90:55470) 2024-04-25T11:37:23.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.330+0000 7f852f7fe640 1 -- 172.21.15.90:0/3152577462 learned_addr learned my addr 172.21.15.90:0/3152577462 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:23.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.330+0000 7f852effd640 1 --2- 172.21.15.90:0/3152577462 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ced68036b0 0x55ced6bcfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:23.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.330+0000 7f852ffff640 1 --2- 172.21.15.90:0/3152577462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced6824390 0x55ced6bdb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:23.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.330+0000 7f852f7fe640 1 -- 172.21.15.90:0/3152577462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced6824390 msgr2=0x55ced6bdb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:23.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.330+0000 7f852f7fe640 1 --2- 172.21.15.90:0/3152577462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced6824390 0x55ced6bdb460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.330+0000 7f852f7fe640 1 -- 172.21.15.90:0/3152577462 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ced68036b0 msgr2=0x55ced6bcfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:23.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.330+0000 7f852f7fe640 1 --2- 172.21.15.90:0/3152577462 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ced68036b0 0x55ced6bcfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.330+0000 7f852f7fe640 1 -- 172.21.15.90:0/3152577462 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ced69067d0 con 0x55ced67fb4d0 2024-04-25T11:37:23.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.331+0000 7f852f7fe640 1 --2- 172.21.15.90:0/3152577462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ced67fb4d0 0x55ced6bcf5b0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f8520009750 tx=0x7f8520003e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=376b4869eb11e8c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:23.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.331+0000 7f850effd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:37:23.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.331+0000 7f850effd640 1 -- 172.21.15.90:0/3152577462 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8520012860 con 0x55ced67fb4d0 2024-04-25T11:37:23.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.331+0000 7f850effd640 1 -- 172.21.15.90:0/3152577462 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f8520012a00 con 0x55ced67fb4d0 2024-04-25T11:37:23.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.331+0000 7f853cc29740 1 -- 172.21.15.90:0/3152577462 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8520025070 con 0x55ced67fb4d0 2024-04-25T11:37:23.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.331+0000 7f850effd640 1 -- 172.21.15.90:0/3152577462 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f85200204a0 con 0x55ced67fb4d0 2024-04-25T11:37:23.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.331+0000 7f853cc29740 10 client.16224 fetch_fsmap learned FSMap version 27 2024-04-25T11:37:23.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.331+0000 7f853cc29740 1 -- 172.21.15.90:0/3152577462 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ced683c3f0 con 0x55ced67fb4d0 2024-04-25T11:37:23.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.332+0000 7f850effd640 1 -- 172.21.15.90:0/3152577462 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8520016070 con 0x55ced67fb4d0 2024-04-25T11:37:23.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.332+0000 7f850effd640 1 -- 172.21.15.90:0/3152577462 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f8520033020 con 0x55ced67fb4d0 2024-04-25T11:37:23.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.332+0000 7f853cc29740 10 client.16224 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:37:23.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.332+0000 7f853cc29740 10 client.16224 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:37:23.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.332+0000 7f853cc29740 20 client.16224 populate_metadata read hostname 'smithi090' 2024-04-25T11:37:23.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.332+0000 7f853cc29740 1 --2- 172.21.15.90:0/3152577462 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ced67e2f60 0x55ced6be1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:23.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.332+0000 7f853cc29740 4 client.16224 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:37:23.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.332+0000 7f853cc29740 1 -- 172.21.15.90:0/3152577462 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8520025070 con 0x55ced67e2f60 2024-04-25T11:37:23.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.333+0000 7f852effd640 1 --2- 172.21.15.90:0/3152577462 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ced67e2f60 0x55ced6be1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:23.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.333+0000 7f850effd640 10 client.16224 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:23.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.333+0000 7f852effd640 1 --2- 172.21.15.90:0/3152577462 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ced67e2f60 0x55ced6be1510 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f3d59eb4cc1df0c server_cookie=a2f7456623fff5d6 in_seq=0 out_seq=0 2024-04-25T11:37:23.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.335+0000 7f850effd640 1 -- 172.21.15.90:0/3152577462 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f8520025070 con 0x55ced67e2f60 2024-04-25T11:37:23.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.335+0000 7f850effd640 10 client.16224 handle_command_reply: tid=0 2024-04-25T11:37:23.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.335+0000 7f853cc29740 1 client.16224 shutdown 2024-04-25T11:37:23.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.336+0000 7f853cc29740 1 -- 172.21.15.90:0/3152577462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ced67fb4d0 msgr2=0x55ced6bcf5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:23.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.336+0000 7f853cc29740 1 --2- 172.21.15.90:0/3152577462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ced67fb4d0 0x55ced6bcf5b0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f8520009750 tx=0x7f8520003e60 comp rx=0 tx=0).stop 2024-04-25T11:37:23.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.336+0000 7f853cc29740 1 -- 172.21.15.90:0/3152577462 shutdown_connections 2024-04-25T11:37:23.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.336+0000 7f853cc29740 1 --2- 172.21.15.90:0/3152577462 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ced67e2f60 0x55ced6be1510 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.336+0000 7f853cc29740 1 --2- 172.21.15.90:0/3152577462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced6824390 0x55ced6bdb460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.336+0000 7f853cc29740 1 --2- 172.21.15.90:0/3152577462 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ced68036b0 0x55ced6bcfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.336+0000 7f853cc29740 1 --2- 172.21.15.90:0/3152577462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ced67fb4d0 0x55ced6bcf5b0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.336+0000 7f853cc29740 1 -- 172.21.15.90:0/3152577462 >> 172.21.15.90:0/3152577462 conn(0x55ced68184e0 msgr2=0x55ced6bc6780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:23.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.336+0000 7f850effd640 0 client.16224 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:23.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.336+0000 7f853cc29740 1 -- 172.21.15.90:0/3152577462 shutdown_connections 2024-04-25T11:37:23.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.336+0000 7f853cc29740 1 -- 172.21.15.90:0/3152577462 wait complete. 2024-04-25T11:37:23.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.336+0000 7f853cc29740 20 client.16224 trim_cache size 0 max 16384 2024-04-25T11:37:23.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.377+0000 7f853cc29740 1 Processor -- start 2024-04-25T11:37:23.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.377+0000 7f853cc29740 1 -- start start 2024-04-25T11:37:23.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.377+0000 7f853cc29740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ced67fb4d0 0x55ced6bce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:23.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.378+0000 7f853cc29740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced68036b0 0x55ced6bd11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:23.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.378+0000 7f853cc29740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ced6824390 0x55ced6bd7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:23.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.378+0000 7f853cc29740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f85301ada20 con 0x55ced6824390 2024-04-25T11:37:23.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.379+0000 7f853cc29740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f85301af590 con 0x55ced67fb4d0 2024-04-25T11:37:23.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.379+0000 7f852f7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ced67fb4d0 0x55ced6bce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:23.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.379+0000 7f852f7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ced67fb4d0 0x55ced6bce6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:46010/0 (socket says 172.21.15.90:46010) 2024-04-25T11:37:23.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.379+0000 7f852f7fe640 1 -- 172.21.15.90:0/2927571903 learned_addr learned my addr 172.21.15.90:0/2927571903 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:23.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.379+0000 7f853cc29740 1 -- 172.21.15.90:0/2927571903 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f85301ae0f0 con 0x55ced68036b0 2024-04-25T11:37:23.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.379+0000 7f852ffff640 1 --2- 172.21.15.90:0/2927571903 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ced6824390 0x55ced6bd7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:23.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.380+0000 7f852ffff640 1 -- 172.21.15.90:0/2927571903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced68036b0 msgr2=0x55ced6bd11d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:37:23.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.380+0000 7f852ffff640 1 --2- 172.21.15.90:0/2927571903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced68036b0 0x55ced6bd11d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.380+0000 7f852ffff640 1 -- 172.21.15.90:0/2927571903 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ced67fb4d0 msgr2=0x55ced6bce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:23.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.380+0000 7f852ffff640 1 --2- 172.21.15.90:0/2927571903 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ced67fb4d0 0x55ced6bce6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.381+0000 7f852ffff640 1 -- 172.21.15.90:0/2927571903 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ced67fc0f0 con 0x55ced6824390 2024-04-25T11:37:23.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.382+0000 7f852ffff640 1 --2- 172.21.15.90:0/2927571903 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ced6824390 0x55ced6bd7430 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f85280dccd0 tx=0x7f8528090040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14c00e5c46ddd32d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:23.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.382+0000 7f85157fa640 1 -- 172.21.15.90:0/2927571903 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85280a7720 con 0x55ced6824390 2024-04-25T11:37:23.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.383+0000 7f85157fa640 1 -- 172.21.15.90:0/2927571903 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85280aee50 con 0x55ced6824390 2024-04-25T11:37:23.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.383+0000 7f85157fa640 1 -- 172.21.15.90:0/2927571903 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85280a39b0 con 0x55ced6824390 2024-04-25T11:37:23.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.384+0000 7f853cc29740 1 -- 172.21.15.90:0/2927571903 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ced6824390 msgr2=0x55ced6bd7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:23.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.384+0000 7f853cc29740 1 --2- 172.21.15.90:0/2927571903 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ced6824390 0x55ced6bd7430 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f85280dccd0 tx=0x7f8528090040 comp rx=0 tx=0).stop 2024-04-25T11:37:23.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.388+0000 7f853cc29740 1 -- 172.21.15.90:0/2927571903 shutdown_connections 2024-04-25T11:37:23.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.388+0000 7f853cc29740 1 --2- 172.21.15.90:0/2927571903 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ced6824390 0x55ced6bd7430 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.388+0000 7f853cc29740 1 --2- 172.21.15.90:0/2927571903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced68036b0 0x55ced6bd11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.388+0000 7f853cc29740 1 --2- 172.21.15.90:0/2927571903 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ced67fb4d0 0x55ced6bce6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.388+0000 7f853cc29740 1 -- 172.21.15.90:0/2927571903 >> 172.21.15.90:0/2927571903 conn(0x55ced68184e0 msgr2=0x55ced6bd0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:23.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.392+0000 7f853cc29740 1 -- 172.21.15.90:0/2927571903 shutdown_connections 2024-04-25T11:37:23.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.392+0000 7f853cc29740 1 -- 172.21.15.90:0/2927571903 wait complete. 2024-04-25T11:37:23.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.393+0000 7f853cc29740 1 -- start start 2024-04-25T11:37:23.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.394+0000 7f853cc29740 1 Processor -- start 2024-04-25T11:37:23.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.395+0000 7f853cc29740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced67fb4d0 0x55ced6bd2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:23.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.395+0000 7f853cc29740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ced68036b0 0x55ced6bd2940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:23.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.395+0000 7f853cc29740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ced6824390 0x55ced6bd2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:23.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.395+0000 7f853cc29740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f85301aee90 con 0x55ced68036b0 2024-04-25T11:37:23.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.395+0000 7f853cc29740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f85301ae7c0 con 0x55ced6824390 2024-04-25T11:37:23.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.395+0000 7f853cc29740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f85301aba00 con 0x55ced67fb4d0 2024-04-25T11:37:23.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.395+0000 7f853cc29740 10 client.0 _ll_register_callbacks cb 0x7f85280093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:37:23.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.395+0000 7f852effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ced68036b0 0x55ced6bd2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:23.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.395+0000 7f852effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ced68036b0 0x55ced6bd2940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55494/0 (socket says 172.21.15.90:55494) 2024-04-25T11:37:23.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.395+0000 7f852effd640 1 -- 172.21.15.90:0/1559511879 learned_addr learned my addr 172.21.15.90:0/1559511879 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:23.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.398+0000 7f852ffff640 1 --2- 172.21.15.90:0/1559511879 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ced6824390 0x55ced6bd2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:23.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.399+0000 7f852ffff640 1 -- 172.21.15.90:0/1559511879 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced67fb4d0 msgr2=0x55ced6bd2400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:37:23.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.399+0000 7f852ffff640 1 --2- 172.21.15.90:0/1559511879 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced67fb4d0 0x55ced6bd2400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.399+0000 7f852ffff640 1 -- 172.21.15.90:0/1559511879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ced68036b0 msgr2=0x55ced6bd2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:23.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.399+0000 7f852ffff640 1 --2- 172.21.15.90:0/1559511879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ced68036b0 0x55ced6bd2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.399+0000 7f852ffff640 1 -- 172.21.15.90:0/1559511879 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ced6aeb4b0 con 0x55ced6824390 2024-04-25T11:37:23.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.400+0000 7f852ffff640 1 --2- 172.21.15.90:0/1559511879 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ced6824390 0x55ced6bd2e80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x55ced6b26d70 tx=0x7f852800efd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48ff8a41dd9f0e8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:23.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.402+0000 7f850effd640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:37:23.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.402+0000 7f850effd640 1 -- 172.21.15.90:0/1559511879 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85280b0580 con 0x55ced6824390 2024-04-25T11:37:23.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.402+0000 7f850effd640 1 -- 172.21.15.90:0/1559511879 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f85280a3d10 con 0x55ced6824390 2024-04-25T11:37:23.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.402+0000 7f853cc29740 1 -- 172.21.15.90:0/1559511879 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85280aee50 con 0x55ced6824390 2024-04-25T11:37:23.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.403+0000 7f850effd640 1 -- 172.21.15.90:0/1559511879 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f85280a7720 con 0x55ced6824390 2024-04-25T11:37:23.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.403+0000 7f853cc29740 10 client.25610 fetch_fsmap learned FSMap version 27 2024-04-25T11:37:23.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.403+0000 7f853cc29740 1 -- 172.21.15.90:0/1559511879 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ced6c51130 con 0x55ced6824390 2024-04-25T11:37:23.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.405+0000 7f850effd640 1 -- 172.21.15.90:0/1559511879 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85280aee50 con 0x55ced6824390 2024-04-25T11:37:23.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.405+0000 7f850effd640 1 -- 172.21.15.90:0/1559511879 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f852808eb70 con 0x55ced6824390 2024-04-25T11:37:23.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.405+0000 7f853cc29740 10 client.25610 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:37:23.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.405+0000 7f853cc29740 10 client.25610 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:37:23.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.405+0000 7f853cc29740 20 client.25610 populate_metadata read hostname 'smithi090' 2024-04-25T11:37:23.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.405+0000 7f853cc29740 1 --2- 172.21.15.90:0/1559511879 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ced67b83a0 0x55ced6c53790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:23.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.405+0000 7f853cc29740 4 client.25610 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:37:23.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.405+0000 7f853cc29740 1 -- 172.21.15.90:0/1559511879 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ced6b38990 con 0x55ced67b83a0 2024-04-25T11:37:23.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.409+0000 7f852f7fe640 1 --2- 172.21.15.90:0/1559511879 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ced67b83a0 0x55ced6c53790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:23.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.430+0000 7f852f7fe640 1 --2- 172.21.15.90:0/1559511879 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ced67b83a0 0x55ced6c53790 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2dbf34d3114df037 server_cookie=e0265c796c6c5e08 in_seq=0 out_seq=0 2024-04-25T11:37:23.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.430+0000 7f850effd640 10 client.25610 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:23.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.434+0000 7f850effd640 1 -- 172.21.15.90:0/1559511879 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ced6b38990 con 0x55ced67b83a0 2024-04-25T11:37:23.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.434+0000 7f850effd640 10 client.25610 handle_command_reply: tid=0 2024-04-25T11:37:23.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.434+0000 7f853cc29740 1 client.25610 shutdown 2024-04-25T11:37:23.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.435+0000 7f853cc29740 1 -- 172.21.15.90:0/1559511879 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ced6824390 msgr2=0x55ced6bd2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:23.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.435+0000 7f853cc29740 1 --2- 172.21.15.90:0/1559511879 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ced6824390 0x55ced6bd2e80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x55ced6b26d70 tx=0x7f852800efd0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.436+0000 7f853cc29740 1 -- 172.21.15.90:0/1559511879 shutdown_connections 2024-04-25T11:37:23.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.436+0000 7f853cc29740 1 --2- 172.21.15.90:0/1559511879 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ced67b83a0 0x55ced6c53790 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.436+0000 7f853cc29740 1 --2- 172.21.15.90:0/1559511879 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ced6824390 0x55ced6bd2e80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.436+0000 7f853cc29740 1 --2- 172.21.15.90:0/1559511879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ced68036b0 0x55ced6bd2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.436+0000 7f853cc29740 1 --2- 172.21.15.90:0/1559511879 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced67fb4d0 0x55ced6bd2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.436+0000 7f853cc29740 1 -- 172.21.15.90:0/1559511879 >> 172.21.15.90:0/1559511879 conn(0x55ced68184e0 msgr2=0x55ced6bd1100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:23.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.436+0000 7f850effd640 0 client.25610 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:23.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.436+0000 7f853cc29740 1 -- 172.21.15.90:0/1559511879 shutdown_connections 2024-04-25T11:37:23.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.436+0000 7f853cc29740 1 -- 172.21.15.90:0/1559511879 wait complete. 2024-04-25T11:37:23.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.436+0000 7f853cc29740 20 client.25610 trim_cache size 0 max 16384 2024-04-25T11:37:23.425 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:37:23.425 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-25T11:37:23.425 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-25T11:37:23.425 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:37:23.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.437+0000 7f8535c96640 1 -- 172.21.15.90:0/3791289048 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f85180afce0 msgr2=0x7f85180b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:23.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.437+0000 7f8535c96640 1 --2- 172.21.15.90:0/3791289048 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f85180afce0 0x7f85180b21f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8520009c20 tx=0x7f8520005f90 comp rx=0 tx=0).stop 2024-04-25T11:37:23.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.437+0000 7f8535c96640 1 -- 172.21.15.90:0/3791289048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8530107450 msgr2=0x7f853019efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:23.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.437+0000 7f8535c96640 1 --2- 172.21.15.90:0/3791289048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8530107450 0x7f853019efa0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f852800eb10 tx=0x7f852800c2d0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.437+0000 7f8535c96640 1 -- 172.21.15.90:0/3791289048 shutdown_connections 2024-04-25T11:37:23.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.437+0000 7f8535c96640 1 --2- 172.21.15.90:0/3791289048 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f85180afce0 0x7f85180b21f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.437+0000 7f8535c96640 1 --2- 172.21.15.90:0/3791289048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8530107450 0x7f853019efa0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.437+0000 7f8535c96640 1 --2- 172.21.15.90:0/3791289048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8530106990 0x7f853019ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.437+0000 7f8535c96640 1 --2- 172.21.15.90:0/3791289048 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8530102400 0x7f853019a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.437+0000 7f8535c96640 1 -- 172.21.15.90:0/3791289048 >> 172.21.15.90:0/3791289048 conn(0x7f8530077880 msgr2=0x7f85301008c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:23.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.438+0000 7f8535c96640 1 -- 172.21.15.90:0/3791289048 shutdown_connections 2024-04-25T11:37:23.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.438+0000 7f8535c96640 1 -- 172.21.15.90:0/3791289048 wait complete. 2024-04-25T11:37:23.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:23 smithi090 ceph-mon[29575]: pgmap v72: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 301 MiB/s wr, 2.86k op/s 2024-04-25T11:37:23.445 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T11:37:23.446 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:37:23.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.633+0000 7feea9c14640 1 -- 172.21.15.90:0/1457792108 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feea4102af0 msgr2=0x7feea4102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:23.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.633+0000 7feea9c14640 1 --2- 172.21.15.90:0/1457792108 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feea4102af0 0x7feea4102f00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fee98009900 tx=0x7fee9802a410 comp rx=0 tx=0).stop 2024-04-25T11:37:23.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.633+0000 7feea9c14640 1 -- 172.21.15.90:0/1457792108 shutdown_connections 2024-04-25T11:37:23.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.633+0000 7feea9c14640 1 --2- 172.21.15.90:0/1457792108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feea410d380 0x7feea410f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.633+0000 7feea9c14640 1 --2- 172.21.15.90:0/1457792108 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feea4102af0 0x7feea4102f00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.633+0000 7feea9c14640 1 --2- 172.21.15.90:0/1457792108 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feea4108a20 0x7feea41025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.633+0000 7feea9c14640 1 -- 172.21.15.90:0/1457792108 >> 172.21.15.90:0/1457792108 conn(0x7feea4078110 msgr2=0x7feea4078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:23.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.633+0000 7feea9c14640 1 -- 172.21.15.90:0/1457792108 shutdown_connections 2024-04-25T11:37:23.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.633+0000 7feea9c14640 1 -- 172.21.15.90:0/1457792108 wait complete. 2024-04-25T11:37:23.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.634+0000 7feea9c14640 1 Processor -- start 2024-04-25T11:37:23.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.634+0000 7feea9c14640 1 -- start start 2024-04-25T11:37:23.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.635+0000 7feea9c14640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feea4102af0 0x7feea419b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:23.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.635+0000 7feea9c14640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feea4108a20 0x7feea419ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:23.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.635+0000 7feea9c14640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feea410d380 0x7feea419bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:23.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.635+0000 7feea9c14640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7feea4112be0 con 0x7feea410d380 2024-04-25T11:37:23.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.635+0000 7feea9c14640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7feea4112a60 con 0x7feea4102af0 2024-04-25T11:37:23.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.635+0000 7feea9c14640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7feea4112d60 con 0x7feea4108a20 2024-04-25T11:37:23.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.635+0000 7feea37fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feea4102af0 0x7feea419b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:23.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.635+0000 7feea3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feea410d380 0x7feea419bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:23.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.635+0000 7feea37fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feea4102af0 0x7feea419b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:46036/0 (socket says 172.21.15.90:46036) 2024-04-25T11:37:23.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.635+0000 7feea2ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feea4108a20 0x7feea419ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:23.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.635+0000 7feea37fe640 1 -- 172.21.15.90:0/1365472094 learned_addr learned my addr 172.21.15.90:0/1365472094 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:23.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.635+0000 7feea2ffd640 1 -- 172.21.15.90:0/1365472094 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feea4102af0 msgr2=0x7feea419b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:23.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.635+0000 7feea2ffd640 1 --2- 172.21.15.90:0/1365472094 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feea4102af0 0x7feea419b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.635+0000 7feea2ffd640 1 -- 172.21.15.90:0/1365472094 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feea410d380 msgr2=0x7feea419bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:23.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.635+0000 7feea2ffd640 1 --2- 172.21.15.90:0/1365472094 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feea410d380 0x7feea419bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.635+0000 7feea2ffd640 1 -- 172.21.15.90:0/1365472094 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feea41a6840 con 0x7feea4108a20 2024-04-25T11:37:23.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.635+0000 7feea3fff640 1 --2- 172.21.15.90:0/1365472094 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feea410d380 0x7feea419bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:37:23.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.636+0000 7feea2ffd640 1 --2- 172.21.15.90:0/1365472094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feea4108a20 0x7feea419ba40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fee9802a930 tx=0x7fee98031a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:23.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.636+0000 7feea0ff9640 1 -- 172.21.15.90:0/1365472094 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9803f070 con 0x7feea4108a20 2024-04-25T11:37:23.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.636+0000 7feea0ff9640 1 -- 172.21.15.90:0/1365472094 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee98004490 con 0x7feea4108a20 2024-04-25T11:37:23.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.636+0000 7feea9c14640 1 -- 172.21.15.90:0/1365472094 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feea41a6ad0 con 0x7feea4108a20 2024-04-25T11:37:23.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.636+0000 7feea0ff9640 1 -- 172.21.15.90:0/1365472094 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee98035070 con 0x7feea4108a20 2024-04-25T11:37:23.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.636+0000 7feea9c14640 1 -- 172.21.15.90:0/1365472094 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feea41a6fc0 con 0x7feea4108a20 2024-04-25T11:37:23.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.637+0000 7feea9c14640 1 -- 172.21.15.90:0/1365472094 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feea4103cd0 con 0x7feea4108a20 2024-04-25T11:37:23.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.638+0000 7feea0ff9640 1 -- 172.21.15.90:0/1365472094 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fee9803b460 con 0x7feea4108a20 2024-04-25T11:37:23.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.639+0000 7feea0ff9640 1 --2- 172.21.15.90:0/1365472094 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fee8c0afc60 0x7fee8c0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:23.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.639+0000 7feea37fe640 1 --2- 172.21.15.90:0/1365472094 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fee8c0afc60 0x7fee8c0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:23.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.639+0000 7feea0ff9640 1 -- 172.21.15.90:0/1365472094 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fee980ff6b0 con 0x7feea4108a20 2024-04-25T11:37:23.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.640+0000 7feea37fe640 1 --2- 172.21.15.90:0/1365472094 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fee8c0afc60 0x7fee8c0b2170 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fee940058c0 tx=0x7fee94005f90 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:23.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.643+0000 7feea0ff9640 1 -- 172.21.15.90:0/1365472094 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fee980c78a0 con 0x7feea4108a20 2024-04-25T11:37:23.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:23 smithi138 ceph-mon[32328]: pgmap v72: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 301 MiB/s wr, 2.86k op/s 2024-04-25T11:37:23.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:23 smithi156 ceph-mon[33045]: pgmap v72: 97 pgs: 97 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 301 MiB/s wr, 2.86k op/s 2024-04-25T11:37:23.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.994+0000 7feea9c14640 1 -- 172.21.15.90:0/1365472094 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feea4106ee0 con 0x7feea4108a20 2024-04-25T11:37:23.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:23.999+0000 7feea0ff9640 1 -- 172.21.15.90:0/1365472094 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11931 (secure 0 0 0) 0x7fee980cbba0 con 0x7feea4108a20 2024-04-25T11:37:23.990 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:37:23.991 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:37:15.460907+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24475,24524,24499,24431,24458]},"id":1}]} 2024-04-25T11:37:23.991 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-25T11:37:23.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.005+0000 7fee8a7fc640 1 -- 172.21.15.90:0/1365472094 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fee8c0afc60 msgr2=0x7fee8c0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:23.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.005+0000 7fee8a7fc640 1 --2- 172.21.15.90:0/1365472094 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fee8c0afc60 0x7fee8c0b2170 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fee940058c0 tx=0x7fee94005f90 comp rx=0 tx=0).stop 2024-04-25T11:37:23.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.006+0000 7fee8a7fc640 1 -- 172.21.15.90:0/1365472094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feea4108a20 msgr2=0x7feea419ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:23.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.006+0000 7fee8a7fc640 1 --2- 172.21.15.90:0/1365472094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feea4108a20 0x7feea419ba40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fee9802a930 tx=0x7fee98031a20 comp rx=0 tx=0).stop 2024-04-25T11:37:23.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.009+0000 7fee8a7fc640 1 -- 172.21.15.90:0/1365472094 shutdown_connections 2024-04-25T11:37:23.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.009+0000 7fee8a7fc640 1 --2- 172.21.15.90:0/1365472094 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fee8c0afc60 0x7fee8c0b2170 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.009+0000 7fee8a7fc640 1 --2- 172.21.15.90:0/1365472094 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feea410d380 0x7feea419bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.009+0000 7fee8a7fc640 1 --2- 172.21.15.90:0/1365472094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feea4108a20 0x7feea419ba40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.009+0000 7fee8a7fc640 1 --2- 172.21.15.90:0/1365472094 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feea4102af0 0x7feea419b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:23.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.009+0000 7fee8a7fc640 1 -- 172.21.15.90:0/1365472094 >> 172.21.15.90:0/1365472094 conn(0x7feea4078110 msgr2=0x7feea41040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:23.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.011+0000 7fee8a7fc640 1 -- 172.21.15.90:0/1365472094 shutdown_connections 2024-04-25T11:37:23.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.011+0000 7fee8a7fc640 1 -- 172.21.15.90:0/1365472094 wait complete. 2024-04-25T11:37:24.036 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T11:37:24.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.379+0000 7f4815d06640 1 -- 172.21.15.90:0/2522012009 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f481010a0b0 msgr2=0x7f481010a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:24.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.379+0000 7f4815d06640 1 --2- 172.21.15.90:0/2522012009 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f481010a0b0 0x7f481010a4c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4808009c10 tx=0x7f480802a790 comp rx=0 tx=0).stop 2024-04-25T11:37:24.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.380+0000 7f4815d06640 1 -- 172.21.15.90:0/2522012009 shutdown_connections 2024-04-25T11:37:24.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.380+0000 7f4815d06640 1 --2- 172.21.15.90:0/2522012009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f481011d120 0x7f481011f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.380+0000 7f4815d06640 1 --2- 172.21.15.90:0/2522012009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f481011a7c0 0x7f481011cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.380+0000 7f4815d06640 1 --2- 172.21.15.90:0/2522012009 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f481010a0b0 0x7f481010a4c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.380+0000 7f4815d06640 1 -- 172.21.15.90:0/2522012009 >> 172.21.15.90:0/2522012009 conn(0x7f481006e7a0 msgr2=0x7f481006ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:24.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.380+0000 7f4815d06640 1 -- 172.21.15.90:0/2522012009 shutdown_connections 2024-04-25T11:37:24.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.380+0000 7f4815d06640 1 -- 172.21.15.90:0/2522012009 wait complete. 2024-04-25T11:37:24.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.381+0000 7f4815d06640 1 Processor -- start 2024-04-25T11:37:24.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.384+0000 7f4815d06640 1 -- start start 2024-04-25T11:37:24.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.384+0000 7f4815d06640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f481011a7c0 0x7f48101144c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:24.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.384+0000 7f4815d06640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f481011d120 0x7f4810114a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:24.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.384+0000 7f4815d06640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4810114f40 0x7f48101bdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:24.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.384+0000 7f4815d06640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4810112870 con 0x7f481011a7c0 2024-04-25T11:37:24.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.384+0000 7f4815d06640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f48101126f0 con 0x7f481011d120 2024-04-25T11:37:24.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.385+0000 7f4815d06640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f48101129f0 con 0x7f4810114f40 2024-04-25T11:37:24.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.387+0000 7f480effd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f481011d120 0x7f4810114a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:24.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.387+0000 7f480ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4810114f40 0x7f48101bdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:24.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.387+0000 7f480f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f481011a7c0 0x7f48101144c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:24.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.387+0000 7f480f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f481011a7c0 0x7f48101144c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55540/0 (socket says 172.21.15.90:55540) 2024-04-25T11:37:24.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.387+0000 7f480f7fe640 1 -- 172.21.15.90:0/753661074 learned_addr learned my addr 172.21.15.90:0/753661074 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:24.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.391+0000 7f480f7fe640 1 -- 172.21.15.90:0/753661074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4810114f40 msgr2=0x7f48101bdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:24.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.391+0000 7f480f7fe640 1 --2- 172.21.15.90:0/753661074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4810114f40 0x7f48101bdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.391+0000 7f480f7fe640 1 -- 172.21.15.90:0/753661074 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f481011d120 msgr2=0x7f4810114a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:24.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.391+0000 7f480f7fe640 1 --2- 172.21.15.90:0/753661074 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f481011d120 0x7f4810114a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.391+0000 7f480f7fe640 1 -- 172.21.15.90:0/753661074 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48101be4f0 con 0x7f481011a7c0 2024-04-25T11:37:24.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.392+0000 7f480f7fe640 1 --2- 172.21.15.90:0/753661074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f481011a7c0 0x7f48101144c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f480802ad00 tx=0x7f4808003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:24.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.400+0000 7f480cff9640 1 -- 172.21.15.90:0/753661074 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4808041020 con 0x7f481011a7c0 2024-04-25T11:37:24.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.405+0000 7f4815d06640 1 -- 172.21.15.90:0/753661074 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48101be780 con 0x7f481011a7c0 2024-04-25T11:37:24.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.406+0000 7f4815d06640 1 -- 172.21.15.90:0/753661074 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48101bebf0 con 0x7f481011a7c0 2024-04-25T11:37:24.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.406+0000 7f480cff9640 1 -- 172.21.15.90:0/753661074 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48080318f0 con 0x7f481011a7c0 2024-04-25T11:37:24.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.406+0000 7f480cff9640 1 -- 172.21.15.90:0/753661074 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4808036040 con 0x7f481011a7c0 2024-04-25T11:37:24.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.411+0000 7f481cbb9740 1 Processor -- start 2024-04-25T11:37:24.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.411+0000 7f481cbb9740 1 -- start start 2024-04-25T11:37:24.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.411+0000 7f481cbb9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558039c8d390 0x55803a02e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:24.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.412+0000 7f481cbb9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558039c644d0 0x55803a02ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:24.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.412+0000 7f481cbb9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c6c6b0 0x55803a03b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:24.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.412+0000 7f481cbb9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558039f93d80 con 0x558039c8d390 2024-04-25T11:37:24.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.412+0000 7f481cbb9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x558039f4da20 con 0x558039c6c6b0 2024-04-25T11:37:24.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.412+0000 7f481cbb9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558039f8fd40 con 0x558039c644d0 2024-04-25T11:37:24.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.413+0000 7f480effd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c6c6b0 0x55803a03b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:24.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.413+0000 7f480effd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c6c6b0 0x55803a03b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:46060/0 (socket says 172.21.15.90:46060) 2024-04-25T11:37:24.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.413+0000 7f480effd640 1 -- 172.21.15.90:0/3690933635 learned_addr learned my addr 172.21.15.90:0/3690933635 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:24.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.415+0000 7f480effd640 1 -- 172.21.15.90:0/3690933635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558039c644d0 msgr2=0x55803a02ed50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:37:24.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.415+0000 7f480effd640 1 --2- 172.21.15.90:0/3690933635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558039c644d0 0x55803a02ed50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.415+0000 7f480effd640 1 -- 172.21.15.90:0/3690933635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558039c8d390 msgr2=0x55803a02e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:24.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.415+0000 7f480effd640 1 --2- 172.21.15.90:0/3690933635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558039c8d390 0x55803a02e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.415+0000 7f480effd640 1 -- 172.21.15.90:0/3690933635 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558039f58c30 con 0x558039c6c6b0 2024-04-25T11:37:24.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.416+0000 7f480effd640 1 --2- 172.21.15.90:0/3690933635 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c6c6b0 0x55803a03b6a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4800009cc0 tx=0x7f48000117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ebf16ab0bd86c65 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:24.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.417+0000 7f47f5ffb640 1 -- 172.21.15.90:0/3690933635 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4800004020 con 0x558039c6c6b0 2024-04-25T11:37:24.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.417+0000 7f47f5ffb640 1 -- 172.21.15.90:0/3690933635 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48000041c0 con 0x558039c6c6b0 2024-04-25T11:37:24.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.418+0000 7f481cbb9740 1 -- 172.21.15.90:0/3690933635 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c6c6b0 msgr2=0x55803a03b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:24.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.418+0000 7f481cbb9740 1 --2- 172.21.15.90:0/3690933635 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c6c6b0 0x55803a03b6a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4800009cc0 tx=0x7f48000117b0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.420+0000 7f481cbb9740 1 -- 172.21.15.90:0/3690933635 shutdown_connections 2024-04-25T11:37:24.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.420+0000 7f481cbb9740 1 --2- 172.21.15.90:0/3690933635 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c6c6b0 0x55803a03b6a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.420+0000 7f481cbb9740 1 --2- 172.21.15.90:0/3690933635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558039c644d0 0x55803a02ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.420+0000 7f481cbb9740 1 --2- 172.21.15.90:0/3690933635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558039c8d390 0x55803a02e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.420+0000 7f481cbb9740 1 -- 172.21.15.90:0/3690933635 >> 172.21.15.90:0/3690933635 conn(0x558039c814e0 msgr2=0x55803a02afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:24.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.456+0000 7f480cff9640 1 -- 172.21.15.90:0/753661074 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f4808044020 con 0x7f481011a7c0 2024-04-25T11:37:24.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.458+0000 7f480cff9640 1 --2- 172.21.15.90:0/753661074 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f47f80afdf0 0x7f47f80b2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:24.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.461+0000 7f480effd640 1 --2- 172.21.15.90:0/753661074 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f47f80afdf0 0x7f47f80b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:24.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.464+0000 7f480effd640 1 --2- 172.21.15.90:0/753661074 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f47f80afdf0 0x7f47f80b2300 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4800009c90 tx=0x7f4800005850 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:24.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.465+0000 7f481cbb9740 1 -- 172.21.15.90:0/3690933635 shutdown_connections 2024-04-25T11:37:24.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.473+0000 7f480cff9640 1 -- 172.21.15.90:0/753661074 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f48080c3c30 con 0x7f481011a7c0 2024-04-25T11:37:24.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.473+0000 7f481cbb9740 1 -- 172.21.15.90:0/3690933635 wait complete. 2024-04-25T11:37:24.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.474+0000 7f481cbb9740 1 -- start start 2024-04-25T11:37:24.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.474+0000 7f481cbb9740 1 Processor -- start 2024-04-25T11:37:24.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.475+0000 7f481cbb9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c6c6b0 0x55803a038de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:24.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.475+0000 7f481cbb9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558039c8d390 0x55803a039320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:24.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.475+0000 7f481cbb9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558039c4bf60 0x55803a039860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:24.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.475+0000 7f481cbb9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55803a03bde0 con 0x558039c4bf60 2024-04-25T11:37:24.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.475+0000 7f481cbb9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x558039f1adc0 con 0x558039c6c6b0 2024-04-25T11:37:24.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.475+0000 7f481cbb9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55803a03bf60 con 0x558039c8d390 2024-04-25T11:37:24.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.475+0000 7f481cbb9740 10 client.0 _ll_register_callbacks cb 0x558039f36600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:37:24.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.477+0000 7f480ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558039c4bf60 0x55803a039860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:24.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.478+0000 7f480ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558039c4bf60 0x55803a039860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55548/0 (socket says 172.21.15.90:55548) 2024-04-25T11:37:24.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.478+0000 7f480ffff640 1 -- 172.21.15.90:0/923026249 learned_addr learned my addr 172.21.15.90:0/923026249 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:24.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.479+0000 7f480effd640 1 --2- 172.21.15.90:0/923026249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558039c8d390 0x55803a039320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:24.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.481+0000 7f480f7fe640 1 --2- 172.21.15.90:0/923026249 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c6c6b0 0x55803a038de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:24.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.482+0000 7f480ffff640 1 -- 172.21.15.90:0/923026249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558039c8d390 msgr2=0x55803a039320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:24.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.482+0000 7f480ffff640 1 --2- 172.21.15.90:0/923026249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558039c8d390 0x55803a039320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.482+0000 7f480ffff640 1 -- 172.21.15.90:0/923026249 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c6c6b0 msgr2=0x55803a038de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:24.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.482+0000 7f480ffff640 1 --2- 172.21.15.90:0/923026249 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c6c6b0 0x55803a038de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.482+0000 7f480ffff640 1 -- 172.21.15.90:0/923026249 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558039ca53f0 con 0x558039c4bf60 2024-04-25T11:37:24.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.482+0000 7f480f7fe640 1 --2- 172.21.15.90:0/923026249 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c6c6b0 0x55803a038de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:37:24.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.483+0000 7f480ffff640 1 --2- 172.21.15.90:0/923026249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558039c4bf60 0x55803a039860 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f4804004700 tx=0x7f480400bd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b831d4d33e5289c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:24.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.488+0000 7f47eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:37:24.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.488+0000 7f47eeffd640 1 -- 172.21.15.90:0/923026249 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f480401a470 con 0x558039c4bf60 2024-04-25T11:37:24.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.488+0000 7f47eeffd640 1 -- 172.21.15.90:0/923026249 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f480401a610 con 0x558039c4bf60 2024-04-25T11:37:24.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.488+0000 7f481cbb9740 1 -- 172.21.15.90:0/923026249 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48000041c0 con 0x558039c4bf60 2024-04-25T11:37:24.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.493+0000 7f47eeffd640 1 -- 172.21.15.90:0/923026249 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f480401a810 con 0x558039c4bf60 2024-04-25T11:37:24.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.493+0000 7f481cbb9740 10 client.16245 fetch_fsmap learned FSMap version 27 2024-04-25T11:37:24.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.493+0000 7f481cbb9740 1 -- 172.21.15.90:0/923026249 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558039d6f7d0 con 0x558039c4bf60 2024-04-25T11:37:24.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.495+0000 7f47eeffd640 1 -- 172.21.15.90:0/923026249 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48000041c0 con 0x558039c4bf60 2024-04-25T11:37:24.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.495+0000 7f47eeffd640 1 -- 172.21.15.90:0/923026249 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f4804011ac0 con 0x558039c4bf60 2024-04-25T11:37:24.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.496+0000 7f481cbb9740 10 client.16245 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:37:24.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.496+0000 7f481cbb9740 10 client.16245 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:37:24.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.496+0000 7f481cbb9740 20 client.16245 populate_metadata read hostname 'smithi090' 2024-04-25T11:37:24.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.496+0000 7f481cbb9740 1 --2- 172.21.15.90:0/923026249 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x558039c213a0 0x55803a04bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:24.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.496+0000 7f481cbb9740 4 client.16245 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:37:24.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.496+0000 7f481cbb9740 1 -- 172.21.15.90:0/923026249 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558039fa1990 con 0x558039c213a0 2024-04-25T11:37:24.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.504+0000 7f480f7fe640 1 --2- 172.21.15.90:0/923026249 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x558039c213a0 0x55803a04bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:24.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.526+0000 7f47eeffd640 10 client.16245 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:37:24.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.526+0000 7f480f7fe640 1 --2- 172.21.15.90:0/923026249 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x558039c213a0 0x55803a04bac0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3ccc6b8eae220600 server_cookie=ab56e0537bea4f94 in_seq=0 out_seq=0 2024-04-25T11:37:24.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.532+0000 7f47eeffd640 1 -- 172.21.15.90:0/923026249 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558039fa1990 con 0x558039c213a0 2024-04-25T11:37:24.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.532+0000 7f47eeffd640 10 client.16245 handle_command_reply: tid=0 2024-04-25T11:37:24.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.532+0000 7f481cbb9740 1 client.16245 shutdown 2024-04-25T11:37:24.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.539+0000 7f481cbb9740 1 -- 172.21.15.90:0/923026249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558039c4bf60 msgr2=0x55803a039860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:24.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.539+0000 7f481cbb9740 1 --2- 172.21.15.90:0/923026249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558039c4bf60 0x55803a039860 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f4804004700 tx=0x7f480400bd10 comp rx=0 tx=0).stop 2024-04-25T11:37:24.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.540+0000 7f481cbb9740 1 -- 172.21.15.90:0/923026249 shutdown_connections 2024-04-25T11:37:24.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.540+0000 7f481cbb9740 1 --2- 172.21.15.90:0/923026249 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x558039c213a0 0x55803a04bac0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.540+0000 7f481cbb9740 1 --2- 172.21.15.90:0/923026249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558039c4bf60 0x55803a039860 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.540+0000 7f481cbb9740 1 --2- 172.21.15.90:0/923026249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558039c8d390 0x55803a039320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.540+0000 7f481cbb9740 1 --2- 172.21.15.90:0/923026249 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c6c6b0 0x55803a038de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.540+0000 7f481cbb9740 1 -- 172.21.15.90:0/923026249 >> 172.21.15.90:0/923026249 conn(0x558039c814e0 msgr2=0x55803a030530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:24.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.541+0000 7f47eeffd640 0 client.16245 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:37:24.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.544+0000 7f481cbb9740 1 -- 172.21.15.90:0/923026249 shutdown_connections 2024-04-25T11:37:24.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.545+0000 7f481cbb9740 1 -- 172.21.15.90:0/923026249 wait complete. 2024-04-25T11:37:24.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.546+0000 7f481cbb9740 20 client.16245 trim_cache size 0 max 16384 2024-04-25T11:37:24.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.608+0000 7f481cbb9740 1 Processor -- start 2024-04-25T11:37:24.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.609+0000 7f481cbb9740 1 -- start start 2024-04-25T11:37:24.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.609+0000 7f481cbb9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c814e0 0x55803a030b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:24.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.610+0000 7f481cbb9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558039c4bf60 0x55803a039330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:24.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.610+0000 7f481cbb9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558039c6c6b0 0x55803a039870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:24.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.610+0000 7f481cbb9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc00eab0 con 0x558039c4bf60 2024-04-25T11:37:24.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.610+0000 7f481cbb9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc010620 con 0x558039c814e0 2024-04-25T11:37:24.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.610+0000 7f480f7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c814e0 0x55803a030b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:24.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.610+0000 7f481cbb9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc00f180 con 0x558039c6c6b0 2024-04-25T11:37:24.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.610+0000 7f480f7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c814e0 0x55803a030b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:46082/0 (socket says 172.21.15.90:46082) 2024-04-25T11:37:24.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.610+0000 7f480f7fe640 1 -- 172.21.15.90:0/1445498195 learned_addr learned my addr 172.21.15.90:0/1445498195 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:24.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.611+0000 7f480effd640 1 --2- 172.21.15.90:0/1445498195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558039c4bf60 0x55803a039330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:24.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.612+0000 7f480f7fe640 1 -- 172.21.15.90:0/1445498195 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558039c6c6b0 msgr2=0x55803a039870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:24.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.612+0000 7f480f7fe640 1 --2- 172.21.15.90:0/1445498195 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558039c6c6b0 0x55803a039870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.612+0000 7f480f7fe640 1 -- 172.21.15.90:0/1445498195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558039c4bf60 msgr2=0x55803a039330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:24.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.612+0000 7f480f7fe640 1 --2- 172.21.15.90:0/1445498195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558039c4bf60 0x55803a039330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.612+0000 7f480f7fe640 1 -- 172.21.15.90:0/1445498195 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55803a02ffc0 con 0x558039c814e0 2024-04-25T11:37:24.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.613+0000 7f480f7fe640 1 --2- 172.21.15.90:0/1445498195 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c814e0 0x55803a030b40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f48080c36c0 tx=0x7f48080093a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3b0eb9619b53494 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:24.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.614+0000 7f47f57fa640 1 -- 172.21.15.90:0/1445498195 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48080cb9f0 con 0x558039c814e0 2024-04-25T11:37:24.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.614+0000 7f47f57fa640 1 -- 172.21.15.90:0/1445498195 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x558039fa1990 con 0x558039c814e0 2024-04-25T11:37:24.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.614+0000 7f47f57fa640 1 -- 172.21.15.90:0/1445498195 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48080fce70 con 0x558039c814e0 2024-04-25T11:37:24.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.616+0000 7f481cbb9740 1 -- 172.21.15.90:0/1445498195 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c814e0 msgr2=0x55803a030b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:24.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.616+0000 7f481cbb9740 1 --2- 172.21.15.90:0/1445498195 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c814e0 0x55803a030b40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f48080c36c0 tx=0x7f48080093a0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.616+0000 7f481cbb9740 1 -- 172.21.15.90:0/1445498195 shutdown_connections 2024-04-25T11:37:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.617+0000 7f481cbb9740 1 --2- 172.21.15.90:0/1445498195 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558039c6c6b0 0x55803a039870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.617+0000 7f481cbb9740 1 --2- 172.21.15.90:0/1445498195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558039c4bf60 0x55803a039330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.617+0000 7f481cbb9740 1 --2- 172.21.15.90:0/1445498195 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c814e0 0x55803a030b40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.617+0000 7f481cbb9740 1 -- 172.21.15.90:0/1445498195 >> 172.21.15.90:0/1445498195 conn(0x55803a02ed70 msgr2=0x55803a030430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:24.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.623+0000 7f481cbb9740 1 -- 172.21.15.90:0/1445498195 shutdown_connections 2024-04-25T11:37:24.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.623+0000 7f481cbb9740 1 -- 172.21.15.90:0/1445498195 wait complete. 2024-04-25T11:37:24.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.624+0000 7f481cbb9740 1 -- start start 2024-04-25T11:37:24.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.624+0000 7f481cbb9740 1 Processor -- start 2024-04-25T11:37:24.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.625+0000 7f481cbb9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558039c4bf60 0x55803a0414d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:24.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.626+0000 7f481cbb9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c6c6b0 0x55803a041a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:24.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.626+0000 7f481cbb9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558039c814e0 0x55803a03ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:24.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.626+0000 7f481cbb9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc00ff20 con 0x558039c4bf60 2024-04-25T11:37:24.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.626+0000 7f481cbb9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc00f850 con 0x558039c6c6b0 2024-04-25T11:37:24.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.626+0000 7f481cbb9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc00ca90 con 0x558039c814e0 2024-04-25T11:37:24.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.626+0000 7f481cbb9740 10 client.0 _ll_register_callbacks cb 0x7f4804004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:37:24.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.627+0000 7f480effd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c6c6b0 0x55803a041a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:24.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.627+0000 7f480effd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c6c6b0 0x55803a041a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:46098/0 (socket says 172.21.15.90:46098) 2024-04-25T11:37:24.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.627+0000 7f480effd640 1 -- 172.21.15.90:0/1799212605 learned_addr learned my addr 172.21.15.90:0/1799212605 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:24.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.627+0000 7f480ffff640 1 --2- 172.21.15.90:0/1799212605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558039c814e0 0x55803a03ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:24.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.627+0000 7f480effd640 1 -- 172.21.15.90:0/1799212605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558039c814e0 msgr2=0x55803a03ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:24.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.627+0000 7f480effd640 1 --2- 172.21.15.90:0/1799212605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558039c814e0 0x55803a03ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.628+0000 7f480effd640 1 -- 172.21.15.90:0/1799212605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558039c4bf60 msgr2=0x55803a0414d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:24.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.628+0000 7f480effd640 1 --2- 172.21.15.90:0/1799212605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558039c4bf60 0x55803a0414d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.628+0000 7f480effd640 1 -- 172.21.15.90:0/1799212605 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55803a03d180 con 0x558039c6c6b0 2024-04-25T11:37:24.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.629+0000 7f480effd640 1 --2- 172.21.15.90:0/1799212605 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c6c6b0 0x55803a041a10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4800003e90 tx=0x7f4800009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bf2c3388e141f37 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:24.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.640+0000 7f47eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:37:24.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.640+0000 7f47eeffd640 1 -- 172.21.15.90:0/1799212605 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f480001f040 con 0x558039c6c6b0 2024-04-25T11:37:24.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.640+0000 7f481cbb9740 1 -- 172.21.15.90:0/1799212605 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558039fa1990 con 0x558039c6c6b0 2024-04-25T11:37:24.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.640+0000 7f47eeffd640 1 -- 172.21.15.90:0/1799212605 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f4800005240 con 0x558039c6c6b0 2024-04-25T11:37:24.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.641+0000 7f47eeffd640 1 -- 172.21.15.90:0/1799212605 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4800041050 con 0x558039c6c6b0 2024-04-25T11:37:24.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.641+0000 7f481cbb9740 10 client.25634 fetch_fsmap learned FSMap version 27 2024-04-25T11:37:24.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.641+0000 7f481cbb9740 1 -- 172.21.15.90:0/1799212605 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55803a0b3950 con 0x558039c6c6b0 2024-04-25T11:37:24.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.645+0000 7f47eeffd640 1 -- 172.21.15.90:0/1799212605 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4800038070 con 0x558039c6c6b0 2024-04-25T11:37:24.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.645+0000 7f47eeffd640 1 -- 172.21.15.90:0/1799212605 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f4800026750 con 0x558039c6c6b0 2024-04-25T11:37:24.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.645+0000 7f481cbb9740 10 client.25634 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:37:24.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.645+0000 7f481cbb9740 10 client.25634 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:37:24.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.645+0000 7f481cbb9740 20 client.25634 populate_metadata read hostname 'smithi090' 2024-04-25T11:37:24.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.645+0000 7f481cbb9740 1 --2- 172.21.15.90:0/1799212605 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x558039c8d390 0x55803a0b5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:24.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.645+0000 7f481cbb9740 4 client.25634 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:37:24.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.645+0000 7f481cbb9740 1 -- 172.21.15.90:0/1799212605 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558039fa1990 con 0x558039c8d390 2024-04-25T11:37:24.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.646+0000 7f480f7fe640 1 --2- 172.21.15.90:0/1799212605 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x558039c8d390 0x55803a0b5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:24.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.648+0000 7f480f7fe640 1 --2- 172.21.15.90:0/1799212605 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x558039c8d390 0x55803a0b5fa0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1eee5a49e91a58c5 server_cookie=4b6279ffaee65647 in_seq=0 out_seq=0 2024-04-25T11:37:24.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.649+0000 7f47eeffd640 10 client.25634 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:37:24.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.651+0000 7f47eeffd640 1 -- 172.21.15.90:0/1799212605 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558039fa1990 con 0x558039c8d390 2024-04-25T11:37:24.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.651+0000 7f47eeffd640 10 client.25634 handle_command_reply: tid=0 2024-04-25T11:37:24.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.652+0000 7f481cbb9740 1 client.25634 shutdown 2024-04-25T11:37:24.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.652+0000 7f481cbb9740 1 -- 172.21.15.90:0/1799212605 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c6c6b0 msgr2=0x55803a041a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:24.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.652+0000 7f481cbb9740 1 --2- 172.21.15.90:0/1799212605 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c6c6b0 0x55803a041a10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4800003e90 tx=0x7f4800009260 comp rx=0 tx=0).stop 2024-04-25T11:37:24.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.653+0000 7f481cbb9740 1 -- 172.21.15.90:0/1799212605 shutdown_connections 2024-04-25T11:37:24.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.653+0000 7f481cbb9740 1 --2- 172.21.15.90:0/1799212605 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x558039c8d390 0x55803a0b5fa0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.653+0000 7f481cbb9740 1 --2- 172.21.15.90:0/1799212605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558039c814e0 0x55803a03ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.653+0000 7f481cbb9740 1 --2- 172.21.15.90:0/1799212605 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558039c6c6b0 0x55803a041a10 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.653+0000 7f481cbb9740 1 --2- 172.21.15.90:0/1799212605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558039c4bf60 0x55803a0414d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.653+0000 7f481cbb9740 1 -- 172.21.15.90:0/1799212605 >> 172.21.15.90:0/1799212605 conn(0x55803a02ed70 msgr2=0x55803a043fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:24.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.657+0000 7f47eeffd640 0 client.25634 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:37:24.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.657+0000 7f481cbb9740 1 -- 172.21.15.90:0/1799212605 shutdown_connections 2024-04-25T11:37:24.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.660+0000 7f481cbb9740 1 -- 172.21.15.90:0/1799212605 wait complete. 2024-04-25T11:37:24.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.661+0000 7f481cbb9740 20 client.25634 trim_cache size 0 max 16384 2024-04-25T11:37:24.653 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:37:24.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.685+0000 7f4815d06640 1 -- 172.21.15.90:0/753661074 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f47f80afdf0 msgr2=0x7f47f80b2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:24.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.685+0000 7f4815d06640 1 --2- 172.21.15.90:0/753661074 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f47f80afdf0 0x7f47f80b2300 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4800009c90 tx=0x7f4800005850 comp rx=0 tx=0).stop 2024-04-25T11:37:24.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.685+0000 7f4815d06640 1 -- 172.21.15.90:0/753661074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f481011a7c0 msgr2=0x7f48101144c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:24.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.685+0000 7f4815d06640 1 --2- 172.21.15.90:0/753661074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f481011a7c0 0x7f48101144c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f480802ad00 tx=0x7f4808003ba0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.691+0000 7f4815d06640 1 -- 172.21.15.90:0/753661074 shutdown_connections 2024-04-25T11:37:24.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.691+0000 7f4815d06640 1 --2- 172.21.15.90:0/753661074 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f47f80afdf0 0x7f47f80b2300 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.691+0000 7f4815d06640 1 --2- 172.21.15.90:0/753661074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4810114f40 0x7f48101bdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.691+0000 7f4815d06640 1 --2- 172.21.15.90:0/753661074 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f481011d120 0x7f4810114a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.691+0000 7f4815d06640 1 --2- 172.21.15.90:0/753661074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f481011a7c0 0x7f48101144c0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:24.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.691+0000 7f4815d06640 1 -- 172.21.15.90:0/753661074 >> 172.21.15.90:0/753661074 conn(0x7f481006e7a0 msgr2=0x7f481011abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:24.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.691+0000 7f4815d06640 1 -- 172.21.15.90:0/753661074 shutdown_connections 2024-04-25T11:37:24.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:24.691+0000 7f4815d06640 1 -- 172.21.15.90:0/753661074 wait complete. 2024-04-25T11:37:24.725 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T11:37:24.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:24 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1365472094' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:37:25.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:24 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1365472094' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:37:25.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.076+0000 7f1971430640 1 -- 172.21.15.90:0/2102007459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f196c11d120 msgr2=0x7f196c11f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:25.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.076+0000 7f1971430640 1 --2- 172.21.15.90:0/2102007459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f196c11d120 0x7f196c11f540 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f1960009900 tx=0x7f196002a4b0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.077+0000 7f1971430640 1 -- 172.21.15.90:0/2102007459 shutdown_connections 2024-04-25T11:37:25.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.077+0000 7f1971430640 1 --2- 172.21.15.90:0/2102007459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f196c11d120 0x7f196c11f540 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.077+0000 7f1971430640 1 --2- 172.21.15.90:0/2102007459 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f196c11a7c0 0x7f196c11cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.077+0000 7f1971430640 1 --2- 172.21.15.90:0/2102007459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f196c10a0b0 0x7f196c10a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.077+0000 7f1971430640 1 -- 172.21.15.90:0/2102007459 >> 172.21.15.90:0/2102007459 conn(0x7f196c06e7a0 msgr2=0x7f196c06ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:25.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.077+0000 7f1971430640 1 -- 172.21.15.90:0/2102007459 shutdown_connections 2024-04-25T11:37:25.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.077+0000 7f1971430640 1 -- 172.21.15.90:0/2102007459 wait complete. 2024-04-25T11:37:25.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.078+0000 7f1971430640 1 Processor -- start 2024-04-25T11:37:25.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.078+0000 7f1971430640 1 -- start start 2024-04-25T11:37:25.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.079+0000 7f1971430640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f196c10a0b0 0x7f196c1a9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.079+0000 7f1971430640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f196c11a7c0 0x7f196c1a98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.079+0000 7f1971430640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f196c11d120 0x7f196c1adba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.079+0000 7f1971430640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f196c122c00 con 0x7f196c10a0b0 2024-04-25T11:37:25.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.079+0000 7f1971430640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f196c122a80 con 0x7f196c11d120 2024-04-25T11:37:25.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.079+0000 7f1971430640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f196c122d80 con 0x7f196c11a7c0 2024-04-25T11:37:25.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.079+0000 7f196affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f196c10a0b0 0x7f196c1a9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.079+0000 7f196affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f196c10a0b0 0x7f196c1a9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55592/0 (socket says 172.21.15.90:55592) 2024-04-25T11:37:25.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.079+0000 7f196affd640 1 -- 172.21.15.90:0/1675952432 learned_addr learned my addr 172.21.15.90:0/1675952432 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:25.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.080+0000 7f196affd640 1 -- 172.21.15.90:0/1675952432 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f196c11a7c0 msgr2=0x7f196c1a98e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:37:25.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.080+0000 7f196affd640 1 --2- 172.21.15.90:0/1675952432 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f196c11a7c0 0x7f196c1a98e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.080+0000 7f196affd640 1 -- 172.21.15.90:0/1675952432 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f196c11d120 msgr2=0x7f196c1adba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:25.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.080+0000 7f196affd640 1 --2- 172.21.15.90:0/1675952432 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f196c11d120 0x7f196c1adba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.080+0000 7f196affd640 1 -- 172.21.15.90:0/1675952432 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f196c1ae1e0 con 0x7f196c10a0b0 2024-04-25T11:37:25.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.080+0000 7f196affd640 1 --2- 172.21.15.90:0/1675952432 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f196c10a0b0 0x7f196c1a9380 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f196400ba50 tx=0x7f196400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:25.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.080+0000 7f1953fff640 1 -- 172.21.15.90:0/1675952432 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1964019070 con 0x7f196c10a0b0 2024-04-25T11:37:25.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.080+0000 7f1953fff640 1 -- 172.21.15.90:0/1675952432 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f196400cbe0 con 0x7f196c10a0b0 2024-04-25T11:37:25.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.080+0000 7f1953fff640 1 -- 172.21.15.90:0/1675952432 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19640075d0 con 0x7f196c10a0b0 2024-04-25T11:37:25.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.080+0000 7f1971430640 1 -- 172.21.15.90:0/1675952432 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f196c1ae4d0 con 0x7f196c10a0b0 2024-04-25T11:37:25.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.081+0000 7f1971430640 1 -- 172.21.15.90:0/1675952432 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f196c1b5e50 con 0x7f196c10a0b0 2024-04-25T11:37:25.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.085+0000 7f19783b4740 1 Processor -- start 2024-04-25T11:37:25.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.085+0000 7f19783b4740 1 -- start start 2024-04-25T11:37:25.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.086+0000 7f1953fff640 1 -- 172.21.15.90:0/1675952432 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f196401c070 con 0x7f196c10a0b0 2024-04-25T11:37:25.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.086+0000 7f19783b4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5575a2186390 0x5575a2527810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.086+0000 7f19783b4740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5575a215d4d0 0x5575a2527d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.086+0000 7f19783b4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5575a21656b0 0x5575a25346a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.086+0000 7f19783b4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5575a248cd80 con 0x5575a21656b0 2024-04-25T11:37:25.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.086+0000 7f19783b4740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5575a2446a20 con 0x5575a215d4d0 2024-04-25T11:37:25.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.086+0000 7f19783b4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5575a2488d40 con 0x5575a2186390 2024-04-25T11:37:25.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.087+0000 7f196a7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5575a21656b0 0x5575a25346a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.087+0000 7f196a7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5575a21656b0 0x5575a25346a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55598/0 (socket says 172.21.15.90:55598) 2024-04-25T11:37:25.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.087+0000 7f196a7fc640 1 -- 172.21.15.90:0/2628782372 learned_addr learned my addr 172.21.15.90:0/2628782372 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:25.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.087+0000 7f196b7fe640 1 --2- 172.21.15.90:0/2628782372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5575a2186390 0x5575a2527810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.087+0000 7f196a7fc640 1 -- 172.21.15.90:0/2628782372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5575a2186390 msgr2=0x5575a2527810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:25.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.087+0000 7f1953fff640 1 --2- 172.21.15.90:0/1675952432 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f19540afce0 0x7f19540b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.087+0000 7f196a7fc640 1 --2- 172.21.15.90:0/2628782372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5575a2186390 0x5575a2527810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.087+0000 7f196a7fc640 1 -- 172.21.15.90:0/2628782372 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5575a215d4d0 msgr2=0x5575a2527d50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:37:25.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.087+0000 7f196a7fc640 1 --2- 172.21.15.90:0/2628782372 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5575a215d4d0 0x5575a2527d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.087+0000 7f196a7fc640 1 -- 172.21.15.90:0/2628782372 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575a2451c30 con 0x5575a21656b0 2024-04-25T11:37:25.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.088+0000 7f1953fff640 1 -- 172.21.15.90:0/1675952432 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f19640de5c0 con 0x7f196c10a0b0 2024-04-25T11:37:25.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.088+0000 7f196b7fe640 1 --2- 172.21.15.90:0/1675952432 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f19540afce0 0x7f19540b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.088+0000 7f196a7fc640 1 --2- 172.21.15.90:0/2628782372 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5575a21656b0 0x5575a25346a0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f195c009740 tx=0x7f195c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a63bb7c89c740fc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:25.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.089+0000 7f19517fa640 1 -- 172.21.15.90:0/2628782372 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195c015020 con 0x5575a21656b0 2024-04-25T11:37:25.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.089+0000 7f196b7fe640 1 --2- 172.21.15.90:0/1675952432 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f19540afce0 0x7f19540b21f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f19600092a0 tx=0x7f1960005fd0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:25.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.089+0000 7f19517fa640 1 -- 172.21.15.90:0/2628782372 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f195c009e50 con 0x5575a21656b0 2024-04-25T11:37:25.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.089+0000 7f19517fa640 1 -- 172.21.15.90:0/2628782372 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195c010380 con 0x5575a21656b0 2024-04-25T11:37:25.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.090+0000 7f19783b4740 1 -- 172.21.15.90:0/2628782372 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5575a21656b0 msgr2=0x5575a25346a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:25.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.090+0000 7f19783b4740 1 --2- 172.21.15.90:0/2628782372 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5575a21656b0 0x5575a25346a0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f195c009740 tx=0x7f195c009c00 comp rx=0 tx=0).stop 2024-04-25T11:37:25.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.090+0000 7f19783b4740 1 -- 172.21.15.90:0/2628782372 shutdown_connections 2024-04-25T11:37:25.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.090+0000 7f19783b4740 1 --2- 172.21.15.90:0/2628782372 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5575a21656b0 0x5575a25346a0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.090+0000 7f19783b4740 1 --2- 172.21.15.90:0/2628782372 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5575a215d4d0 0x5575a2527d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.090+0000 7f19783b4740 1 --2- 172.21.15.90:0/2628782372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5575a2186390 0x5575a2527810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.090+0000 7f19783b4740 1 -- 172.21.15.90:0/2628782372 >> 172.21.15.90:0/2628782372 conn(0x5575a217a4e0 msgr2=0x5575a2523fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:25.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.091+0000 7f19783b4740 1 -- 172.21.15.90:0/2628782372 shutdown_connections 2024-04-25T11:37:25.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.091+0000 7f19783b4740 1 -- 172.21.15.90:0/2628782372 wait complete. 2024-04-25T11:37:25.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.091+0000 7f19783b4740 1 -- start start 2024-04-25T11:37:25.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.091+0000 7f19783b4740 1 Processor -- start 2024-04-25T11:37:25.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.092+0000 7f19783b4740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5575a215d4d0 0x5575a2532210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.092+0000 7f19783b4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5575a21656b0 0x5575a2532750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.092+0000 7f19783b4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5575a2186390 0x5575a253e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.092+0000 7f19783b4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5575a2534de0 con 0x5575a2186390 2024-04-25T11:37:25.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.092+0000 7f19783b4740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5575a2413dc0 con 0x5575a215d4d0 2024-04-25T11:37:25.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.092+0000 7f19783b4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5575a2534f60 con 0x5575a21656b0 2024-04-25T11:37:25.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.092+0000 7f196b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5575a2186390 0x5575a253e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.093+0000 7f196b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5575a2186390 0x5575a253e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55600/0 (socket says 172.21.15.90:55600) 2024-04-25T11:37:25.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.093+0000 7f196b7fe640 1 -- 172.21.15.90:0/3526279311 learned_addr learned my addr 172.21.15.90:0/3526279311 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:25.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.093+0000 7f19783b4740 10 client.0 _ll_register_callbacks cb 0x5575a242f600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:37:25.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.093+0000 7f196b7fe640 1 -- 172.21.15.90:0/3526279311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5575a21656b0 msgr2=0x5575a2532750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:25.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.093+0000 7f196b7fe640 1 --2- 172.21.15.90:0/3526279311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5575a21656b0 0x5575a2532750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.093+0000 7f196b7fe640 1 -- 172.21.15.90:0/3526279311 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5575a215d4d0 msgr2=0x5575a2532210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:25.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.093+0000 7f196b7fe640 1 --2- 172.21.15.90:0/3526279311 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5575a215d4d0 0x5575a2532210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.093+0000 7f196b7fe640 1 -- 172.21.15.90:0/3526279311 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575a219e3f0 con 0x5575a2186390 2024-04-25T11:37:25.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.094+0000 7f196b7fe640 1 --2- 172.21.15.90:0/3526279311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5575a2186390 0x5575a253e320 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f1960009900 tx=0x7f1960009230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6f7d77816e96925 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:25.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.094+0000 7f194eff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:37:25.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.094+0000 7f194eff5640 1 -- 172.21.15.90:0/3526279311 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1960035040 con 0x5575a2186390 2024-04-25T11:37:25.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.094+0000 7f194eff5640 1 -- 172.21.15.90:0/3526279311 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f19600035f0 con 0x5575a2186390 2024-04-25T11:37:25.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.094+0000 7f19783b4740 1 -- 172.21.15.90:0/3526279311 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f195c009e50 con 0x5575a2186390 2024-04-25T11:37:25.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.094+0000 7f194eff5640 1 -- 172.21.15.90:0/3526279311 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f19600037f0 con 0x5575a2186390 2024-04-25T11:37:25.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.094+0000 7f19783b4740 10 client.16260 fetch_fsmap learned FSMap version 27 2024-04-25T11:37:25.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.094+0000 7f19783b4740 1 -- 172.21.15.90:0/3526279311 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5575a22687d0 con 0x5575a2186390 2024-04-25T11:37:25.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.094+0000 7f194eff5640 1 -- 172.21.15.90:0/3526279311 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f196004e070 con 0x5575a2186390 2024-04-25T11:37:25.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.095+0000 7f194eff5640 1 -- 172.21.15.90:0/3526279311 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f196003dbb0 con 0x5575a2186390 2024-04-25T11:37:25.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.095+0000 7f19783b4740 10 client.16260 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:37:25.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.095+0000 7f19783b4740 10 client.16260 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:37:25.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.095+0000 7f19783b4740 20 client.16260 populate_metadata read hostname 'smithi090' 2024-04-25T11:37:25.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.095+0000 7f19783b4740 1 --2- 172.21.15.90:0/3526279311 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5575a2144f60 0x5575a2544870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.095+0000 7f19783b4740 4 client.16260 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:37:25.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.095+0000 7f19783b4740 1 -- 172.21.15.90:0/3526279311 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f195c009e50 con 0x5575a2144f60 2024-04-25T11:37:25.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.097+0000 7f196affd640 1 --2- 172.21.15.90:0/3526279311 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5575a2144f60 0x5575a2544870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:24 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1365472094' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:37:25.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.101+0000 7f194eff5640 10 client.16260 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:37:25.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.101+0000 7f196affd640 1 --2- 172.21.15.90:0/3526279311 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5575a2144f60 0x5575a2544870 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b3fe6226b3c51a18 server_cookie=b15060c7e2c514e2 in_seq=0 out_seq=0 2024-04-25T11:37:25.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.104+0000 7f194eff5640 1 -- 172.21.15.90:0/3526279311 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f195c009e50 con 0x5575a2144f60 2024-04-25T11:37:25.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.104+0000 7f194eff5640 10 client.16260 handle_command_reply: tid=0 2024-04-25T11:37:25.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.104+0000 7f19783b4740 1 client.16260 shutdown 2024-04-25T11:37:25.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.105+0000 7f19783b4740 1 -- 172.21.15.90:0/3526279311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5575a2186390 msgr2=0x5575a253e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:25.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.105+0000 7f19783b4740 1 --2- 172.21.15.90:0/3526279311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5575a2186390 0x5575a253e320 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f1960009900 tx=0x7f1960009230 comp rx=0 tx=0).stop 2024-04-25T11:37:25.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.105+0000 7f19783b4740 1 -- 172.21.15.90:0/3526279311 shutdown_connections 2024-04-25T11:37:25.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.105+0000 7f19783b4740 1 --2- 172.21.15.90:0/3526279311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5575a2186390 0x5575a253e320 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.105+0000 7f19783b4740 1 --2- 172.21.15.90:0/3526279311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5575a21656b0 0x5575a2532750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.105+0000 7f19783b4740 1 --2- 172.21.15.90:0/3526279311 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5575a2144f60 0x5575a2544870 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.105+0000 7f19783b4740 1 --2- 172.21.15.90:0/3526279311 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5575a215d4d0 0x5575a2532210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.105+0000 7f19783b4740 1 -- 172.21.15.90:0/3526279311 >> 172.21.15.90:0/3526279311 conn(0x5575a217a4e0 msgr2=0x5575a2529530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:25.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.105+0000 7f194eff5640 0 client.16260 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:37:25.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.106+0000 7f19783b4740 1 -- 172.21.15.90:0/3526279311 shutdown_connections 2024-04-25T11:37:25.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.106+0000 7f19783b4740 1 -- 172.21.15.90:0/3526279311 wait complete. 2024-04-25T11:37:25.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.106+0000 7f19783b4740 20 client.16260 trim_cache size 0 max 16384 2024-04-25T11:37:25.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.131+0000 7f19783b4740 1 Processor -- start 2024-04-25T11:37:25.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.131+0000 7f19783b4740 1 -- start start 2024-04-25T11:37:25.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.132+0000 7f19783b4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5575a215d4d0 0x5575a251f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.132+0000 7f19783b4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5575a21656b0 0x5575a253dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.132+0000 7f19783b4740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5575a2186390 0x5575a254a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.132+0000 7f19783b4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f196c1be770 con 0x5575a21656b0 2024-04-25T11:37:25.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.132+0000 7f19783b4740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f196c1c02e0 con 0x5575a2186390 2024-04-25T11:37:25.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.132+0000 7f19783b4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f196c1bee40 con 0x5575a215d4d0 2024-04-25T11:37:25.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.132+0000 7f196a7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5575a21656b0 0x5575a253dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.132+0000 7f196a7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5575a21656b0 0x5575a253dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55602/0 (socket says 172.21.15.90:55602) 2024-04-25T11:37:25.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.132+0000 7f196a7fc640 1 -- 172.21.15.90:0/4065169793 learned_addr learned my addr 172.21.15.90:0/4065169793 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:25.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.133+0000 7f196a7fc640 1 -- 172.21.15.90:0/4065169793 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5575a215d4d0 msgr2=0x5575a251f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:25.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.133+0000 7f196a7fc640 1 --2- 172.21.15.90:0/4065169793 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5575a215d4d0 0x5575a251f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.133+0000 7f196a7fc640 1 -- 172.21.15.90:0/4065169793 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5575a2186390 msgr2=0x5575a254a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:25.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.133+0000 7f196a7fc640 1 --2- 172.21.15.90:0/4065169793 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5575a2186390 0x5575a254a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.133+0000 7f196a7fc640 1 -- 172.21.15.90:0/4065169793 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575a2530550 con 0x5575a21656b0 2024-04-25T11:37:25.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.134+0000 7f196a7fc640 1 --2- 172.21.15.90:0/4065169793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5575a21656b0 0x5575a253dec0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f195c009450 tx=0x7f195c009480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27fe2c34a01850ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:25.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.134+0000 7f1950ff9640 1 -- 172.21.15.90:0/4065169793 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195c015070 con 0x5575a21656b0 2024-04-25T11:37:25.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.134+0000 7f1950ff9640 1 -- 172.21.15.90:0/4065169793 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f195c004040 con 0x5575a21656b0 2024-04-25T11:37:25.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.135+0000 7f1950ff9640 1 -- 172.21.15.90:0/4065169793 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195c003bf0 con 0x5575a21656b0 2024-04-25T11:37:25.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.136+0000 7f19783b4740 1 -- 172.21.15.90:0/4065169793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5575a21656b0 msgr2=0x5575a253dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:25.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.136+0000 7f19783b4740 1 --2- 172.21.15.90:0/4065169793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5575a21656b0 0x5575a253dec0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f195c009450 tx=0x7f195c009480 comp rx=0 tx=0).stop 2024-04-25T11:37:25.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.136+0000 7f19783b4740 1 -- 172.21.15.90:0/4065169793 shutdown_connections 2024-04-25T11:37:25.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.136+0000 7f19783b4740 1 --2- 172.21.15.90:0/4065169793 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5575a2186390 0x5575a254a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.136+0000 7f19783b4740 1 --2- 172.21.15.90:0/4065169793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5575a21656b0 0x5575a253dec0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.136+0000 7f19783b4740 1 --2- 172.21.15.90:0/4065169793 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5575a215d4d0 0x5575a251f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.136+0000 7f19783b4740 1 -- 172.21.15.90:0/4065169793 >> 172.21.15.90:0/4065169793 conn(0x5575a217a4e0 msgr2=0x5575a25308d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:25.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.136+0000 7f19783b4740 1 -- 172.21.15.90:0/4065169793 shutdown_connections 2024-04-25T11:37:25.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.136+0000 7f19783b4740 1 -- 172.21.15.90:0/4065169793 wait complete. 2024-04-25T11:37:25.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.137+0000 7f19783b4740 1 -- start start 2024-04-25T11:37:25.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.137+0000 7f19783b4740 1 Processor -- start 2024-04-25T11:37:25.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.138+0000 7f19783b4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5575a215d4d0 0x5575a253a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.138+0000 7f19783b4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5575a21656b0 0x5575a25359f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.138+0000 7f19783b4740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5575a2186390 0x5575a2535f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.138+0000 7f19783b4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f196c1bfbe0 con 0x5575a21656b0 2024-04-25T11:37:25.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.138+0000 7f19783b4740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f196c1bf510 con 0x5575a2186390 2024-04-25T11:37:25.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.138+0000 7f19783b4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f196c1bc750 con 0x5575a215d4d0 2024-04-25T11:37:25.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.138+0000 7f19783b4740 10 client.0 _ll_register_callbacks cb 0x7f1960005e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:37:25.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.139+0000 7f196b7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5575a2186390 0x5575a2535f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.139+0000 7f196b7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5575a2186390 0x5575a2535f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:46142/0 (socket says 172.21.15.90:46142) 2024-04-25T11:37:25.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.139+0000 7f196b7fe640 1 -- 172.21.15.90:0/4188592908 learned_addr learned my addr 172.21.15.90:0/4188592908 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:25.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.139+0000 7f196b7fe640 1 -- 172.21.15.90:0/4188592908 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5575a215d4d0 msgr2=0x5575a253a6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:37:25.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.139+0000 7f196b7fe640 1 --2- 172.21.15.90:0/4188592908 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5575a215d4d0 0x5575a253a6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.139+0000 7f196b7fe640 1 -- 172.21.15.90:0/4188592908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5575a21656b0 msgr2=0x5575a25359f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:25.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.139+0000 7f196b7fe640 1 --2- 172.21.15.90:0/4188592908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5575a21656b0 0x5575a25359f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.139+0000 7f196b7fe640 1 -- 172.21.15.90:0/4188592908 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575a2536540 con 0x5575a2186390 2024-04-25T11:37:25.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.140+0000 7f196b7fe640 1 --2- 172.21.15.90:0/4188592908 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5575a2186390 0x5575a2535f30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1960005f70 tx=0x7f1960049d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=636a1dc789eb6063 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:25.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.140+0000 7f194eff5640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:37:25.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.140+0000 7f194eff5640 1 -- 172.21.15.90:0/4188592908 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196005f070 con 0x5575a2186390 2024-04-25T11:37:25.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.140+0000 7f194eff5640 1 -- 172.21.15.90:0/4188592908 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f1960004e60 con 0x5575a2186390 2024-04-25T11:37:25.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.140+0000 7f19783b4740 1 -- 172.21.15.90:0/4188592908 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f195c004040 con 0x5575a2186390 2024-04-25T11:37:25.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.141+0000 7f194eff5640 1 -- 172.21.15.90:0/4188592908 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f196006b050 con 0x5575a2186390 2024-04-25T11:37:25.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.141+0000 7f19783b4740 10 client.25637 fetch_fsmap learned FSMap version 27 2024-04-25T11:37:25.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.141+0000 7f19783b4740 1 -- 172.21.15.90:0/4188592908 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5575a25ac9b0 con 0x5575a2186390 2024-04-25T11:37:25.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.141+0000 7f194eff5640 1 -- 172.21.15.90:0/4188592908 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f196004e070 con 0x5575a2186390 2024-04-25T11:37:25.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.141+0000 7f194eff5640 1 -- 172.21.15.90:0/4188592908 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f196003dbf0 con 0x5575a2186390 2024-04-25T11:37:25.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.141+0000 7f19783b4740 10 client.25637 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:37:25.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.141+0000 7f19783b4740 10 client.25637 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:37:25.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.141+0000 7f19783b4740 20 client.25637 populate_metadata read hostname 'smithi090' 2024-04-25T11:37:25.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.141+0000 7f19783b4740 1 --2- 172.21.15.90:0/4188592908 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5575a211a3a0 0x5575a25af170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.142+0000 7f19783b4740 4 client.25637 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:37:25.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.142+0000 7f19783b4740 1 -- 172.21.15.90:0/4188592908 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f195c004040 con 0x5575a211a3a0 2024-04-25T11:37:25.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.142+0000 7f196affd640 1 --2- 172.21.15.90:0/4188592908 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5575a211a3a0 0x5575a25af170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.143+0000 7f194eff5640 10 client.25637 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:37:25.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.143+0000 7f196affd640 1 --2- 172.21.15.90:0/4188592908 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5575a211a3a0 0x5575a25af170 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d1542ccd15454177 server_cookie=85f9d5dc92a9fc35 in_seq=0 out_seq=0 2024-04-25T11:37:25.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.143+0000 7f194eff5640 1 -- 172.21.15.90:0/4188592908 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f195c004040 con 0x5575a211a3a0 2024-04-25T11:37:25.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.143+0000 7f194eff5640 10 client.25637 handle_command_reply: tid=0 2024-04-25T11:37:25.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.143+0000 7f19783b4740 1 client.25637 shutdown 2024-04-25T11:37:25.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.143+0000 7f19783b4740 1 -- 172.21.15.90:0/4188592908 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5575a2186390 msgr2=0x5575a2535f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:25.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.143+0000 7f19783b4740 1 --2- 172.21.15.90:0/4188592908 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5575a2186390 0x5575a2535f30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1960005f70 tx=0x7f1960049d40 comp rx=0 tx=0).stop 2024-04-25T11:37:25.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.144+0000 7f19783b4740 1 -- 172.21.15.90:0/4188592908 shutdown_connections 2024-04-25T11:37:25.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.144+0000 7f19783b4740 1 --2- 172.21.15.90:0/4188592908 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5575a211a3a0 0x5575a25af170 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.144+0000 7f19783b4740 1 --2- 172.21.15.90:0/4188592908 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5575a2186390 0x5575a2535f30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.144+0000 7f194eff5640 0 client.25637 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:37:25.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.144+0000 7f19783b4740 1 --2- 172.21.15.90:0/4188592908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5575a21656b0 0x5575a25359f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.144+0000 7f19783b4740 1 --2- 172.21.15.90:0/4188592908 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5575a215d4d0 0x5575a253a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.144+0000 7f19783b4740 1 -- 172.21.15.90:0/4188592908 >> 172.21.15.90:0/4188592908 conn(0x5575a217a4e0 msgr2=0x5575a25275a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:25.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.144+0000 7f19783b4740 1 -- 172.21.15.90:0/4188592908 shutdown_connections 2024-04-25T11:37:25.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.144+0000 7f19783b4740 1 -- 172.21.15.90:0/4188592908 wait complete. 2024-04-25T11:37:25.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.144+0000 7f19783b4740 20 client.25637 trim_cache size 0 max 16384 2024-04-25T11:37:25.134 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:37:25.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.145+0000 7f1971430640 1 -- 172.21.15.90:0/1675952432 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f19540afce0 msgr2=0x7f19540b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:25.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.145+0000 7f1971430640 1 --2- 172.21.15.90:0/1675952432 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f19540afce0 0x7f19540b21f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f19600092a0 tx=0x7f1960005fd0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.145+0000 7f1971430640 1 -- 172.21.15.90:0/1675952432 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f196c10a0b0 msgr2=0x7f196c1a9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:25.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.145+0000 7f1971430640 1 --2- 172.21.15.90:0/1675952432 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f196c10a0b0 0x7f196c1a9380 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f196400ba50 tx=0x7f196400bf10 comp rx=0 tx=0).stop 2024-04-25T11:37:25.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.145+0000 7f1971430640 1 -- 172.21.15.90:0/1675952432 shutdown_connections 2024-04-25T11:37:25.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.145+0000 7f1971430640 1 --2- 172.21.15.90:0/1675952432 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f19540afce0 0x7f19540b21f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.145+0000 7f1971430640 1 --2- 172.21.15.90:0/1675952432 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f196c11d120 0x7f196c1adba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.145+0000 7f1971430640 1 --2- 172.21.15.90:0/1675952432 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f196c11a7c0 0x7f196c1a98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.145+0000 7f1971430640 1 --2- 172.21.15.90:0/1675952432 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f196c10a0b0 0x7f196c1a9380 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.145+0000 7f1971430640 1 -- 172.21.15.90:0/1675952432 >> 172.21.15.90:0/1675952432 conn(0x7f196c06e7a0 msgr2=0x7f196c11bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:25.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.145+0000 7f1971430640 1 -- 172.21.15.90:0/1675952432 shutdown_connections 2024-04-25T11:37:25.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.146+0000 7f1971430640 1 -- 172.21.15.90:0/1675952432 wait complete. 2024-04-25T11:37:25.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T11:37:25.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.364+0000 7fd2ee44f640 1 -- 172.21.15.90:0/3270243619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2e811a7c0 msgr2=0x7fd2e811cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:25.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.364+0000 7fd2ee44f640 1 --2- 172.21.15.90:0/3270243619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2e811a7c0 0x7fd2e811cbe0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8009900 tx=0x7fd2d802a410 comp rx=0 tx=0).stop 2024-04-25T11:37:25.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.365+0000 7fd2ee44f640 1 -- 172.21.15.90:0/3270243619 shutdown_connections 2024-04-25T11:37:25.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.365+0000 7fd2ee44f640 1 --2- 172.21.15.90:0/3270243619 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd2e811d120 0x7fd2e811f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.365+0000 7fd2ee44f640 1 --2- 172.21.15.90:0/3270243619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2e811a7c0 0x7fd2e811cbe0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.365+0000 7fd2ee44f640 1 --2- 172.21.15.90:0/3270243619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd2e8109fc0 0x7fd2e810a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.365+0000 7fd2ee44f640 1 -- 172.21.15.90:0/3270243619 >> 172.21.15.90:0/3270243619 conn(0x7fd2e806f650 msgr2=0x7fd2e806fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:25.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.365+0000 7fd2ee44f640 1 -- 172.21.15.90:0/3270243619 shutdown_connections 2024-04-25T11:37:25.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.365+0000 7fd2ee44f640 1 -- 172.21.15.90:0/3270243619 wait complete. 2024-04-25T11:37:25.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.367+0000 7fd2ee44f640 1 Processor -- start 2024-04-25T11:37:25.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.368+0000 7fd2ee44f640 1 -- start start 2024-04-25T11:37:25.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.369+0000 7fd2ee44f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2e8109fc0 0x7fd2e81a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.370+0000 7fd2ee44f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd2e811a7c0 0x7fd2e81a9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.371+0000 7fd2ee44f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd2e811d120 0x7fd2e81adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.371+0000 7fd2ee44f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e8122aa0 con 0x7fd2e8109fc0 2024-04-25T11:37:25.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.371+0000 7fd2ee44f640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e8122920 con 0x7fd2e811d120 2024-04-25T11:37:25.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.371+0000 7fd2ecc4c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd2e811a7c0 0x7fd2e81a9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.371+0000 7fd2ecc4c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd2e811a7c0 0x7fd2e81a9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:52224/0 (socket says 172.21.15.90:52224) 2024-04-25T11:37:25.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.371+0000 7fd2ecc4c640 1 -- 172.21.15.90:0/694673646 learned_addr learned my addr 172.21.15.90:0/694673646 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:25.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.371+0000 7fd2edc4e640 1 --2- 172.21.15.90:0/694673646 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd2e811d120 0x7fd2e81adcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.371+0000 7fd2ee44f640 1 -- 172.21.15.90:0/694673646 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e8122c20 con 0x7fd2e811a7c0 2024-04-25T11:37:25.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.370+0000 7fd2ed44d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2e8109fc0 0x7fd2e81a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.377+0000 7fd2ed44d640 1 -- 172.21.15.90:0/694673646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd2e811a7c0 msgr2=0x7fd2e81a9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:25.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.377+0000 7fd2ed44d640 1 --2- 172.21.15.90:0/694673646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd2e811a7c0 0x7fd2e81a9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.377+0000 7fd2ed44d640 1 -- 172.21.15.90:0/694673646 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd2e811d120 msgr2=0x7fd2e81adcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:25.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.377+0000 7fd2ed44d640 1 --2- 172.21.15.90:0/694673646 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd2e811d120 0x7fd2e81adcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.378+0000 7fd2ed44d640 1 -- 172.21.15.90:0/694673646 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2e81ae300 con 0x7fd2e8109fc0 2024-04-25T11:37:25.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.378+0000 7fd2ed44d640 1 --2- 172.21.15.90:0/694673646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2e8109fc0 0x7fd2e81a94a0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4009fd0 tx=0x7fd2e400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:25.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.378+0000 7fd2ecc4c640 1 --2- 172.21.15.90:0/694673646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd2e811a7c0 0x7fd2e81a9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:37:25.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.378+0000 7fd2edc4e640 1 --2- 172.21.15.90:0/694673646 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd2e811d120 0x7fd2e81adcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:37:25.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.380+0000 7fd2de7fc640 1 -- 172.21.15.90:0/694673646 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4015020 con 0x7fd2e8109fc0 2024-04-25T11:37:25.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.381+0000 7fd2de7fc640 1 -- 172.21.15.90:0/694673646 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2e4011ce0 con 0x7fd2e8109fc0 2024-04-25T11:37:25.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.381+0000 7fd2ee44f640 1 -- 172.21.15.90:0/694673646 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2e81ae5f0 con 0x7fd2e8109fc0 2024-04-25T11:37:25.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.381+0000 7fd2ee44f640 1 -- 172.21.15.90:0/694673646 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2e81b5f70 con 0x7fd2e8109fc0 2024-04-25T11:37:25.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.381+0000 7fd2de7fc640 1 -- 172.21.15.90:0/694673646 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4010e50 con 0x7fd2e8109fc0 2024-04-25T11:37:25.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.386+0000 7fd2de7fc640 1 -- 172.21.15.90:0/694673646 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fd2e4018020 con 0x7fd2e8109fc0 2024-04-25T11:37:25.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.387+0000 7fd2f61fd740 1 Processor -- start 2024-04-25T11:37:25.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.387+0000 7fd2f61fd740 1 -- start start 2024-04-25T11:37:25.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.388+0000 7fd2f61fd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144dc5390 0x55b145166a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.388+0000 7fd2f61fd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b144d9c4d0 0x55b145166f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.388+0000 7fd2f61fd740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b144da46b0 0x55b145173890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.388+0000 7fd2f61fd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b1450cbd80 con 0x55b144dc5390 2024-04-25T11:37:25.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.388+0000 7fd2f61fd740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55b145085a20 con 0x55b144da46b0 2024-04-25T11:37:25.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.388+0000 7fd2f61fd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b1450c7d40 con 0x55b144d9c4d0 2024-04-25T11:37:25.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.388+0000 7fd2edc4e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144dc5390 0x55b145166a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.388+0000 7fd2edc4e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144dc5390 0x55b145166a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55644/0 (socket says 172.21.15.90:55644) 2024-04-25T11:37:25.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.388+0000 7fd2edc4e640 1 -- 172.21.15.90:0/637092128 learned_addr learned my addr 172.21.15.90:0/637092128 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:25.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.389+0000 7fd2edc4e640 1 -- 172.21.15.90:0/637092128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b144d9c4d0 msgr2=0x55b145166f40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:37:25.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.389+0000 7fd2edc4e640 1 --2- 172.21.15.90:0/637092128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b144d9c4d0 0x55b145166f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.389+0000 7fd2edc4e640 1 -- 172.21.15.90:0/637092128 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b144da46b0 msgr2=0x55b145173890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:25.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.389+0000 7fd2edc4e640 1 --2- 172.21.15.90:0/637092128 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b144da46b0 0x55b145173890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.389+0000 7fd2edc4e640 1 -- 172.21.15.90:0/637092128 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b145090c30 con 0x55b144dc5390 2024-04-25T11:37:25.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.389+0000 7fd2de7fc640 1 --2- 172.21.15.90:0/694673646 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd2d40aff90 0x7fd2d40b24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.390+0000 7fd2ed44d640 1 --2- 172.21.15.90:0/694673646 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd2d40aff90 0x7fd2d40b24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.390+0000 7fd2de7fc640 1 -- 172.21.15.90:0/694673646 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fd2e40de600 con 0x7fd2e8109fc0 2024-04-25T11:37:25.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.390+0000 7fd2edc4e640 1 --2- 172.21.15.90:0/637092128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144dc5390 0x55b145166a00 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0007ed0 tx=0x7fd2e000c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23c55adbd48b3df0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:25.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.390+0000 7fd2ed44d640 1 --2- 172.21.15.90:0/694673646 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd2d40aff90 0x7fd2d40b24a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x55b145164140 tx=0x7fd2e40a6670 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:25.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.391+0000 7fd2d37fe640 1 -- 172.21.15.90:0/637092128 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0014070 con 0x55b144dc5390 2024-04-25T11:37:25.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.391+0000 7fd2d37fe640 1 -- 172.21.15.90:0/637092128 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2e000fd00 con 0x55b144dc5390 2024-04-25T11:37:25.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.391+0000 7fd2d37fe640 1 -- 172.21.15.90:0/637092128 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e000ec60 con 0x55b144dc5390 2024-04-25T11:37:25.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.393+0000 7fd2f61fd740 1 -- 172.21.15.90:0/637092128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144dc5390 msgr2=0x55b145166a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:25.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.393+0000 7fd2f61fd740 1 --2- 172.21.15.90:0/637092128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144dc5390 0x55b145166a00 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0007ed0 tx=0x7fd2e000c690 comp rx=0 tx=0).stop 2024-04-25T11:37:25.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.393+0000 7fd2f61fd740 1 -- 172.21.15.90:0/637092128 shutdown_connections 2024-04-25T11:37:25.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.393+0000 7fd2f61fd740 1 --2- 172.21.15.90:0/637092128 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b144da46b0 0x55b145173890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.393+0000 7fd2f61fd740 1 --2- 172.21.15.90:0/637092128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b144d9c4d0 0x55b145166f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.393+0000 7fd2f61fd740 1 --2- 172.21.15.90:0/637092128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144dc5390 0x55b145166a00 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.393+0000 7fd2f61fd740 1 -- 172.21.15.90:0/637092128 >> 172.21.15.90:0/637092128 conn(0x55b144db94e0 msgr2=0x55b1451631b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:25.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.393+0000 7fd2f61fd740 1 -- 172.21.15.90:0/637092128 shutdown_connections 2024-04-25T11:37:25.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.393+0000 7fd2f61fd740 1 -- 172.21.15.90:0/637092128 wait complete. 2024-04-25T11:37:25.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.394+0000 7fd2f61fd740 1 -- start start 2024-04-25T11:37:25.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.394+0000 7fd2f61fd740 1 Processor -- start 2024-04-25T11:37:25.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.396+0000 7fd2f61fd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144d9c4d0 0x55b1451713e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.396+0000 7fd2f61fd740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b144da46b0 0x55b145171920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.396+0000 7fd2f61fd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b144dc5390 0x55b14517d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.396+0000 7fd2f61fd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b145173f80 con 0x55b144d9c4d0 2024-04-25T11:37:25.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.396+0000 7fd2f61fd740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55b145052dc0 con 0x55b144da46b0 2024-04-25T11:37:25.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.396+0000 7fd2edc4e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144d9c4d0 0x55b1451713e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.396+0000 7fd2edc4e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144d9c4d0 0x55b1451713e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55650/0 (socket says 172.21.15.90:55650) 2024-04-25T11:37:25.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.396+0000 7fd2f61fd740 1 -- 172.21.15.90:0/1910732233 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b145174100 con 0x55b144dc5390 2024-04-25T11:37:25.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.396+0000 7fd2edc4e640 1 -- 172.21.15.90:0/1910732233 learned_addr learned my addr 172.21.15.90:0/1910732233 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:25.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.396+0000 7fd2f61fd740 10 client.0 _ll_register_callbacks cb 0x55b14506e600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:37:25.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.397+0000 7fd2ecc4c640 1 --2- 172.21.15.90:0/1910732233 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b144da46b0 0x55b145171920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.397+0000 7fd2edc4e640 1 -- 172.21.15.90:0/1910732233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b144dc5390 msgr2=0x55b14517d4c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:37:25.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.397+0000 7fd2edc4e640 1 --2- 172.21.15.90:0/1910732233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b144dc5390 0x55b14517d4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.397+0000 7fd2edc4e640 1 -- 172.21.15.90:0/1910732233 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b144da46b0 msgr2=0x55b145171920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:25.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.397+0000 7fd2edc4e640 1 --2- 172.21.15.90:0/1910732233 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b144da46b0 0x55b145171920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.397+0000 7fd2edc4e640 1 -- 172.21.15.90:0/1910732233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b144ddd3f0 con 0x55b144d9c4d0 2024-04-25T11:37:25.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.397+0000 7fd2edc4e640 1 --2- 172.21.15.90:0/1910732233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144d9c4d0 0x55b1451713e0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0004a80 tx=0x7fd2e000efb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88ed78811952fbe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:25.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.397+0000 7fd2d0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:37:25.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.398+0000 7fd2d0ff9640 1 -- 172.21.15.90:0/1910732233 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0014040 con 0x55b144d9c4d0 2024-04-25T11:37:25.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.398+0000 7fd2d0ff9640 1 -- 172.21.15.90:0/1910732233 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fd2e000f850 con 0x55b144d9c4d0 2024-04-25T11:37:25.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.398+0000 7fd2f61fd740 1 -- 172.21.15.90:0/1910732233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2e000fd00 con 0x55b144d9c4d0 2024-04-25T11:37:25.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.399+0000 7fd2d0ff9640 1 -- 172.21.15.90:0/1910732233 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd2e002f050 con 0x55b144d9c4d0 2024-04-25T11:37:25.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.399+0000 7fd2f61fd740 10 client.16275 fetch_fsmap learned FSMap version 27 2024-04-25T11:37:25.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.399+0000 7fd2f61fd740 1 -- 172.21.15.90:0/1910732233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b144ea77d0 con 0x55b144d9c4d0 2024-04-25T11:37:25.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.400+0000 7fd2d0ff9640 1 -- 172.21.15.90:0/1910732233 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2e000fd00 con 0x55b144d9c4d0 2024-04-25T11:37:25.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.400+0000 7fd2d0ff9640 1 -- 172.21.15.90:0/1910732233 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7fd2e0021420 con 0x55b144d9c4d0 2024-04-25T11:37:25.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.400+0000 7fd2f61fd740 10 client.16275 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:37:25.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.400+0000 7fd2f61fd740 10 client.16275 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:37:25.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.400+0000 7fd2f61fd740 20 client.16275 populate_metadata read hostname 'smithi090' 2024-04-25T11:37:25.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.400+0000 7fd2f61fd740 1 --2- 172.21.15.90:0/1910732233 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55b144d83f60 0x55b145183730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.401+0000 7fd2f61fd740 4 client.16275 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:37:25.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.401+0000 7fd2f61fd740 1 -- 172.21.15.90:0/1910732233 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b1450d9990 con 0x55b144d83f60 2024-04-25T11:37:25.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.402+0000 7fd2ecc4c640 1 --2- 172.21.15.90:0/1910732233 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55b144d83f60 0x55b145183730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.403+0000 7fd2ecc4c640 1 --2- 172.21.15.90:0/1910732233 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55b144d83f60 0x55b145183730 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ca9dcc865149110e server_cookie=11420103439968de in_seq=0 out_seq=0 2024-04-25T11:37:25.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.404+0000 7fd2d0ff9640 10 client.16275 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:37:25.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.405+0000 7fd2d0ff9640 1 -- 172.21.15.90:0/1910732233 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b1450d9990 con 0x55b144d83f60 2024-04-25T11:37:25.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.405+0000 7fd2d0ff9640 10 client.16275 handle_command_reply: tid=0 2024-04-25T11:37:25.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.405+0000 7fd2f61fd740 1 client.16275 shutdown 2024-04-25T11:37:25.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.405+0000 7fd2f61fd740 1 -- 172.21.15.90:0/1910732233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144d9c4d0 msgr2=0x55b1451713e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:25.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.406+0000 7fd2f61fd740 1 --2- 172.21.15.90:0/1910732233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144d9c4d0 0x55b1451713e0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0004a80 tx=0x7fd2e000efb0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.407+0000 7fd2f61fd740 1 -- 172.21.15.90:0/1910732233 shutdown_connections 2024-04-25T11:37:25.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.407+0000 7fd2f61fd740 1 --2- 172.21.15.90:0/1910732233 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55b144d83f60 0x55b145183730 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.407+0000 7fd2f61fd740 1 --2- 172.21.15.90:0/1910732233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b144dc5390 0x55b14517d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.407+0000 7fd2d0ff9640 0 client.16275 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:37:25.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.407+0000 7fd2f61fd740 1 --2- 172.21.15.90:0/1910732233 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b144da46b0 0x55b145171920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.407+0000 7fd2f61fd740 1 --2- 172.21.15.90:0/1910732233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144d9c4d0 0x55b1451713e0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.407+0000 7fd2f61fd740 1 -- 172.21.15.90:0/1910732233 >> 172.21.15.90:0/1910732233 conn(0x55b144db94e0 msgr2=0x55b145168720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:25.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.407+0000 7fd2f61fd740 1 -- 172.21.15.90:0/1910732233 shutdown_connections 2024-04-25T11:37:25.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.407+0000 7fd2f61fd740 1 -- 172.21.15.90:0/1910732233 wait complete. 2024-04-25T11:37:25.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.408+0000 7fd2f61fd740 20 client.16275 trim_cache size 0 max 16384 2024-04-25T11:37:25.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.437+0000 7fd2f61fd740 1 Processor -- start 2024-04-25T11:37:25.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.437+0000 7fd2f61fd740 1 -- start start 2024-04-25T11:37:25.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.437+0000 7fd2f61fd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144d83f60 0x55b1451704d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.438+0000 7fd2f61fd740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b144d9c4d0 0x55b1451620d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.438+0000 7fd2f61fd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b144da46b0 0x55b14517d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.438+0000 7fd2f61fd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc00e940 con 0x55b144d83f60 2024-04-25T11:37:25.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.438+0000 7fd2f61fd740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc0104b0 con 0x55b144d9c4d0 2024-04-25T11:37:25.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.438+0000 7fd2f61fd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc00f010 con 0x55b144da46b0 2024-04-25T11:37:25.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.438+0000 7fd2edc4e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144d83f60 0x55b1451704d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.438+0000 7fd2edc4e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144d83f60 0x55b1451704d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55662/0 (socket says 172.21.15.90:55662) 2024-04-25T11:37:25.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.438+0000 7fd2edc4e640 1 -- 172.21.15.90:0/890969719 learned_addr learned my addr 172.21.15.90:0/890969719 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:25.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.438+0000 7fd2edc4e640 1 -- 172.21.15.90:0/890969719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b144da46b0 msgr2=0x55b14517d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:37:25.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.438+0000 7fd2edc4e640 1 --2- 172.21.15.90:0/890969719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b144da46b0 0x55b14517d0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.438+0000 7fd2edc4e640 1 -- 172.21.15.90:0/890969719 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b144d9c4d0 msgr2=0x55b1451620d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:25.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.438+0000 7fd2edc4e640 1 --2- 172.21.15.90:0/890969719 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b144d9c4d0 0x55b1451620d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.438+0000 7fd2edc4e640 1 -- 172.21.15.90:0/890969719 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b14516f720 con 0x55b144d83f60 2024-04-25T11:37:25.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.439+0000 7fd2edc4e640 1 --2- 172.21.15.90:0/890969719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144d83f60 0x55b1451704d0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fd2e000cc10 tx=0x7fd2e0023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1376ef217ceaaec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:25.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.439+0000 7fd2d2ffd640 1 -- 172.21.15.90:0/890969719 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0014040 con 0x55b144d83f60 2024-04-25T11:37:25.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.439+0000 7fd2d2ffd640 1 -- 172.21.15.90:0/890969719 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2e0026070 con 0x55b144d83f60 2024-04-25T11:37:25.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.439+0000 7fd2d2ffd640 1 -- 172.21.15.90:0/890969719 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e000f8b0 con 0x55b144d83f60 2024-04-25T11:37:25.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.440+0000 7fd2f61fd740 1 -- 172.21.15.90:0/890969719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144d83f60 msgr2=0x55b1451704d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:25.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.440+0000 7fd2f61fd740 1 --2- 172.21.15.90:0/890969719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144d83f60 0x55b1451704d0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fd2e000cc10 tx=0x7fd2e0023000 comp rx=0 tx=0).stop 2024-04-25T11:37:25.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.440+0000 7fd2f61fd740 1 -- 172.21.15.90:0/890969719 shutdown_connections 2024-04-25T11:37:25.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.440+0000 7fd2f61fd740 1 --2- 172.21.15.90:0/890969719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b144da46b0 0x55b14517d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.440+0000 7fd2f61fd740 1 --2- 172.21.15.90:0/890969719 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b144d9c4d0 0x55b1451620d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.440+0000 7fd2f61fd740 1 --2- 172.21.15.90:0/890969719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144d83f60 0x55b1451704d0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.440+0000 7fd2f61fd740 1 -- 172.21.15.90:0/890969719 >> 172.21.15.90:0/890969719 conn(0x55b144db94e0 msgr2=0x55b1451619c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:25.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.440+0000 7fd2f61fd740 1 -- 172.21.15.90:0/890969719 shutdown_connections 2024-04-25T11:37:25.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.440+0000 7fd2f61fd740 1 -- 172.21.15.90:0/890969719 wait complete. 2024-04-25T11:37:25.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.441+0000 7fd2f61fd740 1 -- start start 2024-04-25T11:37:25.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.441+0000 7fd2f61fd740 1 Processor -- start 2024-04-25T11:37:25.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.441+0000 7fd2f61fd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b144d83f60 0x55b145179730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.441+0000 7fd2f61fd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144d9c4d0 0x55b145174a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.441+0000 7fd2f61fd740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b144da46b0 0x55b145174fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.441+0000 7fd2f61fd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc00fdb0 con 0x55b144d9c4d0 2024-04-25T11:37:25.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.441+0000 7fd2f61fd740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc00f6e0 con 0x55b144da46b0 2024-04-25T11:37:25.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.441+0000 7fd2f61fd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc00c920 con 0x55b144d83f60 2024-04-25T11:37:25.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.441+0000 7fd2f61fd740 10 client.0 _ll_register_callbacks cb 0x7fd2e0007d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:37:25.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.441+0000 7fd2ecc4c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144d9c4d0 0x55b145174a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.441+0000 7fd2ecc4c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144d9c4d0 0x55b145174a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55664/0 (socket says 172.21.15.90:55664) 2024-04-25T11:37:25.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.441+0000 7fd2ecc4c640 1 -- 172.21.15.90:0/2699243904 learned_addr learned my addr 172.21.15.90:0/2699243904 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:25.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.442+0000 7fd2edc4e640 1 --2- 172.21.15.90:0/2699243904 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b144da46b0 0x55b145174fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.442+0000 7fd2ecc4c640 1 -- 172.21.15.90:0/2699243904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b144d83f60 msgr2=0x55b145179730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:25.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.442+0000 7fd2ecc4c640 1 --2- 172.21.15.90:0/2699243904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b144d83f60 0x55b145179730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.442+0000 7fd2ecc4c640 1 -- 172.21.15.90:0/2699243904 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b144da46b0 msgr2=0x55b145174fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:25.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.442+0000 7fd2ecc4c640 1 --2- 172.21.15.90:0/2699243904 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b144da46b0 0x55b145174fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.442+0000 7fd2ecc4c640 1 -- 172.21.15.90:0/2699243904 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b145175500 con 0x55b144d9c4d0 2024-04-25T11:37:25.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.442+0000 7fd2ecc4c640 1 --2- 172.21.15.90:0/2699243904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144d9c4d0 0x55b145174a80 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80092a0 tx=0x7fd2d8005120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7158906b15b9e23 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:25.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.442+0000 7fd2d0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:37:25.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.442+0000 7fd2d0ff9640 1 -- 172.21.15.90:0/2699243904 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d8035040 con 0x55b144d9c4d0 2024-04-25T11:37:25.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.442+0000 7fd2d0ff9640 1 -- 172.21.15.90:0/2699243904 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fd2d8005420 con 0x55b144d9c4d0 2024-04-25T11:37:25.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.443+0000 7fd2f61fd740 1 -- 172.21.15.90:0/2699243904 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2e0026070 con 0x55b144d9c4d0 2024-04-25T11:37:25.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.443+0000 7fd2d0ff9640 1 -- 172.21.15.90:0/2699243904 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd2d8051050 con 0x55b144d9c4d0 2024-04-25T11:37:25.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.443+0000 7fd2f61fd740 10 client.16281 fetch_fsmap learned FSMap version 27 2024-04-25T11:37:25.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.443+0000 7fd2f61fd740 1 -- 172.21.15.90:0/2699243904 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b1451eb5b0 con 0x55b144d9c4d0 2024-04-25T11:37:25.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.443+0000 7fd2d0ff9640 1 -- 172.21.15.90:0/2699243904 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2e0026070 con 0x55b144d9c4d0 2024-04-25T11:37:25.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.443+0000 7fd2d0ff9640 1 -- 172.21.15.90:0/2699243904 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7fd2d80414c0 con 0x55b144d9c4d0 2024-04-25T11:37:25.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.443+0000 7fd2f61fd740 10 client.16281 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:37:25.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.443+0000 7fd2f61fd740 10 client.16281 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:37:25.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.443+0000 7fd2f61fd740 20 client.16281 populate_metadata read hostname 'smithi090' 2024-04-25T11:37:25.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.443+0000 7fd2f61fd740 1 --2- 172.21.15.90:0/2699243904 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55b144dc5390 0x55b1451edd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.443+0000 7fd2f61fd740 4 client.16281 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:37:25.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.443+0000 7fd2f61fd740 1 -- 172.21.15.90:0/2699243904 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b1450d3800 con 0x55b144dc5390 2024-04-25T11:37:25.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.445+0000 7fd2edc4e640 1 --2- 172.21.15.90:0/2699243904 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55b144dc5390 0x55b1451edd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.447+0000 7fd2edc4e640 1 --2- 172.21.15.90:0/2699243904 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55b144dc5390 0x55b1451edd50 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=505dc9f439759590 server_cookie=4087d0d0271e14ea in_seq=0 out_seq=0 2024-04-25T11:37:25.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.447+0000 7fd2d0ff9640 10 client.16281 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:37:25.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.448+0000 7fd2d0ff9640 1 -- 172.21.15.90:0/2699243904 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b1450d3800 con 0x55b144dc5390 2024-04-25T11:37:25.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.449+0000 7fd2d0ff9640 10 client.16281 handle_command_reply: tid=0 2024-04-25T11:37:25.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.449+0000 7fd2f61fd740 1 client.16281 shutdown 2024-04-25T11:37:25.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.449+0000 7fd2f61fd740 1 -- 172.21.15.90:0/2699243904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144d9c4d0 msgr2=0x55b145174a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:25.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.449+0000 7fd2f61fd740 1 --2- 172.21.15.90:0/2699243904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144d9c4d0 0x55b145174a80 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80092a0 tx=0x7fd2d8005120 comp rx=0 tx=0).stop 2024-04-25T11:37:25.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.449+0000 7fd2f61fd740 1 -- 172.21.15.90:0/2699243904 shutdown_connections 2024-04-25T11:37:25.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.449+0000 7fd2f61fd740 1 --2- 172.21.15.90:0/2699243904 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55b144dc5390 0x55b1451edd50 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.449+0000 7fd2f61fd740 1 --2- 172.21.15.90:0/2699243904 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b144da46b0 0x55b145174fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.449+0000 7fd2f61fd740 1 --2- 172.21.15.90:0/2699243904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b144d9c4d0 0x55b145174a80 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.449+0000 7fd2d0ff9640 0 client.16281 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:37:25.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.449+0000 7fd2f61fd740 1 --2- 172.21.15.90:0/2699243904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b144d83f60 0x55b145179730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.449+0000 7fd2f61fd740 1 -- 172.21.15.90:0/2699243904 >> 172.21.15.90:0/2699243904 conn(0x55b144db94e0 msgr2=0x55b145170230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:25.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.450+0000 7fd2f61fd740 1 -- 172.21.15.90:0/2699243904 shutdown_connections 2024-04-25T11:37:25.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.450+0000 7fd2f61fd740 1 -- 172.21.15.90:0/2699243904 wait complete. 2024-04-25T11:37:25.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.451+0000 7fd2f61fd740 20 client.16281 trim_cache size 0 max 16384 2024-04-25T11:37:25.439 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:37:25.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.452+0000 7fd2ee44f640 1 -- 172.21.15.90:0/694673646 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd2d40aff90 msgr2=0x7fd2d40b24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:25.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.452+0000 7fd2ee44f640 1 --2- 172.21.15.90:0/694673646 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd2d40aff90 0x7fd2d40b24a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x55b145164140 tx=0x7fd2e40a6670 comp rx=0 tx=0).stop 2024-04-25T11:37:25.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.452+0000 7fd2ee44f640 1 -- 172.21.15.90:0/694673646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2e8109fc0 msgr2=0x7fd2e81a94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:25.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.452+0000 7fd2ee44f640 1 --2- 172.21.15.90:0/694673646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2e8109fc0 0x7fd2e81a94a0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4009fd0 tx=0x7fd2e400b870 comp rx=0 tx=0).stop 2024-04-25T11:37:25.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.452+0000 7fd2ee44f640 1 -- 172.21.15.90:0/694673646 shutdown_connections 2024-04-25T11:37:25.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.452+0000 7fd2ee44f640 1 --2- 172.21.15.90:0/694673646 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd2d40aff90 0x7fd2d40b24a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.452+0000 7fd2ee44f640 1 --2- 172.21.15.90:0/694673646 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd2e811d120 0x7fd2e81adcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.452+0000 7fd2ee44f640 1 --2- 172.21.15.90:0/694673646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd2e811a7c0 0x7fd2e81a9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.452+0000 7fd2ee44f640 1 --2- 172.21.15.90:0/694673646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2e8109fc0 0x7fd2e81a94a0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.452+0000 7fd2ee44f640 1 -- 172.21.15.90:0/694673646 >> 172.21.15.90:0/694673646 conn(0x7fd2e806f650 msgr2=0x7fd2e811abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:25.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.452+0000 7fd2ee44f640 1 -- 172.21.15.90:0/694673646 shutdown_connections 2024-04-25T11:37:25.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.452+0000 7fd2ee44f640 1 -- 172.21.15.90:0/694673646 wait complete. 2024-04-25T11:37:25.461 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T11:37:25.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.730+0000 7fb285b96640 1 -- 172.21.15.90:0/54849649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb280109b10 msgr2=0x7fb280103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:25.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.730+0000 7fb285b96640 1 --2- 172.21.15.90:0/54849649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb280109b10 0x7fb280103700 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb270009900 tx=0x7fb27002a440 comp rx=0 tx=0).stop 2024-04-25T11:37:25.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.730+0000 7fb285b96640 1 -- 172.21.15.90:0/54849649 shutdown_connections 2024-04-25T11:37:25.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.730+0000 7fb285b96640 1 --2- 172.21.15.90:0/54849649 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb2801042b0 0x7fb280110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.730+0000 7fb285b96640 1 --2- 172.21.15.90:0/54849649 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb280109fb0 0x7fb280103c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.730+0000 7fb285b96640 1 --2- 172.21.15.90:0/54849649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb280109b10 0x7fb280103700 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.730+0000 7fb285b96640 1 -- 172.21.15.90:0/54849649 >> 172.21.15.90:0/54849649 conn(0x7fb2800ff360 msgr2=0x7fb280101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:25.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.730+0000 7fb285b96640 1 -- 172.21.15.90:0/54849649 shutdown_connections 2024-04-25T11:37:25.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.730+0000 7fb285b96640 1 -- 172.21.15.90:0/54849649 wait complete. 2024-04-25T11:37:25.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.731+0000 7fb285b96640 1 Processor -- start 2024-04-25T11:37:25.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.731+0000 7fb285b96640 1 -- start start 2024-04-25T11:37:25.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.731+0000 7fb285b96640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2801042b0 0x7fb280076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.732+0000 7fb285b96640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb280109b10 0x7fb280074ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.732+0000 7fb285b96640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb280109fb0 0x7fb2800754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.732+0000 7fb285b96640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb280113e50 con 0x7fb280109b10 2024-04-25T11:37:25.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.732+0000 7fb285b96640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb280113cd0 con 0x7fb280109fb0 2024-04-25T11:37:25.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.732+0000 7fb285b96640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb280113fd0 con 0x7fb2801042b0 2024-04-25T11:37:25.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.732+0000 7fb27effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb280109b10 0x7fb280074ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.732+0000 7fb27effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb280109b10 0x7fb280074ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55688/0 (socket says 172.21.15.90:55688) 2024-04-25T11:37:25.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.732+0000 7fb27effd640 1 -- 172.21.15.90:0/377639325 learned_addr learned my addr 172.21.15.90:0/377639325 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:25.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.732+0000 7fb27f7fe640 1 --2- 172.21.15.90:0/377639325 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2801042b0 0x7fb280076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.732+0000 7fb27effd640 1 -- 172.21.15.90:0/377639325 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2801042b0 msgr2=0x7fb280076a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:25.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.732+0000 7fb27effd640 1 --2- 172.21.15.90:0/377639325 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2801042b0 0x7fb280076a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.732+0000 7fb27effd640 1 -- 172.21.15.90:0/377639325 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb280109fb0 msgr2=0x7fb2800754a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:25.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.732+0000 7fb27effd640 1 --2- 172.21.15.90:0/377639325 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb280109fb0 0x7fb2800754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.733+0000 7fb27effd640 1 -- 172.21.15.90:0/377639325 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb280075c90 con 0x7fb280109b10 2024-04-25T11:37:25.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.733+0000 7fb27effd640 1 --2- 172.21.15.90:0/377639325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb280109b10 0x7fb280074ea0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fb274006fd0 tx=0x7fb27400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:25.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.733+0000 7fb27cff9640 1 -- 172.21.15.90:0/377639325 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb274011830 con 0x7fb280109b10 2024-04-25T11:37:25.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.733+0000 7fb27cff9640 1 -- 172.21.15.90:0/377639325 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb274016070 con 0x7fb280109b10 2024-04-25T11:37:25.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.733+0000 7fb285b96640 1 -- 172.21.15.90:0/377639325 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2801ad390 con 0x7fb280109b10 2024-04-25T11:37:25.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.733+0000 7fb27cff9640 1 -- 172.21.15.90:0/377639325 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb274010e00 con 0x7fb280109b10 2024-04-25T11:37:25.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.733+0000 7fb285b96640 1 -- 172.21.15.90:0/377639325 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2801ad8b0 con 0x7fb280109b10 2024-04-25T11:37:25.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.737+0000 7fb27cff9640 1 -- 172.21.15.90:0/377639325 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fb27401b4a0 con 0x7fb280109b10 2024-04-25T11:37:25.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.738+0000 7fb28ca4a740 1 Processor -- start 2024-04-25T11:37:25.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.738+0000 7fb28ca4a740 1 -- start start 2024-04-25T11:37:25.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.738+0000 7fb28ca4a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bda99af390 0x55bda9d50810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.738+0000 7fb28ca4a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bda99864d0 0x55bda9d50d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.738+0000 7fb28ca4a740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bda998e6b0 0x55bda9d5d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.738+0000 7fb28ca4a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55bda9cb5d80 con 0x55bda99af390 2024-04-25T11:37:25.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.738+0000 7fb28ca4a740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55bda9c6fa20 con 0x55bda998e6b0 2024-04-25T11:37:25.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.738+0000 7fb28ca4a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55bda9cb1d40 con 0x55bda99864d0 2024-04-25T11:37:25.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.738+0000 7fb27cff9640 1 --2- 172.21.15.90:0/377639325 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb2680afce0 0x7fb2680b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.738+0000 7fb27ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bda99af390 0x55bda9d50810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.738+0000 7fb27ffff640 1 --2- 172.21.15.90:0/377639325 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb2680afce0 0x7fb2680b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.738+0000 7fb27effd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bda998e6b0 0x55bda9d5d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.738+0000 7fb27ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bda99af390 0x55bda9d50810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55700/0 (socket says 172.21.15.90:55700) 2024-04-25T11:37:25.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.738+0000 7fb27ffff640 1 -- 172.21.15.90:0/2972097074 learned_addr learned my addr 172.21.15.90:0/2972097074 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:25.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.738+0000 7fb27f7fe640 1 --2- 172.21.15.90:0/2972097074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bda99864d0 0x55bda9d50d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.739+0000 7fb27cff9640 1 -- 172.21.15.90:0/377639325 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb2740de640 con 0x7fb280109b10 2024-04-25T11:37:25.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.739+0000 7fb27f7fe640 1 -- 172.21.15.90:0/2972097074 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bda998e6b0 msgr2=0x55bda9d5d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:25.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.739+0000 7fb27f7fe640 1 --2- 172.21.15.90:0/2972097074 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bda998e6b0 0x55bda9d5d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.739+0000 7fb27f7fe640 1 -- 172.21.15.90:0/2972097074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bda99af390 msgr2=0x55bda9d50810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:25.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.739+0000 7fb27f7fe640 1 --2- 172.21.15.90:0/2972097074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bda99af390 0x55bda9d50810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.739+0000 7fb27f7fe640 1 -- 172.21.15.90:0/2972097074 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bda9c7ac30 con 0x55bda99864d0 2024-04-25T11:37:25.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.739+0000 7fb27ffff640 1 --2- 172.21.15.90:0/2972097074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bda99af390 0x55bda9d50810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:37:25.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.739+0000 7fb27ffff640 1 --2- 172.21.15.90:0/377639325 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb2680afce0 0x7fb2680b21f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb27800afd0 tx=0x7fb27800a750 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:25.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.739+0000 7fb27f7fe640 1 --2- 172.21.15.90:0/2972097074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bda99864d0 0x55bda9d50d50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb270009230 tx=0x7fb270004200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ae1456c8ba5faae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:25.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.739+0000 7fb265ffb640 1 -- 172.21.15.90:0/2972097074 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27003f070 con 0x55bda99864d0 2024-04-25T11:37:25.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.739+0000 7fb265ffb640 1 -- 172.21.15.90:0/2972097074 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb270003aa0 con 0x55bda99864d0 2024-04-25T11:37:25.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.740+0000 7fb265ffb640 1 -- 172.21.15.90:0/2972097074 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb270035070 con 0x55bda99864d0 2024-04-25T11:37:25.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.740+0000 7fb28ca4a740 1 -- 172.21.15.90:0/2972097074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bda99864d0 msgr2=0x55bda9d50d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:25.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.740+0000 7fb28ca4a740 1 --2- 172.21.15.90:0/2972097074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bda99864d0 0x55bda9d50d50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb270009230 tx=0x7fb270004200 comp rx=0 tx=0).stop 2024-04-25T11:37:25.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.741+0000 7fb28ca4a740 1 -- 172.21.15.90:0/2972097074 shutdown_connections 2024-04-25T11:37:25.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.741+0000 7fb28ca4a740 1 --2- 172.21.15.90:0/2972097074 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bda998e6b0 0x55bda9d5d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.741+0000 7fb28ca4a740 1 --2- 172.21.15.90:0/2972097074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bda99864d0 0x55bda9d50d50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.741+0000 7fb28ca4a740 1 --2- 172.21.15.90:0/2972097074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bda99af390 0x55bda9d50810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.741+0000 7fb28ca4a740 1 -- 172.21.15.90:0/2972097074 >> 172.21.15.90:0/2972097074 conn(0x55bda99a34e0 msgr2=0x55bda9d4cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:25.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.741+0000 7fb28ca4a740 1 -- 172.21.15.90:0/2972097074 shutdown_connections 2024-04-25T11:37:25.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.741+0000 7fb28ca4a740 1 -- 172.21.15.90:0/2972097074 wait complete. 2024-04-25T11:37:25.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.742+0000 7fb28ca4a740 1 -- start start 2024-04-25T11:37:25.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.742+0000 7fb28ca4a740 1 Processor -- start 2024-04-25T11:37:25.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.742+0000 7fb28ca4a740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bda99864d0 0x55bda9d5b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.742+0000 7fb28ca4a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bda998e6b0 0x55bda9d5b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.742+0000 7fb28ca4a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bda99af390 0x55bda9d671f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.742+0000 7fb28ca4a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55bda9d5dd30 con 0x55bda998e6b0 2024-04-25T11:37:25.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.742+0000 7fb28ca4a740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55bda9c3cdc0 con 0x55bda99864d0 2024-04-25T11:37:25.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.742+0000 7fb28ca4a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55bda9d5deb0 con 0x55bda99af390 2024-04-25T11:37:25.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.743+0000 7fb27ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bda99af390 0x55bda9d671f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.743+0000 7fb27ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bda99af390 0x55bda9d671f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:52300/0 (socket says 172.21.15.90:52300) 2024-04-25T11:37:25.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.743+0000 7fb27ffff640 1 -- 172.21.15.90:0/1896285987 learned_addr learned my addr 172.21.15.90:0/1896285987 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:25.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.743+0000 7fb27effd640 1 --2- 172.21.15.90:0/1896285987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bda998e6b0 0x55bda9d5b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.743+0000 7fb28ca4a740 10 client.0 _ll_register_callbacks cb 0x55bda9c58600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:37:25.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.743+0000 7fb27f7fe640 1 --2- 172.21.15.90:0/1896285987 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bda99864d0 0x55bda9d5b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.743+0000 7fb27ffff640 1 -- 172.21.15.90:0/1896285987 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bda99864d0 msgr2=0x55bda9d5b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:25.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.743+0000 7fb27ffff640 1 --2- 172.21.15.90:0/1896285987 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bda99864d0 0x55bda9d5b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.743+0000 7fb27ffff640 1 -- 172.21.15.90:0/1896285987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bda998e6b0 msgr2=0x55bda9d5b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:25.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.743+0000 7fb27ffff640 1 --2- 172.21.15.90:0/1896285987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bda998e6b0 0x55bda9d5b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.743+0000 7fb27ffff640 1 -- 172.21.15.90:0/1896285987 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bda99c73f0 con 0x55bda99af390 2024-04-25T11:37:25.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.743+0000 7fb27f7fe640 1 --2- 172.21.15.90:0/1896285987 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bda99864d0 0x55bda9d5b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:37:25.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.744+0000 7fb27ffff640 1 --2- 172.21.15.90:0/1896285987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bda99af390 0x55bda9d671f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb27800a0c0 tx=0x7fb278008f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c22fd98e1e2be4b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:25.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.744+0000 7fb25effd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:37:25.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.744+0000 7fb25effd640 1 -- 172.21.15.90:0/1896285987 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb278025060 con 0x55bda99af390 2024-04-25T11:37:25.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.744+0000 7fb28ca4a740 1 -- 172.21.15.90:0/1896285987 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb270003aa0 con 0x55bda99af390 2024-04-25T11:37:25.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.744+0000 7fb25effd640 1 -- 172.21.15.90:0/1896285987 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb278016450 con 0x55bda99af390 2024-04-25T11:37:25.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.744+0000 7fb25effd640 1 -- 172.21.15.90:0/1896285987 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb278016650 con 0x55bda99af390 2024-04-25T11:37:25.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.744+0000 7fb28ca4a740 10 client.25597 fetch_fsmap learned FSMap version 27 2024-04-25T11:37:25.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.744+0000 7fb28ca4a740 1 -- 172.21.15.90:0/1896285987 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bda9a917d0 con 0x55bda99af390 2024-04-25T11:37:25.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.745+0000 7fb25effd640 1 -- 172.21.15.90:0/1896285987 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb270003aa0 con 0x55bda99af390 2024-04-25T11:37:25.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.745+0000 7fb25effd640 1 -- 172.21.15.90:0/1896285987 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7fb27801d070 con 0x55bda99af390 2024-04-25T11:37:25.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.745+0000 7fb28ca4a740 10 client.25597 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:37:25.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.745+0000 7fb28ca4a740 10 client.25597 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:37:25.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.745+0000 7fb28ca4a740 20 client.25597 populate_metadata read hostname 'smithi090' 2024-04-25T11:37:25.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.745+0000 7fb28ca4a740 1 --2- 172.21.15.90:0/1896285987 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55bda996df60 0x55bda9d6d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.746+0000 7fb28ca4a740 4 client.25597 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:37:25.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.746+0000 7fb28ca4a740 1 -- 172.21.15.90:0/1896285987 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bda9cc3990 con 0x55bda996df60 2024-04-25T11:37:25.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.746+0000 7fb27f7fe640 1 --2- 172.21.15.90:0/1896285987 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55bda996df60 0x55bda9d6d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.747+0000 7fb25effd640 10 client.25597 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:37:25.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.747+0000 7fb27f7fe640 1 --2- 172.21.15.90:0/1896285987 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55bda996df60 0x55bda9d6d780 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7a42f5841aff8b75 server_cookie=4dd930142463783b in_seq=0 out_seq=0 2024-04-25T11:37:25.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.747+0000 7fb25effd640 1 -- 172.21.15.90:0/1896285987 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55bda9cc3990 con 0x55bda996df60 2024-04-25T11:37:25.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.747+0000 7fb25effd640 10 client.25597 handle_command_reply: tid=0 2024-04-25T11:37:25.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.748+0000 7fb28ca4a740 1 client.25597 shutdown 2024-04-25T11:37:25.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.748+0000 7fb28ca4a740 1 -- 172.21.15.90:0/1896285987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bda99af390 msgr2=0x55bda9d671f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:25.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.748+0000 7fb28ca4a740 1 --2- 172.21.15.90:0/1896285987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bda99af390 0x55bda9d671f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb27800a0c0 tx=0x7fb278008f90 comp rx=0 tx=0).stop 2024-04-25T11:37:25.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.748+0000 7fb28ca4a740 1 -- 172.21.15.90:0/1896285987 shutdown_connections 2024-04-25T11:37:25.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.748+0000 7fb28ca4a740 1 --2- 172.21.15.90:0/1896285987 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55bda996df60 0x55bda9d6d780 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.748+0000 7fb28ca4a740 1 --2- 172.21.15.90:0/1896285987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bda99af390 0x55bda9d671f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.748+0000 7fb25effd640 0 client.25597 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:37:25.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.748+0000 7fb28ca4a740 1 --2- 172.21.15.90:0/1896285987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bda998e6b0 0x55bda9d5b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.748+0000 7fb28ca4a740 1 --2- 172.21.15.90:0/1896285987 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bda99864d0 0x55bda9d5b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.748+0000 7fb28ca4a740 1 -- 172.21.15.90:0/1896285987 >> 172.21.15.90:0/1896285987 conn(0x55bda99a34e0 msgr2=0x55bda9d52530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:25.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.748+0000 7fb28ca4a740 1 -- 172.21.15.90:0/1896285987 shutdown_connections 2024-04-25T11:37:25.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.748+0000 7fb28ca4a740 1 -- 172.21.15.90:0/1896285987 wait complete. 2024-04-25T11:37:25.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.748+0000 7fb28ca4a740 20 client.25597 trim_cache size 0 max 16384 2024-04-25T11:37:25.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.779+0000 7fb28ca4a740 1 Processor -- start 2024-04-25T11:37:25.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.779+0000 7fb28ca4a740 1 -- start start 2024-04-25T11:37:25.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.779+0000 7fb28ca4a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bda99864d0 0x55bda9d5a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.779+0000 7fb28ca4a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bda998e6b0 0x55bda9d66e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.779+0000 7fb28ca4a740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bda99af390 0x55bda9d55430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.779+0000 7fb28ca4a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb2801b6240 con 0x55bda998e6b0 2024-04-25T11:37:25.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.779+0000 7fb28ca4a740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb2801b7db0 con 0x55bda99af390 2024-04-25T11:37:25.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.779+0000 7fb28ca4a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb2801b6910 con 0x55bda99864d0 2024-04-25T11:37:25.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.779+0000 7fb27effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bda998e6b0 0x55bda9d66e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.779+0000 7fb27effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bda998e6b0 0x55bda9d66e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55726/0 (socket says 172.21.15.90:55726) 2024-04-25T11:37:25.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.779+0000 7fb27effd640 1 -- 172.21.15.90:0/3441529548 learned_addr learned my addr 172.21.15.90:0/3441529548 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:25.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.779+0000 7fb27ffff640 1 --2- 172.21.15.90:0/3441529548 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bda99af390 0x55bda9d55430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.780+0000 7fb27effd640 1 -- 172.21.15.90:0/3441529548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bda99864d0 msgr2=0x55bda9d5a030 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:37:25.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.780+0000 7fb27effd640 1 --2- 172.21.15.90:0/3441529548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bda99864d0 0x55bda9d5a030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.780+0000 7fb27effd640 1 -- 172.21.15.90:0/3441529548 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bda99af390 msgr2=0x55bda9d55430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:25.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.780+0000 7fb27effd640 1 --2- 172.21.15.90:0/3441529548 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bda99af390 0x55bda9d55430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.780+0000 7fb27effd640 1 -- 172.21.15.90:0/3441529548 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bda9d595a0 con 0x55bda998e6b0 2024-04-25T11:37:25.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.780+0000 7fb27effd640 1 --2- 172.21.15.90:0/3441529548 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bda998e6b0 0x55bda9d66e10 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fb27401f000 tx=0x7fb274075390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85f9aa8a7c728081 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:25.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.787+0000 7fb2657fa640 1 -- 172.21.15.90:0/3441529548 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2740a6000 con 0x55bda998e6b0 2024-04-25T11:37:25.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.787+0000 7fb2657fa640 1 -- 172.21.15.90:0/3441529548 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2740751d0 con 0x55bda998e6b0 2024-04-25T11:37:25.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.787+0000 7fb2657fa640 1 -- 172.21.15.90:0/3441529548 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb274021250 con 0x55bda998e6b0 2024-04-25T11:37:25.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.789+0000 7fb28ca4a740 1 -- 172.21.15.90:0/3441529548 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bda998e6b0 msgr2=0x55bda9d66e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:25.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.789+0000 7fb28ca4a740 1 --2- 172.21.15.90:0/3441529548 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bda998e6b0 0x55bda9d66e10 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fb27401f000 tx=0x7fb274075390 comp rx=0 tx=0).stop 2024-04-25T11:37:25.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.789+0000 7fb28ca4a740 1 -- 172.21.15.90:0/3441529548 shutdown_connections 2024-04-25T11:37:25.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.789+0000 7fb28ca4a740 1 --2- 172.21.15.90:0/3441529548 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bda99af390 0x55bda9d55430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.789+0000 7fb28ca4a740 1 --2- 172.21.15.90:0/3441529548 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bda998e6b0 0x55bda9d66e10 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.789+0000 7fb28ca4a740 1 --2- 172.21.15.90:0/3441529548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bda99864d0 0x55bda9d5a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.789+0000 7fb28ca4a740 1 -- 172.21.15.90:0/3441529548 >> 172.21.15.90:0/3441529548 conn(0x55bda99a34e0 msgr2=0x55bda9d59920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:25.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.789+0000 7fb28ca4a740 1 -- 172.21.15.90:0/3441529548 shutdown_connections 2024-04-25T11:37:25.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.789+0000 7fb28ca4a740 1 -- 172.21.15.90:0/3441529548 wait complete. 2024-04-25T11:37:25.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.789+0000 7fb28ca4a740 1 -- start start 2024-04-25T11:37:25.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.790+0000 7fb28ca4a740 1 Processor -- start 2024-04-25T11:37:25.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.790+0000 7fb28ca4a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bda99864d0 0x55bda9d63480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.790+0000 7fb28ca4a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bda998e6b0 0x55bda9d5e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.790+0000 7fb28ca4a740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bda99af390 0x55bda9d5ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.790+0000 7fb28ca4a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb2801b76b0 con 0x55bda998e6b0 2024-04-25T11:37:25.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.790+0000 7fb28ca4a740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb2801b6fe0 con 0x55bda99af390 2024-04-25T11:37:25.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.790+0000 7fb28ca4a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb2801b4220 con 0x55bda99864d0 2024-04-25T11:37:25.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.791+0000 7fb28ca4a740 10 client.0 _ll_register_callbacks cb 0x7fb2780125d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:37:25.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.791+0000 7fb27effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bda998e6b0 0x55bda9d5e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.791+0000 7fb27effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bda998e6b0 0x55bda9d5e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55732/0 (socket says 172.21.15.90:55732) 2024-04-25T11:37:25.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.791+0000 7fb27effd640 1 -- 172.21.15.90:0/3636658559 learned_addr learned my addr 172.21.15.90:0/3636658559 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:25.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.791+0000 7fb27f7fe640 1 --2- 172.21.15.90:0/3636658559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bda99864d0 0x55bda9d63480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.791+0000 7fb27effd640 1 -- 172.21.15.90:0/3636658559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bda99864d0 msgr2=0x55bda9d63480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:25.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.791+0000 7fb27effd640 1 --2- 172.21.15.90:0/3636658559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bda99864d0 0x55bda9d63480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.791+0000 7fb27effd640 1 -- 172.21.15.90:0/3636658559 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bda99af390 msgr2=0x55bda9d5ed10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:37:25.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.791+0000 7fb27effd640 1 --2- 172.21.15.90:0/3636658559 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bda99af390 0x55bda9d5ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.791+0000 7fb27effd640 1 -- 172.21.15.90:0/3636658559 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bda9d5f250 con 0x55bda998e6b0 2024-04-25T11:37:25.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.791+0000 7fb27effd640 1 --2- 172.21.15.90:0/3636658559 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bda998e6b0 0x55bda9d5e7d0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fb2740dc330 tx=0x7fb2740047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4f44da4525c65db server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:25.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.791+0000 7fb25effd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:37:25.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.792+0000 7fb25effd640 1 -- 172.21.15.90:0/3636658559 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb274075030 con 0x55bda998e6b0 2024-04-25T11:37:25.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.792+0000 7fb28ca4a740 1 -- 172.21.15.90:0/3636658559 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bda9d510e0 con 0x55bda998e6b0 2024-04-25T11:37:25.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.792+0000 7fb25effd640 1 -- 172.21.15.90:0/3636658559 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb2740a2b20 con 0x55bda998e6b0 2024-04-25T11:37:25.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.792+0000 7fb25effd640 1 -- 172.21.15.90:0/3636658559 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb2740a63a0 con 0x55bda998e6b0 2024-04-25T11:37:25.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.792+0000 7fb28ca4a740 10 client.16299 fetch_fsmap learned FSMap version 27 2024-04-25T11:37:25.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.792+0000 7fb28ca4a740 1 -- 172.21.15.90:0/3636658559 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bda9dd5740 con 0x55bda998e6b0 2024-04-25T11:37:25.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.792+0000 7fb25effd640 1 -- 172.21.15.90:0/3636658559 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55bda9d510e0 con 0x55bda998e6b0 2024-04-25T11:37:25.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.793+0000 7fb25effd640 1 -- 172.21.15.90:0/3636658559 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7fb2740bbce0 con 0x55bda998e6b0 2024-04-25T11:37:25.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.793+0000 7fb28ca4a740 10 client.16299 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:37:25.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.793+0000 7fb28ca4a740 10 client.16299 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:37:25.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.793+0000 7fb28ca4a740 20 client.16299 populate_metadata read hostname 'smithi090' 2024-04-25T11:37:25.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.793+0000 7fb28ca4a740 1 --2- 172.21.15.90:0/3636658559 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55bda99433a0 0x55bda9dd7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:25.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.793+0000 7fb28ca4a740 4 client.16299 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:37:25.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.793+0000 7fb28ca4a740 1 -- 172.21.15.90:0/3636658559 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bda9cbd800 con 0x55bda99433a0 2024-04-25T11:37:25.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.793+0000 7fb27f7fe640 1 --2- 172.21.15.90:0/3636658559 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55bda99433a0 0x55bda9dd7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:25.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.794+0000 7fb25effd640 10 client.16299 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:37:25.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.794+0000 7fb27f7fe640 1 --2- 172.21.15.90:0/3636658559 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55bda99433a0 0x55bda9dd7d70 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6953f946ead717cf server_cookie=83f5245516765278 in_seq=0 out_seq=0 2024-04-25T11:37:25.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.794+0000 7fb25effd640 1 -- 172.21.15.90:0/3636658559 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bda9cbd800 con 0x55bda99433a0 2024-04-25T11:37:25.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.794+0000 7fb25effd640 10 client.16299 handle_command_reply: tid=0 2024-04-25T11:37:25.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.794+0000 7fb28ca4a740 1 client.16299 shutdown 2024-04-25T11:37:25.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.794+0000 7fb28ca4a740 1 -- 172.21.15.90:0/3636658559 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bda998e6b0 msgr2=0x55bda9d5e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:25.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.794+0000 7fb28ca4a740 1 --2- 172.21.15.90:0/3636658559 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bda998e6b0 0x55bda9d5e7d0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fb2740dc330 tx=0x7fb2740047d0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.795+0000 7fb28ca4a740 1 -- 172.21.15.90:0/3636658559 shutdown_connections 2024-04-25T11:37:25.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.795+0000 7fb28ca4a740 1 --2- 172.21.15.90:0/3636658559 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55bda99433a0 0x55bda9dd7d70 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.795+0000 7fb28ca4a740 1 --2- 172.21.15.90:0/3636658559 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bda99af390 0x55bda9d5ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.795+0000 7fb25effd640 0 client.16299 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:37:25.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.795+0000 7fb28ca4a740 1 --2- 172.21.15.90:0/3636658559 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bda998e6b0 0x55bda9d5e7d0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.795+0000 7fb28ca4a740 1 --2- 172.21.15.90:0/3636658559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bda99864d0 0x55bda9d63480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.795+0000 7fb28ca4a740 1 -- 172.21.15.90:0/3636658559 >> 172.21.15.90:0/3636658559 conn(0x55bda99a34e0 msgr2=0x55bda9d4cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:25.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.795+0000 7fb28ca4a740 1 -- 172.21.15.90:0/3636658559 shutdown_connections 2024-04-25T11:37:25.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.795+0000 7fb28ca4a740 1 -- 172.21.15.90:0/3636658559 wait complete. 2024-04-25T11:37:25.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.795+0000 7fb28ca4a740 20 client.16299 trim_cache size 0 max 16384 2024-04-25T11:37:25.785 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:37:25.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.796+0000 7fb285b96640 1 -- 172.21.15.90:0/377639325 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb2680afce0 msgr2=0x7fb2680b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:25.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.796+0000 7fb285b96640 1 --2- 172.21.15.90:0/377639325 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb2680afce0 0x7fb2680b21f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb27800afd0 tx=0x7fb27800a750 comp rx=0 tx=0).stop 2024-04-25T11:37:25.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.796+0000 7fb285b96640 1 -- 172.21.15.90:0/377639325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb280109b10 msgr2=0x7fb280074ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:25.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.796+0000 7fb285b96640 1 --2- 172.21.15.90:0/377639325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb280109b10 0x7fb280074ea0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fb274006fd0 tx=0x7fb27400b8e0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.796+0000 7fb285b96640 1 -- 172.21.15.90:0/377639325 shutdown_connections 2024-04-25T11:37:25.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.796+0000 7fb285b96640 1 --2- 172.21.15.90:0/377639325 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb2680afce0 0x7fb2680b21f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.796+0000 7fb285b96640 1 --2- 172.21.15.90:0/377639325 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb280109fb0 0x7fb2800754a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.796+0000 7fb285b96640 1 --2- 172.21.15.90:0/377639325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb280109b10 0x7fb280074ea0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.796+0000 7fb285b96640 1 --2- 172.21.15.90:0/377639325 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2801042b0 0x7fb280076a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:25.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.796+0000 7fb285b96640 1 -- 172.21.15.90:0/377639325 >> 172.21.15.90:0/377639325 conn(0x7fb2800ff360 msgr2=0x7fb2800ff770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:25.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.796+0000 7fb285b96640 1 -- 172.21.15.90:0/377639325 shutdown_connections 2024-04-25T11:37:25.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:25.796+0000 7fb285b96640 1 -- 172.21.15.90:0/377639325 wait complete. 2024-04-25T11:37:25.804 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T11:37:26.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.047+0000 7fb047051640 1 -- 172.21.15.90:0/1973952407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb040109b10 msgr2=0x7fb0401036b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:26.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.047+0000 7fb047051640 1 --2- 172.21.15.90:0/1973952407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb040109b10 0x7fb0401036b0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fb030009860 tx=0x7fb03002a340 comp rx=0 tx=0).stop 2024-04-25T11:37:26.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.048+0000 7fb047051640 1 -- 172.21.15.90:0/1973952407 shutdown_connections 2024-04-25T11:37:26.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.048+0000 7fb047051640 1 --2- 172.21.15.90:0/1973952407 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb040104260 0x7fb040110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.048+0000 7fb047051640 1 --2- 172.21.15.90:0/1973952407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb040109fb0 0x7fb040103bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.048+0000 7fb047051640 1 --2- 172.21.15.90:0/1973952407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb040109b10 0x7fb0401036b0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.048+0000 7fb047051640 1 -- 172.21.15.90:0/1973952407 >> 172.21.15.90:0/1973952407 conn(0x7fb0400ff210 msgr2=0x7fb040101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:26.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.048+0000 7fb047051640 1 -- 172.21.15.90:0/1973952407 shutdown_connections 2024-04-25T11:37:26.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.049+0000 7fb047051640 1 -- 172.21.15.90:0/1973952407 wait complete. 2024-04-25T11:37:26.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.050+0000 7fb047051640 1 Processor -- start 2024-04-25T11:37:26.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.050+0000 7fb047051640 1 -- start start 2024-04-25T11:37:26.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.051+0000 7fb047051640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb040104260 0x7fb040113890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:26.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.051+0000 7fb047051640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb040109b10 0x7fb040113dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:26.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.051+0000 7fb047051640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb040109fb0 0x7fb040114310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:26.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.051+0000 7fb047051640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb040076740 con 0x7fb040104260 2024-04-25T11:37:26.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.051+0000 7fb047051640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb0400765c0 con 0x7fb040109b10 2024-04-25T11:37:26.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.051+0000 7fb047051640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb0400768c0 con 0x7fb040109fb0 2024-04-25T11:37:26.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.051+0000 7fb044ded640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb040104260 0x7fb040113890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:26.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.051+0000 7fb044ded640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb040104260 0x7fb040113890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55752/0 (socket says 172.21.15.90:55752) 2024-04-25T11:37:26.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.051+0000 7fb044ded640 1 -- 172.21.15.90:0/4150749039 learned_addr learned my addr 172.21.15.90:0/4150749039 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:26.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.052+0000 7fb044ded640 1 -- 172.21.15.90:0/4150749039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb040109fb0 msgr2=0x7fb040114310 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:37:26.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.052+0000 7fb044ded640 1 --2- 172.21.15.90:0/4150749039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb040109fb0 0x7fb040114310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.052+0000 7fb044ded640 1 -- 172.21.15.90:0/4150749039 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb040109b10 msgr2=0x7fb040113dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:37:26.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.052+0000 7fb044ded640 1 --2- 172.21.15.90:0/4150749039 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb040109b10 0x7fb040113dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.052+0000 7fb044ded640 1 -- 172.21.15.90:0/4150749039 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0401a4710 con 0x7fb040104260 2024-04-25T11:37:26.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.052+0000 7fb044ded640 1 --2- 172.21.15.90:0/4150749039 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb040104260 0x7fb040113890 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fb03002a8b0 tx=0x7fb030032df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:26.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.053+0000 7fb03dffb640 1 -- 172.21.15.90:0/4150749039 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb030042070 con 0x7fb040104260 2024-04-25T11:37:26.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.053+0000 7fb03dffb640 1 -- 172.21.15.90:0/4150749039 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb030030450 con 0x7fb040104260 2024-04-25T11:37:26.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.053+0000 7fb047051640 1 -- 172.21.15.90:0/4150749039 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0401a49a0 con 0x7fb040104260 2024-04-25T11:37:26.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.053+0000 7fb03dffb640 1 -- 172.21.15.90:0/4150749039 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb030037040 con 0x7fb040104260 2024-04-25T11:37:26.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.054+0000 7fb047051640 1 -- 172.21.15.90:0/4150749039 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0401b6090 con 0x7fb040104260 2024-04-25T11:37:26.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.060+0000 7fb04dfe3740 1 Processor -- start 2024-04-25T11:37:26.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.060+0000 7fb04dfe3740 1 -- start start 2024-04-25T11:37:26.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.060+0000 7fb04dfe3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e94e7a4390 0x55e94eb456a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:26.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.060+0000 7fb04dfe3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e94e77b4d0 0x55e94eb45be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:26.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.060+0000 7fb04dfe3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7836b0 0x55e94eb636b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:26.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.060+0000 7fb04dfe3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e94eaaad80 con 0x55e94e7836b0 2024-04-25T11:37:26.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.060+0000 7fb04dfe3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55e94ea64a20 con 0x55e94e7a4390 2024-04-25T11:37:26.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.060+0000 7fb04dfe3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e94eaa6d40 con 0x55e94e77b4d0 2024-04-25T11:37:26.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.061+0000 7fb03dffb640 1 -- 172.21.15.90:0/4150749039 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fb030045070 con 0x7fb040104260 2024-04-25T11:37:26.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.061+0000 7fb03ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7836b0 0x55e94eb636b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:26.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.061+0000 7fb03ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7836b0 0x55e94eb636b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55758/0 (socket says 172.21.15.90:55758) 2024-04-25T11:37:26.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.061+0000 7fb03ffff640 1 -- 172.21.15.90:0/3684299684 learned_addr learned my addr 172.21.15.90:0/3684299684 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:26.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.061+0000 7fb03ffff640 1 -- 172.21.15.90:0/3684299684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e94e77b4d0 msgr2=0x55e94eb45be0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:37:26.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.061+0000 7fb03ffff640 1 --2- 172.21.15.90:0/3684299684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e94e77b4d0 0x55e94eb45be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.061+0000 7fb03ffff640 1 -- 172.21.15.90:0/3684299684 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e94e7a4390 msgr2=0x55e94eb456a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:37:26.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.061+0000 7fb03ffff640 1 --2- 172.21.15.90:0/3684299684 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e94e7a4390 0x55e94eb456a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.061+0000 7fb03ffff640 1 -- 172.21.15.90:0/3684299684 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e94ea6fc30 con 0x55e94e7836b0 2024-04-25T11:37:26.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.062+0000 7fb03ffff640 1 --2- 172.21.15.90:0/3684299684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7836b0 0x55e94eb636b0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fb034009a50 tx=0x7fb034009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c37dceaae32255cc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:26.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.062+0000 7fb026ffd640 1 -- 172.21.15.90:0/3684299684 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03400ec90 con 0x55e94e7836b0 2024-04-25T11:37:26.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.062+0000 7fb026ffd640 1 -- 172.21.15.90:0/3684299684 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb03400ee30 con 0x55e94e7836b0 2024-04-25T11:37:26.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.062+0000 7fb026ffd640 1 -- 172.21.15.90:0/3684299684 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb034010430 con 0x55e94e7836b0 2024-04-25T11:37:26.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.062+0000 7fb03dffb640 1 --2- 172.21.15.90:0/4150749039 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb0280afd90 0x7fb0280b22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:26.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.063+0000 7fb0455ee640 1 --2- 172.21.15.90:0/4150749039 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb0280afd90 0x7fb0280b22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:26.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.063+0000 7fb03dffb640 1 -- 172.21.15.90:0/4150749039 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb0300fe880 con 0x7fb040104260 2024-04-25T11:37:26.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.063+0000 7fb04dfe3740 1 -- 172.21.15.90:0/3684299684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7836b0 msgr2=0x55e94eb636b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:26.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.063+0000 7fb04dfe3740 1 --2- 172.21.15.90:0/3684299684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7836b0 0x55e94eb636b0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fb034009a50 tx=0x7fb034009f10 comp rx=0 tx=0).stop 2024-04-25T11:37:26.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.064+0000 7fb04dfe3740 1 -- 172.21.15.90:0/3684299684 shutdown_connections 2024-04-25T11:37:26.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.064+0000 7fb04dfe3740 1 --2- 172.21.15.90:0/3684299684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7836b0 0x55e94eb636b0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.064+0000 7fb04dfe3740 1 --2- 172.21.15.90:0/3684299684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e94e77b4d0 0x55e94eb45be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.064+0000 7fb04dfe3740 1 --2- 172.21.15.90:0/3684299684 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e94e7a4390 0x55e94eb456a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.064+0000 7fb04dfe3740 1 -- 172.21.15.90:0/3684299684 >> 172.21.15.90:0/3684299684 conn(0x55e94e7984e0 msgr2=0x55e94eb41fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:26.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.064+0000 7fb04dfe3740 1 -- 172.21.15.90:0/3684299684 shutdown_connections 2024-04-25T11:37:26.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.064+0000 7fb0455ee640 1 --2- 172.21.15.90:0/4150749039 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb0280afd90 0x7fb0280b22a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x55e94eb429b0 tx=0x7fb038009250 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:26.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.064+0000 7fb04dfe3740 1 -- 172.21.15.90:0/3684299684 wait complete. 2024-04-25T11:37:26.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:25 smithi138 ceph-mon[32328]: pgmap v73: 97 pgs: 97 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 431 MiB/s wr, 3.05k op/s 2024-04-25T11:37:26.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.065+0000 7fb04dfe3740 1 -- start start 2024-04-25T11:37:26.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.065+0000 7fb04dfe3740 1 Processor -- start 2024-04-25T11:37:26.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.066+0000 7fb04dfe3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e94e77b4d0 0x55e94eb611f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:26.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.066+0000 7fb04dfe3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e94e7836b0 0x55e94eb61730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:26.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.066+0000 7fb04dfe3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7a4390 0x55e94eb6d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:26.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.066+0000 7fb04dfe3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e94eb63da0 con 0x55e94e7a4390 2024-04-25T11:37:26.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.066+0000 7fb04dfe3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55e94ea31dc0 con 0x55e94e7836b0 2024-04-25T11:37:26.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.066+0000 7fb04dfe3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e94eb63f20 con 0x55e94e77b4d0 2024-04-25T11:37:26.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.066+0000 7fb04dfe3740 10 client.0 _ll_register_callbacks cb 0x55e94ea4d600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:37:26.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.066+0000 7fb0455ee640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7a4390 0x55e94eb6d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:26.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.066+0000 7fb0455ee640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7a4390 0x55e94eb6d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55772/0 (socket says 172.21.15.90:55772) 2024-04-25T11:37:26.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.066+0000 7fb0455ee640 1 -- 172.21.15.90:0/1556099382 learned_addr learned my addr 172.21.15.90:0/1556099382 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:26.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.067+0000 7fb0455ee640 1 -- 172.21.15.90:0/1556099382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e94e77b4d0 msgr2=0x55e94eb611f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:26.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.067+0000 7fb0455ee640 1 --2- 172.21.15.90:0/1556099382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e94e77b4d0 0x55e94eb611f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.067+0000 7fb0455ee640 1 -- 172.21.15.90:0/1556099382 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e94e7836b0 msgr2=0x55e94eb61730 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:37:26.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.067+0000 7fb0455ee640 1 --2- 172.21.15.90:0/1556099382 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e94e7836b0 0x55e94eb61730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.067+0000 7fb0455ee640 1 -- 172.21.15.90:0/1556099382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e94e7bc3f0 con 0x55e94e7a4390 2024-04-25T11:37:26.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.067+0000 7fb0455ee640 1 --2- 172.21.15.90:0/1556099382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7a4390 0x55e94eb6d2e0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fb03800da00 tx=0x7fb038004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc2b41a4871bfbd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:26.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.067+0000 7fb01ffff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:37:26.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.068+0000 7fb01ffff640 1 -- 172.21.15.90:0/1556099382 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb038018d10 con 0x55e94e7a4390 2024-04-25T11:37:26.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.068+0000 7fb04dfe3740 1 -- 172.21.15.90:0/1556099382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb03400ee30 con 0x55e94e7a4390 2024-04-25T11:37:26.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.068+0000 7fb01ffff640 1 -- 172.21.15.90:0/1556099382 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb038023480 con 0x55e94e7a4390 2024-04-25T11:37:26.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.068+0000 7fb01ffff640 1 -- 172.21.15.90:0/1556099382 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb038023680 con 0x55e94e7a4390 2024-04-25T11:37:26.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.068+0000 7fb04dfe3740 10 client.16311 fetch_fsmap learned FSMap version 27 2024-04-25T11:37:26.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.068+0000 7fb04dfe3740 1 -- 172.21.15.90:0/1556099382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e94e8867d0 con 0x55e94e7a4390 2024-04-25T11:37:26.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.069+0000 7fb01ffff640 1 -- 172.21.15.90:0/1556099382 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb03400ee30 con 0x55e94e7a4390 2024-04-25T11:37:26.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.069+0000 7fb01ffff640 1 -- 172.21.15.90:0/1556099382 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7fb03801a9b0 con 0x55e94e7a4390 2024-04-25T11:37:26.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.070+0000 7fb04dfe3740 10 client.16311 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:37:26.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.070+0000 7fb04dfe3740 10 client.16311 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:37:26.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.070+0000 7fb04dfe3740 20 client.16311 populate_metadata read hostname 'smithi090' 2024-04-25T11:37:26.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.070+0000 7fb04dfe3740 1 --2- 172.21.15.90:0/1556099382 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e94e762f60 0x55e94eb737c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:26.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.070+0000 7fb04dfe3740 4 client.16311 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:37:26.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.070+0000 7fb04dfe3740 1 -- 172.21.15.90:0/1556099382 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e94eab8990 con 0x55e94e762f60 2024-04-25T11:37:26.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.072+0000 7fb044ded640 1 --2- 172.21.15.90:0/1556099382 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e94e762f60 0x55e94eb737c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:26.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.075+0000 7fb044ded640 1 --2- 172.21.15.90:0/1556099382 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e94e762f60 0x55e94eb737c0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b230fd2393b51abb server_cookie=3e75502095eacfe8 in_seq=0 out_seq=0 2024-04-25T11:37:26.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.075+0000 7fb01ffff640 10 client.16311 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:26.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.079+0000 7fb01ffff640 1 -- 172.21.15.90:0/1556099382 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e94eab8990 con 0x55e94e762f60 2024-04-25T11:37:26.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.079+0000 7fb01ffff640 10 client.16311 handle_command_reply: tid=0 2024-04-25T11:37:26.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.080+0000 7fb04dfe3740 1 client.16311 shutdown 2024-04-25T11:37:26.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.080+0000 7fb04dfe3740 1 -- 172.21.15.90:0/1556099382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7a4390 msgr2=0x55e94eb6d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:26.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.080+0000 7fb04dfe3740 1 --2- 172.21.15.90:0/1556099382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7a4390 0x55e94eb6d2e0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fb03800da00 tx=0x7fb038004b50 comp rx=0 tx=0).stop 2024-04-25T11:37:26.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.080+0000 7fb04dfe3740 1 -- 172.21.15.90:0/1556099382 shutdown_connections 2024-04-25T11:37:26.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.081+0000 7fb04dfe3740 1 --2- 172.21.15.90:0/1556099382 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e94e762f60 0x55e94eb737c0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.081+0000 7fb04dfe3740 1 --2- 172.21.15.90:0/1556099382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7a4390 0x55e94eb6d2e0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.081+0000 7fb04dfe3740 1 --2- 172.21.15.90:0/1556099382 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e94e7836b0 0x55e94eb61730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.081+0000 7fb04dfe3740 1 --2- 172.21.15.90:0/1556099382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e94e77b4d0 0x55e94eb611f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.081+0000 7fb04dfe3740 1 -- 172.21.15.90:0/1556099382 >> 172.21.15.90:0/1556099382 conn(0x55e94e7984e0 msgr2=0x55e94eb47530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:26.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.081+0000 7fb01ffff640 0 client.16311 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:26.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.081+0000 7fb04dfe3740 1 -- 172.21.15.90:0/1556099382 shutdown_connections 2024-04-25T11:37:26.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.081+0000 7fb04dfe3740 1 -- 172.21.15.90:0/1556099382 wait complete. 2024-04-25T11:37:26.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.081+0000 7fb04dfe3740 20 client.16311 trim_cache size 0 max 16384 2024-04-25T11:37:26.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:25 smithi156 ceph-mon[33045]: pgmap v73: 97 pgs: 97 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 431 MiB/s wr, 3.05k op/s 2024-04-25T11:37:26.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.105+0000 7fb04dfe3740 1 Processor -- start 2024-04-25T11:37:26.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.105+0000 7fb04dfe3740 1 -- start start 2024-04-25T11:37:26.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.106+0000 7fb04dfe3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e94e77b4d0 0x55e94eb6cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:26.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.106+0000 7fb04dfe3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7836b0 0x55e94eb5f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:26.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.106+0000 7fb04dfe3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e94e7a4390 0x55e94eb5fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:26.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.106+0000 7fb04dfe3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb0401b6260 con 0x55e94e7836b0 2024-04-25T11:37:26.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.106+0000 7fb04dfe3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb0401b7d50 con 0x55e94e7a4390 2024-04-25T11:37:26.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.106+0000 7fb04dfe3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb0401b68b0 con 0x55e94e77b4d0 2024-04-25T11:37:26.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.106+0000 7fb03ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7836b0 0x55e94eb5f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:26.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.106+0000 7fb03ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7836b0 0x55e94eb5f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55782/0 (socket says 172.21.15.90:55782) 2024-04-25T11:37:26.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.106+0000 7fb03ffff640 1 -- 172.21.15.90:0/2941988507 learned_addr learned my addr 172.21.15.90:0/2941988507 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:26.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.106+0000 7fb03ffff640 1 -- 172.21.15.90:0/2941988507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e94e77b4d0 msgr2=0x55e94eb6cec0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:37:26.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.106+0000 7fb03ffff640 1 --2- 172.21.15.90:0/2941988507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e94e77b4d0 0x55e94eb6cec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.106+0000 7fb03ffff640 1 -- 172.21.15.90:0/2941988507 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e94e7a4390 msgr2=0x55e94eb5fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:26.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.106+0000 7fb03ffff640 1 --2- 172.21.15.90:0/2941988507 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e94e7a4390 0x55e94eb5fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.106+0000 7fb03ffff640 1 -- 172.21.15.90:0/2941988507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e94eb47030 con 0x55e94e7836b0 2024-04-25T11:37:26.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.106+0000 7fb03ffff640 1 --2- 172.21.15.90:0/2941988507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7836b0 0x55e94eb5f8b0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fb034006010 tx=0x7fb034009710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1135e95b8fff9a41 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:26.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.107+0000 7fb0267fc640 1 -- 172.21.15.90:0/2941988507 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb034026020 con 0x55e94e7836b0 2024-04-25T11:37:26.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.107+0000 7fb0267fc640 1 -- 172.21.15.90:0/2941988507 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb034005000 con 0x55e94e7836b0 2024-04-25T11:37:26.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.107+0000 7fb0267fc640 1 -- 172.21.15.90:0/2941988507 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb034004b60 con 0x55e94e7836b0 2024-04-25T11:37:26.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.108+0000 7fb04dfe3740 1 -- 172.21.15.90:0/2941988507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7836b0 msgr2=0x55e94eb5f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:26.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.108+0000 7fb04dfe3740 1 --2- 172.21.15.90:0/2941988507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7836b0 0x55e94eb5f8b0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fb034006010 tx=0x7fb034009710 comp rx=0 tx=0).stop 2024-04-25T11:37:26.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.108+0000 7fb04dfe3740 1 -- 172.21.15.90:0/2941988507 shutdown_connections 2024-04-25T11:37:26.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.108+0000 7fb04dfe3740 1 --2- 172.21.15.90:0/2941988507 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e94e7a4390 0x55e94eb5fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.108+0000 7fb04dfe3740 1 --2- 172.21.15.90:0/2941988507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7836b0 0x55e94eb5f8b0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.108+0000 7fb04dfe3740 1 --2- 172.21.15.90:0/2941988507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e94e77b4d0 0x55e94eb6cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.108+0000 7fb04dfe3740 1 -- 172.21.15.90:0/2941988507 >> 172.21.15.90:0/2941988507 conn(0x55e94e7984e0 msgr2=0x55e94eb611f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:26.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.108+0000 7fb04dfe3740 1 -- 172.21.15.90:0/2941988507 shutdown_connections 2024-04-25T11:37:26.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.108+0000 7fb04dfe3740 1 -- 172.21.15.90:0/2941988507 wait complete. 2024-04-25T11:37:26.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.109+0000 7fb04dfe3740 1 -- start start 2024-04-25T11:37:26.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.109+0000 7fb04dfe3740 1 Processor -- start 2024-04-25T11:37:26.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.109+0000 7fb04dfe3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e94e77b4d0 0x55e94eb65f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:26.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.109+0000 7fb04dfe3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7836b0 0x55e94eb664d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:26.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.110+0000 7fb04dfe3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e94e7a4390 0x55e94eb66a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:26.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.110+0000 7fb04dfe3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb0401b7650 con 0x55e94e7836b0 2024-04-25T11:37:26.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.110+0000 7fb04dfe3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb0401b6f80 con 0x55e94e7a4390 2024-04-25T11:37:26.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.110+0000 7fb04dfe3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb04006a160 con 0x55e94e77b4d0 2024-04-25T11:37:26.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.110+0000 7fb04dfe3740 10 client.0 _ll_register_callbacks cb 0x7fb03800d700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:37:26.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.110+0000 7fb03ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7836b0 0x55e94eb664d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:26.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.110+0000 7fb03ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7836b0 0x55e94eb664d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55792/0 (socket says 172.21.15.90:55792) 2024-04-25T11:37:26.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.110+0000 7fb03ffff640 1 -- 172.21.15.90:0/460869594 learned_addr learned my addr 172.21.15.90:0/460869594 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:26.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.110+0000 7fb0455ee640 1 --2- 172.21.15.90:0/460869594 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e94e7a4390 0x55e94eb66a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:26.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.110+0000 7fb03ffff640 1 -- 172.21.15.90:0/460869594 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e94e77b4d0 msgr2=0x55e94eb65f90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:37:26.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.110+0000 7fb03ffff640 1 --2- 172.21.15.90:0/460869594 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e94e77b4d0 0x55e94eb65f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.110+0000 7fb03ffff640 1 -- 172.21.15.90:0/460869594 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e94e7a4390 msgr2=0x55e94eb66a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:26.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.111+0000 7fb03ffff640 1 --2- 172.21.15.90:0/460869594 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e94e7a4390 0x55e94eb66a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.111+0000 7fb03ffff640 1 -- 172.21.15.90:0/460869594 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e94eb5cba0 con 0x55e94e7836b0 2024-04-25T11:37:26.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.111+0000 7fb03ffff640 1 --2- 172.21.15.90:0/460869594 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7836b0 0x55e94eb664d0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fb034009f50 tx=0x7fb03400e890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e2cd41f69a826b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:26.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.111+0000 7fb01ffff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:37:26.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.111+0000 7fb01ffff640 1 -- 172.21.15.90:0/460869594 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb034004a60 con 0x55e94e7836b0 2024-04-25T11:37:26.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.112+0000 7fb01ffff640 1 -- 172.21.15.90:0/460869594 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb034004c00 con 0x55e94e7836b0 2024-04-25T11:37:26.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.112+0000 7fb04dfe3740 1 -- 172.21.15.90:0/460869594 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb034005000 con 0x55e94e7836b0 2024-04-25T11:37:26.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.112+0000 7fb01ffff640 1 -- 172.21.15.90:0/460869594 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb034004e00 con 0x55e94e7836b0 2024-04-25T11:37:26.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.112+0000 7fb04dfe3740 10 client.16317 fetch_fsmap learned FSMap version 27 2024-04-25T11:37:26.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.112+0000 7fb04dfe3740 1 -- 172.21.15.90:0/460869594 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e94ebdacc0 con 0x55e94e7836b0 2024-04-25T11:37:26.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.113+0000 7fb01ffff640 1 -- 172.21.15.90:0/460869594 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb034005000 con 0x55e94e7836b0 2024-04-25T11:37:26.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.114+0000 7fb01ffff640 1 -- 172.21.15.90:0/460869594 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7fb034022530 con 0x55e94e7836b0 2024-04-25T11:37:26.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.114+0000 7fb04dfe3740 10 client.16317 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:37:26.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.114+0000 7fb04dfe3740 10 client.16317 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:37:26.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.114+0000 7fb04dfe3740 20 client.16317 populate_metadata read hostname 'smithi090' 2024-04-25T11:37:26.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.114+0000 7fb04dfe3740 1 --2- 172.21.15.90:0/460869594 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e94e7383a0 0x55e94ebdd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:26.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.115+0000 7fb04dfe3740 4 client.16317 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:37:26.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.115+0000 7fb04dfe3740 1 -- 172.21.15.90:0/460869594 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e94eab2800 con 0x55e94e7383a0 2024-04-25T11:37:26.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.115+0000 7fb044ded640 1 --2- 172.21.15.90:0/460869594 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e94e7383a0 0x55e94ebdd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:26.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.116+0000 7fb01ffff640 10 client.16317 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:26.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.116+0000 7fb044ded640 1 --2- 172.21.15.90:0/460869594 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e94e7383a0 0x55e94ebdd100 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37e11038525fabf server_cookie=3ac760fa8442f780 in_seq=0 out_seq=0 2024-04-25T11:37:26.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.119+0000 7fb01ffff640 1 -- 172.21.15.90:0/460869594 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e94eab2800 con 0x55e94e7383a0 2024-04-25T11:37:26.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.119+0000 7fb01ffff640 10 client.16317 handle_command_reply: tid=0 2024-04-25T11:37:26.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.120+0000 7fb04dfe3740 1 client.16317 shutdown 2024-04-25T11:37:26.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.120+0000 7fb04dfe3740 1 -- 172.21.15.90:0/460869594 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7836b0 msgr2=0x55e94eb664d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:26.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.120+0000 7fb04dfe3740 1 --2- 172.21.15.90:0/460869594 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7836b0 0x55e94eb664d0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fb034009f50 tx=0x7fb03400e890 comp rx=0 tx=0).stop 2024-04-25T11:37:26.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.120+0000 7fb04dfe3740 1 -- 172.21.15.90:0/460869594 shutdown_connections 2024-04-25T11:37:26.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.120+0000 7fb04dfe3740 1 --2- 172.21.15.90:0/460869594 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e94e7383a0 0x55e94ebdd100 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.120+0000 7fb04dfe3740 1 --2- 172.21.15.90:0/460869594 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e94e7a4390 0x55e94eb66a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.120+0000 7fb04dfe3740 1 --2- 172.21.15.90:0/460869594 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94e7836b0 0x55e94eb664d0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.120+0000 7fb04dfe3740 1 --2- 172.21.15.90:0/460869594 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e94e77b4d0 0x55e94eb65f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.120+0000 7fb04dfe3740 1 -- 172.21.15.90:0/460869594 >> 172.21.15.90:0/460869594 conn(0x55e94e7984e0 msgr2=0x55e94eb611f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:26.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.120+0000 7fb01ffff640 0 client.16317 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:26.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.120+0000 7fb04dfe3740 1 -- 172.21.15.90:0/460869594 shutdown_connections 2024-04-25T11:37:26.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.120+0000 7fb04dfe3740 1 -- 172.21.15.90:0/460869594 wait complete. 2024-04-25T11:37:26.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.121+0000 7fb04dfe3740 20 client.16317 trim_cache size 0 max 16384 2024-04-25T11:37:26.110 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:37:26.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.122+0000 7fb047051640 1 -- 172.21.15.90:0/4150749039 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb0280afd90 msgr2=0x7fb0280b22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:26.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.122+0000 7fb047051640 1 --2- 172.21.15.90:0/4150749039 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb0280afd90 0x7fb0280b22a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x55e94eb429b0 tx=0x7fb038009250 comp rx=0 tx=0).stop 2024-04-25T11:37:26.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.122+0000 7fb047051640 1 -- 172.21.15.90:0/4150749039 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb040104260 msgr2=0x7fb040113890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:26.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.122+0000 7fb047051640 1 --2- 172.21.15.90:0/4150749039 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb040104260 0x7fb040113890 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fb03002a8b0 tx=0x7fb030032df0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.122+0000 7fb047051640 1 -- 172.21.15.90:0/4150749039 shutdown_connections 2024-04-25T11:37:26.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.122+0000 7fb047051640 1 --2- 172.21.15.90:0/4150749039 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb0280afd90 0x7fb0280b22a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.122+0000 7fb047051640 1 --2- 172.21.15.90:0/4150749039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb040109fb0 0x7fb040114310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.122+0000 7fb047051640 1 --2- 172.21.15.90:0/4150749039 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb040109b10 0x7fb040113dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.122+0000 7fb047051640 1 --2- 172.21.15.90:0/4150749039 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb040104260 0x7fb040113890 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:26.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.122+0000 7fb047051640 1 -- 172.21.15.90:0/4150749039 >> 172.21.15.90:0/4150749039 conn(0x7fb0400ff210 msgr2=0x7fb0401017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:26.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.122+0000 7fb047051640 1 -- 172.21.15.90:0/4150749039 shutdown_connections 2024-04-25T11:37:26.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:26.122+0000 7fb047051640 1 -- 172.21.15.90:0/4150749039 wait complete. 2024-04-25T11:37:26.125 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:25 smithi090 ceph-mon[29575]: pgmap v73: 97 pgs: 97 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 431 MiB/s wr, 3.05k op/s 2024-04-25T11:37:27.127 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T11:37:27.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:37:27.127 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T11:37:27.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.451+0000 7f92f7a15640 1 -- 172.21.15.90:0/1542262811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92f010d380 msgr2=0x7f92f010f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:27.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.451+0000 7f92f7a15640 1 --2- 172.21.15.90:0/1542262811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92f010d380 0x7f92f010f7a0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f92e8009ca0 tx=0x7f92e802a8c0 comp rx=0 tx=0).stop 2024-04-25T11:37:27.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.451+0000 7f92f7a15640 1 -- 172.21.15.90:0/1542262811 shutdown_connections 2024-04-25T11:37:27.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.451+0000 7f92f7a15640 1 --2- 172.21.15.90:0/1542262811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92f010d380 0x7f92f010f7a0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:27.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.451+0000 7f92f7a15640 1 --2- 172.21.15.90:0/1542262811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92f0102af0 0x7f92f0102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:27.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.451+0000 7f92f7a15640 1 --2- 172.21.15.90:0/1542262811 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f92f0108a20 0x7f92f01025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:27.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.451+0000 7f92f7a15640 1 -- 172.21.15.90:0/1542262811 >> 172.21.15.90:0/1542262811 conn(0x7f92f0078100 msgr2=0x7f92f0078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:27.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.452+0000 7f92f7a15640 1 -- 172.21.15.90:0/1542262811 shutdown_connections 2024-04-25T11:37:27.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.452+0000 7f92f7a15640 1 -- 172.21.15.90:0/1542262811 wait complete. 2024-04-25T11:37:27.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.453+0000 7f92f7a15640 1 Processor -- start 2024-04-25T11:37:27.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.453+0000 7f92f7a15640 1 -- start start 2024-04-25T11:37:27.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.454+0000 7f92f7a15640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92f0102af0 0x7f92f019d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:27.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.454+0000 7f92f7a15640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f92f0108a20 0x7f92f019db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:27.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.454+0000 7f92f7a15640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92f010d380 0x7f92f01a86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:27.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.454+0000 7f92f5fb2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92f010d380 0x7f92f01a86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:27.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.454+0000 7f92f5fb2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92f010d380 0x7f92f01a86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55806/0 (socket says 172.21.15.90:55806) 2024-04-25T11:37:27.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.454+0000 7f92f5fb2640 1 -- 172.21.15.90:0/4135700000 learned_addr learned my addr 172.21.15.90:0/4135700000 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:27.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.454+0000 7f92f7a15640 1 -- 172.21.15.90:0/4135700000 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f92f0113d50 con 0x7f92f010d380 2024-04-25T11:37:27.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.454+0000 7f92f7a15640 1 -- 172.21.15.90:0/4135700000 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f92f0113bd0 con 0x7f92f0108a20 2024-04-25T11:37:27.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.454+0000 7f92f7a15640 1 -- 172.21.15.90:0/4135700000 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f92f0113ed0 con 0x7f92f0102af0 2024-04-25T11:37:27.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.455+0000 7f92f5fb2640 1 -- 172.21.15.90:0/4135700000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92f0102af0 msgr2=0x7f92f019d630 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:37:27.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.455+0000 7f92f5fb2640 1 --2- 172.21.15.90:0/4135700000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92f0102af0 0x7f92f019d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:27.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.455+0000 7f92f5fb2640 1 -- 172.21.15.90:0/4135700000 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f92f0108a20 msgr2=0x7f92f019db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:27.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.455+0000 7f92f5fb2640 1 --2- 172.21.15.90:0/4135700000 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f92f0108a20 0x7f92f019db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:27.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.455+0000 7f92f5fb2640 1 -- 172.21.15.90:0/4135700000 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92f01a8bf0 con 0x7f92f010d380 2024-04-25T11:37:27.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.455+0000 7f92f5fb2640 1 --2- 172.21.15.90:0/4135700000 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92f010d380 0x7f92f01a86b0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f92e8009430 tx=0x7f92e80317c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:27.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.456+0000 7f92e67fc640 1 -- 172.21.15.90:0/4135700000 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e8003b70 con 0x7f92f010d380 2024-04-25T11:37:27.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.456+0000 7f92f7a15640 1 -- 172.21.15.90:0/4135700000 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92f01a8dc0 con 0x7f92f010d380 2024-04-25T11:37:27.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.456+0000 7f92f7a15640 1 -- 172.21.15.90:0/4135700000 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92f01a92b0 con 0x7f92f010d380 2024-04-25T11:37:27.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.456+0000 7f92e67fc640 1 -- 172.21.15.90:0/4135700000 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92e8003d10 con 0x7f92f010d380 2024-04-25T11:37:27.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.456+0000 7f92e67fc640 1 -- 172.21.15.90:0/4135700000 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e802f9e0 con 0x7f92f010d380 2024-04-25T11:37:27.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.461+0000 7f92fe9a6740 1 Processor -- start 2024-04-25T11:37:27.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.462+0000 7f92fe9a6740 1 -- start start 2024-04-25T11:37:27.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.463+0000 7f92fe9a6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfecee410 0x563bff08e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:27.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.463+0000 7f92fe9a6740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563bfecc5550 0x563bff08ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:27.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.463+0000 7f92fe9a6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563bfeccd730 0x563bff08ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:27.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.463+0000 7f92fe9a6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563bfeff4b00 con 0x563bfecee410 2024-04-25T11:37:27.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.463+0000 7f92fe9a6740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x563bfef90920 con 0x563bfecc5550 2024-04-25T11:37:27.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.463+0000 7f92fe9a6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563bfeff0ac0 con 0x563bfeccd730 2024-04-25T11:37:27.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.464+0000 7f92e67fc640 1 -- 172.21.15.90:0/4135700000 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f92e802fc80 con 0x7f92f010d380 2024-04-25T11:37:27.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.464+0000 7f92f4fb0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfecee410 0x563bff08e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:27.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.464+0000 7f92f4fb0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfecee410 0x563bff08e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55818/0 (socket says 172.21.15.90:55818) 2024-04-25T11:37:27.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.464+0000 7f92f4fb0640 1 -- 172.21.15.90:0/983800349 learned_addr learned my addr 172.21.15.90:0/983800349 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:27.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.465+0000 7f92f4fb0640 1 -- 172.21.15.90:0/983800349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563bfeccd730 msgr2=0x563bff08ef90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:37:27.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.465+0000 7f92f4fb0640 1 --2- 172.21.15.90:0/983800349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563bfeccd730 0x563bff08ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:27.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.465+0000 7f92f4fb0640 1 -- 172.21.15.90:0/983800349 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563bfecc5550 msgr2=0x563bff08ea50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:37:27.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.465+0000 7f92f4fb0640 1 --2- 172.21.15.90:0/983800349 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563bfecc5550 0x563bff08ea50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:27.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.465+0000 7f92f4fb0640 1 -- 172.21.15.90:0/983800349 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bfedce7e0 con 0x563bfecee410 2024-04-25T11:37:27.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.465+0000 7f92e67fc640 1 --2- 172.21.15.90:0/4135700000 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f92dc0afd60 0x7f92dc0b2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:27.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.465+0000 7f92f4fb0640 1 --2- 172.21.15.90:0/983800349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfecee410 0x563bff08e510 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f92e000ba80 tx=0x7f92e000bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb875b5d58e50542 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:27.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.465+0000 7f92e67fc640 1 -- 172.21.15.90:0/4135700000 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f92e8100460 con 0x7f92f010d380 2024-04-25T11:37:27.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.466+0000 7f92f57b1640 1 --2- 172.21.15.90:0/4135700000 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f92dc0afd60 0x7f92dc0b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:27.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.466+0000 7f92db7fe640 1 -- 172.21.15.90:0/983800349 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e0016070 con 0x563bfecee410 2024-04-25T11:37:27.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.466+0000 7f92db7fe640 1 -- 172.21.15.90:0/983800349 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92e00054d0 con 0x563bfecee410 2024-04-25T11:37:27.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.466+0000 7f92db7fe640 1 -- 172.21.15.90:0/983800349 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e0010460 con 0x563bfecee410 2024-04-25T11:37:27.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.467+0000 7f92fe9a6740 1 -- 172.21.15.90:0/983800349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfecee410 msgr2=0x563bff08e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:27.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.468+0000 7f92f57b1640 1 --2- 172.21.15.90:0/4135700000 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f92dc0afd60 0x7f92dc0b2270 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x563bff08c110 tx=0x7f92ec005f90 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:27.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.467+0000 7f92fe9a6740 1 --2- 172.21.15.90:0/983800349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfecee410 0x563bff08e510 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f92e000ba80 tx=0x7f92e000bf40 comp rx=0 tx=0).stop 2024-04-25T11:37:27.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.468+0000 7f92fe9a6740 1 -- 172.21.15.90:0/983800349 shutdown_connections 2024-04-25T11:37:27.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.468+0000 7f92fe9a6740 1 --2- 172.21.15.90:0/983800349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563bfeccd730 0x563bff08ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:27.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.468+0000 7f92fe9a6740 1 --2- 172.21.15.90:0/983800349 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563bfecc5550 0x563bff08ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:27.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.468+0000 7f92fe9a6740 1 --2- 172.21.15.90:0/983800349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfecee410 0x563bff08e510 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:27.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.468+0000 7f92fe9a6740 1 -- 172.21.15.90:0/983800349 >> 172.21.15.90:0/983800349 conn(0x563bfece2560 msgr2=0x563bff08add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:27.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.469+0000 7f92fe9a6740 1 -- 172.21.15.90:0/983800349 shutdown_connections 2024-04-25T11:37:27.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.469+0000 7f92fe9a6740 1 -- 172.21.15.90:0/983800349 wait complete. 2024-04-25T11:37:27.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.469+0000 7f92fe9a6740 1 -- start start 2024-04-25T11:37:27.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.470+0000 7f92fe9a6740 1 Processor -- start 2024-04-25T11:37:27.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.470+0000 7f92fe9a6740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563bfecc5550 0x563bff09df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:27.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.470+0000 7f92fe9a6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563bfeccd730 0x563bff09e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:27.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.470+0000 7f92fe9a6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfecee410 0x563bff099350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:27.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.471+0000 7f92fe9a6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563bff091840 con 0x563bfecee410 2024-04-25T11:37:27.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.471+0000 7f92fe9a6740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x563bfef8e5d0 con 0x563bfecc5550 2024-04-25T11:37:27.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.471+0000 7f92fe9a6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563bff0919c0 con 0x563bfeccd730 2024-04-25T11:37:27.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.471+0000 7f92fe9a6740 10 client.0 _ll_register_callbacks cb 0x563bfef7c080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:37:27.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.471+0000 7f92f5fb2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfecee410 0x563bff099350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:27.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.471+0000 7f92f5fb2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfecee410 0x563bff099350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55820/0 (socket says 172.21.15.90:55820) 2024-04-25T11:37:27.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.471+0000 7f92f5fb2640 1 -- 172.21.15.90:0/2990357839 learned_addr learned my addr 172.21.15.90:0/2990357839 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:27.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.471+0000 7f92f5fb2640 1 -- 172.21.15.90:0/2990357839 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563bfeccd730 msgr2=0x563bff09e4d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:37:27.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.471+0000 7f92f5fb2640 1 --2- 172.21.15.90:0/2990357839 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563bfeccd730 0x563bff09e4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:27.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.471+0000 7f92f5fb2640 1 -- 172.21.15.90:0/2990357839 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563bfecc5550 msgr2=0x563bff09df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:27.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.471+0000 7f92f5fb2640 1 --2- 172.21.15.90:0/2990357839 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563bfecc5550 0x563bff09df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:27.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.471+0000 7f92f5fb2640 1 -- 172.21.15.90:0/2990357839 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bfee2ef70 con 0x563bfecee410 2024-04-25T11:37:27.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.472+0000 7f92f5fb2640 1 --2- 172.21.15.90:0/2990357839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfecee410 0x563bff099350 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f92e80c85e0 tx=0x7f92e80b1280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd83f5208e983240 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:27.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.472+0000 7f92d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:37:27.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.472+0000 7f92d8ff9640 1 -- 172.21.15.90:0/2990357839 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e80e2060 con 0x563bfecee410 2024-04-25T11:37:27.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.472+0000 7f92d8ff9640 1 -- 172.21.15.90:0/2990357839 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f92e80fdd70 con 0x563bfecee410 2024-04-25T11:37:27.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.472+0000 7f92fe9a6740 1 -- 172.21.15.90:0/2990357839 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92e00054d0 con 0x563bfecee410 2024-04-25T11:37:27.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.473+0000 7f92d8ff9640 1 -- 172.21.15.90:0/2990357839 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f92e80fdf70 con 0x563bfecee410 2024-04-25T11:37:27.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.473+0000 7f92fe9a6740 10 client.16329 fetch_fsmap learned FSMap version 27 2024-04-25T11:37:27.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.473+0000 7f92fe9a6740 1 -- 172.21.15.90:0/2990357839 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563bfecc6170 con 0x563bfecee410 2024-04-25T11:37:27.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.473+0000 7f92d8ff9640 1 -- 172.21.15.90:0/2990357839 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92e8036040 con 0x563bfecee410 2024-04-25T11:37:27.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.473+0000 7f92d8ff9640 1 -- 172.21.15.90:0/2990357839 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f92e80c88b0 con 0x563bfecee410 2024-04-25T11:37:27.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.473+0000 7f92fe9a6740 10 client.16329 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:37:27.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.473+0000 7f92fe9a6740 10 client.16329 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:37:27.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.473+0000 7f92fe9a6740 20 client.16329 populate_metadata read hostname 'smithi090' 2024-04-25T11:37:27.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.473+0000 7f92fe9a6740 1 --2- 172.21.15.90:0/2990357839 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563bfecad1f0 0x563bff0bbf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:27.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.473+0000 7f92fe9a6740 4 client.16329 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:37:27.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.473+0000 7f92fe9a6740 1 -- 172.21.15.90:0/2990357839 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f92e00054d0 con 0x563bfecad1f0 2024-04-25T11:37:27.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.475+0000 7f92f57b1640 1 --2- 172.21.15.90:0/2990357839 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563bfecad1f0 0x563bff0bbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:27.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.479+0000 7f92d8ff9640 10 client.16329 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:27.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.479+0000 7f92f57b1640 1 --2- 172.21.15.90:0/2990357839 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563bfecad1f0 0x563bff0bbf30 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3789132cb2704e46 server_cookie=585c9055e89f1c18 in_seq=0 out_seq=0 2024-04-25T11:37:27.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.482+0000 7f92d8ff9640 1 -- 172.21.15.90:0/2990357839 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f92e00054d0 con 0x563bfecad1f0 2024-04-25T11:37:27.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.482+0000 7f92d8ff9640 10 client.16329 handle_command_reply: tid=0 2024-04-25T11:37:27.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.483+0000 7f92fe9a6740 1 client.16329 shutdown 2024-04-25T11:37:27.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.483+0000 7f92fe9a6740 1 -- 172.21.15.90:0/2990357839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfecee410 msgr2=0x563bff099350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:27.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.483+0000 7f92fe9a6740 1 --2- 172.21.15.90:0/2990357839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfecee410 0x563bff099350 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f92e80c85e0 tx=0x7f92e80b1280 comp rx=0 tx=0).stop 2024-04-25T11:37:27.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.483+0000 7f92fe9a6740 1 -- 172.21.15.90:0/2990357839 shutdown_connections 2024-04-25T11:37:27.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.483+0000 7f92fe9a6740 1 --2- 172.21.15.90:0/2990357839 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563bfecad1f0 0x563bff0bbf30 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:27.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.483+0000 7f92fe9a6740 1 --2- 172.21.15.90:0/2990357839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfecee410 0x563bff099350 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:27.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.483+0000 7f92fe9a6740 1 --2- 172.21.15.90:0/2990357839 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563bfeccd730 0x563bff09e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:27.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.483+0000 7f92fe9a6740 1 --2- 172.21.15.90:0/2990357839 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563bfecc5550 0x563bff09df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:27.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.483+0000 7f92fe9a6740 1 -- 172.21.15.90:0/2990357839 >> 172.21.15.90:0/2990357839 conn(0x563bfece2560 msgr2=0x563bff0902d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:27.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.483+0000 7f92d8ff9640 0 client.16329 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:27.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.483+0000 7f92fe9a6740 1 -- 172.21.15.90:0/2990357839 shutdown_connections 2024-04-25T11:37:27.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.483+0000 7f92fe9a6740 1 -- 172.21.15.90:0/2990357839 wait complete. 2024-04-25T11:37:27.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.484+0000 7f92fe9a6740 20 client.16329 trim_cache size 0 max 16384 2024-04-25T11:37:27.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.511+0000 7f92fe9a6740 1 Processor -- start 2024-04-25T11:37:27.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.511+0000 7f92fe9a6740 1 -- start start 2024-04-25T11:37:27.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.511+0000 7f92fe9a6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfecc5550 0x563bff09d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:27.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.511+0000 7f92fe9a6740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563bfeccd730 0x563bff089bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:27.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.511+0000 7f92fe9a6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563bfecee410 0x563bff0c1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:27.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.511+0000 7f92fe9a6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f92d400e900 con 0x563bfecc5550 2024-04-25T11:37:27.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.511+0000 7f92fe9a6740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f92d4010470 con 0x563bfeccd730 2024-04-25T11:37:27.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.511+0000 7f92fe9a6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f92d400efd0 con 0x563bfecee410 2024-04-25T11:37:27.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.511+0000 7f92f57b1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfecc5550 0x563bff09d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:27.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.511+0000 7f92f57b1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfecc5550 0x563bff09d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55834/0 (socket says 172.21.15.90:55834) 2024-04-25T11:37:27.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.511+0000 7f92f57b1640 1 -- 172.21.15.90:0/845286099 learned_addr learned my addr 172.21.15.90:0/845286099 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:27.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.512+0000 7f92f57b1640 1 -- 172.21.15.90:0/845286099 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563bfecee410 msgr2=0x563bff0c1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:27.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.512+0000 7f92f57b1640 1 --2- 172.21.15.90:0/845286099 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563bfecee410 0x563bff0c1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:27.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.512+0000 7f92f57b1640 1 -- 172.21.15.90:0/845286099 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563bfeccd730 msgr2=0x563bff089bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:27.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.512+0000 7f92f57b1640 1 --2- 172.21.15.90:0/845286099 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563bfeccd730 0x563bff089bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:27.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.512+0000 7f92f57b1640 1 -- 172.21.15.90:0/845286099 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bff09a410 con 0x563bfecc5550 2024-04-25T11:37:27.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.512+0000 7f92f57b1640 1 --2- 172.21.15.90:0/845286099 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfecc5550 0x563bff09d8d0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x563bff098bb0 tx=0x7f92ec011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4a91d7422f89c37 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:27.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.512+0000 7f92daffd640 1 -- 172.21.15.90:0/845286099 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ec028030 con 0x563bfecc5550 2024-04-25T11:37:27.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.512+0000 7f92daffd640 1 -- 172.21.15.90:0/845286099 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92ec01de30 con 0x563bfecc5550 2024-04-25T11:37:27.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.512+0000 7f92daffd640 1 -- 172.21.15.90:0/845286099 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ec004920 con 0x563bfecc5550 2024-04-25T11:37:27.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.513+0000 7f92fe9a6740 1 -- 172.21.15.90:0/845286099 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfecc5550 msgr2=0x563bff09d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:27.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.513+0000 7f92fe9a6740 1 --2- 172.21.15.90:0/845286099 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfecc5550 0x563bff09d8d0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x563bff098bb0 tx=0x7f92ec011000 comp rx=0 tx=0).stop 2024-04-25T11:37:27.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.513+0000 7f92fe9a6740 1 -- 172.21.15.90:0/845286099 shutdown_connections 2024-04-25T11:37:27.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.513+0000 7f92fe9a6740 1 --2- 172.21.15.90:0/845286099 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563bfecee410 0x563bff0c1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:27.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.513+0000 7f92fe9a6740 1 --2- 172.21.15.90:0/845286099 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563bfeccd730 0x563bff089bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:27.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.513+0000 7f92fe9a6740 1 --2- 172.21.15.90:0/845286099 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfecc5550 0x563bff09d8d0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:27.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.513+0000 7f92fe9a6740 1 -- 172.21.15.90:0/845286099 >> 172.21.15.90:0/845286099 conn(0x563bfece2560 msgr2=0x563bff0894c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:27.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.514+0000 7f92fe9a6740 1 -- 172.21.15.90:0/845286099 shutdown_connections 2024-04-25T11:37:27.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.514+0000 7f92fe9a6740 1 -- 172.21.15.90:0/845286099 wait complete. 2024-04-25T11:37:27.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.514+0000 7f92fe9a6740 1 -- start start 2024-04-25T11:37:27.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.514+0000 7f92fe9a6740 1 Processor -- start 2024-04-25T11:37:27.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.515+0000 7f92fe9a6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563bfecc5550 0x563bff0cd320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:27.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.515+0000 7f92fe9a6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfeccd730 0x563bff0c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:27.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.515+0000 7f92fe9a6740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563bfecee410 0x563bff0c9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:27.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.515+0000 7f92f4fb0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfeccd730 0x563bff0c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:27.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.515+0000 7f92f4fb0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfeccd730 0x563bff0c96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55848/0 (socket says 172.21.15.90:55848) 2024-04-25T11:37:27.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.515+0000 7f92f4fb0640 1 -- 172.21.15.90:0/819114376 learned_addr learned my addr 172.21.15.90:0/819114376 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:27.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.515+0000 7f92fe9a6740 1 -- 172.21.15.90:0/819114376 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f92d400fd70 con 0x563bfeccd730 2024-04-25T11:37:27.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.515+0000 7f92f57b1640 1 --2- 172.21.15.90:0/819114376 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563bfecc5550 0x563bff0cd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:27.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.515+0000 7f92fe9a6740 1 -- 172.21.15.90:0/819114376 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f92d400f6a0 con 0x563bfecee410 2024-04-25T11:37:27.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.515+0000 7f92f5fb2640 1 --2- 172.21.15.90:0/819114376 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563bfecee410 0x563bff0c9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:27.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.515+0000 7f92fe9a6740 1 -- 172.21.15.90:0/819114376 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f92d400c8e0 con 0x563bfecc5550 2024-04-25T11:37:27.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.516+0000 7f92fe9a6740 10 client.0 _ll_register_callbacks cb 0x7f92e80c8420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:37:27.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.516+0000 7f92f5fb2640 1 -- 172.21.15.90:0/819114376 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563bfecc5550 msgr2=0x563bff0cd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:27.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.516+0000 7f92f5fb2640 1 --2- 172.21.15.90:0/819114376 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563bfecc5550 0x563bff0cd320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:27.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.516+0000 7f92f5fb2640 1 -- 172.21.15.90:0/819114376 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfeccd730 msgr2=0x563bff0c96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:27.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.516+0000 7f92f5fb2640 1 --2- 172.21.15.90:0/819114376 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfeccd730 0x563bff0c96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:27.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.516+0000 7f92f5fb2640 1 -- 172.21.15.90:0/819114376 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bff0ca150 con 0x563bfecee410 2024-04-25T11:37:27.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.516+0000 7f92f5fb2640 1 --2- 172.21.15.90:0/819114376 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563bfecee410 0x563bff0c9c10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f92e8005e30 tx=0x7f92e80c81d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53e9c1e8dfd0359 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:27.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.516+0000 7f92d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:37:27.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.517+0000 7f92d8ff9640 1 -- 172.21.15.90:0/819114376 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e80c4510 con 0x563bfecee410 2024-04-25T11:37:27.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.517+0000 7f92d8ff9640 1 -- 172.21.15.90:0/819114376 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f92e80c78b0 con 0x563bfecee410 2024-04-25T11:37:27.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.517+0000 7f92fe9a6740 1 -- 172.21.15.90:0/819114376 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92ec01de30 con 0x563bfecee410 2024-04-25T11:37:27.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.517+0000 7f92d8ff9640 1 -- 172.21.15.90:0/819114376 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f92e80c8650 con 0x563bfecee410 2024-04-25T11:37:27.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.517+0000 7f92fe9a6740 10 client.25658 fetch_fsmap learned FSMap version 27 2024-04-25T11:37:27.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.517+0000 7f92fe9a6740 1 -- 172.21.15.90:0/819114376 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563bff124ba0 con 0x563bfecee410 2024-04-25T11:37:27.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.517+0000 7f92d8ff9640 1 -- 172.21.15.90:0/819114376 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92e8036070 con 0x563bfecee410 2024-04-25T11:37:27.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.517+0000 7f92d8ff9640 1 -- 172.21.15.90:0/819114376 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f92e80ddaa0 con 0x563bfecee410 2024-04-25T11:37:27.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.517+0000 7f92fe9a6740 10 client.25658 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:37:27.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.518+0000 7f92fe9a6740 10 client.25658 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:37:27.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.518+0000 7f92fe9a6740 20 client.25658 populate_metadata read hostname 'smithi090' 2024-04-25T11:37:27.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.518+0000 7f92fe9a6740 1 --2- 172.21.15.90:0/819114376 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563bfec82630 0x563bff127190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:27.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.518+0000 7f92fe9a6740 4 client.25658 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T11:37:27.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.518+0000 7f92f57b1640 1 --2- 172.21.15.90:0/819114376 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563bfec82630 0x563bff127190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:27.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.518+0000 7f92fe9a6740 1 -- 172.21.15.90:0/819114376 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f92ec01de30 con 0x563bfec82630 2024-04-25T11:37:27.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.519+0000 7f92d8ff9640 10 client.25658 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:27.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:27.519+0000 7f92f57b1640 1 --2- 172.21.15.90:0/819114376 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563bfec82630 0x563bff127190 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9422bb1d17cc50e0 server_cookie=891db27ed4eeac41 in_seq=0 out_seq=0 2024-04-25T11:37:28.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.045+0000 7f92d8ff9640 1 -- 172.21.15.90:0/819114376 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f92ec01de30 con 0x563bfec82630 2024-04-25T11:37:28.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.045+0000 7f92d8ff9640 10 client.25658 handle_command_reply: tid=0 2024-04-25T11:37:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.045+0000 7f92fe9a6740 1 client.25658 shutdown 2024-04-25T11:37:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.046+0000 7f92fe9a6740 1 -- 172.21.15.90:0/819114376 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563bfecee410 msgr2=0x563bff0c9c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.046+0000 7f92fe9a6740 1 --2- 172.21.15.90:0/819114376 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563bfecee410 0x563bff0c9c10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f92e8005e30 tx=0x7f92e80c81d0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.046+0000 7f92fe9a6740 1 -- 172.21.15.90:0/819114376 shutdown_connections 2024-04-25T11:37:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.046+0000 7f92fe9a6740 1 --2- 172.21.15.90:0/819114376 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563bfec82630 0x563bff127190 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.046+0000 7f92fe9a6740 1 --2- 172.21.15.90:0/819114376 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563bfecee410 0x563bff0c9c10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.046+0000 7f92fe9a6740 1 --2- 172.21.15.90:0/819114376 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563bfeccd730 0x563bff0c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.046+0000 7f92fe9a6740 1 --2- 172.21.15.90:0/819114376 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563bfecc5550 0x563bff0cd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.046+0000 7f92fe9a6740 1 -- 172.21.15.90:0/819114376 >> 172.21.15.90:0/819114376 conn(0x563bfece2560 msgr2=0x563bff0a2af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.046+0000 7f92d8ff9640 0 client.25658 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.046+0000 7f92fe9a6740 1 -- 172.21.15.90:0/819114376 shutdown_connections 2024-04-25T11:37:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.046+0000 7f92fe9a6740 1 -- 172.21.15.90:0/819114376 wait complete. 2024-04-25T11:37:28.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.046+0000 7f92fe9a6740 20 client.25658 trim_cache size 0 max 16384 2024-04-25T11:37:28.036 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:37:28.036 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-25T11:37:28.036 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "03145e2b-e022-4391-b671-efcb67331aa4", 2024-04-25T11:37:28.036 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-25T11:37:28.036 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:37:28.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.047+0000 7f92f7a15640 1 -- 172.21.15.90:0/4135700000 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f92dc0afd60 msgr2=0x7f92dc0b2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:28.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.048+0000 7f92f7a15640 1 --2- 172.21.15.90:0/4135700000 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f92dc0afd60 0x7f92dc0b2270 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x563bff08c110 tx=0x7f92ec005f90 comp rx=0 tx=0).stop 2024-04-25T11:37:28.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.048+0000 7f92f7a15640 1 -- 172.21.15.90:0/4135700000 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92f010d380 msgr2=0x7f92f01a86b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:28.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.048+0000 7f92f7a15640 1 --2- 172.21.15.90:0/4135700000 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92f010d380 0x7f92f01a86b0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f92e8009430 tx=0x7f92e80317c0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.048+0000 7f92f7a15640 1 -- 172.21.15.90:0/4135700000 shutdown_connections 2024-04-25T11:37:28.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.048+0000 7f92f7a15640 1 --2- 172.21.15.90:0/4135700000 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f92dc0afd60 0x7f92dc0b2270 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.048+0000 7f92f7a15640 1 --2- 172.21.15.90:0/4135700000 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92f010d380 0x7f92f01a86b0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.048+0000 7f92f7a15640 1 --2- 172.21.15.90:0/4135700000 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f92f0108a20 0x7f92f019db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.049+0000 7f92f7a15640 1 --2- 172.21.15.90:0/4135700000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92f0102af0 0x7f92f019d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.049+0000 7f92f7a15640 1 -- 172.21.15.90:0/4135700000 >> 172.21.15.90:0/4135700000 conn(0x7f92f0078100 msgr2=0x7f92f00ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:28.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.050+0000 7f92f7a15640 1 -- 172.21.15.90:0/4135700000 shutdown_connections 2024-04-25T11:37:28.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.050+0000 7f92f7a15640 1 -- 172.21.15.90:0/4135700000 wait complete. 2024-04-25T11:37:28.068 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:37:28.069 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:37:28.131 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:27 smithi090 ceph-mon[29575]: pgmap v74: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 348 MiB/s wr, 1.75k op/s 2024-04-25T11:37:28.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.285+0000 7f9b6a7e3640 1 -- 172.21.15.90:0/2671912063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b641072c0 msgr2=0x7f9b641076d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:28.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.285+0000 7f9b6a7e3640 1 --2- 172.21.15.90:0/2671912063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b641072c0 0x7f9b641076d0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c009ca0 tx=0x7f9b5c02a960 comp rx=0 tx=0).stop 2024-04-25T11:37:28.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.285+0000 7f9b6a7e3640 1 -- 172.21.15.90:0/2671912063 shutdown_connections 2024-04-25T11:37:28.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.285+0000 7f9b6a7e3640 1 --2- 172.21.15.90:0/2671912063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b641072c0 0x7f9b641076d0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.285+0000 7f9b6a7e3640 1 --2- 172.21.15.90:0/2671912063 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9b64108e30 0x7f9b641020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.285+0000 7f9b6a7e3640 1 --2- 172.21.15.90:0/2671912063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b64108a20 0x7f9b64101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.285+0000 7f9b6a7e3640 1 -- 172.21.15.90:0/2671912063 >> 172.21.15.90:0/2671912063 conn(0x7f9b64077920 msgr2=0x7f9b64077d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:28.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.286+0000 7f9b6a7e3640 1 -- 172.21.15.90:0/2671912063 shutdown_connections 2024-04-25T11:37:28.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.286+0000 7f9b6a7e3640 1 -- 172.21.15.90:0/2671912063 wait complete. 2024-04-25T11:37:28.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.288+0000 7f9b6a7e3640 1 Processor -- start 2024-04-25T11:37:28.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.288+0000 7f9b6a7e3640 1 -- start start 2024-04-25T11:37:28.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.289+0000 7f9b6a7e3640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9b641072c0 0x7f9b641a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:28.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.289+0000 7f9b6a7e3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b64108a20 0x7f9b641a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:28.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.289+0000 7f9b6a7e3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b64108e30 0x7f9b641a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:28.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.289+0000 7f9b6a7e3640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6411a3e0 con 0x7f9b64108e30 2024-04-25T11:37:28.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.290+0000 7f9b6a7e3640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6411a260 con 0x7f9b641072c0 2024-04-25T11:37:28.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.290+0000 7f9b6a7e3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6411a560 con 0x7f9b64108a20 2024-04-25T11:37:28.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.290+0000 7f9b68d80640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b64108e30 0x7f9b641a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:28.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.290+0000 7f9b68d80640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b64108e30 0x7f9b641a5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55858/0 (socket says 172.21.15.90:55858) 2024-04-25T11:37:28.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.290+0000 7f9b68d80640 1 -- 172.21.15.90:0/4237189154 learned_addr learned my addr 172.21.15.90:0/4237189154 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:28.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.290+0000 7f9b63fff640 1 --2- 172.21.15.90:0/4237189154 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9b641072c0 0x7f9b641a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:28.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.290+0000 7f9b68d80640 1 -- 172.21.15.90:0/4237189154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b64108a20 msgr2=0x7f9b641a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:28.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.290+0000 7f9b68d80640 1 --2- 172.21.15.90:0/4237189154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b64108a20 0x7f9b641a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.290+0000 7f9b68d80640 1 -- 172.21.15.90:0/4237189154 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9b641072c0 msgr2=0x7f9b641a0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:28.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.290+0000 7f9b68d80640 1 --2- 172.21.15.90:0/4237189154 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9b641072c0 0x7f9b641a0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.290+0000 7f9b68d80640 1 -- 172.21.15.90:0/4237189154 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b641a5ba0 con 0x7f9b64108e30 2024-04-25T11:37:28.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.291+0000 7f9b68d80640 1 --2- 172.21.15.90:0/4237189154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b64108e30 0x7f9b641a5560 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c005e30 tx=0x7f9b5c003f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:28.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.292+0000 7f9b617fa640 1 -- 172.21.15.90:0/4237189154 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c034030 con 0x7f9b64108e30 2024-04-25T11:37:28.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.292+0000 7f9b617fa640 1 -- 172.21.15.90:0/4237189154 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b5c02fe80 con 0x7f9b64108e30 2024-04-25T11:37:28.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.292+0000 7f9b6a7e3640 1 -- 172.21.15.90:0/4237189154 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b641a5e30 con 0x7f9b64108e30 2024-04-25T11:37:28.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.292+0000 7f9b617fa640 1 -- 172.21.15.90:0/4237189154 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c004660 con 0x7f9b64108e30 2024-04-25T11:37:28.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.292+0000 7f9b6a7e3640 1 -- 172.21.15.90:0/4237189154 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b641ad810 con 0x7f9b64108e30 2024-04-25T11:37:28.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.297+0000 7f9b7176f740 1 Processor -- start 2024-04-25T11:37:28.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.298+0000 7f9b7176f740 1 -- start start 2024-04-25T11:37:28.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.298+0000 7f9b617fa640 1 -- 172.21.15.90:0/4237189154 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f9b5c037020 con 0x7f9b64108e30 2024-04-25T11:37:28.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.299+0000 7f9b7176f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0595ca390 0x55f05996aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:28.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.299+0000 7f9b7176f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f0595a14d0 0x55f05996afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:28.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.299+0000 7f9b7176f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0595a96b0 0x55f0599778f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:28.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.299+0000 7f9b7176f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f0598d0d80 con 0x55f0595a96b0 2024-04-25T11:37:28.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.299+0000 7f9b7176f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f05988a8b0 con 0x55f0595a14d0 2024-04-25T11:37:28.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.299+0000 7f9b7176f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f0598ccd40 con 0x55f0595ca390 2024-04-25T11:37:28.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.299+0000 7f9b63fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f0595a14d0 0x55f05996afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:28.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.299+0000 7f9b63fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f0595a14d0 0x55f05996afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:46376/0 (socket says 172.21.15.90:46376) 2024-04-25T11:37:28.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.299+0000 7f9b63fff640 1 -- 172.21.15.90:0/516105060 learned_addr learned my addr 172.21.15.90:0/516105060 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:28.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.299+0000 7f9b637fe640 1 --2- 172.21.15.90:0/516105060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0595ca390 0x55f05996aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:28.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.300+0000 7f9b63fff640 1 -- 172.21.15.90:0/516105060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0595ca390 msgr2=0x55f05996aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:28.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.300+0000 7f9b63fff640 1 --2- 172.21.15.90:0/516105060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0595ca390 0x55f05996aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.300+0000 7f9b63fff640 1 -- 172.21.15.90:0/516105060 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0595a96b0 msgr2=0x55f0599778f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:28.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.300+0000 7f9b63fff640 1 --2- 172.21.15.90:0/516105060 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0595a96b0 0x55f0599778f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.300+0000 7f9b63fff640 1 -- 172.21.15.90:0/516105060 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f05985ba40 con 0x55f0595a14d0 2024-04-25T11:37:28.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.300+0000 7f9b617fa640 1 --2- 172.21.15.90:0/4237189154 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9b4c0afd60 0x7f9b4c0b2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:28.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.301+0000 7f9b63fff640 1 --2- 172.21.15.90:0/516105060 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f0595a14d0 0x55f05996afa0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9b5400b7f0 tx=0x7f9b5400bcb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e35afde6c83e1379 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:28.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.301+0000 7f9b617fa640 1 -- 172.21.15.90:0/4237189154 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f9b5c0ffd20 con 0x7f9b64108e30 2024-04-25T11:37:28.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.301+0000 7f9b4a7fc640 1 -- 172.21.15.90:0/516105060 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5400d950 con 0x55f0595a14d0 2024-04-25T11:37:28.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.301+0000 7f9b637fe640 1 --2- 172.21.15.90:0/4237189154 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9b4c0afd60 0x7f9b4c0b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:28.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.301+0000 7f9b4a7fc640 1 -- 172.21.15.90:0/516105060 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b54018070 con 0x55f0595a14d0 2024-04-25T11:37:28.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.301+0000 7f9b4a7fc640 1 -- 172.21.15.90:0/516105060 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b540123e0 con 0x55f0595a14d0 2024-04-25T11:37:28.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.303+0000 7f9b637fe640 1 --2- 172.21.15.90:0/4237189154 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9b4c0afd60 0x7f9b4c0b2270 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x55f059967bd0 tx=0x7f9b580096a0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:28.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.303+0000 7f9b7176f740 1 -- 172.21.15.90:0/516105060 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f0595a14d0 msgr2=0x55f05996afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:28.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.303+0000 7f9b7176f740 1 --2- 172.21.15.90:0/516105060 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f0595a14d0 0x55f05996afa0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9b5400b7f0 tx=0x7f9b5400bcb0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.303+0000 7f9b7176f740 1 -- 172.21.15.90:0/516105060 shutdown_connections 2024-04-25T11:37:28.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.303+0000 7f9b7176f740 1 --2- 172.21.15.90:0/516105060 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0595a96b0 0x55f0599778f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.303+0000 7f9b7176f740 1 --2- 172.21.15.90:0/516105060 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f0595a14d0 0x55f05996afa0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.303+0000 7f9b7176f740 1 --2- 172.21.15.90:0/516105060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0595ca390 0x55f05996aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.303+0000 7f9b7176f740 1 -- 172.21.15.90:0/516105060 >> 172.21.15.90:0/516105060 conn(0x55f0595be4e0 msgr2=0x55f059967210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:28.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.303+0000 7f9b7176f740 1 -- 172.21.15.90:0/516105060 shutdown_connections 2024-04-25T11:37:28.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.303+0000 7f9b7176f740 1 -- 172.21.15.90:0/516105060 wait complete. 2024-04-25T11:37:28.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.304+0000 7f9b7176f740 1 -- start start 2024-04-25T11:37:28.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.305+0000 7f9b7176f740 1 Processor -- start 2024-04-25T11:37:28.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.305+0000 7f9b7176f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0595a14d0 0x55f059975640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:28.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.305+0000 7f9b7176f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f0595a96b0 0x55f059975b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:28.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.305+0000 7f9b7176f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0595ca390 0x55f05997b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:28.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.305+0000 7f9b7176f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f059977fe0 con 0x55f0595a14d0 2024-04-25T11:37:28.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.305+0000 7f9b7176f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f059857d10 con 0x55f0595a96b0 2024-04-25T11:37:28.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.305+0000 7f9b7176f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f059978160 con 0x55f0595ca390 2024-04-25T11:37:28.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.305+0000 7f9b7176f740 10 client.0 _ll_register_callbacks cb 0x55f059884210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:37:28.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.306+0000 7f9b63fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0595a14d0 0x55f059975640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:28.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.306+0000 7f9b63fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0595a14d0 0x55f059975640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55888/0 (socket says 172.21.15.90:55888) 2024-04-25T11:37:28.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.306+0000 7f9b63fff640 1 -- 172.21.15.90:0/2893388601 learned_addr learned my addr 172.21.15.90:0/2893388601 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:28.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.306+0000 7f9b63fff640 1 -- 172.21.15.90:0/2893388601 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0595ca390 msgr2=0x55f05997b4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:37:28.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.306+0000 7f9b63fff640 1 --2- 172.21.15.90:0/2893388601 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0595ca390 0x55f05997b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.306+0000 7f9b63fff640 1 -- 172.21.15.90:0/2893388601 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f0595a96b0 msgr2=0x55f059975b80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:37:28.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.306+0000 7f9b63fff640 1 --2- 172.21.15.90:0/2893388601 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f0595a96b0 0x55f059975b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.306+0000 7f9b63fff640 1 -- 172.21.15.90:0/2893388601 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0596ac7d0 con 0x55f0595a14d0 2024-04-25T11:37:28.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.307+0000 7f9b63fff640 1 --2- 172.21.15.90:0/2893388601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0595a14d0 0x55f059975640 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f9b5400b1f0 tx=0x7f9b54014ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10a16cc96954e175 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:28.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.307+0000 7f9b437fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:37:28.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.307+0000 7f9b437fe640 1 -- 172.21.15.90:0/2893388601 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5400d950 con 0x55f0595a14d0 2024-04-25T11:37:28.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.307+0000 7f9b7176f740 1 -- 172.21.15.90:0/2893388601 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b54018070 con 0x55f0595a14d0 2024-04-25T11:37:28.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.307+0000 7f9b437fe640 1 -- 172.21.15.90:0/2893388601 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f9b5400daf0 con 0x55f0595a14d0 2024-04-25T11:37:28.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.307+0000 7f9b437fe640 1 -- 172.21.15.90:0/2893388601 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9b5400dd30 con 0x55f0595a14d0 2024-04-25T11:37:28.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.307+0000 7f9b7176f740 10 client.16344 fetch_fsmap learned FSMap version 27 2024-04-25T11:37:28.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.307+0000 7f9b7176f740 1 -- 172.21.15.90:0/2893388601 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f0595e23f0 con 0x55f0595a14d0 2024-04-25T11:37:28.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.308+0000 7f9b437fe640 1 -- 172.21.15.90:0/2893388601 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b54018070 con 0x55f0595a14d0 2024-04-25T11:37:28.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.308+0000 7f9b437fe640 1 -- 172.21.15.90:0/2893388601 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f9b54024420 con 0x55f0595a14d0 2024-04-25T11:37:28.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.308+0000 7f9b7176f740 10 client.16344 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:37:28.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.308+0000 7f9b7176f740 10 client.16344 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:37:28.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.308+0000 7f9b7176f740 20 client.16344 populate_metadata read hostname 'smithi090' 2024-04-25T11:37:28.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.308+0000 7f9b7176f740 1 --2- 172.21.15.90:0/2893388601 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f059588f60 0x55f059987650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:28.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.308+0000 7f9b7176f740 4 client.16344 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:37:28.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.309+0000 7f9b7176f740 1 -- 172.21.15.90:0/2893388601 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f0598de990 con 0x55f059588f60 2024-04-25T11:37:28.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.309+0000 7f9b637fe640 1 --2- 172.21.15.90:0/2893388601 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f059588f60 0x55f059987650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:28.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.311+0000 7f9b437fe640 10 client.16344 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:28.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.311+0000 7f9b637fe640 1 --2- 172.21.15.90:0/2893388601 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f059588f60 0x55f059987650 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=806a43927f4dcf1d server_cookie=179e9ccafe158d8f in_seq=0 out_seq=0 2024-04-25T11:37:28.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.313+0000 7f9b437fe640 1 -- 172.21.15.90:0/2893388601 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f0598de990 con 0x55f059588f60 2024-04-25T11:37:28.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.313+0000 7f9b437fe640 10 client.16344 handle_command_reply: tid=0 2024-04-25T11:37:28.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.313+0000 7f9b7176f740 1 client.16344 shutdown 2024-04-25T11:37:28.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.313+0000 7f9b7176f740 1 -- 172.21.15.90:0/2893388601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0595a14d0 msgr2=0x55f059975640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:28.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.313+0000 7f9b7176f740 1 --2- 172.21.15.90:0/2893388601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0595a14d0 0x55f059975640 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f9b5400b1f0 tx=0x7f9b54014ee0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.313+0000 7f9b7176f740 1 -- 172.21.15.90:0/2893388601 shutdown_connections 2024-04-25T11:37:28.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.313+0000 7f9b7176f740 1 --2- 172.21.15.90:0/2893388601 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f059588f60 0x55f059987650 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.313+0000 7f9b7176f740 1 --2- 172.21.15.90:0/2893388601 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0595ca390 0x55f05997b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.313+0000 7f9b7176f740 1 --2- 172.21.15.90:0/2893388601 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f0595a96b0 0x55f059975b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.313+0000 7f9b7176f740 1 --2- 172.21.15.90:0/2893388601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0595a14d0 0x55f059975640 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.313+0000 7f9b7176f740 1 -- 172.21.15.90:0/2893388601 >> 172.21.15.90:0/2893388601 conn(0x55f0595be4e0 msgr2=0x55f05996c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:28.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.313+0000 7f9b437fe640 0 client.16344 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:28.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.314+0000 7f9b7176f740 1 -- 172.21.15.90:0/2893388601 shutdown_connections 2024-04-25T11:37:28.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.314+0000 7f9b7176f740 1 -- 172.21.15.90:0/2893388601 wait complete. 2024-04-25T11:37:28.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.314+0000 7f9b7176f740 20 client.16344 trim_cache size 0 max 16384 2024-04-25T11:37:28.303 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:27 smithi138 ceph-mon[32328]: pgmap v74: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 348 MiB/s wr, 1.75k op/s 2024-04-25T11:37:28.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.343+0000 7f9b7176f740 1 Processor -- start 2024-04-25T11:37:28.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.343+0000 7f9b7176f740 1 -- start start 2024-04-25T11:37:28.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.343+0000 7f9b7176f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f0595a14d0 0x55f059967280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:28.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.343+0000 7f9b7176f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0595a96b0 0x55f059970630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:28.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.343+0000 7f9b7176f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0595ca390 0x55f059974410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:28.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.343+0000 7f9b7176f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9b641b6240 con 0x55f0595a96b0 2024-04-25T11:37:28.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.343+0000 7f9b7176f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9b641b7db0 con 0x55f0595a14d0 2024-04-25T11:37:28.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.343+0000 7f9b7176f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9b641b6910 con 0x55f0595ca390 2024-04-25T11:37:28.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.343+0000 7f9b637fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0595a96b0 0x55f059970630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:28.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.343+0000 7f9b63fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f0595a14d0 0x55f059967280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:28.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.343+0000 7f9b637fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0595a96b0 0x55f059970630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55898/0 (socket says 172.21.15.90:55898) 2024-04-25T11:37:28.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.343+0000 7f9b637fe640 1 -- 172.21.15.90:0/57504407 learned_addr learned my addr 172.21.15.90:0/57504407 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:28.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.344+0000 7f9b637fe640 1 -- 172.21.15.90:0/57504407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0595ca390 msgr2=0x55f059974410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:28.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.344+0000 7f9b637fe640 1 --2- 172.21.15.90:0/57504407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0595ca390 0x55f059974410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.344+0000 7f9b637fe640 1 -- 172.21.15.90:0/57504407 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f0595a14d0 msgr2=0x55f059967280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:28.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.344+0000 7f9b637fe640 1 --2- 172.21.15.90:0/57504407 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f0595a14d0 0x55f059967280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.344+0000 7f9b637fe640 1 -- 172.21.15.90:0/57504407 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0595a20f0 con 0x55f0595a96b0 2024-04-25T11:37:28.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.344+0000 7f9b63fff640 1 --2- 172.21.15.90:0/57504407 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f0595a14d0 0x55f059967280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:37:28.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.344+0000 7f9b637fe640 1 --2- 172.21.15.90:0/57504407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0595a96b0 0x55f059970630 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f9b580104a0 tx=0x7f9b58015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d62a8259a9012830 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:28.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.344+0000 7f9b49ffb640 1 -- 172.21.15.90:0/57504407 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5801c7c0 con 0x55f0595a96b0 2024-04-25T11:37:28.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.344+0000 7f9b49ffb640 1 -- 172.21.15.90:0/57504407 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b5801ce10 con 0x55f0595a96b0 2024-04-25T11:37:28.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.344+0000 7f9b49ffb640 1 -- 172.21.15.90:0/57504407 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5801b3e0 con 0x55f0595a96b0 2024-04-25T11:37:28.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.345+0000 7f9b7176f740 1 -- 172.21.15.90:0/57504407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0595a96b0 msgr2=0x55f059970630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:28.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.345+0000 7f9b7176f740 1 --2- 172.21.15.90:0/57504407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0595a96b0 0x55f059970630 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f9b580104a0 tx=0x7f9b58015000 comp rx=0 tx=0).stop 2024-04-25T11:37:28.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.345+0000 7f9b7176f740 1 -- 172.21.15.90:0/57504407 shutdown_connections 2024-04-25T11:37:28.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.345+0000 7f9b7176f740 1 --2- 172.21.15.90:0/57504407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0595ca390 0x55f059974410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.345+0000 7f9b7176f740 1 --2- 172.21.15.90:0/57504407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0595a96b0 0x55f059970630 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.345+0000 7f9b7176f740 1 --2- 172.21.15.90:0/57504407 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f0595a14d0 0x55f059967280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.345+0000 7f9b7176f740 1 -- 172.21.15.90:0/57504407 >> 172.21.15.90:0/57504407 conn(0x55f0595be4e0 msgr2=0x55f059973d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:28.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.345+0000 7f9b7176f740 1 -- 172.21.15.90:0/57504407 shutdown_connections 2024-04-25T11:37:28.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.345+0000 7f9b7176f740 1 -- 172.21.15.90:0/57504407 wait complete. 2024-04-25T11:37:28.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.346+0000 7f9b7176f740 1 -- start start 2024-04-25T11:37:28.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.346+0000 7f9b7176f740 1 Processor -- start 2024-04-25T11:37:28.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.346+0000 7f9b7176f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0595a14d0 0x55f05997c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:28.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.346+0000 7f9b7176f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0595a96b0 0x55f05997c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:28.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.346+0000 7f9b7176f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f0595ca390 0x55f0599ed880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:28.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.346+0000 7f9b7176f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9b641b76b0 con 0x55f0595a96b0 2024-04-25T11:37:28.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.346+0000 7f9b7176f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9b641b6fe0 con 0x55f0595ca390 2024-04-25T11:37:28.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.346+0000 7f9b7176f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9b641b4220 con 0x55f0595a14d0 2024-04-25T11:37:28.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.346+0000 7f9b7176f740 10 client.0 _ll_register_callbacks cb 0x7f9b580095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:37:28.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.346+0000 7f9b637fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0595a96b0 0x55f05997c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:28.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.347+0000 7f9b637fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0595a96b0 0x55f05997c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55904/0 (socket says 172.21.15.90:55904) 2024-04-25T11:37:28.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.347+0000 7f9b637fe640 1 -- 172.21.15.90:0/383157098 learned_addr learned my addr 172.21.15.90:0/383157098 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:28.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.347+0000 7f9b68d80640 1 --2- 172.21.15.90:0/383157098 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f0595ca390 0x55f0599ed880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:28.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.347+0000 7f9b63fff640 1 --2- 172.21.15.90:0/383157098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0595a14d0 0x55f05997c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:28.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.347+0000 7f9b637fe640 1 -- 172.21.15.90:0/383157098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0595a14d0 msgr2=0x55f05997c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:28.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.347+0000 7f9b637fe640 1 --2- 172.21.15.90:0/383157098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0595a14d0 0x55f05997c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.347+0000 7f9b637fe640 1 -- 172.21.15.90:0/383157098 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f0595ca390 msgr2=0x55f0599ed880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:28.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.347+0000 7f9b637fe640 1 --2- 172.21.15.90:0/383157098 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f0595ca390 0x55f0599ed880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.347+0000 7f9b637fe640 1 -- 172.21.15.90:0/383157098 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0598914b0 con 0x55f0595a96b0 2024-04-25T11:37:28.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.347+0000 7f9b68d80640 1 --2- 172.21.15.90:0/383157098 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f0595ca390 0x55f0599ed880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:37:28.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.347+0000 7f9b637fe640 1 --2- 172.21.15.90:0/383157098 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0595a96b0 0x55f05997c900 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f9b58005e00 tx=0x7f9b58010d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=255984c91ad3e6b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:28.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.347+0000 7f9b437fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:37:28.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.347+0000 7f9b437fe640 1 -- 172.21.15.90:0/383157098 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b58025aa0 con 0x55f0595a96b0 2024-04-25T11:37:28.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.347+0000 7f9b437fe640 1 -- 172.21.15.90:0/383157098 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f9b58025250 con 0x55f0595a96b0 2024-04-25T11:37:28.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.347+0000 7f9b7176f740 1 -- 172.21.15.90:0/383157098 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b5801ce10 con 0x55f0595a96b0 2024-04-25T11:37:28.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.348+0000 7f9b437fe640 1 -- 172.21.15.90:0/383157098 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9b58025490 con 0x55f0595a96b0 2024-04-25T11:37:28.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.348+0000 7f9b7176f740 10 client.16350 fetch_fsmap learned FSMap version 27 2024-04-25T11:37:28.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.348+0000 7f9b7176f740 1 -- 172.21.15.90:0/383157098 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f0599eed80 con 0x55f0595a96b0 2024-04-25T11:37:28.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.348+0000 7f9b437fe640 1 -- 172.21.15.90:0/383157098 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b5801ce10 con 0x55f0595a96b0 2024-04-25T11:37:28.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.348+0000 7f9b437fe640 1 -- 172.21.15.90:0/383157098 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f9b58025ce0 con 0x55f0595a96b0 2024-04-25T11:37:28.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.348+0000 7f9b7176f740 10 client.16350 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:37:28.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.348+0000 7f9b7176f740 10 client.16350 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:37:28.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.348+0000 7f9b7176f740 20 client.16350 populate_metadata read hostname 'smithi090' 2024-04-25T11:37:28.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.348+0000 7f9b7176f740 1 --2- 172.21.15.90:0/383157098 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f05955e3a0 0x55f0599f14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:28.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.348+0000 7f9b7176f740 4 client.16350 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:37:28.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.348+0000 7f9b7176f740 1 -- 172.21.15.90:0/383157098 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f0598d8800 con 0x55f05955e3a0 2024-04-25T11:37:28.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.348+0000 7f9b63fff640 1 --2- 172.21.15.90:0/383157098 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f05955e3a0 0x55f0599f14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:28.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.349+0000 7f9b63fff640 1 --2- 172.21.15.90:0/383157098 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f05955e3a0 0x55f0599f14b0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8429c89395e1462 server_cookie=7e2a07b47a196155 in_seq=0 out_seq=0 2024-04-25T11:37:28.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.349+0000 7f9b437fe640 10 client.16350 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:28.341 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:27 smithi156 ceph-mon[33045]: pgmap v74: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 348 MiB/s wr, 1.75k op/s 2024-04-25T11:37:28.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.351+0000 7f9b437fe640 1 -- 172.21.15.90:0/383157098 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55f0598d8800 con 0x55f05955e3a0 2024-04-25T11:37:28.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.351+0000 7f9b437fe640 10 client.16350 handle_command_reply: tid=0 2024-04-25T11:37:28.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.351+0000 7f9b7176f740 1 client.16350 shutdown 2024-04-25T11:37:28.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.351+0000 7f9b7176f740 1 -- 172.21.15.90:0/383157098 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0595a96b0 msgr2=0x55f05997c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:28.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.351+0000 7f9b7176f740 1 --2- 172.21.15.90:0/383157098 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0595a96b0 0x55f05997c900 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f9b58005e00 tx=0x7f9b58010d30 comp rx=0 tx=0).stop 2024-04-25T11:37:28.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.351+0000 7f9b7176f740 1 -- 172.21.15.90:0/383157098 shutdown_connections 2024-04-25T11:37:28.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.351+0000 7f9b7176f740 1 --2- 172.21.15.90:0/383157098 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f05955e3a0 0x55f0599f14b0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.351+0000 7f9b7176f740 1 --2- 172.21.15.90:0/383157098 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f0595ca390 0x55f0599ed880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.351+0000 7f9b437fe640 0 client.16350 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:28.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.351+0000 7f9b7176f740 1 --2- 172.21.15.90:0/383157098 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0595a96b0 0x55f05997c900 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.351+0000 7f9b7176f740 1 --2- 172.21.15.90:0/383157098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0595a14d0 0x55f05997c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.351+0000 7f9b7176f740 1 -- 172.21.15.90:0/383157098 >> 172.21.15.90:0/383157098 conn(0x55f0595be4e0 msgr2=0x55f0599769d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:28.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.351+0000 7f9b7176f740 1 -- 172.21.15.90:0/383157098 shutdown_connections 2024-04-25T11:37:28.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.352+0000 7f9b7176f740 1 -- 172.21.15.90:0/383157098 wait complete. 2024-04-25T11:37:28.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.352+0000 7f9b7176f740 20 client.16350 trim_cache size 0 max 16384 2024-04-25T11:37:28.343 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:37:28.343 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "scrub active (2 inodes in the stack)", 2024-04-25T11:37:28.343 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-25T11:37:28.343 INFO:teuthology.orchestra.run.smithi090.stdout: "03145e2b-e022-4391-b671-efcb67331aa4": { 2024-04-25T11:37:28.343 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-25T11:37:28.343 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "03145e2b-e022-4391-b671-efcb67331aa4", 2024-04-25T11:37:28.344 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-25T11:37:28.344 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:37:28.344 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:37:28.344 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:37:28.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.352+0000 7f9b6a7e3640 1 -- 172.21.15.90:0/4237189154 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9b4c0afd60 msgr2=0x7f9b4c0b2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:28.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.352+0000 7f9b6a7e3640 1 --2- 172.21.15.90:0/4237189154 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9b4c0afd60 0x7f9b4c0b2270 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x55f059967bd0 tx=0x7f9b580096a0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.352+0000 7f9b6a7e3640 1 -- 172.21.15.90:0/4237189154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b64108e30 msgr2=0x7f9b641a5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:28.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.352+0000 7f9b6a7e3640 1 --2- 172.21.15.90:0/4237189154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b64108e30 0x7f9b641a5560 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c005e30 tx=0x7f9b5c003f80 comp rx=0 tx=0).stop 2024-04-25T11:37:28.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.352+0000 7f9b6a7e3640 1 -- 172.21.15.90:0/4237189154 shutdown_connections 2024-04-25T11:37:28.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.352+0000 7f9b6a7e3640 1 --2- 172.21.15.90:0/4237189154 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9b4c0afd60 0x7f9b4c0b2270 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.352+0000 7f9b6a7e3640 1 --2- 172.21.15.90:0/4237189154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b64108e30 0x7f9b641a5560 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.352+0000 7f9b6a7e3640 1 --2- 172.21.15.90:0/4237189154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b64108a20 0x7f9b641a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.352+0000 7f9b6a7e3640 1 --2- 172.21.15.90:0/4237189154 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9b641072c0 0x7f9b641a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.352+0000 7f9b6a7e3640 1 -- 172.21.15.90:0/4237189154 >> 172.21.15.90:0/4237189154 conn(0x7f9b64077920 msgr2=0x7f9b640ffa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:28.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.352+0000 7f9b6a7e3640 1 -- 172.21.15.90:0/4237189154 shutdown_connections 2024-04-25T11:37:28.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:28.352+0000 7f9b6a7e3640 1 -- 172.21.15.90:0/4237189154 wait complete. 2024-04-25T11:37:28.353 INFO:tasks.cephfs.filesystem:scrub status for tag:03145e2b-e022-4391-b671-efcb67331aa4 - {'path': '/', 'tag': '03145e2b-e022-4391-b671-efcb67331aa4', 'options': 'recursive,force'} 2024-04-25T11:37:29.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:28 smithi090 ceph-mon[29575]: scrub queued for path: / 2024-04-25T11:37:29.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:28 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:37:29.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:28 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:37:29.237 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:28 smithi156 ceph-mon[33045]: scrub queued for path: / 2024-04-25T11:37:29.237 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:28 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:37:29.237 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:28 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:37:29.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:28 smithi138 ceph-mon[32328]: scrub queued for path: / 2024-04-25T11:37:29.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:28 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:37:29.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:28 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:37:30.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:30 smithi138 ceph-mon[32328]: pgmap v75: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 318 MiB/s wr, 1.47k op/s 2024-04-25T11:37:30.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:30 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:37:30.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:30 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:37:30.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:30 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:37:30.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:30 smithi156 ceph-mon[33045]: pgmap v75: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 318 MiB/s wr, 1.47k op/s 2024-04-25T11:37:30.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:30 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:37:30.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:30 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:37:30.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:30 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:37:30.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:30 smithi090 ceph-mon[29575]: pgmap v75: 97 pgs: 97 active+clean; 13 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 318 MiB/s wr, 1.47k op/s 2024-04-25T11:37:30.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:30 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:37:30.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:30 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:37:30.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:30 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:37:31.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:31 smithi156 ceph-mon[33045]: pgmap v76: 97 pgs: 97 active+clean; 16 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 611 MiB/s wr, 2.01k op/s 2024-04-25T11:37:31.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:31 smithi090 ceph-mon[29575]: pgmap v76: 97 pgs: 97 active+clean; 16 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 611 MiB/s wr, 2.01k op/s 2024-04-25T11:37:31.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:31 smithi138 ceph-mon[32328]: pgmap v76: 97 pgs: 97 active+clean; 16 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 611 MiB/s wr, 2.01k op/s 2024-04-25T11:37:33.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:33 smithi090 ceph-mon[29575]: pgmap v77: 97 pgs: 97 active+clean; 16 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 480 MiB/s wr, 829 op/s 2024-04-25T11:37:33.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:33 smithi138 ceph-mon[32328]: pgmap v77: 97 pgs: 97 active+clean; 16 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 480 MiB/s wr, 829 op/s 2024-04-25T11:37:33.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:33 smithi156 ceph-mon[33045]: pgmap v77: 97 pgs: 97 active+clean; 16 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 480 MiB/s wr, 829 op/s 2024-04-25T11:37:34.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.985+0000 7f56993b2640 1 -- 172.21.15.90:0/1396816712 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f569410a040 msgr2=0x7f5694103c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:34.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.985+0000 7f56993b2640 1 --2- 172.21.15.90:0/1396816712 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f569410a040 0x7f5694103c80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5688009900 tx=0x7f568802a410 comp rx=0 tx=0).stop 2024-04-25T11:37:34.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.985+0000 7f56993b2640 1 -- 172.21.15.90:0/1396816712 shutdown_connections 2024-04-25T11:37:34.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.985+0000 7f56993b2640 1 --2- 172.21.15.90:0/1396816712 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56941042f0 0x7f56941106c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:34.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.985+0000 7f56993b2640 1 --2- 172.21.15.90:0/1396816712 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f569410a040 0x7f5694103c80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:34.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.985+0000 7f56993b2640 1 --2- 172.21.15.90:0/1396816712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5694109ba0 0x7f5694103740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:34.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.985+0000 7f56993b2640 1 -- 172.21.15.90:0/1396816712 >> 172.21.15.90:0/1396816712 conn(0x7f56940792a0 msgr2=0x7f56940796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:34.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.985+0000 7f56993b2640 1 -- 172.21.15.90:0/1396816712 shutdown_connections 2024-04-25T11:37:34.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.986+0000 7f56993b2640 1 -- 172.21.15.90:0/1396816712 wait complete. 2024-04-25T11:37:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.987+0000 7f56993b2640 1 Processor -- start 2024-04-25T11:37:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.987+0000 7f56993b2640 1 -- start start 2024-04-25T11:37:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.987+0000 7f56993b2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56941042f0 0x7f56941a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.987+0000 7f56993b2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5694109ba0 0x7f56941a0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.987+0000 7f56993b2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f569410a040 0x7f569419a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.987+0000 7f56993b2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5694113d90 con 0x7f5694109ba0 2024-04-25T11:37:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.987+0000 7f56993b2640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5694113c10 con 0x7f569410a040 2024-04-25T11:37:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.987+0000 7f56993b2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5694113f10 con 0x7f56941042f0 2024-04-25T11:37:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.987+0000 7f56927fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5694109ba0 0x7f56941a0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.987+0000 7f56927fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5694109ba0 0x7f56941a0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54768/0 (socket says 172.21.15.90:54768) 2024-04-25T11:37:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.987+0000 7f56927fc640 1 -- 172.21.15.90:0/2447470851 learned_addr learned my addr 172.21.15.90:0/2447470851 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.987+0000 7f56937fe640 1 --2- 172.21.15.90:0/2447470851 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f569410a040 0x7f569419a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.988+0000 7f5692ffd640 1 --2- 172.21.15.90:0/2447470851 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56941042f0 0x7f56941a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.988+0000 7f56937fe640 1 -- 172.21.15.90:0/2447470851 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56941042f0 msgr2=0x7f56941a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.988+0000 7f56937fe640 1 --2- 172.21.15.90:0/2447470851 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56941042f0 0x7f56941a04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.988+0000 7f56937fe640 1 -- 172.21.15.90:0/2447470851 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5694109ba0 msgr2=0x7f56941a0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.988+0000 7f56937fe640 1 --2- 172.21.15.90:0/2447470851 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5694109ba0 0x7f56941a0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.988+0000 7f56937fe640 1 -- 172.21.15.90:0/2447470851 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f569419ad80 con 0x7f569410a040 2024-04-25T11:37:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.988+0000 7f5692ffd640 1 --2- 172.21.15.90:0/2447470851 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56941042f0 0x7f56941a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:37:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.988+0000 7f56937fe640 1 --2- 172.21.15.90:0/2447470851 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f569410a040 0x7f569419a650 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f568c009fd0 tx=0x7f568c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.988+0000 7f567bfff640 1 -- 172.21.15.90:0/2447470851 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568c00ce20 con 0x7f569410a040 2024-04-25T11:37:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.988+0000 7f567bfff640 1 -- 172.21.15.90:0/2447470851 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f568c005020 con 0x7f569410a040 2024-04-25T11:37:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.988+0000 7f567bfff640 1 -- 172.21.15.90:0/2447470851 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568c010690 con 0x7f569410a040 2024-04-25T11:37:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.989+0000 7f56993b2640 1 -- 172.21.15.90:0/2447470851 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f569419b010 con 0x7f569410a040 2024-04-25T11:37:34.978 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.989+0000 7f56993b2640 1 -- 172.21.15.90:0/2447470851 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56941a7270 con 0x7f569410a040 2024-04-25T11:37:34.980 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.991+0000 7f567bfff640 1 -- 172.21.15.90:0/2447470851 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f568c010830 con 0x7f569410a040 2024-04-25T11:37:34.980 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.993+0000 7f56a0263740 1 Processor -- start 2024-04-25T11:37:34.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.993+0000 7f56a0263740 1 -- start start 2024-04-25T11:37:34.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.993+0000 7f567bfff640 1 --2- 172.21.15.90:0/2447470851 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f567c0afcd0 0x7f567c0b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:34.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.994+0000 7f56927fc640 1 --2- 172.21.15.90:0/2447470851 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f567c0afcd0 0x7f567c0b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:34.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.994+0000 7f56a0263740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597aef051b0 0x5597af2b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:34.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.994+0000 7f56a0263740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aeedc2f0 0x5597af2b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:34.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.994+0000 7f56a0263740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597aeee44d0 0x5597af2b2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:34.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.994+0000 7f56a0263740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5597af20b460 con 0x5597aef051b0 2024-04-25T11:37:34.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.994+0000 7f5692ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597aef051b0 0x5597af2b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:34.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.994+0000 7f5692ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597aef051b0 0x5597af2b3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54776/0 (socket says 172.21.15.90:54776) 2024-04-25T11:37:34.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.994+0000 7f5692ffd640 1 -- 172.21.15.90:0/1145071372 learned_addr learned my addr 172.21.15.90:0/1145071372 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:34.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.994+0000 7f567bfff640 1 -- 172.21.15.90:0/2447470851 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f568c0e0710 con 0x7f569410a040 2024-04-25T11:37:34.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.995+0000 7f56a0263740 1 -- 172.21.15.90:0/1145071372 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5597af1c5590 con 0x5597aeee44d0 2024-04-25T11:37:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.995+0000 7f56a0263740 1 -- 172.21.15.90:0/1145071372 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5597af207420 con 0x5597aeedc2f0 2024-04-25T11:37:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.995+0000 7f56927fc640 1 --2- 172.21.15.90:0/2447470851 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f567c0afcd0 0x7f567c0b21e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f569419be80 tx=0x7f5688009560 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.995+0000 7f56927fc640 1 --2- 172.21.15.90:0/1145071372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aeedc2f0 0x5597af2b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.995+0000 7f56937fe640 1 --2- 172.21.15.90:0/1145071372 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597aeee44d0 0x5597af2b2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.996+0000 7f56927fc640 1 -- 172.21.15.90:0/1145071372 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597aeee44d0 msgr2=0x5597af2b2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.996+0000 7f56927fc640 1 --2- 172.21.15.90:0/1145071372 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597aeee44d0 0x5597af2b2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.996+0000 7f56927fc640 1 -- 172.21.15.90:0/1145071372 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597aef051b0 msgr2=0x5597af2b3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.996+0000 7f56927fc640 1 --2- 172.21.15.90:0/1145071372 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597aef051b0 0x5597af2b3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.996+0000 7f56927fc640 1 -- 172.21.15.90:0/1145071372 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597af1d20d0 con 0x5597aeedc2f0 2024-04-25T11:37:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.997+0000 7f56927fc640 1 --2- 172.21.15.90:0/1145071372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aeedc2f0 0x5597af2b40a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5688031440 tx=0x7f568803d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40ff06fe4a4d48b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.998+0000 7f56797fa640 1 -- 172.21.15.90:0/1145071372 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5688035040 con 0x5597aeedc2f0 2024-04-25T11:37:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.998+0000 7f56797fa640 1 -- 172.21.15.90:0/1145071372 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f568804e070 con 0x5597aeedc2f0 2024-04-25T11:37:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:34.998+0000 7f56797fa640 1 -- 172.21.15.90:0/1145071372 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5688003e50 con 0x5597aeedc2f0 2024-04-25T11:37:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.000+0000 7f56a0263740 1 -- 172.21.15.90:0/1145071372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aeedc2f0 msgr2=0x5597af2b40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.000+0000 7f56a0263740 1 --2- 172.21.15.90:0/1145071372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aeedc2f0 0x5597af2b40a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5688031440 tx=0x7f568803d000 comp rx=0 tx=0).stop 2024-04-25T11:37:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.000+0000 7f56a0263740 1 -- 172.21.15.90:0/1145071372 shutdown_connections 2024-04-25T11:37:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.000+0000 7f56a0263740 1 --2- 172.21.15.90:0/1145071372 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597aeee44d0 0x5597af2b2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.000+0000 7f56a0263740 1 --2- 172.21.15.90:0/1145071372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aeedc2f0 0x5597af2b40a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.000+0000 7f56a0263740 1 --2- 172.21.15.90:0/1145071372 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597aef051b0 0x5597af2b3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.000+0000 7f56a0263740 1 -- 172.21.15.90:0/1145071372 >> 172.21.15.90:0/1145071372 conn(0x5597aeef9300 msgr2=0x5597af2a0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.000+0000 7f56a0263740 1 -- 172.21.15.90:0/1145071372 shutdown_connections 2024-04-25T11:37:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.000+0000 7f56a0263740 1 -- 172.21.15.90:0/1145071372 wait complete. 2024-04-25T11:37:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.001+0000 7f56a0263740 1 -- start start 2024-04-25T11:37:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.001+0000 7f56a0263740 1 Processor -- start 2024-04-25T11:37:34.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.002+0000 7f56a0263740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597aeedc2f0 0x5597af2bdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:34.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.002+0000 7f56a0263740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aeee44d0 0x5597af2be060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:34.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.002+0000 7f56a0263740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597aef051b0 0x5597af2b7f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:34.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.002+0000 7f56a0263740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5597af2b31d0 con 0x5597aef051b0 2024-04-25T11:37:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.002+0000 7f56937fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597aef051b0 0x5597af2b7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.002+0000 7f56937fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597aef051b0 0x5597af2b7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54790/0 (socket says 172.21.15.90:54790) 2024-04-25T11:37:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.002+0000 7f56937fe640 1 -- 172.21.15.90:0/3146403076 learned_addr learned my addr 172.21.15.90:0/3146403076 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.002+0000 7f56a0263740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5597af192b60 con 0x5597aeedc2f0 2024-04-25T11:37:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.003+0000 7f56a0263740 1 -- 172.21.15.90:0/3146403076 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5597af2b3350 con 0x5597aeee44d0 2024-04-25T11:37:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.003+0000 7f56a0263740 10 client.0 _ll_register_callbacks cb 0x5597af1b72b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:37:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.003+0000 7f56937fe640 1 -- 172.21.15.90:0/3146403076 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aeee44d0 msgr2=0x5597af2be060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.003+0000 7f56927fc640 1 --2- 172.21.15.90:0/3146403076 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aeee44d0 0x5597af2be060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.003+0000 7f56937fe640 1 --2- 172.21.15.90:0/3146403076 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aeee44d0 0x5597af2be060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.003+0000 7f56937fe640 1 -- 172.21.15.90:0/3146403076 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597aeedc2f0 msgr2=0x5597af2bdb20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:37:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.003+0000 7f56937fe640 1 --2- 172.21.15.90:0/3146403076 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597aeedc2f0 0x5597af2bdb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.003+0000 7f56937fe640 1 -- 172.21.15.90:0/3146403076 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597aef1d210 con 0x5597aef051b0 2024-04-25T11:37:34.992 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.004+0000 7f56937fe640 1 --2- 172.21.15.90:0/3146403076 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597aef051b0 0x5597af2b7f10 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f568c0a4880 tx=0x7f568c0a8260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f21132c8c5f9538a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:34.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.004+0000 7f5676ff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:37:34.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.004+0000 7f5676ff5640 1 -- 172.21.15.90:0/3146403076 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568c0a8850 con 0x5597aef051b0 2024-04-25T11:37:34.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.004+0000 7f5676ff5640 1 -- 172.21.15.90:0/3146403076 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f568c0a4ae0 con 0x5597aef051b0 2024-04-25T11:37:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.004+0000 7f56a0263740 1 -- 172.21.15.90:0/3146403076 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f568804e070 con 0x5597aef051b0 2024-04-25T11:37:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.005+0000 7f5676ff5640 1 -- 172.21.15.90:0/3146403076 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f568c0acc00 con 0x5597aef051b0 2024-04-25T11:37:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.005+0000 7f56a0263740 10 client.16362 fetch_fsmap learned FSMap version 27 2024-04-25T11:37:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.005+0000 7f56a0263740 1 -- 172.21.15.90:0/3146403076 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5597aefe9620 con 0x5597aef051b0 2024-04-25T11:37:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.005+0000 7f5676ff5640 1 -- 172.21.15.90:0/3146403076 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f568804e070 con 0x5597aef051b0 2024-04-25T11:37:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.005+0000 7f5676ff5640 1 -- 172.21.15.90:0/3146403076 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f568c088210 con 0x5597aef051b0 2024-04-25T11:37:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.005+0000 7f56a0263740 10 client.16362 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:37:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.005+0000 7f56a0263740 10 client.16362 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:37:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.005+0000 7f56a0263740 20 client.16362 populate_metadata read hostname 'smithi090' 2024-04-25T11:37:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.005+0000 7f56a0263740 1 --2- 172.21.15.90:0/3146403076 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5597aeec3d80 0x5597af2c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.006+0000 7f56a0263740 4 client.16362 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:37:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.006+0000 7f56a0263740 1 -- 172.21.15.90:0/3146403076 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5597af219070 con 0x5597aeec3d80 2024-04-25T11:37:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.008+0000 7f5692ffd640 1 --2- 172.21.15.90:0/3146403076 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5597aeec3d80 0x5597af2c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:34.999 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.012+0000 7f5676ff5640 10 client.16362 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:34.999 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.012+0000 7f5692ffd640 1 --2- 172.21.15.90:0/3146403076 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5597aeec3d80 0x5597af2c9e20 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f4c2d6f67478f43 server_cookie=761a10fb7a3b5c96 in_seq=0 out_seq=0 2024-04-25T11:37:35.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.013+0000 7f5676ff5640 1 -- 172.21.15.90:0/3146403076 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5597af219070 con 0x5597aeec3d80 2024-04-25T11:37:35.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.013+0000 7f5676ff5640 10 client.16362 handle_command_reply: tid=0 2024-04-25T11:37:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.014+0000 7f56a0263740 1 client.16362 shutdown 2024-04-25T11:37:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.014+0000 7f56a0263740 1 -- 172.21.15.90:0/3146403076 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597aef051b0 msgr2=0x5597af2b7f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.014+0000 7f56a0263740 1 --2- 172.21.15.90:0/3146403076 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597aef051b0 0x5597af2b7f10 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f568c0a4880 tx=0x7f568c0a8260 comp rx=0 tx=0).stop 2024-04-25T11:37:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.015+0000 7f56a0263740 1 -- 172.21.15.90:0/3146403076 shutdown_connections 2024-04-25T11:37:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.015+0000 7f56a0263740 1 --2- 172.21.15.90:0/3146403076 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5597aeec3d80 0x5597af2c9e20 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.015+0000 7f5676ff5640 0 client.16362 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.015+0000 7f56a0263740 1 --2- 172.21.15.90:0/3146403076 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597aef051b0 0x5597af2b7f10 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.015+0000 7f56a0263740 1 --2- 172.21.15.90:0/3146403076 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aeee44d0 0x5597af2be060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.015+0000 7f56a0263740 1 --2- 172.21.15.90:0/3146403076 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597aeedc2f0 0x5597af2bdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.015+0000 7f56a0263740 1 -- 172.21.15.90:0/3146403076 >> 172.21.15.90:0/3146403076 conn(0x5597aeef9300 msgr2=0x5597af2a0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.020+0000 7f56a0263740 1 -- 172.21.15.90:0/3146403076 shutdown_connections 2024-04-25T11:37:35.008 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.020+0000 7f56a0263740 1 -- 172.21.15.90:0/3146403076 wait complete. 2024-04-25T11:37:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.026+0000 7f56a0263740 20 client.16362 trim_cache size 0 max 16384 2024-04-25T11:37:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.066+0000 7f56a0263740 1 Processor -- start 2024-04-25T11:37:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.066+0000 7f56a0263740 1 -- start start 2024-04-25T11:37:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.067+0000 7f56a0263740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aeec3d80 0x5597af2a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.067+0000 7f56a0263740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597aeedc2f0 0x5597af2b56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.068+0000 7f56a0263740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597aeee44d0 0x5597af2b1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.068+0000 7f56a0263740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f56941ada80 con 0x5597aeedc2f0 2024-04-25T11:37:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.068+0000 7f56a0263740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f56941af5f0 con 0x5597aeee44d0 2024-04-25T11:37:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.068+0000 7f5692ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aeec3d80 0x5597af2a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.068+0000 7f5692ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aeec3d80 0x5597af2a45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:50182/0 (socket says 172.21.15.90:50182) 2024-04-25T11:37:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.068+0000 7f5692ffd640 1 -- 172.21.15.90:0/4046355111 learned_addr learned my addr 172.21.15.90:0/4046355111 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.068+0000 7f56a0263740 1 -- 172.21.15.90:0/4046355111 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f56941ae150 con 0x5597aeec3d80 2024-04-25T11:37:35.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.071+0000 7f56927fc640 1 --2- 172.21.15.90:0/4046355111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597aeedc2f0 0x5597af2b56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.073+0000 7f56937fe640 1 --2- 172.21.15.90:0/4046355111 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597aeee44d0 0x5597af2b1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.074+0000 7f5692ffd640 1 -- 172.21.15.90:0/4046355111 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597aeee44d0 msgr2=0x5597af2b1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.074+0000 7f5692ffd640 1 --2- 172.21.15.90:0/4046355111 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597aeee44d0 0x5597af2b1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.074+0000 7f5692ffd640 1 -- 172.21.15.90:0/4046355111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597aeedc2f0 msgr2=0x5597af2b56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.074+0000 7f5692ffd640 1 --2- 172.21.15.90:0/4046355111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597aeedc2f0 0x5597af2b56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.074+0000 7f5692ffd640 1 -- 172.21.15.90:0/4046355111 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597af2be230 con 0x5597aeec3d80 2024-04-25T11:37:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.081+0000 7f5692ffd640 1 --2- 172.21.15.90:0/4046355111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aeec3d80 0x5597af2a45a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5684009fd0 tx=0x7f5684005410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1649c64831cac2f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.082+0000 7f5678ff9640 1 -- 172.21.15.90:0/4046355111 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5684004020 con 0x5597aeec3d80 2024-04-25T11:37:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.082+0000 7f5678ff9640 1 -- 172.21.15.90:0/4046355111 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f568401e8d0 con 0x5597aeec3d80 2024-04-25T11:37:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.084+0000 7f56a0263740 1 -- 172.21.15.90:0/4046355111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aeec3d80 msgr2=0x5597af2a45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.084+0000 7f56a0263740 1 --2- 172.21.15.90:0/4046355111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aeec3d80 0x5597af2a45a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5684009fd0 tx=0x7f5684005410 comp rx=0 tx=0).stop 2024-04-25T11:37:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.086+0000 7f56a0263740 1 -- 172.21.15.90:0/4046355111 shutdown_connections 2024-04-25T11:37:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.086+0000 7f56a0263740 1 --2- 172.21.15.90:0/4046355111 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597aeee44d0 0x5597af2b1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.086+0000 7f56a0263740 1 --2- 172.21.15.90:0/4046355111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597aeedc2f0 0x5597af2b56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.087+0000 7f56a0263740 1 --2- 172.21.15.90:0/4046355111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aeec3d80 0x5597af2a45a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.087+0000 7f56a0263740 1 -- 172.21.15.90:0/4046355111 >> 172.21.15.90:0/4046355111 conn(0x5597aeef9300 msgr2=0x5597af2bdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:35.077 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.089+0000 7f56a0263740 1 -- 172.21.15.90:0/4046355111 shutdown_connections 2024-04-25T11:37:35.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.089+0000 7f56a0263740 1 -- 172.21.15.90:0/4046355111 wait complete. 2024-04-25T11:37:35.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.091+0000 7f56a0263740 1 -- start start 2024-04-25T11:37:35.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.091+0000 7f56a0263740 1 Processor -- start 2024-04-25T11:37:35.080 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.092+0000 7f56a0263740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aeec3d80 0x5597af32a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.093+0000 7f56a0263740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597aeedc2f0 0x5597af32ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.093+0000 7f56a0263740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597aeee44d0 0x5597af2c10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.093+0000 7f56a0263740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f56941aeef0 con 0x5597aeedc2f0 2024-04-25T11:37:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.093+0000 7f56a0263740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f56941ae820 con 0x5597aeee44d0 2024-04-25T11:37:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.093+0000 7f56a0263740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f56941aba60 con 0x5597aeec3d80 2024-04-25T11:37:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.093+0000 7f56a0263740 10 client.0 _ll_register_callbacks cb 0x7f568c0b1050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:37:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.093+0000 7f56927fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597aeedc2f0 0x5597af32ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:35.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.093+0000 7f56927fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597aeedc2f0 0x5597af32ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54808/0 (socket says 172.21.15.90:54808) 2024-04-25T11:37:35.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.093+0000 7f56927fc640 1 -- 172.21.15.90:0/634389385 learned_addr learned my addr 172.21.15.90:0/634389385 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:35.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.094+0000 7f56927fc640 1 -- 172.21.15.90:0/634389385 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aeec3d80 msgr2=0x5597af32a600 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:37:35.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.094+0000 7f56927fc640 1 --2- 172.21.15.90:0/634389385 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aeec3d80 0x5597af32a600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:35.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.094+0000 7f56927fc640 1 -- 172.21.15.90:0/634389385 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597aeee44d0 msgr2=0x5597af2c10b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:37:35.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.094+0000 7f56927fc640 1 --2- 172.21.15.90:0/634389385 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597aeee44d0 0x5597af2c10b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:35.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.094+0000 7f56927fc640 1 -- 172.21.15.90:0/634389385 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597af2c16a0 con 0x5597aeedc2f0 2024-04-25T11:37:35.083 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.095+0000 7f56927fc640 1 --2- 172.21.15.90:0/634389385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597aeedc2f0 0x5597af32ab40 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f5688009850 tx=0x7f56880318d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b6ebc9338826b57 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:35.083 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.096+0000 7f5676ff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:37:35.083 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.096+0000 7f5676ff5640 1 -- 172.21.15.90:0/634389385 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5688040040 con 0x5597aeedc2f0 2024-04-25T11:37:35.084 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.096+0000 7f5676ff5640 1 -- 172.21.15.90:0/634389385 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f5688004450 con 0x5597aeedc2f0 2024-04-25T11:37:35.084 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.097+0000 7f56a0263740 1 -- 172.21.15.90:0/634389385 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f568401e8d0 con 0x5597aeedc2f0 2024-04-25T11:37:35.088 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.100+0000 7f5676ff5640 1 -- 172.21.15.90:0/634389385 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f56880035f0 con 0x5597aeedc2f0 2024-04-25T11:37:35.090 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.102+0000 7f56a0263740 10 client.16368 fetch_fsmap learned FSMap version 27 2024-04-25T11:37:35.090 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.102+0000 7f56a0263740 1 -- 172.21.15.90:0/634389385 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5597af331c50 con 0x5597aeedc2f0 2024-04-25T11:37:35.092 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.104+0000 7f5676ff5640 1 -- 172.21.15.90:0/634389385 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f568401e8d0 con 0x5597aeedc2f0 2024-04-25T11:37:35.094 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.105+0000 7f5676ff5640 1 -- 172.21.15.90:0/634389385 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f5688049a70 con 0x5597aeedc2f0 2024-04-25T11:37:35.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.108+0000 7f56a0263740 10 client.16368 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:37:35.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.108+0000 7f56a0263740 10 client.16368 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:37:35.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.108+0000 7f56a0263740 20 client.16368 populate_metadata read hostname 'smithi090' 2024-04-25T11:37:35.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.108+0000 7f56a0263740 1 --2- 172.21.15.90:0/634389385 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5597aef051b0 0x5597af333ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:35.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.108+0000 7f56a0263740 4 client.16368 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:37:35.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.108+0000 7f56a0263740 1 -- 172.21.15.90:0/634389385 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5597af2a11a0 con 0x5597aef051b0 2024-04-25T11:37:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.114+0000 7f5692ffd640 1 --2- 172.21.15.90:0/634389385 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5597aef051b0 0x5597af333ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:35.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.133+0000 7f5692ffd640 1 --2- 172.21.15.90:0/634389385 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5597aef051b0 0x5597af333ef0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24c8702933560fe1 server_cookie=92ebeb4ab19b8548 in_seq=0 out_seq=0 2024-04-25T11:37:35.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:35.133+0000 7f5676ff5640 10 client.16368 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:35.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:35 smithi090 ceph-mon[29575]: pgmap v78: 97 pgs: 97 active+clean; 17 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 2.8 MiB/s rd, 653 MiB/s wr, 999 op/s 2024-04-25T11:37:35.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:35 smithi138 ceph-mon[32328]: pgmap v78: 97 pgs: 97 active+clean; 17 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 2.8 MiB/s rd, 653 MiB/s wr, 999 op/s 2024-04-25T11:37:35.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:35 smithi156 ceph-mon[33045]: pgmap v78: 97 pgs: 97 active+clean; 17 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 2.8 MiB/s rd, 653 MiB/s wr, 999 op/s 2024-04-25T11:37:36.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:36 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:37:36.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:36 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:37:36.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:36 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:37:36.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:36 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:37:37.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:36 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:37:37.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:36 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:37:37.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:37 smithi156 ceph-mon[33045]: pgmap v79: 97 pgs: 97 active+clean; 19 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 1.6 MiB/s rd, 643 MiB/s wr, 924 op/s 2024-04-25T11:37:37.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:37 smithi090 ceph-mon[29575]: pgmap v79: 97 pgs: 97 active+clean; 19 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 1.6 MiB/s rd, 643 MiB/s wr, 924 op/s 2024-04-25T11:37:38.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:37 smithi138 ceph-mon[32328]: pgmap v79: 97 pgs: 97 active+clean; 19 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 1.6 MiB/s rd, 643 MiB/s wr, 924 op/s 2024-04-25T11:37:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.590+0000 7f5676ff5640 1 -- 172.21.15.90:0/634389385 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5597af2a11a0 con 0x5597aef051b0 2024-04-25T11:37:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.590+0000 7f5676ff5640 10 client.16368 handle_command_reply: tid=0 2024-04-25T11:37:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.590+0000 7f56a0263740 1 client.16368 shutdown 2024-04-25T11:37:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.591+0000 7f56a0263740 1 -- 172.21.15.90:0/634389385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597aeedc2f0 msgr2=0x5597af32ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.591+0000 7f56a0263740 1 --2- 172.21.15.90:0/634389385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597aeedc2f0 0x5597af32ab40 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f5688009850 tx=0x7f56880318d0 comp rx=0 tx=0).stop 2024-04-25T11:37:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.591+0000 7f56a0263740 1 -- 172.21.15.90:0/634389385 shutdown_connections 2024-04-25T11:37:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.591+0000 7f56a0263740 1 --2- 172.21.15.90:0/634389385 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5597aef051b0 0x5597af333ef0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.591+0000 7f56a0263740 1 --2- 172.21.15.90:0/634389385 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5597aeee44d0 0x5597af2c10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.591+0000 7f56a0263740 1 --2- 172.21.15.90:0/634389385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597aeedc2f0 0x5597af32ab40 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.591+0000 7f56a0263740 1 --2- 172.21.15.90:0/634389385 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aeec3d80 0x5597af32a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.591+0000 7f56a0263740 1 -- 172.21.15.90:0/634389385 >> 172.21.15.90:0/634389385 conn(0x5597aeef9300 msgr2=0x5597af2bdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.591+0000 7f5676ff5640 0 client.16368 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.591+0000 7f56a0263740 1 -- 172.21.15.90:0/634389385 shutdown_connections 2024-04-25T11:37:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.591+0000 7f56a0263740 1 -- 172.21.15.90:0/634389385 wait complete. 2024-04-25T11:37:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.591+0000 7f56a0263740 20 client.16368 trim_cache size 0 max 16384 2024-04-25T11:37:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:37:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:37:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:37:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:37:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.594+0000 7f56993b2640 1 -- 172.21.15.90:0/2447470851 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f567c0afcd0 msgr2=0x7f567c0b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.594+0000 7f56993b2640 1 --2- 172.21.15.90:0/2447470851 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f567c0afcd0 0x7f567c0b21e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f569419be80 tx=0x7f5688009560 comp rx=0 tx=0).stop 2024-04-25T11:37:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.594+0000 7f56993b2640 1 -- 172.21.15.90:0/2447470851 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f569410a040 msgr2=0x7f569419a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.594+0000 7f56993b2640 1 --2- 172.21.15.90:0/2447470851 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f569410a040 0x7f569419a650 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f568c009fd0 tx=0x7f568c00ef10 comp rx=0 tx=0).stop 2024-04-25T11:37:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.594+0000 7f56993b2640 1 -- 172.21.15.90:0/2447470851 shutdown_connections 2024-04-25T11:37:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.594+0000 7f56993b2640 1 --2- 172.21.15.90:0/2447470851 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f567c0afcd0 0x7f567c0b21e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.594+0000 7f56993b2640 1 --2- 172.21.15.90:0/2447470851 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f569410a040 0x7f569419a650 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.594+0000 7f56993b2640 1 --2- 172.21.15.90:0/2447470851 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5694109ba0 0x7f56941a0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.594+0000 7f56993b2640 1 --2- 172.21.15.90:0/2447470851 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f56941042f0 0x7f56941a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.594+0000 7f56993b2640 1 -- 172.21.15.90:0/2447470851 >> 172.21.15.90:0/2447470851 conn(0x7f56940792a0 msgr2=0x7f56941022c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:39.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.596+0000 7f56993b2640 1 -- 172.21.15.90:0/2447470851 shutdown_connections 2024-04-25T11:37:39.586 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:39.597+0000 7f56993b2640 1 -- 172.21.15.90:0/2447470851 wait complete. 2024-04-25T11:37:39.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:39 smithi090 ceph-mon[29575]: pgmap v80: 97 pgs: 97 active+clean; 19 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 1.6 MiB/s rd, 586 MiB/s wr, 819 op/s 2024-04-25T11:37:40.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:39 smithi138 ceph-mon[32328]: pgmap v80: 97 pgs: 97 active+clean; 19 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 1.6 MiB/s rd, 586 MiB/s wr, 819 op/s 2024-04-25T11:37:40.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:39 smithi156 ceph-mon[33045]: pgmap v80: 97 pgs: 97 active+clean; 19 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 1.6 MiB/s rd, 586 MiB/s wr, 819 op/s 2024-04-25T11:37:42.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:41 smithi138 ceph-mon[32328]: pgmap v81: 97 pgs: 97 active+clean; 19 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 711 MiB/s wr, 949 op/s 2024-04-25T11:37:42.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:41 smithi156 ceph-mon[33045]: pgmap v81: 97 pgs: 97 active+clean; 19 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 711 MiB/s wr, 949 op/s 2024-04-25T11:37:42.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:41 smithi090 ceph-mon[29575]: pgmap v81: 97 pgs: 97 active+clean; 19 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 711 MiB/s wr, 949 op/s 2024-04-25T11:37:44.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:43 smithi156 ceph-mon[33045]: pgmap v82: 97 pgs: 97 active+clean; 19 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 418 MiB/s wr, 415 op/s 2024-04-25T11:37:44.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:43 smithi090 ceph-mon[29575]: pgmap v82: 97 pgs: 97 active+clean; 19 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 418 MiB/s wr, 415 op/s 2024-04-25T11:37:44.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:43 smithi138 ceph-mon[32328]: pgmap v82: 97 pgs: 97 active+clean; 19 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 418 MiB/s wr, 415 op/s 2024-04-25T11:37:46.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:45 smithi156 ceph-mon[33045]: pgmap v83: 97 pgs: 97 active+clean; 19 GiB data, 59 GiB used, 1014 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 537 MiB/s wr, 493 op/s 2024-04-25T11:37:46.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:45 smithi090 ceph-mon[29575]: pgmap v83: 97 pgs: 97 active+clean; 19 GiB data, 59 GiB used, 1014 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 537 MiB/s wr, 493 op/s 2024-04-25T11:37:46.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:45 smithi138 ceph-mon[32328]: pgmap v83: 97 pgs: 97 active+clean; 19 GiB data, 59 GiB used, 1014 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 537 MiB/s wr, 493 op/s 2024-04-25T11:37:48.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:48 smithi156 ceph-mon[33045]: pgmap v84: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 527 MiB/s wr, 433 op/s 2024-04-25T11:37:48.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:48 smithi090 ceph-mon[29575]: pgmap v84: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 527 MiB/s wr, 433 op/s 2024-04-25T11:37:48.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:48 smithi138 ceph-mon[32328]: pgmap v84: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 527 MiB/s wr, 433 op/s 2024-04-25T11:37:49.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:49 smithi156 ceph-mon[33045]: pgmap v85: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 407 MiB/s wr, 317 op/s 2024-04-25T11:37:49.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:49 smithi090 ceph-mon[29575]: pgmap v85: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 407 MiB/s wr, 317 op/s 2024-04-25T11:37:49.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:49 smithi138 ceph-mon[32328]: pgmap v85: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 407 MiB/s wr, 317 op/s 2024-04-25T11:37:51.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:51 smithi156 ceph-mon[33045]: pgmap v86: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 643 MiB/s wr, 767 op/s 2024-04-25T11:37:51.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:51 smithi090 ceph-mon[29575]: pgmap v86: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 643 MiB/s wr, 767 op/s 2024-04-25T11:37:51.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:51 smithi138 ceph-mon[32328]: pgmap v86: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 643 MiB/s wr, 767 op/s 2024-04-25T11:37:52.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.759+0000 7f8404b72640 1 -- 172.21.15.90:0/2928174249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84001072b0 msgr2=0x7f84001076c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:52.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.759+0000 7f8404b72640 1 --2- 172.21.15.90:0/2928174249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84001072b0 0x7f84001076c0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f83e8009900 tx=0x7f83e802a410 comp rx=0 tx=0).stop 2024-04-25T11:37:52.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.759+0000 7f8404b72640 1 -- 172.21.15.90:0/2928174249 shutdown_connections 2024-04-25T11:37:52.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.759+0000 7f8404b72640 1 --2- 172.21.15.90:0/2928174249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8400107d30 0x7f840010f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:52.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.759+0000 7f8404b72640 1 --2- 172.21.15.90:0/2928174249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84001072b0 0x7f84001076c0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:52.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.759+0000 7f8404b72640 1 --2- 172.21.15.90:0/2928174249 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84001033f0 0x7f8400103800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:52.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.759+0000 7f8404b72640 1 -- 172.21.15.90:0/2928174249 >> 172.21.15.90:0/2928174249 conn(0x7f84000778b0 msgr2=0x7f8400077cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:52.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.759+0000 7f8404b72640 1 -- 172.21.15.90:0/2928174249 shutdown_connections 2024-04-25T11:37:52.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.759+0000 7f8404b72640 1 -- 172.21.15.90:0/2928174249 wait complete. 2024-04-25T11:37:52.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.760+0000 7f8404b72640 1 Processor -- start 2024-04-25T11:37:52.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.760+0000 7f8404b72640 1 -- start start 2024-04-25T11:37:52.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.760+0000 7f8404b72640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84001033f0 0x7f840019a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:52.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.760+0000 7f8404b72640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84001072b0 0x7f840019acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:52.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.760+0000 7f8404b72640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8400107d30 0x7f840019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:52.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.760+0000 7f8404b72640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8400112c30 con 0x7f8400107d30 2024-04-25T11:37:52.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.760+0000 7f8404b72640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8400112ab0 con 0x7f84001033f0 2024-04-25T11:37:52.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.760+0000 7f8404b72640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8400112db0 con 0x7f84001072b0 2024-04-25T11:37:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.761+0000 7f83fe59c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84001033f0 0x7f840019a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.761+0000 7f83fe59c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84001033f0 0x7f840019a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:52798/0 (socket says 172.21.15.90:52798) 2024-04-25T11:37:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.761+0000 7f83fe59c640 1 -- 172.21.15.90:0/602089864 learned_addr learned my addr 172.21.15.90:0/602089864 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.761+0000 7f83fed9d640 1 --2- 172.21.15.90:0/602089864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8400107d30 0x7f840019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.761+0000 7f83f7fff640 1 --2- 172.21.15.90:0/602089864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84001072b0 0x7f840019acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.761+0000 7f83fe59c640 1 -- 172.21.15.90:0/602089864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84001072b0 msgr2=0x7f840019acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.761+0000 7f83fe59c640 1 --2- 172.21.15.90:0/602089864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84001072b0 0x7f840019acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.761+0000 7f83fe59c640 1 -- 172.21.15.90:0/602089864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8400107d30 msgr2=0x7f840019ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.761+0000 7f83fe59c640 1 --2- 172.21.15.90:0/602089864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8400107d30 0x7f840019ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.761+0000 7f83fe59c640 1 -- 172.21.15.90:0/602089864 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f840019f5c0 con 0x7f84001033f0 2024-04-25T11:37:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.761+0000 7f83fe59c640 1 --2- 172.21.15.90:0/602089864 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84001033f0 0x7f840019a760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f83f000e9c0 tx=0x7f83f000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.761+0000 7f83f77fe640 1 -- 172.21.15.90:0/602089864 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f000cd80 con 0x7f84001033f0 2024-04-25T11:37:52.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.761+0000 7f83f77fe640 1 -- 172.21.15.90:0/602089864 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83f00054d0 con 0x7f84001033f0 2024-04-25T11:37:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.761+0000 7f8404b72640 1 -- 172.21.15.90:0/602089864 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f840019f8b0 con 0x7f84001033f0 2024-04-25T11:37:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.761+0000 7f83f77fe640 1 -- 172.21.15.90:0/602089864 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f0010430 con 0x7f84001033f0 2024-04-25T11:37:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.761+0000 7f8404b72640 1 -- 172.21.15.90:0/602089864 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84001a7230 con 0x7f84001033f0 2024-04-25T11:37:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.764+0000 7f83f77fe640 1 -- 172.21.15.90:0/602089864 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f83f00105d0 con 0x7f84001033f0 2024-04-25T11:37:52.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.765+0000 7f840ba1e740 1 Processor -- start 2024-04-25T11:37:52.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.766+0000 7f840ba1e740 1 -- start start 2024-04-25T11:37:52.753 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.766+0000 7f83f77fe640 1 --2- 172.21.15.90:0/602089864 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f83ec0afc50 0x7f83ec0b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:52.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.766+0000 7f83fed9d640 1 --2- 172.21.15.90:0/602089864 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f83ec0afc50 0x7f83ec0b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.766+0000 7f840ba1e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f807fac1b0 0x55f80834baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.766+0000 7f840ba1e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f807f832f0 0x55f80834c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.766+0000 7f840ba1e740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f807f8b4d0 0x55f80834c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.766+0000 7f840ba1e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f8082b2460 con 0x55f807f832f0 2024-04-25T11:37:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.766+0000 7f840ba1e740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f80826c590 con 0x55f807f8b4d0 2024-04-25T11:37:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.766+0000 7f840ba1e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f8082ae420 con 0x55f807fac1b0 2024-04-25T11:37:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.766+0000 7f83f77fe640 1 -- 172.21.15.90:0/602089864 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f83f00df4f0 con 0x7f84001033f0 2024-04-25T11:37:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.766+0000 7f83f7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f807f832f0 0x55f80834c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.766+0000 7f83f7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f807f832f0 0x55f80834c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35208/0 (socket says 172.21.15.90:35208) 2024-04-25T11:37:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.766+0000 7f83f7fff640 1 -- 172.21.15.90:0/3707087831 learned_addr learned my addr 172.21.15.90:0/3707087831 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.767+0000 7f83fe59c640 1 --2- 172.21.15.90:0/3707087831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f807fac1b0 0x55f80834baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.767+0000 7f83fed9d640 1 --2- 172.21.15.90:0/3707087831 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f807f8b4d0 0x55f80834c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.767+0000 7f83f7fff640 1 -- 172.21.15.90:0/3707087831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f807fac1b0 msgr2=0x55f80834baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:52.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.767+0000 7f83f7fff640 1 --2- 172.21.15.90:0/3707087831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f807fac1b0 0x55f80834baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:52.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.767+0000 7f83f7fff640 1 -- 172.21.15.90:0/3707087831 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f807f8b4d0 msgr2=0x55f80834c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:52.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.767+0000 7f83f7fff640 1 --2- 172.21.15.90:0/3707087831 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f807f8b4d0 0x55f80834c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:52.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.767+0000 7f83f7fff640 1 -- 172.21.15.90:0/3707087831 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8082790d0 con 0x55f807f832f0 2024-04-25T11:37:52.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.767+0000 7f83fed9d640 1 --2- 172.21.15.90:0/602089864 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f83ec0afc50 0x7f83ec0b2160 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f83f8005fd0 tx=0x7f83f8003d40 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:52.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.767+0000 7f83f7fff640 1 --2- 172.21.15.90:0/3707087831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f807f832f0 0x55f80834c030 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f84000754a0 tx=0x7f83e8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6871c6f369b3d044 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:52.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.767+0000 7f83f4ff9640 1 -- 172.21.15.90:0/3707087831 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e8040070 con 0x55f807f832f0 2024-04-25T11:37:52.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.767+0000 7f83f4ff9640 1 -- 172.21.15.90:0/3707087831 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83e8004030 con 0x55f807f832f0 2024-04-25T11:37:52.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.767+0000 7f83f4ff9640 1 -- 172.21.15.90:0/3707087831 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e8035040 con 0x55f807f832f0 2024-04-25T11:37:52.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.768+0000 7f840ba1e740 1 -- 172.21.15.90:0/3707087831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f807f832f0 msgr2=0x55f80834c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:52.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.768+0000 7f840ba1e740 1 --2- 172.21.15.90:0/3707087831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f807f832f0 0x55f80834c030 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f84000754a0 tx=0x7f83e8000bc0 comp rx=0 tx=0).stop 2024-04-25T11:37:52.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.768+0000 7f840ba1e740 1 -- 172.21.15.90:0/3707087831 shutdown_connections 2024-04-25T11:37:52.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.768+0000 7f840ba1e740 1 --2- 172.21.15.90:0/3707087831 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f807f8b4d0 0x55f80834c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:52.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.768+0000 7f840ba1e740 1 --2- 172.21.15.90:0/3707087831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f807f832f0 0x55f80834c030 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:52.758 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.768+0000 7f840ba1e740 1 --2- 172.21.15.90:0/3707087831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f807fac1b0 0x55f80834baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:52.758 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.768+0000 7f840ba1e740 1 -- 172.21.15.90:0/3707087831 >> 172.21.15.90:0/3707087831 conn(0x55f807fa0300 msgr2=0x55f8083482a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:52.758 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.769+0000 7f840ba1e740 1 -- 172.21.15.90:0/3707087831 shutdown_connections 2024-04-25T11:37:52.758 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.769+0000 7f840ba1e740 1 -- 172.21.15.90:0/3707087831 wait complete. 2024-04-25T11:37:52.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.769+0000 7f840ba1e740 1 -- start start 2024-04-25T11:37:52.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.769+0000 7f840ba1e740 1 Processor -- start 2024-04-25T11:37:52.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.770+0000 7f840ba1e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f807f832f0 0x55f8083561e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:52.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.770+0000 7f840ba1e740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f807f8b4d0 0x55f808356720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:52.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.770+0000 7f840ba1e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f807fac1b0 0x55f80835c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.770+0000 7f840ba1e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f808358bc0 con 0x55f807fac1b0 2024-04-25T11:37:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.770+0000 7f840ba1e740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f808239b60 con 0x55f807f8b4d0 2024-04-25T11:37:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.770+0000 7f840ba1e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f808358d40 con 0x55f807f832f0 2024-04-25T11:37:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.770+0000 7f840ba1e740 10 client.0 _ll_register_callbacks cb 0x55f80825e2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:37:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.770+0000 7f83fed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f807fac1b0 0x55f80835c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.770+0000 7f83f7fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f807f8b4d0 0x55f808356720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.770+0000 7f83fed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f807fac1b0 0x55f80835c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35216/0 (socket says 172.21.15.90:35216) 2024-04-25T11:37:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.770+0000 7f83fed9d640 1 -- 172.21.15.90:0/1318473473 learned_addr learned my addr 172.21.15.90:0/1318473473 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.770+0000 7f83fe59c640 1 --2- 172.21.15.90:0/1318473473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f807f832f0 0x55f8083561e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.770+0000 7f83f7fff640 1 -- 172.21.15.90:0/1318473473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f807f832f0 msgr2=0x55f8083561e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.770+0000 7f83f7fff640 1 --2- 172.21.15.90:0/1318473473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f807f832f0 0x55f8083561e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.770+0000 7f83f7fff640 1 -- 172.21.15.90:0/1318473473 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f807fac1b0 msgr2=0x55f80835c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:52.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.770+0000 7f83f7fff640 1 --2- 172.21.15.90:0/1318473473 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f807fac1b0 0x55f80835c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:52.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.770+0000 7f83f7fff640 1 -- 172.21.15.90:0/1318473473 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f807fc4210 con 0x55f807f8b4d0 2024-04-25T11:37:52.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.770+0000 7f83fe59c640 1 --2- 172.21.15.90:0/1318473473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f807f832f0 0x55f8083561e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:37:52.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.771+0000 7f83f7fff640 1 --2- 172.21.15.90:0/1318473473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f807f8b4d0 0x55f808356720 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x55f80834cea0 tx=0x7f83e802a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24f3678f3c5ac589 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:52.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.771+0000 7f83e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:37:52.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.771+0000 7f83e5ffb640 1 -- 172.21.15.90:0/1318473473 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e8040070 con 0x55f807f8b4d0 2024-04-25T11:37:52.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.771+0000 7f840ba1e740 1 -- 172.21.15.90:0/1318473473 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83e8004030 con 0x55f807f8b4d0 2024-04-25T11:37:52.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.771+0000 7f83e5ffb640 1 -- 172.21.15.90:0/1318473473 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f83e8005260 con 0x55f807f8b4d0 2024-04-25T11:37:52.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.771+0000 7f83e5ffb640 1 -- 172.21.15.90:0/1318473473 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f83e8054050 con 0x55f807f8b4d0 2024-04-25T11:37:52.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.771+0000 7f840ba1e740 10 client.25697 fetch_fsmap learned FSMap version 27 2024-04-25T11:37:52.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.771+0000 7f840ba1e740 1 -- 172.21.15.90:0/1318473473 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f808090620 con 0x55f807f8b4d0 2024-04-25T11:37:52.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.771+0000 7f83e5ffb640 1 -- 172.21.15.90:0/1318473473 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83e8004030 con 0x55f807f8b4d0 2024-04-25T11:37:52.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.771+0000 7f83e5ffb640 1 -- 172.21.15.90:0/1318473473 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f83e8004b30 con 0x55f807f8b4d0 2024-04-25T11:37:52.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.772+0000 7f840ba1e740 10 client.25697 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:37:52.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.772+0000 7f840ba1e740 10 client.25697 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:37:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.772+0000 7f840ba1e740 20 client.25697 populate_metadata read hostname 'smithi090' 2024-04-25T11:37:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.772+0000 7f840ba1e740 1 --2- 172.21.15.90:0/1318473473 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f807f6ad80 0x55f8083685a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.772+0000 7f840ba1e740 4 client.25697 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:37:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.772+0000 7f840ba1e740 1 -- 172.21.15.90:0/1318473473 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f8082c0070 con 0x55f807f6ad80 2024-04-25T11:37:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.772+0000 7f83fe59c640 1 --2- 172.21.15.90:0/1318473473 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f807f6ad80 0x55f8083685a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.772+0000 7f83e5ffb640 10 client.25697 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.772+0000 7f83fe59c640 1 --2- 172.21.15.90:0/1318473473 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f807f6ad80 0x55f8083685a0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff16241790e28ebb server_cookie=efd8a142af276ce1 in_seq=0 out_seq=0 2024-04-25T11:37:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.773+0000 7f83e5ffb640 1 -- 172.21.15.90:0/1318473473 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f8082c0070 con 0x55f807f6ad80 2024-04-25T11:37:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.773+0000 7f83e5ffb640 10 client.25697 handle_command_reply: tid=0 2024-04-25T11:37:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.773+0000 7f840ba1e740 1 client.25697 shutdown 2024-04-25T11:37:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.773+0000 7f840ba1e740 1 -- 172.21.15.90:0/1318473473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f807f8b4d0 msgr2=0x55f808356720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.773+0000 7f840ba1e740 1 --2- 172.21.15.90:0/1318473473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f807f8b4d0 0x55f808356720 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x55f80834cea0 tx=0x7f83e802a410 comp rx=0 tx=0).stop 2024-04-25T11:37:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.773+0000 7f840ba1e740 1 -- 172.21.15.90:0/1318473473 shutdown_connections 2024-04-25T11:37:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.773+0000 7f840ba1e740 1 --2- 172.21.15.90:0/1318473473 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f807f6ad80 0x55f8083685a0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.773+0000 7f840ba1e740 1 --2- 172.21.15.90:0/1318473473 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f807fac1b0 0x55f80835c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.773+0000 7f840ba1e740 1 --2- 172.21.15.90:0/1318473473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f807f8b4d0 0x55f808356720 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.773+0000 7f83e5ffb640 0 client.25697 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.773+0000 7f840ba1e740 1 --2- 172.21.15.90:0/1318473473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f807f832f0 0x55f8083561e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.773+0000 7f840ba1e740 1 -- 172.21.15.90:0/1318473473 >> 172.21.15.90:0/1318473473 conn(0x55f807fa0300 msgr2=0x55f80834b920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.773+0000 7f840ba1e740 1 -- 172.21.15.90:0/1318473473 shutdown_connections 2024-04-25T11:37:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.773+0000 7f840ba1e740 1 -- 172.21.15.90:0/1318473473 wait complete. 2024-04-25T11:37:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.774+0000 7f840ba1e740 20 client.25697 trim_cache size 0 max 16384 2024-04-25T11:37:52.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.792+0000 7f840ba1e740 1 Processor -- start 2024-04-25T11:37:52.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.792+0000 7f840ba1e740 1 -- start start 2024-04-25T11:37:52.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.792+0000 7f840ba1e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f807f832f0 0x55f80834b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:52.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.792+0000 7f840ba1e740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f807f8b4d0 0x55f80834bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:52.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.792+0000 7f840ba1e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f807fac1b0 0x55f808353210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:52.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.792+0000 7f840ba1e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f84001afbb0 con 0x55f807fac1b0 2024-04-25T11:37:52.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.792+0000 7f840ba1e740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f84001b1720 con 0x55f807f8b4d0 2024-04-25T11:37:52.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.792+0000 7f840ba1e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f84001b0280 con 0x55f807f832f0 2024-04-25T11:37:52.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.792+0000 7f83fed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f807fac1b0 0x55f808353210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:52.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.792+0000 7f83f7fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f807f8b4d0 0x55f80834bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:52.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.792+0000 7f83fed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f807fac1b0 0x55f808353210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35226/0 (socket says 172.21.15.90:35226) 2024-04-25T11:37:52.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.792+0000 7f83fed9d640 1 -- 172.21.15.90:0/1799886606 learned_addr learned my addr 172.21.15.90:0/1799886606 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:52.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.792+0000 7f83fe59c640 1 --2- 172.21.15.90:0/1799886606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f807f832f0 0x55f80834b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:52.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.792+0000 7f83f7fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f807f8b4d0 0x55f80834bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:52818/0 (socket says 172.21.15.90:52818) 2024-04-25T11:37:52.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.793+0000 7f83fe59c640 1 -- 172.21.15.90:0/1799886606 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f807f8b4d0 msgr2=0x55f80834bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:52.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.793+0000 7f83fe59c640 1 --2- 172.21.15.90:0/1799886606 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f807f8b4d0 0x55f80834bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:52.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.793+0000 7f83fe59c640 1 -- 172.21.15.90:0/1799886606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f807fac1b0 msgr2=0x55f808353210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:52.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.793+0000 7f83fe59c640 1 --2- 172.21.15.90:0/1799886606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f807fac1b0 0x55f808353210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:52.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.793+0000 7f83fe59c640 1 -- 172.21.15.90:0/1799886606 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f808354520 con 0x55f807f832f0 2024-04-25T11:37:52.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.793+0000 7f83fe59c640 1 --2- 172.21.15.90:0/1799886606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f807f832f0 0x55f80834b880 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f83f00dcbe0 tx=0x7f83f00049b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae98d6f7ae318a53 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:52.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.793+0000 7f83e7fff640 1 -- 172.21.15.90:0/1799886606 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f00c1070 con 0x55f807f832f0 2024-04-25T11:37:52.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.793+0000 7f83e7fff640 1 -- 172.21.15.90:0/1799886606 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83f00a7630 con 0x55f807f832f0 2024-04-25T11:37:52.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.793+0000 7f83e7fff640 1 -- 172.21.15.90:0/1799886606 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f00b0000 con 0x55f807f832f0 2024-04-25T11:37:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.794+0000 7f840ba1e740 1 -- 172.21.15.90:0/1799886606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f807f832f0 msgr2=0x55f80834b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.794+0000 7f840ba1e740 1 --2- 172.21.15.90:0/1799886606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f807f832f0 0x55f80834b880 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f83f00dcbe0 tx=0x7f83f00049b0 comp rx=0 tx=0).stop 2024-04-25T11:37:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.794+0000 7f840ba1e740 1 -- 172.21.15.90:0/1799886606 shutdown_connections 2024-04-25T11:37:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.794+0000 7f840ba1e740 1 --2- 172.21.15.90:0/1799886606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f807fac1b0 0x55f808353210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.794+0000 7f840ba1e740 1 --2- 172.21.15.90:0/1799886606 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f807f8b4d0 0x55f80834bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.794+0000 7f840ba1e740 1 --2- 172.21.15.90:0/1799886606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f807f832f0 0x55f80834b880 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.794+0000 7f840ba1e740 1 -- 172.21.15.90:0/1799886606 >> 172.21.15.90:0/1799886606 conn(0x55f807fa0300 msgr2=0x55f8083548a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.794+0000 7f840ba1e740 1 -- 172.21.15.90:0/1799886606 shutdown_connections 2024-04-25T11:37:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.794+0000 7f840ba1e740 1 -- 172.21.15.90:0/1799886606 wait complete. 2024-04-25T11:37:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.794+0000 7f840ba1e740 1 -- start start 2024-04-25T11:37:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.795+0000 7f840ba1e740 1 Processor -- start 2024-04-25T11:37:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.795+0000 7f840ba1e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f807f832f0 0x55f8083c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.795+0000 7f840ba1e740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f807f8b4d0 0x55f8083c8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.795+0000 7f840ba1e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f807fac1b0 0x55f808362010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.795+0000 7f840ba1e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f84001b1020 con 0x55f807f832f0 2024-04-25T11:37:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.795+0000 7f840ba1e740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f84001b0950 con 0x55f807f8b4d0 2024-04-25T11:37:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.795+0000 7f840ba1e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f84001adb90 con 0x55f807fac1b0 2024-04-25T11:37:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.795+0000 7f840ba1e740 10 client.0 _ll_register_callbacks cb 0x7f83f000cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:37:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.795+0000 7f83fe59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f807f832f0 0x55f8083c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.795+0000 7f83f7fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f807f8b4d0 0x55f8083c8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.795+0000 7f83fe59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f807f832f0 0x55f8083c7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35228/0 (socket says 172.21.15.90:35228) 2024-04-25T11:37:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.795+0000 7f83fe59c640 1 -- 172.21.15.90:0/1137084606 learned_addr learned my addr 172.21.15.90:0/1137084606 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.795+0000 7f83fed9d640 1 --2- 172.21.15.90:0/1137084606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f807fac1b0 0x55f808362010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.796+0000 7f83fed9d640 1 -- 172.21.15.90:0/1137084606 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f807f8b4d0 msgr2=0x55f8083c8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.796+0000 7f83fed9d640 1 --2- 172.21.15.90:0/1137084606 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f807f8b4d0 0x55f8083c8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.796+0000 7f83fed9d640 1 -- 172.21.15.90:0/1137084606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f807f832f0 msgr2=0x55f8083c7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.796+0000 7f83fed9d640 1 --2- 172.21.15.90:0/1137084606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f807f832f0 0x55f8083c7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.796+0000 7f83fed9d640 1 -- 172.21.15.90:0/1137084606 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8083625d0 con 0x55f807fac1b0 2024-04-25T11:37:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.796+0000 7f83fe59c640 1 --2- 172.21.15.90:0/1137084606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f807f832f0 0x55f8083c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:37:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.796+0000 7f83f7fff640 1 --2- 172.21.15.90:0/1137084606 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f807f8b4d0 0x55f8083c8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:37:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.796+0000 7f83fed9d640 1 --2- 172.21.15.90:0/1137084606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f807fac1b0 0x55f808362010 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f83f8004410 tx=0x7f83f8004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f0b56f6c6c9f715 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.796+0000 7f83e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:37:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.796+0000 7f83e5ffb640 1 -- 172.21.15.90:0/1137084606 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f8011890 con 0x55f807fac1b0 2024-04-25T11:37:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.796+0000 7f83e5ffb640 1 -- 172.21.15.90:0/1137084606 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f83f8011a30 con 0x55f807fac1b0 2024-04-25T11:37:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.796+0000 7f840ba1e740 1 -- 172.21.15.90:0/1137084606 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83f00a7630 con 0x55f807fac1b0 2024-04-25T11:37:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.796+0000 7f83e5ffb640 1 -- 172.21.15.90:0/1137084606 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f83f803d050 con 0x55f807fac1b0 2024-04-25T11:37:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.796+0000 7f840ba1e740 10 client.25645 fetch_fsmap learned FSMap version 27 2024-04-25T11:37:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.796+0000 7f840ba1e740 1 -- 172.21.15.90:0/1137084606 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f8083d0720 con 0x55f807fac1b0 2024-04-25T11:37:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.797+0000 7f83e5ffb640 1 -- 172.21.15.90:0/1137084606 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83f802ed00 con 0x55f807fac1b0 2024-04-25T11:37:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.797+0000 7f83e5ffb640 1 -- 172.21.15.90:0/1137084606 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f83f801f760 con 0x55f807fac1b0 2024-04-25T11:37:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.797+0000 7f840ba1e740 10 client.25645 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:37:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.797+0000 7f840ba1e740 10 client.25645 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:37:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.797+0000 7f840ba1e740 20 client.25645 populate_metadata read hostname 'smithi090' 2024-04-25T11:37:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.797+0000 7f840ba1e740 1 --2- 172.21.15.90:0/1137084606 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f807f401c0 0x55f8083d2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.797+0000 7f840ba1e740 4 client.25645 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:37:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.797+0000 7f840ba1e740 1 -- 172.21.15.90:0/1137084606 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f83f00a7630 con 0x55f807f401c0 2024-04-25T11:37:52.789 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.797+0000 7f83fe59c640 1 --2- 172.21.15.90:0/1137084606 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f807f401c0 0x55f8083d2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:52.789 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.798+0000 7f83e5ffb640 10 client.25645 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:52.789 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:52.798+0000 7f83fe59c640 1 --2- 172.21.15.90:0/1137084606 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f807f401c0 0x55f8083d2f60 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c246934645cf741 server_cookie=5aafba94a34791b1 in_seq=0 out_seq=0 2024-04-25T11:37:53.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:53 smithi090 ceph-mon[29575]: pgmap v87: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.6 MiB/s rd, 518 MiB/s wr, 637 op/s 2024-04-25T11:37:53.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:53 smithi138 ceph-mon[32328]: pgmap v87: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.6 MiB/s rd, 518 MiB/s wr, 637 op/s 2024-04-25T11:37:53.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:53 smithi156 ceph-mon[33045]: pgmap v87: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.6 MiB/s rd, 518 MiB/s wr, 637 op/s 2024-04-25T11:37:55.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:55 smithi090 ceph-mon[29575]: pgmap v88: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 614 MiB/s wr, 775 op/s 2024-04-25T11:37:55.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:55 smithi138 ceph-mon[32328]: pgmap v88: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 614 MiB/s wr, 775 op/s 2024-04-25T11:37:55.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:55 smithi156 ceph-mon[33045]: pgmap v88: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 614 MiB/s wr, 775 op/s 2024-04-25T11:37:57.732 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:57 smithi090 ceph-mon[29575]: pgmap v89: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 632 MiB/s wr, 851 op/s 2024-04-25T11:37:57.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:57 smithi138 ceph-mon[32328]: pgmap v89: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 632 MiB/s wr, 851 op/s 2024-04-25T11:37:57.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:57 smithi156 ceph-mon[33045]: pgmap v89: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 632 MiB/s wr, 851 op/s 2024-04-25T11:37:58.354 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:37:58.354 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:37:58.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.492+0000 7f4aa8a05640 1 -- 172.21.15.90:0/977313427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4aa4108aa0 msgr2=0x7f4aa4102ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:58.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.492+0000 7f4aa8a05640 1 --2- 172.21.15.90:0/977313427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4aa4108aa0 0x7f4aa4102ca0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f4a94009900 tx=0x7f4a9402a410 comp rx=0 tx=0).stop 2024-04-25T11:37:58.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.493+0000 7f4aa8a05640 1 -- 172.21.15.90:0/977313427 shutdown_connections 2024-04-25T11:37:58.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.493+0000 7f4aa8a05640 1 --2- 172.21.15.90:0/977313427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4103850 0x7f4aa410f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:58.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.493+0000 7f4aa8a05640 1 --2- 172.21.15.90:0/977313427 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4aa4108f40 0x7f4aa41031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:58.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.493+0000 7f4aa8a05640 1 --2- 172.21.15.90:0/977313427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4aa4108aa0 0x7f4aa4102ca0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:58.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.493+0000 7f4aa8a05640 1 -- 172.21.15.90:0/977313427 >> 172.21.15.90:0/977313427 conn(0x7f4aa4077880 msgr2=0x7f4aa4077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:58.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.493+0000 7f4aa8a05640 1 -- 172.21.15.90:0/977313427 shutdown_connections 2024-04-25T11:37:58.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.493+0000 7f4aa8a05640 1 -- 172.21.15.90:0/977313427 wait complete. 2024-04-25T11:37:58.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.494+0000 7f4aa8a05640 1 Processor -- start 2024-04-25T11:37:58.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.494+0000 7f4aa8a05640 1 -- start start 2024-04-25T11:37:58.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.494+0000 7f4aa8a05640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4aa4103850 0x7f4aa41985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:58.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.494+0000 7f4aa8a05640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4108aa0 0x7f4aa4198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:58.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.494+0000 7f4aa8a05640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4aa4108f40 0x7f4aa419cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:58.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.494+0000 7f4aa8a05640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa4111bd0 con 0x7f4aa4108f40 2024-04-25T11:37:58.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.494+0000 7f4aa8a05640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa4111a50 con 0x7f4aa4103850 2024-04-25T11:37:58.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.494+0000 7f4aa8a05640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa4111d50 con 0x7f4aa4108aa0 2024-04-25T11:37:58.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.494+0000 7f4aa2d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4aa4108f40 0x7f4aa419cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:58.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.494+0000 7f4aa2d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4aa4108f40 0x7f4aa419cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35248/0 (socket says 172.21.15.90:35248) 2024-04-25T11:37:58.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.494+0000 7f4aa2d9d640 1 -- 172.21.15.90:0/1704467606 learned_addr learned my addr 172.21.15.90:0/1704467606 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:58.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.494+0000 7f4aa259c640 1 --2- 172.21.15.90:0/1704467606 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4aa4103850 0x7f4aa41985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:58.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.494+0000 7f4a9bfff640 1 --2- 172.21.15.90:0/1704467606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4108aa0 0x7f4aa4198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:58.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.495+0000 7f4aa2d9d640 1 -- 172.21.15.90:0/1704467606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4108aa0 msgr2=0x7f4aa4198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:58.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.495+0000 7f4aa2d9d640 1 --2- 172.21.15.90:0/1704467606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4108aa0 0x7f4aa4198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:58.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.495+0000 7f4aa2d9d640 1 -- 172.21.15.90:0/1704467606 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4aa4103850 msgr2=0x7f4aa41985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:58.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.495+0000 7f4aa2d9d640 1 --2- 172.21.15.90:0/1704467606 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4aa4103850 0x7f4aa41985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:58.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.495+0000 7f4aa2d9d640 1 -- 172.21.15.90:0/1704467606 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa419d410 con 0x7f4aa4108f40 2024-04-25T11:37:58.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.495+0000 7f4aa259c640 1 --2- 172.21.15.90:0/1704467606 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4aa4103850 0x7f4aa41985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:37:58.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.495+0000 7f4aa2d9d640 1 --2- 172.21.15.90:0/1704467606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4aa4108f40 0x7f4aa419cdd0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c00eb10 tx=0x7f4a9c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:58.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.495+0000 7f4a9b7fe640 1 -- 172.21.15.90:0/1704467606 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c004b20 con 0x7f4aa4108f40 2024-04-25T11:37:58.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.495+0000 7f4a9b7fe640 1 -- 172.21.15.90:0/1704467606 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a9c004cc0 con 0x7f4aa4108f40 2024-04-25T11:37:58.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.495+0000 7f4aa8a05640 1 -- 172.21.15.90:0/1704467606 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aa419d700 con 0x7f4aa4108f40 2024-04-25T11:37:58.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.495+0000 7f4a9b7fe640 1 -- 172.21.15.90:0/1704467606 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c010670 con 0x7f4aa4108f40 2024-04-25T11:37:58.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.495+0000 7f4aa8a05640 1 -- 172.21.15.90:0/1704467606 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aa41a5080 con 0x7f4aa4108f40 2024-04-25T11:37:58.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.498+0000 7f4a9b7fe640 1 -- 172.21.15.90:0/1704467606 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f4a9c010810 con 0x7f4aa4108f40 2024-04-25T11:37:58.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.498+0000 7f4aaf992740 1 Processor -- start 2024-04-25T11:37:58.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.498+0000 7f4aaf992740 1 -- start start 2024-04-25T11:37:58.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.498+0000 7f4aaf992740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555839eee390 0x55583a28ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:58.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.498+0000 7f4aaf992740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555839ec54d0 0x55583a28efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:58.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.498+0000 7f4aaf992740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839ecd6b0 0x55583a29b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:58.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.498+0000 7f4aaf992740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55583a1f4d80 con 0x555839ecd6b0 2024-04-25T11:37:58.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.498+0000 7f4aaf992740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55583a1ae8b0 con 0x555839eee390 2024-04-25T11:37:58.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.498+0000 7f4aaf992740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55583a1f0d40 con 0x555839ec54d0 2024-04-25T11:37:58.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.499+0000 7f4a9bfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555839eee390 0x55583a28ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:58.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.499+0000 7f4aa259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555839ec54d0 0x55583a28efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:58.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.499+0000 7f4a9bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839ecd6b0 0x55583a29b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:58.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.499+0000 7f4a9bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839ecd6b0 0x55583a29b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35256/0 (socket says 172.21.15.90:35256) 2024-04-25T11:37:58.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.499+0000 7f4a9bfff640 1 -- 172.21.15.90:0/2636496732 learned_addr learned my addr 172.21.15.90:0/2636496732 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:58.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.499+0000 7f4a9b7fe640 1 --2- 172.21.15.90:0/1704467606 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4a900afc60 0x7f4a900b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:58.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.499+0000 7f4aa259c640 1 -- 172.21.15.90:0/2636496732 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555839eee390 msgr2=0x55583a28ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:58.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.499+0000 7f4aa259c640 1 --2- 172.21.15.90:0/2636496732 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555839eee390 0x55583a28ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:58.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.499+0000 7f4aa259c640 1 -- 172.21.15.90:0/2636496732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839ecd6b0 msgr2=0x55583a29b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:58.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.499+0000 7f4aa259c640 1 --2- 172.21.15.90:0/2636496732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839ecd6b0 0x55583a29b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:58.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.499+0000 7f4aa259c640 1 -- 172.21.15.90:0/2636496732 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55583a17fa40 con 0x555839ec54d0 2024-04-25T11:37:58.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.499+0000 7f4a9bfff640 1 --2- 172.21.15.90:0/2636496732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839ecd6b0 0x55583a29b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:37:58.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.499+0000 7f4aa2d9d640 1 --2- 172.21.15.90:0/1704467606 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4a900afc60 0x7f4a900b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:58.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.499+0000 7f4a9b7fe640 1 -- 172.21.15.90:0/1704467606 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f4a9c0df640 con 0x7f4aa4108f40 2024-04-25T11:37:58.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.499+0000 7f4aa259c640 1 --2- 172.21.15.90:0/2636496732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555839ec54d0 0x55583a28efa0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4a9402a930 tx=0x7f4a94004470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bc215585707d9cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:58.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.500+0000 7f4a98ff9640 1 -- 172.21.15.90:0/2636496732 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a94035040 con 0x555839ec54d0 2024-04-25T11:37:58.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.500+0000 7f4aa2d9d640 1 --2- 172.21.15.90:0/1704467606 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4a900afc60 0x7f4a900b2170 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c0dd330 tx=0x7f4a9c05a040 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:58.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.500+0000 7f4a98ff9640 1 -- 172.21.15.90:0/2636496732 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a94004770 con 0x555839ec54d0 2024-04-25T11:37:58.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.500+0000 7f4a98ff9640 1 -- 172.21.15.90:0/2636496732 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a94003830 con 0x555839ec54d0 2024-04-25T11:37:58.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.501+0000 7f4aaf992740 1 -- 172.21.15.90:0/2636496732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555839ec54d0 msgr2=0x55583a28efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:58.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.501+0000 7f4aaf992740 1 --2- 172.21.15.90:0/2636496732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555839ec54d0 0x55583a28efa0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4a9402a930 tx=0x7f4a94004470 comp rx=0 tx=0).stop 2024-04-25T11:37:58.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.501+0000 7f4aaf992740 1 -- 172.21.15.90:0/2636496732 shutdown_connections 2024-04-25T11:37:58.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.501+0000 7f4aaf992740 1 --2- 172.21.15.90:0/2636496732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839ecd6b0 0x55583a29b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:58.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.501+0000 7f4aaf992740 1 --2- 172.21.15.90:0/2636496732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555839ec54d0 0x55583a28efa0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:58.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.501+0000 7f4aaf992740 1 --2- 172.21.15.90:0/2636496732 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555839eee390 0x55583a28ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:58.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.501+0000 7f4aaf992740 1 -- 172.21.15.90:0/2636496732 >> 172.21.15.90:0/2636496732 conn(0x555839ee24e0 msgr2=0x55583a28b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:58.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.501+0000 7f4aaf992740 1 -- 172.21.15.90:0/2636496732 shutdown_connections 2024-04-25T11:37:58.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.501+0000 7f4aaf992740 1 -- 172.21.15.90:0/2636496732 wait complete. 2024-04-25T11:37:58.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.501+0000 7f4aaf992740 1 -- start start 2024-04-25T11:37:58.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.501+0000 7f4aaf992740 1 Processor -- start 2024-04-25T11:37:58.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.502+0000 7f4aaf992740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555839ec54d0 0x55583a29afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:58.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.502+0000 7f4aaf992740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555839ecd6b0 0x55583a29b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:58.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.502+0000 7f4aaf992740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839eee390 0x55583a2994a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:58.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.502+0000 7f4aaf992740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55583a29bf80 con 0x555839eee390 2024-04-25T11:37:58.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.502+0000 7f4aaf992740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55583a17bd10 con 0x555839ecd6b0 2024-04-25T11:37:58.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.502+0000 7f4aaf992740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55583a29c100 con 0x555839ec54d0 2024-04-25T11:37:58.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.502+0000 7f4aaf992740 10 client.0 _ll_register_callbacks cb 0x55583a1a8210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:37:58.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.502+0000 7f4aa259c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555839ecd6b0 0x55583a29b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:58.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.502+0000 7f4a9bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839eee390 0x55583a2994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:58.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.502+0000 7f4aa259c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555839ecd6b0 0x55583a29b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:52862/0 (socket says 172.21.15.90:52862) 2024-04-25T11:37:58.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.502+0000 7f4a9bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839eee390 0x55583a2994a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35266/0 (socket says 172.21.15.90:35266) 2024-04-25T11:37:58.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.502+0000 7f4aa259c640 1 -- 172.21.15.90:0/3396778623 learned_addr learned my addr 172.21.15.90:0/3396778623 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:58.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.502+0000 7f4a9bfff640 1 --2- 172.21.15.90:0/3396778623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555839ec54d0 0x55583a29afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:58.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.502+0000 7f4aa259c640 1 -- 172.21.15.90:0/3396778623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555839ec54d0 msgr2=0x55583a29afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:58.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.502+0000 7f4aa259c640 1 --2- 172.21.15.90:0/3396778623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555839ec54d0 0x55583a29afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:58.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.502+0000 7f4aa259c640 1 -- 172.21.15.90:0/3396778623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839eee390 msgr2=0x55583a2994a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:58.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.502+0000 7f4aa259c640 1 --2- 172.21.15.90:0/3396778623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839eee390 0x55583a2994a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:58.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.502+0000 7f4aa259c640 1 -- 172.21.15.90:0/3396778623 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555839fd07d0 con 0x555839ecd6b0 2024-04-25T11:37:58.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.503+0000 7f4aa259c640 1 --2- 172.21.15.90:0/3396778623 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555839ecd6b0 0x55583a29b4f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4a94005ab0 tx=0x7f4a94003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0061e4783471f82 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:58.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.503+0000 7f4a89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:37:58.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.503+0000 7f4a89ffb640 1 -- 172.21.15.90:0/3396778623 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a94035040 con 0x555839ecd6b0 2024-04-25T11:37:58.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.503+0000 7f4a89ffb640 1 -- 172.21.15.90:0/3396778623 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f4a94003da0 con 0x555839ecd6b0 2024-04-25T11:37:58.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.503+0000 7f4aaf992740 1 -- 172.21.15.90:0/3396778623 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a94004770 con 0x555839ecd6b0 2024-04-25T11:37:58.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.503+0000 7f4a89ffb640 1 -- 172.21.15.90:0/3396778623 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4a9405a050 con 0x555839ecd6b0 2024-04-25T11:37:58.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.503+0000 7f4aaf992740 10 client.25715 fetch_fsmap learned FSMap version 27 2024-04-25T11:37:58.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.503+0000 7f4aaf992740 1 -- 172.21.15.90:0/3396778623 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555839f063f0 con 0x555839ecd6b0 2024-04-25T11:37:58.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.503+0000 7f4a89ffb640 1 -- 172.21.15.90:0/3396778623 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a94004770 con 0x555839ecd6b0 2024-04-25T11:37:58.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.503+0000 7f4a89ffb640 1 -- 172.21.15.90:0/3396778623 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f4a94055030 con 0x555839ecd6b0 2024-04-25T11:37:58.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.503+0000 7f4aaf992740 10 client.25715 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:37:58.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.503+0000 7f4aaf992740 10 client.25715 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:37:58.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.503+0000 7f4aaf992740 20 client.25715 populate_metadata read hostname 'smithi090' 2024-04-25T11:37:58.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.503+0000 7f4aaf992740 1 --2- 172.21.15.90:0/3396778623 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x555839eacf60 0x55583a2ab540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:58.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.503+0000 7f4aaf992740 4 client.25715 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:37:58.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.503+0000 7f4aaf992740 1 -- 172.21.15.90:0/3396778623 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55583a202990 con 0x555839eacf60 2024-04-25T11:37:58.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.504+0000 7f4a9bfff640 1 --2- 172.21.15.90:0/3396778623 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x555839eacf60 0x55583a2ab540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:58.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.504+0000 7f4a89ffb640 10 client.25715 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:58.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:58.504+0000 7f4a9bfff640 1 --2- 172.21.15.90:0/3396778623 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x555839eacf60 0x55583a2ab540 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d17a2fac93ba74b9 server_cookie=d9de1999a150a47c in_seq=0 out_seq=0 2024-04-25T11:37:59.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.462+0000 7f83e5ffb640 1 -- 172.21.15.90:0/1137084606 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f83f00a7630 con 0x55f807f401c0 2024-04-25T11:37:59.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.462+0000 7f83e5ffb640 10 client.25645 handle_command_reply: tid=0 2024-04-25T11:37:59.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.462+0000 7f840ba1e740 1 client.25645 shutdown 2024-04-25T11:37:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.463+0000 7f840ba1e740 1 -- 172.21.15.90:0/1137084606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f807fac1b0 msgr2=0x55f808362010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.463+0000 7f840ba1e740 1 --2- 172.21.15.90:0/1137084606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f807fac1b0 0x55f808362010 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f83f8004410 tx=0x7f83f8004ab0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.463+0000 7f840ba1e740 1 -- 172.21.15.90:0/1137084606 shutdown_connections 2024-04-25T11:37:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.463+0000 7f840ba1e740 1 --2- 172.21.15.90:0/1137084606 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f807f401c0 0x55f8083d2f60 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.463+0000 7f840ba1e740 1 --2- 172.21.15.90:0/1137084606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f807fac1b0 0x55f808362010 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.463+0000 7f840ba1e740 1 --2- 172.21.15.90:0/1137084606 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f807f8b4d0 0x55f8083c8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.463+0000 7f840ba1e740 1 --2- 172.21.15.90:0/1137084606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f807f832f0 0x55f8083c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.463+0000 7f840ba1e740 1 -- 172.21.15.90:0/1137084606 >> 172.21.15.90:0/1137084606 conn(0x55f807fa0300 msgr2=0x55f808354840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.463+0000 7f83e5ffb640 0 client.25645 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.463+0000 7f840ba1e740 1 -- 172.21.15.90:0/1137084606 shutdown_connections 2024-04-25T11:37:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.463+0000 7f840ba1e740 1 -- 172.21.15.90:0/1137084606 wait complete. 2024-04-25T11:37:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.463+0000 7f840ba1e740 20 client.25645 trim_cache size 0 max 16384 2024-04-25T11:37:59.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.463+0000 7f4a89ffb640 1 -- 172.21.15.90:0/3396778623 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55583a202990 con 0x555839eacf60 2024-04-25T11:37:59.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.463+0000 7f4a89ffb640 10 client.25715 handle_command_reply: tid=0 2024-04-25T11:37:59.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.463+0000 7f4aaf992740 1 client.25715 shutdown 2024-04-25T11:37:59.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:37:59.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:37:59.454 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:37:59.454 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:37:59.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.463+0000 7f4aaf992740 1 -- 172.21.15.90:0/3396778623 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555839ecd6b0 msgr2=0x55583a29b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:59.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.463+0000 7f4aaf992740 1 --2- 172.21.15.90:0/3396778623 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555839ecd6b0 0x55583a29b4f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4a94005ab0 tx=0x7f4a94003820 comp rx=0 tx=0).stop 2024-04-25T11:37:59.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.463+0000 7f4aaf992740 1 -- 172.21.15.90:0/3396778623 shutdown_connections 2024-04-25T11:37:59.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.463+0000 7f4aaf992740 1 --2- 172.21.15.90:0/3396778623 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x555839eacf60 0x55583a2ab540 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.463+0000 7f4aaf992740 1 --2- 172.21.15.90:0/3396778623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839eee390 0x55583a2994a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.463+0000 7f4aaf992740 1 --2- 172.21.15.90:0/3396778623 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555839ecd6b0 0x55583a29b4f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.464+0000 7f4aaf992740 1 --2- 172.21.15.90:0/3396778623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555839ec54d0 0x55583a29afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.464+0000 7f4a89ffb640 0 client.25715 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:59.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.464+0000 7f4aaf992740 1 -- 172.21.15.90:0/3396778623 >> 172.21.15.90:0/3396778623 conn(0x555839ee24e0 msgr2=0x55583a290780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:59.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.464+0000 7f4aaf992740 1 -- 172.21.15.90:0/3396778623 shutdown_connections 2024-04-25T11:37:59.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.464+0000 7f4aaf992740 1 -- 172.21.15.90:0/3396778623 wait complete. 2024-04-25T11:37:59.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.464+0000 7f4aaf992740 20 client.25715 trim_cache size 0 max 16384 2024-04-25T11:37:59.456 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.464+0000 7f8404b72640 1 -- 172.21.15.90:0/602089864 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f83ec0afc50 msgr2=0x7f83ec0b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:59.456 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.464+0000 7f8404b72640 1 --2- 172.21.15.90:0/602089864 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f83ec0afc50 0x7f83ec0b2160 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f83f8005fd0 tx=0x7f83f8003d40 comp rx=0 tx=0).stop 2024-04-25T11:37:59.456 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.464+0000 7f8404b72640 1 -- 172.21.15.90:0/602089864 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84001033f0 msgr2=0x7f840019a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:59.456 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.464+0000 7f8404b72640 1 --2- 172.21.15.90:0/602089864 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84001033f0 0x7f840019a760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f83f000e9c0 tx=0x7f83f000ee80 comp rx=0 tx=0).stop 2024-04-25T11:37:59.456 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.464+0000 7f8404b72640 1 -- 172.21.15.90:0/602089864 shutdown_connections 2024-04-25T11:37:59.456 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.464+0000 7f8404b72640 1 --2- 172.21.15.90:0/602089864 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f83ec0afc50 0x7f83ec0b2160 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.456 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.464+0000 7f8404b72640 1 --2- 172.21.15.90:0/602089864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8400107d30 0x7f840019ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.456 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.464+0000 7f8404b72640 1 --2- 172.21.15.90:0/602089864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84001072b0 0x7f840019acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.456 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.464+0000 7f8404b72640 1 --2- 172.21.15.90:0/602089864 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84001033f0 0x7f840019a760 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f83f000e9c0 tx=0x7f83f000ee80 comp rx=0 tx=0).stop 2024-04-25T11:37:59.456 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.464+0000 7f8404b72640 1 -- 172.21.15.90:0/602089864 >> 172.21.15.90:0/602089864 conn(0x7f84000778b0 msgr2=0x7f84000ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:59.457 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.464+0000 7f8404b72640 1 -- 172.21.15.90:0/602089864 shutdown_connections 2024-04-25T11:37:59.457 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:37:59.464+0000 7f8404b72640 1 -- 172.21.15.90:0/602089864 wait complete. 2024-04-25T11:37:59.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.482+0000 7f4aaf992740 1 Processor -- start 2024-04-25T11:37:59.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.483+0000 7f4aaf992740 1 -- start start 2024-04-25T11:37:59.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.483+0000 7f4aaf992740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555839ec54d0 0x55583a294ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:59.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.483+0000 7f4aaf992740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555839ecd6b0 0x55583a299440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:59.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.483+0000 7f4aaf992740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839eee390 0x55583a2ae570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:59.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.483+0000 7f4aaf992740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa41ad980 con 0x555839eee390 2024-04-25T11:37:59.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.483+0000 7f4aaf992740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa41af4f0 con 0x555839ec54d0 2024-04-25T11:37:59.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.483+0000 7f4aaf992740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa41ae050 con 0x555839ecd6b0 2024-04-25T11:37:59.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.483+0000 7f4a9bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839eee390 0x55583a2ae570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:59.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.483+0000 7f4a9bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839eee390 0x55583a2ae570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35270/0 (socket says 172.21.15.90:35270) 2024-04-25T11:37:59.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.483+0000 7f4a9bfff640 1 -- 172.21.15.90:0/1219510981 learned_addr learned my addr 172.21.15.90:0/1219510981 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:59.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.483+0000 7f4aa259c640 1 --2- 172.21.15.90:0/1219510981 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555839ecd6b0 0x55583a299440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:59.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.483+0000 7f4a9bfff640 1 --2- 172.21.15.90:0/1219510981 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555839ec54d0 0x55583a294ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:59.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.483+0000 7f4a9bfff640 1 -- 172.21.15.90:0/1219510981 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555839ecd6b0 msgr2=0x55583a299440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:59.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.483+0000 7f4a9bfff640 1 --2- 172.21.15.90:0/1219510981 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555839ecd6b0 0x55583a299440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.483+0000 7f4a9bfff640 1 -- 172.21.15.90:0/1219510981 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555839ec54d0 msgr2=0x55583a294ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:59.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.483+0000 7f4a9bfff640 1 --2- 172.21.15.90:0/1219510981 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555839ec54d0 0x55583a294ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.483+0000 7f4a9bfff640 1 -- 172.21.15.90:0/1219510981 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555839ec60f0 con 0x555839eee390 2024-04-25T11:37:59.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.484+0000 7f4a9bfff640 1 --2- 172.21.15.90:0/1219510981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839eee390 0x55583a2ae570 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00f010 tx=0x7f4a8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa33c8a732a8310c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:59.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.484+0000 7f4a8bfff640 1 -- 172.21.15.90:0/1219510981 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c02ee60 con 0x555839eee390 2024-04-25T11:37:59.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.484+0000 7f4a8bfff640 1 -- 172.21.15.90:0/1219510981 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a8c032070 con 0x555839eee390 2024-04-25T11:37:59.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.484+0000 7f4a8bfff640 1 -- 172.21.15.90:0/1219510981 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c00c7f0 con 0x555839eee390 2024-04-25T11:37:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.485+0000 7f4aaf992740 1 -- 172.21.15.90:0/1219510981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839eee390 msgr2=0x55583a2ae570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.485+0000 7f4aaf992740 1 --2- 172.21.15.90:0/1219510981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839eee390 0x55583a2ae570 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00f010 tx=0x7f4a8c0046c0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.485+0000 7f4aaf992740 1 -- 172.21.15.90:0/1219510981 shutdown_connections 2024-04-25T11:37:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.485+0000 7f4aaf992740 1 --2- 172.21.15.90:0/1219510981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839eee390 0x55583a2ae570 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.485+0000 7f4aaf992740 1 --2- 172.21.15.90:0/1219510981 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555839ecd6b0 0x55583a299440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.485+0000 7f4aaf992740 1 --2- 172.21.15.90:0/1219510981 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555839ec54d0 0x55583a294ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.485+0000 7f4aaf992740 1 -- 172.21.15.90:0/1219510981 >> 172.21.15.90:0/1219510981 conn(0x555839ee24e0 msgr2=0x55583a298d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.485+0000 7f4aaf992740 1 -- 172.21.15.90:0/1219510981 shutdown_connections 2024-04-25T11:37:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.485+0000 7f4aaf992740 1 -- 172.21.15.90:0/1219510981 wait complete. 2024-04-25T11:37:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.485+0000 7f4aaf992740 1 -- start start 2024-04-25T11:37:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.485+0000 7f4aaf992740 1 Processor -- start 2024-04-25T11:37:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.486+0000 7f4aaf992740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839ec54d0 0x55583a2ad200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.486+0000 7f4aaf992740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555839ecd6b0 0x55583a2ad740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.486+0000 7f4aaf992740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555839eee390 0x55583a2adc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.486+0000 7f4aaf992740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa41aedf0 con 0x555839ec54d0 2024-04-25T11:37:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.486+0000 7f4aaf992740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa41ae720 con 0x555839ecd6b0 2024-04-25T11:37:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.486+0000 7f4aaf992740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa41ab960 con 0x555839eee390 2024-04-25T11:37:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.486+0000 7f4aaf992740 10 client.0 _ll_register_callbacks cb 0x7f4a8c00c370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:37:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.486+0000 7f4a9bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839ec54d0 0x55583a2ad200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.486+0000 7f4a9bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839ec54d0 0x55583a2ad200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35280/0 (socket says 172.21.15.90:35280) 2024-04-25T11:37:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.486+0000 7f4a9bfff640 1 -- 172.21.15.90:0/1924181552 learned_addr learned my addr 172.21.15.90:0/1924181552 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.486+0000 7f4aa259c640 1 --2- 172.21.15.90:0/1924181552 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555839ecd6b0 0x55583a2ad740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.486+0000 7f4a9bfff640 1 --2- 172.21.15.90:0/1924181552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555839eee390 0x55583a2adc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.486+0000 7f4a9bfff640 1 -- 172.21.15.90:0/1924181552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555839eee390 msgr2=0x55583a2adc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.486+0000 7f4a9bfff640 1 --2- 172.21.15.90:0/1924181552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555839eee390 0x55583a2adc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.486+0000 7f4a9bfff640 1 -- 172.21.15.90:0/1924181552 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555839ecd6b0 msgr2=0x55583a2ad740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.486+0000 7f4a9bfff640 1 --2- 172.21.15.90:0/1924181552 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555839ecd6b0 0x55583a2ad740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.486+0000 7f4a9bfff640 1 -- 172.21.15.90:0/1924181552 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55583a1b54b0 con 0x555839ec54d0 2024-04-25T11:37:59.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.486+0000 7f4aa259c640 1 --2- 172.21.15.90:0/1924181552 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555839ecd6b0 0x55583a2ad740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:37:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.486+0000 7f4a9bfff640 1 --2- 172.21.15.90:0/1924181552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839ec54d0 0x55583a2ad200 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c022f40 tx=0x7f4a8c005b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8dc8c216b7904b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.486+0000 7f4a89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:37:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.487+0000 7f4a89ffb640 1 -- 172.21.15.90:0/1924181552 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c004390 con 0x555839ec54d0 2024-04-25T11:37:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.487+0000 7f4a89ffb640 1 -- 172.21.15.90:0/1924181552 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f4a8c005050 con 0x555839ec54d0 2024-04-25T11:37:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.487+0000 7f4aaf992740 1 -- 172.21.15.90:0/1924181552 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a8c032070 con 0x555839ec54d0 2024-04-25T11:37:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.487+0000 7f4a89ffb640 1 -- 172.21.15.90:0/1924181552 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4a8c04b050 con 0x555839ec54d0 2024-04-25T11:37:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.487+0000 7f4aaf992740 10 client.16404 fetch_fsmap learned FSMap version 27 2024-04-25T11:37:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.487+0000 7f4aaf992740 1 -- 172.21.15.90:0/1924181552 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55583a312780 con 0x555839ec54d0 2024-04-25T11:37:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.487+0000 7f4a89ffb640 1 -- 172.21.15.90:0/1924181552 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a8c032070 con 0x555839ec54d0 2024-04-25T11:37:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.487+0000 7f4a89ffb640 1 -- 172.21.15.90:0/1924181552 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f4a8c00cce0 con 0x555839ec54d0 2024-04-25T11:37:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.487+0000 7f4aaf992740 10 client.16404 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:37:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.487+0000 7f4aaf992740 10 client.16404 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:37:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.487+0000 7f4aaf992740 20 client.16404 populate_metadata read hostname 'smithi090' 2024-04-25T11:37:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.487+0000 7f4aaf992740 1 --2- 172.21.15.90:0/1924181552 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x555839e823a0 0x55583a314dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.487+0000 7f4aaf992740 4 client.16404 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:37:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.487+0000 7f4aaf992740 1 -- 172.21.15.90:0/1924181552 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55583a1fc800 con 0x555839e823a0 2024-04-25T11:37:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.487+0000 7f4aa259c640 1 --2- 172.21.15.90:0/1924181552 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x555839e823a0 0x55583a314dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.488+0000 7f4a89ffb640 10 client.16404 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.488+0000 7f4aa259c640 1 --2- 172.21.15.90:0/1924181552 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x555839e823a0 0x55583a314dd0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=994a6c7ee50ac27e server_cookie=b6c98b9df8b6fe5b in_seq=0 out_seq=0 2024-04-25T11:37:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.488+0000 7f4a89ffb640 1 -- 172.21.15.90:0/1924181552 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55583a1fc800 con 0x555839e823a0 2024-04-25T11:37:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.488+0000 7f4a89ffb640 10 client.16404 handle_command_reply: tid=0 2024-04-25T11:37:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.488+0000 7f4aaf992740 1 client.16404 shutdown 2024-04-25T11:37:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.488+0000 7f4aaf992740 1 -- 172.21.15.90:0/1924181552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839ec54d0 msgr2=0x55583a2ad200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.488+0000 7f4aaf992740 1 --2- 172.21.15.90:0/1924181552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839ec54d0 0x55583a2ad200 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c022f40 tx=0x7f4a8c005b90 comp rx=0 tx=0).stop 2024-04-25T11:37:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.488+0000 7f4aaf992740 1 -- 172.21.15.90:0/1924181552 shutdown_connections 2024-04-25T11:37:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.488+0000 7f4aaf992740 1 --2- 172.21.15.90:0/1924181552 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x555839e823a0 0x55583a314dd0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.488+0000 7f4aaf992740 1 --2- 172.21.15.90:0/1924181552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555839eee390 0x55583a2adc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.488+0000 7f4a89ffb640 0 client.16404 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:37:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.488+0000 7f4aaf992740 1 --2- 172.21.15.90:0/1924181552 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555839ecd6b0 0x55583a2ad740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.488+0000 7f4aaf992740 1 --2- 172.21.15.90:0/1924181552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555839ec54d0 0x55583a2ad200 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.488+0000 7f4aaf992740 1 -- 172.21.15.90:0/1924181552 >> 172.21.15.90:0/1924181552 conn(0x555839ee24e0 msgr2=0x55583a29ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.488+0000 7f4aaf992740 1 -- 172.21.15.90:0/1924181552 shutdown_connections 2024-04-25T11:37:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.488+0000 7f4aaf992740 1 -- 172.21.15.90:0/1924181552 wait complete. 2024-04-25T11:37:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.488+0000 7f4aaf992740 20 client.16404 trim_cache size 0 max 16384 2024-04-25T11:37:59.482 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:37:59.482 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-25T11:37:59.482 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-25T11:37:59.482 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:37:59.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.489+0000 7f4aa8a05640 1 -- 172.21.15.90:0/1704467606 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4a900afc60 msgr2=0x7f4a900b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:59.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.489+0000 7f4aa8a05640 1 --2- 172.21.15.90:0/1704467606 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4a900afc60 0x7f4a900b2170 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c0dd330 tx=0x7f4a9c05a040 comp rx=0 tx=0).stop 2024-04-25T11:37:59.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.489+0000 7f4aa8a05640 1 -- 172.21.15.90:0/1704467606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4aa4108f40 msgr2=0x7f4aa419cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:59.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.489+0000 7f4aa8a05640 1 --2- 172.21.15.90:0/1704467606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4aa4108f40 0x7f4aa419cdd0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c00eb10 tx=0x7f4a9c00c2d0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.489+0000 7f4aa8a05640 1 -- 172.21.15.90:0/1704467606 shutdown_connections 2024-04-25T11:37:59.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.489+0000 7f4aa8a05640 1 --2- 172.21.15.90:0/1704467606 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4a900afc60 0x7f4a900b2170 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.489+0000 7f4aa8a05640 1 --2- 172.21.15.90:0/1704467606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4aa4108f40 0x7f4aa419cdd0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.489+0000 7f4aa8a05640 1 --2- 172.21.15.90:0/1704467606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4108aa0 0x7f4aa4198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.489+0000 7f4aa8a05640 1 --2- 172.21.15.90:0/1704467606 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4aa4103850 0x7f4aa41985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.489+0000 7f4aa8a05640 1 -- 172.21.15.90:0/1704467606 >> 172.21.15.90:0/1704467606 conn(0x7f4aa4077880 msgr2=0x7f4aa41094a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:59.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.489+0000 7f4aa8a05640 1 -- 172.21.15.90:0/1704467606 shutdown_connections 2024-04-25T11:37:59.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.489+0000 7f4aa8a05640 1 -- 172.21.15.90:0/1704467606 wait complete. 2024-04-25T11:37:59.489 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T11:37:59.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:37:59.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.622+0000 7fd0bc2e0640 1 -- 172.21.15.90:0/1197451333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0b41090a0 msgr2=0x7fd0b4103210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:59.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.622+0000 7fd0bc2e0640 1 --2- 172.21.15.90:0/1197451333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0b41090a0 0x7fd0b4103210 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4009860 tx=0x7fd0a402a340 comp rx=0 tx=0).stop 2024-04-25T11:37:59.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.622+0000 7fd0bc2e0640 1 -- 172.21.15.90:0/1197451333 shutdown_connections 2024-04-25T11:37:59.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.622+0000 7fd0bc2e0640 1 --2- 172.21.15.90:0/1197451333 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd0b4103880 0x7fd0b410f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.622+0000 7fd0bc2e0640 1 --2- 172.21.15.90:0/1197451333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0b41090a0 0x7fd0b4103210 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.622+0000 7fd0bc2e0640 1 --2- 172.21.15.90:0/1197451333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0b4108c00 0x7fd0b4102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.622+0000 7fd0bc2e0640 1 -- 172.21.15.90:0/1197451333 >> 172.21.15.90:0/1197451333 conn(0x7fd0b40fd810 msgr2=0x7fd0b40ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:59.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.622+0000 7fd0bc2e0640 1 -- 172.21.15.90:0/1197451333 shutdown_connections 2024-04-25T11:37:59.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.622+0000 7fd0bc2e0640 1 -- 172.21.15.90:0/1197451333 wait complete. 2024-04-25T11:37:59.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.623+0000 7fd0bc2e0640 1 Processor -- start 2024-04-25T11:37:59.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.623+0000 7fd0bc2e0640 1 -- start start 2024-04-25T11:37:59.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.623+0000 7fd0bc2e0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0b4103880 0x7fd0b419b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:59.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.623+0000 7fd0bc2e0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0b4108c00 0x7fd0b419bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:59.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.623+0000 7fd0bc2e0640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd0b41090a0 0x7fd0b419c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:59.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.623+0000 7fd0bc2e0640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b4112e20 con 0x7fd0b4108c00 2024-04-25T11:37:59.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.623+0000 7fd0bc2e0640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b4112ca0 con 0x7fd0b41090a0 2024-04-25T11:37:59.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.623+0000 7fd0bc2e0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b4112fa0 con 0x7fd0b4103880 2024-04-25T11:37:59.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.623+0000 7fd0b987b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0b4108c00 0x7fd0b419bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:59.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.623+0000 7fd0ba87d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd0b41090a0 0x7fd0b419c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:59.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.623+0000 7fd0ba87d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd0b41090a0 0x7fd0b419c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:52876/0 (socket says 172.21.15.90:52876) 2024-04-25T11:37:59.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.623+0000 7fd0b987b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0b4108c00 0x7fd0b419bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35302/0 (socket says 172.21.15.90:35302) 2024-04-25T11:37:59.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.623+0000 7fd0ba87d640 1 -- 172.21.15.90:0/2489337666 learned_addr learned my addr 172.21.15.90:0/2489337666 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:37:59.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.624+0000 7fd0ba07c640 1 --2- 172.21.15.90:0/2489337666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0b4103880 0x7fd0b419b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:59.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.624+0000 7fd0ba87d640 1 -- 172.21.15.90:0/2489337666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0b4103880 msgr2=0x7fd0b419b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:59.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.624+0000 7fd0ba87d640 1 --2- 172.21.15.90:0/2489337666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0b4103880 0x7fd0b419b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.624+0000 7fd0ba87d640 1 -- 172.21.15.90:0/2489337666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0b4108c00 msgr2=0x7fd0b419bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:59.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.624+0000 7fd0ba87d640 1 --2- 172.21.15.90:0/2489337666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0b4108c00 0x7fd0b419bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.624+0000 7fd0ba87d640 1 -- 172.21.15.90:0/2489337666 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0b41a6a70 con 0x7fd0b41090a0 2024-04-25T11:37:59.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.624+0000 7fd0b987b640 1 --2- 172.21.15.90:0/2489337666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0b4108c00 0x7fd0b419bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:37:59.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.624+0000 7fd0ba87d640 1 --2- 172.21.15.90:0/2489337666 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd0b41090a0 0x7fd0b419c1e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac00eb10 tx=0x7fd0ac00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:59.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.624+0000 7fd0ab7fe640 1 -- 172.21.15.90:0/2489337666 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac004b20 con 0x7fd0b41090a0 2024-04-25T11:37:59.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.624+0000 7fd0bc2e0640 1 -- 172.21.15.90:0/2489337666 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0b41a6d60 con 0x7fd0b41090a0 2024-04-25T11:37:59.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.624+0000 7fd0ab7fe640 1 -- 172.21.15.90:0/2489337666 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0ac004cc0 con 0x7fd0b41090a0 2024-04-25T11:37:59.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.624+0000 7fd0ab7fe640 1 -- 172.21.15.90:0/2489337666 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac010690 con 0x7fd0b41090a0 2024-04-25T11:37:59.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.624+0000 7fd0bc2e0640 1 -- 172.21.15.90:0/2489337666 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0b41a71c0 con 0x7fd0b41090a0 2024-04-25T11:37:59.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.625+0000 7fd0bc2e0640 1 -- 172.21.15.90:0/2489337666 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0b4069310 con 0x7fd0b41090a0 2024-04-25T11:37:59.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.626+0000 7fd0ab7fe640 1 -- 172.21.15.90:0/2489337666 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fd0ac005000 con 0x7fd0b41090a0 2024-04-25T11:37:59.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.627+0000 7fd0ab7fe640 1 --2- 172.21.15.90:0/2489337666 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd0a00afc60 0x7fd0a00b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:59.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.627+0000 7fd0ba07c640 1 --2- 172.21.15.90:0/2489337666 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd0a00afc60 0x7fd0a00b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:59.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.627+0000 7fd0ab7fe640 1 -- 172.21.15.90:0/2489337666 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fd0ac0df910 con 0x7fd0b41090a0 2024-04-25T11:37:59.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.628+0000 7fd0ba07c640 1 --2- 172.21.15.90:0/2489337666 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd0a00afc60 0x7fd0a00b2170 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4075570 tx=0x7fd0b0009260 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:59.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.631+0000 7fd0ab7fe640 1 -- 172.21.15.90:0/2489337666 <== mon.2 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd0ac053ff0 con 0x7fd0b41090a0 2024-04-25T11:37:59.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:37:59 smithi090 ceph-mon[29575]: pgmap v90: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 469 MiB/s wr, 742 op/s 2024-04-25T11:37:59.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:37:59 smithi138 ceph-mon[32328]: pgmap v90: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 469 MiB/s wr, 742 op/s 2024-04-25T11:37:59.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:37:59 smithi156 ceph-mon[33045]: pgmap v90: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 469 MiB/s wr, 742 op/s 2024-04-25T11:37:59.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.968+0000 7fd0bc2e0640 1 -- 172.21.15.90:0/2489337666 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0b41021b0 con 0x7fd0b41090a0 2024-04-25T11:37:59.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.969+0000 7fd0ab7fe640 1 -- 172.21.15.90:0/2489337666 <== mon.2 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11931 (secure 0 0 0) 0x7fd0ac0a80f0 con 0x7fd0b41090a0 2024-04-25T11:37:59.957 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:37:59.958 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:37:15.460907+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24475,24524,24499,24431,24458]},"id":1}]} 2024-04-25T11:37:59.958 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-25T11:37:59.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.971+0000 7fd0bc2e0640 1 -- 172.21.15.90:0/2489337666 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd0a00afc60 msgr2=0x7fd0a00b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:59.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.971+0000 7fd0bc2e0640 1 --2- 172.21.15.90:0/2489337666 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd0a00afc60 0x7fd0a00b2170 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4075570 tx=0x7fd0b0009260 comp rx=0 tx=0).stop 2024-04-25T11:37:59.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.971+0000 7fd0bc2e0640 1 -- 172.21.15.90:0/2489337666 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd0b41090a0 msgr2=0x7fd0b419c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:59.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.971+0000 7fd0bc2e0640 1 --2- 172.21.15.90:0/2489337666 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd0b41090a0 0x7fd0b419c1e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac00eb10 tx=0x7fd0ac00c2d0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.971+0000 7fd0bc2e0640 1 -- 172.21.15.90:0/2489337666 shutdown_connections 2024-04-25T11:37:59.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.971+0000 7fd0bc2e0640 1 --2- 172.21.15.90:0/2489337666 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd0a00afc60 0x7fd0a00b2170 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.971+0000 7fd0bc2e0640 1 --2- 172.21.15.90:0/2489337666 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd0b41090a0 0x7fd0b419c1e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.971+0000 7fd0bc2e0640 1 --2- 172.21.15.90:0/2489337666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0b4108c00 0x7fd0b419bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.971+0000 7fd0bc2e0640 1 --2- 172.21.15.90:0/2489337666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0b4103880 0x7fd0b419b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:59.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.971+0000 7fd0bc2e0640 1 -- 172.21.15.90:0/2489337666 >> 172.21.15.90:0/2489337666 conn(0x7fd0b40fd810 msgr2=0x7fd0b40ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:59.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.971+0000 7fd0bc2e0640 1 -- 172.21.15.90:0/2489337666 shutdown_connections 2024-04-25T11:37:59.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:37:59.971+0000 7fd0bc2e0640 1 -- 172.21.15.90:0/2489337666 wait complete. 2024-04-25T11:37:59.972 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T11:38:00.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.104+0000 7f31c47b2640 1 -- 172.21.15.90:0/2213226104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31bc10e070 msgr2=0x7f31bc110560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.104+0000 7f31c47b2640 1 --2- 172.21.15.90:0/2213226104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31bc10e070 0x7f31bc110560 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f31b4009ca0 tx=0x7f31b402a950 comp rx=0 tx=0).stop 2024-04-25T11:38:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.104+0000 7f31c47b2640 1 -- 172.21.15.90:0/2213226104 shutdown_connections 2024-04-25T11:38:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.104+0000 7f31c47b2640 1 --2- 172.21.15.90:0/2213226104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31bc10e070 0x7f31bc110560 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.104+0000 7f31c47b2640 1 --2- 172.21.15.90:0/2213226104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31bc10b680 0x7f31bc10daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.104+0000 7f31c47b2640 1 --2- 172.21.15.90:0/2213226104 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31bc104c20 0x7f31bc105030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.104+0000 7f31c47b2640 1 -- 172.21.15.90:0/2213226104 >> 172.21.15.90:0/2213226104 conn(0x7f31bc078180 msgr2=0x7f31bc0785c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.104+0000 7f31c47b2640 1 -- 172.21.15.90:0/2213226104 shutdown_connections 2024-04-25T11:38:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.104+0000 7f31c47b2640 1 -- 172.21.15.90:0/2213226104 wait complete. 2024-04-25T11:38:00.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.105+0000 7f31c47b2640 1 Processor -- start 2024-04-25T11:38:00.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.105+0000 7f31c47b2640 1 -- start start 2024-04-25T11:38:00.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.105+0000 7f31c47b2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31bc104c20 0x7f31bc19a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.105+0000 7f31c47b2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31bc10b680 0x7f31bc19ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.105+0000 7f31c47b2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31bc10e070 0x7f31bc19eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.105+0000 7f31c47b2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f31bc113cd0 con 0x7f31bc10e070 2024-04-25T11:38:00.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.105+0000 7f31c47b2640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f31bc113b50 con 0x7f31bc10b680 2024-04-25T11:38:00.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.105+0000 7f31c47b2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f31bc113e50 con 0x7f31bc104c20 2024-04-25T11:38:00.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.105+0000 7f31c2d4f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31bc10e070 0x7f31bc19eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.105+0000 7f31c2d4f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31bc10e070 0x7f31bc19eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35318/0 (socket says 172.21.15.90:35318) 2024-04-25T11:38:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.105+0000 7f31c2d4f640 1 -- 172.21.15.90:0/245115637 learned_addr learned my addr 172.21.15.90:0/245115637 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.105+0000 7f31c1d4d640 1 --2- 172.21.15.90:0/245115637 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31bc10b680 0x7f31bc19ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.105+0000 7f31c254e640 1 --2- 172.21.15.90:0/245115637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31bc104c20 0x7f31bc19a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.106+0000 7f31c2d4f640 1 -- 172.21.15.90:0/245115637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31bc104c20 msgr2=0x7f31bc19a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.106+0000 7f31c2d4f640 1 --2- 172.21.15.90:0/245115637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31bc104c20 0x7f31bc19a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.106+0000 7f31c2d4f640 1 -- 172.21.15.90:0/245115637 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31bc10b680 msgr2=0x7f31bc19ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.106+0000 7f31c2d4f640 1 --2- 172.21.15.90:0/245115637 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31bc10b680 0x7f31bc19ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.106+0000 7f31c2d4f640 1 -- 172.21.15.90:0/245115637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31bc19f510 con 0x7f31bc10e070 2024-04-25T11:38:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.106+0000 7f31c254e640 1 --2- 172.21.15.90:0/245115637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31bc104c20 0x7f31bc19a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:38:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.106+0000 7f31c2d4f640 1 --2- 172.21.15.90:0/245115637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31bc10e070 0x7f31bc19eed0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f31b4004af0 tx=0x7f31b4031f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.106+0000 7f31b37fe640 1 -- 172.21.15.90:0/245115637 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b4034040 con 0x7f31bc10e070 2024-04-25T11:38:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.106+0000 7f31b37fe640 1 -- 172.21.15.90:0/245115637 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31b40308f0 con 0x7f31bc10e070 2024-04-25T11:38:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.106+0000 7f31c47b2640 1 -- 172.21.15.90:0/245115637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31bc19f7a0 con 0x7f31bc10e070 2024-04-25T11:38:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.106+0000 7f31b37fe640 1 -- 172.21.15.90:0/245115637 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b4004600 con 0x7f31bc10e070 2024-04-25T11:38:00.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.106+0000 7f31c47b2640 1 -- 172.21.15.90:0/245115637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31bc19fce0 con 0x7f31bc10e070 2024-04-25T11:38:00.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.109+0000 7f31cb665740 1 Processor -- start 2024-04-25T11:38:00.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.110+0000 7f31b37fe640 1 -- 172.21.15.90:0/245115637 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f31b4037020 con 0x7f31bc10e070 2024-04-25T11:38:00.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.110+0000 7f31cb665740 1 -- start start 2024-04-25T11:38:00.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.110+0000 7f31cb665740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b742956390 0x55b742cf7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.110+0000 7f31cb665740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b74292d4d0 0x55b742cf7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.110+0000 7f31cb665740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7429356b0 0x55b742d046a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.110+0000 7f31cb665740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b742c5cd80 con 0x55b7429356b0 2024-04-25T11:38:00.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.110+0000 7f31cb665740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55b742c16a20 con 0x55b74292d4d0 2024-04-25T11:38:00.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.110+0000 7f31cb665740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b742c58d40 con 0x55b742956390 2024-04-25T11:38:00.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.110+0000 7f31c1d4d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7429356b0 0x55b742d046a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.110+0000 7f31c254e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b74292d4d0 0x55b742cf7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.110+0000 7f31c254e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b74292d4d0 0x55b742cf7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:52888/0 (socket says 172.21.15.90:52888) 2024-04-25T11:38:00.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.110+0000 7f31c254e640 1 -- 172.21.15.90:0/2043161119 learned_addr learned my addr 172.21.15.90:0/2043161119 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:00.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.110+0000 7f31c1d4d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7429356b0 0x55b742d046a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35334/0 (socket says 172.21.15.90:35334) 2024-04-25T11:38:00.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.110+0000 7f31c1d4d640 1 --2- 172.21.15.90:0/2043161119 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b742956390 0x55b742cf7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.111+0000 7f31b37fe640 1 --2- 172.21.15.90:0/245115637 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f31a80afce0 0x7f31a80b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.111+0000 7f31c254e640 1 -- 172.21.15.90:0/2043161119 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b742956390 msgr2=0x55b742cf7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.111+0000 7f31c254e640 1 --2- 172.21.15.90:0/2043161119 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b742956390 0x55b742cf7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.111+0000 7f31c254e640 1 -- 172.21.15.90:0/2043161119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7429356b0 msgr2=0x55b742d046a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.111+0000 7f31c254e640 1 --2- 172.21.15.90:0/2043161119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7429356b0 0x55b742d046a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.111+0000 7f31c254e640 1 -- 172.21.15.90:0/2043161119 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b742c21c30 con 0x55b74292d4d0 2024-04-25T11:38:00.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.111+0000 7f31c2d4f640 1 --2- 172.21.15.90:0/245115637 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f31a80afce0 0x7f31a80b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.111+0000 7f31b37fe640 1 -- 172.21.15.90:0/245115637 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f31b40ffc80 con 0x7f31bc10e070 2024-04-25T11:38:00.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.111+0000 7f31c254e640 1 --2- 172.21.15.90:0/2043161119 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b74292d4d0 0x55b742cf7d50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f31b8009d50 tx=0x7f31b80117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68b9df3262f30fc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.111+0000 7f31b0ff9640 1 -- 172.21.15.90:0/2043161119 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b8012450 con 0x55b74292d4d0 2024-04-25T11:38:00.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.111+0000 7f31c2d4f640 1 --2- 172.21.15.90:0/245115637 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f31a80afce0 0x7f31a80b21f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f31b40fd970 tx=0x7f31b4095730 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.111+0000 7f31b0ff9640 1 -- 172.21.15.90:0/2043161119 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31b8012aa0 con 0x55b74292d4d0 2024-04-25T11:38:00.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.111+0000 7f31b0ff9640 1 -- 172.21.15.90:0/2043161119 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b800fb60 con 0x55b74292d4d0 2024-04-25T11:38:00.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.112+0000 7f31cb665740 1 -- 172.21.15.90:0/2043161119 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b74292d4d0 msgr2=0x55b742cf7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.112+0000 7f31cb665740 1 --2- 172.21.15.90:0/2043161119 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b74292d4d0 0x55b742cf7d50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f31b8009d50 tx=0x7f31b80117b0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.112+0000 7f31cb665740 1 -- 172.21.15.90:0/2043161119 shutdown_connections 2024-04-25T11:38:00.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.112+0000 7f31cb665740 1 --2- 172.21.15.90:0/2043161119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7429356b0 0x55b742d046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.112+0000 7f31cb665740 1 --2- 172.21.15.90:0/2043161119 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b74292d4d0 0x55b742cf7d50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.112+0000 7f31cb665740 1 --2- 172.21.15.90:0/2043161119 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b742956390 0x55b742cf7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.112+0000 7f31cb665740 1 -- 172.21.15.90:0/2043161119 >> 172.21.15.90:0/2043161119 conn(0x55b74294a4e0 msgr2=0x55b742cf3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.112+0000 7f31cb665740 1 -- 172.21.15.90:0/2043161119 shutdown_connections 2024-04-25T11:38:00.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.112+0000 7f31cb665740 1 -- 172.21.15.90:0/2043161119 wait complete. 2024-04-25T11:38:00.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.113+0000 7f31cb665740 1 -- start start 2024-04-25T11:38:00.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.113+0000 7f31cb665740 1 Processor -- start 2024-04-25T11:38:00.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.113+0000 7f31cb665740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b74292d4d0 0x55b742d02280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.113+0000 7f31cb665740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7429356b0 0x55b742d027c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.113+0000 7f31cb665740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b742956390 0x55b742d0e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.113+0000 7f31cb665740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b742d04d90 con 0x55b742956390 2024-04-25T11:38:00.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.113+0000 7f31cb665740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55b742be3dc0 con 0x55b74292d4d0 2024-04-25T11:38:00.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.113+0000 7f31cb665740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b742d04f10 con 0x55b7429356b0 2024-04-25T11:38:00.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.113+0000 7f31cb665740 10 client.0 _ll_register_callbacks cb 0x55b742bff600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:00.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.113+0000 7f31c1d4d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b74292d4d0 0x55b742d02280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.113+0000 7f31c1d4d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b74292d4d0 0x55b742d02280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:52898/0 (socket says 172.21.15.90:52898) 2024-04-25T11:38:00.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.113+0000 7f31c1d4d640 1 -- 172.21.15.90:0/181951334 learned_addr learned my addr 172.21.15.90:0/181951334 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:00.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.113+0000 7f31c1d4d640 1 --2- 172.21.15.90:0/181951334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b742956390 0x55b742d0e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.113+0000 7f31c254e640 1 --2- 172.21.15.90:0/181951334 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7429356b0 0x55b742d027c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.114+0000 7f31c1d4d640 1 -- 172.21.15.90:0/181951334 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7429356b0 msgr2=0x55b742d027c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.114+0000 7f31c1d4d640 1 --2- 172.21.15.90:0/181951334 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7429356b0 0x55b742d027c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.114+0000 7f31c1d4d640 1 -- 172.21.15.90:0/181951334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b742956390 msgr2=0x55b742d0e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.114+0000 7f31c1d4d640 1 --2- 172.21.15.90:0/181951334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b742956390 0x55b742d0e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.114+0000 7f31c1d4d640 1 -- 172.21.15.90:0/181951334 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b74296e3f0 con 0x55b74292d4d0 2024-04-25T11:38:00.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.114+0000 7f31c1d4d640 1 --2- 172.21.15.90:0/181951334 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b74292d4d0 0x55b742d02280 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f31ac011fc0 tx=0x7f31ac004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bafeb9b8e6ae277a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.114+0000 7f31a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:38:00.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.114+0000 7f31a5ffb640 1 -- 172.21.15.90:0/181951334 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ac004030 con 0x55b74292d4d0 2024-04-25T11:38:00.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.114+0000 7f31a5ffb640 1 -- 172.21.15.90:0/181951334 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f31ac0041d0 con 0x55b74292d4d0 2024-04-25T11:38:00.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.114+0000 7f31cb665740 1 -- 172.21.15.90:0/181951334 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31b8012aa0 con 0x55b74292d4d0 2024-04-25T11:38:00.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.114+0000 7f31a5ffb640 1 -- 172.21.15.90:0/181951334 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f31ac0043d0 con 0x55b74292d4d0 2024-04-25T11:38:00.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.114+0000 7f31cb665740 10 client.25739 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:00.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.115+0000 7f31cb665740 1 -- 172.21.15.90:0/181951334 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b742a387d0 con 0x55b74292d4d0 2024-04-25T11:38:00.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.115+0000 7f31a5ffb640 1 -- 172.21.15.90:0/181951334 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31b8012aa0 con 0x55b74292d4d0 2024-04-25T11:38:00.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.115+0000 7f31a5ffb640 1 -- 172.21.15.90:0/181951334 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f31ac0214d0 con 0x55b74292d4d0 2024-04-25T11:38:00.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.115+0000 7f31cb665740 10 client.25739 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:00.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.115+0000 7f31cb665740 10 client.25739 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:38:00.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.115+0000 7f31cb665740 20 client.25739 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:00.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.115+0000 7f31cb665740 1 --2- 172.21.15.90:0/181951334 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55b742914f60 0x55b742d146c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.115+0000 7f31cb665740 4 client.25739 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:38:00.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.115+0000 7f31cb665740 1 -- 172.21.15.90:0/181951334 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b742c6a990 con 0x55b742914f60 2024-04-25T11:38:00.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.115+0000 7f31c254e640 1 --2- 172.21.15.90:0/181951334 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55b742914f60 0x55b742d146c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.116+0000 7f31c254e640 1 --2- 172.21.15.90:0/181951334 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55b742914f60 0x55b742d146c0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7775c7eecd5cdade server_cookie=d07af5572f19d67c in_seq=0 out_seq=0 2024-04-25T11:38:00.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.116+0000 7f31a5ffb640 10 client.25739 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:38:00.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.117+0000 7f31a5ffb640 1 -- 172.21.15.90:0/181951334 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b742c6a990 con 0x55b742914f60 2024-04-25T11:38:00.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.117+0000 7f31a5ffb640 10 client.25739 handle_command_reply: tid=0 2024-04-25T11:38:00.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.117+0000 7f31cb665740 1 client.25739 shutdown 2024-04-25T11:38:00.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.117+0000 7f31cb665740 1 -- 172.21.15.90:0/181951334 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b74292d4d0 msgr2=0x55b742d02280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.117+0000 7f31cb665740 1 --2- 172.21.15.90:0/181951334 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b74292d4d0 0x55b742d02280 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f31ac011fc0 tx=0x7f31ac004730 comp rx=0 tx=0).stop 2024-04-25T11:38:00.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.117+0000 7f31cb665740 1 -- 172.21.15.90:0/181951334 shutdown_connections 2024-04-25T11:38:00.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.117+0000 7f31cb665740 1 --2- 172.21.15.90:0/181951334 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55b742914f60 0x55b742d146c0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.117+0000 7f31cb665740 1 --2- 172.21.15.90:0/181951334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b742956390 0x55b742d0e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.117+0000 7f31cb665740 1 --2- 172.21.15.90:0/181951334 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7429356b0 0x55b742d027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.117+0000 7f31a5ffb640 0 client.25739 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:38:00.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.117+0000 7f31cb665740 1 --2- 172.21.15.90:0/181951334 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b74292d4d0 0x55b742d02280 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.117+0000 7f31cb665740 1 -- 172.21.15.90:0/181951334 >> 172.21.15.90:0/181951334 conn(0x55b74294a4e0 msgr2=0x55b742cf9530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.117+0000 7f31cb665740 1 -- 172.21.15.90:0/181951334 shutdown_connections 2024-04-25T11:38:00.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.117+0000 7f31cb665740 1 -- 172.21.15.90:0/181951334 wait complete. 2024-04-25T11:38:00.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.117+0000 7f31cb665740 20 client.25739 trim_cache size 0 max 16384 2024-04-25T11:38:00.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.132+0000 7f31cb665740 1 Processor -- start 2024-04-25T11:38:00.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.132+0000 7f31cb665740 1 -- start start 2024-04-25T11:38:00.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.132+0000 7f31cb665740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b74292d4d0 0x55b742cf3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.132+0000 7f31cb665740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7429356b0 0x55b742d0de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.132+0000 7f31cb665740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b742956390 0x55b742d19e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.132+0000 7f31cb665740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f31bc1afbb0 con 0x55b74292d4d0 2024-04-25T11:38:00.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.132+0000 7f31cb665740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f31bc1b1720 con 0x55b742956390 2024-04-25T11:38:00.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.132+0000 7f31cb665740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f31bc1b0280 con 0x55b7429356b0 2024-04-25T11:38:00.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.132+0000 7f31c1d4d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b74292d4d0 0x55b742cf3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.132+0000 7f31c1d4d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b74292d4d0 0x55b742cf3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35342/0 (socket says 172.21.15.90:35342) 2024-04-25T11:38:00.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.132+0000 7f31c1d4d640 1 -- 172.21.15.90:0/1803189324 learned_addr learned my addr 172.21.15.90:0/1803189324 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:00.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.132+0000 7f31c254e640 1 --2- 172.21.15.90:0/1803189324 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7429356b0 0x55b742d0de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.132+0000 7f31c1d4d640 1 --2- 172.21.15.90:0/1803189324 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b742956390 0x55b742d19e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.133+0000 7f31c254e640 1 -- 172.21.15.90:0/1803189324 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b742956390 msgr2=0x55b742d19e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.133+0000 7f31c254e640 1 --2- 172.21.15.90:0/1803189324 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b742956390 0x55b742d19e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.133+0000 7f31c254e640 1 -- 172.21.15.90:0/1803189324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b74292d4d0 msgr2=0x55b742cf3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.133+0000 7f31c254e640 1 --2- 172.21.15.90:0/1803189324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b74292d4d0 0x55b742cf3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.133+0000 7f31c254e640 1 -- 172.21.15.90:0/1803189324 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b742d02f80 con 0x55b7429356b0 2024-04-25T11:38:00.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.133+0000 7f31c1d4d640 1 --2- 172.21.15.90:0/1803189324 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b742956390 0x55b742d19e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:38:00.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.133+0000 7f31c254e640 1 --2- 172.21.15.90:0/1803189324 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7429356b0 0x55b742d0de70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f31b8000c00 tx=0x7f31b80058d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1419051d1cd9a7a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.133+0000 7f31a7fff640 1 -- 172.21.15.90:0/1803189324 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b80096f0 con 0x55b7429356b0 2024-04-25T11:38:00.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.133+0000 7f31a7fff640 1 -- 172.21.15.90:0/1803189324 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31b8012450 con 0x55b7429356b0 2024-04-25T11:38:00.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.133+0000 7f31a7fff640 1 -- 172.21.15.90:0/1803189324 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b800f8c0 con 0x55b7429356b0 2024-04-25T11:38:00.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.134+0000 7f31cb665740 1 -- 172.21.15.90:0/1803189324 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7429356b0 msgr2=0x55b742d0de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.134+0000 7f31cb665740 1 --2- 172.21.15.90:0/1803189324 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7429356b0 0x55b742d0de70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f31b8000c00 tx=0x7f31b80058d0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.134+0000 7f31cb665740 1 -- 172.21.15.90:0/1803189324 shutdown_connections 2024-04-25T11:38:00.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.134+0000 7f31cb665740 1 --2- 172.21.15.90:0/1803189324 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b742956390 0x55b742d19e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.134+0000 7f31cb665740 1 --2- 172.21.15.90:0/1803189324 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7429356b0 0x55b742d0de70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.134+0000 7f31cb665740 1 --2- 172.21.15.90:0/1803189324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b74292d4d0 0x55b742cf3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.134+0000 7f31cb665740 1 -- 172.21.15.90:0/1803189324 >> 172.21.15.90:0/1803189324 conn(0x55b74294a4e0 msgr2=0x55b742cf3860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.134+0000 7f31cb665740 1 -- 172.21.15.90:0/1803189324 shutdown_connections 2024-04-25T11:38:00.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.134+0000 7f31cb665740 1 -- 172.21.15.90:0/1803189324 wait complete. 2024-04-25T11:38:00.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.135+0000 7f31cb665740 1 -- start start 2024-04-25T11:38:00.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.135+0000 7f31cb665740 1 Processor -- start 2024-04-25T11:38:00.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.135+0000 7f31cb665740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b74292d4d0 0x55b742d74e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.135+0000 7f31cb665740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7429356b0 0x55b742d75340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.135+0000 7f31cb665740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b742956390 0x55b742d055c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.135+0000 7f31cb665740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f31bc1b1020 con 0x55b742956390 2024-04-25T11:38:00.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.135+0000 7f31cb665740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f31bc1b0950 con 0x55b7429356b0 2024-04-25T11:38:00.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.135+0000 7f31cb665740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f31bc1adb90 con 0x55b74292d4d0 2024-04-25T11:38:00.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.135+0000 7f31cb665740 10 client.0 _ll_register_callbacks cb 0x7f31ac011de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:00.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.135+0000 7f31c1d4d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b742956390 0x55b742d055c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.135+0000 7f31c254e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7429356b0 0x55b742d75340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.135+0000 7f31c1d4d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b742956390 0x55b742d055c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35344/0 (socket says 172.21.15.90:35344) 2024-04-25T11:38:00.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.135+0000 7f31c1d4d640 1 -- 172.21.15.90:0/917772202 learned_addr learned my addr 172.21.15.90:0/917772202 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:00.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.135+0000 7f31c1d4d640 1 --2- 172.21.15.90:0/917772202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b74292d4d0 0x55b742d74e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.136+0000 7f31c254e640 1 -- 172.21.15.90:0/917772202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b74292d4d0 msgr2=0x55b742d74e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.136+0000 7f31c254e640 1 --2- 172.21.15.90:0/917772202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b74292d4d0 0x55b742d74e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.136+0000 7f31c254e640 1 -- 172.21.15.90:0/917772202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b742956390 msgr2=0x55b742d055c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.136+0000 7f31c254e640 1 --2- 172.21.15.90:0/917772202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b742956390 0x55b742d055c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.136+0000 7f31c254e640 1 -- 172.21.15.90:0/917772202 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b742d05cb0 con 0x55b7429356b0 2024-04-25T11:38:00.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.136+0000 7f31c1d4d640 1 --2- 172.21.15.90:0/917772202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b742956390 0x55b742d055c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:00.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.136+0000 7f31c254e640 1 --2- 172.21.15.90:0/917772202 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7429356b0 0x55b742d75340 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x55b742cf76e0 tx=0x7f31b8009a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c096284e822f50b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.136+0000 7f31a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:38:00.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.136+0000 7f31a5ffb640 1 -- 172.21.15.90:0/917772202 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b803b060 con 0x55b7429356b0 2024-04-25T11:38:00.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.136+0000 7f31a5ffb640 1 -- 172.21.15.90:0/917772202 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f31b8012a10 con 0x55b7429356b0 2024-04-25T11:38:00.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.136+0000 7f31cb665740 1 -- 172.21.15.90:0/917772202 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31b8012450 con 0x55b7429356b0 2024-04-25T11:38:00.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.136+0000 7f31a5ffb640 1 -- 172.21.15.90:0/917772202 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f31b8005050 con 0x55b7429356b0 2024-04-25T11:38:00.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.136+0000 7f31cb665740 10 client.25745 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:00.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.136+0000 7f31cb665740 1 -- 172.21.15.90:0/917772202 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b742d7c2f0 con 0x55b7429356b0 2024-04-25T11:38:00.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.137+0000 7f31a5ffb640 1 -- 172.21.15.90:0/917772202 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31b8012450 con 0x55b7429356b0 2024-04-25T11:38:00.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.137+0000 7f31a5ffb640 1 -- 172.21.15.90:0/917772202 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f31b800fb90 con 0x55b7429356b0 2024-04-25T11:38:00.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.137+0000 7f31cb665740 10 client.25745 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:00.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.137+0000 7f31cb665740 10 client.25745 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:38:00.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.137+0000 7f31cb665740 20 client.25745 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:00.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.137+0000 7f31cb665740 1 --2- 172.21.15.90:0/917772202 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55b7428ea3a0 0x55b742d7e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.137+0000 7f31cb665740 4 client.25745 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:38:00.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.137+0000 7f31cb665740 1 -- 172.21.15.90:0/917772202 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b742c64800 con 0x55b7428ea3a0 2024-04-25T11:38:00.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.137+0000 7f31c1d4d640 1 --2- 172.21.15.90:0/917772202 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55b7428ea3a0 0x55b742d7e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.138+0000 7f31a5ffb640 10 client.25745 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:38:00.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.138+0000 7f31c1d4d640 1 --2- 172.21.15.90:0/917772202 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55b7428ea3a0 0x55b742d7e820 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=dcc8ce6793ae5506 server_cookie=575f2c50f531a2a2 in_seq=0 out_seq=0 2024-04-25T11:38:00.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.138+0000 7f31a5ffb640 1 -- 172.21.15.90:0/917772202 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b742c64800 con 0x55b7428ea3a0 2024-04-25T11:38:00.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.138+0000 7f31a5ffb640 10 client.25745 handle_command_reply: tid=0 2024-04-25T11:38:00.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.138+0000 7f31cb665740 1 client.25745 shutdown 2024-04-25T11:38:00.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.138+0000 7f31cb665740 1 -- 172.21.15.90:0/917772202 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7429356b0 msgr2=0x55b742d75340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.138+0000 7f31cb665740 1 --2- 172.21.15.90:0/917772202 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7429356b0 0x55b742d75340 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x55b742cf76e0 tx=0x7f31b8009a20 comp rx=0 tx=0).stop 2024-04-25T11:38:00.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.138+0000 7f31cb665740 1 -- 172.21.15.90:0/917772202 shutdown_connections 2024-04-25T11:38:00.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.138+0000 7f31cb665740 1 --2- 172.21.15.90:0/917772202 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55b7428ea3a0 0x55b742d7e820 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.138+0000 7f31cb665740 1 --2- 172.21.15.90:0/917772202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b742956390 0x55b742d055c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.138+0000 7f31cb665740 1 --2- 172.21.15.90:0/917772202 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7429356b0 0x55b742d75340 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.138+0000 7f31a5ffb640 0 client.25745 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:38:00.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.138+0000 7f31cb665740 1 --2- 172.21.15.90:0/917772202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b74292d4d0 0x55b742d74e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.138+0000 7f31cb665740 1 -- 172.21.15.90:0/917772202 >> 172.21.15.90:0/917772202 conn(0x55b74294a4e0 msgr2=0x55b742d036a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.138+0000 7f31cb665740 1 -- 172.21.15.90:0/917772202 shutdown_connections 2024-04-25T11:38:00.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.138+0000 7f31cb665740 1 -- 172.21.15.90:0/917772202 wait complete. 2024-04-25T11:38:00.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.138+0000 7f31cb665740 20 client.25745 trim_cache size 0 max 16384 2024-04-25T11:38:00.131 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:38:00.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.139+0000 7f31c47b2640 1 -- 172.21.15.90:0/245115637 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f31a80afce0 msgr2=0x7f31a80b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.139+0000 7f31c47b2640 1 --2- 172.21.15.90:0/245115637 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f31a80afce0 0x7f31a80b21f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f31b40fd970 tx=0x7f31b4095730 comp rx=0 tx=0).stop 2024-04-25T11:38:00.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.139+0000 7f31c47b2640 1 -- 172.21.15.90:0/245115637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31bc10e070 msgr2=0x7f31bc19eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.139+0000 7f31c47b2640 1 --2- 172.21.15.90:0/245115637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31bc10e070 0x7f31bc19eed0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f31b4004af0 tx=0x7f31b4031f50 comp rx=0 tx=0).stop 2024-04-25T11:38:00.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.139+0000 7f31c47b2640 1 -- 172.21.15.90:0/245115637 shutdown_connections 2024-04-25T11:38:00.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.139+0000 7f31c47b2640 1 --2- 172.21.15.90:0/245115637 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f31a80afce0 0x7f31a80b21f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.139+0000 7f31c47b2640 1 --2- 172.21.15.90:0/245115637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31bc10e070 0x7f31bc19eed0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.139+0000 7f31c47b2640 1 --2- 172.21.15.90:0/245115637 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31bc10b680 0x7f31bc19ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.139+0000 7f31c47b2640 1 --2- 172.21.15.90:0/245115637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31bc104c20 0x7f31bc19a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.139+0000 7f31c47b2640 1 -- 172.21.15.90:0/245115637 >> 172.21.15.90:0/245115637 conn(0x7f31bc078180 msgr2=0x7f31bc1011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.139+0000 7f31c47b2640 1 -- 172.21.15.90:0/245115637 shutdown_connections 2024-04-25T11:38:00.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.139+0000 7f31c47b2640 1 -- 172.21.15.90:0/245115637 wait complete. 2024-04-25T11:38:00.145 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T11:38:00.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.337+0000 7f0b9b1f3640 1 -- 172.21.15.90:0/2663517015 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b9410d380 msgr2=0x7f0b9410f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.337+0000 7f0b9b1f3640 1 --2- 172.21.15.90:0/2663517015 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b9410d380 0x7f0b9410f7a0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c009ca0 tx=0x7f0b8c02a8f0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.337+0000 7f0b9b1f3640 1 -- 172.21.15.90:0/2663517015 shutdown_connections 2024-04-25T11:38:00.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.337+0000 7f0b9b1f3640 1 --2- 172.21.15.90:0/2663517015 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b9410d380 0x7f0b9410f7a0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.337+0000 7f0b9b1f3640 1 --2- 172.21.15.90:0/2663517015 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b94102af0 0x7f0b94102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.337+0000 7f0b9b1f3640 1 --2- 172.21.15.90:0/2663517015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b94108a20 0x7f0b941025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.337+0000 7f0b9b1f3640 1 -- 172.21.15.90:0/2663517015 >> 172.21.15.90:0/2663517015 conn(0x7f0b94078100 msgr2=0x7f0b94078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.339+0000 7f0b9b1f3640 1 -- 172.21.15.90:0/2663517015 shutdown_connections 2024-04-25T11:38:00.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.339+0000 7f0b9b1f3640 1 -- 172.21.15.90:0/2663517015 wait complete. 2024-04-25T11:38:00.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.340+0000 7f0b9b1f3640 1 Processor -- start 2024-04-25T11:38:00.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.340+0000 7f0b9b1f3640 1 -- start start 2024-04-25T11:38:00.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.340+0000 7f0b9b1f3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b94102af0 0x7f0b941997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.340+0000 7f0b9b1f3640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b94108a20 0x7f0b94199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.340+0000 7f0b9b1f3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b9410d380 0x7f0b9419dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.340+0000 7f0b9b1f3640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0b94112ce0 con 0x7f0b9410d380 2024-04-25T11:38:00.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.340+0000 7f0b9b1f3640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0b94112b60 con 0x7f0b94108a20 2024-04-25T11:38:00.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.340+0000 7f0b9b1f3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0b94112e60 con 0x7f0b94102af0 2024-04-25T11:38:00.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.341+0000 7f0b99790640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b9410d380 0x7f0b9419dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.341+0000 7f0b99790640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b9410d380 0x7f0b9419dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35376/0 (socket says 172.21.15.90:35376) 2024-04-25T11:38:00.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.341+0000 7f0b99790640 1 -- 172.21.15.90:0/45994122 learned_addr learned my addr 172.21.15.90:0/45994122 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:00.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.341+0000 7f0b93fff640 1 --2- 172.21.15.90:0/45994122 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b94108a20 0x7f0b94199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.341+0000 7f0b99790640 1 -- 172.21.15.90:0/45994122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b94102af0 msgr2=0x7f0b941997c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:38:00.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.341+0000 7f0b99790640 1 --2- 172.21.15.90:0/45994122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b94102af0 0x7f0b941997c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.341+0000 7f0b99790640 1 -- 172.21.15.90:0/45994122 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b94108a20 msgr2=0x7f0b94199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.341+0000 7f0b99790640 1 --2- 172.21.15.90:0/45994122 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b94108a20 0x7f0b94199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.341+0000 7f0b99790640 1 -- 172.21.15.90:0/45994122 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b9419e620 con 0x7f0b9410d380 2024-04-25T11:38:00.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.341+0000 7f0b99790640 1 --2- 172.21.15.90:0/45994122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b9410d380 0x7f0b9419dfe0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c003bd0 tx=0x7f0b8c003c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.342+0000 7f0b91ffb640 1 -- 172.21.15.90:0/45994122 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c004040 con 0x7f0b9410d380 2024-04-25T11:38:00.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.342+0000 7f0b91ffb640 1 -- 172.21.15.90:0/45994122 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b8c02d070 con 0x7f0b9410d380 2024-04-25T11:38:00.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.342+0000 7f0b91ffb640 1 -- 172.21.15.90:0/45994122 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c02fbe0 con 0x7f0b9410d380 2024-04-25T11:38:00.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.342+0000 7f0b9b1f3640 1 -- 172.21.15.90:0/45994122 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b9419e8b0 con 0x7f0b9410d380 2024-04-25T11:38:00.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.342+0000 7f0b9b1f3640 1 -- 172.21.15.90:0/45994122 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b9419edb0 con 0x7f0b9410d380 2024-04-25T11:38:00.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.345+0000 7f0ba217a740 1 Processor -- start 2024-04-25T11:38:00.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.346+0000 7f0b91ffb640 1 -- 172.21.15.90:0/45994122 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f0b8c02fd80 con 0x7f0b9410d380 2024-04-25T11:38:00.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.347+0000 7f0ba217a740 1 -- start start 2024-04-25T11:38:00.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.347+0000 7f0ba217a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6af825390 0x55f6afbc6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.347+0000 7f0ba217a740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6af7fc4d0 0x55f6afbc6d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.347+0000 7f0ba217a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af8046b0 0x55f6afbd36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.347+0000 7f0ba217a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f6afb2bd80 con 0x55f6af8046b0 2024-04-25T11:38:00.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.347+0000 7f0ba217a740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f6afae5a20 con 0x55f6af7fc4d0 2024-04-25T11:38:00.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.347+0000 7f0ba217a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f6afb27d40 con 0x55f6af825390 2024-04-25T11:38:00.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.347+0000 7f0b91ffb640 1 --2- 172.21.15.90:0/45994122 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0b840afd10 0x7f0b840b2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.347+0000 7f0b93fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af8046b0 0x55f6afbd36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.348+0000 7f0b93fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af8046b0 0x55f6afbd36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35392/0 (socket says 172.21.15.90:35392) 2024-04-25T11:38:00.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.348+0000 7f0b93fff640 1 -- 172.21.15.90:0/1170192136 learned_addr learned my addr 172.21.15.90:0/1170192136 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:00.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.348+0000 7f0b91ffb640 1 -- 172.21.15.90:0/45994122 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f0b8c1007a0 con 0x7f0b9410d380 2024-04-25T11:38:00.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.348+0000 7f0b99790640 1 --2- 172.21.15.90:0/45994122 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0b840afd10 0x7f0b840b2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.348+0000 7f0b93fff640 1 -- 172.21.15.90:0/1170192136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6af825390 msgr2=0x55f6afbc6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.348+0000 7f0b93fff640 1 --2- 172.21.15.90:0/1170192136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6af825390 0x55f6afbc6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.348+0000 7f0b93fff640 1 -- 172.21.15.90:0/1170192136 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6af7fc4d0 msgr2=0x55f6afbc6d50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:38:00.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.348+0000 7f0b93fff640 1 --2- 172.21.15.90:0/1170192136 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6af7fc4d0 0x55f6afbc6d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.348+0000 7f0b93fff640 1 -- 172.21.15.90:0/1170192136 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6afaf0c30 con 0x55f6af8046b0 2024-04-25T11:38:00.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.348+0000 7f0b93fff640 1 --2- 172.21.15.90:0/1170192136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af8046b0 0x55f6afbd36a0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f0b8000a830 tx=0x7f0b8000acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3639c3aebbd5d882 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.348+0000 7f0b99790640 1 --2- 172.21.15.90:0/45994122 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0b840afd10 0x7f0b840b2220 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c0c8980 tx=0x7f0b8c003f20 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.349+0000 7f0b7effd640 1 -- 172.21.15.90:0/1170192136 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8000f9c0 con 0x55f6af8046b0 2024-04-25T11:38:00.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.349+0000 7f0b7effd640 1 -- 172.21.15.90:0/1170192136 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b80018070 con 0x55f6af8046b0 2024-04-25T11:38:00.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.349+0000 7f0b7effd640 1 -- 172.21.15.90:0/1170192136 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b80013e50 con 0x55f6af8046b0 2024-04-25T11:38:00.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.350+0000 7f0ba217a740 1 -- 172.21.15.90:0/1170192136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af8046b0 msgr2=0x55f6afbd36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.350+0000 7f0ba217a740 1 --2- 172.21.15.90:0/1170192136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af8046b0 0x55f6afbd36a0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f0b8000a830 tx=0x7f0b8000acf0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.350+0000 7f0ba217a740 1 -- 172.21.15.90:0/1170192136 shutdown_connections 2024-04-25T11:38:00.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.350+0000 7f0ba217a740 1 --2- 172.21.15.90:0/1170192136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af8046b0 0x55f6afbd36a0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.350+0000 7f0ba217a740 1 --2- 172.21.15.90:0/1170192136 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6af7fc4d0 0x55f6afbc6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.350+0000 7f0ba217a740 1 --2- 172.21.15.90:0/1170192136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6af825390 0x55f6afbc6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.350+0000 7f0ba217a740 1 -- 172.21.15.90:0/1170192136 >> 172.21.15.90:0/1170192136 conn(0x55f6af8194e0 msgr2=0x55f6afbc2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.350+0000 7f0ba217a740 1 -- 172.21.15.90:0/1170192136 shutdown_connections 2024-04-25T11:38:00.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.350+0000 7f0ba217a740 1 -- 172.21.15.90:0/1170192136 wait complete. 2024-04-25T11:38:00.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.351+0000 7f0ba217a740 1 -- start start 2024-04-25T11:38:00.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.351+0000 7f0ba217a740 1 Processor -- start 2024-04-25T11:38:00.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.352+0000 7f0ba217a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6af7fc4d0 0x55f6afbd1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.352+0000 7f0ba217a740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6af8046b0 0x55f6afbd1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.352+0000 7f0ba217a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af825390 0x55f6afbdd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.352+0000 7f0ba217a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f6afbd3d90 con 0x55f6af825390 2024-04-25T11:38:00.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.352+0000 7f0ba217a740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f6afab2dc0 con 0x55f6af8046b0 2024-04-25T11:38:00.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.352+0000 7f0ba217a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f6afbd3f10 con 0x55f6af7fc4d0 2024-04-25T11:38:00.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.352+0000 7f0ba217a740 10 client.0 _ll_register_callbacks cb 0x55f6aface600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:00.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.352+0000 7f0b93fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af825390 0x55f6afbdd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.353+0000 7f0b93fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af825390 0x55f6afbdd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35408/0 (socket says 172.21.15.90:35408) 2024-04-25T11:38:00.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.353+0000 7f0b93fff640 1 -- 172.21.15.90:0/374408472 learned_addr learned my addr 172.21.15.90:0/374408472 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:00.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.353+0000 7f0b93fff640 1 -- 172.21.15.90:0/374408472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6af7fc4d0 msgr2=0x55f6afbd1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.353+0000 7f0b93fff640 1 --2- 172.21.15.90:0/374408472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6af7fc4d0 0x55f6afbd1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.353+0000 7f0b93fff640 1 -- 172.21.15.90:0/374408472 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6af8046b0 msgr2=0x55f6afbd1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.353+0000 7f0b93fff640 1 --2- 172.21.15.90:0/374408472 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6af8046b0 0x55f6afbd1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.353+0000 7f0b93fff640 1 -- 172.21.15.90:0/374408472 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6af83d3f0 con 0x55f6af825390 2024-04-25T11:38:00.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.354+0000 7f0b93fff640 1 --2- 172.21.15.90:0/374408472 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af825390 0x55f6afbdd210 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f0b80013990 tx=0x7f0b8000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26712fad081df9bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.354+0000 7f0b73fff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:38:00.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.354+0000 7f0b73fff640 1 -- 172.21.15.90:0/374408472 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b80022070 con 0x55f6af825390 2024-04-25T11:38:00.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.354+0000 7f0b73fff640 1 -- 172.21.15.90:0/374408472 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f0b8000fc90 con 0x55f6af825390 2024-04-25T11:38:00.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.354+0000 7f0ba217a740 1 -- 172.21.15.90:0/374408472 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b80018070 con 0x55f6af825390 2024-04-25T11:38:00.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.354+0000 7f0b73fff640 1 -- 172.21.15.90:0/374408472 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0b8002e050 con 0x55f6af825390 2024-04-25T11:38:00.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.354+0000 7f0ba217a740 10 client.16440 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:00.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.354+0000 7f0ba217a740 1 -- 172.21.15.90:0/374408472 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f6af9077d0 con 0x55f6af825390 2024-04-25T11:38:00.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.355+0000 7f0b73fff640 1 -- 172.21.15.90:0/374408472 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b80018070 con 0x55f6af825390 2024-04-25T11:38:00.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.355+0000 7f0b73fff640 1 -- 172.21.15.90:0/374408472 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f0b8001c430 con 0x55f6af825390 2024-04-25T11:38:00.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.355+0000 7f0ba217a740 10 client.16440 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:00.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.355+0000 7f0ba217a740 10 client.16440 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:38:00.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.355+0000 7f0ba217a740 20 client.16440 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:00.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.355+0000 7f0ba217a740 1 --2- 172.21.15.90:0/374408472 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55f6af7e3f60 0x55f6afbe3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.355+0000 7f0ba217a740 4 client.16440 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:38:00.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.355+0000 7f0ba217a740 1 -- 172.21.15.90:0/374408472 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6afb39990 con 0x55f6af7e3f60 2024-04-25T11:38:00.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.358+0000 7f0b98f8f640 1 --2- 172.21.15.90:0/374408472 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55f6af7e3f60 0x55f6afbe3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.362+0000 7f0b73fff640 10 client.16440 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:38:00.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.362+0000 7f0b98f8f640 1 --2- 172.21.15.90:0/374408472 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55f6af7e3f60 0x55f6afbe3490 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=800d0543ad77e24 server_cookie=74d46b5ff85ae9d5 in_seq=0 out_seq=0 2024-04-25T11:38:00.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.364+0000 7f0b73fff640 1 -- 172.21.15.90:0/374408472 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f6afb39990 con 0x55f6af7e3f60 2024-04-25T11:38:00.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.364+0000 7f0b73fff640 10 client.16440 handle_command_reply: tid=0 2024-04-25T11:38:00.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.365+0000 7f0ba217a740 1 client.16440 shutdown 2024-04-25T11:38:00.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.366+0000 7f0ba217a740 1 -- 172.21.15.90:0/374408472 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af825390 msgr2=0x55f6afbdd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.366+0000 7f0ba217a740 1 --2- 172.21.15.90:0/374408472 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af825390 0x55f6afbdd210 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f0b80013990 tx=0x7f0b8000a620 comp rx=0 tx=0).stop 2024-04-25T11:38:00.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.366+0000 7f0ba217a740 1 -- 172.21.15.90:0/374408472 shutdown_connections 2024-04-25T11:38:00.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.366+0000 7f0ba217a740 1 --2- 172.21.15.90:0/374408472 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af825390 0x55f6afbdd210 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.366+0000 7f0ba217a740 1 --2- 172.21.15.90:0/374408472 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55f6af7e3f60 0x55f6afbe3490 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.366+0000 7f0ba217a740 1 --2- 172.21.15.90:0/374408472 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6af8046b0 0x55f6afbd1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.366+0000 7f0ba217a740 1 --2- 172.21.15.90:0/374408472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6af7fc4d0 0x55f6afbd1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.366+0000 7f0ba217a740 1 -- 172.21.15.90:0/374408472 >> 172.21.15.90:0/374408472 conn(0x55f6af8194e0 msgr2=0x55f6afbc8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.366+0000 7f0b73fff640 0 client.16440 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:38:00.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.366+0000 7f0ba217a740 1 -- 172.21.15.90:0/374408472 shutdown_connections 2024-04-25T11:38:00.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.366+0000 7f0ba217a740 1 -- 172.21.15.90:0/374408472 wait complete. 2024-04-25T11:38:00.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.366+0000 7f0ba217a740 20 client.16440 trim_cache size 0 max 16384 2024-04-25T11:38:00.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.392+0000 7f0ba217a740 1 Processor -- start 2024-04-25T11:38:00.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.392+0000 7f0ba217a740 1 -- start start 2024-04-25T11:38:00.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.393+0000 7f0ba217a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6af7fc4d0 0x55f6afbdd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.393+0000 7f0ba217a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af8046b0 0x55f6afbd0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.393+0000 7f0ba217a740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6af825390 0x55f6afbe64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.393+0000 7f0ba217a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0b941afd20 con 0x55f6af8046b0 2024-04-25T11:38:00.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.393+0000 7f0ba217a740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0b941b1890 con 0x55f6af825390 2024-04-25T11:38:00.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.393+0000 7f0ba217a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0b941b03f0 con 0x55f6af7fc4d0 2024-04-25T11:38:00.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.393+0000 7f0b98f8f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af8046b0 0x55f6afbd0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.393+0000 7f0b98f8f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af8046b0 0x55f6afbd0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35424/0 (socket says 172.21.15.90:35424) 2024-04-25T11:38:00.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.393+0000 7f0b98f8f640 1 -- 172.21.15.90:0/1828140191 learned_addr learned my addr 172.21.15.90:0/1828140191 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:00.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.393+0000 7f0b93fff640 1 --2- 172.21.15.90:0/1828140191 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6af825390 0x55f6afbe64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.393+0000 7f0b98f8f640 1 -- 172.21.15.90:0/1828140191 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6af7fc4d0 msgr2=0x55f6afbdd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.393+0000 7f0b98f8f640 1 --2- 172.21.15.90:0/1828140191 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6af7fc4d0 0x55f6afbdd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.393+0000 7f0b98f8f640 1 -- 172.21.15.90:0/1828140191 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6af825390 msgr2=0x55f6afbe64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.394+0000 7f0b98f8f640 1 --2- 172.21.15.90:0/1828140191 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6af825390 0x55f6afbe64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.394+0000 7f0b98f8f640 1 -- 172.21.15.90:0/1828140191 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6afbd3c70 con 0x55f6af8046b0 2024-04-25T11:38:00.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.394+0000 7f0b98f8f640 1 --2- 172.21.15.90:0/1828140191 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af8046b0 0x55f6afbd0c40 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x55f6afbc7fe0 tx=0x7f0b88009cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c270a62dd9a6a9e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.395+0000 7f0b7e7fc640 1 -- 172.21.15.90:0/1828140191 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b88012790 con 0x55f6af8046b0 2024-04-25T11:38:00.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.395+0000 7f0b7e7fc640 1 -- 172.21.15.90:0/1828140191 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b8801a070 con 0x55f6af8046b0 2024-04-25T11:38:00.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.395+0000 7f0b7e7fc640 1 -- 172.21.15.90:0/1828140191 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8801fca0 con 0x55f6af8046b0 2024-04-25T11:38:00.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.396+0000 7f0ba217a740 1 -- 172.21.15.90:0/1828140191 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af8046b0 msgr2=0x55f6afbd0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.396+0000 7f0ba217a740 1 --2- 172.21.15.90:0/1828140191 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af8046b0 0x55f6afbd0c40 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x55f6afbc7fe0 tx=0x7f0b88009cc0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.396+0000 7f0ba217a740 1 -- 172.21.15.90:0/1828140191 shutdown_connections 2024-04-25T11:38:00.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.396+0000 7f0ba217a740 1 --2- 172.21.15.90:0/1828140191 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6af825390 0x55f6afbe64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.396+0000 7f0ba217a740 1 --2- 172.21.15.90:0/1828140191 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af8046b0 0x55f6afbd0c40 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.396+0000 7f0ba217a740 1 --2- 172.21.15.90:0/1828140191 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6af7fc4d0 0x55f6afbdd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.396+0000 7f0ba217a740 1 -- 172.21.15.90:0/1828140191 >> 172.21.15.90:0/1828140191 conn(0x55f6af8194e0 msgr2=0x55f6afbd0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.396+0000 7f0ba217a740 1 -- 172.21.15.90:0/1828140191 shutdown_connections 2024-04-25T11:38:00.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.396+0000 7f0ba217a740 1 -- 172.21.15.90:0/1828140191 wait complete. 2024-04-25T11:38:00.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.397+0000 7f0ba217a740 1 -- start start 2024-04-25T11:38:00.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.397+0000 7f0ba217a740 1 Processor -- start 2024-04-25T11:38:00.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.398+0000 7f0ba217a740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6af7fc4d0 0x55f6afbe3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.398+0000 7f0ba217a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af825390 0x55f6afbe3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.398+0000 7f0ba217a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6af7b93a0 0x55f6afc4d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.398+0000 7f0ba217a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0b941b1190 con 0x55f6af825390 2024-04-25T11:38:00.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.398+0000 7f0ba217a740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0b941b0ac0 con 0x55f6af7fc4d0 2024-04-25T11:38:00.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.398+0000 7f0ba217a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0b941add00 con 0x55f6af7b93a0 2024-04-25T11:38:00.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.398+0000 7f0ba217a740 10 client.0 _ll_register_callbacks cb 0x7f0b8000a730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:00.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.398+0000 7f0b98f8f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af825390 0x55f6afbe3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.398+0000 7f0b98f8f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af825390 0x55f6afbe3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35426/0 (socket says 172.21.15.90:35426) 2024-04-25T11:38:00.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.398+0000 7f0b98f8f640 1 -- 172.21.15.90:0/1560331647 learned_addr learned my addr 172.21.15.90:0/1560331647 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:00.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.399+0000 7f0b98f8f640 1 -- 172.21.15.90:0/1560331647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6af7b93a0 msgr2=0x55f6afc4d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.399+0000 7f0b98f8f640 1 --2- 172.21.15.90:0/1560331647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6af7b93a0 0x55f6afc4d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.399+0000 7f0b98f8f640 1 -- 172.21.15.90:0/1560331647 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6af7fc4d0 msgr2=0x55f6afbe3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.399+0000 7f0b98f8f640 1 --2- 172.21.15.90:0/1560331647 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6af7fc4d0 0x55f6afbe3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.399+0000 7f0b98f8f640 1 -- 172.21.15.90:0/1560331647 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6afc4df00 con 0x55f6af825390 2024-04-25T11:38:00.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.399+0000 7f0b98f8f640 1 --2- 172.21.15.90:0/1560331647 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af825390 0x55f6afbe3d90 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x55f6afbc7fe0 tx=0x7f0b8801f7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8dd09058b8536e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.399+0000 7f0b73fff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:38:00.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.399+0000 7f0b73fff640 1 -- 172.21.15.90:0/1560331647 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b880044d0 con 0x55f6af825390 2024-04-25T11:38:00.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.400+0000 7f0b73fff640 1 -- 172.21.15.90:0/1560331647 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f0b8801f920 con 0x55f6af825390 2024-04-25T11:38:00.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.400+0000 7f0ba217a740 1 -- 172.21.15.90:0/1560331647 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b8801a070 con 0x55f6af825390 2024-04-25T11:38:00.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.400+0000 7f0b73fff640 1 -- 172.21.15.90:0/1560331647 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0b8801fb20 con 0x55f6af825390 2024-04-25T11:38:00.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.400+0000 7f0ba217a740 10 client.16446 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:00.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.400+0000 7f0ba217a740 1 -- 172.21.15.90:0/1560331647 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f6afc4f150 con 0x55f6af825390 2024-04-25T11:38:00.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.400+0000 7f0b73fff640 1 -- 172.21.15.90:0/1560331647 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b88023070 con 0x55f6af825390 2024-04-25T11:38:00.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.400+0000 7f0b73fff640 1 -- 172.21.15.90:0/1560331647 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f0b88003720 con 0x55f6af825390 2024-04-25T11:38:00.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.401+0000 7f0ba217a740 10 client.16446 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:00.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.401+0000 7f0ba217a740 10 client.16446 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:38:00.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.401+0000 7f0ba217a740 20 client.16446 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:00.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.401+0000 7f0ba217a740 1 --2- 172.21.15.90:0/1560331647 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55f6afc4f720 0x55f6afc51b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.401+0000 7f0ba217a740 4 client.16446 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:38:00.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.401+0000 7f0ba217a740 1 -- 172.21.15.90:0/1560331647 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f0b8801a070 con 0x55f6afc4f720 2024-04-25T11:38:00.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.403+0000 7f0b93fff640 1 --2- 172.21.15.90:0/1560331647 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55f6afc4f720 0x55f6afc51b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.406+0000 7f0b73fff640 10 client.16446 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:38:00.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.406+0000 7f0b93fff640 1 --2- 172.21.15.90:0/1560331647 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55f6afc4f720 0x55f6afc51b40 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=54685ff94f49c7cd server_cookie=4be5aa7b76a1cea6 in_seq=0 out_seq=0 2024-04-25T11:38:00.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.407+0000 7f0b73fff640 1 -- 172.21.15.90:0/1560331647 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f0b8801a070 con 0x55f6afc4f720 2024-04-25T11:38:00.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.407+0000 7f0b73fff640 10 client.16446 handle_command_reply: tid=0 2024-04-25T11:38:00.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.407+0000 7f0ba217a740 1 client.16446 shutdown 2024-04-25T11:38:00.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.407+0000 7f0ba217a740 1 -- 172.21.15.90:0/1560331647 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af825390 msgr2=0x55f6afbe3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.407+0000 7f0ba217a740 1 --2- 172.21.15.90:0/1560331647 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af825390 0x55f6afbe3d90 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x55f6afbc7fe0 tx=0x7f0b8801f7e0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.407+0000 7f0ba217a740 1 -- 172.21.15.90:0/1560331647 shutdown_connections 2024-04-25T11:38:00.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.407+0000 7f0ba217a740 1 --2- 172.21.15.90:0/1560331647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6af7b93a0 0x55f6afc4d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.407+0000 7f0ba217a740 1 --2- 172.21.15.90:0/1560331647 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6af825390 0x55f6afbe3d90 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.407+0000 7f0ba217a740 1 --2- 172.21.15.90:0/1560331647 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55f6afc4f720 0x55f6afc51b40 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.408+0000 7f0ba217a740 1 --2- 172.21.15.90:0/1560331647 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6af7fc4d0 0x55f6afbe3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.408+0000 7f0b73fff640 0 client.16446 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:38:00.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.408+0000 7f0ba217a740 1 -- 172.21.15.90:0/1560331647 >> 172.21.15.90:0/1560331647 conn(0x55f6af8194e0 msgr2=0x55f6afbc65a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.408+0000 7f0ba217a740 1 -- 172.21.15.90:0/1560331647 shutdown_connections 2024-04-25T11:38:00.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.408+0000 7f0ba217a740 1 -- 172.21.15.90:0/1560331647 wait complete. 2024-04-25T11:38:00.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.408+0000 7f0ba217a740 20 client.16446 trim_cache size 0 max 16384 2024-04-25T11:38:00.397 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:38:00.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.409+0000 7f0b9b1f3640 1 -- 172.21.15.90:0/45994122 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0b840afd10 msgr2=0x7f0b840b2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.409+0000 7f0b9b1f3640 1 --2- 172.21.15.90:0/45994122 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0b840afd10 0x7f0b840b2220 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c0c8980 tx=0x7f0b8c003f20 comp rx=0 tx=0).stop 2024-04-25T11:38:00.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.409+0000 7f0b9b1f3640 1 -- 172.21.15.90:0/45994122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b9410d380 msgr2=0x7f0b9419dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.409+0000 7f0b9b1f3640 1 --2- 172.21.15.90:0/45994122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b9410d380 0x7f0b9419dfe0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c003bd0 tx=0x7f0b8c003c00 comp rx=0 tx=0).stop 2024-04-25T11:38:00.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.409+0000 7f0b9b1f3640 1 -- 172.21.15.90:0/45994122 shutdown_connections 2024-04-25T11:38:00.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.409+0000 7f0b9b1f3640 1 --2- 172.21.15.90:0/45994122 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0b840afd10 0x7f0b840b2220 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.409+0000 7f0b9b1f3640 1 --2- 172.21.15.90:0/45994122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b9410d380 0x7f0b9419dfe0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.409+0000 7f0b9b1f3640 1 --2- 172.21.15.90:0/45994122 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b94108a20 0x7f0b94199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.409+0000 7f0b9b1f3640 1 --2- 172.21.15.90:0/45994122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b94102af0 0x7f0b941997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.409+0000 7f0b9b1f3640 1 -- 172.21.15.90:0/45994122 >> 172.21.15.90:0/45994122 conn(0x7f0b94078100 msgr2=0x7f0b941040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.410+0000 7f0b9b1f3640 1 -- 172.21.15.90:0/45994122 shutdown_connections 2024-04-25T11:38:00.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.410+0000 7f0b9b1f3640 1 -- 172.21.15.90:0/45994122 wait complete. 2024-04-25T11:38:00.412 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T11:38:00.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.547+0000 7f66bd225640 1 -- 172.21.15.90:0/2380240295 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66b8108a20 msgr2=0x7f66b81025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.547+0000 7f66bd225640 1 --2- 172.21.15.90:0/2380240295 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66b8108a20 0x7f66b81025b0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009900 tx=0x7f66a802a410 comp rx=0 tx=0).stop 2024-04-25T11:38:00.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.547+0000 7f66bd225640 1 -- 172.21.15.90:0/2380240295 shutdown_connections 2024-04-25T11:38:00.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.547+0000 7f66bd225640 1 --2- 172.21.15.90:0/2380240295 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66b810d380 0x7f66b810f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.547+0000 7f66bd225640 1 --2- 172.21.15.90:0/2380240295 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f66b8102af0 0x7f66b8102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.547+0000 7f66bd225640 1 --2- 172.21.15.90:0/2380240295 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66b8108a20 0x7f66b81025b0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.547+0000 7f66bd225640 1 -- 172.21.15.90:0/2380240295 >> 172.21.15.90:0/2380240295 conn(0x7f66b8078100 msgr2=0x7f66b8078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.547+0000 7f66bd225640 1 -- 172.21.15.90:0/2380240295 shutdown_connections 2024-04-25T11:38:00.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.547+0000 7f66bd225640 1 -- 172.21.15.90:0/2380240295 wait complete. 2024-04-25T11:38:00.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.548+0000 7f66bd225640 1 Processor -- start 2024-04-25T11:38:00.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.548+0000 7f66bd225640 1 -- start start 2024-04-25T11:38:00.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.548+0000 7f66bd225640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66b8102af0 0x7f66b819a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.548+0000 7f66bd225640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f66b8108a20 0x7f66b819ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.548+0000 7f66bd225640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66b810d380 0x7f66b819eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.549+0000 7f66bd225640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f66b8112be0 con 0x7f66b810d380 2024-04-25T11:38:00.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.549+0000 7f66bd225640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f66b8112a60 con 0x7f66b8108a20 2024-04-25T11:38:00.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.549+0000 7f66bd225640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f66b8112d60 con 0x7f66b8102af0 2024-04-25T11:38:00.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.549+0000 7f66b759e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66b810d380 0x7f66b819eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.549+0000 7f66b659c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f66b8108a20 0x7f66b819ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.549+0000 7f66b659c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f66b8108a20 0x7f66b819ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:53006/0 (socket says 172.21.15.90:53006) 2024-04-25T11:38:00.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.549+0000 7f66b659c640 1 -- 172.21.15.90:0/1440953165 learned_addr learned my addr 172.21.15.90:0/1440953165 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:00.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.549+0000 7f66b6d9d640 1 --2- 172.21.15.90:0/1440953165 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66b8102af0 0x7f66b819a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.549+0000 7f66b659c640 1 -- 172.21.15.90:0/1440953165 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66b8102af0 msgr2=0x7f66b819a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.549+0000 7f66b659c640 1 --2- 172.21.15.90:0/1440953165 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66b8102af0 0x7f66b819a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.549+0000 7f66b659c640 1 -- 172.21.15.90:0/1440953165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66b810d380 msgr2=0x7f66b819eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.549+0000 7f66b659c640 1 --2- 172.21.15.90:0/1440953165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66b810d380 0x7f66b819eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.549+0000 7f66b659c640 1 -- 172.21.15.90:0/1440953165 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b819f510 con 0x7f66b8108a20 2024-04-25T11:38:00.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.549+0000 7f66b759e640 1 --2- 172.21.15.90:0/1440953165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66b810d380 0x7f66b819eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:00.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.549+0000 7f66b659c640 1 --2- 172.21.15.90:0/1440953165 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f66b8108a20 0x7f66b819ac10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f66ac00e9f0 tx=0x7f66ac00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.549+0000 7f669ffff640 1 -- 172.21.15.90:0/1440953165 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66ac00ccc0 con 0x7f66b8108a20 2024-04-25T11:38:00.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.549+0000 7f669ffff640 1 -- 172.21.15.90:0/1440953165 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66ac0035f0 con 0x7f66b8108a20 2024-04-25T11:38:00.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.549+0000 7f66bd225640 1 -- 172.21.15.90:0/1440953165 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66b819f800 con 0x7f66b8108a20 2024-04-25T11:38:00.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.550+0000 7f669ffff640 1 -- 172.21.15.90:0/1440953165 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66ac00ce60 con 0x7f66b8108a20 2024-04-25T11:38:00.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.550+0000 7f66bd225640 1 -- 172.21.15.90:0/1440953165 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66b81a7180 con 0x7f66b8108a20 2024-04-25T11:38:00.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.552+0000 7f669ffff640 1 -- 172.21.15.90:0/1440953165 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f66ac010690 con 0x7f66b8108a20 2024-04-25T11:38:00.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.553+0000 7f66c41a9740 1 Processor -- start 2024-04-25T11:38:00.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.553+0000 7f669ffff640 1 --2- 172.21.15.90:0/1440953165 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f66a00afc60 0x7f66a00b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.553+0000 7f66c41a9740 1 -- start start 2024-04-25T11:38:00.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.553+0000 7f66b759e640 1 --2- 172.21.15.90:0/1440953165 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f66a00afc60 0x7f66a00b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.553+0000 7f669ffff640 1 -- 172.21.15.90:0/1440953165 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f66ac0df3e0 con 0x7f66b8108a20 2024-04-25T11:38:00.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.553+0000 7f66c41a9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481c2390 0x559c48563810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.553+0000 7f66c41a9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c481994d0 0x559c48563d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.553+0000 7f66c41a9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c481a16b0 0x559c485706a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.553+0000 7f66c41a9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559c484c8d80 con 0x559c481c2390 2024-04-25T11:38:00.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.553+0000 7f66c41a9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x559c48482a20 con 0x559c481994d0 2024-04-25T11:38:00.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.553+0000 7f66c41a9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559c484c4d40 con 0x559c481a16b0 2024-04-25T11:38:00.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.553+0000 7f66b6d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481c2390 0x559c48563810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.553+0000 7f66b6d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481c2390 0x559c48563810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35460/0 (socket says 172.21.15.90:35460) 2024-04-25T11:38:00.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.553+0000 7f66b6d9d640 1 -- 172.21.15.90:0/2204429732 learned_addr learned my addr 172.21.15.90:0/2204429732 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:00.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.553+0000 7f66b659c640 1 --2- 172.21.15.90:0/2204429732 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c481994d0 0x559c48563d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.553+0000 7f66b759e640 1 --2- 172.21.15.90:0/2204429732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c481a16b0 0x559c485706a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.553+0000 7f66b759e640 1 --2- 172.21.15.90:0/1440953165 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f66a00afc60 0x7f66a00b2170 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f66b0009fd0 tx=0x7f66b0009600 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.554+0000 7f66b6d9d640 1 -- 172.21.15.90:0/2204429732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c481a16b0 msgr2=0x559c485706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.554+0000 7f66b6d9d640 1 --2- 172.21.15.90:0/2204429732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c481a16b0 0x559c485706a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.554+0000 7f66b6d9d640 1 -- 172.21.15.90:0/2204429732 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c481994d0 msgr2=0x559c48563d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.554+0000 7f66b6d9d640 1 --2- 172.21.15.90:0/2204429732 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c481994d0 0x559c48563d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.554+0000 7f66b6d9d640 1 -- 172.21.15.90:0/2204429732 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c4848dc30 con 0x559c481c2390 2024-04-25T11:38:00.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.554+0000 7f66b6d9d640 1 --2- 172.21.15.90:0/2204429732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481c2390 0x559c48563810 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009900 tx=0x7f66a8005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65064626e9b7aa7f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.554+0000 7f669d7fa640 1 -- 172.21.15.90:0/2204429732 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8040070 con 0x559c481c2390 2024-04-25T11:38:00.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.554+0000 7f669d7fa640 1 -- 172.21.15.90:0/2204429732 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66a8032e20 con 0x559c481c2390 2024-04-25T11:38:00.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.554+0000 7f669d7fa640 1 -- 172.21.15.90:0/2204429732 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8035040 con 0x559c481c2390 2024-04-25T11:38:00.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.555+0000 7f66c41a9740 1 -- 172.21.15.90:0/2204429732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481c2390 msgr2=0x559c48563810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.555+0000 7f66c41a9740 1 --2- 172.21.15.90:0/2204429732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481c2390 0x559c48563810 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009900 tx=0x7f66a8005e60 comp rx=0 tx=0).stop 2024-04-25T11:38:00.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.555+0000 7f66c41a9740 1 -- 172.21.15.90:0/2204429732 shutdown_connections 2024-04-25T11:38:00.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.555+0000 7f66c41a9740 1 --2- 172.21.15.90:0/2204429732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c481a16b0 0x559c485706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.555+0000 7f66c41a9740 1 --2- 172.21.15.90:0/2204429732 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c481994d0 0x559c48563d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.555+0000 7f66c41a9740 1 --2- 172.21.15.90:0/2204429732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481c2390 0x559c48563810 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.555+0000 7f66c41a9740 1 -- 172.21.15.90:0/2204429732 >> 172.21.15.90:0/2204429732 conn(0x559c481b64e0 msgr2=0x559c4855ffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.555+0000 7f66c41a9740 1 -- 172.21.15.90:0/2204429732 shutdown_connections 2024-04-25T11:38:00.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.555+0000 7f66c41a9740 1 -- 172.21.15.90:0/2204429732 wait complete. 2024-04-25T11:38:00.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.555+0000 7f66c41a9740 1 -- start start 2024-04-25T11:38:00.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.556+0000 7f66c41a9740 1 Processor -- start 2024-04-25T11:38:00.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.556+0000 7f66c41a9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481994d0 0x559c4856e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.556+0000 7f66c41a9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c481a16b0 0x559c4856e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.556+0000 7f66c41a9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c481c2390 0x559c4857a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.556+0000 7f66c41a9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559c48570d30 con 0x559c481994d0 2024-04-25T11:38:00.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.556+0000 7f66c41a9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x559c4844fdc0 con 0x559c481c2390 2024-04-25T11:38:00.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.556+0000 7f66c41a9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559c48570eb0 con 0x559c481a16b0 2024-04-25T11:38:00.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.556+0000 7f66c41a9740 10 client.0 _ll_register_callbacks cb 0x559c4846b600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:00.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.556+0000 7f66b6d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481994d0 0x559c4856e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.556+0000 7f66b6d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481994d0 0x559c4856e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35468/0 (socket says 172.21.15.90:35468) 2024-04-25T11:38:00.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.556+0000 7f66b6d9d640 1 -- 172.21.15.90:0/3516192625 learned_addr learned my addr 172.21.15.90:0/3516192625 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:00.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.556+0000 7f66b659c640 1 --2- 172.21.15.90:0/3516192625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c481a16b0 0x559c4856e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.556+0000 7f66b759e640 1 --2- 172.21.15.90:0/3516192625 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c481c2390 0x559c4857a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.557+0000 7f66b659c640 1 -- 172.21.15.90:0/3516192625 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c481c2390 msgr2=0x559c4857a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.557+0000 7f66b659c640 1 --2- 172.21.15.90:0/3516192625 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c481c2390 0x559c4857a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.557+0000 7f66b659c640 1 -- 172.21.15.90:0/3516192625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481994d0 msgr2=0x559c4856e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.557+0000 7f66b659c640 1 --2- 172.21.15.90:0/3516192625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481994d0 0x559c4856e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.557+0000 7f66b659c640 1 -- 172.21.15.90:0/3516192625 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c481da3f0 con 0x559c481a16b0 2024-04-25T11:38:00.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.557+0000 7f66b6d9d640 1 --2- 172.21.15.90:0/3516192625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481994d0 0x559c4856e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:00.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.557+0000 7f66b659c640 1 --2- 172.21.15.90:0/3516192625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c481a16b0 0x559c4856e7c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f66ac0dcad0 tx=0x7f66ac0a7ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbf509ee58ed4c24 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.557+0000 7f669aff5640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:38:00.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.557+0000 7f669aff5640 1 -- 172.21.15.90:0/3516192625 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66ac0c0020 con 0x559c481a16b0 2024-04-25T11:38:00.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.557+0000 7f669aff5640 1 -- 172.21.15.90:0/3516192625 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f66ac003aa0 con 0x559c481a16b0 2024-04-25T11:38:00.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.557+0000 7f66c41a9740 1 -- 172.21.15.90:0/3516192625 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66a8032e20 con 0x559c481a16b0 2024-04-25T11:38:00.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.557+0000 7f669aff5640 1 -- 172.21.15.90:0/3516192625 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f66ac003ca0 con 0x559c481a16b0 2024-04-25T11:38:00.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.557+0000 7f66c41a9740 10 client.25699 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:00.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.557+0000 7f66c41a9740 1 -- 172.21.15.90:0/3516192625 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559c482a47d0 con 0x559c481a16b0 2024-04-25T11:38:00.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.558+0000 7f669aff5640 1 -- 172.21.15.90:0/3516192625 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66a8032e20 con 0x559c481a16b0 2024-04-25T11:38:00.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.558+0000 7f669aff5640 1 -- 172.21.15.90:0/3516192625 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f66ac082b10 con 0x559c481a16b0 2024-04-25T11:38:00.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.558+0000 7f66c41a9740 10 client.25699 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:00.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.558+0000 7f66c41a9740 10 client.25699 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:38:00.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.558+0000 7f66c41a9740 20 client.25699 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:00.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.558+0000 7f66c41a9740 1 --2- 172.21.15.90:0/3516192625 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x559c48180f60 0x559c48580730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.558+0000 7f66c41a9740 4 client.25699 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:38:00.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.558+0000 7f66c41a9740 1 -- 172.21.15.90:0/3516192625 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c484d6990 con 0x559c48180f60 2024-04-25T11:38:00.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.558+0000 7f66b6d9d640 1 --2- 172.21.15.90:0/3516192625 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x559c48180f60 0x559c48580730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.559+0000 7f669aff5640 10 client.25699 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:38:00.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.559+0000 7f66b6d9d640 1 --2- 172.21.15.90:0/3516192625 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x559c48180f60 0x559c48580730 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7bba8255349d3460 server_cookie=3dd1601402b0c47d in_seq=0 out_seq=0 2024-04-25T11:38:00.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.560+0000 7f669aff5640 1 -- 172.21.15.90:0/3516192625 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559c484d6990 con 0x559c48180f60 2024-04-25T11:38:00.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.560+0000 7f669aff5640 10 client.25699 handle_command_reply: tid=0 2024-04-25T11:38:00.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.560+0000 7f66c41a9740 1 client.25699 shutdown 2024-04-25T11:38:00.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.560+0000 7f66c41a9740 1 -- 172.21.15.90:0/3516192625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c481a16b0 msgr2=0x559c4856e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.560+0000 7f66c41a9740 1 --2- 172.21.15.90:0/3516192625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c481a16b0 0x559c4856e7c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f66ac0dcad0 tx=0x7f66ac0a7ce0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.560+0000 7f66c41a9740 1 -- 172.21.15.90:0/3516192625 shutdown_connections 2024-04-25T11:38:00.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.560+0000 7f66c41a9740 1 --2- 172.21.15.90:0/3516192625 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x559c48180f60 0x559c48580730 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.560+0000 7f66c41a9740 1 --2- 172.21.15.90:0/3516192625 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c481c2390 0x559c4857a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.560+0000 7f669aff5640 0 client.25699 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:38:00.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.560+0000 7f66c41a9740 1 --2- 172.21.15.90:0/3516192625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c481a16b0 0x559c4856e7c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.560+0000 7f66c41a9740 1 --2- 172.21.15.90:0/3516192625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481994d0 0x559c4856e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.560+0000 7f66c41a9740 1 -- 172.21.15.90:0/3516192625 >> 172.21.15.90:0/3516192625 conn(0x559c481b64e0 msgr2=0x559c48565530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.560+0000 7f66c41a9740 1 -- 172.21.15.90:0/3516192625 shutdown_connections 2024-04-25T11:38:00.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.560+0000 7f66c41a9740 1 -- 172.21.15.90:0/3516192625 wait complete. 2024-04-25T11:38:00.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.561+0000 7f66c41a9740 20 client.25699 trim_cache size 0 max 16384 2024-04-25T11:38:00.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.575+0000 7f66c41a9740 1 Processor -- start 2024-04-25T11:38:00.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.575+0000 7f66c41a9740 1 -- start start 2024-04-25T11:38:00.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.575+0000 7f66c41a9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c48180f60 0x559c48579e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.575+0000 7f66c41a9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c481994d0 0x559c48568510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.575+0000 7f66c41a9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481a16b0 0x559c48585ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.575+0000 7f66c41a9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f66b81af990 con 0x559c481a16b0 2024-04-25T11:38:00.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.575+0000 7f66c41a9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f66b81b1500 con 0x559c48180f60 2024-04-25T11:38:00.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.575+0000 7f66c41a9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f66b81b0060 con 0x559c481994d0 2024-04-25T11:38:00.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.575+0000 7f66b759e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481a16b0 0x559c48585ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.575+0000 7f66b6d9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c48180f60 0x559c48579e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.575+0000 7f66b759e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481a16b0 0x559c48585ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35470/0 (socket says 172.21.15.90:35470) 2024-04-25T11:38:00.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.575+0000 7f66b759e640 1 -- 172.21.15.90:0/1790335815 learned_addr learned my addr 172.21.15.90:0/1790335815 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:00.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.575+0000 7f66b659c640 1 --2- 172.21.15.90:0/1790335815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c481994d0 0x559c48568510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.576+0000 7f66b6d9d640 1 -- 172.21.15.90:0/1790335815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c481994d0 msgr2=0x559c48568510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.576+0000 7f66b6d9d640 1 --2- 172.21.15.90:0/1790335815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c481994d0 0x559c48568510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.576+0000 7f66b6d9d640 1 -- 172.21.15.90:0/1790335815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481a16b0 msgr2=0x559c48585ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.576+0000 7f66b6d9d640 1 --2- 172.21.15.90:0/1790335815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481a16b0 0x559c48585ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.576+0000 7f66b6d9d640 1 -- 172.21.15.90:0/1790335815 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c48564fc0 con 0x559c48180f60 2024-04-25T11:38:00.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.576+0000 7f66b759e640 1 --2- 172.21.15.90:0/1790335815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481a16b0 0x559c48585ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:00.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.576+0000 7f66b6d9d640 1 --2- 172.21.15.90:0/1790335815 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c48180f60 0x559c48579e10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x559c48580e90 tx=0x7f66a80045a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de5e2966f66b215b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.576+0000 7f669cff9640 1 -- 172.21.15.90:0/1790335815 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8054070 con 0x559c48180f60 2024-04-25T11:38:00.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.576+0000 7f669cff9640 1 -- 172.21.15.90:0/1790335815 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66a80054f0 con 0x559c48180f60 2024-04-25T11:38:00.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.576+0000 7f669cff9640 1 -- 172.21.15.90:0/1790335815 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8040030 con 0x559c48180f60 2024-04-25T11:38:00.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.577+0000 7f66c41a9740 1 -- 172.21.15.90:0/1790335815 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c48180f60 msgr2=0x559c48579e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.577+0000 7f66c41a9740 1 --2- 172.21.15.90:0/1790335815 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c48180f60 0x559c48579e10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x559c48580e90 tx=0x7f66a80045a0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.577+0000 7f66c41a9740 1 -- 172.21.15.90:0/1790335815 shutdown_connections 2024-04-25T11:38:00.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.577+0000 7f66c41a9740 1 --2- 172.21.15.90:0/1790335815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481a16b0 0x559c48585ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.577+0000 7f66c41a9740 1 --2- 172.21.15.90:0/1790335815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c481994d0 0x559c48568510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.577+0000 7f66c41a9740 1 --2- 172.21.15.90:0/1790335815 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c48180f60 0x559c48579e10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.577+0000 7f66c41a9740 1 -- 172.21.15.90:0/1790335815 >> 172.21.15.90:0/1790335815 conn(0x559c481b64e0 msgr2=0x559c4856c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.577+0000 7f66c41a9740 1 -- 172.21.15.90:0/1790335815 shutdown_connections 2024-04-25T11:38:00.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.577+0000 7f66c41a9740 1 -- 172.21.15.90:0/1790335815 wait complete. 2024-04-25T11:38:00.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.577+0000 7f66c41a9740 1 -- start start 2024-04-25T11:38:00.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.578+0000 7f66c41a9740 1 Processor -- start 2024-04-25T11:38:00.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.578+0000 7f66c41a9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c48180f60 0x559c485767e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.578+0000 7f66c41a9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481994d0 0x559c48571b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.578+0000 7f66c41a9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c481a16b0 0x559c48572110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.578+0000 7f66c41a9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f66b81b0e00 con 0x559c481994d0 2024-04-25T11:38:00.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.578+0000 7f66c41a9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f66b81b0730 con 0x559c48180f60 2024-04-25T11:38:00.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.578+0000 7f66c41a9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f66b81ad970 con 0x559c481a16b0 2024-04-25T11:38:00.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.578+0000 7f66c41a9740 10 client.0 _ll_register_callbacks cb 0x7f66ac0a6ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:00.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.578+0000 7f66b659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481994d0 0x559c48571b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.578+0000 7f66b6d9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c48180f60 0x559c485767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.578+0000 7f66b6d9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c48180f60 0x559c485767e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:53046/0 (socket says 172.21.15.90:53046) 2024-04-25T11:38:00.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.578+0000 7f66b659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481994d0 0x559c48571b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35480/0 (socket says 172.21.15.90:35480) 2024-04-25T11:38:00.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.578+0000 7f66b6d9d640 1 -- 172.21.15.90:0/1614709902 learned_addr learned my addr 172.21.15.90:0/1614709902 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:00.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.578+0000 7f66b759e640 1 --2- 172.21.15.90:0/1614709902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c481a16b0 0x559c48572110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.578+0000 7f66b6d9d640 1 -- 172.21.15.90:0/1614709902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c481a16b0 msgr2=0x559c48572110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.578+0000 7f66b6d9d640 1 --2- 172.21.15.90:0/1614709902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c481a16b0 0x559c48572110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.578+0000 7f66b6d9d640 1 -- 172.21.15.90:0/1614709902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481994d0 msgr2=0x559c48571b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.578+0000 7f66b6d9d640 1 --2- 172.21.15.90:0/1614709902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481994d0 0x559c48571b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.578+0000 7f66b6d9d640 1 -- 172.21.15.90:0/1614709902 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c48572870 con 0x559c48180f60 2024-04-25T11:38:00.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.578+0000 7f66b759e640 1 --2- 172.21.15.90:0/1614709902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c481a16b0 0x559c48572110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:38:00.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.578+0000 7f66b659c640 1 --2- 172.21.15.90:0/1614709902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481994d0 0x559c48571b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:00.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.579+0000 7f66b6d9d640 1 --2- 172.21.15.90:0/1614709902 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c48180f60 0x559c485767e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f66a802a930 tx=0x7f66a802a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=620ed5eaefc2821c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.579+0000 7f669aff5640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:38:00.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.579+0000 7f669aff5640 1 -- 172.21.15.90:0/1614709902 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8054070 con 0x559c48180f60 2024-04-25T11:38:00.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.579+0000 7f669aff5640 1 -- 172.21.15.90:0/1614709902 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f66a8005250 con 0x559c48180f60 2024-04-25T11:38:00.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.579+0000 7f66c41a9740 1 -- 172.21.15.90:0/1614709902 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66a80054f0 con 0x559c48180f60 2024-04-25T11:38:00.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.579+0000 7f669aff5640 1 -- 172.21.15.90:0/1614709902 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f66a8040070 con 0x559c48180f60 2024-04-25T11:38:00.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.579+0000 7f66c41a9740 10 client.25769 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:00.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.579+0000 7f66c41a9740 1 -- 172.21.15.90:0/1614709902 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559c485e8a50 con 0x559c48180f60 2024-04-25T11:38:00.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.579+0000 7f669aff5640 1 -- 172.21.15.90:0/1614709902 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66a80054f0 con 0x559c48180f60 2024-04-25T11:38:00.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.579+0000 7f669aff5640 1 -- 172.21.15.90:0/1614709902 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f66a804f620 con 0x559c48180f60 2024-04-25T11:38:00.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.579+0000 7f66c41a9740 10 client.25769 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:00.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.579+0000 7f66c41a9740 10 client.25769 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:38:00.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.579+0000 7f66c41a9740 20 client.25769 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:00.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.579+0000 7f66c41a9740 1 --2- 172.21.15.90:0/1614709902 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x559c481c2390 0x559c485ead80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.579+0000 7f66c41a9740 4 client.25769 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:38:00.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.579+0000 7f66c41a9740 1 -- 172.21.15.90:0/1614709902 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559c484d0800 con 0x559c481c2390 2024-04-25T11:38:00.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.580+0000 7f66b659c640 1 --2- 172.21.15.90:0/1614709902 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x559c481c2390 0x559c485ead80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.580+0000 7f66b659c640 1 --2- 172.21.15.90:0/1614709902 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x559c481c2390 0x559c485ead80 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dea549f8f0a541d8 server_cookie=10c14aadc116f323 in_seq=0 out_seq=0 2024-04-25T11:38:00.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.580+0000 7f669aff5640 10 client.25769 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:38:00.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.580+0000 7f669aff5640 1 -- 172.21.15.90:0/1614709902 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559c484d0800 con 0x559c481c2390 2024-04-25T11:38:00.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.580+0000 7f669aff5640 10 client.25769 handle_command_reply: tid=0 2024-04-25T11:38:00.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.580+0000 7f66c41a9740 1 client.25769 shutdown 2024-04-25T11:38:00.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.580+0000 7f66c41a9740 1 -- 172.21.15.90:0/1614709902 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c48180f60 msgr2=0x559c485767e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.580+0000 7f66c41a9740 1 --2- 172.21.15.90:0/1614709902 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c48180f60 0x559c485767e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f66a802a930 tx=0x7f66a802a410 comp rx=0 tx=0).stop 2024-04-25T11:38:00.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.581+0000 7f66c41a9740 1 -- 172.21.15.90:0/1614709902 shutdown_connections 2024-04-25T11:38:00.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.581+0000 7f66c41a9740 1 --2- 172.21.15.90:0/1614709902 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x559c481c2390 0x559c485ead80 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.581+0000 7f66c41a9740 1 --2- 172.21.15.90:0/1614709902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c481a16b0 0x559c48572110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.581+0000 7f66c41a9740 1 --2- 172.21.15.90:0/1614709902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c481994d0 0x559c48571b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.581+0000 7f669aff5640 0 client.25769 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:38:00.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.581+0000 7f66c41a9740 1 --2- 172.21.15.90:0/1614709902 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c48180f60 0x559c485767e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.581+0000 7f66c41a9740 1 -- 172.21.15.90:0/1614709902 >> 172.21.15.90:0/1614709902 conn(0x559c481b64e0 msgr2=0x559c48570130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.581+0000 7f66c41a9740 1 -- 172.21.15.90:0/1614709902 shutdown_connections 2024-04-25T11:38:00.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.581+0000 7f66c41a9740 1 -- 172.21.15.90:0/1614709902 wait complete. 2024-04-25T11:38:00.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.581+0000 7f66c41a9740 20 client.25769 trim_cache size 0 max 16384 2024-04-25T11:38:00.574 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:38:00.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.581+0000 7f66bd225640 1 -- 172.21.15.90:0/1440953165 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f66a00afc60 msgr2=0x7f66a00b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.581+0000 7f66bd225640 1 --2- 172.21.15.90:0/1440953165 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f66a00afc60 0x7f66a00b2170 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f66b0009fd0 tx=0x7f66b0009600 comp rx=0 tx=0).stop 2024-04-25T11:38:00.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.581+0000 7f66bd225640 1 -- 172.21.15.90:0/1440953165 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f66b8108a20 msgr2=0x7f66b819ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.581+0000 7f66bd225640 1 --2- 172.21.15.90:0/1440953165 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f66b8108a20 0x7f66b819ac10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f66ac00e9f0 tx=0x7f66ac00eeb0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.581+0000 7f66bd225640 1 -- 172.21.15.90:0/1440953165 shutdown_connections 2024-04-25T11:38:00.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.581+0000 7f66bd225640 1 --2- 172.21.15.90:0/1440953165 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f66a00afc60 0x7f66a00b2170 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.581+0000 7f66bd225640 1 --2- 172.21.15.90:0/1440953165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66b810d380 0x7f66b819eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.581+0000 7f66bd225640 1 --2- 172.21.15.90:0/1440953165 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f66b8108a20 0x7f66b819ac10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.581+0000 7f66bd225640 1 --2- 172.21.15.90:0/1440953165 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66b8102af0 0x7f66b819a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.581+0000 7f66bd225640 1 -- 172.21.15.90:0/1440953165 >> 172.21.15.90:0/1440953165 conn(0x7f66b8078100 msgr2=0x7f66b8069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.581+0000 7f66bd225640 1 -- 172.21.15.90:0/1440953165 shutdown_connections 2024-04-25T11:38:00.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.582+0000 7f66bd225640 1 -- 172.21.15.90:0/1440953165 wait complete. 2024-04-25T11:38:00.582 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T11:38:00.725 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:00 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2489337666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:38:00.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.779+0000 7f0941bb2640 1 -- 172.21.15.90:0/4141933835 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f093c108a20 msgr2=0x7f093c1025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.779+0000 7f0941bb2640 1 --2- 172.21.15.90:0/4141933835 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f093c108a20 0x7f093c1025b0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f092c009900 tx=0x7f092c02a410 comp rx=0 tx=0).stop 2024-04-25T11:38:00.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.779+0000 7f0941bb2640 1 -- 172.21.15.90:0/4141933835 shutdown_connections 2024-04-25T11:38:00.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.779+0000 7f0941bb2640 1 --2- 172.21.15.90:0/4141933835 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f093c10d380 0x7f093c10f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.779+0000 7f0941bb2640 1 --2- 172.21.15.90:0/4141933835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f093c102af0 0x7f093c102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.779+0000 7f0941bb2640 1 --2- 172.21.15.90:0/4141933835 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f093c108a20 0x7f093c1025b0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.779+0000 7f0941bb2640 1 -- 172.21.15.90:0/4141933835 >> 172.21.15.90:0/4141933835 conn(0x7f093c078100 msgr2=0x7f093c078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.780+0000 7f0941bb2640 1 -- 172.21.15.90:0/4141933835 shutdown_connections 2024-04-25T11:38:00.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.780+0000 7f0941bb2640 1 -- 172.21.15.90:0/4141933835 wait complete. 2024-04-25T11:38:00.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.780+0000 7f0941bb2640 1 Processor -- start 2024-04-25T11:38:00.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.780+0000 7f0941bb2640 1 -- start start 2024-04-25T11:38:00.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.781+0000 7f0941bb2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f093c102af0 0x7f093c1a04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.781+0000 7f0941bb2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f093c108a20 0x7f093c1a0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.781+0000 7f0941bb2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f093c10d380 0x7f093c19a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.781+0000 7f0941bb2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f093c112cc0 con 0x7f093c102af0 2024-04-25T11:38:00.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.781+0000 7f0941bb2640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f093c112b40 con 0x7f093c10d380 2024-04-25T11:38:00.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.781+0000 7f0941bb2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f093c112e40 con 0x7f093c108a20 2024-04-25T11:38:00.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.781+0000 7f093b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f093c102af0 0x7f093c1a04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.781+0000 7f093b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f093c102af0 0x7f093c1a04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35496/0 (socket says 172.21.15.90:35496) 2024-04-25T11:38:00.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.781+0000 7f093b7fe640 1 -- 172.21.15.90:0/4044277163 learned_addr learned my addr 172.21.15.90:0/4044277163 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:00.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.781+0000 7f093b7fe640 1 -- 172.21.15.90:0/4044277163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f093c108a20 msgr2=0x7f093c1a0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.781+0000 7f093b7fe640 1 --2- 172.21.15.90:0/4044277163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f093c108a20 0x7f093c1a0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.781+0000 7f093b7fe640 1 -- 172.21.15.90:0/4044277163 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f093c10d380 msgr2=0x7f093c19a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.781+0000 7f093bfff640 1 --2- 172.21.15.90:0/4044277163 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f093c10d380 0x7f093c19a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.781+0000 7f093b7fe640 1 --2- 172.21.15.90:0/4044277163 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f093c10d380 0x7f093c19a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.781+0000 7f093b7fe640 1 -- 172.21.15.90:0/4044277163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f093c19ada0 con 0x7f093c102af0 2024-04-25T11:38:00.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.781+0000 7f093bfff640 1 --2- 172.21.15.90:0/4044277163 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f093c10d380 0x7f093c19a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:38:00.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.782+0000 7f093b7fe640 1 --2- 172.21.15.90:0/4044277163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f093c102af0 0x7f093c1a04f0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f092c031a90 tx=0x7f092c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.782+0000 7f0938ff9640 1 -- 172.21.15.90:0/4044277163 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f092c03f030 con 0x7f093c102af0 2024-04-25T11:38:00.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.782+0000 7f0941bb2640 1 -- 172.21.15.90:0/4044277163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f093c19afd0 con 0x7f093c102af0 2024-04-25T11:38:00.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.782+0000 7f0938ff9640 1 -- 172.21.15.90:0/4044277163 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f092c003cb0 con 0x7f093c102af0 2024-04-25T11:38:00.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.782+0000 7f0938ff9640 1 -- 172.21.15.90:0/4044277163 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f092c035070 con 0x7f093c102af0 2024-04-25T11:38:00.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.782+0000 7f0941bb2640 1 -- 172.21.15.90:0/4044277163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f093c1a7290 con 0x7f093c102af0 2024-04-25T11:38:00.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.784+0000 7f0948a63740 1 Processor -- start 2024-04-25T11:38:00.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.785+0000 7f0948a63740 1 -- start start 2024-04-25T11:38:00.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.785+0000 7f0948a63740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d1a1e3390 0x559d1a584810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.785+0000 7f0948a63740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1ba4d0 0x559d1a584d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.785+0000 7f0948a63740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d1a1c26b0 0x559d1a5916a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.785+0000 7f0948a63740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559d1a4e9d80 con 0x559d1a1ba4d0 2024-04-25T11:38:00.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.785+0000 7f0948a63740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x559d1a4a3a20 con 0x559d1a1e3390 2024-04-25T11:38:00.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.785+0000 7f0948a63740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559d1a4e5d40 con 0x559d1a1c26b0 2024-04-25T11:38:00.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.785+0000 7f0938ff9640 1 -- 172.21.15.90:0/4044277163 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f092c042030 con 0x7f093c102af0 2024-04-25T11:38:00.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.785+0000 7f093b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1ba4d0 0x559d1a584d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.786+0000 7f093b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1ba4d0 0x559d1a584d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35500/0 (socket says 172.21.15.90:35500) 2024-04-25T11:38:00.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.786+0000 7f093b7fe640 1 -- 172.21.15.90:0/840760132 learned_addr learned my addr 172.21.15.90:0/840760132 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:00.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.786+0000 7f093bfff640 1 --2- 172.21.15.90:0/840760132 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d1a1e3390 0x559d1a584810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.786+0000 7f093affd640 1 --2- 172.21.15.90:0/840760132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d1a1c26b0 0x559d1a5916a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.786+0000 7f093b7fe640 1 -- 172.21.15.90:0/840760132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d1a1c26b0 msgr2=0x559d1a5916a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.786+0000 7f093b7fe640 1 --2- 172.21.15.90:0/840760132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d1a1c26b0 0x559d1a5916a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.786+0000 7f093b7fe640 1 -- 172.21.15.90:0/840760132 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d1a1e3390 msgr2=0x559d1a584810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.786+0000 7f0938ff9640 1 --2- 172.21.15.90:0/4044277163 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f09240afc60 0x7f09240b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.786+0000 7f093b7fe640 1 --2- 172.21.15.90:0/840760132 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d1a1e3390 0x559d1a584810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.786+0000 7f093b7fe640 1 -- 172.21.15.90:0/840760132 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d1a4aec30 con 0x559d1a1ba4d0 2024-04-25T11:38:00.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.786+0000 7f093bfff640 1 --2- 172.21.15.90:0/4044277163 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f09240afc60 0x7f09240b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.786+0000 7f093b7fe640 1 --2- 172.21.15.90:0/840760132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1ba4d0 0x559d1a584d50 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f092c0fcdc0 tx=0x7f092c0c3760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca9e082befe0410b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.787+0000 7f0938ff9640 1 -- 172.21.15.90:0/4044277163 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f092c0ff760 con 0x7f093c102af0 2024-04-25T11:38:00.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.787+0000 7f0921ffb640 1 -- 172.21.15.90:0/840760132 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f092c0c7670 con 0x559d1a1ba4d0 2024-04-25T11:38:00.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.787+0000 7f0921ffb640 1 -- 172.21.15.90:0/840760132 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f092c0c73b0 con 0x559d1a1ba4d0 2024-04-25T11:38:00.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.787+0000 7f0921ffb640 1 -- 172.21.15.90:0/840760132 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f092c0c3030 con 0x559d1a1ba4d0 2024-04-25T11:38:00.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.788+0000 7f0948a63740 1 -- 172.21.15.90:0/840760132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1ba4d0 msgr2=0x559d1a584d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.788+0000 7f093bfff640 1 --2- 172.21.15.90:0/4044277163 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f09240afc60 0x7f09240b2170 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x559d1a581980 tx=0x7f0934004b50 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.788+0000 7f0948a63740 1 --2- 172.21.15.90:0/840760132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1ba4d0 0x559d1a584d50 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f092c0fcdc0 tx=0x7f092c0c3760 comp rx=0 tx=0).stop 2024-04-25T11:38:00.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.788+0000 7f0948a63740 1 -- 172.21.15.90:0/840760132 shutdown_connections 2024-04-25T11:38:00.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.788+0000 7f0948a63740 1 --2- 172.21.15.90:0/840760132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d1a1c26b0 0x559d1a5916a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.788+0000 7f0948a63740 1 --2- 172.21.15.90:0/840760132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1ba4d0 0x559d1a584d50 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.788+0000 7f0948a63740 1 --2- 172.21.15.90:0/840760132 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d1a1e3390 0x559d1a584810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.788+0000 7f0948a63740 1 -- 172.21.15.90:0/840760132 >> 172.21.15.90:0/840760132 conn(0x559d1a1d74e0 msgr2=0x559d1a580fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.788+0000 7f0948a63740 1 -- 172.21.15.90:0/840760132 shutdown_connections 2024-04-25T11:38:00.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.788+0000 7f0948a63740 1 -- 172.21.15.90:0/840760132 wait complete. 2024-04-25T11:38:00.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.789+0000 7f0948a63740 1 -- start start 2024-04-25T11:38:00.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.789+0000 7f0948a63740 1 Processor -- start 2024-04-25T11:38:00.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.789+0000 7f0948a63740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d1a1ba4d0 0x559d1a58f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.789+0000 7f0948a63740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d1a1c26b0 0x559d1a58f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.789+0000 7f0948a63740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1e3390 0x559d1a59b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.789+0000 7f0948a63740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559d1a591c40 con 0x559d1a1e3390 2024-04-25T11:38:00.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.789+0000 7f0948a63740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x559d1a470dc0 con 0x559d1a1c26b0 2024-04-25T11:38:00.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.789+0000 7f0948a63740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559d1a591dc0 con 0x559d1a1ba4d0 2024-04-25T11:38:00.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.789+0000 7f0948a63740 10 client.0 _ll_register_callbacks cb 0x559d1a48c600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:00.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.790+0000 7f093bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1e3390 0x559d1a59b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.790+0000 7f093bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1e3390 0x559d1a59b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35508/0 (socket says 172.21.15.90:35508) 2024-04-25T11:38:00.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.790+0000 7f093bfff640 1 -- 172.21.15.90:0/1921926426 learned_addr learned my addr 172.21.15.90:0/1921926426 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:00.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.790+0000 7f093bfff640 1 -- 172.21.15.90:0/1921926426 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d1a1ba4d0 msgr2=0x559d1a58f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.790+0000 7f093bfff640 1 --2- 172.21.15.90:0/1921926426 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d1a1ba4d0 0x559d1a58f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.790+0000 7f093bfff640 1 -- 172.21.15.90:0/1921926426 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d1a1c26b0 msgr2=0x559d1a58f7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:38:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.790+0000 7f093bfff640 1 --2- 172.21.15.90:0/1921926426 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d1a1c26b0 0x559d1a58f7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.790+0000 7f093bfff640 1 -- 172.21.15.90:0/1921926426 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d1a1fb3f0 con 0x559d1a1e3390 2024-04-25T11:38:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.790+0000 7f093bfff640 1 --2- 172.21.15.90:0/1921926426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1e3390 0x559d1a59b150 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f0934011a90 tx=0x7f09340075f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f3228488f1e164a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.790+0000 7f091affd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:38:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.790+0000 7f091affd640 1 -- 172.21.15.90:0/1921926426 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0934022020 con 0x559d1a1e3390 2024-04-25T11:38:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.790+0000 7f0948a63740 1 -- 172.21.15.90:0/1921926426 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f092c0c73b0 con 0x559d1a1e3390 2024-04-25T11:38:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.790+0000 7f091affd640 1 -- 172.21.15.90:0/1921926426 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f093401e450 con 0x559d1a1e3390 2024-04-25T11:38:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.791+0000 7f091affd640 1 -- 172.21.15.90:0/1921926426 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0934010e10 con 0x559d1a1e3390 2024-04-25T11:38:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.791+0000 7f0948a63740 10 client.16476 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.791+0000 7f0948a63740 1 -- 172.21.15.90:0/1921926426 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559d1a2c57d0 con 0x559d1a1e3390 2024-04-25T11:38:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.791+0000 7f091affd640 1 -- 172.21.15.90:0/1921926426 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f092c0c73b0 con 0x559d1a1e3390 2024-04-25T11:38:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.791+0000 7f091affd640 1 -- 172.21.15.90:0/1921926426 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f093401d780 con 0x559d1a1e3390 2024-04-25T11:38:00.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.791+0000 7f0948a63740 10 client.16476 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:00.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.791+0000 7f0948a63740 10 client.16476 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:38:00.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.791+0000 7f0948a63740 20 client.16476 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:00.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.791+0000 7f0948a63740 1 --2- 172.21.15.90:0/1921926426 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x559d1a1a1f60 0x559d1a5a1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.791+0000 7f0948a63740 4 client.16476 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:38:00.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.791+0000 7f0948a63740 1 -- 172.21.15.90:0/1921926426 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559d1a4f7990 con 0x559d1a1a1f60 2024-04-25T11:38:00.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.792+0000 7f093b7fe640 1 --2- 172.21.15.90:0/1921926426 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x559d1a1a1f60 0x559d1a5a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.793+0000 7f093b7fe640 1 --2- 172.21.15.90:0/1921926426 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x559d1a1a1f60 0x559d1a5a1780 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=41962338496d200f server_cookie=bcb5b4c188081331 in_seq=0 out_seq=0 2024-04-25T11:38:00.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.793+0000 7f091affd640 10 client.16476 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:38:00.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.793+0000 7f091affd640 1 -- 172.21.15.90:0/1921926426 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559d1a4f7990 con 0x559d1a1a1f60 2024-04-25T11:38:00.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.793+0000 7f091affd640 10 client.16476 handle_command_reply: tid=0 2024-04-25T11:38:00.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.794+0000 7f0948a63740 1 client.16476 shutdown 2024-04-25T11:38:00.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.794+0000 7f0948a63740 1 -- 172.21.15.90:0/1921926426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1e3390 msgr2=0x559d1a59b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.794+0000 7f0948a63740 1 --2- 172.21.15.90:0/1921926426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1e3390 0x559d1a59b150 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f0934011a90 tx=0x7f09340075f0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.794+0000 7f0948a63740 1 -- 172.21.15.90:0/1921926426 shutdown_connections 2024-04-25T11:38:00.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.794+0000 7f0948a63740 1 --2- 172.21.15.90:0/1921926426 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x559d1a1a1f60 0x559d1a5a1780 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.794+0000 7f0948a63740 1 --2- 172.21.15.90:0/1921926426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1e3390 0x559d1a59b150 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.794+0000 7f0948a63740 1 --2- 172.21.15.90:0/1921926426 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d1a1c26b0 0x559d1a58f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.794+0000 7f0948a63740 1 --2- 172.21.15.90:0/1921926426 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d1a1ba4d0 0x559d1a58f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.794+0000 7f091affd640 0 client.16476 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:38:00.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.794+0000 7f0948a63740 1 -- 172.21.15.90:0/1921926426 >> 172.21.15.90:0/1921926426 conn(0x559d1a1d74e0 msgr2=0x559d1a586530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.794+0000 7f0948a63740 1 -- 172.21.15.90:0/1921926426 shutdown_connections 2024-04-25T11:38:00.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.794+0000 7f0948a63740 1 -- 172.21.15.90:0/1921926426 wait complete. 2024-04-25T11:38:00.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.794+0000 7f0948a63740 20 client.16476 trim_cache size 0 max 16384 2024-04-25T11:38:00.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:00 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2489337666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:38:00.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.813+0000 7f0948a63740 1 Processor -- start 2024-04-25T11:38:00.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.813+0000 7f0948a63740 1 -- start start 2024-04-25T11:38:00.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.814+0000 7f0948a63740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d1a1a1f60 0x559d1a581010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.814+0000 7f0948a63740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1ba4d0 0x559d1a59adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.814+0000 7f0948a63740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d1a1c26b0 0x559d1a589430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.814+0000 7f0948a63740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f093c1afbe0 con 0x559d1a1ba4d0 2024-04-25T11:38:00.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.814+0000 7f0948a63740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f093c1b1750 con 0x559d1a1c26b0 2024-04-25T11:38:00.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.814+0000 7f0948a63740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f093c1b02b0 con 0x559d1a1a1f60 2024-04-25T11:38:00.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.814+0000 7f093affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1ba4d0 0x559d1a59adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.814+0000 7f093affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1ba4d0 0x559d1a59adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35522/0 (socket says 172.21.15.90:35522) 2024-04-25T11:38:00.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.814+0000 7f093affd640 1 -- 172.21.15.90:0/184818284 learned_addr learned my addr 172.21.15.90:0/184818284 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:00.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.814+0000 7f093affd640 1 -- 172.21.15.90:0/184818284 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d1a1a1f60 msgr2=0x559d1a581010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.814+0000 7f093affd640 1 --2- 172.21.15.90:0/184818284 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d1a1a1f60 0x559d1a581010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.814+0000 7f093affd640 1 -- 172.21.15.90:0/184818284 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d1a1c26b0 msgr2=0x559d1a589430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.814+0000 7f093affd640 1 --2- 172.21.15.90:0/184818284 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d1a1c26b0 0x559d1a589430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.814+0000 7f093affd640 1 -- 172.21.15.90:0/184818284 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d1a58fb90 con 0x559d1a1ba4d0 2024-04-25T11:38:00.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.815+0000 7f093affd640 1 --2- 172.21.15.90:0/184818284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1ba4d0 0x559d1a59adb0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f093000b560 tx=0x7f093000ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cd884206233949b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.815+0000 7f09217fa640 1 -- 172.21.15.90:0/184818284 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09300125f0 con 0x559d1a1ba4d0 2024-04-25T11:38:00.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.815+0000 7f09217fa640 1 -- 172.21.15.90:0/184818284 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0930019070 con 0x559d1a1ba4d0 2024-04-25T11:38:00.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.815+0000 7f09217fa640 1 -- 172.21.15.90:0/184818284 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09300125f0 con 0x559d1a1ba4d0 2024-04-25T11:38:00.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.816+0000 7f0948a63740 1 -- 172.21.15.90:0/184818284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1ba4d0 msgr2=0x559d1a59adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.817+0000 7f0948a63740 1 --2- 172.21.15.90:0/184818284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1ba4d0 0x559d1a59adb0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f093000b560 tx=0x7f093000ba20 comp rx=0 tx=0).stop 2024-04-25T11:38:00.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.817+0000 7f0948a63740 1 -- 172.21.15.90:0/184818284 shutdown_connections 2024-04-25T11:38:00.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.817+0000 7f0948a63740 1 --2- 172.21.15.90:0/184818284 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d1a1c26b0 0x559d1a589430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.817+0000 7f0948a63740 1 --2- 172.21.15.90:0/184818284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1ba4d0 0x559d1a59adb0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.817+0000 7f0948a63740 1 --2- 172.21.15.90:0/184818284 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d1a1a1f60 0x559d1a581010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.817+0000 7f0948a63740 1 -- 172.21.15.90:0/184818284 >> 172.21.15.90:0/184818284 conn(0x559d1a1d74e0 msgr2=0x559d1a58d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.817+0000 7f0948a63740 1 -- 172.21.15.90:0/184818284 shutdown_connections 2024-04-25T11:38:00.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.817+0000 7f0948a63740 1 -- 172.21.15.90:0/184818284 wait complete. 2024-04-25T11:38:00.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.817+0000 7f0948a63740 1 -- start start 2024-04-25T11:38:00.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.818+0000 7f0948a63740 1 Processor -- start 2024-04-25T11:38:00.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.818+0000 7f0948a63740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d1a1a1f60 0x559d1a5923c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.818+0000 7f0948a63740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d1a1ba4d0 0x559d1a592900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.818+0000 7f0948a63740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1c26b0 0x559d1a60bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.818+0000 7f093bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1c26b0 0x559d1a60bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.818+0000 7f093bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1c26b0 0x559d1a60bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35538/0 (socket says 172.21.15.90:35538) 2024-04-25T11:38:00.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.818+0000 7f093bfff640 1 -- 172.21.15.90:0/1637536152 learned_addr learned my addr 172.21.15.90:0/1637536152 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:00.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.818+0000 7f0948a63740 1 -- 172.21.15.90:0/1637536152 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f093c1b1050 con 0x559d1a1c26b0 2024-04-25T11:38:00.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.818+0000 7f0948a63740 1 -- 172.21.15.90:0/1637536152 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f093c1b0980 con 0x559d1a1ba4d0 2024-04-25T11:38:00.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.818+0000 7f0948a63740 1 -- 172.21.15.90:0/1637536152 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f093c1adbc0 con 0x559d1a1a1f60 2024-04-25T11:38:00.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.819+0000 7f0948a63740 10 client.0 _ll_register_callbacks cb 0x7f093400af30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:00.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.819+0000 7f093affd640 1 --2- 172.21.15.90:0/1637536152 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d1a1ba4d0 0x559d1a592900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.819+0000 7f093bfff640 1 -- 172.21.15.90:0/1637536152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d1a1a1f60 msgr2=0x559d1a5923c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.819+0000 7f093bfff640 1 --2- 172.21.15.90:0/1637536152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d1a1a1f60 0x559d1a5923c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.819+0000 7f093bfff640 1 -- 172.21.15.90:0/1637536152 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d1a1ba4d0 msgr2=0x559d1a592900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.819+0000 7f093bfff640 1 --2- 172.21.15.90:0/1637536152 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d1a1ba4d0 0x559d1a592900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.819+0000 7f093bfff640 1 -- 172.21.15.90:0/1637536152 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d1a60c430 con 0x559d1a1c26b0 2024-04-25T11:38:00.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.819+0000 7f093affd640 1 --2- 172.21.15.90:0/1637536152 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d1a1ba4d0 0x559d1a592900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:38:00.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.820+0000 7f093bfff640 1 --2- 172.21.15.90:0/1637536152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1c26b0 0x559d1a60bef0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f0934007220 tx=0x7f0934007250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de9681615bb9bf7e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.820+0000 7f091affd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:38:00.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.820+0000 7f091affd640 1 -- 172.21.15.90:0/1637536152 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0934022040 con 0x559d1a1c26b0 2024-04-25T11:38:00.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.820+0000 7f091affd640 1 -- 172.21.15.90:0/1637536152 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f093401e660 con 0x559d1a1c26b0 2024-04-25T11:38:00.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.820+0000 7f0948a63740 1 -- 172.21.15.90:0/1637536152 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0930019070 con 0x559d1a1c26b0 2024-04-25T11:38:00.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.820+0000 7f091affd640 1 -- 172.21.15.90:0/1637536152 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0934040050 con 0x559d1a1c26b0 2024-04-25T11:38:00.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.820+0000 7f0948a63740 10 client.16482 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:00.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.820+0000 7f0948a63740 1 -- 172.21.15.90:0/1637536152 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559d1a60d610 con 0x559d1a1c26b0 2024-04-25T11:38:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.821+0000 7f091affd640 1 -- 172.21.15.90:0/1637536152 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0930019070 con 0x559d1a1c26b0 2024-04-25T11:38:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.821+0000 7f091affd640 1 -- 172.21.15.90:0/1637536152 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f09340326a0 con 0x559d1a1c26b0 2024-04-25T11:38:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.821+0000 7f0948a63740 10 client.16482 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.821+0000 7f0948a63740 10 client.16482 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:38:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.821+0000 7f0948a63740 20 client.16482 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.821+0000 7f0948a63740 1 --2- 172.21.15.90:0/1637536152 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x559d1a1e3390 0x559d1a60fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.821+0000 7f0948a63740 4 client.16482 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:38:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.821+0000 7f0948a63740 1 -- 172.21.15.90:0/1637536152 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559d1a4f1800 con 0x559d1a1e3390 2024-04-25T11:38:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.822+0000 7f093b7fe640 1 --2- 172.21.15.90:0/1637536152 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x559d1a1e3390 0x559d1a60fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.823+0000 7f091affd640 10 client.16482 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:38:00.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.823+0000 7f093b7fe640 1 --2- 172.21.15.90:0/1637536152 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x559d1a1e3390 0x559d1a60fe20 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=97f38bc88cf068dd server_cookie=a3407f9631f15801 in_seq=0 out_seq=0 2024-04-25T11:38:00.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.824+0000 7f091affd640 1 -- 172.21.15.90:0/1637536152 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559d1a4f1800 con 0x559d1a1e3390 2024-04-25T11:38:00.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.824+0000 7f091affd640 10 client.16482 handle_command_reply: tid=0 2024-04-25T11:38:00.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.824+0000 7f0948a63740 1 client.16482 shutdown 2024-04-25T11:38:00.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.824+0000 7f0948a63740 1 -- 172.21.15.90:0/1637536152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1c26b0 msgr2=0x559d1a60bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.824+0000 7f0948a63740 1 --2- 172.21.15.90:0/1637536152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1c26b0 0x559d1a60bef0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f0934007220 tx=0x7f0934007250 comp rx=0 tx=0).stop 2024-04-25T11:38:00.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.824+0000 7f0948a63740 1 -- 172.21.15.90:0/1637536152 shutdown_connections 2024-04-25T11:38:00.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.824+0000 7f0948a63740 1 --2- 172.21.15.90:0/1637536152 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x559d1a1e3390 0x559d1a60fe20 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.824+0000 7f0948a63740 1 --2- 172.21.15.90:0/1637536152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a1c26b0 0x559d1a60bef0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.824+0000 7f091affd640 0 client.16482 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:38:00.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.824+0000 7f0948a63740 1 --2- 172.21.15.90:0/1637536152 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d1a1ba4d0 0x559d1a592900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.824+0000 7f0948a63740 1 --2- 172.21.15.90:0/1637536152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d1a1a1f60 0x559d1a5923c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.824+0000 7f0948a63740 1 -- 172.21.15.90:0/1637536152 >> 172.21.15.90:0/1637536152 conn(0x559d1a1d74e0 msgr2=0x559d1a58d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.825+0000 7f0948a63740 1 -- 172.21.15.90:0/1637536152 shutdown_connections 2024-04-25T11:38:00.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.825+0000 7f0948a63740 1 -- 172.21.15.90:0/1637536152 wait complete. 2024-04-25T11:38:00.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.825+0000 7f0948a63740 20 client.16482 trim_cache size 0 max 16384 2024-04-25T11:38:00.814 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:38:00.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.826+0000 7f0941bb2640 1 -- 172.21.15.90:0/4044277163 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f09240afc60 msgr2=0x7f09240b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.826+0000 7f0941bb2640 1 --2- 172.21.15.90:0/4044277163 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f09240afc60 0x7f09240b2170 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x559d1a581980 tx=0x7f0934004b50 comp rx=0 tx=0).stop 2024-04-25T11:38:00.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.826+0000 7f0941bb2640 1 -- 172.21.15.90:0/4044277163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f093c102af0 msgr2=0x7f093c1a04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.826+0000 7f0941bb2640 1 --2- 172.21.15.90:0/4044277163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f093c102af0 0x7f093c1a04f0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f092c031a90 tx=0x7f092c004270 comp rx=0 tx=0).stop 2024-04-25T11:38:00.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.826+0000 7f0941bb2640 1 -- 172.21.15.90:0/4044277163 shutdown_connections 2024-04-25T11:38:00.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.826+0000 7f0941bb2640 1 --2- 172.21.15.90:0/4044277163 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f09240afc60 0x7f09240b2170 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.826+0000 7f0941bb2640 1 --2- 172.21.15.90:0/4044277163 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f093c10d380 0x7f093c19a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.826+0000 7f0941bb2640 1 --2- 172.21.15.90:0/4044277163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f093c108a20 0x7f093c1a0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.826+0000 7f0941bb2640 1 --2- 172.21.15.90:0/4044277163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f093c102af0 0x7f093c1a04f0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.826+0000 7f0941bb2640 1 -- 172.21.15.90:0/4044277163 >> 172.21.15.90:0/4044277163 conn(0x7f093c078100 msgr2=0x7f093c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.826+0000 7f0941bb2640 1 -- 172.21.15.90:0/4044277163 shutdown_connections 2024-04-25T11:38:00.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.826+0000 7f0941bb2640 1 -- 172.21.15.90:0/4044277163 wait complete. 2024-04-25T11:38:00.829 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T11:38:00.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:00 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2489337666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:38:00.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.972+0000 7fd02f2f9640 1 -- 172.21.15.90:0/3111896665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd02810d380 msgr2=0x7fd02810f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.972+0000 7fd02f2f9640 1 --2- 172.21.15.90:0/3111896665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd02810d380 0x7fd02810f7a0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fd020009ca0 tx=0x7fd02002a8c0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.972+0000 7fd02f2f9640 1 -- 172.21.15.90:0/3111896665 shutdown_connections 2024-04-25T11:38:00.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.972+0000 7fd02f2f9640 1 --2- 172.21.15.90:0/3111896665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd02810d380 0x7fd02810f7a0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.972+0000 7fd02f2f9640 1 --2- 172.21.15.90:0/3111896665 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd028102af0 0x7fd028102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.972+0000 7fd02f2f9640 1 --2- 172.21.15.90:0/3111896665 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd028108a20 0x7fd0281025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.972+0000 7fd02f2f9640 1 -- 172.21.15.90:0/3111896665 >> 172.21.15.90:0/3111896665 conn(0x7fd028078100 msgr2=0x7fd028078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.972+0000 7fd02f2f9640 1 -- 172.21.15.90:0/3111896665 shutdown_connections 2024-04-25T11:38:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.972+0000 7fd02f2f9640 1 -- 172.21.15.90:0/3111896665 wait complete. 2024-04-25T11:38:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.973+0000 7fd02f2f9640 1 Processor -- start 2024-04-25T11:38:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.973+0000 7fd02f2f9640 1 -- start start 2024-04-25T11:38:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.973+0000 7fd02f2f9640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd028102af0 0x7fd028199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.973+0000 7fd02f2f9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd028108a20 0x7fd028199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.973+0000 7fd02f2f9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd02810d380 0x7fd02819de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.973+0000 7fd02f2f9640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd028112c50 con 0x7fd028108a20 2024-04-25T11:38:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.973+0000 7fd02f2f9640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd028112ad0 con 0x7fd028102af0 2024-04-25T11:38:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.973+0000 7fd02f2f9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd028112dd0 con 0x7fd02810d380 2024-04-25T11:38:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.973+0000 7fd02c894640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd028108a20 0x7fd028199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.973+0000 7fd02c894640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd028108a20 0x7fd028199b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35552/0 (socket says 172.21.15.90:35552) 2024-04-25T11:38:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.973+0000 7fd02c894640 1 -- 172.21.15.90:0/2518038591 learned_addr learned my addr 172.21.15.90:0/2518038591 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.973+0000 7fd02d095640 1 --2- 172.21.15.90:0/2518038591 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd028102af0 0x7fd028199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.973+0000 7fd02d896640 1 --2- 172.21.15.90:0/2518038591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd02810d380 0x7fd02819de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.973+0000 7fd02c894640 1 -- 172.21.15.90:0/2518038591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd02810d380 msgr2=0x7fd02819de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.973+0000 7fd02c894640 1 --2- 172.21.15.90:0/2518038591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd02810d380 0x7fd02819de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.973+0000 7fd02c894640 1 -- 172.21.15.90:0/2518038591 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd028102af0 msgr2=0x7fd028199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.973+0000 7fd02c894640 1 --2- 172.21.15.90:0/2518038591 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd028102af0 0x7fd028199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.973+0000 7fd02c894640 1 -- 172.21.15.90:0/2518038591 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd02819e490 con 0x7fd028108a20 2024-04-25T11:38:00.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.973+0000 7fd02d896640 1 --2- 172.21.15.90:0/2518038591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd02810d380 0x7fd02819de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:38:00.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.974+0000 7fd02c894640 1 --2- 172.21.15.90:0/2518038591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd028108a20 0x7fd028199b90 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fd01800e9f0 tx=0x7fd01800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.974+0000 7fd01e7fc640 1 -- 172.21.15.90:0/2518038591 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd01800ccc0 con 0x7fd028108a20 2024-04-25T11:38:00.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.974+0000 7fd02f2f9640 1 -- 172.21.15.90:0/2518038591 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd02819e780 con 0x7fd028108a20 2024-04-25T11:38:00.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.974+0000 7fd01e7fc640 1 -- 172.21.15.90:0/2518038591 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0180035f0 con 0x7fd028108a20 2024-04-25T11:38:00.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.974+0000 7fd01e7fc640 1 -- 172.21.15.90:0/2518038591 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd01800ce60 con 0x7fd028108a20 2024-04-25T11:38:00.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.974+0000 7fd02f2f9640 1 -- 172.21.15.90:0/2518038591 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0281a6100 con 0x7fd028108a20 2024-04-25T11:38:00.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.977+0000 7fd0361ad740 1 Processor -- start 2024-04-25T11:38:00.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.977+0000 7fd0361ad740 1 -- start start 2024-04-25T11:38:00.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.977+0000 7fd01e7fc640 1 -- 172.21.15.90:0/2518038591 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fd018010690 con 0x7fd028108a20 2024-04-25T11:38:00.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.977+0000 7fd0361ad740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56451acef390 0x56451b090660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.977+0000 7fd0361ad740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56451acc64d0 0x56451b090ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.977+0000 7fd0361ad740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56451acce6b0 0x56451b09f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.977+0000 7fd0361ad740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56451aff5d80 con 0x56451acce6b0 2024-04-25T11:38:00.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.977+0000 7fd0361ad740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56451afafa20 con 0x56451acc64d0 2024-04-25T11:38:00.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.977+0000 7fd0361ad740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56451aff1d40 con 0x56451acef390 2024-04-25T11:38:00.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.977+0000 7fd02d095640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56451acc64d0 0x56451b090ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.977+0000 7fd02c894640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56451acce6b0 0x56451b09f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.977+0000 7fd02d095640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56451acc64d0 0x56451b090ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:53126/0 (socket says 172.21.15.90:53126) 2024-04-25T11:38:00.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.977+0000 7fd02d095640 1 -- 172.21.15.90:0/1251335346 learned_addr learned my addr 172.21.15.90:0/1251335346 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:00.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.977+0000 7fd02d896640 1 --2- 172.21.15.90:0/1251335346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56451acef390 0x56451b090660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.977+0000 7fd02d095640 1 -- 172.21.15.90:0/1251335346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56451acef390 msgr2=0x56451b090660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.977+0000 7fd02d095640 1 --2- 172.21.15.90:0/1251335346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56451acef390 0x56451b090660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.977+0000 7fd02d095640 1 -- 172.21.15.90:0/1251335346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56451acce6b0 msgr2=0x56451b09f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.977+0000 7fd02d095640 1 --2- 172.21.15.90:0/1251335346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56451acce6b0 0x56451b09f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.977+0000 7fd02d095640 1 -- 172.21.15.90:0/1251335346 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56451afbac30 con 0x56451acc64d0 2024-04-25T11:38:00.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.978+0000 7fd02d095640 1 --2- 172.21.15.90:0/1251335346 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56451acc64d0 0x56451b090ba0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd02400b690 tx=0x7fd02400bb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d065de985e5dc7d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.978+0000 7fd0137fe640 1 -- 172.21.15.90:0/1251335346 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd024003bd0 con 0x56451acc64d0 2024-04-25T11:38:00.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.978+0000 7fd01e7fc640 1 --2- 172.21.15.90:0/2518038591 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd0140afc60 0x7fd0140b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.978+0000 7fd0137fe640 1 -- 172.21.15.90:0/1251335346 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd024015070 con 0x56451acc64d0 2024-04-25T11:38:00.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.978+0000 7fd0137fe640 1 -- 172.21.15.90:0/1251335346 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd024004d40 con 0x56451acc64d0 2024-04-25T11:38:00.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.978+0000 7fd02d896640 1 --2- 172.21.15.90:0/2518038591 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd0140afc60 0x7fd0140b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.978+0000 7fd01e7fc640 1 -- 172.21.15.90:0/2518038591 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fd0180df440 con 0x7fd028108a20 2024-04-25T11:38:00.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.979+0000 7fd0361ad740 1 -- 172.21.15.90:0/1251335346 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56451acc64d0 msgr2=0x56451b090ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.979+0000 7fd02d896640 1 --2- 172.21.15.90:0/2518038591 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd0140afc60 0x7fd0140b2170 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd020006010 tx=0x7fd02002f3a0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.979+0000 7fd0361ad740 1 --2- 172.21.15.90:0/1251335346 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56451acc64d0 0x56451b090ba0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd02400b690 tx=0x7fd02400bb50 comp rx=0 tx=0).stop 2024-04-25T11:38:00.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.979+0000 7fd0361ad740 1 -- 172.21.15.90:0/1251335346 shutdown_connections 2024-04-25T11:38:00.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.979+0000 7fd0361ad740 1 --2- 172.21.15.90:0/1251335346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56451acce6b0 0x56451b09f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.979+0000 7fd0361ad740 1 --2- 172.21.15.90:0/1251335346 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56451acc64d0 0x56451b090ba0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.979+0000 7fd0361ad740 1 --2- 172.21.15.90:0/1251335346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56451acef390 0x56451b090660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.979+0000 7fd0361ad740 1 -- 172.21.15.90:0/1251335346 >> 172.21.15.90:0/1251335346 conn(0x56451ace34e0 msgr2=0x56451b08cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.979+0000 7fd0361ad740 1 -- 172.21.15.90:0/1251335346 shutdown_connections 2024-04-25T11:38:00.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.979+0000 7fd0361ad740 1 -- 172.21.15.90:0/1251335346 wait complete. 2024-04-25T11:38:00.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.980+0000 7fd0361ad740 1 -- start start 2024-04-25T11:38:00.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.980+0000 7fd0361ad740 1 Processor -- start 2024-04-25T11:38:00.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.980+0000 7fd0361ad740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56451acc64d0 0x56451b09d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.980+0000 7fd0361ad740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56451acce6b0 0x56451b09d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.980+0000 7fd0361ad740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56451acef390 0x56451b0a9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.980+0000 7fd0361ad740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56451b09ffa0 con 0x56451acc64d0 2024-04-25T11:38:00.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.980+0000 7fd0361ad740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56451af7cdc0 con 0x56451acce6b0 2024-04-25T11:38:00.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.980+0000 7fd0361ad740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56451b0a0120 con 0x56451acef390 2024-04-25T11:38:00.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.980+0000 7fd02d095640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56451acc64d0 0x56451b09d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.980+0000 7fd02d896640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56451acef390 0x56451b0a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.981+0000 7fd02d095640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56451acc64d0 0x56451b09d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35566/0 (socket says 172.21.15.90:35566) 2024-04-25T11:38:00.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.981+0000 7fd02c894640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56451acce6b0 0x56451b09d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.981+0000 7fd02d095640 1 -- 172.21.15.90:0/1668922804 learned_addr learned my addr 172.21.15.90:0/1668922804 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:00.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.981+0000 7fd02d896640 1 --2- 172.21.15.90:0/1668922804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56451acef390 0x56451b0a9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:59242/0 (socket says 172.21.15.90:59242) 2024-04-25T11:38:00.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.981+0000 7fd0361ad740 10 client.0 _ll_register_callbacks cb 0x56451af98600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:00.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.981+0000 7fd02d095640 1 -- 172.21.15.90:0/1668922804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56451acef390 msgr2=0x56451b0a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.981+0000 7fd02d095640 1 --2- 172.21.15.90:0/1668922804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56451acef390 0x56451b0a9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.981+0000 7fd02d095640 1 -- 172.21.15.90:0/1668922804 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56451acce6b0 msgr2=0x56451b09d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.981+0000 7fd02d095640 1 --2- 172.21.15.90:0/1668922804 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56451acce6b0 0x56451b09d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.981+0000 7fd02d095640 1 -- 172.21.15.90:0/1668922804 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56451ad073f0 con 0x56451acc64d0 2024-04-25T11:38:00.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.981+0000 7fd02c894640 1 --2- 172.21.15.90:0/1668922804 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56451acce6b0 0x56451b09d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:00.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.981+0000 7fd02d095640 1 --2- 172.21.15.90:0/1668922804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56451acc64d0 0x56451b09d160 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fd024004230 tx=0x7fd02400bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e7a7bb71cb71adf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.981+0000 7fd010ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:38:00.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.981+0000 7fd010ff9640 1 -- 172.21.15.90:0/1668922804 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd024028060 con 0x56451acc64d0 2024-04-25T11:38:00.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.981+0000 7fd010ff9640 1 -- 172.21.15.90:0/1668922804 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fd024011640 con 0x56451acc64d0 2024-04-25T11:38:00.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.981+0000 7fd0361ad740 1 -- 172.21.15.90:0/1668922804 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd024015070 con 0x56451acc64d0 2024-04-25T11:38:00.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.981+0000 7fd010ff9640 1 -- 172.21.15.90:0/1668922804 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd024011840 con 0x56451acc64d0 2024-04-25T11:38:00.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.981+0000 7fd0361ad740 10 client.16494 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:00.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.981+0000 7fd0361ad740 1 -- 172.21.15.90:0/1668922804 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56451add17d0 con 0x56451acc64d0 2024-04-25T11:38:00.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.982+0000 7fd010ff9640 1 -- 172.21.15.90:0/1668922804 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd024015070 con 0x56451acc64d0 2024-04-25T11:38:00.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.982+0000 7fd010ff9640 1 -- 172.21.15.90:0/1668922804 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7fd024022070 con 0x56451acc64d0 2024-04-25T11:38:00.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.982+0000 7fd0361ad740 10 client.16494 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:00.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.982+0000 7fd0361ad740 10 client.16494 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:38:00.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.982+0000 7fd0361ad740 20 client.16494 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:00.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.982+0000 7fd0361ad740 1 --2- 172.21.15.90:0/1668922804 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56451acadf60 0x56451b0af620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.982+0000 7fd0361ad740 4 client.16494 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:38:00.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.982+0000 7fd0361ad740 1 -- 172.21.15.90:0/1668922804 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56451b003990 con 0x56451acadf60 2024-04-25T11:38:00.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.982+0000 7fd02c894640 1 --2- 172.21.15.90:0/1668922804 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56451acadf60 0x56451b0af620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.982+0000 7fd010ff9640 10 client.16494 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:00.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.982+0000 7fd02c894640 1 --2- 172.21.15.90:0/1668922804 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56451acadf60 0x56451b0af620 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcbe7268039b18db server_cookie=f593a7ff8baa1c64 in_seq=0 out_seq=0 2024-04-25T11:38:00.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.983+0000 7fd010ff9640 1 -- 172.21.15.90:0/1668922804 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56451b003990 con 0x56451acadf60 2024-04-25T11:38:00.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.983+0000 7fd010ff9640 10 client.16494 handle_command_reply: tid=0 2024-04-25T11:38:00.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.983+0000 7fd0361ad740 1 client.16494 shutdown 2024-04-25T11:38:00.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.983+0000 7fd0361ad740 1 -- 172.21.15.90:0/1668922804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56451acc64d0 msgr2=0x56451b09d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.983+0000 7fd0361ad740 1 --2- 172.21.15.90:0/1668922804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56451acc64d0 0x56451b09d160 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fd024004230 tx=0x7fd02400bfd0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.983+0000 7fd0361ad740 1 -- 172.21.15.90:0/1668922804 shutdown_connections 2024-04-25T11:38:00.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.983+0000 7fd0361ad740 1 --2- 172.21.15.90:0/1668922804 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56451acadf60 0x56451b0af620 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.983+0000 7fd0361ad740 1 --2- 172.21.15.90:0/1668922804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56451acef390 0x56451b0a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.983+0000 7fd0361ad740 1 --2- 172.21.15.90:0/1668922804 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56451acce6b0 0x56451b09d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.983+0000 7fd010ff9640 0 client.16494 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:00.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.983+0000 7fd0361ad740 1 --2- 172.21.15.90:0/1668922804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56451acc64d0 0x56451b09d160 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.983+0000 7fd0361ad740 1 -- 172.21.15.90:0/1668922804 >> 172.21.15.90:0/1668922804 conn(0x56451ace34e0 msgr2=0x56451b0924c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.984+0000 7fd0361ad740 1 -- 172.21.15.90:0/1668922804 shutdown_connections 2024-04-25T11:38:00.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.984+0000 7fd0361ad740 1 -- 172.21.15.90:0/1668922804 wait complete. 2024-04-25T11:38:00.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:00.984+0000 7fd0361ad740 20 client.16494 trim_cache size 0 max 16384 2024-04-25T11:38:00.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.002+0000 7fd0361ad740 1 Processor -- start 2024-04-25T11:38:00.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.002+0000 7fd0361ad740 1 -- start start 2024-04-25T11:38:00.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.002+0000 7fd0361ad740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56451acc64d0 0x56451b09f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.002+0000 7fd0361ad740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56451acce6b0 0x56451b0a93c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.002+0000 7fd0361ad740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56451acef390 0x56451b08d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.002+0000 7fd0361ad740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd0281ad9f0 con 0x56451acce6b0 2024-04-25T11:38:00.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.002+0000 7fd0361ad740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd0281af560 con 0x56451acef390 2024-04-25T11:38:00.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.002+0000 7fd0361ad740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd0281ae0c0 con 0x56451acc64d0 2024-04-25T11:38:00.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.002+0000 7fd02c894640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56451acce6b0 0x56451b0a93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.002+0000 7fd02d896640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56451acef390 0x56451b08d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.002+0000 7fd02c894640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56451acce6b0 0x56451b0a93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35570/0 (socket says 172.21.15.90:35570) 2024-04-25T11:38:00.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.002+0000 7fd02c894640 1 -- 172.21.15.90:0/2172996831 learned_addr learned my addr 172.21.15.90:0/2172996831 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:00.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.002+0000 7fd02d095640 1 --2- 172.21.15.90:0/2172996831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56451acc64d0 0x56451b09f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.002+0000 7fd02c894640 1 -- 172.21.15.90:0/2172996831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56451acc64d0 msgr2=0x56451b09f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.002+0000 7fd02c894640 1 --2- 172.21.15.90:0/2172996831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56451acc64d0 0x56451b09f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.002+0000 7fd02c894640 1 -- 172.21.15.90:0/2172996831 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56451acef390 msgr2=0x56451b08d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.002+0000 7fd02c894640 1 --2- 172.21.15.90:0/2172996831 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56451acef390 0x56451b08d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.002+0000 7fd02c894640 1 -- 172.21.15.90:0/2172996831 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56451b09fe80 con 0x56451acce6b0 2024-04-25T11:38:00.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.002+0000 7fd02d896640 1 --2- 172.21.15.90:0/2172996831 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56451acef390 0x56451b08d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:00.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.002+0000 7fd02c894640 1 --2- 172.21.15.90:0/2172996831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56451acce6b0 0x56451b0a93c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fd0180a47f0 tx=0x7fd018090040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9b903771c4496dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.003+0000 7fd012ffd640 1 -- 172.21.15.90:0/2172996831 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0180a6ed0 con 0x56451acce6b0 2024-04-25T11:38:00.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.003+0000 7fd012ffd640 1 -- 172.21.15.90:0/2172996831 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0180a7090 con 0x56451acce6b0 2024-04-25T11:38:00.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.003+0000 7fd012ffd640 1 -- 172.21.15.90:0/2172996831 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0180a6cf0 con 0x56451acce6b0 2024-04-25T11:38:00.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.004+0000 7fd0361ad740 1 -- 172.21.15.90:0/2172996831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56451acce6b0 msgr2=0x56451b0a93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:00.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.004+0000 7fd0361ad740 1 --2- 172.21.15.90:0/2172996831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56451acce6b0 0x56451b0a93c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fd0180a47f0 tx=0x7fd018090040 comp rx=0 tx=0).stop 2024-04-25T11:38:00.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.004+0000 7fd0361ad740 1 -- 172.21.15.90:0/2172996831 shutdown_connections 2024-04-25T11:38:00.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.004+0000 7fd0361ad740 1 --2- 172.21.15.90:0/2172996831 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56451acef390 0x56451b08d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.004+0000 7fd0361ad740 1 --2- 172.21.15.90:0/2172996831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56451acce6b0 0x56451b0a93c0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.004+0000 7fd0361ad740 1 --2- 172.21.15.90:0/2172996831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56451acc64d0 0x56451b09f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.004+0000 7fd0361ad740 1 -- 172.21.15.90:0/2172996831 >> 172.21.15.90:0/2172996831 conn(0x56451ace34e0 msgr2=0x56451b08c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:00.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.004+0000 7fd0361ad740 1 -- 172.21.15.90:0/2172996831 shutdown_connections 2024-04-25T11:38:00.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.004+0000 7fd0361ad740 1 -- 172.21.15.90:0/2172996831 wait complete. 2024-04-25T11:38:00.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.004+0000 7fd0361ad740 1 -- start start 2024-04-25T11:38:00.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.004+0000 7fd0361ad740 1 Processor -- start 2024-04-25T11:38:00.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.005+0000 7fd0361ad740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56451acc64d0 0x56451b0b2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.005+0000 7fd0361ad740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56451acce6b0 0x56451b0b29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.005+0000 7fd0361ad740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56451acef390 0x56451b0af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.005+0000 7fd0361ad740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd0281aee60 con 0x56451acef390 2024-04-25T11:38:00.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.005+0000 7fd0361ad740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd0281ae790 con 0x56451acce6b0 2024-04-25T11:38:00.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.005+0000 7fd0361ad740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd0281ab9d0 con 0x56451acc64d0 2024-04-25T11:38:00.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.005+0000 7fd0361ad740 10 client.0 _ll_register_callbacks cb 0x7fd0180e2e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:00.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.005+0000 7fd02d896640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56451acef390 0x56451b0af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.005+0000 7fd02d095640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56451acc64d0 0x56451b0b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.005+0000 7fd02d896640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56451acef390 0x56451b0af970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35582/0 (socket says 172.21.15.90:35582) 2024-04-25T11:38:00.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.005+0000 7fd02d896640 1 -- 172.21.15.90:0/129896874 learned_addr learned my addr 172.21.15.90:0/129896874 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:00.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.005+0000 7fd02c894640 1 --2- 172.21.15.90:0/129896874 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56451acce6b0 0x56451b0b29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:00.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.005+0000 7fd02d095640 1 -- 172.21.15.90:0/129896874 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56451acce6b0 msgr2=0x56451b0b29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.005+0000 7fd02d095640 1 --2- 172.21.15.90:0/129896874 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56451acce6b0 0x56451b0b29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.005+0000 7fd02d095640 1 -- 172.21.15.90:0/129896874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56451acef390 msgr2=0x56451b0af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:00.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.005+0000 7fd02d095640 1 --2- 172.21.15.90:0/129896874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56451acef390 0x56451b0af970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:00.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.005+0000 7fd02d095640 1 -- 172.21.15.90:0/129896874 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56451b08e110 con 0x56451acc64d0 2024-04-25T11:38:00.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.005+0000 7fd02c894640 1 --2- 172.21.15.90:0/129896874 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56451acce6b0 0x56451b0b29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:00.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.005+0000 7fd02d896640 1 --2- 172.21.15.90:0/129896874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56451acef390 0x56451b0af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:00.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.005+0000 7fd02d095640 1 --2- 172.21.15.90:0/129896874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56451acc64d0 0x56451b0b2470 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd02400b690 tx=0x7fd024004260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8705ae6d6fe8ca32 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:00.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.005+0000 7fd010ff9640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:38:00.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.005+0000 7fd010ff9640 1 -- 172.21.15.90:0/129896874 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd024028070 con 0x56451acc64d0 2024-04-25T11:38:00.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.005+0000 7fd010ff9640 1 -- 172.21.15.90:0/129896874 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fd0240046b0 con 0x56451acc64d0 2024-04-25T11:38:00.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.005+0000 7fd0361ad740 1 -- 172.21.15.90:0/129896874 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0180a7090 con 0x56451acc64d0 2024-04-25T11:38:00.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.006+0000 7fd010ff9640 1 -- 172.21.15.90:0/129896874 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd024010440 con 0x56451acc64d0 2024-04-25T11:38:00.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.006+0000 7fd0361ad740 10 client.25723 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:00.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.006+0000 7fd0361ad740 1 -- 172.21.15.90:0/129896874 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56451b116ea0 con 0x56451acc64d0 2024-04-25T11:38:00.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.006+0000 7fd010ff9640 1 -- 172.21.15.90:0/129896874 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0180a7090 con 0x56451acc64d0 2024-04-25T11:38:00.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.006+0000 7fd010ff9640 1 -- 172.21.15.90:0/129896874 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7fd024022040 con 0x56451acc64d0 2024-04-25T11:38:00.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.006+0000 7fd0361ad740 10 client.25723 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:00.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.006+0000 7fd0361ad740 10 client.25723 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:38:00.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.006+0000 7fd0361ad740 20 client.25723 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:00.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.006+0000 7fd0361ad740 1 --2- 172.21.15.90:0/129896874 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56451ac833a0 0x56451b1195d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:00.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.006+0000 7fd0361ad740 4 client.25723 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:38:00.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.006+0000 7fd0361ad740 1 -- 172.21.15.90:0/129896874 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56451affd800 con 0x56451ac833a0 2024-04-25T11:38:01.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.006+0000 7fd02c894640 1 --2- 172.21.15.90:0/129896874 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56451ac833a0 0x56451b1195d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:01.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.007+0000 7fd010ff9640 10 client.25723 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:01.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.007+0000 7fd02c894640 1 --2- 172.21.15.90:0/129896874 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56451ac833a0 0x56451b1195d0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=819dfd69505c177 server_cookie=142eeafa57ce7244 in_seq=0 out_seq=0 2024-04-25T11:38:01.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.007+0000 7fd010ff9640 1 -- 172.21.15.90:0/129896874 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56451affd800 con 0x56451ac833a0 2024-04-25T11:38:01.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.007+0000 7fd010ff9640 10 client.25723 handle_command_reply: tid=0 2024-04-25T11:38:01.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.007+0000 7fd0361ad740 1 client.25723 shutdown 2024-04-25T11:38:01.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.007+0000 7fd0361ad740 1 -- 172.21.15.90:0/129896874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56451acc64d0 msgr2=0x56451b0b2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:01.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.007+0000 7fd0361ad740 1 --2- 172.21.15.90:0/129896874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56451acc64d0 0x56451b0b2470 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd02400b690 tx=0x7fd024004260 comp rx=0 tx=0).stop 2024-04-25T11:38:01.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.007+0000 7fd0361ad740 1 -- 172.21.15.90:0/129896874 shutdown_connections 2024-04-25T11:38:01.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.007+0000 7fd0361ad740 1 --2- 172.21.15.90:0/129896874 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56451ac833a0 0x56451b1195d0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:01.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.007+0000 7fd0361ad740 1 --2- 172.21.15.90:0/129896874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56451acef390 0x56451b0af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:01.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.007+0000 7fd010ff9640 0 client.25723 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:01.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.007+0000 7fd0361ad740 1 --2- 172.21.15.90:0/129896874 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56451acce6b0 0x56451b0b29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:01.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.007+0000 7fd0361ad740 1 --2- 172.21.15.90:0/129896874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56451acc64d0 0x56451b0b2470 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:01.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.007+0000 7fd0361ad740 1 -- 172.21.15.90:0/129896874 >> 172.21.15.90:0/129896874 conn(0x56451ace34e0 msgr2=0x56451b08b610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:01.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.007+0000 7fd0361ad740 1 -- 172.21.15.90:0/129896874 shutdown_connections 2024-04-25T11:38:01.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.007+0000 7fd0361ad740 1 -- 172.21.15.90:0/129896874 wait complete. 2024-04-25T11:38:01.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.007+0000 7fd0361ad740 20 client.25723 trim_cache size 0 max 16384 2024-04-25T11:38:01.001 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:38:01.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.008+0000 7fd02f2f9640 1 -- 172.21.15.90:0/2518038591 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd0140afc60 msgr2=0x7fd0140b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:01.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.008+0000 7fd02f2f9640 1 --2- 172.21.15.90:0/2518038591 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd0140afc60 0x7fd0140b2170 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd020006010 tx=0x7fd02002f3a0 comp rx=0 tx=0).stop 2024-04-25T11:38:01.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.008+0000 7fd02f2f9640 1 -- 172.21.15.90:0/2518038591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd028108a20 msgr2=0x7fd028199b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:01.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.008+0000 7fd02f2f9640 1 --2- 172.21.15.90:0/2518038591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd028108a20 0x7fd028199b90 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fd01800e9f0 tx=0x7fd01800eeb0 comp rx=0 tx=0).stop 2024-04-25T11:38:01.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.008+0000 7fd02f2f9640 1 -- 172.21.15.90:0/2518038591 shutdown_connections 2024-04-25T11:38:01.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.008+0000 7fd02f2f9640 1 --2- 172.21.15.90:0/2518038591 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd0140afc60 0x7fd0140b2170 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:01.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.008+0000 7fd02f2f9640 1 --2- 172.21.15.90:0/2518038591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd02810d380 0x7fd02819de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:01.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.008+0000 7fd02f2f9640 1 --2- 172.21.15.90:0/2518038591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd028108a20 0x7fd028199b90 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:01.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.008+0000 7fd02f2f9640 1 --2- 172.21.15.90:0/2518038591 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd028102af0 0x7fd028199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:01.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.008+0000 7fd02f2f9640 1 -- 172.21.15.90:0/2518038591 >> 172.21.15.90:0/2518038591 conn(0x7fd028078100 msgr2=0x7fd0281040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:01.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.008+0000 7fd02f2f9640 1 -- 172.21.15.90:0/2518038591 shutdown_connections 2024-04-25T11:38:01.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:01.008+0000 7fd02f2f9640 1 -- 172.21.15.90:0/2518038591 wait complete. 2024-04-25T11:38:01.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:01 smithi090 ceph-mon[29575]: pgmap v91: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 594 MiB/s wr, 817 op/s 2024-04-25T11:38:01.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:01 smithi138 ceph-mon[32328]: pgmap v91: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 594 MiB/s wr, 817 op/s 2024-04-25T11:38:01.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:01 smithi156 ceph-mon[33045]: pgmap v91: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 594 MiB/s wr, 817 op/s 2024-04-25T11:38:02.010 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T11:38:02.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:38:02.010 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T11:38:02.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.210+0000 7f8377f00640 1 -- 172.21.15.90:0/2080140540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8370110200 msgr2=0x7f837010b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:02.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.210+0000 7f8377f00640 1 --2- 172.21.15.90:0/2080140540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8370110200 0x7f837010b990 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f836c009900 tx=0x7f836c02a410 comp rx=0 tx=0).stop 2024-04-25T11:38:02.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.210+0000 7f8377f00640 1 -- 172.21.15.90:0/2080140540 shutdown_connections 2024-04-25T11:38:02.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.210+0000 7f8377f00640 1 --2- 172.21.15.90:0/2080140540 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8370115aa0 0x7f8370117ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.210+0000 7f8377f00640 1 --2- 172.21.15.90:0/2080140540 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f837010bed0 0x7f837010c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.210+0000 7f8377f00640 1 --2- 172.21.15.90:0/2080140540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8370110200 0x7f837010b990 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.210+0000 7f8377f00640 1 -- 172.21.15.90:0/2080140540 >> 172.21.15.90:0/2080140540 conn(0x7f837006e7a0 msgr2=0x7f837006ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:02.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.210+0000 7f8377f00640 1 -- 172.21.15.90:0/2080140540 shutdown_connections 2024-04-25T11:38:02.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.210+0000 7f8377f00640 1 -- 172.21.15.90:0/2080140540 wait complete. 2024-04-25T11:38:02.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.211+0000 7f8377f00640 1 Processor -- start 2024-04-25T11:38:02.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.211+0000 7f8377f00640 1 -- start start 2024-04-25T11:38:02.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.211+0000 7f8377f00640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f837010bed0 0x7f83701a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.211+0000 7f8377f00640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8370110200 0x7f83701a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.211+0000 7f8377f00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8370115aa0 0x7f83701a99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.212+0000 7f8377f00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f837011e750 con 0x7f8370115aa0 2024-04-25T11:38:02.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.212+0000 7f8377f00640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f837011e5d0 con 0x7f837010bed0 2024-04-25T11:38:02.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.212+0000 7f8377f00640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f837011e8d0 con 0x7f8370110200 2024-04-25T11:38:02.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.212+0000 7f837649d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8370115aa0 0x7f83701a99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.212+0000 7f837649d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8370115aa0 0x7f83701a99f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41912/0 (socket says 172.21.15.90:41912) 2024-04-25T11:38:02.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.212+0000 7f837649d640 1 -- 172.21.15.90:0/3063256057 learned_addr learned my addr 172.21.15.90:0/3063256057 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:02.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.212+0000 7f837549b640 1 --2- 172.21.15.90:0/3063256057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8370110200 0x7f83701a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.212+0000 7f837649d640 1 -- 172.21.15.90:0/3063256057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8370110200 msgr2=0x7f83701a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:02.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.212+0000 7f837649d640 1 --2- 172.21.15.90:0/3063256057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8370110200 0x7f83701a5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.212+0000 7f837649d640 1 -- 172.21.15.90:0/3063256057 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f837010bed0 msgr2=0x7f83701a51d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:38:02.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.212+0000 7f837649d640 1 --2- 172.21.15.90:0/3063256057 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f837010bed0 0x7f83701a51d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.212+0000 7f837649d640 1 -- 172.21.15.90:0/3063256057 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83701aa030 con 0x7f8370115aa0 2024-04-25T11:38:02.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.212+0000 7f837549b640 1 --2- 172.21.15.90:0/3063256057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8370110200 0x7f83701a5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:38:02.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.212+0000 7f837649d640 1 --2- 172.21.15.90:0/3063256057 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8370115aa0 0x7f83701a99f0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f836800cad0 tx=0x7f836800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:02.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.212+0000 7f8366ffd640 1 -- 172.21.15.90:0/3063256057 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f836800ae20 con 0x7f8370115aa0 2024-04-25T11:38:02.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.213+0000 7f8366ffd640 1 -- 172.21.15.90:0/3063256057 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8368005020 con 0x7f8370115aa0 2024-04-25T11:38:02.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.213+0000 7f8377f00640 1 -- 172.21.15.90:0/3063256057 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83701aa320 con 0x7f8370115aa0 2024-04-25T11:38:02.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.213+0000 7f8366ffd640 1 -- 172.21.15.90:0/3063256057 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f836800f6b0 con 0x7f8370115aa0 2024-04-25T11:38:02.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.213+0000 7f8377f00640 1 -- 172.21.15.90:0/3063256057 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83701b1ca0 con 0x7f8370115aa0 2024-04-25T11:38:02.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.217+0000 7f837edb3740 1 Processor -- start 2024-04-25T11:38:02.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.217+0000 7f8366ffd640 1 -- 172.21.15.90:0/3063256057 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f836800f850 con 0x7f8370115aa0 2024-04-25T11:38:02.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.218+0000 7f837edb3740 1 -- start start 2024-04-25T11:38:02.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.218+0000 7f837edb3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640570cf410 0x56405746f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.218+0000 7f837edb3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5640570a6550 0x56405746fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.218+0000 7f837edb3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570ae730 0x5640574700a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.218+0000 7f837edb3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5640573d5b00 con 0x5640570ae730 2024-04-25T11:38:02.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.218+0000 7f837edb3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x564057371920 con 0x5640570a6550 2024-04-25T11:38:02.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.218+0000 7f837edb3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5640573d1ac0 con 0x5640570cf410 2024-04-25T11:38:02.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.219+0000 7f837549b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570ae730 0x5640574700a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.219+0000 7f837549b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570ae730 0x5640574700a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41924/0 (socket says 172.21.15.90:41924) 2024-04-25T11:38:02.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.219+0000 7f837549b640 1 -- 172.21.15.90:0/258684188 learned_addr learned my addr 172.21.15.90:0/258684188 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:02.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.219+0000 7f8366ffd640 1 --2- 172.21.15.90:0/3063256057 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f835c0afc40 0x7f835c0b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.219+0000 7f8366ffd640 1 -- 172.21.15.90:0/3063256057 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f83680df180 con 0x7f8370115aa0 2024-04-25T11:38:02.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.219+0000 7f8375c9c640 1 --2- 172.21.15.90:0/258684188 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5640570a6550 0x56405746fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.219+0000 7f837649d640 1 --2- 172.21.15.90:0/3063256057 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f835c0afc40 0x7f835c0b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.220+0000 7f837549b640 1 --2- 172.21.15.90:0/258684188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640570cf410 0x56405746f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.220+0000 7f8375c9c640 1 -- 172.21.15.90:0/258684188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640570cf410 msgr2=0x56405746f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:02.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.220+0000 7f8375c9c640 1 --2- 172.21.15.90:0/258684188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640570cf410 0x56405746f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.220+0000 7f8375c9c640 1 -- 172.21.15.90:0/258684188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570ae730 msgr2=0x5640574700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:02.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.220+0000 7f8375c9c640 1 --2- 172.21.15.90:0/258684188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570ae730 0x5640574700a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.220+0000 7f8375c9c640 1 -- 172.21.15.90:0/258684188 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640571af7e0 con 0x5640570a6550 2024-04-25T11:38:02.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.220+0000 7f837549b640 1 --2- 172.21.15.90:0/258684188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570ae730 0x5640574700a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:02.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.220+0000 7f8375c9c640 1 --2- 172.21.15.90:0/258684188 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5640570a6550 0x56405746fb60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f836c02a930 tx=0x7f836c032b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e8acfa01c5fe959 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:02.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.221+0000 7f837649d640 1 --2- 172.21.15.90:0/3063256057 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f835c0afc40 0x7f835c0b2150 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f836800a920 tx=0x7f836800a890 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:02.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.221+0000 7f835bfff640 1 -- 172.21.15.90:0/258684188 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f836c035040 con 0x5640570a6550 2024-04-25T11:38:02.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.221+0000 7f835bfff640 1 -- 172.21.15.90:0/258684188 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f836c004040 con 0x5640570a6550 2024-04-25T11:38:02.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.221+0000 7f835bfff640 1 -- 172.21.15.90:0/258684188 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f836c003aa0 con 0x5640570a6550 2024-04-25T11:38:02.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.223+0000 7f837edb3740 1 -- 172.21.15.90:0/258684188 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5640570a6550 msgr2=0x56405746fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:02.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.223+0000 7f837edb3740 1 --2- 172.21.15.90:0/258684188 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5640570a6550 0x56405746fb60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f836c02a930 tx=0x7f836c032b20 comp rx=0 tx=0).stop 2024-04-25T11:38:02.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.223+0000 7f837edb3740 1 -- 172.21.15.90:0/258684188 shutdown_connections 2024-04-25T11:38:02.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.223+0000 7f837edb3740 1 --2- 172.21.15.90:0/258684188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570ae730 0x5640574700a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.224+0000 7f837edb3740 1 --2- 172.21.15.90:0/258684188 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5640570a6550 0x56405746fb60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.224+0000 7f837edb3740 1 --2- 172.21.15.90:0/258684188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640570cf410 0x56405746f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.224+0000 7f837edb3740 1 -- 172.21.15.90:0/258684188 >> 172.21.15.90:0/258684188 conn(0x5640570c3560 msgr2=0x56405746bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:02.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.224+0000 7f837edb3740 1 -- 172.21.15.90:0/258684188 shutdown_connections 2024-04-25T11:38:02.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.224+0000 7f837edb3740 1 -- 172.21.15.90:0/258684188 wait complete. 2024-04-25T11:38:02.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.225+0000 7f837edb3740 1 -- start start 2024-04-25T11:38:02.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.225+0000 7f837edb3740 1 Processor -- start 2024-04-25T11:38:02.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.226+0000 7f837edb3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570a6550 0x564057479ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.226+0000 7f837edb3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5640570ae730 0x56405747a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.226+0000 7f837edb3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640570cf410 0x56405747fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.226+0000 7f837edb3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56405747c910 con 0x5640570a6550 2024-04-25T11:38:02.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.226+0000 7f837edb3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56405736f5d0 con 0x5640570ae730 2024-04-25T11:38:02.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.226+0000 7f837edb3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56405747ca90 con 0x5640570cf410 2024-04-25T11:38:02.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.226+0000 7f837edb3740 10 client.0 _ll_register_callbacks cb 0x56405735d080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:02.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.226+0000 7f837549b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570a6550 0x564057479ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.226+0000 7f837549b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570a6550 0x564057479ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41926/0 (socket says 172.21.15.90:41926) 2024-04-25T11:38:02.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.226+0000 7f837549b640 1 -- 172.21.15.90:0/853906856 learned_addr learned my addr 172.21.15.90:0/853906856 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:02.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.226+0000 7f8375c9c640 1 --2- 172.21.15.90:0/853906856 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5640570ae730 0x56405747a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.227+0000 7f837549b640 1 -- 172.21.15.90:0/853906856 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640570cf410 msgr2=0x56405747fe00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:38:02.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.227+0000 7f837549b640 1 --2- 172.21.15.90:0/853906856 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640570cf410 0x56405747fe00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.227+0000 7f837549b640 1 -- 172.21.15.90:0/853906856 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5640570ae730 msgr2=0x56405747a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:02.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.227+0000 7f837549b640 1 --2- 172.21.15.90:0/853906856 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5640570ae730 0x56405747a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.227+0000 7f837549b640 1 -- 172.21.15.90:0/853906856 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56405720ff70 con 0x5640570a6550 2024-04-25T11:38:02.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.227+0000 7f837549b640 1 --2- 172.21.15.90:0/853906856 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570a6550 0x564057479ce0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f8360013590 tx=0x7f8360013a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f75ed500086c61e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:02.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.227+0000 7f83597fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:38:02.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.227+0000 7f83597fa640 1 -- 172.21.15.90:0/853906856 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f836001a070 con 0x5640570a6550 2024-04-25T11:38:02.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.228+0000 7f837edb3740 1 -- 172.21.15.90:0/853906856 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f836c004040 con 0x5640570a6550 2024-04-25T11:38:02.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.228+0000 7f83597fa640 1 -- 172.21.15.90:0/853906856 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f8360011870 con 0x5640570a6550 2024-04-25T11:38:02.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.228+0000 7f83597fa640 1 -- 172.21.15.90:0/853906856 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8360026050 con 0x5640570a6550 2024-04-25T11:38:02.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.228+0000 7f837edb3740 10 client.16512 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:02.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.228+0000 7f837edb3740 1 -- 172.21.15.90:0/853906856 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5640570a7170 con 0x5640570a6550 2024-04-25T11:38:02.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.229+0000 7f83597fa640 1 -- 172.21.15.90:0/853906856 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8360016410 con 0x5640570a6550 2024-04-25T11:38:02.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.229+0000 7f83597fa640 1 -- 172.21.15.90:0/853906856 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f8360015050 con 0x5640570a6550 2024-04-25T11:38:02.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.229+0000 7f837edb3740 10 client.16512 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.229+0000 7f837edb3740 10 client.16512 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:38:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.229+0000 7f837edb3740 20 client.16512 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.230+0000 7f837edb3740 1 --2- 172.21.15.90:0/853906856 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56405708e1f0 0x56405748bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.230+0000 7f837edb3740 4 client.16512 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:38:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.230+0000 7f837edb3740 1 -- 172.21.15.90:0/853906856 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f836c004040 con 0x56405708e1f0 2024-04-25T11:38:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.230+0000 7f8375c9c640 1 --2- 172.21.15.90:0/853906856 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56405708e1f0 0x56405748bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.231+0000 7f83597fa640 10 client.16512 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.232+0000 7f8375c9c640 1 --2- 172.21.15.90:0/853906856 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56405708e1f0 0x56405748bf50 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32dedc86bf458b81 server_cookie=a52f953f1925a86 in_seq=0 out_seq=0 2024-04-25T11:38:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.234+0000 7f83597fa640 1 -- 172.21.15.90:0/853906856 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f836c004040 con 0x56405708e1f0 2024-04-25T11:38:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.234+0000 7f83597fa640 10 client.16512 handle_command_reply: tid=0 2024-04-25T11:38:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.234+0000 7f837edb3740 1 client.16512 shutdown 2024-04-25T11:38:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.235+0000 7f837edb3740 1 -- 172.21.15.90:0/853906856 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570a6550 msgr2=0x564057479ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:02.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.235+0000 7f837edb3740 1 --2- 172.21.15.90:0/853906856 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570a6550 0x564057479ce0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f8360013590 tx=0x7f8360013a50 comp rx=0 tx=0).stop 2024-04-25T11:38:02.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.235+0000 7f837edb3740 1 -- 172.21.15.90:0/853906856 shutdown_connections 2024-04-25T11:38:02.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.235+0000 7f837edb3740 1 --2- 172.21.15.90:0/853906856 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56405708e1f0 0x56405748bf50 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.235+0000 7f837edb3740 1 --2- 172.21.15.90:0/853906856 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640570cf410 0x56405747fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.235+0000 7f837edb3740 1 --2- 172.21.15.90:0/853906856 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5640570ae730 0x56405747a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.235+0000 7f837edb3740 1 --2- 172.21.15.90:0/853906856 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570a6550 0x564057479ce0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.235+0000 7f837edb3740 1 -- 172.21.15.90:0/853906856 >> 172.21.15.90:0/853906856 conn(0x5640570c3560 msgr2=0x56405746f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:02.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.235+0000 7f83597fa640 0 client.16512 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:02.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.235+0000 7f837edb3740 1 -- 172.21.15.90:0/853906856 shutdown_connections 2024-04-25T11:38:02.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.235+0000 7f837edb3740 1 -- 172.21.15.90:0/853906856 wait complete. 2024-04-25T11:38:02.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.236+0000 7f837edb3740 20 client.16512 trim_cache size 0 max 16384 2024-04-25T11:38:02.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.272+0000 7f837edb3740 1 Processor -- start 2024-04-25T11:38:02.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.272+0000 7f837edb3740 1 -- start start 2024-04-25T11:38:02.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.272+0000 7f837edb3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5640570a6550 0x564057491710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.272+0000 7f837edb3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570ae730 0x564057491c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.272+0000 7f837edb3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640570cf410 0x56405747fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.272+0000 7f837edb3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f83701ba790 con 0x5640570ae730 2024-04-25T11:38:02.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.272+0000 7f837edb3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f83701bc300 con 0x5640570a6550 2024-04-25T11:38:02.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.272+0000 7f837edb3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f83701bae60 con 0x5640570cf410 2024-04-25T11:38:02.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.272+0000 7f8375c9c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570ae730 0x564057491c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.272+0000 7f8375c9c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570ae730 0x564057491c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41934/0 (socket says 172.21.15.90:41934) 2024-04-25T11:38:02.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.272+0000 7f8375c9c640 1 -- 172.21.15.90:0/1017807038 learned_addr learned my addr 172.21.15.90:0/1017807038 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:02.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.273+0000 7f8375c9c640 1 -- 172.21.15.90:0/1017807038 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640570cf410 msgr2=0x56405747fd00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:38:02.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.273+0000 7f8375c9c640 1 --2- 172.21.15.90:0/1017807038 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640570cf410 0x56405747fd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.273+0000 7f8375c9c640 1 -- 172.21.15.90:0/1017807038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5640570a6550 msgr2=0x564057491710 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:38:02.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.273+0000 7f8375c9c640 1 --2- 172.21.15.90:0/1017807038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5640570a6550 0x564057491710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.273+0000 7f8375c9c640 1 -- 172.21.15.90:0/1017807038 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564057478040 con 0x5640570ae730 2024-04-25T11:38:02.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.273+0000 7f8375c9c640 1 --2- 172.21.15.90:0/1017807038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570ae730 0x564057491c50 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f836c005f70 tx=0x7f836c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c3b93e025e6059a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:02.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.273+0000 7f835b7fe640 1 -- 172.21.15.90:0/1017807038 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f836c053070 con 0x5640570ae730 2024-04-25T11:38:02.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.273+0000 7f835b7fe640 1 -- 172.21.15.90:0/1017807038 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f836c004030 con 0x5640570ae730 2024-04-25T11:38:02.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.273+0000 7f835b7fe640 1 -- 172.21.15.90:0/1017807038 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f836c035040 con 0x5640570ae730 2024-04-25T11:38:02.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.274+0000 7f837edb3740 1 -- 172.21.15.90:0/1017807038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570ae730 msgr2=0x564057491c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:02.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.274+0000 7f837edb3740 1 --2- 172.21.15.90:0/1017807038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570ae730 0x564057491c50 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f836c005f70 tx=0x7f836c005fd0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.275+0000 7f837edb3740 1 -- 172.21.15.90:0/1017807038 shutdown_connections 2024-04-25T11:38:02.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.275+0000 7f837edb3740 1 --2- 172.21.15.90:0/1017807038 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640570cf410 0x56405747fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.275+0000 7f837edb3740 1 --2- 172.21.15.90:0/1017807038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570ae730 0x564057491c50 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.275+0000 7f837edb3740 1 --2- 172.21.15.90:0/1017807038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5640570a6550 0x564057491710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.275+0000 7f837edb3740 1 -- 172.21.15.90:0/1017807038 >> 172.21.15.90:0/1017807038 conn(0x5640570c3560 msgr2=0x56405747a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:02.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.275+0000 7f837edb3740 1 -- 172.21.15.90:0/1017807038 shutdown_connections 2024-04-25T11:38:02.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.275+0000 7f837edb3740 1 -- 172.21.15.90:0/1017807038 wait complete. 2024-04-25T11:38:02.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.275+0000 7f837edb3740 1 -- start start 2024-04-25T11:38:02.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.276+0000 7f837edb3740 1 Processor -- start 2024-04-25T11:38:02.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.276+0000 7f837edb3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5640570a6550 0x564057486350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.277+0000 7f837edb3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640570ae730 0x564057486890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.277+0000 7f837edb3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570cf410 0x564057481800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.277+0000 7f837549b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570cf410 0x564057481800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.277+0000 7f837549b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570cf410 0x564057481800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41944/0 (socket says 172.21.15.90:41944) 2024-04-25T11:38:02.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.277+0000 7f837549b640 1 -- 172.21.15.90:0/2843932372 learned_addr learned my addr 172.21.15.90:0/2843932372 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:02.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.277+0000 7f837edb3740 1 -- 172.21.15.90:0/2843932372 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f83701bbc00 con 0x5640570cf410 2024-04-25T11:38:02.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.277+0000 7f837edb3740 1 -- 172.21.15.90:0/2843932372 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f83701bb530 con 0x5640570a6550 2024-04-25T11:38:02.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.277+0000 7f837edb3740 1 -- 172.21.15.90:0/2843932372 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f83701b8770 con 0x5640570ae730 2024-04-25T11:38:02.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.277+0000 7f837edb3740 10 client.0 _ll_register_callbacks cb 0x7f836000add0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:02.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.277+0000 7f837549b640 1 -- 172.21.15.90:0/2843932372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640570ae730 msgr2=0x564057486890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:02.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.277+0000 7f837549b640 1 --2- 172.21.15.90:0/2843932372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640570ae730 0x564057486890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.278+0000 7f837549b640 1 -- 172.21.15.90:0/2843932372 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5640570a6550 msgr2=0x564057486350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:02.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.278+0000 7f837549b640 1 --2- 172.21.15.90:0/2843932372 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5640570a6550 0x564057486350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.278+0000 7f837549b640 1 -- 172.21.15.90:0/2843932372 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564057481f70 con 0x5640570cf410 2024-04-25T11:38:02.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.278+0000 7f837549b640 1 --2- 172.21.15.90:0/2843932372 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570cf410 0x564057481800 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f8360000c00 tx=0x7f836000a9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1c8192108b68e36 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:02.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.278+0000 7f83597fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:38:02.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.278+0000 7f83597fa640 1 -- 172.21.15.90:0/2843932372 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f836001a070 con 0x5640570cf410 2024-04-25T11:38:02.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.278+0000 7f83597fa640 1 -- 172.21.15.90:0/2843932372 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f8360004040 con 0x5640570cf410 2024-04-25T11:38:02.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.278+0000 7f837edb3740 1 -- 172.21.15.90:0/2843932372 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f836c004030 con 0x5640570cf410 2024-04-25T11:38:02.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.279+0000 7f83597fa640 1 -- 172.21.15.90:0/2843932372 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f836000f040 con 0x5640570cf410 2024-04-25T11:38:02.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.279+0000 7f837edb3740 10 client.16518 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:02.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.279+0000 7f837edb3740 1 -- 172.21.15.90:0/2843932372 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5640574f3e80 con 0x5640570cf410 2024-04-25T11:38:02.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.279+0000 7f83597fa640 1 -- 172.21.15.90:0/2843932372 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f836c004030 con 0x5640570cf410 2024-04-25T11:38:02.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.279+0000 7f83597fa640 1 -- 172.21.15.90:0/2843932372 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f8360029070 con 0x5640570cf410 2024-04-25T11:38:02.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.280+0000 7f837edb3740 10 client.16518 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:02.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.280+0000 7f837edb3740 10 client.16518 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:38:02.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.280+0000 7f837edb3740 20 client.16518 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:02.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.280+0000 7f837edb3740 1 --2- 172.21.15.90:0/2843932372 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564057063630 0x5640574f6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.280+0000 7f837edb3740 4 client.16518 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T11:38:02.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.280+0000 7f837edb3740 1 -- 172.21.15.90:0/2843932372 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5640573e3710 con 0x564057063630 2024-04-25T11:38:02.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.281+0000 7f8375c9c640 1 --2- 172.21.15.90:0/2843932372 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564057063630 0x5640574f6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.286+0000 7f83597fa640 10 client.16518 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:02.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.286+0000 7f8375c9c640 1 --2- 172.21.15.90:0/2843932372 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564057063630 0x5640574f6660 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e277119be47163e1 server_cookie=3f0a9b6ca69a7879 in_seq=0 out_seq=0 2024-04-25T11:38:02.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.289+0000 7f83597fa640 1 -- 172.21.15.90:0/2843932372 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5640573e3710 con 0x564057063630 2024-04-25T11:38:02.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.289+0000 7f83597fa640 10 client.16518 handle_command_reply: tid=0 2024-04-25T11:38:02.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.289+0000 7f837edb3740 1 client.16518 shutdown 2024-04-25T11:38:02.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.289+0000 7f837edb3740 1 -- 172.21.15.90:0/2843932372 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570cf410 msgr2=0x564057481800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:02.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.289+0000 7f837edb3740 1 --2- 172.21.15.90:0/2843932372 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570cf410 0x564057481800 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f8360000c00 tx=0x7f836000a9b0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.290+0000 7f837edb3740 1 -- 172.21.15.90:0/2843932372 shutdown_connections 2024-04-25T11:38:02.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.290+0000 7f837edb3740 1 --2- 172.21.15.90:0/2843932372 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564057063630 0x5640574f6660 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.290+0000 7f837edb3740 1 --2- 172.21.15.90:0/2843932372 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640570cf410 0x564057481800 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.290+0000 7f83597fa640 0 client.16518 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:02.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.290+0000 7f837edb3740 1 --2- 172.21.15.90:0/2843932372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5640570ae730 0x564057486890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.290+0000 7f837edb3740 1 --2- 172.21.15.90:0/2843932372 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5640570a6550 0x564057486350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.290+0000 7f837edb3740 1 -- 172.21.15.90:0/2843932372 >> 172.21.15.90:0/2843932372 conn(0x5640570c3560 msgr2=0x56405747a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:02.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.290+0000 7f837edb3740 1 -- 172.21.15.90:0/2843932372 shutdown_connections 2024-04-25T11:38:02.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.290+0000 7f837edb3740 1 -- 172.21.15.90:0/2843932372 wait complete. 2024-04-25T11:38:02.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.290+0000 7f837edb3740 20 client.16518 trim_cache size 0 max 16384 2024-04-25T11:38:02.318 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:38:02.318 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-25T11:38:02.318 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "159cea10-261f-4886-899c-64abd991e269", 2024-04-25T11:38:02.319 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-25T11:38:02.319 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:38:02.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.291+0000 7f8377f00640 1 -- 172.21.15.90:0/3063256057 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f835c0afc40 msgr2=0x7f835c0b2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:02.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.291+0000 7f8377f00640 1 --2- 172.21.15.90:0/3063256057 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f835c0afc40 0x7f835c0b2150 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f836800a920 tx=0x7f836800a890 comp rx=0 tx=0).stop 2024-04-25T11:38:02.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.291+0000 7f8377f00640 1 -- 172.21.15.90:0/3063256057 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8370115aa0 msgr2=0x7f83701a99f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:02.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.291+0000 7f8377f00640 1 --2- 172.21.15.90:0/3063256057 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8370115aa0 0x7f83701a99f0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f836800cad0 tx=0x7f836800cf90 comp rx=0 tx=0).stop 2024-04-25T11:38:02.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.292+0000 7f8377f00640 1 -- 172.21.15.90:0/3063256057 shutdown_connections 2024-04-25T11:38:02.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.292+0000 7f8377f00640 1 --2- 172.21.15.90:0/3063256057 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f835c0afc40 0x7f835c0b2150 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.292+0000 7f8377f00640 1 --2- 172.21.15.90:0/3063256057 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8370115aa0 0x7f83701a99f0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.292+0000 7f8377f00640 1 --2- 172.21.15.90:0/3063256057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8370110200 0x7f83701a5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.292+0000 7f8377f00640 1 --2- 172.21.15.90:0/3063256057 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f837010bed0 0x7f83701a51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.292+0000 7f8377f00640 1 -- 172.21.15.90:0/3063256057 >> 172.21.15.90:0/3063256057 conn(0x7f837006e7a0 msgr2=0x7f8370071a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:02.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.292+0000 7f8377f00640 1 -- 172.21.15.90:0/3063256057 shutdown_connections 2024-04-25T11:38:02.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.292+0000 7f8377f00640 1 -- 172.21.15.90:0/3063256057 wait complete. 2024-04-25T11:38:02.321 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:38:02.321 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:38:02.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.458+0000 7f0b19900640 1 -- 172.21.15.90:0/1116509929 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b141037d0 msgr2=0x7f0b1410f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:02.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.458+0000 7f0b19900640 1 --2- 172.21.15.90:0/1116509929 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b141037d0 0x7f0b1410f540 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c009ca0 tx=0x7f0b0c02a8c0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.458+0000 7f0b19900640 1 -- 172.21.15.90:0/1116509929 shutdown_connections 2024-04-25T11:38:02.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.458+0000 7f0b19900640 1 --2- 172.21.15.90:0/1116509929 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b141037d0 0x7f0b1410f540 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.458+0000 7f0b19900640 1 --2- 172.21.15.90:0/1116509929 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b14108ec0 0x7f0b14103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.458+0000 7f0b19900640 1 --2- 172.21.15.90:0/1116509929 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b14108a20 0x7f0b14102c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.458+0000 7f0b19900640 1 -- 172.21.15.90:0/1116509929 >> 172.21.15.90:0/1116509929 conn(0x7f0b14077880 msgr2=0x7f0b14077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:02.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.458+0000 7f0b19900640 1 -- 172.21.15.90:0/1116509929 shutdown_connections 2024-04-25T11:38:02.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.458+0000 7f0b19900640 1 -- 172.21.15.90:0/1116509929 wait complete. 2024-04-25T11:38:02.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.459+0000 7f0b19900640 1 Processor -- start 2024-04-25T11:38:02.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.459+0000 7f0b19900640 1 -- start start 2024-04-25T11:38:02.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.459+0000 7f0b19900640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b141037d0 0x7f0b1419a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.459+0000 7f0b19900640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b14108a20 0x7f0b1419aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.459+0000 7f0b19900640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b14108ec0 0x7f0b1419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.459+0000 7f0b19900640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0b14112c30 con 0x7f0b14108ec0 2024-04-25T11:38:02.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.459+0000 7f0b19900640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0b14112ab0 con 0x7f0b14108a20 2024-04-25T11:38:02.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.459+0000 7f0b19900640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0b14112db0 con 0x7f0b141037d0 2024-04-25T11:38:02.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.460+0000 7f0b0bfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b14108a20 0x7f0b1419aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.460+0000 7f0b12ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b141037d0 0x7f0b1419a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.460+0000 7f0b0bfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b14108a20 0x7f0b1419aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:53654/0 (socket says 172.21.15.90:53654) 2024-04-25T11:38:02.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.460+0000 7f0b12ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b141037d0 0x7f0b1419a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:36032/0 (socket says 172.21.15.90:36032) 2024-04-25T11:38:02.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.460+0000 7f0b0bfff640 1 -- 172.21.15.90:0/3893665921 learned_addr learned my addr 172.21.15.90:0/3893665921 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:02.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.460+0000 7f0b137fe640 1 --2- 172.21.15.90:0/3893665921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b14108ec0 0x7f0b1419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.460+0000 7f0b12ffd640 1 -- 172.21.15.90:0/3893665921 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b14108a20 msgr2=0x7f0b1419aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:02.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.460+0000 7f0b12ffd640 1 --2- 172.21.15.90:0/3893665921 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b14108a20 0x7f0b1419aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.460+0000 7f0b12ffd640 1 -- 172.21.15.90:0/3893665921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b14108ec0 msgr2=0x7f0b1419ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:02.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.460+0000 7f0b12ffd640 1 --2- 172.21.15.90:0/3893665921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b14108ec0 0x7f0b1419ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.460+0000 7f0b12ffd640 1 -- 172.21.15.90:0/3893665921 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b1419f5a0 con 0x7f0b141037d0 2024-04-25T11:38:02.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.460+0000 7f0b12ffd640 1 --2- 172.21.15.90:0/3893665921 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b141037d0 0x7f0b1419a740 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0b0400e9c0 tx=0x7f0b0400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:02.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.460+0000 7f0b188fe640 1 -- 172.21.15.90:0/3893665921 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0400ccc0 con 0x7f0b141037d0 2024-04-25T11:38:02.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.460+0000 7f0b19900640 1 -- 172.21.15.90:0/3893665921 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b1419f890 con 0x7f0b141037d0 2024-04-25T11:38:02.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.460+0000 7f0b188fe640 1 -- 172.21.15.90:0/3893665921 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b040108e0 con 0x7f0b141037d0 2024-04-25T11:38:02.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.461+0000 7f0b188fe640 1 -- 172.21.15.90:0/3893665921 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b040187a0 con 0x7f0b141037d0 2024-04-25T11:38:02.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.461+0000 7f0b19900640 1 -- 172.21.15.90:0/3893665921 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b141a7210 con 0x7f0b141037d0 2024-04-25T11:38:02.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.463+0000 7f0b207b2740 1 Processor -- start 2024-04-25T11:38:02.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.463+0000 7f0b207b2740 1 -- start start 2024-04-25T11:38:02.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.463+0000 7f0b188fe640 1 -- 172.21.15.90:0/3893665921 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f0b04018940 con 0x7f0b141037d0 2024-04-25T11:38:02.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.463+0000 7f0b207b2740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d06102390 0x559d064a2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.463+0000 7f0b207b2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d060d94d0 0x559d064a2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.464+0000 7f0b207b2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d060e16b0 0x559d064b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.464+0000 7f0b207b2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559d06408d80 con 0x559d060d94d0 2024-04-25T11:38:02.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.464+0000 7f0b207b2740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x559d063c28b0 con 0x559d06102390 2024-04-25T11:38:02.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.464+0000 7f0b207b2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559d06404d40 con 0x559d060e16b0 2024-04-25T11:38:02.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.464+0000 7f0b12ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d060d94d0 0x559d064a2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.464+0000 7f0b137fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d06102390 0x559d064a2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.464+0000 7f0b137fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d06102390 0x559d064a2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:53664/0 (socket says 172.21.15.90:53664) 2024-04-25T11:38:02.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.464+0000 7f0b12ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d060d94d0 0x559d064a2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41968/0 (socket says 172.21.15.90:41968) 2024-04-25T11:38:02.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.464+0000 7f0b137fe640 1 -- 172.21.15.90:0/3976047526 learned_addr learned my addr 172.21.15.90:0/3976047526 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:02.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.464+0000 7f0b0bfff640 1 --2- 172.21.15.90:0/3976047526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d060e16b0 0x559d064b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.464+0000 7f0b137fe640 1 -- 172.21.15.90:0/3976047526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d060e16b0 msgr2=0x559d064b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:02.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.464+0000 7f0b137fe640 1 --2- 172.21.15.90:0/3976047526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d060e16b0 0x559d064b3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.464+0000 7f0b188fe640 1 --2- 172.21.15.90:0/3893665921 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0b000afce0 0x7f0b000b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.464+0000 7f0b137fe640 1 -- 172.21.15.90:0/3976047526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d060d94d0 msgr2=0x559d064a2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:02.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.464+0000 7f0b137fe640 1 --2- 172.21.15.90:0/3976047526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d060d94d0 0x559d064a2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.464+0000 7f0b137fe640 1 -- 172.21.15.90:0/3976047526 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d06393a40 con 0x559d06102390 2024-04-25T11:38:02.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.464+0000 7f0b0bfff640 1 --2- 172.21.15.90:0/3976047526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d060e16b0 0x559d064b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:02.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.464+0000 7f0b188fe640 1 -- 172.21.15.90:0/3893665921 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f0b040df420 con 0x7f0b141037d0 2024-04-25T11:38:02.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.464+0000 7f0b137fe640 1 --2- 172.21.15.90:0/3893665921 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0b000afce0 0x7f0b000b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.464+0000 7f0b137fe640 1 --2- 172.21.15.90:0/3976047526 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d06102390 0x559d064a2910 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c005cc0 tx=0x7f0b0c02f420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af52d649919c4ec6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:02.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.465+0000 7f0b137fe640 1 --2- 172.21.15.90:0/3893665921 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0b000afce0 0x7f0b000b21f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c04a000 tx=0x7f0b0c04b000 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:02.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.468+0000 7f0b09ffb640 1 -- 172.21.15.90:0/3976047526 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c04d070 con 0x559d06102390 2024-04-25T11:38:02.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.468+0000 7f0b09ffb640 1 -- 172.21.15.90:0/3976047526 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b0c036040 con 0x559d06102390 2024-04-25T11:38:02.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.468+0000 7f0b09ffb640 1 -- 172.21.15.90:0/3976047526 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c004700 con 0x559d06102390 2024-04-25T11:38:02.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.469+0000 7f0b207b2740 1 -- 172.21.15.90:0/3976047526 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d06102390 msgr2=0x559d064a2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:02.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.469+0000 7f0b207b2740 1 --2- 172.21.15.90:0/3976047526 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d06102390 0x559d064a2910 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c005cc0 tx=0x7f0b0c02f420 comp rx=0 tx=0).stop 2024-04-25T11:38:02.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.469+0000 7f0b207b2740 1 -- 172.21.15.90:0/3976047526 shutdown_connections 2024-04-25T11:38:02.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.469+0000 7f0b207b2740 1 --2- 172.21.15.90:0/3976047526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d060e16b0 0x559d064b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.469+0000 7f0b207b2740 1 --2- 172.21.15.90:0/3976047526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d060d94d0 0x559d064a2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.469+0000 7f0b207b2740 1 --2- 172.21.15.90:0/3976047526 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d06102390 0x559d064a2910 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.469+0000 7f0b207b2740 1 -- 172.21.15.90:0/3976047526 >> 172.21.15.90:0/3976047526 conn(0x559d060f64e0 msgr2=0x559d0649f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:02.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.469+0000 7f0b207b2740 1 -- 172.21.15.90:0/3976047526 shutdown_connections 2024-04-25T11:38:02.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.469+0000 7f0b207b2740 1 -- 172.21.15.90:0/3976047526 wait complete. 2024-04-25T11:38:02.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.469+0000 7f0b207b2740 1 -- start start 2024-04-25T11:38:02.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.470+0000 7f0b207b2740 1 Processor -- start 2024-04-25T11:38:02.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.470+0000 7f0b207b2740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d060d94d0 0x559d064b1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.470+0000 7f0b207b2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d060e16b0 0x559d064b1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.470+0000 7f0b207b2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d06102390 0x559d064b7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.470+0000 7f0b207b2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559d064b43f0 con 0x559d06102390 2024-04-25T11:38:02.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.470+0000 7f0b207b2740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x559d0638fd10 con 0x559d060d94d0 2024-04-25T11:38:02.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.470+0000 7f0b207b2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559d064b4570 con 0x559d060e16b0 2024-04-25T11:38:02.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.470+0000 7f0b207b2740 10 client.0 _ll_register_callbacks cb 0x559d063bc210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:02.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.470+0000 7f0b137fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d06102390 0x559d064b7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.470+0000 7f0b12ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d060d94d0 0x559d064b1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.470+0000 7f0b137fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d06102390 0x559d064b7900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41974/0 (socket says 172.21.15.90:41974) 2024-04-25T11:38:02.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.470+0000 7f0b137fe640 1 -- 172.21.15.90:0/1838910864 learned_addr learned my addr 172.21.15.90:0/1838910864 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:02.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.470+0000 7f0b0bfff640 1 --2- 172.21.15.90:0/1838910864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d060e16b0 0x559d064b1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.470+0000 7f0b137fe640 1 -- 172.21.15.90:0/1838910864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d060e16b0 msgr2=0x559d064b1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:02.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.470+0000 7f0b137fe640 1 --2- 172.21.15.90:0/1838910864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d060e16b0 0x559d064b1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.470+0000 7f0b137fe640 1 -- 172.21.15.90:0/1838910864 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d060d94d0 msgr2=0x559d064b1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:02.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.470+0000 7f0b137fe640 1 --2- 172.21.15.90:0/1838910864 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d060d94d0 0x559d064b1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.470+0000 7f0b137fe640 1 -- 172.21.15.90:0/1838910864 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d061e47d0 con 0x559d06102390 2024-04-25T11:38:02.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.470+0000 7f0b12ffd640 1 --2- 172.21.15.90:0/1838910864 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d060d94d0 0x559d064b1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:02.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.471+0000 7f0b137fe640 1 --2- 172.21.15.90:0/1838910864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d06102390 0x559d064b7900 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c02f4b0 tx=0x7f0b0c0048d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1123a3e43163883 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:02.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.471+0000 7f0af2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:38:02.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.471+0000 7f0af2ffd640 1 -- 172.21.15.90:0/1838910864 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c059020 con 0x559d06102390 2024-04-25T11:38:02.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.471+0000 7f0af2ffd640 1 -- 172.21.15.90:0/1838910864 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f0b0c003b70 con 0x559d06102390 2024-04-25T11:38:02.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.471+0000 7f0b207b2740 1 -- 172.21.15.90:0/1838910864 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b0c036040 con 0x559d06102390 2024-04-25T11:38:02.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.471+0000 7f0af2ffd640 1 -- 172.21.15.90:0/1838910864 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0b0c064050 con 0x559d06102390 2024-04-25T11:38:02.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.471+0000 7f0b207b2740 10 client.16530 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:02.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.471+0000 7f0b207b2740 1 -- 172.21.15.90:0/1838910864 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559d0611a3f0 con 0x559d06102390 2024-04-25T11:38:02.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.471+0000 7f0af2ffd640 1 -- 172.21.15.90:0/1838910864 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b0c036040 con 0x559d06102390 2024-04-25T11:38:02.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.472+0000 7f0af2ffd640 1 -- 172.21.15.90:0/1838910864 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f0b0c043070 con 0x559d06102390 2024-04-25T11:38:02.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.472+0000 7f0b207b2740 10 client.16530 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:02.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.472+0000 7f0b207b2740 10 client.16530 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:38:02.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.472+0000 7f0b207b2740 20 client.16530 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:02.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.472+0000 7f0b207b2740 1 --2- 172.21.15.90:0/1838910864 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559d060c0f60 0x559d064c39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.472+0000 7f0b207b2740 4 client.16530 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:38:02.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.472+0000 7f0b207b2740 1 -- 172.21.15.90:0/1838910864 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559d06416990 con 0x559d060c0f60 2024-04-25T11:38:02.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.472+0000 7f0b12ffd640 1 --2- 172.21.15.90:0/1838910864 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559d060c0f60 0x559d064c39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.472+0000 7f0af2ffd640 10 client.16530 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:02.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.472+0000 7f0b12ffd640 1 --2- 172.21.15.90:0/1838910864 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559d060c0f60 0x559d064c39c0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=761357aa87755825 server_cookie=5bb013191aacf685 in_seq=0 out_seq=0 2024-04-25T11:38:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.473+0000 7f0af2ffd640 1 -- 172.21.15.90:0/1838910864 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559d06416990 con 0x559d060c0f60 2024-04-25T11:38:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.473+0000 7f0af2ffd640 10 client.16530 handle_command_reply: tid=0 2024-04-25T11:38:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.473+0000 7f0b207b2740 1 client.16530 shutdown 2024-04-25T11:38:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.473+0000 7f0b207b2740 1 -- 172.21.15.90:0/1838910864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d06102390 msgr2=0x559d064b7900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.473+0000 7f0b207b2740 1 --2- 172.21.15.90:0/1838910864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d06102390 0x559d064b7900 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c02f4b0 tx=0x7f0b0c0048d0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.473+0000 7f0b207b2740 1 -- 172.21.15.90:0/1838910864 shutdown_connections 2024-04-25T11:38:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.473+0000 7f0b207b2740 1 --2- 172.21.15.90:0/1838910864 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559d060c0f60 0x559d064c39c0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.473+0000 7f0af2ffd640 0 client.16530 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.473+0000 7f0b207b2740 1 --2- 172.21.15.90:0/1838910864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d06102390 0x559d064b7900 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.473+0000 7f0b207b2740 1 --2- 172.21.15.90:0/1838910864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d060e16b0 0x559d064b1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.473+0000 7f0b207b2740 1 --2- 172.21.15.90:0/1838910864 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d060d94d0 0x559d064b1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.473+0000 7f0b207b2740 1 -- 172.21.15.90:0/1838910864 >> 172.21.15.90:0/1838910864 conn(0x559d060f64e0 msgr2=0x559d064a46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.473+0000 7f0b207b2740 1 -- 172.21.15.90:0/1838910864 shutdown_connections 2024-04-25T11:38:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.473+0000 7f0b207b2740 1 -- 172.21.15.90:0/1838910864 wait complete. 2024-04-25T11:38:02.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.474+0000 7f0b207b2740 20 client.16530 trim_cache size 0 max 16384 2024-04-25T11:38:02.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.495+0000 7f0b207b2740 1 Processor -- start 2024-04-25T11:38:02.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.495+0000 7f0b207b2740 1 -- start start 2024-04-25T11:38:02.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.496+0000 7f0b207b2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d060d94d0 0x559d064aca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.496+0000 7f0b207b2740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d060e16b0 0x559d064b09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.496+0000 7f0b207b2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d06102390 0x559d0649e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.496+0000 7f0b207b2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0b141ada40 con 0x559d060d94d0 2024-04-25T11:38:02.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.496+0000 7f0b207b2740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0b141af5b0 con 0x559d060e16b0 2024-04-25T11:38:02.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.496+0000 7f0b207b2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0b141ae110 con 0x559d06102390 2024-04-25T11:38:02.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.496+0000 7f0b12ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d060d94d0 0x559d064aca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.496+0000 7f0b12ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d060d94d0 0x559d064aca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41984/0 (socket says 172.21.15.90:41984) 2024-04-25T11:38:02.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.496+0000 7f0b137fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d06102390 0x559d0649e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.496+0000 7f0b12ffd640 1 -- 172.21.15.90:0/2429092992 learned_addr learned my addr 172.21.15.90:0/2429092992 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:02.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.496+0000 7f0b0bfff640 1 --2- 172.21.15.90:0/2429092992 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d060e16b0 0x559d064b09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.496+0000 7f0b12ffd640 1 -- 172.21.15.90:0/2429092992 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d06102390 msgr2=0x559d0649e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:02.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.496+0000 7f0b12ffd640 1 --2- 172.21.15.90:0/2429092992 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d06102390 0x559d0649e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.496+0000 7f0b12ffd640 1 -- 172.21.15.90:0/2429092992 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d060e16b0 msgr2=0x559d064b09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:02.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.496+0000 7f0b12ffd640 1 --2- 172.21.15.90:0/2429092992 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d060e16b0 0x559d064b09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.496+0000 7f0b12ffd640 1 -- 172.21.15.90:0/2429092992 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d060da0f0 con 0x559d060d94d0 2024-04-25T11:38:02.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.496+0000 7f0b137fe640 1 --2- 172.21.15.90:0/2429092992 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d06102390 0x559d0649e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:02.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.496+0000 7f0b12ffd640 1 --2- 172.21.15.90:0/2429092992 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d060d94d0 0x559d064aca00 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f0b040dcb70 tx=0x7f0b040e1930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b12f595bf26cab88 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:02.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.496+0000 7f0b097fa640 1 -- 172.21.15.90:0/2429092992 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b040a6cd0 con 0x559d060d94d0 2024-04-25T11:38:02.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.496+0000 7f0b097fa640 1 -- 172.21.15.90:0/2429092992 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b040a7070 con 0x559d060d94d0 2024-04-25T11:38:02.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.496+0000 7f0b097fa640 1 -- 172.21.15.90:0/2429092992 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b04010a90 con 0x559d060d94d0 2024-04-25T11:38:02.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.497+0000 7f0b207b2740 1 -- 172.21.15.90:0/2429092992 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d060d94d0 msgr2=0x559d064aca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:02.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.497+0000 7f0b207b2740 1 --2- 172.21.15.90:0/2429092992 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d060d94d0 0x559d064aca00 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f0b040dcb70 tx=0x7f0b040e1930 comp rx=0 tx=0).stop 2024-04-25T11:38:02.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.497+0000 7f0b207b2740 1 -- 172.21.15.90:0/2429092992 shutdown_connections 2024-04-25T11:38:02.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.497+0000 7f0b207b2740 1 --2- 172.21.15.90:0/2429092992 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d06102390 0x559d0649e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.497+0000 7f0b207b2740 1 --2- 172.21.15.90:0/2429092992 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d060e16b0 0x559d064b09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.497+0000 7f0b207b2740 1 --2- 172.21.15.90:0/2429092992 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d060d94d0 0x559d064aca00 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.497+0000 7f0b207b2740 1 -- 172.21.15.90:0/2429092992 >> 172.21.15.90:0/2429092992 conn(0x559d060f64e0 msgr2=0x559d064a4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:02.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.497+0000 7f0b207b2740 1 -- 172.21.15.90:0/2429092992 shutdown_connections 2024-04-25T11:38:02.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.497+0000 7f0b207b2740 1 -- 172.21.15.90:0/2429092992 wait complete. 2024-04-25T11:38:02.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.498+0000 7f0b207b2740 1 -- start start 2024-04-25T11:38:02.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.498+0000 7f0b207b2740 1 Processor -- start 2024-04-25T11:38:02.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.498+0000 7f0b207b2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d060d94d0 0x559d064bdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.498+0000 7f0b207b2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d060e16b0 0x559d064be250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.498+0000 7f0b207b2740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d06102390 0x559d064bb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.498+0000 7f0b207b2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0b141aeeb0 con 0x559d060d94d0 2024-04-25T11:38:02.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.498+0000 7f0b207b2740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0b141ae7e0 con 0x559d06102390 2024-04-25T11:38:02.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.498+0000 7f0b207b2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0b141aba20 con 0x559d060e16b0 2024-04-25T11:38:02.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.498+0000 7f0b207b2740 10 client.0 _ll_register_callbacks cb 0x7f0b0c04b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:02.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.498+0000 7f0b12ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d060d94d0 0x559d064bdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.499+0000 7f0b12ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d060d94d0 0x559d064bdcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41988/0 (socket says 172.21.15.90:41988) 2024-04-25T11:38:02.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.499+0000 7f0b12ffd640 1 -- 172.21.15.90:0/657901257 learned_addr learned my addr 172.21.15.90:0/657901257 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:02.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.499+0000 7f0b0bfff640 1 --2- 172.21.15.90:0/657901257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d060e16b0 0x559d064be250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.499+0000 7f0b137fe640 1 --2- 172.21.15.90:0/657901257 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d06102390 0x559d064bb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.499+0000 7f0b12ffd640 1 -- 172.21.15.90:0/657901257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d060e16b0 msgr2=0x559d064be250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:02.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.499+0000 7f0b12ffd640 1 --2- 172.21.15.90:0/657901257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d060e16b0 0x559d064be250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.499+0000 7f0b12ffd640 1 -- 172.21.15.90:0/657901257 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d06102390 msgr2=0x559d064bb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:02.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.499+0000 7f0b12ffd640 1 --2- 172.21.15.90:0/657901257 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d06102390 0x559d064bb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.499+0000 7f0b12ffd640 1 -- 172.21.15.90:0/657901257 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d063c94b0 con 0x559d060d94d0 2024-04-25T11:38:02.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.499+0000 7f0b12ffd640 1 --2- 172.21.15.90:0/657901257 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d060d94d0 0x559d064bdcf0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f0b040a7260 tx=0x7f0b040b0430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e256b9d1ce37713 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:02.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.499+0000 7f0af2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:38:02.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.499+0000 7f0af2ffd640 1 -- 172.21.15.90:0/657901257 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b040a7560 con 0x559d060d94d0 2024-04-25T11:38:02.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.499+0000 7f0af2ffd640 1 -- 172.21.15.90:0/657901257 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f0b040a37f0 con 0x559d060d94d0 2024-04-25T11:38:02.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.499+0000 7f0b207b2740 1 -- 172.21.15.90:0/657901257 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b040a7070 con 0x559d060d94d0 2024-04-25T11:38:02.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.499+0000 7f0af2ffd640 1 -- 172.21.15.90:0/657901257 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0b040ab910 con 0x559d060d94d0 2024-04-25T11:38:02.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.499+0000 7f0b207b2740 10 client.16536 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:02.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.499+0000 7f0b207b2740 1 -- 172.21.15.90:0/657901257 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559d0652b2b0 con 0x559d060d94d0 2024-04-25T11:38:02.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.500+0000 7f0af2ffd640 1 -- 172.21.15.90:0/657901257 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b040a7070 con 0x559d060d94d0 2024-04-25T11:38:02.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.500+0000 7f0af2ffd640 1 -- 172.21.15.90:0/657901257 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f0b040a67c0 con 0x559d060d94d0 2024-04-25T11:38:02.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.500+0000 7f0b207b2740 10 client.16536 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:02.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.500+0000 7f0b207b2740 10 client.16536 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:38:02.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.500+0000 7f0b207b2740 20 client.16536 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:02.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.500+0000 7f0b207b2740 1 --2- 172.21.15.90:0/657901257 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559d060963a0 0x559d0652da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.500+0000 7f0b207b2740 4 client.16536 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:38:02.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.500+0000 7f0b207b2740 1 -- 172.21.15.90:0/657901257 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559d06410800 con 0x559d060963a0 2024-04-25T11:38:02.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.500+0000 7f0b0bfff640 1 --2- 172.21.15.90:0/657901257 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559d060963a0 0x559d0652da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.500+0000 7f0af2ffd640 10 client.16536 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:02.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.500+0000 7f0b0bfff640 1 --2- 172.21.15.90:0/657901257 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559d060963a0 0x559d0652da90 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a851ed529444233 server_cookie=c306cc29da739908 in_seq=0 out_seq=0 2024-04-25T11:38:02.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.501+0000 7f0af2ffd640 1 -- 172.21.15.90:0/657901257 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559d06410800 con 0x559d060963a0 2024-04-25T11:38:02.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.501+0000 7f0af2ffd640 10 client.16536 handle_command_reply: tid=0 2024-04-25T11:38:02.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.501+0000 7f0b207b2740 1 client.16536 shutdown 2024-04-25T11:38:02.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.501+0000 7f0b207b2740 1 -- 172.21.15.90:0/657901257 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d060d94d0 msgr2=0x559d064bdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:02.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.501+0000 7f0b207b2740 1 --2- 172.21.15.90:0/657901257 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d060d94d0 0x559d064bdcf0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f0b040a7260 tx=0x7f0b040b0430 comp rx=0 tx=0).stop 2024-04-25T11:38:02.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.501+0000 7f0b207b2740 1 -- 172.21.15.90:0/657901257 shutdown_connections 2024-04-25T11:38:02.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.501+0000 7f0b207b2740 1 --2- 172.21.15.90:0/657901257 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559d060963a0 0x559d0652da90 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.501+0000 7f0af2ffd640 0 client.16536 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:02.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.501+0000 7f0b207b2740 1 --2- 172.21.15.90:0/657901257 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559d06102390 0x559d064bb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.501+0000 7f0b207b2740 1 --2- 172.21.15.90:0/657901257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d060e16b0 0x559d064be250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.501+0000 7f0b207b2740 1 --2- 172.21.15.90:0/657901257 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d060d94d0 0x559d064bdcf0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.501+0000 7f0b207b2740 1 -- 172.21.15.90:0/657901257 >> 172.21.15.90:0/657901257 conn(0x559d060f64e0 msgr2=0x559d064a4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:02.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.501+0000 7f0b207b2740 1 -- 172.21.15.90:0/657901257 shutdown_connections 2024-04-25T11:38:02.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.501+0000 7f0b207b2740 1 -- 172.21.15.90:0/657901257 wait complete. 2024-04-25T11:38:02.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.501+0000 7f0b207b2740 20 client.16536 trim_cache size 0 max 16384 2024-04-25T11:38:02.495 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:38:02.495 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T11:38:02.495 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-25T11:38:02.496 INFO:teuthology.orchestra.run.smithi090.stdout: "159cea10-261f-4886-899c-64abd991e269": { 2024-04-25T11:38:02.496 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-25T11:38:02.496 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "159cea10-261f-4886-899c-64abd991e269", 2024-04-25T11:38:02.496 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-25T11:38:02.496 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:38:02.496 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:38:02.496 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:38:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.502+0000 7f0b19900640 1 -- 172.21.15.90:0/3893665921 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0b000afce0 msgr2=0x7f0b000b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.502+0000 7f0b19900640 1 --2- 172.21.15.90:0/3893665921 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0b000afce0 0x7f0b000b21f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c04a000 tx=0x7f0b0c04b000 comp rx=0 tx=0).stop 2024-04-25T11:38:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.502+0000 7f0b19900640 1 -- 172.21.15.90:0/3893665921 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b141037d0 msgr2=0x7f0b1419a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.502+0000 7f0b19900640 1 --2- 172.21.15.90:0/3893665921 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b141037d0 0x7f0b1419a740 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0b0400e9c0 tx=0x7f0b0400ee80 comp rx=0 tx=0).stop 2024-04-25T11:38:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.502+0000 7f0b19900640 1 -- 172.21.15.90:0/3893665921 shutdown_connections 2024-04-25T11:38:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.502+0000 7f0b19900640 1 --2- 172.21.15.90:0/3893665921 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0b000afce0 0x7f0b000b21f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.502+0000 7f0b19900640 1 --2- 172.21.15.90:0/3893665921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b14108ec0 0x7f0b1419ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.502+0000 7f0b19900640 1 --2- 172.21.15.90:0/3893665921 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b14108a20 0x7f0b1419aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.502+0000 7f0b19900640 1 --2- 172.21.15.90:0/3893665921 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b141037d0 0x7f0b1419a740 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.502+0000 7f0b19900640 1 -- 172.21.15.90:0/3893665921 >> 172.21.15.90:0/3893665921 conn(0x7f0b14077880 msgr2=0x7f0b14100c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.502+0000 7f0b19900640 1 -- 172.21.15.90:0/3893665921 shutdown_connections 2024-04-25T11:38:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:02.502+0000 7f0b19900640 1 -- 172.21.15.90:0/3893665921 wait complete. 2024-04-25T11:38:02.504 INFO:tasks.cephfs.filesystem:scrub status for tag:159cea10-261f-4886-899c-64abd991e269 - {'path': '/', 'tag': '159cea10-261f-4886-899c-64abd991e269', 'options': 'recursive,force'} 2024-04-25T11:38:03.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:03 smithi138 ceph-mon[32328]: pgmap v92: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 357 MiB/s wr, 367 op/s 2024-04-25T11:38:03.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:03 smithi138 ceph-mon[32328]: scrub queued for path: / 2024-04-25T11:38:03.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:03 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:38:03.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:03 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:38:03.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:03 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:38:03.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:03 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:38:03.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:03 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:38:03.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:03 smithi156 ceph-mon[33045]: pgmap v92: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 357 MiB/s wr, 367 op/s 2024-04-25T11:38:03.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:03 smithi156 ceph-mon[33045]: scrub queued for path: / 2024-04-25T11:38:03.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:03 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:38:03.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:03 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:38:03.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:03 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:38:03.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:03 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:38:03.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:03 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:38:03.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:03 smithi090 ceph-mon[29575]: pgmap v92: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 357 MiB/s wr, 367 op/s 2024-04-25T11:38:03.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:03 smithi090 ceph-mon[29575]: scrub queued for path: / 2024-04-25T11:38:03.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:03 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:38:03.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:03 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:38:03.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:03 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:38:03.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:03 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:38:03.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:03 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:38:05.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:05 smithi138 ceph-mon[32328]: pgmap v93: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 400 MiB/s wr, 410 op/s 2024-04-25T11:38:05.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:05 smithi156 ceph-mon[33045]: pgmap v93: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 400 MiB/s wr, 410 op/s 2024-04-25T11:38:05.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:05 smithi090 ceph-mon[29575]: pgmap v93: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 400 MiB/s wr, 410 op/s 2024-04-25T11:38:07.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:07 smithi138 ceph-mon[32328]: pgmap v94: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 467 MiB/s wr, 521 op/s 2024-04-25T11:38:07.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:07 smithi156 ceph-mon[33045]: pgmap v94: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 467 MiB/s wr, 521 op/s 2024-04-25T11:38:07.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:07 smithi090 ceph-mon[29575]: pgmap v94: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 467 MiB/s wr, 521 op/s 2024-04-25T11:38:09.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:09 smithi156 ceph-mon[33045]: pgmap v95: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 855 KiB/s rd, 331 MiB/s wr, 367 op/s 2024-04-25T11:38:09.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:09 smithi090 ceph-mon[29575]: pgmap v95: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 855 KiB/s rd, 331 MiB/s wr, 367 op/s 2024-04-25T11:38:10.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:09 smithi138 ceph-mon[32328]: pgmap v95: 97 pgs: 97 active+clean; 20 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 855 KiB/s rd, 331 MiB/s wr, 367 op/s 2024-04-25T11:38:10.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:10 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:38:10.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:10 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:38:11.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:10 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:38:11.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:11 smithi156 ceph-mon[33045]: pgmap v96: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 559 MiB/s wr, 1.85k op/s 2024-04-25T11:38:11.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:11 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:38:11.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:11 smithi090 ceph-mon[29575]: pgmap v96: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 559 MiB/s wr, 1.85k op/s 2024-04-25T11:38:11.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:11 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:38:12.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:11 smithi138 ceph-mon[32328]: pgmap v96: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 559 MiB/s wr, 1.85k op/s 2024-04-25T11:38:12.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:11 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:38:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.573+0000 7f86450e0640 1 -- 172.21.15.90:0/2265397174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8640108a80 msgr2=0x7f8640102620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.573+0000 7f86450e0640 1 --2- 172.21.15.90:0/2265397174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8640108a80 0x7f8640102620 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f8630009900 tx=0x7f863002a410 comp rx=0 tx=0).stop 2024-04-25T11:38:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.573+0000 7f86450e0640 1 -- 172.21.15.90:0/2265397174 shutdown_connections 2024-04-25T11:38:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.573+0000 7f86450e0640 1 --2- 172.21.15.90:0/2265397174 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f86401031d0 0x7f864010f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.573+0000 7f86450e0640 1 --2- 172.21.15.90:0/2265397174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8640108f20 0x7f8640102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.573+0000 7f86450e0640 1 --2- 172.21.15.90:0/2265397174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8640108a80 0x7f8640102620 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.573+0000 7f86450e0640 1 -- 172.21.15.90:0/2265397174 >> 172.21.15.90:0/2265397174 conn(0x7f8640078150 msgr2=0x7f8640078590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.573+0000 7f86450e0640 1 -- 172.21.15.90:0/2265397174 shutdown_connections 2024-04-25T11:38:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.573+0000 7f86450e0640 1 -- 172.21.15.90:0/2265397174 wait complete. 2024-04-25T11:38:12.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.574+0000 7f86450e0640 1 Processor -- start 2024-04-25T11:38:12.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.574+0000 7f86450e0640 1 -- start start 2024-04-25T11:38:12.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.575+0000 7f86450e0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86401031d0 0x7f86400ffd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.575+0000 7f86450e0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8640108a80 0x7f8640100270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.575+0000 7f86450e0640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8640108f20 0x7f86401007b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.575+0000 7f86450e0640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8640112c70 con 0x7f86401031d0 2024-04-25T11:38:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.575+0000 7f86450e0640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8640112af0 con 0x7f8640108f20 2024-04-25T11:38:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.575+0000 7f86450e0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8640112df0 con 0x7f8640108a80 2024-04-25T11:38:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.575+0000 7f863ed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86401031d0 0x7f86400ffd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.575+0000 7f863f59e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8640108f20 0x7f86401007b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.575+0000 7f863f59e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8640108f20 0x7f86401007b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:57796/0 (socket says 172.21.15.90:57796) 2024-04-25T11:38:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.575+0000 7f863f59e640 1 -- 172.21.15.90:0/3944308565 learned_addr learned my addr 172.21.15.90:0/3944308565 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.575+0000 7f8637fff640 1 --2- 172.21.15.90:0/3944308565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8640108a80 0x7f8640100270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.575+0000 7f8637fff640 1 -- 172.21.15.90:0/3944308565 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8640108f20 msgr2=0x7f86401007b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.575+0000 7f8637fff640 1 --2- 172.21.15.90:0/3944308565 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8640108f20 0x7f86401007b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.575+0000 7f8637fff640 1 -- 172.21.15.90:0/3944308565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86401031d0 msgr2=0x7f86400ffd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:12.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.575+0000 7f8637fff640 1 --2- 172.21.15.90:0/3944308565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86401031d0 0x7f86400ffd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:12.566 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.575+0000 7f8637fff640 1 -- 172.21.15.90:0/3944308565 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86401a8b20 con 0x7f8640108a80 2024-04-25T11:38:12.566 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.575+0000 7f8637fff640 1 --2- 172.21.15.90:0/3944308565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8640108a80 0x7f8640100270 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f864006a310 tx=0x7f862800ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:12.566 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.576+0000 7f863cd99640 1 -- 172.21.15.90:0/3944308565 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862800cc40 con 0x7f8640108a80 2024-04-25T11:38:12.566 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.576+0000 7f863cd99640 1 -- 172.21.15.90:0/3944308565 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f862800cde0 con 0x7f8640108a80 2024-04-25T11:38:12.566 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.576+0000 7f863cd99640 1 -- 172.21.15.90:0/3944308565 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8628010430 con 0x7f8640108a80 2024-04-25T11:38:12.566 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.576+0000 7f86450e0640 1 -- 172.21.15.90:0/3944308565 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86401a8e10 con 0x7f8640108a80 2024-04-25T11:38:12.566 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.576+0000 7f86450e0640 1 -- 172.21.15.90:0/3944308565 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86401a9280 con 0x7f8640108a80 2024-04-25T11:38:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.578+0000 7f863cd99640 1 -- 172.21.15.90:0/3944308565 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f86280105d0 con 0x7f8640108a80 2024-04-25T11:38:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.578+0000 7f864bf92740 1 Processor -- start 2024-04-25T11:38:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.579+0000 7f864bf92740 1 -- start start 2024-04-25T11:38:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.579+0000 7f864bf92740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f71057a1b0 0x55f710919af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.579+0000 7f864bf92740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7105512f0 0x55f71091a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.579+0000 7f864bf92740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7105594d0 0x55f71091a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.579+0000 7f864bf92740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f710880460 con 0x55f7105594d0 2024-04-25T11:38:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.579+0000 7f864bf92740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f71083a590 con 0x55f7105512f0 2024-04-25T11:38:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.579+0000 7f864bf92740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f71087c420 con 0x55f71057a1b0 2024-04-25T11:38:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.579+0000 7f863ed9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7105512f0 0x55f71091a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.579+0000 7f8637fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7105594d0 0x55f71091a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.579+0000 7f863ed9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7105512f0 0x55f71091a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:57808/0 (socket says 172.21.15.90:57808) 2024-04-25T11:38:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.579+0000 7f863ed9d640 1 -- 172.21.15.90:0/49185203 learned_addr learned my addr 172.21.15.90:0/49185203 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.579+0000 7f863f59e640 1 --2- 172.21.15.90:0/49185203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f71057a1b0 0x55f710919af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.580+0000 7f863cd99640 1 --2- 172.21.15.90:0/3944308565 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f862c0afcd0 0x7f862c0b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.580+0000 7f863ed9d640 1 -- 172.21.15.90:0/49185203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f71057a1b0 msgr2=0x55f710919af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.580+0000 7f863ed9d640 1 --2- 172.21.15.90:0/49185203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f71057a1b0 0x55f710919af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.580+0000 7f863ed9d640 1 -- 172.21.15.90:0/49185203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7105594d0 msgr2=0x55f71091a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.580+0000 7f863ed9d640 1 --2- 172.21.15.90:0/49185203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7105594d0 0x55f71091a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.580+0000 7f863ed9d640 1 -- 172.21.15.90:0/49185203 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7108470d0 con 0x55f7105512f0 2024-04-25T11:38:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.580+0000 7f863f59e640 1 --2- 172.21.15.90:0/49185203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f71057a1b0 0x55f710919af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:38:12.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.580+0000 7f863f59e640 1 --2- 172.21.15.90:0/3944308565 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f862c0afcd0 0x7f862c0b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:12.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.580+0000 7f863cd99640 1 -- 172.21.15.90:0/3944308565 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f86280df330 con 0x7f8640108a80 2024-04-25T11:38:12.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.580+0000 7f863ed9d640 1 --2- 172.21.15.90:0/49185203 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7105512f0 0x55f71091a030 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8630009900 tx=0x7f8630032b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a6354e0d529d12 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:12.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.580+0000 7f86357fa640 1 -- 172.21.15.90:0/49185203 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8630040070 con 0x55f7105512f0 2024-04-25T11:38:12.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.580+0000 7f86357fa640 1 -- 172.21.15.90:0/49185203 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8630004030 con 0x55f7105512f0 2024-04-25T11:38:12.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.580+0000 7f86357fa640 1 -- 172.21.15.90:0/49185203 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8630035040 con 0x55f7105512f0 2024-04-25T11:38:12.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.582+0000 7f864bf92740 1 -- 172.21.15.90:0/49185203 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7105512f0 msgr2=0x55f71091a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:12.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.582+0000 7f863f59e640 1 --2- 172.21.15.90:0/3944308565 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f862c0afcd0 0x7f862c0b21e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8638009fd0 tx=0x7f8638004a40 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:12.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.582+0000 7f864bf92740 1 --2- 172.21.15.90:0/49185203 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7105512f0 0x55f71091a030 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8630009900 tx=0x7f8630032b20 comp rx=0 tx=0).stop 2024-04-25T11:38:12.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.582+0000 7f864bf92740 1 -- 172.21.15.90:0/49185203 shutdown_connections 2024-04-25T11:38:12.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.582+0000 7f864bf92740 1 --2- 172.21.15.90:0/49185203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7105594d0 0x55f71091a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:12.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.582+0000 7f864bf92740 1 --2- 172.21.15.90:0/49185203 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7105512f0 0x55f71091a030 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:12.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.582+0000 7f864bf92740 1 --2- 172.21.15.90:0/49185203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f71057a1b0 0x55f710919af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:12.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.582+0000 7f864bf92740 1 -- 172.21.15.90:0/49185203 >> 172.21.15.90:0/49185203 conn(0x55f71056e300 msgr2=0x55f7109162a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:12.572 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.582+0000 7f864bf92740 1 -- 172.21.15.90:0/49185203 shutdown_connections 2024-04-25T11:38:12.572 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.582+0000 7f864bf92740 1 -- 172.21.15.90:0/49185203 wait complete. 2024-04-25T11:38:12.572 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.583+0000 7f864bf92740 1 -- start start 2024-04-25T11:38:12.572 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.583+0000 7f864bf92740 1 Processor -- start 2024-04-25T11:38:12.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.584+0000 7f864bf92740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7105512f0 0x55f710925b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:12.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.584+0000 7f864bf92740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7105594d0 0x55f710926040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:12.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.584+0000 7f864bf92740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f71057a1b0 0x55f710923ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:12.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.584+0000 7f864bf92740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f710926bc0 con 0x55f7105512f0 2024-04-25T11:38:12.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.584+0000 7f864bf92740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f710807b60 con 0x55f7105594d0 2024-04-25T11:38:12.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.584+0000 7f864bf92740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f710926d40 con 0x55f71057a1b0 2024-04-25T11:38:12.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.584+0000 7f864bf92740 10 client.0 _ll_register_callbacks cb 0x55f71082c2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:12.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.584+0000 7f863f59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f71057a1b0 0x55f710923ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.584+0000 7f863ed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7105512f0 0x55f710925b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.584+0000 7f863f59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f71057a1b0 0x55f710923ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:59378/0 (socket says 172.21.15.90:59378) 2024-04-25T11:38:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.584+0000 7f8637fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7105594d0 0x55f710926040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.584+0000 7f863ed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7105512f0 0x55f710925b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39426/0 (socket says 172.21.15.90:39426) 2024-04-25T11:38:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.584+0000 7f863f59e640 1 -- 172.21.15.90:0/3492691419 learned_addr learned my addr 172.21.15.90:0/3492691419 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.584+0000 7f863f59e640 1 -- 172.21.15.90:0/3492691419 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7105594d0 msgr2=0x55f710926040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.584+0000 7f863f59e640 1 --2- 172.21.15.90:0/3492691419 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7105594d0 0x55f710926040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.584+0000 7f863f59e640 1 -- 172.21.15.90:0/3492691419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7105512f0 msgr2=0x55f710925b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.584+0000 7f863f59e640 1 --2- 172.21.15.90:0/3492691419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7105512f0 0x55f710925b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.584+0000 7f863f59e640 1 -- 172.21.15.90:0/3492691419 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f710592210 con 0x55f71057a1b0 2024-04-25T11:38:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.584+0000 7f8637fff640 1 --2- 172.21.15.90:0/3492691419 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7105594d0 0x55f710926040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.585+0000 7f863f59e640 1 --2- 172.21.15.90:0/3492691419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f71057a1b0 0x55f710923ff0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f863800faf0 tx=0x7f8638004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b34742eb4a00d55 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.585+0000 7f86267fc640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:38:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.585+0000 7f86267fc640 1 -- 172.21.15.90:0/3492691419 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86380108c0 con 0x55f71057a1b0 2024-04-25T11:38:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.585+0000 7f86267fc640 1 -- 172.21.15.90:0/3492691419 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f8638010a60 con 0x55f71057a1b0 2024-04-25T11:38:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.585+0000 7f864bf92740 1 -- 172.21.15.90:0/3492691419 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8630004030 con 0x55f71057a1b0 2024-04-25T11:38:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.585+0000 7f86267fc640 1 -- 172.21.15.90:0/3492691419 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8638010c60 con 0x55f71057a1b0 2024-04-25T11:38:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.585+0000 7f864bf92740 10 client.25756 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.585+0000 7f864bf92740 1 -- 172.21.15.90:0/3492691419 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f71065e620 con 0x55f71057a1b0 2024-04-25T11:38:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.585+0000 7f86267fc640 1 -- 172.21.15.90:0/3492691419 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8630004030 con 0x55f71057a1b0 2024-04-25T11:38:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.585+0000 7f86267fc640 1 -- 172.21.15.90:0/3492691419 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f8638025070 con 0x55f71057a1b0 2024-04-25T11:38:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.585+0000 7f864bf92740 10 client.25756 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.585+0000 7f864bf92740 10 client.25756 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:38:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.585+0000 7f864bf92740 20 client.25756 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.585+0000 7f864bf92740 1 --2- 172.21.15.90:0/3492691419 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f710538d80 0x55f7109367b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.585+0000 7f864bf92740 4 client.25756 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:38:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.585+0000 7f864bf92740 1 -- 172.21.15.90:0/3492691419 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f71088e070 con 0x55f710538d80 2024-04-25T11:38:12.578 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.586+0000 7f863ed9d640 1 --2- 172.21.15.90:0/3492691419 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f710538d80 0x55f7109367b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:12.578 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.586+0000 7f86267fc640 10 client.25756 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:12.578 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.586+0000 7f863ed9d640 1 --2- 172.21.15.90:0/3492691419 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f710538d80 0x55f7109367b0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22fa508f33951d83 server_cookie=5a8cc59f35d32216 in_seq=0 out_seq=0 2024-04-25T11:38:12.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.587+0000 7f86267fc640 1 -- 172.21.15.90:0/3492691419 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f71088e070 con 0x55f710538d80 2024-04-25T11:38:12.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.587+0000 7f86267fc640 10 client.25756 handle_command_reply: tid=0 2024-04-25T11:38:12.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.588+0000 7f864bf92740 1 client.25756 shutdown 2024-04-25T11:38:12.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.588+0000 7f864bf92740 1 -- 172.21.15.90:0/3492691419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f71057a1b0 msgr2=0x55f710923ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:12.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.588+0000 7f864bf92740 1 --2- 172.21.15.90:0/3492691419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f71057a1b0 0x55f710923ff0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f863800faf0 tx=0x7f8638004ab0 comp rx=0 tx=0).stop 2024-04-25T11:38:12.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.588+0000 7f864bf92740 1 -- 172.21.15.90:0/3492691419 shutdown_connections 2024-04-25T11:38:12.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.588+0000 7f864bf92740 1 --2- 172.21.15.90:0/3492691419 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f710538d80 0x55f7109367b0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:12.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.588+0000 7f864bf92740 1 --2- 172.21.15.90:0/3492691419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f71057a1b0 0x55f710923ff0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:12.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.588+0000 7f864bf92740 1 --2- 172.21.15.90:0/3492691419 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7105594d0 0x55f710926040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:12.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.588+0000 7f864bf92740 1 --2- 172.21.15.90:0/3492691419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7105512f0 0x55f710925b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:12.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.588+0000 7f86267fc640 0 client.25756 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:12.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.588+0000 7f864bf92740 1 -- 172.21.15.90:0/3492691419 >> 172.21.15.90:0/3492691419 conn(0x55f71056e300 msgr2=0x55f71091b810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:12.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.588+0000 7f864bf92740 1 -- 172.21.15.90:0/3492691419 shutdown_connections 2024-04-25T11:38:12.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.588+0000 7f864bf92740 1 -- 172.21.15.90:0/3492691419 wait complete. 2024-04-25T11:38:12.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.588+0000 7f864bf92740 20 client.25756 trim_cache size 0 max 16384 2024-04-25T11:38:12.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.614+0000 7f864bf92740 1 Processor -- start 2024-04-25T11:38:12.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.615+0000 7f864bf92740 1 -- start start 2024-04-25T11:38:12.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.616+0000 7f864bf92740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7105512f0 0x55f71091beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:12.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.616+0000 7f864bf92740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7105594d0 0x55f71093bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:12.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.616+0000 7f864bf92740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f71057a1b0 0x55f71093c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:12.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.617+0000 7f863f59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f71057a1b0 0x55f71093c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:12.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.617+0000 7f863f59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f71057a1b0 0x55f71093c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39438/0 (socket says 172.21.15.90:39438) 2024-04-25T11:38:12.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.617+0000 7f863ed9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7105512f0 0x55f71091beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:12.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.617+0000 7f863ed9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7105512f0 0x55f71091beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:57824/0 (socket says 172.21.15.90:57824) 2024-04-25T11:38:12.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.617+0000 7f863f59e640 1 -- 172.21.15.90:0/3681875397 learned_addr learned my addr 172.21.15.90:0/3681875397 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:12.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.617+0000 7f864bf92740 1 -- 172.21.15.90:0/3681875397 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f86401b1e00 con 0x55f71057a1b0 2024-04-25T11:38:12.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.617+0000 7f864bf92740 1 -- 172.21.15.90:0/3681875397 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f86401b3970 con 0x55f7105512f0 2024-04-25T11:38:12.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.617+0000 7f864bf92740 1 -- 172.21.15.90:0/3681875397 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f86401b24d0 con 0x55f7105594d0 2024-04-25T11:38:12.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.618+0000 7f863f59e640 1 -- 172.21.15.90:0/3681875397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7105594d0 msgr2=0x55f71093bfb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:38:12.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.618+0000 7f863f59e640 1 --2- 172.21.15.90:0/3681875397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7105594d0 0x55f71093bfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:12.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.618+0000 7f863f59e640 1 -- 172.21.15.90:0/3681875397 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7105512f0 msgr2=0x55f71091beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:12.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.618+0000 7f863f59e640 1 --2- 172.21.15.90:0/3681875397 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7105512f0 0x55f71091beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:12.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.618+0000 7f863f59e640 1 -- 172.21.15.90:0/3681875397 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f710930e80 con 0x55f71057a1b0 2024-04-25T11:38:12.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.620+0000 7f863f59e640 1 --2- 172.21.15.90:0/3681875397 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f71057a1b0 0x55f71093c4f0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f863801cfd0 tx=0x7f8638010f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d18919bdb9d3a2e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:12.609 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.621+0000 7f8634ff9640 1 -- 172.21.15.90:0/3681875397 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8638010ac0 con 0x55f71057a1b0 2024-04-25T11:38:12.609 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.621+0000 7f8634ff9640 1 -- 172.21.15.90:0/3681875397 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8638027600 con 0x55f71057a1b0 2024-04-25T11:38:12.609 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.621+0000 7f8634ff9640 1 -- 172.21.15.90:0/3681875397 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863801ca30 con 0x55f71057a1b0 2024-04-25T11:38:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.623+0000 7f864bf92740 1 -- 172.21.15.90:0/3681875397 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f71057a1b0 msgr2=0x55f71093c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.623+0000 7f864bf92740 1 --2- 172.21.15.90:0/3681875397 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f71057a1b0 0x55f71093c4f0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f863801cfd0 tx=0x7f8638010f90 comp rx=0 tx=0).stop 2024-04-25T11:38:12.623 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.634+0000 7f864bf92740 1 -- 172.21.15.90:0/3681875397 shutdown_connections 2024-04-25T11:38:12.623 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.634+0000 7f864bf92740 1 --2- 172.21.15.90:0/3681875397 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f71057a1b0 0x55f71093c4f0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:12.624 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.634+0000 7f864bf92740 1 --2- 172.21.15.90:0/3681875397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7105594d0 0x55f71093bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:12.624 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.634+0000 7f864bf92740 1 --2- 172.21.15.90:0/3681875397 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f7105512f0 0x55f71091beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:12.624 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.634+0000 7f864bf92740 1 -- 172.21.15.90:0/3681875397 >> 172.21.15.90:0/3681875397 conn(0x55f71056e300 msgr2=0x55f71091b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:12.630 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.640+0000 7f864bf92740 1 -- 172.21.15.90:0/3681875397 shutdown_connections 2024-04-25T11:38:12.630 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.642+0000 7f864bf92740 1 -- 172.21.15.90:0/3681875397 wait complete. 2024-04-25T11:38:12.632 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.643+0000 7f864bf92740 1 -- start start 2024-04-25T11:38:12.632 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.643+0000 7f864bf92740 1 Processor -- start 2024-04-25T11:38:12.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.645+0000 7f864bf92740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7105512f0 0x55f71092c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:12.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.645+0000 7f864bf92740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7105594d0 0x55f710929820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:12.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.645+0000 7f864bf92740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f71057a1b0 0x55f710929d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:12.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.645+0000 7f864bf92740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f86401b3270 con 0x55f7105512f0 2024-04-25T11:38:12.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.645+0000 7f864bf92740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f86401b2ba0 con 0x55f71057a1b0 2024-04-25T11:38:12.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.645+0000 7f864bf92740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f86401afde0 con 0x55f7105594d0 2024-04-25T11:38:12.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.645+0000 7f864bf92740 10 client.0 _ll_register_callbacks cb 0x7f8638009c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:12.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.649+0000 7f863ed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7105512f0 0x55f71092c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:12.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.649+0000 7f863ed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7105512f0 0x55f71092c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39448/0 (socket says 172.21.15.90:39448) 2024-04-25T11:38:12.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.649+0000 7f863ed9d640 1 -- 172.21.15.90:0/1164936273 learned_addr learned my addr 172.21.15.90:0/1164936273 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:12.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.650+0000 7f863ed9d640 1 -- 172.21.15.90:0/1164936273 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7105594d0 msgr2=0x55f710929820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:38:12.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.650+0000 7f863ed9d640 1 --2- 172.21.15.90:0/1164936273 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7105594d0 0x55f710929820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:12.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.650+0000 7f863ed9d640 1 -- 172.21.15.90:0/1164936273 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f71057a1b0 msgr2=0x55f710929d60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:38:12.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.650+0000 7f863ed9d640 1 --2- 172.21.15.90:0/1164936273 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f71057a1b0 0x55f710929d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:12.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.650+0000 7f863ed9d640 1 -- 172.21.15.90:0/1164936273 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f71092a400 con 0x55f7105512f0 2024-04-25T11:38:12.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.651+0000 7f863ed9d640 1 --2- 172.21.15.90:0/1164936273 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7105512f0 0x55f71092c4b0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f8630009900 tx=0x7f8630005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e732319d03e96dab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:12.643 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.655+0000 7f86267fc640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:38:12.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.658+0000 7f86267fc640 1 -- 172.21.15.90:0/1164936273 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8630040040 con 0x55f7105512f0 2024-04-25T11:38:12.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.658+0000 7f864bf92740 1 -- 172.21.15.90:0/1164936273 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8638027600 con 0x55f7105512f0 2024-04-25T11:38:12.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.658+0000 7f86267fc640 1 -- 172.21.15.90:0/1164936273 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f86300318b0 con 0x55f7105512f0 2024-04-25T11:38:12.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.659+0000 7f86267fc640 1 -- 172.21.15.90:0/1164936273 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f863005e050 con 0x55f7105512f0 2024-04-25T11:38:12.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.664+0000 7f864bf92740 10 client.16554 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:12.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.664+0000 7f864bf92740 1 -- 172.21.15.90:0/1164936273 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f7109a71e0 con 0x55f7105512f0 2024-04-25T11:38:12.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.665+0000 7f86267fc640 1 -- 172.21.15.90:0/1164936273 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8638027600 con 0x55f7105512f0 2024-04-25T11:38:12.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.666+0000 7f86267fc640 1 -- 172.21.15.90:0/1164936273 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f8630050420 con 0x55f7105512f0 2024-04-25T11:38:12.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.666+0000 7f864bf92740 10 client.16554 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:12.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.666+0000 7f864bf92740 10 client.16554 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:38:12.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.666+0000 7f864bf92740 20 client.16554 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:12.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.666+0000 7f864bf92740 1 --2- 172.21.15.90:0/1164936273 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f71050e1c0 0x55f7109a9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:12.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.666+0000 7f864bf92740 4 client.16554 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:38:12.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.666+0000 7f864bf92740 1 -- 172.21.15.90:0/1164936273 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f710887ee0 con 0x55f71050e1c0 2024-04-25T11:38:12.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.667+0000 7f8637fff640 1 --2- 172.21.15.90:0/1164936273 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f71050e1c0 0x55f7109a9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:12.658 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.668+0000 7f86267fc640 10 client.16554 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:12.659 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:12.669+0000 7f8637fff640 1 --2- 172.21.15.90:0/1164936273 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f71050e1c0 0x55f7109a9610 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc2f1ad7a05d6ab9 server_cookie=c6ea432fe01c16c1 in_seq=0 out_seq=0 2024-04-25T11:38:13.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:13 smithi090 ceph-mon[29575]: pgmap v97: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 657 KiB/s rd, 435 MiB/s wr, 1.77k op/s 2024-04-25T11:38:14.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:13 smithi138 ceph-mon[32328]: pgmap v97: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 657 KiB/s rd, 435 MiB/s wr, 1.77k op/s 2024-04-25T11:38:14.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:13 smithi156 ceph-mon[33045]: pgmap v97: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 657 KiB/s rd, 435 MiB/s wr, 1.77k op/s 2024-04-25T11:38:16.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:15 smithi138 ceph-mon[32328]: pgmap v98: 97 pgs: 97 active+clean; 22 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 472 MiB/s wr, 2.00k op/s 2024-04-25T11:38:16.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:15 smithi156 ceph-mon[33045]: pgmap v98: 97 pgs: 97 active+clean; 22 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 472 MiB/s wr, 2.00k op/s 2024-04-25T11:38:16.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:15 smithi090 ceph-mon[29575]: pgmap v98: 97 pgs: 97 active+clean; 22 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 472 MiB/s wr, 2.00k op/s 2024-04-25T11:38:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.841+0000 7f86267fc640 1 -- 172.21.15.90:0/1164936273 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f710887ee0 con 0x55f71050e1c0 2024-04-25T11:38:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.841+0000 7f86267fc640 10 client.16554 handle_command_reply: tid=0 2024-04-25T11:38:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.841+0000 7f864bf92740 1 client.16554 shutdown 2024-04-25T11:38:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.841+0000 7f864bf92740 1 -- 172.21.15.90:0/1164936273 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7105512f0 msgr2=0x55f71092c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.841+0000 7f864bf92740 1 --2- 172.21.15.90:0/1164936273 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7105512f0 0x55f71092c4b0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f8630009900 tx=0x7f8630005db0 comp rx=0 tx=0).stop 2024-04-25T11:38:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.842+0000 7f864bf92740 1 -- 172.21.15.90:0/1164936273 shutdown_connections 2024-04-25T11:38:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.842+0000 7f864bf92740 1 --2- 172.21.15.90:0/1164936273 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f71050e1c0 0x55f7109a9610 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.842+0000 7f864bf92740 1 --2- 172.21.15.90:0/1164936273 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f71057a1b0 0x55f710929d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.842+0000 7f864bf92740 1 --2- 172.21.15.90:0/1164936273 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f7105594d0 0x55f710929820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.842+0000 7f864bf92740 1 --2- 172.21.15.90:0/1164936273 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f7105512f0 0x55f71092c4b0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.842+0000 7f864bf92740 1 -- 172.21.15.90:0/1164936273 >> 172.21.15.90:0/1164936273 conn(0x55f71056e300 msgr2=0x55f710923530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.842+0000 7f86267fc640 0 client.16554 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.842+0000 7f864bf92740 1 -- 172.21.15.90:0/1164936273 shutdown_connections 2024-04-25T11:38:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.842+0000 7f864bf92740 1 -- 172.21.15.90:0/1164936273 wait complete. 2024-04-25T11:38:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.842+0000 7f864bf92740 20 client.16554 trim_cache size 0 max 16384 2024-04-25T11:38:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:38:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:38:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:38:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:38:16.832 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.844+0000 7f86450e0640 1 -- 172.21.15.90:0/3944308565 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f862c0afcd0 msgr2=0x7f862c0b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:16.832 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.844+0000 7f86450e0640 1 --2- 172.21.15.90:0/3944308565 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f862c0afcd0 0x7f862c0b21e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8638009fd0 tx=0x7f8638004a40 comp rx=0 tx=0).stop 2024-04-25T11:38:16.832 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.844+0000 7f86450e0640 1 -- 172.21.15.90:0/3944308565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8640108a80 msgr2=0x7f8640100270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:16.832 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.844+0000 7f86450e0640 1 --2- 172.21.15.90:0/3944308565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8640108a80 0x7f8640100270 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f864006a310 tx=0x7f862800ee50 comp rx=0 tx=0).stop 2024-04-25T11:38:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.845+0000 7f86450e0640 1 -- 172.21.15.90:0/3944308565 shutdown_connections 2024-04-25T11:38:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.845+0000 7f86450e0640 1 --2- 172.21.15.90:0/3944308565 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f862c0afcd0 0x7f862c0b21e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.845+0000 7f86450e0640 1 --2- 172.21.15.90:0/3944308565 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8640108f20 0x7f86401007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.845+0000 7f86450e0640 1 --2- 172.21.15.90:0/3944308565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8640108a80 0x7f8640100270 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.845+0000 7f86450e0640 1 --2- 172.21.15.90:0/3944308565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86401031d0 0x7f86400ffd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.845+0000 7f86450e0640 1 -- 172.21.15.90:0/3944308565 >> 172.21.15.90:0/3944308565 conn(0x7f8640078150 msgr2=0x7f864006b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.846+0000 7f86450e0640 1 -- 172.21.15.90:0/3944308565 shutdown_connections 2024-04-25T11:38:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:16.846+0000 7f86450e0640 1 -- 172.21.15.90:0/3944308565 wait complete. 2024-04-25T11:38:18.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:17 smithi138 ceph-mon[32328]: pgmap v99: 97 pgs: 97 active+clean; 22 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 554 MiB/s wr, 2.59k op/s 2024-04-25T11:38:18.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:17 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:38:18.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:17 smithi156 ceph-mon[33045]: pgmap v99: 97 pgs: 97 active+clean; 22 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 554 MiB/s wr, 2.59k op/s 2024-04-25T11:38:18.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:17 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:38:18.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:17 smithi090 ceph-mon[29575]: pgmap v99: 97 pgs: 97 active+clean; 22 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 554 MiB/s wr, 2.59k op/s 2024-04-25T11:38:18.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:17 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:38:19.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:18 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:38:19.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:18 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:38:19.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:18 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:38:19.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:18 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:38:19.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:18 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:38:19.087 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:18 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:38:19.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:18 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:38:19.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:18 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:38:19.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:18 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:38:20.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:19 smithi090 ceph-mon[29575]: pgmap v100: 97 pgs: 97 active+clean; 22 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 1.0 MiB/s rd, 391 MiB/s wr, 2.34k op/s 2024-04-25T11:38:20.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:19 smithi138 ceph-mon[32328]: pgmap v100: 97 pgs: 97 active+clean; 22 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 1.0 MiB/s rd, 391 MiB/s wr, 2.34k op/s 2024-04-25T11:38:20.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:19 smithi156 ceph-mon[33045]: pgmap v100: 97 pgs: 97 active+clean; 22 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 1.0 MiB/s rd, 391 MiB/s wr, 2.34k op/s 2024-04-25T11:38:22.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:21 smithi090 ceph-mon[29575]: pgmap v101: 97 pgs: 97 active+clean; 22 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 538 MiB/s wr, 3.56k op/s 2024-04-25T11:38:22.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:21 smithi138 ceph-mon[32328]: pgmap v101: 97 pgs: 97 active+clean; 22 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 538 MiB/s wr, 3.56k op/s 2024-04-25T11:38:22.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:21 smithi156 ceph-mon[33045]: pgmap v101: 97 pgs: 97 active+clean; 22 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 538 MiB/s wr, 3.56k op/s 2024-04-25T11:38:24.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:23 smithi090 ceph-mon[29575]: pgmap v102: 97 pgs: 97 active+clean; 22 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 310 MiB/s wr, 2.08k op/s 2024-04-25T11:38:24.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:23 smithi138 ceph-mon[32328]: pgmap v102: 97 pgs: 97 active+clean; 22 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 310 MiB/s wr, 2.08k op/s 2024-04-25T11:38:24.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:23 smithi156 ceph-mon[33045]: pgmap v102: 97 pgs: 97 active+clean; 22 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 310 MiB/s wr, 2.08k op/s 2024-04-25T11:38:26.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:25 smithi090 ceph-mon[29575]: pgmap v103: 97 pgs: 97 active+clean; 23 GiB data, 70 GiB used, 1003 GiB / 1.0 TiB avail; 1.3 MiB/s rd, 403 MiB/s wr, 2.12k op/s 2024-04-25T11:38:26.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:25 smithi138 ceph-mon[32328]: pgmap v103: 97 pgs: 97 active+clean; 23 GiB data, 70 GiB used, 1003 GiB / 1.0 TiB avail; 1.3 MiB/s rd, 403 MiB/s wr, 2.12k op/s 2024-04-25T11:38:26.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:25 smithi156 ceph-mon[33045]: pgmap v103: 97 pgs: 97 active+clean; 23 GiB data, 70 GiB used, 1003 GiB / 1.0 TiB avail; 1.3 MiB/s rd, 403 MiB/s wr, 2.12k op/s 2024-04-25T11:38:28.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:28 smithi138 ceph-mon[32328]: pgmap v104: 97 pgs: 97 active+clean; 24 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 531 MiB/s wr, 1.96k op/s 2024-04-25T11:38:28.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:27 smithi156 ceph-mon[33045]: pgmap v104: 97 pgs: 97 active+clean; 24 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 531 MiB/s wr, 1.96k op/s 2024-04-25T11:38:28.477 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:27 smithi090 ceph-mon[29575]: pgmap v104: 97 pgs: 97 active+clean; 24 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 531 MiB/s wr, 1.96k op/s 2024-04-25T11:38:30.159 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.163+0000 7f60d359e640 1 -- 172.21.15.90:0/1999404034 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f60d410f8e0 msgr2=0x7f60d410fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:30.159 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.163+0000 7f60d359e640 1 --2- 172.21.15.90:0/1999404034 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f60d410f8e0 0x7f60d410fcf0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f60c8009900 tx=0x7f60c802a410 comp rx=0 tx=0).stop 2024-04-25T11:38:30.160 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.171+0000 7f60d359e640 1 -- 172.21.15.90:0/1999404034 shutdown_connections 2024-04-25T11:38:30.160 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.171+0000 7f60d359e640 1 --2- 172.21.15.90:0/1999404034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60d410a0e0 0x7f60d410a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:30.161 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.171+0000 7f60d359e640 1 --2- 172.21.15.90:0/1999404034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60d41102c0 0x7f60d4109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:30.161 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.171+0000 7f60d359e640 1 --2- 172.21.15.90:0/1999404034 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f60d410f8e0 0x7f60d410fcf0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:30.161 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.171+0000 7f60d359e640 1 -- 172.21.15.90:0/1999404034 >> 172.21.15.90:0/1999404034 conn(0x7f60d406f6e0 msgr2=0x7f60d406faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:30.161 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.172+0000 7f60d359e640 1 -- 172.21.15.90:0/1999404034 shutdown_connections 2024-04-25T11:38:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.173+0000 7f60d359e640 1 -- 172.21.15.90:0/1999404034 wait complete. 2024-04-25T11:38:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.175+0000 7f60d359e640 1 Processor -- start 2024-04-25T11:38:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.175+0000 7f60d359e640 1 -- start start 2024-04-25T11:38:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.176+0000 7f60d359e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f60d410a0e0 0x7f60d41201c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.176+0000 7f60d359e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60d410f8e0 0x7f60d4119330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:30.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.177+0000 7f60d359e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60d41102c0 0x7f60d4119960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:30.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.177+0000 7f60d359e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f60d4122990 con 0x7f60d41102c0 2024-04-25T11:38:30.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.177+0000 7f60d359e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f60d4122810 con 0x7f60d410a0e0 2024-04-25T11:38:30.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.177+0000 7f60d359e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f60d4122b10 con 0x7f60d410f8e0 2024-04-25T11:38:30.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.178+0000 7f60d2d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60d41102c0 0x7f60d4119960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:30.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.178+0000 7f60d2d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60d41102c0 0x7f60d4119960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54940/0 (socket says 172.21.15.90:54940) 2024-04-25T11:38:30.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.178+0000 7f60d2d9d640 1 -- 172.21.15.90:0/1446154786 learned_addr learned my addr 172.21.15.90:0/1446154786 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:30.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.178+0000 7f60d1d9b640 1 --2- 172.21.15.90:0/1446154786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60d410f8e0 0x7f60d4119330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:30.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.179+0000 7f60d259c640 1 --2- 172.21.15.90:0/1446154786 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f60d410a0e0 0x7f60d41201c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:30.172 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.182+0000 7f60d1d9b640 1 -- 172.21.15.90:0/1446154786 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f60d410a0e0 msgr2=0x7f60d41201c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:30.172 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.182+0000 7f60d1d9b640 1 --2- 172.21.15.90:0/1446154786 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f60d410a0e0 0x7f60d41201c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:30.172 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.182+0000 7f60d1d9b640 1 -- 172.21.15.90:0/1446154786 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60d41102c0 msgr2=0x7f60d4119960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:30.173 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.182+0000 7f60d1d9b640 1 --2- 172.21.15.90:0/1446154786 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60d41102c0 0x7f60d4119960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:30.173 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.182+0000 7f60d1d9b640 1 -- 172.21.15.90:0/1446154786 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60d411dce0 con 0x7f60d410f8e0 2024-04-25T11:38:30.178 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.190+0000 7f60d1d9b640 1 --2- 172.21.15.90:0/1446154786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60d410f8e0 0x7f60d4119330 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f60c400e9c0 tx=0x7f60c400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:30.179 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.191+0000 7f60c37fe640 1 -- 172.21.15.90:0/1446154786 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c400ccc0 con 0x7f60d410f8e0 2024-04-25T11:38:30.179 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.191+0000 7f60c37fe640 1 -- 172.21.15.90:0/1446154786 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60c4005020 con 0x7f60d410f8e0 2024-04-25T11:38:30.179 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.191+0000 7f60d359e640 1 -- 172.21.15.90:0/1446154786 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60d411df70 con 0x7f60d410f8e0 2024-04-25T11:38:30.179 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.192+0000 7f60d359e640 1 -- 172.21.15.90:0/1446154786 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60d411e460 con 0x7f60d410f8e0 2024-04-25T11:38:30.183 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.193+0000 7f60c37fe640 1 -- 172.21.15.90:0/1446154786 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c4004a00 con 0x7f60d410f8e0 2024-04-25T11:38:30.186 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.198+0000 7f60df9bf740 1 Processor -- start 2024-04-25T11:38:30.188 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.199+0000 7f60df9bf740 1 -- start start 2024-04-25T11:38:30.188 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.200+0000 7f60df9bf740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb4d6361b0 0x55cb4d9d5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:30.188 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.200+0000 7f60df9bf740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cb4d60d2f0 0x55cb4d9d61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:30.188 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.200+0000 7f60df9bf740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb4d6154d0 0x55cb4d9d6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:30.188 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.200+0000 7f60df9bf740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55cb4d93c460 con 0x55cb4d6154d0 2024-04-25T11:38:30.188 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.200+0000 7f60df9bf740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55cb4d8f6590 con 0x55cb4d60d2f0 2024-04-25T11:38:30.188 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.200+0000 7f60df9bf740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55cb4d938420 con 0x55cb4d6361b0 2024-04-25T11:38:30.191 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.202+0000 7f60d259c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cb4d60d2f0 0x55cb4d9d61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.202+0000 7f60c37fe640 1 -- 172.21.15.90:0/1446154786 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f60c40051d0 con 0x7f60d410f8e0 2024-04-25T11:38:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.203+0000 7f60d259c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cb4d60d2f0 0x55cb4d9d61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:57358/0 (socket says 172.21.15.90:57358) 2024-04-25T11:38:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.203+0000 7f60d259c640 1 -- 172.21.15.90:0/693707355 learned_addr learned my addr 172.21.15.90:0/693707355 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.204+0000 7f60d2d9d640 1 --2- 172.21.15.90:0/693707355 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb4d6361b0 0x55cb4d9d5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.204+0000 7f60c37fe640 1 --2- 172.21.15.90:0/1446154786 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f60bc0afde0 0x7f60bc0b22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:30.194 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.205+0000 7f60c37fe640 1 -- 172.21.15.90:0/1446154786 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f60c40a3790 con 0x7f60d410f8e0 2024-04-25T11:38:30.195 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.206+0000 7f60d259c640 1 -- 172.21.15.90:0/693707355 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb4d6361b0 msgr2=0x55cb4d9d5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:30.195 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.206+0000 7f60d259c640 1 --2- 172.21.15.90:0/693707355 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb4d6361b0 0x55cb4d9d5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:30.195 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.206+0000 7f60d259c640 1 -- 172.21.15.90:0/693707355 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb4d6154d0 msgr2=0x55cb4d9d6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:30.195 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.206+0000 7f60d259c640 1 --2- 172.21.15.90:0/693707355 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb4d6154d0 0x55cb4d9d6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:30.195 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.206+0000 7f60d259c640 1 -- 172.21.15.90:0/693707355 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb4d9030d0 con 0x55cb4d60d2f0 2024-04-25T11:38:30.196 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.207+0000 7f60d2d9d640 1 --2- 172.21.15.90:0/1446154786 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f60bc0afde0 0x7f60bc0b22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:30.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.210+0000 7f60d259c640 1 --2- 172.21.15.90:0/693707355 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cb4d60d2f0 0x55cb4d9d61d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f60c8009230 tx=0x7f60c80040b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e077756a83a128ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:30.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.210+0000 7f60d2d9d640 1 --2- 172.21.15.90:0/1446154786 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f60bc0afde0 0x7f60bc0b22f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x55cb4d9d2e00 tx=0x7f60cc004ab0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:30.203 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.215+0000 7f60c0ff9640 1 -- 172.21.15.90:0/693707355 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c8035030 con 0x55cb4d60d2f0 2024-04-25T11:38:30.203 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.215+0000 7f60c0ff9640 1 -- 172.21.15.90:0/693707355 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60c80054f0 con 0x55cb4d60d2f0 2024-04-25T11:38:30.205 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.217+0000 7f60df9bf740 1 -- 172.21.15.90:0/693707355 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cb4d60d2f0 msgr2=0x55cb4d9d61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:30.205 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.217+0000 7f60df9bf740 1 --2- 172.21.15.90:0/693707355 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cb4d60d2f0 0x55cb4d9d61d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f60c8009230 tx=0x7f60c80040b0 comp rx=0 tx=0).stop 2024-04-25T11:38:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.221+0000 7f60df9bf740 1 -- 172.21.15.90:0/693707355 shutdown_connections 2024-04-25T11:38:30.211 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.221+0000 7f60df9bf740 1 --2- 172.21.15.90:0/693707355 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb4d6154d0 0x55cb4d9d6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:30.211 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.221+0000 7f60df9bf740 1 --2- 172.21.15.90:0/693707355 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cb4d60d2f0 0x55cb4d9d61d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:30.211 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.221+0000 7f60df9bf740 1 --2- 172.21.15.90:0/693707355 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb4d6361b0 0x55cb4d9d5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:30.211 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.221+0000 7f60df9bf740 1 -- 172.21.15.90:0/693707355 >> 172.21.15.90:0/693707355 conn(0x55cb4d62a300 msgr2=0x55cb4d9d2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:30.216 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.226+0000 7f60df9bf740 1 -- 172.21.15.90:0/693707355 shutdown_connections 2024-04-25T11:38:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.226+0000 7f60df9bf740 1 -- 172.21.15.90:0/693707355 wait complete. 2024-04-25T11:38:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.227+0000 7f60df9bf740 1 -- start start 2024-04-25T11:38:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.227+0000 7f60df9bf740 1 Processor -- start 2024-04-25T11:38:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.228+0000 7f60df9bf740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb4d60d2f0 0x55cb4d9e1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.228+0000 7f60df9bf740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cb4d6154d0 0x55cb4d9e22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:30.219 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.228+0000 7f60df9bf740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb4d6361b0 0x55cb4d9e01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:30.219 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.230+0000 7f60d2d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb4d6361b0 0x55cb4d9e01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.230+0000 7f60d2d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb4d6361b0 0x55cb4d9e01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54962/0 (socket says 172.21.15.90:54962) 2024-04-25T11:38:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.230+0000 7f60d2d9d640 1 -- 172.21.15.90:0/1515721197 learned_addr learned my addr 172.21.15.90:0/1515721197 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.230+0000 7f60df9bf740 1 -- 172.21.15.90:0/1515721197 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55cb4d9e2db0 con 0x55cb4d6361b0 2024-04-25T11:38:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.230+0000 7f60df9bf740 1 -- 172.21.15.90:0/1515721197 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55cb4d8c3b60 con 0x55cb4d6154d0 2024-04-25T11:38:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.230+0000 7f60df9bf740 1 -- 172.21.15.90:0/1515721197 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55cb4d9e2f30 con 0x55cb4d60d2f0 2024-04-25T11:38:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.230+0000 7f60df9bf740 10 client.0 _ll_register_callbacks cb 0x55cb4d8e82b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.231+0000 7f60d259c640 1 --2- 172.21.15.90:0/1515721197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb4d60d2f0 0x55cb4d9e1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.231+0000 7f60d1d9b640 1 --2- 172.21.15.90:0/1515721197 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cb4d6154d0 0x55cb4d9e22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.232+0000 7f60d2d9d640 1 -- 172.21.15.90:0/1515721197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb4d60d2f0 msgr2=0x55cb4d9e1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.232+0000 7f60d2d9d640 1 --2- 172.21.15.90:0/1515721197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb4d60d2f0 0x55cb4d9e1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.232+0000 7f60d2d9d640 1 -- 172.21.15.90:0/1515721197 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cb4d6154d0 msgr2=0x55cb4d9e22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.232+0000 7f60d2d9d640 1 --2- 172.21.15.90:0/1515721197 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cb4d6154d0 0x55cb4d9e22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.232+0000 7f60d2d9d640 1 -- 172.21.15.90:0/1515721197 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb4d64e210 con 0x55cb4d6361b0 2024-04-25T11:38:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.232+0000 7f60d2d9d640 1 --2- 172.21.15.90:0/1515721197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb4d6361b0 0x55cb4d9e01c0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00db30 tx=0x7f60cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55da333679490fea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.232+0000 7f60b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:38:30.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.239+0000 7f60b9ffb640 1 -- 172.21.15.90:0/1515721197 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc025070 con 0x55cb4d6361b0 2024-04-25T11:38:30.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.239+0000 7f60df9bf740 1 -- 172.21.15.90:0/1515721197 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60c80054f0 con 0x55cb4d6361b0 2024-04-25T11:38:30.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.239+0000 7f60b9ffb640 1 -- 172.21.15.90:0/1515721197 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f60cc00f420 con 0x55cb4d6361b0 2024-04-25T11:38:30.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.240+0000 7f60b9ffb640 1 -- 172.21.15.90:0/1515721197 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f60cc031050 con 0x55cb4d6361b0 2024-04-25T11:38:30.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.240+0000 7f60df9bf740 10 client.16563 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:30.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.240+0000 7f60df9bf740 1 -- 172.21.15.90:0/1515721197 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cb4d71a620 con 0x55cb4d6361b0 2024-04-25T11:38:30.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.242+0000 7f60b9ffb640 1 -- 172.21.15.90:0/1515721197 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60c80054f0 con 0x55cb4d6361b0 2024-04-25T11:38:30.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.245+0000 7f60b9ffb640 1 -- 172.21.15.90:0/1515721197 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f60cc01d070 con 0x55cb4d6361b0 2024-04-25T11:38:30.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.245+0000 7f60df9bf740 10 client.16563 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:30.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.245+0000 7f60df9bf740 10 client.16563 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:38:30.236 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.246+0000 7f60df9bf740 20 client.16563 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:30.236 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.246+0000 7f60df9bf740 1 --2- 172.21.15.90:0/1515721197 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55cb4d5f4d80 0x55cb4d9f2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:30.236 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.246+0000 7f60df9bf740 4 client.16563 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:38:30.236 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.246+0000 7f60df9bf740 1 -- 172.21.15.90:0/1515721197 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cb4d94a070 con 0x55cb4d5f4d80 2024-04-25T11:38:30.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.254+0000 7f60d259c640 1 --2- 172.21.15.90:0/1515721197 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55cb4d5f4d80 0x55cb4d9f2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:30.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.272+0000 7f60b9ffb640 10 client.16563 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:30.263 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.272+0000 7f60d259c640 1 --2- 172.21.15.90:0/1515721197 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55cb4d5f4d80 0x55cb4d9f2850 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4d57de56526500e server_cookie=928edf6c79a37cbd in_seq=0 out_seq=0 2024-04-25T11:38:30.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.279+0000 7f60b9ffb640 1 -- 172.21.15.90:0/1515721197 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55cb4d94a070 con 0x55cb4d5f4d80 2024-04-25T11:38:30.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.279+0000 7f60b9ffb640 10 client.16563 handle_command_reply: tid=0 2024-04-25T11:38:30.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.279+0000 7f60df9bf740 1 client.16563 shutdown 2024-04-25T11:38:30.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.279+0000 7f60df9bf740 1 -- 172.21.15.90:0/1515721197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb4d6361b0 msgr2=0x55cb4d9e01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:30.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.279+0000 7f60df9bf740 1 --2- 172.21.15.90:0/1515721197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb4d6361b0 0x55cb4d9e01c0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00db30 tx=0x7f60cc004a40 comp rx=0 tx=0).stop 2024-04-25T11:38:30.270 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.280+0000 7f60df9bf740 1 -- 172.21.15.90:0/1515721197 shutdown_connections 2024-04-25T11:38:30.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.280+0000 7f60df9bf740 1 --2- 172.21.15.90:0/1515721197 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55cb4d5f4d80 0x55cb4d9f2850 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:30.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.280+0000 7f60df9bf740 1 --2- 172.21.15.90:0/1515721197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb4d6361b0 0x55cb4d9e01c0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:30.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.280+0000 7f60df9bf740 1 --2- 172.21.15.90:0/1515721197 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cb4d6154d0 0x55cb4d9e22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:30.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.280+0000 7f60df9bf740 1 --2- 172.21.15.90:0/1515721197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb4d60d2f0 0x55cb4d9e1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:30.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.280+0000 7f60df9bf740 1 -- 172.21.15.90:0/1515721197 >> 172.21.15.90:0/1515721197 conn(0x55cb4d62a300 msgr2=0x55cb4d9d79b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:30.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.280+0000 7f60b9ffb640 0 client.16563 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:30.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.280+0000 7f60df9bf740 1 -- 172.21.15.90:0/1515721197 shutdown_connections 2024-04-25T11:38:30.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.280+0000 7f60df9bf740 1 -- 172.21.15.90:0/1515721197 wait complete. 2024-04-25T11:38:30.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.280+0000 7f60df9bf740 20 client.16563 trim_cache size 0 max 16384 2024-04-25T11:38:30.293 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.305+0000 7f60df9bf740 1 Processor -- start 2024-04-25T11:38:30.293 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.305+0000 7f60df9bf740 1 -- start start 2024-04-25T11:38:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.306+0000 7f60df9bf740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb4d60d2f0 0x55cb4d9e0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.306+0000 7f60df9bf740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cb4d6154d0 0x55cb4d9f8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.306+0000 7f60df9bf740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb4d6361b0 0x55cb4d9f8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.307+0000 7f60df9bf740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f60d41c6e90 con 0x55cb4d6361b0 2024-04-25T11:38:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.307+0000 7f60d259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb4d60d2f0 0x55cb4d9e0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.307+0000 7f60d259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb4d60d2f0 0x55cb4d9e0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:40686/0 (socket says 172.21.15.90:40686) 2024-04-25T11:38:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.307+0000 7f60d259c640 1 -- 172.21.15.90:0/4092708246 learned_addr learned my addr 172.21.15.90:0/4092708246 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.307+0000 7f60d1d9b640 1 --2- 172.21.15.90:0/4092708246 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cb4d6154d0 0x55cb4d9f8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.307+0000 7f60df9bf740 1 -- 172.21.15.90:0/4092708246 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f60d41c8a00 con 0x55cb4d6154d0 2024-04-25T11:38:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.307+0000 7f60df9bf740 1 -- 172.21.15.90:0/4092708246 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f60d41c7560 con 0x55cb4d60d2f0 2024-04-25T11:38:30.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.307+0000 7f60d2d9d640 1 --2- 172.21.15.90:0/4092708246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb4d6361b0 0x55cb4d9f8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:30.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.308+0000 7f60d259c640 1 -- 172.21.15.90:0/4092708246 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cb4d6154d0 msgr2=0x55cb4d9f8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:30.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.308+0000 7f60d259c640 1 --2- 172.21.15.90:0/4092708246 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cb4d6154d0 0x55cb4d9f8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.308+0000 7f60d259c640 1 -- 172.21.15.90:0/4092708246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb4d6361b0 msgr2=0x55cb4d9f8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.308+0000 7f60d259c640 1 --2- 172.21.15.90:0/4092708246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb4d6361b0 0x55cb4d9f8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.308+0000 7f60d259c640 1 -- 172.21.15.90:0/4092708246 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb4d9df4c0 con 0x55cb4d60d2f0 2024-04-25T11:38:30.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.310+0000 7f60d259c640 1 --2- 172.21.15.90:0/4092708246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb4d60d2f0 0x55cb4d9e0160 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f60c8009850 tx=0x7f60c802a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df9e04dc002d7175 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:30.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.311+0000 7f60bbfff640 1 -- 172.21.15.90:0/4092708246 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c8035040 con 0x55cb4d60d2f0 2024-04-25T11:38:30.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.311+0000 7f60bbfff640 1 -- 172.21.15.90:0/4092708246 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60c8055070 con 0x55cb4d60d2f0 2024-04-25T11:38:30.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.311+0000 7f60bbfff640 1 -- 172.21.15.90:0/4092708246 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c8005110 con 0x55cb4d60d2f0 2024-04-25T11:38:30.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:30 smithi138 ceph-mon[32328]: pgmap v105: 97 pgs: 97 active+clean; 24 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 1.6 MiB/s rd, 406 MiB/s wr, 1.32k op/s 2024-04-25T11:38:30.302 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.313+0000 7f60df9bf740 1 -- 172.21.15.90:0/4092708246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb4d60d2f0 msgr2=0x55cb4d9e0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:30.302 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.313+0000 7f60df9bf740 1 --2- 172.21.15.90:0/4092708246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb4d60d2f0 0x55cb4d9e0160 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f60c8009850 tx=0x7f60c802a410 comp rx=0 tx=0).stop 2024-04-25T11:38:30.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.317+0000 7f60df9bf740 1 -- 172.21.15.90:0/4092708246 shutdown_connections 2024-04-25T11:38:30.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.317+0000 7f60df9bf740 1 --2- 172.21.15.90:0/4092708246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb4d6361b0 0x55cb4d9f8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:30.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.317+0000 7f60df9bf740 1 --2- 172.21.15.90:0/4092708246 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cb4d6154d0 0x55cb4d9f8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:30.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.317+0000 7f60df9bf740 1 --2- 172.21.15.90:0/4092708246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb4d60d2f0 0x55cb4d9e0160 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:30.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.317+0000 7f60df9bf740 1 -- 172.21.15.90:0/4092708246 >> 172.21.15.90:0/4092708246 conn(0x55cb4d62a300 msgr2=0x55cb4d9dfa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:30.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.318+0000 7f60df9bf740 1 -- 172.21.15.90:0/4092708246 shutdown_connections 2024-04-25T11:38:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.318+0000 7f60df9bf740 1 -- 172.21.15.90:0/4092708246 wait complete. 2024-04-25T11:38:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.319+0000 7f60df9bf740 1 -- start start 2024-04-25T11:38:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.319+0000 7f60df9bf740 1 Processor -- start 2024-04-25T11:38:30.308 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.320+0000 7f60df9bf740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb4d60d2f0 0x55cb4d9e85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:30.308 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.320+0000 7f60df9bf740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cb4d6154d0 0x55cb4d9e5940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:30.308 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.320+0000 7f60df9bf740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb4d6361b0 0x55cb4d9e5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:30.308 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.320+0000 7f60df9bf740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f60d41c8300 con 0x55cb4d60d2f0 2024-04-25T11:38:30.308 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.320+0000 7f60df9bf740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f60d41c7c30 con 0x55cb4d6154d0 2024-04-25T11:38:30.309 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.320+0000 7f60df9bf740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f60d41c4e70 con 0x55cb4d6361b0 2024-04-25T11:38:30.309 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.320+0000 7f60df9bf740 10 client.0 _ll_register_callbacks cb 0x7f60cc013200 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.320+0000 7f60d259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb4d60d2f0 0x55cb4d9e85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.320+0000 7f60d1d9b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cb4d6154d0 0x55cb4d9e5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.320+0000 7f60d2d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb4d6361b0 0x55cb4d9e5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.321+0000 7f60d2d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb4d6361b0 0x55cb4d9e5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:40698/0 (socket says 172.21.15.90:40698) 2024-04-25T11:38:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.321+0000 7f60d1d9b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cb4d6154d0 0x55cb4d9e5940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:57376/0 (socket says 172.21.15.90:57376) 2024-04-25T11:38:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.321+0000 7f60d2d9d640 1 -- 172.21.15.90:0/1197094940 learned_addr learned my addr 172.21.15.90:0/1197094940 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.321+0000 7f60d2d9d640 1 -- 172.21.15.90:0/1197094940 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cb4d6154d0 msgr2=0x55cb4d9e5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.321+0000 7f60d2d9d640 1 --2- 172.21.15.90:0/1197094940 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cb4d6154d0 0x55cb4d9e5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:30.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.321+0000 7f60d2d9d640 1 -- 172.21.15.90:0/1197094940 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb4d60d2f0 msgr2=0x55cb4d9e85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:30.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.321+0000 7f60d2d9d640 1 --2- 172.21.15.90:0/1197094940 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb4d60d2f0 0x55cb4d9e85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:30.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.321+0000 7f60d2d9d640 1 -- 172.21.15.90:0/1197094940 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb4d9e63c0 con 0x55cb4d6361b0 2024-04-25T11:38:30.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.322+0000 7f60d2d9d640 1 --2- 172.21.15.90:0/1197094940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb4d6361b0 0x55cb4d9e5e80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00f650 tx=0x7f60cc00f680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b75fd9f50e6599d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:30.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.322+0000 7f60b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:38:30.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.322+0000 7f60b9ffb640 1 -- 172.21.15.90:0/1197094940 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc025040 con 0x55cb4d6361b0 2024-04-25T11:38:30.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.323+0000 7f60df9bf740 1 -- 172.21.15.90:0/1197094940 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60c8055070 con 0x55cb4d6361b0 2024-04-25T11:38:30.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.323+0000 7f60b9ffb640 1 -- 172.21.15.90:0/1197094940 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f60cc004b20 con 0x55cb4d6361b0 2024-04-25T11:38:30.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.323+0000 7f60b9ffb640 1 -- 172.21.15.90:0/1197094940 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f60cc043050 con 0x55cb4d6361b0 2024-04-25T11:38:30.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.324+0000 7f60df9bf740 10 client.25774 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:30.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.324+0000 7f60df9bf740 1 -- 172.21.15.90:0/1197094940 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cb4da5a8f0 con 0x55cb4d6361b0 2024-04-25T11:38:30.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.324+0000 7f60b9ffb640 1 -- 172.21.15.90:0/1197094940 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60c8055070 con 0x55cb4d6361b0 2024-04-25T11:38:30.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.324+0000 7f60b9ffb640 1 -- 172.21.15.90:0/1197094940 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f60cc01d040 con 0x55cb4d6361b0 2024-04-25T11:38:30.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.324+0000 7f60df9bf740 10 client.25774 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:30.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.324+0000 7f60df9bf740 10 client.25774 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:38:30.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.324+0000 7f60df9bf740 20 client.25774 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:30.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.324+0000 7f60df9bf740 1 --2- 172.21.15.90:0/1197094940 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55cb4d5ca1c0 0x55cb4da5d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:30.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.325+0000 7f60df9bf740 4 client.25774 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:38:30.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.325+0000 7f60df9bf740 1 -- 172.21.15.90:0/1197094940 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cb4d943ee0 con 0x55cb4d5ca1c0 2024-04-25T11:38:30.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.325+0000 7f60d259c640 1 --2- 172.21.15.90:0/1197094940 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55cb4d5ca1c0 0x55cb4da5d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:30.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.326+0000 7f60b9ffb640 10 client.25774 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:30.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:30.328+0000 7f60d259c640 1 --2- 172.21.15.90:0/1197094940 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55cb4d5ca1c0 0x55cb4da5d100 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=edd7f471428641c4 server_cookie=e1081ba9b2837475 in_seq=0 out_seq=0 2024-04-25T11:38:30.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:30 smithi156 ceph-mon[33045]: pgmap v105: 97 pgs: 97 active+clean; 24 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 1.6 MiB/s rd, 406 MiB/s wr, 1.32k op/s 2024-04-25T11:38:30.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:30 smithi090 ceph-mon[29575]: pgmap v105: 97 pgs: 97 active+clean; 24 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 1.6 MiB/s rd, 406 MiB/s wr, 1.32k op/s 2024-04-25T11:38:32.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:32 smithi138 ceph-mon[32328]: pgmap v106: 97 pgs: 97 active+clean; 27 GiB data, 81 GiB used, 992 GiB / 1.0 TiB avail; 2.5 MiB/s rd, 727 MiB/s wr, 1.82k op/s 2024-04-25T11:38:32.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:32 smithi156 ceph-mon[33045]: pgmap v106: 97 pgs: 97 active+clean; 27 GiB data, 81 GiB used, 992 GiB / 1.0 TiB avail; 2.5 MiB/s rd, 727 MiB/s wr, 1.82k op/s 2024-04-25T11:38:32.476 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:32 smithi090 ceph-mon[29575]: pgmap v106: 97 pgs: 97 active+clean; 27 GiB data, 81 GiB used, 992 GiB / 1.0 TiB avail; 2.5 MiB/s rd, 727 MiB/s wr, 1.82k op/s 2024-04-25T11:38:32.505 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:38:32.505 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:38:32.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.707+0000 7fb836cf2640 1 -- 172.21.15.90:0/347376657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb830108ec0 msgr2=0x7fb830102b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:32.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.707+0000 7fb836cf2640 1 --2- 172.21.15.90:0/347376657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb830108ec0 0x7fb830102b10 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fb81c009900 tx=0x7fb81c02a410 comp rx=0 tx=0).stop 2024-04-25T11:38:32.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.707+0000 7fb836cf2640 1 -- 172.21.15.90:0/347376657 shutdown_connections 2024-04-25T11:38:32.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.707+0000 7fb836cf2640 1 --2- 172.21.15.90:0/347376657 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb830103180 0x7fb83010f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:32.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.707+0000 7fb836cf2640 1 --2- 172.21.15.90:0/347376657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb830108ec0 0x7fb830102b10 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:32.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.707+0000 7fb836cf2640 1 --2- 172.21.15.90:0/347376657 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb830108a20 0x7fb8301025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:32.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.707+0000 7fb836cf2640 1 -- 172.21.15.90:0/347376657 >> 172.21.15.90:0/347376657 conn(0x7fb830078110 msgr2=0x7fb830078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:32.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.707+0000 7fb836cf2640 1 -- 172.21.15.90:0/347376657 shutdown_connections 2024-04-25T11:38:32.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.707+0000 7fb836cf2640 1 -- 172.21.15.90:0/347376657 wait complete. 2024-04-25T11:38:32.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.708+0000 7fb836cf2640 1 Processor -- start 2024-04-25T11:38:32.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.708+0000 7fb836cf2640 1 -- start start 2024-04-25T11:38:32.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.708+0000 7fb836cf2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb830103180 0x7fb8301a0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:32.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.708+0000 7fb836cf2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb830108a20 0x7fb8301a0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:32.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.708+0000 7fb836cf2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb830108ec0 0x7fb83019a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:32.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.709+0000 7fb836cf2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb830112d50 con 0x7fb830108ec0 2024-04-25T11:38:32.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.709+0000 7fb836cf2640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb830112bd0 con 0x7fb830103180 2024-04-25T11:38:32.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.709+0000 7fb836cf2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb830112ed0 con 0x7fb830108a20 2024-04-25T11:38:32.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.709+0000 7fb83528f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb830108ec0 0x7fb83019a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:32.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.709+0000 7fb83528f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb830108ec0 0x7fb83019a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34688/0 (socket says 172.21.15.90:34688) 2024-04-25T11:38:32.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.709+0000 7fb83528f640 1 -- 172.21.15.90:0/2699481508 learned_addr learned my addr 172.21.15.90:0/2699481508 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:32.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.709+0000 7fb83528f640 1 -- 172.21.15.90:0/2699481508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb830108a20 msgr2=0x7fb8301a0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:32.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.709+0000 7fb83528f640 1 --2- 172.21.15.90:0/2699481508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb830108a20 0x7fb8301a0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:32.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.709+0000 7fb83528f640 1 -- 172.21.15.90:0/2699481508 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb830103180 msgr2=0x7fb8301a0560 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:38:32.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.709+0000 7fb83528f640 1 --2- 172.21.15.90:0/2699481508 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb830103180 0x7fb8301a0560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:32.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.709+0000 7fb83528f640 1 -- 172.21.15.90:0/2699481508 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb83019ae10 con 0x7fb830108ec0 2024-04-25T11:38:32.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.709+0000 7fb83528f640 1 --2- 172.21.15.90:0/2699481508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb830108ec0 0x7fb83019a6e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fb82800b970 tx=0x7fb82800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:32.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.710+0000 7fb82dffb640 1 -- 172.21.15.90:0/2699481508 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb828019070 con 0x7fb830108ec0 2024-04-25T11:38:32.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.710+0000 7fb836cf2640 1 -- 172.21.15.90:0/2699481508 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb83019b0a0 con 0x7fb830108ec0 2024-04-25T11:38:32.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.710+0000 7fb82dffb640 1 -- 172.21.15.90:0/2699481508 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb828014440 con 0x7fb830108ec0 2024-04-25T11:38:32.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.710+0000 7fb82dffb640 1 -- 172.21.15.90:0/2699481508 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8280075d0 con 0x7fb830108ec0 2024-04-25T11:38:32.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.710+0000 7fb836cf2640 1 -- 172.21.15.90:0/2699481508 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8301a7300 con 0x7fb830108ec0 2024-04-25T11:38:32.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.713+0000 7fb83dba6740 1 Processor -- start 2024-04-25T11:38:32.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.715+0000 7fb82dffb640 1 -- 172.21.15.90:0/2699481508 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fb82801c070 con 0x7fb830108ec0 2024-04-25T11:38:32.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.715+0000 7fb83dba6740 1 -- start start 2024-04-25T11:38:32.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.715+0000 7fb83dba6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55919b616390 0x55919b9b6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:32.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.716+0000 7fb83dba6740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55919b5ed4d0 0x55919b9b6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:32.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.716+0000 7fb83dba6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b5f56b0 0x55919b9c38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:32.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.716+0000 7fb83dba6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55919b91cd80 con 0x55919b5f56b0 2024-04-25T11:38:32.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.716+0000 7fb83dba6740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55919b8d68b0 con 0x55919b5ed4d0 2024-04-25T11:38:32.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.716+0000 7fb83dba6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55919b918d40 con 0x55919b616390 2024-04-25T11:38:32.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.716+0000 7fb82ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b5f56b0 0x55919b9c38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:32.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.716+0000 7fb82ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b5f56b0 0x55919b9c38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34696/0 (socket says 172.21.15.90:34696) 2024-04-25T11:38:32.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.716+0000 7fb82ffff640 1 -- 172.21.15.90:0/2181851534 learned_addr learned my addr 172.21.15.90:0/2181851534 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:32.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.716+0000 7fb82dffb640 1 --2- 172.21.15.90:0/2699481508 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8200afc60 0x7fb8200b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:32.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.716+0000 7fb82ffff640 1 -- 172.21.15.90:0/2181851534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55919b616390 msgr2=0x55919b9b6a60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:38:32.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.716+0000 7fb82ffff640 1 --2- 172.21.15.90:0/2181851534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55919b616390 0x55919b9b6a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:32.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.716+0000 7fb82ffff640 1 -- 172.21.15.90:0/2181851534 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55919b5ed4d0 msgr2=0x55919b9b6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:32.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.716+0000 7fb82ffff640 1 --2- 172.21.15.90:0/2181851534 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55919b5ed4d0 0x55919b9b6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:32.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.716+0000 7fb82ffff640 1 -- 172.21.15.90:0/2181851534 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55919b8a7a40 con 0x55919b5f56b0 2024-04-25T11:38:32.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.717+0000 7fb83528f640 1 --2- 172.21.15.90:0/2699481508 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8200afc60 0x7fb8200b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:32.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.717+0000 7fb82dffb640 1 -- 172.21.15.90:0/2699481508 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb8280de600 con 0x7fb830108ec0 2024-04-25T11:38:32.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.717+0000 7fb82ffff640 1 --2- 172.21.15.90:0/2181851534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b5f56b0 0x55919b9c38f0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fb81c009230 tx=0x7fb81c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98ab02d9a3e6c929 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:32.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.717+0000 7fb81affd640 1 -- 172.21.15.90:0/2181851534 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81c03e070 con 0x55919b5f56b0 2024-04-25T11:38:32.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.717+0000 7fb83528f640 1 --2- 172.21.15.90:0/2699481508 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8200afc60 0x7fb8200b2170 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb8280dbeb0 tx=0x7fb828004b50 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:32.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.717+0000 7fb81affd640 1 -- 172.21.15.90:0/2181851534 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb81c003aa0 con 0x55919b5f56b0 2024-04-25T11:38:32.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.717+0000 7fb81affd640 1 -- 172.21.15.90:0/2181851534 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81c035040 con 0x55919b5f56b0 2024-04-25T11:38:32.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.718+0000 7fb83dba6740 1 -- 172.21.15.90:0/2181851534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b5f56b0 msgr2=0x55919b9c38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:32.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.718+0000 7fb83dba6740 1 --2- 172.21.15.90:0/2181851534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b5f56b0 0x55919b9c38f0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fb81c009230 tx=0x7fb81c000bc0 comp rx=0 tx=0).stop 2024-04-25T11:38:32.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.718+0000 7fb83dba6740 1 -- 172.21.15.90:0/2181851534 shutdown_connections 2024-04-25T11:38:32.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.719+0000 7fb83dba6740 1 --2- 172.21.15.90:0/2181851534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b5f56b0 0x55919b9c38f0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:32.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.719+0000 7fb83dba6740 1 --2- 172.21.15.90:0/2181851534 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55919b5ed4d0 0x55919b9b6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:32.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.719+0000 7fb83dba6740 1 --2- 172.21.15.90:0/2181851534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55919b616390 0x55919b9b6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:32.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.719+0000 7fb83dba6740 1 -- 172.21.15.90:0/2181851534 >> 172.21.15.90:0/2181851534 conn(0x55919b60a4e0 msgr2=0x55919b9b3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:32.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.719+0000 7fb83dba6740 1 -- 172.21.15.90:0/2181851534 shutdown_connections 2024-04-25T11:38:32.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.719+0000 7fb83dba6740 1 -- 172.21.15.90:0/2181851534 wait complete. 2024-04-25T11:38:32.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.719+0000 7fb83dba6740 1 -- start start 2024-04-25T11:38:32.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.719+0000 7fb83dba6740 1 Processor -- start 2024-04-25T11:38:32.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.720+0000 7fb83dba6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55919b5ed4d0 0x55919b9c1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:32.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.720+0000 7fb83dba6740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55919b5f56b0 0x55919b9c1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:32.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.720+0000 7fb83dba6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b616390 0x55919b9c7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:32.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.720+0000 7fb83dba6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55919b9c3f80 con 0x55919b616390 2024-04-25T11:38:32.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.720+0000 7fb83dba6740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55919b8a3d10 con 0x55919b5f56b0 2024-04-25T11:38:32.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.720+0000 7fb83dba6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55919b9c4100 con 0x55919b5ed4d0 2024-04-25T11:38:32.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.720+0000 7fb83dba6740 10 client.0 _ll_register_callbacks cb 0x55919b8d0210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:32.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.720+0000 7fb82ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b616390 0x55919b9c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:32.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.720+0000 7fb82ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b616390 0x55919b9c7440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34704/0 (socket says 172.21.15.90:34704) 2024-04-25T11:38:32.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.720+0000 7fb82ffff640 1 -- 172.21.15.90:0/135251810 learned_addr learned my addr 172.21.15.90:0/135251810 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:32.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.720+0000 7fb82ffff640 1 -- 172.21.15.90:0/135251810 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55919b5ed4d0 msgr2=0x55919b9c1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:32.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.720+0000 7fb82ffff640 1 --2- 172.21.15.90:0/135251810 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55919b5ed4d0 0x55919b9c1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:32.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.720+0000 7fb82ffff640 1 -- 172.21.15.90:0/135251810 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55919b5f56b0 msgr2=0x55919b9c1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:32.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.720+0000 7fb82ffff640 1 --2- 172.21.15.90:0/135251810 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55919b5f56b0 0x55919b9c1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:32.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.720+0000 7fb82ffff640 1 -- 172.21.15.90:0/135251810 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55919b6f87d0 con 0x55919b616390 2024-04-25T11:38:32.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.721+0000 7fb82ffff640 1 --2- 172.21.15.90:0/135251810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b616390 0x55919b9c7440 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fb81c0096c0 tx=0x7fb81c004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd8e3509d042e6a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:32.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.721+0000 7fb80ffff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:38:32.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.721+0000 7fb80ffff640 1 -- 172.21.15.90:0/135251810 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81c035030 con 0x55919b616390 2024-04-25T11:38:32.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.721+0000 7fb83dba6740 1 -- 172.21.15.90:0/135251810 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb81c003aa0 con 0x55919b616390 2024-04-25T11:38:32.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.721+0000 7fb80ffff640 1 -- 172.21.15.90:0/135251810 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb81c0035f0 con 0x55919b616390 2024-04-25T11:38:32.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.721+0000 7fb80ffff640 1 -- 172.21.15.90:0/135251810 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb81c051050 con 0x55919b616390 2024-04-25T11:38:32.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.721+0000 7fb83dba6740 10 client.16581 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:32.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.721+0000 7fb83dba6740 1 -- 172.21.15.90:0/135251810 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55919b62e3f0 con 0x55919b616390 2024-04-25T11:38:32.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.722+0000 7fb80ffff640 1 -- 172.21.15.90:0/135251810 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb81c003aa0 con 0x55919b616390 2024-04-25T11:38:32.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.722+0000 7fb80ffff640 1 -- 172.21.15.90:0/135251810 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7fb81c042700 con 0x55919b616390 2024-04-25T11:38:32.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.722+0000 7fb83dba6740 10 client.16581 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:32.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.722+0000 7fb83dba6740 10 client.16581 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:38:32.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.722+0000 7fb83dba6740 20 client.16581 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:32.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.722+0000 7fb83dba6740 1 --2- 172.21.15.90:0/135251810 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55919b5d4f60 0x55919b9d3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:32.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.722+0000 7fb83dba6740 4 client.16581 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:38:32.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.722+0000 7fb83dba6740 1 -- 172.21.15.90:0/135251810 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55919b92a990 con 0x55919b5d4f60 2024-04-25T11:38:32.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.725+0000 7fb834a8e640 1 --2- 172.21.15.90:0/135251810 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55919b5d4f60 0x55919b9d3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:32.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.730+0000 7fb834a8e640 1 --2- 172.21.15.90:0/135251810 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55919b5d4f60 0x55919b9d3770 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=784b220e38270754 server_cookie=bbc2c0b66eebe60a in_seq=0 out_seq=0 2024-04-25T11:38:32.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:32.730+0000 7fb80ffff640 10 client.16581 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:33.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:33 smithi138 ceph-mon[32328]: pgmap v107: 97 pgs: 97 active+clean; 27 GiB data, 81 GiB used, 992 GiB / 1.0 TiB avail; 2.1 MiB/s rd, 580 MiB/s wr, 598 op/s 2024-04-25T11:38:33.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:33 smithi156 ceph-mon[33045]: pgmap v107: 97 pgs: 97 active+clean; 27 GiB data, 81 GiB used, 992 GiB / 1.0 TiB avail; 2.1 MiB/s rd, 580 MiB/s wr, 598 op/s 2024-04-25T11:38:33.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:33 smithi090 ceph-mon[29575]: pgmap v107: 97 pgs: 97 active+clean; 27 GiB data, 81 GiB used, 992 GiB / 1.0 TiB avail; 2.1 MiB/s rd, 580 MiB/s wr, 598 op/s 2024-04-25T11:38:34.848 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.860+0000 7f60b9ffb640 1 -- 172.21.15.90:0/1197094940 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cb4d943ee0 con 0x55cb4d5ca1c0 2024-04-25T11:38:34.848 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.860+0000 7f60b9ffb640 10 client.25774 handle_command_reply: tid=0 2024-04-25T11:38:34.849 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.860+0000 7f60df9bf740 1 client.25774 shutdown 2024-04-25T11:38:34.849 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.860+0000 7f60df9bf740 1 -- 172.21.15.90:0/1197094940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb4d6361b0 msgr2=0x55cb4d9e5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:34.850 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.860+0000 7f60df9bf740 1 --2- 172.21.15.90:0/1197094940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb4d6361b0 0x55cb4d9e5e80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00f650 tx=0x7f60cc00f680 comp rx=0 tx=0).stop 2024-04-25T11:38:34.850 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.861+0000 7f60df9bf740 1 -- 172.21.15.90:0/1197094940 shutdown_connections 2024-04-25T11:38:34.850 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.861+0000 7f60df9bf740 1 --2- 172.21.15.90:0/1197094940 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55cb4d5ca1c0 0x55cb4da5d100 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.850 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.861+0000 7f60df9bf740 1 --2- 172.21.15.90:0/1197094940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb4d6361b0 0x55cb4d9e5e80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.850 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.861+0000 7f60df9bf740 1 --2- 172.21.15.90:0/1197094940 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cb4d6154d0 0x55cb4d9e5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.850 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.861+0000 7f60df9bf740 1 --2- 172.21.15.90:0/1197094940 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb4d60d2f0 0x55cb4d9e85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.850 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.861+0000 7f60df9bf740 1 -- 172.21.15.90:0/1197094940 >> 172.21.15.90:0/1197094940 conn(0x55cb4d62a300 msgr2=0x55cb4d9de650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:34.851 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.861+0000 7f60b9ffb640 0 client.25774 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:34.851 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.861+0000 7f60df9bf740 1 -- 172.21.15.90:0/1197094940 shutdown_connections 2024-04-25T11:38:34.852 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.861+0000 7f60df9bf740 1 -- 172.21.15.90:0/1197094940 wait complete. 2024-04-25T11:38:34.852 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.861+0000 7f60df9bf740 20 client.25774 trim_cache size 0 max 16384 2024-04-25T11:38:34.852 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:38:34.852 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:38:34.852 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:38:34.853 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:38:34.854 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.862+0000 7f60d359e640 1 -- 172.21.15.90:0/1446154786 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f60bc0afde0 msgr2=0x7f60bc0b22f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:34.854 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.862+0000 7f60d359e640 1 --2- 172.21.15.90:0/1446154786 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f60bc0afde0 0x7f60bc0b22f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x55cb4d9d2e00 tx=0x7f60cc004ab0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.862+0000 7f60d359e640 1 -- 172.21.15.90:0/1446154786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60d410f8e0 msgr2=0x7f60d4119330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.862+0000 7f60d359e640 1 --2- 172.21.15.90:0/1446154786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60d410f8e0 0x7f60d4119330 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f60c400e9c0 tx=0x7f60c400ee80 comp rx=0 tx=0).stop 2024-04-25T11:38:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.863+0000 7f60d359e640 1 -- 172.21.15.90:0/1446154786 shutdown_connections 2024-04-25T11:38:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.863+0000 7f60d359e640 1 --2- 172.21.15.90:0/1446154786 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f60bc0afde0 0x7f60bc0b22f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.863+0000 7f60d359e640 1 --2- 172.21.15.90:0/1446154786 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60d41102c0 0x7f60d4119960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.863+0000 7f60d359e640 1 --2- 172.21.15.90:0/1446154786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60d410f8e0 0x7f60d4119330 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.863+0000 7f60d359e640 1 --2- 172.21.15.90:0/1446154786 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f60d410a0e0 0x7f60d41201c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.863+0000 7f60d359e640 1 -- 172.21.15.90:0/1446154786 >> 172.21.15.90:0/1446154786 conn(0x7f60d406f6e0 msgr2=0x7f60d410d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.864+0000 7f60d359e640 1 -- 172.21.15.90:0/1446154786 shutdown_connections 2024-04-25T11:38:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:34.864+0000 7f60d359e640 1 -- 172.21.15.90:0/1446154786 wait complete. 2024-04-25T11:38:34.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.865+0000 7fb80ffff640 1 -- 172.21.15.90:0/135251810 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55919b92a990 con 0x55919b5d4f60 2024-04-25T11:38:34.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.865+0000 7fb80ffff640 10 client.16581 handle_command_reply: tid=0 2024-04-25T11:38:34.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.865+0000 7fb83dba6740 1 client.16581 shutdown 2024-04-25T11:38:34.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.867+0000 7fb83dba6740 1 -- 172.21.15.90:0/135251810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b616390 msgr2=0x55919b9c7440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:34.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.867+0000 7fb83dba6740 1 --2- 172.21.15.90:0/135251810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b616390 0x55919b9c7440 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fb81c0096c0 tx=0x7fb81c004570 comp rx=0 tx=0).stop 2024-04-25T11:38:34.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.870+0000 7fb83dba6740 1 -- 172.21.15.90:0/135251810 shutdown_connections 2024-04-25T11:38:34.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.870+0000 7fb83dba6740 1 --2- 172.21.15.90:0/135251810 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55919b5d4f60 0x55919b9d3770 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.870+0000 7fb83dba6740 1 --2- 172.21.15.90:0/135251810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b616390 0x55919b9c7440 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.870+0000 7fb83dba6740 1 --2- 172.21.15.90:0/135251810 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55919b5f56b0 0x55919b9c1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.870+0000 7fb83dba6740 1 --2- 172.21.15.90:0/135251810 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55919b5ed4d0 0x55919b9c1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.870+0000 7fb83dba6740 1 -- 172.21.15.90:0/135251810 >> 172.21.15.90:0/135251810 conn(0x55919b60a4e0 msgr2=0x55919b9b8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:34.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.870+0000 7fb80ffff640 0 client.16581 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:34.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.871+0000 7fb83dba6740 1 -- 172.21.15.90:0/135251810 shutdown_connections 2024-04-25T11:38:34.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.871+0000 7fb83dba6740 1 -- 172.21.15.90:0/135251810 wait complete. 2024-04-25T11:38:34.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.871+0000 7fb83dba6740 20 client.16581 trim_cache size 0 max 16384 2024-04-25T11:38:34.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.909+0000 7fb83dba6740 1 Processor -- start 2024-04-25T11:38:34.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.909+0000 7fb83dba6740 1 -- start start 2024-04-25T11:38:34.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.909+0000 7fb83dba6740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55919b5ed4d0 0x55919b9c1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:34.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.909+0000 7fb83dba6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b5f56b0 0x55919b9c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:34.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.909+0000 7fb83dba6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55919b616390 0x55919b9c7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:34.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.909+0000 7fb83dba6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb8301afce0 con 0x55919b5f56b0 2024-04-25T11:38:34.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.909+0000 7fb83dba6740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb8301b1850 con 0x55919b5ed4d0 2024-04-25T11:38:34.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.909+0000 7fb83dba6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb8301b03b0 con 0x55919b616390 2024-04-25T11:38:34.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.909+0000 7fb834a8e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b5f56b0 0x55919b9c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:34.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.909+0000 7fb834a8e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b5f56b0 0x55919b9c73c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34708/0 (socket says 172.21.15.90:34708) 2024-04-25T11:38:34.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.909+0000 7fb834a8e640 1 -- 172.21.15.90:0/50075966 learned_addr learned my addr 172.21.15.90:0/50075966 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:34.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.910+0000 7fb82ffff640 1 --2- 172.21.15.90:0/50075966 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55919b5ed4d0 0x55919b9c1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:34.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.910+0000 7fb82ffff640 1 --2- 172.21.15.90:0/50075966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55919b616390 0x55919b9c7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:34.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.910+0000 7fb834a8e640 1 -- 172.21.15.90:0/50075966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55919b616390 msgr2=0x55919b9c7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:34.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.910+0000 7fb834a8e640 1 --2- 172.21.15.90:0/50075966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55919b616390 0x55919b9c7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.910+0000 7fb834a8e640 1 -- 172.21.15.90:0/50075966 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55919b5ed4d0 msgr2=0x55919b9c1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:34.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.910+0000 7fb834a8e640 1 --2- 172.21.15.90:0/50075966 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55919b5ed4d0 0x55919b9c1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.910+0000 7fb834a8e640 1 -- 172.21.15.90:0/50075966 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55919b5ee0f0 con 0x55919b5f56b0 2024-04-25T11:38:34.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.910+0000 7fb834a8e640 1 --2- 172.21.15.90:0/50075966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b5f56b0 0x55919b9c73c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fb824000c00 tx=0x7fb824009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7871f7d9a2c7f6cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:34.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.910+0000 7fb81a7fc640 1 -- 172.21.15.90:0/50075966 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8240127f0 con 0x55919b5f56b0 2024-04-25T11:38:34.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.910+0000 7fb81a7fc640 1 -- 172.21.15.90:0/50075966 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb82401a070 con 0x55919b5f56b0 2024-04-25T11:38:34.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.910+0000 7fb81a7fc640 1 -- 172.21.15.90:0/50075966 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82401fca0 con 0x55919b5f56b0 2024-04-25T11:38:34.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.912+0000 7fb83dba6740 1 -- 172.21.15.90:0/50075966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b5f56b0 msgr2=0x55919b9c73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:34.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.912+0000 7fb83dba6740 1 --2- 172.21.15.90:0/50075966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b5f56b0 0x55919b9c73c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fb824000c00 tx=0x7fb824009e20 comp rx=0 tx=0).stop 2024-04-25T11:38:34.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.912+0000 7fb83dba6740 1 -- 172.21.15.90:0/50075966 shutdown_connections 2024-04-25T11:38:34.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.912+0000 7fb83dba6740 1 --2- 172.21.15.90:0/50075966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55919b616390 0x55919b9c7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.912+0000 7fb83dba6740 1 --2- 172.21.15.90:0/50075966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b5f56b0 0x55919b9c73c0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.912+0000 7fb83dba6740 1 --2- 172.21.15.90:0/50075966 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55919b5ed4d0 0x55919b9c1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.912+0000 7fb83dba6740 1 -- 172.21.15.90:0/50075966 >> 172.21.15.90:0/50075966 conn(0x55919b60a4e0 msgr2=0x55919b9c1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:34.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.912+0000 7fb83dba6740 1 -- 172.21.15.90:0/50075966 shutdown_connections 2024-04-25T11:38:34.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.912+0000 7fb83dba6740 1 -- 172.21.15.90:0/50075966 wait complete. 2024-04-25T11:38:34.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.913+0000 7fb83dba6740 1 -- start start 2024-04-25T11:38:34.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.913+0000 7fb83dba6740 1 Processor -- start 2024-04-25T11:38:34.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.913+0000 7fb83dba6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b5ed4d0 0x55919b9cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:34.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.914+0000 7fb83dba6740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55919b5f56b0 0x55919b9c8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:34.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.914+0000 7fb83dba6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55919b616390 0x55919b9c8890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:34.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.914+0000 7fb83dba6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb8301b1150 con 0x55919b5ed4d0 2024-04-25T11:38:34.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.914+0000 7fb83dba6740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb8301b0a80 con 0x55919b5f56b0 2024-04-25T11:38:34.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.914+0000 7fb83dba6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb8301adcc0 con 0x55919b616390 2024-04-25T11:38:34.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.914+0000 7fb83dba6740 10 client.0 _ll_register_callbacks cb 0x7fb81c0040d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:34.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.914+0000 7fb82ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b5ed4d0 0x55919b9cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:34.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.914+0000 7fb82ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b5ed4d0 0x55919b9cdf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34714/0 (socket says 172.21.15.90:34714) 2024-04-25T11:38:34.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.914+0000 7fb82ffff640 1 -- 172.21.15.90:0/1338332494 learned_addr learned my addr 172.21.15.90:0/1338332494 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:34.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.914+0000 7fb82ffff640 1 -- 172.21.15.90:0/1338332494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55919b616390 msgr2=0x55919b9c8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:34.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.914+0000 7fb82ffff640 1 --2- 172.21.15.90:0/1338332494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55919b616390 0x55919b9c8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.914+0000 7fb834a8e640 1 --2- 172.21.15.90:0/1338332494 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55919b5f56b0 0x55919b9c8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:34.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.914+0000 7fb82ffff640 1 -- 172.21.15.90:0/1338332494 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55919b5f56b0 msgr2=0x55919b9c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:34.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.914+0000 7fb82ffff640 1 --2- 172.21.15.90:0/1338332494 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55919b5f56b0 0x55919b9c8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.914+0000 7fb82ffff640 1 -- 172.21.15.90:0/1338332494 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55919b8dd4b0 con 0x55919b5ed4d0 2024-04-25T11:38:34.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.914+0000 7fb834a8e640 1 --2- 172.21.15.90:0/1338332494 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55919b5f56b0 0x55919b9c8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:38:34.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.914+0000 7fb82ffff640 1 --2- 172.21.15.90:0/1338332494 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b5ed4d0 0x55919b9cdf30 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fb81c009230 tx=0x7fb81c005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32532ae71aff52c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:34.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.915+0000 7fb80ffff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:38:34.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.915+0000 7fb80ffff640 1 -- 172.21.15.90:0/1338332494 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81c05a070 con 0x55919b5ed4d0 2024-04-25T11:38:34.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.915+0000 7fb80ffff640 1 -- 172.21.15.90:0/1338332494 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb81c003800 con 0x55919b5ed4d0 2024-04-25T11:38:34.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.915+0000 7fb83dba6740 1 -- 172.21.15.90:0/1338332494 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb82401a070 con 0x55919b5ed4d0 2024-04-25T11:38:34.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.915+0000 7fb80ffff640 1 -- 172.21.15.90:0/1338332494 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb81c035040 con 0x55919b5ed4d0 2024-04-25T11:38:34.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.915+0000 7fb83dba6740 10 client.16587 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:34.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.915+0000 7fb83dba6740 1 -- 172.21.15.90:0/1338332494 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55919ba3eca0 con 0x55919b5ed4d0 2024-04-25T11:38:34.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.915+0000 7fb80ffff640 1 -- 172.21.15.90:0/1338332494 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb82401a070 con 0x55919b5ed4d0 2024-04-25T11:38:34.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.916+0000 7fb80ffff640 1 -- 172.21.15.90:0/1338332494 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7fb81c031960 con 0x55919b5ed4d0 2024-04-25T11:38:34.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.916+0000 7fb83dba6740 10 client.16587 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:34.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.916+0000 7fb83dba6740 10 client.16587 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:38:34.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.916+0000 7fb83dba6740 20 client.16587 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:34.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.916+0000 7fb83dba6740 1 --2- 172.21.15.90:0/1338332494 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55919b5aa3a0 0x55919ba41380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:34.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.916+0000 7fb83dba6740 4 client.16587 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:38:34.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.916+0000 7fb83dba6740 1 -- 172.21.15.90:0/1338332494 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55919b924800 con 0x55919b5aa3a0 2024-04-25T11:38:34.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.917+0000 7fb834a8e640 1 --2- 172.21.15.90:0/1338332494 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55919b5aa3a0 0x55919ba41380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:34.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.920+0000 7fb80ffff640 10 client.16587 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:34.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.920+0000 7fb834a8e640 1 --2- 172.21.15.90:0/1338332494 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55919b5aa3a0 0x55919ba41380 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=256e5173c5924f94 server_cookie=869b098f3eb745f3 in_seq=0 out_seq=0 2024-04-25T11:38:34.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.921+0000 7fb80ffff640 1 -- 172.21.15.90:0/1338332494 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55919b924800 con 0x55919b5aa3a0 2024-04-25T11:38:34.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.921+0000 7fb80ffff640 10 client.16587 handle_command_reply: tid=0 2024-04-25T11:38:34.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.921+0000 7fb83dba6740 1 client.16587 shutdown 2024-04-25T11:38:34.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.921+0000 7fb83dba6740 1 -- 172.21.15.90:0/1338332494 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b5ed4d0 msgr2=0x55919b9cdf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:34.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.921+0000 7fb83dba6740 1 --2- 172.21.15.90:0/1338332494 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b5ed4d0 0x55919b9cdf30 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fb81c009230 tx=0x7fb81c005760 comp rx=0 tx=0).stop 2024-04-25T11:38:34.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.921+0000 7fb83dba6740 1 -- 172.21.15.90:0/1338332494 shutdown_connections 2024-04-25T11:38:34.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.921+0000 7fb83dba6740 1 --2- 172.21.15.90:0/1338332494 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55919b5aa3a0 0x55919ba41380 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.921+0000 7fb83dba6740 1 --2- 172.21.15.90:0/1338332494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55919b616390 0x55919b9c8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.921+0000 7fb83dba6740 1 --2- 172.21.15.90:0/1338332494 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55919b5f56b0 0x55919b9c8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.921+0000 7fb83dba6740 1 --2- 172.21.15.90:0/1338332494 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55919b5ed4d0 0x55919b9cdf30 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.921+0000 7fb83dba6740 1 -- 172.21.15.90:0/1338332494 >> 172.21.15.90:0/1338332494 conn(0x55919b60a4e0 msgr2=0x55919b9b2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:34.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.921+0000 7fb80ffff640 0 client.16587 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:34.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.921+0000 7fb83dba6740 1 -- 172.21.15.90:0/1338332494 shutdown_connections 2024-04-25T11:38:34.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.921+0000 7fb83dba6740 1 -- 172.21.15.90:0/1338332494 wait complete. 2024-04-25T11:38:34.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.922+0000 7fb83dba6740 20 client.16587 trim_cache size 0 max 16384 2024-04-25T11:38:34.912 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:38:34.912 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-25T11:38:34.912 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-25T11:38:34.912 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:38:34.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.922+0000 7fb836cf2640 1 -- 172.21.15.90:0/2699481508 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8200afc60 msgr2=0x7fb8200b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:34.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.922+0000 7fb836cf2640 1 --2- 172.21.15.90:0/2699481508 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8200afc60 0x7fb8200b2170 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb8280dbeb0 tx=0x7fb828004b50 comp rx=0 tx=0).stop 2024-04-25T11:38:34.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.922+0000 7fb836cf2640 1 -- 172.21.15.90:0/2699481508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb830108ec0 msgr2=0x7fb83019a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:34.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.922+0000 7fb836cf2640 1 --2- 172.21.15.90:0/2699481508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb830108ec0 0x7fb83019a6e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fb82800b970 tx=0x7fb82800be30 comp rx=0 tx=0).stop 2024-04-25T11:38:34.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.922+0000 7fb836cf2640 1 -- 172.21.15.90:0/2699481508 shutdown_connections 2024-04-25T11:38:34.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.922+0000 7fb836cf2640 1 --2- 172.21.15.90:0/2699481508 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8200afc60 0x7fb8200b2170 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.922+0000 7fb836cf2640 1 --2- 172.21.15.90:0/2699481508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb830108ec0 0x7fb83019a6e0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.922+0000 7fb836cf2640 1 --2- 172.21.15.90:0/2699481508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb830108a20 0x7fb8301a0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.922+0000 7fb836cf2640 1 --2- 172.21.15.90:0/2699481508 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb830103180 0x7fb8301a0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:34.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.922+0000 7fb836cf2640 1 -- 172.21.15.90:0/2699481508 >> 172.21.15.90:0/2699481508 conn(0x7fb830078110 msgr2=0x7fb83006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:34.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.923+0000 7fb836cf2640 1 -- 172.21.15.90:0/2699481508 shutdown_connections 2024-04-25T11:38:34.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:34.923+0000 7fb836cf2640 1 -- 172.21.15.90:0/2699481508 wait complete. 2024-04-25T11:38:34.930 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T11:38:34.930 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:38:35.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.125+0000 7f3769467640 1 -- 172.21.15.90:0/2548304320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3764109b30 msgr2=0x7f37641036d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:35.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.125+0000 7f3769467640 1 --2- 172.21.15.90:0/2548304320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3764109b30 0x7f37641036d0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f3754009900 tx=0x7f375402a3f0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.125+0000 7f3769467640 1 -- 172.21.15.90:0/2548304320 shutdown_connections 2024-04-25T11:38:35.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.125+0000 7f3769467640 1 --2- 172.21.15.90:0/2548304320 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3764104280 0x7f3764110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.125+0000 7f3769467640 1 --2- 172.21.15.90:0/2548304320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3764109fd0 0x7f3764103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.126+0000 7f3769467640 1 --2- 172.21.15.90:0/2548304320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3764109b30 0x7f37641036d0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.126+0000 7f3769467640 1 -- 172.21.15.90:0/2548304320 >> 172.21.15.90:0/2548304320 conn(0x7f37640ff210 msgr2=0x7f3764101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:35.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.126+0000 7f3769467640 1 -- 172.21.15.90:0/2548304320 shutdown_connections 2024-04-25T11:38:35.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.126+0000 7f3769467640 1 -- 172.21.15.90:0/2548304320 wait complete. 2024-04-25T11:38:35.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.127+0000 7f3769467640 1 Processor -- start 2024-04-25T11:38:35.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.128+0000 7f3769467640 1 -- start start 2024-04-25T11:38:35.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.129+0000 7f3769467640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3764104280 0x7f376419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.129+0000 7f3769467640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3764109b30 0x7f376419acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.129+0000 7f3769467640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3764109fd0 0x7f376419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.129+0000 7f3769467640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3764113d20 con 0x7f3764109fd0 2024-04-25T11:38:35.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.129+0000 7f3769467640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3764113ba0 con 0x7f3764104280 2024-04-25T11:38:35.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.129+0000 7f3769467640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3764113ea0 con 0x7f3764109b30 2024-04-25T11:38:35.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.129+0000 7f37637fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3764109fd0 0x7f376419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.129+0000 7f37627fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3764109b30 0x7f376419acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.129+0000 7f37637fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3764109fd0 0x7f376419ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34726/0 (socket says 172.21.15.90:34726) 2024-04-25T11:38:35.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.129+0000 7f37637fe640 1 -- 172.21.15.90:0/2643473179 learned_addr learned my addr 172.21.15.90:0/2643473179 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:35.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.129+0000 7f3762ffd640 1 --2- 172.21.15.90:0/2643473179 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3764104280 0x7f376419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.129+0000 7f37637fe640 1 -- 172.21.15.90:0/2643473179 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3764109b30 msgr2=0x7f376419acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:35.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.129+0000 7f37637fe640 1 --2- 172.21.15.90:0/2643473179 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3764109b30 0x7f376419acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.129+0000 7f37637fe640 1 -- 172.21.15.90:0/2643473179 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3764104280 msgr2=0x7f376419a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:35.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.129+0000 7f37637fe640 1 --2- 172.21.15.90:0/2643473179 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3764104280 0x7f376419a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.129+0000 7f37637fe640 1 -- 172.21.15.90:0/2643473179 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376419f5c0 con 0x7f3764109fd0 2024-04-25T11:38:35.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.130+0000 7f37637fe640 1 --2- 172.21.15.90:0/2643473179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3764109fd0 0x7f376419ef80 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f375c00eb10 tx=0x7f375c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:35.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.130+0000 7f374bfff640 1 -- 172.21.15.90:0/2643473179 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375c004b20 con 0x7f3764109fd0 2024-04-25T11:38:35.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.130+0000 7f3769467640 1 -- 172.21.15.90:0/2643473179 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f376419f8b0 con 0x7f3764109fd0 2024-04-25T11:38:35.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.130+0000 7f374bfff640 1 -- 172.21.15.90:0/2643473179 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f375c004cc0 con 0x7f3764109fd0 2024-04-25T11:38:35.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.130+0000 7f374bfff640 1 -- 172.21.15.90:0/2643473179 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375c010670 con 0x7f3764109fd0 2024-04-25T11:38:35.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.130+0000 7f3769467640 1 -- 172.21.15.90:0/2643473179 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3764075ee0 con 0x7f3764109fd0 2024-04-25T11:38:35.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.132+0000 7f3769467640 1 -- 172.21.15.90:0/2643473179 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3764104e10 con 0x7f3764109fd0 2024-04-25T11:38:35.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.133+0000 7f374bfff640 1 -- 172.21.15.90:0/2643473179 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f375c005000 con 0x7f3764109fd0 2024-04-25T11:38:35.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.134+0000 7f374bfff640 1 --2- 172.21.15.90:0/2643473179 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f374c0b40f0 0x7f374c0b6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.135+0000 7f374bfff640 1 -- 172.21.15.90:0/2643473179 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f375c0df870 con 0x7f3764109fd0 2024-04-25T11:38:35.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.135+0000 7f3762ffd640 1 --2- 172.21.15.90:0/2643473179 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f374c0b40f0 0x7f374c0b6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.135+0000 7f3762ffd640 1 --2- 172.21.15.90:0/2643473179 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f374c0b40f0 0x7f374c0b6600 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3754000c00 tx=0x7f3754031400 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:35.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.140+0000 7f374bfff640 1 -- 172.21.15.90:0/2643473179 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f375c053ff0 con 0x7f3764109fd0 2024-04-25T11:38:35.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.381+0000 7f3769467640 1 -- 172.21.15.90:0/2643473179 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f376406a190 con 0x7f3764109fd0 2024-04-25T11:38:35.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.381+0000 7f374bfff640 1 -- 172.21.15.90:0/2643473179 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11931 (secure 0 0 0) 0x7f375c0a80b0 con 0x7f3764109fd0 2024-04-25T11:38:35.369 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:38:35.369 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:37:15.460907+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24475,24524,24499,24431,24458]},"id":1}]} 2024-04-25T11:38:35.370 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-25T11:38:35.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.384+0000 7f3769467640 1 -- 172.21.15.90:0/2643473179 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f374c0b40f0 msgr2=0x7f374c0b6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:35.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.384+0000 7f3769467640 1 --2- 172.21.15.90:0/2643473179 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f374c0b40f0 0x7f374c0b6600 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3754000c00 tx=0x7f3754031400 comp rx=0 tx=0).stop 2024-04-25T11:38:35.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.384+0000 7f3769467640 1 -- 172.21.15.90:0/2643473179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3764109fd0 msgr2=0x7f376419ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:35.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.384+0000 7f3769467640 1 --2- 172.21.15.90:0/2643473179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3764109fd0 0x7f376419ef80 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f375c00eb10 tx=0x7f375c00c2d0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.384+0000 7f3769467640 1 -- 172.21.15.90:0/2643473179 shutdown_connections 2024-04-25T11:38:35.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.384+0000 7f3769467640 1 --2- 172.21.15.90:0/2643473179 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f374c0b40f0 0x7f374c0b6600 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.384+0000 7f3769467640 1 --2- 172.21.15.90:0/2643473179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3764109fd0 0x7f376419ef80 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.384+0000 7f3769467640 1 --2- 172.21.15.90:0/2643473179 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3764109b30 0x7f376419acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.384+0000 7f3769467640 1 --2- 172.21.15.90:0/2643473179 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3764104280 0x7f376419a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.384+0000 7f3769467640 1 -- 172.21.15.90:0/2643473179 >> 172.21.15.90:0/2643473179 conn(0x7f37640ff210 msgr2=0x7f3764101020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:35.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.384+0000 7f3769467640 1 -- 172.21.15.90:0/2643473179 shutdown_connections 2024-04-25T11:38:35.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.384+0000 7f3769467640 1 -- 172.21.15.90:0/2643473179 wait complete. 2024-04-25T11:38:35.383 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T11:38:35.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.514+0000 7ff13eef9640 1 -- 172.21.15.90:0/2673204660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff138108a20 msgr2=0x7ff1381025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:35.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.514+0000 7ff13eef9640 1 --2- 172.21.15.90:0/2673204660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff138108a20 0x7ff1381025b0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7ff134009900 tx=0x7ff13402a410 comp rx=0 tx=0).stop 2024-04-25T11:38:35.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.515+0000 7ff13eef9640 1 -- 172.21.15.90:0/2673204660 shutdown_connections 2024-04-25T11:38:35.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.515+0000 7ff13eef9640 1 --2- 172.21.15.90:0/2673204660 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff13810d380 0x7ff13810f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.515+0000 7ff13eef9640 1 --2- 172.21.15.90:0/2673204660 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff138102af0 0x7ff138102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.515+0000 7ff13eef9640 1 --2- 172.21.15.90:0/2673204660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff138108a20 0x7ff1381025b0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.515+0000 7ff13eef9640 1 -- 172.21.15.90:0/2673204660 >> 172.21.15.90:0/2673204660 conn(0x7ff138078100 msgr2=0x7ff138078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:35.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.515+0000 7ff13eef9640 1 -- 172.21.15.90:0/2673204660 shutdown_connections 2024-04-25T11:38:35.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.515+0000 7ff13eef9640 1 -- 172.21.15.90:0/2673204660 wait complete. 2024-04-25T11:38:35.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.515+0000 7ff13eef9640 1 Processor -- start 2024-04-25T11:38:35.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.515+0000 7ff13eef9640 1 -- start start 2024-04-25T11:38:35.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.516+0000 7ff13eef9640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff138102af0 0x7ff138199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.516+0000 7ff13eef9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff138108a20 0x7ff138199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.516+0000 7ff13eef9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff13810d380 0x7ff138199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.516+0000 7ff13eef9640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff138112be0 con 0x7ff138108a20 2024-04-25T11:38:35.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.516+0000 7ff13eef9640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff138112a60 con 0x7ff138102af0 2024-04-25T11:38:35.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.516+0000 7ff13eef9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff138112d60 con 0x7ff13810d380 2024-04-25T11:38:35.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.516+0000 7ff12ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff138108a20 0x7ff138199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.516+0000 7ff12ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff138108a20 0x7ff138199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34756/0 (socket says 172.21.15.90:34756) 2024-04-25T11:38:35.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.516+0000 7ff12ffff640 1 -- 172.21.15.90:0/264842710 learned_addr learned my addr 172.21.15.90:0/264842710 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:35.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.516+0000 7ff13cc95640 1 --2- 172.21.15.90:0/264842710 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff138102af0 0x7ff138199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.516+0000 7ff13d496640 1 --2- 172.21.15.90:0/264842710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff13810d380 0x7ff138199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.516+0000 7ff12ffff640 1 -- 172.21.15.90:0/264842710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff13810d380 msgr2=0x7ff138199d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:35.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.516+0000 7ff12ffff640 1 --2- 172.21.15.90:0/264842710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff13810d380 0x7ff138199d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.516+0000 7ff12ffff640 1 -- 172.21.15.90:0/264842710 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff138102af0 msgr2=0x7ff138199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:35.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.516+0000 7ff12ffff640 1 --2- 172.21.15.90:0/264842710 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff138102af0 0x7ff138199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.516+0000 7ff12ffff640 1 -- 172.21.15.90:0/264842710 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1381a89c0 con 0x7ff138108a20 2024-04-25T11:38:35.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.516+0000 7ff13d496640 1 --2- 172.21.15.90:0/264842710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff13810d380 0x7ff138199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:38:35.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.516+0000 7ff12ffff640 1 --2- 172.21.15.90:0/264842710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff138108a20 0x7ff138199850 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff12800e9f0 tx=0x7ff12800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:35.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.516+0000 7ff12dffb640 1 -- 172.21.15.90:0/264842710 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12800ccc0 con 0x7ff138108a20 2024-04-25T11:38:35.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.517+0000 7ff13eef9640 1 -- 172.21.15.90:0/264842710 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1381a8cb0 con 0x7ff138108a20 2024-04-25T11:38:35.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.517+0000 7ff12dffb640 1 -- 172.21.15.90:0/264842710 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1280035f0 con 0x7ff138108a20 2024-04-25T11:38:35.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.517+0000 7ff12dffb640 1 -- 172.21.15.90:0/264842710 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12800ce60 con 0x7ff138108a20 2024-04-25T11:38:35.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.517+0000 7ff13eef9640 1 -- 172.21.15.90:0/264842710 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1381a91b0 con 0x7ff138108a20 2024-04-25T11:38:35.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.519+0000 7ff12dffb640 1 -- 172.21.15.90:0/264842710 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7ff128003aa0 con 0x7ff138108a20 2024-04-25T11:38:35.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.520+0000 7ff12dffb640 1 --2- 172.21.15.90:0/264842710 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff1200afc60 0x7ff1200b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.520+0000 7ff12dffb640 1 -- 172.21.15.90:0/264842710 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7ff1280df480 con 0x7ff138108a20 2024-04-25T11:38:35.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.520+0000 7ff13cc95640 1 --2- 172.21.15.90:0/264842710 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff1200afc60 0x7ff1200b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.521+0000 7ff145daf740 1 Processor -- start 2024-04-25T11:38:35.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.521+0000 7ff13cc95640 1 --2- 172.21.15.90:0/264842710 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff1200afc60 0x7ff1200b2170 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff134000c00 tx=0x7ff134031400 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:35.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.521+0000 7ff145daf740 1 -- start start 2024-04-25T11:38:35.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.521+0000 7ff145daf740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe12403390 0x55fe127a4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.521+0000 7ff145daf740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe123da4d0 0x55fe127a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.521+0000 7ff145daf740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe123e26b0 0x55fe127b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.521+0000 7ff145daf740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55fe12709d80 con 0x55fe123e26b0 2024-04-25T11:38:35.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.521+0000 7ff145daf740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55fe126c3a20 con 0x55fe12403390 2024-04-25T11:38:35.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.521+0000 7ff145daf740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fe12705d40 con 0x55fe123da4d0 2024-04-25T11:38:35.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.521+0000 7ff13d496640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe123e26b0 0x55fe127b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.521+0000 7ff12ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe123da4d0 0x55fe127a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.521+0000 7ff13cc95640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe12403390 0x55fe127a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.521+0000 7ff13cc95640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe12403390 0x55fe127a4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:38850/0 (socket says 172.21.15.90:38850) 2024-04-25T11:38:35.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.521+0000 7ff12ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe123da4d0 0x55fe127a4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:33468/0 (socket says 172.21.15.90:33468) 2024-04-25T11:38:35.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.521+0000 7ff13d496640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe123e26b0 0x55fe127b16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34768/0 (socket says 172.21.15.90:34768) 2024-04-25T11:38:35.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.521+0000 7ff13cc95640 1 -- 172.21.15.90:0/3019389678 learned_addr learned my addr 172.21.15.90:0/3019389678 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:35.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.522+0000 7ff13cc95640 1 -- 172.21.15.90:0/3019389678 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe123da4d0 msgr2=0x55fe127a4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:35.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.522+0000 7ff13cc95640 1 --2- 172.21.15.90:0/3019389678 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe123da4d0 0x55fe127a4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.522+0000 7ff13cc95640 1 -- 172.21.15.90:0/3019389678 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe123e26b0 msgr2=0x55fe127b16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:35.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.522+0000 7ff13cc95640 1 --2- 172.21.15.90:0/3019389678 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe123e26b0 0x55fe127b16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.522+0000 7ff13cc95640 1 -- 172.21.15.90:0/3019389678 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe126cec30 con 0x55fe12403390 2024-04-25T11:38:35.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.522+0000 7ff13d496640 1 --2- 172.21.15.90:0/3019389678 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe123e26b0 0x55fe127b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:35.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.522+0000 7ff13cc95640 1 --2- 172.21.15.90:0/3019389678 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe12403390 0x55fe127a4810 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff134009900 tx=0x7ff13403d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58871369b5fc3c0c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:35.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.522+0000 7ff11effd640 1 -- 172.21.15.90:0/3019389678 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13404c070 con 0x55fe12403390 2024-04-25T11:38:35.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.522+0000 7ff11effd640 1 -- 172.21.15.90:0/3019389678 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1340316e0 con 0x55fe12403390 2024-04-25T11:38:35.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.522+0000 7ff11effd640 1 -- 172.21.15.90:0/3019389678 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff134035040 con 0x55fe12403390 2024-04-25T11:38:35.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.523+0000 7ff145daf740 1 -- 172.21.15.90:0/3019389678 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe12403390 msgr2=0x55fe127a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:35.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.523+0000 7ff145daf740 1 --2- 172.21.15.90:0/3019389678 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe12403390 0x55fe127a4810 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff134009900 tx=0x7ff13403d000 comp rx=0 tx=0).stop 2024-04-25T11:38:35.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.523+0000 7ff145daf740 1 -- 172.21.15.90:0/3019389678 shutdown_connections 2024-04-25T11:38:35.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.523+0000 7ff145daf740 1 --2- 172.21.15.90:0/3019389678 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe123e26b0 0x55fe127b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.523+0000 7ff145daf740 1 --2- 172.21.15.90:0/3019389678 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe123da4d0 0x55fe127a4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.523+0000 7ff145daf740 1 --2- 172.21.15.90:0/3019389678 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe12403390 0x55fe127a4810 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.523+0000 7ff145daf740 1 -- 172.21.15.90:0/3019389678 >> 172.21.15.90:0/3019389678 conn(0x55fe123f74e0 msgr2=0x55fe127a0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:35.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.523+0000 7ff145daf740 1 -- 172.21.15.90:0/3019389678 shutdown_connections 2024-04-25T11:38:35.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.524+0000 7ff145daf740 1 -- 172.21.15.90:0/3019389678 wait complete. 2024-04-25T11:38:35.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.524+0000 7ff145daf740 1 -- start start 2024-04-25T11:38:35.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.524+0000 7ff145daf740 1 Processor -- start 2024-04-25T11:38:35.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.524+0000 7ff145daf740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe123da4d0 0x55fe127af240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.524+0000 7ff145daf740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe123e26b0 0x55fe127af780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.525+0000 7ff145daf740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe12403390 0x55fe127bb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.525+0000 7ff145daf740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55fe127b1d30 con 0x55fe123e26b0 2024-04-25T11:38:35.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.525+0000 7ff145daf740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55fe12690dc0 con 0x55fe123da4d0 2024-04-25T11:38:35.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.525+0000 7ff145daf740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fe127b1eb0 con 0x55fe12403390 2024-04-25T11:38:35.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.525+0000 7ff145daf740 10 client.0 _ll_register_callbacks cb 0x55fe126ac600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:35.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.525+0000 7ff13cc95640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe123da4d0 0x55fe127af240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.525+0000 7ff13d496640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe12403390 0x55fe127bb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.525+0000 7ff13d496640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe12403390 0x55fe127bb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:33482/0 (socket says 172.21.15.90:33482) 2024-04-25T11:38:35.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.525+0000 7ff13cc95640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe123da4d0 0x55fe127af240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:38856/0 (socket says 172.21.15.90:38856) 2024-04-25T11:38:35.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.525+0000 7ff13d496640 1 -- 172.21.15.90:0/1199239843 learned_addr learned my addr 172.21.15.90:0/1199239843 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:35.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.525+0000 7ff12ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe123e26b0 0x55fe127af780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.525+0000 7ff13d496640 1 -- 172.21.15.90:0/1199239843 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe123da4d0 msgr2=0x55fe127af240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:35.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.525+0000 7ff13d496640 1 --2- 172.21.15.90:0/1199239843 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe123da4d0 0x55fe127af240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.525+0000 7ff13d496640 1 -- 172.21.15.90:0/1199239843 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe123e26b0 msgr2=0x55fe127af780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:35.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.525+0000 7ff13d496640 1 --2- 172.21.15.90:0/1199239843 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe123e26b0 0x55fe127af780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.525+0000 7ff13d496640 1 -- 172.21.15.90:0/1199239843 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe1241b3f0 con 0x55fe12403390 2024-04-25T11:38:35.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.525+0000 7ff13d496640 1 --2- 172.21.15.90:0/1199239843 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe12403390 0x55fe127bb1b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff13000afd0 tx=0x7ff13000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89a9914e1061434c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:35.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.525+0000 7ff117fff640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:38:35.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.525+0000 7ff117fff640 1 -- 172.21.15.90:0/1199239843 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13001e060 con 0x55fe12403390 2024-04-25T11:38:35.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.526+0000 7ff117fff640 1 -- 172.21.15.90:0/1199239843 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7ff13000e7c0 con 0x55fe12403390 2024-04-25T11:38:35.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.526+0000 7ff145daf740 1 -- 172.21.15.90:0/1199239843 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1340316e0 con 0x55fe12403390 2024-04-25T11:38:35.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.526+0000 7ff117fff640 1 -- 172.21.15.90:0/1199239843 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff13000e9c0 con 0x55fe12403390 2024-04-25T11:38:35.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.526+0000 7ff145daf740 10 client.25789 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:35.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.526+0000 7ff145daf740 1 -- 172.21.15.90:0/1199239843 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fe124e57d0 con 0x55fe12403390 2024-04-25T11:38:35.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.526+0000 7ff117fff640 1 -- 172.21.15.90:0/1199239843 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1340316e0 con 0x55fe12403390 2024-04-25T11:38:35.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.526+0000 7ff117fff640 1 -- 172.21.15.90:0/1199239843 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7ff130018030 con 0x55fe12403390 2024-04-25T11:38:35.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.526+0000 7ff145daf740 10 client.25789 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:35.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.526+0000 7ff145daf740 10 client.25789 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:38:35.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.526+0000 7ff145daf740 20 client.25789 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:35.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.526+0000 7ff145daf740 1 --2- 172.21.15.90:0/1199239843 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55fe123c1f60 0x55fe127c16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.526+0000 7ff145daf740 4 client.25789 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:38:35.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.526+0000 7ff145daf740 1 -- 172.21.15.90:0/1199239843 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe12717990 con 0x55fe123c1f60 2024-04-25T11:38:35.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.526+0000 7ff13cc95640 1 --2- 172.21.15.90:0/1199239843 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55fe123c1f60 0x55fe127c16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.527+0000 7ff117fff640 10 client.25789 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:38:35.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.527+0000 7ff13cc95640 1 --2- 172.21.15.90:0/1199239843 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55fe123c1f60 0x55fe127c16a0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f4dd8b8e2677ce5a server_cookie=52ce17a465ad5e96 in_seq=0 out_seq=0 2024-04-25T11:38:35.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.527+0000 7ff117fff640 1 -- 172.21.15.90:0/1199239843 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55fe12717990 con 0x55fe123c1f60 2024-04-25T11:38:35.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.527+0000 7ff117fff640 10 client.25789 handle_command_reply: tid=0 2024-04-25T11:38:35.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.528+0000 7ff145daf740 1 client.25789 shutdown 2024-04-25T11:38:35.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.528+0000 7ff145daf740 1 -- 172.21.15.90:0/1199239843 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe12403390 msgr2=0x55fe127bb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:35.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.528+0000 7ff145daf740 1 --2- 172.21.15.90:0/1199239843 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe12403390 0x55fe127bb1b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff13000afd0 tx=0x7ff13000f790 comp rx=0 tx=0).stop 2024-04-25T11:38:35.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.528+0000 7ff145daf740 1 -- 172.21.15.90:0/1199239843 shutdown_connections 2024-04-25T11:38:35.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.528+0000 7ff145daf740 1 --2- 172.21.15.90:0/1199239843 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55fe123c1f60 0x55fe127c16a0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.528+0000 7ff145daf740 1 --2- 172.21.15.90:0/1199239843 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe12403390 0x55fe127bb1b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.528+0000 7ff145daf740 1 --2- 172.21.15.90:0/1199239843 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe123e26b0 0x55fe127af780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.528+0000 7ff117fff640 0 client.25789 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:38:35.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.528+0000 7ff145daf740 1 --2- 172.21.15.90:0/1199239843 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe123da4d0 0x55fe127af240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.528+0000 7ff145daf740 1 -- 172.21.15.90:0/1199239843 >> 172.21.15.90:0/1199239843 conn(0x55fe123f74e0 msgr2=0x55fe127a6530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:35.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.528+0000 7ff145daf740 1 -- 172.21.15.90:0/1199239843 shutdown_connections 2024-04-25T11:38:35.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.528+0000 7ff145daf740 1 -- 172.21.15.90:0/1199239843 wait complete. 2024-04-25T11:38:35.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.528+0000 7ff145daf740 20 client.25789 trim_cache size 0 max 16384 2024-04-25T11:38:35.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.543+0000 7ff145daf740 1 Processor -- start 2024-04-25T11:38:35.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.543+0000 7ff145daf740 1 -- start start 2024-04-25T11:38:35.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.543+0000 7ff145daf740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe123da4d0 0x55fe127ae010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.543+0000 7ff145daf740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe123e26b0 0x55fe127bae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.543+0000 7ff145daf740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe12403390 0x55fe127a94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.543+0000 7ff145daf740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff1381b1c60 con 0x55fe123da4d0 2024-04-25T11:38:35.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.543+0000 7ff145daf740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff1381b37d0 con 0x55fe12403390 2024-04-25T11:38:35.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.543+0000 7ff145daf740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff1381b2330 con 0x55fe123e26b0 2024-04-25T11:38:35.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.543+0000 7ff13cc95640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe123da4d0 0x55fe127ae010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.543+0000 7ff13cc95640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe123da4d0 0x55fe127ae010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34784/0 (socket says 172.21.15.90:34784) 2024-04-25T11:38:35.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.543+0000 7ff13cc95640 1 -- 172.21.15.90:0/3634851092 learned_addr learned my addr 172.21.15.90:0/3634851092 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:35.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.543+0000 7ff13d496640 1 --2- 172.21.15.90:0/3634851092 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe12403390 0x55fe127a94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.543+0000 7ff12ffff640 1 --2- 172.21.15.90:0/3634851092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe123e26b0 0x55fe127bae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.544+0000 7ff13cc95640 1 -- 172.21.15.90:0/3634851092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe123e26b0 msgr2=0x55fe127bae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:35.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.544+0000 7ff13cc95640 1 --2- 172.21.15.90:0/3634851092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe123e26b0 0x55fe127bae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.544+0000 7ff13cc95640 1 -- 172.21.15.90:0/3634851092 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe12403390 msgr2=0x55fe127a94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:35.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.544+0000 7ff13cc95640 1 --2- 172.21.15.90:0/3634851092 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe12403390 0x55fe127a94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.544+0000 7ff13cc95640 1 -- 172.21.15.90:0/3634851092 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe127a5fc0 con 0x55fe123da4d0 2024-04-25T11:38:35.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.544+0000 7ff13d496640 1 --2- 172.21.15.90:0/3634851092 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe12403390 0x55fe127a94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:35.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.544+0000 7ff13cc95640 1 --2- 172.21.15.90:0/3634851092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe123da4d0 0x55fe127ae010 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7ff134009690 tx=0x7ff134003870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcaf052f9a42e534 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:35.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.544+0000 7ff11e7fc640 1 -- 172.21.15.90:0/3634851092 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff134061070 con 0x55fe123da4d0 2024-04-25T11:38:35.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.544+0000 7ff11e7fc640 1 -- 172.21.15.90:0/3634851092 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1340054f0 con 0x55fe123da4d0 2024-04-25T11:38:35.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.544+0000 7ff11e7fc640 1 -- 172.21.15.90:0/3634851092 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff134040070 con 0x55fe123da4d0 2024-04-25T11:38:35.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.545+0000 7ff145daf740 1 -- 172.21.15.90:0/3634851092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe123da4d0 msgr2=0x55fe127ae010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:35.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.545+0000 7ff145daf740 1 --2- 172.21.15.90:0/3634851092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe123da4d0 0x55fe127ae010 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7ff134009690 tx=0x7ff134003870 comp rx=0 tx=0).stop 2024-04-25T11:38:35.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.545+0000 7ff145daf740 1 -- 172.21.15.90:0/3634851092 shutdown_connections 2024-04-25T11:38:35.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.545+0000 7ff145daf740 1 --2- 172.21.15.90:0/3634851092 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe12403390 0x55fe127a94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.545+0000 7ff145daf740 1 --2- 172.21.15.90:0/3634851092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe123e26b0 0x55fe127bae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.545+0000 7ff145daf740 1 --2- 172.21.15.90:0/3634851092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe123da4d0 0x55fe127ae010 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.545+0000 7ff145daf740 1 -- 172.21.15.90:0/3634851092 >> 172.21.15.90:0/3634851092 conn(0x55fe123f74e0 msgr2=0x55fe127ad900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:35.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.545+0000 7ff145daf740 1 -- 172.21.15.90:0/3634851092 shutdown_connections 2024-04-25T11:38:35.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.545+0000 7ff145daf740 1 -- 172.21.15.90:0/3634851092 wait complete. 2024-04-25T11:38:35.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.545+0000 7ff145daf740 1 -- start start 2024-04-25T11:38:35.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.546+0000 7ff145daf740 1 Processor -- start 2024-04-25T11:38:35.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.546+0000 7ff145daf740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe123da4d0 0x55fe127b7820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.546+0000 7ff145daf740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe123e26b0 0x55fe127b2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.546+0000 7ff145daf740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe12403390 0x55fe127b31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.546+0000 7ff145daf740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff1381b30d0 con 0x55fe12403390 2024-04-25T11:38:35.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.546+0000 7ff145daf740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff1381b2a00 con 0x55fe123da4d0 2024-04-25T11:38:35.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.546+0000 7ff145daf740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff1381afc40 con 0x55fe123e26b0 2024-04-25T11:38:35.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.546+0000 7ff145daf740 10 client.0 _ll_register_callbacks cb 0x7ff13000aa90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:35.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.546+0000 7ff13cc95640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe123da4d0 0x55fe127b7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.546+0000 7ff13d496640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe12403390 0x55fe127b31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.546+0000 7ff13cc95640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe123da4d0 0x55fe127b7820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:38872/0 (socket says 172.21.15.90:38872) 2024-04-25T11:38:35.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.546+0000 7ff13cc95640 1 -- 172.21.15.90:0/932790667 learned_addr learned my addr 172.21.15.90:0/932790667 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:35.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.546+0000 7ff12ffff640 1 --2- 172.21.15.90:0/932790667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe123e26b0 0x55fe127b2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.546+0000 7ff13cc95640 1 -- 172.21.15.90:0/932790667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe123e26b0 msgr2=0x55fe127b2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:35.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.546+0000 7ff13cc95640 1 --2- 172.21.15.90:0/932790667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe123e26b0 0x55fe127b2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.547+0000 7ff13cc95640 1 -- 172.21.15.90:0/932790667 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe12403390 msgr2=0x55fe127b31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:35.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.547+0000 7ff13cc95640 1 --2- 172.21.15.90:0/932790667 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe12403390 0x55fe127b31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.547+0000 7ff13cc95640 1 -- 172.21.15.90:0/932790667 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe127b3900 con 0x55fe123da4d0 2024-04-25T11:38:35.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.547+0000 7ff13cc95640 1 --2- 172.21.15.90:0/932790667 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe123da4d0 0x55fe127b7820 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff134005a60 tx=0x7ff134005730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3c072024ffbb654 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:35.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.547+0000 7ff117fff640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:38:35.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.547+0000 7ff117fff640 1 -- 172.21.15.90:0/932790667 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13406b020 con 0x55fe123da4d0 2024-04-25T11:38:35.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.547+0000 7ff117fff640 1 -- 172.21.15.90:0/932790667 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7ff1340316e0 con 0x55fe123da4d0 2024-04-25T11:38:35.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.547+0000 7ff145daf740 1 -- 172.21.15.90:0/932790667 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1340054f0 con 0x55fe123da4d0 2024-04-25T11:38:35.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.547+0000 7ff117fff640 1 -- 172.21.15.90:0/932790667 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff134040070 con 0x55fe123da4d0 2024-04-25T11:38:35.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.547+0000 7ff145daf740 10 client.25874 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:35.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.547+0000 7ff145daf740 1 -- 172.21.15.90:0/932790667 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fe12829a20 con 0x55fe123da4d0 2024-04-25T11:38:35.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.548+0000 7ff117fff640 1 -- 172.21.15.90:0/932790667 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1340054f0 con 0x55fe123da4d0 2024-04-25T11:38:35.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.548+0000 7ff117fff640 1 -- 172.21.15.90:0/932790667 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7ff134067bf0 con 0x55fe123da4d0 2024-04-25T11:38:35.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.548+0000 7ff145daf740 10 client.25874 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:35.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.548+0000 7ff145daf740 10 client.25874 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:38:35.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.548+0000 7ff145daf740 20 client.25874 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:35.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.548+0000 7ff145daf740 1 --2- 172.21.15.90:0/932790667 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55fe123973a0 0x55fe1282bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.548+0000 7ff145daf740 4 client.25874 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:38:35.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.548+0000 7ff145daf740 1 -- 172.21.15.90:0/932790667 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fe127a50e0 con 0x55fe123973a0 2024-04-25T11:38:35.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.548+0000 7ff12ffff640 1 --2- 172.21.15.90:0/932790667 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55fe123973a0 0x55fe1282bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.549+0000 7ff117fff640 10 client.25874 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:38:35.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.549+0000 7ff12ffff640 1 --2- 172.21.15.90:0/932790667 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55fe123973a0 0x55fe1282bcb0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fb42552292c11756 server_cookie=bf2cce51308a2959 in_seq=0 out_seq=0 2024-04-25T11:38:35.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.549+0000 7ff117fff640 1 -- 172.21.15.90:0/932790667 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fe127a50e0 con 0x55fe123973a0 2024-04-25T11:38:35.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.549+0000 7ff117fff640 10 client.25874 handle_command_reply: tid=0 2024-04-25T11:38:35.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.549+0000 7ff145daf740 1 client.25874 shutdown 2024-04-25T11:38:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.549+0000 7ff145daf740 1 -- 172.21.15.90:0/932790667 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe123da4d0 msgr2=0x55fe127b7820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.549+0000 7ff145daf740 1 --2- 172.21.15.90:0/932790667 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe123da4d0 0x55fe127b7820 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff134005a60 tx=0x7ff134005730 comp rx=0 tx=0).stop 2024-04-25T11:38:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.549+0000 7ff145daf740 1 -- 172.21.15.90:0/932790667 shutdown_connections 2024-04-25T11:38:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.549+0000 7ff145daf740 1 --2- 172.21.15.90:0/932790667 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55fe123973a0 0x55fe1282bcb0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.549+0000 7ff145daf740 1 --2- 172.21.15.90:0/932790667 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe12403390 0x55fe127b31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.549+0000 7ff145daf740 1 --2- 172.21.15.90:0/932790667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe123e26b0 0x55fe127b2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.549+0000 7ff145daf740 1 --2- 172.21.15.90:0/932790667 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe123da4d0 0x55fe127b7820 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.549+0000 7ff117fff640 0 client.25874 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:38:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.549+0000 7ff145daf740 1 -- 172.21.15.90:0/932790667 >> 172.21.15.90:0/932790667 conn(0x55fe123f74e0 msgr2=0x55fe1279f610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.549+0000 7ff145daf740 1 -- 172.21.15.90:0/932790667 shutdown_connections 2024-04-25T11:38:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.549+0000 7ff145daf740 1 -- 172.21.15.90:0/932790667 wait complete. 2024-04-25T11:38:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.549+0000 7ff145daf740 20 client.25874 trim_cache size 0 max 16384 2024-04-25T11:38:35.543 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:38:35.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.550+0000 7ff13eef9640 1 -- 172.21.15.90:0/264842710 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff1200afc60 msgr2=0x7ff1200b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:35.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.550+0000 7ff13eef9640 1 --2- 172.21.15.90:0/264842710 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff1200afc60 0x7ff1200b2170 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff134000c00 tx=0x7ff134031400 comp rx=0 tx=0).stop 2024-04-25T11:38:35.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.550+0000 7ff13eef9640 1 -- 172.21.15.90:0/264842710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff138108a20 msgr2=0x7ff138199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:35.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.550+0000 7ff13eef9640 1 --2- 172.21.15.90:0/264842710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff138108a20 0x7ff138199850 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff12800e9f0 tx=0x7ff12800eeb0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.550+0000 7ff13eef9640 1 -- 172.21.15.90:0/264842710 shutdown_connections 2024-04-25T11:38:35.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.550+0000 7ff13eef9640 1 --2- 172.21.15.90:0/264842710 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff1200afc60 0x7ff1200b2170 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.550+0000 7ff13eef9640 1 --2- 172.21.15.90:0/264842710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff13810d380 0x7ff138199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.550+0000 7ff13eef9640 1 --2- 172.21.15.90:0/264842710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff138108a20 0x7ff138199850 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.550+0000 7ff13eef9640 1 --2- 172.21.15.90:0/264842710 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff138102af0 0x7ff138199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.550+0000 7ff13eef9640 1 -- 172.21.15.90:0/264842710 >> 172.21.15.90:0/264842710 conn(0x7ff138078100 msgr2=0x7ff1381040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:35.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.550+0000 7ff13eef9640 1 -- 172.21.15.90:0/264842710 shutdown_connections 2024-04-25T11:38:35.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.550+0000 7ff13eef9640 1 -- 172.21.15.90:0/264842710 wait complete. 2024-04-25T11:38:35.551 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T11:38:35.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.682+0000 7fcb1c2bd640 1 -- 172.21.15.90:0/2351867629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb14109b30 msgr2=0x7fcb14103720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:35.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.682+0000 7fcb1c2bd640 1 --2- 172.21.15.90:0/2351867629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb14109b30 0x7fcb14103720 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fcb10009900 tx=0x7fcb1002a410 comp rx=0 tx=0).stop 2024-04-25T11:38:35.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.682+0000 7fcb1c2bd640 1 -- 172.21.15.90:0/2351867629 shutdown_connections 2024-04-25T11:38:35.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.682+0000 7fcb1c2bd640 1 --2- 172.21.15.90:0/2351867629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb141042d0 0x7fcb14110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.682+0000 7fcb1c2bd640 1 --2- 172.21.15.90:0/2351867629 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb14109fd0 0x7fcb14103c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.682+0000 7fcb1c2bd640 1 --2- 172.21.15.90:0/2351867629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb14109b30 0x7fcb14103720 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.682+0000 7fcb1c2bd640 1 -- 172.21.15.90:0/2351867629 >> 172.21.15.90:0/2351867629 conn(0x7fcb14079100 msgr2=0x7fcb14079540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:35.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.682+0000 7fcb1c2bd640 1 -- 172.21.15.90:0/2351867629 shutdown_connections 2024-04-25T11:38:35.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.682+0000 7fcb1c2bd640 1 -- 172.21.15.90:0/2351867629 wait complete. 2024-04-25T11:38:35.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.683+0000 7fcb1c2bd640 1 Processor -- start 2024-04-25T11:38:35.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.683+0000 7fcb1c2bd640 1 -- start start 2024-04-25T11:38:35.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.683+0000 7fcb1c2bd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb141042d0 0x7fcb1419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.683+0000 7fcb1c2bd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb14109b30 0x7fcb1419acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.683+0000 7fcb1c2bd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb14109fd0 0x7fcb1419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.683+0000 7fcb1c2bd640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcb14113d20 con 0x7fcb14109b30 2024-04-25T11:38:35.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.683+0000 7fcb1c2bd640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fcb14113ba0 con 0x7fcb14109fd0 2024-04-25T11:38:35.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.683+0000 7fcb1c2bd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcb14113ea0 con 0x7fcb141042d0 2024-04-25T11:38:35.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.683+0000 7fcb19858640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb14109b30 0x7fcb1419acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.683+0000 7fcb19858640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb14109b30 0x7fcb1419acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34802/0 (socket says 172.21.15.90:34802) 2024-04-25T11:38:35.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.683+0000 7fcb1a85a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb14109fd0 0x7fcb1419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.683+0000 7fcb19858640 1 -- 172.21.15.90:0/556102340 learned_addr learned my addr 172.21.15.90:0/556102340 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:35.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.684+0000 7fcb1a059640 1 --2- 172.21.15.90:0/556102340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb141042d0 0x7fcb1419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.684+0000 7fcb19858640 1 -- 172.21.15.90:0/556102340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb141042d0 msgr2=0x7fcb1419a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:35.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.684+0000 7fcb19858640 1 --2- 172.21.15.90:0/556102340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb141042d0 0x7fcb1419a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.684+0000 7fcb19858640 1 -- 172.21.15.90:0/556102340 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb14109fd0 msgr2=0x7fcb1419ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:35.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.684+0000 7fcb19858640 1 --2- 172.21.15.90:0/556102340 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb14109fd0 0x7fcb1419ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.684+0000 7fcb19858640 1 -- 172.21.15.90:0/556102340 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb1419f5c0 con 0x7fcb14109b30 2024-04-25T11:38:35.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.684+0000 7fcb1a85a640 1 --2- 172.21.15.90:0/556102340 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb14109fd0 0x7fcb1419ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:35.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.684+0000 7fcb19858640 1 --2- 172.21.15.90:0/556102340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb14109b30 0x7fcb1419acc0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fcb0400e9f0 tx=0x7fcb0400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:35.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.684+0000 7fcb0b7fe640 1 -- 172.21.15.90:0/556102340 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0400ccc0 con 0x7fcb14109b30 2024-04-25T11:38:35.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.684+0000 7fcb1c2bd640 1 -- 172.21.15.90:0/556102340 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb1419f8b0 con 0x7fcb14109b30 2024-04-25T11:38:35.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.684+0000 7fcb0b7fe640 1 -- 172.21.15.90:0/556102340 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb040035f0 con 0x7fcb14109b30 2024-04-25T11:38:35.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.684+0000 7fcb0b7fe640 1 -- 172.21.15.90:0/556102340 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0400ce60 con 0x7fcb14109b30 2024-04-25T11:38:35.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.684+0000 7fcb1c2bd640 1 -- 172.21.15.90:0/556102340 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb14074dd0 con 0x7fcb14109b30 2024-04-25T11:38:35.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.687+0000 7fcb0b7fe640 1 -- 172.21.15.90:0/556102340 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fcb04003aa0 con 0x7fcb14109b30 2024-04-25T11:38:35.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.688+0000 7fcb0b7fe640 1 --2- 172.21.15.90:0/556102340 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcb000afc60 0x7fcb000b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.688+0000 7fcb0b7fe640 1 -- 172.21.15.90:0/556102340 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fcb040df3a0 con 0x7fcb14109b30 2024-04-25T11:38:35.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.688+0000 7fcb1a059640 1 --2- 172.21.15.90:0/556102340 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcb000afc60 0x7fcb000b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.689+0000 7fcb1a059640 1 --2- 172.21.15.90:0/556102340 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcb000afc60 0x7fcb000b2170 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcb10000c00 tx=0x7fcb10031400 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:35.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.689+0000 7fcb23172740 1 Processor -- start 2024-04-25T11:38:35.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.689+0000 7fcb23172740 1 -- start start 2024-04-25T11:38:35.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.689+0000 7fcb23172740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55859bc0b390 0x55859bfac810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.689+0000 7fcb23172740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55859bbe24d0 0x55859bfacd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.689+0000 7fcb23172740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55859bbea6b0 0x55859bfb96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.689+0000 7fcb23172740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55859bf11d80 con 0x55859bc0b390 2024-04-25T11:38:35.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.689+0000 7fcb23172740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55859becba20 con 0x55859bbea6b0 2024-04-25T11:38:35.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.689+0000 7fcb23172740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55859bf0dd40 con 0x55859bbe24d0 2024-04-25T11:38:35.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.690+0000 7fcb1a059640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55859bc0b390 0x55859bfac810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.690+0000 7fcb1a059640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55859bc0b390 0x55859bfac810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34816/0 (socket says 172.21.15.90:34816) 2024-04-25T11:38:35.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.690+0000 7fcb1a059640 1 -- 172.21.15.90:0/4178534633 learned_addr learned my addr 172.21.15.90:0/4178534633 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:35.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.690+0000 7fcb19858640 1 --2- 172.21.15.90:0/4178534633 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55859bbe24d0 0x55859bfacd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.690+0000 7fcb1a85a640 1 --2- 172.21.15.90:0/4178534633 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55859bbea6b0 0x55859bfb96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.690+0000 7fcb1a059640 1 -- 172.21.15.90:0/4178534633 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55859bbe24d0 msgr2=0x55859bfacd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:35.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.690+0000 7fcb1a059640 1 --2- 172.21.15.90:0/4178534633 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55859bbe24d0 0x55859bfacd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.690+0000 7fcb1a059640 1 -- 172.21.15.90:0/4178534633 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55859bbea6b0 msgr2=0x55859bfb96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:35.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.690+0000 7fcb1a059640 1 --2- 172.21.15.90:0/4178534633 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55859bbea6b0 0x55859bfb96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.690+0000 7fcb1a059640 1 -- 172.21.15.90:0/4178534633 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55859bed6c30 con 0x55859bc0b390 2024-04-25T11:38:35.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.690+0000 7fcb19858640 1 --2- 172.21.15.90:0/4178534633 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55859bbe24d0 0x55859bfacd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:35.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.690+0000 7fcb1a059640 1 --2- 172.21.15.90:0/4178534633 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55859bc0b390 0x55859bfac810 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fcb10009900 tx=0x7fcb1003c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad96a70a45092623 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:35.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.690+0000 7fcb08bfe640 1 -- 172.21.15.90:0/4178534633 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb1004b070 con 0x55859bc0b390 2024-04-25T11:38:35.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.690+0000 7fcb08bfe640 1 -- 172.21.15.90:0/4178534633 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb100317a0 con 0x55859bc0b390 2024-04-25T11:38:35.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.690+0000 7fcb08bfe640 1 -- 172.21.15.90:0/4178534633 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb10035050 con 0x55859bc0b390 2024-04-25T11:38:35.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.691+0000 7fcb23172740 1 -- 172.21.15.90:0/4178534633 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55859bc0b390 msgr2=0x55859bfac810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:35.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.691+0000 7fcb23172740 1 --2- 172.21.15.90:0/4178534633 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55859bc0b390 0x55859bfac810 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fcb10009900 tx=0x7fcb1003c000 comp rx=0 tx=0).stop 2024-04-25T11:38:35.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.691+0000 7fcb23172740 1 -- 172.21.15.90:0/4178534633 shutdown_connections 2024-04-25T11:38:35.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.691+0000 7fcb23172740 1 --2- 172.21.15.90:0/4178534633 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55859bbea6b0 0x55859bfb96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.691+0000 7fcb23172740 1 --2- 172.21.15.90:0/4178534633 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55859bbe24d0 0x55859bfacd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.691+0000 7fcb23172740 1 --2- 172.21.15.90:0/4178534633 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55859bc0b390 0x55859bfac810 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.691+0000 7fcb23172740 1 -- 172.21.15.90:0/4178534633 >> 172.21.15.90:0/4178534633 conn(0x55859bbff4e0 msgr2=0x55859bfa8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:35.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.691+0000 7fcb23172740 1 -- 172.21.15.90:0/4178534633 shutdown_connections 2024-04-25T11:38:35.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.692+0000 7fcb23172740 1 -- 172.21.15.90:0/4178534633 wait complete. 2024-04-25T11:38:35.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.692+0000 7fcb23172740 1 -- start start 2024-04-25T11:38:35.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.692+0000 7fcb23172740 1 Processor -- start 2024-04-25T11:38:35.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.692+0000 7fcb23172740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55859bbe24d0 0x55859bfb7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.692+0000 7fcb23172740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55859bbea6b0 0x55859bfb77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.692+0000 7fcb23172740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55859bc0b390 0x55859bfc31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.692+0000 7fcb23172740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55859bfb9d30 con 0x55859bc0b390 2024-04-25T11:38:35.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.692+0000 7fcb23172740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55859be98dc0 con 0x55859bbea6b0 2024-04-25T11:38:35.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.692+0000 7fcb23172740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55859bfb9eb0 con 0x55859bbe24d0 2024-04-25T11:38:35.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.692+0000 7fcb23172740 10 client.0 _ll_register_callbacks cb 0x55859beb4600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:35.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.693+0000 7fcb19858640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55859bbea6b0 0x55859bfb77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.693+0000 7fcb1a85a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55859bc0b390 0x55859bfc31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.693+0000 7fcb19858640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55859bbea6b0 0x55859bfb77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:38904/0 (socket says 172.21.15.90:38904) 2024-04-25T11:38:35.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.693+0000 7fcb19858640 1 -- 172.21.15.90:0/1392962335 learned_addr learned my addr 172.21.15.90:0/1392962335 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:35.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.693+0000 7fcb1a059640 1 --2- 172.21.15.90:0/1392962335 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55859bbe24d0 0x55859bfb7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.693+0000 7fcb1a059640 1 -- 172.21.15.90:0/1392962335 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55859bbea6b0 msgr2=0x55859bfb77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:35.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.693+0000 7fcb1a059640 1 --2- 172.21.15.90:0/1392962335 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55859bbea6b0 0x55859bfb77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.693+0000 7fcb1a059640 1 -- 172.21.15.90:0/1392962335 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55859bc0b390 msgr2=0x55859bfc31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:35.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.693+0000 7fcb1a059640 1 --2- 172.21.15.90:0/1392962335 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55859bc0b390 0x55859bfc31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.693+0000 7fcb1a059640 1 -- 172.21.15.90:0/1392962335 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55859bc233f0 con 0x55859bbe24d0 2024-04-25T11:38:35.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.693+0000 7fcb1a85a640 1 --2- 172.21.15.90:0/1392962335 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55859bc0b390 0x55859bfc31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:35.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.693+0000 7fcb19858640 1 --2- 172.21.15.90:0/1392962335 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55859bbea6b0 0x55859bfb77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:35.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.693+0000 7fcb1a059640 1 --2- 172.21.15.90:0/1392962335 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55859bbe24d0 0x55859bfb7280 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcb10005940 tx=0x7fcb10031950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=804aaf3acd43e58c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:35.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.693+0000 7fcafdffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:38:35.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.694+0000 7fcafdffb640 1 -- 172.21.15.90:0/1392962335 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb10058070 con 0x55859bbe24d0 2024-04-25T11:38:35.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.694+0000 7fcafdffb640 1 -- 172.21.15.90:0/1392962335 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fcb10003fc0 con 0x55859bbe24d0 2024-04-25T11:38:35.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.694+0000 7fcb23172740 1 -- 172.21.15.90:0/1392962335 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb100317a0 con 0x55859bbe24d0 2024-04-25T11:38:35.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.694+0000 7fcafdffb640 1 -- 172.21.15.90:0/1392962335 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcb10064050 con 0x55859bbe24d0 2024-04-25T11:38:35.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.694+0000 7fcb23172740 10 client.25807 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:35.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.694+0000 7fcb23172740 1 -- 172.21.15.90:0/1392962335 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55859bced7d0 con 0x55859bbe24d0 2024-04-25T11:38:35.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.694+0000 7fcafdffb640 1 -- 172.21.15.90:0/1392962335 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb100317a0 con 0x55859bbe24d0 2024-04-25T11:38:35.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.694+0000 7fcafdffb640 1 -- 172.21.15.90:0/1392962335 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7fcb10047430 con 0x55859bbe24d0 2024-04-25T11:38:35.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.694+0000 7fcb23172740 10 client.25807 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:35.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.694+0000 7fcb23172740 10 client.25807 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:38:35.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.694+0000 7fcb23172740 20 client.25807 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:35.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.694+0000 7fcb23172740 1 --2- 172.21.15.90:0/1392962335 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55859bbc9f60 0x55859bfc94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.694+0000 7fcb23172740 4 client.25807 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:38:35.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.694+0000 7fcb23172740 1 -- 172.21.15.90:0/1392962335 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55859bf1f990 con 0x55859bbc9f60 2024-04-25T11:38:35.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.695+0000 7fcb19858640 1 --2- 172.21.15.90:0/1392962335 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55859bbc9f60 0x55859bfc94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.695+0000 7fcafdffb640 10 client.25807 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:38:35.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.695+0000 7fcb19858640 1 --2- 172.21.15.90:0/1392962335 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55859bbc9f60 0x55859bfc94b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8f3727581f15c5e2 server_cookie=8d497f1273478f51 in_seq=0 out_seq=0 2024-04-25T11:38:35.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.696+0000 7fcafdffb640 1 -- 172.21.15.90:0/1392962335 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55859bf1f990 con 0x55859bbc9f60 2024-04-25T11:38:35.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.696+0000 7fcafdffb640 10 client.25807 handle_command_reply: tid=0 2024-04-25T11:38:35.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.696+0000 7fcb23172740 1 client.25807 shutdown 2024-04-25T11:38:35.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.697+0000 7fcb23172740 1 -- 172.21.15.90:0/1392962335 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55859bbe24d0 msgr2=0x55859bfb7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:35.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.697+0000 7fcb23172740 1 --2- 172.21.15.90:0/1392962335 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55859bbe24d0 0x55859bfb7280 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcb10005940 tx=0x7fcb10031950 comp rx=0 tx=0).stop 2024-04-25T11:38:35.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.697+0000 7fcb23172740 1 -- 172.21.15.90:0/1392962335 shutdown_connections 2024-04-25T11:38:35.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.697+0000 7fcb23172740 1 --2- 172.21.15.90:0/1392962335 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55859bc0b390 0x55859bfc31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.697+0000 7fcb23172740 1 --2- 172.21.15.90:0/1392962335 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55859bbc9f60 0x55859bfc94b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.697+0000 7fcb23172740 1 --2- 172.21.15.90:0/1392962335 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55859bbea6b0 0x55859bfb77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.697+0000 7fcb23172740 1 --2- 172.21.15.90:0/1392962335 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55859bbe24d0 0x55859bfb7280 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.697+0000 7fcafdffb640 0 client.25807 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:38:35.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.697+0000 7fcb23172740 1 -- 172.21.15.90:0/1392962335 >> 172.21.15.90:0/1392962335 conn(0x55859bbff4e0 msgr2=0x55859bfae530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:35.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.697+0000 7fcb23172740 1 -- 172.21.15.90:0/1392962335 shutdown_connections 2024-04-25T11:38:35.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.697+0000 7fcb23172740 1 -- 172.21.15.90:0/1392962335 wait complete. 2024-04-25T11:38:35.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.697+0000 7fcb23172740 20 client.25807 trim_cache size 0 max 16384 2024-04-25T11:38:35.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.715+0000 7fcb23172740 1 Processor -- start 2024-04-25T11:38:35.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.715+0000 7fcb23172740 1 -- start start 2024-04-25T11:38:35.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.715+0000 7fcb23172740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55859bbc9f60 0x55859bfc31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.715+0000 7fcb23172740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55859bbe24d0 0x55859bfb6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.715+0000 7fcb23172740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55859bbea6b0 0x55859bfa7d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.715+0000 7fcb23172740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcb141b1e40 con 0x55859bbe24d0 2024-04-25T11:38:35.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.716+0000 7fcb23172740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fcb141b39b0 con 0x55859bbc9f60 2024-04-25T11:38:35.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.716+0000 7fcb23172740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcb141b2510 con 0x55859bbea6b0 2024-04-25T11:38:35.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.717+0000 7fcb1a85a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55859bbea6b0 0x55859bfa7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.717+0000 7fcb1a85a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55859bbea6b0 0x55859bfa7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:33548/0 (socket says 172.21.15.90:33548) 2024-04-25T11:38:35.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.717+0000 7fcb1a85a640 1 -- 172.21.15.90:0/879585419 learned_addr learned my addr 172.21.15.90:0/879585419 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:35.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.717+0000 7fcb19858640 1 --2- 172.21.15.90:0/879585419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55859bbe24d0 0x55859bfb6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.717+0000 7fcb1a85a640 1 -- 172.21.15.90:0/879585419 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55859bbc9f60 msgr2=0x55859bfc31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:35.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.717+0000 7fcb1a85a640 1 --2- 172.21.15.90:0/879585419 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55859bbc9f60 0x55859bfc31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.717+0000 7fcb1a85a640 1 -- 172.21.15.90:0/879585419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55859bbe24d0 msgr2=0x55859bfb6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:35.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.718+0000 7fcb1a85a640 1 --2- 172.21.15.90:0/879585419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55859bbe24d0 0x55859bfb6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.718+0000 7fcb1a85a640 1 -- 172.21.15.90:0/879585419 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55859bfb8530 con 0x55859bbea6b0 2024-04-25T11:38:35.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.718+0000 7fcb19858640 1 --2- 172.21.15.90:0/879585419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55859bbe24d0 0x55859bfb6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:35.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.718+0000 7fcb1a85a640 1 --2- 172.21.15.90:0/879585419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55859bbea6b0 0x55859bfa7d20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c00fd60 tx=0x7fcb0c009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f03b977a6fc14177 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:35.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.719+0000 7fcafffff640 1 -- 172.21.15.90:0/879585419 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c022070 con 0x55859bbea6b0 2024-04-25T11:38:35.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.719+0000 7fcafffff640 1 -- 172.21.15.90:0/879585419 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb0c003e20 con 0x55859bbea6b0 2024-04-25T11:38:35.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.719+0000 7fcafffff640 1 -- 172.21.15.90:0/879585419 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c004860 con 0x55859bbea6b0 2024-04-25T11:38:35.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.720+0000 7fcb23172740 1 -- 172.21.15.90:0/879585419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55859bbea6b0 msgr2=0x55859bfa7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:35.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.720+0000 7fcb23172740 1 --2- 172.21.15.90:0/879585419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55859bbea6b0 0x55859bfa7d20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c00fd60 tx=0x7fcb0c009500 comp rx=0 tx=0).stop 2024-04-25T11:38:35.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.720+0000 7fcb23172740 1 -- 172.21.15.90:0/879585419 shutdown_connections 2024-04-25T11:38:35.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.720+0000 7fcb23172740 1 --2- 172.21.15.90:0/879585419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55859bbea6b0 0x55859bfa7d20 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.720+0000 7fcb23172740 1 --2- 172.21.15.90:0/879585419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55859bbe24d0 0x55859bfb6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.720+0000 7fcb23172740 1 --2- 172.21.15.90:0/879585419 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55859bbc9f60 0x55859bfc31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.720+0000 7fcb23172740 1 -- 172.21.15.90:0/879585419 >> 172.21.15.90:0/879585419 conn(0x55859bbff4e0 msgr2=0x55859bfa7610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:35.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.721+0000 7fcb23172740 1 -- 172.21.15.90:0/879585419 shutdown_connections 2024-04-25T11:38:35.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.721+0000 7fcb23172740 1 -- 172.21.15.90:0/879585419 wait complete. 2024-04-25T11:38:35.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.721+0000 7fcb23172740 1 -- start start 2024-04-25T11:38:35.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.722+0000 7fcb23172740 1 Processor -- start 2024-04-25T11:38:35.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.722+0000 7fcb23172740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55859bbe24d0 0x55859bfc9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.722+0000 7fcb23172740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55859bbea6b0 0x55859bfc9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.722+0000 7fcb23172740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55859bc0b390 0x55859bfc9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.722+0000 7fcb23172740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcb141b32b0 con 0x55859bc0b390 2024-04-25T11:38:35.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.722+0000 7fcb23172740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fcb141b2be0 con 0x55859bbea6b0 2024-04-25T11:38:35.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.722+0000 7fcb23172740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcb141afe20 con 0x55859bbe24d0 2024-04-25T11:38:35.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.722+0000 7fcb23172740 10 client.0 _ll_register_callbacks cb 0x7fcb0c009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:35.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.722+0000 7fcb1a85a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55859bc0b390 0x55859bfc9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.722+0000 7fcb1a85a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55859bc0b390 0x55859bfc9c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34838/0 (socket says 172.21.15.90:34838) 2024-04-25T11:38:35.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.722+0000 7fcb1a85a640 1 -- 172.21.15.90:0/308210488 learned_addr learned my addr 172.21.15.90:0/308210488 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:35.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.722+0000 7fcb1a059640 1 --2- 172.21.15.90:0/308210488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55859bbe24d0 0x55859bfc9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.723+0000 7fcb1a059640 1 -- 172.21.15.90:0/308210488 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55859bbea6b0 msgr2=0x55859bfc9750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:38:35.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.723+0000 7fcb1a059640 1 --2- 172.21.15.90:0/308210488 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55859bbea6b0 0x55859bfc9750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.723+0000 7fcb1a059640 1 -- 172.21.15.90:0/308210488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55859bc0b390 msgr2=0x55859bfc9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:35.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.723+0000 7fcb1a059640 1 --2- 172.21.15.90:0/308210488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55859bc0b390 0x55859bfc9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.723+0000 7fcb1a059640 1 -- 172.21.15.90:0/308210488 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55859c048ec0 con 0x55859bbe24d0 2024-04-25T11:38:35.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.723+0000 7fcb1a059640 1 --2- 172.21.15.90:0/308210488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55859bbe24d0 0x55859bfc9210 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x55859bee66a0 tx=0x7fcb100314a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=531ee98ad872ff0a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:35.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.723+0000 7fcafdffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:38:35.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.723+0000 7fcafdffb640 1 -- 172.21.15.90:0/308210488 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb10061020 con 0x55859bbe24d0 2024-04-25T11:38:35.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.723+0000 7fcafdffb640 1 -- 172.21.15.90:0/308210488 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fcb10003dc0 con 0x55859bbe24d0 2024-04-25T11:38:35.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.723+0000 7fcb23172740 1 -- 172.21.15.90:0/308210488 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb0c003e20 con 0x55859bbe24d0 2024-04-25T11:38:35.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.724+0000 7fcafdffb640 1 -- 172.21.15.90:0/308210488 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcb1006d050 con 0x55859bbe24d0 2024-04-25T11:38:35.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.724+0000 7fcb23172740 10 client.25813 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:35.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.724+0000 7fcb23172740 1 -- 172.21.15.90:0/308210488 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55859c04a120 con 0x55859bbe24d0 2024-04-25T11:38:35.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.724+0000 7fcafdffb640 1 -- 172.21.15.90:0/308210488 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb0c003e20 con 0x55859bbe24d0 2024-04-25T11:38:35.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.724+0000 7fcafdffb640 1 -- 172.21.15.90:0/308210488 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7fcb10050450 con 0x55859bbe24d0 2024-04-25T11:38:35.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.724+0000 7fcb23172740 10 client.25813 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:35.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.724+0000 7fcb23172740 10 client.25813 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:38:35.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.724+0000 7fcb23172740 20 client.25813 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:35.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.724+0000 7fcb23172740 1 --2- 172.21.15.90:0/308210488 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55859bb9f3a0 0x55859c04c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.724+0000 7fcb23172740 4 client.25813 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:38:35.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.724+0000 7fcb23172740 1 -- 172.21.15.90:0/308210488 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55859bf19800 con 0x55859bb9f3a0 2024-04-25T11:38:35.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.725+0000 7fcb19858640 1 --2- 172.21.15.90:0/308210488 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55859bb9f3a0 0x55859c04c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.727+0000 7fcafdffb640 10 client.25813 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:38:35.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.727+0000 7fcb19858640 1 --2- 172.21.15.90:0/308210488 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55859bb9f3a0 0x55859c04c7e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=281b66769d5aee31 server_cookie=682b5ed24e1091cc in_seq=0 out_seq=0 2024-04-25T11:38:35.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.727+0000 7fcafdffb640 1 -- 172.21.15.90:0/308210488 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55859bf19800 con 0x55859bb9f3a0 2024-04-25T11:38:35.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.727+0000 7fcafdffb640 10 client.25813 handle_command_reply: tid=0 2024-04-25T11:38:35.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.727+0000 7fcb23172740 1 client.25813 shutdown 2024-04-25T11:38:35.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.727+0000 7fcb23172740 1 -- 172.21.15.90:0/308210488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55859bbe24d0 msgr2=0x55859bfc9210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:35.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.727+0000 7fcb23172740 1 --2- 172.21.15.90:0/308210488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55859bbe24d0 0x55859bfc9210 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x55859bee66a0 tx=0x7fcb100314a0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.727+0000 7fcb23172740 1 -- 172.21.15.90:0/308210488 shutdown_connections 2024-04-25T11:38:35.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.727+0000 7fcb23172740 1 --2- 172.21.15.90:0/308210488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55859bc0b390 0x55859bfc9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.727+0000 7fcb23172740 1 --2- 172.21.15.90:0/308210488 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55859bb9f3a0 0x55859c04c7e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.727+0000 7fcafdffb640 0 client.25813 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:38:35.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.727+0000 7fcb23172740 1 --2- 172.21.15.90:0/308210488 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55859bbea6b0 0x55859bfc9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.727+0000 7fcb23172740 1 --2- 172.21.15.90:0/308210488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55859bbe24d0 0x55859bfc9210 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x55859bee66a0 tx=0x7fcb100314a0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.727+0000 7fcb23172740 1 -- 172.21.15.90:0/308210488 >> 172.21.15.90:0/308210488 conn(0x55859bbff4e0 msgr2=0x55859bfa8040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:35.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.728+0000 7fcb23172740 1 -- 172.21.15.90:0/308210488 shutdown_connections 2024-04-25T11:38:35.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.728+0000 7fcb23172740 1 -- 172.21.15.90:0/308210488 wait complete. 2024-04-25T11:38:35.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.728+0000 7fcb23172740 20 client.25813 trim_cache size 0 max 16384 2024-04-25T11:38:35.718 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:38:35.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.729+0000 7fcb1c2bd640 1 -- 172.21.15.90:0/556102340 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcb000afc60 msgr2=0x7fcb000b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:35.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.729+0000 7fcb1c2bd640 1 --2- 172.21.15.90:0/556102340 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcb000afc60 0x7fcb000b2170 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcb10000c00 tx=0x7fcb10031400 comp rx=0 tx=0).stop 2024-04-25T11:38:35.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.729+0000 7fcb1c2bd640 1 -- 172.21.15.90:0/556102340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb14109b30 msgr2=0x7fcb1419acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:35.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.729+0000 7fcb1c2bd640 1 --2- 172.21.15.90:0/556102340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb14109b30 0x7fcb1419acc0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fcb0400e9f0 tx=0x7fcb0400eeb0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.729+0000 7fcb1c2bd640 1 -- 172.21.15.90:0/556102340 shutdown_connections 2024-04-25T11:38:35.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.729+0000 7fcb1c2bd640 1 --2- 172.21.15.90:0/556102340 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcb000afc60 0x7fcb000b2170 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.729+0000 7fcb1c2bd640 1 --2- 172.21.15.90:0/556102340 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb14109fd0 0x7fcb1419ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.729+0000 7fcb1c2bd640 1 --2- 172.21.15.90:0/556102340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb14109b30 0x7fcb1419acc0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.729+0000 7fcb1c2bd640 1 --2- 172.21.15.90:0/556102340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb141042d0 0x7fcb1419a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.729+0000 7fcb1c2bd640 1 -- 172.21.15.90:0/556102340 >> 172.21.15.90:0/556102340 conn(0x7fcb14079100 msgr2=0x7fcb141051d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:35.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.730+0000 7fcb1c2bd640 1 -- 172.21.15.90:0/556102340 shutdown_connections 2024-04-25T11:38:35.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:35.730+0000 7fcb1c2bd640 1 -- 172.21.15.90:0/556102340 wait complete. 2024-04-25T11:38:35.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:35 smithi090 ceph-mon[29575]: pgmap v108: 97 pgs: 97 active+clean; 28 GiB data, 85 GiB used, 988 GiB / 1.0 TiB avail; 2.4 MiB/s rd, 671 MiB/s wr, 738 op/s 2024-04-25T11:38:35.755 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T11:38:35.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:35 smithi138 ceph-mon[32328]: pgmap v108: 97 pgs: 97 active+clean; 28 GiB data, 85 GiB used, 988 GiB / 1.0 TiB avail; 2.4 MiB/s rd, 671 MiB/s wr, 738 op/s 2024-04-25T11:38:35.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:35 smithi156 ceph-mon[33045]: pgmap v108: 97 pgs: 97 active+clean; 28 GiB data, 85 GiB used, 988 GiB / 1.0 TiB avail; 2.4 MiB/s rd, 671 MiB/s wr, 738 op/s 2024-04-25T11:38:35.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.009+0000 7fca7abb4640 1 -- 172.21.15.90:0/632233485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca74109b30 msgr2=0x7fca74103720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:35.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.009+0000 7fca7abb4640 1 --2- 172.21.15.90:0/632233485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca74109b30 0x7fca74103720 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fca64009ca0 tx=0x7fca6402a8e0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.009+0000 7fca7abb4640 1 -- 172.21.15.90:0/632233485 shutdown_connections 2024-04-25T11:38:35.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.009+0000 7fca7abb4640 1 --2- 172.21.15.90:0/632233485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca741042d0 0x7fca74110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.009+0000 7fca7abb4640 1 --2- 172.21.15.90:0/632233485 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fca74109fd0 0x7fca74103c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.009+0000 7fca7abb4640 1 --2- 172.21.15.90:0/632233485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca74109b30 0x7fca74103720 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:35.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.009+0000 7fca7abb4640 1 -- 172.21.15.90:0/632233485 >> 172.21.15.90:0/632233485 conn(0x7fca74079280 msgr2=0x7fca740796c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:35.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.009+0000 7fca7abb4640 1 -- 172.21.15.90:0/632233485 shutdown_connections 2024-04-25T11:38:35.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.009+0000 7fca7abb4640 1 -- 172.21.15.90:0/632233485 wait complete. 2024-04-25T11:38:35.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.010+0000 7fca7abb4640 1 Processor -- start 2024-04-25T11:38:35.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.010+0000 7fca7abb4640 1 -- start start 2024-04-25T11:38:35.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.010+0000 7fca7abb4640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fca741042d0 0x7fca741a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.010+0000 7fca7abb4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca74109b30 0x7fca741a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.010+0000 7fca7abb4640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca74109fd0 0x7fca7419a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:35.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.011+0000 7fca7abb4640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fca74113e00 con 0x7fca74109fd0 2024-04-25T11:38:35.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.011+0000 7fca7abb4640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fca74113c80 con 0x7fca741042d0 2024-04-25T11:38:35.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.011+0000 7fca7abb4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fca74113f80 con 0x7fca74109b30 2024-04-25T11:38:35.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.011+0000 7fca79151640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca74109fd0 0x7fca7419a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:35.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.011+0000 7fca79151640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca74109fd0 0x7fca7419a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34854/0 (socket says 172.21.15.90:34854) 2024-04-25T11:38:36.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.011+0000 7fca79151640 1 -- 172.21.15.90:0/295029150 learned_addr learned my addr 172.21.15.90:0/295029150 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:36.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.011+0000 7fca78950640 1 --2- 172.21.15.90:0/295029150 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fca741042d0 0x7fca741a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.011+0000 7fca79151640 1 -- 172.21.15.90:0/295029150 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca74109b30 msgr2=0x7fca741a0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.011+0000 7fca79151640 1 --2- 172.21.15.90:0/295029150 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca74109b30 0x7fca741a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.011+0000 7fca79151640 1 -- 172.21.15.90:0/295029150 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fca741042d0 msgr2=0x7fca741a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.011+0000 7fca79151640 1 --2- 172.21.15.90:0/295029150 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fca741042d0 0x7fca741a05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.011+0000 7fca79151640 1 -- 172.21.15.90:0/295029150 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca7419ae50 con 0x7fca74109fd0 2024-04-25T11:38:36.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.012+0000 7fca79151640 1 --2- 172.21.15.90:0/295029150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca74109fd0 0x7fca7419a720 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fca6c009fd0 tx=0x7fca6c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:36.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.012+0000 7fca727fc640 1 -- 172.21.15.90:0/295029150 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6c011830 con 0x7fca74109fd0 2024-04-25T11:38:36.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.012+0000 7fca727fc640 1 -- 172.21.15.90:0/295029150 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca6c016070 con 0x7fca74109fd0 2024-04-25T11:38:36.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.012+0000 7fca727fc640 1 -- 172.21.15.90:0/295029150 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6c010de0 con 0x7fca74109fd0 2024-04-25T11:38:36.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.012+0000 7fca7abb4640 1 -- 172.21.15.90:0/295029150 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca7419b0e0 con 0x7fca74109fd0 2024-04-25T11:38:36.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.012+0000 7fca7abb4640 1 -- 172.21.15.90:0/295029150 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca74074dd0 con 0x7fca74109fd0 2024-04-25T11:38:36.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.016+0000 7fca727fc640 1 -- 172.21.15.90:0/295029150 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fca6c0119d0 con 0x7fca74109fd0 2024-04-25T11:38:36.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.017+0000 7fca81a61740 1 Processor -- start 2024-04-25T11:38:36.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.017+0000 7fca81a61740 1 -- start start 2024-04-25T11:38:36.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.017+0000 7fca81a61740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ddc7a67390 0x55ddc7e08810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.017+0000 7fca81a61740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ddc7a3e4d0 0x55ddc7e08d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.017+0000 7fca81a61740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a466b0 0x55ddc7e156a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.017+0000 7fca81a61740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ddc7d6dd80 con 0x55ddc7a466b0 2024-04-25T11:38:36.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.017+0000 7fca81a61740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ddc7d27a20 con 0x55ddc7a67390 2024-04-25T11:38:36.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.017+0000 7fca81a61740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ddc7d69d40 con 0x55ddc7a3e4d0 2024-04-25T11:38:36.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.018+0000 7fca6bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a466b0 0x55ddc7e156a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.018+0000 7fca6bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a466b0 0x55ddc7e156a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34856/0 (socket says 172.21.15.90:34856) 2024-04-25T11:38:36.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.018+0000 7fca6bfff640 1 -- 172.21.15.90:0/2540050678 learned_addr learned my addr 172.21.15.90:0/2540050678 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:36.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.018+0000 7fca727fc640 1 --2- 172.21.15.90:0/295029150 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fca600afce0 0x7fca600b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.018+0000 7fca6bfff640 1 -- 172.21.15.90:0/2540050678 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ddc7a3e4d0 msgr2=0x55ddc7e08d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:36.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.018+0000 7fca6bfff640 1 --2- 172.21.15.90:0/2540050678 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ddc7a3e4d0 0x55ddc7e08d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.018+0000 7fca6bfff640 1 -- 172.21.15.90:0/2540050678 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ddc7a67390 msgr2=0x55ddc7e08810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:36.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.018+0000 7fca6bfff640 1 --2- 172.21.15.90:0/2540050678 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ddc7a67390 0x55ddc7e08810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.018+0000 7fca6bfff640 1 -- 172.21.15.90:0/2540050678 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddc7d32c30 con 0x55ddc7a466b0 2024-04-25T11:38:36.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.019+0000 7fca79151640 1 --2- 172.21.15.90:0/295029150 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fca600afce0 0x7fca600b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.019+0000 7fca727fc640 1 -- 172.21.15.90:0/295029150 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fca6c0de680 con 0x7fca74109fd0 2024-04-25T11:38:36.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.019+0000 7fca79151640 1 --2- 172.21.15.90:0/295029150 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fca600afce0 0x7fca600b21f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0dc370 tx=0x7fca6c010800 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:36.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.020+0000 7fca6bfff640 1 --2- 172.21.15.90:0/2540050678 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a466b0 0x55ddc7e156a0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fca5c001730 tx=0x7fca5c001bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5177ef7e6c3a503b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:36.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.020+0000 7fca6affd640 1 -- 172.21.15.90:0/2540050678 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca5c010070 con 0x55ddc7a466b0 2024-04-25T11:38:36.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.020+0000 7fca6affd640 1 -- 172.21.15.90:0/2540050678 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca5c0068e0 con 0x55ddc7a466b0 2024-04-25T11:38:36.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.020+0000 7fca6affd640 1 -- 172.21.15.90:0/2540050678 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca5c007450 con 0x55ddc7a466b0 2024-04-25T11:38:36.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.021+0000 7fca81a61740 1 -- 172.21.15.90:0/2540050678 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a466b0 msgr2=0x55ddc7e156a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.021+0000 7fca81a61740 1 --2- 172.21.15.90:0/2540050678 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a466b0 0x55ddc7e156a0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fca5c001730 tx=0x7fca5c001bf0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.021+0000 7fca81a61740 1 -- 172.21.15.90:0/2540050678 shutdown_connections 2024-04-25T11:38:36.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.021+0000 7fca81a61740 1 --2- 172.21.15.90:0/2540050678 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a466b0 0x55ddc7e156a0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.021+0000 7fca81a61740 1 --2- 172.21.15.90:0/2540050678 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ddc7a3e4d0 0x55ddc7e08d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.021+0000 7fca81a61740 1 --2- 172.21.15.90:0/2540050678 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ddc7a67390 0x55ddc7e08810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.021+0000 7fca81a61740 1 -- 172.21.15.90:0/2540050678 >> 172.21.15.90:0/2540050678 conn(0x55ddc7a5b4e0 msgr2=0x55ddc7e04fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:36.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.022+0000 7fca81a61740 1 -- 172.21.15.90:0/2540050678 shutdown_connections 2024-04-25T11:38:36.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.022+0000 7fca81a61740 1 -- 172.21.15.90:0/2540050678 wait complete. 2024-04-25T11:38:36.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.024+0000 7fca81a61740 1 -- start start 2024-04-25T11:38:36.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.024+0000 7fca81a61740 1 Processor -- start 2024-04-25T11:38:36.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.026+0000 7fca81a61740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ddc7a3e4d0 0x55ddc7e132f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.026+0000 7fca81a61740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ddc7a466b0 0x55ddc7e13830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.026+0000 7fca81a61740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a67390 0x55ddc7e1f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.027+0000 7fca78950640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ddc7a466b0 0x55ddc7e13830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.027+0000 7fca81a61740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ddc7e15d90 con 0x55ddc7a67390 2024-04-25T11:38:36.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.027+0000 7fca81a61740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ddc7cf4dc0 con 0x55ddc7a3e4d0 2024-04-25T11:38:36.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.027+0000 7fca81a61740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ddc7e15f10 con 0x55ddc7a466b0 2024-04-25T11:38:36.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.027+0000 7fca81a61740 10 client.0 _ll_register_callbacks cb 0x55ddc7d10600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:36.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.027+0000 7fca6bfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ddc7a3e4d0 0x55ddc7e132f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.028+0000 7fca78950640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ddc7a466b0 0x55ddc7e13830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:33600/0 (socket says 172.21.15.90:33600) 2024-04-25T11:38:36.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.028+0000 7fca78950640 1 -- 172.21.15.90:0/876754581 learned_addr learned my addr 172.21.15.90:0/876754581 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:36.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.028+0000 7fca6bfff640 1 --2- 172.21.15.90:0/876754581 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a67390 0x55ddc7e1f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.028+0000 7fca6bfff640 1 -- 172.21.15.90:0/876754581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ddc7a466b0 msgr2=0x55ddc7e13830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:36.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.028+0000 7fca6bfff640 1 --2- 172.21.15.90:0/876754581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ddc7a466b0 0x55ddc7e13830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.028+0000 7fca6bfff640 1 -- 172.21.15.90:0/876754581 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ddc7a3e4d0 msgr2=0x55ddc7e132f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:36.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.028+0000 7fca6bfff640 1 --2- 172.21.15.90:0/876754581 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ddc7a3e4d0 0x55ddc7e132f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.028+0000 7fca6bfff640 1 -- 172.21.15.90:0/876754581 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddc7a7f3f0 con 0x55ddc7a67390 2024-04-25T11:38:36.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.028+0000 7fca6bfff640 1 --2- 172.21.15.90:0/876754581 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a67390 0x55ddc7e1f250 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fca5c007dc0 tx=0x7fca5c000e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48022a6dec57de5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:36.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.028+0000 7fca53fff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:38:36.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.029+0000 7fca53fff640 1 -- 172.21.15.90:0/876754581 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca5c026020 con 0x55ddc7a67390 2024-04-25T11:38:36.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.029+0000 7fca53fff640 1 -- 172.21.15.90:0/876754581 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fca5c004220 con 0x55ddc7a67390 2024-04-25T11:38:36.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.029+0000 7fca81a61740 1 -- 172.21.15.90:0/876754581 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca5c0068e0 con 0x55ddc7a67390 2024-04-25T11:38:36.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.029+0000 7fca53fff640 1 -- 172.21.15.90:0/876754581 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fca5c02e050 con 0x55ddc7a67390 2024-04-25T11:38:36.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.029+0000 7fca81a61740 10 client.16641 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:36.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.029+0000 7fca81a61740 1 -- 172.21.15.90:0/876754581 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ddc7b497d0 con 0x55ddc7a67390 2024-04-25T11:38:36.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.029+0000 7fca53fff640 1 -- 172.21.15.90:0/876754581 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca5c0068e0 con 0x55ddc7a67390 2024-04-25T11:38:36.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.030+0000 7fca53fff640 1 -- 172.21.15.90:0/876754581 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7fca5c003850 con 0x55ddc7a67390 2024-04-25T11:38:36.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.030+0000 7fca81a61740 10 client.16641 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:36.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.030+0000 7fca81a61740 10 client.16641 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:38:36.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.030+0000 7fca81a61740 20 client.16641 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:36.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.030+0000 7fca81a61740 1 --2- 172.21.15.90:0/876754581 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55ddc7a25f60 0x55ddc7e25610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.030+0000 7fca81a61740 4 client.16641 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:38:36.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.030+0000 7fca81a61740 1 -- 172.21.15.90:0/876754581 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ddc7d7b990 con 0x55ddc7a25f60 2024-04-25T11:38:36.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.031+0000 7fca78950640 1 --2- 172.21.15.90:0/876754581 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55ddc7a25f60 0x55ddc7e25610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.035+0000 7fca78950640 1 --2- 172.21.15.90:0/876754581 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55ddc7a25f60 0x55ddc7e25610 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=548b471935a5f731 server_cookie=f16c0edf14dc333c in_seq=0 out_seq=0 2024-04-25T11:38:36.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.035+0000 7fca53fff640 10 client.16641 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:38:36.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.037+0000 7fca53fff640 1 -- 172.21.15.90:0/876754581 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ddc7d7b990 con 0x55ddc7a25f60 2024-04-25T11:38:36.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.037+0000 7fca53fff640 10 client.16641 handle_command_reply: tid=0 2024-04-25T11:38:36.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.039+0000 7fca81a61740 1 client.16641 shutdown 2024-04-25T11:38:36.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.039+0000 7fca81a61740 1 -- 172.21.15.90:0/876754581 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a67390 msgr2=0x55ddc7e1f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.039+0000 7fca81a61740 1 --2- 172.21.15.90:0/876754581 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a67390 0x55ddc7e1f250 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fca5c007dc0 tx=0x7fca5c000e40 comp rx=0 tx=0).stop 2024-04-25T11:38:36.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.039+0000 7fca81a61740 1 -- 172.21.15.90:0/876754581 shutdown_connections 2024-04-25T11:38:36.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.039+0000 7fca81a61740 1 --2- 172.21.15.90:0/876754581 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55ddc7a25f60 0x55ddc7e25610 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.039+0000 7fca53fff640 0 client.16641 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:38:36.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.039+0000 7fca81a61740 1 --2- 172.21.15.90:0/876754581 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a67390 0x55ddc7e1f250 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.039+0000 7fca81a61740 1 --2- 172.21.15.90:0/876754581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ddc7a466b0 0x55ddc7e13830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.039+0000 7fca81a61740 1 --2- 172.21.15.90:0/876754581 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ddc7a3e4d0 0x55ddc7e132f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.039+0000 7fca81a61740 1 -- 172.21.15.90:0/876754581 >> 172.21.15.90:0/876754581 conn(0x55ddc7a5b4e0 msgr2=0x55ddc7e0a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:36.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.039+0000 7fca81a61740 1 -- 172.21.15.90:0/876754581 shutdown_connections 2024-04-25T11:38:36.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.039+0000 7fca81a61740 1 -- 172.21.15.90:0/876754581 wait complete. 2024-04-25T11:38:36.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.039+0000 7fca81a61740 20 client.16641 trim_cache size 0 max 16384 2024-04-25T11:38:36.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.059+0000 7fca81a61740 1 Processor -- start 2024-04-25T11:38:36.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.059+0000 7fca81a61740 1 -- start start 2024-04-25T11:38:36.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.060+0000 7fca81a61740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a25f60 0x55ddc7e0d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.060+0000 7fca81a61740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ddc7a3e4d0 0x55ddc7e0d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.060+0000 7fca81a61740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ddc7a466b0 0x55ddc7e10320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.060+0000 7fca81a61740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fca741b1fe0 con 0x55ddc7a25f60 2024-04-25T11:38:36.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.060+0000 7fca81a61740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fca741b3b50 con 0x55ddc7a466b0 2024-04-25T11:38:36.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.060+0000 7fca81a61740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fca741b26b0 con 0x55ddc7a3e4d0 2024-04-25T11:38:36.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.060+0000 7fca6bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a25f60 0x55ddc7e0d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.060+0000 7fca6bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a25f60 0x55ddc7e0d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34878/0 (socket says 172.21.15.90:34878) 2024-04-25T11:38:36.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.060+0000 7fca6bfff640 1 -- 172.21.15.90:0/2745479013 learned_addr learned my addr 172.21.15.90:0/2745479013 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:36.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.061+0000 7fca78950640 1 --2- 172.21.15.90:0/2745479013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ddc7a3e4d0 0x55ddc7e0d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.061+0000 7fca6bfff640 1 -- 172.21.15.90:0/2745479013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ddc7a3e4d0 msgr2=0x55ddc7e0d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:36.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.061+0000 7fca6bfff640 1 --2- 172.21.15.90:0/2745479013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ddc7a3e4d0 0x55ddc7e0d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.061+0000 7fca6bfff640 1 -- 172.21.15.90:0/2745479013 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ddc7a466b0 msgr2=0x55ddc7e10320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:36.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.061+0000 7fca6bfff640 1 --2- 172.21.15.90:0/2745479013 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ddc7a466b0 0x55ddc7e10320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.061+0000 7fca6bfff640 1 -- 172.21.15.90:0/2745479013 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddc7dff2b0 con 0x55ddc7a25f60 2024-04-25T11:38:36.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.061+0000 7fca6bfff640 1 --2- 172.21.15.90:0/2745479013 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a25f60 0x55ddc7e0d430 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fca5c00e530 tx=0x7fca5c009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6838dba138fee2b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:36.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.061+0000 7fca6a7fc640 1 -- 172.21.15.90:0/2745479013 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca5c026040 con 0x55ddc7a25f60 2024-04-25T11:38:36.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.061+0000 7fca6a7fc640 1 -- 172.21.15.90:0/2745479013 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca5c003e60 con 0x55ddc7a25f60 2024-04-25T11:38:36.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.061+0000 7fca6a7fc640 1 -- 172.21.15.90:0/2745479013 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca5c010070 con 0x55ddc7a25f60 2024-04-25T11:38:36.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.062+0000 7fca81a61740 1 -- 172.21.15.90:0/2745479013 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a25f60 msgr2=0x55ddc7e0d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.062+0000 7fca81a61740 1 --2- 172.21.15.90:0/2745479013 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a25f60 0x55ddc7e0d430 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fca5c00e530 tx=0x7fca5c009f10 comp rx=0 tx=0).stop 2024-04-25T11:38:36.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.062+0000 7fca81a61740 1 -- 172.21.15.90:0/2745479013 shutdown_connections 2024-04-25T11:38:36.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.064+0000 7fca81a61740 1 --2- 172.21.15.90:0/2745479013 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ddc7a466b0 0x55ddc7e10320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.064+0000 7fca81a61740 1 --2- 172.21.15.90:0/2745479013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ddc7a3e4d0 0x55ddc7e0d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.064+0000 7fca81a61740 1 --2- 172.21.15.90:0/2745479013 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a25f60 0x55ddc7e0d430 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.064+0000 7fca81a61740 1 -- 172.21.15.90:0/2745479013 >> 172.21.15.90:0/2745479013 conn(0x55ddc7a5b4e0 msgr2=0x55ddc7e04040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:36.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.065+0000 7fca81a61740 1 -- 172.21.15.90:0/2745479013 shutdown_connections 2024-04-25T11:38:36.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.066+0000 7fca81a61740 1 -- 172.21.15.90:0/2745479013 wait complete. 2024-04-25T11:38:36.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.066+0000 7fca81a61740 1 -- start start 2024-04-25T11:38:36.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.066+0000 7fca81a61740 1 Processor -- start 2024-04-25T11:38:36.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.067+0000 7fca81a61740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a25f60 0x55ddc7e27970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.067+0000 7fca81a61740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ddc7a3e4d0 0x55ddc7e27eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.067+0000 7fca81a61740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ddc7a466b0 0x55ddc7e8b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.067+0000 7fca81a61740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fca741b3450 con 0x55ddc7a25f60 2024-04-25T11:38:36.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.067+0000 7fca81a61740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fca741b2d80 con 0x55ddc7a466b0 2024-04-25T11:38:36.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.067+0000 7fca81a61740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fca741affc0 con 0x55ddc7a3e4d0 2024-04-25T11:38:36.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.067+0000 7fca81a61740 10 client.0 _ll_register_callbacks cb 0x7fca5c001630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:36.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.067+0000 7fca6bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a25f60 0x55ddc7e27970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.067+0000 7fca78950640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ddc7a3e4d0 0x55ddc7e27eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.067+0000 7fca78950640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ddc7a3e4d0 0x55ddc7e27eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:33626/0 (socket says 172.21.15.90:33626) 2024-04-25T11:38:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.067+0000 7fca78950640 1 -- 172.21.15.90:0/3525123611 learned_addr learned my addr 172.21.15.90:0/3525123611 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.067+0000 7fca6bfff640 1 --2- 172.21.15.90:0/3525123611 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a25f60 0x55ddc7e27970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34886/0 (socket says 172.21.15.90:34886) 2024-04-25T11:38:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.067+0000 7fca6bfff640 1 -- 172.21.15.90:0/3525123611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ddc7a3e4d0 msgr2=0x55ddc7e27eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.067+0000 7fca6bfff640 1 --2- 172.21.15.90:0/3525123611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ddc7a3e4d0 0x55ddc7e27eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.067+0000 7fca6bfff640 1 -- 172.21.15.90:0/3525123611 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ddc7a466b0 msgr2=0x55ddc7e8b740 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:38:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.067+0000 7fca6bfff640 1 --2- 172.21.15.90:0/3525123611 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ddc7a466b0 0x55ddc7e8b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.067+0000 7fca6bfff640 1 -- 172.21.15.90:0/3525123611 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddc7e085a0 con 0x55ddc7a25f60 2024-04-25T11:38:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.068+0000 7fca6bfff640 1 --2- 172.21.15.90:0/3525123611 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a25f60 0x55ddc7e27970 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fca5c003de0 tx=0x7fca5c0014b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e18c69ba0cc0017 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.068+0000 7fca53fff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:38:36.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.068+0000 7fca53fff640 1 -- 172.21.15.90:0/3525123611 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca5c03e020 con 0x55ddc7a25f60 2024-04-25T11:38:36.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.068+0000 7fca53fff640 1 -- 172.21.15.90:0/3525123611 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fca5c004010 con 0x55ddc7a25f60 2024-04-25T11:38:36.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.068+0000 7fca81a61740 1 -- 172.21.15.90:0/3525123611 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca5c003e60 con 0x55ddc7a25f60 2024-04-25T11:38:36.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.068+0000 7fca53fff640 1 -- 172.21.15.90:0/3525123611 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fca5c010030 con 0x55ddc7a25f60 2024-04-25T11:38:36.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.068+0000 7fca81a61740 10 client.16647 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:36.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.068+0000 7fca81a61740 1 -- 172.21.15.90:0/3525123611 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ddc7e8cce0 con 0x55ddc7a25f60 2024-04-25T11:38:36.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.068+0000 7fca53fff640 1 -- 172.21.15.90:0/3525123611 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca5c003e60 con 0x55ddc7a25f60 2024-04-25T11:38:36.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.071+0000 7fca53fff640 1 -- 172.21.15.90:0/3525123611 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7fca5c039040 con 0x55ddc7a25f60 2024-04-25T11:38:36.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.071+0000 7fca81a61740 10 client.16647 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:36.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.071+0000 7fca81a61740 10 client.16647 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:38:36.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.071+0000 7fca81a61740 20 client.16647 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:36.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.071+0000 7fca81a61740 1 --2- 172.21.15.90:0/3525123611 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55ddc7a67390 0x55ddc7e8f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.071+0000 7fca81a61740 4 client.16647 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:38:36.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.071+0000 7fca81a61740 1 -- 172.21.15.90:0/3525123611 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ddc7d75800 con 0x55ddc7a67390 2024-04-25T11:38:36.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.071+0000 7fca78950640 1 --2- 172.21.15.90:0/3525123611 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55ddc7a67390 0x55ddc7e8f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.072+0000 7fca53fff640 10 client.16647 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:38:36.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.072+0000 7fca78950640 1 --2- 172.21.15.90:0/3525123611 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55ddc7a67390 0x55ddc7e8f370 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6684f70facde0a62 server_cookie=8a5d1f58a5f7b28d in_seq=0 out_seq=0 2024-04-25T11:38:36.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.073+0000 7fca53fff640 1 -- 172.21.15.90:0/3525123611 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ddc7d75800 con 0x55ddc7a67390 2024-04-25T11:38:36.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.073+0000 7fca53fff640 10 client.16647 handle_command_reply: tid=0 2024-04-25T11:38:36.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.073+0000 7fca81a61740 1 client.16647 shutdown 2024-04-25T11:38:36.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.073+0000 7fca81a61740 1 -- 172.21.15.90:0/3525123611 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a25f60 msgr2=0x55ddc7e27970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.073+0000 7fca81a61740 1 --2- 172.21.15.90:0/3525123611 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a25f60 0x55ddc7e27970 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fca5c003de0 tx=0x7fca5c0014b0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.073+0000 7fca81a61740 1 -- 172.21.15.90:0/3525123611 shutdown_connections 2024-04-25T11:38:36.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.073+0000 7fca81a61740 1 --2- 172.21.15.90:0/3525123611 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55ddc7a67390 0x55ddc7e8f370 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.073+0000 7fca81a61740 1 --2- 172.21.15.90:0/3525123611 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ddc7a466b0 0x55ddc7e8b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.073+0000 7fca53fff640 0 client.16647 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:38:36.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.073+0000 7fca81a61740 1 --2- 172.21.15.90:0/3525123611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ddc7a3e4d0 0x55ddc7e27eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.073+0000 7fca81a61740 1 --2- 172.21.15.90:0/3525123611 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddc7a25f60 0x55ddc7e27970 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.073+0000 7fca81a61740 1 -- 172.21.15.90:0/3525123611 >> 172.21.15.90:0/3525123611 conn(0x55ddc7a5b4e0 msgr2=0x55ddc7e14790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:36.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.073+0000 7fca81a61740 1 -- 172.21.15.90:0/3525123611 shutdown_connections 2024-04-25T11:38:36.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.073+0000 7fca81a61740 1 -- 172.21.15.90:0/3525123611 wait complete. 2024-04-25T11:38:36.062 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:38:36.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.073+0000 7fca81a61740 20 client.16647 trim_cache size 0 max 16384 2024-04-25T11:38:36.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.075+0000 7fca7abb4640 1 -- 172.21.15.90:0/295029150 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fca600afce0 msgr2=0x7fca600b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.075+0000 7fca7abb4640 1 --2- 172.21.15.90:0/295029150 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fca600afce0 0x7fca600b21f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0dc370 tx=0x7fca6c010800 comp rx=0 tx=0).stop 2024-04-25T11:38:36.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.075+0000 7fca7abb4640 1 -- 172.21.15.90:0/295029150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca74109fd0 msgr2=0x7fca7419a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.075+0000 7fca7abb4640 1 --2- 172.21.15.90:0/295029150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca74109fd0 0x7fca7419a720 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fca6c009fd0 tx=0x7fca6c00b8e0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.076+0000 7fca7abb4640 1 -- 172.21.15.90:0/295029150 shutdown_connections 2024-04-25T11:38:36.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.076+0000 7fca7abb4640 1 --2- 172.21.15.90:0/295029150 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fca600afce0 0x7fca600b21f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.076+0000 7fca7abb4640 1 --2- 172.21.15.90:0/295029150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca74109fd0 0x7fca7419a720 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.076+0000 7fca7abb4640 1 --2- 172.21.15.90:0/295029150 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca74109b30 0x7fca741a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.076+0000 7fca7abb4640 1 --2- 172.21.15.90:0/295029150 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fca741042d0 0x7fca741a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.076+0000 7fca7abb4640 1 -- 172.21.15.90:0/295029150 >> 172.21.15.90:0/295029150 conn(0x7fca74079280 msgr2=0x7fca741051d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:36.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.076+0000 7fca7abb4640 1 -- 172.21.15.90:0/295029150 shutdown_connections 2024-04-25T11:38:36.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.076+0000 7fca7abb4640 1 -- 172.21.15.90:0/295029150 wait complete. 2024-04-25T11:38:36.083 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T11:38:36.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.235+0000 7fe570334640 1 -- 172.21.15.90:0/3904324745 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe568109b10 msgr2=0x7fe568103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.235+0000 7fe570334640 1 --2- 172.21.15.90:0/3904324745 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe568109b10 0x7fe568103700 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fe564009900 tx=0x7fe56402a410 comp rx=0 tx=0).stop 2024-04-25T11:38:36.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.235+0000 7fe570334640 1 -- 172.21.15.90:0/3904324745 shutdown_connections 2024-04-25T11:38:36.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.235+0000 7fe570334640 1 --2- 172.21.15.90:0/3904324745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5681042b0 0x7fe568110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.235+0000 7fe570334640 1 --2- 172.21.15.90:0/3904324745 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe568109fb0 0x7fe568103c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.235+0000 7fe570334640 1 --2- 172.21.15.90:0/3904324745 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe568109b10 0x7fe568103700 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.235+0000 7fe570334640 1 -- 172.21.15.90:0/3904324745 >> 172.21.15.90:0/3904324745 conn(0x7fe5680ff360 msgr2=0x7fe568101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:36.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.235+0000 7fe570334640 1 -- 172.21.15.90:0/3904324745 shutdown_connections 2024-04-25T11:38:36.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.235+0000 7fe570334640 1 -- 172.21.15.90:0/3904324745 wait complete. 2024-04-25T11:38:36.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.236+0000 7fe570334640 1 Processor -- start 2024-04-25T11:38:36.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.236+0000 7fe570334640 1 -- start start 2024-04-25T11:38:36.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.236+0000 7fe570334640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5681042b0 0x7fe56819a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.236+0000 7fe570334640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe568109b10 0x7fe56819ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.236+0000 7fe570334640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe568109fb0 0x7fe56819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.236+0000 7fe570334640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe568113d60 con 0x7fe568109b10 2024-04-25T11:38:36.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.236+0000 7fe570334640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe568113be0 con 0x7fe568109fb0 2024-04-25T11:38:36.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.236+0000 7fe570334640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe568113ee0 con 0x7fe5681042b0 2024-04-25T11:38:36.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.236+0000 7fe56d8cf640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe568109b10 0x7fe56819ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.236+0000 7fe56d8cf640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe568109b10 0x7fe56819ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34898/0 (socket says 172.21.15.90:34898) 2024-04-25T11:38:36.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.236+0000 7fe56d8cf640 1 -- 172.21.15.90:0/2062475949 learned_addr learned my addr 172.21.15.90:0/2062475949 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:36.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.236+0000 7fe56e0d0640 1 --2- 172.21.15.90:0/2062475949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5681042b0 0x7fe56819a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.236+0000 7fe56e8d1640 1 --2- 172.21.15.90:0/2062475949 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe568109fb0 0x7fe56819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.237+0000 7fe56d8cf640 1 -- 172.21.15.90:0/2062475949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5681042b0 msgr2=0x7fe56819a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.237+0000 7fe56d8cf640 1 --2- 172.21.15.90:0/2062475949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5681042b0 0x7fe56819a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.237+0000 7fe56d8cf640 1 -- 172.21.15.90:0/2062475949 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe568109fb0 msgr2=0x7fe56819efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.237+0000 7fe56d8cf640 1 --2- 172.21.15.90:0/2062475949 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe568109fb0 0x7fe56819efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.237+0000 7fe56d8cf640 1 -- 172.21.15.90:0/2062475949 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe56819f600 con 0x7fe568109b10 2024-04-25T11:38:36.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.237+0000 7fe56d8cf640 1 --2- 172.21.15.90:0/2062475949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe568109b10 0x7fe56819ad00 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fe55800e9f0 tx=0x7fe55800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:36.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.237+0000 7fe55f7fe640 1 -- 172.21.15.90:0/2062475949 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe55800ccc0 con 0x7fe568109b10 2024-04-25T11:38:36.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.237+0000 7fe570334640 1 -- 172.21.15.90:0/2062475949 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe56819f8f0 con 0x7fe568109b10 2024-04-25T11:38:36.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.237+0000 7fe55f7fe640 1 -- 172.21.15.90:0/2062475949 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5580035f0 con 0x7fe568109b10 2024-04-25T11:38:36.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.237+0000 7fe55f7fe640 1 -- 172.21.15.90:0/2062475949 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe55800ce60 con 0x7fe568109b10 2024-04-25T11:38:36.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.237+0000 7fe570334640 1 -- 172.21.15.90:0/2062475949 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5681a7270 con 0x7fe568109b10 2024-04-25T11:38:36.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.240+0000 7fe55f7fe640 1 -- 172.21.15.90:0/2062475949 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fe558010690 con 0x7fe568109b10 2024-04-25T11:38:36.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.240+0000 7fe5771e1740 1 Processor -- start 2024-04-25T11:38:36.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.240+0000 7fe5771e1740 1 -- start start 2024-04-25T11:38:36.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.241+0000 7fe5771e1740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5559ec6eb390 0x5559eca8c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.241+0000 7fe5771e1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559ec6c24d0 0x5559eca8cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.241+0000 7fe5771e1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559ec6ca6b0 0x5559eca996a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.241+0000 7fe5771e1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5559ec9f1d80 con 0x5559ec6ca6b0 2024-04-25T11:38:36.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.241+0000 7fe5771e1740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5559ec9aba20 con 0x5559ec6eb390 2024-04-25T11:38:36.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.241+0000 7fe5771e1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5559ec9edd40 con 0x5559ec6c24d0 2024-04-25T11:38:36.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.241+0000 7fe56d8cf640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559ec6ca6b0 0x5559eca996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.241+0000 7fe56e8d1640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5559ec6eb390 0x5559eca8c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.241+0000 7fe56e8d1640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5559ec6eb390 0x5559eca8c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:39002/0 (socket says 172.21.15.90:39002) 2024-04-25T11:38:36.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.241+0000 7fe56d8cf640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559ec6ca6b0 0x5559eca996a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34912/0 (socket says 172.21.15.90:34912) 2024-04-25T11:38:36.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.241+0000 7fe56e8d1640 1 -- 172.21.15.90:0/2990131576 learned_addr learned my addr 172.21.15.90:0/2990131576 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:36.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.241+0000 7fe56e0d0640 1 --2- 172.21.15.90:0/2990131576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559ec6c24d0 0x5559eca8cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.241+0000 7fe55f7fe640 1 --2- 172.21.15.90:0/2062475949 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe5540afc60 0x7fe5540b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.241+0000 7fe56e8d1640 1 -- 172.21.15.90:0/2990131576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559ec6c24d0 msgr2=0x5559eca8cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:36.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.241+0000 7fe56e8d1640 1 --2- 172.21.15.90:0/2990131576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559ec6c24d0 0x5559eca8cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.241+0000 7fe56e8d1640 1 -- 172.21.15.90:0/2990131576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559ec6ca6b0 msgr2=0x5559eca996a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:36.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.241+0000 7fe56e8d1640 1 --2- 172.21.15.90:0/2990131576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559ec6ca6b0 0x5559eca996a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.241+0000 7fe56e8d1640 1 -- 172.21.15.90:0/2990131576 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559ec9b6c30 con 0x5559ec6eb390 2024-04-25T11:38:36.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.241+0000 7fe56e8d1640 1 --2- 172.21.15.90:0/2062475949 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe5540afc60 0x7fe5540b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.241+0000 7fe55f7fe640 1 -- 172.21.15.90:0/2062475949 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fe5580df3e0 con 0x7fe568109b10 2024-04-25T11:38:36.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.242+0000 7fe56e8d1640 1 --2- 172.21.15.90:0/2990131576 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5559ec6eb390 0x5559eca8c810 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe56000a5d0 tx=0x7fe56000aa90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d4f44f8a935cb1b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:36.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.242+0000 7fe56e8d1640 1 --2- 172.21.15.90:0/2062475949 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe5540afc60 0x7fe5540b2170 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe560020000 tx=0x7fe560021000 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:36.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.242+0000 7fe55cbfe640 1 -- 172.21.15.90:0/2990131576 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56001a040 con 0x5559ec6eb390 2024-04-25T11:38:36.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.242+0000 7fe55cbfe640 1 -- 172.21.15.90:0/2990131576 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe560021c60 con 0x5559ec6eb390 2024-04-25T11:38:36.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.242+0000 7fe55cbfe640 1 -- 172.21.15.90:0/2990131576 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56001e360 con 0x5559ec6eb390 2024-04-25T11:38:36.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.243+0000 7fe5771e1740 1 -- 172.21.15.90:0/2990131576 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5559ec6eb390 msgr2=0x5559eca8c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.243+0000 7fe5771e1740 1 --2- 172.21.15.90:0/2990131576 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5559ec6eb390 0x5559eca8c810 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe56000a5d0 tx=0x7fe56000aa90 comp rx=0 tx=0).stop 2024-04-25T11:38:36.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.243+0000 7fe5771e1740 1 -- 172.21.15.90:0/2990131576 shutdown_connections 2024-04-25T11:38:36.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.243+0000 7fe5771e1740 1 --2- 172.21.15.90:0/2990131576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559ec6ca6b0 0x5559eca996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.243+0000 7fe5771e1740 1 --2- 172.21.15.90:0/2990131576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559ec6c24d0 0x5559eca8cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.243+0000 7fe5771e1740 1 --2- 172.21.15.90:0/2990131576 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5559ec6eb390 0x5559eca8c810 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.243+0000 7fe5771e1740 1 -- 172.21.15.90:0/2990131576 >> 172.21.15.90:0/2990131576 conn(0x5559ec6df4e0 msgr2=0x5559eca88fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:36.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.243+0000 7fe5771e1740 1 -- 172.21.15.90:0/2990131576 shutdown_connections 2024-04-25T11:38:36.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.243+0000 7fe5771e1740 1 -- 172.21.15.90:0/2990131576 wait complete. 2024-04-25T11:38:36.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.243+0000 7fe5771e1740 1 -- start start 2024-04-25T11:38:36.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.243+0000 7fe5771e1740 1 Processor -- start 2024-04-25T11:38:36.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.244+0000 7fe5771e1740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5559ec6c24d0 0x5559eca97260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.244+0000 7fe5771e1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559ec6ca6b0 0x5559eca977a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.244+0000 7fe5771e1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559ec6eb390 0x5559ecaa3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.244+0000 7fe5771e1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5559eca99d90 con 0x5559ec6eb390 2024-04-25T11:38:36.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.244+0000 7fe5771e1740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5559ec978dc0 con 0x5559ec6c24d0 2024-04-25T11:38:36.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.244+0000 7fe5771e1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5559eca99f10 con 0x5559ec6ca6b0 2024-04-25T11:38:36.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.244+0000 7fe5771e1740 10 client.0 _ll_register_callbacks cb 0x5559ec994600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:36.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.244+0000 7fe56e0d0640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5559ec6c24d0 0x5559eca97260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.244+0000 7fe56e8d1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559ec6eb390 0x5559ecaa3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.244+0000 7fe56e0d0640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5559ec6c24d0 0x5559eca97260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:39006/0 (socket says 172.21.15.90:39006) 2024-04-25T11:38:36.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.244+0000 7fe56e0d0640 1 -- 172.21.15.90:0/1239879624 learned_addr learned my addr 172.21.15.90:0/1239879624 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:36.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.244+0000 7fe56d8cf640 1 --2- 172.21.15.90:0/1239879624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559ec6ca6b0 0x5559eca977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.244+0000 7fe56e8d1640 1 -- 172.21.15.90:0/1239879624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559ec6ca6b0 msgr2=0x5559eca977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:36.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.244+0000 7fe56e8d1640 1 --2- 172.21.15.90:0/1239879624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559ec6ca6b0 0x5559eca977a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.244+0000 7fe56e8d1640 1 -- 172.21.15.90:0/1239879624 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5559ec6c24d0 msgr2=0x5559eca97260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:36.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.244+0000 7fe56e8d1640 1 --2- 172.21.15.90:0/1239879624 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5559ec6c24d0 0x5559eca97260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.244+0000 7fe56e8d1640 1 -- 172.21.15.90:0/1239879624 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559ec7033f0 con 0x5559ec6eb390 2024-04-25T11:38:36.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.244+0000 7fe56e0d0640 1 --2- 172.21.15.90:0/1239879624 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5559ec6c24d0 0x5559eca97260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:36.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.244+0000 7fe56e8d1640 1 --2- 172.21.15.90:0/1239879624 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559ec6eb390 0x5559ecaa3210 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x5559eca89980 tx=0x7fe560020a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1050ab89ff53bc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:36.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.244+0000 7fe551ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:38:36.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.245+0000 7fe551ffb640 1 -- 172.21.15.90:0/1239879624 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56001a040 con 0x5559ec6eb390 2024-04-25T11:38:36.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.245+0000 7fe551ffb640 1 -- 172.21.15.90:0/1239879624 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fe560020c00 con 0x5559ec6eb390 2024-04-25T11:38:36.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.245+0000 7fe5771e1740 1 -- 172.21.15.90:0/1239879624 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe560021c60 con 0x5559ec6eb390 2024-04-25T11:38:36.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.245+0000 7fe551ffb640 1 -- 172.21.15.90:0/1239879624 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe560038050 con 0x5559ec6eb390 2024-04-25T11:38:36.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.245+0000 7fe5771e1740 10 client.16659 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:36.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.245+0000 7fe5771e1740 1 -- 172.21.15.90:0/1239879624 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5559ec7cd7d0 con 0x5559ec6eb390 2024-04-25T11:38:36.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.245+0000 7fe551ffb640 1 -- 172.21.15.90:0/1239879624 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe560021c60 con 0x5559ec6eb390 2024-04-25T11:38:36.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.245+0000 7fe551ffb640 1 -- 172.21.15.90:0/1239879624 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7fe560028040 con 0x5559ec6eb390 2024-04-25T11:38:36.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.245+0000 7fe5771e1740 10 client.16659 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:36.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.245+0000 7fe5771e1740 10 client.16659 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:38:36.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.245+0000 7fe5771e1740 20 client.16659 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:36.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.245+0000 7fe5771e1740 1 --2- 172.21.15.90:0/1239879624 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5559ec6a9f60 0x5559ecaa9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.245+0000 7fe5771e1740 4 client.16659 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:38:36.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.245+0000 7fe5771e1740 1 -- 172.21.15.90:0/1239879624 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559ec9ff990 con 0x5559ec6a9f60 2024-04-25T11:38:36.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.246+0000 7fe56e0d0640 1 --2- 172.21.15.90:0/1239879624 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5559ec6a9f60 0x5559ecaa9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.246+0000 7fe551ffb640 10 client.16659 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:38:36.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.246+0000 7fe56e0d0640 1 --2- 172.21.15.90:0/1239879624 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5559ec6a9f60 0x5559ecaa9660 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a60d6c28de3cb4b9 server_cookie=e9d843e14343840a in_seq=0 out_seq=0 2024-04-25T11:38:36.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.247+0000 7fe551ffb640 1 -- 172.21.15.90:0/1239879624 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5559ec9ff990 con 0x5559ec6a9f60 2024-04-25T11:38:36.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.247+0000 7fe551ffb640 10 client.16659 handle_command_reply: tid=0 2024-04-25T11:38:36.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.247+0000 7fe5771e1740 1 client.16659 shutdown 2024-04-25T11:38:36.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.247+0000 7fe5771e1740 1 -- 172.21.15.90:0/1239879624 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559ec6eb390 msgr2=0x5559ecaa3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.247+0000 7fe5771e1740 1 --2- 172.21.15.90:0/1239879624 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559ec6eb390 0x5559ecaa3210 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x5559eca89980 tx=0x7fe560020a80 comp rx=0 tx=0).stop 2024-04-25T11:38:36.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.247+0000 7fe5771e1740 1 -- 172.21.15.90:0/1239879624 shutdown_connections 2024-04-25T11:38:36.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.247+0000 7fe5771e1740 1 --2- 172.21.15.90:0/1239879624 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5559ec6a9f60 0x5559ecaa9660 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.247+0000 7fe5771e1740 1 --2- 172.21.15.90:0/1239879624 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559ec6eb390 0x5559ecaa3210 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.247+0000 7fe5771e1740 1 --2- 172.21.15.90:0/1239879624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559ec6ca6b0 0x5559eca977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.247+0000 7fe551ffb640 0 client.16659 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:38:36.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.247+0000 7fe5771e1740 1 --2- 172.21.15.90:0/1239879624 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5559ec6c24d0 0x5559eca97260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.247+0000 7fe5771e1740 1 -- 172.21.15.90:0/1239879624 >> 172.21.15.90:0/1239879624 conn(0x5559ec6df4e0 msgr2=0x5559eca8e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:36.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.247+0000 7fe5771e1740 1 -- 172.21.15.90:0/1239879624 shutdown_connections 2024-04-25T11:38:36.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.247+0000 7fe5771e1740 1 -- 172.21.15.90:0/1239879624 wait complete. 2024-04-25T11:38:36.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.247+0000 7fe5771e1740 20 client.16659 trim_cache size 0 max 16384 2024-04-25T11:38:36.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.262+0000 7fe5771e1740 1 Processor -- start 2024-04-25T11:38:36.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.262+0000 7fe5771e1740 1 -- start start 2024-04-25T11:38:36.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.262+0000 7fe5771e1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559ec6c24d0 0x5559ecaa2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.262+0000 7fe5771e1740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5559ec6ca6b0 0x5559eca95920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.262+0000 7fe5771e1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559ec6eb390 0x5559eca95e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.262+0000 7fe5771e1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe5681afca0 con 0x5559ec6eb390 2024-04-25T11:38:36.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.262+0000 7fe5771e1740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe5681b1810 con 0x5559ec6ca6b0 2024-04-25T11:38:36.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.262+0000 7fe5771e1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe5681b0370 con 0x5559ec6c24d0 2024-04-25T11:38:36.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.262+0000 7fe56e8d1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559ec6eb390 0x5559eca95e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.262+0000 7fe56d8cf640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5559ec6ca6b0 0x5559eca95920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.262+0000 7fe56e8d1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559ec6eb390 0x5559eca95e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34930/0 (socket says 172.21.15.90:34930) 2024-04-25T11:38:36.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.263+0000 7fe56e8d1640 1 -- 172.21.15.90:0/3723584283 learned_addr learned my addr 172.21.15.90:0/3723584283 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:36.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.263+0000 7fe56e0d0640 1 --2- 172.21.15.90:0/3723584283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559ec6c24d0 0x5559ecaa2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.263+0000 7fe56e8d1640 1 -- 172.21.15.90:0/3723584283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559ec6c24d0 msgr2=0x5559ecaa2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:36.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.263+0000 7fe56e8d1640 1 --2- 172.21.15.90:0/3723584283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559ec6c24d0 0x5559ecaa2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.263+0000 7fe56e8d1640 1 -- 172.21.15.90:0/3723584283 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5559ec6ca6b0 msgr2=0x5559eca95920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:36.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.263+0000 7fe56e8d1640 1 --2- 172.21.15.90:0/3723584283 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5559ec6ca6b0 0x5559eca95920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.263+0000 7fe56e8d1640 1 -- 172.21.15.90:0/3723584283 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559eca832b0 con 0x5559ec6eb390 2024-04-25T11:38:36.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.263+0000 7fe56d8cf640 1 --2- 172.21.15.90:0/3723584283 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5559ec6ca6b0 0x5559eca95920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:36.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.263+0000 7fe56e8d1640 1 --2- 172.21.15.90:0/3723584283 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559ec6eb390 0x5559eca95e60 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fe560009f10 tx=0x7fe560007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90cf579df5c78dc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:36.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.263+0000 7fe553fff640 1 -- 172.21.15.90:0/3723584283 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56001a040 con 0x5559ec6eb390 2024-04-25T11:38:36.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.263+0000 7fe553fff640 1 -- 172.21.15.90:0/3723584283 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe560041070 con 0x5559ec6eb390 2024-04-25T11:38:36.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.263+0000 7fe553fff640 1 -- 172.21.15.90:0/3723584283 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe560029400 con 0x5559ec6eb390 2024-04-25T11:38:36.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.264+0000 7fe5771e1740 1 -- 172.21.15.90:0/3723584283 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559ec6eb390 msgr2=0x5559eca95e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.264+0000 7fe5771e1740 1 --2- 172.21.15.90:0/3723584283 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559ec6eb390 0x5559eca95e60 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fe560009f10 tx=0x7fe560007000 comp rx=0 tx=0).stop 2024-04-25T11:38:36.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.264+0000 7fe5771e1740 1 -- 172.21.15.90:0/3723584283 shutdown_connections 2024-04-25T11:38:36.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.264+0000 7fe5771e1740 1 --2- 172.21.15.90:0/3723584283 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559ec6eb390 0x5559eca95e60 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.264+0000 7fe5771e1740 1 --2- 172.21.15.90:0/3723584283 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5559ec6ca6b0 0x5559eca95920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.264+0000 7fe5771e1740 1 --2- 172.21.15.90:0/3723584283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559ec6c24d0 0x5559ecaa2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.264+0000 7fe5771e1740 1 -- 172.21.15.90:0/3723584283 >> 172.21.15.90:0/3723584283 conn(0x5559ec6df4e0 msgr2=0x5559eca92280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:36.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.264+0000 7fe5771e1740 1 -- 172.21.15.90:0/3723584283 shutdown_connections 2024-04-25T11:38:36.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.264+0000 7fe5771e1740 1 -- 172.21.15.90:0/3723584283 wait complete. 2024-04-25T11:38:36.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.265+0000 7fe5771e1740 1 -- start start 2024-04-25T11:38:36.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.265+0000 7fe5771e1740 1 Processor -- start 2024-04-25T11:38:36.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.265+0000 7fe5771e1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559ec6c24d0 0x5559eca9a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.265+0000 7fe5771e1740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5559ec6ca6b0 0x5559eca9a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.265+0000 7fe5771e1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559ec6eb390 0x5559eca9ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.265+0000 7fe5771e1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe5681b1110 con 0x5559ec6c24d0 2024-04-25T11:38:36.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.265+0000 7fe5771e1740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe5681b0a40 con 0x5559ec6ca6b0 2024-04-25T11:38:36.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.265+0000 7fe5771e1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe5681adc80 con 0x5559ec6eb390 2024-04-25T11:38:36.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.265+0000 7fe5771e1740 10 client.0 _ll_register_callbacks cb 0x7fe5580e2e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:36.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.265+0000 7fe56d8cf640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5559ec6ca6b0 0x5559eca9a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.265+0000 7fe56e0d0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559ec6c24d0 0x5559eca9a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.265+0000 7fe56e0d0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559ec6c24d0 0x5559eca9a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34944/0 (socket says 172.21.15.90:34944) 2024-04-25T11:38:36.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.265+0000 7fe56e0d0640 1 -- 172.21.15.90:0/3142079975 learned_addr learned my addr 172.21.15.90:0/3142079975 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:36.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.266+0000 7fe56e8d1640 1 --2- 172.21.15.90:0/3142079975 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559ec6eb390 0x5559eca9ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.266+0000 7fe56d8cf640 1 -- 172.21.15.90:0/3142079975 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559ec6eb390 msgr2=0x5559eca9ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:36.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.266+0000 7fe56d8cf640 1 --2- 172.21.15.90:0/3142079975 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559ec6eb390 0x5559eca9ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.266+0000 7fe56d8cf640 1 -- 172.21.15.90:0/3142079975 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559ec6c24d0 msgr2=0x5559eca9a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:36.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.266+0000 7fe56d8cf640 1 --2- 172.21.15.90:0/3142079975 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559ec6c24d0 0x5559eca9a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.266+0000 7fe56d8cf640 1 -- 172.21.15.90:0/3142079975 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559ecb13cc0 con 0x5559ec6ca6b0 2024-04-25T11:38:36.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.266+0000 7fe56e8d1640 1 --2- 172.21.15.90:0/3142079975 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559ec6eb390 0x5559eca9ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:38:36.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.266+0000 7fe56d8cf640 1 --2- 172.21.15.90:0/3142079975 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5559ec6ca6b0 0x5559eca9a6f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe5580dcad0 tx=0x7fe558014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33c40ec64a3c9147 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:36.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.266+0000 7fe551ffb640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:38:36.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.266+0000 7fe551ffb640 1 -- 172.21.15.90:0/3142079975 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5580a6c90 con 0x5559ec6ca6b0 2024-04-25T11:38:36.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.266+0000 7fe551ffb640 1 -- 172.21.15.90:0/3142079975 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fe558003aa0 con 0x5559ec6ca6b0 2024-04-25T11:38:36.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.266+0000 7fe5771e1740 1 -- 172.21.15.90:0/3142079975 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe560041070 con 0x5559ec6ca6b0 2024-04-25T11:38:36.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.266+0000 7fe551ffb640 1 -- 172.21.15.90:0/3142079975 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe558003ca0 con 0x5559ec6ca6b0 2024-04-25T11:38:36.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.266+0000 7fe5771e1740 10 client.25907 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:36.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.267+0000 7fe5771e1740 1 -- 172.21.15.90:0/3142079975 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5559ecb14e90 con 0x5559ec6ca6b0 2024-04-25T11:38:36.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.267+0000 7fe551ffb640 1 -- 172.21.15.90:0/3142079975 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe560041070 con 0x5559ec6ca6b0 2024-04-25T11:38:36.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.267+0000 7fe551ffb640 1 -- 172.21.15.90:0/3142079975 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7fe5580afef0 con 0x5559ec6ca6b0 2024-04-25T11:38:36.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.267+0000 7fe5771e1740 10 client.25907 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:36.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.267+0000 7fe5771e1740 10 client.25907 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:38:36.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.267+0000 7fe5771e1740 20 client.25907 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:36.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.267+0000 7fe5771e1740 1 --2- 172.21.15.90:0/3142079975 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5559ec67f3a0 0x5559ecb17600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.267+0000 7fe5771e1740 4 client.25907 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:38:36.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.267+0000 7fe5771e1740 1 -- 172.21.15.90:0/3142079975 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5559eca97b70 con 0x5559ec67f3a0 2024-04-25T11:38:36.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.267+0000 7fe56e0d0640 1 --2- 172.21.15.90:0/3142079975 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5559ec67f3a0 0x5559ecb17600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.268+0000 7fe551ffb640 10 client.25907 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:38:36.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.268+0000 7fe56e0d0640 1 --2- 172.21.15.90:0/3142079975 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5559ec67f3a0 0x5559ecb17600 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=34013be029ffa764 server_cookie=f067ae418c27ac2a in_seq=0 out_seq=0 2024-04-25T11:38:36.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.268+0000 7fe551ffb640 1 -- 172.21.15.90:0/3142079975 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5559eca97b70 con 0x5559ec67f3a0 2024-04-25T11:38:36.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.268+0000 7fe551ffb640 10 client.25907 handle_command_reply: tid=0 2024-04-25T11:38:36.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.268+0000 7fe5771e1740 1 client.25907 shutdown 2024-04-25T11:38:36.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.268+0000 7fe5771e1740 1 -- 172.21.15.90:0/3142079975 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5559ec6ca6b0 msgr2=0x5559eca9a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.268+0000 7fe5771e1740 1 --2- 172.21.15.90:0/3142079975 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5559ec6ca6b0 0x5559eca9a6f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe5580dcad0 tx=0x7fe558014000 comp rx=0 tx=0).stop 2024-04-25T11:38:36.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.268+0000 7fe5771e1740 1 -- 172.21.15.90:0/3142079975 shutdown_connections 2024-04-25T11:38:36.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.268+0000 7fe5771e1740 1 --2- 172.21.15.90:0/3142079975 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5559ec67f3a0 0x5559ecb17600 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.268+0000 7fe5771e1740 1 --2- 172.21.15.90:0/3142079975 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559ec6eb390 0x5559eca9ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.268+0000 7fe5771e1740 1 --2- 172.21.15.90:0/3142079975 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5559ec6ca6b0 0x5559eca9a6f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.268+0000 7fe5771e1740 1 --2- 172.21.15.90:0/3142079975 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559ec6c24d0 0x5559eca9a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.268+0000 7fe551ffb640 0 client.25907 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:38:36.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.268+0000 7fe5771e1740 1 -- 172.21.15.90:0/3142079975 >> 172.21.15.90:0/3142079975 conn(0x5559ec6df4e0 msgr2=0x5559eca954e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:36.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.268+0000 7fe5771e1740 1 -- 172.21.15.90:0/3142079975 shutdown_connections 2024-04-25T11:38:36.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.268+0000 7fe5771e1740 1 -- 172.21.15.90:0/3142079975 wait complete. 2024-04-25T11:38:36.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.268+0000 7fe5771e1740 20 client.25907 trim_cache size 0 max 16384 2024-04-25T11:38:36.261 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:38:36.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.269+0000 7fe570334640 1 -- 172.21.15.90:0/2062475949 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe5540afc60 msgr2=0x7fe5540b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.269+0000 7fe570334640 1 --2- 172.21.15.90:0/2062475949 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe5540afc60 0x7fe5540b2170 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe560020000 tx=0x7fe560021000 comp rx=0 tx=0).stop 2024-04-25T11:38:36.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.269+0000 7fe570334640 1 -- 172.21.15.90:0/2062475949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe568109b10 msgr2=0x7fe56819ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.269+0000 7fe570334640 1 --2- 172.21.15.90:0/2062475949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe568109b10 0x7fe56819ad00 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fe55800e9f0 tx=0x7fe55800eeb0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.269+0000 7fe570334640 1 -- 172.21.15.90:0/2062475949 shutdown_connections 2024-04-25T11:38:36.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.269+0000 7fe570334640 1 --2- 172.21.15.90:0/2062475949 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe5540afc60 0x7fe5540b2170 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.269+0000 7fe570334640 1 --2- 172.21.15.90:0/2062475949 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe568109fb0 0x7fe56819efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.269+0000 7fe570334640 1 --2- 172.21.15.90:0/2062475949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe568109b10 0x7fe56819ad00 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.269+0000 7fe570334640 1 --2- 172.21.15.90:0/2062475949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe5681042b0 0x7fe56819a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.269+0000 7fe570334640 1 -- 172.21.15.90:0/2062475949 >> 172.21.15.90:0/2062475949 conn(0x7fe5680ff360 msgr2=0x7fe568101060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:36.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.269+0000 7fe570334640 1 -- 172.21.15.90:0/2062475949 shutdown_connections 2024-04-25T11:38:36.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.269+0000 7fe570334640 1 -- 172.21.15.90:0/2062475949 wait complete. 2024-04-25T11:38:36.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T11:38:36.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.534+0000 7fe504e05640 1 -- 172.21.15.90:0/3782621110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe500115fe0 msgr2=0x7fe500118400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.534+0000 7fe504e05640 1 --2- 172.21.15.90:0/3782621110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe500115fe0 0x7fe500118400 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4009900 tx=0x7fe4f402a440 comp rx=0 tx=0).stop 2024-04-25T11:38:36.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.534+0000 7fe504e05640 1 -- 172.21.15.90:0/3782621110 shutdown_connections 2024-04-25T11:38:36.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.535+0000 7fe504e05640 1 --2- 172.21.15.90:0/3782621110 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe500118a90 0x7fe50011af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.535+0000 7fe504e05640 1 --2- 172.21.15.90:0/3782621110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe500115fe0 0x7fe500118400 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.535+0000 7fe504e05640 1 --2- 172.21.15.90:0/3782621110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe50010a290 0x7fe50010a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.535+0000 7fe504e05640 1 -- 172.21.15.90:0/3782621110 >> 172.21.15.90:0/3782621110 conn(0x7fe50006e7a0 msgr2=0x7fe50006ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:36.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.535+0000 7fe504e05640 1 -- 172.21.15.90:0/3782621110 shutdown_connections 2024-04-25T11:38:36.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.535+0000 7fe504e05640 1 -- 172.21.15.90:0/3782621110 wait complete. 2024-04-25T11:38:36.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.536+0000 7fe504e05640 1 Processor -- start 2024-04-25T11:38:36.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.536+0000 7fe504e05640 1 -- start start 2024-04-25T11:38:36.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.537+0000 7fe504e05640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe50010a290 0x7fe500115040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.537+0000 7fe504e05640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe500115fe0 0x7fe500115580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.537+0000 7fe504e05640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe500118a90 0x7fe5001adf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.537+0000 7fe504e05640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe50010f190 con 0x7fe500115fe0 2024-04-25T11:38:36.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.537+0000 7fe504e05640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe50010f010 con 0x7fe50010a290 2024-04-25T11:38:36.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.537+0000 7fe504e05640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe50010f310 con 0x7fe500118a90 2024-04-25T11:38:36.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.537+0000 7fe4fdd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe500115fe0 0x7fe500115580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.537+0000 7fe4fdd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe500115fe0 0x7fe500115580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34976/0 (socket says 172.21.15.90:34976) 2024-04-25T11:38:36.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.537+0000 7fe4fdd9b640 1 -- 172.21.15.90:0/1017046258 learned_addr learned my addr 172.21.15.90:0/1017046258 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:36.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.537+0000 7fe4fe59c640 1 --2- 172.21.15.90:0/1017046258 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe50010a290 0x7fe500115040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.538+0000 7fe4fdd9b640 1 -- 172.21.15.90:0/1017046258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe500118a90 msgr2=0x7fe5001adf70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:38:36.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.538+0000 7fe4fdd9b640 1 --2- 172.21.15.90:0/1017046258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe500118a90 0x7fe5001adf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.538+0000 7fe4fdd9b640 1 -- 172.21.15.90:0/1017046258 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe50010a290 msgr2=0x7fe500115040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.538+0000 7fe4fdd9b640 1 --2- 172.21.15.90:0/1017046258 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe50010a290 0x7fe500115040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.538+0000 7fe4fdd9b640 1 -- 172.21.15.90:0/1017046258 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5001ae5b0 con 0x7fe500115fe0 2024-04-25T11:38:36.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.538+0000 7fe4fe59c640 1 --2- 172.21.15.90:0/1017046258 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe50010a290 0x7fe500115040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:38:36.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.538+0000 7fe4fdd9b640 1 --2- 172.21.15.90:0/1017046258 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe500115fe0 0x7fe500115580 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4005ab0 tx=0x7fe4f4003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:36.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.538+0000 7fe4ef7fe640 1 -- 172.21.15.90:0/1017046258 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f403f070 con 0x7fe500115fe0 2024-04-25T11:38:36.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.539+0000 7fe4ef7fe640 1 -- 172.21.15.90:0/1017046258 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4f4031d60 con 0x7fe500115fe0 2024-04-25T11:38:36.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.539+0000 7fe4ef7fe640 1 -- 172.21.15.90:0/1017046258 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f4035030 con 0x7fe500115fe0 2024-04-25T11:38:36.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.539+0000 7fe504e05640 1 -- 172.21.15.90:0/1017046258 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5001ae840 con 0x7fe500115fe0 2024-04-25T11:38:36.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.539+0000 7fe504e05640 1 -- 172.21.15.90:0/1017046258 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5001aece0 con 0x7fe500115fe0 2024-04-25T11:38:36.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.542+0000 7fe50bd94740 1 Processor -- start 2024-04-25T11:38:36.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.543+0000 7fe50bd94740 1 -- start start 2024-04-25T11:38:36.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.543+0000 7fe50bd94740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ffa93390 0x5560ffe347e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.543+0000 7fe50bd94740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560ffa6a4d0 0x5560ffe34d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.543+0000 7fe50bd94740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560ffa726b0 0x5560ffe41670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.543+0000 7fe50bd94740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5560ffd99d80 con 0x5560ffa726b0 2024-04-25T11:38:36.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.543+0000 7fe50bd94740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5560ffd53a20 con 0x5560ffa6a4d0 2024-04-25T11:38:36.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.543+0000 7fe50bd94740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5560ffd95d40 con 0x5560ffa93390 2024-04-25T11:38:36.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.544+0000 7fe4fed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ffa93390 0x5560ffe347e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.544+0000 7fe4fed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ffa93390 0x5560ffe347e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:33720/0 (socket says 172.21.15.90:33720) 2024-04-25T11:38:36.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.544+0000 7fe4fed9d640 1 -- 172.21.15.90:0/405647393 learned_addr learned my addr 172.21.15.90:0/405647393 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:36.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.544+0000 7fe4fe59c640 1 --2- 172.21.15.90:0/405647393 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560ffa6a4d0 0x5560ffe34d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.544+0000 7fe4ef7fe640 1 -- 172.21.15.90:0/1017046258 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fe4f4042070 con 0x7fe500115fe0 2024-04-25T11:38:36.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.545+0000 7fe4fdd9b640 1 --2- 172.21.15.90:0/405647393 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560ffa726b0 0x5560ffe41670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.545+0000 7fe4fed9d640 1 -- 172.21.15.90:0/405647393 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560ffa6a4d0 msgr2=0x5560ffe34d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:36.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.545+0000 7fe4fed9d640 1 --2- 172.21.15.90:0/405647393 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560ffa6a4d0 0x5560ffe34d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.545+0000 7fe4fed9d640 1 -- 172.21.15.90:0/405647393 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560ffa726b0 msgr2=0x5560ffe41670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:36.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.545+0000 7fe4fed9d640 1 --2- 172.21.15.90:0/405647393 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560ffa726b0 0x5560ffe41670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.545+0000 7fe4fed9d640 1 -- 172.21.15.90:0/405647393 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560ffd5ec30 con 0x5560ffa93390 2024-04-25T11:38:36.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.546+0000 7fe4fed9d640 1 --2- 172.21.15.90:0/405647393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ffa93390 0x5560ffe347e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe4f00096f0 tx=0x7fe4f0009bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6daba0a986d4e63c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:36.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.546+0000 7fe4ef7fe640 1 --2- 172.21.15.90:0/1017046258 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe4e80afd60 0x7fe4e80b2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.546+0000 7fe4ecff9640 1 -- 172.21.15.90:0/405647393 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f0015020 con 0x5560ffa93390 2024-04-25T11:38:36.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.546+0000 7fe4ecff9640 1 -- 172.21.15.90:0/405647393 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4f000bd80 con 0x5560ffa93390 2024-04-25T11:38:36.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.546+0000 7fe4ef7fe640 1 -- 172.21.15.90:0/1017046258 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fe4f40ff9f0 con 0x7fe500115fe0 2024-04-25T11:38:36.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.546+0000 7fe4ecff9640 1 -- 172.21.15.90:0/405647393 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f0010380 con 0x5560ffa93390 2024-04-25T11:38:36.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.546+0000 7fe4fe59c640 1 --2- 172.21.15.90:0/1017046258 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe4e80afd60 0x7fe4e80b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.547+0000 7fe50bd94740 1 -- 172.21.15.90:0/405647393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ffa93390 msgr2=0x5560ffe347e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.547+0000 7fe4fe59c640 1 --2- 172.21.15.90:0/1017046258 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe4e80afd60 0x7fe4e80b2270 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x5560ffe35fb0 tx=0x7fe4f800e040 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:36.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.547+0000 7fe50bd94740 1 --2- 172.21.15.90:0/405647393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ffa93390 0x5560ffe347e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe4f00096f0 tx=0x7fe4f0009bb0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.548+0000 7fe50bd94740 1 -- 172.21.15.90:0/405647393 shutdown_connections 2024-04-25T11:38:36.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.548+0000 7fe50bd94740 1 --2- 172.21.15.90:0/405647393 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560ffa726b0 0x5560ffe41670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.548+0000 7fe50bd94740 1 --2- 172.21.15.90:0/405647393 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560ffa6a4d0 0x5560ffe34d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.548+0000 7fe50bd94740 1 --2- 172.21.15.90:0/405647393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ffa93390 0x5560ffe347e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.548+0000 7fe50bd94740 1 -- 172.21.15.90:0/405647393 >> 172.21.15.90:0/405647393 conn(0x5560ffa874e0 msgr2=0x5560ffe30f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:36.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.548+0000 7fe50bd94740 1 -- 172.21.15.90:0/405647393 shutdown_connections 2024-04-25T11:38:36.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.548+0000 7fe50bd94740 1 -- 172.21.15.90:0/405647393 wait complete. 2024-04-25T11:38:36.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.548+0000 7fe50bd94740 1 -- start start 2024-04-25T11:38:36.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.548+0000 7fe50bd94740 1 Processor -- start 2024-04-25T11:38:36.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.549+0000 7fe50bd94740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560ffa6a4d0 0x5560ffe3f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.549+0000 7fe50bd94740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ffa726b0 0x5560ffe3f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.549+0000 7fe50bd94740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560ffa93390 0x5560ffe4b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.549+0000 7fe50bd94740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5560ffe41d60 con 0x5560ffa6a4d0 2024-04-25T11:38:36.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.549+0000 7fe50bd94740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5560ffd20dc0 con 0x5560ffa93390 2024-04-25T11:38:36.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.549+0000 7fe50bd94740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5560ffe41ee0 con 0x5560ffa726b0 2024-04-25T11:38:36.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.549+0000 7fe50bd94740 10 client.0 _ll_register_callbacks cb 0x5560ffd3c600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:36.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.549+0000 7fe4fe59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560ffa6a4d0 0x5560ffe3f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.549+0000 7fe4fe59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560ffa6a4d0 0x5560ffe3f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34988/0 (socket says 172.21.15.90:34988) 2024-04-25T11:38:36.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.549+0000 7fe4fe59c640 1 -- 172.21.15.90:0/708100415 learned_addr learned my addr 172.21.15.90:0/708100415 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:36.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.550+0000 7fe4fed9d640 1 --2- 172.21.15.90:0/708100415 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560ffa93390 0x5560ffe4b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.550+0000 7fe4fe59c640 1 -- 172.21.15.90:0/708100415 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ffa726b0 msgr2=0x5560ffe3f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:36.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.550+0000 7fe4fe59c640 1 --2- 172.21.15.90:0/708100415 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ffa726b0 0x5560ffe3f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.550+0000 7fe4fe59c640 1 -- 172.21.15.90:0/708100415 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560ffa93390 msgr2=0x5560ffe4b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:36.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.550+0000 7fe4fe59c640 1 --2- 172.21.15.90:0/708100415 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560ffa93390 0x5560ffe4b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.550+0000 7fe4fe59c640 1 -- 172.21.15.90:0/708100415 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560ffaab3f0 con 0x5560ffa6a4d0 2024-04-25T11:38:36.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.550+0000 7fe4fe59c640 1 --2- 172.21.15.90:0/708100415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560ffa6a4d0 0x5560ffe3f230 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8007f40 tx=0x7fe4f8004a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=701285a2dfbec552 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:36.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.551+0000 7fe4e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:38:36.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.551+0000 7fe4e5ffb640 1 -- 172.21.15.90:0/708100415 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f80225f0 con 0x5560ffa6a4d0 2024-04-25T11:38:36.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.551+0000 7fe50bd94740 1 -- 172.21.15.90:0/708100415 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4f000bd80 con 0x5560ffa6a4d0 2024-04-25T11:38:36.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.551+0000 7fe4e5ffb640 1 -- 172.21.15.90:0/708100415 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fe4f8022790 con 0x5560ffa6a4d0 2024-04-25T11:38:36.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.551+0000 7fe4e5ffb640 1 -- 172.21.15.90:0/708100415 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe4f8022990 con 0x5560ffa6a4d0 2024-04-25T11:38:36.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.551+0000 7fe50bd94740 10 client.16674 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:36.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.551+0000 7fe50bd94740 1 -- 172.21.15.90:0/708100415 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5560ffb757d0 con 0x5560ffa6a4d0 2024-04-25T11:38:36.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.552+0000 7fe4e5ffb640 1 -- 172.21.15.90:0/708100415 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4f000bd80 con 0x5560ffa6a4d0 2024-04-25T11:38:36.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.552+0000 7fe4e5ffb640 1 -- 172.21.15.90:0/708100415 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7fe4f802bc00 con 0x5560ffa6a4d0 2024-04-25T11:38:36.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.552+0000 7fe50bd94740 10 client.16674 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:36.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.552+0000 7fe50bd94740 10 client.16674 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:38:36.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.552+0000 7fe50bd94740 20 client.16674 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:36.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.552+0000 7fe50bd94740 1 --2- 172.21.15.90:0/708100415 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5560ffa51f60 0x5560ffe51750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.552+0000 7fe50bd94740 4 client.16674 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:38:36.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.552+0000 7fe50bd94740 1 -- 172.21.15.90:0/708100415 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560ffda7990 con 0x5560ffa51f60 2024-04-25T11:38:36.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.553+0000 7fe4fdd9b640 1 --2- 172.21.15.90:0/708100415 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5560ffa51f60 0x5560ffe51750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.555+0000 7fe4e5ffb640 10 client.16674 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:36.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.555+0000 7fe4fdd9b640 1 --2- 172.21.15.90:0/708100415 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5560ffa51f60 0x5560ffe51750 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=170c6cc2053f0f5f server_cookie=714879c5d54541ae in_seq=0 out_seq=0 2024-04-25T11:38:36.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.557+0000 7fe4e5ffb640 1 -- 172.21.15.90:0/708100415 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5560ffda7990 con 0x5560ffa51f60 2024-04-25T11:38:36.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.557+0000 7fe4e5ffb640 10 client.16674 handle_command_reply: tid=0 2024-04-25T11:38:36.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.557+0000 7fe50bd94740 1 client.16674 shutdown 2024-04-25T11:38:36.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.558+0000 7fe50bd94740 1 -- 172.21.15.90:0/708100415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560ffa6a4d0 msgr2=0x5560ffe3f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.558+0000 7fe50bd94740 1 --2- 172.21.15.90:0/708100415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560ffa6a4d0 0x5560ffe3f230 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8007f40 tx=0x7fe4f8004a60 comp rx=0 tx=0).stop 2024-04-25T11:38:36.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.558+0000 7fe50bd94740 1 -- 172.21.15.90:0/708100415 shutdown_connections 2024-04-25T11:38:36.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.558+0000 7fe50bd94740 1 --2- 172.21.15.90:0/708100415 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5560ffa51f60 0x5560ffe51750 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.558+0000 7fe50bd94740 1 --2- 172.21.15.90:0/708100415 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560ffa93390 0x5560ffe4b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.558+0000 7fe50bd94740 1 --2- 172.21.15.90:0/708100415 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ffa726b0 0x5560ffe3f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.558+0000 7fe50bd94740 1 --2- 172.21.15.90:0/708100415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560ffa6a4d0 0x5560ffe3f230 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.558+0000 7fe4e5ffb640 0 client.16674 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:36.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.558+0000 7fe50bd94740 1 -- 172.21.15.90:0/708100415 >> 172.21.15.90:0/708100415 conn(0x5560ffa874e0 msgr2=0x5560ffe36500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:36.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.558+0000 7fe50bd94740 1 -- 172.21.15.90:0/708100415 shutdown_connections 2024-04-25T11:38:36.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.558+0000 7fe50bd94740 1 -- 172.21.15.90:0/708100415 wait complete. 2024-04-25T11:38:36.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.558+0000 7fe50bd94740 20 client.16674 trim_cache size 0 max 16384 2024-04-25T11:38:36.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.580+0000 7fe50bd94740 1 Processor -- start 2024-04-25T11:38:36.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.580+0000 7fe50bd94740 1 -- start start 2024-04-25T11:38:36.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.580+0000 7fe50bd94740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560ffa6a4d0 0x5560ffe4ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.581+0000 7fe50bd94740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ffa726b0 0x5560ffe40e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.581+0000 7fe50bd94740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560ffa93390 0x5560ffe394e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.581+0000 7fe50bd94740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe5001beaf0 con 0x5560ffa93390 2024-04-25T11:38:36.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.581+0000 7fe50bd94740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe5001c0660 con 0x5560ffa6a4d0 2024-04-25T11:38:36.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.581+0000 7fe50bd94740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe5001bf1c0 con 0x5560ffa726b0 2024-04-25T11:38:36.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.581+0000 7fe4fed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560ffa93390 0x5560ffe394e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.581+0000 7fe4fed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560ffa93390 0x5560ffe394e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34990/0 (socket says 172.21.15.90:34990) 2024-04-25T11:38:36.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.581+0000 7fe4fed9d640 1 -- 172.21.15.90:0/4055376645 learned_addr learned my addr 172.21.15.90:0/4055376645 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:36.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.581+0000 7fe4fdd9b640 1 --2- 172.21.15.90:0/4055376645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ffa726b0 0x5560ffe40e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.582+0000 7fe4fed9d640 1 -- 172.21.15.90:0/4055376645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ffa726b0 msgr2=0x5560ffe40e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:36.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.582+0000 7fe4fed9d640 1 --2- 172.21.15.90:0/4055376645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ffa726b0 0x5560ffe40e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.582+0000 7fe4fed9d640 1 -- 172.21.15.90:0/4055376645 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560ffa6a4d0 msgr2=0x5560ffe4ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:36.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.582+0000 7fe4fed9d640 1 --2- 172.21.15.90:0/4055376645 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560ffa6a4d0 0x5560ffe4ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.582+0000 7fe4fed9d640 1 -- 172.21.15.90:0/4055376645 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560ffe3ff10 con 0x5560ffa93390 2024-04-25T11:38:36.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.582+0000 7fe4fed9d640 1 --2- 172.21.15.90:0/4055376645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560ffa93390 0x5560ffe394e0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fe4f00096f0 tx=0x7fe4f0009200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e53f0432036089bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:36.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.582+0000 7fe4e7fff640 1 -- 172.21.15.90:0/4055376645 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f0015050 con 0x5560ffa93390 2024-04-25T11:38:36.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.583+0000 7fe4e7fff640 1 -- 172.21.15.90:0/4055376645 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4f0011d30 con 0x5560ffa93390 2024-04-25T11:38:36.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.583+0000 7fe4e7fff640 1 -- 172.21.15.90:0/4055376645 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f0004040 con 0x5560ffa93390 2024-04-25T11:38:36.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.584+0000 7fe50bd94740 1 -- 172.21.15.90:0/4055376645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560ffa93390 msgr2=0x5560ffe394e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.584+0000 7fe50bd94740 1 --2- 172.21.15.90:0/4055376645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560ffa93390 0x5560ffe394e0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fe4f00096f0 tx=0x7fe4f0009200 comp rx=0 tx=0).stop 2024-04-25T11:38:36.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.584+0000 7fe50bd94740 1 -- 172.21.15.90:0/4055376645 shutdown_connections 2024-04-25T11:38:36.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.584+0000 7fe50bd94740 1 --2- 172.21.15.90:0/4055376645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560ffa93390 0x5560ffe394e0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.584+0000 7fe50bd94740 1 --2- 172.21.15.90:0/4055376645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ffa726b0 0x5560ffe40e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.584+0000 7fe50bd94740 1 --2- 172.21.15.90:0/4055376645 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560ffa6a4d0 0x5560ffe4ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.584+0000 7fe50bd94740 1 -- 172.21.15.90:0/4055376645 >> 172.21.15.90:0/4055376645 conn(0x5560ffa874e0 msgr2=0x5560ffe40740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:36.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.584+0000 7fe50bd94740 1 -- 172.21.15.90:0/4055376645 shutdown_connections 2024-04-25T11:38:36.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.584+0000 7fe50bd94740 1 -- 172.21.15.90:0/4055376645 wait complete. 2024-04-25T11:38:36.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.585+0000 7fe50bd94740 1 -- start start 2024-04-25T11:38:36.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.585+0000 7fe50bd94740 1 Processor -- start 2024-04-25T11:38:36.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.586+0000 7fe50bd94740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ffa6a4d0 0x5560ffe474e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.586+0000 7fe50bd94740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560ffa726b0 0x5560ffe42880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.586+0000 7fe50bd94740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560ffa93390 0x5560ffe42dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.586+0000 7fe50bd94740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe5001bff60 con 0x5560ffa93390 2024-04-25T11:38:36.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.586+0000 7fe50bd94740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe5001bf890 con 0x5560ffa726b0 2024-04-25T11:38:36.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.586+0000 7fe50bd94740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe5001bcad0 con 0x5560ffa6a4d0 2024-04-25T11:38:36.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.586+0000 7fe50bd94740 10 client.0 _ll_register_callbacks cb 0x7fe4f8009ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:36.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.587+0000 7fe4fed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560ffa93390 0x5560ffe42dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.587+0000 7fe4fe59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ffa6a4d0 0x5560ffe474e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.587+0000 7fe4fed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560ffa93390 0x5560ffe42dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35002/0 (socket says 172.21.15.90:35002) 2024-04-25T11:38:36.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.587+0000 7fe4fed9d640 1 -- 172.21.15.90:0/3055000170 learned_addr learned my addr 172.21.15.90:0/3055000170 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:36.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.587+0000 7fe4fed9d640 1 -- 172.21.15.90:0/3055000170 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ffa6a4d0 msgr2=0x5560ffe474e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:36.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.587+0000 7fe4fed9d640 1 --2- 172.21.15.90:0/3055000170 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ffa6a4d0 0x5560ffe474e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.587+0000 7fe4fed9d640 1 -- 172.21.15.90:0/3055000170 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560ffa726b0 msgr2=0x5560ffe42880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:36.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.587+0000 7fe4fed9d640 1 --2- 172.21.15.90:0/3055000170 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560ffa726b0 0x5560ffe42880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.587+0000 7fe4fed9d640 1 -- 172.21.15.90:0/3055000170 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560ffe43300 con 0x5560ffa93390 2024-04-25T11:38:36.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.588+0000 7fe4fed9d640 1 --2- 172.21.15.90:0/3055000170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560ffa93390 0x5560ffe42dc0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fe4f0009450 tx=0x7fe4f0012fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71f22e06dd90a026 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:36.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.588+0000 7fe4e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:38:36.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.588+0000 7fe4e5ffb640 1 -- 172.21.15.90:0/3055000170 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f002a070 con 0x5560ffa93390 2024-04-25T11:38:36.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.588+0000 7fe4e5ffb640 1 -- 172.21.15.90:0/3055000170 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fe4f0004450 con 0x5560ffa93390 2024-04-25T11:38:36.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.588+0000 7fe50bd94740 1 -- 172.21.15.90:0/3055000170 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4f0011d30 con 0x5560ffa93390 2024-04-25T11:38:36.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.589+0000 7fe4e5ffb640 1 -- 172.21.15.90:0/3055000170 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe4f0009e80 con 0x5560ffa93390 2024-04-25T11:38:36.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.589+0000 7fe50bd94740 10 client.16680 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:36.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.589+0000 7fe50bd94740 1 -- 172.21.15.90:0/3055000170 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5560ffeb9750 con 0x5560ffa93390 2024-04-25T11:38:36.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.590+0000 7fe4e5ffb640 1 -- 172.21.15.90:0/3055000170 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4f0011d30 con 0x5560ffa93390 2024-04-25T11:38:36.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.590+0000 7fe4e5ffb640 1 -- 172.21.15.90:0/3055000170 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7fe4f0003960 con 0x5560ffa93390 2024-04-25T11:38:36.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.590+0000 7fe50bd94740 10 client.16680 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:36.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.590+0000 7fe50bd94740 10 client.16680 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:38:36.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.590+0000 7fe50bd94740 20 client.16680 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:36.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.590+0000 7fe50bd94740 1 --2- 172.21.15.90:0/3055000170 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5560ffa273a0 0x5560ffebbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.590+0000 7fe50bd94740 4 client.16680 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:38:36.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.590+0000 7fe50bd94740 1 -- 172.21.15.90:0/3055000170 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5560ffda1800 con 0x5560ffa273a0 2024-04-25T11:38:36.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.591+0000 7fe4fe59c640 1 --2- 172.21.15.90:0/3055000170 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5560ffa273a0 0x5560ffebbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.593+0000 7fe4fe59c640 1 --2- 172.21.15.90:0/3055000170 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5560ffa273a0 0x5560ffebbcc0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8b265ee0dbe019f server_cookie=a1f2c91c8d1a6b0c in_seq=0 out_seq=0 2024-04-25T11:38:36.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.593+0000 7fe4e5ffb640 10 client.16680 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:36.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.599+0000 7fe4e5ffb640 1 -- 172.21.15.90:0/3055000170 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5560ffda1800 con 0x5560ffa273a0 2024-04-25T11:38:36.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.599+0000 7fe4e5ffb640 10 client.16680 handle_command_reply: tid=0 2024-04-25T11:38:36.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.599+0000 7fe50bd94740 1 client.16680 shutdown 2024-04-25T11:38:36.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.599+0000 7fe50bd94740 1 -- 172.21.15.90:0/3055000170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560ffa93390 msgr2=0x5560ffe42dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.599+0000 7fe50bd94740 1 --2- 172.21.15.90:0/3055000170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560ffa93390 0x5560ffe42dc0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fe4f0009450 tx=0x7fe4f0012fc0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.599+0000 7fe50bd94740 1 -- 172.21.15.90:0/3055000170 shutdown_connections 2024-04-25T11:38:36.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.599+0000 7fe50bd94740 1 --2- 172.21.15.90:0/3055000170 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5560ffa273a0 0x5560ffebbcc0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.599+0000 7fe50bd94740 1 --2- 172.21.15.90:0/3055000170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560ffa93390 0x5560ffe42dc0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.599+0000 7fe50bd94740 1 --2- 172.21.15.90:0/3055000170 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560ffa726b0 0x5560ffe42880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.599+0000 7fe50bd94740 1 --2- 172.21.15.90:0/3055000170 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560ffa6a4d0 0x5560ffe474e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.599+0000 7fe50bd94740 1 -- 172.21.15.90:0/3055000170 >> 172.21.15.90:0/3055000170 conn(0x5560ffa874e0 msgr2=0x5560ffe30f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:36.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.599+0000 7fe4e5ffb640 0 client.16680 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:36.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.599+0000 7fe50bd94740 1 -- 172.21.15.90:0/3055000170 shutdown_connections 2024-04-25T11:38:36.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.599+0000 7fe50bd94740 1 -- 172.21.15.90:0/3055000170 wait complete. 2024-04-25T11:38:36.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.600+0000 7fe50bd94740 20 client.16680 trim_cache size 0 max 16384 2024-04-25T11:38:36.588 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:38:36.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.600+0000 7fe504e05640 1 -- 172.21.15.90:0/1017046258 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe4e80afd60 msgr2=0x7fe4e80b2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.600+0000 7fe504e05640 1 --2- 172.21.15.90:0/1017046258 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe4e80afd60 0x7fe4e80b2270 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x5560ffe35fb0 tx=0x7fe4f800e040 comp rx=0 tx=0).stop 2024-04-25T11:38:36.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.600+0000 7fe504e05640 1 -- 172.21.15.90:0/1017046258 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe500115fe0 msgr2=0x7fe500115580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.600+0000 7fe504e05640 1 --2- 172.21.15.90:0/1017046258 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe500115fe0 0x7fe500115580 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4005ab0 tx=0x7fe4f4003cd0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.601+0000 7fe504e05640 1 -- 172.21.15.90:0/1017046258 shutdown_connections 2024-04-25T11:38:36.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.601+0000 7fe504e05640 1 --2- 172.21.15.90:0/1017046258 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe4e80afd60 0x7fe4e80b2270 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.601+0000 7fe504e05640 1 --2- 172.21.15.90:0/1017046258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe500118a90 0x7fe5001adf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.601+0000 7fe504e05640 1 --2- 172.21.15.90:0/1017046258 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe500115fe0 0x7fe500115580 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.601+0000 7fe504e05640 1 --2- 172.21.15.90:0/1017046258 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe50010a290 0x7fe500115040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.601+0000 7fe504e05640 1 -- 172.21.15.90:0/1017046258 >> 172.21.15.90:0/1017046258 conn(0x7fe50006e7a0 msgr2=0x7fe5001163f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:36.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.601+0000 7fe504e05640 1 -- 172.21.15.90:0/1017046258 shutdown_connections 2024-04-25T11:38:36.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:36.601+0000 7fe504e05640 1 -- 172.21.15.90:0/1017046258 wait complete. 2024-04-25T11:38:36.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:36 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2643473179' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:38:36.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:36 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2643473179' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:38:36.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:36 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2643473179' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:38:37.617 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T11:38:37.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:38:37.618 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T11:38:37.703 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:37 smithi090 ceph-mon[29575]: pgmap v109: 97 pgs: 97 active+clean; 30 GiB data, 90 GiB used, 982 GiB / 1.0 TiB avail; 2.6 MiB/s rd, 757 MiB/s wr, 944 op/s 2024-04-25T11:38:37.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:37 smithi138 ceph-mon[32328]: pgmap v109: 97 pgs: 97 active+clean; 30 GiB data, 90 GiB used, 982 GiB / 1.0 TiB avail; 2.6 MiB/s rd, 757 MiB/s wr, 944 op/s 2024-04-25T11:38:37.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:37 smithi156 ceph-mon[33045]: pgmap v109: 97 pgs: 97 active+clean; 30 GiB data, 90 GiB used, 982 GiB / 1.0 TiB avail; 2.6 MiB/s rd, 757 MiB/s wr, 944 op/s 2024-04-25T11:38:37.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.896+0000 7fc21e69a640 1 -- 172.21.15.90:0/1965310769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc218102b10 msgr2=0x7fc218102f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:37.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.896+0000 7fc21e69a640 1 --2- 172.21.15.90:0/1965310769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc218102b10 0x7fc218102f20 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fc20c009900 tx=0x7fc20c02a410 comp rx=0 tx=0).stop 2024-04-25T11:38:37.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.896+0000 7fc21e69a640 1 -- 172.21.15.90:0/1965310769 shutdown_connections 2024-04-25T11:38:37.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.896+0000 7fc21e69a640 1 --2- 172.21.15.90:0/1965310769 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc21810d3a0 0x7fc21810f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.896+0000 7fc21e69a640 1 --2- 172.21.15.90:0/1965310769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc218102b10 0x7fc218102f20 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.896+0000 7fc21e69a640 1 --2- 172.21.15.90:0/1965310769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc218108a40 0x7fc2181025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.896+0000 7fc21e69a640 1 -- 172.21.15.90:0/1965310769 >> 172.21.15.90:0/1965310769 conn(0x7fc218078180 msgr2=0x7fc2180785c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:37.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.896+0000 7fc21e69a640 1 -- 172.21.15.90:0/1965310769 shutdown_connections 2024-04-25T11:38:37.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.896+0000 7fc21e69a640 1 -- 172.21.15.90:0/1965310769 wait complete. 2024-04-25T11:38:37.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.897+0000 7fc21e69a640 1 Processor -- start 2024-04-25T11:38:37.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.897+0000 7fc21e69a640 1 -- start start 2024-04-25T11:38:37.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.898+0000 7fc21e69a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc218102b10 0x7fc2181996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:37.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.898+0000 7fc21e69a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc218108a40 0x7fc218199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:37.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.898+0000 7fc21e69a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc21810d3a0 0x7fc21819de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:37.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.898+0000 7fc21e69a640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc218112c80 con 0x7fc218108a40 2024-04-25T11:38:37.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.898+0000 7fc21e69a640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc218112b00 con 0x7fc218102b10 2024-04-25T11:38:37.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.898+0000 7fc21e69a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc218112e00 con 0x7fc21810d3a0 2024-04-25T11:38:37.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.898+0000 7fc2177fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc218108a40 0x7fc218199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:37.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.898+0000 7fc217fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc218102b10 0x7fc2181996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:37.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.898+0000 7fc217fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc218102b10 0x7fc2181996c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:39106/0 (socket says 172.21.15.90:39106) 2024-04-25T11:38:37.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.898+0000 7fc2177fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc218108a40 0x7fc218199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35020/0 (socket says 172.21.15.90:35020) 2024-04-25T11:38:37.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.898+0000 7fc217fff640 1 -- 172.21.15.90:0/702538974 learned_addr learned my addr 172.21.15.90:0/702538974 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:37.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.898+0000 7fc21cc37640 1 --2- 172.21.15.90:0/702538974 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc21810d3a0 0x7fc21819de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:37.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.899+0000 7fc2177fe640 1 -- 172.21.15.90:0/702538974 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc21810d3a0 msgr2=0x7fc21819de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:37.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.899+0000 7fc2177fe640 1 --2- 172.21.15.90:0/702538974 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc21810d3a0 0x7fc21819de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.899+0000 7fc2177fe640 1 -- 172.21.15.90:0/702538974 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc218102b10 msgr2=0x7fc2181996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:37.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.899+0000 7fc2177fe640 1 --2- 172.21.15.90:0/702538974 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc218102b10 0x7fc2181996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.899+0000 7fc2177fe640 1 -- 172.21.15.90:0/702538974 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc21819e4d0 con 0x7fc218108a40 2024-04-25T11:38:37.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.899+0000 7fc2177fe640 1 --2- 172.21.15.90:0/702538974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc218108a40 0x7fc218199c20 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fc20c02a930 tx=0x7fc20c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:37.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.899+0000 7fc2157fa640 1 -- 172.21.15.90:0/702538974 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc20c03f070 con 0x7fc218108a40 2024-04-25T11:38:37.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.899+0000 7fc21e69a640 1 -- 172.21.15.90:0/702538974 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc21819e760 con 0x7fc218108a40 2024-04-25T11:38:37.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.900+0000 7fc2157fa640 1 -- 172.21.15.90:0/702538974 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc20c003ca0 con 0x7fc218108a40 2024-04-25T11:38:37.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.900+0000 7fc2157fa640 1 -- 172.21.15.90:0/702538974 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc20c03f070 con 0x7fc218108a40 2024-04-25T11:38:37.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.900+0000 7fc21e69a640 1 -- 172.21.15.90:0/702538974 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2181a6140 con 0x7fc218108a40 2024-04-25T11:38:37.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.904+0000 7fc225620740 1 Processor -- start 2024-04-25T11:38:37.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.904+0000 7fc225620740 1 -- start start 2024-04-25T11:38:37.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.905+0000 7fc225620740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b5ce89c410 0x55b5cec3e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:37.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.905+0000 7fc225620740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce873550 0x55b5cec3ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:37.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.905+0000 7fc225620740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b5ce87b730 0x55b5cec3f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:37.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.905+0000 7fc225620740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b5ceba2b00 con 0x55b5ce873550 2024-04-25T11:38:37.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.905+0000 7fc225620740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55b5ceb3e920 con 0x55b5ce89c410 2024-04-25T11:38:37.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.905+0000 7fc225620740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b5ceb9eac0 con 0x55b5ce87b730 2024-04-25T11:38:37.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.905+0000 7fc217fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce873550 0x55b5cec3ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:37.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.905+0000 7fc217fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce873550 0x55b5cec3ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35030/0 (socket says 172.21.15.90:35030) 2024-04-25T11:38:37.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.905+0000 7fc217fff640 1 -- 172.21.15.90:0/1657106438 learned_addr learned my addr 172.21.15.90:0/1657106438 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:37.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.905+0000 7fc217fff640 1 -- 172.21.15.90:0/1657106438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b5ce87b730 msgr2=0x55b5cec3f140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:38:37.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.905+0000 7fc217fff640 1 --2- 172.21.15.90:0/1657106438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b5ce87b730 0x55b5cec3f140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.905+0000 7fc21cc37640 1 --2- 172.21.15.90:0/1657106438 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b5ce89c410 0x55b5cec3e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:37.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.905+0000 7fc217fff640 1 -- 172.21.15.90:0/1657106438 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b5ce89c410 msgr2=0x55b5cec3e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:37.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.905+0000 7fc217fff640 1 --2- 172.21.15.90:0/1657106438 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b5ce89c410 0x55b5cec3e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.905+0000 7fc217fff640 1 -- 172.21.15.90:0/1657106438 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5ce97c7e0 con 0x55b5ce873550 2024-04-25T11:38:37.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.906+0000 7fc21cc37640 1 --2- 172.21.15.90:0/1657106438 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b5ce89c410 0x55b5cec3e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:38:37.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.906+0000 7fc217fff640 1 --2- 172.21.15.90:0/1657106438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce873550 0x55b5cec3ec00 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fc20800bd60 tx=0x7fc2080107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fff37b5c1485d6eb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:37.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.906+0000 7fc1fe7fc640 1 -- 172.21.15.90:0/1657106438 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc208005220 con 0x55b5ce873550 2024-04-25T11:38:37.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.906+0000 7fc2157fa640 1 -- 172.21.15.90:0/702538974 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fc20c03b4f0 con 0x7fc218108a40 2024-04-25T11:38:37.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.906+0000 7fc1fe7fc640 1 -- 172.21.15.90:0/1657106438 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc20800be50 con 0x55b5ce873550 2024-04-25T11:38:37.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.906+0000 7fc1fe7fc640 1 -- 172.21.15.90:0/1657106438 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc208011750 con 0x55b5ce873550 2024-04-25T11:38:37.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.907+0000 7fc2157fa640 1 --2- 172.21.15.90:0/702538974 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc2000afe60 0x7fc2000b2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:37.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.907+0000 7fc21cc37640 1 --2- 172.21.15.90:0/702538974 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc2000afe60 0x7fc2000b2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:37.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.908+0000 7fc225620740 1 -- 172.21.15.90:0/1657106438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce873550 msgr2=0x55b5cec3ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:37.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.908+0000 7fc225620740 1 --2- 172.21.15.90:0/1657106438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce873550 0x55b5cec3ec00 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fc20800bd60 tx=0x7fc2080107a0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.908+0000 7fc21cc37640 1 --2- 172.21.15.90:0/702538974 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc2000afe60 0x7fc2000b2370 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc21000a510 tx=0x7fc210009650 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:37.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.908+0000 7fc225620740 1 -- 172.21.15.90:0/1657106438 shutdown_connections 2024-04-25T11:38:37.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.908+0000 7fc225620740 1 --2- 172.21.15.90:0/1657106438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b5ce87b730 0x55b5cec3f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.908+0000 7fc225620740 1 --2- 172.21.15.90:0/1657106438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce873550 0x55b5cec3ec00 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.908+0000 7fc225620740 1 --2- 172.21.15.90:0/1657106438 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b5ce89c410 0x55b5cec3e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.908+0000 7fc225620740 1 -- 172.21.15.90:0/1657106438 >> 172.21.15.90:0/1657106438 conn(0x55b5ce890560 msgr2=0x55b5cec3afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:37.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.908+0000 7fc2157fa640 1 -- 172.21.15.90:0/702538974 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fc20c0c3360 con 0x7fc218108a40 2024-04-25T11:38:37.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.909+0000 7fc225620740 1 -- 172.21.15.90:0/1657106438 shutdown_connections 2024-04-25T11:38:37.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.909+0000 7fc225620740 1 -- 172.21.15.90:0/1657106438 wait complete. 2024-04-25T11:38:37.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.909+0000 7fc225620740 1 -- start start 2024-04-25T11:38:37.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.909+0000 7fc225620740 1 Processor -- start 2024-04-25T11:38:37.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.910+0000 7fc225620740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b5ce873550 0x55b5cec4b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:37.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.910+0000 7fc225620740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b5ce89c410 0x55b5cec4b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:37.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.910+0000 7fc225620740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce85b1f0 0x55b5cec57220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:37.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.910+0000 7fc225620740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b5cec4ddb0 con 0x55b5ce85b1f0 2024-04-25T11:38:37.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.910+0000 7fc225620740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55b5ceb3c5d0 con 0x55b5ce89c410 2024-04-25T11:38:37.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.910+0000 7fc225620740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b5cec4df30 con 0x55b5ce873550 2024-04-25T11:38:37.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.910+0000 7fc225620740 10 client.0 _ll_register_callbacks cb 0x55b5ceb2a080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:37.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.910+0000 7fc21cc37640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce85b1f0 0x55b5cec57220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:37.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.910+0000 7fc21cc37640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce85b1f0 0x55b5cec57220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35040/0 (socket says 172.21.15.90:35040) 2024-04-25T11:38:37.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.910+0000 7fc21cc37640 1 -- 172.21.15.90:0/3666219740 learned_addr learned my addr 172.21.15.90:0/3666219740 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:37.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.910+0000 7fc217fff640 1 --2- 172.21.15.90:0/3666219740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b5ce873550 0x55b5cec4b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:37.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.910+0000 7fc2177fe640 1 --2- 172.21.15.90:0/3666219740 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b5ce89c410 0x55b5cec4b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:37.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.910+0000 7fc21cc37640 1 -- 172.21.15.90:0/3666219740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b5ce873550 msgr2=0x55b5cec4b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:37.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.911+0000 7fc21cc37640 1 --2- 172.21.15.90:0/3666219740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b5ce873550 0x55b5cec4b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.911+0000 7fc21cc37640 1 -- 172.21.15.90:0/3666219740 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b5ce89c410 msgr2=0x55b5cec4b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:37.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.911+0000 7fc21cc37640 1 --2- 172.21.15.90:0/3666219740 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b5ce89c410 0x55b5cec4b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.911+0000 7fc21cc37640 1 -- 172.21.15.90:0/3666219740 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5ce9dcf70 con 0x55b5ce85b1f0 2024-04-25T11:38:37.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.911+0000 7fc2177fe640 1 --2- 172.21.15.90:0/3666219740 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b5ce89c410 0x55b5cec4b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:38:37.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.911+0000 7fc21cc37640 1 --2- 172.21.15.90:0/3666219740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce85b1f0 0x55b5cec57220 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fc210019e60 tx=0x7fc210004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0084c92c1986f50 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:37.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.911+0000 7fc1f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:38:37.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.911+0000 7fc1f77fe640 1 -- 172.21.15.90:0/3666219740 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2100192e0 con 0x55b5ce85b1f0 2024-04-25T11:38:37.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.911+0000 7fc225620740 1 -- 172.21.15.90:0/3666219740 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc20800be50 con 0x55b5ce85b1f0 2024-04-25T11:38:37.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.911+0000 7fc1f77fe640 1 -- 172.21.15.90:0/3666219740 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fc210016960 con 0x55b5ce85b1f0 2024-04-25T11:38:37.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.911+0000 7fc1f77fe640 1 -- 172.21.15.90:0/3666219740 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc210016ba0 con 0x55b5ce85b1f0 2024-04-25T11:38:37.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.911+0000 7fc225620740 10 client.16692 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:37.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.911+0000 7fc225620740 1 -- 172.21.15.90:0/3666219740 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b5ce874170 con 0x55b5ce85b1f0 2024-04-25T11:38:37.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.912+0000 7fc1f77fe640 1 -- 172.21.15.90:0/3666219740 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc20800be50 con 0x55b5ce85b1f0 2024-04-25T11:38:37.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.913+0000 7fc1f77fe640 1 -- 172.21.15.90:0/3666219740 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7fc210022400 con 0x55b5ce85b1f0 2024-04-25T11:38:37.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.913+0000 7fc225620740 10 client.16692 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:37.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.913+0000 7fc225620740 10 client.16692 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:38:37.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.913+0000 7fc225620740 20 client.16692 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:37.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.913+0000 7fc225620740 1 --2- 172.21.15.90:0/3666219740 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b5ce830630 0x55b5cec5d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:37.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.913+0000 7fc225620740 4 client.16692 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:38:37.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.913+0000 7fc225620740 1 -- 172.21.15.90:0/3666219740 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5cebb0710 con 0x55b5ce830630 2024-04-25T11:38:37.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.914+0000 7fc217fff640 1 --2- 172.21.15.90:0/3666219740 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b5ce830630 0x55b5cec5d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:37.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.915+0000 7fc1f77fe640 10 client.16692 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:37.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.915+0000 7fc217fff640 1 --2- 172.21.15.90:0/3666219740 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b5ce830630 0x55b5cec5d3b0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b5671b7632e85b5 server_cookie=4290bff750142d46 in_seq=0 out_seq=0 2024-04-25T11:38:37.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.917+0000 7fc1f77fe640 1 -- 172.21.15.90:0/3666219740 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b5cebb0710 con 0x55b5ce830630 2024-04-25T11:38:37.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.917+0000 7fc1f77fe640 10 client.16692 handle_command_reply: tid=0 2024-04-25T11:38:37.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.917+0000 7fc225620740 1 client.16692 shutdown 2024-04-25T11:38:37.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.917+0000 7fc225620740 1 -- 172.21.15.90:0/3666219740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce85b1f0 msgr2=0x55b5cec57220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:37.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.917+0000 7fc225620740 1 --2- 172.21.15.90:0/3666219740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce85b1f0 0x55b5cec57220 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fc210019e60 tx=0x7fc210004b50 comp rx=0 tx=0).stop 2024-04-25T11:38:37.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.917+0000 7fc225620740 1 -- 172.21.15.90:0/3666219740 shutdown_connections 2024-04-25T11:38:37.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.917+0000 7fc225620740 1 --2- 172.21.15.90:0/3666219740 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b5ce830630 0x55b5cec5d3b0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.918+0000 7fc225620740 1 --2- 172.21.15.90:0/3666219740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce85b1f0 0x55b5cec57220 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.918+0000 7fc225620740 1 --2- 172.21.15.90:0/3666219740 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b5ce89c410 0x55b5cec4b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.918+0000 7fc225620740 1 --2- 172.21.15.90:0/3666219740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b5ce873550 0x55b5cec4b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.918+0000 7fc1f77fe640 0 client.16692 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:37.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.918+0000 7fc225620740 1 -- 172.21.15.90:0/3666219740 >> 172.21.15.90:0/3666219740 conn(0x55b5ce890560 msgr2=0x55b5cec404e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:37.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.918+0000 7fc225620740 1 -- 172.21.15.90:0/3666219740 shutdown_connections 2024-04-25T11:38:37.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.918+0000 7fc225620740 1 -- 172.21.15.90:0/3666219740 wait complete. 2024-04-25T11:38:37.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.918+0000 7fc225620740 20 client.16692 trim_cache size 0 max 16384 2024-04-25T11:38:37.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.939+0000 7fc225620740 1 Processor -- start 2024-04-25T11:38:37.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.939+0000 7fc225620740 1 -- start start 2024-04-25T11:38:37.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.939+0000 7fc225620740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b5ce85b1f0 0x55b5cec4ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:37.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.939+0000 7fc225620740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b5ce873550 0x55b5cec40b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:37.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.939+0000 7fc225620740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce89c410 0x55b5cec62b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:37.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.939+0000 7fc225620740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc2181ada00 con 0x55b5ce89c410 2024-04-25T11:38:37.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.939+0000 7fc225620740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc2181af570 con 0x55b5ce85b1f0 2024-04-25T11:38:37.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.939+0000 7fc225620740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc2181ae0d0 con 0x55b5ce873550 2024-04-25T11:38:37.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.940+0000 7fc21cc37640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce89c410 0x55b5cec62b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:37.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.940+0000 7fc21cc37640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce89c410 0x55b5cec62b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35046/0 (socket says 172.21.15.90:35046) 2024-04-25T11:38:37.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.940+0000 7fc2177fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b5ce873550 0x55b5cec40b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:37.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.940+0000 7fc21cc37640 1 -- 172.21.15.90:0/1698242510 learned_addr learned my addr 172.21.15.90:0/1698242510 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:37.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.940+0000 7fc217fff640 1 --2- 172.21.15.90:0/1698242510 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b5ce85b1f0 0x55b5cec4ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:37.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.940+0000 7fc21cc37640 1 -- 172.21.15.90:0/1698242510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b5ce873550 msgr2=0x55b5cec40b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:37.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.940+0000 7fc21cc37640 1 --2- 172.21.15.90:0/1698242510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b5ce873550 0x55b5cec40b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.940+0000 7fc21cc37640 1 -- 172.21.15.90:0/1698242510 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b5ce85b1f0 msgr2=0x55b5cec4ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:37.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.940+0000 7fc21cc37640 1 --2- 172.21.15.90:0/1698242510 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b5ce85b1f0 0x55b5cec4ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.940+0000 7fc21cc37640 1 -- 172.21.15.90:0/1698242510 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5cec45dd0 con 0x55b5ce89c410 2024-04-25T11:38:37.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.940+0000 7fc2177fe640 1 --2- 172.21.15.90:0/1698242510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b5ce873550 0x55b5cec40b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:37.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.940+0000 7fc217fff640 1 --2- 172.21.15.90:0/1698242510 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b5ce85b1f0 0x55b5cec4ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:37.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.940+0000 7fc21cc37640 1 --2- 172.21.15.90:0/1698242510 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce89c410 0x55b5cec62b70 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fc21000afd0 tx=0x7fc210004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f4df2ba6fbdfb2c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:37.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.940+0000 7fc1fdffb640 1 -- 172.21.15.90:0/1698242510 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc210016960 con 0x55b5ce89c410 2024-04-25T11:38:37.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.941+0000 7fc1fdffb640 1 -- 172.21.15.90:0/1698242510 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc210016d40 con 0x55b5ce89c410 2024-04-25T11:38:37.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.941+0000 7fc1fdffb640 1 -- 172.21.15.90:0/1698242510 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc21002ab30 con 0x55b5ce89c410 2024-04-25T11:38:37.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.942+0000 7fc225620740 1 -- 172.21.15.90:0/1698242510 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce89c410 msgr2=0x55b5cec62b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:37.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.942+0000 7fc225620740 1 --2- 172.21.15.90:0/1698242510 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce89c410 0x55b5cec62b70 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fc21000afd0 tx=0x7fc210004a40 comp rx=0 tx=0).stop 2024-04-25T11:38:37.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.942+0000 7fc225620740 1 -- 172.21.15.90:0/1698242510 shutdown_connections 2024-04-25T11:38:37.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.942+0000 7fc225620740 1 --2- 172.21.15.90:0/1698242510 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce89c410 0x55b5cec62b70 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.942+0000 7fc225620740 1 --2- 172.21.15.90:0/1698242510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b5ce873550 0x55b5cec40b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.942+0000 7fc225620740 1 --2- 172.21.15.90:0/1698242510 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b5ce85b1f0 0x55b5cec4ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.942+0000 7fc225620740 1 -- 172.21.15.90:0/1698242510 >> 172.21.15.90:0/1698242510 conn(0x55b5ce890560 msgr2=0x55b5cec40470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:37.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.942+0000 7fc225620740 1 -- 172.21.15.90:0/1698242510 shutdown_connections 2024-04-25T11:38:37.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.942+0000 7fc225620740 1 -- 172.21.15.90:0/1698242510 wait complete. 2024-04-25T11:38:37.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.942+0000 7fc225620740 1 -- start start 2024-04-25T11:38:37.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.942+0000 7fc225620740 1 Processor -- start 2024-04-25T11:38:37.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.943+0000 7fc225620740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce85b1f0 0x55b5cec532f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:37.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.943+0000 7fc225620740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b5ce873550 0x55b5cec53830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:37.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.943+0000 7fc225620740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b5ce89c410 0x55b5cec4e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:37.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.943+0000 7fc225620740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc2181aee70 con 0x55b5ce85b1f0 2024-04-25T11:38:37.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.943+0000 7fc225620740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc2181ae7a0 con 0x55b5ce89c410 2024-04-25T11:38:37.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.943+0000 7fc225620740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc2181ab9e0 con 0x55b5ce873550 2024-04-25T11:38:37.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.943+0000 7fc225620740 10 client.0 _ll_register_callbacks cb 0x7fc210019b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:37.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.943+0000 7fc217fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce85b1f0 0x55b5cec532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:37.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.943+0000 7fc21cc37640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b5ce89c410 0x55b5cec4e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:37.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.943+0000 7fc21cc37640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b5ce89c410 0x55b5cec4e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:39126/0 (socket says 172.21.15.90:39126) 2024-04-25T11:38:37.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.943+0000 7fc217fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce85b1f0 0x55b5cec532f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35060/0 (socket says 172.21.15.90:35060) 2024-04-25T11:38:37.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.943+0000 7fc21cc37640 1 -- 172.21.15.90:0/3579932494 learned_addr learned my addr 172.21.15.90:0/3579932494 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:37.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.943+0000 7fc2177fe640 1 --2- 172.21.15.90:0/3579932494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b5ce873550 0x55b5cec53830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:37.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.943+0000 7fc21cc37640 1 -- 172.21.15.90:0/3579932494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b5ce873550 msgr2=0x55b5cec53830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:37.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.943+0000 7fc21cc37640 1 --2- 172.21.15.90:0/3579932494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b5ce873550 0x55b5cec53830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.943+0000 7fc21cc37640 1 -- 172.21.15.90:0/3579932494 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce85b1f0 msgr2=0x55b5cec532f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:37.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.943+0000 7fc21cc37640 1 --2- 172.21.15.90:0/3579932494 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce85b1f0 0x55b5cec532f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.943+0000 7fc21cc37640 1 -- 172.21.15.90:0/3579932494 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5cec4efa0 con 0x55b5ce89c410 2024-04-25T11:38:37.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.944+0000 7fc21cc37640 1 --2- 172.21.15.90:0/3579932494 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b5ce89c410 0x55b5cec4e830 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc210022a60 tx=0x7fc2100228f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1566cb84c9dc3b0d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:37.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.944+0000 7fc1f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:38:37.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.944+0000 7fc1f77fe640 1 -- 172.21.15.90:0/3579932494 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2100192e0 con 0x55b5ce89c410 2024-04-25T11:38:37.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.944+0000 7fc1f77fe640 1 -- 172.21.15.90:0/3579932494 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fc21002a420 con 0x55b5ce89c410 2024-04-25T11:38:37.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.944+0000 7fc225620740 1 -- 172.21.15.90:0/3579932494 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc210016d40 con 0x55b5ce89c410 2024-04-25T11:38:37.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.944+0000 7fc1f77fe640 1 -- 172.21.15.90:0/3579932494 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc21002a640 con 0x55b5ce89c410 2024-04-25T11:38:37.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.944+0000 7fc225620740 10 client.25928 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:37.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.944+0000 7fc225620740 1 -- 172.21.15.90:0/3579932494 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b5cecc5240 con 0x55b5ce89c410 2024-04-25T11:38:37.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.944+0000 7fc1f77fe640 1 -- 172.21.15.90:0/3579932494 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc210016d40 con 0x55b5ce89c410 2024-04-25T11:38:37.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.944+0000 7fc1f77fe640 1 -- 172.21.15.90:0/3579932494 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7fc210017940 con 0x55b5ce89c410 2024-04-25T11:38:37.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.944+0000 7fc225620740 10 client.25928 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:37.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.944+0000 7fc225620740 10 client.25928 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:38:37.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.944+0000 7fc225620740 20 client.25928 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:37.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.944+0000 7fc225620740 1 --2- 172.21.15.90:0/3579932494 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b5cecc5870 0x55b5cecc7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:37.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.944+0000 7fc225620740 4 client.25928 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T11:38:37.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.944+0000 7fc225620740 1 -- 172.21.15.90:0/3579932494 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b5cebaa580 con 0x55b5cecc5870 2024-04-25T11:38:37.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.945+0000 7fc217fff640 1 --2- 172.21.15.90:0/3579932494 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b5cecc5870 0x55b5cecc7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:37.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.945+0000 7fc1f77fe640 10 client.25928 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:37.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.945+0000 7fc217fff640 1 --2- 172.21.15.90:0/3579932494 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b5cecc5870 0x55b5cecc7c90 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bf0f334a01b2d1b server_cookie=3af91ae084c3662f in_seq=0 out_seq=0 2024-04-25T11:38:37.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.946+0000 7fc1f77fe640 1 -- 172.21.15.90:0/3579932494 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b5cebaa580 con 0x55b5cecc5870 2024-04-25T11:38:37.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.946+0000 7fc1f77fe640 10 client.25928 handle_command_reply: tid=0 2024-04-25T11:38:37.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.946+0000 7fc225620740 1 client.25928 shutdown 2024-04-25T11:38:37.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.947+0000 7fc225620740 1 -- 172.21.15.90:0/3579932494 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b5ce89c410 msgr2=0x55b5cec4e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:37.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.947+0000 7fc225620740 1 --2- 172.21.15.90:0/3579932494 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b5ce89c410 0x55b5cec4e830 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc210022a60 tx=0x7fc2100228f0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.947+0000 7fc225620740 1 -- 172.21.15.90:0/3579932494 shutdown_connections 2024-04-25T11:38:37.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.947+0000 7fc225620740 1 --2- 172.21.15.90:0/3579932494 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b5cecc5870 0x55b5cecc7c90 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.947+0000 7fc225620740 1 --2- 172.21.15.90:0/3579932494 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b5ce89c410 0x55b5cec4e830 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.947+0000 7fc225620740 1 --2- 172.21.15.90:0/3579932494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b5ce873550 0x55b5cec53830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.947+0000 7fc1f77fe640 0 client.25928 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:37.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.947+0000 7fc225620740 1 --2- 172.21.15.90:0/3579932494 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5ce85b1f0 0x55b5cec532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.947+0000 7fc225620740 1 -- 172.21.15.90:0/3579932494 >> 172.21.15.90:0/3579932494 conn(0x55b5ce890560 msgr2=0x55b5cec4c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:37.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.947+0000 7fc225620740 1 -- 172.21.15.90:0/3579932494 shutdown_connections 2024-04-25T11:38:37.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.947+0000 7fc225620740 1 -- 172.21.15.90:0/3579932494 wait complete. 2024-04-25T11:38:37.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.947+0000 7fc225620740 20 client.25928 trim_cache size 0 max 16384 2024-04-25T11:38:37.938 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:38:37.938 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-25T11:38:37.938 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "185d3c2c-3876-4b34-b0e9-782c3f7912f8", 2024-04-25T11:38:37.938 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-25T11:38:37.938 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:38:37.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.947+0000 7fc21e69a640 1 -- 172.21.15.90:0/702538974 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc2000afe60 msgr2=0x7fc2000b2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:37.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.947+0000 7fc21e69a640 1 --2- 172.21.15.90:0/702538974 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc2000afe60 0x7fc2000b2370 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc21000a510 tx=0x7fc210009650 comp rx=0 tx=0).stop 2024-04-25T11:38:37.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.947+0000 7fc21e69a640 1 -- 172.21.15.90:0/702538974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc218108a40 msgr2=0x7fc218199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:37.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.947+0000 7fc21e69a640 1 --2- 172.21.15.90:0/702538974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc218108a40 0x7fc218199c20 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fc20c02a930 tx=0x7fc20c004270 comp rx=0 tx=0).stop 2024-04-25T11:38:37.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.947+0000 7fc21e69a640 1 -- 172.21.15.90:0/702538974 shutdown_connections 2024-04-25T11:38:37.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.947+0000 7fc21e69a640 1 --2- 172.21.15.90:0/702538974 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc2000afe60 0x7fc2000b2370 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.947+0000 7fc21e69a640 1 --2- 172.21.15.90:0/702538974 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc21810d3a0 0x7fc21819de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.947+0000 7fc21e69a640 1 --2- 172.21.15.90:0/702538974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc218108a40 0x7fc218199c20 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.947+0000 7fc21e69a640 1 --2- 172.21.15.90:0/702538974 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc218102b10 0x7fc2181996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:37.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.948+0000 7fc21e69a640 1 -- 172.21.15.90:0/702538974 >> 172.21.15.90:0/702538974 conn(0x7fc218078180 msgr2=0x7fc2181040d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:37.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.948+0000 7fc21e69a640 1 -- 172.21.15.90:0/702538974 shutdown_connections 2024-04-25T11:38:37.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:37.948+0000 7fc21e69a640 1 -- 172.21.15.90:0/702538974 wait complete. 2024-04-25T11:38:37.947 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:38:37.948 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:38:38.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.083+0000 7f58c0bf3640 1 -- 172.21.15.90:0/2727216072 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58bc10b720 msgr2=0x7f58bc10db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:38.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.083+0000 7f58c0bf3640 1 --2- 172.21.15.90:0/2727216072 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58bc10b720 0x7f58bc10db40 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f58a4009900 tx=0x7f58a402a480 comp rx=0 tx=0).stop 2024-04-25T11:38:38.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.083+0000 7f58c0bf3640 1 -- 172.21.15.90:0/2727216072 shutdown_connections 2024-04-25T11:38:38.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.083+0000 7f58c0bf3640 1 --2- 172.21.15.90:0/2727216072 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f58bc10e080 0x7f58bc110570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.083+0000 7f58c0bf3640 1 --2- 172.21.15.90:0/2727216072 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58bc10b720 0x7f58bc10db40 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.083+0000 7f58c0bf3640 1 --2- 172.21.15.90:0/2727216072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58bc104cc0 0x7f58bc1050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.083+0000 7f58c0bf3640 1 -- 172.21.15.90:0/2727216072 >> 172.21.15.90:0/2727216072 conn(0x7f58bc078180 msgr2=0x7f58bc0785c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:38.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.083+0000 7f58c0bf3640 1 -- 172.21.15.90:0/2727216072 shutdown_connections 2024-04-25T11:38:38.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.083+0000 7f58c0bf3640 1 -- 172.21.15.90:0/2727216072 wait complete. 2024-04-25T11:38:38.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.084+0000 7f58c0bf3640 1 Processor -- start 2024-04-25T11:38:38.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.084+0000 7f58c0bf3640 1 -- start start 2024-04-25T11:38:38.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.085+0000 7f58c0bf3640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f58bc104cc0 0x7f58bc19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:38.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.085+0000 7f58c0bf3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58bc10b720 0x7f58bc19ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:38.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.085+0000 7f58c0bf3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58bc10e080 0x7f58bc19ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:38.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.085+0000 7f58c0bf3640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f58bc113cd0 con 0x7f58bc10b720 2024-04-25T11:38:38.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.085+0000 7f58c0bf3640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f58bc113b50 con 0x7f58bc104cc0 2024-04-25T11:38:38.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.085+0000 7f58c0bf3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f58bc113e50 con 0x7f58bc10e080 2024-04-25T11:38:38.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.085+0000 7f58b1d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58bc10b720 0x7f58bc19ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:38.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.085+0000 7f58b1d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58bc10b720 0x7f58bc19ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35068/0 (socket says 172.21.15.90:35068) 2024-04-25T11:38:38.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.085+0000 7f58b1d9b640 1 -- 172.21.15.90:0/3247461223 learned_addr learned my addr 172.21.15.90:0/3247461223 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:38.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.085+0000 7f58bad9d640 1 --2- 172.21.15.90:0/3247461223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58bc10e080 0x7f58bc19ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:38.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.085+0000 7f58ba59c640 1 --2- 172.21.15.90:0/3247461223 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f58bc104cc0 0x7f58bc19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:38.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.085+0000 7f58b1d9b640 1 -- 172.21.15.90:0/3247461223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58bc10e080 msgr2=0x7f58bc19ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:38.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.085+0000 7f58b1d9b640 1 --2- 172.21.15.90:0/3247461223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58bc10e080 0x7f58bc19ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.085+0000 7f58b1d9b640 1 -- 172.21.15.90:0/3247461223 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f58bc104cc0 msgr2=0x7f58bc19a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:38.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.085+0000 7f58b1d9b640 1 --2- 172.21.15.90:0/3247461223 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f58bc104cc0 0x7f58bc19a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.085+0000 7f58b1d9b640 1 -- 172.21.15.90:0/3247461223 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58bc19f590 con 0x7f58bc10b720 2024-04-25T11:38:38.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.085+0000 7f58b1d9b640 1 --2- 172.21.15.90:0/3247461223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58bc10b720 0x7f58bc19ac90 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f58a4005940 tx=0x7f58a4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:38.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.085+0000 7f58b3fff640 1 -- 172.21.15.90:0/3247461223 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a403f030 con 0x7f58bc10b720 2024-04-25T11:38:38.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.085+0000 7f58b3fff640 1 -- 172.21.15.90:0/3247461223 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58a4003aa0 con 0x7f58bc10b720 2024-04-25T11:38:38.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.085+0000 7f58c0bf3640 1 -- 172.21.15.90:0/3247461223 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58bc19f820 con 0x7f58bc10b720 2024-04-25T11:38:38.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.086+0000 7f58b3fff640 1 -- 172.21.15.90:0/3247461223 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a4035070 con 0x7f58bc10b720 2024-04-25T11:38:38.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.086+0000 7f58c0bf3640 1 -- 172.21.15.90:0/3247461223 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58bc19fd10 con 0x7f58bc10b720 2024-04-25T11:38:38.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.088+0000 7f58b3fff640 1 -- 172.21.15.90:0/3247461223 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f58a4042030 con 0x7f58bc10b720 2024-04-25T11:38:38.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.088+0000 7f58c7b7c740 1 Processor -- start 2024-04-25T11:38:38.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.088+0000 7f58c7b7c740 1 -- start start 2024-04-25T11:38:38.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.089+0000 7f58c7b7c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588f8563390 0x5588f8903910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:38.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.089+0000 7f58c7b7c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5588f853a4d0 0x5588f8903e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:38.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.089+0000 7f58c7b7c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f85426b0 0x5588f8914d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:38.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.089+0000 7f58c7b7c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5588f8869d80 con 0x5588f8563390 2024-04-25T11:38:38.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.089+0000 7f58c7b7c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5588f88238b0 con 0x5588f853a4d0 2024-04-25T11:38:38.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.089+0000 7f58c7b7c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5588f8865d40 con 0x5588f85426b0 2024-04-25T11:38:38.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.089+0000 7f58bad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588f8563390 0x5588f8903910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:38.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.089+0000 7f58bad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588f8563390 0x5588f8903910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35082/0 (socket says 172.21.15.90:35082) 2024-04-25T11:38:38.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.089+0000 7f58bad9d640 1 -- 172.21.15.90:0/2879038193 learned_addr learned my addr 172.21.15.90:0/2879038193 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:38.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.089+0000 7f58ba59c640 1 --2- 172.21.15.90:0/2879038193 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5588f853a4d0 0x5588f8903e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:38.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.089+0000 7f58b1d9b640 1 --2- 172.21.15.90:0/2879038193 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f85426b0 0x5588f8914d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:38.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.089+0000 7f58b3fff640 1 --2- 172.21.15.90:0/3247461223 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f58a80afc60 0x7f58a80b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:38.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.089+0000 7f58ba59c640 1 -- 172.21.15.90:0/2879038193 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f85426b0 msgr2=0x5588f8914d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:38.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.089+0000 7f58ba59c640 1 --2- 172.21.15.90:0/2879038193 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f85426b0 0x5588f8914d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.089+0000 7f58ba59c640 1 -- 172.21.15.90:0/2879038193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588f8563390 msgr2=0x5588f8903910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:38.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.089+0000 7f58ba59c640 1 --2- 172.21.15.90:0/2879038193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588f8563390 0x5588f8903910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.089+0000 7f58ba59c640 1 -- 172.21.15.90:0/2879038193 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588f87f4a40 con 0x5588f853a4d0 2024-04-25T11:38:38.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.089+0000 7f58bad9d640 1 --2- 172.21.15.90:0/2879038193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588f8563390 0x5588f8903910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:38.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.089+0000 7f58bad9d640 1 --2- 172.21.15.90:0/3247461223 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f58a80afc60 0x7f58a80b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:38.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.089+0000 7f58b3fff640 1 -- 172.21.15.90:0/3247461223 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f58a40ff6f0 con 0x7f58bc10b720 2024-04-25T11:38:38.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.090+0000 7f58ba59c640 1 --2- 172.21.15.90:0/2879038193 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5588f853a4d0 0x5588f8903e50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f58ac00ab40 tx=0x7f58ac00f3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4921ac12decac81b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:38.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.090+0000 7f58bad9d640 1 --2- 172.21.15.90:0/3247461223 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f58a80afc60 0x7f58a80b2170 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f58b40107d0 tx=0x7f58b4004a40 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:38.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.090+0000 7f58b0d99640 1 -- 172.21.15.90:0/2879038193 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58ac004020 con 0x5588f853a4d0 2024-04-25T11:38:38.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.090+0000 7f58b0d99640 1 -- 172.21.15.90:0/2879038193 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58ac0041c0 con 0x5588f853a4d0 2024-04-25T11:38:38.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.090+0000 7f58b0d99640 1 -- 172.21.15.90:0/2879038193 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58ac0186a0 con 0x5588f853a4d0 2024-04-25T11:38:38.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.091+0000 7f58c7b7c740 1 -- 172.21.15.90:0/2879038193 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5588f853a4d0 msgr2=0x5588f8903e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:38.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.091+0000 7f58c7b7c740 1 --2- 172.21.15.90:0/2879038193 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5588f853a4d0 0x5588f8903e50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f58ac00ab40 tx=0x7f58ac00f3c0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.091+0000 7f58c7b7c740 1 -- 172.21.15.90:0/2879038193 shutdown_connections 2024-04-25T11:38:38.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.091+0000 7f58c7b7c740 1 --2- 172.21.15.90:0/2879038193 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f85426b0 0x5588f8914d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.091+0000 7f58c7b7c740 1 --2- 172.21.15.90:0/2879038193 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5588f853a4d0 0x5588f8903e50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.091+0000 7f58c7b7c740 1 --2- 172.21.15.90:0/2879038193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588f8563390 0x5588f8903910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.091+0000 7f58c7b7c740 1 -- 172.21.15.90:0/2879038193 >> 172.21.15.90:0/2879038193 conn(0x5588f85574e0 msgr2=0x5588f8900210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:38.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.091+0000 7f58c7b7c740 1 -- 172.21.15.90:0/2879038193 shutdown_connections 2024-04-25T11:38:38.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.091+0000 7f58c7b7c740 1 -- 172.21.15.90:0/2879038193 wait complete. 2024-04-25T11:38:38.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.091+0000 7f58c7b7c740 1 -- start start 2024-04-25T11:38:38.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.092+0000 7f58c7b7c740 1 Processor -- start 2024-04-25T11:38:38.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.092+0000 7f58c7b7c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f853a4d0 0x5588f8912950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:38.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.092+0000 7f58c7b7c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588f85426b0 0x5588f8912e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:38.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.092+0000 7f58c7b7c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5588f8563390 0x5588f891e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:38.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.092+0000 7f58c7b7c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5588f89153f0 con 0x5588f85426b0 2024-04-25T11:38:38.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.092+0000 7f58c7b7c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5588f87f0d10 con 0x5588f8563390 2024-04-25T11:38:38.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.092+0000 7f58c7b7c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5588f8915570 con 0x5588f853a4d0 2024-04-25T11:38:38.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.092+0000 7f58c7b7c740 10 client.0 _ll_register_callbacks cb 0x5588f881d210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:38.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.092+0000 7f58b1d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588f85426b0 0x5588f8912e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:38.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.092+0000 7f58b1d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588f85426b0 0x5588f8912e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35090/0 (socket says 172.21.15.90:35090) 2024-04-25T11:38:38.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.092+0000 7f58b1d9b640 1 -- 172.21.15.90:0/4080088637 learned_addr learned my addr 172.21.15.90:0/4080088637 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:38.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.092+0000 7f58ba59c640 1 --2- 172.21.15.90:0/4080088637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f853a4d0 0x5588f8912950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:38.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.092+0000 7f58bad9d640 1 --2- 172.21.15.90:0/4080088637 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5588f8563390 0x5588f891e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:38.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.092+0000 7f58b1d9b640 1 -- 172.21.15.90:0/4080088637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f853a4d0 msgr2=0x5588f8912950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:38.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.092+0000 7f58b1d9b640 1 --2- 172.21.15.90:0/4080088637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f853a4d0 0x5588f8912950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.092+0000 7f58b1d9b640 1 -- 172.21.15.90:0/4080088637 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5588f8563390 msgr2=0x5588f891e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:38.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.092+0000 7f58b1d9b640 1 --2- 172.21.15.90:0/4080088637 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5588f8563390 0x5588f891e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.092+0000 7f58b1d9b640 1 -- 172.21.15.90:0/4080088637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588f86457d0 con 0x5588f85426b0 2024-04-25T11:38:38.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.092+0000 7f58ba59c640 1 --2- 172.21.15.90:0/4080088637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f853a4d0 0x5588f8912950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:38.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.093+0000 7f58b1d9b640 1 --2- 172.21.15.90:0/4080088637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588f85426b0 0x5588f8912e90 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f58a40fce40 tx=0x7f58a40c3f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b754a1134d02471 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:38.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.093+0000 7f58a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:38:38.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.093+0000 7f58a1ffb640 1 -- 172.21.15.90:0/4080088637 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a4049070 con 0x5588f85426b0 2024-04-25T11:38:38.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.093+0000 7f58a1ffb640 1 -- 172.21.15.90:0/4080088637 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f58a403bcc0 con 0x5588f85426b0 2024-04-25T11:38:38.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.093+0000 7f58c7b7c740 1 -- 172.21.15.90:0/4080088637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58ac0041c0 con 0x5588f85426b0 2024-04-25T11:38:38.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.093+0000 7f58a1ffb640 1 -- 172.21.15.90:0/4080088637 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f58a40c7230 con 0x5588f85426b0 2024-04-25T11:38:38.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.093+0000 7f58c7b7c740 10 client.16710 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:38.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.093+0000 7f58c7b7c740 1 -- 172.21.15.90:0/4080088637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5588f857b3f0 con 0x5588f85426b0 2024-04-25T11:38:38.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.093+0000 7f58a1ffb640 1 -- 172.21.15.90:0/4080088637 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58ac0041c0 con 0x5588f85426b0 2024-04-25T11:38:38.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.093+0000 7f58a1ffb640 1 -- 172.21.15.90:0/4080088637 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f58a4101ce0 con 0x5588f85426b0 2024-04-25T11:38:38.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.093+0000 7f58c7b7c740 10 client.16710 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:38.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.093+0000 7f58c7b7c740 10 client.16710 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:38:38.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.094+0000 7f58c7b7c740 20 client.16710 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:38.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.094+0000 7f58c7b7c740 1 --2- 172.21.15.90:0/4080088637 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5588f8521f60 0x5588f8924a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:38.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.094+0000 7f58c7b7c740 4 client.16710 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:38:38.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.094+0000 7f58c7b7c740 1 -- 172.21.15.90:0/4080088637 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588f8877990 con 0x5588f8521f60 2024-04-25T11:38:38.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.094+0000 7f58ba59c640 1 --2- 172.21.15.90:0/4080088637 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5588f8521f60 0x5588f8924a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:38.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.094+0000 7f58a1ffb640 10 client.16710 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:38.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.095+0000 7f58ba59c640 1 --2- 172.21.15.90:0/4080088637 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5588f8521f60 0x5588f8924a80 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3599a9b3206c6f6 server_cookie=4ef93f135f548dbe in_seq=0 out_seq=0 2024-04-25T11:38:38.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.096+0000 7f58a1ffb640 1 -- 172.21.15.90:0/4080088637 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5588f8877990 con 0x5588f8521f60 2024-04-25T11:38:38.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.096+0000 7f58a1ffb640 10 client.16710 handle_command_reply: tid=0 2024-04-25T11:38:38.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.096+0000 7f58c7b7c740 1 client.16710 shutdown 2024-04-25T11:38:38.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.097+0000 7f58c7b7c740 1 -- 172.21.15.90:0/4080088637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588f85426b0 msgr2=0x5588f8912e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:38.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.097+0000 7f58c7b7c740 1 --2- 172.21.15.90:0/4080088637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588f85426b0 0x5588f8912e90 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f58a40fce40 tx=0x7f58a40c3f90 comp rx=0 tx=0).stop 2024-04-25T11:38:38.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.097+0000 7f58c7b7c740 1 -- 172.21.15.90:0/4080088637 shutdown_connections 2024-04-25T11:38:38.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.097+0000 7f58c7b7c740 1 --2- 172.21.15.90:0/4080088637 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5588f8521f60 0x5588f8924a80 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.097+0000 7f58c7b7c740 1 --2- 172.21.15.90:0/4080088637 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5588f8563390 0x5588f891e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.097+0000 7f58a1ffb640 0 client.16710 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:38.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.097+0000 7f58c7b7c740 1 --2- 172.21.15.90:0/4080088637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588f85426b0 0x5588f8912e90 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.097+0000 7f58c7b7c740 1 --2- 172.21.15.90:0/4080088637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f853a4d0 0x5588f8912950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.097+0000 7f58c7b7c740 1 -- 172.21.15.90:0/4080088637 >> 172.21.15.90:0/4080088637 conn(0x5588f85574e0 msgr2=0x5588f89056f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:38.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.097+0000 7f58c7b7c740 1 -- 172.21.15.90:0/4080088637 shutdown_connections 2024-04-25T11:38:38.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.097+0000 7f58c7b7c740 1 -- 172.21.15.90:0/4080088637 wait complete. 2024-04-25T11:38:38.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.097+0000 7f58c7b7c740 20 client.16710 trim_cache size 0 max 16384 2024-04-25T11:38:38.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.118+0000 7f58c7b7c740 1 Processor -- start 2024-04-25T11:38:38.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.118+0000 7f58c7b7c740 1 -- start start 2024-04-25T11:38:38.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.119+0000 7f58c7b7c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5588f8521f60 0x5588f891a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:38.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.119+0000 7f58c7b7c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588f853a4d0 0x5588f891adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:38.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.119+0000 7f58c7b7c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f85426b0 0x5588f8905660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:38.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.119+0000 7f58c7b7c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f58bc1ad940 con 0x5588f853a4d0 2024-04-25T11:38:38.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.119+0000 7f58c7b7c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f58bc1af4b0 con 0x5588f8521f60 2024-04-25T11:38:38.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.119+0000 7f58c7b7c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f58bc1ae010 con 0x5588f85426b0 2024-04-25T11:38:38.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.119+0000 7f58b1d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588f853a4d0 0x5588f891adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:38.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.119+0000 7f58b1d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588f853a4d0 0x5588f891adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35094/0 (socket says 172.21.15.90:35094) 2024-04-25T11:38:38.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.119+0000 7f58b1d9b640 1 -- 172.21.15.90:0/1788897752 learned_addr learned my addr 172.21.15.90:0/1788897752 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:38.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.120+0000 7f58bad9d640 1 --2- 172.21.15.90:0/1788897752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f85426b0 0x5588f8905660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:38.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.120+0000 7f58ba59c640 1 --2- 172.21.15.90:0/1788897752 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5588f8521f60 0x5588f891a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:38.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.120+0000 7f58b1d9b640 1 -- 172.21.15.90:0/1788897752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f85426b0 msgr2=0x5588f8905660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:38.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.120+0000 7f58b1d9b640 1 --2- 172.21.15.90:0/1788897752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f85426b0 0x5588f8905660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.120+0000 7f58b1d9b640 1 -- 172.21.15.90:0/1788897752 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5588f8521f60 msgr2=0x5588f891a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:38.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.120+0000 7f58b1d9b640 1 --2- 172.21.15.90:0/1788897752 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5588f8521f60 0x5588f891a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.120+0000 7f58b1d9b640 1 -- 172.21.15.90:0/1788897752 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588f853b0f0 con 0x5588f853a4d0 2024-04-25T11:38:38.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.120+0000 7f58b1d9b640 1 --2- 172.21.15.90:0/1788897752 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588f853a4d0 0x5588f891adc0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f58a40c3960 tx=0x7f58a403b520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c372c57e3a43808 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:38.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.120+0000 7f58a3fff640 1 -- 172.21.15.90:0/1788897752 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a40e0020 con 0x5588f853a4d0 2024-04-25T11:38:38.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.121+0000 7f58a3fff640 1 -- 172.21.15.90:0/1788897752 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58a40c73f0 con 0x5588f853a4d0 2024-04-25T11:38:38.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.121+0000 7f58a3fff640 1 -- 172.21.15.90:0/1788897752 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a4049070 con 0x5588f853a4d0 2024-04-25T11:38:38.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.122+0000 7f58c7b7c740 1 -- 172.21.15.90:0/1788897752 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588f853a4d0 msgr2=0x5588f891adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:38.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.122+0000 7f58c7b7c740 1 --2- 172.21.15.90:0/1788897752 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588f853a4d0 0x5588f891adc0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f58a40c3960 tx=0x7f58a403b520 comp rx=0 tx=0).stop 2024-04-25T11:38:38.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.122+0000 7f58c7b7c740 1 -- 172.21.15.90:0/1788897752 shutdown_connections 2024-04-25T11:38:38.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.122+0000 7f58c7b7c740 1 --2- 172.21.15.90:0/1788897752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f85426b0 0x5588f8905660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.122+0000 7f58c7b7c740 1 --2- 172.21.15.90:0/1788897752 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588f853a4d0 0x5588f891adc0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.122+0000 7f58c7b7c740 1 --2- 172.21.15.90:0/1788897752 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5588f8521f60 0x5588f891a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.122+0000 7f58c7b7c740 1 -- 172.21.15.90:0/1788897752 >> 172.21.15.90:0/1788897752 conn(0x5588f85574e0 msgr2=0x5588f8911010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:38.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.122+0000 7f58c7b7c740 1 -- 172.21.15.90:0/1788897752 shutdown_connections 2024-04-25T11:38:38.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.122+0000 7f58c7b7c740 1 -- 172.21.15.90:0/1788897752 wait complete. 2024-04-25T11:38:38.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.123+0000 7f58c7b7c740 1 -- start start 2024-04-25T11:38:38.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.123+0000 7f58c7b7c740 1 Processor -- start 2024-04-25T11:38:38.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.124+0000 7f58c7b7c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588f8521f60 0x5588f891a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:38.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.124+0000 7f58c7b7c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f853a4d0 0x5588f891aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:38.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.124+0000 7f58c7b7c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5588f85426b0 0x5588f8915ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:38.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.124+0000 7f58c7b7c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f58bc1aedb0 con 0x5588f8521f60 2024-04-25T11:38:38.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.124+0000 7f58c7b7c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f58bc1ae6e0 con 0x5588f85426b0 2024-04-25T11:38:38.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.124+0000 7f58c7b7c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f58bc1ab940 con 0x5588f853a4d0 2024-04-25T11:38:38.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.124+0000 7f58c7b7c740 10 client.0 _ll_register_callbacks cb 0x7f58a40d00b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:38.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.125+0000 7f58b1d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f853a4d0 0x5588f891aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:38.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.125+0000 7f58b1d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f853a4d0 0x5588f891aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:33848/0 (socket says 172.21.15.90:33848) 2024-04-25T11:38:38.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.125+0000 7f58b1d9b640 1 -- 172.21.15.90:0/3640964336 learned_addr learned my addr 172.21.15.90:0/3640964336 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:38.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.125+0000 7f58ba59c640 1 --2- 172.21.15.90:0/3640964336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588f8521f60 0x5588f891a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:38.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.125+0000 7f58b1d9b640 1 -- 172.21.15.90:0/3640964336 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5588f85426b0 msgr2=0x5588f8915ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:38.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.125+0000 7f58b1d9b640 1 --2- 172.21.15.90:0/3640964336 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5588f85426b0 0x5588f8915ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.125+0000 7f58b1d9b640 1 -- 172.21.15.90:0/3640964336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588f8521f60 msgr2=0x5588f891a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:38.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.125+0000 7f58b1d9b640 1 --2- 172.21.15.90:0/3640964336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588f8521f60 0x5588f891a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.125+0000 7f58b1d9b640 1 -- 172.21.15.90:0/3640964336 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588f882a4b0 con 0x5588f853a4d0 2024-04-25T11:38:38.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.127+0000 7f58b1d9b640 1 --2- 172.21.15.90:0/3640964336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f853a4d0 0x5588f891aed0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f58a40d0160 tx=0x7f58a402a480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=451e96a521ef4831 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:38.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.127+0000 7f58a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:38:38.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.128+0000 7f58a1ffb640 1 -- 172.21.15.90:0/3640964336 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a40e0070 con 0x5588f853a4d0 2024-04-25T11:38:38.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.129+0000 7f58a1ffb640 1 -- 172.21.15.90:0/3640964336 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f58a40c8b60 con 0x5588f853a4d0 2024-04-25T11:38:38.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.129+0000 7f58c7b7c740 1 -- 172.21.15.90:0/3640964336 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58a40c73f0 con 0x5588f853a4d0 2024-04-25T11:38:38.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.131+0000 7f58a1ffb640 1 -- 172.21.15.90:0/3640964336 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f58a4049070 con 0x5588f853a4d0 2024-04-25T11:38:38.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.131+0000 7f58c7b7c740 10 client.25876 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:38.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.131+0000 7f58c7b7c740 1 -- 172.21.15.90:0/3640964336 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5588f882a4b0 con 0x5588f853a4d0 2024-04-25T11:38:38.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.132+0000 7f58a1ffb640 1 -- 172.21.15.90:0/3640964336 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58a40c73f0 con 0x5588f853a4d0 2024-04-25T11:38:38.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.132+0000 7f58a1ffb640 1 -- 172.21.15.90:0/3640964336 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f58a40035f0 con 0x5588f853a4d0 2024-04-25T11:38:38.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.132+0000 7f58c7b7c740 10 client.25876 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:38.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.132+0000 7f58c7b7c740 10 client.25876 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:38:38.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.132+0000 7f58c7b7c740 20 client.25876 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:38.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.132+0000 7f58c7b7c740 1 --2- 172.21.15.90:0/3640964336 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5588f8563390 0x5588f898e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:38.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.132+0000 7f58c7b7c740 4 client.25876 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:38:38.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.132+0000 7f58c7b7c740 1 -- 172.21.15.90:0/3640964336 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5588f8871800 con 0x5588f8563390 2024-04-25T11:38:38.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.136+0000 7f58ba59c640 1 --2- 172.21.15.90:0/3640964336 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5588f8563390 0x5588f898e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:38.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.145+0000 7f58ba59c640 1 --2- 172.21.15.90:0/3640964336 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5588f8563390 0x5588f898e9c0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56ab66c84325aef5 server_cookie=cbb3ad8053bc4a15 in_seq=0 out_seq=0 2024-04-25T11:38:38.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.145+0000 7f58a1ffb640 10 client.25876 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:38.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.149+0000 7f58a1ffb640 1 -- 172.21.15.90:0/3640964336 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x5588f8871800 con 0x5588f8563390 2024-04-25T11:38:38.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.150+0000 7f58a1ffb640 10 client.25876 handle_command_reply: tid=0 2024-04-25T11:38:38.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.150+0000 7f58c7b7c740 1 client.25876 shutdown 2024-04-25T11:38:38.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.153+0000 7f58c7b7c740 1 -- 172.21.15.90:0/3640964336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f853a4d0 msgr2=0x5588f891aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:38.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.153+0000 7f58c7b7c740 1 --2- 172.21.15.90:0/3640964336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f853a4d0 0x5588f891aed0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f58a40d0160 tx=0x7f58a402a480 comp rx=0 tx=0).stop 2024-04-25T11:38:38.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.155+0000 7f58c7b7c740 1 -- 172.21.15.90:0/3640964336 shutdown_connections 2024-04-25T11:38:38.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.155+0000 7f58c7b7c740 1 --2- 172.21.15.90:0/3640964336 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5588f8563390 0x5588f898e9c0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.155+0000 7f58c7b7c740 1 --2- 172.21.15.90:0/3640964336 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5588f85426b0 0x5588f8915ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.155+0000 7f58a1ffb640 0 client.25876 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:38.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.155+0000 7f58c7b7c740 1 --2- 172.21.15.90:0/3640964336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f853a4d0 0x5588f891aed0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.155+0000 7f58c7b7c740 1 --2- 172.21.15.90:0/3640964336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588f8521f60 0x5588f891a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.155+0000 7f58c7b7c740 1 -- 172.21.15.90:0/3640964336 >> 172.21.15.90:0/3640964336 conn(0x5588f85574e0 msgr2=0x5588f8905660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:38.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.162+0000 7f58c7b7c740 1 -- 172.21.15.90:0/3640964336 shutdown_connections 2024-04-25T11:38:38.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.170+0000 7f58c7b7c740 1 -- 172.21.15.90:0/3640964336 wait complete. 2024-04-25T11:38:38.162 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:38:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "scrub active (0 inodes in the stack)", 2024-04-25T11:38:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-25T11:38:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: "185d3c2c-3876-4b34-b0e9-782c3f7912f8": { 2024-04-25T11:38:38.163 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-25T11:38:38.163 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "185d3c2c-3876-4b34-b0e9-782c3f7912f8", 2024-04-25T11:38:38.163 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-25T11:38:38.163 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:38:38.163 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:38:38.163 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:38:38.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.174+0000 7f58c7b7c740 20 client.25876 trim_cache size 0 max 16384 2024-04-25T11:38:38.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.179+0000 7f58c0bf3640 1 -- 172.21.15.90:0/3247461223 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f58a80afc60 msgr2=0x7f58a80b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:38.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.179+0000 7f58c0bf3640 1 --2- 172.21.15.90:0/3247461223 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f58a80afc60 0x7f58a80b2170 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f58b40107d0 tx=0x7f58b4004a40 comp rx=0 tx=0).stop 2024-04-25T11:38:38.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.179+0000 7f58c0bf3640 1 -- 172.21.15.90:0/3247461223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58bc10b720 msgr2=0x7f58bc19ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:38.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.179+0000 7f58c0bf3640 1 --2- 172.21.15.90:0/3247461223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58bc10b720 0x7f58bc19ac90 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f58a4005940 tx=0x7f58a4004270 comp rx=0 tx=0).stop 2024-04-25T11:38:38.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.180+0000 7f58c0bf3640 1 -- 172.21.15.90:0/3247461223 shutdown_connections 2024-04-25T11:38:38.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.180+0000 7f58c0bf3640 1 --2- 172.21.15.90:0/3247461223 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f58a80afc60 0x7f58a80b2170 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.180+0000 7f58c0bf3640 1 --2- 172.21.15.90:0/3247461223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58bc10e080 0x7f58bc19ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.180+0000 7f58c0bf3640 1 --2- 172.21.15.90:0/3247461223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58bc10b720 0x7f58bc19ac90 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.180+0000 7f58c0bf3640 1 --2- 172.21.15.90:0/3247461223 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f58bc104cc0 0x7f58bc19a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.180+0000 7f58c0bf3640 1 -- 172.21.15.90:0/3247461223 >> 172.21.15.90:0/3247461223 conn(0x7f58bc078180 msgr2=0x7f58bc1011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:38.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.183+0000 7f58c0bf3640 1 -- 172.21.15.90:0/3247461223 shutdown_connections 2024-04-25T11:38:38.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:38:38.186+0000 7f58c0bf3640 1 -- 172.21.15.90:0/3247461223 wait complete. 2024-04-25T11:38:38.232 INFO:tasks.cephfs.filesystem:scrub status for tag:185d3c2c-3876-4b34-b0e9-782c3f7912f8 - {'path': '/', 'tag': '185d3c2c-3876-4b34-b0e9-782c3f7912f8', 'options': 'recursive,force'} 2024-04-25T11:38:38.725 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:38 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:38:38.725 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:38 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:38:38.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:38 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:38:38.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:38 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:38:38.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:38 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:38:38.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:38 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:38:39.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:39 smithi156 ceph-mon[33045]: scrub queued for path: / 2024-04-25T11:38:39.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:39 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:38:39.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:39 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:38:39.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:39 smithi156 ceph-mon[33045]: pgmap v110: 97 pgs: 97 active+clean; 30 GiB data, 90 GiB used, 982 GiB / 1.0 TiB avail; 1.6 MiB/s rd, 591 MiB/s wr, 873 op/s 2024-04-25T11:38:39.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:39 smithi090 ceph-mon[29575]: scrub queued for path: / 2024-04-25T11:38:39.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:39 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:38:39.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:39 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:38:39.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:39 smithi090 ceph-mon[29575]: pgmap v110: 97 pgs: 97 active+clean; 30 GiB data, 90 GiB used, 982 GiB / 1.0 TiB avail; 1.6 MiB/s rd, 591 MiB/s wr, 873 op/s 2024-04-25T11:38:40.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:39 smithi138 ceph-mon[32328]: scrub queued for path: / 2024-04-25T11:38:40.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:39 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:38:40.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:39 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:38:40.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:39 smithi138 ceph-mon[32328]: pgmap v110: 97 pgs: 97 active+clean; 30 GiB data, 90 GiB used, 982 GiB / 1.0 TiB avail; 1.6 MiB/s rd, 591 MiB/s wr, 873 op/s 2024-04-25T11:38:40.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:40 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:38:41.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:40 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:38:41.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:40 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:38:41.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:41 smithi090 ceph-mon[29575]: pgmap v111: 97 pgs: 97 active+clean; 30 GiB data, 90 GiB used, 983 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 798 MiB/s wr, 1.17k op/s 2024-04-25T11:38:42.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:41 smithi138 ceph-mon[32328]: pgmap v111: 97 pgs: 97 active+clean; 30 GiB data, 90 GiB used, 983 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 798 MiB/s wr, 1.17k op/s 2024-04-25T11:38:42.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:41 smithi156 ceph-mon[33045]: pgmap v111: 97 pgs: 97 active+clean; 30 GiB data, 90 GiB used, 983 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 798 MiB/s wr, 1.17k op/s 2024-04-25T11:38:43.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:43 smithi090 ceph-mon[29575]: pgmap v112: 97 pgs: 97 active+clean; 30 GiB data, 90 GiB used, 983 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 477 MiB/s wr, 674 op/s 2024-04-25T11:38:44.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:43 smithi138 ceph-mon[32328]: pgmap v112: 97 pgs: 97 active+clean; 30 GiB data, 90 GiB used, 983 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 477 MiB/s wr, 674 op/s 2024-04-25T11:38:44.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:43 smithi156 ceph-mon[33045]: pgmap v112: 97 pgs: 97 active+clean; 30 GiB data, 90 GiB used, 983 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 477 MiB/s wr, 674 op/s 2024-04-25T11:38:46.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:45 smithi138 ceph-mon[32328]: pgmap v113: 97 pgs: 97 active+clean; 30 GiB data, 90 GiB used, 982 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 564 MiB/s wr, 766 op/s 2024-04-25T11:38:46.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:45 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:38:46.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:45 smithi156 ceph-mon[33045]: pgmap v113: 97 pgs: 97 active+clean; 30 GiB data, 90 GiB used, 982 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 564 MiB/s wr, 766 op/s 2024-04-25T11:38:46.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:45 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:38:46.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:45 smithi090 ceph-mon[29575]: pgmap v113: 97 pgs: 97 active+clean; 30 GiB data, 90 GiB used, 982 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 564 MiB/s wr, 766 op/s 2024-04-25T11:38:46.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:45 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:38:47.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:46 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:38:47.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:46 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:38:47.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:46 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:38:47.986 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:47.999+0000 7f9769e92640 1 -- 172.21.15.90:0/1558298617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97641069e0 msgr2=0x7f9764106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:47.986 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:47.999+0000 7f9769e92640 1 --2- 172.21.15.90:0/1558298617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97641069e0 0x7f9764106e10 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f9758009900 tx=0x7f975802a410 comp rx=0 tx=0).stop 2024-04-25T11:38:47.986 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:47.999+0000 7f9769e92640 1 -- 172.21.15.90:0/1558298617 shutdown_connections 2024-04-25T11:38:47.986 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:47.999+0000 7f9769e92640 1 --2- 172.21.15.90:0/1558298617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97641074a0 0x7f976410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:47.987 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:47.999+0000 7f9769e92640 1 --2- 172.21.15.90:0/1558298617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97641069e0 0x7f9764106e10 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:47.987 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:47.999+0000 7f9769e92640 1 --2- 172.21.15.90:0/1558298617 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9764102450 0x7f9764102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:47.987 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:47.999+0000 7f9769e92640 1 -- 172.21.15.90:0/1558298617 >> 172.21.15.90:0/1558298617 conn(0x7f97640778b0 msgr2=0x7f9764077cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:47.987 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:47.999+0000 7f9769e92640 1 -- 172.21.15.90:0/1558298617 shutdown_connections 2024-04-25T11:38:47.987 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:47.999+0000 7f9769e92640 1 -- 172.21.15.90:0/1558298617 wait complete. 2024-04-25T11:38:47.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.000+0000 7f9769e92640 1 Processor -- start 2024-04-25T11:38:47.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.000+0000 7f9769e92640 1 -- start start 2024-04-25T11:38:47.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.001+0000 7f9769e92640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9764102450 0x7f9764198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:47.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.001+0000 7f9769e92640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97641069e0 0x7f9764198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:47.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.001+0000 7f9769e92640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97641074a0 0x7f976419cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:47.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.001+0000 7f9769e92640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9764111b30 con 0x7f97641069e0 2024-04-25T11:38:47.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.001+0000 7f9769e92640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f97641119b0 con 0x7f9764102450 2024-04-25T11:38:47.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.001+0000 7f9769e92640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9764111cb0 con 0x7f97641074a0 2024-04-25T11:38:47.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.001+0000 7f9762ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97641069e0 0x7f9764198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:47.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.001+0000 7f9762ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97641069e0 0x7f9764198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41878/0 (socket says 172.21.15.90:41878) 2024-04-25T11:38:47.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.001+0000 7f9762ffd640 1 -- 172.21.15.90:0/219176077 learned_addr learned my addr 172.21.15.90:0/219176077 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:47.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.001+0000 7f97637fe640 1 --2- 172.21.15.90:0/219176077 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9764102450 0x7f9764198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:47.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.001+0000 7f9763fff640 1 --2- 172.21.15.90:0/219176077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97641074a0 0x7f976419cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:47.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.001+0000 7f9763fff640 1 -- 172.21.15.90:0/219176077 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9764102450 msgr2=0x7f9764198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:47.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.001+0000 7f9763fff640 1 --2- 172.21.15.90:0/219176077 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9764102450 0x7f9764198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:47.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.001+0000 7f9763fff640 1 -- 172.21.15.90:0/219176077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97641069e0 msgr2=0x7f9764198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:47.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.001+0000 7f9763fff640 1 --2- 172.21.15.90:0/219176077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97641069e0 0x7f9764198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:47.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.001+0000 7f9763fff640 1 -- 172.21.15.90:0/219176077 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f976419d3f0 con 0x7f97641074a0 2024-04-25T11:38:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.001+0000 7f9762ffd640 1 --2- 172.21.15.90:0/219176077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97641069e0 0x7f9764198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.001+0000 7f97637fe640 1 --2- 172.21.15.90:0/219176077 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9764102450 0x7f9764198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.001+0000 7f9763fff640 1 --2- 172.21.15.90:0/219176077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97641074a0 0x7f976419cdb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f975c009fd0 tx=0x7f975c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.002+0000 7f9760ff9640 1 -- 172.21.15.90:0/219176077 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975c00ce20 con 0x7f97641074a0 2024-04-25T11:38:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.002+0000 7f9769e92640 1 -- 172.21.15.90:0/219176077 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f976419d6e0 con 0x7f97641074a0 2024-04-25T11:38:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.002+0000 7f9760ff9640 1 -- 172.21.15.90:0/219176077 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f975c005020 con 0x7f97641074a0 2024-04-25T11:38:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.002+0000 7f9760ff9640 1 -- 172.21.15.90:0/219176077 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975c010690 con 0x7f97641074a0 2024-04-25T11:38:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.002+0000 7f9769e92640 1 -- 172.21.15.90:0/219176077 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97641a5060 con 0x7f97641074a0 2024-04-25T11:38:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.005+0000 7f9760ff9640 1 -- 172.21.15.90:0/219176077 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f975c010830 con 0x7f97641074a0 2024-04-25T11:38:47.992 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.005+0000 7f9770e18740 1 Processor -- start 2024-04-25T11:38:47.992 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.005+0000 7f9770e18740 1 -- start start 2024-04-25T11:38:47.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.006+0000 7f9770e18740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55649ced21b0 0x55649d271940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:47.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.006+0000 7f9770e18740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55649cea92f0 0x55649d271e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:47.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.006+0000 7f9770e18740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55649ceb14d0 0x55649d2723c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:47.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.006+0000 7f9770e18740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55649d1d8460 con 0x55649ced21b0 2024-04-25T11:38:47.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.006+0000 7f9770e18740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55649d192590 con 0x55649cea92f0 2024-04-25T11:38:47.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.006+0000 7f9770e18740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55649d1d4420 con 0x55649ceb14d0 2024-04-25T11:38:47.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.006+0000 7f9762ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55649ced21b0 0x55649d271940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:47.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.006+0000 7f9762ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55649ced21b0 0x55649d271940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41894/0 (socket says 172.21.15.90:41894) 2024-04-25T11:38:47.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.006+0000 7f9762ffd640 1 -- 172.21.15.90:0/4030424512 learned_addr learned my addr 172.21.15.90:0/4030424512 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:47.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.006+0000 7f97637fe640 1 --2- 172.21.15.90:0/4030424512 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55649cea92f0 0x55649d271e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:47.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.006+0000 7f9760ff9640 1 --2- 172.21.15.90:0/219176077 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f974c0b40f0 0x7f974c0b6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:47.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.006+0000 7f9762ffd640 1 --2- 172.21.15.90:0/4030424512 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55649ceb14d0 0x55649d2723c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:47.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.006+0000 7f97637fe640 1 -- 172.21.15.90:0/4030424512 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55649ceb14d0 msgr2=0x55649d2723c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:47.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.006+0000 7f97637fe640 1 --2- 172.21.15.90:0/4030424512 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55649ceb14d0 0x55649d2723c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:47.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.006+0000 7f97637fe640 1 -- 172.21.15.90:0/4030424512 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55649ced21b0 msgr2=0x55649d271940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:47.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.006+0000 7f97637fe640 1 --2- 172.21.15.90:0/4030424512 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55649ced21b0 0x55649d271940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:47.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.006+0000 7f97637fe640 1 -- 172.21.15.90:0/4030424512 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55649d19f0d0 con 0x55649cea92f0 2024-04-25T11:38:47.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.007+0000 7f9763fff640 1 --2- 172.21.15.90:0/219176077 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f974c0b40f0 0x7f974c0b6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:47.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.007+0000 7f97637fe640 1 --2- 172.21.15.90:0/4030424512 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55649cea92f0 0x55649d271e80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9754009fd0 tx=0x7f97540118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93be3f118f4692a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:47.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.007+0000 7f9760ff9640 1 -- 172.21.15.90:0/219176077 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f975c0df6c0 con 0x7f97641074a0 2024-04-25T11:38:47.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.007+0000 7f9749ffb640 1 -- 172.21.15.90:0/4030424512 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9754020bc0 con 0x55649cea92f0 2024-04-25T11:38:47.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.007+0000 7f9749ffb640 1 -- 172.21.15.90:0/4030424512 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9754020d60 con 0x55649cea92f0 2024-04-25T11:38:47.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.008+0000 7f9749ffb640 1 -- 172.21.15.90:0/4030424512 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9754004a00 con 0x55649cea92f0 2024-04-25T11:38:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.008+0000 7f9770e18740 1 -- 172.21.15.90:0/4030424512 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55649cea92f0 msgr2=0x55649d271e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.008+0000 7f9770e18740 1 --2- 172.21.15.90:0/4030424512 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55649cea92f0 0x55649d271e80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9754009fd0 tx=0x7f97540118d0 comp rx=0 tx=0).stop 2024-04-25T11:38:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.008+0000 7f9763fff640 1 --2- 172.21.15.90:0/219176077 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f974c0b40f0 0x7f974c0b6600 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f975c0a4a70 tx=0x7f975c0d5f50 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.009+0000 7f9770e18740 1 -- 172.21.15.90:0/4030424512 shutdown_connections 2024-04-25T11:38:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.009+0000 7f9770e18740 1 --2- 172.21.15.90:0/4030424512 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55649ceb14d0 0x55649d2723c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.009+0000 7f9770e18740 1 --2- 172.21.15.90:0/4030424512 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55649cea92f0 0x55649d271e80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.009+0000 7f9770e18740 1 --2- 172.21.15.90:0/4030424512 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55649ced21b0 0x55649d271940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.009+0000 7f9770e18740 1 -- 172.21.15.90:0/4030424512 >> 172.21.15.90:0/4030424512 conn(0x55649cec6300 msgr2=0x55649d26e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.009+0000 7f9770e18740 1 -- 172.21.15.90:0/4030424512 shutdown_connections 2024-04-25T11:38:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.009+0000 7f9770e18740 1 -- 172.21.15.90:0/4030424512 wait complete. 2024-04-25T11:38:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.009+0000 7f9770e18740 1 -- start start 2024-04-25T11:38:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.009+0000 7f9770e18740 1 Processor -- start 2024-04-25T11:38:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.010+0000 7f9770e18740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55649cea92f0 0x55649d280260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.010+0000 7f9770e18740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55649ceb14d0 0x55649d27c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.010+0000 7f9770e18740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55649ced21b0 0x55649d27cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.010+0000 7f9770e18740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55649d274ba0 con 0x55649ceb14d0 2024-04-25T11:38:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.010+0000 7f9770e18740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55649d15fb60 con 0x55649ced21b0 2024-04-25T11:38:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.010+0000 7f9770e18740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55649d274d20 con 0x55649cea92f0 2024-04-25T11:38:48.002 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.010+0000 7f9762ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55649cea92f0 0x55649d280260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:48.002 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.010+0000 7f9762ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55649cea92f0 0x55649d280260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:60284/0 (socket says 172.21.15.90:60284) 2024-04-25T11:38:48.002 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.010+0000 7f9762ffd640 1 -- 172.21.15.90:0/2491766228 learned_addr learned my addr 172.21.15.90:0/2491766228 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:48.002 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.010+0000 7f97637fe640 1 --2- 172.21.15.90:0/2491766228 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55649ceb14d0 0x55649d27c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:48.002 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.010+0000 7f9762ffd640 1 --2- 172.21.15.90:0/2491766228 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55649ced21b0 0x55649d27cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:48.002 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.010+0000 7f9770e18740 10 client.0 _ll_register_callbacks cb 0x55649d1842b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:48.002 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.010+0000 7f9762ffd640 1 -- 172.21.15.90:0/2491766228 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55649ced21b0 msgr2=0x55649d27cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:48.002 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.010+0000 7f9762ffd640 1 --2- 172.21.15.90:0/2491766228 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55649ced21b0 0x55649d27cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:48.002 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.010+0000 7f9762ffd640 1 -- 172.21.15.90:0/2491766228 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55649ceb14d0 msgr2=0x55649d27c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:48.002 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.010+0000 7f9762ffd640 1 --2- 172.21.15.90:0/2491766228 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55649ceb14d0 0x55649d27c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:48.002 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.010+0000 7f9762ffd640 1 -- 172.21.15.90:0/2491766228 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55649ceea210 con 0x55649cea92f0 2024-04-25T11:38:48.002 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.011+0000 7f9762ffd640 1 --2- 172.21.15.90:0/2491766228 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55649cea92f0 0x55649d280260 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x55649d268670 tx=0x7f9758032ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56635498cd021e70 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:48.002 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.011+0000 7f9742ffd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:38:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.011+0000 7f9742ffd640 1 -- 172.21.15.90:0/2491766228 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9758035040 con 0x55649cea92f0 2024-04-25T11:38:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.011+0000 7f9770e18740 1 -- 172.21.15.90:0/2491766228 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9754020d60 con 0x55649cea92f0 2024-04-25T11:38:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.011+0000 7f9742ffd640 1 -- 172.21.15.90:0/2491766228 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f9758004340 con 0x55649cea92f0 2024-04-25T11:38:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.011+0000 7f9742ffd640 1 -- 172.21.15.90:0/2491766228 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9758005020 con 0x55649cea92f0 2024-04-25T11:38:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.011+0000 7f9770e18740 10 client.25888 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.011+0000 7f9770e18740 1 -- 172.21.15.90:0/2491766228 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55649cfb6620 con 0x55649cea92f0 2024-04-25T11:38:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.011+0000 7f9742ffd640 1 -- 172.21.15.90:0/2491766228 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9754020d60 con 0x55649cea92f0 2024-04-25T11:38:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.012+0000 7f9742ffd640 1 -- 172.21.15.90:0/2491766228 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f975804a030 con 0x55649cea92f0 2024-04-25T11:38:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.012+0000 7f9770e18740 10 client.25888 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.012+0000 7f9770e18740 10 client.25888 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:38:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.012+0000 7f9770e18740 20 client.25888 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.012+0000 7f9770e18740 1 --2- 172.21.15.90:0/2491766228 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55649ce90d80 0x55649d28e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.012+0000 7f9770e18740 4 client.25888 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:38:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.012+0000 7f9770e18740 1 -- 172.21.15.90:0/2491766228 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55649d1e6070 con 0x55649ce90d80 2024-04-25T11:38:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.012+0000 7f97637fe640 1 --2- 172.21.15.90:0/2491766228 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55649ce90d80 0x55649d28e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.013+0000 7f9742ffd640 10 client.25888 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.013+0000 7f97637fe640 1 --2- 172.21.15.90:0/2491766228 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55649ce90d80 0x55649d28e6d0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e70f84173b17b2cb server_cookie=bb5c9f478e4a6e8b in_seq=0 out_seq=0 2024-04-25T11:38:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.013+0000 7f9742ffd640 1 -- 172.21.15.90:0/2491766228 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55649d1e6070 con 0x55649ce90d80 2024-04-25T11:38:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.013+0000 7f9742ffd640 10 client.25888 handle_command_reply: tid=0 2024-04-25T11:38:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.013+0000 7f9770e18740 1 client.25888 shutdown 2024-04-25T11:38:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.014+0000 7f9770e18740 1 -- 172.21.15.90:0/2491766228 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55649cea92f0 msgr2=0x55649d280260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.014+0000 7f9770e18740 1 --2- 172.21.15.90:0/2491766228 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55649cea92f0 0x55649d280260 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x55649d268670 tx=0x7f9758032ed0 comp rx=0 tx=0).stop 2024-04-25T11:38:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.014+0000 7f9770e18740 1 -- 172.21.15.90:0/2491766228 shutdown_connections 2024-04-25T11:38:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.014+0000 7f9770e18740 1 --2- 172.21.15.90:0/2491766228 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55649ce90d80 0x55649d28e6d0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.014+0000 7f9770e18740 1 --2- 172.21.15.90:0/2491766228 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55649ced21b0 0x55649d27cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.014+0000 7f9742ffd640 0 client.25888 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.014+0000 7f9770e18740 1 --2- 172.21.15.90:0/2491766228 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55649ceb14d0 0x55649d27c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:48.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.014+0000 7f9770e18740 1 --2- 172.21.15.90:0/2491766228 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55649cea92f0 0x55649d280260 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:48.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.014+0000 7f9770e18740 1 -- 172.21.15.90:0/2491766228 >> 172.21.15.90:0/2491766228 conn(0x55649cec6300 msgr2=0x55649d2722b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:48.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.014+0000 7f9770e18740 1 -- 172.21.15.90:0/2491766228 shutdown_connections 2024-04-25T11:38:48.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.014+0000 7f9770e18740 1 -- 172.21.15.90:0/2491766228 wait complete. 2024-04-25T11:38:48.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.014+0000 7f9770e18740 20 client.25888 trim_cache size 0 max 16384 2024-04-25T11:38:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.037+0000 7f9770e18740 1 Processor -- start 2024-04-25T11:38:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.037+0000 7f9770e18740 1 -- start start 2024-04-25T11:38:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.037+0000 7f9770e18740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55649ce90d80 0x55649d272890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.037+0000 7f9770e18740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55649cea92f0 0x55649d275fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.037+0000 7f9770e18740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55649ceb14d0 0x55649d276500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.037+0000 7f9770e18740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f97641ada30 con 0x55649cea92f0 2024-04-25T11:38:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.037+0000 7f9770e18740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f97641af5a0 con 0x55649ce90d80 2024-04-25T11:38:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.037+0000 7f9770e18740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f97641ae100 con 0x55649ceb14d0 2024-04-25T11:38:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.037+0000 7f97637fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55649cea92f0 0x55649d275fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.037+0000 7f9762ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55649ce90d80 0x55649d272890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.037+0000 7f97637fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55649cea92f0 0x55649d275fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41902/0 (socket says 172.21.15.90:41902) 2024-04-25T11:38:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.037+0000 7f97637fe640 1 -- 172.21.15.90:0/544183896 learned_addr learned my addr 172.21.15.90:0/544183896 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.037+0000 7f9762ffd640 1 --2- 172.21.15.90:0/544183896 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55649ce90d80 0x55649d272890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:57780/0 (socket says 172.21.15.90:57780) 2024-04-25T11:38:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.037+0000 7f9762ffd640 1 --2- 172.21.15.90:0/544183896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55649ceb14d0 0x55649d276500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.038+0000 7f9762ffd640 1 -- 172.21.15.90:0/544183896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55649ceb14d0 msgr2=0x55649d276500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.038+0000 7f9762ffd640 1 --2- 172.21.15.90:0/544183896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55649ceb14d0 0x55649d276500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.038+0000 7f9762ffd640 1 -- 172.21.15.90:0/544183896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55649cea92f0 msgr2=0x55649d275fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.038+0000 7f9762ffd640 1 --2- 172.21.15.90:0/544183896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55649cea92f0 0x55649d275fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.038+0000 7f9762ffd640 1 -- 172.21.15.90:0/544183896 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55649d27ab10 con 0x55649ce90d80 2024-04-25T11:38:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.038+0000 7f97637fe640 1 --2- 172.21.15.90:0/544183896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55649cea92f0 0x55649d275fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.038+0000 7f9762ffd640 1 --2- 172.21.15.90:0/544183896 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55649ce90d80 0x55649d272890 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9758004c60 tx=0x7f9758004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f45776da631df76e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.038+0000 7f97497fa640 1 -- 172.21.15.90:0/544183896 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9758042030 con 0x55649ce90d80 2024-04-25T11:38:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.038+0000 7f97497fa640 1 -- 172.21.15.90:0/544183896 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9758004e50 con 0x55649ce90d80 2024-04-25T11:38:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.038+0000 7f97497fa640 1 -- 172.21.15.90:0/544183896 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9758035040 con 0x55649ce90d80 2024-04-25T11:38:48.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.040+0000 7f9770e18740 1 -- 172.21.15.90:0/544183896 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55649ce90d80 msgr2=0x55649d272890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:48.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.040+0000 7f9770e18740 1 --2- 172.21.15.90:0/544183896 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55649ce90d80 0x55649d272890 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9758004c60 tx=0x7f9758004770 comp rx=0 tx=0).stop 2024-04-25T11:38:48.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.040+0000 7f9770e18740 1 -- 172.21.15.90:0/544183896 shutdown_connections 2024-04-25T11:38:48.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.040+0000 7f9770e18740 1 --2- 172.21.15.90:0/544183896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55649ceb14d0 0x55649d276500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.040+0000 7f9770e18740 1 --2- 172.21.15.90:0/544183896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55649cea92f0 0x55649d275fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.040+0000 7f9770e18740 1 --2- 172.21.15.90:0/544183896 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55649ce90d80 0x55649d272890 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.040+0000 7f9770e18740 1 -- 172.21.15.90:0/544183896 >> 172.21.15.90:0/544183896 conn(0x55649cec6300 msgr2=0x55649d27ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.040+0000 7f9770e18740 1 -- 172.21.15.90:0/544183896 shutdown_connections 2024-04-25T11:38:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.040+0000 7f9770e18740 1 -- 172.21.15.90:0/544183896 wait complete. 2024-04-25T11:38:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.040+0000 7f9770e18740 1 -- start start 2024-04-25T11:38:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.041+0000 7f9770e18740 1 Processor -- start 2024-04-25T11:38:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.041+0000 7f9770e18740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55649ce90d80 0x55649d286200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.041+0000 7f9770e18740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55649cea92f0 0x55649d2815f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.041+0000 7f9770e18740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55649ceb14d0 0x55649d281b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.041+0000 7f9770e18740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f97641aeea0 con 0x55649cea92f0 2024-04-25T11:38:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.041+0000 7f9770e18740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f97641ae7d0 con 0x55649ce90d80 2024-04-25T11:38:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.041+0000 7f9770e18740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f97641aba10 con 0x55649ceb14d0 2024-04-25T11:38:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.041+0000 7f9770e18740 10 client.0 _ll_register_callbacks cb 0x7f9758009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:38:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.041+0000 7f9762ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55649ce90d80 0x55649d286200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.041+0000 7f9762ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55649ce90d80 0x55649d286200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:57784/0 (socket says 172.21.15.90:57784) 2024-04-25T11:38:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.041+0000 7f9762ffd640 1 -- 172.21.15.90:0/2407771701 learned_addr learned my addr 172.21.15.90:0/2407771701 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:38:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.041+0000 7f9762ffd640 1 --2- 172.21.15.90:0/2407771701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55649ceb14d0 0x55649d281b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.041+0000 7f97637fe640 1 --2- 172.21.15.90:0/2407771701 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55649cea92f0 0x55649d2815f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.042+0000 7f9762ffd640 1 -- 172.21.15.90:0/2407771701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55649ceb14d0 msgr2=0x55649d281b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.042+0000 7f9762ffd640 1 --2- 172.21.15.90:0/2407771701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55649ceb14d0 0x55649d281b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.042+0000 7f9762ffd640 1 -- 172.21.15.90:0/2407771701 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55649cea92f0 msgr2=0x55649d2815f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.042+0000 7f9762ffd640 1 --2- 172.21.15.90:0/2407771701 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55649cea92f0 0x55649d2815f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.042+0000 7f9762ffd640 1 -- 172.21.15.90:0/2407771701 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55649d282270 con 0x55649ce90d80 2024-04-25T11:38:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.042+0000 7f9762ffd640 1 --2- 172.21.15.90:0/2407771701 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55649ce90d80 0x55649d286200 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9758004900 tx=0x7f9758005950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=377832d38ac1a7d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.042+0000 7f9742ffd640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:38:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.042+0000 7f9742ffd640 1 -- 172.21.15.90:0/2407771701 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9758035070 con 0x55649ce90d80 2024-04-25T11:38:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.042+0000 7f9742ffd640 1 -- 172.21.15.90:0/2407771701 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f9758031a30 con 0x55649ce90d80 2024-04-25T11:38:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.042+0000 7f9770e18740 1 -- 172.21.15.90:0/2407771701 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9758004e50 con 0x55649ce90d80 2024-04-25T11:38:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.042+0000 7f9742ffd640 1 -- 172.21.15.90:0/2407771701 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9758042040 con 0x55649ce90d80 2024-04-25T11:38:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.042+0000 7f9770e18740 10 client.25961 fetch_fsmap learned FSMap version 27 2024-04-25T11:38:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.042+0000 7f9770e18740 1 -- 172.21.15.90:0/2407771701 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55649d2f6960 con 0x55649ce90d80 2024-04-25T11:38:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.043+0000 7f9742ffd640 1 -- 172.21.15.90:0/2407771701 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9758004e50 con 0x55649ce90d80 2024-04-25T11:38:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.043+0000 7f9742ffd640 1 -- 172.21.15.90:0/2407771701 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f975804a040 con 0x55649ce90d80 2024-04-25T11:38:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.043+0000 7f9770e18740 10 client.25961 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:38:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.043+0000 7f9770e18740 10 client.25961 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:38:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.043+0000 7f9770e18740 20 client.25961 populate_metadata read hostname 'smithi090' 2024-04-25T11:38:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.043+0000 7f9770e18740 1 --2- 172.21.15.90:0/2407771701 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55649ced21b0 0x55649d2f9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.043+0000 7f9770e18740 4 client.25961 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:38:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.043+0000 7f9770e18740 1 -- 172.21.15.90:0/2407771701 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55649d1dfee0 con 0x55649ced21b0 2024-04-25T11:38:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.043+0000 7f97637fe640 1 --2- 172.21.15.90:0/2407771701 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55649ced21b0 0x55649d2f9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.044+0000 7f9742ffd640 10 client.25961 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:48.044+0000 7f97637fe640 1 --2- 172.21.15.90:0/2407771701 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55649ced21b0 0x55649d2f9060 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e953b878499c7c28 server_cookie=491493e4a7a1ff1c in_seq=0 out_seq=0 2024-04-25T11:38:48.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:47 smithi138 ceph-mon[32328]: pgmap v114: 97 pgs: 97 active+clean; 30 GiB data, 91 GiB used, 982 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 650 MiB/s wr, 798 op/s 2024-04-25T11:38:48.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:47 smithi156 ceph-mon[33045]: pgmap v114: 97 pgs: 97 active+clean; 30 GiB data, 91 GiB used, 982 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 650 MiB/s wr, 798 op/s 2024-04-25T11:38:48.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:47 smithi090 ceph-mon[29575]: pgmap v114: 97 pgs: 97 active+clean; 30 GiB data, 91 GiB used, 982 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 650 MiB/s wr, 798 op/s 2024-04-25T11:38:50.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:49 smithi138 ceph-mon[32328]: pgmap v115: 97 pgs: 97 active+clean; 30 GiB data, 91 GiB used, 982 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 471 MiB/s wr, 560 op/s 2024-04-25T11:38:50.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:49 smithi156 ceph-mon[33045]: pgmap v115: 97 pgs: 97 active+clean; 30 GiB data, 91 GiB used, 982 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 471 MiB/s wr, 560 op/s 2024-04-25T11:38:50.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:49 smithi090 ceph-mon[29575]: pgmap v115: 97 pgs: 97 active+clean; 30 GiB data, 91 GiB used, 982 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 471 MiB/s wr, 560 op/s 2024-04-25T11:38:51.824 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.836+0000 7f9742ffd640 1 -- 172.21.15.90:0/2407771701 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55649d1dfee0 con 0x55649ced21b0 2024-04-25T11:38:51.824 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.836+0000 7f9742ffd640 10 client.25961 handle_command_reply: tid=0 2024-04-25T11:38:51.824 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.836+0000 7f9770e18740 1 client.25961 shutdown 2024-04-25T11:38:51.824 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.837+0000 7f9770e18740 1 -- 172.21.15.90:0/2407771701 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55649ce90d80 msgr2=0x55649d286200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:51.824 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.837+0000 7f9770e18740 1 --2- 172.21.15.90:0/2407771701 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55649ce90d80 0x55649d286200 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9758004900 tx=0x7f9758005950 comp rx=0 tx=0).stop 2024-04-25T11:38:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.837+0000 7f9770e18740 1 -- 172.21.15.90:0/2407771701 shutdown_connections 2024-04-25T11:38:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.837+0000 7f9770e18740 1 --2- 172.21.15.90:0/2407771701 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55649ced21b0 0x55649d2f9060 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.837+0000 7f9770e18740 1 --2- 172.21.15.90:0/2407771701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55649ceb14d0 0x55649d281b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.837+0000 7f9770e18740 1 --2- 172.21.15.90:0/2407771701 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55649cea92f0 0x55649d2815f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.837+0000 7f9770e18740 1 --2- 172.21.15.90:0/2407771701 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55649ce90d80 0x55649d286200 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.837+0000 7f9770e18740 1 -- 172.21.15.90:0/2407771701 >> 172.21.15.90:0/2407771701 conn(0x55649cec6300 msgr2=0x55649d273730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.837+0000 7f9742ffd640 0 client.25961 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:38:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.838+0000 7f9770e18740 1 -- 172.21.15.90:0/2407771701 shutdown_connections 2024-04-25T11:38:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.838+0000 7f9770e18740 1 -- 172.21.15.90:0/2407771701 wait complete. 2024-04-25T11:38:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.838+0000 7f9770e18740 20 client.25961 trim_cache size 0 max 16384 2024-04-25T11:38:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:38:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:38:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:38:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:38:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.839+0000 7f9769e92640 1 -- 172.21.15.90:0/219176077 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f974c0b40f0 msgr2=0x7f974c0b6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.839+0000 7f9769e92640 1 --2- 172.21.15.90:0/219176077 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f974c0b40f0 0x7f974c0b6600 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f975c0a4a70 tx=0x7f975c0d5f50 comp rx=0 tx=0).stop 2024-04-25T11:38:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.840+0000 7f9769e92640 1 -- 172.21.15.90:0/219176077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97641074a0 msgr2=0x7f976419cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.840+0000 7f9769e92640 1 --2- 172.21.15.90:0/219176077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97641074a0 0x7f976419cdb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f975c009fd0 tx=0x7f975c00ef10 comp rx=0 tx=0).stop 2024-04-25T11:38:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.840+0000 7f9769e92640 1 -- 172.21.15.90:0/219176077 shutdown_connections 2024-04-25T11:38:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.840+0000 7f9769e92640 1 --2- 172.21.15.90:0/219176077 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f974c0b40f0 0x7f974c0b6600 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.840+0000 7f9769e92640 1 --2- 172.21.15.90:0/219176077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97641074a0 0x7f976419cdb0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.840+0000 7f9769e92640 1 --2- 172.21.15.90:0/219176077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97641069e0 0x7f9764198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.840+0000 7f9769e92640 1 --2- 172.21.15.90:0/219176077 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9764102450 0x7f9764198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.840+0000 7f9769e92640 1 -- 172.21.15.90:0/219176077 >> 172.21.15.90:0/219176077 conn(0x7f97640778b0 msgr2=0x7f9764100910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.842+0000 7f9769e92640 1 -- 172.21.15.90:0/219176077 shutdown_connections 2024-04-25T11:38:51.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:38:51.842+0000 7f9769e92640 1 -- 172.21.15.90:0/219176077 wait complete. 2024-04-25T11:38:52.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:51 smithi138 ceph-mon[32328]: pgmap v116: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 2.7 MiB/s rd, 706 MiB/s wr, 1.17k op/s 2024-04-25T11:38:52.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:51 smithi156 ceph-mon[33045]: pgmap v116: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 2.7 MiB/s rd, 706 MiB/s wr, 1.17k op/s 2024-04-25T11:38:52.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:51 smithi090 ceph-mon[29575]: pgmap v116: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 2.7 MiB/s rd, 706 MiB/s wr, 1.17k op/s 2024-04-25T11:38:54.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:53 smithi138 ceph-mon[32328]: pgmap v117: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 499 MiB/s wr, 869 op/s 2024-04-25T11:38:54.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:53 smithi156 ceph-mon[33045]: pgmap v117: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 499 MiB/s wr, 869 op/s 2024-04-25T11:38:54.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:53 smithi090 ceph-mon[29575]: pgmap v117: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 499 MiB/s wr, 869 op/s 2024-04-25T11:38:56.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:55 smithi138 ceph-mon[32328]: pgmap v118: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 2.2 MiB/s rd, 558 MiB/s wr, 914 op/s 2024-04-25T11:38:56.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:55 smithi156 ceph-mon[33045]: pgmap v118: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 2.2 MiB/s rd, 558 MiB/s wr, 914 op/s 2024-04-25T11:38:56.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:55 smithi090 ceph-mon[29575]: pgmap v118: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 2.2 MiB/s rd, 558 MiB/s wr, 914 op/s 2024-04-25T11:38:58.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:57 smithi138 ceph-mon[32328]: pgmap v119: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 611 MiB/s wr, 918 op/s 2024-04-25T11:38:58.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:57 smithi156 ceph-mon[33045]: pgmap v119: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 611 MiB/s wr, 918 op/s 2024-04-25T11:38:58.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:57 smithi090 ceph-mon[29575]: pgmap v119: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 611 MiB/s wr, 918 op/s 2024-04-25T11:39:00.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:38:59 smithi156 ceph-mon[33045]: pgmap v120: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 434 MiB/s wr, 746 op/s 2024-04-25T11:39:00.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:38:59 smithi090 ceph-mon[29575]: pgmap v120: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 434 MiB/s wr, 746 op/s 2024-04-25T11:39:00.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:38:59 smithi138 ceph-mon[32328]: pgmap v120: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 434 MiB/s wr, 746 op/s 2024-04-25T11:39:02.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:01 smithi090 ceph-mon[29575]: pgmap v121: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 622 MiB/s wr, 841 op/s 2024-04-25T11:39:02.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:01 smithi138 ceph-mon[32328]: pgmap v121: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 622 MiB/s wr, 841 op/s 2024-04-25T11:39:02.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:01 smithi156 ceph-mon[33045]: pgmap v121: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 622 MiB/s wr, 841 op/s 2024-04-25T11:39:04.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:03 smithi090 ceph-mon[29575]: pgmap v122: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 1.3 MiB/s rd, 387 MiB/s wr, 234 op/s 2024-04-25T11:39:04.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:03 smithi138 ceph-mon[32328]: pgmap v122: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 1.3 MiB/s rd, 387 MiB/s wr, 234 op/s 2024-04-25T11:39:04.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:03 smithi156 ceph-mon[33045]: pgmap v122: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 1.3 MiB/s rd, 387 MiB/s wr, 234 op/s 2024-04-25T11:39:04.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.009+0000 7f8102c15640 1 -- 172.21.15.90:0/77028994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80fc10e100 msgr2=0x7f80fc1105d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:04.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.009+0000 7f8102c15640 1 --2- 172.21.15.90:0/77028994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80fc10e100 0x7f80fc1105d0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f80f4009ca0 tx=0x7f80f402a950 comp rx=0 tx=0).stop 2024-04-25T11:39:04.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.009+0000 7f8102c15640 1 -- 172.21.15.90:0/77028994 shutdown_connections 2024-04-25T11:39:04.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.009+0000 7f8102c15640 1 --2- 172.21.15.90:0/77028994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80fc10e100 0x7f80fc1105d0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:04.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.009+0000 7f8102c15640 1 --2- 172.21.15.90:0/77028994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80fc10b750 0x7f80fc10db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:04.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.009+0000 7f8102c15640 1 --2- 172.21.15.90:0/77028994 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f80fc104cf0 0x7f80fc105100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:04.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.009+0000 7f8102c15640 1 -- 172.21.15.90:0/77028994 >> 172.21.15.90:0/77028994 conn(0x7f80fc0781b0 msgr2=0x7f80fc0785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:04.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.009+0000 7f8102c15640 1 -- 172.21.15.90:0/77028994 shutdown_connections 2024-04-25T11:39:04.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.009+0000 7f8102c15640 1 -- 172.21.15.90:0/77028994 wait complete. 2024-04-25T11:39:04.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.010+0000 7f8102c15640 1 Processor -- start 2024-04-25T11:39:04.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.010+0000 7f8102c15640 1 -- start start 2024-04-25T11:39:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.010+0000 7f8102c15640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80fc104cf0 0x7f80fc1025b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.010+0000 7f8102c15640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80fc10b750 0x7f80fc102af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.010+0000 7f8102c15640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f80fc10e100 0x7f80fc103030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.010+0000 7f8102c15640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f80fc113d80 con 0x7f80fc10b750 2024-04-25T11:39:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.010+0000 7f8102c15640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f80fc113c00 con 0x7f80fc10e100 2024-04-25T11:39:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.010+0000 7f8102c15640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f80fc113f00 con 0x7f80fc104cf0 2024-04-25T11:39:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.010+0000 7f80fbfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80fc10b750 0x7f80fc102af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.010+0000 7f80fbfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80fc10b750 0x7f80fc102af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42032/0 (socket says 172.21.15.90:42032) 2024-04-25T11:39:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.010+0000 7f80fbfff640 1 -- 172.21.15.90:0/62881302 learned_addr learned my addr 172.21.15.90:0/62881302 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.010+0000 7f81011b2640 1 --2- 172.21.15.90:0/62881302 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f80fc10e100 0x7f80fc103030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.010+0000 7f80fbfff640 1 -- 172.21.15.90:0/62881302 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80fc104cf0 msgr2=0x7f80fc1025b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:39:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.010+0000 7f80fbfff640 1 --2- 172.21.15.90:0/62881302 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80fc104cf0 0x7f80fc1025b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.010+0000 7f80fbfff640 1 -- 172.21.15.90:0/62881302 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f80fc10e100 msgr2=0x7f80fc103030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.010+0000 7f80fbfff640 1 --2- 172.21.15.90:0/62881302 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f80fc10e100 0x7f80fc103030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.010+0000 7f80fbfff640 1 -- 172.21.15.90:0/62881302 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80fc1a8ac0 con 0x7f80fc10b750 2024-04-25T11:39:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.011+0000 7f80fbfff640 1 --2- 172.21.15.90:0/62881302 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80fc10b750 0x7f80fc102af0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f80e800b4a0 tx=0x7f80e800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.011+0000 7f80f9ffb640 1 -- 172.21.15.90:0/62881302 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e8014070 con 0x7f80fc10b750 2024-04-25T11:39:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.011+0000 7f8102c15640 1 -- 172.21.15.90:0/62881302 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80fc1a8db0 con 0x7f80fc10b750 2024-04-25T11:39:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.011+0000 7f80f9ffb640 1 -- 172.21.15.90:0/62881302 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80e8010ac0 con 0x7f80fc10b750 2024-04-25T11:39:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.011+0000 7f80f9ffb640 1 -- 172.21.15.90:0/62881302 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e8017c30 con 0x7f80fc10b750 2024-04-25T11:39:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.011+0000 7f8102c15640 1 -- 172.21.15.90:0/62881302 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80fc1a9220 con 0x7f80fc10b750 2024-04-25T11:39:05.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.014+0000 7f80f9ffb640 1 -- 172.21.15.90:0/62881302 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f80e8017070 con 0x7f80fc10b750 2024-04-25T11:39:05.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.015+0000 7f80f9ffb640 1 --2- 172.21.15.90:0/62881302 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f80ec0afc50 0x7f80ec0b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:05.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.015+0000 7f8109ba7740 1 Processor -- start 2024-04-25T11:39:05.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.015+0000 7f8109ba7740 1 -- start start 2024-04-25T11:39:05.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.015+0000 7f81009b1640 1 --2- 172.21.15.90:0/62881302 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f80ec0afc50 0x7f80ec0b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:05.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.015+0000 7f80f9ffb640 1 -- 172.21.15.90:0/62881302 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f80e80de6a0 con 0x7f80fc10b750 2024-04-25T11:39:05.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.015+0000 7f8109ba7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55beb75a91b0 0x55beb7948af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:05.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.015+0000 7f8109ba7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55beb75802f0 0x55beb7949030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:05.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.015+0000 7f8109ba7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb75884d0 0x55beb7949570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:05.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.015+0000 7f8109ba7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55beb78af460 con 0x55beb75884d0 2024-04-25T11:39:05.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.015+0000 7f8109ba7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55beb7869590 con 0x55beb75a91b0 2024-04-25T11:39:05.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.015+0000 7f8109ba7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55beb78ab420 con 0x55beb75802f0 2024-04-25T11:39:05.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.015+0000 7f81011b2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb75884d0 0x55beb7949570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:05.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.016+0000 7f81009b1640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55beb75a91b0 0x55beb7948af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:05.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.016+0000 7f81011b2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb75884d0 0x55beb7949570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42048/0 (socket says 172.21.15.90:42048) 2024-04-25T11:39:05.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.016+0000 7f81011b2640 1 -- 172.21.15.90:0/4206058315 learned_addr learned my addr 172.21.15.90:0/4206058315 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:05.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.016+0000 7f81009b1640 1 --2- 172.21.15.90:0/62881302 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f80ec0afc50 0x7f80ec0b2160 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f80f000b9e0 tx=0x7f80f000b370 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:05.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.016+0000 7f80fbfff640 1 --2- 172.21.15.90:0/4206058315 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55beb75802f0 0x55beb7949030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:05.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.016+0000 7f81011b2640 1 -- 172.21.15.90:0/4206058315 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55beb75802f0 msgr2=0x55beb7949030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:05.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.016+0000 7f81011b2640 1 --2- 172.21.15.90:0/4206058315 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55beb75802f0 0x55beb7949030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:05.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.016+0000 7f81011b2640 1 -- 172.21.15.90:0/4206058315 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55beb75a91b0 msgr2=0x55beb7948af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:05.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.016+0000 7f81011b2640 1 --2- 172.21.15.90:0/4206058315 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55beb75a91b0 0x55beb7948af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:05.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.016+0000 7f81011b2640 1 -- 172.21.15.90:0/4206058315 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55beb78760d0 con 0x55beb75884d0 2024-04-25T11:39:05.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.016+0000 7f81009b1640 1 --2- 172.21.15.90:0/4206058315 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55beb75a91b0 0x55beb7948af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:05.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.016+0000 7f81011b2640 1 --2- 172.21.15.90:0/4206058315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb75884d0 0x55beb7949570 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f80f4035000 tx=0x7f80f402fdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4ab2344ac87fbfe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:05.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.016+0000 7f80e6ffd640 1 -- 172.21.15.90:0/4206058315 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f4048070 con 0x55beb75884d0 2024-04-25T11:39:05.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.016+0000 7f80e6ffd640 1 -- 172.21.15.90:0/4206058315 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80f4003e20 con 0x55beb75884d0 2024-04-25T11:39:05.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.016+0000 7f80e6ffd640 1 -- 172.21.15.90:0/4206058315 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f4034040 con 0x55beb75884d0 2024-04-25T11:39:05.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.017+0000 7f8109ba7740 1 -- 172.21.15.90:0/4206058315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb75884d0 msgr2=0x55beb7949570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:05.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.017+0000 7f8109ba7740 1 --2- 172.21.15.90:0/4206058315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb75884d0 0x55beb7949570 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f80f4035000 tx=0x7f80f402fdf0 comp rx=0 tx=0).stop 2024-04-25T11:39:05.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.017+0000 7f8109ba7740 1 -- 172.21.15.90:0/4206058315 shutdown_connections 2024-04-25T11:39:05.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.017+0000 7f8109ba7740 1 --2- 172.21.15.90:0/4206058315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb75884d0 0x55beb7949570 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:05.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.017+0000 7f8109ba7740 1 --2- 172.21.15.90:0/4206058315 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55beb75802f0 0x55beb7949030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:05.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.017+0000 7f8109ba7740 1 --2- 172.21.15.90:0/4206058315 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55beb75a91b0 0x55beb7948af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:05.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.017+0000 7f8109ba7740 1 -- 172.21.15.90:0/4206058315 >> 172.21.15.90:0/4206058315 conn(0x55beb759d300 msgr2=0x55beb79452a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:05.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.017+0000 7f8109ba7740 1 -- 172.21.15.90:0/4206058315 shutdown_connections 2024-04-25T11:39:05.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.017+0000 7f8109ba7740 1 -- 172.21.15.90:0/4206058315 wait complete. 2024-04-25T11:39:05.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.018+0000 7f8109ba7740 1 -- start start 2024-04-25T11:39:05.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.018+0000 7f8109ba7740 1 Processor -- start 2024-04-25T11:39:05.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.018+0000 7f8109ba7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb75802f0 0x55beb7954b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:05.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.018+0000 7f8109ba7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55beb75884d0 0x55beb7955040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:05.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.018+0000 7f8109ba7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55beb75a91b0 0x55beb7952ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:05.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.018+0000 7f8109ba7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55beb7955bc0 con 0x55beb75802f0 2024-04-25T11:39:05.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.018+0000 7f8109ba7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55beb7836b60 con 0x55beb75a91b0 2024-04-25T11:39:05.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.018+0000 7f8109ba7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55beb7955d40 con 0x55beb75884d0 2024-04-25T11:39:05.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.018+0000 7f8109ba7740 10 client.0 _ll_register_callbacks cb 0x55beb785b2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:05.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.018+0000 7f81009b1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb75802f0 0x55beb7954b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:05.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.018+0000 7f81009b1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb75802f0 0x55beb7954b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42064/0 (socket says 172.21.15.90:42064) 2024-04-25T11:39:05.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.018+0000 7f81009b1640 1 -- 172.21.15.90:0/3695004035 learned_addr learned my addr 172.21.15.90:0/3695004035 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:05.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.018+0000 7f81011b2640 1 --2- 172.21.15.90:0/3695004035 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55beb75a91b0 0x55beb7952ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:05.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.018+0000 7f80fbfff640 1 --2- 172.21.15.90:0/3695004035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55beb75884d0 0x55beb7955040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:05.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.019+0000 7f81009b1640 1 -- 172.21.15.90:0/3695004035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55beb75884d0 msgr2=0x55beb7955040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:05.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.019+0000 7f81009b1640 1 --2- 172.21.15.90:0/3695004035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55beb75884d0 0x55beb7955040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:05.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.019+0000 7f81009b1640 1 -- 172.21.15.90:0/3695004035 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55beb75a91b0 msgr2=0x55beb7952ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:05.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.019+0000 7f81009b1640 1 --2- 172.21.15.90:0/3695004035 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55beb75a91b0 0x55beb7952ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:05.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.019+0000 7f81009b1640 1 -- 172.21.15.90:0/3695004035 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55beb75c1210 con 0x55beb75802f0 2024-04-25T11:39:05.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.019+0000 7f81011b2640 1 --2- 172.21.15.90:0/3695004035 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55beb75a91b0 0x55beb7952ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:39:05.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.019+0000 7f81009b1640 1 --2- 172.21.15.90:0/3695004035 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb75802f0 0x55beb7954b00 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f80f0010460 tx=0x7f80f0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2ac76cfc6abcb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:05.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.019+0000 7f80dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:05.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.019+0000 7f80dbfff640 1 -- 172.21.15.90:0/3695004035 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f0026600 con 0x55beb75802f0 2024-04-25T11:39:05.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.019+0000 7f80dbfff640 1 -- 172.21.15.90:0/3695004035 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f80f00267a0 con 0x55beb75802f0 2024-04-25T11:39:05.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.019+0000 7f8109ba7740 1 -- 172.21.15.90:0/3695004035 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80f4003e20 con 0x55beb75802f0 2024-04-25T11:39:05.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.019+0000 7f80dbfff640 1 -- 172.21.15.90:0/3695004035 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f80f00269a0 con 0x55beb75802f0 2024-04-25T11:39:05.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.019+0000 7f8109ba7740 10 client.16746 fetch_fsmap learned FSMap version 27 2024-04-25T11:39:05.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.019+0000 7f8109ba7740 1 -- 172.21.15.90:0/3695004035 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55beb768d620 con 0x55beb75802f0 2024-04-25T11:39:05.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.019+0000 7f80dbfff640 1 -- 172.21.15.90:0/3695004035 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80f4003e20 con 0x55beb75802f0 2024-04-25T11:39:05.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.020+0000 7f80dbfff640 1 -- 172.21.15.90:0/3695004035 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f80f001e5b0 con 0x55beb75802f0 2024-04-25T11:39:05.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.020+0000 7f8109ba7740 10 client.16746 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:39:05.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.020+0000 7f8109ba7740 10 client.16746 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:39:05.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.020+0000 7f8109ba7740 20 client.16746 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:05.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.020+0000 7f8109ba7740 1 --2- 172.21.15.90:0/3695004035 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55beb7567d80 0x55beb7965800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:05.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.020+0000 7f8109ba7740 4 client.16746 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:39:05.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.020+0000 7f8109ba7740 1 -- 172.21.15.90:0/3695004035 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55beb78bd070 con 0x55beb7567d80 2024-04-25T11:39:05.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.020+0000 7f80fbfff640 1 --2- 172.21.15.90:0/3695004035 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55beb7567d80 0x55beb7965800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:05.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.020+0000 7f80dbfff640 10 client.16746 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:05.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.020+0000 7f80fbfff640 1 --2- 172.21.15.90:0/3695004035 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55beb7567d80 0x55beb7965800 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bde638f5b3b52917 server_cookie=26ce0969a6606019 in_seq=0 out_seq=0 2024-04-25T11:39:05.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.021+0000 7f80dbfff640 1 -- 172.21.15.90:0/3695004035 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55beb78bd070 con 0x55beb7567d80 2024-04-25T11:39:05.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.021+0000 7f80dbfff640 10 client.16746 handle_command_reply: tid=0 2024-04-25T11:39:05.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.021+0000 7f8109ba7740 1 client.16746 shutdown 2024-04-25T11:39:05.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.021+0000 7f8109ba7740 1 -- 172.21.15.90:0/3695004035 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb75802f0 msgr2=0x55beb7954b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:05.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.021+0000 7f8109ba7740 1 --2- 172.21.15.90:0/3695004035 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb75802f0 0x55beb7954b00 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f80f0010460 tx=0x7f80f0004730 comp rx=0 tx=0).stop 2024-04-25T11:39:05.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.021+0000 7f8109ba7740 1 -- 172.21.15.90:0/3695004035 shutdown_connections 2024-04-25T11:39:05.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.021+0000 7f8109ba7740 1 --2- 172.21.15.90:0/3695004035 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55beb7567d80 0x55beb7965800 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:05.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.021+0000 7f8109ba7740 1 --2- 172.21.15.90:0/3695004035 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55beb75a91b0 0x55beb7952ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:05.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.021+0000 7f80dbfff640 0 client.16746 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:05.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.021+0000 7f8109ba7740 1 --2- 172.21.15.90:0/3695004035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55beb75884d0 0x55beb7955040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:05.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.021+0000 7f8109ba7740 1 --2- 172.21.15.90:0/3695004035 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb75802f0 0x55beb7954b00 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:05.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.021+0000 7f8109ba7740 1 -- 172.21.15.90:0/3695004035 >> 172.21.15.90:0/3695004035 conn(0x55beb759d300 msgr2=0x55beb794a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:05.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.022+0000 7f8109ba7740 1 -- 172.21.15.90:0/3695004035 shutdown_connections 2024-04-25T11:39:05.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.022+0000 7f8109ba7740 1 -- 172.21.15.90:0/3695004035 wait complete. 2024-04-25T11:39:05.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.022+0000 7f8109ba7740 20 client.16746 trim_cache size 0 max 16384 2024-04-25T11:39:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.037+0000 7f8109ba7740 1 Processor -- start 2024-04-25T11:39:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.037+0000 7f8109ba7740 1 -- start start 2024-04-25T11:39:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.037+0000 7f8109ba7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55beb7567d80 0x55beb794aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.037+0000 7f8109ba7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb75802f0 0x55beb794c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.037+0000 7f8109ba7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55beb75884d0 0x55beb794cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.037+0000 7f8109ba7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f80fc1b1d60 con 0x55beb75802f0 2024-04-25T11:39:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.037+0000 7f8109ba7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f80fc1b38d0 con 0x55beb75884d0 2024-04-25T11:39:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.037+0000 7f8109ba7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f80fc1b2430 con 0x55beb7567d80 2024-04-25T11:39:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.037+0000 7f80fbfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb75802f0 0x55beb794c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.037+0000 7f80fbfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb75802f0 0x55beb794c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42076/0 (socket says 172.21.15.90:42076) 2024-04-25T11:39:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.037+0000 7f80fbfff640 1 -- 172.21.15.90:0/239029682 learned_addr learned my addr 172.21.15.90:0/239029682 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.037+0000 7f81011b2640 1 --2- 172.21.15.90:0/239029682 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55beb75884d0 0x55beb794cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.037+0000 7f80fbfff640 1 -- 172.21.15.90:0/239029682 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55beb7567d80 msgr2=0x55beb794aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:05.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.037+0000 7f81009b1640 1 --2- 172.21.15.90:0/239029682 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55beb7567d80 0x55beb794aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:05.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.037+0000 7f80fbfff640 1 --2- 172.21.15.90:0/239029682 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55beb7567d80 0x55beb794aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:05.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.037+0000 7f80fbfff640 1 -- 172.21.15.90:0/239029682 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55beb75884d0 msgr2=0x55beb794cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:05.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.037+0000 7f80fbfff640 1 --2- 172.21.15.90:0/239029682 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55beb75884d0 0x55beb794cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:05.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.037+0000 7f80fbfff640 1 -- 172.21.15.90:0/239029682 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55beb79522f0 con 0x55beb75802f0 2024-04-25T11:39:05.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.037+0000 7f81011b2640 1 --2- 172.21.15.90:0/239029682 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55beb75884d0 0x55beb794cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:39:05.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.038+0000 7f80fbfff640 1 --2- 172.21.15.90:0/239029682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb75802f0 0x55beb794c7a0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f80e80a61d0 tx=0x7f80e808f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6f9cd02382159b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:05.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.038+0000 7f80e67fc640 1 -- 172.21.15.90:0/239029682 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e801e040 con 0x55beb75802f0 2024-04-25T11:39:05.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.038+0000 7f80e67fc640 1 -- 172.21.15.90:0/239029682 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80e80aaca0 con 0x55beb75802f0 2024-04-25T11:39:05.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.038+0000 7f80e67fc640 1 -- 172.21.15.90:0/239029682 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e80a2b80 con 0x55beb75802f0 2024-04-25T11:39:05.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.038+0000 7f8109ba7740 1 -- 172.21.15.90:0/239029682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb75802f0 msgr2=0x55beb794c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:05.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.038+0000 7f8109ba7740 1 --2- 172.21.15.90:0/239029682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb75802f0 0x55beb794c7a0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f80e80a61d0 tx=0x7f80e808f040 comp rx=0 tx=0).stop 2024-04-25T11:39:05.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.038+0000 7f8109ba7740 1 -- 172.21.15.90:0/239029682 shutdown_connections 2024-04-25T11:39:05.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.038+0000 7f8109ba7740 1 --2- 172.21.15.90:0/239029682 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55beb75884d0 0x55beb794cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:05.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.039+0000 7f8109ba7740 1 --2- 172.21.15.90:0/239029682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb75802f0 0x55beb794c7a0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:05.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.039+0000 7f8109ba7740 1 --2- 172.21.15.90:0/239029682 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55beb7567d80 0x55beb794aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:05.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.039+0000 7f8109ba7740 1 -- 172.21.15.90:0/239029682 >> 172.21.15.90:0/239029682 conn(0x55beb759d300 msgr2=0x55beb794a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:05.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.039+0000 7f8109ba7740 1 -- 172.21.15.90:0/239029682 shutdown_connections 2024-04-25T11:39:05.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.039+0000 7f8109ba7740 1 -- 172.21.15.90:0/239029682 wait complete. 2024-04-25T11:39:05.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.039+0000 7f8109ba7740 1 -- start start 2024-04-25T11:39:05.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.039+0000 7f8109ba7740 1 Processor -- start 2024-04-25T11:39:05.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.039+0000 7f8109ba7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb7567d80 0x55beb795af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:05.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.039+0000 7f8109ba7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55beb75802f0 0x55beb795b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:05.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.039+0000 7f8109ba7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55beb75884d0 0x55beb7958340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:05.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.039+0000 7f8109ba7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f80fc1b31d0 con 0x55beb7567d80 2024-04-25T11:39:05.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.039+0000 7f8109ba7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f80fc1b2b00 con 0x55beb75884d0 2024-04-25T11:39:05.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.039+0000 7f8109ba7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f80fc1afd40 con 0x55beb75802f0 2024-04-25T11:39:05.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.039+0000 7f8109ba7740 10 client.0 _ll_register_callbacks cb 0x7f80f000da40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:05.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.039+0000 7f81009b1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb7567d80 0x55beb795af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:05.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.040+0000 7f81009b1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb7567d80 0x55beb795af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42088/0 (socket says 172.21.15.90:42088) 2024-04-25T11:39:05.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.040+0000 7f81009b1640 1 -- 172.21.15.90:0/2846990190 learned_addr learned my addr 172.21.15.90:0/2846990190 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:05.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.040+0000 7f81011b2640 1 --2- 172.21.15.90:0/2846990190 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55beb75884d0 0x55beb7958340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:05.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.040+0000 7f80fbfff640 1 --2- 172.21.15.90:0/2846990190 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55beb75802f0 0x55beb795b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:05.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.040+0000 7f81009b1640 1 -- 172.21.15.90:0/2846990190 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55beb75802f0 msgr2=0x55beb795b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:05.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.040+0000 7f81009b1640 1 --2- 172.21.15.90:0/2846990190 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55beb75802f0 0x55beb795b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:05.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.040+0000 7f81009b1640 1 -- 172.21.15.90:0/2846990190 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55beb75884d0 msgr2=0x55beb7958340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:05.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.040+0000 7f81009b1640 1 --2- 172.21.15.90:0/2846990190 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55beb75884d0 0x55beb7958340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:05.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.040+0000 7f81009b1640 1 -- 172.21.15.90:0/2846990190 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55beb7958a30 con 0x55beb7567d80 2024-04-25T11:39:05.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.040+0000 7f80fbfff640 1 --2- 172.21.15.90:0/2846990190 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55beb75802f0 0x55beb795b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:39:05.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.040+0000 7f81009b1640 1 --2- 172.21.15.90:0/2846990190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb7567d80 0x55beb795af40 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f80f000e010 tx=0x7f80f0026df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68ea2a61276196e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:05.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.040+0000 7f80dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:05.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.040+0000 7f80dbfff640 1 -- 172.21.15.90:0/2846990190 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f0005e50 con 0x55beb7567d80 2024-04-25T11:39:05.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.040+0000 7f80dbfff640 1 -- 172.21.15.90:0/2846990190 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f80f0026400 con 0x55beb7567d80 2024-04-25T11:39:05.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.040+0000 7f8109ba7740 1 -- 172.21.15.90:0/2846990190 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80e80aaca0 con 0x55beb7567d80 2024-04-25T11:39:05.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.040+0000 7f80dbfff640 1 -- 172.21.15.90:0/2846990190 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f80f0045050 con 0x55beb7567d80 2024-04-25T11:39:05.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.040+0000 7f8109ba7740 10 client.16752 fetch_fsmap learned FSMap version 27 2024-04-25T11:39:05.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.040+0000 7f8109ba7740 1 -- 172.21.15.90:0/2846990190 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55beb79cd4a0 con 0x55beb7567d80 2024-04-25T11:39:05.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.040+0000 7f80dbfff640 1 -- 172.21.15.90:0/2846990190 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80e80aaca0 con 0x55beb7567d80 2024-04-25T11:39:05.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.041+0000 7f80dbfff640 1 -- 172.21.15.90:0/2846990190 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f80f0037c90 con 0x55beb7567d80 2024-04-25T11:39:05.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.041+0000 7f8109ba7740 10 client.16752 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:39:05.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.041+0000 7f8109ba7740 10 client.16752 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:39:05.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.041+0000 7f8109ba7740 20 client.16752 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:05.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.041+0000 7f8109ba7740 1 --2- 172.21.15.90:0/2846990190 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55beb75a91b0 0x55beb79cfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:05.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.041+0000 7f8109ba7740 4 client.16752 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:39:05.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.041+0000 7f8109ba7740 1 -- 172.21.15.90:0/2846990190 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55beb78b6ee0 con 0x55beb75a91b0 2024-04-25T11:39:05.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.041+0000 7f80fbfff640 1 --2- 172.21.15.90:0/2846990190 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55beb75a91b0 0x55beb79cfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:05.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.041+0000 7f80dbfff640 10 client.16752 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:05.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:05.041+0000 7f80fbfff640 1 --2- 172.21.15.90:0/2846990190 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55beb75a91b0 0x55beb79cfcb0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bad5ec2f79b56111 server_cookie=1b6d3f4dd4f99455 in_seq=0 out_seq=0 2024-04-25T11:39:06.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:05 smithi138 ceph-mon[32328]: pgmap v123: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 1.3 MiB/s rd, 459 MiB/s wr, 361 op/s 2024-04-25T11:39:06.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:05 smithi156 ceph-mon[33045]: pgmap v123: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 1.3 MiB/s rd, 459 MiB/s wr, 361 op/s 2024-04-25T11:39:06.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:05 smithi090 ceph-mon[29575]: pgmap v123: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 1.3 MiB/s rd, 459 MiB/s wr, 361 op/s 2024-04-25T11:39:08.234 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:39:08.234 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:39:08.259 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:07 smithi090 ceph-mon[29575]: pgmap v124: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 539 MiB/s wr, 570 op/s 2024-04-25T11:39:08.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:07 smithi138 ceph-mon[32328]: pgmap v124: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 539 MiB/s wr, 570 op/s 2024-04-25T11:39:08.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:07 smithi156 ceph-mon[33045]: pgmap v124: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 539 MiB/s wr, 570 op/s 2024-04-25T11:39:08.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.383+0000 7f35fadce640 1 -- 172.21.15.90:0/2688871783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f410e550 msgr2=0x7f35f4110970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:08.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.383+0000 7f35fadce640 1 --2- 172.21.15.90:0/2688871783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f410e550 0x7f35f4110970 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f35ec009be0 tx=0x7f35ec02a6c0 comp rx=0 tx=0).stop 2024-04-25T11:39:08.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.383+0000 7f35fadce640 1 -- 172.21.15.90:0/2688871783 shutdown_connections 2024-04-25T11:39:08.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.383+0000 7f35fadce640 1 --2- 172.21.15.90:0/2688871783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f410e550 0x7f35f4110970 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:08.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.383+0000 7f35fadce640 1 --2- 172.21.15.90:0/2688871783 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f4102050 0x7f35f4102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:08.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.383+0000 7f35fadce640 1 --2- 172.21.15.90:0/2688871783 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35f4109ba0 0x7f35f4101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:08.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.383+0000 7f35fadce640 1 -- 172.21.15.90:0/2688871783 >> 172.21.15.90:0/2688871783 conn(0x7f35f4077880 msgr2=0x7f35f4077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:08.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.383+0000 7f35fadce640 1 -- 172.21.15.90:0/2688871783 shutdown_connections 2024-04-25T11:39:08.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.383+0000 7f35fadce640 1 -- 172.21.15.90:0/2688871783 wait complete. 2024-04-25T11:39:08.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.384+0000 7f35fadce640 1 Processor -- start 2024-04-25T11:39:08.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.384+0000 7f35fadce640 1 -- start start 2024-04-25T11:39:08.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.384+0000 7f35fadce640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f4102050 0x7f35f419d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:08.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.384+0000 7f35fadce640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35f4109ba0 0x7f35f419db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:08.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.384+0000 7f35fadce640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f410e550 0x7f35f41a8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:08.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.384+0000 7f35fadce640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f35f4113d10 con 0x7f35f410e550 2024-04-25T11:39:08.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.384+0000 7f35fadce640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f35f4113b90 con 0x7f35f4109ba0 2024-04-25T11:39:08.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.384+0000 7f35fadce640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f35f4113e90 con 0x7f35f4102050 2024-04-25T11:39:08.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.384+0000 7f35ebfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35f4109ba0 0x7f35f419db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:08.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.384+0000 7f35f936b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f410e550 0x7f35f41a8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:08.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.384+0000 7f35ebfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35f4109ba0 0x7f35f419db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:33846/0 (socket says 172.21.15.90:33846) 2024-04-25T11:39:08.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.384+0000 7f35f936b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f410e550 0x7f35f41a8670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42096/0 (socket says 172.21.15.90:42096) 2024-04-25T11:39:08.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.384+0000 7f35ebfff640 1 -- 172.21.15.90:0/2698895998 learned_addr learned my addr 172.21.15.90:0/2698895998 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:08.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.384+0000 7f35f8b6a640 1 --2- 172.21.15.90:0/2698895998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f4102050 0x7f35f419d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:08.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.384+0000 7f35ebfff640 1 -- 172.21.15.90:0/2698895998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f4102050 msgr2=0x7f35f419d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:08.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.384+0000 7f35ebfff640 1 --2- 172.21.15.90:0/2698895998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f4102050 0x7f35f419d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:08.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.384+0000 7f35ebfff640 1 -- 172.21.15.90:0/2698895998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f410e550 msgr2=0x7f35f41a8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:08.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.384+0000 7f35ebfff640 1 --2- 172.21.15.90:0/2698895998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f410e550 0x7f35f41a8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:08.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.384+0000 7f35ebfff640 1 -- 172.21.15.90:0/2698895998 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35f41a8bb0 con 0x7f35f4109ba0 2024-04-25T11:39:08.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.384+0000 7f35f8b6a640 1 --2- 172.21.15.90:0/2698895998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f4102050 0x7f35f419d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:08.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.385+0000 7f35ebfff640 1 --2- 172.21.15.90:0/2698895998 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35f4109ba0 0x7f35f419db30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f35e400e9f0 tx=0x7f35e400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:08.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.385+0000 7f35e9ffb640 1 -- 172.21.15.90:0/2698895998 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e400ccc0 con 0x7f35f4109ba0 2024-04-25T11:39:08.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.385+0000 7f35e9ffb640 1 -- 172.21.15.90:0/2698895998 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35e40054d0 con 0x7f35f4109ba0 2024-04-25T11:39:08.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.385+0000 7f35fadce640 1 -- 172.21.15.90:0/2698895998 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35f41a8d80 con 0x7f35f4109ba0 2024-04-25T11:39:08.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.385+0000 7f35e9ffb640 1 -- 172.21.15.90:0/2698895998 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e4010430 con 0x7f35f4109ba0 2024-04-25T11:39:08.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.385+0000 7f35fadce640 1 -- 172.21.15.90:0/2698895998 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35f41a9130 con 0x7f35f4109ba0 2024-04-25T11:39:08.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.388+0000 7f3601c83740 1 Processor -- start 2024-04-25T11:39:08.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.388+0000 7f3601c83740 1 -- start start 2024-04-25T11:39:08.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.388+0000 7f3601c83740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5648e9507390 0x5648e98a7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:08.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.388+0000 7f3601c83740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648e94de4d0 0x5648e98a7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:08.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.388+0000 7f3601c83740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648e94e66b0 0x5648e98b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:08.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.388+0000 7f3601c83740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5648e980dd80 con 0x5648e94e66b0 2024-04-25T11:39:08.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.388+0000 7f3601c83740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5648e97c78b0 con 0x5648e9507390 2024-04-25T11:39:08.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.388+0000 7f3601c83740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5648e9809d40 con 0x5648e94de4d0 2024-04-25T11:39:08.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.388+0000 7f35f936b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5648e9507390 0x5648e98a7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:08.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.388+0000 7f35f8b6a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648e94de4d0 0x5648e98a7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:08.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.388+0000 7f35f936b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5648e9507390 0x5648e98a7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:33848/0 (socket says 172.21.15.90:33848) 2024-04-25T11:39:08.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.388+0000 7f35f936b640 1 -- 172.21.15.90:0/3209071725 learned_addr learned my addr 172.21.15.90:0/3209071725 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:08.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.389+0000 7f35f8b6a640 1 -- 172.21.15.90:0/3209071725 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5648e9507390 msgr2=0x5648e98a7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:08.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.389+0000 7f35f8b6a640 1 --2- 172.21.15.90:0/3209071725 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5648e9507390 0x5648e98a7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:08.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.389+0000 7f35f8b6a640 1 -- 172.21.15.90:0/3209071725 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648e94e66b0 msgr2=0x5648e98b48f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:39:08.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.389+0000 7f35f8b6a640 1 --2- 172.21.15.90:0/3209071725 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648e94e66b0 0x5648e98b48f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:08.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.389+0000 7f35f8b6a640 1 -- 172.21.15.90:0/3209071725 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648e9798a40 con 0x5648e94de4d0 2024-04-25T11:39:08.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.389+0000 7f35e9ffb640 1 -- 172.21.15.90:0/2698895998 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f35e40046c0 con 0x7f35f4109ba0 2024-04-25T11:39:08.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.389+0000 7f35f8b6a640 1 --2- 172.21.15.90:0/3209071725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648e94de4d0 0x5648e98a7fa0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f35f4069bf0 tx=0x7f35f00118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5df0397d14dd7f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:08.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.389+0000 7f35daffd640 1 -- 172.21.15.90:0/3209071725 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f0005020 con 0x5648e94de4d0 2024-04-25T11:39:08.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.389+0000 7f35daffd640 1 -- 172.21.15.90:0/3209071725 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35f00047a0 con 0x5648e94de4d0 2024-04-25T11:39:08.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.389+0000 7f35daffd640 1 -- 172.21.15.90:0/3209071725 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f000f830 con 0x5648e94de4d0 2024-04-25T11:39:08.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.390+0000 7f35e9ffb640 1 --2- 172.21.15.90:0/2698895998 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f35dc0afc60 0x7f35dc0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:08.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.390+0000 7f35f936b640 1 --2- 172.21.15.90:0/2698895998 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f35dc0afc60 0x7f35dc0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:08.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.390+0000 7f35e9ffb640 1 -- 172.21.15.90:0/2698895998 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f35e40df410 con 0x7f35f4109ba0 2024-04-25T11:39:08.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.390+0000 7f3601c83740 1 -- 172.21.15.90:0/3209071725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648e94de4d0 msgr2=0x5648e98a7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:08.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.390+0000 7f35f936b640 1 --2- 172.21.15.90:0/2698895998 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f35dc0afc60 0x7f35dc0b2170 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x5648e98a4bd0 tx=0x7f35ec005fd0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:08.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.390+0000 7f3601c83740 1 --2- 172.21.15.90:0/3209071725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648e94de4d0 0x5648e98a7fa0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f35f4069bf0 tx=0x7f35f00118d0 comp rx=0 tx=0).stop 2024-04-25T11:39:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.390+0000 7f3601c83740 1 -- 172.21.15.90:0/3209071725 shutdown_connections 2024-04-25T11:39:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.390+0000 7f3601c83740 1 --2- 172.21.15.90:0/3209071725 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648e94e66b0 0x5648e98b48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.390+0000 7f3601c83740 1 --2- 172.21.15.90:0/3209071725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648e94de4d0 0x5648e98a7fa0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.390+0000 7f3601c83740 1 --2- 172.21.15.90:0/3209071725 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5648e9507390 0x5648e98a7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.390+0000 7f3601c83740 1 -- 172.21.15.90:0/3209071725 >> 172.21.15.90:0/3209071725 conn(0x5648e94fb4e0 msgr2=0x5648e98a4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.390+0000 7f3601c83740 1 -- 172.21.15.90:0/3209071725 shutdown_connections 2024-04-25T11:39:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.390+0000 7f3601c83740 1 -- 172.21.15.90:0/3209071725 wait complete. 2024-04-25T11:39:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.391+0000 7f3601c83740 1 -- start start 2024-04-25T11:39:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.391+0000 7f3601c83740 1 Processor -- start 2024-04-25T11:39:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.391+0000 7f3601c83740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648e94de4d0 0x5648e98b2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.391+0000 7f3601c83740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5648e94e66b0 0x5648e98b2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.391+0000 7f3601c83740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648e9507390 0x5648e98b84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.391+0000 7f3601c83740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5648e98b4fe0 con 0x5648e94de4d0 2024-04-25T11:39:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.391+0000 7f3601c83740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5648e9794d10 con 0x5648e94e66b0 2024-04-25T11:39:08.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.391+0000 7f3601c83740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5648e98b5160 con 0x5648e9507390 2024-04-25T11:39:08.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.391+0000 7f3601c83740 10 client.0 _ll_register_callbacks cb 0x5648e97c1210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:08.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.392+0000 7f35f8b6a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648e94de4d0 0x5648e98b2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:08.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.392+0000 7f35ebfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5648e94e66b0 0x5648e98b2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:08.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.392+0000 7f35ebfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5648e94e66b0 0x5648e98b2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:33852/0 (socket says 172.21.15.90:33852) 2024-04-25T11:39:08.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.392+0000 7f35f8b6a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648e94de4d0 0x5648e98b2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42104/0 (socket says 172.21.15.90:42104) 2024-04-25T11:39:08.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.392+0000 7f35ebfff640 1 -- 172.21.15.90:0/2120538549 learned_addr learned my addr 172.21.15.90:0/2120538549 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:08.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.392+0000 7f35f936b640 1 --2- 172.21.15.90:0/2120538549 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648e9507390 0x5648e98b84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:08.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.392+0000 7f35ebfff640 1 -- 172.21.15.90:0/2120538549 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648e9507390 msgr2=0x5648e98b84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:08.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.392+0000 7f35ebfff640 1 --2- 172.21.15.90:0/2120538549 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648e9507390 0x5648e98b84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:08.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.392+0000 7f35ebfff640 1 -- 172.21.15.90:0/2120538549 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648e94de4d0 msgr2=0x5648e98b2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:08.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.392+0000 7f35ebfff640 1 --2- 172.21.15.90:0/2120538549 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648e94de4d0 0x5648e98b2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:08.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.392+0000 7f35ebfff640 1 -- 172.21.15.90:0/2120538549 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648e95e97d0 con 0x5648e94e66b0 2024-04-25T11:39:08.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.392+0000 7f35f8b6a640 1 --2- 172.21.15.90:0/2120538549 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648e94de4d0 0x5648e98b2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:08.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.392+0000 7f35ebfff640 1 --2- 172.21.15.90:0/2120538549 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5648e94e66b0 0x5648e98b2b80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f35e40dd100 tx=0x7f35e4090050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12afc705d9a7ee8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:08.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.392+0000 7f35d3fff640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:39:08.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.392+0000 7f35d3fff640 1 -- 172.21.15.90:0/2120538549 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e40a7c50 con 0x5648e94e66b0 2024-04-25T11:39:08.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.392+0000 7f35d3fff640 1 -- 172.21.15.90:0/2120538549 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f35e40b0130 con 0x5648e94e66b0 2024-04-25T11:39:08.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.392+0000 7f3601c83740 1 -- 172.21.15.90:0/2120538549 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35f00047a0 con 0x5648e94e66b0 2024-04-25T11:39:08.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.393+0000 7f35d3fff640 1 -- 172.21.15.90:0/2120538549 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f35e40a7550 con 0x5648e94e66b0 2024-04-25T11:39:08.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.393+0000 7f3601c83740 10 client.25988 fetch_fsmap learned FSMap version 27 2024-04-25T11:39:08.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.393+0000 7f3601c83740 1 -- 172.21.15.90:0/2120538549 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5648e951f3f0 con 0x5648e94e66b0 2024-04-25T11:39:08.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.393+0000 7f35d3fff640 1 -- 172.21.15.90:0/2120538549 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35f00047a0 con 0x5648e94e66b0 2024-04-25T11:39:08.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.393+0000 7f35d3fff640 1 -- 172.21.15.90:0/2120538549 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f35e40df050 con 0x5648e94e66b0 2024-04-25T11:39:08.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.393+0000 7f3601c83740 10 client.25988 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:39:08.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.393+0000 7f3601c83740 10 client.25988 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:39:08.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.393+0000 7f3601c83740 20 client.25988 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:08.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.393+0000 7f3601c83740 1 --2- 172.21.15.90:0/2120538549 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5648e94c5f60 0x5648e98c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:08.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.393+0000 7f3601c83740 4 client.25988 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:39:08.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.393+0000 7f3601c83740 1 -- 172.21.15.90:0/2120538549 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648e981b990 con 0x5648e94c5f60 2024-04-25T11:39:08.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.393+0000 7f35f8b6a640 1 --2- 172.21.15.90:0/2120538549 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5648e94c5f60 0x5648e98c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:08.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.394+0000 7f35f8b6a640 1 --2- 172.21.15.90:0/2120538549 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5648e94c5f60 0x5648e98c4760 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=355a40aa9b2eaacf server_cookie=22661454ade9c482 in_seq=0 out_seq=0 2024-04-25T11:39:08.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:08.394+0000 7f35d3fff640 10 client.25988 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:09.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.264+0000 7f80dbfff640 1 -- 172.21.15.90:0/2846990190 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55beb78b6ee0 con 0x55beb75a91b0 2024-04-25T11:39:09.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.264+0000 7f80dbfff640 10 client.16752 handle_command_reply: tid=0 2024-04-25T11:39:09.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.264+0000 7f8109ba7740 1 client.16752 shutdown 2024-04-25T11:39:09.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.264+0000 7f8109ba7740 1 -- 172.21.15.90:0/2846990190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb7567d80 msgr2=0x55beb795af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:09.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.264+0000 7f8109ba7740 1 --2- 172.21.15.90:0/2846990190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb7567d80 0x55beb795af40 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f80f000e010 tx=0x7f80f0026df0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.264+0000 7f8109ba7740 1 -- 172.21.15.90:0/2846990190 shutdown_connections 2024-04-25T11:39:09.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.264+0000 7f8109ba7740 1 --2- 172.21.15.90:0/2846990190 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55beb75a91b0 0x55beb79cfcb0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.264+0000 7f8109ba7740 1 --2- 172.21.15.90:0/2846990190 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55beb75884d0 0x55beb7958340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.264+0000 7f8109ba7740 1 --2- 172.21.15.90:0/2846990190 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55beb75802f0 0x55beb795b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.264+0000 7f8109ba7740 1 --2- 172.21.15.90:0/2846990190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55beb7567d80 0x55beb795af40 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.264+0000 7f8109ba7740 1 -- 172.21.15.90:0/2846990190 >> 172.21.15.90:0/2846990190 conn(0x55beb759d300 msgr2=0x55beb7952880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:09.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.264+0000 7f80dbfff640 0 client.16752 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:09.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.265+0000 7f8109ba7740 1 -- 172.21.15.90:0/2846990190 shutdown_connections 2024-04-25T11:39:09.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.265+0000 7f8109ba7740 1 -- 172.21.15.90:0/2846990190 wait complete. 2024-04-25T11:39:09.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.265+0000 7f8109ba7740 20 client.16752 trim_cache size 0 max 16384 2024-04-25T11:39:09.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.265+0000 7f35d3fff640 1 -- 172.21.15.90:0/2120538549 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5648e981b990 con 0x5648e94c5f60 2024-04-25T11:39:09.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.265+0000 7f35d3fff640 10 client.25988 handle_command_reply: tid=0 2024-04-25T11:39:09.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.265+0000 7f3601c83740 1 client.25988 shutdown 2024-04-25T11:39:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:39:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:39:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:39:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:39:09.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.265+0000 7f3601c83740 1 -- 172.21.15.90:0/2120538549 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5648e94e66b0 msgr2=0x5648e98b2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:09.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.265+0000 7f3601c83740 1 --2- 172.21.15.90:0/2120538549 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5648e94e66b0 0x5648e98b2b80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f35e40dd100 tx=0x7f35e4090050 comp rx=0 tx=0).stop 2024-04-25T11:39:09.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.265+0000 7f3601c83740 1 -- 172.21.15.90:0/2120538549 shutdown_connections 2024-04-25T11:39:09.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.265+0000 7f3601c83740 1 --2- 172.21.15.90:0/2120538549 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5648e94c5f60 0x5648e98c4760 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.265+0000 7f35d3fff640 0 client.25988 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:09.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.265+0000 7f3601c83740 1 --2- 172.21.15.90:0/2120538549 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648e9507390 0x5648e98b84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.265+0000 7f3601c83740 1 --2- 172.21.15.90:0/2120538549 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5648e94e66b0 0x5648e98b2b80 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f35e40dd100 tx=0x7f35e4090050 comp rx=0 tx=0).stop 2024-04-25T11:39:09.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.265+0000 7f3601c83740 1 --2- 172.21.15.90:0/2120538549 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648e94de4d0 0x5648e98b2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.265+0000 7f3601c83740 1 -- 172.21.15.90:0/2120538549 >> 172.21.15.90:0/2120538549 conn(0x5648e94fb4e0 msgr2=0x5648e98a9780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:09.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.266+0000 7f8102c15640 1 -- 172.21.15.90:0/62881302 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f80ec0afc50 msgr2=0x7f80ec0b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:09.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.266+0000 7f8102c15640 1 --2- 172.21.15.90:0/62881302 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f80ec0afc50 0x7f80ec0b2160 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f80f000b9e0 tx=0x7f80f000b370 comp rx=0 tx=0).stop 2024-04-25T11:39:09.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.266+0000 7f8102c15640 1 -- 172.21.15.90:0/62881302 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80fc10b750 msgr2=0x7f80fc102af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:09.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.266+0000 7f8102c15640 1 --2- 172.21.15.90:0/62881302 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80fc10b750 0x7f80fc102af0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f80e800b4a0 tx=0x7f80e800b960 comp rx=0 tx=0).stop 2024-04-25T11:39:09.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.266+0000 7f3601c83740 1 -- 172.21.15.90:0/2120538549 shutdown_connections 2024-04-25T11:39:09.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.266+0000 7f8102c15640 1 -- 172.21.15.90:0/62881302 shutdown_connections 2024-04-25T11:39:09.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.266+0000 7f8102c15640 1 --2- 172.21.15.90:0/62881302 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f80ec0afc50 0x7f80ec0b2160 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.266+0000 7f8102c15640 1 --2- 172.21.15.90:0/62881302 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f80fc10e100 0x7f80fc103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.258 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.266+0000 7f8102c15640 1 --2- 172.21.15.90:0/62881302 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80fc10b750 0x7f80fc102af0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.258 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.266+0000 7f8102c15640 1 --2- 172.21.15.90:0/62881302 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80fc104cf0 0x7f80fc1025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.258 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.266+0000 7f8102c15640 1 -- 172.21.15.90:0/62881302 >> 172.21.15.90:0/62881302 conn(0x7f80fc0781b0 msgr2=0x7f80fc101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:09.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.266+0000 7f3601c83740 1 -- 172.21.15.90:0/2120538549 wait complete. 2024-04-25T11:39:09.258 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.267+0000 7f8102c15640 1 -- 172.21.15.90:0/62881302 shutdown_connections 2024-04-25T11:39:09.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.267+0000 7f3601c83740 20 client.25988 trim_cache size 0 max 16384 2024-04-25T11:39:09.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:09.267+0000 7f8102c15640 1 -- 172.21.15.90:0/62881302 wait complete. 2024-04-25T11:39:09.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.312+0000 7f3601c83740 1 Processor -- start 2024-04-25T11:39:09.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.312+0000 7f3601c83740 1 -- start start 2024-04-25T11:39:09.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.313+0000 7f3601c83740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5648e94c5f60 0x5648e98b8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:09.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.313+0000 7f3601c83740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648e94de4d0 0x5648e98b0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:09.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.313+0000 7f3601c83740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648e9507390 0x5648e98b1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:09.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.313+0000 7f35ebfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648e94de4d0 0x5648e98b0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:09.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.313+0000 7f35ebfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648e94de4d0 0x5648e98b0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42112/0 (socket says 172.21.15.90:42112) 2024-04-25T11:39:09.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.313+0000 7f35ebfff640 1 -- 172.21.15.90:0/253399873 learned_addr learned my addr 172.21.15.90:0/253399873 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:09.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.313+0000 7f3601c83740 1 -- 172.21.15.90:0/253399873 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f35f41b1d80 con 0x5648e94de4d0 2024-04-25T11:39:09.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.313+0000 7f3601c83740 1 -- 172.21.15.90:0/253399873 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f35f41b38f0 con 0x5648e94c5f60 2024-04-25T11:39:09.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.313+0000 7f3601c83740 1 -- 172.21.15.90:0/253399873 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f35f41b2450 con 0x5648e9507390 2024-04-25T11:39:09.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.314+0000 7f35f936b640 1 --2- 172.21.15.90:0/253399873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648e9507390 0x5648e98b1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:09.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.315+0000 7f35ebfff640 1 -- 172.21.15.90:0/253399873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648e9507390 msgr2=0x5648e98b1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:09.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.315+0000 7f35ebfff640 1 --2- 172.21.15.90:0/253399873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648e9507390 0x5648e98b1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.315+0000 7f35ebfff640 1 -- 172.21.15.90:0/253399873 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5648e94c5f60 msgr2=0x5648e98b8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:09.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.315+0000 7f35ebfff640 1 --2- 172.21.15.90:0/253399873 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5648e94c5f60 0x5648e98b8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.315+0000 7f35ebfff640 1 -- 172.21.15.90:0/253399873 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648e94df0f0 con 0x5648e94de4d0 2024-04-25T11:39:09.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.315+0000 7f35ebfff640 1 --2- 172.21.15.90:0/253399873 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648e94de4d0 0x5648e98b0d00 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f35e40a7710 tx=0x7f35e4004a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ce6bc972d35d2a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:09.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.317+0000 7f35da7fc640 1 -- 172.21.15.90:0/253399873 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e40a6ea0 con 0x5648e94de4d0 2024-04-25T11:39:09.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.317+0000 7f35da7fc640 1 -- 172.21.15.90:0/253399873 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35e4005680 con 0x5648e94de4d0 2024-04-25T11:39:09.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.318+0000 7f35da7fc640 1 -- 172.21.15.90:0/253399873 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e40c83b0 con 0x5648e94de4d0 2024-04-25T11:39:09.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.319+0000 7f3601c83740 1 -- 172.21.15.90:0/253399873 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648e94de4d0 msgr2=0x5648e98b0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:09.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.319+0000 7f3601c83740 1 --2- 172.21.15.90:0/253399873 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648e94de4d0 0x5648e98b0d00 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f35e40a7710 tx=0x7f35e4004a60 comp rx=0 tx=0).stop 2024-04-25T11:39:09.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.319+0000 7f3601c83740 1 -- 172.21.15.90:0/253399873 shutdown_connections 2024-04-25T11:39:09.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.319+0000 7f3601c83740 1 --2- 172.21.15.90:0/253399873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648e9507390 0x5648e98b1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.319+0000 7f3601c83740 1 --2- 172.21.15.90:0/253399873 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648e94de4d0 0x5648e98b0d00 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.319+0000 7f3601c83740 1 --2- 172.21.15.90:0/253399873 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5648e94c5f60 0x5648e98b8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.319+0000 7f3601c83740 1 -- 172.21.15.90:0/253399873 >> 172.21.15.90:0/253399873 conn(0x5648e94fb4e0 msgr2=0x5648e98ad5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:09.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.320+0000 7f3601c83740 1 -- 172.21.15.90:0/253399873 shutdown_connections 2024-04-25T11:39:09.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.320+0000 7f3601c83740 1 -- 172.21.15.90:0/253399873 wait complete. 2024-04-25T11:39:09.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.321+0000 7f3601c83740 1 -- start start 2024-04-25T11:39:09.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.321+0000 7f3601c83740 1 Processor -- start 2024-04-25T11:39:09.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.322+0000 7f3601c83740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5648e94c5f60 0x5648e98b9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:09.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.322+0000 7f3601c83740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648e94de4d0 0x5648e98b9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:09.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.322+0000 7f3601c83740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648e9507390 0x5648e993e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:09.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.322+0000 7f3601c83740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f35f41b31f0 con 0x5648e9507390 2024-04-25T11:39:09.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.323+0000 7f3601c83740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f35f41b2b20 con 0x5648e94c5f60 2024-04-25T11:39:09.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.323+0000 7f3601c83740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f35f41afd60 con 0x5648e94de4d0 2024-04-25T11:39:09.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.323+0000 7f35f8b6a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5648e94c5f60 0x5648e98b9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:09.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.323+0000 7f35f8b6a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5648e94c5f60 0x5648e98b9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:33872/0 (socket says 172.21.15.90:33872) 2024-04-25T11:39:09.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.323+0000 7f35f8b6a640 1 -- 172.21.15.90:0/2514992956 learned_addr learned my addr 172.21.15.90:0/2514992956 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:09.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.323+0000 7f3601c83740 10 client.0 _ll_register_callbacks cb 0x7f35e400cef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:09.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.324+0000 7f35ebfff640 1 --2- 172.21.15.90:0/2514992956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648e94de4d0 0x5648e98b9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:09.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.324+0000 7f35f936b640 1 --2- 172.21.15.90:0/2514992956 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648e9507390 0x5648e993e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:09.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.325+0000 7f35f936b640 1 -- 172.21.15.90:0/2514992956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648e94de4d0 msgr2=0x5648e98b9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:09.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.325+0000 7f35f936b640 1 --2- 172.21.15.90:0/2514992956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648e94de4d0 0x5648e98b9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.325+0000 7f35f936b640 1 -- 172.21.15.90:0/2514992956 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5648e94c5f60 msgr2=0x5648e98b9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:09.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.325+0000 7f35f936b640 1 --2- 172.21.15.90:0/2514992956 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5648e94c5f60 0x5648e98b9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.325+0000 7f35f936b640 1 -- 172.21.15.90:0/2514992956 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648e97ce4b0 con 0x5648e9507390 2024-04-25T11:39:09.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.326+0000 7f35f8b6a640 1 --2- 172.21.15.90:0/2514992956 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5648e94c5f60 0x5648e98b9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:09.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.327+0000 7f35f936b640 1 --2- 172.21.15.90:0/2514992956 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648e9507390 0x5648e993e2f0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f35ec009420 tx=0x7f35ec009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=556eb563d8d3f923 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:09.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.327+0000 7f35d3fff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:09.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.330+0000 7f35d3fff640 1 -- 172.21.15.90:0/2514992956 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ec036040 con 0x5648e9507390 2024-04-25T11:39:09.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.331+0000 7f35d3fff640 1 -- 172.21.15.90:0/2514992956 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f35ec004000 con 0x5648e9507390 2024-04-25T11:39:09.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.331+0000 7f3601c83740 1 -- 172.21.15.90:0/2514992956 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35e4005680 con 0x5648e9507390 2024-04-25T11:39:09.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.332+0000 7f35d3fff640 1 -- 172.21.15.90:0/2514992956 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f35ec004200 con 0x5648e9507390 2024-04-25T11:39:09.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.333+0000 7f3601c83740 10 client.16767 fetch_fsmap learned FSMap version 27 2024-04-25T11:39:09.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.333+0000 7f3601c83740 1 -- 172.21.15.90:0/2514992956 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5648e9945950 con 0x5648e9507390 2024-04-25T11:39:09.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.335+0000 7f35d3fff640 1 -- 172.21.15.90:0/2514992956 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35ec004850 con 0x5648e9507390 2024-04-25T11:39:09.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.335+0000 7f35d3fff640 1 -- 172.21.15.90:0/2514992956 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f35ec053020 con 0x5648e9507390 2024-04-25T11:39:09.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.335+0000 7f3601c83740 10 client.16767 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:39:09.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.335+0000 7f3601c83740 10 client.16767 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:39:09.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.335+0000 7f3601c83740 20 client.16767 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:09.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.335+0000 7f3601c83740 1 --2- 172.21.15.90:0/2514992956 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5648e949b3a0 0x5648e99480a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:09.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.335+0000 7f3601c83740 4 client.16767 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:39:09.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.335+0000 7f3601c83740 1 -- 172.21.15.90:0/2514992956 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f35e4005680 con 0x5648e949b3a0 2024-04-25T11:39:09.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.337+0000 7f35f8b6a640 1 --2- 172.21.15.90:0/2514992956 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5648e949b3a0 0x5648e99480a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:09.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.339+0000 7f35f8b6a640 1 --2- 172.21.15.90:0/2514992956 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5648e949b3a0 0x5648e99480a0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b673ccb8ff3ea0d1 server_cookie=ebfa32383e08f060 in_seq=0 out_seq=0 2024-04-25T11:39:09.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.339+0000 7f35d3fff640 10 client.16767 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:09.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.342+0000 7f35d3fff640 1 -- 172.21.15.90:0/2514992956 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f35e4005680 con 0x5648e949b3a0 2024-04-25T11:39:09.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.342+0000 7f35d3fff640 10 client.16767 handle_command_reply: tid=0 2024-04-25T11:39:09.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.342+0000 7f3601c83740 1 client.16767 shutdown 2024-04-25T11:39:09.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.342+0000 7f3601c83740 1 -- 172.21.15.90:0/2514992956 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648e9507390 msgr2=0x5648e993e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:09.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.342+0000 7f3601c83740 1 --2- 172.21.15.90:0/2514992956 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648e9507390 0x5648e993e2f0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f35ec009420 tx=0x7f35ec009500 comp rx=0 tx=0).stop 2024-04-25T11:39:09.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.342+0000 7f3601c83740 1 -- 172.21.15.90:0/2514992956 shutdown_connections 2024-04-25T11:39:09.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.342+0000 7f3601c83740 1 --2- 172.21.15.90:0/2514992956 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5648e949b3a0 0x5648e99480a0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.342+0000 7f3601c83740 1 --2- 172.21.15.90:0/2514992956 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648e9507390 0x5648e993e2f0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.342+0000 7f35d3fff640 0 client.16767 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:09.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.342+0000 7f3601c83740 1 --2- 172.21.15.90:0/2514992956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648e94de4d0 0x5648e98b9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.342+0000 7f3601c83740 1 --2- 172.21.15.90:0/2514992956 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5648e94c5f60 0x5648e98b9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.342+0000 7f3601c83740 1 -- 172.21.15.90:0/2514992956 >> 172.21.15.90:0/2514992956 conn(0x5648e94fb4e0 msgr2=0x5648e98b0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:09.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.342+0000 7f3601c83740 1 -- 172.21.15.90:0/2514992956 shutdown_connections 2024-04-25T11:39:09.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.342+0000 7f3601c83740 1 -- 172.21.15.90:0/2514992956 wait complete. 2024-04-25T11:39:09.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.342+0000 7f3601c83740 20 client.16767 trim_cache size 0 max 16384 2024-04-25T11:39:09.333 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:39:09.333 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-25T11:39:09.333 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-25T11:39:09.333 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:39:09.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.343+0000 7f35fadce640 1 -- 172.21.15.90:0/2698895998 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f35dc0afc60 msgr2=0x7f35dc0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:09.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.343+0000 7f35fadce640 1 --2- 172.21.15.90:0/2698895998 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f35dc0afc60 0x7f35dc0b2170 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x5648e98a4bd0 tx=0x7f35ec005fd0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.343+0000 7f35fadce640 1 -- 172.21.15.90:0/2698895998 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35f4109ba0 msgr2=0x7f35f419db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:09.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.343+0000 7f35fadce640 1 --2- 172.21.15.90:0/2698895998 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35f4109ba0 0x7f35f419db30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f35e400e9f0 tx=0x7f35e400edf0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.344+0000 7f35fadce640 1 -- 172.21.15.90:0/2698895998 shutdown_connections 2024-04-25T11:39:09.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.344+0000 7f35fadce640 1 --2- 172.21.15.90:0/2698895998 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f35dc0afc60 0x7f35dc0b2170 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.344+0000 7f35fadce640 1 --2- 172.21.15.90:0/2698895998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f410e550 0x7f35f41a8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.344+0000 7f35fadce640 1 --2- 172.21.15.90:0/2698895998 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35f4109ba0 0x7f35f419db30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.344+0000 7f35fadce640 1 --2- 172.21.15.90:0/2698895998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f4102050 0x7f35f419d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.344+0000 7f35fadce640 1 -- 172.21.15.90:0/2698895998 >> 172.21.15.90:0/2698895998 conn(0x7f35f4077880 msgr2=0x7f35f4102d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:09.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.344+0000 7f35fadce640 1 -- 172.21.15.90:0/2698895998 shutdown_connections 2024-04-25T11:39:09.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.344+0000 7f35fadce640 1 -- 172.21.15.90:0/2698895998 wait complete. 2024-04-25T11:39:09.372 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T11:39:09.372 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:39:09.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.692+0000 7f1260b00640 1 -- 172.21.15.90:0/3724749756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f125c111ce0 msgr2=0x7f125c112110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:09.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.692+0000 7f1260b00640 1 --2- 172.21.15.90:0/3724749756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f125c111ce0 0x7f125c112110 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f1250009900 tx=0x7f125002a440 comp rx=0 tx=0).stop 2024-04-25T11:39:09.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.692+0000 7f1260b00640 1 -- 172.21.15.90:0/3724749756 shutdown_connections 2024-04-25T11:39:09.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.692+0000 7f1260b00640 1 --2- 172.21.15.90:0/3724749756 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f125c1127a0 0x7f125c118d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.692+0000 7f1260b00640 1 --2- 172.21.15.90:0/3724749756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f125c111ce0 0x7f125c112110 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.692+0000 7f1260b00640 1 --2- 172.21.15.90:0/3724749756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f125c10a350 0x7f125c10a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.692+0000 7f1260b00640 1 -- 172.21.15.90:0/3724749756 >> 172.21.15.90:0/3724749756 conn(0x7f125c06e7b0 msgr2=0x7f125c06ec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:09.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.692+0000 7f1260b00640 1 -- 172.21.15.90:0/3724749756 shutdown_connections 2024-04-25T11:39:09.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.692+0000 7f1260b00640 1 -- 172.21.15.90:0/3724749756 wait complete. 2024-04-25T11:39:09.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.693+0000 7f1260b00640 1 Processor -- start 2024-04-25T11:39:09.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.693+0000 7f1260b00640 1 -- start start 2024-04-25T11:39:09.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.694+0000 7f1260b00640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f125c10a350 0x7f125c10ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:09.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.694+0000 7f1260b00640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f125c111ce0 0x7f125c10bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:09.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.694+0000 7f1260b00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f125c1127a0 0x7f125c10c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:09.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.694+0000 7f1260b00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f125c11c5d0 con 0x7f125c1127a0 2024-04-25T11:39:09.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.694+0000 7f1260b00640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f125c11c450 con 0x7f125c10a350 2024-04-25T11:39:09.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.694+0000 7f1260b00640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f125c11c750 con 0x7f125c111ce0 2024-04-25T11:39:09.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.694+0000 7f125ad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f125c1127a0 0x7f125c10c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:09.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.694+0000 7f125ad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f125c1127a0 0x7f125c10c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42130/0 (socket says 172.21.15.90:42130) 2024-04-25T11:39:09.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.694+0000 7f125ad9d640 1 -- 172.21.15.90:0/2981923313 learned_addr learned my addr 172.21.15.90:0/2981923313 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:09.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.694+0000 7f1259d9b640 1 --2- 172.21.15.90:0/2981923313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f125c111ce0 0x7f125c10bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:09.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.694+0000 7f125ad9d640 1 -- 172.21.15.90:0/2981923313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f125c111ce0 msgr2=0x7f125c10bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:09.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.694+0000 7f125ad9d640 1 --2- 172.21.15.90:0/2981923313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f125c111ce0 0x7f125c10bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.694+0000 7f125ad9d640 1 -- 172.21.15.90:0/2981923313 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f125c10a350 msgr2=0x7f125c10ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:09.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.694+0000 7f125ad9d640 1 --2- 172.21.15.90:0/2981923313 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f125c10a350 0x7f125c10ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.694+0000 7f125ad9d640 1 -- 172.21.15.90:0/2981923313 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f125c1b1430 con 0x7f125c1127a0 2024-04-25T11:39:09.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.695+0000 7f125ad9d640 1 --2- 172.21.15.90:0/2981923313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f125c1127a0 0x7f125c10c4e0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f124c00b3f0 tx=0x7f124c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:09.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.695+0000 7f124b7fe640 1 -- 172.21.15.90:0/2981923313 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124c015020 con 0x7f125c1127a0 2024-04-25T11:39:09.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.695+0000 7f124b7fe640 1 -- 172.21.15.90:0/2981923313 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f124c011ce0 con 0x7f125c1127a0 2024-04-25T11:39:09.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.695+0000 7f124b7fe640 1 -- 172.21.15.90:0/2981923313 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124c010da0 con 0x7f125c1127a0 2024-04-25T11:39:09.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.695+0000 7f1260b00640 1 -- 172.21.15.90:0/2981923313 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f125c1b1720 con 0x7f125c1127a0 2024-04-25T11:39:09.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.696+0000 7f1260b00640 1 -- 172.21.15.90:0/2981923313 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f125c1b1bf0 con 0x7f125c1127a0 2024-04-25T11:39:09.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.697+0000 7f1260b00640 1 -- 172.21.15.90:0/2981923313 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f125c10b3d0 con 0x7f125c1127a0 2024-04-25T11:39:09.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.701+0000 7f124b7fe640 1 -- 172.21.15.90:0/2981923313 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f124c018020 con 0x7f125c1127a0 2024-04-25T11:39:09.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.702+0000 7f124b7fe640 1 --2- 172.21.15.90:0/2981923313 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f12440afce0 0x7f12440b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:09.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.703+0000 7f125a59c640 1 --2- 172.21.15.90:0/2981923313 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f12440afce0 0x7f12440b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:09.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.703+0000 7f124b7fe640 1 -- 172.21.15.90:0/2981923313 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f124c0dede0 con 0x7f125c1127a0 2024-04-25T11:39:09.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.704+0000 7f125a59c640 1 --2- 172.21.15.90:0/2981923313 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f12440afce0 0x7f12440b21f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f125c10add0 tx=0x7f1254009250 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:09.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:09.706+0000 7f124b7fe640 1 -- 172.21.15.90:0/2981923313 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f124c01f020 con 0x7f125c1127a0 2024-04-25T11:39:10.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.164+0000 7f1260b00640 1 -- 172.21.15.90:0/2981923313 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f125c069d40 con 0x7f125c1127a0 2024-04-25T11:39:10.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.166+0000 7f124b7fe640 1 -- 172.21.15.90:0/2981923313 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11931 (secure 0 0 0) 0x7f124c0a7730 con 0x7f125c1127a0 2024-04-25T11:39:10.157 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:39:10.157 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:37:15.460907+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24475,24524,24499,24431,24458]},"id":1}]} 2024-04-25T11:39:10.157 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-25T11:39:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.171+0000 7f1260b00640 1 -- 172.21.15.90:0/2981923313 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f12440afce0 msgr2=0x7f12440b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.171+0000 7f1260b00640 1 --2- 172.21.15.90:0/2981923313 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f12440afce0 0x7f12440b21f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f125c10add0 tx=0x7f1254009250 comp rx=0 tx=0).stop 2024-04-25T11:39:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.171+0000 7f1260b00640 1 -- 172.21.15.90:0/2981923313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f125c1127a0 msgr2=0x7f125c10c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.171+0000 7f1260b00640 1 --2- 172.21.15.90:0/2981923313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f125c1127a0 0x7f125c10c4e0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f124c00b3f0 tx=0x7f124c00b8b0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.171+0000 7f1260b00640 1 -- 172.21.15.90:0/2981923313 shutdown_connections 2024-04-25T11:39:10.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.171+0000 7f1260b00640 1 --2- 172.21.15.90:0/2981923313 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f12440afce0 0x7f12440b21f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.171+0000 7f1260b00640 1 --2- 172.21.15.90:0/2981923313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f125c1127a0 0x7f125c10c4e0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.171+0000 7f1260b00640 1 --2- 172.21.15.90:0/2981923313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f125c111ce0 0x7f125c10bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.171+0000 7f1260b00640 1 --2- 172.21.15.90:0/2981923313 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f125c10a350 0x7f125c10ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.171+0000 7f1260b00640 1 -- 172.21.15.90:0/2981923313 >> 172.21.15.90:0/2981923313 conn(0x7f125c06e7b0 msgr2=0x7f125c10dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:10.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.171+0000 7f1260b00640 1 -- 172.21.15.90:0/2981923313 shutdown_connections 2024-04-25T11:39:10.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.171+0000 7f1260b00640 1 -- 172.21.15.90:0/2981923313 wait complete. 2024-04-25T11:39:10.183 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T11:39:10.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:10 smithi138 ceph-mon[32328]: pgmap v125: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 1.0 MiB/s rd, 399 MiB/s wr, 475 op/s 2024-04-25T11:39:10.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:10 smithi156 ceph-mon[33045]: pgmap v125: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 1.0 MiB/s rd, 399 MiB/s wr, 475 op/s 2024-04-25T11:39:10.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.451+0000 7f5f4364a640 1 -- 172.21.15.90:0/3988964226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f3c107450 msgr2=0x7f5f3c10ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:10.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.451+0000 7f5f4364a640 1 --2- 172.21.15.90:0/3988964226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f3c107450 0x7f5f3c10ebf0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f5f3400adb0 tx=0x7f5f3402b7c0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.451+0000 7f5f4364a640 1 -- 172.21.15.90:0/3988964226 shutdown_connections 2024-04-25T11:39:10.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.451+0000 7f5f4364a640 1 --2- 172.21.15.90:0/3988964226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f3c107450 0x7f5f3c10ebf0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.452+0000 7f5f4364a640 1 --2- 172.21.15.90:0/3988964226 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f3c106990 0x7f5f3c106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.452+0000 7f5f4364a640 1 --2- 172.21.15.90:0/3988964226 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5f3c102400 0x7f5f3c102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.452+0000 7f5f4364a640 1 -- 172.21.15.90:0/3988964226 >> 172.21.15.90:0/3988964226 conn(0x7f5f3c077880 msgr2=0x7f5f3c077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:10.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.452+0000 7f5f4364a640 1 -- 172.21.15.90:0/3988964226 shutdown_connections 2024-04-25T11:39:10.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.452+0000 7f5f4364a640 1 -- 172.21.15.90:0/3988964226 wait complete. 2024-04-25T11:39:10.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.452+0000 7f5f4364a640 1 Processor -- start 2024-04-25T11:39:10.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.453+0000 7f5f4364a640 1 -- start start 2024-04-25T11:39:10.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.453+0000 7f5f4364a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f3c102400 0x7f5f3c0ff030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.453+0000 7f5f4364a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f3c106990 0x7f5f3c0ff570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.453+0000 7f5f4364a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5f3c107450 0x7f5f3c0ffab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.453+0000 7f5f4364a640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3c111bd0 con 0x7f5f3c102400 2024-04-25T11:39:10.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.453+0000 7f5f4364a640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3c111a50 con 0x7f5f3c107450 2024-04-25T11:39:10.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.453+0000 7f5f4364a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3c111d50 con 0x7f5f3c106990 2024-04-25T11:39:10.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.453+0000 7f5f413e6640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f3c102400 0x7f5f3c0ff030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.453+0000 7f5f413e6640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f3c102400 0x7f5f3c0ff030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42162/0 (socket says 172.21.15.90:42162) 2024-04-25T11:39:10.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.453+0000 7f5f413e6640 1 -- 172.21.15.90:0/2456565545 learned_addr learned my addr 172.21.15.90:0/2456565545 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:10.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.453+0000 7f5f40be5640 1 --2- 172.21.15.90:0/2456565545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f3c106990 0x7f5f3c0ff570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.454+0000 7f5f413e6640 1 -- 172.21.15.90:0/2456565545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f3c106990 msgr2=0x7f5f3c0ff570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:10.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.454+0000 7f5f413e6640 1 --2- 172.21.15.90:0/2456565545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f3c106990 0x7f5f3c0ff570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.454+0000 7f5f413e6640 1 -- 172.21.15.90:0/2456565545 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5f3c107450 msgr2=0x7f5f3c0ffab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:10.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.454+0000 7f5f413e6640 1 --2- 172.21.15.90:0/2456565545 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5f3c107450 0x7f5f3c0ffab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.454+0000 7f5f413e6640 1 -- 172.21.15.90:0/2456565545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f3c1a8b10 con 0x7f5f3c102400 2024-04-25T11:39:10.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.454+0000 7f5f40be5640 1 --2- 172.21.15.90:0/2456565545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f3c106990 0x7f5f3c0ff570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:39:10.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.454+0000 7f5f413e6640 1 --2- 172.21.15.90:0/2456565545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f3c102400 0x7f5f3c0ff030 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f5f3800b450 tx=0x7f5f3800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:10.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.455+0000 7f5f327fc640 1 -- 172.21.15.90:0/2456565545 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f38015020 con 0x7f5f3c102400 2024-04-25T11:39:10.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.455+0000 7f5f4364a640 1 -- 172.21.15.90:0/2456565545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f3c1a8e00 con 0x7f5f3c102400 2024-04-25T11:39:10.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.455+0000 7f5f327fc640 1 -- 172.21.15.90:0/2456565545 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f38011d40 con 0x7f5f3c102400 2024-04-25T11:39:10.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.455+0000 7f5f4364a640 1 -- 172.21.15.90:0/2456565545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f3c1a9280 con 0x7f5f3c102400 2024-04-25T11:39:10.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.455+0000 7f5f327fc640 1 -- 172.21.15.90:0/2456565545 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f3800f340 con 0x7f5f3c102400 2024-04-25T11:39:10.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.459+0000 7f5f4a5dd740 1 Processor -- start 2024-04-25T11:39:10.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.459+0000 7f5f4a5dd740 1 -- start start 2024-04-25T11:39:10.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.459+0000 7f5f4a5dd740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559697c39390 0x559697fda810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.459+0000 7f5f4a5dd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559697c104d0 0x559697fdad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.459+0000 7f5f4a5dd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559697c186b0 0x559697fe76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.459+0000 7f5f4a5dd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559697f3fd80 con 0x559697c104d0 2024-04-25T11:39:10.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.459+0000 7f5f4a5dd740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x559697ef9a20 con 0x559697c39390 2024-04-25T11:39:10.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.459+0000 7f5f4a5dd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559697f3bd40 con 0x559697c186b0 2024-04-25T11:39:10.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.460+0000 7f5f40be5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559697c186b0 0x559697fe76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.460+0000 7f5f40be5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559697c186b0 0x559697fe76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:48722/0 (socket says 172.21.15.90:48722) 2024-04-25T11:39:10.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.460+0000 7f5f40be5640 1 -- 172.21.15.90:0/3038548462 learned_addr learned my addr 172.21.15.90:0/3038548462 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:10.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.460+0000 7f5f41be7640 1 --2- 172.21.15.90:0/3038548462 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559697c39390 0x559697fda810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.461+0000 7f5f327fc640 1 -- 172.21.15.90:0/2456565545 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f5f38018020 con 0x7f5f3c102400 2024-04-25T11:39:10.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.461+0000 7f5f413e6640 1 --2- 172.21.15.90:0/3038548462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559697c104d0 0x559697fdad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.461+0000 7f5f40be5640 1 -- 172.21.15.90:0/3038548462 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559697c39390 msgr2=0x559697fda810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:10.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.461+0000 7f5f40be5640 1 --2- 172.21.15.90:0/3038548462 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559697c39390 0x559697fda810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.461+0000 7f5f40be5640 1 -- 172.21.15.90:0/3038548462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559697c104d0 msgr2=0x559697fdad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:10.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.461+0000 7f5f40be5640 1 --2- 172.21.15.90:0/3038548462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559697c104d0 0x559697fdad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.461+0000 7f5f40be5640 1 -- 172.21.15.90:0/3038548462 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559697f04c30 con 0x559697c186b0 2024-04-25T11:39:10.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.461+0000 7f5f413e6640 1 --2- 172.21.15.90:0/3038548462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559697c104d0 0x559697fdad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:39:10.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.462+0000 7f5f40be5640 1 --2- 172.21.15.90:0/3038548462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559697c186b0 0x559697fe76a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c00cc10 tx=0x7f5f2c00e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e071f978b39e7eb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:10.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.462+0000 7f5f277fe640 1 -- 172.21.15.90:0/3038548462 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c014070 con 0x559697c186b0 2024-04-25T11:39:10.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.462+0000 7f5f327fc640 1 --2- 172.21.15.90:0/2456565545 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5f280afce0 0x7f5f280b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.462+0000 7f5f277fe640 1 -- 172.21.15.90:0/3038548462 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f2c0047a0 con 0x559697c186b0 2024-04-25T11:39:10.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.462+0000 7f5f277fe640 1 -- 172.21.15.90:0/3038548462 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c010690 con 0x559697c186b0 2024-04-25T11:39:10.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.462+0000 7f5f41be7640 1 --2- 172.21.15.90:0/2456565545 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5f280afce0 0x7f5f280b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.463+0000 7f5f327fc640 1 -- 172.21.15.90:0/2456565545 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f5f380de9c0 con 0x7f5f3c102400 2024-04-25T11:39:10.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.463+0000 7f5f4a5dd740 1 -- 172.21.15.90:0/3038548462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559697c186b0 msgr2=0x559697fe76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:10.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.463+0000 7f5f4a5dd740 1 --2- 172.21.15.90:0/3038548462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559697c186b0 0x559697fe76a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c00cc10 tx=0x7f5f2c00e720 comp rx=0 tx=0).stop 2024-04-25T11:39:10.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.463+0000 7f5f41be7640 1 --2- 172.21.15.90:0/2456565545 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5f280afce0 0x7f5f280b21f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x559697fd7980 tx=0x7f5f3400aa40 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:10.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.463+0000 7f5f4a5dd740 1 -- 172.21.15.90:0/3038548462 shutdown_connections 2024-04-25T11:39:10.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.463+0000 7f5f4a5dd740 1 --2- 172.21.15.90:0/3038548462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559697c186b0 0x559697fe76a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.463+0000 7f5f4a5dd740 1 --2- 172.21.15.90:0/3038548462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559697c104d0 0x559697fdad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.464+0000 7f5f4a5dd740 1 --2- 172.21.15.90:0/3038548462 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559697c39390 0x559697fda810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.464+0000 7f5f4a5dd740 1 -- 172.21.15.90:0/3038548462 >> 172.21.15.90:0/3038548462 conn(0x559697c2d4e0 msgr2=0x559697fd6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:10.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.464+0000 7f5f4a5dd740 1 -- 172.21.15.90:0/3038548462 shutdown_connections 2024-04-25T11:39:10.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.464+0000 7f5f4a5dd740 1 -- 172.21.15.90:0/3038548462 wait complete. 2024-04-25T11:39:10.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.464+0000 7f5f4a5dd740 1 -- start start 2024-04-25T11:39:10.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.464+0000 7f5f4a5dd740 1 Processor -- start 2024-04-25T11:39:10.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.465+0000 7f5f4a5dd740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559697c104d0 0x559697fe5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.465+0000 7f5f4a5dd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559697c186b0 0x559697fe5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.465+0000 7f5f4a5dd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559697c39390 0x559697ff1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.465+0000 7f5f4a5dd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559697fe7d90 con 0x559697c186b0 2024-04-25T11:39:10.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.465+0000 7f5f4a5dd740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x559697ec6dc0 con 0x559697c104d0 2024-04-25T11:39:10.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.465+0000 7f5f4a5dd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559697fe7f10 con 0x559697c39390 2024-04-25T11:39:10.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.465+0000 7f5f40be5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559697c186b0 0x559697fe5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.465+0000 7f5f40be5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559697c186b0 0x559697fe5740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42178/0 (socket says 172.21.15.90:42178) 2024-04-25T11:39:10.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.465+0000 7f5f40be5640 1 -- 172.21.15.90:0/2793130719 learned_addr learned my addr 172.21.15.90:0/2793130719 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:10.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.465+0000 7f5f4a5dd740 10 client.0 _ll_register_callbacks cb 0x559697ee2600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:10.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.466+0000 7f5f40be5640 1 -- 172.21.15.90:0/2793130719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559697c39390 msgr2=0x559697ff1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:10.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.466+0000 7f5f40be5640 1 --2- 172.21.15.90:0/2793130719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559697c39390 0x559697ff1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.466+0000 7f5f40be5640 1 -- 172.21.15.90:0/2793130719 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559697c104d0 msgr2=0x559697fe5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:10.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.466+0000 7f5f40be5640 1 --2- 172.21.15.90:0/2793130719 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559697c104d0 0x559697fe5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.466+0000 7f5f40be5640 1 -- 172.21.15.90:0/2793130719 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559697c513f0 con 0x559697c186b0 2024-04-25T11:39:10.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.466+0000 7f5f40be5640 1 --2- 172.21.15.90:0/2793130719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559697c186b0 0x559697fe5740 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c010e70 tx=0x7f5f2c010ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b99929f706c251f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:10.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.466+0000 7f5f24ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:10.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.466+0000 7f5f24ff9640 1 -- 172.21.15.90:0/2793130719 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c027060 con 0x559697c186b0 2024-04-25T11:39:10.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.466+0000 7f5f24ff9640 1 -- 172.21.15.90:0/2793130719 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f5f2c018820 con 0x559697c186b0 2024-04-25T11:39:10.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.466+0000 7f5f4a5dd740 1 -- 172.21.15.90:0/2793130719 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f2c0047a0 con 0x559697c186b0 2024-04-25T11:39:10.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.467+0000 7f5f24ff9640 1 -- 172.21.15.90:0/2793130719 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5f2c014070 con 0x559697c186b0 2024-04-25T11:39:10.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.467+0000 7f5f4a5dd740 10 client.16782 fetch_fsmap learned FSMap version 27 2024-04-25T11:39:10.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.467+0000 7f5f4a5dd740 1 -- 172.21.15.90:0/2793130719 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559697d1b7d0 con 0x559697c186b0 2024-04-25T11:39:10.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.467+0000 7f5f24ff9640 1 -- 172.21.15.90:0/2793130719 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f2c0047a0 con 0x559697c186b0 2024-04-25T11:39:10.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.467+0000 7f5f24ff9640 1 -- 172.21.15.90:0/2793130719 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f5f2c022be0 con 0x559697c186b0 2024-04-25T11:39:10.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.467+0000 7f5f4a5dd740 10 client.16782 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:39:10.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.467+0000 7f5f4a5dd740 10 client.16782 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:39:10.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.467+0000 7f5f4a5dd740 20 client.16782 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:10.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.467+0000 7f5f4a5dd740 1 --2- 172.21.15.90:0/2793130719 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x559697bf7f60 0x559697ff7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.467+0000 7f5f4a5dd740 4 client.16782 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:39:10.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.467+0000 7f5f4a5dd740 1 -- 172.21.15.90:0/2793130719 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559697f4d990 con 0x559697bf7f60 2024-04-25T11:39:10.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.469+0000 7f5f413e6640 1 --2- 172.21.15.90:0/2793130719 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x559697bf7f60 0x559697ff7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.470+0000 7f5f413e6640 1 --2- 172.21.15.90:0/2793130719 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x559697bf7f60 0x559697ff7840 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cbb1483119335297 server_cookie=72ec77183950454 in_seq=0 out_seq=0 2024-04-25T11:39:10.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.470+0000 7f5f24ff9640 10 client.16782 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:39:10.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.475+0000 7f5f24ff9640 1 -- 172.21.15.90:0/2793130719 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559697f4d990 con 0x559697bf7f60 2024-04-25T11:39:10.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.475+0000 7f5f24ff9640 10 client.16782 handle_command_reply: tid=0 2024-04-25T11:39:10.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.475+0000 7f5f4a5dd740 1 client.16782 shutdown 2024-04-25T11:39:10.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.476+0000 7f5f4a5dd740 1 -- 172.21.15.90:0/2793130719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559697c186b0 msgr2=0x559697fe5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:10.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.476+0000 7f5f4a5dd740 1 --2- 172.21.15.90:0/2793130719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559697c186b0 0x559697fe5740 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c010e70 tx=0x7f5f2c010ea0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.476+0000 7f5f4a5dd740 1 -- 172.21.15.90:0/2793130719 shutdown_connections 2024-04-25T11:39:10.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.476+0000 7f5f4a5dd740 1 --2- 172.21.15.90:0/2793130719 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x559697bf7f60 0x559697ff7840 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.476+0000 7f5f4a5dd740 1 --2- 172.21.15.90:0/2793130719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559697c39390 0x559697ff1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.476+0000 7f5f4a5dd740 1 --2- 172.21.15.90:0/2793130719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559697c186b0 0x559697fe5740 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.476+0000 7f5f4a5dd740 1 --2- 172.21.15.90:0/2793130719 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559697c104d0 0x559697fe5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.476+0000 7f5f4a5dd740 1 -- 172.21.15.90:0/2793130719 >> 172.21.15.90:0/2793130719 conn(0x559697c2d4e0 msgr2=0x559697fdc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:10.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.476+0000 7f5f24ff9640 0 client.16782 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:39:10.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.476+0000 7f5f4a5dd740 1 -- 172.21.15.90:0/2793130719 shutdown_connections 2024-04-25T11:39:10.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.476+0000 7f5f4a5dd740 1 -- 172.21.15.90:0/2793130719 wait complete. 2024-04-25T11:39:10.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.476+0000 7f5f4a5dd740 20 client.16782 trim_cache size 0 max 16384 2024-04-25T11:39:10.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:10 smithi090 ceph-mon[29575]: pgmap v125: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 1.0 MiB/s rd, 399 MiB/s wr, 475 op/s 2024-04-25T11:39:10.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.501+0000 7f5f4a5dd740 1 Processor -- start 2024-04-25T11:39:10.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.501+0000 7f5f4a5dd740 1 -- start start 2024-04-25T11:39:10.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.502+0000 7f5f4a5dd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559697c104d0 0x559697fe4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.502+0000 7f5f4a5dd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559697c186b0 0x559697ff0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.502+0000 7f5f4a5dd740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559697c39390 0x559697fdf430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.502+0000 7f5f4a5dd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5f2000ea50 con 0x559697c186b0 2024-04-25T11:39:10.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.502+0000 7f5f4a5dd740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5f200105c0 con 0x559697c39390 2024-04-25T11:39:10.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.502+0000 7f5f4a5dd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5f2000f120 con 0x559697c104d0 2024-04-25T11:39:10.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.502+0000 7f5f40be5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559697c186b0 0x559697ff0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.502+0000 7f5f40be5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559697c186b0 0x559697ff0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42192/0 (socket says 172.21.15.90:42192) 2024-04-25T11:39:10.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.502+0000 7f5f40be5640 1 -- 172.21.15.90:0/1231802125 learned_addr learned my addr 172.21.15.90:0/1231802125 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:10.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.502+0000 7f5f413e6640 1 --2- 172.21.15.90:0/1231802125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559697c104d0 0x559697fe4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.502+0000 7f5f40be5640 1 -- 172.21.15.90:0/1231802125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559697c104d0 msgr2=0x559697fe4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:10.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.502+0000 7f5f40be5640 1 --2- 172.21.15.90:0/1231802125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559697c104d0 0x559697fe4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.502+0000 7f5f40be5640 1 -- 172.21.15.90:0/1231802125 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559697c39390 msgr2=0x559697fdf430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:10.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.502+0000 7f5f40be5640 1 --2- 172.21.15.90:0/1231802125 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559697c39390 0x559697fdf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.502+0000 7f5f40be5640 1 -- 172.21.15.90:0/1231802125 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559697fe42f0 con 0x559697c186b0 2024-04-25T11:39:10.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.503+0000 7f5f40be5640 1 --2- 172.21.15.90:0/1231802125 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559697c186b0 0x559697ff0ee0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c021530 tx=0x7f5f2c005f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45edfbad7e95d6fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:10.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.503+0000 7f5f26ffd640 1 -- 172.21.15.90:0/1231802125 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c027070 con 0x559697c186b0 2024-04-25T11:39:10.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.503+0000 7f5f26ffd640 1 -- 172.21.15.90:0/1231802125 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f2c005040 con 0x559697c186b0 2024-04-25T11:39:10.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.503+0000 7f5f26ffd640 1 -- 172.21.15.90:0/1231802125 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c0049e0 con 0x559697c186b0 2024-04-25T11:39:10.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.504+0000 7f5f4a5dd740 1 -- 172.21.15.90:0/1231802125 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559697c186b0 msgr2=0x559697ff0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:10.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.504+0000 7f5f4a5dd740 1 --2- 172.21.15.90:0/1231802125 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559697c186b0 0x559697ff0ee0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c021530 tx=0x7f5f2c005f80 comp rx=0 tx=0).stop 2024-04-25T11:39:10.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.504+0000 7f5f4a5dd740 1 -- 172.21.15.90:0/1231802125 shutdown_connections 2024-04-25T11:39:10.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.504+0000 7f5f4a5dd740 1 --2- 172.21.15.90:0/1231802125 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559697c39390 0x559697fdf430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.504+0000 7f5f4a5dd740 1 --2- 172.21.15.90:0/1231802125 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559697c186b0 0x559697ff0ee0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.504+0000 7f5f4a5dd740 1 --2- 172.21.15.90:0/1231802125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559697c104d0 0x559697fe4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.504+0000 7f5f4a5dd740 1 -- 172.21.15.90:0/1231802125 >> 172.21.15.90:0/1231802125 conn(0x559697c2d4e0 msgr2=0x559697fe4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:10.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.504+0000 7f5f4a5dd740 1 -- 172.21.15.90:0/1231802125 shutdown_connections 2024-04-25T11:39:10.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.504+0000 7f5f4a5dd740 1 -- 172.21.15.90:0/1231802125 wait complete. 2024-04-25T11:39:10.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.505+0000 7f5f4a5dd740 1 -- start start 2024-04-25T11:39:10.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.505+0000 7f5f4a5dd740 1 Processor -- start 2024-04-25T11:39:10.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.505+0000 7f5f4a5dd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559697c104d0 0x559697fe8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.506+0000 7f5f4a5dd740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559697c186b0 0x559697fe8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.506+0000 7f5f4a5dd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559697c39390 0x559697fe8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.506+0000 7f5f4a5dd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5f2000fec0 con 0x559697c104d0 2024-04-25T11:39:10.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.506+0000 7f5f4a5dd740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5f2000f7f0 con 0x559697c186b0 2024-04-25T11:39:10.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.506+0000 7f5f4a5dd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5f2000ca30 con 0x559697c39390 2024-04-25T11:39:10.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.506+0000 7f5f4a5dd740 10 client.0 _ll_register_callbacks cb 0x7f5f2c00ca90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:10.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.506+0000 7f5f41be7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559697c39390 0x559697fe8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.506+0000 7f5f41be7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559697c39390 0x559697fe8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:48746/0 (socket says 172.21.15.90:48746) 2024-04-25T11:39:10.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.506+0000 7f5f41be7640 1 -- 172.21.15.90:0/2163632002 learned_addr learned my addr 172.21.15.90:0/2163632002 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:10.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.506+0000 7f5f40be5640 1 --2- 172.21.15.90:0/2163632002 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559697c186b0 0x559697fe8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.507+0000 7f5f413e6640 1 --2- 172.21.15.90:0/2163632002 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559697c104d0 0x559697fe8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.507+0000 7f5f40be5640 1 -- 172.21.15.90:0/2163632002 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559697c39390 msgr2=0x559697fe8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:10.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.507+0000 7f5f40be5640 1 --2- 172.21.15.90:0/2163632002 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559697c39390 0x559697fe8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.507+0000 7f5f40be5640 1 -- 172.21.15.90:0/2163632002 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559697c104d0 msgr2=0x559697fe8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:10.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.507+0000 7f5f40be5640 1 --2- 172.21.15.90:0/2163632002 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559697c104d0 0x559697fe8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.507+0000 7f5f40be5640 1 -- 172.21.15.90:0/2163632002 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559697fedc70 con 0x559697c186b0 2024-04-25T11:39:10.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.507+0000 7f5f413e6640 1 --2- 172.21.15.90:0/2163632002 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559697c104d0 0x559697fe8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:39:10.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.508+0000 7f5f40be5640 1 --2- 172.21.15.90:0/2163632002 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559697c186b0 0x559697fe8760 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c0183e0 tx=0x7f5f2c0042d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71ace1d5538b4510 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:10.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.508+0000 7f5f24ff9640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:39:10.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.509+0000 7f5f24ff9640 1 -- 172.21.15.90:0/2163632002 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c041060 con 0x559697c186b0 2024-04-25T11:39:10.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.509+0000 7f5f4a5dd740 1 -- 172.21.15.90:0/2163632002 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f2c005040 con 0x559697c186b0 2024-04-25T11:39:10.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.509+0000 7f5f24ff9640 1 -- 172.21.15.90:0/2163632002 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f5f2c004990 con 0x559697c186b0 2024-04-25T11:39:10.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.509+0000 7f5f24ff9640 1 -- 172.21.15.90:0/2163632002 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5f2c027030 con 0x559697c186b0 2024-04-25T11:39:10.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.509+0000 7f5f4a5dd740 10 client.25997 fetch_fsmap learned FSMap version 27 2024-04-25T11:39:10.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.509+0000 7f5f4a5dd740 1 -- 172.21.15.90:0/2163632002 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55969805ebc0 con 0x559697c186b0 2024-04-25T11:39:10.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.510+0000 7f5f24ff9640 1 -- 172.21.15.90:0/2163632002 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f2c005040 con 0x559697c186b0 2024-04-25T11:39:10.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.511+0000 7f5f24ff9640 1 -- 172.21.15.90:0/2163632002 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f5f2c004bd0 con 0x559697c186b0 2024-04-25T11:39:10.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.511+0000 7f5f4a5dd740 10 client.25997 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:39:10.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.511+0000 7f5f4a5dd740 10 client.25997 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:39:10.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.511+0000 7f5f4a5dd740 20 client.25997 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:10.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.511+0000 7f5f4a5dd740 1 --2- 172.21.15.90:0/2163632002 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x559697bcd3a0 0x559698061250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.511+0000 7f5f4a5dd740 4 client.25997 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:39:10.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.511+0000 7f5f4a5dd740 1 -- 172.21.15.90:0/2163632002 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559697f47800 con 0x559697bcd3a0 2024-04-25T11:39:10.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.513+0000 7f5f413e6640 1 --2- 172.21.15.90:0/2163632002 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x559697bcd3a0 0x559698061250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.517+0000 7f5f24ff9640 10 client.25997 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:39:10.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.517+0000 7f5f413e6640 1 --2- 172.21.15.90:0/2163632002 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x559697bcd3a0 0x559698061250 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1fdc1672d51bb893 server_cookie=53aa055eafe6453b in_seq=0 out_seq=0 2024-04-25T11:39:10.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.518+0000 7f5f24ff9640 1 -- 172.21.15.90:0/2163632002 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559697f47800 con 0x559697bcd3a0 2024-04-25T11:39:10.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.518+0000 7f5f24ff9640 10 client.25997 handle_command_reply: tid=0 2024-04-25T11:39:10.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.518+0000 7f5f4a5dd740 1 client.25997 shutdown 2024-04-25T11:39:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.519+0000 7f5f4a5dd740 1 -- 172.21.15.90:0/2163632002 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559697c186b0 msgr2=0x559697fe8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.519+0000 7f5f4a5dd740 1 --2- 172.21.15.90:0/2163632002 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559697c186b0 0x559697fe8760 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c0183e0 tx=0x7f5f2c0042d0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.519+0000 7f5f4a5dd740 1 -- 172.21.15.90:0/2163632002 shutdown_connections 2024-04-25T11:39:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.519+0000 7f5f4a5dd740 1 --2- 172.21.15.90:0/2163632002 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x559697bcd3a0 0x559698061250 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.519+0000 7f5f4a5dd740 1 --2- 172.21.15.90:0/2163632002 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559697c39390 0x559697fe8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.519+0000 7f5f4a5dd740 1 --2- 172.21.15.90:0/2163632002 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559697c186b0 0x559697fe8760 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.519+0000 7f5f4a5dd740 1 --2- 172.21.15.90:0/2163632002 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559697c104d0 0x559697fe8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.519+0000 7f5f4a5dd740 1 -- 172.21.15.90:0/2163632002 >> 172.21.15.90:0/2163632002 conn(0x559697c2d4e0 msgr2=0x559697fd6760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:10.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.519+0000 7f5f24ff9640 0 client.25997 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:39:10.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.520+0000 7f5f4a5dd740 1 -- 172.21.15.90:0/2163632002 shutdown_connections 2024-04-25T11:39:10.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.520+0000 7f5f4a5dd740 1 -- 172.21.15.90:0/2163632002 wait complete. 2024-04-25T11:39:10.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.520+0000 7f5f4a5dd740 20 client.25997 trim_cache size 0 max 16384 2024-04-25T11:39:10.509 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:39:10.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.521+0000 7f5f4364a640 1 -- 172.21.15.90:0/2456565545 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5f280afce0 msgr2=0x7f5f280b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:10.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.521+0000 7f5f4364a640 1 --2- 172.21.15.90:0/2456565545 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5f280afce0 0x7f5f280b21f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x559697fd7980 tx=0x7f5f3400aa40 comp rx=0 tx=0).stop 2024-04-25T11:39:10.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.521+0000 7f5f4364a640 1 -- 172.21.15.90:0/2456565545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f3c102400 msgr2=0x7f5f3c0ff030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:10.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.521+0000 7f5f4364a640 1 --2- 172.21.15.90:0/2456565545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f3c102400 0x7f5f3c0ff030 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f5f3800b450 tx=0x7f5f3800b910 comp rx=0 tx=0).stop 2024-04-25T11:39:10.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.521+0000 7f5f4364a640 1 -- 172.21.15.90:0/2456565545 shutdown_connections 2024-04-25T11:39:10.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.521+0000 7f5f4364a640 1 --2- 172.21.15.90:0/2456565545 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5f280afce0 0x7f5f280b21f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.521+0000 7f5f4364a640 1 --2- 172.21.15.90:0/2456565545 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5f3c107450 0x7f5f3c0ffab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.521+0000 7f5f4364a640 1 --2- 172.21.15.90:0/2456565545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f3c106990 0x7f5f3c0ff570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.521+0000 7f5f4364a640 1 --2- 172.21.15.90:0/2456565545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f3c102400 0x7f5f3c0ff030 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.521+0000 7f5f4364a640 1 -- 172.21.15.90:0/2456565545 >> 172.21.15.90:0/2456565545 conn(0x7f5f3c077880 msgr2=0x7f5f3c103db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:10.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.521+0000 7f5f4364a640 1 -- 172.21.15.90:0/2456565545 shutdown_connections 2024-04-25T11:39:10.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.521+0000 7f5f4364a640 1 -- 172.21.15.90:0/2456565545 wait complete. 2024-04-25T11:39:10.530 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T11:39:10.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.790+0000 7fb0d0e0c640 1 -- 172.21.15.90:0/2904146902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0cc10e550 msgr2=0x7fb0cc110970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:10.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.790+0000 7fb0d0e0c640 1 --2- 172.21.15.90:0/2904146902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0cc10e550 0x7fb0cc110970 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4009ca0 tx=0x7fb0c402a8e0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.790+0000 7fb0d0e0c640 1 -- 172.21.15.90:0/2904146902 shutdown_connections 2024-04-25T11:39:10.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.790+0000 7fb0d0e0c640 1 --2- 172.21.15.90:0/2904146902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0cc10e550 0x7fb0cc110970 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.790+0000 7fb0d0e0c640 1 --2- 172.21.15.90:0/2904146902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0cc102050 0x7fb0cc102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.790+0000 7fb0d0e0c640 1 --2- 172.21.15.90:0/2904146902 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb0cc109ba0 0x7fb0cc101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.790+0000 7fb0d0e0c640 1 -- 172.21.15.90:0/2904146902 >> 172.21.15.90:0/2904146902 conn(0x7fb0cc077880 msgr2=0x7fb0cc077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:10.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.790+0000 7fb0d0e0c640 1 -- 172.21.15.90:0/2904146902 shutdown_connections 2024-04-25T11:39:10.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.790+0000 7fb0d0e0c640 1 -- 172.21.15.90:0/2904146902 wait complete. 2024-04-25T11:39:10.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.792+0000 7fb0d0e0c640 1 Processor -- start 2024-04-25T11:39:10.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.792+0000 7fb0d0e0c640 1 -- start start 2024-04-25T11:39:10.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.793+0000 7fb0d0e0c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0cc102050 0x7fb0cc1a0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.793+0000 7fb0d0e0c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb0cc109ba0 0x7fb0cc1a0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.793+0000 7fb0d0e0c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0cc10e550 0x7fb0cc19a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.793+0000 7fb0d0e0c640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb0cc113e40 con 0x7fb0cc102050 2024-04-25T11:39:10.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.793+0000 7fb0d0e0c640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb0cc113cc0 con 0x7fb0cc109ba0 2024-04-25T11:39:10.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.793+0000 7fb0d0e0c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb0cc113fc0 con 0x7fb0cc10e550 2024-04-25T11:39:10.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.793+0000 7fb0ca59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0cc102050 0x7fb0cc1a0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.793+0000 7fb0ca59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0cc102050 0x7fb0cc1a0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42214/0 (socket says 172.21.15.90:42214) 2024-04-25T11:39:10.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.793+0000 7fb0ca59c640 1 -- 172.21.15.90:0/2311136815 learned_addr learned my addr 172.21.15.90:0/2311136815 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:10.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.793+0000 7fb0ca59c640 1 -- 172.21.15.90:0/2311136815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0cc10e550 msgr2=0x7fb0cc19a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:10.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.793+0000 7fb0ca59c640 1 --2- 172.21.15.90:0/2311136815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0cc10e550 0x7fb0cc19a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.793+0000 7fb0ca59c640 1 -- 172.21.15.90:0/2311136815 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb0cc109ba0 msgr2=0x7fb0cc1a0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:10.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.793+0000 7fb0ca59c640 1 --2- 172.21.15.90:0/2311136815 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb0cc109ba0 0x7fb0cc1a0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.793+0000 7fb0ca59c640 1 -- 172.21.15.90:0/2311136815 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0cc19adf0 con 0x7fb0cc102050 2024-04-25T11:39:10.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.794+0000 7fb0ca59c640 1 --2- 172.21.15.90:0/2311136815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0cc102050 0x7fb0cc1a0540 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc00b450 tx=0x7fb0bc00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:10.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.794+0000 7fb0bb7fe640 1 -- 172.21.15.90:0/2311136815 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc015020 con 0x7fb0cc102050 2024-04-25T11:39:10.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.794+0000 7fb0bb7fe640 1 -- 172.21.15.90:0/2311136815 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0bc011d40 con 0x7fb0cc102050 2024-04-25T11:39:10.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.794+0000 7fb0bb7fe640 1 -- 172.21.15.90:0/2311136815 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc00f340 con 0x7fb0cc102050 2024-04-25T11:39:10.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.794+0000 7fb0d0e0c640 1 -- 172.21.15.90:0/2311136815 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0cc19b080 con 0x7fb0cc102050 2024-04-25T11:39:10.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.794+0000 7fb0d0e0c640 1 -- 172.21.15.90:0/2311136815 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0cc1a72e0 con 0x7fb0cc102050 2024-04-25T11:39:10.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.798+0000 7fb0d7d99740 1 Processor -- start 2024-04-25T11:39:10.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.798+0000 7fb0d7d99740 1 -- start start 2024-04-25T11:39:10.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.799+0000 7fb0d7d99740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e8641390 0x55c8e89e2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.799+0000 7fb0d7d99740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e86184d0 0x55c8e89e2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.799+0000 7fb0d7d99740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e86206b0 0x55c8e89e3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.799+0000 7fb0d7d99740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c8e8947d80 con 0x55c8e86184d0 2024-04-25T11:39:10.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.799+0000 7fb0d7d99740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c8e8901a20 con 0x55c8e86206b0 2024-04-25T11:39:10.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.799+0000 7fb0d7d99740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c8e8943d40 con 0x55c8e8641390 2024-04-25T11:39:10.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.800+0000 7fb0bb7fe640 1 -- 172.21.15.90:0/2311136815 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fb0bc018020 con 0x7fb0cc102050 2024-04-25T11:39:10.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.800+0000 7fb0ca59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e86184d0 0x55c8e89e2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.800+0000 7fb0ca59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e86184d0 0x55c8e89e2bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42226/0 (socket says 172.21.15.90:42226) 2024-04-25T11:39:10.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.800+0000 7fb0ca59c640 1 -- 172.21.15.90:0/2816713840 learned_addr learned my addr 172.21.15.90:0/2816713840 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:10.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.800+0000 7fb0cad9d640 1 --2- 172.21.15.90:0/2816713840 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e8641390 0x55c8e89e2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.800+0000 7fb0ca59c640 1 -- 172.21.15.90:0/2816713840 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e8641390 msgr2=0x55c8e89e2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:10.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.800+0000 7fb0ca59c640 1 --2- 172.21.15.90:0/2816713840 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e8641390 0x55c8e89e2680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.800+0000 7fb0ca59c640 1 -- 172.21.15.90:0/2816713840 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e86206b0 msgr2=0x55c8e89e3100 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:39:10.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.800+0000 7fb0ca59c640 1 --2- 172.21.15.90:0/2816713840 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e86206b0 0x55c8e89e3100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.800+0000 7fb0ca59c640 1 -- 172.21.15.90:0/2816713840 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8e890cc30 con 0x55c8e86184d0 2024-04-25T11:39:10.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.801+0000 7fb0ca59c640 1 --2- 172.21.15.90:0/2816713840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e86184d0 0x55c8e89e2bc0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc0dc200 tx=0x7fb0bc0e75c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1353bcf16fe79ba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:10.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.801+0000 7fb0b8ff9640 1 -- 172.21.15.90:0/2816713840 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc0f2070 con 0x55c8e86184d0 2024-04-25T11:39:10.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.801+0000 7fb0b8ff9640 1 -- 172.21.15.90:0/2816713840 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0bc0eba80 con 0x55c8e86184d0 2024-04-25T11:39:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.801+0000 7fb0b8ff9640 1 -- 172.21.15.90:0/2816713840 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc0e6040 con 0x55c8e86184d0 2024-04-25T11:39:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.802+0000 7fb0bb7fe640 1 --2- 172.21.15.90:0/2311136815 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb0b40afd60 0x7fb0b40b2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.802+0000 7fb0cad9d640 1 --2- 172.21.15.90:0/2311136815 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb0b40afd60 0x7fb0b40b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.802+0000 7fb0bb7fe640 1 -- 172.21.15.90:0/2311136815 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb0bc0deb20 con 0x7fb0cc102050 2024-04-25T11:39:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.803+0000 7fb0cad9d640 1 --2- 172.21.15.90:0/2311136815 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb0b40afd60 0x7fb0b40b2270 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4009a60 tx=0x7fb0c4032fe0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.803+0000 7fb0d7d99740 1 -- 172.21.15.90:0/2816713840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e86184d0 msgr2=0x55c8e89e2bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.803+0000 7fb0d7d99740 1 --2- 172.21.15.90:0/2816713840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e86184d0 0x55c8e89e2bc0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc0dc200 tx=0x7fb0bc0e75c0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.803+0000 7fb0d7d99740 1 -- 172.21.15.90:0/2816713840 shutdown_connections 2024-04-25T11:39:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.804+0000 7fb0d7d99740 1 --2- 172.21.15.90:0/2816713840 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e86206b0 0x55c8e89e3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.804+0000 7fb0d7d99740 1 --2- 172.21.15.90:0/2816713840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e86184d0 0x55c8e89e2bc0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.804+0000 7fb0d7d99740 1 --2- 172.21.15.90:0/2816713840 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e8641390 0x55c8e89e2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.804+0000 7fb0d7d99740 1 -- 172.21.15.90:0/2816713840 >> 172.21.15.90:0/2816713840 conn(0x55c8e86354e0 msgr2=0x55c8e89defc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:10.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.804+0000 7fb0d7d99740 1 -- 172.21.15.90:0/2816713840 shutdown_connections 2024-04-25T11:39:10.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.804+0000 7fb0d7d99740 1 -- 172.21.15.90:0/2816713840 wait complete. 2024-04-25T11:39:10.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.805+0000 7fb0d7d99740 1 -- start start 2024-04-25T11:39:10.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.805+0000 7fb0d7d99740 1 Processor -- start 2024-04-25T11:39:10.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.806+0000 7fb0d7d99740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e86184d0 0x55c8e89f21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.806+0000 7fb0d7d99740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e86206b0 0x55c8e89f2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.806+0000 7fb0d7d99740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e8641390 0x55c8e89ed620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.807+0000 7fb0cad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e8641390 0x55c8e89ed620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.807+0000 7fb0cad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e8641390 0x55c8e89ed620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42228/0 (socket says 172.21.15.90:42228) 2024-04-25T11:39:10.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.807+0000 7fb0ca59c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e86184d0 0x55c8e89f21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.807+0000 7fb0cad9d640 1 -- 172.21.15.90:0/1249809676 learned_addr learned my addr 172.21.15.90:0/1249809676 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:10.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.807+0000 7fb0d7d99740 1 -- 172.21.15.90:0/1249809676 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c8e89e9d40 con 0x55c8e8641390 2024-04-25T11:39:10.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.807+0000 7fb0d7d99740 1 -- 172.21.15.90:0/1249809676 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c8e88cedc0 con 0x55c8e86184d0 2024-04-25T11:39:10.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.807+0000 7fb0d7d99740 1 -- 172.21.15.90:0/1249809676 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c8e89e9ec0 con 0x55c8e86206b0 2024-04-25T11:39:10.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.807+0000 7fb0d7d99740 10 client.0 _ll_register_callbacks cb 0x55c8e88ea600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:10.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.807+0000 7fb0c9d9b640 1 --2- 172.21.15.90:0/1249809676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e86206b0 0x55c8e89f2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.808+0000 7fb0cad9d640 1 -- 172.21.15.90:0/1249809676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e86206b0 msgr2=0x55c8e89f2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:10.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.808+0000 7fb0cad9d640 1 --2- 172.21.15.90:0/1249809676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e86206b0 0x55c8e89f2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.808+0000 7fb0cad9d640 1 -- 172.21.15.90:0/1249809676 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e86184d0 msgr2=0x55c8e89f21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:10.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.808+0000 7fb0cad9d640 1 --2- 172.21.15.90:0/1249809676 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e86184d0 0x55c8e89f21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.808+0000 7fb0cad9d640 1 -- 172.21.15.90:0/1249809676 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8e86593f0 con 0x55c8e8641390 2024-04-25T11:39:10.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.809+0000 7fb0cad9d640 1 --2- 172.21.15.90:0/1249809676 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e8641390 0x55c8e89ed620 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4009ca0 tx=0x7fb0c40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=793db69ba04a61b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:10.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.809+0000 7fb0b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:10.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.809+0000 7fb0b1ffb640 1 -- 172.21.15.90:0/1249809676 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c40305e0 con 0x55c8e8641390 2024-04-25T11:39:10.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.809+0000 7fb0b1ffb640 1 -- 172.21.15.90:0/1249809676 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb0c4030780 con 0x55c8e8641390 2024-04-25T11:39:10.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.809+0000 7fb0d7d99740 1 -- 172.21.15.90:0/1249809676 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0bc0eba80 con 0x55c8e8641390 2024-04-25T11:39:10.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.810+0000 7fb0b1ffb640 1 -- 172.21.15.90:0/1249809676 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb0c4030980 con 0x55c8e8641390 2024-04-25T11:39:10.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.810+0000 7fb0d7d99740 10 client.16800 fetch_fsmap learned FSMap version 27 2024-04-25T11:39:10.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.810+0000 7fb0d7d99740 1 -- 172.21.15.90:0/1249809676 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c8e87237d0 con 0x55c8e8641390 2024-04-25T11:39:10.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.810+0000 7fb0b1ffb640 1 -- 172.21.15.90:0/1249809676 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0bc0eba80 con 0x55c8e8641390 2024-04-25T11:39:10.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.810+0000 7fb0b1ffb640 1 -- 172.21.15.90:0/1249809676 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7fb0c404e070 con 0x55c8e8641390 2024-04-25T11:39:10.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.811+0000 7fb0d7d99740 10 client.16800 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:39:10.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.811+0000 7fb0d7d99740 10 client.16800 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:39:10.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.811+0000 7fb0d7d99740 20 client.16800 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:10.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.811+0000 7fb0d7d99740 1 --2- 172.21.15.90:0/1249809676 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55c8e85fff60 0x55c8e8a10a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.811+0000 7fb0d7d99740 4 client.16800 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:39:10.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.811+0000 7fb0d7d99740 1 -- 172.21.15.90:0/1249809676 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8e8955990 con 0x55c8e85fff60 2024-04-25T11:39:10.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.812+0000 7fb0ca59c640 1 --2- 172.21.15.90:0/1249809676 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55c8e85fff60 0x55c8e8a10a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.812+0000 7fb0b1ffb640 10 client.16800 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:39:10.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.812+0000 7fb0ca59c640 1 --2- 172.21.15.90:0/1249809676 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55c8e85fff60 0x55c8e8a10a40 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=485b10e303f85b6f server_cookie=36fc0fbae0e9f8c1 in_seq=0 out_seq=0 2024-04-25T11:39:10.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.814+0000 7fb0b1ffb640 1 -- 172.21.15.90:0/1249809676 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c8e8955990 con 0x55c8e85fff60 2024-04-25T11:39:10.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.814+0000 7fb0b1ffb640 10 client.16800 handle_command_reply: tid=0 2024-04-25T11:39:10.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.814+0000 7fb0d7d99740 1 client.16800 shutdown 2024-04-25T11:39:10.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.815+0000 7fb0d7d99740 1 -- 172.21.15.90:0/1249809676 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e8641390 msgr2=0x55c8e89ed620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:10.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.815+0000 7fb0d7d99740 1 --2- 172.21.15.90:0/1249809676 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e8641390 0x55c8e89ed620 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4009ca0 tx=0x7fb0c40095c0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.815+0000 7fb0d7d99740 1 -- 172.21.15.90:0/1249809676 shutdown_connections 2024-04-25T11:39:10.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.815+0000 7fb0d7d99740 1 --2- 172.21.15.90:0/1249809676 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e8641390 0x55c8e89ed620 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.815+0000 7fb0d7d99740 1 --2- 172.21.15.90:0/1249809676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e86206b0 0x55c8e89f2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.815+0000 7fb0d7d99740 1 --2- 172.21.15.90:0/1249809676 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55c8e85fff60 0x55c8e8a10a40 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.815+0000 7fb0d7d99740 1 --2- 172.21.15.90:0/1249809676 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e86184d0 0x55c8e89f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.815+0000 7fb0d7d99740 1 -- 172.21.15.90:0/1249809676 >> 172.21.15.90:0/1249809676 conn(0x55c8e86354e0 msgr2=0x55c8e89dd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:10.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.815+0000 7fb0b1ffb640 0 client.16800 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:39:10.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.815+0000 7fb0d7d99740 1 -- 172.21.15.90:0/1249809676 shutdown_connections 2024-04-25T11:39:10.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.816+0000 7fb0d7d99740 1 -- 172.21.15.90:0/1249809676 wait complete. 2024-04-25T11:39:10.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.816+0000 7fb0d7d99740 20 client.16800 trim_cache size 0 max 16384 2024-04-25T11:39:10.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.836+0000 7fb0d7d99740 1 Processor -- start 2024-04-25T11:39:10.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.836+0000 7fb0d7d99740 1 -- start start 2024-04-25T11:39:10.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.836+0000 7fb0d7d99740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e86184d0 0x55c8e89eb180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.837+0000 7fb0d7d99740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e86206b0 0x55c8e89e6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.837+0000 7fb0d7d99740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e8641390 0x55c8e89f1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.837+0000 7fb0d7d99740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb0cc1adb90 con 0x55c8e86184d0 2024-04-25T11:39:10.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.837+0000 7fb0d7d99740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb0cc1af700 con 0x55c8e8641390 2024-04-25T11:39:10.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.837+0000 7fb0c9d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e86206b0 0x55c8e89e6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.837+0000 7fb0c9d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e86206b0 0x55c8e89e6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:48788/0 (socket says 172.21.15.90:48788) 2024-04-25T11:39:10.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.837+0000 7fb0c9d9b640 1 -- 172.21.15.90:0/3542562303 learned_addr learned my addr 172.21.15.90:0/3542562303 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:10.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.837+0000 7fb0d7d99740 1 -- 172.21.15.90:0/3542562303 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb0cc1ae260 con 0x55c8e86206b0 2024-04-25T11:39:10.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.838+0000 7fb0cad9d640 1 --2- 172.21.15.90:0/3542562303 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e8641390 0x55c8e89f1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.838+0000 7fb0ca59c640 1 --2- 172.21.15.90:0/3542562303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e86184d0 0x55c8e89eb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.838+0000 7fb0ca59c640 1 -- 172.21.15.90:0/3542562303 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e86206b0 msgr2=0x55c8e89e6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:10.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.838+0000 7fb0ca59c640 1 --2- 172.21.15.90:0/3542562303 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e86206b0 0x55c8e89e6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.839+0000 7fb0ca59c640 1 -- 172.21.15.90:0/3542562303 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e8641390 msgr2=0x55c8e89f1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:10.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.839+0000 7fb0ca59c640 1 --2- 172.21.15.90:0/3542562303 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e8641390 0x55c8e89f1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.839+0000 7fb0ca59c640 1 -- 172.21.15.90:0/3542562303 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8e89da7f0 con 0x55c8e86184d0 2024-04-25T11:39:10.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.839+0000 7fb0ca59c640 1 --2- 172.21.15.90:0/3542562303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e86184d0 0x55c8e89eb180 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc00be10 tx=0x7fb0bc0108d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39ea7820b9405b3f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:10.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.843+0000 7fb0b3fff640 1 -- 172.21.15.90:0/3542562303 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc0a2bd0 con 0x55c8e86184d0 2024-04-25T11:39:10.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.844+0000 7fb0b3fff640 1 -- 172.21.15.90:0/3542562303 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0bc0a6820 con 0x55c8e86184d0 2024-04-25T11:39:10.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.844+0000 7fb0b3fff640 1 -- 172.21.15.90:0/3542562303 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc0a6480 con 0x55c8e86184d0 2024-04-25T11:39:10.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.845+0000 7fb0d7d99740 1 -- 172.21.15.90:0/3542562303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e86184d0 msgr2=0x55c8e89eb180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:10.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.845+0000 7fb0d7d99740 1 --2- 172.21.15.90:0/3542562303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e86184d0 0x55c8e89eb180 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc00be10 tx=0x7fb0bc0108d0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.845+0000 7fb0d7d99740 1 -- 172.21.15.90:0/3542562303 shutdown_connections 2024-04-25T11:39:10.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.845+0000 7fb0d7d99740 1 --2- 172.21.15.90:0/3542562303 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e8641390 0x55c8e89f1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.845+0000 7fb0d7d99740 1 --2- 172.21.15.90:0/3542562303 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e86206b0 0x55c8e89e6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.845+0000 7fb0d7d99740 1 --2- 172.21.15.90:0/3542562303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e86184d0 0x55c8e89eb180 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.845+0000 7fb0d7d99740 1 -- 172.21.15.90:0/3542562303 >> 172.21.15.90:0/3542562303 conn(0x55c8e86354e0 msgr2=0x55c8e89ebef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:10.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.855+0000 7fb0d7d99740 1 -- 172.21.15.90:0/3542562303 shutdown_connections 2024-04-25T11:39:10.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.855+0000 7fb0d7d99740 1 -- 172.21.15.90:0/3542562303 wait complete. 2024-04-25T11:39:10.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.856+0000 7fb0d7d99740 1 -- start start 2024-04-25T11:39:10.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.856+0000 7fb0d7d99740 1 Processor -- start 2024-04-25T11:39:10.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.857+0000 7fb0d7d99740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e86184d0 0x55c8e8a21b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.857+0000 7fb0d7d99740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e86206b0 0x55c8e8a22050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.857+0000 7fb0d7d99740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e8641390 0x55c8e8a151f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.857+0000 7fb0d7d99740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb0cc1af000 con 0x55c8e86184d0 2024-04-25T11:39:10.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.857+0000 7fb0d7d99740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb0cc1ae930 con 0x55c8e86206b0 2024-04-25T11:39:10.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.857+0000 7fb0d7d99740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb0cc1abb70 con 0x55c8e8641390 2024-04-25T11:39:10.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.857+0000 7fb0d7d99740 10 client.0 _ll_register_callbacks cb 0x7fb0c4031470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:10.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.857+0000 7fb0ca59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e86184d0 0x55c8e8a21b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.857+0000 7fb0ca59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e86184d0 0x55c8e8a21b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42250/0 (socket says 172.21.15.90:42250) 2024-04-25T11:39:10.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.857+0000 7fb0ca59c640 1 -- 172.21.15.90:0/139708135 learned_addr learned my addr 172.21.15.90:0/139708135 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:10.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.858+0000 7fb0c9d9b640 1 --2- 172.21.15.90:0/139708135 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e86206b0 0x55c8e8a22050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.858+0000 7fb0cad9d640 1 --2- 172.21.15.90:0/139708135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e8641390 0x55c8e8a151f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.858+0000 7fb0ca59c640 1 -- 172.21.15.90:0/139708135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e8641390 msgr2=0x55c8e8a151f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:10.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.858+0000 7fb0ca59c640 1 --2- 172.21.15.90:0/139708135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e8641390 0x55c8e8a151f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.858+0000 7fb0ca59c640 1 -- 172.21.15.90:0/139708135 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e86206b0 msgr2=0x55c8e8a22050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:10.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.858+0000 7fb0ca59c640 1 --2- 172.21.15.90:0/139708135 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e86206b0 0x55c8e8a22050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.858+0000 7fb0ca59c640 1 -- 172.21.15.90:0/139708135 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8e8a15730 con 0x55c8e86184d0 2024-04-25T11:39:10.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.859+0000 7fb0ca59c640 1 --2- 172.21.15.90:0/139708135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e86184d0 0x55c8e8a21b10 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc0a6ca0 tx=0x7fb0bc0e2c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e1bb91483bab69d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:10.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.859+0000 7fb0b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:10.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.860+0000 7fb0b1ffb640 1 -- 172.21.15.90:0/139708135 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc08c020 con 0x55c8e86184d0 2024-04-25T11:39:10.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.860+0000 7fb0b1ffb640 1 -- 172.21.15.90:0/139708135 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb0bc0a3300 con 0x55c8e86184d0 2024-04-25T11:39:10.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.865+0000 7fb0d7d99740 1 -- 172.21.15.90:0/139708135 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0bc0a6820 con 0x55c8e86184d0 2024-04-25T11:39:10.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.869+0000 7fb0b1ffb640 1 -- 172.21.15.90:0/139708135 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb0bc0a6660 con 0x55c8e86184d0 2024-04-25T11:39:10.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.869+0000 7fb0d7d99740 10 client.16806 fetch_fsmap learned FSMap version 27 2024-04-25T11:39:10.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.869+0000 7fb0d7d99740 1 -- 172.21.15.90:0/139708135 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c8e8a89070 con 0x55c8e86184d0 2024-04-25T11:39:10.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.878+0000 7fb0b1ffb640 1 -- 172.21.15.90:0/139708135 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0bc0a6820 con 0x55c8e86184d0 2024-04-25T11:39:10.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.879+0000 7fb0b1ffb640 1 -- 172.21.15.90:0/139708135 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7fb0bc0dc430 con 0x55c8e86184d0 2024-04-25T11:39:10.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.880+0000 7fb0d7d99740 10 client.16806 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:39:10.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.880+0000 7fb0d7d99740 10 client.16806 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:39:10.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.880+0000 7fb0d7d99740 20 client.16806 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:10.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.880+0000 7fb0d7d99740 1 --2- 172.21.15.90:0/139708135 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55c8e85d53a0 0x55c8e8a8b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:10.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.880+0000 7fb0d7d99740 4 client.16806 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:39:10.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.880+0000 7fb0d7d99740 1 -- 172.21.15.90:0/139708135 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c8e894f800 con 0x55c8e85d53a0 2024-04-25T11:39:10.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.883+0000 7fb0c9d9b640 1 --2- 172.21.15.90:0/139708135 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55c8e85d53a0 0x55c8e8a8b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:10.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.888+0000 7fb0c9d9b640 1 --2- 172.21.15.90:0/139708135 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55c8e85d53a0 0x55c8e8a8b250 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a74cc49124c9bdb6 server_cookie=5f2b98c4d1c8602a in_seq=0 out_seq=0 2024-04-25T11:39:10.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.888+0000 7fb0b1ffb640 10 client.16806 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:39:10.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.888+0000 7fb0b1ffb640 1 -- 172.21.15.90:0/139708135 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c8e894f800 con 0x55c8e85d53a0 2024-04-25T11:39:10.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.888+0000 7fb0b1ffb640 10 client.16806 handle_command_reply: tid=0 2024-04-25T11:39:10.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.889+0000 7fb0d7d99740 1 client.16806 shutdown 2024-04-25T11:39:10.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.889+0000 7fb0d7d99740 1 -- 172.21.15.90:0/139708135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e86184d0 msgr2=0x55c8e8a21b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:10.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.889+0000 7fb0d7d99740 1 --2- 172.21.15.90:0/139708135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e86184d0 0x55c8e8a21b10 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc0a6ca0 tx=0x7fb0bc0e2c20 comp rx=0 tx=0).stop 2024-04-25T11:39:10.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.889+0000 7fb0d7d99740 1 -- 172.21.15.90:0/139708135 shutdown_connections 2024-04-25T11:39:10.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.889+0000 7fb0d7d99740 1 --2- 172.21.15.90:0/139708135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e8641390 0x55c8e8a151f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.889+0000 7fb0d7d99740 1 --2- 172.21.15.90:0/139708135 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55c8e85d53a0 0x55c8e8a8b250 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.889+0000 7fb0d7d99740 1 --2- 172.21.15.90:0/139708135 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e86206b0 0x55c8e8a22050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.889+0000 7fb0d7d99740 1 --2- 172.21.15.90:0/139708135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e86184d0 0x55c8e8a21b10 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.889+0000 7fb0d7d99740 1 -- 172.21.15.90:0/139708135 >> 172.21.15.90:0/139708135 conn(0x55c8e86354e0 msgr2=0x55c8e89de980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:10.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.889+0000 7fb0b1ffb640 0 client.16806 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:39:10.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.889+0000 7fb0d7d99740 1 -- 172.21.15.90:0/139708135 shutdown_connections 2024-04-25T11:39:10.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.889+0000 7fb0d7d99740 1 -- 172.21.15.90:0/139708135 wait complete. 2024-04-25T11:39:10.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.890+0000 7fb0d7d99740 20 client.16806 trim_cache size 0 max 16384 2024-04-25T11:39:10.880 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:39:10.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.895+0000 7fb0d0e0c640 1 -- 172.21.15.90:0/2311136815 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb0b40afd60 msgr2=0x7fb0b40b2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:10.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.895+0000 7fb0d0e0c640 1 --2- 172.21.15.90:0/2311136815 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb0b40afd60 0x7fb0b40b2270 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4009a60 tx=0x7fb0c4032fe0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.895+0000 7fb0d0e0c640 1 -- 172.21.15.90:0/2311136815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0cc102050 msgr2=0x7fb0cc1a0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:10.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.895+0000 7fb0d0e0c640 1 --2- 172.21.15.90:0/2311136815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0cc102050 0x7fb0cc1a0540 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc00b450 tx=0x7fb0bc00b910 comp rx=0 tx=0).stop 2024-04-25T11:39:10.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.896+0000 7fb0d0e0c640 1 -- 172.21.15.90:0/2311136815 shutdown_connections 2024-04-25T11:39:10.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.896+0000 7fb0d0e0c640 1 --2- 172.21.15.90:0/2311136815 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb0b40afd60 0x7fb0b40b2270 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.896+0000 7fb0d0e0c640 1 --2- 172.21.15.90:0/2311136815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0cc10e550 0x7fb0cc19a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.897+0000 7fb0d0e0c640 1 --2- 172.21.15.90:0/2311136815 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb0cc109ba0 0x7fb0cc1a0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.897+0000 7fb0d0e0c640 1 --2- 172.21.15.90:0/2311136815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0cc102050 0x7fb0cc1a0540 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.897+0000 7fb0d0e0c640 1 -- 172.21.15.90:0/2311136815 >> 172.21.15.90:0/2311136815 conn(0x7fb0cc077880 msgr2=0x7fb0cc10a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:10.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.897+0000 7fb0d0e0c640 1 -- 172.21.15.90:0/2311136815 shutdown_connections 2024-04-25T11:39:10.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:10.897+0000 7fb0d0e0c640 1 -- 172.21.15.90:0/2311136815 wait complete. 2024-04-25T11:39:10.924 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T11:39:11.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.187+0000 7f1620dfe640 1 -- 172.21.15.90:0/671254418 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f161c11d120 msgr2=0x7f161c11f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:11.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.187+0000 7f1620dfe640 1 --2- 172.21.15.90:0/671254418 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f161c11d120 0x7f161c11f540 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f160c009900 tx=0x7f160c02a4d0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.188+0000 7f1620dfe640 1 -- 172.21.15.90:0/671254418 shutdown_connections 2024-04-25T11:39:11.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.188+0000 7f1620dfe640 1 --2- 172.21.15.90:0/671254418 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f161c11d120 0x7f161c11f540 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.188+0000 7f1620dfe640 1 --2- 172.21.15.90:0/671254418 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f161c11a7c0 0x7f161c11cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.188+0000 7f1620dfe640 1 --2- 172.21.15.90:0/671254418 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f161c10a0b0 0x7f161c10a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.188+0000 7f1620dfe640 1 -- 172.21.15.90:0/671254418 >> 172.21.15.90:0/671254418 conn(0x7f161c06e7a0 msgr2=0x7f161c06ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:11.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.197+0000 7f1620dfe640 1 -- 172.21.15.90:0/671254418 shutdown_connections 2024-04-25T11:39:11.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.197+0000 7f1620dfe640 1 -- 172.21.15.90:0/671254418 wait complete. 2024-04-25T11:39:11.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.199+0000 7f1620dfe640 1 Processor -- start 2024-04-25T11:39:11.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.199+0000 7f1620dfe640 1 -- start start 2024-04-25T11:39:11.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.200+0000 7f1620dfe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f161c10a0b0 0x7f161c1a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.200+0000 7f1620dfe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f161c11a7c0 0x7f161c1a9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.200+0000 7f1620dfe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f161c11d120 0x7f161c1adce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.200+0000 7f1620dfe640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f161c122b80 con 0x7f161c11a7c0 2024-04-25T11:39:11.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.200+0000 7f1620dfe640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f161c122a00 con 0x7f161c11d120 2024-04-25T11:39:11.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.200+0000 7f1620dfe640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f161c122d00 con 0x7f161c10a0b0 2024-04-25T11:39:11.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.201+0000 7f1619d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f161c11a7c0 0x7f161c1a9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.201+0000 7f1619d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f161c11a7c0 0x7f161c1a9a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42272/0 (socket says 172.21.15.90:42272) 2024-04-25T11:39:11.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.201+0000 7f1619d9b640 1 -- 172.21.15.90:0/4219505416 learned_addr learned my addr 172.21.15.90:0/4219505416 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:11.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.202+0000 7f1619d9b640 1 -- 172.21.15.90:0/4219505416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f161c10a0b0 msgr2=0x7f161c1a94c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:39:11.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.202+0000 7f1619d9b640 1 --2- 172.21.15.90:0/4219505416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f161c10a0b0 0x7f161c1a94c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.202+0000 7f1619d9b640 1 -- 172.21.15.90:0/4219505416 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f161c11d120 msgr2=0x7f161c1adce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:11.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.202+0000 7f1619d9b640 1 --2- 172.21.15.90:0/4219505416 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f161c11d120 0x7f161c1adce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.202+0000 7f1619d9b640 1 -- 172.21.15.90:0/4219505416 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f161c1ae320 con 0x7f161c11a7c0 2024-04-25T11:39:11.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.204+0000 7f1619d9b640 1 --2- 172.21.15.90:0/4219505416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f161c11a7c0 0x7f161c1a9a20 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f161400eab0 tx=0x7f161400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:11.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.205+0000 7f16137fe640 1 -- 172.21.15.90:0/4219505416 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f161400ce40 con 0x7f161c11a7c0 2024-04-25T11:39:11.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.205+0000 7f1620dfe640 1 -- 172.21.15.90:0/4219505416 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f161c1ae610 con 0x7f161c11a7c0 2024-04-25T11:39:11.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.206+0000 7f1620dfe640 1 -- 172.21.15.90:0/4219505416 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f161c1b5f90 con 0x7f161c11a7c0 2024-04-25T11:39:11.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.206+0000 7f16137fe640 1 -- 172.21.15.90:0/4219505416 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1614011920 con 0x7f161c11a7c0 2024-04-25T11:39:11.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.206+0000 7f16137fe640 1 -- 172.21.15.90:0/4219505416 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1614010990 con 0x7f161c11a7c0 2024-04-25T11:39:11.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.211+0000 7f16137fe640 1 -- 172.21.15.90:0/4219505416 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f1614011470 con 0x7f161c11a7c0 2024-04-25T11:39:11.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.213+0000 7f16137fe640 1 --2- 172.21.15.90:0/4219505416 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f16080afe70 0x7f16080b2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.214+0000 7f16137fe640 1 -- 172.21.15.90:0/4219505416 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f16140df6a0 con 0x7f161c11a7c0 2024-04-25T11:39:11.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.217+0000 7f161a59c640 1 --2- 172.21.15.90:0/4219505416 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f16080afe70 0x7f16080b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.218+0000 7f1627d83740 1 Processor -- start 2024-04-25T11:39:11.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.219+0000 7f1627d83740 1 -- start start 2024-04-25T11:39:11.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.219+0000 7f1627d83740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558961794390 0x558961b35810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.219+0000 7f1627d83740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55896176b4d0 0x558961b35d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.219+0000 7f1627d83740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 0x558961b426a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.219+0000 7f1627d83740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558961a9ad80 con 0x5589617736b0 2024-04-25T11:39:11.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.219+0000 7f1627d83740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x558961a54a20 con 0x558961794390 2024-04-25T11:39:11.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.219+0000 7f1627d83740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558961a96d40 con 0x55896176b4d0 2024-04-25T11:39:11.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.220+0000 7f161ad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 0x558961b426a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.220+0000 7f161ad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 0x558961b426a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42282/0 (socket says 172.21.15.90:42282) 2024-04-25T11:39:11.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.220+0000 7f161ad9d640 1 -- 172.21.15.90:0/1939292764 learned_addr learned my addr 172.21.15.90:0/1939292764 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:11.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.221+0000 7f1619d9b640 1 --2- 172.21.15.90:0/1939292764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55896176b4d0 0x558961b35d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.221+0000 7f161ad9d640 1 -- 172.21.15.90:0/1939292764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55896176b4d0 msgr2=0x558961b35d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:11.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.221+0000 7f161ad9d640 1 --2- 172.21.15.90:0/1939292764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55896176b4d0 0x558961b35d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.221+0000 7f161ad9d640 1 -- 172.21.15.90:0/1939292764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558961794390 msgr2=0x558961b35810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:39:11.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.221+0000 7f161ad9d640 1 --2- 172.21.15.90:0/1939292764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558961794390 0x558961b35810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.221+0000 7f161ad9d640 1 -- 172.21.15.90:0/1939292764 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558961a5fc30 con 0x5589617736b0 2024-04-25T11:39:11.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.222+0000 7f161ad9d640 1 --2- 172.21.15.90:0/1939292764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 0x558961b426a0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f160c005ab0 tx=0x7f160c005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4a95745a09f3537 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:11.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.222+0000 7f1610ff9640 1 -- 172.21.15.90:0/1939292764 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f160c062030 con 0x5589617736b0 2024-04-25T11:39:11.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.222+0000 7f1610ff9640 1 -- 172.21.15.90:0/1939292764 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f160c0047c0 con 0x5589617736b0 2024-04-25T11:39:11.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.223+0000 7f1610ff9640 1 -- 172.21.15.90:0/1939292764 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f160c062030 con 0x5589617736b0 2024-04-25T11:39:11.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.224+0000 7f1627d83740 1 -- 172.21.15.90:0/1939292764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 msgr2=0x558961b426a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:11.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.224+0000 7f1627d83740 1 --2- 172.21.15.90:0/1939292764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 0x558961b426a0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f160c005ab0 tx=0x7f160c005f30 comp rx=0 tx=0).stop 2024-04-25T11:39:11.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.224+0000 7f1627d83740 1 -- 172.21.15.90:0/1939292764 shutdown_connections 2024-04-25T11:39:11.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.224+0000 7f1627d83740 1 --2- 172.21.15.90:0/1939292764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 0x558961b426a0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.224+0000 7f1627d83740 1 --2- 172.21.15.90:0/1939292764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55896176b4d0 0x558961b35d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.224+0000 7f1627d83740 1 --2- 172.21.15.90:0/1939292764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558961794390 0x558961b35810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.224+0000 7f1627d83740 1 -- 172.21.15.90:0/1939292764 >> 172.21.15.90:0/1939292764 conn(0x5589617884e0 msgr2=0x558961b31fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:11.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.228+0000 7f1627d83740 1 -- 172.21.15.90:0/1939292764 shutdown_connections 2024-04-25T11:39:11.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.230+0000 7f1627d83740 1 -- 172.21.15.90:0/1939292764 wait complete. 2024-04-25T11:39:11.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.230+0000 7f161a59c640 1 --2- 172.21.15.90:0/4219505416 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f16080afe70 0x7f16080b2380 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x558961b32980 tx=0x7f1604006bd0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:11.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.231+0000 7f1627d83740 1 -- start start 2024-04-25T11:39:11.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.231+0000 7f1627d83740 1 Processor -- start 2024-04-25T11:39:11.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.232+0000 7f1627d83740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55896176b4d0 0x558961b3ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.232+0000 7f1627d83740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 0x558961b40450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.233+0000 7f1627d83740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558961794390 0x558961b4c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.233+0000 7f1627d83740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558961b42d80 con 0x5589617736b0 2024-04-25T11:39:11.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.233+0000 7f1619d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 0x558961b40450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.233+0000 7f1619d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 0x558961b40450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42292/0 (socket says 172.21.15.90:42292) 2024-04-25T11:39:11.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.233+0000 7f1619d9b640 1 -- 172.21.15.90:0/990114509 learned_addr learned my addr 172.21.15.90:0/990114509 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:11.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.233+0000 7f1627d83740 1 -- 172.21.15.90:0/990114509 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x558961a21dc0 con 0x558961794390 2024-04-25T11:39:11.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.233+0000 7f1627d83740 1 -- 172.21.15.90:0/990114509 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558961b42f00 con 0x55896176b4d0 2024-04-25T11:39:11.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.233+0000 7f1627d83740 10 client.0 _ll_register_callbacks cb 0x558961a3d600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:11.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.234+0000 7f161ad9d640 1 --2- 172.21.15.90:0/990114509 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558961794390 0x558961b4c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.234+0000 7f161a59c640 1 --2- 172.21.15.90:0/990114509 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55896176b4d0 0x558961b3ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.234+0000 7f1619d9b640 1 -- 172.21.15.90:0/990114509 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55896176b4d0 msgr2=0x558961b3ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:11.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.234+0000 7f1619d9b640 1 --2- 172.21.15.90:0/990114509 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55896176b4d0 0x558961b3ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.234+0000 7f1619d9b640 1 -- 172.21.15.90:0/990114509 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558961794390 msgr2=0x558961b4c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:11.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.234+0000 7f1619d9b640 1 --2- 172.21.15.90:0/990114509 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558961794390 0x558961b4c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.234+0000 7f1619d9b640 1 -- 172.21.15.90:0/990114509 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589617ac3f0 con 0x5589617736b0 2024-04-25T11:39:11.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.235+0000 7f1619d9b640 1 --2- 172.21.15.90:0/990114509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 0x558961b40450 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f16140dcd90 tx=0x7f16140a7350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71366c4913266479 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:11.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.235+0000 7f1601ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:11.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.235+0000 7f1601ffb640 1 -- 172.21.15.90:0/990114509 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16140204c0 con 0x5589617736b0 2024-04-25T11:39:11.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.235+0000 7f1601ffb640 1 -- 172.21.15.90:0/990114509 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f16140a3d70 con 0x5589617736b0 2024-04-25T11:39:11.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.235+0000 7f1627d83740 1 -- 172.21.15.90:0/990114509 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f160c0047c0 con 0x5589617736b0 2024-04-25T11:39:11.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.236+0000 7f1601ffb640 1 -- 172.21.15.90:0/990114509 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f16140abb30 con 0x5589617736b0 2024-04-25T11:39:11.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.236+0000 7f1627d83740 10 client.16818 fetch_fsmap learned FSMap version 27 2024-04-25T11:39:11.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.236+0000 7f1627d83740 1 -- 172.21.15.90:0/990114509 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5589618767d0 con 0x5589617736b0 2024-04-25T11:39:11.226 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:11 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2981923313' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:39:11.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.237+0000 7f1601ffb640 1 -- 172.21.15.90:0/990114509 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f160c0047c0 con 0x5589617736b0 2024-04-25T11:39:11.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.237+0000 7f1601ffb640 1 -- 172.21.15.90:0/990114509 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f161400f040 con 0x5589617736b0 2024-04-25T11:39:11.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.237+0000 7f1627d83740 10 client.16818 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:39:11.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.237+0000 7f1627d83740 10 client.16818 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:39:11.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.237+0000 7f1627d83740 20 client.16818 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:11.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.237+0000 7f1627d83740 1 --2- 172.21.15.90:0/990114509 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x558961752f60 0x558961b52670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.237+0000 7f1627d83740 4 client.16818 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:39:11.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.237+0000 7f1627d83740 1 -- 172.21.15.90:0/990114509 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558961aa8990 con 0x558961752f60 2024-04-25T11:39:11.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.237+0000 7f161a59c640 1 --2- 172.21.15.90:0/990114509 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x558961752f60 0x558961b52670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.243+0000 7f1601ffb640 10 client.16818 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:39:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.243+0000 7f161a59c640 1 --2- 172.21.15.90:0/990114509 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x558961752f60 0x558961b52670 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1255251c5e9aa432 server_cookie=ad51c876f33b965f in_seq=0 out_seq=0 2024-04-25T11:39:11.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.248+0000 7f1601ffb640 1 -- 172.21.15.90:0/990114509 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558961aa8990 con 0x558961752f60 2024-04-25T11:39:11.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.248+0000 7f1601ffb640 10 client.16818 handle_command_reply: tid=0 2024-04-25T11:39:11.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.248+0000 7f1627d83740 1 client.16818 shutdown 2024-04-25T11:39:11.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.248+0000 7f1627d83740 1 -- 172.21.15.90:0/990114509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 msgr2=0x558961b40450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:11.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.248+0000 7f1627d83740 1 --2- 172.21.15.90:0/990114509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 0x558961b40450 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f16140dcd90 tx=0x7f16140a7350 comp rx=0 tx=0).stop 2024-04-25T11:39:11.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.248+0000 7f1627d83740 1 -- 172.21.15.90:0/990114509 shutdown_connections 2024-04-25T11:39:11.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.248+0000 7f1627d83740 1 --2- 172.21.15.90:0/990114509 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x558961752f60 0x558961b52670 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.248+0000 7f1601ffb640 0 client.16818 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:39:11.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.248+0000 7f1627d83740 1 --2- 172.21.15.90:0/990114509 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558961794390 0x558961b4c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.248+0000 7f1627d83740 1 --2- 172.21.15.90:0/990114509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 0x558961b40450 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.248+0000 7f1627d83740 1 --2- 172.21.15.90:0/990114509 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55896176b4d0 0x558961b3ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.248+0000 7f1627d83740 1 -- 172.21.15.90:0/990114509 >> 172.21.15.90:0/990114509 conn(0x5589617884e0 msgr2=0x558961b37530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:11.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.248+0000 7f1627d83740 1 -- 172.21.15.90:0/990114509 shutdown_connections 2024-04-25T11:39:11.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.248+0000 7f1627d83740 1 -- 172.21.15.90:0/990114509 wait complete. 2024-04-25T11:39:11.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.249+0000 7f1627d83740 20 client.16818 trim_cache size 0 max 16384 2024-04-25T11:39:11.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.277+0000 7f1627d83740 1 Processor -- start 2024-04-25T11:39:11.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.277+0000 7f1627d83740 1 -- start start 2024-04-25T11:39:11.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.277+0000 7f1627d83740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55896176b4d0 0x558961b4beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.278+0000 7f1627d83740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 0x558961b37b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.278+0000 7f1627d83740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558961794390 0x558961b57e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.278+0000 7f1627d83740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f15fc00e940 con 0x5589617736b0 2024-04-25T11:39:11.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.278+0000 7f1627d83740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f15fc0104b0 con 0x55896176b4d0 2024-04-25T11:39:11.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.278+0000 7f1627d83740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f15fc00f010 con 0x558961794390 2024-04-25T11:39:11.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.278+0000 7f1619d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 0x558961b37b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.278+0000 7f1619d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 0x558961b37b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42300/0 (socket says 172.21.15.90:42300) 2024-04-25T11:39:11.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.278+0000 7f1619d9b640 1 -- 172.21.15.90:0/534384760 learned_addr learned my addr 172.21.15.90:0/534384760 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:11.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.278+0000 7f1619d9b640 1 -- 172.21.15.90:0/534384760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558961794390 msgr2=0x558961b57e70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:39:11.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.278+0000 7f1619d9b640 1 --2- 172.21.15.90:0/534384760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558961794390 0x558961b57e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.278+0000 7f1619d9b640 1 -- 172.21.15.90:0/534384760 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55896176b4d0 msgr2=0x558961b4beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:11.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.278+0000 7f1619d9b640 1 --2- 172.21.15.90:0/534384760 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55896176b4d0 0x558961b4beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.278+0000 7f1619d9b640 1 -- 172.21.15.90:0/534384760 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558961b3e460 con 0x5589617736b0 2024-04-25T11:39:11.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.279+0000 7f1619d9b640 1 --2- 172.21.15.90:0/534384760 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 0x558961b37b20 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f16140b02c0 tx=0x7f16140103d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8732bf5c0c99bad1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:11.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.279+0000 7f1603fff640 1 -- 172.21.15.90:0/534384760 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16140a7550 con 0x5589617736b0 2024-04-25T11:39:11.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.279+0000 7f1603fff640 1 -- 172.21.15.90:0/534384760 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16140b05e0 con 0x5589617736b0 2024-04-25T11:39:11.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.279+0000 7f1603fff640 1 -- 172.21.15.90:0/534384760 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1614011b90 con 0x5589617736b0 2024-04-25T11:39:11.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.280+0000 7f1627d83740 1 -- 172.21.15.90:0/534384760 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 msgr2=0x558961b37b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:11.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.280+0000 7f1627d83740 1 --2- 172.21.15.90:0/534384760 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 0x558961b37b20 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f16140b02c0 tx=0x7f16140103d0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.280+0000 7f1627d83740 1 -- 172.21.15.90:0/534384760 shutdown_connections 2024-04-25T11:39:11.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.280+0000 7f1627d83740 1 --2- 172.21.15.90:0/534384760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558961794390 0x558961b57e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.280+0000 7f1627d83740 1 --2- 172.21.15.90:0/534384760 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 0x558961b37b20 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.280+0000 7f1627d83740 1 --2- 172.21.15.90:0/534384760 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55896176b4d0 0x558961b4beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.280+0000 7f1627d83740 1 -- 172.21.15.90:0/534384760 >> 172.21.15.90:0/534384760 conn(0x5589617884e0 msgr2=0x558961b37410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:11.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.281+0000 7f1627d83740 1 -- 172.21.15.90:0/534384760 shutdown_connections 2024-04-25T11:39:11.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.281+0000 7f1627d83740 1 -- 172.21.15.90:0/534384760 wait complete. 2024-04-25T11:39:11.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.281+0000 7f1627d83740 1 -- start start 2024-04-25T11:39:11.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.282+0000 7f1627d83740 1 Processor -- start 2024-04-25T11:39:11.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.282+0000 7f1627d83740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55896176b4d0 0x558961b487b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.283+0000 7f1627d83740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 0x558961b45b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.283+0000 7f1627d83740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558961794390 0x558961b460b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.283+0000 7f1627d83740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f15fc00fdb0 con 0x5589617736b0 2024-04-25T11:39:11.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.283+0000 7f1627d83740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f15fc00f6e0 con 0x558961794390 2024-04-25T11:39:11.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.283+0000 7f1619d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 0x558961b45b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.283+0000 7f1627d83740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f15fc00c920 con 0x55896176b4d0 2024-04-25T11:39:11.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.283+0000 7f1619d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 0x558961b45b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42310/0 (socket says 172.21.15.90:42310) 2024-04-25T11:39:11.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.283+0000 7f1619d9b640 1 -- 172.21.15.90:0/2100000410 learned_addr learned my addr 172.21.15.90:0/2100000410 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:11.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.283+0000 7f1627d83740 10 client.0 _ll_register_callbacks cb 0x7f1614011760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:11.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.284+0000 7f161a59c640 1 --2- 172.21.15.90:0/2100000410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55896176b4d0 0x558961b487b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.284+0000 7f161ad9d640 1 --2- 172.21.15.90:0/2100000410 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558961794390 0x558961b460b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.284+0000 7f1619d9b640 1 -- 172.21.15.90:0/2100000410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55896176b4d0 msgr2=0x558961b487b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:11.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.284+0000 7f1619d9b640 1 --2- 172.21.15.90:0/2100000410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55896176b4d0 0x558961b487b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.284+0000 7f1619d9b640 1 -- 172.21.15.90:0/2100000410 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558961794390 msgr2=0x558961b460b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:11.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.284+0000 7f1619d9b640 1 --2- 172.21.15.90:0/2100000410 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558961794390 0x558961b460b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.284+0000 7f1619d9b640 1 -- 172.21.15.90:0/2100000410 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558961b467f0 con 0x5589617736b0 2024-04-25T11:39:11.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.286+0000 7f1619d9b640 1 --2- 172.21.15.90:0/2100000410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 0x558961b45b70 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f161400c700 tx=0x7f16140a6fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=314d7933e56b5ae7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:11.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.286+0000 7f1601ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:11.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.287+0000 7f1601ffb640 1 -- 172.21.15.90:0/2100000410 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1614011b90 con 0x5589617736b0 2024-04-25T11:39:11.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.287+0000 7f1601ffb640 1 -- 172.21.15.90:0/2100000410 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f16140c1070 con 0x5589617736b0 2024-04-25T11:39:11.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.287+0000 7f1627d83740 1 -- 172.21.15.90:0/2100000410 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16140b05e0 con 0x5589617736b0 2024-04-25T11:39:11.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.288+0000 7f1601ffb640 1 -- 172.21.15.90:0/2100000410 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1614011d30 con 0x5589617736b0 2024-04-25T11:39:11.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.288+0000 7f1627d83740 10 client.16824 fetch_fsmap learned FSMap version 27 2024-04-25T11:39:11.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.288+0000 7f1627d83740 1 -- 172.21.15.90:0/2100000410 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558961bba900 con 0x5589617736b0 2024-04-25T11:39:11.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.289+0000 7f1601ffb640 1 -- 172.21.15.90:0/2100000410 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16140b05e0 con 0x5589617736b0 2024-04-25T11:39:11.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.289+0000 7f1601ffb640 1 -- 172.21.15.90:0/2100000410 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f1614010b30 con 0x5589617736b0 2024-04-25T11:39:11.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.289+0000 7f1627d83740 10 client.16824 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:39:11.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.289+0000 7f1627d83740 10 client.16824 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:39:11.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.289+0000 7f1627d83740 20 client.16824 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:11.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.289+0000 7f1627d83740 1 --2- 172.21.15.90:0/2100000410 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x5589617283a0 0x558961bbcd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.289+0000 7f1627d83740 4 client.16824 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:39:11.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.289+0000 7f1627d83740 1 -- 172.21.15.90:0/2100000410 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558961aa2800 con 0x5589617283a0 2024-04-25T11:39:11.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.292+0000 7f161a59c640 1 --2- 172.21.15.90:0/2100000410 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x5589617283a0 0x558961bbcd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.296+0000 7f1601ffb640 10 client.16824 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:39:11.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.296+0000 7f161a59c640 1 --2- 172.21.15.90:0/2100000410 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x5589617283a0 0x558961bbcd40 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b4b11be1dc797bdd server_cookie=4fe89f9622e93bec in_seq=0 out_seq=0 2024-04-25T11:39:11.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.296+0000 7f1601ffb640 1 -- 172.21.15.90:0/2100000410 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558961aa2800 con 0x5589617283a0 2024-04-25T11:39:11.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.296+0000 7f1601ffb640 10 client.16824 handle_command_reply: tid=0 2024-04-25T11:39:11.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.297+0000 7f1627d83740 1 client.16824 shutdown 2024-04-25T11:39:11.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.297+0000 7f1627d83740 1 -- 172.21.15.90:0/2100000410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 msgr2=0x558961b45b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:11.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.297+0000 7f1627d83740 1 --2- 172.21.15.90:0/2100000410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 0x558961b45b70 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f161400c700 tx=0x7f16140a6fa0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.297+0000 7f1627d83740 1 -- 172.21.15.90:0/2100000410 shutdown_connections 2024-04-25T11:39:11.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.297+0000 7f1627d83740 1 --2- 172.21.15.90:0/2100000410 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x5589617283a0 0x558961bbcd40 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.297+0000 7f1601ffb640 0 client.16824 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:39:11.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.297+0000 7f1627d83740 1 --2- 172.21.15.90:0/2100000410 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558961794390 0x558961b460b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.297+0000 7f1627d83740 1 --2- 172.21.15.90:0/2100000410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5589617736b0 0x558961b45b70 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.297+0000 7f1627d83740 1 --2- 172.21.15.90:0/2100000410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55896176b4d0 0x558961b487b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.297+0000 7f1627d83740 1 -- 172.21.15.90:0/2100000410 >> 172.21.15.90:0/2100000410 conn(0x5589617884e0 msgr2=0x558961b37410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:11.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.297+0000 7f1627d83740 1 -- 172.21.15.90:0/2100000410 shutdown_connections 2024-04-25T11:39:11.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.297+0000 7f1627d83740 1 -- 172.21.15.90:0/2100000410 wait complete. 2024-04-25T11:39:11.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.298+0000 7f1627d83740 20 client.16824 trim_cache size 0 max 16384 2024-04-25T11:39:11.287 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:39:11.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.298+0000 7f1620dfe640 1 -- 172.21.15.90:0/4219505416 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f16080afe70 msgr2=0x7f16080b2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:11.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.298+0000 7f1620dfe640 1 --2- 172.21.15.90:0/4219505416 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f16080afe70 0x7f16080b2380 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x558961b32980 tx=0x7f1604006bd0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.298+0000 7f1620dfe640 1 -- 172.21.15.90:0/4219505416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f161c11a7c0 msgr2=0x7f161c1a9a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:11.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.298+0000 7f1620dfe640 1 --2- 172.21.15.90:0/4219505416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f161c11a7c0 0x7f161c1a9a20 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f161400eab0 tx=0x7f161400ef70 comp rx=0 tx=0).stop 2024-04-25T11:39:11.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.299+0000 7f1620dfe640 1 -- 172.21.15.90:0/4219505416 shutdown_connections 2024-04-25T11:39:11.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.299+0000 7f1620dfe640 1 --2- 172.21.15.90:0/4219505416 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f16080afe70 0x7f16080b2380 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.299+0000 7f1620dfe640 1 --2- 172.21.15.90:0/4219505416 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f161c11d120 0x7f161c1adce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.299+0000 7f1620dfe640 1 --2- 172.21.15.90:0/4219505416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f161c11a7c0 0x7f161c1a9a20 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.299+0000 7f1620dfe640 1 --2- 172.21.15.90:0/4219505416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f161c10a0b0 0x7f161c1a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.299+0000 7f1620dfe640 1 -- 172.21.15.90:0/4219505416 >> 172.21.15.90:0/4219505416 conn(0x7f161c06e7a0 msgr2=0x7f161c11abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:11.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.299+0000 7f1620dfe640 1 -- 172.21.15.90:0/4219505416 shutdown_connections 2024-04-25T11:39:11.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.299+0000 7f1620dfe640 1 -- 172.21.15.90:0/4219505416 wait complete. 2024-04-25T11:39:11.310 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T11:39:11.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:11 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2981923313' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:39:11.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.519+0000 7f5ef8cdd640 1 -- 172.21.15.90:0/3737273553 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ef41166a0 msgr2=0x7f5ef4118bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:11.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.519+0000 7f5ef8cdd640 1 --2- 172.21.15.90:0/3737273553 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ef41166a0 0x7f5ef4118bb0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8009900 tx=0x7f5ee802a440 comp rx=0 tx=0).stop 2024-04-25T11:39:11.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.522+0000 7f5ef8cdd640 1 -- 172.21.15.90:0/3737273553 shutdown_connections 2024-04-25T11:39:11.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.522+0000 7f5ef8cdd640 1 --2- 172.21.15.90:0/3737273553 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ef41166a0 0x7f5ef4118bb0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.522+0000 7f5ef8cdd640 1 --2- 172.21.15.90:0/3737273553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ef4113bf0 0x7f5ef4116010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.522+0000 7f5ef8cdd640 1 --2- 172.21.15.90:0/3737273553 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ef410a260 0x7f5ef410a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.522+0000 7f5ef8cdd640 1 -- 172.21.15.90:0/3737273553 >> 172.21.15.90:0/3737273553 conn(0x7f5ef406e5d0 msgr2=0x7f5ef406ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:11.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.523+0000 7f5ef8cdd640 1 -- 172.21.15.90:0/3737273553 shutdown_connections 2024-04-25T11:39:11.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.523+0000 7f5ef8cdd640 1 -- 172.21.15.90:0/3737273553 wait complete. 2024-04-25T11:39:11.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.525+0000 7f5ef8cdd640 1 Processor -- start 2024-04-25T11:39:11.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.526+0000 7f5ef8cdd640 1 -- start start 2024-04-25T11:39:11.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.529+0000 7f5ef8cdd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ef410a260 0x7f5ef4119720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.529+0000 7f5ef8cdd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ef4113bf0 0x7f5ef4119c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.529+0000 7f5ef8cdd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ef41166a0 0x7f5ef411a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.529+0000 7f5ef8cdd640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef411e670 con 0x7f5ef4113bf0 2024-04-25T11:39:11.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.529+0000 7f5ef8cdd640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef411e4f0 con 0x7f5ef41166a0 2024-04-25T11:39:11.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.529+0000 7f5ef8cdd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef411e7f0 con 0x7f5ef410a260 2024-04-25T11:39:11.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.530+0000 7f5ef2ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ef4113bf0 0x7f5ef4119c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.530+0000 7f5ef2ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ef4113bf0 0x7f5ef4119c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44438/0 (socket says 172.21.15.90:44438) 2024-04-25T11:39:11.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.530+0000 7f5ef2ffd640 1 -- 172.21.15.90:0/2411882486 learned_addr learned my addr 172.21.15.90:0/2411882486 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:11.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.531+0000 7f5ef2ffd640 1 -- 172.21.15.90:0/2411882486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ef410a260 msgr2=0x7f5ef4119720 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:39:11.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.531+0000 7f5ef2ffd640 1 --2- 172.21.15.90:0/2411882486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ef410a260 0x7f5ef4119720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.531+0000 7f5ef2ffd640 1 -- 172.21.15.90:0/2411882486 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ef41166a0 msgr2=0x7f5ef411a1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:39:11.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.531+0000 7f5ef2ffd640 1 --2- 172.21.15.90:0/2411882486 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ef41166a0 0x7f5ef411a1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.531+0000 7f5ef2ffd640 1 -- 172.21.15.90:0/2411882486 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ef41b5680 con 0x7f5ef4113bf0 2024-04-25T11:39:11.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.531+0000 7f5ef2ffd640 1 --2- 172.21.15.90:0/2411882486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ef4113bf0 0x7f5ef4119c60 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f5eec00b9b0 tx=0x7f5eec00be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:11.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.532+0000 7f5ef0ff9640 1 -- 172.21.15.90:0/2411882486 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eec019070 con 0x7f5ef4113bf0 2024-04-25T11:39:11.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.532+0000 7f5ef0ff9640 1 -- 172.21.15.90:0/2411882486 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5eec00cba0 con 0x7f5ef4113bf0 2024-04-25T11:39:11.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.532+0000 7f5ef0ff9640 1 -- 172.21.15.90:0/2411882486 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eec0125a0 con 0x7f5ef4113bf0 2024-04-25T11:39:11.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.532+0000 7f5ef8cdd640 1 -- 172.21.15.90:0/2411882486 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ef41b5910 con 0x7f5ef4113bf0 2024-04-25T11:39:11.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.532+0000 7f5ef8cdd640 1 -- 172.21.15.90:0/2411882486 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ef41b5e10 con 0x7f5ef4113bf0 2024-04-25T11:39:11.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.539+0000 7f5f00df3740 1 Processor -- start 2024-04-25T11:39:11.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.541+0000 7f5f00df3740 1 -- start start 2024-04-25T11:39:11.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.541+0000 7f5f00df3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd68390 0x55a7fd1099e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.541+0000 7f5f00df3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7fcd3f4d0 0x55a7fd109f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.541+0000 7f5f00df3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7fcd476b0 0x55a7fd116870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.541+0000 7f5f00df3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a7fd06ed80 con 0x55a7fcd68390 2024-04-25T11:39:11.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.541+0000 7f5f00df3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55a7fd028a20 con 0x55a7fcd3f4d0 2024-04-25T11:39:11.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.541+0000 7f5f00df3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55a7fd06ad40 con 0x55a7fcd476b0 2024-04-25T11:39:11.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.542+0000 7f5ef0ff9640 1 -- 172.21.15.90:0/2411882486 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f5eec01c070 con 0x7f5ef4113bf0 2024-04-25T11:39:11.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.544+0000 7f5ef0ff9640 1 --2- 172.21.15.90:0/2411882486 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5edc0afd50 0x7f5edc0b2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.544+0000 7f5ef37fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7fcd3f4d0 0x55a7fd109f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.544+0000 7f5ef0ff9640 1 -- 172.21.15.90:0/2411882486 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f5eec0de5b0 con 0x7f5ef4113bf0 2024-04-25T11:39:11.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.544+0000 7f5ef2ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7fcd476b0 0x55a7fd116870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.545+0000 7f5ef3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd68390 0x55a7fd1099e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.545+0000 7f5ef3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd68390 0x55a7fd1099e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44442/0 (socket says 172.21.15.90:44442) 2024-04-25T11:39:11.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.545+0000 7f5ef3fff640 1 -- 172.21.15.90:0/699411615 learned_addr learned my addr 172.21.15.90:0/699411615 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:11.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.546+0000 7f5ef3fff640 1 --2- 172.21.15.90:0/2411882486 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5edc0afd50 0x7f5edc0b2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.547+0000 7f5ef3fff640 1 -- 172.21.15.90:0/699411615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7fcd476b0 msgr2=0x55a7fd116870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:11.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.547+0000 7f5ef3fff640 1 --2- 172.21.15.90:0/699411615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7fcd476b0 0x55a7fd116870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.547+0000 7f5ef3fff640 1 -- 172.21.15.90:0/699411615 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7fcd3f4d0 msgr2=0x55a7fd109f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:11.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.547+0000 7f5ef3fff640 1 --2- 172.21.15.90:0/699411615 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7fcd3f4d0 0x55a7fd109f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.547+0000 7f5ef3fff640 1 -- 172.21.15.90:0/699411615 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7fd033c30 con 0x55a7fcd68390 2024-04-25T11:39:11.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.547+0000 7f5ef37fe640 1 --2- 172.21.15.90:0/699411615 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7fcd3f4d0 0x55a7fd109f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:39:11.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.548+0000 7f5ef3fff640 1 --2- 172.21.15.90:0/699411615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd68390 0x55a7fd1099e0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8005ab0 tx=0x7f5ee8003aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e6b5e435c467ced server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:11.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.548+0000 7f5ef3fff640 1 --2- 172.21.15.90:0/2411882486 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5edc0afd50 0x7f5edc0b2260 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5ee80057c0 tx=0x7f5ee8039000 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:11.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.548+0000 7f5ed9ffb640 1 -- 172.21.15.90:0/699411615 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee8035040 con 0x55a7fcd68390 2024-04-25T11:39:11.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.548+0000 7f5ed9ffb640 1 -- 172.21.15.90:0/699411615 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ee8004180 con 0x55a7fcd68390 2024-04-25T11:39:11.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.548+0000 7f5ed9ffb640 1 -- 172.21.15.90:0/699411615 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee8040420 con 0x55a7fcd68390 2024-04-25T11:39:11.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.550+0000 7f5f00df3740 1 -- 172.21.15.90:0/699411615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd68390 msgr2=0x55a7fd1099e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:11.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.550+0000 7f5f00df3740 1 --2- 172.21.15.90:0/699411615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd68390 0x55a7fd1099e0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8005ab0 tx=0x7f5ee8003aa0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.550+0000 7f5f00df3740 1 -- 172.21.15.90:0/699411615 shutdown_connections 2024-04-25T11:39:11.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.550+0000 7f5f00df3740 1 --2- 172.21.15.90:0/699411615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7fcd476b0 0x55a7fd116870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.550+0000 7f5f00df3740 1 --2- 172.21.15.90:0/699411615 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7fcd3f4d0 0x55a7fd109f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.550+0000 7f5f00df3740 1 --2- 172.21.15.90:0/699411615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd68390 0x55a7fd1099e0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.550+0000 7f5f00df3740 1 -- 172.21.15.90:0/699411615 >> 172.21.15.90:0/699411615 conn(0x55a7fcd5c4e0 msgr2=0x55a7fd106190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:11.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.551+0000 7f5f00df3740 1 -- 172.21.15.90:0/699411615 shutdown_connections 2024-04-25T11:39:11.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.551+0000 7f5f00df3740 1 -- 172.21.15.90:0/699411615 wait complete. 2024-04-25T11:39:11.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.552+0000 7f5f00df3740 1 -- start start 2024-04-25T11:39:11.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.552+0000 7f5f00df3740 1 Processor -- start 2024-04-25T11:39:11.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.553+0000 7f5f00df3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd3f4d0 0x55a7fd114500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.554+0000 7f5f00df3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7fcd476b0 0x55a7fd114a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.554+0000 7f5f00df3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7fcd68390 0x55a7fd120420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.554+0000 7f5f00df3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a7fd116fa0 con 0x55a7fcd3f4d0 2024-04-25T11:39:11.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.554+0000 7f5f00df3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55a7fcff5dc0 con 0x55a7fcd476b0 2024-04-25T11:39:11.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.554+0000 7f5f00df3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55a7fd117120 con 0x55a7fcd68390 2024-04-25T11:39:11.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.554+0000 7f5f00df3740 10 client.0 _ll_register_callbacks cb 0x55a7fd011600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:11.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.555+0000 7f5ef37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd3f4d0 0x55a7fd114500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.555+0000 7f5ef37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd3f4d0 0x55a7fd114500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44444/0 (socket says 172.21.15.90:44444) 2024-04-25T11:39:11.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.555+0000 7f5ef37fe640 1 -- 172.21.15.90:0/2931194285 learned_addr learned my addr 172.21.15.90:0/2931194285 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:11.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.555+0000 7f5ef3fff640 1 --2- 172.21.15.90:0/2931194285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7fcd68390 0x55a7fd120420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.556+0000 7f5ef37fe640 1 -- 172.21.15.90:0/2931194285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7fcd68390 msgr2=0x55a7fd120420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:11.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.556+0000 7f5ef37fe640 1 --2- 172.21.15.90:0/2931194285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7fcd68390 0x55a7fd120420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.556+0000 7f5ef37fe640 1 -- 172.21.15.90:0/2931194285 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7fcd476b0 msgr2=0x55a7fd114a40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:39:11.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.556+0000 7f5ef37fe640 1 --2- 172.21.15.90:0/2931194285 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7fcd476b0 0x55a7fd114a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.556+0000 7f5ef37fe640 1 -- 172.21.15.90:0/2931194285 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7fcd803f0 con 0x55a7fcd3f4d0 2024-04-25T11:39:11.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:11 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2981923313' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:39:11.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.563+0000 7f5ef37fe640 1 --2- 172.21.15.90:0/2931194285 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd3f4d0 0x55a7fd114500 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f5ee400bf30 tx=0x7f5ee400d5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=959a1d149b704a6c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:11.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.563+0000 7f5ed2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:11.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.564+0000 7f5ed2ffd640 1 -- 172.21.15.90:0/2931194285 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee4014430 con 0x55a7fcd3f4d0 2024-04-25T11:39:11.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.564+0000 7f5f00df3740 1 -- 172.21.15.90:0/2931194285 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ee8004180 con 0x55a7fcd3f4d0 2024-04-25T11:39:11.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.564+0000 7f5ed2ffd640 1 -- 172.21.15.90:0/2931194285 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f5ee40145d0 con 0x55a7fcd3f4d0 2024-04-25T11:39:11.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.564+0000 7f5ed2ffd640 1 -- 172.21.15.90:0/2931194285 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5ee4022050 con 0x55a7fcd3f4d0 2024-04-25T11:39:11.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.564+0000 7f5f00df3740 10 client.16836 fetch_fsmap learned FSMap version 27 2024-04-25T11:39:11.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.564+0000 7f5f00df3740 1 -- 172.21.15.90:0/2931194285 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a7fce4a7d0 con 0x55a7fcd3f4d0 2024-04-25T11:39:11.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.565+0000 7f5ed2ffd640 1 -- 172.21.15.90:0/2931194285 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ee8004180 con 0x55a7fcd3f4d0 2024-04-25T11:39:11.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.565+0000 7f5ed2ffd640 1 -- 172.21.15.90:0/2931194285 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f5ee4013070 con 0x55a7fcd3f4d0 2024-04-25T11:39:11.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.565+0000 7f5f00df3740 10 client.16836 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:39:11.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.565+0000 7f5f00df3740 10 client.16836 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:39:11.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.565+0000 7f5f00df3740 20 client.16836 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:11.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.565+0000 7f5f00df3740 1 --2- 172.21.15.90:0/2931194285 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55a7fcd26f60 0x55a7fd126dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.565+0000 7f5f00df3740 4 client.16836 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:39:11.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.565+0000 7f5f00df3740 1 -- 172.21.15.90:0/2931194285 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7fd07c990 con 0x55a7fcd26f60 2024-04-25T11:39:11.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.568+0000 7f5ef2ffd640 1 --2- 172.21.15.90:0/2931194285 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55a7fcd26f60 0x55a7fd126dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.570+0000 7f5ef2ffd640 1 --2- 172.21.15.90:0/2931194285 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55a7fcd26f60 0x55a7fd126dd0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7c1095e9d983db25 server_cookie=5b6229c0a5bf3920 in_seq=0 out_seq=0 2024-04-25T11:39:11.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.570+0000 7f5ed2ffd640 10 client.16836 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:39:11.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.572+0000 7f5ed2ffd640 1 -- 172.21.15.90:0/2931194285 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a7fd07c990 con 0x55a7fcd26f60 2024-04-25T11:39:11.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.572+0000 7f5ed2ffd640 10 client.16836 handle_command_reply: tid=0 2024-04-25T11:39:11.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.573+0000 7f5f00df3740 1 client.16836 shutdown 2024-04-25T11:39:11.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.575+0000 7f5f00df3740 1 -- 172.21.15.90:0/2931194285 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd3f4d0 msgr2=0x55a7fd114500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:11.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.575+0000 7f5f00df3740 1 --2- 172.21.15.90:0/2931194285 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd3f4d0 0x55a7fd114500 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f5ee400bf30 tx=0x7f5ee400d5f0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.580+0000 7f5f00df3740 1 -- 172.21.15.90:0/2931194285 shutdown_connections 2024-04-25T11:39:11.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.580+0000 7f5f00df3740 1 --2- 172.21.15.90:0/2931194285 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55a7fcd26f60 0x55a7fd126dd0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.580+0000 7f5f00df3740 1 --2- 172.21.15.90:0/2931194285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7fcd68390 0x55a7fd120420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.580+0000 7f5f00df3740 1 --2- 172.21.15.90:0/2931194285 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7fcd476b0 0x55a7fd114a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.580+0000 7f5ed2ffd640 0 client.16836 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:39:11.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.580+0000 7f5f00df3740 1 --2- 172.21.15.90:0/2931194285 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd3f4d0 0x55a7fd114500 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.580+0000 7f5f00df3740 1 -- 172.21.15.90:0/2931194285 >> 172.21.15.90:0/2931194285 conn(0x55a7fcd5c4e0 msgr2=0x55a7fd10b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:11.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.585+0000 7f5f00df3740 1 -- 172.21.15.90:0/2931194285 shutdown_connections 2024-04-25T11:39:11.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.587+0000 7f5f00df3740 1 -- 172.21.15.90:0/2931194285 wait complete. 2024-04-25T11:39:11.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.588+0000 7f5f00df3740 20 client.16836 trim_cache size 0 max 16384 2024-04-25T11:39:11.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.614+0000 7f5f00df3740 1 Processor -- start 2024-04-25T11:39:11.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.615+0000 7f5f00df3740 1 -- start start 2024-04-25T11:39:11.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.616+0000 7f5f00df3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7fcd3f4d0 0x55a7fd1135f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.616+0000 7f5f00df3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7fcd476b0 0x55a7fd116120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.616+0000 7f5f00df3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd68390 0x55a7fd120080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.616+0000 7f5f00df3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef41be8f0 con 0x55a7fcd68390 2024-04-25T11:39:11.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.616+0000 7f5f00df3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef41c0460 con 0x55a7fcd476b0 2024-04-25T11:39:11.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.616+0000 7f5f00df3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef41befc0 con 0x55a7fcd3f4d0 2024-04-25T11:39:11.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.616+0000 7f5ef3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd68390 0x55a7fd120080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.616+0000 7f5ef3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd68390 0x55a7fd120080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44452/0 (socket says 172.21.15.90:44452) 2024-04-25T11:39:11.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.616+0000 7f5ef3fff640 1 -- 172.21.15.90:0/1813394888 learned_addr learned my addr 172.21.15.90:0/1813394888 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:11.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.617+0000 7f5ef2ffd640 1 --2- 172.21.15.90:0/1813394888 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7fcd476b0 0x55a7fd116120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.617+0000 7f5ef3fff640 1 -- 172.21.15.90:0/1813394888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7fcd3f4d0 msgr2=0x55a7fd1135f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:39:11.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.617+0000 7f5ef3fff640 1 --2- 172.21.15.90:0/1813394888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7fcd3f4d0 0x55a7fd1135f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.617+0000 7f5ef3fff640 1 -- 172.21.15.90:0/1813394888 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7fcd476b0 msgr2=0x55a7fd116120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:11.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.617+0000 7f5ef3fff640 1 --2- 172.21.15.90:0/1813394888 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7fcd476b0 0x55a7fd116120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.617+0000 7f5ef3fff640 1 -- 172.21.15.90:0/1813394888 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7fd112840 con 0x55a7fcd68390 2024-04-25T11:39:11.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.618+0000 7f5ef3fff640 1 --2- 172.21.15.90:0/1813394888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd68390 0x55a7fd120080 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8005f70 tx=0x7f5ee802a440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efecf37b9b1a5c70 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:11.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.618+0000 7f5ed97fa640 1 -- 172.21.15.90:0/1813394888 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee8035040 con 0x55a7fcd68390 2024-04-25T11:39:11.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.619+0000 7f5ed97fa640 1 -- 172.21.15.90:0/1813394888 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ee8009780 con 0x55a7fcd68390 2024-04-25T11:39:11.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.619+0000 7f5ed97fa640 1 -- 172.21.15.90:0/1813394888 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee8004a80 con 0x55a7fcd68390 2024-04-25T11:39:11.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.620+0000 7f5f00df3740 1 -- 172.21.15.90:0/1813394888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd68390 msgr2=0x55a7fd120080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:11.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.620+0000 7f5f00df3740 1 --2- 172.21.15.90:0/1813394888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd68390 0x55a7fd120080 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8005f70 tx=0x7f5ee802a440 comp rx=0 tx=0).stop 2024-04-25T11:39:11.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.620+0000 7f5f00df3740 1 -- 172.21.15.90:0/1813394888 shutdown_connections 2024-04-25T11:39:11.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.620+0000 7f5f00df3740 1 --2- 172.21.15.90:0/1813394888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd68390 0x55a7fd120080 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.620+0000 7f5f00df3740 1 --2- 172.21.15.90:0/1813394888 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7fcd476b0 0x55a7fd116120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.620+0000 7f5f00df3740 1 --2- 172.21.15.90:0/1813394888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7fcd3f4d0 0x55a7fd1135f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.620+0000 7f5f00df3740 1 -- 172.21.15.90:0/1813394888 >> 172.21.15.90:0/1813394888 conn(0x55a7fcd5c4e0 msgr2=0x55a7fd115a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:11.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.621+0000 7f5f00df3740 1 -- 172.21.15.90:0/1813394888 shutdown_connections 2024-04-25T11:39:11.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.621+0000 7f5f00df3740 1 -- 172.21.15.90:0/1813394888 wait complete. 2024-04-25T11:39:11.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.621+0000 7f5f00df3740 1 -- start start 2024-04-25T11:39:11.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.622+0000 7f5f00df3740 1 Processor -- start 2024-04-25T11:39:11.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.622+0000 7f5f00df3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7fcd3f4d0 0x55a7fd117670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.622+0000 7f5f00df3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd476b0 0x55a7fd117bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.622+0000 7f5f00df3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7fcd68390 0x55a7fd18d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.622+0000 7f5f00df3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef41bfd60 con 0x55a7fcd476b0 2024-04-25T11:39:11.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.622+0000 7f5f00df3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef41bf690 con 0x55a7fcd3f4d0 2024-04-25T11:39:11.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.622+0000 7f5f00df3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef41bc8d0 con 0x55a7fcd68390 2024-04-25T11:39:11.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.622+0000 7f5f00df3740 10 client.0 _ll_register_callbacks cb 0x7f5ee4006f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:11.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.622+0000 7f5ef2ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd476b0 0x55a7fd117bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.623+0000 7f5ef2ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd476b0 0x55a7fd117bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44468/0 (socket says 172.21.15.90:44468) 2024-04-25T11:39:11.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.623+0000 7f5ef2ffd640 1 -- 172.21.15.90:0/172731271 learned_addr learned my addr 172.21.15.90:0/172731271 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:11.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.623+0000 7f5ef37fe640 1 --2- 172.21.15.90:0/172731271 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7fcd3f4d0 0x55a7fd117670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.623+0000 7f5ef3fff640 1 --2- 172.21.15.90:0/172731271 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7fcd68390 0x55a7fd18d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.623+0000 7f5ef2ffd640 1 -- 172.21.15.90:0/172731271 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7fcd68390 msgr2=0x55a7fd18d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:11.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.623+0000 7f5ef2ffd640 1 --2- 172.21.15.90:0/172731271 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7fcd68390 0x55a7fd18d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.623+0000 7f5ef2ffd640 1 -- 172.21.15.90:0/172731271 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7fcd3f4d0 msgr2=0x55a7fd117670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:11.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.623+0000 7f5ef2ffd640 1 --2- 172.21.15.90:0/172731271 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7fcd3f4d0 0x55a7fd117670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.623+0000 7f5ef2ffd640 1 -- 172.21.15.90:0/172731271 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7fd18d6b0 con 0x55a7fcd476b0 2024-04-25T11:39:11.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.624+0000 7f5ef2ffd640 1 --2- 172.21.15.90:0/172731271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd476b0 0x55a7fd117bb0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f5eec0a6730 tx=0x7f5eec090040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9181b5396f3d2feb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:11.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.624+0000 7f5ed2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:11.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.624+0000 7f5ed2ffd640 1 -- 172.21.15.90:0/172731271 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eec0c90b0 con 0x55a7fcd476b0 2024-04-25T11:39:11.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.624+0000 7f5ed2ffd640 1 -- 172.21.15.90:0/172731271 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f5eec0dbec0 con 0x55a7fcd476b0 2024-04-25T11:39:11.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.624+0000 7f5f00df3740 1 -- 172.21.15.90:0/172731271 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ee8009780 con 0x55a7fcd476b0 2024-04-25T11:39:11.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.625+0000 7f5ed2ffd640 1 -- 172.21.15.90:0/172731271 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5eec0a2a30 con 0x55a7fcd476b0 2024-04-25T11:39:11.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.625+0000 7f5f00df3740 10 client.16842 fetch_fsmap learned FSMap version 27 2024-04-25T11:39:11.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.625+0000 7f5f00df3740 1 -- 172.21.15.90:0/172731271 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a7fd18e7b0 con 0x55a7fcd476b0 2024-04-25T11:39:11.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.626+0000 7f5ed2ffd640 1 -- 172.21.15.90:0/172731271 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ee8009780 con 0x55a7fcd476b0 2024-04-25T11:39:11.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.626+0000 7f5ed2ffd640 1 -- 172.21.15.90:0/172731271 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f5eec0af170 con 0x55a7fcd476b0 2024-04-25T11:39:11.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.626+0000 7f5f00df3740 10 client.16842 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:39:11.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.626+0000 7f5f00df3740 10 client.16842 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:39:11.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.626+0000 7f5f00df3740 20 client.16842 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:11.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.626+0000 7f5f00df3740 1 --2- 172.21.15.90:0/172731271 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55a7fccfc3a0 0x55a7fd191060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.627+0000 7f5f00df3740 4 client.16842 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:39:11.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.627+0000 7f5f00df3740 1 -- 172.21.15.90:0/172731271 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a7fd076800 con 0x55a7fccfc3a0 2024-04-25T11:39:11.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.627+0000 7f5ef37fe640 1 --2- 172.21.15.90:0/172731271 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55a7fccfc3a0 0x55a7fd191060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.629+0000 7f5ed2ffd640 10 client.16842 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:39:11.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.629+0000 7f5ef37fe640 1 --2- 172.21.15.90:0/172731271 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55a7fccfc3a0 0x55a7fd191060 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8c7ee4068b5f9909 server_cookie=e9b62b5546d8c76f in_seq=0 out_seq=0 2024-04-25T11:39:11.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.629+0000 7f5ed2ffd640 1 -- 172.21.15.90:0/172731271 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a7fd076800 con 0x55a7fccfc3a0 2024-04-25T11:39:11.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.629+0000 7f5ed2ffd640 10 client.16842 handle_command_reply: tid=0 2024-04-25T11:39:11.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.629+0000 7f5f00df3740 1 client.16842 shutdown 2024-04-25T11:39:11.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.630+0000 7f5f00df3740 1 -- 172.21.15.90:0/172731271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd476b0 msgr2=0x55a7fd117bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:11.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.630+0000 7f5f00df3740 1 --2- 172.21.15.90:0/172731271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd476b0 0x55a7fd117bb0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f5eec0a6730 tx=0x7f5eec090040 comp rx=0 tx=0).stop 2024-04-25T11:39:11.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.630+0000 7f5f00df3740 1 -- 172.21.15.90:0/172731271 shutdown_connections 2024-04-25T11:39:11.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.630+0000 7f5f00df3740 1 --2- 172.21.15.90:0/172731271 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55a7fccfc3a0 0x55a7fd191060 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.630+0000 7f5f00df3740 1 --2- 172.21.15.90:0/172731271 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7fcd68390 0x55a7fd18d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.630+0000 7f5f00df3740 1 --2- 172.21.15.90:0/172731271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7fcd476b0 0x55a7fd117bb0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.630+0000 7f5f00df3740 1 --2- 172.21.15.90:0/172731271 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7fcd3f4d0 0x55a7fd117670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.630+0000 7f5f00df3740 1 -- 172.21.15.90:0/172731271 >> 172.21.15.90:0/172731271 conn(0x55a7fcd5c4e0 msgr2=0x55a7fd113350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:11.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.630+0000 7f5ed2ffd640 0 client.16842 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:39:11.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.630+0000 7f5f00df3740 1 -- 172.21.15.90:0/172731271 shutdown_connections 2024-04-25T11:39:11.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.630+0000 7f5f00df3740 1 -- 172.21.15.90:0/172731271 wait complete. 2024-04-25T11:39:11.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.630+0000 7f5f00df3740 20 client.16842 trim_cache size 0 max 16384 2024-04-25T11:39:11.620 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:39:11.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.631+0000 7f5ef8cdd640 1 -- 172.21.15.90:0/2411882486 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5edc0afd50 msgr2=0x7f5edc0b2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:11.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.631+0000 7f5ef8cdd640 1 --2- 172.21.15.90:0/2411882486 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5edc0afd50 0x7f5edc0b2260 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5ee80057c0 tx=0x7f5ee8039000 comp rx=0 tx=0).stop 2024-04-25T11:39:11.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.631+0000 7f5ef8cdd640 1 -- 172.21.15.90:0/2411882486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ef4113bf0 msgr2=0x7f5ef4119c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:11.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.631+0000 7f5ef8cdd640 1 --2- 172.21.15.90:0/2411882486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ef4113bf0 0x7f5ef4119c60 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f5eec00b9b0 tx=0x7f5eec00be70 comp rx=0 tx=0).stop 2024-04-25T11:39:11.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.631+0000 7f5ef8cdd640 1 -- 172.21.15.90:0/2411882486 shutdown_connections 2024-04-25T11:39:11.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.631+0000 7f5ef8cdd640 1 --2- 172.21.15.90:0/2411882486 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5edc0afd50 0x7f5edc0b2260 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.631+0000 7f5ef8cdd640 1 --2- 172.21.15.90:0/2411882486 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ef41166a0 0x7f5ef411a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.631+0000 7f5ef8cdd640 1 --2- 172.21.15.90:0/2411882486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ef4113bf0 0x7f5ef4119c60 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.631+0000 7f5ef8cdd640 1 --2- 172.21.15.90:0/2411882486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ef410a260 0x7f5ef4119720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.631+0000 7f5ef8cdd640 1 -- 172.21.15.90:0/2411882486 >> 172.21.15.90:0/2411882486 conn(0x7f5ef406e5d0 msgr2=0x7f5ef4073910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:11.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.631+0000 7f5ef8cdd640 1 -- 172.21.15.90:0/2411882486 shutdown_connections 2024-04-25T11:39:11.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.631+0000 7f5ef8cdd640 1 -- 172.21.15.90:0/2411882486 wait complete. 2024-04-25T11:39:11.651 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T11:39:11.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.953+0000 7f7a9359e640 1 -- 172.21.15.90:0/4083764862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a94113c80 msgr2=0x7f7a941160a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:11.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.953+0000 7f7a9359e640 1 --2- 172.21.15.90:0/4083764862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a94113c80 0x7f7a941160a0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f7a88009900 tx=0x7f7a8802a440 comp rx=0 tx=0).stop 2024-04-25T11:39:11.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.953+0000 7f7a9359e640 1 -- 172.21.15.90:0/4083764862 shutdown_connections 2024-04-25T11:39:11.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.953+0000 7f7a9359e640 1 --2- 172.21.15.90:0/4083764862 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7a941166c0 0x7f7a94118bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.953+0000 7f7a9359e640 1 --2- 172.21.15.90:0/4083764862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a94113c80 0x7f7a941160a0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.953+0000 7f7a9359e640 1 --2- 172.21.15.90:0/4083764862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a9410a260 0x7f7a9410a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.953+0000 7f7a9359e640 1 -- 172.21.15.90:0/4083764862 >> 172.21.15.90:0/4083764862 conn(0x7f7a9406f650 msgr2=0x7f7a9406fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:11.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.953+0000 7f7a9359e640 1 -- 172.21.15.90:0/4083764862 shutdown_connections 2024-04-25T11:39:11.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.953+0000 7f7a9359e640 1 -- 172.21.15.90:0/4083764862 wait complete. 2024-04-25T11:39:11.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.954+0000 7f7a9359e640 1 Processor -- start 2024-04-25T11:39:11.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.955+0000 7f7a9359e640 1 -- start start 2024-04-25T11:39:11.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.955+0000 7f7a9359e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a9410a260 0x7f7a941aa330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.955+0000 7f7a9359e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a94113c80 0x7f7a941aa870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.955+0000 7f7a9359e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7a941166c0 0x7f7a941b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.955+0000 7f7a9359e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9411c470 con 0x7f7a94113c80 2024-04-25T11:39:11.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.955+0000 7f7a91d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a94113c80 0x7f7a941aa870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.955+0000 7f7a91d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a94113c80 0x7f7a941aa870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44482/0 (socket says 172.21.15.90:44482) 2024-04-25T11:39:11.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.955+0000 7f7a91d9b640 1 -- 172.21.15.90:0/639615382 learned_addr learned my addr 172.21.15.90:0/639615382 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:11.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.955+0000 7f7a9359e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9411c2f0 con 0x7f7a941166c0 2024-04-25T11:39:11.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.956+0000 7f7a9359e640 1 -- 172.21.15.90:0/639615382 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9411c5f0 con 0x7f7a9410a260 2024-04-25T11:39:11.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.956+0000 7f7a9259c640 1 --2- 172.21.15.90:0/639615382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a9410a260 0x7f7a941aa330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.956+0000 7f7a91d9b640 1 -- 172.21.15.90:0/639615382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a9410a260 msgr2=0x7f7a941aa330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:11.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.956+0000 7f7a91d9b640 1 --2- 172.21.15.90:0/639615382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a9410a260 0x7f7a941aa330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.956+0000 7f7a91d9b640 1 -- 172.21.15.90:0/639615382 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7a941166c0 msgr2=0x7f7a941b51d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:39:11.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.956+0000 7f7a91d9b640 1 --2- 172.21.15.90:0/639615382 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7a941166c0 0x7f7a941b51d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.956+0000 7f7a91d9b640 1 -- 172.21.15.90:0/639615382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a941b5740 con 0x7f7a94113c80 2024-04-25T11:39:11.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.956+0000 7f7a91d9b640 1 --2- 172.21.15.90:0/639615382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a94113c80 0x7f7a941aa870 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f7a88005ab0 tx=0x7f7a88003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:11.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.956+0000 7f7a837fe640 1 -- 172.21.15.90:0/639615382 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a8803f070 con 0x7f7a94113c80 2024-04-25T11:39:11.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.957+0000 7f7a9359e640 1 -- 172.21.15.90:0/639615382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a941b59d0 con 0x7f7a94113c80 2024-04-25T11:39:11.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.957+0000 7f7a837fe640 1 -- 172.21.15.90:0/639615382 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a88031d60 con 0x7f7a94113c80 2024-04-25T11:39:11.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.957+0000 7f7a837fe640 1 -- 172.21.15.90:0/639615382 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a88035030 con 0x7f7a94113c80 2024-04-25T11:39:11.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.957+0000 7f7a9359e640 1 -- 172.21.15.90:0/639615382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a941b5e60 con 0x7f7a94113c80 2024-04-25T11:39:11.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.961+0000 7f7a9fa2b740 1 Processor -- start 2024-04-25T11:39:11.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.962+0000 7f7a9fa2b740 1 -- start start 2024-04-25T11:39:11.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.964+0000 7f7a837fe640 1 -- 172.21.15.90:0/639615382 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f7a88042070 con 0x7f7a94113c80 2024-04-25T11:39:11.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.966+0000 7f7a837fe640 1 --2- 172.21.15.90:0/639615382 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f7a7c0afce0 0x7f7a7c0b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.966+0000 7f7a837fe640 1 -- 172.21.15.90:0/639615382 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f7a880ffb10 con 0x7f7a94113c80 2024-04-25T11:39:11.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.967+0000 7f7a92d9d640 1 --2- 172.21.15.90:0/639615382 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f7a7c0afce0 0x7f7a7c0b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.967+0000 7f7a9fa2b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7d4dcc390 0x55c7d516da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.968+0000 7f7a9fa2b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d4da34d0 0x55c7d516df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.968+0000 7f7a9fa2b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7d4dab6b0 0x55c7d517a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.968+0000 7f7a9fa2b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c7d50d2d80 con 0x55c7d4da34d0 2024-04-25T11:39:11.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.968+0000 7f7a9fa2b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c7d508ca20 con 0x55c7d4dcc390 2024-04-25T11:39:11.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.968+0000 7f7a9fa2b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c7d50ced40 con 0x55c7d4dab6b0 2024-04-25T11:39:11.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.968+0000 7f7a91d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d4da34d0 0x55c7d516df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.968+0000 7f7a91d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d4da34d0 0x55c7d516df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44486/0 (socket says 172.21.15.90:44486) 2024-04-25T11:39:11.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.968+0000 7f7a91d9b640 1 -- 172.21.15.90:0/3251103425 learned_addr learned my addr 172.21.15.90:0/3251103425 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:11.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.969+0000 7f7a92d9d640 1 --2- 172.21.15.90:0/3251103425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7d4dab6b0 0x55c7d517a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.969+0000 7f7a91d9b640 1 -- 172.21.15.90:0/3251103425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7d4dab6b0 msgr2=0x55c7d517a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:11.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.969+0000 7f7a91d9b640 1 --2- 172.21.15.90:0/3251103425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7d4dab6b0 0x55c7d517a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.969+0000 7f7a91d9b640 1 -- 172.21.15.90:0/3251103425 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7d4dcc390 msgr2=0x55c7d516da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:11.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.969+0000 7f7a91d9b640 1 --2- 172.21.15.90:0/3251103425 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7d4dcc390 0x55c7d516da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.969+0000 7f7a91d9b640 1 -- 172.21.15.90:0/3251103425 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7d5097c30 con 0x55c7d4da34d0 2024-04-25T11:39:11.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.969+0000 7f7a91d9b640 1 --2- 172.21.15.90:0/3251103425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d4da34d0 0x55c7d516df40 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f7a880c7b80 tx=0x7f7a8803b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15687807a59ee047 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:11.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.970+0000 7f7a92d9d640 1 --2- 172.21.15.90:0/639615382 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f7a7c0afce0 0x7f7a7c0b21f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7a9406c4e0 tx=0x7f7a8c00a610 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:11.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.970+0000 7f7a80ff9640 1 -- 172.21.15.90:0/3251103425 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a88049070 con 0x55c7d4da34d0 2024-04-25T11:39:11.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.970+0000 7f7a80ff9640 1 -- 172.21.15.90:0/3251103425 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a880c82f0 con 0x55c7d4da34d0 2024-04-25T11:39:11.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.970+0000 7f7a80ff9640 1 -- 172.21.15.90:0/3251103425 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a880fd420 con 0x55c7d4da34d0 2024-04-25T11:39:11.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.972+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/3251103425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d4da34d0 msgr2=0x55c7d516df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:11.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.972+0000 7f7a9fa2b740 1 --2- 172.21.15.90:0/3251103425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d4da34d0 0x55c7d516df40 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f7a880c7b80 tx=0x7f7a8803b460 comp rx=0 tx=0).stop 2024-04-25T11:39:11.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.974+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/3251103425 shutdown_connections 2024-04-25T11:39:11.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.974+0000 7f7a9fa2b740 1 --2- 172.21.15.90:0/3251103425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7d4dab6b0 0x55c7d517a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.974+0000 7f7a9fa2b740 1 --2- 172.21.15.90:0/3251103425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d4da34d0 0x55c7d516df40 secure :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f7a880c7b80 tx=0x7f7a8803b460 comp rx=0 tx=0).stop 2024-04-25T11:39:11.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.974+0000 7f7a9fa2b740 1 --2- 172.21.15.90:0/3251103425 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7d4dcc390 0x55c7d516da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.974+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/3251103425 >> 172.21.15.90:0/3251103425 conn(0x55c7d4dc04e0 msgr2=0x55c7d516a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:11.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.976+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/3251103425 shutdown_connections 2024-04-25T11:39:11.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.978+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/3251103425 wait complete. 2024-04-25T11:39:11.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.978+0000 7f7a9fa2b740 1 -- start start 2024-04-25T11:39:11.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.979+0000 7f7a9fa2b740 1 Processor -- start 2024-04-25T11:39:11.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.979+0000 7f7a9fa2b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7d4dab6b0 0x55c7d51785a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.979+0000 7f7a9fa2b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7d4dcc390 0x55c7d5178ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.980+0000 7f7a9fa2b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d4d8af60 0x55c7d517e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.980+0000 7f7a9fa2b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c7d517add0 con 0x55c7d4d8af60 2024-04-25T11:39:11.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.980+0000 7f7a9fa2b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c7d5059dc0 con 0x55c7d4dcc390 2024-04-25T11:39:11.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.980+0000 7f7a92d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d4d8af60 0x55c7d517e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.980+0000 7f7a9fa2b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c7d517af50 con 0x55c7d4dab6b0 2024-04-25T11:39:11.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.980+0000 7f7a92d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d4d8af60 0x55c7d517e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44488/0 (socket says 172.21.15.90:44488) 2024-04-25T11:39:11.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.980+0000 7f7a92d9d640 1 -- 172.21.15.90:0/2964489128 learned_addr learned my addr 172.21.15.90:0/2964489128 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:11.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.980+0000 7f7a9fa2b740 10 client.0 _ll_register_callbacks cb 0x55c7d5075600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:11.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.980+0000 7f7a92d9d640 1 -- 172.21.15.90:0/2964489128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7d4dab6b0 msgr2=0x55c7d51785a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:11.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.980+0000 7f7a92d9d640 1 --2- 172.21.15.90:0/2964489128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7d4dab6b0 0x55c7d51785a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.980+0000 7f7a92d9d640 1 -- 172.21.15.90:0/2964489128 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7d4dcc390 msgr2=0x55c7d5178ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:11.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.980+0000 7f7a92d9d640 1 --2- 172.21.15.90:0/2964489128 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7d4dcc390 0x55c7d5178ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.980+0000 7f7a92d9d640 1 -- 172.21.15.90:0/2964489128 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7d4de43f0 con 0x55c7d4d8af60 2024-04-25T11:39:11.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.981+0000 7f7a92d9d640 1 --2- 172.21.15.90:0/2964489128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d4d8af60 0x55c7d517e100 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c01bdc0 tx=0x7f7a8c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58b97c6b904a5b2e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:11.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.981+0000 7f7a79ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:11.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.981+0000 7f7a79ffb640 1 -- 172.21.15.90:0/2964489128 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c015070 con 0x55c7d4d8af60 2024-04-25T11:39:11.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.981+0000 7f7a79ffb640 1 -- 172.21.15.90:0/2964489128 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f7a8c00fdd0 con 0x55c7d4d8af60 2024-04-25T11:39:11.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.982+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/2964489128 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a880c82f0 con 0x55c7d4d8af60 2024-04-25T11:39:11.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.982+0000 7f7a79ffb640 1 -- 172.21.15.90:0/2964489128 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7a8c003a10 con 0x55c7d4d8af60 2024-04-25T11:39:11.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.982+0000 7f7a9fa2b740 10 client.16854 fetch_fsmap learned FSMap version 27 2024-04-25T11:39:11.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.982+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/2964489128 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c7d4eae7d0 con 0x55c7d4d8af60 2024-04-25T11:39:11.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.982+0000 7f7a79ffb640 1 -- 172.21.15.90:0/2964489128 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a880c82f0 con 0x55c7d4d8af60 2024-04-25T11:39:11.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.983+0000 7f7a79ffb640 1 -- 172.21.15.90:0/2964489128 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f7a8c019430 con 0x55c7d4d8af60 2024-04-25T11:39:11.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.983+0000 7f7a9fa2b740 10 client.16854 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:39:11.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.983+0000 7f7a9fa2b740 10 client.16854 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:39:11.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.983+0000 7f7a9fa2b740 20 client.16854 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:11.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.983+0000 7f7a9fa2b740 1 --2- 172.21.15.90:0/2964489128 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c7d4d603a0 0x55c7d518a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:11.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.983+0000 7f7a9fa2b740 4 client.16854 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:39:11.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.983+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/2964489128 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7d50e0990 con 0x55c7d4d603a0 2024-04-25T11:39:11.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.984+0000 7f7a9259c640 1 --2- 172.21.15.90:0/2964489128 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c7d4d603a0 0x55c7d518a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:11.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.987+0000 7f7a9259c640 1 --2- 172.21.15.90:0/2964489128 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c7d4d603a0 0x55c7d518a930 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e413d1c41215aabe server_cookie=d1aac2c6fd0d975 in_seq=0 out_seq=0 2024-04-25T11:39:11.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.987+0000 7f7a79ffb640 10 client.16854 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:11.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.992+0000 7f7a79ffb640 1 -- 172.21.15.90:0/2964489128 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c7d50e0990 con 0x55c7d4d603a0 2024-04-25T11:39:11.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.992+0000 7f7a79ffb640 10 client.16854 handle_command_reply: tid=0 2024-04-25T11:39:11.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.992+0000 7f7a9fa2b740 1 client.16854 shutdown 2024-04-25T11:39:11.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.993+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/2964489128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d4d8af60 msgr2=0x55c7d517e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:11.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.993+0000 7f7a9fa2b740 1 --2- 172.21.15.90:0/2964489128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d4d8af60 0x55c7d517e100 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c01bdc0 tx=0x7f7a8c004ad0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.993+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/2964489128 shutdown_connections 2024-04-25T11:39:11.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.993+0000 7f7a9fa2b740 1 --2- 172.21.15.90:0/2964489128 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c7d4d603a0 0x55c7d518a930 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.993+0000 7f7a9fa2b740 1 --2- 172.21.15.90:0/2964489128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d4d8af60 0x55c7d517e100 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.993+0000 7f7a9fa2b740 1 --2- 172.21.15.90:0/2964489128 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7d4dcc390 0x55c7d5178ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.993+0000 7f7a9fa2b740 1 --2- 172.21.15.90:0/2964489128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7d4dab6b0 0x55c7d51785a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:11.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.993+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/2964489128 >> 172.21.15.90:0/2964489128 conn(0x55c7d4dc04e0 msgr2=0x55c7d51689c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:11.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.993+0000 7f7a79ffb640 0 client.16854 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:11.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.993+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/2964489128 shutdown_connections 2024-04-25T11:39:11.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.993+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/2964489128 wait complete. 2024-04-25T11:39:11.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:11.994+0000 7f7a9fa2b740 20 client.16854 trim_cache size 0 max 16384 2024-04-25T11:39:12.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.020+0000 7f7a9fa2b740 1 Processor -- start 2024-04-25T11:39:12.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.020+0000 7f7a9fa2b740 1 -- start start 2024-04-25T11:39:12.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.021+0000 7f7a9fa2b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d4d8af60 0x55c7d517e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:12.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.021+0000 7f7a9fa2b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7d4dab6b0 0x55c7d5172620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:12.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.021+0000 7f7a9fa2b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7d4dcc390 0x55c7d5172b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:12.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.021+0000 7f7a9259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d4d8af60 0x55c7d517e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:12.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.021+0000 7f7a9259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d4d8af60 0x55c7d517e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44500/0 (socket says 172.21.15.90:44500) 2024-04-25T11:39:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.022+0000 7f7a9fa2b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7a941be8a0 con 0x55c7d4d8af60 2024-04-25T11:39:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.022+0000 7f7a9259c640 1 -- 172.21.15.90:0/716400219 learned_addr learned my addr 172.21.15.90:0/716400219 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.022+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/716400219 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7a941c0410 con 0x55c7d4dab6b0 2024-04-25T11:39:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.022+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/716400219 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7a941bef70 con 0x55c7d4dcc390 2024-04-25T11:39:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.022+0000 7f7a91d9b640 1 --2- 172.21.15.90:0/716400219 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7d4dab6b0 0x55c7d5172620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.022+0000 7f7a92d9d640 1 --2- 172.21.15.90:0/716400219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7d4dcc390 0x55c7d5172b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:12.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.023+0000 7f7a9259c640 1 -- 172.21.15.90:0/716400219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7d4dcc390 msgr2=0x55c7d5172b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:12.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.023+0000 7f7a9259c640 1 --2- 172.21.15.90:0/716400219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7d4dcc390 0x55c7d5172b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:12.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.023+0000 7f7a9259c640 1 -- 172.21.15.90:0/716400219 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7d4dab6b0 msgr2=0x55c7d5172620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:12.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.023+0000 7f7a9259c640 1 --2- 172.21.15.90:0/716400219 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7d4dab6b0 0x55c7d5172620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:12.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.023+0000 7f7a9259c640 1 -- 172.21.15.90:0/716400219 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7d5173f50 con 0x55c7d4d8af60 2024-04-25T11:39:12.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.023+0000 7f7a9259c640 1 --2- 172.21.15.90:0/716400219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d4d8af60 0x55c7d517e0c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f7a9410ad50 tx=0x7f7a84009d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6cb090ebc561407 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:12.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.024+0000 7f7a7bfff640 1 -- 172.21.15.90:0/716400219 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a8401e070 con 0x55c7d4d8af60 2024-04-25T11:39:12.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.024+0000 7f7a7bfff640 1 -- 172.21.15.90:0/716400219 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a84014dc0 con 0x55c7d4d8af60 2024-04-25T11:39:12.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.024+0000 7f7a7bfff640 1 -- 172.21.15.90:0/716400219 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a84017a70 con 0x55c7d4d8af60 2024-04-25T11:39:12.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.027+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/716400219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d4d8af60 msgr2=0x55c7d517e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:12.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.027+0000 7f7a9fa2b740 1 --2- 172.21.15.90:0/716400219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d4d8af60 0x55c7d517e0c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f7a9410ad50 tx=0x7f7a84009d50 comp rx=0 tx=0).stop 2024-04-25T11:39:12.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.027+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/716400219 shutdown_connections 2024-04-25T11:39:12.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.027+0000 7f7a9fa2b740 1 --2- 172.21.15.90:0/716400219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7d4dcc390 0x55c7d5172b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:12.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.027+0000 7f7a9fa2b740 1 --2- 172.21.15.90:0/716400219 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7d4dab6b0 0x55c7d5172620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:12.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.027+0000 7f7a9fa2b740 1 --2- 172.21.15.90:0/716400219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d4d8af60 0x55c7d517e0c0 secure :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f7a9410ad50 tx=0x7f7a84009d50 comp rx=0 tx=0).stop 2024-04-25T11:39:12.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.027+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/716400219 >> 172.21.15.90:0/716400219 conn(0x55c7d4dc04e0 msgr2=0x55c7d5176c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:12.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.029+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/716400219 shutdown_connections 2024-04-25T11:39:12.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.029+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/716400219 wait complete. 2024-04-25T11:39:12.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.031+0000 7f7a9fa2b740 1 -- start start 2024-04-25T11:39:12.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.031+0000 7f7a9fa2b740 1 Processor -- start 2024-04-25T11:39:12.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.032+0000 7f7a9fa2b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7d4dab6b0 0x55c7d51847d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:12.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.032+0000 7f7a9fa2b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7d4dcc390 0x55c7d517fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:12.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.032+0000 7f7a9fa2b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d51800b0 0x55c7d51804c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:12.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.032+0000 7f7a9fa2b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7a941bfd10 con 0x55c7d51800b0 2024-04-25T11:39:12.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.032+0000 7f7a9fa2b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7a941bf640 con 0x55c7d4dab6b0 2024-04-25T11:39:12.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.032+0000 7f7a9fa2b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7a941bc880 con 0x55c7d4dcc390 2024-04-25T11:39:12.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.032+0000 7f7a9fa2b740 10 client.0 _ll_register_callbacks cb 0x7f7a8c01bac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:12.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.033+0000 7f7a91d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7d4dcc390 0x55c7d517fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:12.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.033+0000 7f7a92d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d51800b0 0x55c7d51804c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:12.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.033+0000 7f7a91d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7d4dcc390 0x55c7d517fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:59232/0 (socket says 172.21.15.90:59232) 2024-04-25T11:39:12.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.033+0000 7f7a91d9b640 1 -- 172.21.15.90:0/1689804896 learned_addr learned my addr 172.21.15.90:0/1689804896 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:12.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.033+0000 7f7a9259c640 1 --2- 172.21.15.90:0/1689804896 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7d4dab6b0 0x55c7d51847d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:12.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.033+0000 7f7a92d9d640 1 -- 172.21.15.90:0/1689804896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7d4dcc390 msgr2=0x55c7d517fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:12.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.033+0000 7f7a92d9d640 1 --2- 172.21.15.90:0/1689804896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7d4dcc390 0x55c7d517fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:12.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.033+0000 7f7a92d9d640 1 -- 172.21.15.90:0/1689804896 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7d4dab6b0 msgr2=0x55c7d51847d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:12.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.033+0000 7f7a92d9d640 1 --2- 172.21.15.90:0/1689804896 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7d4dab6b0 0x55c7d51847d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:12.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.033+0000 7f7a92d9d640 1 -- 172.21.15.90:0/1689804896 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7d51f1ca0 con 0x55c7d51800b0 2024-04-25T11:39:12.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.033+0000 7f7a91d9b640 1 --2- 172.21.15.90:0/1689804896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7d4dcc390 0x55c7d517fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:12.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.033+0000 7f7a92d9d640 1 --2- 172.21.15.90:0/1689804896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d51800b0 0x55c7d51804c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c019920 tx=0x7f7a8c019dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20983083ccd1f53e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:12.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.033+0000 7f7a79ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:12.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.034+0000 7f7a79ffb640 1 -- 172.21.15.90:0/1689804896 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c033070 con 0x55c7d51800b0 2024-04-25T11:39:12.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.034+0000 7f7a79ffb640 1 -- 172.21.15.90:0/1689804896 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f7a8c00fdd0 con 0x55c7d51800b0 2024-04-25T11:39:12.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.034+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/1689804896 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a84014dc0 con 0x55c7d51800b0 2024-04-25T11:39:12.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.034+0000 7f7a79ffb640 1 -- 172.21.15.90:0/1689804896 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7a8c015050 con 0x55c7d51800b0 2024-04-25T11:39:12.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.034+0000 7f7a9fa2b740 10 client.16860 fetch_fsmap learned FSMap version 27 2024-04-25T11:39:12.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.034+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/1689804896 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c7d51f2d70 con 0x55c7d51800b0 2024-04-25T11:39:12.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.035+0000 7f7a79ffb640 1 -- 172.21.15.90:0/1689804896 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a84014dc0 con 0x55c7d51800b0 2024-04-25T11:39:12.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.035+0000 7f7a79ffb640 1 -- 172.21.15.90:0/1689804896 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7f7a8c02f450 con 0x55c7d51800b0 2024-04-25T11:39:12.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.035+0000 7f7a9fa2b740 10 client.16860 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:39:12.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.035+0000 7f7a9fa2b740 10 client.16860 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:39:12.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.035+0000 7f7a9fa2b740 20 client.16860 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:12.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.035+0000 7f7a9fa2b740 1 --2- 172.21.15.90:0/1689804896 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c7d51f34b0 0x55c7d51f58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:12.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.035+0000 7f7a9fa2b740 4 client.16860 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:39:12.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.035+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/1689804896 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c7d50da800 con 0x55c7d51f34b0 2024-04-25T11:39:12.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.036+0000 7f7a9259c640 1 --2- 172.21.15.90:0/1689804896 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c7d51f34b0 0x55c7d51f58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:12.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.036+0000 7f7a79ffb640 10 client.16860 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:12.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.036+0000 7f7a9259c640 1 --2- 172.21.15.90:0/1689804896 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c7d51f34b0 0x55c7d51f58d0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6d178c2e95b69ac server_cookie=855bfd169f1c0057 in_seq=0 out_seq=0 2024-04-25T11:39:12.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.037+0000 7f7a79ffb640 1 -- 172.21.15.90:0/1689804896 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c7d50da800 con 0x55c7d51f34b0 2024-04-25T11:39:12.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.037+0000 7f7a79ffb640 10 client.16860 handle_command_reply: tid=0 2024-04-25T11:39:12.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.037+0000 7f7a9fa2b740 1 client.16860 shutdown 2024-04-25T11:39:12.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.037+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/1689804896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d51800b0 msgr2=0x55c7d51804c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:12.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.037+0000 7f7a9fa2b740 1 --2- 172.21.15.90:0/1689804896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d51800b0 0x55c7d51804c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c019920 tx=0x7f7a8c019dc0 comp rx=0 tx=0).stop 2024-04-25T11:39:12.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.037+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/1689804896 shutdown_connections 2024-04-25T11:39:12.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.037+0000 7f7a9fa2b740 1 --2- 172.21.15.90:0/1689804896 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c7d51f34b0 0x55c7d51f58d0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:12.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.037+0000 7f7a9fa2b740 1 --2- 172.21.15.90:0/1689804896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7d51800b0 0x55c7d51804c0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:12.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.037+0000 7f7a9fa2b740 1 --2- 172.21.15.90:0/1689804896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7d4dcc390 0x55c7d517fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:12.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.037+0000 7f7a9fa2b740 1 --2- 172.21.15.90:0/1689804896 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c7d4dab6b0 0x55c7d51847d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:12.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.037+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/1689804896 >> 172.21.15.90:0/1689804896 conn(0x55c7d4dc04e0 msgr2=0x55c7d5179820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:12.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.037+0000 7f7a79ffb640 0 client.16860 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:12.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.038+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/1689804896 shutdown_connections 2024-04-25T11:39:12.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.038+0000 7f7a9fa2b740 1 -- 172.21.15.90:0/1689804896 wait complete. 2024-04-25T11:39:12.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.038+0000 7f7a9fa2b740 20 client.16860 trim_cache size 0 max 16384 2024-04-25T11:39:12.027 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:39:12.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.041+0000 7f7a9359e640 1 -- 172.21.15.90:0/639615382 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f7a7c0afce0 msgr2=0x7f7a7c0b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:12.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.041+0000 7f7a9359e640 1 --2- 172.21.15.90:0/639615382 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f7a7c0afce0 0x7f7a7c0b21f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7a9406c4e0 tx=0x7f7a8c00a610 comp rx=0 tx=0).stop 2024-04-25T11:39:12.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.041+0000 7f7a9359e640 1 -- 172.21.15.90:0/639615382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a94113c80 msgr2=0x7f7a941aa870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:12.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.041+0000 7f7a9359e640 1 --2- 172.21.15.90:0/639615382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a94113c80 0x7f7a941aa870 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f7a88005ab0 tx=0x7f7a88003cd0 comp rx=0 tx=0).stop 2024-04-25T11:39:12.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.042+0000 7f7a9359e640 1 -- 172.21.15.90:0/639615382 shutdown_connections 2024-04-25T11:39:12.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.042+0000 7f7a9359e640 1 --2- 172.21.15.90:0/639615382 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f7a7c0afce0 0x7f7a7c0b21f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:12.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.042+0000 7f7a9359e640 1 --2- 172.21.15.90:0/639615382 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7a941166c0 0x7f7a941b51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:12.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.042+0000 7f7a9359e640 1 --2- 172.21.15.90:0/639615382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a94113c80 0x7f7a941aa870 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:12.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.042+0000 7f7a9359e640 1 --2- 172.21.15.90:0/639615382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a9410a260 0x7f7a941aa330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:12.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.042+0000 7f7a9359e640 1 -- 172.21.15.90:0/639615382 >> 172.21.15.90:0/639615382 conn(0x7f7a9406f650 msgr2=0x7f7a94116b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:12.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.044+0000 7f7a9359e640 1 -- 172.21.15.90:0/639615382 shutdown_connections 2024-04-25T11:39:12.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:12.046+0000 7f7a9359e640 1 -- 172.21.15.90:0/639615382 wait complete. 2024-04-25T11:39:12.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:12 smithi090 ceph-mon[29575]: pgmap v126: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 571 MiB/s wr, 926 op/s 2024-04-25T11:39:12.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:12 smithi138 ceph-mon[32328]: pgmap v126: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 571 MiB/s wr, 926 op/s 2024-04-25T11:39:12.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:12 smithi156 ceph-mon[33045]: pgmap v126: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 571 MiB/s wr, 926 op/s 2024-04-25T11:39:13.069 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T11:39:13.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:39:13.069 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T11:39:13.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.209+0000 7fe09f851640 1 -- 172.21.15.90:0/2966476863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe098103850 msgr2=0x7fe09810f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:13.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.209+0000 7fe09f851640 1 --2- 172.21.15.90:0/2966476863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe098103850 0x7fe09810f5c0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fe090009ca0 tx=0x7fe09002a8c0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.209+0000 7fe09f851640 1 -- 172.21.15.90:0/2966476863 shutdown_connections 2024-04-25T11:39:13.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.209+0000 7fe09f851640 1 --2- 172.21.15.90:0/2966476863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe098103850 0x7fe09810f5c0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.209+0000 7fe09f851640 1 --2- 172.21.15.90:0/2966476863 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe098108f40 0x7fe0981031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.209+0000 7fe09f851640 1 --2- 172.21.15.90:0/2966476863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe098108aa0 0x7fe098102ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.209+0000 7fe09f851640 1 -- 172.21.15.90:0/2966476863 >> 172.21.15.90:0/2966476863 conn(0x7fe098077880 msgr2=0x7fe098077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:13.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.209+0000 7fe09f851640 1 -- 172.21.15.90:0/2966476863 shutdown_connections 2024-04-25T11:39:13.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.209+0000 7fe09f851640 1 -- 172.21.15.90:0/2966476863 wait complete. 2024-04-25T11:39:13.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.210+0000 7fe09f851640 1 Processor -- start 2024-04-25T11:39:13.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.210+0000 7fe09f851640 1 -- start start 2024-04-25T11:39:13.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.210+0000 7fe09f851640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe098103850 0x7fe09819b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.210+0000 7fe09f851640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe098108aa0 0x7fe09819bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.210+0000 7fe09f851640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe098108f40 0x7fe09819c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.210+0000 7fe09f851640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe098112cb0 con 0x7fe098103850 2024-04-25T11:39:13.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.210+0000 7fe09f851640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe098112b30 con 0x7fe098108aa0 2024-04-25T11:39:13.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.210+0000 7fe09f851640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe098112e30 con 0x7fe098108f40 2024-04-25T11:39:13.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.210+0000 7fe09d5ed640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe098103850 0x7fe09819b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.210+0000 7fe09d5ed640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe098103850 0x7fe09819b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44542/0 (socket says 172.21.15.90:44542) 2024-04-25T11:39:13.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.210+0000 7fe09d5ed640 1 -- 172.21.15.90:0/3354346246 learned_addr learned my addr 172.21.15.90:0/3354346246 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:13.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.210+0000 7fe09cdec640 1 --2- 172.21.15.90:0/3354346246 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe098108aa0 0x7fe09819bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.210+0000 7fe09ddee640 1 --2- 172.21.15.90:0/3354346246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe098108f40 0x7fe09819c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.210+0000 7fe09cdec640 1 -- 172.21.15.90:0/3354346246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe098108f40 msgr2=0x7fe09819c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:13.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.210+0000 7fe09cdec640 1 --2- 172.21.15.90:0/3354346246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe098108f40 0x7fe09819c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.210+0000 7fe09cdec640 1 -- 172.21.15.90:0/3354346246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe098103850 msgr2=0x7fe09819b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:13.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.210+0000 7fe09cdec640 1 --2- 172.21.15.90:0/3354346246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe098103850 0x7fe09819b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.210+0000 7fe09cdec640 1 -- 172.21.15.90:0/3354346246 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0981a6900 con 0x7fe098108aa0 2024-04-25T11:39:13.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.211+0000 7fe09ddee640 1 --2- 172.21.15.90:0/3354346246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe098108f40 0x7fe09819c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:13.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.211+0000 7fe09d5ed640 1 --2- 172.21.15.90:0/3354346246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe098103850 0x7fe09819b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:13.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.211+0000 7fe09cdec640 1 --2- 172.21.15.90:0/3354346246 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe098108aa0 0x7fe09819bb30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe0980754b0 tx=0x7fe08800ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:13.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.211+0000 7fe08e7fc640 1 -- 172.21.15.90:0/3354346246 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08800cd00 con 0x7fe098108aa0 2024-04-25T11:39:13.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.211+0000 7fe08e7fc640 1 -- 172.21.15.90:0/3354346246 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0880108e0 con 0x7fe098108aa0 2024-04-25T11:39:13.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.211+0000 7fe09f851640 1 -- 172.21.15.90:0/3354346246 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0981a6bf0 con 0x7fe098108aa0 2024-04-25T11:39:13.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.211+0000 7fe08e7fc640 1 -- 172.21.15.90:0/3354346246 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0880188c0 con 0x7fe098108aa0 2024-04-25T11:39:13.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.211+0000 7fe09f851640 1 -- 172.21.15.90:0/3354346246 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0981a7140 con 0x7fe098108aa0 2024-04-25T11:39:13.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.214+0000 7fe0a67d7740 1 Processor -- start 2024-04-25T11:39:13.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.214+0000 7fe0a67d7740 1 -- start start 2024-04-25T11:39:13.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.214+0000 7fe0a67d7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acd4f4f410 0x55acd52ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.214+0000 7fe0a67d7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd4f26550 0x55acd52efb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.214+0000 7fe0a67d7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acd4f2e730 0x55acd52f00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.214+0000 7fe0a67d7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55acd5255b00 con 0x55acd4f2e730 2024-04-25T11:39:13.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.214+0000 7fe0a67d7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55acd51f1920 con 0x55acd4f4f410 2024-04-25T11:39:13.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.214+0000 7fe0a67d7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55acd5251ac0 con 0x55acd4f26550 2024-04-25T11:39:13.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.214+0000 7fe09ddee640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acd4f4f410 0x55acd52ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.214+0000 7fe09ddee640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acd4f4f410 0x55acd52ef620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:50630/0 (socket says 172.21.15.90:50630) 2024-04-25T11:39:13.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.214+0000 7fe09ddee640 1 -- 172.21.15.90:0/2528960615 learned_addr learned my addr 172.21.15.90:0/2528960615 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:13.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.214+0000 7fe09d5ed640 1 --2- 172.21.15.90:0/2528960615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd4f26550 0x55acd52efb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.215+0000 7fe09ddee640 1 -- 172.21.15.90:0/2528960615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd4f26550 msgr2=0x55acd52efb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:13.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.215+0000 7fe09ddee640 1 --2- 172.21.15.90:0/2528960615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd4f26550 0x55acd52efb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.215+0000 7fe09ddee640 1 -- 172.21.15.90:0/2528960615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acd4f2e730 msgr2=0x55acd52f00a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:39:13.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.215+0000 7fe09ddee640 1 --2- 172.21.15.90:0/2528960615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acd4f2e730 0x55acd52f00a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.215+0000 7fe09ddee640 1 -- 172.21.15.90:0/2528960615 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55acd502f7e0 con 0x55acd4f4f410 2024-04-25T11:39:13.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.215+0000 7fe08e7fc640 1 -- 172.21.15.90:0/3354346246 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fe08801f020 con 0x7fe098108aa0 2024-04-25T11:39:13.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.215+0000 7fe09ddee640 1 --2- 172.21.15.90:0/2528960615 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acd4f4f410 0x55acd52ef620 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe090035000 tx=0x7fe09002f310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cec395d4da38b326 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:13.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.215+0000 7fe0837fe640 1 -- 172.21.15.90:0/2528960615 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe090031650 con 0x55acd4f4f410 2024-04-25T11:39:13.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.215+0000 7fe0837fe640 1 -- 172.21.15.90:0/2528960615 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe090036040 con 0x55acd4f4f410 2024-04-25T11:39:13.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.215+0000 7fe0837fe640 1 -- 172.21.15.90:0/2528960615 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe090004420 con 0x55acd4f4f410 2024-04-25T11:39:13.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.216+0000 7fe08e7fc640 1 --2- 172.21.15.90:0/3354346246 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe0840afc60 0x7fe0840b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.216+0000 7fe09d5ed640 1 --2- 172.21.15.90:0/3354346246 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe0840afc60 0x7fe0840b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.216+0000 7fe08e7fc640 1 -- 172.21.15.90:0/3354346246 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fe0880df0a0 con 0x7fe098108aa0 2024-04-25T11:39:13.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.217+0000 7fe09d5ed640 1 --2- 172.21.15.90:0/3354346246 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe0840afc60 0x7fe0840b2170 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x55acd52f0be0 tx=0x7fe094005f90 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:13.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.217+0000 7fe0a67d7740 1 -- 172.21.15.90:0/2528960615 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acd4f4f410 msgr2=0x55acd52ef620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:13.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.217+0000 7fe0a67d7740 1 --2- 172.21.15.90:0/2528960615 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acd4f4f410 0x55acd52ef620 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe090035000 tx=0x7fe09002f310 comp rx=0 tx=0).stop 2024-04-25T11:39:13.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.217+0000 7fe0a67d7740 1 -- 172.21.15.90:0/2528960615 shutdown_connections 2024-04-25T11:39:13.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.217+0000 7fe0a67d7740 1 --2- 172.21.15.90:0/2528960615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acd4f2e730 0x55acd52f00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.217+0000 7fe0a67d7740 1 --2- 172.21.15.90:0/2528960615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd4f26550 0x55acd52efb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.217+0000 7fe0a67d7740 1 --2- 172.21.15.90:0/2528960615 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acd4f4f410 0x55acd52ef620 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.217+0000 7fe0a67d7740 1 -- 172.21.15.90:0/2528960615 >> 172.21.15.90:0/2528960615 conn(0x55acd4f43560 msgr2=0x55acd52ebdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:13.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.217+0000 7fe0a67d7740 1 -- 172.21.15.90:0/2528960615 shutdown_connections 2024-04-25T11:39:13.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.217+0000 7fe0a67d7740 1 -- 172.21.15.90:0/2528960615 wait complete. 2024-04-25T11:39:13.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.217+0000 7fe0a67d7740 1 -- start start 2024-04-25T11:39:13.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.217+0000 7fe0a67d7740 1 Processor -- start 2024-04-25T11:39:13.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.218+0000 7fe0a67d7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd4f26550 0x55acd52f9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.218+0000 7fe0a67d7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acd4f2e730 0x55acd52fa190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.218+0000 7fe0a67d7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acd4f4f410 0x55acd52ffc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.218+0000 7fe0a67d7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55acd52fc790 con 0x55acd4f4f410 2024-04-25T11:39:13.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.218+0000 7fe0a67d7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55acd51ef5d0 con 0x55acd4f2e730 2024-04-25T11:39:13.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.218+0000 7fe0a67d7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55acd52fc910 con 0x55acd4f26550 2024-04-25T11:39:13.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.218+0000 7fe09ddee640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acd4f4f410 0x55acd52ffc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.218+0000 7fe09d5ed640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd4f26550 0x55acd52f9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.218+0000 7fe09ddee640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acd4f4f410 0x55acd52ffc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44558/0 (socket says 172.21.15.90:44558) 2024-04-25T11:39:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.218+0000 7fe09ddee640 1 -- 172.21.15.90:0/3874948994 learned_addr learned my addr 172.21.15.90:0/3874948994 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.218+0000 7fe0a67d7740 10 client.0 _ll_register_callbacks cb 0x55acd51dd080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.218+0000 7fe09cdec640 1 --2- 172.21.15.90:0/3874948994 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acd4f2e730 0x55acd52fa190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.219+0000 7fe09d5ed640 1 -- 172.21.15.90:0/3874948994 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acd4f2e730 msgr2=0x55acd52fa190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.219+0000 7fe09d5ed640 1 --2- 172.21.15.90:0/3874948994 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acd4f2e730 0x55acd52fa190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.219+0000 7fe09d5ed640 1 -- 172.21.15.90:0/3874948994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acd4f4f410 msgr2=0x55acd52ffc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.219+0000 7fe09d5ed640 1 --2- 172.21.15.90:0/3874948994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acd4f4f410 0x55acd52ffc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.219+0000 7fe09d5ed640 1 -- 172.21.15.90:0/3874948994 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55acd508ff70 con 0x55acd4f26550 2024-04-25T11:39:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.219+0000 7fe09ddee640 1 --2- 172.21.15.90:0/3874948994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acd4f4f410 0x55acd52ffc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.219+0000 7fe09d5ed640 1 --2- 172.21.15.90:0/3874948994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd4f26550 0x55acd52f9c50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe094009ca0 tx=0x7fe094004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=347cc5500509b243 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.219+0000 7fe080ff9640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:39:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.219+0000 7fe080ff9640 1 -- 172.21.15.90:0/3874948994 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe094011910 con 0x55acd4f26550 2024-04-25T11:39:13.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.219+0000 7fe0a67d7740 1 -- 172.21.15.90:0/3874948994 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe090036040 con 0x55acd4f26550 2024-04-25T11:39:13.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.219+0000 7fe080ff9640 1 -- 172.21.15.90:0/3874948994 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fe094011ab0 con 0x55acd4f26550 2024-04-25T11:39:13.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.219+0000 7fe080ff9640 1 -- 172.21.15.90:0/3874948994 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe094011cb0 con 0x55acd4f26550 2024-04-25T11:39:13.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.220+0000 7fe0a67d7740 10 client.25981 fetch_fsmap learned FSMap version 27 2024-04-25T11:39:13.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.220+0000 7fe0a67d7740 1 -- 172.21.15.90:0/3874948994 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55acd4f27170 con 0x55acd4f26550 2024-04-25T11:39:13.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.220+0000 7fe080ff9640 1 -- 172.21.15.90:0/3874948994 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe090036040 con 0x55acd4f26550 2024-04-25T11:39:13.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.220+0000 7fe080ff9640 1 -- 172.21.15.90:0/3874948994 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7fe094027070 con 0x55acd4f26550 2024-04-25T11:39:13.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.220+0000 7fe0a67d7740 10 client.25981 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:39:13.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.220+0000 7fe0a67d7740 10 client.25981 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:39:13.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.220+0000 7fe0a67d7740 20 client.25981 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:13.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.220+0000 7fe0a67d7740 1 --2- 172.21.15.90:0/3874948994 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55acd4f0e1f0 0x55acd530bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.220+0000 7fe0a67d7740 4 client.25981 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:39:13.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.220+0000 7fe0a67d7740 1 -- 172.21.15.90:0/3874948994 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55acd5263710 con 0x55acd4f0e1f0 2024-04-25T11:39:13.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.221+0000 7fe09cdec640 1 --2- 172.21.15.90:0/3874948994 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55acd4f0e1f0 0x55acd530bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.221+0000 7fe080ff9640 10 client.25981 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:13.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.221+0000 7fe09cdec640 1 --2- 172.21.15.90:0/3874948994 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55acd4f0e1f0 0x55acd530bf40 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5286704aa655bbe server_cookie=eb615e705d7ca199 in_seq=0 out_seq=0 2024-04-25T11:39:13.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.222+0000 7fe080ff9640 1 -- 172.21.15.90:0/3874948994 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55acd5263710 con 0x55acd4f0e1f0 2024-04-25T11:39:13.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.222+0000 7fe080ff9640 10 client.25981 handle_command_reply: tid=0 2024-04-25T11:39:13.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.222+0000 7fe0a67d7740 1 client.25981 shutdown 2024-04-25T11:39:13.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.222+0000 7fe0a67d7740 1 -- 172.21.15.90:0/3874948994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd4f26550 msgr2=0x55acd52f9c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:13.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.222+0000 7fe0a67d7740 1 --2- 172.21.15.90:0/3874948994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd4f26550 0x55acd52f9c50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe094009ca0 tx=0x7fe094004730 comp rx=0 tx=0).stop 2024-04-25T11:39:13.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.222+0000 7fe0a67d7740 1 -- 172.21.15.90:0/3874948994 shutdown_connections 2024-04-25T11:39:13.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.222+0000 7fe0a67d7740 1 --2- 172.21.15.90:0/3874948994 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55acd4f0e1f0 0x55acd530bf40 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.222+0000 7fe0a67d7740 1 --2- 172.21.15.90:0/3874948994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acd4f4f410 0x55acd52ffc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.222+0000 7fe080ff9640 0 client.25981 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:13.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.222+0000 7fe0a67d7740 1 --2- 172.21.15.90:0/3874948994 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acd4f2e730 0x55acd52fa190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.222+0000 7fe0a67d7740 1 --2- 172.21.15.90:0/3874948994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd4f26550 0x55acd52f9c50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.222+0000 7fe0a67d7740 1 -- 172.21.15.90:0/3874948994 >> 172.21.15.90:0/3874948994 conn(0x55acd4f43560 msgr2=0x55acd52ef450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:13.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.222+0000 7fe0a67d7740 1 -- 172.21.15.90:0/3874948994 shutdown_connections 2024-04-25T11:39:13.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.222+0000 7fe0a67d7740 1 -- 172.21.15.90:0/3874948994 wait complete. 2024-04-25T11:39:13.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.223+0000 7fe0a67d7740 20 client.25981 trim_cache size 0 max 16384 2024-04-25T11:39:13.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.256+0000 7fe0a67d7740 1 Processor -- start 2024-04-25T11:39:13.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.256+0000 7fe0a67d7740 1 -- start start 2024-04-25T11:39:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.257+0000 7fe0a67d7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd4f0e1f0 0x55acd52f4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.257+0000 7fe0a67d7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acd4f26550 0x55acd52fbb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.257+0000 7fe0a67d7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acd4f2e730 0x55acd52ffb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.257+0000 7fe0a67d7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe0981afb40 con 0x55acd4f2e730 2024-04-25T11:39:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.257+0000 7fe0a67d7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe0981b16b0 con 0x55acd4f26550 2024-04-25T11:39:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.257+0000 7fe0a67d7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe0981b0210 con 0x55acd4f0e1f0 2024-04-25T11:39:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.257+0000 7fe09ddee640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acd4f2e730 0x55acd52ffb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.257+0000 7fe09cdec640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acd4f26550 0x55acd52fbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.257+0000 7fe09ddee640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acd4f2e730 0x55acd52ffb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44574/0 (socket says 172.21.15.90:44574) 2024-04-25T11:39:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.257+0000 7fe09cdec640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acd4f26550 0x55acd52fbb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:50648/0 (socket says 172.21.15.90:50648) 2024-04-25T11:39:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.257+0000 7fe09ddee640 1 -- 172.21.15.90:0/3254243754 learned_addr learned my addr 172.21.15.90:0/3254243754 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.257+0000 7fe09d5ed640 1 --2- 172.21.15.90:0/3254243754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd4f0e1f0 0x55acd52f4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.257+0000 7fe09cdec640 1 -- 172.21.15.90:0/3254243754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd4f0e1f0 msgr2=0x55acd52f4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:13.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.257+0000 7fe09cdec640 1 --2- 172.21.15.90:0/3254243754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd4f0e1f0 0x55acd52f4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.257+0000 7fe09cdec640 1 -- 172.21.15.90:0/3254243754 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acd4f2e730 msgr2=0x55acd52ffb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:13.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.257+0000 7fe09cdec640 1 --2- 172.21.15.90:0/3254243754 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acd4f2e730 0x55acd52ffb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.257+0000 7fe09cdec640 1 -- 172.21.15.90:0/3254243754 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55acd5306c50 con 0x55acd4f26550 2024-04-25T11:39:13.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.257+0000 7fe09d5ed640 1 --2- 172.21.15.90:0/3254243754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd4f0e1f0 0x55acd52f4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:13.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.257+0000 7fe09ddee640 1 --2- 172.21.15.90:0/3254243754 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acd4f2e730 0x55acd52ffb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:13.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.258+0000 7fe09cdec640 1 --2- 172.21.15.90:0/3254243754 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acd4f26550 0x55acd52fbb90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe0880dc790 tx=0x7fe0880a6ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6689db23aefe342f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:13.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.258+0000 7fe082ffd640 1 -- 172.21.15.90:0/3254243754 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe088010db0 con 0x55acd4f26550 2024-04-25T11:39:13.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.258+0000 7fe082ffd640 1 -- 172.21.15.90:0/3254243754 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0880093c0 con 0x55acd4f26550 2024-04-25T11:39:13.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.258+0000 7fe082ffd640 1 -- 172.21.15.90:0/3254243754 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0880ab170 con 0x55acd4f26550 2024-04-25T11:39:13.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.259+0000 7fe0a67d7740 1 -- 172.21.15.90:0/3254243754 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acd4f26550 msgr2=0x55acd52fbb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:13.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.259+0000 7fe0a67d7740 1 --2- 172.21.15.90:0/3254243754 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acd4f26550 0x55acd52fbb90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe0880dc790 tx=0x7fe0880a6ba0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.259+0000 7fe0a67d7740 1 -- 172.21.15.90:0/3254243754 shutdown_connections 2024-04-25T11:39:13.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.259+0000 7fe0a67d7740 1 --2- 172.21.15.90:0/3254243754 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acd4f2e730 0x55acd52ffb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.259+0000 7fe0a67d7740 1 --2- 172.21.15.90:0/3254243754 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acd4f26550 0x55acd52fbb90 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.259+0000 7fe0a67d7740 1 --2- 172.21.15.90:0/3254243754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd4f0e1f0 0x55acd52f4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.259+0000 7fe0a67d7740 1 -- 172.21.15.90:0/3254243754 >> 172.21.15.90:0/3254243754 conn(0x55acd4f43560 msgr2=0x55acd52eab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:13.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.259+0000 7fe0a67d7740 1 -- 172.21.15.90:0/3254243754 shutdown_connections 2024-04-25T11:39:13.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.259+0000 7fe0a67d7740 1 -- 172.21.15.90:0/3254243754 wait complete. 2024-04-25T11:39:13.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.259+0000 7fe0a67d7740 1 -- start start 2024-04-25T11:39:13.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.259+0000 7fe0a67d7740 1 Processor -- start 2024-04-25T11:39:13.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.260+0000 7fe0a67d7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd4f0e1f0 0x55acd53062f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.260+0000 7fe0a67d7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acd4f26550 0x55acd5301640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.260+0000 7fe0a67d7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acd4f2e730 0x55acd5301b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.260+0000 7fe0a67d7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe0981b0fb0 con 0x55acd4f2e730 2024-04-25T11:39:13.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.260+0000 7fe0a67d7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe0981b08e0 con 0x55acd4f26550 2024-04-25T11:39:13.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.260+0000 7fe0a67d7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe0981adb20 con 0x55acd4f0e1f0 2024-04-25T11:39:13.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.260+0000 7fe0a67d7740 10 client.0 _ll_register_callbacks cb 0x7fe09400f680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:13.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.260+0000 7fe09cdec640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acd4f26550 0x55acd5301640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.260+0000 7fe09ddee640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acd4f2e730 0x55acd5301b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.260+0000 7fe09cdec640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acd4f26550 0x55acd5301640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:50656/0 (socket says 172.21.15.90:50656) 2024-04-25T11:39:13.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.260+0000 7fe09cdec640 1 -- 172.21.15.90:0/4242415122 learned_addr learned my addr 172.21.15.90:0/4242415122 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:13.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.260+0000 7fe09d5ed640 1 --2- 172.21.15.90:0/4242415122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd4f0e1f0 0x55acd53062f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.260+0000 7fe09cdec640 1 -- 172.21.15.90:0/4242415122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd4f0e1f0 msgr2=0x55acd53062f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:13.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.260+0000 7fe09cdec640 1 --2- 172.21.15.90:0/4242415122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd4f0e1f0 0x55acd53062f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.260+0000 7fe09cdec640 1 -- 172.21.15.90:0/4242415122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acd4f2e730 msgr2=0x55acd5301b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:13.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.260+0000 7fe09cdec640 1 --2- 172.21.15.90:0/4242415122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acd4f2e730 0x55acd5301b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.260+0000 7fe09cdec640 1 -- 172.21.15.90:0/4242415122 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55acd53020c0 con 0x55acd4f26550 2024-04-25T11:39:13.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.260+0000 7fe09d5ed640 1 --2- 172.21.15.90:0/4242415122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd4f0e1f0 0x55acd53062f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:39:13.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.261+0000 7fe09ddee640 1 --2- 172.21.15.90:0/4242415122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acd4f2e730 0x55acd5301b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:13.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.261+0000 7fe09cdec640 1 --2- 172.21.15.90:0/4242415122 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acd4f26550 0x55acd5301640 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe0880dcd90 tx=0x7fe0880a6da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1a93754d23ceef0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:13.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.261+0000 7fe080ff9640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:39:13.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.261+0000 7fe080ff9640 1 -- 172.21.15.90:0/4242415122 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0880a3470 con 0x55acd4f26550 2024-04-25T11:39:13.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.261+0000 7fe080ff9640 1 -- 172.21.15.90:0/4242415122 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fe0880ab170 con 0x55acd4f26550 2024-04-25T11:39:13.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.261+0000 7fe0a67d7740 1 -- 172.21.15.90:0/4242415122 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0880093c0 con 0x55acd4f26550 2024-04-25T11:39:13.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.261+0000 7fe080ff9640 1 -- 172.21.15.90:0/4242415122 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe0880a3610 con 0x55acd4f26550 2024-04-25T11:39:13.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.261+0000 7fe0a67d7740 10 client.26045 fetch_fsmap learned FSMap version 27 2024-04-25T11:39:13.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.261+0000 7fe0a67d7740 1 -- 172.21.15.90:0/4242415122 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55acd5378360 con 0x55acd4f26550 2024-04-25T11:39:13.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.261+0000 7fe080ff9640 1 -- 172.21.15.90:0/4242415122 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0880093c0 con 0x55acd4f26550 2024-04-25T11:39:13.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.262+0000 7fe080ff9640 1 -- 172.21.15.90:0/4242415122 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7fe0880a6440 con 0x55acd4f26550 2024-04-25T11:39:13.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.262+0000 7fe0a67d7740 10 client.26045 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:39:13.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.262+0000 7fe0a67d7740 10 client.26045 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:39:13.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.262+0000 7fe0a67d7740 20 client.26045 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:13.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.262+0000 7fe0a67d7740 1 --2- 172.21.15.90:0/4242415122 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55acd4f4f410 0x55acd537aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.262+0000 7fe0a67d7740 4 client.26045 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T11:39:13.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.262+0000 7fe0a67d7740 1 -- 172.21.15.90:0/4242415122 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55acd525d580 con 0x55acd4f4f410 2024-04-25T11:39:13.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.262+0000 7fe09d5ed640 1 --2- 172.21.15.90:0/4242415122 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55acd4f4f410 0x55acd537aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.262+0000 7fe080ff9640 10 client.26045 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:13.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.262+0000 7fe09d5ed640 1 --2- 172.21.15.90:0/4242415122 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55acd4f4f410 0x55acd537aa70 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11d25f41cd8ea303 server_cookie=a93523e99abf98c0 in_seq=0 out_seq=0 2024-04-25T11:39:13.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.264+0000 7fe080ff9640 1 -- 172.21.15.90:0/4242415122 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55acd525d580 con 0x55acd4f4f410 2024-04-25T11:39:13.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.264+0000 7fe080ff9640 10 client.26045 handle_command_reply: tid=0 2024-04-25T11:39:13.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.264+0000 7fe0a67d7740 1 client.26045 shutdown 2024-04-25T11:39:13.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.264+0000 7fe0a67d7740 1 -- 172.21.15.90:0/4242415122 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acd4f26550 msgr2=0x55acd5301640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:13.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.264+0000 7fe0a67d7740 1 --2- 172.21.15.90:0/4242415122 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acd4f26550 0x55acd5301640 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe0880dcd90 tx=0x7fe0880a6da0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.264+0000 7fe0a67d7740 1 -- 172.21.15.90:0/4242415122 shutdown_connections 2024-04-25T11:39:13.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.264+0000 7fe0a67d7740 1 --2- 172.21.15.90:0/4242415122 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55acd4f4f410 0x55acd537aa70 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.264+0000 7fe0a67d7740 1 --2- 172.21.15.90:0/4242415122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acd4f2e730 0x55acd5301b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.264+0000 7fe0a67d7740 1 --2- 172.21.15.90:0/4242415122 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acd4f26550 0x55acd5301640 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.264+0000 7fe080ff9640 0 client.26045 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:13.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.264+0000 7fe0a67d7740 1 --2- 172.21.15.90:0/4242415122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd4f0e1f0 0x55acd53062f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.264+0000 7fe0a67d7740 1 -- 172.21.15.90:0/4242415122 >> 172.21.15.90:0/4242415122 conn(0x55acd4f43560 msgr2=0x55acd52efde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:13.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.264+0000 7fe0a67d7740 1 -- 172.21.15.90:0/4242415122 shutdown_connections 2024-04-25T11:39:13.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.264+0000 7fe0a67d7740 1 -- 172.21.15.90:0/4242415122 wait complete. 2024-04-25T11:39:13.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.264+0000 7fe0a67d7740 20 client.26045 trim_cache size 0 max 16384 2024-04-25T11:39:13.257 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:39:13.257 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-25T11:39:13.257 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "fb8351bb-9bb1-4281-b402-7aabcdfdc8d7", 2024-04-25T11:39:13.257 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-25T11:39:13.257 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:39:13.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.265+0000 7fe09f851640 1 -- 172.21.15.90:0/3354346246 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe0840afc60 msgr2=0x7fe0840b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:13.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.265+0000 7fe09f851640 1 --2- 172.21.15.90:0/3354346246 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe0840afc60 0x7fe0840b2170 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x55acd52f0be0 tx=0x7fe094005f90 comp rx=0 tx=0).stop 2024-04-25T11:39:13.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.265+0000 7fe09f851640 1 -- 172.21.15.90:0/3354346246 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe098108aa0 msgr2=0x7fe09819bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:13.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.265+0000 7fe09f851640 1 --2- 172.21.15.90:0/3354346246 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe098108aa0 0x7fe09819bb30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe0980754b0 tx=0x7fe08800ee50 comp rx=0 tx=0).stop 2024-04-25T11:39:13.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.265+0000 7fe09f851640 1 -- 172.21.15.90:0/3354346246 shutdown_connections 2024-04-25T11:39:13.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.265+0000 7fe09f851640 1 --2- 172.21.15.90:0/3354346246 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe0840afc60 0x7fe0840b2170 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.265+0000 7fe09f851640 1 --2- 172.21.15.90:0/3354346246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe098108f40 0x7fe09819c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.265+0000 7fe09f851640 1 --2- 172.21.15.90:0/3354346246 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe098108aa0 0x7fe09819bb30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.265+0000 7fe09f851640 1 --2- 172.21.15.90:0/3354346246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe098103850 0x7fe09819b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.265+0000 7fe09f851640 1 -- 172.21.15.90:0/3354346246 >> 172.21.15.90:0/3354346246 conn(0x7fe098077880 msgr2=0x7fe0980ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:13.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.265+0000 7fe09f851640 1 -- 172.21.15.90:0/3354346246 shutdown_connections 2024-04-25T11:39:13.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.265+0000 7fe09f851640 1 -- 172.21.15.90:0/3354346246 wait complete. 2024-04-25T11:39:13.267 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:39:13.267 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:39:13.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.405+0000 7fb90b625640 1 -- 172.21.15.90:0/3311696961 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb90410d3a0 msgr2=0x7fb90410f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:13.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.405+0000 7fb90b625640 1 --2- 172.21.15.90:0/3311696961 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb90410d3a0 0x7fb90410f7c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc009be0 tx=0x7fb8fc02a6c0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.405+0000 7fb90b625640 1 -- 172.21.15.90:0/3311696961 shutdown_connections 2024-04-25T11:39:13.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.405+0000 7fb90b625640 1 --2- 172.21.15.90:0/3311696961 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb90410d3a0 0x7fb90410f7c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.405+0000 7fb90b625640 1 --2- 172.21.15.90:0/3311696961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb904102b10 0x7fb904102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.405+0000 7fb90b625640 1 --2- 172.21.15.90:0/3311696961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb904108a40 0x7fb9041025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.405+0000 7fb90b625640 1 -- 172.21.15.90:0/3311696961 >> 172.21.15.90:0/3311696961 conn(0x7fb904078180 msgr2=0x7fb9040785c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:13.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.405+0000 7fb90b625640 1 -- 172.21.15.90:0/3311696961 shutdown_connections 2024-04-25T11:39:13.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.405+0000 7fb90b625640 1 -- 172.21.15.90:0/3311696961 wait complete. 2024-04-25T11:39:13.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.406+0000 7fb90b625640 1 Processor -- start 2024-04-25T11:39:13.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.406+0000 7fb90b625640 1 -- start start 2024-04-25T11:39:13.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.406+0000 7fb90b625640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb904102b10 0x7fb90419d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.406+0000 7fb90b625640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb904108a40 0x7fb90419db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.406+0000 7fb90b625640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb90410d3a0 0x7fb9041a8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.406+0000 7fb90b625640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb904113d30 con 0x7fb904102b10 2024-04-25T11:39:13.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.406+0000 7fb90b625640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb904113bb0 con 0x7fb90410d3a0 2024-04-25T11:39:13.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.406+0000 7fb90b625640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb904113eb0 con 0x7fb904108a40 2024-04-25T11:39:13.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.407+0000 7fb909bc2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb90410d3a0 0x7fb9041a8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.407+0000 7fb909bc2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb90410d3a0 0x7fb9041a8690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:50670/0 (socket says 172.21.15.90:50670) 2024-04-25T11:39:13.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.407+0000 7fb9093c1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb904102b10 0x7fb90419d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.407+0000 7fb909bc2640 1 -- 172.21.15.90:0/1921219526 learned_addr learned my addr 172.21.15.90:0/1921219526 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:13.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.407+0000 7fb908bc0640 1 --2- 172.21.15.90:0/1921219526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb904108a40 0x7fb90419db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.407+0000 7fb9093c1640 1 -- 172.21.15.90:0/1921219526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb904108a40 msgr2=0x7fb90419db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:13.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.407+0000 7fb9093c1640 1 --2- 172.21.15.90:0/1921219526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb904108a40 0x7fb90419db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.407+0000 7fb9093c1640 1 -- 172.21.15.90:0/1921219526 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb90410d3a0 msgr2=0x7fb9041a8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:13.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.407+0000 7fb9093c1640 1 --2- 172.21.15.90:0/1921219526 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb90410d3a0 0x7fb9041a8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.407+0000 7fb9093c1640 1 -- 172.21.15.90:0/1921219526 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9041a8bd0 con 0x7fb904102b10 2024-04-25T11:39:13.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.407+0000 7fb909bc2640 1 --2- 172.21.15.90:0/1921219526 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb90410d3a0 0x7fb9041a8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:13.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.407+0000 7fb9093c1640 1 --2- 172.21.15.90:0/1921219526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb904102b10 0x7fb90419d610 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fb90000e9c0 tx=0x7fb90000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:13.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.407+0000 7fb8fa7fc640 1 -- 172.21.15.90:0/1921219526 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90000ccc0 con 0x7fb904102b10 2024-04-25T11:39:13.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.408+0000 7fb8fa7fc640 1 -- 172.21.15.90:0/1921219526 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb900005020 con 0x7fb904102b10 2024-04-25T11:39:13.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.408+0000 7fb90b625640 1 -- 172.21.15.90:0/1921219526 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9041a8e00 con 0x7fb904102b10 2024-04-25T11:39:13.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.408+0000 7fb8fa7fc640 1 -- 172.21.15.90:0/1921219526 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90000ce60 con 0x7fb904102b10 2024-04-25T11:39:13.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.408+0000 7fb90b625640 1 -- 172.21.15.90:0/1921219526 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9041a9270 con 0x7fb904102b10 2024-04-25T11:39:13.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.410+0000 7fb9125a9740 1 Processor -- start 2024-04-25T11:39:13.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.410+0000 7fb9125a9740 1 -- start start 2024-04-25T11:39:13.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.411+0000 7fb9125a9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560cacfd9390 0x560cad379a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.411+0000 7fb9125a9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cacfb04d0 0x560cad379fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.411+0000 7fb9125a9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560cacfb86b0 0x560cad3868f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.411+0000 7fb9125a9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560cad2dfd80 con 0x560cacfb04d0 2024-04-25T11:39:13.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.411+0000 7fb9125a9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x560cad2998b0 con 0x560cacfb86b0 2024-04-25T11:39:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.411+0000 7fb9125a9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560cad2dbd40 con 0x560cacfd9390 2024-04-25T11:39:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.411+0000 7fb908bc0640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560cacfb86b0 0x560cad3868f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.411+0000 7fb908bc0640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560cacfb86b0 0x560cad3868f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:50672/0 (socket says 172.21.15.90:50672) 2024-04-25T11:39:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.411+0000 7fb908bc0640 1 -- 172.21.15.90:0/2465331941 learned_addr learned my addr 172.21.15.90:0/2465331941 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.411+0000 7fb909bc2640 1 --2- 172.21.15.90:0/2465331941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560cacfd9390 0x560cad379a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.411+0000 7fb908bc0640 1 -- 172.21.15.90:0/2465331941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560cacfd9390 msgr2=0x560cad379a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.411+0000 7fb908bc0640 1 --2- 172.21.15.90:0/2465331941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560cacfd9390 0x560cad379a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.411+0000 7fb908bc0640 1 -- 172.21.15.90:0/2465331941 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cacfb04d0 msgr2=0x560cad379fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:39:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.411+0000 7fb908bc0640 1 --2- 172.21.15.90:0/2465331941 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cacfb04d0 0x560cad379fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.411+0000 7fb8fa7fc640 1 -- 172.21.15.90:0/1921219526 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fb900004ac0 con 0x7fb904102b10 2024-04-25T11:39:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.411+0000 7fb908bc0640 1 -- 172.21.15.90:0/2465331941 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cad26aa40 con 0x560cacfb86b0 2024-04-25T11:39:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.412+0000 7fb908bc0640 1 --2- 172.21.15.90:0/2465331941 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560cacfb86b0 0x560cad3868f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4009c30 tx=0x7fb8f4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7e76cf255a9038d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.412+0000 7fb8ef7fe640 1 -- 172.21.15.90:0/2465331941 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f401e070 con 0x560cacfb86b0 2024-04-25T11:39:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.412+0000 7fb8ef7fe640 1 -- 172.21.15.90:0/2465331941 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8f4009de0 con 0x560cacfb86b0 2024-04-25T11:39:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.412+0000 7fb8ef7fe640 1 -- 172.21.15.90:0/2465331941 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f4019450 con 0x560cacfb86b0 2024-04-25T11:39:13.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.412+0000 7fb8fa7fc640 1 --2- 172.21.15.90:0/1921219526 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8f00afc60 0x7fb8f00b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.412+0000 7fb909bc2640 1 --2- 172.21.15.90:0/1921219526 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8f00afc60 0x7fb8f00b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.412+0000 7fb8fa7fc640 1 -- 172.21.15.90:0/1921219526 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb9000def70 con 0x7fb904102b10 2024-04-25T11:39:13.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.413+0000 7fb9125a9740 1 -- 172.21.15.90:0/2465331941 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560cacfb86b0 msgr2=0x560cad3868f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:13.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.413+0000 7fb909bc2640 1 --2- 172.21.15.90:0/1921219526 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8f00afc60 0x7fb8f00b2170 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc009420 tx=0x7fb8fc004a40 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:13.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.413+0000 7fb9125a9740 1 --2- 172.21.15.90:0/2465331941 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560cacfb86b0 0x560cad3868f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4009c30 tx=0x7fb8f4011810 comp rx=0 tx=0).stop 2024-04-25T11:39:13.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.413+0000 7fb9125a9740 1 -- 172.21.15.90:0/2465331941 shutdown_connections 2024-04-25T11:39:13.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.413+0000 7fb9125a9740 1 --2- 172.21.15.90:0/2465331941 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560cacfb86b0 0x560cad3868f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.413+0000 7fb9125a9740 1 --2- 172.21.15.90:0/2465331941 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cacfb04d0 0x560cad379fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.413+0000 7fb9125a9740 1 --2- 172.21.15.90:0/2465331941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560cacfd9390 0x560cad379a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.413+0000 7fb9125a9740 1 -- 172.21.15.90:0/2465331941 >> 172.21.15.90:0/2465331941 conn(0x560cacfcd4e0 msgr2=0x560cad376210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:13.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.413+0000 7fb9125a9740 1 -- 172.21.15.90:0/2465331941 shutdown_connections 2024-04-25T11:39:13.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.413+0000 7fb9125a9740 1 -- 172.21.15.90:0/2465331941 wait complete. 2024-04-25T11:39:13.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.413+0000 7fb9125a9740 1 -- start start 2024-04-25T11:39:13.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.414+0000 7fb9125a9740 1 Processor -- start 2024-04-25T11:39:13.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.414+0000 7fb9125a9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cacfb04d0 0x560cad3845d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.414+0000 7fb9125a9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560cacfb86b0 0x560cad384b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.414+0000 7fb9125a9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560cacfd9390 0x560cad38a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.414+0000 7fb9125a9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560cad386fe0 con 0x560cacfb04d0 2024-04-25T11:39:13.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.414+0000 7fb9125a9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x560cad266d10 con 0x560cacfd9390 2024-04-25T11:39:13.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.414+0000 7fb9125a9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560cad387160 con 0x560cacfb86b0 2024-04-25T11:39:13.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.414+0000 7fb9125a9740 10 client.0 _ll_register_callbacks cb 0x560cad293210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:13.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.414+0000 7fb9093c1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cacfb04d0 0x560cad3845d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.414+0000 7fb9093c1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cacfb04d0 0x560cad3845d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44606/0 (socket says 172.21.15.90:44606) 2024-04-25T11:39:13.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.414+0000 7fb9093c1640 1 -- 172.21.15.90:0/2760274775 learned_addr learned my addr 172.21.15.90:0/2760274775 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:13.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.414+0000 7fb909bc2640 1 --2- 172.21.15.90:0/2760274775 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560cacfd9390 0x560cad38a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.414+0000 7fb908bc0640 1 --2- 172.21.15.90:0/2760274775 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560cacfb86b0 0x560cad384b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.415+0000 7fb9093c1640 1 -- 172.21.15.90:0/2760274775 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560cacfb86b0 msgr2=0x560cad384b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:13.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.415+0000 7fb9093c1640 1 --2- 172.21.15.90:0/2760274775 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560cacfb86b0 0x560cad384b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.415+0000 7fb9093c1640 1 -- 172.21.15.90:0/2760274775 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560cacfd9390 msgr2=0x560cad38a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:13.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.415+0000 7fb9093c1640 1 --2- 172.21.15.90:0/2760274775 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560cacfd9390 0x560cad38a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.415+0000 7fb9093c1640 1 -- 172.21.15.90:0/2760274775 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cad0bb7d0 con 0x560cacfb04d0 2024-04-25T11:39:13.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.415+0000 7fb9093c1640 1 --2- 172.21.15.90:0/2760274775 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cacfb04d0 0x560cad3845d0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fb90000c6a0 tx=0x7fb9000a6cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6954aed1c7f17158 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:13.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.415+0000 7fb8ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:13.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.415+0000 7fb8ecff9640 1 -- 172.21.15.90:0/2760274775 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90001c040 con 0x560cacfb04d0 2024-04-25T11:39:13.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.415+0000 7fb8ecff9640 1 -- 172.21.15.90:0/2760274775 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb9000051d0 con 0x560cacfb04d0 2024-04-25T11:39:13.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.415+0000 7fb9125a9740 1 -- 172.21.15.90:0/2760274775 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8f4009de0 con 0x560cacfb04d0 2024-04-25T11:39:13.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.415+0000 7fb8ecff9640 1 -- 172.21.15.90:0/2760274775 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb9000aff10 con 0x560cacfb04d0 2024-04-25T11:39:13.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.415+0000 7fb9125a9740 10 client.16884 fetch_fsmap learned FSMap version 27 2024-04-25T11:39:13.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.415+0000 7fb9125a9740 1 -- 172.21.15.90:0/2760274775 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560cacff13f0 con 0x560cacfb04d0 2024-04-25T11:39:13.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.416+0000 7fb8ecff9640 1 -- 172.21.15.90:0/2760274775 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8f4009de0 con 0x560cacfb04d0 2024-04-25T11:39:13.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.416+0000 7fb8ecff9640 1 -- 172.21.15.90:0/2760274775 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7fb90001f030 con 0x560cacfb04d0 2024-04-25T11:39:13.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.416+0000 7fb9125a9740 10 client.16884 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:39:13.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.416+0000 7fb9125a9740 10 client.16884 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:39:13.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.416+0000 7fb9125a9740 20 client.16884 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:13.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.416+0000 7fb9125a9740 1 --2- 172.21.15.90:0/2760274775 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x560cacf97f60 0x560cad396770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.416+0000 7fb9125a9740 4 client.16884 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:39:13.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.416+0000 7fb9125a9740 1 -- 172.21.15.90:0/2760274775 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560cad2ed990 con 0x560cacf97f60 2024-04-25T11:39:13.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.416+0000 7fb908bc0640 1 --2- 172.21.15.90:0/2760274775 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x560cacf97f60 0x560cad396770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.416+0000 7fb8ecff9640 10 client.16884 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:13.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.416+0000 7fb908bc0640 1 --2- 172.21.15.90:0/2760274775 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x560cacf97f60 0x560cad396770 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8eab140e48b99594 server_cookie=20d02fbb049b2203 in_seq=0 out_seq=0 2024-04-25T11:39:13.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.417+0000 7fb8ecff9640 1 -- 172.21.15.90:0/2760274775 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560cad2ed990 con 0x560cacf97f60 2024-04-25T11:39:13.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.417+0000 7fb8ecff9640 10 client.16884 handle_command_reply: tid=0 2024-04-25T11:39:13.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.417+0000 7fb9125a9740 1 client.16884 shutdown 2024-04-25T11:39:13.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.417+0000 7fb9125a9740 1 -- 172.21.15.90:0/2760274775 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cacfb04d0 msgr2=0x560cad3845d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:13.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.417+0000 7fb9125a9740 1 --2- 172.21.15.90:0/2760274775 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cacfb04d0 0x560cad3845d0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fb90000c6a0 tx=0x7fb9000a6cd0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.417+0000 7fb9125a9740 1 -- 172.21.15.90:0/2760274775 shutdown_connections 2024-04-25T11:39:13.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.417+0000 7fb9125a9740 1 --2- 172.21.15.90:0/2760274775 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x560cacf97f60 0x560cad396770 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.417+0000 7fb9125a9740 1 --2- 172.21.15.90:0/2760274775 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560cacfd9390 0x560cad38a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.417+0000 7fb8ecff9640 0 client.16884 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:13.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.417+0000 7fb9125a9740 1 --2- 172.21.15.90:0/2760274775 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560cacfb86b0 0x560cad384b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.417+0000 7fb9125a9740 1 --2- 172.21.15.90:0/2760274775 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cacfb04d0 0x560cad3845d0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.417+0000 7fb9125a9740 1 -- 172.21.15.90:0/2760274775 >> 172.21.15.90:0/2760274775 conn(0x560cacfcd4e0 msgr2=0x560cad37b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:13.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.417+0000 7fb9125a9740 1 -- 172.21.15.90:0/2760274775 shutdown_connections 2024-04-25T11:39:13.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.418+0000 7fb9125a9740 1 -- 172.21.15.90:0/2760274775 wait complete. 2024-04-25T11:39:13.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.418+0000 7fb9125a9740 20 client.16884 trim_cache size 0 max 16384 2024-04-25T11:39:13.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.437+0000 7fb9125a9740 1 Processor -- start 2024-04-25T11:39:13.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.437+0000 7fb9125a9740 1 -- start start 2024-04-25T11:39:13.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.437+0000 7fb9125a9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560cacf97f60 0x560cad383270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.437+0000 7fb9125a9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cacfb04d0 0x560cad39bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.437+0000 7fb9125a9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560cacfb86b0 0x560cad39c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.437+0000 7fb9125a9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb9041b1d60 con 0x560cacfb04d0 2024-04-25T11:39:13.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.437+0000 7fb9125a9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb9041b38d0 con 0x560cacfb86b0 2024-04-25T11:39:13.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.437+0000 7fb9125a9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb9041b2430 con 0x560cacf97f60 2024-04-25T11:39:13.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.437+0000 7fb908bc0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cacfb04d0 0x560cad39bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.437+0000 7fb909bc2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560cacfb86b0 0x560cad39c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.438+0000 7fb909bc2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560cacfb86b0 0x560cad39c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:50690/0 (socket says 172.21.15.90:50690) 2024-04-25T11:39:13.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.438+0000 7fb909bc2640 1 -- 172.21.15.90:0/3052563378 learned_addr learned my addr 172.21.15.90:0/3052563378 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:13.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.438+0000 7fb908bc0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cacfb04d0 0x560cad39bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44610/0 (socket says 172.21.15.90:44610) 2024-04-25T11:39:13.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.438+0000 7fb9093c1640 1 --2- 172.21.15.90:0/3052563378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560cacf97f60 0x560cad383270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.438+0000 7fb909bc2640 1 -- 172.21.15.90:0/3052563378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560cacf97f60 msgr2=0x560cad383270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:13.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.438+0000 7fb909bc2640 1 --2- 172.21.15.90:0/3052563378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560cacf97f60 0x560cad383270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.438+0000 7fb909bc2640 1 -- 172.21.15.90:0/3052563378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cacfb04d0 msgr2=0x560cad39bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:13.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.438+0000 7fb909bc2640 1 --2- 172.21.15.90:0/3052563378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cacfb04d0 0x560cad39bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.438+0000 7fb909bc2640 1 -- 172.21.15.90:0/3052563378 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cacfb10f0 con 0x560cacfb86b0 2024-04-25T11:39:13.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.438+0000 7fb909bc2640 1 --2- 172.21.15.90:0/3052563378 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560cacfb86b0 0x560cad39c440 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc034040 tx=0x7fb8fc03f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98e1bb1ab4014f2b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:13.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.438+0000 7fb8eeffd640 1 -- 172.21.15.90:0/3052563378 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc036040 con 0x560cacfb86b0 2024-04-25T11:39:13.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.438+0000 7fb8eeffd640 1 -- 172.21.15.90:0/3052563378 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8fc003e50 con 0x560cacfb86b0 2024-04-25T11:39:13.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.438+0000 7fb8eeffd640 1 -- 172.21.15.90:0/3052563378 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc031960 con 0x560cacfb86b0 2024-04-25T11:39:13.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.440+0000 7fb9125a9740 1 -- 172.21.15.90:0/3052563378 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560cacfb86b0 msgr2=0x560cad39c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:13.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.440+0000 7fb9125a9740 1 --2- 172.21.15.90:0/3052563378 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560cacfb86b0 0x560cad39c440 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc034040 tx=0x7fb8fc03f000 comp rx=0 tx=0).stop 2024-04-25T11:39:13.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.440+0000 7fb9125a9740 1 -- 172.21.15.90:0/3052563378 shutdown_connections 2024-04-25T11:39:13.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.440+0000 7fb9125a9740 1 --2- 172.21.15.90:0/3052563378 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560cacfb86b0 0x560cad39c440 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.440+0000 7fb9125a9740 1 --2- 172.21.15.90:0/3052563378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cacfb04d0 0x560cad39bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.440+0000 7fb9125a9740 1 --2- 172.21.15.90:0/3052563378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560cacf97f60 0x560cad383270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.440+0000 7fb9125a9740 1 -- 172.21.15.90:0/3052563378 >> 172.21.15.90:0/3052563378 conn(0x560cacfcd4e0 msgr2=0x560cad382b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:13.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.440+0000 7fb9125a9740 1 -- 172.21.15.90:0/3052563378 shutdown_connections 2024-04-25T11:39:13.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.440+0000 7fb9125a9740 1 -- 172.21.15.90:0/3052563378 wait complete. 2024-04-25T11:39:13.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.441+0000 7fb9125a9740 1 -- start start 2024-04-25T11:39:13.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.441+0000 7fb9125a9740 1 Processor -- start 2024-04-25T11:39:13.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.442+0000 7fb9125a9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560cacf97f60 0x560cad390ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.442+0000 7fb9125a9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560cacfb04d0 0x560cad38c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.442+0000 7fb9125a9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cacfb86b0 0x560cad38c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.442+0000 7fb9125a9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb9041b31d0 con 0x560cacfb86b0 2024-04-25T11:39:13.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.442+0000 7fb9125a9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb9041b2b00 con 0x560cacfb04d0 2024-04-25T11:39:13.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.442+0000 7fb9125a9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb9041afd40 con 0x560cacf97f60 2024-04-25T11:39:13.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.442+0000 7fb908bc0640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560cacfb04d0 0x560cad38c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.442+0000 7fb9093c1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560cacf97f60 0x560cad390ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.442+0000 7fb908bc0640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560cacfb04d0 0x560cad38c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:50706/0 (socket says 172.21.15.90:50706) 2024-04-25T11:39:13.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.442+0000 7fb908bc0640 1 -- 172.21.15.90:0/2669834055 learned_addr learned my addr 172.21.15.90:0/2669834055 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:13.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.442+0000 7fb909bc2640 1 --2- 172.21.15.90:0/2669834055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cacfb86b0 0x560cad38c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.442+0000 7fb9125a9740 10 client.0 _ll_register_callbacks cb 0x7fb9000a2f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:13.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.442+0000 7fb9093c1640 1 -- 172.21.15.90:0/2669834055 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560cacfb04d0 msgr2=0x560cad38c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:13.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.442+0000 7fb9093c1640 1 --2- 172.21.15.90:0/2669834055 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560cacfb04d0 0x560cad38c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.442+0000 7fb9093c1640 1 -- 172.21.15.90:0/2669834055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cacfb86b0 msgr2=0x560cad38c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:13.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.442+0000 7fb9093c1640 1 --2- 172.21.15.90:0/2669834055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cacfb86b0 0x560cad38c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.442+0000 7fb9093c1640 1 -- 172.21.15.90:0/2669834055 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cad2a04b0 con 0x560cacf97f60 2024-04-25T11:39:13.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.442+0000 7fb908bc0640 1 --2- 172.21.15.90:0/2669834055 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560cacfb04d0 0x560cad38c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:39:13.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.443+0000 7fb9093c1640 1 --2- 172.21.15.90:0/2669834055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560cacf97f60 0x560cad390ce0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb9000a7040 tx=0x7fb9000a2e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=900bb0969ec06e4d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:13.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.443+0000 7fb8ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:39:13.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.443+0000 7fb8ecff9640 1 -- 172.21.15.90:0/2669834055 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90001c070 con 0x560cacf97f60 2024-04-25T11:39:13.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.443+0000 7fb8ecff9640 1 -- 172.21.15.90:0/2669834055 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb9000a3340 con 0x560cacf97f60 2024-04-25T11:39:13.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.443+0000 7fb9125a9740 1 -- 172.21.15.90:0/2669834055 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8fc003e50 con 0x560cacf97f60 2024-04-25T11:39:13.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.443+0000 7fb8ecff9640 1 -- 172.21.15.90:0/2669834055 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb9000a6820 con 0x560cacf97f60 2024-04-25T11:39:13.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.443+0000 7fb9125a9740 10 client.26005 fetch_fsmap learned FSMap version 27 2024-04-25T11:39:13.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.443+0000 7fb9125a9740 1 -- 172.21.15.90:0/2669834055 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560cad3fe4e0 con 0x560cacf97f60 2024-04-25T11:39:13.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.445+0000 7fb8ecff9640 1 -- 172.21.15.90:0/2669834055 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8fc003e50 con 0x560cacf97f60 2024-04-25T11:39:13.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.445+0000 7fb8ecff9640 1 -- 172.21.15.90:0/2669834055 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 27) ==== 8358+0+0 (secure 0 0 0) 0x7fb90001f040 con 0x560cacf97f60 2024-04-25T11:39:13.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.445+0000 7fb9125a9740 10 client.26005 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T11:39:13.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.445+0000 7fb9125a9740 10 client.26005 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:39:13.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.445+0000 7fb9125a9740 20 client.26005 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:13.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.445+0000 7fb9125a9740 1 --2- 172.21.15.90:0/2669834055 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x560cacfd9390 0x560cad400b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.445+0000 7fb9125a9740 4 client.26005 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:39:13.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.445+0000 7fb9125a9740 1 -- 172.21.15.90:0/2669834055 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560cad2e7800 con 0x560cacfd9390 2024-04-25T11:39:13.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.446+0000 7fb908bc0640 1 --2- 172.21.15.90:0/2669834055 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x560cacfd9390 0x560cad400b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.449+0000 7fb8ecff9640 10 client.26005 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:13.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.449+0000 7fb908bc0640 1 --2- 172.21.15.90:0/2669834055 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x560cacfd9390 0x560cad400b10 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f468124be775face server_cookie=204ef99040897810 in_seq=0 out_seq=0 2024-04-25T11:39:13.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.450+0000 7fb8ecff9640 1 -- 172.21.15.90:0/2669834055 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x560cad2e7800 con 0x560cacfd9390 2024-04-25T11:39:13.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.450+0000 7fb8ecff9640 10 client.26005 handle_command_reply: tid=0 2024-04-25T11:39:13.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.450+0000 7fb9125a9740 1 client.26005 shutdown 2024-04-25T11:39:13.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.450+0000 7fb9125a9740 1 -- 172.21.15.90:0/2669834055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560cacf97f60 msgr2=0x560cad390ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:13.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.450+0000 7fb9125a9740 1 --2- 172.21.15.90:0/2669834055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560cacf97f60 0x560cad390ce0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb9000a7040 tx=0x7fb9000a2e80 comp rx=0 tx=0).stop 2024-04-25T11:39:13.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.450+0000 7fb9125a9740 1 -- 172.21.15.90:0/2669834055 shutdown_connections 2024-04-25T11:39:13.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.450+0000 7fb9125a9740 1 --2- 172.21.15.90:0/2669834055 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x560cacfd9390 0x560cad400b10 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.451+0000 7fb8ecff9640 0 client.26005 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:13.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.451+0000 7fb9125a9740 1 --2- 172.21.15.90:0/2669834055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cacfb86b0 0x560cad38c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.451+0000 7fb9125a9740 1 --2- 172.21.15.90:0/2669834055 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560cacfb04d0 0x560cad38c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.451+0000 7fb9125a9740 1 --2- 172.21.15.90:0/2669834055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560cacf97f60 0x560cad390ce0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.451+0000 7fb9125a9740 1 -- 172.21.15.90:0/2669834055 >> 172.21.15.90:0/2669834055 conn(0x560cacfcd4e0 msgr2=0x560cad3862b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:13.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.451+0000 7fb9125a9740 1 -- 172.21.15.90:0/2669834055 shutdown_connections 2024-04-25T11:39:13.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.451+0000 7fb9125a9740 1 -- 172.21.15.90:0/2669834055 wait complete. 2024-04-25T11:39:13.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.451+0000 7fb9125a9740 20 client.26005 trim_cache size 0 max 16384 2024-04-25T11:39:13.440 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:39:13.440 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "scrub active (0 inodes in the stack)", 2024-04-25T11:39:13.440 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-25T11:39:13.440 INFO:teuthology.orchestra.run.smithi090.stdout: "fb8351bb-9bb1-4281-b402-7aabcdfdc8d7": { 2024-04-25T11:39:13.440 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-25T11:39:13.440 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "fb8351bb-9bb1-4281-b402-7aabcdfdc8d7", 2024-04-25T11:39:13.440 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-25T11:39:13.441 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:39:13.441 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:39:13.441 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:39:13.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.452+0000 7fb90b625640 1 -- 172.21.15.90:0/1921219526 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8f00afc60 msgr2=0x7fb8f00b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:13.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.452+0000 7fb90b625640 1 --2- 172.21.15.90:0/1921219526 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8f00afc60 0x7fb8f00b2170 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc009420 tx=0x7fb8fc004a40 comp rx=0 tx=0).stop 2024-04-25T11:39:13.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.452+0000 7fb90b625640 1 -- 172.21.15.90:0/1921219526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb904102b10 msgr2=0x7fb90419d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:13.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.452+0000 7fb90b625640 1 --2- 172.21.15.90:0/1921219526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb904102b10 0x7fb90419d610 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fb90000e9c0 tx=0x7fb90000edc0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.452+0000 7fb90b625640 1 -- 172.21.15.90:0/1921219526 shutdown_connections 2024-04-25T11:39:13.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.453+0000 7fb90b625640 1 --2- 172.21.15.90:0/1921219526 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8f00afc60 0x7fb8f00b2170 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.453+0000 7fb90b625640 1 --2- 172.21.15.90:0/1921219526 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb90410d3a0 0x7fb9041a8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.453+0000 7fb90b625640 1 --2- 172.21.15.90:0/1921219526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb904108a40 0x7fb90419db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.453+0000 7fb90b625640 1 --2- 172.21.15.90:0/1921219526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb904102b10 0x7fb90419d610 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:13.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.453+0000 7fb90b625640 1 -- 172.21.15.90:0/1921219526 >> 172.21.15.90:0/1921219526 conn(0x7fb904078180 msgr2=0x7fb9041011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:13.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.453+0000 7fb90b625640 1 -- 172.21.15.90:0/1921219526 shutdown_connections 2024-04-25T11:39:13.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:13.453+0000 7fb90b625640 1 -- 172.21.15.90:0/1921219526 wait complete. 2024-04-25T11:39:13.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:13 smithi090 ceph-mon[29575]: pgmap v127: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 791 KiB/s rd, 383 MiB/s wr, 831 op/s 2024-04-25T11:39:13.474 INFO:tasks.cephfs.filesystem:scrub status for tag:fb8351bb-9bb1-4281-b402-7aabcdfdc8d7 - {'path': '/', 'tag': 'fb8351bb-9bb1-4281-b402-7aabcdfdc8d7', 'options': 'recursive,force'} 2024-04-25T11:39:13.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:13 smithi138 ceph-mon[32328]: pgmap v127: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 791 KiB/s rd, 383 MiB/s wr, 831 op/s 2024-04-25T11:39:13.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:13 smithi156 ceph-mon[33045]: pgmap v127: 97 pgs: 97 active+clean; 30 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 791 KiB/s rd, 383 MiB/s wr, 831 op/s 2024-04-25T11:39:14.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:14 smithi090 ceph-mon[29575]: scrub queued for path: / 2024-04-25T11:39:14.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:14 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:39:14.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:14 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:39:14.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:14 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:39:14.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:14 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:39:14.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:14 smithi138 ceph-mon[32328]: scrub queued for path: / 2024-04-25T11:39:14.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:14 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:39:14.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:14 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:39:14.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:14 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:39:14.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:14 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:39:14.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:14 smithi156 ceph-mon[33045]: scrub queued for path: / 2024-04-25T11:39:14.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:14 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:39:14.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:14 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:39:14.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:14 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:39:14.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:14 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:39:15.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:15 smithi156 ceph-mon[33045]: pgmap v128: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 977 GiB / 1.0 TiB avail; 1.0 MiB/s rd, 481 MiB/s wr, 1.20k op/s 2024-04-25T11:39:15.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:15 smithi090 ceph-mon[29575]: pgmap v128: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 977 GiB / 1.0 TiB avail; 1.0 MiB/s rd, 481 MiB/s wr, 1.20k op/s 2024-04-25T11:39:16.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:15 smithi138 ceph-mon[32328]: pgmap v128: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 977 GiB / 1.0 TiB avail; 1.0 MiB/s rd, 481 MiB/s wr, 1.20k op/s 2024-04-25T11:39:17.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:17 smithi138 ceph-mon[32328]: pgmap v129: 97 pgs: 97 active+clean; 33 GiB data, 101 GiB used, 971 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 601 MiB/s wr, 1.85k op/s 2024-04-25T11:39:17.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:17 smithi156 ceph-mon[33045]: pgmap v129: 97 pgs: 97 active+clean; 33 GiB data, 101 GiB used, 971 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 601 MiB/s wr, 1.85k op/s 2024-04-25T11:39:17.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:17 smithi090 ceph-mon[29575]: pgmap v129: 97 pgs: 97 active+clean; 33 GiB data, 101 GiB used, 971 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 601 MiB/s wr, 1.85k op/s 2024-04-25T11:39:18.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:18 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:39:18.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:18 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:39:19.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:18 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:39:19.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:19 smithi156 ceph-mon[33045]: pgmap v130: 97 pgs: 97 active+clean; 33 GiB data, 101 GiB used, 971 GiB / 1.0 TiB avail; 1017 KiB/s rd, 461 MiB/s wr, 1.60k op/s 2024-04-25T11:39:19.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:19 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:39:19.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:19 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:39:19.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:19 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:39:19.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:19 smithi090 ceph-mon[29575]: pgmap v130: 97 pgs: 97 active+clean; 33 GiB data, 101 GiB used, 971 GiB / 1.0 TiB avail; 1017 KiB/s rd, 461 MiB/s wr, 1.60k op/s 2024-04-25T11:39:19.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:19 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:39:19.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:19 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:39:19.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:19 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:39:20.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:19 smithi138 ceph-mon[32328]: pgmap v130: 97 pgs: 97 active+clean; 33 GiB data, 101 GiB used, 971 GiB / 1.0 TiB avail; 1017 KiB/s rd, 461 MiB/s wr, 1.60k op/s 2024-04-25T11:39:20.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:19 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:39:20.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:19 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:39:20.052 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:19 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:39:21.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:21 smithi156 ceph-mon[33045]: pgmap v131: 97 pgs: 97 active+clean; 33 GiB data, 102 GiB used, 971 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 604 MiB/s wr, 2.82k op/s 2024-04-25T11:39:21.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:21 smithi156 ceph-mon[33045]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T11:39:21.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:21 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:39:21.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:21 smithi090 ceph-mon[29575]: pgmap v131: 97 pgs: 97 active+clean; 33 GiB data, 102 GiB used, 971 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 604 MiB/s wr, 2.82k op/s 2024-04-25T11:39:21.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:21 smithi090 ceph-mon[29575]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T11:39:21.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:21 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:39:22.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:21 smithi138 ceph-mon[32328]: pgmap v131: 97 pgs: 97 active+clean; 33 GiB data, 102 GiB used, 971 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 604 MiB/s wr, 2.82k op/s 2024-04-25T11:39:22.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:21 smithi138 ceph-mon[32328]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T11:39:22.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:21 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:39:22.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.475+0000 7f060152d640 1 -- 172.21.15.90:0/3471626560 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f05fc11a7c0 msgr2=0x7f05fc11cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:22.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.475+0000 7f060152d640 1 --2- 172.21.15.90:0/3471626560 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f05fc11a7c0 0x7f05fc11cbe0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f05f4009ca0 tx=0x7f05f402a980 comp rx=0 tx=0).stop 2024-04-25T11:39:22.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.475+0000 7f060152d640 1 -- 172.21.15.90:0/3471626560 shutdown_connections 2024-04-25T11:39:22.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.475+0000 7f060152d640 1 --2- 172.21.15.90:0/3471626560 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05fc11d120 0x7f05fc11f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:22.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.475+0000 7f060152d640 1 --2- 172.21.15.90:0/3471626560 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f05fc11a7c0 0x7f05fc11cbe0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:22.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.475+0000 7f060152d640 1 --2- 172.21.15.90:0/3471626560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05fc109fc0 0x7f05fc10a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:22.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.475+0000 7f060152d640 1 -- 172.21.15.90:0/3471626560 >> 172.21.15.90:0/3471626560 conn(0x7f05fc06f650 msgr2=0x7f05fc06fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.478+0000 7f060152d640 1 -- 172.21.15.90:0/3471626560 shutdown_connections 2024-04-25T11:39:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.479+0000 7f060152d640 1 -- 172.21.15.90:0/3471626560 wait complete. 2024-04-25T11:39:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.480+0000 7f060152d640 1 Processor -- start 2024-04-25T11:39:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.481+0000 7f060152d640 1 -- start start 2024-04-25T11:39:22.470 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.481+0000 7f060152d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f05fc109fc0 0x7f05fc11a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:22.470 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.481+0000 7f060152d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05fc11a7c0 0x7f05fc113390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:22.470 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.481+0000 7f060152d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05fc11d120 0x7f05fc113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:22.470 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.481+0000 7f060152d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f05fc122a80 con 0x7f05fc11d120 2024-04-25T11:39:22.470 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.481+0000 7f060152d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f05fc122900 con 0x7f05fc109fc0 2024-04-25T11:39:22.470 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.481+0000 7f060152d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f05fc122c00 con 0x7f05fc11a7c0 2024-04-25T11:39:22.471 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.483+0000 7f05fa7fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05fc11a7c0 0x7f05fc113390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:22.471 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.483+0000 7f05fa7fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05fc11a7c0 0x7f05fc113390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:42142/0 (socket says 172.21.15.90:42142) 2024-04-25T11:39:22.471 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.483+0000 7f05fa7fc640 1 -- 172.21.15.90:0/2406035530 learned_addr learned my addr 172.21.15.90:0/2406035530 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:22.471 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.484+0000 7f05faffd640 1 --2- 172.21.15.90:0/2406035530 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f05fc109fc0 0x7f05fc11a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.485+0000 7f05fa7fc640 1 -- 172.21.15.90:0/2406035530 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f05fc109fc0 msgr2=0x7f05fc11a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.485+0000 7f05fa7fc640 1 --2- 172.21.15.90:0/2406035530 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f05fc109fc0 0x7f05fc11a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.485+0000 7f05fa7fc640 1 -- 172.21.15.90:0/2406035530 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05fc11d120 msgr2=0x7f05fc113a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.485+0000 7f05fa7fc640 1 --2- 172.21.15.90:0/2406035530 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05fc11d120 0x7f05fc113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.485+0000 7f05fa7fc640 1 -- 172.21.15.90:0/2406035530 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05fc117d70 con 0x7f05fc11a7c0 2024-04-25T11:39:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.486+0000 7f05fa7fc640 1 --2- 172.21.15.90:0/2406035530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05fc11a7c0 0x7f05fc113390 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f05f4009dc0 tx=0x7f05f40308b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.487+0000 7f05e3fff640 1 -- 172.21.15.90:0/2406035530 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f4034050 con 0x7f05fc11a7c0 2024-04-25T11:39:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.487+0000 7f060152d640 1 -- 172.21.15.90:0/2406035530 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05fc118000 con 0x7f05fc11a7c0 2024-04-25T11:39:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.487+0000 7f060152d640 1 -- 172.21.15.90:0/2406035530 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05fc1184f0 con 0x7f05fc11a7c0 2024-04-25T11:39:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.493+0000 7f05e3fff640 1 -- 172.21.15.90:0/2406035530 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05f40398d0 con 0x7f05fc11a7c0 2024-04-25T11:39:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.493+0000 7f05e3fff640 1 -- 172.21.15.90:0/2406035530 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f40427c0 con 0x7f05fc11a7c0 2024-04-25T11:39:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.495+0000 7f06083dd740 1 Processor -- start 2024-04-25T11:39:22.489 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.500+0000 7f05e3fff640 1 -- 172.21.15.90:0/2406035530 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f05f4037020 con 0x7f05fc11a7c0 2024-04-25T11:39:22.491 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.501+0000 7f06083dd740 1 -- start start 2024-04-25T11:39:22.491 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.501+0000 7f06083dd740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acc87691b0 0x55acc8b08ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:22.491 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.501+0000 7f06083dd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acc87402f0 0x55acc8b091e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:22.491 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.501+0000 7f06083dd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acc87484d0 0x55acc8b09720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:22.491 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.501+0000 7f06083dd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55acc8a6f460 con 0x55acc87484d0 2024-04-25T11:39:22.491 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.501+0000 7f06083dd740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55acc8a29590 con 0x55acc87691b0 2024-04-25T11:39:22.491 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.501+0000 7f06083dd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55acc8a6b420 con 0x55acc87402f0 2024-04-25T11:39:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.503+0000 7f05fa7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acc87484d0 0x55acc8b09720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.504+0000 7f05fa7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acc87484d0 0x55acc8b09720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44178/0 (socket says 172.21.15.90:44178) 2024-04-25T11:39:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.504+0000 7f05fa7fc640 1 -- 172.21.15.90:0/3151517897 learned_addr learned my addr 172.21.15.90:0/3151517897 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.505+0000 7f05e3fff640 1 --2- 172.21.15.90:0/2406035530 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f05e40afe60 0x7f05e40b2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.506+0000 7f05faffd640 1 --2- 172.21.15.90:0/3151517897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acc87402f0 0x55acc8b091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.506+0000 7f05e3fff640 1 -- 172.21.15.90:0/2406035530 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f05f4100050 con 0x7f05fc11a7c0 2024-04-25T11:39:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.506+0000 7f05fa7fc640 1 -- 172.21.15.90:0/3151517897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acc87402f0 msgr2=0x55acc8b091e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.506+0000 7f05fa7fc640 1 --2- 172.21.15.90:0/3151517897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acc87402f0 0x55acc8b091e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.506+0000 7f05fa7fc640 1 -- 172.21.15.90:0/3151517897 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acc87691b0 msgr2=0x55acc8b08ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.510+0000 7f05fb7fe640 1 --2- 172.21.15.90:0/3151517897 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acc87691b0 0x55acc8b08ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.510+0000 7f05fb7fe640 1 --2- 172.21.15.90:0/2406035530 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f05e40afe60 0x7f05e40b2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.510+0000 7f05fa7fc640 1 --2- 172.21.15.90:0/3151517897 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acc87691b0 0x55acc8b08ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.510+0000 7f05fa7fc640 1 -- 172.21.15.90:0/3151517897 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55acc8a360d0 con 0x55acc87484d0 2024-04-25T11:39:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.514+0000 7f05fa7fc640 1 --2- 172.21.15.90:0/3151517897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acc87484d0 0x55acc8b09720 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f05f40fd650 tx=0x7f05f4000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58b8b5fb2a85aece server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.515+0000 7f05fb7fe640 1 --2- 172.21.15.90:0/2406035530 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f05e40afe60 0x7f05e40b2370 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x55acc8b05e10 tx=0x7f05f0001040 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.516+0000 7f05e17fa640 1 -- 172.21.15.90:0/3151517897 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f40c78a0 con 0x55acc87484d0 2024-04-25T11:39:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.516+0000 7f05e17fa640 1 -- 172.21.15.90:0/3151517897 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05f40098d0 con 0x55acc87484d0 2024-04-25T11:39:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.517+0000 7f06083dd740 1 -- 172.21.15.90:0/3151517897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acc87484d0 msgr2=0x55acc8b09720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.517+0000 7f06083dd740 1 --2- 172.21.15.90:0/3151517897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acc87484d0 0x55acc8b09720 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f05f40fd650 tx=0x7f05f4000f40 comp rx=0 tx=0).stop 2024-04-25T11:39:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.518+0000 7f06083dd740 1 -- 172.21.15.90:0/3151517897 shutdown_connections 2024-04-25T11:39:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.518+0000 7f06083dd740 1 --2- 172.21.15.90:0/3151517897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acc87484d0 0x55acc8b09720 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.518+0000 7f06083dd740 1 --2- 172.21.15.90:0/3151517897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acc87402f0 0x55acc8b091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.518+0000 7f06083dd740 1 --2- 172.21.15.90:0/3151517897 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acc87691b0 0x55acc8b08ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.518+0000 7f06083dd740 1 -- 172.21.15.90:0/3151517897 >> 172.21.15.90:0/3151517897 conn(0x55acc875d300 msgr2=0x55acc8b05450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.519+0000 7f06083dd740 1 -- 172.21.15.90:0/3151517897 shutdown_connections 2024-04-25T11:39:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.520+0000 7f06083dd740 1 -- 172.21.15.90:0/3151517897 wait complete. 2024-04-25T11:39:22.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.520+0000 7f06083dd740 1 -- start start 2024-04-25T11:39:22.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.521+0000 7f06083dd740 1 Processor -- start 2024-04-25T11:39:22.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.522+0000 7f06083dd740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acc87402f0 0x55acc8b132d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:22.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.522+0000 7f06083dd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acc87484d0 0x55acc8b13810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:22.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.522+0000 7f06083dd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acc87691b0 0x55acc8b18ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:22.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.522+0000 7f06083dd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55acc8a6f460 con 0x55acc87484d0 2024-04-25T11:39:22.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.522+0000 7f06083dd740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55acc89f6b60 con 0x55acc87402f0 2024-04-25T11:39:22.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.522+0000 7f06083dd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55acc8b15b90 con 0x55acc87691b0 2024-04-25T11:39:22.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.522+0000 7f06083dd740 10 client.0 _ll_register_callbacks cb 0x55acc8a1b2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:22.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.523+0000 7f05fa7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acc87484d0 0x55acc8b13810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:22.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.523+0000 7f05fa7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acc87484d0 0x55acc8b13810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44188/0 (socket says 172.21.15.90:44188) 2024-04-25T11:39:22.514 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.523+0000 7f05fa7fc640 1 -- 172.21.15.90:0/4029567067 learned_addr learned my addr 172.21.15.90:0/4029567067 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:22.514 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.523+0000 7f05fb7fe640 1 --2- 172.21.15.90:0/4029567067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acc87691b0 0x55acc8b18ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:22.514 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.523+0000 7f05faffd640 1 --2- 172.21.15.90:0/4029567067 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acc87402f0 0x55acc8b132d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:22.514 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.524+0000 7f05fa7fc640 1 -- 172.21.15.90:0/4029567067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acc87691b0 msgr2=0x55acc8b18ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:22.514 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.524+0000 7f05fa7fc640 1 --2- 172.21.15.90:0/4029567067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acc87691b0 0x55acc8b18ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:22.514 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.524+0000 7f05fa7fc640 1 -- 172.21.15.90:0/4029567067 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acc87402f0 msgr2=0x55acc8b132d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:22.514 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.524+0000 7f05fa7fc640 1 --2- 172.21.15.90:0/4029567067 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acc87402f0 0x55acc8b132d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:22.514 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.524+0000 7f05fa7fc640 1 -- 172.21.15.90:0/4029567067 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55acc8781210 con 0x55acc87484d0 2024-04-25T11:39:22.514 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.525+0000 7f05fa7fc640 1 --2- 172.21.15.90:0/4029567067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acc87484d0 0x55acc8b13810 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f05f4030c70 tx=0x7f05f4103310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca63521f53f86b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:22.515 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.526+0000 7f05deff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:22.515 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.526+0000 7f05deff5640 1 -- 172.21.15.90:0/4029567067 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f40e2060 con 0x55acc87484d0 2024-04-25T11:39:22.515 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.526+0000 7f06083dd740 1 -- 172.21.15.90:0/4029567067 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55acc8a7d070 con 0x55acc87484d0 2024-04-25T11:39:22.515 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.527+0000 7f05deff5640 1 -- 172.21.15.90:0/4029567067 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f05f4103d60 con 0x55acc87484d0 2024-04-25T11:39:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.527+0000 7f05deff5640 1 -- 172.21.15.90:0/4029567067 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f05f40e9050 con 0x55acc87484d0 2024-04-25T11:39:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.527+0000 7f06083dd740 10 client.16896 fetch_fsmap learned FSMap version 28 2024-04-25T11:39:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.528+0000 7f06083dd740 1 -- 172.21.15.90:0/4029567067 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55acc884d620 con 0x55acc87484d0 2024-04-25T11:39:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.528+0000 7f05deff5640 1 -- 172.21.15.90:0/4029567067 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55acc8a7d070 con 0x55acc87484d0 2024-04-25T11:39:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.529+0000 7f05deff5640 1 -- 172.21.15.90:0/4029567067 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f05f4041070 con 0x55acc87484d0 2024-04-25T11:39:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.530+0000 7f06083dd740 10 client.16896 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:39:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.530+0000 7f06083dd740 10 client.16896 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:39:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.530+0000 7f06083dd740 20 client.16896 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.530+0000 7f06083dd740 1 --2- 172.21.15.90:0/4029567067 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55acc8727d80 0x55acc8b252c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.530+0000 7f06083dd740 4 client.16896 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:39:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.530+0000 7f06083dd740 1 -- 172.21.15.90:0/4029567067 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55acc8a76ee0 con 0x55acc8727d80 2024-04-25T11:39:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.532+0000 7f05faffd640 1 --2- 172.21.15.90:0/4029567067 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55acc8727d80 0x55acc8b252c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:22.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.542+0000 7f05faffd640 1 --2- 172.21.15.90:0/4029567067 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55acc8727d80 0x55acc8b252c0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d42cf0f8545183c server_cookie=11af8f9ef33e34c1 in_seq=0 out_seq=0 2024-04-25T11:39:22.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.543+0000 7f05deff5640 10 client.16896 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:22.537 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.548+0000 7f05deff5640 1 -- 172.21.15.90:0/4029567067 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55acc8a76ee0 con 0x55acc8727d80 2024-04-25T11:39:22.537 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.548+0000 7f05deff5640 10 client.16896 handle_command_reply: tid=0 2024-04-25T11:39:22.537 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.548+0000 7f06083dd740 1 client.16896 shutdown 2024-04-25T11:39:22.538 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.550+0000 7f06083dd740 1 -- 172.21.15.90:0/4029567067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acc87484d0 msgr2=0x55acc8b13810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.550+0000 7f06083dd740 1 --2- 172.21.15.90:0/4029567067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acc87484d0 0x55acc8b13810 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f05f4030c70 tx=0x7f05f4103310 comp rx=0 tx=0).stop 2024-04-25T11:39:22.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.552+0000 7f06083dd740 1 -- 172.21.15.90:0/4029567067 shutdown_connections 2024-04-25T11:39:22.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.552+0000 7f06083dd740 1 --2- 172.21.15.90:0/4029567067 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55acc8727d80 0x55acc8b252c0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:22.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.552+0000 7f06083dd740 1 --2- 172.21.15.90:0/4029567067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acc87691b0 0x55acc8b18ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:22.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.552+0000 7f06083dd740 1 --2- 172.21.15.90:0/4029567067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acc87484d0 0x55acc8b13810 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:22.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.552+0000 7f06083dd740 1 --2- 172.21.15.90:0/4029567067 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acc87402f0 0x55acc8b132d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:22.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.552+0000 7f06083dd740 1 -- 172.21.15.90:0/4029567067 >> 172.21.15.90:0/4029567067 conn(0x55acc875d300 msgr2=0x55acc8b0a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:22.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.552+0000 7f05deff5640 0 client.16896 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:22.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.557+0000 7f06083dd740 1 -- 172.21.15.90:0/4029567067 shutdown_connections 2024-04-25T11:39:22.546 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.557+0000 7f06083dd740 1 -- 172.21.15.90:0/4029567067 wait complete. 2024-04-25T11:39:22.546 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.557+0000 7f06083dd740 20 client.16896 trim_cache size 0 max 16384 2024-04-25T11:39:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.595+0000 7f06083dd740 1 Processor -- start 2024-04-25T11:39:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.596+0000 7f06083dd740 1 -- start start 2024-04-25T11:39:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.596+0000 7f06083dd740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acc87402f0 0x55acc8b18e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.596+0000 7f06083dd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acc87484d0 0x55acc8b0aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.597+0000 7f06083dd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acc87691b0 0x55acc8b0c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.597+0000 7f06083dd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f05e800eac0 con 0x55acc87691b0 2024-04-25T11:39:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.597+0000 7f06083dd740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f05e8010600 con 0x55acc87402f0 2024-04-25T11:39:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.597+0000 7f05faffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acc87402f0 0x55acc8b18e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.597+0000 7f05faffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acc87402f0 0x55acc8b18e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:42888/0 (socket says 172.21.15.90:42888) 2024-04-25T11:39:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.597+0000 7f05faffd640 1 -- 172.21.15.90:0/4127866152 learned_addr learned my addr 172.21.15.90:0/4127866152 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.597+0000 7f05fa7fc640 1 --2- 172.21.15.90:0/4127866152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acc87484d0 0x55acc8b0aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.597+0000 7f06083dd740 1 -- 172.21.15.90:0/4127866152 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f05e800f190 con 0x55acc87484d0 2024-04-25T11:39:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.598+0000 7f05fb7fe640 1 --2- 172.21.15.90:0/4127866152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acc87691b0 0x55acc8b0c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.598+0000 7f05fa7fc640 1 -- 172.21.15.90:0/4127866152 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acc87402f0 msgr2=0x55acc8b18e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.598+0000 7f05fa7fc640 1 --2- 172.21.15.90:0/4127866152 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acc87402f0 0x55acc8b18e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.598+0000 7f05fa7fc640 1 -- 172.21.15.90:0/4127866152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acc87691b0 msgr2=0x55acc8b0c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.598+0000 7f05fa7fc640 1 --2- 172.21.15.90:0/4127866152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acc87691b0 0x55acc8b0c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.598+0000 7f05fa7fc640 1 -- 172.21.15.90:0/4127866152 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55acc8b0b530 con 0x55acc87484d0 2024-04-25T11:39:22.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.600+0000 7f05fa7fc640 1 --2- 172.21.15.90:0/4127866152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acc87484d0 0x55acc8b0aff0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f05f402af70 tx=0x7f05f4000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96c3c0383ff4e006 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:22.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.601+0000 7f05e0ff9640 1 -- 172.21.15.90:0/4127866152 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f40e2040 con 0x55acc87484d0 2024-04-25T11:39:22.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.602+0000 7f05e0ff9640 1 -- 172.21.15.90:0/4127866152 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05f4030d10 con 0x55acc87484d0 2024-04-25T11:39:22.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.603+0000 7f05e0ff9640 1 -- 172.21.15.90:0/4127866152 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f4103810 con 0x55acc87484d0 2024-04-25T11:39:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.603+0000 7f06083dd740 1 -- 172.21.15.90:0/4127866152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acc87484d0 msgr2=0x55acc8b0aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.603+0000 7f06083dd740 1 --2- 172.21.15.90:0/4127866152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acc87484d0 0x55acc8b0aff0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f05f402af70 tx=0x7f05f4000f40 comp rx=0 tx=0).stop 2024-04-25T11:39:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.604+0000 7f06083dd740 1 -- 172.21.15.90:0/4127866152 shutdown_connections 2024-04-25T11:39:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.604+0000 7f06083dd740 1 --2- 172.21.15.90:0/4127866152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acc87691b0 0x55acc8b0c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.604+0000 7f06083dd740 1 --2- 172.21.15.90:0/4127866152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acc87484d0 0x55acc8b0aff0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.604+0000 7f06083dd740 1 --2- 172.21.15.90:0/4127866152 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acc87402f0 0x55acc8b18e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.604+0000 7f06083dd740 1 -- 172.21.15.90:0/4127866152 >> 172.21.15.90:0/4127866152 conn(0x55acc875d300 msgr2=0x55acc8b0a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.604+0000 7f06083dd740 1 -- 172.21.15.90:0/4127866152 shutdown_connections 2024-04-25T11:39:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.604+0000 7f06083dd740 1 -- 172.21.15.90:0/4127866152 wait complete. 2024-04-25T11:39:22.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.605+0000 7f06083dd740 1 -- start start 2024-04-25T11:39:22.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.605+0000 7f06083dd740 1 Processor -- start 2024-04-25T11:39:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.606+0000 7f06083dd740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acc87402f0 0x55acc8b1a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.606+0000 7f06083dd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acc87484d0 0x55acc8b1a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.607+0000 7f06083dd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acc87691b0 0x55acc8b1af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.607+0000 7f05faffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acc87402f0 0x55acc8b1a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.607+0000 7f05fa7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acc87484d0 0x55acc8b1a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.608+0000 7f05fa7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acc87484d0 0x55acc8b1a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44198/0 (socket says 172.21.15.90:44198) 2024-04-25T11:39:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.608+0000 7f05faffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acc87402f0 0x55acc8b1a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:42890/0 (socket says 172.21.15.90:42890) 2024-04-25T11:39:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.608+0000 7f05fa7fc640 1 -- 172.21.15.90:0/3206442269 learned_addr learned my addr 172.21.15.90:0/3206442269 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.608+0000 7f06083dd740 1 -- 172.21.15.90:0/3206442269 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f05e800ff30 con 0x55acc87484d0 2024-04-25T11:39:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.608+0000 7f06083dd740 1 -- 172.21.15.90:0/3206442269 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f05e800f860 con 0x55acc87402f0 2024-04-25T11:39:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.608+0000 7f06083dd740 1 -- 172.21.15.90:0/3206442269 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f05e800caa0 con 0x55acc87691b0 2024-04-25T11:39:22.597 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.609+0000 7f05fb7fe640 1 --2- 172.21.15.90:0/3206442269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acc87691b0 0x55acc8b1af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:22.597 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.609+0000 7f06083dd740 10 client.0 _ll_register_callbacks cb 0x7f05f40c40a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:22.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.610+0000 7f05fb7fe640 1 -- 172.21.15.90:0/3206442269 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acc87402f0 msgr2=0x55acc8b1a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:22.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.610+0000 7f05fb7fe640 1 --2- 172.21.15.90:0/3206442269 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acc87402f0 0x55acc8b1a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:22.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.610+0000 7f05fb7fe640 1 -- 172.21.15.90:0/3206442269 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acc87484d0 msgr2=0x55acc8b1a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:22.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.610+0000 7f05fb7fe640 1 --2- 172.21.15.90:0/3206442269 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acc87484d0 0x55acc8b1a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:22.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.610+0000 7f05fb7fe640 1 -- 172.21.15.90:0/3206442269 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55acc8b8b5d0 con 0x55acc87691b0 2024-04-25T11:39:22.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.611+0000 7f05fb7fe640 1 --2- 172.21.15.90:0/3206442269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acc87691b0 0x55acc8b1af20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x55acc8af73f0 tx=0x7f05f0000fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f153e90d3286ce6f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:22.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.611+0000 7f05deff5640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:39:22.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.613+0000 7f05deff5640 1 -- 172.21.15.90:0/3206442269 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f0017850 con 0x55acc87691b0 2024-04-25T11:39:22.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.614+0000 7f05deff5640 1 -- 172.21.15.90:0/3206442269 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f05f00179f0 con 0x55acc87691b0 2024-04-25T11:39:22.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.614+0000 7f06083dd740 1 -- 172.21.15.90:0/3206442269 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05f4030d10 con 0x55acc87691b0 2024-04-25T11:39:22.664 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.676+0000 7f05deff5640 1 -- 172.21.15.90:0/3206442269 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f05f0017bf0 con 0x55acc87691b0 2024-04-25T11:39:22.664 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.676+0000 7f06083dd740 10 client.26017 fetch_fsmap learned FSMap version 28 2024-04-25T11:39:22.664 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.676+0000 7f06083dd740 1 -- 172.21.15.90:0/3206442269 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55acc8b8c840 con 0x55acc87691b0 2024-04-25T11:39:22.664 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.677+0000 7f05deff5640 1 -- 172.21.15.90:0/3206442269 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05f0017d90 con 0x55acc87691b0 2024-04-25T11:39:22.665 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.677+0000 7f05deff5640 1 -- 172.21.15.90:0/3206442269 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f05f0023410 con 0x55acc87691b0 2024-04-25T11:39:22.665 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.677+0000 7f06083dd740 10 client.26017 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:39:22.665 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.677+0000 7f06083dd740 10 client.26017 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:39:22.665 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.677+0000 7f06083dd740 20 client.26017 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:22.665 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.677+0000 7f06083dd740 1 --2- 172.21.15.90:0/3206442269 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55acc86fd1c0 0x55acc8b8efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:22.665 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.677+0000 7f06083dd740 4 client.26017 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:39:22.665 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.677+0000 7f06083dd740 1 -- 172.21.15.90:0/3206442269 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f05f4030d10 con 0x55acc86fd1c0 2024-04-25T11:39:22.666 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.678+0000 7f05faffd640 1 --2- 172.21.15.90:0/3206442269 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55acc86fd1c0 0x55acc8b8efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:22.667 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.679+0000 7f05deff5640 10 client.26017 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:22.667 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:22.679+0000 7f05faffd640 1 --2- 172.21.15.90:0/3206442269 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55acc86fd1c0 0x55acc8b8efe0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3e19f8753ddede8 server_cookie=5fc59658cb0e6990 in_seq=0 out_seq=0 2024-04-25T11:39:24.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:23 smithi138 ceph-mon[32328]: pgmap v132: 97 pgs: 97 active+clean; 33 GiB data, 102 GiB used, 971 GiB / 1.0 TiB avail; 977 KiB/s rd, 432 MiB/s wr, 2.37k op/s 2024-04-25T11:39:24.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:23 smithi156 ceph-mon[33045]: pgmap v132: 97 pgs: 97 active+clean; 33 GiB data, 102 GiB used, 971 GiB / 1.0 TiB avail; 977 KiB/s rd, 432 MiB/s wr, 2.37k op/s 2024-04-25T11:39:24.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:23 smithi090 ceph-mon[29575]: pgmap v132: 97 pgs: 97 active+clean; 33 GiB data, 102 GiB used, 971 GiB / 1.0 TiB avail; 977 KiB/s rd, 432 MiB/s wr, 2.37k op/s 2024-04-25T11:39:26.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:25 smithi090 ceph-mon[29575]: pgmap v133: 97 pgs: 97 active+clean; 33 GiB data, 102 GiB used, 971 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 515 MiB/s wr, 2.42k op/s 2024-04-25T11:39:26.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:25 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:39:26.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:25 smithi138 ceph-mon[32328]: pgmap v133: 97 pgs: 97 active+clean; 33 GiB data, 102 GiB used, 971 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 515 MiB/s wr, 2.42k op/s 2024-04-25T11:39:26.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:25 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:39:26.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:25 smithi156 ceph-mon[33045]: pgmap v133: 97 pgs: 97 active+clean; 33 GiB data, 102 GiB used, 971 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 515 MiB/s wr, 2.42k op/s 2024-04-25T11:39:26.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:25 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:39:26.908 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.919+0000 7f05deff5640 1 -- 172.21.15.90:0/3206442269 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f05f4030d10 con 0x55acc86fd1c0 2024-04-25T11:39:26.909 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.919+0000 7f05deff5640 10 client.26017 handle_command_reply: tid=0 2024-04-25T11:39:26.909 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.919+0000 7f06083dd740 1 client.26017 shutdown 2024-04-25T11:39:26.909 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.920+0000 7f06083dd740 1 -- 172.21.15.90:0/3206442269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acc87691b0 msgr2=0x55acc8b1af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:26.909 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.920+0000 7f06083dd740 1 --2- 172.21.15.90:0/3206442269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acc87691b0 0x55acc8b1af20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x55acc8af73f0 tx=0x7f05f0000fd0 comp rx=0 tx=0).stop 2024-04-25T11:39:26.909 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.920+0000 7f06083dd740 1 -- 172.21.15.90:0/3206442269 shutdown_connections 2024-04-25T11:39:26.909 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.920+0000 7f06083dd740 1 --2- 172.21.15.90:0/3206442269 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55acc86fd1c0 0x55acc8b8efe0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:26.909 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.920+0000 7f06083dd740 1 --2- 172.21.15.90:0/3206442269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acc87691b0 0x55acc8b1af20 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:26.909 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.920+0000 7f06083dd740 1 --2- 172.21.15.90:0/3206442269 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55acc87484d0 0x55acc8b1a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:26.910 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.920+0000 7f06083dd740 1 --2- 172.21.15.90:0/3206442269 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55acc87402f0 0x55acc8b1a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:26.910 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.920+0000 7f06083dd740 1 -- 172.21.15.90:0/3206442269 >> 172.21.15.90:0/3206442269 conn(0x55acc875d300 msgr2=0x55acc8b14fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:26.910 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.920+0000 7f05deff5640 0 client.26017 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:26.910 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.920+0000 7f06083dd740 1 -- 172.21.15.90:0/3206442269 shutdown_connections 2024-04-25T11:39:26.910 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.920+0000 7f06083dd740 1 -- 172.21.15.90:0/3206442269 wait complete. 2024-04-25T11:39:26.910 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.921+0000 7f06083dd740 20 client.26017 trim_cache size 0 max 16384 2024-04-25T11:39:26.910 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:39:26.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:39:26.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:39:26.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:39:26.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.922+0000 7f060152d640 1 -- 172.21.15.90:0/2406035530 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f05e40afe60 msgr2=0x7f05e40b2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:26.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.922+0000 7f060152d640 1 --2- 172.21.15.90:0/2406035530 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f05e40afe60 0x7f05e40b2370 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x55acc8b05e10 tx=0x7f05f0001040 comp rx=0 tx=0).stop 2024-04-25T11:39:26.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.922+0000 7f060152d640 1 -- 172.21.15.90:0/2406035530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05fc11a7c0 msgr2=0x7f05fc113390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:26.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.922+0000 7f060152d640 1 --2- 172.21.15.90:0/2406035530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05fc11a7c0 0x7f05fc113390 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f05f4009dc0 tx=0x7f05f40308b0 comp rx=0 tx=0).stop 2024-04-25T11:39:26.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.923+0000 7f060152d640 1 -- 172.21.15.90:0/2406035530 shutdown_connections 2024-04-25T11:39:26.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.923+0000 7f060152d640 1 --2- 172.21.15.90:0/2406035530 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f05e40afe60 0x7f05e40b2370 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:26.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.923+0000 7f060152d640 1 --2- 172.21.15.90:0/2406035530 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05fc11d120 0x7f05fc113a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:26.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.923+0000 7f060152d640 1 --2- 172.21.15.90:0/2406035530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05fc11a7c0 0x7f05fc113390 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:26.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.923+0000 7f060152d640 1 --2- 172.21.15.90:0/2406035530 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f05fc109fc0 0x7f05fc11a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:26.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.923+0000 7f060152d640 1 -- 172.21.15.90:0/2406035530 >> 172.21.15.90:0/2406035530 conn(0x7f05fc06f650 msgr2=0x7f05fc11abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:26.914 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.924+0000 7f060152d640 1 -- 172.21.15.90:0/2406035530 shutdown_connections 2024-04-25T11:39:26.914 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:26.924+0000 7f060152d640 1 -- 172.21.15.90:0/2406035530 wait complete. 2024-04-25T11:39:27.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:27 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:39:27.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:26 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:39:27.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:26 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:39:28.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:28 smithi138 ceph-mon[32328]: pgmap v134: 97 pgs: 97 active+clean; 34 GiB data, 102 GiB used, 970 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 571 MiB/s wr, 2.10k op/s 2024-04-25T11:39:28.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:27 smithi156 ceph-mon[33045]: pgmap v134: 97 pgs: 97 active+clean; 34 GiB data, 102 GiB used, 970 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 571 MiB/s wr, 2.10k op/s 2024-04-25T11:39:28.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:27 smithi090 ceph-mon[29575]: pgmap v134: 97 pgs: 97 active+clean; 34 GiB data, 102 GiB used, 970 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 571 MiB/s wr, 2.10k op/s 2024-04-25T11:39:30.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:30 smithi138 ceph-mon[32328]: pgmap v135: 97 pgs: 97 active+clean; 34 GiB data, 102 GiB used, 970 GiB / 1.0 TiB avail; 1.3 MiB/s rd, 380 MiB/s wr, 1.33k op/s 2024-04-25T11:39:30.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:30 smithi156 ceph-mon[33045]: pgmap v135: 97 pgs: 97 active+clean; 34 GiB data, 102 GiB used, 970 GiB / 1.0 TiB avail; 1.3 MiB/s rd, 380 MiB/s wr, 1.33k op/s 2024-04-25T11:39:30.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:30 smithi090 ceph-mon[29575]: pgmap v135: 97 pgs: 97 active+clean; 34 GiB data, 102 GiB used, 970 GiB / 1.0 TiB avail; 1.3 MiB/s rd, 380 MiB/s wr, 1.33k op/s 2024-04-25T11:39:31.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:31 smithi090 ceph-mon[29575]: pgmap v136: 97 pgs: 97 active+clean; 34 GiB data, 104 GiB used, 969 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 561 MiB/s wr, 1.99k op/s 2024-04-25T11:39:31.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:31 smithi138 ceph-mon[32328]: pgmap v136: 97 pgs: 97 active+clean; 34 GiB data, 104 GiB used, 969 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 561 MiB/s wr, 1.99k op/s 2024-04-25T11:39:31.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:31 smithi156 ceph-mon[33045]: pgmap v136: 97 pgs: 97 active+clean; 34 GiB data, 104 GiB used, 969 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 561 MiB/s wr, 1.99k op/s 2024-04-25T11:39:33.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:33 smithi090 ceph-mon[29575]: pgmap v137: 97 pgs: 97 active+clean; 34 GiB data, 104 GiB used, 969 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 417 MiB/s wr, 768 op/s 2024-04-25T11:39:33.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:33 smithi138 ceph-mon[32328]: pgmap v137: 97 pgs: 97 active+clean; 34 GiB data, 104 GiB used, 969 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 417 MiB/s wr, 768 op/s 2024-04-25T11:39:33.837 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:33 smithi156 ceph-mon[33045]: pgmap v137: 97 pgs: 97 active+clean; 34 GiB data, 104 GiB used, 969 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 417 MiB/s wr, 768 op/s 2024-04-25T11:39:35.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:35 smithi090 ceph-mon[29575]: pgmap v138: 97 pgs: 97 active+clean; 36 GiB data, 108 GiB used, 965 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 517 MiB/s wr, 944 op/s 2024-04-25T11:39:35.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:35 smithi138 ceph-mon[32328]: pgmap v138: 97 pgs: 97 active+clean; 36 GiB data, 108 GiB used, 965 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 517 MiB/s wr, 944 op/s 2024-04-25T11:39:35.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:35 smithi156 ceph-mon[33045]: pgmap v138: 97 pgs: 97 active+clean; 36 GiB data, 108 GiB used, 965 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 517 MiB/s wr, 944 op/s 2024-04-25T11:39:37.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:37 smithi138 ceph-mon[32328]: pgmap v139: 97 pgs: 97 active+clean; 38 GiB data, 115 GiB used, 958 GiB / 1.0 TiB avail; 2.7 MiB/s rd, 670 MiB/s wr, 1.28k op/s 2024-04-25T11:39:37.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:37 smithi156 ceph-mon[33045]: pgmap v139: 97 pgs: 97 active+clean; 38 GiB data, 115 GiB used, 958 GiB / 1.0 TiB avail; 2.7 MiB/s rd, 670 MiB/s wr, 1.28k op/s 2024-04-25T11:39:37.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:37 smithi090 ceph-mon[29575]: pgmap v139: 97 pgs: 97 active+clean; 38 GiB data, 115 GiB used, 958 GiB / 1.0 TiB avail; 2.7 MiB/s rd, 670 MiB/s wr, 1.28k op/s 2024-04-25T11:39:39.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:39 smithi138 ceph-mon[32328]: pgmap v140: 97 pgs: 97 active+clean; 38 GiB data, 115 GiB used, 958 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 516 MiB/s wr, 1.23k op/s 2024-04-25T11:39:39.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:39 smithi156 ceph-mon[33045]: pgmap v140: 97 pgs: 97 active+clean; 38 GiB data, 115 GiB used, 958 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 516 MiB/s wr, 1.23k op/s 2024-04-25T11:39:39.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:39 smithi090 ceph-mon[29575]: pgmap v140: 97 pgs: 97 active+clean; 38 GiB data, 115 GiB used, 958 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 516 MiB/s wr, 1.23k op/s 2024-04-25T11:39:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.112+0000 7fbe9b15e640 1 -- 172.21.15.90:0/3135167756 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe9406ce20 msgr2=0x7fbe9406d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.112+0000 7fbe9b15e640 1 --2- 172.21.15.90:0/3135167756 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe9406ce20 0x7fbe9406d230 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbe8001a910 tx=0x7fbe8003b500 comp rx=0 tx=0).stop 2024-04-25T11:39:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.118+0000 7fbe9b15e640 1 -- 172.21.15.90:0/3135167756 shutdown_connections 2024-04-25T11:39:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.118+0000 7fbe9b15e640 1 --2- 172.21.15.90:0/3135167756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe9411d140 0x7fbe9411f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.118+0000 7fbe9b15e640 1 --2- 172.21.15.90:0/3135167756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe9411a7e0 0x7fbe9411cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.118+0000 7fbe9b15e640 1 --2- 172.21.15.90:0/3135167756 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe9406ce20 0x7fbe9406d230 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.118+0000 7fbe9b15e640 1 -- 172.21.15.90:0/3135167756 >> 172.21.15.90:0/3135167756 conn(0x7fbe94071970 msgr2=0x7fbe94071d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.118+0000 7fbe9b15e640 1 -- 172.21.15.90:0/3135167756 shutdown_connections 2024-04-25T11:39:40.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.118+0000 7fbe9b15e640 1 -- 172.21.15.90:0/3135167756 wait complete. 2024-04-25T11:39:40.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.120+0000 7fbe9b15e640 1 Processor -- start 2024-04-25T11:39:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.121+0000 7fbe9b15e640 1 -- start start 2024-04-25T11:39:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.121+0000 7fbe9b15e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe9406ce20 0x7fbe94116560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.121+0000 7fbe9b15e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe9411a7e0 0x7fbe94116aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.121+0000 7fbe9b15e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe9411d140 0x7fbe94116fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.121+0000 7fbe9b15e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbe94112910 con 0x7fbe9411d140 2024-04-25T11:39:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.121+0000 7fbe9b15e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbe94112790 con 0x7fbe9411a7e0 2024-04-25T11:39:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.121+0000 7fbe9b15e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbe94112a90 con 0x7fbe9406ce20 2024-04-25T11:39:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.122+0000 7fbe98efa640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe9406ce20 0x7fbe94116560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.122+0000 7fbe98efa640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe9406ce20 0x7fbe94116560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:57206/0 (socket says 172.21.15.90:57206) 2024-04-25T11:39:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.122+0000 7fbe98efa640 1 -- 172.21.15.90:0/254738947 learned_addr learned my addr 172.21.15.90:0/254738947 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.122+0000 7fbe93fff640 1 --2- 172.21.15.90:0/254738947 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe9411a7e0 0x7fbe94116aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.123+0000 7fbe98efa640 1 -- 172.21.15.90:0/254738947 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe9411a7e0 msgr2=0x7fbe94116aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.123+0000 7fbe98efa640 1 --2- 172.21.15.90:0/254738947 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe9411a7e0 0x7fbe94116aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.123+0000 7fbe98efa640 1 -- 172.21.15.90:0/254738947 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe9411d140 msgr2=0x7fbe94116fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.123+0000 7fbe98efa640 1 --2- 172.21.15.90:0/254738947 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe9411d140 0x7fbe94116fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.123+0000 7fbe98efa640 1 -- 172.21.15.90:0/254738947 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe941ace90 con 0x7fbe9406ce20 2024-04-25T11:39:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.123+0000 7fbe98efa640 1 --2- 172.21.15.90:0/254738947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe9406ce20 0x7fbe94116560 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbe8003baf0 tx=0x7fbe80004a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.128+0000 7fbe91ffb640 1 -- 172.21.15.90:0/254738947 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8004f070 con 0x7fbe9406ce20 2024-04-25T11:39:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.128+0000 7fbe91ffb640 1 -- 172.21.15.90:0/254738947 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe80040bd0 con 0x7fbe9406ce20 2024-04-25T11:39:40.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.128+0000 7fbe91ffb640 1 -- 172.21.15.90:0/254738947 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe80046030 con 0x7fbe9406ce20 2024-04-25T11:39:40.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.128+0000 7fbe9b15e640 1 -- 172.21.15.90:0/254738947 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe941ad120 con 0x7fbe9406ce20 2024-04-25T11:39:40.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.129+0000 7fbe9b15e640 1 -- 172.21.15.90:0/254738947 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe941ad5c0 con 0x7fbe9406ce20 2024-04-25T11:39:40.136 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.142+0000 7fbe91ffb640 1 -- 172.21.15.90:0/254738947 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fbe80052070 con 0x7fbe9406ce20 2024-04-25T11:39:40.136 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.144+0000 7fbe91ffb640 1 --2- 172.21.15.90:0/254738947 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbe840afe60 0x7fbe840b2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:40.136 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.145+0000 7fbe91ffb640 1 -- 172.21.15.90:0/254738947 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fbe80110450 con 0x7fbe9406ce20 2024-04-25T11:39:40.137 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.145+0000 7fbea200f740 1 Processor -- start 2024-04-25T11:39:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.149+0000 7fbea200f740 1 -- start start 2024-04-25T11:39:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.149+0000 7fbea200f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557eebd111b0 0x557eec0b0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.149+0000 7fbea200f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eebce82f0 0x557eec0b11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.150+0000 7fbea200f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557eebcf04d0 0x557eec0b1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.150+0000 7fbea200f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557eec017460 con 0x557eebcf04d0 2024-04-25T11:39:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.150+0000 7fbea200f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x557eebfd1590 con 0x557eebd111b0 2024-04-25T11:39:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.150+0000 7fbea200f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x557eec013420 con 0x557eebce82f0 2024-04-25T11:39:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.156+0000 7fbe93fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eebce82f0 0x557eec0b11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.156+0000 7fbe93fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eebce82f0 0x557eec0b11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:57218/0 (socket says 172.21.15.90:57218) 2024-04-25T11:39:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.156+0000 7fbe93fff640 1 -- 172.21.15.90:0/2600943191 learned_addr learned my addr 172.21.15.90:0/2600943191 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.157+0000 7fbe98efa640 1 --2- 172.21.15.90:0/2600943191 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557eebd111b0 0x557eec0b0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.157+0000 7fbe93fff640 1 -- 172.21.15.90:0/2600943191 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557eebd111b0 msgr2=0x557eec0b0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.157+0000 7fbe93fff640 1 --2- 172.21.15.90:0/2600943191 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557eebd111b0 0x557eec0b0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.157+0000 7fbe93fff640 1 -- 172.21.15.90:0/2600943191 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557eebcf04d0 msgr2=0x557eec0b1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.157+0000 7fbe93fff640 1 --2- 172.21.15.90:0/2600943191 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557eebcf04d0 0x557eec0b1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.157+0000 7fbe93fff640 1 -- 172.21.15.90:0/2600943191 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557eebfde0d0 con 0x557eebce82f0 2024-04-25T11:39:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.158+0000 7fbe93fff640 1 --2- 172.21.15.90:0/2600943191 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eebce82f0 0x557eec0b11e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbe88009af0 tx=0x7fbe88010250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2987438e38e2bfc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.158+0000 7fbe7effd640 1 -- 172.21.15.90:0/2600943191 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe88018020 con 0x557eebce82f0 2024-04-25T11:39:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.159+0000 7fbe7effd640 1 -- 172.21.15.90:0/2600943191 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe88010c90 con 0x557eebce82f0 2024-04-25T11:39:40.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.160+0000 7fbea200f740 1 -- 172.21.15.90:0/2600943191 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eebce82f0 msgr2=0x557eec0b11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:40.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.160+0000 7fbea200f740 1 --2- 172.21.15.90:0/2600943191 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eebce82f0 0x557eec0b11e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbe88009af0 tx=0x7fbe88010250 comp rx=0 tx=0).stop 2024-04-25T11:39:40.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.165+0000 7fbe93fff640 1 --2- 172.21.15.90:0/254738947 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbe840afe60 0x7fbe840b2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:40.155 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.168+0000 7fbea200f740 1 -- 172.21.15.90:0/2600943191 shutdown_connections 2024-04-25T11:39:40.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.168+0000 7fbea200f740 1 --2- 172.21.15.90:0/2600943191 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557eebcf04d0 0x557eec0b1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:40.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.168+0000 7fbea200f740 1 --2- 172.21.15.90:0/2600943191 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eebce82f0 0x557eec0b11e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:40.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.168+0000 7fbea200f740 1 --2- 172.21.15.90:0/2600943191 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557eebd111b0 0x557eec0b0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:40.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.168+0000 7fbea200f740 1 -- 172.21.15.90:0/2600943191 >> 172.21.15.90:0/2600943191 conn(0x557eebd05300 msgr2=0x557eec0ad450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:40.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.168+0000 7fbea200f740 1 -- 172.21.15.90:0/2600943191 shutdown_connections 2024-04-25T11:39:40.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.168+0000 7fbea200f740 1 -- 172.21.15.90:0/2600943191 wait complete. 2024-04-25T11:39:40.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.169+0000 7fbea200f740 1 -- start start 2024-04-25T11:39:40.157 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.169+0000 7fbea200f740 1 Processor -- start 2024-04-25T11:39:40.157 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.169+0000 7fbea200f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557eebce82f0 0x557eec0bb200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:40.157 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.169+0000 7fbea200f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557eebcf04d0 0x557eec0bb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:40.157 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.169+0000 7fbea200f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eebd111b0 0x557eec0c73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:40.157 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.170+0000 7fbea200f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557eec0bde70 con 0x557eebcf04d0 2024-04-25T11:39:40.157 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.170+0000 7fbea200f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x557eebf9eb60 con 0x557eebce82f0 2024-04-25T11:39:40.157 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.170+0000 7fbea200f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x557eec0bdff0 con 0x557eebd111b0 2024-04-25T11:39:40.158 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.170+0000 7fbea200f740 10 client.0 _ll_register_callbacks cb 0x557eebfc32b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:40.158 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.170+0000 7fbe98efa640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557eebce82f0 0x557eec0bb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:40.158 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.170+0000 7fbe98efa640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557eebce82f0 0x557eec0bb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:52732/0 (socket says 172.21.15.90:52732) 2024-04-25T11:39:40.158 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.170+0000 7fbe98efa640 1 -- 172.21.15.90:0/3392416168 learned_addr learned my addr 172.21.15.90:0/3392416168 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:40.158 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.170+0000 7fbe996fb640 1 --2- 172.21.15.90:0/3392416168 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eebd111b0 0x557eec0c73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:40.159 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.171+0000 7fbe98efa640 1 -- 172.21.15.90:0/3392416168 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eebd111b0 msgr2=0x557eec0c73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:40.159 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.171+0000 7fbe98efa640 1 --2- 172.21.15.90:0/3392416168 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eebd111b0 0x557eec0c73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:40.159 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.171+0000 7fbe98efa640 1 -- 172.21.15.90:0/3392416168 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557eebcf04d0 msgr2=0x557eec0bb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:40.159 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.171+0000 7fbe98efa640 1 --2- 172.21.15.90:0/3392416168 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557eebcf04d0 0x557eec0bb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:40.159 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.171+0000 7fbe98efa640 1 -- 172.21.15.90:0/3392416168 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557eebd29210 con 0x557eebce82f0 2024-04-25T11:39:40.159 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.172+0000 7fbe98efa640 1 --2- 172.21.15.90:0/3392416168 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557eebce82f0 0x557eec0bb200 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbe800d45b0 tx=0x7fbe800d7fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea7b8c2e80761399 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:40.170 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.175+0000 7fbe93fff640 1 --2- 172.21.15.90:0/254738947 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbe840afe60 0x7fbe840b2370 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x557eec0acd70 tx=0x7fbe88014dd0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:40.170 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.183+0000 7fbe77fff640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:39:40.171 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.183+0000 7fbe77fff640 1 -- 172.21.15.90:0/3392416168 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe800f1020 con 0x557eebce82f0 2024-04-25T11:39:40.171 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.183+0000 7fbe77fff640 1 -- 172.21.15.90:0/3392416168 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fbe8010dd60 con 0x557eebce82f0 2024-04-25T11:39:40.171 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.183+0000 7fbea200f740 1 -- 172.21.15.90:0/3392416168 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe88010c90 con 0x557eebce82f0 2024-04-25T11:39:40.172 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.185+0000 7fbe77fff640 1 -- 172.21.15.90:0/3392416168 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbe80059040 con 0x557eebce82f0 2024-04-25T11:39:40.173 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.185+0000 7fbea200f740 10 client.26090 fetch_fsmap learned FSMap version 28 2024-04-25T11:39:40.173 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.185+0000 7fbea200f740 1 -- 172.21.15.90:0/3392416168 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557eebdf5620 con 0x557eebce82f0 2024-04-25T11:39:40.173 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.186+0000 7fbe77fff640 1 -- 172.21.15.90:0/3392416168 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe88010c90 con 0x557eebce82f0 2024-04-25T11:39:40.174 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.186+0000 7fbe77fff640 1 -- 172.21.15.90:0/3392416168 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fbe800f93c0 con 0x557eebce82f0 2024-04-25T11:39:40.176 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.186+0000 7fbea200f740 10 client.26090 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:39:40.176 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.186+0000 7fbea200f740 10 client.26090 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:39:40.176 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.186+0000 7fbea200f740 20 client.26090 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:40.176 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.186+0000 7fbea200f740 1 --2- 172.21.15.90:0/3392416168 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x557eebccfd80 0x557eec0cdb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:40.176 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.187+0000 7fbea200f740 4 client.26090 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:39:40.176 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.187+0000 7fbea200f740 1 -- 172.21.15.90:0/3392416168 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557eec025070 con 0x557eebccfd80 2024-04-25T11:39:40.177 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.189+0000 7fbe93fff640 1 --2- 172.21.15.90:0/3392416168 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x557eebccfd80 0x557eec0cdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:40.179 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.191+0000 7fbe93fff640 1 --2- 172.21.15.90:0/3392416168 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x557eebccfd80 0x557eec0cdb00 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5494238340ffebde server_cookie=92ec35b3d7c50448 in_seq=0 out_seq=0 2024-04-25T11:39:40.179 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.191+0000 7fbe77fff640 10 client.26090 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:40.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.202+0000 7fbe77fff640 1 -- 172.21.15.90:0/3392416168 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557eec025070 con 0x557eebccfd80 2024-04-25T11:39:40.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.202+0000 7fbe77fff640 10 client.26090 handle_command_reply: tid=0 2024-04-25T11:39:40.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.202+0000 7fbea200f740 1 client.26090 shutdown 2024-04-25T11:39:40.191 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.203+0000 7fbea200f740 1 -- 172.21.15.90:0/3392416168 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557eebce82f0 msgr2=0x557eec0bb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:40.191 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.203+0000 7fbea200f740 1 --2- 172.21.15.90:0/3392416168 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557eebce82f0 0x557eec0bb200 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbe800d45b0 tx=0x7fbe800d7fb0 comp rx=0 tx=0).stop 2024-04-25T11:39:40.191 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.203+0000 7fbea200f740 1 -- 172.21.15.90:0/3392416168 shutdown_connections 2024-04-25T11:39:40.191 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.203+0000 7fbea200f740 1 --2- 172.21.15.90:0/3392416168 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x557eebccfd80 0x557eec0cdb00 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:40.192 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.203+0000 7fbea200f740 1 --2- 172.21.15.90:0/3392416168 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eebd111b0 0x557eec0c73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:40.192 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.203+0000 7fbea200f740 1 --2- 172.21.15.90:0/3392416168 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557eebcf04d0 0x557eec0bb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:40.192 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.203+0000 7fbea200f740 1 --2- 172.21.15.90:0/3392416168 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557eebce82f0 0x557eec0bb200 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:40.192 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.203+0000 7fbea200f740 1 -- 172.21.15.90:0/3392416168 >> 172.21.15.90:0/3392416168 conn(0x557eebd05300 msgr2=0x557eec0abd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:40.192 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.203+0000 7fbe77fff640 0 client.26090 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:40.192 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.203+0000 7fbea200f740 1 -- 172.21.15.90:0/3392416168 shutdown_connections 2024-04-25T11:39:40.192 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.203+0000 7fbea200f740 1 -- 172.21.15.90:0/3392416168 wait complete. 2024-04-25T11:39:40.192 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.204+0000 7fbea200f740 20 client.26090 trim_cache size 0 max 16384 2024-04-25T11:39:40.263 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.276+0000 7fbea200f740 1 Processor -- start 2024-04-25T11:39:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.282+0000 7fbea200f740 1 -- start start 2024-04-25T11:39:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.283+0000 7fbea200f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557eebce82f0 0x557eec0ac750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:40.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.284+0000 7fbe98efa640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557eebce82f0 0x557eec0ac750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:40.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.284+0000 7fbe98efa640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557eebce82f0 0x557eec0ac750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50688/0 (socket says 172.21.15.90:50688) 2024-04-25T11:39:40.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.284+0000 7fbea200f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eebcf04d0 0x557eec0c6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:40.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.284+0000 7fbe98efa640 1 -- 172.21.15.90:0/4106424402 learned_addr learned my addr 172.21.15.90:0/4106424402 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:40.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.284+0000 7fbea200f740 1 --2- 172.21.15.90:0/4106424402 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557eebd111b0 0x557eec0d32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:40.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.284+0000 7fbea200f740 1 -- 172.21.15.90:0/4106424402 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7800ea60 con 0x557eebce82f0 2024-04-25T11:39:40.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.284+0000 7fbea200f740 1 -- 172.21.15.90:0/4106424402 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbe780105a0 con 0x557eebd111b0 2024-04-25T11:39:40.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.284+0000 7fbea200f740 1 -- 172.21.15.90:0/4106424402 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7800f130 con 0x557eebcf04d0 2024-04-25T11:39:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.285+0000 7fbe98efa640 1 -- 172.21.15.90:0/4106424402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eebcf04d0 msgr2=0x557eec0c6fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:39:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.285+0000 7fbe98efa640 1 --2- 172.21.15.90:0/4106424402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eebcf04d0 0x557eec0c6fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.285+0000 7fbe98efa640 1 -- 172.21.15.90:0/4106424402 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557eebd111b0 msgr2=0x557eec0d32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.285+0000 7fbe98efa640 1 --2- 172.21.15.90:0/4106424402 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557eebd111b0 0x557eec0d32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.285+0000 7fbe98efa640 1 -- 172.21.15.90:0/4106424402 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557eebccffb0 con 0x557eebce82f0 2024-04-25T11:39:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.288+0000 7fbe98efa640 1 --2- 172.21.15.90:0/4106424402 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557eebce82f0 0x557eec0ac750 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fbe8001aa30 tx=0x7fbe800d4520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=127965de7638a26b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:40.277 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.289+0000 7fbe7e7fc640 1 -- 172.21.15.90:0/4106424402 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe800f1040 con 0x557eebce82f0 2024-04-25T11:39:40.277 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.289+0000 7fbe7e7fc640 1 -- 172.21.15.90:0/4106424402 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe800dc8e0 con 0x557eebce82f0 2024-04-25T11:39:40.277 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.289+0000 7fbe7e7fc640 1 -- 172.21.15.90:0/4106424402 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe800dfc60 con 0x557eebce82f0 2024-04-25T11:39:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.291+0000 7fbea200f740 1 -- 172.21.15.90:0/4106424402 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557eebce82f0 msgr2=0x557eec0ac750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.291+0000 7fbea200f740 1 --2- 172.21.15.90:0/4106424402 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557eebce82f0 0x557eec0ac750 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fbe8001aa30 tx=0x7fbe800d4520 comp rx=0 tx=0).stop 2024-04-25T11:39:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.292+0000 7fbea200f740 1 -- 172.21.15.90:0/4106424402 shutdown_connections 2024-04-25T11:39:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.292+0000 7fbea200f740 1 --2- 172.21.15.90:0/4106424402 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557eebd111b0 0x557eec0d32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.292+0000 7fbea200f740 1 --2- 172.21.15.90:0/4106424402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eebcf04d0 0x557eec0c6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.292+0000 7fbea200f740 1 --2- 172.21.15.90:0/4106424402 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557eebce82f0 0x557eec0ac750 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.292+0000 7fbea200f740 1 -- 172.21.15.90:0/4106424402 >> 172.21.15.90:0/4106424402 conn(0x557eebd05300 msgr2=0x557eec0b9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.292+0000 7fbea200f740 1 -- 172.21.15.90:0/4106424402 shutdown_connections 2024-04-25T11:39:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.292+0000 7fbea200f740 1 -- 172.21.15.90:0/4106424402 wait complete. 2024-04-25T11:39:40.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.293+0000 7fbea200f740 1 -- start start 2024-04-25T11:39:40.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.293+0000 7fbea200f740 1 Processor -- start 2024-04-25T11:39:40.282 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.294+0000 7fbea200f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eebce82f0 0x557eec0be2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:40.282 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.295+0000 7fbea200f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557eebcf04d0 0x557eec0be890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:40.282 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.295+0000 7fbea200f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557eebd111b0 0x557eec133e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:40.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.295+0000 7fbea200f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7800fed0 con 0x557eebd111b0 2024-04-25T11:39:40.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.295+0000 7fbea200f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7800f800 con 0x557eebcf04d0 2024-04-25T11:39:40.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.295+0000 7fbea200f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7800ca40 con 0x557eebce82f0 2024-04-25T11:39:40.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.295+0000 7fbe93fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557eebcf04d0 0x557eec0be890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:40.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.295+0000 7fbe93fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557eebcf04d0 0x557eec0be890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:52746/0 (socket says 172.21.15.90:52746) 2024-04-25T11:39:40.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.295+0000 7fbe93fff640 1 -- 172.21.15.90:0/934197066 learned_addr learned my addr 172.21.15.90:0/934197066 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:40.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.295+0000 7fbe98efa640 1 --2- 172.21.15.90:0/934197066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eebce82f0 0x557eec0be2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:40.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.295+0000 7fbea200f740 10 client.0 _ll_register_callbacks cb 0x7fbe800d8300 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:40.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.296+0000 7fbe996fb640 1 --2- 172.21.15.90:0/934197066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557eebd111b0 0x557eec133e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:40.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.296+0000 7fbe93fff640 1 -- 172.21.15.90:0/934197066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eebce82f0 msgr2=0x557eec0be2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:40.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.296+0000 7fbe93fff640 1 --2- 172.21.15.90:0/934197066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eebce82f0 0x557eec0be2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:40.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.296+0000 7fbe93fff640 1 -- 172.21.15.90:0/934197066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557eebd111b0 msgr2=0x557eec133e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:40.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.296+0000 7fbe93fff640 1 --2- 172.21.15.90:0/934197066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557eebd111b0 0x557eec133e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:40.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.296+0000 7fbe93fff640 1 -- 172.21.15.90:0/934197066 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557eec1343a0 con 0x557eebcf04d0 2024-04-25T11:39:40.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.297+0000 7fbe93fff640 1 --2- 172.21.15.90:0/934197066 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557eebcf04d0 0x557eec0be890 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbe88014e40 tx=0x7fbe88004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ba15e91d00accab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:40.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.298+0000 7fbe77fff640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:39:40.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.300+0000 7fbe77fff640 1 -- 172.21.15.90:0/934197066 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe88038060 con 0x557eebcf04d0 2024-04-25T11:39:40.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.300+0000 7fbe77fff640 1 -- 172.21.15.90:0/934197066 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fbe880322f0 con 0x557eebcf04d0 2024-04-25T11:39:40.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.301+0000 7fbea200f740 1 -- 172.21.15.90:0/934197066 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe800dc8e0 con 0x557eebcf04d0 2024-04-25T11:39:40.291 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.303+0000 7fbe77fff640 1 -- 172.21.15.90:0/934197066 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbe88018070 con 0x557eebcf04d0 2024-04-25T11:39:40.292 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.303+0000 7fbea200f740 10 client.26093 fetch_fsmap learned FSMap version 28 2024-04-25T11:39:40.292 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.304+0000 7fbea200f740 1 -- 172.21.15.90:0/934197066 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557eec135460 con 0x557eebcf04d0 2024-04-25T11:39:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.306+0000 7fbe77fff640 1 -- 172.21.15.90:0/934197066 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe800dc8e0 con 0x557eebcf04d0 2024-04-25T11:39:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.306+0000 7fbe77fff640 1 -- 172.21.15.90:0/934197066 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fbe88032ce0 con 0x557eebcf04d0 2024-04-25T11:39:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.306+0000 7fbea200f740 10 client.26093 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:39:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.306+0000 7fbea200f740 10 client.26093 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:39:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.306+0000 7fbea200f740 20 client.26093 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.306+0000 7fbea200f740 1 --2- 172.21.15.90:0/934197066 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x557eebca51c0 0x557eec137810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.307+0000 7fbea200f740 4 client.26093 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:39:40.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.307+0000 7fbea200f740 1 -- 172.21.15.90:0/934197066 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557eec01eee0 con 0x557eebca51c0 2024-04-25T11:39:40.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.309+0000 7fbe98efa640 1 --2- 172.21.15.90:0/934197066 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x557eebca51c0 0x557eec137810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:40.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.317+0000 7fbe77fff640 10 client.26093 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:40.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:40.317+0000 7fbe98efa640 1 --2- 172.21.15.90:0/934197066 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x557eebca51c0 0x557eec137810 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dadd07be5ede298e server_cookie=23814ef06ec64917 in_seq=0 out_seq=0 2024-04-25T11:39:41.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:41 smithi156 ceph-mon[33045]: pgmap v141: 97 pgs: 97 active+clean; 40 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 3.0 MiB/s rd, 759 MiB/s wr, 2.00k op/s 2024-04-25T11:39:41.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:41 smithi090 ceph-mon[29575]: pgmap v141: 97 pgs: 97 active+clean; 40 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 3.0 MiB/s rd, 759 MiB/s wr, 2.00k op/s 2024-04-25T11:39:42.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:41 smithi138 ceph-mon[32328]: pgmap v141: 97 pgs: 97 active+clean; 40 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 3.0 MiB/s rd, 759 MiB/s wr, 2.00k op/s 2024-04-25T11:39:43.475 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:39:43.476 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:39:43.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.611+0000 7fce6e0a7640 1 -- 172.21.15.90:0/1175955739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce6810d380 msgr2=0x7fce6810f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:43.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.611+0000 7fce6e0a7640 1 --2- 172.21.15.90:0/1175955739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce6810d380 0x7fce6810f7a0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fce60009ca0 tx=0x7fce6002a8c0 comp rx=0 tx=0).stop 2024-04-25T11:39:43.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.611+0000 7fce6e0a7640 1 -- 172.21.15.90:0/1175955739 shutdown_connections 2024-04-25T11:39:43.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.611+0000 7fce6e0a7640 1 --2- 172.21.15.90:0/1175955739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce6810d380 0x7fce6810f7a0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:43.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.611+0000 7fce6e0a7640 1 --2- 172.21.15.90:0/1175955739 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fce68102af0 0x7fce68102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:43.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.611+0000 7fce6e0a7640 1 --2- 172.21.15.90:0/1175955739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce68108a20 0x7fce681025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:43.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.611+0000 7fce6e0a7640 1 -- 172.21.15.90:0/1175955739 >> 172.21.15.90:0/1175955739 conn(0x7fce68078100 msgr2=0x7fce68078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:43.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.611+0000 7fce6e0a7640 1 -- 172.21.15.90:0/1175955739 shutdown_connections 2024-04-25T11:39:43.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.611+0000 7fce6e0a7640 1 -- 172.21.15.90:0/1175955739 wait complete. 2024-04-25T11:39:43.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.612+0000 7fce6e0a7640 1 Processor -- start 2024-04-25T11:39:43.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.612+0000 7fce6e0a7640 1 -- start start 2024-04-25T11:39:43.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.612+0000 7fce6e0a7640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce68102af0 0x7fce6819b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:43.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.612+0000 7fce6e0a7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce68108a20 0x7fce6819baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:43.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.612+0000 7fce6e0a7640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fce6810d380 0x7fce6819bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:43.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.612+0000 7fce6e0a7640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fce68112c00 con 0x7fce68102af0 2024-04-25T11:39:43.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.612+0000 7fce6e0a7640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fce68112a80 con 0x7fce6810d380 2024-04-25T11:39:43.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.612+0000 7fce6e0a7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fce68112d80 con 0x7fce68108a20 2024-04-25T11:39:43.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.612+0000 7fce677fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce68102af0 0x7fce6819b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:43.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.612+0000 7fce677fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce68102af0 0x7fce6819b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49086/0 (socket says 172.21.15.90:49086) 2024-04-25T11:39:43.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.612+0000 7fce677fe640 1 -- 172.21.15.90:0/3265380889 learned_addr learned my addr 172.21.15.90:0/3265380889 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:43.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.612+0000 7fce66ffd640 1 --2- 172.21.15.90:0/3265380889 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce68108a20 0x7fce6819baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:43.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.612+0000 7fce67fff640 1 --2- 172.21.15.90:0/3265380889 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fce6810d380 0x7fce6819bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:43.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.613+0000 7fce677fe640 1 -- 172.21.15.90:0/3265380889 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce68108a20 msgr2=0x7fce6819baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:43.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.613+0000 7fce677fe640 1 --2- 172.21.15.90:0/3265380889 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce68108a20 0x7fce6819baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:43.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.613+0000 7fce677fe640 1 -- 172.21.15.90:0/3265380889 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fce6810d380 msgr2=0x7fce6819bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:43.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.613+0000 7fce677fe640 1 --2- 172.21.15.90:0/3265380889 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fce6810d380 0x7fce6819bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:43.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.613+0000 7fce677fe640 1 -- 172.21.15.90:0/3265380889 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce681a6870 con 0x7fce68102af0 2024-04-25T11:39:43.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.613+0000 7fce67fff640 1 --2- 172.21.15.90:0/3265380889 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fce6810d380 0x7fce6819bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:39:43.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.613+0000 7fce677fe640 1 --2- 172.21.15.90:0/3265380889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce68102af0 0x7fce6819b560 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fce5800e9f0 tx=0x7fce5800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:43.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.613+0000 7fce64ff9640 1 -- 172.21.15.90:0/3265380889 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5800ccc0 con 0x7fce68102af0 2024-04-25T11:39:43.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.613+0000 7fce6e0a7640 1 -- 172.21.15.90:0/3265380889 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce681a6b60 con 0x7fce68102af0 2024-04-25T11:39:43.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.613+0000 7fce64ff9640 1 -- 172.21.15.90:0/3265380889 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce580054d0 con 0x7fce68102af0 2024-04-25T11:39:43.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.613+0000 7fce64ff9640 1 -- 172.21.15.90:0/3265380889 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce58010430 con 0x7fce68102af0 2024-04-25T11:39:43.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.613+0000 7fce6e0a7640 1 -- 172.21.15.90:0/3265380889 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce681a70b0 con 0x7fce68102af0 2024-04-25T11:39:43.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.616+0000 7fce64ff9640 1 -- 172.21.15.90:0/3265380889 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fce580046c0 con 0x7fce68102af0 2024-04-25T11:39:43.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.616+0000 7fce75031740 1 Processor -- start 2024-04-25T11:39:43.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.616+0000 7fce75031740 1 -- start start 2024-04-25T11:39:43.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.616+0000 7fce75031740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56305e969390 0x56305ed09a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:43.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.616+0000 7fce75031740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56305e9404d0 0x56305ed09fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:43.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.616+0000 7fce75031740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56305e9486b0 0x56305ed168f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:43.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.616+0000 7fce75031740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56305ec6fd80 con 0x56305e9486b0 2024-04-25T11:39:43.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.616+0000 7fce75031740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56305ec298b0 con 0x56305e969390 2024-04-25T11:39:43.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.616+0000 7fce75031740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56305ec6bd40 con 0x56305e9404d0 2024-04-25T11:39:43.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.616+0000 7fce66ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56305e9486b0 0x56305ed168f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:43.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.616+0000 7fce67fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56305e969390 0x56305ed09a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:43.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.616+0000 7fce67fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56305e969390 0x56305ed09a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:56148/0 (socket says 172.21.15.90:56148) 2024-04-25T11:39:43.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.616+0000 7fce67fff640 1 -- 172.21.15.90:0/2991777792 learned_addr learned my addr 172.21.15.90:0/2991777792 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:43.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.616+0000 7fce677fe640 1 --2- 172.21.15.90:0/2991777792 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56305e9404d0 0x56305ed09fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:43.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.616+0000 7fce67fff640 1 -- 172.21.15.90:0/2991777792 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56305e9404d0 msgr2=0x56305ed09fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:43.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.616+0000 7fce67fff640 1 --2- 172.21.15.90:0/2991777792 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56305e9404d0 0x56305ed09fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:43.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.616+0000 7fce67fff640 1 -- 172.21.15.90:0/2991777792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56305e9486b0 msgr2=0x56305ed168f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:43.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.617+0000 7fce67fff640 1 --2- 172.21.15.90:0/2991777792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56305e9486b0 0x56305ed168f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:43.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.617+0000 7fce67fff640 1 -- 172.21.15.90:0/2991777792 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56305ebfaa40 con 0x56305e969390 2024-04-25T11:39:43.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.617+0000 7fce64ff9640 1 --2- 172.21.15.90:0/3265380889 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fce500afc60 0x7fce500b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:43.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.617+0000 7fce67fff640 1 --2- 172.21.15.90:0/2991777792 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56305e969390 0x56305ed09a60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fce600094d0 tx=0x7fce6002f310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b441697113e013c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:43.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.617+0000 7fce677fe640 1 --2- 172.21.15.90:0/3265380889 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fce500afc60 0x7fce500b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:43.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.617+0000 7fce4dffb640 1 -- 172.21.15.90:0/2991777792 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce60031cf0 con 0x56305e969390 2024-04-25T11:39:43.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.617+0000 7fce4dffb640 1 -- 172.21.15.90:0/2991777792 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce60036040 con 0x56305e969390 2024-04-25T11:39:43.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.617+0000 7fce4dffb640 1 -- 172.21.15.90:0/2991777792 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce600042a0 con 0x56305e969390 2024-04-25T11:39:43.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.617+0000 7fce64ff9640 1 -- 172.21.15.90:0/3265380889 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fce580df430 con 0x7fce68102af0 2024-04-25T11:39:43.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.618+0000 7fce75031740 1 -- 172.21.15.90:0/2991777792 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56305e969390 msgr2=0x56305ed09a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:43.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.618+0000 7fce75031740 1 --2- 172.21.15.90:0/2991777792 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56305e969390 0x56305ed09a60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fce600094d0 tx=0x7fce6002f310 comp rx=0 tx=0).stop 2024-04-25T11:39:43.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.618+0000 7fce677fe640 1 --2- 172.21.15.90:0/3265380889 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fce500afc60 0x7fce500b2170 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fce580dcbd0 tx=0x7fce580e1040 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:43.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.618+0000 7fce75031740 1 -- 172.21.15.90:0/2991777792 shutdown_connections 2024-04-25T11:39:43.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.618+0000 7fce75031740 1 --2- 172.21.15.90:0/2991777792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56305e9486b0 0x56305ed168f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:43.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.618+0000 7fce75031740 1 --2- 172.21.15.90:0/2991777792 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56305e9404d0 0x56305ed09fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:43.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.618+0000 7fce75031740 1 --2- 172.21.15.90:0/2991777792 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56305e969390 0x56305ed09a60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:43.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.618+0000 7fce75031740 1 -- 172.21.15.90:0/2991777792 >> 172.21.15.90:0/2991777792 conn(0x56305e95d4e0 msgr2=0x56305ed06210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:43.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.618+0000 7fce75031740 1 -- 172.21.15.90:0/2991777792 shutdown_connections 2024-04-25T11:39:43.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.618+0000 7fce75031740 1 -- 172.21.15.90:0/2991777792 wait complete. 2024-04-25T11:39:43.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.619+0000 7fce75031740 1 -- start start 2024-04-25T11:39:43.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.619+0000 7fce75031740 1 Processor -- start 2024-04-25T11:39:43.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.619+0000 7fce75031740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56305e9404d0 0x56305ed145e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:43.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.619+0000 7fce75031740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56305e9486b0 0x56305ed14b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:43.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.619+0000 7fce75031740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56305e969390 0x56305ed1a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:43.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.619+0000 7fce75031740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56305ed16f80 con 0x56305e9486b0 2024-04-25T11:39:43.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.619+0000 7fce75031740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56305ebf6d10 con 0x56305e9404d0 2024-04-25T11:39:43.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.619+0000 7fce75031740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56305ed17100 con 0x56305e969390 2024-04-25T11:39:43.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.619+0000 7fce75031740 10 client.0 _ll_register_callbacks cb 0x56305ec23210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:43.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.619+0000 7fce66ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56305e9486b0 0x56305ed14b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.619+0000 7fce67fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56305e9404d0 0x56305ed145e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.619+0000 7fce67fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56305e9404d0 0x56305ed145e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:56152/0 (socket says 172.21.15.90:56152) 2024-04-25T11:39:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.619+0000 7fce66ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56305e9486b0 0x56305ed14b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49102/0 (socket says 172.21.15.90:49102) 2024-04-25T11:39:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.619+0000 7fce67fff640 1 -- 172.21.15.90:0/2865773588 learned_addr learned my addr 172.21.15.90:0/2865773588 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.620+0000 7fce67fff640 1 --2- 172.21.15.90:0/2865773588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56305e969390 0x56305ed1a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.620+0000 7fce66ffd640 1 -- 172.21.15.90:0/2865773588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56305e969390 msgr2=0x56305ed1a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.620+0000 7fce66ffd640 1 --2- 172.21.15.90:0/2865773588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56305e969390 0x56305ed1a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.620+0000 7fce66ffd640 1 -- 172.21.15.90:0/2865773588 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56305e9404d0 msgr2=0x56305ed145e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.620+0000 7fce66ffd640 1 --2- 172.21.15.90:0/2865773588 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56305e9404d0 0x56305ed145e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.620+0000 7fce66ffd640 1 -- 172.21.15.90:0/2865773588 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56305ea4b7d0 con 0x56305e9486b0 2024-04-25T11:39:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.620+0000 7fce67fff640 1 --2- 172.21.15.90:0/2865773588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56305e969390 0x56305ed1a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:39:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.621+0000 7fce66ffd640 1 --2- 172.21.15.90:0/2865773588 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56305e9486b0 0x56305ed14b20 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fce68075fd0 tx=0x7fce5c012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7266cf1fade0c99 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.621+0000 7fce46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.621+0000 7fce46ffd640 1 -- 172.21.15.90:0/2865773588 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5c00f8f0 con 0x56305e9486b0 2024-04-25T11:39:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.621+0000 7fce46ffd640 1 -- 172.21.15.90:0/2865773588 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fce5c00fa90 con 0x56305e9486b0 2024-04-25T11:39:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.621+0000 7fce75031740 1 -- 172.21.15.90:0/2865773588 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce60036040 con 0x56305e9486b0 2024-04-25T11:39:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.621+0000 7fce46ffd640 1 -- 172.21.15.90:0/2865773588 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fce5c00fc90 con 0x56305e9486b0 2024-04-25T11:39:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.621+0000 7fce75031740 10 client.16920 fetch_fsmap learned FSMap version 28 2024-04-25T11:39:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.621+0000 7fce75031740 1 -- 172.21.15.90:0/2865773588 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56305e9813f0 con 0x56305e9486b0 2024-04-25T11:39:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.621+0000 7fce46ffd640 1 -- 172.21.15.90:0/2865773588 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce60036040 con 0x56305e9486b0 2024-04-25T11:39:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.621+0000 7fce46ffd640 1 -- 172.21.15.90:0/2865773588 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fce5c01c420 con 0x56305e9486b0 2024-04-25T11:39:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.621+0000 7fce75031740 10 client.16920 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:39:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.621+0000 7fce75031740 10 client.16920 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:39:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.622+0000 7fce75031740 20 client.16920 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.622+0000 7fce75031740 1 --2- 172.21.15.90:0/2865773588 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56305e927f60 0x56305ed266f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.622+0000 7fce75031740 4 client.16920 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:39:43.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.622+0000 7fce75031740 1 -- 172.21.15.90:0/2865773588 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56305ec7d990 con 0x56305e927f60 2024-04-25T11:39:43.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.622+0000 7fce67fff640 1 --2- 172.21.15.90:0/2865773588 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56305e927f60 0x56305ed266f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:43.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.622+0000 7fce46ffd640 10 client.16920 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:43.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:43.622+0000 7fce67fff640 1 --2- 172.21.15.90:0/2865773588 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56305e927f60 0x56305ed266f0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f31dfb9c9e6f9c40 server_cookie=b14d1a1934a71948 in_seq=0 out_seq=0 2024-04-25T11:39:43.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:43 smithi156 ceph-mon[33045]: pgmap v142: 97 pgs: 97 active+clean; 40 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 579 MiB/s wr, 1.33k op/s 2024-04-25T11:39:43.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:43 smithi090 ceph-mon[29575]: pgmap v142: 97 pgs: 97 active+clean; 40 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 579 MiB/s wr, 1.33k op/s 2024-04-25T11:39:44.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:43 smithi138 ceph-mon[32328]: pgmap v142: 97 pgs: 97 active+clean; 40 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 579 MiB/s wr, 1.33k op/s 2024-04-25T11:39:45.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:45 smithi156 ceph-mon[33045]: pgmap v143: 97 pgs: 97 active+clean; 40 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 638 MiB/s wr, 1.40k op/s 2024-04-25T11:39:45.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:45 smithi090 ceph-mon[29575]: pgmap v143: 97 pgs: 97 active+clean; 40 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 638 MiB/s wr, 1.40k op/s 2024-04-25T11:39:46.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:45 smithi138 ceph-mon[32328]: pgmap v143: 97 pgs: 97 active+clean; 40 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 638 MiB/s wr, 1.40k op/s 2024-04-25T11:39:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.572+0000 7fbe77fff640 1 -- 172.21.15.90:0/934197066 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557eec01eee0 con 0x557eebca51c0 2024-04-25T11:39:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.572+0000 7fbe77fff640 10 client.26093 handle_command_reply: tid=0 2024-04-25T11:39:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.572+0000 7fbea200f740 1 client.26093 shutdown 2024-04-25T11:39:46.563 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.573+0000 7fbea200f740 1 -- 172.21.15.90:0/934197066 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557eebcf04d0 msgr2=0x557eec0be890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:46.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.573+0000 7fbea200f740 1 --2- 172.21.15.90:0/934197066 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557eebcf04d0 0x557eec0be890 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbe88014e40 tx=0x7fbe88004750 comp rx=0 tx=0).stop 2024-04-25T11:39:46.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.573+0000 7fbea200f740 1 -- 172.21.15.90:0/934197066 shutdown_connections 2024-04-25T11:39:46.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.573+0000 7fbea200f740 1 --2- 172.21.15.90:0/934197066 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x557eebca51c0 0x557eec137810 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.573+0000 7fbea200f740 1 --2- 172.21.15.90:0/934197066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557eebd111b0 0x557eec133e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.573+0000 7fbea200f740 1 --2- 172.21.15.90:0/934197066 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557eebcf04d0 0x557eec0be890 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.573+0000 7fbea200f740 1 --2- 172.21.15.90:0/934197066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eebce82f0 0x557eec0be2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.573+0000 7fbea200f740 1 -- 172.21.15.90:0/934197066 >> 172.21.15.90:0/934197066 conn(0x557eebd05300 msgr2=0x557eec0b3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:46.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.573+0000 7fbe77fff640 0 client.26093 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:46.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.573+0000 7fbea200f740 1 -- 172.21.15.90:0/934197066 shutdown_connections 2024-04-25T11:39:46.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.573+0000 7fbea200f740 1 -- 172.21.15.90:0/934197066 wait complete. 2024-04-25T11:39:46.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.577+0000 7fbea200f740 20 client.26093 trim_cache size 0 max 16384 2024-04-25T11:39:46.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:39:46.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:39:46.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:39:46.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:39:46.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.578+0000 7fbe9b15e640 1 -- 172.21.15.90:0/254738947 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbe840afe60 msgr2=0x7fbe840b2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:46.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.578+0000 7fbe9b15e640 1 --2- 172.21.15.90:0/254738947 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbe840afe60 0x7fbe840b2370 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x557eec0acd70 tx=0x7fbe88014dd0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.579+0000 7fbe9b15e640 1 -- 172.21.15.90:0/254738947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe9406ce20 msgr2=0x7fbe94116560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:46.566 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.579+0000 7fbe9b15e640 1 --2- 172.21.15.90:0/254738947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe9406ce20 0x7fbe94116560 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbe8003baf0 tx=0x7fbe80004a30 comp rx=0 tx=0).stop 2024-04-25T11:39:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.580+0000 7fbe9b15e640 1 -- 172.21.15.90:0/254738947 shutdown_connections 2024-04-25T11:39:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.580+0000 7fbe9b15e640 1 --2- 172.21.15.90:0/254738947 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbe840afe60 0x7fbe840b2370 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.580+0000 7fbe9b15e640 1 --2- 172.21.15.90:0/254738947 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe9411d140 0x7fbe94116fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.580+0000 7fbe9b15e640 1 --2- 172.21.15.90:0/254738947 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe9411a7e0 0x7fbe94116aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.580+0000 7fbe9b15e640 1 --2- 172.21.15.90:0/254738947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe9406ce20 0x7fbe94116560 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.580+0000 7fbe9b15e640 1 -- 172.21.15.90:0/254738947 >> 172.21.15.90:0/254738947 conn(0x7fbe94071970 msgr2=0x7fbe9411b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.581+0000 7fbe9b15e640 1 -- 172.21.15.90:0/254738947 shutdown_connections 2024-04-25T11:39:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:46.581+0000 7fbe9b15e640 1 -- 172.21.15.90:0/254738947 wait complete. 2024-04-25T11:39:46.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.579+0000 7fce46ffd640 1 -- 172.21.15.90:0/2865773588 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56305ec7d990 con 0x56305e927f60 2024-04-25T11:39:46.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.579+0000 7fce46ffd640 10 client.16920 handle_command_reply: tid=0 2024-04-25T11:39:46.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.579+0000 7fce75031740 1 client.16920 shutdown 2024-04-25T11:39:46.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.583+0000 7fce75031740 1 -- 172.21.15.90:0/2865773588 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56305e9486b0 msgr2=0x56305ed14b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:46.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.583+0000 7fce75031740 1 --2- 172.21.15.90:0/2865773588 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56305e9486b0 0x56305ed14b20 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fce68075fd0 tx=0x7fce5c012b60 comp rx=0 tx=0).stop 2024-04-25T11:39:46.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.583+0000 7fce75031740 1 -- 172.21.15.90:0/2865773588 shutdown_connections 2024-04-25T11:39:46.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.583+0000 7fce75031740 1 --2- 172.21.15.90:0/2865773588 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56305e927f60 0x56305ed266f0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.583+0000 7fce75031740 1 --2- 172.21.15.90:0/2865773588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56305e969390 0x56305ed1a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.583+0000 7fce75031740 1 --2- 172.21.15.90:0/2865773588 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56305e9486b0 0x56305ed14b20 secure :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fce68075fd0 tx=0x7fce5c012b60 comp rx=0 tx=0).stop 2024-04-25T11:39:46.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.583+0000 7fce75031740 1 --2- 172.21.15.90:0/2865773588 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56305e9404d0 0x56305ed145e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.583+0000 7fce75031740 1 -- 172.21.15.90:0/2865773588 >> 172.21.15.90:0/2865773588 conn(0x56305e95d4e0 msgr2=0x56305ed0b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:46.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.583+0000 7fce46ffd640 0 client.16920 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:46.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.588+0000 7fce75031740 1 -- 172.21.15.90:0/2865773588 shutdown_connections 2024-04-25T11:39:46.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.589+0000 7fce75031740 1 -- 172.21.15.90:0/2865773588 wait complete. 2024-04-25T11:39:46.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.589+0000 7fce75031740 20 client.16920 trim_cache size 0 max 16384 2024-04-25T11:39:46.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.673+0000 7fce75031740 1 Processor -- start 2024-04-25T11:39:46.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.674+0000 7fce75031740 1 -- start start 2024-04-25T11:39:46.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.674+0000 7fce75031740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56305e9404d0 0x56305ed0f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:46.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.674+0000 7fce75031740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56305e969390 0x56305ed09e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:46.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.674+0000 7fce75031740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56305e8fd3a0 0x56305ed133b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:46.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.674+0000 7fce75031740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fce681afa60 con 0x56305e9404d0 2024-04-25T11:39:46.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.674+0000 7fce75031740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fce681b15a0 con 0x56305e8fd3a0 2024-04-25T11:39:46.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.674+0000 7fce75031740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fce681b0130 con 0x56305e969390 2024-04-25T11:39:46.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.674+0000 7fce67fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56305e9404d0 0x56305ed0f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:46.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.674+0000 7fce67fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56305e9404d0 0x56305ed0f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49108/0 (socket says 172.21.15.90:49108) 2024-04-25T11:39:46.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.674+0000 7fce67fff640 1 -- 172.21.15.90:0/497126459 learned_addr learned my addr 172.21.15.90:0/497126459 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:46.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.675+0000 7fce67fff640 1 -- 172.21.15.90:0/497126459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56305e969390 msgr2=0x56305ed09e60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:39:46.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.675+0000 7fce67fff640 1 --2- 172.21.15.90:0/497126459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56305e969390 0x56305ed09e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.675+0000 7fce67fff640 1 -- 172.21.15.90:0/497126459 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56305e8fd3a0 msgr2=0x56305ed133b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:39:46.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.675+0000 7fce67fff640 1 --2- 172.21.15.90:0/497126459 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56305e8fd3a0 0x56305ed133b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.675+0000 7fce67fff640 1 -- 172.21.15.90:0/497126459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56305e9410f0 con 0x56305e9404d0 2024-04-25T11:39:46.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.675+0000 7fce67fff640 1 --2- 172.21.15.90:0/497126459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56305e9404d0 0x56305ed0f640 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fce6002f380 tx=0x7fce60005c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4abc1e8bde03cbba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:46.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.675+0000 7fce4d7fa640 1 -- 172.21.15.90:0/497126459 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6005d070 con 0x56305e9404d0 2024-04-25T11:39:46.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.675+0000 7fce4d7fa640 1 -- 172.21.15.90:0/497126459 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce60036040 con 0x56305e9404d0 2024-04-25T11:39:46.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.676+0000 7fce4d7fa640 1 -- 172.21.15.90:0/497126459 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce60058020 con 0x56305e9404d0 2024-04-25T11:39:46.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.677+0000 7fce75031740 1 -- 172.21.15.90:0/497126459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56305e9404d0 msgr2=0x56305ed0f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:46.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.677+0000 7fce75031740 1 --2- 172.21.15.90:0/497126459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56305e9404d0 0x56305ed0f640 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fce6002f380 tx=0x7fce60005c30 comp rx=0 tx=0).stop 2024-04-25T11:39:46.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.677+0000 7fce75031740 1 -- 172.21.15.90:0/497126459 shutdown_connections 2024-04-25T11:39:46.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.677+0000 7fce75031740 1 --2- 172.21.15.90:0/497126459 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56305e8fd3a0 0x56305ed133b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.677+0000 7fce75031740 1 --2- 172.21.15.90:0/497126459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56305e969390 0x56305ed09e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.677+0000 7fce75031740 1 --2- 172.21.15.90:0/497126459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56305e9404d0 0x56305ed0f640 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.677+0000 7fce75031740 1 -- 172.21.15.90:0/497126459 >> 172.21.15.90:0/497126459 conn(0x56305e95d4e0 msgr2=0x56305ed12ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:46.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.677+0000 7fce75031740 1 -- 172.21.15.90:0/497126459 shutdown_connections 2024-04-25T11:39:46.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.677+0000 7fce75031740 1 -- 172.21.15.90:0/497126459 wait complete. 2024-04-25T11:39:46.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.678+0000 7fce75031740 1 -- start start 2024-04-25T11:39:46.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.678+0000 7fce75031740 1 Processor -- start 2024-04-25T11:39:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.679+0000 7fce75031740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56305e8fd3a0 0x56305ed207a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.679+0000 7fce75031740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56305e9404d0 0x56305ed20d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.679+0000 7fce75031740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56305e969390 0x56305ed1bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.679+0000 7fce75031740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fce681b0ed0 con 0x56305e8fd3a0 2024-04-25T11:39:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.679+0000 7fce75031740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fce681b0800 con 0x56305e9404d0 2024-04-25T11:39:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.679+0000 7fce75031740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fce681ada40 con 0x56305e969390 2024-04-25T11:39:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.679+0000 7fce75031740 10 client.0 _ll_register_callbacks cb 0x7fce5c009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.679+0000 7fce67fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56305e8fd3a0 0x56305ed207a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.679+0000 7fce67fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56305e8fd3a0 0x56305ed207a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49110/0 (socket says 172.21.15.90:49110) 2024-04-25T11:39:46.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.679+0000 7fce67fff640 1 -- 172.21.15.90:0/862218533 learned_addr learned my addr 172.21.15.90:0/862218533 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:46.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.680+0000 7fce67fff640 1 -- 172.21.15.90:0/862218533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56305e969390 msgr2=0x56305ed1bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:46.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.680+0000 7fce67fff640 1 --2- 172.21.15.90:0/862218533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56305e969390 0x56305ed1bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.680+0000 7fce67fff640 1 -- 172.21.15.90:0/862218533 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56305e9404d0 msgr2=0x56305ed20d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:46.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.680+0000 7fce67fff640 1 --2- 172.21.15.90:0/862218533 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56305e9404d0 0x56305ed20d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.680+0000 7fce67fff640 1 -- 172.21.15.90:0/862218533 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56305ec304b0 con 0x56305e8fd3a0 2024-04-25T11:39:46.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.681+0000 7fce67fff640 1 --2- 172.21.15.90:0/862218533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56305e8fd3a0 0x56305ed207a0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fce60009620 tx=0x7fce600319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4adf2289d356f51 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:46.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.681+0000 7fce46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:46.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.681+0000 7fce46ffd640 1 -- 172.21.15.90:0/862218533 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce60058030 con 0x56305e8fd3a0 2024-04-25T11:39:46.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.681+0000 7fce46ffd640 1 -- 172.21.15.90:0/862218533 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fce60031450 con 0x56305e8fd3a0 2024-04-25T11:39:46.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.681+0000 7fce75031740 1 -- 172.21.15.90:0/862218533 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce60036040 con 0x56305e8fd3a0 2024-04-25T11:39:46.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.681+0000 7fce46ffd640 1 -- 172.21.15.90:0/862218533 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fce6005d040 con 0x56305e8fd3a0 2024-04-25T11:39:46.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.681+0000 7fce75031740 10 client.16926 fetch_fsmap learned FSMap version 28 2024-04-25T11:39:46.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.681+0000 7fce75031740 1 -- 172.21.15.90:0/862218533 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56305ed96a20 con 0x56305e8fd3a0 2024-04-25T11:39:46.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.682+0000 7fce46ffd640 1 -- 172.21.15.90:0/862218533 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce60036040 con 0x56305e8fd3a0 2024-04-25T11:39:46.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.682+0000 7fce46ffd640 1 -- 172.21.15.90:0/862218533 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fce60050040 con 0x56305e8fd3a0 2024-04-25T11:39:46.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.682+0000 7fce75031740 10 client.16926 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:39:46.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.682+0000 7fce75031740 10 client.16926 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:39:46.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.682+0000 7fce75031740 20 client.16926 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:46.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.682+0000 7fce75031740 1 --2- 172.21.15.90:0/862218533 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56305ed96bf0 0x56305ed99010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:46.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.682+0000 7fce75031740 4 client.16926 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:39:46.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.682+0000 7fce75031740 1 -- 172.21.15.90:0/862218533 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56305ec77800 con 0x56305ed96bf0 2024-04-25T11:39:46.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.683+0000 7fce66ffd640 1 --2- 172.21.15.90:0/862218533 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56305ed96bf0 0x56305ed99010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:46.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.688+0000 7fce46ffd640 10 client.16926 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:46.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.688+0000 7fce66ffd640 1 --2- 172.21.15.90:0/862218533 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56305ed96bf0 0x56305ed99010 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2ee50e10c49385b server_cookie=22c0c44db02d0dd9 in_seq=0 out_seq=0 2024-04-25T11:39:46.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.692+0000 7fce46ffd640 1 -- 172.21.15.90:0/862218533 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56305ec77800 con 0x56305ed96bf0 2024-04-25T11:39:46.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.692+0000 7fce46ffd640 10 client.16926 handle_command_reply: tid=0 2024-04-25T11:39:46.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.692+0000 7fce75031740 1 client.16926 shutdown 2024-04-25T11:39:46.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.692+0000 7fce75031740 1 -- 172.21.15.90:0/862218533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56305e8fd3a0 msgr2=0x56305ed207a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:46.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.692+0000 7fce75031740 1 --2- 172.21.15.90:0/862218533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56305e8fd3a0 0x56305ed207a0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fce60009620 tx=0x7fce600319f0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.692+0000 7fce75031740 1 -- 172.21.15.90:0/862218533 shutdown_connections 2024-04-25T11:39:46.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.692+0000 7fce75031740 1 --2- 172.21.15.90:0/862218533 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56305ed96bf0 0x56305ed99010 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.692+0000 7fce75031740 1 --2- 172.21.15.90:0/862218533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56305e969390 0x56305ed1bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.692+0000 7fce75031740 1 --2- 172.21.15.90:0/862218533 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56305e9404d0 0x56305ed20d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.692+0000 7fce75031740 1 --2- 172.21.15.90:0/862218533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56305e8fd3a0 0x56305ed207a0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.692+0000 7fce75031740 1 -- 172.21.15.90:0/862218533 >> 172.21.15.90:0/862218533 conn(0x56305e95d4e0 msgr2=0x56305ed15a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:46.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.692+0000 7fce46ffd640 0 client.16926 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:46.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.692+0000 7fce75031740 1 -- 172.21.15.90:0/862218533 shutdown_connections 2024-04-25T11:39:46.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.692+0000 7fce75031740 1 -- 172.21.15.90:0/862218533 wait complete. 2024-04-25T11:39:46.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.693+0000 7fce75031740 20 client.16926 trim_cache size 0 max 16384 2024-04-25T11:39:46.682 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:39:46.682 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-25T11:39:46.682 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-25T11:39:46.682 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:39:46.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.693+0000 7fce6e0a7640 1 -- 172.21.15.90:0/3265380889 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fce500afc60 msgr2=0x7fce500b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:46.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.693+0000 7fce6e0a7640 1 --2- 172.21.15.90:0/3265380889 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fce500afc60 0x7fce500b2170 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fce580dcbd0 tx=0x7fce580e1040 comp rx=0 tx=0).stop 2024-04-25T11:39:46.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.693+0000 7fce6e0a7640 1 -- 172.21.15.90:0/3265380889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce68102af0 msgr2=0x7fce6819b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:46.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.693+0000 7fce6e0a7640 1 --2- 172.21.15.90:0/3265380889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce68102af0 0x7fce6819b560 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fce5800e9f0 tx=0x7fce5800eeb0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.693+0000 7fce6e0a7640 1 -- 172.21.15.90:0/3265380889 shutdown_connections 2024-04-25T11:39:46.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.693+0000 7fce6e0a7640 1 --2- 172.21.15.90:0/3265380889 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fce500afc60 0x7fce500b2170 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.693+0000 7fce6e0a7640 1 --2- 172.21.15.90:0/3265380889 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fce6810d380 0x7fce6819bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.693+0000 7fce6e0a7640 1 --2- 172.21.15.90:0/3265380889 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce68108a20 0x7fce6819baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.693+0000 7fce6e0a7640 1 --2- 172.21.15.90:0/3265380889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce68102af0 0x7fce6819b560 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.693+0000 7fce6e0a7640 1 -- 172.21.15.90:0/3265380889 >> 172.21.15.90:0/3265380889 conn(0x7fce68078100 msgr2=0x7fce681040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:46.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.694+0000 7fce6e0a7640 1 -- 172.21.15.90:0/3265380889 shutdown_connections 2024-04-25T11:39:46.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.694+0000 7fce6e0a7640 1 -- 172.21.15.90:0/3265380889 wait complete. 2024-04-25T11:39:46.702 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T11:39:46.702 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:39:46.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.873+0000 7f316ecb6640 1 -- 172.21.15.90:0/439200820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3168102020 msgr2=0x7f3168102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:46.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.873+0000 7f316ecb6640 1 --2- 172.21.15.90:0/439200820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3168102020 0x7f3168102430 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f3158009900 tx=0x7f315802a410 comp rx=0 tx=0).stop 2024-04-25T11:39:46.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.873+0000 7f316ecb6640 1 -- 172.21.15.90:0/439200820 shutdown_connections 2024-04-25T11:39:46.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.873+0000 7f316ecb6640 1 --2- 172.21.15.90:0/439200820 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f316810e520 0x7f3168110940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.873+0000 7f316ecb6640 1 --2- 172.21.15.90:0/439200820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3168102020 0x7f3168102430 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.873+0000 7f316ecb6640 1 --2- 172.21.15.90:0/439200820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3168109b70 0x7f3168101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.873+0000 7f316ecb6640 1 -- 172.21.15.90:0/439200820 >> 172.21.15.90:0/439200820 conn(0x7f31680fd810 msgr2=0x7f31680ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:46.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.873+0000 7f316ecb6640 1 -- 172.21.15.90:0/439200820 shutdown_connections 2024-04-25T11:39:46.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.873+0000 7f316ecb6640 1 -- 172.21.15.90:0/439200820 wait complete. 2024-04-25T11:39:46.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.874+0000 7f316ecb6640 1 Processor -- start 2024-04-25T11:39:46.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.874+0000 7f316ecb6640 1 -- start start 2024-04-25T11:39:46.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.874+0000 7f316ecb6640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3168102020 0x7f316819a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:46.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.874+0000 7f316ecb6640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3168109b70 0x7f316819ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:46.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.874+0000 7f316ecb6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f316810e520 0x7f316819ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:46.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.874+0000 7f316ecb6640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3168113d30 con 0x7f3168102020 2024-04-25T11:39:46.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.874+0000 7f316ecb6640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3168113bb0 con 0x7f3168109b70 2024-04-25T11:39:46.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.874+0000 7f316ecb6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3168113eb0 con 0x7f316810e520 2024-04-25T11:39:46.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.874+0000 7f316ca52640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3168102020 0x7f316819a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:46.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.874+0000 7f315ffff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3168109b70 0x7f316819ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:46.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.874+0000 7f315ffff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3168109b70 0x7f316819ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:56188/0 (socket says 172.21.15.90:56188) 2024-04-25T11:39:46.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.874+0000 7f316ca52640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3168102020 0x7f316819a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49140/0 (socket says 172.21.15.90:49140) 2024-04-25T11:39:46.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.875+0000 7f315ffff640 1 -- 172.21.15.90:0/1177614694 learned_addr learned my addr 172.21.15.90:0/1177614694 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:46.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.875+0000 7f316d253640 1 --2- 172.21.15.90:0/1177614694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f316810e520 0x7f316819ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:46.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.875+0000 7f315ffff640 1 -- 172.21.15.90:0/1177614694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f316810e520 msgr2=0x7f316819ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:46.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.875+0000 7f315ffff640 1 --2- 172.21.15.90:0/1177614694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f316810e520 0x7f316819ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.875+0000 7f315ffff640 1 -- 172.21.15.90:0/1177614694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3168102020 msgr2=0x7f316819a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:46.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.875+0000 7f315ffff640 1 --2- 172.21.15.90:0/1177614694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3168102020 0x7f316819a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:46.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.875+0000 7f315ffff640 1 -- 172.21.15.90:0/1177614694 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f316819f550 con 0x7f3168109b70 2024-04-25T11:39:46.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.875+0000 7f316ca52640 1 --2- 172.21.15.90:0/1177614694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3168102020 0x7f316819a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:46.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.875+0000 7f315ffff640 1 --2- 172.21.15.90:0/1177614694 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3168109b70 0x7f316819ac50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3158005ab0 tx=0x7f3158003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:46.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.875+0000 7f315dffb640 1 -- 172.21.15.90:0/1177614694 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315803f070 con 0x7f3168109b70 2024-04-25T11:39:46.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.876+0000 7f316ecb6640 1 -- 172.21.15.90:0/1177614694 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f316819f7e0 con 0x7f3168109b70 2024-04-25T11:39:46.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.876+0000 7f315dffb640 1 -- 172.21.15.90:0/1177614694 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3158031b50 con 0x7f3168109b70 2024-04-25T11:39:46.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.876+0000 7f315dffb640 1 -- 172.21.15.90:0/1177614694 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3158035030 con 0x7f3168109b70 2024-04-25T11:39:46.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.876+0000 7f316ecb6640 1 -- 172.21.15.90:0/1177614694 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f316819fd20 con 0x7f3168109b70 2024-04-25T11:39:46.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.877+0000 7f316ecb6640 1 -- 172.21.15.90:0/1177614694 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3168069310 con 0x7f3168109b70 2024-04-25T11:39:46.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.879+0000 7f315dffb640 1 -- 172.21.15.90:0/1177614694 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f3158042070 con 0x7f3168109b70 2024-04-25T11:39:46.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.880+0000 7f315dffb640 1 --2- 172.21.15.90:0/1177614694 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f31500afce0 0x7f31500b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:46.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.880+0000 7f316ca52640 1 --2- 172.21.15.90:0/1177614694 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f31500afce0 0x7f31500b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:46.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.880+0000 7f315dffb640 1 -- 172.21.15.90:0/1177614694 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f31580ffe80 con 0x7f3168109b70 2024-04-25T11:39:46.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.881+0000 7f316ca52640 1 --2- 172.21.15.90:0/1177614694 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f31500afce0 0x7f31500b21f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3164005fd0 tx=0x7f316400b390 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:46.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:46.883+0000 7f315dffb640 1 -- 172.21.15.90:0/1177614694 <== mon.2 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f31580c86c0 con 0x7f3168109b70 2024-04-25T11:39:47.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.122+0000 7f316ecb6640 1 -- 172.21.15.90:0/1177614694 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31681a72e0 con 0x7f3168109b70 2024-04-25T11:39:47.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.123+0000 7f315dffb640 1 -- 172.21.15.90:0/1177614694 <== mon.2 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11930 (secure 0 0 0) 0x7f31580c7fc0 con 0x7f3168109b70 2024-04-25T11:39:47.109 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:39:47.110 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:39:20.469254+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24458,24431,24499,24475,24524]},"id":1}]} 2024-04-25T11:39:47.110 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 28 2024-04-25T11:39:47.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.125+0000 7f316ecb6640 1 -- 172.21.15.90:0/1177614694 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f31500afce0 msgr2=0x7f31500b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.125+0000 7f316ecb6640 1 --2- 172.21.15.90:0/1177614694 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f31500afce0 0x7f31500b21f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3164005fd0 tx=0x7f316400b390 comp rx=0 tx=0).stop 2024-04-25T11:39:47.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.125+0000 7f316ecb6640 1 -- 172.21.15.90:0/1177614694 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3168109b70 msgr2=0x7f316819ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.125+0000 7f316ecb6640 1 --2- 172.21.15.90:0/1177614694 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3168109b70 0x7f316819ac50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3158005ab0 tx=0x7f3158003b90 comp rx=0 tx=0).stop 2024-04-25T11:39:47.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.125+0000 7f316ecb6640 1 -- 172.21.15.90:0/1177614694 shutdown_connections 2024-04-25T11:39:47.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.125+0000 7f316ecb6640 1 --2- 172.21.15.90:0/1177614694 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f31500afce0 0x7f31500b21f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.125+0000 7f316ecb6640 1 --2- 172.21.15.90:0/1177614694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f316810e520 0x7f316819ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.125+0000 7f316ecb6640 1 --2- 172.21.15.90:0/1177614694 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3168109b70 0x7f316819ac50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.125+0000 7f316ecb6640 1 --2- 172.21.15.90:0/1177614694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3168102020 0x7f316819a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.125+0000 7f316ecb6640 1 -- 172.21.15.90:0/1177614694 >> 172.21.15.90:0/1177614694 conn(0x7f31680fd810 msgr2=0x7f31680fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:47.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.125+0000 7f316ecb6640 1 -- 172.21.15.90:0/1177614694 shutdown_connections 2024-04-25T11:39:47.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.125+0000 7f316ecb6640 1 -- 172.21.15.90:0/1177614694 wait complete. 2024-04-25T11:39:47.126 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T11:39:47.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.302+0000 7f35fd72d640 1 -- 172.21.15.90:0/1322630781 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f811d120 msgr2=0x7f35f811f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.302+0000 7f35fd72d640 1 --2- 172.21.15.90:0/1322630781 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f811d120 0x7f35f811f540 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f35f0009cd0 tx=0x7f35f002a930 comp rx=0 tx=0).stop 2024-04-25T11:39:47.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.310+0000 7f35fd72d640 1 -- 172.21.15.90:0/1322630781 shutdown_connections 2024-04-25T11:39:47.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.310+0000 7f35fd72d640 1 --2- 172.21.15.90:0/1322630781 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f811d120 0x7f35f811f540 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.310+0000 7f35fd72d640 1 --2- 172.21.15.90:0/1322630781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f811a7c0 0x7f35f811cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.310+0000 7f35fd72d640 1 --2- 172.21.15.90:0/1322630781 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35f8109fc0 0x7f35f810a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.310+0000 7f35fd72d640 1 -- 172.21.15.90:0/1322630781 >> 172.21.15.90:0/1322630781 conn(0x7f35f806f650 msgr2=0x7f35f806fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:47.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.314+0000 7f35fd72d640 1 -- 172.21.15.90:0/1322630781 shutdown_connections 2024-04-25T11:39:47.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.314+0000 7f35fd72d640 1 -- 172.21.15.90:0/1322630781 wait complete. 2024-04-25T11:39:47.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.315+0000 7f35fd72d640 1 Processor -- start 2024-04-25T11:39:47.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.315+0000 7f35fd72d640 1 -- start start 2024-04-25T11:39:47.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.316+0000 7f35fd72d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f8109fc0 0x7f35f81a92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.317+0000 7f35fd72d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f811a7c0 0x7f35f81a9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.317+0000 7f35f7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f8109fc0 0x7f35f81a92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.317+0000 7f35f7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f8109fc0 0x7f35f81a92b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49160/0 (socket says 172.21.15.90:49160) 2024-04-25T11:39:47.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.317+0000 7f35f7fff640 1 -- 172.21.15.90:0/4056470994 learned_addr learned my addr 172.21.15.90:0/4056470994 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:47.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.317+0000 7f35fd72d640 1 --2- 172.21.15.90:0/4056470994 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35f811d120 0x7f35f81adad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.317+0000 7f35fd72d640 1 -- 172.21.15.90:0/4056470994 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f35f81229d0 con 0x7f35f8109fc0 2024-04-25T11:39:47.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.317+0000 7f35fd72d640 1 -- 172.21.15.90:0/4056470994 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f35f8122850 con 0x7f35f811d120 2024-04-25T11:39:47.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.317+0000 7f35fd72d640 1 -- 172.21.15.90:0/4056470994 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f35f8122b50 con 0x7f35f811a7c0 2024-04-25T11:39:47.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.318+0000 7f35f7fff640 1 -- 172.21.15.90:0/4056470994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f811a7c0 msgr2=0x7f35f81a9810 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:39:47.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.318+0000 7f35f7fff640 1 --2- 172.21.15.90:0/4056470994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f811a7c0 0x7f35f81a9810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.318+0000 7f35f7fff640 1 -- 172.21.15.90:0/4056470994 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35f811d120 msgr2=0x7f35f81adad0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:39:47.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.318+0000 7f35f7fff640 1 --2- 172.21.15.90:0/4056470994 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35f811d120 0x7f35f81adad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.318+0000 7f35f7fff640 1 -- 172.21.15.90:0/4056470994 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35f81ae110 con 0x7f35f8109fc0 2024-04-25T11:39:47.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.319+0000 7f35f7fff640 1 --2- 172.21.15.90:0/4056470994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f8109fc0 0x7f35f81a92b0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f35ec00b470 tx=0x7f35ec00b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:47.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.319+0000 7f35f57fa640 1 -- 172.21.15.90:0/4056470994 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ec015020 con 0x7f35f8109fc0 2024-04-25T11:39:47.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.319+0000 7f35fd72d640 1 -- 172.21.15.90:0/4056470994 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35f81ae400 con 0x7f35f8109fc0 2024-04-25T11:39:47.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.319+0000 7f35fd72d640 1 -- 172.21.15.90:0/4056470994 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35f81ae8f0 con 0x7f35f8109fc0 2024-04-25T11:39:47.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.321+0000 7f35f57fa640 1 -- 172.21.15.90:0/4056470994 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35ec011d40 con 0x7f35f8109fc0 2024-04-25T11:39:47.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.321+0000 7f35f57fa640 1 -- 172.21.15.90:0/4056470994 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ec01ad60 con 0x7f35f8109fc0 2024-04-25T11:39:47.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.323+0000 7f3605843740 1 Processor -- start 2024-04-25T11:39:47.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.324+0000 7f3605843740 1 -- start start 2024-04-25T11:39:47.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.324+0000 7f3605843740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563b3eb2e390 0x563b3eecfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.324+0000 7f3605843740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eb054d0 0x563b3eecff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.324+0000 7f3605843740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b3eb0d6b0 0x563b3eedc890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.324+0000 7f3605843740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563b3ee34d80 con 0x563b3eb054d0 2024-04-25T11:39:47.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.324+0000 7f3605843740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x563b3edeea20 con 0x563b3eb2e390 2024-04-25T11:39:47.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.324+0000 7f3605843740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563b3ee30d40 con 0x563b3eb0d6b0 2024-04-25T11:39:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.325+0000 7f35f77fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b3eb0d6b0 0x563b3eedc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.325+0000 7f35f77fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b3eb0d6b0 0x563b3eedc890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:46678/0 (socket says 172.21.15.90:46678) 2024-04-25T11:39:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.325+0000 7f35f77fe640 1 -- 172.21.15.90:0/1055705705 learned_addr learned my addr 172.21.15.90:0/1055705705 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.325+0000 7f35f57fa640 1 -- 172.21.15.90:0/4056470994 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f35ec018020 con 0x7f35f8109fc0 2024-04-25T11:39:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.325+0000 7f35f7fff640 1 --2- 172.21.15.90:0/1055705705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eb054d0 0x563b3eecff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.325+0000 7f35fcf2c640 1 --2- 172.21.15.90:0/1055705705 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563b3eb2e390 0x563b3eecfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.325+0000 7f35f7fff640 1 -- 172.21.15.90:0/1055705705 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b3eb0d6b0 msgr2=0x563b3eedc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.325+0000 7f35f7fff640 1 --2- 172.21.15.90:0/1055705705 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b3eb0d6b0 0x563b3eedc890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.325+0000 7f35f7fff640 1 -- 172.21.15.90:0/1055705705 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563b3eb2e390 msgr2=0x563b3eecfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.325+0000 7f35f7fff640 1 --2- 172.21.15.90:0/1055705705 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563b3eb2e390 0x563b3eecfa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.325+0000 7f35f7fff640 1 -- 172.21.15.90:0/1055705705 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b3edf9c30 con 0x563b3eb054d0 2024-04-25T11:39:47.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.325+0000 7f35f77fe640 1 --2- 172.21.15.90:0/1055705705 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b3eb0d6b0 0x563b3eedc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.326+0000 7f35f7fff640 1 --2- 172.21.15.90:0/1055705705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eb054d0 0x563b3eecff40 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f35ec0dc170 tx=0x7f35ec0e6750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a866a3c33386a52 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.326+0000 7f35f57fa640 1 --2- 172.21.15.90:0/4056470994 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f35e00afd10 0x7f35e00b2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.326+0000 7f35de7fc640 1 -- 172.21.15.90:0/1055705705 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ec0a3210 con 0x563b3eb054d0 2024-04-25T11:39:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.326+0000 7f35fcf2c640 1 --2- 172.21.15.90:0/4056470994 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f35e00afd10 0x7f35e00b2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.326+0000 7f35f57fa640 1 -- 172.21.15.90:0/4056470994 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f35ec0de970 con 0x7f35f8109fc0 2024-04-25T11:39:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.326+0000 7f35de7fc640 1 -- 172.21.15.90:0/1055705705 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35ec0eb070 con 0x563b3eb054d0 2024-04-25T11:39:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.326+0000 7f35de7fc640 1 -- 172.21.15.90:0/1055705705 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ec0a6330 con 0x563b3eb054d0 2024-04-25T11:39:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.328+0000 7f35fcf2c640 1 --2- 172.21.15.90:0/4056470994 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f35e00afd10 0x7f35e00b2220 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f35f002aea0 tx=0x7f35f002ae30 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.328+0000 7f3605843740 1 -- 172.21.15.90:0/1055705705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eb054d0 msgr2=0x563b3eecff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.328+0000 7f3605843740 1 --2- 172.21.15.90:0/1055705705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eb054d0 0x563b3eecff40 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f35ec0dc170 tx=0x7f35ec0e6750 comp rx=0 tx=0).stop 2024-04-25T11:39:47.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.328+0000 7f3605843740 1 -- 172.21.15.90:0/1055705705 shutdown_connections 2024-04-25T11:39:47.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.328+0000 7f3605843740 1 --2- 172.21.15.90:0/1055705705 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b3eb0d6b0 0x563b3eedc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.328+0000 7f3605843740 1 --2- 172.21.15.90:0/1055705705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eb054d0 0x563b3eecff40 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.328+0000 7f3605843740 1 --2- 172.21.15.90:0/1055705705 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563b3eb2e390 0x563b3eecfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.328+0000 7f3605843740 1 -- 172.21.15.90:0/1055705705 >> 172.21.15.90:0/1055705705 conn(0x563b3eb224e0 msgr2=0x563b3eecc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:47.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.328+0000 7f3605843740 1 -- 172.21.15.90:0/1055705705 shutdown_connections 2024-04-25T11:39:47.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.328+0000 7f3605843740 1 -- 172.21.15.90:0/1055705705 wait complete. 2024-04-25T11:39:47.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.329+0000 7f3605843740 1 -- start start 2024-04-25T11:39:47.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.330+0000 7f3605843740 1 Processor -- start 2024-04-25T11:39:47.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.330+0000 7f3605843740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eb054d0 0x563b3eeda380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.330+0000 7f3605843740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563b3eb0d6b0 0x563b3eeda8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.331+0000 7f3605843740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b3eb2e390 0x563b3eee64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.331+0000 7f3605843740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563b3eedcf80 con 0x563b3eb054d0 2024-04-25T11:39:47.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.331+0000 7f3605843740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x563b3edbbdc0 con 0x563b3eb0d6b0 2024-04-25T11:39:47.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.331+0000 7f35f7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eb054d0 0x563b3eeda380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.331+0000 7f3605843740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563b3eedd100 con 0x563b3eb2e390 2024-04-25T11:39:47.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.331+0000 7f35f7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eb054d0 0x563b3eeda380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49180/0 (socket says 172.21.15.90:49180) 2024-04-25T11:39:47.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.331+0000 7f35f7fff640 1 -- 172.21.15.90:0/1843505316 learned_addr learned my addr 172.21.15.90:0/1843505316 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:47.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.331+0000 7f3605843740 10 client.0 _ll_register_callbacks cb 0x563b3edd7600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:47.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.331+0000 7f35f77fe640 1 --2- 172.21.15.90:0/1843505316 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563b3eb0d6b0 0x563b3eeda8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.331+0000 7f35fcf2c640 1 --2- 172.21.15.90:0/1843505316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b3eb2e390 0x563b3eee64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.331+0000 7f35f7fff640 1 -- 172.21.15.90:0/1843505316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b3eb2e390 msgr2=0x563b3eee64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:47.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.331+0000 7f35f7fff640 1 --2- 172.21.15.90:0/1843505316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b3eb2e390 0x563b3eee64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.331+0000 7f35f7fff640 1 -- 172.21.15.90:0/1843505316 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563b3eb0d6b0 msgr2=0x563b3eeda8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:47.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.331+0000 7f35f7fff640 1 --2- 172.21.15.90:0/1843505316 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563b3eb0d6b0 0x563b3eeda8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.331+0000 7f35f7fff640 1 -- 172.21.15.90:0/1843505316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b3eb463f0 con 0x563b3eb054d0 2024-04-25T11:39:47.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.331+0000 7f35fcf2c640 1 --2- 172.21.15.90:0/1843505316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b3eb2e390 0x563b3eee64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:39:47.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.332+0000 7f35f7fff640 1 --2- 172.21.15.90:0/1843505316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eb054d0 0x563b3eeda380 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f35ec0ded20 tx=0x7f35ec0e43e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0b0df6dd579465a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:47.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.332+0000 7f35d77fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:47.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.332+0000 7f35d77fe640 1 -- 172.21.15.90:0/1843505316 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ec0a6510 con 0x563b3eb054d0 2024-04-25T11:39:47.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.332+0000 7f35d77fe640 1 -- 172.21.15.90:0/1843505316 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f35ec00fdc0 con 0x563b3eb054d0 2024-04-25T11:39:47.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.332+0000 7f3605843740 1 -- 172.21.15.90:0/1843505316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35ec0eb070 con 0x563b3eb054d0 2024-04-25T11:39:47.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.332+0000 7f35d77fe640 1 -- 172.21.15.90:0/1843505316 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f35ec0a6700 con 0x563b3eb054d0 2024-04-25T11:39:47.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.332+0000 7f3605843740 10 client.16944 fetch_fsmap learned FSMap version 28 2024-04-25T11:39:47.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.332+0000 7f3605843740 1 -- 172.21.15.90:0/1843505316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563b3ec107d0 con 0x563b3eb054d0 2024-04-25T11:39:47.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.333+0000 7f35d77fe640 1 -- 172.21.15.90:0/1843505316 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35ec0eb070 con 0x563b3eb054d0 2024-04-25T11:39:47.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.333+0000 7f35d77fe640 1 -- 172.21.15.90:0/1843505316 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f35ec0af040 con 0x563b3eb054d0 2024-04-25T11:39:47.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.333+0000 7f3605843740 10 client.16944 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:39:47.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.333+0000 7f3605843740 10 client.16944 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:39:47.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.333+0000 7f3605843740 20 client.16944 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:47.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.333+0000 7f3605843740 1 --2- 172.21.15.90:0/1843505316 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x563b3eaecf60 0x563b3eeec6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.333+0000 7f3605843740 4 client.16944 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:39:47.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.333+0000 7f3605843740 1 -- 172.21.15.90:0/1843505316 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b3ee42990 con 0x563b3eaecf60 2024-04-25T11:39:47.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.335+0000 7f35f77fe640 1 --2- 172.21.15.90:0/1843505316 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x563b3eaecf60 0x563b3eeec6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.339+0000 7f35d77fe640 10 client.16944 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:39:47.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.339+0000 7f35f77fe640 1 --2- 172.21.15.90:0/1843505316 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x563b3eaecf60 0x563b3eeec6b0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b13754480a21f7c8 server_cookie=adee76ddcdc4b5c6 in_seq=0 out_seq=0 2024-04-25T11:39:47.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.341+0000 7f35d77fe640 1 -- 172.21.15.90:0/1843505316 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563b3ee42990 con 0x563b3eaecf60 2024-04-25T11:39:47.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.341+0000 7f35d77fe640 10 client.16944 handle_command_reply: tid=0 2024-04-25T11:39:47.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.341+0000 7f3605843740 1 client.16944 shutdown 2024-04-25T11:39:47.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.341+0000 7f3605843740 1 -- 172.21.15.90:0/1843505316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eb054d0 msgr2=0x563b3eeda380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.341+0000 7f3605843740 1 --2- 172.21.15.90:0/1843505316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eb054d0 0x563b3eeda380 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f35ec0ded20 tx=0x7f35ec0e43e0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.341+0000 7f3605843740 1 -- 172.21.15.90:0/1843505316 shutdown_connections 2024-04-25T11:39:47.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.341+0000 7f3605843740 1 --2- 172.21.15.90:0/1843505316 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x563b3eaecf60 0x563b3eeec6b0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.341+0000 7f35d77fe640 0 client.16944 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:39:47.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.341+0000 7f3605843740 1 --2- 172.21.15.90:0/1843505316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b3eb2e390 0x563b3eee64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.341+0000 7f3605843740 1 --2- 172.21.15.90:0/1843505316 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563b3eb0d6b0 0x563b3eeda8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.341+0000 7f3605843740 1 --2- 172.21.15.90:0/1843505316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eb054d0 0x563b3eeda380 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.341+0000 7f3605843740 1 -- 172.21.15.90:0/1843505316 >> 172.21.15.90:0/1843505316 conn(0x563b3eb224e0 msgr2=0x563b3eed1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:47.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.342+0000 7f3605843740 1 -- 172.21.15.90:0/1843505316 shutdown_connections 2024-04-25T11:39:47.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.342+0000 7f3605843740 1 -- 172.21.15.90:0/1843505316 wait complete. 2024-04-25T11:39:47.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.342+0000 7f3605843740 20 client.16944 trim_cache size 0 max 16384 2024-04-25T11:39:47.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.370+0000 7f3605843740 1 Processor -- start 2024-04-25T11:39:47.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.371+0000 7f3605843740 1 -- start start 2024-04-25T11:39:47.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.371+0000 7f3605843740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eaecf60 0x563b3eed9470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.371+0000 7f3605843740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b3eb054d0 0x563b3eee6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.371+0000 7f3605843740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563b3eb0d6b0 0x563b3eed1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.372+0000 7f35f7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eaecf60 0x563b3eed9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.372+0000 7f35f7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eaecf60 0x563b3eed9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49182/0 (socket says 172.21.15.90:49182) 2024-04-25T11:39:47.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.372+0000 7f3605843740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f35e400e940 con 0x563b3eaecf60 2024-04-25T11:39:47.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.372+0000 7f35f7fff640 1 -- 172.21.15.90:0/1569390963 learned_addr learned my addr 172.21.15.90:0/1569390963 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:47.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.372+0000 7f3605843740 1 -- 172.21.15.90:0/1569390963 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f35e4010480 con 0x563b3eb0d6b0 2024-04-25T11:39:47.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.372+0000 7f3605843740 1 -- 172.21.15.90:0/1569390963 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f35e400f010 con 0x563b3eb054d0 2024-04-25T11:39:47.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.372+0000 7f35f77fe640 1 --2- 172.21.15.90:0/1569390963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b3eb054d0 0x563b3eee6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.372+0000 7f35f7fff640 1 -- 172.21.15.90:0/1569390963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b3eb054d0 msgr2=0x563b3eee6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:47.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.372+0000 7f35f7fff640 1 --2- 172.21.15.90:0/1569390963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b3eb054d0 0x563b3eee6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.372+0000 7f35f7fff640 1 -- 172.21.15.90:0/1569390963 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563b3eb0d6b0 msgr2=0x563b3eed1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:47.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.372+0000 7f35f7fff640 1 --2- 172.21.15.90:0/1569390963 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563b3eb0d6b0 0x563b3eed1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.372+0000 7f35f7fff640 1 -- 172.21.15.90:0/1569390963 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b3eedc8a0 con 0x563b3eaecf60 2024-04-25T11:39:47.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.372+0000 7f35f7fff640 1 --2- 172.21.15.90:0/1569390963 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eaecf60 0x563b3eed9470 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f35ec0a3e30 tx=0x7f35ec004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6da5daff913e3cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:47.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.373+0000 7f35ddffb640 1 -- 172.21.15.90:0/1569390963 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ec0a6990 con 0x563b3eaecf60 2024-04-25T11:39:47.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.373+0000 7f35ddffb640 1 -- 172.21.15.90:0/1569390963 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35ec0eb070 con 0x563b3eaecf60 2024-04-25T11:39:47.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.373+0000 7f35ddffb640 1 -- 172.21.15.90:0/1569390963 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ec0a2530 con 0x563b3eaecf60 2024-04-25T11:39:47.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.374+0000 7f3605843740 1 -- 172.21.15.90:0/1569390963 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eaecf60 msgr2=0x563b3eed9470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.374+0000 7f3605843740 1 --2- 172.21.15.90:0/1569390963 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eaecf60 0x563b3eed9470 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f35ec0a3e30 tx=0x7f35ec004700 comp rx=0 tx=0).stop 2024-04-25T11:39:47.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.375+0000 7f3605843740 1 -- 172.21.15.90:0/1569390963 shutdown_connections 2024-04-25T11:39:47.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.375+0000 7f3605843740 1 --2- 172.21.15.90:0/1569390963 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563b3eb0d6b0 0x563b3eed1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.375+0000 7f3605843740 1 --2- 172.21.15.90:0/1569390963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b3eb054d0 0x563b3eee6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.375+0000 7f3605843740 1 --2- 172.21.15.90:0/1569390963 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eaecf60 0x563b3eed9470 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.375+0000 7f3605843740 1 -- 172.21.15.90:0/1569390963 >> 172.21.15.90:0/1569390963 conn(0x563b3eb224e0 msgr2=0x563b3eed1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:47.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.375+0000 7f3605843740 1 -- 172.21.15.90:0/1569390963 shutdown_connections 2024-04-25T11:39:47.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.375+0000 7f3605843740 1 -- 172.21.15.90:0/1569390963 wait complete. 2024-04-25T11:39:47.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.375+0000 7f3605843740 1 -- start start 2024-04-25T11:39:47.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.376+0000 7f3605843740 1 Processor -- start 2024-04-25T11:39:47.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.376+0000 7f3605843740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b3eaecf60 0x563b3eeefa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.376+0000 7f3605843740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eb054d0 0x563b3eeecdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.376+0000 7f3605843740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563b3eb0d6b0 0x563b3eeed330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.377+0000 7f35f77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eb054d0 0x563b3eeecdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.377+0000 7f35f77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eb054d0 0x563b3eeecdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49194/0 (socket says 172.21.15.90:49194) 2024-04-25T11:39:47.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.377+0000 7f35f77fe640 1 -- 172.21.15.90:0/2831226469 learned_addr learned my addr 172.21.15.90:0/2831226469 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:47.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.377+0000 7f3605843740 1 -- 172.21.15.90:0/2831226469 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f35e400fdb0 con 0x563b3eb054d0 2024-04-25T11:39:47.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.377+0000 7f3605843740 1 -- 172.21.15.90:0/2831226469 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f35e400f6e0 con 0x563b3eb0d6b0 2024-04-25T11:39:47.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.377+0000 7f35f7fff640 1 --2- 172.21.15.90:0/2831226469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b3eaecf60 0x563b3eeefa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.377+0000 7f3605843740 1 -- 172.21.15.90:0/2831226469 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f35e400c920 con 0x563b3eaecf60 2024-04-25T11:39:47.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.377+0000 7f35fcf2c640 1 --2- 172.21.15.90:0/2831226469 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563b3eb0d6b0 0x563b3eeed330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.377+0000 7f3605843740 10 client.0 _ll_register_callbacks cb 0x7f35e8005f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:47.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.377+0000 7f35f77fe640 1 -- 172.21.15.90:0/2831226469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b3eaecf60 msgr2=0x563b3eeefa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:47.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.377+0000 7f35f77fe640 1 --2- 172.21.15.90:0/2831226469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b3eaecf60 0x563b3eeefa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.377+0000 7f35f77fe640 1 -- 172.21.15.90:0/2831226469 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563b3eb0d6b0 msgr2=0x563b3eeed330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:47.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.377+0000 7f35f77fe640 1 --2- 172.21.15.90:0/2831226469 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563b3eb0d6b0 0x563b3eeed330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.377+0000 7f35f77fe640 1 -- 172.21.15.90:0/2831226469 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b3eeed8a0 con 0x563b3eb054d0 2024-04-25T11:39:47.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.378+0000 7f35f77fe640 1 --2- 172.21.15.90:0/2831226469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eb054d0 0x563b3eeecdf0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f35e8000c00 tx=0x7f35e8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef2696d6e05b6698 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:47.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.378+0000 7f35d77fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:47.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.378+0000 7f35d77fe640 1 -- 172.21.15.90:0/2831226469 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e801e070 con 0x563b3eb054d0 2024-04-25T11:39:47.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.378+0000 7f3605843740 1 -- 172.21.15.90:0/2831226469 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35ec0eb070 con 0x563b3eb054d0 2024-04-25T11:39:47.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.378+0000 7f35d77fe640 1 -- 172.21.15.90:0/2831226469 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f35e8005220 con 0x563b3eb054d0 2024-04-25T11:39:47.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.379+0000 7f35d77fe640 1 -- 172.21.15.90:0/2831226469 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f35e8005420 con 0x563b3eb054d0 2024-04-25T11:39:47.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.379+0000 7f3605843740 10 client.16950 fetch_fsmap learned FSMap version 28 2024-04-25T11:39:47.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.379+0000 7f3605843740 1 -- 172.21.15.90:0/2831226469 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563b3ef54510 con 0x563b3eb054d0 2024-04-25T11:39:47.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.379+0000 7f35d77fe640 1 -- 172.21.15.90:0/2831226469 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35e80055c0 con 0x563b3eb054d0 2024-04-25T11:39:47.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.380+0000 7f35d77fe640 1 -- 172.21.15.90:0/2831226469 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f35e8004b20 con 0x563b3eb054d0 2024-04-25T11:39:47.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.380+0000 7f3605843740 10 client.16950 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:39:47.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.381+0000 7f3605843740 10 client.16950 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:39:47.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.381+0000 7f3605843740 20 client.16950 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:47.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.381+0000 7f3605843740 1 --2- 172.21.15.90:0/2831226469 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x563b3eb2e390 0x563b3ef56c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.381+0000 7f3605843740 4 client.16950 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:39:47.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.381+0000 7f3605843740 1 -- 172.21.15.90:0/2831226469 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f35ec0eb070 con 0x563b3eb2e390 2024-04-25T11:39:47.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.381+0000 7f35f7fff640 1 --2- 172.21.15.90:0/2831226469 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x563b3eb2e390 0x563b3ef56c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.382+0000 7f35f7fff640 1 --2- 172.21.15.90:0/2831226469 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x563b3eb2e390 0x563b3ef56c80 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2307a73c0acab293 server_cookie=a5fd0c8478902341 in_seq=0 out_seq=0 2024-04-25T11:39:47.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.382+0000 7f35d77fe640 10 client.16950 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:39:47.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.382+0000 7f35d77fe640 1 -- 172.21.15.90:0/2831226469 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f35ec0eb070 con 0x563b3eb2e390 2024-04-25T11:39:47.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.382+0000 7f35d77fe640 10 client.16950 handle_command_reply: tid=0 2024-04-25T11:39:47.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.382+0000 7f3605843740 1 client.16950 shutdown 2024-04-25T11:39:47.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.383+0000 7f3605843740 1 -- 172.21.15.90:0/2831226469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eb054d0 msgr2=0x563b3eeecdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.383+0000 7f3605843740 1 --2- 172.21.15.90:0/2831226469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eb054d0 0x563b3eeecdf0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f35e8000c00 tx=0x7f35e8011810 comp rx=0 tx=0).stop 2024-04-25T11:39:47.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.383+0000 7f3605843740 1 -- 172.21.15.90:0/2831226469 shutdown_connections 2024-04-25T11:39:47.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.383+0000 7f3605843740 1 --2- 172.21.15.90:0/2831226469 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x563b3eb2e390 0x563b3ef56c80 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.383+0000 7f3605843740 1 --2- 172.21.15.90:0/2831226469 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563b3eb0d6b0 0x563b3eeed330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.383+0000 7f35d77fe640 0 client.16950 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:39:47.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.383+0000 7f3605843740 1 --2- 172.21.15.90:0/2831226469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563b3eb054d0 0x563b3eeecdf0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.383+0000 7f3605843740 1 --2- 172.21.15.90:0/2831226469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b3eaecf60 0x563b3eeefa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.383+0000 7f3605843740 1 -- 172.21.15.90:0/2831226469 >> 172.21.15.90:0/2831226469 conn(0x563b3eb224e0 msgr2=0x563b3eed91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:47.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.383+0000 7f3605843740 1 -- 172.21.15.90:0/2831226469 shutdown_connections 2024-04-25T11:39:47.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.383+0000 7f3605843740 1 -- 172.21.15.90:0/2831226469 wait complete. 2024-04-25T11:39:47.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.383+0000 7f3605843740 20 client.16950 trim_cache size 0 max 16384 2024-04-25T11:39:47.374 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:39:47.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.384+0000 7f35fd72d640 1 -- 172.21.15.90:0/4056470994 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f35e00afd10 msgr2=0x7f35e00b2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.384+0000 7f35fd72d640 1 --2- 172.21.15.90:0/4056470994 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f35e00afd10 0x7f35e00b2220 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f35f002aea0 tx=0x7f35f002ae30 comp rx=0 tx=0).stop 2024-04-25T11:39:47.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.384+0000 7f35fd72d640 1 -- 172.21.15.90:0/4056470994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f8109fc0 msgr2=0x7f35f81a92b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.384+0000 7f35fd72d640 1 --2- 172.21.15.90:0/4056470994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f8109fc0 0x7f35f81a92b0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f35ec00b470 tx=0x7f35ec00b930 comp rx=0 tx=0).stop 2024-04-25T11:39:47.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.385+0000 7f35fd72d640 1 -- 172.21.15.90:0/4056470994 shutdown_connections 2024-04-25T11:39:47.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.385+0000 7f35fd72d640 1 --2- 172.21.15.90:0/4056470994 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f35e00afd10 0x7f35e00b2220 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.385+0000 7f35fd72d640 1 --2- 172.21.15.90:0/4056470994 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35f811d120 0x7f35f81adad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.385+0000 7f35fd72d640 1 --2- 172.21.15.90:0/4056470994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f811a7c0 0x7f35f81a9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.385+0000 7f35fd72d640 1 --2- 172.21.15.90:0/4056470994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f8109fc0 0x7f35f81a92b0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.385+0000 7f35fd72d640 1 -- 172.21.15.90:0/4056470994 >> 172.21.15.90:0/4056470994 conn(0x7f35f806f650 msgr2=0x7f35f811abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:47.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.385+0000 7f35fd72d640 1 -- 172.21.15.90:0/4056470994 shutdown_connections 2024-04-25T11:39:47.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.385+0000 7f35fd72d640 1 -- 172.21.15.90:0/4056470994 wait complete. 2024-04-25T11:39:47.394 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T11:39:47.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.582+0000 7fef97fff640 1 -- 172.21.15.90:0/364519339 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c030840 con 0x7fefa010f400 2024-04-25T11:39:47.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.583+0000 7fefa7372640 1 -- 172.21.15.90:0/364519339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fefa010f400 msgr2=0x7fefa0116ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.583+0000 7fefa7372640 1 --2- 172.21.15.90:0/364519339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fefa010f400 0x7fefa0116ba0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fef9c009860 tx=0x7fef9c02a380 comp rx=0 tx=0).stop 2024-04-25T11:39:47.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.583+0000 7fefa7372640 1 -- 172.21.15.90:0/364519339 shutdown_connections 2024-04-25T11:39:47.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.583+0000 7fefa7372640 1 --2- 172.21.15.90:0/364519339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fefa010f400 0x7fefa0116ba0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.583+0000 7fefa7372640 1 --2- 172.21.15.90:0/364519339 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefa010e940 0x7fefa010ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.583+0000 7fefa7372640 1 --2- 172.21.15.90:0/364519339 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fefa010a3b0 0x7fefa010a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.583+0000 7fefa7372640 1 -- 172.21.15.90:0/364519339 >> 172.21.15.90:0/364519339 conn(0x7fefa0076ff0 msgr2=0x7fefa0077450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.590+0000 7fefa7372640 1 -- 172.21.15.90:0/364519339 shutdown_connections 2024-04-25T11:39:47.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.592+0000 7fefa7372640 1 -- 172.21.15.90:0/364519339 wait complete. 2024-04-25T11:39:47.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.594+0000 7fefa7372640 1 Processor -- start 2024-04-25T11:39:47.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.596+0000 7fefa7372640 1 -- start start 2024-04-25T11:39:47.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.597+0000 7fefa7372640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fefa010a3b0 0x7fefa01a69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.597+0000 7fefa7372640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fefa010e940 0x7fefa01a6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.598+0000 7fefa7372640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefa010f400 0x7fefa01a0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.598+0000 7fefa7372640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fefa011a340 con 0x7fefa010f400 2024-04-25T11:39:47.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.598+0000 7fefa7372640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fefa011a1c0 con 0x7fefa010a3b0 2024-04-25T11:39:47.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.598+0000 7fefa7372640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fefa011a4c0 con 0x7fefa010e940 2024-04-25T11:39:47.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.600+0000 7fefa490d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fefa010e940 0x7fefa01a6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.601+0000 7fefa490d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fefa010e940 0x7fefa01a6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:46734/0 (socket says 172.21.15.90:46734) 2024-04-25T11:39:47.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.601+0000 7fefa490d640 1 -- 172.21.15.90:0/4068767743 learned_addr learned my addr 172.21.15.90:0/4068767743 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:47.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.601+0000 7fefa590f640 1 --2- 172.21.15.90:0/4068767743 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefa010f400 0x7fefa01a0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.602+0000 7fefa490d640 1 -- 172.21.15.90:0/4068767743 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fefa010a3b0 msgr2=0x7fefa01a69c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:39:47.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.602+0000 7fefa490d640 1 --2- 172.21.15.90:0/4068767743 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fefa010a3b0 0x7fefa01a69c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.602+0000 7fefa490d640 1 -- 172.21.15.90:0/4068767743 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefa010f400 msgr2=0x7fefa01a0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.602+0000 7fefa490d640 1 --2- 172.21.15.90:0/4068767743 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefa010f400 0x7fefa01a0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.602+0000 7fefa490d640 1 -- 172.21.15.90:0/4068767743 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefa01a1270 con 0x7fefa010e940 2024-04-25T11:39:47.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.603+0000 7fefa490d640 1 --2- 172.21.15.90:0/4068767743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fefa010e940 0x7fefa01a6f00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fef9000e9f0 tx=0x7fef9000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:47.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.603+0000 7fef967fc640 1 -- 172.21.15.90:0/4068767743 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9000ccc0 con 0x7fefa010e940 2024-04-25T11:39:47.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.604+0000 7fefa7372640 1 -- 172.21.15.90:0/4068767743 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefa01a1500 con 0x7fefa010e940 2024-04-25T11:39:47.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.604+0000 7fefa7372640 1 -- 172.21.15.90:0/4068767743 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefa01ad760 con 0x7fefa010e940 2024-04-25T11:39:47.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.604+0000 7fef967fc640 1 -- 172.21.15.90:0/4068767743 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef900108e0 con 0x7fefa010e940 2024-04-25T11:39:47.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.604+0000 7fef967fc640 1 -- 172.21.15.90:0/4068767743 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef90018880 con 0x7fefa010e940 2024-04-25T11:39:47.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.611+0000 7fef967fc640 1 -- 172.21.15.90:0/4068767743 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fef90018a20 con 0x7fefa010e940 2024-04-25T11:39:47.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.611+0000 7fefae21d740 1 Processor -- start 2024-04-25T11:39:47.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.612+0000 7fefae21d740 1 -- start start 2024-04-25T11:39:47.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.612+0000 7fefae21d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5576236e5390 0x557623a86810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.612+0000 7fefae21d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236bc4d0 0x557623a86d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.612+0000 7fefae21d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576236c46b0 0x557623a936a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.612+0000 7fefae21d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5576239ebd80 con 0x5576236bc4d0 2024-04-25T11:39:47.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.612+0000 7fefae21d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5576239a5a20 con 0x5576236e5390 2024-04-25T11:39:47.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.612+0000 7fefae21d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5576239e7d40 con 0x5576236c46b0 2024-04-25T11:39:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.613+0000 7fefa510e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236bc4d0 0x557623a86d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.613+0000 7fefa510e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236bc4d0 0x557623a86d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49222/0 (socket says 172.21.15.90:49222) 2024-04-25T11:39:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.613+0000 7fefa510e640 1 -- 172.21.15.90:0/2005640975 learned_addr learned my addr 172.21.15.90:0/2005640975 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.613+0000 7fefa590f640 1 --2- 172.21.15.90:0/2005640975 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5576236e5390 0x557623a86810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.613+0000 7fefa490d640 1 --2- 172.21.15.90:0/2005640975 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576236c46b0 0x557623a936a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.613+0000 7fef967fc640 1 --2- 172.21.15.90:0/4068767743 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fef8c0afce0 0x7fef8c0b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.614+0000 7fefa590f640 1 -- 172.21.15.90:0/2005640975 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576236c46b0 msgr2=0x557623a936a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.614+0000 7fefa590f640 1 --2- 172.21.15.90:0/2005640975 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576236c46b0 0x557623a936a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.614+0000 7fefa590f640 1 -- 172.21.15.90:0/2005640975 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236bc4d0 msgr2=0x557623a86d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.614+0000 7fefa590f640 1 --2- 172.21.15.90:0/2005640975 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236bc4d0 0x557623a86d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.614+0000 7fefa590f640 1 -- 172.21.15.90:0/2005640975 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576239b0c30 con 0x5576236e5390 2024-04-25T11:39:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.614+0000 7fefa590f640 1 --2- 172.21.15.90:0/4068767743 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fef8c0afce0 0x7fef8c0b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.614+0000 7fefa510e640 1 --2- 172.21.15.90:0/2005640975 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236bc4d0 0x557623a86d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:39:47.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.615+0000 7fef967fc640 1 -- 172.21.15.90:0/4068767743 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fef90010490 con 0x7fefa010e940 2024-04-25T11:39:47.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.615+0000 7fefa590f640 1 --2- 172.21.15.90:0/4068767743 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fef8c0afce0 0x7fef8c0b21f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fef9c009830 tx=0x7fef9c041040 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:47.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.620+0000 7fefa590f640 1 --2- 172.21.15.90:0/2005640975 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5576236e5390 0x557623a86810 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fef9c02a8a0 tx=0x7fef9c009550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5693506c68fb6e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:47.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.623+0000 7fef8b7fe640 1 -- 172.21.15.90:0/2005640975 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c037040 con 0x5576236e5390 2024-04-25T11:39:47.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.623+0000 7fef8b7fe640 1 -- 172.21.15.90:0/2005640975 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef9c04c070 con 0x5576236e5390 2024-04-25T11:39:47.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.623+0000 7fef8b7fe640 1 -- 172.21.15.90:0/2005640975 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c003850 con 0x5576236e5390 2024-04-25T11:39:47.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.624+0000 7fefae21d740 1 -- 172.21.15.90:0/2005640975 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5576236e5390 msgr2=0x557623a86810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.624+0000 7fefae21d740 1 --2- 172.21.15.90:0/2005640975 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5576236e5390 0x557623a86810 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fef9c02a8a0 tx=0x7fef9c009550 comp rx=0 tx=0).stop 2024-04-25T11:39:47.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.624+0000 7fefae21d740 1 -- 172.21.15.90:0/2005640975 shutdown_connections 2024-04-25T11:39:47.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.624+0000 7fefae21d740 1 --2- 172.21.15.90:0/2005640975 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576236c46b0 0x557623a936a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.624+0000 7fefae21d740 1 --2- 172.21.15.90:0/2005640975 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236bc4d0 0x557623a86d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.624+0000 7fefae21d740 1 --2- 172.21.15.90:0/2005640975 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5576236e5390 0x557623a86810 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.624+0000 7fefae21d740 1 -- 172.21.15.90:0/2005640975 >> 172.21.15.90:0/2005640975 conn(0x5576236d94e0 msgr2=0x557623a82fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:47.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.624+0000 7fefae21d740 1 -- 172.21.15.90:0/2005640975 shutdown_connections 2024-04-25T11:39:47.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.624+0000 7fefae21d740 1 -- 172.21.15.90:0/2005640975 wait complete. 2024-04-25T11:39:47.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.625+0000 7fefae21d740 1 -- start start 2024-04-25T11:39:47.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.625+0000 7fefae21d740 1 Processor -- start 2024-04-25T11:39:47.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.626+0000 7fefae21d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236bc4d0 0x557623a911c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.626+0000 7fefae21d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5576236c46b0 0x557623a91700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.626+0000 7fefae21d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576236e5390 0x557623aae2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.626+0000 7fefae21d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557623aa4d40 con 0x5576236bc4d0 2024-04-25T11:39:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.626+0000 7fefae21d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x557623972dc0 con 0x5576236c46b0 2024-04-25T11:39:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.626+0000 7fefae21d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x557623aa4ec0 con 0x5576236e5390 2024-04-25T11:39:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.626+0000 7fefa510e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236bc4d0 0x557623a911c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.626+0000 7fefae21d740 10 client.0 _ll_register_callbacks cb 0x55762398e600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.626+0000 7fefa510e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236bc4d0 0x557623a911c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49226/0 (socket says 172.21.15.90:49226) 2024-04-25T11:39:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.626+0000 7fefa510e640 1 -- 172.21.15.90:0/3984912782 learned_addr learned my addr 172.21.15.90:0/3984912782 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.627+0000 7fefa510e640 1 -- 172.21.15.90:0/3984912782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576236e5390 msgr2=0x557623aae2e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:39:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.627+0000 7fefa510e640 1 --2- 172.21.15.90:0/3984912782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576236e5390 0x557623aae2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.627+0000 7fefa510e640 1 -- 172.21.15.90:0/3984912782 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5576236c46b0 msgr2=0x557623a91700 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:39:47.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.627+0000 7fefa510e640 1 --2- 172.21.15.90:0/3984912782 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5576236c46b0 0x557623a91700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.627+0000 7fefa510e640 1 -- 172.21.15.90:0/3984912782 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576236fd3f0 con 0x5576236bc4d0 2024-04-25T11:39:47.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.627+0000 7fefa510e640 1 --2- 172.21.15.90:0/3984912782 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236bc4d0 0x557623a911c0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fef980116f0 tx=0x7fef98011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd94ce36728c4ee8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:47.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.627+0000 7fef88ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:47.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.628+0000 7fef88ff9640 1 -- 172.21.15.90:0/3984912782 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef98004d20 con 0x5576236bc4d0 2024-04-25T11:39:47.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.628+0000 7fef88ff9640 1 -- 172.21.15.90:0/3984912782 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fef9800fb50 con 0x5576236bc4d0 2024-04-25T11:39:47.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.628+0000 7fefae21d740 1 -- 172.21.15.90:0/3984912782 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef9c04c070 con 0x5576236bc4d0 2024-04-25T11:39:47.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.628+0000 7fef88ff9640 1 -- 172.21.15.90:0/3984912782 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fef9800fd50 con 0x5576236bc4d0 2024-04-25T11:39:47.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.628+0000 7fefae21d740 10 client.16962 fetch_fsmap learned FSMap version 28 2024-04-25T11:39:47.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.628+0000 7fefae21d740 1 -- 172.21.15.90:0/3984912782 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5576237c77d0 con 0x5576236bc4d0 2024-04-25T11:39:47.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.629+0000 7fef88ff9640 1 -- 172.21.15.90:0/3984912782 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef9c04c070 con 0x5576236bc4d0 2024-04-25T11:39:47.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.629+0000 7fef88ff9640 1 -- 172.21.15.90:0/3984912782 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fef9801a440 con 0x5576236bc4d0 2024-04-25T11:39:47.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.629+0000 7fefae21d740 10 client.16962 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:39:47.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.629+0000 7fefae21d740 10 client.16962 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:39:47.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.629+0000 7fefae21d740 20 client.16962 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:47.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.629+0000 7fefae21d740 1 --2- 172.21.15.90:0/3984912782 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5576236a3f60 0x557623ab4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.629+0000 7fefae21d740 4 client.16962 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:39:47.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.629+0000 7fefae21d740 1 -- 172.21.15.90:0/3984912782 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5576239f9990 con 0x5576236a3f60 2024-04-25T11:39:47.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.634+0000 7fefa490d640 1 --2- 172.21.15.90:0/3984912782 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5576236a3f60 0x557623ab4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.644+0000 7fefa490d640 1 --2- 172.21.15.90:0/3984912782 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5576236a3f60 0x557623ab4810 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=86d30876f657ad3d server_cookie=d4c478168a41a4bd in_seq=0 out_seq=0 2024-04-25T11:39:47.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.644+0000 7fef88ff9640 10 client.16962 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:39:47.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.649+0000 7fef88ff9640 1 -- 172.21.15.90:0/3984912782 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5576239f9990 con 0x5576236a3f60 2024-04-25T11:39:47.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.649+0000 7fef88ff9640 10 client.16962 handle_command_reply: tid=0 2024-04-25T11:39:47.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.649+0000 7fefae21d740 1 client.16962 shutdown 2024-04-25T11:39:47.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.649+0000 7fefae21d740 1 -- 172.21.15.90:0/3984912782 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236bc4d0 msgr2=0x557623a911c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.649+0000 7fefae21d740 1 --2- 172.21.15.90:0/3984912782 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236bc4d0 0x557623a911c0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fef980116f0 tx=0x7fef98011bb0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.649+0000 7fefae21d740 1 -- 172.21.15.90:0/3984912782 shutdown_connections 2024-04-25T11:39:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.649+0000 7fefae21d740 1 --2- 172.21.15.90:0/3984912782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576236e5390 0x557623aae2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.649+0000 7fefae21d740 1 --2- 172.21.15.90:0/3984912782 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5576236a3f60 0x557623ab4810 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.649+0000 7fefae21d740 1 --2- 172.21.15.90:0/3984912782 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5576236c46b0 0x557623a91700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.649+0000 7fefae21d740 1 --2- 172.21.15.90:0/3984912782 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236bc4d0 0x557623a911c0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.649+0000 7fefae21d740 1 -- 172.21.15.90:0/3984912782 >> 172.21.15.90:0/3984912782 conn(0x5576236d94e0 msgr2=0x557623a88530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.649+0000 7fef88ff9640 0 client.16962 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:39:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.649+0000 7fefae21d740 1 -- 172.21.15.90:0/3984912782 shutdown_connections 2024-04-25T11:39:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.649+0000 7fefae21d740 1 -- 172.21.15.90:0/3984912782 wait complete. 2024-04-25T11:39:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.649+0000 7fefae21d740 20 client.16962 trim_cache size 0 max 16384 2024-04-25T11:39:47.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.666+0000 7fefae21d740 1 Processor -- start 2024-04-25T11:39:47.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.666+0000 7fefae21d740 1 -- start start 2024-04-25T11:39:47.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.667+0000 7fefae21d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236bc4d0 0x557623a902b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.667+0000 7fefae21d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576236c46b0 0x557623aadec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.667+0000 7fefae21d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5576236e5390 0x557623a8b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.667+0000 7fefae21d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fef8400e940 con 0x5576236bc4d0 2024-04-25T11:39:47.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.667+0000 7fefae21d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fef84010480 con 0x5576236e5390 2024-04-25T11:39:47.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.667+0000 7fefae21d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fef8400f010 con 0x5576236c46b0 2024-04-25T11:39:47.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.667+0000 7fefa510e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236bc4d0 0x557623a902b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.667+0000 7fefa510e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236bc4d0 0x557623a902b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49230/0 (socket says 172.21.15.90:49230) 2024-04-25T11:39:47.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.667+0000 7fefa510e640 1 -- 172.21.15.90:0/2781850841 learned_addr learned my addr 172.21.15.90:0/2781850841 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:47.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.667+0000 7fefa510e640 1 -- 172.21.15.90:0/2781850841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576236c46b0 msgr2=0x557623aadec0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:39:47.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.667+0000 7fefa510e640 1 --2- 172.21.15.90:0/2781850841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576236c46b0 0x557623aadec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.667+0000 7fefa510e640 1 -- 172.21.15.90:0/2781850841 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5576236e5390 msgr2=0x557623a8b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:47.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.667+0000 7fefa510e640 1 --2- 172.21.15.90:0/2781850841 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5576236e5390 0x557623a8b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.667+0000 7fefa510e640 1 -- 172.21.15.90:0/2781850841 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557623a87080 con 0x5576236bc4d0 2024-04-25T11:39:47.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.668+0000 7fefa510e640 1 --2- 172.21.15.90:0/2781850841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236bc4d0 0x557623a902b0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fef9800a010 tx=0x7fef98011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c0a2d0fb01c785 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:47.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.668+0000 7fef8affd640 1 -- 172.21.15.90:0/2781850841 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef98032070 con 0x5576236bc4d0 2024-04-25T11:39:47.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.668+0000 7fef8affd640 1 -- 172.21.15.90:0/2781850841 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef98003e20 con 0x5576236bc4d0 2024-04-25T11:39:47.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.668+0000 7fef8affd640 1 -- 172.21.15.90:0/2781850841 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9801a670 con 0x5576236bc4d0 2024-04-25T11:39:47.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.669+0000 7fefae21d740 1 -- 172.21.15.90:0/2781850841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236bc4d0 msgr2=0x557623a902b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.669+0000 7fefae21d740 1 --2- 172.21.15.90:0/2781850841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236bc4d0 0x557623a902b0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fef9800a010 tx=0x7fef98011ec0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.669+0000 7fefae21d740 1 -- 172.21.15.90:0/2781850841 shutdown_connections 2024-04-25T11:39:47.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.669+0000 7fefae21d740 1 --2- 172.21.15.90:0/2781850841 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5576236e5390 0x557623a8b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.669+0000 7fefae21d740 1 --2- 172.21.15.90:0/2781850841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576236c46b0 0x557623aadec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.669+0000 7fefae21d740 1 --2- 172.21.15.90:0/2781850841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236bc4d0 0x557623a902b0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.669+0000 7fefae21d740 1 -- 172.21.15.90:0/2781850841 >> 172.21.15.90:0/2781850841 conn(0x5576236d94e0 msgr2=0x557623a82dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:47.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.669+0000 7fefae21d740 1 -- 172.21.15.90:0/2781850841 shutdown_connections 2024-04-25T11:39:47.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.669+0000 7fefae21d740 1 -- 172.21.15.90:0/2781850841 wait complete. 2024-04-25T11:39:47.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.669+0000 7fefae21d740 1 -- start start 2024-04-25T11:39:47.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.670+0000 7fefae21d740 1 Processor -- start 2024-04-25T11:39:47.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.670+0000 7fefae21d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576236bc4d0 0x557623aa5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.670+0000 7fefae21d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5576236c46b0 0x557623aa5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.670+0000 7fefae21d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236e5390 0x557623aa5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.670+0000 7fefae21d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fef8400fdb0 con 0x5576236e5390 2024-04-25T11:39:47.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.670+0000 7fefae21d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fef8400f6e0 con 0x5576236c46b0 2024-04-25T11:39:47.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.670+0000 7fefae21d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fef8400c920 con 0x5576236bc4d0 2024-04-25T11:39:47.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.670+0000 7fefae21d740 10 client.0 _ll_register_callbacks cb 0x7fef980099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:47.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.671+0000 7fefa590f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236e5390 0x557623aa5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.671+0000 7fefa590f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236e5390 0x557623aa5c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49238/0 (socket says 172.21.15.90:49238) 2024-04-25T11:39:47.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.671+0000 7fefa590f640 1 -- 172.21.15.90:0/3604012966 learned_addr learned my addr 172.21.15.90:0/3604012966 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:47.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.671+0000 7fefa490d640 1 --2- 172.21.15.90:0/3604012966 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5576236c46b0 0x557623aa5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.671+0000 7fefa590f640 1 -- 172.21.15.90:0/3604012966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576236bc4d0 msgr2=0x557623aa5200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:39:47.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.671+0000 7fefa590f640 1 --2- 172.21.15.90:0/3604012966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576236bc4d0 0x557623aa5200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.671+0000 7fefa590f640 1 -- 172.21.15.90:0/3604012966 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5576236c46b0 msgr2=0x557623aa5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:47.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.671+0000 7fefa590f640 1 --2- 172.21.15.90:0/3604012966 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5576236c46b0 0x557623aa5740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.671+0000 7fefa590f640 1 -- 172.21.15.90:0/3604012966 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557623b1abb0 con 0x5576236e5390 2024-04-25T11:39:47.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.671+0000 7fefa590f640 1 --2- 172.21.15.90:0/3604012966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236e5390 0x557623aa5c80 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fef9c035040 tx=0x7fef9c0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63653ec10a6294fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.671+0000 7fef88ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.672+0000 7fef88ff9640 1 -- 172.21.15.90:0/3604012966 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c055070 con 0x5576236e5390 2024-04-25T11:39:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.672+0000 7fef88ff9640 1 -- 172.21.15.90:0/3604012966 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fef9c0035f0 con 0x5576236e5390 2024-04-25T11:39:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.672+0000 7fefae21d740 1 -- 172.21.15.90:0/3604012966 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef98003e20 con 0x5576236e5390 2024-04-25T11:39:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.672+0000 7fef88ff9640 1 -- 172.21.15.90:0/3604012966 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fef9c0037f0 con 0x5576236e5390 2024-04-25T11:39:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.672+0000 7fefae21d740 10 client.16968 fetch_fsmap learned FSMap version 28 2024-04-25T11:39:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.672+0000 7fefae21d740 1 -- 172.21.15.90:0/3604012966 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557623b1bda0 con 0x5576236e5390 2024-04-25T11:39:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.672+0000 7fef88ff9640 1 -- 172.21.15.90:0/3604012966 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef98003e20 con 0x5576236e5390 2024-04-25T11:39:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.673+0000 7fef88ff9640 1 -- 172.21.15.90:0/3604012966 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fef9c0326c0 con 0x5576236e5390 2024-04-25T11:39:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.673+0000 7fefae21d740 10 client.16968 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:39:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.673+0000 7fefae21d740 10 client.16968 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:39:47.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.673+0000 7fefae21d740 20 client.16968 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:47.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.673+0000 7fefae21d740 1 --2- 172.21.15.90:0/3604012966 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5576236793a0 0x557623b1e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.673+0000 7fefae21d740 4 client.16968 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:39:47.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.673+0000 7fefae21d740 1 -- 172.21.15.90:0/3604012966 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5576239f3800 con 0x5576236793a0 2024-04-25T11:39:47.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.674+0000 7fefa510e640 1 --2- 172.21.15.90:0/3604012966 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5576236793a0 0x557623b1e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.675+0000 7fefa510e640 1 --2- 172.21.15.90:0/3604012966 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5576236793a0 0x557623b1e0e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5f089d4160f76183 server_cookie=27a97398ad231f90 in_seq=0 out_seq=0 2024-04-25T11:39:47.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.675+0000 7fef88ff9640 10 client.16968 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:39:47.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.675+0000 7fef88ff9640 1 -- 172.21.15.90:0/3604012966 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5576239f3800 con 0x5576236793a0 2024-04-25T11:39:47.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.675+0000 7fef88ff9640 10 client.16968 handle_command_reply: tid=0 2024-04-25T11:39:47.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.675+0000 7fefae21d740 1 client.16968 shutdown 2024-04-25T11:39:47.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.676+0000 7fefae21d740 1 -- 172.21.15.90:0/3604012966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236e5390 msgr2=0x557623aa5c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.676+0000 7fefae21d740 1 --2- 172.21.15.90:0/3604012966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236e5390 0x557623aa5c80 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fef9c035040 tx=0x7fef9c0048f0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.676+0000 7fefae21d740 1 -- 172.21.15.90:0/3604012966 shutdown_connections 2024-04-25T11:39:47.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.676+0000 7fefae21d740 1 --2- 172.21.15.90:0/3604012966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576236e5390 0x557623aa5c80 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.676+0000 7fefae21d740 1 --2- 172.21.15.90:0/3604012966 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5576236793a0 0x557623b1e0e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.676+0000 7fef88ff9640 0 client.16968 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:39:47.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.676+0000 7fefae21d740 1 --2- 172.21.15.90:0/3604012966 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5576236c46b0 0x557623aa5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.676+0000 7fefae21d740 1 --2- 172.21.15.90:0/3604012966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576236bc4d0 0x557623aa5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.676+0000 7fefae21d740 1 -- 172.21.15.90:0/3604012966 >> 172.21.15.90:0/3604012966 conn(0x5576236d94e0 msgr2=0x557623a82dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:47.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.676+0000 7fefae21d740 1 -- 172.21.15.90:0/3604012966 shutdown_connections 2024-04-25T11:39:47.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.676+0000 7fefae21d740 1 -- 172.21.15.90:0/3604012966 wait complete. 2024-04-25T11:39:47.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.676+0000 7fefae21d740 20 client.16968 trim_cache size 0 max 16384 2024-04-25T11:39:47.667 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:39:47.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.677+0000 7fefa7372640 1 -- 172.21.15.90:0/4068767743 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fef8c0afce0 msgr2=0x7fef8c0b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.677+0000 7fefa7372640 1 --2- 172.21.15.90:0/4068767743 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fef8c0afce0 0x7fef8c0b21f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fef9c009830 tx=0x7fef9c041040 comp rx=0 tx=0).stop 2024-04-25T11:39:47.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.677+0000 7fefa7372640 1 -- 172.21.15.90:0/4068767743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fefa010e940 msgr2=0x7fefa01a6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.677+0000 7fefa7372640 1 --2- 172.21.15.90:0/4068767743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fefa010e940 0x7fefa01a6f00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fef9000e9f0 tx=0x7fef9000eeb0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.677+0000 7fefa7372640 1 -- 172.21.15.90:0/4068767743 shutdown_connections 2024-04-25T11:39:47.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.677+0000 7fefa7372640 1 --2- 172.21.15.90:0/4068767743 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fef8c0afce0 0x7fef8c0b21f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.677+0000 7fefa7372640 1 --2- 172.21.15.90:0/4068767743 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefa010f400 0x7fefa01a0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.678+0000 7fefa7372640 1 --2- 172.21.15.90:0/4068767743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fefa010e940 0x7fefa01a6f00 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.678+0000 7fefa7372640 1 --2- 172.21.15.90:0/4068767743 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fefa010a3b0 0x7fefa01a69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.678+0000 7fefa7372640 1 -- 172.21.15.90:0/4068767743 >> 172.21.15.90:0/4068767743 conn(0x7fefa0076ff0 msgr2=0x7fefa007a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:47.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.678+0000 7fefa7372640 1 -- 172.21.15.90:0/4068767743 shutdown_connections 2024-04-25T11:39:47.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.678+0000 7fefa7372640 1 -- 172.21.15.90:0/4068767743 wait complete. 2024-04-25T11:39:47.685 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T11:39:47.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.832+0000 7f18a4986640 1 -- 172.21.15.90:0/2519329663 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a0108ec0 msgr2=0x7f18a0102b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.832+0000 7f18a4986640 1 --2- 172.21.15.90:0/2519329663 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a0108ec0 0x7f18a0102b10 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f1894009900 tx=0x7f189402a410 comp rx=0 tx=0).stop 2024-04-25T11:39:47.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.832+0000 7f18a4986640 1 -- 172.21.15.90:0/2519329663 shutdown_connections 2024-04-25T11:39:47.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.832+0000 7f18a4986640 1 --2- 172.21.15.90:0/2519329663 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18a0103180 0x7f18a010f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.832+0000 7f18a4986640 1 --2- 172.21.15.90:0/2519329663 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a0108ec0 0x7f18a0102b10 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.832+0000 7f18a4986640 1 --2- 172.21.15.90:0/2519329663 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18a0108a20 0x7f18a01025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.832+0000 7f18a4986640 1 -- 172.21.15.90:0/2519329663 >> 172.21.15.90:0/2519329663 conn(0x7f18a0078110 msgr2=0x7f18a0078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:47.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.832+0000 7f18a4986640 1 -- 172.21.15.90:0/2519329663 shutdown_connections 2024-04-25T11:39:47.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.832+0000 7f18a4986640 1 -- 172.21.15.90:0/2519329663 wait complete. 2024-04-25T11:39:47.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.833+0000 7f18a4986640 1 Processor -- start 2024-04-25T11:39:47.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.833+0000 7f18a4986640 1 -- start start 2024-04-25T11:39:47.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.833+0000 7f18a4986640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a0103180 0x7f18a019b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.833+0000 7f18a4986640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18a0108a20 0x7f18a019bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.833+0000 7f18a4986640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18a0108ec0 0x7f18a019c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.833+0000 7f18a4986640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f18a0112c40 con 0x7f18a0103180 2024-04-25T11:39:47.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.833+0000 7f18a4986640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f18a0112ac0 con 0x7f18a0108ec0 2024-04-25T11:39:47.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.833+0000 7f18a4986640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f18a0112dc0 con 0x7f18a0108a20 2024-04-25T11:39:47.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.833+0000 7f189e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a0103180 0x7f18a019b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.834+0000 7f189ed9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18a0108ec0 0x7f18a019c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.834+0000 7f189e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a0103180 0x7f18a019b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49258/0 (socket says 172.21.15.90:49258) 2024-04-25T11:39:47.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.834+0000 7f189dd9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18a0108a20 0x7f18a019bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.834+0000 7f189ed9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18a0108ec0 0x7f18a019c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:56302/0 (socket says 172.21.15.90:56302) 2024-04-25T11:39:47.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.834+0000 7f189e59c640 1 -- 172.21.15.90:0/2465622558 learned_addr learned my addr 172.21.15.90:0/2465622558 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:47.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.834+0000 7f189ed9d640 1 -- 172.21.15.90:0/2465622558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18a0108a20 msgr2=0x7f18a019bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.834+0000 7f189ed9d640 1 --2- 172.21.15.90:0/2465622558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18a0108a20 0x7f18a019bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.834+0000 7f189ed9d640 1 -- 172.21.15.90:0/2465622558 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a0103180 msgr2=0x7f18a019b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.834+0000 7f189ed9d640 1 --2- 172.21.15.90:0/2465622558 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a0103180 0x7f18a019b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.834+0000 7f189ed9d640 1 -- 172.21.15.90:0/2465622558 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a01a6920 con 0x7f18a0108ec0 2024-04-25T11:39:47.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.834+0000 7f189e59c640 1 --2- 172.21.15.90:0/2465622558 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a0103180 0x7f18a019b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:47.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.834+0000 7f189ed9d640 1 --2- 172.21.15.90:0/2465622558 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18a0108ec0 0x7f18a019c080 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f189800ec50 tx=0x7f189800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:47.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.834+0000 7f188f7fe640 1 -- 172.21.15.90:0/2465622558 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1898003b70 con 0x7f18a0108ec0 2024-04-25T11:39:47.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.834+0000 7f188f7fe640 1 -- 172.21.15.90:0/2465622558 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1898003d10 con 0x7f18a0108ec0 2024-04-25T11:39:47.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.834+0000 7f18a4986640 1 -- 172.21.15.90:0/2465622558 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18a01a6c10 con 0x7f18a0108ec0 2024-04-25T11:39:47.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.834+0000 7f188f7fe640 1 -- 172.21.15.90:0/2465622558 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1898010830 con 0x7f18a0108ec0 2024-04-25T11:39:47.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.834+0000 7f18a4986640 1 -- 172.21.15.90:0/2465622558 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18a01a70e0 con 0x7f18a0108ec0 2024-04-25T11:39:47.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.837+0000 7f18ab83f740 1 Processor -- start 2024-04-25T11:39:47.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.837+0000 7f188f7fe640 1 -- 172.21.15.90:0/2465622558 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f18980109d0 con 0x7f18a0108ec0 2024-04-25T11:39:47.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.837+0000 7f18ab83f740 1 -- start start 2024-04-25T11:39:47.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.838+0000 7f18ab83f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56018c65e390 0x56018c9ff810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.838+0000 7f18ab83f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56018c6354d0 0x56018c9ffd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.838+0000 7f18ab83f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c63d6b0 0x56018ca0c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.838+0000 7f18ab83f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56018c964d80 con 0x56018c63d6b0 2024-04-25T11:39:47.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.838+0000 7f18ab83f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56018c91ea20 con 0x56018c6354d0 2024-04-25T11:39:47.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.838+0000 7f18ab83f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56018c960d40 con 0x56018c65e390 2024-04-25T11:39:47.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.838+0000 7f189dd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c63d6b0 0x56018ca0c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.838+0000 7f189dd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c63d6b0 0x56018ca0c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49260/0 (socket says 172.21.15.90:49260) 2024-04-25T11:39:47.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.838+0000 7f189dd9b640 1 -- 172.21.15.90:0/4279937216 learned_addr learned my addr 172.21.15.90:0/4279937216 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:47.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.838+0000 7f189e59c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56018c6354d0 0x56018c9ffd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.838+0000 7f189dd9b640 1 --2- 172.21.15.90:0/4279937216 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56018c65e390 0x56018c9ff810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.838+0000 7f188f7fe640 1 --2- 172.21.15.90:0/2465622558 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f18880afc60 0x7f18880b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.838+0000 7f189e59c640 1 -- 172.21.15.90:0/4279937216 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56018c65e390 msgr2=0x56018c9ff810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:47.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.838+0000 7f189e59c640 1 --2- 172.21.15.90:0/4279937216 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56018c65e390 0x56018c9ff810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.838+0000 7f189e59c640 1 -- 172.21.15.90:0/4279937216 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c63d6b0 msgr2=0x56018ca0c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:47.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.838+0000 7f189e59c640 1 --2- 172.21.15.90:0/4279937216 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c63d6b0 0x56018ca0c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.838+0000 7f189e59c640 1 -- 172.21.15.90:0/4279937216 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56018c929c30 con 0x56018c6354d0 2024-04-25T11:39:47.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.838+0000 7f189ed9d640 1 --2- 172.21.15.90:0/2465622558 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f18880afc60 0x7f18880b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.838+0000 7f189dd9b640 1 --2- 172.21.15.90:0/4279937216 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c63d6b0 0x56018ca0c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:47.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.838+0000 7f188f7fe640 1 -- 172.21.15.90:0/2465622558 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f18980df5e0 con 0x7f18a0108ec0 2024-04-25T11:39:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.838+0000 7f189e59c640 1 --2- 172.21.15.90:0/4279937216 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56018c6354d0 0x56018c9ffd50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1890004650 tx=0x7f18900118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=273d733ac9bae9f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.839+0000 7f189ed9d640 1 --2- 172.21.15.90:0/2465622558 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f18880afc60 0x7f18880b2170 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f18980a4990 tx=0x7f189800ce70 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.839+0000 7f188cff9640 1 -- 172.21.15.90:0/4279937216 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189001e070 con 0x56018c6354d0 2024-04-25T11:39:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.839+0000 7f188cff9640 1 -- 172.21.15.90:0/4279937216 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f189000fa50 con 0x56018c6354d0 2024-04-25T11:39:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.839+0000 7f188cff9640 1 -- 172.21.15.90:0/4279937216 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1890019a90 con 0x56018c6354d0 2024-04-25T11:39:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.840+0000 7f18ab83f740 1 -- 172.21.15.90:0/4279937216 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56018c6354d0 msgr2=0x56018c9ffd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.840+0000 7f18ab83f740 1 --2- 172.21.15.90:0/4279937216 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56018c6354d0 0x56018c9ffd50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1890004650 tx=0x7f18900118d0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.840+0000 7f18ab83f740 1 -- 172.21.15.90:0/4279937216 shutdown_connections 2024-04-25T11:39:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.840+0000 7f18ab83f740 1 --2- 172.21.15.90:0/4279937216 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c63d6b0 0x56018ca0c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.840+0000 7f18ab83f740 1 --2- 172.21.15.90:0/4279937216 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56018c6354d0 0x56018c9ffd50 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.840+0000 7f18ab83f740 1 --2- 172.21.15.90:0/4279937216 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56018c65e390 0x56018c9ff810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.840+0000 7f18ab83f740 1 -- 172.21.15.90:0/4279937216 >> 172.21.15.90:0/4279937216 conn(0x56018c6524e0 msgr2=0x56018c9fbfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.840+0000 7f18ab83f740 1 -- 172.21.15.90:0/4279937216 shutdown_connections 2024-04-25T11:39:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.840+0000 7f18ab83f740 1 -- 172.21.15.90:0/4279937216 wait complete. 2024-04-25T11:39:47.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.840+0000 7f18ab83f740 1 -- start start 2024-04-25T11:39:47.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.840+0000 7f18ab83f740 1 Processor -- start 2024-04-25T11:39:47.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.840+0000 7f18ab83f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56018c6354d0 0x56018ca0a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.840+0000 7f18ab83f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c63d6b0 0x56018ca0a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.840+0000 7f18ab83f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56018c65e390 0x56018ca16210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.841+0000 7f18ab83f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56018ca0cd90 con 0x56018c63d6b0 2024-04-25T11:39:47.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.841+0000 7f18ab83f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56018c8ebdc0 con 0x56018c65e390 2024-04-25T11:39:47.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.841+0000 7f18ab83f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56018ca0cf10 con 0x56018c6354d0 2024-04-25T11:39:47.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.841+0000 7f18ab83f740 10 client.0 _ll_register_callbacks cb 0x56018c907600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:47.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.841+0000 7f189e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c63d6b0 0x56018ca0a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.841+0000 7f189e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c63d6b0 0x56018ca0a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49268/0 (socket says 172.21.15.90:49268) 2024-04-25T11:39:47.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.841+0000 7f189e59c640 1 -- 172.21.15.90:0/6291396 learned_addr learned my addr 172.21.15.90:0/6291396 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:47.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.841+0000 7f189dd9b640 1 --2- 172.21.15.90:0/6291396 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56018c65e390 0x56018ca16210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.841+0000 7f189dd9b640 1 --2- 172.21.15.90:0/6291396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56018c6354d0 0x56018ca0a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.841+0000 7f189e59c640 1 -- 172.21.15.90:0/6291396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56018c6354d0 msgr2=0x56018ca0a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.841+0000 7f189e59c640 1 --2- 172.21.15.90:0/6291396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56018c6354d0 0x56018ca0a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.841+0000 7f189e59c640 1 -- 172.21.15.90:0/6291396 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56018c65e390 msgr2=0x56018ca16210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.841+0000 7f189e59c640 1 --2- 172.21.15.90:0/6291396 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56018c65e390 0x56018ca16210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.841+0000 7f189e59c640 1 -- 172.21.15.90:0/6291396 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56018c6763f0 con 0x56018c63d6b0 2024-04-25T11:39:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.841+0000 7f189e59c640 1 --2- 172.21.15.90:0/6291396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c63d6b0 0x56018ca0a7a0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f189000ffc0 tx=0x7f1890009cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a1f3ca2c90faf09 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.841+0000 7f1885ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.841+0000 7f1885ffb640 1 -- 172.21.15.90:0/6291396 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189001e040 con 0x56018c63d6b0 2024-04-25T11:39:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.841+0000 7f1885ffb640 1 -- 172.21.15.90:0/6291396 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f189000f620 con 0x56018c63d6b0 2024-04-25T11:39:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.841+0000 7f18ab83f740 1 -- 172.21.15.90:0/6291396 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f189000fa50 con 0x56018c63d6b0 2024-04-25T11:39:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.842+0000 7f1885ffb640 1 -- 172.21.15.90:0/6291396 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1890032050 con 0x56018c63d6b0 2024-04-25T11:39:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.842+0000 7f18ab83f740 10 client.16980 fetch_fsmap learned FSMap version 28 2024-04-25T11:39:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.842+0000 7f18ab83f740 1 -- 172.21.15.90:0/6291396 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56018c7407d0 con 0x56018c63d6b0 2024-04-25T11:39:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.842+0000 7f1885ffb640 1 -- 172.21.15.90:0/6291396 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f189000fa50 con 0x56018c63d6b0 2024-04-25T11:39:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.842+0000 7f1885ffb640 1 -- 172.21.15.90:0/6291396 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f1890021030 con 0x56018c63d6b0 2024-04-25T11:39:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.842+0000 7f18ab83f740 10 client.16980 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:39:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.842+0000 7f18ab83f740 10 client.16980 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:39:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.842+0000 7f18ab83f740 20 client.16980 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.842+0000 7f18ab83f740 1 --2- 172.21.15.90:0/6291396 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x56018c61cf60 0x56018ca1c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.842+0000 7f18ab83f740 4 client.16980 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:39:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.842+0000 7f18ab83f740 1 -- 172.21.15.90:0/6291396 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56018c972990 con 0x56018c61cf60 2024-04-25T11:39:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.842+0000 7f189dd9b640 1 --2- 172.21.15.90:0/6291396 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x56018c61cf60 0x56018ca1c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.843+0000 7f1885ffb640 10 client.16980 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:39:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.843+0000 7f189dd9b640 1 --2- 172.21.15.90:0/6291396 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x56018c61cf60 0x56018ca1c5a0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d5058e6b6b7f9892 server_cookie=99728a01f7bd5385 in_seq=0 out_seq=0 2024-04-25T11:39:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.843+0000 7f1885ffb640 1 -- 172.21.15.90:0/6291396 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56018c972990 con 0x56018c61cf60 2024-04-25T11:39:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.843+0000 7f1885ffb640 10 client.16980 handle_command_reply: tid=0 2024-04-25T11:39:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.844+0000 7f18ab83f740 1 client.16980 shutdown 2024-04-25T11:39:47.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.844+0000 7f18ab83f740 1 -- 172.21.15.90:0/6291396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c63d6b0 msgr2=0x56018ca0a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.844+0000 7f18ab83f740 1 --2- 172.21.15.90:0/6291396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c63d6b0 0x56018ca0a7a0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f189000ffc0 tx=0x7f1890009cc0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.844+0000 7f18ab83f740 1 -- 172.21.15.90:0/6291396 shutdown_connections 2024-04-25T11:39:47.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.844+0000 7f18ab83f740 1 --2- 172.21.15.90:0/6291396 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x56018c61cf60 0x56018ca1c5a0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.844+0000 7f18ab83f740 1 --2- 172.21.15.90:0/6291396 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56018c65e390 0x56018ca16210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.844+0000 7f18ab83f740 1 --2- 172.21.15.90:0/6291396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c63d6b0 0x56018ca0a7a0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.844+0000 7f1885ffb640 0 client.16980 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:39:47.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.844+0000 7f18ab83f740 1 --2- 172.21.15.90:0/6291396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56018c6354d0 0x56018ca0a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.844+0000 7f18ab83f740 1 -- 172.21.15.90:0/6291396 >> 172.21.15.90:0/6291396 conn(0x56018c6524e0 msgr2=0x56018ca01530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:47.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.844+0000 7f18ab83f740 1 -- 172.21.15.90:0/6291396 shutdown_connections 2024-04-25T11:39:47.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.844+0000 7f18ab83f740 1 -- 172.21.15.90:0/6291396 wait complete. 2024-04-25T11:39:47.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.844+0000 7f18ab83f740 20 client.16980 trim_cache size 0 max 16384 2024-04-25T11:39:47.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.859+0000 7f18ab83f740 1 Processor -- start 2024-04-25T11:39:47.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.859+0000 7f18ab83f740 1 -- start start 2024-04-25T11:39:47.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.859+0000 7f18ab83f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56018c6354d0 0x56018ca0be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.859+0000 7f18ab83f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c63d6b0 0x56018ca09800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.859+0000 7f18ab83f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56018c65e390 0x56018ca04430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.859+0000 7f18ab83f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f18a01afbe0 con 0x56018c63d6b0 2024-04-25T11:39:47.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.859+0000 7f18ab83f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f18a01b1720 con 0x56018c65e390 2024-04-25T11:39:47.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.859+0000 7f18ab83f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f18a01b02b0 con 0x56018c6354d0 2024-04-25T11:39:47.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.859+0000 7f189e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c63d6b0 0x56018ca09800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.860+0000 7f189e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c63d6b0 0x56018ca09800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49270/0 (socket says 172.21.15.90:49270) 2024-04-25T11:39:47.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.860+0000 7f189e59c640 1 -- 172.21.15.90:0/3700462297 learned_addr learned my addr 172.21.15.90:0/3700462297 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:47.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.860+0000 7f189dd9b640 1 --2- 172.21.15.90:0/3700462297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56018c6354d0 0x56018ca0be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.860+0000 7f189dd9b640 1 --2- 172.21.15.90:0/3700462297 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56018c65e390 0x56018ca04430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.860+0000 7f189e59c640 1 -- 172.21.15.90:0/3700462297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56018c6354d0 msgr2=0x56018ca0be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:47.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.860+0000 7f189e59c640 1 --2- 172.21.15.90:0/3700462297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56018c6354d0 0x56018ca0be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.860+0000 7f189e59c640 1 -- 172.21.15.90:0/3700462297 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56018c65e390 msgr2=0x56018ca04430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:47.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.860+0000 7f189e59c640 1 --2- 172.21.15.90:0/3700462297 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56018c65e390 0x56018ca04430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.860+0000 7f189e59c640 1 -- 172.21.15.90:0/3700462297 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56018c9fd0f0 con 0x56018c63d6b0 2024-04-25T11:39:47.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.860+0000 7f189dd9b640 1 --2- 172.21.15.90:0/3700462297 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56018c65e390 0x56018ca04430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:39:47.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.860+0000 7f189e59c640 1 --2- 172.21.15.90:0/3700462297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c63d6b0 0x56018ca09800 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f1890009f40 tx=0x7f18900118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa9cfa2da56855d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:47.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.860+0000 7f1887fff640 1 -- 172.21.15.90:0/3700462297 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189001e070 con 0x56018c63d6b0 2024-04-25T11:39:47.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.860+0000 7f1887fff640 1 -- 172.21.15.90:0/3700462297 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1890029070 con 0x56018c63d6b0 2024-04-25T11:39:47.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.860+0000 7f1887fff640 1 -- 172.21.15.90:0/3700462297 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1890005540 con 0x56018c63d6b0 2024-04-25T11:39:47.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.861+0000 7f18ab83f740 1 -- 172.21.15.90:0/3700462297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c63d6b0 msgr2=0x56018ca09800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.861+0000 7f18ab83f740 1 --2- 172.21.15.90:0/3700462297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c63d6b0 0x56018ca09800 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f1890009f40 tx=0x7f18900118d0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.861+0000 7f18ab83f740 1 -- 172.21.15.90:0/3700462297 shutdown_connections 2024-04-25T11:39:47.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.861+0000 7f18ab83f740 1 --2- 172.21.15.90:0/3700462297 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56018c65e390 0x56018ca04430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.861+0000 7f18ab83f740 1 --2- 172.21.15.90:0/3700462297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c63d6b0 0x56018ca09800 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.861+0000 7f18ab83f740 1 --2- 172.21.15.90:0/3700462297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56018c6354d0 0x56018ca0be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.861+0000 7f18ab83f740 1 -- 172.21.15.90:0/3700462297 >> 172.21.15.90:0/3700462297 conn(0x56018c6524e0 msgr2=0x56018ca0b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:47.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.861+0000 7f18ab83f740 1 -- 172.21.15.90:0/3700462297 shutdown_connections 2024-04-25T11:39:47.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.861+0000 7f18ab83f740 1 -- 172.21.15.90:0/3700462297 wait complete. 2024-04-25T11:39:47.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.861+0000 7f18ab83f740 1 -- start start 2024-04-25T11:39:47.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.862+0000 7f18ab83f740 1 Processor -- start 2024-04-25T11:39:47.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.862+0000 7f18ab83f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56018c6354d0 0x56018ca0d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.862+0000 7f18ab83f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56018c63d6b0 0x56018ca0da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.862+0000 7f18ab83f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c65e390 0x56018ca827e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.862+0000 7f18ab83f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f18a01b1050 con 0x56018c65e390 2024-04-25T11:39:47.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.862+0000 7f18ab83f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f18a01b0980 con 0x56018c63d6b0 2024-04-25T11:39:47.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.862+0000 7f18ab83f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f18a01adbc0 con 0x56018c6354d0 2024-04-25T11:39:47.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.862+0000 7f18ab83f740 10 client.0 _ll_register_callbacks cb 0x7f189001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:47.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.862+0000 7f189dd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c65e390 0x56018ca827e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.862+0000 7f189dd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c65e390 0x56018ca827e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49284/0 (socket says 172.21.15.90:49284) 2024-04-25T11:39:47.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.862+0000 7f189dd9b640 1 -- 172.21.15.90:0/4044998474 learned_addr learned my addr 172.21.15.90:0/4044998474 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:47.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.862+0000 7f189dd9b640 1 --2- 172.21.15.90:0/4044998474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56018c6354d0 0x56018ca0d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.862+0000 7f189e59c640 1 --2- 172.21.15.90:0/4044998474 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56018c63d6b0 0x56018ca0da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.862+0000 7f189dd9b640 1 -- 172.21.15.90:0/4044998474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56018c6354d0 msgr2=0x56018ca0d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:47.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.862+0000 7f189dd9b640 1 --2- 172.21.15.90:0/4044998474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56018c6354d0 0x56018ca0d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.862+0000 7f189dd9b640 1 -- 172.21.15.90:0/4044998474 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56018c63d6b0 msgr2=0x56018ca0da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:47.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.862+0000 7f189dd9b640 1 --2- 172.21.15.90:0/4044998474 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56018c63d6b0 0x56018ca0da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.862+0000 7f189dd9b640 1 -- 172.21.15.90:0/4044998474 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56018ca82d20 con 0x56018c65e390 2024-04-25T11:39:47.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.862+0000 7f189dd9b640 1 --2- 172.21.15.90:0/4044998474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c65e390 0x56018ca827e0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f1894000c00 tx=0x7f189000ff10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86ead52daafa3d96 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:47.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.862+0000 7f1885ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:47.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.863+0000 7f1885ffb640 1 -- 172.21.15.90:0/4044998474 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1894035040 con 0x56018c65e390 2024-04-25T11:39:47.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.863+0000 7f1885ffb640 1 -- 172.21.15.90:0/4044998474 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f1894004390 con 0x56018c65e390 2024-04-25T11:39:47.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.863+0000 7f18ab83f740 1 -- 172.21.15.90:0/4044998474 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1890029070 con 0x56018c65e390 2024-04-25T11:39:47.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.863+0000 7f1885ffb640 1 -- 172.21.15.90:0/4044998474 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1894062050 con 0x56018c65e390 2024-04-25T11:39:47.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.863+0000 7f18ab83f740 10 client.16986 fetch_fsmap learned FSMap version 28 2024-04-25T11:39:47.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.863+0000 7f18ab83f740 1 -- 172.21.15.90:0/4044998474 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56018ca83f20 con 0x56018c65e390 2024-04-25T11:39:47.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.863+0000 7f1885ffb640 1 -- 172.21.15.90:0/4044998474 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1890029070 con 0x56018c65e390 2024-04-25T11:39:47.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.863+0000 7f1885ffb640 1 -- 172.21.15.90:0/4044998474 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f189404e810 con 0x56018c65e390 2024-04-25T11:39:47.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.863+0000 7f18ab83f740 10 client.16986 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:39:47.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.863+0000 7f18ab83f740 10 client.16986 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:39:47.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.863+0000 7f18ab83f740 20 client.16986 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:47.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.863+0000 7f18ab83f740 1 --2- 172.21.15.90:0/4044998474 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x56018c5f23a0 0x56018ca86660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.863+0000 7f18ab83f740 4 client.16986 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:39:47.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.863+0000 7f18ab83f740 1 -- 172.21.15.90:0/4044998474 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56018ca000e0 con 0x56018c5f23a0 2024-04-25T11:39:47.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.864+0000 7f189e59c640 1 --2- 172.21.15.90:0/4044998474 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x56018c5f23a0 0x56018ca86660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.864+0000 7f1885ffb640 10 client.16986 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:39:47.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.864+0000 7f189e59c640 1 --2- 172.21.15.90:0/4044998474 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x56018c5f23a0 0x56018ca86660 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=afa63cdad73469ab server_cookie=ffa1f86d46f5b081 in_seq=0 out_seq=0 2024-04-25T11:39:47.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.864+0000 7f1885ffb640 1 -- 172.21.15.90:0/4044998474 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56018ca000e0 con 0x56018c5f23a0 2024-04-25T11:39:47.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.864+0000 7f1885ffb640 10 client.16986 handle_command_reply: tid=0 2024-04-25T11:39:47.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.864+0000 7f18ab83f740 1 client.16986 shutdown 2024-04-25T11:39:47.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.864+0000 7f18ab83f740 1 -- 172.21.15.90:0/4044998474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c65e390 msgr2=0x56018ca827e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.864+0000 7f18ab83f740 1 --2- 172.21.15.90:0/4044998474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c65e390 0x56018ca827e0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f1894000c00 tx=0x7f189000ff10 comp rx=0 tx=0).stop 2024-04-25T11:39:47.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.864+0000 7f18ab83f740 1 -- 172.21.15.90:0/4044998474 shutdown_connections 2024-04-25T11:39:47.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.864+0000 7f18ab83f740 1 --2- 172.21.15.90:0/4044998474 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x56018c5f23a0 0x56018ca86660 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.864+0000 7f18ab83f740 1 --2- 172.21.15.90:0/4044998474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56018c65e390 0x56018ca827e0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.864+0000 7f18ab83f740 1 --2- 172.21.15.90:0/4044998474 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56018c63d6b0 0x56018ca0da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.864+0000 7f18ab83f740 1 --2- 172.21.15.90:0/4044998474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56018c6354d0 0x56018ca0d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.864+0000 7f18ab83f740 1 -- 172.21.15.90:0/4044998474 >> 172.21.15.90:0/4044998474 conn(0x56018c6524e0 msgr2=0x56018c9fbe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:47.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.864+0000 7f1885ffb640 0 client.16986 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:39:47.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.864+0000 7f18ab83f740 1 -- 172.21.15.90:0/4044998474 shutdown_connections 2024-04-25T11:39:47.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.864+0000 7f18ab83f740 1 -- 172.21.15.90:0/4044998474 wait complete. 2024-04-25T11:39:47.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.865+0000 7f18ab83f740 20 client.16986 trim_cache size 0 max 16384 2024-04-25T11:39:47.858 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:39:47.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.865+0000 7f18a4986640 1 -- 172.21.15.90:0/2465622558 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f18880afc60 msgr2=0x7f18880b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.865+0000 7f18a4986640 1 --2- 172.21.15.90:0/2465622558 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f18880afc60 0x7f18880b2170 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f18980a4990 tx=0x7f189800ce70 comp rx=0 tx=0).stop 2024-04-25T11:39:47.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.865+0000 7f18a4986640 1 -- 172.21.15.90:0/2465622558 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18a0108ec0 msgr2=0x7f18a019c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.865+0000 7f18a4986640 1 --2- 172.21.15.90:0/2465622558 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18a0108ec0 0x7f18a019c080 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f189800ec50 tx=0x7f189800c6a0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.865+0000 7f18a4986640 1 -- 172.21.15.90:0/2465622558 shutdown_connections 2024-04-25T11:39:47.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.865+0000 7f18a4986640 1 --2- 172.21.15.90:0/2465622558 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f18880afc60 0x7f18880b2170 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.865+0000 7f18a4986640 1 --2- 172.21.15.90:0/2465622558 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18a0108ec0 0x7f18a019c080 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.865+0000 7f18a4986640 1 --2- 172.21.15.90:0/2465622558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18a0108a20 0x7f18a019bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.865+0000 7f18a4986640 1 --2- 172.21.15.90:0/2465622558 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a0103180 0x7f18a019b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.865+0000 7f18a4986640 1 -- 172.21.15.90:0/2465622558 >> 172.21.15.90:0/2465622558 conn(0x7f18a0078110 msgr2=0x7f18a0104100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:47.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.865+0000 7f18a4986640 1 -- 172.21.15.90:0/2465622558 shutdown_connections 2024-04-25T11:39:47.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:47.865+0000 7f18a4986640 1 -- 172.21.15.90:0/2465622558 wait complete. 2024-04-25T11:39:47.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T11:39:47.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:47 smithi090 ceph-mon[29575]: pgmap v144: 97 pgs: 97 active+clean; 40 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 2.8 MiB/s rd, 644 MiB/s wr, 1.34k op/s 2024-04-25T11:39:47.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:47 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1177614694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:39:47.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.003+0000 7f0ec3964640 1 -- 172.21.15.90:0/1152189751 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ebc109af0 msgr2=0x7f0ebc101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.003+0000 7f0ec3964640 1 --2- 172.21.15.90:0/1152189751 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ebc109af0 0x7f0ebc101b70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8009900 tx=0x7f0eb802a410 comp rx=0 tx=0).stop 2024-04-25T11:39:47.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.003+0000 7f0ec3964640 1 -- 172.21.15.90:0/1152189751 shutdown_connections 2024-04-25T11:39:47.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.003+0000 7f0ec3964640 1 --2- 172.21.15.90:0/1152189751 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ebc102720 0x7f0ebc110570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.003+0000 7f0ec3964640 1 --2- 172.21.15.90:0/1152189751 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ebc109f90 0x7f0ebc1020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.003+0000 7f0ec3964640 1 --2- 172.21.15.90:0/1152189751 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ebc109af0 0x7f0ebc101b70 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.003+0000 7f0ec3964640 1 -- 172.21.15.90:0/1152189751 >> 172.21.15.90:0/1152189751 conn(0x7f0ebc077880 msgr2=0x7f0ebc077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:47.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.003+0000 7f0ec3964640 1 -- 172.21.15.90:0/1152189751 shutdown_connections 2024-04-25T11:39:47.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.003+0000 7f0ec3964640 1 -- 172.21.15.90:0/1152189751 wait complete. 2024-04-25T11:39:47.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.004+0000 7f0ec3964640 1 Processor -- start 2024-04-25T11:39:47.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.004+0000 7f0ec3964640 1 -- start start 2024-04-25T11:39:47.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.004+0000 7f0ec3964640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ebc102720 0x7f0ebc19a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.004+0000 7f0ec3964640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ebc109af0 0x7f0ebc19ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.004+0000 7f0ec3964640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ebc109f90 0x7f0ebc19ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.004+0000 7f0ec3964640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc113d00 con 0x7f0ebc109af0 2024-04-25T11:39:47.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.004+0000 7f0ec3964640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc113b80 con 0x7f0ebc102720 2024-04-25T11:39:47.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.004+0000 7f0ec3964640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc113e80 con 0x7f0ebc109f90 2024-04-25T11:39:47.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.004+0000 7f0ec0eff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ebc109af0 0x7f0ebc19ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.004+0000 7f0ec0eff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ebc109af0 0x7f0ebc19ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49302/0 (socket says 172.21.15.90:49302) 2024-04-25T11:39:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.004+0000 7f0ec0eff640 1 -- 172.21.15.90:0/3174231850 learned_addr learned my addr 172.21.15.90:0/3174231850 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.004+0000 7f0ec1700640 1 --2- 172.21.15.90:0/3174231850 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ebc102720 0x7f0ebc19a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.004+0000 7f0ec1f01640 1 --2- 172.21.15.90:0/3174231850 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ebc109f90 0x7f0ebc19ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.005+0000 7f0ec0eff640 1 -- 172.21.15.90:0/3174231850 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ebc109f90 msgr2=0x7f0ebc19ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.005+0000 7f0ec0eff640 1 --2- 172.21.15.90:0/3174231850 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ebc109f90 0x7f0ebc19ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.005+0000 7f0ec0eff640 1 -- 172.21.15.90:0/3174231850 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ebc102720 msgr2=0x7f0ebc19a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.005+0000 7f0ec0eff640 1 --2- 172.21.15.90:0/3174231850 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ebc102720 0x7f0ebc19a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.005+0000 7f0ec0eff640 1 -- 172.21.15.90:0/3174231850 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ebc19f580 con 0x7f0ebc109af0 2024-04-25T11:39:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.005+0000 7f0ec0eff640 1 --2- 172.21.15.90:0/3174231850 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ebc109af0 0x7f0ebc19ac80 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f0eac00e9f0 tx=0x7f0eac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.005+0000 7f0eb27fc640 1 -- 172.21.15.90:0/3174231850 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eac00ccc0 con 0x7f0ebc109af0 2024-04-25T11:39:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.005+0000 7f0ec3964640 1 -- 172.21.15.90:0/3174231850 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ebc19f870 con 0x7f0ebc109af0 2024-04-25T11:39:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.005+0000 7f0eb27fc640 1 -- 172.21.15.90:0/3174231850 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0eac0054d0 con 0x7f0ebc109af0 2024-04-25T11:39:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.005+0000 7f0ec3964640 1 -- 172.21.15.90:0/3174231850 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ebc1a71f0 con 0x7f0ebc109af0 2024-04-25T11:39:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.005+0000 7f0eb27fc640 1 -- 172.21.15.90:0/3174231850 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eac010430 con 0x7f0ebc109af0 2024-04-25T11:39:48.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.007+0000 7f0eca812740 1 Processor -- start 2024-04-25T11:39:48.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.008+0000 7f0eb27fc640 1 -- 172.21.15.90:0/3174231850 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f0eac0105d0 con 0x7f0ebc109af0 2024-04-25T11:39:48.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.008+0000 7f0eca812740 1 -- start start 2024-04-25T11:39:48.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.008+0000 7f0eca812740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ea10879390 0x55ea10c1a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.008+0000 7f0eca812740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ea108504d0 0x55ea10c1ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.008+0000 7f0eca812740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea108586b0 0x55ea10c276a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.008+0000 7f0eca812740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ea10b7fd80 con 0x55ea108586b0 2024-04-25T11:39:48.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.008+0000 7f0eca812740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ea10b39a20 con 0x55ea10879390 2024-04-25T11:39:48.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.008+0000 7f0eca812740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ea10b7bd40 con 0x55ea108504d0 2024-04-25T11:39:48.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.008+0000 7f0ec0eff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea108586b0 0x55ea10c276a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.008+0000 7f0ec0eff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea108586b0 0x55ea10c276a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49308/0 (socket says 172.21.15.90:49308) 2024-04-25T11:39:48.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.008+0000 7f0ec0eff640 1 -- 172.21.15.90:0/4129449794 learned_addr learned my addr 172.21.15.90:0/4129449794 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:48.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.008+0000 7f0ec1f01640 1 --2- 172.21.15.90:0/4129449794 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ea10879390 0x55ea10c1a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.008+0000 7f0ec1700640 1 --2- 172.21.15.90:0/4129449794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ea108504d0 0x55ea10c1ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.008+0000 7f0eb27fc640 1 --2- 172.21.15.90:0/3174231850 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0ea80afc60 0x7f0ea80b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.008+0000 7f0ec0eff640 1 -- 172.21.15.90:0/4129449794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ea108504d0 msgr2=0x55ea10c1ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:48.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.008+0000 7f0ec0eff640 1 --2- 172.21.15.90:0/4129449794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ea108504d0 0x55ea10c1ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.008+0000 7f0ec0eff640 1 -- 172.21.15.90:0/4129449794 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ea10879390 msgr2=0x55ea10c1a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:48.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.008+0000 7f0ec0eff640 1 --2- 172.21.15.90:0/4129449794 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ea10879390 0x55ea10c1a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.008+0000 7f0ec0eff640 1 -- 172.21.15.90:0/4129449794 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea10b44c30 con 0x55ea108586b0 2024-04-25T11:39:48.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.008+0000 7f0ec1f01640 1 --2- 172.21.15.90:0/4129449794 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ea10879390 0x55ea10c1a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:48.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.008+0000 7f0ec1f01640 1 --2- 172.21.15.90:0/3174231850 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0ea80afc60 0x7f0ea80b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.009+0000 7f0eb27fc640 1 -- 172.21.15.90:0/3174231850 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f0eac0df4e0 con 0x7f0ebc109af0 2024-04-25T11:39:48.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.009+0000 7f0ec0eff640 1 --2- 172.21.15.90:0/4129449794 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea108586b0 0x55ea10c276a0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f0eac0dcb40 tx=0x7f0eac021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ebba321e6b8f599 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:48.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.009+0000 7f0ea77fe640 1 -- 172.21.15.90:0/4129449794 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eac0a73f0 con 0x55ea108586b0 2024-04-25T11:39:48.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.009+0000 7f0ec1f01640 1 --2- 172.21.15.90:0/3174231850 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0ea80afc60 0x7f0ea80b2170 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x55ea10c17980 tx=0x7f0eb4004a40 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:48.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.009+0000 7f0ea77fe640 1 -- 172.21.15.90:0/4129449794 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0eac0a7130 con 0x55ea108586b0 2024-04-25T11:39:48.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.009+0000 7f0ea77fe640 1 -- 172.21.15.90:0/4129449794 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eac0a6d90 con 0x55ea108586b0 2024-04-25T11:39:48.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.010+0000 7f0eca812740 1 -- 172.21.15.90:0/4129449794 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea108586b0 msgr2=0x55ea10c276a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:48.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.010+0000 7f0eca812740 1 --2- 172.21.15.90:0/4129449794 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea108586b0 0x55ea10c276a0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f0eac0dcb40 tx=0x7f0eac021000 comp rx=0 tx=0).stop 2024-04-25T11:39:48.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.010+0000 7f0eca812740 1 -- 172.21.15.90:0/4129449794 shutdown_connections 2024-04-25T11:39:48.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.010+0000 7f0eca812740 1 --2- 172.21.15.90:0/4129449794 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea108586b0 0x55ea10c276a0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.010+0000 7f0eca812740 1 --2- 172.21.15.90:0/4129449794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ea108504d0 0x55ea10c1ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.010+0000 7f0eca812740 1 --2- 172.21.15.90:0/4129449794 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ea10879390 0x55ea10c1a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.010+0000 7f0eca812740 1 -- 172.21.15.90:0/4129449794 >> 172.21.15.90:0/4129449794 conn(0x55ea1086d4e0 msgr2=0x55ea10c16fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:48.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.010+0000 7f0eca812740 1 -- 172.21.15.90:0/4129449794 shutdown_connections 2024-04-25T11:39:48.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.010+0000 7f0eca812740 1 -- 172.21.15.90:0/4129449794 wait complete. 2024-04-25T11:39:48.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.010+0000 7f0eca812740 1 -- start start 2024-04-25T11:39:48.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.011+0000 7f0eca812740 1 Processor -- start 2024-04-25T11:39:48.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.011+0000 7f0eca812740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea108504d0 0x55ea10c25220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.011+0000 7f0eca812740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ea108586b0 0x55ea10c25760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.011+0000 7f0eca812740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ea10879390 0x55ea10c311b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.011+0000 7f0eca812740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ea10c27d30 con 0x55ea108504d0 2024-04-25T11:39:48.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.011+0000 7f0eca812740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ea10b06dc0 con 0x55ea10879390 2024-04-25T11:39:48.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.011+0000 7f0eca812740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ea10c27eb0 con 0x55ea108586b0 2024-04-25T11:39:48.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.011+0000 7f0ec1700640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea108504d0 0x55ea10c25220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.011+0000 7f0eca812740 10 client.0 _ll_register_callbacks cb 0x55ea10b22600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:48.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.011+0000 7f0ec1700640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea108504d0 0x55ea10c25220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49316/0 (socket says 172.21.15.90:49316) 2024-04-25T11:39:48.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.011+0000 7f0ec1700640 1 -- 172.21.15.90:0/3263087199 learned_addr learned my addr 172.21.15.90:0/3263087199 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:48.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.011+0000 7f0ec1f01640 1 --2- 172.21.15.90:0/3263087199 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ea10879390 0x55ea10c311b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.011+0000 7f0ec0eff640 1 --2- 172.21.15.90:0/3263087199 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ea108586b0 0x55ea10c25760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.011+0000 7f0ec1700640 1 -- 172.21.15.90:0/3263087199 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ea108586b0 msgr2=0x55ea10c25760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:48.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.011+0000 7f0ec1700640 1 --2- 172.21.15.90:0/3263087199 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ea108586b0 0x55ea10c25760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.011+0000 7f0ec1700640 1 -- 172.21.15.90:0/3263087199 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ea10879390 msgr2=0x55ea10c311b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:48.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.011+0000 7f0ec1700640 1 --2- 172.21.15.90:0/3263087199 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ea10879390 0x55ea10c311b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.011+0000 7f0ec1700640 1 -- 172.21.15.90:0/3263087199 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea108913f0 con 0x55ea108504d0 2024-04-25T11:39:48.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.012+0000 7f0ec1700640 1 --2- 172.21.15.90:0/3263087199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea108504d0 0x55ea10c25220 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8009900 tx=0x7f0eb8005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f108ffe5c79073dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:48.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.012+0000 7f0ea4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:48.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.012+0000 7f0ea4ff9640 1 -- 172.21.15.90:0/3263087199 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8035040 con 0x55ea108504d0 2024-04-25T11:39:48.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.012+0000 7f0ea4ff9640 1 -- 172.21.15.90:0/3263087199 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f0eb80317a0 con 0x55ea108504d0 2024-04-25T11:39:48.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.012+0000 7f0eca812740 1 -- 172.21.15.90:0/3263087199 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0eac0a7130 con 0x55ea108504d0 2024-04-25T11:39:48.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.012+0000 7f0ea4ff9640 1 -- 172.21.15.90:0/3263087199 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0eb8004040 con 0x55ea108504d0 2024-04-25T11:39:48.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.012+0000 7f0eca812740 10 client.16998 fetch_fsmap learned FSMap version 28 2024-04-25T11:39:48.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.012+0000 7f0eca812740 1 -- 172.21.15.90:0/3263087199 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ea1095b7d0 con 0x55ea108504d0 2024-04-25T11:39:48.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.012+0000 7f0ea4ff9640 1 -- 172.21.15.90:0/3263087199 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0eac0a7130 con 0x55ea108504d0 2024-04-25T11:39:48.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.012+0000 7f0ea4ff9640 1 -- 172.21.15.90:0/3263087199 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f0eb80047e0 con 0x55ea108504d0 2024-04-25T11:39:48.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.012+0000 7f0eca812740 10 client.16998 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:39:48.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.012+0000 7f0eca812740 10 client.16998 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:39:48.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.012+0000 7f0eca812740 20 client.16998 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:48.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.012+0000 7f0eca812740 1 --2- 172.21.15.90:0/3263087199 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55ea10837f60 0x55ea10c375e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.012+0000 7f0eca812740 4 client.16998 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:39:48.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.012+0000 7f0eca812740 1 -- 172.21.15.90:0/3263087199 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea10b8d990 con 0x55ea10837f60 2024-04-25T11:39:48.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.013+0000 7f0ec0eff640 1 --2- 172.21.15.90:0/3263087199 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55ea10837f60 0x55ea10c375e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.013+0000 7f0ea4ff9640 10 client.16998 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:39:48.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.013+0000 7f0ec0eff640 1 --2- 172.21.15.90:0/3263087199 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55ea10837f60 0x55ea10c375e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=84fa240af0c26020 server_cookie=5ed50c43eeddbf87 in_seq=0 out_seq=0 2024-04-25T11:39:48.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.014+0000 7f0ea4ff9640 1 -- 172.21.15.90:0/3263087199 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ea10b8d990 con 0x55ea10837f60 2024-04-25T11:39:48.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.014+0000 7f0ea4ff9640 10 client.16998 handle_command_reply: tid=0 2024-04-25T11:39:48.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.014+0000 7f0eca812740 1 client.16998 shutdown 2024-04-25T11:39:48.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.014+0000 7f0eca812740 1 -- 172.21.15.90:0/3263087199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea108504d0 msgr2=0x55ea10c25220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:48.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.014+0000 7f0eca812740 1 --2- 172.21.15.90:0/3263087199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea108504d0 0x55ea10c25220 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8009900 tx=0x7f0eb8005f00 comp rx=0 tx=0).stop 2024-04-25T11:39:48.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.014+0000 7f0eca812740 1 -- 172.21.15.90:0/3263087199 shutdown_connections 2024-04-25T11:39:48.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.014+0000 7f0eca812740 1 --2- 172.21.15.90:0/3263087199 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55ea10837f60 0x55ea10c375e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.014+0000 7f0eca812740 1 --2- 172.21.15.90:0/3263087199 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ea10879390 0x55ea10c311b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.014+0000 7f0ea4ff9640 0 client.16998 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:39:48.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.014+0000 7f0eca812740 1 --2- 172.21.15.90:0/3263087199 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ea108586b0 0x55ea10c25760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.014+0000 7f0eca812740 1 --2- 172.21.15.90:0/3263087199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea108504d0 0x55ea10c25220 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.014+0000 7f0eca812740 1 -- 172.21.15.90:0/3263087199 >> 172.21.15.90:0/3263087199 conn(0x55ea1086d4e0 msgr2=0x55ea10c1c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:48.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.014+0000 7f0eca812740 1 -- 172.21.15.90:0/3263087199 shutdown_connections 2024-04-25T11:39:48.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.014+0000 7f0eca812740 1 -- 172.21.15.90:0/3263087199 wait complete. 2024-04-25T11:39:48.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.014+0000 7f0eca812740 20 client.16998 trim_cache size 0 max 16384 2024-04-25T11:39:48.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.028+0000 7f0eca812740 1 Processor -- start 2024-04-25T11:39:48.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.028+0000 7f0eca812740 1 -- start start 2024-04-25T11:39:48.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.028+0000 7f0eca812740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ea10837f60 0x55ea10c16ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.028+0000 7f0eca812740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ea108504d0 0x55ea10c30e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.028+0000 7f0eca812740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea108586b0 0x55ea10c3cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.028+0000 7f0eca812740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc1afba0 con 0x55ea108586b0 2024-04-25T11:39:48.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.028+0000 7f0eca812740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc1b16e0 con 0x55ea108504d0 2024-04-25T11:39:48.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.028+0000 7f0eca812740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc1b0270 con 0x55ea10837f60 2024-04-25T11:39:48.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.028+0000 7f0ec1f01640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea108586b0 0x55ea10c3cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.028+0000 7f0ec0eff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ea108504d0 0x55ea10c30e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.028+0000 7f0ec1700640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ea10837f60 0x55ea10c16ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.028+0000 7f0ec1f01640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea108586b0 0x55ea10c3cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49322/0 (socket says 172.21.15.90:49322) 2024-04-25T11:39:48.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.028+0000 7f0ec0eff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ea108504d0 0x55ea10c30e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:56380/0 (socket says 172.21.15.90:56380) 2024-04-25T11:39:48.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.028+0000 7f0ec1f01640 1 -- 172.21.15.90:0/3854368686 learned_addr learned my addr 172.21.15.90:0/3854368686 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:48.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.029+0000 7f0ec1700640 1 -- 172.21.15.90:0/3854368686 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ea108504d0 msgr2=0x55ea10c30e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:48.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.029+0000 7f0ec1700640 1 --2- 172.21.15.90:0/3854368686 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ea108504d0 0x55ea10c30e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.029+0000 7f0ec1700640 1 -- 172.21.15.90:0/3854368686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea108586b0 msgr2=0x55ea10c3cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:48.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.029+0000 7f0ec1700640 1 --2- 172.21.15.90:0/3854368686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea108586b0 0x55ea10c3cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.029+0000 7f0ec1700640 1 -- 172.21.15.90:0/3854368686 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea10c18030 con 0x55ea10837f60 2024-04-25T11:39:48.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.029+0000 7f0ec1700640 1 --2- 172.21.15.90:0/3854368686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ea10837f60 0x55ea10c16ed0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8004bb0 tx=0x7f0eb8005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6380c003a5b3648 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:48.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.029+0000 7f0ea6ffd640 1 -- 172.21.15.90:0/3854368686 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8035030 con 0x55ea10837f60 2024-04-25T11:39:48.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.029+0000 7f0ea6ffd640 1 -- 172.21.15.90:0/3854368686 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0eb80048c0 con 0x55ea10837f60 2024-04-25T11:39:48.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.029+0000 7f0ea6ffd640 1 -- 172.21.15.90:0/3854368686 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8003ab0 con 0x55ea10837f60 2024-04-25T11:39:48.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.030+0000 7f0eca812740 1 -- 172.21.15.90:0/3854368686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ea10837f60 msgr2=0x55ea10c16ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:48.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.030+0000 7f0eca812740 1 --2- 172.21.15.90:0/3854368686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ea10837f60 0x55ea10c16ed0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8004bb0 tx=0x7f0eb8005fd0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.030+0000 7f0eca812740 1 -- 172.21.15.90:0/3854368686 shutdown_connections 2024-04-25T11:39:48.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.030+0000 7f0eca812740 1 --2- 172.21.15.90:0/3854368686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea108586b0 0x55ea10c3cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.030+0000 7f0eca812740 1 --2- 172.21.15.90:0/3854368686 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ea108504d0 0x55ea10c30e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.030+0000 7f0eca812740 1 --2- 172.21.15.90:0/3854368686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ea10837f60 0x55ea10c16ed0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.030+0000 7f0eca812740 1 -- 172.21.15.90:0/3854368686 >> 172.21.15.90:0/3854368686 conn(0x55ea1086d4e0 msgr2=0x55ea10c167c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:48.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.030+0000 7f0eca812740 1 -- 172.21.15.90:0/3854368686 shutdown_connections 2024-04-25T11:39:48.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.030+0000 7f0eca812740 1 -- 172.21.15.90:0/3854368686 wait complete. 2024-04-25T11:39:48.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.030+0000 7f0eca812740 1 -- start start 2024-04-25T11:39:48.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.031+0000 7f0eca812740 1 Processor -- start 2024-04-25T11:39:48.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.031+0000 7f0eca812740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea10837f60 0x55ea10c2d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.031+0000 7f0eca812740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ea108504d0 0x55ea10c28910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.031+0000 7f0eca812740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ea108586b0 0x55ea10c28e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.031+0000 7f0eca812740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc1b1010 con 0x55ea10837f60 2024-04-25T11:39:48.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.031+0000 7f0eca812740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc1b0940 con 0x55ea108504d0 2024-04-25T11:39:48.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.031+0000 7f0eca812740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc1adb80 con 0x55ea108586b0 2024-04-25T11:39:48.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.031+0000 7f0eca812740 10 client.0 _ll_register_callbacks cb 0x7f0eb8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:48.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.031+0000 7f0ec1700640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea10837f60 0x55ea10c2d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.031+0000 7f0ec1700640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea10837f60 0x55ea10c2d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49326/0 (socket says 172.21.15.90:49326) 2024-04-25T11:39:48.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.031+0000 7f0ec1700640 1 -- 172.21.15.90:0/1763203410 learned_addr learned my addr 172.21.15.90:0/1763203410 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:48.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.031+0000 7f0ec0eff640 1 --2- 172.21.15.90:0/1763203410 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ea108504d0 0x55ea10c28910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.031+0000 7f0ec1f01640 1 --2- 172.21.15.90:0/1763203410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ea108586b0 0x55ea10c28e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.031+0000 7f0ec1700640 1 -- 172.21.15.90:0/1763203410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ea108586b0 msgr2=0x55ea10c28e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:48.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.031+0000 7f0ec1700640 1 --2- 172.21.15.90:0/1763203410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ea108586b0 0x55ea10c28e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.031+0000 7f0ec1700640 1 -- 172.21.15.90:0/1763203410 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ea108504d0 msgr2=0x55ea10c28910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:48.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.031+0000 7f0ec1700640 1 --2- 172.21.15.90:0/1763203410 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ea108504d0 0x55ea10c28910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.031+0000 7f0ec1700640 1 -- 172.21.15.90:0/1763203410 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea10c29440 con 0x55ea10837f60 2024-04-25T11:39:48.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.032+0000 7f0ec1700640 1 --2- 172.21.15.90:0/1763203410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea10837f60 0x55ea10c2d5c0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8031440 tx=0x7f0eb8032b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f840ce5a6c57c671 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:48.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.032+0000 7f0ea4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:48.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.032+0000 7f0ea4ff9640 1 -- 172.21.15.90:0/1763203410 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8035070 con 0x55ea10837f60 2024-04-25T11:39:48.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.032+0000 7f0ea4ff9640 1 -- 172.21.15.90:0/1763203410 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f0eb8005210 con 0x55ea10837f60 2024-04-25T11:39:48.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.032+0000 7f0eca812740 1 -- 172.21.15.90:0/1763203410 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0eb80048c0 con 0x55ea10837f60 2024-04-25T11:39:48.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.032+0000 7f0ea4ff9640 1 -- 172.21.15.90:0/1763203410 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0eb8068050 con 0x55ea10837f60 2024-04-25T11:39:48.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.032+0000 7f0eca812740 10 client.17004 fetch_fsmap learned FSMap version 28 2024-04-25T11:39:48.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.032+0000 7f0eca812740 1 -- 172.21.15.90:0/1763203410 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ea10c9f6f0 con 0x55ea10837f60 2024-04-25T11:39:48.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.032+0000 7f0ea4ff9640 1 -- 172.21.15.90:0/1763203410 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0eb80048c0 con 0x55ea10837f60 2024-04-25T11:39:48.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.032+0000 7f0ea4ff9640 1 -- 172.21.15.90:0/1763203410 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f0eb8056030 con 0x55ea10837f60 2024-04-25T11:39:48.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.032+0000 7f0eca812740 10 client.17004 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:39:48.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.032+0000 7f0eca812740 10 client.17004 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:39:48.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.032+0000 7f0eca812740 20 client.17004 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:48.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.032+0000 7f0eca812740 1 --2- 172.21.15.90:0/1763203410 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55ea10879390 0x55ea10ca1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.032+0000 7f0eca812740 4 client.17004 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:39:48.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.032+0000 7f0eca812740 1 -- 172.21.15.90:0/1763203410 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ea10b87800 con 0x55ea10879390 2024-04-25T11:39:48.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.033+0000 7f0ec0eff640 1 --2- 172.21.15.90:0/1763203410 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55ea10879390 0x55ea10ca1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.033+0000 7f0ea4ff9640 10 client.17004 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:39:48.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.033+0000 7f0ec0eff640 1 --2- 172.21.15.90:0/1763203410 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55ea10879390 0x55ea10ca1d00 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=181913cccc61faa1 server_cookie=17e2499433109c94 in_seq=0 out_seq=0 2024-04-25T11:39:48.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.033+0000 7f0ea4ff9640 1 -- 172.21.15.90:0/1763203410 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ea10b87800 con 0x55ea10879390 2024-04-25T11:39:48.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.033+0000 7f0ea4ff9640 10 client.17004 handle_command_reply: tid=0 2024-04-25T11:39:48.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.033+0000 7f0eca812740 1 client.17004 shutdown 2024-04-25T11:39:48.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.033+0000 7f0eca812740 1 -- 172.21.15.90:0/1763203410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea10837f60 msgr2=0x55ea10c2d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:48.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.033+0000 7f0eca812740 1 --2- 172.21.15.90:0/1763203410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea10837f60 0x55ea10c2d5c0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8031440 tx=0x7f0eb8032b10 comp rx=0 tx=0).stop 2024-04-25T11:39:48.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.034+0000 7f0eca812740 1 -- 172.21.15.90:0/1763203410 shutdown_connections 2024-04-25T11:39:48.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.034+0000 7f0eca812740 1 --2- 172.21.15.90:0/1763203410 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55ea10879390 0x55ea10ca1d00 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.034+0000 7f0eca812740 1 --2- 172.21.15.90:0/1763203410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ea108586b0 0x55ea10c28e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.034+0000 7f0eca812740 1 --2- 172.21.15.90:0/1763203410 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ea108504d0 0x55ea10c28910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.034+0000 7f0eca812740 1 --2- 172.21.15.90:0/1763203410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ea10837f60 0x55ea10c2d5c0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.034+0000 7f0eca812740 1 -- 172.21.15.90:0/1763203410 >> 172.21.15.90:0/1763203410 conn(0x55ea1086d4e0 msgr2=0x55ea10c266c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:48.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.034+0000 7f0ea4ff9640 0 client.17004 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:39:48.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.034+0000 7f0eca812740 1 -- 172.21.15.90:0/1763203410 shutdown_connections 2024-04-25T11:39:48.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.034+0000 7f0eca812740 1 -- 172.21.15.90:0/1763203410 wait complete. 2024-04-25T11:39:48.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.034+0000 7f0eca812740 20 client.17004 trim_cache size 0 max 16384 2024-04-25T11:39:48.028 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:39:48.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.034+0000 7f0ec3964640 1 -- 172.21.15.90:0/3174231850 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0ea80afc60 msgr2=0x7f0ea80b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:48.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.034+0000 7f0ec3964640 1 --2- 172.21.15.90:0/3174231850 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0ea80afc60 0x7f0ea80b2170 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x55ea10c17980 tx=0x7f0eb4004a40 comp rx=0 tx=0).stop 2024-04-25T11:39:48.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.034+0000 7f0ec3964640 1 -- 172.21.15.90:0/3174231850 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ebc109af0 msgr2=0x7f0ebc19ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:48.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.034+0000 7f0ec3964640 1 --2- 172.21.15.90:0/3174231850 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ebc109af0 0x7f0ebc19ac80 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f0eac00e9f0 tx=0x7f0eac00eeb0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.034+0000 7f0ec3964640 1 -- 172.21.15.90:0/3174231850 shutdown_connections 2024-04-25T11:39:48.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.034+0000 7f0ec3964640 1 --2- 172.21.15.90:0/3174231850 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0ea80afc60 0x7f0ea80b2170 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.034+0000 7f0ec3964640 1 --2- 172.21.15.90:0/3174231850 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ebc109f90 0x7f0ebc19ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.034+0000 7f0ec3964640 1 --2- 172.21.15.90:0/3174231850 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ebc109af0 0x7f0ebc19ac80 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.034+0000 7f0ec3964640 1 --2- 172.21.15.90:0/3174231850 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ebc102720 0x7f0ebc19a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.034+0000 7f0ec3964640 1 -- 172.21.15.90:0/3174231850 >> 172.21.15.90:0/3174231850 conn(0x7f0ebc077880 msgr2=0x7f0ebc1051f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:48.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.034+0000 7f0ec3964640 1 -- 172.21.15.90:0/3174231850 shutdown_connections 2024-04-25T11:39:48.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.034+0000 7f0ec3964640 1 -- 172.21.15.90:0/3174231850 wait complete. 2024-04-25T11:39:48.033 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T11:39:48.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:47 smithi138 ceph-mon[32328]: pgmap v144: 97 pgs: 97 active+clean; 40 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 2.8 MiB/s rd, 644 MiB/s wr, 1.34k op/s 2024-04-25T11:39:48.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:47 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1177614694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:39:48.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:47 smithi156 ceph-mon[33045]: pgmap v144: 97 pgs: 97 active+clean; 40 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 2.8 MiB/s rd, 644 MiB/s wr, 1.34k op/s 2024-04-25T11:39:48.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:47 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1177614694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:39:48.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.167+0000 7f7b0c296640 1 -- 172.21.15.90:0/715920635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b0410e550 msgr2=0x7f7b04110970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:48.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.167+0000 7f7b0c296640 1 --2- 172.21.15.90:0/715920635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b0410e550 0x7f7b04110970 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f7afc009ca0 tx=0x7f7afc02a8c0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.167+0000 7f7b0c296640 1 -- 172.21.15.90:0/715920635 shutdown_connections 2024-04-25T11:39:48.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.167+0000 7f7b0c296640 1 --2- 172.21.15.90:0/715920635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b0410e550 0x7f7b04110970 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.167+0000 7f7b0c296640 1 --2- 172.21.15.90:0/715920635 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7b04102050 0x7f7b04102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.167+0000 7f7b0c296640 1 --2- 172.21.15.90:0/715920635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b04109ba0 0x7f7b04101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.167+0000 7f7b0c296640 1 -- 172.21.15.90:0/715920635 >> 172.21.15.90:0/715920635 conn(0x7f7b04077880 msgr2=0x7f7b04077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:48.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.167+0000 7f7b0c296640 1 -- 172.21.15.90:0/715920635 shutdown_connections 2024-04-25T11:39:48.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.167+0000 7f7b0c296640 1 -- 172.21.15.90:0/715920635 wait complete. 2024-04-25T11:39:48.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.168+0000 7f7b0c296640 1 Processor -- start 2024-04-25T11:39:48.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.168+0000 7f7b0c296640 1 -- start start 2024-04-25T11:39:48.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.168+0000 7f7b0c296640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b04102050 0x7f7b0419a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.168+0000 7f7b0c296640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7b04109ba0 0x7f7b0419acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.168+0000 7f7b0c296640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b0410e550 0x7f7b0419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.168+0000 7f7b0c296640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7b04113d10 con 0x7f7b0410e550 2024-04-25T11:39:48.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.168+0000 7f7b0c296640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7b04113b90 con 0x7f7b04109ba0 2024-04-25T11:39:48.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.168+0000 7f7b0c296640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7b04113e90 con 0x7f7b04102050 2024-04-25T11:39:48.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.168+0000 7f7b0a833640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b0410e550 0x7f7b0419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.168+0000 7f7b0a833640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b0410e550 0x7f7b0419efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49350/0 (socket says 172.21.15.90:49350) 2024-04-25T11:39:48.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.168+0000 7f7b0a833640 1 -- 172.21.15.90:0/3262445482 learned_addr learned my addr 172.21.15.90:0/3262445482 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:48.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.168+0000 7f7b0a032640 1 --2- 172.21.15.90:0/3262445482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b04102050 0x7f7b0419a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.168+0000 7f7b09831640 1 --2- 172.21.15.90:0/3262445482 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7b04109ba0 0x7f7b0419acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.168+0000 7f7b0a833640 1 -- 172.21.15.90:0/3262445482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b04102050 msgr2=0x7f7b0419a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:48.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.168+0000 7f7b0a833640 1 --2- 172.21.15.90:0/3262445482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b04102050 0x7f7b0419a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.168+0000 7f7b0a833640 1 -- 172.21.15.90:0/3262445482 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7b04109ba0 msgr2=0x7f7b0419acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:48.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.168+0000 7f7b0a833640 1 --2- 172.21.15.90:0/3262445482 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7b04109ba0 0x7f7b0419acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.169+0000 7f7b0a833640 1 -- 172.21.15.90:0/3262445482 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b0419f5f0 con 0x7f7b0410e550 2024-04-25T11:39:48.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.169+0000 7f7b0a833640 1 --2- 172.21.15.90:0/3262445482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b0410e550 0x7f7b0419efb0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f7afc009430 tx=0x7f7afc031660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:48.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.169+0000 7f7afb7fe640 1 -- 172.21.15.90:0/3262445482 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7afc003b70 con 0x7f7b0410e550 2024-04-25T11:39:48.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.169+0000 7f7afb7fe640 1 -- 172.21.15.90:0/3262445482 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7afc003d10 con 0x7f7b0410e550 2024-04-25T11:39:48.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.169+0000 7f7b0c296640 1 -- 172.21.15.90:0/3262445482 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b0419f880 con 0x7f7b0410e550 2024-04-25T11:39:48.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.169+0000 7f7afb7fe640 1 -- 172.21.15.90:0/3262445482 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7afc02fa50 con 0x7f7b0410e550 2024-04-25T11:39:48.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.169+0000 7f7b0c296640 1 -- 172.21.15.90:0/3262445482 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b0419fdd0 con 0x7f7b0410e550 2024-04-25T11:39:48.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.172+0000 7f7b13228740 1 Processor -- start 2024-04-25T11:39:48.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.172+0000 7f7afb7fe640 1 -- 172.21.15.90:0/3262445482 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f7afc02fbf0 con 0x7f7b0410e550 2024-04-25T11:39:48.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.172+0000 7f7b13228740 1 -- start start 2024-04-25T11:39:48.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.172+0000 7f7b13228740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558482bf6390 0x558482f976a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.172+0000 7f7b13228740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558482bcd4d0 0x558482f97be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.172+0000 7f7b13228740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558482bd56b0 0x558482fa8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.172+0000 7f7b13228740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558482efcd80 con 0x558482bf6390 2024-04-25T11:39:48.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.172+0000 7f7b13228740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x558482eb6a20 con 0x558482bcd4d0 2024-04-25T11:39:48.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.172+0000 7f7b13228740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558482ef8d40 con 0x558482bd56b0 2024-04-25T11:39:48.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.173+0000 7f7b09831640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558482bf6390 0x558482f976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.173+0000 7f7b0a032640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558482bcd4d0 0x558482f97be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.173+0000 7f7b09831640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558482bf6390 0x558482f976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49352/0 (socket says 172.21.15.90:49352) 2024-04-25T11:39:48.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.173+0000 7f7b09831640 1 -- 172.21.15.90:0/1201796692 learned_addr learned my addr 172.21.15.90:0/1201796692 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:48.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.173+0000 7f7afb7fe640 1 --2- 172.21.15.90:0/3262445482 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f7af00afc60 0x7f7af00b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.173+0000 7f7b09831640 1 --2- 172.21.15.90:0/1201796692 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558482bd56b0 0x558482fa8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.173+0000 7f7b0a032640 1 -- 172.21.15.90:0/1201796692 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558482bd56b0 msgr2=0x558482fa8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:48.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.173+0000 7f7b0a032640 1 --2- 172.21.15.90:0/1201796692 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558482bd56b0 0x558482fa8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.173+0000 7f7b0a032640 1 -- 172.21.15.90:0/1201796692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558482bf6390 msgr2=0x558482f976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:48.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.173+0000 7f7b0a032640 1 --2- 172.21.15.90:0/1201796692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558482bf6390 0x558482f976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.173+0000 7f7b0a032640 1 -- 172.21.15.90:0/1201796692 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558482ec1c30 con 0x558482bcd4d0 2024-04-25T11:39:48.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.173+0000 7f7b09831640 1 --2- 172.21.15.90:0/1201796692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558482bf6390 0x558482f976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:48.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.173+0000 7f7b0a833640 1 --2- 172.21.15.90:0/3262445482 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f7af00afc60 0x7f7af00b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.173+0000 7f7afb7fe640 1 -- 172.21.15.90:0/3262445482 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f7afc100460 con 0x7f7b0410e550 2024-04-25T11:39:48.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.173+0000 7f7b0a032640 1 --2- 172.21.15.90:0/1201796692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558482bcd4d0 0x558482f97be0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7b04069bf0 tx=0x7f7af4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9bbde00d2ed04f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:48.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.173+0000 7f7af8bfe640 1 -- 172.21.15.90:0/1201796692 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af401e070 con 0x558482bcd4d0 2024-04-25T11:39:48.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.174+0000 7f7af8bfe640 1 -- 172.21.15.90:0/1201796692 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7af4009d50 con 0x558482bcd4d0 2024-04-25T11:39:48.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.174+0000 7f7af8bfe640 1 -- 172.21.15.90:0/1201796692 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af4019450 con 0x558482bcd4d0 2024-04-25T11:39:48.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.174+0000 7f7b13228740 1 -- 172.21.15.90:0/1201796692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558482bcd4d0 msgr2=0x558482f97be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:48.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.174+0000 7f7b13228740 1 --2- 172.21.15.90:0/1201796692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558482bcd4d0 0x558482f97be0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7b04069bf0 tx=0x7f7af4011810 comp rx=0 tx=0).stop 2024-04-25T11:39:48.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.174+0000 7f7b0a833640 1 --2- 172.21.15.90:0/3262445482 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f7af00afc60 0x7f7af00b2170 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7afc0fdd10 tx=0x7f7afc041b80 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:48.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.175+0000 7f7b13228740 1 -- 172.21.15.90:0/1201796692 shutdown_connections 2024-04-25T11:39:48.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.175+0000 7f7b13228740 1 --2- 172.21.15.90:0/1201796692 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558482bd56b0 0x558482fa8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.175+0000 7f7b13228740 1 --2- 172.21.15.90:0/1201796692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558482bcd4d0 0x558482f97be0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.175+0000 7f7b13228740 1 --2- 172.21.15.90:0/1201796692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558482bf6390 0x558482f976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.175+0000 7f7b13228740 1 -- 172.21.15.90:0/1201796692 >> 172.21.15.90:0/1201796692 conn(0x558482bea4e0 msgr2=0x558482f93fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:48.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.175+0000 7f7b13228740 1 -- 172.21.15.90:0/1201796692 shutdown_connections 2024-04-25T11:39:48.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.175+0000 7f7b13228740 1 -- 172.21.15.90:0/1201796692 wait complete. 2024-04-25T11:39:48.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.175+0000 7f7b13228740 1 -- start start 2024-04-25T11:39:48.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.175+0000 7f7b13228740 1 Processor -- start 2024-04-25T11:39:48.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.176+0000 7f7b13228740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558482bcd4d0 0x558482fa6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.176+0000 7f7b13228740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558482bd56b0 0x558482fa6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.176+0000 7f7b13228740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558482bf6390 0x558482fb26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.176+0000 7f7b13228740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558482fa91a0 con 0x558482bd56b0 2024-04-25T11:39:48.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.176+0000 7f7b13228740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x558482e83dc0 con 0x558482bf6390 2024-04-25T11:39:48.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.176+0000 7f7b13228740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558482fa9320 con 0x558482bcd4d0 2024-04-25T11:39:48.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.176+0000 7f7b0a032640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558482bd56b0 0x558482fa6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.176+0000 7f7b09831640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558482bf6390 0x558482fb26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.176+0000 7f7b13228740 10 client.0 _ll_register_callbacks cb 0x558482e9f600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:48.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.176+0000 7f7b09831640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558482bf6390 0x558482fb26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:56422/0 (socket says 172.21.15.90:56422) 2024-04-25T11:39:48.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.176+0000 7f7b09831640 1 -- 172.21.15.90:0/248102857 learned_addr learned my addr 172.21.15.90:0/248102857 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:48.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.176+0000 7f7b0a032640 1 --2- 172.21.15.90:0/248102857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558482bd56b0 0x558482fa6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49360/0 (socket says 172.21.15.90:49360) 2024-04-25T11:39:48.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.176+0000 7f7b09831640 1 --2- 172.21.15.90:0/248102857 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558482bcd4d0 0x558482fa6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.176+0000 7f7b09831640 1 -- 172.21.15.90:0/248102857 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558482bcd4d0 msgr2=0x558482fa6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:48.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.176+0000 7f7b09831640 1 --2- 172.21.15.90:0/248102857 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558482bcd4d0 0x558482fa6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.176+0000 7f7b09831640 1 -- 172.21.15.90:0/248102857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558482bd56b0 msgr2=0x558482fa6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:48.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.176+0000 7f7b09831640 1 --2- 172.21.15.90:0/248102857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558482bd56b0 0x558482fa6bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.176+0000 7f7b09831640 1 -- 172.21.15.90:0/248102857 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558482c0e3f0 con 0x558482bf6390 2024-04-25T11:39:48.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.176+0000 7f7b09831640 1 --2- 172.21.15.90:0/248102857 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558482bf6390 0x558482fb26e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7b00009970 tx=0x7f7b00005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d232ac43b0cfbe1c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:48.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.176+0000 7f7aedffb640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:39:48.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.177+0000 7f7aedffb640 1 -- 172.21.15.90:0/248102857 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b00013070 con 0x558482bf6390 2024-04-25T11:39:48.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.177+0000 7f7aedffb640 1 -- 172.21.15.90:0/248102857 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f7b00013210 con 0x558482bf6390 2024-04-25T11:39:48.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.177+0000 7f7b13228740 1 -- 172.21.15.90:0/248102857 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7af4009d50 con 0x558482bf6390 2024-04-25T11:39:48.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.177+0000 7f7aedffb640 1 -- 172.21.15.90:0/248102857 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7b00004050 con 0x558482bf6390 2024-04-25T11:39:48.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.177+0000 7f7b13228740 10 client.26171 fetch_fsmap learned FSMap version 28 2024-04-25T11:39:48.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.177+0000 7f7b13228740 1 -- 172.21.15.90:0/248102857 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558482cd87d0 con 0x558482bf6390 2024-04-25T11:39:48.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.177+0000 7f7aedffb640 1 -- 172.21.15.90:0/248102857 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7af4009d50 con 0x558482bf6390 2024-04-25T11:39:48.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.177+0000 7f7aedffb640 1 -- 172.21.15.90:0/248102857 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f7b00003ca0 con 0x558482bf6390 2024-04-25T11:39:48.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.177+0000 7f7b13228740 10 client.26171 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:39:48.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.177+0000 7f7b13228740 10 client.26171 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:39:48.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.177+0000 7f7b13228740 20 client.26171 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:48.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.177+0000 7f7b13228740 1 --2- 172.21.15.90:0/248102857 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x558482bb4f60 0x558482fb8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.177+0000 7f7b13228740 4 client.26171 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:39:48.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.177+0000 7f7b13228740 1 -- 172.21.15.90:0/248102857 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558482f0a990 con 0x558482bb4f60 2024-04-25T11:39:48.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.178+0000 7f7b0a032640 1 --2- 172.21.15.90:0/248102857 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x558482bb4f60 0x558482fb8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.178+0000 7f7aedffb640 10 client.26171 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:48.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.178+0000 7f7b0a032640 1 --2- 172.21.15.90:0/248102857 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x558482bb4f60 0x558482fb8c40 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cae73254f5996c30 server_cookie=d7ab164563dd6e4d in_seq=0 out_seq=0 2024-04-25T11:39:48.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.179+0000 7f7aedffb640 1 -- 172.21.15.90:0/248102857 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558482f0a990 con 0x558482bb4f60 2024-04-25T11:39:48.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.179+0000 7f7aedffb640 10 client.26171 handle_command_reply: tid=0 2024-04-25T11:39:48.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.179+0000 7f7b13228740 1 client.26171 shutdown 2024-04-25T11:39:48.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.179+0000 7f7b13228740 1 -- 172.21.15.90:0/248102857 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558482bf6390 msgr2=0x558482fb26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:48.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.179+0000 7f7b13228740 1 --2- 172.21.15.90:0/248102857 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558482bf6390 0x558482fb26e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7b00009970 tx=0x7f7b00005f90 comp rx=0 tx=0).stop 2024-04-25T11:39:48.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.179+0000 7f7b13228740 1 -- 172.21.15.90:0/248102857 shutdown_connections 2024-04-25T11:39:48.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.179+0000 7f7b13228740 1 --2- 172.21.15.90:0/248102857 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x558482bb4f60 0x558482fb8c40 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.179+0000 7f7b13228740 1 --2- 172.21.15.90:0/248102857 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558482bf6390 0x558482fb26e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.179+0000 7f7b13228740 1 --2- 172.21.15.90:0/248102857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558482bd56b0 0x558482fa6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.179+0000 7f7b13228740 1 --2- 172.21.15.90:0/248102857 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558482bcd4d0 0x558482fa6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.179+0000 7f7aedffb640 0 client.26171 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:48.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.179+0000 7f7b13228740 1 -- 172.21.15.90:0/248102857 >> 172.21.15.90:0/248102857 conn(0x558482bea4e0 msgr2=0x558482f99530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:48.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.179+0000 7f7b13228740 1 -- 172.21.15.90:0/248102857 shutdown_connections 2024-04-25T11:39:48.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.179+0000 7f7b13228740 1 -- 172.21.15.90:0/248102857 wait complete. 2024-04-25T11:39:48.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.179+0000 7f7b13228740 20 client.26171 trim_cache size 0 max 16384 2024-04-25T11:39:48.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.196+0000 7f7b13228740 1 Processor -- start 2024-04-25T11:39:48.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.196+0000 7f7b13228740 1 -- start start 2024-04-25T11:39:48.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.197+0000 7f7b13228740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558482bb4f60 0x558482fa1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.197+0000 7f7b13228740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558482bcd4d0 0x558482f99b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.197+0000 7f7b13228740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558482bd56b0 0x558482fa5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.197+0000 7f7b13228740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7b041ad950 con 0x558482bd56b0 2024-04-25T11:39:48.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.197+0000 7f7b13228740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7b041af490 con 0x558482bb4f60 2024-04-25T11:39:48.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.197+0000 7f7b13228740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7b041ae020 con 0x558482bcd4d0 2024-04-25T11:39:48.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.197+0000 7f7b09831640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558482bb4f60 0x558482fa1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.197+0000 7f7b09831640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558482bb4f60 0x558482fa1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:56436/0 (socket says 172.21.15.90:56436) 2024-04-25T11:39:48.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.197+0000 7f7b0a032640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558482bcd4d0 0x558482f99b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.197+0000 7f7b09831640 1 -- 172.21.15.90:0/3484332485 learned_addr learned my addr 172.21.15.90:0/3484332485 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:48.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.197+0000 7f7b09831640 1 --2- 172.21.15.90:0/3484332485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558482bd56b0 0x558482fa5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.197+0000 7f7b09831640 1 -- 172.21.15.90:0/3484332485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558482bcd4d0 msgr2=0x558482f99b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:48.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.197+0000 7f7b09831640 1 --2- 172.21.15.90:0/3484332485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558482bcd4d0 0x558482f99b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.197+0000 7f7b09831640 1 -- 172.21.15.90:0/3484332485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558482bd56b0 msgr2=0x558482fa5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:48.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.197+0000 7f7b09831640 1 --2- 172.21.15.90:0/3484332485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558482bd56b0 0x558482fa5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.198+0000 7f7b09831640 1 -- 172.21.15.90:0/3484332485 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558482fa7390 con 0x558482bb4f60 2024-04-25T11:39:48.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.198+0000 7f7b0a032640 1 --2- 172.21.15.90:0/3484332485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558482bcd4d0 0x558482f99b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:48.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.198+0000 7f7b09831640 1 --2- 172.21.15.90:0/3484332485 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558482bb4f60 0x558482fa1c80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7b000092e0 tx=0x7f7b00009850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=837f4c06a6b59fce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:48.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.198+0000 7f7aeffff640 1 -- 172.21.15.90:0/3484332485 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b000227b0 con 0x558482bb4f60 2024-04-25T11:39:48.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.198+0000 7f7aeffff640 1 -- 172.21.15.90:0/3484332485 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b00004310 con 0x558482bb4f60 2024-04-25T11:39:48.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.198+0000 7f7aeffff640 1 -- 172.21.15.90:0/3484332485 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b000054d0 con 0x558482bb4f60 2024-04-25T11:39:48.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.199+0000 7f7b13228740 1 -- 172.21.15.90:0/3484332485 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558482bb4f60 msgr2=0x558482fa1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:48.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.199+0000 7f7b13228740 1 --2- 172.21.15.90:0/3484332485 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558482bb4f60 0x558482fa1c80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7b000092e0 tx=0x7f7b00009850 comp rx=0 tx=0).stop 2024-04-25T11:39:48.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.199+0000 7f7b13228740 1 -- 172.21.15.90:0/3484332485 shutdown_connections 2024-04-25T11:39:48.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.199+0000 7f7b13228740 1 --2- 172.21.15.90:0/3484332485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558482bd56b0 0x558482fa5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.199+0000 7f7b13228740 1 --2- 172.21.15.90:0/3484332485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558482bcd4d0 0x558482f99b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.199+0000 7f7b13228740 1 --2- 172.21.15.90:0/3484332485 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558482bb4f60 0x558482fa1c80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.199+0000 7f7b13228740 1 -- 172.21.15.90:0/3484332485 >> 172.21.15.90:0/3484332485 conn(0x558482bea4e0 msgr2=0x558482f99480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:48.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.199+0000 7f7b13228740 1 -- 172.21.15.90:0/3484332485 shutdown_connections 2024-04-25T11:39:48.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.199+0000 7f7b13228740 1 -- 172.21.15.90:0/3484332485 wait complete. 2024-04-25T11:39:48.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.199+0000 7f7b13228740 1 -- start start 2024-04-25T11:39:48.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.199+0000 7f7b13228740 1 Processor -- start 2024-04-25T11:39:48.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.200+0000 7f7b13228740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558482bb4f60 0x558482fae490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.200+0000 7f7b13228740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558482bcd4d0 0x558482fae9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.200+0000 7f7b13228740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558482bd56b0 0x558482fa9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.200+0000 7f7b13228740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7b041aedc0 con 0x558482bcd4d0 2024-04-25T11:39:48.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.200+0000 7f7b13228740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7b041ae6f0 con 0x558482bd56b0 2024-04-25T11:39:48.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.200+0000 7f7b13228740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7b041ab930 con 0x558482bb4f60 2024-04-25T11:39:48.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.200+0000 7f7b13228740 10 client.0 _ll_register_callbacks cb 0x7f7b00013d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:48.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.200+0000 7f7b0a032640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558482bcd4d0 0x558482fae9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.200+0000 7f7b0a032640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558482bcd4d0 0x558482fae9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49384/0 (socket says 172.21.15.90:49384) 2024-04-25T11:39:48.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.200+0000 7f7b0a032640 1 -- 172.21.15.90:0/1350694086 learned_addr learned my addr 172.21.15.90:0/1350694086 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:48.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.200+0000 7f7b09831640 1 --2- 172.21.15.90:0/1350694086 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558482bb4f60 0x558482fae490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.200+0000 7f7b0a032640 1 -- 172.21.15.90:0/1350694086 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558482bb4f60 msgr2=0x558482fae490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:48.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.200+0000 7f7b0a032640 1 --2- 172.21.15.90:0/1350694086 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558482bb4f60 0x558482fae490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.200+0000 7f7b0a032640 1 -- 172.21.15.90:0/1350694086 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558482bd56b0 msgr2=0x558482fa9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:48.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.200+0000 7f7b0a032640 1 --2- 172.21.15.90:0/1350694086 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558482bd56b0 0x558482fa9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.200+0000 7f7b0a032640 1 -- 172.21.15.90:0/1350694086 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558482faa0b0 con 0x558482bcd4d0 2024-04-25T11:39:48.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.201+0000 7f7b0a032640 1 --2- 172.21.15.90:0/1350694086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558482bcd4d0 0x558482fae9d0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f7af4003b00 tx=0x7f7af4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f46dc496d370422 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:48.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.201+0000 7f7aedffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:48.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.201+0000 7f7aedffb640 1 -- 172.21.15.90:0/1350694086 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af401e040 con 0x558482bcd4d0 2024-04-25T11:39:48.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.201+0000 7f7b13228740 1 -- 172.21.15.90:0/1350694086 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b00004310 con 0x558482bcd4d0 2024-04-25T11:39:48.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.201+0000 7f7aedffb640 1 -- 172.21.15.90:0/1350694086 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f7af4005040 con 0x558482bcd4d0 2024-04-25T11:39:48.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.201+0000 7f7aedffb640 1 -- 172.21.15.90:0/1350694086 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7af4005c50 con 0x558482bcd4d0 2024-04-25T11:39:48.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.201+0000 7f7b13228740 10 client.17022 fetch_fsmap learned FSMap version 28 2024-04-25T11:39:48.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.201+0000 7f7b13228740 1 -- 172.21.15.90:0/1350694086 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558483020850 con 0x558482bcd4d0 2024-04-25T11:39:48.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.201+0000 7f7aedffb640 1 -- 172.21.15.90:0/1350694086 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b00004310 con 0x558482bcd4d0 2024-04-25T11:39:48.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.202+0000 7f7aedffb640 1 -- 172.21.15.90:0/1350694086 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f7af4004b70 con 0x558482bcd4d0 2024-04-25T11:39:48.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.202+0000 7f7b13228740 10 client.17022 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:39:48.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.202+0000 7f7b13228740 10 client.17022 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:39:48.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.202+0000 7f7b13228740 20 client.17022 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:48.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.202+0000 7f7b13228740 1 --2- 172.21.15.90:0/1350694086 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x558482bf6390 0x558483022e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:48.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.202+0000 7f7b13228740 4 client.17022 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:39:48.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.202+0000 7f7b13228740 1 -- 172.21.15.90:0/1350694086 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558482f04800 con 0x558482bf6390 2024-04-25T11:39:48.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.203+0000 7f7b09831640 1 --2- 172.21.15.90:0/1350694086 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x558482bf6390 0x558483022e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:48.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.204+0000 7f7aedffb640 10 client.17022 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:48.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.204+0000 7f7b09831640 1 --2- 172.21.15.90:0/1350694086 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x558482bf6390 0x558483022e70 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f3acf2b1f0386c3 server_cookie=c5373daec68b7ab9 in_seq=0 out_seq=0 2024-04-25T11:39:48.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.204+0000 7f7aedffb640 1 -- 172.21.15.90:0/1350694086 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558482f04800 con 0x558482bf6390 2024-04-25T11:39:48.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.204+0000 7f7aedffb640 10 client.17022 handle_command_reply: tid=0 2024-04-25T11:39:48.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.204+0000 7f7b13228740 1 client.17022 shutdown 2024-04-25T11:39:48.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.204+0000 7f7b13228740 1 -- 172.21.15.90:0/1350694086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558482bcd4d0 msgr2=0x558482fae9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:48.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.204+0000 7f7b13228740 1 --2- 172.21.15.90:0/1350694086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558482bcd4d0 0x558482fae9d0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f7af4003b00 tx=0x7f7af4009780 comp rx=0 tx=0).stop 2024-04-25T11:39:48.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.204+0000 7f7b13228740 1 -- 172.21.15.90:0/1350694086 shutdown_connections 2024-04-25T11:39:48.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.204+0000 7f7b13228740 1 --2- 172.21.15.90:0/1350694086 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x558482bf6390 0x558483022e70 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.204+0000 7f7b13228740 1 --2- 172.21.15.90:0/1350694086 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558482bd56b0 0x558482fa9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.204+0000 7f7b13228740 1 --2- 172.21.15.90:0/1350694086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558482bcd4d0 0x558482fae9d0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.204+0000 7f7b13228740 1 --2- 172.21.15.90:0/1350694086 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558482bb4f60 0x558482fae490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.204+0000 7f7aedffb640 0 client.17022 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:48.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.204+0000 7f7b13228740 1 -- 172.21.15.90:0/1350694086 >> 172.21.15.90:0/1350694086 conn(0x558482bea4e0 msgr2=0x558482f99c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:48.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.204+0000 7f7b13228740 1 -- 172.21.15.90:0/1350694086 shutdown_connections 2024-04-25T11:39:48.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.204+0000 7f7b13228740 1 -- 172.21.15.90:0/1350694086 wait complete. 2024-04-25T11:39:48.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.205+0000 7f7b13228740 20 client.17022 trim_cache size 0 max 16384 2024-04-25T11:39:48.196 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:39:48.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.206+0000 7f7b0c296640 1 -- 172.21.15.90:0/3262445482 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f7af00afc60 msgr2=0x7f7af00b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:48.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.206+0000 7f7b0c296640 1 --2- 172.21.15.90:0/3262445482 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f7af00afc60 0x7f7af00b2170 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7afc0fdd10 tx=0x7f7afc041b80 comp rx=0 tx=0).stop 2024-04-25T11:39:48.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.206+0000 7f7b0c296640 1 -- 172.21.15.90:0/3262445482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b0410e550 msgr2=0x7f7b0419efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:48.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.206+0000 7f7b0c296640 1 --2- 172.21.15.90:0/3262445482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b0410e550 0x7f7b0419efb0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f7afc009430 tx=0x7f7afc031660 comp rx=0 tx=0).stop 2024-04-25T11:39:48.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.206+0000 7f7b0c296640 1 -- 172.21.15.90:0/3262445482 shutdown_connections 2024-04-25T11:39:48.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.206+0000 7f7b0c296640 1 --2- 172.21.15.90:0/3262445482 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f7af00afc60 0x7f7af00b2170 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.206+0000 7f7b0c296640 1 --2- 172.21.15.90:0/3262445482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b0410e550 0x7f7b0419efb0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.206+0000 7f7b0c296640 1 --2- 172.21.15.90:0/3262445482 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7b04109ba0 0x7f7b0419acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.206+0000 7f7b0c296640 1 --2- 172.21.15.90:0/3262445482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b04102050 0x7f7b0419a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.206+0000 7f7b0c296640 1 -- 172.21.15.90:0/3262445482 >> 172.21.15.90:0/3262445482 conn(0x7f7b04077880 msgr2=0x7f7b0410a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:48.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.206+0000 7f7b0c296640 1 -- 172.21.15.90:0/3262445482 shutdown_connections 2024-04-25T11:39:48.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:48.206+0000 7f7b0c296640 1 -- 172.21.15.90:0/3262445482 wait complete. 2024-04-25T11:39:49.220 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T11:39:49.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:39:49.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T11:39:49.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.354+0000 7f4140184640 1 -- 172.21.15.90:0/1550891797 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4138102b60 msgr2=0x7f4138102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:49.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.354+0000 7f4140184640 1 --2- 172.21.15.90:0/1550891797 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4138102b60 0x7f4138102f70 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f4128009900 tx=0x7f412802a410 comp rx=0 tx=0).stop 2024-04-25T11:39:49.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.354+0000 7f4140184640 1 -- 172.21.15.90:0/1550891797 shutdown_connections 2024-04-25T11:39:49.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.354+0000 7f4140184640 1 --2- 172.21.15.90:0/1550891797 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f413810d3a0 0x7f413810f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.354+0000 7f4140184640 1 --2- 172.21.15.90:0/1550891797 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4138102b60 0x7f4138102f70 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.354+0000 7f4140184640 1 --2- 172.21.15.90:0/1550891797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4138108a40 0x7f4138102620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.354+0000 7f4140184640 1 -- 172.21.15.90:0/1550891797 >> 172.21.15.90:0/1550891797 conn(0x7f41380fe260 msgr2=0x7f4138100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:49.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.354+0000 7f4140184640 1 -- 172.21.15.90:0/1550891797 shutdown_connections 2024-04-25T11:39:49.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.354+0000 7f4140184640 1 -- 172.21.15.90:0/1550891797 wait complete. 2024-04-25T11:39:49.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.355+0000 7f4140184640 1 Processor -- start 2024-04-25T11:39:49.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.355+0000 7f4140184640 1 -- start start 2024-04-25T11:39:49.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.355+0000 7f4140184640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4138102b60 0x7f413819b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.355+0000 7f4140184640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4138108a40 0x7f413819bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.355+0000 7f4140184640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f413810d3a0 0x7f41381a6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.355+0000 7f4140184640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4138112c00 con 0x7f4138108a40 2024-04-25T11:39:49.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.355+0000 7f4140184640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4138112a80 con 0x7f4138102b60 2024-04-25T11:39:49.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.355+0000 7f4140184640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4138112d80 con 0x7f413810d3a0 2024-04-25T11:39:49.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.355+0000 7f413d71f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4138108a40 0x7f413819bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.355+0000 7f413df20640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4138102b60 0x7f413819b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.355+0000 7f413df20640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4138102b60 0x7f413819b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:56454/0 (socket says 172.21.15.90:56454) 2024-04-25T11:39:49.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.355+0000 7f413d71f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4138108a40 0x7f413819bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49388/0 (socket says 172.21.15.90:49388) 2024-04-25T11:39:49.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.355+0000 7f413df20640 1 -- 172.21.15.90:0/1396209096 learned_addr learned my addr 172.21.15.90:0/1396209096 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:49.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.355+0000 7f413e721640 1 --2- 172.21.15.90:0/1396209096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f413810d3a0 0x7f41381a6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.356+0000 7f413df20640 1 -- 172.21.15.90:0/1396209096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f413810d3a0 msgr2=0x7f41381a6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:49.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.356+0000 7f413df20640 1 --2- 172.21.15.90:0/1396209096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f413810d3a0 0x7f41381a6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.356+0000 7f413df20640 1 -- 172.21.15.90:0/1396209096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4138108a40 msgr2=0x7f413819bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:49.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.356+0000 7f413df20640 1 --2- 172.21.15.90:0/1396209096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4138108a40 0x7f413819bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.356+0000 7f413df20640 1 -- 172.21.15.90:0/1396209096 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f413819c340 con 0x7f4138102b60 2024-04-25T11:39:49.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.356+0000 7f413d71f640 1 --2- 172.21.15.90:0/1396209096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4138108a40 0x7f413819bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:49.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.356+0000 7f413df20640 1 --2- 172.21.15.90:0/1396209096 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4138102b60 0x7f413819b580 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f413400e9f0 tx=0x7f413400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:49.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.356+0000 7f412effd640 1 -- 172.21.15.90:0/1396209096 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413400ccc0 con 0x7f4138102b60 2024-04-25T11:39:49.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.356+0000 7f412effd640 1 -- 172.21.15.90:0/1396209096 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41340054d0 con 0x7f4138102b60 2024-04-25T11:39:49.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.356+0000 7f4140184640 1 -- 172.21.15.90:0/1396209096 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41381a6c00 con 0x7f4138102b60 2024-04-25T11:39:49.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.356+0000 7f412effd640 1 -- 172.21.15.90:0/1396209096 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4134010430 con 0x7f4138102b60 2024-04-25T11:39:49.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.356+0000 7f4140184640 1 -- 172.21.15.90:0/1396209096 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41381a70d0 con 0x7f4138102b60 2024-04-25T11:39:49.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.358+0000 7f414703c740 1 Processor -- start 2024-04-25T11:39:49.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.359+0000 7f414703c740 1 -- start start 2024-04-25T11:39:49.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.359+0000 7f412effd640 1 -- 172.21.15.90:0/1396209096 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f41340105d0 con 0x7f4138102b60 2024-04-25T11:39:49.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.359+0000 7f414703c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e3f2db3410 0x55e3f3153620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.359+0000 7f414703c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3f2d8a550 0x55e3f3153b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.359+0000 7f414703c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2d92730 0x55e3f31540a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.359+0000 7f414703c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e3f30b9b00 con 0x55e3f2d92730 2024-04-25T11:39:49.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.359+0000 7f414703c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55e3f3055920 con 0x55e3f2db3410 2024-04-25T11:39:49.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.359+0000 7f414703c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e3f30b5ac0 con 0x55e3f2d8a550 2024-04-25T11:39:49.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.359+0000 7f413e721640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e3f2db3410 0x55e3f3153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.359+0000 7f413d71f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2d92730 0x55e3f31540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.359+0000 7f413e721640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e3f2db3410 0x55e3f3153620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:56464/0 (socket says 172.21.15.90:56464) 2024-04-25T11:39:49.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.359+0000 7f413e721640 1 -- 172.21.15.90:0/1401645630 learned_addr learned my addr 172.21.15.90:0/1401645630 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:49.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.359+0000 7f413df20640 1 --2- 172.21.15.90:0/1401645630 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3f2d8a550 0x55e3f3153b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.360+0000 7f412effd640 1 --2- 172.21.15.90:0/1396209096 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f41240afc60 0x7f41240b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.360+0000 7f413d71f640 1 -- 172.21.15.90:0/1401645630 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3f2d8a550 msgr2=0x55e3f3153b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:49.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.360+0000 7f413d71f640 1 --2- 172.21.15.90:0/1401645630 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3f2d8a550 0x55e3f3153b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.360+0000 7f413d71f640 1 -- 172.21.15.90:0/1401645630 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e3f2db3410 msgr2=0x55e3f3153620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:49.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.360+0000 7f413d71f640 1 --2- 172.21.15.90:0/1401645630 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e3f2db3410 0x55e3f3153620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.360+0000 7f413d71f640 1 -- 172.21.15.90:0/1401645630 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3f2e937e0 con 0x55e3f2d92730 2024-04-25T11:39:49.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.360+0000 7f413df20640 1 --2- 172.21.15.90:0/1401645630 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3f2d8a550 0x55e3f3153b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:49.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.360+0000 7f413e721640 1 --2- 172.21.15.90:0/1401645630 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e3f2db3410 0x55e3f3153620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:49.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.360+0000 7f413e721640 1 --2- 172.21.15.90:0/1396209096 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f41240afc60 0x7f41240b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.360+0000 7f412effd640 1 -- 172.21.15.90:0/1396209096 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f41340df4a0 con 0x7f4138102b60 2024-04-25T11:39:49.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.360+0000 7f413d71f640 1 --2- 172.21.15.90:0/1401645630 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2d92730 0x55e3f31540a0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f412802a930 tx=0x7f4128000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5134e7ec0bfa281d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:49.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.360+0000 7f4123fff640 1 -- 172.21.15.90:0/1401645630 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4128040070 con 0x55e3f2d92730 2024-04-25T11:39:49.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.360+0000 7f4123fff640 1 -- 172.21.15.90:0/1401645630 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41280316a0 con 0x55e3f2d92730 2024-04-25T11:39:49.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.360+0000 7f4123fff640 1 -- 172.21.15.90:0/1401645630 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4128035040 con 0x55e3f2d92730 2024-04-25T11:39:49.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.361+0000 7f414703c740 1 -- 172.21.15.90:0/1401645630 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2d92730 msgr2=0x55e3f31540a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:49.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.361+0000 7f413e721640 1 --2- 172.21.15.90:0/1396209096 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f41240afc60 0x7f41240b2170 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x55e3f3150790 tx=0x7f4130004a40 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:49.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.361+0000 7f414703c740 1 --2- 172.21.15.90:0/1401645630 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2d92730 0x55e3f31540a0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f412802a930 tx=0x7f4128000bc0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.361+0000 7f414703c740 1 -- 172.21.15.90:0/1401645630 shutdown_connections 2024-04-25T11:39:49.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.361+0000 7f414703c740 1 --2- 172.21.15.90:0/1401645630 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2d92730 0x55e3f31540a0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.361+0000 7f414703c740 1 --2- 172.21.15.90:0/1401645630 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3f2d8a550 0x55e3f3153b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.361+0000 7f414703c740 1 --2- 172.21.15.90:0/1401645630 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e3f2db3410 0x55e3f3153620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.361+0000 7f414703c740 1 -- 172.21.15.90:0/1401645630 >> 172.21.15.90:0/1401645630 conn(0x55e3f2da7560 msgr2=0x55e3f314fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:49.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.361+0000 7f414703c740 1 -- 172.21.15.90:0/1401645630 shutdown_connections 2024-04-25T11:39:49.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.361+0000 7f414703c740 1 -- 172.21.15.90:0/1401645630 wait complete. 2024-04-25T11:39:49.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.361+0000 7f414703c740 1 -- start start 2024-04-25T11:39:49.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.362+0000 7f414703c740 1 Processor -- start 2024-04-25T11:39:49.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.362+0000 7f414703c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2d8a550 0x55e3f315db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.362+0000 7f414703c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3f2d92730 0x55e3f315e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.362+0000 7f414703c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e3f2db3410 0x55e3f3169dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.362+0000 7f414703c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e3f3160910 con 0x55e3f2d8a550 2024-04-25T11:39:49.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.362+0000 7f414703c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55e3f30535d0 con 0x55e3f2db3410 2024-04-25T11:39:49.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.362+0000 7f414703c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e3f3160a90 con 0x55e3f2d92730 2024-04-25T11:39:49.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.362+0000 7f414703c740 10 client.0 _ll_register_callbacks cb 0x55e3f3041080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:49.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.362+0000 7f413df20640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2d8a550 0x55e3f315db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.362+0000 7f413df20640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2d8a550 0x55e3f315db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49396/0 (socket says 172.21.15.90:49396) 2024-04-25T11:39:49.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.362+0000 7f413df20640 1 -- 172.21.15.90:0/3420070364 learned_addr learned my addr 172.21.15.90:0/3420070364 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:49.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.362+0000 7f413e721640 1 --2- 172.21.15.90:0/3420070364 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e3f2db3410 0x55e3f3169dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.362+0000 7f413d71f640 1 --2- 172.21.15.90:0/3420070364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3f2d92730 0x55e3f315e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.362+0000 7f413df20640 1 -- 172.21.15.90:0/3420070364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3f2d92730 msgr2=0x55e3f315e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:49.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.362+0000 7f413df20640 1 --2- 172.21.15.90:0/3420070364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3f2d92730 0x55e3f315e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.362+0000 7f413df20640 1 -- 172.21.15.90:0/3420070364 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e3f2db3410 msgr2=0x55e3f3169dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:49.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.362+0000 7f413df20640 1 --2- 172.21.15.90:0/3420070364 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e3f2db3410 0x55e3f3169dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.363+0000 7f413df20640 1 -- 172.21.15.90:0/3420070364 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3f2ef3f70 con 0x55e3f2d8a550 2024-04-25T11:39:49.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.363+0000 7f413e721640 1 --2- 172.21.15.90:0/3420070364 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e3f2db3410 0x55e3f3169dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:49.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.363+0000 7f413df20640 1 --2- 172.21.15.90:0/3420070364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2d8a550 0x55e3f315db80 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f41340a3600 tx=0x7f4134009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db6a8ca272c74007 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:49.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.363+0000 7f41217fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:49.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.363+0000 7f41217fa640 1 -- 172.21.15.90:0/3420070364 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41340a6cf0 con 0x55e3f2d8a550 2024-04-25T11:39:49.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.363+0000 7f41217fa640 1 -- 172.21.15.90:0/3420070364 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f41340aff50 con 0x55e3f2d8a550 2024-04-25T11:39:49.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.363+0000 7f414703c740 1 -- 172.21.15.90:0/3420070364 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41280316a0 con 0x55e3f2d8a550 2024-04-25T11:39:49.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.363+0000 7f41217fa640 1 -- 172.21.15.90:0/3420070364 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f41340dcfc0 con 0x55e3f2d8a550 2024-04-25T11:39:49.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.363+0000 7f414703c740 10 client.17034 fetch_fsmap learned FSMap version 28 2024-04-25T11:39:49.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.363+0000 7f414703c740 1 -- 172.21.15.90:0/3420070364 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e3f2d8b170 con 0x55e3f2d8a550 2024-04-25T11:39:49.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.364+0000 7f41217fa640 1 -- 172.21.15.90:0/3420070364 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41280316a0 con 0x55e3f2d8a550 2024-04-25T11:39:49.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.364+0000 7f41217fa640 1 -- 172.21.15.90:0/3420070364 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f41340a67e0 con 0x55e3f2d8a550 2024-04-25T11:39:49.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.364+0000 7f414703c740 10 client.17034 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:39:49.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.364+0000 7f414703c740 10 client.17034 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:39:49.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.364+0000 7f414703c740 20 client.17034 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:49.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.364+0000 7f414703c740 1 --2- 172.21.15.90:0/3420070364 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e3f2d721f0 0x55e3f316ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.364+0000 7f414703c740 4 client.17034 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:39:49.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.364+0000 7f414703c740 1 -- 172.21.15.90:0/3420070364 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e3f30c7710 con 0x55e3f2d721f0 2024-04-25T11:39:49.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.364+0000 7f413d71f640 1 --2- 172.21.15.90:0/3420070364 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e3f2d721f0 0x55e3f316ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.364+0000 7f41217fa640 10 client.17034 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:49.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.364+0000 7f413d71f640 1 --2- 172.21.15.90:0/3420070364 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e3f2d721f0 0x55e3f316ff50 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e8b8b96dcc551c server_cookie=499cd4b78830b5a6 in_seq=0 out_seq=0 2024-04-25T11:39:49.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.365+0000 7f41217fa640 1 -- 172.21.15.90:0/3420070364 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e3f30c7710 con 0x55e3f2d721f0 2024-04-25T11:39:49.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.365+0000 7f41217fa640 10 client.17034 handle_command_reply: tid=0 2024-04-25T11:39:49.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.365+0000 7f414703c740 1 client.17034 shutdown 2024-04-25T11:39:49.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.365+0000 7f414703c740 1 -- 172.21.15.90:0/3420070364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2d8a550 msgr2=0x55e3f315db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:49.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.365+0000 7f414703c740 1 --2- 172.21.15.90:0/3420070364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2d8a550 0x55e3f315db80 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f41340a3600 tx=0x7f4134009750 comp rx=0 tx=0).stop 2024-04-25T11:39:49.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.365+0000 7f414703c740 1 -- 172.21.15.90:0/3420070364 shutdown_connections 2024-04-25T11:39:49.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.365+0000 7f414703c740 1 --2- 172.21.15.90:0/3420070364 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e3f2d721f0 0x55e3f316ff50 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.365+0000 7f414703c740 1 --2- 172.21.15.90:0/3420070364 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e3f2db3410 0x55e3f3169dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.365+0000 7f414703c740 1 --2- 172.21.15.90:0/3420070364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3f2d92730 0x55e3f315e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.365+0000 7f414703c740 1 --2- 172.21.15.90:0/3420070364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2d8a550 0x55e3f315db80 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.365+0000 7f41217fa640 0 client.17034 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:49.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.365+0000 7f414703c740 1 -- 172.21.15.90:0/3420070364 >> 172.21.15.90:0/3420070364 conn(0x55e3f2da7560 msgr2=0x55e3f3155340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:49.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.366+0000 7f414703c740 1 -- 172.21.15.90:0/3420070364 shutdown_connections 2024-04-25T11:39:49.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.366+0000 7f414703c740 1 -- 172.21.15.90:0/3420070364 wait complete. 2024-04-25T11:39:49.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.366+0000 7f414703c740 20 client.17034 trim_cache size 0 max 16384 2024-04-25T11:39:49.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.384+0000 7f414703c740 1 Processor -- start 2024-04-25T11:39:49.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.384+0000 7f414703c740 1 -- start start 2024-04-25T11:39:49.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.384+0000 7f414703c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e3f2d8a550 0x55e3f315cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.384+0000 7f414703c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3f2d92730 0x55e3f31580e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.384+0000 7f414703c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2db3410 0x55e3f315f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.384+0000 7f414703c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f41381af9c0 con 0x55e3f2db3410 2024-04-25T11:39:49.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.384+0000 7f414703c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f41381b1500 con 0x55e3f2d8a550 2024-04-25T11:39:49.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.384+0000 7f414703c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f41381b0090 con 0x55e3f2d92730 2024-04-25T11:39:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.384+0000 7f413e721640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2db3410 0x55e3f315f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.384+0000 7f413df20640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e3f2d8a550 0x55e3f315cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.384+0000 7f413d71f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3f2d92730 0x55e3f31580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.384+0000 7f413e721640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2db3410 0x55e3f315f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49412/0 (socket says 172.21.15.90:49412) 2024-04-25T11:39:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.384+0000 7f413df20640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e3f2d8a550 0x55e3f315cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:56476/0 (socket says 172.21.15.90:56476) 2024-04-25T11:39:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.384+0000 7f413e721640 1 -- 172.21.15.90:0/2165191864 learned_addr learned my addr 172.21.15.90:0/2165191864 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.385+0000 7f413e721640 1 -- 172.21.15.90:0/2165191864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3f2d92730 msgr2=0x55e3f31580e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.385+0000 7f413e721640 1 --2- 172.21.15.90:0/2165191864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3f2d92730 0x55e3f31580e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.385+0000 7f413e721640 1 -- 172.21.15.90:0/2165191864 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e3f2d8a550 msgr2=0x55e3f315cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.385+0000 7f413e721640 1 --2- 172.21.15.90:0/2165191864 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e3f2d8a550 0x55e3f315cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.385+0000 7f413e721640 1 -- 172.21.15.90:0/2165191864 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3f314f300 con 0x55e3f2db3410 2024-04-25T11:39:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.385+0000 7f413d71f640 1 --2- 172.21.15.90:0/2165191864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3f2d92730 0x55e3f31580e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.385+0000 7f413df20640 1 --2- 172.21.15.90:0/2165191864 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e3f2d8a550 0x55e3f315cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:49.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.385+0000 7f413e721640 1 --2- 172.21.15.90:0/2165191864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2db3410 0x55e3f315f7a0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f4130020b70 tx=0x7f4130013030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e72695008f6a12e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:49.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.385+0000 7f41237fe640 1 -- 172.21.15.90:0/2165191864 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413002a630 con 0x55e3f2db3410 2024-04-25T11:39:49.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.385+0000 7f41237fe640 1 -- 172.21.15.90:0/2165191864 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f413001ee40 con 0x55e3f2db3410 2024-04-25T11:39:49.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.385+0000 7f41237fe640 1 -- 172.21.15.90:0/2165191864 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413001b880 con 0x55e3f2db3410 2024-04-25T11:39:49.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.386+0000 7f414703c740 1 -- 172.21.15.90:0/2165191864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2db3410 msgr2=0x55e3f315f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:49.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.386+0000 7f414703c740 1 --2- 172.21.15.90:0/2165191864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2db3410 0x55e3f315f7a0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f4130020b70 tx=0x7f4130013030 comp rx=0 tx=0).stop 2024-04-25T11:39:49.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.386+0000 7f414703c740 1 -- 172.21.15.90:0/2165191864 shutdown_connections 2024-04-25T11:39:49.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.386+0000 7f414703c740 1 --2- 172.21.15.90:0/2165191864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2db3410 0x55e3f315f7a0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.386+0000 7f414703c740 1 --2- 172.21.15.90:0/2165191864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3f2d92730 0x55e3f31580e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.386+0000 7f414703c740 1 --2- 172.21.15.90:0/2165191864 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e3f2d8a550 0x55e3f315cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.386+0000 7f414703c740 1 -- 172.21.15.90:0/2165191864 >> 172.21.15.90:0/2165191864 conn(0x55e3f2da7560 msgr2=0x55e3f315f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:49.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.386+0000 7f414703c740 1 -- 172.21.15.90:0/2165191864 shutdown_connections 2024-04-25T11:39:49.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.386+0000 7f414703c740 1 -- 172.21.15.90:0/2165191864 wait complete. 2024-04-25T11:39:49.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.386+0000 7f414703c740 1 -- start start 2024-04-25T11:39:49.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.386+0000 7f414703c740 1 Processor -- start 2024-04-25T11:39:49.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.387+0000 7f414703c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e3f2d8a550 0x55e3f31610d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.387+0000 7f414703c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2d92730 0x55e3f3161610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.387+0000 7f414703c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3f2db3410 0x55e3f31d62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.387+0000 7f414703c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f41381b0e30 con 0x55e3f2d92730 2024-04-25T11:39:49.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.387+0000 7f414703c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f41381b0760 con 0x55e3f2d8a550 2024-04-25T11:39:49.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.387+0000 7f414703c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f41381ad9a0 con 0x55e3f2db3410 2024-04-25T11:39:49.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.387+0000 7f414703c740 10 client.0 _ll_register_callbacks cb 0x7f41340e2ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:49.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.387+0000 7f413d71f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2d92730 0x55e3f3161610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.387+0000 7f413d71f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2d92730 0x55e3f3161610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49420/0 (socket says 172.21.15.90:49420) 2024-04-25T11:39:49.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.387+0000 7f413df20640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e3f2d8a550 0x55e3f31610d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.387+0000 7f413d71f640 1 -- 172.21.15.90:0/1844853067 learned_addr learned my addr 172.21.15.90:0/1844853067 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:49.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.387+0000 7f413e721640 1 --2- 172.21.15.90:0/1844853067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3f2db3410 0x55e3f31d62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.387+0000 7f413df20640 1 -- 172.21.15.90:0/1844853067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3f2db3410 msgr2=0x55e3f31d62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:49.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.387+0000 7f413df20640 1 --2- 172.21.15.90:0/1844853067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3f2db3410 0x55e3f31d62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.387+0000 7f413df20640 1 -- 172.21.15.90:0/1844853067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2d92730 msgr2=0x55e3f3161610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:49.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.387+0000 7f413df20640 1 --2- 172.21.15.90:0/1844853067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2d92730 0x55e3f3161610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.387+0000 7f413df20640 1 -- 172.21.15.90:0/1844853067 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3f31d6820 con 0x55e3f2d8a550 2024-04-25T11:39:49.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.387+0000 7f413d71f640 1 --2- 172.21.15.90:0/1844853067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2d92730 0x55e3f3161610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:49.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.388+0000 7f413df20640 1 --2- 172.21.15.90:0/1844853067 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e3f2d8a550 0x55e3f31610d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f413400c6a0 tx=0x7f413400ce60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4d17d29979dc4df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:49.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.388+0000 7f41217fa640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:39:49.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.388+0000 7f41217fa640 1 -- 172.21.15.90:0/1844853067 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41340a6cf0 con 0x55e3f2d8a550 2024-04-25T11:39:49.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.388+0000 7f41217fa640 1 -- 172.21.15.90:0/1844853067 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f41340a3b70 con 0x55e3f2d8a550 2024-04-25T11:39:49.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.388+0000 7f414703c740 1 -- 172.21.15.90:0/1844853067 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e3f30c1580 con 0x55e3f2d8a550 2024-04-25T11:39:49.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.388+0000 7f41217fa640 1 -- 172.21.15.90:0/1844853067 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f41340a7350 con 0x55e3f2d8a550 2024-04-25T11:39:49.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.388+0000 7f414703c740 10 client.26192 fetch_fsmap learned FSMap version 28 2024-04-25T11:39:49.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.388+0000 7f414703c740 1 -- 172.21.15.90:0/1844853067 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e3f31d7ac0 con 0x55e3f2d8a550 2024-04-25T11:39:49.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.388+0000 7f41217fa640 1 -- 172.21.15.90:0/1844853067 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55e3f30c1580 con 0x55e3f2d8a550 2024-04-25T11:39:49.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.388+0000 7f41217fa640 1 -- 172.21.15.90:0/1844853067 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f41340c83d0 con 0x55e3f2d8a550 2024-04-25T11:39:49.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.388+0000 7f414703c740 10 client.26192 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:39:49.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.388+0000 7f414703c740 10 client.26192 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:39:49.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.388+0000 7f414703c740 20 client.26192 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:49.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.388+0000 7f414703c740 1 --2- 172.21.15.90:0/1844853067 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e3f2d47630 0x55e3f31da300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.388+0000 7f414703c740 4 client.26192 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T11:39:49.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.388+0000 7f414703c740 1 -- 172.21.15.90:0/1844853067 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e3f303cac0 con 0x55e3f2d47630 2024-04-25T11:39:49.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.389+0000 7f413d71f640 1 --2- 172.21.15.90:0/1844853067 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e3f2d47630 0x55e3f31da300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.389+0000 7f41217fa640 10 client.26192 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:49.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.389+0000 7f413d71f640 1 --2- 172.21.15.90:0/1844853067 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e3f2d47630 0x55e3f31da300 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=611066eaf3696b17 server_cookie=1d16b46806e1bd8d in_seq=0 out_seq=0 2024-04-25T11:39:49.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.391+0000 7f41217fa640 1 -- 172.21.15.90:0/1844853067 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e3f303cac0 con 0x55e3f2d47630 2024-04-25T11:39:49.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.391+0000 7f41217fa640 10 client.26192 handle_command_reply: tid=0 2024-04-25T11:39:49.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.391+0000 7f414703c740 1 client.26192 shutdown 2024-04-25T11:39:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.391+0000 7f414703c740 1 -- 172.21.15.90:0/1844853067 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e3f2d8a550 msgr2=0x55e3f31610d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.391+0000 7f414703c740 1 --2- 172.21.15.90:0/1844853067 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e3f2d8a550 0x55e3f31610d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f413400c6a0 tx=0x7f413400ce60 comp rx=0 tx=0).stop 2024-04-25T11:39:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.391+0000 7f414703c740 1 -- 172.21.15.90:0/1844853067 shutdown_connections 2024-04-25T11:39:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.391+0000 7f414703c740 1 --2- 172.21.15.90:0/1844853067 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e3f2d47630 0x55e3f31da300 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.391+0000 7f414703c740 1 --2- 172.21.15.90:0/1844853067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3f2db3410 0x55e3f31d62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.391+0000 7f414703c740 1 --2- 172.21.15.90:0/1844853067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3f2d92730 0x55e3f3161610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.391+0000 7f41217fa640 0 client.26192 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.391+0000 7f414703c740 1 --2- 172.21.15.90:0/1844853067 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e3f2d8a550 0x55e3f31610d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.391+0000 7f414703c740 1 -- 172.21.15.90:0/1844853067 >> 172.21.15.90:0/1844853067 conn(0x55e3f2da7560 msgr2=0x55e3f315f950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.391+0000 7f414703c740 1 -- 172.21.15.90:0/1844853067 shutdown_connections 2024-04-25T11:39:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.391+0000 7f414703c740 1 -- 172.21.15.90:0/1844853067 wait complete. 2024-04-25T11:39:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.391+0000 7f414703c740 20 client.26192 trim_cache size 0 max 16384 2024-04-25T11:39:49.384 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:39:49.384 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-25T11:39:49.384 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "70f0b13e-d0ae-4306-b1df-a0002a7111d8", 2024-04-25T11:39:49.384 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-25T11:39:49.384 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:39:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.392+0000 7f4140184640 1 -- 172.21.15.90:0/1396209096 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f41240afc60 msgr2=0x7f41240b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.392+0000 7f4140184640 1 --2- 172.21.15.90:0/1396209096 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f41240afc60 0x7f41240b2170 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x55e3f3150790 tx=0x7f4130004a40 comp rx=0 tx=0).stop 2024-04-25T11:39:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.392+0000 7f4140184640 1 -- 172.21.15.90:0/1396209096 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4138102b60 msgr2=0x7f413819b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.392+0000 7f4140184640 1 --2- 172.21.15.90:0/1396209096 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4138102b60 0x7f413819b580 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f413400e9f0 tx=0x7f413400eeb0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.392+0000 7f4140184640 1 -- 172.21.15.90:0/1396209096 shutdown_connections 2024-04-25T11:39:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.392+0000 7f4140184640 1 --2- 172.21.15.90:0/1396209096 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f41240afc60 0x7f41240b2170 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.392+0000 7f4140184640 1 --2- 172.21.15.90:0/1396209096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f413810d3a0 0x7f41381a6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.392+0000 7f4140184640 1 --2- 172.21.15.90:0/1396209096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4138108a40 0x7f413819bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.392+0000 7f4140184640 1 --2- 172.21.15.90:0/1396209096 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4138102b60 0x7f413819b580 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.392+0000 7f4140184640 1 -- 172.21.15.90:0/1396209096 >> 172.21.15.90:0/1396209096 conn(0x7f41380fe260 msgr2=0x7f41381040d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:49.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.392+0000 7f4140184640 1 -- 172.21.15.90:0/1396209096 shutdown_connections 2024-04-25T11:39:49.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.392+0000 7f4140184640 1 -- 172.21.15.90:0/1396209096 wait complete. 2024-04-25T11:39:49.391 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:39:49.391 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:39:49.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.529+0000 7f138466e640 1 -- 172.21.15.90:0/3986830111 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f137c104bf0 msgr2=0x7f137c105000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:49.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.529+0000 7f138466e640 1 --2- 172.21.15.90:0/3986830111 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f137c104bf0 0x7f137c105000 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1378009900 tx=0x7f137802a410 comp rx=0 tx=0).stop 2024-04-25T11:39:49.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.529+0000 7f138466e640 1 -- 172.21.15.90:0/3986830111 shutdown_connections 2024-04-25T11:39:49.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.529+0000 7f138466e640 1 --2- 172.21.15.90:0/3986830111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f137c10e040 0x7f137c110530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.529+0000 7f138466e640 1 --2- 172.21.15.90:0/3986830111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f137c10b650 0x7f137c10da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.529+0000 7f138466e640 1 --2- 172.21.15.90:0/3986830111 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f137c104bf0 0x7f137c105000 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.529+0000 7f138466e640 1 -- 172.21.15.90:0/3986830111 >> 172.21.15.90:0/3986830111 conn(0x7f137c078100 msgr2=0x7f137c078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:49.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.529+0000 7f138466e640 1 -- 172.21.15.90:0/3986830111 shutdown_connections 2024-04-25T11:39:49.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.529+0000 7f138466e640 1 -- 172.21.15.90:0/3986830111 wait complete. 2024-04-25T11:39:49.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.530+0000 7f138466e640 1 Processor -- start 2024-04-25T11:39:49.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.530+0000 7f138466e640 1 -- start start 2024-04-25T11:39:49.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.530+0000 7f138466e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f137c104bf0 0x7f137c19a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.530+0000 7f138466e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f137c10b650 0x7f137c19acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.530+0000 7f138466e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f137c10e040 0x7f137c19ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.530+0000 7f138466e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f137c113ca0 con 0x7f137c10b650 2024-04-25T11:39:49.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.530+0000 7f138466e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f137c113b20 con 0x7f137c104bf0 2024-04-25T11:39:49.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.530+0000 7f138466e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f137c113e20 con 0x7f137c10e040 2024-04-25T11:39:49.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.530+0000 7f138240a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f137c104bf0 0x7f137c19a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.530+0000 7f1381c09640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f137c10b650 0x7f137c19acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.530+0000 7f138240a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f137c104bf0 0x7f137c19a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:56500/0 (socket says 172.21.15.90:56500) 2024-04-25T11:39:49.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.530+0000 7f138240a640 1 -- 172.21.15.90:0/3203511362 learned_addr learned my addr 172.21.15.90:0/3203511362 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:49.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.530+0000 7f1382c0b640 1 --2- 172.21.15.90:0/3203511362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f137c10e040 0x7f137c19ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.530+0000 7f138240a640 1 -- 172.21.15.90:0/3203511362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f137c10e040 msgr2=0x7f137c19ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:49.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.530+0000 7f138240a640 1 --2- 172.21.15.90:0/3203511362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f137c10e040 0x7f137c19ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.530+0000 7f138240a640 1 -- 172.21.15.90:0/3203511362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f137c10b650 msgr2=0x7f137c19acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:49.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.530+0000 7f138240a640 1 --2- 172.21.15.90:0/3203511362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f137c10b650 0x7f137c19acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.531+0000 7f138240a640 1 -- 172.21.15.90:0/3203511362 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f137c19f5c0 con 0x7f137c104bf0 2024-04-25T11:39:49.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.531+0000 7f138240a640 1 --2- 172.21.15.90:0/3203511362 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f137c104bf0 0x7f137c19a760 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1378005940 tx=0x7f1378004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:49.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.531+0000 7f13737fe640 1 -- 172.21.15.90:0/3203511362 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137803f030 con 0x7f137c104bf0 2024-04-25T11:39:49.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.531+0000 7f13737fe640 1 -- 172.21.15.90:0/3203511362 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1378003aa0 con 0x7f137c104bf0 2024-04-25T11:39:49.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.531+0000 7f138466e640 1 -- 172.21.15.90:0/3203511362 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f137c19f850 con 0x7f137c104bf0 2024-04-25T11:39:49.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.531+0000 7f13737fe640 1 -- 172.21.15.90:0/3203511362 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1378035070 con 0x7f137c104bf0 2024-04-25T11:39:49.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.531+0000 7f138466e640 1 -- 172.21.15.90:0/3203511362 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f137c19fd90 con 0x7f137c104bf0 2024-04-25T11:39:49.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.534+0000 7f138b601740 1 Processor -- start 2024-04-25T11:39:49.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.534+0000 7f138b601740 1 -- start start 2024-04-25T11:39:49.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.534+0000 7f13737fe640 1 -- 172.21.15.90:0/3203511362 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f1378042030 con 0x7f137c104bf0 2024-04-25T11:39:49.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.534+0000 7f138b601740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55637d671390 0x55637da11a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.534+0000 7f138b601740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55637d6484d0 0x55637da11fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.534+0000 7f138b601740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55637d6506b0 0x55637da1e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.534+0000 7f138b601740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55637d977d80 con 0x55637d6506b0 2024-04-25T11:39:49.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.534+0000 7f138b601740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55637d9318b0 con 0x55637d671390 2024-04-25T11:39:49.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.534+0000 7f138b601740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55637d973d40 con 0x55637d6484d0 2024-04-25T11:39:49.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.534+0000 7f1381c09640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55637d6506b0 0x55637da1e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.534+0000 7f1382c0b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55637d671390 0x55637da11a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.534+0000 7f1382c0b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55637d671390 0x55637da11a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:56512/0 (socket says 172.21.15.90:56512) 2024-04-25T11:39:49.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.534+0000 7f1381c09640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55637d6506b0 0x55637da1e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49442/0 (socket says 172.21.15.90:49442) 2024-04-25T11:39:49.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.534+0000 7f1382c0b640 1 -- 172.21.15.90:0/3995998425 learned_addr learned my addr 172.21.15.90:0/3995998425 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:49.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.534+0000 7f138240a640 1 --2- 172.21.15.90:0/3995998425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55637d6484d0 0x55637da11fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.535+0000 7f1382c0b640 1 -- 172.21.15.90:0/3995998425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55637d6484d0 msgr2=0x55637da11fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:49.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.535+0000 7f1382c0b640 1 --2- 172.21.15.90:0/3995998425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55637d6484d0 0x55637da11fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.535+0000 7f1382c0b640 1 -- 172.21.15.90:0/3995998425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55637d6506b0 msgr2=0x55637da1e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:49.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.535+0000 7f1382c0b640 1 --2- 172.21.15.90:0/3995998425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55637d6506b0 0x55637da1e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.535+0000 7f1382c0b640 1 -- 172.21.15.90:0/3995998425 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55637d902a40 con 0x55637d671390 2024-04-25T11:39:49.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.535+0000 7f138240a640 1 --2- 172.21.15.90:0/3995998425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55637d6484d0 0x55637da11fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:39:49.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.535+0000 7f13737fe640 1 --2- 172.21.15.90:0/3203511362 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f13680afce0 0x7f13680b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.535+0000 7f1382c0b640 1 --2- 172.21.15.90:0/3995998425 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55637d671390 0x55637da11a60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1374009f60 tx=0x7f1374009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11380717a5d3669b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:49.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.535+0000 7f138240a640 1 --2- 172.21.15.90:0/3203511362 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f13680afce0 0x7f13680b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.535+0000 7f13737fe640 1 -- 172.21.15.90:0/3203511362 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f13780ff690 con 0x7f137c104bf0 2024-04-25T11:39:49.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.535+0000 7f1370bfe640 1 -- 172.21.15.90:0/3995998425 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137401e070 con 0x55637d671390 2024-04-25T11:39:49.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.535+0000 7f1370bfe640 1 -- 172.21.15.90:0/3995998425 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1374003c30 con 0x55637d671390 2024-04-25T11:39:49.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.535+0000 7f1370bfe640 1 -- 172.21.15.90:0/3995998425 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137400f8c0 con 0x55637d671390 2024-04-25T11:39:49.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.536+0000 7f138b601740 1 -- 172.21.15.90:0/3995998425 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55637d671390 msgr2=0x55637da11a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:49.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.536+0000 7f138240a640 1 --2- 172.21.15.90:0/3203511362 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f13680afce0 0x7f13680b21f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f13780fce30 tx=0x7f13780b0040 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:49.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.536+0000 7f138b601740 1 --2- 172.21.15.90:0/3995998425 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55637d671390 0x55637da11a60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1374009f60 tx=0x7f1374009f90 comp rx=0 tx=0).stop 2024-04-25T11:39:49.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.536+0000 7f138b601740 1 -- 172.21.15.90:0/3995998425 shutdown_connections 2024-04-25T11:39:49.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.536+0000 7f138b601740 1 --2- 172.21.15.90:0/3995998425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55637d6506b0 0x55637da1e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.536+0000 7f138b601740 1 --2- 172.21.15.90:0/3995998425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55637d6484d0 0x55637da11fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.536+0000 7f138b601740 1 --2- 172.21.15.90:0/3995998425 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55637d671390 0x55637da11a60 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.536+0000 7f138b601740 1 -- 172.21.15.90:0/3995998425 >> 172.21.15.90:0/3995998425 conn(0x55637d6654e0 msgr2=0x55637da0e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:49.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.536+0000 7f138b601740 1 -- 172.21.15.90:0/3995998425 shutdown_connections 2024-04-25T11:39:49.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.536+0000 7f138b601740 1 -- 172.21.15.90:0/3995998425 wait complete. 2024-04-25T11:39:49.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.537+0000 7f138b601740 1 -- start start 2024-04-25T11:39:49.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.537+0000 7f138b601740 1 Processor -- start 2024-04-25T11:39:49.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.537+0000 7f138b601740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55637d6484d0 0x55637da1c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.537+0000 7f138b601740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55637d6506b0 0x55637da1cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.537+0000 7f138b601740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55637d671390 0x55637da224a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.537+0000 7f138b601740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55637da1efe0 con 0x55637d6484d0 2024-04-25T11:39:49.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.537+0000 7f138b601740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55637d8fed10 con 0x55637d671390 2024-04-25T11:39:49.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.537+0000 7f138b601740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55637da1f160 con 0x55637d6506b0 2024-04-25T11:39:49.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.537+0000 7f138b601740 10 client.0 _ll_register_callbacks cb 0x55637d92b210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:49.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.537+0000 7f1382c0b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55637d6484d0 0x55637da1c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.537+0000 7f1382c0b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55637d6484d0 0x55637da1c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49454/0 (socket says 172.21.15.90:49454) 2024-04-25T11:39:49.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.537+0000 7f1382c0b640 1 -- 172.21.15.90:0/2289976669 learned_addr learned my addr 172.21.15.90:0/2289976669 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:49.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.537+0000 7f1382c0b640 1 --2- 172.21.15.90:0/2289976669 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55637d671390 0x55637da224a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.537+0000 7f1381c09640 1 --2- 172.21.15.90:0/2289976669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55637d6506b0 0x55637da1cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.538+0000 7f1382c0b640 1 -- 172.21.15.90:0/2289976669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55637d6506b0 msgr2=0x55637da1cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:49.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.538+0000 7f1382c0b640 1 --2- 172.21.15.90:0/2289976669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55637d6506b0 0x55637da1cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.538+0000 7f1382c0b640 1 -- 172.21.15.90:0/2289976669 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55637d671390 msgr2=0x55637da224a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:49.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.538+0000 7f1382c0b640 1 --2- 172.21.15.90:0/2289976669 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55637d671390 0x55637da224a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.538+0000 7f1382c0b640 1 -- 172.21.15.90:0/2289976669 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55637d7537d0 con 0x55637d6484d0 2024-04-25T11:39:49.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.538+0000 7f1382c0b640 1 --2- 172.21.15.90:0/2289976669 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55637d6484d0 0x55637da1c640 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f13740048f0 tx=0x7f137400f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22719ec41916d749 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:49.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.538+0000 7f1365ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:49.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.538+0000 7f1365ffb640 1 -- 172.21.15.90:0/2289976669 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137402e060 con 0x55637d6484d0 2024-04-25T11:39:49.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.538+0000 7f138b601740 1 -- 172.21.15.90:0/2289976669 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1374003c30 con 0x55637d6484d0 2024-04-25T11:39:49.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.538+0000 7f1365ffb640 1 -- 172.21.15.90:0/2289976669 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f1374004540 con 0x55637d6484d0 2024-04-25T11:39:49.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.538+0000 7f1365ffb640 1 -- 172.21.15.90:0/2289976669 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f137401e070 con 0x55637d6484d0 2024-04-25T11:39:49.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.538+0000 7f138b601740 10 client.17052 fetch_fsmap learned FSMap version 28 2024-04-25T11:39:49.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.538+0000 7f138b601740 1 -- 172.21.15.90:0/2289976669 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55637d6893f0 con 0x55637d6484d0 2024-04-25T11:39:49.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.538+0000 7f1365ffb640 1 -- 172.21.15.90:0/2289976669 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1374003c30 con 0x55637d6484d0 2024-04-25T11:39:49.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.538+0000 7f1365ffb640 1 -- 172.21.15.90:0/2289976669 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f1374025070 con 0x55637d6484d0 2024-04-25T11:39:49.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.539+0000 7f138b601740 10 client.17052 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:39:49.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.539+0000 7f138b601740 10 client.17052 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:39:49.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.539+0000 7f138b601740 20 client.17052 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:49.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.539+0000 7f138b601740 1 --2- 172.21.15.90:0/2289976669 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55637d62ff60 0x55637da2e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.539+0000 7f138b601740 4 client.17052 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:39:49.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.539+0000 7f138b601740 1 -- 172.21.15.90:0/2289976669 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55637d985990 con 0x55637d62ff60 2024-04-25T11:39:49.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.539+0000 7f1381c09640 1 --2- 172.21.15.90:0/2289976669 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55637d62ff60 0x55637da2e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.539+0000 7f1365ffb640 10 client.17052 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:49.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.539+0000 7f1381c09640 1 --2- 172.21.15.90:0/2289976669 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55637d62ff60 0x55637da2e5f0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf12ba71f0d59fd9 server_cookie=a623679ba4433d0e in_seq=0 out_seq=0 2024-04-25T11:39:49.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.540+0000 7f1365ffb640 1 -- 172.21.15.90:0/2289976669 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55637d985990 con 0x55637d62ff60 2024-04-25T11:39:49.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.540+0000 7f1365ffb640 10 client.17052 handle_command_reply: tid=0 2024-04-25T11:39:49.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.540+0000 7f138b601740 1 client.17052 shutdown 2024-04-25T11:39:49.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.540+0000 7f138b601740 1 -- 172.21.15.90:0/2289976669 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55637d6484d0 msgr2=0x55637da1c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:49.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.540+0000 7f138b601740 1 --2- 172.21.15.90:0/2289976669 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55637d6484d0 0x55637da1c640 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f13740048f0 tx=0x7f137400f520 comp rx=0 tx=0).stop 2024-04-25T11:39:49.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.540+0000 7f138b601740 1 -- 172.21.15.90:0/2289976669 shutdown_connections 2024-04-25T11:39:49.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.540+0000 7f138b601740 1 --2- 172.21.15.90:0/2289976669 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55637d62ff60 0x55637da2e5f0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.540+0000 7f138b601740 1 --2- 172.21.15.90:0/2289976669 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55637d671390 0x55637da224a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.540+0000 7f138b601740 1 --2- 172.21.15.90:0/2289976669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55637d6506b0 0x55637da1cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.540+0000 7f138b601740 1 --2- 172.21.15.90:0/2289976669 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55637d6484d0 0x55637da1c640 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.540+0000 7f1365ffb640 0 client.17052 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:49.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.540+0000 7f138b601740 1 -- 172.21.15.90:0/2289976669 >> 172.21.15.90:0/2289976669 conn(0x55637d6654e0 msgr2=0x55637da13780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:49.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.540+0000 7f138b601740 1 -- 172.21.15.90:0/2289976669 shutdown_connections 2024-04-25T11:39:49.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.540+0000 7f138b601740 1 -- 172.21.15.90:0/2289976669 wait complete. 2024-04-25T11:39:49.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.540+0000 7f138b601740 20 client.17052 trim_cache size 0 max 16384 2024-04-25T11:39:49.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.559+0000 7f138b601740 1 Processor -- start 2024-04-25T11:39:49.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.559+0000 7f138b601740 1 -- start start 2024-04-25T11:39:49.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.560+0000 7f138b601740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55637d6484d0 0x55637da1b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.560+0000 7f138b601740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55637d6506b0 0x55637da11e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.560+0000 7f138b601740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55637d671390 0x55637da22420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.560+0000 7f138b601740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f137c1afbb0 con 0x55637d671390 2024-04-25T11:39:49.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.560+0000 7f138b601740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f137c1b16f0 con 0x55637d6506b0 2024-04-25T11:39:49.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.560+0000 7f138b601740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f137c1b0280 con 0x55637d6484d0 2024-04-25T11:39:49.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.560+0000 7f1381c09640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55637d6506b0 0x55637da11e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.560+0000 7f1381c09640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55637d6506b0 0x55637da11e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:56524/0 (socket says 172.21.15.90:56524) 2024-04-25T11:39:49.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.560+0000 7f1381c09640 1 -- 172.21.15.90:0/3290026539 learned_addr learned my addr 172.21.15.90:0/3290026539 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:49.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.560+0000 7f1382c0b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55637d671390 0x55637da22420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.560+0000 7f1382c0b640 1 --2- 172.21.15.90:0/3290026539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55637d6484d0 0x55637da1b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.560+0000 7f1381c09640 1 -- 172.21.15.90:0/3290026539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55637d6484d0 msgr2=0x55637da1b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:49.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.560+0000 7f1381c09640 1 --2- 172.21.15.90:0/3290026539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55637d6484d0 0x55637da1b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.560+0000 7f1381c09640 1 -- 172.21.15.90:0/3290026539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55637d671390 msgr2=0x55637da22420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:49.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.560+0000 7f1381c09640 1 --2- 172.21.15.90:0/3290026539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55637d671390 0x55637da22420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.560+0000 7f1381c09640 1 -- 172.21.15.90:0/3290026539 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55637d6490f0 con 0x55637d6506b0 2024-04-25T11:39:49.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.560+0000 7f1381c09640 1 --2- 172.21.15.90:0/3290026539 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55637d6506b0 0x55637da11e80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f136c00b780 tx=0x7f136c00b390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98c329200679fa6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:49.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.561+0000 7f1367fff640 1 -- 172.21.15.90:0/3290026539 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f136c003bc0 con 0x55637d6506b0 2024-04-25T11:39:49.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.561+0000 7f1367fff640 1 -- 172.21.15.90:0/3290026539 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f136c023020 con 0x55637d6506b0 2024-04-25T11:39:49.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.561+0000 7f1367fff640 1 -- 172.21.15.90:0/3290026539 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f136c0116a0 con 0x55637d6506b0 2024-04-25T11:39:49.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.561+0000 7f138b601740 1 -- 172.21.15.90:0/3290026539 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55637d6506b0 msgr2=0x55637da11e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:49.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.561+0000 7f138b601740 1 --2- 172.21.15.90:0/3290026539 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55637d6506b0 0x55637da11e80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f136c00b780 tx=0x7f136c00b390 comp rx=0 tx=0).stop 2024-04-25T11:39:49.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.561+0000 7f138b601740 1 -- 172.21.15.90:0/3290026539 shutdown_connections 2024-04-25T11:39:49.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.562+0000 7f138b601740 1 --2- 172.21.15.90:0/3290026539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55637d671390 0x55637da22420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.562+0000 7f138b601740 1 --2- 172.21.15.90:0/3290026539 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55637d6506b0 0x55637da11e80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.562+0000 7f138b601740 1 --2- 172.21.15.90:0/3290026539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55637d6484d0 0x55637da1b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.562+0000 7f138b601740 1 -- 172.21.15.90:0/3290026539 >> 172.21.15.90:0/3290026539 conn(0x55637d6654e0 msgr2=0x55637da13cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:49.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.562+0000 7f138b601740 1 -- 172.21.15.90:0/3290026539 shutdown_connections 2024-04-25T11:39:49.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.562+0000 7f138b601740 1 -- 172.21.15.90:0/3290026539 wait complete. 2024-04-25T11:39:49.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.562+0000 7f138b601740 1 -- start start 2024-04-25T11:39:49.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.562+0000 7f138b601740 1 Processor -- start 2024-04-25T11:39:49.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.563+0000 7f138b601740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55637d6484d0 0x55637da234e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.563+0000 7f138b601740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55637d6506b0 0x55637da23a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.563+0000 7f138b601740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55637d671390 0x55637da23f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.563+0000 7f138b601740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f137c1b1020 con 0x55637d6484d0 2024-04-25T11:39:49.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.563+0000 7f138b601740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f137c1b0950 con 0x55637d6506b0 2024-04-25T11:39:49.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.563+0000 7f138b601740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f137c1adb90 con 0x55637d671390 2024-04-25T11:39:49.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.563+0000 7f138b601740 10 client.0 _ll_register_callbacks cb 0x7f137401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:49.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.563+0000 7f1382c0b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55637d6484d0 0x55637da234e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.563+0000 7f1382c0b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55637d6484d0 0x55637da234e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49470/0 (socket says 172.21.15.90:49470) 2024-04-25T11:39:49.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.563+0000 7f1382c0b640 1 -- 172.21.15.90:0/3216228541 learned_addr learned my addr 172.21.15.90:0/3216228541 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:49.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.563+0000 7f1381c09640 1 --2- 172.21.15.90:0/3216228541 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55637d6506b0 0x55637da23a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.563+0000 7f1382c0b640 1 --2- 172.21.15.90:0/3216228541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55637d671390 0x55637da23f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.563+0000 7f1382c0b640 1 -- 172.21.15.90:0/3216228541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55637d671390 msgr2=0x55637da23f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:49.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.563+0000 7f1382c0b640 1 --2- 172.21.15.90:0/3216228541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55637d671390 0x55637da23f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.563+0000 7f1382c0b640 1 -- 172.21.15.90:0/3216228541 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55637d6506b0 msgr2=0x55637da23a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:49.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.563+0000 7f1382c0b640 1 --2- 172.21.15.90:0/3216228541 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55637d6506b0 0x55637da23a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.563+0000 7f1382c0b640 1 -- 172.21.15.90:0/3216228541 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55637d9384b0 con 0x55637d6484d0 2024-04-25T11:39:49.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.563+0000 7f1381c09640 1 --2- 172.21.15.90:0/3216228541 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55637d6506b0 0x55637da23a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:49.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.563+0000 7f1382c0b640 1 --2- 172.21.15.90:0/3216228541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55637d6484d0 0x55637da234e0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f1374004920 tx=0x7f1374003ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca3a2438b55ac1c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:49.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.563+0000 7f1365ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:49.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.563+0000 7f1365ffb640 1 -- 172.21.15.90:0/3216228541 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137402e070 con 0x55637d6484d0 2024-04-25T11:39:49.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.563+0000 7f1365ffb640 1 -- 172.21.15.90:0/3216228541 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f1374004340 con 0x55637d6484d0 2024-04-25T11:39:49.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.563+0000 7f138b601740 1 -- 172.21.15.90:0/3216228541 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f136c023020 con 0x55637d6484d0 2024-04-25T11:39:49.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.564+0000 7f1365ffb640 1 -- 172.21.15.90:0/3216228541 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f137400f810 con 0x55637d6484d0 2024-04-25T11:39:49.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.564+0000 7f138b601740 10 client.17058 fetch_fsmap learned FSMap version 28 2024-04-25T11:39:49.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.564+0000 7f138b601740 1 -- 172.21.15.90:0/3216228541 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55637da957c0 con 0x55637d6484d0 2024-04-25T11:39:49.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.564+0000 7f1365ffb640 1 -- 172.21.15.90:0/3216228541 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f136c023020 con 0x55637d6484d0 2024-04-25T11:39:49.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.564+0000 7f1365ffb640 1 -- 172.21.15.90:0/3216228541 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f1374025040 con 0x55637d6484d0 2024-04-25T11:39:49.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.564+0000 7f138b601740 10 client.17058 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:39:49.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.564+0000 7f138b601740 10 client.17058 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:39:49.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.564+0000 7f138b601740 20 client.17058 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:49.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.564+0000 7f138b601740 1 --2- 172.21.15.90:0/3216228541 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55637d6053a0 0x55637da97e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:49.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.564+0000 7f138b601740 4 client.17058 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:39:49.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.564+0000 7f138b601740 1 -- 172.21.15.90:0/3216228541 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55637d97f800 con 0x55637d6053a0 2024-04-25T11:39:49.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.564+0000 7f1381c09640 1 --2- 172.21.15.90:0/3216228541 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55637d6053a0 0x55637da97e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:49.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.565+0000 7f1365ffb640 10 client.17058 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:49.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.565+0000 7f1381c09640 1 --2- 172.21.15.90:0/3216228541 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55637d6053a0 0x55637da97e60 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d01e7161bcd510a6 server_cookie=be2f74ddc2883272 in_seq=0 out_seq=0 2024-04-25T11:39:49.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.565+0000 7f1365ffb640 1 -- 172.21.15.90:0/3216228541 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55637d97f800 con 0x55637d6053a0 2024-04-25T11:39:49.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.565+0000 7f1365ffb640 10 client.17058 handle_command_reply: tid=0 2024-04-25T11:39:49.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.565+0000 7f138b601740 1 client.17058 shutdown 2024-04-25T11:39:49.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.565+0000 7f138b601740 1 -- 172.21.15.90:0/3216228541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55637d6484d0 msgr2=0x55637da234e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:49.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.565+0000 7f138b601740 1 --2- 172.21.15.90:0/3216228541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55637d6484d0 0x55637da234e0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f1374004920 tx=0x7f1374003ab0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.565+0000 7f138b601740 1 -- 172.21.15.90:0/3216228541 shutdown_connections 2024-04-25T11:39:49.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.565+0000 7f138b601740 1 --2- 172.21.15.90:0/3216228541 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55637d6053a0 0x55637da97e60 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.565+0000 7f138b601740 1 --2- 172.21.15.90:0/3216228541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55637d671390 0x55637da23f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.565+0000 7f138b601740 1 --2- 172.21.15.90:0/3216228541 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55637d6506b0 0x55637da23a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.565+0000 7f1365ffb640 0 client.17058 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:49.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.565+0000 7f138b601740 1 --2- 172.21.15.90:0/3216228541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55637d6484d0 0x55637da234e0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.565+0000 7f138b601740 1 -- 172.21.15.90:0/3216228541 >> 172.21.15.90:0/3216228541 conn(0x55637d6654e0 msgr2=0x55637da1d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:49.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.565+0000 7f138b601740 1 -- 172.21.15.90:0/3216228541 shutdown_connections 2024-04-25T11:39:49.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.565+0000 7f138b601740 1 -- 172.21.15.90:0/3216228541 wait complete. 2024-04-25T11:39:49.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.565+0000 7f138b601740 20 client.17058 trim_cache size 0 max 16384 2024-04-25T11:39:49.559 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:39:49.559 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "scrub active (0 inodes in the stack)", 2024-04-25T11:39:49.559 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-25T11:39:49.559 INFO:teuthology.orchestra.run.smithi090.stdout: "70f0b13e-d0ae-4306-b1df-a0002a7111d8": { 2024-04-25T11:39:49.559 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-25T11:39:49.560 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "70f0b13e-d0ae-4306-b1df-a0002a7111d8", 2024-04-25T11:39:49.560 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-25T11:39:49.560 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:39:49.560 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:39:49.560 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:39:49.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.566+0000 7f138466e640 1 -- 172.21.15.90:0/3203511362 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f13680afce0 msgr2=0x7f13680b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:49.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.566+0000 7f138466e640 1 --2- 172.21.15.90:0/3203511362 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f13680afce0 0x7f13680b21f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f13780fce30 tx=0x7f13780b0040 comp rx=0 tx=0).stop 2024-04-25T11:39:49.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.566+0000 7f138466e640 1 -- 172.21.15.90:0/3203511362 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f137c104bf0 msgr2=0x7f137c19a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:49.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.566+0000 7f138466e640 1 --2- 172.21.15.90:0/3203511362 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f137c104bf0 0x7f137c19a760 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1378005940 tx=0x7f1378004270 comp rx=0 tx=0).stop 2024-04-25T11:39:49.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.566+0000 7f138466e640 1 -- 172.21.15.90:0/3203511362 shutdown_connections 2024-04-25T11:39:49.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.566+0000 7f138466e640 1 --2- 172.21.15.90:0/3203511362 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f13680afce0 0x7f13680b21f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.566+0000 7f138466e640 1 --2- 172.21.15.90:0/3203511362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f137c10e040 0x7f137c19ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.566+0000 7f138466e640 1 --2- 172.21.15.90:0/3203511362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f137c10b650 0x7f137c19acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.566+0000 7f138466e640 1 --2- 172.21.15.90:0/3203511362 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f137c104bf0 0x7f137c19a760 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:49.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.566+0000 7f138466e640 1 -- 172.21.15.90:0/3203511362 >> 172.21.15.90:0/3203511362 conn(0x7f137c078100 msgr2=0x7f137c101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:49.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.566+0000 7f138466e640 1 -- 172.21.15.90:0/3203511362 shutdown_connections 2024-04-25T11:39:49.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:39:49.566+0000 7f138466e640 1 -- 172.21.15.90:0/3203511362 wait complete. 2024-04-25T11:39:49.564 INFO:tasks.cephfs.filesystem:scrub status for tag:70f0b13e-d0ae-4306-b1df-a0002a7111d8 - {'path': '/', 'tag': '70f0b13e-d0ae-4306-b1df-a0002a7111d8', 'options': 'recursive,force'} 2024-04-25T11:39:49.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:49 smithi090 ceph-mon[29575]: pgmap v145: 97 pgs: 97 active+clean; 40 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 408 MiB/s wr, 956 op/s 2024-04-25T11:39:49.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:49 smithi090 ceph-mon[29575]: scrub queued for path: / 2024-04-25T11:39:49.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:49 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:39:49.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:49 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:39:50.053 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:49 smithi138 ceph-mon[32328]: pgmap v145: 97 pgs: 97 active+clean; 40 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 408 MiB/s wr, 956 op/s 2024-04-25T11:39:50.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:49 smithi138 ceph-mon[32328]: scrub queued for path: / 2024-04-25T11:39:50.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:49 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:39:50.054 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:49 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:39:50.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:49 smithi156 ceph-mon[33045]: pgmap v145: 97 pgs: 97 active+clean; 40 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 408 MiB/s wr, 956 op/s 2024-04-25T11:39:50.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:49 smithi156 ceph-mon[33045]: scrub queued for path: / 2024-04-25T11:39:50.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:49 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:39:50.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:49 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:39:51.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:50 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:39:51.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:50 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:39:51.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:50 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:39:51.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:50 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:39:51.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:50 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:39:51.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:50 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:39:52.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:51 smithi156 ceph-mon[33045]: pgmap v146: 97 pgs: 97 active+clean; 40 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 547 MiB/s wr, 1.04k op/s 2024-04-25T11:39:52.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:51 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:39:52.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:51 smithi090 ceph-mon[29575]: pgmap v146: 97 pgs: 97 active+clean; 40 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 547 MiB/s wr, 1.04k op/s 2024-04-25T11:39:52.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:51 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:39:52.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:51 smithi138 ceph-mon[32328]: pgmap v146: 97 pgs: 97 active+clean; 40 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 547 MiB/s wr, 1.04k op/s 2024-04-25T11:39:52.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:51 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:39:54.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:53 smithi090 ceph-mon[29575]: pgmap v147: 97 pgs: 97 active+clean; 40 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 1.0 MiB/s rd, 304 MiB/s wr, 276 op/s 2024-04-25T11:39:54.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:53 smithi138 ceph-mon[32328]: pgmap v147: 97 pgs: 97 active+clean; 40 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 1.0 MiB/s rd, 304 MiB/s wr, 276 op/s 2024-04-25T11:39:54.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:53 smithi156 ceph-mon[33045]: pgmap v147: 97 pgs: 97 active+clean; 40 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 1.0 MiB/s rd, 304 MiB/s wr, 276 op/s 2024-04-25T11:39:56.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:56 smithi156 ceph-mon[33045]: pgmap v148: 97 pgs: 97 active+clean; 40 GiB data, 121 GiB used, 951 GiB / 1.0 TiB avail; 1.0 MiB/s rd, 380 MiB/s wr, 399 op/s 2024-04-25T11:39:56.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:56 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:39:56.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:56 smithi090 ceph-mon[29575]: pgmap v148: 97 pgs: 97 active+clean; 40 GiB data, 121 GiB used, 951 GiB / 1.0 TiB avail; 1.0 MiB/s rd, 380 MiB/s wr, 399 op/s 2024-04-25T11:39:56.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:56 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:39:56.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:56 smithi138 ceph-mon[32328]: pgmap v148: 97 pgs: 97 active+clean; 40 GiB data, 121 GiB used, 951 GiB / 1.0 TiB avail; 1.0 MiB/s rd, 380 MiB/s wr, 399 op/s 2024-04-25T11:39:56.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:56 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:39:57.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:57 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:39:57.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:57 smithi090 ceph-mon[29575]: pgmap v149: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 499 MiB/s wr, 639 op/s 2024-04-25T11:39:57.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:57 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:39:57.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:57 smithi138 ceph-mon[32328]: pgmap v149: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 499 MiB/s wr, 639 op/s 2024-04-25T11:39:57.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:57 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:39:57.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:57 smithi156 ceph-mon[33045]: pgmap v149: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 499 MiB/s wr, 639 op/s 2024-04-25T11:39:59.723 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.736+0000 7f265703c640 1 -- 172.21.15.90:0/2326779462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2650108a60 msgr2=0x7f2650102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:59.723 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.736+0000 7f265703c640 1 --2- 172.21.15.90:0/2326779462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2650108a60 0x7f2650102610 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f2644009900 tx=0x7f264402a410 comp rx=0 tx=0).stop 2024-04-25T11:39:59.723 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.736+0000 7f265703c640 1 -- 172.21.15.90:0/2326779462 shutdown_connections 2024-04-25T11:39:59.723 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.736+0000 7f265703c640 1 --2- 172.21.15.90:0/2326779462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26501031c0 0x7f265010f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:59.723 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.736+0000 7f265703c640 1 --2- 172.21.15.90:0/2326779462 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2650108f00 0x7f2650102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:59.723 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.736+0000 7f265703c640 1 --2- 172.21.15.90:0/2326779462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2650108a60 0x7f2650102610 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:59.724 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.736+0000 7f265703c640 1 -- 172.21.15.90:0/2326779462 >> 172.21.15.90:0/2326779462 conn(0x7f26500781b0 msgr2=0x7f26500785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:59.724 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.736+0000 7f265703c640 1 -- 172.21.15.90:0/2326779462 shutdown_connections 2024-04-25T11:39:59.724 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.736+0000 7f265703c640 1 -- 172.21.15.90:0/2326779462 wait complete. 2024-04-25T11:39:59.724 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.737+0000 7f265703c640 1 Processor -- start 2024-04-25T11:39:59.724 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.737+0000 7f265703c640 1 -- start start 2024-04-25T11:39:59.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:39:59 smithi090 ceph-mon[29575]: pgmap v150: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 393 MiB/s wr, 520 op/s 2024-04-25T11:39:59.726 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.737+0000 7f265703c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26501031c0 0x7f265019b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:59.726 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.737+0000 7f265703c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2650108a60 0x7f265019bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:59.726 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.737+0000 7f265703c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2650108f00 0x7f265019c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:59.726 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.737+0000 7f265703c640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2650112c60 con 0x7f2650108f00 2024-04-25T11:39:59.726 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.737+0000 7f265703c640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2650112ae0 con 0x7f2650108a60 2024-04-25T11:39:59.726 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.737+0000 7f265703c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2650112de0 con 0x7f26501031c0 2024-04-25T11:39:59.726 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.737+0000 7f26555d9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2650108f00 0x7f265019c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:59.726 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.737+0000 7f26555d9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2650108f00 0x7f265019c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55194/0 (socket says 172.21.15.90:55194) 2024-04-25T11:39:59.726 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.737+0000 7f26555d9640 1 -- 172.21.15.90:0/1181299465 learned_addr learned my addr 172.21.15.90:0/1181299465 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:59.726 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.737+0000 7f264ffff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2650108a60 0x7f265019bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:59.726 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.737+0000 7f2654dd8640 1 --2- 172.21.15.90:0/1181299465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26501031c0 0x7f265019b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:59.726 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.738+0000 7f264ffff640 1 -- 172.21.15.90:0/1181299465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26501031c0 msgr2=0x7f265019b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:59.726 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.738+0000 7f264ffff640 1 --2- 172.21.15.90:0/1181299465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26501031c0 0x7f265019b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:59.726 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.738+0000 7f264ffff640 1 -- 172.21.15.90:0/1181299465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2650108f00 msgr2=0x7f265019c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:59.727 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.738+0000 7f264ffff640 1 --2- 172.21.15.90:0/1181299465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2650108f00 0x7f265019c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:59.727 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.738+0000 7f264ffff640 1 -- 172.21.15.90:0/1181299465 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26501a68f0 con 0x7f2650108a60 2024-04-25T11:39:59.727 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.738+0000 7f26555d9640 1 --2- 172.21.15.90:0/1181299465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2650108f00 0x7f265019c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:59.727 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.738+0000 7f264ffff640 1 --2- 172.21.15.90:0/1181299465 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2650108a60 0x7f265019bb20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f265006c520 tx=0x7f263c00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:59.727 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.738+0000 7f264dffb640 1 -- 172.21.15.90:0/1181299465 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f263c00cc40 con 0x7f2650108a60 2024-04-25T11:39:59.727 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.738+0000 7f265703c640 1 -- 172.21.15.90:0/1181299465 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26501a6be0 con 0x7f2650108a60 2024-04-25T11:39:59.727 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.738+0000 7f264dffb640 1 -- 172.21.15.90:0/1181299465 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f263c00cde0 con 0x7f2650108a60 2024-04-25T11:39:59.727 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.739+0000 7f264dffb640 1 -- 172.21.15.90:0/1181299465 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f263c010430 con 0x7f2650108a60 2024-04-25T11:39:59.727 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.739+0000 7f265703c640 1 -- 172.21.15.90:0/1181299465 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26501a7180 con 0x7f2650108a60 2024-04-25T11:39:59.728 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.742+0000 7f264dffb640 1 -- 172.21.15.90:0/1181299465 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f263c003ed0 con 0x7f2650108a60 2024-04-25T11:39:59.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.742+0000 7f264dffb640 1 --2- 172.21.15.90:0/1181299465 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f26400afc50 0x7f26400b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:59.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.742+0000 7f265dfc5740 1 Processor -- start 2024-04-25T11:39:59.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.742+0000 7f265dfc5740 1 -- start start 2024-04-25T11:39:59.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.743+0000 7f2654dd8640 1 --2- 172.21.15.90:0/1181299465 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f26400afc50 0x7f26400b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:59.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.743+0000 7f264dffb640 1 -- 172.21.15.90:0/1181299465 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f263c0df460 con 0x7f2650108a60 2024-04-25T11:39:59.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.743+0000 7f265dfc5740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6f525c1b0 0x55f6f55fbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:59.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.743+0000 7f265dfc5740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6f52332f0 0x55f6f55fc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:59.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.743+0000 7f265dfc5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f523b4d0 0x55f6f55fc570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:59.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.743+0000 7f265dfc5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f6f5562460 con 0x55f6f523b4d0 2024-04-25T11:39:59.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.743+0000 7f265dfc5740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f6f551c590 con 0x55f6f52332f0 2024-04-25T11:39:59.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.743+0000 7f265dfc5740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f6f555e420 con 0x55f6f525c1b0 2024-04-25T11:39:59.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.743+0000 7f26555d9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f523b4d0 0x55f6f55fc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:59.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.743+0000 7f26555d9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f523b4d0 0x55f6f55fc570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55202/0 (socket says 172.21.15.90:55202) 2024-04-25T11:39:59.732 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.743+0000 7f2654dd8640 1 --2- 172.21.15.90:0/707288558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6f525c1b0 0x55f6f55fbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:59.732 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.743+0000 7f26555d9640 1 -- 172.21.15.90:0/707288558 learned_addr learned my addr 172.21.15.90:0/707288558 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:59.732 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.743+0000 7f264ffff640 1 --2- 172.21.15.90:0/707288558 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6f52332f0 0x55f6f55fc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:59.732 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.743+0000 7f2654dd8640 1 --2- 172.21.15.90:0/1181299465 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f26400afc50 0x7f26400b2160 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2644009230 tx=0x7f2644031400 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:59.732 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.743+0000 7f2654dd8640 1 -- 172.21.15.90:0/707288558 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6f52332f0 msgr2=0x55f6f55fc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:59.732 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.743+0000 7f2654dd8640 1 --2- 172.21.15.90:0/707288558 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6f52332f0 0x55f6f55fc030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:59.732 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.743+0000 7f2654dd8640 1 -- 172.21.15.90:0/707288558 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f523b4d0 msgr2=0x55f6f55fc570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:59.732 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.743+0000 7f2654dd8640 1 --2- 172.21.15.90:0/707288558 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f523b4d0 0x55f6f55fc570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:59.732 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.743+0000 7f2654dd8640 1 -- 172.21.15.90:0/707288558 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6f55290d0 con 0x55f6f525c1b0 2024-04-25T11:39:59.732 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.743+0000 7f264ffff640 1 --2- 172.21.15.90:0/707288558 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6f52332f0 0x55f6f55fc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:59.732 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.743+0000 7f26555d9640 1 --2- 172.21.15.90:0/707288558 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f523b4d0 0x55f6f55fc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:59.732 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.744+0000 7f2654dd8640 1 --2- 172.21.15.90:0/707288558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6f525c1b0 0x55f6f55fbaf0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f264402a930 tx=0x7f264403d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad81ccea1363b8ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:59.732 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.744+0000 7f263affd640 1 -- 172.21.15.90:0/707288558 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264404c070 con 0x55f6f525c1b0 2024-04-25T11:39:59.732 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.744+0000 7f263affd640 1 -- 172.21.15.90:0/707288558 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26440318a0 con 0x55f6f525c1b0 2024-04-25T11:39:59.732 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.744+0000 7f263affd640 1 -- 172.21.15.90:0/707288558 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2644035040 con 0x55f6f525c1b0 2024-04-25T11:39:59.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.745+0000 7f265dfc5740 1 -- 172.21.15.90:0/707288558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6f525c1b0 msgr2=0x55f6f55fbaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:59.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.745+0000 7f265dfc5740 1 --2- 172.21.15.90:0/707288558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6f525c1b0 0x55f6f55fbaf0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f264402a930 tx=0x7f264403d000 comp rx=0 tx=0).stop 2024-04-25T11:39:59.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.745+0000 7f265dfc5740 1 -- 172.21.15.90:0/707288558 shutdown_connections 2024-04-25T11:39:59.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.745+0000 7f265dfc5740 1 --2- 172.21.15.90:0/707288558 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f523b4d0 0x55f6f55fc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:59.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.745+0000 7f265dfc5740 1 --2- 172.21.15.90:0/707288558 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6f52332f0 0x55f6f55fc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:59.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.745+0000 7f265dfc5740 1 --2- 172.21.15.90:0/707288558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6f525c1b0 0x55f6f55fbaf0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:59.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.745+0000 7f265dfc5740 1 -- 172.21.15.90:0/707288558 >> 172.21.15.90:0/707288558 conn(0x55f6f5250300 msgr2=0x55f6f55f82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:59.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.745+0000 7f265dfc5740 1 -- 172.21.15.90:0/707288558 shutdown_connections 2024-04-25T11:39:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.745+0000 7f265dfc5740 1 -- 172.21.15.90:0/707288558 wait complete. 2024-04-25T11:39:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.745+0000 7f265dfc5740 1 -- start start 2024-04-25T11:39:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.745+0000 7f265dfc5740 1 Processor -- start 2024-04-25T11:39:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.746+0000 7f265dfc5740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6f52332f0 0x55f6f5607ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.746+0000 7f265dfc5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f523b4d0 0x55f6f5608100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.746+0000 7f265dfc5740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6f525c1b0 0x55f6f5606020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.746+0000 7f265dfc5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f6f5608bc0 con 0x55f6f523b4d0 2024-04-25T11:39:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.746+0000 7f265dfc5740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f6f54e9b60 con 0x55f6f525c1b0 2024-04-25T11:39:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.746+0000 7f265dfc5740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f6f5608d40 con 0x55f6f52332f0 2024-04-25T11:39:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.746+0000 7f264ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f523b4d0 0x55f6f5608100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.746+0000 7f264ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f523b4d0 0x55f6f5608100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55210/0 (socket says 172.21.15.90:55210) 2024-04-25T11:39:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.746+0000 7f264ffff640 1 -- 172.21.15.90:0/4240042901 learned_addr learned my addr 172.21.15.90:0/4240042901 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.746+0000 7f26555d9640 1 --2- 172.21.15.90:0/4240042901 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6f525c1b0 0x55f6f5606020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.746+0000 7f2654dd8640 1 --2- 172.21.15.90:0/4240042901 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6f52332f0 0x55f6f5607ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.746+0000 7f265dfc5740 10 client.0 _ll_register_callbacks cb 0x55f6f550e2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:59.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.746+0000 7f264ffff640 1 -- 172.21.15.90:0/4240042901 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6f52332f0 msgr2=0x55f6f5607ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:59.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.746+0000 7f264ffff640 1 --2- 172.21.15.90:0/4240042901 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6f52332f0 0x55f6f5607ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:59.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.746+0000 7f264ffff640 1 -- 172.21.15.90:0/4240042901 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6f525c1b0 msgr2=0x55f6f5606020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:59.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.746+0000 7f264ffff640 1 --2- 172.21.15.90:0/4240042901 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6f525c1b0 0x55f6f5606020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:59.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.746+0000 7f264ffff640 1 -- 172.21.15.90:0/4240042901 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6f5274210 con 0x55f6f523b4d0 2024-04-25T11:39:59.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.746+0000 7f26555d9640 1 --2- 172.21.15.90:0/4240042901 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6f525c1b0 0x55f6f5606020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:59.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.747+0000 7f264ffff640 1 --2- 172.21.15.90:0/4240042901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f523b4d0 0x55f6f5608100 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f263c0a4810 tx=0x7f263c0a70b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb36a4bdb442b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:59.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.747+0000 7f262ffff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:59.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.747+0000 7f262ffff640 1 -- 172.21.15.90:0/4240042901 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f263c0ab950 con 0x55f6f523b4d0 2024-04-25T11:39:59.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.747+0000 7f262ffff640 1 -- 172.21.15.90:0/4240042901 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f263c0aff70 con 0x55f6f523b4d0 2024-04-25T11:39:59.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.747+0000 7f265dfc5740 1 -- 172.21.15.90:0/4240042901 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26440318a0 con 0x55f6f523b4d0 2024-04-25T11:39:59.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.747+0000 7f262ffff640 1 -- 172.21.15.90:0/4240042901 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f263c0b0400 con 0x55f6f523b4d0 2024-04-25T11:39:59.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.747+0000 7f265dfc5740 10 client.17070 fetch_fsmap learned FSMap version 28 2024-04-25T11:39:59.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.747+0000 7f265dfc5740 1 -- 172.21.15.90:0/4240042901 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f6f5340620 con 0x55f6f523b4d0 2024-04-25T11:39:59.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.747+0000 7f262ffff640 1 -- 172.21.15.90:0/4240042901 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26440318a0 con 0x55f6f523b4d0 2024-04-25T11:39:59.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.747+0000 7f262ffff640 1 -- 172.21.15.90:0/4240042901 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f263c0a2f10 con 0x55f6f523b4d0 2024-04-25T11:39:59.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.747+0000 7f265dfc5740 10 client.17070 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:39:59.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.748+0000 7f265dfc5740 10 client.17070 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:39:59.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.748+0000 7f265dfc5740 20 client.17070 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:59.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.748+0000 7f265dfc5740 1 --2- 172.21.15.90:0/4240042901 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f6f521ad80 0x55f6f5618800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:59.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.748+0000 7f265dfc5740 4 client.17070 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:39:59.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.748+0000 7f265dfc5740 1 -- 172.21.15.90:0/4240042901 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6f5570070 con 0x55f6f521ad80 2024-04-25T11:39:59.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.748+0000 7f2654dd8640 1 --2- 172.21.15.90:0/4240042901 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f6f521ad80 0x55f6f5618800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:59.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.748+0000 7f262ffff640 10 client.17070 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:59.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.749+0000 7f2654dd8640 1 --2- 172.21.15.90:0/4240042901 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f6f521ad80 0x55f6f5618800 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31ad5f979408de8c server_cookie=8d3bfdac0628b422 in_seq=0 out_seq=0 2024-04-25T11:39:59.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.750+0000 7f262ffff640 1 -- 172.21.15.90:0/4240042901 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f6f5570070 con 0x55f6f521ad80 2024-04-25T11:39:59.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.750+0000 7f262ffff640 10 client.17070 handle_command_reply: tid=0 2024-04-25T11:39:59.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.750+0000 7f265dfc5740 1 client.17070 shutdown 2024-04-25T11:39:59.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.750+0000 7f265dfc5740 1 -- 172.21.15.90:0/4240042901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f523b4d0 msgr2=0x55f6f5608100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:59.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.750+0000 7f265dfc5740 1 --2- 172.21.15.90:0/4240042901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f523b4d0 0x55f6f5608100 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f263c0a4810 tx=0x7f263c0a70b0 comp rx=0 tx=0).stop 2024-04-25T11:39:59.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.750+0000 7f265dfc5740 1 -- 172.21.15.90:0/4240042901 shutdown_connections 2024-04-25T11:39:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.750+0000 7f265dfc5740 1 --2- 172.21.15.90:0/4240042901 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f6f521ad80 0x55f6f5618800 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.750+0000 7f265dfc5740 1 --2- 172.21.15.90:0/4240042901 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6f525c1b0 0x55f6f5606020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.750+0000 7f262ffff640 0 client.17070 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.750+0000 7f265dfc5740 1 --2- 172.21.15.90:0/4240042901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f523b4d0 0x55f6f5608100 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.750+0000 7f265dfc5740 1 --2- 172.21.15.90:0/4240042901 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6f52332f0 0x55f6f5607ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.750+0000 7f265dfc5740 1 -- 172.21.15.90:0/4240042901 >> 172.21.15.90:0/4240042901 conn(0x55f6f5250300 msgr2=0x55f6f55fb920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.750+0000 7f265dfc5740 1 -- 172.21.15.90:0/4240042901 shutdown_connections 2024-04-25T11:39:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.750+0000 7f265dfc5740 1 -- 172.21.15.90:0/4240042901 wait complete. 2024-04-25T11:39:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.751+0000 7f265dfc5740 20 client.17070 trim_cache size 0 max 16384 2024-04-25T11:39:59.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.768+0000 7f265dfc5740 1 Processor -- start 2024-04-25T11:39:59.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.768+0000 7f265dfc5740 1 -- start start 2024-04-25T11:39:59.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.768+0000 7f265dfc5740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6f52332f0 0x55f6f55fbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:59.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.768+0000 7f265dfc5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f525c1b0 0x55f6f561dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:59.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.768+0000 7f265dfc5740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6f51f01c0 0x55f6f561e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:59.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.768+0000 7f265dfc5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f26501afb10 con 0x55f6f525c1b0 2024-04-25T11:39:59.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.768+0000 7f265dfc5740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f26501b1650 con 0x55f6f52332f0 2024-04-25T11:39:59.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.768+0000 7f265dfc5740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f26501b01e0 con 0x55f6f51f01c0 2024-04-25T11:39:59.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.768+0000 7f264ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f525c1b0 0x55f6f561dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:59.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.769+0000 7f264ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f525c1b0 0x55f6f561dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55212/0 (socket says 172.21.15.90:55212) 2024-04-25T11:39:59.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.769+0000 7f264ffff640 1 -- 172.21.15.90:0/593882318 learned_addr learned my addr 172.21.15.90:0/593882318 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:59.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.769+0000 7f2654dd8640 1 --2- 172.21.15.90:0/593882318 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6f52332f0 0x55f6f55fbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:59.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.769+0000 7f26555d9640 1 --2- 172.21.15.90:0/593882318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6f51f01c0 0x55f6f561e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:59.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.769+0000 7f264ffff640 1 -- 172.21.15.90:0/593882318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6f51f01c0 msgr2=0x55f6f561e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:59.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.769+0000 7f264ffff640 1 --2- 172.21.15.90:0/593882318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6f51f01c0 0x55f6f561e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:59.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.769+0000 7f264ffff640 1 -- 172.21.15.90:0/593882318 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6f52332f0 msgr2=0x55f6f55fbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:59.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.769+0000 7f264ffff640 1 --2- 172.21.15.90:0/593882318 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6f52332f0 0x55f6f55fbf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:59.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.769+0000 7f264ffff640 1 -- 172.21.15.90:0/593882318 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6f55f8c90 con 0x55f6f525c1b0 2024-04-25T11:39:59.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.769+0000 7f264ffff640 1 --2- 172.21.15.90:0/593882318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f525c1b0 0x55f6f561dfe0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f263c0a7530 tx=0x7f263c0045d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38de11b39340b604 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:59.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.769+0000 7f263a7fc640 1 -- 172.21.15.90:0/593882318 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f263c0a6ef0 con 0x55f6f525c1b0 2024-04-25T11:39:59.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.769+0000 7f263a7fc640 1 -- 172.21.15.90:0/593882318 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f263c0ab950 con 0x55f6f525c1b0 2024-04-25T11:39:59.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.769+0000 7f263a7fc640 1 -- 172.21.15.90:0/593882318 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f263c0aff70 con 0x55f6f525c1b0 2024-04-25T11:39:59.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.770+0000 7f265dfc5740 1 -- 172.21.15.90:0/593882318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f525c1b0 msgr2=0x55f6f561dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:59.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.770+0000 7f265dfc5740 1 --2- 172.21.15.90:0/593882318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f525c1b0 0x55f6f561dfe0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f263c0a7530 tx=0x7f263c0045d0 comp rx=0 tx=0).stop 2024-04-25T11:39:59.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.770+0000 7f265dfc5740 1 -- 172.21.15.90:0/593882318 shutdown_connections 2024-04-25T11:39:59.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.770+0000 7f265dfc5740 1 --2- 172.21.15.90:0/593882318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6f51f01c0 0x55f6f561e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:59.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.770+0000 7f265dfc5740 1 --2- 172.21.15.90:0/593882318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f525c1b0 0x55f6f561dfe0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:59.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.770+0000 7f265dfc5740 1 --2- 172.21.15.90:0/593882318 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6f52332f0 0x55f6f55fbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:59.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.770+0000 7f265dfc5740 1 -- 172.21.15.90:0/593882318 >> 172.21.15.90:0/593882318 conn(0x55f6f5250300 msgr2=0x55f6f55fb880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:59.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.770+0000 7f265dfc5740 1 -- 172.21.15.90:0/593882318 shutdown_connections 2024-04-25T11:39:59.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.770+0000 7f265dfc5740 1 -- 172.21.15.90:0/593882318 wait complete. 2024-04-25T11:39:59.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.771+0000 7f265dfc5740 1 -- start start 2024-04-25T11:39:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.771+0000 7f265dfc5740 1 Processor -- start 2024-04-25T11:39:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.771+0000 7f265dfc5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f51f01c0 0x55f6f560e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.771+0000 7f265dfc5740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6f52332f0 0x55f6f560b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.771+0000 7f265dfc5740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6f525c1b0 0x55f6f560bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.771+0000 7f265dfc5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f26501b0f80 con 0x55f6f51f01c0 2024-04-25T11:39:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.771+0000 7f265dfc5740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f26501b08b0 con 0x55f6f525c1b0 2024-04-25T11:39:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.771+0000 7f265dfc5740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f26501adaf0 con 0x55f6f52332f0 2024-04-25T11:39:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.771+0000 7f2654dd8640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f51f01c0 0x55f6f560e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.771+0000 7f2654dd8640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f51f01c0 0x55f6f560e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55218/0 (socket says 172.21.15.90:55218) 2024-04-25T11:39:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.771+0000 7f26555d9640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6f525c1b0 0x55f6f560bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.771+0000 7f2654dd8640 1 -- 172.21.15.90:0/2673529507 learned_addr learned my addr 172.21.15.90:0/2673529507 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:39:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.771+0000 7f265dfc5740 10 client.0 _ll_register_callbacks cb 0x7f263c0a7400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:39:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.772+0000 7f2654dd8640 1 -- 172.21.15.90:0/2673529507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6f52332f0 msgr2=0x55f6f560b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:59.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.772+0000 7f2654dd8640 1 --2- 172.21.15.90:0/2673529507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6f52332f0 0x55f6f560b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:59.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.772+0000 7f2654dd8640 1 -- 172.21.15.90:0/2673529507 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6f525c1b0 msgr2=0x55f6f560bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:59.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.772+0000 7f2654dd8640 1 --2- 172.21.15.90:0/2673529507 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6f525c1b0 0x55f6f560bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:59.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.772+0000 7f2654dd8640 1 -- 172.21.15.90:0/2673529507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6f560c050 con 0x55f6f51f01c0 2024-04-25T11:39:59.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.772+0000 7f262ffff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:39:59.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.772+0000 7f2654dd8640 1 --2- 172.21.15.90:0/2673529507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f51f01c0 0x55f6f560e210 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f2644009850 tx=0x7f2644031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f6d621e2ed46a9e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:59.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.772+0000 7f262ffff640 1 -- 172.21.15.90:0/2673529507 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2644065060 con 0x55f6f51f01c0 2024-04-25T11:39:59.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.772+0000 7f262ffff640 1 -- 172.21.15.90:0/2673529507 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f2644004450 con 0x55f6f51f01c0 2024-04-25T11:39:59.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.772+0000 7f265dfc5740 1 -- 172.21.15.90:0/2673529507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f263c0ab950 con 0x55f6f51f01c0 2024-04-25T11:39:59.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.773+0000 7f262ffff640 1 -- 172.21.15.90:0/2673529507 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2644040040 con 0x55f6f51f01c0 2024-04-25T11:39:59.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.773+0000 7f265dfc5740 10 client.17076 fetch_fsmap learned FSMap version 28 2024-04-25T11:39:59.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.773+0000 7f265dfc5740 1 -- 172.21.15.90:0/2673529507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f6f56807c0 con 0x55f6f51f01c0 2024-04-25T11:39:59.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.773+0000 7f262ffff640 1 -- 172.21.15.90:0/2673529507 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f263c0ab950 con 0x55f6f51f01c0 2024-04-25T11:39:59.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.774+0000 7f262ffff640 1 -- 172.21.15.90:0/2673529507 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f26440038c0 con 0x55f6f51f01c0 2024-04-25T11:39:59.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.774+0000 7f265dfc5740 10 client.17076 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:39:59.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.774+0000 7f265dfc5740 10 client.17076 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:39:59.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.774+0000 7f265dfc5740 20 client.17076 populate_metadata read hostname 'smithi090' 2024-04-25T11:39:59.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.774+0000 7f265dfc5740 1 --2- 172.21.15.90:0/2673529507 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f6f5680ab0 0x55f6f5682ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:59.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.774+0000 7f265dfc5740 4 client.17076 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:39:59.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.774+0000 7f265dfc5740 1 -- 172.21.15.90:0/2673529507 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f6f5569ee0 con 0x55f6f5680ab0 2024-04-25T11:39:59.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.774+0000 7f264ffff640 1 --2- 172.21.15.90:0/2673529507 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f6f5680ab0 0x55f6f5682ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:59.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.774+0000 7f262ffff640 10 client.17076 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:39:59.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:39:59.775+0000 7f264ffff640 1 --2- 172.21.15.90:0/2673529507 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f6f5680ab0 0x55f6f5682ef0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1345951aad69fe94 server_cookie=d9781c7dd35025fb in_seq=0 out_seq=0 2024-04-25T11:39:59.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:39:59 smithi138 ceph-mon[32328]: pgmap v150: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 393 MiB/s wr, 520 op/s 2024-04-25T11:39:59.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:39:59 smithi156 ceph-mon[33045]: pgmap v150: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 393 MiB/s wr, 520 op/s 2024-04-25T11:40:00.723 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:00 smithi090 ceph-mon[29575]: overall HEALTH_OK 2024-04-25T11:40:00.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:00 smithi138 ceph-mon[32328]: overall HEALTH_OK 2024-04-25T11:40:00.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:00 smithi156 ceph-mon[33045]: overall HEALTH_OK 2024-04-25T11:40:01.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:01 smithi138 ceph-mon[32328]: pgmap v151: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 579 MiB/s wr, 741 op/s 2024-04-25T11:40:01.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:01 smithi156 ceph-mon[33045]: pgmap v151: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 579 MiB/s wr, 741 op/s 2024-04-25T11:40:01.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:01 smithi090 ceph-mon[29575]: pgmap v151: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 579 MiB/s wr, 741 op/s 2024-04-25T11:40:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.492+0000 7f262ffff640 1 -- 172.21.15.90:0/2673529507 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f6f5569ee0 con 0x55f6f5680ab0 2024-04-25T11:40:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.492+0000 7f262ffff640 10 client.17076 handle_command_reply: tid=0 2024-04-25T11:40:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.492+0000 7f265dfc5740 1 client.17076 shutdown 2024-04-25T11:40:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.492+0000 7f265dfc5740 1 -- 172.21.15.90:0/2673529507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f51f01c0 msgr2=0x55f6f560e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.492+0000 7f265dfc5740 1 --2- 172.21.15.90:0/2673529507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f51f01c0 0x55f6f560e210 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f2644009850 tx=0x7f2644031a50 comp rx=0 tx=0).stop 2024-04-25T11:40:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.493+0000 7f265dfc5740 1 -- 172.21.15.90:0/2673529507 shutdown_connections 2024-04-25T11:40:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.493+0000 7f265dfc5740 1 --2- 172.21.15.90:0/2673529507 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f6f5680ab0 0x55f6f5682ef0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.493+0000 7f265dfc5740 1 --2- 172.21.15.90:0/2673529507 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f6f525c1b0 0x55f6f560bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.493+0000 7f265dfc5740 1 --2- 172.21.15.90:0/2673529507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6f52332f0 0x55f6f560b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.493+0000 7f265dfc5740 1 --2- 172.21.15.90:0/2673529507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f6f51f01c0 0x55f6f560e210 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.493+0000 7f265dfc5740 1 -- 172.21.15.90:0/2673529507 >> 172.21.15.90:0/2673529507 conn(0x55f6f5250300 msgr2=0x55f6f5607630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.493+0000 7f262ffff640 0 client.17076 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.493+0000 7f265dfc5740 1 -- 172.21.15.90:0/2673529507 shutdown_connections 2024-04-25T11:40:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.493+0000 7f265dfc5740 1 -- 172.21.15.90:0/2673529507 wait complete. 2024-04-25T11:40:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.493+0000 7f265dfc5740 20 client.17076 trim_cache size 0 max 16384 2024-04-25T11:40:03.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:40:03.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:40:03.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:40:03.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:40:03.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.494+0000 7f265703c640 1 -- 172.21.15.90:0/1181299465 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f26400afc50 msgr2=0x7f26400b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:03.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.494+0000 7f265703c640 1 --2- 172.21.15.90:0/1181299465 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f26400afc50 0x7f26400b2160 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2644009230 tx=0x7f2644031400 comp rx=0 tx=0).stop 2024-04-25T11:40:03.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.494+0000 7f265703c640 1 -- 172.21.15.90:0/1181299465 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2650108a60 msgr2=0x7f265019bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:03.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.494+0000 7f265703c640 1 --2- 172.21.15.90:0/1181299465 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2650108a60 0x7f265019bb20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f265006c520 tx=0x7f263c00ee50 comp rx=0 tx=0).stop 2024-04-25T11:40:03.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.494+0000 7f265703c640 1 -- 172.21.15.90:0/1181299465 shutdown_connections 2024-04-25T11:40:03.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.494+0000 7f265703c640 1 --2- 172.21.15.90:0/1181299465 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f26400afc50 0x7f26400b2160 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:03.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.494+0000 7f265703c640 1 --2- 172.21.15.90:0/1181299465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2650108f00 0x7f265019c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:03.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.494+0000 7f265703c640 1 --2- 172.21.15.90:0/1181299465 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2650108a60 0x7f265019bb20 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:03.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.494+0000 7f265703c640 1 --2- 172.21.15.90:0/1181299465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26501031c0 0x7f265019b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:03.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.494+0000 7f265703c640 1 -- 172.21.15.90:0/1181299465 >> 172.21.15.90:0/1181299465 conn(0x7f26500781b0 msgr2=0x7f2650104140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:03.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.495+0000 7f265703c640 1 -- 172.21.15.90:0/1181299465 shutdown_connections 2024-04-25T11:40:03.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:03.495+0000 7f265703c640 1 -- 172.21.15.90:0/1181299465 wait complete. 2024-04-25T11:40:03.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:03 smithi138 ceph-mon[32328]: pgmap v152: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 440 MiB/s wr, 652 op/s 2024-04-25T11:40:03.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:03 smithi156 ceph-mon[33045]: pgmap v152: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 440 MiB/s wr, 652 op/s 2024-04-25T11:40:03.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:03 smithi090 ceph-mon[29575]: pgmap v152: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 440 MiB/s wr, 652 op/s 2024-04-25T11:40:05.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:05 smithi156 ceph-mon[33045]: pgmap v153: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 503 MiB/s wr, 707 op/s 2024-04-25T11:40:05.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:05 smithi090 ceph-mon[29575]: pgmap v153: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 503 MiB/s wr, 707 op/s 2024-04-25T11:40:06.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:05 smithi138 ceph-mon[32328]: pgmap v153: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 503 MiB/s wr, 707 op/s 2024-04-25T11:40:07.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:07 smithi090 ceph-mon[29575]: pgmap v154: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 550 MiB/s wr, 700 op/s 2024-04-25T11:40:08.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:07 smithi138 ceph-mon[32328]: pgmap v154: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 550 MiB/s wr, 700 op/s 2024-04-25T11:40:08.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:07 smithi156 ceph-mon[33045]: pgmap v154: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 550 MiB/s wr, 700 op/s 2024-04-25T11:40:10.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:09 smithi156 ceph-mon[33045]: pgmap v155: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 1.3 MiB/s rd, 372 MiB/s wr, 391 op/s 2024-04-25T11:40:10.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:09 smithi090 ceph-mon[29575]: pgmap v155: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 1.3 MiB/s rd, 372 MiB/s wr, 391 op/s 2024-04-25T11:40:10.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:09 smithi138 ceph-mon[32328]: pgmap v155: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 1.3 MiB/s rd, 372 MiB/s wr, 391 op/s 2024-04-25T11:40:12.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:11 smithi090 ceph-mon[29575]: pgmap v156: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 1.5 MiB/s rd, 488 MiB/s wr, 654 op/s 2024-04-25T11:40:12.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:11 smithi138 ceph-mon[32328]: pgmap v156: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 1.5 MiB/s rd, 488 MiB/s wr, 654 op/s 2024-04-25T11:40:12.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:11 smithi156 ceph-mon[33045]: pgmap v156: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 1.5 MiB/s rd, 488 MiB/s wr, 654 op/s 2024-04-25T11:40:14.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:13 smithi090 ceph-mon[29575]: pgmap v157: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 890 KiB/s rd, 302 MiB/s wr, 433 op/s 2024-04-25T11:40:14.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:13 smithi138 ceph-mon[32328]: pgmap v157: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 890 KiB/s rd, 302 MiB/s wr, 433 op/s 2024-04-25T11:40:14.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:13 smithi156 ceph-mon[33045]: pgmap v157: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 890 KiB/s rd, 302 MiB/s wr, 433 op/s 2024-04-25T11:40:16.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:15 smithi090 ceph-mon[29575]: pgmap v158: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 922 KiB/s rd, 345 MiB/s wr, 624 op/s 2024-04-25T11:40:16.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:15 smithi138 ceph-mon[32328]: pgmap v158: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 922 KiB/s rd, 345 MiB/s wr, 624 op/s 2024-04-25T11:40:16.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:15 smithi156 ceph-mon[33045]: pgmap v158: 97 pgs: 97 active+clean; 40 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 922 KiB/s rd, 345 MiB/s wr, 624 op/s 2024-04-25T11:40:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.610+0000 7f7c74b34640 1 -- 172.21.15.90:0/564811914 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7c701031c0 msgr2=0x7f7c7010f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.610+0000 7f7c74b34640 1 --2- 172.21.15.90:0/564811914 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7c701031c0 0x7f7c7010f4f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7c68009be0 tx=0x7f7c6802a6c0 comp rx=0 tx=0).stop 2024-04-25T11:40:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.610+0000 7f7c74b34640 1 -- 172.21.15.90:0/564811914 shutdown_connections 2024-04-25T11:40:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.610+0000 7f7c74b34640 1 --2- 172.21.15.90:0/564811914 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7c701031c0 0x7f7c7010f4f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.610+0000 7f7c74b34640 1 --2- 172.21.15.90:0/564811914 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c70108f00 0x7f7c70102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.610+0000 7f7c74b34640 1 --2- 172.21.15.90:0/564811914 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c70108a60 0x7f7c70102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.610+0000 7f7c74b34640 1 -- 172.21.15.90:0/564811914 >> 172.21.15.90:0/564811914 conn(0x7f7c700781b0 msgr2=0x7f7c700785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.611+0000 7f7c74b34640 1 -- 172.21.15.90:0/564811914 shutdown_connections 2024-04-25T11:40:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.611+0000 7f7c74b34640 1 -- 172.21.15.90:0/564811914 wait complete. 2024-04-25T11:40:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.612+0000 7f7c74b34640 1 Processor -- start 2024-04-25T11:40:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.612+0000 7f7c74b34640 1 -- start start 2024-04-25T11:40:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.612+0000 7f7c74b34640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c701031c0 0x7f7c701a2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.612+0000 7f7c74b34640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7c70108a60 0x7f7c701a2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.612+0000 7f7c74b34640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c70108f00 0x7f7c7019c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.612+0000 7f7c74b34640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7c70112ca0 con 0x7f7c701031c0 2024-04-25T11:40:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.612+0000 7f7c74b34640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7c70112b20 con 0x7f7c70108a60 2024-04-25T11:40:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.612+0000 7f7c74b34640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7c70112e20 con 0x7f7c70108f00 2024-04-25T11:40:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.612+0000 7f7c6dd9b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7c70108a60 0x7f7c701a2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.612+0000 7f7c6e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c701031c0 0x7f7c701a2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.612+0000 7f7c6dd9b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7c70108a60 0x7f7c701a2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:36418/0 (socket says 172.21.15.90:36418) 2024-04-25T11:40:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.612+0000 7f7c6dd9b640 1 -- 172.21.15.90:0/4226948219 learned_addr learned my addr 172.21.15.90:0/4226948219 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.612+0000 7f7c6ed9d640 1 --2- 172.21.15.90:0/4226948219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c70108f00 0x7f7c7019c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.612+0000 7f7c6dd9b640 1 -- 172.21.15.90:0/4226948219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c70108f00 msgr2=0x7f7c7019c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.612+0000 7f7c6dd9b640 1 --2- 172.21.15.90:0/4226948219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c70108f00 0x7f7c7019c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.613+0000 7f7c6dd9b640 1 -- 172.21.15.90:0/4226948219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c701031c0 msgr2=0x7f7c701a2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.613+0000 7f7c6dd9b640 1 --2- 172.21.15.90:0/4226948219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c701031c0 0x7f7c701a2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.613+0000 7f7c6dd9b640 1 -- 172.21.15.90:0/4226948219 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c7019cf20 con 0x7f7c70108a60 2024-04-25T11:40:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.613+0000 7f7c6dd9b640 1 --2- 172.21.15.90:0/4226948219 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7c70108a60 0x7f7c701a2bb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7c6400e980 tx=0x7f7c6400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.613+0000 7f7c5f7fe640 1 -- 172.21.15.90:0/4226948219 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c6400cd80 con 0x7f7c70108a60 2024-04-25T11:40:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.613+0000 7f7c5f7fe640 1 -- 172.21.15.90:0/4226948219 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c640108e0 con 0x7f7c70108a60 2024-04-25T11:40:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.613+0000 7f7c5f7fe640 1 -- 172.21.15.90:0/4226948219 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c64018940 con 0x7f7c70108a60 2024-04-25T11:40:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.613+0000 7f7c74b34640 1 -- 172.21.15.90:0/4226948219 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c7019d210 con 0x7f7c70108a60 2024-04-25T11:40:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.613+0000 7f7c74b34640 1 -- 172.21.15.90:0/4226948219 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c701a94f0 con 0x7f7c70108a60 2024-04-25T11:40:16.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.616+0000 7f7c5f7fe640 1 -- 172.21.15.90:0/4226948219 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f7c6401f020 con 0x7f7c70108a60 2024-04-25T11:40:16.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.616+0000 7f7c7b9e4740 1 Processor -- start 2024-04-25T11:40:16.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.616+0000 7f7c7b9e4740 1 -- start start 2024-04-25T11:40:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.617+0000 7f7c7b9e4740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7942661b0 0x55d79460e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.617+0000 7f7c7b9e4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d79423d2f0 0x55d79460e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.617+0000 7f7c7b9e4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d7942454d0 0x55d79460ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.617+0000 7f7c7b9e4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d79456c460 con 0x55d79423d2f0 2024-04-25T11:40:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.617+0000 7f7c7b9e4740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55d794526590 con 0x55d7942661b0 2024-04-25T11:40:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.617+0000 7f7c7b9e4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d794568420 con 0x55d7942454d0 2024-04-25T11:40:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.617+0000 7f7c6ed9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7942661b0 0x55d79460e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.617+0000 7f7c6ed9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7942661b0 0x55d79460e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:36428/0 (socket says 172.21.15.90:36428) 2024-04-25T11:40:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.617+0000 7f7c6ed9d640 1 -- 172.21.15.90:0/2004486927 learned_addr learned my addr 172.21.15.90:0/2004486927 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.617+0000 7f7c6e59c640 1 --2- 172.21.15.90:0/2004486927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d79423d2f0 0x55d79460e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.617+0000 7f7c6dd9b640 1 --2- 172.21.15.90:0/2004486927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d7942454d0 0x55d79460ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:16.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.617+0000 7f7c6ed9d640 1 -- 172.21.15.90:0/2004486927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d7942454d0 msgr2=0x55d79460ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:16.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.617+0000 7f7c6ed9d640 1 --2- 172.21.15.90:0/2004486927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d7942454d0 0x55d79460ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:16.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.617+0000 7f7c6ed9d640 1 -- 172.21.15.90:0/2004486927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d79423d2f0 msgr2=0x55d79460e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:16.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.617+0000 7f7c6ed9d640 1 --2- 172.21.15.90:0/2004486927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d79423d2f0 0x55d79460e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:16.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.617+0000 7f7c6ed9d640 1 -- 172.21.15.90:0/2004486927 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7945330d0 con 0x55d7942661b0 2024-04-25T11:40:16.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.617+0000 7f7c5f7fe640 1 --2- 172.21.15.90:0/4226948219 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f7c580afc50 0x7f7c580b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:16.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.618+0000 7f7c6ed9d640 1 --2- 172.21.15.90:0/2004486927 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7942661b0 0x55d79460e300 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7c68009be0 tx=0x7f7c68030430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1778cf7c1b18c792 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:16.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.618+0000 7f7c6e59c640 1 --2- 172.21.15.90:0/4226948219 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f7c580afc50 0x7f7c580b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:16.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.618+0000 7f7c5f7fe640 1 -- 172.21.15.90:0/4226948219 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f7c640df170 con 0x7f7c70108a60 2024-04-25T11:40:16.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.618+0000 7f7c5cff9640 1 -- 172.21.15.90:0/2004486927 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c68041030 con 0x55d7942661b0 2024-04-25T11:40:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.618+0000 7f7c5cff9640 1 -- 172.21.15.90:0/2004486927 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c68003e20 con 0x55d7942661b0 2024-04-25T11:40:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.618+0000 7f7c5cff9640 1 -- 172.21.15.90:0/2004486927 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c68036040 con 0x55d7942661b0 2024-04-25T11:40:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.619+0000 7f7c6e59c640 1 --2- 172.21.15.90:0/4226948219 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f7c580afc50 0x7f7c580b2160 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x55d79460f6b0 tx=0x7f7c60005f90 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.619+0000 7f7c7b9e4740 1 -- 172.21.15.90:0/2004486927 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7942661b0 msgr2=0x55d79460e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.619+0000 7f7c7b9e4740 1 --2- 172.21.15.90:0/2004486927 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7942661b0 0x55d79460e300 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7c68009be0 tx=0x7f7c68030430 comp rx=0 tx=0).stop 2024-04-25T11:40:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.619+0000 7f7c7b9e4740 1 -- 172.21.15.90:0/2004486927 shutdown_connections 2024-04-25T11:40:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.619+0000 7f7c7b9e4740 1 --2- 172.21.15.90:0/2004486927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d7942454d0 0x55d79460ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.619+0000 7f7c7b9e4740 1 --2- 172.21.15.90:0/2004486927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d79423d2f0 0x55d79460e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.619+0000 7f7c7b9e4740 1 --2- 172.21.15.90:0/2004486927 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7942661b0 0x55d79460e300 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.619+0000 7f7c7b9e4740 1 -- 172.21.15.90:0/2004486927 >> 172.21.15.90:0/2004486927 conn(0x55d79425a300 msgr2=0x55d79460aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.619+0000 7f7c7b9e4740 1 -- 172.21.15.90:0/2004486927 shutdown_connections 2024-04-25T11:40:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.619+0000 7f7c7b9e4740 1 -- 172.21.15.90:0/2004486927 wait complete. 2024-04-25T11:40:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.620+0000 7f7c7b9e4740 1 -- start start 2024-04-25T11:40:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.620+0000 7f7c7b9e4740 1 Processor -- start 2024-04-25T11:40:16.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.620+0000 7f7c7b9e4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d79423d2f0 0x55d7946189f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:16.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.620+0000 7f7c7b9e4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7942454d0 0x55d794618f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:16.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.620+0000 7f7c7b9e4740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7942661b0 0x55d79461e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:16.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.620+0000 7f7c7b9e4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d79461b3d0 con 0x55d7942454d0 2024-04-25T11:40:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.620+0000 7f7c7b9e4740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55d7944f3b60 con 0x55d7942661b0 2024-04-25T11:40:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.620+0000 7f7c7b9e4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d79461b550 con 0x55d79423d2f0 2024-04-25T11:40:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.620+0000 7f7c7b9e4740 10 client.0 _ll_register_callbacks cb 0x55d7945182b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.620+0000 7f7c6dd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7942454d0 0x55d794618f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.620+0000 7f7c6dd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7942454d0 0x55d794618f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48552/0 (socket says 172.21.15.90:48552) 2024-04-25T11:40:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.620+0000 7f7c6dd9b640 1 -- 172.21.15.90:0/3687709407 learned_addr learned my addr 172.21.15.90:0/3687709407 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.620+0000 7f7c6ed9d640 1 --2- 172.21.15.90:0/3687709407 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7942661b0 0x55d79461e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.620+0000 7f7c6e59c640 1 --2- 172.21.15.90:0/3687709407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d79423d2f0 0x55d7946189f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.621+0000 7f7c6dd9b640 1 -- 172.21.15.90:0/3687709407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d79423d2f0 msgr2=0x55d7946189f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.621+0000 7f7c6dd9b640 1 --2- 172.21.15.90:0/3687709407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d79423d2f0 0x55d7946189f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.621+0000 7f7c6dd9b640 1 -- 172.21.15.90:0/3687709407 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7942661b0 msgr2=0x55d79461e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.621+0000 7f7c6dd9b640 1 --2- 172.21.15.90:0/3687709407 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7942661b0 0x55d79461e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.621+0000 7f7c6dd9b640 1 -- 172.21.15.90:0/3687709407 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d79427e210 con 0x55d7942454d0 2024-04-25T11:40:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.621+0000 7f7c6ed9d640 1 --2- 172.21.15.90:0/3687709407 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7942661b0 0x55d79461e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:40:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.621+0000 7f7c6dd9b640 1 --2- 172.21.15.90:0/3687709407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7942454d0 0x55d794618f30 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f7c640dc8c0 tx=0x7f7c64009670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be699e371680f30d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.621+0000 7f7c55ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:40:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.621+0000 7f7c55ffb640 1 -- 172.21.15.90:0/3687709407 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c640a6a20 con 0x55d7942454d0 2024-04-25T11:40:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.621+0000 7f7c55ffb640 1 -- 172.21.15.90:0/3687709407 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f7c640dcae0 con 0x55d7942454d0 2024-04-25T11:40:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.621+0000 7f7c7b9e4740 1 -- 172.21.15.90:0/3687709407 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c68003e20 con 0x55d7942454d0 2024-04-25T11:40:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.621+0000 7f7c55ffb640 1 -- 172.21.15.90:0/3687709407 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7c640e2e60 con 0x55d7942454d0 2024-04-25T11:40:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.621+0000 7f7c7b9e4740 10 client.17088 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.621+0000 7f7c7b9e4740 1 -- 172.21.15.90:0/3687709407 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d79434a620 con 0x55d7942454d0 2024-04-25T11:40:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.621+0000 7f7c55ffb640 1 -- 172.21.15.90:0/3687709407 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c68003e20 con 0x55d7942454d0 2024-04-25T11:40:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.622+0000 7f7c55ffb640 1 -- 172.21.15.90:0/3687709407 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f7c640afc80 con 0x55d7942454d0 2024-04-25T11:40:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.622+0000 7f7c7b9e4740 10 client.17088 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.622+0000 7f7c7b9e4740 10 client.17088 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:40:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.622+0000 7f7c7b9e4740 20 client.17088 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.622+0000 7f7c7b9e4740 1 --2- 172.21.15.90:0/3687709407 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d794224d80 0x55d79462afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.622+0000 7f7c7b9e4740 4 client.17088 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:40:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.622+0000 7f7c7b9e4740 1 -- 172.21.15.90:0/3687709407 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d79457a070 con 0x55d794224d80 2024-04-25T11:40:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.622+0000 7f7c6e59c640 1 --2- 172.21.15.90:0/3687709407 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d794224d80 0x55d79462afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.623+0000 7f7c6e59c640 1 --2- 172.21.15.90:0/3687709407 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d794224d80 0x55d79462afb0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d9d93e7723cf9eb server_cookie=6fef53cdd3125e9c in_seq=0 out_seq=0 2024-04-25T11:40:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.623+0000 7f7c55ffb640 10 client.17088 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.624+0000 7f7c55ffb640 1 -- 172.21.15.90:0/3687709407 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d79457a070 con 0x55d794224d80 2024-04-25T11:40:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.624+0000 7f7c55ffb640 10 client.17088 handle_command_reply: tid=0 2024-04-25T11:40:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.624+0000 7f7c7b9e4740 1 client.17088 shutdown 2024-04-25T11:40:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.625+0000 7f7c7b9e4740 1 -- 172.21.15.90:0/3687709407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7942454d0 msgr2=0x55d794618f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.625+0000 7f7c7b9e4740 1 --2- 172.21.15.90:0/3687709407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7942454d0 0x55d794618f30 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f7c640dc8c0 tx=0x7f7c64009670 comp rx=0 tx=0).stop 2024-04-25T11:40:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.625+0000 7f7c7b9e4740 1 -- 172.21.15.90:0/3687709407 shutdown_connections 2024-04-25T11:40:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.625+0000 7f7c7b9e4740 1 --2- 172.21.15.90:0/3687709407 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d794224d80 0x55d79462afb0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.625+0000 7f7c7b9e4740 1 --2- 172.21.15.90:0/3687709407 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7942661b0 0x55d79461e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.625+0000 7f7c7b9e4740 1 --2- 172.21.15.90:0/3687709407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7942454d0 0x55d794618f30 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.625+0000 7f7c7b9e4740 1 --2- 172.21.15.90:0/3687709407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d79423d2f0 0x55d7946189f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:16.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.625+0000 7f7c55ffb640 0 client.17088 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:16.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.625+0000 7f7c7b9e4740 1 -- 172.21.15.90:0/3687709407 >> 172.21.15.90:0/3687709407 conn(0x55d79425a300 msgr2=0x55d79460e130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:16.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.625+0000 7f7c7b9e4740 1 -- 172.21.15.90:0/3687709407 shutdown_connections 2024-04-25T11:40:16.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.625+0000 7f7c7b9e4740 1 -- 172.21.15.90:0/3687709407 wait complete. 2024-04-25T11:40:16.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.625+0000 7f7c7b9e4740 20 client.17088 trim_cache size 0 max 16384 2024-04-25T11:40:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.642+0000 7f7c7b9e4740 1 Processor -- start 2024-04-25T11:40:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.642+0000 7f7c7b9e4740 1 -- start start 2024-04-25T11:40:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.642+0000 7f7c7b9e4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d79423d2f0 0x55d794618430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.642+0000 7f7c7b9e4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d7942454d0 0x55d79461e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.642+0000 7f7c7b9e4740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7942661b0 0x55d794630710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.642+0000 7f7c7b9e4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7c701ada70 con 0x55d79423d2f0 2024-04-25T11:40:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.642+0000 7f7c7b9e4740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7c701af5b0 con 0x55d7942661b0 2024-04-25T11:40:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.642+0000 7f7c7b9e4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7c701ae140 con 0x55d7942454d0 2024-04-25T11:40:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.642+0000 7f7c6e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d79423d2f0 0x55d794618430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.642+0000 7f7c6ed9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7942661b0 0x55d794630710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.642+0000 7f7c6e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d79423d2f0 0x55d794618430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48560/0 (socket says 172.21.15.90:48560) 2024-04-25T11:40:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.642+0000 7f7c6e59c640 1 -- 172.21.15.90:0/700576560 learned_addr learned my addr 172.21.15.90:0/700576560 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.642+0000 7f7c6dd9b640 1 --2- 172.21.15.90:0/700576560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d7942454d0 0x55d79461e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.642+0000 7f7c6ed9d640 1 -- 172.21.15.90:0/700576560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d7942454d0 msgr2=0x55d79461e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.642+0000 7f7c6ed9d640 1 --2- 172.21.15.90:0/700576560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d7942454d0 0x55d79461e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.642+0000 7f7c6ed9d640 1 -- 172.21.15.90:0/700576560 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d79423d2f0 msgr2=0x55d794618430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.642+0000 7f7c6ed9d640 1 --2- 172.21.15.90:0/700576560 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d79423d2f0 0x55d794618430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.642+0000 7f7c6ed9d640 1 -- 172.21.15.90:0/700576560 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d794619700 con 0x55d7942661b0 2024-04-25T11:40:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.642+0000 7f7c6dd9b640 1 --2- 172.21.15.90:0/700576560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d7942454d0 0x55d79461e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:40:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.642+0000 7f7c6e59c640 1 --2- 172.21.15.90:0/700576560 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d79423d2f0 0x55d794618430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:40:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.643+0000 7f7c6ed9d640 1 --2- 172.21.15.90:0/700576560 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7942661b0 0x55d794630710 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7c6802abe0 tx=0x7f7c68009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=631cebb1f2f9a3b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.643+0000 7f7c57fff640 1 -- 172.21.15.90:0/700576560 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c68036070 con 0x55d7942661b0 2024-04-25T11:40:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.643+0000 7f7c57fff640 1 -- 172.21.15.90:0/700576560 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c68003e20 con 0x55d7942661b0 2024-04-25T11:40:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.643+0000 7f7c57fff640 1 -- 172.21.15.90:0/700576560 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c68041040 con 0x55d7942661b0 2024-04-25T11:40:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.644+0000 7f7c7b9e4740 1 -- 172.21.15.90:0/700576560 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7942661b0 msgr2=0x55d794630710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.644+0000 7f7c7b9e4740 1 --2- 172.21.15.90:0/700576560 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7942661b0 0x55d794630710 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7c6802abe0 tx=0x7f7c68009500 comp rx=0 tx=0).stop 2024-04-25T11:40:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.644+0000 7f7c7b9e4740 1 -- 172.21.15.90:0/700576560 shutdown_connections 2024-04-25T11:40:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.644+0000 7f7c7b9e4740 1 --2- 172.21.15.90:0/700576560 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7942661b0 0x55d794630710 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.644+0000 7f7c7b9e4740 1 --2- 172.21.15.90:0/700576560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d7942454d0 0x55d79461e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.644+0000 7f7c7b9e4740 1 --2- 172.21.15.90:0/700576560 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d79423d2f0 0x55d794618430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.644+0000 7f7c7b9e4740 1 -- 172.21.15.90:0/700576560 >> 172.21.15.90:0/700576560 conn(0x55d79425a300 msgr2=0x55d794617d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.645+0000 7f7c7b9e4740 1 -- 172.21.15.90:0/700576560 shutdown_connections 2024-04-25T11:40:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.645+0000 7f7c7b9e4740 1 -- 172.21.15.90:0/700576560 wait complete. 2024-04-25T11:40:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.645+0000 7f7c7b9e4740 1 -- start start 2024-04-25T11:40:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.645+0000 7f7c7b9e4740 1 Processor -- start 2024-04-25T11:40:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.646+0000 7f7c7b9e4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d79423d2f0 0x55d794624e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.646+0000 7f7c7b9e4740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7942454d0 0x55d7946253b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.646+0000 7f7c7b9e4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7942661b0 0x55d794620320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.646+0000 7f7c7b9e4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7c701aeee0 con 0x55d7942661b0 2024-04-25T11:40:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.646+0000 7f7c7b9e4740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7c701ae810 con 0x55d7942454d0 2024-04-25T11:40:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.646+0000 7f7c7b9e4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7c701abaf0 con 0x55d79423d2f0 2024-04-25T11:40:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.646+0000 7f7c7b9e4740 10 client.0 _ll_register_callbacks cb 0x7f7c6400ef70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.646+0000 7f7c6dd9b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7942454d0 0x55d7946253b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.646+0000 7f7c6dd9b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7942454d0 0x55d7946253b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:36448/0 (socket says 172.21.15.90:36448) 2024-04-25T11:40:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.646+0000 7f7c6dd9b640 1 -- 172.21.15.90:0/1637700066 learned_addr learned my addr 172.21.15.90:0/1637700066 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.646+0000 7f7c6ed9d640 1 --2- 172.21.15.90:0/1637700066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7942661b0 0x55d794620320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.646+0000 7f7c6e59c640 1 --2- 172.21.15.90:0/1637700066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d79423d2f0 0x55d794624e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:16.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.646+0000 7f7c6dd9b640 1 -- 172.21.15.90:0/1637700066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d79423d2f0 msgr2=0x55d794624e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:16.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.646+0000 7f7c6dd9b640 1 --2- 172.21.15.90:0/1637700066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d79423d2f0 0x55d794624e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:16.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.646+0000 7f7c6dd9b640 1 -- 172.21.15.90:0/1637700066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7942661b0 msgr2=0x55d794620320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:16.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.646+0000 7f7c6dd9b640 1 --2- 172.21.15.90:0/1637700066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7942661b0 0x55d794620320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:16.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.646+0000 7f7c6dd9b640 1 -- 172.21.15.90:0/1637700066 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d794620ab0 con 0x55d7942454d0 2024-04-25T11:40:16.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.647+0000 7f7c6dd9b640 1 --2- 172.21.15.90:0/1637700066 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7942454d0 0x55d7946253b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7c640a7240 tx=0x7f7c640ab1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea848f2d1f992cb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:16.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.647+0000 7f7c55ffb640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:40:16.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.647+0000 7f7c55ffb640 1 -- 172.21.15.90:0/1637700066 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c640dccf0 con 0x55d7942454d0 2024-04-25T11:40:16.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.647+0000 7f7c55ffb640 1 -- 172.21.15.90:0/1637700066 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f7c64010430 con 0x55d7942454d0 2024-04-25T11:40:16.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.647+0000 7f7c7b9e4740 1 -- 172.21.15.90:0/1637700066 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c68003e20 con 0x55d7942454d0 2024-04-25T11:40:16.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.647+0000 7f7c55ffb640 1 -- 172.21.15.90:0/1637700066 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7c640a6dc0 con 0x55d7942454d0 2024-04-25T11:40:16.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.647+0000 7f7c7b9e4740 10 client.26246 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:16.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.647+0000 7f7c7b9e4740 1 -- 172.21.15.90:0/1637700066 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d794692df0 con 0x55d7942454d0 2024-04-25T11:40:16.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.647+0000 7f7c55ffb640 1 -- 172.21.15.90:0/1637700066 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c68003e20 con 0x55d7942454d0 2024-04-25T11:40:16.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.647+0000 7f7c55ffb640 1 -- 172.21.15.90:0/1637700066 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f7c64018b00 con 0x55d7942454d0 2024-04-25T11:40:16.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.648+0000 7f7c7b9e4740 10 client.26246 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:16.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.648+0000 7f7c7b9e4740 10 client.26246 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:40:16.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.648+0000 7f7c7b9e4740 20 client.26246 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:16.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.648+0000 7f7c7b9e4740 1 --2- 172.21.15.90:0/1637700066 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d7941fa1c0 0x55d794695330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:16.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.648+0000 7f7c7b9e4740 4 client.26246 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:40:16.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.648+0000 7f7c7b9e4740 1 -- 172.21.15.90:0/1637700066 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d794573ee0 con 0x55d7941fa1c0 2024-04-25T11:40:16.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.648+0000 7f7c6e59c640 1 --2- 172.21.15.90:0/1637700066 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d7941fa1c0 0x55d794695330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:16.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.648+0000 7f7c55ffb640 10 client.26246 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:16.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:16.648+0000 7f7c6e59c640 1 --2- 172.21.15.90:0/1637700066 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d7941fa1c0 0x55d794695330 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38f1d49f35b5d251 server_cookie=1ccdbfe99cae96d7 in_seq=0 out_seq=0 2024-04-25T11:40:18.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:17 smithi138 ceph-mon[32328]: pgmap v159: 97 pgs: 97 active+clean; 40 GiB data, 123 GiB used, 950 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 394 MiB/s wr, 1.19k op/s 2024-04-25T11:40:18.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:17 smithi156 ceph-mon[33045]: pgmap v159: 97 pgs: 97 active+clean; 40 GiB data, 123 GiB used, 950 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 394 MiB/s wr, 1.19k op/s 2024-04-25T11:40:18.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:17 smithi090 ceph-mon[29575]: pgmap v159: 97 pgs: 97 active+clean; 40 GiB data, 123 GiB used, 950 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 394 MiB/s wr, 1.19k op/s 2024-04-25T11:40:19.566 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:40:19.566 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:40:19.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.799+0000 7f2201ab6640 1 -- 172.21.15.90:0/1574554188 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f21fc102af0 msgr2=0x7f21fc102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:19.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.799+0000 7f2201ab6640 1 --2- 172.21.15.90:0/1574554188 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f21fc102af0 0x7f21fc102f00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f21f0009860 tx=0x7f21f002a340 comp rx=0 tx=0).stop 2024-04-25T11:40:19.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.799+0000 7f2201ab6640 1 -- 172.21.15.90:0/1574554188 shutdown_connections 2024-04-25T11:40:19.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.799+0000 7f2201ab6640 1 --2- 172.21.15.90:0/1574554188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21fc10d380 0x7f21fc10f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:19.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.799+0000 7f2201ab6640 1 --2- 172.21.15.90:0/1574554188 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f21fc102af0 0x7f21fc102f00 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:19.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.799+0000 7f2201ab6640 1 --2- 172.21.15.90:0/1574554188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21fc108a20 0x7f21fc1025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:19.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.799+0000 7f2201ab6640 1 -- 172.21.15.90:0/1574554188 >> 172.21.15.90:0/1574554188 conn(0x7f21fc078100 msgr2=0x7f21fc078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:19.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.799+0000 7f2201ab6640 1 -- 172.21.15.90:0/1574554188 shutdown_connections 2024-04-25T11:40:19.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.799+0000 7f2201ab6640 1 -- 172.21.15.90:0/1574554188 wait complete. 2024-04-25T11:40:19.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.800+0000 7f2201ab6640 1 Processor -- start 2024-04-25T11:40:19.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.800+0000 7f2201ab6640 1 -- start start 2024-04-25T11:40:19.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.800+0000 7f2201ab6640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f21fc102af0 0x7f21fc199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:19.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.800+0000 7f2201ab6640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21fc108a20 0x7f21fc199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:19.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.800+0000 7f2201ab6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21fc10d380 0x7f21fc19de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:19.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.800+0000 7f2201ab6640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc112be0 con 0x7f21fc108a20 2024-04-25T11:40:19.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.800+0000 7f2201ab6640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc112a60 con 0x7f21fc102af0 2024-04-25T11:40:19.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.800+0000 7f2201ab6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc112d60 con 0x7f21fc10d380 2024-04-25T11:40:19.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.801+0000 7f21fb7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f21fc102af0 0x7f21fc199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:19.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.801+0000 7f21faffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21fc108a20 0x7f21fc199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:19.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.801+0000 7f21faffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21fc108a20 0x7f21fc199b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48594/0 (socket says 172.21.15.90:48594) 2024-04-25T11:40:19.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.801+0000 7f21fbfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21fc10d380 0x7f21fc19de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:19.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.801+0000 7f21faffd640 1 -- 172.21.15.90:0/2155315508 learned_addr learned my addr 172.21.15.90:0/2155315508 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.801+0000 7f21fb7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f21fc102af0 0x7f21fc199620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:36456/0 (socket says 172.21.15.90:36456) 2024-04-25T11:40:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.801+0000 7f21fb7fe640 1 -- 172.21.15.90:0/2155315508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21fc10d380 msgr2=0x7f21fc19de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.801+0000 7f21fb7fe640 1 --2- 172.21.15.90:0/2155315508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21fc10d380 0x7f21fc19de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.801+0000 7f21fb7fe640 1 -- 172.21.15.90:0/2155315508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21fc108a20 msgr2=0x7f21fc199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.801+0000 7f21fb7fe640 1 --2- 172.21.15.90:0/2155315508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21fc108a20 0x7f21fc199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.801+0000 7f21fb7fe640 1 -- 172.21.15.90:0/2155315508 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21fc19e480 con 0x7f21fc102af0 2024-04-25T11:40:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.801+0000 7f21faffd640 1 --2- 172.21.15.90:0/2155315508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21fc108a20 0x7f21fc199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:40:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.801+0000 7f21fb7fe640 1 --2- 172.21.15.90:0/2155315508 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f21fc102af0 0x7f21fc199620 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f21fc103a20 tx=0x7f21ec00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.802+0000 7f21f8ff9640 1 -- 172.21.15.90:0/2155315508 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ec00ccd0 con 0x7f21fc102af0 2024-04-25T11:40:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.802+0000 7f2201ab6640 1 -- 172.21.15.90:0/2155315508 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21fc19e770 con 0x7f21fc102af0 2024-04-25T11:40:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.802+0000 7f21f8ff9640 1 -- 172.21.15.90:0/2155315508 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21ec0035f0 con 0x7f21fc102af0 2024-04-25T11:40:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.802+0000 7f21f8ff9640 1 -- 172.21.15.90:0/2155315508 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ec010690 con 0x7f21fc102af0 2024-04-25T11:40:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.802+0000 7f2201ab6640 1 -- 172.21.15.90:0/2155315508 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21fc1a60f0 con 0x7f21fc102af0 2024-04-25T11:40:19.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.804+0000 7f2208969740 1 Processor -- start 2024-04-25T11:40:19.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.804+0000 7f2208969740 1 -- start start 2024-04-25T11:40:19.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.805+0000 7f2208969740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d4fe622390 0x55d4fe9c2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:19.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.805+0000 7f2208969740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d4fe5f94d0 0x55d4fe9c2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:19.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.805+0000 7f2208969740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4fe6016b0 0x55d4fe9d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:19.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.805+0000 7f2208969740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d4fe928d80 con 0x55d4fe6016b0 2024-04-25T11:40:19.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.805+0000 7f2208969740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55d4fe8e28b0 con 0x55d4fe622390 2024-04-25T11:40:19.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.805+0000 7f2208969740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d4fe924d40 con 0x55d4fe5f94d0 2024-04-25T11:40:19.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.805+0000 7f21faffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4fe6016b0 0x55d4fe9d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:19.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.805+0000 7f21fbfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d4fe622390 0x55d4fe9c2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:19.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.805+0000 7f21fbfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d4fe622390 0x55d4fe9c2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:36464/0 (socket says 172.21.15.90:36464) 2024-04-25T11:40:19.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.805+0000 7f21faffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4fe6016b0 0x55d4fe9d1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48598/0 (socket says 172.21.15.90:48598) 2024-04-25T11:40:19.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.805+0000 7f21fbfff640 1 -- 172.21.15.90:0/3485877282 learned_addr learned my addr 172.21.15.90:0/3485877282 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:19.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.805+0000 7f21fbfff640 1 -- 172.21.15.90:0/3485877282 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d4fe5f94d0 msgr2=0x55d4fe9c2e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:40:19.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.805+0000 7f21fbfff640 1 --2- 172.21.15.90:0/3485877282 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d4fe5f94d0 0x55d4fe9c2e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:19.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.805+0000 7f21fbfff640 1 -- 172.21.15.90:0/3485877282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4fe6016b0 msgr2=0x55d4fe9d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:19.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.805+0000 7f21fbfff640 1 --2- 172.21.15.90:0/3485877282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4fe6016b0 0x55d4fe9d1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:19.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.805+0000 7f21fbfff640 1 -- 172.21.15.90:0/3485877282 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4fe8b3a40 con 0x55d4fe622390 2024-04-25T11:40:19.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.805+0000 7f21faffd640 1 --2- 172.21.15.90:0/3485877282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4fe6016b0 0x55d4fe9d1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:40:19.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.805+0000 7f21f8ff9640 1 -- 172.21.15.90:0/2155315508 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f21ec004020 con 0x7f21fc102af0 2024-04-25T11:40:19.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.805+0000 7f21fbfff640 1 --2- 172.21.15.90:0/3485877282 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d4fe622390 0x55d4fe9c2950 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f21f4009fd0 tx=0x7f21f40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5478e9aa4a872808 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:19.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.806+0000 7f21e1ffb640 1 -- 172.21.15.90:0/3485877282 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f4004d20 con 0x55d4fe622390 2024-04-25T11:40:19.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.806+0000 7f21e1ffb640 1 -- 172.21.15.90:0/3485877282 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21f4004050 con 0x55d4fe622390 2024-04-25T11:40:19.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.806+0000 7f21e1ffb640 1 -- 172.21.15.90:0/3485877282 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f4005280 con 0x55d4fe622390 2024-04-25T11:40:19.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.806+0000 7f21f8ff9640 1 --2- 172.21.15.90:0/2155315508 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f21e40afc60 0x7f21e40b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:19.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.807+0000 7f21fb7fe640 1 --2- 172.21.15.90:0/2155315508 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f21e40afc60 0x7f21e40b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:19.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.807+0000 7f21f8ff9640 1 -- 172.21.15.90:0/2155315508 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f21ec0df2e0 con 0x7f21fc102af0 2024-04-25T11:40:19.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.807+0000 7f2208969740 1 -- 172.21.15.90:0/3485877282 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d4fe622390 msgr2=0x55d4fe9c2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:19.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.807+0000 7f2208969740 1 --2- 172.21.15.90:0/3485877282 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d4fe622390 0x55d4fe9c2950 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f21f4009fd0 tx=0x7f21f40118d0 comp rx=0 tx=0).stop 2024-04-25T11:40:19.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.807+0000 7f2208969740 1 -- 172.21.15.90:0/3485877282 shutdown_connections 2024-04-25T11:40:19.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.807+0000 7f2208969740 1 --2- 172.21.15.90:0/3485877282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4fe6016b0 0x55d4fe9d1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:19.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.807+0000 7f2208969740 1 --2- 172.21.15.90:0/3485877282 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d4fe5f94d0 0x55d4fe9c2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:19.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.807+0000 7f2208969740 1 --2- 172.21.15.90:0/3485877282 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d4fe622390 0x55d4fe9c2950 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:19.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.807+0000 7f2208969740 1 -- 172.21.15.90:0/3485877282 >> 172.21.15.90:0/3485877282 conn(0x55d4fe6164e0 msgr2=0x55d4fe9bf210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:19.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.807+0000 7f2208969740 1 -- 172.21.15.90:0/3485877282 shutdown_connections 2024-04-25T11:40:19.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.807+0000 7f2208969740 1 -- 172.21.15.90:0/3485877282 wait complete. 2024-04-25T11:40:19.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.807+0000 7f21fb7fe640 1 --2- 172.21.15.90:0/2155315508 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f21e40afc60 0x7f21e40b2170 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x55d4fe9c02c0 tx=0x7f21ec007d80 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:19.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.808+0000 7f2208969740 1 -- start start 2024-04-25T11:40:19.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.808+0000 7f2208969740 1 Processor -- start 2024-04-25T11:40:19.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.809+0000 7f2208969740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d4fe5f94d0 0x55d4fe9cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:19.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.809+0000 7f2208969740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4fe6016b0 0x55d4fe9cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:19.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.809+0000 7f2208969740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d4fe622390 0x55d4fe9db730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:19.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.809+0000 7f2208969740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d4fe9d21f0 con 0x55d4fe6016b0 2024-04-25T11:40:19.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.809+0000 7f2208969740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55d4fe8afd10 con 0x55d4fe622390 2024-04-25T11:40:19.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.809+0000 7f2208969740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d4fe9d2370 con 0x55d4fe5f94d0 2024-04-25T11:40:19.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.809+0000 7f2208969740 10 client.0 _ll_register_callbacks cb 0x55d4fe8dc210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:19.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.809+0000 7f21fbfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d4fe5f94d0 0x55d4fe9cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:19.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.809+0000 7f21fbfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d4fe5f94d0 0x55d4fe9cf780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:40862/0 (socket says 172.21.15.90:40862) 2024-04-25T11:40:19.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.809+0000 7f21fbfff640 1 -- 172.21.15.90:0/2353157412 learned_addr learned my addr 172.21.15.90:0/2353157412 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:19.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.809+0000 7f21faffd640 1 --2- 172.21.15.90:0/2353157412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4fe6016b0 0x55d4fe9cfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:19.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.809+0000 7f21fbfff640 1 --2- 172.21.15.90:0/2353157412 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d4fe622390 0x55d4fe9db730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:19.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.809+0000 7f21fbfff640 1 -- 172.21.15.90:0/2353157412 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d4fe622390 msgr2=0x55d4fe9db730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:19.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.809+0000 7f21fbfff640 1 --2- 172.21.15.90:0/2353157412 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d4fe622390 0x55d4fe9db730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:19.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.809+0000 7f21fbfff640 1 -- 172.21.15.90:0/2353157412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4fe6016b0 msgr2=0x55d4fe9cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:19.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.809+0000 7f21fbfff640 1 --2- 172.21.15.90:0/2353157412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4fe6016b0 0x55d4fe9cfcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:19.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.809+0000 7f21fbfff640 1 -- 172.21.15.90:0/2353157412 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4fe7047d0 con 0x55d4fe5f94d0 2024-04-25T11:40:19.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.810+0000 7f21fbfff640 1 --2- 172.21.15.90:0/2353157412 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d4fe5f94d0 0x55d4fe9cf780 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f21f4005fd0 tx=0x7f21f4004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ab0d61d34b0e40 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:19.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.810+0000 7f21daffd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:40:19.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.810+0000 7f21daffd640 1 -- 172.21.15.90:0/2353157412 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f402d060 con 0x55d4fe5f94d0 2024-04-25T11:40:19.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.810+0000 7f21daffd640 1 -- 172.21.15.90:0/2353157412 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f21f400f4e0 con 0x55d4fe5f94d0 2024-04-25T11:40:19.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.810+0000 7f2208969740 1 -- 172.21.15.90:0/2353157412 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21f4004050 con 0x55d4fe5f94d0 2024-04-25T11:40:19.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.810+0000 7f21daffd640 1 -- 172.21.15.90:0/2353157412 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f21f400f6e0 con 0x55d4fe5f94d0 2024-04-25T11:40:19.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.810+0000 7f2208969740 10 client.26194 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:19.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.810+0000 7f2208969740 1 -- 172.21.15.90:0/2353157412 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d4fe63a3f0 con 0x55d4fe5f94d0 2024-04-25T11:40:19.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.810+0000 7f21daffd640 1 -- 172.21.15.90:0/2353157412 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21f4004050 con 0x55d4fe5f94d0 2024-04-25T11:40:19.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.811+0000 7f21daffd640 1 -- 172.21.15.90:0/2353157412 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f21f4025040 con 0x55d4fe5f94d0 2024-04-25T11:40:19.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.812+0000 7f2208969740 10 client.26194 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:19.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.812+0000 7f2208969740 10 client.26194 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:40:19.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.812+0000 7f2208969740 20 client.26194 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:19.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.812+0000 7f2208969740 1 --2- 172.21.15.90:0/2353157412 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d4fe5e0f60 0x55d4fe9e17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:19.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.812+0000 7f2208969740 4 client.26194 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:40:19.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.812+0000 7f2208969740 1 -- 172.21.15.90:0/2353157412 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d4fe936990 con 0x55d4fe5e0f60 2024-04-25T11:40:19.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.812+0000 7f21faffd640 1 --2- 172.21.15.90:0/2353157412 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d4fe5e0f60 0x55d4fe9e17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:19.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.812+0000 7f21faffd640 1 --2- 172.21.15.90:0/2353157412 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d4fe5e0f60 0x55d4fe9e17a0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37f164e03266c714 server_cookie=a899ddcdf8fd41ae in_seq=0 out_seq=0 2024-04-25T11:40:19.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:19.813+0000 7f21daffd640 10 client.26194 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:20.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:20 smithi138 ceph-mon[32328]: pgmap v160: 97 pgs: 97 active+clean; 40 GiB data, 123 GiB used, 950 GiB / 1.0 TiB avail; 448 KiB/s rd, 272 MiB/s wr, 1.07k op/s 2024-04-25T11:40:20.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:20 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:40:20.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:20 smithi156 ceph-mon[33045]: pgmap v160: 97 pgs: 97 active+clean; 40 GiB data, 123 GiB used, 950 GiB / 1.0 TiB avail; 448 KiB/s rd, 272 MiB/s wr, 1.07k op/s 2024-04-25T11:40:20.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:20 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:40:20.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:20 smithi090 ceph-mon[29575]: pgmap v160: 97 pgs: 97 active+clean; 40 GiB data, 123 GiB used, 950 GiB / 1.0 TiB avail; 448 KiB/s rd, 272 MiB/s wr, 1.07k op/s 2024-04-25T11:40:20.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:20 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:40:21.198 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.210+0000 7f7c55ffb640 1 -- 172.21.15.90:0/1637700066 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d794573ee0 con 0x55d7941fa1c0 2024-04-25T11:40:21.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.210+0000 7f7c55ffb640 10 client.26246 handle_command_reply: tid=0 2024-04-25T11:40:21.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.210+0000 7f7c7b9e4740 1 client.26246 shutdown 2024-04-25T11:40:21.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.210+0000 7f7c7b9e4740 1 -- 172.21.15.90:0/1637700066 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7942454d0 msgr2=0x55d7946253b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:21.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.210+0000 7f7c7b9e4740 1 --2- 172.21.15.90:0/1637700066 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7942454d0 0x55d7946253b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7c640a7240 tx=0x7f7c640ab1e0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.210+0000 7f7c7b9e4740 1 -- 172.21.15.90:0/1637700066 shutdown_connections 2024-04-25T11:40:21.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.210+0000 7f7c7b9e4740 1 --2- 172.21.15.90:0/1637700066 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d7941fa1c0 0x55d794695330 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.210+0000 7f7c7b9e4740 1 --2- 172.21.15.90:0/1637700066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7942661b0 0x55d794620320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.210+0000 7f7c7b9e4740 1 --2- 172.21.15.90:0/1637700066 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7942454d0 0x55d7946253b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.210+0000 7f7c7b9e4740 1 --2- 172.21.15.90:0/1637700066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d79423d2f0 0x55d794624e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.210+0000 7f7c7b9e4740 1 -- 172.21.15.90:0/1637700066 >> 172.21.15.90:0/1637700066 conn(0x55d79425a300 msgr2=0x55d79460e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.210+0000 7f7c55ffb640 0 client.26246 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.210+0000 7f7c7b9e4740 1 -- 172.21.15.90:0/1637700066 shutdown_connections 2024-04-25T11:40:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.211+0000 7f7c7b9e4740 1 -- 172.21.15.90:0/1637700066 wait complete. 2024-04-25T11:40:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.211+0000 7f7c7b9e4740 20 client.26246 trim_cache size 0 max 16384 2024-04-25T11:40:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:40:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:40:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:40:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:40:21.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.212+0000 7f21daffd640 1 -- 172.21.15.90:0/2353157412 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d4fe936990 con 0x55d4fe5e0f60 2024-04-25T11:40:21.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.212+0000 7f21daffd640 10 client.26194 handle_command_reply: tid=0 2024-04-25T11:40:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.212+0000 7f7c74b34640 1 -- 172.21.15.90:0/4226948219 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f7c580afc50 msgr2=0x7f7c580b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.212+0000 7f7c74b34640 1 --2- 172.21.15.90:0/4226948219 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f7c580afc50 0x7f7c580b2160 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x55d79460f6b0 tx=0x7f7c60005f90 comp rx=0 tx=0).stop 2024-04-25T11:40:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.212+0000 7f7c74b34640 1 -- 172.21.15.90:0/4226948219 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7c70108a60 msgr2=0x7f7c701a2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.212+0000 7f7c74b34640 1 --2- 172.21.15.90:0/4226948219 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7c70108a60 0x7f7c701a2bb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7c6400e980 tx=0x7f7c6400ee40 comp rx=0 tx=0).stop 2024-04-25T11:40:21.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.213+0000 7f2208969740 1 client.26194 shutdown 2024-04-25T11:40:21.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.213+0000 7f2208969740 1 -- 172.21.15.90:0/2353157412 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d4fe5f94d0 msgr2=0x55d4fe9cf780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:21.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.213+0000 7f2208969740 1 --2- 172.21.15.90:0/2353157412 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d4fe5f94d0 0x55d4fe9cf780 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f21f4005fd0 tx=0x7f21f4004950 comp rx=0 tx=0).stop 2024-04-25T11:40:21.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.213+0000 7f2208969740 1 -- 172.21.15.90:0/2353157412 shutdown_connections 2024-04-25T11:40:21.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.213+0000 7f2208969740 1 --2- 172.21.15.90:0/2353157412 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d4fe5e0f60 0x55d4fe9e17a0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.213+0000 7f2208969740 1 --2- 172.21.15.90:0/2353157412 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d4fe622390 0x55d4fe9db730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.213+0000 7f2208969740 1 --2- 172.21.15.90:0/2353157412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4fe6016b0 0x55d4fe9cfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.213+0000 7f2208969740 1 --2- 172.21.15.90:0/2353157412 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d4fe5f94d0 0x55d4fe9cf780 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.213+0000 7f2208969740 1 -- 172.21.15.90:0/2353157412 >> 172.21.15.90:0/2353157412 conn(0x55d4fe6164e0 msgr2=0x55d4fe9c4780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:21.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.213+0000 7f21daffd640 0 client.26194 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:21.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.213+0000 7f2208969740 1 -- 172.21.15.90:0/2353157412 shutdown_connections 2024-04-25T11:40:21.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.213+0000 7f2208969740 1 -- 172.21.15.90:0/2353157412 wait complete. 2024-04-25T11:40:21.203 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.213+0000 7f7c74b34640 1 -- 172.21.15.90:0/4226948219 shutdown_connections 2024-04-25T11:40:21.203 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.213+0000 7f7c74b34640 1 --2- 172.21.15.90:0/4226948219 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f7c580afc50 0x7f7c580b2160 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.203 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.213+0000 7f7c74b34640 1 --2- 172.21.15.90:0/4226948219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c70108f00 0x7f7c7019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.204 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.213+0000 7f7c74b34640 1 --2- 172.21.15.90:0/4226948219 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7c70108a60 0x7f7c701a2bb0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.204 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.213+0000 7f7c74b34640 1 --2- 172.21.15.90:0/4226948219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c701031c0 0x7f7c701a2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.204 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.213+0000 7f7c74b34640 1 -- 172.21.15.90:0/4226948219 >> 172.21.15.90:0/4226948219 conn(0x7f7c700781b0 msgr2=0x7f7c7006a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:21.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.214+0000 7f2208969740 20 client.26194 trim_cache size 0 max 16384 2024-04-25T11:40:21.204 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.214+0000 7f7c74b34640 1 -- 172.21.15.90:0/4226948219 shutdown_connections 2024-04-25T11:40:21.204 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:21.214+0000 7f7c74b34640 1 -- 172.21.15.90:0/4226948219 wait complete. 2024-04-25T11:40:21.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.284+0000 7f2208969740 1 Processor -- start 2024-04-25T11:40:21.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.285+0000 7f2208969740 1 -- start start 2024-04-25T11:40:21.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.286+0000 7f2208969740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d4fe5f94d0 0x55d4fe9d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:21.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.286+0000 7f2208969740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4fe6016b0 0x55d4fe9d18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:21.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.286+0000 7f2208969740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d4fe622390 0x55d4fe9be9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:21.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.286+0000 7f2208969740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc1adac0 con 0x55d4fe6016b0 2024-04-25T11:40:21.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.286+0000 7f2208969740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc1af600 con 0x55d4fe622390 2024-04-25T11:40:21.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.286+0000 7f2208969740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc1ae190 con 0x55d4fe5f94d0 2024-04-25T11:40:21.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.295+0000 7f21fbfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d4fe5f94d0 0x55d4fe9d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:21.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.295+0000 7f21fbfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d4fe622390 0x55d4fe9be9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:21.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.295+0000 7f21faffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4fe6016b0 0x55d4fe9d18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:21.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.295+0000 7f21faffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4fe6016b0 0x55d4fe9d18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48630/0 (socket says 172.21.15.90:48630) 2024-04-25T11:40:21.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.295+0000 7f21faffd640 1 -- 172.21.15.90:0/2010600067 learned_addr learned my addr 172.21.15.90:0/2010600067 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:21.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.302+0000 7f21faffd640 1 -- 172.21.15.90:0/2010600067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d4fe5f94d0 msgr2=0x55d4fe9d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:21.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.302+0000 7f21faffd640 1 --2- 172.21.15.90:0/2010600067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d4fe5f94d0 0x55d4fe9d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.302+0000 7f21faffd640 1 -- 172.21.15.90:0/2010600067 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d4fe622390 msgr2=0x55d4fe9be9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:21.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.302+0000 7f21faffd640 1 --2- 172.21.15.90:0/2010600067 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d4fe622390 0x55d4fe9be9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.302+0000 7f21faffd640 1 -- 172.21.15.90:0/2010600067 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4fe5fa0f0 con 0x55d4fe6016b0 2024-04-25T11:40:21.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.305+0000 7f21faffd640 1 --2- 172.21.15.90:0/2010600067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4fe6016b0 0x55d4fe9d18e0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f21f0035040 tx=0x7f21f0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1807a42bdb385305 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:21.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.307+0000 7f21e17fa640 1 -- 172.21.15.90:0/2010600067 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f004b070 con 0x55d4fe6016b0 2024-04-25T11:40:21.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.307+0000 7f21e17fa640 1 -- 172.21.15.90:0/2010600067 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21f0030e20 con 0x55d4fe6016b0 2024-04-25T11:40:21.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.307+0000 7f21e17fa640 1 -- 172.21.15.90:0/2010600067 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f0037040 con 0x55d4fe6016b0 2024-04-25T11:40:21.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.311+0000 7f2208969740 1 -- 172.21.15.90:0/2010600067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4fe6016b0 msgr2=0x55d4fe9d18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:21.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.311+0000 7f2208969740 1 --2- 172.21.15.90:0/2010600067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4fe6016b0 0x55d4fe9d18e0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f21f0035040 tx=0x7f21f0005f90 comp rx=0 tx=0).stop 2024-04-25T11:40:21.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.316+0000 7f2208969740 1 -- 172.21.15.90:0/2010600067 shutdown_connections 2024-04-25T11:40:21.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.316+0000 7f2208969740 1 --2- 172.21.15.90:0/2010600067 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d4fe622390 0x55d4fe9be9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.316+0000 7f2208969740 1 --2- 172.21.15.90:0/2010600067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4fe6016b0 0x55d4fe9d18e0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.316+0000 7f2208969740 1 --2- 172.21.15.90:0/2010600067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d4fe5f94d0 0x55d4fe9d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.316+0000 7f2208969740 1 -- 172.21.15.90:0/2010600067 >> 172.21.15.90:0/2010600067 conn(0x55d4fe6164e0 msgr2=0x55d4fe9d0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:21.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.321+0000 7f2208969740 1 -- 172.21.15.90:0/2010600067 shutdown_connections 2024-04-25T11:40:21.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.321+0000 7f2208969740 1 -- 172.21.15.90:0/2010600067 wait complete. 2024-04-25T11:40:21.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.322+0000 7f2208969740 1 -- start start 2024-04-25T11:40:21.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.322+0000 7f2208969740 1 Processor -- start 2024-04-25T11:40:21.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.323+0000 7f2208969740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d4fe5f94d0 0x55d4fea4ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:21.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.324+0000 7f2208969740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d4fe6016b0 0x55d4fea48080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:21.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.324+0000 7f2208969740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4fe622390 0x55d4fea48600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:21.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.324+0000 7f2208969740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc1aef30 con 0x55d4fe622390 2024-04-25T11:40:21.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.324+0000 7f2208969740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc1ae860 con 0x55d4fe6016b0 2024-04-25T11:40:21.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.324+0000 7f2208969740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc1abaa0 con 0x55d4fe5f94d0 2024-04-25T11:40:21.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.324+0000 7f2208969740 10 client.0 _ll_register_callbacks cb 0x7f21f40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:21.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.329+0000 7f21fbfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4fe622390 0x55d4fea48600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:21.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.329+0000 7f21fbfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4fe622390 0x55d4fea48600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48640/0 (socket says 172.21.15.90:48640) 2024-04-25T11:40:21.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.329+0000 7f21fbfff640 1 -- 172.21.15.90:0/4122480880 learned_addr learned my addr 172.21.15.90:0/4122480880 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:21.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.329+0000 7f21fbfff640 1 --2- 172.21.15.90:0/4122480880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d4fe5f94d0 0x55d4fea4ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:21.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.329+0000 7f21faffd640 1 --2- 172.21.15.90:0/4122480880 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d4fe6016b0 0x55d4fea48080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:21.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.333+0000 7f21fbfff640 1 -- 172.21.15.90:0/4122480880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d4fe5f94d0 msgr2=0x55d4fea4ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:21.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.333+0000 7f21fbfff640 1 --2- 172.21.15.90:0/4122480880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d4fe5f94d0 0x55d4fea4ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.333+0000 7f21fbfff640 1 -- 172.21.15.90:0/4122480880 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d4fe6016b0 msgr2=0x55d4fea48080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:21.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.333+0000 7f21fbfff640 1 --2- 172.21.15.90:0/4122480880 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d4fe6016b0 0x55d4fea48080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.333+0000 7f21fbfff640 1 -- 172.21.15.90:0/4122480880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4fe8e94b0 con 0x55d4fe622390 2024-04-25T11:40:21.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.334+0000 7f21fbfff640 1 --2- 172.21.15.90:0/4122480880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4fe622390 0x55d4fea48600 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f21f4014040 tx=0x7f21f4004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2237ddba7acb99 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:21.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.334+0000 7f21daffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:40:21.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.334+0000 7f21daffd640 1 -- 172.21.15.90:0/4122480880 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f4040060 con 0x55d4fe622390 2024-04-25T11:40:21.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.334+0000 7f21daffd640 1 -- 172.21.15.90:0/4122480880 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f21f4019640 con 0x55d4fe622390 2024-04-25T11:40:21.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.334+0000 7f2208969740 1 -- 172.21.15.90:0/4122480880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21f0030e20 con 0x55d4fe622390 2024-04-25T11:40:21.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.335+0000 7f21daffd640 1 -- 172.21.15.90:0/4122480880 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f21f402d040 con 0x55d4fe622390 2024-04-25T11:40:21.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.335+0000 7f2208969740 10 client.17112 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:21.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.336+0000 7f2208969740 1 -- 172.21.15.90:0/4122480880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d4fe9e4500 con 0x55d4fe622390 2024-04-25T11:40:21.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.336+0000 7f21daffd640 1 -- 172.21.15.90:0/4122480880 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21f4003b90 con 0x55d4fe622390 2024-04-25T11:40:21.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.336+0000 7f21daffd640 1 -- 172.21.15.90:0/4122480880 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f21f403b6a0 con 0x55d4fe622390 2024-04-25T11:40:21.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.336+0000 7f2208969740 10 client.17112 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:21.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.336+0000 7f2208969740 10 client.17112 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:40:21.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.336+0000 7f2208969740 20 client.17112 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:21.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.337+0000 7f2208969740 1 --2- 172.21.15.90:0/4122480880 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d4fe5b63a0 0x55d4fe9e4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:21.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.337+0000 7f2208969740 4 client.17112 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:40:21.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.337+0000 7f2208969740 1 -- 172.21.15.90:0/4122480880 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f21f0030e20 con 0x55d4fe5b63a0 2024-04-25T11:40:21.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.338+0000 7f21faffd640 1 --2- 172.21.15.90:0/4122480880 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d4fe5b63a0 0x55d4fe9e4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:21.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.341+0000 7f21faffd640 1 --2- 172.21.15.90:0/4122480880 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d4fe5b63a0 0x55d4fe9e4c00 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b79167f27847fa9 server_cookie=5dbe60c1848c6330 in_seq=0 out_seq=0 2024-04-25T11:40:21.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.341+0000 7f21daffd640 10 client.17112 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:21.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.344+0000 7f21daffd640 1 -- 172.21.15.90:0/4122480880 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f21f0030e20 con 0x55d4fe5b63a0 2024-04-25T11:40:21.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.344+0000 7f21daffd640 10 client.17112 handle_command_reply: tid=0 2024-04-25T11:40:21.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.344+0000 7f2208969740 1 client.17112 shutdown 2024-04-25T11:40:21.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.344+0000 7f2208969740 1 -- 172.21.15.90:0/4122480880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4fe622390 msgr2=0x55d4fea48600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:21.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.344+0000 7f2208969740 1 --2- 172.21.15.90:0/4122480880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4fe622390 0x55d4fea48600 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f21f4014040 tx=0x7f21f4004950 comp rx=0 tx=0).stop 2024-04-25T11:40:21.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.344+0000 7f2208969740 1 -- 172.21.15.90:0/4122480880 shutdown_connections 2024-04-25T11:40:21.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.344+0000 7f2208969740 1 --2- 172.21.15.90:0/4122480880 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d4fe5b63a0 0x55d4fe9e4c00 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.345+0000 7f2208969740 1 --2- 172.21.15.90:0/4122480880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4fe622390 0x55d4fea48600 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.345+0000 7f2208969740 1 --2- 172.21.15.90:0/4122480880 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d4fe6016b0 0x55d4fea48080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.345+0000 7f2208969740 1 --2- 172.21.15.90:0/4122480880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d4fe5f94d0 0x55d4fea4ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.345+0000 7f2208969740 1 -- 172.21.15.90:0/4122480880 >> 172.21.15.90:0/4122480880 conn(0x55d4fe6164e0 msgr2=0x55d4fe9ce4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:21.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.345+0000 7f21daffd640 0 client.17112 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:21.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.345+0000 7f2208969740 1 -- 172.21.15.90:0/4122480880 shutdown_connections 2024-04-25T11:40:21.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.346+0000 7f2208969740 1 -- 172.21.15.90:0/4122480880 wait complete. 2024-04-25T11:40:21.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.346+0000 7f2208969740 20 client.17112 trim_cache size 0 max 16384 2024-04-25T11:40:21.334 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:40:21.334 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-25T11:40:21.334 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-25T11:40:21.334 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:40:21.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.347+0000 7f2201ab6640 1 -- 172.21.15.90:0/2155315508 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f21e40afc60 msgr2=0x7f21e40b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:21.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.347+0000 7f2201ab6640 1 --2- 172.21.15.90:0/2155315508 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f21e40afc60 0x7f21e40b2170 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x55d4fe9c02c0 tx=0x7f21ec007d80 comp rx=0 tx=0).stop 2024-04-25T11:40:21.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.347+0000 7f2201ab6640 1 -- 172.21.15.90:0/2155315508 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f21fc102af0 msgr2=0x7f21fc199620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:21.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.347+0000 7f2201ab6640 1 --2- 172.21.15.90:0/2155315508 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f21fc102af0 0x7f21fc199620 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f21fc103a20 tx=0x7f21ec00ee50 comp rx=0 tx=0).stop 2024-04-25T11:40:21.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.348+0000 7f2201ab6640 1 -- 172.21.15.90:0/2155315508 shutdown_connections 2024-04-25T11:40:21.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.348+0000 7f2201ab6640 1 --2- 172.21.15.90:0/2155315508 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f21e40afc60 0x7f21e40b2170 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.348+0000 7f2201ab6640 1 --2- 172.21.15.90:0/2155315508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21fc10d380 0x7f21fc19de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.348+0000 7f2201ab6640 1 --2- 172.21.15.90:0/2155315508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21fc108a20 0x7f21fc199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.348+0000 7f2201ab6640 1 --2- 172.21.15.90:0/2155315508 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f21fc102af0 0x7f21fc199620 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.348+0000 7f2201ab6640 1 -- 172.21.15.90:0/2155315508 >> 172.21.15.90:0/2155315508 conn(0x7f21fc078100 msgr2=0x7f21fc1040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:21.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.348+0000 7f2201ab6640 1 -- 172.21.15.90:0/2155315508 shutdown_connections 2024-04-25T11:40:21.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:21 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:40:21.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:21 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:40:21.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:21 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:40:21.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.348+0000 7f2201ab6640 1 -- 172.21.15.90:0/2155315508 wait complete. 2024-04-25T11:40:21.362 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:21 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:40:21.363 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:21 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:40:21.363 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:21 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:40:21.364 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T11:40:21.364 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:40:21.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:21 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:40:21.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:21 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:40:21.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:21 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:40:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.632+0000 7f84eaffd640 1 -- 172.21.15.90:0/3500016175 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e4031cd0 con 0x7f84ec111c80 2024-04-25T11:40:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.633+0000 7f84f1391640 1 -- 172.21.15.90:0/3500016175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84ec111c80 msgr2=0x7f84ec1140a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.633+0000 7f84f1391640 1 --2- 172.21.15.90:0/3500016175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84ec111c80 0x7f84ec1140a0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f84e4009ca0 tx=0x7f84e402a8e0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.633+0000 7f84f1391640 1 -- 172.21.15.90:0/3500016175 shutdown_connections 2024-04-25T11:40:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.633+0000 7f84f1391640 1 --2- 172.21.15.90:0/3500016175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84ec1145e0 0x7f84ec116a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.633+0000 7f84f1391640 1 --2- 172.21.15.90:0/3500016175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84ec111c80 0x7f84ec1140a0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.633+0000 7f84f1391640 1 --2- 172.21.15.90:0/3500016175 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84ec1018d0 0x7f84ec101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.633+0000 7f84f1391640 1 -- 172.21.15.90:0/3500016175 >> 172.21.15.90:0/3500016175 conn(0x7f84ec0fd200 msgr2=0x7f84ec0ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:21.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.633+0000 7f84f1391640 1 -- 172.21.15.90:0/3500016175 shutdown_connections 2024-04-25T11:40:21.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.633+0000 7f84f1391640 1 -- 172.21.15.90:0/3500016175 wait complete. 2024-04-25T11:40:21.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.634+0000 7f84f1391640 1 Processor -- start 2024-04-25T11:40:21.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.635+0000 7f84f1391640 1 -- start start 2024-04-25T11:40:21.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.635+0000 7f84f1391640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84ec1018d0 0x7f84ec10a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:21.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.635+0000 7f84f1391640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84ec111c80 0x7f84ec10ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:21.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.635+0000 7f84f1391640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84ec1145e0 0x7f84ec10b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:21.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.635+0000 7f84f1391640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec11a270 con 0x7f84ec1018d0 2024-04-25T11:40:21.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.635+0000 7f84f1391640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec11a0f0 con 0x7f84ec111c80 2024-04-25T11:40:21.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.635+0000 7f84f1391640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec11a3f0 con 0x7f84ec1145e0 2024-04-25T11:40:21.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.635+0000 7f84ebfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84ec1018d0 0x7f84ec10a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:21.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.635+0000 7f84ebfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84ec1018d0 0x7f84ec10a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60762/0 (socket says 172.21.15.90:60762) 2024-04-25T11:40:21.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.636+0000 7f84ebfff640 1 -- 172.21.15.90:0/3336889429 learned_addr learned my addr 172.21.15.90:0/3336889429 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:21.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.636+0000 7f84ebfff640 1 -- 172.21.15.90:0/3336889429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84ec1145e0 msgr2=0x7f84ec10b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:21.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.636+0000 7f84ebfff640 1 --2- 172.21.15.90:0/3336889429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84ec1145e0 0x7f84ec10b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.636+0000 7f84ebfff640 1 -- 172.21.15.90:0/3336889429 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84ec111c80 msgr2=0x7f84ec10ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:21.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.636+0000 7f84ebfff640 1 --2- 172.21.15.90:0/3336889429 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84ec111c80 0x7f84ec10ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:21.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.636+0000 7f84ebfff640 1 -- 172.21.15.90:0/3336889429 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84ec10f2e0 con 0x7f84ec1018d0 2024-04-25T11:40:21.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.636+0000 7f84ebfff640 1 --2- 172.21.15.90:0/3336889429 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84ec1018d0 0x7f84ec10a8e0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00b450 tx=0x7f84dc00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:21.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.637+0000 7f84e97fa640 1 -- 172.21.15.90:0/3336889429 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84dc015020 con 0x7f84ec1018d0 2024-04-25T11:40:21.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.637+0000 7f84f1391640 1 -- 172.21.15.90:0/3336889429 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84ec10f570 con 0x7f84ec1018d0 2024-04-25T11:40:21.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.637+0000 7f84f1391640 1 -- 172.21.15.90:0/3336889429 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84ec10fb10 con 0x7f84ec1018d0 2024-04-25T11:40:21.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.639+0000 7f84e97fa640 1 -- 172.21.15.90:0/3336889429 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84dc011d40 con 0x7f84ec1018d0 2024-04-25T11:40:21.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.639+0000 7f84e97fa640 1 -- 172.21.15.90:0/3336889429 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84dc00f340 con 0x7f84ec1018d0 2024-04-25T11:40:21.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.640+0000 7f84f1391640 1 -- 172.21.15.90:0/3336889429 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84ec102a10 con 0x7f84ec1018d0 2024-04-25T11:40:21.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.641+0000 7f84e97fa640 1 -- 172.21.15.90:0/3336889429 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f84dc018020 con 0x7f84ec1018d0 2024-04-25T11:40:21.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.642+0000 7f84e97fa640 1 --2- 172.21.15.90:0/3336889429 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f84d40afcd0 0x7f84d40b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:21.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.642+0000 7f84eb7fe640 1 --2- 172.21.15.90:0/3336889429 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f84d40afcd0 0x7f84d40b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:21.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.643+0000 7f84eb7fe640 1 --2- 172.21.15.90:0/3336889429 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f84d40afcd0 0x7f84d40b21e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f84e402aed0 tx=0x7f84e4000f40 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:21.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.643+0000 7f84e97fa640 1 -- 172.21.15.90:0/3336889429 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f84dc0deab0 con 0x7f84ec1018d0 2024-04-25T11:40:21.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:21.648+0000 7f84e97fa640 1 -- 172.21.15.90:0/3336889429 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f84dc01f020 con 0x7f84ec1018d0 2024-04-25T11:40:22.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.127+0000 7f84f1391640 1 -- 172.21.15.90:0/3336889429 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f84ec0ffd30 con 0x7f84ec1018d0 2024-04-25T11:40:22.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.128+0000 7f84e97fa640 1 -- 172.21.15.90:0/3336889429 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11930 (secure 0 0 0) 0x7f84dc00d040 con 0x7f84ec1018d0 2024-04-25T11:40:22.116 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:40:22.116 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:39:20.469254+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24458,24431,24499,24475,24524]},"id":1}]} 2024-04-25T11:40:22.116 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 28 2024-04-25T11:40:22.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.132+0000 7f84f1391640 1 -- 172.21.15.90:0/3336889429 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f84d40afcd0 msgr2=0x7f84d40b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:22.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.132+0000 7f84f1391640 1 --2- 172.21.15.90:0/3336889429 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f84d40afcd0 0x7f84d40b21e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f84e402aed0 tx=0x7f84e4000f40 comp rx=0 tx=0).stop 2024-04-25T11:40:22.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.132+0000 7f84f1391640 1 -- 172.21.15.90:0/3336889429 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84ec1018d0 msgr2=0x7f84ec10a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:22.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.132+0000 7f84f1391640 1 --2- 172.21.15.90:0/3336889429 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84ec1018d0 0x7f84ec10a8e0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00b450 tx=0x7f84dc00b910 comp rx=0 tx=0).stop 2024-04-25T11:40:22.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.133+0000 7f84f1391640 1 -- 172.21.15.90:0/3336889429 shutdown_connections 2024-04-25T11:40:22.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.133+0000 7f84f1391640 1 --2- 172.21.15.90:0/3336889429 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f84d40afcd0 0x7f84d40b21e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.133+0000 7f84f1391640 1 --2- 172.21.15.90:0/3336889429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84ec1145e0 0x7f84ec10b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.133+0000 7f84f1391640 1 --2- 172.21.15.90:0/3336889429 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84ec111c80 0x7f84ec10ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.133+0000 7f84f1391640 1 --2- 172.21.15.90:0/3336889429 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84ec1018d0 0x7f84ec10a8e0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.133+0000 7f84f1391640 1 -- 172.21.15.90:0/3336889429 >> 172.21.15.90:0/3336889429 conn(0x7f84ec0fd200 msgr2=0x7f84ec1089f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:22.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.133+0000 7f84f1391640 1 -- 172.21.15.90:0/3336889429 shutdown_connections 2024-04-25T11:40:22.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.133+0000 7f84f1391640 1 -- 172.21.15.90:0/3336889429 wait complete. 2024-04-25T11:40:22.140 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T11:40:22.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.483+0000 7ff1a112d640 1 -- 172.21.15.90:0/2007433718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff19c10a0e0 msgr2=0x7ff19c10a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:22.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.483+0000 7ff1a112d640 1 --2- 172.21.15.90:0/2007433718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff19c10a0e0 0x7ff19c10a570 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7ff190009860 tx=0x7ff19002a380 comp rx=0 tx=0).stop 2024-04-25T11:40:22.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:22 smithi090 ceph-mon[29575]: pgmap v161: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 404 MiB/s wr, 2.28k op/s 2024-04-25T11:40:22.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.493+0000 7ff1a112d640 1 -- 172.21.15.90:0/2007433718 shutdown_connections 2024-04-25T11:40:22.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.493+0000 7ff1a112d640 1 --2- 172.21.15.90:0/2007433718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff19c10a0e0 0x7ff19c10a570 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.494+0000 7ff1a112d640 1 --2- 172.21.15.90:0/2007433718 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff19c1102c0 0x7ff19c109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.494+0000 7ff1a112d640 1 --2- 172.21.15.90:0/2007433718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff19c10f8e0 0x7ff19c10fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.494+0000 7ff1a112d640 1 -- 172.21.15.90:0/2007433718 >> 172.21.15.90:0/2007433718 conn(0x7ff19c06e5d0 msgr2=0x7ff19c06ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:22.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.497+0000 7ff1a112d640 1 -- 172.21.15.90:0/2007433718 shutdown_connections 2024-04-25T11:40:22.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.499+0000 7ff1a112d640 1 -- 172.21.15.90:0/2007433718 wait complete. 2024-04-25T11:40:22.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.501+0000 7ff1a112d640 1 Processor -- start 2024-04-25T11:40:22.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.502+0000 7ff1a112d640 1 -- start start 2024-04-25T11:40:22.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.503+0000 7ff1a112d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff19c10a0e0 0x7ff19c1a9360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.503+0000 7ff1a112d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff19c10f8e0 0x7ff19c1a98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.503+0000 7ff1a112d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff19c1102c0 0x7ff19c1adb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.503+0000 7ff1a112d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff19c122940 con 0x7ff19c1102c0 2024-04-25T11:40:22.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.504+0000 7ff1a112d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff19c1227c0 con 0x7ff19c10a0e0 2024-04-25T11:40:22.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.504+0000 7ff1a112d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff19c122ac0 con 0x7ff19c10f8e0 2024-04-25T11:40:22.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.504+0000 7ff19b7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff19c10f8e0 0x7ff19c1a98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:22.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.504+0000 7ff1a092c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff19c1102c0 0x7ff19c1adb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:22.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.504+0000 7ff1a092c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff19c1102c0 0x7ff19c1adb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60786/0 (socket says 172.21.15.90:60786) 2024-04-25T11:40:22.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.504+0000 7ff1a092c640 1 -- 172.21.15.90:0/3738622022 learned_addr learned my addr 172.21.15.90:0/3738622022 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:22.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.505+0000 7ff19b7fe640 1 -- 172.21.15.90:0/3738622022 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff19c10a0e0 msgr2=0x7ff19c1a9360 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:40:22.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.505+0000 7ff19b7fe640 1 --2- 172.21.15.90:0/3738622022 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff19c10a0e0 0x7ff19c1a9360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.505+0000 7ff19b7fe640 1 -- 172.21.15.90:0/3738622022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff19c1102c0 msgr2=0x7ff19c1adb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:22.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.505+0000 7ff19b7fe640 1 --2- 172.21.15.90:0/3738622022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff19c1102c0 0x7ff19c1adb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.505+0000 7ff19b7fe640 1 -- 172.21.15.90:0/3738622022 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff19c1ae1c0 con 0x7ff19c10f8e0 2024-04-25T11:40:22.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.506+0000 7ff19b7fe640 1 --2- 172.21.15.90:0/3738622022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff19c10f8e0 0x7ff19c1a98c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff18c00d6a0 tx=0x7ff18c00db60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:22.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.507+0000 7ff1997fa640 1 -- 172.21.15.90:0/3738622022 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18c017070 con 0x7ff19c10f8e0 2024-04-25T11:40:22.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.507+0000 7ff1997fa640 1 -- 172.21.15.90:0/3738622022 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff18c009820 con 0x7ff19c10f8e0 2024-04-25T11:40:22.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.507+0000 7ff1997fa640 1 -- 172.21.15.90:0/3738622022 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18c004c40 con 0x7ff19c10f8e0 2024-04-25T11:40:22.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.507+0000 7ff1a112d640 1 -- 172.21.15.90:0/3738622022 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff19c1ae4b0 con 0x7ff19c10f8e0 2024-04-25T11:40:22.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.507+0000 7ff1a112d640 1 -- 172.21.15.90:0/3738622022 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff19c1ae9b0 con 0x7ff19c10f8e0 2024-04-25T11:40:22.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.519+0000 7ff1a9240740 1 Processor -- start 2024-04-25T11:40:22.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.519+0000 7ff1a9240740 1 -- start start 2024-04-25T11:40:22.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.520+0000 7ff1a9240740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55559d5a2390 0x55559d9439e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.520+0000 7ff1a9240740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55559d5794d0 0x55559d943f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.520+0000 7ff1a9240740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55559d5816b0 0x55559d950870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.521+0000 7ff1a9240740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55559d8a8d80 con 0x55559d5816b0 2024-04-25T11:40:22.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.521+0000 7ff1a092c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55559d5a2390 0x55559d9439e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:22.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.521+0000 7ff1a092c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55559d5a2390 0x55559d9439e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:59504/0 (socket says 172.21.15.90:59504) 2024-04-25T11:40:22.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.521+0000 7ff1a092c640 1 -- 172.21.15.90:0/1748499148 learned_addr learned my addr 172.21.15.90:0/1748499148 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:22.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.521+0000 7ff1a9240740 1 -- 172.21.15.90:0/1748499148 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55559d862a20 con 0x55559d5794d0 2024-04-25T11:40:22.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.521+0000 7ff1a9240740 1 -- 172.21.15.90:0/1748499148 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55559d8a4d40 con 0x55559d5a2390 2024-04-25T11:40:22.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.521+0000 7ff1997fa640 1 -- 172.21.15.90:0/3738622022 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7ff18c010460 con 0x7ff19c10f8e0 2024-04-25T11:40:22.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.522+0000 7ff1a092c640 1 -- 172.21.15.90:0/1748499148 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55559d5794d0 msgr2=0x55559d943f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:40:22.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.522+0000 7ff1a092c640 1 --2- 172.21.15.90:0/1748499148 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55559d5794d0 0x55559d943f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.522+0000 7ff1a092c640 1 -- 172.21.15.90:0/1748499148 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55559d5816b0 msgr2=0x55559d950870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:22.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.522+0000 7ff1a092c640 1 --2- 172.21.15.90:0/1748499148 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55559d5816b0 0x55559d950870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.522+0000 7ff1a092c640 1 -- 172.21.15.90:0/1748499148 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55559d86dc30 con 0x55559d5a2390 2024-04-25T11:40:22.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.523+0000 7ff1a092c640 1 --2- 172.21.15.90:0/1748499148 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55559d5a2390 0x55559d9439e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff190035040 tx=0x7ff190009490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1992a6cdefbffd6e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:22.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.524+0000 7ff1997fa640 1 --2- 172.21.15.90:0/3738622022 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff1840afe60 0x7ff1840b2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.525+0000 7ff1997fa640 1 -- 172.21.15.90:0/3738622022 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7ff18c005280 con 0x7ff19c10f8e0 2024-04-25T11:40:22.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.525+0000 7ff1827fc640 1 -- 172.21.15.90:0/1748499148 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff190042020 con 0x55559d5a2390 2024-04-25T11:40:22.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.526+0000 7ff1827fc640 1 -- 172.21.15.90:0/1748499148 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff190038040 con 0x55559d5a2390 2024-04-25T11:40:22.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.527+0000 7ff1a9240740 1 -- 172.21.15.90:0/1748499148 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55559d5a2390 msgr2=0x55559d9439e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:22.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.527+0000 7ff1a9240740 1 --2- 172.21.15.90:0/1748499148 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55559d5a2390 0x55559d9439e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff190035040 tx=0x7ff190009490 comp rx=0 tx=0).stop 2024-04-25T11:40:22.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.527+0000 7ff1a9240740 1 -- 172.21.15.90:0/1748499148 shutdown_connections 2024-04-25T11:40:22.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.528+0000 7ff1a9240740 1 --2- 172.21.15.90:0/1748499148 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55559d5816b0 0x55559d950870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.528+0000 7ff1a9240740 1 --2- 172.21.15.90:0/1748499148 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55559d5794d0 0x55559d943f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.528+0000 7ff1a9240740 1 --2- 172.21.15.90:0/1748499148 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55559d5a2390 0x55559d9439e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.528+0000 7ff1a9240740 1 -- 172.21.15.90:0/1748499148 >> 172.21.15.90:0/1748499148 conn(0x55559d5964e0 msgr2=0x55559d940190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:22.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.528+0000 7ff1a9240740 1 -- 172.21.15.90:0/1748499148 shutdown_connections 2024-04-25T11:40:22.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.529+0000 7ff19bfff640 1 --2- 172.21.15.90:0/3738622022 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff1840afe60 0x7ff1840b2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:22.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.530+0000 7ff1a9240740 1 -- 172.21.15.90:0/1748499148 wait complete. 2024-04-25T11:40:22.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.530+0000 7ff1a9240740 1 -- start start 2024-04-25T11:40:22.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.531+0000 7ff1a9240740 1 Processor -- start 2024-04-25T11:40:22.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.532+0000 7ff1a9240740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55559d5794d0 0x55559d94de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.532+0000 7ff1a9240740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55559d5816b0 0x55559d94e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.532+0000 7ff1a9240740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55559d560f60 0x55559d94e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.532+0000 7ff1a9240740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55559d950f50 con 0x55559d560f60 2024-04-25T11:40:22.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.532+0000 7ff1a9240740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55559d82fdc0 con 0x55559d5794d0 2024-04-25T11:40:22.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.532+0000 7ff1a9240740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55559d9510d0 con 0x55559d5816b0 2024-04-25T11:40:22.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.532+0000 7ff1a9240740 10 client.0 _ll_register_callbacks cb 0x55559d84b600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:22.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.532+0000 7ff1a092c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55559d560f60 0x55559d94e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:22.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.532+0000 7ff1a092c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55559d560f60 0x55559d94e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60804/0 (socket says 172.21.15.90:60804) 2024-04-25T11:40:22.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.532+0000 7ff1a092c640 1 -- 172.21.15.90:0/1654899316 learned_addr learned my addr 172.21.15.90:0/1654899316 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:22.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.533+0000 7ff19b7fe640 1 --2- 172.21.15.90:0/1654899316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55559d5816b0 0x55559d94e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:22.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.533+0000 7ff19bfff640 1 --2- 172.21.15.90:0/1654899316 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55559d5794d0 0x55559d94de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:22.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.535+0000 7ff19bfff640 1 --2- 172.21.15.90:0/3738622022 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff1840afe60 0x7ff1840b2370 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff194009330 tx=0x7ff194012800 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:22.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.535+0000 7ff1a092c640 1 -- 172.21.15.90:0/1654899316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55559d5816b0 msgr2=0x55559d94e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:22.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.535+0000 7ff1a092c640 1 --2- 172.21.15.90:0/1654899316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55559d5816b0 0x55559d94e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.535+0000 7ff1a092c640 1 -- 172.21.15.90:0/1654899316 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55559d5794d0 msgr2=0x55559d94de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:22.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.535+0000 7ff1a092c640 1 --2- 172.21.15.90:0/1654899316 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55559d5794d0 0x55559d94de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.535+0000 7ff1a092c640 1 -- 172.21.15.90:0/1654899316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55559d5ba3f0 con 0x55559d560f60 2024-04-25T11:40:22.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.536+0000 7ff1a092c640 1 --2- 172.21.15.90:0/1654899316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55559d560f60 0x55559d94e8a0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7ff19002a970 tx=0x7ff190004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e10e9d5f88c51518 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:22.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.537+0000 7ff17b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:40:22.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.537+0000 7ff17b7fe640 1 -- 172.21.15.90:0/1654899316 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff190042030 con 0x55559d560f60 2024-04-25T11:40:22.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.537+0000 7ff1a9240740 1 -- 172.21.15.90:0/1654899316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55559d8b6990 con 0x55559d560f60 2024-04-25T11:40:22.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.538+0000 7ff17b7fe640 1 -- 172.21.15.90:0/1654899316 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7ff1900035f0 con 0x55559d560f60 2024-04-25T11:40:22.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.541+0000 7ff17b7fe640 1 -- 172.21.15.90:0/1654899316 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff1900037f0 con 0x55559d560f60 2024-04-25T11:40:22.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.541+0000 7ff1a9240740 10 client.17127 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:22.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.541+0000 7ff1a9240740 1 -- 172.21.15.90:0/1654899316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55559d6847d0 con 0x55559d560f60 2024-04-25T11:40:22.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.542+0000 7ff17b7fe640 1 -- 172.21.15.90:0/1654899316 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55559d8b6990 con 0x55559d560f60 2024-04-25T11:40:22.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.543+0000 7ff17b7fe640 1 -- 172.21.15.90:0/1654899316 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7ff190005370 con 0x55559d560f60 2024-04-25T11:40:22.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.543+0000 7ff1a9240740 10 client.17127 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:22.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.543+0000 7ff1a9240740 10 client.17127 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:40:22.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.543+0000 7ff1a9240740 20 client.17127 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:22.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.543+0000 7ff1a9240740 1 --2- 172.21.15.90:0/1654899316 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55559d5363a0 0x55559d960490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.543+0000 7ff1a9240740 4 client.17127 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:40:22.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.543+0000 7ff1a9240740 1 -- 172.21.15.90:0/1654899316 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55559d8b0800 con 0x55559d5363a0 2024-04-25T11:40:22.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.545+0000 7ff19bfff640 1 --2- 172.21.15.90:0/1654899316 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55559d5363a0 0x55559d960490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:22.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.551+0000 7ff19bfff640 1 --2- 172.21.15.90:0/1654899316 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55559d5363a0 0x55559d960490 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=469592f4f17128d2 server_cookie=daeacf026fd18f44 in_seq=0 out_seq=0 2024-04-25T11:40:22.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.551+0000 7ff17b7fe640 10 client.17127 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:40:22.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.554+0000 7ff17b7fe640 1 -- 172.21.15.90:0/1654899316 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55559d8b0800 con 0x55559d5363a0 2024-04-25T11:40:22.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.554+0000 7ff17b7fe640 10 client.17127 handle_command_reply: tid=0 2024-04-25T11:40:22.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.554+0000 7ff1a9240740 1 client.17127 shutdown 2024-04-25T11:40:22.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.555+0000 7ff1a9240740 1 -- 172.21.15.90:0/1654899316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55559d560f60 msgr2=0x55559d94e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:22.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.555+0000 7ff1a9240740 1 --2- 172.21.15.90:0/1654899316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55559d560f60 0x55559d94e8a0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7ff19002a970 tx=0x7ff190004270 comp rx=0 tx=0).stop 2024-04-25T11:40:22.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.555+0000 7ff1a9240740 1 -- 172.21.15.90:0/1654899316 shutdown_connections 2024-04-25T11:40:22.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.555+0000 7ff1a9240740 1 --2- 172.21.15.90:0/1654899316 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55559d5363a0 0x55559d960490 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.555+0000 7ff1a9240740 1 --2- 172.21.15.90:0/1654899316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55559d560f60 0x55559d94e8a0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.555+0000 7ff1a9240740 1 --2- 172.21.15.90:0/1654899316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55559d5816b0 0x55559d94e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.555+0000 7ff1a9240740 1 --2- 172.21.15.90:0/1654899316 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55559d5794d0 0x55559d94de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.555+0000 7ff17b7fe640 0 client.17127 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:40:22.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.555+0000 7ff1a9240740 1 -- 172.21.15.90:0/1654899316 >> 172.21.15.90:0/1654899316 conn(0x55559d5964e0 msgr2=0x55559d945700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:22.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.555+0000 7ff1a9240740 1 -- 172.21.15.90:0/1654899316 shutdown_connections 2024-04-25T11:40:22.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.555+0000 7ff1a9240740 1 -- 172.21.15.90:0/1654899316 wait complete. 2024-04-25T11:40:22.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.555+0000 7ff1a9240740 20 client.17127 trim_cache size 0 max 16384 2024-04-25T11:40:22.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:22 smithi138 ceph-mon[32328]: pgmap v161: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 404 MiB/s wr, 2.28k op/s 2024-04-25T11:40:22.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:22 smithi156 ceph-mon[33045]: pgmap v161: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 404 MiB/s wr, 2.28k op/s 2024-04-25T11:40:22.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.607+0000 7ff1a9240740 1 Processor -- start 2024-04-25T11:40:22.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.607+0000 7ff1a9240740 1 -- start start 2024-04-25T11:40:22.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.608+0000 7ff1a9240740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55559d560f60 0x55559d950550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.608+0000 7ff1a9240740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55559d5794d0 0x55559d94e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.608+0000 7ff1a9240740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55559d5816b0 0x55559d945cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.608+0000 7ff1a9240740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff18800ea60 con 0x55559d5816b0 2024-04-25T11:40:22.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.608+0000 7ff1a9240740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff1880105a0 con 0x55559d560f60 2024-04-25T11:40:22.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.608+0000 7ff1a9240740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff18800f130 con 0x55559d5794d0 2024-04-25T11:40:22.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.608+0000 7ff1a092c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55559d5816b0 0x55559d945cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:22.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.608+0000 7ff1a092c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55559d5816b0 0x55559d945cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60816/0 (socket says 172.21.15.90:60816) 2024-04-25T11:40:22.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.608+0000 7ff1a092c640 1 -- 172.21.15.90:0/2862355569 learned_addr learned my addr 172.21.15.90:0/2862355569 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:22.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.609+0000 7ff19b7fe640 1 --2- 172.21.15.90:0/2862355569 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55559d5794d0 0x55559d94e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:22.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.609+0000 7ff19bfff640 1 --2- 172.21.15.90:0/2862355569 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55559d560f60 0x55559d950550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:22.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.609+0000 7ff1a092c640 1 -- 172.21.15.90:0/2862355569 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55559d5794d0 msgr2=0x55559d94e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:22.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.609+0000 7ff1a092c640 1 --2- 172.21.15.90:0/2862355569 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55559d5794d0 0x55559d94e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.609+0000 7ff1a092c640 1 -- 172.21.15.90:0/2862355569 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55559d560f60 msgr2=0x55559d950550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:22.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.609+0000 7ff1a092c640 1 --2- 172.21.15.90:0/2862355569 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55559d560f60 0x55559d950550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.609+0000 7ff1a092c640 1 -- 172.21.15.90:0/2862355569 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55559d9451e0 con 0x55559d5816b0 2024-04-25T11:40:22.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.610+0000 7ff1a092c640 1 --2- 172.21.15.90:0/2862355569 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55559d5816b0 0x55559d945cf0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7ff190035040 tx=0x7ff190030940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f31838ab2a03adeb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:22.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.611+0000 7ff181ffb640 1 -- 172.21.15.90:0/2862355569 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff190042040 con 0x55559d5816b0 2024-04-25T11:40:22.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.611+0000 7ff181ffb640 1 -- 172.21.15.90:0/2862355569 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff19005c070 con 0x55559d5816b0 2024-04-25T11:40:22.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.611+0000 7ff181ffb640 1 -- 172.21.15.90:0/2862355569 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff190003bf0 con 0x55559d5816b0 2024-04-25T11:40:22.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.612+0000 7ff1a9240740 1 -- 172.21.15.90:0/2862355569 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55559d5816b0 msgr2=0x55559d945cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:22.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.612+0000 7ff1a9240740 1 --2- 172.21.15.90:0/2862355569 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55559d5816b0 0x55559d945cf0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7ff190035040 tx=0x7ff190030940 comp rx=0 tx=0).stop 2024-04-25T11:40:22.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.612+0000 7ff1a9240740 1 -- 172.21.15.90:0/2862355569 shutdown_connections 2024-04-25T11:40:22.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.613+0000 7ff1a9240740 1 --2- 172.21.15.90:0/2862355569 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55559d5816b0 0x55559d945cf0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.613+0000 7ff1a9240740 1 --2- 172.21.15.90:0/2862355569 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55559d5794d0 0x55559d94e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.613+0000 7ff1a9240740 1 --2- 172.21.15.90:0/2862355569 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55559d560f60 0x55559d950550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.613+0000 7ff1a9240740 1 -- 172.21.15.90:0/2862355569 >> 172.21.15.90:0/2862355569 conn(0x55559d5964e0 msgr2=0x55559d9455e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:22.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.613+0000 7ff1a9240740 1 -- 172.21.15.90:0/2862355569 shutdown_connections 2024-04-25T11:40:22.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.613+0000 7ff1a9240740 1 -- 172.21.15.90:0/2862355569 wait complete. 2024-04-25T11:40:22.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.614+0000 7ff1a9240740 1 -- start start 2024-04-25T11:40:22.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.614+0000 7ff1a9240740 1 Processor -- start 2024-04-25T11:40:22.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.615+0000 7ff1a9240740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55559d560f60 0x55559d960710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.615+0000 7ff1a9240740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55559d5794d0 0x55559d960c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.615+0000 7ff1a9240740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55559d5816b0 0x55559d9c65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.615+0000 7ff1a9240740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff18800fed0 con 0x55559d5794d0 2024-04-25T11:40:22.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.615+0000 7ff1a9240740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff18800f800 con 0x55559d5816b0 2024-04-25T11:40:22.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.615+0000 7ff1a9240740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff18800ca40 con 0x55559d560f60 2024-04-25T11:40:22.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.615+0000 7ff1a9240740 10 client.0 _ll_register_callbacks cb 0x7ff1900303d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:22.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.615+0000 7ff19b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55559d5794d0 0x55559d960c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:22.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.615+0000 7ff19b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55559d5794d0 0x55559d960c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60826/0 (socket says 172.21.15.90:60826) 2024-04-25T11:40:22.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.615+0000 7ff19b7fe640 1 -- 172.21.15.90:0/2947460288 learned_addr learned my addr 172.21.15.90:0/2947460288 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:22.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.615+0000 7ff19bfff640 1 --2- 172.21.15.90:0/2947460288 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55559d560f60 0x55559d960710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:22.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.615+0000 7ff1a092c640 1 --2- 172.21.15.90:0/2947460288 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55559d5816b0 0x55559d9c65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:22.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.616+0000 7ff19b7fe640 1 -- 172.21.15.90:0/2947460288 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55559d560f60 msgr2=0x55559d960710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:22.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.616+0000 7ff19b7fe640 1 --2- 172.21.15.90:0/2947460288 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55559d560f60 0x55559d960710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.616+0000 7ff19b7fe640 1 -- 172.21.15.90:0/2947460288 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55559d5816b0 msgr2=0x55559d9c65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:22.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.616+0000 7ff19b7fe640 1 --2- 172.21.15.90:0/2947460288 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55559d5816b0 0x55559d9c65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.616+0000 7ff19b7fe640 1 -- 172.21.15.90:0/2947460288 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55559d9c6af0 con 0x55559d5794d0 2024-04-25T11:40:22.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.616+0000 7ff1a092c640 1 --2- 172.21.15.90:0/2947460288 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55559d5816b0 0x55559d9c65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:40:22.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.616+0000 7ff19b7fe640 1 --2- 172.21.15.90:0/2947460288 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55559d5794d0 0x55559d960c50 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7ff18c009610 tx=0x7ff18c0a3040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcee8e6e2fdb8d06 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:22.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.616+0000 7ff17b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:40:22.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.616+0000 7ff17b7fe640 1 -- 172.21.15.90:0/2947460288 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18c020040 con 0x55559d5794d0 2024-04-25T11:40:22.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.617+0000 7ff17b7fe640 1 -- 172.21.15.90:0/2947460288 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7ff18c0a3b40 con 0x55559d5794d0 2024-04-25T11:40:22.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.617+0000 7ff1a9240740 1 -- 172.21.15.90:0/2947460288 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff19005c070 con 0x55559d5794d0 2024-04-25T11:40:22.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.617+0000 7ff17b7fe640 1 -- 172.21.15.90:0/2947460288 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff18c0abc60 con 0x55559d5794d0 2024-04-25T11:40:22.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.617+0000 7ff1a9240740 10 client.17133 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:22.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.617+0000 7ff1a9240740 1 -- 172.21.15.90:0/2947460288 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55559d9c7bf0 con 0x55559d5794d0 2024-04-25T11:40:22.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.618+0000 7ff17b7fe640 1 -- 172.21.15.90:0/2947460288 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff19005c070 con 0x55559d5794d0 2024-04-25T11:40:22.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.618+0000 7ff17b7fe640 1 -- 172.21.15.90:0/2947460288 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7ff18c0c8b60 con 0x55559d5794d0 2024-04-25T11:40:22.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.618+0000 7ff1a9240740 10 client.17133 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:22.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.618+0000 7ff1a9240740 10 client.17133 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:40:22.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.618+0000 7ff1a9240740 20 client.17133 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:22.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.618+0000 7ff1a9240740 1 --2- 172.21.15.90:0/2947460288 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55559d9c8320 0x55559d9ca740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.618+0000 7ff1a9240740 4 client.17133 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:40:22.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.618+0000 7ff1a9240740 1 -- 172.21.15.90:0/2947460288 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55559d82b610 con 0x55559d9c8320 2024-04-25T11:40:22.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.618+0000 7ff19bfff640 1 --2- 172.21.15.90:0/2947460288 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55559d9c8320 0x55559d9ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:22.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.620+0000 7ff17b7fe640 10 client.17133 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:40:22.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.620+0000 7ff19bfff640 1 --2- 172.21.15.90:0/2947460288 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55559d9c8320 0x55559d9ca740 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d98868612edb804f server_cookie=c7e945eea83a9ffd in_seq=0 out_seq=0 2024-04-25T11:40:22.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.621+0000 7ff17b7fe640 1 -- 172.21.15.90:0/2947460288 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55559d82b610 con 0x55559d9c8320 2024-04-25T11:40:22.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.621+0000 7ff17b7fe640 10 client.17133 handle_command_reply: tid=0 2024-04-25T11:40:22.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.621+0000 7ff1a9240740 1 client.17133 shutdown 2024-04-25T11:40:22.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.622+0000 7ff1a9240740 1 -- 172.21.15.90:0/2947460288 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55559d5794d0 msgr2=0x55559d960c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:22.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.622+0000 7ff1a9240740 1 --2- 172.21.15.90:0/2947460288 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55559d5794d0 0x55559d960c50 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7ff18c009610 tx=0x7ff18c0a3040 comp rx=0 tx=0).stop 2024-04-25T11:40:22.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.622+0000 7ff1a9240740 1 -- 172.21.15.90:0/2947460288 shutdown_connections 2024-04-25T11:40:22.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.622+0000 7ff1a9240740 1 --2- 172.21.15.90:0/2947460288 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55559d9c8320 0x55559d9ca740 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.622+0000 7ff1a9240740 1 --2- 172.21.15.90:0/2947460288 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55559d5816b0 0x55559d9c65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.622+0000 7ff1a9240740 1 --2- 172.21.15.90:0/2947460288 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55559d5794d0 0x55559d960c50 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.622+0000 7ff1a9240740 1 --2- 172.21.15.90:0/2947460288 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55559d560f60 0x55559d960710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.622+0000 7ff17b7fe640 0 client.17133 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:40:22.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.622+0000 7ff1a9240740 1 -- 172.21.15.90:0/2947460288 >> 172.21.15.90:0/2947460288 conn(0x55559d5964e0 msgr2=0x55559d93e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:22.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.622+0000 7ff1a9240740 1 -- 172.21.15.90:0/2947460288 shutdown_connections 2024-04-25T11:40:22.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.623+0000 7ff1a9240740 1 -- 172.21.15.90:0/2947460288 wait complete. 2024-04-25T11:40:22.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.623+0000 7ff1a9240740 20 client.17133 trim_cache size 0 max 16384 2024-04-25T11:40:22.611 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:40:22.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.624+0000 7ff1a112d640 1 -- 172.21.15.90:0/3738622022 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff1840afe60 msgr2=0x7ff1840b2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:22.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.624+0000 7ff1a112d640 1 --2- 172.21.15.90:0/3738622022 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff1840afe60 0x7ff1840b2370 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff194009330 tx=0x7ff194012800 comp rx=0 tx=0).stop 2024-04-25T11:40:22.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.624+0000 7ff1a112d640 1 -- 172.21.15.90:0/3738622022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff19c10f8e0 msgr2=0x7ff19c1a98c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:22.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.624+0000 7ff1a112d640 1 --2- 172.21.15.90:0/3738622022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff19c10f8e0 0x7ff19c1a98c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff18c00d6a0 tx=0x7ff18c00db60 comp rx=0 tx=0).stop 2024-04-25T11:40:22.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.625+0000 7ff1a112d640 1 -- 172.21.15.90:0/3738622022 shutdown_connections 2024-04-25T11:40:22.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.625+0000 7ff1a112d640 1 --2- 172.21.15.90:0/3738622022 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff1840afe60 0x7ff1840b2370 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.625+0000 7ff1a112d640 1 --2- 172.21.15.90:0/3738622022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff19c1102c0 0x7ff19c1adb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.625+0000 7ff1a112d640 1 --2- 172.21.15.90:0/3738622022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff19c10f8e0 0x7ff19c1a98c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.625+0000 7ff1a112d640 1 --2- 172.21.15.90:0/3738622022 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff19c10a0e0 0x7ff19c1a9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.625+0000 7ff1a112d640 1 -- 172.21.15.90:0/3738622022 >> 172.21.15.90:0/3738622022 conn(0x7ff19c06e5d0 msgr2=0x7ff19c073910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:22.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.625+0000 7ff1a112d640 1 -- 172.21.15.90:0/3738622022 shutdown_connections 2024-04-25T11:40:22.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.625+0000 7ff1a112d640 1 -- 172.21.15.90:0/3738622022 wait complete. 2024-04-25T11:40:22.630 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T11:40:22.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.924+0000 7fa2450ca640 1 -- 172.21.15.90:0/3277806354 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa240102400 msgr2=0x7fa240102810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:22.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.924+0000 7fa2450ca640 1 --2- 172.21.15.90:0/3277806354 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa240102400 0x7fa240102810 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fa234009900 tx=0x7fa23402a410 comp rx=0 tx=0).stop 2024-04-25T11:40:22.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.925+0000 7fa2450ca640 1 -- 172.21.15.90:0/3277806354 shutdown_connections 2024-04-25T11:40:22.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.925+0000 7fa2450ca640 1 --2- 172.21.15.90:0/3277806354 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa240107450 0x7fa24010ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.925+0000 7fa2450ca640 1 --2- 172.21.15.90:0/3277806354 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa240106990 0x7fa240106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.925+0000 7fa2450ca640 1 --2- 172.21.15.90:0/3277806354 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa240102400 0x7fa240102810 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.925+0000 7fa2450ca640 1 -- 172.21.15.90:0/3277806354 >> 172.21.15.90:0/3277806354 conn(0x7fa240077880 msgr2=0x7fa240077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:22.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.926+0000 7fa2450ca640 1 -- 172.21.15.90:0/3277806354 shutdown_connections 2024-04-25T11:40:22.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.926+0000 7fa2450ca640 1 -- 172.21.15.90:0/3277806354 wait complete. 2024-04-25T11:40:22.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.928+0000 7fa2450ca640 1 Processor -- start 2024-04-25T11:40:22.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.928+0000 7fa2450ca640 1 -- start start 2024-04-25T11:40:22.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.929+0000 7fa2450ca640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa240102400 0x7fa24019e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.929+0000 7fa2450ca640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa240106990 0x7fa24019ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.929+0000 7fa2450ca640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa240107450 0x7fa240198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.929+0000 7fa2450ca640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa240112430 con 0x7fa240107450 2024-04-25T11:40:22.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.929+0000 7fa2450ca640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa2401122b0 con 0x7fa240106990 2024-04-25T11:40:22.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.929+0000 7fa2450ca640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa2401125b0 con 0x7fa240102400 2024-04-25T11:40:22.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.930+0000 7fa23ed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa240102400 0x7fa24019e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:22.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.930+0000 7fa23ed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa240102400 0x7fa24019e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:59540/0 (socket says 172.21.15.90:59540) 2024-04-25T11:40:22.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.930+0000 7fa23ed9d640 1 -- 172.21.15.90:0/4070423393 learned_addr learned my addr 172.21.15.90:0/4070423393 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:22.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.930+0000 7fa23e59c640 1 --2- 172.21.15.90:0/4070423393 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa240106990 0x7fa24019ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:22.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.930+0000 7fa23f59e640 1 --2- 172.21.15.90:0/4070423393 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa240107450 0x7fa240198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:22.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.931+0000 7fa23f59e640 1 -- 172.21.15.90:0/4070423393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa240102400 msgr2=0x7fa24019e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:22.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.931+0000 7fa23f59e640 1 --2- 172.21.15.90:0/4070423393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa240102400 0x7fa24019e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.931+0000 7fa23f59e640 1 -- 172.21.15.90:0/4070423393 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa240106990 msgr2=0x7fa24019ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:22.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.931+0000 7fa23f59e640 1 --2- 172.21.15.90:0/4070423393 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa240106990 0x7fa24019ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.931+0000 7fa23f59e640 1 -- 172.21.15.90:0/4070423393 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa240199190 con 0x7fa240107450 2024-04-25T11:40:22.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.931+0000 7fa23f59e640 1 --2- 172.21.15.90:0/4070423393 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa240107450 0x7fa240198af0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fa238009fd0 tx=0x7fa23800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:22.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.932+0000 7fa227fff640 1 -- 172.21.15.90:0/4070423393 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23800ce20 con 0x7fa240107450 2024-04-25T11:40:22.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.932+0000 7fa227fff640 1 -- 172.21.15.90:0/4070423393 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa238005020 con 0x7fa240107450 2024-04-25T11:40:22.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.932+0000 7fa227fff640 1 -- 172.21.15.90:0/4070423393 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa238010690 con 0x7fa240107450 2024-04-25T11:40:22.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.934+0000 7fa2450ca640 1 -- 172.21.15.90:0/4070423393 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa240199480 con 0x7fa240107450 2024-04-25T11:40:22.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.934+0000 7fa2450ca640 1 -- 172.21.15.90:0/4070423393 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2401998f0 con 0x7fa240107450 2024-04-25T11:40:22.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.942+0000 7fa227fff640 1 -- 172.21.15.90:0/4070423393 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fa2380054d0 con 0x7fa240107450 2024-04-25T11:40:22.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.946+0000 7fa227fff640 1 --2- 172.21.15.90:0/4070423393 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa2280afe70 0x7fa2280b2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.947+0000 7fa227fff640 1 -- 172.21.15.90:0/4070423393 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fa2380dfac0 con 0x7fa240107450 2024-04-25T11:40:22.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.951+0000 7fa24bf7b740 1 Processor -- start 2024-04-25T11:40:22.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.951+0000 7fa24bf7b740 1 -- start start 2024-04-25T11:40:22.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.952+0000 7fa24bf7b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5642894f5390 0x564289896810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.952+0000 7fa24bf7b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894cc4d0 0x564289896d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.952+0000 7fa24bf7b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642894d46b0 0x5642898a36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.952+0000 7fa24bf7b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5642897fbd80 con 0x5642894cc4d0 2024-04-25T11:40:22.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.952+0000 7fa24bf7b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5642897b5a20 con 0x5642894f5390 2024-04-25T11:40:22.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.952+0000 7fa24bf7b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5642897f7d40 con 0x5642894d46b0 2024-04-25T11:40:22.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.953+0000 7fa23e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894cc4d0 0x564289896d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:22.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.953+0000 7fa23e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894cc4d0 0x564289896d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60836/0 (socket says 172.21.15.90:60836) 2024-04-25T11:40:22.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.953+0000 7fa23e59c640 1 -- 172.21.15.90:0/3128900160 learned_addr learned my addr 172.21.15.90:0/3128900160 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:22.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.953+0000 7fa23f59e640 1 --2- 172.21.15.90:0/3128900160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642894d46b0 0x5642898a36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:22.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.954+0000 7fa23e59c640 1 -- 172.21.15.90:0/3128900160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642894d46b0 msgr2=0x5642898a36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:22.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.954+0000 7fa23e59c640 1 --2- 172.21.15.90:0/3128900160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642894d46b0 0x5642898a36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.954+0000 7fa23e59c640 1 -- 172.21.15.90:0/3128900160 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5642894f5390 msgr2=0x564289896810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:40:22.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.954+0000 7fa23e59c640 1 --2- 172.21.15.90:0/3128900160 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5642894f5390 0x564289896810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.954+0000 7fa23e59c640 1 -- 172.21.15.90:0/3128900160 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642897c0c30 con 0x5642894cc4d0 2024-04-25T11:40:22.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.955+0000 7fa23e59c640 1 --2- 172.21.15.90:0/3128900160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894cc4d0 0x564289896d50 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fa23000aa30 tx=0x7fa23000aef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f65144b149a87e9b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:22.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.955+0000 7fa23ed9d640 1 --2- 172.21.15.90:0/4070423393 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa2280afe70 0x7fa2280b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:22.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.957+0000 7fa23ed9d640 1 --2- 172.21.15.90:0/4070423393 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa2280afe70 0x7fa2280b2380 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa2340096c0 tx=0x7fa234031400 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:22.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.967+0000 7fa2257fa640 1 -- 172.21.15.90:0/3128900160 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa230016070 con 0x5642894cc4d0 2024-04-25T11:40:22.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.968+0000 7fa2257fa640 1 -- 172.21.15.90:0/3128900160 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa23000ebd0 con 0x5642894cc4d0 2024-04-25T11:40:22.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.970+0000 7fa24bf7b740 1 -- 172.21.15.90:0/3128900160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894cc4d0 msgr2=0x564289896d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:22.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.970+0000 7fa24bf7b740 1 --2- 172.21.15.90:0/3128900160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894cc4d0 0x564289896d50 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fa23000aa30 tx=0x7fa23000aef0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.973+0000 7fa24bf7b740 1 -- 172.21.15.90:0/3128900160 shutdown_connections 2024-04-25T11:40:22.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.973+0000 7fa24bf7b740 1 --2- 172.21.15.90:0/3128900160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642894d46b0 0x5642898a36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.973+0000 7fa24bf7b740 1 --2- 172.21.15.90:0/3128900160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894cc4d0 0x564289896d50 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.973+0000 7fa24bf7b740 1 --2- 172.21.15.90:0/3128900160 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5642894f5390 0x564289896810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.973+0000 7fa24bf7b740 1 -- 172.21.15.90:0/3128900160 >> 172.21.15.90:0/3128900160 conn(0x5642894e94e0 msgr2=0x564289892fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:22.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.974+0000 7fa24bf7b740 1 -- 172.21.15.90:0/3128900160 shutdown_connections 2024-04-25T11:40:22.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.974+0000 7fa24bf7b740 1 -- 172.21.15.90:0/3128900160 wait complete. 2024-04-25T11:40:22.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.975+0000 7fa24bf7b740 1 -- start start 2024-04-25T11:40:22.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.975+0000 7fa24bf7b740 1 Processor -- start 2024-04-25T11:40:22.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.976+0000 7fa24bf7b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894cc4d0 0x5642898a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.976+0000 7fa24bf7b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642894d46b0 0x5642898a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.976+0000 7fa24bf7b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5642894f5390 0x5642898b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.976+0000 7fa24bf7b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5642898a3de0 con 0x5642894cc4d0 2024-04-25T11:40:22.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.976+0000 7fa24bf7b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x564289782dc0 con 0x5642894f5390 2024-04-25T11:40:22.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.976+0000 7fa24bf7b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5642898a3f60 con 0x5642894d46b0 2024-04-25T11:40:22.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.976+0000 7fa24bf7b740 10 client.0 _ll_register_callbacks cb 0x56428979e600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:22.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.977+0000 7fa23ed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894cc4d0 0x5642898a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:22.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.977+0000 7fa23ed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894cc4d0 0x5642898a11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60844/0 (socket says 172.21.15.90:60844) 2024-04-25T11:40:22.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.977+0000 7fa23ed9d640 1 -- 172.21.15.90:0/432143455 learned_addr learned my addr 172.21.15.90:0/432143455 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:22.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.977+0000 7fa23e59c640 1 --2- 172.21.15.90:0/432143455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642894d46b0 0x5642898a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:22.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.977+0000 7fa23f59e640 1 --2- 172.21.15.90:0/432143455 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5642894f5390 0x5642898b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:22.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.981+0000 7fa23ed9d640 1 -- 172.21.15.90:0/432143455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642894d46b0 msgr2=0x5642898a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:22.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.981+0000 7fa23ed9d640 1 --2- 172.21.15.90:0/432143455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642894d46b0 0x5642898a16e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.981+0000 7fa23ed9d640 1 -- 172.21.15.90:0/432143455 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5642894f5390 msgr2=0x5642898b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:22.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.981+0000 7fa23ed9d640 1 --2- 172.21.15.90:0/432143455 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5642894f5390 0x5642898b5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:22.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.981+0000 7fa23ed9d640 1 -- 172.21.15.90:0/432143455 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56428950d3f0 con 0x5642894cc4d0 2024-04-25T11:40:22.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.983+0000 7fa23ed9d640 1 --2- 172.21.15.90:0/432143455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894cc4d0 0x5642898a11a0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fa234009850 tx=0x7fa234005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64dbd1c4a3e3e000 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:22.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.987+0000 7fa222ff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:40:22.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.987+0000 7fa222ff5640 1 -- 172.21.15.90:0/432143455 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23404d020 con 0x5642894cc4d0 2024-04-25T11:40:22.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.987+0000 7fa222ff5640 1 -- 172.21.15.90:0/432143455 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fa234004250 con 0x5642894cc4d0 2024-04-25T11:40:22.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.988+0000 7fa24bf7b740 1 -- 172.21.15.90:0/432143455 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa23000ebd0 con 0x5642894cc4d0 2024-04-25T11:40:22.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.995+0000 7fa222ff5640 1 -- 172.21.15.90:0/432143455 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa234035040 con 0x5642894cc4d0 2024-04-25T11:40:22.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.995+0000 7fa24bf7b740 10 client.17145 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:22.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.995+0000 7fa24bf7b740 1 -- 172.21.15.90:0/432143455 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5642895d77d0 con 0x5642894cc4d0 2024-04-25T11:40:22.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.997+0000 7fa222ff5640 1 -- 172.21.15.90:0/432143455 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa23000ebd0 con 0x5642894cc4d0 2024-04-25T11:40:22.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:22.999+0000 7fa222ff5640 1 -- 172.21.15.90:0/432143455 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fa23405a020 con 0x5642894cc4d0 2024-04-25T11:40:22.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.000+0000 7fa24bf7b740 10 client.17145 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:22.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.000+0000 7fa24bf7b740 10 client.17145 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:40:22.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.000+0000 7fa24bf7b740 20 client.17145 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:22.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.000+0000 7fa24bf7b740 1 --2- 172.21.15.90:0/432143455 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5642894b3f60 0x5642898bc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:22.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.000+0000 7fa24bf7b740 4 client.17145 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:40:22.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.000+0000 7fa24bf7b740 1 -- 172.21.15.90:0/432143455 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564289809990 con 0x5642894b3f60 2024-04-25T11:40:22.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.004+0000 7fa23e59c640 1 --2- 172.21.15.90:0/432143455 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5642894b3f60 0x5642898bc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:23.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.016+0000 7fa23e59c640 1 --2- 172.21.15.90:0/432143455 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5642894b3f60 0x5642898bc410 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5dfd9b75f1ddb91e server_cookie=1325b971d6d0187 in_seq=0 out_seq=0 2024-04-25T11:40:23.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.017+0000 7fa222ff5640 10 client.17145 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:40:23.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.021+0000 7fa222ff5640 1 -- 172.21.15.90:0/432143455 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564289809990 con 0x5642894b3f60 2024-04-25T11:40:23.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.021+0000 7fa222ff5640 10 client.17145 handle_command_reply: tid=0 2024-04-25T11:40:23.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.021+0000 7fa24bf7b740 1 client.17145 shutdown 2024-04-25T11:40:23.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.025+0000 7fa24bf7b740 1 -- 172.21.15.90:0/432143455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894cc4d0 msgr2=0x5642898a11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:23.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.025+0000 7fa24bf7b740 1 --2- 172.21.15.90:0/432143455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894cc4d0 0x5642898a11a0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fa234009850 tx=0x7fa234005fd0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.027+0000 7fa24bf7b740 1 -- 172.21.15.90:0/432143455 shutdown_connections 2024-04-25T11:40:23.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.027+0000 7fa24bf7b740 1 --2- 172.21.15.90:0/432143455 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5642894b3f60 0x5642898bc410 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.027+0000 7fa24bf7b740 1 --2- 172.21.15.90:0/432143455 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5642894f5390 0x5642898b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.027+0000 7fa24bf7b740 1 --2- 172.21.15.90:0/432143455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642894d46b0 0x5642898a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.027+0000 7fa24bf7b740 1 --2- 172.21.15.90:0/432143455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894cc4d0 0x5642898a11a0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.027+0000 7fa24bf7b740 1 -- 172.21.15.90:0/432143455 >> 172.21.15.90:0/432143455 conn(0x5642894e94e0 msgr2=0x564289898530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:23.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.027+0000 7fa222ff5640 0 client.17145 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:40:23.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.027+0000 7fa24bf7b740 1 -- 172.21.15.90:0/432143455 shutdown_connections 2024-04-25T11:40:23.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.027+0000 7fa24bf7b740 1 -- 172.21.15.90:0/432143455 wait complete. 2024-04-25T11:40:23.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.027+0000 7fa24bf7b740 20 client.17145 trim_cache size 0 max 16384 2024-04-25T11:40:23.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.065+0000 7fa24bf7b740 1 Processor -- start 2024-04-25T11:40:23.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.065+0000 7fa24bf7b740 1 -- start start 2024-04-25T11:40:23.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.067+0000 7fa24bf7b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5642894cc4d0 0x56428989b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:23.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.067+0000 7fa24bf7b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894d46b0 0x5642898c1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:23.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.068+0000 7fa24bf7b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642894f5390 0x5642898c2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:23.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.069+0000 7fa24bf7b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa2401ad710 con 0x5642894d46b0 2024-04-25T11:40:23.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.069+0000 7fa23ed9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5642894cc4d0 0x56428989b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:23.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.069+0000 7fa23ed9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5642894cc4d0 0x56428989b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:32780/0 (socket says 172.21.15.90:32780) 2024-04-25T11:40:23.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.069+0000 7fa23ed9d640 1 -- 172.21.15.90:0/1560964333 learned_addr learned my addr 172.21.15.90:0/1560964333 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:23.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.069+0000 7fa24bf7b740 1 -- 172.21.15.90:0/1560964333 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa2401af250 con 0x5642894cc4d0 2024-04-25T11:40:23.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.069+0000 7fa24bf7b740 1 -- 172.21.15.90:0/1560964333 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa2401adde0 con 0x5642894f5390 2024-04-25T11:40:23.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.070+0000 7fa23e59c640 1 --2- 172.21.15.90:0/1560964333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894d46b0 0x5642898c1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:23.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.070+0000 7fa23f59e640 1 --2- 172.21.15.90:0/1560964333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642894f5390 0x5642898c2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:23.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.070+0000 7fa23e59c640 1 -- 172.21.15.90:0/1560964333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642894f5390 msgr2=0x5642898c2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:23.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.070+0000 7fa23e59c640 1 --2- 172.21.15.90:0/1560964333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642894f5390 0x5642898c2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.070+0000 7fa23e59c640 1 -- 172.21.15.90:0/1560964333 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5642894cc4d0 msgr2=0x56428989b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:23.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.070+0000 7fa23e59c640 1 --2- 172.21.15.90:0/1560964333 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5642894cc4d0 0x56428989b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.070+0000 7fa23e59c640 1 -- 172.21.15.90:0/1560964333 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642898bc9e0 con 0x5642894d46b0 2024-04-25T11:40:23.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.071+0000 7fa23e59c640 1 --2- 172.21.15.90:0/1560964333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894d46b0 0x5642898c1c10 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x56428988d3a0 tx=0x7fa23000a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e3e0c90de5f2bc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:23.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.072+0000 7fa224ff9640 1 -- 172.21.15.90:0/1560964333 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa230016040 con 0x5642894d46b0 2024-04-25T11:40:23.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.076+0000 7fa224ff9640 1 -- 172.21.15.90:0/1560964333 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa23000ee20 con 0x5642894d46b0 2024-04-25T11:40:23.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.076+0000 7fa224ff9640 1 -- 172.21.15.90:0/1560964333 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa230011810 con 0x5642894d46b0 2024-04-25T11:40:23.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.078+0000 7fa24bf7b740 1 -- 172.21.15.90:0/1560964333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894d46b0 msgr2=0x5642898c1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:23.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.079+0000 7fa24bf7b740 1 --2- 172.21.15.90:0/1560964333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894d46b0 0x5642898c1c10 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x56428988d3a0 tx=0x7fa23000a1f0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.081+0000 7fa24bf7b740 1 -- 172.21.15.90:0/1560964333 shutdown_connections 2024-04-25T11:40:23.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.081+0000 7fa24bf7b740 1 --2- 172.21.15.90:0/1560964333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642894f5390 0x5642898c2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.081+0000 7fa24bf7b740 1 --2- 172.21.15.90:0/1560964333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894d46b0 0x5642898c1c10 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.081+0000 7fa24bf7b740 1 --2- 172.21.15.90:0/1560964333 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5642894cc4d0 0x56428989b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.081+0000 7fa24bf7b740 1 -- 172.21.15.90:0/1560964333 >> 172.21.15.90:0/1560964333 conn(0x5642894e94e0 msgr2=0x5642898a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:23.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.091+0000 7fa24bf7b740 1 -- 172.21.15.90:0/1560964333 shutdown_connections 2024-04-25T11:40:23.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.092+0000 7fa24bf7b740 1 -- 172.21.15.90:0/1560964333 wait complete. 2024-04-25T11:40:23.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.093+0000 7fa24bf7b740 1 -- start start 2024-04-25T11:40:23.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.094+0000 7fa24bf7b740 1 Processor -- start 2024-04-25T11:40:23.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.094+0000 7fa24bf7b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5642894cc4d0 0x5642898acb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:23.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.095+0000 7fa24bf7b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894d46b0 0x5642898ad060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:23.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.095+0000 7fa23e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894d46b0 0x5642898ad060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:23.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.095+0000 7fa24bf7b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642894f5390 0x5642898ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:23.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.096+0000 7fa23ed9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5642894cc4d0 0x5642898acb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:23.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.096+0000 7fa23ed9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5642894cc4d0 0x5642898acb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:32792/0 (socket says 172.21.15.90:32792) 2024-04-25T11:40:23.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.096+0000 7fa23e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894d46b0 0x5642898ad060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60862/0 (socket says 172.21.15.90:60862) 2024-04-25T11:40:23.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.096+0000 7fa23f59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642894f5390 0x5642898ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:23.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.096+0000 7fa23f59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642894f5390 0x5642898ad5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:59578/0 (socket says 172.21.15.90:59578) 2024-04-25T11:40:23.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.097+0000 7fa23ed9d640 1 -- 172.21.15.90:0/939181162 learned_addr learned my addr 172.21.15.90:0/939181162 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:23.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.097+0000 7fa24bf7b740 1 -- 172.21.15.90:0/939181162 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa2401aeb80 con 0x5642894d46b0 2024-04-25T11:40:23.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.097+0000 7fa24bf7b740 1 -- 172.21.15.90:0/939181162 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa2401ae4b0 con 0x5642894cc4d0 2024-04-25T11:40:23.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.097+0000 7fa24bf7b740 1 -- 172.21.15.90:0/939181162 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa2401ab7a0 con 0x5642894f5390 2024-04-25T11:40:23.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.097+0000 7fa24bf7b740 10 client.0 _ll_register_callbacks cb 0x7fa234040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:23.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.098+0000 7fa23f59e640 1 -- 172.21.15.90:0/939181162 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5642894cc4d0 msgr2=0x5642898acb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:23.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.098+0000 7fa23f59e640 1 --2- 172.21.15.90:0/939181162 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5642894cc4d0 0x5642898acb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.098+0000 7fa23f59e640 1 -- 172.21.15.90:0/939181162 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894d46b0 msgr2=0x5642898ad060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:23.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.098+0000 7fa23f59e640 1 --2- 172.21.15.90:0/939181162 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894d46b0 0x5642898ad060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.098+0000 7fa23f59e640 1 -- 172.21.15.90:0/939181162 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564289922930 con 0x5642894f5390 2024-04-25T11:40:23.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.098+0000 7fa23e59c640 1 --2- 172.21.15.90:0/939181162 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894d46b0 0x5642898ad060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:40:23.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.099+0000 7fa23f59e640 1 --2- 172.21.15.90:0/939181162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642894f5390 0x5642898ad5a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa2380dd150 tx=0x7fa2380dd710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b244401df48ca6b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:23.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.099+0000 7fa222ff5640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:40:23.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.099+0000 7fa222ff5640 1 -- 172.21.15.90:0/939181162 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2380a82a0 con 0x5642894f5390 2024-04-25T11:40:23.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.099+0000 7fa222ff5640 1 -- 172.21.15.90:0/939181162 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fa2380a5af0 con 0x5642894f5390 2024-04-25T11:40:23.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.100+0000 7fa24bf7b740 1 -- 172.21.15.90:0/939181162 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa23000ee20 con 0x5642894f5390 2024-04-25T11:40:23.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.101+0000 7fa222ff5640 1 -- 172.21.15.90:0/939181162 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa2380a7ba0 con 0x5642894f5390 2024-04-25T11:40:23.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.101+0000 7fa24bf7b740 10 client.26230 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:23.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.101+0000 7fa24bf7b740 1 -- 172.21.15.90:0/939181162 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564289923c10 con 0x5642894f5390 2024-04-25T11:40:23.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.102+0000 7fa222ff5640 1 -- 172.21.15.90:0/939181162 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa23000ee20 con 0x5642894f5390 2024-04-25T11:40:23.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.102+0000 7fa222ff5640 1 -- 172.21.15.90:0/939181162 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fa238020070 con 0x5642894f5390 2024-04-25T11:40:23.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.103+0000 7fa24bf7b740 10 client.26230 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:23.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.103+0000 7fa24bf7b740 10 client.26230 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:40:23.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.103+0000 7fa24bf7b740 20 client.26230 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:23.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.103+0000 7fa24bf7b740 1 --2- 172.21.15.90:0/939181162 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5642894893a0 0x5642899263a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:23.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.103+0000 7fa24bf7b740 4 client.26230 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:40:23.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.103+0000 7fa24bf7b740 1 -- 172.21.15.90:0/939181162 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564289803800 con 0x5642894893a0 2024-04-25T11:40:23.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.104+0000 7fa23ed9d640 1 --2- 172.21.15.90:0/939181162 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5642894893a0 0x5642899263a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:23.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.108+0000 7fa222ff5640 10 client.26230 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:40:23.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.107+0000 7fa23ed9d640 1 --2- 172.21.15.90:0/939181162 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5642894893a0 0x5642899263a0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7f73bf91127e9f8a server_cookie=636d69b6babaa07c in_seq=0 out_seq=0 2024-04-25T11:40:23.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.109+0000 7fa222ff5640 1 -- 172.21.15.90:0/939181162 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564289803800 con 0x5642894893a0 2024-04-25T11:40:23.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.109+0000 7fa222ff5640 10 client.26230 handle_command_reply: tid=0 2024-04-25T11:40:23.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.109+0000 7fa24bf7b740 1 client.26230 shutdown 2024-04-25T11:40:23.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.110+0000 7fa24bf7b740 1 -- 172.21.15.90:0/939181162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642894f5390 msgr2=0x5642898ad5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:23.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.110+0000 7fa24bf7b740 1 --2- 172.21.15.90:0/939181162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642894f5390 0x5642898ad5a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa2380dd150 tx=0x7fa2380dd710 comp rx=0 tx=0).stop 2024-04-25T11:40:23.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.110+0000 7fa24bf7b740 1 -- 172.21.15.90:0/939181162 shutdown_connections 2024-04-25T11:40:23.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.110+0000 7fa24bf7b740 1 --2- 172.21.15.90:0/939181162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5642894f5390 0x5642898ad5a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.110+0000 7fa24bf7b740 1 --2- 172.21.15.90:0/939181162 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642894d46b0 0x5642898ad060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.110+0000 7fa24bf7b740 1 --2- 172.21.15.90:0/939181162 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5642894893a0 0x5642899263a0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.110+0000 7fa24bf7b740 1 --2- 172.21.15.90:0/939181162 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5642894cc4d0 0x5642898acb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.110+0000 7fa24bf7b740 1 -- 172.21.15.90:0/939181162 >> 172.21.15.90:0/939181162 conn(0x5642894e94e0 msgr2=0x5642898b47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:23.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.110+0000 7fa222ff5640 0 client.26230 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:40:23.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.110+0000 7fa24bf7b740 1 -- 172.21.15.90:0/939181162 shutdown_connections 2024-04-25T11:40:23.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.110+0000 7fa24bf7b740 1 -- 172.21.15.90:0/939181162 wait complete. 2024-04-25T11:40:23.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.110+0000 7fa24bf7b740 20 client.26230 trim_cache size 0 max 16384 2024-04-25T11:40:23.099 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:40:23.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.111+0000 7fa2450ca640 1 -- 172.21.15.90:0/4070423393 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa2280afe70 msgr2=0x7fa2280b2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:23.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.111+0000 7fa2450ca640 1 --2- 172.21.15.90:0/4070423393 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa2280afe70 0x7fa2280b2380 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa2340096c0 tx=0x7fa234031400 comp rx=0 tx=0).stop 2024-04-25T11:40:23.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.111+0000 7fa2450ca640 1 -- 172.21.15.90:0/4070423393 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa240107450 msgr2=0x7fa240198af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:23.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.111+0000 7fa2450ca640 1 --2- 172.21.15.90:0/4070423393 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa240107450 0x7fa240198af0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fa238009fd0 tx=0x7fa23800ef10 comp rx=0 tx=0).stop 2024-04-25T11:40:23.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.111+0000 7fa2450ca640 1 -- 172.21.15.90:0/4070423393 shutdown_connections 2024-04-25T11:40:23.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.111+0000 7fa2450ca640 1 --2- 172.21.15.90:0/4070423393 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa2280afe70 0x7fa2280b2380 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.111+0000 7fa2450ca640 1 --2- 172.21.15.90:0/4070423393 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa240107450 0x7fa240198af0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.111+0000 7fa2450ca640 1 --2- 172.21.15.90:0/4070423393 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa240106990 0x7fa24019ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.111+0000 7fa2450ca640 1 --2- 172.21.15.90:0/4070423393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa240102400 0x7fa24019e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.111+0000 7fa2450ca640 1 -- 172.21.15.90:0/4070423393 >> 172.21.15.90:0/4070423393 conn(0x7fa240077880 msgr2=0x7fa2400ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:23.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.111+0000 7fa2450ca640 1 -- 172.21.15.90:0/4070423393 shutdown_connections 2024-04-25T11:40:23.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.112+0000 7fa2450ca640 1 -- 172.21.15.90:0/4070423393 wait complete. 2024-04-25T11:40:23.117 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T11:40:23.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.448+0000 7f41d5f91640 1 -- 172.21.15.90:0/1726464400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f41c80a0f60 msgr2=0x7f41c80a3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:23.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.448+0000 7f41d5f91640 1 --2- 172.21.15.90:0/1726464400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f41c80a0f60 0x7f41c80a3450 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f41d00511c0 tx=0x7f41d0097600 comp rx=0 tx=0).stop 2024-04-25T11:40:23.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.448+0000 7f41d5f91640 1 -- 172.21.15.90:0/1726464400 shutdown_connections 2024-04-25T11:40:23.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.448+0000 7f41d5f91640 1 --2- 172.21.15.90:0/1726464400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f41c80a0f60 0x7f41c80a3450 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.448+0000 7f41d5f91640 1 --2- 172.21.15.90:0/1726464400 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41c809e5b0 0x7f41c80a09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.448+0000 7f41d5f91640 1 --2- 172.21.15.90:0/1726464400 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f41c8097b50 0x7f41c8097f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.448+0000 7f41d5f91640 1 -- 172.21.15.90:0/1726464400 >> 172.21.15.90:0/1726464400 conn(0x7f41c800aff0 msgr2=0x7f41c800b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:23.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.448+0000 7f41d5f91640 1 -- 172.21.15.90:0/1726464400 shutdown_connections 2024-04-25T11:40:23.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.448+0000 7f41d5f91640 1 -- 172.21.15.90:0/1726464400 wait complete. 2024-04-25T11:40:23.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.449+0000 7f41d5f91640 1 Processor -- start 2024-04-25T11:40:23.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.449+0000 7f41d5f91640 1 -- start start 2024-04-25T11:40:23.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.449+0000 7f41d5f91640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41c8097b50 0x7f41c812d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:23.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.449+0000 7f41d5f91640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f41c809e5b0 0x7f41c812da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:23.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.449+0000 7f41d5f91640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f41c80a0f60 0x7f41c8131d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:23.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.449+0000 7f41d5f91640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f41c80a6b20 con 0x7f41c80a0f60 2024-04-25T11:40:23.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.449+0000 7f41d5f91640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f41c80a69a0 con 0x7f41c809e5b0 2024-04-25T11:40:23.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.449+0000 7f41d5f91640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f41c80a6ca0 con 0x7f41c8097b50 2024-04-25T11:40:23.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.450+0000 7f41cffff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f41c809e5b0 0x7f41c812da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:23.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.450+0000 7f41d5790640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f41c80a0f60 0x7f41c8131d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:23.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.450+0000 7f41cffff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f41c809e5b0 0x7f41c812da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:32818/0 (socket says 172.21.15.90:32818) 2024-04-25T11:40:23.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.450+0000 7f41cffff640 1 -- 172.21.15.90:0/2047494146 learned_addr learned my addr 172.21.15.90:0/2047494146 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:23.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.450+0000 7f41d4f8f640 1 --2- 172.21.15.90:0/2047494146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41c8097b50 0x7f41c812d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:23.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.450+0000 7f41d4f8f640 1 -- 172.21.15.90:0/2047494146 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f41c809e5b0 msgr2=0x7f41c812da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:23.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.450+0000 7f41d4f8f640 1 --2- 172.21.15.90:0/2047494146 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f41c809e5b0 0x7f41c812da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.450+0000 7f41d4f8f640 1 -- 172.21.15.90:0/2047494146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f41c80a0f60 msgr2=0x7f41c8131d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:23.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.450+0000 7f41d4f8f640 1 --2- 172.21.15.90:0/2047494146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f41c80a0f60 0x7f41c8131d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.450+0000 7f41d4f8f640 1 -- 172.21.15.90:0/2047494146 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41c8132360 con 0x7f41c8097b50 2024-04-25T11:40:23.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.451+0000 7f41d5790640 1 --2- 172.21.15.90:0/2047494146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f41c80a0f60 0x7f41c8131d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:40:23.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.451+0000 7f41d4f8f640 1 --2- 172.21.15.90:0/2047494146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41c8097b50 0x7f41c812d500 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f41c400e9f0 tx=0x7f41c400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:23.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.451+0000 7f41cdffb640 1 -- 172.21.15.90:0/2047494146 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c400ccc0 con 0x7f41c8097b50 2024-04-25T11:40:23.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.451+0000 7f41d5f91640 1 -- 172.21.15.90:0/2047494146 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41c8132650 con 0x7f41c8097b50 2024-04-25T11:40:23.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.451+0000 7f41cdffb640 1 -- 172.21.15.90:0/2047494146 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41c40108e0 con 0x7f41c8097b50 2024-04-25T11:40:23.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.452+0000 7f41cdffb640 1 -- 172.21.15.90:0/2047494146 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c4018880 con 0x7f41c8097b50 2024-04-25T11:40:23.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.452+0000 7f41d5f91640 1 -- 172.21.15.90:0/2047494146 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41c8095340 con 0x7f41c8097b50 2024-04-25T11:40:23.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.455+0000 7f41cdffb640 1 -- 172.21.15.90:0/2047494146 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f41c4018a20 con 0x7f41c8097b50 2024-04-25T11:40:23.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.456+0000 7f41de17e740 1 Processor -- start 2024-04-25T11:40:23.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.456+0000 7f41de17e740 1 -- start start 2024-04-25T11:40:23.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.456+0000 7f41de17e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd5d2e6390 0x55dd5d6878d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:23.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.456+0000 7f41de17e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd5d2bd4d0 0x55dd5d687e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:23.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.457+0000 7f41d5790640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd5d2e6390 0x55dd5d6878d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:23.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.457+0000 7f41d5790640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd5d2e6390 0x55dd5d6878d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60892/0 (socket says 172.21.15.90:60892) 2024-04-25T11:40:23.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.457+0000 7f41de17e740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55dd5d2c56b0 0x55dd5d69d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:23.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.457+0000 7f41de17e740 1 -- 172.21.15.90:0/2425506764 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55dd5d5ecd80 con 0x55dd5d2e6390 2024-04-25T11:40:23.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.457+0000 7f41d5790640 1 -- 172.21.15.90:0/2425506764 learned_addr learned my addr 172.21.15.90:0/2425506764 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:23.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.457+0000 7f41de17e740 1 -- 172.21.15.90:0/2425506764 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55dd5d5a6a20 con 0x55dd5d2c56b0 2024-04-25T11:40:23.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.457+0000 7f41de17e740 1 -- 172.21.15.90:0/2425506764 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55dd5d5e8d40 con 0x55dd5d2bd4d0 2024-04-25T11:40:23.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.457+0000 7f41cdffb640 1 --2- 172.21.15.90:0/2047494146 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f41c00afce0 0x7f41c00b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:23.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.457+0000 7f41cffff640 1 --2- 172.21.15.90:0/2425506764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55dd5d2c56b0 0x55dd5d69d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:23.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.457+0000 7f41d4f8f640 1 --2- 172.21.15.90:0/2425506764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd5d2bd4d0 0x55dd5d687e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:23.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.458+0000 7f41cffff640 1 -- 172.21.15.90:0/2425506764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd5d2bd4d0 msgr2=0x55dd5d687e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:23.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.458+0000 7f41cffff640 1 --2- 172.21.15.90:0/2425506764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd5d2bd4d0 0x55dd5d687e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.458+0000 7f41cffff640 1 -- 172.21.15.90:0/2425506764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd5d2e6390 msgr2=0x55dd5d6878d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:23.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.458+0000 7f41cffff640 1 --2- 172.21.15.90:0/2425506764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd5d2e6390 0x55dd5d6878d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.458+0000 7f41cffff640 1 -- 172.21.15.90:0/2425506764 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd5d5b1c30 con 0x55dd5d2c56b0 2024-04-25T11:40:23.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.458+0000 7f41cdffb640 1 -- 172.21.15.90:0/2047494146 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f41c40df370 con 0x7f41c8097b50 2024-04-25T11:40:23.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.459+0000 7f41cffff640 1 --2- 172.21.15.90:0/2425506764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55dd5d2c56b0 0x55dd5d69d080 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f41bc010e40 tx=0x7f41bc005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c4796f72ef463d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:23.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.459+0000 7f41d5790640 1 --2- 172.21.15.90:0/2047494146 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f41c00afce0 0x7f41c00b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:23.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.459+0000 7f41baffd640 1 -- 172.21.15.90:0/2425506764 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc005020 con 0x55dd5d2c56b0 2024-04-25T11:40:23.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.460+0000 7f41baffd640 1 -- 172.21.15.90:0/2425506764 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41bc0047a0 con 0x55dd5d2c56b0 2024-04-25T11:40:23.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.460+0000 7f41baffd640 1 -- 172.21.15.90:0/2425506764 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc0196a0 con 0x55dd5d2c56b0 2024-04-25T11:40:23.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.463+0000 7f41de17e740 1 -- 172.21.15.90:0/2425506764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55dd5d2c56b0 msgr2=0x55dd5d69d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:23.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.463+0000 7f41de17e740 1 --2- 172.21.15.90:0/2425506764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55dd5d2c56b0 0x55dd5d69d080 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f41bc010e40 tx=0x7f41bc005f90 comp rx=0 tx=0).stop 2024-04-25T11:40:23.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.466+0000 7f41de17e740 1 -- 172.21.15.90:0/2425506764 shutdown_connections 2024-04-25T11:40:23.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.466+0000 7f41de17e740 1 --2- 172.21.15.90:0/2425506764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55dd5d2c56b0 0x55dd5d69d080 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.466+0000 7f41de17e740 1 --2- 172.21.15.90:0/2425506764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd5d2bd4d0 0x55dd5d687e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.466+0000 7f41de17e740 1 --2- 172.21.15.90:0/2425506764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd5d2e6390 0x55dd5d6878d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.466+0000 7f41de17e740 1 -- 172.21.15.90:0/2425506764 >> 172.21.15.90:0/2425506764 conn(0x55dd5d2da4e0 msgr2=0x55dd5d684190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:23.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.466+0000 7f41d5790640 1 --2- 172.21.15.90:0/2047494146 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f41c00afce0 0x7f41c00b21f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f41d0068910 tx=0x7f41d0097600 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:23.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.466+0000 7f41de17e740 1 -- 172.21.15.90:0/2425506764 shutdown_connections 2024-04-25T11:40:23.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.467+0000 7f41de17e740 1 -- 172.21.15.90:0/2425506764 wait complete. 2024-04-25T11:40:23.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.467+0000 7f41de17e740 1 -- start start 2024-04-25T11:40:23.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.467+0000 7f41de17e740 1 Processor -- start 2024-04-25T11:40:23.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.468+0000 7f41de17e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd5d2bd4d0 0x55dd5d69abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:23.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.468+0000 7f41de17e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd5d2c56b0 0x55dd5d69b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:23.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.468+0000 7f41de17e740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55dd5d2e6390 0x55dd5d6a6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:23.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.468+0000 7f41de17e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55dd5d69d770 con 0x55dd5d2bd4d0 2024-04-25T11:40:23.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.468+0000 7f41de17e740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55dd5d573dc0 con 0x55dd5d2e6390 2024-04-25T11:40:23.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.468+0000 7f41de17e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55dd5d69d8f0 con 0x55dd5d2c56b0 2024-04-25T11:40:23.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.468+0000 7f41de17e740 10 client.0 _ll_register_callbacks cb 0x55dd5d58f600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:23.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.468+0000 7f41d4f8f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd5d2bd4d0 0x55dd5d69abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:23.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.469+0000 7f41d4f8f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd5d2bd4d0 0x55dd5d69abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60902/0 (socket says 172.21.15.90:60902) 2024-04-25T11:40:23.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.469+0000 7f41d4f8f640 1 -- 172.21.15.90:0/3455763257 learned_addr learned my addr 172.21.15.90:0/3455763257 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:23.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.469+0000 7f41cffff640 1 --2- 172.21.15.90:0/3455763257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd5d2c56b0 0x55dd5d69b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:23.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.469+0000 7f41d5790640 1 --2- 172.21.15.90:0/3455763257 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55dd5d2e6390 0x55dd5d6a6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:23.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.469+0000 7f41cffff640 1 -- 172.21.15.90:0/3455763257 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55dd5d2e6390 msgr2=0x55dd5d6a6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:23.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.469+0000 7f41cffff640 1 --2- 172.21.15.90:0/3455763257 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55dd5d2e6390 0x55dd5d6a6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.469+0000 7f41cffff640 1 -- 172.21.15.90:0/3455763257 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd5d2bd4d0 msgr2=0x55dd5d69abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:23.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.469+0000 7f41cffff640 1 --2- 172.21.15.90:0/3455763257 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd5d2bd4d0 0x55dd5d69abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.469+0000 7f41cffff640 1 -- 172.21.15.90:0/3455763257 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd5d2fe3f0 con 0x55dd5d2c56b0 2024-04-25T11:40:23.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.469+0000 7f41cffff640 1 --2- 172.21.15.90:0/3455763257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd5d2c56b0 0x55dd5d69b130 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f41bc010e40 tx=0x7f41bc0050f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7a6377d44745e69 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:23.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.481+0000 7f41affff640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:40:23.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.481+0000 7f41affff640 1 -- 172.21.15.90:0/3455763257 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc029070 con 0x55dd5d2c56b0 2024-04-25T11:40:23.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.481+0000 7f41de17e740 1 -- 172.21.15.90:0/3455763257 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41bc0047a0 con 0x55dd5d2c56b0 2024-04-25T11:40:23.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.481+0000 7f41affff640 1 -- 172.21.15.90:0/3455763257 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f41bc025450 con 0x55dd5d2c56b0 2024-04-25T11:40:23.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.482+0000 7f41affff640 1 -- 172.21.15.90:0/3455763257 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f41bc035050 con 0x55dd5d2c56b0 2024-04-25T11:40:23.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:23 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3336889429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:40:23.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:23 smithi090 ceph-mon[29575]: pgmap v162: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 947 GiB / 1.0 TiB avail; 1.5 MiB/s rd, 321 MiB/s wr, 2.33k op/s 2024-04-25T11:40:23.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.487+0000 7f41de17e740 10 client.26242 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:23.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.487+0000 7f41de17e740 1 -- 172.21.15.90:0/3455763257 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dd5d3c87d0 con 0x55dd5d2c56b0 2024-04-25T11:40:23.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.489+0000 7f41affff640 1 -- 172.21.15.90:0/3455763257 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41bc0047a0 con 0x55dd5d2c56b0 2024-04-25T11:40:23.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.491+0000 7f41affff640 1 -- 172.21.15.90:0/3455763257 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f41bc019460 con 0x55dd5d2c56b0 2024-04-25T11:40:23.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.493+0000 7f41de17e740 10 client.26242 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:23.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.493+0000 7f41de17e740 10 client.26242 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:40:23.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.493+0000 7f41de17e740 20 client.26242 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:23.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.493+0000 7f41de17e740 1 --2- 172.21.15.90:0/3455763257 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55dd5d2a4f60 0x55dd5d6ace90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:23.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.493+0000 7f41de17e740 4 client.26242 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:40:23.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.493+0000 7f41de17e740 1 -- 172.21.15.90:0/3455763257 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd5d5fa990 con 0x55dd5d2a4f60 2024-04-25T11:40:23.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.497+0000 7f41d4f8f640 1 --2- 172.21.15.90:0/3455763257 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55dd5d2a4f60 0x55dd5d6ace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:23.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.505+0000 7f41d4f8f640 1 --2- 172.21.15.90:0/3455763257 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55dd5d2a4f60 0x55dd5d6ace90 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ad8b986b995bd0c7 server_cookie=be962a916f33933e in_seq=0 out_seq=0 2024-04-25T11:40:23.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.511+0000 7f41affff640 10 client.26242 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:40:23.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.511+0000 7f41affff640 1 -- 172.21.15.90:0/3455763257 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55dd5d5fa990 con 0x55dd5d2a4f60 2024-04-25T11:40:23.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.511+0000 7f41affff640 10 client.26242 handle_command_reply: tid=0 2024-04-25T11:40:23.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.511+0000 7f41de17e740 1 client.26242 shutdown 2024-04-25T11:40:23.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.511+0000 7f41de17e740 1 -- 172.21.15.90:0/3455763257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd5d2c56b0 msgr2=0x55dd5d69b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:23.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.511+0000 7f41de17e740 1 --2- 172.21.15.90:0/3455763257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd5d2c56b0 0x55dd5d69b130 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f41bc010e40 tx=0x7f41bc0050f0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.517+0000 7f41de17e740 1 -- 172.21.15.90:0/3455763257 shutdown_connections 2024-04-25T11:40:23.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.517+0000 7f41de17e740 1 --2- 172.21.15.90:0/3455763257 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55dd5d2a4f60 0x55dd5d6ace90 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.517+0000 7f41de17e740 1 --2- 172.21.15.90:0/3455763257 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55dd5d2e6390 0x55dd5d6a6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.517+0000 7f41de17e740 1 --2- 172.21.15.90:0/3455763257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd5d2c56b0 0x55dd5d69b130 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.517+0000 7f41de17e740 1 --2- 172.21.15.90:0/3455763257 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd5d2bd4d0 0x55dd5d69abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.517+0000 7f41de17e740 1 -- 172.21.15.90:0/3455763257 >> 172.21.15.90:0/3455763257 conn(0x55dd5d2da4e0 msgr2=0x55dd5d6829c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:23.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.517+0000 7f41affff640 0 client.26242 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:40:23.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.530+0000 7f41de17e740 1 -- 172.21.15.90:0/3455763257 shutdown_connections 2024-04-25T11:40:23.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.530+0000 7f41de17e740 1 -- 172.21.15.90:0/3455763257 wait complete. 2024-04-25T11:40:23.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.534+0000 7f41de17e740 20 client.26242 trim_cache size 0 max 16384 2024-04-25T11:40:23.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:23 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3336889429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:40:23.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:23 smithi138 ceph-mon[32328]: pgmap v162: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 947 GiB / 1.0 TiB avail; 1.5 MiB/s rd, 321 MiB/s wr, 2.33k op/s 2024-04-25T11:40:23.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.582+0000 7f41de17e740 1 Processor -- start 2024-04-25T11:40:23.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.585+0000 7f41de17e740 1 -- start start 2024-04-25T11:40:23.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.586+0000 7f41de17e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd5d2bd4d0 0x55dd5d6a6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:23.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.586+0000 7f41de17e740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55dd5d2c56b0 0x55dd5d687770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:23.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.586+0000 7f41de17e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd5d2e6390 0x55dd5d687cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:23.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.586+0000 7f41de17e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f41c81406c0 con 0x55dd5d2bd4d0 2024-04-25T11:40:23.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.586+0000 7f41de17e740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f41c8142200 con 0x55dd5d2c56b0 2024-04-25T11:40:23.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.586+0000 7f41de17e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f41c8140d90 con 0x55dd5d2e6390 2024-04-25T11:40:23.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.587+0000 7f41d4f8f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd5d2bd4d0 0x55dd5d6a6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:23.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.587+0000 7f41d4f8f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd5d2bd4d0 0x55dd5d6a6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60904/0 (socket says 172.21.15.90:60904) 2024-04-25T11:40:23.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.587+0000 7f41d4f8f640 1 -- 172.21.15.90:0/3979286122 learned_addr learned my addr 172.21.15.90:0/3979286122 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:23.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.588+0000 7f41d4f8f640 1 -- 172.21.15.90:0/3979286122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd5d2e6390 msgr2=0x55dd5d687cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:40:23.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.588+0000 7f41d4f8f640 1 --2- 172.21.15.90:0/3979286122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd5d2e6390 0x55dd5d687cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.588+0000 7f41d4f8f640 1 -- 172.21.15.90:0/3979286122 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55dd5d2c56b0 msgr2=0x55dd5d687770 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:40:23.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.588+0000 7f41d4f8f640 1 --2- 172.21.15.90:0/3979286122 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55dd5d2c56b0 0x55dd5d687770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.588+0000 7f41d4f8f640 1 -- 172.21.15.90:0/3979286122 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd5d675ba0 con 0x55dd5d2bd4d0 2024-04-25T11:40:23.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.588+0000 7f41d4f8f640 1 --2- 172.21.15.90:0/3979286122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd5d2bd4d0 0x55dd5d6a6c20 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f41c40dd060 tx=0x7f41c40a34d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff82bfeac4049328 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:23.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.591+0000 7f41ba7fc640 1 -- 172.21.15.90:0/3979286122 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c40a6cd0 con 0x55dd5d2bd4d0 2024-04-25T11:40:23.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.592+0000 7f41ba7fc640 1 -- 172.21.15.90:0/3979286122 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41c40a7070 con 0x55dd5d2bd4d0 2024-04-25T11:40:23.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.593+0000 7f41de17e740 1 -- 172.21.15.90:0/3979286122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd5d2bd4d0 msgr2=0x55dd5d6a6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:23.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.593+0000 7f41de17e740 1 --2- 172.21.15.90:0/3979286122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd5d2bd4d0 0x55dd5d6a6c20 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f41c40dd060 tx=0x7f41c40a34d0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:23 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3336889429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:40:23.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:23 smithi156 ceph-mon[33045]: pgmap v162: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 947 GiB / 1.0 TiB avail; 1.5 MiB/s rd, 321 MiB/s wr, 2.33k op/s 2024-04-25T11:40:23.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.600+0000 7f41de17e740 1 -- 172.21.15.90:0/3979286122 shutdown_connections 2024-04-25T11:40:23.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.600+0000 7f41de17e740 1 --2- 172.21.15.90:0/3979286122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd5d2e6390 0x55dd5d687cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.600+0000 7f41de17e740 1 --2- 172.21.15.90:0/3979286122 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55dd5d2c56b0 0x55dd5d687770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.600+0000 7f41de17e740 1 --2- 172.21.15.90:0/3979286122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd5d2bd4d0 0x55dd5d6a6c20 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.600+0000 7f41de17e740 1 -- 172.21.15.90:0/3979286122 >> 172.21.15.90:0/3979286122 conn(0x55dd5d2da4e0 msgr2=0x55dd5d695ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:23.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.602+0000 7f41de17e740 1 -- 172.21.15.90:0/3979286122 shutdown_connections 2024-04-25T11:40:23.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.602+0000 7f41de17e740 1 -- 172.21.15.90:0/3979286122 wait complete. 2024-04-25T11:40:23.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.603+0000 7f41de17e740 1 -- start start 2024-04-25T11:40:23.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.603+0000 7f41de17e740 1 Processor -- start 2024-04-25T11:40:23.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.604+0000 7f41de17e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd5d2bd4d0 0x55dd5d6b0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:23.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.604+0000 7f41de17e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd5d2c56b0 0x55dd5d70d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:23.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.604+0000 7f41de17e740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55dd5d2e6390 0x55dd5d70dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:23.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.604+0000 7f41de17e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f41c8141b30 con 0x55dd5d2c56b0 2024-04-25T11:40:23.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.604+0000 7f41de17e740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f41c8141460 con 0x55dd5d2e6390 2024-04-25T11:40:23.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.604+0000 7f41de17e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f41c813e750 con 0x55dd5d2bd4d0 2024-04-25T11:40:23.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.604+0000 7f41de17e740 10 client.0 _ll_register_callbacks cb 0x7f41c4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:23.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.607+0000 7f41d5790640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55dd5d2e6390 0x55dd5d70dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:23.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.607+0000 7f41cffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd5d2c56b0 0x55dd5d70d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:23.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.607+0000 7f41cffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd5d2c56b0 0x55dd5d70d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60920/0 (socket says 172.21.15.90:60920) 2024-04-25T11:40:23.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.607+0000 7f41cffff640 1 -- 172.21.15.90:0/590969733 learned_addr learned my addr 172.21.15.90:0/590969733 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:23.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.607+0000 7f41d4f8f640 1 --2- 172.21.15.90:0/590969733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd5d2bd4d0 0x55dd5d6b0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:23.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.610+0000 7f41d4f8f640 1 -- 172.21.15.90:0/590969733 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55dd5d2e6390 msgr2=0x55dd5d70dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:23.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.610+0000 7f41d4f8f640 1 --2- 172.21.15.90:0/590969733 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55dd5d2e6390 0x55dd5d70dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.610+0000 7f41d4f8f640 1 -- 172.21.15.90:0/590969733 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd5d2c56b0 msgr2=0x55dd5d70d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:23.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.610+0000 7f41d4f8f640 1 --2- 172.21.15.90:0/590969733 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd5d2c56b0 0x55dd5d70d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.610+0000 7f41d4f8f640 1 -- 172.21.15.90:0/590969733 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd5d70e010 con 0x55dd5d2bd4d0 2024-04-25T11:40:23.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.611+0000 7f41d4f8f640 1 --2- 172.21.15.90:0/590969733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd5d2bd4d0 0x55dd5d6b0150 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f41c40c3010 tx=0x7f41c40a3a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb2e09b62a49ff72 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:23.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.611+0000 7f41affff640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:40:23.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.613+0000 7f41affff640 1 -- 172.21.15.90:0/590969733 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c40900c0 con 0x55dd5d2bd4d0 2024-04-25T11:40:23.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.613+0000 7f41affff640 1 -- 172.21.15.90:0/590969733 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f41c40b0090 con 0x55dd5d2bd4d0 2024-04-25T11:40:23.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.615+0000 7f41de17e740 1 -- 172.21.15.90:0/590969733 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41c40a7070 con 0x55dd5d2bd4d0 2024-04-25T11:40:23.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.617+0000 7f41affff640 1 -- 172.21.15.90:0/590969733 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f41c40ab860 con 0x55dd5d2bd4d0 2024-04-25T11:40:23.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.618+0000 7f41de17e740 10 client.26245 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:23.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.618+0000 7f41de17e740 1 -- 172.21.15.90:0/590969733 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dd5d6afa90 con 0x55dd5d2bd4d0 2024-04-25T11:40:23.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.621+0000 7f41affff640 1 -- 172.21.15.90:0/590969733 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41c40a7070 con 0x55dd5d2bd4d0 2024-04-25T11:40:23.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.622+0000 7f41affff640 1 -- 172.21.15.90:0/590969733 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f41c40a2ed0 con 0x55dd5d2bd4d0 2024-04-25T11:40:23.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.623+0000 7f41de17e740 10 client.26245 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:23.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.623+0000 7f41de17e740 10 client.26245 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:40:23.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.623+0000 7f41de17e740 20 client.26245 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:23.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.623+0000 7f41de17e740 1 --2- 172.21.15.90:0/590969733 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55dd5d27a3a0 0x55dd5d68cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:23.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.623+0000 7f41de17e740 4 client.26245 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:40:23.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.623+0000 7f41de17e740 1 -- 172.21.15.90:0/590969733 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dd5d5f4800 con 0x55dd5d27a3a0 2024-04-25T11:40:23.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.624+0000 7f41cffff640 1 --2- 172.21.15.90:0/590969733 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55dd5d27a3a0 0x55dd5d68cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:23.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.628+0000 7f41cffff640 1 --2- 172.21.15.90:0/590969733 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55dd5d27a3a0 0x55dd5d68cc60 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=49b1913b2aa67e3e server_cookie=a2d9cbbe863a14cd in_seq=0 out_seq=0 2024-04-25T11:40:23.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.628+0000 7f41affff640 10 client.26245 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:40:23.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.630+0000 7f41affff640 1 -- 172.21.15.90:0/590969733 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dd5d5f4800 con 0x55dd5d27a3a0 2024-04-25T11:40:23.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.630+0000 7f41affff640 10 client.26245 handle_command_reply: tid=0 2024-04-25T11:40:23.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.630+0000 7f41de17e740 1 client.26245 shutdown 2024-04-25T11:40:23.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.633+0000 7f41de17e740 1 -- 172.21.15.90:0/590969733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd5d2bd4d0 msgr2=0x55dd5d6b0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:23.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.633+0000 7f41de17e740 1 --2- 172.21.15.90:0/590969733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd5d2bd4d0 0x55dd5d6b0150 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f41c40c3010 tx=0x7f41c40a3a20 comp rx=0 tx=0).stop 2024-04-25T11:40:23.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.635+0000 7f41de17e740 1 -- 172.21.15.90:0/590969733 shutdown_connections 2024-04-25T11:40:23.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.635+0000 7f41de17e740 1 --2- 172.21.15.90:0/590969733 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55dd5d27a3a0 0x55dd5d68cc60 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.635+0000 7f41de17e740 1 --2- 172.21.15.90:0/590969733 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55dd5d2e6390 0x55dd5d70dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.635+0000 7f41de17e740 1 --2- 172.21.15.90:0/590969733 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd5d2c56b0 0x55dd5d70d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.635+0000 7f41de17e740 1 --2- 172.21.15.90:0/590969733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd5d2bd4d0 0x55dd5d6b0150 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.635+0000 7f41de17e740 1 -- 172.21.15.90:0/590969733 >> 172.21.15.90:0/590969733 conn(0x55dd5d2da4e0 msgr2=0x55dd5d695ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:23.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.635+0000 7f41affff640 0 client.26245 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:40:23.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.635+0000 7f41de17e740 1 -- 172.21.15.90:0/590969733 shutdown_connections 2024-04-25T11:40:23.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.639+0000 7f41de17e740 1 -- 172.21.15.90:0/590969733 wait complete. 2024-04-25T11:40:23.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.639+0000 7f41de17e740 20 client.26245 trim_cache size 0 max 16384 2024-04-25T11:40:23.629 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:40:23.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.643+0000 7f41d5f91640 1 -- 172.21.15.90:0/2047494146 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f41c00afce0 msgr2=0x7f41c00b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:23.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.643+0000 7f41d5f91640 1 --2- 172.21.15.90:0/2047494146 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f41c00afce0 0x7f41c00b21f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f41d0068910 tx=0x7f41d0097600 comp rx=0 tx=0).stop 2024-04-25T11:40:23.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.643+0000 7f41d5f91640 1 -- 172.21.15.90:0/2047494146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41c8097b50 msgr2=0x7f41c812d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:23.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.643+0000 7f41d5f91640 1 --2- 172.21.15.90:0/2047494146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41c8097b50 0x7f41c812d500 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f41c400e9f0 tx=0x7f41c400eeb0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.648+0000 7f41d5f91640 1 -- 172.21.15.90:0/2047494146 shutdown_connections 2024-04-25T11:40:23.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.649+0000 7f41d5f91640 1 --2- 172.21.15.90:0/2047494146 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f41c00afce0 0x7f41c00b21f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.649+0000 7f41d5f91640 1 --2- 172.21.15.90:0/2047494146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f41c80a0f60 0x7f41c8131d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.649+0000 7f41d5f91640 1 --2- 172.21.15.90:0/2047494146 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f41c809e5b0 0x7f41c812da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.649+0000 7f41d5f91640 1 --2- 172.21.15.90:0/2047494146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41c8097b50 0x7f41c812d500 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.649+0000 7f41d5f91640 1 -- 172.21.15.90:0/2047494146 >> 172.21.15.90:0/2047494146 conn(0x7f41c800aff0 msgr2=0x7f41c8094070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:23.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.649+0000 7f41d5f91640 1 -- 172.21.15.90:0/2047494146 shutdown_connections 2024-04-25T11:40:23.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:23.649+0000 7f41d5f91640 1 -- 172.21.15.90:0/2047494146 wait complete. 2024-04-25T11:40:23.671 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T11:40:23.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.000+0000 7f6c011b2640 1 -- 172.21.15.90:0/2991533865 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bfc10a5d0 msgr2=0x7f6bfc10a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:23.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.000+0000 7f6c011b2640 1 --2- 172.21.15.90:0/2991533865 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bfc10a5d0 0x7f6bfc10a9e0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4005fd0 tx=0x7f6bf402a590 comp rx=0 tx=0).stop 2024-04-25T11:40:23.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.004+0000 7f6c011b2640 1 -- 172.21.15.90:0/2991533865 shutdown_connections 2024-04-25T11:40:23.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.004+0000 7f6c011b2640 1 --2- 172.21.15.90:0/2991533865 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6bfc10f620 0x7f6bfc116dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.004+0000 7f6c011b2640 1 --2- 172.21.15.90:0/2991533865 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6bfc10eb60 0x7f6bfc10ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.004+0000 7f6c011b2640 1 --2- 172.21.15.90:0/2991533865 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bfc10a5d0 0x7f6bfc10a9e0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:23.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.004+0000 7f6c011b2640 1 -- 172.21.15.90:0/2991533865 >> 172.21.15.90:0/2991533865 conn(0x7f6bfc06eac0 msgr2=0x7f6bfc06eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:23.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.012+0000 7f6c011b2640 1 -- 172.21.15.90:0/2991533865 shutdown_connections 2024-04-25T11:40:24.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.016+0000 7f6c011b2640 1 -- 172.21.15.90:0/2991533865 wait complete. 2024-04-25T11:40:24.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.017+0000 7f6c011b2640 1 Processor -- start 2024-04-25T11:40:24.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.017+0000 7f6c011b2640 1 -- start start 2024-04-25T11:40:24.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.017+0000 7f6c011b2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bfc10a5d0 0x7f6bfc1a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.017+0000 7f6c011b2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6bfc10eb60 0x7f6bfc1a99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.017+0000 7f6c011b2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6bfc10f620 0x7f6bfc1adc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.018+0000 7f6c011b2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc1229b0 con 0x7f6bfc10a5d0 2024-04-25T11:40:24.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.018+0000 7f6c011b2640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc122830 con 0x7f6bfc10eb60 2024-04-25T11:40:24.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.018+0000 7f6c011b2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc122b30 con 0x7f6bfc10f620 2024-04-25T11:40:24.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.018+0000 7f6c009b1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6bfc10f620 0x7f6bfc1adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:24.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.018+0000 7f6c009b1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6bfc10f620 0x7f6bfc1adc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:59654/0 (socket says 172.21.15.90:59654) 2024-04-25T11:40:24.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.018+0000 7f6c009b1640 1 -- 172.21.15.90:0/2978658821 learned_addr learned my addr 172.21.15.90:0/2978658821 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:24.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.019+0000 7f6c009b1640 1 -- 172.21.15.90:0/2978658821 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6bfc10eb60 msgr2=0x7f6bfc1a99b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:40:24.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.019+0000 7f6c009b1640 1 --2- 172.21.15.90:0/2978658821 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6bfc10eb60 0x7f6bfc1a99b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.019+0000 7f6c009b1640 1 -- 172.21.15.90:0/2978658821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bfc10a5d0 msgr2=0x7f6bfc1a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:24.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.019+0000 7f6c009b1640 1 --2- 172.21.15.90:0/2978658821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bfc10a5d0 0x7f6bfc1a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.019+0000 7f6c009b1640 1 -- 172.21.15.90:0/2978658821 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bfc1ae2b0 con 0x7f6bfc10f620 2024-04-25T11:40:24.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.023+0000 7f6c009b1640 1 --2- 172.21.15.90:0/2978658821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6bfc10f620 0x7f6bfc1adc70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6bec00bed0 tx=0x7f6bec0095b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:24.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.023+0000 7f6bf97fa640 1 -- 172.21.15.90:0/2978658821 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bec01a020 con 0x7f6bfc10f620 2024-04-25T11:40:24.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.023+0000 7f6c011b2640 1 -- 172.21.15.90:0/2978658821 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bfc1ae5a0 con 0x7f6bfc10f620 2024-04-25T11:40:24.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.023+0000 7f6c011b2640 1 -- 172.21.15.90:0/2978658821 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bfc1aea90 con 0x7f6bfc10f620 2024-04-25T11:40:24.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.024+0000 7f6bf97fa640 1 -- 172.21.15.90:0/2978658821 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bec009e40 con 0x7f6bfc10f620 2024-04-25T11:40:24.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.024+0000 7f6bf97fa640 1 -- 172.21.15.90:0/2978658821 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bec004b10 con 0x7f6bfc10f620 2024-04-25T11:40:24.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.028+0000 7f6bf97fa640 1 -- 172.21.15.90:0/2978658821 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f6bec004cb0 con 0x7f6bfc10f620 2024-04-25T11:40:24.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.029+0000 7f6bf97fa640 1 --2- 172.21.15.90:0/2978658821 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f6be40afe60 0x7f6be40b2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.030+0000 7f6bf97fa640 1 -- 172.21.15.90:0/2978658821 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f6bec0dfbf0 con 0x7f6bfc10f620 2024-04-25T11:40:24.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.033+0000 7f6bfbfff640 1 --2- 172.21.15.90:0/2978658821 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f6be40afe60 0x7f6be40b2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:24.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.033+0000 7f6c0939b740 1 Processor -- start 2024-04-25T11:40:24.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.034+0000 7f6c0939b740 1 -- start start 2024-04-25T11:40:24.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.034+0000 7f6c0939b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555df4233390 0x555df45d49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.034+0000 7f6c0939b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555df420a4d0 0x555df45d4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.034+0000 7f6c0939b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555df42126b0 0x555df45e1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.034+0000 7f6c0939b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555df4539d80 con 0x555df420a4d0 2024-04-25T11:40:24.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.034+0000 7f6c0939b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x555df44f3a20 con 0x555df4233390 2024-04-25T11:40:24.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.034+0000 7f6c0939b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x555df4535d40 con 0x555df42126b0 2024-04-25T11:40:24.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.035+0000 7f6c009b1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555df42126b0 0x555df45e1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:24.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.035+0000 7f6c009b1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555df42126b0 0x555df45e1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:59666/0 (socket says 172.21.15.90:59666) 2024-04-25T11:40:24.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.035+0000 7f6c009b1640 1 -- 172.21.15.90:0/3962377740 learned_addr learned my addr 172.21.15.90:0/3962377740 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:24.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.035+0000 7f6c009b1640 1 -- 172.21.15.90:0/3962377740 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555df4233390 msgr2=0x555df45d49e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:40:24.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.035+0000 7f6c009b1640 1 --2- 172.21.15.90:0/3962377740 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555df4233390 0x555df45d49e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.035+0000 7f6c009b1640 1 -- 172.21.15.90:0/3962377740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555df420a4d0 msgr2=0x555df45d4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:24.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.035+0000 7f6c009b1640 1 --2- 172.21.15.90:0/3962377740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555df420a4d0 0x555df45d4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.035+0000 7f6c009b1640 1 -- 172.21.15.90:0/3962377740 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555df44fec30 con 0x555df42126b0 2024-04-25T11:40:24.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.036+0000 7f6c009b1640 1 --2- 172.21.15.90:0/3962377740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555df42126b0 0x555df45e1870 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6bec009ab0 tx=0x7f6bec0a7870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b14d49514cdea91e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:24.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.036+0000 7f6be27fc640 1 -- 172.21.15.90:0/3962377740 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bec0a7d60 con 0x555df42126b0 2024-04-25T11:40:24.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.036+0000 7f6be27fc640 1 -- 172.21.15.90:0/3962377740 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bec0ac110 con 0x555df42126b0 2024-04-25T11:40:24.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.038+0000 7f6c0939b740 1 -- 172.21.15.90:0/3962377740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555df42126b0 msgr2=0x555df45e1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:24.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.038+0000 7f6c0939b740 1 --2- 172.21.15.90:0/3962377740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555df42126b0 0x555df45e1870 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6bec009ab0 tx=0x7f6bec0a7870 comp rx=0 tx=0).stop 2024-04-25T11:40:24.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.038+0000 7f6c0939b740 1 -- 172.21.15.90:0/3962377740 shutdown_connections 2024-04-25T11:40:24.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.038+0000 7f6c0939b740 1 --2- 172.21.15.90:0/3962377740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555df42126b0 0x555df45e1870 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.038+0000 7f6c0939b740 1 --2- 172.21.15.90:0/3962377740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555df420a4d0 0x555df45d4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.038+0000 7f6c0939b740 1 --2- 172.21.15.90:0/3962377740 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555df4233390 0x555df45d49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.038+0000 7f6c0939b740 1 -- 172.21.15.90:0/3962377740 >> 172.21.15.90:0/3962377740 conn(0x555df42274e0 msgr2=0x555df45d1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:24.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.042+0000 7f6c0939b740 1 -- 172.21.15.90:0/3962377740 shutdown_connections 2024-04-25T11:40:24.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.046+0000 7f6c0939b740 1 -- 172.21.15.90:0/3962377740 wait complete. 2024-04-25T11:40:24.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.047+0000 7f6c0939b740 1 -- start start 2024-04-25T11:40:24.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.047+0000 7f6c0939b740 1 Processor -- start 2024-04-25T11:40:24.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.048+0000 7f6c0939b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555df420a4d0 0x555df45ded70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.048+0000 7f6c0939b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555df4233390 0x555df45df2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.048+0000 7f6c0939b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555df41f1f60 0x555df45df7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.048+0000 7f6c0939b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555df45e2000 con 0x555df420a4d0 2024-04-25T11:40:24.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.048+0000 7f6c0939b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x555df44c0dc0 con 0x555df4233390 2024-04-25T11:40:24.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.048+0000 7f6c0939b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x555df45e2180 con 0x555df41f1f60 2024-04-25T11:40:24.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.048+0000 7f6c0939b740 10 client.0 _ll_register_callbacks cb 0x555df44dc600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:24.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.048+0000 7f6bfbfff640 1 --2- 172.21.15.90:0/2978658821 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f6be40afe60 0x7f6be40b2370 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x555df45d1b50 tx=0x7f6bf4000f40 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:24.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.049+0000 7f6bfb7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555df4233390 0x555df45df2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:24.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.049+0000 7f6bfb7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555df4233390 0x555df45df2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:32876/0 (socket says 172.21.15.90:32876) 2024-04-25T11:40:24.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.049+0000 7f6bfb7fe640 1 -- 172.21.15.90:0/3343734620 learned_addr learned my addr 172.21.15.90:0/3343734620 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:24.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.050+0000 7f6c009b1640 1 --2- 172.21.15.90:0/3343734620 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555df41f1f60 0x555df45df7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:24.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.051+0000 7f6bfb7fe640 1 -- 172.21.15.90:0/3343734620 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555df41f1f60 msgr2=0x555df45df7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:24.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.051+0000 7f6bfb7fe640 1 --2- 172.21.15.90:0/3343734620 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555df41f1f60 0x555df45df7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.051+0000 7f6bfb7fe640 1 -- 172.21.15.90:0/3343734620 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555df420a4d0 msgr2=0x555df45ded70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:24.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.051+0000 7f6bfb7fe640 1 --2- 172.21.15.90:0/3343734620 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555df420a4d0 0x555df45ded70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.051+0000 7f6bfb7fe640 1 -- 172.21.15.90:0/3343734620 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555df424b3f0 con 0x555df4233390 2024-04-25T11:40:24.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.053+0000 7f6bfb7fe640 1 --2- 172.21.15.90:0/3343734620 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555df4233390 0x555df45df2b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0009db0 tx=0x7f6bf000b4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ed45f1eb611d43e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:24.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.053+0000 7f6bdb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:40:24.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.054+0000 7f6bdb7fe640 1 -- 172.21.15.90:0/3343734620 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf0015020 con 0x555df4233390 2024-04-25T11:40:24.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.054+0000 7f6c0939b740 1 -- 172.21.15.90:0/3343734620 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555df4547990 con 0x555df4233390 2024-04-25T11:40:24.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.054+0000 7f6bdb7fe640 1 -- 172.21.15.90:0/3343734620 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f6bf0011420 con 0x555df4233390 2024-04-25T11:40:24.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.057+0000 7f6bdb7fe640 1 -- 172.21.15.90:0/3343734620 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6bf0011620 con 0x555df4233390 2024-04-25T11:40:24.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.057+0000 7f6c0939b740 10 client.26303 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:24.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.057+0000 7f6c0939b740 1 -- 172.21.15.90:0/3343734620 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555df43157d0 con 0x555df4233390 2024-04-25T11:40:24.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.061+0000 7f6bdb7fe640 1 -- 172.21.15.90:0/3343734620 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x555df4547990 con 0x555df4233390 2024-04-25T11:40:24.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.061+0000 7f6bdb7fe640 1 -- 172.21.15.90:0/3343734620 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f6bf0022020 con 0x555df4233390 2024-04-25T11:40:24.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.061+0000 7f6c0939b740 10 client.26303 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:24.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.061+0000 7f6c0939b740 10 client.26303 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:40:24.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.061+0000 7f6c0939b740 20 client.26303 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:24.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.061+0000 7f6c0939b740 1 --2- 172.21.15.90:0/3343734620 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x555df41c73a0 0x555df45f1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.061+0000 7f6c0939b740 4 client.26303 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:40:24.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.061+0000 7f6c0939b740 1 -- 172.21.15.90:0/3343734620 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555df4541800 con 0x555df41c73a0 2024-04-25T11:40:24.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.070+0000 7f6bfbfff640 1 --2- 172.21.15.90:0/3343734620 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x555df41c73a0 0x555df45f1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:24.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.080+0000 7f6bfbfff640 1 --2- 172.21.15.90:0/3343734620 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x555df41c73a0 0x555df45f1bd0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2de02a8b7892612e server_cookie=53cf783fe482df11 in_seq=0 out_seq=0 2024-04-25T11:40:24.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.081+0000 7f6bdb7fe640 10 client.26303 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:40:24.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.083+0000 7f6bdb7fe640 1 -- 172.21.15.90:0/3343734620 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x555df4541800 con 0x555df41c73a0 2024-04-25T11:40:24.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.084+0000 7f6bdb7fe640 10 client.26303 handle_command_reply: tid=0 2024-04-25T11:40:24.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.085+0000 7f6c0939b740 1 client.26303 shutdown 2024-04-25T11:40:24.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.089+0000 7f6c0939b740 1 -- 172.21.15.90:0/3343734620 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555df4233390 msgr2=0x555df45df2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:24.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.089+0000 7f6c0939b740 1 --2- 172.21.15.90:0/3343734620 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555df4233390 0x555df45df2b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0009db0 tx=0x7f6bf000b4b0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.092+0000 7f6c0939b740 1 -- 172.21.15.90:0/3343734620 shutdown_connections 2024-04-25T11:40:24.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.092+0000 7f6c0939b740 1 --2- 172.21.15.90:0/3343734620 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x555df41c73a0 0x555df45f1bd0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.092+0000 7f6c0939b740 1 --2- 172.21.15.90:0/3343734620 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555df41f1f60 0x555df45df7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.092+0000 7f6c0939b740 1 --2- 172.21.15.90:0/3343734620 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555df4233390 0x555df45df2b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.092+0000 7f6c0939b740 1 --2- 172.21.15.90:0/3343734620 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555df420a4d0 0x555df45ded70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.092+0000 7f6c0939b740 1 -- 172.21.15.90:0/3343734620 >> 172.21.15.90:0/3343734620 conn(0x555df42274e0 msgr2=0x555df45d6700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:24.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.093+0000 7f6bdb7fe640 0 client.26303 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:40:24.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.097+0000 7f6c0939b740 1 -- 172.21.15.90:0/3343734620 shutdown_connections 2024-04-25T11:40:24.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.100+0000 7f6c0939b740 1 -- 172.21.15.90:0/3343734620 wait complete. 2024-04-25T11:40:24.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.100+0000 7f6c0939b740 20 client.26303 trim_cache size 0 max 16384 2024-04-25T11:40:24.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.159+0000 7f6c0939b740 1 Processor -- start 2024-04-25T11:40:24.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.160+0000 7f6c0939b740 1 -- start start 2024-04-25T11:40:24.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.160+0000 7f6c0939b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555df41f1f60 0x555df45e14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.161+0000 7f6c0939b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555df420a4d0 0x555df45d9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.161+0000 7f6c0939b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555df4233390 0x555df45df2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.161+0000 7f6c0939b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6be800eab0 con 0x555df41f1f60 2024-04-25T11:40:24.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.161+0000 7f6c0939b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6be80105f0 con 0x555df4233390 2024-04-25T11:40:24.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.161+0000 7f6c0939b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6be800f180 con 0x555df420a4d0 2024-04-25T11:40:24.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.162+0000 7f6c009b1640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555df4233390 0x555df45df2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:24.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.162+0000 7f6bfbfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555df41f1f60 0x555df45e14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:24.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.162+0000 7f6bfbfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555df41f1f60 0x555df45e14a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60978/0 (socket says 172.21.15.90:60978) 2024-04-25T11:40:24.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.162+0000 7f6bfb7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555df420a4d0 0x555df45d9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:24.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.162+0000 7f6bfbfff640 1 -- 172.21.15.90:0/1268195912 learned_addr learned my addr 172.21.15.90:0/1268195912 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:24.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.165+0000 7f6c009b1640 1 -- 172.21.15.90:0/1268195912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555df420a4d0 msgr2=0x555df45d9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:24.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.165+0000 7f6c009b1640 1 --2- 172.21.15.90:0/1268195912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555df420a4d0 0x555df45d9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.165+0000 7f6c009b1640 1 -- 172.21.15.90:0/1268195912 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555df41f1f60 msgr2=0x555df45e14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:24.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.165+0000 7f6c009b1640 1 --2- 172.21.15.90:0/1268195912 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555df41f1f60 0x555df45e14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.165+0000 7f6c009b1640 1 -- 172.21.15.90:0/1268195912 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555df45c2ba0 con 0x555df4233390 2024-04-25T11:40:24.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.167+0000 7f6c009b1640 1 --2- 172.21.15.90:0/1268195912 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555df4233390 0x555df45df2c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6bec0a7a60 tx=0x7f6bec0a74d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cda7807ca92e3ced server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:24.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.169+0000 7f6be1ffb640 1 -- 172.21.15.90:0/1268195912 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bec0c6070 con 0x555df4233390 2024-04-25T11:40:24.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.169+0000 7f6be1ffb640 1 -- 172.21.15.90:0/1268195912 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bec0a6fc0 con 0x555df4233390 2024-04-25T11:40:24.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.169+0000 7f6be1ffb640 1 -- 172.21.15.90:0/1268195912 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bec00cb30 con 0x555df4233390 2024-04-25T11:40:24.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.172+0000 7f6c0939b740 1 -- 172.21.15.90:0/1268195912 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555df4233390 msgr2=0x555df45df2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:24.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.172+0000 7f6c0939b740 1 --2- 172.21.15.90:0/1268195912 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555df4233390 0x555df45df2c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6bec0a7a60 tx=0x7f6bec0a74d0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.175+0000 7f6c0939b740 1 -- 172.21.15.90:0/1268195912 shutdown_connections 2024-04-25T11:40:24.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.175+0000 7f6c0939b740 1 --2- 172.21.15.90:0/1268195912 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555df4233390 0x555df45df2c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.175+0000 7f6c0939b740 1 --2- 172.21.15.90:0/1268195912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555df420a4d0 0x555df45d9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.175+0000 7f6c0939b740 1 --2- 172.21.15.90:0/1268195912 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555df41f1f60 0x555df45e14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.175+0000 7f6c0939b740 1 -- 172.21.15.90:0/1268195912 >> 172.21.15.90:0/1268195912 conn(0x555df42274e0 msgr2=0x555df45dd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:24.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.176+0000 7f6c0939b740 1 -- 172.21.15.90:0/1268195912 shutdown_connections 2024-04-25T11:40:24.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.177+0000 7f6c0939b740 1 -- 172.21.15.90:0/1268195912 wait complete. 2024-04-25T11:40:24.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.177+0000 7f6c0939b740 1 -- start start 2024-04-25T11:40:24.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.178+0000 7f6c0939b740 1 Processor -- start 2024-04-25T11:40:24.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.179+0000 7f6c0939b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555df41f1f60 0x555df45e2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.179+0000 7f6c0939b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555df420a4d0 0x555df45e2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.179+0000 7f6c0939b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555df4233390 0x555df45e2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.179+0000 7f6c0939b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6be800ff20 con 0x555df420a4d0 2024-04-25T11:40:24.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.179+0000 7f6c0939b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6be800f850 con 0x555df4233390 2024-04-25T11:40:24.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.179+0000 7f6c0939b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6be800ca90 con 0x555df41f1f60 2024-04-25T11:40:24.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.180+0000 7f6c0939b740 10 client.0 _ll_register_callbacks cb 0x7f6bf0006e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:24.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.180+0000 7f6bfb7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555df420a4d0 0x555df45e2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:24.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.180+0000 7f6bfb7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555df420a4d0 0x555df45e2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60992/0 (socket says 172.21.15.90:60992) 2024-04-25T11:40:24.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.180+0000 7f6bfb7fe640 1 -- 172.21.15.90:0/342635471 learned_addr learned my addr 172.21.15.90:0/342635471 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:24.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.183+0000 7f6c009b1640 1 --2- 172.21.15.90:0/342635471 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555df4233390 0x555df45e2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:24.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.183+0000 7f6bfbfff640 1 --2- 172.21.15.90:0/342635471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555df41f1f60 0x555df45e2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:24.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.183+0000 7f6bfb7fe640 1 -- 172.21.15.90:0/342635471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555df41f1f60 msgr2=0x555df45e2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:24.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.183+0000 7f6bfb7fe640 1 --2- 172.21.15.90:0/342635471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555df41f1f60 0x555df45e2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.183+0000 7f6bfb7fe640 1 -- 172.21.15.90:0/342635471 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555df4233390 msgr2=0x555df45e2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:24.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.183+0000 7f6bfb7fe640 1 --2- 172.21.15.90:0/342635471 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555df4233390 0x555df45e2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.183+0000 7f6bfb7fe640 1 -- 172.21.15.90:0/342635471 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555df4657f70 con 0x555df420a4d0 2024-04-25T11:40:24.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.184+0000 7f6c009b1640 1 --2- 172.21.15.90:0/342635471 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555df4233390 0x555df45e2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:40:24.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.185+0000 7f6bfb7fe640 1 --2- 172.21.15.90:0/342635471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555df420a4d0 0x555df45e2a50 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f6bf001ad80 tx=0x7f6bf001a3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca8349efb9057d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:24.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.186+0000 7f6bdb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:40:24.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.186+0000 7f6bdb7fe640 1 -- 172.21.15.90:0/342635471 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf0015040 con 0x555df420a4d0 2024-04-25T11:40:24.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.186+0000 7f6bdb7fe640 1 -- 172.21.15.90:0/342635471 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f6bf001a8f0 con 0x555df420a4d0 2024-04-25T11:40:24.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.187+0000 7f6c0939b740 1 -- 172.21.15.90:0/342635471 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555df44bc610 con 0x555df420a4d0 2024-04-25T11:40:24.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.190+0000 7f6bdb7fe640 1 -- 172.21.15.90:0/342635471 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6bf0004040 con 0x555df420a4d0 2024-04-25T11:40:24.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.190+0000 7f6c0939b740 10 client.17178 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:24.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.190+0000 7f6c0939b740 1 -- 172.21.15.90:0/342635471 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555df46591f0 con 0x555df420a4d0 2024-04-25T11:40:24.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.191+0000 7f6bdb7fe640 1 -- 172.21.15.90:0/342635471 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x555df44bc610 con 0x555df420a4d0 2024-04-25T11:40:24.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.191+0000 7f6bdb7fe640 1 -- 172.21.15.90:0/342635471 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f6bf0022040 con 0x555df420a4d0 2024-04-25T11:40:24.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.192+0000 7f6c0939b740 10 client.17178 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:24.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.192+0000 7f6c0939b740 10 client.17178 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:40:24.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.192+0000 7f6c0939b740 20 client.17178 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:24.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.192+0000 7f6c0939b740 1 --2- 172.21.15.90:0/342635471 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x555df46599a0 0x555df465bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.192+0000 7f6c0939b740 4 client.17178 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:40:24.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.192+0000 7f6c0939b740 1 -- 172.21.15.90:0/342635471 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555df450a240 con 0x555df46599a0 2024-04-25T11:40:24.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.194+0000 7f6bfbfff640 1 --2- 172.21.15.90:0/342635471 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x555df46599a0 0x555df465bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:24.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.198+0000 7f6bfbfff640 1 --2- 172.21.15.90:0/342635471 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x555df46599a0 0x555df465bdc0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7cb3eb6e1d7d619a server_cookie=a62b25e99ce229a2 in_seq=0 out_seq=0 2024-04-25T11:40:24.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.198+0000 7f6bdb7fe640 10 client.17178 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:40:24.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.199+0000 7f6bdb7fe640 1 -- 172.21.15.90:0/342635471 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555df450a240 con 0x555df46599a0 2024-04-25T11:40:24.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.199+0000 7f6bdb7fe640 10 client.17178 handle_command_reply: tid=0 2024-04-25T11:40:24.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.199+0000 7f6c0939b740 1 client.17178 shutdown 2024-04-25T11:40:24.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.200+0000 7f6c0939b740 1 -- 172.21.15.90:0/342635471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555df420a4d0 msgr2=0x555df45e2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:24.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.200+0000 7f6c0939b740 1 --2- 172.21.15.90:0/342635471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555df420a4d0 0x555df45e2a50 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f6bf001ad80 tx=0x7f6bf001a3f0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.200+0000 7f6c0939b740 1 -- 172.21.15.90:0/342635471 shutdown_connections 2024-04-25T11:40:24.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.200+0000 7f6c0939b740 1 --2- 172.21.15.90:0/342635471 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x555df46599a0 0x555df465bdc0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.201+0000 7f6c0939b740 1 --2- 172.21.15.90:0/342635471 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555df4233390 0x555df45e2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.201+0000 7f6bdb7fe640 0 client.17178 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:40:24.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.201+0000 7f6c0939b740 1 --2- 172.21.15.90:0/342635471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555df420a4d0 0x555df45e2a50 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.201+0000 7f6c0939b740 1 --2- 172.21.15.90:0/342635471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555df41f1f60 0x555df45e2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.201+0000 7f6c0939b740 1 -- 172.21.15.90:0/342635471 >> 172.21.15.90:0/342635471 conn(0x555df42274e0 msgr2=0x555df45cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:24.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.201+0000 7f6c0939b740 1 -- 172.21.15.90:0/342635471 shutdown_connections 2024-04-25T11:40:24.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.201+0000 7f6c0939b740 1 -- 172.21.15.90:0/342635471 wait complete. 2024-04-25T11:40:24.191 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:40:24.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.203+0000 7f6c0939b740 20 client.17178 trim_cache size 0 max 16384 2024-04-25T11:40:24.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.206+0000 7f6c011b2640 1 -- 172.21.15.90:0/2978658821 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f6be40afe60 msgr2=0x7f6be40b2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:24.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.206+0000 7f6c011b2640 1 --2- 172.21.15.90:0/2978658821 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f6be40afe60 0x7f6be40b2370 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x555df45d1b50 tx=0x7f6bf4000f40 comp rx=0 tx=0).stop 2024-04-25T11:40:24.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.206+0000 7f6c011b2640 1 -- 172.21.15.90:0/2978658821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6bfc10f620 msgr2=0x7f6bfc1adc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:24.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.206+0000 7f6c011b2640 1 --2- 172.21.15.90:0/2978658821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6bfc10f620 0x7f6bfc1adc70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6bec00bed0 tx=0x7f6bec0095b0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.206+0000 7f6c011b2640 1 -- 172.21.15.90:0/2978658821 shutdown_connections 2024-04-25T11:40:24.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.206+0000 7f6c011b2640 1 --2- 172.21.15.90:0/2978658821 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f6be40afe60 0x7f6be40b2370 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.206+0000 7f6c011b2640 1 --2- 172.21.15.90:0/2978658821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6bfc10f620 0x7f6bfc1adc70 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.206+0000 7f6c011b2640 1 --2- 172.21.15.90:0/2978658821 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6bfc10eb60 0x7f6bfc1a99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.206+0000 7f6c011b2640 1 --2- 172.21.15.90:0/2978658821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bfc10a5d0 0x7f6bfc1a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.207+0000 7f6c011b2640 1 -- 172.21.15.90:0/2978658821 >> 172.21.15.90:0/2978658821 conn(0x7f6bfc06eac0 msgr2=0x7f6bfc10dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:24.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.207+0000 7f6c011b2640 1 -- 172.21.15.90:0/2978658821 shutdown_connections 2024-04-25T11:40:24.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.207+0000 7f6c011b2640 1 -- 172.21.15.90:0/2978658821 wait complete. 2024-04-25T11:40:24.220 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T11:40:24.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.564+0000 7ff48e1ff640 1 -- 172.21.15.90:0/684807676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff488107440 msgr2=0x7ff48810ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:24.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.564+0000 7ff48e1ff640 1 --2- 172.21.15.90:0/684807676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff488107440 0x7ff48810ebe0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff48000adb0 tx=0x7ff48002b7c0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.564+0000 7ff48e1ff640 1 -- 172.21.15.90:0/684807676 shutdown_connections 2024-04-25T11:40:24.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.564+0000 7ff48e1ff640 1 --2- 172.21.15.90:0/684807676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff488107440 0x7ff48810ebe0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.564+0000 7ff48e1ff640 1 --2- 172.21.15.90:0/684807676 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff488106980 0x7ff488106db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.564+0000 7ff48e1ff640 1 --2- 172.21.15.90:0/684807676 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff488102440 0x7ff488102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.564+0000 7ff48e1ff640 1 -- 172.21.15.90:0/684807676 >> 172.21.15.90:0/684807676 conn(0x7ff488077880 msgr2=0x7ff488077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:24.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.566+0000 7ff48e1ff640 1 -- 172.21.15.90:0/684807676 shutdown_connections 2024-04-25T11:40:24.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.566+0000 7ff48e1ff640 1 -- 172.21.15.90:0/684807676 wait complete. 2024-04-25T11:40:24.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.567+0000 7ff48e1ff640 1 Processor -- start 2024-04-25T11:40:24.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.570+0000 7ff48e1ff640 1 -- start start 2024-04-25T11:40:24.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.571+0000 7ff48e1ff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff488102440 0x7ff488198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.571+0000 7ff48e1ff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff488106980 0x7ff488198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.571+0000 7ff48e1ff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff488107440 0x7ff48819cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.571+0000 7ff48e1ff640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff488111bb0 con 0x7ff488107440 2024-04-25T11:40:24.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.571+0000 7ff487fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff488107440 0x7ff48819cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:24.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.571+0000 7ff487fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff488107440 0x7ff48819cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:32774/0 (socket says 172.21.15.90:32774) 2024-04-25T11:40:24.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.571+0000 7ff487fff640 1 -- 172.21.15.90:0/3957259899 learned_addr learned my addr 172.21.15.90:0/3957259899 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:24.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.571+0000 7ff48e1ff640 1 -- 172.21.15.90:0/3957259899 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff488111a30 con 0x7ff488102440 2024-04-25T11:40:24.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.571+0000 7ff48e1ff640 1 -- 172.21.15.90:0/3957259899 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff488111d30 con 0x7ff488106980 2024-04-25T11:40:24.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.572+0000 7ff487fff640 1 -- 172.21.15.90:0/3957259899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff488106980 msgr2=0x7ff488198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:24.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.572+0000 7ff487fff640 1 --2- 172.21.15.90:0/3957259899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff488106980 0x7ff488198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.572+0000 7ff487fff640 1 -- 172.21.15.90:0/3957259899 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff488102440 msgr2=0x7ff488198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:24.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.572+0000 7ff487fff640 1 --2- 172.21.15.90:0/3957259899 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff488102440 0x7ff488198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.572+0000 7ff487fff640 1 -- 172.21.15.90:0/3957259899 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff48819d290 con 0x7ff488107440 2024-04-25T11:40:24.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.573+0000 7ff487fff640 1 --2- 172.21.15.90:0/3957259899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff488107440 0x7ff48819cc50 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7ff480030740 tx=0x7ff480030770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:24.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.573+0000 7ff484ff9640 1 -- 172.21.15.90:0/3957259899 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff480040070 con 0x7ff488107440 2024-04-25T11:40:24.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.573+0000 7ff484ff9640 1 -- 172.21.15.90:0/3957259899 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff480045b10 con 0x7ff488107440 2024-04-25T11:40:24.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.573+0000 7ff484ff9640 1 -- 172.21.15.90:0/3957259899 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48003cac0 con 0x7ff488107440 2024-04-25T11:40:24.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.573+0000 7ff48e1ff640 1 -- 172.21.15.90:0/3957259899 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff48819d520 con 0x7ff488107440 2024-04-25T11:40:24.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.573+0000 7ff48e1ff640 1 -- 172.21.15.90:0/3957259899 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff48819d9a0 con 0x7ff488107440 2024-04-25T11:40:24.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.577+0000 7ff484ff9640 1 -- 172.21.15.90:0/3957259899 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7ff480045660 con 0x7ff488107440 2024-04-25T11:40:24.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.578+0000 7ff484ff9640 1 --2- 172.21.15.90:0/3957259899 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff4700b41a0 0x7ff4700b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.579+0000 7ff484ff9640 1 -- 172.21.15.90:0/3957259899 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7ff480100940 con 0x7ff488107440 2024-04-25T11:40:24.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.579+0000 7ff4877fe640 1 --2- 172.21.15.90:0/3957259899 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff4700b41a0 0x7ff4700b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:24.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.580+0000 7ff49518f740 1 Processor -- start 2024-04-25T11:40:24.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.580+0000 7ff4877fe640 1 --2- 172.21.15.90:0/3957259899 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff4700b41a0 0x7ff4700b66b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff478009710 tx=0x7ff478006c90 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:24.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.580+0000 7ff49518f740 1 -- start start 2024-04-25T11:40:24.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.581+0000 7ff49518f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b2fef7a390 0x55b2ff31b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.581+0000 7ff49518f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2fef514d0 0x55b2ff31bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.581+0000 7ff49518f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef596b0 0x55b2ff3286a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.581+0000 7ff49518f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b2ff280d80 con 0x55b2fef596b0 2024-04-25T11:40:24.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.581+0000 7ff49518f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55b2ff23aa20 con 0x55b2fef7a390 2024-04-25T11:40:24.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.581+0000 7ff49518f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b2ff27cd40 con 0x55b2fef514d0 2024-04-25T11:40:24.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.581+0000 7ff487fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef596b0 0x55b2ff3286a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:24.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.581+0000 7ff487fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef596b0 0x55b2ff3286a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:32786/0 (socket says 172.21.15.90:32786) 2024-04-25T11:40:24.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.581+0000 7ff487fff640 1 -- 172.21.15.90:0/8786303 learned_addr learned my addr 172.21.15.90:0/8786303 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:24.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.581+0000 7ff487fff640 1 -- 172.21.15.90:0/8786303 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2fef514d0 msgr2=0x55b2ff31bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:24.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.581+0000 7ff487fff640 1 --2- 172.21.15.90:0/8786303 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2fef514d0 0x55b2ff31bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.581+0000 7ff487fff640 1 -- 172.21.15.90:0/8786303 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b2fef7a390 msgr2=0x55b2ff31b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:24.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.581+0000 7ff487fff640 1 --2- 172.21.15.90:0/8786303 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b2fef7a390 0x55b2ff31b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.581+0000 7ff487fff640 1 -- 172.21.15.90:0/8786303 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2ff245c30 con 0x55b2fef596b0 2024-04-25T11:40:24.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.582+0000 7ff487fff640 1 --2- 172.21.15.90:0/8786303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef596b0 0x55b2ff3286a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7ff4800c8720 tx=0x7ff480005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75697540677da6d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:24.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.582+0000 7ff46dffb640 1 -- 172.21.15.90:0/8786303 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff480045950 con 0x55b2fef596b0 2024-04-25T11:40:24.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.582+0000 7ff46dffb640 1 -- 172.21.15.90:0/8786303 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff48000a9e0 con 0x55b2fef596b0 2024-04-25T11:40:24.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.582+0000 7ff46dffb640 1 -- 172.21.15.90:0/8786303 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4800d1880 con 0x55b2fef596b0 2024-04-25T11:40:24.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.583+0000 7ff49518f740 1 -- 172.21.15.90:0/8786303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef596b0 msgr2=0x55b2ff3286a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:24.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.583+0000 7ff49518f740 1 --2- 172.21.15.90:0/8786303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef596b0 0x55b2ff3286a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7ff4800c8720 tx=0x7ff480005fd0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.583+0000 7ff49518f740 1 -- 172.21.15.90:0/8786303 shutdown_connections 2024-04-25T11:40:24.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.583+0000 7ff49518f740 1 --2- 172.21.15.90:0/8786303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef596b0 0x55b2ff3286a0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.583+0000 7ff49518f740 1 --2- 172.21.15.90:0/8786303 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2fef514d0 0x55b2ff31bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.583+0000 7ff49518f740 1 --2- 172.21.15.90:0/8786303 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b2fef7a390 0x55b2ff31b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.583+0000 7ff49518f740 1 -- 172.21.15.90:0/8786303 >> 172.21.15.90:0/8786303 conn(0x55b2fef6e4e0 msgr2=0x55b2ff317fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:24.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.583+0000 7ff49518f740 1 -- 172.21.15.90:0/8786303 shutdown_connections 2024-04-25T11:40:24.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.583+0000 7ff49518f740 1 -- 172.21.15.90:0/8786303 wait complete. 2024-04-25T11:40:24.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.584+0000 7ff49518f740 1 -- start start 2024-04-25T11:40:24.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.584+0000 7ff49518f740 1 Processor -- start 2024-04-25T11:40:24.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.585+0000 7ff49518f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b2fef514d0 0x55b2ff3261c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.585+0000 7ff49518f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef596b0 0x55b2ff326700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.585+0000 7ff49518f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2fef7a390 0x55b2ff3320a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.585+0000 7ff49518f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b2ff328be0 con 0x55b2fef596b0 2024-04-25T11:40:24.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.585+0000 7ff49518f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55b2ff207dc0 con 0x55b2fef514d0 2024-04-25T11:40:24.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.585+0000 7ff49518f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b2ff328d60 con 0x55b2fef7a390 2024-04-25T11:40:24.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.585+0000 7ff49518f740 10 client.0 _ll_register_callbacks cb 0x55b2ff223600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:24.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.585+0000 7ff486ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef596b0 0x55b2ff326700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:24.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.585+0000 7ff486ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef596b0 0x55b2ff326700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:32792/0 (socket says 172.21.15.90:32792) 2024-04-25T11:40:24.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.585+0000 7ff486ffd640 1 -- 172.21.15.90:0/1188874575 learned_addr learned my addr 172.21.15.90:0/1188874575 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:24.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.586+0000 7ff486ffd640 1 -- 172.21.15.90:0/1188874575 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2fef7a390 msgr2=0x55b2ff3320a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:24.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.586+0000 7ff486ffd640 1 --2- 172.21.15.90:0/1188874575 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2fef7a390 0x55b2ff3320a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.586+0000 7ff486ffd640 1 -- 172.21.15.90:0/1188874575 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b2fef514d0 msgr2=0x55b2ff3261c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:24.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.586+0000 7ff486ffd640 1 --2- 172.21.15.90:0/1188874575 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b2fef514d0 0x55b2ff3261c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.586+0000 7ff486ffd640 1 -- 172.21.15.90:0/1188874575 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2fef923f0 con 0x55b2fef596b0 2024-04-25T11:40:24.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.586+0000 7ff486ffd640 1 --2- 172.21.15.90:0/1188874575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef596b0 0x55b2ff326700 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7ff47c009db0 tx=0x7ff47c00b4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60e666dae93d364c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:24.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.586+0000 7ff466ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:40:24.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.587+0000 7ff466ffd640 1 -- 172.21.15.90:0/1188874575 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47c015020 con 0x55b2fef596b0 2024-04-25T11:40:24.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.587+0000 7ff466ffd640 1 -- 172.21.15.90:0/1188874575 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7ff47c011420 con 0x55b2fef596b0 2024-04-25T11:40:24.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.587+0000 7ff49518f740 1 -- 172.21.15.90:0/1188874575 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b2ff28e990 con 0x55b2fef596b0 2024-04-25T11:40:24.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.588+0000 7ff466ffd640 1 -- 172.21.15.90:0/1188874575 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff47c020050 con 0x55b2fef596b0 2024-04-25T11:40:24.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.588+0000 7ff49518f740 10 client.17187 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:24.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.588+0000 7ff49518f740 1 -- 172.21.15.90:0/1188874575 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b2ff05c7d0 con 0x55b2fef596b0 2024-04-25T11:40:24.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.589+0000 7ff466ffd640 1 -- 172.21.15.90:0/1188874575 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55b2ff28e990 con 0x55b2fef596b0 2024-04-25T11:40:24.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.589+0000 7ff466ffd640 1 -- 172.21.15.90:0/1188874575 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7ff47c019830 con 0x55b2fef596b0 2024-04-25T11:40:24.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.589+0000 7ff49518f740 10 client.17187 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:24.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.589+0000 7ff49518f740 10 client.17187 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:40:24.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.589+0000 7ff49518f740 20 client.17187 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:24.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.589+0000 7ff49518f740 1 --2- 172.21.15.90:0/1188874575 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b2fef38f60 0x55b2ff338400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.589+0000 7ff49518f740 4 client.17187 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:40:24.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.590+0000 7ff49518f740 1 -- 172.21.15.90:0/1188874575 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b2ff288800 con 0x55b2fef38f60 2024-04-25T11:40:24.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.590+0000 7ff4877fe640 1 --2- 172.21.15.90:0/1188874575 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b2fef38f60 0x55b2ff338400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:24.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.591+0000 7ff466ffd640 10 client.17187 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:24.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.591+0000 7ff4877fe640 1 --2- 172.21.15.90:0/1188874575 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b2fef38f60 0x55b2ff338400 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1f0b0bcda073578 server_cookie=21f92d0ed41617d6 in_seq=0 out_seq=0 2024-04-25T11:40:24.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.595+0000 7ff466ffd640 1 -- 172.21.15.90:0/1188874575 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b2ff288800 con 0x55b2fef38f60 2024-04-25T11:40:24.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.595+0000 7ff466ffd640 10 client.17187 handle_command_reply: tid=0 2024-04-25T11:40:24.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.595+0000 7ff49518f740 1 client.17187 shutdown 2024-04-25T11:40:24.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.595+0000 7ff49518f740 1 -- 172.21.15.90:0/1188874575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef596b0 msgr2=0x55b2ff326700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:24.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.595+0000 7ff49518f740 1 --2- 172.21.15.90:0/1188874575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef596b0 0x55b2ff326700 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7ff47c009db0 tx=0x7ff47c00b4b0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.595+0000 7ff49518f740 1 -- 172.21.15.90:0/1188874575 shutdown_connections 2024-04-25T11:40:24.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.595+0000 7ff49518f740 1 --2- 172.21.15.90:0/1188874575 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b2fef38f60 0x55b2ff338400 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.595+0000 7ff49518f740 1 --2- 172.21.15.90:0/1188874575 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2fef7a390 0x55b2ff3320a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.595+0000 7ff466ffd640 0 client.17187 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:24.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.595+0000 7ff49518f740 1 --2- 172.21.15.90:0/1188874575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef596b0 0x55b2ff326700 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.595+0000 7ff49518f740 1 --2- 172.21.15.90:0/1188874575 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b2fef514d0 0x55b2ff3261c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.595+0000 7ff49518f740 1 -- 172.21.15.90:0/1188874575 >> 172.21.15.90:0/1188874575 conn(0x55b2fef6e4e0 msgr2=0x55b2ff31d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:24.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.596+0000 7ff49518f740 1 -- 172.21.15.90:0/1188874575 shutdown_connections 2024-04-25T11:40:24.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.596+0000 7ff49518f740 1 -- 172.21.15.90:0/1188874575 wait complete. 2024-04-25T11:40:24.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.596+0000 7ff49518f740 20 client.17187 trim_cache size 0 max 16384 2024-04-25T11:40:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.617+0000 7ff49518f740 1 Processor -- start 2024-04-25T11:40:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.617+0000 7ff49518f740 1 -- start start 2024-04-25T11:40:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.617+0000 7ff49518f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2fef38f60 0x55b2ff313190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.617+0000 7ff49518f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef514d0 0x55b2ff331cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.617+0000 7ff49518f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b2fef596b0 0x55b2ff324880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.617+0000 7ff49518f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff4881ad820 con 0x55b2fef514d0 2024-04-25T11:40:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.617+0000 7ff49518f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff4881af360 con 0x55b2fef596b0 2024-04-25T11:40:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.617+0000 7ff49518f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff4881adef0 con 0x55b2fef38f60 2024-04-25T11:40:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.617+0000 7ff486ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef514d0 0x55b2ff331cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.617+0000 7ff486ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef514d0 0x55b2ff331cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:32796/0 (socket says 172.21.15.90:32796) 2024-04-25T11:40:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.617+0000 7ff486ffd640 1 -- 172.21.15.90:0/3300151666 learned_addr learned my addr 172.21.15.90:0/3300151666 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:24.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.617+0000 7ff487fff640 1 --2- 172.21.15.90:0/3300151666 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b2fef596b0 0x55b2ff324880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:24.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.617+0000 7ff4877fe640 1 --2- 172.21.15.90:0/3300151666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2fef38f60 0x55b2ff313190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:24.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.618+0000 7ff486ffd640 1 -- 172.21.15.90:0/3300151666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2fef38f60 msgr2=0x55b2ff313190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:24.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.618+0000 7ff486ffd640 1 --2- 172.21.15.90:0/3300151666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2fef38f60 0x55b2ff313190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.618+0000 7ff486ffd640 1 -- 172.21.15.90:0/3300151666 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b2fef596b0 msgr2=0x55b2ff324880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:24.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.618+0000 7ff486ffd640 1 --2- 172.21.15.90:0/3300151666 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b2fef596b0 0x55b2ff324880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.618+0000 7ff486ffd640 1 -- 172.21.15.90:0/3300151666 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2ff324500 con 0x55b2fef514d0 2024-04-25T11:40:24.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.618+0000 7ff486ffd640 1 --2- 172.21.15.90:0/3300151666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef514d0 0x55b2ff331cc0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7ff47c019e40 tx=0x7ff47c00ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11fe259440b22e89 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:24.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.618+0000 7ff46d7fa640 1 -- 172.21.15.90:0/3300151666 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47c029070 con 0x55b2fef514d0 2024-04-25T11:40:24.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.618+0000 7ff46d7fa640 1 -- 172.21.15.90:0/3300151666 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff47c019850 con 0x55b2fef514d0 2024-04-25T11:40:24.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.618+0000 7ff46d7fa640 1 -- 172.21.15.90:0/3300151666 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47c015040 con 0x55b2fef514d0 2024-04-25T11:40:24.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.619+0000 7ff49518f740 1 -- 172.21.15.90:0/3300151666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef514d0 msgr2=0x55b2ff331cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:24.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.619+0000 7ff49518f740 1 --2- 172.21.15.90:0/3300151666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef514d0 0x55b2ff331cc0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7ff47c019e40 tx=0x7ff47c00ba70 comp rx=0 tx=0).stop 2024-04-25T11:40:24.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.619+0000 7ff49518f740 1 -- 172.21.15.90:0/3300151666 shutdown_connections 2024-04-25T11:40:24.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.619+0000 7ff49518f740 1 --2- 172.21.15.90:0/3300151666 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b2fef596b0 0x55b2ff324880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.619+0000 7ff49518f740 1 --2- 172.21.15.90:0/3300151666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef514d0 0x55b2ff331cc0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.619+0000 7ff49518f740 1 --2- 172.21.15.90:0/3300151666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2fef38f60 0x55b2ff313190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.619+0000 7ff49518f740 1 -- 172.21.15.90:0/3300151666 >> 172.21.15.90:0/3300151666 conn(0x55b2fef6e4e0 msgr2=0x55b2ff31b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:24.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.620+0000 7ff49518f740 1 -- 172.21.15.90:0/3300151666 shutdown_connections 2024-04-25T11:40:24.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.620+0000 7ff49518f740 1 -- 172.21.15.90:0/3300151666 wait complete. 2024-04-25T11:40:24.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.620+0000 7ff49518f740 1 -- start start 2024-04-25T11:40:24.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.621+0000 7ff49518f740 1 Processor -- start 2024-04-25T11:40:24.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.621+0000 7ff49518f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2fef38f60 0x55b2ff329830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.621+0000 7ff49518f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef514d0 0x55b2ff329d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.622+0000 7ff49518f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b2fef596b0 0x55b2ff32dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.622+0000 7ff486ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef514d0 0x55b2ff329d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:24.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.622+0000 7ff486ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef514d0 0x55b2ff329d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:32806/0 (socket says 172.21.15.90:32806) 2024-04-25T11:40:24.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.622+0000 7ff486ffd640 1 -- 172.21.15.90:0/792840320 learned_addr learned my addr 172.21.15.90:0/792840320 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:24.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.622+0000 7ff49518f740 1 -- 172.21.15.90:0/792840320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff4881aec90 con 0x55b2fef514d0 2024-04-25T11:40:24.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.622+0000 7ff49518f740 1 -- 172.21.15.90:0/792840320 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff4881ae5c0 con 0x55b2fef596b0 2024-04-25T11:40:24.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.622+0000 7ff49518f740 1 -- 172.21.15.90:0/792840320 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff4881ab800 con 0x55b2fef38f60 2024-04-25T11:40:24.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.622+0000 7ff487fff640 1 --2- 172.21.15.90:0/792840320 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b2fef596b0 0x55b2ff32dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:24.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.622+0000 7ff49518f740 10 client.0 _ll_register_callbacks cb 0x7ff47c006e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:24.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.623+0000 7ff4877fe640 1 --2- 172.21.15.90:0/792840320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2fef38f60 0x55b2ff329830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:24.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.623+0000 7ff486ffd640 1 -- 172.21.15.90:0/792840320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2fef38f60 msgr2=0x55b2ff329830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:24.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.623+0000 7ff486ffd640 1 --2- 172.21.15.90:0/792840320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2fef38f60 0x55b2ff329830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.623+0000 7ff486ffd640 1 -- 172.21.15.90:0/792840320 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b2fef596b0 msgr2=0x55b2ff32dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:24.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.623+0000 7ff486ffd640 1 --2- 172.21.15.90:0/792840320 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b2fef596b0 0x55b2ff32dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.623+0000 7ff486ffd640 1 -- 172.21.15.90:0/792840320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2ff32e1c0 con 0x55b2fef514d0 2024-04-25T11:40:24.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.623+0000 7ff487fff640 1 --2- 172.21.15.90:0/792840320 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b2fef596b0 0x55b2ff32dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:40:24.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.623+0000 7ff486ffd640 1 --2- 172.21.15.90:0/792840320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef514d0 0x55b2ff329d70 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7ff47c019e90 tx=0x7ff47c00b4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abfccd461420e5d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:24.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.624+0000 7ff466ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:40:24.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.624+0000 7ff466ffd640 1 -- 172.21.15.90:0/792840320 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47c029070 con 0x55b2fef514d0 2024-04-25T11:40:24.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.624+0000 7ff466ffd640 1 -- 172.21.15.90:0/792840320 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7ff47c0193a0 con 0x55b2fef514d0 2024-04-25T11:40:24.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.624+0000 7ff49518f740 1 -- 172.21.15.90:0/792840320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff47c019850 con 0x55b2fef514d0 2024-04-25T11:40:24.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.624+0000 7ff466ffd640 1 -- 172.21.15.90:0/792840320 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff47c015040 con 0x55b2fef514d0 2024-04-25T11:40:24.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.624+0000 7ff49518f740 10 client.17193 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:24.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.624+0000 7ff49518f740 1 -- 172.21.15.90:0/792840320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b2ff3a0130 con 0x55b2fef514d0 2024-04-25T11:40:24.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.625+0000 7ff466ffd640 1 -- 172.21.15.90:0/792840320 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff47c019850 con 0x55b2fef514d0 2024-04-25T11:40:24.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.625+0000 7ff466ffd640 1 -- 172.21.15.90:0/792840320 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7ff47c004020 con 0x55b2fef514d0 2024-04-25T11:40:24.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.625+0000 7ff49518f740 10 client.17193 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:24.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.625+0000 7ff49518f740 10 client.17193 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:40:24.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.625+0000 7ff49518f740 20 client.17193 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:24.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.625+0000 7ff49518f740 1 --2- 172.21.15.90:0/792840320 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b2fef7a390 0x55b2ff3a2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:24.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.625+0000 7ff49518f740 4 client.17193 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:40:24.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.625+0000 7ff49518f740 1 -- 172.21.15.90:0/792840320 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b2ff203610 con 0x55b2fef7a390 2024-04-25T11:40:24.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.627+0000 7ff4877fe640 1 --2- 172.21.15.90:0/792840320 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b2fef7a390 0x55b2ff3a2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:24.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.628+0000 7ff4877fe640 1 --2- 172.21.15.90:0/792840320 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b2fef7a390 0x55b2ff3a2850 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db1592385e129f0c server_cookie=563dbe0b861a30e2 in_seq=0 out_seq=0 2024-04-25T11:40:24.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.628+0000 7ff466ffd640 10 client.17193 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:24.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.633+0000 7ff466ffd640 1 -- 172.21.15.90:0/792840320 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b2ff203610 con 0x55b2fef7a390 2024-04-25T11:40:24.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.633+0000 7ff466ffd640 10 client.17193 handle_command_reply: tid=0 2024-04-25T11:40:24.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.633+0000 7ff49518f740 1 client.17193 shutdown 2024-04-25T11:40:24.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.633+0000 7ff49518f740 1 -- 172.21.15.90:0/792840320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef514d0 msgr2=0x55b2ff329d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:24.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.633+0000 7ff49518f740 1 --2- 172.21.15.90:0/792840320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef514d0 0x55b2ff329d70 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7ff47c019e90 tx=0x7ff47c00b4b0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.634+0000 7ff49518f740 1 -- 172.21.15.90:0/792840320 shutdown_connections 2024-04-25T11:40:24.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.634+0000 7ff49518f740 1 --2- 172.21.15.90:0/792840320 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b2fef7a390 0x55b2ff3a2850 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.634+0000 7ff49518f740 1 --2- 172.21.15.90:0/792840320 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b2fef596b0 0x55b2ff32dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.634+0000 7ff49518f740 1 --2- 172.21.15.90:0/792840320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b2fef514d0 0x55b2ff329d70 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.634+0000 7ff49518f740 1 --2- 172.21.15.90:0/792840320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2fef38f60 0x55b2ff329830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.634+0000 7ff49518f740 1 -- 172.21.15.90:0/792840320 >> 172.21.15.90:0/792840320 conn(0x55b2fef6e4e0 msgr2=0x55b2ff3247b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:24.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.634+0000 7ff466ffd640 0 client.17193 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:24.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.634+0000 7ff49518f740 1 -- 172.21.15.90:0/792840320 shutdown_connections 2024-04-25T11:40:24.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.634+0000 7ff49518f740 1 -- 172.21.15.90:0/792840320 wait complete. 2024-04-25T11:40:24.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.635+0000 7ff49518f740 20 client.17193 trim_cache size 0 max 16384 2024-04-25T11:40:24.623 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:40:24.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.635+0000 7ff48e1ff640 1 -- 172.21.15.90:0/3957259899 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff4700b41a0 msgr2=0x7ff4700b66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:24.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.635+0000 7ff48e1ff640 1 --2- 172.21.15.90:0/3957259899 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff4700b41a0 0x7ff4700b66b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff478009710 tx=0x7ff478006c90 comp rx=0 tx=0).stop 2024-04-25T11:40:24.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.635+0000 7ff48e1ff640 1 -- 172.21.15.90:0/3957259899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff488107440 msgr2=0x7ff48819cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:24.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.635+0000 7ff48e1ff640 1 --2- 172.21.15.90:0/3957259899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff488107440 0x7ff48819cc50 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7ff480030740 tx=0x7ff480030770 comp rx=0 tx=0).stop 2024-04-25T11:40:24.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.636+0000 7ff48e1ff640 1 -- 172.21.15.90:0/3957259899 shutdown_connections 2024-04-25T11:40:24.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.636+0000 7ff48e1ff640 1 --2- 172.21.15.90:0/3957259899 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff4700b41a0 0x7ff4700b66b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.636+0000 7ff48e1ff640 1 --2- 172.21.15.90:0/3957259899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff488107440 0x7ff48819cc50 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.636+0000 7ff48e1ff640 1 --2- 172.21.15.90:0/3957259899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff488106980 0x7ff488198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.636+0000 7ff48e1ff640 1 --2- 172.21.15.90:0/3957259899 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff488102440 0x7ff488198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:24.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.636+0000 7ff48e1ff640 1 -- 172.21.15.90:0/3957259899 >> 172.21.15.90:0/3957259899 conn(0x7ff488077880 msgr2=0x7ff488100910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:24.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.636+0000 7ff48e1ff640 1 -- 172.21.15.90:0/3957259899 shutdown_connections 2024-04-25T11:40:24.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:24.636+0000 7ff48e1ff640 1 -- 172.21.15.90:0/3957259899 wait complete. 2024-04-25T11:40:25.641 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T11:40:25.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:40:25.641 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T11:40:25.674 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:25 smithi090 ceph-mon[29575]: pgmap v163: 97 pgs: 97 active+clean; 42 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 2.2 MiB/s rd, 403 MiB/s wr, 2.64k op/s 2024-04-25T11:40:25.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:25 smithi138 ceph-mon[32328]: pgmap v163: 97 pgs: 97 active+clean; 42 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 2.2 MiB/s rd, 403 MiB/s wr, 2.64k op/s 2024-04-25T11:40:25.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:25 smithi156 ceph-mon[33045]: pgmap v163: 97 pgs: 97 active+clean; 42 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 2.2 MiB/s rd, 403 MiB/s wr, 2.64k op/s 2024-04-25T11:40:25.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.936+0000 7f24761ce640 1 -- 172.21.15.90:0/1032504615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2470102400 msgr2=0x7f2470102810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:25.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.936+0000 7f24761ce640 1 --2- 172.21.15.90:0/1032504615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2470102400 0x7f2470102810 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f2460009900 tx=0x7f246002a410 comp rx=0 tx=0).stop 2024-04-25T11:40:25.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.937+0000 7f24761ce640 1 -- 172.21.15.90:0/1032504615 shutdown_connections 2024-04-25T11:40:25.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.937+0000 7f24761ce640 1 --2- 172.21.15.90:0/1032504615 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2470107450 0x7f247010ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:25.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.937+0000 7f24761ce640 1 --2- 172.21.15.90:0/1032504615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2470106990 0x7f2470106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:25.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.937+0000 7f24761ce640 1 --2- 172.21.15.90:0/1032504615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2470102400 0x7f2470102810 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:25.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.937+0000 7f24761ce640 1 -- 172.21.15.90:0/1032504615 >> 172.21.15.90:0/1032504615 conn(0x7f2470077880 msgr2=0x7f2470077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:25.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.937+0000 7f24761ce640 1 -- 172.21.15.90:0/1032504615 shutdown_connections 2024-04-25T11:40:25.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.937+0000 7f24761ce640 1 -- 172.21.15.90:0/1032504615 wait complete. 2024-04-25T11:40:25.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.938+0000 7f24761ce640 1 Processor -- start 2024-04-25T11:40:25.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.939+0000 7f24761ce640 1 -- start start 2024-04-25T11:40:25.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.939+0000 7f24761ce640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2470102400 0x7f24701a25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:25.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.939+0000 7f24761ce640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2470106990 0x7f24701a2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:25.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.939+0000 7f24761ce640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2470107450 0x7f247019c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:25.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.939+0000 7f24761ce640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2470113d50 con 0x7f2470107450 2024-04-25T11:40:25.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.939+0000 7f24761ce640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2470113bd0 con 0x7f2470102400 2024-04-25T11:40:25.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.939+0000 7f24761ce640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2470113ed0 con 0x7f2470106990 2024-04-25T11:40:25.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.939+0000 7f246ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2470107450 0x7f247019c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:25.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.940+0000 7f246ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2470107450 0x7f247019c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:32814/0 (socket says 172.21.15.90:32814) 2024-04-25T11:40:25.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.940+0000 7f246ffff640 1 -- 172.21.15.90:0/3900041073 learned_addr learned my addr 172.21.15.90:0/3900041073 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:25.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.940+0000 7f246f7fe640 1 --2- 172.21.15.90:0/3900041073 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2470102400 0x7f24701a25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:25.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.940+0000 7f246ffff640 1 -- 172.21.15.90:0/3900041073 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2470106990 msgr2=0x7f24701a2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:25.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.940+0000 7f246ffff640 1 --2- 172.21.15.90:0/3900041073 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2470106990 0x7f24701a2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:25.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.940+0000 7f246ffff640 1 -- 172.21.15.90:0/3900041073 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2470102400 msgr2=0x7f24701a25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:25.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.940+0000 7f246ffff640 1 --2- 172.21.15.90:0/3900041073 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2470102400 0x7f24701a25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:25.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.940+0000 7f246ffff640 1 -- 172.21.15.90:0/3900041073 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f247019cf20 con 0x7f2470107450 2024-04-25T11:40:25.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.940+0000 7f246f7fe640 1 --2- 172.21.15.90:0/3900041073 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2470102400 0x7f24701a25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:40:25.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.941+0000 7f246ffff640 1 --2- 172.21.15.90:0/3900041073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2470107450 0x7f247019c8b0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f246800b970 tx=0x7f246800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:25.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.941+0000 7f246cff9640 1 -- 172.21.15.90:0/3900041073 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2468019070 con 0x7f2470107450 2024-04-25T11:40:25.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.941+0000 7f24761ce640 1 -- 172.21.15.90:0/3900041073 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f247019d210 con 0x7f2470107450 2024-04-25T11:40:25.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.941+0000 7f246cff9640 1 -- 172.21.15.90:0/3900041073 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2468014440 con 0x7f2470107450 2024-04-25T11:40:25.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.941+0000 7f246cff9640 1 -- 172.21.15.90:0/3900041073 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24680075d0 con 0x7f2470107450 2024-04-25T11:40:25.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.941+0000 7f24761ce640 1 -- 172.21.15.90:0/3900041073 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f247019d710 con 0x7f2470107450 2024-04-25T11:40:25.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.946+0000 7f247d080740 1 Processor -- start 2024-04-25T11:40:25.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.948+0000 7f246cff9640 1 -- 172.21.15.90:0/3900041073 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f246801c070 con 0x7f2470107450 2024-04-25T11:40:25.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.948+0000 7f247d080740 1 -- start start 2024-04-25T11:40:25.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.949+0000 7f247d080740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e99678d410 0x55e996b2d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:25.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.949+0000 7f247d080740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e996764550 0x55e996b2db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:25.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.949+0000 7f247d080740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99676c730 0x55e996b2e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:25.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.949+0000 7f247d080740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e996a93b00 con 0x55e99676c730 2024-04-25T11:40:25.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.949+0000 7f247d080740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55e996a2f920 con 0x55e99678d410 2024-04-25T11:40:25.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.949+0000 7f247d080740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e996a8fac0 con 0x55e996764550 2024-04-25T11:40:25.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.949+0000 7f246effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99676c730 0x55e996b2e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:25.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.949+0000 7f246effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99676c730 0x55e996b2e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:32818/0 (socket says 172.21.15.90:32818) 2024-04-25T11:40:25.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.949+0000 7f246effd640 1 -- 172.21.15.90:0/3141511601 learned_addr learned my addr 172.21.15.90:0/3141511601 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:25.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.949+0000 7f246f7fe640 1 --2- 172.21.15.90:0/3141511601 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e996764550 0x55e996b2db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:25.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.950+0000 7f246cff9640 1 --2- 172.21.15.90:0/3900041073 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f24580afce0 0x7f24580b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:25.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.950+0000 7f246effd640 1 -- 172.21.15.90:0/3141511601 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e996764550 msgr2=0x55e996b2db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:25.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.950+0000 7f246effd640 1 --2- 172.21.15.90:0/3141511601 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e996764550 0x55e996b2db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:25.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.950+0000 7f246effd640 1 -- 172.21.15.90:0/3141511601 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e99678d410 msgr2=0x55e996b2d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:25.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.950+0000 7f246effd640 1 --2- 172.21.15.90:0/3141511601 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e99678d410 0x55e996b2d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:25.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.950+0000 7f246effd640 1 -- 172.21.15.90:0/3141511601 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e99686d7e0 con 0x55e99676c730 2024-04-25T11:40:25.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.950+0000 7f246cff9640 1 -- 172.21.15.90:0/3900041073 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f24680de670 con 0x7f2470107450 2024-04-25T11:40:25.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.951+0000 7f246ffff640 1 --2- 172.21.15.90:0/3900041073 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f24580afce0 0x7f24580b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:25.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.951+0000 7f246effd640 1 --2- 172.21.15.90:0/3141511601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99676c730 0x55e996b2e0a0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f246400ab20 tx=0x7f246400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2a6aa208d3f270c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:25.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.952+0000 7f2455ffb640 1 -- 172.21.15.90:0/3141511601 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2464004020 con 0x55e99676c730 2024-04-25T11:40:25.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.952+0000 7f2455ffb640 1 -- 172.21.15.90:0/3141511601 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24640041c0 con 0x55e99676c730 2024-04-25T11:40:25.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.952+0000 7f2455ffb640 1 -- 172.21.15.90:0/3141511601 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246400f6b0 con 0x55e99676c730 2024-04-25T11:40:25.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.953+0000 7f246ffff640 1 --2- 172.21.15.90:0/3900041073 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f24580afce0 0x7f24580b21f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f24680a28e0 tx=0x7f24680af040 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:25.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.953+0000 7f247d080740 1 -- 172.21.15.90:0/3141511601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99676c730 msgr2=0x55e996b2e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:25.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.953+0000 7f247d080740 1 --2- 172.21.15.90:0/3141511601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99676c730 0x55e996b2e0a0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f246400ab20 tx=0x7f246400c2d0 comp rx=0 tx=0).stop 2024-04-25T11:40:25.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.953+0000 7f247d080740 1 -- 172.21.15.90:0/3141511601 shutdown_connections 2024-04-25T11:40:25.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.953+0000 7f247d080740 1 --2- 172.21.15.90:0/3141511601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99676c730 0x55e996b2e0a0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:25.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.953+0000 7f247d080740 1 --2- 172.21.15.90:0/3141511601 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e996764550 0x55e996b2db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:25.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.953+0000 7f247d080740 1 --2- 172.21.15.90:0/3141511601 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e99678d410 0x55e996b2d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:25.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.953+0000 7f247d080740 1 -- 172.21.15.90:0/3141511601 >> 172.21.15.90:0/3141511601 conn(0x55e996781560 msgr2=0x55e996b29dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:25.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.953+0000 7f247d080740 1 -- 172.21.15.90:0/3141511601 shutdown_connections 2024-04-25T11:40:25.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.953+0000 7f247d080740 1 -- 172.21.15.90:0/3141511601 wait complete. 2024-04-25T11:40:25.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.954+0000 7f247d080740 1 -- start start 2024-04-25T11:40:25.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.954+0000 7f247d080740 1 Processor -- start 2024-04-25T11:40:25.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.954+0000 7f247d080740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e996764550 0x55e996b37c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:25.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.954+0000 7f247d080740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99676c730 0x55e996b381c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:25.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.955+0000 7f247d080740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e99678d410 0x55e996b3ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:25.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.955+0000 7f247d080740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e996b3a970 con 0x55e99676c730 2024-04-25T11:40:25.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.955+0000 7f247d080740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55e996a2d5d0 con 0x55e99678d410 2024-04-25T11:40:25.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.955+0000 7f247d080740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e996b3aaf0 con 0x55e996764550 2024-04-25T11:40:25.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.955+0000 7f247d080740 10 client.0 _ll_register_callbacks cb 0x55e996a1b080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:25.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.955+0000 7f246f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99676c730 0x55e996b381c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:25.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.955+0000 7f246f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99676c730 0x55e996b381c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:32834/0 (socket says 172.21.15.90:32834) 2024-04-25T11:40:25.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.955+0000 7f246f7fe640 1 -- 172.21.15.90:0/4233854787 learned_addr learned my addr 172.21.15.90:0/4233854787 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:25.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.955+0000 7f246effd640 1 --2- 172.21.15.90:0/4233854787 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e99678d410 0x55e996b3ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:25.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.955+0000 7f246effd640 1 --2- 172.21.15.90:0/4233854787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e996764550 0x55e996b37c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:25.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.955+0000 7f246f7fe640 1 -- 172.21.15.90:0/4233854787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e996764550 msgr2=0x55e996b37c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:25.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.955+0000 7f246f7fe640 1 --2- 172.21.15.90:0/4233854787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e996764550 0x55e996b37c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:25.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.955+0000 7f246f7fe640 1 -- 172.21.15.90:0/4233854787 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e99678d410 msgr2=0x55e996b3ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:25.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.955+0000 7f246f7fe640 1 --2- 172.21.15.90:0/4233854787 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e99678d410 0x55e996b3ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:25.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.955+0000 7f246f7fe640 1 -- 172.21.15.90:0/4233854787 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9968cdf70 con 0x55e99676c730 2024-04-25T11:40:25.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.956+0000 7f246effd640 1 --2- 172.21.15.90:0/4233854787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e996764550 0x55e996b37c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:40:25.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.956+0000 7f246f7fe640 1 --2- 172.21.15.90:0/4233854787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99676c730 0x55e996b381c0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f247019db40 tx=0x7f2460031890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fa62b67ee3f88a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:25.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.956+0000 7f244effd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:40:25.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.956+0000 7f244effd640 1 -- 172.21.15.90:0/4233854787 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246003f070 con 0x55e99676c730 2024-04-25T11:40:25.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.956+0000 7f247d080740 1 -- 172.21.15.90:0/4233854787 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24640041c0 con 0x55e99676c730 2024-04-25T11:40:25.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.957+0000 7f244effd640 1 -- 172.21.15.90:0/4233854787 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f2460004450 con 0x55e99676c730 2024-04-25T11:40:25.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.957+0000 7f244effd640 1 -- 172.21.15.90:0/4233854787 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2460035070 con 0x55e99676c730 2024-04-25T11:40:25.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.957+0000 7f247d080740 10 client.17205 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:25.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.957+0000 7f247d080740 1 -- 172.21.15.90:0/4233854787 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e996765170 con 0x55e99676c730 2024-04-25T11:40:25.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.958+0000 7f244effd640 1 -- 172.21.15.90:0/4233854787 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24600054e0 con 0x55e99676c730 2024-04-25T11:40:25.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.958+0000 7f244effd640 1 -- 172.21.15.90:0/4233854787 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f24600035f0 con 0x55e99676c730 2024-04-25T11:40:25.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.958+0000 7f247d080740 10 client.17205 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:25.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.958+0000 7f247d080740 10 client.17205 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:40:25.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.958+0000 7f247d080740 20 client.17205 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:25.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.959+0000 7f247d080740 1 --2- 172.21.15.90:0/4233854787 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e99674c1f0 0x55e996b4a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:25.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.959+0000 7f247d080740 4 client.17205 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:40:25.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.959+0000 7f247d080740 1 -- 172.21.15.90:0/4233854787 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f24640041c0 con 0x55e99674c1f0 2024-04-25T11:40:25.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.960+0000 7f246effd640 1 --2- 172.21.15.90:0/4233854787 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e99674c1f0 0x55e996b4a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:25.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.963+0000 7f246effd640 1 --2- 172.21.15.90:0/4233854787 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e99674c1f0 0x55e996b4a240 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd7a3c80524f14c9 server_cookie=c2d7bab0b6e7a0de in_seq=0 out_seq=0 2024-04-25T11:40:25.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.963+0000 7f244effd640 10 client.17205 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:25.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.968+0000 7f244effd640 1 -- 172.21.15.90:0/4233854787 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f24640041c0 con 0x55e99674c1f0 2024-04-25T11:40:25.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.968+0000 7f244effd640 10 client.17205 handle_command_reply: tid=0 2024-04-25T11:40:25.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.968+0000 7f247d080740 1 client.17205 shutdown 2024-04-25T11:40:25.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.969+0000 7f247d080740 1 -- 172.21.15.90:0/4233854787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99676c730 msgr2=0x55e996b381c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:25.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.969+0000 7f247d080740 1 --2- 172.21.15.90:0/4233854787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99676c730 0x55e996b381c0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f247019db40 tx=0x7f2460031890 comp rx=0 tx=0).stop 2024-04-25T11:40:25.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.969+0000 7f247d080740 1 -- 172.21.15.90:0/4233854787 shutdown_connections 2024-04-25T11:40:25.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.970+0000 7f247d080740 1 --2- 172.21.15.90:0/4233854787 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e99674c1f0 0x55e996b4a240 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:25.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.970+0000 7f247d080740 1 --2- 172.21.15.90:0/4233854787 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e99678d410 0x55e996b3ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:25.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.970+0000 7f244effd640 0 client.17205 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:25.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.970+0000 7f247d080740 1 --2- 172.21.15.90:0/4233854787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99676c730 0x55e996b381c0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:25.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.970+0000 7f247d080740 1 --2- 172.21.15.90:0/4233854787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e996764550 0x55e996b37c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:25.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.970+0000 7f247d080740 1 -- 172.21.15.90:0/4233854787 >> 172.21.15.90:0/4233854787 conn(0x55e996781560 msgr2=0x55e996b2f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:25.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.970+0000 7f247d080740 1 -- 172.21.15.90:0/4233854787 shutdown_connections 2024-04-25T11:40:25.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.970+0000 7f247d080740 1 -- 172.21.15.90:0/4233854787 wait complete. 2024-04-25T11:40:25.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.970+0000 7f247d080740 20 client.17205 trim_cache size 0 max 16384 2024-04-25T11:40:25.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.997+0000 7f247d080740 1 Processor -- start 2024-04-25T11:40:25.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.997+0000 7f247d080740 1 -- start start 2024-04-25T11:40:25.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.997+0000 7f247d080740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99674c1f0 0x55e996b32190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:25.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.998+0000 7f247d080740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e996764550 0x55e996b28bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:25.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.998+0000 7f247d080740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e99676c730 0x55e996b3dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:25.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.998+0000 7f247d080740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f24701b1e20 con 0x55e99674c1f0 2024-04-25T11:40:25.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.998+0000 7f247d080740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f24701b3960 con 0x55e99676c730 2024-04-25T11:40:25.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.998+0000 7f247d080740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f24701b24f0 con 0x55e996764550 2024-04-25T11:40:25.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.998+0000 7f246effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99674c1f0 0x55e996b32190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:25.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.998+0000 7f246effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99674c1f0 0x55e996b32190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:32838/0 (socket says 172.21.15.90:32838) 2024-04-25T11:40:25.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.998+0000 7f246effd640 1 -- 172.21.15.90:0/791205188 learned_addr learned my addr 172.21.15.90:0/791205188 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:25.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.998+0000 7f246effd640 1 -- 172.21.15.90:0/791205188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e996764550 msgr2=0x55e996b28bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:25.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.998+0000 7f246effd640 1 --2- 172.21.15.90:0/791205188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e996764550 0x55e996b28bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:25.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.998+0000 7f246effd640 1 -- 172.21.15.90:0/791205188 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e99676c730 msgr2=0x55e996b3dd60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:40:25.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.998+0000 7f246effd640 1 --2- 172.21.15.90:0/791205188 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e99676c730 0x55e996b3dd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:25.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.998+0000 7f246effd640 1 -- 172.21.15.90:0/791205188 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e996b326d0 con 0x55e99674c1f0 2024-04-25T11:40:25.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.999+0000 7f246effd640 1 --2- 172.21.15.90:0/791205188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99674c1f0 0x55e996b32190 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f2464016000 tx=0x7f246400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69adb4e56fc96f7c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:25.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.999+0000 7f24557fa640 1 -- 172.21.15.90:0/791205188 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246402b070 con 0x55e99674c1f0 2024-04-25T11:40:25.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.999+0000 7f24557fa640 1 -- 172.21.15.90:0/791205188 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f246400ccb0 con 0x55e99674c1f0 2024-04-25T11:40:25.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:25.999+0000 7f24557fa640 1 -- 172.21.15.90:0/791205188 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24640266a0 con 0x55e99674c1f0 2024-04-25T11:40:25.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.000+0000 7f247d080740 1 -- 172.21.15.90:0/791205188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99674c1f0 msgr2=0x55e996b32190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:25.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.000+0000 7f247d080740 1 --2- 172.21.15.90:0/791205188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99674c1f0 0x55e996b32190 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f2464016000 tx=0x7f246400c2d0 comp rx=0 tx=0).stop 2024-04-25T11:40:25.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.000+0000 7f247d080740 1 -- 172.21.15.90:0/791205188 shutdown_connections 2024-04-25T11:40:25.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.000+0000 7f247d080740 1 --2- 172.21.15.90:0/791205188 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e99676c730 0x55e996b3dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:25.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.000+0000 7f247d080740 1 --2- 172.21.15.90:0/791205188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e996764550 0x55e996b28bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:25.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.000+0000 7f247d080740 1 --2- 172.21.15.90:0/791205188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99674c1f0 0x55e996b32190 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:25.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.000+0000 7f247d080740 1 -- 172.21.15.90:0/791205188 >> 172.21.15.90:0/791205188 conn(0x55e996781560 msgr2=0x55e996b284c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:25.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.000+0000 7f247d080740 1 -- 172.21.15.90:0/791205188 shutdown_connections 2024-04-25T11:40:25.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.000+0000 7f247d080740 1 -- 172.21.15.90:0/791205188 wait complete. 2024-04-25T11:40:25.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.001+0000 7f247d080740 1 -- start start 2024-04-25T11:40:25.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.001+0000 7f247d080740 1 Processor -- start 2024-04-25T11:40:25.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.002+0000 7f247d080740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e99674c1f0 0x55e996b445c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:25.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.002+0000 7f247d080740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e996764550 0x55e996b41930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:25.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.002+0000 7f247d080740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99676c730 0x55e996b41e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:25.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.002+0000 7f247d080740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f24701b3290 con 0x55e99676c730 2024-04-25T11:40:25.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.002+0000 7f247d080740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f24701b2bc0 con 0x55e996764550 2024-04-25T11:40:25.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.002+0000 7f247d080740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f24701afe00 con 0x55e99674c1f0 2024-04-25T11:40:25.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.002+0000 7f246effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99676c730 0x55e996b41e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:25.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.002+0000 7f246effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99676c730 0x55e996b41e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:32846/0 (socket says 172.21.15.90:32846) 2024-04-25T11:40:25.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.002+0000 7f246effd640 1 -- 172.21.15.90:0/808519799 learned_addr learned my addr 172.21.15.90:0/808519799 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:25.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.002+0000 7f247d080740 10 client.0 _ll_register_callbacks cb 0x7f24600314a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:25.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.002+0000 7f246effd640 1 -- 172.21.15.90:0/808519799 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e99674c1f0 msgr2=0x55e996b445c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:25.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.002+0000 7f246effd640 1 --2- 172.21.15.90:0/808519799 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e99674c1f0 0x55e996b445c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:25.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.002+0000 7f246effd640 1 -- 172.21.15.90:0/808519799 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e996764550 msgr2=0x55e996b41930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:25.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.002+0000 7f246effd640 1 --2- 172.21.15.90:0/808519799 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e996764550 0x55e996b41930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:25.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.003+0000 7f246effd640 1 -- 172.21.15.90:0/808519799 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e996b423b0 con 0x55e99676c730 2024-04-25T11:40:25.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.003+0000 7f246effd640 1 --2- 172.21.15.90:0/808519799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99676c730 0x55e996b41e70 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f246400a490 tx=0x7f2464005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70568d94c379e78c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:25.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.003+0000 7f244effd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:40:25.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.003+0000 7f244effd640 1 -- 172.21.15.90:0/808519799 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246402b070 con 0x55e99676c730 2024-04-25T11:40:25.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.003+0000 7f244effd640 1 -- 172.21.15.90:0/808519799 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f2464026640 con 0x55e99676c730 2024-04-25T11:40:25.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.003+0000 7f247d080740 1 -- 172.21.15.90:0/808519799 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f246400ccb0 con 0x55e99676c730 2024-04-25T11:40:25.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.003+0000 7f244effd640 1 -- 172.21.15.90:0/808519799 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f24640035f0 con 0x55e99676c730 2024-04-25T11:40:25.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.003+0000 7f247d080740 10 client.17211 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:25.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.003+0000 7f247d080740 1 -- 172.21.15.90:0/808519799 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e996bb0050 con 0x55e99676c730 2024-04-25T11:40:25.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.004+0000 7f244effd640 1 -- 172.21.15.90:0/808519799 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2464003790 con 0x55e99676c730 2024-04-25T11:40:25.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.004+0000 7f244effd640 1 -- 172.21.15.90:0/808519799 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f24640047a0 con 0x55e99676c730 2024-04-25T11:40:25.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.004+0000 7f247d080740 10 client.17211 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:25.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.004+0000 7f247d080740 10 client.17211 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:40:25.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.004+0000 7f247d080740 20 client.17211 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:25.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.004+0000 7f247d080740 1 --2- 172.21.15.90:0/808519799 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e99678d410 0x55e996bb05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:25.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.004+0000 7f247d080740 4 client.17211 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T11:40:25.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.004+0000 7f247d080740 1 -- 172.21.15.90:0/808519799 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f246400ccb0 con 0x55e99678d410 2024-04-25T11:40:25.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.006+0000 7f246f7fe640 1 --2- 172.21.15.90:0/808519799 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e99678d410 0x55e996bb05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:26.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.013+0000 7f244effd640 10 client.17211 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:26.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.013+0000 7f246f7fe640 1 --2- 172.21.15.90:0/808519799 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e99678d410 0x55e996bb05c0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=706c1a3b50adc343 server_cookie=75471968488f9240 in_seq=0 out_seq=0 2024-04-25T11:40:26.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.512+0000 7f244effd640 1 -- 172.21.15.90:0/808519799 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f246400ccb0 con 0x55e99678d410 2024-04-25T11:40:26.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.512+0000 7f244effd640 10 client.17211 handle_command_reply: tid=0 2024-04-25T11:40:26.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.512+0000 7f247d080740 1 client.17211 shutdown 2024-04-25T11:40:26.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.516+0000 7f247d080740 1 -- 172.21.15.90:0/808519799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99676c730 msgr2=0x55e996b41e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:26.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.516+0000 7f247d080740 1 --2- 172.21.15.90:0/808519799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99676c730 0x55e996b41e70 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f246400a490 tx=0x7f2464005c50 comp rx=0 tx=0).stop 2024-04-25T11:40:26.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.516+0000 7f247d080740 1 -- 172.21.15.90:0/808519799 shutdown_connections 2024-04-25T11:40:26.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.516+0000 7f247d080740 1 --2- 172.21.15.90:0/808519799 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e99678d410 0x55e996bb05c0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:26.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.516+0000 7f247d080740 1 --2- 172.21.15.90:0/808519799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e99676c730 0x55e996b41e70 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:26.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.516+0000 7f247d080740 1 --2- 172.21.15.90:0/808519799 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e996764550 0x55e996b41930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:26.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.516+0000 7f247d080740 1 --2- 172.21.15.90:0/808519799 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e99674c1f0 0x55e996b445c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:26.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.516+0000 7f247d080740 1 -- 172.21.15.90:0/808519799 >> 172.21.15.90:0/808519799 conn(0x55e996781560 msgr2=0x55e996b2d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:26.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.517+0000 7f244effd640 0 client.17211 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:26.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.518+0000 7f247d080740 1 -- 172.21.15.90:0/808519799 shutdown_connections 2024-04-25T11:40:26.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.518+0000 7f247d080740 1 -- 172.21.15.90:0/808519799 wait complete. 2024-04-25T11:40:26.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.518+0000 7f247d080740 20 client.17211 trim_cache size 0 max 16384 2024-04-25T11:40:26.508 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:40:26.508 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-25T11:40:26.508 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "6783de6e-89d2-4996-a92f-f4a92e99c570", 2024-04-25T11:40:26.508 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-25T11:40:26.508 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:40:26.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.520+0000 7f24761ce640 1 -- 172.21.15.90:0/3900041073 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f24580afce0 msgr2=0x7f24580b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:26.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.520+0000 7f24761ce640 1 --2- 172.21.15.90:0/3900041073 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f24580afce0 0x7f24580b21f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f24680a28e0 tx=0x7f24680af040 comp rx=0 tx=0).stop 2024-04-25T11:40:26.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.520+0000 7f24761ce640 1 -- 172.21.15.90:0/3900041073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2470107450 msgr2=0x7f247019c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:26.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.520+0000 7f24761ce640 1 --2- 172.21.15.90:0/3900041073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2470107450 0x7f247019c8b0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f246800b970 tx=0x7f246800be30 comp rx=0 tx=0).stop 2024-04-25T11:40:26.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.520+0000 7f24761ce640 1 -- 172.21.15.90:0/3900041073 shutdown_connections 2024-04-25T11:40:26.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.520+0000 7f24761ce640 1 --2- 172.21.15.90:0/3900041073 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f24580afce0 0x7f24580b21f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:26.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.520+0000 7f24761ce640 1 --2- 172.21.15.90:0/3900041073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2470107450 0x7f247019c8b0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:26.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.520+0000 7f24761ce640 1 --2- 172.21.15.90:0/3900041073 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2470106990 0x7f24701a2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:26.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.520+0000 7f24761ce640 1 --2- 172.21.15.90:0/3900041073 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2470102400 0x7f24701a25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:26.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.520+0000 7f24761ce640 1 -- 172.21.15.90:0/3900041073 >> 172.21.15.90:0/3900041073 conn(0x7f2470077880 msgr2=0x7f24700ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:26.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.522+0000 7f24761ce640 1 -- 172.21.15.90:0/3900041073 shutdown_connections 2024-04-25T11:40:26.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.524+0000 7f24761ce640 1 -- 172.21.15.90:0/3900041073 wait complete. 2024-04-25T11:40:26.567 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:40:26.567 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:40:26.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.845+0000 7f4e8f59e640 1 -- 172.21.15.90:0/694186405 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e9011a7c0 msgr2=0x7f4e9011cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:26.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.845+0000 7f4e8f59e640 1 --2- 172.21.15.90:0/694186405 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e9011a7c0 0x7f4e9011cbe0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f4e88009ca0 tx=0x7f4e8802a8f0 comp rx=0 tx=0).stop 2024-04-25T11:40:26.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.845+0000 7f4e8f59e640 1 -- 172.21.15.90:0/694186405 shutdown_connections 2024-04-25T11:40:26.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.845+0000 7f4e8f59e640 1 --2- 172.21.15.90:0/694186405 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4e9011d120 0x7f4e9011f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:26.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.845+0000 7f4e8f59e640 1 --2- 172.21.15.90:0/694186405 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e9011a7c0 0x7f4e9011cbe0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:26.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.845+0000 7f4e8f59e640 1 --2- 172.21.15.90:0/694186405 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e90109fc0 0x7f4e9010a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:26.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.845+0000 7f4e8f59e640 1 -- 172.21.15.90:0/694186405 >> 172.21.15.90:0/694186405 conn(0x7f4e9006f650 msgr2=0x7f4e9006fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:26.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.846+0000 7f4e8f59e640 1 -- 172.21.15.90:0/694186405 shutdown_connections 2024-04-25T11:40:26.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.847+0000 7f4e8f59e640 1 -- 172.21.15.90:0/694186405 wait complete. 2024-04-25T11:40:26.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.849+0000 7f4e8f59e640 1 Processor -- start 2024-04-25T11:40:26.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.849+0000 7f4e8f59e640 1 -- start start 2024-04-25T11:40:26.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.849+0000 7f4e8f59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e90109fc0 0x7f4e901a9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:26.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.849+0000 7f4e8f59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e9011a7c0 0x7f4e901a9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:26.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.849+0000 7f4e8f59e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4e9011d120 0x7f4e901adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:26.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.849+0000 7f4e8f59e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4e90122aa0 con 0x7f4e9011a7c0 2024-04-25T11:40:26.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.849+0000 7f4e8f59e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4e90122920 con 0x7f4e9011d120 2024-04-25T11:40:26.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.849+0000 7f4e8f59e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4e90122c20 con 0x7f4e90109fc0 2024-04-25T11:40:26.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.850+0000 7f4e8dd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e9011a7c0 0x7f4e901a9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:26.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.850+0000 7f4e8dd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e9011a7c0 0x7f4e901a9ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:32856/0 (socket says 172.21.15.90:32856) 2024-04-25T11:40:26.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.850+0000 7f4e8dd9b640 1 -- 172.21.15.90:0/3130669314 learned_addr learned my addr 172.21.15.90:0/3130669314 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:26.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.850+0000 7f4e8ed9d640 1 --2- 172.21.15.90:0/3130669314 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4e9011d120 0x7f4e901adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:26.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.851+0000 7f4e8dd9b640 1 -- 172.21.15.90:0/3130669314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e90109fc0 msgr2=0x7f4e901a9580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:40:26.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.851+0000 7f4e8dd9b640 1 --2- 172.21.15.90:0/3130669314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e90109fc0 0x7f4e901a9580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:26.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.851+0000 7f4e8dd9b640 1 -- 172.21.15.90:0/3130669314 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4e9011d120 msgr2=0x7f4e901adda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:26.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.851+0000 7f4e8dd9b640 1 --2- 172.21.15.90:0/3130669314 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4e9011d120 0x7f4e901adda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:26.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.851+0000 7f4e8dd9b640 1 -- 172.21.15.90:0/3130669314 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e901ae3e0 con 0x7f4e9011a7c0 2024-04-25T11:40:26.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.852+0000 7f4e8dd9b640 1 --2- 172.21.15.90:0/3130669314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e9011a7c0 0x7f4e901a9ae0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f4e8802f570 tx=0x7f4e88031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:26.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.859+0000 7f4e7f7fe640 1 -- 172.21.15.90:0/3130669314 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e88003d60 con 0x7f4e9011a7c0 2024-04-25T11:40:26.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.859+0000 7f4e8f59e640 1 -- 172.21.15.90:0/3130669314 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e901ae670 con 0x7f4e9011a7c0 2024-04-25T11:40:26.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.859+0000 7f4e8f59e640 1 -- 172.21.15.90:0/3130669314 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e901aebc0 con 0x7f4e9011a7c0 2024-04-25T11:40:26.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.864+0000 7f4e9bf88740 1 Processor -- start 2024-04-25T11:40:26.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.871+0000 7f4e9bf88740 1 -- start start 2024-04-25T11:40:26.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.872+0000 7f4e9bf88740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5636a7065390 0x5636a7405c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:26.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.872+0000 7f4e9bf88740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5636a703c4d0 0x5636a74061b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:26.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.872+0000 7f4e9bf88740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636a70446b0 0x5636a7412b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:26.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.872+0000 7f4e9bf88740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5636a736bd80 con 0x5636a70446b0 2024-04-25T11:40:26.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.872+0000 7f4e9bf88740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5636a73258b0 con 0x5636a7065390 2024-04-25T11:40:26.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.872+0000 7f4e9bf88740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5636a7367d40 con 0x5636a703c4d0 2024-04-25T11:40:26.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.873+0000 7f4e7f7fe640 1 -- 172.21.15.90:0/3130669314 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e8802d070 con 0x7f4e9011a7c0 2024-04-25T11:40:26.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.873+0000 7f4e7f7fe640 1 -- 172.21.15.90:0/3130669314 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8802fb70 con 0x7f4e9011a7c0 2024-04-25T11:40:26.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.873+0000 7f4e7f7fe640 1 -- 172.21.15.90:0/3130669314 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f4e8802fd50 con 0x7f4e9011a7c0 2024-04-25T11:40:26.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.874+0000 7f4e8ed9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5636a7065390 0x5636a7405c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:26.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.874+0000 7f4e8ed9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5636a7065390 0x5636a7405c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:33034/0 (socket says 172.21.15.90:33034) 2024-04-25T11:40:26.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.874+0000 7f4e8ed9d640 1 -- 172.21.15.90:0/2934242371 learned_addr learned my addr 172.21.15.90:0/2934242371 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:26.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.875+0000 7f4e8dd9b640 1 --2- 172.21.15.90:0/2934242371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636a70446b0 0x5636a7412b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:26.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.877+0000 7f4e8ed9d640 1 -- 172.21.15.90:0/2934242371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5636a703c4d0 msgr2=0x5636a74061b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:40:26.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.877+0000 7f4e8ed9d640 1 --2- 172.21.15.90:0/2934242371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5636a703c4d0 0x5636a74061b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:26.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.877+0000 7f4e8ed9d640 1 -- 172.21.15.90:0/2934242371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636a70446b0 msgr2=0x5636a7412b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:26.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.877+0000 7f4e8ed9d640 1 --2- 172.21.15.90:0/2934242371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636a70446b0 0x5636a7412b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:26.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.877+0000 7f4e8ed9d640 1 -- 172.21.15.90:0/2934242371 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636a72f6a40 con 0x5636a7065390 2024-04-25T11:40:26.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.877+0000 7f4e7f7fe640 1 --2- 172.21.15.90:0/3130669314 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4e780afea0 0x7f4e780b23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:26.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.878+0000 7f4e7f7fe640 1 -- 172.21.15.90:0/3130669314 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f4e88101720 con 0x7f4e9011a7c0 2024-04-25T11:40:26.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.878+0000 7f4e8ed9d640 1 --2- 172.21.15.90:0/2934242371 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5636a7065390 0x5636a7405c70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4e80009960 tx=0x7f4e80009e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc46cba426aed95e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:26.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.887+0000 7f4e7cff9640 1 -- 172.21.15.90:0/2934242371 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e80017070 con 0x5636a7065390 2024-04-25T11:40:26.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.888+0000 7f4e8e59c640 1 --2- 172.21.15.90:0/3130669314 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4e780afea0 0x7f4e780b23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:26.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.888+0000 7f4e7cff9640 1 -- 172.21.15.90:0/2934242371 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e80013470 con 0x5636a7065390 2024-04-25T11:40:26.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.890+0000 7f4e9bf88740 1 -- 172.21.15.90:0/2934242371 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5636a7065390 msgr2=0x5636a7405c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:26.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.890+0000 7f4e9bf88740 1 --2- 172.21.15.90:0/2934242371 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5636a7065390 0x5636a7405c70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4e80009960 tx=0x7f4e80009e20 comp rx=0 tx=0).stop 2024-04-25T11:40:26.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.890+0000 7f4e9bf88740 1 -- 172.21.15.90:0/2934242371 shutdown_connections 2024-04-25T11:40:26.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.890+0000 7f4e9bf88740 1 --2- 172.21.15.90:0/2934242371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636a70446b0 0x5636a7412b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:26.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.890+0000 7f4e9bf88740 1 --2- 172.21.15.90:0/2934242371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5636a703c4d0 0x5636a74061b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:26.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.890+0000 7f4e9bf88740 1 --2- 172.21.15.90:0/2934242371 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5636a7065390 0x5636a7405c70 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:26.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.890+0000 7f4e9bf88740 1 -- 172.21.15.90:0/2934242371 >> 172.21.15.90:0/2934242371 conn(0x5636a70594e0 msgr2=0x5636a7402420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:26.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.890+0000 7f4e9bf88740 1 -- 172.21.15.90:0/2934242371 shutdown_connections 2024-04-25T11:40:26.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.891+0000 7f4e9bf88740 1 -- 172.21.15.90:0/2934242371 wait complete. 2024-04-25T11:40:26.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.892+0000 7f4e9bf88740 1 -- start start 2024-04-25T11:40:26.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.892+0000 7f4e9bf88740 1 Processor -- start 2024-04-25T11:40:26.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.894+0000 7f4e9bf88740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636a703c4d0 0x5636a7410700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:26.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.894+0000 7f4e9bf88740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5636a70446b0 0x5636a7410c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:26.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.894+0000 7f4e9bf88740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5636a7065390 0x5636a741c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:26.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.894+0000 7f4e9bf88740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5636a7413240 con 0x5636a703c4d0 2024-04-25T11:40:26.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.895+0000 7f4e9bf88740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5636a72f2d10 con 0x5636a7065390 2024-04-25T11:40:26.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.895+0000 7f4e9bf88740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5636a74133c0 con 0x5636a70446b0 2024-04-25T11:40:26.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.895+0000 7f4e9bf88740 10 client.0 _ll_register_callbacks cb 0x5636a731f210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:26.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.896+0000 7f4e8e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636a703c4d0 0x5636a7410700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:26.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.897+0000 7f4e8dd9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5636a70446b0 0x5636a7410c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:26.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.897+0000 7f4e8ed9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5636a7065390 0x5636a741c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:26.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.897+0000 7f4e8ed9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5636a7065390 0x5636a741c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:33038/0 (socket says 172.21.15.90:33038) 2024-04-25T11:40:26.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.897+0000 7f4e8ed9d640 1 -- 172.21.15.90:0/3348568492 learned_addr learned my addr 172.21.15.90:0/3348568492 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:26.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.897+0000 7f4e8e59c640 1 --2- 172.21.15.90:0/3130669314 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4e780afea0 0x7f4e780b23b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x5636a7407440 tx=0x7f4e84006bb0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:26.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.899+0000 7f4e8e59c640 1 -- 172.21.15.90:0/3348568492 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5636a70446b0 msgr2=0x5636a7410c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:26.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.899+0000 7f4e8e59c640 1 --2- 172.21.15.90:0/3348568492 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5636a70446b0 0x5636a7410c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:26.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.899+0000 7f4e8e59c640 1 -- 172.21.15.90:0/3348568492 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5636a7065390 msgr2=0x5636a741c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:26.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.899+0000 7f4e8e59c640 1 --2- 172.21.15.90:0/3348568492 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5636a7065390 0x5636a741c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:26.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.899+0000 7f4e8e59c640 1 -- 172.21.15.90:0/3348568492 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636a71477d0 con 0x5636a703c4d0 2024-04-25T11:40:26.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.900+0000 7f4e8e59c640 1 --2- 172.21.15.90:0/3348568492 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636a703c4d0 0x5636a7410700 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f4e9010ad50 tx=0x7f4e84004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=478ebbeab96e6ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:26.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.901+0000 7f4e75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:40:26.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.901+0000 7f4e75ffb640 1 -- 172.21.15.90:0/3348568492 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e84019040 con 0x5636a703c4d0 2024-04-25T11:40:26.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.901+0000 7f4e9bf88740 1 -- 172.21.15.90:0/3348568492 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e80013470 con 0x5636a703c4d0 2024-04-25T11:40:26.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.901+0000 7f4e75ffb640 1 -- 172.21.15.90:0/3348568492 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f4e8401d420 con 0x5636a703c4d0 2024-04-25T11:40:26.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.903+0000 7f4e75ffb640 1 -- 172.21.15.90:0/3348568492 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4e84005990 con 0x5636a703c4d0 2024-04-25T11:40:26.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.904+0000 7f4e9bf88740 10 client.17223 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:26.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.904+0000 7f4e9bf88740 1 -- 172.21.15.90:0/3348568492 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5636a707d3f0 con 0x5636a703c4d0 2024-04-25T11:40:26.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.905+0000 7f4e75ffb640 1 -- 172.21.15.90:0/3348568492 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e80013470 con 0x5636a703c4d0 2024-04-25T11:40:26.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.905+0000 7f4e75ffb640 1 -- 172.21.15.90:0/3348568492 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f4e84012070 con 0x5636a703c4d0 2024-04-25T11:40:26.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.905+0000 7f4e9bf88740 10 client.17223 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:26.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.905+0000 7f4e9bf88740 10 client.17223 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:40:26.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.905+0000 7f4e9bf88740 20 client.17223 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:26.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.905+0000 7f4e9bf88740 1 --2- 172.21.15.90:0/3348568492 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5636a7023f60 0x5636a7422ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:26.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.905+0000 7f4e9bf88740 4 client.17223 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:40:26.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.905+0000 7f4e9bf88740 1 -- 172.21.15.90:0/3348568492 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636a7379990 con 0x5636a7023f60 2024-04-25T11:40:26.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.915+0000 7f4e8dd9b640 1 --2- 172.21.15.90:0/3348568492 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5636a7023f60 0x5636a7422ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:26.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.928+0000 7f4e8dd9b640 1 --2- 172.21.15.90:0/3348568492 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5636a7023f60 0x5636a7422ed0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a490f7edc0cc3ea server_cookie=12730cc834cc5e63 in_seq=0 out_seq=0 2024-04-25T11:40:26.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.931+0000 7f4e75ffb640 10 client.17223 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:26.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.934+0000 7f4e75ffb640 1 -- 172.21.15.90:0/3348568492 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5636a7379990 con 0x5636a7023f60 2024-04-25T11:40:26.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.934+0000 7f4e75ffb640 10 client.17223 handle_command_reply: tid=0 2024-04-25T11:40:26.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.935+0000 7f4e9bf88740 1 client.17223 shutdown 2024-04-25T11:40:26.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.948+0000 7f4e9bf88740 1 -- 172.21.15.90:0/3348568492 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636a703c4d0 msgr2=0x5636a7410700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:26.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.948+0000 7f4e9bf88740 1 --2- 172.21.15.90:0/3348568492 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636a703c4d0 0x5636a7410700 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f4e9010ad50 tx=0x7f4e84004770 comp rx=0 tx=0).stop 2024-04-25T11:40:26.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.948+0000 7f4e9bf88740 1 -- 172.21.15.90:0/3348568492 shutdown_connections 2024-04-25T11:40:26.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.948+0000 7f4e9bf88740 1 --2- 172.21.15.90:0/3348568492 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5636a7023f60 0x5636a7422ed0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:26.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.948+0000 7f4e9bf88740 1 --2- 172.21.15.90:0/3348568492 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5636a7065390 0x5636a741c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:26.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.948+0000 7f4e9bf88740 1 --2- 172.21.15.90:0/3348568492 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5636a70446b0 0x5636a7410c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:26.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.948+0000 7f4e9bf88740 1 --2- 172.21.15.90:0/3348568492 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636a703c4d0 0x5636a7410700 secure :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f4e9010ad50 tx=0x7f4e84004770 comp rx=0 tx=0).stop 2024-04-25T11:40:26.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.948+0000 7f4e9bf88740 1 -- 172.21.15.90:0/3348568492 >> 172.21.15.90:0/3348568492 conn(0x5636a70594e0 msgr2=0x5636a7407990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:26.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.950+0000 7f4e75ffb640 0 client.17223 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:26.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.950+0000 7f4e9bf88740 1 -- 172.21.15.90:0/3348568492 shutdown_connections 2024-04-25T11:40:26.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.953+0000 7f4e9bf88740 1 -- 172.21.15.90:0/3348568492 wait complete. 2024-04-25T11:40:26.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:26.954+0000 7f4e9bf88740 20 client.17223 trim_cache size 0 max 16384 2024-04-25T11:40:27.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.033+0000 7f4e9bf88740 1 Processor -- start 2024-04-25T11:40:27.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.036+0000 7f4e9bf88740 1 -- start start 2024-04-25T11:40:27.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.037+0000 7f4e9bf88740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636a70446b0 0x5636a740f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:27.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.037+0000 7f4e9bf88740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5636a7065390 0x5636a740b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:27.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.037+0000 7f4e9bf88740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5636a6ff93a0 0x5636a7407fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:27.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.037+0000 7f4e9bf88740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7000eac0 con 0x5636a70446b0 2024-04-25T11:40:27.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.037+0000 7f4e9bf88740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4e70010600 con 0x5636a7065390 2024-04-25T11:40:27.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.037+0000 7f4e9bf88740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7000f190 con 0x5636a6ff93a0 2024-04-25T11:40:27.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.038+0000 7f4e8ed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5636a6ff93a0 0x5636a7407fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:27.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.038+0000 7f4e8dd9b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5636a7065390 0x5636a740b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:27.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.039+0000 7f4e8dd9b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5636a7065390 0x5636a740b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:33052/0 (socket says 172.21.15.90:33052) 2024-04-25T11:40:27.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.039+0000 7f4e8dd9b640 1 -- 172.21.15.90:0/1949558367 learned_addr learned my addr 172.21.15.90:0/1949558367 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:27.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.039+0000 7f4e8ed9d640 1 -- 172.21.15.90:0/1949558367 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5636a7065390 msgr2=0x5636a740b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:27.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.039+0000 7f4e8ed9d640 1 --2- 172.21.15.90:0/1949558367 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5636a7065390 0x5636a740b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:27.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.039+0000 7f4e8ed9d640 1 -- 172.21.15.90:0/1949558367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636a70446b0 msgr2=0x5636a740f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:27.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.039+0000 7f4e8ed9d640 1 --2- 172.21.15.90:0/1949558367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636a70446b0 0x5636a740f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:27.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.040+0000 7f4e8ed9d640 1 -- 172.21.15.90:0/1949558367 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636a703d0f0 con 0x5636a6ff93a0 2024-04-25T11:40:27.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.040+0000 7f4e8ed9d640 1 --2- 172.21.15.90:0/1949558367 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5636a6ff93a0 0x5636a7407fa0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4e80006fd0 tx=0x7f4e80009660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7de92a4db60dedf2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:27.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.041+0000 7f4e77fff640 1 -- 172.21.15.90:0/1949558367 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e80017070 con 0x5636a6ff93a0 2024-04-25T11:40:27.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.041+0000 7f4e77fff640 1 -- 172.21.15.90:0/1949558367 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e800044f0 con 0x5636a6ff93a0 2024-04-25T11:40:27.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.041+0000 7f4e77fff640 1 -- 172.21.15.90:0/1949558367 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e80012e80 con 0x5636a6ff93a0 2024-04-25T11:40:27.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.042+0000 7f4e9bf88740 1 -- 172.21.15.90:0/1949558367 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5636a6ff93a0 msgr2=0x5636a7407fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:27.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.042+0000 7f4e9bf88740 1 --2- 172.21.15.90:0/1949558367 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5636a6ff93a0 0x5636a7407fa0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4e80006fd0 tx=0x7f4e80009660 comp rx=0 tx=0).stop 2024-04-25T11:40:27.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.042+0000 7f4e9bf88740 1 -- 172.21.15.90:0/1949558367 shutdown_connections 2024-04-25T11:40:27.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.042+0000 7f4e9bf88740 1 --2- 172.21.15.90:0/1949558367 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5636a6ff93a0 0x5636a7407fa0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:27.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.043+0000 7f4e9bf88740 1 --2- 172.21.15.90:0/1949558367 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5636a7065390 0x5636a740b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:27.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.043+0000 7f4e9bf88740 1 --2- 172.21.15.90:0/1949558367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636a70446b0 0x5636a740f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:27.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.043+0000 7f4e9bf88740 1 -- 172.21.15.90:0/1949558367 >> 172.21.15.90:0/1949558367 conn(0x5636a70594e0 msgr2=0x5636a7407890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:27.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.045+0000 7f4e9bf88740 1 -- 172.21.15.90:0/1949558367 shutdown_connections 2024-04-25T11:40:27.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.045+0000 7f4e9bf88740 1 -- 172.21.15.90:0/1949558367 wait complete. 2024-04-25T11:40:27.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.046+0000 7f4e9bf88740 1 -- start start 2024-04-25T11:40:27.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.046+0000 7f4e9bf88740 1 Processor -- start 2024-04-25T11:40:27.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.047+0000 7f4e9bf88740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636a6ff93a0 0x5636a7413780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:27.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.047+0000 7f4e9bf88740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5636a70446b0 0x5636a7413cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:27.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.048+0000 7f4e9bf88740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5636a7065390 0x5636a7414200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:27.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.048+0000 7f4e9bf88740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7000ff30 con 0x5636a6ff93a0 2024-04-25T11:40:27.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.048+0000 7f4e8dd9b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5636a70446b0 0x5636a7413cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:27.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.048+0000 7f4e8dd9b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5636a70446b0 0x5636a7413cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:33064/0 (socket says 172.21.15.90:33064) 2024-04-25T11:40:27.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.048+0000 7f4e8dd9b640 1 -- 172.21.15.90:0/1636529050 learned_addr learned my addr 172.21.15.90:0/1636529050 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:27.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.048+0000 7f4e9bf88740 1 -- 172.21.15.90:0/1636529050 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7000f860 con 0x5636a70446b0 2024-04-25T11:40:27.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.048+0000 7f4e9bf88740 1 -- 172.21.15.90:0/1636529050 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7000caa0 con 0x5636a7065390 2024-04-25T11:40:27.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.048+0000 7f4e9bf88740 10 client.0 _ll_register_callbacks cb 0x7f4e84016040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:27.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.049+0000 7f4e8ed9d640 1 --2- 172.21.15.90:0/1636529050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5636a7065390 0x5636a7414200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:27.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.049+0000 7f4e8e59c640 1 --2- 172.21.15.90:0/1636529050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636a6ff93a0 0x5636a7413780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:27.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.049+0000 7f4e8dd9b640 1 -- 172.21.15.90:0/1636529050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5636a7065390 msgr2=0x5636a7414200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:27.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.049+0000 7f4e8dd9b640 1 --2- 172.21.15.90:0/1636529050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5636a7065390 0x5636a7414200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:27.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.049+0000 7f4e8dd9b640 1 -- 172.21.15.90:0/1636529050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636a6ff93a0 msgr2=0x5636a7413780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:27.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.049+0000 7f4e8dd9b640 1 --2- 172.21.15.90:0/1636529050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636a6ff93a0 0x5636a7413780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:27.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.049+0000 7f4e8dd9b640 1 -- 172.21.15.90:0/1636529050 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636a732c4b0 con 0x5636a70446b0 2024-04-25T11:40:27.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.049+0000 7f4e8ed9d640 1 --2- 172.21.15.90:0/1636529050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5636a7065390 0x5636a7414200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:40:27.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.049+0000 7f4e8e59c640 1 --2- 172.21.15.90:0/1636529050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636a6ff93a0 0x5636a7413780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:40:27.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.049+0000 7f4e9bf88740 1 -- 172.21.15.90:0/1636529050 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e800044f0 con 0x5636a70446b0 2024-04-25T11:40:27.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.049+0000 7f4e8dd9b640 1 --2- 172.21.15.90:0/1636529050 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5636a70446b0 0x5636a7413cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4e880c9300 tx=0x7f4e8803e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7e8ec1ada29e13c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:27.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.049+0000 7f4e75ffb640 10 client.26330 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:40:27.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.050+0000 7f4e75ffb640 1 -- 172.21.15.90:0/1636529050 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e880d2130 con 0x5636a70446b0 2024-04-25T11:40:27.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.050+0000 7f4e75ffb640 1 -- 172.21.15.90:0/1636529050 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f4e880ec280 con 0x5636a70446b0 2024-04-25T11:40:27.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.051+0000 7f4e75ffb640 1 -- 172.21.15.90:0/1636529050 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4e880c94a0 con 0x5636a70446b0 2024-04-25T11:40:27.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.054+0000 7f4e9bf88740 10 client.26330 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:27.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.054+0000 7f4e9bf88740 1 -- 172.21.15.90:0/1636529050 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({fsmap=28}) -- 0x5636a748a470 con 0x5636a70446b0 2024-04-25T11:40:27.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.055+0000 7f4e75ffb640 1 -- 172.21.15.90:0/1636529050 <== mon.2 v2:172.21.15.138:3300/0 4 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f4e880c9bf0 con 0x5636a70446b0 2024-04-25T11:40:27.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.056+0000 7f4e9bf88740 10 client.26330 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:27.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.056+0000 7f4e9bf88740 10 client.26330 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:40:27.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.057+0000 7f4e9bf88740 20 client.26330 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:27.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.057+0000 7f4e9bf88740 1 --2- 172.21.15.90:0/1636529050 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5636a748aa60 0x5636a748cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:27.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.057+0000 7f4e9bf88740 4 client.26330 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:40:27.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.057+0000 7f4e9bf88740 1 -- 172.21.15.90:0/1636529050 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5636a7373800 con 0x5636a748aa60 2024-04-25T11:40:27.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.058+0000 7f4e8e59c640 1 --2- 172.21.15.90:0/1636529050 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5636a748aa60 0x5636a748cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:27.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.058+0000 7f4e8e59c640 1 --2- 172.21.15.90:0/1636529050 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5636a748aa60 0x5636a748cea0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1ce449bf2be6cab server_cookie=d160ceea8fe1368b in_seq=0 out_seq=0 2024-04-25T11:40:27.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.058+0000 7f4e75ffb640 10 client.26330 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:27.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.061+0000 7f4e75ffb640 1 -- 172.21.15.90:0/1636529050 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x5636a7373800 con 0x5636a748aa60 2024-04-25T11:40:27.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.061+0000 7f4e75ffb640 10 client.26330 handle_command_reply: tid=0 2024-04-25T11:40:27.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.062+0000 7f4e9bf88740 1 client.26330 shutdown 2024-04-25T11:40:27.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.063+0000 7f4e9bf88740 1 -- 172.21.15.90:0/1636529050 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5636a70446b0 msgr2=0x5636a7413cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:27.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.063+0000 7f4e9bf88740 1 --2- 172.21.15.90:0/1636529050 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5636a70446b0 0x5636a7413cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4e880c9300 tx=0x7f4e8803e000 comp rx=0 tx=0).stop 2024-04-25T11:40:27.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.064+0000 7f4e9bf88740 1 -- 172.21.15.90:0/1636529050 shutdown_connections 2024-04-25T11:40:27.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.064+0000 7f4e9bf88740 1 --2- 172.21.15.90:0/1636529050 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5636a748aa60 0x5636a748cea0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:27.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.064+0000 7f4e75ffb640 0 client.26330 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:27.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.064+0000 7f4e9bf88740 1 --2- 172.21.15.90:0/1636529050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5636a7065390 0x5636a7414200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:27.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.064+0000 7f4e9bf88740 1 --2- 172.21.15.90:0/1636529050 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5636a70446b0 0x5636a7413cc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:27.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.064+0000 7f4e9bf88740 1 --2- 172.21.15.90:0/1636529050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636a6ff93a0 0x5636a7413780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:27.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.064+0000 7f4e9bf88740 1 -- 172.21.15.90:0/1636529050 >> 172.21.15.90:0/1636529050 conn(0x5636a70594e0 msgr2=0x5636a7411b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:27.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.066+0000 7f4e9bf88740 1 -- 172.21.15.90:0/1636529050 shutdown_connections 2024-04-25T11:40:27.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.067+0000 7f4e9bf88740 1 -- 172.21.15.90:0/1636529050 wait complete. 2024-04-25T11:40:27.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.068+0000 7f4e9bf88740 20 client.26330 trim_cache size 0 max 16384 2024-04-25T11:40:27.059 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:40:27.059 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "scrub active (0 inodes in the stack)", 2024-04-25T11:40:27.060 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-25T11:40:27.060 INFO:teuthology.orchestra.run.smithi090.stdout: "6783de6e-89d2-4996-a92f-f4a92e99c570": { 2024-04-25T11:40:27.060 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-25T11:40:27.060 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "6783de6e-89d2-4996-a92f-f4a92e99c570", 2024-04-25T11:40:27.060 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-25T11:40:27.060 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:40:27.060 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:40:27.060 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:40:27.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.074+0000 7f4e8f59e640 1 -- 172.21.15.90:0/3130669314 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4e780afea0 msgr2=0x7f4e780b23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:27.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.074+0000 7f4e8f59e640 1 --2- 172.21.15.90:0/3130669314 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4e780afea0 0x7f4e780b23b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x5636a7407440 tx=0x7f4e84006bb0 comp rx=0 tx=0).stop 2024-04-25T11:40:27.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.074+0000 7f4e8f59e640 1 -- 172.21.15.90:0/3130669314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e9011a7c0 msgr2=0x7f4e901a9ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:27.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.074+0000 7f4e8f59e640 1 --2- 172.21.15.90:0/3130669314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e9011a7c0 0x7f4e901a9ae0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f4e8802f570 tx=0x7f4e88031a20 comp rx=0 tx=0).stop 2024-04-25T11:40:27.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.074+0000 7f4e8f59e640 1 -- 172.21.15.90:0/3130669314 shutdown_connections 2024-04-25T11:40:27.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.074+0000 7f4e8f59e640 1 --2- 172.21.15.90:0/3130669314 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4e780afea0 0x7f4e780b23b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:27.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.074+0000 7f4e8f59e640 1 --2- 172.21.15.90:0/3130669314 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4e9011d120 0x7f4e901adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:27.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.074+0000 7f4e8f59e640 1 --2- 172.21.15.90:0/3130669314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e9011a7c0 0x7f4e901a9ae0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:27.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.074+0000 7f4e8f59e640 1 --2- 172.21.15.90:0/3130669314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e90109fc0 0x7f4e901a9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:27.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.075+0000 7f4e8f59e640 1 -- 172.21.15.90:0/3130669314 >> 172.21.15.90:0/3130669314 conn(0x7f4e9006f650 msgr2=0x7f4e9011abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:27.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.075+0000 7f4e8f59e640 1 -- 172.21.15.90:0/3130669314 shutdown_connections 2024-04-25T11:40:27.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:27.075+0000 7f4e8f59e640 1 -- 172.21.15.90:0/3130669314 wait complete. 2024-04-25T11:40:27.089 INFO:tasks.cephfs.filesystem:scrub status for tag:6783de6e-89d2-4996-a92f-f4a92e99c570 - {'path': '/', 'tag': '6783de6e-89d2-4996-a92f-f4a92e99c570', 'options': 'recursive,force'} 2024-04-25T11:40:27.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:27 smithi090 ceph-mon[29575]: pgmap v164: 97 pgs: 97 active+clean; 44 GiB data, 133 GiB used, 940 GiB / 1.0 TiB avail; 2.5 MiB/s rd, 556 MiB/s wr, 2.98k op/s 2024-04-25T11:40:27.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:27 smithi090 ceph-mon[29575]: scrub queued for path: / 2024-04-25T11:40:27.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:27 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:40:27.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:27 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:40:27.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:27 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:40:27.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:27 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:40:27.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:27 smithi138 ceph-mon[32328]: pgmap v164: 97 pgs: 97 active+clean; 44 GiB data, 133 GiB used, 940 GiB / 1.0 TiB avail; 2.5 MiB/s rd, 556 MiB/s wr, 2.98k op/s 2024-04-25T11:40:27.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:27 smithi138 ceph-mon[32328]: scrub queued for path: / 2024-04-25T11:40:27.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:27 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:40:27.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:27 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:40:27.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:27 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:40:27.801 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:27 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:40:27.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:27 smithi156 ceph-mon[33045]: pgmap v164: 97 pgs: 97 active+clean; 44 GiB data, 133 GiB used, 940 GiB / 1.0 TiB avail; 2.5 MiB/s rd, 556 MiB/s wr, 2.98k op/s 2024-04-25T11:40:27.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:27 smithi156 ceph-mon[33045]: scrub queued for path: / 2024-04-25T11:40:27.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:27 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:40:27.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:27 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:40:27.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:27 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:40:27.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:27 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:40:29.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:29 smithi156 ceph-mon[33045]: pgmap v165: 97 pgs: 97 active+clean; 44 GiB data, 133 GiB used, 940 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 443 MiB/s wr, 2.35k op/s 2024-04-25T11:40:29.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:29 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:40:29.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:29 smithi090 ceph-mon[29575]: pgmap v165: 97 pgs: 97 active+clean; 44 GiB data, 133 GiB used, 940 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 443 MiB/s wr, 2.35k op/s 2024-04-25T11:40:29.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:29 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:40:30.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:29 smithi138 ceph-mon[32328]: pgmap v165: 97 pgs: 97 active+clean; 44 GiB data, 133 GiB used, 940 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 443 MiB/s wr, 2.35k op/s 2024-04-25T11:40:30.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:29 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:40:31.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:31 smithi090 ceph-mon[29575]: pgmap v166: 97 pgs: 97 active+clean; 46 GiB data, 140 GiB used, 933 GiB / 1.0 TiB avail; 3.7 MiB/s rd, 650 MiB/s wr, 3.09k op/s 2024-04-25T11:40:32.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:31 smithi138 ceph-mon[32328]: pgmap v166: 97 pgs: 97 active+clean; 46 GiB data, 140 GiB used, 933 GiB / 1.0 TiB avail; 3.7 MiB/s rd, 650 MiB/s wr, 3.09k op/s 2024-04-25T11:40:32.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:31 smithi156 ceph-mon[33045]: pgmap v166: 97 pgs: 97 active+clean; 46 GiB data, 140 GiB used, 933 GiB / 1.0 TiB avail; 3.7 MiB/s rd, 650 MiB/s wr, 3.09k op/s 2024-04-25T11:40:33.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:33 smithi090 ceph-mon[29575]: pgmap v167: 97 pgs: 97 active+clean; 46 GiB data, 141 GiB used, 932 GiB / 1.0 TiB avail; 2.8 MiB/s rd, 568 MiB/s wr, 2.07k op/s 2024-04-25T11:40:34.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:33 smithi138 ceph-mon[32328]: pgmap v167: 97 pgs: 97 active+clean; 46 GiB data, 141 GiB used, 932 GiB / 1.0 TiB avail; 2.8 MiB/s rd, 568 MiB/s wr, 2.07k op/s 2024-04-25T11:40:34.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:33 smithi156 ceph-mon[33045]: pgmap v167: 97 pgs: 97 active+clean; 46 GiB data, 141 GiB used, 932 GiB / 1.0 TiB avail; 2.8 MiB/s rd, 568 MiB/s wr, 2.07k op/s 2024-04-25T11:40:34.438 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.451+0000 7fac5291f640 1 -- 172.21.15.90:0/1213615279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac4c1031b0 msgr2=0x7fac4c10f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:34.438 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.451+0000 7fac5291f640 1 --2- 172.21.15.90:0/1213615279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac4c1031b0 0x7fac4c10f4e0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fac44009ca0 tx=0x7fac4402a8f0 comp rx=0 tx=0).stop 2024-04-25T11:40:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.451+0000 7fac5291f640 1 -- 172.21.15.90:0/1213615279 shutdown_connections 2024-04-25T11:40:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.451+0000 7fac5291f640 1 --2- 172.21.15.90:0/1213615279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac4c1031b0 0x7fac4c10f4e0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.451+0000 7fac5291f640 1 --2- 172.21.15.90:0/1213615279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac4c108ef0 0x7fac4c102b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.451+0000 7fac5291f640 1 --2- 172.21.15.90:0/1213615279 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fac4c108a50 0x7fac4c102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.451+0000 7fac5291f640 1 -- 172.21.15.90:0/1213615279 >> 172.21.15.90:0/1213615279 conn(0x7fac4c0fe260 msgr2=0x7fac4c100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.451+0000 7fac5291f640 1 -- 172.21.15.90:0/1213615279 shutdown_connections 2024-04-25T11:40:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.451+0000 7fac5291f640 1 -- 172.21.15.90:0/1213615279 wait complete. 2024-04-25T11:40:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.453+0000 7fac5291f640 1 Processor -- start 2024-04-25T11:40:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.453+0000 7fac5291f640 1 -- start start 2024-04-25T11:40:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.454+0000 7fac5291f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac4c1031b0 0x7fac4c1997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.454+0000 7fac5291f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fac4c108a50 0x7fac4c199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.454+0000 7fac5291f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac4c108ef0 0x7fac4c19dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.454+0000 7fac5291f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c112d50 con 0x7fac4c1031b0 2024-04-25T11:40:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.454+0000 7fac5291f640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c112bd0 con 0x7fac4c108a50 2024-04-25T11:40:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.454+0000 7fac5291f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c112ed0 con 0x7fac4c108ef0 2024-04-25T11:40:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.455+0000 7fac4bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac4c1031b0 0x7fac4c1997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.455+0000 7fac4bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac4c1031b0 0x7fac4c1997b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35762/0 (socket says 172.21.15.90:35762) 2024-04-25T11:40:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.455+0000 7fac4bfff640 1 -- 172.21.15.90:0/1712335662 learned_addr learned my addr 172.21.15.90:0/1712335662 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.455+0000 7fac50ebc640 1 --2- 172.21.15.90:0/1712335662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac4c108ef0 0x7fac4c19dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.455+0000 7fac4bfff640 1 -- 172.21.15.90:0/1712335662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac4c108ef0 msgr2=0x7fac4c19dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.455+0000 7fac4bfff640 1 --2- 172.21.15.90:0/1712335662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac4c108ef0 0x7fac4c19dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.455+0000 7fac4bfff640 1 -- 172.21.15.90:0/1712335662 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fac4c108a50 msgr2=0x7fac4c199d10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:40:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.455+0000 7fac4bfff640 1 --2- 172.21.15.90:0/1712335662 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fac4c108a50 0x7fac4c199d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.455+0000 7fac4bfff640 1 -- 172.21.15.90:0/1712335662 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac4c19e610 con 0x7fac4c1031b0 2024-04-25T11:40:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.456+0000 7fac4bfff640 1 --2- 172.21.15.90:0/1712335662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac4c1031b0 0x7fac4c1997b0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fac3c00b450 tx=0x7fac3c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:34.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.456+0000 7fac497fa640 1 -- 172.21.15.90:0/1712335662 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c015020 con 0x7fac4c1031b0 2024-04-25T11:40:34.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.456+0000 7fac497fa640 1 -- 172.21.15.90:0/1712335662 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac3c011d40 con 0x7fac4c1031b0 2024-04-25T11:40:34.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.456+0000 7fac5291f640 1 -- 172.21.15.90:0/1712335662 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac4c19e900 con 0x7fac4c1031b0 2024-04-25T11:40:34.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.456+0000 7fac497fa640 1 -- 172.21.15.90:0/1712335662 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c00f340 con 0x7fac4c1031b0 2024-04-25T11:40:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.457+0000 7fac5291f640 1 -- 172.21.15.90:0/1712335662 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac4c1a6280 con 0x7fac4c1031b0 2024-04-25T11:40:34.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.463+0000 7fac597d1740 1 Processor -- start 2024-04-25T11:40:34.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.463+0000 7fac597d1740 1 -- start start 2024-04-25T11:40:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.463+0000 7fac597d1740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564c581111b0 0x564c584b0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.463+0000 7fac597d1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c580e82f0 0x564c584b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.463+0000 7fac497fa640 1 -- 172.21.15.90:0/1712335662 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fac3c018020 con 0x7fac4c1031b0 2024-04-25T11:40:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.463+0000 7fac597d1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564c580f04d0 0x564c584b1380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.464+0000 7fac597d1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564c58417460 con 0x564c580e82f0 2024-04-25T11:40:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.464+0000 7fac597d1740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x564c583d1590 con 0x564c581111b0 2024-04-25T11:40:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.464+0000 7fac597d1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x564c58413420 con 0x564c580f04d0 2024-04-25T11:40:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.464+0000 7fac4bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c580e82f0 0x564c584b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.464+0000 7fac4bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c580e82f0 0x564c584b0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35776/0 (socket says 172.21.15.90:35776) 2024-04-25T11:40:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.464+0000 7fac4bfff640 1 -- 172.21.15.90:0/1577257289 learned_addr learned my addr 172.21.15.90:0/1577257289 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.464+0000 7fac4b7fe640 1 --2- 172.21.15.90:0/1577257289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564c580f04d0 0x564c584b1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:34.454 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.465+0000 7fac50ebc640 1 --2- 172.21.15.90:0/1577257289 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564c581111b0 0x564c584b0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:34.454 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.465+0000 7fac4bfff640 1 -- 172.21.15.90:0/1577257289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564c580f04d0 msgr2=0x564c584b1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:34.454 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.465+0000 7fac4bfff640 1 --2- 172.21.15.90:0/1577257289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564c580f04d0 0x564c584b1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:34.454 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.465+0000 7fac4bfff640 1 -- 172.21.15.90:0/1577257289 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564c581111b0 msgr2=0x564c584b0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:34.454 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.465+0000 7fac4bfff640 1 --2- 172.21.15.90:0/1577257289 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564c581111b0 0x564c584b0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:34.454 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.465+0000 7fac4bfff640 1 -- 172.21.15.90:0/1577257289 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c583de0d0 con 0x564c580e82f0 2024-04-25T11:40:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.465+0000 7fac4bfff640 1 --2- 172.21.15.90:0/1577257289 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c580e82f0 0x564c584b0e40 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0db490 tx=0x7fac3c0e1800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2e90c5f18b588e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.466+0000 7fac497fa640 1 --2- 172.21.15.90:0/1712335662 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fac340afda0 0x7fac340b22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.466+0000 7fac327fc640 1 -- 172.21.15.90:0/1577257289 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c0e2060 con 0x564c580e82f0 2024-04-25T11:40:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.466+0000 7fac327fc640 1 -- 172.21.15.90:0/1577257289 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac3c09d990 con 0x564c580e82f0 2024-04-25T11:40:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.466+0000 7fac327fc640 1 -- 172.21.15.90:0/1577257289 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c0a5fe0 con 0x564c580e82f0 2024-04-25T11:40:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.466+0000 7fac497fa640 1 -- 172.21.15.90:0/1712335662 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fac3c0e1170 con 0x7fac4c1031b0 2024-04-25T11:40:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.468+0000 7fac597d1740 1 -- 172.21.15.90:0/1577257289 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c580e82f0 msgr2=0x564c584b0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.468+0000 7fac597d1740 1 --2- 172.21.15.90:0/1577257289 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c580e82f0 0x564c584b0e40 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0db490 tx=0x7fac3c0e1800 comp rx=0 tx=0).stop 2024-04-25T11:40:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.468+0000 7fac597d1740 1 -- 172.21.15.90:0/1577257289 shutdown_connections 2024-04-25T11:40:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.468+0000 7fac597d1740 1 --2- 172.21.15.90:0/1577257289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564c580f04d0 0x564c584b1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:34.458 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.468+0000 7fac597d1740 1 --2- 172.21.15.90:0/1577257289 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c580e82f0 0x564c584b0e40 secure :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0db490 tx=0x7fac3c0e1800 comp rx=0 tx=0).stop 2024-04-25T11:40:34.458 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.468+0000 7fac597d1740 1 --2- 172.21.15.90:0/1577257289 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564c581111b0 0x564c584b0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:34.458 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.468+0000 7fac597d1740 1 -- 172.21.15.90:0/1577257289 >> 172.21.15.90:0/1577257289 conn(0x564c58105300 msgr2=0x564c584ad2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:34.458 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.469+0000 7fac50ebc640 1 --2- 172.21.15.90:0/1712335662 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fac340afda0 0x7fac340b22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.472+0000 7fac50ebc640 1 --2- 172.21.15.90:0/1712335662 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fac340afda0 0x7fac340b22b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fac440094e0 tx=0x7fac4402f4d0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.472+0000 7fac597d1740 1 -- 172.21.15.90:0/1577257289 shutdown_connections 2024-04-25T11:40:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.472+0000 7fac597d1740 1 -- 172.21.15.90:0/1577257289 wait complete. 2024-04-25T11:40:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.473+0000 7fac597d1740 1 -- start start 2024-04-25T11:40:34.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.474+0000 7fac597d1740 1 Processor -- start 2024-04-25T11:40:34.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.475+0000 7fac597d1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564c580f04d0 0x564c584bf300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:34.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.475+0000 7fac597d1740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564c581111b0 0x564c584bb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:34.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.475+0000 7fac597d1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c580cfd80 0x564c584bbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:34.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.475+0000 7fac597d1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564c584b3b50 con 0x564c580cfd80 2024-04-25T11:40:34.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.475+0000 7fac597d1740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x564c5839eb60 con 0x564c581111b0 2024-04-25T11:40:34.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.475+0000 7fac597d1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x564c584b3cd0 con 0x564c580f04d0 2024-04-25T11:40:34.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.475+0000 7fac597d1740 10 client.0 _ll_register_callbacks cb 0x564c583c32b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:34.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.475+0000 7fac50ebc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c580cfd80 0x564c584bbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.475+0000 7fac50ebc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c580cfd80 0x564c584bbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35786/0 (socket says 172.21.15.90:35786) 2024-04-25T11:40:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.475+0000 7fac50ebc640 1 -- 172.21.15.90:0/2823973326 learned_addr learned my addr 172.21.15.90:0/2823973326 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.476+0000 7fac4bfff640 1 --2- 172.21.15.90:0/2823973326 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564c580f04d0 0x564c584bf300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.476+0000 7fac50ebc640 1 -- 172.21.15.90:0/2823973326 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564c580f04d0 msgr2=0x564c584bf300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.476+0000 7fac50ebc640 1 --2- 172.21.15.90:0/2823973326 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564c580f04d0 0x564c584bf300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:34.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.476+0000 7fac50ebc640 1 -- 172.21.15.90:0/2823973326 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564c581111b0 msgr2=0x564c584bb660 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:40:34.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.476+0000 7fac50ebc640 1 --2- 172.21.15.90:0/2823973326 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564c581111b0 0x564c584bb660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:34.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.476+0000 7fac50ebc640 1 -- 172.21.15.90:0/2823973326 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c58129210 con 0x564c580cfd80 2024-04-25T11:40:34.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.476+0000 7fac50ebc640 1 --2- 172.21.15.90:0/2823973326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c580cfd80 0x564c584bbba0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fac44035000 tx=0x7fac440095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6841b3cac2004b38 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.476+0000 7fac2b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:40:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.477+0000 7fac2b7fe640 1 -- 172.21.15.90:0/2823973326 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac440047c0 con 0x564c580cfd80 2024-04-25T11:40:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.477+0000 7fac597d1740 1 -- 172.21.15.90:0/2823973326 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac3c09d990 con 0x564c580cfd80 2024-04-25T11:40:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.477+0000 7fac2b7fe640 1 -- 172.21.15.90:0/2823973326 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fac4402f640 con 0x564c580cfd80 2024-04-25T11:40:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.477+0000 7fac2b7fe640 1 -- 172.21.15.90:0/2823973326 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fac44031910 con 0x564c580cfd80 2024-04-25T11:40:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.477+0000 7fac597d1740 10 client.17235 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.477+0000 7fac597d1740 1 -- 172.21.15.90:0/2823973326 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564c581f5620 con 0x564c580cfd80 2024-04-25T11:40:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.477+0000 7fac2b7fe640 1 -- 172.21.15.90:0/2823973326 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac3c09d990 con 0x564c580cfd80 2024-04-25T11:40:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.478+0000 7fac2b7fe640 1 -- 172.21.15.90:0/2823973326 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fac4404e070 con 0x564c580cfd80 2024-04-25T11:40:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.478+0000 7fac597d1740 10 client.17235 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.478+0000 7fac597d1740 10 client.17235 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:40:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.478+0000 7fac597d1740 20 client.17235 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.478+0000 7fac597d1740 1 --2- 172.21.15.90:0/2823973326 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564c580a51c0 0x564c584d5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.478+0000 7fac597d1740 4 client.17235 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:40:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.478+0000 7fac597d1740 1 -- 172.21.15.90:0/2823973326 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c58425070 con 0x564c580a51c0 2024-04-25T11:40:34.468 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.480+0000 7fac4bfff640 1 --2- 172.21.15.90:0/2823973326 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564c580a51c0 0x564c584d5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:34.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.486+0000 7fac2b7fe640 10 client.17235 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:34.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.486+0000 7fac4bfff640 1 --2- 172.21.15.90:0/2823973326 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564c580a51c0 0x564c584d5fc0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=466f27b13d725b02 server_cookie=a181e15825ddde4b in_seq=0 out_seq=0 2024-04-25T11:40:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.492+0000 7fac2b7fe640 1 -- 172.21.15.90:0/2823973326 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564c58425070 con 0x564c580a51c0 2024-04-25T11:40:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.492+0000 7fac2b7fe640 10 client.17235 handle_command_reply: tid=0 2024-04-25T11:40:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.492+0000 7fac597d1740 1 client.17235 shutdown 2024-04-25T11:40:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.493+0000 7fac597d1740 1 -- 172.21.15.90:0/2823973326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c580cfd80 msgr2=0x564c584bbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.493+0000 7fac597d1740 1 --2- 172.21.15.90:0/2823973326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c580cfd80 0x564c584bbba0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fac44035000 tx=0x7fac440095c0 comp rx=0 tx=0).stop 2024-04-25T11:40:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.493+0000 7fac597d1740 1 -- 172.21.15.90:0/2823973326 shutdown_connections 2024-04-25T11:40:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.493+0000 7fac597d1740 1 --2- 172.21.15.90:0/2823973326 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564c580a51c0 0x564c584d5fc0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.493+0000 7fac597d1740 1 --2- 172.21.15.90:0/2823973326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c580cfd80 0x564c584bbba0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.493+0000 7fac597d1740 1 --2- 172.21.15.90:0/2823973326 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564c581111b0 0x564c584bb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.493+0000 7fac597d1740 1 --2- 172.21.15.90:0/2823973326 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564c580f04d0 0x564c584bf300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.493+0000 7fac597d1740 1 -- 172.21.15.90:0/2823973326 >> 172.21.15.90:0/2823973326 conn(0x564c58105300 msgr2=0x564c584b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.493+0000 7fac2b7fe640 0 client.17235 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.493+0000 7fac597d1740 1 -- 172.21.15.90:0/2823973326 shutdown_connections 2024-04-25T11:40:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.493+0000 7fac597d1740 1 -- 172.21.15.90:0/2823973326 wait complete. 2024-04-25T11:40:34.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.493+0000 7fac597d1740 20 client.17235 trim_cache size 0 max 16384 2024-04-25T11:40:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.514+0000 7fac597d1740 1 Processor -- start 2024-04-25T11:40:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.514+0000 7fac597d1740 1 -- start start 2024-04-25T11:40:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.514+0000 7fac597d1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564c580cfd80 0x564c584b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.514+0000 7fac597d1740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564c580f04d0 0x564c584b5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.514+0000 7fac597d1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c581111b0 0x564c584db7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.514+0000 7fac597d1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c1add00 con 0x564c581111b0 2024-04-25T11:40:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.514+0000 7fac597d1740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c1af840 con 0x564c580f04d0 2024-04-25T11:40:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.514+0000 7fac597d1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c1ae3d0 con 0x564c580cfd80 2024-04-25T11:40:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.514+0000 7fac50ebc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c581111b0 0x564c584db7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.514+0000 7fac50ebc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c581111b0 0x564c584db7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35800/0 (socket says 172.21.15.90:35800) 2024-04-25T11:40:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.514+0000 7fac50ebc640 1 -- 172.21.15.90:0/1755711820 learned_addr learned my addr 172.21.15.90:0/1755711820 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.514+0000 7fac50ebc640 1 -- 172.21.15.90:0/1755711820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564c580cfd80 msgr2=0x564c584b4f30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:40:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.514+0000 7fac50ebc640 1 --2- 172.21.15.90:0/1755711820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564c580cfd80 0x564c584b4f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.514+0000 7fac50ebc640 1 -- 172.21.15.90:0/1755711820 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564c580f04d0 msgr2=0x564c584b5470 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:40:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.514+0000 7fac50ebc640 1 --2- 172.21.15.90:0/1755711820 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564c580f04d0 0x564c584b5470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.514+0000 7fac50ebc640 1 -- 172.21.15.90:0/1755711820 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c584ae3d0 con 0x564c581111b0 2024-04-25T11:40:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.515+0000 7fac50ebc640 1 --2- 172.21.15.90:0/1755711820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c581111b0 0x564c584db7c0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fac44030a90 tx=0x7fac44031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe58f5798b583a4d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.515+0000 7fac31ffb640 1 -- 172.21.15.90:0/1755711820 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac440305f0 con 0x564c581111b0 2024-04-25T11:40:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.515+0000 7fac31ffb640 1 -- 172.21.15.90:0/1755711820 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac44036070 con 0x564c581111b0 2024-04-25T11:40:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.515+0000 7fac31ffb640 1 -- 172.21.15.90:0/1755711820 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4402f880 con 0x564c581111b0 2024-04-25T11:40:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.516+0000 7fac597d1740 1 -- 172.21.15.90:0/1755711820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c581111b0 msgr2=0x564c584db7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.516+0000 7fac597d1740 1 --2- 172.21.15.90:0/1755711820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c581111b0 0x564c584db7c0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fac44030a90 tx=0x7fac44031ee0 comp rx=0 tx=0).stop 2024-04-25T11:40:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.516+0000 7fac597d1740 1 -- 172.21.15.90:0/1755711820 shutdown_connections 2024-04-25T11:40:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.516+0000 7fac597d1740 1 --2- 172.21.15.90:0/1755711820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c581111b0 0x564c584db7c0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.516+0000 7fac597d1740 1 --2- 172.21.15.90:0/1755711820 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564c580f04d0 0x564c584b5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.516+0000 7fac597d1740 1 --2- 172.21.15.90:0/1755711820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564c580cfd80 0x564c584b4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.516+0000 7fac597d1740 1 -- 172.21.15.90:0/1755711820 >> 172.21.15.90:0/1755711820 conn(0x564c58105300 msgr2=0x564c584b9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.516+0000 7fac597d1740 1 -- 172.21.15.90:0/1755711820 shutdown_connections 2024-04-25T11:40:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.516+0000 7fac597d1740 1 -- 172.21.15.90:0/1755711820 wait complete. 2024-04-25T11:40:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.516+0000 7fac597d1740 1 -- start start 2024-04-25T11:40:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.517+0000 7fac597d1740 1 Processor -- start 2024-04-25T11:40:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.517+0000 7fac597d1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564c580cfd80 0x564c584c5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.517+0000 7fac597d1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c580f04d0 0x564c584c26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.517+0000 7fac597d1740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564c581111b0 0x564c584c2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.517+0000 7fac597d1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c1af170 con 0x564c580f04d0 2024-04-25T11:40:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.517+0000 7fac597d1740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c1aeaa0 con 0x564c581111b0 2024-04-25T11:40:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.517+0000 7fac597d1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c1abce0 con 0x564c580cfd80 2024-04-25T11:40:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.517+0000 7fac597d1740 10 client.0 _ll_register_callbacks cb 0x7fac44009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.517+0000 7fac4b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c580f04d0 0x564c584c26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:34.508 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.517+0000 7fac4b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c580f04d0 0x564c584c26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35806/0 (socket says 172.21.15.90:35806) 2024-04-25T11:40:34.508 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.518+0000 7fac4b7fe640 1 -- 172.21.15.90:0/1978943850 learned_addr learned my addr 172.21.15.90:0/1978943850 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:34.508 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.518+0000 7fac4b7fe640 1 -- 172.21.15.90:0/1978943850 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564c580cfd80 msgr2=0x564c584c5380 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:40:34.508 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.518+0000 7fac4b7fe640 1 --2- 172.21.15.90:0/1978943850 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564c580cfd80 0x564c584c5380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:34.508 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.518+0000 7fac4b7fe640 1 -- 172.21.15.90:0/1978943850 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564c581111b0 msgr2=0x564c584c2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:34.508 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.518+0000 7fac4b7fe640 1 --2- 172.21.15.90:0/1978943850 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564c581111b0 0x564c584c2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:34.508 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.518+0000 7fac4b7fe640 1 -- 172.21.15.90:0/1978943850 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c584c34d0 con 0x564c580f04d0 2024-04-25T11:40:34.508 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.518+0000 7fac4b7fe640 1 --2- 172.21.15.90:0/1978943850 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c580f04d0 0x564c584c26f0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fac4c06c4c0 tx=0x7fac4000e660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5b3bebce0924f25 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:34.508 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.518+0000 7fac2b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:40:34.508 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.518+0000 7fac2b7fe640 1 -- 172.21.15.90:0/1978943850 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac40017070 con 0x564c580f04d0 2024-04-25T11:40:34.508 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.518+0000 7fac2b7fe640 1 -- 172.21.15.90:0/1978943850 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fac40013470 con 0x564c580f04d0 2024-04-25T11:40:34.508 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.519+0000 7fac597d1740 1 -- 172.21.15.90:0/1978943850 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac44036070 con 0x564c580f04d0 2024-04-25T11:40:34.509 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.519+0000 7fac2b7fe640 1 -- 172.21.15.90:0/1978943850 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fac40021050 con 0x564c580f04d0 2024-04-25T11:40:34.509 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.519+0000 7fac597d1740 10 client.17241 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:34.509 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.519+0000 7fac597d1740 1 -- 172.21.15.90:0/1978943850 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564c5853e440 con 0x564c580f04d0 2024-04-25T11:40:34.509 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.519+0000 7fac2b7fe640 1 -- 172.21.15.90:0/1978943850 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac44036070 con 0x564c580f04d0 2024-04-25T11:40:34.509 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.520+0000 7fac2b7fe640 1 -- 172.21.15.90:0/1978943850 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fac4001c7e0 con 0x564c580f04d0 2024-04-25T11:40:34.510 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.520+0000 7fac597d1740 10 client.17241 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:34.510 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.520+0000 7fac597d1740 10 client.17241 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:40:34.510 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.520+0000 7fac597d1740 20 client.17241 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:34.510 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.520+0000 7fac597d1740 1 --2- 172.21.15.90:0/1978943850 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564c5853ec70 0x564c58541100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:34.510 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.520+0000 7fac597d1740 4 client.17241 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:40:34.510 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.520+0000 7fac597d1740 1 -- 172.21.15.90:0/1978943850 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564c5841eee0 con 0x564c5853ec70 2024-04-25T11:40:34.510 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.521+0000 7fac4bfff640 1 --2- 172.21.15.90:0/1978943850 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564c5853ec70 0x564c58541100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:34.511 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.523+0000 7fac2b7fe640 10 client.17241 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:34.511 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:34.523+0000 7fac4bfff640 1 --2- 172.21.15.90:0/1978943850 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564c5853ec70 0x564c58541100 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9125a8223a26060 server_cookie=fd4f01c2a6c12156 in_seq=0 out_seq=0 2024-04-25T11:40:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.982+0000 7fac2b7fe640 1 -- 172.21.15.90:0/1978943850 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564c5841eee0 con 0x564c5853ec70 2024-04-25T11:40:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.982+0000 7fac2b7fe640 10 client.17241 handle_command_reply: tid=0 2024-04-25T11:40:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.982+0000 7fac597d1740 1 client.17241 shutdown 2024-04-25T11:40:35.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.982+0000 7fac597d1740 1 -- 172.21.15.90:0/1978943850 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c580f04d0 msgr2=0x564c584c26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:35.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.982+0000 7fac597d1740 1 --2- 172.21.15.90:0/1978943850 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c580f04d0 0x564c584c26f0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fac4c06c4c0 tx=0x7fac4000e660 comp rx=0 tx=0).stop 2024-04-25T11:40:35.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.983+0000 7fac597d1740 1 -- 172.21.15.90:0/1978943850 shutdown_connections 2024-04-25T11:40:35.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.983+0000 7fac597d1740 1 --2- 172.21.15.90:0/1978943850 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564c5853ec70 0x564c58541100 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:35.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.983+0000 7fac597d1740 1 --2- 172.21.15.90:0/1978943850 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564c581111b0 0x564c584c2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:35.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.983+0000 7fac597d1740 1 --2- 172.21.15.90:0/1978943850 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564c580f04d0 0x564c584c26f0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:35.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.983+0000 7fac597d1740 1 --2- 172.21.15.90:0/1978943850 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564c580cfd80 0x564c584c5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:35.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.983+0000 7fac597d1740 1 -- 172.21.15.90:0/1978943850 >> 172.21.15.90:0/1978943850 conn(0x564c58105300 msgr2=0x564c584b9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:35.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.983+0000 7fac2b7fe640 0 client.17241 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:35.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.983+0000 7fac597d1740 1 -- 172.21.15.90:0/1978943850 shutdown_connections 2024-04-25T11:40:35.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.983+0000 7fac597d1740 1 -- 172.21.15.90:0/1978943850 wait complete. 2024-04-25T11:40:35.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.983+0000 7fac597d1740 20 client.17241 trim_cache size 0 max 16384 2024-04-25T11:40:35.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:40:35.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:40:35.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:40:35.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:40:35.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.991+0000 7fac5291f640 1 -- 172.21.15.90:0/1712335662 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fac340afda0 msgr2=0x7fac340b22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:35.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.991+0000 7fac5291f640 1 --2- 172.21.15.90:0/1712335662 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fac340afda0 0x7fac340b22b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fac440094e0 tx=0x7fac4402f4d0 comp rx=0 tx=0).stop 2024-04-25T11:40:35.980 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.991+0000 7fac5291f640 1 -- 172.21.15.90:0/1712335662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac4c1031b0 msgr2=0x7fac4c1997b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:35.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.991+0000 7fac5291f640 1 --2- 172.21.15.90:0/1712335662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac4c1031b0 0x7fac4c1997b0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fac3c00b450 tx=0x7fac3c00b910 comp rx=0 tx=0).stop 2024-04-25T11:40:35.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.991+0000 7fac5291f640 1 -- 172.21.15.90:0/1712335662 shutdown_connections 2024-04-25T11:40:35.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.991+0000 7fac5291f640 1 --2- 172.21.15.90:0/1712335662 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fac340afda0 0x7fac340b22b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:35.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.991+0000 7fac5291f640 1 --2- 172.21.15.90:0/1712335662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac4c108ef0 0x7fac4c19dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:35.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.991+0000 7fac5291f640 1 --2- 172.21.15.90:0/1712335662 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fac4c108a50 0x7fac4c199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:35.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.991+0000 7fac5291f640 1 --2- 172.21.15.90:0/1712335662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac4c1031b0 0x7fac4c1997b0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:35.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.992+0000 7fac5291f640 1 -- 172.21.15.90:0/1712335662 >> 172.21.15.90:0/1712335662 conn(0x7fac4c0fe260 msgr2=0x7fac4c100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:35.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.992+0000 7fac5291f640 1 -- 172.21.15.90:0/1712335662 shutdown_connections 2024-04-25T11:40:35.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:35.992+0000 7fac5291f640 1 -- 172.21.15.90:0/1712335662 wait complete. 2024-04-25T11:40:36.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:35 smithi156 ceph-mon[33045]: pgmap v168: 97 pgs: 97 active+clean; 47 GiB data, 144 GiB used, 929 GiB / 1.0 TiB avail; 3.7 MiB/s rd, 636 MiB/s wr, 1.82k op/s 2024-04-25T11:40:36.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:35 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:40:36.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:35 smithi090 ceph-mon[29575]: pgmap v168: 97 pgs: 97 active+clean; 47 GiB data, 144 GiB used, 929 GiB / 1.0 TiB avail; 3.7 MiB/s rd, 636 MiB/s wr, 1.82k op/s 2024-04-25T11:40:36.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:35 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:40:36.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:35 smithi138 ceph-mon[32328]: pgmap v168: 97 pgs: 97 active+clean; 47 GiB data, 144 GiB used, 929 GiB / 1.0 TiB avail; 3.7 MiB/s rd, 636 MiB/s wr, 1.82k op/s 2024-04-25T11:40:36.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:35 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:40:37.724 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:37 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:40:37.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:37 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:40:37.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:37 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:40:38.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:38 smithi138 ceph-mon[32328]: pgmap v169: 97 pgs: 97 active+clean; 49 GiB data, 148 GiB used, 925 GiB / 1.0 TiB avail; 3.0 MiB/s rd, 732 MiB/s wr, 1.61k op/s 2024-04-25T11:40:38.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:38 smithi156 ceph-mon[33045]: pgmap v169: 97 pgs: 97 active+clean; 49 GiB data, 148 GiB used, 925 GiB / 1.0 TiB avail; 3.0 MiB/s rd, 732 MiB/s wr, 1.61k op/s 2024-04-25T11:40:38.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:38 smithi090 ceph-mon[29575]: pgmap v169: 97 pgs: 97 active+clean; 49 GiB data, 148 GiB used, 925 GiB / 1.0 TiB avail; 3.0 MiB/s rd, 732 MiB/s wr, 1.61k op/s 2024-04-25T11:40:39.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:39 smithi156 ceph-mon[33045]: pgmap v170: 97 pgs: 97 active+clean; 49 GiB data, 148 GiB used, 925 GiB / 1.0 TiB avail; 2.6 MiB/s rd, 537 MiB/s wr, 1.08k op/s 2024-04-25T11:40:39.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:39 smithi090 ceph-mon[29575]: pgmap v170: 97 pgs: 97 active+clean; 49 GiB data, 148 GiB used, 925 GiB / 1.0 TiB avail; 2.6 MiB/s rd, 537 MiB/s wr, 1.08k op/s 2024-04-25T11:40:40.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:39 smithi138 ceph-mon[32328]: pgmap v170: 97 pgs: 97 active+clean; 49 GiB data, 148 GiB used, 925 GiB / 1.0 TiB avail; 2.6 MiB/s rd, 537 MiB/s wr, 1.08k op/s 2024-04-25T11:40:41.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:41 smithi156 ceph-mon[33045]: pgmap v171: 97 pgs: 97 active+clean; 49 GiB data, 149 GiB used, 924 GiB / 1.0 TiB avail; 4.3 MiB/s rd, 725 MiB/s wr, 1.32k op/s 2024-04-25T11:40:41.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:41 smithi090 ceph-mon[29575]: pgmap v171: 97 pgs: 97 active+clean; 49 GiB data, 149 GiB used, 924 GiB / 1.0 TiB avail; 4.3 MiB/s rd, 725 MiB/s wr, 1.32k op/s 2024-04-25T11:40:42.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:41 smithi138 ceph-mon[32328]: pgmap v171: 97 pgs: 97 active+clean; 49 GiB data, 149 GiB used, 924 GiB / 1.0 TiB avail; 4.3 MiB/s rd, 725 MiB/s wr, 1.32k op/s 2024-04-25T11:40:43.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:43 smithi090 ceph-mon[29575]: pgmap v172: 97 pgs: 97 active+clean; 49 GiB data, 149 GiB used, 924 GiB / 1.0 TiB avail; 2.9 MiB/s rd, 573 MiB/s wr, 665 op/s 2024-04-25T11:40:44.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:43 smithi138 ceph-mon[32328]: pgmap v172: 97 pgs: 97 active+clean; 49 GiB data, 149 GiB used, 924 GiB / 1.0 TiB avail; 2.9 MiB/s rd, 573 MiB/s wr, 665 op/s 2024-04-25T11:40:44.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:43 smithi156 ceph-mon[33045]: pgmap v172: 97 pgs: 97 active+clean; 49 GiB data, 149 GiB used, 924 GiB / 1.0 TiB avail; 2.9 MiB/s rd, 573 MiB/s wr, 665 op/s 2024-04-25T11:40:45.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:45 smithi090 ceph-mon[29575]: pgmap v173: 97 pgs: 97 active+clean; 49 GiB data, 150 GiB used, 923 GiB / 1.0 TiB avail; 2.9 MiB/s rd, 602 MiB/s wr, 581 op/s 2024-04-25T11:40:46.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:45 smithi138 ceph-mon[32328]: pgmap v173: 97 pgs: 97 active+clean; 49 GiB data, 150 GiB used, 923 GiB / 1.0 TiB avail; 2.9 MiB/s rd, 602 MiB/s wr, 581 op/s 2024-04-25T11:40:46.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:45 smithi156 ceph-mon[33045]: pgmap v173: 97 pgs: 97 active+clean; 49 GiB data, 150 GiB used, 923 GiB / 1.0 TiB avail; 2.9 MiB/s rd, 602 MiB/s wr, 581 op/s 2024-04-25T11:40:48.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:47 smithi090 ceph-mon[29575]: pgmap v174: 97 pgs: 97 active+clean; 50 GiB data, 152 GiB used, 921 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 656 MiB/s wr, 693 op/s 2024-04-25T11:40:48.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:47 smithi138 ceph-mon[32328]: pgmap v174: 97 pgs: 97 active+clean; 50 GiB data, 152 GiB used, 921 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 656 MiB/s wr, 693 op/s 2024-04-25T11:40:48.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:47 smithi156 ceph-mon[33045]: pgmap v174: 97 pgs: 97 active+clean; 50 GiB data, 152 GiB used, 921 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 656 MiB/s wr, 693 op/s 2024-04-25T11:40:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.194+0000 7fe605634640 1 -- 172.21.15.90:0/1171600712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe600108a60 msgr2=0x7fe600102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.194+0000 7fe605634640 1 --2- 172.21.15.90:0/1171600712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe600108a60 0x7fe600102610 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0009900 tx=0x7fe5f002a410 comp rx=0 tx=0).stop 2024-04-25T11:40:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.194+0000 7fe605634640 1 -- 172.21.15.90:0/1171600712 shutdown_connections 2024-04-25T11:40:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.194+0000 7fe605634640 1 --2- 172.21.15.90:0/1171600712 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6001031c0 0x7fe60010f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.194+0000 7fe605634640 1 --2- 172.21.15.90:0/1171600712 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe600108f00 0x7fe600102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:49.183 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.194+0000 7fe605634640 1 --2- 172.21.15.90:0/1171600712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe600108a60 0x7fe600102610 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:49.183 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.194+0000 7fe605634640 1 -- 172.21.15.90:0/1171600712 >> 172.21.15.90:0/1171600712 conn(0x7fe6000781b0 msgr2=0x7fe6000785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:49.183 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.195+0000 7fe605634640 1 -- 172.21.15.90:0/1171600712 shutdown_connections 2024-04-25T11:40:49.183 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.195+0000 7fe605634640 1 -- 172.21.15.90:0/1171600712 wait complete. 2024-04-25T11:40:49.183 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.196+0000 7fe605634640 1 Processor -- start 2024-04-25T11:40:49.183 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.196+0000 7fe605634640 1 -- start start 2024-04-25T11:40:49.184 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.196+0000 7fe605634640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe6001031c0 0x7fe6001994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:49.184 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.196+0000 7fe605634640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe600108a60 0x7fe600199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:49.184 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.196+0000 7fe605634640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe600108f00 0x7fe600199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:49.185 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.197+0000 7fe605634640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe600112d10 con 0x7fe600108f00 2024-04-25T11:40:49.185 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.197+0000 7fe605634640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe600112b90 con 0x7fe6001031c0 2024-04-25T11:40:49.185 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.197+0000 7fe605634640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe600112e90 con 0x7fe600108a60 2024-04-25T11:40:49.185 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.197+0000 7fe5ff7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe600108f00 0x7fe600199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:49.185 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.197+0000 7fe5ff7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe600108f00 0x7fe600199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57366/0 (socket says 172.21.15.90:57366) 2024-04-25T11:40:49.185 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.197+0000 7fe5ff7fe640 1 -- 172.21.15.90:0/2449442404 learned_addr learned my addr 172.21.15.90:0/2449442404 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:49.185 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.197+0000 7fe5feffd640 1 --2- 172.21.15.90:0/2449442404 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe6001031c0 0x7fe6001994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:49.186 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.198+0000 7fe5ff7fe640 1 -- 172.21.15.90:0/2449442404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe600108a60 msgr2=0x7fe600199a20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:40:49.186 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.198+0000 7fe5ff7fe640 1 --2- 172.21.15.90:0/2449442404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe600108a60 0x7fe600199a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:49.186 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.198+0000 7fe5ff7fe640 1 -- 172.21.15.90:0/2449442404 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe6001031c0 msgr2=0x7fe6001994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:49.186 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.198+0000 7fe5ff7fe640 1 --2- 172.21.15.90:0/2449442404 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe6001031c0 0x7fe6001994e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:49.186 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.198+0000 7fe5ff7fe640 1 -- 172.21.15.90:0/2449442404 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6001a8b60 con 0x7fe600108f00 2024-04-25T11:40:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.198+0000 7fe5ff7fe640 1 --2- 172.21.15.90:0/2449442404 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe600108f00 0x7fe600199f60 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fe5f800cce0 tx=0x7fe5f800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.199+0000 7fe5fcff9640 1 -- 172.21.15.90:0/2449442404 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f800f450 con 0x7fe600108f00 2024-04-25T11:40:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.199+0000 7fe5fcff9640 1 -- 172.21.15.90:0/2449442404 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5f800faa0 con 0x7fe600108f00 2024-04-25T11:40:49.187 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.199+0000 7fe605634640 1 -- 172.21.15.90:0/2449442404 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6001a8df0 con 0x7fe600108f00 2024-04-25T11:40:49.188 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.199+0000 7fe5fcff9640 1 -- 172.21.15.90:0/2449442404 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f8018be0 con 0x7fe600108f00 2024-04-25T11:40:49.188 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.199+0000 7fe605634640 1 -- 172.21.15.90:0/2449442404 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6001a92e0 con 0x7fe600108f00 2024-04-25T11:40:49.191 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.204+0000 7fe60c5bf740 1 Processor -- start 2024-04-25T11:40:49.192 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.205+0000 7fe5fcff9640 1 -- 172.21.15.90:0/2449442404 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fe5f800f5f0 con 0x7fe600108f00 2024-04-25T11:40:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.205+0000 7fe60c5bf740 1 -- start start 2024-04-25T11:40:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.206+0000 7fe60c5bf740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240598b1b0 0x562405d2aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.206+0000 7fe60c5bf740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624059622f0 0x562405d2b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.206+0000 7fe60c5bf740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56240596a4d0 0x562405d2b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.206+0000 7fe60c5bf740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562405c91460 con 0x56240598b1b0 2024-04-25T11:40:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.206+0000 7fe60c5bf740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x562405c4b590 con 0x56240596a4d0 2024-04-25T11:40:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.206+0000 7fe60c5bf740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562405c8d420 con 0x5624059622f0 2024-04-25T11:40:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.206+0000 7fe5fcff9640 1 --2- 172.21.15.90:0/2449442404 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe5ec0afcd0 0x7fe5ec0b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.206+0000 7fe5f67fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240598b1b0 0x562405d2aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.206+0000 7fe5f67fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240598b1b0 0x562405d2aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57370/0 (socket says 172.21.15.90:57370) 2024-04-25T11:40:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.206+0000 7fe5f67fc640 1 -- 172.21.15.90:0/3526516110 learned_addr learned my addr 172.21.15.90:0/3526516110 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.207+0000 7fe5fcff9640 1 -- 172.21.15.90:0/2449442404 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fe5f80df4f0 con 0x7fe600108f00 2024-04-25T11:40:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.207+0000 7fe5ff7fe640 1 --2- 172.21.15.90:0/2449442404 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe5ec0afcd0 0x7fe5ec0b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.207+0000 7fe5f67fc640 1 -- 172.21.15.90:0/3526516110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624059622f0 msgr2=0x562405d2b030 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:40:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.207+0000 7fe5f67fc640 1 --2- 172.21.15.90:0/3526516110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624059622f0 0x562405d2b030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.207+0000 7fe5f67fc640 1 -- 172.21.15.90:0/3526516110 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56240596a4d0 msgr2=0x562405d2b570 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:40:49.197 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.207+0000 7fe5f67fc640 1 --2- 172.21.15.90:0/3526516110 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56240596a4d0 0x562405d2b570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:49.197 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.207+0000 7fe5f67fc640 1 -- 172.21.15.90:0/3526516110 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562405c580d0 con 0x56240598b1b0 2024-04-25T11:40:49.197 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.207+0000 7fe5f67fc640 1 --2- 172.21.15.90:0/3526516110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240598b1b0 0x562405d2aaf0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8009710 tx=0x7fe5e8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d9796ee80100939 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.207+0000 7fe5ff7fe640 1 --2- 172.21.15.90:0/2449442404 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe5ec0afcd0 0x7fe5ec0b21e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe5f800ccb0 tx=0x7fe5f800ff50 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.208+0000 7fe5f57fa640 1 -- 172.21.15.90:0/3526516110 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8015020 con 0x56240598b1b0 2024-04-25T11:40:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.208+0000 7fe5f57fa640 1 -- 172.21.15.90:0/3526516110 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5e8009e20 con 0x56240598b1b0 2024-04-25T11:40:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.208+0000 7fe5f57fa640 1 -- 172.21.15.90:0/3526516110 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8010380 con 0x56240598b1b0 2024-04-25T11:40:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.209+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3526516110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240598b1b0 msgr2=0x562405d2aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.209+0000 7fe60c5bf740 1 --2- 172.21.15.90:0/3526516110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240598b1b0 0x562405d2aaf0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8009710 tx=0x7fe5e8009bd0 comp rx=0 tx=0).stop 2024-04-25T11:40:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.209+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3526516110 shutdown_connections 2024-04-25T11:40:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.210+0000 7fe60c5bf740 1 --2- 172.21.15.90:0/3526516110 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56240596a4d0 0x562405d2b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.210+0000 7fe60c5bf740 1 --2- 172.21.15.90:0/3526516110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624059622f0 0x562405d2b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.210+0000 7fe60c5bf740 1 --2- 172.21.15.90:0/3526516110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240598b1b0 0x562405d2aaf0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.210+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3526516110 >> 172.21.15.90:0/3526516110 conn(0x56240597f300 msgr2=0x562405d272a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.210+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3526516110 shutdown_connections 2024-04-25T11:40:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.210+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3526516110 wait complete. 2024-04-25T11:40:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.211+0000 7fe60c5bf740 1 -- start start 2024-04-25T11:40:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.211+0000 7fe60c5bf740 1 Processor -- start 2024-04-25T11:40:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.212+0000 7fe60c5bf740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624059622f0 0x562405d35190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.212+0000 7fe5f67fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624059622f0 0x562405d35190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.212+0000 7fe5f67fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624059622f0 0x562405d35190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57372/0 (socket says 172.21.15.90:57372) 2024-04-25T11:40:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.212+0000 7fe5f67fc640 1 -- 172.21.15.90:0/3824612702 learned_addr learned my addr 172.21.15.90:0/3824612702 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.213+0000 7fe60c5bf740 1 --2- 172.21.15.90:0/3824612702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56240596a4d0 0x562405d356d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.213+0000 7fe60c5bf740 1 --2- 172.21.15.90:0/3824612702 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56240598b1b0 0x562405d3b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.213+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3824612702 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562405d37c20 con 0x5624059622f0 2024-04-25T11:40:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.213+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3824612702 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x562405c18b60 con 0x56240598b1b0 2024-04-25T11:40:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.213+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3824612702 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562405d37da0 con 0x56240596a4d0 2024-04-25T11:40:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.214+0000 7fe5f67fc640 1 -- 172.21.15.90:0/3824612702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56240596a4d0 msgr2=0x562405d356d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.214+0000 7fe5f67fc640 1 --2- 172.21.15.90:0/3824612702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56240596a4d0 0x562405d356d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.214+0000 7fe5f67fc640 1 -- 172.21.15.90:0/3824612702 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56240598b1b0 msgr2=0x562405d3b160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:40:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.214+0000 7fe5f67fc640 1 --2- 172.21.15.90:0/3824612702 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56240598b1b0 0x562405d3b160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.214+0000 7fe5f67fc640 1 -- 172.21.15.90:0/3824612702 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624059a3210 con 0x5624059622f0 2024-04-25T11:40:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.214+0000 7fe60c5bf740 10 client.0 _ll_register_callbacks cb 0x562405c3d2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.214+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3824612702 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5e8009e20 con 0x5624059622f0 2024-04-25T11:40:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.214+0000 7fe5f67fc640 1 --2- 172.21.15.90:0/3824612702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624059622f0 0x562405d35190 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8011b50 tx=0x7fe5e800bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c34f7bc2a151a68d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:49.204 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.214+0000 7fe5e67fc640 10 client.17253 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:40:49.204 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.215+0000 7fe5e67fc640 1 -- 172.21.15.90:0/3824612702 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8015030 con 0x5624059622f0 2024-04-25T11:40:49.204 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.215+0000 7fe5e67fc640 1 -- 172.21.15.90:0/3824612702 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fe5e8011420 con 0x5624059622f0 2024-04-25T11:40:49.204 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.215+0000 7fe5e67fc640 1 -- 172.21.15.90:0/3824612702 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe5e8012e90 con 0x5624059622f0 2024-04-25T11:40:49.204 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.215+0000 7fe60c5bf740 10 client.17253 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:49.204 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.215+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3824612702 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=28}) -- 0x562405a6f620 con 0x5624059622f0 2024-04-25T11:40:49.205 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.216+0000 7fe5e67fc640 1 -- 172.21.15.90:0/3824612702 <== mon.0 v2:172.21.15.90:3300/0 4 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fe5e8011680 con 0x5624059622f0 2024-04-25T11:40:49.205 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.216+0000 7fe60c5bf740 10 client.17253 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:49.205 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.216+0000 7fe60c5bf740 10 client.17253 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:40:49.206 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.216+0000 7fe60c5bf740 20 client.17253 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:49.206 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.216+0000 7fe60c5bf740 1 --2- 172.21.15.90:0/3824612702 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562405949d80 0x562405d50040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:49.206 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.216+0000 7fe60c5bf740 4 client.17253 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:40:49.206 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.216+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3824612702 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562405c9f070 con 0x562405949d80 2024-04-25T11:40:49.206 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.217+0000 7fe5feffd640 1 --2- 172.21.15.90:0/3824612702 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562405949d80 0x562405d50040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:49.206 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.219+0000 7fe5e67fc640 10 client.17253 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:49.206 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.219+0000 7fe5feffd640 1 --2- 172.21.15.90:0/3824612702 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562405949d80 0x562405d50040 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8347bc3fa83cc62 server_cookie=2e577504262b1bf in_seq=0 out_seq=0 2024-04-25T11:40:49.207 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.220+0000 7fe5e67fc640 1 -- 172.21.15.90:0/3824612702 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562405c9f070 con 0x562405949d80 2024-04-25T11:40:49.207 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.220+0000 7fe5e67fc640 10 client.17253 handle_command_reply: tid=0 2024-04-25T11:40:49.207 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.220+0000 7fe60c5bf740 1 client.17253 shutdown 2024-04-25T11:40:49.208 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.220+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3824612702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624059622f0 msgr2=0x562405d35190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:49.208 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.220+0000 7fe60c5bf740 1 --2- 172.21.15.90:0/3824612702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624059622f0 0x562405d35190 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8011b50 tx=0x7fe5e800bb00 comp rx=0 tx=0).stop 2024-04-25T11:40:49.209 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.221+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3824612702 shutdown_connections 2024-04-25T11:40:49.209 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.221+0000 7fe60c5bf740 1 --2- 172.21.15.90:0/3824612702 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562405949d80 0x562405d50040 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:49.209 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.221+0000 7fe60c5bf740 1 --2- 172.21.15.90:0/3824612702 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56240598b1b0 0x562405d3b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:49.209 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.221+0000 7fe5e67fc640 0 client.17253 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:49.209 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.221+0000 7fe60c5bf740 1 --2- 172.21.15.90:0/3824612702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56240596a4d0 0x562405d356d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:49.209 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.221+0000 7fe60c5bf740 1 --2- 172.21.15.90:0/3824612702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624059622f0 0x562405d35190 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:49.209 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.221+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3824612702 >> 172.21.15.90:0/3824612702 conn(0x56240597f300 msgr2=0x562405d2a920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:49.209 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.221+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3824612702 shutdown_connections 2024-04-25T11:40:49.209 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.221+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3824612702 wait complete. 2024-04-25T11:40:49.209 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.221+0000 7fe60c5bf740 20 client.17253 trim_cache size 0 max 16384 2024-04-25T11:40:49.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.240+0000 7fe60c5bf740 1 Processor -- start 2024-04-25T11:40:49.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.240+0000 7fe60c5bf740 1 -- start start 2024-04-25T11:40:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.241+0000 7fe60c5bf740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624059622f0 0x562405d3b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.241+0000 7fe60c5bf740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56240596a4d0 0x562405d2af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.241+0000 7fe60c5bf740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240598b1b0 0x562405d55840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.241+0000 7fe60c5bf740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe6001ada60 con 0x56240598b1b0 2024-04-25T11:40:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.241+0000 7fe60c5bf740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe6001af5a0 con 0x56240596a4d0 2024-04-25T11:40:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.241+0000 7fe60c5bf740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe6001ae130 con 0x5624059622f0 2024-04-25T11:40:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.241+0000 7fe5f67fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240598b1b0 0x562405d55840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.241+0000 7fe5f67fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240598b1b0 0x562405d55840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57380/0 (socket says 172.21.15.90:57380) 2024-04-25T11:40:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.241+0000 7fe5f67fc640 1 -- 172.21.15.90:0/3648812316 learned_addr learned my addr 172.21.15.90:0/3648812316 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.241+0000 7fe5feffd640 1 --2- 172.21.15.90:0/3648812316 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56240596a4d0 0x562405d2af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.241+0000 7fe5f67fc640 1 -- 172.21.15.90:0/3648812316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624059622f0 msgr2=0x562405d3b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.241+0000 7fe5f67fc640 1 --2- 172.21.15.90:0/3648812316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624059622f0 0x562405d3b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:49.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.241+0000 7fe5f67fc640 1 -- 172.21.15.90:0/3648812316 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56240596a4d0 msgr2=0x562405d2af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:49.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.241+0000 7fe5f67fc640 1 --2- 172.21.15.90:0/3648812316 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56240596a4d0 0x562405d2af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:49.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.241+0000 7fe5f67fc640 1 -- 172.21.15.90:0/3648812316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562405d2b8a0 con 0x56240598b1b0 2024-04-25T11:40:49.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.241+0000 7fe5f67fc640 1 --2- 172.21.15.90:0/3648812316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240598b1b0 0x562405d55840 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800bb40 tx=0x7fe5e801c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68f16f7ae1d7de72 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:49.233 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.242+0000 7fe5f4ff9640 1 -- 172.21.15.90:0/3648812316 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8015070 con 0x56240598b1b0 2024-04-25T11:40:49.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.242+0000 7fe5f4ff9640 1 -- 172.21.15.90:0/3648812316 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5e8032020 con 0x56240598b1b0 2024-04-25T11:40:49.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.242+0000 7fe5f4ff9640 1 -- 172.21.15.90:0/3648812316 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8003690 con 0x56240598b1b0 2024-04-25T11:40:49.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.244+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3648812316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240598b1b0 msgr2=0x562405d55840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:49.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.244+0000 7fe60c5bf740 1 --2- 172.21.15.90:0/3648812316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240598b1b0 0x562405d55840 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800bb40 tx=0x7fe5e801c040 comp rx=0 tx=0).stop 2024-04-25T11:40:49.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.244+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3648812316 shutdown_connections 2024-04-25T11:40:49.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.244+0000 7fe60c5bf740 1 --2- 172.21.15.90:0/3648812316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56240598b1b0 0x562405d55840 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:49.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.244+0000 7fe60c5bf740 1 --2- 172.21.15.90:0/3648812316 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56240596a4d0 0x562405d2af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:49.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.244+0000 7fe60c5bf740 1 --2- 172.21.15.90:0/3648812316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624059622f0 0x562405d3b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:49.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.244+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3648812316 >> 172.21.15.90:0/3648812316 conn(0x56240597f300 msgr2=0x562405d2a880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:49.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.244+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3648812316 shutdown_connections 2024-04-25T11:40:49.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.244+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3648812316 wait complete. 2024-04-25T11:40:49.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.245+0000 7fe60c5bf740 1 -- start start 2024-04-25T11:40:49.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.245+0000 7fe60c5bf740 1 Processor -- start 2024-04-25T11:40:49.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.245+0000 7fe60c5bf740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624059622f0 0x562405d41a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:49.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.246+0000 7fe5f67fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624059622f0 0x562405d41a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:49.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.246+0000 7fe5f67fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624059622f0 0x562405d41a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57386/0 (socket says 172.21.15.90:57386) 2024-04-25T11:40:49.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.246+0000 7fe60c5bf740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56240596a4d0 0x562405d3dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:49.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.246+0000 7fe5f67fc640 1 -- 172.21.15.90:0/3421202429 learned_addr learned my addr 172.21.15.90:0/3421202429 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:49.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.246+0000 7fe60c5bf740 1 --2- 172.21.15.90:0/3421202429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56240598b1b0 0x562405d3e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:49.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.246+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3421202429 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe6001aeed0 con 0x5624059622f0 2024-04-25T11:40:49.236 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.246+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3421202429 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe6001ae800 con 0x56240596a4d0 2024-04-25T11:40:49.236 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.246+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3421202429 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe6001aba40 con 0x56240598b1b0 2024-04-25T11:40:49.236 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.246+0000 7fe60c5bf740 10 client.0 _ll_register_callbacks cb 0x7fe5e8012d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:49.236 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.246+0000 7fe5feffd640 1 --2- 172.21.15.90:0/3421202429 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56240596a4d0 0x562405d3dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:49.236 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.246+0000 7fe5f67fc640 1 -- 172.21.15.90:0/3421202429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56240598b1b0 msgr2=0x562405d3e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:49.236 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.246+0000 7fe5f67fc640 1 --2- 172.21.15.90:0/3421202429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56240598b1b0 0x562405d3e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:49.236 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.246+0000 7fe5f67fc640 1 -- 172.21.15.90:0/3421202429 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56240596a4d0 msgr2=0x562405d3dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:49.236 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.246+0000 7fe5f67fc640 1 --2- 172.21.15.90:0/3421202429 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56240596a4d0 0x562405d3dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:49.236 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.246+0000 7fe5f67fc640 1 -- 172.21.15.90:0/3421202429 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562405d3e7e0 con 0x5624059622f0 2024-04-25T11:40:49.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.247+0000 7fe5f67fc640 1 --2- 172.21.15.90:0/3421202429 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624059622f0 0x562405d41a00 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8004170 tx=0x7fe5e8009090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37b681fa7f47cd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:49.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.247+0000 7fe5e67fc640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:40:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.247+0000 7fe5e67fc640 1 -- 172.21.15.90:0/3421202429 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8015040 con 0x5624059622f0 2024-04-25T11:40:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.247+0000 7fe5e67fc640 1 -- 172.21.15.90:0/3421202429 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fe5e8004630 con 0x5624059622f0 2024-04-25T11:40:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.247+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3421202429 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5e8032020 con 0x5624059622f0 2024-04-25T11:40:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.248+0000 7fe5e67fc640 1 -- 172.21.15.90:0/3421202429 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe5e80093b0 con 0x5624059622f0 2024-04-25T11:40:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.248+0000 7fe60c5bf740 10 client.17259 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.248+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3421202429 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562405db70b0 con 0x5624059622f0 2024-04-25T11:40:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.248+0000 7fe5e67fc640 1 -- 172.21.15.90:0/3421202429 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5e8009e20 con 0x5624059622f0 2024-04-25T11:40:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.248+0000 7fe5e67fc640 1 -- 172.21.15.90:0/3421202429 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fe5e802c6c0 con 0x5624059622f0 2024-04-25T11:40:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.248+0000 7fe60c5bf740 10 client.17259 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.248+0000 7fe60c5bf740 10 client.17259 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:40:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.248+0000 7fe60c5bf740 20 client.17259 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.248+0000 7fe60c5bf740 1 --2- 172.21.15.90:0/3421202429 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56240591f1c0 0x562405db9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.248+0000 7fe60c5bf740 4 client.17259 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:40:49.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.248+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3421202429 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fe5e8032020 con 0x56240591f1c0 2024-04-25T11:40:49.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.250+0000 7fe5feffd640 1 --2- 172.21.15.90:0/3421202429 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56240591f1c0 0x562405db9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:49.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.251+0000 7fe5e67fc640 10 client.17259 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:49.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:49.251+0000 7fe5feffd640 1 --2- 172.21.15.90:0/3421202429 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56240591f1c0 0x562405db9790 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12b8078b150ea408 server_cookie=7ac4a9be9fb79463 in_seq=0 out_seq=0 2024-04-25T11:40:50.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:50 smithi138 ceph-mon[32328]: pgmap v175: 97 pgs: 97 active+clean; 50 GiB data, 152 GiB used, 921 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 478 MiB/s wr, 588 op/s 2024-04-25T11:40:50.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:50 smithi156 ceph-mon[33045]: pgmap v175: 97 pgs: 97 active+clean; 50 GiB data, 152 GiB used, 921 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 478 MiB/s wr, 588 op/s 2024-04-25T11:40:50.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:50 smithi090 ceph-mon[29575]: pgmap v175: 97 pgs: 97 active+clean; 50 GiB data, 152 GiB used, 921 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 478 MiB/s wr, 588 op/s 2024-04-25T11:40:51.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:40:51.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:40:51.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:40:51.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:40:51.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.663+0000 7fe5e67fc640 1 -- 172.21.15.90:0/3421202429 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fe5e8032020 con 0x56240591f1c0 2024-04-25T11:40:51.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.663+0000 7fe5e67fc640 10 client.17259 handle_command_reply: tid=0 2024-04-25T11:40:51.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.663+0000 7fe60c5bf740 1 client.17259 shutdown 2024-04-25T11:40:51.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.663+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3421202429 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624059622f0 msgr2=0x562405d41a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:51.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.663+0000 7fe60c5bf740 1 --2- 172.21.15.90:0/3421202429 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624059622f0 0x562405d41a00 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8004170 tx=0x7fe5e8009090 comp rx=0 tx=0).stop 2024-04-25T11:40:51.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.663+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3421202429 shutdown_connections 2024-04-25T11:40:51.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.663+0000 7fe60c5bf740 1 --2- 172.21.15.90:0/3421202429 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56240591f1c0 0x562405db9790 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:51.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.663+0000 7fe60c5bf740 1 --2- 172.21.15.90:0/3421202429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56240598b1b0 0x562405d3e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:51.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.663+0000 7fe60c5bf740 1 --2- 172.21.15.90:0/3421202429 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56240596a4d0 0x562405d3dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:51.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.663+0000 7fe60c5bf740 1 --2- 172.21.15.90:0/3421202429 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624059622f0 0x562405d41a00 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:51.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.663+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3421202429 >> 172.21.15.90:0/3421202429 conn(0x56240597f300 msgr2=0x562405d36440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:51.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.663+0000 7fe5e67fc640 0 client.17259 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:51.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.664+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3421202429 shutdown_connections 2024-04-25T11:40:51.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.664+0000 7fe60c5bf740 1 -- 172.21.15.90:0/3421202429 wait complete. 2024-04-25T11:40:51.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.664+0000 7fe60c5bf740 20 client.17259 trim_cache size 0 max 16384 2024-04-25T11:40:51.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.672+0000 7fe605634640 1 -- 172.21.15.90:0/2449442404 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe5ec0afcd0 msgr2=0x7fe5ec0b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:51.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.672+0000 7fe605634640 1 --2- 172.21.15.90:0/2449442404 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe5ec0afcd0 0x7fe5ec0b21e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe5f800ccb0 tx=0x7fe5f800ff50 comp rx=0 tx=0).stop 2024-04-25T11:40:51.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.672+0000 7fe605634640 1 -- 172.21.15.90:0/2449442404 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe600108f00 msgr2=0x7fe600199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:51.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.672+0000 7fe605634640 1 --2- 172.21.15.90:0/2449442404 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe600108f00 0x7fe600199f60 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fe5f800cce0 tx=0x7fe5f800a620 comp rx=0 tx=0).stop 2024-04-25T11:40:51.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.672+0000 7fe605634640 1 -- 172.21.15.90:0/2449442404 shutdown_connections 2024-04-25T11:40:51.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.672+0000 7fe605634640 1 --2- 172.21.15.90:0/2449442404 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe5ec0afcd0 0x7fe5ec0b21e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:51.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.672+0000 7fe605634640 1 --2- 172.21.15.90:0/2449442404 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe600108f00 0x7fe600199f60 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:51.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.672+0000 7fe605634640 1 --2- 172.21.15.90:0/2449442404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe600108a60 0x7fe600199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:51.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.672+0000 7fe605634640 1 --2- 172.21.15.90:0/2449442404 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe6001031c0 0x7fe6001994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:51.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.672+0000 7fe605634640 1 -- 172.21.15.90:0/2449442404 >> 172.21.15.90:0/2449442404 conn(0x7fe6000781b0 msgr2=0x7fe600104140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:51.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.672+0000 7fe605634640 1 -- 172.21.15.90:0/2449442404 shutdown_connections 2024-04-25T11:40:51.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:40:51.672+0000 7fe605634640 1 -- 172.21.15.90:0/2449442404 wait complete. 2024-04-25T11:40:52.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:52 smithi156 ceph-mon[33045]: pgmap v176: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 3.3 MiB/s rd, 669 MiB/s wr, 761 op/s 2024-04-25T11:40:52.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:52 smithi090 ceph-mon[29575]: pgmap v176: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 3.3 MiB/s rd, 669 MiB/s wr, 761 op/s 2024-04-25T11:40:52.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:52 smithi138 ceph-mon[32328]: pgmap v176: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 3.3 MiB/s rd, 669 MiB/s wr, 761 op/s 2024-04-25T11:40:53.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:53 smithi138 ceph-mon[32328]: pgmap v177: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 533 MiB/s wr, 577 op/s 2024-04-25T11:40:53.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:53 smithi156 ceph-mon[33045]: pgmap v177: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 533 MiB/s wr, 577 op/s 2024-04-25T11:40:53.723 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:53 smithi090 ceph-mon[29575]: pgmap v177: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 533 MiB/s wr, 577 op/s 2024-04-25T11:40:55.723 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:55 smithi090 ceph-mon[29575]: pgmap v178: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 561 MiB/s wr, 592 op/s 2024-04-25T11:40:55.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:55 smithi138 ceph-mon[32328]: pgmap v178: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 561 MiB/s wr, 592 op/s 2024-04-25T11:40:55.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:55 smithi156 ceph-mon[33045]: pgmap v178: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 561 MiB/s wr, 592 op/s 2024-04-25T11:40:57.090 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:40:57.091 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:40:57.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.228+0000 7fa79c346640 1 -- 172.21.15.90:0/2490711358 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa794108ec0 msgr2=0x7fa794102b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:57.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.228+0000 7fa79c346640 1 --2- 172.21.15.90:0/2490711358 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa794108ec0 0x7fa794102b10 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fa784009900 tx=0x7fa78402a410 comp rx=0 tx=0).stop 2024-04-25T11:40:57.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.229+0000 7fa79c346640 1 -- 172.21.15.90:0/2490711358 shutdown_connections 2024-04-25T11:40:57.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.229+0000 7fa79c346640 1 --2- 172.21.15.90:0/2490711358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa794103180 0x7fa79410f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.229+0000 7fa79c346640 1 --2- 172.21.15.90:0/2490711358 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa794108ec0 0x7fa794102b10 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.229+0000 7fa79c346640 1 --2- 172.21.15.90:0/2490711358 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa794108a20 0x7fa7941025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.229+0000 7fa79c346640 1 -- 172.21.15.90:0/2490711358 >> 172.21.15.90:0/2490711358 conn(0x7fa794078110 msgr2=0x7fa794078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:57.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.229+0000 7fa79c346640 1 -- 172.21.15.90:0/2490711358 shutdown_connections 2024-04-25T11:40:57.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.229+0000 7fa79c346640 1 -- 172.21.15.90:0/2490711358 wait complete. 2024-04-25T11:40:57.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.229+0000 7fa79c346640 1 Processor -- start 2024-04-25T11:40:57.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.229+0000 7fa79c346640 1 -- start start 2024-04-25T11:40:57.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.229+0000 7fa79c346640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa794103180 0x7fa79419b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:57.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.229+0000 7fa79c346640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa794108a20 0x7fa79419bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:57.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.229+0000 7fa79c346640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa794108ec0 0x7fa79419c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:57.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.229+0000 7fa79c346640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa794112c20 con 0x7fa794108ec0 2024-04-25T11:40:57.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.229+0000 7fa79c346640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa794112aa0 con 0x7fa794103180 2024-04-25T11:40:57.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.229+0000 7fa79c346640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa794112da0 con 0x7fa794108a20 2024-04-25T11:40:57.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.230+0000 7fa79a8e3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa794108ec0 0x7fa79419c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:57.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.230+0000 7fa79a0e2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa794103180 0x7fa79419b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:57.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.230+0000 7fa79a8e3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa794108ec0 0x7fa79419c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46050/0 (socket says 172.21.15.90:46050) 2024-04-25T11:40:57.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.230+0000 7fa79a8e3640 1 -- 172.21.15.90:0/3533057507 learned_addr learned my addr 172.21.15.90:0/3533057507 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:57.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.230+0000 7fa7998e1640 1 --2- 172.21.15.90:0/3533057507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa794108a20 0x7fa79419bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:57.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.230+0000 7fa79a0e2640 1 -- 172.21.15.90:0/3533057507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa794108a20 msgr2=0x7fa79419bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:57.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.230+0000 7fa79a0e2640 1 --2- 172.21.15.90:0/3533057507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa794108a20 0x7fa79419bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.230+0000 7fa79a0e2640 1 -- 172.21.15.90:0/3533057507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa794108ec0 msgr2=0x7fa79419c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:57.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.230+0000 7fa79a0e2640 1 --2- 172.21.15.90:0/3533057507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa794108ec0 0x7fa79419c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.230+0000 7fa79a0e2640 1 -- 172.21.15.90:0/3533057507 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7941a6850 con 0x7fa794103180 2024-04-25T11:40:57.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.230+0000 7fa79a0e2640 1 --2- 172.21.15.90:0/3533057507 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa794103180 0x7fa79419b5a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa794103710 tx=0x7fa79000cdd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:57.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.230+0000 7fa78b7fe640 1 -- 172.21.15.90:0/3533057507 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa790007bb0 con 0x7fa794103180 2024-04-25T11:40:57.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.230+0000 7fa79c346640 1 -- 172.21.15.90:0/3533057507 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7941a6b40 con 0x7fa794103180 2024-04-25T11:40:57.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.230+0000 7fa78b7fe640 1 -- 172.21.15.90:0/3533057507 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa790007d50 con 0x7fa794103180 2024-04-25T11:40:57.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.230+0000 7fa78b7fe640 1 -- 172.21.15.90:0/3533057507 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7900186f0 con 0x7fa794103180 2024-04-25T11:40:57.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.230+0000 7fa79c346640 1 -- 172.21.15.90:0/3533057507 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7941a7060 con 0x7fa794103180 2024-04-25T11:40:57.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.233+0000 7fa78b7fe640 1 -- 172.21.15.90:0/3533057507 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fa79000f450 con 0x7fa794103180 2024-04-25T11:40:57.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.233+0000 7fa7a31f9740 1 Processor -- start 2024-04-25T11:40:57.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.233+0000 7fa7a31f9740 1 -- start start 2024-04-25T11:40:57.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.234+0000 7fa7a31f9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9a3351390 0x55e9a36f1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:57.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.234+0000 7fa7a31f9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9a33284d0 0x55e9a36f1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:57.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.234+0000 7fa7a31f9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9a33306b0 0x55e9a36fe8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:57.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.234+0000 7fa7a31f9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e9a3657d80 con 0x55e9a33306b0 2024-04-25T11:40:57.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.234+0000 7fa7a31f9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55e9a36118b0 con 0x55e9a33284d0 2024-04-25T11:40:57.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.234+0000 7fa7a31f9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e9a3653d40 con 0x55e9a3351390 2024-04-25T11:40:57.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.234+0000 7fa79a0e2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9a33284d0 0x55e9a36f1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:57.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.234+0000 7fa7998e1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9a33306b0 0x55e9a36fe8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:57.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.234+0000 7fa79a0e2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9a33284d0 0x55e9a36f1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:40836/0 (socket says 172.21.15.90:40836) 2024-04-25T11:40:57.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.234+0000 7fa7998e1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9a33306b0 0x55e9a36fe8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46058/0 (socket says 172.21.15.90:46058) 2024-04-25T11:40:57.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.234+0000 7fa79a0e2640 1 -- 172.21.15.90:0/1699991293 learned_addr learned my addr 172.21.15.90:0/1699991293 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:57.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.234+0000 7fa79a8e3640 1 --2- 172.21.15.90:0/1699991293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9a3351390 0x55e9a36f1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:57.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.234+0000 7fa78b7fe640 1 --2- 172.21.15.90:0/3533057507 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa7800afc60 0x7fa7800b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:57.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.234+0000 7fa79a0e2640 1 -- 172.21.15.90:0/1699991293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9a3351390 msgr2=0x55e9a36f1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:57.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.234+0000 7fa79a0e2640 1 --2- 172.21.15.90:0/1699991293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9a3351390 0x55e9a36f1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.234+0000 7fa79a0e2640 1 -- 172.21.15.90:0/1699991293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9a33306b0 msgr2=0x55e9a36fe8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:57.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.234+0000 7fa79a0e2640 1 --2- 172.21.15.90:0/1699991293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9a33306b0 0x55e9a36fe8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.234+0000 7fa79a0e2640 1 -- 172.21.15.90:0/1699991293 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9a35e2a40 con 0x55e9a33284d0 2024-04-25T11:40:57.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.234+0000 7fa79a8e3640 1 --2- 172.21.15.90:0/3533057507 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa7800afc60 0x7fa7800b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:57.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.235+0000 7fa78b7fe640 1 -- 172.21.15.90:0/3533057507 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fa7900df080 con 0x7fa794103180 2024-04-25T11:40:57.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.235+0000 7fa79a0e2640 1 --2- 172.21.15.90:0/1699991293 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9a33284d0 0x55e9a36f1fa0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa7900dc6e0 tx=0x7fa79009bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ada0050892d78d3e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:57.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.235+0000 7fa788bfe640 1 -- 172.21.15.90:0/1699991293 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7900a6f90 con 0x55e9a33284d0 2024-04-25T11:40:57.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.235+0000 7fa788bfe640 1 -- 172.21.15.90:0/1699991293 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7900a6cd0 con 0x55e9a33284d0 2024-04-25T11:40:57.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.235+0000 7fa79a8e3640 1 --2- 172.21.15.90:0/3533057507 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa7800afc60 0x7fa7800b2170 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x55e9a36eebd0 tx=0x7fa78c004a40 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:57.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.235+0000 7fa788bfe640 1 -- 172.21.15.90:0/1699991293 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7900a6930 con 0x55e9a33284d0 2024-04-25T11:40:57.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.236+0000 7fa7a31f9740 1 -- 172.21.15.90:0/1699991293 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9a33284d0 msgr2=0x55e9a36f1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:57.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.236+0000 7fa7a31f9740 1 --2- 172.21.15.90:0/1699991293 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9a33284d0 0x55e9a36f1fa0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa7900dc6e0 tx=0x7fa79009bab0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.236+0000 7fa7a31f9740 1 -- 172.21.15.90:0/1699991293 shutdown_connections 2024-04-25T11:40:57.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.236+0000 7fa7a31f9740 1 --2- 172.21.15.90:0/1699991293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9a33306b0 0x55e9a36fe8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.236+0000 7fa7a31f9740 1 --2- 172.21.15.90:0/1699991293 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9a33284d0 0x55e9a36f1fa0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.236+0000 7fa7a31f9740 1 --2- 172.21.15.90:0/1699991293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9a3351390 0x55e9a36f1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.236+0000 7fa7a31f9740 1 -- 172.21.15.90:0/1699991293 >> 172.21.15.90:0/1699991293 conn(0x55e9a33454e0 msgr2=0x55e9a36ee210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:57.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.236+0000 7fa7a31f9740 1 -- 172.21.15.90:0/1699991293 shutdown_connections 2024-04-25T11:40:57.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.236+0000 7fa7a31f9740 1 -- 172.21.15.90:0/1699991293 wait complete. 2024-04-25T11:40:57.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.236+0000 7fa7a31f9740 1 -- start start 2024-04-25T11:40:57.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.236+0000 7fa7a31f9740 1 Processor -- start 2024-04-25T11:40:57.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.237+0000 7fa7a31f9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9a33284d0 0x55e9a36fc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:57.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.237+0000 7fa7a31f9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9a33306b0 0x55e9a36fcb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:57.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.237+0000 7fa7a31f9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9a3351390 0x55e9a3702440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:57.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.237+0000 7fa7a31f9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e9a36fef80 con 0x55e9a3351390 2024-04-25T11:40:57.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.237+0000 7fa7a31f9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55e9a35ded10 con 0x55e9a33284d0 2024-04-25T11:40:57.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.237+0000 7fa7a31f9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e9a36ff100 con 0x55e9a33306b0 2024-04-25T11:40:57.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.237+0000 7fa7a31f9740 10 client.0 _ll_register_callbacks cb 0x55e9a360b210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:57.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.237+0000 7fa79a8e3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9a3351390 0x55e9a3702440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:57.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.237+0000 7fa79a8e3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9a3351390 0x55e9a3702440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46070/0 (socket says 172.21.15.90:46070) 2024-04-25T11:40:57.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.237+0000 7fa79a8e3640 1 -- 172.21.15.90:0/2501400751 learned_addr learned my addr 172.21.15.90:0/2501400751 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:57.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.237+0000 7fa7998e1640 1 --2- 172.21.15.90:0/2501400751 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9a33306b0 0x55e9a36fcb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:57.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.237+0000 7fa79a0e2640 1 --2- 172.21.15.90:0/2501400751 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9a33284d0 0x55e9a36fc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:57.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.237+0000 7fa7998e1640 1 -- 172.21.15.90:0/2501400751 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9a33284d0 msgr2=0x55e9a36fc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:57.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.237+0000 7fa7998e1640 1 --2- 172.21.15.90:0/2501400751 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9a33284d0 0x55e9a36fc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.237+0000 7fa7998e1640 1 -- 172.21.15.90:0/2501400751 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9a3351390 msgr2=0x55e9a3702440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:57.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.237+0000 7fa7998e1640 1 --2- 172.21.15.90:0/2501400751 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9a3351390 0x55e9a3702440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.237+0000 7fa7998e1640 1 -- 172.21.15.90:0/2501400751 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9a34337d0 con 0x55e9a33306b0 2024-04-25T11:40:57.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.237+0000 7fa79a8e3640 1 --2- 172.21.15.90:0/2501400751 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9a3351390 0x55e9a3702440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:40:57.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.237+0000 7fa7998e1640 1 --2- 172.21.15.90:0/2501400751 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9a33306b0 0x55e9a36fcb80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa78402a930 tx=0x7fa784031740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=facf394d0a3ccaa9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:57.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.237+0000 7fa77dffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:40:57.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.238+0000 7fa77dffb640 1 -- 172.21.15.90:0/2501400751 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa784035040 con 0x55e9a33306b0 2024-04-25T11:40:57.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.238+0000 7fa7a31f9740 1 -- 172.21.15.90:0/2501400751 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7900a6cd0 con 0x55e9a33306b0 2024-04-25T11:40:57.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.238+0000 7fa77dffb640 1 -- 172.21.15.90:0/2501400751 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fa784004240 con 0x55e9a33306b0 2024-04-25T11:40:57.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.238+0000 7fa77dffb640 1 -- 172.21.15.90:0/2501400751 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa784004440 con 0x55e9a33306b0 2024-04-25T11:40:57.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.238+0000 7fa7a31f9740 10 client.26293 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:57.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.238+0000 7fa7a31f9740 1 -- 172.21.15.90:0/2501400751 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e9a33693f0 con 0x55e9a33306b0 2024-04-25T11:40:57.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.238+0000 7fa77dffb640 1 -- 172.21.15.90:0/2501400751 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7900a6cd0 con 0x55e9a33306b0 2024-04-25T11:40:57.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.239+0000 7fa77dffb640 1 -- 172.21.15.90:0/2501400751 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fa784004ad0 con 0x55e9a33306b0 2024-04-25T11:40:57.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.239+0000 7fa7a31f9740 10 client.26293 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:57.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.239+0000 7fa7a31f9740 10 client.26293 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:40:57.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.239+0000 7fa7a31f9740 20 client.26293 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:57.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.239+0000 7fa7a31f9740 1 --2- 172.21.15.90:0/2501400751 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e9a330ff60 0x55e9a370e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:57.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.239+0000 7fa7a31f9740 4 client.26293 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:40:57.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.239+0000 7fa7a31f9740 1 -- 172.21.15.90:0/2501400751 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9a3665990 con 0x55e9a330ff60 2024-04-25T11:40:57.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.239+0000 7fa79a0e2640 1 --2- 172.21.15.90:0/2501400751 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e9a330ff60 0x55e9a370e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:57.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.239+0000 7fa77dffb640 10 client.26293 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:57.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.240+0000 7fa79a0e2640 1 --2- 172.21.15.90:0/2501400751 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e9a330ff60 0x55e9a370e6a0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1ff241c433e6da2 server_cookie=f8d81142ff4e17c1 in_seq=0 out_seq=0 2024-04-25T11:40:57.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.241+0000 7fa77dffb640 1 -- 172.21.15.90:0/2501400751 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e9a3665990 con 0x55e9a330ff60 2024-04-25T11:40:57.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.241+0000 7fa77dffb640 10 client.26293 handle_command_reply: tid=0 2024-04-25T11:40:57.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.241+0000 7fa7a31f9740 1 client.26293 shutdown 2024-04-25T11:40:57.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.241+0000 7fa7a31f9740 1 -- 172.21.15.90:0/2501400751 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9a33306b0 msgr2=0x55e9a36fcb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:57.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.241+0000 7fa7a31f9740 1 --2- 172.21.15.90:0/2501400751 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9a33306b0 0x55e9a36fcb80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa78402a930 tx=0x7fa784031740 comp rx=0 tx=0).stop 2024-04-25T11:40:57.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.241+0000 7fa7a31f9740 1 -- 172.21.15.90:0/2501400751 shutdown_connections 2024-04-25T11:40:57.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.241+0000 7fa7a31f9740 1 --2- 172.21.15.90:0/2501400751 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e9a330ff60 0x55e9a370e6a0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.241+0000 7fa7a31f9740 1 --2- 172.21.15.90:0/2501400751 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9a3351390 0x55e9a3702440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.241+0000 7fa77dffb640 0 client.26293 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:57.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.241+0000 7fa7a31f9740 1 --2- 172.21.15.90:0/2501400751 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9a33306b0 0x55e9a36fcb80 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.241+0000 7fa7a31f9740 1 --2- 172.21.15.90:0/2501400751 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9a33284d0 0x55e9a36fc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.241+0000 7fa7a31f9740 1 -- 172.21.15.90:0/2501400751 >> 172.21.15.90:0/2501400751 conn(0x55e9a33454e0 msgr2=0x55e9a36f3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:57.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.241+0000 7fa7a31f9740 1 -- 172.21.15.90:0/2501400751 shutdown_connections 2024-04-25T11:40:57.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.241+0000 7fa7a31f9740 1 -- 172.21.15.90:0/2501400751 wait complete. 2024-04-25T11:40:57.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.241+0000 7fa7a31f9740 20 client.26293 trim_cache size 0 max 16384 2024-04-25T11:40:57.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.261+0000 7fa7a31f9740 1 Processor -- start 2024-04-25T11:40:57.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.261+0000 7fa7a31f9740 1 -- start start 2024-04-25T11:40:57.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.261+0000 7fa7a31f9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9a330ff60 0x55e9a36fb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:57.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.261+0000 7fa7a31f9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9a33284d0 0x55e9a37023c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:57.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.261+0000 7fa7a31f9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9a33306b0 0x55e9a3702900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:57.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.261+0000 7fa7a31f9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa7941afc30 con 0x55e9a33284d0 2024-04-25T11:40:57.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.261+0000 7fa7a31f9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa7941b1770 con 0x55e9a330ff60 2024-04-25T11:40:57.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.261+0000 7fa7a31f9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa7941b0300 con 0x55e9a33306b0 2024-04-25T11:40:57.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.261+0000 7fa79a0e2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9a330ff60 0x55e9a36fb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:57.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.261+0000 7fa79a0e2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9a330ff60 0x55e9a36fb410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:40856/0 (socket says 172.21.15.90:40856) 2024-04-25T11:40:57.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.261+0000 7fa79a0e2640 1 -- 172.21.15.90:0/1808610439 learned_addr learned my addr 172.21.15.90:0/1808610439 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:57.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.261+0000 7fa7998e1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9a33284d0 0x55e9a37023c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:57.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.261+0000 7fa79a8e3640 1 --2- 172.21.15.90:0/1808610439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9a33306b0 0x55e9a3702900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:57.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.262+0000 7fa79a0e2640 1 -- 172.21.15.90:0/1808610439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9a33306b0 msgr2=0x55e9a3702900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:57.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.262+0000 7fa79a0e2640 1 --2- 172.21.15.90:0/1808610439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9a33306b0 0x55e9a3702900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.262+0000 7fa79a0e2640 1 -- 172.21.15.90:0/1808610439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9a33284d0 msgr2=0x55e9a37023c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:57.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.262+0000 7fa79a0e2640 1 --2- 172.21.15.90:0/1808610439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9a33284d0 0x55e9a37023c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.262+0000 7fa79a0e2640 1 -- 172.21.15.90:0/1808610439 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9a33290f0 con 0x55e9a330ff60 2024-04-25T11:40:57.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.262+0000 7fa79a8e3640 1 --2- 172.21.15.90:0/1808610439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9a33306b0 0x55e9a3702900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:40:57.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.262+0000 7fa79a0e2640 1 --2- 172.21.15.90:0/1808610439 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9a330ff60 0x55e9a36fb410 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa790007590 tx=0x7fa790007f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dccff7e6400aae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:57.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.262+0000 7fa77ffff640 1 -- 172.21.15.90:0/1808610439 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7900a6b10 con 0x55e9a330ff60 2024-04-25T11:40:57.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.262+0000 7fa77ffff640 1 -- 172.21.15.90:0/1808610439 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7900a6cd0 con 0x55e9a330ff60 2024-04-25T11:40:57.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.262+0000 7fa77ffff640 1 -- 172.21.15.90:0/1808610439 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7900b0020 con 0x55e9a330ff60 2024-04-25T11:40:57.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.263+0000 7fa7a31f9740 1 -- 172.21.15.90:0/1808610439 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9a330ff60 msgr2=0x55e9a36fb410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:57.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.263+0000 7fa7a31f9740 1 --2- 172.21.15.90:0/1808610439 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9a330ff60 0x55e9a36fb410 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa790007590 tx=0x7fa790007f00 comp rx=0 tx=0).stop 2024-04-25T11:40:57.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.263+0000 7fa7a31f9740 1 -- 172.21.15.90:0/1808610439 shutdown_connections 2024-04-25T11:40:57.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.263+0000 7fa7a31f9740 1 --2- 172.21.15.90:0/1808610439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9a33306b0 0x55e9a3702900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.263+0000 7fa7a31f9740 1 --2- 172.21.15.90:0/1808610439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9a33284d0 0x55e9a37023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.263+0000 7fa7a31f9740 1 --2- 172.21.15.90:0/1808610439 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9a330ff60 0x55e9a36fb410 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.263+0000 7fa7a31f9740 1 -- 172.21.15.90:0/1808610439 >> 172.21.15.90:0/1808610439 conn(0x55e9a33454e0 msgr2=0x55e9a36fad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:57.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.263+0000 7fa7a31f9740 1 -- 172.21.15.90:0/1808610439 shutdown_connections 2024-04-25T11:40:57.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.263+0000 7fa7a31f9740 1 -- 172.21.15.90:0/1808610439 wait complete. 2024-04-25T11:40:57.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.264+0000 7fa7a31f9740 1 -- start start 2024-04-25T11:40:57.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.264+0000 7fa7a31f9740 1 Processor -- start 2024-04-25T11:40:57.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.264+0000 7fa7a31f9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9a33284d0 0x55e9a3703c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:57.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.264+0000 7fa7a31f9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9a33306b0 0x55e9a37041b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:57.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.264+0000 7fa7a31f9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9a3351390 0x55e9a3774a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:57.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.264+0000 7fa7a31f9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa7941b10a0 con 0x55e9a3351390 2024-04-25T11:40:57.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.264+0000 7fa7a31f9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa7941b09d0 con 0x55e9a33284d0 2024-04-25T11:40:57.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.264+0000 7fa7a31f9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa7941adc10 con 0x55e9a33306b0 2024-04-25T11:40:57.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.264+0000 7fa7a31f9740 10 client.0 _ll_register_callbacks cb 0x7fa784009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:57.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.264+0000 7fa79a0e2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9a33284d0 0x55e9a3703c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:57.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.264+0000 7fa79a8e3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9a3351390 0x55e9a3774a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:57.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.264+0000 7fa79a0e2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9a33284d0 0x55e9a3703c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:40866/0 (socket says 172.21.15.90:40866) 2024-04-25T11:40:57.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.264+0000 7fa79a8e3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9a3351390 0x55e9a3774a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46088/0 (socket says 172.21.15.90:46088) 2024-04-25T11:40:57.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.264+0000 7fa79a0e2640 1 -- 172.21.15.90:0/1996821077 learned_addr learned my addr 172.21.15.90:0/1996821077 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:57.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.264+0000 7fa7998e1640 1 --2- 172.21.15.90:0/1996821077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9a33306b0 0x55e9a37041b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:57.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.265+0000 7fa79a0e2640 1 -- 172.21.15.90:0/1996821077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9a33306b0 msgr2=0x55e9a37041b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:57.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.265+0000 7fa79a0e2640 1 --2- 172.21.15.90:0/1996821077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9a33306b0 0x55e9a37041b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.265+0000 7fa79a0e2640 1 -- 172.21.15.90:0/1996821077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9a3351390 msgr2=0x55e9a3774a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:57.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.265+0000 7fa79a0e2640 1 --2- 172.21.15.90:0/1996821077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9a3351390 0x55e9a3774a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.265+0000 7fa79a0e2640 1 -- 172.21.15.90:0/1996821077 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9a36184b0 con 0x55e9a33284d0 2024-04-25T11:40:57.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.265+0000 7fa79a8e3640 1 --2- 172.21.15.90:0/1996821077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9a3351390 0x55e9a3774a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:40:57.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.265+0000 7fa79a0e2640 1 --2- 172.21.15.90:0/1996821077 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9a33284d0 0x55e9a3703c70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa7900ab600 tx=0x7fa7900a6b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3115f2036b307529 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:57.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.265+0000 7fa77dffb640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:40:57.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.265+0000 7fa77dffb640 1 -- 172.21.15.90:0/1996821077 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7900e2a90 con 0x55e9a33284d0 2024-04-25T11:40:57.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.265+0000 7fa77dffb640 1 -- 172.21.15.90:0/1996821077 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fa7900a3450 con 0x55e9a33284d0 2024-04-25T11:40:57.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.265+0000 7fa7a31f9740 1 -- 172.21.15.90:0/1996821077 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7900a6cd0 con 0x55e9a33284d0 2024-04-25T11:40:57.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.266+0000 7fa77dffb640 1 -- 172.21.15.90:0/1996821077 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa7900b0020 con 0x55e9a33284d0 2024-04-25T11:40:57.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.266+0000 7fa7a31f9740 10 client.26354 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:57.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.266+0000 7fa7a31f9740 1 -- 172.21.15.90:0/1996821077 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e9a3776000 con 0x55e9a33284d0 2024-04-25T11:40:57.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.266+0000 7fa77dffb640 1 -- 172.21.15.90:0/1996821077 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7900a6cd0 con 0x55e9a33284d0 2024-04-25T11:40:57.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.266+0000 7fa77dffb640 1 -- 172.21.15.90:0/1996821077 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fa7900b80b0 con 0x55e9a33284d0 2024-04-25T11:40:57.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.266+0000 7fa7a31f9740 10 client.26354 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:57.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.266+0000 7fa7a31f9740 10 client.26354 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:40:57.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.266+0000 7fa7a31f9740 20 client.26354 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:57.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.266+0000 7fa7a31f9740 1 --2- 172.21.15.90:0/1996821077 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e9a32e53a0 0x55e9a3778290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:57.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.266+0000 7fa7a31f9740 4 client.26354 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:40:57.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.266+0000 7fa7a31f9740 1 -- 172.21.15.90:0/1996821077 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e9a36f2330 con 0x55e9a32e53a0 2024-04-25T11:40:57.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.266+0000 7fa7998e1640 1 --2- 172.21.15.90:0/1996821077 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e9a32e53a0 0x55e9a3778290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:57.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.267+0000 7fa77dffb640 10 client.26354 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:57.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.267+0000 7fa7998e1640 1 --2- 172.21.15.90:0/1996821077 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e9a32e53a0 0x55e9a3778290 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a71629cea3d97a1 server_cookie=619dfe5a10d54064 in_seq=0 out_seq=0 2024-04-25T11:40:57.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.267+0000 7fa77dffb640 1 -- 172.21.15.90:0/1996821077 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e9a36f2330 con 0x55e9a32e53a0 2024-04-25T11:40:57.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.267+0000 7fa77dffb640 10 client.26354 handle_command_reply: tid=0 2024-04-25T11:40:57.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.267+0000 7fa7a31f9740 1 client.26354 shutdown 2024-04-25T11:40:57.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.267+0000 7fa7a31f9740 1 -- 172.21.15.90:0/1996821077 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9a33284d0 msgr2=0x55e9a3703c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:57.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.267+0000 7fa7a31f9740 1 --2- 172.21.15.90:0/1996821077 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9a33284d0 0x55e9a3703c70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa7900ab600 tx=0x7fa7900a6b10 comp rx=0 tx=0).stop 2024-04-25T11:40:57.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.267+0000 7fa7a31f9740 1 -- 172.21.15.90:0/1996821077 shutdown_connections 2024-04-25T11:40:57.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.267+0000 7fa7a31f9740 1 --2- 172.21.15.90:0/1996821077 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e9a32e53a0 0x55e9a3778290 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.267+0000 7fa7a31f9740 1 --2- 172.21.15.90:0/1996821077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9a3351390 0x55e9a3774a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.267+0000 7fa77dffb640 0 client.26354 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:57.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.267+0000 7fa7a31f9740 1 --2- 172.21.15.90:0/1996821077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e9a33306b0 0x55e9a37041b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.267+0000 7fa7a31f9740 1 --2- 172.21.15.90:0/1996821077 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e9a33284d0 0x55e9a3703c70 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.268+0000 7fa7a31f9740 1 -- 172.21.15.90:0/1996821077 >> 172.21.15.90:0/1996821077 conn(0x55e9a33454e0 msgr2=0x55e9a36ed3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:57.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.268+0000 7fa7a31f9740 1 -- 172.21.15.90:0/1996821077 shutdown_connections 2024-04-25T11:40:57.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.268+0000 7fa7a31f9740 1 -- 172.21.15.90:0/1996821077 wait complete. 2024-04-25T11:40:57.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.268+0000 7fa7a31f9740 20 client.26354 trim_cache size 0 max 16384 2024-04-25T11:40:57.261 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:40:57.261 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-25T11:40:57.261 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-25T11:40:57.261 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:40:57.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.268+0000 7fa79c346640 1 -- 172.21.15.90:0/3533057507 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa7800afc60 msgr2=0x7fa7800b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:57.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.268+0000 7fa79c346640 1 --2- 172.21.15.90:0/3533057507 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa7800afc60 0x7fa7800b2170 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x55e9a36eebd0 tx=0x7fa78c004a40 comp rx=0 tx=0).stop 2024-04-25T11:40:57.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.268+0000 7fa79c346640 1 -- 172.21.15.90:0/3533057507 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa794103180 msgr2=0x7fa79419b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:57.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.268+0000 7fa79c346640 1 --2- 172.21.15.90:0/3533057507 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa794103180 0x7fa79419b5a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa794103710 tx=0x7fa79000cdd0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.268+0000 7fa79c346640 1 -- 172.21.15.90:0/3533057507 shutdown_connections 2024-04-25T11:40:57.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.268+0000 7fa79c346640 1 --2- 172.21.15.90:0/3533057507 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa7800afc60 0x7fa7800b2170 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.268+0000 7fa79c346640 1 --2- 172.21.15.90:0/3533057507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa794108ec0 0x7fa79419c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.268+0000 7fa79c346640 1 --2- 172.21.15.90:0/3533057507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa794108a20 0x7fa79419bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.268+0000 7fa79c346640 1 --2- 172.21.15.90:0/3533057507 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa794103180 0x7fa79419b5a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.268+0000 7fa79c346640 1 -- 172.21.15.90:0/3533057507 >> 172.21.15.90:0/3533057507 conn(0x7fa794078110 msgr2=0x7fa794104100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:57.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.269+0000 7fa79c346640 1 -- 172.21.15.90:0/3533057507 shutdown_connections 2024-04-25T11:40:57.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.269+0000 7fa79c346640 1 -- 172.21.15.90:0/3533057507 wait complete. 2024-04-25T11:40:57.266 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T11:40:57.266 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:40:57.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.541+0000 7f6fab7e3640 1 -- 172.21.15.90:0/60286207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fa41023d0 msgr2=0x7f6fa41027e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:57.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.541+0000 7f6fab7e3640 1 --2- 172.21.15.90:0/60286207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fa41023d0 0x7f6fa41027e0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0009900 tx=0x7f6fa002a410 comp rx=0 tx=0).stop 2024-04-25T11:40:57.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.542+0000 7f6fab7e3640 1 -- 172.21.15.90:0/60286207 shutdown_connections 2024-04-25T11:40:57.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.542+0000 7f6fab7e3640 1 --2- 172.21.15.90:0/60286207 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6fa4107420 0x7f6fa410ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.542+0000 7f6fab7e3640 1 --2- 172.21.15.90:0/60286207 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fa4106960 0x7f6fa4106d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.542+0000 7f6fab7e3640 1 --2- 172.21.15.90:0/60286207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fa41023d0 0x7f6fa41027e0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.542+0000 7f6fab7e3640 1 -- 172.21.15.90:0/60286207 >> 172.21.15.90:0/60286207 conn(0x7f6fa40fd810 msgr2=0x7f6fa40ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:57.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.542+0000 7f6fab7e3640 1 -- 172.21.15.90:0/60286207 shutdown_connections 2024-04-25T11:40:57.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.542+0000 7f6fab7e3640 1 -- 172.21.15.90:0/60286207 wait complete. 2024-04-25T11:40:57.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.543+0000 7f6fab7e3640 1 Processor -- start 2024-04-25T11:40:57.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.543+0000 7f6fab7e3640 1 -- start start 2024-04-25T11:40:57.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.543+0000 7f6fab7e3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fa41023d0 0x7f6fa419a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:57.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.543+0000 7f6fab7e3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fa4106960 0x7f6fa419ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:57.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.543+0000 7f6fab7e3640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6fa4107420 0x7f6fa419ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:57.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.544+0000 7f6fab7e3640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa4111b00 con 0x7f6fa41023d0 2024-04-25T11:40:57.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.544+0000 7f6fab7e3640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa4111980 con 0x7f6fa4107420 2024-04-25T11:40:57.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.544+0000 7f6fab7e3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa4111c80 con 0x7f6fa4106960 2024-04-25T11:40:57.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.544+0000 7f6fa957f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fa41023d0 0x7f6fa419a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:57.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.544+0000 7f6fa957f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fa41023d0 0x7f6fa419a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46096/0 (socket says 172.21.15.90:46096) 2024-04-25T11:40:57.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.544+0000 7f6fa957f640 1 -- 172.21.15.90:0/757934653 learned_addr learned my addr 172.21.15.90:0/757934653 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:57.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.544+0000 7f6fa957f640 1 -- 172.21.15.90:0/757934653 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fa4106960 msgr2=0x7f6fa419ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:57.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.544+0000 7f6fa957f640 1 --2- 172.21.15.90:0/757934653 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fa4106960 0x7f6fa419ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.544+0000 7f6fa957f640 1 -- 172.21.15.90:0/757934653 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6fa4107420 msgr2=0x7f6fa419ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:57.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.544+0000 7f6fa957f640 1 --2- 172.21.15.90:0/757934653 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6fa4107420 0x7f6fa419ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.544+0000 7f6fa957f640 1 -- 172.21.15.90:0/757934653 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fa419f580 con 0x7f6fa41023d0 2024-04-25T11:40:57.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.545+0000 7f6fa957f640 1 --2- 172.21.15.90:0/757934653 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fa41023d0 0x7f6fa419a720 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0005ab0 tx=0x7f6fa0003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:57.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.556+0000 7f6f9a7fc640 1 -- 172.21.15.90:0/757934653 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa003f070 con 0x7f6fa41023d0 2024-04-25T11:40:57.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.557+0000 7f6f9a7fc640 1 -- 172.21.15.90:0/757934653 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6fa0031d60 con 0x7f6fa41023d0 2024-04-25T11:40:57.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.557+0000 7f6fab7e3640 1 -- 172.21.15.90:0/757934653 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fa419f810 con 0x7f6fa41023d0 2024-04-25T11:40:57.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.557+0000 7f6f9a7fc640 1 -- 172.21.15.90:0/757934653 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0035030 con 0x7f6fa41023d0 2024-04-25T11:40:57.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.557+0000 7f6fab7e3640 1 -- 172.21.15.90:0/757934653 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fa419fd50 con 0x7f6fa41023d0 2024-04-25T11:40:57.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.559+0000 7f6fab7e3640 1 -- 172.21.15.90:0/757934653 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fa4069310 con 0x7f6fa41023d0 2024-04-25T11:40:57.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.561+0000 7f6f9a7fc640 1 -- 172.21.15.90:0/757934653 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f6fa0042070 con 0x7f6fa41023d0 2024-04-25T11:40:57.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.563+0000 7f6f9a7fc640 1 --2- 172.21.15.90:0/757934653 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f6f900afce0 0x7f6f900b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:57.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.563+0000 7f6f9a7fc640 1 -- 172.21.15.90:0/757934653 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f6fa00fffd0 con 0x7f6fa41023d0 2024-04-25T11:40:57.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.564+0000 7f6fa8d7e640 1 --2- 172.21.15.90:0/757934653 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f6f900afce0 0x7f6f900b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:57.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.564+0000 7f6fa8d7e640 1 --2- 172.21.15.90:0/757934653 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f6f900afce0 0x7f6f900b21f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6f94005fd0 tx=0x7f6f94005850 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:57.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.569+0000 7f6f9a7fc640 1 -- 172.21.15.90:0/757934653 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6fa00c8810 con 0x7f6fa41023d0 2024-04-25T11:40:57.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:57 smithi138 ceph-mon[32328]: pgmap v179: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 2.7 MiB/s rd, 626 MiB/s wr, 670 op/s 2024-04-25T11:40:57.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:57 smithi156 ceph-mon[33045]: pgmap v179: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 2.7 MiB/s rd, 626 MiB/s wr, 670 op/s 2024-04-25T11:40:57.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.860+0000 7f6fab7e3640 1 -- 172.21.15.90:0/757934653 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6fa419ba70 con 0x7f6fa41023d0 2024-04-25T11:40:57.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.861+0000 7f6f9a7fc640 1 -- 172.21.15.90:0/757934653 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11930 (secure 0 0 0) 0x7f6fa00c8110 con 0x7f6fa41023d0 2024-04-25T11:40:57.848 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:40:57.848 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:39:20.469254+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24458,24431,24499,24475,24524]},"id":1}]} 2024-04-25T11:40:57.848 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 28 2024-04-25T11:40:57.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.863+0000 7f6fab7e3640 1 -- 172.21.15.90:0/757934653 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f6f900afce0 msgr2=0x7f6f900b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:57.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.863+0000 7f6fab7e3640 1 --2- 172.21.15.90:0/757934653 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f6f900afce0 0x7f6f900b21f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6f94005fd0 tx=0x7f6f94005850 comp rx=0 tx=0).stop 2024-04-25T11:40:57.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.863+0000 7f6fab7e3640 1 -- 172.21.15.90:0/757934653 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fa41023d0 msgr2=0x7f6fa419a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:57.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.863+0000 7f6fab7e3640 1 --2- 172.21.15.90:0/757934653 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fa41023d0 0x7f6fa419a720 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0005ab0 tx=0x7f6fa0003cd0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.863+0000 7f6fab7e3640 1 -- 172.21.15.90:0/757934653 shutdown_connections 2024-04-25T11:40:57.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.863+0000 7f6fab7e3640 1 --2- 172.21.15.90:0/757934653 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f6f900afce0 0x7f6f900b21f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.863+0000 7f6fab7e3640 1 --2- 172.21.15.90:0/757934653 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6fa4107420 0x7f6fa419ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.863+0000 7f6fab7e3640 1 --2- 172.21.15.90:0/757934653 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fa4106960 0x7f6fa419ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.863+0000 7f6fab7e3640 1 --2- 172.21.15.90:0/757934653 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fa41023d0 0x7f6fa419a720 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:57.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.863+0000 7f6fab7e3640 1 -- 172.21.15.90:0/757934653 >> 172.21.15.90:0/757934653 conn(0x7f6fa40fd810 msgr2=0x7f6fa410cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:57.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.863+0000 7f6fab7e3640 1 -- 172.21.15.90:0/757934653 shutdown_connections 2024-04-25T11:40:57.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:57.863+0000 7f6fab7e3640 1 -- 172.21.15.90:0/757934653 wait complete. 2024-04-25T11:40:57.861 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:57 smithi090 ceph-mon[29575]: pgmap v179: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 2.7 MiB/s rd, 626 MiB/s wr, 670 op/s 2024-04-25T11:40:57.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T11:40:58.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.101+0000 7f4a48346640 1 -- 172.21.15.90:0/1021571397 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a40103180 msgr2=0x7f4a4010f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.101+0000 7f4a48346640 1 --2- 172.21.15.90:0/1021571397 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a40103180 0x7f4a4010f4b0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f4a38009ca0 tx=0x7f4a3802a8e0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.102+0000 7f4a48346640 1 -- 172.21.15.90:0/1021571397 shutdown_connections 2024-04-25T11:40:58.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.102+0000 7f4a48346640 1 --2- 172.21.15.90:0/1021571397 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a40103180 0x7f4a4010f4b0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.102+0000 7f4a48346640 1 --2- 172.21.15.90:0/1021571397 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4a40108ec0 0x7f4a40102b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.102+0000 7f4a48346640 1 --2- 172.21.15.90:0/1021571397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a40108a20 0x7f4a401025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.102+0000 7f4a48346640 1 -- 172.21.15.90:0/1021571397 >> 172.21.15.90:0/1021571397 conn(0x7f4a40078110 msgr2=0x7f4a40078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:58.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.102+0000 7f4a48346640 1 -- 172.21.15.90:0/1021571397 shutdown_connections 2024-04-25T11:40:58.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.102+0000 7f4a48346640 1 -- 172.21.15.90:0/1021571397 wait complete. 2024-04-25T11:40:58.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.103+0000 7f4a48346640 1 Processor -- start 2024-04-25T11:40:58.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.103+0000 7f4a48346640 1 -- start start 2024-04-25T11:40:58.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.103+0000 7f4a48346640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4a40103180 0x7f4a4019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.103+0000 7f4a48346640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a40108a20 0x7f4a4019aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.103+0000 7f4a48346640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a40108ec0 0x7f4a4019ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.103+0000 7f4a48346640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4a40112d10 con 0x7f4a40108ec0 2024-04-25T11:40:58.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.103+0000 7f4a48346640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4a40112b90 con 0x7f4a40103180 2024-04-25T11:40:58.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.103+0000 7f4a48346640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4a40112e90 con 0x7f4a40108a20 2024-04-25T11:40:58.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.103+0000 7f4a468e3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a40108ec0 0x7f4a4019ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.103+0000 7f4a468e3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a40108ec0 0x7f4a4019ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46120/0 (socket says 172.21.15.90:46120) 2024-04-25T11:40:58.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.103+0000 7f4a468e3640 1 -- 172.21.15.90:0/3859862619 learned_addr learned my addr 172.21.15.90:0/3859862619 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:58.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.104+0000 7f4a468e3640 1 -- 172.21.15.90:0/3859862619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a40108a20 msgr2=0x7f4a4019aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.104+0000 7f4a468e3640 1 --2- 172.21.15.90:0/3859862619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a40108a20 0x7f4a4019aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.104+0000 7f4a468e3640 1 -- 172.21.15.90:0/3859862619 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4a40103180 msgr2=0x7f4a4019a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.104+0000 7f4a468e3640 1 --2- 172.21.15.90:0/3859862619 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4a40103180 0x7f4a4019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.104+0000 7f4a468e3640 1 -- 172.21.15.90:0/3859862619 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a4019f5a0 con 0x7f4a40108ec0 2024-04-25T11:40:58.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.104+0000 7f4a468e3640 1 --2- 172.21.15.90:0/3859862619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a40108ec0 0x7f4a4019ef60 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f4a38003bd0 tx=0x7f4a38003c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:58.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.104+0000 7f4a377fe640 1 -- 172.21.15.90:0/3859862619 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a38004240 con 0x7f4a40108ec0 2024-04-25T11:40:58.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.104+0000 7f4a48346640 1 -- 172.21.15.90:0/3859862619 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a4019f830 con 0x7f4a40108ec0 2024-04-25T11:40:58.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.104+0000 7f4a48346640 1 -- 172.21.15.90:0/3859862619 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a40104100 con 0x7f4a40108ec0 2024-04-25T11:40:58.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.104+0000 7f4a377fe640 1 -- 172.21.15.90:0/3859862619 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a38003e20 con 0x7f4a40108ec0 2024-04-25T11:40:58.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.104+0000 7f4a377fe640 1 -- 172.21.15.90:0/3859862619 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3803a400 con 0x7f4a40108ec0 2024-04-25T11:40:58.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.108+0000 7f4a377fe640 1 -- 172.21.15.90:0/3859862619 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f4a3803a5a0 con 0x7f4a40108ec0 2024-04-25T11:40:58.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.109+0000 7f4a4f1fc740 1 Processor -- start 2024-04-25T11:40:58.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.109+0000 7f4a4f1fc740 1 -- start start 2024-04-25T11:40:58.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.110+0000 7f4a4f1fc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d303334390 0x55d3036d5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.110+0000 7f4a4f1fc740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d30330b4d0 0x55d3036d5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.110+0000 7f4a377fe640 1 --2- 172.21.15.90:0/3859862619 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4a2c0afc60 0x7f4a2c0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.110+0000 7f4a4f1fc740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d3033136b0 0x55d3036e26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.110+0000 7f4a458e1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d303334390 0x55d3036d5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.110+0000 7f4a458e1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d303334390 0x55d3036d5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46130/0 (socket says 172.21.15.90:46130) 2024-04-25T11:40:58.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.110+0000 7f4a458e1640 1 -- 172.21.15.90:0/440747522 learned_addr learned my addr 172.21.15.90:0/440747522 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:58.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.110+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/440747522 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d30363ad80 con 0x55d303334390 2024-04-25T11:40:58.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.110+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/440747522 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55d3035f4a20 con 0x55d3033136b0 2024-04-25T11:40:58.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.111+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/440747522 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d303636d40 con 0x55d30330b4d0 2024-04-25T11:40:58.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.111+0000 7f4a458e1640 1 --2- 172.21.15.90:0/3859862619 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4a2c0afc60 0x7f4a2c0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.111+0000 7f4a377fe640 1 -- 172.21.15.90:0/3859862619 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f4a38100380 con 0x7f4a40108ec0 2024-04-25T11:40:58.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.111+0000 7f4a460e2640 1 --2- 172.21.15.90:0/440747522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d30330b4d0 0x55d3036d5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.111+0000 7f4a458e1640 1 -- 172.21.15.90:0/440747522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d30330b4d0 msgr2=0x55d3036d5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.111+0000 7f4a468e3640 1 --2- 172.21.15.90:0/440747522 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d3033136b0 0x55d3036e26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.111+0000 7f4a458e1640 1 --2- 172.21.15.90:0/440747522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d30330b4d0 0x55d3036d5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.111+0000 7f4a458e1640 1 -- 172.21.15.90:0/440747522 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d3033136b0 msgr2=0x55d3036e26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.111+0000 7f4a458e1640 1 --2- 172.21.15.90:0/440747522 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d3033136b0 0x55d3036e26a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.111+0000 7f4a458e1640 1 -- 172.21.15.90:0/440747522 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3035ffc30 con 0x55d303334390 2024-04-25T11:40:58.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.112+0000 7f4a458e1640 1 --2- 172.21.15.90:0/440747522 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d303334390 0x55d3036d5810 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f4a3000dac0 tx=0x7f4a3000df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2bde114154889 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:58.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.112+0000 7f4a458e1640 1 --2- 172.21.15.90:0/3859862619 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4a2c0afc60 0x7f4a2c0b2170 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4a3000a000 tx=0x7f4a3001a000 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:58.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.112+0000 7f4a34bfe640 1 -- 172.21.15.90:0/440747522 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3000ba10 con 0x55d303334390 2024-04-25T11:40:58.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.112+0000 7f4a34bfe640 1 -- 172.21.15.90:0/440747522 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a3001d070 con 0x55d303334390 2024-04-25T11:40:58.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.112+0000 7f4a34bfe640 1 -- 172.21.15.90:0/440747522 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a30019a50 con 0x55d303334390 2024-04-25T11:40:58.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.114+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/440747522 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d303334390 msgr2=0x55d3036d5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.114+0000 7f4a4f1fc740 1 --2- 172.21.15.90:0/440747522 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d303334390 0x55d3036d5810 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f4a3000dac0 tx=0x7f4a3000df80 comp rx=0 tx=0).stop 2024-04-25T11:40:58.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.114+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/440747522 shutdown_connections 2024-04-25T11:40:58.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.114+0000 7f4a4f1fc740 1 --2- 172.21.15.90:0/440747522 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d3033136b0 0x55d3036e26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.114+0000 7f4a4f1fc740 1 --2- 172.21.15.90:0/440747522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d30330b4d0 0x55d3036d5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.114+0000 7f4a4f1fc740 1 --2- 172.21.15.90:0/440747522 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d303334390 0x55d3036d5810 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.114+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/440747522 >> 172.21.15.90:0/440747522 conn(0x55d3033284e0 msgr2=0x55d3036d1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:58.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.114+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/440747522 shutdown_connections 2024-04-25T11:40:58.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.114+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/440747522 wait complete. 2024-04-25T11:40:58.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.116+0000 7f4a4f1fc740 1 -- start start 2024-04-25T11:40:58.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.116+0000 7f4a4f1fc740 1 Processor -- start 2024-04-25T11:40:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.117+0000 7f4a4f1fc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d30330b4d0 0x55d3036e0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.117+0000 7f4a4f1fc740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3033136b0 0x55d3036e0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.117+0000 7f4a4f1fc740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d303334390 0x55d3036ec2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.117+0000 7f4a4f1fc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d3036e2d30 con 0x55d30330b4d0 2024-04-25T11:40:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.117+0000 7f4a4f1fc740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55d3035c1dc0 con 0x55d303334390 2024-04-25T11:40:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.117+0000 7f4a4f1fc740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d3036e2eb0 con 0x55d3033136b0 2024-04-25T11:40:58.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.117+0000 7f4a4f1fc740 10 client.0 _ll_register_callbacks cb 0x55d3035dd600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:58.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.117+0000 7f4a460e2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d30330b4d0 0x55d3036e0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.117+0000 7f4a460e2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d30330b4d0 0x55d3036e0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46136/0 (socket says 172.21.15.90:46136) 2024-04-25T11:40:58.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.117+0000 7f4a460e2640 1 -- 172.21.15.90:0/1694360272 learned_addr learned my addr 172.21.15.90:0/1694360272 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:58.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.118+0000 7f4a468e3640 1 --2- 172.21.15.90:0/1694360272 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d303334390 0x55d3036ec2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.118+0000 7f4a458e1640 1 --2- 172.21.15.90:0/1694360272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3033136b0 0x55d3036e0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.118+0000 7f4a460e2640 1 -- 172.21.15.90:0/1694360272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3033136b0 msgr2=0x55d3036e0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.118+0000 7f4a460e2640 1 --2- 172.21.15.90:0/1694360272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3033136b0 0x55d3036e0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.118+0000 7f4a460e2640 1 -- 172.21.15.90:0/1694360272 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d303334390 msgr2=0x55d3036ec2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.118+0000 7f4a460e2640 1 --2- 172.21.15.90:0/1694360272 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d303334390 0x55d3036ec2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.118+0000 7f4a460e2640 1 -- 172.21.15.90:0/1694360272 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d30334c3f0 con 0x55d30330b4d0 2024-04-25T11:40:58.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.118+0000 7f4a468e3640 1 --2- 172.21.15.90:0/1694360272 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d303334390 0x55d3036ec2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:40:58.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.118+0000 7f4a460e2640 1 --2- 172.21.15.90:0/1694360272 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d30330b4d0 0x55d3036e0250 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c00e650 tx=0x7f4a3c00eb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=415383001e2e279b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:58.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.118+0000 7f4a29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:40:58.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.119+0000 7f4a29ffb640 1 -- 172.21.15.90:0/1694360272 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c004220 con 0x55d30330b4d0 2024-04-25T11:40:58.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.119+0000 7f4a29ffb640 1 -- 172.21.15.90:0/1694360272 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f4a3c0043c0 con 0x55d30330b4d0 2024-04-25T11:40:58.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.119+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/1694360272 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a3001d070 con 0x55d30330b4d0 2024-04-25T11:40:58.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.119+0000 7f4a29ffb640 1 -- 172.21.15.90:0/1694360272 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4a3c00ee50 con 0x55d30330b4d0 2024-04-25T11:40:58.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.119+0000 7f4a4f1fc740 10 client.17295 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:58.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.119+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/1694360272 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d3034167d0 con 0x55d30330b4d0 2024-04-25T11:40:58.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.120+0000 7f4a29ffb640 1 -- 172.21.15.90:0/1694360272 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a3001d070 con 0x55d30330b4d0 2024-04-25T11:40:58.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.120+0000 7f4a29ffb640 1 -- 172.21.15.90:0/1694360272 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f4a3c01a440 con 0x55d30330b4d0 2024-04-25T11:40:58.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.120+0000 7f4a4f1fc740 10 client.17295 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:58.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.120+0000 7f4a4f1fc740 10 client.17295 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:40:58.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.120+0000 7f4a4f1fc740 20 client.17295 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:58.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.120+0000 7f4a4f1fc740 1 --2- 172.21.15.90:0/1694360272 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55d3032f2f60 0x55d3036f2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.120+0000 7f4a4f1fc740 4 client.17295 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:40:58.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.120+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/1694360272 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d303648990 con 0x55d3032f2f60 2024-04-25T11:40:58.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.120+0000 7f4a458e1640 1 --2- 172.21.15.90:0/1694360272 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55d3032f2f60 0x55d3036f2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.122+0000 7f4a29ffb640 10 client.17295 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:40:58.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.122+0000 7f4a458e1640 1 --2- 172.21.15.90:0/1694360272 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55d3032f2f60 0x55d3036f2850 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=570832f3da72f667 server_cookie=57da37530f64d7b4 in_seq=0 out_seq=0 2024-04-25T11:40:58.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.123+0000 7f4a29ffb640 1 -- 172.21.15.90:0/1694360272 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d303648990 con 0x55d3032f2f60 2024-04-25T11:40:58.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.123+0000 7f4a29ffb640 10 client.17295 handle_command_reply: tid=0 2024-04-25T11:40:58.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.123+0000 7f4a4f1fc740 1 client.17295 shutdown 2024-04-25T11:40:58.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.124+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/1694360272 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d30330b4d0 msgr2=0x55d3036e0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.124+0000 7f4a4f1fc740 1 --2- 172.21.15.90:0/1694360272 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d30330b4d0 0x55d3036e0250 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c00e650 tx=0x7f4a3c00eb10 comp rx=0 tx=0).stop 2024-04-25T11:40:58.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.124+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/1694360272 shutdown_connections 2024-04-25T11:40:58.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.124+0000 7f4a4f1fc740 1 --2- 172.21.15.90:0/1694360272 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55d3032f2f60 0x55d3036f2850 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.124+0000 7f4a4f1fc740 1 --2- 172.21.15.90:0/1694360272 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d303334390 0x55d3036ec2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.124+0000 7f4a4f1fc740 1 --2- 172.21.15.90:0/1694360272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3033136b0 0x55d3036e0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.124+0000 7f4a4f1fc740 1 --2- 172.21.15.90:0/1694360272 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d30330b4d0 0x55d3036e0250 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.124+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/1694360272 >> 172.21.15.90:0/1694360272 conn(0x55d3033284e0 msgr2=0x55d3036d7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:58.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.124+0000 7f4a29ffb640 0 client.17295 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:40:58.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.124+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/1694360272 shutdown_connections 2024-04-25T11:40:58.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.124+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/1694360272 wait complete. 2024-04-25T11:40:58.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.124+0000 7f4a4f1fc740 20 client.17295 trim_cache size 0 max 16384 2024-04-25T11:40:58.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.147+0000 7f4a4f1fc740 1 Processor -- start 2024-04-25T11:40:58.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.148+0000 7f4a4f1fc740 1 -- start start 2024-04-25T11:40:58.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.148+0000 7f4a4f1fc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d30330b4d0 0x55d3036df340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.148+0000 7f4a4f1fc740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d3033136b0 0x55d3036d5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.148+0000 7f4a4f1fc740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d303334390 0x55d3036d1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.148+0000 7f4a4f1fc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2400e990 con 0x55d30330b4d0 2024-04-25T11:40:58.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.148+0000 7f4a4f1fc740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4a240104d0 con 0x55d3033136b0 2024-04-25T11:40:58.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.148+0000 7f4a4f1fc740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2400f060 con 0x55d303334390 2024-04-25T11:40:58.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.148+0000 7f4a460e2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d30330b4d0 0x55d3036df340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.148+0000 7f4a460e2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d30330b4d0 0x55d3036df340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46144/0 (socket says 172.21.15.90:46144) 2024-04-25T11:40:58.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.148+0000 7f4a460e2640 1 -- 172.21.15.90:0/3710451226 learned_addr learned my addr 172.21.15.90:0/3710451226 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:58.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.149+0000 7f4a460e2640 1 -- 172.21.15.90:0/3710451226 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d303334390 msgr2=0x55d3036d1ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:40:58.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.149+0000 7f4a460e2640 1 --2- 172.21.15.90:0/3710451226 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d303334390 0x55d3036d1ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.149+0000 7f4a460e2640 1 -- 172.21.15.90:0/3710451226 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d3033136b0 msgr2=0x55d3036d5c30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:40:58.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.149+0000 7f4a460e2640 1 --2- 172.21.15.90:0/3710451226 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d3033136b0 0x55d3036d5c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.149+0000 7f4a460e2640 1 -- 172.21.15.90:0/3710451226 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3036de590 con 0x55d30330b4d0 2024-04-25T11:40:58.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.149+0000 7f4a460e2640 1 --2- 172.21.15.90:0/3710451226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d30330b4d0 0x55d3036df340 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c01a860 tx=0x7f4a3c00ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf27d1c521dd84dc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:58.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.149+0000 7f4a2bfff640 1 -- 172.21.15.90:0/3710451226 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c01a440 con 0x55d30330b4d0 2024-04-25T11:40:58.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.149+0000 7f4a2bfff640 1 -- 172.21.15.90:0/3710451226 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a3c01a5e0 con 0x55d30330b4d0 2024-04-25T11:40:58.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.149+0000 7f4a2bfff640 1 -- 172.21.15.90:0/3710451226 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c02cad0 con 0x55d30330b4d0 2024-04-25T11:40:58.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.151+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/3710451226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d30330b4d0 msgr2=0x55d3036df340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.151+0000 7f4a4f1fc740 1 --2- 172.21.15.90:0/3710451226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d30330b4d0 0x55d3036df340 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c01a860 tx=0x7f4a3c00ca80 comp rx=0 tx=0).stop 2024-04-25T11:40:58.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.151+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/3710451226 shutdown_connections 2024-04-25T11:40:58.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.151+0000 7f4a4f1fc740 1 --2- 172.21.15.90:0/3710451226 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d303334390 0x55d3036d1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.151+0000 7f4a4f1fc740 1 --2- 172.21.15.90:0/3710451226 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d3033136b0 0x55d3036d5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.151+0000 7f4a4f1fc740 1 --2- 172.21.15.90:0/3710451226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d30330b4d0 0x55d3036df340 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.151+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/3710451226 >> 172.21.15.90:0/3710451226 conn(0x55d3033284e0 msgr2=0x55d3036d0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:58.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.151+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/3710451226 shutdown_connections 2024-04-25T11:40:58.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.151+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/3710451226 wait complete. 2024-04-25T11:40:58.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.152+0000 7f4a4f1fc740 1 -- start start 2024-04-25T11:40:58.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.152+0000 7f4a4f1fc740 1 Processor -- start 2024-04-25T11:40:58.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.152+0000 7f4a4f1fc740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d30330b4d0 0x55d3036e31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.153+0000 7f4a4f1fc740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3033136b0 0x55d3036e3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.153+0000 7f4a4f1fc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d303334390 0x55d3036e3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.153+0000 7f4a4f1fc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2400fe00 con 0x55d303334390 2024-04-25T11:40:58.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.153+0000 7f4a4f1fc740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2400f730 con 0x55d30330b4d0 2024-04-25T11:40:58.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.153+0000 7f4a4f1fc740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2400c970 con 0x55d3033136b0 2024-04-25T11:40:58.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.153+0000 7f4a4f1fc740 10 client.0 _ll_register_callbacks cb 0x7f4a3c00c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:58.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.153+0000 7f4a468e3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d303334390 0x55d3036e3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.153+0000 7f4a468e3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d303334390 0x55d3036e3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46146/0 (socket says 172.21.15.90:46146) 2024-04-25T11:40:58.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.153+0000 7f4a468e3640 1 -- 172.21.15.90:0/2288881433 learned_addr learned my addr 172.21.15.90:0/2288881433 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:58.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.153+0000 7f4a460e2640 1 --2- 172.21.15.90:0/2288881433 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d30330b4d0 0x55d3036e31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.154+0000 7f4a458e1640 1 --2- 172.21.15.90:0/2288881433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3033136b0 0x55d3036e3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.154+0000 7f4a468e3640 1 -- 172.21.15.90:0/2288881433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3033136b0 msgr2=0x55d3036e3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.154+0000 7f4a468e3640 1 --2- 172.21.15.90:0/2288881433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3033136b0 0x55d3036e3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.154+0000 7f4a468e3640 1 -- 172.21.15.90:0/2288881433 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d30330b4d0 msgr2=0x55d3036e31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.154+0000 7f4a468e3640 1 --2- 172.21.15.90:0/2288881433 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d30330b4d0 0x55d3036e31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.154+0000 7f4a468e3640 1 -- 172.21.15.90:0/2288881433 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d303758bf0 con 0x55d303334390 2024-04-25T11:40:58.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.154+0000 7f4a458e1640 1 --2- 172.21.15.90:0/2288881433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3033136b0 0x55d3036e3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:40:58.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.154+0000 7f4a468e3640 1 --2- 172.21.15.90:0/2288881433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d303334390 0x55d3036e3c40 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f4a38006010 tx=0x7f4a380c80e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d855970b550bfa38 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:58.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.155+0000 7f4a29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:40:58.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.155+0000 7f4a29ffb640 1 -- 172.21.15.90:0/2288881433 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a380b10c0 con 0x55d303334390 2024-04-25T11:40:58.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.155+0000 7f4a29ffb640 1 -- 172.21.15.90:0/2288881433 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f4a38009890 con 0x55d303334390 2024-04-25T11:40:58.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.155+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/2288881433 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a3c01a5e0 con 0x55d303334390 2024-04-25T11:40:58.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.156+0000 7f4a29ffb640 1 -- 172.21.15.90:0/2288881433 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4a380c85d0 con 0x55d303334390 2024-04-25T11:40:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.158+0000 7f4a4f1fc740 10 client.17301 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.158+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/2288881433 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d303759d90 con 0x55d303334390 2024-04-25T11:40:58.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.159+0000 7f4a29ffb640 1 -- 172.21.15.90:0/2288881433 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a38036040 con 0x55d303334390 2024-04-25T11:40:58.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.159+0000 7f4a29ffb640 1 -- 172.21.15.90:0/2288881433 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f4a380c4860 con 0x55d303334390 2024-04-25T11:40:58.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.159+0000 7f4a4f1fc740 10 client.17301 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:58.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.159+0000 7f4a4f1fc740 10 client.17301 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:40:58.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.159+0000 7f4a4f1fc740 20 client.17301 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:58.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.159+0000 7f4a4f1fc740 1 --2- 172.21.15.90:0/2288881433 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55d3032c83a0 0x55d30375c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.159+0000 7f4a4f1fc740 4 client.17301 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:40:58.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.159+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/2288881433 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f4a3c01a5e0 con 0x55d3032c83a0 2024-04-25T11:40:58.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.161+0000 7f4a460e2640 1 --2- 172.21.15.90:0/2288881433 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55d3032c83a0 0x55d30375c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.166+0000 7f4a460e2640 1 --2- 172.21.15.90:0/2288881433 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55d3032c83a0 0x55d30375c4c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9e556ee3fc7327dc server_cookie=54abd2a46677338 in_seq=0 out_seq=0 2024-04-25T11:40:58.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.166+0000 7f4a29ffb640 10 client.17301 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:40:58.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.166+0000 7f4a29ffb640 1 -- 172.21.15.90:0/2288881433 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f4a3c01a5e0 con 0x55d3032c83a0 2024-04-25T11:40:58.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.166+0000 7f4a29ffb640 10 client.17301 handle_command_reply: tid=0 2024-04-25T11:40:58.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.166+0000 7f4a4f1fc740 1 client.17301 shutdown 2024-04-25T11:40:58.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.167+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/2288881433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d303334390 msgr2=0x55d3036e3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.167+0000 7f4a4f1fc740 1 --2- 172.21.15.90:0/2288881433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d303334390 0x55d3036e3c40 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f4a38006010 tx=0x7f4a380c80e0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.167+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/2288881433 shutdown_connections 2024-04-25T11:40:58.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.167+0000 7f4a4f1fc740 1 --2- 172.21.15.90:0/2288881433 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55d3032c83a0 0x55d30375c4c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.167+0000 7f4a29ffb640 0 client.17301 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:40:58.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.167+0000 7f4a4f1fc740 1 --2- 172.21.15.90:0/2288881433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d303334390 0x55d3036e3c40 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.167+0000 7f4a4f1fc740 1 --2- 172.21.15.90:0/2288881433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3033136b0 0x55d3036e3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.167+0000 7f4a4f1fc740 1 --2- 172.21.15.90:0/2288881433 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d30330b4d0 0x55d3036e31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.167+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/2288881433 >> 172.21.15.90:0/2288881433 conn(0x55d3033284e0 msgr2=0x55d3036df0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:58.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.167+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/2288881433 shutdown_connections 2024-04-25T11:40:58.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.167+0000 7f4a4f1fc740 1 -- 172.21.15.90:0/2288881433 wait complete. 2024-04-25T11:40:58.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.167+0000 7f4a4f1fc740 20 client.17301 trim_cache size 0 max 16384 2024-04-25T11:40:58.155 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:40:58.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.168+0000 7f4a48346640 1 -- 172.21.15.90:0/3859862619 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4a2c0afc60 msgr2=0x7f4a2c0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.168+0000 7f4a48346640 1 --2- 172.21.15.90:0/3859862619 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4a2c0afc60 0x7f4a2c0b2170 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4a3000a000 tx=0x7f4a3001a000 comp rx=0 tx=0).stop 2024-04-25T11:40:58.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.168+0000 7f4a48346640 1 -- 172.21.15.90:0/3859862619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a40108ec0 msgr2=0x7f4a4019ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.168+0000 7f4a48346640 1 --2- 172.21.15.90:0/3859862619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a40108ec0 0x7f4a4019ef60 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f4a38003bd0 tx=0x7f4a38003c00 comp rx=0 tx=0).stop 2024-04-25T11:40:58.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.169+0000 7f4a48346640 1 -- 172.21.15.90:0/3859862619 shutdown_connections 2024-04-25T11:40:58.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.169+0000 7f4a48346640 1 --2- 172.21.15.90:0/3859862619 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4a2c0afc60 0x7f4a2c0b2170 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.169+0000 7f4a48346640 1 --2- 172.21.15.90:0/3859862619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a40108ec0 0x7f4a4019ef60 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.169+0000 7f4a48346640 1 --2- 172.21.15.90:0/3859862619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a40108a20 0x7f4a4019aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.169+0000 7f4a48346640 1 --2- 172.21.15.90:0/3859862619 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4a40103180 0x7f4a4019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.169+0000 7f4a48346640 1 -- 172.21.15.90:0/3859862619 >> 172.21.15.90:0/3859862619 conn(0x7f4a40078110 msgr2=0x7f4a40106610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:58.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.169+0000 7f4a48346640 1 -- 172.21.15.90:0/3859862619 shutdown_connections 2024-04-25T11:40:58.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.169+0000 7f4a48346640 1 -- 172.21.15.90:0/3859862619 wait complete. 2024-04-25T11:40:58.183 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T11:40:58.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.341+0000 7f2e1c90d640 1 -- 172.21.15.90:0/1537376918 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e18106990 msgr2=0x7f2e18106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.341+0000 7f2e1c90d640 1 --2- 172.21.15.90:0/1537376918 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e18106990 0x7f2e18106dc0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c009900 tx=0x7f2e0c02a410 comp rx=0 tx=0).stop 2024-04-25T11:40:58.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.341+0000 7f2e1c90d640 1 -- 172.21.15.90:0/1537376918 shutdown_connections 2024-04-25T11:40:58.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.341+0000 7f2e1c90d640 1 --2- 172.21.15.90:0/1537376918 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2e18107450 0x7f2e1810ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.341+0000 7f2e1c90d640 1 --2- 172.21.15.90:0/1537376918 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e18106990 0x7f2e18106dc0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.341+0000 7f2e1c90d640 1 --2- 172.21.15.90:0/1537376918 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e18102400 0x7f2e18102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.341+0000 7f2e1c90d640 1 -- 172.21.15.90:0/1537376918 >> 172.21.15.90:0/1537376918 conn(0x7f2e18077880 msgr2=0x7f2e18077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:58.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.341+0000 7f2e1c90d640 1 -- 172.21.15.90:0/1537376918 shutdown_connections 2024-04-25T11:40:58.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.341+0000 7f2e1c90d640 1 -- 172.21.15.90:0/1537376918 wait complete. 2024-04-25T11:40:58.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.342+0000 7f2e1c90d640 1 Processor -- start 2024-04-25T11:40:58.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.342+0000 7f2e1c90d640 1 -- start start 2024-04-25T11:40:58.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.342+0000 7f2e1c90d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e18102400 0x7f2e1819e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.342+0000 7f2e1c90d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e18106990 0x7f2e1819e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.342+0000 7f2e1c90d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2e18107450 0x7f2e18199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.342+0000 7f2e1c90d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2e18112350 con 0x7f2e18102400 2024-04-25T11:40:58.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.342+0000 7f2e1c90d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2e181121d0 con 0x7f2e18107450 2024-04-25T11:40:58.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.342+0000 7f2e1c90d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2e181124d0 con 0x7f2e18106990 2024-04-25T11:40:58.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.343+0000 7f2e1659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e18102400 0x7f2e1819e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.343+0000 7f2e1659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e18102400 0x7f2e1819e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46162/0 (socket says 172.21.15.90:46162) 2024-04-25T11:40:58.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.343+0000 7f2e16d9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2e18107450 0x7f2e18199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.343+0000 7f2e1659c640 1 -- 172.21.15.90:0/4138126362 learned_addr learned my addr 172.21.15.90:0/4138126362 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:58.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.343+0000 7f2e15d9b640 1 --2- 172.21.15.90:0/4138126362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e18106990 0x7f2e1819e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.343+0000 7f2e1659c640 1 -- 172.21.15.90:0/4138126362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e18106990 msgr2=0x7f2e1819e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.343+0000 7f2e1659c640 1 --2- 172.21.15.90:0/4138126362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e18106990 0x7f2e1819e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.343+0000 7f2e1659c640 1 -- 172.21.15.90:0/4138126362 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2e18107450 msgr2=0x7f2e18199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.343+0000 7f2e1659c640 1 --2- 172.21.15.90:0/4138126362 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2e18107450 0x7f2e18199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.343+0000 7f2e1659c640 1 -- 172.21.15.90:0/4138126362 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e1819a2a0 con 0x7f2e18102400 2024-04-25T11:40:58.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.343+0000 7f2e16d9d640 1 --2- 172.21.15.90:0/4138126362 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2e18107450 0x7f2e18199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:40:58.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.343+0000 7f2e1659c640 1 --2- 172.21.15.90:0/4138126362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e18102400 0x7f2e1819e240 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f2e0800e9f0 tx=0x7f2e0800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:58.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.343+0000 7f2e077fe640 1 -- 172.21.15.90:0/4138126362 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0800cd80 con 0x7f2e18102400 2024-04-25T11:40:58.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.343+0000 7f2e1c90d640 1 -- 172.21.15.90:0/4138126362 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e1819a590 con 0x7f2e18102400 2024-04-25T11:40:58.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.343+0000 7f2e077fe640 1 -- 172.21.15.90:0/4138126362 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e080054d0 con 0x7f2e18102400 2024-04-25T11:40:58.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.343+0000 7f2e077fe640 1 -- 172.21.15.90:0/4138126362 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e08010690 con 0x7f2e18102400 2024-04-25T11:40:58.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.343+0000 7f2e1c90d640 1 -- 172.21.15.90:0/4138126362 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e18198b20 con 0x7f2e18102400 2024-04-25T11:40:58.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.347+0000 7f2e237c4740 1 Processor -- start 2024-04-25T11:40:58.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.347+0000 7f2e237c4740 1 -- start start 2024-04-25T11:40:58.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.347+0000 7f2e077fe640 1 -- 172.21.15.90:0/4138126362 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f2e08010830 con 0x7f2e18102400 2024-04-25T11:40:58.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.347+0000 7f2e237c4740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb987f9390 0x55eb98b9a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.347+0000 7f2e237c4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb987d04d0 0x55eb98b9ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.347+0000 7f2e237c4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb987d86b0 0x55eb98ba76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.347+0000 7f2e237c4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55eb98affd80 con 0x55eb987d86b0 2024-04-25T11:40:58.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.347+0000 7f2e237c4740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55eb98ab9a20 con 0x55eb987f9390 2024-04-25T11:40:58.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.347+0000 7f2e237c4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55eb98afbd40 con 0x55eb987d04d0 2024-04-25T11:40:58.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.347+0000 7f2e1659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb987d04d0 0x55eb98b9ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.347+0000 7f2e15d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb987d86b0 0x55eb98ba76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.347+0000 7f2e16d9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb987f9390 0x55eb98b9a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.348+0000 7f2e16d9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb987f9390 0x55eb98b9a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:40962/0 (socket says 172.21.15.90:40962) 2024-04-25T11:40:58.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.348+0000 7f2e1659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb987d04d0 0x55eb98b9ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:44298/0 (socket says 172.21.15.90:44298) 2024-04-25T11:40:58.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.348+0000 7f2e16d9d640 1 -- 172.21.15.90:0/1781978066 learned_addr learned my addr 172.21.15.90:0/1781978066 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:58.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.348+0000 7f2e16d9d640 1 -- 172.21.15.90:0/1781978066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb987d04d0 msgr2=0x55eb98b9ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.348+0000 7f2e16d9d640 1 --2- 172.21.15.90:0/1781978066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb987d04d0 0x55eb98b9ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.348+0000 7f2e16d9d640 1 -- 172.21.15.90:0/1781978066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb987d86b0 msgr2=0x55eb98ba76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.348+0000 7f2e16d9d640 1 --2- 172.21.15.90:0/1781978066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb987d86b0 0x55eb98ba76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.348+0000 7f2e16d9d640 1 -- 172.21.15.90:0/1781978066 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb98ac4c30 con 0x55eb987f9390 2024-04-25T11:40:58.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.348+0000 7f2e1659c640 1 --2- 172.21.15.90:0/1781978066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb987d04d0 0x55eb98b9ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:40:58.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.348+0000 7f2e16d9d640 1 --2- 172.21.15.90:0/1781978066 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb987f9390 0x55eb98b9a810 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2e10011790 tx=0x7f2e10011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e1eadbb67c691d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:58.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.348+0000 7f2e077fe640 1 --2- 172.21.15.90:0/4138126362 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f2e000afc60 0x7f2e000b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.348+0000 7f2e04ff9640 1 -- 172.21.15.90:0/1781978066 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e10003c30 con 0x55eb987f9390 2024-04-25T11:40:58.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.348+0000 7f2e1659c640 1 --2- 172.21.15.90:0/4138126362 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f2e000afc60 0x7f2e000b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.348+0000 7f2e04ff9640 1 -- 172.21.15.90:0/1781978066 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e1000fe50 con 0x55eb987f9390 2024-04-25T11:40:58.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.348+0000 7f2e04ff9640 1 -- 172.21.15.90:0/1781978066 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e100193e0 con 0x55eb987f9390 2024-04-25T11:40:58.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.349+0000 7f2e077fe640 1 -- 172.21.15.90:0/4138126362 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f2e080df4c0 con 0x7f2e18102400 2024-04-25T11:40:58.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.349+0000 7f2e237c4740 1 -- 172.21.15.90:0/1781978066 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb987f9390 msgr2=0x55eb98b9a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.349+0000 7f2e1659c640 1 --2- 172.21.15.90:0/4138126362 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f2e000afc60 0x7f2e000b2170 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2e080dcc60 tx=0x7f2e080e2f90 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:58.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.349+0000 7f2e237c4740 1 --2- 172.21.15.90:0/1781978066 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb987f9390 0x55eb98b9a810 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2e10011790 tx=0x7f2e10011c50 comp rx=0 tx=0).stop 2024-04-25T11:40:58.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.349+0000 7f2e237c4740 1 -- 172.21.15.90:0/1781978066 shutdown_connections 2024-04-25T11:40:58.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.349+0000 7f2e237c4740 1 --2- 172.21.15.90:0/1781978066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb987d86b0 0x55eb98ba76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.349+0000 7f2e237c4740 1 --2- 172.21.15.90:0/1781978066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb987d04d0 0x55eb98b9ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.349+0000 7f2e237c4740 1 --2- 172.21.15.90:0/1781978066 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb987f9390 0x55eb98b9a810 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.349+0000 7f2e237c4740 1 -- 172.21.15.90:0/1781978066 >> 172.21.15.90:0/1781978066 conn(0x55eb987ed4e0 msgr2=0x55eb98b96fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:58.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.349+0000 7f2e237c4740 1 -- 172.21.15.90:0/1781978066 shutdown_connections 2024-04-25T11:40:58.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.349+0000 7f2e237c4740 1 -- 172.21.15.90:0/1781978066 wait complete. 2024-04-25T11:40:58.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.350+0000 7f2e237c4740 1 -- start start 2024-04-25T11:40:58.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.350+0000 7f2e237c4740 1 Processor -- start 2024-04-25T11:40:58.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.350+0000 7f2e237c4740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb987d04d0 0x55eb98ba5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.350+0000 7f2e237c4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb987d86b0 0x55eb98ba5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.350+0000 7f2e237c4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb987f9390 0x55eb98bb1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.350+0000 7f2e237c4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55eb98ba7d90 con 0x55eb987f9390 2024-04-25T11:40:58.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.350+0000 7f2e237c4740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55eb98a86dc0 con 0x55eb987d04d0 2024-04-25T11:40:58.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.350+0000 7f2e237c4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55eb98ba7f10 con 0x55eb987d86b0 2024-04-25T11:40:58.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.350+0000 7f2e237c4740 10 client.0 _ll_register_callbacks cb 0x55eb98aa2600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:58.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.350+0000 7f2e16d9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb987d04d0 0x55eb98ba5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.350+0000 7f2e16d9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb987d04d0 0x55eb98ba5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:40974/0 (socket says 172.21.15.90:40974) 2024-04-25T11:40:58.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.350+0000 7f2e16d9d640 1 -- 172.21.15.90:0/3039065022 learned_addr learned my addr 172.21.15.90:0/3039065022 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:58.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.351+0000 7f2e16d9d640 1 --2- 172.21.15.90:0/3039065022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb987f9390 0x55eb98bb1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.351+0000 7f2e15d9b640 1 --2- 172.21.15.90:0/3039065022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb987d86b0 0x55eb98ba5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.351+0000 7f2e16d9d640 1 -- 172.21.15.90:0/3039065022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb987d86b0 msgr2=0x55eb98ba5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.351+0000 7f2e16d9d640 1 --2- 172.21.15.90:0/3039065022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb987d86b0 0x55eb98ba5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.351+0000 7f2e16d9d640 1 -- 172.21.15.90:0/3039065022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb987f9390 msgr2=0x55eb98bb1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.351+0000 7f2e16d9d640 1 --2- 172.21.15.90:0/3039065022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb987f9390 0x55eb98bb1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.351+0000 7f2e16d9d640 1 -- 172.21.15.90:0/3039065022 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb988113f0 con 0x55eb987d04d0 2024-04-25T11:40:58.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.351+0000 7f2e16d9d640 1 --2- 172.21.15.90:0/3039065022 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb987d04d0 0x55eb98ba5210 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2e100115a0 tx=0x7f2e10005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17bd30d1acabecb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:58.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.351+0000 7f2dfdffb640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:40:58.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.351+0000 7f2dfdffb640 1 -- 172.21.15.90:0/3039065022 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1002d060 con 0x55eb987d04d0 2024-04-25T11:40:58.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.351+0000 7f2dfdffb640 1 -- 172.21.15.90:0/3039065022 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f2e10011050 con 0x55eb987d04d0 2024-04-25T11:40:58.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.351+0000 7f2e237c4740 1 -- 172.21.15.90:0/3039065022 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e1000fe50 con 0x55eb987d04d0 2024-04-25T11:40:58.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.351+0000 7f2dfdffb640 1 -- 172.21.15.90:0/3039065022 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2e10011250 con 0x55eb987d04d0 2024-04-25T11:40:58.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.351+0000 7f2e237c4740 10 client.26375 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:58.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.351+0000 7f2e237c4740 1 -- 172.21.15.90:0/3039065022 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55eb988db7d0 con 0x55eb987d04d0 2024-04-25T11:40:58.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.352+0000 7f2dfdffb640 1 -- 172.21.15.90:0/3039065022 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e1000fe50 con 0x55eb987d04d0 2024-04-25T11:40:58.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.352+0000 7f2dfdffb640 1 -- 172.21.15.90:0/3039065022 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f2e10018070 con 0x55eb987d04d0 2024-04-25T11:40:58.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.352+0000 7f2e237c4740 10 client.26375 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:58.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.352+0000 7f2e237c4740 10 client.26375 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:40:58.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.352+0000 7f2e237c4740 20 client.26375 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:58.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.352+0000 7f2e237c4740 1 --2- 172.21.15.90:0/3039065022 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55eb987b7f60 0x55eb98bb74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.352+0000 7f2e237c4740 4 client.26375 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:40:58.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.352+0000 7f2e237c4740 1 -- 172.21.15.90:0/3039065022 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb98b0d990 con 0x55eb987b7f60 2024-04-25T11:40:58.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.352+0000 7f2e15d9b640 1 --2- 172.21.15.90:0/3039065022 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55eb987b7f60 0x55eb98bb74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.353+0000 7f2dfdffb640 10 client.26375 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:40:58.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.353+0000 7f2e15d9b640 1 --2- 172.21.15.90:0/3039065022 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55eb987b7f60 0x55eb98bb74f0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=795db34287d29d75 server_cookie=8926a5c86c03ddaf in_seq=0 out_seq=0 2024-04-25T11:40:58.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.353+0000 7f2dfdffb640 1 -- 172.21.15.90:0/3039065022 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55eb98b0d990 con 0x55eb987b7f60 2024-04-25T11:40:58.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.353+0000 7f2dfdffb640 10 client.26375 handle_command_reply: tid=0 2024-04-25T11:40:58.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.353+0000 7f2e237c4740 1 client.26375 shutdown 2024-04-25T11:40:58.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.353+0000 7f2e237c4740 1 -- 172.21.15.90:0/3039065022 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb987d04d0 msgr2=0x55eb98ba5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.353+0000 7f2e237c4740 1 --2- 172.21.15.90:0/3039065022 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb987d04d0 0x55eb98ba5210 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2e100115a0 tx=0x7f2e10005bd0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.354+0000 7f2e237c4740 1 -- 172.21.15.90:0/3039065022 shutdown_connections 2024-04-25T11:40:58.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.354+0000 7f2e237c4740 1 --2- 172.21.15.90:0/3039065022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb987f9390 0x55eb98bb1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.354+0000 7f2e237c4740 1 --2- 172.21.15.90:0/3039065022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb987d86b0 0x55eb98ba5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.354+0000 7f2e237c4740 1 --2- 172.21.15.90:0/3039065022 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55eb987b7f60 0x55eb98bb74f0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.354+0000 7f2e237c4740 1 --2- 172.21.15.90:0/3039065022 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb987d04d0 0x55eb98ba5210 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.354+0000 7f2dfdffb640 0 client.26375 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:40:58.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.354+0000 7f2e237c4740 1 -- 172.21.15.90:0/3039065022 >> 172.21.15.90:0/3039065022 conn(0x55eb987ed4e0 msgr2=0x55eb98b9c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:58.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.354+0000 7f2e237c4740 1 -- 172.21.15.90:0/3039065022 shutdown_connections 2024-04-25T11:40:58.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.354+0000 7f2e237c4740 1 -- 172.21.15.90:0/3039065022 wait complete. 2024-04-25T11:40:58.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.354+0000 7f2e237c4740 20 client.26375 trim_cache size 0 max 16384 2024-04-25T11:40:58.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.370+0000 7f2e237c4740 1 Processor -- start 2024-04-25T11:40:58.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.370+0000 7f2e237c4740 1 -- start start 2024-04-25T11:40:58.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.370+0000 7f2e237c4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb987d04d0 0x55eb98ba7150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.370+0000 7f2e237c4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb987d86b0 0x55eb98b95d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.370+0000 7f2e237c4740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb987f9390 0x55eb98b96260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.370+0000 7f2e237c4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2e181ada80 con 0x55eb987d04d0 2024-04-25T11:40:58.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.370+0000 7f2e237c4740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2e181af5c0 con 0x55eb987f9390 2024-04-25T11:40:58.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.370+0000 7f2e237c4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2e181ae150 con 0x55eb987d86b0 2024-04-25T11:40:58.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.371+0000 7f2e16d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb987d04d0 0x55eb98ba7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.371+0000 7f2e16d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb987d04d0 0x55eb98ba7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46182/0 (socket says 172.21.15.90:46182) 2024-04-25T11:40:58.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.371+0000 7f2e16d9d640 1 -- 172.21.15.90:0/3979608852 learned_addr learned my addr 172.21.15.90:0/3979608852 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:58.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.371+0000 7f2e16d9d640 1 --2- 172.21.15.90:0/3979608852 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb987f9390 0x55eb98b96260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.371+0000 7f2e15d9b640 1 --2- 172.21.15.90:0/3979608852 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb987d86b0 0x55eb98b95d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.371+0000 7f2e16d9d640 1 -- 172.21.15.90:0/3979608852 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb987d86b0 msgr2=0x55eb98b95d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.371+0000 7f2e16d9d640 1 --2- 172.21.15.90:0/3979608852 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb987d86b0 0x55eb98b95d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.371+0000 7f2e16d9d640 1 -- 172.21.15.90:0/3979608852 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb987f9390 msgr2=0x55eb98b96260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.371+0000 7f2e16d9d640 1 --2- 172.21.15.90:0/3979608852 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb987f9390 0x55eb98b96260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.371+0000 7f2e16d9d640 1 -- 172.21.15.90:0/3979608852 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb98bb7860 con 0x55eb987d04d0 2024-04-25T11:40:58.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.371+0000 7f2e16d9d640 1 --2- 172.21.15.90:0/3979608852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb987d04d0 0x55eb98ba7150 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f2e10005d70 tx=0x7f2e10011680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d16a234cd8c2028e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:58.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.371+0000 7f2dfffff640 1 -- 172.21.15.90:0/3979608852 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1002d070 con 0x55eb987d04d0 2024-04-25T11:40:58.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.371+0000 7f2dfffff640 1 -- 172.21.15.90:0/3979608852 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e100113f0 con 0x55eb987d04d0 2024-04-25T11:40:58.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.371+0000 7f2dfffff640 1 -- 172.21.15.90:0/3979608852 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e10004870 con 0x55eb987d04d0 2024-04-25T11:40:58.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.372+0000 7f2e237c4740 1 -- 172.21.15.90:0/3979608852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb987d04d0 msgr2=0x55eb98ba7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.372+0000 7f2e237c4740 1 --2- 172.21.15.90:0/3979608852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb987d04d0 0x55eb98ba7150 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f2e10005d70 tx=0x7f2e10011680 comp rx=0 tx=0).stop 2024-04-25T11:40:58.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.372+0000 7f2e237c4740 1 -- 172.21.15.90:0/3979608852 shutdown_connections 2024-04-25T11:40:58.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.372+0000 7f2e237c4740 1 --2- 172.21.15.90:0/3979608852 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb987f9390 0x55eb98b96260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.372+0000 7f2e237c4740 1 --2- 172.21.15.90:0/3979608852 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb987d86b0 0x55eb98b95d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.372+0000 7f2e237c4740 1 --2- 172.21.15.90:0/3979608852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb987d04d0 0x55eb98ba7150 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.372+0000 7f2e237c4740 1 -- 172.21.15.90:0/3979608852 >> 172.21.15.90:0/3979608852 conn(0x55eb987ed4e0 msgr2=0x55eb98b95610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:58.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.372+0000 7f2e237c4740 1 -- 172.21.15.90:0/3979608852 shutdown_connections 2024-04-25T11:40:58.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.372+0000 7f2e237c4740 1 -- 172.21.15.90:0/3979608852 wait complete. 2024-04-25T11:40:58.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.373+0000 7f2e237c4740 1 -- start start 2024-04-25T11:40:58.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.373+0000 7f2e237c4740 1 Processor -- start 2024-04-25T11:40:58.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.373+0000 7f2e237c4740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb987d04d0 0x55eb98ba87f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.373+0000 7f2e237c4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb987d86b0 0x55eb98ba8d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.373+0000 7f2e237c4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb987f9390 0x55eb98c17600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.373+0000 7f2e237c4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2e181aeef0 con 0x55eb987f9390 2024-04-25T11:40:58.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.373+0000 7f2e237c4740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2e181ae820 con 0x55eb987d04d0 2024-04-25T11:40:58.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.373+0000 7f2e237c4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2e181aba60 con 0x55eb987d86b0 2024-04-25T11:40:58.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.373+0000 7f2e237c4740 10 client.0 _ll_register_callbacks cb 0x7f2e08005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:58.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.373+0000 7f2e16d9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb987d04d0 0x55eb98ba87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.373+0000 7f2e16d9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb987d04d0 0x55eb98ba87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:40996/0 (socket says 172.21.15.90:40996) 2024-04-25T11:40:58.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.373+0000 7f2e16d9d640 1 -- 172.21.15.90:0/1434191281 learned_addr learned my addr 172.21.15.90:0/1434191281 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:58.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.373+0000 7f2e15d9b640 1 --2- 172.21.15.90:0/1434191281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb987d86b0 0x55eb98ba8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.373+0000 7f2e16d9d640 1 --2- 172.21.15.90:0/1434191281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb987f9390 0x55eb98c17600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.374+0000 7f2e15d9b640 1 -- 172.21.15.90:0/1434191281 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb987d04d0 msgr2=0x55eb98ba87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.374+0000 7f2e15d9b640 1 --2- 172.21.15.90:0/1434191281 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb987d04d0 0x55eb98ba87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.374+0000 7f2e15d9b640 1 -- 172.21.15.90:0/1434191281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb987f9390 msgr2=0x55eb98c17600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.374+0000 7f2e15d9b640 1 --2- 172.21.15.90:0/1434191281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb987f9390 0x55eb98c17600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.374+0000 7f2e15d9b640 1 -- 172.21.15.90:0/1434191281 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb98c17bc0 con 0x55eb987d86b0 2024-04-25T11:40:58.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.374+0000 7f2e16d9d640 1 --2- 172.21.15.90:0/1434191281 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb987d04d0 0x55eb98ba87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:40:58.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.374+0000 7f2e15d9b640 1 --2- 172.21.15.90:0/1434191281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb987d86b0 0x55eb98ba8d50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c006010 tx=0x7f2e0c032f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d8cb45f4afacb22 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:58.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.374+0000 7f2dfdffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:40:58.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.374+0000 7f2dfdffb640 1 -- 172.21.15.90:0/1434191281 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c04b070 con 0x55eb987d86b0 2024-04-25T11:40:58.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.374+0000 7f2e237c4740 1 -- 172.21.15.90:0/1434191281 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e100113f0 con 0x55eb987d86b0 2024-04-25T11:40:58.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.374+0000 7f2dfdffb640 1 -- 172.21.15.90:0/1434191281 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f2e0c005040 con 0x55eb987d86b0 2024-04-25T11:40:58.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.374+0000 7f2dfdffb640 1 -- 172.21.15.90:0/1434191281 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2e0c057050 con 0x55eb987d86b0 2024-04-25T11:40:58.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.374+0000 7f2e237c4740 10 client.26326 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:58.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.374+0000 7f2e237c4740 1 -- 172.21.15.90:0/1434191281 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55eb98c1efe0 con 0x55eb987d86b0 2024-04-25T11:40:58.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.374+0000 7f2dfdffb640 1 -- 172.21.15.90:0/1434191281 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e100113f0 con 0x55eb987d86b0 2024-04-25T11:40:58.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.375+0000 7f2dfdffb640 1 -- 172.21.15.90:0/1434191281 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f2e0c004ce0 con 0x55eb987d86b0 2024-04-25T11:40:58.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.375+0000 7f2e237c4740 10 client.26326 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:58.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.375+0000 7f2e237c4740 10 client.26326 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:40:58.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.375+0000 7f2e237c4740 20 client.26326 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:58.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.375+0000 7f2e237c4740 1 --2- 172.21.15.90:0/1434191281 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55eb9878d3a0 0x55eb98c21780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.375+0000 7f2e237c4740 4 client.26326 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:40:58.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.375+0000 7f2e237c4740 1 -- 172.21.15.90:0/1434191281 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55eb98b07800 con 0x55eb9878d3a0 2024-04-25T11:40:58.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.375+0000 7f2e16d9d640 1 --2- 172.21.15.90:0/1434191281 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55eb9878d3a0 0x55eb98c21780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.375+0000 7f2dfdffb640 10 client.26326 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:40:58.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.376+0000 7f2e16d9d640 1 --2- 172.21.15.90:0/1434191281 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55eb9878d3a0 0x55eb98c21780 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e75942138ca0d394 server_cookie=259d64b9ea0af5a6 in_seq=0 out_seq=0 2024-04-25T11:40:58.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.376+0000 7f2dfdffb640 1 -- 172.21.15.90:0/1434191281 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55eb98b07800 con 0x55eb9878d3a0 2024-04-25T11:40:58.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.376+0000 7f2dfdffb640 10 client.26326 handle_command_reply: tid=0 2024-04-25T11:40:58.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.376+0000 7f2e237c4740 1 client.26326 shutdown 2024-04-25T11:40:58.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.376+0000 7f2e237c4740 1 -- 172.21.15.90:0/1434191281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb987d86b0 msgr2=0x55eb98ba8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.376+0000 7f2e237c4740 1 --2- 172.21.15.90:0/1434191281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb987d86b0 0x55eb98ba8d50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c006010 tx=0x7f2e0c032f40 comp rx=0 tx=0).stop 2024-04-25T11:40:58.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.376+0000 7f2e237c4740 1 -- 172.21.15.90:0/1434191281 shutdown_connections 2024-04-25T11:40:58.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.376+0000 7f2e237c4740 1 --2- 172.21.15.90:0/1434191281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb987f9390 0x55eb98c17600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.376+0000 7f2e237c4740 1 --2- 172.21.15.90:0/1434191281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb987d86b0 0x55eb98ba8d50 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.376+0000 7f2e237c4740 1 --2- 172.21.15.90:0/1434191281 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55eb9878d3a0 0x55eb98c21780 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.376+0000 7f2e237c4740 1 --2- 172.21.15.90:0/1434191281 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb987d04d0 0x55eb98ba87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.376+0000 7f2e237c4740 1 -- 172.21.15.90:0/1434191281 >> 172.21.15.90:0/1434191281 conn(0x55eb987ed4e0 msgr2=0x55eb98ba64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:58.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.376+0000 7f2dfdffb640 0 client.26326 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:40:58.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.376+0000 7f2e237c4740 1 -- 172.21.15.90:0/1434191281 shutdown_connections 2024-04-25T11:40:58.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.376+0000 7f2e237c4740 1 -- 172.21.15.90:0/1434191281 wait complete. 2024-04-25T11:40:58.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.376+0000 7f2e237c4740 20 client.26326 trim_cache size 0 max 16384 2024-04-25T11:40:58.372 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:40:58.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.377+0000 7f2e1c90d640 1 -- 172.21.15.90:0/4138126362 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f2e000afc60 msgr2=0x7f2e000b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.377+0000 7f2e1c90d640 1 --2- 172.21.15.90:0/4138126362 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f2e000afc60 0x7f2e000b2170 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2e080dcc60 tx=0x7f2e080e2f90 comp rx=0 tx=0).stop 2024-04-25T11:40:58.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.377+0000 7f2e1c90d640 1 -- 172.21.15.90:0/4138126362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e18102400 msgr2=0x7f2e1819e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.377+0000 7f2e1c90d640 1 --2- 172.21.15.90:0/4138126362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e18102400 0x7f2e1819e240 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f2e0800e9f0 tx=0x7f2e0800edf0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.377+0000 7f2e1c90d640 1 -- 172.21.15.90:0/4138126362 shutdown_connections 2024-04-25T11:40:58.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.377+0000 7f2e1c90d640 1 --2- 172.21.15.90:0/4138126362 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f2e000afc60 0x7f2e000b2170 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.377+0000 7f2e1c90d640 1 --2- 172.21.15.90:0/4138126362 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2e18107450 0x7f2e18199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.377+0000 7f2e1c90d640 1 --2- 172.21.15.90:0/4138126362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e18106990 0x7f2e1819e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.377+0000 7f2e1c90d640 1 --2- 172.21.15.90:0/4138126362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e18102400 0x7f2e1819e240 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.377+0000 7f2e1c90d640 1 -- 172.21.15.90:0/4138126362 >> 172.21.15.90:0/4138126362 conn(0x7f2e18077880 msgr2=0x7f2e181008c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:58.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.377+0000 7f2e1c90d640 1 -- 172.21.15.90:0/4138126362 shutdown_connections 2024-04-25T11:40:58.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.377+0000 7f2e1c90d640 1 -- 172.21.15.90:0/4138126362 wait complete. 2024-04-25T11:40:58.377 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T11:40:58.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.521+0000 7f8aa8e8c640 1 -- 172.21.15.90:0/444652211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8aa4109b30 msgr2=0x7f8aa4103720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.521+0000 7f8aa8e8c640 1 --2- 172.21.15.90:0/444652211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8aa4109b30 0x7f8aa4103720 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f8a94009860 tx=0x7f8a9402a340 comp rx=0 tx=0).stop 2024-04-25T11:40:58.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.521+0000 7f8aa8e8c640 1 -- 172.21.15.90:0/444652211 shutdown_connections 2024-04-25T11:40:58.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.521+0000 7f8aa8e8c640 1 --2- 172.21.15.90:0/444652211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8aa41042d0 0x7f8aa4110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.521+0000 7f8aa8e8c640 1 --2- 172.21.15.90:0/444652211 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8aa4109fd0 0x7f8aa4103c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.521+0000 7f8aa8e8c640 1 --2- 172.21.15.90:0/444652211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8aa4109b30 0x7f8aa4103720 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.521+0000 7f8aa8e8c640 1 -- 172.21.15.90:0/444652211 >> 172.21.15.90:0/444652211 conn(0x7f8aa4079280 msgr2=0x7f8aa40796c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:58.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.521+0000 7f8aa8e8c640 1 -- 172.21.15.90:0/444652211 shutdown_connections 2024-04-25T11:40:58.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.521+0000 7f8aa8e8c640 1 -- 172.21.15.90:0/444652211 wait complete. 2024-04-25T11:40:58.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.523+0000 7f8aa8e8c640 1 Processor -- start 2024-04-25T11:40:58.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.523+0000 7f8aa8e8c640 1 -- start start 2024-04-25T11:40:58.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.523+0000 7f8aa8e8c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8aa41042d0 0x7f8aa419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.523+0000 7f8aa8e8c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8aa4109b30 0x7f8aa419acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.523+0000 7f8aa8e8c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8aa4109fd0 0x7f8aa419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.523+0000 7f8aa8e8c640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa4113d20 con 0x7f8aa4109b30 2024-04-25T11:40:58.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.523+0000 7f8aa8e8c640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa4113ba0 con 0x7f8aa4109fd0 2024-04-25T11:40:58.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.523+0000 7f8aa8e8c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa4113ea0 con 0x7f8aa41042d0 2024-04-25T11:40:58.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.523+0000 7f8a9bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8aa4109b30 0x7f8aa419acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.523+0000 7f8a9bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8aa4109b30 0x7f8aa419acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46196/0 (socket says 172.21.15.90:46196) 2024-04-25T11:40:58.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.523+0000 7f8aa259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8aa41042d0 0x7f8aa419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.523+0000 7f8a9bfff640 1 -- 172.21.15.90:0/3915526408 learned_addr learned my addr 172.21.15.90:0/3915526408 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:58.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.523+0000 7f8aa2d9d640 1 --2- 172.21.15.90:0/3915526408 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8aa4109fd0 0x7f8aa419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.524+0000 7f8a9bfff640 1 -- 172.21.15.90:0/3915526408 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8aa41042d0 msgr2=0x7f8aa419a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.524+0000 7f8a9bfff640 1 --2- 172.21.15.90:0/3915526408 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8aa41042d0 0x7f8aa419a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.524+0000 7f8a9bfff640 1 -- 172.21.15.90:0/3915526408 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8aa4109fd0 msgr2=0x7f8aa419ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.524+0000 7f8a9bfff640 1 --2- 172.21.15.90:0/3915526408 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8aa4109fd0 0x7f8aa419ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.524+0000 7f8a9bfff640 1 -- 172.21.15.90:0/3915526408 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aa419f5c0 con 0x7f8aa4109b30 2024-04-25T11:40:58.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.524+0000 7f8aa2d9d640 1 --2- 172.21.15.90:0/3915526408 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8aa4109fd0 0x7f8aa419ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:40:58.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.524+0000 7f8a9bfff640 1 --2- 172.21.15.90:0/3915526408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8aa4109b30 0x7f8aa419acc0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c00e980 tx=0x7f8a8c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:58.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.524+0000 7f8a9b7fe640 1 -- 172.21.15.90:0/3915526408 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c00cd80 con 0x7f8aa4109b30 2024-04-25T11:40:58.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.524+0000 7f8aa8e8c640 1 -- 172.21.15.90:0/3915526408 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8aa419f8b0 con 0x7f8aa4109b30 2024-04-25T11:40:58.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.524+0000 7f8aa8e8c640 1 -- 172.21.15.90:0/3915526408 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8aa41a7230 con 0x7f8aa4109b30 2024-04-25T11:40:58.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.524+0000 7f8a9b7fe640 1 -- 172.21.15.90:0/3915526408 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a8c0108e0 con 0x7f8aa4109b30 2024-04-25T11:40:58.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.524+0000 7f8a9b7fe640 1 -- 172.21.15.90:0/3915526408 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c0188f0 con 0x7f8aa4109b30 2024-04-25T11:40:58.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.527+0000 7f8aafe10740 1 Processor -- start 2024-04-25T11:40:58.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.528+0000 7f8aafe10740 1 -- start start 2024-04-25T11:40:58.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.528+0000 7f8aafe10740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f68b3eb390 0x55f68b78c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.528+0000 7f8aafe10740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f68b3c24d0 0x55f68b78cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.528+0000 7f8aafe10740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f68b3ca6b0 0x55f68b78d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.528+0000 7f8aafe10740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f68b6f1d80 con 0x55f68b3eb390 2024-04-25T11:40:58.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.528+0000 7f8aafe10740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f68b6aba20 con 0x55f68b3ca6b0 2024-04-25T11:40:58.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.528+0000 7f8aafe10740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f68b6edd40 con 0x55f68b3c24d0 2024-04-25T11:40:58.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.528+0000 7f8aa2d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f68b3eb390 0x55f68b78c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.528+0000 7f8aa2d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f68b3eb390 0x55f68b78c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46204/0 (socket says 172.21.15.90:46204) 2024-04-25T11:40:58.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.528+0000 7f8aa2d9d640 1 -- 172.21.15.90:0/3811875703 learned_addr learned my addr 172.21.15.90:0/3811875703 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:58.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.529+0000 7f8aa2d9d640 1 -- 172.21.15.90:0/3811875703 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f68b3c24d0 msgr2=0x55f68b78cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.529+0000 7f8aa2d9d640 1 --2- 172.21.15.90:0/3811875703 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f68b3c24d0 0x55f68b78cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.529+0000 7f8aa2d9d640 1 -- 172.21.15.90:0/3811875703 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f68b3ca6b0 msgr2=0x55f68b78d100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:40:58.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.529+0000 7f8aa2d9d640 1 --2- 172.21.15.90:0/3811875703 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f68b3ca6b0 0x55f68b78d100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.529+0000 7f8aa2d9d640 1 -- 172.21.15.90:0/3811875703 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f68b6b6c30 con 0x55f68b3eb390 2024-04-25T11:40:58.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.529+0000 7f8aa2d9d640 1 --2- 172.21.15.90:0/3811875703 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f68b3eb390 0x55f68b78c680 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c0107e0 tx=0x7f8a9c010ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9037fe9157766a09 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:58.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.530+0000 7f8a9b7fe640 1 -- 172.21.15.90:0/3915526408 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f8a8c01f020 con 0x7f8aa4109b30 2024-04-25T11:40:58.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.530+0000 7f8a98ff9640 1 -- 172.21.15.90:0/3811875703 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c003b70 con 0x55f68b3eb390 2024-04-25T11:40:58.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.531+0000 7f8a98ff9640 1 -- 172.21.15.90:0/3811875703 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a9c00ede0 con 0x55f68b3eb390 2024-04-25T11:40:58.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.531+0000 7f8a98ff9640 1 -- 172.21.15.90:0/3811875703 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c004770 con 0x55f68b3eb390 2024-04-25T11:40:58.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.531+0000 7f8a9b7fe640 1 --2- 172.21.15.90:0/3915526408 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8a900afce0 0x7f8a900b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.532+0000 7f8a9b7fe640 1 -- 172.21.15.90:0/3915526408 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f8a8c0def60 con 0x7f8aa4109b30 2024-04-25T11:40:58.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.532+0000 7f8aa259c640 1 --2- 172.21.15.90:0/3915526408 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8a900afce0 0x7f8a900b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.534+0000 7f8aafe10740 1 -- 172.21.15.90:0/3811875703 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f68b3eb390 msgr2=0x55f68b78c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.534+0000 7f8aa259c640 1 --2- 172.21.15.90:0/3915526408 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8a900afce0 0x7f8a900b21f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8a94009830 tx=0x7f8a94000bc0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:58.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.534+0000 7f8aafe10740 1 --2- 172.21.15.90:0/3811875703 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f68b3eb390 0x55f68b78c680 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c0107e0 tx=0x7f8a9c010ca0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.534+0000 7f8aafe10740 1 -- 172.21.15.90:0/3811875703 shutdown_connections 2024-04-25T11:40:58.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.534+0000 7f8aafe10740 1 --2- 172.21.15.90:0/3811875703 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f68b3ca6b0 0x55f68b78d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.534+0000 7f8aafe10740 1 --2- 172.21.15.90:0/3811875703 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f68b3c24d0 0x55f68b78cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.534+0000 7f8aafe10740 1 --2- 172.21.15.90:0/3811875703 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f68b3eb390 0x55f68b78c680 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.534+0000 7f8aafe10740 1 -- 172.21.15.90:0/3811875703 >> 172.21.15.90:0/3811875703 conn(0x55f68b3df4e0 msgr2=0x55f68b788fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:58.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.534+0000 7f8aafe10740 1 -- 172.21.15.90:0/3811875703 shutdown_connections 2024-04-25T11:40:58.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.535+0000 7f8aafe10740 1 -- 172.21.15.90:0/3811875703 wait complete. 2024-04-25T11:40:58.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.536+0000 7f8aafe10740 1 -- start start 2024-04-25T11:40:58.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.536+0000 7f8aafe10740 1 Processor -- start 2024-04-25T11:40:58.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.537+0000 7f8aafe10740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f68b3c24d0 0x55f68b79f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.538+0000 7f8aafe10740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f68b3ca6b0 0x55f68b79fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.538+0000 7f8aa259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f68b3c24d0 0x55f68b79f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.538+0000 7f8aa259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f68b3c24d0 0x55f68b79f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46220/0 (socket says 172.21.15.90:46220) 2024-04-25T11:40:58.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.538+0000 7f8aa259c640 1 -- 172.21.15.90:0/904252203 learned_addr learned my addr 172.21.15.90:0/904252203 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:58.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.538+0000 7f8aafe10740 1 --2- 172.21.15.90:0/904252203 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f68b3eb390 0x55f68b7aba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.538+0000 7f8aafe10740 1 -- 172.21.15.90:0/904252203 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f68b7a2580 con 0x55f68b3c24d0 2024-04-25T11:40:58.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.538+0000 7f8aafe10740 1 -- 172.21.15.90:0/904252203 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f68b678dc0 con 0x55f68b3eb390 2024-04-25T11:40:58.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.538+0000 7f8aafe10740 1 -- 172.21.15.90:0/904252203 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f68b7a2700 con 0x55f68b3ca6b0 2024-04-25T11:40:58.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.538+0000 7f8aafe10740 10 client.0 _ll_register_callbacks cb 0x55f68b694600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:58.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.539+0000 7f8a9bfff640 1 --2- 172.21.15.90:0/904252203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f68b3ca6b0 0x55f68b79fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.539+0000 7f8aa259c640 1 -- 172.21.15.90:0/904252203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f68b3ca6b0 msgr2=0x55f68b79fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.539+0000 7f8aa259c640 1 --2- 172.21.15.90:0/904252203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f68b3ca6b0 0x55f68b79fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.539+0000 7f8aa259c640 1 -- 172.21.15.90:0/904252203 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f68b3eb390 msgr2=0x55f68b7aba90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:40:58.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.539+0000 7f8aa259c640 1 --2- 172.21.15.90:0/904252203 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f68b3eb390 0x55f68b7aba90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.539+0000 7f8aa259c640 1 -- 172.21.15.90:0/904252203 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f68b4033f0 con 0x55f68b3c24d0 2024-04-25T11:40:58.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.539+0000 7f8a89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:40:58.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.540+0000 7f8aa259c640 1 --2- 172.21.15.90:0/904252203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f68b3c24d0 0x55f68b79f850 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f8a94009230 tx=0x7f8a9402a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a3e6c5f87819545 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:58.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.541+0000 7f8a89ffb640 1 -- 172.21.15.90:0/904252203 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9404c020 con 0x55f68b3c24d0 2024-04-25T11:40:58.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.542+0000 7f8a89ffb640 1 -- 172.21.15.90:0/904252203 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f8a940035f0 con 0x55f68b3c24d0 2024-04-25T11:40:58.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.542+0000 7f8aafe10740 1 -- 172.21.15.90:0/904252203 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a9c00ede0 con 0x55f68b3c24d0 2024-04-25T11:40:58.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.542+0000 7f8a89ffb640 1 -- 172.21.15.90:0/904252203 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8a9404c020 con 0x55f68b3c24d0 2024-04-25T11:40:58.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.542+0000 7f8aafe10740 10 client.17331 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:58.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.542+0000 7f8aafe10740 1 -- 172.21.15.90:0/904252203 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f68b4cd7d0 con 0x55f68b3c24d0 2024-04-25T11:40:58.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.543+0000 7f8a89ffb640 1 -- 172.21.15.90:0/904252203 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a9c00ede0 con 0x55f68b3c24d0 2024-04-25T11:40:58.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.543+0000 7f8a89ffb640 1 -- 172.21.15.90:0/904252203 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f8a94032950 con 0x55f68b3c24d0 2024-04-25T11:40:58.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.545+0000 7f8aafe10740 10 client.17331 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:58.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.545+0000 7f8aafe10740 10 client.17331 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:40:58.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.545+0000 7f8aafe10740 20 client.17331 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:58.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.545+0000 7f8aafe10740 1 --2- 172.21.15.90:0/904252203 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f68b3a9f60 0x55f68b7b1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.545+0000 7f8aafe10740 4 client.17331 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:40:58.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.545+0000 7f8aafe10740 1 -- 172.21.15.90:0/904252203 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f68b6ff990 con 0x55f68b3a9f60 2024-04-25T11:40:58.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.550+0000 7f8a9bfff640 1 --2- 172.21.15.90:0/904252203 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f68b3a9f60 0x55f68b7b1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.564+0000 7f8a9bfff640 1 --2- 172.21.15.90:0/904252203 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f68b3a9f60 0x55f68b7b1d70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c1f32d63808860b2 server_cookie=643ef75ab2c87ebb in_seq=0 out_seq=0 2024-04-25T11:40:58.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.564+0000 7f8a89ffb640 10 client.17331 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:40:58.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.576+0000 7f8a89ffb640 1 -- 172.21.15.90:0/904252203 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f68b6ff990 con 0x55f68b3a9f60 2024-04-25T11:40:58.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.576+0000 7f8a89ffb640 10 client.17331 handle_command_reply: tid=0 2024-04-25T11:40:58.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.576+0000 7f8aafe10740 1 client.17331 shutdown 2024-04-25T11:40:58.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.578+0000 7f8aafe10740 1 -- 172.21.15.90:0/904252203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f68b3c24d0 msgr2=0x55f68b79f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.578+0000 7f8aafe10740 1 --2- 172.21.15.90:0/904252203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f68b3c24d0 0x55f68b79f850 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f8a94009230 tx=0x7f8a9402a340 comp rx=0 tx=0).stop 2024-04-25T11:40:58.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.578+0000 7f8aafe10740 1 -- 172.21.15.90:0/904252203 shutdown_connections 2024-04-25T11:40:58.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.578+0000 7f8aafe10740 1 --2- 172.21.15.90:0/904252203 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f68b3a9f60 0x55f68b7b1d70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.579+0000 7f8a89ffb640 0 client.17331 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:40:58.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.579+0000 7f8aafe10740 1 --2- 172.21.15.90:0/904252203 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f68b3eb390 0x55f68b7aba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.579+0000 7f8aafe10740 1 --2- 172.21.15.90:0/904252203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f68b3ca6b0 0x55f68b79fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.579+0000 7f8aafe10740 1 --2- 172.21.15.90:0/904252203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f68b3c24d0 0x55f68b79f850 secure :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f8a94009230 tx=0x7f8a9402a340 comp rx=0 tx=0).stop 2024-04-25T11:40:58.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.579+0000 7f8aafe10740 1 -- 172.21.15.90:0/904252203 >> 172.21.15.90:0/904252203 conn(0x55f68b3df4e0 msgr2=0x55f68b787610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:58.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.586+0000 7f8aafe10740 1 -- 172.21.15.90:0/904252203 shutdown_connections 2024-04-25T11:40:58.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.586+0000 7f8aafe10740 1 -- 172.21.15.90:0/904252203 wait complete. 2024-04-25T11:40:58.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.586+0000 7f8aafe10740 20 client.17331 trim_cache size 0 max 16384 2024-04-25T11:40:58.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.626+0000 7f8aafe10740 1 Processor -- start 2024-04-25T11:40:58.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.626+0000 7f8aafe10740 1 -- start start 2024-04-25T11:40:58.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.627+0000 7f8aafe10740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f68b3a9f60 0x55f68b789160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.627+0000 7f8aafe10740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f68b3ca6b0 0x55f68b7ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.627+0000 7f8aafe10740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f68b3eb390 0x55f68b7b7530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.627+0000 7f8aafe10740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8a8400ea00 con 0x55f68b3a9f60 2024-04-25T11:40:58.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.627+0000 7f8aafe10740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8a84010540 con 0x55f68b3eb390 2024-04-25T11:40:58.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.627+0000 7f8aafe10740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8a8400f0d0 con 0x55f68b3ca6b0 2024-04-25T11:40:58.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.628+0000 7f8a9bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f68b3ca6b0 0x55f68b7ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.628+0000 7f8a9bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f68b3ca6b0 0x55f68b7ab670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:44374/0 (socket says 172.21.15.90:44374) 2024-04-25T11:40:58.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.628+0000 7f8a9bfff640 1 -- 172.21.15.90:0/1941639474 learned_addr learned my addr 172.21.15.90:0/1941639474 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:58.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.628+0000 7f8aa2d9d640 1 --2- 172.21.15.90:0/1941639474 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f68b3eb390 0x55f68b7b7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.628+0000 7f8aa259c640 1 --2- 172.21.15.90:0/1941639474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f68b3a9f60 0x55f68b789160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.629+0000 7f8aa2d9d640 1 -- 172.21.15.90:0/1941639474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f68b3ca6b0 msgr2=0x55f68b7ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.629+0000 7f8aa2d9d640 1 --2- 172.21.15.90:0/1941639474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f68b3ca6b0 0x55f68b7ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.629+0000 7f8aa2d9d640 1 -- 172.21.15.90:0/1941639474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f68b3a9f60 msgr2=0x55f68b789160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.629+0000 7f8aa2d9d640 1 --2- 172.21.15.90:0/1941639474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f68b3a9f60 0x55f68b789160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.629+0000 7f8aa2d9d640 1 -- 172.21.15.90:0/1941639474 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f68b784780 con 0x55f68b3eb390 2024-04-25T11:40:58.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.630+0000 7f8aa2d9d640 1 --2- 172.21.15.90:0/1941639474 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f68b3eb390 0x55f68b7b7530 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c00e720 tx=0x7f8a9c000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd9bbf3401dfb0b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:58.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.633+0000 7f8a8bfff640 1 -- 172.21.15.90:0/1941639474 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c003970 con 0x55f68b3eb390 2024-04-25T11:40:58.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.633+0000 7f8a8bfff640 1 -- 172.21.15.90:0/1941639474 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a9c00e930 con 0x55f68b3eb390 2024-04-25T11:40:58.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.633+0000 7f8a8bfff640 1 -- 172.21.15.90:0/1941639474 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c005430 con 0x55f68b3eb390 2024-04-25T11:40:58.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.636+0000 7f8aafe10740 1 -- 172.21.15.90:0/1941639474 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f68b3eb390 msgr2=0x55f68b7b7530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.636+0000 7f8aafe10740 1 --2- 172.21.15.90:0/1941639474 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f68b3eb390 0x55f68b7b7530 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c00e720 tx=0x7f8a9c000f40 comp rx=0 tx=0).stop 2024-04-25T11:40:58.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.640+0000 7f8aafe10740 1 -- 172.21.15.90:0/1941639474 shutdown_connections 2024-04-25T11:40:58.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.640+0000 7f8aafe10740 1 --2- 172.21.15.90:0/1941639474 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f68b3eb390 0x55f68b7b7530 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.640+0000 7f8aafe10740 1 --2- 172.21.15.90:0/1941639474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f68b3ca6b0 0x55f68b7ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.640+0000 7f8aafe10740 1 --2- 172.21.15.90:0/1941639474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f68b3a9f60 0x55f68b789160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.640+0000 7f8aafe10740 1 -- 172.21.15.90:0/1941639474 >> 172.21.15.90:0/1941639474 conn(0x55f68b3df4e0 msgr2=0x55f68b788a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:58.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.641+0000 7f8aafe10740 1 -- 172.21.15.90:0/1941639474 shutdown_connections 2024-04-25T11:40:58.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.642+0000 7f8aafe10740 1 -- 172.21.15.90:0/1941639474 wait complete. 2024-04-25T11:40:58.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.643+0000 7f8aafe10740 1 -- start start 2024-04-25T11:40:58.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.643+0000 7f8aafe10740 1 Processor -- start 2024-04-25T11:40:58.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.644+0000 7f8aafe10740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f68b3a9f60 0x55f68b7a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.646+0000 7f8aafe10740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f68b3ca6b0 0x55f68b7a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.646+0000 7f8aafe10740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f68b3eb390 0x55f68b7a5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.646+0000 7f8aa259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f68b3a9f60 0x55f68b7a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.646+0000 7f8aa259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f68b3a9f60 0x55f68b7a47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46236/0 (socket says 172.21.15.90:46236) 2024-04-25T11:40:58.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.646+0000 7f8aa259c640 1 -- 172.21.15.90:0/153650042 learned_addr learned my addr 172.21.15.90:0/153650042 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:58.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.646+0000 7f8aafe10740 1 -- 172.21.15.90:0/153650042 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8a8400fe70 con 0x55f68b3a9f60 2024-04-25T11:40:58.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.647+0000 7f8aafe10740 1 -- 172.21.15.90:0/153650042 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8a8400f7a0 con 0x55f68b3eb390 2024-04-25T11:40:58.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.647+0000 7f8aafe10740 1 -- 172.21.15.90:0/153650042 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8a8400c9e0 con 0x55f68b3ca6b0 2024-04-25T11:40:58.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.647+0000 7f8aafe10740 10 client.0 _ll_register_callbacks cb 0x7f8a8c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:58.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.647+0000 7f8aa2d9d640 1 --2- 172.21.15.90:0/153650042 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f68b3eb390 0x55f68b7a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.647+0000 7f8a9bfff640 1 --2- 172.21.15.90:0/153650042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f68b3ca6b0 0x55f68b7a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.649+0000 7f8aa259c640 1 -- 172.21.15.90:0/153650042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f68b3ca6b0 msgr2=0x55f68b7a4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.649+0000 7f8aa259c640 1 --2- 172.21.15.90:0/153650042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f68b3ca6b0 0x55f68b7a4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.649+0000 7f8aa259c640 1 -- 172.21.15.90:0/153650042 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f68b3eb390 msgr2=0x55f68b7a5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.649+0000 7f8aa259c640 1 --2- 172.21.15.90:0/153650042 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f68b3eb390 0x55f68b7a5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.649+0000 7f8aa259c640 1 -- 172.21.15.90:0/153650042 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f68b7926d0 con 0x55f68b3a9f60 2024-04-25T11:40:58.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.650+0000 7f8aa259c640 1 --2- 172.21.15.90:0/153650042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f68b3a9f60 0x55f68b7a47f0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f8a9402a860 tx=0x7f8a94030820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f31628a31b36f399 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:58.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.650+0000 7f8a89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:40:58.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.650+0000 7f8a89ffb640 1 -- 172.21.15.90:0/153650042 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a94037040 con 0x55f68b3a9f60 2024-04-25T11:40:58.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.650+0000 7f8a89ffb640 1 -- 172.21.15.90:0/153650042 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f8a94032850 con 0x55f68b3a9f60 2024-04-25T11:40:58.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.651+0000 7f8aafe10740 1 -- 172.21.15.90:0/153650042 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a9c00e930 con 0x55f68b3a9f60 2024-04-25T11:40:58.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.651+0000 7f8a89ffb640 1 -- 172.21.15.90:0/153650042 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8a94032a50 con 0x55f68b3a9f60 2024-04-25T11:40:58.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.652+0000 7f8aafe10740 10 client.17337 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:58.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.652+0000 7f8aafe10740 1 -- 172.21.15.90:0/153650042 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f68b79cbb0 con 0x55f68b3a9f60 2024-04-25T11:40:58.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.654+0000 7f8a89ffb640 1 -- 172.21.15.90:0/153650042 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a9c00e930 con 0x55f68b3a9f60 2024-04-25T11:40:58.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.654+0000 7f8a89ffb640 1 -- 172.21.15.90:0/153650042 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f8a940036c0 con 0x55f68b3a9f60 2024-04-25T11:40:58.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.654+0000 7f8aafe10740 10 client.17337 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:58.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.654+0000 7f8aafe10740 10 client.17337 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:40:58.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.654+0000 7f8aafe10740 20 client.17337 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:58.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.654+0000 7f8aafe10740 1 --2- 172.21.15.90:0/153650042 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f68b37f3a0 0x55f68b79d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.654+0000 7f8aafe10740 4 client.17337 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:40:58.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.654+0000 7f8aafe10740 1 -- 172.21.15.90:0/153650042 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f68b6f9800 con 0x55f68b37f3a0 2024-04-25T11:40:58.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.655+0000 7f8a9bfff640 1 --2- 172.21.15.90:0/153650042 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f68b37f3a0 0x55f68b79d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.662+0000 7f8a89ffb640 10 client.17337 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:40:58.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.662+0000 7f8a9bfff640 1 --2- 172.21.15.90:0/153650042 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f68b37f3a0 0x55f68b79d360 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c3910b7440cd049f server_cookie=6cc3cbefa1f94d6 in_seq=0 out_seq=0 2024-04-25T11:40:58.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.665+0000 7f8a89ffb640 1 -- 172.21.15.90:0/153650042 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f68b6f9800 con 0x55f68b37f3a0 2024-04-25T11:40:58.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.665+0000 7f8a89ffb640 10 client.17337 handle_command_reply: tid=0 2024-04-25T11:40:58.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.665+0000 7f8aafe10740 1 client.17337 shutdown 2024-04-25T11:40:58.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.666+0000 7f8aafe10740 1 -- 172.21.15.90:0/153650042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f68b3a9f60 msgr2=0x55f68b7a47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.666+0000 7f8aafe10740 1 --2- 172.21.15.90:0/153650042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f68b3a9f60 0x55f68b7a47f0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f8a9402a860 tx=0x7f8a94030820 comp rx=0 tx=0).stop 2024-04-25T11:40:58.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.666+0000 7f8aafe10740 1 -- 172.21.15.90:0/153650042 shutdown_connections 2024-04-25T11:40:58.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.666+0000 7f8aafe10740 1 --2- 172.21.15.90:0/153650042 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f68b37f3a0 0x55f68b79d360 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.666+0000 7f8aafe10740 1 --2- 172.21.15.90:0/153650042 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f68b3eb390 0x55f68b7a5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.666+0000 7f8aafe10740 1 --2- 172.21.15.90:0/153650042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f68b3ca6b0 0x55f68b7a4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.666+0000 7f8aafe10740 1 --2- 172.21.15.90:0/153650042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f68b3a9f60 0x55f68b7a47f0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.666+0000 7f8aafe10740 1 -- 172.21.15.90:0/153650042 >> 172.21.15.90:0/153650042 conn(0x55f68b3df4e0 msgr2=0x55f68b7a0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:58.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.666+0000 7f8a89ffb640 0 client.17337 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:40:58.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.666+0000 7f8aafe10740 1 -- 172.21.15.90:0/153650042 shutdown_connections 2024-04-25T11:40:58.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.666+0000 7f8aafe10740 1 -- 172.21.15.90:0/153650042 wait complete. 2024-04-25T11:40:58.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.668+0000 7f8aafe10740 20 client.17337 trim_cache size 0 max 16384 2024-04-25T11:40:58.657 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:40:58.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.671+0000 7f8aa8e8c640 1 -- 172.21.15.90:0/3915526408 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8a900afce0 msgr2=0x7f8a900b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.671+0000 7f8aa8e8c640 1 --2- 172.21.15.90:0/3915526408 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8a900afce0 0x7f8a900b21f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8a94009830 tx=0x7f8a94000bc0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.671+0000 7f8aa8e8c640 1 -- 172.21.15.90:0/3915526408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8aa4109b30 msgr2=0x7f8aa419acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.671+0000 7f8aa8e8c640 1 --2- 172.21.15.90:0/3915526408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8aa4109b30 0x7f8aa419acc0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c00e980 tx=0x7f8a8c00ee40 comp rx=0 tx=0).stop 2024-04-25T11:40:58.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.671+0000 7f8aa8e8c640 1 -- 172.21.15.90:0/3915526408 shutdown_connections 2024-04-25T11:40:58.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.671+0000 7f8aa8e8c640 1 --2- 172.21.15.90:0/3915526408 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8a900afce0 0x7f8a900b21f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.671+0000 7f8aa8e8c640 1 --2- 172.21.15.90:0/3915526408 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8aa4109fd0 0x7f8aa419ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.671+0000 7f8aa8e8c640 1 --2- 172.21.15.90:0/3915526408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8aa4109b30 0x7f8aa419acc0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.672+0000 7f8aa8e8c640 1 --2- 172.21.15.90:0/3915526408 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8aa41042d0 0x7f8aa419a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.672+0000 7f8aa8e8c640 1 -- 172.21.15.90:0/3915526408 >> 172.21.15.90:0/3915526408 conn(0x7f8aa4079280 msgr2=0x7f8aa41051d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:58.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.673+0000 7f8aa8e8c640 1 -- 172.21.15.90:0/3915526408 shutdown_connections 2024-04-25T11:40:58.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.673+0000 7f8aa8e8c640 1 -- 172.21.15.90:0/3915526408 wait complete. 2024-04-25T11:40:58.685 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T11:40:58.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:58 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/757934653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:40:58.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.859+0000 7f3b6bee2640 1 -- 172.21.15.90:0/3622235206 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b64108a40 msgr2=0x7f3b64102ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.859+0000 7f3b6bee2640 1 --2- 172.21.15.90:0/3622235206 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b64108a40 0x7f3b64102ca0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f3b60009860 tx=0x7f3b6002a3d0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.859+0000 7f3b6bee2640 1 -- 172.21.15.90:0/3622235206 shutdown_connections 2024-04-25T11:40:58.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.859+0000 7f3b6bee2640 1 --2- 172.21.15.90:0/3622235206 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3b64103890 0x7f3b6410f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.859+0000 7f3b6bee2640 1 --2- 172.21.15.90:0/3622235206 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b64108e50 0x7f3b64103200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.859+0000 7f3b6bee2640 1 --2- 172.21.15.90:0/3622235206 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b64108a40 0x7f3b64102ca0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.859+0000 7f3b6bee2640 1 -- 172.21.15.90:0/3622235206 >> 172.21.15.90:0/3622235206 conn(0x7f3b64077880 msgr2=0x7f3b64077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:58.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.859+0000 7f3b6bee2640 1 -- 172.21.15.90:0/3622235206 shutdown_connections 2024-04-25T11:40:58.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.859+0000 7f3b6bee2640 1 -- 172.21.15.90:0/3622235206 wait complete. 2024-04-25T11:40:58.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.860+0000 7f3b6bee2640 1 Processor -- start 2024-04-25T11:40:58.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.860+0000 7f3b6bee2640 1 -- start start 2024-04-25T11:40:58.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.860+0000 7f3b6bee2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b64108a40 0x7f3b641102a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.860+0000 7f3b6bee2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3b64108e50 0x7f3b641107e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.860+0000 7f3b6bee2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b64110d20 0x7f3b641a8560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.860+0000 7f3b6bee2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3b64113d50 con 0x7f3b64108a40 2024-04-25T11:40:58.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.860+0000 7f3b6bee2640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3b64113bd0 con 0x7f3b64108e50 2024-04-25T11:40:58.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.860+0000 7f3b6bee2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3b64113ed0 con 0x7f3b64110d20 2024-04-25T11:40:58.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.861+0000 7f3b69c7e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b64108a40 0x7f3b641102a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.861+0000 7f3b69c7e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b64108a40 0x7f3b641102a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46248/0 (socket says 172.21.15.90:46248) 2024-04-25T11:40:58.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.861+0000 7f3b69c7e640 1 -- 172.21.15.90:0/1254482672 learned_addr learned my addr 172.21.15.90:0/1254482672 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:58.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.861+0000 7f3b6947d640 1 --2- 172.21.15.90:0/1254482672 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3b64108e50 0x7f3b641107e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.861+0000 7f3b6a47f640 1 --2- 172.21.15.90:0/1254482672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b64110d20 0x7f3b641a8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.861+0000 7f3b6947d640 1 -- 172.21.15.90:0/1254482672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b64110d20 msgr2=0x7f3b641a8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.861+0000 7f3b6947d640 1 --2- 172.21.15.90:0/1254482672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b64110d20 0x7f3b641a8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.861+0000 7f3b6947d640 1 -- 172.21.15.90:0/1254482672 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b64108a40 msgr2=0x7f3b641102a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.861+0000 7f3b6947d640 1 --2- 172.21.15.90:0/1254482672 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b64108a40 0x7f3b641102a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.861+0000 7f3b6947d640 1 -- 172.21.15.90:0/1254482672 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b641a8b20 con 0x7f3b64108e50 2024-04-25T11:40:58.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.861+0000 7f3b69c7e640 1 --2- 172.21.15.90:0/1254482672 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b64108a40 0x7f3b641102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:40:58.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.861+0000 7f3b6947d640 1 --2- 172.21.15.90:0/1254482672 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3b64108e50 0x7f3b641107e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3b5400d6a0 tx=0x7f3b5400db60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:58.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.861+0000 7f3b5affd640 1 -- 172.21.15.90:0/1254482672 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b54017070 con 0x7f3b64108e50 2024-04-25T11:40:58.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.862+0000 7f3b6bee2640 1 -- 172.21.15.90:0/1254482672 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b641a8e10 con 0x7f3b64108e50 2024-04-25T11:40:58.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.862+0000 7f3b5affd640 1 -- 172.21.15.90:0/1254482672 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b54009820 con 0x7f3b64108e50 2024-04-25T11:40:58.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.862+0000 7f3b5affd640 1 -- 172.21.15.90:0/1254482672 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b54004c40 con 0x7f3b64108e50 2024-04-25T11:40:58.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.862+0000 7f3b6bee2640 1 -- 172.21.15.90:0/1254482672 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b641a9270 con 0x7f3b64108e50 2024-04-25T11:40:58.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.864+0000 7f3b5affd640 1 -- 172.21.15.90:0/1254482672 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f3b54010460 con 0x7f3b64108e50 2024-04-25T11:40:58.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.864+0000 7f3b72d8c740 1 Processor -- start 2024-04-25T11:40:58.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.865+0000 7f3b72d8c740 1 -- start start 2024-04-25T11:40:58.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.865+0000 7f3b72d8c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd3e653390 0x55fd3e9f4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.865+0000 7f3b72d8c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd3e62a4d0 0x55fd3e9f4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.865+0000 7f3b72d8c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd3e6326b0 0x55fd3ea016a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.865+0000 7f3b72d8c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55fd3e959d80 con 0x55fd3e62a4d0 2024-04-25T11:40:58.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.865+0000 7f3b72d8c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55fd3e913a20 con 0x55fd3e6326b0 2024-04-25T11:40:58.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.865+0000 7f3b72d8c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fd3e955d40 con 0x55fd3e653390 2024-04-25T11:40:58.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.865+0000 7f3b69c7e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd3e62a4d0 0x55fd3e9f4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.865+0000 7f3b6947d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd3e6326b0 0x55fd3ea016a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.865+0000 7f3b69c7e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd3e62a4d0 0x55fd3e9f4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46260/0 (socket says 172.21.15.90:46260) 2024-04-25T11:40:58.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.865+0000 7f3b69c7e640 1 -- 172.21.15.90:0/798727053 learned_addr learned my addr 172.21.15.90:0/798727053 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:58.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.865+0000 7f3b6a47f640 1 --2- 172.21.15.90:0/798727053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd3e653390 0x55fd3e9f4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.865+0000 7f3b5affd640 1 --2- 172.21.15.90:0/1254482672 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3b500afc60 0x7f3b500b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.866+0000 7f3b69c7e640 1 -- 172.21.15.90:0/798727053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd3e653390 msgr2=0x55fd3e9f4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.866+0000 7f3b69c7e640 1 --2- 172.21.15.90:0/798727053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd3e653390 0x55fd3e9f4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.866+0000 7f3b69c7e640 1 -- 172.21.15.90:0/798727053 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd3e6326b0 msgr2=0x55fd3ea016a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.866+0000 7f3b69c7e640 1 --2- 172.21.15.90:0/798727053 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd3e6326b0 0x55fd3ea016a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.866+0000 7f3b69c7e640 1 -- 172.21.15.90:0/798727053 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd3e91ec30 con 0x55fd3e62a4d0 2024-04-25T11:40:58.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.866+0000 7f3b6947d640 1 --2- 172.21.15.90:0/798727053 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd3e6326b0 0x55fd3ea016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:40:58.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.866+0000 7f3b6a47f640 1 --2- 172.21.15.90:0/1254482672 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3b500afc60 0x7f3b500b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.866+0000 7f3b69c7e640 1 --2- 172.21.15.90:0/798727053 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd3e62a4d0 0x55fd3e9f4d50 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f3b60009980 tx=0x7f3b60030980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13de9f50a9e094cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:58.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.866+0000 7f3b5affd640 1 -- 172.21.15.90:0/1254482672 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f3b540de890 con 0x7f3b64108e50 2024-04-25T11:40:58.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.866+0000 7f3b4ffff640 1 -- 172.21.15.90:0/798727053 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b60030da0 con 0x55fd3e62a4d0 2024-04-25T11:40:58.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.866+0000 7f3b4ffff640 1 -- 172.21.15.90:0/798727053 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b60005040 con 0x55fd3e62a4d0 2024-04-25T11:40:58.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.866+0000 7f3b4ffff640 1 -- 172.21.15.90:0/798727053 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b60004940 con 0x55fd3e62a4d0 2024-04-25T11:40:58.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.867+0000 7f3b72d8c740 1 -- 172.21.15.90:0/798727053 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd3e62a4d0 msgr2=0x55fd3e9f4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.867+0000 7f3b6a47f640 1 --2- 172.21.15.90:0/1254482672 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3b500afc60 0x7f3b500b2170 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x55fd3e9f1980 tx=0x7f3b5c004a40 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:58.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.867+0000 7f3b72d8c740 1 --2- 172.21.15.90:0/798727053 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd3e62a4d0 0x55fd3e9f4d50 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f3b60009980 tx=0x7f3b60030980 comp rx=0 tx=0).stop 2024-04-25T11:40:58.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.867+0000 7f3b72d8c740 1 -- 172.21.15.90:0/798727053 shutdown_connections 2024-04-25T11:40:58.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.867+0000 7f3b72d8c740 1 --2- 172.21.15.90:0/798727053 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd3e6326b0 0x55fd3ea016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.867+0000 7f3b72d8c740 1 --2- 172.21.15.90:0/798727053 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd3e62a4d0 0x55fd3e9f4d50 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.867+0000 7f3b72d8c740 1 --2- 172.21.15.90:0/798727053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd3e653390 0x55fd3e9f4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.867+0000 7f3b72d8c740 1 -- 172.21.15.90:0/798727053 >> 172.21.15.90:0/798727053 conn(0x55fd3e6474e0 msgr2=0x55fd3e9f0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:58.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.867+0000 7f3b72d8c740 1 -- 172.21.15.90:0/798727053 shutdown_connections 2024-04-25T11:40:58.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.867+0000 7f3b72d8c740 1 -- 172.21.15.90:0/798727053 wait complete. 2024-04-25T11:40:58.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.868+0000 7f3b72d8c740 1 -- start start 2024-04-25T11:40:58.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.868+0000 7f3b72d8c740 1 Processor -- start 2024-04-25T11:40:58.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.868+0000 7f3b72d8c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd3e62a4d0 0x55fd3e9ff2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.868+0000 7f3b72d8c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd3e6326b0 0x55fd3e9ff7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.868+0000 7f3b72d8c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd3e653390 0x55fd3ea0b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.868+0000 7f3b72d8c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55fd3ea01d30 con 0x55fd3e6326b0 2024-04-25T11:40:58.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.868+0000 7f3b72d8c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55fd3e8e0dc0 con 0x55fd3e653390 2024-04-25T11:40:58.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.868+0000 7f3b72d8c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fd3ea01eb0 con 0x55fd3e62a4d0 2024-04-25T11:40:58.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.869+0000 7f3b72d8c740 10 client.0 _ll_register_callbacks cb 0x55fd3e8fc600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:58.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.869+0000 7f3b6947d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd3e6326b0 0x55fd3e9ff7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.869+0000 7f3b6a47f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd3e653390 0x55fd3ea0b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.869+0000 7f3b6947d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd3e6326b0 0x55fd3e9ff7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46264/0 (socket says 172.21.15.90:46264) 2024-04-25T11:40:58.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.869+0000 7f3b6947d640 1 -- 172.21.15.90:0/875397649 learned_addr learned my addr 172.21.15.90:0/875397649 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:58.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.869+0000 7f3b69c7e640 1 --2- 172.21.15.90:0/875397649 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd3e62a4d0 0x55fd3e9ff2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.869+0000 7f3b6a47f640 1 -- 172.21.15.90:0/875397649 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd3e62a4d0 msgr2=0x55fd3e9ff2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.869+0000 7f3b6a47f640 1 --2- 172.21.15.90:0/875397649 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd3e62a4d0 0x55fd3e9ff2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.869+0000 7f3b6a47f640 1 -- 172.21.15.90:0/875397649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd3e6326b0 msgr2=0x55fd3e9ff7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.869+0000 7f3b6a47f640 1 --2- 172.21.15.90:0/875397649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd3e6326b0 0x55fd3e9ff7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.869+0000 7f3b6a47f640 1 -- 172.21.15.90:0/875397649 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd3e66b3f0 con 0x55fd3e653390 2024-04-25T11:40:58.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.869+0000 7f3b6947d640 1 --2- 172.21.15.90:0/875397649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd3e6326b0 0x55fd3e9ff7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:40:58.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.869+0000 7f3b69c7e640 1 --2- 172.21.15.90:0/875397649 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd3e62a4d0 0x55fd3e9ff2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:40:58.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.869+0000 7f3b6a47f640 1 --2- 172.21.15.90:0/875397649 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd3e653390 0x55fd3ea0b1b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x55fd3e9f5910 tx=0x7f3b5c010040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc7ea5678b433914 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:58.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.869+0000 7f3b4d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:40:58.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.869+0000 7f3b4d7fa640 1 -- 172.21.15.90:0/875397649 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c02b060 con 0x55fd3e653390 2024-04-25T11:40:58.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.870+0000 7f3b4d7fa640 1 -- 172.21.15.90:0/875397649 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f3b5c01a6f0 con 0x55fd3e653390 2024-04-25T11:40:58.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.870+0000 7f3b72d8c740 1 -- 172.21.15.90:0/875397649 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b60005040 con 0x55fd3e653390 2024-04-25T11:40:58.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.870+0000 7f3b4d7fa640 1 -- 172.21.15.90:0/875397649 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3b5c01a8f0 con 0x55fd3e653390 2024-04-25T11:40:58.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.870+0000 7f3b72d8c740 10 client.26405 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:58.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.870+0000 7f3b72d8c740 1 -- 172.21.15.90:0/875397649 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fd3e7357d0 con 0x55fd3e653390 2024-04-25T11:40:58.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.870+0000 7f3b4d7fa640 1 -- 172.21.15.90:0/875397649 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b60005040 con 0x55fd3e653390 2024-04-25T11:40:58.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.870+0000 7f3b4d7fa640 1 -- 172.21.15.90:0/875397649 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f3b5c00b040 con 0x55fd3e653390 2024-04-25T11:40:58.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.870+0000 7f3b72d8c740 10 client.26405 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:58.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.870+0000 7f3b72d8c740 10 client.26405 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:40:58.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.870+0000 7f3b72d8c740 20 client.26405 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:58.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.870+0000 7f3b72d8c740 1 --2- 172.21.15.90:0/875397649 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55fd3e611f60 0x55fd3ea11760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.870+0000 7f3b72d8c740 4 client.26405 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:40:58.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.870+0000 7f3b72d8c740 1 -- 172.21.15.90:0/875397649 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd3e967990 con 0x55fd3e611f60 2024-04-25T11:40:58.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.871+0000 7f3b69c7e640 1 --2- 172.21.15.90:0/875397649 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55fd3e611f60 0x55fd3ea11760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.871+0000 7f3b4d7fa640 10 client.26405 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:40:58.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.871+0000 7f3b69c7e640 1 --2- 172.21.15.90:0/875397649 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55fd3e611f60 0x55fd3ea11760 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=35d73485b0167b43 server_cookie=426406d6606ac68e in_seq=0 out_seq=0 2024-04-25T11:40:58.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.872+0000 7f3b4d7fa640 1 -- 172.21.15.90:0/875397649 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55fd3e967990 con 0x55fd3e611f60 2024-04-25T11:40:58.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.872+0000 7f3b4d7fa640 10 client.26405 handle_command_reply: tid=0 2024-04-25T11:40:58.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.872+0000 7f3b72d8c740 1 client.26405 shutdown 2024-04-25T11:40:58.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.872+0000 7f3b72d8c740 1 -- 172.21.15.90:0/875397649 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd3e653390 msgr2=0x55fd3ea0b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.872+0000 7f3b72d8c740 1 --2- 172.21.15.90:0/875397649 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd3e653390 0x55fd3ea0b1b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x55fd3e9f5910 tx=0x7f3b5c010040 comp rx=0 tx=0).stop 2024-04-25T11:40:58.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.872+0000 7f3b72d8c740 1 -- 172.21.15.90:0/875397649 shutdown_connections 2024-04-25T11:40:58.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.872+0000 7f3b72d8c740 1 --2- 172.21.15.90:0/875397649 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55fd3e611f60 0x55fd3ea11760 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.872+0000 7f3b72d8c740 1 --2- 172.21.15.90:0/875397649 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd3e653390 0x55fd3ea0b1b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.872+0000 7f3b72d8c740 1 --2- 172.21.15.90:0/875397649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd3e6326b0 0x55fd3e9ff7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.872+0000 7f3b4d7fa640 0 client.26405 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:40:58.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.872+0000 7f3b72d8c740 1 --2- 172.21.15.90:0/875397649 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd3e62a4d0 0x55fd3e9ff2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.872+0000 7f3b72d8c740 1 -- 172.21.15.90:0/875397649 >> 172.21.15.90:0/875397649 conn(0x55fd3e6474e0 msgr2=0x55fd3e9f6530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:58.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.872+0000 7f3b72d8c740 1 -- 172.21.15.90:0/875397649 shutdown_connections 2024-04-25T11:40:58.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.872+0000 7f3b72d8c740 1 -- 172.21.15.90:0/875397649 wait complete. 2024-04-25T11:40:58.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.873+0000 7f3b72d8c740 20 client.26405 trim_cache size 0 max 16384 2024-04-25T11:40:58.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.889+0000 7f3b72d8c740 1 Processor -- start 2024-04-25T11:40:58.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.889+0000 7f3b72d8c740 1 -- start start 2024-04-25T11:40:58.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.890+0000 7f3b72d8c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd3e62a4d0 0x55fd3e9f0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.890+0000 7f3b72d8c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd3e6326b0 0x55fd3ea0ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.890+0000 7f3b72d8c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd3e653390 0x55fd3e9f9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.890+0000 7f3b72d8c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3b641b1e00 con 0x55fd3e62a4d0 2024-04-25T11:40:58.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.890+0000 7f3b72d8c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3b641b3940 con 0x55fd3e653390 2024-04-25T11:40:58.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.890+0000 7f3b72d8c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3b641b24d0 con 0x55fd3e6326b0 2024-04-25T11:40:58.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.890+0000 7f3b69c7e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd3e62a4d0 0x55fd3e9f0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.890+0000 7f3b6a47f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd3e653390 0x55fd3e9f9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.890+0000 7f3b6a47f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd3e653390 0x55fd3e9f9430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:41070/0 (socket says 172.21.15.90:41070) 2024-04-25T11:40:58.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.890+0000 7f3b69c7e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd3e62a4d0 0x55fd3e9f0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46270/0 (socket says 172.21.15.90:46270) 2024-04-25T11:40:58.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.890+0000 7f3b6a47f640 1 -- 172.21.15.90:0/2860839820 learned_addr learned my addr 172.21.15.90:0/2860839820 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:58.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.890+0000 7f3b6947d640 1 --2- 172.21.15.90:0/2860839820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd3e6326b0 0x55fd3ea0ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.890+0000 7f3b6a47f640 1 -- 172.21.15.90:0/2860839820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd3e6326b0 msgr2=0x55fd3ea0ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.890+0000 7f3b6a47f640 1 --2- 172.21.15.90:0/2860839820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd3e6326b0 0x55fd3ea0ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.890+0000 7f3b6a47f640 1 -- 172.21.15.90:0/2860839820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd3e62a4d0 msgr2=0x55fd3e9f0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.890+0000 7f3b6a47f640 1 --2- 172.21.15.90:0/2860839820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd3e62a4d0 0x55fd3e9f0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.890+0000 7f3b6a47f640 1 -- 172.21.15.90:0/2860839820 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd3e9f6010 con 0x55fd3e653390 2024-04-25T11:40:58.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.891+0000 7f3b6a47f640 1 --2- 172.21.15.90:0/2860839820 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd3e653390 0x55fd3e9f9430 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c000f80 tx=0x7f3b5c009540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=caa224bd8baef79e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:58.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.891+0000 7f3b4f7fe640 1 -- 172.21.15.90:0/2860839820 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c02b020 con 0x55fd3e653390 2024-04-25T11:40:58.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.891+0000 7f3b4f7fe640 1 -- 172.21.15.90:0/2860839820 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b5c021050 con 0x55fd3e653390 2024-04-25T11:40:58.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.891+0000 7f3b4f7fe640 1 -- 172.21.15.90:0/2860839820 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c026b30 con 0x55fd3e653390 2024-04-25T11:40:58.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.892+0000 7f3b72d8c740 1 -- 172.21.15.90:0/2860839820 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd3e653390 msgr2=0x55fd3e9f9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.892+0000 7f3b72d8c740 1 --2- 172.21.15.90:0/2860839820 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd3e653390 0x55fd3e9f9430 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c000f80 tx=0x7f3b5c009540 comp rx=0 tx=0).stop 2024-04-25T11:40:58.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.892+0000 7f3b72d8c740 1 -- 172.21.15.90:0/2860839820 shutdown_connections 2024-04-25T11:40:58.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.892+0000 7f3b72d8c740 1 --2- 172.21.15.90:0/2860839820 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd3e653390 0x55fd3e9f9430 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.892+0000 7f3b72d8c740 1 --2- 172.21.15.90:0/2860839820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd3e6326b0 0x55fd3ea0ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.892+0000 7f3b72d8c740 1 --2- 172.21.15.90:0/2860839820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd3e62a4d0 0x55fd3e9f0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.892+0000 7f3b72d8c740 1 -- 172.21.15.90:0/2860839820 >> 172.21.15.90:0/2860839820 conn(0x55fd3e6474e0 msgr2=0x55fd3e9f08a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:58.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.892+0000 7f3b72d8c740 1 -- 172.21.15.90:0/2860839820 shutdown_connections 2024-04-25T11:40:58.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.892+0000 7f3b72d8c740 1 -- 172.21.15.90:0/2860839820 wait complete. 2024-04-25T11:40:58.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.892+0000 7f3b72d8c740 1 -- start start 2024-04-25T11:40:58.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.892+0000 7f3b72d8c740 1 Processor -- start 2024-04-25T11:40:58.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.893+0000 7f3b72d8c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd3e62a4d0 0x55fd3ea077e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.893+0000 7f3b72d8c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd3e6326b0 0x55fd3ea02150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.893+0000 7f3b72d8c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd3e653390 0x55fd3ea02750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.893+0000 7f3b72d8c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3b641b3270 con 0x55fd3e653390 2024-04-25T11:40:58.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.893+0000 7f3b72d8c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3b641b2ba0 con 0x55fd3e6326b0 2024-04-25T11:40:58.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.893+0000 7f3b72d8c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3b641afde0 con 0x55fd3e62a4d0 2024-04-25T11:40:58.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.893+0000 7f3b72d8c740 10 client.0 _ll_register_callbacks cb 0x7f3b5c01d040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:58.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.893+0000 7f3b6947d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd3e6326b0 0x55fd3ea02150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.893+0000 7f3b6a47f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd3e653390 0x55fd3ea02750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.893+0000 7f3b6947d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd3e6326b0 0x55fd3ea02150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:41074/0 (socket says 172.21.15.90:41074) 2024-04-25T11:40:58.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.893+0000 7f3b6947d640 1 -- 172.21.15.90:0/3019037602 learned_addr learned my addr 172.21.15.90:0/3019037602 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:58.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.893+0000 7f3b69c7e640 1 --2- 172.21.15.90:0/3019037602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd3e62a4d0 0x55fd3ea077e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.894+0000 7f3b6947d640 1 -- 172.21.15.90:0/3019037602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd3e62a4d0 msgr2=0x55fd3ea077e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.894+0000 7f3b6947d640 1 --2- 172.21.15.90:0/3019037602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd3e62a4d0 0x55fd3ea077e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.894+0000 7f3b6947d640 1 -- 172.21.15.90:0/3019037602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd3e653390 msgr2=0x55fd3ea02750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:58.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.894+0000 7f3b6947d640 1 --2- 172.21.15.90:0/3019037602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd3e653390 0x55fd3ea02750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.894+0000 7f3b6947d640 1 -- 172.21.15.90:0/3019037602 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd3ea02f30 con 0x55fd3e6326b0 2024-04-25T11:40:58.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.894+0000 7f3b6947d640 1 --2- 172.21.15.90:0/3019037602 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd3e6326b0 0x55fd3ea02150 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3b540dc580 tx=0x7f3b54009f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4733cfd098531367 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:58.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.894+0000 7f3b4d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:40:58.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.894+0000 7f3b4d7fa640 1 -- 172.21.15.90:0/3019037602 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b54020040 con 0x55fd3e6326b0 2024-04-25T11:40:58.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.894+0000 7f3b4d7fa640 1 -- 172.21.15.90:0/3019037602 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f3b540af3a0 con 0x55fd3e6326b0 2024-04-25T11:40:58.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.894+0000 7f3b72d8c740 1 -- 172.21.15.90:0/3019037602 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b5c021050 con 0x55fd3e6326b0 2024-04-25T11:40:58.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.895+0000 7f3b4d7fa640 1 -- 172.21.15.90:0/3019037602 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3b540a63d0 con 0x55fd3e6326b0 2024-04-25T11:40:58.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.895+0000 7f3b72d8c740 10 client.26411 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:58.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.895+0000 7f3b72d8c740 1 -- 172.21.15.90:0/3019037602 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fd3ea78b00 con 0x55fd3e6326b0 2024-04-25T11:40:58.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.895+0000 7f3b4d7fa640 1 -- 172.21.15.90:0/3019037602 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b5c021050 con 0x55fd3e6326b0 2024-04-25T11:40:58.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.895+0000 7f3b4d7fa640 1 -- 172.21.15.90:0/3019037602 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f3b540a23f0 con 0x55fd3e6326b0 2024-04-25T11:40:58.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.895+0000 7f3b72d8c740 10 client.26411 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:58.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.895+0000 7f3b72d8c740 10 client.26411 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:40:58.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.895+0000 7f3b72d8c740 20 client.26411 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:58.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.895+0000 7f3b72d8c740 1 --2- 172.21.15.90:0/3019037602 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55fd3e5e73a0 0x55fd3ea7b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:58.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.895+0000 7f3b72d8c740 4 client.26411 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:40:58.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.895+0000 7f3b72d8c740 1 -- 172.21.15.90:0/3019037602 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fd3e961800 con 0x55fd3e5e73a0 2024-04-25T11:40:58.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.895+0000 7f3b69c7e640 1 --2- 172.21.15.90:0/3019037602 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55fd3e5e73a0 0x55fd3ea7b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:58.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.896+0000 7f3b4d7fa640 10 client.26411 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:40:58.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.896+0000 7f3b69c7e640 1 --2- 172.21.15.90:0/3019037602 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55fd3e5e73a0 0x55fd3ea7b250 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e73b65c63c437c8e server_cookie=d1f536dcabf81917 in_seq=0 out_seq=0 2024-04-25T11:40:58.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.896+0000 7f3b4d7fa640 1 -- 172.21.15.90:0/3019037602 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fd3e961800 con 0x55fd3e5e73a0 2024-04-25T11:40:58.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.896+0000 7f3b4d7fa640 10 client.26411 handle_command_reply: tid=0 2024-04-25T11:40:58.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.896+0000 7f3b72d8c740 1 client.26411 shutdown 2024-04-25T11:40:58.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.896+0000 7f3b72d8c740 1 -- 172.21.15.90:0/3019037602 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd3e6326b0 msgr2=0x55fd3ea02150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.896+0000 7f3b72d8c740 1 --2- 172.21.15.90:0/3019037602 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd3e6326b0 0x55fd3ea02150 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3b540dc580 tx=0x7f3b54009f00 comp rx=0 tx=0).stop 2024-04-25T11:40:58.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.896+0000 7f3b72d8c740 1 -- 172.21.15.90:0/3019037602 shutdown_connections 2024-04-25T11:40:58.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.896+0000 7f3b72d8c740 1 --2- 172.21.15.90:0/3019037602 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55fd3e5e73a0 0x55fd3ea7b250 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.896+0000 7f3b72d8c740 1 --2- 172.21.15.90:0/3019037602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd3e653390 0x55fd3ea02750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.896+0000 7f3b72d8c740 1 --2- 172.21.15.90:0/3019037602 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd3e6326b0 0x55fd3ea02150 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.896+0000 7f3b4d7fa640 0 client.26411 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:40:58.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.896+0000 7f3b72d8c740 1 --2- 172.21.15.90:0/3019037602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd3e62a4d0 0x55fd3ea077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.896+0000 7f3b72d8c740 1 -- 172.21.15.90:0/3019037602 >> 172.21.15.90:0/3019037602 conn(0x55fd3e6474e0 msgr2=0x55fd3ea006e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:58.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.896+0000 7f3b72d8c740 1 -- 172.21.15.90:0/3019037602 shutdown_connections 2024-04-25T11:40:58.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.896+0000 7f3b72d8c740 1 -- 172.21.15.90:0/3019037602 wait complete. 2024-04-25T11:40:58.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.897+0000 7f3b72d8c740 20 client.26411 trim_cache size 0 max 16384 2024-04-25T11:40:58.887 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:40:58.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.897+0000 7f3b6bee2640 1 -- 172.21.15.90:0/1254482672 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3b500afc60 msgr2=0x7f3b500b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.897+0000 7f3b6bee2640 1 --2- 172.21.15.90:0/1254482672 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3b500afc60 0x7f3b500b2170 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x55fd3e9f1980 tx=0x7f3b5c004a40 comp rx=0 tx=0).stop 2024-04-25T11:40:58.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.897+0000 7f3b6bee2640 1 -- 172.21.15.90:0/1254482672 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3b64108e50 msgr2=0x7f3b641107e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:58.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.897+0000 7f3b6bee2640 1 --2- 172.21.15.90:0/1254482672 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3b64108e50 0x7f3b641107e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3b5400d6a0 tx=0x7f3b5400db60 comp rx=0 tx=0).stop 2024-04-25T11:40:58.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.897+0000 7f3b6bee2640 1 -- 172.21.15.90:0/1254482672 shutdown_connections 2024-04-25T11:40:58.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.897+0000 7f3b6bee2640 1 --2- 172.21.15.90:0/1254482672 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3b500afc60 0x7f3b500b2170 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.897+0000 7f3b6bee2640 1 --2- 172.21.15.90:0/1254482672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b64110d20 0x7f3b641a8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.897+0000 7f3b6bee2640 1 --2- 172.21.15.90:0/1254482672 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3b64108e50 0x7f3b641107e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.897+0000 7f3b6bee2640 1 --2- 172.21.15.90:0/1254482672 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b64108a40 0x7f3b641102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:58.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.897+0000 7f3b6bee2640 1 -- 172.21.15.90:0/1254482672 >> 172.21.15.90:0/1254482672 conn(0x7f3b64077880 msgr2=0x7f3b641040d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:58.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.897+0000 7f3b6bee2640 1 -- 172.21.15.90:0/1254482672 shutdown_connections 2024-04-25T11:40:58.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:58.897+0000 7f3b6bee2640 1 -- 172.21.15.90:0/1254482672 wait complete. 2024-04-25T11:40:58.897 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:58 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/757934653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:40:58.899 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T11:40:59.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.038+0000 7fc585625640 1 -- 172.21.15.90:0/4015267240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc580108e70 msgr2=0x7fc580103220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:59.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.038+0000 7fc585625640 1 --2- 172.21.15.90:0/4015267240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc580108e70 0x7fc580103220 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fc574009900 tx=0x7fc57402a480 comp rx=0 tx=0).stop 2024-04-25T11:40:59.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.038+0000 7fc585625640 1 -- 172.21.15.90:0/4015267240 shutdown_connections 2024-04-25T11:40:59.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.038+0000 7fc585625640 1 --2- 172.21.15.90:0/4015267240 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5801038b0 0x7fc58010f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.038+0000 7fc585625640 1 --2- 172.21.15.90:0/4015267240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc580108e70 0x7fc580103220 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.038+0000 7fc585625640 1 --2- 172.21.15.90:0/4015267240 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc580108a60 0x7fc580102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.038+0000 7fc585625640 1 -- 172.21.15.90:0/4015267240 >> 172.21.15.90:0/4015267240 conn(0x7fc580077880 msgr2=0x7fc580077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:59.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.038+0000 7fc585625640 1 -- 172.21.15.90:0/4015267240 shutdown_connections 2024-04-25T11:40:59.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.038+0000 7fc585625640 1 -- 172.21.15.90:0/4015267240 wait complete. 2024-04-25T11:40:59.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.039+0000 7fc585625640 1 Processor -- start 2024-04-25T11:40:59.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.039+0000 7fc585625640 1 -- start start 2024-04-25T11:40:59.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.040+0000 7fc585625640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc580108a60 0x7fc580199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:59.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.040+0000 7fc585625640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc580108e70 0x7fc5801997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:59.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.040+0000 7fc585625640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc58019da60 0x7fc58019de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:59.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.040+0000 7fc585625640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc580112c20 con 0x7fc580108a60 2024-04-25T11:40:59.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.040+0000 7fc585625640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc580112aa0 con 0x7fc580108e70 2024-04-25T11:40:59.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.040+0000 7fc585625640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc580112da0 con 0x7fc58019da60 2024-04-25T11:40:59.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.040+0000 7fc57effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc580108a60 0x7fc580199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:59.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.040+0000 7fc57effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc580108a60 0x7fc580199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46286/0 (socket says 172.21.15.90:46286) 2024-04-25T11:40:59.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.040+0000 7fc57effd640 1 -- 172.21.15.90:0/704554465 learned_addr learned my addr 172.21.15.90:0/704554465 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:59.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.040+0000 7fc57f7fe640 1 --2- 172.21.15.90:0/704554465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc58019da60 0x7fc58019de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:59.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.041+0000 7fc57effd640 1 -- 172.21.15.90:0/704554465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc58019da60 msgr2=0x7fc58019de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:59.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.041+0000 7fc57effd640 1 --2- 172.21.15.90:0/704554465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc58019da60 0x7fc58019de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.041+0000 7fc57effd640 1 -- 172.21.15.90:0/704554465 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc580108e70 msgr2=0x7fc5801997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:59.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.041+0000 7fc57effd640 1 --2- 172.21.15.90:0/704554465 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc580108e70 0x7fc5801997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.041+0000 7fc57effd640 1 -- 172.21.15.90:0/704554465 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc58019e430 con 0x7fc580108a60 2024-04-25T11:40:59.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.041+0000 7fc57effd640 1 --2- 172.21.15.90:0/704554465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc580108a60 0x7fc580199240 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fc57000c9a0 tx=0x7fc57000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:59.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.041+0000 7fc567fff640 1 -- 172.21.15.90:0/704554465 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc570014070 con 0x7fc580108a60 2024-04-25T11:40:59.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.041+0000 7fc585625640 1 -- 172.21.15.90:0/704554465 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc58019e720 con 0x7fc580108a60 2024-04-25T11:40:59.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.041+0000 7fc567fff640 1 -- 172.21.15.90:0/704554465 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc570004440 con 0x7fc580108a60 2024-04-25T11:40:59.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.041+0000 7fc567fff640 1 -- 172.21.15.90:0/704554465 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57000f450 con 0x7fc580108a60 2024-04-25T11:40:59.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.041+0000 7fc585625640 1 -- 172.21.15.90:0/704554465 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5801a5f40 con 0x7fc580108a60 2024-04-25T11:40:59.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.048+0000 7fc567fff640 1 -- 172.21.15.90:0/704554465 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fc570017070 con 0x7fc580108a60 2024-04-25T11:40:59.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.050+0000 7fc567fff640 1 --2- 172.21.15.90:0/704554465 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc5680afce0 0x7fc5680b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:59.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.050+0000 7fc567fff640 1 -- 172.21.15.90:0/704554465 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fc5700debb0 con 0x7fc580108a60 2024-04-25T11:40:59.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.050+0000 7fc57e7fc640 1 --2- 172.21.15.90:0/704554465 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc5680afce0 0x7fc5680b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:59.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.052+0000 7fc57e7fc640 1 --2- 172.21.15.90:0/704554465 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc5680afce0 0x7fc5680b21f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc57402a9a0 tx=0x7fc574031400 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:59.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.054+0000 7fc58c5aa740 1 Processor -- start 2024-04-25T11:40:59.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.057+0000 7fc58c5aa740 1 -- start start 2024-04-25T11:40:59.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.058+0000 7fc58c5aa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d2f3305390 0x55d2f36a6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:59.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.058+0000 7fc58c5aa740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2f32dc4d0 0x55d2f36a6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:59.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.058+0000 7fc58c5aa740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d2f32e46b0 0x55d2f36a7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:59.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.058+0000 7fc58c5aa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d2f360bd80 con 0x55d2f3305390 2024-04-25T11:40:59.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.058+0000 7fc58c5aa740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55d2f35c5a20 con 0x55d2f32e46b0 2024-04-25T11:40:59.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.058+0000 7fc58c5aa740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d2f3607d40 con 0x55d2f32dc4d0 2024-04-25T11:40:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.058+0000 7fc57effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d2f3305390 0x55d2f36a6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.058+0000 7fc57effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d2f3305390 0x55d2f36a6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46296/0 (socket says 172.21.15.90:46296) 2024-04-25T11:40:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.058+0000 7fc57effd640 1 -- 172.21.15.90:0/2856773906 learned_addr learned my addr 172.21.15.90:0/2856773906 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.059+0000 7fc57effd640 1 -- 172.21.15.90:0/2856773906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2f32dc4d0 msgr2=0x55d2f36a6bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:40:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.059+0000 7fc57effd640 1 --2- 172.21.15.90:0/2856773906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2f32dc4d0 0x55d2f36a6bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.059+0000 7fc57effd640 1 -- 172.21.15.90:0/2856773906 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d2f32e46b0 msgr2=0x55d2f36a7100 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:40:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.059+0000 7fc57effd640 1 --2- 172.21.15.90:0/2856773906 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d2f32e46b0 0x55d2f36a7100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.059+0000 7fc57effd640 1 -- 172.21.15.90:0/2856773906 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2f35d0c30 con 0x55d2f3305390 2024-04-25T11:40:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.059+0000 7fc57effd640 1 --2- 172.21.15.90:0/2856773906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d2f3305390 0x55d2f36a6680 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fc5700a2d10 tx=0x7fc570000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e0b73f40ba7093e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.060+0000 7fc5657fa640 1 -- 172.21.15.90:0/2856773906 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5700af440 con 0x55d2f3305390 2024-04-25T11:40:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.060+0000 7fc5657fa640 1 -- 172.21.15.90:0/2856773906 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5700be070 con 0x55d2f3305390 2024-04-25T11:40:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.060+0000 7fc5657fa640 1 -- 172.21.15.90:0/2856773906 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5700dc4e0 con 0x55d2f3305390 2024-04-25T11:40:59.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.062+0000 7fc58c5aa740 1 -- 172.21.15.90:0/2856773906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d2f3305390 msgr2=0x55d2f36a6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:59.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.062+0000 7fc58c5aa740 1 --2- 172.21.15.90:0/2856773906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d2f3305390 0x55d2f36a6680 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fc5700a2d10 tx=0x7fc570000bc0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.063+0000 7fc58c5aa740 1 -- 172.21.15.90:0/2856773906 shutdown_connections 2024-04-25T11:40:59.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.063+0000 7fc58c5aa740 1 --2- 172.21.15.90:0/2856773906 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d2f32e46b0 0x55d2f36a7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.063+0000 7fc58c5aa740 1 --2- 172.21.15.90:0/2856773906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2f32dc4d0 0x55d2f36a6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.063+0000 7fc58c5aa740 1 --2- 172.21.15.90:0/2856773906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d2f3305390 0x55d2f36a6680 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.063+0000 7fc58c5aa740 1 -- 172.21.15.90:0/2856773906 >> 172.21.15.90:0/2856773906 conn(0x55d2f32f94e0 msgr2=0x55d2f36a2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:59.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:58 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/757934653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:40:59.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.065+0000 7fc58c5aa740 1 -- 172.21.15.90:0/2856773906 shutdown_connections 2024-04-25T11:40:59.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.065+0000 7fc58c5aa740 1 -- 172.21.15.90:0/2856773906 wait complete. 2024-04-25T11:40:59.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.066+0000 7fc58c5aa740 1 -- start start 2024-04-25T11:40:59.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.066+0000 7fc58c5aa740 1 Processor -- start 2024-04-25T11:40:59.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.067+0000 7fc58c5aa740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d2f32dc4d0 0x55d2f36b6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:59.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.068+0000 7fc58c5aa740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2f32e46b0 0x55d2f36b6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:59.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.068+0000 7fc58c5aa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d2f3305390 0x55d2f36b15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:59.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.068+0000 7fc58c5aa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d2f36adce0 con 0x55d2f3305390 2024-04-25T11:40:59.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.068+0000 7fc58c5aa740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55d2f3592dc0 con 0x55d2f32dc4d0 2024-04-25T11:40:59.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.068+0000 7fc58c5aa740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d2f36ade60 con 0x55d2f32e46b0 2024-04-25T11:40:59.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.068+0000 7fc58c5aa740 10 client.0 _ll_register_callbacks cb 0x55d2f35ae600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:59.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.069+0000 7fc57e7fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2f32e46b0 0x55d2f36b6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:59.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.069+0000 7fc57e7fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2f32e46b0 0x55d2f36b6770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:44466/0 (socket says 172.21.15.90:44466) 2024-04-25T11:40:59.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.069+0000 7fc57e7fc640 1 -- 172.21.15.90:0/2368715395 learned_addr learned my addr 172.21.15.90:0/2368715395 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:59.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.069+0000 7fc57effd640 1 --2- 172.21.15.90:0/2368715395 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d2f32dc4d0 0x55d2f36b6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:59.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.070+0000 7fc57effd640 1 -- 172.21.15.90:0/2368715395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2f32e46b0 msgr2=0x55d2f36b6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:59.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.070+0000 7fc57effd640 1 --2- 172.21.15.90:0/2368715395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2f32e46b0 0x55d2f36b6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.070+0000 7fc57effd640 1 -- 172.21.15.90:0/2368715395 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d2f3305390 msgr2=0x55d2f36b15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:40:59.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.070+0000 7fc57effd640 1 --2- 172.21.15.90:0/2368715395 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d2f3305390 0x55d2f36b15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.070+0000 7fc57effd640 1 -- 172.21.15.90:0/2368715395 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2f331d3f0 con 0x55d2f32dc4d0 2024-04-25T11:40:59.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.071+0000 7fc57effd640 1 --2- 172.21.15.90:0/2368715395 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d2f32dc4d0 0x55d2f36b6210 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x55d2f36a8390 tx=0x7fc570003920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c110c36162f32c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:59.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.071+0000 7fc562ff5640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:40:59.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.072+0000 7fc562ff5640 1 -- 172.21.15.90:0/2368715395 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57006b070 con 0x55d2f32dc4d0 2024-04-25T11:40:59.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.072+0000 7fc562ff5640 1 -- 172.21.15.90:0/2368715395 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fc5700a5ef0 con 0x55d2f32dc4d0 2024-04-25T11:40:59.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.072+0000 7fc58c5aa740 1 -- 172.21.15.90:0/2368715395 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5700be070 con 0x55d2f32dc4d0 2024-04-25T11:40:59.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.073+0000 7fc562ff5640 1 -- 172.21.15.90:0/2368715395 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc5700a6c90 con 0x55d2f32dc4d0 2024-04-25T11:40:59.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.073+0000 7fc58c5aa740 10 client.26414 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:59.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.073+0000 7fc58c5aa740 1 -- 172.21.15.90:0/2368715395 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d2f33e77d0 con 0x55d2f32dc4d0 2024-04-25T11:40:59.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.073+0000 7fc562ff5640 1 -- 172.21.15.90:0/2368715395 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5700ab040 con 0x55d2f32dc4d0 2024-04-25T11:40:59.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.073+0000 7fc562ff5640 1 -- 172.21.15.90:0/2368715395 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fc5700a5960 con 0x55d2f32dc4d0 2024-04-25T11:40:59.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.074+0000 7fc58c5aa740 10 client.26414 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:59.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.074+0000 7fc58c5aa740 10 client.26414 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:40:59.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.074+0000 7fc58c5aa740 20 client.26414 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:59.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.075+0000 7fc58c5aa740 1 --2- 172.21.15.90:0/2368715395 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d2f32c3f60 0x55d2f36d4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:59.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.075+0000 7fc58c5aa740 4 client.26414 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:40:59.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.075+0000 7fc58c5aa740 1 -- 172.21.15.90:0/2368715395 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc5700be070 con 0x55d2f32c3f60 2024-04-25T11:40:59.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.075+0000 7fc57e7fc640 1 --2- 172.21.15.90:0/2368715395 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d2f32c3f60 0x55d2f36d4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:59.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.084+0000 7fc57e7fc640 1 --2- 172.21.15.90:0/2368715395 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d2f32c3f60 0x55d2f36d4530 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd324990ec324cc4 server_cookie=f3b4d4da7b3522c7 in_seq=0 out_seq=0 2024-04-25T11:40:59.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.084+0000 7fc562ff5640 10 client.26414 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:59.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.091+0000 7fc562ff5640 1 -- 172.21.15.90:0/2368715395 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7fc5700be070 con 0x55d2f32c3f60 2024-04-25T11:40:59.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.091+0000 7fc562ff5640 10 client.26414 handle_command_reply: tid=0 2024-04-25T11:40:59.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.091+0000 7fc58c5aa740 1 client.26414 shutdown 2024-04-25T11:40:59.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.099+0000 7fc58c5aa740 1 -- 172.21.15.90:0/2368715395 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d2f32dc4d0 msgr2=0x55d2f36b6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:59.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.099+0000 7fc58c5aa740 1 --2- 172.21.15.90:0/2368715395 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d2f32dc4d0 0x55d2f36b6210 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x55d2f36a8390 tx=0x7fc570003920 comp rx=0 tx=0).stop 2024-04-25T11:40:59.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.102+0000 7fc58c5aa740 1 -- 172.21.15.90:0/2368715395 shutdown_connections 2024-04-25T11:40:59.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.102+0000 7fc58c5aa740 1 --2- 172.21.15.90:0/2368715395 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d2f32c3f60 0x55d2f36d4530 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.102+0000 7fc58c5aa740 1 --2- 172.21.15.90:0/2368715395 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d2f3305390 0x55d2f36b15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.102+0000 7fc58c5aa740 1 --2- 172.21.15.90:0/2368715395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2f32e46b0 0x55d2f36b6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.102+0000 7fc58c5aa740 1 --2- 172.21.15.90:0/2368715395 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d2f32dc4d0 0x55d2f36b6210 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.102+0000 7fc58c5aa740 1 -- 172.21.15.90:0/2368715395 >> 172.21.15.90:0/2368715395 conn(0x55d2f32f94e0 msgr2=0x55d2f36a1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:59.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.103+0000 7fc562ff5640 0 client.26414 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:59.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.103+0000 7fc58c5aa740 1 -- 172.21.15.90:0/2368715395 shutdown_connections 2024-04-25T11:40:59.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.103+0000 7fc58c5aa740 1 -- 172.21.15.90:0/2368715395 wait complete. 2024-04-25T11:40:59.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.104+0000 7fc58c5aa740 20 client.26414 trim_cache size 0 max 16384 2024-04-25T11:40:59.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.141+0000 7fc58c5aa740 1 Processor -- start 2024-04-25T11:40:59.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.141+0000 7fc58c5aa740 1 -- start start 2024-04-25T11:40:59.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.141+0000 7fc58c5aa740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d2f32dc4d0 0x55d2f36b1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:59.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.142+0000 7fc58c5aa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d2f32e46b0 0x55d2f36b0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:59.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.142+0000 7fc58c5aa740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2f3305390 0x55d2f36af120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:59.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.142+0000 7fc58c5aa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc5801adaf0 con 0x55d2f32e46b0 2024-04-25T11:40:59.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.142+0000 7fc58c5aa740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc5801af630 con 0x55d2f32dc4d0 2024-04-25T11:40:59.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.142+0000 7fc58c5aa740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc5801ae1c0 con 0x55d2f3305390 2024-04-25T11:40:59.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.142+0000 7fc57e7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d2f32e46b0 0x55d2f36b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:59.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.142+0000 7fc57e7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d2f32e46b0 0x55d2f36b0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46314/0 (socket says 172.21.15.90:46314) 2024-04-25T11:40:59.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.142+0000 7fc57e7fc640 1 -- 172.21.15.90:0/1966347384 learned_addr learned my addr 172.21.15.90:0/1966347384 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:59.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.142+0000 7fc57e7fc640 1 -- 172.21.15.90:0/1966347384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2f3305390 msgr2=0x55d2f36af120 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:40:59.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.142+0000 7fc57e7fc640 1 --2- 172.21.15.90:0/1966347384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2f3305390 0x55d2f36af120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.142+0000 7fc57e7fc640 1 -- 172.21.15.90:0/1966347384 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d2f32dc4d0 msgr2=0x55d2f36b1590 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:40:59.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.142+0000 7fc57e7fc640 1 --2- 172.21.15.90:0/1966347384 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d2f32dc4d0 0x55d2f36b1590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.143+0000 7fc57e7fc640 1 -- 172.21.15.90:0/1966347384 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2f32c4190 con 0x55d2f32e46b0 2024-04-25T11:40:59.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.143+0000 7fc57e7fc640 1 --2- 172.21.15.90:0/1966347384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d2f32e46b0 0x55d2f36b0530 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fc574009230 tx=0x7fc57403a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88eb35c9708600bc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:59.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.144+0000 7fc564ff9640 1 -- 172.21.15.90:0/1966347384 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc574035040 con 0x55d2f32e46b0 2024-04-25T11:40:59.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.144+0000 7fc564ff9640 1 -- 172.21.15.90:0/1966347384 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc574003aa0 con 0x55d2f32e46b0 2024-04-25T11:40:59.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.144+0000 7fc564ff9640 1 -- 172.21.15.90:0/1966347384 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57404f790 con 0x55d2f32e46b0 2024-04-25T11:40:59.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.145+0000 7fc58c5aa740 1 -- 172.21.15.90:0/1966347384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d2f32e46b0 msgr2=0x55d2f36b0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:59.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.145+0000 7fc58c5aa740 1 --2- 172.21.15.90:0/1966347384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d2f32e46b0 0x55d2f36b0530 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fc574009230 tx=0x7fc57403a000 comp rx=0 tx=0).stop 2024-04-25T11:40:59.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.145+0000 7fc58c5aa740 1 -- 172.21.15.90:0/1966347384 shutdown_connections 2024-04-25T11:40:59.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.145+0000 7fc58c5aa740 1 --2- 172.21.15.90:0/1966347384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2f3305390 0x55d2f36af120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.145+0000 7fc58c5aa740 1 --2- 172.21.15.90:0/1966347384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d2f32e46b0 0x55d2f36b0530 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.145+0000 7fc58c5aa740 1 --2- 172.21.15.90:0/1966347384 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d2f32dc4d0 0x55d2f36b1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.145+0000 7fc58c5aa740 1 -- 172.21.15.90:0/1966347384 >> 172.21.15.90:0/1966347384 conn(0x55d2f32f94e0 msgr2=0x55d2f36b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:59.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.145+0000 7fc58c5aa740 1 -- 172.21.15.90:0/1966347384 shutdown_connections 2024-04-25T11:40:59.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.145+0000 7fc58c5aa740 1 -- 172.21.15.90:0/1966347384 wait complete. 2024-04-25T11:40:59.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.146+0000 7fc58c5aa740 1 -- start start 2024-04-25T11:40:59.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.146+0000 7fc58c5aa740 1 Processor -- start 2024-04-25T11:40:59.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.147+0000 7fc58c5aa740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2f32dc4d0 0x55d2f36c1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:59.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.147+0000 7fc58c5aa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d2f32e46b0 0x55d2f36c1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:59.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.147+0000 7fc58c5aa740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d2f3305390 0x55d2f36bc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:59.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.147+0000 7fc58c5aa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc5801aef60 con 0x55d2f32e46b0 2024-04-25T11:40:59.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.147+0000 7fc58c5aa740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc5801ae890 con 0x55d2f3305390 2024-04-25T11:40:59.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.147+0000 7fc58c5aa740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc5801abad0 con 0x55d2f32dc4d0 2024-04-25T11:40:59.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.147+0000 7fc58c5aa740 10 client.0 _ll_register_callbacks cb 0x7fc5700a6670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:40:59.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.147+0000 7fc57e7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d2f32e46b0 0x55d2f36c1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:59.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.147+0000 7fc57e7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d2f32e46b0 0x55d2f36c1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46318/0 (socket says 172.21.15.90:46318) 2024-04-25T11:40:59.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.147+0000 7fc57e7fc640 1 -- 172.21.15.90:0/265149978 learned_addr learned my addr 172.21.15.90:0/265149978 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:40:59.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.148+0000 7fc57e7fc640 1 -- 172.21.15.90:0/265149978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2f32dc4d0 msgr2=0x55d2f36c1200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:40:59.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.148+0000 7fc57e7fc640 1 --2- 172.21.15.90:0/265149978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2f32dc4d0 0x55d2f36c1200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.148+0000 7fc57e7fc640 1 -- 172.21.15.90:0/265149978 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d2f3305390 msgr2=0x55d2f36bc6e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:40:59.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.148+0000 7fc57e7fc640 1 --2- 172.21.15.90:0/265149978 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d2f3305390 0x55d2f36bc6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.148+0000 7fc57e7fc640 1 -- 172.21.15.90:0/265149978 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2f36bcdf0 con 0x55d2f32e46b0 2024-04-25T11:40:59.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.148+0000 7fc57e7fc640 1 --2- 172.21.15.90:0/265149978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d2f32e46b0 0x55d2f36c1740 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fc574009850 tx=0x7fc574032e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9979e053bb349a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:40:59.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.148+0000 7fc562ff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:40:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.148+0000 7fc562ff5640 1 -- 172.21.15.90:0/265149978 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc574080070 con 0x55d2f32e46b0 2024-04-25T11:40:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.148+0000 7fc562ff5640 1 -- 172.21.15.90:0/265149978 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fc5740035f0 con 0x55d2f32e46b0 2024-04-25T11:40:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.148+0000 7fc58c5aa740 1 -- 172.21.15.90:0/265149978 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc574003aa0 con 0x55d2f32e46b0 2024-04-25T11:40:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.149+0000 7fc562ff5640 1 -- 172.21.15.90:0/265149978 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc57408c050 con 0x55d2f32e46b0 2024-04-25T11:40:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.149+0000 7fc58c5aa740 10 client.17370 fetch_fsmap learned FSMap version 28 2024-04-25T11:40:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.149+0000 7fc58c5aa740 1 -- 172.21.15.90:0/265149978 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d2f36c9870 con 0x55d2f32e46b0 2024-04-25T11:40:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.149+0000 7fc562ff5640 1 -- 172.21.15.90:0/265149978 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc574003aa0 con 0x55d2f32e46b0 2024-04-25T11:40:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.149+0000 7fc562ff5640 1 -- 172.21.15.90:0/265149978 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fc574049450 con 0x55d2f32e46b0 2024-04-25T11:40:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.149+0000 7fc58c5aa740 10 client.17370 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:40:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.149+0000 7fc58c5aa740 10 client.17370 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:40:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.149+0000 7fc58c5aa740 20 client.17370 populate_metadata read hostname 'smithi090' 2024-04-25T11:40:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.149+0000 7fc58c5aa740 1 --2- 172.21.15.90:0/265149978 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d2f32993a0 0x55d2f36cbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:40:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.149+0000 7fc58c5aa740 4 client.17370 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:40:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.150+0000 7fc58c5aa740 1 -- 172.21.15.90:0/265149978 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d2f3619990 con 0x55d2f32993a0 2024-04-25T11:40:59.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.153+0000 7fc57effd640 1 --2- 172.21.15.90:0/265149978 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d2f32993a0 0x55d2f36cbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:40:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.164+0000 7fc562ff5640 10 client.17370 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.164+0000 7fc57effd640 1 --2- 172.21.15.90:0/265149978 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d2f32993a0 0x55d2f36cbfb0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=192779647887e001 server_cookie=c6b6f36f0f778768 in_seq=0 out_seq=0 2024-04-25T11:40:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.166+0000 7fc562ff5640 1 -- 172.21.15.90:0/265149978 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d2f3619990 con 0x55d2f32993a0 2024-04-25T11:40:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.166+0000 7fc562ff5640 10 client.17370 handle_command_reply: tid=0 2024-04-25T11:40:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.166+0000 7fc58c5aa740 1 client.17370 shutdown 2024-04-25T11:40:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.166+0000 7fc58c5aa740 1 -- 172.21.15.90:0/265149978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d2f32e46b0 msgr2=0x55d2f36c1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.166+0000 7fc58c5aa740 1 --2- 172.21.15.90:0/265149978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d2f32e46b0 0x55d2f36c1740 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fc574009850 tx=0x7fc574032e10 comp rx=0 tx=0).stop 2024-04-25T11:40:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.167+0000 7fc58c5aa740 1 -- 172.21.15.90:0/265149978 shutdown_connections 2024-04-25T11:40:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.167+0000 7fc58c5aa740 1 --2- 172.21.15.90:0/265149978 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d2f32993a0 0x55d2f36cbfb0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.167+0000 7fc58c5aa740 1 --2- 172.21.15.90:0/265149978 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d2f3305390 0x55d2f36bc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.167+0000 7fc58c5aa740 1 --2- 172.21.15.90:0/265149978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d2f32e46b0 0x55d2f36c1740 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.167+0000 7fc562ff5640 0 client.17370 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:40:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.167+0000 7fc58c5aa740 1 --2- 172.21.15.90:0/265149978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2f32dc4d0 0x55d2f36c1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.167+0000 7fc58c5aa740 1 -- 172.21.15.90:0/265149978 >> 172.21.15.90:0/265149978 conn(0x55d2f32f94e0 msgr2=0x55d2f36a65a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.167+0000 7fc58c5aa740 1 -- 172.21.15.90:0/265149978 shutdown_connections 2024-04-25T11:40:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.167+0000 7fc58c5aa740 1 -- 172.21.15.90:0/265149978 wait complete. 2024-04-25T11:40:59.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.167+0000 7fc58c5aa740 20 client.17370 trim_cache size 0 max 16384 2024-04-25T11:40:59.157 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:40:59.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.167+0000 7fc585625640 1 -- 172.21.15.90:0/704554465 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc5680afce0 msgr2=0x7fc5680b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:59.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.167+0000 7fc585625640 1 --2- 172.21.15.90:0/704554465 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc5680afce0 0x7fc5680b21f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc57402a9a0 tx=0x7fc574031400 comp rx=0 tx=0).stop 2024-04-25T11:40:59.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.167+0000 7fc585625640 1 -- 172.21.15.90:0/704554465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc580108a60 msgr2=0x7fc580199240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:40:59.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.167+0000 7fc585625640 1 --2- 172.21.15.90:0/704554465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc580108a60 0x7fc580199240 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fc57000c9a0 tx=0x7fc57000ce60 comp rx=0 tx=0).stop 2024-04-25T11:40:59.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.168+0000 7fc585625640 1 -- 172.21.15.90:0/704554465 shutdown_connections 2024-04-25T11:40:59.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.168+0000 7fc585625640 1 --2- 172.21.15.90:0/704554465 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc5680afce0 0x7fc5680b21f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.168+0000 7fc585625640 1 --2- 172.21.15.90:0/704554465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc58019da60 0x7fc58019de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.168+0000 7fc585625640 1 --2- 172.21.15.90:0/704554465 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc580108e70 0x7fc5801997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.168+0000 7fc585625640 1 --2- 172.21.15.90:0/704554465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc580108a60 0x7fc580199240 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:40:59.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.168+0000 7fc585625640 1 -- 172.21.15.90:0/704554465 >> 172.21.15.90:0/704554465 conn(0x7fc580077880 msgr2=0x7fc5801040f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:40:59.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.168+0000 7fc585625640 1 -- 172.21.15.90:0/704554465 shutdown_connections 2024-04-25T11:40:59.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:40:59.168+0000 7fc585625640 1 -- 172.21.15.90:0/704554465 wait complete. 2024-04-25T11:40:59.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:40:59 smithi156 ceph-mon[33045]: pgmap v180: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 471 MiB/s wr, 499 op/s 2024-04-25T11:40:59.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:40:59 smithi090 ceph-mon[29575]: pgmap v180: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 471 MiB/s wr, 499 op/s 2024-04-25T11:41:00.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:40:59 smithi138 ceph-mon[32328]: pgmap v180: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 471 MiB/s wr, 499 op/s 2024-04-25T11:41:00.171 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T11:41:00.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:41:00.171 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T11:41:00.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.311+0000 7f887ec67640 1 -- 172.21.15.90:0/3247900078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f887810daa0 msgr2=0x7f887810fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:00.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.311+0000 7f887ec67640 1 --2- 172.21.15.90:0/3247900078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f887810daa0 0x7f887810fec0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8864009900 tx=0x7f886402a410 comp rx=0 tx=0).stop 2024-04-25T11:41:00.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.311+0000 7f887ec67640 1 -- 172.21.15.90:0/3247900078 shutdown_connections 2024-04-25T11:41:00.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.311+0000 7f887ec67640 1 --2- 172.21.15.90:0/3247900078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8878110400 0x7f8878112820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.311+0000 7f887ec67640 1 --2- 172.21.15.90:0/3247900078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f887810daa0 0x7f887810fec0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.311+0000 7f887ec67640 1 --2- 172.21.15.90:0/3247900078 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8878102860 0x7f8878102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.311+0000 7f887ec67640 1 -- 172.21.15.90:0/3247900078 >> 172.21.15.90:0/3247900078 conn(0x7f8878078100 msgr2=0x7f8878078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:00.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.311+0000 7f887ec67640 1 -- 172.21.15.90:0/3247900078 shutdown_connections 2024-04-25T11:41:00.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.311+0000 7f887ec67640 1 -- 172.21.15.90:0/3247900078 wait complete. 2024-04-25T11:41:00.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.312+0000 7f887ec67640 1 Processor -- start 2024-04-25T11:41:00.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.312+0000 7f887ec67640 1 -- start start 2024-04-25T11:41:00.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.313+0000 7f887ec67640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8878102860 0x7f88781144d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.313+0000 7f887ec67640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f887810daa0 0x7f8878114a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.313+0000 7f887ec67640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8878110400 0x7f887811b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.313+0000 7f887ec67640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8878106220 con 0x7f8878110400 2024-04-25T11:41:00.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.313+0000 7f887ec67640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f88781060a0 con 0x7f8878102860 2024-04-25T11:41:00.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.313+0000 7f887ec67640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f88781063a0 con 0x7f887810daa0 2024-04-25T11:41:00.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.313+0000 7f887d204640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8878110400 0x7f887811b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.313+0000 7f887d204640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8878110400 0x7f887811b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46340/0 (socket says 172.21.15.90:46340) 2024-04-25T11:41:00.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.313+0000 7f887d204640 1 -- 172.21.15.90:0/1251564228 learned_addr learned my addr 172.21.15.90:0/1251564228 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:00.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.313+0000 7f887ca03640 1 --2- 172.21.15.90:0/1251564228 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8878102860 0x7f88781144d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.313+0000 7f887d204640 1 -- 172.21.15.90:0/1251564228 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f887810daa0 msgr2=0x7f8878114a10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:41:00.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.314+0000 7f887d204640 1 --2- 172.21.15.90:0/1251564228 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f887810daa0 0x7f8878114a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.314+0000 7f887d204640 1 -- 172.21.15.90:0/1251564228 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8878102860 msgr2=0x7f88781144d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:00.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.314+0000 7f887d204640 1 --2- 172.21.15.90:0/1251564228 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8878102860 0x7f88781144d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.314+0000 7f887d204640 1 -- 172.21.15.90:0/1251564228 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f887811b940 con 0x7f8878110400 2024-04-25T11:41:00.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.314+0000 7f887ca03640 1 --2- 172.21.15.90:0/1251564228 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8878102860 0x7f88781144d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:41:00.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.314+0000 7f887d204640 1 --2- 172.21.15.90:0/1251564228 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8878110400 0x7f887811b350 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f887000eb10 tx=0x7f887000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:00.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.314+0000 7f8875ffb640 1 -- 172.21.15.90:0/1251564228 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8870004b20 con 0x7f8878110400 2024-04-25T11:41:00.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.314+0000 7f887ec67640 1 -- 172.21.15.90:0/1251564228 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f887811bc30 con 0x7f8878110400 2024-04-25T11:41:00.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.314+0000 7f8875ffb640 1 -- 172.21.15.90:0/1251564228 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8870004cc0 con 0x7f8878110400 2024-04-25T11:41:00.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.316+0000 7f8875ffb640 1 -- 172.21.15.90:0/1251564228 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8870010670 con 0x7f8878110400 2024-04-25T11:41:00.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.321+0000 7f887ec67640 1 -- 172.21.15.90:0/1251564228 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f887811c180 con 0x7f8878110400 2024-04-25T11:41:00.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.321+0000 7f8875ffb640 1 -- 172.21.15.90:0/1251564228 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f8870005000 con 0x7f8878110400 2024-04-25T11:41:00.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.322+0000 7f8875ffb640 1 --2- 172.21.15.90:0/1251564228 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f88680afb40 0x7f88680b2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.323+0000 7f887ca03640 1 --2- 172.21.15.90:0/1251564228 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f88680afb40 0x7f88680b2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.323+0000 7f8875ffb640 1 -- 172.21.15.90:0/1251564228 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f88700e1180 con 0x7f8878110400 2024-04-25T11:41:00.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.323+0000 7f887ca03640 1 --2- 172.21.15.90:0/1251564228 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f88680afb40 0x7f88680b2050 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f886c009c20 tx=0x7f886c005f90 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:00.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.325+0000 7f8885bf7740 1 Processor -- start 2024-04-25T11:41:00.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.325+0000 7f8885bf7740 1 -- start start 2024-04-25T11:41:00.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.325+0000 7f8885bf7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620f5e29410 0x5620f61c9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.325+0000 7f8885bf7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5620f5e00550 0x5620f61c9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.325+0000 7f8885bf7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e08730 0x5620f61ca0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.325+0000 7f8885bf7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5620f612fb00 con 0x5620f5e08730 2024-04-25T11:41:00.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.325+0000 7f8885bf7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5620f60cb920 con 0x5620f5e00550 2024-04-25T11:41:00.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.326+0000 7f8885bf7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5620f612bac0 con 0x5620f5e29410 2024-04-25T11:41:00.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.326+0000 7f887d204640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e08730 0x5620f61ca0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.326+0000 7f887d204640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e08730 0x5620f61ca0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46350/0 (socket says 172.21.15.90:46350) 2024-04-25T11:41:00.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.326+0000 7f887d204640 1 -- 172.21.15.90:0/1381531616 learned_addr learned my addr 172.21.15.90:0/1381531616 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:00.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.327+0000 7f887d204640 1 -- 172.21.15.90:0/1381531616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620f5e29410 msgr2=0x5620f61c9620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:41:00.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.327+0000 7f887d204640 1 --2- 172.21.15.90:0/1381531616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620f5e29410 0x5620f61c9620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.327+0000 7f887d204640 1 -- 172.21.15.90:0/1381531616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5620f5e00550 msgr2=0x5620f61c9b60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:41:00.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.327+0000 7f887d204640 1 --2- 172.21.15.90:0/1381531616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5620f5e00550 0x5620f61c9b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.327+0000 7f887d204640 1 -- 172.21.15.90:0/1381531616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620f5f097e0 con 0x5620f5e08730 2024-04-25T11:41:00.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.327+0000 7f887d204640 1 --2- 172.21.15.90:0/1381531616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e08730 0x5620f61ca0a0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f88700a4440 tx=0x7f8870004aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3190b3ae5fc6f215 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:00.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.329+0000 7f8862ffd640 1 -- 172.21.15.90:0/1381531616 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88700e3060 con 0x5620f5e08730 2024-04-25T11:41:00.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.329+0000 7f8862ffd640 1 -- 172.21.15.90:0/1381531616 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88700c1070 con 0x5620f5e08730 2024-04-25T11:41:00.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.329+0000 7f8862ffd640 1 -- 172.21.15.90:0/1381531616 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88700b0030 con 0x5620f5e08730 2024-04-25T11:41:00.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.330+0000 7f8885bf7740 1 -- 172.21.15.90:0/1381531616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e08730 msgr2=0x5620f61ca0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:00.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.330+0000 7f8885bf7740 1 --2- 172.21.15.90:0/1381531616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e08730 0x5620f61ca0a0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f88700a4440 tx=0x7f8870004aa0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.331+0000 7f8885bf7740 1 -- 172.21.15.90:0/1381531616 shutdown_connections 2024-04-25T11:41:00.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.331+0000 7f8885bf7740 1 --2- 172.21.15.90:0/1381531616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e08730 0x5620f61ca0a0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.331+0000 7f8885bf7740 1 --2- 172.21.15.90:0/1381531616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5620f5e00550 0x5620f61c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.331+0000 7f8885bf7740 1 --2- 172.21.15.90:0/1381531616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620f5e29410 0x5620f61c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.331+0000 7f8885bf7740 1 -- 172.21.15.90:0/1381531616 >> 172.21.15.90:0/1381531616 conn(0x5620f5e1d560 msgr2=0x5620f61c5dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:00.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.331+0000 7f8885bf7740 1 -- 172.21.15.90:0/1381531616 shutdown_connections 2024-04-25T11:41:00.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.331+0000 7f8885bf7740 1 -- 172.21.15.90:0/1381531616 wait complete. 2024-04-25T11:41:00.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.332+0000 7f8885bf7740 1 -- start start 2024-04-25T11:41:00.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.333+0000 7f8885bf7740 1 Processor -- start 2024-04-25T11:41:00.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.334+0000 7f8885bf7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e00550 0x5620f61d3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.334+0000 7f8885bf7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620f5e08730 0x5620f61d4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.334+0000 7f8885bf7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5620f5e29410 0x5620f61eae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.334+0000 7f8885bf7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5620f61e7920 con 0x5620f5e00550 2024-04-25T11:41:00.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.334+0000 7f887ca03640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e00550 0x5620f61d3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.334+0000 7f887ca03640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e00550 0x5620f61d3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46358/0 (socket says 172.21.15.90:46358) 2024-04-25T11:41:00.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.334+0000 7f887ca03640 1 -- 172.21.15.90:0/588951046 learned_addr learned my addr 172.21.15.90:0/588951046 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:00.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.334+0000 7f8885bf7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5620f60c95d0 con 0x5620f5e29410 2024-04-25T11:41:00.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.335+0000 7f8885bf7740 1 -- 172.21.15.90:0/588951046 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5620f61e7aa0 con 0x5620f5e08730 2024-04-25T11:41:00.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.335+0000 7f8885bf7740 10 client.0 _ll_register_callbacks cb 0x5620f60b7080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:00.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.336+0000 7f887ca03640 1 -- 172.21.15.90:0/588951046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620f5e08730 msgr2=0x5620f61d4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:00.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.336+0000 7f887ca03640 1 --2- 172.21.15.90:0/588951046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620f5e08730 0x5620f61d4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.336+0000 7f887ca03640 1 -- 172.21.15.90:0/588951046 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5620f5e29410 msgr2=0x5620f61eae10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:41:00.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.336+0000 7f887ca03640 1 --2- 172.21.15.90:0/588951046 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5620f5e29410 0x5620f61eae10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.336+0000 7f887ca03640 1 -- 172.21.15.90:0/588951046 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620f5f69f70 con 0x5620f5e00550 2024-04-25T11:41:00.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.338+0000 7f887ca03640 1 --2- 172.21.15.90:0/588951046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e00550 0x5620f61d3cd0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f886c012c10 tx=0x7f886c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56317320383647e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:00.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.338+0000 7f8857fff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:41:00.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.338+0000 7f8857fff640 1 -- 172.21.15.90:0/588951046 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886c012e40 con 0x5620f5e00550 2024-04-25T11:41:00.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.339+0000 7f8857fff640 1 -- 172.21.15.90:0/588951046 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f886c01f420 con 0x5620f5e00550 2024-04-25T11:41:00.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.339+0000 7f8885bf7740 1 -- 172.21.15.90:0/588951046 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88700c1070 con 0x5620f5e00550 2024-04-25T11:41:00.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.343+0000 7f8857fff640 1 -- 172.21.15.90:0/588951046 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f886c009e10 con 0x5620f5e00550 2024-04-25T11:41:00.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.343+0000 7f8885bf7740 10 client.17382 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:00.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.344+0000 7f8885bf7740 1 -- 172.21.15.90:0/588951046 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5620f5e01170 con 0x5620f5e00550 2024-04-25T11:41:00.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.350+0000 7f8857fff640 1 -- 172.21.15.90:0/588951046 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f886c01fad0 con 0x5620f5e00550 2024-04-25T11:41:00.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.350+0000 7f8857fff640 1 -- 172.21.15.90:0/588951046 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f886c0279f0 con 0x5620f5e00550 2024-04-25T11:41:00.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.350+0000 7f8885bf7740 10 client.17382 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:00.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.350+0000 7f8885bf7740 10 client.17382 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:41:00.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.350+0000 7f8885bf7740 20 client.17382 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:00.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.351+0000 7f8885bf7740 1 --2- 172.21.15.90:0/588951046 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5620f5de81f0 0x5620f61f70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.351+0000 7f8885bf7740 4 client.17382 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:41:00.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.351+0000 7f8885bf7740 1 -- 172.21.15.90:0/588951046 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f88700c1070 con 0x5620f5de81f0 2024-04-25T11:41:00.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.358+0000 7f8877fff640 1 --2- 172.21.15.90:0/588951046 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5620f5de81f0 0x5620f61f70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.371+0000 7f8877fff640 1 --2- 172.21.15.90:0/588951046 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5620f5de81f0 0x5620f61f70f0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eeb7015ca69f951 server_cookie=bab8a9cd78480a9f in_seq=0 out_seq=0 2024-04-25T11:41:00.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.372+0000 7f8857fff640 10 client.17382 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:00.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.376+0000 7f8857fff640 1 -- 172.21.15.90:0/588951046 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f88700c1070 con 0x5620f5de81f0 2024-04-25T11:41:00.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.376+0000 7f8857fff640 10 client.17382 handle_command_reply: tid=0 2024-04-25T11:41:00.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.376+0000 7f8885bf7740 1 client.17382 shutdown 2024-04-25T11:41:00.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.378+0000 7f8885bf7740 1 -- 172.21.15.90:0/588951046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e00550 msgr2=0x5620f61d3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:00.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.378+0000 7f8885bf7740 1 --2- 172.21.15.90:0/588951046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e00550 0x5620f61d3cd0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f886c012c10 tx=0x7f886c004730 comp rx=0 tx=0).stop 2024-04-25T11:41:00.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.383+0000 7f8885bf7740 1 -- 172.21.15.90:0/588951046 shutdown_connections 2024-04-25T11:41:00.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.383+0000 7f8885bf7740 1 --2- 172.21.15.90:0/588951046 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5620f5de81f0 0x5620f61f70f0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.383+0000 7f8885bf7740 1 --2- 172.21.15.90:0/588951046 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5620f5e29410 0x5620f61eae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.383+0000 7f8885bf7740 1 --2- 172.21.15.90:0/588951046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620f5e08730 0x5620f61d4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.383+0000 7f8885bf7740 1 --2- 172.21.15.90:0/588951046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e00550 0x5620f61d3cd0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.383+0000 7f8885bf7740 1 -- 172.21.15.90:0/588951046 >> 172.21.15.90:0/588951046 conn(0x5620f5e1d560 msgr2=0x5620f61cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:00.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.384+0000 7f8857fff640 0 client.17382 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:00.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.384+0000 7f8885bf7740 1 -- 172.21.15.90:0/588951046 shutdown_connections 2024-04-25T11:41:00.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.385+0000 7f8885bf7740 1 -- 172.21.15.90:0/588951046 wait complete. 2024-04-25T11:41:00.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.387+0000 7f8885bf7740 20 client.17382 trim_cache size 0 max 16384 2024-04-25T11:41:00.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.438+0000 7f8885bf7740 1 Processor -- start 2024-04-25T11:41:00.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.438+0000 7f8885bf7740 1 -- start start 2024-04-25T11:41:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.438+0000 7f8885bf7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5620f5e00550 0x5620f61c4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.438+0000 7f8885bf7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e08730 0x5620f61ead10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.438+0000 7f8885bf7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620f5e29410 0x5620f61eb250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.439+0000 7f8885bf7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8878074610 con 0x5620f5e08730 2024-04-25T11:41:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.439+0000 7f8885bf7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f88781b7bc0 con 0x5620f5e00550 2024-04-25T11:41:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.439+0000 7f8885bf7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f88781b6840 con 0x5620f5e29410 2024-04-25T11:41:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.439+0000 7f8877fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e08730 0x5620f61ead10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.439+0000 7f8877fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e08730 0x5620f61ead10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46372/0 (socket says 172.21.15.90:46372) 2024-04-25T11:41:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.439+0000 7f8877fff640 1 -- 172.21.15.90:0/1185032960 learned_addr learned my addr 172.21.15.90:0/1185032960 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.439+0000 7f887d204640 1 --2- 172.21.15.90:0/1185032960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620f5e29410 0x5620f61eb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.439+0000 7f8877fff640 1 -- 172.21.15.90:0/1185032960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620f5e29410 msgr2=0x5620f61eb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.439+0000 7f8877fff640 1 --2- 172.21.15.90:0/1185032960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620f5e29410 0x5620f61eb250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.439+0000 7f8877fff640 1 -- 172.21.15.90:0/1185032960 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5620f5e00550 msgr2=0x5620f61c4b30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:41:00.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.439+0000 7f8877fff640 1 --2- 172.21.15.90:0/1185032960 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5620f5e00550 0x5620f61c4b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.439+0000 7f8877fff640 1 -- 172.21.15.90:0/1185032960 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620f5de8420 con 0x5620f5e08730 2024-04-25T11:41:00.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.439+0000 7f8877fff640 1 --2- 172.21.15.90:0/1185032960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e08730 0x5620f61ead10 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f8878075fd0 tx=0x7f88640042b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1cb627b7833745b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:00.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.440+0000 7f88627fc640 1 -- 172.21.15.90:0/1185032960 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8864046020 con 0x5620f5e08730 2024-04-25T11:41:00.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.440+0000 7f88627fc640 1 -- 172.21.15.90:0/1185032960 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8864042070 con 0x5620f5e08730 2024-04-25T11:41:00.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.440+0000 7f88627fc640 1 -- 172.21.15.90:0/1185032960 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8864035040 con 0x5620f5e08730 2024-04-25T11:41:00.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.441+0000 7f8885bf7740 1 -- 172.21.15.90:0/1185032960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e08730 msgr2=0x5620f61ead10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:00.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.441+0000 7f8885bf7740 1 --2- 172.21.15.90:0/1185032960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e08730 0x5620f61ead10 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f8878075fd0 tx=0x7f88640042b0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.441+0000 7f8885bf7740 1 -- 172.21.15.90:0/1185032960 shutdown_connections 2024-04-25T11:41:00.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.441+0000 7f8885bf7740 1 --2- 172.21.15.90:0/1185032960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620f5e29410 0x5620f61eb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.441+0000 7f8885bf7740 1 --2- 172.21.15.90:0/1185032960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e08730 0x5620f61ead10 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.441+0000 7f8885bf7740 1 --2- 172.21.15.90:0/1185032960 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5620f5e00550 0x5620f61c4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.441+0000 7f8885bf7740 1 -- 172.21.15.90:0/1185032960 >> 172.21.15.90:0/1185032960 conn(0x5620f5e1d560 msgr2=0x5620f61cbba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:00.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.441+0000 7f8885bf7740 1 -- 172.21.15.90:0/1185032960 shutdown_connections 2024-04-25T11:41:00.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.442+0000 7f8885bf7740 1 -- 172.21.15.90:0/1185032960 wait complete. 2024-04-25T11:41:00.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.442+0000 7f8885bf7740 1 -- start start 2024-04-25T11:41:00.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.442+0000 7f8885bf7740 1 Processor -- start 2024-04-25T11:41:00.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.443+0000 7f8885bf7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5620f5e00550 0x5620f61f1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.443+0000 7f8885bf7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620f5e08730 0x5620f61ec8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.443+0000 7f8885bf7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e29410 0x5620f61ecdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.443+0000 7f8885bf7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f88781b74f0 con 0x5620f5e29410 2024-04-25T11:41:00.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.443+0000 7f8885bf7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f88781b6e20 con 0x5620f5e00550 2024-04-25T11:41:00.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.443+0000 7f8885bf7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f887806a710 con 0x5620f5e08730 2024-04-25T11:41:00.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.443+0000 7f8885bf7740 10 client.0 _ll_register_callbacks cb 0x7f886c0128c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:00.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.443+0000 7f887d204640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e29410 0x5620f61ecdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.443+0000 7f887d204640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e29410 0x5620f61ecdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46388/0 (socket says 172.21.15.90:46388) 2024-04-25T11:41:00.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.443+0000 7f887d204640 1 -- 172.21.15.90:0/311303974 learned_addr learned my addr 172.21.15.90:0/311303974 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:00.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.444+0000 7f887ca03640 1 --2- 172.21.15.90:0/311303974 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5620f5e00550 0x5620f61f1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.444+0000 7f887d204640 1 -- 172.21.15.90:0/311303974 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620f5e08730 msgr2=0x5620f61ec8b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:41:00.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.444+0000 7f887d204640 1 --2- 172.21.15.90:0/311303974 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620f5e08730 0x5620f61ec8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.444+0000 7f887d204640 1 -- 172.21.15.90:0/311303974 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5620f5e00550 msgr2=0x5620f61f1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:00.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.444+0000 7f887d204640 1 --2- 172.21.15.90:0/311303974 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5620f5e00550 0x5620f61f1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.444+0000 7f887d204640 1 -- 172.21.15.90:0/311303974 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620f61ed330 con 0x5620f5e29410 2024-04-25T11:41:00.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.444+0000 7f887ca03640 1 --2- 172.21.15.90:0/311303974 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5620f5e00550 0x5620f61f1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:41:00.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.444+0000 7f8857fff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:41:00.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.444+0000 7f887d204640 1 --2- 172.21.15.90:0/311303974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e29410 0x5620f61ecdf0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f88700e10f0 tx=0x7f88700ab3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca96d3e13e08c255 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:00.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.445+0000 7f8857fff640 1 -- 172.21.15.90:0/311303974 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887006b020 con 0x5620f5e29410 2024-04-25T11:41:00.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.445+0000 7f8857fff640 1 -- 172.21.15.90:0/311303974 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f8870067da0 con 0x5620f5e29410 2024-04-25T11:41:00.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.445+0000 7f8885bf7740 1 -- 172.21.15.90:0/311303974 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8864042070 con 0x5620f5e29410 2024-04-25T11:41:00.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.445+0000 7f8857fff640 1 -- 172.21.15.90:0/311303974 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f88700a6ce0 con 0x5620f5e29410 2024-04-25T11:41:00.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.445+0000 7f8885bf7740 10 client.17388 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:00.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.445+0000 7f8885bf7740 1 -- 172.21.15.90:0/311303974 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5620f625f040 con 0x5620f5e29410 2024-04-25T11:41:00.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.445+0000 7f8857fff640 1 -- 172.21.15.90:0/311303974 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8864042070 con 0x5620f5e29410 2024-04-25T11:41:00.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.446+0000 7f8857fff640 1 -- 172.21.15.90:0/311303974 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f88700a6430 con 0x5620f5e29410 2024-04-25T11:41:00.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.446+0000 7f8885bf7740 10 client.17388 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:00.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.446+0000 7f8885bf7740 10 client.17388 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:41:00.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.446+0000 7f8885bf7740 20 client.17388 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:00.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.446+0000 7f8885bf7740 1 --2- 172.21.15.90:0/311303974 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5620f5dbd630 0x5620f6261330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.446+0000 7f8885bf7740 4 client.17388 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T11:41:00.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.446+0000 7f8885bf7740 1 -- 172.21.15.90:0/311303974 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5620f613d710 con 0x5620f5dbd630 2024-04-25T11:41:00.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.450+0000 7f887ca03640 1 --2- 172.21.15.90:0/311303974 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5620f5dbd630 0x5620f6261330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.456+0000 7f887ca03640 1 --2- 172.21.15.90:0/311303974 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5620f5dbd630 0x5620f6261330 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8100baf8a9e7927 server_cookie=1a79a41a66235bf8 in_seq=0 out_seq=0 2024-04-25T11:41:00.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.456+0000 7f8857fff640 10 client.17388 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:00.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.612+0000 7f8857fff640 1 -- 172.21.15.90:0/311303974 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5620f613d710 con 0x5620f5dbd630 2024-04-25T11:41:00.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.612+0000 7f8857fff640 10 client.17388 handle_command_reply: tid=0 2024-04-25T11:41:00.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.612+0000 7f8885bf7740 1 client.17388 shutdown 2024-04-25T11:41:00.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.612+0000 7f8885bf7740 1 -- 172.21.15.90:0/311303974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e29410 msgr2=0x5620f61ecdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:00.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.612+0000 7f8885bf7740 1 --2- 172.21.15.90:0/311303974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e29410 0x5620f61ecdf0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f88700e10f0 tx=0x7f88700ab3a0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.612+0000 7f8885bf7740 1 -- 172.21.15.90:0/311303974 shutdown_connections 2024-04-25T11:41:00.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.612+0000 7f8885bf7740 1 --2- 172.21.15.90:0/311303974 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5620f5dbd630 0x5620f6261330 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.612+0000 7f8885bf7740 1 --2- 172.21.15.90:0/311303974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5620f5e29410 0x5620f61ecdf0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.612+0000 7f8885bf7740 1 --2- 172.21.15.90:0/311303974 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620f5e08730 0x5620f61ec8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.612+0000 7f8885bf7740 1 --2- 172.21.15.90:0/311303974 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5620f5e00550 0x5620f61f1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.612+0000 7f8857fff640 0 client.17388 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:00.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.612+0000 7f8885bf7740 1 -- 172.21.15.90:0/311303974 >> 172.21.15.90:0/311303974 conn(0x5620f5e1d560 msgr2=0x5620f61cbaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:00.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.612+0000 7f8885bf7740 1 -- 172.21.15.90:0/311303974 shutdown_connections 2024-04-25T11:41:00.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.612+0000 7f8885bf7740 1 -- 172.21.15.90:0/311303974 wait complete. 2024-04-25T11:41:00.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.612+0000 7f8885bf7740 20 client.17388 trim_cache size 0 max 16384 2024-04-25T11:41:00.600 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:41:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-25T11:41:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "170cd0c0-9c0e-42a6-bb1a-42fd52717626", 2024-04-25T11:41:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-25T11:41:00.601 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:41:00.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.613+0000 7f887ec67640 1 -- 172.21.15.90:0/1251564228 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f88680afb40 msgr2=0x7f88680b2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:00.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.613+0000 7f887ec67640 1 --2- 172.21.15.90:0/1251564228 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f88680afb40 0x7f88680b2050 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f886c009c20 tx=0x7f886c005f90 comp rx=0 tx=0).stop 2024-04-25T11:41:00.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.613+0000 7f887ec67640 1 -- 172.21.15.90:0/1251564228 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8878110400 msgr2=0x7f887811b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:00.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.613+0000 7f887ec67640 1 --2- 172.21.15.90:0/1251564228 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8878110400 0x7f887811b350 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f887000eb10 tx=0x7f887000c2d0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.613+0000 7f887ec67640 1 -- 172.21.15.90:0/1251564228 shutdown_connections 2024-04-25T11:41:00.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.613+0000 7f887ec67640 1 --2- 172.21.15.90:0/1251564228 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f88680afb40 0x7f88680b2050 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.613+0000 7f887ec67640 1 --2- 172.21.15.90:0/1251564228 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8878110400 0x7f887811b350 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.613+0000 7f887ec67640 1 --2- 172.21.15.90:0/1251564228 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f887810daa0 0x7f8878114a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.613+0000 7f887ec67640 1 --2- 172.21.15.90:0/1251564228 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8878102860 0x7f88781144d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.613+0000 7f887ec67640 1 -- 172.21.15.90:0/1251564228 >> 172.21.15.90:0/1251564228 conn(0x7f8878078100 msgr2=0x7f8878101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:00.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.613+0000 7f887ec67640 1 -- 172.21.15.90:0/1251564228 shutdown_connections 2024-04-25T11:41:00.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.613+0000 7f887ec67640 1 -- 172.21.15.90:0/1251564228 wait complete. 2024-04-25T11:41:00.616 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:41:00.616 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:41:00.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.760+0000 7f79ebb1f640 1 -- 172.21.15.90:0/2460568223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f79e4108a20 msgr2=0x7f79e41025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:00.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.760+0000 7f79ebb1f640 1 --2- 172.21.15.90:0/2460568223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f79e4108a20 0x7f79e41025b0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f79e0009900 tx=0x7f79e002a410 comp rx=0 tx=0).stop 2024-04-25T11:41:00.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.760+0000 7f79ebb1f640 1 -- 172.21.15.90:0/2460568223 shutdown_connections 2024-04-25T11:41:00.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.760+0000 7f79ebb1f640 1 --2- 172.21.15.90:0/2460568223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79e410d380 0x7f79e410f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.760+0000 7f79ebb1f640 1 --2- 172.21.15.90:0/2460568223 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f79e4102af0 0x7f79e4102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.760+0000 7f79ebb1f640 1 --2- 172.21.15.90:0/2460568223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f79e4108a20 0x7f79e41025b0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.760+0000 7f79ebb1f640 1 -- 172.21.15.90:0/2460568223 >> 172.21.15.90:0/2460568223 conn(0x7f79e4078100 msgr2=0x7f79e4078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:00.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.760+0000 7f79ebb1f640 1 -- 172.21.15.90:0/2460568223 shutdown_connections 2024-04-25T11:41:00.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.760+0000 7f79ebb1f640 1 -- 172.21.15.90:0/2460568223 wait complete. 2024-04-25T11:41:00.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.761+0000 7f79ebb1f640 1 Processor -- start 2024-04-25T11:41:00.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.761+0000 7f79ebb1f640 1 -- start start 2024-04-25T11:41:00.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.761+0000 7f79ebb1f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f79e4102af0 0x7f79e419b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.761+0000 7f79ebb1f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79e4108a20 0x7f79e419ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.761+0000 7f79ebb1f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f79e410d380 0x7f79e419bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.761+0000 7f79ebb1f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f79e4112be0 con 0x7f79e410d380 2024-04-25T11:41:00.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.761+0000 7f79ebb1f640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f79e4112a60 con 0x7f79e4102af0 2024-04-25T11:41:00.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.761+0000 7f79ebb1f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f79e4112d60 con 0x7f79e4108a20 2024-04-25T11:41:00.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.761+0000 7f79e98bb640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f79e4102af0 0x7f79e419b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.761+0000 7f79ea0bc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f79e410d380 0x7f79e419bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.761+0000 7f79e98bb640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f79e4102af0 0x7f79e419b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:41166/0 (socket says 172.21.15.90:41166) 2024-04-25T11:41:00.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.761+0000 7f79e98bb640 1 -- 172.21.15.90:0/835679164 learned_addr learned my addr 172.21.15.90:0/835679164 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:00.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.761+0000 7f79e90ba640 1 --2- 172.21.15.90:0/835679164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79e4108a20 0x7f79e419ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.762+0000 7f79e98bb640 1 -- 172.21.15.90:0/835679164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79e4108a20 msgr2=0x7f79e419ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:00.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.762+0000 7f79e98bb640 1 --2- 172.21.15.90:0/835679164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79e4108a20 0x7f79e419ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.762+0000 7f79e98bb640 1 -- 172.21.15.90:0/835679164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f79e410d380 msgr2=0x7f79e419bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:00.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.762+0000 7f79e98bb640 1 --2- 172.21.15.90:0/835679164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f79e410d380 0x7f79e419bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.762+0000 7f79e98bb640 1 -- 172.21.15.90:0/835679164 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79e41a6830 con 0x7f79e4102af0 2024-04-25T11:41:00.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.762+0000 7f79e98bb640 1 --2- 172.21.15.90:0/835679164 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f79e4102af0 0x7f79e419b520 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f79e0006010 tx=0x7f79e0004060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:00.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.762+0000 7f79daffd640 1 -- 172.21.15.90:0/835679164 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e003f030 con 0x7f79e4102af0 2024-04-25T11:41:00.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.762+0000 7f79daffd640 1 -- 172.21.15.90:0/835679164 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79e0003aa0 con 0x7f79e4102af0 2024-04-25T11:41:00.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.762+0000 7f79daffd640 1 -- 172.21.15.90:0/835679164 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e0035070 con 0x7f79e4102af0 2024-04-25T11:41:00.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.762+0000 7f79ebb1f640 1 -- 172.21.15.90:0/835679164 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79e41a6ac0 con 0x7f79e4102af0 2024-04-25T11:41:00.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.763+0000 7f79ebb1f640 1 -- 172.21.15.90:0/835679164 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79e41a6ea0 con 0x7f79e4102af0 2024-04-25T11:41:00.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.765+0000 7f79daffd640 1 -- 172.21.15.90:0/835679164 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f79e0042030 con 0x7f79e4102af0 2024-04-25T11:41:00.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.766+0000 7f79f29d2740 1 Processor -- start 2024-04-25T11:41:00.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.766+0000 7f79f29d2740 1 -- start start 2024-04-25T11:41:00.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.766+0000 7f79daffd640 1 --2- 172.21.15.90:0/835679164 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f79d00afce0 0x7f79d00b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.766+0000 7f79f29d2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9dc9390 0x5627ea169a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.766+0000 7f79f29d2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627e9da04d0 0x5627ea169fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.766+0000 7f79f29d2740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5627e9da86b0 0x5627ea1768f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.766+0000 7f79f29d2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5627ea0cfd80 con 0x5627e9dc9390 2024-04-25T11:41:00.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.766+0000 7f79f29d2740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5627ea0898b0 con 0x5627e9da86b0 2024-04-25T11:41:00.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.766+0000 7f79f29d2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5627ea0cbd40 con 0x5627e9da04d0 2024-04-25T11:41:00.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.766+0000 7f79ea0bc640 1 --2- 172.21.15.90:0/835679164 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f79d00afce0 0x7f79d00b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.767+0000 7f79e98bb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9dc9390 0x5627ea169a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.767+0000 7f79e98bb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9dc9390 0x5627ea169a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46424/0 (socket says 172.21.15.90:46424) 2024-04-25T11:41:00.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.767+0000 7f79ea0bc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5627e9da86b0 0x5627ea1768f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.767+0000 7f79e98bb640 1 -- 172.21.15.90:0/3234188767 learned_addr learned my addr 172.21.15.90:0/3234188767 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:00.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.767+0000 7f79daffd640 1 -- 172.21.15.90:0/835679164 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f79e00ff400 con 0x7f79e4102af0 2024-04-25T11:41:00.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.767+0000 7f79e90ba640 1 --2- 172.21.15.90:0/3234188767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627e9da04d0 0x5627ea169fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.767+0000 7f79e98bb640 1 -- 172.21.15.90:0/3234188767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627e9da04d0 msgr2=0x5627ea169fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:00.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.767+0000 7f79e98bb640 1 --2- 172.21.15.90:0/3234188767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627e9da04d0 0x5627ea169fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.767+0000 7f79e98bb640 1 -- 172.21.15.90:0/3234188767 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5627e9da86b0 msgr2=0x5627ea1768f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:00.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.767+0000 7f79e98bb640 1 --2- 172.21.15.90:0/3234188767 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5627e9da86b0 0x5627ea1768f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.767+0000 7f79e98bb640 1 -- 172.21.15.90:0/3234188767 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627ea05aa40 con 0x5627e9dc9390 2024-04-25T11:41:00.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.767+0000 7f79ea0bc640 1 --2- 172.21.15.90:0/3234188767 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5627e9da86b0 0x5627ea1768f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:00.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.767+0000 7f79ea0bc640 1 --2- 172.21.15.90:0/835679164 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f79d00afce0 0x7f79d00b21f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f79dc009fd0 tx=0x7f79dc009da0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:00.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.767+0000 7f79e98bb640 1 --2- 172.21.15.90:0/3234188767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9dc9390 0x5627ea169a60 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f79e0031440 tx=0x7f79e003d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d40ddf2e65549f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:00.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.767+0000 7f79cffff640 1 -- 172.21.15.90:0/3234188767 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e0049020 con 0x5627e9dc9390 2024-04-25T11:41:00.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.767+0000 7f79cffff640 1 -- 172.21.15.90:0/3234188767 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79e003b460 con 0x5627e9dc9390 2024-04-25T11:41:00.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.767+0000 7f79cffff640 1 -- 172.21.15.90:0/3234188767 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e00c75f0 con 0x5627e9dc9390 2024-04-25T11:41:00.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.768+0000 7f79f29d2740 1 -- 172.21.15.90:0/3234188767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9dc9390 msgr2=0x5627ea169a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:00.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.768+0000 7f79f29d2740 1 --2- 172.21.15.90:0/3234188767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9dc9390 0x5627ea169a60 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f79e0031440 tx=0x7f79e003d000 comp rx=0 tx=0).stop 2024-04-25T11:41:00.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.768+0000 7f79f29d2740 1 -- 172.21.15.90:0/3234188767 shutdown_connections 2024-04-25T11:41:00.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.768+0000 7f79f29d2740 1 --2- 172.21.15.90:0/3234188767 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5627e9da86b0 0x5627ea1768f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.768+0000 7f79f29d2740 1 --2- 172.21.15.90:0/3234188767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627e9da04d0 0x5627ea169fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.768+0000 7f79f29d2740 1 --2- 172.21.15.90:0/3234188767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9dc9390 0x5627ea169a60 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.768+0000 7f79f29d2740 1 -- 172.21.15.90:0/3234188767 >> 172.21.15.90:0/3234188767 conn(0x5627e9dbd4e0 msgr2=0x5627ea166210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:00.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.769+0000 7f79f29d2740 1 -- 172.21.15.90:0/3234188767 shutdown_connections 2024-04-25T11:41:00.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.769+0000 7f79f29d2740 1 -- 172.21.15.90:0/3234188767 wait complete. 2024-04-25T11:41:00.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.769+0000 7f79f29d2740 1 -- start start 2024-04-25T11:41:00.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.769+0000 7f79f29d2740 1 Processor -- start 2024-04-25T11:41:00.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.769+0000 7f79f29d2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627e9da04d0 0x5627ea174690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.769+0000 7f79f29d2740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5627e9da86b0 0x5627ea174bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.769+0000 7f79f29d2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9dc9390 0x5627ea17a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.769+0000 7f79f29d2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5627ea176e90 con 0x5627e9dc9390 2024-04-25T11:41:00.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.769+0000 7f79f29d2740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5627ea056d10 con 0x5627e9da86b0 2024-04-25T11:41:00.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.769+0000 7f79f29d2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5627ea177010 con 0x5627e9da04d0 2024-04-25T11:41:00.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.770+0000 7f79f29d2740 10 client.0 _ll_register_callbacks cb 0x5627ea083210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:00.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.770+0000 7f79ea0bc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9dc9390 0x5627ea17a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.770+0000 7f79e90ba640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5627e9da86b0 0x5627ea174bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.770+0000 7f79ea0bc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9dc9390 0x5627ea17a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46438/0 (socket says 172.21.15.90:46438) 2024-04-25T11:41:00.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.770+0000 7f79ea0bc640 1 -- 172.21.15.90:0/3806498573 learned_addr learned my addr 172.21.15.90:0/3806498573 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:00.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.770+0000 7f79e98bb640 1 --2- 172.21.15.90:0/3806498573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627e9da04d0 0x5627ea174690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.770+0000 7f79ea0bc640 1 -- 172.21.15.90:0/3806498573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627e9da04d0 msgr2=0x5627ea174690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:00.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.770+0000 7f79ea0bc640 1 --2- 172.21.15.90:0/3806498573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627e9da04d0 0x5627ea174690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.770+0000 7f79ea0bc640 1 -- 172.21.15.90:0/3806498573 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5627e9da86b0 msgr2=0x5627ea174bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:00.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.770+0000 7f79ea0bc640 1 --2- 172.21.15.90:0/3806498573 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5627e9da86b0 0x5627ea174bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.770+0000 7f79ea0bc640 1 -- 172.21.15.90:0/3806498573 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627e9eab7d0 con 0x5627e9dc9390 2024-04-25T11:41:00.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.770+0000 7f79ea0bc640 1 --2- 172.21.15.90:0/3806498573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9dc9390 0x5627ea17a350 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f79dc010e40 tx=0x7f79dc009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=811a5ec1501e76c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:00.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.770+0000 7f79cd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:41:00.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.770+0000 7f79cd7fa640 1 -- 172.21.15.90:0/3806498573 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79dc003e00 con 0x5627e9dc9390 2024-04-25T11:41:00.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.770+0000 7f79cd7fa640 1 -- 172.21.15.90:0/3806498573 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f79dc004740 con 0x5627e9dc9390 2024-04-25T11:41:00.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.770+0000 7f79f29d2740 1 -- 172.21.15.90:0/3806498573 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5627ea0dd990 con 0x5627e9dc9390 2024-04-25T11:41:00.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.770+0000 7f79cd7fa640 1 -- 172.21.15.90:0/3806498573 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f79dc030410 con 0x5627e9dc9390 2024-04-25T11:41:00.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.770+0000 7f79f29d2740 10 client.17400 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:00.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.770+0000 7f79f29d2740 1 -- 172.21.15.90:0/3806498573 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5627e9de13f0 con 0x5627e9dc9390 2024-04-25T11:41:00.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.771+0000 7f79cd7fa640 1 -- 172.21.15.90:0/3806498573 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5627ea0dd990 con 0x5627e9dc9390 2024-04-25T11:41:00.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.771+0000 7f79cd7fa640 1 -- 172.21.15.90:0/3806498573 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f79dc01f070 con 0x5627e9dc9390 2024-04-25T11:41:00.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.771+0000 7f79f29d2740 10 client.17400 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:00.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.771+0000 7f79f29d2740 10 client.17400 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:41:00.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.771+0000 7f79f29d2740 20 client.17400 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:00.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.771+0000 7f79f29d2740 1 --2- 172.21.15.90:0/3806498573 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5627e9d87f60 0x5627ea186a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.771+0000 7f79f29d2740 4 client.17400 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:41:00.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.771+0000 7f79f29d2740 1 -- 172.21.15.90:0/3806498573 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627ea0d7800 con 0x5627e9d87f60 2024-04-25T11:41:00.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.771+0000 7f79e98bb640 1 --2- 172.21.15.90:0/3806498573 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5627e9d87f60 0x5627ea186a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.771+0000 7f79cd7fa640 10 client.17400 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:00.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.771+0000 7f79e98bb640 1 --2- 172.21.15.90:0/3806498573 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5627e9d87f60 0x5627ea186a40 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a3ff97053240f31 server_cookie=aeac779b3f11a42a in_seq=0 out_seq=0 2024-04-25T11:41:00.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.772+0000 7f79cd7fa640 1 -- 172.21.15.90:0/3806498573 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5627ea0d7800 con 0x5627e9d87f60 2024-04-25T11:41:00.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.772+0000 7f79cd7fa640 10 client.17400 handle_command_reply: tid=0 2024-04-25T11:41:00.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.772+0000 7f79f29d2740 1 client.17400 shutdown 2024-04-25T11:41:00.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.772+0000 7f79f29d2740 1 -- 172.21.15.90:0/3806498573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9dc9390 msgr2=0x5627ea17a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:00.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.772+0000 7f79f29d2740 1 --2- 172.21.15.90:0/3806498573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9dc9390 0x5627ea17a350 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f79dc010e40 tx=0x7f79dc009600 comp rx=0 tx=0).stop 2024-04-25T11:41:00.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.772+0000 7f79f29d2740 1 -- 172.21.15.90:0/3806498573 shutdown_connections 2024-04-25T11:41:00.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.772+0000 7f79f29d2740 1 --2- 172.21.15.90:0/3806498573 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5627e9d87f60 0x5627ea186a40 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.772+0000 7f79f29d2740 1 --2- 172.21.15.90:0/3806498573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9dc9390 0x5627ea17a350 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.772+0000 7f79f29d2740 1 --2- 172.21.15.90:0/3806498573 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5627e9da86b0 0x5627ea174bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.772+0000 7f79f29d2740 1 --2- 172.21.15.90:0/3806498573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627e9da04d0 0x5627ea174690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.772+0000 7f79cd7fa640 0 client.17400 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:00.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.772+0000 7f79f29d2740 1 -- 172.21.15.90:0/3806498573 >> 172.21.15.90:0/3806498573 conn(0x5627e9dbd4e0 msgr2=0x5627ea16b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:00.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.772+0000 7f79f29d2740 1 -- 172.21.15.90:0/3806498573 shutdown_connections 2024-04-25T11:41:00.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.772+0000 7f79f29d2740 1 -- 172.21.15.90:0/3806498573 wait complete. 2024-04-25T11:41:00.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.772+0000 7f79f29d2740 20 client.17400 trim_cache size 0 max 16384 2024-04-25T11:41:00.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.793+0000 7f79f29d2740 1 Processor -- start 2024-04-25T11:41:00.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.793+0000 7f79f29d2740 1 -- start start 2024-04-25T11:41:00.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.793+0000 7f79f29d2740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5627e9da04d0 0x5627ea17a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.793+0000 7f79f29d2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9da86b0 0x5627ea17a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.793+0000 7f79f29d2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627e9dc9390 0x5627ea18c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.793+0000 7f79f29d2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f79e41afae0 con 0x5627e9da86b0 2024-04-25T11:41:00.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.793+0000 7f79f29d2740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f79e41b1620 con 0x5627e9da04d0 2024-04-25T11:41:00.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.793+0000 7f79f29d2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f79e41b01b0 con 0x5627e9dc9390 2024-04-25T11:41:00.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.794+0000 7f79e90ba640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9da86b0 0x5627ea17a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.794+0000 7f79e90ba640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9da86b0 0x5627ea17a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46450/0 (socket says 172.21.15.90:46450) 2024-04-25T11:41:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.794+0000 7f79e90ba640 1 -- 172.21.15.90:0/1426311228 learned_addr learned my addr 172.21.15.90:0/1426311228 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.794+0000 7f79e98bb640 1 --2- 172.21.15.90:0/1426311228 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5627e9da04d0 0x5627ea17a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.794+0000 7f79ea0bc640 1 --2- 172.21.15.90:0/1426311228 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627e9dc9390 0x5627ea18c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.794+0000 7f79e90ba640 1 -- 172.21.15.90:0/1426311228 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627e9dc9390 msgr2=0x5627ea18c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.794+0000 7f79e90ba640 1 --2- 172.21.15.90:0/1426311228 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627e9dc9390 0x5627ea18c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.794+0000 7f79e90ba640 1 -- 172.21.15.90:0/1426311228 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5627e9da04d0 msgr2=0x5627ea17a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.794+0000 7f79e90ba640 1 --2- 172.21.15.90:0/1426311228 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5627e9da04d0 0x5627ea17a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.794+0000 7f79e90ba640 1 -- 172.21.15.90:0/1426311228 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627e9da10f0 con 0x5627e9da86b0 2024-04-25T11:41:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.794+0000 7f79e98bb640 1 --2- 172.21.15.90:0/1426311228 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5627e9da04d0 0x5627ea17a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.794+0000 7f79e90ba640 1 --2- 172.21.15.90:0/1426311228 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9da86b0 0x5627ea17a810 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f79d4009e90 tx=0x7f79d4005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=857d23d99f86fdd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.794+0000 7f79cf7fe640 1 -- 172.21.15.90:0/1426311228 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d40120d0 con 0x5627e9da86b0 2024-04-25T11:41:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.794+0000 7f79cf7fe640 1 -- 172.21.15.90:0/1426311228 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79d4012270 con 0x5627e9da86b0 2024-04-25T11:41:00.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.794+0000 7f79cf7fe640 1 -- 172.21.15.90:0/1426311228 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d4004e20 con 0x5627e9da86b0 2024-04-25T11:41:00.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.795+0000 7f79f29d2740 1 -- 172.21.15.90:0/1426311228 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9da86b0 msgr2=0x5627ea17a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:00.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.795+0000 7f79f29d2740 1 --2- 172.21.15.90:0/1426311228 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9da86b0 0x5627ea17a810 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f79d4009e90 tx=0x7f79d4005f90 comp rx=0 tx=0).stop 2024-04-25T11:41:00.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.795+0000 7f79f29d2740 1 -- 172.21.15.90:0/1426311228 shutdown_connections 2024-04-25T11:41:00.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.795+0000 7f79f29d2740 1 --2- 172.21.15.90:0/1426311228 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627e9dc9390 0x5627ea18c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.795+0000 7f79f29d2740 1 --2- 172.21.15.90:0/1426311228 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9da86b0 0x5627ea17a810 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.795+0000 7f79f29d2740 1 --2- 172.21.15.90:0/1426311228 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5627e9da04d0 0x5627ea17a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.795+0000 7f79f29d2740 1 -- 172.21.15.90:0/1426311228 >> 172.21.15.90:0/1426311228 conn(0x5627e9dbd4e0 msgr2=0x5627ea172d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:00.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.795+0000 7f79f29d2740 1 -- 172.21.15.90:0/1426311228 shutdown_connections 2024-04-25T11:41:00.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.795+0000 7f79f29d2740 1 -- 172.21.15.90:0/1426311228 wait complete. 2024-04-25T11:41:00.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.796+0000 7f79f29d2740 1 -- start start 2024-04-25T11:41:00.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.796+0000 7f79f29d2740 1 Processor -- start 2024-04-25T11:41:00.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.796+0000 7f79f29d2740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5627e9da04d0 0x5627ea17b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.796+0000 7f79f29d2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627e9da86b0 0x5627ea17be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.796+0000 7f79f29d2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9dc9390 0x5627ea1f11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.796+0000 7f79f29d2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f79e41b0f50 con 0x5627e9dc9390 2024-04-25T11:41:00.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.796+0000 7f79f29d2740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f79e41b0880 con 0x5627e9da04d0 2024-04-25T11:41:00.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.796+0000 7f79f29d2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f79e41adac0 con 0x5627e9da86b0 2024-04-25T11:41:00.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.796+0000 7f79f29d2740 10 client.0 _ll_register_callbacks cb 0x7f79dc0124f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:00.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.796+0000 7f79ea0bc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9dc9390 0x5627ea1f11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.796+0000 7f79ea0bc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9dc9390 0x5627ea1f11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46466/0 (socket says 172.21.15.90:46466) 2024-04-25T11:41:00.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.796+0000 7f79ea0bc640 1 -- 172.21.15.90:0/917577419 learned_addr learned my addr 172.21.15.90:0/917577419 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:00.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.796+0000 7f79e98bb640 1 --2- 172.21.15.90:0/917577419 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5627e9da04d0 0x5627ea17b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.796+0000 7f79e90ba640 1 --2- 172.21.15.90:0/917577419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627e9da86b0 0x5627ea17be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.797+0000 7f79ea0bc640 1 -- 172.21.15.90:0/917577419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627e9da86b0 msgr2=0x5627ea17be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:00.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.797+0000 7f79ea0bc640 1 --2- 172.21.15.90:0/917577419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627e9da86b0 0x5627ea17be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.797+0000 7f79ea0bc640 1 -- 172.21.15.90:0/917577419 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5627e9da04d0 msgr2=0x5627ea17b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:00.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.797+0000 7f79ea0bc640 1 --2- 172.21.15.90:0/917577419 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5627e9da04d0 0x5627ea17b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.797+0000 7f79ea0bc640 1 -- 172.21.15.90:0/917577419 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627ea0904b0 con 0x5627e9dc9390 2024-04-25T11:41:00.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.797+0000 7f79e98bb640 1 --2- 172.21.15.90:0/917577419 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5627e9da04d0 0x5627ea17b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:00.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.797+0000 7f79ea0bc640 1 --2- 172.21.15.90:0/917577419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9dc9390 0x5627ea1f11f0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f79dc01e800 tx=0x7f79dc004480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b085604fc6d98962 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:00.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.797+0000 7f79cd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:41:00.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.797+0000 7f79cd7fa640 1 -- 172.21.15.90:0/917577419 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79dc01ea80 con 0x5627e9dc9390 2024-04-25T11:41:00.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.797+0000 7f79cd7fa640 1 -- 172.21.15.90:0/917577419 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f79dc004740 con 0x5627e9dc9390 2024-04-25T11:41:00.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.797+0000 7f79f29d2740 1 -- 172.21.15.90:0/917577419 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79d4012270 con 0x5627e9dc9390 2024-04-25T11:41:00.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.797+0000 7f79cd7fa640 1 -- 172.21.15.90:0/917577419 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f79dc004940 con 0x5627e9dc9390 2024-04-25T11:41:00.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.797+0000 7f79f29d2740 10 client.17406 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:00.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.797+0000 7f79f29d2740 1 -- 172.21.15.90:0/917577419 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5627ea1f2570 con 0x5627e9dc9390 2024-04-25T11:41:00.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.797+0000 7f79cd7fa640 1 -- 172.21.15.90:0/917577419 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79d4012270 con 0x5627e9dc9390 2024-04-25T11:41:00.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.798+0000 7f79cd7fa640 1 -- 172.21.15.90:0/917577419 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f79dc037020 con 0x5627e9dc9390 2024-04-25T11:41:00.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.798+0000 7f79f29d2740 10 client.17406 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:00.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.798+0000 7f79f29d2740 10 client.17406 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:41:00.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.798+0000 7f79f29d2740 20 client.17406 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:00.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.798+0000 7f79f29d2740 1 --2- 172.21.15.90:0/917577419 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5627e9d5d3a0 0x5627ea1f4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:00.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.798+0000 7f79f29d2740 4 client.17406 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:41:00.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.798+0000 7f79f29d2740 1 -- 172.21.15.90:0/917577419 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5627ea0524a0 con 0x5627e9d5d3a0 2024-04-25T11:41:00.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.798+0000 7f79e98bb640 1 --2- 172.21.15.90:0/917577419 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5627e9d5d3a0 0x5627ea1f4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:00.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.798+0000 7f79cd7fa640 10 client.17406 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:00.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.798+0000 7f79e98bb640 1 --2- 172.21.15.90:0/917577419 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5627e9d5d3a0 0x5627ea1f4c70 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=998840d21a90dd61 server_cookie=173539f17209adc4 in_seq=0 out_seq=0 2024-04-25T11:41:00.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.798+0000 7f79cd7fa640 1 -- 172.21.15.90:0/917577419 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x5627ea0524a0 con 0x5627e9d5d3a0 2024-04-25T11:41:00.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.798+0000 7f79cd7fa640 10 client.17406 handle_command_reply: tid=0 2024-04-25T11:41:00.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.799+0000 7f79f29d2740 1 client.17406 shutdown 2024-04-25T11:41:00.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.799+0000 7f79f29d2740 1 -- 172.21.15.90:0/917577419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9dc9390 msgr2=0x5627ea1f11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:00.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.799+0000 7f79f29d2740 1 --2- 172.21.15.90:0/917577419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9dc9390 0x5627ea1f11f0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f79dc01e800 tx=0x7f79dc004480 comp rx=0 tx=0).stop 2024-04-25T11:41:00.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.799+0000 7f79f29d2740 1 -- 172.21.15.90:0/917577419 shutdown_connections 2024-04-25T11:41:00.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.799+0000 7f79f29d2740 1 --2- 172.21.15.90:0/917577419 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5627e9d5d3a0 0x5627ea1f4c70 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.799+0000 7f79f29d2740 1 --2- 172.21.15.90:0/917577419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5627e9dc9390 0x5627ea1f11f0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.799+0000 7f79f29d2740 1 --2- 172.21.15.90:0/917577419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5627e9da86b0 0x5627ea17be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.799+0000 7f79f29d2740 1 --2- 172.21.15.90:0/917577419 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5627e9da04d0 0x5627ea17b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.799+0000 7f79cd7fa640 0 client.17406 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:00.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.799+0000 7f79f29d2740 1 -- 172.21.15.90:0/917577419 >> 172.21.15.90:0/917577419 conn(0x5627e9dbd4e0 msgr2=0x5627ea16b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:00.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.799+0000 7f79f29d2740 1 -- 172.21.15.90:0/917577419 shutdown_connections 2024-04-25T11:41:00.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.799+0000 7f79f29d2740 1 -- 172.21.15.90:0/917577419 wait complete. 2024-04-25T11:41:00.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.799+0000 7f79f29d2740 20 client.17406 trim_cache size 0 max 16384 2024-04-25T11:41:00.791 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:41:00.791 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "scrub active (0 inodes in the stack)", 2024-04-25T11:41:00.791 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-25T11:41:00.791 INFO:teuthology.orchestra.run.smithi090.stdout: "170cd0c0-9c0e-42a6-bb1a-42fd52717626": { 2024-04-25T11:41:00.792 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-25T11:41:00.792 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "170cd0c0-9c0e-42a6-bb1a-42fd52717626", 2024-04-25T11:41:00.792 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-25T11:41:00.792 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:41:00.792 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:41:00.792 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:41:00.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.799+0000 7f79ebb1f640 1 -- 172.21.15.90:0/835679164 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f79d00afce0 msgr2=0x7f79d00b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:00.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.799+0000 7f79ebb1f640 1 --2- 172.21.15.90:0/835679164 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f79d00afce0 0x7f79d00b21f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f79dc009fd0 tx=0x7f79dc009da0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.799+0000 7f79ebb1f640 1 -- 172.21.15.90:0/835679164 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f79e4102af0 msgr2=0x7f79e419b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:00.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.799+0000 7f79ebb1f640 1 --2- 172.21.15.90:0/835679164 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f79e4102af0 0x7f79e419b520 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f79e0006010 tx=0x7f79e0004060 comp rx=0 tx=0).stop 2024-04-25T11:41:00.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.799+0000 7f79ebb1f640 1 -- 172.21.15.90:0/835679164 shutdown_connections 2024-04-25T11:41:00.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.799+0000 7f79ebb1f640 1 --2- 172.21.15.90:0/835679164 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f79d00afce0 0x7f79d00b21f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.799+0000 7f79ebb1f640 1 --2- 172.21.15.90:0/835679164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f79e410d380 0x7f79e419bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.799+0000 7f79ebb1f640 1 --2- 172.21.15.90:0/835679164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79e4108a20 0x7f79e419ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.799+0000 7f79ebb1f640 1 --2- 172.21.15.90:0/835679164 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f79e4102af0 0x7f79e419b520 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:00.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.799+0000 7f79ebb1f640 1 -- 172.21.15.90:0/835679164 >> 172.21.15.90:0/835679164 conn(0x7f79e4078100 msgr2=0x7f79e41040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:00.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.800+0000 7f79ebb1f640 1 -- 172.21.15.90:0/835679164 shutdown_connections 2024-04-25T11:41:00.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:00.800+0000 7f79ebb1f640 1 -- 172.21.15.90:0/835679164 wait complete. 2024-04-25T11:41:00.800 INFO:tasks.cephfs.filesystem:scrub status for tag:170cd0c0-9c0e-42a6-bb1a-42fd52717626 - {'path': '/', 'tag': '170cd0c0-9c0e-42a6-bb1a-42fd52717626', 'options': 'recursive,force'} 2024-04-25T11:41:01.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:01 smithi090 ceph-mon[29575]: pgmap v181: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 2.7 MiB/s rd, 619 MiB/s wr, 700 op/s 2024-04-25T11:41:01.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:01 smithi090 ceph-mon[29575]: scrub queued for path: / 2024-04-25T11:41:01.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:01 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:41:01.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:01 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:41:01.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:01 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:41:01.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:01 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:41:02.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:01 smithi138 ceph-mon[32328]: pgmap v181: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 2.7 MiB/s rd, 619 MiB/s wr, 700 op/s 2024-04-25T11:41:02.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:01 smithi138 ceph-mon[32328]: scrub queued for path: / 2024-04-25T11:41:02.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:01 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:41:02.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:01 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:41:02.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:01 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:41:02.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:01 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:41:02.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:01 smithi156 ceph-mon[33045]: pgmap v181: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 2.7 MiB/s rd, 619 MiB/s wr, 700 op/s 2024-04-25T11:41:02.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:01 smithi156 ceph-mon[33045]: scrub queued for path: / 2024-04-25T11:41:02.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:01 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:41:02.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:01 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:41:02.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:01 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:41:02.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:01 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:41:02.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:02 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:41:03.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:02 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:41:03.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:02 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:41:03.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:03 smithi090 ceph-mon[29575]: pgmap v182: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 468 MiB/s wr, 576 op/s 2024-04-25T11:41:04.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:03 smithi138 ceph-mon[32328]: pgmap v182: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 468 MiB/s wr, 576 op/s 2024-04-25T11:41:04.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:03 smithi156 ceph-mon[33045]: pgmap v182: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 468 MiB/s wr, 576 op/s 2024-04-25T11:41:04.826 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.840+0000 7f28d8e30640 1 -- 172.21.15.90:0/1370380306 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28d41064c0 msgr2=0x7f28d4112890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.840+0000 7f28d8e30640 1 --2- 172.21.15.90:0/1370380306 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28d41064c0 0x7f28d4112890 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f28cc009ca0 tx=0x7f28cc02a8e0 comp rx=0 tx=0).stop 2024-04-25T11:41:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.840+0000 7f28d8e30640 1 -- 172.21.15.90:0/1370380306 shutdown_connections 2024-04-25T11:41:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.840+0000 7f28d8e30640 1 --2- 172.21.15.90:0/1370380306 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28d41064c0 0x7f28d4112890 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.840+0000 7f28d8e30640 1 --2- 172.21.15.90:0/1370380306 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f28d410c210 0x7f28d4105e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.840+0000 7f28d8e30640 1 --2- 172.21.15.90:0/1370380306 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28d410bd70 0x7f28d4105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.840+0000 7f28d8e30640 1 -- 172.21.15.90:0/1370380306 >> 172.21.15.90:0/1370380306 conn(0x7f28d4077000 msgr2=0x7f28d4077440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.840+0000 7f28d8e30640 1 -- 172.21.15.90:0/1370380306 shutdown_connections 2024-04-25T11:41:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.840+0000 7f28d8e30640 1 -- 172.21.15.90:0/1370380306 wait complete. 2024-04-25T11:41:04.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.841+0000 7f28d8e30640 1 Processor -- start 2024-04-25T11:41:04.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.841+0000 7f28d8e30640 1 -- start start 2024-04-25T11:41:04.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.842+0000 7f28d8e30640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28d41064c0 0x7f28d41a26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:04.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.842+0000 7f28d8e30640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f28d410bd70 0x7f28d41a2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:04.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.842+0000 7f28d8e30640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28d410c210 0x7f28d419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:04.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.842+0000 7f28d8e30640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f28d41160b0 con 0x7f28d41064c0 2024-04-25T11:41:04.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.842+0000 7f28d8e30640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f28d4115f30 con 0x7f28d410bd70 2024-04-25T11:41:04.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.842+0000 7f28d8e30640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f28d4116230 con 0x7f28d410c210 2024-04-25T11:41:04.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.842+0000 7f28d259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28d41064c0 0x7f28d41a26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:04.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.842+0000 7f28d259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28d41064c0 0x7f28d41a26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37456/0 (socket says 172.21.15.90:37456) 2024-04-25T11:41:04.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.842+0000 7f28d259c640 1 -- 172.21.15.90:0/142557947 learned_addr learned my addr 172.21.15.90:0/142557947 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:04.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.842+0000 7f28d259c640 1 -- 172.21.15.90:0/142557947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28d410c210 msgr2=0x7f28d419c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:04.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.842+0000 7f28d259c640 1 --2- 172.21.15.90:0/142557947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28d410c210 0x7f28d419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:04.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.842+0000 7f28d259c640 1 -- 172.21.15.90:0/142557947 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f28d410bd70 msgr2=0x7f28d41a2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:04.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.842+0000 7f28d259c640 1 --2- 172.21.15.90:0/142557947 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f28d410bd70 0x7f28d41a2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:04.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.842+0000 7f28d259c640 1 -- 172.21.15.90:0/142557947 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d419cf70 con 0x7f28d41064c0 2024-04-25T11:41:04.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.843+0000 7f28d259c640 1 --2- 172.21.15.90:0/142557947 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28d41064c0 0x7f28d41a26c0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f28c400b450 tx=0x7f28c400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:04.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.843+0000 7f28c37fe640 1 -- 172.21.15.90:0/142557947 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c4015020 con 0x7f28d41064c0 2024-04-25T11:41:04.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.843+0000 7f28c37fe640 1 -- 172.21.15.90:0/142557947 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28c4011d40 con 0x7f28d41064c0 2024-04-25T11:41:04.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.843+0000 7f28c37fe640 1 -- 172.21.15.90:0/142557947 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c400f340 con 0x7f28d41064c0 2024-04-25T11:41:04.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.843+0000 7f28d8e30640 1 -- 172.21.15.90:0/142557947 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28d419d260 con 0x7f28d41064c0 2024-04-25T11:41:04.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.843+0000 7f28d8e30640 1 -- 172.21.15.90:0/142557947 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28d41a9540 con 0x7f28d41064c0 2024-04-25T11:41:04.833 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.847+0000 7f28dfdb3740 1 Processor -- start 2024-04-25T11:41:04.834 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.848+0000 7f28dfdb3740 1 -- start start 2024-04-25T11:41:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.848+0000 7f28dfdb3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d1aabb41b0 0x55d1aaf53aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.848+0000 7f28dfdb3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1aab8b2f0 0x55d1aaf53fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.848+0000 7f28dfdb3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab934d0 0x55d1aaf54520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.848+0000 7f28c37fe640 1 -- 172.21.15.90:0/142557947 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f28c4018020 con 0x7f28d41064c0 2024-04-25T11:41:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.848+0000 7f28dfdb3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d1aaeba460 con 0x55d1aab934d0 2024-04-25T11:41:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.848+0000 7f28dfdb3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55d1aae74590 con 0x55d1aabb41b0 2024-04-25T11:41:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.848+0000 7f28dfdb3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d1aaeb6420 con 0x55d1aab8b2f0 2024-04-25T11:41:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.848+0000 7f28d1d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab934d0 0x55d1aaf54520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.849+0000 7f28d1d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab934d0 0x55d1aaf54520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37460/0 (socket says 172.21.15.90:37460) 2024-04-25T11:41:04.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.849+0000 7f28d1d9b640 1 -- 172.21.15.90:0/2920997503 learned_addr learned my addr 172.21.15.90:0/2920997503 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:04.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.849+0000 7f28d1d9b640 1 -- 172.21.15.90:0/2920997503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1aab8b2f0 msgr2=0x55d1aaf53fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:41:04.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.849+0000 7f28d1d9b640 1 --2- 172.21.15.90:0/2920997503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1aab8b2f0 0x55d1aaf53fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:04.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.849+0000 7f28d1d9b640 1 -- 172.21.15.90:0/2920997503 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d1aabb41b0 msgr2=0x55d1aaf53aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:04.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.849+0000 7f28c37fe640 1 --2- 172.21.15.90:0/142557947 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f28bc0afcd0 0x7f28bc0b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:04.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.849+0000 7f28d1d9b640 1 --2- 172.21.15.90:0/2920997503 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d1aabb41b0 0x55d1aaf53aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:04.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.849+0000 7f28d1d9b640 1 -- 172.21.15.90:0/2920997503 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1aae810d0 con 0x55d1aab934d0 2024-04-25T11:41:04.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.850+0000 7f28d2d9d640 1 --2- 172.21.15.90:0/142557947 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f28bc0afcd0 0x7f28bc0b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:04.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.850+0000 7f28c37fe640 1 -- 172.21.15.90:0/142557947 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f28c40deab0 con 0x7f28d41064c0 2024-04-25T11:41:04.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.850+0000 7f28d1d9b640 1 --2- 172.21.15.90:0/2920997503 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab934d0 0x55d1aaf54520 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f28c8009750 tx=0x7f28c8009c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bdea8ebfecefd0d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:04.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.850+0000 7f28d2d9d640 1 --2- 172.21.15.90:0/142557947 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f28bc0afcd0 0x7f28bc0b21e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f28cc0094e0 tx=0x7f28cc009470 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:04.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.851+0000 7f28c0ff9640 1 -- 172.21.15.90:0/2920997503 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c8015020 con 0x55d1aab934d0 2024-04-25T11:41:04.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.851+0000 7f28c0ff9640 1 -- 172.21.15.90:0/2920997503 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28c8011420 con 0x55d1aab934d0 2024-04-25T11:41:04.839 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.852+0000 7f28c0ff9640 1 -- 172.21.15.90:0/2920997503 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c8009e60 con 0x55d1aab934d0 2024-04-25T11:41:04.839 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.853+0000 7f28dfdb3740 1 -- 172.21.15.90:0/2920997503 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab934d0 msgr2=0x55d1aaf54520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:04.840 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.853+0000 7f28dfdb3740 1 --2- 172.21.15.90:0/2920997503 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab934d0 0x55d1aaf54520 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f28c8009750 tx=0x7f28c8009c10 comp rx=0 tx=0).stop 2024-04-25T11:41:04.840 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.853+0000 7f28dfdb3740 1 -- 172.21.15.90:0/2920997503 shutdown_connections 2024-04-25T11:41:04.840 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.853+0000 7f28dfdb3740 1 --2- 172.21.15.90:0/2920997503 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab934d0 0x55d1aaf54520 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:04.840 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.853+0000 7f28dfdb3740 1 --2- 172.21.15.90:0/2920997503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1aab8b2f0 0x55d1aaf53fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:04.840 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.853+0000 7f28dfdb3740 1 --2- 172.21.15.90:0/2920997503 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d1aabb41b0 0x55d1aaf53aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:04.840 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.853+0000 7f28dfdb3740 1 -- 172.21.15.90:0/2920997503 >> 172.21.15.90:0/2920997503 conn(0x55d1aaba8300 msgr2=0x55d1aaf50250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:04.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.854+0000 7f28dfdb3740 1 -- 172.21.15.90:0/2920997503 shutdown_connections 2024-04-25T11:41:04.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.854+0000 7f28dfdb3740 1 -- 172.21.15.90:0/2920997503 wait complete. 2024-04-25T11:41:04.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.854+0000 7f28dfdb3740 1 -- start start 2024-04-25T11:41:04.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.855+0000 7f28dfdb3740 1 Processor -- start 2024-04-25T11:41:04.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.855+0000 7f28dfdb3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1aab8b2f0 0x55d1aaf5fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:04.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.855+0000 7f28dfdb3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab934d0 0x55d1aaf60020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:04.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.856+0000 7f28dfdb3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d1aabb41b0 0x55d1aaf5df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:04.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.856+0000 7f28dfdb3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d1aaf60c20 con 0x55d1aab934d0 2024-04-25T11:41:04.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.856+0000 7f28d1d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab934d0 0x55d1aaf60020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:04.843 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.856+0000 7f28dfdb3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55d1aae41b60 con 0x55d1aabb41b0 2024-04-25T11:41:04.843 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.856+0000 7f28d1d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab934d0 0x55d1aaf60020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37470/0 (socket says 172.21.15.90:37470) 2024-04-25T11:41:04.843 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.856+0000 7f28dfdb3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d1aaf60da0 con 0x55d1aab8b2f0 2024-04-25T11:41:04.843 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.856+0000 7f28d1d9b640 1 -- 172.21.15.90:0/1367686557 learned_addr learned my addr 172.21.15.90:0/1367686557 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:04.843 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.856+0000 7f28dfdb3740 10 client.0 _ll_register_callbacks cb 0x55d1aae662b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:04.843 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.857+0000 7f28d1d9b640 1 -- 172.21.15.90:0/1367686557 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1aab8b2f0 msgr2=0x55d1aaf5fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:04.844 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.857+0000 7f28d1d9b640 1 --2- 172.21.15.90:0/1367686557 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1aab8b2f0 0x55d1aaf5fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:04.844 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.857+0000 7f28d1d9b640 1 -- 172.21.15.90:0/1367686557 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d1aabb41b0 msgr2=0x55d1aaf5df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:04.844 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.857+0000 7f28d1d9b640 1 --2- 172.21.15.90:0/1367686557 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d1aabb41b0 0x55d1aaf5df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:04.844 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.857+0000 7f28d1d9b640 1 -- 172.21.15.90:0/1367686557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1aabcc210 con 0x55d1aab934d0 2024-04-25T11:41:04.844 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.857+0000 7f28d1d9b640 1 --2- 172.21.15.90:0/1367686557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab934d0 0x55d1aaf60020 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f28c8011c30 tx=0x7f28c8011c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=738046578714da83 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:04.844 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.857+0000 7f28b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:41:04.844 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.858+0000 7f28b9ffb640 1 -- 172.21.15.90:0/1367686557 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c800bb80 con 0x55d1aab934d0 2024-04-25T11:41:04.844 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.858+0000 7f28b9ffb640 1 -- 172.21.15.90:0/1367686557 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f28c800bd20 con 0x55d1aab934d0 2024-04-25T11:41:04.845 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.859+0000 7f28dfdb3740 1 -- 172.21.15.90:0/1367686557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28c8011420 con 0x55d1aab934d0 2024-04-25T11:41:04.845 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.859+0000 7f28b9ffb640 1 -- 172.21.15.90:0/1367686557 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f28c801a420 con 0x55d1aab934d0 2024-04-25T11:41:04.845 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.859+0000 7f28dfdb3740 10 client.17418 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:04.845 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.859+0000 7f28dfdb3740 1 -- 172.21.15.90:0/1367686557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d1aac98620 con 0x55d1aab934d0 2024-04-25T11:41:04.846 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.860+0000 7f28b9ffb640 1 -- 172.21.15.90:0/1367686557 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28c8011420 con 0x55d1aab934d0 2024-04-25T11:41:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.860+0000 7f28b9ffb640 1 -- 172.21.15.90:0/1367686557 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f28c8022700 con 0x55d1aab934d0 2024-04-25T11:41:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.860+0000 7f28dfdb3740 10 client.17418 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.860+0000 7f28dfdb3740 10 client.17418 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:41:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.860+0000 7f28dfdb3740 20 client.17418 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.860+0000 7f28dfdb3740 1 --2- 172.21.15.90:0/1367686557 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d1aab72d80 0x55d1aaf70470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.860+0000 7f28dfdb3740 4 client.17418 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:41:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.860+0000 7f28dfdb3740 1 -- 172.21.15.90:0/1367686557 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d1aaec8070 con 0x55d1aab72d80 2024-04-25T11:41:04.848 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.862+0000 7f28d259c640 1 --2- 172.21.15.90:0/1367686557 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d1aab72d80 0x55d1aaf70470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:04.851 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.865+0000 7f28b9ffb640 10 client.17418 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:04.851 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.865+0000 7f28d259c640 1 --2- 172.21.15.90:0/1367686557 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d1aab72d80 0x55d1aaf70470 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f40b00a8ac8836ab server_cookie=8122b54e16d1a28d in_seq=0 out_seq=0 2024-04-25T11:41:04.856 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.868+0000 7f28b9ffb640 1 -- 172.21.15.90:0/1367686557 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d1aaec8070 con 0x55d1aab72d80 2024-04-25T11:41:04.856 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.868+0000 7f28b9ffb640 10 client.17418 handle_command_reply: tid=0 2024-04-25T11:41:04.856 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.869+0000 7f28dfdb3740 1 client.17418 shutdown 2024-04-25T11:41:04.856 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.869+0000 7f28dfdb3740 1 -- 172.21.15.90:0/1367686557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab934d0 msgr2=0x55d1aaf60020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:04.856 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.869+0000 7f28dfdb3740 1 --2- 172.21.15.90:0/1367686557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab934d0 0x55d1aaf60020 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f28c8011c30 tx=0x7f28c8011c60 comp rx=0 tx=0).stop 2024-04-25T11:41:04.856 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.869+0000 7f28dfdb3740 1 -- 172.21.15.90:0/1367686557 shutdown_connections 2024-04-25T11:41:04.856 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.869+0000 7f28dfdb3740 1 --2- 172.21.15.90:0/1367686557 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d1aab72d80 0x55d1aaf70470 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:04.856 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.869+0000 7f28dfdb3740 1 --2- 172.21.15.90:0/1367686557 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d1aabb41b0 0x55d1aaf5df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:04.856 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.869+0000 7f28dfdb3740 1 --2- 172.21.15.90:0/1367686557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab934d0 0x55d1aaf60020 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:04.856 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.869+0000 7f28b9ffb640 0 client.17418 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:04.856 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.869+0000 7f28dfdb3740 1 --2- 172.21.15.90:0/1367686557 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1aab8b2f0 0x55d1aaf5fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:04.856 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.869+0000 7f28dfdb3740 1 -- 172.21.15.90:0/1367686557 >> 172.21.15.90:0/1367686557 conn(0x55d1aaba8300 msgr2=0x55d1aaf557c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:04.857 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.869+0000 7f28dfdb3740 1 -- 172.21.15.90:0/1367686557 shutdown_connections 2024-04-25T11:41:04.857 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.869+0000 7f28dfdb3740 1 -- 172.21.15.90:0/1367686557 wait complete. 2024-04-25T11:41:04.857 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.869+0000 7f28dfdb3740 20 client.17418 trim_cache size 0 max 16384 2024-04-25T11:41:04.876 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.889+0000 7f28dfdb3740 1 Processor -- start 2024-04-25T11:41:04.876 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.890+0000 7f28dfdb3740 1 -- start start 2024-04-25T11:41:04.877 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.890+0000 7f28dfdb3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1aab72d80 0x55d1aaf55e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:04.877 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.890+0000 7f28dfdb3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab8b2f0 0x55d1aaf75b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:04.877 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.890+0000 7f28dfdb3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d1aab934d0 0x55d1aaf760d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:04.877 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.890+0000 7f28dfdb3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f28d41b1f20 con 0x55d1aab8b2f0 2024-04-25T11:41:04.877 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.890+0000 7f28dfdb3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f28d41b3a60 con 0x55d1aab934d0 2024-04-25T11:41:04.877 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.890+0000 7f28dfdb3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f28d41b25f0 con 0x55d1aab72d80 2024-04-25T11:41:04.877 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.890+0000 7f28d1d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab8b2f0 0x55d1aaf75b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:04.878 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.890+0000 7f28d1d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab8b2f0 0x55d1aaf75b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37486/0 (socket says 172.21.15.90:37486) 2024-04-25T11:41:04.878 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.890+0000 7f28d1d9b640 1 -- 172.21.15.90:0/4263455488 learned_addr learned my addr 172.21.15.90:0/4263455488 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:04.878 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.891+0000 7f28d1d9b640 1 -- 172.21.15.90:0/4263455488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1aab72d80 msgr2=0x55d1aaf55e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:04.878 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.891+0000 7f28d1d9b640 1 --2- 172.21.15.90:0/4263455488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1aab72d80 0x55d1aaf55e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:04.878 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.891+0000 7f28d1d9b640 1 -- 172.21.15.90:0/4263455488 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d1aab934d0 msgr2=0x55d1aaf760d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:04.878 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.891+0000 7f28d1d9b640 1 --2- 172.21.15.90:0/4263455488 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d1aab934d0 0x55d1aaf760d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:04.878 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.891+0000 7f28d1d9b640 1 -- 172.21.15.90:0/4263455488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1aaf4a4c0 con 0x55d1aab8b2f0 2024-04-25T11:41:04.879 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.891+0000 7f28d1d9b640 1 --2- 172.21.15.90:0/4263455488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab8b2f0 0x55d1aaf75b90 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f28c8011c00 tx=0x7f28c8006f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94a9d30e6019d1ba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:04.879 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.892+0000 7f28bbfff640 1 -- 172.21.15.90:0/4263455488 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c8015040 con 0x55d1aab8b2f0 2024-04-25T11:41:04.879 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.892+0000 7f28bbfff640 1 -- 172.21.15.90:0/4263455488 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28c8011e50 con 0x55d1aab8b2f0 2024-04-25T11:41:04.879 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.892+0000 7f28bbfff640 1 -- 172.21.15.90:0/4263455488 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c800be10 con 0x55d1aab8b2f0 2024-04-25T11:41:04.880 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.893+0000 7f28dfdb3740 1 -- 172.21.15.90:0/4263455488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab8b2f0 msgr2=0x55d1aaf75b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:04.880 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.893+0000 7f28dfdb3740 1 --2- 172.21.15.90:0/4263455488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab8b2f0 0x55d1aaf75b90 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f28c8011c00 tx=0x7f28c8006f60 comp rx=0 tx=0).stop 2024-04-25T11:41:04.880 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.893+0000 7f28dfdb3740 1 -- 172.21.15.90:0/4263455488 shutdown_connections 2024-04-25T11:41:04.880 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.893+0000 7f28dfdb3740 1 --2- 172.21.15.90:0/4263455488 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d1aab934d0 0x55d1aaf760d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:04.880 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.893+0000 7f28dfdb3740 1 --2- 172.21.15.90:0/4263455488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab8b2f0 0x55d1aaf75b90 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:04.881 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.893+0000 7f28dfdb3740 1 --2- 172.21.15.90:0/4263455488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1aab72d80 0x55d1aaf55e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:04.881 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.893+0000 7f28dfdb3740 1 -- 172.21.15.90:0/4263455488 >> 172.21.15.90:0/4263455488 conn(0x55d1aaba8300 msgr2=0x55d1aaf55750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:04.881 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.893+0000 7f28dfdb3740 1 -- 172.21.15.90:0/4263455488 shutdown_connections 2024-04-25T11:41:04.881 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.893+0000 7f28dfdb3740 1 -- 172.21.15.90:0/4263455488 wait complete. 2024-04-25T11:41:04.881 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.894+0000 7f28dfdb3740 1 -- start start 2024-04-25T11:41:04.881 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.894+0000 7f28dfdb3740 1 Processor -- start 2024-04-25T11:41:04.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.895+0000 7f28dfdb3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d1aab72d80 0x55d1aaf665d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:04.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.895+0000 7f28dfdb3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1aab8b2f0 0x55d1aaf61920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:04.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.895+0000 7f28dfdb3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab934d0 0x55d1aaf61e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:04.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.895+0000 7f28dfdb3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f28d41b3390 con 0x55d1aab934d0 2024-04-25T11:41:04.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.895+0000 7f28dfdb3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f28d41b2cc0 con 0x55d1aab72d80 2024-04-25T11:41:04.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.895+0000 7f28dfdb3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f28d41aff00 con 0x55d1aab8b2f0 2024-04-25T11:41:04.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.895+0000 7f28dfdb3740 10 client.0 _ll_register_callbacks cb 0x7f28c80095a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:04.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.895+0000 7f28d2d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab934d0 0x55d1aaf61e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:04.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.895+0000 7f28d2d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab934d0 0x55d1aaf61e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37488/0 (socket says 172.21.15.90:37488) 2024-04-25T11:41:04.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.895+0000 7f28d2d9d640 1 -- 172.21.15.90:0/1303863506 learned_addr learned my addr 172.21.15.90:0/1303863506 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:04.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.896+0000 7f28d1d9b640 1 --2- 172.21.15.90:0/1303863506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1aab8b2f0 0x55d1aaf61920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:04.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.896+0000 7f28d2d9d640 1 -- 172.21.15.90:0/1303863506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1aab8b2f0 msgr2=0x55d1aaf61920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:04.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.896+0000 7f28d2d9d640 1 --2- 172.21.15.90:0/1303863506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1aab8b2f0 0x55d1aaf61920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:04.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.896+0000 7f28d259c640 1 --2- 172.21.15.90:0/1303863506 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d1aab72d80 0x55d1aaf665d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:04.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.896+0000 7f28d2d9d640 1 -- 172.21.15.90:0/1303863506 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d1aab72d80 msgr2=0x55d1aaf665d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:04.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.896+0000 7f28d2d9d640 1 --2- 172.21.15.90:0/1303863506 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d1aab72d80 0x55d1aaf665d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:04.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.896+0000 7f28d2d9d640 1 -- 172.21.15.90:0/1303863506 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1aaf625a0 con 0x55d1aab934d0 2024-04-25T11:41:04.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.896+0000 7f28d1d9b640 1 --2- 172.21.15.90:0/1303863506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1aab8b2f0 0x55d1aaf61920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:41:04.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.896+0000 7f28d2d9d640 1 --2- 172.21.15.90:0/1303863506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab934d0 0x55d1aaf61e60 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x55d1aaf50c10 tx=0x7f28cc0309b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e3392fe9f215b58 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:04.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.896+0000 7f28b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:41:04.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.896+0000 7f28b9ffb640 1 -- 172.21.15.90:0/1303863506 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28cc04a070 con 0x55d1aab934d0 2024-04-25T11:41:04.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.896+0000 7f28b9ffb640 1 -- 172.21.15.90:0/1303863506 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f28cc046830 con 0x55d1aab934d0 2024-04-25T11:41:04.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.896+0000 7f28dfdb3740 1 -- 172.21.15.90:0/1303863506 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28c8011e50 con 0x55d1aab934d0 2024-04-25T11:41:04.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.897+0000 7f28b9ffb640 1 -- 172.21.15.90:0/1303863506 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f28cc046a30 con 0x55d1aab934d0 2024-04-25T11:41:04.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.897+0000 7f28dfdb3740 10 client.17424 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:04.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.897+0000 7f28dfdb3740 1 -- 172.21.15.90:0/1303863506 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d1aafd8600 con 0x55d1aab934d0 2024-04-25T11:41:04.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.897+0000 7f28b9ffb640 1 -- 172.21.15.90:0/1303863506 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28c8011e50 con 0x55d1aab934d0 2024-04-25T11:41:04.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.897+0000 7f28b9ffb640 1 -- 172.21.15.90:0/1303863506 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f28cc03a3f0 con 0x55d1aab934d0 2024-04-25T11:41:04.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.897+0000 7f28dfdb3740 10 client.17424 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:04.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.897+0000 7f28dfdb3740 10 client.17424 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:41:04.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.897+0000 7f28dfdb3740 20 client.17424 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:04.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.897+0000 7f28dfdb3740 1 --2- 172.21.15.90:0/1303863506 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d1aabb41b0 0x55d1aafdadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:04.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.897+0000 7f28dfdb3740 4 client.17424 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:41:04.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.897+0000 7f28dfdb3740 1 -- 172.21.15.90:0/1303863506 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d1aaec1ee0 con 0x55d1aabb41b0 2024-04-25T11:41:04.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.898+0000 7f28d259c640 1 --2- 172.21.15.90:0/1303863506 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d1aabb41b0 0x55d1aafdadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:04.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.899+0000 7f28b9ffb640 10 client.17424 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:04.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:04.899+0000 7f28d259c640 1 --2- 172.21.15.90:0/1303863506 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d1aabb41b0 0x55d1aafdadf0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7690ed45ef512a19 server_cookie=d7144a0d8dc5bd82 in_seq=0 out_seq=0 2024-04-25T11:41:05.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:05 smithi090 ceph-mon[29575]: pgmap v183: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 490 MiB/s wr, 592 op/s 2024-04-25T11:41:06.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:05 smithi138 ceph-mon[32328]: pgmap v183: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 490 MiB/s wr, 592 op/s 2024-04-25T11:41:06.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:05 smithi156 ceph-mon[33045]: pgmap v183: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 490 MiB/s wr, 592 op/s 2024-04-25T11:41:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.715+0000 7f28b9ffb640 1 -- 172.21.15.90:0/1303863506 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d1aaec1ee0 con 0x55d1aabb41b0 2024-04-25T11:41:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.715+0000 7f28b9ffb640 10 client.17424 handle_command_reply: tid=0 2024-04-25T11:41:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.715+0000 7f28dfdb3740 1 client.17424 shutdown 2024-04-25T11:41:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.716+0000 7f28dfdb3740 1 -- 172.21.15.90:0/1303863506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab934d0 msgr2=0x55d1aaf61e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.716+0000 7f28dfdb3740 1 --2- 172.21.15.90:0/1303863506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab934d0 0x55d1aaf61e60 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x55d1aaf50c10 tx=0x7f28cc0309b0 comp rx=0 tx=0).stop 2024-04-25T11:41:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.716+0000 7f28dfdb3740 1 -- 172.21.15.90:0/1303863506 shutdown_connections 2024-04-25T11:41:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.716+0000 7f28dfdb3740 1 --2- 172.21.15.90:0/1303863506 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d1aabb41b0 0x55d1aafdadf0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.716+0000 7f28dfdb3740 1 --2- 172.21.15.90:0/1303863506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d1aab934d0 0x55d1aaf61e60 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.716+0000 7f28dfdb3740 1 --2- 172.21.15.90:0/1303863506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1aab8b2f0 0x55d1aaf61920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.716+0000 7f28dfdb3740 1 --2- 172.21.15.90:0/1303863506 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d1aab72d80 0x55d1aaf665d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.716+0000 7f28dfdb3740 1 -- 172.21.15.90:0/1303863506 >> 172.21.15.90:0/1303863506 conn(0x55d1aaba8300 msgr2=0x55d1aaf53830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.716+0000 7f28b9ffb640 0 client.17424 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.716+0000 7f28dfdb3740 1 -- 172.21.15.90:0/1303863506 shutdown_connections 2024-04-25T11:41:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.716+0000 7f28dfdb3740 1 -- 172.21.15.90:0/1303863506 wait complete. 2024-04-25T11:41:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.717+0000 7f28dfdb3740 20 client.17424 trim_cache size 0 max 16384 2024-04-25T11:41:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:41:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:41:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:41:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:41:07.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.720+0000 7f28d8e30640 1 -- 172.21.15.90:0/142557947 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f28bc0afcd0 msgr2=0x7f28bc0b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:07.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.720+0000 7f28d8e30640 1 --2- 172.21.15.90:0/142557947 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f28bc0afcd0 0x7f28bc0b21e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f28cc0094e0 tx=0x7f28cc009470 comp rx=0 tx=0).stop 2024-04-25T11:41:07.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.720+0000 7f28d8e30640 1 -- 172.21.15.90:0/142557947 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28d41064c0 msgr2=0x7f28d41a26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:07.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.720+0000 7f28d8e30640 1 --2- 172.21.15.90:0/142557947 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28d41064c0 0x7f28d41a26c0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f28c400b450 tx=0x7f28c400b910 comp rx=0 tx=0).stop 2024-04-25T11:41:07.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.720+0000 7f28d8e30640 1 -- 172.21.15.90:0/142557947 shutdown_connections 2024-04-25T11:41:07.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.720+0000 7f28d8e30640 1 --2- 172.21.15.90:0/142557947 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f28bc0afcd0 0x7f28bc0b21e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:07.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.720+0000 7f28d8e30640 1 --2- 172.21.15.90:0/142557947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28d410c210 0x7f28d419c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:07.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.720+0000 7f28d8e30640 1 --2- 172.21.15.90:0/142557947 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f28d410bd70 0x7f28d41a2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:07.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.720+0000 7f28d8e30640 1 --2- 172.21.15.90:0/142557947 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28d41064c0 0x7f28d41a26c0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:07.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.720+0000 7f28d8e30640 1 -- 172.21.15.90:0/142557947 >> 172.21.15.90:0/142557947 conn(0x7f28d4077000 msgr2=0x7f28d41031d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:07.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.720+0000 7f28d8e30640 1 -- 172.21.15.90:0/142557947 shutdown_connections 2024-04-25T11:41:07.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:07.720+0000 7f28d8e30640 1 -- 172.21.15.90:0/142557947 wait complete. 2024-04-25T11:41:08.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:07 smithi090 ceph-mon[29575]: pgmap v184: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 539 MiB/s wr, 641 op/s 2024-04-25T11:41:08.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:07 smithi138 ceph-mon[32328]: pgmap v184: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 539 MiB/s wr, 641 op/s 2024-04-25T11:41:08.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:07 smithi156 ceph-mon[33045]: pgmap v184: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 539 MiB/s wr, 641 op/s 2024-04-25T11:41:10.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:10 smithi138 ceph-mon[32328]: pgmap v185: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 394 MiB/s wr, 464 op/s 2024-04-25T11:41:10.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:10 smithi156 ceph-mon[33045]: pgmap v185: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 394 MiB/s wr, 464 op/s 2024-04-25T11:41:10.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:10 smithi090 ceph-mon[29575]: pgmap v185: 97 pgs: 97 active+clean; 50 GiB data, 153 GiB used, 920 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 394 MiB/s wr, 464 op/s 2024-04-25T11:41:11.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:11 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:41:11.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:11 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:41:11.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:11 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:41:12.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:12 smithi090 ceph-mon[29575]: pgmap v186: 97 pgs: 97 active+clean; 51 GiB data, 156 GiB used, 917 GiB / 1.0 TiB avail; 2.1 MiB/s rd, 597 MiB/s wr, 1.23k op/s 2024-04-25T11:41:12.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:12 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:41:12.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:12 smithi138 ceph-mon[32328]: pgmap v186: 97 pgs: 97 active+clean; 51 GiB data, 156 GiB used, 917 GiB / 1.0 TiB avail; 2.1 MiB/s rd, 597 MiB/s wr, 1.23k op/s 2024-04-25T11:41:12.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:12 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:41:12.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:12 smithi156 ceph-mon[33045]: pgmap v186: 97 pgs: 97 active+clean; 51 GiB data, 156 GiB used, 917 GiB / 1.0 TiB avail; 2.1 MiB/s rd, 597 MiB/s wr, 1.23k op/s 2024-04-25T11:41:12.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:12 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:41:13.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:13 smithi090 ceph-mon[29575]: pgmap v187: 97 pgs: 97 active+clean; 52 GiB data, 157 GiB used, 916 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 502 MiB/s wr, 1.25k op/s 2024-04-25T11:41:13.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:13 smithi138 ceph-mon[32328]: pgmap v187: 97 pgs: 97 active+clean; 52 GiB data, 157 GiB used, 916 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 502 MiB/s wr, 1.25k op/s 2024-04-25T11:41:13.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:13 smithi156 ceph-mon[33045]: pgmap v187: 97 pgs: 97 active+clean; 52 GiB data, 157 GiB used, 916 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 502 MiB/s wr, 1.25k op/s 2024-04-25T11:41:15.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:15 smithi156 ceph-mon[33045]: pgmap v188: 97 pgs: 97 active+clean; 52 GiB data, 157 GiB used, 916 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 479 MiB/s wr, 1.31k op/s 2024-04-25T11:41:15.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:15 smithi090 ceph-mon[29575]: pgmap v188: 97 pgs: 97 active+clean; 52 GiB data, 157 GiB used, 916 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 479 MiB/s wr, 1.31k op/s 2024-04-25T11:41:16.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:15 smithi138 ceph-mon[32328]: pgmap v188: 97 pgs: 97 active+clean; 52 GiB data, 157 GiB used, 916 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 479 MiB/s wr, 1.31k op/s 2024-04-25T11:41:17.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:17 smithi138 ceph-mon[32328]: pgmap v189: 97 pgs: 97 active+clean; 52 GiB data, 159 GiB used, 914 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 548 MiB/s wr, 2.19k op/s 2024-04-25T11:41:17.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:17 smithi156 ceph-mon[33045]: pgmap v189: 97 pgs: 97 active+clean; 52 GiB data, 159 GiB used, 914 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 548 MiB/s wr, 2.19k op/s 2024-04-25T11:41:17.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:17 smithi090 ceph-mon[29575]: pgmap v189: 97 pgs: 97 active+clean; 52 GiB data, 159 GiB used, 914 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 548 MiB/s wr, 2.19k op/s 2024-04-25T11:41:19.977 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:19 smithi090 ceph-mon[29575]: pgmap v190: 97 pgs: 97 active+clean; 52 GiB data, 159 GiB used, 914 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 417 MiB/s wr, 2.04k op/s 2024-04-25T11:41:20.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:19 smithi138 ceph-mon[32328]: pgmap v190: 97 pgs: 97 active+clean; 52 GiB data, 159 GiB used, 914 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 417 MiB/s wr, 2.04k op/s 2024-04-25T11:41:20.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:19 smithi156 ceph-mon[33045]: pgmap v190: 97 pgs: 97 active+clean; 52 GiB data, 159 GiB used, 914 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 417 MiB/s wr, 2.04k op/s 2024-04-25T11:41:20.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.931+0000 7f3414d96640 1 -- 172.21.15.90:0/3458850420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f341010e970 msgr2=0x7f341010eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:20.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.931+0000 7f3414d96640 1 --2- 172.21.15.90:0/3458850420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f341010e970 0x7f341010eda0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f3404009900 tx=0x7f340402a410 comp rx=0 tx=0).stop 2024-04-25T11:41:20.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.931+0000 7f3414d96640 1 -- 172.21.15.90:0/3458850420 shutdown_connections 2024-04-25T11:41:20.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.931+0000 7f3414d96640 1 --2- 172.21.15.90:0/3458850420 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f341010f430 0x7f3410116bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:20.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.931+0000 7f3414d96640 1 --2- 172.21.15.90:0/3458850420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f341010e970 0x7f341010eda0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:20.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.931+0000 7f3414d96640 1 --2- 172.21.15.90:0/3458850420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f341010a3e0 0x7f341010a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:20.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.931+0000 7f3414d96640 1 -- 172.21.15.90:0/3458850420 >> 172.21.15.90:0/3458850420 conn(0x7f3410077070 msgr2=0x7f34100774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:20.920 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.931+0000 7f3414d96640 1 -- 172.21.15.90:0/3458850420 shutdown_connections 2024-04-25T11:41:20.920 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.931+0000 7f3414d96640 1 -- 172.21.15.90:0/3458850420 wait complete. 2024-04-25T11:41:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.932+0000 7f3414d96640 1 Processor -- start 2024-04-25T11:41:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.932+0000 7f3414d96640 1 -- start start 2024-04-25T11:41:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.932+0000 7f3414d96640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f341010a3e0 0x7f34101a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.933+0000 7f3414d96640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f341010e970 0x7f34101a12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.933+0000 7f3414d96640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f341010f430 0x7f34101a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.933+0000 7f3414d96640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f341011a330 con 0x7f341010f430 2024-04-25T11:41:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.933+0000 7f3414d96640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f341011a1b0 con 0x7f341010a3e0 2024-04-25T11:41:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.933+0000 7f3414d96640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f341011a4b0 con 0x7f341010e970 2024-04-25T11:41:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.933+0000 7f340e59c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f341010a3e0 0x7f34101a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.933+0000 7f340dd9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f341010e970 0x7f34101a12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.933+0000 7f340e59c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f341010a3e0 0x7f34101a0d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:57974/0 (socket says 172.21.15.90:57974) 2024-04-25T11:41:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.933+0000 7f340e59c640 1 -- 172.21.15.90:0/3783243495 learned_addr learned my addr 172.21.15.90:0/3783243495 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.933+0000 7f340ed9d640 1 --2- 172.21.15.90:0/3783243495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f341010f430 0x7f34101a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.933+0000 7f340e59c640 1 -- 172.21.15.90:0/3783243495 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f341010e970 msgr2=0x7f34101a12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.933+0000 7f340e59c640 1 --2- 172.21.15.90:0/3783243495 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f341010e970 0x7f34101a12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.933+0000 7f340e59c640 1 -- 172.21.15.90:0/3783243495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f341010f430 msgr2=0x7f34101a5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.933+0000 7f340e59c640 1 --2- 172.21.15.90:0/3783243495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f341010f430 0x7f34101a5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.933+0000 7f340e59c640 1 -- 172.21.15.90:0/3783243495 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34101a5bd0 con 0x7f341010a3e0 2024-04-25T11:41:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.933+0000 7f340dd9b640 1 --2- 172.21.15.90:0/3783243495 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f341010e970 0x7f34101a12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.933+0000 7f340ed9d640 1 --2- 172.21.15.90:0/3783243495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f341010f430 0x7f34101a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.934+0000 7f340e59c640 1 --2- 172.21.15.90:0/3783243495 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f341010a3e0 0x7f34101a0d70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f340000e9f0 tx=0x7f340000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.934+0000 7f33ff7fe640 1 -- 172.21.15.90:0/3783243495 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f340000cd80 con 0x7f341010a3e0 2024-04-25T11:41:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.934+0000 7f3414d96640 1 -- 172.21.15.90:0/3783243495 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34101a5ec0 con 0x7f341010a3e0 2024-04-25T11:41:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.934+0000 7f33ff7fe640 1 -- 172.21.15.90:0/3783243495 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34000054d0 con 0x7f341010a3e0 2024-04-25T11:41:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.934+0000 7f33ff7fe640 1 -- 172.21.15.90:0/3783243495 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3400010690 con 0x7f341010a3e0 2024-04-25T11:41:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.934+0000 7f3414d96640 1 -- 172.21.15.90:0/3783243495 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34101ad840 con 0x7f341010a3e0 2024-04-25T11:41:20.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.938+0000 7f341bc49740 1 Processor -- start 2024-04-25T11:41:20.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.938+0000 7f341bc49740 1 -- start start 2024-04-25T11:41:20.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.938+0000 7f33ff7fe640 1 -- 172.21.15.90:0/3783243495 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f3400010830 con 0x7f341010a3e0 2024-04-25T11:41:20.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.938+0000 7f341bc49740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc548351b0 0x55cc54bd4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:20.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.938+0000 7f341bc49740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc5480c2f0 0x55cc54bd5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:20.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.938+0000 7f341bc49740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc548144d0 0x55cc54bd5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:20.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.938+0000 7f341bc49740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55cc54b3b460 con 0x55cc548144d0 2024-04-25T11:41:20.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.938+0000 7f341bc49740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55cc54af5590 con 0x55cc548351b0 2024-04-25T11:41:20.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.938+0000 7f341bc49740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55cc54b37420 con 0x55cc5480c2f0 2024-04-25T11:41:20.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.938+0000 7f340e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc5480c2f0 0x55cc54bd5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:20.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.939+0000 7f340dd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc548144d0 0x55cc54bd5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:20.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.939+0000 7f340ed9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc548351b0 0x55cc54bd4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:20.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.939+0000 7f340e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc5480c2f0 0x55cc54bd5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:49976/0 (socket says 172.21.15.90:49976) 2024-04-25T11:41:20.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.939+0000 7f340ed9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc548351b0 0x55cc54bd4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:57988/0 (socket says 172.21.15.90:57988) 2024-04-25T11:41:20.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.939+0000 7f340e59c640 1 -- 172.21.15.90:0/1897522187 learned_addr learned my addr 172.21.15.90:0/1897522187 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:20.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.939+0000 7f340dd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc548144d0 0x55cc54bd5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36550/0 (socket says 172.21.15.90:36550) 2024-04-25T11:41:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.939+0000 7f340dd9b640 1 -- 172.21.15.90:0/1897522187 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc5480c2f0 msgr2=0x55cc54bd5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.939+0000 7f340dd9b640 1 --2- 172.21.15.90:0/1897522187 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc5480c2f0 0x55cc54bd5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.939+0000 7f340dd9b640 1 -- 172.21.15.90:0/1897522187 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc548351b0 msgr2=0x55cc54bd4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.939+0000 7f340dd9b640 1 --2- 172.21.15.90:0/1897522187 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc548351b0 0x55cc54bd4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.939+0000 7f340dd9b640 1 -- 172.21.15.90:0/1897522187 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc54b020d0 con 0x55cc548144d0 2024-04-25T11:41:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.939+0000 7f340e59c640 1 --2- 172.21.15.90:0/1897522187 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc5480c2f0 0x55cc54bd5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.939+0000 7f340dd9b640 1 --2- 172.21.15.90:0/1897522187 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc548144d0 0x55cc54bd5570 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f3404006010 tx=0x7f3404005e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c330d25067c7cbe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.939+0000 7f33ff7fe640 1 --2- 172.21.15.90:0/3783243495 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f33f80afc50 0x7f33f80b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.939+0000 7f33ff7fe640 1 -- 172.21.15.90:0/3783243495 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f34000df4c0 con 0x7f341010a3e0 2024-04-25T11:41:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.939+0000 7f340ed9d640 1 --2- 172.21.15.90:0/3783243495 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f33f80afc50 0x7f33f80b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.940+0000 7f33fcff9640 1 -- 172.21.15.90:0/1897522187 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3404040070 con 0x55cc548144d0 2024-04-25T11:41:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.940+0000 7f33fcff9640 1 -- 172.21.15.90:0/1897522187 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3404031720 con 0x55cc548144d0 2024-04-25T11:41:20.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.940+0000 7f33fcff9640 1 -- 172.21.15.90:0/1897522187 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3404035040 con 0x55cc548144d0 2024-04-25T11:41:20.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.941+0000 7f341bc49740 1 -- 172.21.15.90:0/1897522187 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc548144d0 msgr2=0x55cc54bd5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:20.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.941+0000 7f341bc49740 1 --2- 172.21.15.90:0/1897522187 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc548144d0 0x55cc54bd5570 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f3404006010 tx=0x7f3404005e20 comp rx=0 tx=0).stop 2024-04-25T11:41:20.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.941+0000 7f340ed9d640 1 --2- 172.21.15.90:0/3783243495 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f33f80afc50 0x7f33f80b2160 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x55cc54bd1c60 tx=0x7f3408009500 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:20.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.941+0000 7f341bc49740 1 -- 172.21.15.90:0/1897522187 shutdown_connections 2024-04-25T11:41:20.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.941+0000 7f341bc49740 1 --2- 172.21.15.90:0/1897522187 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc548144d0 0x55cc54bd5570 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:20.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.941+0000 7f341bc49740 1 --2- 172.21.15.90:0/1897522187 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc5480c2f0 0x55cc54bd5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:20.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.941+0000 7f341bc49740 1 --2- 172.21.15.90:0/1897522187 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc548351b0 0x55cc54bd4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:20.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.941+0000 7f341bc49740 1 -- 172.21.15.90:0/1897522187 >> 172.21.15.90:0/1897522187 conn(0x55cc54829300 msgr2=0x55cc54bd12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:20.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.941+0000 7f341bc49740 1 -- 172.21.15.90:0/1897522187 shutdown_connections 2024-04-25T11:41:20.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.941+0000 7f341bc49740 1 -- 172.21.15.90:0/1897522187 wait complete. 2024-04-25T11:41:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.942+0000 7f341bc49740 1 -- start start 2024-04-25T11:41:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.942+0000 7f341bc49740 1 Processor -- start 2024-04-25T11:41:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.942+0000 7f341bc49740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc5480c2f0 0x55cc54be0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.942+0000 7f341bc49740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc548144d0 0x55cc54be1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.942+0000 7f341bc49740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc548351b0 0x55cc54bdf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.942+0000 7f341bc49740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55cc54be1c10 con 0x55cc548351b0 2024-04-25T11:41:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.942+0000 7f341bc49740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55cc54ac2b60 con 0x55cc5480c2f0 2024-04-25T11:41:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.942+0000 7f341bc49740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55cc54be1d90 con 0x55cc548144d0 2024-04-25T11:41:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.943+0000 7f341bc49740 10 client.0 _ll_register_callbacks cb 0x55cc54ae72b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.943+0000 7f340e59c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc5480c2f0 0x55cc54be0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.943+0000 7f340e59c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc5480c2f0 0x55cc54be0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:57996/0 (socket says 172.21.15.90:57996) 2024-04-25T11:41:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.943+0000 7f340e59c640 1 -- 172.21.15.90:0/2509765312 learned_addr learned my addr 172.21.15.90:0/2509765312 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.943+0000 7f340ed9d640 1 --2- 172.21.15.90:0/2509765312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc548351b0 0x55cc54bdf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.943+0000 7f340dd9b640 1 --2- 172.21.15.90:0/2509765312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc548144d0 0x55cc54be1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.943+0000 7f340e59c640 1 -- 172.21.15.90:0/2509765312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc548144d0 msgr2=0x55cc54be1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.943+0000 7f340e59c640 1 --2- 172.21.15.90:0/2509765312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc548144d0 0x55cc54be1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.943+0000 7f340e59c640 1 -- 172.21.15.90:0/2509765312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc548351b0 msgr2=0x55cc54bdf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.943+0000 7f340e59c640 1 --2- 172.21.15.90:0/2509765312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc548351b0 0x55cc54bdf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.943+0000 7f340e59c640 1 -- 172.21.15.90:0/2509765312 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc5484d210 con 0x55cc5480c2f0 2024-04-25T11:41:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.943+0000 7f340ed9d640 1 --2- 172.21.15.90:0/2509765312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc548351b0 0x55cc54bdf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:41:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.944+0000 7f340e59c640 1 --2- 172.21.15.90:0/2509765312 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc5480c2f0 0x55cc54be0b40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f340000c6a0 tx=0x7f3400009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5897d7774f27d084 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.944+0000 7f33f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:41:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.944+0000 7f33f5ffb640 1 -- 172.21.15.90:0/2509765312 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34000a6f50 con 0x55cc5480c2f0 2024-04-25T11:41:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.944+0000 7f33f5ffb640 1 -- 172.21.15.90:0/2509765312 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f34000affd0 con 0x55cc5480c2f0 2024-04-25T11:41:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.944+0000 7f341bc49740 1 -- 172.21.15.90:0/2509765312 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3404031720 con 0x55cc5480c2f0 2024-04-25T11:41:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.944+0000 7f33f5ffb640 1 -- 172.21.15.90:0/2509765312 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f34000a6d70 con 0x55cc5480c2f0 2024-04-25T11:41:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.944+0000 7f341bc49740 10 client.26450 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.944+0000 7f341bc49740 1 -- 172.21.15.90:0/2509765312 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cc54919620 con 0x55cc5480c2f0 2024-04-25T11:41:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.944+0000 7f33f5ffb640 1 -- 172.21.15.90:0/2509765312 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3404031720 con 0x55cc5480c2f0 2024-04-25T11:41:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.945+0000 7f33f5ffb640 1 -- 172.21.15.90:0/2509765312 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f3400010b20 con 0x55cc5480c2f0 2024-04-25T11:41:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.945+0000 7f341bc49740 10 client.26450 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.945+0000 7f341bc49740 10 client.26450 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:41:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.945+0000 7f341bc49740 20 client.26450 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.945+0000 7f341bc49740 1 --2- 172.21.15.90:0/2509765312 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55cc547f3d80 0x55cc54bf1770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.945+0000 7f341bc49740 4 client.26450 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:41:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.945+0000 7f341bc49740 1 -- 172.21.15.90:0/2509765312 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc54b49070 con 0x55cc547f3d80 2024-04-25T11:41:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.946+0000 7f340dd9b640 1 --2- 172.21.15.90:0/2509765312 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55cc547f3d80 0x55cc54bf1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.946+0000 7f33f5ffb640 10 client.26450 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.946+0000 7f340dd9b640 1 --2- 172.21.15.90:0/2509765312 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55cc547f3d80 0x55cc54bf1770 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52705d97546444f server_cookie=37b490ae003b98bc in_seq=0 out_seq=0 2024-04-25T11:41:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.948+0000 7f33f5ffb640 1 -- 172.21.15.90:0/2509765312 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55cc54b49070 con 0x55cc547f3d80 2024-04-25T11:41:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.948+0000 7f33f5ffb640 10 client.26450 handle_command_reply: tid=0 2024-04-25T11:41:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.948+0000 7f341bc49740 1 client.26450 shutdown 2024-04-25T11:41:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.949+0000 7f341bc49740 1 -- 172.21.15.90:0/2509765312 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc5480c2f0 msgr2=0x55cc54be0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.949+0000 7f341bc49740 1 --2- 172.21.15.90:0/2509765312 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc5480c2f0 0x55cc54be0b40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f340000c6a0 tx=0x7f3400009310 comp rx=0 tx=0).stop 2024-04-25T11:41:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.949+0000 7f341bc49740 1 -- 172.21.15.90:0/2509765312 shutdown_connections 2024-04-25T11:41:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.949+0000 7f341bc49740 1 --2- 172.21.15.90:0/2509765312 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55cc547f3d80 0x55cc54bf1770 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.949+0000 7f341bc49740 1 --2- 172.21.15.90:0/2509765312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc548351b0 0x55cc54bdf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.949+0000 7f33f5ffb640 0 client.26450 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.949+0000 7f341bc49740 1 --2- 172.21.15.90:0/2509765312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc548144d0 0x55cc54be1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.949+0000 7f341bc49740 1 --2- 172.21.15.90:0/2509765312 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc5480c2f0 0x55cc54be0b40 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.949+0000 7f341bc49740 1 -- 172.21.15.90:0/2509765312 >> 172.21.15.90:0/2509765312 conn(0x55cc54829300 msgr2=0x55cc54bd6810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.949+0000 7f341bc49740 1 -- 172.21.15.90:0/2509765312 shutdown_connections 2024-04-25T11:41:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.949+0000 7f341bc49740 1 -- 172.21.15.90:0/2509765312 wait complete. 2024-04-25T11:41:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.949+0000 7f341bc49740 20 client.26450 trim_cache size 0 max 16384 2024-04-25T11:41:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.966+0000 7f341bc49740 1 Processor -- start 2024-04-25T11:41:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.966+0000 7f341bc49740 1 -- start start 2024-04-25T11:41:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.966+0000 7f341bc49740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc5480c2f0 0x55cc54bd5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.966+0000 7f341bc49740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc548144d0 0x55cc54be0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.966+0000 7f341bc49740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc548351b0 0x55cc54bd87a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.966+0000 7f341bc49740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f34101ada10 con 0x55cc548144d0 2024-04-25T11:41:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.966+0000 7f341bc49740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f34101af460 con 0x55cc548351b0 2024-04-25T11:41:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.966+0000 7f341bc49740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f34101adff0 con 0x55cc5480c2f0 2024-04-25T11:41:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.967+0000 7f340dd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc548144d0 0x55cc54be0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.967+0000 7f340e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc5480c2f0 0x55cc54bd5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.967+0000 7f340ed9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc548351b0 0x55cc54bd87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.967+0000 7f340dd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc548144d0 0x55cc54be0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36560/0 (socket says 172.21.15.90:36560) 2024-04-25T11:41:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.967+0000 7f340e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc5480c2f0 0x55cc54bd5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:49996/0 (socket says 172.21.15.90:49996) 2024-04-25T11:41:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.967+0000 7f340dd9b640 1 -- 172.21.15.90:0/1495652458 learned_addr learned my addr 172.21.15.90:0/1495652458 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:20.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.967+0000 7f340dd9b640 1 -- 172.21.15.90:0/1495652458 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc5480c2f0 msgr2=0x55cc54bd5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:20.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.967+0000 7f340dd9b640 1 --2- 172.21.15.90:0/1495652458 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc5480c2f0 0x55cc54bd5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:20.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.967+0000 7f340dd9b640 1 -- 172.21.15.90:0/1495652458 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc548351b0 msgr2=0x55cc54bd87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:20.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.967+0000 7f340dd9b640 1 --2- 172.21.15.90:0/1495652458 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc548351b0 0x55cc54bd87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:20.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.967+0000 7f340dd9b640 1 -- 172.21.15.90:0/1495652458 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc54bcca60 con 0x55cc548144d0 2024-04-25T11:41:20.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.967+0000 7f340e59c640 1 --2- 172.21.15.90:0/1495652458 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc5480c2f0 0x55cc54bd5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:20.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.967+0000 7f340dd9b640 1 --2- 172.21.15.90:0/1495652458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc548144d0 0x55cc54be0e80 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f3404009900 tx=0x7f340402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d061253872241211 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:20.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.967+0000 7f33f7fff640 1 -- 172.21.15.90:0/1495652458 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3404057070 con 0x55cc548144d0 2024-04-25T11:41:20.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.968+0000 7f33f7fff640 1 -- 172.21.15.90:0/1495652458 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34040318c0 con 0x55cc548144d0 2024-04-25T11:41:20.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.968+0000 7f33f7fff640 1 -- 172.21.15.90:0/1495652458 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3404035040 con 0x55cc548144d0 2024-04-25T11:41:20.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.968+0000 7f341bc49740 1 -- 172.21.15.90:0/1495652458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc548144d0 msgr2=0x55cc54be0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:20.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.968+0000 7f341bc49740 1 --2- 172.21.15.90:0/1495652458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc548144d0 0x55cc54be0e80 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f3404009900 tx=0x7f340402a410 comp rx=0 tx=0).stop 2024-04-25T11:41:20.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.969+0000 7f341bc49740 1 -- 172.21.15.90:0/1495652458 shutdown_connections 2024-04-25T11:41:20.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.969+0000 7f341bc49740 1 --2- 172.21.15.90:0/1495652458 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc548351b0 0x55cc54bd87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:20.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.969+0000 7f341bc49740 1 --2- 172.21.15.90:0/1495652458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc548144d0 0x55cc54be0e80 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:20.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.969+0000 7f341bc49740 1 --2- 172.21.15.90:0/1495652458 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc5480c2f0 0x55cc54bd5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:20.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.969+0000 7f341bc49740 1 -- 172.21.15.90:0/1495652458 >> 172.21.15.90:0/1495652458 conn(0x55cc54829300 msgr2=0x55cc54be0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:20.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.969+0000 7f341bc49740 1 -- 172.21.15.90:0/1495652458 shutdown_connections 2024-04-25T11:41:20.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.969+0000 7f341bc49740 1 -- 172.21.15.90:0/1495652458 wait complete. 2024-04-25T11:41:20.960 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.969+0000 7f341bc49740 1 -- start start 2024-04-25T11:41:20.960 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.970+0000 7f341bc49740 1 Processor -- start 2024-04-25T11:41:20.960 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.970+0000 7f341bc49740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc5480c2f0 0x55cc54be7500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.970+0000 7f341bc49740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc548144d0 0x55cc54be4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.970+0000 7f341bc49740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc548351b0 0x55cc54be4db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.970+0000 7f341bc49740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f34101aed90 con 0x55cc548351b0 2024-04-25T11:41:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.970+0000 7f341bc49740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f34101ae6c0 con 0x55cc548144d0 2024-04-25T11:41:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.970+0000 7f341bc49740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f341010b4a0 con 0x55cc5480c2f0 2024-04-25T11:41:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.970+0000 7f341bc49740 10 client.0 _ll_register_callbacks cb 0x7f340000cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.970+0000 7f340dd9b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc548144d0 0x55cc54be4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.970+0000 7f340ed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc548351b0 0x55cc54be4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.970+0000 7f340dd9b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc548144d0 0x55cc54be4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:58014/0 (socket says 172.21.15.90:58014) 2024-04-25T11:41:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.970+0000 7f340dd9b640 1 -- 172.21.15.90:0/3391045777 learned_addr learned my addr 172.21.15.90:0/3391045777 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.970+0000 7f340e59c640 1 --2- 172.21.15.90:0/3391045777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc5480c2f0 0x55cc54be7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.971+0000 7f340dd9b640 1 -- 172.21.15.90:0/3391045777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc5480c2f0 msgr2=0x55cc54be7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.971+0000 7f340dd9b640 1 --2- 172.21.15.90:0/3391045777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc5480c2f0 0x55cc54be7500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.971+0000 7f340dd9b640 1 -- 172.21.15.90:0/3391045777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc548351b0 msgr2=0x55cc54be4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.971+0000 7f340dd9b640 1 --2- 172.21.15.90:0/3391045777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc548351b0 0x55cc54be4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.971+0000 7f340dd9b640 1 -- 172.21.15.90:0/3391045777 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc54be5450 con 0x55cc548144d0 2024-04-25T11:41:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.971+0000 7f340ed9d640 1 --2- 172.21.15.90:0/3391045777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc548351b0 0x55cc54be4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.971+0000 7f340e59c640 1 --2- 172.21.15.90:0/3391045777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc5480c2f0 0x55cc54be7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.971+0000 7f340dd9b640 1 --2- 172.21.15.90:0/3391045777 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc548144d0 0x55cc54be4870 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f340403cf30 tx=0x7f3404009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccacdfe4208dc657 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.971+0000 7f33f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:41:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.971+0000 7f33f5ffb640 1 -- 172.21.15.90:0/3391045777 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3404035040 con 0x55cc548144d0 2024-04-25T11:41:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.971+0000 7f341bc49740 1 -- 172.21.15.90:0/3391045777 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34040318c0 con 0x55cc548144d0 2024-04-25T11:41:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.971+0000 7f33f5ffb640 1 -- 172.21.15.90:0/3391045777 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f340403c650 con 0x55cc548144d0 2024-04-25T11:41:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.972+0000 7f33f5ffb640 1 -- 172.21.15.90:0/3391045777 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f340406a050 con 0x55cc548144d0 2024-04-25T11:41:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.972+0000 7f341bc49740 10 client.26456 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:20.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.972+0000 7f341bc49740 1 -- 172.21.15.90:0/3391045777 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cc54c59950 con 0x55cc548144d0 2024-04-25T11:41:20.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.972+0000 7f33f5ffb640 1 -- 172.21.15.90:0/3391045777 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34040318c0 con 0x55cc548144d0 2024-04-25T11:41:20.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.972+0000 7f33f5ffb640 1 -- 172.21.15.90:0/3391045777 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f3404065030 con 0x55cc548144d0 2024-04-25T11:41:20.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.972+0000 7f341bc49740 10 client.26456 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:20.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.972+0000 7f341bc49740 10 client.26456 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:41:20.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.972+0000 7f341bc49740 20 client.26456 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:20.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.972+0000 7f341bc49740 1 --2- 172.21.15.90:0/3391045777 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55cc547c91c0 0x55cc54c5bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:20.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.972+0000 7f341bc49740 4 client.26456 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:41:20.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.972+0000 7f341bc49740 1 -- 172.21.15.90:0/3391045777 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cc54b42ee0 con 0x55cc547c91c0 2024-04-25T11:41:20.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.973+0000 7f340e59c640 1 --2- 172.21.15.90:0/3391045777 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55cc547c91c0 0x55cc54c5bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:20.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.973+0000 7f340e59c640 1 --2- 172.21.15.90:0/3391045777 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55cc547c91c0 0x55cc54c5bba0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa02d85219873948 server_cookie=74943a498cc8e2 in_seq=0 out_seq=0 2024-04-25T11:41:20.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:20.973+0000 7f33f5ffb640 10 client.26456 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:21.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:21 smithi090 ceph-mon[29575]: pgmap v191: 97 pgs: 97 active+clean; 53 GiB data, 161 GiB used, 912 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 568 MiB/s wr, 2.71k op/s 2024-04-25T11:41:21.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:21 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:41:22.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:21 smithi138 ceph-mon[32328]: pgmap v191: 97 pgs: 97 active+clean; 53 GiB data, 161 GiB used, 912 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 568 MiB/s wr, 2.71k op/s 2024-04-25T11:41:22.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:21 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:41:22.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:21 smithi156 ceph-mon[33045]: pgmap v191: 97 pgs: 97 active+clean; 53 GiB data, 161 GiB used, 912 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 568 MiB/s wr, 2.71k op/s 2024-04-25T11:41:22.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:21 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:41:23.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:22 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:41:23.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:22 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:41:23.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:22 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:41:23.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:22 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:41:23.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:22 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:41:23.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:22 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:41:23.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:22 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:41:23.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:22 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:41:23.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:22 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:41:24.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:23 smithi090 ceph-mon[29575]: pgmap v192: 97 pgs: 97 active+clean; 53 GiB data, 162 GiB used, 911 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 413 MiB/s wr, 2.08k op/s 2024-04-25T11:41:24.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:23 smithi138 ceph-mon[32328]: pgmap v192: 97 pgs: 97 active+clean; 53 GiB data, 162 GiB used, 911 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 413 MiB/s wr, 2.08k op/s 2024-04-25T11:41:24.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:23 smithi156 ceph-mon[33045]: pgmap v192: 97 pgs: 97 active+clean; 53 GiB data, 162 GiB used, 911 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 413 MiB/s wr, 2.08k op/s 2024-04-25T11:41:25.470 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.482+0000 7f33f5ffb640 1 -- 172.21.15.90:0/3391045777 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cc54b42ee0 con 0x55cc547c91c0 2024-04-25T11:41:25.471 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.482+0000 7f33f5ffb640 10 client.26456 handle_command_reply: tid=0 2024-04-25T11:41:25.471 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.482+0000 7f341bc49740 1 client.26456 shutdown 2024-04-25T11:41:25.471 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.482+0000 7f341bc49740 1 -- 172.21.15.90:0/3391045777 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc548144d0 msgr2=0x55cc54be4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:25.471 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.482+0000 7f341bc49740 1 --2- 172.21.15.90:0/3391045777 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc548144d0 0x55cc54be4870 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f340403cf30 tx=0x7f3404009260 comp rx=0 tx=0).stop 2024-04-25T11:41:25.471 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.483+0000 7f341bc49740 1 -- 172.21.15.90:0/3391045777 shutdown_connections 2024-04-25T11:41:25.472 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.483+0000 7f341bc49740 1 --2- 172.21.15.90:0/3391045777 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55cc547c91c0 0x55cc54c5bba0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:25.472 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.483+0000 7f341bc49740 1 --2- 172.21.15.90:0/3391045777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc548351b0 0x55cc54be4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:25.472 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.483+0000 7f341bc49740 1 --2- 172.21.15.90:0/3391045777 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc548144d0 0x55cc54be4870 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:25.472 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.483+0000 7f341bc49740 1 --2- 172.21.15.90:0/3391045777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc5480c2f0 0x55cc54be7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:25.472 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.483+0000 7f33f5ffb640 0 client.26456 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:25.472 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.483+0000 7f341bc49740 1 -- 172.21.15.90:0/3391045777 >> 172.21.15.90:0/3391045777 conn(0x55cc54829300 msgr2=0x55cc54bd6e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:25.472 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.483+0000 7f341bc49740 1 -- 172.21.15.90:0/3391045777 shutdown_connections 2024-04-25T11:41:25.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.489+0000 7f341bc49740 1 -- 172.21.15.90:0/3391045777 wait complete. 2024-04-25T11:41:25.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.489+0000 7f341bc49740 20 client.26456 trim_cache size 0 max 16384 2024-04-25T11:41:25.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:41:25.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:41:25.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:41:25.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:41:25.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.502+0000 7f3414d96640 1 -- 172.21.15.90:0/3783243495 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f33f80afc50 msgr2=0x7f33f80b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:25.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.502+0000 7f3414d96640 1 --2- 172.21.15.90:0/3783243495 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f33f80afc50 0x7f33f80b2160 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x55cc54bd1c60 tx=0x7f3408009500 comp rx=0 tx=0).stop 2024-04-25T11:41:25.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.503+0000 7f3414d96640 1 -- 172.21.15.90:0/3783243495 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f341010a3e0 msgr2=0x7f34101a0d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:25.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.503+0000 7f3414d96640 1 --2- 172.21.15.90:0/3783243495 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f341010a3e0 0x7f34101a0d70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f340000e9f0 tx=0x7f340000edf0 comp rx=0 tx=0).stop 2024-04-25T11:41:25.491 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.503+0000 7f3414d96640 1 -- 172.21.15.90:0/3783243495 shutdown_connections 2024-04-25T11:41:25.491 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.503+0000 7f3414d96640 1 --2- 172.21.15.90:0/3783243495 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f33f80afc50 0x7f33f80b2160 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:25.491 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.503+0000 7f3414d96640 1 --2- 172.21.15.90:0/3783243495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f341010f430 0x7f34101a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:25.491 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.503+0000 7f3414d96640 1 --2- 172.21.15.90:0/3783243495 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f341010e970 0x7f34101a12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:25.491 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.503+0000 7f3414d96640 1 --2- 172.21.15.90:0/3783243495 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f341010a3e0 0x7f34101a0d70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:25.491 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.503+0000 7f3414d96640 1 -- 172.21.15.90:0/3783243495 >> 172.21.15.90:0/3783243495 conn(0x7f3410077070 msgr2=0x7f34101079c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:25.492 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.504+0000 7f3414d96640 1 -- 172.21.15.90:0/3783243495 shutdown_connections 2024-04-25T11:41:25.493 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:25.505+0000 7f3414d96640 1 -- 172.21.15.90:0/3783243495 wait complete. 2024-04-25T11:41:26.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:26 smithi138 ceph-mon[32328]: pgmap v193: 97 pgs: 97 active+clean; 53 GiB data, 162 GiB used, 911 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 391 MiB/s wr, 1.86k op/s 2024-04-25T11:41:26.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:25 smithi156 ceph-mon[33045]: pgmap v193: 97 pgs: 97 active+clean; 53 GiB data, 162 GiB used, 911 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 391 MiB/s wr, 1.86k op/s 2024-04-25T11:41:26.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:25 smithi090 ceph-mon[29575]: pgmap v193: 97 pgs: 97 active+clean; 53 GiB data, 162 GiB used, 911 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 391 MiB/s wr, 1.86k op/s 2024-04-25T11:41:28.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:28 smithi156 ceph-mon[33045]: pgmap v194: 97 pgs: 97 active+clean; 55 GiB data, 167 GiB used, 906 GiB / 1.0 TiB avail; 3.9 MiB/s rd, 627 MiB/s wr, 1.87k op/s 2024-04-25T11:41:28.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:28 smithi090 ceph-mon[29575]: pgmap v194: 97 pgs: 97 active+clean; 55 GiB data, 167 GiB used, 906 GiB / 1.0 TiB avail; 3.9 MiB/s rd, 627 MiB/s wr, 1.87k op/s 2024-04-25T11:41:28.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:28 smithi138 ceph-mon[32328]: pgmap v194: 97 pgs: 97 active+clean; 55 GiB data, 167 GiB used, 906 GiB / 1.0 TiB avail; 3.9 MiB/s rd, 627 MiB/s wr, 1.87k op/s 2024-04-25T11:41:30.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:30 smithi090 ceph-mon[29575]: pgmap v195: 97 pgs: 97 active+clean; 55 GiB data, 167 GiB used, 906 GiB / 1.0 TiB avail; 2.6 MiB/s rd, 484 MiB/s wr, 927 op/s 2024-04-25T11:41:30.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:30 smithi138 ceph-mon[32328]: pgmap v195: 97 pgs: 97 active+clean; 55 GiB data, 167 GiB used, 906 GiB / 1.0 TiB avail; 2.6 MiB/s rd, 484 MiB/s wr, 927 op/s 2024-04-25T11:41:30.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:30 smithi156 ceph-mon[33045]: pgmap v195: 97 pgs: 97 active+clean; 55 GiB data, 167 GiB used, 906 GiB / 1.0 TiB avail; 2.6 MiB/s rd, 484 MiB/s wr, 927 op/s 2024-04-25T11:41:30.801 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:41:30.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:41:31.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.070+0000 7fa3cc251640 1 -- 172.21.15.90:0/3034512231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3c4104c30 msgr2=0x7fa3c4105040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:31.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.070+0000 7fa3cc251640 1 --2- 172.21.15.90:0/3034512231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3c4104c30 0x7fa3c4105040 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0009900 tx=0x7fa3c002a420 comp rx=0 tx=0).stop 2024-04-25T11:41:31.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.070+0000 7fa3cc251640 1 -- 172.21.15.90:0/3034512231 shutdown_connections 2024-04-25T11:41:31.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.070+0000 7fa3cc251640 1 --2- 172.21.15.90:0/3034512231 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3c410e080 0x7fa3c4110570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.070+0000 7fa3cc251640 1 --2- 172.21.15.90:0/3034512231 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa3c410b690 0x7fa3c410dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.070+0000 7fa3cc251640 1 --2- 172.21.15.90:0/3034512231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3c4104c30 0x7fa3c4105040 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.070+0000 7fa3cc251640 1 -- 172.21.15.90:0/3034512231 >> 172.21.15.90:0/3034512231 conn(0x7fa3c4078110 msgr2=0x7fa3c4078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:31.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.071+0000 7fa3cc251640 1 -- 172.21.15.90:0/3034512231 shutdown_connections 2024-04-25T11:41:31.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.071+0000 7fa3cc251640 1 -- 172.21.15.90:0/3034512231 wait complete. 2024-04-25T11:41:31.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.072+0000 7fa3cc251640 1 Processor -- start 2024-04-25T11:41:31.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.073+0000 7fa3cc251640 1 -- start start 2024-04-25T11:41:31.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.073+0000 7fa3cc251640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa3c4104c30 0x7fa3c41a1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.073+0000 7fa3cc251640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3c410b690 0x7fa3c41a21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.073+0000 7fa3cc251640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3c410e080 0x7fa3c41acb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.073+0000 7fa3cc251640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c4113dc0 con 0x7fa3c410b690 2024-04-25T11:41:31.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.073+0000 7fa3cc251640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c4113c40 con 0x7fa3c4104c30 2024-04-25T11:41:31.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.073+0000 7fa3cc251640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c4113f40 con 0x7fa3c410e080 2024-04-25T11:41:31.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.073+0000 7fa3c97ec640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3c410b690 0x7fa3c41a21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:31.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.074+0000 7fa3c97ec640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3c410b690 0x7fa3c41a21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39412/0 (socket says 172.21.15.90:39412) 2024-04-25T11:41:31.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.074+0000 7fa3c97ec640 1 -- 172.21.15.90:0/1498654869 learned_addr learned my addr 172.21.15.90:0/1498654869 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:31.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.074+0000 7fa3c9fed640 1 --2- 172.21.15.90:0/1498654869 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa3c4104c30 0x7fa3c41a1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:31.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.074+0000 7fa3ca7ee640 1 --2- 172.21.15.90:0/1498654869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3c410e080 0x7fa3c41acb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:31.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.074+0000 7fa3c97ec640 1 -- 172.21.15.90:0/1498654869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3c410e080 msgr2=0x7fa3c41acb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:31.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.074+0000 7fa3c97ec640 1 --2- 172.21.15.90:0/1498654869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3c410e080 0x7fa3c41acb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.074+0000 7fa3c97ec640 1 -- 172.21.15.90:0/1498654869 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa3c4104c30 msgr2=0x7fa3c41a1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:31.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.074+0000 7fa3c97ec640 1 --2- 172.21.15.90:0/1498654869 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa3c4104c30 0x7fa3c41a1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.074+0000 7fa3c97ec640 1 -- 172.21.15.90:0/1498654869 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3c41ad060 con 0x7fa3c410b690 2024-04-25T11:41:31.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.075+0000 7fa3c97ec640 1 --2- 172.21.15.90:0/1498654869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3c410b690 0x7fa3c41a21c0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fa3b400b450 tx=0x7fa3b400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:31.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.076+0000 7fa3baffd640 1 -- 172.21.15.90:0/1498654869 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b4015020 con 0x7fa3c410b690 2024-04-25T11:41:31.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.076+0000 7fa3baffd640 1 -- 172.21.15.90:0/1498654869 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3b4011d40 con 0x7fa3c410b690 2024-04-25T11:41:31.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.077+0000 7fa3baffd640 1 -- 172.21.15.90:0/1498654869 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b400f340 con 0x7fa3c410b690 2024-04-25T11:41:31.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.077+0000 7fa3cc251640 1 -- 172.21.15.90:0/1498654869 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3c41ad350 con 0x7fa3c410b690 2024-04-25T11:41:31.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.077+0000 7fa3cc251640 1 -- 172.21.15.90:0/1498654869 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3c41ad8c0 con 0x7fa3c410b690 2024-04-25T11:41:31.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.080+0000 7fa3d31e7740 1 Processor -- start 2024-04-25T11:41:31.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.080+0000 7fa3d31e7740 1 -- start start 2024-04-25T11:41:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.081+0000 7fa3d31e7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2bb9390 0x5612a2f59a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.081+0000 7fa3d31e7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612a2b904d0 0x5612a2f59fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.081+0000 7fa3d31e7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612a2b986b0 0x5612a2f668f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.081+0000 7fa3d31e7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5612a2ebfd80 con 0x5612a2bb9390 2024-04-25T11:41:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.081+0000 7fa3d31e7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5612a2e798b0 con 0x5612a2b904d0 2024-04-25T11:41:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.081+0000 7fa3d31e7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5612a2ebbd40 con 0x5612a2b986b0 2024-04-25T11:41:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.081+0000 7fa3ca7ee640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2bb9390 0x5612a2f59a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.081+0000 7fa3ca7ee640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2bb9390 0x5612a2f59a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39422/0 (socket says 172.21.15.90:39422) 2024-04-25T11:41:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.081+0000 7fa3ca7ee640 1 -- 172.21.15.90:0/623762870 learned_addr learned my addr 172.21.15.90:0/623762870 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.081+0000 7fa3ca7ee640 1 -- 172.21.15.90:0/623762870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612a2b986b0 msgr2=0x5612a2f668f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:41:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.081+0000 7fa3ca7ee640 1 --2- 172.21.15.90:0/623762870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612a2b986b0 0x5612a2f668f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.081+0000 7fa3ca7ee640 1 -- 172.21.15.90:0/623762870 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612a2b904d0 msgr2=0x5612a2f59fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.081+0000 7fa3ca7ee640 1 --2- 172.21.15.90:0/623762870 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612a2b904d0 0x5612a2f59fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.081+0000 7fa3baffd640 1 -- 172.21.15.90:0/1498654869 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fa3b4018020 con 0x7fa3c410b690 2024-04-25T11:41:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.081+0000 7fa3ca7ee640 1 -- 172.21.15.90:0/623762870 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612a2e4aa40 con 0x5612a2bb9390 2024-04-25T11:41:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.082+0000 7fa3ca7ee640 1 --2- 172.21.15.90:0/623762870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2bb9390 0x5612a2f59a60 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc007d90 tx=0x7fa3bc00c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1b03f35cd4e4f30 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.082+0000 7fa3affff640 1 -- 172.21.15.90:0/623762870 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc015020 con 0x5612a2bb9390 2024-04-25T11:41:31.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.082+0000 7fa3affff640 1 -- 172.21.15.90:0/623762870 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3bc011ae0 con 0x5612a2bb9390 2024-04-25T11:41:31.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.082+0000 7fa3affff640 1 -- 172.21.15.90:0/623762870 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc0109d0 con 0x5612a2bb9390 2024-04-25T11:41:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.082+0000 7fa3baffd640 1 --2- 172.21.15.90:0/1498654869 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa3b00afd60 0x7fa3b00b2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.082+0000 7fa3c9fed640 1 --2- 172.21.15.90:0/1498654869 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa3b00afd60 0x7fa3b00b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.082+0000 7fa3baffd640 1 -- 172.21.15.90:0/1498654869 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fa3b40deb80 con 0x7fa3c410b690 2024-04-25T11:41:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.083+0000 7fa3d31e7740 1 -- 172.21.15.90:0/623762870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2bb9390 msgr2=0x5612a2f59a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.083+0000 7fa3d31e7740 1 --2- 172.21.15.90:0/623762870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2bb9390 0x5612a2f59a60 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc007d90 tx=0x7fa3bc00c560 comp rx=0 tx=0).stop 2024-04-25T11:41:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.083+0000 7fa3d31e7740 1 -- 172.21.15.90:0/623762870 shutdown_connections 2024-04-25T11:41:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.083+0000 7fa3d31e7740 1 --2- 172.21.15.90:0/623762870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612a2b986b0 0x5612a2f668f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.083+0000 7fa3d31e7740 1 --2- 172.21.15.90:0/623762870 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612a2b904d0 0x5612a2f59fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.083+0000 7fa3d31e7740 1 --2- 172.21.15.90:0/623762870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2bb9390 0x5612a2f59a60 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.083+0000 7fa3d31e7740 1 -- 172.21.15.90:0/623762870 >> 172.21.15.90:0/623762870 conn(0x5612a2bad4e0 msgr2=0x5612a2f56210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.083+0000 7fa3d31e7740 1 -- 172.21.15.90:0/623762870 shutdown_connections 2024-04-25T11:41:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.083+0000 7fa3d31e7740 1 -- 172.21.15.90:0/623762870 wait complete. 2024-04-25T11:41:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.083+0000 7fa3c9fed640 1 --2- 172.21.15.90:0/1498654869 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa3b00afd60 0x7fa3b00b2270 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x5612a2f5b230 tx=0x7fa3c0031400 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.084+0000 7fa3d31e7740 1 -- start start 2024-04-25T11:41:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.084+0000 7fa3d31e7740 1 Processor -- start 2024-04-25T11:41:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.084+0000 7fa3d31e7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612a2b904d0 0x5612a2f64500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.084+0000 7fa3d31e7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2b986b0 0x5612a2f64a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.084+0000 7fa3d31e7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612a2bb9390 0x5612a2f704d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.084+0000 7fa3d31e7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5612a2f66fe0 con 0x5612a2b986b0 2024-04-25T11:41:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.084+0000 7fa3d31e7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5612a2e46d10 con 0x5612a2b904d0 2024-04-25T11:41:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.084+0000 7fa3d31e7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5612a2f67160 con 0x5612a2bb9390 2024-04-25T11:41:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.084+0000 7fa3c97ec640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2b986b0 0x5612a2f64a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.084+0000 7fa3d31e7740 10 client.0 _ll_register_callbacks cb 0x5612a2e73210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.085+0000 7fa3c97ec640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2b986b0 0x5612a2f64a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39428/0 (socket says 172.21.15.90:39428) 2024-04-25T11:41:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.085+0000 7fa3c97ec640 1 -- 172.21.15.90:0/151207268 learned_addr learned my addr 172.21.15.90:0/151207268 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.085+0000 7fa3c9fed640 1 --2- 172.21.15.90:0/151207268 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612a2b904d0 0x5612a2f64500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.085+0000 7fa3ca7ee640 1 --2- 172.21.15.90:0/151207268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612a2bb9390 0x5612a2f704d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.085+0000 7fa3c97ec640 1 -- 172.21.15.90:0/151207268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612a2bb9390 msgr2=0x5612a2f704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.085+0000 7fa3c97ec640 1 --2- 172.21.15.90:0/151207268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612a2bb9390 0x5612a2f704d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.085+0000 7fa3c97ec640 1 -- 172.21.15.90:0/151207268 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612a2b904d0 msgr2=0x5612a2f64500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.085+0000 7fa3c97ec640 1 --2- 172.21.15.90:0/151207268 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612a2b904d0 0x5612a2f64500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.085+0000 7fa3c97ec640 1 -- 172.21.15.90:0/151207268 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612a2c9b7d0 con 0x5612a2b986b0 2024-04-25T11:41:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.085+0000 7fa3ca7ee640 1 --2- 172.21.15.90:0/151207268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612a2bb9390 0x5612a2f704d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:41:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.085+0000 7fa3c9fed640 1 --2- 172.21.15.90:0/151207268 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612a2b904d0 0x5612a2f64500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:41:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.085+0000 7fa3c97ec640 1 --2- 172.21.15.90:0/151207268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2b986b0 0x5612a2f64a40 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fa3b40a2df0 tx=0x7fa3b4010730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f946c1a145c94c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.085+0000 7fa3ad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:41:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.085+0000 7fa3ad7fa640 1 -- 172.21.15.90:0/151207268 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b40dc6a0 con 0x5612a2b986b0 2024-04-25T11:41:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.085+0000 7fa3ad7fa640 1 -- 172.21.15.90:0/151207268 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fa3b4011890 con 0x5612a2b986b0 2024-04-25T11:41:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.085+0000 7fa3d31e7740 1 -- 172.21.15.90:0/151207268 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3bc011ae0 con 0x5612a2b986b0 2024-04-25T11:41:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.086+0000 7fa3ad7fa640 1 -- 172.21.15.90:0/151207268 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa3b40ab120 con 0x5612a2b986b0 2024-04-25T11:41:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.086+0000 7fa3d31e7740 10 client.17454 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.086+0000 7fa3d31e7740 1 -- 172.21.15.90:0/151207268 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5612a2bd13f0 con 0x5612a2b986b0 2024-04-25T11:41:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.086+0000 7fa3ad7fa640 1 -- 172.21.15.90:0/151207268 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3bc011ae0 con 0x5612a2b986b0 2024-04-25T11:41:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.086+0000 7fa3ad7fa640 1 -- 172.21.15.90:0/151207268 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fa3b40a2870 con 0x5612a2b986b0 2024-04-25T11:41:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.086+0000 7fa3d31e7740 10 client.17454 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.086+0000 7fa3d31e7740 10 client.17454 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:41:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.086+0000 7fa3d31e7740 20 client.17454 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.086+0000 7fa3d31e7740 1 --2- 172.21.15.90:0/151207268 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5612a2b77f60 0x5612a2f76640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.086+0000 7fa3d31e7740 4 client.17454 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:41:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.086+0000 7fa3d31e7740 1 -- 172.21.15.90:0/151207268 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612a2ecd990 con 0x5612a2b77f60 2024-04-25T11:41:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.087+0000 7fa3c9fed640 1 --2- 172.21.15.90:0/151207268 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5612a2b77f60 0x5612a2f76640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:31.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.088+0000 7fa3c9fed640 1 --2- 172.21.15.90:0/151207268 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5612a2b77f60 0x5612a2f76640 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77425af5edbbefaf server_cookie=dc928a04beffe730 in_seq=0 out_seq=0 2024-04-25T11:41:31.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.088+0000 7fa3ad7fa640 10 client.17454 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:31.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.093+0000 7fa3ad7fa640 1 -- 172.21.15.90:0/151207268 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5612a2ecd990 con 0x5612a2b77f60 2024-04-25T11:41:31.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.093+0000 7fa3ad7fa640 10 client.17454 handle_command_reply: tid=0 2024-04-25T11:41:31.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.093+0000 7fa3d31e7740 1 client.17454 shutdown 2024-04-25T11:41:31.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.094+0000 7fa3d31e7740 1 -- 172.21.15.90:0/151207268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2b986b0 msgr2=0x5612a2f64a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:31.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.094+0000 7fa3d31e7740 1 --2- 172.21.15.90:0/151207268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2b986b0 0x5612a2f64a40 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fa3b40a2df0 tx=0x7fa3b4010730 comp rx=0 tx=0).stop 2024-04-25T11:41:31.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.094+0000 7fa3d31e7740 1 -- 172.21.15.90:0/151207268 shutdown_connections 2024-04-25T11:41:31.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.094+0000 7fa3d31e7740 1 --2- 172.21.15.90:0/151207268 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5612a2b77f60 0x5612a2f76640 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.094+0000 7fa3d31e7740 1 --2- 172.21.15.90:0/151207268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612a2bb9390 0x5612a2f704d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.094+0000 7fa3d31e7740 1 --2- 172.21.15.90:0/151207268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2b986b0 0x5612a2f64a40 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.094+0000 7fa3d31e7740 1 --2- 172.21.15.90:0/151207268 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612a2b904d0 0x5612a2f64500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.094+0000 7fa3d31e7740 1 -- 172.21.15.90:0/151207268 >> 172.21.15.90:0/151207268 conn(0x5612a2bad4e0 msgr2=0x5612a2f5b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:31.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.094+0000 7fa3ad7fa640 0 client.17454 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:31.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.094+0000 7fa3d31e7740 1 -- 172.21.15.90:0/151207268 shutdown_connections 2024-04-25T11:41:31.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.094+0000 7fa3d31e7740 1 -- 172.21.15.90:0/151207268 wait complete. 2024-04-25T11:41:31.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.094+0000 7fa3d31e7740 20 client.17454 trim_cache size 0 max 16384 2024-04-25T11:41:31.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.117+0000 7fa3d31e7740 1 Processor -- start 2024-04-25T11:41:31.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.117+0000 7fa3d31e7740 1 -- start start 2024-04-25T11:41:31.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.117+0000 7fa3d31e7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2b904d0 0x5612a2f553f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.117+0000 7fa3d31e7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612a2b986b0 0x5612a2f56280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.117+0000 7fa3d31e7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612a2bb9390 0x5612a2f63bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.117+0000 7fa3d31e7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c41b6290 con 0x5612a2b904d0 2024-04-25T11:41:31.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.117+0000 7fa3d31e7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c41b7dd0 con 0x5612a2b986b0 2024-04-25T11:41:31.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.117+0000 7fa3d31e7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c41b6960 con 0x5612a2bb9390 2024-04-25T11:41:31.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.118+0000 7fa3c9fed640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2b904d0 0x5612a2f553f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:31.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.118+0000 7fa3c9fed640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2b904d0 0x5612a2f553f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39442/0 (socket says 172.21.15.90:39442) 2024-04-25T11:41:31.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.118+0000 7fa3c9fed640 1 -- 172.21.15.90:0/823489163 learned_addr learned my addr 172.21.15.90:0/823489163 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:31.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.118+0000 7fa3c9fed640 1 -- 172.21.15.90:0/823489163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612a2bb9390 msgr2=0x5612a2f63bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:31.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.118+0000 7fa3c9fed640 1 --2- 172.21.15.90:0/823489163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612a2bb9390 0x5612a2f63bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.118+0000 7fa3c9fed640 1 -- 172.21.15.90:0/823489163 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612a2b986b0 msgr2=0x5612a2f56280 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:41:31.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.118+0000 7fa3c9fed640 1 --2- 172.21.15.90:0/823489163 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612a2b986b0 0x5612a2f56280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.118+0000 7fa3c9fed640 1 -- 172.21.15.90:0/823489163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612a2b910f0 con 0x5612a2b904d0 2024-04-25T11:41:31.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.118+0000 7fa3c9fed640 1 --2- 172.21.15.90:0/823489163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2b904d0 0x5612a2f553f0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0009230 tx=0x7fa3c003b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d73e95f73ed86d52 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:31.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.120+0000 7fa3af7fe640 1 -- 172.21.15.90:0/823489163 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c0035050 con 0x5612a2b904d0 2024-04-25T11:41:31.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.120+0000 7fa3af7fe640 1 -- 172.21.15.90:0/823489163 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3c0041070 con 0x5612a2b904d0 2024-04-25T11:41:31.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.120+0000 7fa3af7fe640 1 -- 172.21.15.90:0/823489163 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c0004cf0 con 0x5612a2b904d0 2024-04-25T11:41:31.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.121+0000 7fa3d31e7740 1 -- 172.21.15.90:0/823489163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2b904d0 msgr2=0x5612a2f553f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:31.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.121+0000 7fa3d31e7740 1 --2- 172.21.15.90:0/823489163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2b904d0 0x5612a2f553f0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0009230 tx=0x7fa3c003b000 comp rx=0 tx=0).stop 2024-04-25T11:41:31.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.121+0000 7fa3d31e7740 1 -- 172.21.15.90:0/823489163 shutdown_connections 2024-04-25T11:41:31.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.121+0000 7fa3d31e7740 1 --2- 172.21.15.90:0/823489163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612a2bb9390 0x5612a2f63bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.121+0000 7fa3d31e7740 1 --2- 172.21.15.90:0/823489163 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612a2b986b0 0x5612a2f56280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.121+0000 7fa3d31e7740 1 --2- 172.21.15.90:0/823489163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2b904d0 0x5612a2f553f0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.121+0000 7fa3d31e7740 1 -- 172.21.15.90:0/823489163 >> 172.21.15.90:0/823489163 conn(0x5612a2bad4e0 msgr2=0x5612a2f65a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:31.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.122+0000 7fa3d31e7740 1 -- 172.21.15.90:0/823489163 shutdown_connections 2024-04-25T11:41:31.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.122+0000 7fa3d31e7740 1 -- 172.21.15.90:0/823489163 wait complete. 2024-04-25T11:41:31.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.122+0000 7fa3d31e7740 1 -- start start 2024-04-25T11:41:31.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.122+0000 7fa3d31e7740 1 Processor -- start 2024-04-25T11:41:31.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.123+0000 7fa3d31e7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2b904d0 0x5612a2f67430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.123+0000 7fa3d31e7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612a2b986b0 0x5612a2f67970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.123+0000 7fa3d31e7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612a2bb9390 0x5612a2f67eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.123+0000 7fa3d31e7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c41b7700 con 0x5612a2b904d0 2024-04-25T11:41:31.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.123+0000 7fa3d31e7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c41b7030 con 0x5612a2b986b0 2024-04-25T11:41:31.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.123+0000 7fa3d31e7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c41b4270 con 0x5612a2bb9390 2024-04-25T11:41:31.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.123+0000 7fa3d31e7740 10 client.0 _ll_register_callbacks cb 0x7fa3b40092c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:31.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.123+0000 7fa3c9fed640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2b904d0 0x5612a2f67430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:31.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.124+0000 7fa3c9fed640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2b904d0 0x5612a2f67430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39456/0 (socket says 172.21.15.90:39456) 2024-04-25T11:41:31.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.124+0000 7fa3c9fed640 1 -- 172.21.15.90:0/1764206342 learned_addr learned my addr 172.21.15.90:0/1764206342 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:31.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.124+0000 7fa3c9fed640 1 -- 172.21.15.90:0/1764206342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612a2bb9390 msgr2=0x5612a2f67eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:31.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.124+0000 7fa3c9fed640 1 --2- 172.21.15.90:0/1764206342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612a2bb9390 0x5612a2f67eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.124+0000 7fa3c9fed640 1 -- 172.21.15.90:0/1764206342 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612a2b986b0 msgr2=0x5612a2f67970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:31.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.124+0000 7fa3c9fed640 1 --2- 172.21.15.90:0/1764206342 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612a2b986b0 0x5612a2f67970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.124+0000 7fa3c9fed640 1 -- 172.21.15.90:0/1764206342 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612a2e804b0 con 0x5612a2b904d0 2024-04-25T11:41:31.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.124+0000 7fa3c9fed640 1 --2- 172.21.15.90:0/1764206342 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2b904d0 0x5612a2f67430 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fa3c002a460 tx=0x7fa3c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aa69e957751fd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:31.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.124+0000 7fa3ad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:41:31.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.124+0000 7fa3ad7fa640 1 -- 172.21.15.90:0/1764206342 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c0035040 con 0x5612a2b904d0 2024-04-25T11:41:31.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.125+0000 7fa3d31e7740 1 -- 172.21.15.90:0/1764206342 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3c0041070 con 0x5612a2b904d0 2024-04-25T11:41:31.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.125+0000 7fa3ad7fa640 1 -- 172.21.15.90:0/1764206342 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fa3c0004a70 con 0x5612a2b904d0 2024-04-25T11:41:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.125+0000 7fa3ad7fa640 1 -- 172.21.15.90:0/1764206342 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa3c0004c70 con 0x5612a2b904d0 2024-04-25T11:41:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.125+0000 7fa3d31e7740 10 client.17460 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.125+0000 7fa3d31e7740 1 -- 172.21.15.90:0/1764206342 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5612a2fddaa0 con 0x5612a2b904d0 2024-04-25T11:41:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.125+0000 7fa3ad7fa640 1 -- 172.21.15.90:0/1764206342 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3c0041070 con 0x5612a2b904d0 2024-04-25T11:41:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.125+0000 7fa3ad7fa640 1 -- 172.21.15.90:0/1764206342 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fa3c0003930 con 0x5612a2b904d0 2024-04-25T11:41:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.126+0000 7fa3d31e7740 10 client.17460 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.126+0000 7fa3d31e7740 10 client.17460 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:41:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.126+0000 7fa3d31e7740 20 client.17460 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.126+0000 7fa3d31e7740 1 --2- 172.21.15.90:0/1764206342 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5612a2b4d3a0 0x5612a2fe02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.126+0000 7fa3d31e7740 4 client.17460 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:41:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.126+0000 7fa3d31e7740 1 -- 172.21.15.90:0/1764206342 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5612a2ec7800 con 0x5612a2b4d3a0 2024-04-25T11:41:31.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.127+0000 7fa3c97ec640 1 --2- 172.21.15.90:0/1764206342 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5612a2b4d3a0 0x5612a2fe02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:31.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.131+0000 7fa3ad7fa640 10 client.17460 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:31.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.131+0000 7fa3c97ec640 1 --2- 172.21.15.90:0/1764206342 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5612a2b4d3a0 0x5612a2fe02c0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80e1cff3ceb206a1 server_cookie=4c84c4ba0f0d60da in_seq=0 out_seq=0 2024-04-25T11:41:31.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.135+0000 7fa3ad7fa640 1 -- 172.21.15.90:0/1764206342 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5612a2ec7800 con 0x5612a2b4d3a0 2024-04-25T11:41:31.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.135+0000 7fa3ad7fa640 10 client.17460 handle_command_reply: tid=0 2024-04-25T11:41:31.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.135+0000 7fa3d31e7740 1 client.17460 shutdown 2024-04-25T11:41:31.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.136+0000 7fa3d31e7740 1 -- 172.21.15.90:0/1764206342 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2b904d0 msgr2=0x5612a2f67430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:31.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.136+0000 7fa3d31e7740 1 --2- 172.21.15.90:0/1764206342 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2b904d0 0x5612a2f67430 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fa3c002a460 tx=0x7fa3c00046c0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.136+0000 7fa3d31e7740 1 -- 172.21.15.90:0/1764206342 shutdown_connections 2024-04-25T11:41:31.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.136+0000 7fa3d31e7740 1 --2- 172.21.15.90:0/1764206342 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5612a2b4d3a0 0x5612a2fe02c0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.136+0000 7fa3d31e7740 1 --2- 172.21.15.90:0/1764206342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612a2bb9390 0x5612a2f67eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.136+0000 7fa3ad7fa640 0 client.17460 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:31.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.136+0000 7fa3d31e7740 1 --2- 172.21.15.90:0/1764206342 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612a2b986b0 0x5612a2f67970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.136+0000 7fa3d31e7740 1 --2- 172.21.15.90:0/1764206342 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612a2b904d0 0x5612a2f67430 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.136+0000 7fa3d31e7740 1 -- 172.21.15.90:0/1764206342 >> 172.21.15.90:0/1764206342 conn(0x5612a2bad4e0 msgr2=0x5612a2f55150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:31.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.136+0000 7fa3d31e7740 1 -- 172.21.15.90:0/1764206342 shutdown_connections 2024-04-25T11:41:31.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.136+0000 7fa3d31e7740 1 -- 172.21.15.90:0/1764206342 wait complete. 2024-04-25T11:41:31.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.136+0000 7fa3d31e7740 20 client.17460 trim_cache size 0 max 16384 2024-04-25T11:41:31.125 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:41:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-25T11:41:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-25T11:41:31.126 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:41:31.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.137+0000 7fa3cc251640 1 -- 172.21.15.90:0/1498654869 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa3b00afd60 msgr2=0x7fa3b00b2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:31.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.137+0000 7fa3cc251640 1 --2- 172.21.15.90:0/1498654869 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa3b00afd60 0x7fa3b00b2270 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x5612a2f5b230 tx=0x7fa3c0031400 comp rx=0 tx=0).stop 2024-04-25T11:41:31.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.137+0000 7fa3cc251640 1 -- 172.21.15.90:0/1498654869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3c410b690 msgr2=0x7fa3c41a21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:31.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.137+0000 7fa3cc251640 1 --2- 172.21.15.90:0/1498654869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3c410b690 0x7fa3c41a21c0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fa3b400b450 tx=0x7fa3b400b910 comp rx=0 tx=0).stop 2024-04-25T11:41:31.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.137+0000 7fa3cc251640 1 -- 172.21.15.90:0/1498654869 shutdown_connections 2024-04-25T11:41:31.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.137+0000 7fa3cc251640 1 --2- 172.21.15.90:0/1498654869 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa3b00afd60 0x7fa3b00b2270 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.137+0000 7fa3cc251640 1 --2- 172.21.15.90:0/1498654869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3c410e080 0x7fa3c41acb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.137+0000 7fa3cc251640 1 --2- 172.21.15.90:0/1498654869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3c410b690 0x7fa3c41a21c0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.137+0000 7fa3cc251640 1 --2- 172.21.15.90:0/1498654869 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa3c4104c30 0x7fa3c41a1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.137+0000 7fa3cc251640 1 -- 172.21.15.90:0/1498654869 >> 172.21.15.90:0/1498654869 conn(0x7fa3c4078110 msgr2=0x7fa3c4102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:31.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.137+0000 7fa3cc251640 1 -- 172.21.15.90:0/1498654869 shutdown_connections 2024-04-25T11:41:31.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.137+0000 7fa3cc251640 1 -- 172.21.15.90:0/1498654869 wait complete. 2024-04-25T11:41:31.144 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T11:41:31.144 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:41:31.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.325+0000 7f6dea7c5640 1 -- 172.21.15.90:0/953742006 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6de4102af0 msgr2=0x7f6de4102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:31.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.325+0000 7f6dea7c5640 1 --2- 172.21.15.90:0/953742006 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6de4102af0 0x7f6de4102f00 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8009900 tx=0x7f6dd802a410 comp rx=0 tx=0).stop 2024-04-25T11:41:31.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.325+0000 7f6dea7c5640 1 -- 172.21.15.90:0/953742006 shutdown_connections 2024-04-25T11:41:31.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.325+0000 7f6dea7c5640 1 --2- 172.21.15.90:0/953742006 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6de410d380 0x7f6de410f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.325+0000 7f6dea7c5640 1 --2- 172.21.15.90:0/953742006 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6de4102af0 0x7f6de4102f00 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.325+0000 7f6dea7c5640 1 --2- 172.21.15.90:0/953742006 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6de4108a20 0x7f6de41025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.325+0000 7f6dea7c5640 1 -- 172.21.15.90:0/953742006 >> 172.21.15.90:0/953742006 conn(0x7f6de4078110 msgr2=0x7f6de4078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:31.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.325+0000 7f6dea7c5640 1 -- 172.21.15.90:0/953742006 shutdown_connections 2024-04-25T11:41:31.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.325+0000 7f6dea7c5640 1 -- 172.21.15.90:0/953742006 wait complete. 2024-04-25T11:41:31.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.326+0000 7f6dea7c5640 1 Processor -- start 2024-04-25T11:41:31.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.326+0000 7f6dea7c5640 1 -- start start 2024-04-25T11:41:31.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.326+0000 7f6dea7c5640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6de4102af0 0x7f6de41a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.326+0000 7f6dea7c5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6de4108a20 0x7f6de41a2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.326+0000 7f6dea7c5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6de410d380 0x7f6de419c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.326+0000 7f6dea7c5640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6de4115ed0 con 0x7f6de410d380 2024-04-25T11:41:31.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.326+0000 7f6dea7c5640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6de4115d50 con 0x7f6de4102af0 2024-04-25T11:41:31.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.326+0000 7f6dea7c5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6de4116050 con 0x7f6de4108a20 2024-04-25T11:41:31.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.326+0000 7f6de8d62640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6de410d380 0x7f6de419c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:31.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.326+0000 7f6de8d62640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6de410d380 0x7f6de419c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39484/0 (socket says 172.21.15.90:39484) 2024-04-25T11:41:31.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.326+0000 7f6de8d62640 1 -- 172.21.15.90:0/3151972026 learned_addr learned my addr 172.21.15.90:0/3151972026 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:31.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.326+0000 7f6de3fff640 1 --2- 172.21.15.90:0/3151972026 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6de4102af0 0x7f6de41a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:31.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.326+0000 7f6de37fe640 1 --2- 172.21.15.90:0/3151972026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6de4108a20 0x7f6de41a2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:31.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.327+0000 7f6de8d62640 1 -- 172.21.15.90:0/3151972026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6de4108a20 msgr2=0x7f6de41a2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:31.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.327+0000 7f6de8d62640 1 --2- 172.21.15.90:0/3151972026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6de4108a20 0x7f6de41a2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.327+0000 7f6de8d62640 1 -- 172.21.15.90:0/3151972026 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6de4102af0 msgr2=0x7f6de41a2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:31.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.327+0000 7f6de8d62640 1 --2- 172.21.15.90:0/3151972026 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6de4102af0 0x7f6de41a2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.327+0000 7f6de8d62640 1 -- 172.21.15.90:0/3151972026 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6de419ce20 con 0x7f6de410d380 2024-04-25T11:41:31.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.327+0000 7f6de3fff640 1 --2- 172.21.15.90:0/3151972026 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6de4102af0 0x7f6de41a2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:31.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.327+0000 7f6de8d62640 1 --2- 172.21.15.90:0/3151972026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6de410d380 0x7f6de419c7b0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc00cce0 tx=0x7f6ddc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:31.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.327+0000 7f6de17fa640 1 -- 172.21.15.90:0/3151972026 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ddc00f450 con 0x7f6de410d380 2024-04-25T11:41:31.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.327+0000 7f6de17fa640 1 -- 172.21.15.90:0/3151972026 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ddc00faa0 con 0x7f6de410d380 2024-04-25T11:41:31.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.327+0000 7f6dea7c5640 1 -- 172.21.15.90:0/3151972026 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6de419d110 con 0x7f6de410d380 2024-04-25T11:41:31.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.327+0000 7f6de17fa640 1 -- 172.21.15.90:0/3151972026 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ddc018b70 con 0x7f6de410d380 2024-04-25T11:41:31.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.327+0000 7f6dea7c5640 1 -- 172.21.15.90:0/3151972026 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6de419d5e0 con 0x7f6de410d380 2024-04-25T11:41:31.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.328+0000 7f6dea7c5640 1 -- 172.21.15.90:0/3151972026 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6de4103cd0 con 0x7f6de410d380 2024-04-25T11:41:31.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.331+0000 7f6de17fa640 1 -- 172.21.15.90:0/3151972026 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f6ddc018d10 con 0x7f6de410d380 2024-04-25T11:41:31.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.331+0000 7f6de17fa640 1 --2- 172.21.15.90:0/3151972026 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f6dcc0afc40 0x7f6dcc0b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.332+0000 7f6de17fa640 1 -- 172.21.15.90:0/3151972026 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f6ddc0df460 con 0x7f6de410d380 2024-04-25T11:41:31.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.332+0000 7f6de3fff640 1 --2- 172.21.15.90:0/3151972026 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f6dcc0afc40 0x7f6dcc0b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:31.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.332+0000 7f6de3fff640 1 --2- 172.21.15.90:0/3151972026 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f6dcc0afc40 0x7f6dcc0b2150 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6de419dae0 tx=0x7f6dd4005f90 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:31.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.337+0000 7f6de17fa640 1 -- 172.21.15.90:0/3151972026 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6ddc014030 con 0x7f6de410d380 2024-04-25T11:41:31.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:31 smithi090 ceph-mon[29575]: pgmap v196: 97 pgs: 97 active+clean; 58 GiB data, 175 GiB used, 898 GiB / 1.0 TiB avail; 3.7 MiB/s rd, 739 MiB/s wr, 1.33k op/s 2024-04-25T11:41:31.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:31 smithi138 ceph-mon[32328]: pgmap v196: 97 pgs: 97 active+clean; 58 GiB data, 175 GiB used, 898 GiB / 1.0 TiB avail; 3.7 MiB/s rd, 739 MiB/s wr, 1.33k op/s 2024-04-25T11:41:31.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.573+0000 7f6dea7c5640 1 -- 172.21.15.90:0/3151972026 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6de406afa0 con 0x7f6de410d380 2024-04-25T11:41:31.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.573+0000 7f6de17fa640 1 -- 172.21.15.90:0/3151972026 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11930 (secure 0 0 0) 0x7f6de406afa0 con 0x7f6de410d380 2024-04-25T11:41:31.561 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:41:31.561 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:39:20.469254+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24458,24431,24499,24475,24524]},"id":1}]} 2024-04-25T11:41:31.561 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 28 2024-04-25T11:41:31.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.576+0000 7f6dea7c5640 1 -- 172.21.15.90:0/3151972026 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f6dcc0afc40 msgr2=0x7f6dcc0b2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:31.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.576+0000 7f6dea7c5640 1 --2- 172.21.15.90:0/3151972026 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f6dcc0afc40 0x7f6dcc0b2150 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6de419dae0 tx=0x7f6dd4005f90 comp rx=0 tx=0).stop 2024-04-25T11:41:31.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.576+0000 7f6dea7c5640 1 -- 172.21.15.90:0/3151972026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6de410d380 msgr2=0x7f6de419c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:31.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.576+0000 7f6dea7c5640 1 --2- 172.21.15.90:0/3151972026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6de410d380 0x7f6de419c7b0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc00cce0 tx=0x7f6ddc00a620 comp rx=0 tx=0).stop 2024-04-25T11:41:31.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.577+0000 7f6dea7c5640 1 -- 172.21.15.90:0/3151972026 shutdown_connections 2024-04-25T11:41:31.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.577+0000 7f6dea7c5640 1 --2- 172.21.15.90:0/3151972026 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f6dcc0afc40 0x7f6dcc0b2150 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.577+0000 7f6dea7c5640 1 --2- 172.21.15.90:0/3151972026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6de410d380 0x7f6de419c7b0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.577+0000 7f6dea7c5640 1 --2- 172.21.15.90:0/3151972026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6de4108a20 0x7f6de41a2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.577+0000 7f6dea7c5640 1 --2- 172.21.15.90:0/3151972026 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6de4102af0 0x7f6de41a2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.577+0000 7f6dea7c5640 1 -- 172.21.15.90:0/3151972026 >> 172.21.15.90:0/3151972026 conn(0x7f6de4078110 msgr2=0x7f6de41040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:31.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.577+0000 7f6dea7c5640 1 -- 172.21.15.90:0/3151972026 shutdown_connections 2024-04-25T11:41:31.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.577+0000 7f6dea7c5640 1 -- 172.21.15.90:0/3151972026 wait complete. 2024-04-25T11:41:31.581 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T11:41:31.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:31 smithi156 ceph-mon[33045]: pgmap v196: 97 pgs: 97 active+clean; 58 GiB data, 175 GiB used, 898 GiB / 1.0 TiB avail; 3.7 MiB/s rd, 739 MiB/s wr, 1.33k op/s 2024-04-25T11:41:31.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.844+0000 7f1e9dd9b640 1 -- 172.21.15.90:0/824537879 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e94035020 con 0x7f1ea01127a0 2024-04-25T11:41:31.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.845+0000 7f1ea50f9640 1 -- 172.21.15.90:0/824537879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ea01127a0 msgr2=0x7f1ea0118d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:31.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.845+0000 7f1ea50f9640 1 --2- 172.21.15.90:0/824537879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ea01127a0 0x7f1ea0118d60 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f1e94009900 tx=0x7f1e9402a420 comp rx=0 tx=0).stop 2024-04-25T11:41:31.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.845+0000 7f1ea50f9640 1 -- 172.21.15.90:0/824537879 shutdown_connections 2024-04-25T11:41:31.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.845+0000 7f1ea50f9640 1 --2- 172.21.15.90:0/824537879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ea01127a0 0x7f1ea0118d60 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.845+0000 7f1ea50f9640 1 --2- 172.21.15.90:0/824537879 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1ea0111ce0 0x7f1ea0112110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.845+0000 7f1ea50f9640 1 --2- 172.21.15.90:0/824537879 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ea010a350 0x7f1ea010a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.845+0000 7f1ea50f9640 1 -- 172.21.15.90:0/824537879 >> 172.21.15.90:0/824537879 conn(0x7f1ea006e7a0 msgr2=0x7f1ea006ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:31.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.846+0000 7f1ea50f9640 1 -- 172.21.15.90:0/824537879 shutdown_connections 2024-04-25T11:41:31.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.846+0000 7f1ea50f9640 1 -- 172.21.15.90:0/824537879 wait complete. 2024-04-25T11:41:31.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.847+0000 7f1ea50f9640 1 Processor -- start 2024-04-25T11:41:31.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.848+0000 7f1ea50f9640 1 -- start start 2024-04-25T11:41:31.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.848+0000 7f1ea50f9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ea010a350 0x7f1ea0116480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.848+0000 7f1ea50f9640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1ea0111ce0 0x7f1ea01169c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.848+0000 7f1ea50f9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ea01127a0 0x7f1ea01b5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.848+0000 7f1ea50f9640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea010d4c0 con 0x7f1ea01127a0 2024-04-25T11:41:31.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.848+0000 7f1ea50f9640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea010d340 con 0x7f1ea0111ce0 2024-04-25T11:41:31.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.848+0000 7f1ea50f9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea010d640 con 0x7f1ea010a350 2024-04-25T11:41:31.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.848+0000 7f1e9f59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ea01127a0 0x7f1ea01b5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:31.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.848+0000 7f1e9f59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ea01127a0 0x7f1ea01b5480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38058/0 (socket says 172.21.15.90:38058) 2024-04-25T11:41:31.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.848+0000 7f1e9f59e640 1 -- 172.21.15.90:0/456541455 learned_addr learned my addr 172.21.15.90:0/456541455 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:31.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.849+0000 7f1e9f59e640 1 -- 172.21.15.90:0/456541455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ea010a350 msgr2=0x7f1ea0116480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:31.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.849+0000 7f1e9f59e640 1 --2- 172.21.15.90:0/456541455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ea010a350 0x7f1ea0116480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.849+0000 7f1e9f59e640 1 -- 172.21.15.90:0/456541455 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1ea0111ce0 msgr2=0x7f1ea01169c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:41:31.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.849+0000 7f1e9f59e640 1 --2- 172.21.15.90:0/456541455 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1ea0111ce0 0x7f1ea01169c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.849+0000 7f1e9f59e640 1 -- 172.21.15.90:0/456541455 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea01b59c0 con 0x7f1ea01127a0 2024-04-25T11:41:31.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.849+0000 7f1e9f59e640 1 --2- 172.21.15.90:0/456541455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ea01127a0 0x7f1ea01b5480 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f1e94005d90 tx=0x7f1e940045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:31.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.849+0000 7f1e87fff640 1 -- 172.21.15.90:0/456541455 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e94040070 con 0x7f1ea01127a0 2024-04-25T11:41:31.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.849+0000 7f1e87fff640 1 -- 172.21.15.90:0/456541455 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e94004ce0 con 0x7f1ea01127a0 2024-04-25T11:41:31.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.849+0000 7f1e87fff640 1 -- 172.21.15.90:0/456541455 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e94035020 con 0x7f1ea01127a0 2024-04-25T11:41:31.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.849+0000 7f1ea50f9640 1 -- 172.21.15.90:0/456541455 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ea01b5b90 con 0x7f1ea01127a0 2024-04-25T11:41:31.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.849+0000 7f1ea50f9640 1 -- 172.21.15.90:0/456541455 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ea01b5f40 con 0x7f1ea01127a0 2024-04-25T11:41:31.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.853+0000 7f1eabfaa740 1 Processor -- start 2024-04-25T11:41:31.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.854+0000 7f1e87fff640 1 -- 172.21.15.90:0/456541455 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f1e94043070 con 0x7f1ea01127a0 2024-04-25T11:41:31.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.854+0000 7f1eabfaa740 1 -- start start 2024-04-25T11:41:31.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.854+0000 7f1eabfaa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b93390 0x561051f34810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.854+0000 7f1eabfaa740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561051b6a4d0 0x561051f34d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.854+0000 7f1eabfaa740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561051b726b0 0x561051f416a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.854+0000 7f1eabfaa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561051e99d80 con 0x561051b93390 2024-04-25T11:41:31.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.854+0000 7f1eabfaa740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x561051e53a20 con 0x561051b6a4d0 2024-04-25T11:41:31.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.854+0000 7f1eabfaa740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561051e95d40 con 0x561051b726b0 2024-04-25T11:41:31.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.855+0000 7f1e9e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b93390 0x561051f34810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:31.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.855+0000 7f1e9e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b93390 0x561051f34810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38064/0 (socket says 172.21.15.90:38064) 2024-04-25T11:41:31.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.855+0000 7f1e9e59c640 1 -- 172.21.15.90:0/3363682816 learned_addr learned my addr 172.21.15.90:0/3363682816 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:31.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.855+0000 7f1e9e59c640 1 -- 172.21.15.90:0/3363682816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561051b726b0 msgr2=0x561051f416a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:41:31.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.855+0000 7f1e9e59c640 1 --2- 172.21.15.90:0/3363682816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561051b726b0 0x561051f416a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.855+0000 7f1e9e59c640 1 -- 172.21.15.90:0/3363682816 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561051b6a4d0 msgr2=0x561051f34d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:31.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.855+0000 7f1e9e59c640 1 --2- 172.21.15.90:0/3363682816 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561051b6a4d0 0x561051f34d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.855+0000 7f1e9e59c640 1 -- 172.21.15.90:0/3363682816 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561051e5ec30 con 0x561051b93390 2024-04-25T11:41:31.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.856+0000 7f1e87fff640 1 --2- 172.21.15.90:0/456541455 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1e880afce0 0x7f1e880b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.856+0000 7f1e9ed9d640 1 --2- 172.21.15.90:0/456541455 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1e880afce0 0x7f1e880b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:31.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.856+0000 7f1e9e59c640 1 --2- 172.21.15.90:0/3363682816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b93390 0x561051f34810 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f1e90009700 tx=0x7f1e90009bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b10f7583979d0552 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:31.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.856+0000 7f1e87fff640 1 -- 172.21.15.90:0/456541455 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f1e940ffd60 con 0x7f1ea01127a0 2024-04-25T11:41:31.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.856+0000 7f1e857fa640 1 -- 172.21.15.90:0/3363682816 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e90015020 con 0x561051b93390 2024-04-25T11:41:31.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.857+0000 7f1e857fa640 1 -- 172.21.15.90:0/3363682816 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e90009e10 con 0x561051b93390 2024-04-25T11:41:31.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.857+0000 7f1e857fa640 1 -- 172.21.15.90:0/3363682816 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e90010380 con 0x561051b93390 2024-04-25T11:41:31.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.858+0000 7f1eabfaa740 1 -- 172.21.15.90:0/3363682816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b93390 msgr2=0x561051f34810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:31.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.858+0000 7f1eabfaa740 1 --2- 172.21.15.90:0/3363682816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b93390 0x561051f34810 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f1e90009700 tx=0x7f1e90009bc0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.858+0000 7f1e9ed9d640 1 --2- 172.21.15.90:0/456541455 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1e880afce0 0x7f1e880b21f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x561051f35fe0 tx=0x7f1e98009250 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:31.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.858+0000 7f1eabfaa740 1 -- 172.21.15.90:0/3363682816 shutdown_connections 2024-04-25T11:41:31.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.858+0000 7f1eabfaa740 1 --2- 172.21.15.90:0/3363682816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561051b726b0 0x561051f416a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.858+0000 7f1eabfaa740 1 --2- 172.21.15.90:0/3363682816 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561051b6a4d0 0x561051f34d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.858+0000 7f1eabfaa740 1 --2- 172.21.15.90:0/3363682816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b93390 0x561051f34810 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.858+0000 7f1eabfaa740 1 -- 172.21.15.90:0/3363682816 >> 172.21.15.90:0/3363682816 conn(0x561051b874e0 msgr2=0x561051f30fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:31.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.858+0000 7f1eabfaa740 1 -- 172.21.15.90:0/3363682816 shutdown_connections 2024-04-25T11:41:31.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.858+0000 7f1eabfaa740 1 -- 172.21.15.90:0/3363682816 wait complete. 2024-04-25T11:41:31.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.859+0000 7f1eabfaa740 1 -- start start 2024-04-25T11:41:31.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.859+0000 7f1eabfaa740 1 Processor -- start 2024-04-25T11:41:31.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.859+0000 7f1eabfaa740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561051b6a4d0 0x561051f3f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.859+0000 7f1eabfaa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b726b0 0x561051f3f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.859+0000 7f1eabfaa740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561051b93390 0x561051f4b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.859+0000 7f1eabfaa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561051f41d90 con 0x561051b726b0 2024-04-25T11:41:31.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.859+0000 7f1eabfaa740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x561051e20dc0 con 0x561051b6a4d0 2024-04-25T11:41:31.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.859+0000 7f1eabfaa740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561051f41f10 con 0x561051b93390 2024-04-25T11:41:31.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.860+0000 7f1eabfaa740 10 client.0 _ll_register_callbacks cb 0x561051e3c600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:31.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.860+0000 7f1e9e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b726b0 0x561051f3f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:31.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.860+0000 7f1e9e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b726b0 0x561051f3f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38080/0 (socket says 172.21.15.90:38080) 2024-04-25T11:41:31.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.860+0000 7f1e9e59c640 1 -- 172.21.15.90:0/3556961816 learned_addr learned my addr 172.21.15.90:0/3556961816 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:31.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.860+0000 7f1e9e59c640 1 -- 172.21.15.90:0/3556961816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561051b93390 msgr2=0x561051f4b210 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:41:31.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.860+0000 7f1e9e59c640 1 --2- 172.21.15.90:0/3556961816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561051b93390 0x561051f4b210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.860+0000 7f1e9e59c640 1 -- 172.21.15.90:0/3556961816 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561051b6a4d0 msgr2=0x561051f3f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:31.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.860+0000 7f1e9e59c640 1 --2- 172.21.15.90:0/3556961816 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561051b6a4d0 0x561051f3f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.860+0000 7f1e9e59c640 1 -- 172.21.15.90:0/3556961816 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561051bab3f0 con 0x561051b726b0 2024-04-25T11:41:31.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.860+0000 7f1e9e59c640 1 --2- 172.21.15.90:0/3556961816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b726b0 0x561051f3f880 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f1e90011b20 tx=0x7f1e900094d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2479b018d074695 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:31.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.860+0000 7f1e82ff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:41:31.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.861+0000 7f1e82ff5640 1 -- 172.21.15.90:0/3556961816 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9000bce0 con 0x561051b726b0 2024-04-25T11:41:31.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.861+0000 7f1eabfaa740 1 -- 172.21.15.90:0/3556961816 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e90009e10 con 0x561051b726b0 2024-04-25T11:41:31.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.861+0000 7f1e82ff5640 1 -- 172.21.15.90:0/3556961816 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f1e90024470 con 0x561051b726b0 2024-04-25T11:41:31.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.861+0000 7f1e82ff5640 1 -- 172.21.15.90:0/3556961816 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1e90015030 con 0x561051b726b0 2024-04-25T11:41:31.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.861+0000 7f1eabfaa740 10 client.17478 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:31.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.861+0000 7f1eabfaa740 1 -- 172.21.15.90:0/3556961816 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561051c757d0 con 0x561051b726b0 2024-04-25T11:41:31.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.862+0000 7f1e82ff5640 1 -- 172.21.15.90:0/3556961816 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e90009e10 con 0x561051b726b0 2024-04-25T11:41:31.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.862+0000 7f1e82ff5640 1 -- 172.21.15.90:0/3556961816 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f1e9002c630 con 0x561051b726b0 2024-04-25T11:41:31.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.862+0000 7f1eabfaa740 10 client.17478 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:31.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.862+0000 7f1eabfaa740 10 client.17478 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:41:31.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.862+0000 7f1eabfaa740 20 client.17478 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:31.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.862+0000 7f1eabfaa740 1 --2- 172.21.15.90:0/3556961816 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x561051b51f60 0x561051f51620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.862+0000 7f1eabfaa740 4 client.17478 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:41:31.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.862+0000 7f1eabfaa740 1 -- 172.21.15.90:0/3556961816 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561051ea7990 con 0x561051b51f60 2024-04-25T11:41:31.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.863+0000 7f1e9ed9d640 1 --2- 172.21.15.90:0/3556961816 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x561051b51f60 0x561051f51620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:31.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.865+0000 7f1e82ff5640 10 client.17478 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:41:31.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.865+0000 7f1e9ed9d640 1 --2- 172.21.15.90:0/3556961816 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x561051b51f60 0x561051f51620 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=362568a7cb5ef367 server_cookie=1f21b3c53ebc692d in_seq=0 out_seq=0 2024-04-25T11:41:31.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.866+0000 7f1e82ff5640 1 -- 172.21.15.90:0/3556961816 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561051ea7990 con 0x561051b51f60 2024-04-25T11:41:31.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.866+0000 7f1e82ff5640 10 client.17478 handle_command_reply: tid=0 2024-04-25T11:41:31.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.866+0000 7f1eabfaa740 1 client.17478 shutdown 2024-04-25T11:41:31.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.866+0000 7f1eabfaa740 1 -- 172.21.15.90:0/3556961816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b726b0 msgr2=0x561051f3f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:31.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.867+0000 7f1eabfaa740 1 --2- 172.21.15.90:0/3556961816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b726b0 0x561051f3f880 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f1e90011b20 tx=0x7f1e900094d0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.867+0000 7f1eabfaa740 1 -- 172.21.15.90:0/3556961816 shutdown_connections 2024-04-25T11:41:31.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.867+0000 7f1eabfaa740 1 --2- 172.21.15.90:0/3556961816 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x561051b51f60 0x561051f51620 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.867+0000 7f1eabfaa740 1 --2- 172.21.15.90:0/3556961816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561051b93390 0x561051f4b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.867+0000 7f1eabfaa740 1 --2- 172.21.15.90:0/3556961816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b726b0 0x561051f3f880 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.867+0000 7f1e82ff5640 0 client.17478 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:41:31.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.867+0000 7f1eabfaa740 1 --2- 172.21.15.90:0/3556961816 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561051b6a4d0 0x561051f3f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.867+0000 7f1eabfaa740 1 -- 172.21.15.90:0/3556961816 >> 172.21.15.90:0/3556961816 conn(0x561051b874e0 msgr2=0x561051f36530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:31.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.867+0000 7f1eabfaa740 1 -- 172.21.15.90:0/3556961816 shutdown_connections 2024-04-25T11:41:31.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.867+0000 7f1eabfaa740 1 -- 172.21.15.90:0/3556961816 wait complete. 2024-04-25T11:41:31.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.867+0000 7f1eabfaa740 20 client.17478 trim_cache size 0 max 16384 2024-04-25T11:41:31.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.886+0000 7f1eabfaa740 1 Processor -- start 2024-04-25T11:41:31.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.886+0000 7f1eabfaa740 1 -- start start 2024-04-25T11:41:31.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.886+0000 7f1eabfaa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b51f60 0x561051f3e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.886+0000 7f1eabfaa740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561051b6a4d0 0x561051f31010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.886+0000 7f1eabfaa740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561051b726b0 0x561051f4ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.886+0000 7f1eabfaa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea01beab0 con 0x561051b51f60 2024-04-25T11:41:31.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.886+0000 7f1eabfaa740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea01c05f0 con 0x561051b726b0 2024-04-25T11:41:31.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.886+0000 7f1eabfaa740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea01bf180 con 0x561051b6a4d0 2024-04-25T11:41:31.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.887+0000 7f1e9ed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b51f60 0x561051f3e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:31.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.887+0000 7f1e9ed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b51f60 0x561051f3e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38092/0 (socket says 172.21.15.90:38092) 2024-04-25T11:41:31.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.887+0000 7f1e9ed9d640 1 -- 172.21.15.90:0/4132671947 learned_addr learned my addr 172.21.15.90:0/4132671947 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:31.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.887+0000 7f1e9ed9d640 1 -- 172.21.15.90:0/4132671947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561051b6a4d0 msgr2=0x561051f31010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:31.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.887+0000 7f1e9ed9d640 1 --2- 172.21.15.90:0/4132671947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561051b6a4d0 0x561051f31010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.887+0000 7f1e9ed9d640 1 -- 172.21.15.90:0/4132671947 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561051b726b0 msgr2=0x561051f4ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:31.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.887+0000 7f1e9ed9d640 1 --2- 172.21.15.90:0/4132671947 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561051b726b0 0x561051f4ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.887+0000 7f1e9ed9d640 1 -- 172.21.15.90:0/4132671947 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561051f41c70 con 0x561051b51f60 2024-04-25T11:41:31.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.887+0000 7f1e9ed9d640 1 --2- 172.21.15.90:0/4132671947 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b51f60 0x561051f3e070 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x561051f31f50 tx=0x7f1e9800f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=242ac339564341df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:31.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.888+0000 7f1e84ff9640 1 -- 172.21.15.90:0/4132671947 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e98021450 con 0x561051b51f60 2024-04-25T11:41:31.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.888+0000 7f1e84ff9640 1 -- 172.21.15.90:0/4132671947 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e98021aa0 con 0x561051b51f60 2024-04-25T11:41:31.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.888+0000 7f1e84ff9640 1 -- 172.21.15.90:0/4132671947 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e980193b0 con 0x561051b51f60 2024-04-25T11:41:31.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.888+0000 7f1eabfaa740 1 -- 172.21.15.90:0/4132671947 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b51f60 msgr2=0x561051f3e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:31.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.888+0000 7f1eabfaa740 1 --2- 172.21.15.90:0/4132671947 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b51f60 0x561051f3e070 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x561051f31f50 tx=0x7f1e9800f000 comp rx=0 tx=0).stop 2024-04-25T11:41:31.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.889+0000 7f1eabfaa740 1 -- 172.21.15.90:0/4132671947 shutdown_connections 2024-04-25T11:41:31.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.889+0000 7f1eabfaa740 1 --2- 172.21.15.90:0/4132671947 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561051b726b0 0x561051f4ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.889+0000 7f1eabfaa740 1 --2- 172.21.15.90:0/4132671947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561051b6a4d0 0x561051f31010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.889+0000 7f1eabfaa740 1 --2- 172.21.15.90:0/4132671947 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b51f60 0x561051f3e070 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.889+0000 7f1eabfaa740 1 -- 172.21.15.90:0/4132671947 >> 172.21.15.90:0/4132671947 conn(0x561051b874e0 msgr2=0x561051f30900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:31.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.889+0000 7f1eabfaa740 1 -- 172.21.15.90:0/4132671947 shutdown_connections 2024-04-25T11:41:31.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.889+0000 7f1eabfaa740 1 -- 172.21.15.90:0/4132671947 wait complete. 2024-04-25T11:41:31.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.889+0000 7f1eabfaa740 1 -- start start 2024-04-25T11:41:31.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.889+0000 7f1eabfaa740 1 Processor -- start 2024-04-25T11:41:31.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.890+0000 7f1eabfaa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b51f60 0x561051f421b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.890+0000 7f1eabfaa740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561051b6a4d0 0x561051f426f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.890+0000 7f1eabfaa740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561051b726b0 0x561051f42c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.890+0000 7f1eabfaa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea01bff20 con 0x561051b51f60 2024-04-25T11:41:31.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.890+0000 7f1eabfaa740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea01bf850 con 0x561051b6a4d0 2024-04-25T11:41:31.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.890+0000 7f1eabfaa740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea01bca90 con 0x561051b726b0 2024-04-25T11:41:31.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.890+0000 7f1e9ed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b51f60 0x561051f421b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:31.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.890+0000 7f1e9ed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b51f60 0x561051f421b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38098/0 (socket says 172.21.15.90:38098) 2024-04-25T11:41:31.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.890+0000 7f1e9ed9d640 1 -- 172.21.15.90:0/2723924399 learned_addr learned my addr 172.21.15.90:0/2723924399 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:31.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.890+0000 7f1eabfaa740 10 client.0 _ll_register_callbacks cb 0x7f1e900095d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:31.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.891+0000 7f1e9ed9d640 1 -- 172.21.15.90:0/2723924399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561051b726b0 msgr2=0x561051f42c30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:41:31.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.891+0000 7f1e9ed9d640 1 --2- 172.21.15.90:0/2723924399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561051b726b0 0x561051f42c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.891+0000 7f1e9ed9d640 1 -- 172.21.15.90:0/2723924399 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561051b6a4d0 msgr2=0x561051f426f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:31.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.891+0000 7f1e9ed9d640 1 --2- 172.21.15.90:0/2723924399 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561051b6a4d0 0x561051f426f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.891+0000 7f1e9ed9d640 1 -- 172.21.15.90:0/2723924399 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561051fb7a50 con 0x561051b51f60 2024-04-25T11:41:31.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.891+0000 7f1e9ed9d640 1 --2- 172.21.15.90:0/2723924399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b51f60 0x561051f421b0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f1e9800d010 tx=0x7f1e9800d650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eef33d658ac20e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:31.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.891+0000 7f1e82ff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:41:31.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.891+0000 7f1e82ff5640 1 -- 172.21.15.90:0/2723924399 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e98021450 con 0x561051b51f60 2024-04-25T11:41:31.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.891+0000 7f1e82ff5640 1 -- 172.21.15.90:0/2723924399 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f1e980215f0 con 0x561051b51f60 2024-04-25T11:41:31.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.891+0000 7f1eabfaa740 1 -- 172.21.15.90:0/2723924399 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e98021aa0 con 0x561051b51f60 2024-04-25T11:41:31.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.891+0000 7f1e82ff5640 1 -- 172.21.15.90:0/2723924399 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1e9800ee40 con 0x561051b51f60 2024-04-25T11:41:31.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.891+0000 7f1eabfaa740 10 client.17484 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:31.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.891+0000 7f1eabfaa740 1 -- 172.21.15.90:0/2723924399 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561051fb8c90 con 0x561051b51f60 2024-04-25T11:41:31.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.892+0000 7f1e82ff5640 1 -- 172.21.15.90:0/2723924399 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e98021aa0 con 0x561051b51f60 2024-04-25T11:41:31.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.892+0000 7f1e82ff5640 1 -- 172.21.15.90:0/2723924399 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f1e98021ce0 con 0x561051b51f60 2024-04-25T11:41:31.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.892+0000 7f1eabfaa740 10 client.17484 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:31.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.892+0000 7f1eabfaa740 10 client.17484 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:41:31.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.892+0000 7f1eabfaa740 20 client.17484 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:31.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.892+0000 7f1eabfaa740 1 --2- 172.21.15.90:0/2723924399 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x561051b93390 0x561051fbb380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:31.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.892+0000 7f1eabfaa740 4 client.17484 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:41:31.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.892+0000 7f1eabfaa740 1 -- 172.21.15.90:0/2723924399 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561051ea1800 con 0x561051b93390 2024-04-25T11:41:31.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.893+0000 7f1e9e59c640 1 --2- 172.21.15.90:0/2723924399 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x561051b93390 0x561051fbb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:31.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.894+0000 7f1e82ff5640 10 client.17484 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:41:31.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.894+0000 7f1e9e59c640 1 --2- 172.21.15.90:0/2723924399 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x561051b93390 0x561051fbb380 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6672b71250da9e2 server_cookie=a26a5ddee0cab19a in_seq=0 out_seq=0 2024-04-25T11:41:31.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.894+0000 7f1e82ff5640 1 -- 172.21.15.90:0/2723924399 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561051ea1800 con 0x561051b93390 2024-04-25T11:41:31.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.894+0000 7f1e82ff5640 10 client.17484 handle_command_reply: tid=0 2024-04-25T11:41:31.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.894+0000 7f1eabfaa740 1 client.17484 shutdown 2024-04-25T11:41:31.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.894+0000 7f1eabfaa740 1 -- 172.21.15.90:0/2723924399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b51f60 msgr2=0x561051f421b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:31.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.894+0000 7f1eabfaa740 1 --2- 172.21.15.90:0/2723924399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b51f60 0x561051f421b0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f1e9800d010 tx=0x7f1e9800d650 comp rx=0 tx=0).stop 2024-04-25T11:41:31.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.895+0000 7f1eabfaa740 1 -- 172.21.15.90:0/2723924399 shutdown_connections 2024-04-25T11:41:31.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.895+0000 7f1eabfaa740 1 --2- 172.21.15.90:0/2723924399 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x561051b93390 0x561051fbb380 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.895+0000 7f1eabfaa740 1 --2- 172.21.15.90:0/2723924399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561051b726b0 0x561051f42c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.895+0000 7f1e82ff5640 0 client.17484 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:41:31.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.895+0000 7f1eabfaa740 1 --2- 172.21.15.90:0/2723924399 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561051b6a4d0 0x561051f426f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.895+0000 7f1eabfaa740 1 --2- 172.21.15.90:0/2723924399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561051b51f60 0x561051f421b0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.895+0000 7f1eabfaa740 1 -- 172.21.15.90:0/2723924399 >> 172.21.15.90:0/2723924399 conn(0x561051b874e0 msgr2=0x561051f41210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:31.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.895+0000 7f1eabfaa740 1 -- 172.21.15.90:0/2723924399 shutdown_connections 2024-04-25T11:41:31.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.895+0000 7f1eabfaa740 1 -- 172.21.15.90:0/2723924399 wait complete. 2024-04-25T11:41:31.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.895+0000 7f1eabfaa740 20 client.17484 trim_cache size 0 max 16384 2024-04-25T11:41:31.886 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:41:31.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.896+0000 7f1ea50f9640 1 -- 172.21.15.90:0/456541455 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1e880afce0 msgr2=0x7f1e880b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:31.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.896+0000 7f1ea50f9640 1 --2- 172.21.15.90:0/456541455 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1e880afce0 0x7f1e880b21f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x561051f35fe0 tx=0x7f1e98009250 comp rx=0 tx=0).stop 2024-04-25T11:41:31.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.896+0000 7f1ea50f9640 1 -- 172.21.15.90:0/456541455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ea01127a0 msgr2=0x7f1ea01b5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:31.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.896+0000 7f1ea50f9640 1 --2- 172.21.15.90:0/456541455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ea01127a0 0x7f1ea01b5480 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f1e94005d90 tx=0x7f1e940045e0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.896+0000 7f1ea50f9640 1 -- 172.21.15.90:0/456541455 shutdown_connections 2024-04-25T11:41:31.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.896+0000 7f1ea50f9640 1 --2- 172.21.15.90:0/456541455 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1e880afce0 0x7f1e880b21f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.896+0000 7f1ea50f9640 1 --2- 172.21.15.90:0/456541455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ea01127a0 0x7f1ea01b5480 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.896+0000 7f1ea50f9640 1 --2- 172.21.15.90:0/456541455 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1ea0111ce0 0x7f1ea01169c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.896+0000 7f1ea50f9640 1 --2- 172.21.15.90:0/456541455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ea010a350 0x7f1ea0116480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:31.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.896+0000 7f1ea50f9640 1 -- 172.21.15.90:0/456541455 >> 172.21.15.90:0/456541455 conn(0x7f1ea006e7a0 msgr2=0x7f1ea0073a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:31.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.897+0000 7f1ea50f9640 1 -- 172.21.15.90:0/456541455 shutdown_connections 2024-04-25T11:41:31.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:31.897+0000 7f1ea50f9640 1 -- 172.21.15.90:0/456541455 wait complete. 2024-04-25T11:41:31.902 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T11:41:32.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.099+0000 7f4b11f14640 1 -- 172.21.15.90:0/3020097741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0c10d380 msgr2=0x7f4b0c10f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.099+0000 7f4b11f14640 1 --2- 172.21.15.90:0/3020097741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0c10d380 0x7f4b0c10f7a0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f4b04009ca0 tx=0x7f4b0402a8c0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.099+0000 7f4b11f14640 1 -- 172.21.15.90:0/3020097741 shutdown_connections 2024-04-25T11:41:32.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.100+0000 7f4b11f14640 1 --2- 172.21.15.90:0/3020097741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0c10d380 0x7f4b0c10f7a0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.100+0000 7f4b11f14640 1 --2- 172.21.15.90:0/3020097741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b0c102af0 0x7f4b0c102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.100+0000 7f4b11f14640 1 --2- 172.21.15.90:0/3020097741 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4b0c108a20 0x7f4b0c1025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.100+0000 7f4b11f14640 1 -- 172.21.15.90:0/3020097741 >> 172.21.15.90:0/3020097741 conn(0x7f4b0c078100 msgr2=0x7f4b0c078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:32.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.100+0000 7f4b11f14640 1 -- 172.21.15.90:0/3020097741 shutdown_connections 2024-04-25T11:41:32.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.100+0000 7f4b11f14640 1 -- 172.21.15.90:0/3020097741 wait complete. 2024-04-25T11:41:32.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.100+0000 7f4b11f14640 1 Processor -- start 2024-04-25T11:41:32.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.100+0000 7f4b11f14640 1 -- start start 2024-04-25T11:41:32.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.101+0000 7f4b11f14640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4b0c102af0 0x7f4b0c19b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.101+0000 7f4b11f14640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b0c108a20 0x7f4b0c19ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.101+0000 7f4b11f14640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0c10d380 0x7f4b0c19bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.101+0000 7f4b11f14640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0c112c50 con 0x7f4b0c10d380 2024-04-25T11:41:32.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.101+0000 7f4b11f14640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0c112ad0 con 0x7f4b0c102af0 2024-04-25T11:41:32.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.101+0000 7f4b11f14640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0c112dd0 con 0x7f4b0c108a20 2024-04-25T11:41:32.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.101+0000 7f4b0bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0c10d380 0x7f4b0c19bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.101+0000 7f4b0bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0c10d380 0x7f4b0c19bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38118/0 (socket says 172.21.15.90:38118) 2024-04-25T11:41:32.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.101+0000 7f4b0bfff640 1 -- 172.21.15.90:0/247315122 learned_addr learned my addr 172.21.15.90:0/247315122 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:32.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.101+0000 7f4b0b7fe640 1 --2- 172.21.15.90:0/247315122 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4b0c102af0 0x7f4b0c19b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.101+0000 7f4b0affd640 1 --2- 172.21.15.90:0/247315122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b0c108a20 0x7f4b0c19ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.101+0000 7f4b0bfff640 1 -- 172.21.15.90:0/247315122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b0c108a20 msgr2=0x7f4b0c19ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.101+0000 7f4b0bfff640 1 --2- 172.21.15.90:0/247315122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b0c108a20 0x7f4b0c19ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.101+0000 7f4b0bfff640 1 -- 172.21.15.90:0/247315122 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4b0c102af0 msgr2=0x7f4b0c19b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.101+0000 7f4b0bfff640 1 --2- 172.21.15.90:0/247315122 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4b0c102af0 0x7f4b0c19b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.101+0000 7f4b0bfff640 1 -- 172.21.15.90:0/247315122 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b0c1a6870 con 0x7f4b0c10d380 2024-04-25T11:41:32.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.102+0000 7f4b0bfff640 1 --2- 172.21.15.90:0/247315122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0c10d380 0x7f4b0c19bfd0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f4b04009430 tx=0x7f4b040317c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:32.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.102+0000 7f4b08ff9640 1 -- 172.21.15.90:0/247315122 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b04003b70 con 0x7f4b0c10d380 2024-04-25T11:41:32.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.102+0000 7f4b11f14640 1 -- 172.21.15.90:0/247315122 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b0c1a6b00 con 0x7f4b0c10d380 2024-04-25T11:41:32.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.102+0000 7f4b08ff9640 1 -- 172.21.15.90:0/247315122 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b04003d10 con 0x7f4b0c10d380 2024-04-25T11:41:32.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.102+0000 7f4b08ff9640 1 -- 172.21.15.90:0/247315122 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0402fa30 con 0x7f4b0c10d380 2024-04-25T11:41:32.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.102+0000 7f4b11f14640 1 -- 172.21.15.90:0/247315122 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b0c1a70a0 con 0x7f4b0c10d380 2024-04-25T11:41:32.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.105+0000 7f4b18dc7740 1 Processor -- start 2024-04-25T11:41:32.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.105+0000 7f4b08ff9640 1 -- 172.21.15.90:0/247315122 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f4b0402fbd0 con 0x7f4b0c10d380 2024-04-25T11:41:32.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.105+0000 7f4b18dc7740 1 -- start start 2024-04-25T11:41:32.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.106+0000 7f4b18dc7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffd0c41390 0x55ffd0fe2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.106+0000 7f4b18dc7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffd0c184d0 0x55ffd0fe2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.106+0000 7f4b18dc7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffd0c206b0 0x55ffd0fef6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.106+0000 7f4b18dc7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ffd0f47d80 con 0x55ffd0c184d0 2024-04-25T11:41:32.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.106+0000 7f4b18dc7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ffd0f01a20 con 0x55ffd0c206b0 2024-04-25T11:41:32.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.106+0000 7f4b18dc7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ffd0f43d40 con 0x55ffd0c41390 2024-04-25T11:41:32.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.106+0000 7f4b0b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffd0c184d0 0x55ffd0fe2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.106+0000 7f4b0b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffd0c184d0 0x55ffd0fe2d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38126/0 (socket says 172.21.15.90:38126) 2024-04-25T11:41:32.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.106+0000 7f4b0b7fe640 1 -- 172.21.15.90:0/2393909360 learned_addr learned my addr 172.21.15.90:0/2393909360 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:32.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.106+0000 7f4b0affd640 1 --2- 172.21.15.90:0/2393909360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffd0c41390 0x55ffd0fe2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.106+0000 7f4b08ff9640 1 --2- 172.21.15.90:0/247315122 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4af40afc60 0x7f4af40b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.106+0000 7f4b0affd640 1 --2- 172.21.15.90:0/2393909360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffd0c206b0 0x55ffd0fef6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.106+0000 7f4b0b7fe640 1 -- 172.21.15.90:0/2393909360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffd0c41390 msgr2=0x55ffd0fe2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.106+0000 7f4b0b7fe640 1 --2- 172.21.15.90:0/2393909360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffd0c41390 0x55ffd0fe2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.106+0000 7f4b0b7fe640 1 -- 172.21.15.90:0/2393909360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffd0c206b0 msgr2=0x55ffd0fef6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.106+0000 7f4b0b7fe640 1 --2- 172.21.15.90:0/2393909360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffd0c206b0 0x55ffd0fef6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.106+0000 7f4b0b7fe640 1 -- 172.21.15.90:0/2393909360 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffd0f0cc30 con 0x55ffd0c184d0 2024-04-25T11:41:32.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.106+0000 7f4b0affd640 1 --2- 172.21.15.90:0/2393909360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffd0c41390 0x55ffd0fe2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:41:32.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.106+0000 7f4b0bfff640 1 --2- 172.21.15.90:0/247315122 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4af40afc60 0x7f4af40b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.106+0000 7f4b0b7fe640 1 --2- 172.21.15.90:0/2393909360 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffd0c184d0 0x55ffd0fe2d50 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f4afc009c10 tx=0x7f4afc011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5730286058966d6d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:32.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.107+0000 7f4b08ff9640 1 -- 172.21.15.90:0/247315122 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f4b04100420 con 0x7f4b0c10d380 2024-04-25T11:41:32.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.107+0000 7f4af1ffb640 1 -- 172.21.15.90:0/2393909360 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4afc004230 con 0x55ffd0c184d0 2024-04-25T11:41:32.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.107+0000 7f4af1ffb640 1 -- 172.21.15.90:0/2393909360 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4afc0043d0 con 0x55ffd0c184d0 2024-04-25T11:41:32.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.107+0000 7f4af1ffb640 1 -- 172.21.15.90:0/2393909360 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4afc004d50 con 0x55ffd0c184d0 2024-04-25T11:41:32.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.108+0000 7f4b18dc7740 1 -- 172.21.15.90:0/2393909360 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffd0c184d0 msgr2=0x55ffd0fe2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.108+0000 7f4b18dc7740 1 --2- 172.21.15.90:0/2393909360 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffd0c184d0 0x55ffd0fe2d50 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f4afc009c10 tx=0x7f4afc011810 comp rx=0 tx=0).stop 2024-04-25T11:41:32.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.108+0000 7f4b0bfff640 1 --2- 172.21.15.90:0/247315122 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4af40afc60 0x7f4af40b2170 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4b040fdcd0 tx=0x7f4b040bcf60 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:32.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.108+0000 7f4b18dc7740 1 -- 172.21.15.90:0/2393909360 shutdown_connections 2024-04-25T11:41:32.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.108+0000 7f4b18dc7740 1 --2- 172.21.15.90:0/2393909360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffd0c206b0 0x55ffd0fef6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.108+0000 7f4b18dc7740 1 --2- 172.21.15.90:0/2393909360 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffd0c184d0 0x55ffd0fe2d50 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.108+0000 7f4b18dc7740 1 --2- 172.21.15.90:0/2393909360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffd0c41390 0x55ffd0fe2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.108+0000 7f4b18dc7740 1 -- 172.21.15.90:0/2393909360 >> 172.21.15.90:0/2393909360 conn(0x55ffd0c354e0 msgr2=0x55ffd0fdefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:32.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.108+0000 7f4b18dc7740 1 -- 172.21.15.90:0/2393909360 shutdown_connections 2024-04-25T11:41:32.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.108+0000 7f4b18dc7740 1 -- 172.21.15.90:0/2393909360 wait complete. 2024-04-25T11:41:32.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.108+0000 7f4b18dc7740 1 -- start start 2024-04-25T11:41:32.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.108+0000 7f4b18dc7740 1 Processor -- start 2024-04-25T11:41:32.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.109+0000 7f4b18dc7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffd0c184d0 0x55ffd0fed2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.109+0000 7f4b18dc7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffd0c206b0 0x55ffd0fed7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.109+0000 7f4b18dc7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffd0c41390 0x55ffd0ff9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.109+0000 7f4b18dc7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ffd0fefd90 con 0x55ffd0c184d0 2024-04-25T11:41:32.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.109+0000 7f4b18dc7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ffd0ecedc0 con 0x55ffd0c41390 2024-04-25T11:41:32.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.109+0000 7f4b18dc7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ffd0feff10 con 0x55ffd0c206b0 2024-04-25T11:41:32.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.109+0000 7f4b18dc7740 10 client.0 _ll_register_callbacks cb 0x55ffd0eea600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:32.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.109+0000 7f4b0affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffd0c184d0 0x55ffd0fed2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.109+0000 7f4b0affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffd0c184d0 0x55ffd0fed2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38128/0 (socket says 172.21.15.90:38128) 2024-04-25T11:41:32.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.109+0000 7f4b0affd640 1 -- 172.21.15.90:0/312954190 learned_addr learned my addr 172.21.15.90:0/312954190 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:32.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.109+0000 7f4b0b7fe640 1 --2- 172.21.15.90:0/312954190 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffd0c206b0 0x55ffd0fed7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.109+0000 7f4b0affd640 1 --2- 172.21.15.90:0/312954190 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffd0c41390 0x55ffd0ff9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.109+0000 7f4b0affd640 1 -- 172.21.15.90:0/312954190 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffd0c206b0 msgr2=0x55ffd0fed7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.109+0000 7f4b0affd640 1 --2- 172.21.15.90:0/312954190 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffd0c206b0 0x55ffd0fed7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.109+0000 7f4b0affd640 1 -- 172.21.15.90:0/312954190 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffd0c41390 msgr2=0x55ffd0ff9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.109+0000 7f4b0affd640 1 --2- 172.21.15.90:0/312954190 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffd0c41390 0x55ffd0ff9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.109+0000 7f4b0affd640 1 -- 172.21.15.90:0/312954190 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffd0c593f0 con 0x55ffd0c184d0 2024-04-25T11:41:32.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.109+0000 7f4b0affd640 1 --2- 172.21.15.90:0/312954190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffd0c184d0 0x55ffd0fed2a0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c075fd0 tx=0x7f4b000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c00e9fb67ff7e3bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:32.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.109+0000 7f4aeaffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:41:32.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.109+0000 7f4aeaffd640 1 -- 172.21.15.90:0/312954190 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0001b040 con 0x55ffd0c184d0 2024-04-25T11:41:32.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.110+0000 7f4b18dc7740 1 -- 172.21.15.90:0/312954190 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4afc0043d0 con 0x55ffd0c184d0 2024-04-25T11:41:32.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.110+0000 7f4aeaffd640 1 -- 172.21.15.90:0/312954190 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f4b000140d0 con 0x55ffd0c184d0 2024-04-25T11:41:32.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.110+0000 7f4aeaffd640 1 -- 172.21.15.90:0/312954190 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4b00010c30 con 0x55ffd0c184d0 2024-04-25T11:41:32.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.110+0000 7f4b18dc7740 10 client.17496 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:32.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.110+0000 7f4b18dc7740 1 -- 172.21.15.90:0/312954190 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ffd0d237d0 con 0x55ffd0c184d0 2024-04-25T11:41:32.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.110+0000 7f4aeaffd640 1 -- 172.21.15.90:0/312954190 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4afc0043d0 con 0x55ffd0c184d0 2024-04-25T11:41:32.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.110+0000 7f4aeaffd640 1 -- 172.21.15.90:0/312954190 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f4b00012c40 con 0x55ffd0c184d0 2024-04-25T11:41:32.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.110+0000 7f4b18dc7740 10 client.17496 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:32.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.110+0000 7f4b18dc7740 10 client.17496 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:41:32.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.110+0000 7f4b18dc7740 20 client.17496 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:32.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.110+0000 7f4b18dc7740 1 --2- 172.21.15.90:0/312954190 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55ffd0bfff60 0x55ffd0fff6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.110+0000 7f4b18dc7740 4 client.17496 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:41:32.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.110+0000 7f4b18dc7740 1 -- 172.21.15.90:0/312954190 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ffd0f55990 con 0x55ffd0bfff60 2024-04-25T11:41:32.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.110+0000 7f4b0b7fe640 1 --2- 172.21.15.90:0/312954190 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55ffd0bfff60 0x55ffd0fff6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.111+0000 7f4aeaffd640 10 client.17496 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:41:32.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.111+0000 7f4b0b7fe640 1 --2- 172.21.15.90:0/312954190 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55ffd0bfff60 0x55ffd0fff6f0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a580664f0b7ab6ca server_cookie=2618ba9faaa63afb in_seq=0 out_seq=0 2024-04-25T11:41:32.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.112+0000 7f4aeaffd640 1 -- 172.21.15.90:0/312954190 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ffd0f55990 con 0x55ffd0bfff60 2024-04-25T11:41:32.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.112+0000 7f4aeaffd640 10 client.17496 handle_command_reply: tid=0 2024-04-25T11:41:32.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.112+0000 7f4b18dc7740 1 client.17496 shutdown 2024-04-25T11:41:32.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.112+0000 7f4b18dc7740 1 -- 172.21.15.90:0/312954190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffd0c184d0 msgr2=0x55ffd0fed2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.112+0000 7f4b18dc7740 1 --2- 172.21.15.90:0/312954190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffd0c184d0 0x55ffd0fed2a0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c075fd0 tx=0x7f4b000046c0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.112+0000 7f4b18dc7740 1 -- 172.21.15.90:0/312954190 shutdown_connections 2024-04-25T11:41:32.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.112+0000 7f4b18dc7740 1 --2- 172.21.15.90:0/312954190 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55ffd0bfff60 0x55ffd0fff6f0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.112+0000 7f4b18dc7740 1 --2- 172.21.15.90:0/312954190 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffd0c41390 0x55ffd0ff9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.112+0000 7f4b18dc7740 1 --2- 172.21.15.90:0/312954190 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffd0c206b0 0x55ffd0fed7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.112+0000 7f4b18dc7740 1 --2- 172.21.15.90:0/312954190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffd0c184d0 0x55ffd0fed2a0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.112+0000 7f4aeaffd640 0 client.17496 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:41:32.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.112+0000 7f4b18dc7740 1 -- 172.21.15.90:0/312954190 >> 172.21.15.90:0/312954190 conn(0x55ffd0c354e0 msgr2=0x55ffd0fe4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:32.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.112+0000 7f4b18dc7740 1 -- 172.21.15.90:0/312954190 shutdown_connections 2024-04-25T11:41:32.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.112+0000 7f4b18dc7740 1 -- 172.21.15.90:0/312954190 wait complete. 2024-04-25T11:41:32.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.112+0000 7f4b18dc7740 20 client.17496 trim_cache size 0 max 16384 2024-04-25T11:41:32.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.126+0000 7f4b18dc7740 1 Processor -- start 2024-04-25T11:41:32.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.126+0000 7f4b18dc7740 1 -- start start 2024-04-25T11:41:32.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.127+0000 7f4b18dc7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffd0bfff60 0x55ffd0ff8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.127+0000 7f4b18dc7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffd0c184d0 0x55ffd0fddd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.127+0000 7f4b18dc7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffd0c206b0 0x55ffd0fde260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.127+0000 7f4b18dc7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0c1afc10 con 0x55ffd0c206b0 2024-04-25T11:41:32.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.127+0000 7f4b18dc7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0c1b1750 con 0x55ffd0c184d0 2024-04-25T11:41:32.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.127+0000 7f4b18dc7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0c1b02e0 con 0x55ffd0bfff60 2024-04-25T11:41:32.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.127+0000 7f4b0b7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffd0c184d0 0x55ffd0fddd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.127+0000 7f4b0affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffd0c206b0 0x55ffd0fde260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.127+0000 7f4b0b7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffd0c184d0 0x55ffd0fddd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:58168/0 (socket says 172.21.15.90:58168) 2024-04-25T11:41:32.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.127+0000 7f4b0b7fe640 1 -- 172.21.15.90:0/3672402946 learned_addr learned my addr 172.21.15.90:0/3672402946 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:32.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.127+0000 7f4b0affd640 1 --2- 172.21.15.90:0/3672402946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffd0bfff60 0x55ffd0ff8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.127+0000 7f4b0b7fe640 1 -- 172.21.15.90:0/3672402946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffd0bfff60 msgr2=0x55ffd0ff8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.127+0000 7f4b0b7fe640 1 --2- 172.21.15.90:0/3672402946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffd0bfff60 0x55ffd0ff8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.127+0000 7f4b0b7fe640 1 -- 172.21.15.90:0/3672402946 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffd0c206b0 msgr2=0x55ffd0fde260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.127+0000 7f4b0b7fe640 1 --2- 172.21.15.90:0/3672402946 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffd0c206b0 0x55ffd0fde260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.127+0000 7f4b0b7fe640 1 -- 172.21.15.90:0/3672402946 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffd0fe8680 con 0x55ffd0c184d0 2024-04-25T11:41:32.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.128+0000 7f4b0b7fe640 1 --2- 172.21.15.90:0/3672402946 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffd0c184d0 0x55ffd0fddd20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x55ffd0fdff50 tx=0x7f4afc004220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5397daba5ed405c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:32.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.128+0000 7f4af17fa640 1 -- 172.21.15.90:0/3672402946 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4afc01e040 con 0x55ffd0c184d0 2024-04-25T11:41:32.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.128+0000 7f4af17fa640 1 -- 172.21.15.90:0/3672402946 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4afc01f040 con 0x55ffd0c184d0 2024-04-25T11:41:32.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.128+0000 7f4af17fa640 1 -- 172.21.15.90:0/3672402946 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4afc003850 con 0x55ffd0c184d0 2024-04-25T11:41:32.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.129+0000 7f4b18dc7740 1 -- 172.21.15.90:0/3672402946 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffd0c184d0 msgr2=0x55ffd0fddd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.129+0000 7f4b18dc7740 1 --2- 172.21.15.90:0/3672402946 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffd0c184d0 0x55ffd0fddd20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x55ffd0fdff50 tx=0x7f4afc004220 comp rx=0 tx=0).stop 2024-04-25T11:41:32.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.129+0000 7f4b18dc7740 1 -- 172.21.15.90:0/3672402946 shutdown_connections 2024-04-25T11:41:32.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.129+0000 7f4b18dc7740 1 --2- 172.21.15.90:0/3672402946 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffd0c206b0 0x55ffd0fde260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.129+0000 7f4b18dc7740 1 --2- 172.21.15.90:0/3672402946 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffd0c184d0 0x55ffd0fddd20 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.129+0000 7f4b18dc7740 1 --2- 172.21.15.90:0/3672402946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffd0bfff60 0x55ffd0ff8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.129+0000 7f4b18dc7740 1 -- 172.21.15.90:0/3672402946 >> 172.21.15.90:0/3672402946 conn(0x55ffd0c354e0 msgr2=0x55ffd0fdd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:32.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.129+0000 7f4b18dc7740 1 -- 172.21.15.90:0/3672402946 shutdown_connections 2024-04-25T11:41:32.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.129+0000 7f4b18dc7740 1 -- 172.21.15.90:0/3672402946 wait complete. 2024-04-25T11:41:32.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.129+0000 7f4b18dc7740 1 -- start start 2024-04-25T11:41:32.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.129+0000 7f4b18dc7740 1 Processor -- start 2024-04-25T11:41:32.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.130+0000 7f4b18dc7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffd0bfff60 0x55ffd0ff5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.130+0000 7f4b18dc7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffd0c184d0 0x55ffd0ff55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.130+0000 7f4b18dc7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffd0c206b0 0x55ffd0ff05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.130+0000 7f4b18dc7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0c1b1080 con 0x55ffd0c206b0 2024-04-25T11:41:32.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.130+0000 7f4b18dc7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0c1b09b0 con 0x55ffd0c184d0 2024-04-25T11:41:32.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.130+0000 7f4b18dc7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0c1adbf0 con 0x55ffd0bfff60 2024-04-25T11:41:32.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.130+0000 7f4b18dc7740 10 client.0 _ll_register_callbacks cb 0x7f4b00014890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:32.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.130+0000 7f4b0b7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffd0c184d0 0x55ffd0ff55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.130+0000 7f4b0affd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffd0bfff60 0x55ffd0ff5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.130+0000 7f4b0b7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffd0c184d0 0x55ffd0ff55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:58176/0 (socket says 172.21.15.90:58176) 2024-04-25T11:41:32.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.130+0000 7f4b0b7fe640 1 -- 172.21.15.90:0/1950429223 learned_addr learned my addr 172.21.15.90:0/1950429223 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:32.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.130+0000 7f4b0affd640 1 --2- 172.21.15.90:0/1950429223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffd0c206b0 0x55ffd0ff05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.130+0000 7f4b0b7fe640 1 -- 172.21.15.90:0/1950429223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffd0bfff60 msgr2=0x55ffd0ff5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.130+0000 7f4b0b7fe640 1 --2- 172.21.15.90:0/1950429223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffd0bfff60 0x55ffd0ff5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.130+0000 7f4b0b7fe640 1 -- 172.21.15.90:0/1950429223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffd0c206b0 msgr2=0x55ffd0ff05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.130+0000 7f4b0b7fe640 1 --2- 172.21.15.90:0/1950429223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffd0c206b0 0x55ffd0ff05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.130+0000 7f4b0b7fe640 1 -- 172.21.15.90:0/1950429223 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffd0ff0cb0 con 0x55ffd0c184d0 2024-04-25T11:41:32.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.130+0000 7f4b0b7fe640 1 --2- 172.21.15.90:0/1950429223 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffd0c184d0 0x55ffd0ff55c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4afc02c9e0 tx=0x7f4afc009e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afa01a17b1a452a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:32.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.130+0000 7f4aeaffd640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:41:32.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.131+0000 7f4aeaffd640 1 -- 172.21.15.90:0/1950429223 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4afc02c4e0 con 0x55ffd0c184d0 2024-04-25T11:41:32.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.131+0000 7f4aeaffd640 1 -- 172.21.15.90:0/1950429223 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f4afc02c680 con 0x55ffd0c184d0 2024-04-25T11:41:32.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.131+0000 7f4b18dc7740 1 -- 172.21.15.90:0/1950429223 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4afc01f040 con 0x55ffd0c184d0 2024-04-25T11:41:32.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.131+0000 7f4aeaffd640 1 -- 172.21.15.90:0/1950429223 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4afc0035f0 con 0x55ffd0c184d0 2024-04-25T11:41:32.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.131+0000 7f4b18dc7740 10 client.26480 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:32.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.131+0000 7f4b18dc7740 1 -- 172.21.15.90:0/1950429223 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ffd1067320 con 0x55ffd0c184d0 2024-04-25T11:41:32.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.131+0000 7f4aeaffd640 1 -- 172.21.15.90:0/1950429223 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4afc01f040 con 0x55ffd0c184d0 2024-04-25T11:41:32.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.131+0000 7f4aeaffd640 1 -- 172.21.15.90:0/1950429223 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f4afc024040 con 0x55ffd0c184d0 2024-04-25T11:41:32.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.131+0000 7f4b18dc7740 10 client.26480 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:32.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.131+0000 7f4b18dc7740 10 client.26480 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:41:32.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.131+0000 7f4b18dc7740 20 client.26480 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:32.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.131+0000 7f4b18dc7740 1 --2- 172.21.15.90:0/1950429223 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55ffd0c41390 0x55ffd1069820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.131+0000 7f4b18dc7740 4 client.26480 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:41:32.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.131+0000 7f4b18dc7740 1 -- 172.21.15.90:0/1950429223 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ffd0f4f800 con 0x55ffd0c41390 2024-04-25T11:41:32.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.132+0000 7f4b0affd640 1 --2- 172.21.15.90:0/1950429223 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55ffd0c41390 0x55ffd1069820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.132+0000 7f4aeaffd640 10 client.26480 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:41:32.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.132+0000 7f4b0affd640 1 --2- 172.21.15.90:0/1950429223 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55ffd0c41390 0x55ffd1069820 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3593b55d9281ffc8 server_cookie=1ddebed09e087bb5 in_seq=0 out_seq=0 2024-04-25T11:41:32.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.132+0000 7f4aeaffd640 1 -- 172.21.15.90:0/1950429223 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ffd0f4f800 con 0x55ffd0c41390 2024-04-25T11:41:32.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.132+0000 7f4aeaffd640 10 client.26480 handle_command_reply: tid=0 2024-04-25T11:41:32.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.132+0000 7f4b18dc7740 1 client.26480 shutdown 2024-04-25T11:41:32.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.133+0000 7f4b18dc7740 1 -- 172.21.15.90:0/1950429223 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffd0c184d0 msgr2=0x55ffd0ff55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.133+0000 7f4b18dc7740 1 --2- 172.21.15.90:0/1950429223 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffd0c184d0 0x55ffd0ff55c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4afc02c9e0 tx=0x7f4afc009e80 comp rx=0 tx=0).stop 2024-04-25T11:41:32.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.133+0000 7f4b18dc7740 1 -- 172.21.15.90:0/1950429223 shutdown_connections 2024-04-25T11:41:32.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.133+0000 7f4b18dc7740 1 --2- 172.21.15.90:0/1950429223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffd0c206b0 0x55ffd0ff05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.133+0000 7f4b18dc7740 1 --2- 172.21.15.90:0/1950429223 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55ffd0c41390 0x55ffd1069820 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.133+0000 7f4b18dc7740 1 --2- 172.21.15.90:0/1950429223 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffd0c184d0 0x55ffd0ff55c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.133+0000 7f4b18dc7740 1 --2- 172.21.15.90:0/1950429223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffd0bfff60 0x55ffd0ff5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.133+0000 7f4aeaffd640 0 client.26480 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:41:32.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.133+0000 7f4b18dc7740 1 -- 172.21.15.90:0/1950429223 >> 172.21.15.90:0/1950429223 conn(0x55ffd0c354e0 msgr2=0x55ffd0fee710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:32.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.133+0000 7f4b18dc7740 1 -- 172.21.15.90:0/1950429223 shutdown_connections 2024-04-25T11:41:32.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.133+0000 7f4b18dc7740 1 -- 172.21.15.90:0/1950429223 wait complete. 2024-04-25T11:41:32.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.133+0000 7f4b18dc7740 20 client.26480 trim_cache size 0 max 16384 2024-04-25T11:41:32.125 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:41:32.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.133+0000 7f4b11f14640 1 -- 172.21.15.90:0/247315122 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4af40afc60 msgr2=0x7f4af40b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.133+0000 7f4b11f14640 1 --2- 172.21.15.90:0/247315122 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4af40afc60 0x7f4af40b2170 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4b040fdcd0 tx=0x7f4b040bcf60 comp rx=0 tx=0).stop 2024-04-25T11:41:32.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.133+0000 7f4b11f14640 1 -- 172.21.15.90:0/247315122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0c10d380 msgr2=0x7f4b0c19bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.133+0000 7f4b11f14640 1 --2- 172.21.15.90:0/247315122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0c10d380 0x7f4b0c19bfd0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f4b04009430 tx=0x7f4b040317c0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.133+0000 7f4b11f14640 1 -- 172.21.15.90:0/247315122 shutdown_connections 2024-04-25T11:41:32.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.133+0000 7f4b11f14640 1 --2- 172.21.15.90:0/247315122 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4af40afc60 0x7f4af40b2170 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.133+0000 7f4b11f14640 1 --2- 172.21.15.90:0/247315122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0c10d380 0x7f4b0c19bfd0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.133+0000 7f4b11f14640 1 --2- 172.21.15.90:0/247315122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b0c108a20 0x7f4b0c19ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.133+0000 7f4b11f14640 1 --2- 172.21.15.90:0/247315122 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4b0c102af0 0x7f4b0c19b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.133+0000 7f4b11f14640 1 -- 172.21.15.90:0/247315122 >> 172.21.15.90:0/247315122 conn(0x7f4b0c078100 msgr2=0x7f4b0c1040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:32.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.134+0000 7f4b11f14640 1 -- 172.21.15.90:0/247315122 shutdown_connections 2024-04-25T11:41:32.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.134+0000 7f4b11f14640 1 -- 172.21.15.90:0/247315122 wait complete. 2024-04-25T11:41:32.134 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T11:41:32.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.266+0000 7f49bdc61640 1 -- 172.21.15.90:0/3370790699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49b810d3a0 msgr2=0x7f49b810f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.266+0000 7f49bdc61640 1 --2- 172.21.15.90:0/3370790699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49b810d3a0 0x7f49b810f7c0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f49b0009900 tx=0x7f49b002a410 comp rx=0 tx=0).stop 2024-04-25T11:41:32.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.266+0000 7f49bdc61640 1 -- 172.21.15.90:0/3370790699 shutdown_connections 2024-04-25T11:41:32.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.266+0000 7f49bdc61640 1 --2- 172.21.15.90:0/3370790699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49b810d3a0 0x7f49b810f7c0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.266+0000 7f49bdc61640 1 --2- 172.21.15.90:0/3370790699 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f49b8102b10 0x7f49b8102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.266+0000 7f49bdc61640 1 --2- 172.21.15.90:0/3370790699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49b8108a40 0x7f49b81025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.266+0000 7f49bdc61640 1 -- 172.21.15.90:0/3370790699 >> 172.21.15.90:0/3370790699 conn(0x7f49b8078180 msgr2=0x7f49b80785c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:32.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.266+0000 7f49bdc61640 1 -- 172.21.15.90:0/3370790699 shutdown_connections 2024-04-25T11:41:32.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.266+0000 7f49bdc61640 1 -- 172.21.15.90:0/3370790699 wait complete. 2024-04-25T11:41:32.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.267+0000 7f49bdc61640 1 Processor -- start 2024-04-25T11:41:32.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.267+0000 7f49bdc61640 1 -- start start 2024-04-25T11:41:32.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.267+0000 7f49bdc61640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49b8102b10 0x7f49b819b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.267+0000 7f49bdc61640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f49b8108a40 0x7f49b819b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.267+0000 7f49bdc61640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49b810d3a0 0x7f49b819bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.267+0000 7f49bdc61640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f49b8112c00 con 0x7f49b810d3a0 2024-04-25T11:41:32.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.267+0000 7f49bdc61640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f49b8112a80 con 0x7f49b8108a40 2024-04-25T11:41:32.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.267+0000 7f49bdc61640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f49b8112d80 con 0x7f49b8102b10 2024-04-25T11:41:32.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.267+0000 7f49b7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49b810d3a0 0x7f49b819bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.267+0000 7f49b7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49b810d3a0 0x7f49b819bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38164/0 (socket says 172.21.15.90:38164) 2024-04-25T11:41:32.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.267+0000 7f49b7fff640 1 -- 172.21.15.90:0/2593425790 learned_addr learned my addr 172.21.15.90:0/2593425790 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:32.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.267+0000 7f49b6ffd640 1 --2- 172.21.15.90:0/2593425790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f49b8108a40 0x7f49b819b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.268+0000 7f49b77fe640 1 --2- 172.21.15.90:0/2593425790 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49b8102b10 0x7f49b819b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.268+0000 7f49b6ffd640 1 -- 172.21.15.90:0/2593425790 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49b8102b10 msgr2=0x7f49b819b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.268+0000 7f49b6ffd640 1 --2- 172.21.15.90:0/2593425790 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49b8102b10 0x7f49b819b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.268+0000 7f49b6ffd640 1 -- 172.21.15.90:0/2593425790 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49b810d3a0 msgr2=0x7f49b819bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.268+0000 7f49b6ffd640 1 --2- 172.21.15.90:0/2593425790 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49b810d3a0 0x7f49b819bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.268+0000 7f49b6ffd640 1 -- 172.21.15.90:0/2593425790 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49b81a67e0 con 0x7f49b8108a40 2024-04-25T11:41:32.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.268+0000 7f49b7fff640 1 --2- 172.21.15.90:0/2593425790 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49b810d3a0 0x7f49b819bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:32.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.268+0000 7f49b6ffd640 1 --2- 172.21.15.90:0/2593425790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f49b8108a40 0x7f49b819b9e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f49b8069de0 tx=0x7f49ac00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:32.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.268+0000 7f49b4ff9640 1 -- 172.21.15.90:0/2593425790 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ac00cc40 con 0x7f49b8108a40 2024-04-25T11:41:32.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.268+0000 7f49b4ff9640 1 -- 172.21.15.90:0/2593425790 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49ac00cde0 con 0x7f49b8108a40 2024-04-25T11:41:32.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.268+0000 7f49bdc61640 1 -- 172.21.15.90:0/2593425790 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49b81a6ad0 con 0x7f49b8108a40 2024-04-25T11:41:32.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.268+0000 7f49b4ff9640 1 -- 172.21.15.90:0/2593425790 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ac010430 con 0x7f49b8108a40 2024-04-25T11:41:32.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.268+0000 7f49bdc61640 1 -- 172.21.15.90:0/2593425790 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49b81a7020 con 0x7f49b8108a40 2024-04-25T11:41:32.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.271+0000 7f49b4ff9640 1 -- 172.21.15.90:0/2593425790 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f49ac003ed0 con 0x7f49b8108a40 2024-04-25T11:41:32.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.271+0000 7f49b4ff9640 1 --2- 172.21.15.90:0/2593425790 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f49a00afc60 0x7f49a00b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.271+0000 7f49c4bef740 1 Processor -- start 2024-04-25T11:41:32.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.272+0000 7f49c4bef740 1 -- start start 2024-04-25T11:41:32.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.272+0000 7f49b4ff9640 1 -- 172.21.15.90:0/2593425790 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f49ac0df490 con 0x7f49b8108a40 2024-04-25T11:41:32.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.272+0000 7f49b77fe640 1 --2- 172.21.15.90:0/2593425790 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f49a00afc60 0x7f49a00b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.272+0000 7f49c4bef740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb202cd390 0x55eb2066e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.272+0000 7f49c4bef740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202a44d0 0x55eb2066ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.272+0000 7f49c4bef740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb202ac6b0 0x55eb2067b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.272+0000 7f49c4bef740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55eb205d3d80 con 0x55eb202a44d0 2024-04-25T11:41:32.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.272+0000 7f49c4bef740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55eb2058da20 con 0x55eb202cd390 2024-04-25T11:41:32.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.272+0000 7f49c4bef740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55eb205cfd40 con 0x55eb202ac6b0 2024-04-25T11:41:32.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.272+0000 7f49b77fe640 1 --2- 172.21.15.90:0/2593425790 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f49a00afc60 0x7f49a00b2170 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f49b8103a40 tx=0x7f49a8009760 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:32.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.272+0000 7f49b6ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202a44d0 0x55eb2066ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.272+0000 7f49b6ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202a44d0 0x55eb2066ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38176/0 (socket says 172.21.15.90:38176) 2024-04-25T11:41:32.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.272+0000 7f49b6ffd640 1 -- 172.21.15.90:0/2408925 learned_addr learned my addr 172.21.15.90:0/2408925 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:32.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.273+0000 7f49b7fff640 1 --2- 172.21.15.90:0/2408925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb202ac6b0 0x55eb2067b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.273+0000 7f49b77fe640 1 --2- 172.21.15.90:0/2408925 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb202cd390 0x55eb2066e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.273+0000 7f49b6ffd640 1 -- 172.21.15.90:0/2408925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb202ac6b0 msgr2=0x55eb2067b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.273+0000 7f49b6ffd640 1 --2- 172.21.15.90:0/2408925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb202ac6b0 0x55eb2067b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.273+0000 7f49b6ffd640 1 -- 172.21.15.90:0/2408925 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb202cd390 msgr2=0x55eb2066e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.273+0000 7f49b6ffd640 1 --2- 172.21.15.90:0/2408925 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb202cd390 0x55eb2066e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.273+0000 7f49b6ffd640 1 -- 172.21.15.90:0/2408925 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb20598c30 con 0x55eb202a44d0 2024-04-25T11:41:32.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.273+0000 7f49b6ffd640 1 --2- 172.21.15.90:0/2408925 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202a44d0 0x55eb2066ed50 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f49ac0a3650 tx=0x7f49ac0a70e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38a572a5097f56c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:32.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.273+0000 7f499dffb640 1 -- 172.21.15.90:0/2408925 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ac0900a0 con 0x55eb202a44d0 2024-04-25T11:41:32.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.273+0000 7f499dffb640 1 -- 172.21.15.90:0/2408925 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49ac0a75d0 con 0x55eb202a44d0 2024-04-25T11:41:32.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.273+0000 7f499dffb640 1 -- 172.21.15.90:0/2408925 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ac0b01a0 con 0x55eb202a44d0 2024-04-25T11:41:32.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.274+0000 7f49c4bef740 1 -- 172.21.15.90:0/2408925 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202a44d0 msgr2=0x55eb2066ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.274+0000 7f49c4bef740 1 --2- 172.21.15.90:0/2408925 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202a44d0 0x55eb2066ed50 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f49ac0a3650 tx=0x7f49ac0a70e0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.274+0000 7f49c4bef740 1 -- 172.21.15.90:0/2408925 shutdown_connections 2024-04-25T11:41:32.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.274+0000 7f49c4bef740 1 --2- 172.21.15.90:0/2408925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb202ac6b0 0x55eb2067b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.274+0000 7f49c4bef740 1 --2- 172.21.15.90:0/2408925 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202a44d0 0x55eb2066ed50 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.274+0000 7f49c4bef740 1 --2- 172.21.15.90:0/2408925 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb202cd390 0x55eb2066e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.274+0000 7f49c4bef740 1 -- 172.21.15.90:0/2408925 >> 172.21.15.90:0/2408925 conn(0x55eb202c14e0 msgr2=0x55eb2066afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:32.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.274+0000 7f49c4bef740 1 -- 172.21.15.90:0/2408925 shutdown_connections 2024-04-25T11:41:32.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.274+0000 7f49c4bef740 1 -- 172.21.15.90:0/2408925 wait complete. 2024-04-25T11:41:32.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.275+0000 7f49c4bef740 1 -- start start 2024-04-25T11:41:32.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.275+0000 7f49c4bef740 1 Processor -- start 2024-04-25T11:41:32.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.275+0000 7f49c4bef740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202a44d0 0x55eb20679300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.275+0000 7f49c4bef740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb202ac6b0 0x55eb20679840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.275+0000 7f49c4bef740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb202cd390 0x55eb20685210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.275+0000 7f49c4bef740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55eb2067bd90 con 0x55eb202a44d0 2024-04-25T11:41:32.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.275+0000 7f49c4bef740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55eb2055adc0 con 0x55eb202ac6b0 2024-04-25T11:41:32.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.275+0000 7f49c4bef740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55eb2067bf10 con 0x55eb202cd390 2024-04-25T11:41:32.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.275+0000 7f49c4bef740 10 client.0 _ll_register_callbacks cb 0x55eb20576600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:32.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.275+0000 7f49b77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202a44d0 0x55eb20679300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.275+0000 7f49b77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202a44d0 0x55eb20679300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38182/0 (socket says 172.21.15.90:38182) 2024-04-25T11:41:32.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.275+0000 7f49b77fe640 1 -- 172.21.15.90:0/4283618814 learned_addr learned my addr 172.21.15.90:0/4283618814 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:32.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.275+0000 7f49b6ffd640 1 --2- 172.21.15.90:0/4283618814 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb202ac6b0 0x55eb20679840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.275+0000 7f49b7fff640 1 --2- 172.21.15.90:0/4283618814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb202cd390 0x55eb20685210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.276+0000 7f49b77fe640 1 -- 172.21.15.90:0/4283618814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb202cd390 msgr2=0x55eb20685210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.276+0000 7f49b77fe640 1 --2- 172.21.15.90:0/4283618814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb202cd390 0x55eb20685210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.276+0000 7f49b77fe640 1 -- 172.21.15.90:0/4283618814 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb202ac6b0 msgr2=0x55eb20679840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.276+0000 7f49b77fe640 1 --2- 172.21.15.90:0/4283618814 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb202ac6b0 0x55eb20679840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.276+0000 7f49b77fe640 1 -- 172.21.15.90:0/4283618814 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb202e53f0 con 0x55eb202a44d0 2024-04-25T11:41:32.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.276+0000 7f49b6ffd640 1 --2- 172.21.15.90:0/4283618814 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb202ac6b0 0x55eb20679840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:32.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.276+0000 7f49b77fe640 1 --2- 172.21.15.90:0/4283618814 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202a44d0 0x55eb20679300 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f49a8011960 tx=0x7f49a8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e0293f6fe77916d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:32.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.276+0000 7f4996ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:41:32.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.276+0000 7f4996ffd640 1 -- 172.21.15.90:0/4283618814 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a801dd40 con 0x55eb202a44d0 2024-04-25T11:41:32.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.276+0000 7f4996ffd640 1 -- 172.21.15.90:0/4283618814 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f49a8003970 con 0x55eb202a44d0 2024-04-25T11:41:32.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.276+0000 7f49c4bef740 1 -- 172.21.15.90:0/4283618814 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49ac0a75d0 con 0x55eb202a44d0 2024-04-25T11:41:32.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.276+0000 7f4996ffd640 1 -- 172.21.15.90:0/4283618814 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f49a8003b70 con 0x55eb202a44d0 2024-04-25T11:41:32.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.276+0000 7f49c4bef740 10 client.17514 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:32.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.276+0000 7f49c4bef740 1 -- 172.21.15.90:0/4283618814 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55eb203af7d0 con 0x55eb202a44d0 2024-04-25T11:41:32.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.276+0000 7f4996ffd640 1 -- 172.21.15.90:0/4283618814 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49ac0a75d0 con 0x55eb202a44d0 2024-04-25T11:41:32.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.277+0000 7f4996ffd640 1 -- 172.21.15.90:0/4283618814 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f49a801e420 con 0x55eb202a44d0 2024-04-25T11:41:32.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.277+0000 7f49c4bef740 10 client.17514 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:32.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.277+0000 7f49c4bef740 10 client.17514 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:41:32.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.277+0000 7f49c4bef740 20 client.17514 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:32.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.277+0000 7f49c4bef740 1 --2- 172.21.15.90:0/4283618814 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55eb2028bf60 0x55eb2068b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.277+0000 7f49c4bef740 4 client.17514 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:41:32.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.277+0000 7f49c4bef740 1 -- 172.21.15.90:0/4283618814 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb205e1990 con 0x55eb2028bf60 2024-04-25T11:41:32.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.277+0000 7f49b6ffd640 1 --2- 172.21.15.90:0/4283618814 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55eb2028bf60 0x55eb2068b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.277+0000 7f4996ffd640 10 client.17514 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:41:32.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.277+0000 7f49b6ffd640 1 --2- 172.21.15.90:0/4283618814 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55eb2028bf60 0x55eb2068b810 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c413bdc847d58d6d server_cookie=fdf797081bded89a in_seq=0 out_seq=0 2024-04-25T11:41:32.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.278+0000 7f4996ffd640 1 -- 172.21.15.90:0/4283618814 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55eb205e1990 con 0x55eb2028bf60 2024-04-25T11:41:32.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.278+0000 7f4996ffd640 10 client.17514 handle_command_reply: tid=0 2024-04-25T11:41:32.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.278+0000 7f49c4bef740 1 client.17514 shutdown 2024-04-25T11:41:32.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.278+0000 7f49c4bef740 1 -- 172.21.15.90:0/4283618814 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202a44d0 msgr2=0x55eb20679300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.278+0000 7f49c4bef740 1 --2- 172.21.15.90:0/4283618814 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202a44d0 0x55eb20679300 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f49a8011960 tx=0x7f49a8004ab0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.278+0000 7f49c4bef740 1 -- 172.21.15.90:0/4283618814 shutdown_connections 2024-04-25T11:41:32.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.278+0000 7f49c4bef740 1 --2- 172.21.15.90:0/4283618814 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55eb2028bf60 0x55eb2068b810 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.278+0000 7f49c4bef740 1 --2- 172.21.15.90:0/4283618814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb202cd390 0x55eb20685210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.278+0000 7f49c4bef740 1 --2- 172.21.15.90:0/4283618814 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb202ac6b0 0x55eb20679840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.278+0000 7f4996ffd640 0 client.17514 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:41:32.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.278+0000 7f49c4bef740 1 --2- 172.21.15.90:0/4283618814 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202a44d0 0x55eb20679300 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.278+0000 7f49c4bef740 1 -- 172.21.15.90:0/4283618814 >> 172.21.15.90:0/4283618814 conn(0x55eb202c14e0 msgr2=0x55eb20670530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:32.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.278+0000 7f49c4bef740 1 -- 172.21.15.90:0/4283618814 shutdown_connections 2024-04-25T11:41:32.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.278+0000 7f49c4bef740 1 -- 172.21.15.90:0/4283618814 wait complete. 2024-04-25T11:41:32.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.278+0000 7f49c4bef740 20 client.17514 trim_cache size 0 max 16384 2024-04-25T11:41:32.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.292+0000 7f49c4bef740 1 Processor -- start 2024-04-25T11:41:32.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.292+0000 7f49c4bef740 1 -- start start 2024-04-25T11:41:32.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.293+0000 7f49c4bef740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb202a44d0 0x55eb2066afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.293+0000 7f49c4bef740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202ac6b0 0x55eb20684e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.293+0000 7f49c4bef740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb202cd390 0x55eb20690fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.293+0000 7f49c4bef740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f49b81afc80 con 0x55eb202ac6b0 2024-04-25T11:41:32.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.293+0000 7f49c4bef740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f49b81b17c0 con 0x55eb202cd390 2024-04-25T11:41:32.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.293+0000 7f49c4bef740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f49b81b0350 con 0x55eb202a44d0 2024-04-25T11:41:32.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.293+0000 7f49b6ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202ac6b0 0x55eb20684e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.293+0000 7f49b6ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202ac6b0 0x55eb20684e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38188/0 (socket says 172.21.15.90:38188) 2024-04-25T11:41:32.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.293+0000 7f49b6ffd640 1 -- 172.21.15.90:0/2294228796 learned_addr learned my addr 172.21.15.90:0/2294228796 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:32.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.293+0000 7f49b7fff640 1 --2- 172.21.15.90:0/2294228796 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb202cd390 0x55eb20690fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.293+0000 7f49b77fe640 1 --2- 172.21.15.90:0/2294228796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb202a44d0 0x55eb2066afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.293+0000 7f49b6ffd640 1 -- 172.21.15.90:0/2294228796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb202a44d0 msgr2=0x55eb2066afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.293+0000 7f49b6ffd640 1 --2- 172.21.15.90:0/2294228796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb202a44d0 0x55eb2066afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.293+0000 7f49b6ffd640 1 -- 172.21.15.90:0/2294228796 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb202cd390 msgr2=0x55eb20690fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.293+0000 7f49b6ffd640 1 --2- 172.21.15.90:0/2294228796 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb202cd390 0x55eb20690fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.293+0000 7f49b6ffd640 1 -- 172.21.15.90:0/2294228796 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb206783f0 con 0x55eb202ac6b0 2024-04-25T11:41:32.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.293+0000 7f49b77fe640 1 --2- 172.21.15.90:0/2294228796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb202a44d0 0x55eb2066afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:41:32.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.293+0000 7f49b7fff640 1 --2- 172.21.15.90:0/2294228796 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb202cd390 0x55eb20690fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:41:32.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.293+0000 7f49b6ffd640 1 --2- 172.21.15.90:0/2294228796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202ac6b0 0x55eb20684e70 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f49ac0dcc30 tx=0x7f49ac0a3510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88c9abb9bd2de7f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:32.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.294+0000 7f499d7fa640 1 -- 172.21.15.90:0/2294228796 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ac0a75d0 con 0x55eb202ac6b0 2024-04-25T11:41:32.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.294+0000 7f499d7fa640 1 -- 172.21.15.90:0/2294228796 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49ac0a3860 con 0x55eb202ac6b0 2024-04-25T11:41:32.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.294+0000 7f499d7fa640 1 -- 172.21.15.90:0/2294228796 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ac0a2f40 con 0x55eb202ac6b0 2024-04-25T11:41:32.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.295+0000 7f49c4bef740 1 -- 172.21.15.90:0/2294228796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202ac6b0 msgr2=0x55eb20684e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.295+0000 7f49c4bef740 1 --2- 172.21.15.90:0/2294228796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202ac6b0 0x55eb20684e70 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f49ac0dcc30 tx=0x7f49ac0a3510 comp rx=0 tx=0).stop 2024-04-25T11:41:32.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.295+0000 7f49c4bef740 1 -- 172.21.15.90:0/2294228796 shutdown_connections 2024-04-25T11:41:32.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.295+0000 7f49c4bef740 1 --2- 172.21.15.90:0/2294228796 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb202cd390 0x55eb20690fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.295+0000 7f49c4bef740 1 --2- 172.21.15.90:0/2294228796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202ac6b0 0x55eb20684e70 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.295+0000 7f49c4bef740 1 --2- 172.21.15.90:0/2294228796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb202a44d0 0x55eb2066afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.295+0000 7f49c4bef740 1 -- 172.21.15.90:0/2294228796 >> 172.21.15.90:0/2294228796 conn(0x55eb202c14e0 msgr2=0x55eb2066a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:32.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.295+0000 7f49c4bef740 1 -- 172.21.15.90:0/2294228796 shutdown_connections 2024-04-25T11:41:32.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.295+0000 7f49c4bef740 1 -- 172.21.15.90:0/2294228796 wait complete. 2024-04-25T11:41:32.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.295+0000 7f49c4bef740 1 -- start start 2024-04-25T11:41:32.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.295+0000 7f49c4bef740 1 Processor -- start 2024-04-25T11:41:32.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.296+0000 7f49c4bef740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb202a44d0 0x55eb2067c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.296+0000 7f49c4bef740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202ac6b0 0x55eb2067c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.296+0000 7f49c4bef740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb202cd390 0x55eb206f1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.296+0000 7f49c4bef740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f49b81b10f0 con 0x55eb202ac6b0 2024-04-25T11:41:32.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.296+0000 7f49c4bef740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f49b81b0a20 con 0x55eb202a44d0 2024-04-25T11:41:32.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.296+0000 7f49c4bef740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f49b81adc60 con 0x55eb202cd390 2024-04-25T11:41:32.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.296+0000 7f49c4bef740 10 client.0 _ll_register_callbacks cb 0x7f49a8009bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:32.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.296+0000 7f49b6ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202ac6b0 0x55eb2067c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.296+0000 7f49b7fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb202cd390 0x55eb206f1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.296+0000 7f49b77fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb202a44d0 0x55eb2067c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.296+0000 7f49b6ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202ac6b0 0x55eb2067c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38190/0 (socket says 172.21.15.90:38190) 2024-04-25T11:41:32.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.296+0000 7f49b6ffd640 1 -- 172.21.15.90:0/647393103 learned_addr learned my addr 172.21.15.90:0/647393103 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:32.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.296+0000 7f49b7fff640 1 -- 172.21.15.90:0/647393103 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb202a44d0 msgr2=0x55eb2067c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.296+0000 7f49b7fff640 1 --2- 172.21.15.90:0/647393103 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb202a44d0 0x55eb2067c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.296+0000 7f49b7fff640 1 -- 172.21.15.90:0/647393103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202ac6b0 msgr2=0x55eb2067c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.296+0000 7f49b7fff640 1 --2- 172.21.15.90:0/647393103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202ac6b0 0x55eb2067c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.296+0000 7f49b7fff640 1 -- 172.21.15.90:0/647393103 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb2067cfe0 con 0x55eb202cd390 2024-04-25T11:41:32.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.296+0000 7f49b6ffd640 1 --2- 172.21.15.90:0/647393103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202ac6b0 0x55eb2067c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:32.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.296+0000 7f49b77fe640 1 --2- 172.21.15.90:0/647393103 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb202a44d0 0x55eb2067c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:32.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.297+0000 7f49b7fff640 1 --2- 172.21.15.90:0/647393103 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb202cd390 0x55eb206f1b70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f49b819c480 tx=0x7f49b0031a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0d1442d126474c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:32.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.297+0000 7f4996ffd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:41:32.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.297+0000 7f4996ffd640 1 -- 172.21.15.90:0/647393103 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b004c020 con 0x55eb202cd390 2024-04-25T11:41:32.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.297+0000 7f4996ffd640 1 -- 172.21.15.90:0/647393103 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f49b0005020 con 0x55eb202cd390 2024-04-25T11:41:32.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.297+0000 7f49c4bef740 1 -- 172.21.15.90:0/647393103 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49ac0a3860 con 0x55eb202cd390 2024-04-25T11:41:32.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.297+0000 7f4996ffd640 1 -- 172.21.15.90:0/647393103 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f49b0054050 con 0x55eb202cd390 2024-04-25T11:41:32.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.297+0000 7f49c4bef740 10 client.26443 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:32.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.297+0000 7f49c4bef740 1 -- 172.21.15.90:0/647393103 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55eb206f3160 con 0x55eb202cd390 2024-04-25T11:41:32.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.297+0000 7f4996ffd640 1 -- 172.21.15.90:0/647393103 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49ac0a3860 con 0x55eb202cd390 2024-04-25T11:41:32.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.297+0000 7f4996ffd640 1 -- 172.21.15.90:0/647393103 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f49b0047070 con 0x55eb202cd390 2024-04-25T11:41:32.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.297+0000 7f49c4bef740 10 client.26443 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:32.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.297+0000 7f49c4bef740 10 client.26443 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:41:32.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.297+0000 7f49c4bef740 20 client.26443 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:32.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.297+0000 7f49c4bef740 1 --2- 172.21.15.90:0/647393103 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55eb202613a0 0x55eb206f5910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.297+0000 7f49c4bef740 4 client.26443 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:41:32.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.297+0000 7f49c4bef740 1 -- 172.21.15.90:0/647393103 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55eb205db800 con 0x55eb202613a0 2024-04-25T11:41:32.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.298+0000 7f49b77fe640 1 --2- 172.21.15.90:0/647393103 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55eb202613a0 0x55eb206f5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.298+0000 7f4996ffd640 10 client.26443 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:41:32.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.298+0000 7f49b77fe640 1 --2- 172.21.15.90:0/647393103 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55eb202613a0 0x55eb206f5910 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f37941e5edc98a5d server_cookie=ddb1e4371d37d270 in_seq=0 out_seq=0 2024-04-25T11:41:32.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.298+0000 7f4996ffd640 1 -- 172.21.15.90:0/647393103 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55eb205db800 con 0x55eb202613a0 2024-04-25T11:41:32.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.298+0000 7f4996ffd640 10 client.26443 handle_command_reply: tid=0 2024-04-25T11:41:32.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.298+0000 7f49c4bef740 1 client.26443 shutdown 2024-04-25T11:41:32.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.299+0000 7f49c4bef740 1 -- 172.21.15.90:0/647393103 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb202cd390 msgr2=0x55eb206f1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.299+0000 7f49c4bef740 1 --2- 172.21.15.90:0/647393103 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb202cd390 0x55eb206f1b70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f49b819c480 tx=0x7f49b0031a20 comp rx=0 tx=0).stop 2024-04-25T11:41:32.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.299+0000 7f49c4bef740 1 -- 172.21.15.90:0/647393103 shutdown_connections 2024-04-25T11:41:32.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.299+0000 7f49c4bef740 1 --2- 172.21.15.90:0/647393103 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55eb202613a0 0x55eb206f5910 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.299+0000 7f49c4bef740 1 --2- 172.21.15.90:0/647393103 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb202cd390 0x55eb206f1b70 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.299+0000 7f49c4bef740 1 --2- 172.21.15.90:0/647393103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb202ac6b0 0x55eb2067c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.299+0000 7f49c4bef740 1 --2- 172.21.15.90:0/647393103 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb202a44d0 0x55eb2067c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.299+0000 7f4996ffd640 0 client.26443 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:41:32.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.299+0000 7f49c4bef740 1 -- 172.21.15.90:0/647393103 >> 172.21.15.90:0/647393103 conn(0x55eb202c14e0 msgr2=0x55eb20670410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:32.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.299+0000 7f49c4bef740 1 -- 172.21.15.90:0/647393103 shutdown_connections 2024-04-25T11:41:32.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.299+0000 7f49c4bef740 1 -- 172.21.15.90:0/647393103 wait complete. 2024-04-25T11:41:32.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.299+0000 7f49c4bef740 20 client.26443 trim_cache size 0 max 16384 2024-04-25T11:41:32.293 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:41:32.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.299+0000 7f49bdc61640 1 -- 172.21.15.90:0/2593425790 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f49a00afc60 msgr2=0x7f49a00b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.299+0000 7f49bdc61640 1 --2- 172.21.15.90:0/2593425790 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f49a00afc60 0x7f49a00b2170 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f49b8103a40 tx=0x7f49a8009760 comp rx=0 tx=0).stop 2024-04-25T11:41:32.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.299+0000 7f49bdc61640 1 -- 172.21.15.90:0/2593425790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f49b8108a40 msgr2=0x7f49b819b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.299+0000 7f49bdc61640 1 --2- 172.21.15.90:0/2593425790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f49b8108a40 0x7f49b819b9e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f49b8069de0 tx=0x7f49ac00ee50 comp rx=0 tx=0).stop 2024-04-25T11:41:32.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.299+0000 7f49bdc61640 1 -- 172.21.15.90:0/2593425790 shutdown_connections 2024-04-25T11:41:32.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.299+0000 7f49bdc61640 1 --2- 172.21.15.90:0/2593425790 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f49a00afc60 0x7f49a00b2170 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.299+0000 7f49bdc61640 1 --2- 172.21.15.90:0/2593425790 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49b810d3a0 0x7f49b819bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.299+0000 7f49bdc61640 1 --2- 172.21.15.90:0/2593425790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f49b8108a40 0x7f49b819b9e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.299+0000 7f49bdc61640 1 --2- 172.21.15.90:0/2593425790 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49b8102b10 0x7f49b819b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.299+0000 7f49bdc61640 1 -- 172.21.15.90:0/2593425790 >> 172.21.15.90:0/2593425790 conn(0x7f49b8078180 msgr2=0x7f49b81040d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:32.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.300+0000 7f49bdc61640 1 -- 172.21.15.90:0/2593425790 shutdown_connections 2024-04-25T11:41:32.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.300+0000 7f49bdc61640 1 -- 172.21.15.90:0/2593425790 wait complete. 2024-04-25T11:41:32.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T11:41:32.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.436+0000 7f835390d640 1 -- 172.21.15.90:0/2801838213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f834c106990 msgr2=0x7f834c106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.436+0000 7f835390d640 1 --2- 172.21.15.90:0/2801838213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f834c106990 0x7f834c106dc0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f833c009900 tx=0x7f833c02a410 comp rx=0 tx=0).stop 2024-04-25T11:41:32.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.436+0000 7f835390d640 1 -- 172.21.15.90:0/2801838213 shutdown_connections 2024-04-25T11:41:32.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.436+0000 7f835390d640 1 --2- 172.21.15.90:0/2801838213 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f834c107450 0x7f834c10ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.436+0000 7f835390d640 1 --2- 172.21.15.90:0/2801838213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f834c106990 0x7f834c106dc0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.436+0000 7f835390d640 1 --2- 172.21.15.90:0/2801838213 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f834c102400 0x7f834c102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.436+0000 7f835390d640 1 -- 172.21.15.90:0/2801838213 >> 172.21.15.90:0/2801838213 conn(0x7f834c077880 msgr2=0x7f834c077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:32.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.436+0000 7f835390d640 1 -- 172.21.15.90:0/2801838213 shutdown_connections 2024-04-25T11:41:32.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.436+0000 7f835390d640 1 -- 172.21.15.90:0/2801838213 wait complete. 2024-04-25T11:41:32.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.436+0000 7f835390d640 1 Processor -- start 2024-04-25T11:41:32.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.436+0000 7f835390d640 1 -- start start 2024-04-25T11:41:32.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.437+0000 7f835390d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f834c102400 0x7f834c19a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.437+0000 7f835390d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f834c106990 0x7f834c19ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.437+0000 7f835390d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f834c107450 0x7f834c19efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.437+0000 7f835390d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f834c111ae0 con 0x7f834c107450 2024-04-25T11:41:32.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.437+0000 7f835390d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f834c111960 con 0x7f834c102400 2024-04-25T11:41:32.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.437+0000 7f835390d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f834c111c60 con 0x7f834c106990 2024-04-25T11:41:32.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.437+0000 7f8351eaa640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f834c107450 0x7f834c19efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.437+0000 7f8351eaa640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f834c107450 0x7f834c19efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38200/0 (socket says 172.21.15.90:38200) 2024-04-25T11:41:32.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.437+0000 7f8351eaa640 1 -- 172.21.15.90:0/201319347 learned_addr learned my addr 172.21.15.90:0/201319347 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:32.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.437+0000 7f83516a9640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f834c102400 0x7f834c19a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.437+0000 7f8350ea8640 1 --2- 172.21.15.90:0/201319347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f834c106990 0x7f834c19ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.437+0000 7f8351eaa640 1 -- 172.21.15.90:0/201319347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f834c106990 msgr2=0x7f834c19ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.437+0000 7f8351eaa640 1 --2- 172.21.15.90:0/201319347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f834c106990 0x7f834c19ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.437+0000 7f8351eaa640 1 -- 172.21.15.90:0/201319347 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f834c102400 msgr2=0x7f834c19a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.437+0000 7f8351eaa640 1 --2- 172.21.15.90:0/201319347 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f834c102400 0x7f834c19a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.437+0000 7f8351eaa640 1 -- 172.21.15.90:0/201319347 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f834c19f600 con 0x7f834c107450 2024-04-25T11:41:32.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.437+0000 7f8350ea8640 1 --2- 172.21.15.90:0/201319347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f834c106990 0x7f834c19ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:41:32.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.437+0000 7f83516a9640 1 --2- 172.21.15.90:0/201319347 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f834c102400 0x7f834c19a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:32.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.437+0000 7f8351eaa640 1 --2- 172.21.15.90:0/201319347 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f834c107450 0x7f834c19efc0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f834400ec60 tx=0x7f834400c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:32.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.437+0000 7f83427fc640 1 -- 172.21.15.90:0/201319347 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8344004d20 con 0x7f834c107450 2024-04-25T11:41:32.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.438+0000 7f835390d640 1 -- 172.21.15.90:0/201319347 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f834c19f8f0 con 0x7f834c107450 2024-04-25T11:41:32.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.438+0000 7f83427fc640 1 -- 172.21.15.90:0/201319347 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f834400ba20 con 0x7f834c107450 2024-04-25T11:41:32.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.438+0000 7f83427fc640 1 -- 172.21.15.90:0/201319347 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83440056c0 con 0x7f834c107450 2024-04-25T11:41:32.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.438+0000 7f835390d640 1 -- 172.21.15.90:0/201319347 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f834c1a7270 con 0x7f834c107450 2024-04-25T11:41:32.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.440+0000 7f835a7be740 1 Processor -- start 2024-04-25T11:41:32.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.440+0000 7f83427fc640 1 -- 172.21.15.90:0/201319347 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f8344005860 con 0x7f834c107450 2024-04-25T11:41:32.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.440+0000 7f835a7be740 1 -- start start 2024-04-25T11:41:32.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.440+0000 7f835a7be740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608772f0390 0x560877691810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.440+0000 7f835a7be740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772c74d0 0x560877691d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.440+0000 7f835a7be740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5608772cf6b0 0x56087769e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.440+0000 7f835a7be740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5608775f6d80 con 0x5608772c74d0 2024-04-25T11:41:32.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.441+0000 7f835a7be740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5608775b0a20 con 0x5608772cf6b0 2024-04-25T11:41:32.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.441+0000 7f835a7be740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5608775f2d40 con 0x5608772f0390 2024-04-25T11:41:32.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.441+0000 7f83516a9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772c74d0 0x560877691d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.441+0000 7f8350ea8640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5608772cf6b0 0x56087769e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.441+0000 7f8350ea8640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5608772cf6b0 0x56087769e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:58284/0 (socket says 172.21.15.90:58284) 2024-04-25T11:41:32.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.441+0000 7f8350ea8640 1 -- 172.21.15.90:0/815600818 learned_addr learned my addr 172.21.15.90:0/815600818 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.441+0000 7f83516a9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772c74d0 0x560877691d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38204/0 (socket says 172.21.15.90:38204) 2024-04-25T11:41:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.441+0000 7f8350ea8640 1 --2- 172.21.15.90:0/815600818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608772f0390 0x560877691810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.441+0000 7f83427fc640 1 --2- 172.21.15.90:0/201319347 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f83380afc60 0x7f83380b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.441+0000 7f8350ea8640 1 -- 172.21.15.90:0/815600818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608772f0390 msgr2=0x560877691810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.441+0000 7f8350ea8640 1 --2- 172.21.15.90:0/815600818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608772f0390 0x560877691810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.441+0000 7f8350ea8640 1 -- 172.21.15.90:0/815600818 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772c74d0 msgr2=0x560877691d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.441+0000 7f8350ea8640 1 --2- 172.21.15.90:0/815600818 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772c74d0 0x560877691d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.441+0000 7f8350ea8640 1 -- 172.21.15.90:0/815600818 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608775bbc30 con 0x5608772cf6b0 2024-04-25T11:41:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.441+0000 7f8351eaa640 1 --2- 172.21.15.90:0/201319347 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f83380afc60 0x7f83380b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.441+0000 7f83427fc640 1 -- 172.21.15.90:0/201319347 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f83440df590 con 0x7f834c107450 2024-04-25T11:41:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.441+0000 7f8350ea8640 1 --2- 172.21.15.90:0/815600818 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5608772cf6b0 0x56087769e6a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f833c032fd0 tx=0x7f833c0058b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a17750f8bb027fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.442+0000 7f83377fe640 1 -- 172.21.15.90:0/815600818 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c035040 con 0x5608772cf6b0 2024-04-25T11:41:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.442+0000 7f83377fe640 1 -- 172.21.15.90:0/815600818 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f833c005df0 con 0x5608772cf6b0 2024-04-25T11:41:32.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.442+0000 7f83377fe640 1 -- 172.21.15.90:0/815600818 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c0039b0 con 0x5608772cf6b0 2024-04-25T11:41:32.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.442+0000 7f835a7be740 1 -- 172.21.15.90:0/815600818 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5608772cf6b0 msgr2=0x56087769e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.442+0000 7f835a7be740 1 --2- 172.21.15.90:0/815600818 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5608772cf6b0 0x56087769e6a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f833c032fd0 tx=0x7f833c0058b0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.442+0000 7f8351eaa640 1 --2- 172.21.15.90:0/201319347 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f83380afc60 0x7f83380b2170 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f83440a73d0 tx=0x7f83440a3610 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:32.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.443+0000 7f835a7be740 1 -- 172.21.15.90:0/815600818 shutdown_connections 2024-04-25T11:41:32.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.443+0000 7f835a7be740 1 --2- 172.21.15.90:0/815600818 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5608772cf6b0 0x56087769e6a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.443+0000 7f835a7be740 1 --2- 172.21.15.90:0/815600818 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772c74d0 0x560877691d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.443+0000 7f835a7be740 1 --2- 172.21.15.90:0/815600818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608772f0390 0x560877691810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.443+0000 7f835a7be740 1 -- 172.21.15.90:0/815600818 >> 172.21.15.90:0/815600818 conn(0x5608772e44e0 msgr2=0x56087768dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:32.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.443+0000 7f835a7be740 1 -- 172.21.15.90:0/815600818 shutdown_connections 2024-04-25T11:41:32.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.443+0000 7f835a7be740 1 -- 172.21.15.90:0/815600818 wait complete. 2024-04-25T11:41:32.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.443+0000 7f835a7be740 1 -- start start 2024-04-25T11:41:32.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.443+0000 7f835a7be740 1 Processor -- start 2024-04-25T11:41:32.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.444+0000 7f835a7be740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608772c74d0 0x56087769c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.444+0000 7f835a7be740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5608772cf6b0 0x56087769c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.444+0000 7f835a7be740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772f0390 0x5608776a81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.444+0000 7f835a7be740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56087769ed30 con 0x5608772f0390 2024-04-25T11:41:32.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.444+0000 7f835a7be740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56087757ddc0 con 0x5608772cf6b0 2024-04-25T11:41:32.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.444+0000 7f835a7be740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56087769eeb0 con 0x5608772c74d0 2024-04-25T11:41:32.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.444+0000 7f835a7be740 10 client.0 _ll_register_callbacks cb 0x560877599600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:32.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.444+0000 7f8350ea8640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772f0390 0x5608776a81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.444+0000 7f8350ea8640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772f0390 0x5608776a81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38220/0 (socket says 172.21.15.90:38220) 2024-04-25T11:41:32.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.444+0000 7f8350ea8640 1 -- 172.21.15.90:0/3292828648 learned_addr learned my addr 172.21.15.90:0/3292828648 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:32.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.444+0000 7f83516a9640 1 --2- 172.21.15.90:0/3292828648 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5608772cf6b0 0x56087769c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.444+0000 7f8350ea8640 1 --2- 172.21.15.90:0/3292828648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608772c74d0 0x56087769c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.444+0000 7f83516a9640 1 -- 172.21.15.90:0/3292828648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608772c74d0 msgr2=0x56087769c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.444+0000 7f83516a9640 1 --2- 172.21.15.90:0/3292828648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608772c74d0 0x56087769c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.444+0000 7f83516a9640 1 -- 172.21.15.90:0/3292828648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772f0390 msgr2=0x5608776a81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.444+0000 7f83516a9640 1 --2- 172.21.15.90:0/3292828648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772f0390 0x5608776a81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.444+0000 7f83516a9640 1 -- 172.21.15.90:0/3292828648 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608773083f0 con 0x5608772cf6b0 2024-04-25T11:41:32.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.444+0000 7f8350ea8640 1 --2- 172.21.15.90:0/3292828648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772f0390 0x5608776a81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:32.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.444+0000 7f83516a9640 1 --2- 172.21.15.90:0/3292828648 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5608772cf6b0 0x56087769c7a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x56087768ef50 tx=0x7f834800f420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d7bda0cc663883 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:32.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.445+0000 7f8334ff9640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:41:32.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.445+0000 7f8334ff9640 1 -- 172.21.15.90:0/3292828648 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834800fe70 con 0x5608772cf6b0 2024-04-25T11:41:32.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.445+0000 7f8334ff9640 1 -- 172.21.15.90:0/3292828648 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f8348005020 con 0x5608772cf6b0 2024-04-25T11:41:32.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.445+0000 7f835a7be740 1 -- 172.21.15.90:0/3292828648 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f833c005df0 con 0x5608772cf6b0 2024-04-25T11:41:32.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.445+0000 7f8334ff9640 1 -- 172.21.15.90:0/3292828648 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8348005220 con 0x5608772cf6b0 2024-04-25T11:41:32.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.445+0000 7f835a7be740 10 client.26510 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:32.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.445+0000 7f835a7be740 1 -- 172.21.15.90:0/3292828648 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5608773d27d0 con 0x5608772cf6b0 2024-04-25T11:41:32.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.445+0000 7f8334ff9640 1 -- 172.21.15.90:0/3292828648 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f833c005df0 con 0x5608772cf6b0 2024-04-25T11:41:32.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.445+0000 7f8334ff9640 1 -- 172.21.15.90:0/3292828648 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f8348021600 con 0x5608772cf6b0 2024-04-25T11:41:32.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.445+0000 7f835a7be740 10 client.26510 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:32.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.445+0000 7f835a7be740 10 client.26510 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:41:32.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.445+0000 7f835a7be740 20 client.26510 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:32.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.445+0000 7f835a7be740 1 --2- 172.21.15.90:0/3292828648 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5608772aef60 0x5608776ae740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.445+0000 7f835a7be740 4 client.26510 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:41:32.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.445+0000 7f835a7be740 1 -- 172.21.15.90:0/3292828648 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560877604990 con 0x5608772aef60 2024-04-25T11:41:32.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.446+0000 7f8350ea8640 1 --2- 172.21.15.90:0/3292828648 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5608772aef60 0x5608776ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.446+0000 7f8334ff9640 10 client.26510 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:41:32.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.446+0000 7f8350ea8640 1 --2- 172.21.15.90:0/3292828648 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5608772aef60 0x5608776ae740 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=60714685d7d65e42 server_cookie=2392b868dcc76f7e in_seq=0 out_seq=0 2024-04-25T11:41:32.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.447+0000 7f8334ff9640 1 -- 172.21.15.90:0/3292828648 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560877604990 con 0x5608772aef60 2024-04-25T11:41:32.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.447+0000 7f8334ff9640 10 client.26510 handle_command_reply: tid=0 2024-04-25T11:41:32.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.447+0000 7f835a7be740 1 client.26510 shutdown 2024-04-25T11:41:32.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.447+0000 7f835a7be740 1 -- 172.21.15.90:0/3292828648 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5608772cf6b0 msgr2=0x56087769c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.447+0000 7f835a7be740 1 --2- 172.21.15.90:0/3292828648 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5608772cf6b0 0x56087769c7a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x56087768ef50 tx=0x7f834800f420 comp rx=0 tx=0).stop 2024-04-25T11:41:32.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.447+0000 7f835a7be740 1 -- 172.21.15.90:0/3292828648 shutdown_connections 2024-04-25T11:41:32.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.447+0000 7f835a7be740 1 --2- 172.21.15.90:0/3292828648 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5608772aef60 0x5608776ae740 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.447+0000 7f835a7be740 1 --2- 172.21.15.90:0/3292828648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772f0390 0x5608776a81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.447+0000 7f8334ff9640 0 client.26510 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:41:32.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.447+0000 7f835a7be740 1 --2- 172.21.15.90:0/3292828648 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5608772cf6b0 0x56087769c7a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.447+0000 7f835a7be740 1 --2- 172.21.15.90:0/3292828648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608772c74d0 0x56087769c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.447+0000 7f835a7be740 1 -- 172.21.15.90:0/3292828648 >> 172.21.15.90:0/3292828648 conn(0x5608772e44e0 msgr2=0x560877693530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:32.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.448+0000 7f835a7be740 1 -- 172.21.15.90:0/3292828648 shutdown_connections 2024-04-25T11:41:32.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.448+0000 7f835a7be740 1 -- 172.21.15.90:0/3292828648 wait complete. 2024-04-25T11:41:32.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.448+0000 7f835a7be740 20 client.26510 trim_cache size 0 max 16384 2024-04-25T11:41:32.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.462+0000 7f835a7be740 1 Processor -- start 2024-04-25T11:41:32.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.462+0000 7f835a7be740 1 -- start start 2024-04-25T11:41:32.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.462+0000 7f835a7be740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608772aef60 0x560877697850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.462+0000 7f835a7be740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772c74d0 0x560877691cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.462+0000 7f835a7be740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5608772cf6b0 0x5608776a7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.462+0000 7f835a7be740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f834c1afc10 con 0x5608772c74d0 2024-04-25T11:41:32.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.462+0000 7f835a7be740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f834c1b1750 con 0x5608772cf6b0 2024-04-25T11:41:32.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.462+0000 7f835a7be740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f834c1b02e0 con 0x5608772aef60 2024-04-25T11:41:32.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.462+0000 7f83516a9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772c74d0 0x560877691cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.462+0000 7f83516a9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772c74d0 0x560877691cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38230/0 (socket says 172.21.15.90:38230) 2024-04-25T11:41:32.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.462+0000 7f83516a9640 1 -- 172.21.15.90:0/2562843734 learned_addr learned my addr 172.21.15.90:0/2562843734 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:32.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.462+0000 7f8350ea8640 1 --2- 172.21.15.90:0/2562843734 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608772aef60 0x560877697850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.462+0000 7f8350ea8640 1 --2- 172.21.15.90:0/2562843734 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5608772cf6b0 0x5608776a7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.462+0000 7f83516a9640 1 -- 172.21.15.90:0/2562843734 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608772aef60 msgr2=0x560877697850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.462+0000 7f83516a9640 1 --2- 172.21.15.90:0/2562843734 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608772aef60 0x560877697850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.462+0000 7f83516a9640 1 -- 172.21.15.90:0/2562843734 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5608772cf6b0 msgr2=0x5608776a7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.462+0000 7f83516a9640 1 --2- 172.21.15.90:0/2562843734 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5608772cf6b0 0x5608776a7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.463+0000 7f83516a9640 1 -- 172.21.15.90:0/2562843734 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608776897f0 con 0x5608772c74d0 2024-04-25T11:41:32.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.463+0000 7f83516a9640 1 --2- 172.21.15.90:0/2562843734 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772c74d0 0x560877691cb0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f8348024040 tx=0x7f8348000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edcb3a6bbba7286 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:32.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.463+0000 7f8336ffd640 1 -- 172.21.15.90:0/2562843734 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8348021c60 con 0x5608772c74d0 2024-04-25T11:41:32.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.463+0000 7f8336ffd640 1 -- 172.21.15.90:0/2562843734 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8348021e00 con 0x5608772c74d0 2024-04-25T11:41:32.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.463+0000 7f8336ffd640 1 -- 172.21.15.90:0/2562843734 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8348004f40 con 0x5608772c74d0 2024-04-25T11:41:32.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.464+0000 7f835a7be740 1 -- 172.21.15.90:0/2562843734 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772c74d0 msgr2=0x560877691cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.464+0000 7f835a7be740 1 --2- 172.21.15.90:0/2562843734 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772c74d0 0x560877691cb0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f8348024040 tx=0x7f8348000bc0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.464+0000 7f835a7be740 1 -- 172.21.15.90:0/2562843734 shutdown_connections 2024-04-25T11:41:32.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.464+0000 7f835a7be740 1 --2- 172.21.15.90:0/2562843734 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5608772cf6b0 0x5608776a7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.464+0000 7f835a7be740 1 --2- 172.21.15.90:0/2562843734 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772c74d0 0x560877691cb0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.464+0000 7f835a7be740 1 --2- 172.21.15.90:0/2562843734 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608772aef60 0x560877697850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.464+0000 7f835a7be740 1 -- 172.21.15.90:0/2562843734 >> 172.21.15.90:0/2562843734 conn(0x5608772e44e0 msgr2=0x5608776915a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:32.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.464+0000 7f835a7be740 1 -- 172.21.15.90:0/2562843734 shutdown_connections 2024-04-25T11:41:32.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.464+0000 7f835a7be740 1 -- 172.21.15.90:0/2562843734 wait complete. 2024-04-25T11:41:32.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.464+0000 7f835a7be740 1 -- start start 2024-04-25T11:41:32.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.464+0000 7f835a7be740 1 Processor -- start 2024-04-25T11:41:32.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.465+0000 7f835a7be740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772aef60 0x56087769f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.465+0000 7f835a7be740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5608772c74d0 0x56087769f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.465+0000 7f835a7be740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608772cf6b0 0x56087769fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.465+0000 7f835a7be740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f834c1b1080 con 0x5608772aef60 2024-04-25T11:41:32.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.465+0000 7f835a7be740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f834c1b09b0 con 0x5608772c74d0 2024-04-25T11:41:32.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.465+0000 7f835a7be740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f834c1adbf0 con 0x5608772cf6b0 2024-04-25T11:41:32.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.465+0000 7f8350ea8640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772aef60 0x56087769f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.465+0000 7f835a7be740 10 client.0 _ll_register_callbacks cb 0x7f834800ab90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:32.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.465+0000 7f8350ea8640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772aef60 0x56087769f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38234/0 (socket says 172.21.15.90:38234) 2024-04-25T11:41:32.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.465+0000 7f8350ea8640 1 -- 172.21.15.90:0/1190604149 learned_addr learned my addr 172.21.15.90:0/1190604149 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:32.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.465+0000 7f83516a9640 1 --2- 172.21.15.90:0/1190604149 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5608772c74d0 0x56087769f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.465+0000 7f8350ea8640 1 --2- 172.21.15.90:0/1190604149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608772cf6b0 0x56087769fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.465+0000 7f8350ea8640 1 -- 172.21.15.90:0/1190604149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608772cf6b0 msgr2=0x56087769fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.465+0000 7f8350ea8640 1 --2- 172.21.15.90:0/1190604149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608772cf6b0 0x56087769fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.465+0000 7f8350ea8640 1 -- 172.21.15.90:0/1190604149 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5608772c74d0 msgr2=0x56087769f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.465+0000 7f8350ea8640 1 --2- 172.21.15.90:0/1190604149 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5608772c74d0 0x56087769f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.465+0000 7f8350ea8640 1 -- 172.21.15.90:0/1190604149 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560877714aa0 con 0x5608772aef60 2024-04-25T11:41:32.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.465+0000 7f83516a9640 1 --2- 172.21.15.90:0/1190604149 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5608772c74d0 0x56087769f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:32.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.465+0000 7f8350ea8640 1 --2- 172.21.15.90:0/1190604149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772aef60 0x56087769f150 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f833c031960 tx=0x7f833c0058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1fe0630b09c0368 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:32.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.465+0000 7f8334ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:41:32.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.465+0000 7f8334ff9640 1 -- 172.21.15.90:0/1190604149 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c035040 con 0x5608772aef60 2024-04-25T11:41:32.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.466+0000 7f8334ff9640 1 -- 172.21.15.90:0/1190604149 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f833c03d460 con 0x5608772aef60 2024-04-25T11:41:32.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.466+0000 7f835a7be740 1 -- 172.21.15.90:0/1190604149 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8348021e00 con 0x5608772aef60 2024-04-25T11:41:32.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.466+0000 7f8334ff9640 1 -- 172.21.15.90:0/1190604149 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f833c03d660 con 0x5608772aef60 2024-04-25T11:41:32.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.466+0000 7f835a7be740 10 client.17538 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:32.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.466+0000 7f835a7be740 1 -- 172.21.15.90:0/1190604149 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560877715c50 con 0x5608772aef60 2024-04-25T11:41:32.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.466+0000 7f8334ff9640 1 -- 172.21.15.90:0/1190604149 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8348021e00 con 0x5608772aef60 2024-04-25T11:41:32.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.466+0000 7f8334ff9640 1 -- 172.21.15.90:0/1190604149 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f833c05c040 con 0x5608772aef60 2024-04-25T11:41:32.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.466+0000 7f835a7be740 10 client.17538 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:32.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.466+0000 7f835a7be740 10 client.17538 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:41:32.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.466+0000 7f835a7be740 20 client.17538 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:32.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.466+0000 7f835a7be740 1 --2- 172.21.15.90:0/1190604149 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5608772f0390 0x560877717f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.466+0000 7f835a7be740 4 client.17538 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:41:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.466+0000 7f835a7be740 1 -- 172.21.15.90:0/1190604149 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5608775fe800 con 0x5608772f0390 2024-04-25T11:41:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.466+0000 7f83516a9640 1 --2- 172.21.15.90:0/1190604149 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5608772f0390 0x560877717f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.467+0000 7f8334ff9640 10 client.17538 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:41:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.467+0000 7f83516a9640 1 --2- 172.21.15.90:0/1190604149 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5608772f0390 0x560877717f30 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4a93a3819b59db07 server_cookie=577a5332e1f08612 in_seq=0 out_seq=0 2024-04-25T11:41:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.467+0000 7f8334ff9640 1 -- 172.21.15.90:0/1190604149 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5608775fe800 con 0x5608772f0390 2024-04-25T11:41:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.467+0000 7f8334ff9640 10 client.17538 handle_command_reply: tid=0 2024-04-25T11:41:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.467+0000 7f835a7be740 1 client.17538 shutdown 2024-04-25T11:41:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.467+0000 7f835a7be740 1 -- 172.21.15.90:0/1190604149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772aef60 msgr2=0x56087769f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.467+0000 7f835a7be740 1 --2- 172.21.15.90:0/1190604149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772aef60 0x56087769f150 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f833c031960 tx=0x7f833c0058b0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.467+0000 7f835a7be740 1 -- 172.21.15.90:0/1190604149 shutdown_connections 2024-04-25T11:41:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.467+0000 7f835a7be740 1 --2- 172.21.15.90:0/1190604149 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5608772f0390 0x560877717f30 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.467+0000 7f835a7be740 1 --2- 172.21.15.90:0/1190604149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608772cf6b0 0x56087769fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.467+0000 7f8334ff9640 0 client.17538 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:41:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.467+0000 7f835a7be740 1 --2- 172.21.15.90:0/1190604149 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5608772c74d0 0x56087769f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.467+0000 7f835a7be740 1 --2- 172.21.15.90:0/1190604149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5608772aef60 0x56087769f150 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.467+0000 7f835a7be740 1 -- 172.21.15.90:0/1190604149 >> 172.21.15.90:0/1190604149 conn(0x5608772e44e0 msgr2=0x56087769a920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:32.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.467+0000 7f835a7be740 1 -- 172.21.15.90:0/1190604149 shutdown_connections 2024-04-25T11:41:32.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.467+0000 7f835a7be740 1 -- 172.21.15.90:0/1190604149 wait complete. 2024-04-25T11:41:32.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.468+0000 7f835a7be740 20 client.17538 trim_cache size 0 max 16384 2024-04-25T11:41:32.461 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:41:32.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.468+0000 7f835390d640 1 -- 172.21.15.90:0/201319347 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f83380afc60 msgr2=0x7f83380b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.468+0000 7f835390d640 1 --2- 172.21.15.90:0/201319347 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f83380afc60 0x7f83380b2170 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f83440a73d0 tx=0x7f83440a3610 comp rx=0 tx=0).stop 2024-04-25T11:41:32.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.468+0000 7f835390d640 1 -- 172.21.15.90:0/201319347 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f834c107450 msgr2=0x7f834c19efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.468+0000 7f835390d640 1 --2- 172.21.15.90:0/201319347 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f834c107450 0x7f834c19efc0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f834400ec60 tx=0x7f834400c7a0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.468+0000 7f835390d640 1 -- 172.21.15.90:0/201319347 shutdown_connections 2024-04-25T11:41:32.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.468+0000 7f835390d640 1 --2- 172.21.15.90:0/201319347 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f83380afc60 0x7f83380b2170 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.468+0000 7f835390d640 1 --2- 172.21.15.90:0/201319347 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f834c107450 0x7f834c19efc0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.468+0000 7f835390d640 1 --2- 172.21.15.90:0/201319347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f834c106990 0x7f834c19ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.468+0000 7f835390d640 1 --2- 172.21.15.90:0/201319347 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f834c102400 0x7f834c19a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.468+0000 7f835390d640 1 -- 172.21.15.90:0/201319347 >> 172.21.15.90:0/201319347 conn(0x7f834c077880 msgr2=0x7f834c10def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:32.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.468+0000 7f835390d640 1 -- 172.21.15.90:0/201319347 shutdown_connections 2024-04-25T11:41:32.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.468+0000 7f835390d640 1 -- 172.21.15.90:0/201319347 wait complete. 2024-04-25T11:41:32.468 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:32 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3151972026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:41:32.469 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T11:41:32.487 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:32 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3151972026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:41:32.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:32 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3151972026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:41:32.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.613+0000 7fd612892640 1 -- 172.21.15.90:0/4052334164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd60c10d380 msgr2=0x7fd60c10f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.613+0000 7fd612892640 1 --2- 172.21.15.90:0/4052334164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd60c10d380 0x7fd60c10f7a0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fd604009ca0 tx=0x7fd60402a8c0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.613+0000 7fd612892640 1 -- 172.21.15.90:0/4052334164 shutdown_connections 2024-04-25T11:41:32.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.613+0000 7fd612892640 1 --2- 172.21.15.90:0/4052334164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd60c10d380 0x7fd60c10f7a0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.613+0000 7fd612892640 1 --2- 172.21.15.90:0/4052334164 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd60c102af0 0x7fd60c102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.613+0000 7fd612892640 1 --2- 172.21.15.90:0/4052334164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd60c108a20 0x7fd60c1025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.613+0000 7fd612892640 1 -- 172.21.15.90:0/4052334164 >> 172.21.15.90:0/4052334164 conn(0x7fd60c078100 msgr2=0x7fd60c078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:32.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.613+0000 7fd612892640 1 -- 172.21.15.90:0/4052334164 shutdown_connections 2024-04-25T11:41:32.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.613+0000 7fd612892640 1 -- 172.21.15.90:0/4052334164 wait complete. 2024-04-25T11:41:32.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.614+0000 7fd612892640 1 Processor -- start 2024-04-25T11:41:32.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.614+0000 7fd612892640 1 -- start start 2024-04-25T11:41:32.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.614+0000 7fd612892640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd60c102af0 0x7fd60c199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.614+0000 7fd612892640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd60c108a20 0x7fd60c199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.615+0000 7fd612892640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd60c10d380 0x7fd60c19dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.615+0000 7fd612892640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c112be0 con 0x7fd60c10d380 2024-04-25T11:41:32.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.615+0000 7fd612892640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c112a60 con 0x7fd60c108a20 2024-04-25T11:41:32.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.615+0000 7fd612892640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c112d60 con 0x7fd60c102af0 2024-04-25T11:41:32.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.615+0000 7fd60b7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd60c108a20 0x7fd60c199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.615+0000 7fd610e2f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd60c10d380 0x7fd60c19dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.615+0000 7fd60b7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd60c108a20 0x7fd60c199ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:58332/0 (socket says 172.21.15.90:58332) 2024-04-25T11:41:32.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.615+0000 7fd60b7fe640 1 -- 172.21.15.90:0/1307257763 learned_addr learned my addr 172.21.15.90:0/1307257763 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:32.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.615+0000 7fd610e2f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd60c10d380 0x7fd60c19dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38250/0 (socket says 172.21.15.90:38250) 2024-04-25T11:41:32.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.615+0000 7fd60bfff640 1 --2- 172.21.15.90:0/1307257763 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd60c102af0 0x7fd60c199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.615+0000 7fd60b7fe640 1 -- 172.21.15.90:0/1307257763 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd60c102af0 msgr2=0x7fd60c199580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.615+0000 7fd60b7fe640 1 --2- 172.21.15.90:0/1307257763 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd60c102af0 0x7fd60c199580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.615+0000 7fd60b7fe640 1 -- 172.21.15.90:0/1307257763 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd60c10d380 msgr2=0x7fd60c19dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.615+0000 7fd60b7fe640 1 --2- 172.21.15.90:0/1307257763 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd60c10d380 0x7fd60c19dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.615+0000 7fd60b7fe640 1 -- 172.21.15.90:0/1307257763 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd60c19e3e0 con 0x7fd60c108a20 2024-04-25T11:41:32.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.615+0000 7fd60b7fe640 1 --2- 172.21.15.90:0/1307257763 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd60c108a20 0x7fd60c199ae0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd60000e9f0 tx=0x7fd60000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:32.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.615+0000 7fd6097fa640 1 -- 172.21.15.90:0/1307257763 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60000ccc0 con 0x7fd60c108a20 2024-04-25T11:41:32.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.616+0000 7fd612892640 1 -- 172.21.15.90:0/1307257763 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd60c19e6d0 con 0x7fd60c108a20 2024-04-25T11:41:32.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.616+0000 7fd6097fa640 1 -- 172.21.15.90:0/1307257763 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6000035f0 con 0x7fd60c108a20 2024-04-25T11:41:32.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.616+0000 7fd6097fa640 1 -- 172.21.15.90:0/1307257763 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60000ce60 con 0x7fd60c108a20 2024-04-25T11:41:32.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.616+0000 7fd612892640 1 -- 172.21.15.90:0/1307257763 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd60c1a6050 con 0x7fd60c108a20 2024-04-25T11:41:32.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.618+0000 7fd619814740 1 Processor -- start 2024-04-25T11:41:32.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.618+0000 7fd6097fa640 1 -- 172.21.15.90:0/1307257763 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fd600010690 con 0x7fd60c108a20 2024-04-25T11:41:32.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.618+0000 7fd619814740 1 -- start start 2024-04-25T11:41:32.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.619+0000 7fd619814740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f605eea390 0x55f60628b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.619+0000 7fd619814740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f605ec14d0 0x55f60628bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.619+0000 7fd619814740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605ec96b0 0x55f60629a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.619+0000 7fd619814740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f6061f0d80 con 0x55f605ec96b0 2024-04-25T11:41:32.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.619+0000 7fd619814740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f6061aaa20 con 0x55f605eea390 2024-04-25T11:41:32.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.619+0000 7fd619814740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f6061ecd40 con 0x55f605ec14d0 2024-04-25T11:41:32.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.619+0000 7fd60b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605ec96b0 0x55f60629a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.619+0000 7fd610e2f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f605eea390 0x55f60628b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.619+0000 7fd60b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605ec96b0 0x55f60629a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38260/0 (socket says 172.21.15.90:38260) 2024-04-25T11:41:32.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.619+0000 7fd610e2f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f605eea390 0x55f60628b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:58338/0 (socket says 172.21.15.90:58338) 2024-04-25T11:41:32.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.619+0000 7fd60b7fe640 1 -- 172.21.15.90:0/2804166858 learned_addr learned my addr 172.21.15.90:0/2804166858 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:32.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.619+0000 7fd60bfff640 1 --2- 172.21.15.90:0/2804166858 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f605ec14d0 0x55f60628bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.619+0000 7fd6097fa640 1 --2- 172.21.15.90:0/1307257763 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd5f40afc60 0x7fd5f40b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.619+0000 7fd60b7fe640 1 -- 172.21.15.90:0/2804166858 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f605ec14d0 msgr2=0x55f60628bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.619+0000 7fd60b7fe640 1 --2- 172.21.15.90:0/2804166858 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f605ec14d0 0x55f60628bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.619+0000 7fd60b7fe640 1 -- 172.21.15.90:0/2804166858 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f605eea390 msgr2=0x55f60628b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.619+0000 7fd60b7fe640 1 --2- 172.21.15.90:0/2804166858 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f605eea390 0x55f60628b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.619+0000 7fd60b7fe640 1 -- 172.21.15.90:0/2804166858 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6061b5c30 con 0x55f605ec96b0 2024-04-25T11:41:32.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.619+0000 7fd610e2f640 1 --2- 172.21.15.90:0/2804166858 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f605eea390 0x55f60628b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:32.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.619+0000 7fd610e2f640 1 --2- 172.21.15.90:0/1307257763 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd5f40afc60 0x7fd5f40b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.620+0000 7fd6097fa640 1 -- 172.21.15.90:0/1307257763 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fd6000df3d0 con 0x7fd60c108a20 2024-04-25T11:41:32.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.620+0000 7fd60b7fe640 1 --2- 172.21.15.90:0/2804166858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605ec96b0 0x55f60629a8b0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fd6000dca60 tx=0x7fd60009be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d733f785ccf2a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:32.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.620+0000 7fd5f27fc640 1 -- 172.21.15.90:0/2804166858 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6000d0070 con 0x55f605ec96b0 2024-04-25T11:41:32.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.620+0000 7fd610e2f640 1 --2- 172.21.15.90:0/1307257763 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd5f40afc60 0x7fd5f40b2170 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x55f606288ee0 tx=0x7fd6040095c0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:32.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.620+0000 7fd5f27fc640 1 -- 172.21.15.90:0/2804166858 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6000cef00 con 0x55f605ec96b0 2024-04-25T11:41:32.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.620+0000 7fd5f27fc640 1 -- 172.21.15.90:0/2804166858 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6000a7c10 con 0x55f605ec96b0 2024-04-25T11:41:32.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.621+0000 7fd619814740 1 -- 172.21.15.90:0/2804166858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605ec96b0 msgr2=0x55f60629a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.621+0000 7fd619814740 1 --2- 172.21.15.90:0/2804166858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605ec96b0 0x55f60629a8b0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fd6000dca60 tx=0x7fd60009be00 comp rx=0 tx=0).stop 2024-04-25T11:41:32.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.621+0000 7fd619814740 1 -- 172.21.15.90:0/2804166858 shutdown_connections 2024-04-25T11:41:32.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.621+0000 7fd619814740 1 --2- 172.21.15.90:0/2804166858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605ec96b0 0x55f60629a8b0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.621+0000 7fd619814740 1 --2- 172.21.15.90:0/2804166858 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f605ec14d0 0x55f60628bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.621+0000 7fd619814740 1 --2- 172.21.15.90:0/2804166858 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f605eea390 0x55f60628b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.621+0000 7fd619814740 1 -- 172.21.15.90:0/2804166858 >> 172.21.15.90:0/2804166858 conn(0x55f605ede4e0 msgr2=0x55f606287fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:32.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.621+0000 7fd619814740 1 -- 172.21.15.90:0/2804166858 shutdown_connections 2024-04-25T11:41:32.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.621+0000 7fd619814740 1 -- 172.21.15.90:0/2804166858 wait complete. 2024-04-25T11:41:32.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.621+0000 7fd619814740 1 -- start start 2024-04-25T11:41:32.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.622+0000 7fd619814740 1 Processor -- start 2024-04-25T11:41:32.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.622+0000 7fd619814740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f605ec14d0 0x55f606298120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.622+0000 7fd619814740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f605ec96b0 0x55f606298660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.622+0000 7fd619814740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605eea390 0x55f6062a4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.622+0000 7fd619814740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f60629aee0 con 0x55f605eea390 2024-04-25T11:41:32.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.622+0000 7fd619814740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f606177dc0 con 0x55f605ec14d0 2024-04-25T11:41:32.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.622+0000 7fd619814740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f60629b060 con 0x55f605ec96b0 2024-04-25T11:41:32.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.622+0000 7fd619814740 10 client.0 _ll_register_callbacks cb 0x55f606193600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:32.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.622+0000 7fd60bfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f605ec14d0 0x55f606298120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.622+0000 7fd610e2f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605eea390 0x55f6062a4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.622+0000 7fd60bfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f605ec14d0 0x55f606298120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:58344/0 (socket says 172.21.15.90:58344) 2024-04-25T11:41:32.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.622+0000 7fd60bfff640 1 -- 172.21.15.90:0/1967235566 learned_addr learned my addr 172.21.15.90:0/1967235566 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:32.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.622+0000 7fd60b7fe640 1 --2- 172.21.15.90:0/1967235566 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f605ec96b0 0x55f606298660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.623+0000 7fd60bfff640 1 -- 172.21.15.90:0/1967235566 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f605ec96b0 msgr2=0x55f606298660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.623+0000 7fd60bfff640 1 --2- 172.21.15.90:0/1967235566 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f605ec96b0 0x55f606298660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.623+0000 7fd60bfff640 1 -- 172.21.15.90:0/1967235566 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605eea390 msgr2=0x55f6062a4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.623+0000 7fd60bfff640 1 --2- 172.21.15.90:0/1967235566 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605eea390 0x55f6062a4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.623+0000 7fd60bfff640 1 -- 172.21.15.90:0/1967235566 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f605f023f0 con 0x55f605ec14d0 2024-04-25T11:41:32.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.623+0000 7fd610e2f640 1 --2- 172.21.15.90:0/1967235566 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605eea390 0x55f6062a4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:32.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.623+0000 7fd60bfff640 1 --2- 172.21.15.90:0/1967235566 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f605ec14d0 0x55f606298120 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00bad0 tx=0x7fd5fc00bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5723e2f50d8459cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:32.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.623+0000 7fd5eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:41:32.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.623+0000 7fd5eb7fe640 1 -- 172.21.15.90:0/1967235566 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc005020 con 0x55f605ec14d0 2024-04-25T11:41:32.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.623+0000 7fd5eb7fe640 1 -- 172.21.15.90:0/1967235566 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fd5fc0051c0 con 0x55f605ec14d0 2024-04-25T11:41:32.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.623+0000 7fd619814740 1 -- 172.21.15.90:0/1967235566 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6000cef00 con 0x55f605ec14d0 2024-04-25T11:41:32.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.623+0000 7fd5eb7fe640 1 -- 172.21.15.90:0/1967235566 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd5fc0053c0 con 0x55f605ec14d0 2024-04-25T11:41:32.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.623+0000 7fd619814740 10 client.26528 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:32.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.623+0000 7fd619814740 1 -- 172.21.15.90:0/1967235566 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f605fcc7d0 con 0x55f605ec14d0 2024-04-25T11:41:32.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.623+0000 7fd5eb7fe640 1 -- 172.21.15.90:0/1967235566 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6000cef00 con 0x55f605ec14d0 2024-04-25T11:41:32.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.624+0000 7fd5eb7fe640 1 -- 172.21.15.90:0/1967235566 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fd5fc01b670 con 0x55f605ec14d0 2024-04-25T11:41:32.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.624+0000 7fd619814740 10 client.26528 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:32.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.624+0000 7fd619814740 10 client.26528 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:41:32.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.624+0000 7fd619814740 20 client.26528 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:32.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.624+0000 7fd619814740 1 --2- 172.21.15.90:0/1967235566 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f605ea8f60 0x55f6062aa700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.624+0000 7fd619814740 4 client.26528 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:41:32.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.624+0000 7fd619814740 1 -- 172.21.15.90:0/1967235566 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6061fe990 con 0x55f605ea8f60 2024-04-25T11:41:32.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.624+0000 7fd60b7fe640 1 --2- 172.21.15.90:0/1967235566 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f605ea8f60 0x55f6062aa700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.624+0000 7fd5eb7fe640 10 client.26528 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:32.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.624+0000 7fd60b7fe640 1 --2- 172.21.15.90:0/1967235566 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f605ea8f60 0x55f6062aa700 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b460666b784ec67 server_cookie=acee7409825125d2 in_seq=0 out_seq=0 2024-04-25T11:41:32.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.625+0000 7fd5eb7fe640 1 -- 172.21.15.90:0/1967235566 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f6061fe990 con 0x55f605ea8f60 2024-04-25T11:41:32.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.625+0000 7fd5eb7fe640 10 client.26528 handle_command_reply: tid=0 2024-04-25T11:41:32.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.625+0000 7fd619814740 1 client.26528 shutdown 2024-04-25T11:41:32.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.625+0000 7fd619814740 1 -- 172.21.15.90:0/1967235566 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f605ec14d0 msgr2=0x55f606298120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.625+0000 7fd619814740 1 --2- 172.21.15.90:0/1967235566 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f605ec14d0 0x55f606298120 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00bad0 tx=0x7fd5fc00bed0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.625+0000 7fd619814740 1 -- 172.21.15.90:0/1967235566 shutdown_connections 2024-04-25T11:41:32.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.625+0000 7fd619814740 1 --2- 172.21.15.90:0/1967235566 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f605ea8f60 0x55f6062aa700 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.625+0000 7fd619814740 1 --2- 172.21.15.90:0/1967235566 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605eea390 0x55f6062a4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.625+0000 7fd619814740 1 --2- 172.21.15.90:0/1967235566 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f605ec96b0 0x55f606298660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.625+0000 7fd619814740 1 --2- 172.21.15.90:0/1967235566 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f605ec14d0 0x55f606298120 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.625+0000 7fd619814740 1 -- 172.21.15.90:0/1967235566 >> 172.21.15.90:0/1967235566 conn(0x55f605ede4e0 msgr2=0x55f60628d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:32.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.625+0000 7fd5eb7fe640 0 client.26528 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:32.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.626+0000 7fd619814740 1 -- 172.21.15.90:0/1967235566 shutdown_connections 2024-04-25T11:41:32.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.626+0000 7fd619814740 1 -- 172.21.15.90:0/1967235566 wait complete. 2024-04-25T11:41:32.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.626+0000 7fd619814740 20 client.26528 trim_cache size 0 max 16384 2024-04-25T11:41:32.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.647+0000 7fd619814740 1 Processor -- start 2024-04-25T11:41:32.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.647+0000 7fd619814740 1 -- start start 2024-04-25T11:41:32.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.647+0000 7fd619814740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605ede4e0 0x55f60628db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.647+0000 7fd619814740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f605ec14d0 0x55f6062a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.647+0000 7fd619814740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f605ec96b0 0x55f6062afec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.647+0000 7fd619814740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c1ad920 con 0x55f605ede4e0 2024-04-25T11:41:32.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.647+0000 7fd619814740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c1af460 con 0x55f605ec96b0 2024-04-25T11:41:32.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.647+0000 7fd619814740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c1adff0 con 0x55f605ec14d0 2024-04-25T11:41:32.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.647+0000 7fd60bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605ede4e0 0x55f60628db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.647+0000 7fd60bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605ede4e0 0x55f60628db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38286/0 (socket says 172.21.15.90:38286) 2024-04-25T11:41:32.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.647+0000 7fd610e2f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f605ec96b0 0x55f6062afec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.647+0000 7fd60bfff640 1 -- 172.21.15.90:0/316381516 learned_addr learned my addr 172.21.15.90:0/316381516 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:32.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.647+0000 7fd60b7fe640 1 --2- 172.21.15.90:0/316381516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f605ec14d0 0x55f6062a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.647+0000 7fd60bfff640 1 -- 172.21.15.90:0/316381516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f605ec14d0 msgr2=0x55f6062a3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.647+0000 7fd60bfff640 1 --2- 172.21.15.90:0/316381516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f605ec14d0 0x55f6062a3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.647+0000 7fd60bfff640 1 -- 172.21.15.90:0/316381516 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f605ec96b0 msgr2=0x55f6062afec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.647+0000 7fd60bfff640 1 --2- 172.21.15.90:0/316381516 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f605ec96b0 0x55f6062afec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.647+0000 7fd60bfff640 1 -- 172.21.15.90:0/316381516 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f606293ce0 con 0x55f605ede4e0 2024-04-25T11:41:32.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.648+0000 7fd60bfff640 1 --2- 172.21.15.90:0/316381516 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605ede4e0 0x55f60628db70 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc003a70 tx=0x7fd5fc004460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c2e2b39a5794674 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:32.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.648+0000 7fd5f1ffb640 1 -- 172.21.15.90:0/316381516 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc030070 con 0x55f605ede4e0 2024-04-25T11:41:32.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.648+0000 7fd5f1ffb640 1 -- 172.21.15.90:0/316381516 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5fc005620 con 0x55f605ede4e0 2024-04-25T11:41:32.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.648+0000 7fd5f1ffb640 1 -- 172.21.15.90:0/316381516 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc027040 con 0x55f605ede4e0 2024-04-25T11:41:32.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.649+0000 7fd619814740 1 -- 172.21.15.90:0/316381516 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605ede4e0 msgr2=0x55f60628db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.649+0000 7fd619814740 1 --2- 172.21.15.90:0/316381516 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605ede4e0 0x55f60628db70 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc003a70 tx=0x7fd5fc004460 comp rx=0 tx=0).stop 2024-04-25T11:41:32.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.649+0000 7fd619814740 1 -- 172.21.15.90:0/316381516 shutdown_connections 2024-04-25T11:41:32.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.649+0000 7fd619814740 1 --2- 172.21.15.90:0/316381516 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f605ec96b0 0x55f6062afec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.649+0000 7fd619814740 1 --2- 172.21.15.90:0/316381516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f605ec14d0 0x55f6062a3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.649+0000 7fd619814740 1 --2- 172.21.15.90:0/316381516 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605ede4e0 0x55f60628db70 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.649+0000 7fd619814740 1 -- 172.21.15.90:0/316381516 >> 172.21.15.90:0/316381516 conn(0x55f6062aaa50 msgr2=0x55f60628d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:32.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.649+0000 7fd619814740 1 -- 172.21.15.90:0/316381516 shutdown_connections 2024-04-25T11:41:32.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.649+0000 7fd619814740 1 -- 172.21.15.90:0/316381516 wait complete. 2024-04-25T11:41:32.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.649+0000 7fd619814740 1 -- start start 2024-04-25T11:41:32.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.649+0000 7fd619814740 1 Processor -- start 2024-04-25T11:41:32.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.650+0000 7fd619814740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f605ec14d0 0x55f60629d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.650+0000 7fd619814740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f605ec96b0 0x55f60629d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.650+0000 7fd619814740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605ede4e0 0x55f60629dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.650+0000 7fd619814740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c1aed90 con 0x55f605ede4e0 2024-04-25T11:41:32.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.650+0000 7fd619814740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c1ae6c0 con 0x55f605ec96b0 2024-04-25T11:41:32.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.650+0000 7fd619814740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c1ab900 con 0x55f605ec14d0 2024-04-25T11:41:32.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.650+0000 7fd619814740 10 client.0 _ll_register_callbacks cb 0x7fd5fc0040a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:32.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.650+0000 7fd610e2f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605ede4e0 0x55f60629dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.650+0000 7fd60b7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f605ec96b0 0x55f60629d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.650+0000 7fd610e2f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605ede4e0 0x55f60629dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38294/0 (socket says 172.21.15.90:38294) 2024-04-25T11:41:32.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.650+0000 7fd610e2f640 1 -- 172.21.15.90:0/1145221176 learned_addr learned my addr 172.21.15.90:0/1145221176 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:32.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.650+0000 7fd60bfff640 1 --2- 172.21.15.90:0/1145221176 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f605ec14d0 0x55f60629d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.650+0000 7fd610e2f640 1 -- 172.21.15.90:0/1145221176 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f605ec14d0 msgr2=0x55f60629d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.650+0000 7fd610e2f640 1 --2- 172.21.15.90:0/1145221176 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f605ec14d0 0x55f60629d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.650+0000 7fd610e2f640 1 -- 172.21.15.90:0/1145221176 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f605ec96b0 msgr2=0x55f60629d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:32.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.650+0000 7fd610e2f640 1 --2- 172.21.15.90:0/1145221176 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f605ec96b0 0x55f60629d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.650+0000 7fd610e2f640 1 -- 172.21.15.90:0/1145221176 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f606310c70 con 0x55f605ede4e0 2024-04-25T11:41:32.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.650+0000 7fd610e2f640 1 --2- 172.21.15.90:0/1145221176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605ede4e0 0x55f60629dbb0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fd604009ca0 tx=0x7fd60402a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7daa5820fe5825af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:32.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.650+0000 7fd5eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:41:32.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.651+0000 7fd5eb7fe640 1 -- 172.21.15.90:0/1145221176 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd604004510 con 0x55f605ede4e0 2024-04-25T11:41:32.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.651+0000 7fd5eb7fe640 1 -- 172.21.15.90:0/1145221176 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fd6040046b0 con 0x55f605ede4e0 2024-04-25T11:41:32.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.651+0000 7fd619814740 1 -- 172.21.15.90:0/1145221176 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5fc005620 con 0x55f605ede4e0 2024-04-25T11:41:32.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.651+0000 7fd5eb7fe640 1 -- 172.21.15.90:0/1145221176 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd604063050 con 0x55f605ede4e0 2024-04-25T11:41:32.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.651+0000 7fd619814740 10 client.17556 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:32.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.651+0000 7fd619814740 1 -- 172.21.15.90:0/1145221176 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f606311eb0 con 0x55f605ede4e0 2024-04-25T11:41:32.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.651+0000 7fd5eb7fe640 1 -- 172.21.15.90:0/1145221176 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5fc005620 con 0x55f605ede4e0 2024-04-25T11:41:32.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.651+0000 7fd5eb7fe640 1 -- 172.21.15.90:0/1145221176 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fd604030750 con 0x55f605ede4e0 2024-04-25T11:41:32.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.651+0000 7fd619814740 10 client.17556 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:32.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.651+0000 7fd619814740 10 client.17556 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:41:32.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.651+0000 7fd619814740 20 client.17556 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:32.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.651+0000 7fd619814740 1 --2- 172.21.15.90:0/1145221176 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f605eea390 0x55f6063146f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:32.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.651+0000 7fd619814740 4 client.17556 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:41:32.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.651+0000 7fd619814740 1 -- 172.21.15.90:0/1145221176 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f6061f8800 con 0x55f605eea390 2024-04-25T11:41:32.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.651+0000 7fd60bfff640 1 --2- 172.21.15.90:0/1145221176 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f605eea390 0x55f6063146f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:32.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.652+0000 7fd5eb7fe640 10 client.17556 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:32.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.652+0000 7fd60bfff640 1 --2- 172.21.15.90:0/1145221176 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f605eea390 0x55f6063146f0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c00ee6182410206e server_cookie=146aeeeb9f2be92b in_seq=0 out_seq=0 2024-04-25T11:41:32.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.652+0000 7fd5eb7fe640 1 -- 172.21.15.90:0/1145221176 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f6061f8800 con 0x55f605eea390 2024-04-25T11:41:32.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.652+0000 7fd5eb7fe640 10 client.17556 handle_command_reply: tid=0 2024-04-25T11:41:32.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.652+0000 7fd619814740 1 client.17556 shutdown 2024-04-25T11:41:32.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.653+0000 7fd619814740 1 -- 172.21.15.90:0/1145221176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605ede4e0 msgr2=0x55f60629dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.653+0000 7fd619814740 1 --2- 172.21.15.90:0/1145221176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605ede4e0 0x55f60629dbb0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fd604009ca0 tx=0x7fd60402a8c0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.653+0000 7fd619814740 1 -- 172.21.15.90:0/1145221176 shutdown_connections 2024-04-25T11:41:32.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.653+0000 7fd619814740 1 --2- 172.21.15.90:0/1145221176 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f605eea390 0x55f6063146f0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.653+0000 7fd619814740 1 --2- 172.21.15.90:0/1145221176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f605ede4e0 0x55f60629dbb0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.653+0000 7fd5eb7fe640 0 client.17556 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:32.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.653+0000 7fd619814740 1 --2- 172.21.15.90:0/1145221176 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f605ec96b0 0x55f60629d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.653+0000 7fd619814740 1 --2- 172.21.15.90:0/1145221176 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f605ec14d0 0x55f60629d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.653+0000 7fd619814740 1 -- 172.21.15.90:0/1145221176 >> 172.21.15.90:0/1145221176 conn(0x55f6062aaa50 msgr2=0x55f606297890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:32.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.653+0000 7fd619814740 1 -- 172.21.15.90:0/1145221176 shutdown_connections 2024-04-25T11:41:32.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.653+0000 7fd619814740 1 -- 172.21.15.90:0/1145221176 wait complete. 2024-04-25T11:41:32.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.653+0000 7fd619814740 20 client.17556 trim_cache size 0 max 16384 2024-04-25T11:41:32.646 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:41:32.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.653+0000 7fd612892640 1 -- 172.21.15.90:0/1307257763 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd5f40afc60 msgr2=0x7fd5f40b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.653+0000 7fd612892640 1 --2- 172.21.15.90:0/1307257763 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd5f40afc60 0x7fd5f40b2170 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x55f606288ee0 tx=0x7fd6040095c0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.653+0000 7fd612892640 1 -- 172.21.15.90:0/1307257763 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd60c108a20 msgr2=0x7fd60c199ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:32.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.653+0000 7fd612892640 1 --2- 172.21.15.90:0/1307257763 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd60c108a20 0x7fd60c199ae0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd60000e9f0 tx=0x7fd60000eeb0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.654+0000 7fd612892640 1 -- 172.21.15.90:0/1307257763 shutdown_connections 2024-04-25T11:41:32.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.654+0000 7fd612892640 1 --2- 172.21.15.90:0/1307257763 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd5f40afc60 0x7fd5f40b2170 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.654+0000 7fd612892640 1 --2- 172.21.15.90:0/1307257763 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd60c10d380 0x7fd60c19dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.654+0000 7fd612892640 1 --2- 172.21.15.90:0/1307257763 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd60c108a20 0x7fd60c199ae0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.654+0000 7fd612892640 1 --2- 172.21.15.90:0/1307257763 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd60c102af0 0x7fd60c199580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:32.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.654+0000 7fd612892640 1 -- 172.21.15.90:0/1307257763 >> 172.21.15.90:0/1307257763 conn(0x7fd60c078100 msgr2=0x7fd60c1040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:32.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.654+0000 7fd612892640 1 -- 172.21.15.90:0/1307257763 shutdown_connections 2024-04-25T11:41:32.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:32.654+0000 7fd612892640 1 -- 172.21.15.90:0/1307257763 wait complete. 2024-04-25T11:41:33.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:33 smithi156 ceph-mon[33045]: pgmap v197: 97 pgs: 97 active+clean; 59 GiB data, 178 GiB used, 895 GiB / 1.0 TiB avail; 3.0 MiB/s rd, 662 MiB/s wr, 796 op/s 2024-04-25T11:41:33.656 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T11:41:33.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:41:33.657 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T11:41:33.679 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:33 smithi090 ceph-mon[29575]: pgmap v197: 97 pgs: 97 active+clean; 59 GiB data, 178 GiB used, 895 GiB / 1.0 TiB avail; 3.0 MiB/s rd, 662 MiB/s wr, 796 op/s 2024-04-25T11:41:33.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.793+0000 7feb1a164640 1 -- 172.21.15.90:0/3219014128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb1410e570 msgr2=0x7feb14110990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:33.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.793+0000 7feb1a164640 1 --2- 172.21.15.90:0/3219014128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb1410e570 0x7feb14110990 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7feb0c009ca0 tx=0x7feb0c02a8c0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.793+0000 7feb1a164640 1 -- 172.21.15.90:0/3219014128 shutdown_connections 2024-04-25T11:41:33.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.793+0000 7feb1a164640 1 --2- 172.21.15.90:0/3219014128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb1410e570 0x7feb14110990 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.793+0000 7feb1a164640 1 --2- 172.21.15.90:0/3219014128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb14102070 0x7feb14102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.793+0000 7feb1a164640 1 --2- 172.21.15.90:0/3219014128 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feb14109bc0 0x7feb14101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.793+0000 7feb1a164640 1 -- 172.21.15.90:0/3219014128 >> 172.21.15.90:0/3219014128 conn(0x7feb14077920 msgr2=0x7feb14077d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:33.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.794+0000 7feb1a164640 1 -- 172.21.15.90:0/3219014128 shutdown_connections 2024-04-25T11:41:33.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.794+0000 7feb1a164640 1 -- 172.21.15.90:0/3219014128 wait complete. 2024-04-25T11:41:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.794+0000 7feb1a164640 1 Processor -- start 2024-04-25T11:41:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.795+0000 7feb1a164640 1 -- start start 2024-04-25T11:41:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.795+0000 7feb1a164640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feb14102070 0x7feb1419a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.795+0000 7feb1a164640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb14109bc0 0x7feb1419ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.795+0000 7feb1a164640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb1410e570 0x7feb1419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.795+0000 7feb1a164640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7feb14113d30 con 0x7feb14109bc0 2024-04-25T11:41:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.795+0000 7feb1a164640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7feb14113bb0 con 0x7feb14102070 2024-04-25T11:41:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.795+0000 7feb1a164640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7feb14113eb0 con 0x7feb1410e570 2024-04-25T11:41:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.795+0000 7feb137fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feb14102070 0x7feb1419a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.795+0000 7feb137fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feb14102070 0x7feb1419a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:58388/0 (socket says 172.21.15.90:58388) 2024-04-25T11:41:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.795+0000 7feb137fe640 1 -- 172.21.15.90:0/56146794 learned_addr learned my addr 172.21.15.90:0/56146794 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.795+0000 7feb12ffd640 1 --2- 172.21.15.90:0/56146794 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb14109bc0 0x7feb1419ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.795+0000 7feb13fff640 1 --2- 172.21.15.90:0/56146794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb1410e570 0x7feb1419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.795+0000 7feb137fe640 1 -- 172.21.15.90:0/56146794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb1410e570 msgr2=0x7feb1419efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.795+0000 7feb137fe640 1 --2- 172.21.15.90:0/56146794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb1410e570 0x7feb1419efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.795+0000 7feb137fe640 1 -- 172.21.15.90:0/56146794 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb14109bc0 msgr2=0x7feb1419ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.795+0000 7feb137fe640 1 --2- 172.21.15.90:0/56146794 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb14109bc0 0x7feb1419ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.795+0000 7feb137fe640 1 -- 172.21.15.90:0/56146794 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb1419f610 con 0x7feb14102070 2024-04-25T11:41:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.796+0000 7feb137fe640 1 --2- 172.21.15.90:0/56146794 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feb14102070 0x7feb1419a7b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7feb0400e9f0 tx=0x7feb0400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.796+0000 7feb10ff9640 1 -- 172.21.15.90:0/56146794 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0400ccc0 con 0x7feb14102070 2024-04-25T11:41:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.796+0000 7feb1a164640 1 -- 172.21.15.90:0/56146794 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb1419f900 con 0x7feb14102070 2024-04-25T11:41:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.796+0000 7feb10ff9640 1 -- 172.21.15.90:0/56146794 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb040035f0 con 0x7feb14102070 2024-04-25T11:41:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.796+0000 7feb10ff9640 1 -- 172.21.15.90:0/56146794 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0400ce60 con 0x7feb14102070 2024-04-25T11:41:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.796+0000 7feb1a164640 1 -- 172.21.15.90:0/56146794 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb141a7280 con 0x7feb14102070 2024-04-25T11:41:33.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.798+0000 7feb21015740 1 Processor -- start 2024-04-25T11:41:33.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.798+0000 7feb21015740 1 -- start start 2024-04-25T11:41:33.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.798+0000 7feb21015740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c260b6410 0x559c26456620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:33.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.798+0000 7feb21015740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c2608d550 0x559c26456b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:33.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.798+0000 7feb21015740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c26095730 0x559c264570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:33.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.798+0000 7feb21015740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559c263bcb00 con 0x559c260b6410 2024-04-25T11:41:33.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.799+0000 7feb21015740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x559c26358920 con 0x559c26095730 2024-04-25T11:41:33.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.799+0000 7feb21015740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559c263b8ac0 con 0x559c2608d550 2024-04-25T11:41:33.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.799+0000 7feb13fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c260b6410 0x559c26456620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:33.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.799+0000 7feb13fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c260b6410 0x559c26456620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38320/0 (socket says 172.21.15.90:38320) 2024-04-25T11:41:33.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.799+0000 7feb13fff640 1 -- 172.21.15.90:0/4189978445 learned_addr learned my addr 172.21.15.90:0/4189978445 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:33.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.799+0000 7feb12ffd640 1 --2- 172.21.15.90:0/4189978445 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c26095730 0x559c264570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:33.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.799+0000 7feb10ff9640 1 -- 172.21.15.90:0/56146794 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7feb04004020 con 0x7feb14102070 2024-04-25T11:41:33.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.799+0000 7feb13fff640 1 -- 172.21.15.90:0/4189978445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c2608d550 msgr2=0x559c26456b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:33.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.799+0000 7feb13fff640 1 --2- 172.21.15.90:0/4189978445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c2608d550 0x559c26456b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.799+0000 7feb13fff640 1 -- 172.21.15.90:0/4189978445 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c26095730 msgr2=0x559c264570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:33.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.799+0000 7feb13fff640 1 --2- 172.21.15.90:0/4189978445 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c26095730 0x559c264570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.799+0000 7feb13fff640 1 -- 172.21.15.90:0/4189978445 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c261967e0 con 0x559c260b6410 2024-04-25T11:41:33.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.799+0000 7feb13fff640 1 --2- 172.21.15.90:0/4189978445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c260b6410 0x559c26456620 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7feb0c005cc0 tx=0x7feb0c02f420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8052295a22eb512f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:33.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.799+0000 7feaf9ffb640 1 -- 172.21.15.90:0/4189978445 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c031cf0 con 0x559c260b6410 2024-04-25T11:41:33.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.799+0000 7feaf9ffb640 1 -- 172.21.15.90:0/4189978445 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb0c036040 con 0x559c260b6410 2024-04-25T11:41:33.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.799+0000 7feb10ff9640 1 --2- 172.21.15.90:0/56146794 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7feafc0afc60 0x7feafc0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:33.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.799+0000 7feaf9ffb640 1 -- 172.21.15.90:0/4189978445 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c004040 con 0x559c260b6410 2024-04-25T11:41:33.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.800+0000 7feb137fe640 1 --2- 172.21.15.90:0/56146794 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7feafc0afc60 0x7feafc0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:33.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.800+0000 7feb10ff9640 1 -- 172.21.15.90:0/56146794 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7feb040df3e0 con 0x7feb14102070 2024-04-25T11:41:33.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.800+0000 7feb21015740 1 -- 172.21.15.90:0/4189978445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c260b6410 msgr2=0x559c26456620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:33.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.800+0000 7feb137fe640 1 --2- 172.21.15.90:0/56146794 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7feafc0afc60 0x7feafc0b2170 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x559c26454110 tx=0x7feb0404b040 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:33.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.800+0000 7feb21015740 1 --2- 172.21.15.90:0/4189978445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c260b6410 0x559c26456620 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7feb0c005cc0 tx=0x7feb0c02f420 comp rx=0 tx=0).stop 2024-04-25T11:41:33.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.800+0000 7feb21015740 1 -- 172.21.15.90:0/4189978445 shutdown_connections 2024-04-25T11:41:33.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.801+0000 7feb21015740 1 --2- 172.21.15.90:0/4189978445 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c26095730 0x559c264570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.801+0000 7feb21015740 1 --2- 172.21.15.90:0/4189978445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c2608d550 0x559c26456b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.801+0000 7feb21015740 1 --2- 172.21.15.90:0/4189978445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c260b6410 0x559c26456620 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.801+0000 7feb21015740 1 -- 172.21.15.90:0/4189978445 >> 172.21.15.90:0/4189978445 conn(0x559c260aa560 msgr2=0x559c26452dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:33.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.801+0000 7feb21015740 1 -- 172.21.15.90:0/4189978445 shutdown_connections 2024-04-25T11:41:33.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.801+0000 7feb21015740 1 -- 172.21.15.90:0/4189978445 wait complete. 2024-04-25T11:41:33.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.801+0000 7feb21015740 1 -- start start 2024-04-25T11:41:33.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.801+0000 7feb21015740 1 Processor -- start 2024-04-25T11:41:33.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.802+0000 7feb21015740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c2608d550 0x559c26460bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:33.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.802+0000 7feb21015740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c26095730 0x559c264610f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:33.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.802+0000 7feb21015740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c260b6410 0x559c2646cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:33.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.802+0000 7feb21015740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559c26463910 con 0x559c2608d550 2024-04-25T11:41:33.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.802+0000 7feb21015740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x559c263565d0 con 0x559c260b6410 2024-04-25T11:41:33.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.802+0000 7feb21015740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559c26463a90 con 0x559c26095730 2024-04-25T11:41:33.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.802+0000 7feb21015740 10 client.0 _ll_register_callbacks cb 0x559c26344080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:33.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.802+0000 7feb13fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c2608d550 0x559c26460bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:33.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.802+0000 7feb13fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c2608d550 0x559c26460bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38336/0 (socket says 172.21.15.90:38336) 2024-04-25T11:41:33.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.802+0000 7feb13fff640 1 -- 172.21.15.90:0/4036467670 learned_addr learned my addr 172.21.15.90:0/4036467670 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:33.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.802+0000 7feb13fff640 1 --2- 172.21.15.90:0/4036467670 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c260b6410 0x559c2646cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:33.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.802+0000 7feb12ffd640 1 --2- 172.21.15.90:0/4036467670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c26095730 0x559c264610f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:33.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.803+0000 7feb13fff640 1 -- 172.21.15.90:0/4036467670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c26095730 msgr2=0x559c264610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:33.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.803+0000 7feb13fff640 1 --2- 172.21.15.90:0/4036467670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c26095730 0x559c264610f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.803+0000 7feb13fff640 1 -- 172.21.15.90:0/4036467670 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c260b6410 msgr2=0x559c2646cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:33.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.803+0000 7feb13fff640 1 --2- 172.21.15.90:0/4036467670 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c260b6410 0x559c2646cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.803+0000 7feb13fff640 1 -- 172.21.15.90:0/4036467670 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c261f6f70 con 0x559c2608d550 2024-04-25T11:41:33.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.803+0000 7feb13fff640 1 --2- 172.21.15.90:0/4036467670 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c2608d550 0x559c26460bb0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7feb0c0096c0 tx=0x7feb0c003c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e172e700f786522 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:33.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.803+0000 7feaf2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:41:33.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.803+0000 7feaf2ffd640 1 -- 172.21.15.90:0/4036467670 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c058060 con 0x559c2608d550 2024-04-25T11:41:33.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.803+0000 7feaf2ffd640 1 -- 172.21.15.90:0/4036467670 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7feb0c031650 con 0x559c2608d550 2024-04-25T11:41:33.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.803+0000 7feb21015740 1 -- 172.21.15.90:0/4036467670 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb0c036040 con 0x559c2608d550 2024-04-25T11:41:33.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.803+0000 7feaf2ffd640 1 -- 172.21.15.90:0/4036467670 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7feb0c04a040 con 0x559c2608d550 2024-04-25T11:41:33.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.803+0000 7feb21015740 10 client.17568 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:33.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.803+0000 7feb21015740 1 -- 172.21.15.90:0/4036467670 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559c2608e170 con 0x559c2608d550 2024-04-25T11:41:33.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.803+0000 7feaf2ffd640 1 -- 172.21.15.90:0/4036467670 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb0c036040 con 0x559c2608d550 2024-04-25T11:41:33.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.804+0000 7feaf2ffd640 1 -- 172.21.15.90:0/4036467670 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7feb0c050050 con 0x559c2608d550 2024-04-25T11:41:33.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.804+0000 7feb21015740 10 client.17568 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:33.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.804+0000 7feb21015740 10 client.17568 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:41:33.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.804+0000 7feb21015740 20 client.17568 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:33.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.804+0000 7feb21015740 1 --2- 172.21.15.90:0/4036467670 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559c260751f0 0x559c26472d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:33.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.804+0000 7feb21015740 4 client.17568 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:41:33.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.804+0000 7feb21015740 1 -- 172.21.15.90:0/4036467670 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c263ca710 con 0x559c260751f0 2024-04-25T11:41:33.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.804+0000 7feb12ffd640 1 --2- 172.21.15.90:0/4036467670 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559c260751f0 0x559c26472d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:33.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.804+0000 7feaf2ffd640 10 client.17568 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:33.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.804+0000 7feb12ffd640 1 --2- 172.21.15.90:0/4036467670 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559c260751f0 0x559c26472d90 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc1b32f8eaf588d4 server_cookie=c14c95aa90bcd98d in_seq=0 out_seq=0 2024-04-25T11:41:33.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.805+0000 7feaf2ffd640 1 -- 172.21.15.90:0/4036467670 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559c263ca710 con 0x559c260751f0 2024-04-25T11:41:33.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.805+0000 7feaf2ffd640 10 client.17568 handle_command_reply: tid=0 2024-04-25T11:41:33.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.805+0000 7feb21015740 1 client.17568 shutdown 2024-04-25T11:41:33.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.805+0000 7feb21015740 1 -- 172.21.15.90:0/4036467670 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c2608d550 msgr2=0x559c26460bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:33.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.805+0000 7feb21015740 1 --2- 172.21.15.90:0/4036467670 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c2608d550 0x559c26460bb0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7feb0c0096c0 tx=0x7feb0c003c30 comp rx=0 tx=0).stop 2024-04-25T11:41:33.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.805+0000 7feb21015740 1 -- 172.21.15.90:0/4036467670 shutdown_connections 2024-04-25T11:41:33.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.805+0000 7feb21015740 1 --2- 172.21.15.90:0/4036467670 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559c260751f0 0x559c26472d90 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.805+0000 7feb21015740 1 --2- 172.21.15.90:0/4036467670 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c260b6410 0x559c2646cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.806+0000 7feb21015740 1 --2- 172.21.15.90:0/4036467670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c26095730 0x559c264610f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.806+0000 7feb21015740 1 --2- 172.21.15.90:0/4036467670 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c2608d550 0x559c26460bb0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.806+0000 7feaf2ffd640 0 client.17568 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:33.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.806+0000 7feb21015740 1 -- 172.21.15.90:0/4036467670 >> 172.21.15.90:0/4036467670 conn(0x559c260aa560 msgr2=0x559c264563f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:33.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.806+0000 7feb21015740 1 -- 172.21.15.90:0/4036467670 shutdown_connections 2024-04-25T11:41:33.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.806+0000 7feb21015740 1 -- 172.21.15.90:0/4036467670 wait complete. 2024-04-25T11:41:33.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.806+0000 7feb21015740 20 client.17568 trim_cache size 0 max 16384 2024-04-25T11:41:33.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:33 smithi138 ceph-mon[32328]: pgmap v197: 97 pgs: 97 active+clean; 59 GiB data, 178 GiB used, 895 GiB / 1.0 TiB avail; 3.0 MiB/s rd, 662 MiB/s wr, 796 op/s 2024-04-25T11:41:33.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.859+0000 7feb21015740 1 Processor -- start 2024-04-25T11:41:33.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.860+0000 7feb21015740 1 -- start start 2024-04-25T11:41:33.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.861+0000 7feb21015740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c2608d550 0x559c2645b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:33.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.861+0000 7feb21015740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c26095730 0x559c26456de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:33.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.861+0000 7feb21015740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c260b6410 0x559c2646cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:33.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.862+0000 7feb21015740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7feb141afdc0 con 0x559c26095730 2024-04-25T11:41:33.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.862+0000 7feb21015740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7feb141b1900 con 0x559c260b6410 2024-04-25T11:41:33.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.862+0000 7feb21015740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7feb141b0490 con 0x559c2608d550 2024-04-25T11:41:33.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.863+0000 7feb12ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c26095730 0x559c26456de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:33.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.863+0000 7feb12ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c26095730 0x559c26456de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38346/0 (socket says 172.21.15.90:38346) 2024-04-25T11:41:33.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.863+0000 7feb12ffd640 1 -- 172.21.15.90:0/639442832 learned_addr learned my addr 172.21.15.90:0/639442832 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:33.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.870+0000 7feb13fff640 1 --2- 172.21.15.90:0/639442832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c2608d550 0x559c2645b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:33.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.870+0000 7feb13fff640 1 --2- 172.21.15.90:0/639442832 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c260b6410 0x559c2646cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.874+0000 7feb12ffd640 1 -- 172.21.15.90:0/639442832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c2608d550 msgr2=0x559c2645b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:33.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.874+0000 7feb12ffd640 1 --2- 172.21.15.90:0/639442832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c2608d550 0x559c2645b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.874+0000 7feb12ffd640 1 -- 172.21.15.90:0/639442832 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c260b6410 msgr2=0x559c2646cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:33.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.874+0000 7feb12ffd640 1 --2- 172.21.15.90:0/639442832 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c260b6410 0x559c2646cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.874+0000 7feb12ffd640 1 -- 172.21.15.90:0/639442832 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c264618c0 con 0x559c26095730 2024-04-25T11:41:33.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.877+0000 7feb12ffd640 1 --2- 172.21.15.90:0/639442832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c26095730 0x559c26456de0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7feb1410f8c0 tx=0x7feb0800b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=338d04de9041fbfe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:33.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.878+0000 7feaf97fa640 1 -- 172.21.15.90:0/639442832 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb08026070 con 0x559c26095730 2024-04-25T11:41:33.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.878+0000 7feaf97fa640 1 -- 172.21.15.90:0/639442832 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb08020660 con 0x559c26095730 2024-04-25T11:41:33.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.880+0000 7feb21015740 1 -- 172.21.15.90:0/639442832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c26095730 msgr2=0x559c26456de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:33.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.880+0000 7feb21015740 1 --2- 172.21.15.90:0/639442832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c26095730 0x559c26456de0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7feb1410f8c0 tx=0x7feb0800b390 comp rx=0 tx=0).stop 2024-04-25T11:41:33.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.882+0000 7feb21015740 1 -- 172.21.15.90:0/639442832 shutdown_connections 2024-04-25T11:41:33.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.882+0000 7feb21015740 1 --2- 172.21.15.90:0/639442832 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c260b6410 0x559c2646cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.882+0000 7feb21015740 1 --2- 172.21.15.90:0/639442832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c26095730 0x559c26456de0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.882+0000 7feb21015740 1 --2- 172.21.15.90:0/639442832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c2608d550 0x559c2645b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.882+0000 7feb21015740 1 -- 172.21.15.90:0/639442832 >> 172.21.15.90:0/639442832 conn(0x559c260aa560 msgr2=0x559c26451b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:33.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.883+0000 7feb21015740 1 -- 172.21.15.90:0/639442832 shutdown_connections 2024-04-25T11:41:33.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.884+0000 7feb21015740 1 -- 172.21.15.90:0/639442832 wait complete. 2024-04-25T11:41:33.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.884+0000 7feb21015740 1 -- start start 2024-04-25T11:41:33.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.885+0000 7feb21015740 1 Processor -- start 2024-04-25T11:41:33.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.886+0000 7feb21015740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c2608d550 0x559c26475d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:33.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.886+0000 7feb21015740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c26095730 0x559c26476260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:33.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.886+0000 7feb21015740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c260b6410 0x559c26473310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:33.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.886+0000 7feb21015740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7feb141b1230 con 0x559c26095730 2024-04-25T11:41:33.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.886+0000 7feb21015740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7feb141b0b60 con 0x559c260b6410 2024-04-25T11:41:33.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.886+0000 7feb21015740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7feb141adda0 con 0x559c2608d550 2024-04-25T11:41:33.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.886+0000 7feb21015740 10 client.0 _ll_register_callbacks cb 0x7feb0c031d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:33.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.888+0000 7feb12ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c26095730 0x559c26476260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:33.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.889+0000 7feb12ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c26095730 0x559c26476260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38352/0 (socket says 172.21.15.90:38352) 2024-04-25T11:41:33.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.889+0000 7feb12ffd640 1 -- 172.21.15.90:0/1690082441 learned_addr learned my addr 172.21.15.90:0/1690082441 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:33.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.889+0000 7feb13fff640 1 --2- 172.21.15.90:0/1690082441 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c260b6410 0x559c26473310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:33.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.889+0000 7feb13fff640 1 --2- 172.21.15.90:0/1690082441 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c2608d550 0x559c26475d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:33.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.889+0000 7feb12ffd640 1 -- 172.21.15.90:0/1690082441 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c2608d550 msgr2=0x559c26475d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:33.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.890+0000 7feb12ffd640 1 --2- 172.21.15.90:0/1690082441 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c2608d550 0x559c26475d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.890+0000 7feb12ffd640 1 -- 172.21.15.90:0/1690082441 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c260b6410 msgr2=0x559c26473310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:33.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.890+0000 7feb12ffd640 1 --2- 172.21.15.90:0/1690082441 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c260b6410 0x559c26473310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.890+0000 7feb12ffd640 1 -- 172.21.15.90:0/1690082441 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c2645c560 con 0x559c26095730 2024-04-25T11:41:33.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.890+0000 7feb12ffd640 1 --2- 172.21.15.90:0/1690082441 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c26095730 0x559c26476260 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7feb08024040 tx=0x7feb08003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6d8bd604391e824 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:33.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.891+0000 7feaf2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:41:33.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.892+0000 7feaf2ffd640 1 -- 172.21.15.90:0/1690082441 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb08026040 con 0x559c26095730 2024-04-25T11:41:33.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.892+0000 7feb21015740 1 -- 172.21.15.90:0/1690082441 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb08020660 con 0x559c26095730 2024-04-25T11:41:33.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.892+0000 7feaf2ffd640 1 -- 172.21.15.90:0/1690082441 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7feb08020810 con 0x559c26095730 2024-04-25T11:41:33.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.893+0000 7feaf2ffd640 1 -- 172.21.15.90:0/1690082441 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7feb08020a10 con 0x559c26095730 2024-04-25T11:41:33.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.893+0000 7feb21015740 10 client.17574 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:33.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.893+0000 7feb21015740 1 -- 172.21.15.90:0/1690082441 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559c264d4490 con 0x559c26095730 2024-04-25T11:41:33.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.896+0000 7feaf2ffd640 1 -- 172.21.15.90:0/1690082441 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb08020660 con 0x559c26095730 2024-04-25T11:41:33.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.897+0000 7feaf2ffd640 1 -- 172.21.15.90:0/1690082441 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7feb0802a6d0 con 0x559c26095730 2024-04-25T11:41:33.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.899+0000 7feb21015740 10 client.17574 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:33.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.899+0000 7feb21015740 10 client.17574 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:41:33.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.899+0000 7feb21015740 20 client.17574 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:33.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.899+0000 7feb21015740 1 --2- 172.21.15.90:0/1690082441 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559c2604a630 0x559c264d7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:33.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.899+0000 7feb21015740 4 client.17574 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T11:41:33.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.899+0000 7feb21015740 1 -- 172.21.15.90:0/1690082441 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559c263c4580 con 0x559c2604a630 2024-04-25T11:41:33.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.905+0000 7feb13fff640 1 --2- 172.21.15.90:0/1690082441 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559c2604a630 0x559c264d7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:33.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.919+0000 7feb13fff640 1 --2- 172.21.15.90:0/1690082441 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559c2604a630 0x559c264d7280 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfcd341272f956c7 server_cookie=31001c8bb8a45862 in_seq=0 out_seq=0 2024-04-25T11:41:33.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.919+0000 7feaf2ffd640 10 client.17574 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:33.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.925+0000 7feaf2ffd640 1 -- 172.21.15.90:0/1690082441 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559c263c4580 con 0x559c2604a630 2024-04-25T11:41:33.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.925+0000 7feaf2ffd640 10 client.17574 handle_command_reply: tid=0 2024-04-25T11:41:33.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.925+0000 7feb21015740 1 client.17574 shutdown 2024-04-25T11:41:33.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.931+0000 7feb21015740 1 -- 172.21.15.90:0/1690082441 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c26095730 msgr2=0x559c26476260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:33.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.931+0000 7feb21015740 1 --2- 172.21.15.90:0/1690082441 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c26095730 0x559c26476260 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7feb08024040 tx=0x7feb08003f90 comp rx=0 tx=0).stop 2024-04-25T11:41:33.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.931+0000 7feb21015740 1 -- 172.21.15.90:0/1690082441 shutdown_connections 2024-04-25T11:41:33.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.931+0000 7feb21015740 1 --2- 172.21.15.90:0/1690082441 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559c2604a630 0x559c264d7280 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.931+0000 7feb21015740 1 --2- 172.21.15.90:0/1690082441 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559c260b6410 0x559c26473310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.932+0000 7feb21015740 1 --2- 172.21.15.90:0/1690082441 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559c26095730 0x559c26476260 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.932+0000 7feb21015740 1 --2- 172.21.15.90:0/1690082441 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c2608d550 0x559c26475d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.932+0000 7feb21015740 1 -- 172.21.15.90:0/1690082441 >> 172.21.15.90:0/1690082441 conn(0x559c260aa560 msgr2=0x559c26460230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:33.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.932+0000 7feaf2ffd640 0 client.17574 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:33.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.936+0000 7feb21015740 1 -- 172.21.15.90:0/1690082441 shutdown_connections 2024-04-25T11:41:33.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.939+0000 7feb21015740 1 -- 172.21.15.90:0/1690082441 wait complete. 2024-04-25T11:41:33.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.939+0000 7feb21015740 20 client.17574 trim_cache size 0 max 16384 2024-04-25T11:41:33.931 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:41:33.931 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-25T11:41:33.931 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "ebcb6268-39f5-4327-ac0e-d7d98cfcdf44", 2024-04-25T11:41:33.931 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-25T11:41:33.931 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:41:33.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.945+0000 7feb1a164640 1 -- 172.21.15.90:0/56146794 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7feafc0afc60 msgr2=0x7feafc0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:33.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.945+0000 7feb1a164640 1 --2- 172.21.15.90:0/56146794 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7feafc0afc60 0x7feafc0b2170 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x559c26454110 tx=0x7feb0404b040 comp rx=0 tx=0).stop 2024-04-25T11:41:33.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.945+0000 7feb1a164640 1 -- 172.21.15.90:0/56146794 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feb14102070 msgr2=0x7feb1419a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:33.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.945+0000 7feb1a164640 1 --2- 172.21.15.90:0/56146794 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feb14102070 0x7feb1419a7b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7feb0400e9f0 tx=0x7feb0400eeb0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.947+0000 7feb1a164640 1 -- 172.21.15.90:0/56146794 shutdown_connections 2024-04-25T11:41:33.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.947+0000 7feb1a164640 1 --2- 172.21.15.90:0/56146794 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7feafc0afc60 0x7feafc0b2170 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.947+0000 7feb1a164640 1 --2- 172.21.15.90:0/56146794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb1410e570 0x7feb1419efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.947+0000 7feb1a164640 1 --2- 172.21.15.90:0/56146794 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb14109bc0 0x7feb1419ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.947+0000 7feb1a164640 1 --2- 172.21.15.90:0/56146794 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feb14102070 0x7feb1419a7b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:33.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.947+0000 7feb1a164640 1 -- 172.21.15.90:0/56146794 >> 172.21.15.90:0/56146794 conn(0x7feb14077920 msgr2=0x7feb1410a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:33.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.949+0000 7feb1a164640 1 -- 172.21.15.90:0/56146794 shutdown_connections 2024-04-25T11:41:33.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:33.949+0000 7feb1a164640 1 -- 172.21.15.90:0/56146794 wait complete. 2024-04-25T11:41:33.969 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:41:33.970 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:41:34.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.188+0000 7f9fa9613640 1 -- 172.21.15.90:0/2255144730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9fa410d3a0 msgr2=0x7f9fa410f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:34.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.188+0000 7f9fa9613640 1 --2- 172.21.15.90:0/2255144730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9fa410d3a0 0x7f9fa410f7c0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c009ca0 tx=0x7f9f9c02a8c0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.189+0000 7f9fa9613640 1 -- 172.21.15.90:0/2255144730 shutdown_connections 2024-04-25T11:41:34.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.189+0000 7f9fa9613640 1 --2- 172.21.15.90:0/2255144730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9fa410d3a0 0x7f9fa410f7c0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.189+0000 7f9fa9613640 1 --2- 172.21.15.90:0/2255144730 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fa4102b10 0x7f9fa4102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.189+0000 7f9fa9613640 1 --2- 172.21.15.90:0/2255144730 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9fa4108a40 0x7f9fa41025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.189+0000 7f9fa9613640 1 -- 172.21.15.90:0/2255144730 >> 172.21.15.90:0/2255144730 conn(0x7f9fa4078180 msgr2=0x7f9fa40785c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:34.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.189+0000 7f9fa9613640 1 -- 172.21.15.90:0/2255144730 shutdown_connections 2024-04-25T11:41:34.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.189+0000 7f9fa9613640 1 -- 172.21.15.90:0/2255144730 wait complete. 2024-04-25T11:41:34.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.190+0000 7f9fa9613640 1 Processor -- start 2024-04-25T11:41:34.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.190+0000 7f9fa9613640 1 -- start start 2024-04-25T11:41:34.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.190+0000 7f9fa9613640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9fa4102b10 0x7f9fa419a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.190+0000 7f9fa9613640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fa4108a40 0x7f9fa419ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.190+0000 7f9fa9613640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9fa410d3a0 0x7f9fa419ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.190+0000 7f9fa9613640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa4112c20 con 0x7f9fa410d3a0 2024-04-25T11:41:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.190+0000 7f9fa9613640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa4112aa0 con 0x7f9fa4102b10 2024-04-25T11:41:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.190+0000 7f9fa9613640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa4112da0 con 0x7f9fa4108a40 2024-04-25T11:41:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.190+0000 7f9fa2ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9fa4102b10 0x7f9fa419a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.190+0000 7f9fa37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9fa410d3a0 0x7f9fa419ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.190+0000 7f9fa2ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9fa4102b10 0x7f9fa419a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:58432/0 (socket says 172.21.15.90:58432) 2024-04-25T11:41:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.190+0000 7f9fa2ffd640 1 -- 172.21.15.90:0/1663117181 learned_addr learned my addr 172.21.15.90:0/1663117181 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.191+0000 7f9f9a7fc640 1 --2- 172.21.15.90:0/1663117181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fa4108a40 0x7f9fa419ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.191+0000 7f9fa2ffd640 1 -- 172.21.15.90:0/1663117181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fa4108a40 msgr2=0x7f9fa419ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.191+0000 7f9fa2ffd640 1 --2- 172.21.15.90:0/1663117181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fa4108a40 0x7f9fa419ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.191+0000 7f9fa2ffd640 1 -- 172.21.15.90:0/1663117181 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9fa410d3a0 msgr2=0x7f9fa419ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:34.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.191+0000 7f9fa2ffd640 1 --2- 172.21.15.90:0/1663117181 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9fa410d3a0 0x7f9fa419ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.191+0000 7f9fa2ffd640 1 -- 172.21.15.90:0/1663117181 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fa419f4f0 con 0x7f9fa4102b10 2024-04-25T11:41:34.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.191+0000 7f9fa37fe640 1 --2- 172.21.15.90:0/1663117181 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9fa410d3a0 0x7f9fa419ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:34.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.191+0000 7f9fa2ffd640 1 --2- 172.21.15.90:0/1663117181 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9fa4102b10 0x7f9fa419a710 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9f9400c970 tx=0x7f9f9400ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:34.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.191+0000 7f9fa0ff9640 1 -- 172.21.15.90:0/1663117181 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f94014070 con 0x7f9fa4102b10 2024-04-25T11:41:34.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.191+0000 7f9fa9613640 1 -- 172.21.15.90:0/1663117181 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fa419f7e0 con 0x7f9fa4102b10 2024-04-25T11:41:34.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.191+0000 7f9fa9613640 1 -- 172.21.15.90:0/1663117181 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fa41a71e0 con 0x7f9fa4102b10 2024-04-25T11:41:34.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.191+0000 7f9fa0ff9640 1 -- 172.21.15.90:0/1663117181 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f94004440 con 0x7f9fa4102b10 2024-04-25T11:41:34.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.192+0000 7f9fa0ff9640 1 -- 172.21.15.90:0/1663117181 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f94003e40 con 0x7f9fa4102b10 2024-04-25T11:41:34.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.194+0000 7f9fb059f740 1 Processor -- start 2024-04-25T11:41:34.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.194+0000 7f9fb059f740 1 -- start start 2024-04-25T11:41:34.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.194+0000 7f9fa0ff9640 1 -- 172.21.15.90:0/1663117181 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f9f94017070 con 0x7f9fa4102b10 2024-04-25T11:41:34.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.195+0000 7f9fb059f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55571db82390 0x55571df22a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:34.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.195+0000 7f9fb059f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55571db594d0 0x55571df22fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:34.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.195+0000 7f9fb059f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db616b0 0x55571df2f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:34.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.195+0000 7f9fb059f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55571de88d80 con 0x55571db616b0 2024-04-25T11:41:34.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.195+0000 7f9fb059f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55571de428b0 con 0x55571db82390 2024-04-25T11:41:34.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.195+0000 7f9fb059f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55571de84d40 con 0x55571db594d0 2024-04-25T11:41:34.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.195+0000 7f9f9a7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db616b0 0x55571df2f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:34.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.195+0000 7f9fa37fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55571db82390 0x55571df22a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:34.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.195+0000 7f9fa2ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55571db594d0 0x55571df22fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:34.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.195+0000 7f9f9a7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db616b0 0x55571df2f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38378/0 (socket says 172.21.15.90:38378) 2024-04-25T11:41:34.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.195+0000 7f9f9a7fc640 1 -- 172.21.15.90:0/1564659511 learned_addr learned my addr 172.21.15.90:0/1564659511 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:34.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.195+0000 7f9fa37fe640 1 --2- 172.21.15.90:0/1564659511 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55571db82390 0x55571df22a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:58446/0 (socket says 172.21.15.90:58446) 2024-04-25T11:41:34.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.195+0000 7f9fa2ffd640 1 --2- 172.21.15.90:0/1564659511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55571db594d0 0x55571df22fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:37200/0 (socket says 172.21.15.90:37200) 2024-04-25T11:41:34.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.195+0000 7f9f9a7fc640 1 -- 172.21.15.90:0/1564659511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55571db594d0 msgr2=0x55571df22fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:34.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.195+0000 7f9f9a7fc640 1 --2- 172.21.15.90:0/1564659511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55571db594d0 0x55571df22fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.195+0000 7f9f9a7fc640 1 -- 172.21.15.90:0/1564659511 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55571db82390 msgr2=0x55571df22a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:34.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.195+0000 7f9f9a7fc640 1 --2- 172.21.15.90:0/1564659511 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55571db82390 0x55571df22a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.195+0000 7f9f9a7fc640 1 -- 172.21.15.90:0/1564659511 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55571de13a40 con 0x55571db616b0 2024-04-25T11:41:34.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.195+0000 7f9fa37fe640 1 --2- 172.21.15.90:0/1564659511 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55571db82390 0x55571df22a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:34.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.195+0000 7f9fa2ffd640 1 --2- 172.21.15.90:0/1564659511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55571db594d0 0x55571df22fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:34.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.195+0000 7f9fa0ff9640 1 --2- 172.21.15.90:0/1663117181 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9f900afce0 0x7f9f900b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:34.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.196+0000 7f9f9a7fc640 1 --2- 172.21.15.90:0/1564659511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db616b0 0x55571df2f8f0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c009c20 tx=0x7f9f8c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fcb05f814c052c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:34.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.196+0000 7f9fa37fe640 1 --2- 172.21.15.90:0/1663117181 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9f900afce0 0x7f9f900b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:34.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.196+0000 7f9fa0ff9640 1 -- 172.21.15.90:0/1663117181 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f9f940de880 con 0x7f9fa4102b10 2024-04-25T11:41:34.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.196+0000 7f9f997fa640 1 -- 172.21.15.90:0/1564659511 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c004230 con 0x55571db616b0 2024-04-25T11:41:34.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.196+0000 7f9f997fa640 1 -- 172.21.15.90:0/1564659511 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f8c0043d0 con 0x55571db616b0 2024-04-25T11:41:34.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.196+0000 7f9f997fa640 1 -- 172.21.15.90:0/1564659511 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c019660 con 0x55571db616b0 2024-04-25T11:41:34.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.197+0000 7f9fb059f740 1 -- 172.21.15.90:0/1564659511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db616b0 msgr2=0x55571df2f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:34.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.197+0000 7f9fa37fe640 1 --2- 172.21.15.90:0/1663117181 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9f900afce0 0x7f9f900b21f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x55571df1fbd0 tx=0x7f9f9c02f310 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:34.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.197+0000 7f9fb059f740 1 --2- 172.21.15.90:0/1564659511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db616b0 0x55571df2f8f0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c009c20 tx=0x7f9f8c011810 comp rx=0 tx=0).stop 2024-04-25T11:41:34.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.197+0000 7f9fb059f740 1 -- 172.21.15.90:0/1564659511 shutdown_connections 2024-04-25T11:41:34.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.197+0000 7f9fb059f740 1 --2- 172.21.15.90:0/1564659511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db616b0 0x55571df2f8f0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.197+0000 7f9fb059f740 1 --2- 172.21.15.90:0/1564659511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55571db594d0 0x55571df22fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.197+0000 7f9fb059f740 1 --2- 172.21.15.90:0/1564659511 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55571db82390 0x55571df22a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.197+0000 7f9fb059f740 1 -- 172.21.15.90:0/1564659511 >> 172.21.15.90:0/1564659511 conn(0x55571db764e0 msgr2=0x55571df1f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:34.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.197+0000 7f9fb059f740 1 -- 172.21.15.90:0/1564659511 shutdown_connections 2024-04-25T11:41:34.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.197+0000 7f9fb059f740 1 -- 172.21.15.90:0/1564659511 wait complete. 2024-04-25T11:41:34.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.197+0000 7f9fb059f740 1 -- start start 2024-04-25T11:41:34.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.197+0000 7f9fb059f740 1 Processor -- start 2024-04-25T11:41:34.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.198+0000 7f9fb059f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db594d0 0x55571df2d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:34.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.198+0000 7f9fb059f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55571db616b0 0x55571df2da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:34.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.198+0000 7f9fb059f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55571db82390 0x55571df39550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:34.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.198+0000 7f9fb059f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55571df2ffe0 con 0x55571db594d0 2024-04-25T11:41:34.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.198+0000 7f9fb059f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55571de0fd10 con 0x55571db616b0 2024-04-25T11:41:34.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.198+0000 7f9fb059f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55571df30160 con 0x55571db82390 2024-04-25T11:41:34.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.198+0000 7f9fb059f740 10 client.0 _ll_register_callbacks cb 0x55571de3c210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:34.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.198+0000 7f9fa2ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db594d0 0x55571df2d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:34.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.198+0000 7f9fa2ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db594d0 0x55571df2d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38380/0 (socket says 172.21.15.90:38380) 2024-04-25T11:41:34.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.198+0000 7f9fa2ffd640 1 -- 172.21.15.90:0/3312957195 learned_addr learned my addr 172.21.15.90:0/3312957195 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:34.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.198+0000 7f9f9a7fc640 1 --2- 172.21.15.90:0/3312957195 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55571db616b0 0x55571df2da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:34.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.198+0000 7f9fa37fe640 1 --2- 172.21.15.90:0/3312957195 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55571db82390 0x55571df39550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:34.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.198+0000 7f9fa2ffd640 1 -- 172.21.15.90:0/3312957195 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55571db82390 msgr2=0x55571df39550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:34.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.198+0000 7f9fa2ffd640 1 --2- 172.21.15.90:0/3312957195 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55571db82390 0x55571df39550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.198+0000 7f9fa2ffd640 1 -- 172.21.15.90:0/3312957195 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55571db616b0 msgr2=0x55571df2da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:34.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.198+0000 7f9fa2ffd640 1 --2- 172.21.15.90:0/3312957195 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55571db616b0 0x55571df2da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.198+0000 7f9fa2ffd640 1 -- 172.21.15.90:0/3312957195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55571dc647d0 con 0x55571db594d0 2024-04-25T11:41:34.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.198+0000 7f9fa37fe640 1 --2- 172.21.15.90:0/3312957195 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55571db82390 0x55571df39550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:41:34.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.198+0000 7f9f9a7fc640 1 --2- 172.21.15.90:0/3312957195 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55571db616b0 0x55571df2da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:34.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.198+0000 7f9fa2ffd640 1 --2- 172.21.15.90:0/3312957195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db594d0 0x55571df2d540 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f9f940dbfd0 tx=0x7f9f9400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d31734fde4efd60 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:34.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.198+0000 7f9f8a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:41:34.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.198+0000 7f9f8a7fc640 1 -- 172.21.15.90:0/3312957195 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f940a6310 con 0x55571db594d0 2024-04-25T11:41:34.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.199+0000 7f9fb059f740 1 -- 172.21.15.90:0/3312957195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f8c0043d0 con 0x55571db594d0 2024-04-25T11:41:34.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.199+0000 7f9f8a7fc640 1 -- 172.21.15.90:0/3312957195 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f9f940af390 con 0x55571db594d0 2024-04-25T11:41:34.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.199+0000 7f9f8a7fc640 1 -- 172.21.15.90:0/3312957195 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9f940a6130 con 0x55571db594d0 2024-04-25T11:41:34.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.199+0000 7f9fb059f740 10 client.17586 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:34.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.199+0000 7f9fb059f740 1 -- 172.21.15.90:0/3312957195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55571db9a3f0 con 0x55571db594d0 2024-04-25T11:41:34.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.199+0000 7f9f8a7fc640 1 -- 172.21.15.90:0/3312957195 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f8c0043d0 con 0x55571db594d0 2024-04-25T11:41:34.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.199+0000 7f9f8a7fc640 1 -- 172.21.15.90:0/3312957195 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f9f940a2330 con 0x55571db594d0 2024-04-25T11:41:34.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.199+0000 7f9fb059f740 10 client.17586 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:34.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.199+0000 7f9fb059f740 10 client.17586 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:41:34.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.199+0000 7f9fb059f740 20 client.17586 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:34.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.199+0000 7f9fb059f740 1 --2- 172.21.15.90:0/3312957195 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55571db40f60 0x55571df3f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:34.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.199+0000 7f9fb059f740 4 client.17586 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:41:34.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.199+0000 7f9fb059f740 1 -- 172.21.15.90:0/3312957195 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55571de96990 con 0x55571db40f60 2024-04-25T11:41:34.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.200+0000 7f9f9a7fc640 1 --2- 172.21.15.90:0/3312957195 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55571db40f60 0x55571df3f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:34.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.200+0000 7f9f8a7fc640 10 client.17586 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:34.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.200+0000 7f9f9a7fc640 1 --2- 172.21.15.90:0/3312957195 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55571db40f60 0x55571df3f6c0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d09a3cccb9b59f13 server_cookie=59bc7ed365644648 in_seq=0 out_seq=0 2024-04-25T11:41:34.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.201+0000 7f9f8a7fc640 1 -- 172.21.15.90:0/3312957195 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55571de96990 con 0x55571db40f60 2024-04-25T11:41:34.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.201+0000 7f9f8a7fc640 10 client.17586 handle_command_reply: tid=0 2024-04-25T11:41:34.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.201+0000 7f9fb059f740 1 client.17586 shutdown 2024-04-25T11:41:34.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.201+0000 7f9fb059f740 1 -- 172.21.15.90:0/3312957195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db594d0 msgr2=0x55571df2d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:34.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.201+0000 7f9fb059f740 1 --2- 172.21.15.90:0/3312957195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db594d0 0x55571df2d540 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f9f940dbfd0 tx=0x7f9f9400cf60 comp rx=0 tx=0).stop 2024-04-25T11:41:34.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.201+0000 7f9fb059f740 1 -- 172.21.15.90:0/3312957195 shutdown_connections 2024-04-25T11:41:34.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.201+0000 7f9fb059f740 1 --2- 172.21.15.90:0/3312957195 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55571db40f60 0x55571df3f6c0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.201+0000 7f9f8a7fc640 0 client.17586 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:34.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.201+0000 7f9fb059f740 1 --2- 172.21.15.90:0/3312957195 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55571db82390 0x55571df39550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.201+0000 7f9fb059f740 1 --2- 172.21.15.90:0/3312957195 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55571db616b0 0x55571df2da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.201+0000 7f9fb059f740 1 --2- 172.21.15.90:0/3312957195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db594d0 0x55571df2d540 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.201+0000 7f9fb059f740 1 -- 172.21.15.90:0/3312957195 >> 172.21.15.90:0/3312957195 conn(0x55571db764e0 msgr2=0x55571df24780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:34.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.201+0000 7f9fb059f740 1 -- 172.21.15.90:0/3312957195 shutdown_connections 2024-04-25T11:41:34.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.201+0000 7f9fb059f740 1 -- 172.21.15.90:0/3312957195 wait complete. 2024-04-25T11:41:34.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.202+0000 7f9fb059f740 20 client.17586 trim_cache size 0 max 16384 2024-04-25T11:41:34.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.221+0000 7f9fb059f740 1 Processor -- start 2024-04-25T11:41:34.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.221+0000 7f9fb059f740 1 -- start start 2024-04-25T11:41:34.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.222+0000 7f9fb059f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55571db40f60 0x55571df27680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:34.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.222+0000 7f9fb059f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55571db594d0 0x55571df27bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:34.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.222+0000 7f9fb059f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db616b0 0x55571df354a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:34.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.222+0000 7f9fb059f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8400e8a0 con 0x55571db616b0 2024-04-25T11:41:34.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.222+0000 7f9fb059f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9f840103e0 con 0x55571db40f60 2024-04-25T11:41:34.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.222+0000 7f9fb059f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8400ef70 con 0x55571db594d0 2024-04-25T11:41:34.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.222+0000 7f9fa37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db616b0 0x55571df354a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:34.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.222+0000 7f9fa37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db616b0 0x55571df354a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38388/0 (socket says 172.21.15.90:38388) 2024-04-25T11:41:34.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.222+0000 7f9fa2ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55571db40f60 0x55571df27680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:34.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.222+0000 7f9fa37fe640 1 -- 172.21.15.90:0/973691935 learned_addr learned my addr 172.21.15.90:0/973691935 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:34.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.222+0000 7f9f9a7fc640 1 --2- 172.21.15.90:0/973691935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55571db594d0 0x55571df27bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:34.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.222+0000 7f9fa37fe640 1 -- 172.21.15.90:0/973691935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55571db594d0 msgr2=0x55571df27bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:34.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.222+0000 7f9fa37fe640 1 --2- 172.21.15.90:0/973691935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55571db594d0 0x55571df27bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.222+0000 7f9fa37fe640 1 -- 172.21.15.90:0/973691935 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55571db40f60 msgr2=0x55571df27680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:34.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.222+0000 7f9fa37fe640 1 --2- 172.21.15.90:0/973691935 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55571db40f60 0x55571df27680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.222+0000 7f9fa37fe640 1 -- 172.21.15.90:0/973691935 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55571db5a0f0 con 0x55571db616b0 2024-04-25T11:41:34.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.222+0000 7f9fa2ffd640 1 --2- 172.21.15.90:0/973691935 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55571db40f60 0x55571df27680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:34.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.222+0000 7f9fa37fe640 1 --2- 172.21.15.90:0/973691935 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db616b0 0x55571df354a0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c009ca0 tx=0x7f9f9c042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74453a0288b3cbbb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:34.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.223+0000 7f9f98ff9640 1 -- 172.21.15.90:0/973691935 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c031830 con 0x55571db616b0 2024-04-25T11:41:34.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.223+0000 7f9f98ff9640 1 -- 172.21.15.90:0/973691935 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f9c036040 con 0x55571db616b0 2024-04-25T11:41:34.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.223+0000 7f9f98ff9640 1 -- 172.21.15.90:0/973691935 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c004300 con 0x55571db616b0 2024-04-25T11:41:34.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.224+0000 7f9fb059f740 1 -- 172.21.15.90:0/973691935 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db616b0 msgr2=0x55571df354a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:34.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.224+0000 7f9fb059f740 1 --2- 172.21.15.90:0/973691935 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db616b0 0x55571df354a0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c009ca0 tx=0x7f9f9c042000 comp rx=0 tx=0).stop 2024-04-25T11:41:34.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.224+0000 7f9fb059f740 1 -- 172.21.15.90:0/973691935 shutdown_connections 2024-04-25T11:41:34.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.224+0000 7f9fb059f740 1 --2- 172.21.15.90:0/973691935 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db616b0 0x55571df354a0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.224+0000 7f9fb059f740 1 --2- 172.21.15.90:0/973691935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55571db594d0 0x55571df27bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.224+0000 7f9fb059f740 1 --2- 172.21.15.90:0/973691935 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55571db40f60 0x55571df27680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.224+0000 7f9fb059f740 1 -- 172.21.15.90:0/973691935 >> 172.21.15.90:0/973691935 conn(0x55571db764e0 msgr2=0x55571df2cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:34.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.224+0000 7f9fb059f740 1 -- 172.21.15.90:0/973691935 shutdown_connections 2024-04-25T11:41:34.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.224+0000 7f9fb059f740 1 -- 172.21.15.90:0/973691935 wait complete. 2024-04-25T11:41:34.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.224+0000 7f9fb059f740 1 -- start start 2024-04-25T11:41:34.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.224+0000 7f9fb059f740 1 Processor -- start 2024-04-25T11:41:34.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.224+0000 7f9fb059f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55571db40f60 0x55571df30470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:34.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.224+0000 7f9fb059f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55571db594d0 0x55571df309b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:34.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.224+0000 7f9fb059f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db616b0 0x55571dfa59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:34.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.225+0000 7f9fb059f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8400fd10 con 0x55571db616b0 2024-04-25T11:41:34.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.225+0000 7f9fb059f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8400f640 con 0x55571db40f60 2024-04-25T11:41:34.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.225+0000 7f9fa2ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55571db40f60 0x55571df30470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:34.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.225+0000 7f9fa2ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55571db40f60 0x55571df30470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:58468/0 (socket says 172.21.15.90:58468) 2024-04-25T11:41:34.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.225+0000 7f9f9a7fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55571db594d0 0x55571df309b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:34.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.225+0000 7f9fa37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db616b0 0x55571dfa59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:34.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.225+0000 7f9fa2ffd640 1 -- 172.21.15.90:0/2142253211 learned_addr learned my addr 172.21.15.90:0/2142253211 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:34.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.225+0000 7f9fb059f740 1 -- 172.21.15.90:0/2142253211 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8400c880 con 0x55571db594d0 2024-04-25T11:41:34.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.225+0000 7f9fb059f740 10 client.0 _ll_register_callbacks cb 0x7f9f940e4240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:34.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.225+0000 7f9fa2ffd640 1 -- 172.21.15.90:0/2142253211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55571db594d0 msgr2=0x55571df309b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:34.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.225+0000 7f9fa2ffd640 1 --2- 172.21.15.90:0/2142253211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55571db594d0 0x55571df309b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.225+0000 7f9fa2ffd640 1 -- 172.21.15.90:0/2142253211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db616b0 msgr2=0x55571dfa59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:34.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.225+0000 7f9fa2ffd640 1 --2- 172.21.15.90:0/2142253211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db616b0 0x55571dfa59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.225+0000 7f9fa2ffd640 1 -- 172.21.15.90:0/2142253211 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55571de494b0 con 0x55571db40f60 2024-04-25T11:41:34.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.225+0000 7f9f9a7fc640 1 --2- 172.21.15.90:0/2142253211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55571db594d0 0x55571df309b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:34.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.225+0000 7f9fa37fe640 1 --2- 172.21.15.90:0/2142253211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db616b0 0x55571dfa59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:34.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.225+0000 7f9fa2ffd640 1 --2- 172.21.15.90:0/2142253211 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55571db40f60 0x55571df30470 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9f940a3c30 tx=0x7f9f94000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9dbb7586858b90a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:34.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.225+0000 7f9f8a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:41:34.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.226+0000 7f9f8a7fc640 1 -- 172.21.15.90:0/2142253211 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f940bf020 con 0x55571db40f60 2024-04-25T11:41:34.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.226+0000 7f9f8a7fc640 1 -- 172.21.15.90:0/2142253211 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f9f940a5c20 con 0x55571db40f60 2024-04-25T11:41:34.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.226+0000 7f9fb059f740 1 -- 172.21.15.90:0/2142253211 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f9c036040 con 0x55571db40f60 2024-04-25T11:41:34.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.226+0000 7f9f8a7fc640 1 -- 172.21.15.90:0/2142253211 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9f940a6310 con 0x55571db40f60 2024-04-25T11:41:34.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.226+0000 7f9fb059f740 10 client.26564 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:34.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.226+0000 7f9fb059f740 1 -- 172.21.15.90:0/2142253211 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55571dfa6db0 con 0x55571db40f60 2024-04-25T11:41:34.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.226+0000 7f9f8a7fc640 1 -- 172.21.15.90:0/2142253211 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f9c036040 con 0x55571db40f60 2024-04-25T11:41:34.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.226+0000 7f9f8a7fc640 1 -- 172.21.15.90:0/2142253211 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f9f940abf00 con 0x55571db40f60 2024-04-25T11:41:34.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.226+0000 7f9fb059f740 10 client.26564 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:34.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.226+0000 7f9fb059f740 10 client.26564 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:41:34.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.226+0000 7f9fb059f740 20 client.26564 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:34.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.226+0000 7f9fb059f740 1 --2- 172.21.15.90:0/2142253211 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55571db82390 0x55571dfa9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:34.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.226+0000 7f9fb059f740 4 client.26564 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:41:34.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.226+0000 7f9fb059f740 1 -- 172.21.15.90:0/2142253211 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55571de90800 con 0x55571db82390 2024-04-25T11:41:34.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.226+0000 7f9f9a7fc640 1 --2- 172.21.15.90:0/2142253211 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55571db82390 0x55571dfa9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:34.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.227+0000 7f9f9a7fc640 1 --2- 172.21.15.90:0/2142253211 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55571db82390 0x55571dfa9560 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13315edd4fd119de server_cookie=746b98c13646ec19 in_seq=0 out_seq=0 2024-04-25T11:41:34.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.227+0000 7f9f8a7fc640 10 client.26564 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:34.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.228+0000 7f9f8a7fc640 1 -- 172.21.15.90:0/2142253211 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55571de90800 con 0x55571db82390 2024-04-25T11:41:34.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.228+0000 7f9f8a7fc640 10 client.26564 handle_command_reply: tid=0 2024-04-25T11:41:34.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.228+0000 7f9fb059f740 1 client.26564 shutdown 2024-04-25T11:41:34.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.228+0000 7f9fb059f740 1 -- 172.21.15.90:0/2142253211 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55571db40f60 msgr2=0x55571df30470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:34.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.228+0000 7f9fb059f740 1 --2- 172.21.15.90:0/2142253211 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55571db40f60 0x55571df30470 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9f940a3c30 tx=0x7f9f94000bc0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.228+0000 7f9fb059f740 1 -- 172.21.15.90:0/2142253211 shutdown_connections 2024-04-25T11:41:34.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.228+0000 7f9fb059f740 1 --2- 172.21.15.90:0/2142253211 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55571db82390 0x55571dfa9560 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.228+0000 7f9fb059f740 1 --2- 172.21.15.90:0/2142253211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55571db616b0 0x55571dfa59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.228+0000 7f9fb059f740 1 --2- 172.21.15.90:0/2142253211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55571db594d0 0x55571df309b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.228+0000 7f9f8a7fc640 0 client.26564 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:34.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.228+0000 7f9fb059f740 1 --2- 172.21.15.90:0/2142253211 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55571db40f60 0x55571df30470 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.228+0000 7f9fb059f740 1 -- 172.21.15.90:0/2142253211 >> 172.21.15.90:0/2142253211 conn(0x55571db764e0 msgr2=0x55571df2d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:34.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.228+0000 7f9fb059f740 1 -- 172.21.15.90:0/2142253211 shutdown_connections 2024-04-25T11:41:34.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.228+0000 7f9fb059f740 1 -- 172.21.15.90:0/2142253211 wait complete. 2024-04-25T11:41:34.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.228+0000 7f9fb059f740 20 client.26564 trim_cache size 0 max 16384 2024-04-25T11:41:34.221 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:41:34.221 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "scrub active (0 inodes in the stack)", 2024-04-25T11:41:34.222 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-25T11:41:34.222 INFO:teuthology.orchestra.run.smithi090.stdout: "ebcb6268-39f5-4327-ac0e-d7d98cfcdf44": { 2024-04-25T11:41:34.222 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-25T11:41:34.222 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "ebcb6268-39f5-4327-ac0e-d7d98cfcdf44", 2024-04-25T11:41:34.222 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-25T11:41:34.222 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:41:34.222 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:41:34.222 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:41:34.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.228+0000 7f9fa9613640 1 -- 172.21.15.90:0/1663117181 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9f900afce0 msgr2=0x7f9f900b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:34.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.228+0000 7f9fa9613640 1 --2- 172.21.15.90:0/1663117181 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9f900afce0 0x7f9f900b21f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x55571df1fbd0 tx=0x7f9f9c02f310 comp rx=0 tx=0).stop 2024-04-25T11:41:34.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.228+0000 7f9fa9613640 1 -- 172.21.15.90:0/1663117181 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9fa4102b10 msgr2=0x7f9fa419a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:34.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.228+0000 7f9fa9613640 1 --2- 172.21.15.90:0/1663117181 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9fa4102b10 0x7f9fa419a710 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9f9400c970 tx=0x7f9f9400ce30 comp rx=0 tx=0).stop 2024-04-25T11:41:34.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.229+0000 7f9fa9613640 1 -- 172.21.15.90:0/1663117181 shutdown_connections 2024-04-25T11:41:34.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.229+0000 7f9fa9613640 1 --2- 172.21.15.90:0/1663117181 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9f900afce0 0x7f9f900b21f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.229+0000 7f9fa9613640 1 --2- 172.21.15.90:0/1663117181 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9fa410d3a0 0x7f9fa419ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.229+0000 7f9fa9613640 1 --2- 172.21.15.90:0/1663117181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fa4108a40 0x7f9fa419ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.229+0000 7f9fa9613640 1 --2- 172.21.15.90:0/1663117181 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9fa4102b10 0x7f9fa419a710 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:34.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.229+0000 7f9fa9613640 1 -- 172.21.15.90:0/1663117181 >> 172.21.15.90:0/1663117181 conn(0x7f9fa4078180 msgr2=0x7f9fa406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:34.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.229+0000 7f9fa9613640 1 -- 172.21.15.90:0/1663117181 shutdown_connections 2024-04-25T11:41:34.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:41:34.229+0000 7f9fa9613640 1 -- 172.21.15.90:0/1663117181 wait complete. 2024-04-25T11:41:34.230 INFO:tasks.cephfs.filesystem:scrub status for tag:ebcb6268-39f5-4327-ac0e-d7d98cfcdf44 - {'path': '/', 'tag': 'ebcb6268-39f5-4327-ac0e-d7d98cfcdf44', 'options': 'recursive,force'} 2024-04-25T11:41:34.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:34 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:41:34.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:34 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:41:34.723 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:34 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:41:34.723 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:34 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:41:34.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:34 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:41:34.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:34 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:41:35.701 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:35 smithi138 ceph-mon[32328]: scrub queued for path: / 2024-04-25T11:41:35.701 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:35 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:41:35.701 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:35 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:41:35.702 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:35 smithi138 ceph-mon[32328]: pgmap v198: 97 pgs: 97 active+clean; 59 GiB data, 178 GiB used, 895 GiB / 1.0 TiB avail; 3.7 MiB/s rd, 637 MiB/s wr, 700 op/s 2024-04-25T11:41:35.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:35 smithi090 ceph-mon[29575]: scrub queued for path: / 2024-04-25T11:41:35.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:35 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:41:35.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:35 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:41:35.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:35 smithi090 ceph-mon[29575]: pgmap v198: 97 pgs: 97 active+clean; 59 GiB data, 178 GiB used, 895 GiB / 1.0 TiB avail; 3.7 MiB/s rd, 637 MiB/s wr, 700 op/s 2024-04-25T11:41:35.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:35 smithi156 ceph-mon[33045]: scrub queued for path: / 2024-04-25T11:41:35.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:35 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:41:35.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:35 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:41:35.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:35 smithi156 ceph-mon[33045]: pgmap v198: 97 pgs: 97 active+clean; 59 GiB data, 178 GiB used, 895 GiB / 1.0 TiB avail; 3.7 MiB/s rd, 637 MiB/s wr, 700 op/s 2024-04-25T11:41:36.725 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:36 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:41:36.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:36 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:41:36.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:36 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:41:37.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:37 smithi138 ceph-mon[32328]: pgmap v199: 97 pgs: 97 active+clean; 60 GiB data, 182 GiB used, 891 GiB / 1.0 TiB avail; 4.0 MiB/s rd, 790 MiB/s wr, 933 op/s 2024-04-25T11:41:37.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:37 smithi156 ceph-mon[33045]: pgmap v199: 97 pgs: 97 active+clean; 60 GiB data, 182 GiB used, 891 GiB / 1.0 TiB avail; 4.0 MiB/s rd, 790 MiB/s wr, 933 op/s 2024-04-25T11:41:37.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:37 smithi090 ceph-mon[29575]: pgmap v199: 97 pgs: 97 active+clean; 60 GiB data, 182 GiB used, 891 GiB / 1.0 TiB avail; 4.0 MiB/s rd, 790 MiB/s wr, 933 op/s 2024-04-25T11:41:38.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.659+0000 7f0b05fb2640 1 -- 172.21.15.90:0/3117682674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b001032f0 msgr2=0x7f0b00103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:38.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.659+0000 7f0b05fb2640 1 --2- 172.21.15.90:0/3117682674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b001032f0 0x7f0b00103700 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f0af4009900 tx=0x7f0af402a410 comp rx=0 tx=0).stop 2024-04-25T11:41:38.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.659+0000 7f0b05fb2640 1 -- 172.21.15.90:0/3117682674 shutdown_connections 2024-04-25T11:41:38.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.659+0000 7f0b05fb2640 1 --2- 172.21.15.90:0/3117682674 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b0010d470 0x7f0b0010f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:38.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.659+0000 7f0b05fb2640 1 --2- 172.21.15.90:0/3117682674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b001032f0 0x7f0b00103700 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:38.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.659+0000 7f0b05fb2640 1 --2- 172.21.15.90:0/3117682674 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b00107980 0x7f0b00102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:38.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.659+0000 7f0b05fb2640 1 -- 172.21.15.90:0/3117682674 >> 172.21.15.90:0/3117682674 conn(0x7f0b000778b0 msgr2=0x7f0b00077cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:38.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.659+0000 7f0b05fb2640 1 -- 172.21.15.90:0/3117682674 shutdown_connections 2024-04-25T11:41:38.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.659+0000 7f0b05fb2640 1 -- 172.21.15.90:0/3117682674 wait complete. 2024-04-25T11:41:38.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.660+0000 7f0b05fb2640 1 Processor -- start 2024-04-25T11:41:38.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.660+0000 7f0b05fb2640 1 -- start start 2024-04-25T11:41:38.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.660+0000 7f0b05fb2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b001032f0 0x7f0b0019b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:38.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.660+0000 7f0b05fb2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b00107980 0x7f0b0019bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:38.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.660+0000 7f0b05fb2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b0010d470 0x7f0b0019c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:38.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.660+0000 7f0b05fb2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0b00112ce0 con 0x7f0b00107980 2024-04-25T11:41:38.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.660+0000 7f0b05fb2640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0b00112b60 con 0x7f0b001032f0 2024-04-25T11:41:38.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.660+0000 7f0b05fb2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0b00112e60 con 0x7f0b0010d470 2024-04-25T11:41:38.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.661+0000 7f0afeffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b00107980 0x7f0b0019bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:38.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.661+0000 7f0afeffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b00107980 0x7f0b0019bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38408/0 (socket says 172.21.15.90:38408) 2024-04-25T11:41:38.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.661+0000 7f0afeffd640 1 -- 172.21.15.90:0/644154022 learned_addr learned my addr 172.21.15.90:0/644154022 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:38.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.661+0000 7f0afeffd640 1 -- 172.21.15.90:0/644154022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b0010d470 msgr2=0x7f0b0019c080 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:41:38.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.661+0000 7f0afeffd640 1 --2- 172.21.15.90:0/644154022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b0010d470 0x7f0b0019c080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:38.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.661+0000 7f0afeffd640 1 -- 172.21.15.90:0/644154022 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b001032f0 msgr2=0x7f0b0019b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:38.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.661+0000 7f0afeffd640 1 --2- 172.21.15.90:0/644154022 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b001032f0 0x7f0b0019b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:38.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.661+0000 7f0afeffd640 1 -- 172.21.15.90:0/644154022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b001a6880 con 0x7f0b00107980 2024-04-25T11:41:38.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.661+0000 7f0afeffd640 1 --2- 172.21.15.90:0/644154022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b00107980 0x7f0b0019bb40 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f0af4031440 tx=0x7f0af4003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:38.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.661+0000 7f0afcff9640 1 -- 172.21.15.90:0/644154022 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af403f070 con 0x7f0b00107980 2024-04-25T11:41:38.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.661+0000 7f0b05fb2640 1 -- 172.21.15.90:0/644154022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b001a6b10 con 0x7f0b00107980 2024-04-25T11:41:38.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.661+0000 7f0afcff9640 1 -- 172.21.15.90:0/644154022 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0af4031d60 con 0x7f0b00107980 2024-04-25T11:41:38.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.661+0000 7f0afcff9640 1 -- 172.21.15.90:0/644154022 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af4035030 con 0x7f0b00107980 2024-04-25T11:41:38.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.661+0000 7f0b05fb2640 1 -- 172.21.15.90:0/644154022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b001a7050 con 0x7f0b00107980 2024-04-25T11:41:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.664+0000 7f0b0ce65740 1 Processor -- start 2024-04-25T11:41:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.664+0000 7f0b0ce65740 1 -- start start 2024-04-25T11:41:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.664+0000 7f0b0ce65740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7886d51b0 0x55d788a74af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.664+0000 7f0b0ce65740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 0x55d788a75030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.664+0000 7f0b0ce65740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d7886b44d0 0x55d788a75570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.664+0000 7f0b0ce65740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d7889db460 con 0x55d7886ac2f0 2024-04-25T11:41:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.664+0000 7f0b0ce65740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55d788995590 con 0x55d7886d51b0 2024-04-25T11:41:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.664+0000 7f0b0ce65740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d7889d7420 con 0x55d7886b44d0 2024-04-25T11:41:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.664+0000 7f0afcff9640 1 -- 172.21.15.90:0/644154022 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f0af4042070 con 0x7f0b00107980 2024-04-25T11:41:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.664+0000 7f0aff7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 0x55d788a75030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.665+0000 7f0aff7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 0x55d788a75030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38410/0 (socket says 172.21.15.90:38410) 2024-04-25T11:41:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.665+0000 7f0aff7fe640 1 -- 172.21.15.90:0/159892468 learned_addr learned my addr 172.21.15.90:0/159892468 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.665+0000 7f0aff7fe640 1 -- 172.21.15.90:0/159892468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d7886b44d0 msgr2=0x55d788a75570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.665+0000 7f0aff7fe640 1 --2- 172.21.15.90:0/159892468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d7886b44d0 0x55d788a75570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.665+0000 7f0aff7fe640 1 -- 172.21.15.90:0/159892468 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7886d51b0 msgr2=0x55d788a74af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.665+0000 7f0aff7fe640 1 --2- 172.21.15.90:0/159892468 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7886d51b0 0x55d788a74af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.665+0000 7f0aff7fe640 1 -- 172.21.15.90:0/159892468 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7889a20d0 con 0x55d7886ac2f0 2024-04-25T11:41:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.665+0000 7f0aff7fe640 1 --2- 172.21.15.90:0/159892468 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 0x55d788a75030 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f0af0009820 tx=0x7f0af0009ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a71e904af2ba9b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.665+0000 7f0afcff9640 1 --2- 172.21.15.90:0/644154022 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0ae80afc50 0x7f0ae80b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.665+0000 7f0ae5ffb640 1 -- 172.21.15.90:0/159892468 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af0015020 con 0x55d7886ac2f0 2024-04-25T11:41:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.665+0000 7f0afffff640 1 --2- 172.21.15.90:0/644154022 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0ae80afc50 0x7f0ae80b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.666+0000 7f0ae5ffb640 1 -- 172.21.15.90:0/159892468 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0af0011420 con 0x55d7886ac2f0 2024-04-25T11:41:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.666+0000 7f0ae5ffb640 1 -- 172.21.15.90:0/159892468 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af00105c0 con 0x55d7886ac2f0 2024-04-25T11:41:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.666+0000 7f0afcff9640 1 -- 172.21.15.90:0/644154022 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f0af40ffa60 con 0x7f0b00107980 2024-04-25T11:41:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.666+0000 7f0b0ce65740 1 -- 172.21.15.90:0/159892468 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 msgr2=0x55d788a75030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.666+0000 7f0afffff640 1 --2- 172.21.15.90:0/644154022 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0ae80afc50 0x7f0ae80b2160 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0af8005fd0 tx=0x7f0af80071c0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.666+0000 7f0b0ce65740 1 --2- 172.21.15.90:0/159892468 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 0x55d788a75030 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f0af0009820 tx=0x7f0af0009ce0 comp rx=0 tx=0).stop 2024-04-25T11:41:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.667+0000 7f0b0ce65740 1 -- 172.21.15.90:0/159892468 shutdown_connections 2024-04-25T11:41:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.667+0000 7f0b0ce65740 1 --2- 172.21.15.90:0/159892468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d7886b44d0 0x55d788a75570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.667+0000 7f0b0ce65740 1 --2- 172.21.15.90:0/159892468 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 0x55d788a75030 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.667+0000 7f0b0ce65740 1 --2- 172.21.15.90:0/159892468 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7886d51b0 0x55d788a74af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.667+0000 7f0b0ce65740 1 -- 172.21.15.90:0/159892468 >> 172.21.15.90:0/159892468 conn(0x55d7886c9300 msgr2=0x55d788a712a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.667+0000 7f0b0ce65740 1 -- 172.21.15.90:0/159892468 shutdown_connections 2024-04-25T11:41:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.667+0000 7f0b0ce65740 1 -- 172.21.15.90:0/159892468 wait complete. 2024-04-25T11:41:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.668+0000 7f0b0ce65740 1 -- start start 2024-04-25T11:41:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.668+0000 7f0b0ce65740 1 Processor -- start 2024-04-25T11:41:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.668+0000 7f0b0ce65740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 0x55d788a80b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.668+0000 7f0b0ce65740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7886b44d0 0x55d788a81070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.668+0000 7f0b0ce65740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d7886d51b0 0x55d788a7efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.668+0000 7f0b0ce65740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d788a81c20 con 0x55d7886ac2f0 2024-04-25T11:41:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.668+0000 7f0b0ce65740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55d788962b60 con 0x55d7886b44d0 2024-04-25T11:41:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.668+0000 7f0b0ce65740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d788a81da0 con 0x55d7886d51b0 2024-04-25T11:41:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.669+0000 7f0b0ce65740 10 client.0 _ll_register_callbacks cb 0x55d7889872b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.669+0000 7f0aff7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 0x55d788a80b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.669+0000 7f0aff7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 0x55d788a80b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38424/0 (socket says 172.21.15.90:38424) 2024-04-25T11:41:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.669+0000 7f0aff7fe640 1 -- 172.21.15.90:0/2670709059 learned_addr learned my addr 172.21.15.90:0/2670709059 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.669+0000 7f0afffff640 1 --2- 172.21.15.90:0/2670709059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d7886d51b0 0x55d788a7efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.669+0000 7f0aff7fe640 1 -- 172.21.15.90:0/2670709059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d7886d51b0 msgr2=0x55d788a7efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.669+0000 7f0aff7fe640 1 --2- 172.21.15.90:0/2670709059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d7886d51b0 0x55d788a7efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.669+0000 7f0aff7fe640 1 -- 172.21.15.90:0/2670709059 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7886b44d0 msgr2=0x55d788a81070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.669+0000 7f0aff7fe640 1 --2- 172.21.15.90:0/2670709059 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7886b44d0 0x55d788a81070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.669+0000 7f0aff7fe640 1 -- 172.21.15.90:0/2670709059 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7886ed210 con 0x55d7886ac2f0 2024-04-25T11:41:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.669+0000 7f0afffff640 1 --2- 172.21.15.90:0/2670709059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d7886d51b0 0x55d788a7efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:41:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.669+0000 7f0aff7fe640 1 --2- 172.21.15.90:0/2670709059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 0x55d788a80b10 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f0af0011e20 tx=0x7f0af00115d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d45d34df1502cfc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.669+0000 7f0adeffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:41:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.670+0000 7f0adeffd640 1 -- 172.21.15.90:0/2670709059 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af0027060 con 0x55d7886ac2f0 2024-04-25T11:41:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.670+0000 7f0adeffd640 1 -- 172.21.15.90:0/2670709059 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f0af000b8d0 con 0x55d7886ac2f0 2024-04-25T11:41:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.670+0000 7f0b0ce65740 1 -- 172.21.15.90:0/2670709059 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0af0011420 con 0x55d7886ac2f0 2024-04-25T11:41:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.670+0000 7f0adeffd640 1 -- 172.21.15.90:0/2670709059 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0af0015020 con 0x55d7886ac2f0 2024-04-25T11:41:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.670+0000 7f0b0ce65740 10 client.17604 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.670+0000 7f0b0ce65740 1 -- 172.21.15.90:0/2670709059 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d7887b9620 con 0x55d7886ac2f0 2024-04-25T11:41:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.671+0000 7f0adeffd640 1 -- 172.21.15.90:0/2670709059 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0af0011420 con 0x55d7886ac2f0 2024-04-25T11:41:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.671+0000 7f0adeffd640 1 -- 172.21.15.90:0/2670709059 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f0af001ab20 con 0x55d7886ac2f0 2024-04-25T11:41:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.671+0000 7f0b0ce65740 10 client.17604 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.671+0000 7f0b0ce65740 10 client.17604 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:41:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.671+0000 7f0b0ce65740 20 client.17604 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.671+0000 7f0b0ce65740 1 --2- 172.21.15.90:0/2670709059 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d788693d80 0x55d788a91680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.671+0000 7f0b0ce65740 4 client.17604 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:41:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.671+0000 7f0b0ce65740 1 -- 172.21.15.90:0/2670709059 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d7889e9070 con 0x55d788693d80 2024-04-25T11:41:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.672+0000 7f0afeffd640 1 --2- 172.21.15.90:0/2670709059 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d788693d80 0x55d788a91680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.674+0000 7f0adeffd640 10 client.17604 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.674+0000 7f0afeffd640 1 --2- 172.21.15.90:0/2670709059 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d788693d80 0x55d788a91680 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a590b0f6e66e021 server_cookie=146210c892efca04 in_seq=0 out_seq=0 2024-04-25T11:41:38.667 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.678+0000 7f0adeffd640 1 -- 172.21.15.90:0/2670709059 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d7889e9070 con 0x55d788693d80 2024-04-25T11:41:38.667 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.678+0000 7f0adeffd640 10 client.17604 handle_command_reply: tid=0 2024-04-25T11:41:38.667 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.678+0000 7f0b0ce65740 1 client.17604 shutdown 2024-04-25T11:41:38.667 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.678+0000 7f0b0ce65740 1 -- 172.21.15.90:0/2670709059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 msgr2=0x55d788a80b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:38.667 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.678+0000 7f0b0ce65740 1 --2- 172.21.15.90:0/2670709059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 0x55d788a80b10 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f0af0011e20 tx=0x7f0af00115d0 comp rx=0 tx=0).stop 2024-04-25T11:41:38.667 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.679+0000 7f0b0ce65740 1 -- 172.21.15.90:0/2670709059 shutdown_connections 2024-04-25T11:41:38.667 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.679+0000 7f0b0ce65740 1 --2- 172.21.15.90:0/2670709059 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d788693d80 0x55d788a91680 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:38.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.679+0000 7f0adeffd640 0 client.17604 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:38.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.679+0000 7f0b0ce65740 1 --2- 172.21.15.90:0/2670709059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d7886d51b0 0x55d788a7efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:38.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.679+0000 7f0b0ce65740 1 --2- 172.21.15.90:0/2670709059 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7886b44d0 0x55d788a81070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:38.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.679+0000 7f0b0ce65740 1 --2- 172.21.15.90:0/2670709059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 0x55d788a80b10 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:38.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.679+0000 7f0b0ce65740 1 -- 172.21.15.90:0/2670709059 >> 172.21.15.90:0/2670709059 conn(0x55d7886c9300 msgr2=0x55d788a76810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:38.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.679+0000 7f0b0ce65740 1 -- 172.21.15.90:0/2670709059 shutdown_connections 2024-04-25T11:41:38.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.680+0000 7f0b0ce65740 1 -- 172.21.15.90:0/2670709059 wait complete. 2024-04-25T11:41:38.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.680+0000 7f0b0ce65740 20 client.17604 trim_cache size 0 max 16384 2024-04-25T11:41:38.697 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.708+0000 7f0b0ce65740 1 Processor -- start 2024-04-25T11:41:38.697 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.708+0000 7f0b0ce65740 1 -- start start 2024-04-25T11:41:38.697 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.709+0000 7f0b0ce65740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7886c9300 0x55d788a78850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:38.697 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.709+0000 7f0b0ce65740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d788693d80 0x55d788a78d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:38.697 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.709+0000 7f0b0ce65740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 0x55d788a96e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:38.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.709+0000 7f0b0ce65740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0b001afc40 con 0x55d7886ac2f0 2024-04-25T11:41:38.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.709+0000 7f0b0ce65740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0b001b1780 con 0x55d7886c9300 2024-04-25T11:41:38.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.709+0000 7f0b0ce65740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0b001b0310 con 0x55d788693d80 2024-04-25T11:41:38.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.709+0000 7f0afffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 0x55d788a96e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:38.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.709+0000 7f0afffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 0x55d788a96e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38434/0 (socket says 172.21.15.90:38434) 2024-04-25T11:41:38.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.709+0000 7f0afffff640 1 -- 172.21.15.90:0/3229124699 learned_addr learned my addr 172.21.15.90:0/3229124699 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:38.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.710+0000 7f0afffff640 1 -- 172.21.15.90:0/3229124699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d788693d80 msgr2=0x55d788a78d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:38.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.710+0000 7f0afffff640 1 --2- 172.21.15.90:0/3229124699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d788693d80 0x55d788a78d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:38.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.710+0000 7f0afffff640 1 -- 172.21.15.90:0/3229124699 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7886c9300 msgr2=0x55d788a78850 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:41:38.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.710+0000 7f0afffff640 1 --2- 172.21.15.90:0/3229124699 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7886c9300 0x55d788a78850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:38.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.710+0000 7f0afffff640 1 -- 172.21.15.90:0/3229124699 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d788a91be0 con 0x55d7886ac2f0 2024-04-25T11:41:38.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.710+0000 7f0afffff640 1 --2- 172.21.15.90:0/3229124699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 0x55d788a96e20 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f0af800ce20 tx=0x7f0af800f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cdb7825a5ee996d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:38.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.710+0000 7f0ae57fa640 1 -- 172.21.15.90:0/3229124699 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af8003e00 con 0x55d7886ac2f0 2024-04-25T11:41:38.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.710+0000 7f0ae57fa640 1 -- 172.21.15.90:0/3229124699 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0af8018040 con 0x55d7886ac2f0 2024-04-25T11:41:38.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.710+0000 7f0ae57fa640 1 -- 172.21.15.90:0/3229124699 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af80057a0 con 0x55d7886ac2f0 2024-04-25T11:41:38.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.711+0000 7f0b0ce65740 1 -- 172.21.15.90:0/3229124699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 msgr2=0x55d788a96e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:38.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.711+0000 7f0b0ce65740 1 --2- 172.21.15.90:0/3229124699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 0x55d788a96e20 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f0af800ce20 tx=0x7f0af800f000 comp rx=0 tx=0).stop 2024-04-25T11:41:38.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.711+0000 7f0b0ce65740 1 -- 172.21.15.90:0/3229124699 shutdown_connections 2024-04-25T11:41:38.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.711+0000 7f0b0ce65740 1 --2- 172.21.15.90:0/3229124699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 0x55d788a96e20 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:38.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.711+0000 7f0b0ce65740 1 --2- 172.21.15.90:0/3229124699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d788693d80 0x55d788a78d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:38.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.711+0000 7f0b0ce65740 1 --2- 172.21.15.90:0/3229124699 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d7886c9300 0x55d788a78850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:38.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.711+0000 7f0b0ce65740 1 -- 172.21.15.90:0/3229124699 >> 172.21.15.90:0/3229124699 conn(0x55d788a80c40 msgr2=0x55d788a7d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:38.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.711+0000 7f0b0ce65740 1 -- 172.21.15.90:0/3229124699 shutdown_connections 2024-04-25T11:41:38.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.711+0000 7f0b0ce65740 1 -- 172.21.15.90:0/3229124699 wait complete. 2024-04-25T11:41:38.702 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.712+0000 7f0b0ce65740 1 -- start start 2024-04-25T11:41:38.702 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.712+0000 7f0b0ce65740 1 Processor -- start 2024-04-25T11:41:38.702 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.713+0000 7f0b0ce65740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d788693d80 0x55d788a82dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:38.702 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.713+0000 7f0b0ce65740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 0x55d788a83300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:38.702 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.713+0000 7f0afeffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 0x55d788a83300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:38.702 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.713+0000 7f0afeffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 0x55d788a83300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38450/0 (socket says 172.21.15.90:38450) 2024-04-25T11:41:38.702 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.713+0000 7f0afeffd640 1 -- 172.21.15.90:0/2232913699 learned_addr learned my addr 172.21.15.90:0/2232913699 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:38.702 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.713+0000 7f0b0ce65740 1 --2- 172.21.15.90:0/2232913699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d7886c9300 0x55d788a83840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:38.702 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.713+0000 7f0b0ce65740 1 -- 172.21.15.90:0/2232913699 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0b001b10b0 con 0x55d7886ac2f0 2024-04-25T11:41:38.702 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.713+0000 7f0b0ce65740 1 -- 172.21.15.90:0/2232913699 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0b001b09e0 con 0x55d788693d80 2024-04-25T11:41:38.702 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.714+0000 7f0b0ce65740 1 -- 172.21.15.90:0/2232913699 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0b001adc20 con 0x55d7886c9300 2024-04-25T11:41:38.702 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.714+0000 7f0b0ce65740 10 client.0 _ll_register_callbacks cb 0x7f0af0009670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:38.705 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.714+0000 7f0afeffd640 1 -- 172.21.15.90:0/2232913699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d7886c9300 msgr2=0x55d788a83840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:38.705 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.714+0000 7f0afeffd640 1 --2- 172.21.15.90:0/2232913699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d7886c9300 0x55d788a83840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:38.705 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.714+0000 7f0afeffd640 1 -- 172.21.15.90:0/2232913699 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d788693d80 msgr2=0x55d788a82dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:38.705 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.714+0000 7f0afeffd640 1 --2- 172.21.15.90:0/2232913699 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d788693d80 0x55d788a82dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:38.705 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.714+0000 7f0afeffd640 1 -- 172.21.15.90:0/2232913699 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d788af8aa0 con 0x55d7886ac2f0 2024-04-25T11:41:38.705 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.715+0000 7f0afeffd640 1 --2- 172.21.15.90:0/2232913699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 0x55d788a83300 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f0af40c3c20 tx=0x7f0af40c75b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6264c30e0123342 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:38.705 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.715+0000 7f0adeffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:41:38.705 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.715+0000 7f0adeffd640 1 -- 172.21.15.90:0/2232913699 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af4049040 con 0x55d7886ac2f0 2024-04-25T11:41:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.715+0000 7f0adeffd640 1 -- 172.21.15.90:0/2232913699 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f0af40c4190 con 0x55d7886ac2f0 2024-04-25T11:41:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.715+0000 7f0b0ce65740 1 -- 172.21.15.90:0/2232913699 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0af8018040 con 0x55d7886ac2f0 2024-04-25T11:41:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.715+0000 7f0adeffd640 1 -- 172.21.15.90:0/2232913699 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0af40c76b0 con 0x55d7886ac2f0 2024-04-25T11:41:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.715+0000 7f0b0ce65740 10 client.17610 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.715+0000 7f0b0ce65740 1 -- 172.21.15.90:0/2232913699 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d788af9d10 con 0x55d7886ac2f0 2024-04-25T11:41:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.716+0000 7f0adeffd640 1 -- 172.21.15.90:0/2232913699 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0af8018040 con 0x55d7886ac2f0 2024-04-25T11:41:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.716+0000 7f0adeffd640 1 -- 172.21.15.90:0/2232913699 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f0af403b610 con 0x55d7886ac2f0 2024-04-25T11:41:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.717+0000 7f0b0ce65740 10 client.17610 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.717+0000 7f0b0ce65740 10 client.17610 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:41:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.717+0000 7f0b0ce65740 20 client.17610 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.717+0000 7f0b0ce65740 1 --2- 172.21.15.90:0/2232913699 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d7886b44d0 0x55d788afc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:38.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.717+0000 7f0b0ce65740 4 client.17610 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:41:38.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.717+0000 7f0b0ce65740 1 -- 172.21.15.90:0/2232913699 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d7889e2ee0 con 0x55d7886b44d0 2024-04-25T11:41:38.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.718+0000 7f0aff7fe640 1 --2- 172.21.15.90:0/2232913699 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d7886b44d0 0x55d788afc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:38.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.719+0000 7f0aff7fe640 1 --2- 172.21.15.90:0/2232913699 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d7886b44d0 0x55d788afc4c0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91fda8f280c94ee4 server_cookie=197d12b15eb8979b in_seq=0 out_seq=0 2024-04-25T11:41:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:38.719+0000 7f0adeffd640 10 client.17610 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:39.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.839+0000 7f0adeffd640 1 -- 172.21.15.90:0/2232913699 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d7889e2ee0 con 0x55d7886b44d0 2024-04-25T11:41:39.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.839+0000 7f0adeffd640 10 client.17610 handle_command_reply: tid=0 2024-04-25T11:41:39.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.839+0000 7f0b0ce65740 1 client.17610 shutdown 2024-04-25T11:41:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.840+0000 7f0b0ce65740 1 -- 172.21.15.90:0/2232913699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 msgr2=0x55d788a83300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.840+0000 7f0b0ce65740 1 --2- 172.21.15.90:0/2232913699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 0x55d788a83300 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f0af40c3c20 tx=0x7f0af40c75b0 comp rx=0 tx=0).stop 2024-04-25T11:41:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.840+0000 7f0b0ce65740 1 -- 172.21.15.90:0/2232913699 shutdown_connections 2024-04-25T11:41:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.840+0000 7f0b0ce65740 1 --2- 172.21.15.90:0/2232913699 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d7886b44d0 0x55d788afc4c0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.840+0000 7f0b0ce65740 1 --2- 172.21.15.90:0/2232913699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d7886c9300 0x55d788a83840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.840+0000 7f0b0ce65740 1 --2- 172.21.15.90:0/2232913699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d7886ac2f0 0x55d788a83300 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.840+0000 7f0b0ce65740 1 --2- 172.21.15.90:0/2232913699 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d788693d80 0x55d788a82dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.840+0000 7f0b0ce65740 1 -- 172.21.15.90:0/2232913699 >> 172.21.15.90:0/2232913699 conn(0x55d788a80c40 msgr2=0x55d788a76730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.840+0000 7f0adeffd640 0 client.17610 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.841+0000 7f0b0ce65740 1 -- 172.21.15.90:0/2232913699 shutdown_connections 2024-04-25T11:41:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.841+0000 7f0b0ce65740 1 -- 172.21.15.90:0/2232913699 wait complete. 2024-04-25T11:41:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.841+0000 7f0b0ce65740 20 client.17610 trim_cache size 0 max 16384 2024-04-25T11:41:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:41:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:41:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:41:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:41:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.845+0000 7f0b05fb2640 1 -- 172.21.15.90:0/644154022 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0ae80afc50 msgr2=0x7f0ae80b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.845+0000 7f0b05fb2640 1 --2- 172.21.15.90:0/644154022 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0ae80afc50 0x7f0ae80b2160 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0af8005fd0 tx=0x7f0af80071c0 comp rx=0 tx=0).stop 2024-04-25T11:41:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.845+0000 7f0b05fb2640 1 -- 172.21.15.90:0/644154022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b00107980 msgr2=0x7f0b0019bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.845+0000 7f0b05fb2640 1 --2- 172.21.15.90:0/644154022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b00107980 0x7f0b0019bb40 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f0af4031440 tx=0x7f0af4003d50 comp rx=0 tx=0).stop 2024-04-25T11:41:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.846+0000 7f0b05fb2640 1 -- 172.21.15.90:0/644154022 shutdown_connections 2024-04-25T11:41:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.846+0000 7f0b05fb2640 1 --2- 172.21.15.90:0/644154022 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0ae80afc50 0x7f0ae80b2160 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.846+0000 7f0b05fb2640 1 --2- 172.21.15.90:0/644154022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b0010d470 0x7f0b0019c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.846+0000 7f0b05fb2640 1 --2- 172.21.15.90:0/644154022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b00107980 0x7f0b0019bb40 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.846+0000 7f0b05fb2640 1 --2- 172.21.15.90:0/644154022 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b001032f0 0x7f0b0019b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.846+0000 7f0b05fb2640 1 -- 172.21.15.90:0/644154022 >> 172.21.15.90:0/644154022 conn(0x7f0b000778b0 msgr2=0x7f0b00100910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:39.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.850+0000 7f0b05fb2640 1 -- 172.21.15.90:0/644154022 shutdown_connections 2024-04-25T11:41:39.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:39.850+0000 7f0b05fb2640 1 -- 172.21.15.90:0/644154022 wait complete. 2024-04-25T11:41:39.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:39 smithi090 ceph-mon[29575]: pgmap v200: 97 pgs: 97 active+clean; 60 GiB data, 182 GiB used, 891 GiB / 1.0 TiB avail; 2.1 MiB/s rd, 537 MiB/s wr, 816 op/s 2024-04-25T11:41:40.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:39 smithi138 ceph-mon[32328]: pgmap v200: 97 pgs: 97 active+clean; 60 GiB data, 182 GiB used, 891 GiB / 1.0 TiB avail; 2.1 MiB/s rd, 537 MiB/s wr, 816 op/s 2024-04-25T11:41:40.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:39 smithi156 ceph-mon[33045]: pgmap v200: 97 pgs: 97 active+clean; 60 GiB data, 182 GiB used, 891 GiB / 1.0 TiB avail; 2.1 MiB/s rd, 537 MiB/s wr, 816 op/s 2024-04-25T11:41:40.976 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:40 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:41:41.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:40 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:41:41.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:40 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:41:41.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:41 smithi090 ceph-mon[29575]: pgmap v201: 97 pgs: 97 active+clean; 60 GiB data, 182 GiB used, 891 GiB / 1.0 TiB avail; 3.7 MiB/s rd, 745 MiB/s wr, 1.06k op/s 2024-04-25T11:41:41.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:41 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:41:42.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:41 smithi138 ceph-mon[32328]: pgmap v201: 97 pgs: 97 active+clean; 60 GiB data, 182 GiB used, 891 GiB / 1.0 TiB avail; 3.7 MiB/s rd, 745 MiB/s wr, 1.06k op/s 2024-04-25T11:41:42.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:41 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:41:42.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:41 smithi156 ceph-mon[33045]: pgmap v201: 97 pgs: 97 active+clean; 60 GiB data, 182 GiB used, 891 GiB / 1.0 TiB avail; 3.7 MiB/s rd, 745 MiB/s wr, 1.06k op/s 2024-04-25T11:41:42.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:41 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:41:44.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:44 smithi156 ceph-mon[33045]: pgmap v202: 97 pgs: 97 active+clean; 60 GiB data, 182 GiB used, 891 GiB / 1.0 TiB avail; 2.6 MiB/s rd, 547 MiB/s wr, 708 op/s 2024-04-25T11:41:44.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:44 smithi090 ceph-mon[29575]: pgmap v202: 97 pgs: 97 active+clean; 60 GiB data, 182 GiB used, 891 GiB / 1.0 TiB avail; 2.6 MiB/s rd, 547 MiB/s wr, 708 op/s 2024-04-25T11:41:44.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:44 smithi138 ceph-mon[32328]: pgmap v202: 97 pgs: 97 active+clean; 60 GiB data, 182 GiB used, 891 GiB / 1.0 TiB avail; 2.6 MiB/s rd, 547 MiB/s wr, 708 op/s 2024-04-25T11:41:45.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:45 smithi138 ceph-mon[32328]: pgmap v203: 97 pgs: 97 active+clean; 60 GiB data, 182 GiB used, 891 GiB / 1.0 TiB avail; 2.6 MiB/s rd, 503 MiB/s wr, 607 op/s 2024-04-25T11:41:45.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:45 smithi156 ceph-mon[33045]: pgmap v203: 97 pgs: 97 active+clean; 60 GiB data, 182 GiB used, 891 GiB / 1.0 TiB avail; 2.6 MiB/s rd, 503 MiB/s wr, 607 op/s 2024-04-25T11:41:45.723 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:45 smithi090 ceph-mon[29575]: pgmap v203: 97 pgs: 97 active+clean; 60 GiB data, 182 GiB used, 891 GiB / 1.0 TiB avail; 2.6 MiB/s rd, 503 MiB/s wr, 607 op/s 2024-04-25T11:41:47.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:47 smithi090 ceph-mon[29575]: pgmap v204: 97 pgs: 97 active+clean; 60 GiB data, 182 GiB used, 890 GiB / 1.0 TiB avail; 3.0 MiB/s rd, 715 MiB/s wr, 827 op/s 2024-04-25T11:41:48.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:47 smithi138 ceph-mon[32328]: pgmap v204: 97 pgs: 97 active+clean; 60 GiB data, 182 GiB used, 890 GiB / 1.0 TiB avail; 3.0 MiB/s rd, 715 MiB/s wr, 827 op/s 2024-04-25T11:41:48.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:47 smithi156 ceph-mon[33045]: pgmap v204: 97 pgs: 97 active+clean; 60 GiB data, 182 GiB used, 890 GiB / 1.0 TiB avail; 3.0 MiB/s rd, 715 MiB/s wr, 827 op/s 2024-04-25T11:41:49.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:49 smithi090 ceph-mon[29575]: pgmap v205: 97 pgs: 97 active+clean; 60 GiB data, 182 GiB used, 890 GiB / 1.0 TiB avail; 2.7 MiB/s rd, 531 MiB/s wr, 584 op/s 2024-04-25T11:41:50.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:49 smithi138 ceph-mon[32328]: pgmap v205: 97 pgs: 97 active+clean; 60 GiB data, 182 GiB used, 890 GiB / 1.0 TiB avail; 2.7 MiB/s rd, 531 MiB/s wr, 584 op/s 2024-04-25T11:41:50.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:49 smithi156 ceph-mon[33045]: pgmap v205: 97 pgs: 97 active+clean; 60 GiB data, 182 GiB used, 890 GiB / 1.0 TiB avail; 2.7 MiB/s rd, 531 MiB/s wr, 584 op/s 2024-04-25T11:41:52.051 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:51 smithi138 ceph-mon[32328]: pgmap v206: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 4.0 MiB/s rd, 701 MiB/s wr, 1.13k op/s 2024-04-25T11:41:52.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:51 smithi156 ceph-mon[33045]: pgmap v206: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 4.0 MiB/s rd, 701 MiB/s wr, 1.13k op/s 2024-04-25T11:41:52.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:51 smithi090 ceph-mon[29575]: pgmap v206: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 4.0 MiB/s rd, 701 MiB/s wr, 1.13k op/s 2024-04-25T11:41:53.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.038+0000 7f213c2cd640 1 -- 172.21.15.90:0/3100801647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2134104d20 msgr2=0x7f2134105130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:53.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.038+0000 7f213c2cd640 1 --2- 172.21.15.90:0/3100801647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2134104d20 0x7f2134105130 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2128009860 tx=0x7f212802a3d0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.038+0000 7f213c2cd640 1 -- 172.21.15.90:0/3100801647 shutdown_connections 2024-04-25T11:41:53.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.038+0000 7f213c2cd640 1 --2- 172.21.15.90:0/3100801647 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f213410e0e0 0x7f21341105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.038+0000 7f213c2cd640 1 --2- 172.21.15.90:0/3100801647 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f213410b780 0x7f213410dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.038+0000 7f213c2cd640 1 --2- 172.21.15.90:0/3100801647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2134104d20 0x7f2134105130 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.038+0000 7f213c2cd640 1 -- 172.21.15.90:0/3100801647 >> 172.21.15.90:0/3100801647 conn(0x7f2134078150 msgr2=0x7f2134078590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:53.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.038+0000 7f213c2cd640 1 -- 172.21.15.90:0/3100801647 shutdown_connections 2024-04-25T11:41:53.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.038+0000 7f213c2cd640 1 -- 172.21.15.90:0/3100801647 wait complete. 2024-04-25T11:41:53.028 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.039+0000 7f213c2cd640 1 Processor -- start 2024-04-25T11:41:53.028 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.039+0000 7f213c2cd640 1 -- start start 2024-04-25T11:41:53.028 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.039+0000 7f213c2cd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2134104d20 0x7f213419a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:53.028 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.039+0000 7f213c2cd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f213410b780 0x7f213419ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:53.028 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.039+0000 7f213c2cd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f213410e0e0 0x7f213419ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:53.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.039+0000 7f213c2cd640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2134113d90 con 0x7f213410e0e0 2024-04-25T11:41:53.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.039+0000 7f213c2cd640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2134113c10 con 0x7f213410b780 2024-04-25T11:41:53.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.039+0000 7f2139868640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f213410b780 0x7f213419ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:53.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.039+0000 7f2139868640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f213410b780 0x7f213419ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:59568/0 (socket says 172.21.15.90:59568) 2024-04-25T11:41:53.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.039+0000 7f213c2cd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2134113f10 con 0x7f2134104d20 2024-04-25T11:41:53.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.039+0000 7f2139868640 1 -- 172.21.15.90:0/22962514 learned_addr learned my addr 172.21.15.90:0/22962514 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:53.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.039+0000 7f213a069640 1 --2- 172.21.15.90:0/22962514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2134104d20 0x7f213419a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:53.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.039+0000 7f213a86a640 1 --2- 172.21.15.90:0/22962514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f213410e0e0 0x7f213419ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:53.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.040+0000 7f2139868640 1 -- 172.21.15.90:0/22962514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2134104d20 msgr2=0x7f213419a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:53.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.040+0000 7f2139868640 1 --2- 172.21.15.90:0/22962514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2134104d20 0x7f213419a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.040+0000 7f2139868640 1 -- 172.21.15.90:0/22962514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f213410e0e0 msgr2=0x7f213419ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:53.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.040+0000 7f2139868640 1 --2- 172.21.15.90:0/22962514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f213410e0e0 0x7f213419ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.040+0000 7f2139868640 1 -- 172.21.15.90:0/22962514 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f213419f590 con 0x7f213410b780 2024-04-25T11:41:53.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.040+0000 7f2139868640 1 --2- 172.21.15.90:0/22962514 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f213410b780 0x7f213419ac90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f213000e9c0 tx=0x7f213000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:53.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.040+0000 7f21277fe640 1 -- 172.21.15.90:0/22962514 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f213000cde0 con 0x7f213410b780 2024-04-25T11:41:53.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.040+0000 7f21277fe640 1 -- 172.21.15.90:0/22962514 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21300035f0 con 0x7f213410b780 2024-04-25T11:41:53.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.040+0000 7f21277fe640 1 -- 172.21.15.90:0/22962514 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2130010690 con 0x7f213410b780 2024-04-25T11:41:53.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.040+0000 7f213c2cd640 1 -- 172.21.15.90:0/22962514 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f213419f880 con 0x7f213410b780 2024-04-25T11:41:53.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.040+0000 7f213c2cd640 1 -- 172.21.15.90:0/22962514 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f213419fd80 con 0x7f213410b780 2024-04-25T11:41:53.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.043+0000 7f21277fe640 1 -- 172.21.15.90:0/22962514 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f2130010830 con 0x7f213410b780 2024-04-25T11:41:53.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.043+0000 7f2143178740 1 Processor -- start 2024-04-25T11:41:53.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.044+0000 7f2143178740 1 -- start start 2024-04-25T11:41:53.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.044+0000 7f2143178740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e8161b0 0x555c4ebb5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:53.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.044+0000 7f21277fe640 1 --2- 172.21.15.90:0/22962514 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f21200afc50 0x7f21200b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:53.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.044+0000 7f2143178740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c4e7ed2f0 0x555c4ebb6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:53.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.044+0000 7f2143178740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c4e7f54d0 0x555c4ebb6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:53.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.044+0000 7f2143178740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555c4eb1c460 con 0x555c4e7f54d0 2024-04-25T11:41:53.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.044+0000 7f2143178740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x555c4ead6590 con 0x555c4e8161b0 2024-04-25T11:41:53.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.044+0000 7f2143178740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x555c4eb18420 con 0x555c4e7ed2f0 2024-04-25T11:41:53.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.044+0000 7f213a86a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e8161b0 0x555c4ebb5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:53.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.044+0000 7f213a86a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e8161b0 0x555c4ebb5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:59584/0 (socket says 172.21.15.90:59584) 2024-04-25T11:41:53.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.044+0000 7f213a86a640 1 -- 172.21.15.90:0/161393388 learned_addr learned my addr 172.21.15.90:0/161393388 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:53.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.044+0000 7f213a86a640 1 --2- 172.21.15.90:0/161393388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c4e7f54d0 0x555c4ebb6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:53.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.044+0000 7f213a069640 1 --2- 172.21.15.90:0/22962514 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f21200afc50 0x7f21200b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:53.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.044+0000 7f21277fe640 1 -- 172.21.15.90:0/22962514 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f21300df3c0 con 0x7f213410b780 2024-04-25T11:41:53.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.045+0000 7f2139868640 1 --2- 172.21.15.90:0/161393388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c4e7ed2f0 0x555c4ebb6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:53.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.045+0000 7f213a86a640 1 -- 172.21.15.90:0/161393388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c4e7ed2f0 msgr2=0x555c4ebb6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:53.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.045+0000 7f213a86a640 1 --2- 172.21.15.90:0/161393388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c4e7ed2f0 0x555c4ebb6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.045+0000 7f213a86a640 1 -- 172.21.15.90:0/161393388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c4e7f54d0 msgr2=0x555c4ebb6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:53.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.045+0000 7f213a86a640 1 --2- 172.21.15.90:0/161393388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c4e7f54d0 0x555c4ebb6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.045+0000 7f213a86a640 1 -- 172.21.15.90:0/161393388 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c4eae30d0 con 0x555c4e8161b0 2024-04-25T11:41:53.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.045+0000 7f213a86a640 1 --2- 172.21.15.90:0/161393388 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e8161b0 0x555c4ebb5af0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f212c005f40 tx=0x7f212c005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=107fa490961bd5d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:53.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.045+0000 7f213a069640 1 --2- 172.21.15.90:0/22962514 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f21200afc50 0x7f21200b2160 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f212802a9c0 tx=0x7f21280058b0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:53.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.045+0000 7f2124bfe640 1 -- 172.21.15.90:0/161393388 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f212c011130 con 0x555c4e8161b0 2024-04-25T11:41:53.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.045+0000 7f2124bfe640 1 -- 172.21.15.90:0/161393388 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f212c0112d0 con 0x555c4e8161b0 2024-04-25T11:41:53.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.045+0000 7f2124bfe640 1 -- 172.21.15.90:0/161393388 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f212c00fc00 con 0x555c4e8161b0 2024-04-25T11:41:53.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.046+0000 7f2143178740 1 -- 172.21.15.90:0/161393388 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e8161b0 msgr2=0x555c4ebb5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:53.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.046+0000 7f2143178740 1 --2- 172.21.15.90:0/161393388 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e8161b0 0x555c4ebb5af0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f212c005f40 tx=0x7f212c005bd0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.046+0000 7f2143178740 1 -- 172.21.15.90:0/161393388 shutdown_connections 2024-04-25T11:41:53.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.046+0000 7f2143178740 1 --2- 172.21.15.90:0/161393388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c4e7f54d0 0x555c4ebb6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.046+0000 7f2143178740 1 --2- 172.21.15.90:0/161393388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c4e7ed2f0 0x555c4ebb6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.046+0000 7f2143178740 1 --2- 172.21.15.90:0/161393388 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e8161b0 0x555c4ebb5af0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.046+0000 7f2143178740 1 -- 172.21.15.90:0/161393388 >> 172.21.15.90:0/161393388 conn(0x555c4e80a300 msgr2=0x555c4ebb22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:53.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.047+0000 7f2143178740 1 -- 172.21.15.90:0/161393388 shutdown_connections 2024-04-25T11:41:53.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.047+0000 7f2143178740 1 -- 172.21.15.90:0/161393388 wait complete. 2024-04-25T11:41:53.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.047+0000 7f2143178740 1 -- start start 2024-04-25T11:41:53.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.047+0000 7f2143178740 1 Processor -- start 2024-04-25T11:41:53.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.047+0000 7f2143178740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c4e7ed2f0 0x555c4ebc1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:53.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.047+0000 7f2143178740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c4e7f54d0 0x555c4ebc20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:53.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.047+0000 7f2143178740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e8161b0 0x555c4ebc0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:53.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.047+0000 7f2143178740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555c4ebc2c20 con 0x555c4e7f54d0 2024-04-25T11:41:53.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.047+0000 7f2143178740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x555c4eaa3b60 con 0x555c4e8161b0 2024-04-25T11:41:53.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.047+0000 7f2143178740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x555c4ebc2da0 con 0x555c4e7ed2f0 2024-04-25T11:41:53.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.047+0000 7f2143178740 10 client.0 _ll_register_callbacks cb 0x555c4eac82b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:53.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.048+0000 7f213a86a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e8161b0 0x555c4ebc0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:53.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.048+0000 7f2139868640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c4e7f54d0 0x555c4ebc20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:53.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.048+0000 7f213a86a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e8161b0 0x555c4ebc0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:59594/0 (socket says 172.21.15.90:59594) 2024-04-25T11:41:53.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.048+0000 7f213a86a640 1 -- 172.21.15.90:0/2971762578 learned_addr learned my addr 172.21.15.90:0/2971762578 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:53.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.048+0000 7f213a069640 1 --2- 172.21.15.90:0/2971762578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c4e7ed2f0 0x555c4ebc1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:53.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.048+0000 7f213a86a640 1 -- 172.21.15.90:0/2971762578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c4e7ed2f0 msgr2=0x555c4ebc1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:53.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.048+0000 7f213a86a640 1 --2- 172.21.15.90:0/2971762578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c4e7ed2f0 0x555c4ebc1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.048+0000 7f213a86a640 1 -- 172.21.15.90:0/2971762578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c4e7f54d0 msgr2=0x555c4ebc20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:53.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.048+0000 7f213a86a640 1 --2- 172.21.15.90:0/2971762578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c4e7f54d0 0x555c4ebc20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.048+0000 7f213a86a640 1 -- 172.21.15.90:0/2971762578 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c4e82e210 con 0x555c4e8161b0 2024-04-25T11:41:53.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.048+0000 7f2139868640 1 --2- 172.21.15.90:0/2971762578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c4e7f54d0 0x555c4ebc20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:53.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.048+0000 7f213a069640 1 --2- 172.21.15.90:0/2971762578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c4e7ed2f0 0x555c4ebc1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:53.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.048+0000 7f213a86a640 1 --2- 172.21.15.90:0/2971762578 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e8161b0 0x555c4ebc0030 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f212c009a70 tx=0x7f212c003ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0ded96fe9b2911d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:53.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.048+0000 7f211dffb640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:41:53.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.048+0000 7f211dffb640 1 -- 172.21.15.90:0/2971762578 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f212c011130 con 0x555c4e8161b0 2024-04-25T11:41:53.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.048+0000 7f211dffb640 1 -- 172.21.15.90:0/2971762578 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f212c00fb80 con 0x555c4e8161b0 2024-04-25T11:41:53.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.048+0000 7f2143178740 1 -- 172.21.15.90:0/2971762578 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f212c0112d0 con 0x555c4e8161b0 2024-04-25T11:41:53.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.049+0000 7f211dffb640 1 -- 172.21.15.90:0/2971762578 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f212c00fd80 con 0x555c4e8161b0 2024-04-25T11:41:53.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.049+0000 7f2143178740 10 client.26576 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:53.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.049+0000 7f2143178740 1 -- 172.21.15.90:0/2971762578 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555c4e8fa620 con 0x555c4e8161b0 2024-04-25T11:41:53.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.049+0000 7f211dffb640 1 -- 172.21.15.90:0/2971762578 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f212c0112d0 con 0x555c4e8161b0 2024-04-25T11:41:53.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.049+0000 7f211dffb640 1 -- 172.21.15.90:0/2971762578 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f212c027020 con 0x555c4e8161b0 2024-04-25T11:41:53.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.049+0000 7f2143178740 10 client.26576 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:53.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.049+0000 7f2143178740 10 client.26576 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:41:53.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.049+0000 7f2143178740 20 client.26576 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:53.043 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.049+0000 7f2143178740 1 --2- 172.21.15.90:0/2971762578 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x555c4e7d4d80 0x555c4ebd2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:53.043 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.049+0000 7f2143178740 4 client.26576 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:41:53.043 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.049+0000 7f2143178740 1 -- 172.21.15.90:0/2971762578 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c4eb2a070 con 0x555c4e7d4d80 2024-04-25T11:41:53.043 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.049+0000 7f213a069640 1 --2- 172.21.15.90:0/2971762578 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x555c4e7d4d80 0x555c4ebd2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:53.043 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.050+0000 7f211dffb640 10 client.26576 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:53.043 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.050+0000 7f213a069640 1 --2- 172.21.15.90:0/2971762578 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x555c4e7d4d80 0x555c4ebd2650 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b935b80886b65720 server_cookie=5414d220f58cfd62 in_seq=0 out_seq=0 2024-04-25T11:41:53.043 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.050+0000 7f211dffb640 1 -- 172.21.15.90:0/2971762578 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x555c4eb2a070 con 0x555c4e7d4d80 2024-04-25T11:41:53.043 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.051+0000 7f211dffb640 10 client.26576 handle_command_reply: tid=0 2024-04-25T11:41:53.043 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.051+0000 7f2143178740 1 client.26576 shutdown 2024-04-25T11:41:53.043 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.051+0000 7f2143178740 1 -- 172.21.15.90:0/2971762578 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e8161b0 msgr2=0x555c4ebc0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:53.043 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.051+0000 7f2143178740 1 --2- 172.21.15.90:0/2971762578 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e8161b0 0x555c4ebc0030 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f212c009a70 tx=0x7f212c003ce0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.043 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.051+0000 7f2143178740 1 -- 172.21.15.90:0/2971762578 shutdown_connections 2024-04-25T11:41:53.043 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.051+0000 7f2143178740 1 --2- 172.21.15.90:0/2971762578 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x555c4e7d4d80 0x555c4ebd2650 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.044 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.051+0000 7f2143178740 1 --2- 172.21.15.90:0/2971762578 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e8161b0 0x555c4ebc0030 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.044 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.051+0000 7f2143178740 1 --2- 172.21.15.90:0/2971762578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c4e7f54d0 0x555c4ebc20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.044 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.051+0000 7f2143178740 1 --2- 172.21.15.90:0/2971762578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c4e7ed2f0 0x555c4ebc1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.044 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.051+0000 7f2143178740 1 -- 172.21.15.90:0/2971762578 >> 172.21.15.90:0/2971762578 conn(0x555c4e80a300 msgr2=0x555c4ebb58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:53.044 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.051+0000 7f211dffb640 0 client.26576 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:53.044 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.051+0000 7f2143178740 1 -- 172.21.15.90:0/2971762578 shutdown_connections 2024-04-25T11:41:53.044 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.051+0000 7f2143178740 1 -- 172.21.15.90:0/2971762578 wait complete. 2024-04-25T11:41:53.044 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.051+0000 7f2143178740 20 client.26576 trim_cache size 0 max 16384 2024-04-25T11:41:53.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.068+0000 7f2143178740 1 Processor -- start 2024-04-25T11:41:53.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.068+0000 7f2143178740 1 -- start start 2024-04-25T11:41:53.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.068+0000 7f2143178740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c4e7ed2f0 0x555c4ebd7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:53.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.068+0000 7f2143178740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e7f54d0 0x555c4ebd8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:53.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.068+0000 7f2143178740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c4e8161b0 0x555c4ebb1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:53.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.068+0000 7f2143178740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f21341ada40 con 0x555c4e8161b0 2024-04-25T11:41:53.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.068+0000 7f2143178740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f21341af580 con 0x555c4e7f54d0 2024-04-25T11:41:53.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.068+0000 7f2143178740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f21341ae110 con 0x555c4e7ed2f0 2024-04-25T11:41:53.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.068+0000 7f213a86a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c4e8161b0 0x555c4ebb1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:53.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.068+0000 7f2139868640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e7f54d0 0x555c4ebd8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:53.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.068+0000 7f213a069640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c4e7ed2f0 0x555c4ebd7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:53.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.068+0000 7f213a86a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c4e8161b0 0x555c4ebb1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46288/0 (socket says 172.21.15.90:46288) 2024-04-25T11:41:53.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.068+0000 7f2139868640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e7f54d0 0x555c4ebd8240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:59598/0 (socket says 172.21.15.90:59598) 2024-04-25T11:41:53.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.068+0000 7f213a86a640 1 -- 172.21.15.90:0/1558261211 learned_addr learned my addr 172.21.15.90:0/1558261211 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:53.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.068+0000 7f213a069640 1 -- 172.21.15.90:0/1558261211 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e7f54d0 msgr2=0x555c4ebd8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:53.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.068+0000 7f213a069640 1 --2- 172.21.15.90:0/1558261211 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e7f54d0 0x555c4ebd8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.068+0000 7f213a069640 1 -- 172.21.15.90:0/1558261211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c4e8161b0 msgr2=0x555c4ebb1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:53.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.068+0000 7f213a069640 1 --2- 172.21.15.90:0/1558261211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c4e8161b0 0x555c4ebb1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.068+0000 7f213a069640 1 -- 172.21.15.90:0/1558261211 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c4ebc1f00 con 0x555c4e7ed2f0 2024-04-25T11:41:53.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.068+0000 7f213a86a640 1 --2- 172.21.15.90:0/1558261211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c4e8161b0 0x555c4ebb1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:53.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.069+0000 7f213a069640 1 --2- 172.21.15.90:0/1558261211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c4e7ed2f0 0x555c4ebd7d00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2128005940 tx=0x7f2128040000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=878b1a185c922983 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:53.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.069+0000 7f211ffff640 1 -- 172.21.15.90:0/1558261211 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21280046c0 con 0x555c4e7ed2f0 2024-04-25T11:41:53.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.069+0000 7f211ffff640 1 -- 172.21.15.90:0/1558261211 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2128004d10 con 0x555c4e7ed2f0 2024-04-25T11:41:53.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.069+0000 7f211ffff640 1 -- 172.21.15.90:0/1558261211 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2128054990 con 0x555c4e7ed2f0 2024-04-25T11:41:53.062 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.070+0000 7f2143178740 1 -- 172.21.15.90:0/1558261211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c4e7ed2f0 msgr2=0x555c4ebd7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:53.062 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.070+0000 7f2143178740 1 --2- 172.21.15.90:0/1558261211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c4e7ed2f0 0x555c4ebd7d00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2128005940 tx=0x7f2128040000 comp rx=0 tx=0).stop 2024-04-25T11:41:53.062 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.070+0000 7f2143178740 1 -- 172.21.15.90:0/1558261211 shutdown_connections 2024-04-25T11:41:53.062 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.070+0000 7f2143178740 1 --2- 172.21.15.90:0/1558261211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c4e8161b0 0x555c4ebb1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.062 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.070+0000 7f2143178740 1 --2- 172.21.15.90:0/1558261211 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e7f54d0 0x555c4ebd8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.062 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.070+0000 7f2143178740 1 --2- 172.21.15.90:0/1558261211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c4e7ed2f0 0x555c4ebd7d00 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.062 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.070+0000 7f2143178740 1 -- 172.21.15.90:0/1558261211 >> 172.21.15.90:0/1558261211 conn(0x555c4e80a300 msgr2=0x555c4ebbe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:53.062 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.070+0000 7f2143178740 1 -- 172.21.15.90:0/1558261211 shutdown_connections 2024-04-25T11:41:53.063 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.070+0000 7f2143178740 1 -- 172.21.15.90:0/1558261211 wait complete. 2024-04-25T11:41:53.063 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.071+0000 7f2143178740 1 -- start start 2024-04-25T11:41:53.063 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.071+0000 7f2143178740 1 Processor -- start 2024-04-25T11:41:53.063 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.072+0000 7f2143178740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e7ed2f0 0x555c4ebc85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:53.063 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.072+0000 7f2143178740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c4e7f54d0 0x555c4ebc4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:53.063 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.072+0000 7f2143178740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c4e8161b0 0x555c4ebc4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:53.063 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.072+0000 7f2143178740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f21341aeeb0 con 0x555c4e8161b0 2024-04-25T11:41:53.063 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.072+0000 7f2143178740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f21341ae7e0 con 0x555c4e7ed2f0 2024-04-25T11:41:53.063 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.072+0000 7f2143178740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f21341aba20 con 0x555c4e7f54d0 2024-04-25T11:41:53.063 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.072+0000 7f2143178740 10 client.0 _ll_register_callbacks cb 0x7f212c0115d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:41:53.063 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.072+0000 7f213a069640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e7ed2f0 0x555c4ebc85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:53.063 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.072+0000 7f2139868640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c4e7f54d0 0x555c4ebc4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:53.063 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.072+0000 7f213a069640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e7ed2f0 0x555c4ebc85b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:59614/0 (socket says 172.21.15.90:59614) 2024-04-25T11:41:53.064 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.072+0000 7f213a069640 1 -- 172.21.15.90:0/1291131073 learned_addr learned my addr 172.21.15.90:0/1291131073 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:41:53.064 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.072+0000 7f213a86a640 1 --2- 172.21.15.90:0/1291131073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c4e8161b0 0x555c4ebc4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:53.064 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.072+0000 7f2139868640 1 -- 172.21.15.90:0/1291131073 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e7ed2f0 msgr2=0x555c4ebc85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:53.064 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.072+0000 7f2139868640 1 --2- 172.21.15.90:0/1291131073 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e7ed2f0 0x555c4ebc85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.064 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.072+0000 7f2139868640 1 -- 172.21.15.90:0/1291131073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c4e8161b0 msgr2=0x555c4ebc4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:53.064 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.072+0000 7f2139868640 1 --2- 172.21.15.90:0/1291131073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c4e8161b0 0x555c4ebc4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.064 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.072+0000 7f2139868640 1 -- 172.21.15.90:0/1291131073 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c4ebc55e0 con 0x555c4e7f54d0 2024-04-25T11:41:53.064 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.072+0000 7f213a069640 1 --2- 172.21.15.90:0/1291131073 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e7ed2f0 0x555c4ebc85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:53.064 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.072+0000 7f2139868640 1 --2- 172.21.15.90:0/1291131073 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c4e7f54d0 0x555c4ebc4960 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f21300dcab0 tx=0x7f2130003cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=375534da98add2b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:53.064 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.072+0000 7f211dffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:41:53.064 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.072+0000 7f211dffb640 1 -- 172.21.15.90:0/1291131073 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21300dcf40 con 0x555c4e7f54d0 2024-04-25T11:41:53.064 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.073+0000 7f211dffb640 1 -- 172.21.15.90:0/1291131073 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f21300094f0 con 0x555c4e7f54d0 2024-04-25T11:41:53.064 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.073+0000 7f2143178740 1 -- 172.21.15.90:0/1291131073 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2128004d10 con 0x555c4e7f54d0 2024-04-25T11:41:53.065 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.073+0000 7f211dffb640 1 -- 172.21.15.90:0/1291131073 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f21300ab8b0 con 0x555c4e7f54d0 2024-04-25T11:41:53.065 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.073+0000 7f2143178740 10 client.26524 fetch_fsmap learned FSMap version 28 2024-04-25T11:41:53.065 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.073+0000 7f2143178740 1 -- 172.21.15.90:0/1291131073 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555c4ec39810 con 0x555c4e7f54d0 2024-04-25T11:41:53.065 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.073+0000 7f211dffb640 1 -- 172.21.15.90:0/1291131073 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2128004d10 con 0x555c4e7f54d0 2024-04-25T11:41:53.065 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.073+0000 7f211dffb640 1 -- 172.21.15.90:0/1291131073 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f21300a7760 con 0x555c4e7f54d0 2024-04-25T11:41:53.065 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.073+0000 7f2143178740 10 client.26524 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:41:53.065 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.073+0000 7f2143178740 10 client.26524 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:41:53.065 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.073+0000 7f2143178740 20 client.26524 populate_metadata read hostname 'smithi090' 2024-04-25T11:41:53.065 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.073+0000 7f2143178740 1 --2- 172.21.15.90:0/1291131073 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x555c4e7aa1c0 0x555c4ec3c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:53.065 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.073+0000 7f2143178740 4 client.26524 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:41:53.065 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.073+0000 7f2143178740 1 -- 172.21.15.90:0/1291131073 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555c4eb23ee0 con 0x555c4e7aa1c0 2024-04-25T11:41:53.065 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.073+0000 7f213a069640 1 --2- 172.21.15.90:0/1291131073 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x555c4e7aa1c0 0x555c4ec3c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:53.065 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.074+0000 7f211dffb640 10 client.26524 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:53.066 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:53.074+0000 7f213a069640 1 --2- 172.21.15.90:0/1291131073 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x555c4e7aa1c0 0x555c4ec3c020 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b20e494dc7a6de61 server_cookie=1962e893f39b5279 in_seq=0 out_seq=0 2024-04-25T11:41:54.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:53 smithi138 ceph-mon[32328]: pgmap v207: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 2.4 MiB/s rd, 537 MiB/s wr, 1.03k op/s 2024-04-25T11:41:54.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:53 smithi156 ceph-mon[33045]: pgmap v207: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 2.4 MiB/s rd, 537 MiB/s wr, 1.03k op/s 2024-04-25T11:41:54.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:53 smithi090 ceph-mon[29575]: pgmap v207: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 2.4 MiB/s rd, 537 MiB/s wr, 1.03k op/s 2024-04-25T11:41:55.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.623+0000 7f211dffb640 1 -- 172.21.15.90:0/1291131073 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555c4eb23ee0 con 0x555c4e7aa1c0 2024-04-25T11:41:55.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.623+0000 7f211dffb640 10 client.26524 handle_command_reply: tid=0 2024-04-25T11:41:55.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.623+0000 7f2143178740 1 client.26524 shutdown 2024-04-25T11:41:55.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.623+0000 7f2143178740 1 -- 172.21.15.90:0/1291131073 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c4e7f54d0 msgr2=0x555c4ebc4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:55.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.623+0000 7f2143178740 1 --2- 172.21.15.90:0/1291131073 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c4e7f54d0 0x555c4ebc4960 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f21300dcab0 tx=0x7f2130003cd0 comp rx=0 tx=0).stop 2024-04-25T11:41:55.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.623+0000 7f2143178740 1 -- 172.21.15.90:0/1291131073 shutdown_connections 2024-04-25T11:41:55.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.623+0000 7f2143178740 1 --2- 172.21.15.90:0/1291131073 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x555c4e7aa1c0 0x555c4ec3c020 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:55.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.623+0000 7f2143178740 1 --2- 172.21.15.90:0/1291131073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555c4e8161b0 0x555c4ebc4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:55.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.623+0000 7f2143178740 1 --2- 172.21.15.90:0/1291131073 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555c4e7f54d0 0x555c4ebc4960 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:55.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.623+0000 7f2143178740 1 --2- 172.21.15.90:0/1291131073 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555c4e7ed2f0 0x555c4ebc85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:55.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.623+0000 7f2143178740 1 -- 172.21.15.90:0/1291131073 >> 172.21.15.90:0/1291131073 conn(0x555c4e80a300 msgr2=0x555c4ebb5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:55.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.623+0000 7f211dffb640 0 client.26524 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:41:55.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.623+0000 7f2143178740 1 -- 172.21.15.90:0/1291131073 shutdown_connections 2024-04-25T11:41:55.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.624+0000 7f2143178740 1 -- 172.21.15.90:0/1291131073 wait complete. 2024-04-25T11:41:55.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.624+0000 7f2143178740 20 client.26524 trim_cache size 0 max 16384 2024-04-25T11:41:55.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:41:55.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:41:55.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:41:55.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:41:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.627+0000 7f213c2cd640 1 -- 172.21.15.90:0/22962514 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f21200afc50 msgr2=0x7f21200b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.627+0000 7f213c2cd640 1 --2- 172.21.15.90:0/22962514 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f21200afc50 0x7f21200b2160 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f212802a9c0 tx=0x7f21280058b0 comp rx=0 tx=0).stop 2024-04-25T11:41:55.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.627+0000 7f213c2cd640 1 -- 172.21.15.90:0/22962514 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f213410b780 msgr2=0x7f213419ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:55.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.627+0000 7f213c2cd640 1 --2- 172.21.15.90:0/22962514 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f213410b780 0x7f213419ac90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f213000e9c0 tx=0x7f213000ee80 comp rx=0 tx=0).stop 2024-04-25T11:41:55.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.628+0000 7f213c2cd640 1 -- 172.21.15.90:0/22962514 shutdown_connections 2024-04-25T11:41:55.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.628+0000 7f213c2cd640 1 --2- 172.21.15.90:0/22962514 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f21200afc50 0x7f21200b2160 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:55.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.628+0000 7f213c2cd640 1 --2- 172.21.15.90:0/22962514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f213410e0e0 0x7f213419ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:55.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.628+0000 7f213c2cd640 1 --2- 172.21.15.90:0/22962514 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f213410b780 0x7f213419ac90 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:55.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.628+0000 7f213c2cd640 1 --2- 172.21.15.90:0/22962514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2134104d20 0x7f213419a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:55.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.628+0000 7f213c2cd640 1 -- 172.21.15.90:0/22962514 >> 172.21.15.90:0/22962514 conn(0x7f2134078150 msgr2=0x7f2134101240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:55.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.628+0000 7f213c2cd640 1 -- 172.21.15.90:0/22962514 shutdown_connections 2024-04-25T11:41:55.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:41:55.628+0000 7f213c2cd640 1 -- 172.21.15.90:0/22962514 wait complete. 2024-04-25T11:41:56.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:55 smithi090 ceph-mon[29575]: pgmap v208: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 2.4 MiB/s rd, 505 MiB/s wr, 987 op/s 2024-04-25T11:41:56.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:55 smithi138 ceph-mon[32328]: pgmap v208: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 2.4 MiB/s rd, 505 MiB/s wr, 987 op/s 2024-04-25T11:41:56.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:55 smithi156 ceph-mon[33045]: pgmap v208: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 2.4 MiB/s rd, 505 MiB/s wr, 987 op/s 2024-04-25T11:41:58.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:41:57 smithi090 ceph-mon[29575]: pgmap v209: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 889 GiB / 1.0 TiB avail; 3.4 MiB/s rd, 621 MiB/s wr, 1.03k op/s 2024-04-25T11:41:58.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:41:58 smithi138 ceph-mon[32328]: pgmap v209: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 889 GiB / 1.0 TiB avail; 3.4 MiB/s rd, 621 MiB/s wr, 1.03k op/s 2024-04-25T11:41:58.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:41:57 smithi156 ceph-mon[33045]: pgmap v209: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 889 GiB / 1.0 TiB avail; 3.4 MiB/s rd, 621 MiB/s wr, 1.03k op/s 2024-04-25T11:42:00.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:00 smithi138 ceph-mon[32328]: pgmap v210: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 889 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 386 MiB/s wr, 776 op/s 2024-04-25T11:42:00.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:00 smithi156 ceph-mon[33045]: pgmap v210: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 889 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 386 MiB/s wr, 776 op/s 2024-04-25T11:42:00.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:00 smithi090 ceph-mon[29575]: pgmap v210: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 889 GiB / 1.0 TiB avail; 2.3 MiB/s rd, 386 MiB/s wr, 776 op/s 2024-04-25T11:42:02.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:02 smithi156 ceph-mon[33045]: pgmap v211: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 3.5 MiB/s rd, 564 MiB/s wr, 842 op/s 2024-04-25T11:42:02.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:02 smithi090 ceph-mon[29575]: pgmap v211: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 3.5 MiB/s rd, 564 MiB/s wr, 842 op/s 2024-04-25T11:42:02.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:02 smithi138 ceph-mon[32328]: pgmap v211: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 3.5 MiB/s rd, 564 MiB/s wr, 842 op/s 2024-04-25T11:42:03.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:03 smithi090 ceph-mon[29575]: pgmap v212: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 2.2 MiB/s rd, 442 MiB/s wr, 316 op/s 2024-04-25T11:42:03.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:03 smithi138 ceph-mon[32328]: pgmap v212: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 2.2 MiB/s rd, 442 MiB/s wr, 316 op/s 2024-04-25T11:42:03.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:03 smithi156 ceph-mon[33045]: pgmap v212: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 2.2 MiB/s rd, 442 MiB/s wr, 316 op/s 2024-04-25T11:42:04.231 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:42:04.231 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:42:04.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.457+0000 7fec4e630640 1 -- 172.21.15.90:0/1546017919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec48107930 msgr2=0x7fec48102d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:04.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.457+0000 7fec4e630640 1 --2- 172.21.15.90:0/1546017919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec48107930 0x7fec48102d60 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fec38009900 tx=0x7fec3802a440 comp rx=0 tx=0).stop 2024-04-25T11:42:04.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.457+0000 7fec4e630640 1 -- 172.21.15.90:0/1546017919 shutdown_connections 2024-04-25T11:42:04.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.457+0000 7fec4e630640 1 --2- 172.21.15.90:0/1546017919 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fec4810d420 0x7fec4810f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.457+0000 7fec4e630640 1 --2- 172.21.15.90:0/1546017919 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec481032a0 0x7fec481036b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.457+0000 7fec4e630640 1 --2- 172.21.15.90:0/1546017919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec48107930 0x7fec48102d60 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.457+0000 7fec4e630640 1 -- 172.21.15.90:0/1546017919 >> 172.21.15.90:0/1546017919 conn(0x7fec48077880 msgr2=0x7fec48077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:04.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.457+0000 7fec4e630640 1 -- 172.21.15.90:0/1546017919 shutdown_connections 2024-04-25T11:42:04.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.457+0000 7fec4e630640 1 -- 172.21.15.90:0/1546017919 wait complete. 2024-04-25T11:42:04.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.458+0000 7fec4e630640 1 Processor -- start 2024-04-25T11:42:04.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.458+0000 7fec4e630640 1 -- start start 2024-04-25T11:42:04.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.458+0000 7fec4e630640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec481032a0 0x7fec48110280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.458+0000 7fec4e630640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fec48107930 0x7fec481107c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.458+0000 7fec4e630640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec4810d420 0x7fec48110d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.458+0000 7fec4e630640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fec48113e30 con 0x7fec481032a0 2024-04-25T11:42:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.458+0000 7fec4e630640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fec48113cb0 con 0x7fec48107930 2024-04-25T11:42:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.458+0000 7fec4e630640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fec48113fb0 con 0x7fec4810d420 2024-04-25T11:42:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.458+0000 7fec47fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec481032a0 0x7fec48110280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.458+0000 7fec47fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec481032a0 0x7fec48110280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58096/0 (socket says 172.21.15.90:58096) 2024-04-25T11:42:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.458+0000 7fec47fff640 1 -- 172.21.15.90:0/10397869 learned_addr learned my addr 172.21.15.90:0/10397869 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.459+0000 7fec47fff640 1 -- 172.21.15.90:0/10397869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec4810d420 msgr2=0x7fec48110d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.459+0000 7fec47fff640 1 --2- 172.21.15.90:0/10397869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec4810d420 0x7fec48110d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.459+0000 7fec47fff640 1 -- 172.21.15.90:0/10397869 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fec48107930 msgr2=0x7fec481107c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:42:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.459+0000 7fec47fff640 1 --2- 172.21.15.90:0/10397869 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fec48107930 0x7fec481107c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.459+0000 7fec47fff640 1 -- 172.21.15.90:0/10397869 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec481a8bf0 con 0x7fec481032a0 2024-04-25T11:42:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.459+0000 7fec47fff640 1 --2- 172.21.15.90:0/10397869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec481032a0 0x7fec48110280 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fec3802a9b0 tx=0x7fec38003e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.459+0000 7fec457fa640 1 -- 172.21.15.90:0/10397869 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3803f030 con 0x7fec481032a0 2024-04-25T11:42:04.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.460+0000 7fec457fa640 1 -- 172.21.15.90:0/10397869 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec38032bf0 con 0x7fec481032a0 2024-04-25T11:42:04.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.460+0000 7fec4e630640 1 -- 172.21.15.90:0/10397869 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec481a8e80 con 0x7fec481032a0 2024-04-25T11:42:04.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.460+0000 7fec457fa640 1 -- 172.21.15.90:0/10397869 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec38035070 con 0x7fec481032a0 2024-04-25T11:42:04.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.460+0000 7fec4e630640 1 -- 172.21.15.90:0/10397869 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec481a92e0 con 0x7fec481032a0 2024-04-25T11:42:04.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.464+0000 7fec457fa640 1 -- 172.21.15.90:0/10397869 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fec38042030 con 0x7fec481032a0 2024-04-25T11:42:04.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.464+0000 7fec555b1740 1 Processor -- start 2024-04-25T11:42:04.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.464+0000 7fec555b1740 1 -- start start 2024-04-25T11:42:04.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.465+0000 7fec555b1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c800e390 0x55e2c83b7270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:04.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.466+0000 7fec555b1740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e2c7fe54d0 0x55e2c83b77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:04.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.466+0000 7fec555b1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2c7fed6b0 0x55e2c83c4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:04.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.466+0000 7fec555b1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e2c8314d80 con 0x55e2c800e390 2024-04-25T11:42:04.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.466+0000 7fec555b1740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55e2c82ce8b0 con 0x55e2c7fe54d0 2024-04-25T11:42:04.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.466+0000 7fec555b1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e2c8310d40 con 0x55e2c7fed6b0 2024-04-25T11:42:04.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.466+0000 7fec457fa640 1 --2- 172.21.15.90:0/10397869 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fec300afd10 0x7fec300b2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:04.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.466+0000 7fec457fa640 1 -- 172.21.15.90:0/10397869 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fec380ff720 con 0x7fec481032a0 2024-04-25T11:42:04.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.466+0000 7fec4cbcd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c800e390 0x55e2c83b7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:04.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.466+0000 7fec4cbcd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c800e390 0x55e2c83b7270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58104/0 (socket says 172.21.15.90:58104) 2024-04-25T11:42:04.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.466+0000 7fec4cbcd640 1 -- 172.21.15.90:0/1643088365 learned_addr learned my addr 172.21.15.90:0/1643088365 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:04.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.466+0000 7fec4cbcd640 1 --2- 172.21.15.90:0/10397869 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fec300afd10 0x7fec300b2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:04.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.466+0000 7fec47fff640 1 --2- 172.21.15.90:0/1643088365 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e2c7fe54d0 0x55e2c83b77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:04.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.467+0000 7fec4cbcd640 1 -- 172.21.15.90:0/1643088365 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2c7fed6b0 msgr2=0x55e2c83c4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:04.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.467+0000 7fec4cbcd640 1 --2- 172.21.15.90:0/1643088365 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2c7fed6b0 0x55e2c83c4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.467+0000 7fec4cbcd640 1 -- 172.21.15.90:0/1643088365 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e2c7fe54d0 msgr2=0x55e2c83b77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:04.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.467+0000 7fec4cbcd640 1 --2- 172.21.15.90:0/1643088365 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e2c7fe54d0 0x55e2c83b77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.467+0000 7fec4cbcd640 1 -- 172.21.15.90:0/1643088365 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2c829fa40 con 0x55e2c800e390 2024-04-25T11:42:04.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.468+0000 7fec4cbcd640 1 --2- 172.21.15.90:0/10397869 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fec300afd10 0x7fec300b2220 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fec400145b0 tx=0x7fec40019000 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:04.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.468+0000 7fec4cbcd640 1 --2- 172.21.15.90:0/1643088365 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c800e390 0x55e2c83b7270 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fec40012760 tx=0x7fec40012c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28a91c114a3f7ceb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:04.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.480+0000 7fec2e7fc640 1 -- 172.21.15.90:0/1643088365 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec40014d10 con 0x55e2c800e390 2024-04-25T11:42:04.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.480+0000 7fec2e7fc640 1 -- 172.21.15.90:0/1643088365 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec4000b8c0 con 0x55e2c800e390 2024-04-25T11:42:04.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.480+0000 7fec2e7fc640 1 -- 172.21.15.90:0/1643088365 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec40006090 con 0x55e2c800e390 2024-04-25T11:42:04.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.481+0000 7fec555b1740 1 -- 172.21.15.90:0/1643088365 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c800e390 msgr2=0x55e2c83b7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:04.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.481+0000 7fec555b1740 1 --2- 172.21.15.90:0/1643088365 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c800e390 0x55e2c83b7270 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fec40012760 tx=0x7fec40012c20 comp rx=0 tx=0).stop 2024-04-25T11:42:04.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.481+0000 7fec555b1740 1 -- 172.21.15.90:0/1643088365 shutdown_connections 2024-04-25T11:42:04.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.481+0000 7fec555b1740 1 --2- 172.21.15.90:0/1643088365 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2c7fed6b0 0x55e2c83c4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.482+0000 7fec555b1740 1 --2- 172.21.15.90:0/1643088365 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e2c7fe54d0 0x55e2c83b77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.482+0000 7fec555b1740 1 --2- 172.21.15.90:0/1643088365 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c800e390 0x55e2c83b7270 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.482+0000 7fec555b1740 1 -- 172.21.15.90:0/1643088365 >> 172.21.15.90:0/1643088365 conn(0x55e2c80024e0 msgr2=0x55e2c83b3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:04.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.482+0000 7fec555b1740 1 -- 172.21.15.90:0/1643088365 shutdown_connections 2024-04-25T11:42:04.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.482+0000 7fec555b1740 1 -- 172.21.15.90:0/1643088365 wait complete. 2024-04-25T11:42:04.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.482+0000 7fec555b1740 1 -- start start 2024-04-25T11:42:04.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.483+0000 7fec555b1740 1 Processor -- start 2024-04-25T11:42:04.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.483+0000 7fec555b1740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e2c7fe54d0 0x55e2c83c1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:04.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.483+0000 7fec555b1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c7fed6b0 0x55e2c83c2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:04.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.483+0000 7fec555b1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2c800e390 0x55e2c83c7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:04.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.484+0000 7fec555b1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e2c83c4790 con 0x55e2c7fed6b0 2024-04-25T11:42:04.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.484+0000 7fec555b1740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55e2c829bd10 con 0x55e2c7fe54d0 2024-04-25T11:42:04.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.484+0000 7fec555b1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e2c83c4910 con 0x55e2c800e390 2024-04-25T11:42:04.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.484+0000 7fec555b1740 10 client.0 _ll_register_callbacks cb 0x55e2c82c8210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:04.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.484+0000 7fec477fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c7fed6b0 0x55e2c83c2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:04.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.484+0000 7fec477fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c7fed6b0 0x55e2c83c2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58114/0 (socket says 172.21.15.90:58114) 2024-04-25T11:42:04.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.484+0000 7fec477fe640 1 -- 172.21.15.90:0/2451348520 learned_addr learned my addr 172.21.15.90:0/2451348520 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:04.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.484+0000 7fec477fe640 1 -- 172.21.15.90:0/2451348520 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2c800e390 msgr2=0x55e2c83c7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:04.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.484+0000 7fec477fe640 1 --2- 172.21.15.90:0/2451348520 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2c800e390 0x55e2c83c7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.484+0000 7fec477fe640 1 -- 172.21.15.90:0/2451348520 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e2c7fe54d0 msgr2=0x55e2c83c1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:04.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.484+0000 7fec477fe640 1 --2- 172.21.15.90:0/2451348520 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e2c7fe54d0 0x55e2c83c1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.484+0000 7fec477fe640 1 -- 172.21.15.90:0/2451348520 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2c80f07d0 con 0x55e2c7fed6b0 2024-04-25T11:42:04.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.485+0000 7fec477fe640 1 --2- 172.21.15.90:0/2451348520 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c7fed6b0 0x55e2c83c2340 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fec3c009dd0 tx=0x7fec3c00b4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eba250a9ac35d155 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:04.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.485+0000 7fec277fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:04.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.485+0000 7fec277fe640 1 -- 172.21.15.90:0/2451348520 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3c015020 con 0x55e2c7fed6b0 2024-04-25T11:42:04.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.485+0000 7fec277fe640 1 -- 172.21.15.90:0/2451348520 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fec3c011420 con 0x55e2c7fed6b0 2024-04-25T11:42:04.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.485+0000 7fec555b1740 1 -- 172.21.15.90:0/2451348520 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec4000b8c0 con 0x55e2c7fed6b0 2024-04-25T11:42:04.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.485+0000 7fec277fe640 1 -- 172.21.15.90:0/2451348520 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fec3c020050 con 0x55e2c7fed6b0 2024-04-25T11:42:04.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.485+0000 7fec555b1740 10 client.17640 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:04.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.485+0000 7fec555b1740 1 -- 172.21.15.90:0/2451348520 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e2c80263f0 con 0x55e2c7fed6b0 2024-04-25T11:42:04.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.486+0000 7fec277fe640 1 -- 172.21.15.90:0/2451348520 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec4000b8c0 con 0x55e2c7fed6b0 2024-04-25T11:42:04.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.486+0000 7fec277fe640 1 -- 172.21.15.90:0/2451348520 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fec3c0197e0 con 0x55e2c7fed6b0 2024-04-25T11:42:04.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.486+0000 7fec555b1740 10 client.17640 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:04.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.486+0000 7fec555b1740 10 client.17640 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:42:04.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.486+0000 7fec555b1740 20 client.17640 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:04.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.486+0000 7fec555b1740 1 --2- 172.21.15.90:0/2451348520 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e2c7fccf60 0x55e2c83d40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:04.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.486+0000 7fec555b1740 4 client.17640 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:42:04.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.486+0000 7fec555b1740 1 -- 172.21.15.90:0/2451348520 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e2c8322990 con 0x55e2c7fccf60 2024-04-25T11:42:04.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.487+0000 7fec47fff640 1 --2- 172.21.15.90:0/2451348520 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e2c7fccf60 0x55e2c83d40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:04.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.489+0000 7fec277fe640 10 client.17640 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:04.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.489+0000 7fec47fff640 1 --2- 172.21.15.90:0/2451348520 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e2c7fccf60 0x55e2c83d40f0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c730abb77c061b7 server_cookie=3e7107fee3c4900f in_seq=0 out_seq=0 2024-04-25T11:42:04.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.490+0000 7fec277fe640 1 -- 172.21.15.90:0/2451348520 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7fec380a2e50 con 0x55e2c7fccf60 2024-04-25T11:42:04.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.490+0000 7fec277fe640 10 client.17640 handle_command_reply: tid=0 2024-04-25T11:42:04.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.490+0000 7fec555b1740 1 client.17640 shutdown 2024-04-25T11:42:04.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.490+0000 7fec555b1740 1 -- 172.21.15.90:0/2451348520 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c7fed6b0 msgr2=0x55e2c83c2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:04.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.490+0000 7fec555b1740 1 --2- 172.21.15.90:0/2451348520 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c7fed6b0 0x55e2c83c2340 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fec3c009dd0 tx=0x7fec3c00b4b0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.490+0000 7fec555b1740 1 -- 172.21.15.90:0/2451348520 shutdown_connections 2024-04-25T11:42:04.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.490+0000 7fec555b1740 1 --2- 172.21.15.90:0/2451348520 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e2c7fccf60 0x55e2c83d40f0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.490+0000 7fec555b1740 1 --2- 172.21.15.90:0/2451348520 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2c800e390 0x55e2c83c7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.490+0000 7fec555b1740 1 --2- 172.21.15.90:0/2451348520 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c7fed6b0 0x55e2c83c2340 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.491+0000 7fec555b1740 1 --2- 172.21.15.90:0/2451348520 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e2c7fe54d0 0x55e2c83c1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.491+0000 7fec555b1740 1 -- 172.21.15.90:0/2451348520 >> 172.21.15.90:0/2451348520 conn(0x55e2c80024e0 msgr2=0x55e2c83b8f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:04.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.491+0000 7fec277fe640 0 client.17640 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:04.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.491+0000 7fec555b1740 1 -- 172.21.15.90:0/2451348520 shutdown_connections 2024-04-25T11:42:04.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.491+0000 7fec555b1740 1 -- 172.21.15.90:0/2451348520 wait complete. 2024-04-25T11:42:04.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.491+0000 7fec555b1740 20 client.17640 trim_cache size 0 max 16384 2024-04-25T11:42:04.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.510+0000 7fec555b1740 1 Processor -- start 2024-04-25T11:42:04.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.510+0000 7fec555b1740 1 -- start start 2024-04-25T11:42:04.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.510+0000 7fec555b1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2c7fe54d0 0x55e2c83aa3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:04.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.510+0000 7fec555b1740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e2c800e390 0x55e2c83bbf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:04.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.510+0000 7fec555b1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c7fa23a0 0x55e2c83bc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:04.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.510+0000 7fec555b1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fec481ad9d0 con 0x55e2c7fa23a0 2024-04-25T11:42:04.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.510+0000 7fec555b1740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fec481af510 con 0x55e2c800e390 2024-04-25T11:42:04.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.510+0000 7fec555b1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fec481ae0a0 con 0x55e2c7fe54d0 2024-04-25T11:42:04.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.511+0000 7fec4cbcd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c7fa23a0 0x55e2c83bc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:04.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.511+0000 7fec4cbcd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c7fa23a0 0x55e2c83bc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58126/0 (socket says 172.21.15.90:58126) 2024-04-25T11:42:04.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.511+0000 7fec4cbcd640 1 -- 172.21.15.90:0/4173207323 learned_addr learned my addr 172.21.15.90:0/4173207323 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:04.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.511+0000 7fec477fe640 1 --2- 172.21.15.90:0/4173207323 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e2c800e390 0x55e2c83bbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:04.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.511+0000 7fec47fff640 1 --2- 172.21.15.90:0/4173207323 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2c7fe54d0 0x55e2c83aa3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:04.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.511+0000 7fec4cbcd640 1 -- 172.21.15.90:0/4173207323 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2c7fe54d0 msgr2=0x55e2c83aa3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:04.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.511+0000 7fec4cbcd640 1 --2- 172.21.15.90:0/4173207323 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2c7fe54d0 0x55e2c83aa3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.511+0000 7fec4cbcd640 1 -- 172.21.15.90:0/4173207323 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e2c800e390 msgr2=0x55e2c83bbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:04.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.511+0000 7fec4cbcd640 1 --2- 172.21.15.90:0/4173207323 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e2c800e390 0x55e2c83bbf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.511+0000 7fec4cbcd640 1 -- 172.21.15.90:0/4173207323 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2c7fe60f0 con 0x55e2c7fa23a0 2024-04-25T11:42:04.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.511+0000 7fec47fff640 1 --2- 172.21.15.90:0/4173207323 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2c7fe54d0 0x55e2c83aa3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:42:04.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.511+0000 7fec4cbcd640 1 --2- 172.21.15.90:0/4173207323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c7fa23a0 0x55e2c83bc4b0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fec40021000 tx=0x7fec40009010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d7adc6ec4519983 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:04.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.514+0000 7fec2dffb640 1 -- 172.21.15.90:0/4173207323 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec40014d10 con 0x55e2c7fa23a0 2024-04-25T11:42:04.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.514+0000 7fec2dffb640 1 -- 172.21.15.90:0/4173207323 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec40018c60 con 0x55e2c7fa23a0 2024-04-25T11:42:04.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.514+0000 7fec2dffb640 1 -- 172.21.15.90:0/4173207323 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec400196e0 con 0x55e2c7fa23a0 2024-04-25T11:42:04.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.515+0000 7fec555b1740 1 -- 172.21.15.90:0/4173207323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c7fa23a0 msgr2=0x55e2c83bc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:04.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.515+0000 7fec555b1740 1 --2- 172.21.15.90:0/4173207323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c7fa23a0 0x55e2c83bc4b0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fec40021000 tx=0x7fec40009010 comp rx=0 tx=0).stop 2024-04-25T11:42:04.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.515+0000 7fec555b1740 1 -- 172.21.15.90:0/4173207323 shutdown_connections 2024-04-25T11:42:04.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.515+0000 7fec555b1740 1 --2- 172.21.15.90:0/4173207323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c7fa23a0 0x55e2c83bc4b0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.515+0000 7fec555b1740 1 --2- 172.21.15.90:0/4173207323 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e2c800e390 0x55e2c83bbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.515+0000 7fec555b1740 1 --2- 172.21.15.90:0/4173207323 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2c7fe54d0 0x55e2c83aa3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.515+0000 7fec555b1740 1 -- 172.21.15.90:0/4173207323 >> 172.21.15.90:0/4173207323 conn(0x55e2c80024e0 msgr2=0x55e2c83c0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:04.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.515+0000 7fec555b1740 1 -- 172.21.15.90:0/4173207323 shutdown_connections 2024-04-25T11:42:04.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.515+0000 7fec555b1740 1 -- 172.21.15.90:0/4173207323 wait complete. 2024-04-25T11:42:04.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.515+0000 7fec555b1740 1 -- start start 2024-04-25T11:42:04.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.516+0000 7fec555b1740 1 Processor -- start 2024-04-25T11:42:04.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.516+0000 7fec555b1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2c7fa23a0 0x55e2c83c9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:04.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.516+0000 7fec555b1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c7fe54d0 0x55e2c83c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:04.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.516+0000 7fec555b1740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e2c800e390 0x55e2c83c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:04.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.516+0000 7fec555b1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fec481aee40 con 0x55e2c7fe54d0 2024-04-25T11:42:04.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.516+0000 7fec555b1740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fec481ae770 con 0x55e2c800e390 2024-04-25T11:42:04.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.516+0000 7fec555b1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fec481aba30 con 0x55e2c7fa23a0 2024-04-25T11:42:04.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.516+0000 7fec555b1740 10 client.0 _ll_register_callbacks cb 0x7fec3c006e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:04.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.516+0000 7fec477fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c7fe54d0 0x55e2c83c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:04.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.516+0000 7fec4cbcd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e2c800e390 0x55e2c83c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:04.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.516+0000 7fec47fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2c7fa23a0 0x55e2c83c9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:04.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.516+0000 7fec477fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c7fe54d0 0x55e2c83c96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58134/0 (socket says 172.21.15.90:58134) 2024-04-25T11:42:04.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.516+0000 7fec4cbcd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e2c800e390 0x55e2c83c9c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:34534/0 (socket says 172.21.15.90:34534) 2024-04-25T11:42:04.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.516+0000 7fec477fe640 1 -- 172.21.15.90:0/1701670217 learned_addr learned my addr 172.21.15.90:0/1701670217 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:04.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.516+0000 7fec4cbcd640 1 -- 172.21.15.90:0/1701670217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2c7fa23a0 msgr2=0x55e2c83c9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:04.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.516+0000 7fec4cbcd640 1 --2- 172.21.15.90:0/1701670217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2c7fa23a0 0x55e2c83c9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.516+0000 7fec4cbcd640 1 -- 172.21.15.90:0/1701670217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c7fe54d0 msgr2=0x55e2c83c96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:04.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.516+0000 7fec4cbcd640 1 --2- 172.21.15.90:0/1701670217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c7fe54d0 0x55e2c83c96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.516+0000 7fec4cbcd640 1 -- 172.21.15.90:0/1701670217 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2c82d54b0 con 0x55e2c800e390 2024-04-25T11:42:04.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.517+0000 7fec4cbcd640 1 --2- 172.21.15.90:0/1701670217 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e2c800e390 0x55e2c83c9c00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fec4000bd50 tx=0x7fec400189e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f14e0b463057dde3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:04.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.517+0000 7fec277fe640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:42:04.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.521+0000 7fec277fe640 1 -- 172.21.15.90:0/1701670217 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec40004e50 con 0x55e2c800e390 2024-04-25T11:42:04.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.521+0000 7fec277fe640 1 -- 172.21.15.90:0/1701670217 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fec40012f50 con 0x55e2c800e390 2024-04-25T11:42:04.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.521+0000 7fec555b1740 1 -- 172.21.15.90:0/1701670217 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec40018c60 con 0x55e2c800e390 2024-04-25T11:42:04.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.521+0000 7fec277fe640 1 -- 172.21.15.90:0/1701670217 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fec40042050 con 0x55e2c800e390 2024-04-25T11:42:04.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.521+0000 7fec555b1740 10 client.26588 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:04.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.521+0000 7fec555b1740 1 -- 172.21.15.90:0/1701670217 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e2c843b490 con 0x55e2c800e390 2024-04-25T11:42:04.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.521+0000 7fec277fe640 1 -- 172.21.15.90:0/1701670217 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec40018c60 con 0x55e2c800e390 2024-04-25T11:42:04.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.521+0000 7fec277fe640 1 -- 172.21.15.90:0/1701670217 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fec400364f0 con 0x55e2c800e390 2024-04-25T11:42:04.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.521+0000 7fec555b1740 10 client.26588 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:04.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.521+0000 7fec555b1740 10 client.26588 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:42:04.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.521+0000 7fec555b1740 20 client.26588 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:04.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.521+0000 7fec555b1740 1 --2- 172.21.15.90:0/1701670217 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e2c843b900 0x55e2c843dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:04.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.521+0000 7fec555b1740 4 client.26588 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:42:04.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.521+0000 7fec555b1740 1 -- 172.21.15.90:0/1701670217 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e2c831c800 con 0x55e2c843b900 2024-04-25T11:42:04.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.522+0000 7fec47fff640 1 --2- 172.21.15.90:0/1701670217 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e2c843b900 0x55e2c843dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:04.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.522+0000 7fec277fe640 10 client.26588 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:04.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.522+0000 7fec47fff640 1 --2- 172.21.15.90:0/1701670217 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e2c843b900 0x55e2c843dd20 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc34057005d4b09d server_cookie=865e27e705a51325 in_seq=0 out_seq=0 2024-04-25T11:42:04.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.522+0000 7fec277fe640 1 -- 172.21.15.90:0/1701670217 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e2c831c800 con 0x55e2c843b900 2024-04-25T11:42:04.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.522+0000 7fec277fe640 10 client.26588 handle_command_reply: tid=0 2024-04-25T11:42:04.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.522+0000 7fec555b1740 1 client.26588 shutdown 2024-04-25T11:42:04.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.522+0000 7fec555b1740 1 -- 172.21.15.90:0/1701670217 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e2c800e390 msgr2=0x55e2c83c9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:04.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.522+0000 7fec555b1740 1 --2- 172.21.15.90:0/1701670217 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e2c800e390 0x55e2c83c9c00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fec4000bd50 tx=0x7fec400189e0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.522+0000 7fec555b1740 1 -- 172.21.15.90:0/1701670217 shutdown_connections 2024-04-25T11:42:04.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.522+0000 7fec555b1740 1 --2- 172.21.15.90:0/1701670217 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e2c843b900 0x55e2c843dd20 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.522+0000 7fec555b1740 1 --2- 172.21.15.90:0/1701670217 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e2c800e390 0x55e2c83c9c00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.522+0000 7fec555b1740 1 --2- 172.21.15.90:0/1701670217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e2c7fe54d0 0x55e2c83c96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.522+0000 7fec555b1740 1 --2- 172.21.15.90:0/1701670217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e2c7fa23a0 0x55e2c83c9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.522+0000 7fec277fe640 0 client.26588 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:04.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.522+0000 7fec555b1740 1 -- 172.21.15.90:0/1701670217 >> 172.21.15.90:0/1701670217 conn(0x55e2c80024e0 msgr2=0x55e2c83aa150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:04.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.522+0000 7fec555b1740 1 -- 172.21.15.90:0/1701670217 shutdown_connections 2024-04-25T11:42:04.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.522+0000 7fec555b1740 1 -- 172.21.15.90:0/1701670217 wait complete. 2024-04-25T11:42:04.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.523+0000 7fec555b1740 20 client.26588 trim_cache size 0 max 16384 2024-04-25T11:42:04.511 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:42:04.511 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-25T11:42:04.511 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-25T11:42:04.511 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:42:04.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.523+0000 7fec4e630640 1 -- 172.21.15.90:0/10397869 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fec300afd10 msgr2=0x7fec300b2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:04.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.523+0000 7fec4e630640 1 --2- 172.21.15.90:0/10397869 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fec300afd10 0x7fec300b2220 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fec400145b0 tx=0x7fec40019000 comp rx=0 tx=0).stop 2024-04-25T11:42:04.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.523+0000 7fec4e630640 1 -- 172.21.15.90:0/10397869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec481032a0 msgr2=0x7fec48110280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:04.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.523+0000 7fec4e630640 1 --2- 172.21.15.90:0/10397869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec481032a0 0x7fec48110280 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fec3802a9b0 tx=0x7fec38003e30 comp rx=0 tx=0).stop 2024-04-25T11:42:04.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.523+0000 7fec4e630640 1 -- 172.21.15.90:0/10397869 shutdown_connections 2024-04-25T11:42:04.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.523+0000 7fec4e630640 1 --2- 172.21.15.90:0/10397869 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fec300afd10 0x7fec300b2220 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.523+0000 7fec4e630640 1 --2- 172.21.15.90:0/10397869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec4810d420 0x7fec48110d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.523+0000 7fec4e630640 1 --2- 172.21.15.90:0/10397869 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fec48107930 0x7fec481107c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.523+0000 7fec4e630640 1 --2- 172.21.15.90:0/10397869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec481032a0 0x7fec48110280 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.523+0000 7fec4e630640 1 -- 172.21.15.90:0/10397869 >> 172.21.15.90:0/10397869 conn(0x7fec48077880 msgr2=0x7fec481008c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:04.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.523+0000 7fec4e630640 1 -- 172.21.15.90:0/10397869 shutdown_connections 2024-04-25T11:42:04.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.523+0000 7fec4e630640 1 -- 172.21.15.90:0/10397869 wait complete. 2024-04-25T11:42:04.521 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T11:42:04.521 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:42:04.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.647+0000 7fdb0b842640 1 -- 172.21.15.90:0/3451354219 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb04108a20 msgr2=0x7fdb041025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:04.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.647+0000 7fdb0b842640 1 --2- 172.21.15.90:0/3451354219 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb04108a20 0x7fdb041025b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fdb00009900 tx=0x7fdb0002a410 comp rx=0 tx=0).stop 2024-04-25T11:42:04.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.647+0000 7fdb0b842640 1 -- 172.21.15.90:0/3451354219 shutdown_connections 2024-04-25T11:42:04.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.647+0000 7fdb0b842640 1 --2- 172.21.15.90:0/3451354219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb0410d380 0x7fdb0410f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.647+0000 7fdb0b842640 1 --2- 172.21.15.90:0/3451354219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdb04102af0 0x7fdb04102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.647+0000 7fdb0b842640 1 --2- 172.21.15.90:0/3451354219 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb04108a20 0x7fdb041025b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.647+0000 7fdb0b842640 1 -- 172.21.15.90:0/3451354219 >> 172.21.15.90:0/3451354219 conn(0x7fdb04078110 msgr2=0x7fdb04078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:04.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.647+0000 7fdb0b842640 1 -- 172.21.15.90:0/3451354219 shutdown_connections 2024-04-25T11:42:04.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.647+0000 7fdb0b842640 1 -- 172.21.15.90:0/3451354219 wait complete. 2024-04-25T11:42:04.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.648+0000 7fdb0b842640 1 Processor -- start 2024-04-25T11:42:04.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.648+0000 7fdb0b842640 1 -- start start 2024-04-25T11:42:04.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.648+0000 7fdb0b842640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb04102af0 0x7fdb04199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:04.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.648+0000 7fdb0b842640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb04108a20 0x7fdb04199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:04.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.648+0000 7fdb0b842640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdb0410d380 0x7fdb0419de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:04.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.648+0000 7fdb0b842640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdb04112be0 con 0x7fdb04102af0 2024-04-25T11:42:04.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.648+0000 7fdb0b842640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fdb04112a60 con 0x7fdb04108a20 2024-04-25T11:42:04.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.648+0000 7fdb0b842640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdb04112d60 con 0x7fdb0410d380 2024-04-25T11:42:04.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.648+0000 7fdb095de640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb04102af0 0x7fdb04199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:04.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.648+0000 7fdb095de640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb04102af0 0x7fdb04199600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58162/0 (socket says 172.21.15.90:58162) 2024-04-25T11:42:04.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.648+0000 7fdb095de640 1 -- 172.21.15.90:0/167949890 learned_addr learned my addr 172.21.15.90:0/167949890 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:04.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.648+0000 7fdb08ddd640 1 --2- 172.21.15.90:0/167949890 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb04108a20 0x7fdb04199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:04.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.648+0000 7fdb09ddf640 1 --2- 172.21.15.90:0/167949890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdb0410d380 0x7fdb0419de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:04.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.649+0000 7fdb095de640 1 -- 172.21.15.90:0/167949890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdb0410d380 msgr2=0x7fdb0419de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:04.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.649+0000 7fdb095de640 1 --2- 172.21.15.90:0/167949890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdb0410d380 0x7fdb0419de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.649+0000 7fdb095de640 1 -- 172.21.15.90:0/167949890 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb04108a20 msgr2=0x7fdb04199b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:04.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.649+0000 7fdb095de640 1 --2- 172.21.15.90:0/167949890 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb04108a20 0x7fdb04199b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.649+0000 7fdb095de640 1 -- 172.21.15.90:0/167949890 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb0419e460 con 0x7fdb04102af0 2024-04-25T11:42:04.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.649+0000 7fdb095de640 1 --2- 172.21.15.90:0/167949890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb04102af0 0x7fdb04199600 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fdb00032fd0 tx=0x7fdb00031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:04.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.649+0000 7fdafa7fc640 1 -- 172.21.15.90:0/167949890 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb0003f030 con 0x7fdb04102af0 2024-04-25T11:42:04.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.649+0000 7fdb0b842640 1 -- 172.21.15.90:0/167949890 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb0419e6f0 con 0x7fdb04102af0 2024-04-25T11:42:04.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.649+0000 7fdafa7fc640 1 -- 172.21.15.90:0/167949890 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb00004490 con 0x7fdb04102af0 2024-04-25T11:42:04.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.649+0000 7fdafa7fc640 1 -- 172.21.15.90:0/167949890 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb00035070 con 0x7fdb04102af0 2024-04-25T11:42:04.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.649+0000 7fdb0b842640 1 -- 172.21.15.90:0/167949890 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb0419ebe0 con 0x7fdb04102af0 2024-04-25T11:42:04.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.650+0000 7fdb0b842640 1 -- 172.21.15.90:0/167949890 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb04103cd0 con 0x7fdb04102af0 2024-04-25T11:42:04.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.651+0000 7fdafa7fc640 1 -- 172.21.15.90:0/167949890 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fdb00042030 con 0x7fdb04102af0 2024-04-25T11:42:04.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.652+0000 7fdafa7fc640 1 --2- 172.21.15.90:0/167949890 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fdaf00afc60 0x7fdaf00b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:04.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.653+0000 7fdb08ddd640 1 --2- 172.21.15.90:0/167949890 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fdaf00afc60 0x7fdaf00b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:04.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.653+0000 7fdafa7fc640 1 -- 172.21.15.90:0/167949890 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fdb000ff790 con 0x7fdb04102af0 2024-04-25T11:42:04.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.653+0000 7fdb08ddd640 1 --2- 172.21.15.90:0/167949890 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fdaf00afc60 0x7fdaf00b2170 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdb0410e7b0 tx=0x7fdaf4005f90 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:04.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.656+0000 7fdafa7fc640 1 -- 172.21.15.90:0/167949890 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdb000c7980 con 0x7fdb04102af0 2024-04-25T11:42:04.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.998+0000 7fdb0b842640 1 -- 172.21.15.90:0/167949890 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb0406afa0 con 0x7fdb04102af0 2024-04-25T11:42:04.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:04.999+0000 7fdafa7fc640 1 -- 172.21.15.90:0/167949890 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11930 (secure 0 0 0) 0x7fdb000cbc80 con 0x7fdb04102af0 2024-04-25T11:42:04.985 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:42:04.985 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:39:20.469254+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24458,24431,24499,24475,24524]},"id":1}]} 2024-04-25T11:42:04.986 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 28 2024-04-25T11:42:04.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.001+0000 7fdb0b842640 1 -- 172.21.15.90:0/167949890 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fdaf00afc60 msgr2=0x7fdaf00b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:04.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.001+0000 7fdb0b842640 1 --2- 172.21.15.90:0/167949890 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fdaf00afc60 0x7fdaf00b2170 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdb0410e7b0 tx=0x7fdaf4005f90 comp rx=0 tx=0).stop 2024-04-25T11:42:04.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.001+0000 7fdb0b842640 1 -- 172.21.15.90:0/167949890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb04102af0 msgr2=0x7fdb04199600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:04.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.001+0000 7fdb0b842640 1 --2- 172.21.15.90:0/167949890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb04102af0 0x7fdb04199600 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fdb00032fd0 tx=0x7fdb00031a20 comp rx=0 tx=0).stop 2024-04-25T11:42:04.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.002+0000 7fdb0b842640 1 -- 172.21.15.90:0/167949890 shutdown_connections 2024-04-25T11:42:04.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.002+0000 7fdb0b842640 1 --2- 172.21.15.90:0/167949890 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fdaf00afc60 0x7fdaf00b2170 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.002+0000 7fdb0b842640 1 --2- 172.21.15.90:0/167949890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdb0410d380 0x7fdb0419de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.002+0000 7fdb0b842640 1 --2- 172.21.15.90:0/167949890 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb04108a20 0x7fdb04199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.002+0000 7fdb0b842640 1 --2- 172.21.15.90:0/167949890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb04102af0 0x7fdb04199600 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:04.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.002+0000 7fdb0b842640 1 -- 172.21.15.90:0/167949890 >> 172.21.15.90:0/167949890 conn(0x7fdb04078110 msgr2=0x7fdb041040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:04.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.002+0000 7fdb0b842640 1 -- 172.21.15.90:0/167949890 shutdown_connections 2024-04-25T11:42:04.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.002+0000 7fdb0b842640 1 -- 172.21.15.90:0/167949890 wait complete. 2024-04-25T11:42:05.005 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T11:42:05.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.151+0000 7f72fb5ba640 1 -- 172.21.15.90:0/3575937261 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72f4104c30 msgr2=0x7f72f4105040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.151+0000 7f72fb5ba640 1 --2- 172.21.15.90:0/3575937261 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72f4104c30 0x7f72f4105040 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f72f0009900 tx=0x7f72f002a480 comp rx=0 tx=0).stop 2024-04-25T11:42:05.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.151+0000 7f72fb5ba640 1 -- 172.21.15.90:0/3575937261 shutdown_connections 2024-04-25T11:42:05.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.151+0000 7f72fb5ba640 1 --2- 172.21.15.90:0/3575937261 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f72f410e080 0x7f72f4110550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.151+0000 7f72fb5ba640 1 --2- 172.21.15.90:0/3575937261 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72f410b690 0x7f72f410dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.151+0000 7f72fb5ba640 1 --2- 172.21.15.90:0/3575937261 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72f4104c30 0x7f72f4105040 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.151+0000 7f72fb5ba640 1 -- 172.21.15.90:0/3575937261 >> 172.21.15.90:0/3575937261 conn(0x7f72f4078100 msgr2=0x7f72f4078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:05.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.151+0000 7f72fb5ba640 1 -- 172.21.15.90:0/3575937261 shutdown_connections 2024-04-25T11:42:05.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.151+0000 7f72fb5ba640 1 -- 172.21.15.90:0/3575937261 wait complete. 2024-04-25T11:42:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.151+0000 7f72fb5ba640 1 Processor -- start 2024-04-25T11:42:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.151+0000 7f72fb5ba640 1 -- start start 2024-04-25T11:42:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.152+0000 7f72fb5ba640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72f4104c30 0x7f72f41a2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.152+0000 7f72fb5ba640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f72f410b690 0x7f72f41a2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.152+0000 7f72fb5ba640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72f410e080 0x7f72f419c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.152+0000 7f72fb5ba640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f72f4113cc0 con 0x7f72f410e080 2024-04-25T11:42:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.152+0000 7f72fb5ba640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f72f4113b40 con 0x7f72f410b690 2024-04-25T11:42:05.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.152+0000 7f72fb5ba640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f72f4113e40 con 0x7f72f4104c30 2024-04-25T11:42:05.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.152+0000 7f72f9b57640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72f410e080 0x7f72f419c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.152+0000 7f72f9356640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72f4104c30 0x7f72f41a2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.152+0000 7f72f9b57640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72f410e080 0x7f72f419c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58182/0 (socket says 172.21.15.90:58182) 2024-04-25T11:42:05.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.152+0000 7f72f9356640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72f4104c30 0x7f72f41a2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:54366/0 (socket says 172.21.15.90:54366) 2024-04-25T11:42:05.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.152+0000 7f72f9b57640 1 -- 172.21.15.90:0/1532224630 learned_addr learned my addr 172.21.15.90:0/1532224630 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:05.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.152+0000 7f72f8b55640 1 --2- 172.21.15.90:0/1532224630 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f72f410b690 0x7f72f41a2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.152+0000 7f72f9b57640 1 -- 172.21.15.90:0/1532224630 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72f4104c30 msgr2=0x7f72f41a2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.152+0000 7f72f9b57640 1 --2- 172.21.15.90:0/1532224630 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72f4104c30 0x7f72f41a2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.152+0000 7f72f9b57640 1 -- 172.21.15.90:0/1532224630 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f72f410b690 msgr2=0x7f72f41a2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.152+0000 7f72f9b57640 1 --2- 172.21.15.90:0/1532224630 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f72f410b690 0x7f72f41a2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.152+0000 7f72f9b57640 1 -- 172.21.15.90:0/1532224630 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72f419cf40 con 0x7f72f410e080 2024-04-25T11:42:05.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.152+0000 7f72f9356640 1 --2- 172.21.15.90:0/1532224630 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72f4104c30 0x7f72f41a2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:42:05.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.152+0000 7f72f8b55640 1 --2- 172.21.15.90:0/1532224630 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f72f410b690 0x7f72f41a2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:42:05.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.152+0000 7f72f9b57640 1 --2- 172.21.15.90:0/1532224630 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72f410e080 0x7f72f419c860 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f72ec00eb10 tx=0x7f72ec00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:05.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.152+0000 7f72ea7fc640 1 -- 172.21.15.90:0/1532224630 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ec004b20 con 0x7f72f410e080 2024-04-25T11:42:05.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.153+0000 7f72ea7fc640 1 -- 172.21.15.90:0/1532224630 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72ec004cc0 con 0x7f72f410e080 2024-04-25T11:42:05.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.153+0000 7f72fb5ba640 1 -- 172.21.15.90:0/1532224630 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72f419d230 con 0x7f72f410e080 2024-04-25T11:42:05.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.153+0000 7f72ea7fc640 1 -- 172.21.15.90:0/1532224630 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ec010670 con 0x7f72f410e080 2024-04-25T11:42:05.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.153+0000 7f72fb5ba640 1 -- 172.21.15.90:0/1532224630 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72f41a9480 con 0x7f72f410e080 2024-04-25T11:42:05.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.156+0000 7f730246a740 1 Processor -- start 2024-04-25T11:42:05.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.156+0000 7f72ea7fc640 1 -- 172.21.15.90:0/1532224630 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f72ec005020 con 0x7f72f410e080 2024-04-25T11:42:05.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.156+0000 7f730246a740 1 -- start start 2024-04-25T11:42:05.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.156+0000 7f730246a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be508e390 0x558be542f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.156+0000 7f730246a740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558be50654d0 0x558be542fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.156+0000 7f730246a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be506d6b0 0x558be543c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.156+0000 7f730246a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558be5394d80 con 0x558be506d6b0 2024-04-25T11:42:05.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.156+0000 7f730246a740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x558be534ea20 con 0x558be50654d0 2024-04-25T11:42:05.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.156+0000 7f730246a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558be5390d40 con 0x558be508e390 2024-04-25T11:42:05.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.157+0000 7f72f9356640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558be50654d0 0x558be542fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.157+0000 7f72f8b55640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be506d6b0 0x558be543c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.157+0000 7f72f9356640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558be50654d0 0x558be542fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:34584/0 (socket says 172.21.15.90:34584) 2024-04-25T11:42:05.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.157+0000 7f72f9356640 1 -- 172.21.15.90:0/3525053943 learned_addr learned my addr 172.21.15.90:0/3525053943 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:05.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.157+0000 7f72f8b55640 1 --2- 172.21.15.90:0/3525053943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be508e390 0x558be542f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.157+0000 7f72ea7fc640 1 --2- 172.21.15.90:0/1532224630 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f72e00afc60 0x7f72e00b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.157+0000 7f72f9356640 1 -- 172.21.15.90:0/3525053943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be508e390 msgr2=0x558be542f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.157+0000 7f72f9356640 1 --2- 172.21.15.90:0/3525053943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be508e390 0x558be542f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.157+0000 7f72f9356640 1 -- 172.21.15.90:0/3525053943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be506d6b0 msgr2=0x558be543c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.157+0000 7f72f9356640 1 --2- 172.21.15.90:0/3525053943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be506d6b0 0x558be543c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.157+0000 7f72f9356640 1 -- 172.21.15.90:0/3525053943 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558be5359c30 con 0x558be50654d0 2024-04-25T11:42:05.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.157+0000 7f72f8b55640 1 --2- 172.21.15.90:0/3525053943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be506d6b0 0x558be543c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:42:05.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.157+0000 7f72f9b57640 1 --2- 172.21.15.90:0/1532224630 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f72e00afc60 0x7f72e00b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.157+0000 7f72ea7fc640 1 -- 172.21.15.90:0/1532224630 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f72ec0df610 con 0x7f72f410e080 2024-04-25T11:42:05.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.157+0000 7f72f9356640 1 --2- 172.21.15.90:0/3525053943 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558be50654d0 0x558be542fd50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f72f002a9a0 tx=0x7f72f0005e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b85396220e05bdb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:05.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.157+0000 7f72f9b57640 1 --2- 172.21.15.90:0/1532224630 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f72e00afc60 0x7f72e00b2170 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f72ec0dcdb0 tx=0x7f72ec090040 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:05.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.157+0000 7f72df7fe640 1 -- 172.21.15.90:0/3525053943 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f0040070 con 0x558be50654d0 2024-04-25T11:42:05.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.158+0000 7f72df7fe640 1 -- 172.21.15.90:0/3525053943 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72f0031720 con 0x558be50654d0 2024-04-25T11:42:05.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.158+0000 7f72df7fe640 1 -- 172.21.15.90:0/3525053943 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f0035040 con 0x558be50654d0 2024-04-25T11:42:05.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.158+0000 7f730246a740 1 -- 172.21.15.90:0/3525053943 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558be50654d0 msgr2=0x558be542fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.158+0000 7f730246a740 1 --2- 172.21.15.90:0/3525053943 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558be50654d0 0x558be542fd50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f72f002a9a0 tx=0x7f72f0005e20 comp rx=0 tx=0).stop 2024-04-25T11:42:05.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.158+0000 7f730246a740 1 -- 172.21.15.90:0/3525053943 shutdown_connections 2024-04-25T11:42:05.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.158+0000 7f730246a740 1 --2- 172.21.15.90:0/3525053943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be506d6b0 0x558be543c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.158+0000 7f730246a740 1 --2- 172.21.15.90:0/3525053943 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558be50654d0 0x558be542fd50 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.158+0000 7f730246a740 1 --2- 172.21.15.90:0/3525053943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be508e390 0x558be542f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.158+0000 7f730246a740 1 -- 172.21.15.90:0/3525053943 >> 172.21.15.90:0/3525053943 conn(0x558be50824e0 msgr2=0x558be542bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:05.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.159+0000 7f730246a740 1 -- 172.21.15.90:0/3525053943 shutdown_connections 2024-04-25T11:42:05.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.159+0000 7f730246a740 1 -- 172.21.15.90:0/3525053943 wait complete. 2024-04-25T11:42:05.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.159+0000 7f730246a740 1 -- start start 2024-04-25T11:42:05.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.159+0000 7f730246a740 1 Processor -- start 2024-04-25T11:42:05.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.159+0000 7f730246a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be50654d0 0x558be543a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.159+0000 7f730246a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be506d6b0 0x558be543a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.159+0000 7f730246a740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558be508e390 0x558be54461f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.159+0000 7f730246a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558be543cd30 con 0x558be506d6b0 2024-04-25T11:42:05.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.159+0000 7f730246a740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x558be531bdc0 con 0x558be508e390 2024-04-25T11:42:05.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.159+0000 7f730246a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558be543ceb0 con 0x558be50654d0 2024-04-25T11:42:05.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.160+0000 7f730246a740 10 client.0 _ll_register_callbacks cb 0x558be5337600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:05.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.160+0000 7f72f9356640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be506d6b0 0x558be543a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.160+0000 7f72f8b55640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558be508e390 0x558be54461f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.160+0000 7f72f8b55640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558be508e390 0x558be54461f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:34596/0 (socket says 172.21.15.90:34596) 2024-04-25T11:42:05.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.160+0000 7f72f8b55640 1 -- 172.21.15.90:0/3815891156 learned_addr learned my addr 172.21.15.90:0/3815891156 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:05.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.160+0000 7f72f9356640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be506d6b0 0x558be543a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58198/0 (socket says 172.21.15.90:58198) 2024-04-25T11:42:05.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.160+0000 7f72f8b55640 1 --2- 172.21.15.90:0/3815891156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be50654d0 0x558be543a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.160+0000 7f72f8b55640 1 -- 172.21.15.90:0/3815891156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be50654d0 msgr2=0x558be543a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.160+0000 7f72f8b55640 1 --2- 172.21.15.90:0/3815891156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be50654d0 0x558be543a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.160+0000 7f72f8b55640 1 -- 172.21.15.90:0/3815891156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be506d6b0 msgr2=0x558be543a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.160+0000 7f72f8b55640 1 --2- 172.21.15.90:0/3815891156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be506d6b0 0x558be543a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.160+0000 7f72f8b55640 1 -- 172.21.15.90:0/3815891156 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558be50a63f0 con 0x558be508e390 2024-04-25T11:42:05.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.160+0000 7f72f9356640 1 --2- 172.21.15.90:0/3815891156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be506d6b0 0x558be543a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:42:05.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.160+0000 7f72f8b55640 1 --2- 172.21.15.90:0/3815891156 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558be508e390 0x558be54461f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f72f410d010 tx=0x7f72e4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=740a03b1aacaa514 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:05.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.160+0000 7f72dcff9640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:42:05.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.160+0000 7f72dcff9640 1 -- 172.21.15.90:0/3815891156 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e4004230 con 0x558be508e390 2024-04-25T11:42:05.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.160+0000 7f72dcff9640 1 -- 172.21.15.90:0/3815891156 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f72e40043d0 con 0x558be508e390 2024-04-25T11:42:05.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.160+0000 7f730246a740 1 -- 172.21.15.90:0/3815891156 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72f0031720 con 0x558be508e390 2024-04-25T11:42:05.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.160+0000 7f72dcff9640 1 -- 172.21.15.90:0/3815891156 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f72e4027050 con 0x558be508e390 2024-04-25T11:42:05.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.160+0000 7f730246a740 10 client.26606 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:05.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.160+0000 7f730246a740 1 -- 172.21.15.90:0/3815891156 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558be51707d0 con 0x558be508e390 2024-04-25T11:42:05.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.161+0000 7f72dcff9640 1 -- 172.21.15.90:0/3815891156 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72f0031720 con 0x558be508e390 2024-04-25T11:42:05.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.161+0000 7f72dcff9640 1 -- 172.21.15.90:0/3815891156 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f72e400e830 con 0x558be508e390 2024-04-25T11:42:05.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.161+0000 7f730246a740 10 client.26606 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:05.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.161+0000 7f730246a740 10 client.26606 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:42:05.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.161+0000 7f730246a740 20 client.26606 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:05.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.161+0000 7f730246a740 1 --2- 172.21.15.90:0/3815891156 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x558be504cf60 0x558be544c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.161+0000 7f730246a740 4 client.26606 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:42:05.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.161+0000 7f730246a740 1 -- 172.21.15.90:0/3815891156 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558be53a2990 con 0x558be504cf60 2024-04-25T11:42:05.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.161+0000 7f72f9356640 1 --2- 172.21.15.90:0/3815891156 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x558be504cf60 0x558be544c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.161+0000 7f72dcff9640 10 client.26606 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:42:05.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.161+0000 7f72f9356640 1 --2- 172.21.15.90:0/3815891156 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x558be504cf60 0x558be544c780 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=254bc12d0bc08790 server_cookie=bc02adf137cbaa5a in_seq=0 out_seq=0 2024-04-25T11:42:05.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.162+0000 7f72dcff9640 1 -- 172.21.15.90:0/3815891156 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558be53a2990 con 0x558be504cf60 2024-04-25T11:42:05.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.162+0000 7f72dcff9640 10 client.26606 handle_command_reply: tid=0 2024-04-25T11:42:05.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.162+0000 7f730246a740 1 client.26606 shutdown 2024-04-25T11:42:05.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.162+0000 7f730246a740 1 -- 172.21.15.90:0/3815891156 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558be508e390 msgr2=0x558be54461f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.162+0000 7f730246a740 1 --2- 172.21.15.90:0/3815891156 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558be508e390 0x558be54461f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f72f410d010 tx=0x7f72e4004730 comp rx=0 tx=0).stop 2024-04-25T11:42:05.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.162+0000 7f730246a740 1 -- 172.21.15.90:0/3815891156 shutdown_connections 2024-04-25T11:42:05.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.162+0000 7f730246a740 1 --2- 172.21.15.90:0/3815891156 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x558be504cf60 0x558be544c780 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.162+0000 7f730246a740 1 --2- 172.21.15.90:0/3815891156 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558be508e390 0x558be54461f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.162+0000 7f730246a740 1 --2- 172.21.15.90:0/3815891156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be506d6b0 0x558be543a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.162+0000 7f730246a740 1 --2- 172.21.15.90:0/3815891156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be50654d0 0x558be543a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.162+0000 7f72dcff9640 0 client.26606 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:42:05.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.162+0000 7f730246a740 1 -- 172.21.15.90:0/3815891156 >> 172.21.15.90:0/3815891156 conn(0x558be50824e0 msgr2=0x558be5431530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:05.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.163+0000 7f730246a740 1 -- 172.21.15.90:0/3815891156 shutdown_connections 2024-04-25T11:42:05.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.163+0000 7f730246a740 1 -- 172.21.15.90:0/3815891156 wait complete. 2024-04-25T11:42:05.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.163+0000 7f730246a740 20 client.26606 trim_cache size 0 max 16384 2024-04-25T11:42:05.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.178+0000 7f730246a740 1 Processor -- start 2024-04-25T11:42:05.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.178+0000 7f730246a740 1 -- start start 2024-04-25T11:42:05.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.179+0000 7f730246a740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558be50654d0 0x558be5445e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.179+0000 7f730246a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be506d6b0 0x558be5434510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.179+0000 7f730246a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be508e390 0x558be542ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.179+0000 7f730246a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f72f41b1e10 con 0x558be506d6b0 2024-04-25T11:42:05.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.179+0000 7f730246a740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f72f41b3950 con 0x558be50654d0 2024-04-25T11:42:05.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.179+0000 7f730246a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f72f41b24e0 con 0x558be508e390 2024-04-25T11:42:05.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.179+0000 7f72f9356640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be506d6b0 0x558be5434510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.179+0000 7f72f9356640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be506d6b0 0x558be5434510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58214/0 (socket says 172.21.15.90:58214) 2024-04-25T11:42:05.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.179+0000 7f72f9356640 1 -- 172.21.15.90:0/2268769158 learned_addr learned my addr 172.21.15.90:0/2268769158 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:05.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.179+0000 7f72f8b55640 1 --2- 172.21.15.90:0/2268769158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be508e390 0x558be542ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.179+0000 7f72f8b55640 1 --2- 172.21.15.90:0/2268769158 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558be50654d0 0x558be5445e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.179+0000 7f72f9356640 1 -- 172.21.15.90:0/2268769158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be508e390 msgr2=0x558be542ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.179+0000 7f72f9356640 1 --2- 172.21.15.90:0/2268769158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be508e390 0x558be542ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.179+0000 7f72f9356640 1 -- 172.21.15.90:0/2268769158 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558be50654d0 msgr2=0x558be5445e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.179+0000 7f72f9356640 1 --2- 172.21.15.90:0/2268769158 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558be50654d0 0x558be5445e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.179+0000 7f72f9356640 1 -- 172.21.15.90:0/2268769158 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558be5431010 con 0x558be506d6b0 2024-04-25T11:42:05.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.179+0000 7f72f9356640 1 --2- 172.21.15.90:0/2268769158 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be506d6b0 0x558be5434510 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f72f0033000 tx=0x7f72f0005500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bceea978bdb8b6f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:05.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.180+0000 7f72deffd640 1 -- 172.21.15.90:0/2268769158 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f0035040 con 0x558be506d6b0 2024-04-25T11:42:05.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.180+0000 7f72deffd640 1 -- 172.21.15.90:0/2268769158 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72f0031900 con 0x558be506d6b0 2024-04-25T11:42:05.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.180+0000 7f72deffd640 1 -- 172.21.15.90:0/2268769158 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f0040040 con 0x558be506d6b0 2024-04-25T11:42:05.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.180+0000 7f730246a740 1 -- 172.21.15.90:0/2268769158 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be506d6b0 msgr2=0x558be5434510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.180+0000 7f730246a740 1 --2- 172.21.15.90:0/2268769158 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be506d6b0 0x558be5434510 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f72f0033000 tx=0x7f72f0005500 comp rx=0 tx=0).stop 2024-04-25T11:42:05.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.180+0000 7f730246a740 1 -- 172.21.15.90:0/2268769158 shutdown_connections 2024-04-25T11:42:05.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.180+0000 7f730246a740 1 --2- 172.21.15.90:0/2268769158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be508e390 0x558be542ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.180+0000 7f730246a740 1 --2- 172.21.15.90:0/2268769158 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be506d6b0 0x558be5434510 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.180+0000 7f730246a740 1 --2- 172.21.15.90:0/2268769158 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558be50654d0 0x558be5445e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.180+0000 7f730246a740 1 -- 172.21.15.90:0/2268769158 >> 172.21.15.90:0/2268769158 conn(0x558be50824e0 msgr2=0x558be542a610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:05.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.181+0000 7f730246a740 1 -- 172.21.15.90:0/2268769158 shutdown_connections 2024-04-25T11:42:05.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.181+0000 7f730246a740 1 -- 172.21.15.90:0/2268769158 wait complete. 2024-04-25T11:42:05.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.181+0000 7f730246a740 1 -- start start 2024-04-25T11:42:05.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.181+0000 7f730246a740 1 Processor -- start 2024-04-25T11:42:05.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.182+0000 7f730246a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be50654d0 0x558be543d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.182+0000 7f730246a740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558be506d6b0 0x558be543d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.182+0000 7f730246a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be508e390 0x558be543dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.182+0000 7f730246a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f72f41b3280 con 0x558be50654d0 2024-04-25T11:42:05.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.182+0000 7f730246a740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f72f41b2bb0 con 0x558be506d6b0 2024-04-25T11:42:05.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.182+0000 7f730246a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f72f41afdf0 con 0x558be508e390 2024-04-25T11:42:05.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.182+0000 7f730246a740 10 client.0 _ll_register_callbacks cb 0x7f72e4010430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:05.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.182+0000 7f72f8b55640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be50654d0 0x558be543d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.182+0000 7f72f8b55640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be50654d0 0x558be543d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58224/0 (socket says 172.21.15.90:58224) 2024-04-25T11:42:05.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.182+0000 7f72f8b55640 1 -- 172.21.15.90:0/751465269 learned_addr learned my addr 172.21.15.90:0/751465269 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:05.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.182+0000 7f72f9356640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558be506d6b0 0x558be543d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.182+0000 7f72f8b55640 1 --2- 172.21.15.90:0/751465269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be508e390 0x558be543dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.182+0000 7f72f9356640 1 -- 172.21.15.90:0/751465269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be508e390 msgr2=0x558be543dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.182+0000 7f72f9356640 1 --2- 172.21.15.90:0/751465269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be508e390 0x558be543dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.182+0000 7f72f9356640 1 -- 172.21.15.90:0/751465269 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be50654d0 msgr2=0x558be543d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.182+0000 7f72f9356640 1 --2- 172.21.15.90:0/751465269 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be50654d0 0x558be543d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.182+0000 7f72f9356640 1 -- 172.21.15.90:0/751465269 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558be54b2bb0 con 0x558be506d6b0 2024-04-25T11:42:05.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.182+0000 7f72f8b55640 1 --2- 172.21.15.90:0/751465269 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be50654d0 0x558be543d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:42:05.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.183+0000 7f72f9356640 1 --2- 172.21.15.90:0/751465269 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558be506d6b0 0x558be543d690 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x558be5430580 tx=0x7f72f0032f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65152ed60f18bbca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:05.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.183+0000 7f72dcff9640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:42:05.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.183+0000 7f72dcff9640 1 -- 172.21.15.90:0/751465269 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f005c070 con 0x558be506d6b0 2024-04-25T11:42:05.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.183+0000 7f72dcff9640 1 -- 172.21.15.90:0/751465269 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f72f004f620 con 0x558be506d6b0 2024-04-25T11:42:05.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.183+0000 7f730246a740 1 -- 172.21.15.90:0/751465269 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558be539c800 con 0x558be506d6b0 2024-04-25T11:42:05.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.183+0000 7f72dcff9640 1 -- 172.21.15.90:0/751465269 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f72f0035070 con 0x558be506d6b0 2024-04-25T11:42:05.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.183+0000 7f730246a740 10 client.26612 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:05.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.183+0000 7f730246a740 1 -- 172.21.15.90:0/751465269 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558be54b3e30 con 0x558be506d6b0 2024-04-25T11:42:05.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.184+0000 7f72dcff9640 1 -- 172.21.15.90:0/751465269 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x558be539c800 con 0x558be506d6b0 2024-04-25T11:42:05.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.184+0000 7f72dcff9640 1 -- 172.21.15.90:0/751465269 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f72f003ccb0 con 0x558be506d6b0 2024-04-25T11:42:05.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.184+0000 7f730246a740 10 client.26612 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:05.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.184+0000 7f730246a740 10 client.26612 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:42:05.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.184+0000 7f730246a740 20 client.26612 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:05.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.184+0000 7f730246a740 1 --2- 172.21.15.90:0/751465269 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x558be50223a0 0x558be54b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.184+0000 7f730246a740 4 client.26612 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:42:05.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.184+0000 7f730246a740 1 -- 172.21.15.90:0/751465269 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558be5317610 con 0x558be50223a0 2024-04-25T11:42:05.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.184+0000 7f72f8b55640 1 --2- 172.21.15.90:0/751465269 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x558be50223a0 0x558be54b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.185+0000 7f72dcff9640 10 client.26612 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:42:05.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.185+0000 7f72f8b55640 1 --2- 172.21.15.90:0/751465269 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x558be50223a0 0x558be54b60e0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8871c8a92a94d35a server_cookie=c9dec820dad21c19 in_seq=0 out_seq=0 2024-04-25T11:42:05.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.185+0000 7f72dcff9640 1 -- 172.21.15.90:0/751465269 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558be5317610 con 0x558be50223a0 2024-04-25T11:42:05.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.185+0000 7f72dcff9640 10 client.26612 handle_command_reply: tid=0 2024-04-25T11:42:05.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.185+0000 7f730246a740 1 client.26612 shutdown 2024-04-25T11:42:05.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.186+0000 7f730246a740 1 -- 172.21.15.90:0/751465269 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558be506d6b0 msgr2=0x558be543d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.186+0000 7f730246a740 1 --2- 172.21.15.90:0/751465269 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558be506d6b0 0x558be543d690 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x558be5430580 tx=0x7f72f0032f90 comp rx=0 tx=0).stop 2024-04-25T11:42:05.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.186+0000 7f730246a740 1 -- 172.21.15.90:0/751465269 shutdown_connections 2024-04-25T11:42:05.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.186+0000 7f730246a740 1 --2- 172.21.15.90:0/751465269 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x558be50223a0 0x558be54b60e0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.186+0000 7f730246a740 1 --2- 172.21.15.90:0/751465269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be508e390 0x558be543dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.186+0000 7f72dcff9640 0 client.26612 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:42:05.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.186+0000 7f730246a740 1 --2- 172.21.15.90:0/751465269 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x558be506d6b0 0x558be543d690 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.186+0000 7f730246a740 1 --2- 172.21.15.90:0/751465269 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558be50654d0 0x558be543d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.186+0000 7f730246a740 1 -- 172.21.15.90:0/751465269 >> 172.21.15.90:0/751465269 conn(0x558be50824e0 msgr2=0x558be5444e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:05.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.186+0000 7f730246a740 1 -- 172.21.15.90:0/751465269 shutdown_connections 2024-04-25T11:42:05.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.186+0000 7f730246a740 1 -- 172.21.15.90:0/751465269 wait complete. 2024-04-25T11:42:05.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.187+0000 7f730246a740 20 client.26612 trim_cache size 0 max 16384 2024-04-25T11:42:05.178 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:42:05.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.187+0000 7f72fb5ba640 1 -- 172.21.15.90:0/1532224630 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f72e00afc60 msgr2=0x7f72e00b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.187+0000 7f72fb5ba640 1 --2- 172.21.15.90:0/1532224630 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f72e00afc60 0x7f72e00b2170 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f72ec0dcdb0 tx=0x7f72ec090040 comp rx=0 tx=0).stop 2024-04-25T11:42:05.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.187+0000 7f72fb5ba640 1 -- 172.21.15.90:0/1532224630 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72f410e080 msgr2=0x7f72f419c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.187+0000 7f72fb5ba640 1 --2- 172.21.15.90:0/1532224630 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72f410e080 0x7f72f419c860 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f72ec00eb10 tx=0x7f72ec00c2d0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.187+0000 7f72fb5ba640 1 -- 172.21.15.90:0/1532224630 shutdown_connections 2024-04-25T11:42:05.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.187+0000 7f72fb5ba640 1 --2- 172.21.15.90:0/1532224630 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f72e00afc60 0x7f72e00b2170 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.187+0000 7f72fb5ba640 1 --2- 172.21.15.90:0/1532224630 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72f410e080 0x7f72f419c860 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.187+0000 7f72fb5ba640 1 --2- 172.21.15.90:0/1532224630 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f72f410b690 0x7f72f41a2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.187+0000 7f72fb5ba640 1 --2- 172.21.15.90:0/1532224630 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72f4104c30 0x7f72f41a2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.187+0000 7f72fb5ba640 1 -- 172.21.15.90:0/1532224630 >> 172.21.15.90:0/1532224630 conn(0x7f72f4078100 msgr2=0x7f72f4109af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:05.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.188+0000 7f72fb5ba640 1 -- 172.21.15.90:0/1532224630 shutdown_connections 2024-04-25T11:42:05.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.188+0000 7f72fb5ba640 1 -- 172.21.15.90:0/1532224630 wait complete. 2024-04-25T11:42:05.189 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T11:42:05.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.377+0000 7fb28d867640 1 -- 172.21.15.90:0/3730541140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb288108ec0 msgr2=0x7fb288103160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.377+0000 7fb28d867640 1 --2- 172.21.15.90:0/3730541140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb288108ec0 0x7fb288103160 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fb27c009900 tx=0x7fb27c02a410 comp rx=0 tx=0).stop 2024-04-25T11:42:05.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.377+0000 7fb28d867640 1 -- 172.21.15.90:0/3730541140 shutdown_connections 2024-04-25T11:42:05.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.377+0000 7fb28d867640 1 --2- 172.21.15.90:0/3730541140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2881037d0 0x7fb28810f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.377+0000 7fb28d867640 1 --2- 172.21.15.90:0/3730541140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb288108ec0 0x7fb288103160 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.377+0000 7fb28d867640 1 --2- 172.21.15.90:0/3730541140 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb288108a20 0x7fb288102c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.377+0000 7fb28d867640 1 -- 172.21.15.90:0/3730541140 >> 172.21.15.90:0/3730541140 conn(0x7fb288077880 msgr2=0x7fb288077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:05.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.378+0000 7fb28d867640 1 -- 172.21.15.90:0/3730541140 shutdown_connections 2024-04-25T11:42:05.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.378+0000 7fb28d867640 1 -- 172.21.15.90:0/3730541140 wait complete. 2024-04-25T11:42:05.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.378+0000 7fb28d867640 1 Processor -- start 2024-04-25T11:42:05.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.379+0000 7fb28d867640 1 -- start start 2024-04-25T11:42:05.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.379+0000 7fb28d867640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2881037d0 0x7fb2881a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.379+0000 7fb28d867640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb288108a20 0x7fb2881a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.379+0000 7fb28d867640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb288108ec0 0x7fb28819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.379+0000 7fb28d867640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb288112cf0 con 0x7fb2881037d0 2024-04-25T11:42:05.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.379+0000 7fb28d867640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb288112b70 con 0x7fb288108a20 2024-04-25T11:42:05.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.379+0000 7fb28d867640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb288112e70 con 0x7fb288108ec0 2024-04-25T11:42:05.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.379+0000 7fb286ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2881037d0 0x7fb2881a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.379+0000 7fb286ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2881037d0 0x7fb2881a05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58232/0 (socket says 172.21.15.90:58232) 2024-04-25T11:42:05.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.379+0000 7fb286ffd640 1 -- 172.21.15.90:0/2898683750 learned_addr learned my addr 172.21.15.90:0/2898683750 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:05.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.379+0000 7fb286ffd640 1 -- 172.21.15.90:0/2898683750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb288108ec0 msgr2=0x7fb28819a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.380+0000 7fb286ffd640 1 --2- 172.21.15.90:0/2898683750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb288108ec0 0x7fb28819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.380+0000 7fb286ffd640 1 -- 172.21.15.90:0/2898683750 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb288108a20 msgr2=0x7fb2881a0b00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:42:05.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.380+0000 7fb286ffd640 1 --2- 172.21.15.90:0/2898683750 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb288108a20 0x7fb2881a0b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.380+0000 7fb286ffd640 1 -- 172.21.15.90:0/2898683750 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb28819ae70 con 0x7fb2881037d0 2024-04-25T11:42:05.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.380+0000 7fb286ffd640 1 --2- 172.21.15.90:0/2898683750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2881037d0 0x7fb2881a05c0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fb27800b470 tx=0x7fb27800b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:05.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.380+0000 7fb28c865640 1 -- 172.21.15.90:0/2898683750 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb278015020 con 0x7fb2881037d0 2024-04-25T11:42:05.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.380+0000 7fb28c865640 1 -- 172.21.15.90:0/2898683750 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb278011d40 con 0x7fb2881037d0 2024-04-25T11:42:05.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.380+0000 7fb28d867640 1 -- 172.21.15.90:0/2898683750 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb28819b100 con 0x7fb2881037d0 2024-04-25T11:42:05.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.380+0000 7fb28c865640 1 -- 172.21.15.90:0/2898683750 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27800f340 con 0x7fb2881037d0 2024-04-25T11:42:05.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.380+0000 7fb28d867640 1 -- 172.21.15.90:0/2898683750 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2881a7360 con 0x7fb2881037d0 2024-04-25T11:42:05.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.384+0000 7fb28c865640 1 -- 172.21.15.90:0/2898683750 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fb278018020 con 0x7fb2881037d0 2024-04-25T11:42:05.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.384+0000 7fb2947f6740 1 Processor -- start 2024-04-25T11:42:05.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.384+0000 7fb2947f6740 1 -- start start 2024-04-25T11:42:05.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.384+0000 7fb2947f6740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd9f0a0390 0x55fd9f4416a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.384+0000 7fb2947f6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd9f0774d0 0x55fd9f441be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.384+0000 7fb2947f6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 0x55fd9f452ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.384+0000 7fb2947f6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55fd9f3a6d80 con 0x55fd9f07f6b0 2024-04-25T11:42:05.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.384+0000 7fb2947f6740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55fd9f360a20 con 0x55fd9f0a0390 2024-04-25T11:42:05.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.384+0000 7fb2947f6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fd9f3a2d40 con 0x55fd9f0774d0 2024-04-25T11:42:05.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.384+0000 7fb2867fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 0x55fd9f452ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.384+0000 7fb2867fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 0x55fd9f452ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58242/0 (socket says 172.21.15.90:58242) 2024-04-25T11:42:05.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.384+0000 7fb2867fc640 1 -- 172.21.15.90:0/334409140 learned_addr learned my addr 172.21.15.90:0/334409140 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:05.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.385+0000 7fb28c865640 1 --2- 172.21.15.90:0/2898683750 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb2700afc60 0x7fb2700b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.385+0000 7fb2877fe640 1 --2- 172.21.15.90:0/2898683750 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb2700afc60 0x7fb2700b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.385+0000 7fb28c865640 1 -- 172.21.15.90:0/2898683750 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb2780deb10 con 0x7fb2881037d0 2024-04-25T11:42:05.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.385+0000 7fb2867fc640 1 -- 172.21.15.90:0/334409140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd9f0774d0 msgr2=0x55fd9f441be0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:42:05.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.385+0000 7fb2867fc640 1 --2- 172.21.15.90:0/334409140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd9f0774d0 0x55fd9f441be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.385+0000 7fb2867fc640 1 -- 172.21.15.90:0/334409140 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd9f0a0390 msgr2=0x55fd9f4416a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:42:05.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.385+0000 7fb2867fc640 1 --2- 172.21.15.90:0/334409140 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd9f0a0390 0x55fd9f4416a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.385+0000 7fb2867fc640 1 -- 172.21.15.90:0/334409140 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd9f36bc30 con 0x55fd9f07f6b0 2024-04-25T11:42:05.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.385+0000 7fb2867fc640 1 --2- 172.21.15.90:0/334409140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 0x55fd9f452ab0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fb27c006010 tx=0x7fb27c032b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8f57bc2954d18a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:05.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.385+0000 7fb26dffb640 1 -- 172.21.15.90:0/334409140 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27c040070 con 0x55fd9f07f6b0 2024-04-25T11:42:05.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.385+0000 7fb2877fe640 1 --2- 172.21.15.90:0/2898683750 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb2700afc60 0x7fb2700b2170 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x55fd9f43e9b0 tx=0x7fb280001640 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:05.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.385+0000 7fb26dffb640 1 -- 172.21.15.90:0/334409140 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb27c004040 con 0x55fd9f07f6b0 2024-04-25T11:42:05.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.385+0000 7fb26dffb640 1 -- 172.21.15.90:0/334409140 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27c035040 con 0x55fd9f07f6b0 2024-04-25T11:42:05.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.386+0000 7fb2947f6740 1 -- 172.21.15.90:0/334409140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 msgr2=0x55fd9f452ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.386+0000 7fb2947f6740 1 --2- 172.21.15.90:0/334409140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 0x55fd9f452ab0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fb27c006010 tx=0x7fb27c032b20 comp rx=0 tx=0).stop 2024-04-25T11:42:05.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.387+0000 7fb2947f6740 1 -- 172.21.15.90:0/334409140 shutdown_connections 2024-04-25T11:42:05.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.387+0000 7fb2947f6740 1 --2- 172.21.15.90:0/334409140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 0x55fd9f452ab0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.387+0000 7fb2947f6740 1 --2- 172.21.15.90:0/334409140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd9f0774d0 0x55fd9f441be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.387+0000 7fb2947f6740 1 --2- 172.21.15.90:0/334409140 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd9f0a0390 0x55fd9f4416a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.387+0000 7fb2947f6740 1 -- 172.21.15.90:0/334409140 >> 172.21.15.90:0/334409140 conn(0x55fd9f0944e0 msgr2=0x55fd9f43dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:05.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.387+0000 7fb2947f6740 1 -- 172.21.15.90:0/334409140 shutdown_connections 2024-04-25T11:42:05.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.387+0000 7fb2947f6740 1 -- 172.21.15.90:0/334409140 wait complete. 2024-04-25T11:42:05.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.387+0000 7fb2947f6740 1 -- start start 2024-04-25T11:42:05.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.388+0000 7fb2947f6740 1 Processor -- start 2024-04-25T11:42:05.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.388+0000 7fb2947f6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd9f0774d0 0x55fd9f4506d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.388+0000 7fb2947f6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 0x55fd9f450c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.388+0000 7fb2947f6740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd9f0a0390 0x55fd9f45c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.388+0000 7fb2947f6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55fd9f453140 con 0x55fd9f07f6b0 2024-04-25T11:42:05.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.388+0000 7fb2947f6740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55fd9f32ddc0 con 0x55fd9f0a0390 2024-04-25T11:42:05.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.388+0000 7fb2947f6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fd9f4532c0 con 0x55fd9f0774d0 2024-04-25T11:42:05.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.388+0000 7fb2947f6740 10 client.0 _ll_register_callbacks cb 0x55fd9f349600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:05.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.388+0000 7fb2867fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 0x55fd9f450c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.388+0000 7fb2867fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 0x55fd9f450c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58252/0 (socket says 172.21.15.90:58252) 2024-04-25T11:42:05.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.388+0000 7fb2867fc640 1 -- 172.21.15.90:0/2473460469 learned_addr learned my addr 172.21.15.90:0/2473460469 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:05.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.388+0000 7fb2867fc640 1 -- 172.21.15.90:0/2473460469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd9f0774d0 msgr2=0x55fd9f4506d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:42:05.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.388+0000 7fb2867fc640 1 --2- 172.21.15.90:0/2473460469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd9f0774d0 0x55fd9f4506d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.389+0000 7fb2867fc640 1 -- 172.21.15.90:0/2473460469 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd9f0a0390 msgr2=0x55fd9f45c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.389+0000 7fb2867fc640 1 --2- 172.21.15.90:0/2473460469 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd9f0a0390 0x55fd9f45c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.389+0000 7fb2867fc640 1 -- 172.21.15.90:0/2473460469 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd9f0b83f0 con 0x55fd9f07f6b0 2024-04-25T11:42:05.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.389+0000 7fb2867fc640 1 --2- 172.21.15.90:0/2473460469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 0x55fd9f450c10 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fb27c0096c0 tx=0x7fb27c004420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a0b2e3a9fe49ef5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:05.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.389+0000 7fb266ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:05.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.389+0000 7fb266ffd640 1 -- 172.21.15.90:0/2473460469 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27c040050 con 0x55fd9f07f6b0 2024-04-25T11:42:05.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.389+0000 7fb266ffd640 1 -- 172.21.15.90:0/2473460469 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb27c0318b0 con 0x55fd9f07f6b0 2024-04-25T11:42:05.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.389+0000 7fb2947f6740 1 -- 172.21.15.90:0/2473460469 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb27c004040 con 0x55fd9f07f6b0 2024-04-25T11:42:05.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.389+0000 7fb266ffd640 1 -- 172.21.15.90:0/2473460469 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb27c035040 con 0x55fd9f07f6b0 2024-04-25T11:42:05.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.389+0000 7fb2947f6740 10 client.17682 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:05.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.389+0000 7fb2947f6740 1 -- 172.21.15.90:0/2473460469 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fd9f1827d0 con 0x55fd9f07f6b0 2024-04-25T11:42:05.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.390+0000 7fb266ffd640 1 -- 172.21.15.90:0/2473460469 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb27c004040 con 0x55fd9f07f6b0 2024-04-25T11:42:05.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.390+0000 7fb266ffd640 1 -- 172.21.15.90:0/2473460469 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fb27c045830 con 0x55fd9f07f6b0 2024-04-25T11:42:05.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.390+0000 7fb2947f6740 10 client.17682 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:05.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.390+0000 7fb2947f6740 10 client.17682 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:42:05.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.390+0000 7fb2947f6740 20 client.17682 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:05.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.390+0000 7fb2947f6740 1 --2- 172.21.15.90:0/2473460469 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55fd9f05ef60 0x55fd9f4629a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.390+0000 7fb2947f6740 4 client.17682 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:42:05.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.390+0000 7fb2947f6740 1 -- 172.21.15.90:0/2473460469 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd9f3b4990 con 0x55fd9f05ef60 2024-04-25T11:42:05.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.391+0000 7fb286ffd640 1 --2- 172.21.15.90:0/2473460469 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55fd9f05ef60 0x55fd9f4629a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.392+0000 7fb286ffd640 1 --2- 172.21.15.90:0/2473460469 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55fd9f05ef60 0x55fd9f4629a0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=adff9cf21935f39f server_cookie=2188185955cd3a6b in_seq=0 out_seq=0 2024-04-25T11:42:05.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.392+0000 7fb266ffd640 10 client.17682 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:42:05.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.394+0000 7fb266ffd640 1 -- 172.21.15.90:0/2473460469 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55fd9f3b4990 con 0x55fd9f05ef60 2024-04-25T11:42:05.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.394+0000 7fb266ffd640 10 client.17682 handle_command_reply: tid=0 2024-04-25T11:42:05.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.394+0000 7fb2947f6740 1 client.17682 shutdown 2024-04-25T11:42:05.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.394+0000 7fb2947f6740 1 -- 172.21.15.90:0/2473460469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 msgr2=0x55fd9f450c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.394+0000 7fb2947f6740 1 --2- 172.21.15.90:0/2473460469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 0x55fd9f450c10 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fb27c0096c0 tx=0x7fb27c004420 comp rx=0 tx=0).stop 2024-04-25T11:42:05.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.395+0000 7fb2947f6740 1 -- 172.21.15.90:0/2473460469 shutdown_connections 2024-04-25T11:42:05.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.395+0000 7fb2947f6740 1 --2- 172.21.15.90:0/2473460469 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55fd9f05ef60 0x55fd9f4629a0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.395+0000 7fb2947f6740 1 --2- 172.21.15.90:0/2473460469 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd9f0a0390 0x55fd9f45c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.395+0000 7fb266ffd640 0 client.17682 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:42:05.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.395+0000 7fb2947f6740 1 --2- 172.21.15.90:0/2473460469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 0x55fd9f450c10 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.395+0000 7fb2947f6740 1 --2- 172.21.15.90:0/2473460469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd9f0774d0 0x55fd9f4506d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.395+0000 7fb2947f6740 1 -- 172.21.15.90:0/2473460469 >> 172.21.15.90:0/2473460469 conn(0x55fd9f0944e0 msgr2=0x55fd9f443530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:05.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.395+0000 7fb2947f6740 1 -- 172.21.15.90:0/2473460469 shutdown_connections 2024-04-25T11:42:05.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.395+0000 7fb2947f6740 1 -- 172.21.15.90:0/2473460469 wait complete. 2024-04-25T11:42:05.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.395+0000 7fb2947f6740 20 client.17682 trim_cache size 0 max 16384 2024-04-25T11:42:05.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.413+0000 7fb2947f6740 1 Processor -- start 2024-04-25T11:42:05.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.413+0000 7fb2947f6740 1 -- start start 2024-04-25T11:42:05.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.413+0000 7fb2947f6740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd9f05ef60 0x55fd9f450110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.413+0000 7fb2947f6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd9f0774d0 0x55fd9f45c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.413+0000 7fb2947f6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 0x55fd9f4659c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.413+0000 7fb2947f6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb2881adb40 con 0x55fd9f07f6b0 2024-04-25T11:42:05.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.413+0000 7fb2947f6740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb2881af680 con 0x55fd9f05ef60 2024-04-25T11:42:05.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.413+0000 7fb2947f6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb2881ae210 con 0x55fd9f0774d0 2024-04-25T11:42:05.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.413+0000 7fb2877fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 0x55fd9f4659c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.413+0000 7fb2877fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 0x55fd9f4659c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58258/0 (socket says 172.21.15.90:58258) 2024-04-25T11:42:05.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.413+0000 7fb2877fe640 1 -- 172.21.15.90:0/748627895 learned_addr learned my addr 172.21.15.90:0/748627895 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:05.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.413+0000 7fb2867fc640 1 --2- 172.21.15.90:0/748627895 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd9f0774d0 0x55fd9f45c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.414+0000 7fb286ffd640 1 --2- 172.21.15.90:0/748627895 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd9f05ef60 0x55fd9f450110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.414+0000 7fb2877fe640 1 -- 172.21.15.90:0/748627895 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd9f0774d0 msgr2=0x55fd9f45c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.414+0000 7fb2877fe640 1 --2- 172.21.15.90:0/748627895 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd9f0774d0 0x55fd9f45c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.414+0000 7fb2877fe640 1 -- 172.21.15.90:0/748627895 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd9f05ef60 msgr2=0x55fd9f450110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.414+0000 7fb2877fe640 1 --2- 172.21.15.90:0/748627895 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd9f05ef60 0x55fd9f450110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.414+0000 7fb2877fe640 1 -- 172.21.15.90:0/748627895 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd9f4382b0 con 0x55fd9f07f6b0 2024-04-25T11:42:05.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.414+0000 7fb286ffd640 1 --2- 172.21.15.90:0/748627895 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd9f05ef60 0x55fd9f450110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:42:05.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.414+0000 7fb2877fe640 1 --2- 172.21.15.90:0/748627895 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 0x55fd9f4659c0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fb2800019a0 tx=0x7fb280001350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbe975ed48d873b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:05.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.414+0000 7fb26d7fa640 1 -- 172.21.15.90:0/748627895 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb280024070 con 0x55fd9f07f6b0 2024-04-25T11:42:05.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.414+0000 7fb26d7fa640 1 -- 172.21.15.90:0/748627895 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2800160d0 con 0x55fd9f07f6b0 2024-04-25T11:42:05.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.414+0000 7fb26d7fa640 1 -- 172.21.15.90:0/748627895 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28001f670 con 0x55fd9f07f6b0 2024-04-25T11:42:05.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.415+0000 7fb2947f6740 1 -- 172.21.15.90:0/748627895 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 msgr2=0x55fd9f4659c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.415+0000 7fb2947f6740 1 --2- 172.21.15.90:0/748627895 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 0x55fd9f4659c0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fb2800019a0 tx=0x7fb280001350 comp rx=0 tx=0).stop 2024-04-25T11:42:05.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.416+0000 7fb2947f6740 1 -- 172.21.15.90:0/748627895 shutdown_connections 2024-04-25T11:42:05.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.416+0000 7fb2947f6740 1 --2- 172.21.15.90:0/748627895 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 0x55fd9f4659c0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.416+0000 7fb2947f6740 1 --2- 172.21.15.90:0/748627895 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd9f0774d0 0x55fd9f45c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.416+0000 7fb2947f6740 1 --2- 172.21.15.90:0/748627895 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd9f05ef60 0x55fd9f450110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.416+0000 7fb2947f6740 1 -- 172.21.15.90:0/748627895 >> 172.21.15.90:0/748627895 conn(0x55fd9f0944e0 msgr2=0x55fd9f44fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:05.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.416+0000 7fb2947f6740 1 -- 172.21.15.90:0/748627895 shutdown_connections 2024-04-25T11:42:05.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.416+0000 7fb2947f6740 1 -- 172.21.15.90:0/748627895 wait complete. 2024-04-25T11:42:05.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.416+0000 7fb2947f6740 1 -- start start 2024-04-25T11:42:05.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.416+0000 7fb2947f6740 1 Processor -- start 2024-04-25T11:42:05.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.417+0000 7fb2947f6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd9f05ef60 0x55fd9f466010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.417+0000 7fb2947f6740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd9f0774d0 0x55fd9f463380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.417+0000 7fb2947f6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 0x55fd9f463940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.417+0000 7fb2947f6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb2881aefb0 con 0x55fd9f07f6b0 2024-04-25T11:42:05.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.417+0000 7fb2947f6740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb2881ae8e0 con 0x55fd9f0774d0 2024-04-25T11:42:05.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.417+0000 7fb2947f6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb2881abb20 con 0x55fd9f05ef60 2024-04-25T11:42:05.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.417+0000 7fb2947f6740 10 client.0 _ll_register_callbacks cb 0x7fb27c005740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:05.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.417+0000 7fb2877fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 0x55fd9f463940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.417+0000 7fb286ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd9f05ef60 0x55fd9f466010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.417+0000 7fb2877fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 0x55fd9f463940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58268/0 (socket says 172.21.15.90:58268) 2024-04-25T11:42:05.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.417+0000 7fb2877fe640 1 -- 172.21.15.90:0/4157679217 learned_addr learned my addr 172.21.15.90:0/4157679217 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:05.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.417+0000 7fb2867fc640 1 --2- 172.21.15.90:0/4157679217 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd9f0774d0 0x55fd9f463380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.418+0000 7fb286ffd640 1 -- 172.21.15.90:0/4157679217 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd9f0774d0 msgr2=0x55fd9f463380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.418+0000 7fb286ffd640 1 --2- 172.21.15.90:0/4157679217 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd9f0774d0 0x55fd9f463380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.418+0000 7fb286ffd640 1 -- 172.21.15.90:0/4157679217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 msgr2=0x55fd9f463940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.418+0000 7fb286ffd640 1 --2- 172.21.15.90:0/4157679217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 0x55fd9f463940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.418+0000 7fb286ffd640 1 -- 172.21.15.90:0/4157679217 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd9f44c080 con 0x55fd9f05ef60 2024-04-25T11:42:05.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.418+0000 7fb286ffd640 1 --2- 172.21.15.90:0/4157679217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd9f05ef60 0x55fd9f466010 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb2780dc310 tx=0x7fb2780046e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=573957e60e8f7a8d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:05.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.418+0000 7fb266ffd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:42:05.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.418+0000 7fb266ffd640 1 -- 172.21.15.90:0/4157679217 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2780a6d60 con 0x55fd9f05ef60 2024-04-25T11:42:05.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.418+0000 7fb266ffd640 1 -- 172.21.15.90:0/4157679217 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fb2780a2ff0 con 0x55fd9f05ef60 2024-04-25T11:42:05.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.418+0000 7fb2947f6740 1 -- 172.21.15.90:0/4157679217 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2800160d0 con 0x55fd9f05ef60 2024-04-25T11:42:05.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.418+0000 7fb266ffd640 1 -- 172.21.15.90:0/4157679217 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb2780ab110 con 0x55fd9f05ef60 2024-04-25T11:42:05.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.418+0000 7fb2947f6740 10 client.26557 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:05.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.419+0000 7fb2947f6740 1 -- 172.21.15.90:0/4157679217 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fd9f4ca870 con 0x55fd9f05ef60 2024-04-25T11:42:05.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.419+0000 7fb266ffd640 1 -- 172.21.15.90:0/4157679217 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2800160d0 con 0x55fd9f05ef60 2024-04-25T11:42:05.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.419+0000 7fb266ffd640 1 -- 172.21.15.90:0/4157679217 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fb2780a26d0 con 0x55fd9f05ef60 2024-04-25T11:42:05.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.419+0000 7fb2947f6740 10 client.26557 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:05.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.419+0000 7fb2947f6740 10 client.26557 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:42:05.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.419+0000 7fb2947f6740 20 client.26557 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:05.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.419+0000 7fb2947f6740 1 --2- 172.21.15.90:0/4157679217 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55fd9f0a0390 0x55fd9f4ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.419+0000 7fb2947f6740 4 client.26557 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:42:05.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.419+0000 7fb2947f6740 1 -- 172.21.15.90:0/4157679217 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fd9f3ae800 con 0x55fd9f0a0390 2024-04-25T11:42:05.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.419+0000 7fb2867fc640 1 --2- 172.21.15.90:0/4157679217 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55fd9f0a0390 0x55fd9f4ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.420+0000 7fb2867fc640 1 --2- 172.21.15.90:0/4157679217 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55fd9f0a0390 0x55fd9f4ccf60 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=758d1ffbb22af3ab server_cookie=9e654c631f616d48 in_seq=0 out_seq=0 2024-04-25T11:42:05.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.420+0000 7fb266ffd640 10 client.26557 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:42:05.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.421+0000 7fb266ffd640 1 -- 172.21.15.90:0/4157679217 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fd9f3ae800 con 0x55fd9f0a0390 2024-04-25T11:42:05.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.421+0000 7fb266ffd640 10 client.26557 handle_command_reply: tid=0 2024-04-25T11:42:05.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.421+0000 7fb2947f6740 1 client.26557 shutdown 2024-04-25T11:42:05.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.421+0000 7fb2947f6740 1 -- 172.21.15.90:0/4157679217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd9f05ef60 msgr2=0x55fd9f466010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.421+0000 7fb2947f6740 1 --2- 172.21.15.90:0/4157679217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd9f05ef60 0x55fd9f466010 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb2780dc310 tx=0x7fb2780046e0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.421+0000 7fb2947f6740 1 -- 172.21.15.90:0/4157679217 shutdown_connections 2024-04-25T11:42:05.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.421+0000 7fb2947f6740 1 --2- 172.21.15.90:0/4157679217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd9f07f6b0 0x55fd9f463940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.421+0000 7fb2947f6740 1 --2- 172.21.15.90:0/4157679217 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55fd9f0a0390 0x55fd9f4ccf60 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.421+0000 7fb2947f6740 1 --2- 172.21.15.90:0/4157679217 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fd9f0774d0 0x55fd9f463380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.421+0000 7fb2947f6740 1 --2- 172.21.15.90:0/4157679217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd9f05ef60 0x55fd9f466010 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.421+0000 7fb266ffd640 0 client.26557 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:42:05.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.421+0000 7fb2947f6740 1 -- 172.21.15.90:0/4157679217 >> 172.21.15.90:0/4157679217 conn(0x55fd9f0944e0 msgr2=0x55fd9f43d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:05.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.421+0000 7fb2947f6740 1 -- 172.21.15.90:0/4157679217 shutdown_connections 2024-04-25T11:42:05.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.421+0000 7fb2947f6740 1 -- 172.21.15.90:0/4157679217 wait complete. 2024-04-25T11:42:05.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.422+0000 7fb2947f6740 20 client.26557 trim_cache size 0 max 16384 2024-04-25T11:42:05.411 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:42:05.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.422+0000 7fb28d867640 1 -- 172.21.15.90:0/2898683750 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb2700afc60 msgr2=0x7fb2700b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.422+0000 7fb28d867640 1 --2- 172.21.15.90:0/2898683750 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb2700afc60 0x7fb2700b2170 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x55fd9f43e9b0 tx=0x7fb280001640 comp rx=0 tx=0).stop 2024-04-25T11:42:05.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.422+0000 7fb28d867640 1 -- 172.21.15.90:0/2898683750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2881037d0 msgr2=0x7fb2881a05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.422+0000 7fb28d867640 1 --2- 172.21.15.90:0/2898683750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2881037d0 0x7fb2881a05c0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fb27800b470 tx=0x7fb27800b930 comp rx=0 tx=0).stop 2024-04-25T11:42:05.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.422+0000 7fb28d867640 1 -- 172.21.15.90:0/2898683750 shutdown_connections 2024-04-25T11:42:05.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.422+0000 7fb28d867640 1 --2- 172.21.15.90:0/2898683750 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb2700afc60 0x7fb2700b2170 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.422+0000 7fb28d867640 1 --2- 172.21.15.90:0/2898683750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb288108ec0 0x7fb28819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.422+0000 7fb28d867640 1 --2- 172.21.15.90:0/2898683750 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb288108a20 0x7fb2881a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.422+0000 7fb28d867640 1 --2- 172.21.15.90:0/2898683750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2881037d0 0x7fb2881a05c0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.422+0000 7fb28d867640 1 -- 172.21.15.90:0/2898683750 >> 172.21.15.90:0/2898683750 conn(0x7fb288077880 msgr2=0x7fb2881060d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:05.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.422+0000 7fb28d867640 1 -- 172.21.15.90:0/2898683750 shutdown_connections 2024-04-25T11:42:05.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.422+0000 7fb28d867640 1 -- 172.21.15.90:0/2898683750 wait complete. 2024-04-25T11:42:05.420 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T11:42:05.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.556+0000 7fd643dab640 1 -- 172.21.15.90:0/2324884575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd63c108a20 msgr2=0x7fd63c1025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.557+0000 7fd643dab640 1 --2- 172.21.15.90:0/2324884575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd63c108a20 0x7fd63c1025b0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fd62c009900 tx=0x7fd62c02a410 comp rx=0 tx=0).stop 2024-04-25T11:42:05.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.557+0000 7fd643dab640 1 -- 172.21.15.90:0/2324884575 shutdown_connections 2024-04-25T11:42:05.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.557+0000 7fd643dab640 1 --2- 172.21.15.90:0/2324884575 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd63c10d380 0x7fd63c10f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.557+0000 7fd643dab640 1 --2- 172.21.15.90:0/2324884575 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd63c102af0 0x7fd63c102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.557+0000 7fd643dab640 1 --2- 172.21.15.90:0/2324884575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd63c108a20 0x7fd63c1025b0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.557+0000 7fd643dab640 1 -- 172.21.15.90:0/2324884575 >> 172.21.15.90:0/2324884575 conn(0x7fd63c078100 msgr2=0x7fd63c078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:05.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.557+0000 7fd643dab640 1 -- 172.21.15.90:0/2324884575 shutdown_connections 2024-04-25T11:42:05.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.557+0000 7fd643dab640 1 -- 172.21.15.90:0/2324884575 wait complete. 2024-04-25T11:42:05.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.557+0000 7fd643dab640 1 Processor -- start 2024-04-25T11:42:05.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.557+0000 7fd643dab640 1 -- start start 2024-04-25T11:42:05.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.558+0000 7fd643dab640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd63c102af0 0x7fd63c19a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.558+0000 7fd643dab640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd63c108a20 0x7fd63c19ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.558+0000 7fd643dab640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd63c10d380 0x7fd63c19eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.558+0000 7fd643dab640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c112be0 con 0x7fd63c108a20 2024-04-25T11:42:05.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.558+0000 7fd643dab640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c112a60 con 0x7fd63c102af0 2024-04-25T11:42:05.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.558+0000 7fd643dab640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c112d60 con 0x7fd63c10d380 2024-04-25T11:42:05.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.558+0000 7fd641346640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd63c108a20 0x7fd63c19ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.558+0000 7fd641346640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd63c108a20 0x7fd63c19ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58298/0 (socket says 172.21.15.90:58298) 2024-04-25T11:42:05.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.558+0000 7fd641346640 1 -- 172.21.15.90:0/1676461847 learned_addr learned my addr 172.21.15.90:0/1676461847 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:05.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.558+0000 7fd641b47640 1 --2- 172.21.15.90:0/1676461847 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd63c102af0 0x7fd63c19a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.558+0000 7fd642348640 1 --2- 172.21.15.90:0/1676461847 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd63c10d380 0x7fd63c19eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.558+0000 7fd641346640 1 -- 172.21.15.90:0/1676461847 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd63c10d380 msgr2=0x7fd63c19eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.558+0000 7fd641346640 1 --2- 172.21.15.90:0/1676461847 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd63c10d380 0x7fd63c19eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.558+0000 7fd641346640 1 -- 172.21.15.90:0/1676461847 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd63c102af0 msgr2=0x7fd63c19a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.558+0000 7fd641346640 1 --2- 172.21.15.90:0/1676461847 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd63c102af0 0x7fd63c19a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.558+0000 7fd641346640 1 -- 172.21.15.90:0/1676461847 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd63c19f530 con 0x7fd63c108a20 2024-04-25T11:42:05.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.558+0000 7fd641346640 1 --2- 172.21.15.90:0/1676461847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd63c108a20 0x7fd63c19ac30 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fd63800e980 tx=0x7fd63800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:05.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.558+0000 7fd632ffd640 1 -- 172.21.15.90:0/1676461847 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63800cd80 con 0x7fd63c108a20 2024-04-25T11:42:05.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.558+0000 7fd632ffd640 1 -- 172.21.15.90:0/1676461847 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6380108e0 con 0x7fd63c108a20 2024-04-25T11:42:05.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.558+0000 7fd643dab640 1 -- 172.21.15.90:0/1676461847 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd63c19f820 con 0x7fd63c108a20 2024-04-25T11:42:05.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.558+0000 7fd632ffd640 1 -- 172.21.15.90:0/1676461847 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd638018990 con 0x7fd63c108a20 2024-04-25T11:42:05.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.558+0000 7fd643dab640 1 -- 172.21.15.90:0/1676461847 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd63c1a71a0 con 0x7fd63c108a20 2024-04-25T11:42:05.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.561+0000 7fd64ac58740 1 Processor -- start 2024-04-25T11:42:05.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.561+0000 7fd64ac58740 1 -- start start 2024-04-25T11:42:05.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.561+0000 7fd632ffd640 1 -- 172.21.15.90:0/1676461847 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fd63801f020 con 0x7fd63c108a20 2024-04-25T11:42:05.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.562+0000 7fd64ac58740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd40390 0x55f1cc0e1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.562+0000 7fd64ac58740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f1cbd174d0 0x55f1cc0e1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.562+0000 7fd64ac58740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f1cbd1f6b0 0x55f1cc0ee6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.562+0000 7fd64ac58740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f1cc046d80 con 0x55f1cbd40390 2024-04-25T11:42:05.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.562+0000 7fd64ac58740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f1cc000a20 con 0x55f1cbd174d0 2024-04-25T11:42:05.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.562+0000 7fd64ac58740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f1cc042d40 con 0x55f1cbd1f6b0 2024-04-25T11:42:05.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.562+0000 7fd642348640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd40390 0x55f1cc0e1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.562+0000 7fd642348640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd40390 0x55f1cc0e1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58312/0 (socket says 172.21.15.90:58312) 2024-04-25T11:42:05.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.562+0000 7fd641b47640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f1cbd174d0 0x55f1cc0e1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.562+0000 7fd642348640 1 -- 172.21.15.90:0/3449496021 learned_addr learned my addr 172.21.15.90:0/3449496021 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:05.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.562+0000 7fd641346640 1 --2- 172.21.15.90:0/3449496021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f1cbd1f6b0 0x55f1cc0ee6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.562+0000 7fd642348640 1 -- 172.21.15.90:0/3449496021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f1cbd1f6b0 msgr2=0x55f1cc0ee6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.562+0000 7fd642348640 1 --2- 172.21.15.90:0/3449496021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f1cbd1f6b0 0x55f1cc0ee6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.562+0000 7fd642348640 1 -- 172.21.15.90:0/3449496021 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f1cbd174d0 msgr2=0x55f1cc0e1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.562+0000 7fd642348640 1 --2- 172.21.15.90:0/3449496021 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f1cbd174d0 0x55f1cc0e1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.562+0000 7fd642348640 1 -- 172.21.15.90:0/3449496021 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1cc00bc30 con 0x55f1cbd40390 2024-04-25T11:42:05.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.562+0000 7fd641b47640 1 --2- 172.21.15.90:0/3449496021 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f1cbd174d0 0x55f1cc0e1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:42:05.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.562+0000 7fd641346640 1 --2- 172.21.15.90:0/3449496021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f1cbd1f6b0 0x55f1cc0ee6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:42:05.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.563+0000 7fd632ffd640 1 --2- 172.21.15.90:0/1676461847 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd6280afc60 0x7fd6280b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.563+0000 7fd642348640 1 --2- 172.21.15.90:0/3449496021 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd40390 0x55f1cc0e1810 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fd634010370 tx=0x7fd634010830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1532a23689a51ea6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:05.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.563+0000 7fd641b47640 1 --2- 172.21.15.90:0/1676461847 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd6280afc60 0x7fd6280b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.563+0000 7fd632ffd640 1 -- 172.21.15.90:0/1676461847 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fd6380df240 con 0x7fd63c108a20 2024-04-25T11:42:05.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.563+0000 7fd627fff640 1 -- 172.21.15.90:0/3449496021 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd634016070 con 0x55f1cbd40390 2024-04-25T11:42:05.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.563+0000 7fd627fff640 1 -- 172.21.15.90:0/3449496021 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd634004d20 con 0x55f1cbd40390 2024-04-25T11:42:05.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.563+0000 7fd627fff640 1 -- 172.21.15.90:0/3449496021 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd634005350 con 0x55f1cbd40390 2024-04-25T11:42:05.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.564+0000 7fd64ac58740 1 -- 172.21.15.90:0/3449496021 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd40390 msgr2=0x55f1cc0e1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.564+0000 7fd64ac58740 1 --2- 172.21.15.90:0/3449496021 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd40390 0x55f1cc0e1810 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fd634010370 tx=0x7fd634010830 comp rx=0 tx=0).stop 2024-04-25T11:42:05.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.564+0000 7fd641b47640 1 --2- 172.21.15.90:0/1676461847 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd6280afc60 0x7fd6280b2170 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd62c000c00 tx=0x7fd62c031400 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:05.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.564+0000 7fd64ac58740 1 -- 172.21.15.90:0/3449496021 shutdown_connections 2024-04-25T11:42:05.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.564+0000 7fd64ac58740 1 --2- 172.21.15.90:0/3449496021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f1cbd1f6b0 0x55f1cc0ee6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.564+0000 7fd64ac58740 1 --2- 172.21.15.90:0/3449496021 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f1cbd174d0 0x55f1cc0e1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.564+0000 7fd64ac58740 1 --2- 172.21.15.90:0/3449496021 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd40390 0x55f1cc0e1810 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.564+0000 7fd64ac58740 1 -- 172.21.15.90:0/3449496021 >> 172.21.15.90:0/3449496021 conn(0x55f1cbd344e0 msgr2=0x55f1cc0ddfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:05.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.564+0000 7fd64ac58740 1 -- 172.21.15.90:0/3449496021 shutdown_connections 2024-04-25T11:42:05.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.564+0000 7fd64ac58740 1 -- 172.21.15.90:0/3449496021 wait complete. 2024-04-25T11:42:05.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.564+0000 7fd64ac58740 1 -- start start 2024-04-25T11:42:05.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.564+0000 7fd64ac58740 1 Processor -- start 2024-04-25T11:42:05.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.565+0000 7fd64ac58740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f1cbd174d0 0x55f1cc0ec280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.565+0000 7fd64ac58740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f1cbd1f6b0 0x55f1cc0ec7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.565+0000 7fd64ac58740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd40390 0x55f1cc0f8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.565+0000 7fd64ac58740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f1cc0eed90 con 0x55f1cbd40390 2024-04-25T11:42:05.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.565+0000 7fd64ac58740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f1cbfcddc0 con 0x55f1cbd174d0 2024-04-25T11:42:05.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.565+0000 7fd64ac58740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f1cc0eef10 con 0x55f1cbd1f6b0 2024-04-25T11:42:05.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.565+0000 7fd64ac58740 10 client.0 _ll_register_callbacks cb 0x55f1cbfe9600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:05.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.565+0000 7fd642348640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd40390 0x55f1cc0f8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.565+0000 7fd641b47640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f1cbd174d0 0x55f1cc0ec280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.565+0000 7fd641b47640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f1cbd174d0 0x55f1cc0ec280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:34720/0 (socket says 172.21.15.90:34720) 2024-04-25T11:42:05.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.565+0000 7fd642348640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd40390 0x55f1cc0f8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58322/0 (socket says 172.21.15.90:58322) 2024-04-25T11:42:05.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.565+0000 7fd641b47640 1 -- 172.21.15.90:0/1529009329 learned_addr learned my addr 172.21.15.90:0/1529009329 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:05.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.565+0000 7fd641346640 1 --2- 172.21.15.90:0/1529009329 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f1cbd1f6b0 0x55f1cc0ec7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.565+0000 7fd642348640 1 -- 172.21.15.90:0/1529009329 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f1cbd1f6b0 msgr2=0x55f1cc0ec7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.565+0000 7fd642348640 1 --2- 172.21.15.90:0/1529009329 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f1cbd1f6b0 0x55f1cc0ec7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.565+0000 7fd642348640 1 -- 172.21.15.90:0/1529009329 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f1cbd174d0 msgr2=0x55f1cc0ec280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.565+0000 7fd642348640 1 --2- 172.21.15.90:0/1529009329 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f1cbd174d0 0x55f1cc0ec280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.565+0000 7fd642348640 1 -- 172.21.15.90:0/1529009329 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1cbd583f0 con 0x55f1cbd40390 2024-04-25T11:42:05.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.565+0000 7fd641b47640 1 --2- 172.21.15.90:0/1529009329 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f1cbd174d0 0x55f1cc0ec280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:42:05.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.565+0000 7fd642348640 1 --2- 172.21.15.90:0/1529009329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd40390 0x55f1cc0f8210 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x55f1cc0de980 tx=0x7fd634004c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f387d4c567285d87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:05.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.565+0000 7fd6257fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:05.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.566+0000 7fd6257fa640 1 -- 172.21.15.90:0/1529009329 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd634029060 con 0x55f1cbd40390 2024-04-25T11:42:05.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.566+0000 7fd6257fa640 1 -- 172.21.15.90:0/1529009329 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fd634012820 con 0x55f1cbd40390 2024-04-25T11:42:05.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.566+0000 7fd64ac58740 1 -- 172.21.15.90:0/1529009329 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd634004d20 con 0x55f1cbd40390 2024-04-25T11:42:05.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.566+0000 7fd6257fa640 1 -- 172.21.15.90:0/1529009329 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd634016070 con 0x55f1cbd40390 2024-04-25T11:42:05.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.566+0000 7fd64ac58740 10 client.17700 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:05.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.566+0000 7fd64ac58740 1 -- 172.21.15.90:0/1529009329 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f1cbe227d0 con 0x55f1cbd40390 2024-04-25T11:42:05.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.566+0000 7fd6257fa640 1 -- 172.21.15.90:0/1529009329 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd634004d20 con 0x55f1cbd40390 2024-04-25T11:42:05.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.566+0000 7fd6257fa640 1 -- 172.21.15.90:0/1529009329 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fd63400f040 con 0x55f1cbd40390 2024-04-25T11:42:05.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.566+0000 7fd64ac58740 10 client.17700 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:05.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.566+0000 7fd64ac58740 10 client.17700 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:42:05.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.566+0000 7fd64ac58740 20 client.17700 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:05.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.566+0000 7fd64ac58740 1 --2- 172.21.15.90:0/1529009329 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f1cbcfef60 0x55f1cc0fe570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.566+0000 7fd64ac58740 4 client.17700 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:42:05.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.566+0000 7fd64ac58740 1 -- 172.21.15.90:0/1529009329 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1cc054990 con 0x55f1cbcfef60 2024-04-25T11:42:05.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.567+0000 7fd641b47640 1 --2- 172.21.15.90:0/1529009329 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f1cbcfef60 0x55f1cc0fe570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.567+0000 7fd6257fa640 10 client.17700 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:42:05.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.567+0000 7fd641b47640 1 --2- 172.21.15.90:0/1529009329 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f1cbcfef60 0x55f1cc0fe570 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a93e9318715ec960 server_cookie=efe628231f154b54 in_seq=0 out_seq=0 2024-04-25T11:42:05.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.568+0000 7fd6257fa640 1 -- 172.21.15.90:0/1529009329 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f1cc054990 con 0x55f1cbcfef60 2024-04-25T11:42:05.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.568+0000 7fd6257fa640 10 client.17700 handle_command_reply: tid=0 2024-04-25T11:42:05.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.568+0000 7fd64ac58740 1 client.17700 shutdown 2024-04-25T11:42:05.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.568+0000 7fd64ac58740 1 -- 172.21.15.90:0/1529009329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd40390 msgr2=0x55f1cc0f8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.568+0000 7fd64ac58740 1 --2- 172.21.15.90:0/1529009329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd40390 0x55f1cc0f8210 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x55f1cc0de980 tx=0x7fd634004c70 comp rx=0 tx=0).stop 2024-04-25T11:42:05.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.568+0000 7fd64ac58740 1 -- 172.21.15.90:0/1529009329 shutdown_connections 2024-04-25T11:42:05.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.568+0000 7fd64ac58740 1 --2- 172.21.15.90:0/1529009329 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f1cbcfef60 0x55f1cc0fe570 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.568+0000 7fd64ac58740 1 --2- 172.21.15.90:0/1529009329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd40390 0x55f1cc0f8210 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.568+0000 7fd64ac58740 1 --2- 172.21.15.90:0/1529009329 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f1cbd1f6b0 0x55f1cc0ec7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.568+0000 7fd6257fa640 0 client.17700 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:42:05.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.568+0000 7fd64ac58740 1 --2- 172.21.15.90:0/1529009329 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f1cbd174d0 0x55f1cc0ec280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.568+0000 7fd64ac58740 1 -- 172.21.15.90:0/1529009329 >> 172.21.15.90:0/1529009329 conn(0x55f1cbd344e0 msgr2=0x55f1cc0e3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:05.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.568+0000 7fd64ac58740 1 -- 172.21.15.90:0/1529009329 shutdown_connections 2024-04-25T11:42:05.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.568+0000 7fd64ac58740 1 -- 172.21.15.90:0/1529009329 wait complete. 2024-04-25T11:42:05.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.568+0000 7fd64ac58740 20 client.17700 trim_cache size 0 max 16384 2024-04-25T11:42:05.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.581+0000 7fd64ac58740 1 Processor -- start 2024-04-25T11:42:05.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.581+0000 7fd64ac58740 1 -- start start 2024-04-25T11:42:05.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.582+0000 7fd64ac58740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f1cbd174d0 0x55f1cc0f8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.582+0000 7fd64ac58740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd1f6b0 0x55f1cc0e1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.582+0000 7fd64ac58740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f1cbd40390 0x55f1cc0e6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.582+0000 7fd64ac58740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c1afa50 con 0x55f1cbd1f6b0 2024-04-25T11:42:05.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.582+0000 7fd64ac58740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c1b1590 con 0x55f1cbd174d0 2024-04-25T11:42:05.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.582+0000 7fd64ac58740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c1b0120 con 0x55f1cbd40390 2024-04-25T11:42:05.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.582+0000 7fd641346640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd1f6b0 0x55f1cc0e1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.582+0000 7fd641346640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd1f6b0 0x55f1cc0e1cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58336/0 (socket says 172.21.15.90:58336) 2024-04-25T11:42:05.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.582+0000 7fd641346640 1 -- 172.21.15.90:0/2506212025 learned_addr learned my addr 172.21.15.90:0/2506212025 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:05.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.582+0000 7fd641b47640 1 --2- 172.21.15.90:0/2506212025 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f1cbd174d0 0x55f1cc0f8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.582+0000 7fd642348640 1 --2- 172.21.15.90:0/2506212025 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f1cbd40390 0x55f1cc0e6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.582+0000 7fd641346640 1 -- 172.21.15.90:0/2506212025 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f1cbd40390 msgr2=0x55f1cc0e6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.582+0000 7fd641346640 1 --2- 172.21.15.90:0/2506212025 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f1cbd40390 0x55f1cc0e6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.582+0000 7fd641346640 1 -- 172.21.15.90:0/2506212025 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f1cbd174d0 msgr2=0x55f1cc0f8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.582+0000 7fd641346640 1 --2- 172.21.15.90:0/2506212025 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f1cbd174d0 0x55f1cc0f8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.582+0000 7fd641346640 1 -- 172.21.15.90:0/2506212025 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1cc0e7c30 con 0x55f1cbd1f6b0 2024-04-25T11:42:05.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.582+0000 7fd641346640 1 --2- 172.21.15.90:0/2506212025 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd1f6b0 0x55f1cc0e1cb0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fd6380a3400 tx=0x7fd638018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23a80cdd4d2976c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:05.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.582+0000 7fd6277fe640 1 -- 172.21.15.90:0/2506212025 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6380a7150 con 0x55f1cbd1f6b0 2024-04-25T11:42:05.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.582+0000 7fd6277fe640 1 -- 172.21.15.90:0/2506212025 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6380a6e90 con 0x55f1cbd1f6b0 2024-04-25T11:42:05.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.582+0000 7fd6277fe640 1 -- 172.21.15.90:0/2506212025 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6380a6af0 con 0x55f1cbd1f6b0 2024-04-25T11:42:05.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.583+0000 7fd64ac58740 1 -- 172.21.15.90:0/2506212025 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd1f6b0 msgr2=0x55f1cc0e1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.583+0000 7fd64ac58740 1 --2- 172.21.15.90:0/2506212025 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd1f6b0 0x55f1cc0e1cb0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fd6380a3400 tx=0x7fd638018420 comp rx=0 tx=0).stop 2024-04-25T11:42:05.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.583+0000 7fd64ac58740 1 -- 172.21.15.90:0/2506212025 shutdown_connections 2024-04-25T11:42:05.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.583+0000 7fd64ac58740 1 --2- 172.21.15.90:0/2506212025 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f1cbd40390 0x55f1cc0e6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.583+0000 7fd64ac58740 1 --2- 172.21.15.90:0/2506212025 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd1f6b0 0x55f1cc0e1cb0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.583+0000 7fd64ac58740 1 --2- 172.21.15.90:0/2506212025 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f1cbd174d0 0x55f1cc0f8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.583+0000 7fd64ac58740 1 -- 172.21.15.90:0/2506212025 >> 172.21.15.90:0/2506212025 conn(0x55f1cbd344e0 msgr2=0x55f1cc0e15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:05.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.583+0000 7fd64ac58740 1 -- 172.21.15.90:0/2506212025 shutdown_connections 2024-04-25T11:42:05.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.583+0000 7fd64ac58740 1 -- 172.21.15.90:0/2506212025 wait complete. 2024-04-25T11:42:05.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.583+0000 7fd64ac58740 1 -- start start 2024-04-25T11:42:05.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.584+0000 7fd64ac58740 1 Processor -- start 2024-04-25T11:42:05.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.584+0000 7fd64ac58740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f1cbd174d0 0x55f1cc0fe910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.584+0000 7fd64ac58740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd1f6b0 0x55f1cc0fee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.584+0000 7fd64ac58740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f1cbd40390 0x55f1cc164690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.584+0000 7fd64ac58740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c1b0ec0 con 0x55f1cbd1f6b0 2024-04-25T11:42:05.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.584+0000 7fd64ac58740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c1b07f0 con 0x55f1cbd40390 2024-04-25T11:42:05.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.584+0000 7fd64ac58740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c1ada30 con 0x55f1cbd174d0 2024-04-25T11:42:05.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.584+0000 7fd64ac58740 10 client.0 _ll_register_callbacks cb 0x7fd62c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:05.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.584+0000 7fd641346640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd1f6b0 0x55f1cc0fee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.584+0000 7fd641346640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd1f6b0 0x55f1cc0fee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58340/0 (socket says 172.21.15.90:58340) 2024-04-25T11:42:05.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.584+0000 7fd641346640 1 -- 172.21.15.90:0/274837159 learned_addr learned my addr 172.21.15.90:0/274837159 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:05.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.584+0000 7fd642348640 1 --2- 172.21.15.90:0/274837159 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f1cbd40390 0x55f1cc164690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.584+0000 7fd641b47640 1 --2- 172.21.15.90:0/274837159 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f1cbd174d0 0x55f1cc0fe910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.585+0000 7fd641346640 1 -- 172.21.15.90:0/274837159 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f1cbd174d0 msgr2=0x55f1cc0fe910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.585+0000 7fd641346640 1 --2- 172.21.15.90:0/274837159 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f1cbd174d0 0x55f1cc0fe910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.585+0000 7fd641346640 1 -- 172.21.15.90:0/274837159 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f1cbd40390 msgr2=0x55f1cc164690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.585+0000 7fd641346640 1 --2- 172.21.15.90:0/274837159 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f1cbd40390 0x55f1cc164690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.585+0000 7fd641346640 1 -- 172.21.15.90:0/274837159 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1cc164bd0 con 0x55f1cbd1f6b0 2024-04-25T11:42:05.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.585+0000 7fd641346640 1 --2- 172.21.15.90:0/274837159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd1f6b0 0x55f1cc0fee50 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fd6380a7310 tx=0x7fd638009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37fb485744a86c84 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:05.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.585+0000 7fd6257fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:05.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.585+0000 7fd6257fa640 1 -- 172.21.15.90:0/274837159 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6380a6cd0 con 0x55f1cbd1f6b0 2024-04-25T11:42:05.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.585+0000 7fd6257fa640 1 -- 172.21.15.90:0/274837159 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fd6380afd50 con 0x55f1cbd1f6b0 2024-04-25T11:42:05.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.585+0000 7fd64ac58740 1 -- 172.21.15.90:0/274837159 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6380a6e90 con 0x55f1cbd1f6b0 2024-04-25T11:42:05.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.585+0000 7fd6257fa640 1 -- 172.21.15.90:0/274837159 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd6380a7150 con 0x55f1cbd1f6b0 2024-04-25T11:42:05.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.585+0000 7fd64ac58740 10 client.17706 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:05.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.585+0000 7fd64ac58740 1 -- 172.21.15.90:0/274837159 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f1cc165de0 con 0x55f1cbd1f6b0 2024-04-25T11:42:05.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.585+0000 7fd6257fa640 1 -- 172.21.15.90:0/274837159 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6380a6e90 con 0x55f1cbd1f6b0 2024-04-25T11:42:05.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.586+0000 7fd6257fa640 1 -- 172.21.15.90:0/274837159 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fd638018b30 con 0x55f1cbd1f6b0 2024-04-25T11:42:05.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.586+0000 7fd64ac58740 10 client.17706 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:05.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.586+0000 7fd64ac58740 10 client.17706 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:42:05.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.586+0000 7fd64ac58740 20 client.17706 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:05.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.586+0000 7fd64ac58740 1 --2- 172.21.15.90:0/274837159 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f1cbcd43a0 0x55f1cc168460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.586+0000 7fd64ac58740 4 client.17706 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:42:05.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.586+0000 7fd64ac58740 1 -- 172.21.15.90:0/274837159 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f1cc04e800 con 0x55f1cbcd43a0 2024-04-25T11:42:05.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.586+0000 7fd641b47640 1 --2- 172.21.15.90:0/274837159 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f1cbcd43a0 0x55f1cc168460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.586+0000 7fd6257fa640 10 client.17706 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:42:05.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.586+0000 7fd641b47640 1 --2- 172.21.15.90:0/274837159 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f1cbcd43a0 0x55f1cc168460 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7f8dd663f3b6effd server_cookie=8b254b6d4e62eec7 in_seq=0 out_seq=0 2024-04-25T11:42:05.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.586+0000 7fd6257fa640 1 -- 172.21.15.90:0/274837159 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f1cc04e800 con 0x55f1cbcd43a0 2024-04-25T11:42:05.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.586+0000 7fd6257fa640 10 client.17706 handle_command_reply: tid=0 2024-04-25T11:42:05.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.586+0000 7fd64ac58740 1 client.17706 shutdown 2024-04-25T11:42:05.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.586+0000 7fd64ac58740 1 -- 172.21.15.90:0/274837159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd1f6b0 msgr2=0x55f1cc0fee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.586+0000 7fd64ac58740 1 --2- 172.21.15.90:0/274837159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd1f6b0 0x55f1cc0fee50 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fd6380a7310 tx=0x7fd638009580 comp rx=0 tx=0).stop 2024-04-25T11:42:05.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.587+0000 7fd64ac58740 1 -- 172.21.15.90:0/274837159 shutdown_connections 2024-04-25T11:42:05.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.587+0000 7fd64ac58740 1 --2- 172.21.15.90:0/274837159 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f1cbcd43a0 0x55f1cc168460 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.587+0000 7fd64ac58740 1 --2- 172.21.15.90:0/274837159 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f1cbd40390 0x55f1cc164690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.587+0000 7fd64ac58740 1 --2- 172.21.15.90:0/274837159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1cbd1f6b0 0x55f1cc0fee50 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.587+0000 7fd64ac58740 1 --2- 172.21.15.90:0/274837159 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f1cbd174d0 0x55f1cc0fe910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.587+0000 7fd64ac58740 1 -- 172.21.15.90:0/274837159 >> 172.21.15.90:0/274837159 conn(0x55f1cbd344e0 msgr2=0x55f1cc0dc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:05.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.587+0000 7fd6257fa640 0 client.17706 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:42:05.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.587+0000 7fd64ac58740 1 -- 172.21.15.90:0/274837159 shutdown_connections 2024-04-25T11:42:05.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.587+0000 7fd64ac58740 1 -- 172.21.15.90:0/274837159 wait complete. 2024-04-25T11:42:05.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.587+0000 7fd64ac58740 20 client.17706 trim_cache size 0 max 16384 2024-04-25T11:42:05.580 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:42:05.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.587+0000 7fd643dab640 1 -- 172.21.15.90:0/1676461847 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd6280afc60 msgr2=0x7fd6280b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.587+0000 7fd643dab640 1 --2- 172.21.15.90:0/1676461847 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd6280afc60 0x7fd6280b2170 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd62c000c00 tx=0x7fd62c031400 comp rx=0 tx=0).stop 2024-04-25T11:42:05.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.587+0000 7fd643dab640 1 -- 172.21.15.90:0/1676461847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd63c108a20 msgr2=0x7fd63c19ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.587+0000 7fd643dab640 1 --2- 172.21.15.90:0/1676461847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd63c108a20 0x7fd63c19ac30 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fd63800e980 tx=0x7fd63800ee40 comp rx=0 tx=0).stop 2024-04-25T11:42:05.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.587+0000 7fd643dab640 1 -- 172.21.15.90:0/1676461847 shutdown_connections 2024-04-25T11:42:05.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.587+0000 7fd643dab640 1 --2- 172.21.15.90:0/1676461847 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd6280afc60 0x7fd6280b2170 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.587+0000 7fd643dab640 1 --2- 172.21.15.90:0/1676461847 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd63c10d380 0x7fd63c19eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.587+0000 7fd643dab640 1 --2- 172.21.15.90:0/1676461847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd63c108a20 0x7fd63c19ac30 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.587+0000 7fd643dab640 1 --2- 172.21.15.90:0/1676461847 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd63c102af0 0x7fd63c19a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.587+0000 7fd643dab640 1 -- 172.21.15.90:0/1676461847 >> 172.21.15.90:0/1676461847 conn(0x7fd63c078100 msgr2=0x7fd63c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:05.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.587+0000 7fd643dab640 1 -- 172.21.15.90:0/1676461847 shutdown_connections 2024-04-25T11:42:05.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.587+0000 7fd643dab640 1 -- 172.21.15.90:0/1676461847 wait complete. 2024-04-25T11:42:05.584 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T11:42:05.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:05 smithi090 ceph-mon[29575]: pgmap v213: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 2.2 MiB/s rd, 419 MiB/s wr, 204 op/s 2024-04-25T11:42:05.726 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:05 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/167949890' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:42:05.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.772+0000 7f8861ffb640 1 -- 172.21.15.90:0/3081797387 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8858031810 con 0x7f885c0a57f0 2024-04-25T11:42:05.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.772+0000 7f8863fff640 1 -- 172.21.15.90:0/3081797387 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f885c0a57f0 msgr2=0x7f885c0a5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.772+0000 7f8863fff640 1 --2- 172.21.15.90:0/3081797387 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f885c0a57f0 0x7f885c0a5c00 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f8858009900 tx=0x7f885802a4d0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.773+0000 7f8863fff640 1 -- 172.21.15.90:0/3081797387 shutdown_connections 2024-04-25T11:42:05.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.773+0000 7f8863fff640 1 --2- 172.21.15.90:0/3081797387 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f885c0b86c0 0x7f885c0baae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.773+0000 7f8863fff640 1 --2- 172.21.15.90:0/3081797387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f885c0b5d60 0x7f885c0b8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.773+0000 7f8863fff640 1 --2- 172.21.15.90:0/3081797387 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f885c0a57f0 0x7f885c0a5c00 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.773+0000 7f8863fff640 1 -- 172.21.15.90:0/3081797387 >> 172.21.15.90:0/3081797387 conn(0x7f885c01aed0 msgr2=0x7f885c01b310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:05.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.774+0000 7f8863fff640 1 -- 172.21.15.90:0/3081797387 shutdown_connections 2024-04-25T11:42:05.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.774+0000 7f8863fff640 1 -- 172.21.15.90:0/3081797387 wait complete. 2024-04-25T11:42:05.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.776+0000 7f8863fff640 1 Processor -- start 2024-04-25T11:42:05.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.776+0000 7f8863fff640 1 -- start start 2024-04-25T11:42:05.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.777+0000 7f8863fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f885c0b5d60 0x7f885c0ae910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.777+0000 7f8863fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f885c0b86c0 0x7f885c0aee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.777+0000 7f8863fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f885c0af390 0x7f885c0b3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.777+0000 7f8863fff640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f885c0be300 con 0x7f885c0af390 2024-04-25T11:42:05.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.777+0000 7f8863fff640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f885c0be180 con 0x7f885c0b5d60 2024-04-25T11:42:05.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.777+0000 7f8863fff640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f885c0be480 con 0x7f885c0b86c0 2024-04-25T11:42:05.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.781+0000 7f88637fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f885c0af390 0x7f885c0b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.781+0000 7f88637fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f885c0af390 0x7f885c0b3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58356/0 (socket says 172.21.15.90:58356) 2024-04-25T11:42:05.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.781+0000 7f88637fe640 1 -- 172.21.15.90:0/4268968331 learned_addr learned my addr 172.21.15.90:0/4268968331 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:05.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.782+0000 7f88637fe640 1 -- 172.21.15.90:0/4268968331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f885c0b86c0 msgr2=0x7f885c0aee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.782+0000 7f88637fe640 1 --2- 172.21.15.90:0/4268968331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f885c0b86c0 0x7f885c0aee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.782+0000 7f88637fe640 1 -- 172.21.15.90:0/4268968331 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f885c0b5d60 msgr2=0x7f885c0ae910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.782+0000 7f88637fe640 1 --2- 172.21.15.90:0/4268968331 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f885c0b5d60 0x7f885c0ae910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.782+0000 7f88637fe640 1 -- 172.21.15.90:0/4268968331 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f885c0b3850 con 0x7f885c0af390 2024-04-25T11:42:05.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.782+0000 7f88637fe640 1 --2- 172.21.15.90:0/4268968331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f885c0af390 0x7f885c0b3310 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f8864066010 tx=0x7f886406fb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:05.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.783+0000 7f8868a03640 1 -- 172.21.15.90:0/4268968331 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886407e070 con 0x7f885c0af390 2024-04-25T11:42:05.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.783+0000 7f8868a03640 1 -- 172.21.15.90:0/4268968331 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f886406cd50 con 0x7f885c0af390 2024-04-25T11:42:05.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.785+0000 7f8868a03640 1 -- 172.21.15.90:0/4268968331 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8864079e00 con 0x7f885c0af390 2024-04-25T11:42:05.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.786+0000 7f8863fff640 1 -- 172.21.15.90:0/4268968331 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f885c0b3b40 con 0x7f885c0af390 2024-04-25T11:42:05.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.786+0000 7f8863fff640 1 -- 172.21.15.90:0/4268968331 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f885c0b4040 con 0x7f885c0af390 2024-04-25T11:42:05.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.797+0000 7f8870992740 1 Processor -- start 2024-04-25T11:42:05.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.800+0000 7f8868a03640 1 -- 172.21.15.90:0/4268968331 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f8864079410 con 0x7f885c0af390 2024-04-25T11:42:05.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.802+0000 7f8870992740 1 -- start start 2024-04-25T11:42:05.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.802+0000 7f8868a03640 1 --2- 172.21.15.90:0/4268968331 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f884c0afc10 0x7f884c0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.803+0000 7f8870992740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556d07047390 0x556d073e8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.803+0000 7f8868a03640 1 -- 172.21.15.90:0/4268968331 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f8864142fa0 con 0x7f885c0af390 2024-04-25T11:42:05.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.803+0000 7f8870992740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d0701e4d0 0x556d073e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.803+0000 7f8870992740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d070266b0 0x556d073f5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.803+0000 7f8870992740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556d0734dd80 con 0x556d070266b0 2024-04-25T11:42:05.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.803+0000 7f8870992740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x556d07307a20 con 0x556d07047390 2024-04-25T11:42:05.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.803+0000 7f8870992740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x556d07349d40 con 0x556d0701e4d0 2024-04-25T11:42:05.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.803+0000 7f88627fc640 1 --2- 172.21.15.90:0/4268968331 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f884c0afc10 0x7f884c0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.803+0000 7f88637fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d070266b0 0x556d073f5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.803+0000 7f88637fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d070266b0 0x556d073f5890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58360/0 (socket says 172.21.15.90:58360) 2024-04-25T11:42:05.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.803+0000 7f88637fe640 1 -- 172.21.15.90:0/1885337180 learned_addr learned my addr 172.21.15.90:0/1885337180 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:05.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.803+0000 7f88637fe640 1 -- 172.21.15.90:0/1885337180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d0701e4d0 msgr2=0x556d073e8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.803+0000 7f88637fe640 1 --2- 172.21.15.90:0/1885337180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d0701e4d0 0x556d073e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.803+0000 7f88637fe640 1 -- 172.21.15.90:0/1885337180 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556d07047390 msgr2=0x556d073e8a00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:42:05.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.803+0000 7f88637fe640 1 --2- 172.21.15.90:0/1885337180 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556d07047390 0x556d073e8a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.804+0000 7f88637fe640 1 -- 172.21.15.90:0/1885337180 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d07312c30 con 0x556d070266b0 2024-04-25T11:42:05.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.804+0000 7f88627fc640 1 --2- 172.21.15.90:0/4268968331 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f884c0afc10 0x7f884c0b2120 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f885c0b7bf0 tx=0x7f8854009200 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:05.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.804+0000 7f88637fe640 1 --2- 172.21.15.90:0/1885337180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d070266b0 0x556d073f5890 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f886410b010 tx=0x7f886410ab90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fba4ccd0129cb4a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:05.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.804+0000 7f8849ffb640 1 -- 172.21.15.90:0/1885337180 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886410b080 con 0x556d070266b0 2024-04-25T11:42:05.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.804+0000 7f8849ffb640 1 -- 172.21.15.90:0/1885337180 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f886410f430 con 0x556d070266b0 2024-04-25T11:42:05.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.804+0000 7f8849ffb640 1 -- 172.21.15.90:0/1885337180 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886410a520 con 0x556d070266b0 2024-04-25T11:42:05.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.805+0000 7f8870992740 1 -- 172.21.15.90:0/1885337180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d070266b0 msgr2=0x556d073f5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.805+0000 7f8870992740 1 --2- 172.21.15.90:0/1885337180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d070266b0 0x556d073f5890 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f886410b010 tx=0x7f886410ab90 comp rx=0 tx=0).stop 2024-04-25T11:42:05.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.805+0000 7f8870992740 1 -- 172.21.15.90:0/1885337180 shutdown_connections 2024-04-25T11:42:05.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.805+0000 7f8870992740 1 --2- 172.21.15.90:0/1885337180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d070266b0 0x556d073f5890 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.805+0000 7f8870992740 1 --2- 172.21.15.90:0/1885337180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d0701e4d0 0x556d073e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.805+0000 7f8870992740 1 --2- 172.21.15.90:0/1885337180 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556d07047390 0x556d073e8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.805+0000 7f8870992740 1 -- 172.21.15.90:0/1885337180 >> 172.21.15.90:0/1885337180 conn(0x556d0703b4e0 msgr2=0x556d073e51b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:05.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.805+0000 7f8870992740 1 -- 172.21.15.90:0/1885337180 shutdown_connections 2024-04-25T11:42:05.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.806+0000 7f8870992740 1 -- 172.21.15.90:0/1885337180 wait complete. 2024-04-25T11:42:05.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.806+0000 7f8870992740 1 -- start start 2024-04-25T11:42:05.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.806+0000 7f8870992740 1 Processor -- start 2024-04-25T11:42:05.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.807+0000 7f8870992740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556d0701e4d0 0x556d073f3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.807+0000 7f8870992740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d070266b0 0x556d073f36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.807+0000 7f8870992740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d07047390 0x556d073ff430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.807+0000 7f8870992740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556d073f5ec0 con 0x556d070266b0 2024-04-25T11:42:05.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.807+0000 7f8870992740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x556d072d4dc0 con 0x556d0701e4d0 2024-04-25T11:42:05.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.807+0000 7f8870992740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x556d073f6040 con 0x556d07047390 2024-04-25T11:42:05.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.807+0000 7f8870992740 10 client.0 _ll_register_callbacks cb 0x556d072f0600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:05.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.807+0000 7f88627fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d070266b0 0x556d073f36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.807+0000 7f88627fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d070266b0 0x556d073f36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58364/0 (socket says 172.21.15.90:58364) 2024-04-25T11:42:05.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.807+0000 7f88627fc640 1 -- 172.21.15.90:0/1732038811 learned_addr learned my addr 172.21.15.90:0/1732038811 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:05.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.807+0000 7f88637fe640 1 --2- 172.21.15.90:0/1732038811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d07047390 0x556d073ff430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.807+0000 7f8862ffd640 1 --2- 172.21.15.90:0/1732038811 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556d0701e4d0 0x556d073f3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.808+0000 7f88627fc640 1 -- 172.21.15.90:0/1732038811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d07047390 msgr2=0x556d073ff430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.808+0000 7f88627fc640 1 --2- 172.21.15.90:0/1732038811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d07047390 0x556d073ff430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.808+0000 7f88627fc640 1 -- 172.21.15.90:0/1732038811 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556d0701e4d0 msgr2=0x556d073f3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.808+0000 7f88627fc640 1 --2- 172.21.15.90:0/1732038811 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556d0701e4d0 0x556d073f3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.808+0000 7f88627fc640 1 -- 172.21.15.90:0/1732038811 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d0705f3f0 con 0x556d070266b0 2024-04-25T11:42:05.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.808+0000 7f88627fc640 1 --2- 172.21.15.90:0/1732038811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d070266b0 0x556d073f36a0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f885400def0 tx=0x7f8854004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=441ee4267faef31a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:05.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.808+0000 7f8842ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:05.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.809+0000 7f8842ffd640 1 -- 172.21.15.90:0/1732038811 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8854026060 con 0x556d070266b0 2024-04-25T11:42:05.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.809+0000 7f8842ffd640 1 -- 172.21.15.90:0/1732038811 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f8854016dc0 con 0x556d070266b0 2024-04-25T11:42:05.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.809+0000 7f8870992740 1 -- 172.21.15.90:0/1732038811 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f886410f430 con 0x556d070266b0 2024-04-25T11:42:05.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.809+0000 7f8842ffd640 1 -- 172.21.15.90:0/1732038811 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f885400f040 con 0x556d070266b0 2024-04-25T11:42:05.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.809+0000 7f8870992740 10 client.17718 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:05.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.809+0000 7f8870992740 1 -- 172.21.15.90:0/1732038811 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556d071297d0 con 0x556d070266b0 2024-04-25T11:42:05.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.809+0000 7f8842ffd640 1 -- 172.21.15.90:0/1732038811 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f886410f430 con 0x556d070266b0 2024-04-25T11:42:05.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.810+0000 7f8842ffd640 1 -- 172.21.15.90:0/1732038811 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f88540124e0 con 0x556d070266b0 2024-04-25T11:42:05.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.810+0000 7f8870992740 10 client.17718 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:05.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.810+0000 7f8870992740 10 client.17718 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:42:05.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.810+0000 7f8870992740 20 client.17718 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:05.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.810+0000 7f8870992740 1 --2- 172.21.15.90:0/1732038811 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x556d07005f60 0x556d074057e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.810+0000 7f8870992740 4 client.17718 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:42:05.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.810+0000 7f8870992740 1 -- 172.21.15.90:0/1732038811 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d0735b990 con 0x556d07005f60 2024-04-25T11:42:05.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.811+0000 7f8862ffd640 1 --2- 172.21.15.90:0/1732038811 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x556d07005f60 0x556d074057e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:05 smithi138 ceph-mon[32328]: pgmap v213: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 2.2 MiB/s rd, 419 MiB/s wr, 204 op/s 2024-04-25T11:42:05.802 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:05 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/167949890' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:42:05.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.813+0000 7f8862ffd640 1 --2- 172.21.15.90:0/1732038811 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x556d07005f60 0x556d074057e0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4114ea2483dda568 server_cookie=4d591a59743ec04f in_seq=0 out_seq=0 2024-04-25T11:42:05.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.813+0000 7f8842ffd640 10 client.17718 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:42:05.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.815+0000 7f8842ffd640 1 -- 172.21.15.90:0/1732038811 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556d0735b990 con 0x556d07005f60 2024-04-25T11:42:05.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.815+0000 7f8842ffd640 10 client.17718 handle_command_reply: tid=0 2024-04-25T11:42:05.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.815+0000 7f8870992740 1 client.17718 shutdown 2024-04-25T11:42:05.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.815+0000 7f8870992740 1 -- 172.21.15.90:0/1732038811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d070266b0 msgr2=0x556d073f36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.815+0000 7f8870992740 1 --2- 172.21.15.90:0/1732038811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d070266b0 0x556d073f36a0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f885400def0 tx=0x7f8854004770 comp rx=0 tx=0).stop 2024-04-25T11:42:05.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.815+0000 7f8870992740 1 -- 172.21.15.90:0/1732038811 shutdown_connections 2024-04-25T11:42:05.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.815+0000 7f8870992740 1 --2- 172.21.15.90:0/1732038811 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x556d07005f60 0x556d074057e0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.815+0000 7f8842ffd640 0 client.17718 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:42:05.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.815+0000 7f8870992740 1 --2- 172.21.15.90:0/1732038811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d07047390 0x556d073ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.815+0000 7f8870992740 1 --2- 172.21.15.90:0/1732038811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d070266b0 0x556d073f36a0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.815+0000 7f8870992740 1 --2- 172.21.15.90:0/1732038811 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556d0701e4d0 0x556d073f3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.815+0000 7f8870992740 1 -- 172.21.15.90:0/1732038811 >> 172.21.15.90:0/1732038811 conn(0x556d0703b4e0 msgr2=0x556d073ea720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:05.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.815+0000 7f8870992740 1 -- 172.21.15.90:0/1732038811 shutdown_connections 2024-04-25T11:42:05.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.815+0000 7f8870992740 1 -- 172.21.15.90:0/1732038811 wait complete. 2024-04-25T11:42:05.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.816+0000 7f8870992740 20 client.17718 trim_cache size 0 max 16384 2024-04-25T11:42:05.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.837+0000 7f8870992740 1 Processor -- start 2024-04-25T11:42:05.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.837+0000 7f8870992740 1 -- start start 2024-04-25T11:42:05.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.837+0000 7f8870992740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d0701e4d0 0x556d073f5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.837+0000 7f8870992740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556d070266b0 0x556d073e8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.837+0000 7f8870992740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d07047390 0x556d073ff010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.837+0000 7f8870992740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f885c151ca0 con 0x556d0701e4d0 2024-04-25T11:42:05.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.837+0000 7f8870992740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f885c1537e0 con 0x556d070266b0 2024-04-25T11:42:05.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.837+0000 7f8870992740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f885c152370 con 0x556d07047390 2024-04-25T11:42:05.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.837+0000 7f8862ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d0701e4d0 0x556d073f5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.837+0000 7f8862ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d0701e4d0 0x556d073f5890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58368/0 (socket says 172.21.15.90:58368) 2024-04-25T11:42:05.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.837+0000 7f8862ffd640 1 -- 172.21.15.90:0/3378295216 learned_addr learned my addr 172.21.15.90:0/3378295216 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:05.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.838+0000 7f8862ffd640 1 -- 172.21.15.90:0/3378295216 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d07047390 msgr2=0x556d073ff010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.838+0000 7f8862ffd640 1 --2- 172.21.15.90:0/3378295216 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d07047390 0x556d073ff010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.838+0000 7f8862ffd640 1 -- 172.21.15.90:0/3378295216 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556d070266b0 msgr2=0x556d073e8ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:42:05.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.838+0000 7f8862ffd640 1 --2- 172.21.15.90:0/3378295216 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556d070266b0 0x556d073e8ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.838+0000 7f8862ffd640 1 -- 172.21.15.90:0/3378295216 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d073e0540 con 0x556d0701e4d0 2024-04-25T11:42:05.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.838+0000 7f8862ffd640 1 --2- 172.21.15.90:0/3378295216 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d0701e4d0 0x556d073f5890 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f8858009230 tx=0x7f88580043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8e2c67d5d2359cc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:05.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.839+0000 7f88497fa640 1 -- 172.21.15.90:0/3378295216 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8858035040 con 0x556d0701e4d0 2024-04-25T11:42:05.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.839+0000 7f88497fa640 1 -- 172.21.15.90:0/3378295216 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88580044b0 con 0x556d0701e4d0 2024-04-25T11:42:05.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.839+0000 7f88497fa640 1 -- 172.21.15.90:0/3378295216 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885803c9a0 con 0x556d0701e4d0 2024-04-25T11:42:05.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.841+0000 7f8870992740 1 -- 172.21.15.90:0/3378295216 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d0701e4d0 msgr2=0x556d073f5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.841+0000 7f8870992740 1 --2- 172.21.15.90:0/3378295216 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d0701e4d0 0x556d073f5890 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f8858009230 tx=0x7f88580043c0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.841+0000 7f8870992740 1 -- 172.21.15.90:0/3378295216 shutdown_connections 2024-04-25T11:42:05.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.841+0000 7f8870992740 1 --2- 172.21.15.90:0/3378295216 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d07047390 0x556d073ff010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.841+0000 7f8870992740 1 --2- 172.21.15.90:0/3378295216 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556d070266b0 0x556d073e8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.841+0000 7f8870992740 1 --2- 172.21.15.90:0/3378295216 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d0701e4d0 0x556d073f5890 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.841+0000 7f8870992740 1 -- 172.21.15.90:0/3378295216 >> 172.21.15.90:0/3378295216 conn(0x556d0703b4e0 msgr2=0x556d073e8790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:05.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.841+0000 7f8870992740 1 -- 172.21.15.90:0/3378295216 shutdown_connections 2024-04-25T11:42:05.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.841+0000 7f8870992740 1 -- 172.21.15.90:0/3378295216 wait complete. 2024-04-25T11:42:05.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.842+0000 7f8870992740 1 -- start start 2024-04-25T11:42:05.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.842+0000 7f8870992740 1 Processor -- start 2024-04-25T11:42:05.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.842+0000 7f8870992740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d0701e4d0 0x556d073fb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.842+0000 7f8870992740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556d070266b0 0x556d073f6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.843+0000 7f8870992740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d07047390 0x556d073f6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.843+0000 7f8870992740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f885c153110 con 0x556d07047390 2024-04-25T11:42:05.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.843+0000 7f8870992740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f885c152a40 con 0x556d070266b0 2024-04-25T11:42:05.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.843+0000 7f8870992740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f885c0a6ac0 con 0x556d0701e4d0 2024-04-25T11:42:05.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.843+0000 7f8870992740 10 client.0 _ll_register_callbacks cb 0x7f8854009550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:05.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.843+0000 7f88637fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d07047390 0x556d073f6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.843+0000 7f88637fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d07047390 0x556d073f6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58380/0 (socket says 172.21.15.90:58380) 2024-04-25T11:42:05.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.843+0000 7f88637fe640 1 -- 172.21.15.90:0/189742001 learned_addr learned my addr 172.21.15.90:0/189742001 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:05.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.843+0000 7f8862ffd640 1 --2- 172.21.15.90:0/189742001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d0701e4d0 0x556d073fb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.843+0000 7f88637fe640 1 -- 172.21.15.90:0/189742001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d0701e4d0 msgr2=0x556d073fb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.843+0000 7f88637fe640 1 --2- 172.21.15.90:0/189742001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d0701e4d0 0x556d073fb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.843+0000 7f88637fe640 1 -- 172.21.15.90:0/189742001 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556d070266b0 msgr2=0x556d073f6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:05.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.843+0000 7f88637fe640 1 --2- 172.21.15.90:0/189742001 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556d070266b0 0x556d073f6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.843+0000 7f88637fe640 1 -- 172.21.15.90:0/189742001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d073f74d0 con 0x556d07047390 2024-04-25T11:42:05.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.844+0000 7f8862ffd640 1 --2- 172.21.15.90:0/189742001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d0701e4d0 0x556d073fb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:42:05.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.844+0000 7f88637fe640 1 --2- 172.21.15.90:0/189742001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d07047390 0x556d073f6f90 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f88641082f0 tx=0x7f886410f300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ec01bf391d80cdb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:05.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.844+0000 7f8842ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:05.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.844+0000 7f8842ffd640 1 -- 172.21.15.90:0/189742001 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886412a060 con 0x556d07047390 2024-04-25T11:42:05.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.844+0000 7f8842ffd640 1 -- 172.21.15.90:0/189742001 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f886410a4f0 con 0x556d07047390 2024-04-25T11:42:05.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.844+0000 7f8870992740 1 -- 172.21.15.90:0/189742001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88580044b0 con 0x556d07047390 2024-04-25T11:42:05.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.844+0000 7f8842ffd640 1 -- 172.21.15.90:0/189742001 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8864113ee0 con 0x556d07047390 2024-04-25T11:42:05.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.844+0000 7f8870992740 10 client.17724 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:05.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.844+0000 7f8870992740 1 -- 172.21.15.90:0/189742001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556d0746d7c0 con 0x556d07047390 2024-04-25T11:42:05.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.845+0000 7f8842ffd640 1 -- 172.21.15.90:0/189742001 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8864116030 con 0x556d07047390 2024-04-25T11:42:05.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.845+0000 7f8842ffd640 1 -- 172.21.15.90:0/189742001 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f8864117070 con 0x556d07047390 2024-04-25T11:42:05.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.845+0000 7f8870992740 10 client.17724 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:05.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.845+0000 7f8870992740 10 client.17724 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:42:05.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.845+0000 7f8870992740 20 client.17724 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:05.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.845+0000 7f8870992740 1 --2- 172.21.15.90:0/189742001 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x556d06fdb3a0 0x556d0746fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:05.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.845+0000 7f8870992740 4 client.17724 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:42:05.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.845+0000 7f8870992740 1 -- 172.21.15.90:0/189742001 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556d07355800 con 0x556d06fdb3a0 2024-04-25T11:42:05.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.846+0000 7f8862ffd640 1 --2- 172.21.15.90:0/189742001 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x556d06fdb3a0 0x556d0746fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:05.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.847+0000 7f8862ffd640 1 --2- 172.21.15.90:0/189742001 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x556d06fdb3a0 0x556d0746fb60 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bd378c9c1f058a05 server_cookie=1017232a7daf3bf1 in_seq=0 out_seq=0 2024-04-25T11:42:05.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.847+0000 7f8842ffd640 10 client.17724 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:42:05.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.848+0000 7f8842ffd640 1 -- 172.21.15.90:0/189742001 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556d07355800 con 0x556d06fdb3a0 2024-04-25T11:42:05.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.848+0000 7f8842ffd640 10 client.17724 handle_command_reply: tid=0 2024-04-25T11:42:05.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.848+0000 7f8870992740 1 client.17724 shutdown 2024-04-25T11:42:05.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:05 smithi156 ceph-mon[33045]: pgmap v213: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 2.2 MiB/s rd, 419 MiB/s wr, 204 op/s 2024-04-25T11:42:05.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:05 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/167949890' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:42:05.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.848+0000 7f8870992740 1 -- 172.21.15.90:0/189742001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d07047390 msgr2=0x556d073f6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.848+0000 7f8870992740 1 --2- 172.21.15.90:0/189742001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d07047390 0x556d073f6f90 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f88641082f0 tx=0x7f886410f300 comp rx=0 tx=0).stop 2024-04-25T11:42:05.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.848+0000 7f8870992740 1 -- 172.21.15.90:0/189742001 shutdown_connections 2024-04-25T11:42:05.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.848+0000 7f8870992740 1 --2- 172.21.15.90:0/189742001 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x556d06fdb3a0 0x556d0746fb60 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.849+0000 7f8870992740 1 --2- 172.21.15.90:0/189742001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d07047390 0x556d073f6f90 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.849+0000 7f8870992740 1 --2- 172.21.15.90:0/189742001 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556d070266b0 0x556d073f6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.849+0000 7f8842ffd640 0 client.17724 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:42:05.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.849+0000 7f8870992740 1 --2- 172.21.15.90:0/189742001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d0701e4d0 0x556d073fb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.849+0000 7f8870992740 1 -- 172.21.15.90:0/189742001 >> 172.21.15.90:0/189742001 conn(0x556d0703b4e0 msgr2=0x556d073e39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:05.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.849+0000 7f8870992740 1 -- 172.21.15.90:0/189742001 shutdown_connections 2024-04-25T11:42:05.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.849+0000 7f8870992740 1 -- 172.21.15.90:0/189742001 wait complete. 2024-04-25T11:42:05.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.849+0000 7f8870992740 20 client.17724 trim_cache size 0 max 16384 2024-04-25T11:42:05.838 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:42:05.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.850+0000 7f8863fff640 1 -- 172.21.15.90:0/4268968331 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f884c0afc10 msgr2=0x7f884c0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.850+0000 7f8863fff640 1 --2- 172.21.15.90:0/4268968331 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f884c0afc10 0x7f884c0b2120 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f885c0b7bf0 tx=0x7f8854009200 comp rx=0 tx=0).stop 2024-04-25T11:42:05.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.850+0000 7f8863fff640 1 -- 172.21.15.90:0/4268968331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f885c0af390 msgr2=0x7f885c0b3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:05.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.850+0000 7f8863fff640 1 --2- 172.21.15.90:0/4268968331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f885c0af390 0x7f885c0b3310 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f8864066010 tx=0x7f886406fb20 comp rx=0 tx=0).stop 2024-04-25T11:42:05.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.850+0000 7f8863fff640 1 -- 172.21.15.90:0/4268968331 shutdown_connections 2024-04-25T11:42:05.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.850+0000 7f8863fff640 1 --2- 172.21.15.90:0/4268968331 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f884c0afc10 0x7f884c0b2120 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.850+0000 7f8863fff640 1 --2- 172.21.15.90:0/4268968331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f885c0af390 0x7f885c0b3310 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.850+0000 7f8863fff640 1 --2- 172.21.15.90:0/4268968331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f885c0b86c0 0x7f885c0aee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.850+0000 7f8863fff640 1 --2- 172.21.15.90:0/4268968331 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f885c0b5d60 0x7f885c0ae910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:05.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.850+0000 7f8863fff640 1 -- 172.21.15.90:0/4268968331 >> 172.21.15.90:0/4268968331 conn(0x7f885c01aed0 msgr2=0x7f885c0b6170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:05.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.850+0000 7f8863fff640 1 -- 172.21.15.90:0/4268968331 shutdown_connections 2024-04-25T11:42:05.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:05.850+0000 7f8863fff640 1 -- 172.21.15.90:0/4268968331 wait complete. 2024-04-25T11:42:05.858 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T11:42:06.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.033+0000 7f5d841ce640 1 -- 172.21.15.90:0/476531369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d7c10d380 msgr2=0x7f5d7c10f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:06.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.033+0000 7f5d841ce640 1 --2- 172.21.15.90:0/476531369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d7c10d380 0x7f5d7c10f7a0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f5d74009ca0 tx=0x7f5d7402a8c0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.033+0000 7f5d841ce640 1 -- 172.21.15.90:0/476531369 shutdown_connections 2024-04-25T11:42:06.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.033+0000 7f5d841ce640 1 --2- 172.21.15.90:0/476531369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d7c10d380 0x7f5d7c10f7a0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.033+0000 7f5d841ce640 1 --2- 172.21.15.90:0/476531369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d7c102af0 0x7f5d7c102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.033+0000 7f5d841ce640 1 --2- 172.21.15.90:0/476531369 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d7c108a20 0x7f5d7c1025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.033+0000 7f5d841ce640 1 -- 172.21.15.90:0/476531369 >> 172.21.15.90:0/476531369 conn(0x7f5d7c078100 msgr2=0x7f5d7c078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:06.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.033+0000 7f5d841ce640 1 -- 172.21.15.90:0/476531369 shutdown_connections 2024-04-25T11:42:06.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.033+0000 7f5d841ce640 1 -- 172.21.15.90:0/476531369 wait complete. 2024-04-25T11:42:06.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.034+0000 7f5d841ce640 1 Processor -- start 2024-04-25T11:42:06.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.034+0000 7f5d841ce640 1 -- start start 2024-04-25T11:42:06.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.034+0000 7f5d841ce640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d7c102af0 0x7f5d7c1995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:06.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.034+0000 7f5d841ce640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d7c108a20 0x7f5d7c199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:06.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.034+0000 7f5d841ce640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d7c10d380 0x7f5d7c19de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:06.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.034+0000 7f5d841ce640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7c112c00 con 0x7f5d7c102af0 2024-04-25T11:42:06.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.034+0000 7f5d841ce640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7c112a80 con 0x7f5d7c108a20 2024-04-25T11:42:06.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.034+0000 7f5d841ce640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7c112d80 con 0x7f5d7c10d380 2024-04-25T11:42:06.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.034+0000 7f5d81f6a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d7c102af0 0x7f5d7c1995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:06.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.034+0000 7f5d81769640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d7c108a20 0x7f5d7c199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:06.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.034+0000 7f5d81f6a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d7c102af0 0x7f5d7c1995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58392/0 (socket says 172.21.15.90:58392) 2024-04-25T11:42:06.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.034+0000 7f5d81769640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d7c108a20 0x7f5d7c199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:34792/0 (socket says 172.21.15.90:34792) 2024-04-25T11:42:06.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.034+0000 7f5d81f6a640 1 -- 172.21.15.90:0/2738482831 learned_addr learned my addr 172.21.15.90:0/2738482831 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:06.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.035+0000 7f5d8276b640 1 --2- 172.21.15.90:0/2738482831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d7c10d380 0x7f5d7c19de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:06.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.035+0000 7f5d81769640 1 -- 172.21.15.90:0/2738482831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d7c10d380 msgr2=0x7f5d7c19de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:06.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.035+0000 7f5d81769640 1 --2- 172.21.15.90:0/2738482831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d7c10d380 0x7f5d7c19de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.035+0000 7f5d81769640 1 -- 172.21.15.90:0/2738482831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d7c102af0 msgr2=0x7f5d7c1995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:06.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.035+0000 7f5d81769640 1 --2- 172.21.15.90:0/2738482831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d7c102af0 0x7f5d7c1995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.035+0000 7f5d81769640 1 -- 172.21.15.90:0/2738482831 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d7c19e440 con 0x7f5d7c108a20 2024-04-25T11:42:06.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.035+0000 7f5d81769640 1 --2- 172.21.15.90:0/2738482831 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d7c108a20 0x7f5d7c199b40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c06c4d0 tx=0x7f5d6c00ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:06.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.035+0000 7f5d72ffd640 1 -- 172.21.15.90:0/2738482831 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c01c070 con 0x7f5d7c108a20 2024-04-25T11:42:06.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.035+0000 7f5d72ffd640 1 -- 172.21.15.90:0/2738482831 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d6c004030 con 0x7f5d7c108a20 2024-04-25T11:42:06.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.035+0000 7f5d841ce640 1 -- 172.21.15.90:0/2738482831 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d7c19e730 con 0x7f5d7c108a20 2024-04-25T11:42:06.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.035+0000 7f5d72ffd640 1 -- 172.21.15.90:0/2738482831 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c003d00 con 0x7f5d7c108a20 2024-04-25T11:42:06.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.035+0000 7f5d841ce640 1 -- 172.21.15.90:0/2738482831 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d7c19ec30 con 0x7f5d7c108a20 2024-04-25T11:42:06.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.038+0000 7f5d8b07e740 1 Processor -- start 2024-04-25T11:42:06.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.038+0000 7f5d8b07e740 1 -- start start 2024-04-25T11:42:06.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.038+0000 7f5d72ffd640 1 -- 172.21.15.90:0/2738482831 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f5d6c01f070 con 0x7f5d7c108a20 2024-04-25T11:42:06.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.038+0000 7f5d8b07e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a9b390 0x562352e3c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:06.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.038+0000 7f5d8b07e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562352a724d0 0x562352e3cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:06.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.038+0000 7f5d8b07e740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562352a7a6b0 0x562352e496a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:06.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.038+0000 7f5d8b07e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562352da1d80 con 0x562352a9b390 2024-04-25T11:42:06.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.038+0000 7f5d8b07e740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x562352d5ba20 con 0x562352a7a6b0 2024-04-25T11:42:06.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.038+0000 7f5d8b07e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562352d9dd40 con 0x562352a724d0 2024-04-25T11:42:06.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.039+0000 7f5d8276b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a9b390 0x562352e3c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:06.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.039+0000 7f5d8276b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a9b390 0x562352e3c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58398/0 (socket says 172.21.15.90:58398) 2024-04-25T11:42:06.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.039+0000 7f5d8276b640 1 -- 172.21.15.90:0/2178728155 learned_addr learned my addr 172.21.15.90:0/2178728155 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:06.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.039+0000 7f5d81769640 1 --2- 172.21.15.90:0/2178728155 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562352a7a6b0 0x562352e496a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:06.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.039+0000 7f5d81f6a640 1 --2- 172.21.15.90:0/2178728155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562352a724d0 0x562352e3cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:06.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.039+0000 7f5d81f6a640 1 -- 172.21.15.90:0/2178728155 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562352a7a6b0 msgr2=0x562352e496a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:06.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.039+0000 7f5d81f6a640 1 --2- 172.21.15.90:0/2178728155 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562352a7a6b0 0x562352e496a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.039+0000 7f5d81f6a640 1 -- 172.21.15.90:0/2178728155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a9b390 msgr2=0x562352e3c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:06.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.039+0000 7f5d81f6a640 1 --2- 172.21.15.90:0/2178728155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a9b390 0x562352e3c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.039+0000 7f5d81f6a640 1 -- 172.21.15.90:0/2178728155 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562352d66c30 con 0x562352a724d0 2024-04-25T11:42:06.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.039+0000 7f5d8276b640 1 --2- 172.21.15.90:0/2178728155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a9b390 0x562352e3c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:42:06.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.039+0000 7f5d72ffd640 1 --2- 172.21.15.90:0/2738482831 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5d680afc60 0x7f5d680b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:06.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.039+0000 7f5d81f6a640 1 --2- 172.21.15.90:0/2178728155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562352a724d0 0x562352e3cd50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5d7800bc90 tx=0x7f5d780107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f2e3cbb4c733afd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:06.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.039+0000 7f5d8276b640 1 --2- 172.21.15.90:0/2738482831 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5d680afc60 0x7f5d680b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:06.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.040+0000 7f5d67fff640 1 -- 172.21.15.90:0/2178728155 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d78004230 con 0x562352a724d0 2024-04-25T11:42:06.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.040+0000 7f5d67fff640 1 -- 172.21.15.90:0/2178728155 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d7800be40 con 0x562352a724d0 2024-04-25T11:42:06.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.040+0000 7f5d67fff640 1 -- 172.21.15.90:0/2178728155 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d780116a0 con 0x562352a724d0 2024-04-25T11:42:06.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.040+0000 7f5d72ffd640 1 -- 172.21.15.90:0/2738482831 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f5d6c0e0830 con 0x7f5d7c108a20 2024-04-25T11:42:06.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.041+0000 7f5d8b07e740 1 -- 172.21.15.90:0/2178728155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562352a724d0 msgr2=0x562352e3cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:06.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.041+0000 7f5d8276b640 1 --2- 172.21.15.90:0/2738482831 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5d680afc60 0x7f5d680b2170 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5d7402f500 tx=0x7f5d7402f330 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:06.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.041+0000 7f5d8b07e740 1 --2- 172.21.15.90:0/2178728155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562352a724d0 0x562352e3cd50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5d7800bc90 tx=0x7f5d780107a0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.041+0000 7f5d8b07e740 1 -- 172.21.15.90:0/2178728155 shutdown_connections 2024-04-25T11:42:06.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.041+0000 7f5d8b07e740 1 --2- 172.21.15.90:0/2178728155 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562352a7a6b0 0x562352e496a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.041+0000 7f5d8b07e740 1 --2- 172.21.15.90:0/2178728155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562352a724d0 0x562352e3cd50 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.041+0000 7f5d8b07e740 1 --2- 172.21.15.90:0/2178728155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a9b390 0x562352e3c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.041+0000 7f5d8b07e740 1 -- 172.21.15.90:0/2178728155 >> 172.21.15.90:0/2178728155 conn(0x562352a8f4e0 msgr2=0x562352e38fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:06.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.041+0000 7f5d8b07e740 1 -- 172.21.15.90:0/2178728155 shutdown_connections 2024-04-25T11:42:06.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.041+0000 7f5d8b07e740 1 -- 172.21.15.90:0/2178728155 wait complete. 2024-04-25T11:42:06.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.041+0000 7f5d8b07e740 1 -- start start 2024-04-25T11:42:06.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.041+0000 7f5d8b07e740 1 Processor -- start 2024-04-25T11:42:06.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.042+0000 7f5d8b07e740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562352a724d0 0x562352e47210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:06.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.042+0000 7f5d8b07e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562352a7a6b0 0x562352e47750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:06.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.042+0000 7f5d8b07e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a9b390 0x562352e53210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:06.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.042+0000 7f5d8b07e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562352e49d90 con 0x562352a9b390 2024-04-25T11:42:06.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.042+0000 7f5d8b07e740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x562352d28dc0 con 0x562352a724d0 2024-04-25T11:42:06.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.042+0000 7f5d8b07e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562352e49f10 con 0x562352a7a6b0 2024-04-25T11:42:06.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.042+0000 7f5d8b07e740 10 client.0 _ll_register_callbacks cb 0x562352d44600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:06.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.042+0000 7f5d8276b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a9b390 0x562352e53210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:06.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.042+0000 7f5d81f6a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562352a724d0 0x562352e47210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:06.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.042+0000 7f5d8276b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a9b390 0x562352e53210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58408/0 (socket says 172.21.15.90:58408) 2024-04-25T11:42:06.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.042+0000 7f5d8276b640 1 -- 172.21.15.90:0/477829616 learned_addr learned my addr 172.21.15.90:0/477829616 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:06.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.042+0000 7f5d81f6a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562352a724d0 0x562352e47210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:34818/0 (socket says 172.21.15.90:34818) 2024-04-25T11:42:06.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.042+0000 7f5d81769640 1 --2- 172.21.15.90:0/477829616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562352a7a6b0 0x562352e47750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:06.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.043+0000 7f5d8276b640 1 -- 172.21.15.90:0/477829616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562352a7a6b0 msgr2=0x562352e47750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:06.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.043+0000 7f5d8276b640 1 --2- 172.21.15.90:0/477829616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562352a7a6b0 0x562352e47750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.043+0000 7f5d8276b640 1 -- 172.21.15.90:0/477829616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562352a724d0 msgr2=0x562352e47210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:06.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.043+0000 7f5d8276b640 1 --2- 172.21.15.90:0/477829616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562352a724d0 0x562352e47210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.043+0000 7f5d8276b640 1 -- 172.21.15.90:0/477829616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562352ab33f0 con 0x562352a9b390 2024-04-25T11:42:06.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.043+0000 7f5d81f6a640 1 --2- 172.21.15.90:0/477829616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562352a724d0 0x562352e47210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:42:06.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.043+0000 7f5d8276b640 1 --2- 172.21.15.90:0/477829616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a9b390 0x562352e53210 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f5d7402ade0 tx=0x7f5d7402a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c30c3f8c26125b07 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:06.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.043+0000 7f5d657fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:06.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.043+0000 7f5d657fa640 1 -- 172.21.15.90:0/477829616 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d74003d00 con 0x562352a9b390 2024-04-25T11:42:06.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.043+0000 7f5d8b07e740 1 -- 172.21.15.90:0/477829616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562352daf990 con 0x562352a9b390 2024-04-25T11:42:06.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.043+0000 7f5d657fa640 1 -- 172.21.15.90:0/477829616 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f5d74004040 con 0x562352a9b390 2024-04-25T11:42:06.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.043+0000 7f5d657fa640 1 -- 172.21.15.90:0/477829616 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5d74004240 con 0x562352a9b390 2024-04-25T11:42:06.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.043+0000 7f5d8b07e740 10 client.17736 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:06.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.043+0000 7f5d8b07e740 1 -- 172.21.15.90:0/477829616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562352b7d7d0 con 0x562352a9b390 2024-04-25T11:42:06.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.043+0000 7f5d657fa640 1 -- 172.21.15.90:0/477829616 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x562352daf990 con 0x562352a9b390 2024-04-25T11:42:06.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.044+0000 7f5d657fa640 1 -- 172.21.15.90:0/477829616 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f5d74051070 con 0x562352a9b390 2024-04-25T11:42:06.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.044+0000 7f5d8b07e740 10 client.17736 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:06.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.044+0000 7f5d8b07e740 10 client.17736 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:42:06.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.044+0000 7f5d8b07e740 20 client.17736 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:06.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.044+0000 7f5d8b07e740 1 --2- 172.21.15.90:0/477829616 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562352a59f60 0x562352e59670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:06.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.044+0000 7f5d8b07e740 4 client.17736 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:42:06.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.044+0000 7f5d8b07e740 1 -- 172.21.15.90:0/477829616 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562352da9800 con 0x562352a59f60 2024-04-25T11:42:06.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.044+0000 7f5d81f6a640 1 --2- 172.21.15.90:0/477829616 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562352a59f60 0x562352e59670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:06.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.044+0000 7f5d657fa640 10 client.17736 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:06.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.044+0000 7f5d81f6a640 1 --2- 172.21.15.90:0/477829616 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562352a59f60 0x562352e59670 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48a551ea98b5fe66 server_cookie=c6ca51bffe543de6 in_seq=0 out_seq=0 2024-04-25T11:42:06.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.045+0000 7f5d657fa640 1 -- 172.21.15.90:0/477829616 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562352da9800 con 0x562352a59f60 2024-04-25T11:42:06.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.045+0000 7f5d657fa640 10 client.17736 handle_command_reply: tid=0 2024-04-25T11:42:06.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.045+0000 7f5d8b07e740 1 client.17736 shutdown 2024-04-25T11:42:06.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.045+0000 7f5d8b07e740 1 -- 172.21.15.90:0/477829616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a9b390 msgr2=0x562352e53210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:06.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.045+0000 7f5d8b07e740 1 --2- 172.21.15.90:0/477829616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a9b390 0x562352e53210 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f5d7402ade0 tx=0x7f5d7402a8c0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.046+0000 7f5d8b07e740 1 -- 172.21.15.90:0/477829616 shutdown_connections 2024-04-25T11:42:06.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.046+0000 7f5d8b07e740 1 --2- 172.21.15.90:0/477829616 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562352a59f60 0x562352e59670 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.046+0000 7f5d8b07e740 1 --2- 172.21.15.90:0/477829616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a9b390 0x562352e53210 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.046+0000 7f5d657fa640 0 client.17736 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:06.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.046+0000 7f5d8b07e740 1 --2- 172.21.15.90:0/477829616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562352a7a6b0 0x562352e47750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.046+0000 7f5d8b07e740 1 --2- 172.21.15.90:0/477829616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562352a724d0 0x562352e47210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.046+0000 7f5d8b07e740 1 -- 172.21.15.90:0/477829616 >> 172.21.15.90:0/477829616 conn(0x562352a8f4e0 msgr2=0x562352e3e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:06.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.046+0000 7f5d8b07e740 1 -- 172.21.15.90:0/477829616 shutdown_connections 2024-04-25T11:42:06.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.046+0000 7f5d8b07e740 1 -- 172.21.15.90:0/477829616 wait complete. 2024-04-25T11:42:06.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.046+0000 7f5d8b07e740 20 client.17736 trim_cache size 0 max 16384 2024-04-25T11:42:06.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.074+0000 7f5d8b07e740 1 Processor -- start 2024-04-25T11:42:06.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.074+0000 7f5d8b07e740 1 -- start start 2024-04-25T11:42:06.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.074+0000 7f5d8b07e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a59f60 0x562352e468e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:06.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.074+0000 7f5d8b07e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562352a724d0 0x562352e48e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:06.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.074+0000 7f5d8b07e740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562352a7a6b0 0x562352e41430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:06.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.074+0000 7f5d8b07e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7c1ad770 con 0x562352a59f60 2024-04-25T11:42:06.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.074+0000 7f5d8b07e740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7c1af2b0 con 0x562352a7a6b0 2024-04-25T11:42:06.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.074+0000 7f5d8b07e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7c1ade40 con 0x562352a724d0 2024-04-25T11:42:06.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.075+0000 7f5d8276b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562352a7a6b0 0x562352e41430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:06.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.075+0000 7f5d8276b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562352a7a6b0 0x562352e41430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:34832/0 (socket says 172.21.15.90:34832) 2024-04-25T11:42:06.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.075+0000 7f5d8276b640 1 -- 172.21.15.90:0/3456585477 learned_addr learned my addr 172.21.15.90:0/3456585477 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:06.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.075+0000 7f5d81769640 1 --2- 172.21.15.90:0/3456585477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562352a724d0 0x562352e48e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:06.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.075+0000 7f5d81f6a640 1 --2- 172.21.15.90:0/3456585477 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a59f60 0x562352e468e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:06.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.075+0000 7f5d8276b640 1 -- 172.21.15.90:0/3456585477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562352a724d0 msgr2=0x562352e48e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:06.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.075+0000 7f5d8276b640 1 --2- 172.21.15.90:0/3456585477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562352a724d0 0x562352e48e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.075+0000 7f5d8276b640 1 -- 172.21.15.90:0/3456585477 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a59f60 msgr2=0x562352e468e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:06.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.075+0000 7f5d8276b640 1 --2- 172.21.15.90:0/3456585477 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a59f60 0x562352e468e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.075+0000 7f5d8276b640 1 -- 172.21.15.90:0/3456585477 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562352e34780 con 0x562352a7a6b0 2024-04-25T11:42:06.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.075+0000 7f5d81f6a640 1 --2- 172.21.15.90:0/3456585477 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a59f60 0x562352e468e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:42:06.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.075+0000 7f5d8276b640 1 --2- 172.21.15.90:0/3456585477 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562352a7a6b0 0x562352e41430 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5d7402fc60 tx=0x7f5d74005d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93ce431fc5fb2a68 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:06.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.075+0000 7f5d677fe640 1 -- 172.21.15.90:0/3456585477 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d74004b40 con 0x562352a7a6b0 2024-04-25T11:42:06.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.075+0000 7f5d677fe640 1 -- 172.21.15.90:0/3456585477 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d74036050 con 0x562352a7a6b0 2024-04-25T11:42:06.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.076+0000 7f5d677fe640 1 -- 172.21.15.90:0/3456585477 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d740043e0 con 0x562352a7a6b0 2024-04-25T11:42:06.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.076+0000 7f5d8b07e740 1 -- 172.21.15.90:0/3456585477 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562352a7a6b0 msgr2=0x562352e41430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:06.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.076+0000 7f5d8b07e740 1 --2- 172.21.15.90:0/3456585477 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562352a7a6b0 0x562352e41430 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5d7402fc60 tx=0x7f5d74005d10 comp rx=0 tx=0).stop 2024-04-25T11:42:06.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.076+0000 7f5d8b07e740 1 -- 172.21.15.90:0/3456585477 shutdown_connections 2024-04-25T11:42:06.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.076+0000 7f5d8b07e740 1 --2- 172.21.15.90:0/3456585477 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562352a7a6b0 0x562352e41430 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.076+0000 7f5d8b07e740 1 --2- 172.21.15.90:0/3456585477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562352a724d0 0x562352e48e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.076+0000 7f5d8b07e740 1 --2- 172.21.15.90:0/3456585477 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a59f60 0x562352e468e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.076+0000 7f5d8b07e740 1 -- 172.21.15.90:0/3456585477 >> 172.21.15.90:0/3456585477 conn(0x562352a8f4e0 msgr2=0x562352e48720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:06.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.076+0000 7f5d8b07e740 1 -- 172.21.15.90:0/3456585477 shutdown_connections 2024-04-25T11:42:06.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.076+0000 7f5d8b07e740 1 -- 172.21.15.90:0/3456585477 wait complete. 2024-04-25T11:42:06.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.077+0000 7f5d8b07e740 1 -- start start 2024-04-25T11:42:06.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.077+0000 7f5d8b07e740 1 Processor -- start 2024-04-25T11:42:06.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.077+0000 7f5d8b07e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a59f60 0x562352eb9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:06.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.077+0000 7f5d8b07e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562352a724d0 0x562352eba2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:06.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.077+0000 7f5d8b07e740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562352a7a6b0 0x562352e4a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:06.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.077+0000 7f5d8b07e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7c1aebe0 con 0x562352a59f60 2024-04-25T11:42:06.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.077+0000 7f5d8b07e740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7c1ae510 con 0x562352a7a6b0 2024-04-25T11:42:06.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.077+0000 7f5d8b07e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7c1ab750 con 0x562352a724d0 2024-04-25T11:42:06.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.078+0000 7f5d8b07e740 10 client.0 _ll_register_callbacks cb 0x7f5d74009c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:06.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.078+0000 7f5d81f6a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a59f60 0x562352eb9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:06.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.078+0000 7f5d8276b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562352a7a6b0 0x562352e4a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:06.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.078+0000 7f5d81f6a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a59f60 0x562352eb9d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58424/0 (socket says 172.21.15.90:58424) 2024-04-25T11:42:06.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.078+0000 7f5d81f6a640 1 -- 172.21.15.90:0/4071121687 learned_addr learned my addr 172.21.15.90:0/4071121687 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:06.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.078+0000 7f5d8276b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562352a7a6b0 0x562352e4a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:34836/0 (socket says 172.21.15.90:34836) 2024-04-25T11:42:06.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.078+0000 7f5d81769640 1 --2- 172.21.15.90:0/4071121687 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562352a724d0 0x562352eba2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:06.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.078+0000 7f5d81769640 1 -- 172.21.15.90:0/4071121687 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562352a7a6b0 msgr2=0x562352e4a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:06.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.078+0000 7f5d81769640 1 --2- 172.21.15.90:0/4071121687 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562352a7a6b0 0x562352e4a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.078+0000 7f5d81769640 1 -- 172.21.15.90:0/4071121687 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a59f60 msgr2=0x562352eb9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:06.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.078+0000 7f5d81769640 1 --2- 172.21.15.90:0/4071121687 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a59f60 0x562352eb9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.078+0000 7f5d81769640 1 -- 172.21.15.90:0/4071121687 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562352e4ada0 con 0x562352a724d0 2024-04-25T11:42:06.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.078+0000 7f5d8276b640 1 --2- 172.21.15.90:0/4071121687 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562352a7a6b0 0x562352e4a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:42:06.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.078+0000 7f5d81f6a640 1 --2- 172.21.15.90:0/4071121687 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a59f60 0x562352eb9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:42:06.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.078+0000 7f5d81769640 1 --2- 172.21.15.90:0/4071121687 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562352a724d0 0x562352eba2d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c0ddf20 tx=0x7f5d6c009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbc5b216f14b140a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:06.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.078+0000 7f5d657fa640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:42:06.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.078+0000 7f5d657fa640 1 -- 172.21.15.90:0/4071121687 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c0a87f0 con 0x562352a724d0 2024-04-25T11:42:06.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.078+0000 7f5d657fa640 1 -- 172.21.15.90:0/4071121687 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f5d6c00bb40 con 0x562352a724d0 2024-04-25T11:42:06.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.078+0000 7f5d8b07e740 1 -- 172.21.15.90:0/4071121687 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d74036050 con 0x562352a724d0 2024-04-25T11:42:06.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.079+0000 7f5d657fa640 1 -- 172.21.15.90:0/4071121687 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5d6c0a8190 con 0x562352a724d0 2024-04-25T11:42:06.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.079+0000 7f5d8b07e740 10 client.26596 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:06.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.079+0000 7f5d8b07e740 1 -- 172.21.15.90:0/4071121687 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562352ec13d0 con 0x562352a724d0 2024-04-25T11:42:06.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.079+0000 7f5d657fa640 1 -- 172.21.15.90:0/4071121687 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d74036050 con 0x562352a724d0 2024-04-25T11:42:06.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.079+0000 7f5d657fa640 1 -- 172.21.15.90:0/4071121687 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f5d6c0e0ce0 con 0x562352a724d0 2024-04-25T11:42:06.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.079+0000 7f5d8b07e740 10 client.26596 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:06.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.079+0000 7f5d8b07e740 10 client.26596 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:42:06.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.079+0000 7f5d8b07e740 20 client.26596 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:06.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.079+0000 7f5d8b07e740 1 --2- 172.21.15.90:0/4071121687 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562352a9b390 0x562352ec3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:06.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.079+0000 7f5d8b07e740 4 client.26596 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:42:06.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.079+0000 7f5d8b07e740 1 -- 172.21.15.90:0/4071121687 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562352e3d0e0 con 0x562352a9b390 2024-04-25T11:42:06.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.079+0000 7f5d81f6a640 1 --2- 172.21.15.90:0/4071121687 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562352a9b390 0x562352ec3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:06.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.080+0000 7f5d657fa640 10 client.26596 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:06.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.080+0000 7f5d81f6a640 1 --2- 172.21.15.90:0/4071121687 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562352a9b390 0x562352ec3b40 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0c8299bc72a41be server_cookie=f7cf0ae69470ea74 in_seq=0 out_seq=0 2024-04-25T11:42:06.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.080+0000 7f5d657fa640 1 -- 172.21.15.90:0/4071121687 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562352e3d0e0 con 0x562352a9b390 2024-04-25T11:42:06.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.080+0000 7f5d657fa640 10 client.26596 handle_command_reply: tid=0 2024-04-25T11:42:06.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.080+0000 7f5d8b07e740 1 client.26596 shutdown 2024-04-25T11:42:06.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.080+0000 7f5d8b07e740 1 -- 172.21.15.90:0/4071121687 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562352a724d0 msgr2=0x562352eba2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:06.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.080+0000 7f5d8b07e740 1 --2- 172.21.15.90:0/4071121687 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562352a724d0 0x562352eba2d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c0ddf20 tx=0x7f5d6c009310 comp rx=0 tx=0).stop 2024-04-25T11:42:06.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.080+0000 7f5d8b07e740 1 -- 172.21.15.90:0/4071121687 shutdown_connections 2024-04-25T11:42:06.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.080+0000 7f5d8b07e740 1 --2- 172.21.15.90:0/4071121687 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562352a9b390 0x562352ec3b40 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.080+0000 7f5d8b07e740 1 --2- 172.21.15.90:0/4071121687 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562352a7a6b0 0x562352e4a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.080+0000 7f5d8b07e740 1 --2- 172.21.15.90:0/4071121687 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562352a724d0 0x562352eba2d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.080+0000 7f5d657fa640 0 client.26596 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:06.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.080+0000 7f5d8b07e740 1 --2- 172.21.15.90:0/4071121687 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562352a59f60 0x562352eb9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.080+0000 7f5d8b07e740 1 -- 172.21.15.90:0/4071121687 >> 172.21.15.90:0/4071121687 conn(0x562352a8f4e0 msgr2=0x562352e51e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:06.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.081+0000 7f5d8b07e740 1 -- 172.21.15.90:0/4071121687 shutdown_connections 2024-04-25T11:42:06.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.081+0000 7f5d8b07e740 1 -- 172.21.15.90:0/4071121687 wait complete. 2024-04-25T11:42:06.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.081+0000 7f5d8b07e740 20 client.26596 trim_cache size 0 max 16384 2024-04-25T11:42:06.074 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:42:06.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.081+0000 7f5d841ce640 1 -- 172.21.15.90:0/2738482831 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5d680afc60 msgr2=0x7f5d680b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:06.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.081+0000 7f5d841ce640 1 --2- 172.21.15.90:0/2738482831 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5d680afc60 0x7f5d680b2170 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5d7402f500 tx=0x7f5d7402f330 comp rx=0 tx=0).stop 2024-04-25T11:42:06.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.081+0000 7f5d841ce640 1 -- 172.21.15.90:0/2738482831 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d7c108a20 msgr2=0x7f5d7c199b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:06.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.081+0000 7f5d841ce640 1 --2- 172.21.15.90:0/2738482831 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d7c108a20 0x7f5d7c199b40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c06c4d0 tx=0x7f5d6c00ef60 comp rx=0 tx=0).stop 2024-04-25T11:42:06.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.081+0000 7f5d841ce640 1 -- 172.21.15.90:0/2738482831 shutdown_connections 2024-04-25T11:42:06.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.081+0000 7f5d841ce640 1 --2- 172.21.15.90:0/2738482831 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5d680afc60 0x7f5d680b2170 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.081+0000 7f5d841ce640 1 --2- 172.21.15.90:0/2738482831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d7c10d380 0x7f5d7c19de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.081+0000 7f5d841ce640 1 --2- 172.21.15.90:0/2738482831 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d7c108a20 0x7f5d7c199b40 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.081+0000 7f5d841ce640 1 --2- 172.21.15.90:0/2738482831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d7c102af0 0x7f5d7c1995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.081+0000 7f5d841ce640 1 -- 172.21.15.90:0/2738482831 >> 172.21.15.90:0/2738482831 conn(0x7f5d7c078100 msgr2=0x7f5d7c1040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:06.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.082+0000 7f5d841ce640 1 -- 172.21.15.90:0/2738482831 shutdown_connections 2024-04-25T11:42:06.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:06.082+0000 7f5d841ce640 1 -- 172.21.15.90:0/2738482831 wait complete. 2024-04-25T11:42:07.080 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T11:42:07.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:42:07.081 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T11:42:07.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.209+0000 7f3aa600d640 1 -- 172.21.15.90:0/2773701541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3aa0108a20 msgr2=0x7f3aa01025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:07.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.209+0000 7f3aa600d640 1 --2- 172.21.15.90:0/2773701541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3aa0108a20 0x7f3aa01025b0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f3a90009900 tx=0x7f3a9002a410 comp rx=0 tx=0).stop 2024-04-25T11:42:07.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.209+0000 7f3aa600d640 1 -- 172.21.15.90:0/2773701541 shutdown_connections 2024-04-25T11:42:07.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.209+0000 7f3aa600d640 1 --2- 172.21.15.90:0/2773701541 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3aa010d380 0x7f3aa010f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.209+0000 7f3aa600d640 1 --2- 172.21.15.90:0/2773701541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3aa0102af0 0x7f3aa0102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.209+0000 7f3aa600d640 1 --2- 172.21.15.90:0/2773701541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3aa0108a20 0x7f3aa01025b0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.209+0000 7f3aa600d640 1 -- 172.21.15.90:0/2773701541 >> 172.21.15.90:0/2773701541 conn(0x7f3aa0078100 msgr2=0x7f3aa0078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:07.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.209+0000 7f3aa600d640 1 -- 172.21.15.90:0/2773701541 shutdown_connections 2024-04-25T11:42:07.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.209+0000 7f3aa600d640 1 -- 172.21.15.90:0/2773701541 wait complete. 2024-04-25T11:42:07.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.210+0000 7f3aa600d640 1 Processor -- start 2024-04-25T11:42:07.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.210+0000 7f3aa600d640 1 -- start start 2024-04-25T11:42:07.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.210+0000 7f3aa600d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3aa0102af0 0x7f3aa019b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.210+0000 7f3aa600d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3aa0108a20 0x7f3aa019baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.210+0000 7f3aa600d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3aa010d380 0x7f3aa019bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.210+0000 7f3aa600d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa0112be0 con 0x7f3aa0102af0 2024-04-25T11:42:07.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.210+0000 7f3aa600d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa0112a60 con 0x7f3aa0108a20 2024-04-25T11:42:07.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.210+0000 7f3aa600d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa0112d60 con 0x7f3aa010d380 2024-04-25T11:42:07.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.210+0000 7f3a9f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3aa0102af0 0x7f3aa019b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.211+0000 7f3a9f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3aa0102af0 0x7f3aa019b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58440/0 (socket says 172.21.15.90:58440) 2024-04-25T11:42:07.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.211+0000 7f3a9f7fe640 1 -- 172.21.15.90:0/485514566 learned_addr learned my addr 172.21.15.90:0/485514566 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:07.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.211+0000 7f3a9effd640 1 --2- 172.21.15.90:0/485514566 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3aa0108a20 0x7f3aa019baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.211+0000 7f3a9ffff640 1 --2- 172.21.15.90:0/485514566 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3aa010d380 0x7f3aa019bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.211+0000 7f3a9f7fe640 1 -- 172.21.15.90:0/485514566 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3aa010d380 msgr2=0x7f3aa019bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:07.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.211+0000 7f3a9f7fe640 1 --2- 172.21.15.90:0/485514566 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3aa010d380 0x7f3aa019bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.211+0000 7f3a9f7fe640 1 -- 172.21.15.90:0/485514566 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3aa0108a20 msgr2=0x7f3aa019baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:07.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.211+0000 7f3a9f7fe640 1 --2- 172.21.15.90:0/485514566 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3aa0108a20 0x7f3aa019baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.211+0000 7f3a9f7fe640 1 -- 172.21.15.90:0/485514566 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aa01a6870 con 0x7f3aa0102af0 2024-04-25T11:42:07.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.211+0000 7f3a9f7fe640 1 --2- 172.21.15.90:0/485514566 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3aa0102af0 0x7f3aa019b560 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f3a90032fd0 tx=0x7f3a90004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:07.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.211+0000 7f3a9cff9640 1 -- 172.21.15.90:0/485514566 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9003f030 con 0x7f3aa0102af0 2024-04-25T11:42:07.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.211+0000 7f3aa600d640 1 -- 172.21.15.90:0/485514566 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3aa01a6b00 con 0x7f3aa0102af0 2024-04-25T11:42:07.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.211+0000 7f3a9cff9640 1 -- 172.21.15.90:0/485514566 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a90004490 con 0x7f3aa0102af0 2024-04-25T11:42:07.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.211+0000 7f3a9cff9640 1 -- 172.21.15.90:0/485514566 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a90035070 con 0x7f3aa0102af0 2024-04-25T11:42:07.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.211+0000 7f3aa600d640 1 -- 172.21.15.90:0/485514566 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3aa01a7040 con 0x7f3aa0102af0 2024-04-25T11:42:07.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.213+0000 7f3a9cff9640 1 -- 172.21.15.90:0/485514566 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f3a90042030 con 0x7f3aa0102af0 2024-04-25T11:42:07.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.214+0000 7f3aacf95740 1 Processor -- start 2024-04-25T11:42:07.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.214+0000 7f3aacf95740 1 -- start start 2024-04-25T11:42:07.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.214+0000 7f3aacf95740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f5b66e2410 0x55f5b6a82620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.214+0000 7f3aacf95740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66b9550 0x55f5b6a82b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.214+0000 7f3aacf95740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5b66c1730 0x55f5b6a830a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.214+0000 7f3aacf95740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f5b69e8b00 con 0x55f5b66b9550 2024-04-25T11:42:07.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.214+0000 7f3aacf95740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f5b6984920 con 0x55f5b66e2410 2024-04-25T11:42:07.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.214+0000 7f3aacf95740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f5b69e4ac0 con 0x55f5b66c1730 2024-04-25T11:42:07.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.214+0000 7f3a9f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66b9550 0x55f5b6a82b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.214+0000 7f3a9f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66b9550 0x55f5b6a82b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58452/0 (socket says 172.21.15.90:58452) 2024-04-25T11:42:07.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.214+0000 7f3a9f7fe640 1 -- 172.21.15.90:0/3689522143 learned_addr learned my addr 172.21.15.90:0/3689522143 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:07.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.214+0000 7f3a9cff9640 1 --2- 172.21.15.90:0/485514566 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3a880afc60 0x7f3a880b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.214+0000 7f3a9ffff640 1 --2- 172.21.15.90:0/3689522143 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f5b66e2410 0x55f5b6a82620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.214+0000 7f3a9effd640 1 --2- 172.21.15.90:0/3689522143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5b66c1730 0x55f5b6a830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.214+0000 7f3a9ffff640 1 --2- 172.21.15.90:0/485514566 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3a880afc60 0x7f3a880b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.214+0000 7f3a9cff9640 1 -- 172.21.15.90:0/485514566 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f3a900ff5a0 con 0x7f3aa0102af0 2024-04-25T11:42:07.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.214+0000 7f3a9f7fe640 1 -- 172.21.15.90:0/3689522143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5b66c1730 msgr2=0x55f5b6a830a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:07.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.214+0000 7f3a9f7fe640 1 --2- 172.21.15.90:0/3689522143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5b66c1730 0x55f5b6a830a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.214+0000 7f3a9f7fe640 1 -- 172.21.15.90:0/3689522143 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f5b66e2410 msgr2=0x55f5b6a82620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:07.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.214+0000 7f3a9f7fe640 1 --2- 172.21.15.90:0/3689522143 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f5b66e2410 0x55f5b6a82620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.214+0000 7f3a9f7fe640 1 -- 172.21.15.90:0/3689522143 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5b67c27e0 con 0x55f5b66b9550 2024-04-25T11:42:07.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.214+0000 7f3a9ffff640 1 --2- 172.21.15.90:0/3689522143 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f5b66e2410 0x55f5b6a82620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:42:07.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.215+0000 7f3a9f7fe640 1 --2- 172.21.15.90:0/3689522143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66b9550 0x55f5b6a82b60 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f3a900c73e0 tx=0x7f3a900c3e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b84e9d5d8146754 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:07.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.215+0000 7f3a85ffb640 1 -- 172.21.15.90:0/3689522143 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a90049020 con 0x55f5b66b9550 2024-04-25T11:42:07.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.215+0000 7f3a9ffff640 1 --2- 172.21.15.90:0/485514566 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3a880afc60 0x7f3a880b2170 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x55f5b6a7f790 tx=0x7f3a98004ab0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:07.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.215+0000 7f3a85ffb640 1 -- 172.21.15.90:0/3689522143 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a900c71f0 con 0x55f5b66b9550 2024-04-25T11:42:07.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.215+0000 7f3a85ffb640 1 -- 172.21.15.90:0/3689522143 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a900c76e0 con 0x55f5b66b9550 2024-04-25T11:42:07.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.216+0000 7f3aacf95740 1 -- 172.21.15.90:0/3689522143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66b9550 msgr2=0x55f5b6a82b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:07.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.216+0000 7f3aacf95740 1 --2- 172.21.15.90:0/3689522143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66b9550 0x55f5b6a82b60 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f3a900c73e0 tx=0x7f3a900c3e60 comp rx=0 tx=0).stop 2024-04-25T11:42:07.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.216+0000 7f3aacf95740 1 -- 172.21.15.90:0/3689522143 shutdown_connections 2024-04-25T11:42:07.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.216+0000 7f3aacf95740 1 --2- 172.21.15.90:0/3689522143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5b66c1730 0x55f5b6a830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.216+0000 7f3aacf95740 1 --2- 172.21.15.90:0/3689522143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66b9550 0x55f5b6a82b60 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.216+0000 7f3aacf95740 1 --2- 172.21.15.90:0/3689522143 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f5b66e2410 0x55f5b6a82620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.216+0000 7f3aacf95740 1 -- 172.21.15.90:0/3689522143 >> 172.21.15.90:0/3689522143 conn(0x55f5b66d6560 msgr2=0x55f5b6a7edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:07.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.216+0000 7f3aacf95740 1 -- 172.21.15.90:0/3689522143 shutdown_connections 2024-04-25T11:42:07.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.216+0000 7f3aacf95740 1 -- 172.21.15.90:0/3689522143 wait complete. 2024-04-25T11:42:07.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.216+0000 7f3aacf95740 1 -- start start 2024-04-25T11:42:07.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.216+0000 7f3aacf95740 1 Processor -- start 2024-04-25T11:42:07.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.217+0000 7f3aacf95740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5b66b9550 0x55f5b6a8cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.217+0000 7f3aacf95740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66c1730 0x55f5b6a8d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.217+0000 7f3aacf95740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f5b66e2410 0x55f5b6a98c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.217+0000 7f3aacf95740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f5b6a8f730 con 0x55f5b66c1730 2024-04-25T11:42:07.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.217+0000 7f3aacf95740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f5b69825d0 con 0x55f5b66e2410 2024-04-25T11:42:07.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.217+0000 7f3aacf95740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f5b6a8f8b0 con 0x55f5b66b9550 2024-04-25T11:42:07.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.217+0000 7f3aacf95740 10 client.0 _ll_register_callbacks cb 0x55f5b6970080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:07.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.217+0000 7f3a9effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66c1730 0x55f5b6a8d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.217+0000 7f3a9effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66c1730 0x55f5b6a8d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58458/0 (socket says 172.21.15.90:58458) 2024-04-25T11:42:07.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.217+0000 7f3a9effd640 1 -- 172.21.15.90:0/731021675 learned_addr learned my addr 172.21.15.90:0/731021675 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:07.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.217+0000 7f3a9ffff640 1 --2- 172.21.15.90:0/731021675 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f5b66e2410 0x55f5b6a98c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.217+0000 7f3a9f7fe640 1 --2- 172.21.15.90:0/731021675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5b66b9550 0x55f5b6a8cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.217+0000 7f3a9effd640 1 -- 172.21.15.90:0/731021675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5b66b9550 msgr2=0x55f5b6a8cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:07.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.217+0000 7f3a9effd640 1 --2- 172.21.15.90:0/731021675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5b66b9550 0x55f5b6a8cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.217+0000 7f3a9effd640 1 -- 172.21.15.90:0/731021675 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f5b66e2410 msgr2=0x55f5b6a98c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:07.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.217+0000 7f3a9effd640 1 --2- 172.21.15.90:0/731021675 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f5b66e2410 0x55f5b6a98c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.217+0000 7f3a9effd640 1 -- 172.21.15.90:0/731021675 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5b6822f70 con 0x55f5b66c1730 2024-04-25T11:42:07.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.217+0000 7f3a9effd640 1 --2- 172.21.15.90:0/731021675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66c1730 0x55f5b6a8d0b0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f3a9400be20 tx=0x7f3a94012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dafdbc3ca4af11c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:07.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.217+0000 7f3a7effd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:07.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.217+0000 7f3a7effd640 1 -- 172.21.15.90:0/731021675 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a94005220 con 0x55f5b66c1730 2024-04-25T11:42:07.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.217+0000 7f3a7effd640 1 -- 172.21.15.90:0/731021675 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f3a940053c0 con 0x55f5b66c1730 2024-04-25T11:42:07.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.217+0000 7f3aacf95740 1 -- 172.21.15.90:0/731021675 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a900c71f0 con 0x55f5b66c1730 2024-04-25T11:42:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.218+0000 7f3a7effd640 1 -- 172.21.15.90:0/731021675 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3a940055e0 con 0x55f5b66c1730 2024-04-25T11:42:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.218+0000 7f3aacf95740 10 client.17754 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.218+0000 7f3aacf95740 1 -- 172.21.15.90:0/731021675 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f5b66ba170 con 0x55f5b66c1730 2024-04-25T11:42:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.218+0000 7f3a7effd640 1 -- 172.21.15.90:0/731021675 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a900c71f0 con 0x55f5b66c1730 2024-04-25T11:42:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.218+0000 7f3a7effd640 1 -- 172.21.15.90:0/731021675 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f3a94010b30 con 0x55f5b66c1730 2024-04-25T11:42:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.218+0000 7f3aacf95740 10 client.17754 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.218+0000 7f3aacf95740 10 client.17754 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:42:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.218+0000 7f3aacf95740 20 client.17754 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.218+0000 7f3aacf95740 1 --2- 172.21.15.90:0/731021675 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f5b66a11f0 0x55f5b6a9edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.218+0000 7f3aacf95740 4 client.17754 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:42:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.218+0000 7f3aacf95740 1 -- 172.21.15.90:0/731021675 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f5b69f6710 con 0x55f5b66a11f0 2024-04-25T11:42:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.218+0000 7f3a9f7fe640 1 --2- 172.21.15.90:0/731021675 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f5b66a11f0 0x55f5b6a9edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.219+0000 7f3a7effd640 10 client.17754 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:07.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.219+0000 7f3a9f7fe640 1 --2- 172.21.15.90:0/731021675 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f5b66a11f0 0x55f5b6a9edf0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a28409a10b1adf6e server_cookie=f904963dd438f97d in_seq=0 out_seq=0 2024-04-25T11:42:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.219+0000 7f3a7effd640 1 -- 172.21.15.90:0/731021675 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f5b69f6710 con 0x55f5b66a11f0 2024-04-25T11:42:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.219+0000 7f3a7effd640 10 client.17754 handle_command_reply: tid=0 2024-04-25T11:42:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.219+0000 7f3aacf95740 1 client.17754 shutdown 2024-04-25T11:42:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.220+0000 7f3aacf95740 1 -- 172.21.15.90:0/731021675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66c1730 msgr2=0x55f5b6a8d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.220+0000 7f3aacf95740 1 --2- 172.21.15.90:0/731021675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66c1730 0x55f5b6a8d0b0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f3a9400be20 tx=0x7f3a94012820 comp rx=0 tx=0).stop 2024-04-25T11:42:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.220+0000 7f3aacf95740 1 -- 172.21.15.90:0/731021675 shutdown_connections 2024-04-25T11:42:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.220+0000 7f3aacf95740 1 --2- 172.21.15.90:0/731021675 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f5b66a11f0 0x55f5b6a9edf0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.220+0000 7f3aacf95740 1 --2- 172.21.15.90:0/731021675 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f5b66e2410 0x55f5b6a98c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.220+0000 7f3aacf95740 1 --2- 172.21.15.90:0/731021675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66c1730 0x55f5b6a8d0b0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.220+0000 7f3aacf95740 1 --2- 172.21.15.90:0/731021675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5b66b9550 0x55f5b6a8cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.220+0000 7f3a7effd640 0 client.17754 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.220+0000 7f3aacf95740 1 -- 172.21.15.90:0/731021675 >> 172.21.15.90:0/731021675 conn(0x55f5b66d6560 msgr2=0x55f5b6a825c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.220+0000 7f3aacf95740 1 -- 172.21.15.90:0/731021675 shutdown_connections 2024-04-25T11:42:07.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.220+0000 7f3aacf95740 1 -- 172.21.15.90:0/731021675 wait complete. 2024-04-25T11:42:07.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.220+0000 7f3aacf95740 20 client.17754 trim_cache size 0 max 16384 2024-04-25T11:42:07.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.238+0000 7f3aacf95740 1 Processor -- start 2024-04-25T11:42:07.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.238+0000 7f3aacf95740 1 -- start start 2024-04-25T11:42:07.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.238+0000 7f3aacf95740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66b9550 0x55f5b6a8bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.238+0000 7f3aacf95740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5b66c1730 0x55f5b6a7ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.238+0000 7f3aacf95740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f5b66e2410 0x55f5b6a7dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.238+0000 7f3aacf95740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa01afac0 con 0x55f5b66b9550 2024-04-25T11:42:07.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.238+0000 7f3aacf95740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa01b1600 con 0x55f5b66e2410 2024-04-25T11:42:07.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.238+0000 7f3aacf95740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa01b0190 con 0x55f5b66c1730 2024-04-25T11:42:07.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.238+0000 7f3a9f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66b9550 0x55f5b6a8bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.238+0000 7f3a9f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66b9550 0x55f5b6a8bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58460/0 (socket says 172.21.15.90:58460) 2024-04-25T11:42:07.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.238+0000 7f3a9f7fe640 1 -- 172.21.15.90:0/1533034134 learned_addr learned my addr 172.21.15.90:0/1533034134 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:07.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.238+0000 7f3a9ffff640 1 --2- 172.21.15.90:0/1533034134 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f5b66e2410 0x55f5b6a7dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.238+0000 7f3a9effd640 1 --2- 172.21.15.90:0/1533034134 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5b66c1730 0x55f5b6a7ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.238+0000 7f3a9f7fe640 1 -- 172.21.15.90:0/1533034134 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5b66c1730 msgr2=0x55f5b6a7ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:07.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.238+0000 7f3a9f7fe640 1 --2- 172.21.15.90:0/1533034134 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5b66c1730 0x55f5b6a7ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.238+0000 7f3a9f7fe640 1 -- 172.21.15.90:0/1533034134 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f5b66e2410 msgr2=0x55f5b6a7dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:07.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.238+0000 7f3a9f7fe640 1 --2- 172.21.15.90:0/1533034134 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f5b66e2410 0x55f5b6a7dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.238+0000 7f3a9f7fe640 1 -- 172.21.15.90:0/1533034134 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5b6a833c0 con 0x55f5b66b9550 2024-04-25T11:42:07.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.238+0000 7f3a9f7fe640 1 --2- 172.21.15.90:0/1533034134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66b9550 0x55f5b6a8bc60 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f3a900c4950 tx=0x7f3a90000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0d1a0f521f94c81 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:07.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.238+0000 7f3a857fa640 1 -- 172.21.15.90:0/1533034134 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a90049040 con 0x55f5b66b9550 2024-04-25T11:42:07.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.239+0000 7f3a857fa640 1 -- 172.21.15.90:0/1533034134 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a900c71f0 con 0x55f5b66b9550 2024-04-25T11:42:07.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.239+0000 7f3a857fa640 1 -- 172.21.15.90:0/1533034134 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a900c76e0 con 0x55f5b66b9550 2024-04-25T11:42:07.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.239+0000 7f3aacf95740 1 -- 172.21.15.90:0/1533034134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66b9550 msgr2=0x55f5b6a8bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:07.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.239+0000 7f3aacf95740 1 --2- 172.21.15.90:0/1533034134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66b9550 0x55f5b6a8bc60 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f3a900c4950 tx=0x7f3a90000bc0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.239+0000 7f3aacf95740 1 -- 172.21.15.90:0/1533034134 shutdown_connections 2024-04-25T11:42:07.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.239+0000 7f3aacf95740 1 --2- 172.21.15.90:0/1533034134 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f5b66e2410 0x55f5b6a7dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.239+0000 7f3aacf95740 1 --2- 172.21.15.90:0/1533034134 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5b66c1730 0x55f5b6a7ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.239+0000 7f3aacf95740 1 --2- 172.21.15.90:0/1533034134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66b9550 0x55f5b6a8bc60 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.239+0000 7f3aacf95740 1 -- 172.21.15.90:0/1533034134 >> 172.21.15.90:0/1533034134 conn(0x55f5b66d6560 msgr2=0x55f5b6a7d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:07.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.239+0000 7f3aacf95740 1 -- 172.21.15.90:0/1533034134 shutdown_connections 2024-04-25T11:42:07.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.239+0000 7f3aacf95740 1 -- 172.21.15.90:0/1533034134 wait complete. 2024-04-25T11:42:07.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.240+0000 7f3aacf95740 1 -- start start 2024-04-25T11:42:07.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.240+0000 7f3aacf95740 1 Processor -- start 2024-04-25T11:42:07.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.240+0000 7f3aacf95740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5b66b9550 0x55f5b6a94d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.240+0000 7f3aacf95740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f5b66c1730 0x55f5b6a90110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.240+0000 7f3aacf95740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66e2410 0x55f5b6a90650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.240+0000 7f3aacf95740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa01b0f30 con 0x55f5b66e2410 2024-04-25T11:42:07.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.240+0000 7f3aacf95740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa01b0860 con 0x55f5b66c1730 2024-04-25T11:42:07.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.240+0000 7f3aacf95740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa01adaa0 con 0x55f5b66b9550 2024-04-25T11:42:07.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.240+0000 7f3aacf95740 10 client.0 _ll_register_callbacks cb 0x7f3a9400ba30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:07.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.240+0000 7f3a9ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66e2410 0x55f5b6a90650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.240+0000 7f3a9ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66e2410 0x55f5b6a90650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58472/0 (socket says 172.21.15.90:58472) 2024-04-25T11:42:07.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.240+0000 7f3a9ffff640 1 -- 172.21.15.90:0/1363291568 learned_addr learned my addr 172.21.15.90:0/1363291568 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:07.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.240+0000 7f3a9f7fe640 1 --2- 172.21.15.90:0/1363291568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5b66b9550 0x55f5b6a94d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.240+0000 7f3a9effd640 1 --2- 172.21.15.90:0/1363291568 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f5b66c1730 0x55f5b6a90110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.241+0000 7f3a9ffff640 1 -- 172.21.15.90:0/1363291568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5b66b9550 msgr2=0x55f5b6a94d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:07.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.241+0000 7f3a9ffff640 1 --2- 172.21.15.90:0/1363291568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5b66b9550 0x55f5b6a94d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.241+0000 7f3a9ffff640 1 -- 172.21.15.90:0/1363291568 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f5b66c1730 msgr2=0x55f5b6a90110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:07.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.241+0000 7f3a9ffff640 1 --2- 172.21.15.90:0/1363291568 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f5b66c1730 0x55f5b6a90110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.241+0000 7f3a9ffff640 1 -- 172.21.15.90:0/1363291568 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5b6a90b90 con 0x55f5b66e2410 2024-04-25T11:42:07.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.241+0000 7f3a9ffff640 1 --2- 172.21.15.90:0/1363291568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66e2410 0x55f5b6a90650 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f3a98003dc0 tx=0x7f3a98004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0fb7ba258f9adcc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:07.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.241+0000 7f3a7effd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:07.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.241+0000 7f3a7effd640 1 -- 172.21.15.90:0/1363291568 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a98019040 con 0x55f5b66e2410 2024-04-25T11:42:07.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.241+0000 7f3a7effd640 1 -- 172.21.15.90:0/1363291568 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f3a980045e0 con 0x55f5b66e2410 2024-04-25T11:42:07.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.241+0000 7f3aacf95740 1 -- 172.21.15.90:0/1363291568 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a900c71f0 con 0x55f5b66e2410 2024-04-25T11:42:07.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.241+0000 7f3a7effd640 1 -- 172.21.15.90:0/1363291568 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3a980047e0 con 0x55f5b66e2410 2024-04-25T11:42:07.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.241+0000 7f3aacf95740 10 client.17760 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:07.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.241+0000 7f3aacf95740 1 -- 172.21.15.90:0/1363291568 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f5b6b06d60 con 0x55f5b66e2410 2024-04-25T11:42:07.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.241+0000 7f3a7effd640 1 -- 172.21.15.90:0/1363291568 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a900c71f0 con 0x55f5b66e2410 2024-04-25T11:42:07.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.241+0000 7f3a7effd640 1 -- 172.21.15.90:0/1363291568 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f3a9802c4f0 con 0x55f5b66e2410 2024-04-25T11:42:07.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.241+0000 7f3aacf95740 10 client.17760 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:07.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.241+0000 7f3aacf95740 10 client.17760 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:42:07.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.241+0000 7f3aacf95740 20 client.17760 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:07.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.241+0000 7f3aacf95740 1 --2- 172.21.15.90:0/1363291568 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f5b6676630 0x55f5b6b09550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.241+0000 7f3aacf95740 4 client.17760 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T11:42:07.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.241+0000 7f3aacf95740 1 -- 172.21.15.90:0/1363291568 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f5b69f0580 con 0x55f5b6676630 2024-04-25T11:42:07.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.242+0000 7f3a9f7fe640 1 --2- 172.21.15.90:0/1363291568 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f5b6676630 0x55f5b6b09550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.242+0000 7f3a7effd640 10 client.17760 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:07.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.242+0000 7f3a9f7fe640 1 --2- 172.21.15.90:0/1363291568 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f5b6676630 0x55f5b6b09550 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=326fb1443c000256 server_cookie=e041feb7986f3fb6 in_seq=0 out_seq=0 2024-04-25T11:42:07.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.243+0000 7f3a7effd640 1 -- 172.21.15.90:0/1363291568 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f5b69f0580 con 0x55f5b6676630 2024-04-25T11:42:07.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.243+0000 7f3a7effd640 10 client.17760 handle_command_reply: tid=0 2024-04-25T11:42:07.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.244+0000 7f3aacf95740 1 client.17760 shutdown 2024-04-25T11:42:07.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.244+0000 7f3aacf95740 1 -- 172.21.15.90:0/1363291568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66e2410 msgr2=0x55f5b6a90650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:07.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.244+0000 7f3aacf95740 1 --2- 172.21.15.90:0/1363291568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66e2410 0x55f5b6a90650 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f3a98003dc0 tx=0x7f3a98004a40 comp rx=0 tx=0).stop 2024-04-25T11:42:07.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.244+0000 7f3aacf95740 1 -- 172.21.15.90:0/1363291568 shutdown_connections 2024-04-25T11:42:07.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.244+0000 7f3aacf95740 1 --2- 172.21.15.90:0/1363291568 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f5b6676630 0x55f5b6b09550 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.244+0000 7f3aacf95740 1 --2- 172.21.15.90:0/1363291568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f5b66e2410 0x55f5b6a90650 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.244+0000 7f3aacf95740 1 --2- 172.21.15.90:0/1363291568 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f5b66c1730 0x55f5b6a90110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.244+0000 7f3aacf95740 1 --2- 172.21.15.90:0/1363291568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f5b66b9550 0x55f5b6a94d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.244+0000 7f3a7effd640 0 client.17760 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:07.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.244+0000 7f3aacf95740 1 -- 172.21.15.90:0/1363291568 >> 172.21.15.90:0/1363291568 conn(0x55f5b66d6560 msgr2=0x55f5b6a8b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:07.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.244+0000 7f3aacf95740 1 -- 172.21.15.90:0/1363291568 shutdown_connections 2024-04-25T11:42:07.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.244+0000 7f3aacf95740 1 -- 172.21.15.90:0/1363291568 wait complete. 2024-04-25T11:42:07.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.244+0000 7f3aacf95740 20 client.17760 trim_cache size 0 max 16384 2024-04-25T11:42:07.237 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:42:07.237 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-25T11:42:07.237 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "674ae53a-aa58-48fc-98d5-b5ed725812dd", 2024-04-25T11:42:07.237 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-25T11:42:07.237 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:42:07.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.244+0000 7f3aa600d640 1 -- 172.21.15.90:0/485514566 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3a880afc60 msgr2=0x7f3a880b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:07.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.244+0000 7f3aa600d640 1 --2- 172.21.15.90:0/485514566 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3a880afc60 0x7f3a880b2170 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x55f5b6a7f790 tx=0x7f3a98004ab0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.244+0000 7f3aa600d640 1 -- 172.21.15.90:0/485514566 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3aa0102af0 msgr2=0x7f3aa019b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:07.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.244+0000 7f3aa600d640 1 --2- 172.21.15.90:0/485514566 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3aa0102af0 0x7f3aa019b560 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f3a90032fd0 tx=0x7f3a90004270 comp rx=0 tx=0).stop 2024-04-25T11:42:07.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.244+0000 7f3aa600d640 1 -- 172.21.15.90:0/485514566 shutdown_connections 2024-04-25T11:42:07.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.244+0000 7f3aa600d640 1 --2- 172.21.15.90:0/485514566 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3a880afc60 0x7f3a880b2170 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.244+0000 7f3aa600d640 1 --2- 172.21.15.90:0/485514566 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3aa010d380 0x7f3aa019bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.245+0000 7f3aa600d640 1 --2- 172.21.15.90:0/485514566 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3aa0108a20 0x7f3aa019baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.245+0000 7f3aa600d640 1 --2- 172.21.15.90:0/485514566 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3aa0102af0 0x7f3aa019b560 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.245+0000 7f3aa600d640 1 -- 172.21.15.90:0/485514566 >> 172.21.15.90:0/485514566 conn(0x7f3aa0078100 msgr2=0x7f3aa01040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:07.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.245+0000 7f3aa600d640 1 -- 172.21.15.90:0/485514566 shutdown_connections 2024-04-25T11:42:07.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.245+0000 7f3aa600d640 1 -- 172.21.15.90:0/485514566 wait complete. 2024-04-25T11:42:07.245 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:42:07.245 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:42:07.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.374+0000 7f97e18d9640 1 -- 172.21.15.90:0/2402621995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97dc108eb0 msgr2=0x7f97dc102b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:07.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.374+0000 7f97e18d9640 1 --2- 172.21.15.90:0/2402621995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97dc108eb0 0x7f97dc102b50 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f97c4009900 tx=0x7f97c402a410 comp rx=0 tx=0).stop 2024-04-25T11:42:07.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.374+0000 7f97e18d9640 1 -- 172.21.15.90:0/2402621995 shutdown_connections 2024-04-25T11:42:07.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.374+0000 7f97e18d9640 1 --2- 172.21.15.90:0/2402621995 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97dc1031c0 0x7f97dc10f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.374+0000 7f97e18d9640 1 --2- 172.21.15.90:0/2402621995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97dc108eb0 0x7f97dc102b50 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.374+0000 7f97e18d9640 1 --2- 172.21.15.90:0/2402621995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97dc108a10 0x7f97dc102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.374+0000 7f97e18d9640 1 -- 172.21.15.90:0/2402621995 >> 172.21.15.90:0/2402621995 conn(0x7f97dc078110 msgr2=0x7f97dc078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:07.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.374+0000 7f97e18d9640 1 -- 172.21.15.90:0/2402621995 shutdown_connections 2024-04-25T11:42:07.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.374+0000 7f97e18d9640 1 -- 172.21.15.90:0/2402621995 wait complete. 2024-04-25T11:42:07.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.374+0000 7f97e18d9640 1 Processor -- start 2024-04-25T11:42:07.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.374+0000 7f97e18d9640 1 -- start start 2024-04-25T11:42:07.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.375+0000 7f97e18d9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97dc1031c0 0x7f97dc199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.375+0000 7f97e18d9640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97dc108a10 0x7f97dc199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.375+0000 7f97e18d9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97dc108eb0 0x7f97dc19de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.375+0000 7f97e18d9640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f97dc112c30 con 0x7f97dc108eb0 2024-04-25T11:42:07.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.375+0000 7f97e18d9640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f97dc112ab0 con 0x7f97dc108a10 2024-04-25T11:42:07.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.375+0000 7f97e18d9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f97dc112db0 con 0x7f97dc1031c0 2024-04-25T11:42:07.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.375+0000 7f97db7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97dc108eb0 0x7f97dc19de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.375+0000 7f97db7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97dc108eb0 0x7f97dc19de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58480/0 (socket says 172.21.15.90:58480) 2024-04-25T11:42:07.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.375+0000 7f97db7fe640 1 -- 172.21.15.90:0/142537091 learned_addr learned my addr 172.21.15.90:0/142537091 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:07.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.375+0000 7f97daffd640 1 --2- 172.21.15.90:0/142537091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97dc1031c0 0x7f97dc199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.375+0000 7f97d3fff640 1 --2- 172.21.15.90:0/142537091 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97dc108a10 0x7f97dc199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.375+0000 7f97db7fe640 1 -- 172.21.15.90:0/142537091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97dc1031c0 msgr2=0x7f97dc199650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:07.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.375+0000 7f97db7fe640 1 --2- 172.21.15.90:0/142537091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97dc1031c0 0x7f97dc199650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.375+0000 7f97db7fe640 1 -- 172.21.15.90:0/142537091 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97dc108a10 msgr2=0x7f97dc199bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:07.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.375+0000 7f97db7fe640 1 --2- 172.21.15.90:0/142537091 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97dc108a10 0x7f97dc199bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.375+0000 7f97db7fe640 1 -- 172.21.15.90:0/142537091 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97dc19e4b0 con 0x7f97dc108eb0 2024-04-25T11:42:07.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.375+0000 7f97d3fff640 1 --2- 172.21.15.90:0/142537091 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97dc108a10 0x7f97dc199bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:42:07.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.375+0000 7f97db7fe640 1 --2- 172.21.15.90:0/142537091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97dc108eb0 0x7f97dc19de70 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f97d400eb10 tx=0x7f97d400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:07.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.375+0000 7f97e08d7640 1 -- 172.21.15.90:0/142537091 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d4004b20 con 0x7f97dc108eb0 2024-04-25T11:42:07.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.375+0000 7f97e08d7640 1 -- 172.21.15.90:0/142537091 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97d4004cc0 con 0x7f97dc108eb0 2024-04-25T11:42:07.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.375+0000 7f97e18d9640 1 -- 172.21.15.90:0/142537091 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97dc19e7a0 con 0x7f97dc108eb0 2024-04-25T11:42:07.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.375+0000 7f97e08d7640 1 -- 172.21.15.90:0/142537091 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d4010670 con 0x7f97dc108eb0 2024-04-25T11:42:07.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.375+0000 7f97e18d9640 1 -- 172.21.15.90:0/142537091 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97dc19ec70 con 0x7f97dc108eb0 2024-04-25T11:42:07.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.378+0000 7f97e8782740 1 Processor -- start 2024-04-25T11:42:07.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.378+0000 7f97e08d7640 1 -- 172.21.15.90:0/142537091 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f97d4005020 con 0x7f97dc108eb0 2024-04-25T11:42:07.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.378+0000 7f97e8782740 1 -- start start 2024-04-25T11:42:07.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.378+0000 7f97e8782740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf943f390 0x559cf97dfa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.378+0000 7f97e8782740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559cf94164d0 0x559cf97dffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.378+0000 7f97e8782740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559cf941e6b0 0x559cf97ec8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.378+0000 7f97e8782740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559cf9745d80 con 0x559cf943f390 2024-04-25T11:42:07.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.378+0000 7f97e8782740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x559cf96ff8b0 con 0x559cf941e6b0 2024-04-25T11:42:07.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.378+0000 7f97e8782740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559cf9741d40 con 0x559cf94164d0 2024-04-25T11:42:07.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.378+0000 7f97d3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf943f390 0x559cf97dfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.378+0000 7f97d3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf943f390 0x559cf97dfa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58494/0 (socket says 172.21.15.90:58494) 2024-04-25T11:42:07.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.378+0000 7f97d3fff640 1 -- 172.21.15.90:0/2269212667 learned_addr learned my addr 172.21.15.90:0/2269212667 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:07.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.378+0000 7f97d3fff640 1 --2- 172.21.15.90:0/2269212667 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559cf941e6b0 0x559cf97ec8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.378+0000 7f97daffd640 1 --2- 172.21.15.90:0/2269212667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559cf94164d0 0x559cf97dffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.379+0000 7f97e08d7640 1 --2- 172.21.15.90:0/142537091 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f97c80afc60 0x7f97c80b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.379+0000 7f97d3fff640 1 -- 172.21.15.90:0/2269212667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559cf94164d0 msgr2=0x559cf97dffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:07.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.379+0000 7f97d3fff640 1 --2- 172.21.15.90:0/2269212667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559cf94164d0 0x559cf97dffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.379+0000 7f97d3fff640 1 -- 172.21.15.90:0/2269212667 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559cf941e6b0 msgr2=0x559cf97ec8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:07.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.379+0000 7f97d3fff640 1 --2- 172.21.15.90:0/2269212667 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559cf941e6b0 0x559cf97ec8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.379+0000 7f97d3fff640 1 -- 172.21.15.90:0/2269212667 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cf96d0a40 con 0x559cf943f390 2024-04-25T11:42:07.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.379+0000 7f97db7fe640 1 --2- 172.21.15.90:0/142537091 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f97c80afc60 0x7f97c80b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.379+0000 7f97d3fff640 1 --2- 172.21.15.90:0/2269212667 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf943f390 0x559cf97dfa60 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f97c4031440 tx=0x7f97c40058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfd919782ccd2a60 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:07.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.379+0000 7f97e08d7640 1 -- 172.21.15.90:0/142537091 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f97d40df6e0 con 0x7f97dc108eb0 2024-04-25T11:42:07.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.379+0000 7f97d1ffb640 1 -- 172.21.15.90:0/2269212667 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c4035050 con 0x559cf943f390 2024-04-25T11:42:07.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.379+0000 7f97d1ffb640 1 -- 172.21.15.90:0/2269212667 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97c40316d0 con 0x559cf943f390 2024-04-25T11:42:07.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.379+0000 7f97d1ffb640 1 -- 172.21.15.90:0/2269212667 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c40046c0 con 0x559cf943f390 2024-04-25T11:42:07.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.380+0000 7f97e8782740 1 -- 172.21.15.90:0/2269212667 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf943f390 msgr2=0x559cf97dfa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:07.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.380+0000 7f97db7fe640 1 --2- 172.21.15.90:0/142537091 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f97c80afc60 0x7f97c80b2170 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f97d40a4a90 tx=0x7f97d40ca040 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:07.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.380+0000 7f97e8782740 1 --2- 172.21.15.90:0/2269212667 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf943f390 0x559cf97dfa60 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f97c4031440 tx=0x7f97c40058b0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.380+0000 7f97e8782740 1 -- 172.21.15.90:0/2269212667 shutdown_connections 2024-04-25T11:42:07.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.380+0000 7f97e8782740 1 --2- 172.21.15.90:0/2269212667 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559cf941e6b0 0x559cf97ec8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.380+0000 7f97e8782740 1 --2- 172.21.15.90:0/2269212667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559cf94164d0 0x559cf97dffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.380+0000 7f97e8782740 1 --2- 172.21.15.90:0/2269212667 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf943f390 0x559cf97dfa60 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.380+0000 7f97e8782740 1 -- 172.21.15.90:0/2269212667 >> 172.21.15.90:0/2269212667 conn(0x559cf94334e0 msgr2=0x559cf97dc210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:07.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.380+0000 7f97e8782740 1 -- 172.21.15.90:0/2269212667 shutdown_connections 2024-04-25T11:42:07.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.380+0000 7f97e8782740 1 -- 172.21.15.90:0/2269212667 wait complete. 2024-04-25T11:42:07.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.380+0000 7f97e8782740 1 -- start start 2024-04-25T11:42:07.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.381+0000 7f97e8782740 1 Processor -- start 2024-04-25T11:42:07.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.381+0000 7f97e8782740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559cf94164d0 0x559cf97ea640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.381+0000 7f97e8782740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf941e6b0 0x559cf97eab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.381+0000 7f97e8782740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559cf943f390 0x559cf97f2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.381+0000 7f97e8782740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559cf97ef130 con 0x559cf941e6b0 2024-04-25T11:42:07.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.381+0000 7f97e8782740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x559cf96ccd10 con 0x559cf943f390 2024-04-25T11:42:07.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.381+0000 7f97e8782740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559cf97ef2b0 con 0x559cf94164d0 2024-04-25T11:42:07.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.381+0000 7f97daffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf941e6b0 0x559cf97eab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.381+0000 7f97e8782740 10 client.0 _ll_register_callbacks cb 0x559cf96f9210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:07.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.381+0000 7f97daffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf941e6b0 0x559cf97eab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58500/0 (socket says 172.21.15.90:58500) 2024-04-25T11:42:07.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.381+0000 7f97daffd640 1 -- 172.21.15.90:0/2337210378 learned_addr learned my addr 172.21.15.90:0/2337210378 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:07.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.381+0000 7f97d3fff640 1 --2- 172.21.15.90:0/2337210378 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559cf943f390 0x559cf97f2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.381+0000 7f97d3fff640 1 --2- 172.21.15.90:0/2337210378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559cf94164d0 0x559cf97ea640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.381+0000 7f97daffd640 1 -- 172.21.15.90:0/2337210378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559cf94164d0 msgr2=0x559cf97ea640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:07.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.381+0000 7f97daffd640 1 --2- 172.21.15.90:0/2337210378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559cf94164d0 0x559cf97ea640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.381+0000 7f97daffd640 1 -- 172.21.15.90:0/2337210378 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559cf943f390 msgr2=0x559cf97f2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:07.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.381+0000 7f97daffd640 1 --2- 172.21.15.90:0/2337210378 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559cf943f390 0x559cf97f2680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.381+0000 7f97daffd640 1 -- 172.21.15.90:0/2337210378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cf95217d0 con 0x559cf941e6b0 2024-04-25T11:42:07.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.381+0000 7f97daffd640 1 --2- 172.21.15.90:0/2337210378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf941e6b0 0x559cf97eab80 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f97cc0126d0 tx=0x7f97cc00af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=746567068909db04 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:07.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.381+0000 7f97baffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:07.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.382+0000 7f97baffd640 1 -- 172.21.15.90:0/2337210378 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97cc010e80 con 0x559cf941e6b0 2024-04-25T11:42:07.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.382+0000 7f97baffd640 1 -- 172.21.15.90:0/2337210378 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f97cc005020 con 0x559cf941e6b0 2024-04-25T11:42:07.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.382+0000 7f97e8782740 1 -- 172.21.15.90:0/2337210378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97c40316d0 con 0x559cf941e6b0 2024-04-25T11:42:07.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.382+0000 7f97baffd640 1 -- 172.21.15.90:0/2337210378 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f97cc005260 con 0x559cf941e6b0 2024-04-25T11:42:07.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.382+0000 7f97e8782740 10 client.17772 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:07.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.382+0000 7f97e8782740 1 -- 172.21.15.90:0/2337210378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559cf94573f0 con 0x559cf941e6b0 2024-04-25T11:42:07.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.382+0000 7f97baffd640 1 -- 172.21.15.90:0/2337210378 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97c40316d0 con 0x559cf941e6b0 2024-04-25T11:42:07.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.382+0000 7f97baffd640 1 -- 172.21.15.90:0/2337210378 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f97cc01a4c0 con 0x559cf941e6b0 2024-04-25T11:42:07.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.382+0000 7f97e8782740 10 client.17772 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:07.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.382+0000 7f97e8782740 10 client.17772 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:42:07.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.382+0000 7f97e8782740 20 client.17772 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:07.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.382+0000 7f97e8782740 1 --2- 172.21.15.90:0/2337210378 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559cf93fdf60 0x559cf97fe9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.382+0000 7f97e8782740 4 client.17772 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:42:07.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.382+0000 7f97e8782740 1 -- 172.21.15.90:0/2337210378 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559cf9753990 con 0x559cf93fdf60 2024-04-25T11:42:07.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.382+0000 7f97d3fff640 1 --2- 172.21.15.90:0/2337210378 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559cf93fdf60 0x559cf97fe9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.383+0000 7f97baffd640 10 client.17772 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:07.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.383+0000 7f97d3fff640 1 --2- 172.21.15.90:0/2337210378 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559cf93fdf60 0x559cf97fe9b0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33c0faa9b60e2b87 server_cookie=d1a6366ca7e8a866 in_seq=0 out_seq=0 2024-04-25T11:42:07.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.383+0000 7f97baffd640 1 -- 172.21.15.90:0/2337210378 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559cf9753990 con 0x559cf93fdf60 2024-04-25T11:42:07.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.383+0000 7f97baffd640 10 client.17772 handle_command_reply: tid=0 2024-04-25T11:42:07.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.383+0000 7f97e8782740 1 client.17772 shutdown 2024-04-25T11:42:07.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.384+0000 7f97e8782740 1 -- 172.21.15.90:0/2337210378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf941e6b0 msgr2=0x559cf97eab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:07.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.384+0000 7f97e8782740 1 --2- 172.21.15.90:0/2337210378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf941e6b0 0x559cf97eab80 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f97cc0126d0 tx=0x7f97cc00af90 comp rx=0 tx=0).stop 2024-04-25T11:42:07.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.384+0000 7f97e8782740 1 -- 172.21.15.90:0/2337210378 shutdown_connections 2024-04-25T11:42:07.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.384+0000 7f97e8782740 1 --2- 172.21.15.90:0/2337210378 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559cf93fdf60 0x559cf97fe9b0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.384+0000 7f97e8782740 1 --2- 172.21.15.90:0/2337210378 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559cf943f390 0x559cf97f2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.384+0000 7f97e8782740 1 --2- 172.21.15.90:0/2337210378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf941e6b0 0x559cf97eab80 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.384+0000 7f97baffd640 0 client.17772 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:07.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.384+0000 7f97e8782740 1 --2- 172.21.15.90:0/2337210378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559cf94164d0 0x559cf97ea640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.384+0000 7f97e8782740 1 -- 172.21.15.90:0/2337210378 >> 172.21.15.90:0/2337210378 conn(0x559cf94334e0 msgr2=0x559cf97e1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:07.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.384+0000 7f97e8782740 1 -- 172.21.15.90:0/2337210378 shutdown_connections 2024-04-25T11:42:07.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.384+0000 7f97e8782740 1 -- 172.21.15.90:0/2337210378 wait complete. 2024-04-25T11:42:07.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.384+0000 7f97e8782740 20 client.17772 trim_cache size 0 max 16384 2024-04-25T11:42:07.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.401+0000 7f97e8782740 1 Processor -- start 2024-04-25T11:42:07.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.401+0000 7f97e8782740 1 -- start start 2024-04-25T11:42:07.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.402+0000 7f97e8782740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559cf94164d0 0x559cf97e9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.402+0000 7f97e8782740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559cf941e6b0 0x559cf97f2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.402+0000 7f97e8782740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf943f390 0x559cf97f2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.402+0000 7f97e8782740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f97dc1ad8e0 con 0x559cf943f390 2024-04-25T11:42:07.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.402+0000 7f97e8782740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f97dc1af420 con 0x559cf941e6b0 2024-04-25T11:42:07.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.402+0000 7f97e8782740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f97dc1adfb0 con 0x559cf94164d0 2024-04-25T11:42:07.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.402+0000 7f97d3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf943f390 0x559cf97f2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.402+0000 7f97d3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf943f390 0x559cf97f2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58514/0 (socket says 172.21.15.90:58514) 2024-04-25T11:42:07.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.402+0000 7f97d3fff640 1 -- 172.21.15.90:0/4176651511 learned_addr learned my addr 172.21.15.90:0/4176651511 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:07.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.402+0000 7f97daffd640 1 --2- 172.21.15.90:0/4176651511 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559cf941e6b0 0x559cf97f2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.402+0000 7f97d3fff640 1 --2- 172.21.15.90:0/4176651511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559cf94164d0 0x559cf97e9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.402+0000 7f97d3fff640 1 -- 172.21.15.90:0/4176651511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559cf94164d0 msgr2=0x559cf97e9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:07.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.402+0000 7f97d3fff640 1 --2- 172.21.15.90:0/4176651511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559cf94164d0 0x559cf97e9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.402+0000 7f97d3fff640 1 -- 172.21.15.90:0/4176651511 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559cf941e6b0 msgr2=0x559cf97f2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:07.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.402+0000 7f97d3fff640 1 --2- 172.21.15.90:0/4176651511 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559cf941e6b0 0x559cf97f2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.402+0000 7f97d3fff640 1 -- 172.21.15.90:0/4176651511 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cf94170f0 con 0x559cf943f390 2024-04-25T11:42:07.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.402+0000 7f97daffd640 1 --2- 172.21.15.90:0/4176651511 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559cf941e6b0 0x559cf97f2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:42:07.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.402+0000 7f97d3fff640 1 --2- 172.21.15.90:0/4176651511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf943f390 0x559cf97f2ac0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f97c4000c00 tx=0x7f97c40058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b261fc5ab47805d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:07.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.402+0000 7f97d17fa640 1 -- 172.21.15.90:0/4176651511 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c4035040 con 0x559cf943f390 2024-04-25T11:42:07.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.402+0000 7f97d17fa640 1 -- 172.21.15.90:0/4176651511 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97c4050020 con 0x559cf943f390 2024-04-25T11:42:07.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.402+0000 7f97d17fa640 1 -- 172.21.15.90:0/4176651511 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c4003b70 con 0x559cf943f390 2024-04-25T11:42:07.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.403+0000 7f97e8782740 1 -- 172.21.15.90:0/4176651511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf943f390 msgr2=0x559cf97f2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:07.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.403+0000 7f97e8782740 1 --2- 172.21.15.90:0/4176651511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf943f390 0x559cf97f2ac0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f97c4000c00 tx=0x7f97c40058b0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.403+0000 7f97e8782740 1 -- 172.21.15.90:0/4176651511 shutdown_connections 2024-04-25T11:42:07.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.403+0000 7f97e8782740 1 --2- 172.21.15.90:0/4176651511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf943f390 0x559cf97f2ac0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.403+0000 7f97e8782740 1 --2- 172.21.15.90:0/4176651511 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559cf941e6b0 0x559cf97f2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.403+0000 7f97e8782740 1 --2- 172.21.15.90:0/4176651511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559cf94164d0 0x559cf97e9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.403+0000 7f97e8782740 1 -- 172.21.15.90:0/4176651511 >> 172.21.15.90:0/4176651511 conn(0x559cf94334e0 msgr2=0x559cf97e8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:07.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.403+0000 7f97e8782740 1 -- 172.21.15.90:0/4176651511 shutdown_connections 2024-04-25T11:42:07.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.403+0000 7f97e8782740 1 -- 172.21.15.90:0/4176651511 wait complete. 2024-04-25T11:42:07.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.404+0000 7f97e8782740 1 -- start start 2024-04-25T11:42:07.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.404+0000 7f97e8782740 1 Processor -- start 2024-04-25T11:42:07.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.404+0000 7f97e8782740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559cf94164d0 0x559cf97f3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.404+0000 7f97e8782740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559cf941e6b0 0x559cf97f4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.404+0000 7f97e8782740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf943f390 0x559cf97f4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.404+0000 7f97e8782740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f97dc1aed50 con 0x559cf943f390 2024-04-25T11:42:07.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.404+0000 7f97e8782740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f97dc1ae680 con 0x559cf94164d0 2024-04-25T11:42:07.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.404+0000 7f97e8782740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f97dc1ab8c0 con 0x559cf941e6b0 2024-04-25T11:42:07.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.404+0000 7f97e8782740 10 client.0 _ll_register_callbacks cb 0x7f97cc00a650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:07.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.404+0000 7f97d3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf943f390 0x559cf97f4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.404+0000 7f97d3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf943f390 0x559cf97f4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58524/0 (socket says 172.21.15.90:58524) 2024-04-25T11:42:07.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.404+0000 7f97d3fff640 1 -- 172.21.15.90:0/808076858 learned_addr learned my addr 172.21.15.90:0/808076858 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:07.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.404+0000 7f97daffd640 1 --2- 172.21.15.90:0/808076858 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559cf941e6b0 0x559cf97f4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.404+0000 7f97d3fff640 1 -- 172.21.15.90:0/808076858 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559cf941e6b0 msgr2=0x559cf97f4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:07.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.404+0000 7f97d3fff640 1 --2- 172.21.15.90:0/808076858 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559cf941e6b0 0x559cf97f4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.404+0000 7f97d3fff640 1 -- 172.21.15.90:0/808076858 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559cf94164d0 msgr2=0x559cf97f3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:07.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.404+0000 7f97d3fff640 1 --2- 172.21.15.90:0/808076858 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559cf94164d0 0x559cf97f3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.404+0000 7f97d3fff640 1 -- 172.21.15.90:0/808076858 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cf97064b0 con 0x559cf943f390 2024-04-25T11:42:07.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.405+0000 7f97d3fff640 1 --2- 172.21.15.90:0/808076858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf943f390 0x559cf97f4580 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f97c402a930 tx=0x7f97c4005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4bb9f7ec0dde4e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:07.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.405+0000 7f97baffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:07.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.405+0000 7f97baffd640 1 -- 172.21.15.90:0/808076858 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c4035030 con 0x559cf943f390 2024-04-25T11:42:07.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.405+0000 7f97baffd640 1 -- 172.21.15.90:0/808076858 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7f97c4004100 con 0x559cf943f390 2024-04-25T11:42:07.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.405+0000 7f97e8782740 1 -- 172.21.15.90:0/808076858 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97c4050020 con 0x559cf943f390 2024-04-25T11:42:07.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.405+0000 7f97baffd640 1 -- 172.21.15.90:0/808076858 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f97c4004300 con 0x559cf943f390 2024-04-25T11:42:07.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.405+0000 7f97e8782740 10 client.17778 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:07.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.405+0000 7f97e8782740 1 -- 172.21.15.90:0/808076858 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559cf9866020 con 0x559cf943f390 2024-04-25T11:42:07.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.405+0000 7f97baffd640 1 -- 172.21.15.90:0/808076858 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97c4050020 con 0x559cf943f390 2024-04-25T11:42:07.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.405+0000 7f97baffd640 1 -- 172.21.15.90:0/808076858 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f97c4059020 con 0x559cf943f390 2024-04-25T11:42:07.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.405+0000 7f97e8782740 10 client.17778 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:07.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.405+0000 7f97e8782740 10 client.17778 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:42:07.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.405+0000 7f97e8782740 20 client.17778 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:07.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.405+0000 7f97e8782740 1 --2- 172.21.15.90:0/808076858 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559cf93d33a0 0x559cf98682b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:07.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.405+0000 7f97e8782740 4 client.17778 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:42:07.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.405+0000 7f97e8782740 1 -- 172.21.15.90:0/808076858 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559cf97e0330 con 0x559cf93d33a0 2024-04-25T11:42:07.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.406+0000 7f97daffd640 1 --2- 172.21.15.90:0/808076858 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559cf93d33a0 0x559cf98682b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:07.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.406+0000 7f97baffd640 10 client.17778 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:07.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.406+0000 7f97daffd640 1 --2- 172.21.15.90:0/808076858 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559cf93d33a0 0x559cf98682b0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a90760c3a2bf905 server_cookie=8be034142977e68 in_seq=0 out_seq=0 2024-04-25T11:42:07.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.406+0000 7f97baffd640 1 -- 172.21.15.90:0/808076858 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559cf97e0330 con 0x559cf93d33a0 2024-04-25T11:42:07.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.406+0000 7f97baffd640 10 client.17778 handle_command_reply: tid=0 2024-04-25T11:42:07.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.406+0000 7f97e8782740 1 client.17778 shutdown 2024-04-25T11:42:07.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.406+0000 7f97e8782740 1 -- 172.21.15.90:0/808076858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf943f390 msgr2=0x559cf97f4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:07.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.406+0000 7f97e8782740 1 --2- 172.21.15.90:0/808076858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf943f390 0x559cf97f4580 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f97c402a930 tx=0x7f97c4005640 comp rx=0 tx=0).stop 2024-04-25T11:42:07.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.406+0000 7f97e8782740 1 -- 172.21.15.90:0/808076858 shutdown_connections 2024-04-25T11:42:07.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.406+0000 7f97e8782740 1 --2- 172.21.15.90:0/808076858 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559cf93d33a0 0x559cf98682b0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.406+0000 7f97e8782740 1 --2- 172.21.15.90:0/808076858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cf943f390 0x559cf97f4580 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.406+0000 7f97e8782740 1 --2- 172.21.15.90:0/808076858 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559cf941e6b0 0x559cf97f4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.406+0000 7f97baffd640 0 client.17778 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:07.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.406+0000 7f97e8782740 1 --2- 172.21.15.90:0/808076858 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559cf94164d0 0x559cf97f3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.406+0000 7f97e8782740 1 -- 172.21.15.90:0/808076858 >> 172.21.15.90:0/808076858 conn(0x559cf94334e0 msgr2=0x559cf97dc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:07.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.406+0000 7f97e8782740 1 -- 172.21.15.90:0/808076858 shutdown_connections 2024-04-25T11:42:07.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.407+0000 7f97e8782740 1 -- 172.21.15.90:0/808076858 wait complete. 2024-04-25T11:42:07.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.407+0000 7f97e8782740 20 client.17778 trim_cache size 0 max 16384 2024-04-25T11:42:07.399 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:42:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T11:42:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-25T11:42:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: "674ae53a-aa58-48fc-98d5-b5ed725812dd": { 2024-04-25T11:42:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-25T11:42:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "674ae53a-aa58-48fc-98d5-b5ed725812dd", 2024-04-25T11:42:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-25T11:42:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:42:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:42:07.400 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:42:07.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.407+0000 7f97e18d9640 1 -- 172.21.15.90:0/142537091 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f97c80afc60 msgr2=0x7f97c80b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:07.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.407+0000 7f97e18d9640 1 --2- 172.21.15.90:0/142537091 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f97c80afc60 0x7f97c80b2170 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f97d40a4a90 tx=0x7f97d40ca040 comp rx=0 tx=0).stop 2024-04-25T11:42:07.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.407+0000 7f97e18d9640 1 -- 172.21.15.90:0/142537091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97dc108eb0 msgr2=0x7f97dc19de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:07.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.407+0000 7f97e18d9640 1 --2- 172.21.15.90:0/142537091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97dc108eb0 0x7f97dc19de70 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f97d400eb10 tx=0x7f97d400c2d0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.407+0000 7f97e18d9640 1 -- 172.21.15.90:0/142537091 shutdown_connections 2024-04-25T11:42:07.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.407+0000 7f97e18d9640 1 --2- 172.21.15.90:0/142537091 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f97c80afc60 0x7f97c80b2170 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.407+0000 7f97e18d9640 1 --2- 172.21.15.90:0/142537091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97dc108eb0 0x7f97dc19de70 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.407+0000 7f97e18d9640 1 --2- 172.21.15.90:0/142537091 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97dc108a10 0x7f97dc199bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.407+0000 7f97e18d9640 1 --2- 172.21.15.90:0/142537091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97dc1031c0 0x7f97dc199650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:07.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.407+0000 7f97e18d9640 1 -- 172.21.15.90:0/142537091 >> 172.21.15.90:0/142537091 conn(0x7f97dc078110 msgr2=0x7f97dc1040f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:07.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.407+0000 7f97e18d9640 1 -- 172.21.15.90:0/142537091 shutdown_connections 2024-04-25T11:42:07.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:07.407+0000 7f97e18d9640 1 -- 172.21.15.90:0/142537091 wait complete. 2024-04-25T11:42:07.405 INFO:tasks.cephfs.filesystem:scrub status for tag:674ae53a-aa58-48fc-98d5-b5ed725812dd - {'path': '/', 'tag': '674ae53a-aa58-48fc-98d5-b5ed725812dd', 'options': 'recursive,force'} 2024-04-25T11:42:07.723 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:07 smithi090 ceph-mon[29575]: pgmap v214: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 2.8 MiB/s rd, 574 MiB/s wr, 581 op/s 2024-04-25T11:42:07.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:07 smithi138 ceph-mon[32328]: pgmap v214: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 2.8 MiB/s rd, 574 MiB/s wr, 581 op/s 2024-04-25T11:42:07.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:07 smithi156 ceph-mon[33045]: pgmap v214: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 2.8 MiB/s rd, 574 MiB/s wr, 581 op/s 2024-04-25T11:42:08.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.771+0000 7fbceca51640 1 -- 172.21.15.90:0/3795357182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbce8109c00 msgr2=0x7fbce8101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:08.758 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.771+0000 7fbceca51640 1 --2- 172.21.15.90:0/3795357182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbce8109c00 0x7fbce8101b70 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8009900 tx=0x7fbcd802a410 comp rx=0 tx=0).stop 2024-04-25T11:42:08.758 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.771+0000 7fbceca51640 1 -- 172.21.15.90:0/3795357182 shutdown_connections 2024-04-25T11:42:08.758 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.771+0000 7fbceca51640 1 --2- 172.21.15.90:0/3795357182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbce810e5b0 0x7fbce81109d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.758 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.771+0000 7fbceca51640 1 --2- 172.21.15.90:0/3795357182 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbce81020b0 0x7fbce81024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.758 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.771+0000 7fbceca51640 1 --2- 172.21.15.90:0/3795357182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbce8109c00 0x7fbce8101b70 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.758 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.771+0000 7fbceca51640 1 -- 172.21.15.90:0/3795357182 >> 172.21.15.90:0/3795357182 conn(0x7fbce80fd8c0 msgr2=0x7fbce80ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:08.758 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.771+0000 7fbceca51640 1 -- 172.21.15.90:0/3795357182 shutdown_connections 2024-04-25T11:42:08.758 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.771+0000 7fbceca51640 1 -- 172.21.15.90:0/3795357182 wait complete. 2024-04-25T11:42:08.760 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.772+0000 7fbceca51640 1 Processor -- start 2024-04-25T11:42:08.760 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.772+0000 7fbceca51640 1 -- start start 2024-04-25T11:42:08.760 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.772+0000 7fbceca51640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbce81020b0 0x7fbce819a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.772+0000 7fbceca51640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbce8109c00 0x7fbce819acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.772+0000 7fbceca51640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbce810e5b0 0x7fbce819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.772+0000 7fbceca51640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbce8113d70 con 0x7fbce81020b0 2024-04-25T11:42:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.772+0000 7fbceca51640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbce8113bf0 con 0x7fbce8109c00 2024-04-25T11:42:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.772+0000 7fbceca51640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbce8113ef0 con 0x7fbce810e5b0 2024-04-25T11:42:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.772+0000 7fbce659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbce81020b0 0x7fbce819a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.772+0000 7fbce659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbce81020b0 0x7fbce819a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58542/0 (socket says 172.21.15.90:58542) 2024-04-25T11:42:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.772+0000 7fbce659c640 1 -- 172.21.15.90:0/2707242898 learned_addr learned my addr 172.21.15.90:0/2707242898 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.773+0000 7fbce5d9b640 1 --2- 172.21.15.90:0/2707242898 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbce8109c00 0x7fbce819acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.773+0000 7fbce6d9d640 1 --2- 172.21.15.90:0/2707242898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbce810e5b0 0x7fbce819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.773+0000 7fbce659c640 1 -- 172.21.15.90:0/2707242898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbce810e5b0 msgr2=0x7fbce819ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.773+0000 7fbce659c640 1 --2- 172.21.15.90:0/2707242898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbce810e5b0 0x7fbce819ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.773+0000 7fbce659c640 1 -- 172.21.15.90:0/2707242898 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbce8109c00 msgr2=0x7fbce819acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.773+0000 7fbce659c640 1 --2- 172.21.15.90:0/2707242898 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbce8109c00 0x7fbce819acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.773+0000 7fbce659c640 1 -- 172.21.15.90:0/2707242898 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce819f5d0 con 0x7fbce81020b0 2024-04-25T11:42:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.773+0000 7fbce659c640 1 --2- 172.21.15.90:0/2707242898 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbce81020b0 0x7fbce819a770 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fbcd802a930 tx=0x7fbcd8031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.773+0000 7fbcd77fe640 1 -- 172.21.15.90:0/2707242898 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd803f030 con 0x7fbce81020b0 2024-04-25T11:42:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.773+0000 7fbcd77fe640 1 -- 172.21.15.90:0/2707242898 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcd8004490 con 0x7fbce81020b0 2024-04-25T11:42:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.773+0000 7fbceca51640 1 -- 172.21.15.90:0/2707242898 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbce819f860 con 0x7fbce81020b0 2024-04-25T11:42:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.773+0000 7fbcd77fe640 1 -- 172.21.15.90:0/2707242898 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd8035070 con 0x7fbce81020b0 2024-04-25T11:42:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.773+0000 7fbceca51640 1 -- 172.21.15.90:0/2707242898 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbce819fd50 con 0x7fbce81020b0 2024-04-25T11:42:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.776+0000 7fbcd77fe640 1 -- 172.21.15.90:0/2707242898 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fbcd8042030 con 0x7fbce81020b0 2024-04-25T11:42:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.776+0000 7fbcf39e2740 1 Processor -- start 2024-04-25T11:42:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.776+0000 7fbcf39e2740 1 -- start start 2024-04-25T11:42:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.776+0000 7fbcf39e2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d461b0 0x55f1710e59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.776+0000 7fbcf39e2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f170d1d2f0 0x55f1710e5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.776+0000 7fbcf39e2740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f170d254d0 0x55f1710e6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.776+0000 7fbcf39e2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f17104c460 con 0x55f170d461b0 2024-04-25T11:42:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.776+0000 7fbcf39e2740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f171006590 con 0x55f170d254d0 2024-04-25T11:42:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.776+0000 7fbcf39e2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f171048420 con 0x55f170d1d2f0 2024-04-25T11:42:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.776+0000 7fbce6d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d461b0 0x55f1710e59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.776+0000 7fbce6d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d461b0 0x55f1710e59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58546/0 (socket says 172.21.15.90:58546) 2024-04-25T11:42:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.776+0000 7fbce6d9d640 1 -- 172.21.15.90:0/2077000892 learned_addr learned my addr 172.21.15.90:0/2077000892 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:08.771 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.776+0000 7fbce659c640 1 --2- 172.21.15.90:0/2077000892 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f170d1d2f0 0x55f1710e5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:08.771 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.776+0000 7fbce5d9b640 1 --2- 172.21.15.90:0/2077000892 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f170d254d0 0x55f1710e6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:08.771 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.777+0000 7fbcd77fe640 1 --2- 172.21.15.90:0/2707242898 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbcd00afc50 0x7fbcd00b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:08.771 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.777+0000 7fbce6d9d640 1 -- 172.21.15.90:0/2077000892 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f170d1d2f0 msgr2=0x55f1710e5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:08.771 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.777+0000 7fbce6d9d640 1 --2- 172.21.15.90:0/2077000892 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f170d1d2f0 0x55f1710e5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.771 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.777+0000 7fbce6d9d640 1 -- 172.21.15.90:0/2077000892 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f170d254d0 msgr2=0x55f1710e6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:08.771 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.777+0000 7fbce6d9d640 1 --2- 172.21.15.90:0/2077000892 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f170d254d0 0x55f1710e6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.771 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.777+0000 7fbce6d9d640 1 -- 172.21.15.90:0/2077000892 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1710130d0 con 0x55f170d461b0 2024-04-25T11:42:08.771 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.777+0000 7fbcd77fe640 1 -- 172.21.15.90:0/2707242898 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fbcd80ff500 con 0x7fbce81020b0 2024-04-25T11:42:08.771 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.777+0000 7fbce6d9d640 1 --2- 172.21.15.90:0/2707242898 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbcd00afc50 0x7fbcd00b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:08.771 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.777+0000 7fbce6d9d640 1 --2- 172.21.15.90:0/2077000892 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d461b0 0x55f1710e59a0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fbce000f3e0 tx=0x7fbce000f8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5316091d4b0a16da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:08.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.777+0000 7fbcd4ff9640 1 -- 172.21.15.90:0/2077000892 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce0012e60 con 0x55f170d461b0 2024-04-25T11:42:08.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.777+0000 7fbcd4ff9640 1 -- 172.21.15.90:0/2077000892 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbce0005050 con 0x55f170d461b0 2024-04-25T11:42:08.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.777+0000 7fbce6d9d640 1 --2- 172.21.15.90:0/2707242898 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbcd00afc50 0x7fbcd00b2160 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbce0020000 tx=0x7fbce001d000 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:08.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.777+0000 7fbcd4ff9640 1 -- 172.21.15.90:0/2077000892 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce001c660 con 0x55f170d461b0 2024-04-25T11:42:08.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.778+0000 7fbcf39e2740 1 -- 172.21.15.90:0/2077000892 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d461b0 msgr2=0x55f1710e59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:08.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.778+0000 7fbcf39e2740 1 --2- 172.21.15.90:0/2077000892 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d461b0 0x55f1710e59a0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fbce000f3e0 tx=0x7fbce000f8a0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.778+0000 7fbcf39e2740 1 -- 172.21.15.90:0/2077000892 shutdown_connections 2024-04-25T11:42:08.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.778+0000 7fbcf39e2740 1 --2- 172.21.15.90:0/2077000892 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f170d254d0 0x55f1710e6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.778+0000 7fbcf39e2740 1 --2- 172.21.15.90:0/2077000892 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f170d1d2f0 0x55f1710e5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.778+0000 7fbcf39e2740 1 --2- 172.21.15.90:0/2077000892 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d461b0 0x55f1710e59a0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.778+0000 7fbcf39e2740 1 -- 172.21.15.90:0/2077000892 >> 172.21.15.90:0/2077000892 conn(0x55f170d3a300 msgr2=0x55f1710e22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:08.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.778+0000 7fbcf39e2740 1 -- 172.21.15.90:0/2077000892 shutdown_connections 2024-04-25T11:42:08.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.778+0000 7fbcf39e2740 1 -- 172.21.15.90:0/2077000892 wait complete. 2024-04-25T11:42:08.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.779+0000 7fbcf39e2740 1 -- start start 2024-04-25T11:42:08.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.779+0000 7fbcf39e2740 1 Processor -- start 2024-04-25T11:42:08.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.779+0000 7fbcf39e2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d1d2f0 0x55f1710f5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:08.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.779+0000 7fbcf39e2740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f170d254d0 0x55f1710f6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:08.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.779+0000 7fbcf39e2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f170d461b0 0x55f1710f4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:08.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.779+0000 7fbcf39e2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f1710f7030 con 0x55f170d1d2f0 2024-04-25T11:42:08.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.779+0000 7fbcf39e2740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f170fd3b60 con 0x55f170d254d0 2024-04-25T11:42:08.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.779+0000 7fbcf39e2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f1710f71b0 con 0x55f170d461b0 2024-04-25T11:42:08.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.779+0000 7fbcf39e2740 10 client.0 _ll_register_callbacks cb 0x55f170ff82b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:08.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.779+0000 7fbce659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d1d2f0 0x55f1710f5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:08.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.779+0000 7fbce659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d1d2f0 0x55f1710f5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58554/0 (socket says 172.21.15.90:58554) 2024-04-25T11:42:08.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.779+0000 7fbce659c640 1 -- 172.21.15.90:0/1332635805 learned_addr learned my addr 172.21.15.90:0/1332635805 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:08.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.780+0000 7fbce6d9d640 1 --2- 172.21.15.90:0/1332635805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f170d461b0 0x55f1710f4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:08.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.780+0000 7fbce5d9b640 1 --2- 172.21.15.90:0/1332635805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f170d254d0 0x55f1710f6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:08.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.780+0000 7fbce659c640 1 -- 172.21.15.90:0/1332635805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f170d461b0 msgr2=0x55f1710f4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:08.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.780+0000 7fbce659c640 1 --2- 172.21.15.90:0/1332635805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f170d461b0 0x55f1710f4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.780+0000 7fbce659c640 1 -- 172.21.15.90:0/1332635805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f170d254d0 msgr2=0x55f1710f6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:08.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.780+0000 7fbce659c640 1 --2- 172.21.15.90:0/1332635805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f170d254d0 0x55f1710f6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.780+0000 7fbce659c640 1 -- 172.21.15.90:0/1332635805 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f170d5e210 con 0x55f170d1d2f0 2024-04-25T11:42:08.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.780+0000 7fbce6d9d640 1 --2- 172.21.15.90:0/1332635805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f170d461b0 0x55f1710f4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:42:08.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.780+0000 7fbce5d9b640 1 --2- 172.21.15.90:0/1332635805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f170d254d0 0x55f1710f6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:42:08.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.780+0000 7fbce659c640 1 --2- 172.21.15.90:0/1332635805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d1d2f0 0x55f1710f5f40 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8006010 tx=0x7fbcd8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e17419bc14be5dbd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:08.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.780+0000 7fbccdffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:08.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.780+0000 7fbccdffb640 1 -- 172.21.15.90:0/1332635805 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd8049040 con 0x55f170d1d2f0 2024-04-25T11:42:08.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.780+0000 7fbccdffb640 1 -- 172.21.15.90:0/1332635805 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fbcd80c38d0 con 0x55f170d1d2f0 2024-04-25T11:42:08.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.780+0000 7fbcf39e2740 1 -- 172.21.15.90:0/1332635805 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbce0005050 con 0x55f170d1d2f0 2024-04-25T11:42:08.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.780+0000 7fbccdffb640 1 -- 172.21.15.90:0/1332635805 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbcd80c7410 con 0x55f170d1d2f0 2024-04-25T11:42:08.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.780+0000 7fbcf39e2740 10 client.17790 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:08.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.780+0000 7fbcf39e2740 1 -- 172.21.15.90:0/1332635805 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f170e2a620 con 0x55f170d1d2f0 2024-04-25T11:42:08.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.780+0000 7fbccdffb640 1 -- 172.21.15.90:0/1332635805 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbce0005050 con 0x55f170d1d2f0 2024-04-25T11:42:08.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.780+0000 7fbccdffb640 1 -- 172.21.15.90:0/1332635805 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fbcd803ba30 con 0x55f170d1d2f0 2024-04-25T11:42:08.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.780+0000 7fbcf39e2740 10 client.17790 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:08.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.780+0000 7fbcf39e2740 10 client.17790 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:42:08.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.781+0000 7fbcf39e2740 20 client.17790 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:08.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.781+0000 7fbcf39e2740 1 --2- 172.21.15.90:0/1332635805 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f170d04d80 0x55f171106b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:08.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.781+0000 7fbcf39e2740 4 client.17790 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:42:08.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.781+0000 7fbcf39e2740 1 -- 172.21.15.90:0/1332635805 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f17105a070 con 0x55f170d04d80 2024-04-25T11:42:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.781+0000 7fbce5d9b640 1 --2- 172.21.15.90:0/1332635805 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f170d04d80 0x55f171106b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.781+0000 7fbccdffb640 10 client.17790 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.781+0000 7fbce5d9b640 1 --2- 172.21.15.90:0/1332635805 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f170d04d80 0x55f171106b70 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e11e2ab18294e4dc server_cookie=7dae526a0891d9cd in_seq=0 out_seq=0 2024-04-25T11:42:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.782+0000 7fbccdffb640 1 -- 172.21.15.90:0/1332635805 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f17105a070 con 0x55f170d04d80 2024-04-25T11:42:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.782+0000 7fbccdffb640 10 client.17790 handle_command_reply: tid=0 2024-04-25T11:42:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.782+0000 7fbcf39e2740 1 client.17790 shutdown 2024-04-25T11:42:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.782+0000 7fbcf39e2740 1 -- 172.21.15.90:0/1332635805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d1d2f0 msgr2=0x55f1710f5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.782+0000 7fbcf39e2740 1 --2- 172.21.15.90:0/1332635805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d1d2f0 0x55f1710f5f40 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8006010 tx=0x7fbcd8000bc0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.782+0000 7fbcf39e2740 1 -- 172.21.15.90:0/1332635805 shutdown_connections 2024-04-25T11:42:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.782+0000 7fbcf39e2740 1 --2- 172.21.15.90:0/1332635805 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f170d04d80 0x55f171106b70 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.782+0000 7fbcf39e2740 1 --2- 172.21.15.90:0/1332635805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f170d461b0 0x55f1710f4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.782+0000 7fbcf39e2740 1 --2- 172.21.15.90:0/1332635805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f170d254d0 0x55f1710f6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.782+0000 7fbcf39e2740 1 --2- 172.21.15.90:0/1332635805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d1d2f0 0x55f1710f5f40 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.782+0000 7fbccdffb640 0 client.17790 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:08.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.782+0000 7fbcf39e2740 1 -- 172.21.15.90:0/1332635805 >> 172.21.15.90:0/1332635805 conn(0x55f170d3a300 msgr2=0x55f1710e77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:08.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.782+0000 7fbcf39e2740 1 -- 172.21.15.90:0/1332635805 shutdown_connections 2024-04-25T11:42:08.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.782+0000 7fbcf39e2740 1 -- 172.21.15.90:0/1332635805 wait complete. 2024-04-25T11:42:08.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.782+0000 7fbcf39e2740 20 client.17790 trim_cache size 0 max 16384 2024-04-25T11:42:08.788 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.797+0000 7fbcf39e2740 1 Processor -- start 2024-04-25T11:42:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.797+0000 7fbcf39e2740 1 -- start start 2024-04-25T11:42:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.798+0000 7fbcf39e2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f170d1d2f0 0x55f1710e7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.798+0000 7fbcf39e2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d254d0 0x55f17110c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.798+0000 7fbcf39e2740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f170d461b0 0x55f17110c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.798+0000 7fbcf39e2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbce81ad980 con 0x55f170d254d0 2024-04-25T11:42:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.798+0000 7fbcf39e2740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbce81af4c0 con 0x55f170d461b0 2024-04-25T11:42:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.798+0000 7fbcf39e2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbce81ae050 con 0x55f170d1d2f0 2024-04-25T11:42:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.798+0000 7fbce5d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d254d0 0x55f17110c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.798+0000 7fbce5d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d254d0 0x55f17110c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58560/0 (socket says 172.21.15.90:58560) 2024-04-25T11:42:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.798+0000 7fbce5d9b640 1 -- 172.21.15.90:0/536031785 learned_addr learned my addr 172.21.15.90:0/536031785 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.798+0000 7fbce6d9d640 1 --2- 172.21.15.90:0/536031785 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f170d461b0 0x55f17110c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.798+0000 7fbce5d9b640 1 -- 172.21.15.90:0/536031785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f170d1d2f0 msgr2=0x55f1710e7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.798+0000 7fbce5d9b640 1 --2- 172.21.15.90:0/536031785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f170d1d2f0 0x55f1710e7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.798+0000 7fbce5d9b640 1 -- 172.21.15.90:0/536031785 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f170d461b0 msgr2=0x55f17110c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.798+0000 7fbce5d9b640 1 --2- 172.21.15.90:0/536031785 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f170d461b0 0x55f17110c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.798+0000 7fbce5d9b640 1 -- 172.21.15.90:0/536031785 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1710dc4c0 con 0x55f170d254d0 2024-04-25T11:42:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.798+0000 7fbce6d9d640 1 --2- 172.21.15.90:0/536031785 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f170d461b0 0x55f17110c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:42:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.798+0000 7fbce5d9b640 1 --2- 172.21.15.90:0/536031785 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d254d0 0x55f17110c370 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc000c00 tx=0x7fbcdc0044d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8585dc7d188807a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.798+0000 7fbccffff640 1 -- 172.21.15.90:0/536031785 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc003e40 con 0x55f170d254d0 2024-04-25T11:42:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.798+0000 7fbccffff640 1 -- 172.21.15.90:0/536031785 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcdc023070 con 0x55f170d254d0 2024-04-25T11:42:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.798+0000 7fbccffff640 1 -- 172.21.15.90:0/536031785 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc012be0 con 0x55f170d254d0 2024-04-25T11:42:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.799+0000 7fbcf39e2740 1 -- 172.21.15.90:0/536031785 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d254d0 msgr2=0x55f17110c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.799+0000 7fbcf39e2740 1 --2- 172.21.15.90:0/536031785 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d254d0 0x55f17110c370 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc000c00 tx=0x7fbcdc0044d0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.799+0000 7fbcf39e2740 1 -- 172.21.15.90:0/536031785 shutdown_connections 2024-04-25T11:42:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.799+0000 7fbcf39e2740 1 --2- 172.21.15.90:0/536031785 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f170d461b0 0x55f17110c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.799+0000 7fbcf39e2740 1 --2- 172.21.15.90:0/536031785 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d254d0 0x55f17110c370 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.799+0000 7fbcf39e2740 1 --2- 172.21.15.90:0/536031785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f170d1d2f0 0x55f1710e7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.799+0000 7fbcf39e2740 1 -- 172.21.15.90:0/536031785 >> 172.21.15.90:0/536031785 conn(0x55f170d3a300 msgr2=0x55f1710e77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.799+0000 7fbcf39e2740 1 -- 172.21.15.90:0/536031785 shutdown_connections 2024-04-25T11:42:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.799+0000 7fbcf39e2740 1 -- 172.21.15.90:0/536031785 wait complete. 2024-04-25T11:42:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.799+0000 7fbcf39e2740 1 -- start start 2024-04-25T11:42:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.800+0000 7fbcf39e2740 1 Processor -- start 2024-04-25T11:42:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.800+0000 7fbcf39e2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f170d1d2f0 0x55f1710fa040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.800+0000 7fbcf39e2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d254d0 0x55f1710fa580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.800+0000 7fbcf39e2740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f170d461b0 0x55f1710faac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.800+0000 7fbcf39e2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbce81aedf0 con 0x55f170d254d0 2024-04-25T11:42:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.800+0000 7fbcf39e2740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbce81ae720 con 0x55f170d461b0 2024-04-25T11:42:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.800+0000 7fbcf39e2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbce81ab960 con 0x55f170d1d2f0 2024-04-25T11:42:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.800+0000 7fbcf39e2740 10 client.0 _ll_register_callbacks cb 0x7fbcd8005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.800+0000 7fbce5d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d254d0 0x55f1710fa580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.800+0000 7fbce5d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d254d0 0x55f1710fa580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58572/0 (socket says 172.21.15.90:58572) 2024-04-25T11:42:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.800+0000 7fbce5d9b640 1 -- 172.21.15.90:0/2888124591 learned_addr learned my addr 172.21.15.90:0/2888124591 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.800+0000 7fbce6d9d640 1 --2- 172.21.15.90:0/2888124591 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f170d461b0 0x55f1710faac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.800+0000 7fbce659c640 1 --2- 172.21.15.90:0/2888124591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f170d1d2f0 0x55f1710fa040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.800+0000 7fbce5d9b640 1 -- 172.21.15.90:0/2888124591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f170d1d2f0 msgr2=0x55f1710fa040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.800+0000 7fbce5d9b640 1 --2- 172.21.15.90:0/2888124591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f170d1d2f0 0x55f1710fa040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.800+0000 7fbce5d9b640 1 -- 172.21.15.90:0/2888124591 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f170d461b0 msgr2=0x55f1710faac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.800+0000 7fbce5d9b640 1 --2- 172.21.15.90:0/2888124591 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f170d461b0 0x55f1710faac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.800+0000 7fbce5d9b640 1 -- 172.21.15.90:0/2888124591 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f17116df30 con 0x55f170d254d0 2024-04-25T11:42:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.800+0000 7fbce5d9b640 1 --2- 172.21.15.90:0/2888124591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d254d0 0x55f1710fa580 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc009cc0 tx=0x7fbcdc003c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3428a80d61c03534 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.800+0000 7fbccdffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.801+0000 7fbccdffb640 1 -- 172.21.15.90:0/2888124591 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc012960 con 0x55f170d254d0 2024-04-25T11:42:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.801+0000 7fbccdffb640 1 -- 172.21.15.90:0/2888124591 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 8614+0+0 (secure 0 0 0) 0x7fbcdc012b00 con 0x55f170d254d0 2024-04-25T11:42:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.801+0000 7fbcf39e2740 1 -- 172.21.15.90:0/2888124591 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbcdc023070 con 0x55f170d254d0 2024-04-25T11:42:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.801+0000 7fbccdffb640 1 -- 172.21.15.90:0/2888124591 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbcdc03a050 con 0x55f170d254d0 2024-04-25T11:42:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.801+0000 7fbcf39e2740 10 client.17796 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.801+0000 7fbcf39e2740 1 -- 172.21.15.90:0/2888124591 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f17116f130 con 0x55f170d254d0 2024-04-25T11:42:08.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.801+0000 7fbccdffb640 1 -- 172.21.15.90:0/2888124591 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcdc023070 con 0x55f170d254d0 2024-04-25T11:42:08.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.801+0000 7fbccdffb640 1 -- 172.21.15.90:0/2888124591 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fbcdc02a540 con 0x55f170d254d0 2024-04-25T11:42:08.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.801+0000 7fbcf39e2740 10 client.17796 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:08.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.801+0000 7fbcf39e2740 10 client.17796 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:42:08.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.801+0000 7fbcf39e2740 20 client.17796 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:08.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.801+0000 7fbcf39e2740 1 --2- 172.21.15.90:0/2888124591 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f170cda1c0 0x55f1711718c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:08.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.801+0000 7fbcf39e2740 4 client.17796 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:42:08.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.801+0000 7fbcf39e2740 1 -- 172.21.15.90:0/2888124591 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f171053ee0 con 0x55f170cda1c0 2024-04-25T11:42:08.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.802+0000 7fbce659c640 1 --2- 172.21.15.90:0/2888124591 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f170cda1c0 0x55f1711718c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:08.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.802+0000 7fbce659c640 1 --2- 172.21.15.90:0/2888124591 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f170cda1c0 0x55f1711718c0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64eec6c35cdaef server_cookie=62ab5ab8b4f65049 in_seq=0 out_seq=0 2024-04-25T11:42:08.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.802+0000 7fbccdffb640 10 client.17796 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:08.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:08 smithi138 ceph-mon[32328]: scrub queued for path: / 2024-04-25T11:42:08.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:08 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:42:08.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:08 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:42:08.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:08 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:42:08.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:08 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:42:08.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:08 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:42:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.821+0000 7fbccdffb640 1 -- 172.21.15.90:0/2888124591 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f171053ee0 con 0x55f170cda1c0 2024-04-25T11:42:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.821+0000 7fbccdffb640 10 client.17796 handle_command_reply: tid=0 2024-04-25T11:42:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.821+0000 7fbcf39e2740 1 client.17796 shutdown 2024-04-25T11:42:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.821+0000 7fbcf39e2740 1 -- 172.21.15.90:0/2888124591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d254d0 msgr2=0x55f1710fa580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.821+0000 7fbcf39e2740 1 --2- 172.21.15.90:0/2888124591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d254d0 0x55f1710fa580 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc009cc0 tx=0x7fbcdc003c40 comp rx=0 tx=0).stop 2024-04-25T11:42:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.822+0000 7fbcf39e2740 1 -- 172.21.15.90:0/2888124591 shutdown_connections 2024-04-25T11:42:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.822+0000 7fbcf39e2740 1 --2- 172.21.15.90:0/2888124591 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55f170cda1c0 0x55f1711718c0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.822+0000 7fbcf39e2740 1 --2- 172.21.15.90:0/2888124591 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f170d461b0 0x55f1710faac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.822+0000 7fbccdffb640 0 client.17796 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.822+0000 7fbcf39e2740 1 --2- 172.21.15.90:0/2888124591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f170d254d0 0x55f1710fa580 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.822+0000 7fbcf39e2740 1 --2- 172.21.15.90:0/2888124591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f170d1d2f0 0x55f1710fa040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.822+0000 7fbcf39e2740 1 -- 172.21.15.90:0/2888124591 >> 172.21.15.90:0/2888124591 conn(0x55f170d3a300 msgr2=0x55f1710f5c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.822+0000 7fbcf39e2740 1 -- 172.21.15.90:0/2888124591 shutdown_connections 2024-04-25T11:42:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.823+0000 7fbcf39e2740 1 -- 172.21.15.90:0/2888124591 wait complete. 2024-04-25T11:42:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.823+0000 7fbcf39e2740 20 client.17796 trim_cache size 0 max 16384 2024-04-25T11:42:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:42:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:42:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:42:08.811 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:42:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.824+0000 7fbceca51640 1 -- 172.21.15.90:0/2707242898 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbcd00afc50 msgr2=0x7fbcd00b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.824+0000 7fbceca51640 1 --2- 172.21.15.90:0/2707242898 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbcd00afc50 0x7fbcd00b2160 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbce0020000 tx=0x7fbce001d000 comp rx=0 tx=0).stop 2024-04-25T11:42:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.824+0000 7fbceca51640 1 -- 172.21.15.90:0/2707242898 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbce81020b0 msgr2=0x7fbce819a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.824+0000 7fbceca51640 1 --2- 172.21.15.90:0/2707242898 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbce81020b0 0x7fbce819a770 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fbcd802a930 tx=0x7fbcd8031a20 comp rx=0 tx=0).stop 2024-04-25T11:42:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.824+0000 7fbceca51640 1 -- 172.21.15.90:0/2707242898 shutdown_connections 2024-04-25T11:42:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.824+0000 7fbceca51640 1 --2- 172.21.15.90:0/2707242898 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbcd00afc50 0x7fbcd00b2160 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.824+0000 7fbceca51640 1 --2- 172.21.15.90:0/2707242898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbce810e5b0 0x7fbce819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.824+0000 7fbceca51640 1 --2- 172.21.15.90:0/2707242898 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbce8109c00 0x7fbce819acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.824+0000 7fbceca51640 1 --2- 172.21.15.90:0/2707242898 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbce81020b0 0x7fbce819a770 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.813 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.824+0000 7fbceca51640 1 -- 172.21.15.90:0/2707242898 >> 172.21.15.90:0/2707242898 conn(0x7fbce80fd8c0 msgr2=0x7fbce80fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:08.813 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.824+0000 7fbceca51640 1 -- 172.21.15.90:0/2707242898 shutdown_connections 2024-04-25T11:42:08.813 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:08.824+0000 7fbceca51640 1 -- 172.21.15.90:0/2707242898 wait complete. 2024-04-25T11:42:08.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:08 smithi156 ceph-mon[33045]: scrub queued for path: / 2024-04-25T11:42:08.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:08 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:42:08.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:08 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:42:08.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:08 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:42:08.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:08 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:42:08.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:08 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:42:08.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:08 smithi090 ceph-mon[29575]: scrub queued for path: / 2024-04-25T11:42:08.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:08 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:42:08.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:08 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:42:08.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:08 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:42:08.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:08 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:42:08.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:08 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:42:09.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:09 smithi138 ceph-mon[32328]: pgmap v215: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 427 MiB/s wr, 508 op/s 2024-04-25T11:42:09.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:09 smithi156 ceph-mon[33045]: pgmap v215: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 427 MiB/s wr, 508 op/s 2024-04-25T11:42:09.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:09 smithi090 ceph-mon[29575]: pgmap v215: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 427 MiB/s wr, 508 op/s 2024-04-25T11:42:11.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:11 smithi138 ceph-mon[32328]: pgmap v216: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 482 MiB/s wr, 700 op/s 2024-04-25T11:42:11.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:11 smithi138 ceph-mon[32328]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T11:42:11.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:11 smithi156 ceph-mon[33045]: pgmap v216: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 482 MiB/s wr, 700 op/s 2024-04-25T11:42:11.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:11 smithi156 ceph-mon[33045]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T11:42:11.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:11 smithi090 ceph-mon[29575]: pgmap v216: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 482 MiB/s wr, 700 op/s 2024-04-25T11:42:11.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:11 smithi090 ceph-mon[29575]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T11:42:13.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:13 smithi138 ceph-mon[32328]: pgmap v218: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 1017 KiB/s rd, 310 MiB/s wr, 756 op/s 2024-04-25T11:42:13.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:13 smithi156 ceph-mon[33045]: pgmap v218: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 1017 KiB/s rd, 310 MiB/s wr, 756 op/s 2024-04-25T11:42:13.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:13 smithi090 ceph-mon[29575]: pgmap v218: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 1017 KiB/s rd, 310 MiB/s wr, 756 op/s 2024-04-25T11:42:15.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:15 smithi138 ceph-mon[32328]: pgmap v219: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 1017 KiB/s rd, 287 MiB/s wr, 741 op/s 2024-04-25T11:42:15.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:15 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:42:15.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:15 smithi156 ceph-mon[33045]: pgmap v219: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 1017 KiB/s rd, 287 MiB/s wr, 741 op/s 2024-04-25T11:42:15.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:15 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:42:15.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:15 smithi090 ceph-mon[29575]: pgmap v219: 97 pgs: 97 active+clean; 60 GiB data, 183 GiB used, 890 GiB / 1.0 TiB avail; 1017 KiB/s rd, 287 MiB/s wr, 741 op/s 2024-04-25T11:42:15.974 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:15 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:42:17.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:16 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:42:17.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:16 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:42:17.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:16 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:42:18.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:17 smithi090 ceph-mon[29575]: pgmap v220: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 237 KiB/s rd, 88 MiB/s wr, 483 op/s 2024-04-25T11:42:18.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:18 smithi138 ceph-mon[32328]: pgmap v220: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 237 KiB/s rd, 88 MiB/s wr, 483 op/s 2024-04-25T11:42:18.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:17 smithi156 ceph-mon[33045]: pgmap v220: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 237 KiB/s rd, 88 MiB/s wr, 483 op/s 2024-04-25T11:42:20.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:19 smithi090 ceph-mon[29575]: pgmap v221: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 237 KiB/s rd, 97 MiB/s wr, 576 op/s 2024-04-25T11:42:20.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:19 smithi090 ceph-mon[29575]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T11:42:20.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:19 smithi138 ceph-mon[32328]: pgmap v221: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 237 KiB/s rd, 97 MiB/s wr, 576 op/s 2024-04-25T11:42:20.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:19 smithi138 ceph-mon[32328]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T11:42:20.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:19 smithi156 ceph-mon[33045]: pgmap v221: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 237 KiB/s rd, 97 MiB/s wr, 576 op/s 2024-04-25T11:42:20.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:19 smithi156 ceph-mon[33045]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T11:42:21.915 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.926+0000 7ff5aea0d640 1 -- 172.21.15.90:0/3606165083 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5a810a020 msgr2=0x7ff5a8103c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:21.915 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.926+0000 7ff5aea0d640 1 --2- 172.21.15.90:0/3606165083 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5a810a020 0x7ff5a8103c60 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7ff59c009900 tx=0x7ff59c02a410 comp rx=0 tx=0).stop 2024-04-25T11:42:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.926+0000 7ff5aea0d640 1 -- 172.21.15.90:0/3606165083 shutdown_connections 2024-04-25T11:42:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.926+0000 7ff5aea0d640 1 --2- 172.21.15.90:0/3606165083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5a81042d0 0x7ff5a81106a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.926+0000 7ff5aea0d640 1 --2- 172.21.15.90:0/3606165083 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5a810a020 0x7ff5a8103c60 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.926+0000 7ff5aea0d640 1 --2- 172.21.15.90:0/3606165083 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff5a8109b80 0x7ff5a8103720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.926+0000 7ff5aea0d640 1 -- 172.21.15.90:0/3606165083 >> 172.21.15.90:0/3606165083 conn(0x7ff5a8079270 msgr2=0x7ff5a80796b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.926+0000 7ff5aea0d640 1 -- 172.21.15.90:0/3606165083 shutdown_connections 2024-04-25T11:42:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.926+0000 7ff5aea0d640 1 -- 172.21.15.90:0/3606165083 wait complete. 2024-04-25T11:42:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.927+0000 7ff5aea0d640 1 Processor -- start 2024-04-25T11:42:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.927+0000 7ff5aea0d640 1 -- start start 2024-04-25T11:42:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.927+0000 7ff5aea0d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5a81042d0 0x7ff5a819a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.927+0000 7ff5aea0d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5a8109b80 0x7ff5a819acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.928+0000 7ff5aea0d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff5a810a020 0x7ff5a819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.928+0000 7ff5aea0d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a8113d70 con 0x7ff5a8109b80 2024-04-25T11:42:21.916 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.928+0000 7ff5aea0d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a8113bf0 con 0x7ff5a810a020 2024-04-25T11:42:21.917 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.928+0000 7ff5aea0d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a8113ef0 con 0x7ff5a81042d0 2024-04-25T11:42:21.917 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.928+0000 7ff5a77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5a8109b80 0x7ff5a819acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:21.917 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.928+0000 7ff5a77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5a8109b80 0x7ff5a819acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40234/0 (socket says 172.21.15.90:40234) 2024-04-25T11:42:21.917 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.928+0000 7ff5a77fe640 1 -- 172.21.15.90:0/1417334969 learned_addr learned my addr 172.21.15.90:0/1417334969 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:21.917 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.928+0000 7ff5acfaa640 1 --2- 172.21.15.90:0/1417334969 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff5a810a020 0x7ff5a819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:21.917 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.928+0000 7ff5a7fff640 1 --2- 172.21.15.90:0/1417334969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5a81042d0 0x7ff5a819a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:21.917 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.928+0000 7ff5acfaa640 1 -- 172.21.15.90:0/1417334969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5a81042d0 msgr2=0x7ff5a819a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:21.917 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.928+0000 7ff5acfaa640 1 --2- 172.21.15.90:0/1417334969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5a81042d0 0x7ff5a819a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.917 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.928+0000 7ff5acfaa640 1 -- 172.21.15.90:0/1417334969 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5a8109b80 msgr2=0x7ff5a819acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:21.917 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.928+0000 7ff5acfaa640 1 --2- 172.21.15.90:0/1417334969 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5a8109b80 0x7ff5a819acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.917 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.928+0000 7ff5acfaa640 1 -- 172.21.15.90:0/1417334969 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5a819f5d0 con 0x7ff5a810a020 2024-04-25T11:42:21.917 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.928+0000 7ff5a77fe640 1 --2- 172.21.15.90:0/1417334969 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5a8109b80 0x7ff5a819acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:42:21.917 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.928+0000 7ff5acfaa640 1 --2- 172.21.15.90:0/1417334969 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff5a810a020 0x7ff5a819ef90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff5a000da10 tx=0x7ff5a000ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:21.917 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.928+0000 7ff5a57fa640 1 -- 172.21.15.90:0/1417334969 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a000bd70 con 0x7ff5a810a020 2024-04-25T11:42:21.918 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.928+0000 7ff5a57fa640 1 -- 172.21.15.90:0/1417334969 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5a0003e20 con 0x7ff5a810a020 2024-04-25T11:42:21.918 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.928+0000 7ff5aea0d640 1 -- 172.21.15.90:0/1417334969 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5a819f8c0 con 0x7ff5a810a020 2024-04-25T11:42:21.918 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.928+0000 7ff5a57fa640 1 -- 172.21.15.90:0/1417334969 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a00106c0 con 0x7ff5a810a020 2024-04-25T11:42:21.918 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.929+0000 7ff5aea0d640 1 -- 172.21.15.90:0/1417334969 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5a819fdc0 con 0x7ff5a810a020 2024-04-25T11:42:21.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.931+0000 7ff5a57fa640 1 -- 172.21.15.90:0/1417334969 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7ff5a0004270 con 0x7ff5a810a020 2024-04-25T11:42:21.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.931+0000 7ff5b5995740 1 Processor -- start 2024-04-25T11:42:21.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.931+0000 7ff5b5995740 1 -- start start 2024-04-25T11:42:21.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.931+0000 7ff5b5995740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5577bdac41b0 0x5577bde639a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:21.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.931+0000 7ff5b5995740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5577bda9b2f0 0x5577bde63ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:21.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.931+0000 7ff5b5995740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5577bdaa34d0 0x5577bde64420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:21.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.931+0000 7ff5b5995740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5577bddca460 con 0x5577bda9b2f0 2024-04-25T11:42:21.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.931+0000 7ff5b5995740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5577bdd84590 con 0x5577bdaa34d0 2024-04-25T11:42:21.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.931+0000 7ff5b5995740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5577bddc6420 con 0x5577bdac41b0 2024-04-25T11:42:21.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.931+0000 7ff5a57fa640 1 --2- 172.21.15.90:0/1417334969 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff5900afc50 0x7ff5900b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:21.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.932+0000 7ff5a77fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5577bdaa34d0 0x5577bde64420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:21.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.932+0000 7ff5a77fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5577bdaa34d0 0x5577bde64420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:46476/0 (socket says 172.21.15.90:46476) 2024-04-25T11:42:21.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.932+0000 7ff5a77fe640 1 -- 172.21.15.90:0/1914756555 learned_addr learned my addr 172.21.15.90:0/1914756555 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:21.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.932+0000 7ff5a7fff640 1 --2- 172.21.15.90:0/1914756555 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5577bda9b2f0 0x5577bde63ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:21.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.932+0000 7ff5acfaa640 1 --2- 172.21.15.90:0/1417334969 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff5900afc50 0x7ff5900b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:21.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.932+0000 7ff5a77fe640 1 --2- 172.21.15.90:0/1914756555 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5577bdac41b0 0x5577bde639a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:21.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.932+0000 7ff5a57fa640 1 -- 172.21.15.90:0/1417334969 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7ff5a00df530 con 0x7ff5a810a020 2024-04-25T11:42:21.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.932+0000 7ff5a77fe640 1 -- 172.21.15.90:0/1914756555 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5577bdac41b0 msgr2=0x5577bde639a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:21.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.932+0000 7ff5a77fe640 1 --2- 172.21.15.90:0/1914756555 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5577bdac41b0 0x5577bde639a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.932+0000 7ff5a77fe640 1 -- 172.21.15.90:0/1914756555 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5577bda9b2f0 msgr2=0x5577bde63ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:21.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.932+0000 7ff5a77fe640 1 --2- 172.21.15.90:0/1914756555 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5577bda9b2f0 0x5577bde63ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.932+0000 7ff5a77fe640 1 -- 172.21.15.90:0/1914756555 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577bdd910d0 con 0x5577bdaa34d0 2024-04-25T11:42:21.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.932+0000 7ff5a77fe640 1 --2- 172.21.15.90:0/1914756555 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5577bdaa34d0 0x5577bde64420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff59c009230 tx=0x7ff59c032f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69f9fdd18f29554 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:21.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.932+0000 7ff5acfaa640 1 --2- 172.21.15.90:0/1417334969 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff5900afc50 0x7ff5900b2160 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff5a00dcca0 tx=0x7ff5a006da60 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:21.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.932+0000 7ff58e7fc640 1 -- 172.21.15.90:0/1914756555 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59c035070 con 0x5577bdaa34d0 2024-04-25T11:42:21.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.932+0000 7ff58e7fc640 1 -- 172.21.15.90:0/1914756555 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff59c0318a0 con 0x5577bdaa34d0 2024-04-25T11:42:21.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.932+0000 7ff58e7fc640 1 -- 172.21.15.90:0/1914756555 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59c004a00 con 0x5577bdaa34d0 2024-04-25T11:42:21.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.933+0000 7ff5b5995740 1 -- 172.21.15.90:0/1914756555 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5577bdaa34d0 msgr2=0x5577bde64420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:21.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.933+0000 7ff5b5995740 1 --2- 172.21.15.90:0/1914756555 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5577bdaa34d0 0x5577bde64420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff59c009230 tx=0x7ff59c032f40 comp rx=0 tx=0).stop 2024-04-25T11:42:21.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.933+0000 7ff5b5995740 1 -- 172.21.15.90:0/1914756555 shutdown_connections 2024-04-25T11:42:21.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.933+0000 7ff5b5995740 1 --2- 172.21.15.90:0/1914756555 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5577bdaa34d0 0x5577bde64420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.928 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.933+0000 7ff5b5995740 1 --2- 172.21.15.90:0/1914756555 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5577bda9b2f0 0x5577bde63ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.928 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.933+0000 7ff5b5995740 1 --2- 172.21.15.90:0/1914756555 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5577bdac41b0 0x5577bde639a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.928 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.933+0000 7ff5b5995740 1 -- 172.21.15.90:0/1914756555 >> 172.21.15.90:0/1914756555 conn(0x5577bdab8300 msgr2=0x5577bde602a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:21.928 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.933+0000 7ff5b5995740 1 -- 172.21.15.90:0/1914756555 shutdown_connections 2024-04-25T11:42:21.928 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.933+0000 7ff5b5995740 1 -- 172.21.15.90:0/1914756555 wait complete. 2024-04-25T11:42:21.928 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.934+0000 7ff5b5995740 1 -- start start 2024-04-25T11:42:21.928 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.934+0000 7ff5b5995740 1 Processor -- start 2024-04-25T11:42:21.928 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.934+0000 7ff5b5995740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5577bda9b2f0 0x5577bde74000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:21.928 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.934+0000 7ff5b5995740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5577bdaa34d0 0x5577bde74540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:21.928 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.934+0000 7ff5b5995740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5577bdac41b0 0x5577bde72480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:21.928 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.934+0000 7ff5b5995740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5577bde74fd0 con 0x5577bda9b2f0 2024-04-25T11:42:21.928 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.934+0000 7ff5b5995740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5577bdd51b60 con 0x5577bdaa34d0 2024-04-25T11:42:21.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.934+0000 7ff5b5995740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5577bde75150 con 0x5577bdac41b0 2024-04-25T11:42:21.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.934+0000 7ff5b5995740 10 client.0 _ll_register_callbacks cb 0x5577bdd762b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:21.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.934+0000 7ff5a77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5577bda9b2f0 0x5577bde74000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:21.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.934+0000 7ff5a77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5577bda9b2f0 0x5577bde74000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40250/0 (socket says 172.21.15.90:40250) 2024-04-25T11:42:21.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.934+0000 7ff5a77fe640 1 -- 172.21.15.90:0/283886450 learned_addr learned my addr 172.21.15.90:0/283886450 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:21.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.934+0000 7ff5a7fff640 1 --2- 172.21.15.90:0/283886450 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5577bdaa34d0 0x5577bde74540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:21.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.934+0000 7ff5a77fe640 1 --2- 172.21.15.90:0/283886450 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5577bdac41b0 0x5577bde72480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:21.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.935+0000 7ff5a77fe640 1 -- 172.21.15.90:0/283886450 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5577bdac41b0 msgr2=0x5577bde72480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:21.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.935+0000 7ff5a77fe640 1 --2- 172.21.15.90:0/283886450 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5577bdac41b0 0x5577bde72480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.935+0000 7ff5a77fe640 1 -- 172.21.15.90:0/283886450 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5577bdaa34d0 msgr2=0x5577bde74540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:21.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.935+0000 7ff5a77fe640 1 --2- 172.21.15.90:0/283886450 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5577bdaa34d0 0x5577bde74540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.935+0000 7ff5a77fe640 1 -- 172.21.15.90:0/283886450 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577bdadc210 con 0x5577bda9b2f0 2024-04-25T11:42:21.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.935+0000 7ff5a7fff640 1 --2- 172.21.15.90:0/283886450 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5577bdaa34d0 0x5577bde74540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:42:21.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.935+0000 7ff5a77fe640 1 --2- 172.21.15.90:0/283886450 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5577bda9b2f0 0x5577bde74000 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7ff59c005620 tx=0x7ff59c005520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d57bc9cb8e0ee2cb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.935+0000 7ff5877fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.935+0000 7ff5877fe640 1 -- 172.21.15.90:0/283886450 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59c050070 con 0x5577bda9b2f0 2024-04-25T11:42:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.935+0000 7ff5877fe640 1 -- 172.21.15.90:0/283886450 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7ff59c004240 con 0x5577bda9b2f0 2024-04-25T11:42:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.935+0000 7ff5b5995740 1 -- 172.21.15.90:0/283886450 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff59c0318a0 con 0x5577bda9b2f0 2024-04-25T11:42:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.935+0000 7ff5877fe640 1 -- 172.21.15.90:0/283886450 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff59c05c050 con 0x5577bda9b2f0 2024-04-25T11:42:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.935+0000 7ff5b5995740 10 client.17808 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.935+0000 7ff5b5995740 1 -- 172.21.15.90:0/283886450 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5577bdba8620 con 0x5577bda9b2f0 2024-04-25T11:42:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.935+0000 7ff5877fe640 1 -- 172.21.15.90:0/283886450 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff59c0318a0 con 0x5577bda9b2f0 2024-04-25T11:42:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.935+0000 7ff5877fe640 1 -- 172.21.15.90:0/283886450 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7ff59c0038b0 con 0x5577bda9b2f0 2024-04-25T11:42:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.935+0000 7ff5b5995740 10 client.17808 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.935+0000 7ff5b5995740 10 client.17808 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:42:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.936+0000 7ff5b5995740 20 client.17808 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.936+0000 7ff5b5995740 1 --2- 172.21.15.90:0/283886450 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5577bda82d80 0x5577bde849f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.936+0000 7ff5b5995740 4 client.17808 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:42:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.936+0000 7ff5b5995740 1 -- 172.21.15.90:0/283886450 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5577bddd8070 con 0x5577bda82d80 2024-04-25T11:42:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.936+0000 7ff5a7fff640 1 --2- 172.21.15.90:0/283886450 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5577bda82d80 0x5577bde849f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.936+0000 7ff5877fe640 10 client.17808 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.936+0000 7ff5a7fff640 1 --2- 172.21.15.90:0/283886450 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5577bda82d80 0x5577bde849f0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ea233a6fb521668 server_cookie=a70a6c8a9e91bd1 in_seq=0 out_seq=0 2024-04-25T11:42:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.937+0000 7ff5877fe640 1 -- 172.21.15.90:0/283886450 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5577bddd8070 con 0x5577bda82d80 2024-04-25T11:42:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.937+0000 7ff5877fe640 10 client.17808 handle_command_reply: tid=0 2024-04-25T11:42:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.937+0000 7ff5b5995740 1 client.17808 shutdown 2024-04-25T11:42:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.937+0000 7ff5b5995740 1 -- 172.21.15.90:0/283886450 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5577bda9b2f0 msgr2=0x5577bde74000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.937+0000 7ff5b5995740 1 --2- 172.21.15.90:0/283886450 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5577bda9b2f0 0x5577bde74000 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7ff59c005620 tx=0x7ff59c005520 comp rx=0 tx=0).stop 2024-04-25T11:42:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.937+0000 7ff5b5995740 1 -- 172.21.15.90:0/283886450 shutdown_connections 2024-04-25T11:42:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.937+0000 7ff5b5995740 1 --2- 172.21.15.90:0/283886450 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5577bda82d80 0x5577bde849f0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.937+0000 7ff5b5995740 1 --2- 172.21.15.90:0/283886450 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5577bdac41b0 0x5577bde72480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.937+0000 7ff5b5995740 1 --2- 172.21.15.90:0/283886450 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5577bdaa34d0 0x5577bde74540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.937+0000 7ff5877fe640 0 client.17808 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.937+0000 7ff5b5995740 1 --2- 172.21.15.90:0/283886450 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5577bda9b2f0 0x5577bde74000 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.937+0000 7ff5b5995740 1 -- 172.21.15.90:0/283886450 >> 172.21.15.90:0/283886450 conn(0x5577bdab8300 msgr2=0x5577bde657a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.937+0000 7ff5b5995740 1 -- 172.21.15.90:0/283886450 shutdown_connections 2024-04-25T11:42:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.937+0000 7ff5b5995740 1 -- 172.21.15.90:0/283886450 wait complete. 2024-04-25T11:42:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.937+0000 7ff5b5995740 20 client.17808 trim_cache size 0 max 16384 2024-04-25T11:42:21.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.953+0000 7ff5b5995740 1 Processor -- start 2024-04-25T11:42:21.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.953+0000 7ff5b5995740 1 -- start start 2024-04-25T11:42:21.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.953+0000 7ff5b5995740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5577bda82d80 0x5577bde65eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:21.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.953+0000 7ff5b5995740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5577bda9b2f0 0x5577bde6bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:21.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.953+0000 7ff5b5995740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5577bdaa34d0 0x5577bde6c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:21.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.953+0000 7ff5b5995740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a81ada30 con 0x5577bda82d80 2024-04-25T11:42:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.953+0000 7ff5b5995740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a81af570 con 0x5577bdaa34d0 2024-04-25T11:42:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.953+0000 7ff5b5995740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a81ae100 con 0x5577bda9b2f0 2024-04-25T11:42:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.953+0000 7ff5a77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5577bda82d80 0x5577bde65eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.953+0000 7ff5a77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5577bda82d80 0x5577bde65eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40254/0 (socket says 172.21.15.90:40254) 2024-04-25T11:42:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.953+0000 7ff5a77fe640 1 -- 172.21.15.90:0/2116178194 learned_addr learned my addr 172.21.15.90:0/2116178194 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.953+0000 7ff5a77fe640 1 --2- 172.21.15.90:0/2116178194 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5577bdaa34d0 0x5577bde6c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.953+0000 7ff5a7fff640 1 --2- 172.21.15.90:0/2116178194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5577bda9b2f0 0x5577bde6bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.953+0000 7ff5a77fe640 1 -- 172.21.15.90:0/2116178194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5577bda9b2f0 msgr2=0x5577bde6bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.953+0000 7ff5a77fe640 1 --2- 172.21.15.90:0/2116178194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5577bda9b2f0 0x5577bde6bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.953+0000 7ff5a77fe640 1 -- 172.21.15.90:0/2116178194 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5577bdaa34d0 msgr2=0x5577bde6c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.953+0000 7ff5a77fe640 1 --2- 172.21.15.90:0/2116178194 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5577bdaa34d0 0x5577bde6c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.953+0000 7ff5a77fe640 1 -- 172.21.15.90:0/2116178194 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577bde66720 con 0x5577bda82d80 2024-04-25T11:42:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.953+0000 7ff5a77fe640 1 --2- 172.21.15.90:0/2116178194 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5577bda82d80 0x5577bde65eb0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7ff59c02a930 tx=0x7ff59c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa32bab6a8381806 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:21.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.954+0000 7ff58dffb640 1 -- 172.21.15.90:0/2116178194 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59c050050 con 0x5577bda82d80 2024-04-25T11:42:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.954+0000 7ff58dffb640 1 -- 172.21.15.90:0/2116178194 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff59c0040f0 con 0x5577bda82d80 2024-04-25T11:42:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.954+0000 7ff58dffb640 1 -- 172.21.15.90:0/2116178194 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59c035040 con 0x5577bda82d80 2024-04-25T11:42:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.954+0000 7ff5b5995740 1 -- 172.21.15.90:0/2116178194 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5577bda82d80 msgr2=0x5577bde65eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.954+0000 7ff5b5995740 1 --2- 172.21.15.90:0/2116178194 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5577bda82d80 0x5577bde65eb0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7ff59c02a930 tx=0x7ff59c000bc0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.954+0000 7ff5b5995740 1 -- 172.21.15.90:0/2116178194 shutdown_connections 2024-04-25T11:42:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.954+0000 7ff5b5995740 1 --2- 172.21.15.90:0/2116178194 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5577bdaa34d0 0x5577bde6c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.954+0000 7ff5b5995740 1 --2- 172.21.15.90:0/2116178194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5577bda9b2f0 0x5577bde6bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.954+0000 7ff5b5995740 1 --2- 172.21.15.90:0/2116178194 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5577bda82d80 0x5577bde65eb0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.954+0000 7ff5b5995740 1 -- 172.21.15.90:0/2116178194 >> 172.21.15.90:0/2116178194 conn(0x5577bdab8300 msgr2=0x5577bde657a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.954+0000 7ff5b5995740 1 -- 172.21.15.90:0/2116178194 shutdown_connections 2024-04-25T11:42:21.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.954+0000 7ff5b5995740 1 -- 172.21.15.90:0/2116178194 wait complete. 2024-04-25T11:42:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.955+0000 7ff5b5995740 1 -- start start 2024-04-25T11:42:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.955+0000 7ff5b5995740 1 Processor -- start 2024-04-25T11:42:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.955+0000 7ff5b5995740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5577bda82d80 0x5577bde84e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.955+0000 7ff5b5995740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5577bda9b2f0 0x5577bde853b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.955+0000 7ff5b5995740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5577bdaa34d0 0x5577bdee4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.955+0000 7ff5b5995740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a81aeea0 con 0x5577bda82d80 2024-04-25T11:42:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.955+0000 7ff5b5995740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a81ae7d0 con 0x5577bdaa34d0 2024-04-25T11:42:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.955+0000 7ff5b5995740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a81aba30 con 0x5577bda9b2f0 2024-04-25T11:42:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.955+0000 7ff5b5995740 10 client.0 _ll_register_callbacks cb 0x7ff598009470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.955+0000 7ff5a77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5577bda82d80 0x5577bde84e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.955+0000 7ff5a77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5577bda82d80 0x5577bde84e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40266/0 (socket says 172.21.15.90:40266) 2024-04-25T11:42:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.955+0000 7ff5a77fe640 1 -- 172.21.15.90:0/3068354456 learned_addr learned my addr 172.21.15.90:0/3068354456 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:21.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.956+0000 7ff5a77fe640 1 --2- 172.21.15.90:0/3068354456 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5577bdaa34d0 0x5577bdee4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:21.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.956+0000 7ff5a7fff640 1 --2- 172.21.15.90:0/3068354456 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5577bda9b2f0 0x5577bde853b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:21.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.956+0000 7ff5a77fe640 1 -- 172.21.15.90:0/3068354456 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5577bda9b2f0 msgr2=0x5577bde853b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:21.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.956+0000 7ff5a77fe640 1 --2- 172.21.15.90:0/3068354456 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5577bda9b2f0 0x5577bde853b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.956+0000 7ff5a77fe640 1 -- 172.21.15.90:0/3068354456 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5577bdaa34d0 msgr2=0x5577bdee4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:21.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.956+0000 7ff5a77fe640 1 --2- 172.21.15.90:0/3068354456 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5577bdaa34d0 0x5577bdee4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.956+0000 7ff5a77fe640 1 -- 172.21.15.90:0/3068354456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577bdee5050 con 0x5577bda82d80 2024-04-25T11:42:21.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.956+0000 7ff5a77fe640 1 --2- 172.21.15.90:0/3068354456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5577bda82d80 0x5577bde84e70 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7ff59c032ed0 tx=0x7ff59c003850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5647e8d33ceee6bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:21.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.956+0000 7ff5877fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:21.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.956+0000 7ff5877fe640 1 -- 172.21.15.90:0/3068354456 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59c035070 con 0x5577bda82d80 2024-04-25T11:42:21.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.956+0000 7ff5877fe640 1 -- 172.21.15.90:0/3068354456 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7ff59c0314a0 con 0x5577bda82d80 2024-04-25T11:42:21.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.956+0000 7ff5b5995740 1 -- 172.21.15.90:0/3068354456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff59c0040f0 con 0x5577bda82d80 2024-04-25T11:42:21.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.956+0000 7ff5877fe640 1 -- 172.21.15.90:0/3068354456 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff59c050040 con 0x5577bda82d80 2024-04-25T11:42:21.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.956+0000 7ff5b5995740 10 client.17814 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:21.949 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.956+0000 7ff5b5995740 1 -- 172.21.15.90:0/3068354456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5577bdeec2d0 con 0x5577bda82d80 2024-04-25T11:42:21.949 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.957+0000 7ff5877fe640 1 -- 172.21.15.90:0/3068354456 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff59c0040f0 con 0x5577bda82d80 2024-04-25T11:42:21.949 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.957+0000 7ff5877fe640 1 -- 172.21.15.90:0/3068354456 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7ff59c004c20 con 0x5577bda82d80 2024-04-25T11:42:21.949 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.957+0000 7ff5b5995740 10 client.17814 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:21.949 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.957+0000 7ff5b5995740 10 client.17814 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:42:21.949 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.957+0000 7ff5b5995740 20 client.17814 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:21.949 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.957+0000 7ff5b5995740 1 --2- 172.21.15.90:0/3068354456 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5577bdac41b0 0x5577bdeee9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:21.949 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.957+0000 7ff5b5995740 4 client.17814 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:42:21.949 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.957+0000 7ff5b5995740 1 -- 172.21.15.90:0/3068354456 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5577bddd1ee0 con 0x5577bdac41b0 2024-04-25T11:42:21.949 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.957+0000 7ff5a7fff640 1 --2- 172.21.15.90:0/3068354456 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5577bdac41b0 0x5577bdeee9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:21.949 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.957+0000 7ff5877fe640 10 client.17814 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:21.949 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:21.957+0000 7ff5a7fff640 1 --2- 172.21.15.90:0/3068354456 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5577bdac41b0 0x5577bdeee9a0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92873b008e4ab17d server_cookie=6b1ee49c45eb9f37 in_seq=0 out_seq=0 2024-04-25T11:42:22.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:21 smithi090 ceph-mon[29575]: pgmap v223: 97 pgs: 97 active+clean; 60 GiB data, 187 GiB used, 886 GiB / 1.0 TiB avail; 19 KiB/s rd, 204 MiB/s wr, 1.48k op/s 2024-04-25T11:42:22.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:21 smithi138 ceph-mon[32328]: pgmap v223: 97 pgs: 97 active+clean; 60 GiB data, 187 GiB used, 886 GiB / 1.0 TiB avail; 19 KiB/s rd, 204 MiB/s wr, 1.48k op/s 2024-04-25T11:42:22.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:21 smithi156 ceph-mon[33045]: pgmap v223: 97 pgs: 97 active+clean; 60 GiB data, 187 GiB used, 886 GiB / 1.0 TiB avail; 19 KiB/s rd, 204 MiB/s wr, 1.48k op/s 2024-04-25T11:42:23.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:22 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:42:23.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:22 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:42:23.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:22 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:42:23.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:22 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:42:23.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:22 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:42:23.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:22 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:42:23.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:22 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:42:23.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:22 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:42:23.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:22 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:42:23.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:22 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:42:23.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:22 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:42:23.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:22 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:42:24.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:23 smithi090 ceph-mon[29575]: pgmap v224: 97 pgs: 97 active+clean; 60 GiB data, 188 GiB used, 885 GiB / 1.0 TiB avail; 18 KiB/s rd, 261 MiB/s wr, 1.83k op/s 2024-04-25T11:42:24.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:23 smithi138 ceph-mon[32328]: pgmap v224: 97 pgs: 97 active+clean; 60 GiB data, 188 GiB used, 885 GiB / 1.0 TiB avail; 18 KiB/s rd, 261 MiB/s wr, 1.83k op/s 2024-04-25T11:42:24.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:23 smithi156 ceph-mon[33045]: pgmap v224: 97 pgs: 97 active+clean; 60 GiB data, 188 GiB used, 885 GiB / 1.0 TiB avail; 18 KiB/s rd, 261 MiB/s wr, 1.83k op/s 2024-04-25T11:42:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.500+0000 7ff5877fe640 1 -- 172.21.15.90:0/3068354456 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5577bddd1ee0 con 0x5577bdac41b0 2024-04-25T11:42:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.500+0000 7ff5877fe640 10 client.17814 handle_command_reply: tid=0 2024-04-25T11:42:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.500+0000 7ff5b5995740 1 client.17814 shutdown 2024-04-25T11:42:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.500+0000 7ff5b5995740 1 -- 172.21.15.90:0/3068354456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5577bda82d80 msgr2=0x5577bde84e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.500+0000 7ff5b5995740 1 --2- 172.21.15.90:0/3068354456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5577bda82d80 0x5577bde84e70 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7ff59c032ed0 tx=0x7ff59c003850 comp rx=0 tx=0).stop 2024-04-25T11:42:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.500+0000 7ff5b5995740 1 -- 172.21.15.90:0/3068354456 shutdown_connections 2024-04-25T11:42:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.500+0000 7ff5b5995740 1 --2- 172.21.15.90:0/3068354456 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5577bdac41b0 0x5577bdeee9a0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.500+0000 7ff5b5995740 1 --2- 172.21.15.90:0/3068354456 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5577bdaa34d0 0x5577bdee4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.500+0000 7ff5b5995740 1 --2- 172.21.15.90:0/3068354456 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5577bda9b2f0 0x5577bde853b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.500+0000 7ff5b5995740 1 --2- 172.21.15.90:0/3068354456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5577bda82d80 0x5577bde84e70 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.500+0000 7ff5877fe640 0 client.17814 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:25.488 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.500+0000 7ff5b5995740 1 -- 172.21.15.90:0/3068354456 >> 172.21.15.90:0/3068354456 conn(0x5577bdab8300 msgr2=0x5577bde719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:25.488 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.500+0000 7ff5b5995740 1 -- 172.21.15.90:0/3068354456 shutdown_connections 2024-04-25T11:42:25.488 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.500+0000 7ff5b5995740 1 -- 172.21.15.90:0/3068354456 wait complete. 2024-04-25T11:42:25.488 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.500+0000 7ff5b5995740 20 client.17814 trim_cache size 0 max 16384 2024-04-25T11:42:25.488 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:42:25.488 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:42:25.488 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:42:25.488 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:42:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.501+0000 7ff5aea0d640 1 -- 172.21.15.90:0/1417334969 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff5900afc50 msgr2=0x7ff5900b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.501+0000 7ff5aea0d640 1 --2- 172.21.15.90:0/1417334969 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff5900afc50 0x7ff5900b2160 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff5a00dcca0 tx=0x7ff5a006da60 comp rx=0 tx=0).stop 2024-04-25T11:42:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.501+0000 7ff5aea0d640 1 -- 172.21.15.90:0/1417334969 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff5a810a020 msgr2=0x7ff5a819ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.501+0000 7ff5aea0d640 1 --2- 172.21.15.90:0/1417334969 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff5a810a020 0x7ff5a819ef90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff5a000da10 tx=0x7ff5a000ded0 comp rx=0 tx=0).stop 2024-04-25T11:42:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.501+0000 7ff5aea0d640 1 -- 172.21.15.90:0/1417334969 shutdown_connections 2024-04-25T11:42:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.501+0000 7ff5aea0d640 1 --2- 172.21.15.90:0/1417334969 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff5900afc50 0x7ff5900b2160 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:25.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.501+0000 7ff5aea0d640 1 --2- 172.21.15.90:0/1417334969 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff5a810a020 0x7ff5a819ef90 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:25.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.501+0000 7ff5aea0d640 1 --2- 172.21.15.90:0/1417334969 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5a8109b80 0x7ff5a819acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:25.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.501+0000 7ff5aea0d640 1 --2- 172.21.15.90:0/1417334969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5a81042d0 0x7ff5a819a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:25.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.501+0000 7ff5aea0d640 1 -- 172.21.15.90:0/1417334969 >> 172.21.15.90:0/1417334969 conn(0x7ff5a8079270 msgr2=0x7ff5a8102270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:25.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.501+0000 7ff5aea0d640 1 -- 172.21.15.90:0/1417334969 shutdown_connections 2024-04-25T11:42:25.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:25.501+0000 7ff5aea0d640 1 -- 172.21.15.90:0/1417334969 wait complete. 2024-04-25T11:42:26.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:25 smithi090 ceph-mon[29575]: pgmap v225: 97 pgs: 97 active+clean; 60 GiB data, 188 GiB used, 885 GiB / 1.0 TiB avail; 18 KiB/s rd, 268 MiB/s wr, 1.86k op/s 2024-04-25T11:42:26.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:25 smithi138 ceph-mon[32328]: pgmap v225: 97 pgs: 97 active+clean; 60 GiB data, 188 GiB used, 885 GiB / 1.0 TiB avail; 18 KiB/s rd, 268 MiB/s wr, 1.86k op/s 2024-04-25T11:42:26.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:25 smithi156 ceph-mon[33045]: pgmap v225: 97 pgs: 97 active+clean; 60 GiB data, 188 GiB used, 885 GiB / 1.0 TiB avail; 18 KiB/s rd, 268 MiB/s wr, 1.86k op/s 2024-04-25T11:42:28.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:27 smithi090 ceph-mon[29575]: pgmap v226: 97 pgs: 97 active+clean; 60 GiB data, 185 GiB used, 888 GiB / 1.0 TiB avail; 22 KiB/s rd, 309 MiB/s wr, 2.02k op/s 2024-04-25T11:42:28.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:27 smithi138 ceph-mon[32328]: pgmap v226: 97 pgs: 97 active+clean; 60 GiB data, 185 GiB used, 888 GiB / 1.0 TiB avail; 22 KiB/s rd, 309 MiB/s wr, 2.02k op/s 2024-04-25T11:42:28.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:27 smithi156 ceph-mon[33045]: pgmap v226: 97 pgs: 97 active+clean; 60 GiB data, 185 GiB used, 888 GiB / 1.0 TiB avail; 22 KiB/s rd, 309 MiB/s wr, 2.02k op/s 2024-04-25T11:42:30.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:29 smithi090 ceph-mon[29575]: pgmap v227: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 22 KiB/s rd, 300 MiB/s wr, 1.93k op/s 2024-04-25T11:42:30.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:29 smithi138 ceph-mon[32328]: pgmap v227: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 22 KiB/s rd, 300 MiB/s wr, 1.93k op/s 2024-04-25T11:42:30.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:29 smithi156 ceph-mon[33045]: pgmap v227: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 22 KiB/s rd, 300 MiB/s wr, 1.93k op/s 2024-04-25T11:42:32.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:31 smithi090 ceph-mon[29575]: pgmap v228: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 29 KiB/s rd, 260 MiB/s wr, 1.69k op/s 2024-04-25T11:42:32.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:31 smithi138 ceph-mon[32328]: pgmap v228: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 29 KiB/s rd, 260 MiB/s wr, 1.69k op/s 2024-04-25T11:42:32.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:31 smithi156 ceph-mon[33045]: pgmap v228: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 29 KiB/s rd, 260 MiB/s wr, 1.69k op/s 2024-04-25T11:42:34.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:33 smithi090 ceph-mon[29575]: pgmap v229: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 14 KiB/s rd, 109 MiB/s wr, 686 op/s 2024-04-25T11:42:34.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:33 smithi138 ceph-mon[32328]: pgmap v229: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 14 KiB/s rd, 109 MiB/s wr, 686 op/s 2024-04-25T11:42:34.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:33 smithi156 ceph-mon[33045]: pgmap v229: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 14 KiB/s rd, 109 MiB/s wr, 686 op/s 2024-04-25T11:42:36.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:35 smithi090 ceph-mon[29575]: pgmap v230: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 14 KiB/s rd, 57 MiB/s wr, 373 op/s 2024-04-25T11:42:36.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:35 smithi138 ceph-mon[32328]: pgmap v230: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 14 KiB/s rd, 57 MiB/s wr, 373 op/s 2024-04-25T11:42:36.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:35 smithi156 ceph-mon[33045]: pgmap v230: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 14 KiB/s rd, 57 MiB/s wr, 373 op/s 2024-04-25T11:42:37.405 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:42:37.406 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:42:37.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.534+0000 7fa9b4964640 1 -- 172.21.15.90:0/2647997114 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9b0109f90 msgr2=0x7fa9b01020b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:37.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.534+0000 7fa9b4964640 1 --2- 172.21.15.90:0/2647997114 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9b0109f90 0x7fa9b01020b0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fa998009900 tx=0x7fa99802a410 comp rx=0 tx=0).stop 2024-04-25T11:42:37.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.534+0000 7fa9b4964640 1 -- 172.21.15.90:0/2647997114 shutdown_connections 2024-04-25T11:42:37.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.534+0000 7fa9b4964640 1 --2- 172.21.15.90:0/2647997114 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa9b0102720 0x7fa9b0110570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.534+0000 7fa9b4964640 1 --2- 172.21.15.90:0/2647997114 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9b0109f90 0x7fa9b01020b0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.534+0000 7fa9b4964640 1 --2- 172.21.15.90:0/2647997114 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9b0109af0 0x7fa9b0101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.534+0000 7fa9b4964640 1 -- 172.21.15.90:0/2647997114 >> 172.21.15.90:0/2647997114 conn(0x7fa9b0077880 msgr2=0x7fa9b0077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:37.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.534+0000 7fa9b4964640 1 -- 172.21.15.90:0/2647997114 shutdown_connections 2024-04-25T11:42:37.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.534+0000 7fa9b4964640 1 -- 172.21.15.90:0/2647997114 wait complete. 2024-04-25T11:42:37.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.535+0000 7fa9b4964640 1 Processor -- start 2024-04-25T11:42:37.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.535+0000 7fa9b4964640 1 -- start start 2024-04-25T11:42:37.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.535+0000 7fa9b4964640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9b0102720 0x7fa9b019a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.535+0000 7fa9b4964640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9b0109af0 0x7fa9b019ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.535+0000 7fa9b4964640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa9b0109f90 0x7fa9b019ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.535+0000 7fa9b4964640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b0113ce0 con 0x7fa9b0102720 2024-04-25T11:42:37.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.535+0000 7fa9b4964640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b0113b60 con 0x7fa9b0109f90 2024-04-25T11:42:37.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.535+0000 7fa9b4964640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b0113e60 con 0x7fa9b0109af0 2024-04-25T11:42:37.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.536+0000 7fa9ae59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9b0102720 0x7fa9b019a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.536+0000 7fa9ae59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9b0102720 0x7fa9b019a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60084/0 (socket says 172.21.15.90:60084) 2024-04-25T11:42:37.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.536+0000 7fa9ae59c640 1 -- 172.21.15.90:0/3018938907 learned_addr learned my addr 172.21.15.90:0/3018938907 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:37.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.536+0000 7fa9aed9d640 1 --2- 172.21.15.90:0/3018938907 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa9b0109f90 0x7fa9b019ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.536+0000 7fa9a5d9b640 1 --2- 172.21.15.90:0/3018938907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9b0109af0 0x7fa9b019ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.536+0000 7fa9ae59c640 1 -- 172.21.15.90:0/3018938907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9b0109af0 msgr2=0x7fa9b019ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:37.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.536+0000 7fa9ae59c640 1 --2- 172.21.15.90:0/3018938907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9b0109af0 0x7fa9b019ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.536+0000 7fa9ae59c640 1 -- 172.21.15.90:0/3018938907 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa9b0109f90 msgr2=0x7fa9b019ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:37.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.536+0000 7fa9ae59c640 1 --2- 172.21.15.90:0/3018938907 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa9b0109f90 0x7fa9b019ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.536+0000 7fa9ae59c640 1 -- 172.21.15.90:0/3018938907 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b019f4e0 con 0x7fa9b0102720 2024-04-25T11:42:37.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.536+0000 7fa9ae59c640 1 --2- 172.21.15.90:0/3018938907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9b0102720 0x7fa9b019a700 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fa9a000c9a0 tx=0x7fa9a000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:37.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.536+0000 7fa9a7fff640 1 -- 172.21.15.90:0/3018938907 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0007bb0 con 0x7fa9b0102720 2024-04-25T11:42:37.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.536+0000 7fa9b4964640 1 -- 172.21.15.90:0/3018938907 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9b019f7d0 con 0x7fa9b0102720 2024-04-25T11:42:37.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.536+0000 7fa9a7fff640 1 -- 172.21.15.90:0/3018938907 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9a0007d50 con 0x7fa9b0102720 2024-04-25T11:42:37.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.536+0000 7fa9a7fff640 1 -- 172.21.15.90:0/3018938907 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a000f450 con 0x7fa9b0102720 2024-04-25T11:42:37.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.536+0000 7fa9b4964640 1 -- 172.21.15.90:0/3018938907 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9b019fd20 con 0x7fa9b0102720 2024-04-25T11:42:37.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.539+0000 7fa9a7fff640 1 -- 172.21.15.90:0/3018938907 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fa9a00046c0 con 0x7fa9b0102720 2024-04-25T11:42:37.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.539+0000 7fa9bb811740 1 Processor -- start 2024-04-25T11:42:37.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.539+0000 7fa9bb811740 1 -- start start 2024-04-25T11:42:37.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.539+0000 7fa9bb811740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ab4eb65390 0x55ab4ef05a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.539+0000 7fa9bb811740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ab4eb3c4d0 0x55ab4ef05fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.539+0000 7fa9bb811740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb446b0 0x55ab4ef128f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.539+0000 7fa9bb811740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ab4ee6bd80 con 0x55ab4eb446b0 2024-04-25T11:42:37.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.539+0000 7fa9bb811740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ab4ee258b0 con 0x55ab4eb65390 2024-04-25T11:42:37.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.539+0000 7fa9bb811740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ab4ee67d40 con 0x55ab4eb3c4d0 2024-04-25T11:42:37.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.539+0000 7fa9a5d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb446b0 0x55ab4ef128f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.540+0000 7fa9a5d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb446b0 0x55ab4ef128f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60098/0 (socket says 172.21.15.90:60098) 2024-04-25T11:42:37.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.540+0000 7fa9a5d9b640 1 -- 172.21.15.90:0/3868073231 learned_addr learned my addr 172.21.15.90:0/3868073231 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:37.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.540+0000 7fa9aed9d640 1 --2- 172.21.15.90:0/3868073231 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ab4eb65390 0x55ab4ef05a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.540+0000 7fa9a7fff640 1 --2- 172.21.15.90:0/3018938907 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa99c0afc60 0x7fa99c0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.540+0000 7fa9ae59c640 1 --2- 172.21.15.90:0/3868073231 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ab4eb3c4d0 0x55ab4ef05fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.540+0000 7fa9a5d9b640 1 -- 172.21.15.90:0/3868073231 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ab4eb3c4d0 msgr2=0x55ab4ef05fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:37.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.540+0000 7fa9a5d9b640 1 --2- 172.21.15.90:0/3868073231 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ab4eb3c4d0 0x55ab4ef05fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.540+0000 7fa9a5d9b640 1 -- 172.21.15.90:0/3868073231 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ab4eb65390 msgr2=0x55ab4ef05a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:37.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.540+0000 7fa9a5d9b640 1 --2- 172.21.15.90:0/3868073231 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ab4eb65390 0x55ab4ef05a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.540+0000 7fa9a5d9b640 1 -- 172.21.15.90:0/3868073231 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab4edf6a40 con 0x55ab4eb446b0 2024-04-25T11:42:37.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.540+0000 7fa9aed9d640 1 --2- 172.21.15.90:0/3868073231 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ab4eb65390 0x55ab4ef05a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:42:37.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.540+0000 7fa9ae59c640 1 --2- 172.21.15.90:0/3868073231 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ab4eb3c4d0 0x55ab4ef05fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:42:37.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.540+0000 7fa9aed9d640 1 --2- 172.21.15.90:0/3018938907 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa99c0afc60 0x7fa99c0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.540+0000 7fa9a7fff640 1 -- 172.21.15.90:0/3018938907 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7fa9a00dee30 con 0x7fa9b0102720 2024-04-25T11:42:37.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.540+0000 7fa9a5d9b640 1 --2- 172.21.15.90:0/3868073231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb446b0 0x55ab4ef128f0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fa9980096c0 tx=0x7fa998032cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d7167236a97c0a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:37.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.540+0000 7fa9a4d99640 1 -- 172.21.15.90:0/3868073231 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99803f070 con 0x55ab4eb446b0 2024-04-25T11:42:37.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.540+0000 7fa9a4d99640 1 -- 172.21.15.90:0/3868073231 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9980316a0 con 0x55ab4eb446b0 2024-04-25T11:42:37.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.540+0000 7fa9a4d99640 1 -- 172.21.15.90:0/3868073231 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa998035070 con 0x55ab4eb446b0 2024-04-25T11:42:37.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.540+0000 7fa9aed9d640 1 --2- 172.21.15.90:0/3018938907 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa99c0afc60 0x7fa99c0b2170 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8007310 tx=0x7fa9a8004a40 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:37.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.541+0000 7fa9bb811740 1 -- 172.21.15.90:0/3868073231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb446b0 msgr2=0x55ab4ef128f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:37.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.541+0000 7fa9bb811740 1 --2- 172.21.15.90:0/3868073231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb446b0 0x55ab4ef128f0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fa9980096c0 tx=0x7fa998032cf0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.541+0000 7fa9bb811740 1 -- 172.21.15.90:0/3868073231 shutdown_connections 2024-04-25T11:42:37.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.541+0000 7fa9bb811740 1 --2- 172.21.15.90:0/3868073231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb446b0 0x55ab4ef128f0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.541+0000 7fa9bb811740 1 --2- 172.21.15.90:0/3868073231 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ab4eb3c4d0 0x55ab4ef05fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.541+0000 7fa9bb811740 1 --2- 172.21.15.90:0/3868073231 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ab4eb65390 0x55ab4ef05a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.541+0000 7fa9bb811740 1 -- 172.21.15.90:0/3868073231 >> 172.21.15.90:0/3868073231 conn(0x55ab4eb594e0 msgr2=0x55ab4ef02210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:37.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.541+0000 7fa9bb811740 1 -- 172.21.15.90:0/3868073231 shutdown_connections 2024-04-25T11:42:37.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.541+0000 7fa9bb811740 1 -- 172.21.15.90:0/3868073231 wait complete. 2024-04-25T11:42:37.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.542+0000 7fa9bb811740 1 -- start start 2024-04-25T11:42:37.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.542+0000 7fa9bb811740 1 Processor -- start 2024-04-25T11:42:37.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.542+0000 7fa9bb811740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ab4eb3c4d0 0x55ab4ef105b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.542+0000 7fa9bb811740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ab4eb446b0 0x55ab4ef10af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.542+0000 7fa9bb811740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb65390 0x55ab4ef1c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.542+0000 7fa9bb811740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ab4ef12f80 con 0x55ab4eb65390 2024-04-25T11:42:37.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.542+0000 7fa9bb811740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ab4edf2d10 con 0x55ab4eb446b0 2024-04-25T11:42:37.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.542+0000 7fa9bb811740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ab4ef13100 con 0x55ab4eb3c4d0 2024-04-25T11:42:37.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.542+0000 7fa9bb811740 10 client.0 _ll_register_callbacks cb 0x55ab4ee1f210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:37.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.542+0000 7fa9aed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb65390 0x55ab4ef1c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.542+0000 7fa9aed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb65390 0x55ab4ef1c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60108/0 (socket says 172.21.15.90:60108) 2024-04-25T11:42:37.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.542+0000 7fa9aed9d640 1 -- 172.21.15.90:0/4163746618 learned_addr learned my addr 172.21.15.90:0/4163746618 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:37.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.542+0000 7fa9ae59c640 1 --2- 172.21.15.90:0/4163746618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ab4eb3c4d0 0x55ab4ef105b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.542+0000 7fa9a5d9b640 1 --2- 172.21.15.90:0/4163746618 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ab4eb446b0 0x55ab4ef10af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.542+0000 7fa9aed9d640 1 -- 172.21.15.90:0/4163746618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ab4eb3c4d0 msgr2=0x55ab4ef105b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:37.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.542+0000 7fa9aed9d640 1 --2- 172.21.15.90:0/4163746618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ab4eb3c4d0 0x55ab4ef105b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.542+0000 7fa9aed9d640 1 -- 172.21.15.90:0/4163746618 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ab4eb446b0 msgr2=0x55ab4ef10af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:37.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.542+0000 7fa9aed9d640 1 --2- 172.21.15.90:0/4163746618 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ab4eb446b0 0x55ab4ef10af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.543+0000 7fa9aed9d640 1 -- 172.21.15.90:0/4163746618 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab4ec477d0 con 0x55ab4eb65390 2024-04-25T11:42:37.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.543+0000 7fa9ae59c640 1 --2- 172.21.15.90:0/4163746618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ab4eb3c4d0 0x55ab4ef105b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:42:37.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.543+0000 7fa9a5d9b640 1 --2- 172.21.15.90:0/4163746618 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ab4eb446b0 0x55ab4ef10af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:42:37.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.543+0000 7fa9aed9d640 1 --2- 172.21.15.90:0/4163746618 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb65390 0x55ab4ef1c400 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fa9a80077f0 tx=0x7fa9a8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7166c7e56001f568 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:37.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.543+0000 7fa995ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:37.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.543+0000 7fa995ffb640 1 -- 172.21.15.90:0/4163746618 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8027670 con 0x55ab4eb65390 2024-04-25T11:42:37.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.543+0000 7fa995ffb640 1 -- 172.21.15.90:0/4163746618 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7fa9a801e450 con 0x55ab4eb65390 2024-04-25T11:42:37.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.543+0000 7fa9bb811740 1 -- 172.21.15.90:0/4163746618 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9980316a0 con 0x55ab4eb65390 2024-04-25T11:42:37.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.543+0000 7fa995ffb640 1 -- 172.21.15.90:0/4163746618 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa9a8027810 con 0x55ab4eb65390 2024-04-25T11:42:37.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.543+0000 7fa9bb811740 10 client.17826 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:37.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.543+0000 7fa9bb811740 1 -- 172.21.15.90:0/4163746618 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ab4eb7d3f0 con 0x55ab4eb65390 2024-04-25T11:42:37.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.543+0000 7fa995ffb640 1 -- 172.21.15.90:0/4163746618 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9980316a0 con 0x55ab4eb65390 2024-04-25T11:42:37.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.543+0000 7fa995ffb640 1 -- 172.21.15.90:0/4163746618 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fa9a801b440 con 0x55ab4eb65390 2024-04-25T11:42:37.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.543+0000 7fa9bb811740 10 client.17826 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:37.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.543+0000 7fa9bb811740 10 client.17826 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:42:37.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.543+0000 7fa9bb811740 20 client.17826 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:37.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.543+0000 7fa9bb811740 1 --2- 172.21.15.90:0/4163746618 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ab4eb23f60 0x55ab4ef22680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.543+0000 7fa9bb811740 4 client.17826 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:42:37.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.543+0000 7fa9bb811740 1 -- 172.21.15.90:0/4163746618 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab4ee79990 con 0x55ab4eb23f60 2024-04-25T11:42:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.544+0000 7fa9ae59c640 1 --2- 172.21.15.90:0/4163746618 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ab4eb23f60 0x55ab4ef22680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.544+0000 7fa995ffb640 10 client.17826 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.544+0000 7fa9ae59c640 1 --2- 172.21.15.90:0/4163746618 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ab4eb23f60 0x55ab4ef22680 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e85a206348b90ef server_cookie=fd08aa9c13d33c6d in_seq=0 out_seq=0 2024-04-25T11:42:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.545+0000 7fa995ffb640 1 -- 172.21.15.90:0/4163746618 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ab4ee79990 con 0x55ab4eb23f60 2024-04-25T11:42:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.545+0000 7fa995ffb640 10 client.17826 handle_command_reply: tid=0 2024-04-25T11:42:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.545+0000 7fa9bb811740 1 client.17826 shutdown 2024-04-25T11:42:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.545+0000 7fa9bb811740 1 -- 172.21.15.90:0/4163746618 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb65390 msgr2=0x55ab4ef1c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.545+0000 7fa9bb811740 1 --2- 172.21.15.90:0/4163746618 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb65390 0x55ab4ef1c400 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fa9a80077f0 tx=0x7fa9a8004ab0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.545+0000 7fa9bb811740 1 -- 172.21.15.90:0/4163746618 shutdown_connections 2024-04-25T11:42:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.545+0000 7fa9bb811740 1 --2- 172.21.15.90:0/4163746618 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ab4eb23f60 0x55ab4ef22680 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.545+0000 7fa9bb811740 1 --2- 172.21.15.90:0/4163746618 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb65390 0x55ab4ef1c400 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.545+0000 7fa995ffb640 0 client.17826 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.545+0000 7fa9bb811740 1 --2- 172.21.15.90:0/4163746618 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ab4eb446b0 0x55ab4ef10af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.545+0000 7fa9bb811740 1 --2- 172.21.15.90:0/4163746618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ab4eb3c4d0 0x55ab4ef105b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.545+0000 7fa9bb811740 1 -- 172.21.15.90:0/4163746618 >> 172.21.15.90:0/4163746618 conn(0x55ab4eb594e0 msgr2=0x55ab4ef07780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:37.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.545+0000 7fa9bb811740 1 -- 172.21.15.90:0/4163746618 shutdown_connections 2024-04-25T11:42:37.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.545+0000 7fa9bb811740 1 -- 172.21.15.90:0/4163746618 wait complete. 2024-04-25T11:42:37.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.546+0000 7fa9bb811740 20 client.17826 trim_cache size 0 max 16384 2024-04-25T11:42:37.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.563+0000 7fa9bb811740 1 Processor -- start 2024-04-25T11:42:37.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.563+0000 7fa9bb811740 1 -- start start 2024-04-25T11:42:37.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.563+0000 7fa9bb811740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb23f60 0x55ab4ef183d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.563+0000 7fa9bb811740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ab4eb3c4d0 0x55ab4ef18910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.563+0000 7fa9bb811740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ab4eb446b0 0x55ab4ef07660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.563+0000 7fa9bb811740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b01afbf0 con 0x55ab4eb23f60 2024-04-25T11:42:37.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.563+0000 7fa9bb811740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b01b1730 con 0x55ab4eb3c4d0 2024-04-25T11:42:37.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.563+0000 7fa9bb811740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b01b02c0 con 0x55ab4eb446b0 2024-04-25T11:42:37.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.564+0000 7fa9ae59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb23f60 0x55ab4ef183d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.564+0000 7fa9ae59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb23f60 0x55ab4ef183d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60110/0 (socket says 172.21.15.90:60110) 2024-04-25T11:42:37.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.564+0000 7fa9ae59c640 1 -- 172.21.15.90:0/1214517915 learned_addr learned my addr 172.21.15.90:0/1214517915 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:37.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.564+0000 7fa9a5d9b640 1 --2- 172.21.15.90:0/1214517915 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ab4eb3c4d0 0x55ab4ef18910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.564+0000 7fa9ae59c640 1 -- 172.21.15.90:0/1214517915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ab4eb446b0 msgr2=0x55ab4ef07660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:37.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.564+0000 7fa9ae59c640 1 --2- 172.21.15.90:0/1214517915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ab4eb446b0 0x55ab4ef07660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.564+0000 7fa9ae59c640 1 -- 172.21.15.90:0/1214517915 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ab4eb3c4d0 msgr2=0x55ab4ef18910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:37.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.564+0000 7fa9ae59c640 1 --2- 172.21.15.90:0/1214517915 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ab4eb3c4d0 0x55ab4ef18910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.564+0000 7fa9ae59c640 1 -- 172.21.15.90:0/1214517915 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab4eb3d0f0 con 0x55ab4eb23f60 2024-04-25T11:42:37.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.564+0000 7fa9ae59c640 1 --2- 172.21.15.90:0/1214517915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb23f60 0x55ab4ef183d0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fa9a00dc550 tx=0x7fa9a00d5690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80e7c41bdc5b959 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:37.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.564+0000 7fa997fff640 1 -- 172.21.15.90:0/1214517915 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a001d040 con 0x55ab4eb23f60 2024-04-25T11:42:37.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.564+0000 7fa997fff640 1 -- 172.21.15.90:0/1214517915 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9a00ab2f0 con 0x55ab4eb23f60 2024-04-25T11:42:37.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.564+0000 7fa997fff640 1 -- 172.21.15.90:0/1214517915 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a00a2b10 con 0x55ab4eb23f60 2024-04-25T11:42:37.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.565+0000 7fa9bb811740 1 -- 172.21.15.90:0/1214517915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb23f60 msgr2=0x55ab4ef183d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:37.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.565+0000 7fa9bb811740 1 --2- 172.21.15.90:0/1214517915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb23f60 0x55ab4ef183d0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fa9a00dc550 tx=0x7fa9a00d5690 comp rx=0 tx=0).stop 2024-04-25T11:42:37.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.565+0000 7fa9bb811740 1 -- 172.21.15.90:0/1214517915 shutdown_connections 2024-04-25T11:42:37.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.565+0000 7fa9bb811740 1 --2- 172.21.15.90:0/1214517915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ab4eb446b0 0x55ab4ef07660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.565+0000 7fa9bb811740 1 --2- 172.21.15.90:0/1214517915 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ab4eb3c4d0 0x55ab4ef18910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.565+0000 7fa9bb811740 1 --2- 172.21.15.90:0/1214517915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb23f60 0x55ab4ef183d0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.565+0000 7fa9bb811740 1 -- 172.21.15.90:0/1214517915 >> 172.21.15.90:0/1214517915 conn(0x55ab4eb594e0 msgr2=0x55ab4ef057f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.565+0000 7fa9bb811740 1 -- 172.21.15.90:0/1214517915 shutdown_connections 2024-04-25T11:42:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.565+0000 7fa9bb811740 1 -- 172.21.15.90:0/1214517915 wait complete. 2024-04-25T11:42:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.565+0000 7fa9bb811740 1 -- start start 2024-04-25T11:42:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.565+0000 7fa9bb811740 1 Processor -- start 2024-04-25T11:42:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.566+0000 7fa9bb811740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ab4eb23f60 0x55ab4ef133a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.566+0000 7fa9bb811740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ab4eb3c4d0 0x55ab4ef138e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.566+0000 7fa9bb811740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb446b0 0x55ab4ef88990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.566+0000 7fa9bb811740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b01b1060 con 0x55ab4eb446b0 2024-04-25T11:42:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.566+0000 7fa9bb811740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b01b0990 con 0x55ab4eb3c4d0 2024-04-25T11:42:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.566+0000 7fa9bb811740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b01adbf0 con 0x55ab4eb23f60 2024-04-25T11:42:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.566+0000 7fa9bb811740 10 client.0 _ll_register_callbacks cb 0x7fa9a8007d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.566+0000 7fa9aed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb446b0 0x55ab4ef88990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.566+0000 7fa9aed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb446b0 0x55ab4ef88990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60124/0 (socket says 172.21.15.90:60124) 2024-04-25T11:42:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.566+0000 7fa9aed9d640 1 -- 172.21.15.90:0/1940029071 learned_addr learned my addr 172.21.15.90:0/1940029071 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.566+0000 7fa9a5d9b640 1 --2- 172.21.15.90:0/1940029071 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ab4eb3c4d0 0x55ab4ef138e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.566+0000 7fa9ae59c640 1 --2- 172.21.15.90:0/1940029071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ab4eb23f60 0x55ab4ef133a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.566+0000 7fa9aed9d640 1 -- 172.21.15.90:0/1940029071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ab4eb23f60 msgr2=0x55ab4ef133a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.566+0000 7fa9aed9d640 1 --2- 172.21.15.90:0/1940029071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ab4eb23f60 0x55ab4ef133a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.566+0000 7fa9aed9d640 1 -- 172.21.15.90:0/1940029071 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ab4eb3c4d0 msgr2=0x55ab4ef138e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.566+0000 7fa9aed9d640 1 --2- 172.21.15.90:0/1940029071 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ab4eb3c4d0 0x55ab4ef138e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.566+0000 7fa9aed9d640 1 -- 172.21.15.90:0/1940029071 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab4ee2c4b0 con 0x55ab4eb446b0 2024-04-25T11:42:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.566+0000 7fa9aed9d640 1 --2- 172.21.15.90:0/1940029071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb446b0 0x55ab4ef88990 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fa9a801b8b0 tx=0x7fa9a801b750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70857938d39ef124 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.566+0000 7fa995ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.566+0000 7fa995ffb640 1 -- 172.21.15.90:0/1940029071 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8007a20 con 0x55ab4eb446b0 2024-04-25T11:42:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.566+0000 7fa995ffb640 1 -- 172.21.15.90:0/1940029071 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7fa9a8004ab0 con 0x55ab4eb446b0 2024-04-25T11:42:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.566+0000 7fa9bb811740 1 -- 172.21.15.90:0/1940029071 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9a00ab2f0 con 0x55ab4eb446b0 2024-04-25T11:42:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.566+0000 7fa995ffb640 1 -- 172.21.15.90:0/1940029071 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa9a8004cf0 con 0x55ab4eb446b0 2024-04-25T11:42:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.566+0000 7fa9bb811740 10 client.17832 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.566+0000 7fa9bb811740 1 -- 172.21.15.90:0/1940029071 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ab4ef89d50 con 0x55ab4eb446b0 2024-04-25T11:42:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.567+0000 7fa995ffb640 1 -- 172.21.15.90:0/1940029071 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9a00ab2f0 con 0x55ab4eb446b0 2024-04-25T11:42:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.567+0000 7fa995ffb640 1 -- 172.21.15.90:0/1940029071 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fa9a8035070 con 0x55ab4eb446b0 2024-04-25T11:42:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.567+0000 7fa9bb811740 10 client.17832 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.567+0000 7fa9bb811740 10 client.17832 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:42:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.567+0000 7fa9bb811740 20 client.17832 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.567+0000 7fa9bb811740 1 --2- 172.21.15.90:0/1940029071 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ab4eb65390 0x55ab4ef8c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.567+0000 7fa9bb811740 4 client.17832 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:42:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.567+0000 7fa9bb811740 1 -- 172.21.15.90:0/1940029071 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ab4ee73800 con 0x55ab4eb65390 2024-04-25T11:42:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.567+0000 7fa9ae59c640 1 --2- 172.21.15.90:0/1940029071 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ab4eb65390 0x55ab4ef8c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.567+0000 7fa995ffb640 10 client.17832 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.567+0000 7fa9ae59c640 1 --2- 172.21.15.90:0/1940029071 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ab4eb65390 0x55ab4ef8c560 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=470e1310bdd715f server_cookie=47fd135f501286e6 in_seq=0 out_seq=0 2024-04-25T11:42:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.568+0000 7fa995ffb640 1 -- 172.21.15.90:0/1940029071 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ab4ee73800 con 0x55ab4eb65390 2024-04-25T11:42:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.568+0000 7fa995ffb640 10 client.17832 handle_command_reply: tid=0 2024-04-25T11:42:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.568+0000 7fa9bb811740 1 client.17832 shutdown 2024-04-25T11:42:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.568+0000 7fa9bb811740 1 -- 172.21.15.90:0/1940029071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb446b0 msgr2=0x55ab4ef88990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.568+0000 7fa9bb811740 1 --2- 172.21.15.90:0/1940029071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb446b0 0x55ab4ef88990 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fa9a801b8b0 tx=0x7fa9a801b750 comp rx=0 tx=0).stop 2024-04-25T11:42:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.568+0000 7fa9bb811740 1 -- 172.21.15.90:0/1940029071 shutdown_connections 2024-04-25T11:42:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.568+0000 7fa9bb811740 1 --2- 172.21.15.90:0/1940029071 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ab4eb65390 0x55ab4ef8c560 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.568+0000 7fa9bb811740 1 --2- 172.21.15.90:0/1940029071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4eb446b0 0x55ab4ef88990 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.568+0000 7fa9bb811740 1 --2- 172.21.15.90:0/1940029071 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ab4eb3c4d0 0x55ab4ef138e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.568+0000 7fa9bb811740 1 --2- 172.21.15.90:0/1940029071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ab4eb23f60 0x55ab4ef133a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.568+0000 7fa995ffb640 0 client.17832 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.568+0000 7fa9bb811740 1 -- 172.21.15.90:0/1940029071 >> 172.21.15.90:0/1940029071 conn(0x55ab4eb594e0 msgr2=0x55ab4ef07660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.568+0000 7fa9bb811740 1 -- 172.21.15.90:0/1940029071 shutdown_connections 2024-04-25T11:42:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.568+0000 7fa9bb811740 1 -- 172.21.15.90:0/1940029071 wait complete. 2024-04-25T11:42:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.568+0000 7fa9bb811740 20 client.17832 trim_cache size 0 max 16384 2024-04-25T11:42:37.564 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:42:37.564 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-25T11:42:37.564 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-25T11:42:37.564 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:42:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.568+0000 7fa9b4964640 1 -- 172.21.15.90:0/3018938907 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa99c0afc60 msgr2=0x7fa99c0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.568+0000 7fa9b4964640 1 --2- 172.21.15.90:0/3018938907 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa99c0afc60 0x7fa99c0b2170 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8007310 tx=0x7fa9a8004a40 comp rx=0 tx=0).stop 2024-04-25T11:42:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.568+0000 7fa9b4964640 1 -- 172.21.15.90:0/3018938907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9b0102720 msgr2=0x7fa9b019a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.568+0000 7fa9b4964640 1 --2- 172.21.15.90:0/3018938907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9b0102720 0x7fa9b019a700 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fa9a000c9a0 tx=0x7fa9a000ce60 comp rx=0 tx=0).stop 2024-04-25T11:42:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.568+0000 7fa9b4964640 1 -- 172.21.15.90:0/3018938907 shutdown_connections 2024-04-25T11:42:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.568+0000 7fa9b4964640 1 --2- 172.21.15.90:0/3018938907 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa99c0afc60 0x7fa99c0b2170 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.568+0000 7fa9b4964640 1 --2- 172.21.15.90:0/3018938907 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa9b0109f90 0x7fa9b019ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.568+0000 7fa9b4964640 1 --2- 172.21.15.90:0/3018938907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9b0109af0 0x7fa9b019ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.568+0000 7fa9b4964640 1 --2- 172.21.15.90:0/3018938907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9b0102720 0x7fa9b019a700 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.568+0000 7fa9b4964640 1 -- 172.21.15.90:0/3018938907 >> 172.21.15.90:0/3018938907 conn(0x7fa9b0077880 msgr2=0x7fa9b01051f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:37.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.569+0000 7fa9b4964640 1 -- 172.21.15.90:0/3018938907 shutdown_connections 2024-04-25T11:42:37.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.569+0000 7fa9b4964640 1 -- 172.21.15.90:0/3018938907 wait complete. 2024-04-25T11:42:37.567 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T11:42:37.567 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:42:37.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.689+0000 7fe786f9d640 1 -- 172.21.15.90:0/251384083 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7801072a0 msgr2=0x7fe7801076b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:37.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.689+0000 7fe786f9d640 1 --2- 172.21.15.90:0/251384083 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7801072a0 0x7fe7801076b0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fe770009900 tx=0x7fe77002a410 comp rx=0 tx=0).stop 2024-04-25T11:42:37.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.689+0000 7fe786f9d640 1 -- 172.21.15.90:0/251384083 shutdown_connections 2024-04-25T11:42:37.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.689+0000 7fe786f9d640 1 --2- 172.21.15.90:0/251384083 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe780107cd0 0x7fe780108160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.689+0000 7fe786f9d640 1 --2- 172.21.15.90:0/251384083 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7801072a0 0x7fe7801076b0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.689+0000 7fe786f9d640 1 --2- 172.21.15.90:0/251384083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe780102850 0x7fe780102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.689+0000 7fe786f9d640 1 -- 172.21.15.90:0/251384083 >> 172.21.15.90:0/251384083 conn(0x7fe7800fe080 msgr2=0x7fe7801004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:37.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.689+0000 7fe786f9d640 1 -- 172.21.15.90:0/251384083 shutdown_connections 2024-04-25T11:42:37.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.690+0000 7fe786f9d640 1 -- 172.21.15.90:0/251384083 wait complete. 2024-04-25T11:42:37.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.690+0000 7fe786f9d640 1 Processor -- start 2024-04-25T11:42:37.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.690+0000 7fe786f9d640 1 -- start start 2024-04-25T11:42:37.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.690+0000 7fe786f9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe780102850 0x7fe7801a6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.690+0000 7fe786f9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7801072a0 0x7fe7801a6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.690+0000 7fe786f9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe780107cd0 0x7fe7801a0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.690+0000 7fe786f9d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe78011a100 con 0x7fe7801072a0 2024-04-25T11:42:37.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.690+0000 7fe786f9d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe780119f80 con 0x7fe780102850 2024-04-25T11:42:37.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.690+0000 7fe786f9d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe78011a280 con 0x7fe780107cd0 2024-04-25T11:42:37.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.690+0000 7fe78579a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7801072a0 0x7fe7801a6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.691+0000 7fe78579a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7801072a0 0x7fe7801a6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60136/0 (socket says 172.21.15.90:60136) 2024-04-25T11:42:37.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.691+0000 7fe78579a640 1 -- 172.21.15.90:0/444633888 learned_addr learned my addr 172.21.15.90:0/444633888 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:37.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.691+0000 7fe785f9b640 1 --2- 172.21.15.90:0/444633888 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe780102850 0x7fe7801a6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.691+0000 7fe78679c640 1 --2- 172.21.15.90:0/444633888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe780107cd0 0x7fe7801a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.691+0000 7fe78579a640 1 -- 172.21.15.90:0/444633888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe780107cd0 msgr2=0x7fe7801a0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:37.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.691+0000 7fe78579a640 1 --2- 172.21.15.90:0/444633888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe780107cd0 0x7fe7801a0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.691+0000 7fe78579a640 1 -- 172.21.15.90:0/444633888 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe780102850 msgr2=0x7fe7801a6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:37.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.691+0000 7fe78579a640 1 --2- 172.21.15.90:0/444633888 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe780102850 0x7fe7801a6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.691+0000 7fe78579a640 1 -- 172.21.15.90:0/444633888 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7801a1020 con 0x7fe7801072a0 2024-04-25T11:42:37.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.691+0000 7fe78579a640 1 --2- 172.21.15.90:0/444633888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7801072a0 0x7fe7801a6cb0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fe770005940 tx=0x7fe770004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:37.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.691+0000 7fe776ffd640 1 -- 172.21.15.90:0/444633888 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77003f030 con 0x7fe7801072a0 2024-04-25T11:42:37.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.691+0000 7fe786f9d640 1 -- 172.21.15.90:0/444633888 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7801a12b0 con 0x7fe7801072a0 2024-04-25T11:42:37.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.691+0000 7fe776ffd640 1 -- 172.21.15.90:0/444633888 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe770003aa0 con 0x7fe7801072a0 2024-04-25T11:42:37.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.691+0000 7fe776ffd640 1 -- 172.21.15.90:0/444633888 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe770035070 con 0x7fe7801072a0 2024-04-25T11:42:37.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.691+0000 7fe786f9d640 1 -- 172.21.15.90:0/444633888 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7801a17a0 con 0x7fe7801072a0 2024-04-25T11:42:37.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.692+0000 7fe786f9d640 1 -- 172.21.15.90:0/444633888 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe78006a030 con 0x7fe7801072a0 2024-04-25T11:42:37.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.694+0000 7fe776ffd640 1 -- 172.21.15.90:0/444633888 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fe770042030 con 0x7fe7801072a0 2024-04-25T11:42:37.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.694+0000 7fe776ffd640 1 --2- 172.21.15.90:0/444633888 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe76c0afc50 0x7fe76c0b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.694+0000 7fe785f9b640 1 --2- 172.21.15.90:0/444633888 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe76c0afc50 0x7fe76c0b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.694+0000 7fe776ffd640 1 -- 172.21.15.90:0/444633888 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7fe7700ffad0 con 0x7fe7801072a0 2024-04-25T11:42:37.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.695+0000 7fe785f9b640 1 --2- 172.21.15.90:0/444633888 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe76c0afc50 0x7fe76c0b2160 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe7801a1ac0 tx=0x7fe77c005f90 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:37.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.698+0000 7fe776ffd640 1 -- 172.21.15.90:0/444633888 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe7700c82e0 con 0x7fe7801072a0 2024-04-25T11:42:37.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.911+0000 7fe786f9d640 1 -- 172.21.15.90:0/444633888 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7801ad680 con 0x7fe7801072a0 2024-04-25T11:42:37.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.912+0000 7fe776ffd640 1 -- 172.21.15.90:0/444633888 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11930 (secure 0 0 0) 0x7fe7700c7be0 con 0x7fe7801072a0 2024-04-25T11:42:37.899 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:42:37.899 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:39:20.469254+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24458,24431,24499,24475,24524]},"id":1}]} 2024-04-25T11:42:37.900 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 28 2024-04-25T11:42:37.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.914+0000 7fe786f9d640 1 -- 172.21.15.90:0/444633888 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe76c0afc50 msgr2=0x7fe76c0b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:37.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.914+0000 7fe786f9d640 1 --2- 172.21.15.90:0/444633888 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe76c0afc50 0x7fe76c0b2160 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe7801a1ac0 tx=0x7fe77c005f90 comp rx=0 tx=0).stop 2024-04-25T11:42:37.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.914+0000 7fe786f9d640 1 -- 172.21.15.90:0/444633888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7801072a0 msgr2=0x7fe7801a6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:37.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.914+0000 7fe786f9d640 1 --2- 172.21.15.90:0/444633888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7801072a0 0x7fe7801a6cb0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fe770005940 tx=0x7fe770004270 comp rx=0 tx=0).stop 2024-04-25T11:42:37.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.914+0000 7fe786f9d640 1 -- 172.21.15.90:0/444633888 shutdown_connections 2024-04-25T11:42:37.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.914+0000 7fe786f9d640 1 --2- 172.21.15.90:0/444633888 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe76c0afc50 0x7fe76c0b2160 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.914+0000 7fe786f9d640 1 --2- 172.21.15.90:0/444633888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe780107cd0 0x7fe7801a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.914+0000 7fe786f9d640 1 --2- 172.21.15.90:0/444633888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7801072a0 0x7fe7801a6cb0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.914+0000 7fe786f9d640 1 --2- 172.21.15.90:0/444633888 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe780102850 0x7fe7801a6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.914+0000 7fe786f9d640 1 -- 172.21.15.90:0/444633888 >> 172.21.15.90:0/444633888 conn(0x7fe7800fe080 msgr2=0x7fe780105220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:37.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.914+0000 7fe786f9d640 1 -- 172.21.15.90:0/444633888 shutdown_connections 2024-04-25T11:42:37.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:37.914+0000 7fe786f9d640 1 -- 172.21.15.90:0/444633888 wait complete. 2024-04-25T11:42:37.913 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T11:42:38.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.037+0000 7efca44ee640 1 -- 172.21.15.90:0/1828862366 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc9c108a30 msgr2=0x7efc9c1025c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.037+0000 7efca44ee640 1 --2- 172.21.15.90:0/1828862366 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc9c108a30 0x7efc9c1025c0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7efc8c009900 tx=0x7efc8c02a410 comp rx=0 tx=0).stop 2024-04-25T11:42:38.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.037+0000 7efca44ee640 1 -- 172.21.15.90:0/1828862366 shutdown_connections 2024-04-25T11:42:38.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.037+0000 7efca44ee640 1 --2- 172.21.15.90:0/1828862366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efc9c10d390 0x7efc9c10f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.037+0000 7efca44ee640 1 --2- 172.21.15.90:0/1828862366 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efc9c102b00 0x7efc9c102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.037+0000 7efca44ee640 1 --2- 172.21.15.90:0/1828862366 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc9c108a30 0x7efc9c1025c0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.037+0000 7efca44ee640 1 -- 172.21.15.90:0/1828862366 >> 172.21.15.90:0/1828862366 conn(0x7efc9c0fe260 msgr2=0x7efc9c100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.037+0000 7efca44ee640 1 -- 172.21.15.90:0/1828862366 shutdown_connections 2024-04-25T11:42:38.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.037+0000 7efca44ee640 1 -- 172.21.15.90:0/1828862366 wait complete. 2024-04-25T11:42:38.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.038+0000 7efca44ee640 1 Processor -- start 2024-04-25T11:42:38.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.038+0000 7efca44ee640 1 -- start start 2024-04-25T11:42:38.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.038+0000 7efca44ee640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc9c102b00 0x7efc9c19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.038+0000 7efca44ee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efc9c108a30 0x7efc9c19acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.038+0000 7efca44ee640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efc9c10d390 0x7efc9c19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.038+0000 7efca44ee640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7efc9c112c40 con 0x7efc9c102b00 2024-04-25T11:42:38.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.038+0000 7efca44ee640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7efc9c112ac0 con 0x7efc9c10d390 2024-04-25T11:42:38.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.038+0000 7efca44ee640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7efc9c112dc0 con 0x7efc9c108a30 2024-04-25T11:42:38.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.038+0000 7efca228a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc9c102b00 0x7efc9c19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.038+0000 7efca228a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc9c102b00 0x7efc9c19a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60152/0 (socket says 172.21.15.90:60152) 2024-04-25T11:42:38.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.038+0000 7efca228a640 1 -- 172.21.15.90:0/2147789114 learned_addr learned my addr 172.21.15.90:0/2147789114 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.039+0000 7efca2a8b640 1 --2- 172.21.15.90:0/2147789114 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efc9c10d390 0x7efc9c19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.039+0000 7efca228a640 1 -- 172.21.15.90:0/2147789114 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efc9c108a30 msgr2=0x7efc9c19acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.039+0000 7efca228a640 1 --2- 172.21.15.90:0/2147789114 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efc9c108a30 0x7efc9c19acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.039+0000 7efca228a640 1 -- 172.21.15.90:0/2147789114 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efc9c10d390 msgr2=0x7efc9c19efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.039+0000 7efca228a640 1 --2- 172.21.15.90:0/2147789114 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efc9c10d390 0x7efc9c19efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.039+0000 7efca228a640 1 -- 172.21.15.90:0/2147789114 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc9c19f5f0 con 0x7efc9c102b00 2024-04-25T11:42:38.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.039+0000 7efca2a8b640 1 --2- 172.21.15.90:0/2147789114 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efc9c10d390 0x7efc9c19efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:42:38.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.039+0000 7efca228a640 1 --2- 172.21.15.90:0/2147789114 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc9c102b00 0x7efc9c19a790 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7efc8c031440 tx=0x7efc8c003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.039+0000 7efc937fe640 1 -- 172.21.15.90:0/2147789114 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8c03f070 con 0x7efc9c102b00 2024-04-25T11:42:38.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.039+0000 7efc937fe640 1 -- 172.21.15.90:0/2147789114 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc8c031d60 con 0x7efc9c102b00 2024-04-25T11:42:38.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.039+0000 7efca44ee640 1 -- 172.21.15.90:0/2147789114 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc9c19f880 con 0x7efc9c102b00 2024-04-25T11:42:38.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.039+0000 7efc937fe640 1 -- 172.21.15.90:0/2147789114 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8c035030 con 0x7efc9c102b00 2024-04-25T11:42:38.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.039+0000 7efca44ee640 1 -- 172.21.15.90:0/2147789114 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc9c19fd70 con 0x7efc9c102b00 2024-04-25T11:42:38.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.042+0000 7efcab3a0740 1 Processor -- start 2024-04-25T11:42:38.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.042+0000 7efcab3a0740 1 -- start start 2024-04-25T11:42:38.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.043+0000 7efcab3a0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85023d390 0x55f8505de810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.043+0000 7efcab3a0740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f8502144d0 0x55f8505ded50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.043+0000 7efcab3a0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85021c6b0 0x55f8505eb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.043+0000 7efcab3a0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f850543d80 con 0x55f85021c6b0 2024-04-25T11:42:38.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.043+0000 7efcab3a0740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f8504fda20 con 0x55f8502144d0 2024-04-25T11:42:38.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.043+0000 7efcab3a0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f85053fd40 con 0x55f85023d390 2024-04-25T11:42:38.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.043+0000 7efc937fe640 1 -- 172.21.15.90:0/2147789114 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7efc8c042070 con 0x7efc9c102b00 2024-04-25T11:42:38.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.043+0000 7efca1a89640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85021c6b0 0x55f8505eb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.043+0000 7efca1a89640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85021c6b0 0x55f8505eb6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60160/0 (socket says 172.21.15.90:60160) 2024-04-25T11:42:38.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.043+0000 7efca1a89640 1 -- 172.21.15.90:0/1084766718 learned_addr learned my addr 172.21.15.90:0/1084766718 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.043+0000 7efca2a8b640 1 --2- 172.21.15.90:0/1084766718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85023d390 0x55f8505de810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.043+0000 7efca228a640 1 --2- 172.21.15.90:0/1084766718 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f8502144d0 0x55f8505ded50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.043+0000 7efca1a89640 1 -- 172.21.15.90:0/1084766718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85023d390 msgr2=0x55f8505de810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.043+0000 7efca1a89640 1 --2- 172.21.15.90:0/1084766718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85023d390 0x55f8505de810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.043+0000 7efca1a89640 1 -- 172.21.15.90:0/1084766718 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f8502144d0 msgr2=0x55f8505ded50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.043+0000 7efca1a89640 1 --2- 172.21.15.90:0/1084766718 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f8502144d0 0x55f8505ded50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.043+0000 7efca1a89640 1 -- 172.21.15.90:0/1084766718 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f850508c30 con 0x55f85021c6b0 2024-04-25T11:42:38.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.043+0000 7efca228a640 1 --2- 172.21.15.90:0/1084766718 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f8502144d0 0x55f8505ded50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:42:38.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.043+0000 7efca1a89640 1 --2- 172.21.15.90:0/1084766718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85021c6b0 0x55f8505eb6a0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7efc9800a980 tx=0x7efc9800ae40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2148256b8a017903 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.044+0000 7efc90bfe640 1 -- 172.21.15.90:0/1084766718 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9800cd80 con 0x55f85021c6b0 2024-04-25T11:42:38.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.044+0000 7efc90bfe640 1 -- 172.21.15.90:0/1084766718 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc980054d0 con 0x55f85021c6b0 2024-04-25T11:42:38.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.044+0000 7efc90bfe640 1 -- 172.21.15.90:0/1084766718 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9800f6b0 con 0x55f85021c6b0 2024-04-25T11:42:38.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.044+0000 7efc937fe640 1 --2- 172.21.15.90:0/2147789114 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7efc880afc60 0x7efc880b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.044+0000 7efca2a8b640 1 --2- 172.21.15.90:0/2147789114 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7efc880afc60 0x7efc880b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.044+0000 7efc937fe640 1 -- 172.21.15.90:0/2147789114 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7efc8c0ffb20 con 0x7efc9c102b00 2024-04-25T11:42:38.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.045+0000 7efcab3a0740 1 -- 172.21.15.90:0/1084766718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85021c6b0 msgr2=0x55f8505eb6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.045+0000 7efca2a8b640 1 --2- 172.21.15.90:0/2147789114 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7efc880afc60 0x7efc880b2170 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x55f8505db980 tx=0x7efc94008630 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.045+0000 7efcab3a0740 1 --2- 172.21.15.90:0/1084766718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85021c6b0 0x55f8505eb6a0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7efc9800a980 tx=0x7efc9800ae40 comp rx=0 tx=0).stop 2024-04-25T11:42:38.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.045+0000 7efcab3a0740 1 -- 172.21.15.90:0/1084766718 shutdown_connections 2024-04-25T11:42:38.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.045+0000 7efcab3a0740 1 --2- 172.21.15.90:0/1084766718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85021c6b0 0x55f8505eb6a0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.045+0000 7efcab3a0740 1 --2- 172.21.15.90:0/1084766718 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f8502144d0 0x55f8505ded50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.045+0000 7efcab3a0740 1 --2- 172.21.15.90:0/1084766718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85023d390 0x55f8505de810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.045+0000 7efcab3a0740 1 -- 172.21.15.90:0/1084766718 >> 172.21.15.90:0/1084766718 conn(0x55f8502314e0 msgr2=0x55f8505dafc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.045+0000 7efcab3a0740 1 -- 172.21.15.90:0/1084766718 shutdown_connections 2024-04-25T11:42:38.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.045+0000 7efcab3a0740 1 -- 172.21.15.90:0/1084766718 wait complete. 2024-04-25T11:42:38.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.045+0000 7efcab3a0740 1 -- start start 2024-04-25T11:42:38.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.045+0000 7efcab3a0740 1 Processor -- start 2024-04-25T11:42:38.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.045+0000 7efcab3a0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8502144d0 0x55f8505e9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.046+0000 7efcab3a0740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85021c6b0 0x55f8505e97a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.046+0000 7efcab3a0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85023d390 0x55f8505f5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.046+0000 7efcab3a0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f8505ebd90 con 0x55f8502144d0 2024-04-25T11:42:38.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.046+0000 7efcab3a0740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f8504cadc0 con 0x55f85021c6b0 2024-04-25T11:42:38.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.046+0000 7efcab3a0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f8505ebf10 con 0x55f85023d390 2024-04-25T11:42:38.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.046+0000 7efcab3a0740 10 client.0 _ll_register_callbacks cb 0x55f8504e6600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:38.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.046+0000 7efca228a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8502144d0 0x55f8505e9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.046+0000 7efca228a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8502144d0 0x55f8505e9260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60168/0 (socket says 172.21.15.90:60168) 2024-04-25T11:42:38.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.046+0000 7efca228a640 1 -- 172.21.15.90:0/2982687715 learned_addr learned my addr 172.21.15.90:0/2982687715 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.046+0000 7efca1a89640 1 --2- 172.21.15.90:0/2982687715 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85021c6b0 0x55f8505e97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.046+0000 7efca2a8b640 1 --2- 172.21.15.90:0/2982687715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85023d390 0x55f8505f5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.046+0000 7efca228a640 1 -- 172.21.15.90:0/2982687715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85023d390 msgr2=0x55f8505f5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.046+0000 7efca228a640 1 --2- 172.21.15.90:0/2982687715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85023d390 0x55f8505f5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.046+0000 7efca228a640 1 -- 172.21.15.90:0/2982687715 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85021c6b0 msgr2=0x55f8505e97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.046+0000 7efca228a640 1 --2- 172.21.15.90:0/2982687715 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85021c6b0 0x55f8505e97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.046+0000 7efca228a640 1 -- 172.21.15.90:0/2982687715 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8502553f0 con 0x55f8502144d0 2024-04-25T11:42:38.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.046+0000 7efca2a8b640 1 --2- 172.21.15.90:0/2982687715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85023d390 0x55f8505f5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:42:38.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.046+0000 7efca228a640 1 --2- 172.21.15.90:0/2982687715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8502144d0 0x55f8505e9260 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7efc8c0fd1e0 tx=0x7efc8c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85cebda5f8d118c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.046+0000 7efc85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:38.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.046+0000 7efc85ffb640 1 -- 172.21.15.90:0/2982687715 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8c105060 con 0x55f8502144d0 2024-04-25T11:42:38.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.046+0000 7efc85ffb640 1 -- 172.21.15.90:0/2982687715 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7efc8c0c7e30 con 0x55f8502144d0 2024-04-25T11:42:38.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.046+0000 7efcab3a0740 1 -- 172.21.15.90:0/2982687715 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc980054d0 con 0x55f8502144d0 2024-04-25T11:42:38.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.046+0000 7efc85ffb640 1 -- 172.21.15.90:0/2982687715 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7efc8c049070 con 0x55f8502144d0 2024-04-25T11:42:38.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.046+0000 7efcab3a0740 10 client.17850 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:38.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.046+0000 7efcab3a0740 1 -- 172.21.15.90:0/2982687715 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f85031f7d0 con 0x55f8502144d0 2024-04-25T11:42:38.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.046+0000 7efc85ffb640 1 -- 172.21.15.90:0/2982687715 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc980054d0 con 0x55f8502144d0 2024-04-25T11:42:38.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.047+0000 7efc85ffb640 1 -- 172.21.15.90:0/2982687715 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7efc8c0d05a0 con 0x55f8502144d0 2024-04-25T11:42:38.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.047+0000 7efcab3a0740 10 client.17850 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:38.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.047+0000 7efcab3a0740 10 client.17850 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:42:38.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.047+0000 7efcab3a0740 20 client.17850 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:38.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.047+0000 7efcab3a0740 1 --2- 172.21.15.90:0/2982687715 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55f8501fbf60 0x55f8505fb550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.047+0000 7efcab3a0740 4 client.17850 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:42:38.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.047+0000 7efcab3a0740 1 -- 172.21.15.90:0/2982687715 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f850551990 con 0x55f8501fbf60 2024-04-25T11:42:38.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.047+0000 7efca1a89640 1 --2- 172.21.15.90:0/2982687715 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55f8501fbf60 0x55f8505fb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.048+0000 7efc85ffb640 10 client.17850 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:42:38.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.048+0000 7efca1a89640 1 --2- 172.21.15.90:0/2982687715 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55f8501fbf60 0x55f8505fb550 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=36103236ea6163a7 server_cookie=7803f36bb6b66492 in_seq=0 out_seq=0 2024-04-25T11:42:38.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.048+0000 7efc85ffb640 1 -- 172.21.15.90:0/2982687715 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f850551990 con 0x55f8501fbf60 2024-04-25T11:42:38.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.048+0000 7efc85ffb640 10 client.17850 handle_command_reply: tid=0 2024-04-25T11:42:38.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.048+0000 7efcab3a0740 1 client.17850 shutdown 2024-04-25T11:42:38.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.048+0000 7efcab3a0740 1 -- 172.21.15.90:0/2982687715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8502144d0 msgr2=0x55f8505e9260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.048+0000 7efcab3a0740 1 --2- 172.21.15.90:0/2982687715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8502144d0 0x55f8505e9260 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7efc8c0fd1e0 tx=0x7efc8c009260 comp rx=0 tx=0).stop 2024-04-25T11:42:38.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.048+0000 7efcab3a0740 1 -- 172.21.15.90:0/2982687715 shutdown_connections 2024-04-25T11:42:38.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.048+0000 7efcab3a0740 1 --2- 172.21.15.90:0/2982687715 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55f8501fbf60 0x55f8505fb550 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.048+0000 7efcab3a0740 1 --2- 172.21.15.90:0/2982687715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85023d390 0x55f8505f5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.048+0000 7efcab3a0740 1 --2- 172.21.15.90:0/2982687715 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85021c6b0 0x55f8505e97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.048+0000 7efc85ffb640 0 client.17850 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:42:38.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.048+0000 7efcab3a0740 1 --2- 172.21.15.90:0/2982687715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8502144d0 0x55f8505e9260 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.048+0000 7efcab3a0740 1 -- 172.21.15.90:0/2982687715 >> 172.21.15.90:0/2982687715 conn(0x55f8502314e0 msgr2=0x55f8505e0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.049+0000 7efcab3a0740 1 -- 172.21.15.90:0/2982687715 shutdown_connections 2024-04-25T11:42:38.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.049+0000 7efcab3a0740 1 -- 172.21.15.90:0/2982687715 wait complete. 2024-04-25T11:42:38.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.049+0000 7efcab3a0740 20 client.17850 trim_cache size 0 max 16384 2024-04-25T11:42:38.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.062+0000 7efcab3a0740 1 Processor -- start 2024-04-25T11:42:38.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.062+0000 7efcab3a0740 1 -- start start 2024-04-25T11:42:38.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.062+0000 7efcab3a0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8502144d0 0x55f8505daf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.062+0000 7efcab3a0740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85021c6b0 0x55f8505f4e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.063+0000 7efcab3a0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85023d390 0x55f850600ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.063+0000 7efcab3a0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7efc9c1afbe0 con 0x55f85023d390 2024-04-25T11:42:38.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.063+0000 7efcab3a0740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7efc9c1b1720 con 0x55f85021c6b0 2024-04-25T11:42:38.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.063+0000 7efcab3a0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7efc9c1b02b0 con 0x55f8502144d0 2024-04-25T11:42:38.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.063+0000 7efca2a8b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85023d390 0x55f850600ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.063+0000 7efca2a8b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85023d390 0x55f850600ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60172/0 (socket says 172.21.15.90:60172) 2024-04-25T11:42:38.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.063+0000 7efca2a8b640 1 -- 172.21.15.90:0/1739584787 learned_addr learned my addr 172.21.15.90:0/1739584787 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.063+0000 7efca1a89640 1 --2- 172.21.15.90:0/1739584787 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85021c6b0 0x55f8505f4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.063+0000 7efca228a640 1 --2- 172.21.15.90:0/1739584787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8502144d0 0x55f8505daf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.063+0000 7efca2a8b640 1 -- 172.21.15.90:0/1739584787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8502144d0 msgr2=0x55f8505daf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.063+0000 7efca2a8b640 1 --2- 172.21.15.90:0/1739584787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8502144d0 0x55f8505daf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.063+0000 7efca2a8b640 1 -- 172.21.15.90:0/1739584787 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85021c6b0 msgr2=0x55f8505f4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.063+0000 7efca2a8b640 1 --2- 172.21.15.90:0/1739584787 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85021c6b0 0x55f8505f4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.063+0000 7efca2a8b640 1 -- 172.21.15.90:0/1739584787 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8505dfd90 con 0x55f85023d390 2024-04-25T11:42:38.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.063+0000 7efca1a89640 1 --2- 172.21.15.90:0/1739584787 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85021c6b0 0x55f8505f4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:42:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.063+0000 7efca2a8b640 1 --2- 172.21.15.90:0/1739584787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85023d390 0x55f850600ce0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7efc9401b700 tx=0x7efc94013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcabd4fb77ab11eb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.063+0000 7efc87fff640 1 -- 172.21.15.90:0/1739584787 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9401b020 con 0x55f85023d390 2024-04-25T11:42:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.063+0000 7efc87fff640 1 -- 172.21.15.90:0/1739584787 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc9401b1c0 con 0x55f85023d390 2024-04-25T11:42:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.063+0000 7efc87fff640 1 -- 172.21.15.90:0/1739584787 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc94024c40 con 0x55f85023d390 2024-04-25T11:42:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.064+0000 7efcab3a0740 1 -- 172.21.15.90:0/1739584787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85023d390 msgr2=0x55f850600ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.064+0000 7efcab3a0740 1 --2- 172.21.15.90:0/1739584787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85023d390 0x55f850600ce0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7efc9401b700 tx=0x7efc94013000 comp rx=0 tx=0).stop 2024-04-25T11:42:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.064+0000 7efcab3a0740 1 -- 172.21.15.90:0/1739584787 shutdown_connections 2024-04-25T11:42:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.064+0000 7efcab3a0740 1 --2- 172.21.15.90:0/1739584787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f85023d390 0x55f850600ce0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.064+0000 7efcab3a0740 1 --2- 172.21.15.90:0/1739584787 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85021c6b0 0x55f8505f4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.064+0000 7efcab3a0740 1 --2- 172.21.15.90:0/1739584787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8502144d0 0x55f8505daf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.064+0000 7efcab3a0740 1 -- 172.21.15.90:0/1739584787 >> 172.21.15.90:0/1739584787 conn(0x55f8502314e0 msgr2=0x55f8505da800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.064+0000 7efcab3a0740 1 -- 172.21.15.90:0/1739584787 shutdown_connections 2024-04-25T11:42:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.064+0000 7efcab3a0740 1 -- 172.21.15.90:0/1739584787 wait complete. 2024-04-25T11:42:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.064+0000 7efcab3a0740 1 -- start start 2024-04-25T11:42:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.065+0000 7efcab3a0740 1 Processor -- start 2024-04-25T11:42:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.065+0000 7efcab3a0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8502144d0 0x55f8505f1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.065+0000 7efcab3a0740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85021c6b0 0x55f8505ec7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.065+0000 7efcab3a0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85023d390 0x55f8505eccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.065+0000 7efcab3a0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7efc9c1b1050 con 0x55f8502144d0 2024-04-25T11:42:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.065+0000 7efcab3a0740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7efc9c1b0980 con 0x55f85021c6b0 2024-04-25T11:42:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.065+0000 7efcab3a0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7efc9c1adbe0 con 0x55f85023d390 2024-04-25T11:42:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.065+0000 7efcab3a0740 10 client.0 _ll_register_callbacks cb 0x7efc8c1033c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.065+0000 7efca228a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8502144d0 0x55f8505f1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.065+0000 7efca228a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8502144d0 0x55f8505f1410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60180/0 (socket says 172.21.15.90:60180) 2024-04-25T11:42:38.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.065+0000 7efca228a640 1 -- 172.21.15.90:0/3530128254 learned_addr learned my addr 172.21.15.90:0/3530128254 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.065+0000 7efca1a89640 1 --2- 172.21.15.90:0/3530128254 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85021c6b0 0x55f8505ec7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.065+0000 7efca2a8b640 1 --2- 172.21.15.90:0/3530128254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85023d390 0x55f8505eccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.065+0000 7efca228a640 1 -- 172.21.15.90:0/3530128254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85023d390 msgr2=0x55f8505eccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.065+0000 7efca228a640 1 --2- 172.21.15.90:0/3530128254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85023d390 0x55f8505eccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.065+0000 7efca228a640 1 -- 172.21.15.90:0/3530128254 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85021c6b0 msgr2=0x55f8505ec7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.065+0000 7efca228a640 1 --2- 172.21.15.90:0/3530128254 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85021c6b0 0x55f8505ec7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.065+0000 7efca228a640 1 -- 172.21.15.90:0/3530128254 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8505ed230 con 0x55f8502144d0 2024-04-25T11:42:38.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.065+0000 7efca228a640 1 --2- 172.21.15.90:0/3530128254 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8502144d0 0x55f8505f1410 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7efc8c006010 tx=0x7efc8c0c8060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=704be74bcd4e7b67 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.065+0000 7efc85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:38.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.065+0000 7efc85ffb640 1 -- 172.21.15.90:0/3530128254 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8c0fd610 con 0x55f8502144d0 2024-04-25T11:42:38.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.066+0000 7efc85ffb640 1 -- 172.21.15.90:0/3530128254 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7efc8c031ac0 con 0x55f8502144d0 2024-04-25T11:42:38.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.066+0000 7efcab3a0740 1 -- 172.21.15.90:0/3530128254 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc9401b1c0 con 0x55f8502144d0 2024-04-25T11:42:38.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.066+0000 7efc85ffb640 1 -- 172.21.15.90:0/3530128254 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7efc8c105070 con 0x55f8502144d0 2024-04-25T11:42:38.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.066+0000 7efcab3a0740 10 client.17856 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:38.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.066+0000 7efcab3a0740 1 -- 172.21.15.90:0/3530128254 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f850663410 con 0x55f8502144d0 2024-04-25T11:42:38.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.066+0000 7efc85ffb640 1 -- 172.21.15.90:0/3530128254 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc9401b1c0 con 0x55f8502144d0 2024-04-25T11:42:38.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.066+0000 7efc85ffb640 1 -- 172.21.15.90:0/3530128254 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7efc8c0c3540 con 0x55f8502144d0 2024-04-25T11:42:38.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.066+0000 7efcab3a0740 10 client.17856 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:38.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.066+0000 7efcab3a0740 10 client.17856 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:42:38.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.066+0000 7efcab3a0740 20 client.17856 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:38.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.066+0000 7efcab3a0740 1 --2- 172.21.15.90:0/3530128254 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55f8501d13a0 0x55f850665ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.066+0000 7efcab3a0740 4 client.17856 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:42:38.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.066+0000 7efcab3a0740 1 -- 172.21.15.90:0/3530128254 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f85054b800 con 0x55f8501d13a0 2024-04-25T11:42:38.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.066+0000 7efca1a89640 1 --2- 172.21.15.90:0/3530128254 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55f8501d13a0 0x55f850665ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.067+0000 7efc85ffb640 10 client.17856 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:42:38.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.067+0000 7efca1a89640 1 --2- 172.21.15.90:0/3530128254 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55f8501d13a0 0x55f850665ac0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a0915be1cc0ba593 server_cookie=719444f1b4253f0e in_seq=0 out_seq=0 2024-04-25T11:42:38.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.067+0000 7efc85ffb640 1 -- 172.21.15.90:0/3530128254 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f85054b800 con 0x55f8501d13a0 2024-04-25T11:42:38.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.067+0000 7efc85ffb640 10 client.17856 handle_command_reply: tid=0 2024-04-25T11:42:38.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.067+0000 7efcab3a0740 1 client.17856 shutdown 2024-04-25T11:42:38.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.067+0000 7efcab3a0740 1 -- 172.21.15.90:0/3530128254 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8502144d0 msgr2=0x55f8505f1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.067+0000 7efcab3a0740 1 --2- 172.21.15.90:0/3530128254 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8502144d0 0x55f8505f1410 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7efc8c006010 tx=0x7efc8c0c8060 comp rx=0 tx=0).stop 2024-04-25T11:42:38.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.067+0000 7efcab3a0740 1 -- 172.21.15.90:0/3530128254 shutdown_connections 2024-04-25T11:42:38.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.067+0000 7efcab3a0740 1 --2- 172.21.15.90:0/3530128254 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55f8501d13a0 0x55f850665ac0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.067+0000 7efcab3a0740 1 --2- 172.21.15.90:0/3530128254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f85023d390 0x55f8505eccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.067+0000 7efcab3a0740 1 --2- 172.21.15.90:0/3530128254 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f85021c6b0 0x55f8505ec7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.067+0000 7efc85ffb640 0 client.17856 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:42:38.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.067+0000 7efcab3a0740 1 --2- 172.21.15.90:0/3530128254 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8502144d0 0x55f8505f1410 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.067+0000 7efcab3a0740 1 -- 172.21.15.90:0/3530128254 >> 172.21.15.90:0/3530128254 conn(0x55f8502314e0 msgr2=0x55f8505e0410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.067+0000 7efcab3a0740 1 -- 172.21.15.90:0/3530128254 shutdown_connections 2024-04-25T11:42:38.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.067+0000 7efcab3a0740 1 -- 172.21.15.90:0/3530128254 wait complete. 2024-04-25T11:42:38.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.067+0000 7efcab3a0740 20 client.17856 trim_cache size 0 max 16384 2024-04-25T11:42:38.062 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:42:38.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.068+0000 7efca44ee640 1 -- 172.21.15.90:0/2147789114 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7efc880afc60 msgr2=0x7efc880b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.068+0000 7efca44ee640 1 --2- 172.21.15.90:0/2147789114 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7efc880afc60 0x7efc880b2170 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x55f8505db980 tx=0x7efc94008630 comp rx=0 tx=0).stop 2024-04-25T11:42:38.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.068+0000 7efca44ee640 1 -- 172.21.15.90:0/2147789114 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc9c102b00 msgr2=0x7efc9c19a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.068+0000 7efca44ee640 1 --2- 172.21.15.90:0/2147789114 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc9c102b00 0x7efc9c19a790 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7efc8c031440 tx=0x7efc8c003d50 comp rx=0 tx=0).stop 2024-04-25T11:42:38.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.068+0000 7efca44ee640 1 -- 172.21.15.90:0/2147789114 shutdown_connections 2024-04-25T11:42:38.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.068+0000 7efca44ee640 1 --2- 172.21.15.90:0/2147789114 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7efc880afc60 0x7efc880b2170 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.068+0000 7efca44ee640 1 --2- 172.21.15.90:0/2147789114 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efc9c10d390 0x7efc9c19efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.068+0000 7efca44ee640 1 --2- 172.21.15.90:0/2147789114 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efc9c108a30 0x7efc9c19acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.068+0000 7efca44ee640 1 --2- 172.21.15.90:0/2147789114 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc9c102b00 0x7efc9c19a790 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.068+0000 7efca44ee640 1 -- 172.21.15.90:0/2147789114 >> 172.21.15.90:0/2147789114 conn(0x7efc9c0fe260 msgr2=0x7efc9c100a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.068+0000 7efca44ee640 1 -- 172.21.15.90:0/2147789114 shutdown_connections 2024-04-25T11:42:38.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.068+0000 7efca44ee640 1 -- 172.21.15.90:0/2147789114 wait complete. 2024-04-25T11:42:38.066 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T11:42:38.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.190+0000 7f950f842640 1 -- 172.21.15.90:0/3254775018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9508108a10 msgr2=0x7f9508102c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.190+0000 7f950f842640 1 --2- 172.21.15.90:0/3254775018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9508108a10 0x7f9508102c80 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f9504009900 tx=0x7f950402a410 comp rx=0 tx=0).stop 2024-04-25T11:42:38.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.190+0000 7f950f842640 1 -- 172.21.15.90:0/3254775018 shutdown_connections 2024-04-25T11:42:38.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.190+0000 7f950f842640 1 --2- 172.21.15.90:0/3254775018 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9508103830 0x7f950810f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.190+0000 7f950f842640 1 --2- 172.21.15.90:0/3254775018 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9508108eb0 0x7f95081031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.190+0000 7f950f842640 1 --2- 172.21.15.90:0/3254775018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9508108a10 0x7f9508102c80 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.190+0000 7f950f842640 1 -- 172.21.15.90:0/3254775018 >> 172.21.15.90:0/3254775018 conn(0x7f9508077880 msgr2=0x7f9508077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.190+0000 7f950f842640 1 -- 172.21.15.90:0/3254775018 shutdown_connections 2024-04-25T11:42:38.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.190+0000 7f950f842640 1 -- 172.21.15.90:0/3254775018 wait complete. 2024-04-25T11:42:38.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.190+0000 7f950f842640 1 Processor -- start 2024-04-25T11:42:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.191+0000 7f950f842640 1 -- start start 2024-04-25T11:42:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.191+0000 7f950f842640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9508103830 0x7f950819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.191+0000 7f950f842640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9508108a10 0x7f950819acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.191+0000 7f950f842640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9508108eb0 0x7f950819ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.191+0000 7f950f842640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9508112c00 con 0x7f9508108a10 2024-04-25T11:42:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.191+0000 7f950f842640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9508112a80 con 0x7f9508103830 2024-04-25T11:42:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.191+0000 7f950f842640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9508112d80 con 0x7f9508108eb0 2024-04-25T11:42:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.191+0000 7f950cddd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9508108a10 0x7f950819acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.191+0000 7f950cddd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9508108a10 0x7f950819acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60200/0 (socket says 172.21.15.90:60200) 2024-04-25T11:42:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.191+0000 7f950d5de640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9508103830 0x7f950819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.191+0000 7f950cddd640 1 -- 172.21.15.90:0/1444115914 learned_addr learned my addr 172.21.15.90:0/1444115914 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.191+0000 7f950dddf640 1 --2- 172.21.15.90:0/1444115914 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9508108eb0 0x7f950819ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.191+0000 7f950cddd640 1 -- 172.21.15.90:0/1444115914 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9508108eb0 msgr2=0x7f950819ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.191+0000 7f950cddd640 1 --2- 172.21.15.90:0/1444115914 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9508108eb0 0x7f950819ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.191+0000 7f950cddd640 1 -- 172.21.15.90:0/1444115914 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9508103830 msgr2=0x7f950819a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.191+0000 7f950cddd640 1 --2- 172.21.15.90:0/1444115914 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9508103830 0x7f950819a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.191+0000 7f950cddd640 1 -- 172.21.15.90:0/1444115914 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f950819f5b0 con 0x7f9508108a10 2024-04-25T11:42:38.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.191+0000 7f950d5de640 1 --2- 172.21.15.90:0/1444115914 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9508103830 0x7f950819a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:42:38.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.192+0000 7f950cddd640 1 --2- 172.21.15.90:0/1444115914 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9508108a10 0x7f950819acb0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f94f800d740 tx=0x7f94f800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.192+0000 7f94fe7fc640 1 -- 172.21.15.90:0/1444115914 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f8017070 con 0x7f9508108a10 2024-04-25T11:42:38.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.192+0000 7f94fe7fc640 1 -- 172.21.15.90:0/1444115914 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94f800de50 con 0x7f9508108a10 2024-04-25T11:42:38.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.192+0000 7f950f842640 1 -- 172.21.15.90:0/1444115914 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f950819f8a0 con 0x7f9508108a10 2024-04-25T11:42:38.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.192+0000 7f94fe7fc640 1 -- 172.21.15.90:0/1444115914 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f8003c20 con 0x7f9508108a10 2024-04-25T11:42:38.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.192+0000 7f950f842640 1 -- 172.21.15.90:0/1444115914 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95081a7220 con 0x7f9508108a10 2024-04-25T11:42:38.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.194+0000 7f94fe7fc640 1 -- 172.21.15.90:0/1444115914 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f94f8003dc0 con 0x7f9508108a10 2024-04-25T11:42:38.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.195+0000 7f95167d0740 1 Processor -- start 2024-04-25T11:42:38.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.195+0000 7f95167d0740 1 -- start start 2024-04-25T11:42:38.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.195+0000 7f95167d0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16af7390 0x560c16e98810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.195+0000 7f95167d0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c16ace4d0 0x560c16e98d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.195+0000 7f95167d0740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560c16ad66b0 0x560c16ea56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.195+0000 7f95167d0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560c16dfdd80 con 0x560c16af7390 2024-04-25T11:42:38.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.195+0000 7f95167d0740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x560c16db7a20 con 0x560c16ad66b0 2024-04-25T11:42:38.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.195+0000 7f95167d0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560c16df9d40 con 0x560c16ace4d0 2024-04-25T11:42:38.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.195+0000 7f950dddf640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16af7390 0x560c16e98810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.195+0000 7f950dddf640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16af7390 0x560c16e98810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60208/0 (socket says 172.21.15.90:60208) 2024-04-25T11:42:38.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.195+0000 7f950dddf640 1 -- 172.21.15.90:0/3530920917 learned_addr learned my addr 172.21.15.90:0/3530920917 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.195+0000 7f950cddd640 1 --2- 172.21.15.90:0/3530920917 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560c16ad66b0 0x560c16ea56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.195+0000 7f950d5de640 1 --2- 172.21.15.90:0/3530920917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c16ace4d0 0x560c16e98d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.195+0000 7f950dddf640 1 -- 172.21.15.90:0/3530920917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c16ace4d0 msgr2=0x560c16e98d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.195+0000 7f950dddf640 1 --2- 172.21.15.90:0/3530920917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c16ace4d0 0x560c16e98d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.195+0000 7f94fe7fc640 1 --2- 172.21.15.90:0/1444115914 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f94f40afc60 0x7f94f40b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.195+0000 7f950dddf640 1 -- 172.21.15.90:0/3530920917 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560c16ad66b0 msgr2=0x560c16ea56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.195+0000 7f950dddf640 1 --2- 172.21.15.90:0/3530920917 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560c16ad66b0 0x560c16ea56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.195+0000 7f950dddf640 1 -- 172.21.15.90:0/3530920917 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c16dc2c30 con 0x560c16af7390 2024-04-25T11:42:38.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.195+0000 7f950cddd640 1 --2- 172.21.15.90:0/3530920917 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560c16ad66b0 0x560c16ea56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:42:38.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.195+0000 7f950dddf640 1 --2- 172.21.15.90:0/3530920917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16af7390 0x560c16e98810 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f95000039f0 tx=0x7f9500003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=392efd58933a4309 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.195+0000 7f94fe7fc640 1 -- 172.21.15.90:0/1444115914 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7f94f80dea00 con 0x7f9508108a10 2024-04-25T11:42:38.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.196+0000 7f950dddf640 1 --2- 172.21.15.90:0/1444115914 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f94f40afc60 0x7f94f40b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.196+0000 7f94f37fe640 1 -- 172.21.15.90:0/3530920917 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9500004860 con 0x560c16af7390 2024-04-25T11:42:38.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.196+0000 7f94f37fe640 1 -- 172.21.15.90:0/3530920917 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9500015070 con 0x560c16af7390 2024-04-25T11:42:38.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.196+0000 7f94f37fe640 1 -- 172.21.15.90:0/3530920917 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9500010440 con 0x560c16af7390 2024-04-25T11:42:38.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.197+0000 7f95167d0740 1 -- 172.21.15.90:0/3530920917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16af7390 msgr2=0x560c16e98810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.197+0000 7f95167d0740 1 --2- 172.21.15.90:0/3530920917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16af7390 0x560c16e98810 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f95000039f0 tx=0x7f9500003eb0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.197+0000 7f950dddf640 1 --2- 172.21.15.90:0/1444115914 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f94f40afc60 0x7f94f40b2170 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9500009fd0 tx=0x7f9500010b60 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.197+0000 7f95167d0740 1 -- 172.21.15.90:0/3530920917 shutdown_connections 2024-04-25T11:42:38.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.197+0000 7f95167d0740 1 --2- 172.21.15.90:0/3530920917 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560c16ad66b0 0x560c16ea56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.197+0000 7f95167d0740 1 --2- 172.21.15.90:0/3530920917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c16ace4d0 0x560c16e98d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.197+0000 7f95167d0740 1 --2- 172.21.15.90:0/3530920917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16af7390 0x560c16e98810 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.197+0000 7f95167d0740 1 -- 172.21.15.90:0/3530920917 >> 172.21.15.90:0/3530920917 conn(0x560c16aeb4e0 msgr2=0x560c16e94fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.197+0000 7f95167d0740 1 -- 172.21.15.90:0/3530920917 shutdown_connections 2024-04-25T11:42:38.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.197+0000 7f95167d0740 1 -- 172.21.15.90:0/3530920917 wait complete. 2024-04-25T11:42:38.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.197+0000 7f95167d0740 1 -- start start 2024-04-25T11:42:38.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.197+0000 7f95167d0740 1 Processor -- start 2024-04-25T11:42:38.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.198+0000 7f95167d0740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560c16ace4d0 0x560c16ea3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.198+0000 7f95167d0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16ad66b0 0x560c16ea3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.198+0000 7f95167d0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c16af7390 0x560c16eaf210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.198+0000 7f95167d0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560c16ea5d90 con 0x560c16ad66b0 2024-04-25T11:42:38.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.198+0000 7f95167d0740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x560c16d84dc0 con 0x560c16ace4d0 2024-04-25T11:42:38.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.198+0000 7f95167d0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560c16ea5f10 con 0x560c16af7390 2024-04-25T11:42:38.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.198+0000 7f95167d0740 10 client.0 _ll_register_callbacks cb 0x560c16da0600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:38.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.198+0000 7f950cddd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16ad66b0 0x560c16ea3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.198+0000 7f950cddd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16ad66b0 0x560c16ea3790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60216/0 (socket says 172.21.15.90:60216) 2024-04-25T11:42:38.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.198+0000 7f950cddd640 1 -- 172.21.15.90:0/3409970643 learned_addr learned my addr 172.21.15.90:0/3409970643 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.198+0000 7f950d5de640 1 --2- 172.21.15.90:0/3409970643 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560c16ace4d0 0x560c16ea3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.198+0000 7f950dddf640 1 --2- 172.21.15.90:0/3409970643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c16af7390 0x560c16eaf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.198+0000 7f950cddd640 1 -- 172.21.15.90:0/3409970643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c16af7390 msgr2=0x560c16eaf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.198+0000 7f950cddd640 1 --2- 172.21.15.90:0/3409970643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c16af7390 0x560c16eaf210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.198+0000 7f950cddd640 1 -- 172.21.15.90:0/3409970643 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560c16ace4d0 msgr2=0x560c16ea3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.198+0000 7f950cddd640 1 --2- 172.21.15.90:0/3409970643 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560c16ace4d0 0x560c16ea3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.198+0000 7f950cddd640 1 -- 172.21.15.90:0/3409970643 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c16b0f3f0 con 0x560c16ad66b0 2024-04-25T11:42:38.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.198+0000 7f950cddd640 1 --2- 172.21.15.90:0/3409970643 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16ad66b0 0x560c16ea3790 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f94f80a3e30 tx=0x7f94f80a6990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afc9f0092636bf1b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.198+0000 7f94f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:38.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.198+0000 7f94f0ff9640 1 -- 172.21.15.90:0/3409970643 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f8020050 con 0x560c16ad66b0 2024-04-25T11:42:38.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.198+0000 7f94f0ff9640 1 -- 172.21.15.90:0/3409970643 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7f94f80a2e50 con 0x560c16ad66b0 2024-04-25T11:42:38.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.198+0000 7f95167d0740 1 -- 172.21.15.90:0/3409970643 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9500015070 con 0x560c16ad66b0 2024-04-25T11:42:38.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.199+0000 7f94f0ff9640 1 -- 172.21.15.90:0/3409970643 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f94f80a66d0 con 0x560c16ad66b0 2024-04-25T11:42:38.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.199+0000 7f95167d0740 10 client.17868 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:38.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.199+0000 7f95167d0740 1 -- 172.21.15.90:0/3409970643 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560c16bd97d0 con 0x560c16ad66b0 2024-04-25T11:42:38.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.199+0000 7f94f0ff9640 1 -- 172.21.15.90:0/3409970643 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9500015070 con 0x560c16ad66b0 2024-04-25T11:42:38.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.199+0000 7f94f0ff9640 1 -- 172.21.15.90:0/3409970643 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f94f80a2530 con 0x560c16ad66b0 2024-04-25T11:42:38.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.199+0000 7f95167d0740 10 client.17868 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:38.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.199+0000 7f95167d0740 10 client.17868 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:42:38.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.199+0000 7f95167d0740 20 client.17868 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:38.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.199+0000 7f95167d0740 1 --2- 172.21.15.90:0/3409970643 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x560c16ab5f60 0x560c16eb55a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.199+0000 7f95167d0740 4 client.17868 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:42:38.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.199+0000 7f95167d0740 1 -- 172.21.15.90:0/3409970643 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c16e0b990 con 0x560c16ab5f60 2024-04-25T11:42:38.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.199+0000 7f950d5de640 1 --2- 172.21.15.90:0/3409970643 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x560c16ab5f60 0x560c16eb55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.200+0000 7f94f0ff9640 10 client.17868 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:42:38.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.200+0000 7f950d5de640 1 --2- 172.21.15.90:0/3409970643 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x560c16ab5f60 0x560c16eb55a0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=df8a2d145305fb36 server_cookie=a928c3bd41f50656 in_seq=0 out_seq=0 2024-04-25T11:42:38.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.200+0000 7f94f0ff9640 1 -- 172.21.15.90:0/3409970643 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560c16e0b990 con 0x560c16ab5f60 2024-04-25T11:42:38.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.200+0000 7f94f0ff9640 10 client.17868 handle_command_reply: tid=0 2024-04-25T11:42:38.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.200+0000 7f95167d0740 1 client.17868 shutdown 2024-04-25T11:42:38.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.201+0000 7f95167d0740 1 -- 172.21.15.90:0/3409970643 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16ad66b0 msgr2=0x560c16ea3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.201+0000 7f95167d0740 1 --2- 172.21.15.90:0/3409970643 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16ad66b0 0x560c16ea3790 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f94f80a3e30 tx=0x7f94f80a6990 comp rx=0 tx=0).stop 2024-04-25T11:42:38.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.201+0000 7f95167d0740 1 -- 172.21.15.90:0/3409970643 shutdown_connections 2024-04-25T11:42:38.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.201+0000 7f95167d0740 1 --2- 172.21.15.90:0/3409970643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c16af7390 0x560c16eaf210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.201+0000 7f95167d0740 1 --2- 172.21.15.90:0/3409970643 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16ad66b0 0x560c16ea3790 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.201+0000 7f95167d0740 1 --2- 172.21.15.90:0/3409970643 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x560c16ab5f60 0x560c16eb55a0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.201+0000 7f95167d0740 1 --2- 172.21.15.90:0/3409970643 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560c16ace4d0 0x560c16ea3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.201+0000 7f94f0ff9640 0 client.17868 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:42:38.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.201+0000 7f95167d0740 1 -- 172.21.15.90:0/3409970643 >> 172.21.15.90:0/3409970643 conn(0x560c16aeb4e0 msgr2=0x560c16e9a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.201+0000 7f95167d0740 1 -- 172.21.15.90:0/3409970643 shutdown_connections 2024-04-25T11:42:38.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.201+0000 7f95167d0740 1 -- 172.21.15.90:0/3409970643 wait complete. 2024-04-25T11:42:38.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.201+0000 7f95167d0740 20 client.17868 trim_cache size 0 max 16384 2024-04-25T11:42:38.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.214+0000 7f95167d0740 1 Processor -- start 2024-04-25T11:42:38.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.214+0000 7f95167d0740 1 -- start start 2024-04-25T11:42:38.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.214+0000 7f95167d0740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560c16ace4d0 0x560c16ea2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.214+0000 7f95167d0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16ad66b0 0x560c16e93d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.214+0000 7f95167d0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c16af7390 0x560c16e9d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.214+0000 7f95167d0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f95081afd30 con 0x560c16ad66b0 2024-04-25T11:42:38.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.214+0000 7f95167d0740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f95081b1870 con 0x560c16ace4d0 2024-04-25T11:42:38.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.214+0000 7f95167d0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f95081b0400 con 0x560c16af7390 2024-04-25T11:42:38.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.215+0000 7f950cddd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16ad66b0 0x560c16e93d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.215+0000 7f950cddd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16ad66b0 0x560c16e93d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60226/0 (socket says 172.21.15.90:60226) 2024-04-25T11:42:38.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.215+0000 7f950cddd640 1 -- 172.21.15.90:0/145945312 learned_addr learned my addr 172.21.15.90:0/145945312 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.215+0000 7f950cddd640 1 -- 172.21.15.90:0/145945312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c16af7390 msgr2=0x560c16e9d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.215+0000 7f950cddd640 1 --2- 172.21.15.90:0/145945312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c16af7390 0x560c16e9d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.215+0000 7f950cddd640 1 -- 172.21.15.90:0/145945312 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560c16ace4d0 msgr2=0x560c16ea2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.215+0000 7f950cddd640 1 --2- 172.21.15.90:0/145945312 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560c16ace4d0 0x560c16ea2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.215+0000 7f950cddd640 1 -- 172.21.15.90:0/145945312 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c16e9ec00 con 0x560c16ad66b0 2024-04-25T11:42:38.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.215+0000 7f950cddd640 1 --2- 172.21.15.90:0/145945312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16ad66b0 0x560c16e93d20 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f94f80a6870 tx=0x7f94f80035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=befa32933c36ea39 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.215+0000 7f94f2ffd640 1 -- 172.21.15.90:0/145945312 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f8020050 con 0x560c16ad66b0 2024-04-25T11:42:38.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.215+0000 7f94f2ffd640 1 -- 172.21.15.90:0/145945312 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94f80e2030 con 0x560c16ad66b0 2024-04-25T11:42:38.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.215+0000 7f94f2ffd640 1 -- 172.21.15.90:0/145945312 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f80aaec0 con 0x560c16ad66b0 2024-04-25T11:42:38.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.216+0000 7f95167d0740 1 -- 172.21.15.90:0/145945312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16ad66b0 msgr2=0x560c16e93d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.216+0000 7f95167d0740 1 --2- 172.21.15.90:0/145945312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16ad66b0 0x560c16e93d20 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f94f80a6870 tx=0x7f94f80035f0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.216+0000 7f95167d0740 1 -- 172.21.15.90:0/145945312 shutdown_connections 2024-04-25T11:42:38.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.216+0000 7f95167d0740 1 --2- 172.21.15.90:0/145945312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c16af7390 0x560c16e9d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.216+0000 7f95167d0740 1 --2- 172.21.15.90:0/145945312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16ad66b0 0x560c16e93d20 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.216+0000 7f95167d0740 1 --2- 172.21.15.90:0/145945312 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560c16ace4d0 0x560c16ea2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.216+0000 7f95167d0740 1 -- 172.21.15.90:0/145945312 >> 172.21.15.90:0/145945312 conn(0x560c16aeb4e0 msgr2=0x560c16e93610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.216+0000 7f95167d0740 1 -- 172.21.15.90:0/145945312 shutdown_connections 2024-04-25T11:42:38.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.216+0000 7f95167d0740 1 -- 172.21.15.90:0/145945312 wait complete. 2024-04-25T11:42:38.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.216+0000 7f95167d0740 1 -- start start 2024-04-25T11:42:38.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.216+0000 7f95167d0740 1 Processor -- start 2024-04-25T11:42:38.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.217+0000 7f95167d0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16ace4d0 0x560c16eab870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.217+0000 7f95167d0740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560c16ad66b0 0x560c16ea6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.217+0000 7f95167d0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c16af7390 0x560c16ea71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.217+0000 7f95167d0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f95081b11a0 con 0x560c16ace4d0 2024-04-25T11:42:38.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.217+0000 7f95167d0740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f95081b0ad0 con 0x560c16ad66b0 2024-04-25T11:42:38.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.217+0000 7f95167d0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f95081add30 con 0x560c16af7390 2024-04-25T11:42:38.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.217+0000 7f95167d0740 10 client.0 _ll_register_callbacks cb 0x7f94f8005d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:38.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.217+0000 7f950d5de640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16ace4d0 0x560c16eab870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.217+0000 7f950d5de640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16ace4d0 0x560c16eab870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60240/0 (socket says 172.21.15.90:60240) 2024-04-25T11:42:38.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.217+0000 7f950d5de640 1 -- 172.21.15.90:0/2116138989 learned_addr learned my addr 172.21.15.90:0/2116138989 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.217+0000 7f950cddd640 1 --2- 172.21.15.90:0/2116138989 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560c16ad66b0 0x560c16ea6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.217+0000 7f950d5de640 1 -- 172.21.15.90:0/2116138989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c16af7390 msgr2=0x560c16ea71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.217+0000 7f950dddf640 1 --2- 172.21.15.90:0/2116138989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c16af7390 0x560c16ea71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.217+0000 7f950d5de640 1 --2- 172.21.15.90:0/2116138989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c16af7390 0x560c16ea71c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.217+0000 7f950d5de640 1 -- 172.21.15.90:0/2116138989 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560c16ad66b0 msgr2=0x560c16ea6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.217+0000 7f950d5de640 1 --2- 172.21.15.90:0/2116138989 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560c16ad66b0 0x560c16ea6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.217+0000 7f950d5de640 1 -- 172.21.15.90:0/2116138989 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c16ea7920 con 0x560c16ace4d0 2024-04-25T11:42:38.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.217+0000 7f950cddd640 1 --2- 172.21.15.90:0/2116138989 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560c16ad66b0 0x560c16ea6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:42:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.217+0000 7f950d5de640 1 --2- 172.21.15.90:0/2116138989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16ace4d0 0x560c16eab870 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f9508078990 tx=0x7f9504032a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e62551ada32da7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.217+0000 7f94f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.217+0000 7f94f0ff9640 1 -- 172.21.15.90:0/2116138989 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9504035040 con 0x560c16ace4d0 2024-04-25T11:42:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.217+0000 7f94f0ff9640 1 -- 172.21.15.90:0/2116138989 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7f9504005040 con 0x560c16ace4d0 2024-04-25T11:42:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.217+0000 7f95167d0740 1 -- 172.21.15.90:0/2116138989 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94f80e2030 con 0x560c16ace4d0 2024-04-25T11:42:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.218+0000 7f94f0ff9640 1 -- 172.21.15.90:0/2116138989 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9504053050 con 0x560c16ace4d0 2024-04-25T11:42:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.218+0000 7f95167d0740 10 client.17874 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.218+0000 7f95167d0740 1 -- 172.21.15.90:0/2116138989 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560c16f1d8c0 con 0x560c16ace4d0 2024-04-25T11:42:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.218+0000 7f94f0ff9640 1 -- 172.21.15.90:0/2116138989 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94f80e2030 con 0x560c16ace4d0 2024-04-25T11:42:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.218+0000 7f94f0ff9640 1 -- 172.21.15.90:0/2116138989 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f95040452c0 con 0x560c16ace4d0 2024-04-25T11:42:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.218+0000 7f95167d0740 10 client.17874 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.218+0000 7f95167d0740 10 client.17874 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:42:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.218+0000 7f95167d0740 20 client.17874 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.218+0000 7f95167d0740 1 --2- 172.21.15.90:0/2116138989 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x560c16a8b3a0 0x560c16f200b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.218+0000 7f95167d0740 4 client.17874 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:42:38.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.218+0000 7f95167d0740 1 -- 172.21.15.90:0/2116138989 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560c16e05800 con 0x560c16a8b3a0 2024-04-25T11:42:38.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.218+0000 7f950cddd640 1 --2- 172.21.15.90:0/2116138989 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x560c16a8b3a0 0x560c16f200b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.219+0000 7f94f0ff9640 10 client.17874 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:42:38.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.219+0000 7f950cddd640 1 --2- 172.21.15.90:0/2116138989 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x560c16a8b3a0 0x560c16f200b0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=46acddf4638d539f server_cookie=15dbea1f5a6023ae in_seq=0 out_seq=0 2024-04-25T11:42:38.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.219+0000 7f94f0ff9640 1 -- 172.21.15.90:0/2116138989 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560c16e05800 con 0x560c16a8b3a0 2024-04-25T11:42:38.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.219+0000 7f94f0ff9640 10 client.17874 handle_command_reply: tid=0 2024-04-25T11:42:38.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.219+0000 7f95167d0740 1 client.17874 shutdown 2024-04-25T11:42:38.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.219+0000 7f95167d0740 1 -- 172.21.15.90:0/2116138989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16ace4d0 msgr2=0x560c16eab870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.219+0000 7f95167d0740 1 --2- 172.21.15.90:0/2116138989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16ace4d0 0x560c16eab870 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f9508078990 tx=0x7f9504032a90 comp rx=0 tx=0).stop 2024-04-25T11:42:38.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.219+0000 7f95167d0740 1 -- 172.21.15.90:0/2116138989 shutdown_connections 2024-04-25T11:42:38.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.219+0000 7f95167d0740 1 --2- 172.21.15.90:0/2116138989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c16af7390 0x560c16ea71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.219+0000 7f95167d0740 1 --2- 172.21.15.90:0/2116138989 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x560c16a8b3a0 0x560c16f200b0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.219+0000 7f95167d0740 1 --2- 172.21.15.90:0/2116138989 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560c16ad66b0 0x560c16ea6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.219+0000 7f95167d0740 1 --2- 172.21.15.90:0/2116138989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560c16ace4d0 0x560c16eab870 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.219+0000 7f94f0ff9640 0 client.17874 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:42:38.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.219+0000 7f95167d0740 1 -- 172.21.15.90:0/2116138989 >> 172.21.15.90:0/2116138989 conn(0x560c16aeb4e0 msgr2=0x560c16ea20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.219+0000 7f95167d0740 1 -- 172.21.15.90:0/2116138989 shutdown_connections 2024-04-25T11:42:38.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.219+0000 7f95167d0740 1 -- 172.21.15.90:0/2116138989 wait complete. 2024-04-25T11:42:38.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.219+0000 7f95167d0740 20 client.17874 trim_cache size 0 max 16384 2024-04-25T11:42:38.212 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:42:38.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.220+0000 7f950f842640 1 -- 172.21.15.90:0/1444115914 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f94f40afc60 msgr2=0x7f94f40b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.220+0000 7f950f842640 1 --2- 172.21.15.90:0/1444115914 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f94f40afc60 0x7f94f40b2170 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9500009fd0 tx=0x7f9500010b60 comp rx=0 tx=0).stop 2024-04-25T11:42:38.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.220+0000 7f950f842640 1 -- 172.21.15.90:0/1444115914 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9508108a10 msgr2=0x7f950819acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.220+0000 7f950f842640 1 --2- 172.21.15.90:0/1444115914 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9508108a10 0x7f950819acb0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f94f800d740 tx=0x7f94f800dc00 comp rx=0 tx=0).stop 2024-04-25T11:42:38.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.220+0000 7f950f842640 1 -- 172.21.15.90:0/1444115914 shutdown_connections 2024-04-25T11:42:38.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.220+0000 7f950f842640 1 --2- 172.21.15.90:0/1444115914 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f94f40afc60 0x7f94f40b2170 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.220+0000 7f950f842640 1 --2- 172.21.15.90:0/1444115914 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9508108eb0 0x7f950819ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.220+0000 7f950f842640 1 --2- 172.21.15.90:0/1444115914 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9508108a10 0x7f950819acb0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.220+0000 7f950f842640 1 --2- 172.21.15.90:0/1444115914 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9508103830 0x7f950819a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.220+0000 7f950f842640 1 -- 172.21.15.90:0/1444115914 >> 172.21.15.90:0/1444115914 conn(0x7f9508077880 msgr2=0x7f9508101b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.220+0000 7f950f842640 1 -- 172.21.15.90:0/1444115914 shutdown_connections 2024-04-25T11:42:38.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.220+0000 7f950f842640 1 -- 172.21.15.90:0/1444115914 wait complete. 2024-04-25T11:42:38.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:37 smithi090 ceph-mon[29575]: pgmap v231: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 18 KiB/s rd, 49 MiB/s wr, 328 op/s 2024-04-25T11:42:38.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:37 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/444633888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:42:38.218 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T11:42:38.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:37 smithi138 ceph-mon[32328]: pgmap v231: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 18 KiB/s rd, 49 MiB/s wr, 328 op/s 2024-04-25T11:42:38.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:37 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/444633888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:42:38.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.340+0000 7fe24b830640 1 -- 172.21.15.90:0/3992228649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe244104260 msgr2=0x7fe244110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.340+0000 7fe24b830640 1 --2- 172.21.15.90:0/3992228649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe244104260 0x7fe244110630 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fe23c009ca0 tx=0x7fe23c02a8c0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.340+0000 7fe24b830640 1 -- 172.21.15.90:0/3992228649 shutdown_connections 2024-04-25T11:42:38.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.340+0000 7fe24b830640 1 --2- 172.21.15.90:0/3992228649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe244104260 0x7fe244110630 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.340+0000 7fe24b830640 1 --2- 172.21.15.90:0/3992228649 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe244109fb0 0x7fe244103bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.340+0000 7fe24b830640 1 --2- 172.21.15.90:0/3992228649 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe244109b10 0x7fe2441036b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.340+0000 7fe24b830640 1 -- 172.21.15.90:0/3992228649 >> 172.21.15.90:0/3992228649 conn(0x7fe2440ff1d0 msgr2=0x7fe244101620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.340+0000 7fe24b830640 1 -- 172.21.15.90:0/3992228649 shutdown_connections 2024-04-25T11:42:38.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.341+0000 7fe24b830640 1 -- 172.21.15.90:0/3992228649 wait complete. 2024-04-25T11:42:38.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.341+0000 7fe24b830640 1 Processor -- start 2024-04-25T11:42:38.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.341+0000 7fe24b830640 1 -- start start 2024-04-25T11:42:38.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.341+0000 7fe24b830640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe244104260 0x7fe24419a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.341+0000 7fe24b830640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe244109b10 0x7fe24419ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.341+0000 7fe24b830640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe244109fb0 0x7fe24419ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.341+0000 7fe24b830640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe244113d20 con 0x7fe244109b10 2024-04-25T11:42:38.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.341+0000 7fe24b830640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe244113ba0 con 0x7fe244104260 2024-04-25T11:42:38.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.341+0000 7fe24b830640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe244113ea0 con 0x7fe244109fb0 2024-04-25T11:42:38.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.342+0000 7fe248dcb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe244109b10 0x7fe24419ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.342+0000 7fe248dcb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe244109b10 0x7fe24419ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60260/0 (socket says 172.21.15.90:60260) 2024-04-25T11:42:38.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.342+0000 7fe248dcb640 1 -- 172.21.15.90:0/723854997 learned_addr learned my addr 172.21.15.90:0/723854997 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.342+0000 7fe2495cc640 1 --2- 172.21.15.90:0/723854997 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe244104260 0x7fe24419a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.342+0000 7fe249dcd640 1 --2- 172.21.15.90:0/723854997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe244109fb0 0x7fe24419ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.342+0000 7fe248dcb640 1 -- 172.21.15.90:0/723854997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe244109fb0 msgr2=0x7fe24419ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.342+0000 7fe248dcb640 1 --2- 172.21.15.90:0/723854997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe244109fb0 0x7fe24419ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.342+0000 7fe248dcb640 1 -- 172.21.15.90:0/723854997 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe244104260 msgr2=0x7fe24419a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.342+0000 7fe248dcb640 1 --2- 172.21.15.90:0/723854997 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe244104260 0x7fe24419a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.342+0000 7fe248dcb640 1 -- 172.21.15.90:0/723854997 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe24419f540 con 0x7fe244109b10 2024-04-25T11:42:38.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.342+0000 7fe248dcb640 1 --2- 172.21.15.90:0/723854997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe244109b10 0x7fe24419ac40 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fe23400c970 tx=0x7fe23400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.342+0000 7fe23a7fc640 1 -- 172.21.15.90:0/723854997 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe234004220 con 0x7fe244109b10 2024-04-25T11:42:38.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.342+0000 7fe23a7fc640 1 -- 172.21.15.90:0/723854997 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2340043c0 con 0x7fe244109b10 2024-04-25T11:42:38.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.342+0000 7fe23a7fc640 1 -- 172.21.15.90:0/723854997 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe234004a00 con 0x7fe244109b10 2024-04-25T11:42:38.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.342+0000 7fe24b830640 1 -- 172.21.15.90:0/723854997 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe24419f830 con 0x7fe244109b10 2024-04-25T11:42:38.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.342+0000 7fe24b830640 1 -- 172.21.15.90:0/723854997 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe24419fd30 con 0x7fe244109b10 2024-04-25T11:42:38.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.345+0000 7fe2527b6740 1 Processor -- start 2024-04-25T11:42:38.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.345+0000 7fe2527b6740 1 -- start start 2024-04-25T11:42:38.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.345+0000 7fe23a7fc640 1 -- 172.21.15.90:0/723854997 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fe2340079b0 con 0x7fe244109b10 2024-04-25T11:42:38.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.345+0000 7fe2527b6740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb224c5390 0x55eb228666e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.345+0000 7fe2527b6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb2249c4d0 0x55eb22866c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.345+0000 7fe2527b6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224a46b0 0x55eb22877ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.345+0000 7fe2527b6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55eb227cbd80 con 0x55eb224a46b0 2024-04-25T11:42:38.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.345+0000 7fe2527b6740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55eb22785a20 con 0x55eb224c5390 2024-04-25T11:42:38.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.345+0000 7fe2527b6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55eb227c7d40 con 0x55eb2249c4d0 2024-04-25T11:42:38.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.345+0000 7fe248dcb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224a46b0 0x55eb22877ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.345+0000 7fe248dcb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224a46b0 0x55eb22877ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60272/0 (socket says 172.21.15.90:60272) 2024-04-25T11:42:38.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.345+0000 7fe248dcb640 1 -- 172.21.15.90:0/2203458148 learned_addr learned my addr 172.21.15.90:0/2203458148 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.345+0000 7fe249dcd640 1 --2- 172.21.15.90:0/2203458148 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb224c5390 0x55eb228666e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.345+0000 7fe2495cc640 1 --2- 172.21.15.90:0/2203458148 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb2249c4d0 0x55eb22866c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.345+0000 7fe248dcb640 1 -- 172.21.15.90:0/2203458148 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb2249c4d0 msgr2=0x55eb22866c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.345+0000 7fe248dcb640 1 --2- 172.21.15.90:0/2203458148 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb2249c4d0 0x55eb22866c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.345+0000 7fe248dcb640 1 -- 172.21.15.90:0/2203458148 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb224c5390 msgr2=0x55eb228666e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.345+0000 7fe248dcb640 1 --2- 172.21.15.90:0/2203458148 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb224c5390 0x55eb228666e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.346+0000 7fe248dcb640 1 -- 172.21.15.90:0/2203458148 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb22790c30 con 0x55eb224a46b0 2024-04-25T11:42:38.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.346+0000 7fe23a7fc640 1 --2- 172.21.15.90:0/723854997 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe2300afc60 0x7fe2300b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.346+0000 7fe249dcd640 1 --2- 172.21.15.90:0/723854997 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe2300afc60 0x7fe2300b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.346+0000 7fe248dcb640 1 --2- 172.21.15.90:0/2203458148 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224a46b0 0x55eb22877ab0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fe2340dc490 tx=0x7fe2340e7570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5042a4c24d9480a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.346+0000 7fe23a7fc640 1 -- 172.21.15.90:0/723854997 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7fe2340dee60 con 0x7fe244109b10 2024-04-25T11:42:38.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.346+0000 7fe22f7fe640 1 -- 172.21.15.90:0/2203458148 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe23401d030 con 0x55eb224a46b0 2024-04-25T11:42:38.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.346+0000 7fe22f7fe640 1 -- 172.21.15.90:0/2203458148 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe234005690 con 0x55eb224a46b0 2024-04-25T11:42:38.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.346+0000 7fe22f7fe640 1 -- 172.21.15.90:0/2203458148 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2340a6d40 con 0x55eb224a46b0 2024-04-25T11:42:38.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.347+0000 7fe2527b6740 1 -- 172.21.15.90:0/2203458148 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224a46b0 msgr2=0x55eb22877ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.347+0000 7fe2527b6740 1 --2- 172.21.15.90:0/2203458148 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224a46b0 0x55eb22877ab0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fe2340dc490 tx=0x7fe2340e7570 comp rx=0 tx=0).stop 2024-04-25T11:42:38.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.347+0000 7fe249dcd640 1 --2- 172.21.15.90:0/723854997 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe2300afc60 0x7fe2300b2170 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe23c005cc0 tx=0x7fe23c005c30 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.347+0000 7fe2527b6740 1 -- 172.21.15.90:0/2203458148 shutdown_connections 2024-04-25T11:42:38.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.347+0000 7fe2527b6740 1 --2- 172.21.15.90:0/2203458148 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224a46b0 0x55eb22877ab0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.347+0000 7fe2527b6740 1 --2- 172.21.15.90:0/2203458148 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb2249c4d0 0x55eb22866c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.347+0000 7fe2527b6740 1 --2- 172.21.15.90:0/2203458148 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb224c5390 0x55eb228666e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.347+0000 7fe2527b6740 1 -- 172.21.15.90:0/2203458148 >> 172.21.15.90:0/2203458148 conn(0x55eb224b94e0 msgr2=0x55eb22862fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.347+0000 7fe2527b6740 1 -- 172.21.15.90:0/2203458148 shutdown_connections 2024-04-25T11:42:38.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.347+0000 7fe2527b6740 1 -- 172.21.15.90:0/2203458148 wait complete. 2024-04-25T11:42:38.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.347+0000 7fe2527b6740 1 -- start start 2024-04-25T11:42:38.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.347+0000 7fe2527b6740 1 Processor -- start 2024-04-25T11:42:38.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.348+0000 7fe2527b6740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb2249c4d0 0x55eb22875370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.348+0000 7fe2527b6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb224a46b0 0x55eb228758b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.348+0000 7fe2527b6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224c5390 0x55eb228816e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.348+0000 7fe2527b6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55eb22878230 con 0x55eb224c5390 2024-04-25T11:42:38.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.348+0000 7fe2527b6740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55eb22752dc0 con 0x55eb2249c4d0 2024-04-25T11:42:38.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.348+0000 7fe2527b6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55eb228783b0 con 0x55eb224a46b0 2024-04-25T11:42:38.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.348+0000 7fe2527b6740 10 client.0 _ll_register_callbacks cb 0x55eb2276e600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:38.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.348+0000 7fe249dcd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224c5390 0x55eb228816e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.348+0000 7fe249dcd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224c5390 0x55eb228816e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60274/0 (socket says 172.21.15.90:60274) 2024-04-25T11:42:38.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.348+0000 7fe249dcd640 1 -- 172.21.15.90:0/658757768 learned_addr learned my addr 172.21.15.90:0/658757768 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.348+0000 7fe248dcb640 1 --2- 172.21.15.90:0/658757768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb224a46b0 0x55eb228758b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.348+0000 7fe2495cc640 1 --2- 172.21.15.90:0/658757768 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb2249c4d0 0x55eb22875370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.348+0000 7fe249dcd640 1 -- 172.21.15.90:0/658757768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb224a46b0 msgr2=0x55eb228758b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.348+0000 7fe249dcd640 1 --2- 172.21.15.90:0/658757768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb224a46b0 0x55eb228758b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.348+0000 7fe249dcd640 1 -- 172.21.15.90:0/658757768 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb2249c4d0 msgr2=0x55eb22875370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.348+0000 7fe249dcd640 1 --2- 172.21.15.90:0/658757768 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb2249c4d0 0x55eb22875370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.348+0000 7fe249dcd640 1 -- 172.21.15.90:0/658757768 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb224dd3f0 con 0x55eb224c5390 2024-04-25T11:42:38.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.348+0000 7fe249dcd640 1 --2- 172.21.15.90:0/658757768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224c5390 0x55eb228816e0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fe23c031450 tx=0x7fe23c02a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ba5be11ce2d8bcf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.348+0000 7fe22cff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:38.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.349+0000 7fe22cff9640 1 -- 172.21.15.90:0/658757768 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe23c003d00 con 0x55eb224c5390 2024-04-25T11:42:38.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.349+0000 7fe22cff9640 1 -- 172.21.15.90:0/658757768 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7fe23c004040 con 0x55eb224c5390 2024-04-25T11:42:38.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.349+0000 7fe2527b6740 1 -- 172.21.15.90:0/658757768 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55eb227d9990 con 0x55eb224c5390 2024-04-25T11:42:38.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.349+0000 7fe22cff9640 1 -- 172.21.15.90:0/658757768 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe23c004240 con 0x55eb224c5390 2024-04-25T11:42:38.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.349+0000 7fe2527b6740 10 client.17886 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:38.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.349+0000 7fe2527b6740 1 -- 172.21.15.90:0/658757768 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55eb225a77d0 con 0x55eb224c5390 2024-04-25T11:42:38.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.349+0000 7fe22cff9640 1 -- 172.21.15.90:0/658757768 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55eb227d9990 con 0x55eb224c5390 2024-04-25T11:42:38.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.349+0000 7fe22cff9640 1 -- 172.21.15.90:0/658757768 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fe23c051070 con 0x55eb224c5390 2024-04-25T11:42:38.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.349+0000 7fe2527b6740 10 client.17886 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:38.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.349+0000 7fe2527b6740 10 client.17886 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:42:38.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.349+0000 7fe2527b6740 20 client.17886 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:38.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.349+0000 7fe2527b6740 1 --2- 172.21.15.90:0/658757768 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55eb22483f60 0x55eb22887940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.349+0000 7fe2527b6740 4 client.17886 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:42:38.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.349+0000 7fe2527b6740 1 -- 172.21.15.90:0/658757768 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb227d3800 con 0x55eb22483f60 2024-04-25T11:42:38.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.349+0000 7fe2495cc640 1 --2- 172.21.15.90:0/658757768 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55eb22483f60 0x55eb22887940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.350+0000 7fe22cff9640 10 client.17886 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:42:38.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.350+0000 7fe2495cc640 1 --2- 172.21.15.90:0/658757768 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55eb22483f60 0x55eb22887940 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=75d857c6c575f5a6 server_cookie=87dbac252e990898 in_seq=0 out_seq=0 2024-04-25T11:42:38.343 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:37 smithi156 ceph-mon[33045]: pgmap v231: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 18 KiB/s rd, 49 MiB/s wr, 328 op/s 2024-04-25T11:42:38.343 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:37 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/444633888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:42:38.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.351+0000 7fe22cff9640 1 -- 172.21.15.90:0/658757768 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55eb227d3800 con 0x55eb22483f60 2024-04-25T11:42:38.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.351+0000 7fe22cff9640 10 client.17886 handle_command_reply: tid=0 2024-04-25T11:42:38.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.351+0000 7fe2527b6740 1 client.17886 shutdown 2024-04-25T11:42:38.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.351+0000 7fe2527b6740 1 -- 172.21.15.90:0/658757768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224c5390 msgr2=0x55eb228816e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.351+0000 7fe2527b6740 1 --2- 172.21.15.90:0/658757768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224c5390 0x55eb228816e0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fe23c031450 tx=0x7fe23c02a8c0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.351+0000 7fe2527b6740 1 -- 172.21.15.90:0/658757768 shutdown_connections 2024-04-25T11:42:38.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.351+0000 7fe2527b6740 1 --2- 172.21.15.90:0/658757768 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55eb22483f60 0x55eb22887940 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.351+0000 7fe2527b6740 1 --2- 172.21.15.90:0/658757768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224c5390 0x55eb228816e0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.351+0000 7fe2527b6740 1 --2- 172.21.15.90:0/658757768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb224a46b0 0x55eb228758b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.351+0000 7fe2527b6740 1 --2- 172.21.15.90:0/658757768 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb2249c4d0 0x55eb22875370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.351+0000 7fe22cff9640 0 client.17886 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:42:38.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.351+0000 7fe2527b6740 1 -- 172.21.15.90:0/658757768 >> 172.21.15.90:0/658757768 conn(0x55eb224b94e0 msgr2=0x55eb228684c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.351+0000 7fe2527b6740 1 -- 172.21.15.90:0/658757768 shutdown_connections 2024-04-25T11:42:38.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.351+0000 7fe2527b6740 1 -- 172.21.15.90:0/658757768 wait complete. 2024-04-25T11:42:38.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.351+0000 7fe2527b6740 20 client.17886 trim_cache size 0 max 16384 2024-04-25T11:42:38.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.364+0000 7fe2527b6740 1 Processor -- start 2024-04-25T11:42:38.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.364+0000 7fe2527b6740 1 -- start start 2024-04-25T11:42:38.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.365+0000 7fe2527b6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb2249c4d0 0x55eb22881280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.365+0000 7fe2527b6740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb224a46b0 0x55eb2288d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.365+0000 7fe2527b6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224c5390 0x55eb2288d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.365+0000 7fe2527b6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe2441adab0 con 0x55eb224c5390 2024-04-25T11:42:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.365+0000 7fe2527b6740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe2441af5f0 con 0x55eb224a46b0 2024-04-25T11:42:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.365+0000 7fe2527b6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe2441ae180 con 0x55eb2249c4d0 2024-04-25T11:42:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.365+0000 7fe249dcd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224c5390 0x55eb2288d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.365+0000 7fe249dcd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224c5390 0x55eb2288d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60284/0 (socket says 172.21.15.90:60284) 2024-04-25T11:42:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.365+0000 7fe249dcd640 1 -- 172.21.15.90:0/2114333531 learned_addr learned my addr 172.21.15.90:0/2114333531 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.365+0000 7fe248dcb640 1 --2- 172.21.15.90:0/2114333531 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb224a46b0 0x55eb2288d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.365+0000 7fe2495cc640 1 --2- 172.21.15.90:0/2114333531 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb2249c4d0 0x55eb22881280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.365+0000 7fe249dcd640 1 -- 172.21.15.90:0/2114333531 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb2249c4d0 msgr2=0x55eb22881280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.365+0000 7fe249dcd640 1 --2- 172.21.15.90:0/2114333531 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb2249c4d0 0x55eb22881280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.365+0000 7fe249dcd640 1 -- 172.21.15.90:0/2114333531 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb224a46b0 msgr2=0x55eb2288d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.365+0000 7fe249dcd640 1 --2- 172.21.15.90:0/2114333531 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb224a46b0 0x55eb2288d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.365+0000 7fe249dcd640 1 -- 172.21.15.90:0/2114333531 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb22866c40 con 0x55eb224c5390 2024-04-25T11:42:38.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.365+0000 7fe2495cc640 1 --2- 172.21.15.90:0/2114333531 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb2249c4d0 0x55eb22881280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:42:38.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.365+0000 7fe248dcb640 1 --2- 172.21.15.90:0/2114333531 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb224a46b0 0x55eb2288d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:42:38.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.365+0000 7fe249dcd640 1 --2- 172.21.15.90:0/2114333531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224c5390 0x55eb2288d640 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fe23c009940 tx=0x7fe23c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5e2b197946baaa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.365+0000 7fe22effd640 1 -- 172.21.15.90:0/2114333531 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe23c003d00 con 0x55eb224c5390 2024-04-25T11:42:38.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.365+0000 7fe22effd640 1 -- 172.21.15.90:0/2114333531 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe23c036040 con 0x55eb224c5390 2024-04-25T11:42:38.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.365+0000 7fe22effd640 1 -- 172.21.15.90:0/2114333531 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe23c02fb90 con 0x55eb224c5390 2024-04-25T11:42:38.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.366+0000 7fe2527b6740 1 -- 172.21.15.90:0/2114333531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224c5390 msgr2=0x55eb2288d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.366+0000 7fe2527b6740 1 --2- 172.21.15.90:0/2114333531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224c5390 0x55eb2288d640 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fe23c009940 tx=0x7fe23c0095c0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.366+0000 7fe2527b6740 1 -- 172.21.15.90:0/2114333531 shutdown_connections 2024-04-25T11:42:38.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.366+0000 7fe2527b6740 1 --2- 172.21.15.90:0/2114333531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224c5390 0x55eb2288d640 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.366+0000 7fe2527b6740 1 --2- 172.21.15.90:0/2114333531 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb224a46b0 0x55eb2288d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.366+0000 7fe2527b6740 1 --2- 172.21.15.90:0/2114333531 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb2249c4d0 0x55eb22881280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.366+0000 7fe2527b6740 1 -- 172.21.15.90:0/2114333531 >> 172.21.15.90:0/2114333531 conn(0x55eb224b94e0 msgr2=0x55eb22868eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.366+0000 7fe2527b6740 1 -- 172.21.15.90:0/2114333531 shutdown_connections 2024-04-25T11:42:38.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.366+0000 7fe2527b6740 1 -- 172.21.15.90:0/2114333531 wait complete. 2024-04-25T11:42:38.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.366+0000 7fe2527b6740 1 -- start start 2024-04-25T11:42:38.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.367+0000 7fe2527b6740 1 Processor -- start 2024-04-25T11:42:38.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.367+0000 7fe2527b6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb2249c4d0 0x55eb2287db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.367+0000 7fe2527b6740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb224a46b0 0x55eb2287af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.367+0000 7fe2527b6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224c5390 0x55eb2287b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.367+0000 7fe2527b6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe2441aef20 con 0x55eb224c5390 2024-04-25T11:42:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.367+0000 7fe2527b6740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe2441ae850 con 0x55eb224a46b0 2024-04-25T11:42:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.367+0000 7fe2527b6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe2441abab0 con 0x55eb2249c4d0 2024-04-25T11:42:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.367+0000 7fe2527b6740 10 client.0 _ll_register_callbacks cb 0x7fe23c0093f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.367+0000 7fe249dcd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224c5390 0x55eb2287b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.367+0000 7fe249dcd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224c5390 0x55eb2287b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60300/0 (socket says 172.21.15.90:60300) 2024-04-25T11:42:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.367+0000 7fe249dcd640 1 -- 172.21.15.90:0/3403627244 learned_addr learned my addr 172.21.15.90:0/3403627244 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.367+0000 7fe248dcb640 1 --2- 172.21.15.90:0/3403627244 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb224a46b0 0x55eb2287af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.367+0000 7fe2495cc640 1 --2- 172.21.15.90:0/3403627244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb2249c4d0 0x55eb2287db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.367+0000 7fe249dcd640 1 -- 172.21.15.90:0/3403627244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb2249c4d0 msgr2=0x55eb2287db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.367+0000 7fe249dcd640 1 --2- 172.21.15.90:0/3403627244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb2249c4d0 0x55eb2287db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.367+0000 7fe249dcd640 1 -- 172.21.15.90:0/3403627244 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb224a46b0 msgr2=0x55eb2287af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.367+0000 7fe249dcd640 1 --2- 172.21.15.90:0/3403627244 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb224a46b0 0x55eb2287af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.367+0000 7fe249dcd640 1 -- 172.21.15.90:0/3403627244 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb2287bbc0 con 0x55eb224c5390 2024-04-25T11:42:38.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.368+0000 7fe249dcd640 1 --2- 172.21.15.90:0/3403627244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224c5390 0x55eb2287b480 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fe23c0094c0 tx=0x7fe23c02fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afdf36640f5da67f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.368+0000 7fe22cff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:38.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.368+0000 7fe22cff9640 1 -- 172.21.15.90:0/3403627244 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe23c0047f0 con 0x55eb224c5390 2024-04-25T11:42:38.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.368+0000 7fe22cff9640 1 -- 172.21.15.90:0/3403627244 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7fe23c0305c0 con 0x55eb224c5390 2024-04-25T11:42:38.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.368+0000 7fe2527b6740 1 -- 172.21.15.90:0/3403627244 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe23c036040 con 0x55eb224c5390 2024-04-25T11:42:38.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.368+0000 7fe22cff9640 1 -- 172.21.15.90:0/3403627244 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe23c032ea0 con 0x55eb224c5390 2024-04-25T11:42:38.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.368+0000 7fe2527b6740 10 client.17892 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:38.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.368+0000 7fe2527b6740 1 -- 172.21.15.90:0/3403627244 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55eb228efb20 con 0x55eb224c5390 2024-04-25T11:42:38.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.368+0000 7fe22cff9640 1 -- 172.21.15.90:0/3403627244 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe23c036040 con 0x55eb224c5390 2024-04-25T11:42:38.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.368+0000 7fe22cff9640 1 -- 172.21.15.90:0/3403627244 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fe23c074030 con 0x55eb224c5390 2024-04-25T11:42:38.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.368+0000 7fe2527b6740 10 client.17892 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:38.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.368+0000 7fe2527b6740 10 client.17892 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:42:38.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.368+0000 7fe2527b6740 20 client.17892 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:38.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.368+0000 7fe2527b6740 1 --2- 172.21.15.90:0/3403627244 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55eb224593a0 0x55eb228f2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.368+0000 7fe2527b6740 4 client.17892 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:42:38.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.368+0000 7fe2527b6740 1 -- 172.21.15.90:0/3403627244 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55eb2274e610 con 0x55eb224593a0 2024-04-25T11:42:38.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.369+0000 7fe2495cc640 1 --2- 172.21.15.90:0/3403627244 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55eb224593a0 0x55eb228f2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.369+0000 7fe22cff9640 10 client.17892 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:42:38.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.369+0000 7fe2495cc640 1 --2- 172.21.15.90:0/3403627244 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55eb224593a0 0x55eb228f2170 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d0f7d334b8ba5183 server_cookie=e87b58f2522f4c8e in_seq=0 out_seq=0 2024-04-25T11:42:38.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.369+0000 7fe22cff9640 1 -- 172.21.15.90:0/3403627244 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55eb2274e610 con 0x55eb224593a0 2024-04-25T11:42:38.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.369+0000 7fe22cff9640 10 client.17892 handle_command_reply: tid=0 2024-04-25T11:42:38.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.369+0000 7fe2527b6740 1 client.17892 shutdown 2024-04-25T11:42:38.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.369+0000 7fe2527b6740 1 -- 172.21.15.90:0/3403627244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224c5390 msgr2=0x55eb2287b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.369+0000 7fe2527b6740 1 --2- 172.21.15.90:0/3403627244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224c5390 0x55eb2287b480 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fe23c0094c0 tx=0x7fe23c02fce0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.369+0000 7fe2527b6740 1 -- 172.21.15.90:0/3403627244 shutdown_connections 2024-04-25T11:42:38.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.369+0000 7fe2527b6740 1 --2- 172.21.15.90:0/3403627244 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55eb224593a0 0x55eb228f2170 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.369+0000 7fe2527b6740 1 --2- 172.21.15.90:0/3403627244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb224c5390 0x55eb2287b480 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.369+0000 7fe2527b6740 1 --2- 172.21.15.90:0/3403627244 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55eb224a46b0 0x55eb2287af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.369+0000 7fe22cff9640 0 client.17892 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:42:38.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.369+0000 7fe2527b6740 1 --2- 172.21.15.90:0/3403627244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb2249c4d0 0x55eb2287db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.369+0000 7fe2527b6740 1 -- 172.21.15.90:0/3403627244 >> 172.21.15.90:0/3403627244 conn(0x55eb224b94e0 msgr2=0x55eb22877450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.370+0000 7fe2527b6740 1 -- 172.21.15.90:0/3403627244 shutdown_connections 2024-04-25T11:42:38.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.370+0000 7fe2527b6740 1 -- 172.21.15.90:0/3403627244 wait complete. 2024-04-25T11:42:38.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.370+0000 7fe2527b6740 20 client.17892 trim_cache size 0 max 16384 2024-04-25T11:42:38.363 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:42:38.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.370+0000 7fe24b830640 1 -- 172.21.15.90:0/723854997 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe2300afc60 msgr2=0x7fe2300b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.370+0000 7fe24b830640 1 --2- 172.21.15.90:0/723854997 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe2300afc60 0x7fe2300b2170 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe23c005cc0 tx=0x7fe23c005c30 comp rx=0 tx=0).stop 2024-04-25T11:42:38.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.370+0000 7fe24b830640 1 -- 172.21.15.90:0/723854997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe244109b10 msgr2=0x7fe24419ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.370+0000 7fe24b830640 1 --2- 172.21.15.90:0/723854997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe244109b10 0x7fe24419ac40 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fe23400c970 tx=0x7fe23400ce30 comp rx=0 tx=0).stop 2024-04-25T11:42:38.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.370+0000 7fe24b830640 1 -- 172.21.15.90:0/723854997 shutdown_connections 2024-04-25T11:42:38.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.370+0000 7fe24b830640 1 --2- 172.21.15.90:0/723854997 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe2300afc60 0x7fe2300b2170 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.370+0000 7fe24b830640 1 --2- 172.21.15.90:0/723854997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe244109fb0 0x7fe24419ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.370+0000 7fe24b830640 1 --2- 172.21.15.90:0/723854997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe244109b10 0x7fe24419ac40 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.370+0000 7fe24b830640 1 --2- 172.21.15.90:0/723854997 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe244104260 0x7fe24419a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.370+0000 7fe24b830640 1 -- 172.21.15.90:0/723854997 >> 172.21.15.90:0/723854997 conn(0x7fe2440ff1d0 msgr2=0x7fe244101a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.370+0000 7fe24b830640 1 -- 172.21.15.90:0/723854997 shutdown_connections 2024-04-25T11:42:38.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.370+0000 7fe24b830640 1 -- 172.21.15.90:0/723854997 wait complete. 2024-04-25T11:42:38.368 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T11:42:38.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.492+0000 7f9156834640 1 -- 172.21.15.90:0/3245276611 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9150106990 msgr2=0x7f9150106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.492+0000 7f9156834640 1 --2- 172.21.15.90:0/3245276611 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9150106990 0x7f9150106dc0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f9138009900 tx=0x7f913802a410 comp rx=0 tx=0).stop 2024-04-25T11:42:38.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.492+0000 7f9156834640 1 -- 172.21.15.90:0/3245276611 shutdown_connections 2024-04-25T11:42:38.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.492+0000 7f9156834640 1 --2- 172.21.15.90:0/3245276611 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9150107450 0x7f915010ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.492+0000 7f9156834640 1 --2- 172.21.15.90:0/3245276611 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9150106990 0x7f9150106dc0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.492+0000 7f9156834640 1 --2- 172.21.15.90:0/3245276611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9150102400 0x7f9150102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.492+0000 7f9156834640 1 -- 172.21.15.90:0/3245276611 >> 172.21.15.90:0/3245276611 conn(0x7f9150077880 msgr2=0x7f9150077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.492+0000 7f9156834640 1 -- 172.21.15.90:0/3245276611 shutdown_connections 2024-04-25T11:42:38.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.492+0000 7f9156834640 1 -- 172.21.15.90:0/3245276611 wait complete. 2024-04-25T11:42:38.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.493+0000 7f9156834640 1 Processor -- start 2024-04-25T11:42:38.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.493+0000 7f9156834640 1 -- start start 2024-04-25T11:42:38.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.493+0000 7f9156834640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9150102400 0x7f915019a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.493+0000 7f9156834640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9150106990 0x7f915019acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.493+0000 7f9156834640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9150107450 0x7f915019ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.493+0000 7f9156834640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9150113cf0 con 0x7f9150102400 2024-04-25T11:42:38.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.493+0000 7f9156834640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9150113b70 con 0x7f9150106990 2024-04-25T11:42:38.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.493+0000 7f9156834640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9150113e70 con 0x7f9150107450 2024-04-25T11:42:38.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.493+0000 7f914ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9150102400 0x7f915019a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.493+0000 7f914ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9150102400 0x7f915019a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60318/0 (socket says 172.21.15.90:60318) 2024-04-25T11:42:38.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.493+0000 7f914ffff640 1 -- 172.21.15.90:0/177157151 learned_addr learned my addr 172.21.15.90:0/177157151 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.493+0000 7f914f7fe640 1 --2- 172.21.15.90:0/177157151 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9150106990 0x7f915019acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.493+0000 7f914ffff640 1 -- 172.21.15.90:0/177157151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9150107450 msgr2=0x7f915019ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.493+0000 7f914ffff640 1 --2- 172.21.15.90:0/177157151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9150107450 0x7f915019ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.493+0000 7f914ffff640 1 -- 172.21.15.90:0/177157151 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9150106990 msgr2=0x7f915019acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.493+0000 7f914ffff640 1 --2- 172.21.15.90:0/177157151 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9150106990 0x7f915019acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.493+0000 7f914ffff640 1 -- 172.21.15.90:0/177157151 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f915019f5d0 con 0x7f9150102400 2024-04-25T11:42:38.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.494+0000 7f914ffff640 1 --2- 172.21.15.90:0/177157151 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9150102400 0x7f915019a770 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f914000d950 tx=0x7f914000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.494+0000 7f914d7fa640 1 -- 172.21.15.90:0/177157151 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9140017070 con 0x7f9150102400 2024-04-25T11:42:38.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.494+0000 7f9156834640 1 -- 172.21.15.90:0/177157151 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f915019f8c0 con 0x7f9150102400 2024-04-25T11:42:38.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.494+0000 7f914d7fa640 1 -- 172.21.15.90:0/177157151 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9140009c30 con 0x7f9150102400 2024-04-25T11:42:38.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.494+0000 7f914d7fa640 1 -- 172.21.15.90:0/177157151 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914000f620 con 0x7f9150102400 2024-04-25T11:42:38.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.494+0000 7f9156834640 1 -- 172.21.15.90:0/177157151 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91501a7240 con 0x7f9150102400 2024-04-25T11:42:38.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.497+0000 7f915d7c3740 1 Processor -- start 2024-04-25T11:42:38.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.497+0000 7f915d7c3740 1 -- start start 2024-04-25T11:42:38.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.497+0000 7f914d7fa640 1 -- 172.21.15.90:0/177157151 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f9140010460 con 0x7f9150102400 2024-04-25T11:42:38.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.497+0000 7f915d7c3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c8619b390 0x561c8653c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.497+0000 7f915d7c3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c861724d0 0x561c8653cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.497+0000 7f915d7c3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c8617a6b0 0x561c865496a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.497+0000 7f915d7c3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561c864a1d80 con 0x561c8619b390 2024-04-25T11:42:38.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.497+0000 7f915d7c3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x561c8645ba20 con 0x561c861724d0 2024-04-25T11:42:38.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.497+0000 7f915d7c3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561c8649dd40 con 0x561c8617a6b0 2024-04-25T11:42:38.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.497+0000 7f9154dd1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c8619b390 0x561c8653c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.497+0000 7f9154dd1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c8619b390 0x561c8653c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60332/0 (socket says 172.21.15.90:60332) 2024-04-25T11:42:38.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.497+0000 7f9154dd1640 1 -- 172.21.15.90:0/2257625252 learned_addr learned my addr 172.21.15.90:0/2257625252 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.497+0000 7f914ffff640 1 --2- 172.21.15.90:0/2257625252 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c861724d0 0x561c8653cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.498+0000 7f9154dd1640 1 -- 172.21.15.90:0/2257625252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c8617a6b0 msgr2=0x561c865496a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.498+0000 7f914f7fe640 1 --2- 172.21.15.90:0/2257625252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c8617a6b0 0x561c865496a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.498+0000 7f9154dd1640 1 --2- 172.21.15.90:0/2257625252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c8617a6b0 0x561c865496a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.498+0000 7f9154dd1640 1 -- 172.21.15.90:0/2257625252 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c861724d0 msgr2=0x561c8653cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.498+0000 7f9154dd1640 1 --2- 172.21.15.90:0/2257625252 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c861724d0 0x561c8653cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.498+0000 7f9154dd1640 1 -- 172.21.15.90:0/2257625252 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c86466c30 con 0x561c8619b390 2024-04-25T11:42:38.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.498+0000 7f9154dd1640 1 --2- 172.21.15.90:0/2257625252 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c8619b390 0x561c8653c810 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f9148009e20 tx=0x7f914800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cf141ebd27b3ebf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.498+0000 7f914d7fa640 1 --2- 172.21.15.90:0/177157151 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f913c0afc60 0x7f913c0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.498+0000 7f914ffff640 1 --2- 172.21.15.90:0/177157151 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f913c0afc60 0x7f913c0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.498+0000 7f91467fc640 1 -- 172.21.15.90:0/2257625252 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9148004d20 con 0x561c8619b390 2024-04-25T11:42:38.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.498+0000 7f91467fc640 1 -- 172.21.15.90:0/2257625252 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f914800ee70 con 0x561c8619b390 2024-04-25T11:42:38.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.498+0000 7f91467fc640 1 -- 172.21.15.90:0/2257625252 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9148018870 con 0x561c8619b390 2024-04-25T11:42:38.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.498+0000 7f914d7fa640 1 -- 172.21.15.90:0/177157151 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7f91400ded20 con 0x7f9150102400 2024-04-25T11:42:38.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.499+0000 7f915d7c3740 1 -- 172.21.15.90:0/2257625252 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c8619b390 msgr2=0x561c8653c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.499+0000 7f914ffff640 1 --2- 172.21.15.90:0/177157151 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f913c0afc60 0x7f913c0b2170 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f91400dc490 tx=0x7f9140000bc0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.499+0000 7f915d7c3740 1 --2- 172.21.15.90:0/2257625252 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c8619b390 0x561c8653c810 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f9148009e20 tx=0x7f914800e720 comp rx=0 tx=0).stop 2024-04-25T11:42:38.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.499+0000 7f915d7c3740 1 -- 172.21.15.90:0/2257625252 shutdown_connections 2024-04-25T11:42:38.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.499+0000 7f915d7c3740 1 --2- 172.21.15.90:0/2257625252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c8617a6b0 0x561c865496a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.499+0000 7f915d7c3740 1 --2- 172.21.15.90:0/2257625252 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c861724d0 0x561c8653cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.499+0000 7f915d7c3740 1 --2- 172.21.15.90:0/2257625252 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c8619b390 0x561c8653c810 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.499+0000 7f915d7c3740 1 -- 172.21.15.90:0/2257625252 >> 172.21.15.90:0/2257625252 conn(0x561c8618f4e0 msgr2=0x561c86538fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.499+0000 7f915d7c3740 1 -- 172.21.15.90:0/2257625252 shutdown_connections 2024-04-25T11:42:38.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.499+0000 7f915d7c3740 1 -- 172.21.15.90:0/2257625252 wait complete. 2024-04-25T11:42:38.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.499+0000 7f915d7c3740 1 -- start start 2024-04-25T11:42:38.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.500+0000 7f915d7c3740 1 Processor -- start 2024-04-25T11:42:38.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.500+0000 7f915d7c3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c861724d0 0x561c86547260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.500+0000 7f915d7c3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c8617a6b0 0x561c865477a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.500+0000 7f915d7c3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c8619b390 0x561c86553210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.500+0000 7f915d7c3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561c86549d90 con 0x561c8619b390 2024-04-25T11:42:38.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.500+0000 7f915d7c3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x561c86428dc0 con 0x561c8617a6b0 2024-04-25T11:42:38.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.500+0000 7f915d7c3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561c86549f10 con 0x561c861724d0 2024-04-25T11:42:38.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.500+0000 7f915d7c3740 10 client.0 _ll_register_callbacks cb 0x561c86444600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:38.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.500+0000 7f9154dd1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c8619b390 0x561c86553210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.500+0000 7f9154dd1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c8619b390 0x561c86553210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60342/0 (socket says 172.21.15.90:60342) 2024-04-25T11:42:38.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.500+0000 7f9154dd1640 1 -- 172.21.15.90:0/3578324337 learned_addr learned my addr 172.21.15.90:0/3578324337 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.500+0000 7f914f7fe640 1 --2- 172.21.15.90:0/3578324337 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c8617a6b0 0x561c865477a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.500+0000 7f9154dd1640 1 --2- 172.21.15.90:0/3578324337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c861724d0 0x561c86547260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.500+0000 7f9154dd1640 1 -- 172.21.15.90:0/3578324337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c861724d0 msgr2=0x561c86547260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.500+0000 7f9154dd1640 1 --2- 172.21.15.90:0/3578324337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c861724d0 0x561c86547260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.500+0000 7f9154dd1640 1 -- 172.21.15.90:0/3578324337 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c8617a6b0 msgr2=0x561c865477a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.500+0000 7f9154dd1640 1 --2- 172.21.15.90:0/3578324337 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c8617a6b0 0x561c865477a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.500+0000 7f9154dd1640 1 -- 172.21.15.90:0/3578324337 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c861b33f0 con 0x561c8619b390 2024-04-25T11:42:38.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.500+0000 7f914f7fe640 1 --2- 172.21.15.90:0/3578324337 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c8617a6b0 0x561c865477a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:42:38.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.500+0000 7f9154dd1640 1 --2- 172.21.15.90:0/3578324337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c8619b390 0x561c86553210 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f9148000f80 tx=0x7f9148018510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd794348727eefb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.500+0000 7f912f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:38.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.501+0000 7f912f7fe640 1 -- 172.21.15.90:0/3578324337 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914801c070 con 0x561c8619b390 2024-04-25T11:42:38.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.501+0000 7f912f7fe640 1 -- 172.21.15.90:0/3578324337 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7f9148003c40 con 0x561c8619b390 2024-04-25T11:42:38.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.501+0000 7f915d7c3740 1 -- 172.21.15.90:0/3578324337 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f914800ee70 con 0x561c8619b390 2024-04-25T11:42:38.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.501+0000 7f912f7fe640 1 -- 172.21.15.90:0/3578324337 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f914802d050 con 0x561c8619b390 2024-04-25T11:42:38.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.501+0000 7f915d7c3740 10 client.17904 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:38.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.501+0000 7f915d7c3740 1 -- 172.21.15.90:0/3578324337 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561c8627d7d0 con 0x561c8619b390 2024-04-25T11:42:38.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.501+0000 7f912f7fe640 1 -- 172.21.15.90:0/3578324337 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f914800ee70 con 0x561c8619b390 2024-04-25T11:42:38.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.501+0000 7f912f7fe640 1 -- 172.21.15.90:0/3578324337 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f91480054d0 con 0x561c8619b390 2024-04-25T11:42:38.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.501+0000 7f915d7c3740 10 client.17904 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:38.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.501+0000 7f915d7c3740 10 client.17904 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:42:38.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.501+0000 7f915d7c3740 20 client.17904 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:38.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.501+0000 7f915d7c3740 1 --2- 172.21.15.90:0/3578324337 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x561c86159f60 0x561c865594f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.501+0000 7f915d7c3740 4 client.17904 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:42:38.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.501+0000 7f915d7c3740 1 -- 172.21.15.90:0/3578324337 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c864af990 con 0x561c86159f60 2024-04-25T11:42:38.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.501+0000 7f914f7fe640 1 --2- 172.21.15.90:0/3578324337 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x561c86159f60 0x561c865594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.502+0000 7f912f7fe640 10 client.17904 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:42:38.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.502+0000 7f914f7fe640 1 --2- 172.21.15.90:0/3578324337 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x561c86159f60 0x561c865594f0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2f02a7ab7291ed98 server_cookie=a5b1236c8a577a03 in_seq=0 out_seq=0 2024-04-25T11:42:38.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.503+0000 7f912f7fe640 1 -- 172.21.15.90:0/3578324337 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561c864af990 con 0x561c86159f60 2024-04-25T11:42:38.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.503+0000 7f912f7fe640 10 client.17904 handle_command_reply: tid=0 2024-04-25T11:42:38.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.503+0000 7f915d7c3740 1 client.17904 shutdown 2024-04-25T11:42:38.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.503+0000 7f915d7c3740 1 -- 172.21.15.90:0/3578324337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c8619b390 msgr2=0x561c86553210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.503+0000 7f915d7c3740 1 --2- 172.21.15.90:0/3578324337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c8619b390 0x561c86553210 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f9148000f80 tx=0x7f9148018510 comp rx=0 tx=0).stop 2024-04-25T11:42:38.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.503+0000 7f915d7c3740 1 -- 172.21.15.90:0/3578324337 shutdown_connections 2024-04-25T11:42:38.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.503+0000 7f915d7c3740 1 --2- 172.21.15.90:0/3578324337 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x561c86159f60 0x561c865594f0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.503+0000 7f915d7c3740 1 --2- 172.21.15.90:0/3578324337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c8619b390 0x561c86553210 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.503+0000 7f915d7c3740 1 --2- 172.21.15.90:0/3578324337 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c8617a6b0 0x561c865477a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.503+0000 7f915d7c3740 1 --2- 172.21.15.90:0/3578324337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c861724d0 0x561c86547260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.503+0000 7f912f7fe640 0 client.17904 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:42:38.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.503+0000 7f915d7c3740 1 -- 172.21.15.90:0/3578324337 >> 172.21.15.90:0/3578324337 conn(0x561c8618f4e0 msgr2=0x561c8653e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.503+0000 7f915d7c3740 1 -- 172.21.15.90:0/3578324337 shutdown_connections 2024-04-25T11:42:38.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.503+0000 7f915d7c3740 1 -- 172.21.15.90:0/3578324337 wait complete. 2024-04-25T11:42:38.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.503+0000 7f915d7c3740 20 client.17904 trim_cache size 0 max 16384 2024-04-25T11:42:38.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.516+0000 7f915d7c3740 1 Processor -- start 2024-04-25T11:42:38.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.516+0000 7f915d7c3740 1 -- start start 2024-04-25T11:42:38.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.517+0000 7f915d7c3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c861724d0 0x561c865491a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.517+0000 7f915d7c3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c8617a6b0 0x561c86552e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.517+0000 7f915d7c3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c8619b390 0x561c86537d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.517+0000 7f915d7c3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f91501afd90 con 0x561c8617a6b0 2024-04-25T11:42:38.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.517+0000 7f915d7c3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f91501b18d0 con 0x561c8619b390 2024-04-25T11:42:38.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.517+0000 7f915d7c3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f91501b0460 con 0x561c861724d0 2024-04-25T11:42:38.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.517+0000 7f914f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c8617a6b0 0x561c86552e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.517+0000 7f914f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c8617a6b0 0x561c86552e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60348/0 (socket says 172.21.15.90:60348) 2024-04-25T11:42:38.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.517+0000 7f914f7fe640 1 -- 172.21.15.90:0/3338891189 learned_addr learned my addr 172.21.15.90:0/3338891189 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.517+0000 7f9154dd1640 1 --2- 172.21.15.90:0/3338891189 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c8619b390 0x561c86537d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.517+0000 7f914f7fe640 1 -- 172.21.15.90:0/3338891189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c861724d0 msgr2=0x561c865491a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.517+0000 7f9154dd1640 1 --2- 172.21.15.90:0/3338891189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c861724d0 0x561c865491a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.517+0000 7f914f7fe640 1 --2- 172.21.15.90:0/3338891189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c861724d0 0x561c865491a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.517+0000 7f914f7fe640 1 -- 172.21.15.90:0/3338891189 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c8619b390 msgr2=0x561c86537d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.517+0000 7f914f7fe640 1 --2- 172.21.15.90:0/3338891189 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c8619b390 0x561c86537d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.517+0000 7f914f7fe640 1 -- 172.21.15.90:0/3338891189 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c86549eb0 con 0x561c8617a6b0 2024-04-25T11:42:38.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.517+0000 7f9154dd1640 1 --2- 172.21.15.90:0/3338891189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c861724d0 0x561c865491a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:42:38.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.517+0000 7f914f7fe640 1 --2- 172.21.15.90:0/3338891189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c8617a6b0 0x561c86552e70 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f9138000c00 tx=0x7f9138031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1331fceb20dfb245 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.517+0000 7f9145ffb640 1 -- 172.21.15.90:0/3338891189 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9138035040 con 0x561c8617a6b0 2024-04-25T11:42:38.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.517+0000 7f9145ffb640 1 -- 172.21.15.90:0/3338891189 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9138048070 con 0x561c8617a6b0 2024-04-25T11:42:38.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.517+0000 7f9145ffb640 1 -- 172.21.15.90:0/3338891189 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91380049b0 con 0x561c8617a6b0 2024-04-25T11:42:38.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.518+0000 7f915d7c3740 1 -- 172.21.15.90:0/3338891189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c8617a6b0 msgr2=0x561c86552e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.518+0000 7f915d7c3740 1 --2- 172.21.15.90:0/3338891189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c8617a6b0 0x561c86552e70 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f9138000c00 tx=0x7f9138031400 comp rx=0 tx=0).stop 2024-04-25T11:42:38.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.518+0000 7f915d7c3740 1 -- 172.21.15.90:0/3338891189 shutdown_connections 2024-04-25T11:42:38.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.518+0000 7f915d7c3740 1 --2- 172.21.15.90:0/3338891189 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c8619b390 0x561c86537d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.518+0000 7f915d7c3740 1 --2- 172.21.15.90:0/3338891189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c8617a6b0 0x561c86552e70 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.518+0000 7f915d7c3740 1 --2- 172.21.15.90:0/3338891189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c861724d0 0x561c865491a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.518+0000 7f915d7c3740 1 -- 172.21.15.90:0/3338891189 >> 172.21.15.90:0/3338891189 conn(0x561c8618f4e0 msgr2=0x561c86537610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.518+0000 7f915d7c3740 1 -- 172.21.15.90:0/3338891189 shutdown_connections 2024-04-25T11:42:38.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.518+0000 7f915d7c3740 1 -- 172.21.15.90:0/3338891189 wait complete. 2024-04-25T11:42:38.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.519+0000 7f915d7c3740 1 -- start start 2024-04-25T11:42:38.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.519+0000 7f915d7c3740 1 Processor -- start 2024-04-25T11:42:38.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.519+0000 7f915d7c3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c861724d0 0x561c8654c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.519+0000 7f915d7c3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c8617a6b0 0x561c8654cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.519+0000 7f915d7c3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c8619b390 0x561c865bf700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.519+0000 7f915d7c3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f91501b1200 con 0x561c861724d0 2024-04-25T11:42:38.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.519+0000 7f915d7c3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f91501b0b30 con 0x561c8617a6b0 2024-04-25T11:42:38.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.519+0000 7f915d7c3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f91501add90 con 0x561c8619b390 2024-04-25T11:42:38.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.519+0000 7f915d7c3740 10 client.0 _ll_register_callbacks cb 0x7f9148009d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:38.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.519+0000 7f9154dd1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c861724d0 0x561c8654c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.519+0000 7f9154dd1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c861724d0 0x561c8654c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60350/0 (socket says 172.21.15.90:60350) 2024-04-25T11:42:38.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.519+0000 7f9154dd1640 1 -- 172.21.15.90:0/798190942 learned_addr learned my addr 172.21.15.90:0/798190942 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.519+0000 7f914f7fe640 1 --2- 172.21.15.90:0/798190942 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c8617a6b0 0x561c8654cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.520+0000 7f9154dd1640 1 -- 172.21.15.90:0/798190942 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c8619b390 msgr2=0x561c865bf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.520+0000 7f9154dd1640 1 --2- 172.21.15.90:0/798190942 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c8619b390 0x561c865bf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.520+0000 7f9154dd1640 1 -- 172.21.15.90:0/798190942 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c8617a6b0 msgr2=0x561c8654cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.520+0000 7f9154dd1640 1 --2- 172.21.15.90:0/798190942 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c8617a6b0 0x561c8654cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.520+0000 7f9154dd1640 1 -- 172.21.15.90:0/798190942 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c865bfc40 con 0x561c861724d0 2024-04-25T11:42:38.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.520+0000 7f9154dd1640 1 --2- 172.21.15.90:0/798190942 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c861724d0 0x561c8654c5c0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f9148004210 tx=0x7f9148004b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4db1b21863b5699 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.520+0000 7f912f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:38.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.520+0000 7f912f7fe640 1 -- 172.21.15.90:0/798190942 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9148035020 con 0x561c861724d0 2024-04-25T11:42:38.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.520+0000 7f912f7fe640 1 -- 172.21.15.90:0/798190942 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7f9148003c40 con 0x561c861724d0 2024-04-25T11:42:38.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.520+0000 7f915d7c3740 1 -- 172.21.15.90:0/798190942 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9138048070 con 0x561c861724d0 2024-04-25T11:42:38.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.520+0000 7f912f7fe640 1 -- 172.21.15.90:0/798190942 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9148040050 con 0x561c861724d0 2024-04-25T11:42:38.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.520+0000 7f915d7c3740 10 client.17910 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:38.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.520+0000 7f915d7c3740 1 -- 172.21.15.90:0/798190942 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561c865c0dd0 con 0x561c861724d0 2024-04-25T11:42:38.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.520+0000 7f912f7fe640 1 -- 172.21.15.90:0/798190942 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9138048070 con 0x561c861724d0 2024-04-25T11:42:38.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.521+0000 7f912f7fe640 1 -- 172.21.15.90:0/798190942 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f914803c4b0 con 0x561c861724d0 2024-04-25T11:42:38.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.521+0000 7f915d7c3740 10 client.17910 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:38.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.521+0000 7f915d7c3740 10 client.17910 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:42:38.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.521+0000 7f915d7c3740 20 client.17910 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:38.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.521+0000 7f915d7c3740 1 --2- 172.21.15.90:0/798190942 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x561c8612f3a0 0x561c865c3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.521+0000 7f915d7c3740 4 client.17910 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:42:38.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.521+0000 7f915d7c3740 1 -- 172.21.15.90:0/798190942 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561c864a9800 con 0x561c8612f3a0 2024-04-25T11:42:38.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.521+0000 7f914f7fe640 1 --2- 172.21.15.90:0/798190942 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x561c8612f3a0 0x561c865c3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.521+0000 7f912f7fe640 10 client.17910 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:42:38.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.521+0000 7f914f7fe640 1 --2- 172.21.15.90:0/798190942 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x561c8612f3a0 0x561c865c3530 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b62ea80009775810 server_cookie=b994bdebc23175c5 in_seq=0 out_seq=0 2024-04-25T11:42:38.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.521+0000 7f912f7fe640 1 -- 172.21.15.90:0/798190942 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561c864a9800 con 0x561c8612f3a0 2024-04-25T11:42:38.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.521+0000 7f912f7fe640 10 client.17910 handle_command_reply: tid=0 2024-04-25T11:42:38.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.521+0000 7f915d7c3740 1 client.17910 shutdown 2024-04-25T11:42:38.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.522+0000 7f915d7c3740 1 -- 172.21.15.90:0/798190942 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c861724d0 msgr2=0x561c8654c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.522+0000 7f915d7c3740 1 --2- 172.21.15.90:0/798190942 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c861724d0 0x561c8654c5c0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f9148004210 tx=0x7f9148004b20 comp rx=0 tx=0).stop 2024-04-25T11:42:38.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.522+0000 7f915d7c3740 1 -- 172.21.15.90:0/798190942 shutdown_connections 2024-04-25T11:42:38.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.522+0000 7f915d7c3740 1 --2- 172.21.15.90:0/798190942 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x561c8612f3a0 0x561c865c3530 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.522+0000 7f915d7c3740 1 --2- 172.21.15.90:0/798190942 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c8619b390 0x561c865bf700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.522+0000 7f915d7c3740 1 --2- 172.21.15.90:0/798190942 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c8617a6b0 0x561c8654cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.522+0000 7f912f7fe640 0 client.17910 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:42:38.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.522+0000 7f915d7c3740 1 --2- 172.21.15.90:0/798190942 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c861724d0 0x561c8654c5c0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.522+0000 7f915d7c3740 1 -- 172.21.15.90:0/798190942 >> 172.21.15.90:0/798190942 conn(0x561c8618f4e0 msgr2=0x561c86538040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.522+0000 7f915d7c3740 1 -- 172.21.15.90:0/798190942 shutdown_connections 2024-04-25T11:42:38.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.522+0000 7f915d7c3740 1 -- 172.21.15.90:0/798190942 wait complete. 2024-04-25T11:42:38.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.522+0000 7f915d7c3740 20 client.17910 trim_cache size 0 max 16384 2024-04-25T11:42:38.514 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:42:38.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.522+0000 7f9156834640 1 -- 172.21.15.90:0/177157151 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f913c0afc60 msgr2=0x7f913c0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.522+0000 7f9156834640 1 --2- 172.21.15.90:0/177157151 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f913c0afc60 0x7f913c0b2170 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f91400dc490 tx=0x7f9140000bc0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.522+0000 7f9156834640 1 -- 172.21.15.90:0/177157151 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9150102400 msgr2=0x7f915019a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.522+0000 7f9156834640 1 --2- 172.21.15.90:0/177157151 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9150102400 0x7f915019a770 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f914000d950 tx=0x7f914000de10 comp rx=0 tx=0).stop 2024-04-25T11:42:38.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.523+0000 7f9156834640 1 -- 172.21.15.90:0/177157151 shutdown_connections 2024-04-25T11:42:38.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.523+0000 7f9156834640 1 --2- 172.21.15.90:0/177157151 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f913c0afc60 0x7f913c0b2170 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.523+0000 7f9156834640 1 --2- 172.21.15.90:0/177157151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9150107450 0x7f915019ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.523+0000 7f9156834640 1 --2- 172.21.15.90:0/177157151 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9150106990 0x7f915019acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.523+0000 7f9156834640 1 --2- 172.21.15.90:0/177157151 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9150102400 0x7f915019a770 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.523+0000 7f9156834640 1 -- 172.21.15.90:0/177157151 >> 172.21.15.90:0/177157151 conn(0x7f9150077880 msgr2=0x7f91501008c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.523+0000 7f9156834640 1 -- 172.21.15.90:0/177157151 shutdown_connections 2024-04-25T11:42:38.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.523+0000 7f9156834640 1 -- 172.21.15.90:0/177157151 wait complete. 2024-04-25T11:42:38.521 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T11:42:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.602+0000 7fb570cd8640 1 -- 172.21.15.90:0/1821115255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb56c1031e0 msgr2=0x7fb56c10f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.602+0000 7fb570cd8640 1 --2- 172.21.15.90:0/1821115255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb56c1031e0 0x7fb56c10f5b0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fb564009ca0 tx=0x7fb56402a8c0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.602+0000 7fb570cd8640 1 -- 172.21.15.90:0/1821115255 shutdown_connections 2024-04-25T11:42:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.602+0000 7fb570cd8640 1 --2- 172.21.15.90:0/1821115255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb56c1031e0 0x7fb56c10f5b0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.602+0000 7fb570cd8640 1 --2- 172.21.15.90:0/1821115255 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb56c108f30 0x7fb56c102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.602+0000 7fb570cd8640 1 --2- 172.21.15.90:0/1821115255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb56c108a90 0x7fb56c102630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.602+0000 7fb570cd8640 1 -- 172.21.15.90:0/1821115255 >> 172.21.15.90:0/1821115255 conn(0x7fb56c078080 msgr2=0x7fb56c0784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.602+0000 7fb570cd8640 1 -- 172.21.15.90:0/1821115255 shutdown_connections 2024-04-25T11:42:38.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.602+0000 7fb570cd8640 1 -- 172.21.15.90:0/1821115255 wait complete. 2024-04-25T11:42:38.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.603+0000 7fb570cd8640 1 Processor -- start 2024-04-25T11:42:38.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.603+0000 7fb570cd8640 1 -- start start 2024-04-25T11:42:38.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.603+0000 7fb570cd8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb56c1031e0 0x7fb56c0ff9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.603+0000 7fb570cd8640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb56c108a90 0x7fb56c0ffee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.603+0000 7fb570cd8640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb56c108f30 0x7fb56c100420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.603+0000 7fb570cd8640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb56c112ca0 con 0x7fb56c108f30 2024-04-25T11:42:38.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.603+0000 7fb570cd8640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb56c112b20 con 0x7fb56c108a90 2024-04-25T11:42:38.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.603+0000 7fb570cd8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb56c112e20 con 0x7fb56c1031e0 2024-04-25T11:42:38.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.603+0000 7fb56ad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb56c108f30 0x7fb56c100420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.603+0000 7fb56ad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb56c108f30 0x7fb56c100420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60376/0 (socket says 172.21.15.90:60376) 2024-04-25T11:42:38.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.603+0000 7fb56ad9d640 1 -- 172.21.15.90:0/1544744786 learned_addr learned my addr 172.21.15.90:0/1544744786 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.603+0000 7fb569d9b640 1 --2- 172.21.15.90:0/1544744786 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb56c108a90 0x7fb56c0ffee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.604+0000 7fb56a59c640 1 --2- 172.21.15.90:0/1544744786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb56c1031e0 0x7fb56c0ff9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.604+0000 7fb56ad9d640 1 -- 172.21.15.90:0/1544744786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb56c1031e0 msgr2=0x7fb56c0ff9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.604+0000 7fb56ad9d640 1 --2- 172.21.15.90:0/1544744786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb56c1031e0 0x7fb56c0ff9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.604+0000 7fb56ad9d640 1 -- 172.21.15.90:0/1544744786 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb56c108a90 msgr2=0x7fb56c0ffee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.604+0000 7fb56ad9d640 1 --2- 172.21.15.90:0/1544744786 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb56c108a90 0x7fb56c0ffee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.604+0000 7fb56ad9d640 1 -- 172.21.15.90:0/1544744786 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb56c1a8b10 con 0x7fb56c108f30 2024-04-25T11:42:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.604+0000 7fb56ad9d640 1 --2- 172.21.15.90:0/1544744786 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb56c108f30 0x7fb56c100420 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fb5640098d0 tx=0x7fb5640317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.604+0000 7fb55b7fe640 1 -- 172.21.15.90:0/1544744786 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb564003b70 con 0x7fb56c108f30 2024-04-25T11:42:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.604+0000 7fb55b7fe640 1 -- 172.21.15.90:0/1544744786 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb564003d10 con 0x7fb56c108f30 2024-04-25T11:42:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.604+0000 7fb570cd8640 1 -- 172.21.15.90:0/1544744786 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb56c1a8da0 con 0x7fb56c108f30 2024-04-25T11:42:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.604+0000 7fb55b7fe640 1 -- 172.21.15.90:0/1544744786 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56402fa20 con 0x7fb56c108f30 2024-04-25T11:42:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.604+0000 7fb570cd8640 1 -- 172.21.15.90:0/1544744786 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb56c1a9260 con 0x7fb56c108f30 2024-04-25T11:42:38.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.608+0000 7fb577b85740 1 Processor -- start 2024-04-25T11:42:38.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.608+0000 7fb55b7fe640 1 -- 172.21.15.90:0/1544744786 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fb56402fc20 con 0x7fb56c108f30 2024-04-25T11:42:38.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.608+0000 7fb577b85740 1 -- start start 2024-04-25T11:42:38.597 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.608+0000 7fb577b85740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55df1b0 0x5560c597eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.597 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.608+0000 7fb577b85740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c55b62f0 0x5560c597f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.597 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.608+0000 7fb577b85740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560c55be4d0 0x5560c597f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.597 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.608+0000 7fb577b85740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5560c58e5460 con 0x5560c55df1b0 2024-04-25T11:42:38.597 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.608+0000 7fb577b85740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5560c589f590 con 0x5560c55be4d0 2024-04-25T11:42:38.597 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.608+0000 7fb577b85740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5560c58e1420 con 0x5560c55b62f0 2024-04-25T11:42:38.597 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.608+0000 7fb569d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55df1b0 0x5560c597eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.597 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.608+0000 7fb569d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55df1b0 0x5560c597eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60384/0 (socket says 172.21.15.90:60384) 2024-04-25T11:42:38.597 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.608+0000 7fb569d9b640 1 -- 172.21.15.90:0/2036611109 learned_addr learned my addr 172.21.15.90:0/2036611109 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.597 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.608+0000 7fb55b7fe640 1 --2- 172.21.15.90:0/1544744786 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb5540afc50 0x7fb5540b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.597 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.609+0000 7fb569d9b640 1 --2- 172.21.15.90:0/2036611109 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560c55be4d0 0x5560c597f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.597 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.609+0000 7fb56a59c640 1 --2- 172.21.15.90:0/2036611109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c55b62f0 0x5560c597f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.609+0000 7fb56ad9d640 1 --2- 172.21.15.90:0/1544744786 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb5540afc50 0x7fb5540b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.609+0000 7fb569d9b640 1 -- 172.21.15.90:0/2036611109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c55b62f0 msgr2=0x5560c597f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.609+0000 7fb569d9b640 1 --2- 172.21.15.90:0/2036611109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c55b62f0 0x5560c597f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.609+0000 7fb569d9b640 1 -- 172.21.15.90:0/2036611109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55df1b0 msgr2=0x5560c597eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.609+0000 7fb569d9b640 1 --2- 172.21.15.90:0/2036611109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55df1b0 0x5560c597eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.609+0000 7fb569d9b640 1 -- 172.21.15.90:0/2036611109 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560c58ac0d0 con 0x5560c55be4d0 2024-04-25T11:42:38.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.609+0000 7fb55b7fe640 1 -- 172.21.15.90:0/1544744786 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7fb564100500 con 0x7fb56c108f30 2024-04-25T11:42:38.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.609+0000 7fb569d9b640 1 --2- 172.21.15.90:0/2036611109 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560c55be4d0 0x5560c597f570 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb560013720 tx=0x7fb560004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d757a67819ab510 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.609+0000 7fb56ad9d640 1 --2- 172.21.15.90:0/1544744786 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb5540afc50 0x7fb5540b2160 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb5640c5990 tx=0x7fb56404a000 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.609+0000 7fb558ff9640 1 -- 172.21.15.90:0/2036611109 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb560013050 con 0x5560c55be4d0 2024-04-25T11:42:38.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.609+0000 7fb558ff9640 1 -- 172.21.15.90:0/2036611109 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5600131f0 con 0x5560c55be4d0 2024-04-25T11:42:38.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.609+0000 7fb558ff9640 1 -- 172.21.15.90:0/2036611109 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56000fe50 con 0x5560c55be4d0 2024-04-25T11:42:38.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.610+0000 7fb577b85740 1 -- 172.21.15.90:0/2036611109 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560c55be4d0 msgr2=0x5560c597f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.610+0000 7fb577b85740 1 --2- 172.21.15.90:0/2036611109 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560c55be4d0 0x5560c597f570 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb560013720 tx=0x7fb560004730 comp rx=0 tx=0).stop 2024-04-25T11:42:38.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.610+0000 7fb577b85740 1 -- 172.21.15.90:0/2036611109 shutdown_connections 2024-04-25T11:42:38.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.610+0000 7fb577b85740 1 --2- 172.21.15.90:0/2036611109 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560c55be4d0 0x5560c597f570 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.610+0000 7fb577b85740 1 --2- 172.21.15.90:0/2036611109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c55b62f0 0x5560c597f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.610+0000 7fb577b85740 1 --2- 172.21.15.90:0/2036611109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55df1b0 0x5560c597eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.610+0000 7fb577b85740 1 -- 172.21.15.90:0/2036611109 >> 172.21.15.90:0/2036611109 conn(0x5560c55d3300 msgr2=0x5560c597b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.611+0000 7fb577b85740 1 -- 172.21.15.90:0/2036611109 shutdown_connections 2024-04-25T11:42:38.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.611+0000 7fb577b85740 1 -- 172.21.15.90:0/2036611109 wait complete. 2024-04-25T11:42:38.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.611+0000 7fb577b85740 1 -- start start 2024-04-25T11:42:38.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.611+0000 7fb577b85740 1 Processor -- start 2024-04-25T11:42:38.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.611+0000 7fb577b85740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55b62f0 0x5560c598ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.611+0000 7fb577b85740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560c55be4d0 0x5560c598b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.611+0000 7fb577b85740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c55df1b0 0x5560c5989030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.611+0000 7fb577b85740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5560c598bbc0 con 0x5560c55b62f0 2024-04-25T11:42:38.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.611+0000 7fb577b85740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5560c586cb60 con 0x5560c55be4d0 2024-04-25T11:42:38.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.611+0000 7fb577b85740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5560c598bd40 con 0x5560c55df1b0 2024-04-25T11:42:38.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.611+0000 7fb577b85740 10 client.0 _ll_register_callbacks cb 0x5560c58912b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:38.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.612+0000 7fb56a59c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560c55be4d0 0x5560c598b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.612+0000 7fb569d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55b62f0 0x5560c598ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.612+0000 7fb56a59c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560c55be4d0 0x5560c598b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:38838/0 (socket says 172.21.15.90:38838) 2024-04-25T11:42:38.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.612+0000 7fb56a59c640 1 -- 172.21.15.90:0/3534491194 learned_addr learned my addr 172.21.15.90:0/3534491194 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.612+0000 7fb569d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55b62f0 0x5560c598ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60386/0 (socket says 172.21.15.90:60386) 2024-04-25T11:42:38.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.612+0000 7fb569d9b640 1 --2- 172.21.15.90:0/3534491194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c55df1b0 0x5560c5989030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.612+0000 7fb56a59c640 1 -- 172.21.15.90:0/3534491194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c55df1b0 msgr2=0x5560c5989030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.612+0000 7fb56a59c640 1 --2- 172.21.15.90:0/3534491194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c55df1b0 0x5560c5989030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.612+0000 7fb56a59c640 1 -- 172.21.15.90:0/3534491194 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55b62f0 msgr2=0x5560c598ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.612+0000 7fb56a59c640 1 --2- 172.21.15.90:0/3534491194 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55b62f0 0x5560c598ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.612+0000 7fb56a59c640 1 -- 172.21.15.90:0/3534491194 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560c55f7210 con 0x5560c55be4d0 2024-04-25T11:42:38.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.612+0000 7fb569d9b640 1 --2- 172.21.15.90:0/3534491194 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55b62f0 0x5560c598ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:42:38.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.612+0000 7fb56a59c640 1 --2- 172.21.15.90:0/3534491194 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560c55be4d0 0x5560c598b0d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb55c00be40 tx=0x7fb55c012820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19661c1d97e794bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.612+0000 7fb551ffb640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:42:38.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.612+0000 7fb551ffb640 1 -- 172.21.15.90:0/3534491194 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c01f060 con 0x5560c55be4d0 2024-04-25T11:42:38.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.612+0000 7fb577b85740 1 -- 172.21.15.90:0/3534491194 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5600131f0 con 0x5560c55be4d0 2024-04-25T11:42:38.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.612+0000 7fb551ffb640 1 -- 172.21.15.90:0/3534491194 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7fb55c010960 con 0x5560c55be4d0 2024-04-25T11:42:38.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.613+0000 7fb551ffb640 1 -- 172.21.15.90:0/3534491194 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb55c026050 con 0x5560c55be4d0 2024-04-25T11:42:38.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.613+0000 7fb577b85740 10 client.26813 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:38.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.613+0000 7fb577b85740 1 -- 172.21.15.90:0/3534491194 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5560c56c3620 con 0x5560c55be4d0 2024-04-25T11:42:38.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.613+0000 7fb551ffb640 1 -- 172.21.15.90:0/3534491194 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5600131f0 con 0x5560c55be4d0 2024-04-25T11:42:38.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.613+0000 7fb551ffb640 1 -- 172.21.15.90:0/3534491194 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fb55c019030 con 0x5560c55be4d0 2024-04-25T11:42:38.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.613+0000 7fb577b85740 10 client.26813 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:38.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.613+0000 7fb577b85740 10 client.26813 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:42:38.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.613+0000 7fb577b85740 20 client.26813 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:38.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.613+0000 7fb577b85740 1 --2- 172.21.15.90:0/3534491194 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5560c559dd80 0x5560c599b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.613+0000 7fb577b85740 4 client.26813 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:42:38.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.613+0000 7fb577b85740 1 -- 172.21.15.90:0/3534491194 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560c58f3070 con 0x5560c559dd80 2024-04-25T11:42:38.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.613+0000 7fb569d9b640 1 --2- 172.21.15.90:0/3534491194 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5560c559dd80 0x5560c599b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.614+0000 7fb551ffb640 10 client.26813 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:38.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.614+0000 7fb569d9b640 1 --2- 172.21.15.90:0/3534491194 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5560c559dd80 0x5560c599b7d0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf9c469bfb51a28a server_cookie=ff46b4c278f39fb5 in_seq=0 out_seq=0 2024-04-25T11:42:38.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.614+0000 7fb551ffb640 1 -- 172.21.15.90:0/3534491194 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5560c58f3070 con 0x5560c559dd80 2024-04-25T11:42:38.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.614+0000 7fb551ffb640 10 client.26813 handle_command_reply: tid=0 2024-04-25T11:42:38.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.614+0000 7fb577b85740 1 client.26813 shutdown 2024-04-25T11:42:38.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.615+0000 7fb577b85740 1 -- 172.21.15.90:0/3534491194 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560c55be4d0 msgr2=0x5560c598b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.615+0000 7fb577b85740 1 --2- 172.21.15.90:0/3534491194 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560c55be4d0 0x5560c598b0d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb55c00be40 tx=0x7fb55c012820 comp rx=0 tx=0).stop 2024-04-25T11:42:38.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.615+0000 7fb577b85740 1 -- 172.21.15.90:0/3534491194 shutdown_connections 2024-04-25T11:42:38.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.615+0000 7fb577b85740 1 --2- 172.21.15.90:0/3534491194 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5560c559dd80 0x5560c599b7d0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.615+0000 7fb577b85740 1 --2- 172.21.15.90:0/3534491194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c55df1b0 0x5560c5989030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.615+0000 7fb577b85740 1 --2- 172.21.15.90:0/3534491194 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560c55be4d0 0x5560c598b0d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.615+0000 7fb551ffb640 0 client.26813 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:38.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.615+0000 7fb577b85740 1 --2- 172.21.15.90:0/3534491194 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55b62f0 0x5560c598ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.615+0000 7fb577b85740 1 -- 172.21.15.90:0/3534491194 >> 172.21.15.90:0/3534491194 conn(0x5560c55d3300 msgr2=0x5560c5980810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.615+0000 7fb577b85740 1 -- 172.21.15.90:0/3534491194 shutdown_connections 2024-04-25T11:42:38.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.615+0000 7fb577b85740 1 -- 172.21.15.90:0/3534491194 wait complete. 2024-04-25T11:42:38.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.615+0000 7fb577b85740 20 client.26813 trim_cache size 0 max 16384 2024-04-25T11:42:38.634 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.646+0000 7fb577b85740 1 Processor -- start 2024-04-25T11:42:38.634 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.646+0000 7fb577b85740 1 -- start start 2024-04-25T11:42:38.634 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.646+0000 7fb577b85740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55b62f0 0x5560c59827a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.634 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.646+0000 7fb577b85740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560c55be4d0 0x5560c5982ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.634 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.646+0000 7fb577b85740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c55df1b0 0x5560c5980730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.634 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.646+0000 7fb577b85740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb56c1b1d80 con 0x5560c55b62f0 2024-04-25T11:42:38.634 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.646+0000 7fb577b85740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb56c1b38c0 con 0x5560c55be4d0 2024-04-25T11:42:38.634 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.646+0000 7fb577b85740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb56c1b2450 con 0x5560c55df1b0 2024-04-25T11:42:38.634 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.646+0000 7fb569d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55b62f0 0x5560c59827a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.634 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.646+0000 7fb569d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55b62f0 0x5560c59827a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60402/0 (socket says 172.21.15.90:60402) 2024-04-25T11:42:38.634 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.646+0000 7fb569d9b640 1 -- 172.21.15.90:0/2370387866 learned_addr learned my addr 172.21.15.90:0/2370387866 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.634 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.646+0000 7fb56a59c640 1 --2- 172.21.15.90:0/2370387866 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560c55be4d0 0x5560c5982ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.634 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.646+0000 7fb569d9b640 1 -- 172.21.15.90:0/2370387866 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c55df1b0 msgr2=0x5560c5980730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.635 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.646+0000 7fb569d9b640 1 --2- 172.21.15.90:0/2370387866 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c55df1b0 0x5560c5980730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.635 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.646+0000 7fb569d9b640 1 -- 172.21.15.90:0/2370387866 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560c55be4d0 msgr2=0x5560c5982ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.635 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.646+0000 7fb569d9b640 1 --2- 172.21.15.90:0/2370387866 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560c55be4d0 0x5560c5982ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.635 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.646+0000 7fb569d9b640 1 -- 172.21.15.90:0/2370387866 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560c59832d0 con 0x5560c55b62f0 2024-04-25T11:42:38.635 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.647+0000 7fb569d9b640 1 --2- 172.21.15.90:0/2370387866 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55b62f0 0x5560c59827a0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fb5600136f0 tx=0x7fb560009d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7779563ba06244a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.635 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.647+0000 7fb553fff640 1 -- 172.21.15.90:0/2370387866 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb560017070 con 0x5560c55b62f0 2024-04-25T11:42:38.635 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.647+0000 7fb553fff640 1 -- 172.21.15.90:0/2370387866 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb560005560 con 0x5560c55b62f0 2024-04-25T11:42:38.635 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.647+0000 7fb553fff640 1 -- 172.21.15.90:0/2370387866 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56002b6f0 con 0x5560c55b62f0 2024-04-25T11:42:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.648+0000 7fb577b85740 1 -- 172.21.15.90:0/2370387866 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55b62f0 msgr2=0x5560c59827a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.648+0000 7fb577b85740 1 --2- 172.21.15.90:0/2370387866 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55b62f0 0x5560c59827a0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fb5600136f0 tx=0x7fb560009d60 comp rx=0 tx=0).stop 2024-04-25T11:42:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.648+0000 7fb577b85740 1 -- 172.21.15.90:0/2370387866 shutdown_connections 2024-04-25T11:42:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.648+0000 7fb577b85740 1 --2- 172.21.15.90:0/2370387866 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c55df1b0 0x5560c5980730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.648+0000 7fb577b85740 1 --2- 172.21.15.90:0/2370387866 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560c55be4d0 0x5560c5982ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.648+0000 7fb577b85740 1 --2- 172.21.15.90:0/2370387866 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55b62f0 0x5560c59827a0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.648+0000 7fb577b85740 1 -- 172.21.15.90:0/2370387866 >> 172.21.15.90:0/2370387866 conn(0x5560c55d3300 msgr2=0x5560c5987900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.648+0000 7fb577b85740 1 -- 172.21.15.90:0/2370387866 shutdown_connections 2024-04-25T11:42:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.648+0000 7fb577b85740 1 -- 172.21.15.90:0/2370387866 wait complete. 2024-04-25T11:42:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.648+0000 7fb577b85740 1 -- start start 2024-04-25T11:42:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.648+0000 7fb577b85740 1 Processor -- start 2024-04-25T11:42:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.648+0000 7fb577b85740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55b62f0 0x5560c598ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.648+0000 7fb577b85740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c55be4d0 0x5560c598f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.648+0000 7fb577b85740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560c55df1b0 0x5560c598f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.648+0000 7fb577b85740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb56c1b31f0 con 0x5560c55b62f0 2024-04-25T11:42:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.648+0000 7fb577b85740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb56c1b2b20 con 0x5560c55df1b0 2024-04-25T11:42:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.648+0000 7fb577b85740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb56c1afd80 con 0x5560c55be4d0 2024-04-25T11:42:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.649+0000 7fb577b85740 10 client.0 _ll_register_callbacks cb 0x7fb55c00b730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.649+0000 7fb569d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55b62f0 0x5560c598ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.649+0000 7fb569d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55b62f0 0x5560c598ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60410/0 (socket says 172.21.15.90:60410) 2024-04-25T11:42:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.649+0000 7fb569d9b640 1 -- 172.21.15.90:0/3516832481 learned_addr learned my addr 172.21.15.90:0/3516832481 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.649+0000 7fb56a59c640 1 --2- 172.21.15.90:0/3516832481 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c55be4d0 0x5560c598f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.649+0000 7fb569d9b640 1 -- 172.21.15.90:0/3516832481 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c55be4d0 msgr2=0x5560c598f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.649+0000 7fb569d9b640 1 --2- 172.21.15.90:0/3516832481 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c55be4d0 0x5560c598f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.649+0000 7fb569d9b640 1 -- 172.21.15.90:0/3516832481 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560c55df1b0 msgr2=0x5560c598f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.649+0000 7fb569d9b640 1 --2- 172.21.15.90:0/3516832481 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560c55df1b0 0x5560c598f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.649+0000 7fb569d9b640 1 -- 172.21.15.90:0/3516832481 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560c5a02b90 con 0x5560c55b62f0 2024-04-25T11:42:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.649+0000 7fb569d9b640 1 --2- 172.21.15.90:0/3516832481 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55b62f0 0x5560c598ebe0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fb56002b350 tx=0x7fb560009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3343451ebdd18fd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.649+0000 7fb551ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.649+0000 7fb551ffb640 1 -- 172.21.15.90:0/3516832481 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb560017040 con 0x5560c55b62f0 2024-04-25T11:42:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.649+0000 7fb551ffb640 1 -- 172.21.15.90:0/3516832481 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7fb5600052c0 con 0x5560c55b62f0 2024-04-25T11:42:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.649+0000 7fb577b85740 1 -- 172.21.15.90:0/3516832481 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb560005560 con 0x5560c55b62f0 2024-04-25T11:42:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.649+0000 7fb551ffb640 1 -- 172.21.15.90:0/3516832481 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb56002b3f0 con 0x5560c55b62f0 2024-04-25T11:42:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.649+0000 7fb577b85740 10 client.17928 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.649+0000 7fb577b85740 1 -- 172.21.15.90:0/3516832481 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5560c5a03db0 con 0x5560c55b62f0 2024-04-25T11:42:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.650+0000 7fb551ffb640 1 -- 172.21.15.90:0/3516832481 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb560005560 con 0x5560c55b62f0 2024-04-25T11:42:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.650+0000 7fb551ffb640 1 -- 172.21.15.90:0/3516832481 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fb560003970 con 0x5560c55b62f0 2024-04-25T11:42:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.650+0000 7fb577b85740 10 client.17928 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.650+0000 7fb577b85740 10 client.17928 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:42:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.650+0000 7fb577b85740 20 client.17928 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.650+0000 7fb577b85740 1 --2- 172.21.15.90:0/3516832481 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5560c55731c0 0x5560c5a062a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.650+0000 7fb577b85740 4 client.17928 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:42:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.650+0000 7fb577b85740 1 -- 172.21.15.90:0/3516832481 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5560c58ecee0 con 0x5560c55731c0 2024-04-25T11:42:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.650+0000 7fb56a59c640 1 --2- 172.21.15.90:0/3516832481 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5560c55731c0 0x5560c5a062a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.651+0000 7fb551ffb640 10 client.17928 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.651+0000 7fb56a59c640 1 --2- 172.21.15.90:0/3516832481 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5560c55731c0 0x5560c5a062a0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf1504c69d8859c8 server_cookie=6040f4ea901c7643 in_seq=0 out_seq=0 2024-04-25T11:42:38.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.662+0000 7fc33e870640 1 -- 172.21.15.90:0/2381702353 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc338109ba0 msgr2=0x7fc338101b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.662+0000 7fc33e870640 1 --2- 172.21.15.90:0/2381702353 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc338109ba0 0x7fc338101b10 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fc328009900 tx=0x7fc32802a410 comp rx=0 tx=0).stop 2024-04-25T11:42:38.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.662+0000 7fc33e870640 1 -- 172.21.15.90:0/2381702353 shutdown_connections 2024-04-25T11:42:38.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.662+0000 7fc33e870640 1 --2- 172.21.15.90:0/2381702353 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc33810e550 0x7fc338110970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.662+0000 7fc33e870640 1 --2- 172.21.15.90:0/2381702353 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc338102050 0x7fc338102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.662+0000 7fc33e870640 1 --2- 172.21.15.90:0/2381702353 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc338109ba0 0x7fc338101b10 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.662+0000 7fc33e870640 1 -- 172.21.15.90:0/2381702353 >> 172.21.15.90:0/2381702353 conn(0x7fc338077880 msgr2=0x7fc338077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.662+0000 7fc33e870640 1 -- 172.21.15.90:0/2381702353 shutdown_connections 2024-04-25T11:42:38.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.662+0000 7fc33e870640 1 -- 172.21.15.90:0/2381702353 wait complete. 2024-04-25T11:42:38.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.663+0000 7fc33e870640 1 Processor -- start 2024-04-25T11:42:38.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.663+0000 7fc33e870640 1 -- start start 2024-04-25T11:42:38.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.663+0000 7fc33e870640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc338102050 0x7fc33819a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fc33e870640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc338109ba0 0x7fc33819acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fc33e870640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc33810e550 0x7fc33819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fc33e870640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc338113d10 con 0x7fc338102050 2024-04-25T11:42:38.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fc33e870640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc338113b90 con 0x7fc33810e550 2024-04-25T11:42:38.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fc33e870640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc338113e90 con 0x7fc338109ba0 2024-04-25T11:42:38.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fc337fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc338102050 0x7fc33819a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fc337fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc338102050 0x7fc33819a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60438/0 (socket says 172.21.15.90:60438) 2024-04-25T11:42:38.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fc337fff640 1 -- 172.21.15.90:0/1072829617 learned_addr learned my addr 172.21.15.90:0/1072829617 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fb551ffb640 1 -- 172.21.15.90:0/3516832481 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5560c58ecee0 con 0x5560c55731c0 2024-04-25T11:42:38.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fb551ffb640 10 client.17928 handle_command_reply: tid=0 2024-04-25T11:42:38.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fb577b85740 1 client.17928 shutdown 2024-04-25T11:42:38.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fc3377fe640 1 --2- 172.21.15.90:0/1072829617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc338109ba0 0x7fc33819acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fc33ce0d640 1 --2- 172.21.15.90:0/1072829617 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc33810e550 0x7fc33819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fb577b85740 1 -- 172.21.15.90:0/3516832481 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55b62f0 msgr2=0x5560c598ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fb577b85740 1 --2- 172.21.15.90:0/3516832481 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55b62f0 0x5560c598ebe0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fb56002b350 tx=0x7fb560009f60 comp rx=0 tx=0).stop 2024-04-25T11:42:38.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fb577b85740 1 -- 172.21.15.90:0/3516832481 shutdown_connections 2024-04-25T11:42:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fb577b85740 1 --2- 172.21.15.90:0/3516832481 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5560c55731c0 0x5560c5a062a0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fb577b85740 1 --2- 172.21.15.90:0/3516832481 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5560c55df1b0 0x5560c598f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fb577b85740 1 --2- 172.21.15.90:0/3516832481 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c55be4d0 0x5560c598f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fb577b85740 1 --2- 172.21.15.90:0/3516832481 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560c55b62f0 0x5560c598ebe0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fc337fff640 1 -- 172.21.15.90:0/1072829617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc338109ba0 msgr2=0x7fc33819acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fc337fff640 1 --2- 172.21.15.90:0/1072829617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc338109ba0 0x7fc33819acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fb577b85740 1 -- 172.21.15.90:0/3516832481 >> 172.21.15.90:0/3516832481 conn(0x5560c55d3300 msgr2=0x5560c59806a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fb551ffb640 0 client.17928 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:38.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fc337fff640 1 -- 172.21.15.90:0/1072829617 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc33810e550 msgr2=0x7fc33819ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fc337fff640 1 --2- 172.21.15.90:0/1072829617 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc33810e550 0x7fc33819ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fc337fff640 1 -- 172.21.15.90:0/1072829617 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc33819f5d0 con 0x7fc338102050 2024-04-25T11:42:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fb577b85740 1 -- 172.21.15.90:0/3516832481 shutdown_connections 2024-04-25T11:42:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fb577b85740 1 -- 172.21.15.90:0/3516832481 wait complete. 2024-04-25T11:42:38.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fc337fff640 1 --2- 172.21.15.90:0/1072829617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc338102050 0x7fc33819a770 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fc32802a930 tx=0x7fc328004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fb577b85740 20 client.17928 trim_cache size 0 max 16384 2024-04-25T11:42:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:42:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:42:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:42:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:42:38.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fc3357fa640 1 -- 172.21.15.90:0/1072829617 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32803f030 con 0x7fc338102050 2024-04-25T11:42:38.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fc3357fa640 1 -- 172.21.15.90:0/1072829617 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc328003aa0 con 0x7fc338102050 2024-04-25T11:42:38.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fc3357fa640 1 -- 172.21.15.90:0/1072829617 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc328035070 con 0x7fc338102050 2024-04-25T11:42:38.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fc33e870640 1 -- 172.21.15.90:0/1072829617 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc33819f860 con 0x7fc338102050 2024-04-25T11:42:38.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.664+0000 7fc33e870640 1 -- 172.21.15.90:0/1072829617 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc33819fda0 con 0x7fc338102050 2024-04-25T11:42:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.665+0000 7fb570cd8640 1 -- 172.21.15.90:0/1544744786 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb5540afc50 msgr2=0x7fb5540b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.665+0000 7fb570cd8640 1 --2- 172.21.15.90:0/1544744786 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb5540afc50 0x7fb5540b2160 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb5640c5990 tx=0x7fb56404a000 comp rx=0 tx=0).stop 2024-04-25T11:42:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.665+0000 7fb570cd8640 1 -- 172.21.15.90:0/1544744786 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb56c108f30 msgr2=0x7fb56c100420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.665+0000 7fb570cd8640 1 --2- 172.21.15.90:0/1544744786 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb56c108f30 0x7fb56c100420 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fb5640098d0 tx=0x7fb5640317b0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.665+0000 7fb570cd8640 1 -- 172.21.15.90:0/1544744786 shutdown_connections 2024-04-25T11:42:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.665+0000 7fb570cd8640 1 --2- 172.21.15.90:0/1544744786 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb5540afc50 0x7fb5540b2160 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.665+0000 7fb570cd8640 1 --2- 172.21.15.90:0/1544744786 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb56c108f30 0x7fb56c100420 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.665+0000 7fb570cd8640 1 --2- 172.21.15.90:0/1544744786 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb56c108a90 0x7fb56c0ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.665+0000 7fb570cd8640 1 --2- 172.21.15.90:0/1544744786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb56c1031e0 0x7fb56c0ff9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.665+0000 7fb570cd8640 1 -- 172.21.15.90:0/1544744786 >> 172.21.15.90:0/1544744786 conn(0x7fb56c078080 msgr2=0x7fb56c06b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.665+0000 7fb570cd8640 1 -- 172.21.15.90:0/1544744786 shutdown_connections 2024-04-25T11:42:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:38.665+0000 7fb570cd8640 1 -- 172.21.15.90:0/1544744786 wait complete. 2024-04-25T11:42:38.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.667+0000 7fc3357fa640 1 -- 172.21.15.90:0/1072829617 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fc328042030 con 0x7fc338102050 2024-04-25T11:42:38.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.667+0000 7fc3357fa640 1 --2- 172.21.15.90:0/1072829617 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc3200afce0 0x7fc3200b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.668+0000 7fc3457f4740 1 Processor -- start 2024-04-25T11:42:38.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.668+0000 7fc3377fe640 1 --2- 172.21.15.90:0/1072829617 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc3200afce0 0x7fc3200b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.668+0000 7fc3457f4740 1 -- start start 2024-04-25T11:42:38.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.668+0000 7fc3357fa640 1 -- 172.21.15.90:0/1072829617 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7fc3280ff6c0 con 0x7fc338102050 2024-04-25T11:42:38.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.668+0000 7fc3457f4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ffad390 0x560a1034e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.668+0000 7fc3457f4740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a0ff844d0 0x560a1034ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.668+0000 7fc3457f4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0ff8c6b0 0x560a1035b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.668+0000 7fc3457f4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560a102b3d80 con 0x560a0ffad390 2024-04-25T11:42:38.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.668+0000 7fc3457f4740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x560a1026da20 con 0x560a0ff844d0 2024-04-25T11:42:38.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.668+0000 7fc3457f4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560a102afd40 con 0x560a0ff8c6b0 2024-04-25T11:42:38.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.668+0000 7fc337fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ffad390 0x560a1034e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.668+0000 7fc337fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ffad390 0x560a1034e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60444/0 (socket says 172.21.15.90:60444) 2024-04-25T11:42:38.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.668+0000 7fc337fff640 1 -- 172.21.15.90:0/3007780704 learned_addr learned my addr 172.21.15.90:0/3007780704 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.668+0000 7fc3377fe640 1 --2- 172.21.15.90:0/1072829617 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc3200afce0 0x7fc3200b21f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc33810fa50 tx=0x7fc32c00b410 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.668+0000 7fc33ce0d640 1 --2- 172.21.15.90:0/3007780704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0ff8c6b0 0x560a1035b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.668+0000 7fc337fff640 1 -- 172.21.15.90:0/3007780704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0ff8c6b0 msgr2=0x560a1035b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.668+0000 7fc337fff640 1 --2- 172.21.15.90:0/3007780704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0ff8c6b0 0x560a1035b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.668+0000 7fc337fff640 1 -- 172.21.15.90:0/3007780704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a0ff844d0 msgr2=0x560a1034ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.668+0000 7fc337fff640 1 --2- 172.21.15.90:0/3007780704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a0ff844d0 0x560a1034ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.668+0000 7fc337fff640 1 -- 172.21.15.90:0/3007780704 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a10278c30 con 0x560a0ffad390 2024-04-25T11:42:38.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.669+0000 7fc337fff640 1 --2- 172.21.15.90:0/3007780704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ffad390 0x560a1034e810 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fc3280c7580 tx=0x7fc32802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=651a518a3156dfd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.669+0000 7fc31e7fc640 1 -- 172.21.15.90:0/3007780704 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc328049070 con 0x560a0ffad390 2024-04-25T11:42:38.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.669+0000 7fc31e7fc640 1 -- 172.21.15.90:0/3007780704 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3280cbb80 con 0x560a0ffad390 2024-04-25T11:42:38.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.669+0000 7fc31e7fc640 1 -- 172.21.15.90:0/3007780704 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3280fd000 con 0x560a0ffad390 2024-04-25T11:42:38.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.670+0000 7fc3457f4740 1 -- 172.21.15.90:0/3007780704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ffad390 msgr2=0x560a1034e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.670+0000 7fc3457f4740 1 --2- 172.21.15.90:0/3007780704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ffad390 0x560a1034e810 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fc3280c7580 tx=0x7fc32802a410 comp rx=0 tx=0).stop 2024-04-25T11:42:38.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.670+0000 7fc3457f4740 1 -- 172.21.15.90:0/3007780704 shutdown_connections 2024-04-25T11:42:38.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.670+0000 7fc3457f4740 1 --2- 172.21.15.90:0/3007780704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0ff8c6b0 0x560a1035b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.670+0000 7fc3457f4740 1 --2- 172.21.15.90:0/3007780704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a0ff844d0 0x560a1034ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.670+0000 7fc3457f4740 1 --2- 172.21.15.90:0/3007780704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ffad390 0x560a1034e810 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.670+0000 7fc3457f4740 1 -- 172.21.15.90:0/3007780704 >> 172.21.15.90:0/3007780704 conn(0x560a0ffa14e0 msgr2=0x560a1034afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.670+0000 7fc3457f4740 1 -- 172.21.15.90:0/3007780704 shutdown_connections 2024-04-25T11:42:38.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.670+0000 7fc3457f4740 1 -- 172.21.15.90:0/3007780704 wait complete. 2024-04-25T11:42:38.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.670+0000 7fc3457f4740 1 -- start start 2024-04-25T11:42:38.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.671+0000 7fc3457f4740 1 Processor -- start 2024-04-25T11:42:38.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.671+0000 7fc3457f4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0ff844d0 0x560a10359340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.671+0000 7fc3457f4740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a0ff8c6b0 0x560a10359880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.671+0000 7fc3457f4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ffad390 0x560a10365010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.671+0000 7fc3457f4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560a1035bbe0 con 0x560a0ffad390 2024-04-25T11:42:38.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.671+0000 7fc3457f4740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x560a1023adc0 con 0x560a0ff8c6b0 2024-04-25T11:42:38.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.671+0000 7fc3457f4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560a1035bd60 con 0x560a0ff844d0 2024-04-25T11:42:38.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.671+0000 7fc3457f4740 10 client.0 _ll_register_callbacks cb 0x560a10256600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:38.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.671+0000 7fc33ce0d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ffad390 0x560a10365010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.671+0000 7fc33ce0d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ffad390 0x560a10365010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60448/0 (socket says 172.21.15.90:60448) 2024-04-25T11:42:38.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.671+0000 7fc33ce0d640 1 -- 172.21.15.90:0/3540527537 learned_addr learned my addr 172.21.15.90:0/3540527537 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.671+0000 7fc3377fe640 1 --2- 172.21.15.90:0/3540527537 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a0ff8c6b0 0x560a10359880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.671+0000 7fc337fff640 1 --2- 172.21.15.90:0/3540527537 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0ff844d0 0x560a10359340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.671+0000 7fc33ce0d640 1 -- 172.21.15.90:0/3540527537 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0ff844d0 msgr2=0x560a10359340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.671+0000 7fc33ce0d640 1 --2- 172.21.15.90:0/3540527537 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0ff844d0 0x560a10359340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.671+0000 7fc33ce0d640 1 -- 172.21.15.90:0/3540527537 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a0ff8c6b0 msgr2=0x560a10359880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.671+0000 7fc33ce0d640 1 --2- 172.21.15.90:0/3540527537 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a0ff8c6b0 0x560a10359880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.671+0000 7fc33ce0d640 1 -- 172.21.15.90:0/3540527537 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a0ffc53f0 con 0x560a0ffad390 2024-04-25T11:42:38.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.672+0000 7fc33ce0d640 1 --2- 172.21.15.90:0/3540527537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ffad390 0x560a10365010 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fc330011690 tx=0x7fc330011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5644e24e2fe9f03 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.672+0000 7fc3177fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:38.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.672+0000 7fc3177fe640 1 -- 172.21.15.90:0/3540527537 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc33000fb40 con 0x560a0ffad390 2024-04-25T11:42:38.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.672+0000 7fc3177fe640 1 -- 172.21.15.90:0/3540527537 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7fc33000fce0 con 0x560a0ffad390 2024-04-25T11:42:38.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.672+0000 7fc3457f4740 1 -- 172.21.15.90:0/3540527537 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3280cbb80 con 0x560a0ffad390 2024-04-25T11:42:38.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.672+0000 7fc3177fe640 1 -- 172.21.15.90:0/3540527537 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc33001d480 con 0x560a0ffad390 2024-04-25T11:42:38.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.672+0000 7fc3457f4740 10 client.17940 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:38.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.672+0000 7fc3457f4740 1 -- 172.21.15.90:0/3540527537 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560a1008f7d0 con 0x560a0ffad390 2024-04-25T11:42:38.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.672+0000 7fc3177fe640 1 -- 172.21.15.90:0/3540527537 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3280cbb80 con 0x560a0ffad390 2024-04-25T11:42:38.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.672+0000 7fc3177fe640 1 -- 172.21.15.90:0/3540527537 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fc330023030 con 0x560a0ffad390 2024-04-25T11:42:38.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.672+0000 7fc3457f4740 10 client.17940 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:38.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.672+0000 7fc3457f4740 10 client.17940 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:42:38.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.672+0000 7fc3457f4740 20 client.17940 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:38.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.672+0000 7fc3457f4740 1 --2- 172.21.15.90:0/3540527537 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x560a0ff6bf60 0x560a1036b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.672+0000 7fc3457f4740 4 client.17940 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:42:38.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.672+0000 7fc3457f4740 1 -- 172.21.15.90:0/3540527537 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a102c1990 con 0x560a0ff6bf60 2024-04-25T11:42:38.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.673+0000 7fc337fff640 1 --2- 172.21.15.90:0/3540527537 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x560a0ff6bf60 0x560a1036b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.673+0000 7fc3177fe640 10 client.17940 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:38.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.673+0000 7fc337fff640 1 --2- 172.21.15.90:0/3540527537 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x560a0ff6bf60 0x560a1036b590 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9edd764d3fb6c0b server_cookie=f9c4af6858a1000b in_seq=0 out_seq=0 2024-04-25T11:42:38.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.674+0000 7fc3177fe640 1 -- 172.21.15.90:0/3540527537 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560a102c1990 con 0x560a0ff6bf60 2024-04-25T11:42:38.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.674+0000 7fc3177fe640 10 client.17940 handle_command_reply: tid=0 2024-04-25T11:42:38.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.674+0000 7fc3457f4740 1 client.17940 shutdown 2024-04-25T11:42:38.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.674+0000 7fc3457f4740 1 -- 172.21.15.90:0/3540527537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ffad390 msgr2=0x560a10365010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.674+0000 7fc3457f4740 1 --2- 172.21.15.90:0/3540527537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ffad390 0x560a10365010 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fc330011690 tx=0x7fc330011b50 comp rx=0 tx=0).stop 2024-04-25T11:42:38.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.674+0000 7fc3457f4740 1 -- 172.21.15.90:0/3540527537 shutdown_connections 2024-04-25T11:42:38.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.674+0000 7fc3457f4740 1 --2- 172.21.15.90:0/3540527537 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x560a0ff6bf60 0x560a1036b590 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.674+0000 7fc3457f4740 1 --2- 172.21.15.90:0/3540527537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ffad390 0x560a10365010 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.674+0000 7fc3457f4740 1 --2- 172.21.15.90:0/3540527537 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a0ff8c6b0 0x560a10359880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.674+0000 7fc3177fe640 0 client.17940 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:38.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.674+0000 7fc3457f4740 1 --2- 172.21.15.90:0/3540527537 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0ff844d0 0x560a10359340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.674+0000 7fc3457f4740 1 -- 172.21.15.90:0/3540527537 >> 172.21.15.90:0/3540527537 conn(0x560a0ffa14e0 msgr2=0x560a10350530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.674+0000 7fc3457f4740 1 -- 172.21.15.90:0/3540527537 shutdown_connections 2024-04-25T11:42:38.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.674+0000 7fc3457f4740 1 -- 172.21.15.90:0/3540527537 wait complete. 2024-04-25T11:42:38.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.674+0000 7fc3457f4740 20 client.17940 trim_cache size 0 max 16384 2024-04-25T11:42:38.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.688+0000 7fc3457f4740 1 Processor -- start 2024-04-25T11:42:38.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.688+0000 7fc3457f4740 1 -- start start 2024-04-25T11:42:38.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.688+0000 7fc3457f4740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a0ff844d0 0x560a10358110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.688+0000 7fc3457f4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ff8c6b0 0x560a10364c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.688+0000 7fc3457f4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0ffad390 0x560a10353450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.688+0000 7fc3457f4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc3381afc90 con 0x560a0ff8c6b0 2024-04-25T11:42:38.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.688+0000 7fc3457f4740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc3381b17d0 con 0x560a0ff844d0 2024-04-25T11:42:38.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.688+0000 7fc3457f4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc3381b0360 con 0x560a0ffad390 2024-04-25T11:42:38.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.688+0000 7fc3377fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ff8c6b0 0x560a10364c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.688+0000 7fc3377fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ff8c6b0 0x560a10364c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60454/0 (socket says 172.21.15.90:60454) 2024-04-25T11:42:38.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.688+0000 7fc3377fe640 1 -- 172.21.15.90:0/147009131 learned_addr learned my addr 172.21.15.90:0/147009131 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.688+0000 7fc337fff640 1 --2- 172.21.15.90:0/147009131 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a0ff844d0 0x560a10358110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.689+0000 7fc33ce0d640 1 --2- 172.21.15.90:0/147009131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0ffad390 0x560a10353450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.689+0000 7fc3377fe640 1 -- 172.21.15.90:0/147009131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0ffad390 msgr2=0x560a10353450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.689+0000 7fc3377fe640 1 --2- 172.21.15.90:0/147009131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0ffad390 0x560a10353450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.689+0000 7fc3377fe640 1 -- 172.21.15.90:0/147009131 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a0ff844d0 msgr2=0x560a10358110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.689+0000 7fc3377fe640 1 --2- 172.21.15.90:0/147009131 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a0ff844d0 0x560a10358110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.689+0000 7fc3377fe640 1 -- 172.21.15.90:0/147009131 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a10357680 con 0x560a0ff8c6b0 2024-04-25T11:42:38.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.689+0000 7fc3377fe640 1 --2- 172.21.15.90:0/147009131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ff8c6b0 0x560a10364c30 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fc32c005fd0 tx=0x7fc32c013070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9d4cbac1bbff7c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.689+0000 7fc31dffb640 1 -- 172.21.15.90:0/147009131 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32c01f6f0 con 0x560a0ff8c6b0 2024-04-25T11:42:38.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.689+0000 7fc31dffb640 1 -- 172.21.15.90:0/147009131 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc32c023070 con 0x560a0ff8c6b0 2024-04-25T11:42:38.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.689+0000 7fc31dffb640 1 -- 172.21.15.90:0/147009131 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32c028b10 con 0x560a0ff8c6b0 2024-04-25T11:42:38.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.690+0000 7fc3457f4740 1 -- 172.21.15.90:0/147009131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ff8c6b0 msgr2=0x560a10364c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.690+0000 7fc3457f4740 1 --2- 172.21.15.90:0/147009131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ff8c6b0 0x560a10364c30 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fc32c005fd0 tx=0x7fc32c013070 comp rx=0 tx=0).stop 2024-04-25T11:42:38.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.690+0000 7fc3457f4740 1 -- 172.21.15.90:0/147009131 shutdown_connections 2024-04-25T11:42:38.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.690+0000 7fc3457f4740 1 --2- 172.21.15.90:0/147009131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0ffad390 0x560a10353450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.690+0000 7fc3457f4740 1 --2- 172.21.15.90:0/147009131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ff8c6b0 0x560a10364c30 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.690+0000 7fc3457f4740 1 --2- 172.21.15.90:0/147009131 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a0ff844d0 0x560a10358110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.690+0000 7fc3457f4740 1 -- 172.21.15.90:0/147009131 >> 172.21.15.90:0/147009131 conn(0x560a0ffa14e0 msgr2=0x560a10357a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.690+0000 7fc3457f4740 1 -- 172.21.15.90:0/147009131 shutdown_connections 2024-04-25T11:42:38.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.690+0000 7fc3457f4740 1 -- 172.21.15.90:0/147009131 wait complete. 2024-04-25T11:42:38.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.690+0000 7fc3457f4740 1 -- start start 2024-04-25T11:42:38.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.691+0000 7fc3457f4740 1 Processor -- start 2024-04-25T11:42:38.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.691+0000 7fc3457f4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ff844d0 0x560a103610e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.691+0000 7fc3457f4740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a0ff8c6b0 0x560a10361620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.691+0000 7fc3457f4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0ffad390 0x560a1035c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.691+0000 7fc3457f4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc3381b1100 con 0x560a0ff844d0 2024-04-25T11:42:38.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.691+0000 7fc3457f4740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc3381b0a30 con 0x560a0ff8c6b0 2024-04-25T11:42:38.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.691+0000 7fc3457f4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc3381adc90 con 0x560a0ffad390 2024-04-25T11:42:38.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.691+0000 7fc3457f4740 10 client.0 _ll_register_callbacks cb 0x7fc330009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:38.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.691+0000 7fc337fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ff844d0 0x560a103610e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.691+0000 7fc337fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ff844d0 0x560a103610e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60468/0 (socket says 172.21.15.90:60468) 2024-04-25T11:42:38.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.691+0000 7fc337fff640 1 -- 172.21.15.90:0/2183647789 learned_addr learned my addr 172.21.15.90:0/2183647789 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:38.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.691+0000 7fc33ce0d640 1 --2- 172.21.15.90:0/2183647789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0ffad390 0x560a1035c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.691+0000 7fc3377fe640 1 --2- 172.21.15.90:0/2183647789 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a0ff8c6b0 0x560a10361620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.691+0000 7fc337fff640 1 -- 172.21.15.90:0/2183647789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0ffad390 msgr2=0x560a1035c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.691+0000 7fc337fff640 1 --2- 172.21.15.90:0/2183647789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0ffad390 0x560a1035c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.691+0000 7fc337fff640 1 -- 172.21.15.90:0/2183647789 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a0ff8c6b0 msgr2=0x560a10361620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:38.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.691+0000 7fc337fff640 1 --2- 172.21.15.90:0/2183647789 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a0ff8c6b0 0x560a10361620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.691+0000 7fc337fff640 1 -- 172.21.15.90:0/2183647789 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a1035cde0 con 0x560a0ff844d0 2024-04-25T11:42:38.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.691+0000 7fc33ce0d640 1 --2- 172.21.15.90:0/2183647789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0ffad390 0x560a1035c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:42:38.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.691+0000 7fc3377fe640 1 --2- 172.21.15.90:0/2183647789 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a0ff8c6b0 0x560a10361620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:42:38.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.691+0000 7fc337fff640 1 --2- 172.21.15.90:0/2183647789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ff844d0 0x560a103610e0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fc328032af0 tx=0x7fc32803b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c3fdf68d9d32b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:38.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.691+0000 7fc3177fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:38.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.692+0000 7fc3177fe640 1 -- 172.21.15.90:0/2183647789 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3280e0070 con 0x560a0ff844d0 2024-04-25T11:42:38.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.692+0000 7fc3177fe640 1 -- 172.21.15.90:0/2183647789 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7fc3280fd000 con 0x560a0ff844d0 2024-04-25T11:42:38.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.692+0000 7fc3457f4740 1 -- 172.21.15.90:0/2183647789 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc32c023070 con 0x560a0ff844d0 2024-04-25T11:42:38.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.692+0000 7fc3177fe640 1 -- 172.21.15.90:0/2183647789 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc328049040 con 0x560a0ff844d0 2024-04-25T11:42:38.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.692+0000 7fc3457f4740 10 client.17946 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:38.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.692+0000 7fc3457f4740 1 -- 172.21.15.90:0/2183647789 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560a103d3380 con 0x560a0ff844d0 2024-04-25T11:42:38.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.692+0000 7fc3177fe640 1 -- 172.21.15.90:0/2183647789 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc32c023070 con 0x560a0ff844d0 2024-04-25T11:42:38.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.692+0000 7fc3177fe640 1 -- 172.21.15.90:0/2183647789 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fc3280c7ae0 con 0x560a0ff844d0 2024-04-25T11:42:38.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.692+0000 7fc3457f4740 10 client.17946 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:38.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.692+0000 7fc3457f4740 10 client.17946 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:42:38.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.692+0000 7fc3457f4740 20 client.17946 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:38.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.692+0000 7fc3457f4740 1 --2- 172.21.15.90:0/2183647789 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x560a0ff413a0 0x560a103d5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:38.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.692+0000 7fc3457f4740 4 client.17946 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:42:38.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.692+0000 7fc3457f4740 1 -- 172.21.15.90:0/2183647789 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560a102bb800 con 0x560a0ff413a0 2024-04-25T11:42:38.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.692+0000 7fc3377fe640 1 --2- 172.21.15.90:0/2183647789 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x560a0ff413a0 0x560a103d5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:38.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.693+0000 7fc3177fe640 10 client.17946 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:38.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.693+0000 7fc3377fe640 1 --2- 172.21.15.90:0/2183647789 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x560a0ff413a0 0x560a103d5950 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7405a3aa6c9b357 server_cookie=41f2884a2bda8da2 in_seq=0 out_seq=0 2024-04-25T11:42:38.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.693+0000 7fc3177fe640 1 -- 172.21.15.90:0/2183647789 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560a102bb800 con 0x560a0ff413a0 2024-04-25T11:42:38.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.693+0000 7fc3177fe640 10 client.17946 handle_command_reply: tid=0 2024-04-25T11:42:38.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.693+0000 7fc3457f4740 1 client.17946 shutdown 2024-04-25T11:42:38.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.693+0000 7fc3457f4740 1 -- 172.21.15.90:0/2183647789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ff844d0 msgr2=0x560a103610e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.693+0000 7fc3457f4740 1 --2- 172.21.15.90:0/2183647789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ff844d0 0x560a103610e0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fc328032af0 tx=0x7fc32803b6b0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.693+0000 7fc3457f4740 1 -- 172.21.15.90:0/2183647789 shutdown_connections 2024-04-25T11:42:38.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.693+0000 7fc3457f4740 1 --2- 172.21.15.90:0/2183647789 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x560a0ff413a0 0x560a103d5950 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.693+0000 7fc3457f4740 1 --2- 172.21.15.90:0/2183647789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0ffad390 0x560a1035c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.693+0000 7fc3457f4740 1 --2- 172.21.15.90:0/2183647789 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560a0ff8c6b0 0x560a10361620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.693+0000 7fc3177fe640 0 client.17946 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:38.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.693+0000 7fc3457f4740 1 --2- 172.21.15.90:0/2183647789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a0ff844d0 0x560a103610e0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.693+0000 7fc3457f4740 1 -- 172.21.15.90:0/2183647789 >> 172.21.15.90:0/2183647789 conn(0x560a0ffa14e0 msgr2=0x560a10358190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.693+0000 7fc3457f4740 1 -- 172.21.15.90:0/2183647789 shutdown_connections 2024-04-25T11:42:38.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.693+0000 7fc3457f4740 1 -- 172.21.15.90:0/2183647789 wait complete. 2024-04-25T11:42:38.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.693+0000 7fc3457f4740 20 client.17946 trim_cache size 0 max 16384 2024-04-25T11:42:38.688 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:42:38.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.694+0000 7fc33e870640 1 -- 172.21.15.90:0/1072829617 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc3200afce0 msgr2=0x7fc3200b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.694+0000 7fc33e870640 1 --2- 172.21.15.90:0/1072829617 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc3200afce0 0x7fc3200b21f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc33810fa50 tx=0x7fc32c00b410 comp rx=0 tx=0).stop 2024-04-25T11:42:38.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.694+0000 7fc33e870640 1 -- 172.21.15.90:0/1072829617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc338102050 msgr2=0x7fc33819a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.694+0000 7fc33e870640 1 --2- 172.21.15.90:0/1072829617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc338102050 0x7fc33819a770 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fc32802a930 tx=0x7fc328004270 comp rx=0 tx=0).stop 2024-04-25T11:42:38.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.694+0000 7fc33e870640 1 -- 172.21.15.90:0/1072829617 shutdown_connections 2024-04-25T11:42:38.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.694+0000 7fc33e870640 1 --2- 172.21.15.90:0/1072829617 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc3200afce0 0x7fc3200b21f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.694+0000 7fc33e870640 1 --2- 172.21.15.90:0/1072829617 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc33810e550 0x7fc33819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.694+0000 7fc33e870640 1 --2- 172.21.15.90:0/1072829617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc338109ba0 0x7fc33819acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.694+0000 7fc33e870640 1 --2- 172.21.15.90:0/1072829617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc338102050 0x7fc33819a770 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.694+0000 7fc33e870640 1 -- 172.21.15.90:0/1072829617 >> 172.21.15.90:0/1072829617 conn(0x7fc338077880 msgr2=0x7fc33810a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.694+0000 7fc33e870640 1 -- 172.21.15.90:0/1072829617 shutdown_connections 2024-04-25T11:42:38.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:38.694+0000 7fc33e870640 1 -- 172.21.15.90:0/1072829617 wait complete. 2024-04-25T11:42:39.693 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T11:42:39.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:42:39.693 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T11:42:39.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.834+0000 7fbe3f630640 1 -- 172.21.15.90:0/2069321694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe38108c50 msgr2=0x7fbe38102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:39.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.834+0000 7fbe3f630640 1 --2- 172.21.15.90:0/2069321694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe38108c50 0x7fbe38102d20 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fbe28009900 tx=0x7fbe2802a410 comp rx=0 tx=0).stop 2024-04-25T11:42:39.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.834+0000 7fbe3f630640 1 -- 172.21.15.90:0/2069321694 shutdown_connections 2024-04-25T11:42:39.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.834+0000 7fbe3f630640 1 --2- 172.21.15.90:0/2069321694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe381038d0 0x7fbe3810f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.834+0000 7fbe3f630640 1 --2- 172.21.15.90:0/2069321694 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe381090f0 0x7fbe38103260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.834+0000 7fbe3f630640 1 --2- 172.21.15.90:0/2069321694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe38108c50 0x7fbe38102d20 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.834+0000 7fbe3f630640 1 -- 172.21.15.90:0/2069321694 >> 172.21.15.90:0/2069321694 conn(0x7fbe38077920 msgr2=0x7fbe38077d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:39.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.834+0000 7fbe3f630640 1 -- 172.21.15.90:0/2069321694 shutdown_connections 2024-04-25T11:42:39.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.834+0000 7fbe3f630640 1 -- 172.21.15.90:0/2069321694 wait complete. 2024-04-25T11:42:39.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.835+0000 7fbe3f630640 1 Processor -- start 2024-04-25T11:42:39.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.835+0000 7fbe3f630640 1 -- start start 2024-04-25T11:42:39.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.835+0000 7fbe3f630640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe381038d0 0x7fbe3819f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.835+0000 7fbe3f630640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe38108c50 0x7fbe3819fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.835+0000 7fbe3f630640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe381090f0 0x7fbe381996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.835+0000 7fbe3f630640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbe38112e50 con 0x7fbe381038d0 2024-04-25T11:42:39.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.835+0000 7fbe3f630640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbe38112cd0 con 0x7fbe381090f0 2024-04-25T11:42:39.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.835+0000 7fbe3f630640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbe38112fd0 con 0x7fbe38108c50 2024-04-25T11:42:39.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.835+0000 7fbe3d3cc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe381038d0 0x7fbe3819f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.835+0000 7fbe3d3cc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe381038d0 0x7fbe3819f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60496/0 (socket says 172.21.15.90:60496) 2024-04-25T11:42:39.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.835+0000 7fbe3d3cc640 1 -- 172.21.15.90:0/3802568332 learned_addr learned my addr 172.21.15.90:0/3802568332 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:39.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.835+0000 7fbe3dbcd640 1 --2- 172.21.15.90:0/3802568332 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe381090f0 0x7fbe381996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.836+0000 7fbe3cbcb640 1 --2- 172.21.15.90:0/3802568332 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe38108c50 0x7fbe3819fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.836+0000 7fbe3d3cc640 1 -- 172.21.15.90:0/3802568332 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe38108c50 msgr2=0x7fbe3819fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:39.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.836+0000 7fbe3d3cc640 1 --2- 172.21.15.90:0/3802568332 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe38108c50 0x7fbe3819fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.836+0000 7fbe3d3cc640 1 -- 172.21.15.90:0/3802568332 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe381090f0 msgr2=0x7fbe381996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:39.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.836+0000 7fbe3d3cc640 1 --2- 172.21.15.90:0/3802568332 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe381090f0 0x7fbe381996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.836+0000 7fbe3d3cc640 1 -- 172.21.15.90:0/3802568332 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe38199de0 con 0x7fbe381038d0 2024-04-25T11:42:39.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.836+0000 7fbe3cbcb640 1 --2- 172.21.15.90:0/3802568332 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe38108c50 0x7fbe3819fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:42:39.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.836+0000 7fbe3d3cc640 1 --2- 172.21.15.90:0/3802568332 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe381038d0 0x7fbe3819f530 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fbe28006010 tx=0x7fbe28004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:39.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.836+0000 7fbe2e7fc640 1 -- 172.21.15.90:0/3802568332 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe2803f030 con 0x7fbe381038d0 2024-04-25T11:42:39.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.836+0000 7fbe2e7fc640 1 -- 172.21.15.90:0/3802568332 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe28003aa0 con 0x7fbe381038d0 2024-04-25T11:42:39.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.836+0000 7fbe3f630640 1 -- 172.21.15.90:0/3802568332 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe3819a010 con 0x7fbe381038d0 2024-04-25T11:42:39.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.836+0000 7fbe2e7fc640 1 -- 172.21.15.90:0/3802568332 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe28035070 con 0x7fbe381038d0 2024-04-25T11:42:39.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.836+0000 7fbe3f630640 1 -- 172.21.15.90:0/3802568332 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe3819a500 con 0x7fbe381038d0 2024-04-25T11:42:39.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.839+0000 7fbe465b5740 1 Processor -- start 2024-04-25T11:42:39.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.839+0000 7fbe465b5740 1 -- start start 2024-04-25T11:42:39.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.839+0000 7fbe2e7fc640 1 -- 172.21.15.90:0/3802568332 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fbe28042030 con 0x7fbe381038d0 2024-04-25T11:42:39.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.839+0000 7fbe465b5740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562556beb410 0x562556f8b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.839+0000 7fbe465b5740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562556bc2550 0x562556f8b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.839+0000 7fbe465b5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bca730 0x562556f8bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.839+0000 7fbe465b5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562556ef1b00 con 0x562556bca730 2024-04-25T11:42:39.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.839+0000 7fbe465b5740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x562556e8d920 con 0x562556beb410 2024-04-25T11:42:39.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.839+0000 7fbe465b5740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562556eedac0 con 0x562556bc2550 2024-04-25T11:42:39.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.839+0000 7fbe3cbcb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bca730 0x562556f8bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.839+0000 7fbe3cbcb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bca730 0x562556f8bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60512/0 (socket says 172.21.15.90:60512) 2024-04-25T11:42:39.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.839+0000 7fbe3cbcb640 1 -- 172.21.15.90:0/2301112304 learned_addr learned my addr 172.21.15.90:0/2301112304 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:39.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.839+0000 7fbe3dbcd640 1 --2- 172.21.15.90:0/2301112304 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562556beb410 0x562556f8b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.839+0000 7fbe3d3cc640 1 --2- 172.21.15.90:0/2301112304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562556bc2550 0x562556f8b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.840+0000 7fbe2e7fc640 1 --2- 172.21.15.90:0/3802568332 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbe240afc60 0x7fbe240b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.840+0000 7fbe3cbcb640 1 -- 172.21.15.90:0/2301112304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562556bc2550 msgr2=0x562556f8b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:39.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.840+0000 7fbe3cbcb640 1 --2- 172.21.15.90:0/2301112304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562556bc2550 0x562556f8b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.840+0000 7fbe3cbcb640 1 -- 172.21.15.90:0/2301112304 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562556beb410 msgr2=0x562556f8b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:39.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.840+0000 7fbe3cbcb640 1 --2- 172.21.15.90:0/2301112304 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562556beb410 0x562556f8b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.840+0000 7fbe3cbcb640 1 -- 172.21.15.90:0/2301112304 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562556ccb7e0 con 0x562556bca730 2024-04-25T11:42:39.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.840+0000 7fbe3d3cc640 1 --2- 172.21.15.90:0/2301112304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562556bc2550 0x562556f8b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:42:39.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.840+0000 7fbe3dbcd640 1 --2- 172.21.15.90:0/3802568332 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbe240afc60 0x7fbe240b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.840+0000 7fbe2e7fc640 1 -- 172.21.15.90:0/3802568332 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7fbe280ff660 con 0x7fbe381038d0 2024-04-25T11:42:39.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.840+0000 7fbe3cbcb640 1 --2- 172.21.15.90:0/2301112304 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bca730 0x562556f8bf10 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fbe34009c60 tx=0x7fbe34010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50ea0193a27448bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:39.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.840+0000 7fbe237fe640 1 -- 172.21.15.90:0/2301112304 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe34005020 con 0x562556bca730 2024-04-25T11:42:39.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.840+0000 7fbe237fe640 1 -- 172.21.15.90:0/2301112304 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe340047a0 con 0x562556bca730 2024-04-25T11:42:39.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.840+0000 7fbe237fe640 1 -- 172.21.15.90:0/2301112304 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe340126b0 con 0x562556bca730 2024-04-25T11:42:39.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.841+0000 7fbe465b5740 1 -- 172.21.15.90:0/2301112304 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bca730 msgr2=0x562556f8bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:39.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.841+0000 7fbe3dbcd640 1 --2- 172.21.15.90:0/3802568332 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbe240afc60 0x7fbe240b2170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x562556f88cf0 tx=0x7fbe30004a40 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:39.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.841+0000 7fbe465b5740 1 --2- 172.21.15.90:0/2301112304 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bca730 0x562556f8bf10 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fbe34009c60 tx=0x7fbe34010740 comp rx=0 tx=0).stop 2024-04-25T11:42:39.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.841+0000 7fbe465b5740 1 -- 172.21.15.90:0/2301112304 shutdown_connections 2024-04-25T11:42:39.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.841+0000 7fbe465b5740 1 --2- 172.21.15.90:0/2301112304 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bca730 0x562556f8bf10 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.841+0000 7fbe465b5740 1 --2- 172.21.15.90:0/2301112304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562556bc2550 0x562556f8b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.841+0000 7fbe465b5740 1 --2- 172.21.15.90:0/2301112304 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562556beb410 0x562556f8b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.841+0000 7fbe465b5740 1 -- 172.21.15.90:0/2301112304 >> 172.21.15.90:0/2301112304 conn(0x562556bdf560 msgr2=0x562556f87dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:39.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.841+0000 7fbe465b5740 1 -- 172.21.15.90:0/2301112304 shutdown_connections 2024-04-25T11:42:39.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.841+0000 7fbe465b5740 1 -- 172.21.15.90:0/2301112304 wait complete. 2024-04-25T11:42:39.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.841+0000 7fbe465b5740 1 -- start start 2024-04-25T11:42:39.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.842+0000 7fbe465b5740 1 Processor -- start 2024-04-25T11:42:39.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.842+0000 7fbe465b5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bc2550 0x562556f97dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.842+0000 7fbe465b5740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562556bca730 0x562556f98300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.842+0000 7fbe465b5740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562556beb410 0x562556fa4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.842+0000 7fbe465b5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562556f9ab80 con 0x562556bc2550 2024-04-25T11:42:39.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.842+0000 7fbe465b5740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x562556e8b5d0 con 0x562556bca730 2024-04-25T11:42:39.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.842+0000 7fbe465b5740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562556f9ad00 con 0x562556beb410 2024-04-25T11:42:39.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.842+0000 7fbe465b5740 10 client.0 _ll_register_callbacks cb 0x562556e79080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:39.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.842+0000 7fbe3d3cc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bc2550 0x562556f97dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.842+0000 7fbe3d3cc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bc2550 0x562556f97dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60528/0 (socket says 172.21.15.90:60528) 2024-04-25T11:42:39.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.842+0000 7fbe3d3cc640 1 -- 172.21.15.90:0/2823558800 learned_addr learned my addr 172.21.15.90:0/2823558800 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:39.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.842+0000 7fbe3dbcd640 1 --2- 172.21.15.90:0/2823558800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562556beb410 0x562556fa4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.842+0000 7fbe3cbcb640 1 --2- 172.21.15.90:0/2823558800 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562556bca730 0x562556f98300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.842+0000 7fbe3d3cc640 1 -- 172.21.15.90:0/2823558800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562556beb410 msgr2=0x562556fa4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:39.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.842+0000 7fbe3d3cc640 1 --2- 172.21.15.90:0/2823558800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562556beb410 0x562556fa4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.842+0000 7fbe3d3cc640 1 -- 172.21.15.90:0/2823558800 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562556bca730 msgr2=0x562556f98300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:39.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.842+0000 7fbe3d3cc640 1 --2- 172.21.15.90:0/2823558800 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562556bca730 0x562556f98300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.842+0000 7fbe3d3cc640 1 -- 172.21.15.90:0/2823558800 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562556d2bf70 con 0x562556bc2550 2024-04-25T11:42:39.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.843+0000 7fbe3d3cc640 1 --2- 172.21.15.90:0/2823558800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bc2550 0x562556f97dc0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fbe28031440 tx=0x7fbe28009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d8ce54e64133d38 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:39.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.843+0000 7fbe20ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:39.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.843+0000 7fbe20ff9640 1 -- 172.21.15.90:0/2823558800 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe28049070 con 0x562556bc2550 2024-04-25T11:42:39.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.843+0000 7fbe20ff9640 1 -- 172.21.15.90:0/2823558800 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7fbe280c3a00 con 0x562556bc2550 2024-04-25T11:42:39.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.843+0000 7fbe465b5740 1 -- 172.21.15.90:0/2823558800 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe340047a0 con 0x562556bc2550 2024-04-25T11:42:39.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.843+0000 7fbe20ff9640 1 -- 172.21.15.90:0/2823558800 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbe280d05d0 con 0x562556bc2550 2024-04-25T11:42:39.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.843+0000 7fbe465b5740 10 client.17958 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:39.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.843+0000 7fbe465b5740 1 -- 172.21.15.90:0/2823558800 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562556bc3170 con 0x562556bc2550 2024-04-25T11:42:39.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.843+0000 7fbe20ff9640 1 -- 172.21.15.90:0/2823558800 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe340047a0 con 0x562556bc2550 2024-04-25T11:42:39.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.843+0000 7fbe20ff9640 1 -- 172.21.15.90:0/2823558800 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fbe280b00b0 con 0x562556bc2550 2024-04-25T11:42:39.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.843+0000 7fbe465b5740 10 client.17958 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:39.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.843+0000 7fbe465b5740 10 client.17958 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:42:39.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.843+0000 7fbe465b5740 20 client.17958 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:39.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.843+0000 7fbe465b5740 1 --2- 172.21.15.90:0/2823558800 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562556baa1f0 0x562556faa090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.843+0000 7fbe465b5740 4 client.17958 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:42:39.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.843+0000 7fbe465b5740 1 -- 172.21.15.90:0/2823558800 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562556eff710 con 0x562556baa1f0 2024-04-25T11:42:39.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.844+0000 7fbe3cbcb640 1 --2- 172.21.15.90:0/2823558800 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562556baa1f0 0x562556faa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.844+0000 7fbe20ff9640 10 client.17958 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:39.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.844+0000 7fbe3cbcb640 1 --2- 172.21.15.90:0/2823558800 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562556baa1f0 0x562556faa090 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be6de6b648d6e4c7 server_cookie=3a5120705f0db123 in_seq=0 out_seq=0 2024-04-25T11:42:39.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.845+0000 7fbe20ff9640 1 -- 172.21.15.90:0/2823558800 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562556eff710 con 0x562556baa1f0 2024-04-25T11:42:39.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.845+0000 7fbe20ff9640 10 client.17958 handle_command_reply: tid=0 2024-04-25T11:42:39.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.845+0000 7fbe465b5740 1 client.17958 shutdown 2024-04-25T11:42:39.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.845+0000 7fbe465b5740 1 -- 172.21.15.90:0/2823558800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bc2550 msgr2=0x562556f97dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:39.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.845+0000 7fbe465b5740 1 --2- 172.21.15.90:0/2823558800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bc2550 0x562556f97dc0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fbe28031440 tx=0x7fbe28009260 comp rx=0 tx=0).stop 2024-04-25T11:42:39.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.845+0000 7fbe465b5740 1 -- 172.21.15.90:0/2823558800 shutdown_connections 2024-04-25T11:42:39.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.845+0000 7fbe465b5740 1 --2- 172.21.15.90:0/2823558800 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562556baa1f0 0x562556faa090 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.845+0000 7fbe465b5740 1 --2- 172.21.15.90:0/2823558800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562556beb410 0x562556fa4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.845+0000 7fbe465b5740 1 --2- 172.21.15.90:0/2823558800 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562556bca730 0x562556f98300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.845+0000 7fbe465b5740 1 --2- 172.21.15.90:0/2823558800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bc2550 0x562556f97dc0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.845+0000 7fbe20ff9640 0 client.17958 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:39.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.845+0000 7fbe465b5740 1 -- 172.21.15.90:0/2823558800 >> 172.21.15.90:0/2823558800 conn(0x562556bdf560 msgr2=0x562556f8d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:39.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.845+0000 7fbe465b5740 1 -- 172.21.15.90:0/2823558800 shutdown_connections 2024-04-25T11:42:39.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.845+0000 7fbe465b5740 1 -- 172.21.15.90:0/2823558800 wait complete. 2024-04-25T11:42:39.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.845+0000 7fbe465b5740 20 client.17958 trim_cache size 0 max 16384 2024-04-25T11:42:39.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.863+0000 7fbe465b5740 1 Processor -- start 2024-04-25T11:42:39.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.863+0000 7fbe465b5740 1 -- start start 2024-04-25T11:42:39.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.864+0000 7fbe465b5740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562556bc2550 0x562556f86bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.864+0000 7fbe465b5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bca730 0x562556faf800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.864+0000 7fbe465b5740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562556beb410 0x562556fafd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.864+0000 7fbe465b5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbe381ad9b0 con 0x562556bca730 2024-04-25T11:42:39.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.864+0000 7fbe465b5740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbe381af4f0 con 0x562556bc2550 2024-04-25T11:42:39.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.864+0000 7fbe465b5740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbe381ae080 con 0x562556beb410 2024-04-25T11:42:39.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.864+0000 7fbe3cbcb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bca730 0x562556faf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.864+0000 7fbe3cbcb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bca730 0x562556faf800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60544/0 (socket says 172.21.15.90:60544) 2024-04-25T11:42:39.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.864+0000 7fbe3cbcb640 1 -- 172.21.15.90:0/1167813253 learned_addr learned my addr 172.21.15.90:0/1167813253 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:39.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.864+0000 7fbe3d3cc640 1 --2- 172.21.15.90:0/1167813253 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562556bc2550 0x562556f86bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.864+0000 7fbe3cbcb640 1 -- 172.21.15.90:0/1167813253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562556beb410 msgr2=0x562556fafd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:39.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.864+0000 7fbe3cbcb640 1 --2- 172.21.15.90:0/1167813253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562556beb410 0x562556fafd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.864+0000 7fbe3cbcb640 1 -- 172.21.15.90:0/1167813253 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562556bc2550 msgr2=0x562556f86bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:39.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.864+0000 7fbe3cbcb640 1 --2- 172.21.15.90:0/1167813253 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562556bc2550 0x562556f86bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.864+0000 7fbe3cbcb640 1 -- 172.21.15.90:0/1167813253 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562556f889c0 con 0x562556bca730 2024-04-25T11:42:39.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.864+0000 7fbe3d3cc640 1 --2- 172.21.15.90:0/1167813253 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562556bc2550 0x562556f86bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:42:39.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.864+0000 7fbe3cbcb640 1 --2- 172.21.15.90:0/1167813253 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bca730 0x562556faf800 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fbe34000c00 tx=0x7fbe34009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94684fcbc6fed8be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:39.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.864+0000 7fbe22ffd640 1 -- 172.21.15.90:0/1167813253 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3402d070 con 0x562556bca730 2024-04-25T11:42:39.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.864+0000 7fbe22ffd640 1 -- 172.21.15.90:0/1167813253 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe34004640 con 0x562556bca730 2024-04-25T11:42:39.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.864+0000 7fbe22ffd640 1 -- 172.21.15.90:0/1167813253 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe340283d0 con 0x562556bca730 2024-04-25T11:42:39.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.865+0000 7fbe465b5740 1 -- 172.21.15.90:0/1167813253 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bca730 msgr2=0x562556faf800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:39.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.865+0000 7fbe465b5740 1 --2- 172.21.15.90:0/1167813253 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bca730 0x562556faf800 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fbe34000c00 tx=0x7fbe34009f90 comp rx=0 tx=0).stop 2024-04-25T11:42:39.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.865+0000 7fbe465b5740 1 -- 172.21.15.90:0/1167813253 shutdown_connections 2024-04-25T11:42:39.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.865+0000 7fbe465b5740 1 --2- 172.21.15.90:0/1167813253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562556beb410 0x562556fafd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.865+0000 7fbe465b5740 1 --2- 172.21.15.90:0/1167813253 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bca730 0x562556faf800 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.865+0000 7fbe465b5740 1 --2- 172.21.15.90:0/1167813253 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562556bc2550 0x562556f86bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.865+0000 7fbe465b5740 1 -- 172.21.15.90:0/1167813253 >> 172.21.15.90:0/1167813253 conn(0x562556bdf560 msgr2=0x562556f864c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:39.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.865+0000 7fbe465b5740 1 -- 172.21.15.90:0/1167813253 shutdown_connections 2024-04-25T11:42:39.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.865+0000 7fbe465b5740 1 -- 172.21.15.90:0/1167813253 wait complete. 2024-04-25T11:42:39.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.866+0000 7fbe465b5740 1 -- start start 2024-04-25T11:42:39.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.866+0000 7fbe465b5740 1 Processor -- start 2024-04-25T11:42:39.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.866+0000 7fbe465b5740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562556bc2550 0x562556fa0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.866+0000 7fbe465b5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bca730 0x562556fa0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.866+0000 7fbe465b5740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562556beb410 0x562556f9b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.866+0000 7fbe465b5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbe381aee20 con 0x562556bca730 2024-04-25T11:42:39.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.866+0000 7fbe465b5740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbe381ae750 con 0x562556beb410 2024-04-25T11:42:39.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.866+0000 7fbe465b5740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbe381ab9b0 con 0x562556bc2550 2024-04-25T11:42:39.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.866+0000 7fbe465b5740 10 client.0 _ll_register_callbacks cb 0x7fbe280c2f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:39.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.866+0000 7fbe3cbcb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bca730 0x562556fa0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.866+0000 7fbe3cbcb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bca730 0x562556fa0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60550/0 (socket says 172.21.15.90:60550) 2024-04-25T11:42:39.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.866+0000 7fbe3cbcb640 1 -- 172.21.15.90:0/3737999451 learned_addr learned my addr 172.21.15.90:0/3737999451 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:39.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.866+0000 7fbe3dbcd640 1 --2- 172.21.15.90:0/3737999451 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562556beb410 0x562556f9b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.866+0000 7fbe3d3cc640 1 --2- 172.21.15.90:0/3737999451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562556bc2550 0x562556fa0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.866+0000 7fbe3cbcb640 1 -- 172.21.15.90:0/3737999451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562556bc2550 msgr2=0x562556fa0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:39.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.866+0000 7fbe3cbcb640 1 --2- 172.21.15.90:0/3737999451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562556bc2550 0x562556fa0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.866+0000 7fbe3cbcb640 1 -- 172.21.15.90:0/3737999451 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562556beb410 msgr2=0x562556f9b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:39.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.866+0000 7fbe3cbcb640 1 --2- 172.21.15.90:0/3737999451 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562556beb410 0x562556f9b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.867+0000 7fbe3cbcb640 1 -- 172.21.15.90:0/3737999451 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562556f9bdb0 con 0x562556bca730 2024-04-25T11:42:39.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.867+0000 7fbe3d3cc640 1 --2- 172.21.15.90:0/3737999451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562556bc2550 0x562556fa0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:42:39.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.867+0000 7fbe3dbcd640 1 --2- 172.21.15.90:0/3737999451 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562556beb410 0x562556f9b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:42:39.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.867+0000 7fbe3cbcb640 1 --2- 172.21.15.90:0/3737999451 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bca730 0x562556fa0640 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fbe34004aa0 tx=0x7fbe34004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6d0d119335f4696 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:39.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.867+0000 7fbe20ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:39.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.867+0000 7fbe20ff9640 1 -- 172.21.15.90:0/3737999451 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3402d070 con 0x562556bca730 2024-04-25T11:42:39.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.867+0000 7fbe20ff9640 1 -- 172.21.15.90:0/3737999451 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7fbe34005200 con 0x562556bca730 2024-04-25T11:42:39.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.867+0000 7fbe465b5740 1 -- 172.21.15.90:0/3737999451 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe34004640 con 0x562556bca730 2024-04-25T11:42:39.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.867+0000 7fbe20ff9640 1 -- 172.21.15.90:0/3737999451 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbe34009870 con 0x562556bca730 2024-04-25T11:42:39.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.867+0000 7fbe465b5740 10 client.17964 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:39.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.867+0000 7fbe465b5740 1 -- 172.21.15.90:0/3737999451 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562557011e40 con 0x562556bca730 2024-04-25T11:42:39.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.867+0000 7fbe20ff9640 1 -- 172.21.15.90:0/3737999451 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe34004640 con 0x562556bca730 2024-04-25T11:42:39.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.867+0000 7fbe20ff9640 1 -- 172.21.15.90:0/3737999451 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fbe34037030 con 0x562556bca730 2024-04-25T11:42:39.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.867+0000 7fbe465b5740 10 client.17964 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:39.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.867+0000 7fbe465b5740 10 client.17964 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:42:39.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.867+0000 7fbe465b5740 20 client.17964 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:39.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.867+0000 7fbe465b5740 1 --2- 172.21.15.90:0/3737999451 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562556b7f630 0x5625570143b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.867+0000 7fbe465b5740 4 client.17964 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T11:42:39.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.867+0000 7fbe465b5740 1 -- 172.21.15.90:0/3737999451 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562556ef9580 con 0x562556b7f630 2024-04-25T11:42:39.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.868+0000 7fbe3d3cc640 1 --2- 172.21.15.90:0/3737999451 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562556b7f630 0x5625570143b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.868+0000 7fbe20ff9640 10 client.17964 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:39.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.868+0000 7fbe3d3cc640 1 --2- 172.21.15.90:0/3737999451 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562556b7f630 0x5625570143b0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c245a29a4a592647 server_cookie=1dc3c9464a97a210 in_seq=0 out_seq=0 2024-04-25T11:42:39.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.869+0000 7fbe20ff9640 1 -- 172.21.15.90:0/3737999451 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562556ef9580 con 0x562556b7f630 2024-04-25T11:42:39.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.869+0000 7fbe20ff9640 10 client.17964 handle_command_reply: tid=0 2024-04-25T11:42:39.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.869+0000 7fbe465b5740 1 client.17964 shutdown 2024-04-25T11:42:39.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.869+0000 7fbe465b5740 1 -- 172.21.15.90:0/3737999451 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bca730 msgr2=0x562556fa0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:39.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.869+0000 7fbe465b5740 1 --2- 172.21.15.90:0/3737999451 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bca730 0x562556fa0640 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fbe34004aa0 tx=0x7fbe34004ad0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.869+0000 7fbe465b5740 1 -- 172.21.15.90:0/3737999451 shutdown_connections 2024-04-25T11:42:39.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.869+0000 7fbe465b5740 1 --2- 172.21.15.90:0/3737999451 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562556b7f630 0x5625570143b0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.869+0000 7fbe465b5740 1 --2- 172.21.15.90:0/3737999451 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562556beb410 0x562556f9b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.869+0000 7fbe465b5740 1 --2- 172.21.15.90:0/3737999451 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562556bca730 0x562556fa0640 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.869+0000 7fbe465b5740 1 --2- 172.21.15.90:0/3737999451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562556bc2550 0x562556fa0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.869+0000 7fbe20ff9640 0 client.17964 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:39.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.869+0000 7fbe465b5740 1 -- 172.21.15.90:0/3737999451 >> 172.21.15.90:0/3737999451 conn(0x562556bdf560 msgr2=0x562556f8d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:39.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.870+0000 7fbe465b5740 1 -- 172.21.15.90:0/3737999451 shutdown_connections 2024-04-25T11:42:39.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.870+0000 7fbe465b5740 1 -- 172.21.15.90:0/3737999451 wait complete. 2024-04-25T11:42:39.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.870+0000 7fbe465b5740 20 client.17964 trim_cache size 0 max 16384 2024-04-25T11:42:39.863 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:42:39.863 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-25T11:42:39.863 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "3f7b92e1-94cb-491d-ac35-f599fb1c4b6a", 2024-04-25T11:42:39.863 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-25T11:42:39.863 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:42:39.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.870+0000 7fbe3f630640 1 -- 172.21.15.90:0/3802568332 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbe240afc60 msgr2=0x7fbe240b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:39.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.870+0000 7fbe3f630640 1 --2- 172.21.15.90:0/3802568332 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbe240afc60 0x7fbe240b2170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x562556f88cf0 tx=0x7fbe30004a40 comp rx=0 tx=0).stop 2024-04-25T11:42:39.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.870+0000 7fbe3f630640 1 -- 172.21.15.90:0/3802568332 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe381038d0 msgr2=0x7fbe3819f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:39.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.870+0000 7fbe3f630640 1 --2- 172.21.15.90:0/3802568332 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe381038d0 0x7fbe3819f530 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fbe28006010 tx=0x7fbe28004060 comp rx=0 tx=0).stop 2024-04-25T11:42:39.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.870+0000 7fbe3f630640 1 -- 172.21.15.90:0/3802568332 shutdown_connections 2024-04-25T11:42:39.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.870+0000 7fbe3f630640 1 --2- 172.21.15.90:0/3802568332 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbe240afc60 0x7fbe240b2170 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.870+0000 7fbe3f630640 1 --2- 172.21.15.90:0/3802568332 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe381090f0 0x7fbe381996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.870+0000 7fbe3f630640 1 --2- 172.21.15.90:0/3802568332 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe38108c50 0x7fbe3819fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.870+0000 7fbe3f630640 1 --2- 172.21.15.90:0/3802568332 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe381038d0 0x7fbe3819f530 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.870+0000 7fbe3f630640 1 -- 172.21.15.90:0/3802568332 >> 172.21.15.90:0/3802568332 conn(0x7fbe38077920 msgr2=0x7fbe38104330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:39.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.870+0000 7fbe3f630640 1 -- 172.21.15.90:0/3802568332 shutdown_connections 2024-04-25T11:42:39.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.870+0000 7fbe3f630640 1 -- 172.21.15.90:0/3802568332 wait complete. 2024-04-25T11:42:39.869 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:42:39.869 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:42:39.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:39 smithi090 ceph-mon[29575]: pgmap v232: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 13 KiB/s rd, 224 KiB/s wr, 13 op/s 2024-04-25T11:42:39.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.994+0000 7fdf118e2640 1 -- 172.21.15.90:0/1694844422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf0c109b30 msgr2=0x7fdf0c103720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:39.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.994+0000 7fdf118e2640 1 --2- 172.21.15.90:0/1694844422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf0c109b30 0x7fdf0c103720 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fdefc009900 tx=0x7fdefc02a410 comp rx=0 tx=0).stop 2024-04-25T11:42:39.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.994+0000 7fdf118e2640 1 -- 172.21.15.90:0/1694844422 shutdown_connections 2024-04-25T11:42:39.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.994+0000 7fdf118e2640 1 --2- 172.21.15.90:0/1694844422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf0c1042d0 0x7fdf0c110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.994+0000 7fdf118e2640 1 --2- 172.21.15.90:0/1694844422 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdf0c109fd0 0x7fdf0c103c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.994+0000 7fdf118e2640 1 --2- 172.21.15.90:0/1694844422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf0c109b30 0x7fdf0c103720 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.994+0000 7fdf118e2640 1 -- 172.21.15.90:0/1694844422 >> 172.21.15.90:0/1694844422 conn(0x7fdf0c079280 msgr2=0x7fdf0c0796c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:39.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.994+0000 7fdf118e2640 1 -- 172.21.15.90:0/1694844422 shutdown_connections 2024-04-25T11:42:39.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.994+0000 7fdf118e2640 1 -- 172.21.15.90:0/1694844422 wait complete. 2024-04-25T11:42:39.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.994+0000 7fdf118e2640 1 Processor -- start 2024-04-25T11:42:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.995+0000 7fdf118e2640 1 -- start start 2024-04-25T11:42:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.995+0000 7fdf118e2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf0c1042d0 0x7fdf0c19d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.995+0000 7fdf118e2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdf0c109b30 0x7fdf0c19dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.995+0000 7fdf118e2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf0c109fd0 0x7fdf0c1a86e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.995+0000 7fdf118e2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdf0c113d80 con 0x7fdf0c1042d0 2024-04-25T11:42:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.995+0000 7fdf118e2640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fdf0c113c00 con 0x7fdf0c109b30 2024-04-25T11:42:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.995+0000 7fdf118e2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdf0c113f00 con 0x7fdf0c109fd0 2024-04-25T11:42:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.995+0000 7fdf0affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf0c1042d0 0x7fdf0c19d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.995+0000 7fdf0affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf0c1042d0 0x7fdf0c19d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60582/0 (socket says 172.21.15.90:60582) 2024-04-25T11:42:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.995+0000 7fdf0affd640 1 -- 172.21.15.90:0/1573881474 learned_addr learned my addr 172.21.15.90:0/1573881474 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.995+0000 7fdf0a7fc640 1 --2- 172.21.15.90:0/1573881474 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdf0c109b30 0x7fdf0c19dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.995+0000 7fdf0affd640 1 -- 172.21.15.90:0/1573881474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf0c109fd0 msgr2=0x7fdf0c1a86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.995+0000 7fdf0affd640 1 --2- 172.21.15.90:0/1573881474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf0c109fd0 0x7fdf0c1a86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.995+0000 7fdf0affd640 1 -- 172.21.15.90:0/1573881474 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdf0c109b30 msgr2=0x7fdf0c19dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.995+0000 7fdf0affd640 1 --2- 172.21.15.90:0/1573881474 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdf0c109b30 0x7fdf0c19dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.995+0000 7fdf0affd640 1 -- 172.21.15.90:0/1573881474 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf0c1a8c20 con 0x7fdf0c1042d0 2024-04-25T11:42:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.995+0000 7fdf0affd640 1 --2- 172.21.15.90:0/1573881474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf0c1042d0 0x7fdf0c19d660 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fdefc031a90 tx=0x7fdefc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.996+0000 7fdf108e0640 1 -- 172.21.15.90:0/1573881474 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdefc03f030 con 0x7fdf0c1042d0 2024-04-25T11:42:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.996+0000 7fdf108e0640 1 -- 172.21.15.90:0/1573881474 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdefc003cb0 con 0x7fdf0c1042d0 2024-04-25T11:42:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.996+0000 7fdf118e2640 1 -- 172.21.15.90:0/1573881474 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf0c1a8df0 con 0x7fdf0c1042d0 2024-04-25T11:42:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.996+0000 7fdf108e0640 1 -- 172.21.15.90:0/1573881474 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdefc035070 con 0x7fdf0c1042d0 2024-04-25T11:42:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.996+0000 7fdf118e2640 1 -- 172.21.15.90:0/1573881474 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf0c1a91a0 con 0x7fdf0c1042d0 2024-04-25T11:42:39.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.998+0000 7fdf108e0640 1 -- 172.21.15.90:0/1573881474 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fdefc042030 con 0x7fdf0c1042d0 2024-04-25T11:42:39.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.998+0000 7fdf1878d740 1 Processor -- start 2024-04-25T11:42:39.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.998+0000 7fdf1878d740 1 -- start start 2024-04-25T11:42:39.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.999+0000 7fdf1878d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7567e1390 0x55a756b81a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.999+0000 7fdf1878d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7567b84d0 0x55a756b81fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.999+0000 7fdf1878d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567c06b0 0x55a756b8e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.999+0000 7fdf1878d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a756ae7d80 con 0x55a7567c06b0 2024-04-25T11:42:39.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.999+0000 7fdf1878d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55a756aa18b0 con 0x55a7567e1390 2024-04-25T11:42:39.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.999+0000 7fdf1878d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55a756ae3d40 con 0x55a7567b84d0 2024-04-25T11:42:39.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.999+0000 7fdf0a7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567c06b0 0x55a756b8e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.999+0000 7fdf0a7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567c06b0 0x55a756b8e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60590/0 (socket says 172.21.15.90:60590) 2024-04-25T11:42:39.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.999+0000 7fdf0a7fc640 1 -- 172.21.15.90:0/3233409979 learned_addr learned my addr 172.21.15.90:0/3233409979 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:39.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.999+0000 7fdf0affd640 1 --2- 172.21.15.90:0/3233409979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7567b84d0 0x55a756b81fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.999+0000 7fdf108e0640 1 --2- 172.21.15.90:0/1573881474 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fdef40afc60 0x7fdef40b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.999+0000 7fdf0b7fe640 1 --2- 172.21.15.90:0/3233409979 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7567e1390 0x55a756b81a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.999+0000 7fdf0a7fc640 1 -- 172.21.15.90:0/3233409979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7567b84d0 msgr2=0x55a756b81fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:39.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.999+0000 7fdf0a7fc640 1 --2- 172.21.15.90:0/3233409979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7567b84d0 0x55a756b81fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.999+0000 7fdf0a7fc640 1 -- 172.21.15.90:0/3233409979 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7567e1390 msgr2=0x55a756b81a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:39.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.999+0000 7fdf0a7fc640 1 --2- 172.21.15.90:0/3233409979 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7567e1390 0x55a756b81a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.999+0000 7fdf0a7fc640 1 -- 172.21.15.90:0/3233409979 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a756a72a40 con 0x55a7567c06b0 2024-04-25T11:42:39.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.999+0000 7fdf0b7fe640 1 --2- 172.21.15.90:0/3233409979 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7567e1390 0x55a756b81a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:42:39.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.999+0000 7fdf0b7fe640 1 --2- 172.21.15.90:0/1573881474 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fdef40afc60 0x7fdef40b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.999+0000 7fdf108e0640 1 -- 172.21.15.90:0/1573881474 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7fdefc0ff780 con 0x7fdf0c1042d0 2024-04-25T11:42:39.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.999+0000 7fdf0a7fc640 1 --2- 172.21.15.90:0/3233409979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567c06b0 0x55a756b8e8f0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fdf0000a880 tx=0x7fdf0000ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78de4c6eb160a950 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:39.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:39.999+0000 7fdef1ffb640 1 -- 172.21.15.90:0/3233409979 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf00015070 con 0x55a7567c06b0 2024-04-25T11:42:39.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.000+0000 7fdef1ffb640 1 -- 172.21.15.90:0/3233409979 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf000113f0 con 0x55a7567c06b0 2024-04-25T11:42:39.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.000+0000 7fdef1ffb640 1 -- 172.21.15.90:0/3233409979 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf000185c0 con 0x55a7567c06b0 2024-04-25T11:42:39.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.000+0000 7fdf1878d740 1 -- 172.21.15.90:0/3233409979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567c06b0 msgr2=0x55a756b8e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:39.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.000+0000 7fdf0b7fe640 1 --2- 172.21.15.90:0/1573881474 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fdef40afc60 0x7fdef40b2170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdf04005fd0 tx=0x7fdf04004a40 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:39.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.000+0000 7fdf1878d740 1 --2- 172.21.15.90:0/3233409979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567c06b0 0x55a756b8e8f0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fdf0000a880 tx=0x7fdf0000ad40 comp rx=0 tx=0).stop 2024-04-25T11:42:39.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.000+0000 7fdf1878d740 1 -- 172.21.15.90:0/3233409979 shutdown_connections 2024-04-25T11:42:39.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.000+0000 7fdf1878d740 1 --2- 172.21.15.90:0/3233409979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567c06b0 0x55a756b8e8f0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.000+0000 7fdf1878d740 1 --2- 172.21.15.90:0/3233409979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7567b84d0 0x55a756b81fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.000+0000 7fdf1878d740 1 --2- 172.21.15.90:0/3233409979 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7567e1390 0x55a756b81a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.000+0000 7fdf1878d740 1 -- 172.21.15.90:0/3233409979 >> 172.21.15.90:0/3233409979 conn(0x55a7567d54e0 msgr2=0x55a756b7e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:39.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.000+0000 7fdf1878d740 1 -- 172.21.15.90:0/3233409979 shutdown_connections 2024-04-25T11:42:39.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.000+0000 7fdf1878d740 1 -- 172.21.15.90:0/3233409979 wait complete. 2024-04-25T11:42:39.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.001+0000 7fdf1878d740 1 -- start start 2024-04-25T11:42:39.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.001+0000 7fdf1878d740 1 Processor -- start 2024-04-25T11:42:39.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.001+0000 7fdf1878d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7567b84d0 0x55a756b8c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.001+0000 7fdf1878d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7567c06b0 0x55a756b8ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.001+0000 7fdf1878d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567e1390 0x55a756b984a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.001+0000 7fdf1878d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a756b8efe0 con 0x55a7567e1390 2024-04-25T11:42:39.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.001+0000 7fdf1878d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55a756a6ed10 con 0x55a7567c06b0 2024-04-25T11:42:39.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.001+0000 7fdf1878d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55a756b8f160 con 0x55a7567b84d0 2024-04-25T11:42:39.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.001+0000 7fdf1878d740 10 client.0 _ll_register_callbacks cb 0x55a756a9b210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:39.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.001+0000 7fdf0b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567e1390 0x55a756b984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.001+0000 7fdf0b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567e1390 0x55a756b984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60604/0 (socket says 172.21.15.90:60604) 2024-04-25T11:42:39.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.001+0000 7fdf0b7fe640 1 -- 172.21.15.90:0/523721766 learned_addr learned my addr 172.21.15.90:0/523721766 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:39.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.002+0000 7fdf0affd640 1 --2- 172.21.15.90:0/523721766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7567b84d0 0x55a756b8c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.002+0000 7fdf0a7fc640 1 --2- 172.21.15.90:0/523721766 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7567c06b0 0x55a756b8ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.002+0000 7fdf0b7fe640 1 -- 172.21.15.90:0/523721766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7567b84d0 msgr2=0x55a756b8c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:39.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.002+0000 7fdf0b7fe640 1 --2- 172.21.15.90:0/523721766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7567b84d0 0x55a756b8c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.002+0000 7fdf0b7fe640 1 -- 172.21.15.90:0/523721766 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7567c06b0 msgr2=0x55a756b8ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:39.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.002+0000 7fdf0b7fe640 1 --2- 172.21.15.90:0/523721766 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7567c06b0 0x55a756b8ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.002+0000 7fdf0b7fe640 1 -- 172.21.15.90:0/523721766 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7568c37d0 con 0x55a7567e1390 2024-04-25T11:42:39.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.002+0000 7fdf0affd640 1 --2- 172.21.15.90:0/523721766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7567b84d0 0x55a756b8c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:42:39.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.002+0000 7fdf0b7fe640 1 --2- 172.21.15.90:0/523721766 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567e1390 0x55a756b984a0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fdf0400fd70 tx=0x7fdf04005f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbc5f5c8140279da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:39.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.002+0000 7fdeeaffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:39.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.002+0000 7fdeeaffd640 1 -- 172.21.15.90:0/523721766 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0401f070 con 0x55a7567e1390 2024-04-25T11:42:39.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.002+0000 7fdeeaffd640 1 -- 172.21.15.90:0/523721766 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7fdf0401b430 con 0x55a7567e1390 2024-04-25T11:42:39.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.002+0000 7fdf1878d740 1 -- 172.21.15.90:0/523721766 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf000113f0 con 0x55a7567e1390 2024-04-25T11:42:39.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.002+0000 7fdeeaffd640 1 -- 172.21.15.90:0/523721766 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdf0402b050 con 0x55a7567e1390 2024-04-25T11:42:39.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.002+0000 7fdf1878d740 10 client.17976 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:39.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.002+0000 7fdf1878d740 1 -- 172.21.15.90:0/523721766 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a7567f93f0 con 0x55a7567e1390 2024-04-25T11:42:39.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.002+0000 7fdeeaffd640 1 -- 172.21.15.90:0/523721766 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf000113f0 con 0x55a7567e1390 2024-04-25T11:42:39.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.003+0000 7fdeeaffd640 1 -- 172.21.15.90:0/523721766 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fdf040195b0 con 0x55a7567e1390 2024-04-25T11:42:39.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.003+0000 7fdf1878d740 10 client.17976 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:39.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.003+0000 7fdf1878d740 10 client.17976 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:42:39.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.003+0000 7fdf1878d740 20 client.17976 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:39.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.003+0000 7fdf1878d740 1 --2- 172.21.15.90:0/523721766 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55a75679ff60 0x55a756b9e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.003+0000 7fdf1878d740 4 client.17976 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:42:39.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.003+0000 7fdf1878d740 1 -- 172.21.15.90:0/523721766 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a756af5990 con 0x55a75679ff60 2024-04-25T11:42:39.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.003+0000 7fdf0affd640 1 --2- 172.21.15.90:0/523721766 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55a75679ff60 0x55a756b9e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.003+0000 7fdeeaffd640 10 client.17976 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:39.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.003+0000 7fdf0affd640 1 --2- 172.21.15.90:0/523721766 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55a75679ff60 0x55a756b9e620 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ddc161520ed7e20 server_cookie=5e6fde3c8b10dc1a in_seq=0 out_seq=0 2024-04-25T11:42:39.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.004+0000 7fdeeaffd640 1 -- 172.21.15.90:0/523721766 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a756af5990 con 0x55a75679ff60 2024-04-25T11:42:39.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.004+0000 7fdeeaffd640 10 client.17976 handle_command_reply: tid=0 2024-04-25T11:42:39.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.004+0000 7fdf1878d740 1 client.17976 shutdown 2024-04-25T11:42:39.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.004+0000 7fdf1878d740 1 -- 172.21.15.90:0/523721766 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567e1390 msgr2=0x55a756b984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:39.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.004+0000 7fdf1878d740 1 --2- 172.21.15.90:0/523721766 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567e1390 0x55a756b984a0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fdf0400fd70 tx=0x7fdf04005f40 comp rx=0 tx=0).stop 2024-04-25T11:42:39.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.005+0000 7fdf1878d740 1 -- 172.21.15.90:0/523721766 shutdown_connections 2024-04-25T11:42:39.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.005+0000 7fdf1878d740 1 --2- 172.21.15.90:0/523721766 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55a75679ff60 0x55a756b9e620 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.005+0000 7fdf1878d740 1 --2- 172.21.15.90:0/523721766 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567e1390 0x55a756b984a0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.005+0000 7fdeeaffd640 0 client.17976 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:39.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.005+0000 7fdf1878d740 1 --2- 172.21.15.90:0/523721766 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7567c06b0 0x55a756b8ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.005+0000 7fdf1878d740 1 --2- 172.21.15.90:0/523721766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7567b84d0 0x55a756b8c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.005+0000 7fdf1878d740 1 -- 172.21.15.90:0/523721766 >> 172.21.15.90:0/523721766 conn(0x55a7567d54e0 msgr2=0x55a756b83780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:39.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.005+0000 7fdf1878d740 1 -- 172.21.15.90:0/523721766 shutdown_connections 2024-04-25T11:42:39.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.005+0000 7fdf1878d740 1 -- 172.21.15.90:0/523721766 wait complete. 2024-04-25T11:42:39.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.005+0000 7fdf1878d740 20 client.17976 trim_cache size 0 max 16384 2024-04-25T11:42:40.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.022+0000 7fdf1878d740 1 Processor -- start 2024-04-25T11:42:40.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.022+0000 7fdf1878d740 1 -- start start 2024-04-25T11:42:40.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.022+0000 7fdf1878d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a75679ff60 0x55a756b94430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:40.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.022+0000 7fdf1878d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7567b84d0 0x55a756b94970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:40.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.022+0000 7fdf1878d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567c06b0 0x55a756b7d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:40.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.022+0000 7fdf1878d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdf0c1b1d90 con 0x55a7567c06b0 2024-04-25T11:42:40.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.022+0000 7fdf1878d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fdf0c1b38d0 con 0x55a75679ff60 2024-04-25T11:42:40.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.022+0000 7fdf1878d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdf0c1b2460 con 0x55a7567b84d0 2024-04-25T11:42:40.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.023+0000 7fdf0b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567c06b0 0x55a756b7d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:40.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.023+0000 7fdf0b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567c06b0 0x55a756b7d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60608/0 (socket says 172.21.15.90:60608) 2024-04-25T11:42:40.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.023+0000 7fdf0b7fe640 1 -- 172.21.15.90:0/917669702 learned_addr learned my addr 172.21.15.90:0/917669702 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:40.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.023+0000 7fdf0affd640 1 --2- 172.21.15.90:0/917669702 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a75679ff60 0x55a756b94430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:40.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.023+0000 7fdf0a7fc640 1 --2- 172.21.15.90:0/917669702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7567b84d0 0x55a756b94970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:40.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.023+0000 7fdf0b7fe640 1 -- 172.21.15.90:0/917669702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7567b84d0 msgr2=0x55a756b94970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:40.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.023+0000 7fdf0b7fe640 1 --2- 172.21.15.90:0/917669702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7567b84d0 0x55a756b94970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:40.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.023+0000 7fdf0b7fe640 1 -- 172.21.15.90:0/917669702 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a75679ff60 msgr2=0x55a756b94430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:40.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.023+0000 7fdf0b7fe640 1 --2- 172.21.15.90:0/917669702 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a75679ff60 0x55a756b94430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:40.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.023+0000 7fdf0b7fe640 1 -- 172.21.15.90:0/917669702 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7567b90f0 con 0x55a7567c06b0 2024-04-25T11:42:40.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.023+0000 7fdf0b7fe640 1 --2- 172.21.15.90:0/917669702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567c06b0 0x55a756b7d9d0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fdf0400dc10 tx=0x7fdf04007aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7ee3e4d34c2c6d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:40.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.023+0000 7fdef17fa640 1 -- 172.21.15.90:0/917669702 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0401f040 con 0x55a7567c06b0 2024-04-25T11:42:40.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.023+0000 7fdef17fa640 1 -- 172.21.15.90:0/917669702 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf04033030 con 0x55a7567c06b0 2024-04-25T11:42:40.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.023+0000 7fdef17fa640 1 -- 172.21.15.90:0/917669702 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf04005280 con 0x55a7567c06b0 2024-04-25T11:42:40.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.024+0000 7fdf1878d740 1 -- 172.21.15.90:0/917669702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567c06b0 msgr2=0x55a756b7d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:40.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.024+0000 7fdf1878d740 1 --2- 172.21.15.90:0/917669702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567c06b0 0x55a756b7d9d0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fdf0400dc10 tx=0x7fdf04007aa0 comp rx=0 tx=0).stop 2024-04-25T11:42:40.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.024+0000 7fdf1878d740 1 -- 172.21.15.90:0/917669702 shutdown_connections 2024-04-25T11:42:40.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.024+0000 7fdf1878d740 1 --2- 172.21.15.90:0/917669702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567c06b0 0x55a756b7d9d0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:40.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.024+0000 7fdf1878d740 1 --2- 172.21.15.90:0/917669702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a7567b84d0 0x55a756b94970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:40.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.024+0000 7fdf1878d740 1 --2- 172.21.15.90:0/917669702 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a75679ff60 0x55a756b94430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:40.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.024+0000 7fdf1878d740 1 -- 172.21.15.90:0/917669702 >> 172.21.15.90:0/917669702 conn(0x55a7567d54e0 msgr2=0x55a756b87530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:40.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.024+0000 7fdf1878d740 1 -- 172.21.15.90:0/917669702 shutdown_connections 2024-04-25T11:42:40.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.024+0000 7fdf1878d740 1 -- 172.21.15.90:0/917669702 wait complete. 2024-04-25T11:42:40.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.024+0000 7fdf1878d740 1 -- start start 2024-04-25T11:42:40.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.025+0000 7fdf1878d740 1 Processor -- start 2024-04-25T11:42:40.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.025+0000 7fdf1878d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a75679ff60 0x55a756b949b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:40.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.025+0000 7fdf1878d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567b84d0 0x55a756b8fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:40.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.025+0000 7fdf1878d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7567c06b0 0x55a756b90240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:40.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.025+0000 7fdf1878d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdf0c1b3200 con 0x55a7567b84d0 2024-04-25T11:42:40.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.025+0000 7fdf1878d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fdf0c1b2b30 con 0x55a7567c06b0 2024-04-25T11:42:40.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.025+0000 7fdf1878d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdf0c1afd90 con 0x55a75679ff60 2024-04-25T11:42:40.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.025+0000 7fdf1878d740 10 client.0 _ll_register_callbacks cb 0x7fdf0400f9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:40.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.025+0000 7fdf0a7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567b84d0 0x55a756b8fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:40.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.025+0000 7fdf0a7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567b84d0 0x55a756b8fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60622/0 (socket says 172.21.15.90:60622) 2024-04-25T11:42:40.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.025+0000 7fdf0a7fc640 1 -- 172.21.15.90:0/4198678747 learned_addr learned my addr 172.21.15.90:0/4198678747 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:40.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.025+0000 7fdf0affd640 1 --2- 172.21.15.90:0/4198678747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a75679ff60 0x55a756b949b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:40.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.025+0000 7fdf0b7fe640 1 --2- 172.21.15.90:0/4198678747 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7567c06b0 0x55a756b90240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:40.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.025+0000 7fdf0a7fc640 1 -- 172.21.15.90:0/4198678747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a75679ff60 msgr2=0x55a756b949b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:40.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.025+0000 7fdf0a7fc640 1 --2- 172.21.15.90:0/4198678747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a75679ff60 0x55a756b949b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:40.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.025+0000 7fdf0a7fc640 1 -- 172.21.15.90:0/4198678747 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7567c06b0 msgr2=0x55a756b90240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:40.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.025+0000 7fdf0a7fc640 1 --2- 172.21.15.90:0/4198678747 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7567c06b0 0x55a756b90240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:40.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.025+0000 7fdf0a7fc640 1 -- 172.21.15.90:0/4198678747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a756aa84b0 con 0x55a7567b84d0 2024-04-25T11:42:40.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.025+0000 7fdf0b7fe640 1 --2- 172.21.15.90:0/4198678747 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7567c06b0 0x55a756b90240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:42:40.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.025+0000 7fdf0a7fc640 1 --2- 172.21.15.90:0/4198678747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567b84d0 0x55a756b8fd00 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fdf00008010 tx=0x7fdf0000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9b45243d43b0d46 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:40.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.025+0000 7fdeeaffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:40.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.026+0000 7fdeeaffd640 1 -- 172.21.15.90:0/4198678747 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf00015040 con 0x55a7567b84d0 2024-04-25T11:42:40.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.026+0000 7fdeeaffd640 1 -- 172.21.15.90:0/4198678747 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7fdf0000ce40 con 0x55a7567b84d0 2024-04-25T11:42:40.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.026+0000 7fdf1878d740 1 -- 172.21.15.90:0/4198678747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf04033030 con 0x55a7567b84d0 2024-04-25T11:42:40.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.026+0000 7fdeeaffd640 1 -- 172.21.15.90:0/4198678747 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdf00031050 con 0x55a7567b84d0 2024-04-25T11:42:40.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.026+0000 7fdf1878d740 10 client.17982 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:40.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.026+0000 7fdf1878d740 1 -- 172.21.15.90:0/4198678747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a756c066b0 con 0x55a7567b84d0 2024-04-25T11:42:40.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.026+0000 7fdeeaffd640 1 -- 172.21.15.90:0/4198678747 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf04033030 con 0x55a7567b84d0 2024-04-25T11:42:40.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.026+0000 7fdeeaffd640 1 -- 172.21.15.90:0/4198678747 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fdf0001aa20 con 0x55a7567b84d0 2024-04-25T11:42:40.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.026+0000 7fdf1878d740 10 client.17982 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:40.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.026+0000 7fdf1878d740 10 client.17982 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:42:40.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.026+0000 7fdf1878d740 20 client.17982 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:40.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.026+0000 7fdf1878d740 1 --2- 172.21.15.90:0/4198678747 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55a7567e1390 0x55a756c08ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:40.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.026+0000 7fdf1878d740 4 client.17982 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:42:40.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.026+0000 7fdf1878d740 1 -- 172.21.15.90:0/4198678747 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a756aef800 con 0x55a7567e1390 2024-04-25T11:42:40.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.026+0000 7fdf0affd640 1 --2- 172.21.15.90:0/4198678747 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55a7567e1390 0x55a756c08ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:40.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.027+0000 7fdf0affd640 1 --2- 172.21.15.90:0/4198678747 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55a7567e1390 0x55a756c08ac0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25d3414e897cb64a server_cookie=3c8a218b2f18d639 in_seq=0 out_seq=0 2024-04-25T11:42:40.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.027+0000 7fdeeaffd640 10 client.17982 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:40.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.027+0000 7fdeeaffd640 1 -- 172.21.15.90:0/4198678747 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a756aef800 con 0x55a7567e1390 2024-04-25T11:42:40.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.027+0000 7fdeeaffd640 10 client.17982 handle_command_reply: tid=0 2024-04-25T11:42:40.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.027+0000 7fdf1878d740 1 client.17982 shutdown 2024-04-25T11:42:40.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.027+0000 7fdf1878d740 1 -- 172.21.15.90:0/4198678747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567b84d0 msgr2=0x55a756b8fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:40.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.027+0000 7fdf1878d740 1 --2- 172.21.15.90:0/4198678747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567b84d0 0x55a756b8fd00 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fdf00008010 tx=0x7fdf0000a5c0 comp rx=0 tx=0).stop 2024-04-25T11:42:40.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.027+0000 7fdf1878d740 1 -- 172.21.15.90:0/4198678747 shutdown_connections 2024-04-25T11:42:40.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.027+0000 7fdf1878d740 1 --2- 172.21.15.90:0/4198678747 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55a7567e1390 0x55a756c08ac0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:40.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.027+0000 7fdf1878d740 1 --2- 172.21.15.90:0/4198678747 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a7567c06b0 0x55a756b90240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:40.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.027+0000 7fdf1878d740 1 --2- 172.21.15.90:0/4198678747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7567b84d0 0x55a756b8fd00 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:40.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.027+0000 7fdeeaffd640 0 client.17982 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:40.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.027+0000 7fdf1878d740 1 --2- 172.21.15.90:0/4198678747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a75679ff60 0x55a756b949b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:40.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.027+0000 7fdf1878d740 1 -- 172.21.15.90:0/4198678747 >> 172.21.15.90:0/4198678747 conn(0x55a7567d54e0 msgr2=0x55a756b87530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:40.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.027+0000 7fdf1878d740 1 -- 172.21.15.90:0/4198678747 shutdown_connections 2024-04-25T11:42:40.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.027+0000 7fdf1878d740 1 -- 172.21.15.90:0/4198678747 wait complete. 2024-04-25T11:42:40.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.027+0000 7fdf1878d740 20 client.17982 trim_cache size 0 max 16384 2024-04-25T11:42:40.021 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:42:40.022 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T11:42:40.022 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-25T11:42:40.022 INFO:teuthology.orchestra.run.smithi090.stdout: "3f7b92e1-94cb-491d-ac35-f599fb1c4b6a": { 2024-04-25T11:42:40.022 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-25T11:42:40.022 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "3f7b92e1-94cb-491d-ac35-f599fb1c4b6a", 2024-04-25T11:42:40.022 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-25T11:42:40.022 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:42:40.022 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:42:40.022 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:42:40.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.028+0000 7fdf118e2640 1 -- 172.21.15.90:0/1573881474 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fdef40afc60 msgr2=0x7fdef40b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:40.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.028+0000 7fdf118e2640 1 --2- 172.21.15.90:0/1573881474 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fdef40afc60 0x7fdef40b2170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdf04005fd0 tx=0x7fdf04004a40 comp rx=0 tx=0).stop 2024-04-25T11:42:40.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.028+0000 7fdf118e2640 1 -- 172.21.15.90:0/1573881474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf0c1042d0 msgr2=0x7fdf0c19d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:40.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.028+0000 7fdf118e2640 1 --2- 172.21.15.90:0/1573881474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf0c1042d0 0x7fdf0c19d660 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fdefc031a90 tx=0x7fdefc004270 comp rx=0 tx=0).stop 2024-04-25T11:42:40.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.028+0000 7fdf118e2640 1 -- 172.21.15.90:0/1573881474 shutdown_connections 2024-04-25T11:42:40.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.028+0000 7fdf118e2640 1 --2- 172.21.15.90:0/1573881474 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fdef40afc60 0x7fdef40b2170 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:40.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.028+0000 7fdf118e2640 1 --2- 172.21.15.90:0/1573881474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf0c109fd0 0x7fdf0c1a86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:40.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.028+0000 7fdf118e2640 1 --2- 172.21.15.90:0/1573881474 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdf0c109b30 0x7fdf0c19dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:40.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.028+0000 7fdf118e2640 1 --2- 172.21.15.90:0/1573881474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf0c1042d0 0x7fdf0c19d660 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:40.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.028+0000 7fdf118e2640 1 -- 172.21.15.90:0/1573881474 >> 172.21.15.90:0/1573881474 conn(0x7fdf0c079280 msgr2=0x7fdf0c075070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:40.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.028+0000 7fdf118e2640 1 -- 172.21.15.90:0/1573881474 shutdown_connections 2024-04-25T11:42:40.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:42:40.028+0000 7fdf118e2640 1 -- 172.21.15.90:0/1573881474 wait complete. 2024-04-25T11:42:40.026 INFO:tasks.cephfs.filesystem:scrub status for tag:3f7b92e1-94cb-491d-ac35-f599fb1c4b6a - {'path': '/', 'tag': '3f7b92e1-94cb-491d-ac35-f599fb1c4b6a', 'options': 'recursive,force'} 2024-04-25T11:42:40.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:39 smithi138 ceph-mon[32328]: pgmap v232: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 13 KiB/s rd, 224 KiB/s wr, 13 op/s 2024-04-25T11:42:40.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:39 smithi156 ceph-mon[33045]: pgmap v232: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 13 KiB/s rd, 224 KiB/s wr, 13 op/s 2024-04-25T11:42:41.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:40 smithi090 ceph-mon[29575]: scrub queued for path: / 2024-04-25T11:42:41.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:40 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:42:41.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:40 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:42:41.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:40 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:42:41.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:40 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:42:41.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:40 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:42:41.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:40 smithi138 ceph-mon[32328]: scrub queued for path: / 2024-04-25T11:42:41.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:40 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:42:41.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:40 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:42:41.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:40 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:42:41.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:40 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:42:41.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:40 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:42:41.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:40 smithi156 ceph-mon[33045]: scrub queued for path: / 2024-04-25T11:42:41.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:40 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:42:41.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:40 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:42:41.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:40 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:42:41.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:40 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:42:41.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:40 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:42:42.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:41 smithi090 ceph-mon[29575]: pgmap v233: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 21 KiB/s rd, 147 KiB/s wr, 18 op/s 2024-04-25T11:42:42.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:41 smithi138 ceph-mon[32328]: pgmap v233: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 21 KiB/s rd, 147 KiB/s wr, 18 op/s 2024-04-25T11:42:42.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:41 smithi156 ceph-mon[33045]: pgmap v233: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 21 KiB/s rd, 147 KiB/s wr, 18 op/s 2024-04-25T11:42:44.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:43 smithi090 ceph-mon[29575]: pgmap v234: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T11:42:44.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:43 smithi138 ceph-mon[32328]: pgmap v234: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T11:42:44.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:43 smithi156 ceph-mon[33045]: pgmap v234: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T11:42:46.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:45 smithi090 ceph-mon[29575]: pgmap v235: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T11:42:46.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:45 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:42:46.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:45 smithi138 ceph-mon[32328]: pgmap v235: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T11:42:46.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:45 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:42:46.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:45 smithi156 ceph-mon[33045]: pgmap v235: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 12 op/s 2024-04-25T11:42:46.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:45 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:42:47.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:46 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:42:47.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:46 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:42:47.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:46 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:42:48.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:47 smithi090 ceph-mon[29575]: pgmap v236: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 17 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-04-25T11:42:48.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:47 smithi138 ceph-mon[32328]: pgmap v236: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 17 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-04-25T11:42:48.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:47 smithi156 ceph-mon[33045]: pgmap v236: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 17 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-04-25T11:42:50.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:49 smithi090 ceph-mon[29575]: pgmap v237: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 12 KiB/s rd, 8.3 KiB/s wr, 14 op/s 2024-04-25T11:42:50.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:49 smithi138 ceph-mon[32328]: pgmap v237: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 12 KiB/s rd, 8.3 KiB/s wr, 14 op/s 2024-04-25T11:42:50.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:49 smithi156 ceph-mon[33045]: pgmap v237: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 12 KiB/s rd, 8.3 KiB/s wr, 14 op/s 2024-04-25T11:42:51.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.773+0000 7fb582267640 1 -- 172.21.15.90:0/1699695705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb57c109bd0 msgr2=0x7fb57c101b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:51.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.773+0000 7fb582267640 1 --2- 172.21.15.90:0/1699695705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb57c109bd0 0x7fb57c101b40 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fb56c009900 tx=0x7fb56c02a410 comp rx=0 tx=0).stop 2024-04-25T11:42:51.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.773+0000 7fb582267640 1 -- 172.21.15.90:0/1699695705 shutdown_connections 2024-04-25T11:42:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.773+0000 7fb582267640 1 --2- 172.21.15.90:0/1699695705 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb57c10e580 0x7fb57c1109a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.774+0000 7fb582267640 1 --2- 172.21.15.90:0/1699695705 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb57c102080 0x7fb57c102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.774+0000 7fb582267640 1 --2- 172.21.15.90:0/1699695705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb57c109bd0 0x7fb57c101b40 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.774+0000 7fb582267640 1 -- 172.21.15.90:0/1699695705 >> 172.21.15.90:0/1699695705 conn(0x7fb57c0fd8f0 msgr2=0x7fb57c0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.774+0000 7fb582267640 1 -- 172.21.15.90:0/1699695705 shutdown_connections 2024-04-25T11:42:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.774+0000 7fb582267640 1 -- 172.21.15.90:0/1699695705 wait complete. 2024-04-25T11:42:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.774+0000 7fb582267640 1 Processor -- start 2024-04-25T11:42:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.774+0000 7fb582267640 1 -- start start 2024-04-25T11:42:51.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.775+0000 7fb582267640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb57c102080 0x7fb57c19a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:51.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.775+0000 7fb582267640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb57c109bd0 0x7fb57c19ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:51.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.775+0000 7fb582267640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb57c10e580 0x7fb57c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.775+0000 7fb582267640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb57c113d40 con 0x7fb57c102080 2024-04-25T11:42:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.775+0000 7fb582267640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb57c113bc0 con 0x7fb57c10e580 2024-04-25T11:42:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.775+0000 7fb582267640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb57c113ec0 con 0x7fb57c109bd0 2024-04-25T11:42:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.775+0000 7fb57bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb57c102080 0x7fb57c19a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.775+0000 7fb57bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb57c102080 0x7fb57c19a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41180/0 (socket says 172.21.15.90:41180) 2024-04-25T11:42:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.775+0000 7fb57bfff640 1 -- 172.21.15.90:0/2789212981 learned_addr learned my addr 172.21.15.90:0/2789212981 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.775+0000 7fb57b7fe640 1 --2- 172.21.15.90:0/2789212981 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb57c109bd0 0x7fb57c19ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.775+0000 7fb580804640 1 --2- 172.21.15.90:0/2789212981 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb57c10e580 0x7fb57c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.775+0000 7fb57bfff640 1 -- 172.21.15.90:0/2789212981 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb57c109bd0 msgr2=0x7fb57c19ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.775+0000 7fb57bfff640 1 --2- 172.21.15.90:0/2789212981 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb57c109bd0 0x7fb57c19ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.775+0000 7fb57bfff640 1 -- 172.21.15.90:0/2789212981 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb57c10e580 msgr2=0x7fb57c19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.775+0000 7fb57bfff640 1 --2- 172.21.15.90:0/2789212981 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb57c10e580 0x7fb57c19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.775+0000 7fb57bfff640 1 -- 172.21.15.90:0/2789212981 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb57c19f620 con 0x7fb57c102080 2024-04-25T11:42:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.775+0000 7fb580804640 1 --2- 172.21.15.90:0/2789212981 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb57c10e580 0x7fb57c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:42:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.775+0000 7fb57bfff640 1 --2- 172.21.15.90:0/2789212981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb57c102080 0x7fb57c19a7c0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fb56c02a930 tx=0x7fb56c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.776+0000 7fb5797fa640 1 -- 172.21.15.90:0/2789212981 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56c03f030 con 0x7fb57c102080 2024-04-25T11:42:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.776+0000 7fb582267640 1 -- 172.21.15.90:0/2789212981 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb57c19f8b0 con 0x7fb57c102080 2024-04-25T11:42:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.776+0000 7fb5797fa640 1 -- 172.21.15.90:0/2789212981 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb56c003aa0 con 0x7fb57c102080 2024-04-25T11:42:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.776+0000 7fb5797fa640 1 -- 172.21.15.90:0/2789212981 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56c035070 con 0x7fb57c102080 2024-04-25T11:42:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.776+0000 7fb582267640 1 -- 172.21.15.90:0/2789212981 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb57c19fdf0 con 0x7fb57c102080 2024-04-25T11:42:51.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.778+0000 7fb5797fa640 1 -- 172.21.15.90:0/2789212981 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fb56c042030 con 0x7fb57c102080 2024-04-25T11:42:51.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.778+0000 7fb5891fc740 1 Processor -- start 2024-04-25T11:42:51.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.778+0000 7fb5891fc740 1 -- start start 2024-04-25T11:42:51.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.779+0000 7fb5891fc740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bbea8611b0 0x55bbeac00960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:51.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.779+0000 7fb5891fc740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bbea8382f0 0x55bbeac00ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:51.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.779+0000 7fb5891fc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8404d0 0x55bbeac013e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:51.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.779+0000 7fb5891fc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55bbeab67460 con 0x55bbea8404d0 2024-04-25T11:42:51.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.779+0000 7fb5891fc740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55bbeab21590 con 0x55bbea8382f0 2024-04-25T11:42:51.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.779+0000 7fb5891fc740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55bbeab63420 con 0x55bbea8611b0 2024-04-25T11:42:51.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.779+0000 7fb57b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8404d0 0x55bbeac013e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:51.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.779+0000 7fb57b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8404d0 0x55bbeac013e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41194/0 (socket says 172.21.15.90:41194) 2024-04-25T11:42:51.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.779+0000 7fb57b7fe640 1 -- 172.21.15.90:0/1441398577 learned_addr learned my addr 172.21.15.90:0/1441398577 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:51.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.779+0000 7fb580804640 1 --2- 172.21.15.90:0/1441398577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bbea8611b0 0x55bbeac00960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:51.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.779+0000 7fb57bfff640 1 --2- 172.21.15.90:0/1441398577 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bbea8382f0 0x55bbeac00ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:51.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.779+0000 7fb5797fa640 1 --2- 172.21.15.90:0/2789212981 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb5640afc50 0x7fb5640b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:51.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.779+0000 7fb57b7fe640 1 -- 172.21.15.90:0/1441398577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bbea8611b0 msgr2=0x55bbeac00960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:51.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.779+0000 7fb57b7fe640 1 --2- 172.21.15.90:0/1441398577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bbea8611b0 0x55bbeac00960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.779+0000 7fb57b7fe640 1 -- 172.21.15.90:0/1441398577 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bbea8382f0 msgr2=0x55bbeac00ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:51.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.779+0000 7fb57b7fe640 1 --2- 172.21.15.90:0/1441398577 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bbea8382f0 0x55bbeac00ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.779+0000 7fb57b7fe640 1 -- 172.21.15.90:0/1441398577 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbeab2e0d0 con 0x55bbea8404d0 2024-04-25T11:42:51.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.779+0000 7fb57bfff640 1 --2- 172.21.15.90:0/1441398577 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bbea8382f0 0x55bbeac00ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:42:51.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.779+0000 7fb580804640 1 --2- 172.21.15.90:0/2789212981 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb5640afc50 0x7fb5640b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:51.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.779+0000 7fb5797fa640 1 -- 172.21.15.90:0/2789212981 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7fb56c0ff5d0 con 0x7fb57c102080 2024-04-25T11:42:51.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.779+0000 7fb57b7fe640 1 --2- 172.21.15.90:0/1441398577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8404d0 0x55bbeac013e0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fb57000b6c0 tx=0x7fb57000bb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee6391455c225541 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:51.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.780+0000 7fb5627fc640 1 -- 172.21.15.90:0/1441398577 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb570003bd0 con 0x55bbea8404d0 2024-04-25T11:42:51.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.780+0000 7fb5627fc640 1 -- 172.21.15.90:0/1441398577 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb570015070 con 0x55bbea8404d0 2024-04-25T11:42:51.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.780+0000 7fb5627fc640 1 -- 172.21.15.90:0/1441398577 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb570004d10 con 0x55bbea8404d0 2024-04-25T11:42:51.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.780+0000 7fb5891fc740 1 -- 172.21.15.90:0/1441398577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8404d0 msgr2=0x55bbeac013e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:51.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.780+0000 7fb5891fc740 1 --2- 172.21.15.90:0/1441398577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8404d0 0x55bbeac013e0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fb57000b6c0 tx=0x7fb57000bb80 comp rx=0 tx=0).stop 2024-04-25T11:42:51.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.780+0000 7fb580804640 1 --2- 172.21.15.90:0/2789212981 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb5640afc50 0x7fb5640b2160 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x55bbeabfe1c0 tx=0x7fb574004a40 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:51.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.780+0000 7fb5891fc740 1 -- 172.21.15.90:0/1441398577 shutdown_connections 2024-04-25T11:42:51.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.780+0000 7fb5891fc740 1 --2- 172.21.15.90:0/1441398577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8404d0 0x55bbeac013e0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.780+0000 7fb5891fc740 1 --2- 172.21.15.90:0/1441398577 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bbea8382f0 0x55bbeac00ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.780+0000 7fb5891fc740 1 --2- 172.21.15.90:0/1441398577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bbea8611b0 0x55bbeac00960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.780+0000 7fb5891fc740 1 -- 172.21.15.90:0/1441398577 >> 172.21.15.90:0/1441398577 conn(0x55bbea855300 msgr2=0x55bbeabfd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:51.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.781+0000 7fb5891fc740 1 -- 172.21.15.90:0/1441398577 shutdown_connections 2024-04-25T11:42:51.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.781+0000 7fb5891fc740 1 -- 172.21.15.90:0/1441398577 wait complete. 2024-04-25T11:42:51.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.781+0000 7fb5891fc740 1 -- start start 2024-04-25T11:42:51.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.781+0000 7fb5891fc740 1 Processor -- start 2024-04-25T11:42:51.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.781+0000 7fb5891fc740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bbea8382f0 0x55bbeac0f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:51.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.781+0000 7fb5891fc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8404d0 0x55bbeac0fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:51.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.781+0000 7fb5891fc740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bbea8611b0 0x55bbeac154b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:51.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.781+0000 7fb5891fc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55bbeac12030 con 0x55bbea8404d0 2024-04-25T11:42:51.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.781+0000 7fb5891fc740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55bbeaaeeb60 con 0x55bbea8382f0 2024-04-25T11:42:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.781+0000 7fb5891fc740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55bbeac121b0 con 0x55bbea8611b0 2024-04-25T11:42:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.782+0000 7fb5891fc740 10 client.0 _ll_register_callbacks cb 0x55bbeab132b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.782+0000 7fb57b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8404d0 0x55bbeac0fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.782+0000 7fb57b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8404d0 0x55bbeac0fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41196/0 (socket says 172.21.15.90:41196) 2024-04-25T11:42:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.782+0000 7fb57b7fe640 1 -- 172.21.15.90:0/3547461480 learned_addr learned my addr 172.21.15.90:0/3547461480 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.782+0000 7fb580804640 1 --2- 172.21.15.90:0/3547461480 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bbea8611b0 0x55bbeac154b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.782+0000 7fb57bfff640 1 --2- 172.21.15.90:0/3547461480 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bbea8382f0 0x55bbeac0f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.782+0000 7fb57b7fe640 1 -- 172.21.15.90:0/3547461480 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bbea8611b0 msgr2=0x55bbeac154b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.782+0000 7fb57b7fe640 1 --2- 172.21.15.90:0/3547461480 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bbea8611b0 0x55bbeac154b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.782+0000 7fb57b7fe640 1 -- 172.21.15.90:0/3547461480 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bbea8382f0 msgr2=0x55bbeac0f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.782+0000 7fb57b7fe640 1 --2- 172.21.15.90:0/3547461480 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bbea8382f0 0x55bbeac0f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.782+0000 7fb57b7fe640 1 -- 172.21.15.90:0/3547461480 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbea879210 con 0x55bbea8404d0 2024-04-25T11:42:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.782+0000 7fb57bfff640 1 --2- 172.21.15.90:0/3547461480 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bbea8382f0 0x55bbeac0f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:42:51.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.782+0000 7fb57b7fe640 1 --2- 172.21.15.90:0/3547461480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8404d0 0x55bbeac0fa80 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fb570005cd0 tx=0x7fb570003e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12d241c708702d7b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.782+0000 7fb55b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.782+0000 7fb55b7fe640 1 -- 172.21.15.90:0/3547461480 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57000bdd0 con 0x55bbea8404d0 2024-04-25T11:42:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.782+0000 7fb55b7fe640 1 -- 172.21.15.90:0/3547461480 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7fb570011650 con 0x55bbea8404d0 2024-04-25T11:42:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.782+0000 7fb5891fc740 1 -- 172.21.15.90:0/3547461480 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb570015070 con 0x55bbea8404d0 2024-04-25T11:42:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.782+0000 7fb55b7fe640 1 -- 172.21.15.90:0/3547461480 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb570011850 con 0x55bbea8404d0 2024-04-25T11:42:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.782+0000 7fb5891fc740 10 client.17994 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.782+0000 7fb5891fc740 1 -- 172.21.15.90:0/3547461480 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bbea945620 con 0x55bbea8404d0 2024-04-25T11:42:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.783+0000 7fb55b7fe640 1 -- 172.21.15.90:0/3547461480 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb570015070 con 0x55bbea8404d0 2024-04-25T11:42:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.783+0000 7fb55b7fe640 1 -- 172.21.15.90:0/3547461480 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fb570025a40 con 0x55bbea8404d0 2024-04-25T11:42:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.783+0000 7fb5891fc740 10 client.17994 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.783+0000 7fb5891fc740 10 client.17994 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:42:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.783+0000 7fb5891fc740 20 client.17994 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.783+0000 7fb5891fc740 1 --2- 172.21.15.90:0/3547461480 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55bbea81fd80 0x55bbeac219b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:51.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.783+0000 7fb5891fc740 4 client.17994 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:42:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.783+0000 7fb5891fc740 1 -- 172.21.15.90:0/3547461480 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bbeab75070 con 0x55bbea81fd80 2024-04-25T11:42:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.783+0000 7fb57bfff640 1 --2- 172.21.15.90:0/3547461480 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55bbea81fd80 0x55bbeac219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.784+0000 7fb57bfff640 1 --2- 172.21.15.90:0/3547461480 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55bbea81fd80 0x55bbeac219b0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62a5a1604576732e server_cookie=b8cece78bf7264bc in_seq=0 out_seq=0 2024-04-25T11:42:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.784+0000 7fb55b7fe640 10 client.17994 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.784+0000 7fb55b7fe640 1 -- 172.21.15.90:0/3547461480 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55bbeab75070 con 0x55bbea81fd80 2024-04-25T11:42:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.784+0000 7fb55b7fe640 10 client.17994 handle_command_reply: tid=0 2024-04-25T11:42:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.784+0000 7fb5891fc740 1 client.17994 shutdown 2024-04-25T11:42:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.785+0000 7fb5891fc740 1 -- 172.21.15.90:0/3547461480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8404d0 msgr2=0x55bbeac0fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.785+0000 7fb5891fc740 1 --2- 172.21.15.90:0/3547461480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8404d0 0x55bbeac0fa80 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fb570005cd0 tx=0x7fb570003e00 comp rx=0 tx=0).stop 2024-04-25T11:42:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.785+0000 7fb5891fc740 1 -- 172.21.15.90:0/3547461480 shutdown_connections 2024-04-25T11:42:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.785+0000 7fb5891fc740 1 --2- 172.21.15.90:0/3547461480 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55bbea81fd80 0x55bbeac219b0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.785+0000 7fb5891fc740 1 --2- 172.21.15.90:0/3547461480 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bbea8611b0 0x55bbeac154b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.785+0000 7fb5891fc740 1 --2- 172.21.15.90:0/3547461480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8404d0 0x55bbeac0fa80 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.779 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.785+0000 7fb55b7fe640 0 client.17994 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.785+0000 7fb5891fc740 1 --2- 172.21.15.90:0/3547461480 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bbea8382f0 0x55bbeac0f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.785+0000 7fb5891fc740 1 -- 172.21.15.90:0/3547461480 >> 172.21.15.90:0/3547461480 conn(0x55bbea855300 msgr2=0x55bbeac02700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.785+0000 7fb5891fc740 1 -- 172.21.15.90:0/3547461480 shutdown_connections 2024-04-25T11:42:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.785+0000 7fb5891fc740 1 -- 172.21.15.90:0/3547461480 wait complete. 2024-04-25T11:42:51.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.785+0000 7fb5891fc740 20 client.17994 trim_cache size 0 max 16384 2024-04-25T11:42:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.800+0000 7fb5891fc740 1 Processor -- start 2024-04-25T11:42:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.800+0000 7fb5891fc740 1 -- start start 2024-04-25T11:42:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.800+0000 7fb5891fc740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bbea8382f0 0x55bbeac15420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.800+0000 7fb5891fc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8404d0 0x55bbeac012b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.800+0000 7fb5891fc740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bbea8611b0 0x55bbeac08d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.800+0000 7fb5891fc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb57c1adab0 con 0x55bbea8404d0 2024-04-25T11:42:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.800+0000 7fb5891fc740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb57c1af5f0 con 0x55bbea8382f0 2024-04-25T11:42:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.800+0000 7fb5891fc740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb57c1ae180 con 0x55bbea8611b0 2024-04-25T11:42:51.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.800+0000 7fb57b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8404d0 0x55bbeac012b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:51.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.800+0000 7fb57b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8404d0 0x55bbeac012b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41200/0 (socket says 172.21.15.90:41200) 2024-04-25T11:42:51.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.800+0000 7fb57b7fe640 1 -- 172.21.15.90:0/2750136637 learned_addr learned my addr 172.21.15.90:0/2750136637 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:51.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.800+0000 7fb57b7fe640 1 -- 172.21.15.90:0/2750136637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bbea8611b0 msgr2=0x55bbeac08d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:51.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.800+0000 7fb57b7fe640 1 --2- 172.21.15.90:0/2750136637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bbea8611b0 0x55bbeac08d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.800+0000 7fb57b7fe640 1 -- 172.21.15.90:0/2750136637 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bbea8382f0 msgr2=0x55bbeac15420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:51.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.800+0000 7fb57bfff640 1 --2- 172.21.15.90:0/2750136637 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bbea8382f0 0x55bbeac15420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:51.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.800+0000 7fb57b7fe640 1 --2- 172.21.15.90:0/2750136637 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bbea8382f0 0x55bbeac15420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.800+0000 7fb57b7fe640 1 -- 172.21.15.90:0/2750136637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbeac21e10 con 0x55bbea8404d0 2024-04-25T11:42:51.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.800+0000 7fb57b7fe640 1 --2- 172.21.15.90:0/2750136637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8404d0 0x55bbeac012b0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fb570025de0 tx=0x7fb57000bb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=242a7e2e32daabb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:51.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.801+0000 7fb561ffb640 1 -- 172.21.15.90:0/2750136637 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb570003e70 con 0x55bbea8404d0 2024-04-25T11:42:51.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.801+0000 7fb561ffb640 1 -- 172.21.15.90:0/2750136637 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb570015040 con 0x55bbea8404d0 2024-04-25T11:42:51.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.801+0000 7fb561ffb640 1 -- 172.21.15.90:0/2750136637 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5700198c0 con 0x55bbea8404d0 2024-04-25T11:42:51.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.801+0000 7fb5891fc740 1 -- 172.21.15.90:0/2750136637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8404d0 msgr2=0x55bbeac012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.801+0000 7fb5891fc740 1 --2- 172.21.15.90:0/2750136637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8404d0 0x55bbeac012b0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fb570025de0 tx=0x7fb57000bb80 comp rx=0 tx=0).stop 2024-04-25T11:42:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.801+0000 7fb5891fc740 1 -- 172.21.15.90:0/2750136637 shutdown_connections 2024-04-25T11:42:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.801+0000 7fb5891fc740 1 --2- 172.21.15.90:0/2750136637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bbea8611b0 0x55bbeac08d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.801+0000 7fb5891fc740 1 --2- 172.21.15.90:0/2750136637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8404d0 0x55bbeac012b0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.801+0000 7fb5891fc740 1 --2- 172.21.15.90:0/2750136637 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bbea8382f0 0x55bbeac15420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.801+0000 7fb5891fc740 1 -- 172.21.15.90:0/2750136637 >> 172.21.15.90:0/2750136637 conn(0x55bbea855300 msgr2=0x55bbeac10a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.801+0000 7fb5891fc740 1 -- 172.21.15.90:0/2750136637 shutdown_connections 2024-04-25T11:42:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.801+0000 7fb5891fc740 1 -- 172.21.15.90:0/2750136637 wait complete. 2024-04-25T11:42:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.802+0000 7fb5891fc740 1 -- start start 2024-04-25T11:42:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.802+0000 7fb5891fc740 1 Processor -- start 2024-04-25T11:42:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.802+0000 7fb5891fc740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bbea8382f0 0x55bbeac1c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.802+0000 7fb5891fc740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bbea8404d0 0x55bbeac176b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.802+0000 7fb5891fc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8611b0 0x55bbeac17bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.802+0000 7fb5891fc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb57c1aef20 con 0x55bbea8611b0 2024-04-25T11:42:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.802+0000 7fb5891fc740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb57c1ae850 con 0x55bbea8382f0 2024-04-25T11:42:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.802+0000 7fb5891fc740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb57c1abab0 con 0x55bbea8404d0 2024-04-25T11:42:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.802+0000 7fb5891fc740 10 client.0 _ll_register_callbacks cb 0x7fb5740084d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:42:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.802+0000 7fb580804640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8611b0 0x55bbeac17bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.802+0000 7fb580804640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8611b0 0x55bbeac17bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41212/0 (socket says 172.21.15.90:41212) 2024-04-25T11:42:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.802+0000 7fb580804640 1 -- 172.21.15.90:0/1008060388 learned_addr learned my addr 172.21.15.90:0/1008060388 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:42:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.803+0000 7fb57b7fe640 1 --2- 172.21.15.90:0/1008060388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bbea8404d0 0x55bbeac176b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.803+0000 7fb57bfff640 1 --2- 172.21.15.90:0/1008060388 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bbea8382f0 0x55bbeac1c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.803+0000 7fb580804640 1 -- 172.21.15.90:0/1008060388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bbea8404d0 msgr2=0x55bbeac176b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.803+0000 7fb580804640 1 --2- 172.21.15.90:0/1008060388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bbea8404d0 0x55bbeac176b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.803+0000 7fb580804640 1 -- 172.21.15.90:0/1008060388 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bbea8382f0 msgr2=0x55bbeac1c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.803+0000 7fb580804640 1 --2- 172.21.15.90:0/1008060388 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bbea8382f0 0x55bbeac1c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.803+0000 7fb580804640 1 -- 172.21.15.90:0/1008060388 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbeac18330 con 0x55bbea8611b0 2024-04-25T11:42:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.803+0000 7fb57b7fe640 1 --2- 172.21.15.90:0/1008060388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bbea8404d0 0x55bbeac176b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:42:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.803+0000 7fb580804640 1 --2- 172.21.15.90:0/1008060388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8611b0 0x55bbeac17bf0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fb57401ce60 tx=0x7fb574004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b0bad97eb36abfc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.803+0000 7fb55b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:42:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.803+0000 7fb55b7fe640 1 -- 172.21.15.90:0/1008060388 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57401c220 con 0x55bbea8611b0 2024-04-25T11:42:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.803+0000 7fb55b7fe640 1 -- 172.21.15.90:0/1008060388 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7fb574019620 con 0x55bbea8611b0 2024-04-25T11:42:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.803+0000 7fb5891fc740 1 -- 172.21.15.90:0/1008060388 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb570015040 con 0x55bbea8611b0 2024-04-25T11:42:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.803+0000 7fb55b7fe640 1 -- 172.21.15.90:0/1008060388 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb574019820 con 0x55bbea8611b0 2024-04-25T11:42:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.803+0000 7fb5891fc740 10 client.18000 fetch_fsmap learned FSMap version 28 2024-04-25T11:42:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.803+0000 7fb5891fc740 1 -- 172.21.15.90:0/1008060388 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bbeac89b10 con 0x55bbea8611b0 2024-04-25T11:42:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.803+0000 7fb55b7fe640 1 -- 172.21.15.90:0/1008060388 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb570015040 con 0x55bbea8611b0 2024-04-25T11:42:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.804+0000 7fb55b7fe640 1 -- 172.21.15.90:0/1008060388 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fb5740269a0 con 0x55bbea8611b0 2024-04-25T11:42:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.804+0000 7fb5891fc740 10 client.18000 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:42:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.804+0000 7fb5891fc740 10 client.18000 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:42:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.804+0000 7fb5891fc740 20 client.18000 populate_metadata read hostname 'smithi090' 2024-04-25T11:42:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.804+0000 7fb5891fc740 1 --2- 172.21.15.90:0/1008060388 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55bbea7f51c0 0x55bbeac8c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.804+0000 7fb5891fc740 4 client.18000 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:42:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.804+0000 7fb5891fc740 1 -- 172.21.15.90:0/1008060388 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bbeab6eee0 con 0x55bbea7f51c0 2024-04-25T11:42:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.804+0000 7fb57bfff640 1 --2- 172.21.15.90:0/1008060388 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55bbea7f51c0 0x55bbeac8c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.804+0000 7fb57bfff640 1 --2- 172.21.15.90:0/1008060388 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55bbea7f51c0 0x55bbeac8c380 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ec741899c235396 server_cookie=a10850094d5e6ff8 in_seq=0 out_seq=0 2024-04-25T11:42:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.804+0000 7fb55b7fe640 10 client.18000 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:51.802 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.814+0000 7fb55b7fe640 1 -- 172.21.15.90:0/1008060388 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bbeab6eee0 con 0x55bbea7f51c0 2024-04-25T11:42:51.802 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.814+0000 7fb55b7fe640 10 client.18000 handle_command_reply: tid=0 2024-04-25T11:42:51.802 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.814+0000 7fb5891fc740 1 client.18000 shutdown 2024-04-25T11:42:51.802 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.814+0000 7fb5891fc740 1 -- 172.21.15.90:0/1008060388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8611b0 msgr2=0x55bbeac17bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:51.802 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.814+0000 7fb5891fc740 1 --2- 172.21.15.90:0/1008060388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8611b0 0x55bbeac17bf0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fb57401ce60 tx=0x7fb574004ab0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.802 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.814+0000 7fb5891fc740 1 -- 172.21.15.90:0/1008060388 shutdown_connections 2024-04-25T11:42:51.802 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.814+0000 7fb5891fc740 1 --2- 172.21.15.90:0/1008060388 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55bbea7f51c0 0x55bbeac8c380 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.802 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.814+0000 7fb5891fc740 1 --2- 172.21.15.90:0/1008060388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bbea8611b0 0x55bbeac17bf0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.802 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.814+0000 7fb5891fc740 1 --2- 172.21.15.90:0/1008060388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bbea8404d0 0x55bbeac176b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.803 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.814+0000 7fb55b7fe640 0 client.18000 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:42:51.803 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.814+0000 7fb5891fc740 1 --2- 172.21.15.90:0/1008060388 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55bbea8382f0 0x55bbeac1c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.803 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.815+0000 7fb5891fc740 1 -- 172.21.15.90:0/1008060388 >> 172.21.15.90:0/1008060388 conn(0x55bbea855300 msgr2=0x55bbeac01010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:51.803 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.815+0000 7fb5891fc740 1 -- 172.21.15.90:0/1008060388 shutdown_connections 2024-04-25T11:42:51.803 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.815+0000 7fb5891fc740 1 -- 172.21.15.90:0/1008060388 wait complete. 2024-04-25T11:42:51.803 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.815+0000 7fb5891fc740 20 client.18000 trim_cache size 0 max 16384 2024-04-25T11:42:51.803 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:42:51.803 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:42:51.803 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:42:51.803 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:42:51.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.815+0000 7fb582267640 1 -- 172.21.15.90:0/2789212981 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb5640afc50 msgr2=0x7fb5640b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:51.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.815+0000 7fb582267640 1 --2- 172.21.15.90:0/2789212981 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb5640afc50 0x7fb5640b2160 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x55bbeabfe1c0 tx=0x7fb574004a40 comp rx=0 tx=0).stop 2024-04-25T11:42:51.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.815+0000 7fb582267640 1 -- 172.21.15.90:0/2789212981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb57c102080 msgr2=0x7fb57c19a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:51.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.815+0000 7fb582267640 1 --2- 172.21.15.90:0/2789212981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb57c102080 0x7fb57c19a7c0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fb56c02a930 tx=0x7fb56c004270 comp rx=0 tx=0).stop 2024-04-25T11:42:51.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.815+0000 7fb582267640 1 -- 172.21.15.90:0/2789212981 shutdown_connections 2024-04-25T11:42:51.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.815+0000 7fb582267640 1 --2- 172.21.15.90:0/2789212981 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb5640afc50 0x7fb5640b2160 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.815+0000 7fb582267640 1 --2- 172.21.15.90:0/2789212981 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb57c10e580 0x7fb57c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.815+0000 7fb582267640 1 --2- 172.21.15.90:0/2789212981 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb57c109bd0 0x7fb57c19ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.815+0000 7fb582267640 1 --2- 172.21.15.90:0/2789212981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb57c102080 0x7fb57c19a7c0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:51.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.815+0000 7fb582267640 1 -- 172.21.15.90:0/2789212981 >> 172.21.15.90:0/2789212981 conn(0x7fb57c0fd8f0 msgr2=0x7fb57c0fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:51.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.815+0000 7fb582267640 1 -- 172.21.15.90:0/2789212981 shutdown_connections 2024-04-25T11:42:51.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:42:51.815+0000 7fb582267640 1 -- 172.21.15.90:0/2789212981 wait complete. 2024-04-25T11:42:52.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:51 smithi090 ceph-mon[29575]: pgmap v238: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 15 KiB/s rd, 8.3 KiB/s wr, 18 op/s 2024-04-25T11:42:52.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:51 smithi138 ceph-mon[32328]: pgmap v238: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 15 KiB/s rd, 8.3 KiB/s wr, 18 op/s 2024-04-25T11:42:52.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:51 smithi156 ceph-mon[33045]: pgmap v238: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 15 KiB/s rd, 8.3 KiB/s wr, 18 op/s 2024-04-25T11:42:54.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:53 smithi090 ceph-mon[29575]: pgmap v239: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 7.2 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-25T11:42:54.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:53 smithi138 ceph-mon[32328]: pgmap v239: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 7.2 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-25T11:42:54.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:53 smithi156 ceph-mon[33045]: pgmap v239: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 7.2 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-25T11:42:56.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:55 smithi090 ceph-mon[29575]: pgmap v240: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.5 MiB/s wr, 28 op/s 2024-04-25T11:42:56.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:55 smithi138 ceph-mon[32328]: pgmap v240: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.5 MiB/s wr, 28 op/s 2024-04-25T11:42:56.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:55 smithi156 ceph-mon[33045]: pgmap v240: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 889 GiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.5 MiB/s wr, 28 op/s 2024-04-25T11:42:58.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:57 smithi090 ceph-mon[29575]: pgmap v241: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 888 GiB / 1.0 TiB avail; 11 KiB/s rd, 14 MiB/s wr, 182 op/s 2024-04-25T11:42:58.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:57 smithi138 ceph-mon[32328]: pgmap v241: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 888 GiB / 1.0 TiB avail; 11 KiB/s rd, 14 MiB/s wr, 182 op/s 2024-04-25T11:42:58.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:57 smithi156 ceph-mon[33045]: pgmap v241: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 888 GiB / 1.0 TiB avail; 11 KiB/s rd, 14 MiB/s wr, 182 op/s 2024-04-25T11:43:00.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:42:59 smithi090 ceph-mon[29575]: pgmap v242: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 888 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 18 MiB/s wr, 226 op/s 2024-04-25T11:43:00.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:42:59 smithi138 ceph-mon[32328]: pgmap v242: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 888 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 18 MiB/s wr, 226 op/s 2024-04-25T11:43:00.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:42:59 smithi156 ceph-mon[33045]: pgmap v242: 97 pgs: 97 active+clean; 60 GiB data, 184 GiB used, 888 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 18 MiB/s wr, 226 op/s 2024-04-25T11:43:02.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:01 smithi090 ceph-mon[29575]: pgmap v243: 97 pgs: 97 active+clean; 61 GiB data, 185 GiB used, 888 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 22 MiB/s wr, 282 op/s 2024-04-25T11:43:02.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:01 smithi138 ceph-mon[32328]: pgmap v243: 97 pgs: 97 active+clean; 61 GiB data, 185 GiB used, 888 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 22 MiB/s wr, 282 op/s 2024-04-25T11:43:02.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:01 smithi156 ceph-mon[33045]: pgmap v243: 97 pgs: 97 active+clean; 61 GiB data, 185 GiB used, 888 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 22 MiB/s wr, 282 op/s 2024-04-25T11:43:04.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:03 smithi090 ceph-mon[29575]: pgmap v244: 97 pgs: 97 active+clean; 61 GiB data, 185 GiB used, 888 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 24 MiB/s wr, 292 op/s 2024-04-25T11:43:04.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:03 smithi138 ceph-mon[32328]: pgmap v244: 97 pgs: 97 active+clean; 61 GiB data, 185 GiB used, 888 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 24 MiB/s wr, 292 op/s 2024-04-25T11:43:04.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:03 smithi156 ceph-mon[33045]: pgmap v244: 97 pgs: 97 active+clean; 61 GiB data, 185 GiB used, 888 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 24 MiB/s wr, 292 op/s 2024-04-25T11:43:04.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.922+0000 7fad7af34640 1 -- 172.21.15.90:0/89803082 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad74102090 msgr2=0x7fad741024a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:04.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.922+0000 7fad7af34640 1 --2- 172.21.15.90:0/89803082 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad74102090 0x7fad741024a0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fad60009900 tx=0x7fad6002a410 comp rx=0 tx=0).stop 2024-04-25T11:43:04.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.923+0000 7fad7af34640 1 -- 172.21.15.90:0/89803082 shutdown_connections 2024-04-25T11:43:04.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.923+0000 7fad7af34640 1 --2- 172.21.15.90:0/89803082 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad7410e590 0x7fad741109b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.923+0000 7fad7af34640 1 --2- 172.21.15.90:0/89803082 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad74102090 0x7fad741024a0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.923+0000 7fad7af34640 1 --2- 172.21.15.90:0/89803082 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fad74109be0 0x7fad74101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.923+0000 7fad7af34640 1 -- 172.21.15.90:0/89803082 >> 172.21.15.90:0/89803082 conn(0x7fad740fd8a0 msgr2=0x7fad740ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:04.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.923+0000 7fad7af34640 1 -- 172.21.15.90:0/89803082 shutdown_connections 2024-04-25T11:43:04.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.923+0000 7fad7af34640 1 -- 172.21.15.90:0/89803082 wait complete. 2024-04-25T11:43:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.923+0000 7fad7af34640 1 Processor -- start 2024-04-25T11:43:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.923+0000 7fad7af34640 1 -- start start 2024-04-25T11:43:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.923+0000 7fad7af34640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad74102090 0x7fad74102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.923+0000 7fad7af34640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fad74109be0 0x7fad74103270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.924+0000 7fad7af34640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad7410e590 0x7fad741037b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.924+0000 7fad7af34640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fad74113d50 con 0x7fad7410e590 2024-04-25T11:43:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.924+0000 7fad7af34640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fad74113bd0 con 0x7fad74109be0 2024-04-25T11:43:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.924+0000 7fad7af34640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fad74113ed0 con 0x7fad74102090 2024-04-25T11:43:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.924+0000 7fad794d1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad7410e590 0x7fad741037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.924+0000 7fad794d1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad7410e590 0x7fad741037b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52536/0 (socket says 172.21.15.90:52536) 2024-04-25T11:43:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.924+0000 7fad794d1640 1 -- 172.21.15.90:0/1998642381 learned_addr learned my addr 172.21.15.90:0/1998642381 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.924+0000 7fad78cd0640 1 --2- 172.21.15.90:0/1998642381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad74102090 0x7fad74102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.924+0000 7fad73fff640 1 --2- 172.21.15.90:0/1998642381 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fad74109be0 0x7fad74103270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.924+0000 7fad794d1640 1 -- 172.21.15.90:0/1998642381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad74102090 msgr2=0x7fad74102d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.924+0000 7fad794d1640 1 --2- 172.21.15.90:0/1998642381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad74102090 0x7fad74102d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.924+0000 7fad794d1640 1 -- 172.21.15.90:0/1998642381 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fad74109be0 msgr2=0x7fad74103270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.924+0000 7fad794d1640 1 --2- 172.21.15.90:0/1998642381 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fad74109be0 0x7fad74103270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.924+0000 7fad794d1640 1 -- 172.21.15.90:0/1998642381 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad741a8ad0 con 0x7fad7410e590 2024-04-25T11:43:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.924+0000 7fad794d1640 1 --2- 172.21.15.90:0/1998642381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad7410e590 0x7fad741037b0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fad6c00cab0 tx=0x7fad6c00cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.924+0000 7fad71ffb640 1 -- 172.21.15.90:0/1998642381 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6c00ae20 con 0x7fad7410e590 2024-04-25T11:43:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.924+0000 7fad71ffb640 1 -- 172.21.15.90:0/1998642381 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad6c00f900 con 0x7fad7410e590 2024-04-25T11:43:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.924+0000 7fad7af34640 1 -- 172.21.15.90:0/1998642381 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad741a8dc0 con 0x7fad7410e590 2024-04-25T11:43:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.924+0000 7fad71ffb640 1 -- 172.21.15.90:0/1998642381 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6c018a50 con 0x7fad7410e590 2024-04-25T11:43:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.925+0000 7fad7af34640 1 -- 172.21.15.90:0/1998642381 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad741a9220 con 0x7fad7410e590 2024-04-25T11:43:04.920 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.927+0000 7fad71ffb640 1 -- 172.21.15.90:0/1998642381 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fad6c00f450 con 0x7fad7410e590 2024-04-25T11:43:04.920 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.927+0000 7fad81de9740 1 Processor -- start 2024-04-25T11:43:04.920 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.927+0000 7fad81de9740 1 -- start start 2024-04-25T11:43:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.928+0000 7fad81de9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a5fb4a1b0 0x559a5fee9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.928+0000 7fad81de9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb212f0 0x559a5feea030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.928+0000 7fad81de9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559a5fb294d0 0x559a5feea570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.928+0000 7fad81de9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559a5fe50460 con 0x559a5fb212f0 2024-04-25T11:43:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.928+0000 7fad81de9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x559a5fe0a590 con 0x559a5fb294d0 2024-04-25T11:43:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.928+0000 7fad81de9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559a5fe4c420 con 0x559a5fb4a1b0 2024-04-25T11:43:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.928+0000 7fad71ffb640 1 --2- 172.21.15.90:0/1998642381 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fad640afc50 0x7fad640b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.928+0000 7fad78cd0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb212f0 0x559a5feea030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.928+0000 7fad78cd0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb212f0 0x559a5feea030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52542/0 (socket says 172.21.15.90:52542) 2024-04-25T11:43:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.928+0000 7fad78cd0640 1 -- 172.21.15.90:0/2196387330 learned_addr learned my addr 172.21.15.90:0/2196387330 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.928+0000 7fad73fff640 1 --2- 172.21.15.90:0/2196387330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a5fb4a1b0 0x559a5fee9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.928+0000 7fad794d1640 1 --2- 172.21.15.90:0/1998642381 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fad640afc50 0x7fad640b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.928+0000 7fad71ffb640 1 -- 172.21.15.90:0/1998642381 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7fad6c0df320 con 0x7fad7410e590 2024-04-25T11:43:04.922 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.928+0000 7fad73fff640 1 --2- 172.21.15.90:0/2196387330 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559a5fb294d0 0x559a5feea570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:04.922 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.928+0000 7fad78cd0640 1 -- 172.21.15.90:0/2196387330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a5fb4a1b0 msgr2=0x559a5fee9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:04.922 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.928+0000 7fad78cd0640 1 --2- 172.21.15.90:0/2196387330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a5fb4a1b0 0x559a5fee9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.922 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.928+0000 7fad78cd0640 1 -- 172.21.15.90:0/2196387330 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559a5fb294d0 msgr2=0x559a5feea570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:04.922 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.928+0000 7fad78cd0640 1 --2- 172.21.15.90:0/2196387330 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559a5fb294d0 0x559a5feea570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.922 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.928+0000 7fad78cd0640 1 -- 172.21.15.90:0/2196387330 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a5fe170d0 con 0x559a5fb212f0 2024-04-25T11:43:04.922 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.928+0000 7fad73fff640 1 --2- 172.21.15.90:0/2196387330 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559a5fb294d0 0x559a5feea570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:04.922 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.928+0000 7fad78cd0640 1 --2- 172.21.15.90:0/2196387330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb212f0 0x559a5feea030 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fad6800bc70 tx=0x7fad680107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df57bc7174ce02e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:04.922 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.928+0000 7fad794d1640 1 --2- 172.21.15.90:0/1998642381 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fad640afc50 0x7fad640b2160 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fad6c012040 tx=0x7fad6c0071c0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:04.922 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.929+0000 7fad5effd640 1 -- 172.21.15.90:0/2196387330 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad68004220 con 0x559a5fb212f0 2024-04-25T11:43:04.922 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.929+0000 7fad5effd640 1 -- 172.21.15.90:0/2196387330 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad6800be20 con 0x559a5fb212f0 2024-04-25T11:43:04.922 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.929+0000 7fad5effd640 1 -- 172.21.15.90:0/2196387330 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad68004c70 con 0x559a5fb212f0 2024-04-25T11:43:04.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.929+0000 7fad81de9740 1 -- 172.21.15.90:0/2196387330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb212f0 msgr2=0x559a5feea030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:04.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.929+0000 7fad81de9740 1 --2- 172.21.15.90:0/2196387330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb212f0 0x559a5feea030 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fad6800bc70 tx=0x7fad680107a0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.929+0000 7fad81de9740 1 -- 172.21.15.90:0/2196387330 shutdown_connections 2024-04-25T11:43:04.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.929+0000 7fad81de9740 1 --2- 172.21.15.90:0/2196387330 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559a5fb294d0 0x559a5feea570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.929+0000 7fad81de9740 1 --2- 172.21.15.90:0/2196387330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb212f0 0x559a5feea030 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.929+0000 7fad81de9740 1 --2- 172.21.15.90:0/2196387330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a5fb4a1b0 0x559a5fee9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.930+0000 7fad81de9740 1 -- 172.21.15.90:0/2196387330 >> 172.21.15.90:0/2196387330 conn(0x559a5fb3e300 msgr2=0x559a5fee62a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:04.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.930+0000 7fad81de9740 1 -- 172.21.15.90:0/2196387330 shutdown_connections 2024-04-25T11:43:04.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.930+0000 7fad81de9740 1 -- 172.21.15.90:0/2196387330 wait complete. 2024-04-25T11:43:04.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.930+0000 7fad81de9740 1 -- start start 2024-04-25T11:43:04.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.930+0000 7fad81de9740 1 Processor -- start 2024-04-25T11:43:04.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.930+0000 7fad81de9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb212f0 0x559a5fef4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:04.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.930+0000 7fad81de9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a5fb294d0 0x559a5fef46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:04.923 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.930+0000 7fad81de9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559a5fb4a1b0 0x559a5fefa0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:04.924 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.930+0000 7fad81de9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559a5fef6c20 con 0x559a5fb212f0 2024-04-25T11:43:04.924 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.930+0000 7fad81de9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x559a5fdd7b60 con 0x559a5fb4a1b0 2024-04-25T11:43:04.924 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.930+0000 7fad81de9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559a5fef6da0 con 0x559a5fb294d0 2024-04-25T11:43:04.924 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.931+0000 7fad81de9740 10 client.0 _ll_register_callbacks cb 0x559a5fdfc2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:04.924 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.931+0000 7fad73fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb212f0 0x559a5fef4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:04.924 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.931+0000 7fad73fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb212f0 0x559a5fef4180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52546/0 (socket says 172.21.15.90:52546) 2024-04-25T11:43:04.924 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.931+0000 7fad73fff640 1 -- 172.21.15.90:0/1371228716 learned_addr learned my addr 172.21.15.90:0/1371228716 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:04.924 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.931+0000 7fad78cd0640 1 --2- 172.21.15.90:0/1371228716 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a5fb294d0 0x559a5fef46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:04.924 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.931+0000 7fad73fff640 1 --2- 172.21.15.90:0/1371228716 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559a5fb4a1b0 0x559a5fefa0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:04.924 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.931+0000 7fad73fff640 1 -- 172.21.15.90:0/1371228716 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a5fb294d0 msgr2=0x559a5fef46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:04.924 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.931+0000 7fad73fff640 1 --2- 172.21.15.90:0/1371228716 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a5fb294d0 0x559a5fef46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.924 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.931+0000 7fad73fff640 1 -- 172.21.15.90:0/1371228716 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559a5fb4a1b0 msgr2=0x559a5fefa0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:04.924 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.931+0000 7fad73fff640 1 --2- 172.21.15.90:0/1371228716 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559a5fb4a1b0 0x559a5fefa0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.931+0000 7fad73fff640 1 -- 172.21.15.90:0/1371228716 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a5fb62210 con 0x559a5fb212f0 2024-04-25T11:43:04.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.931+0000 7fad73fff640 1 --2- 172.21.15.90:0/1371228716 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb212f0 0x559a5fef4180 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fad7406bf10 tx=0x7fad60009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bcd7d59708ccf9b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:04.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.931+0000 7fad53fff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:04.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.931+0000 7fad53fff640 1 -- 172.21.15.90:0/1371228716 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad60035070 con 0x559a5fb212f0 2024-04-25T11:43:04.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.931+0000 7fad53fff640 1 -- 172.21.15.90:0/1371228716 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7fad60004240 con 0x559a5fb212f0 2024-04-25T11:43:04.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.931+0000 7fad81de9740 1 -- 172.21.15.90:0/1371228716 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad6800be20 con 0x559a5fb212f0 2024-04-25T11:43:04.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.931+0000 7fad53fff640 1 -- 172.21.15.90:0/1371228716 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fad60004440 con 0x559a5fb212f0 2024-04-25T11:43:04.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.931+0000 7fad81de9740 10 client.18012 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:04.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.931+0000 7fad81de9740 1 -- 172.21.15.90:0/1371228716 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559a5fc2e620 con 0x559a5fb212f0 2024-04-25T11:43:04.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.932+0000 7fad53fff640 1 -- 172.21.15.90:0/1371228716 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad6800be20 con 0x559a5fb212f0 2024-04-25T11:43:04.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.932+0000 7fad53fff640 1 -- 172.21.15.90:0/1371228716 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fad6003d660 con 0x559a5fb212f0 2024-04-25T11:43:04.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.932+0000 7fad81de9740 10 client.18012 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:04.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.932+0000 7fad81de9740 10 client.18012 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:43:04.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.932+0000 7fad81de9740 20 client.18012 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:04.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.932+0000 7fad81de9740 1 --2- 172.21.15.90:0/1371228716 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559a5fb08d80 0x559a5ff06690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:04.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.932+0000 7fad81de9740 4 client.18012 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:43:04.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.932+0000 7fad81de9740 1 -- 172.21.15.90:0/1371228716 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a5fe5e070 con 0x559a5fb08d80 2024-04-25T11:43:04.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.932+0000 7fad78cd0640 1 --2- 172.21.15.90:0/1371228716 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559a5fb08d80 0x559a5ff06690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:04.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.933+0000 7fad53fff640 10 client.18012 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:04.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.933+0000 7fad78cd0640 1 --2- 172.21.15.90:0/1371228716 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559a5fb08d80 0x559a5ff06690 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d57628cdab0ca0ff server_cookie=28cce6652520f558 in_seq=0 out_seq=0 2024-04-25T11:43:04.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.934+0000 7fad53fff640 1 -- 172.21.15.90:0/1371228716 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559a5fe5e070 con 0x559a5fb08d80 2024-04-25T11:43:04.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.934+0000 7fad53fff640 10 client.18012 handle_command_reply: tid=0 2024-04-25T11:43:04.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.934+0000 7fad81de9740 1 client.18012 shutdown 2024-04-25T11:43:04.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.934+0000 7fad81de9740 1 -- 172.21.15.90:0/1371228716 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb212f0 msgr2=0x559a5fef4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:04.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.934+0000 7fad81de9740 1 --2- 172.21.15.90:0/1371228716 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb212f0 0x559a5fef4180 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fad7406bf10 tx=0x7fad60009260 comp rx=0 tx=0).stop 2024-04-25T11:43:04.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.934+0000 7fad81de9740 1 -- 172.21.15.90:0/1371228716 shutdown_connections 2024-04-25T11:43:04.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.934+0000 7fad81de9740 1 --2- 172.21.15.90:0/1371228716 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559a5fb08d80 0x559a5ff06690 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.934+0000 7fad81de9740 1 --2- 172.21.15.90:0/1371228716 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559a5fb4a1b0 0x559a5fefa0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.934+0000 7fad81de9740 1 --2- 172.21.15.90:0/1371228716 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a5fb294d0 0x559a5fef46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.934+0000 7fad81de9740 1 --2- 172.21.15.90:0/1371228716 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb212f0 0x559a5fef4180 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.934+0000 7fad53fff640 0 client.18012 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:04.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.934+0000 7fad81de9740 1 -- 172.21.15.90:0/1371228716 >> 172.21.15.90:0/1371228716 conn(0x559a5fb3e300 msgr2=0x559a5fee9920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:04.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.934+0000 7fad81de9740 1 -- 172.21.15.90:0/1371228716 shutdown_connections 2024-04-25T11:43:04.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.934+0000 7fad81de9740 1 -- 172.21.15.90:0/1371228716 wait complete. 2024-04-25T11:43:04.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.934+0000 7fad81de9740 20 client.18012 trim_cache size 0 max 16384 2024-04-25T11:43:04.936 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.949+0000 7fad81de9740 1 Processor -- start 2024-04-25T11:43:04.936 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.949+0000 7fad81de9740 1 -- start start 2024-04-25T11:43:04.936 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.949+0000 7fad81de9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a5fb212f0 0x559a5fefa010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:04.936 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.949+0000 7fad81de9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559a5fb294d0 0x559a5fee9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:04.936 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.949+0000 7fad81de9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb4a1b0 0x559a5ff0be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:04.936 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.949+0000 7fad81de9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fad741b1ed0 con 0x559a5fb4a1b0 2024-04-25T11:43:04.936 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.949+0000 7fad81de9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fad741b3a10 con 0x559a5fb294d0 2024-04-25T11:43:04.936 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.949+0000 7fad81de9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fad741b25a0 con 0x559a5fb212f0 2024-04-25T11:43:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.949+0000 7fad73fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb4a1b0 0x559a5ff0be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.949+0000 7fad73fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb4a1b0 0x559a5ff0be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52552/0 (socket says 172.21.15.90:52552) 2024-04-25T11:43:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.949+0000 7fad73fff640 1 -- 172.21.15.90:0/1193750928 learned_addr learned my addr 172.21.15.90:0/1193750928 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.949+0000 7fad73fff640 1 -- 172.21.15.90:0/1193750928 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a5fb212f0 msgr2=0x559a5fefa010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.949+0000 7fad73fff640 1 --2- 172.21.15.90:0/1193750928 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a5fb212f0 0x559a5fefa010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.950+0000 7fad73fff640 1 -- 172.21.15.90:0/1193750928 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559a5fb294d0 msgr2=0x559a5fee9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.950+0000 7fad73fff640 1 --2- 172.21.15.90:0/1193750928 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559a5fb294d0 0x559a5fee9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.950+0000 7fad73fff640 1 -- 172.21.15.90:0/1193750928 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a5fef5430 con 0x559a5fb4a1b0 2024-04-25T11:43:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.950+0000 7fad73fff640 1 --2- 172.21.15.90:0/1193750928 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb4a1b0 0x559a5ff0be50 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fad60006010 tx=0x7fad6002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ea234cb7be91f0d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.950+0000 7fad5e7fc640 1 -- 172.21.15.90:0/1193750928 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad60035040 con 0x559a5fb4a1b0 2024-04-25T11:43:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.950+0000 7fad5e7fc640 1 -- 172.21.15.90:0/1193750928 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad60031c70 con 0x559a5fb4a1b0 2024-04-25T11:43:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.950+0000 7fad5e7fc640 1 -- 172.21.15.90:0/1193750928 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad60004750 con 0x559a5fb4a1b0 2024-04-25T11:43:04.941 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.951+0000 7fad81de9740 1 -- 172.21.15.90:0/1193750928 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb4a1b0 msgr2=0x559a5ff0be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:04.941 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.951+0000 7fad81de9740 1 --2- 172.21.15.90:0/1193750928 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb4a1b0 0x559a5ff0be50 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fad60006010 tx=0x7fad6002a410 comp rx=0 tx=0).stop 2024-04-25T11:43:04.941 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.951+0000 7fad81de9740 1 -- 172.21.15.90:0/1193750928 shutdown_connections 2024-04-25T11:43:04.941 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.951+0000 7fad81de9740 1 --2- 172.21.15.90:0/1193750928 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb4a1b0 0x559a5ff0be50 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.941 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.951+0000 7fad81de9740 1 --2- 172.21.15.90:0/1193750928 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559a5fb294d0 0x559a5fee9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.941 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.951+0000 7fad81de9740 1 --2- 172.21.15.90:0/1193750928 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a5fb212f0 0x559a5fefa010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.941 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.951+0000 7fad81de9740 1 -- 172.21.15.90:0/1193750928 >> 172.21.15.90:0/1193750928 conn(0x559a5fb3e300 msgr2=0x559a5fee9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:04.941 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.951+0000 7fad81de9740 1 -- 172.21.15.90:0/1193750928 shutdown_connections 2024-04-25T11:43:04.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.951+0000 7fad81de9740 1 -- 172.21.15.90:0/1193750928 wait complete. 2024-04-25T11:43:04.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.951+0000 7fad81de9740 1 -- start start 2024-04-25T11:43:04.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.951+0000 7fad81de9740 1 Processor -- start 2024-04-25T11:43:04.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.951+0000 7fad81de9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a5fb212f0 0x559a5fefd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:04.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.951+0000 7fad81de9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559a5fb294d0 0x559a5fefdf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:04.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.952+0000 7fad81de9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb4a1b0 0x559a5ff66a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:04.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.952+0000 7fad81de9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fad741b3340 con 0x559a5fb4a1b0 2024-04-25T11:43:04.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.952+0000 7fad81de9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fad741b2c70 con 0x559a5fb294d0 2024-04-25T11:43:04.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.952+0000 7fad81de9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fad741afed0 con 0x559a5fb212f0 2024-04-25T11:43:04.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.952+0000 7fad81de9740 10 client.0 _ll_register_callbacks cb 0x7fad60009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:04.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.952+0000 7fad73fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb4a1b0 0x559a5ff66a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:04.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.952+0000 7fad73fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb4a1b0 0x559a5ff66a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52556/0 (socket says 172.21.15.90:52556) 2024-04-25T11:43:04.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.952+0000 7fad73fff640 1 -- 172.21.15.90:0/2706691341 learned_addr learned my addr 172.21.15.90:0/2706691341 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:04.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.952+0000 7fad78cd0640 1 --2- 172.21.15.90:0/2706691341 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559a5fb294d0 0x559a5fefdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:04.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.952+0000 7fad73fff640 1 --2- 172.21.15.90:0/2706691341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a5fb212f0 0x559a5fefd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:04.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.952+0000 7fad73fff640 1 -- 172.21.15.90:0/2706691341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a5fb212f0 msgr2=0x559a5fefd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:04.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.952+0000 7fad73fff640 1 --2- 172.21.15.90:0/2706691341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a5fb212f0 0x559a5fefd9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.952+0000 7fad73fff640 1 -- 172.21.15.90:0/2706691341 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559a5fb294d0 msgr2=0x559a5fefdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:04.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.952+0000 7fad73fff640 1 --2- 172.21.15.90:0/2706691341 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559a5fb294d0 0x559a5fefdf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.952+0000 7fad73fff640 1 -- 172.21.15.90:0/2706691341 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a5ff66fb0 con 0x559a5fb4a1b0 2024-04-25T11:43:04.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.952+0000 7fad73fff640 1 --2- 172.21.15.90:0/2706691341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb4a1b0 0x559a5ff66a40 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fad600059d0 tx=0x7fad600045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff30223de57377d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:04.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.952+0000 7fad53fff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:04.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.952+0000 7fad53fff640 1 -- 172.21.15.90:0/2706691341 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad60035070 con 0x559a5fb4a1b0 2024-04-25T11:43:04.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.952+0000 7fad53fff640 1 -- 172.21.15.90:0/2706691341 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8658+0+0 (secure 0 0 0) 0x7fad60005650 con 0x559a5fb4a1b0 2024-04-25T11:43:04.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.952+0000 7fad81de9740 1 -- 172.21.15.90:0/2706691341 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad60031c70 con 0x559a5fb4a1b0 2024-04-25T11:43:04.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.952+0000 7fad53fff640 1 -- 172.21.15.90:0/2706691341 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fad600048e0 con 0x559a5fb4a1b0 2024-04-25T11:43:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.952+0000 7fad81de9740 10 client.18018 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.952+0000 7fad81de9740 1 -- 172.21.15.90:0/2706691341 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559a5ff6e2e0 con 0x559a5fb4a1b0 2024-04-25T11:43:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.953+0000 7fad53fff640 1 -- 172.21.15.90:0/2706691341 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad60031c70 con 0x559a5fb4a1b0 2024-04-25T11:43:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.953+0000 7fad53fff640 1 -- 172.21.15.90:0/2706691341 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fad6005e970 con 0x559a5fb4a1b0 2024-04-25T11:43:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.953+0000 7fad81de9740 10 client.18018 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.953+0000 7fad81de9740 10 client.18018 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:43:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.953+0000 7fad81de9740 20 client.18018 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.953+0000 7fad81de9740 1 --2- 172.21.15.90:0/2706691341 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559a5fade1c0 0x559a5ff70930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.953+0000 7fad81de9740 4 client.18018 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:43:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.953+0000 7fad81de9740 1 -- 172.21.15.90:0/2706691341 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559a5fe57ee0 con 0x559a5fade1c0 2024-04-25T11:43:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.953+0000 7fad78cd0640 1 --2- 172.21.15.90:0/2706691341 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559a5fade1c0 0x559a5ff70930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.954+0000 7fad53fff640 10 client.18018 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.954+0000 7fad78cd0640 1 --2- 172.21.15.90:0/2706691341 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559a5fade1c0 0x559a5ff70930 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dac2da2471f72cef server_cookie=523a0d49f9961055 in_seq=0 out_seq=0 2024-04-25T11:43:04.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.970+0000 7fad53fff640 1 -- 172.21.15.90:0/2706691341 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559a5fe57ee0 con 0x559a5fade1c0 2024-04-25T11:43:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.970+0000 7fad53fff640 10 client.18018 handle_command_reply: tid=0 2024-04-25T11:43:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.970+0000 7fad81de9740 1 client.18018 shutdown 2024-04-25T11:43:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.970+0000 7fad81de9740 1 -- 172.21.15.90:0/2706691341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb4a1b0 msgr2=0x559a5ff66a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.970+0000 7fad81de9740 1 --2- 172.21.15.90:0/2706691341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb4a1b0 0x559a5ff66a40 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fad600059d0 tx=0x7fad600045e0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.970+0000 7fad81de9740 1 -- 172.21.15.90:0/2706691341 shutdown_connections 2024-04-25T11:43:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.970+0000 7fad81de9740 1 --2- 172.21.15.90:0/2706691341 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x559a5fade1c0 0x559a5ff70930 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.970+0000 7fad81de9740 1 --2- 172.21.15.90:0/2706691341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a5fb4a1b0 0x559a5ff66a40 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.970+0000 7fad53fff640 0 client.18018 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.970+0000 7fad81de9740 1 --2- 172.21.15.90:0/2706691341 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x559a5fb294d0 0x559a5fefdf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.970+0000 7fad81de9740 1 --2- 172.21.15.90:0/2706691341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a5fb212f0 0x559a5fefd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.970+0000 7fad81de9740 1 -- 172.21.15.90:0/2706691341 >> 172.21.15.90:0/2706691341 conn(0x559a5fb3e300 msgr2=0x559a5fef5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.970+0000 7fad81de9740 1 -- 172.21.15.90:0/2706691341 shutdown_connections 2024-04-25T11:43:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.970+0000 7fad81de9740 1 -- 172.21.15.90:0/2706691341 wait complete. 2024-04-25T11:43:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.971+0000 7fad81de9740 20 client.18018 trim_cache size 0 max 16384 2024-04-25T11:43:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:43:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:43:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:43:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:43:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.971+0000 7fad7af34640 1 -- 172.21.15.90:0/1998642381 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fad640afc50 msgr2=0x7fad640b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.971+0000 7fad7af34640 1 --2- 172.21.15.90:0/1998642381 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fad640afc50 0x7fad640b2160 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fad6c012040 tx=0x7fad6c0071c0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.971+0000 7fad7af34640 1 -- 172.21.15.90:0/1998642381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad7410e590 msgr2=0x7fad741037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.971+0000 7fad7af34640 1 --2- 172.21.15.90:0/1998642381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad7410e590 0x7fad741037b0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fad6c00cab0 tx=0x7fad6c00cf70 comp rx=0 tx=0).stop 2024-04-25T11:43:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.971+0000 7fad7af34640 1 -- 172.21.15.90:0/1998642381 shutdown_connections 2024-04-25T11:43:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.971+0000 7fad7af34640 1 --2- 172.21.15.90:0/1998642381 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fad640afc50 0x7fad640b2160 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.971+0000 7fad7af34640 1 --2- 172.21.15.90:0/1998642381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad7410e590 0x7fad741037b0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.971+0000 7fad7af34640 1 --2- 172.21.15.90:0/1998642381 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fad74109be0 0x7fad74103270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.971+0000 7fad7af34640 1 --2- 172.21.15.90:0/1998642381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad74102090 0x7fad74102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.971+0000 7fad7af34640 1 -- 172.21.15.90:0/1998642381 >> 172.21.15.90:0/1998642381 conn(0x7fad740fd8a0 msgr2=0x7fad740fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.971+0000 7fad7af34640 1 -- 172.21.15.90:0/1998642381 shutdown_connections 2024-04-25T11:43:04.960 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:04.971+0000 7fad7af34640 1 -- 172.21.15.90:0/1998642381 wait complete. 2024-04-25T11:43:06.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:05 smithi138 ceph-mon[32328]: pgmap v245: 97 pgs: 97 active+clean; 61 GiB data, 185 GiB used, 888 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 24 MiB/s wr, 292 op/s 2024-04-25T11:43:06.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:05 smithi156 ceph-mon[33045]: pgmap v245: 97 pgs: 97 active+clean; 61 GiB data, 185 GiB used, 888 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 24 MiB/s wr, 292 op/s 2024-04-25T11:43:06.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:05 smithi090 ceph-mon[29575]: pgmap v245: 97 pgs: 97 active+clean; 61 GiB data, 185 GiB used, 888 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 24 MiB/s wr, 292 op/s 2024-04-25T11:43:07.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:07 smithi138 ceph-mon[32328]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T11:43:07.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:07 smithi156 ceph-mon[33045]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T11:43:07.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:07 smithi090 ceph-mon[29575]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T11:43:08.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:08 smithi138 ceph-mon[32328]: pgmap v247: 97 pgs: 97 active+clean; 61 GiB data, 185 GiB used, 888 GiB / 1.0 TiB avail; 13 KiB/s rd, 12 MiB/s wr, 159 op/s 2024-04-25T11:43:08.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:08 smithi156 ceph-mon[33045]: pgmap v247: 97 pgs: 97 active+clean; 61 GiB data, 185 GiB used, 888 GiB / 1.0 TiB avail; 13 KiB/s rd, 12 MiB/s wr, 159 op/s 2024-04-25T11:43:08.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:08 smithi090 ceph-mon[29575]: pgmap v247: 97 pgs: 97 active+clean; 61 GiB data, 185 GiB used, 888 GiB / 1.0 TiB avail; 13 KiB/s rd, 12 MiB/s wr, 159 op/s 2024-04-25T11:43:10.027 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:43:10.027 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:43:10.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.164+0000 7fcb7e18f640 1 -- 172.21.15.90:0/842490421 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb78102420 msgr2=0x7fcb78102830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.164+0000 7fcb7e18f640 1 --2- 172.21.15.90:0/842490421 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb78102420 0x7fcb78102830 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fcb68009900 tx=0x7fcb6802a410 comp rx=0 tx=0).stop 2024-04-25T11:43:10.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.164+0000 7fcb7e18f640 1 -- 172.21.15.90:0/842490421 shutdown_connections 2024-04-25T11:43:10.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.164+0000 7fcb7e18f640 1 --2- 172.21.15.90:0/842490421 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb78107470 0x7fcb7810ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.164+0000 7fcb7e18f640 1 --2- 172.21.15.90:0/842490421 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb781069b0 0x7fcb78106de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.164+0000 7fcb7e18f640 1 --2- 172.21.15.90:0/842490421 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb78102420 0x7fcb78102830 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.164+0000 7fcb7e18f640 1 -- 172.21.15.90:0/842490421 >> 172.21.15.90:0/842490421 conn(0x7fcb78077920 msgr2=0x7fcb78077d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:10.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.164+0000 7fcb7e18f640 1 -- 172.21.15.90:0/842490421 shutdown_connections 2024-04-25T11:43:10.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.164+0000 7fcb7e18f640 1 -- 172.21.15.90:0/842490421 wait complete. 2024-04-25T11:43:10.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.165+0000 7fcb7e18f640 1 Processor -- start 2024-04-25T11:43:10.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.165+0000 7fcb7e18f640 1 -- start start 2024-04-25T11:43:10.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.165+0000 7fcb7e18f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb78102420 0x7fcb7819b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.165+0000 7fcb7e18f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb781069b0 0x7fcb7819bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.165+0000 7fcb7e18f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb78107470 0x7fcb7819bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.165+0000 7fcb7e18f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcb781123a0 con 0x7fcb78102420 2024-04-25T11:43:10.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.165+0000 7fcb7e18f640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fcb78112220 con 0x7fcb78107470 2024-04-25T11:43:10.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.165+0000 7fcb7e18f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcb78112520 con 0x7fcb781069b0 2024-04-25T11:43:10.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.166+0000 7fcb777fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb78102420 0x7fcb7819b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.166+0000 7fcb777fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb78102420 0x7fcb7819b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52574/0 (socket says 172.21.15.90:52574) 2024-04-25T11:43:10.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.166+0000 7fcb777fe640 1 -- 172.21.15.90:0/1587244308 learned_addr learned my addr 172.21.15.90:0/1587244308 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:10.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.166+0000 7fcb77fff640 1 --2- 172.21.15.90:0/1587244308 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb78107470 0x7fcb7819bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.166+0000 7fcb76ffd640 1 --2- 172.21.15.90:0/1587244308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb781069b0 0x7fcb7819bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.166+0000 7fcb777fe640 1 -- 172.21.15.90:0/1587244308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb781069b0 msgr2=0x7fcb7819bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.166+0000 7fcb777fe640 1 --2- 172.21.15.90:0/1587244308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb781069b0 0x7fcb7819bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.166+0000 7fcb777fe640 1 -- 172.21.15.90:0/1587244308 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb78107470 msgr2=0x7fcb7819bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.166+0000 7fcb777fe640 1 --2- 172.21.15.90:0/1587244308 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb78107470 0x7fcb7819bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.166+0000 7fcb777fe640 1 -- 172.21.15.90:0/1587244308 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb781a6880 con 0x7fcb78102420 2024-04-25T11:43:10.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.166+0000 7fcb77fff640 1 --2- 172.21.15.90:0/1587244308 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb78107470 0x7fcb7819bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:43:10.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.166+0000 7fcb777fe640 1 --2- 172.21.15.90:0/1587244308 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb78102420 0x7fcb7819b570 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fcb68005940 tx=0x7fcb680040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:10.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.166+0000 7fcb74ff9640 1 -- 172.21.15.90:0/1587244308 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb6803f030 con 0x7fcb78102420 2024-04-25T11:43:10.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.166+0000 7fcb7e18f640 1 -- 172.21.15.90:0/1587244308 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb781a6b10 con 0x7fcb78102420 2024-04-25T11:43:10.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.166+0000 7fcb74ff9640 1 -- 172.21.15.90:0/1587244308 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb680044c0 con 0x7fcb78102420 2024-04-25T11:43:10.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.166+0000 7fcb74ff9640 1 -- 172.21.15.90:0/1587244308 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb68035070 con 0x7fcb78102420 2024-04-25T11:43:10.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.166+0000 7fcb7e18f640 1 -- 172.21.15.90:0/1587244308 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb781a7000 con 0x7fcb78102420 2024-04-25T11:43:10.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.169+0000 7fcb74ff9640 1 -- 172.21.15.90:0/1587244308 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fcb68042030 con 0x7fcb78102420 2024-04-25T11:43:10.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.169+0000 7fcb85044740 1 Processor -- start 2024-04-25T11:43:10.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.169+0000 7fcb85044740 1 -- start start 2024-04-25T11:43:10.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.169+0000 7fcb85044740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563dc9044390 0x563dc93e4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.169+0000 7fcb85044740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563dc901b4d0 0x563dc93e4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.169+0000 7fcb85044740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563dc90236b0 0x563dc93f18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.169+0000 7fcb85044740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563dc934ad80 con 0x563dc901b4d0 2024-04-25T11:43:10.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.169+0000 7fcb85044740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x563dc93048b0 con 0x563dc9044390 2024-04-25T11:43:10.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.169+0000 7fcb85044740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563dc9346d40 con 0x563dc90236b0 2024-04-25T11:43:10.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.169+0000 7fcb777fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563dc901b4d0 0x563dc93e4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.169+0000 7fcb77fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563dc9044390 0x563dc93e4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.169+0000 7fcb777fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563dc901b4d0 0x563dc93e4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52584/0 (socket says 172.21.15.90:52584) 2024-04-25T11:43:10.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.169+0000 7fcb777fe640 1 -- 172.21.15.90:0/1038152200 learned_addr learned my addr 172.21.15.90:0/1038152200 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:10.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.169+0000 7fcb77fff640 1 --2- 172.21.15.90:0/1038152200 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563dc9044390 0x563dc93e4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:60104/0 (socket says 172.21.15.90:60104) 2024-04-25T11:43:10.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.170+0000 7fcb76ffd640 1 --2- 172.21.15.90:0/1038152200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563dc90236b0 0x563dc93f18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.170+0000 7fcb74ff9640 1 --2- 172.21.15.90:0/1587244308 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcb600afc60 0x7fcb600b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.170+0000 7fcb777fe640 1 -- 172.21.15.90:0/1038152200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563dc90236b0 msgr2=0x563dc93f18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:10.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.170+0000 7fcb777fe640 1 --2- 172.21.15.90:0/1038152200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563dc90236b0 0x563dc93f18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.170+0000 7fcb777fe640 1 -- 172.21.15.90:0/1038152200 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563dc9044390 msgr2=0x563dc93e4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:10.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.170+0000 7fcb777fe640 1 --2- 172.21.15.90:0/1038152200 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563dc9044390 0x563dc93e4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.170+0000 7fcb777fe640 1 -- 172.21.15.90:0/1038152200 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563dc92d5a40 con 0x563dc901b4d0 2024-04-25T11:43:10.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.170+0000 7fcb77fff640 1 --2- 172.21.15.90:0/1038152200 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563dc9044390 0x563dc93e4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:43:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.170+0000 7fcb77fff640 1 --2- 172.21.15.90:0/1587244308 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcb600afc60 0x7fcb600b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.170+0000 7fcb74ff9640 1 -- 172.21.15.90:0/1587244308 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7fcb680ff420 con 0x7fcb78102420 2024-04-25T11:43:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.170+0000 7fcb777fe640 1 --2- 172.21.15.90:0/1038152200 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563dc901b4d0 0x563dc93e4fa0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fcb680fca30 tx=0x7fcb6808da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4ba5772989fa52b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.170+0000 7fcb5dffb640 1 -- 172.21.15.90:0/1038152200 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb680c6f10 con 0x563dc901b4d0 2024-04-25T11:43:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.170+0000 7fcb77fff640 1 --2- 172.21.15.90:0/1587244308 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcb600afc60 0x7fcb600b2170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcb70005c40 tx=0x7fcb70004a40 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.170+0000 7fcb5dffb640 1 -- 172.21.15.90:0/1038152200 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb680e0030 con 0x563dc901b4d0 2024-04-25T11:43:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.170+0000 7fcb5dffb640 1 -- 172.21.15.90:0/1038152200 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb680ea130 con 0x563dc901b4d0 2024-04-25T11:43:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.171+0000 7fcb85044740 1 -- 172.21.15.90:0/1038152200 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563dc901b4d0 msgr2=0x563dc93e4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.171+0000 7fcb85044740 1 --2- 172.21.15.90:0/1038152200 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563dc901b4d0 0x563dc93e4fa0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fcb680fca30 tx=0x7fcb6808da90 comp rx=0 tx=0).stop 2024-04-25T11:43:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.171+0000 7fcb85044740 1 -- 172.21.15.90:0/1038152200 shutdown_connections 2024-04-25T11:43:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.171+0000 7fcb85044740 1 --2- 172.21.15.90:0/1038152200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563dc90236b0 0x563dc93f18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.171+0000 7fcb85044740 1 --2- 172.21.15.90:0/1038152200 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563dc901b4d0 0x563dc93e4fa0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.171+0000 7fcb85044740 1 --2- 172.21.15.90:0/1038152200 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563dc9044390 0x563dc93e4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.171+0000 7fcb85044740 1 -- 172.21.15.90:0/1038152200 >> 172.21.15.90:0/1038152200 conn(0x563dc90384e0 msgr2=0x563dc93e1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.171+0000 7fcb85044740 1 -- 172.21.15.90:0/1038152200 shutdown_connections 2024-04-25T11:43:10.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.171+0000 7fcb85044740 1 -- 172.21.15.90:0/1038152200 wait complete. 2024-04-25T11:43:10.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.172+0000 7fcb85044740 1 -- start start 2024-04-25T11:43:10.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.172+0000 7fcb85044740 1 Processor -- start 2024-04-25T11:43:10.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.172+0000 7fcb85044740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563dc901b4d0 0x563dc93ef500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.172+0000 7fcb85044740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563dc90236b0 0x563dc93efa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.172+0000 7fcb85044740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563dc9044390 0x563dc93ff580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.172+0000 7fcb85044740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563dc934ad80 con 0x563dc901b4d0 2024-04-25T11:43:10.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.172+0000 7fcb85044740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x563dc92d1d10 con 0x563dc9044390 2024-04-25T11:43:10.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.172+0000 7fcb85044740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563dc93f1e30 con 0x563dc90236b0 2024-04-25T11:43:10.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.172+0000 7fcb777fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563dc901b4d0 0x563dc93ef500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.172+0000 7fcb85044740 10 client.0 _ll_register_callbacks cb 0x563dc92fe210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:10.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.172+0000 7fcb777fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563dc901b4d0 0x563dc93ef500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52600/0 (socket says 172.21.15.90:52600) 2024-04-25T11:43:10.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.172+0000 7fcb777fe640 1 -- 172.21.15.90:0/122990538 learned_addr learned my addr 172.21.15.90:0/122990538 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:10.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.173+0000 7fcb77fff640 1 --2- 172.21.15.90:0/122990538 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563dc9044390 0x563dc93ff580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.173+0000 7fcb76ffd640 1 --2- 172.21.15.90:0/122990538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563dc90236b0 0x563dc93efa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.173+0000 7fcb777fe640 1 -- 172.21.15.90:0/122990538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563dc90236b0 msgr2=0x563dc93efa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:10.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.173+0000 7fcb777fe640 1 --2- 172.21.15.90:0/122990538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563dc90236b0 0x563dc93efa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.173+0000 7fcb777fe640 1 -- 172.21.15.90:0/122990538 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563dc9044390 msgr2=0x563dc93ff580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:10.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.173+0000 7fcb777fe640 1 --2- 172.21.15.90:0/122990538 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563dc9044390 0x563dc93ff580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.173+0000 7fcb777fe640 1 -- 172.21.15.90:0/122990538 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563dc91267d0 con 0x563dc901b4d0 2024-04-25T11:43:10.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.173+0000 7fcb77fff640 1 --2- 172.21.15.90:0/122990538 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563dc9044390 0x563dc93ff580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:10.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.173+0000 7fcb777fe640 1 --2- 172.21.15.90:0/122990538 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563dc901b4d0 0x563dc93ef500 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fcb680c4830 tx=0x7fcb6802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4a7084e918b4846 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:10.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.173+0000 7fcb56ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:10.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.173+0000 7fcb56ffd640 1 -- 172.21.15.90:0/122990538 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb68049070 con 0x563dc901b4d0 2024-04-25T11:43:10.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.173+0000 7fcb56ffd640 1 -- 172.21.15.90:0/122990538 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7fcb680c3850 con 0x563dc901b4d0 2024-04-25T11:43:10.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.173+0000 7fcb85044740 1 -- 172.21.15.90:0/122990538 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb680e0030 con 0x563dc901b4d0 2024-04-25T11:43:10.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.173+0000 7fcb56ffd640 1 -- 172.21.15.90:0/122990538 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcb680c6d30 con 0x563dc901b4d0 2024-04-25T11:43:10.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.173+0000 7fcb85044740 10 client.18030 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:10.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.173+0000 7fcb85044740 1 -- 172.21.15.90:0/122990538 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563dc905c3f0 con 0x563dc901b4d0 2024-04-25T11:43:10.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.173+0000 7fcb56ffd640 1 -- 172.21.15.90:0/122990538 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb680e0030 con 0x563dc901b4d0 2024-04-25T11:43:10.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.174+0000 7fcb56ffd640 1 -- 172.21.15.90:0/122990538 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fcb6808d670 con 0x563dc901b4d0 2024-04-25T11:43:10.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.174+0000 7fcb85044740 10 client.18030 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:10.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.174+0000 7fcb85044740 10 client.18030 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:43:10.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.174+0000 7fcb85044740 20 client.18030 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:10.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.174+0000 7fcb85044740 1 --2- 172.21.15.90:0/122990538 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563dc9002f60 0x563dc9405520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.174+0000 7fcb85044740 4 client.18030 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:43:10.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.174+0000 7fcb85044740 1 -- 172.21.15.90:0/122990538 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563dc9358990 con 0x563dc9002f60 2024-04-25T11:43:10.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.174+0000 7fcb76ffd640 1 --2- 172.21.15.90:0/122990538 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563dc9002f60 0x563dc9405520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.174+0000 7fcb56ffd640 10 client.18030 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:10.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.174+0000 7fcb76ffd640 1 --2- 172.21.15.90:0/122990538 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563dc9002f60 0x563dc9405520 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c66bb11876dcf68 server_cookie=a2b5aac1fee7562b in_seq=0 out_seq=0 2024-04-25T11:43:10.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.175+0000 7fcb56ffd640 1 -- 172.21.15.90:0/122990538 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563dc9358990 con 0x563dc9002f60 2024-04-25T11:43:10.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.175+0000 7fcb56ffd640 10 client.18030 handle_command_reply: tid=0 2024-04-25T11:43:10.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.175+0000 7fcb85044740 1 client.18030 shutdown 2024-04-25T11:43:10.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.175+0000 7fcb85044740 1 -- 172.21.15.90:0/122990538 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563dc901b4d0 msgr2=0x563dc93ef500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.175+0000 7fcb85044740 1 --2- 172.21.15.90:0/122990538 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563dc901b4d0 0x563dc93ef500 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fcb680c4830 tx=0x7fcb6802a410 comp rx=0 tx=0).stop 2024-04-25T11:43:10.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.175+0000 7fcb85044740 1 -- 172.21.15.90:0/122990538 shutdown_connections 2024-04-25T11:43:10.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.175+0000 7fcb85044740 1 --2- 172.21.15.90:0/122990538 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563dc9002f60 0x563dc9405520 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.175+0000 7fcb85044740 1 --2- 172.21.15.90:0/122990538 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563dc9044390 0x563dc93ff580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.175+0000 7fcb85044740 1 --2- 172.21.15.90:0/122990538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563dc90236b0 0x563dc93efa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.175+0000 7fcb56ffd640 0 client.18030 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:10.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.175+0000 7fcb85044740 1 --2- 172.21.15.90:0/122990538 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563dc901b4d0 0x563dc93ef500 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.175+0000 7fcb85044740 1 -- 172.21.15.90:0/122990538 >> 172.21.15.90:0/122990538 conn(0x563dc90384e0 msgr2=0x563dc93e6780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:10.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.175+0000 7fcb85044740 1 -- 172.21.15.90:0/122990538 shutdown_connections 2024-04-25T11:43:10.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.175+0000 7fcb85044740 1 -- 172.21.15.90:0/122990538 wait complete. 2024-04-25T11:43:10.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.175+0000 7fcb85044740 20 client.18030 trim_cache size 0 max 16384 2024-04-25T11:43:10.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.195+0000 7fcb85044740 1 Processor -- start 2024-04-25T11:43:10.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.195+0000 7fcb85044740 1 -- start start 2024-04-25T11:43:10.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.195+0000 7fcb85044740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563dc901b4d0 0x563dc93f1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.195+0000 7fcb85044740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563dc90236b0 0x563dc940ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.195+0000 7fcb85044740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563dc9044390 0x563dc940b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.195+0000 7fcb85044740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcb781adaf0 con 0x563dc90236b0 2024-04-25T11:43:10.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.195+0000 7fcb85044740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fcb781af630 con 0x563dc901b4d0 2024-04-25T11:43:10.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.195+0000 7fcb85044740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcb781ae1c0 con 0x563dc9044390 2024-04-25T11:43:10.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.195+0000 7fcb76ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563dc90236b0 0x563dc940ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.195+0000 7fcb76ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563dc90236b0 0x563dc940ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52602/0 (socket says 172.21.15.90:52602) 2024-04-25T11:43:10.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.195+0000 7fcb76ffd640 1 -- 172.21.15.90:0/1200010741 learned_addr learned my addr 172.21.15.90:0/1200010741 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:10.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.195+0000 7fcb77fff640 1 --2- 172.21.15.90:0/1200010741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563dc9044390 0x563dc940b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.196+0000 7fcb777fe640 1 --2- 172.21.15.90:0/1200010741 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563dc901b4d0 0x563dc93f1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.196+0000 7fcb76ffd640 1 -- 172.21.15.90:0/1200010741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563dc9044390 msgr2=0x563dc940b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:10.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.196+0000 7fcb76ffd640 1 --2- 172.21.15.90:0/1200010741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563dc9044390 0x563dc940b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.196+0000 7fcb76ffd640 1 -- 172.21.15.90:0/1200010741 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563dc901b4d0 msgr2=0x563dc93f1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:10.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.196+0000 7fcb76ffd640 1 --2- 172.21.15.90:0/1200010741 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563dc901b4d0 0x563dc93f1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.196+0000 7fcb76ffd640 1 -- 172.21.15.90:0/1200010741 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563dc901c0f0 con 0x563dc90236b0 2024-04-25T11:43:10.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.196+0000 7fcb76ffd640 1 --2- 172.21.15.90:0/1200010741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563dc90236b0 0x563dc940ac40 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c009da0 tx=0x7fcb6c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fd8f107580e3da4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:10.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.196+0000 7fcb5d7fa640 1 -- 172.21.15.90:0/1200010741 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb6c004230 con 0x563dc90236b0 2024-04-25T11:43:10.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.196+0000 7fcb5d7fa640 1 -- 172.21.15.90:0/1200010741 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb6c0043d0 con 0x563dc90236b0 2024-04-25T11:43:10.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.196+0000 7fcb5d7fa640 1 -- 172.21.15.90:0/1200010741 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb6c004e40 con 0x563dc90236b0 2024-04-25T11:43:10.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.197+0000 7fcb85044740 1 -- 172.21.15.90:0/1200010741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563dc90236b0 msgr2=0x563dc940ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.197+0000 7fcb85044740 1 --2- 172.21.15.90:0/1200010741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563dc90236b0 0x563dc940ac40 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c009da0 tx=0x7fcb6c005f90 comp rx=0 tx=0).stop 2024-04-25T11:43:10.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.197+0000 7fcb85044740 1 -- 172.21.15.90:0/1200010741 shutdown_connections 2024-04-25T11:43:10.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.197+0000 7fcb85044740 1 --2- 172.21.15.90:0/1200010741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563dc9044390 0x563dc940b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.197+0000 7fcb85044740 1 --2- 172.21.15.90:0/1200010741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563dc90236b0 0x563dc940ac40 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.197+0000 7fcb85044740 1 --2- 172.21.15.90:0/1200010741 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563dc901b4d0 0x563dc93f1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.197+0000 7fcb85044740 1 -- 172.21.15.90:0/1200010741 >> 172.21.15.90:0/1200010741 conn(0x563dc90384e0 msgr2=0x563dc93f0a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:10.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.197+0000 7fcb85044740 1 -- 172.21.15.90:0/1200010741 shutdown_connections 2024-04-25T11:43:10.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.197+0000 7fcb85044740 1 -- 172.21.15.90:0/1200010741 wait complete. 2024-04-25T11:43:10.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.198+0000 7fcb85044740 1 -- start start 2024-04-25T11:43:10.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.198+0000 7fcb85044740 1 Processor -- start 2024-04-25T11:43:10.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.198+0000 7fcb85044740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563dc901b4d0 0x563dc93fb670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.198+0000 7fcb85044740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563dc90236b0 0x563dc93fbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.198+0000 7fcb85044740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563dc9044390 0x563dc93f6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.198+0000 7fcb85044740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcb781aef60 con 0x563dc9044390 2024-04-25T11:43:10.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.198+0000 7fcb85044740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fcb781ae890 con 0x563dc90236b0 2024-04-25T11:43:10.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.198+0000 7fcb85044740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcb781abaf0 con 0x563dc901b4d0 2024-04-25T11:43:10.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.198+0000 7fcb85044740 10 client.0 _ll_register_callbacks cb 0x7fcb70009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.198+0000 7fcb777fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563dc901b4d0 0x563dc93fb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.198+0000 7fcb777fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563dc901b4d0 0x563dc93fb670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:44872/0 (socket says 172.21.15.90:44872) 2024-04-25T11:43:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.198+0000 7fcb777fe640 1 -- 172.21.15.90:0/1745429524 learned_addr learned my addr 172.21.15.90:0/1745429524 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.198+0000 7fcb77fff640 1 --2- 172.21.15.90:0/1745429524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563dc9044390 0x563dc93f6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.199+0000 7fcb76ffd640 1 --2- 172.21.15.90:0/1745429524 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563dc90236b0 0x563dc93fbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.199+0000 7fcb777fe640 1 -- 172.21.15.90:0/1745429524 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563dc90236b0 msgr2=0x563dc93fbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.199+0000 7fcb777fe640 1 --2- 172.21.15.90:0/1745429524 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563dc90236b0 0x563dc93fbbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.199+0000 7fcb777fe640 1 -- 172.21.15.90:0/1745429524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563dc9044390 msgr2=0x563dc93f6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.199+0000 7fcb777fe640 1 --2- 172.21.15.90:0/1745429524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563dc9044390 0x563dc93f6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.199+0000 7fcb777fe640 1 -- 172.21.15.90:0/1745429524 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563dc930b4b0 con 0x563dc901b4d0 2024-04-25T11:43:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.199+0000 7fcb76ffd640 1 --2- 172.21.15.90:0/1745429524 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563dc90236b0 0x563dc93fbbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.199+0000 7fcb777fe640 1 --2- 172.21.15.90:0/1745429524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563dc901b4d0 0x563dc93fb670 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcb680fd0c0 tx=0x7fcb680c7200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f66de9297a2a5e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.199+0000 7fcb56ffd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:43:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.199+0000 7fcb56ffd640 1 -- 172.21.15.90:0/1745429524 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb68049040 con 0x563dc901b4d0 2024-04-25T11:43:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.199+0000 7fcb56ffd640 1 -- 172.21.15.90:0/1745429524 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7fcb680c7a20 con 0x563dc901b4d0 2024-04-25T11:43:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.199+0000 7fcb85044740 1 -- 172.21.15.90:0/1745429524 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb6c0043d0 con 0x563dc901b4d0 2024-04-25T11:43:10.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.199+0000 7fcb56ffd640 1 -- 172.21.15.90:0/1745429524 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcb680c7390 con 0x563dc901b4d0 2024-04-25T11:43:10.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.199+0000 7fcb85044740 10 client.26782 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:10.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.199+0000 7fcb85044740 1 -- 172.21.15.90:0/1745429524 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563dc930b4b0 con 0x563dc901b4d0 2024-04-25T11:43:10.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.200+0000 7fcb56ffd640 1 -- 172.21.15.90:0/1745429524 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb6c0043d0 con 0x563dc901b4d0 2024-04-25T11:43:10.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.200+0000 7fcb56ffd640 1 -- 172.21.15.90:0/1745429524 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fcb680cfee0 con 0x563dc901b4d0 2024-04-25T11:43:10.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.200+0000 7fcb85044740 10 client.26782 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:10.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.200+0000 7fcb85044740 10 client.26782 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:43:10.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.200+0000 7fcb85044740 20 client.26782 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:10.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.200+0000 7fcb85044740 1 --2- 172.21.15.90:0/1745429524 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563dc8fd83a0 0x563dc946f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.200+0000 7fcb85044740 4 client.26782 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:43:10.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.200+0000 7fcb85044740 1 -- 172.21.15.90:0/1745429524 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563dc9352800 con 0x563dc8fd83a0 2024-04-25T11:43:10.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.200+0000 7fcb76ffd640 1 --2- 172.21.15.90:0/1745429524 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563dc8fd83a0 0x563dc946f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.201+0000 7fcb56ffd640 10 client.26782 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:10.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.201+0000 7fcb76ffd640 1 --2- 172.21.15.90:0/1745429524 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563dc8fd83a0 0x563dc946f3d0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3803d6e828412c1 server_cookie=936855d10016efc in_seq=0 out_seq=0 2024-04-25T11:43:10.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.201+0000 7fcb56ffd640 1 -- 172.21.15.90:0/1745429524 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563dc9352800 con 0x563dc8fd83a0 2024-04-25T11:43:10.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.201+0000 7fcb56ffd640 10 client.26782 handle_command_reply: tid=0 2024-04-25T11:43:10.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.201+0000 7fcb85044740 1 client.26782 shutdown 2024-04-25T11:43:10.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.201+0000 7fcb85044740 1 -- 172.21.15.90:0/1745429524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563dc901b4d0 msgr2=0x563dc93fb670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.201+0000 7fcb85044740 1 --2- 172.21.15.90:0/1745429524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563dc901b4d0 0x563dc93fb670 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcb680fd0c0 tx=0x7fcb680c7200 comp rx=0 tx=0).stop 2024-04-25T11:43:10.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.201+0000 7fcb85044740 1 -- 172.21.15.90:0/1745429524 shutdown_connections 2024-04-25T11:43:10.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.201+0000 7fcb85044740 1 --2- 172.21.15.90:0/1745429524 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563dc8fd83a0 0x563dc946f3d0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.201+0000 7fcb85044740 1 --2- 172.21.15.90:0/1745429524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563dc9044390 0x563dc93f6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.201+0000 7fcb56ffd640 0 client.26782 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:10.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.201+0000 7fcb85044740 1 --2- 172.21.15.90:0/1745429524 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563dc90236b0 0x563dc93fbbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.201+0000 7fcb85044740 1 --2- 172.21.15.90:0/1745429524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563dc901b4d0 0x563dc93fb670 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.201+0000 7fcb85044740 1 -- 172.21.15.90:0/1745429524 >> 172.21.15.90:0/1745429524 conn(0x563dc90384e0 msgr2=0x563dc93f09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:10.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.201+0000 7fcb85044740 1 -- 172.21.15.90:0/1745429524 shutdown_connections 2024-04-25T11:43:10.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.201+0000 7fcb85044740 1 -- 172.21.15.90:0/1745429524 wait complete. 2024-04-25T11:43:10.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.202+0000 7fcb85044740 20 client.26782 trim_cache size 0 max 16384 2024-04-25T11:43:10.192 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:43:10.192 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-25T11:43:10.192 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-25T11:43:10.192 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:43:10.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.202+0000 7fcb7e18f640 1 -- 172.21.15.90:0/1587244308 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcb600afc60 msgr2=0x7fcb600b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.202+0000 7fcb7e18f640 1 --2- 172.21.15.90:0/1587244308 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcb600afc60 0x7fcb600b2170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcb70005c40 tx=0x7fcb70004a40 comp rx=0 tx=0).stop 2024-04-25T11:43:10.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.202+0000 7fcb7e18f640 1 -- 172.21.15.90:0/1587244308 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb78102420 msgr2=0x7fcb7819b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.202+0000 7fcb7e18f640 1 --2- 172.21.15.90:0/1587244308 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb78102420 0x7fcb7819b570 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fcb68005940 tx=0x7fcb680040c0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.202+0000 7fcb7e18f640 1 -- 172.21.15.90:0/1587244308 shutdown_connections 2024-04-25T11:43:10.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.202+0000 7fcb7e18f640 1 --2- 172.21.15.90:0/1587244308 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcb600afc60 0x7fcb600b2170 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.202+0000 7fcb7e18f640 1 --2- 172.21.15.90:0/1587244308 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb78107470 0x7fcb7819bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.202+0000 7fcb7e18f640 1 --2- 172.21.15.90:0/1587244308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb781069b0 0x7fcb7819bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.202+0000 7fcb7e18f640 1 --2- 172.21.15.90:0/1587244308 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb78102420 0x7fcb7819b570 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.202+0000 7fcb7e18f640 1 -- 172.21.15.90:0/1587244308 >> 172.21.15.90:0/1587244308 conn(0x7fcb78077920 msgr2=0x7fcb7810d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:10.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.202+0000 7fcb7e18f640 1 -- 172.21.15.90:0/1587244308 shutdown_connections 2024-04-25T11:43:10.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.203+0000 7fcb7e18f640 1 -- 172.21.15.90:0/1587244308 wait complete. 2024-04-25T11:43:10.200 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T11:43:10.201 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:43:10.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:10 smithi138 ceph-mon[32328]: pgmap v248: 97 pgs: 4 active+clean+snaptrim, 83 active+clean, 10 active+clean+snaptrim_wait; 60 GiB data, 185 GiB used, 888 GiB / 1.0 TiB avail; 13 KiB/s rd, 6.9 MiB/s wr, 99 op/s 2024-04-25T11:43:10.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.330+0000 7f9d88d64640 1 -- 172.21.15.90:0/1275768252 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d8410d380 msgr2=0x7f9d8410f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.330+0000 7f9d88d64640 1 --2- 172.21.15.90:0/1275768252 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d8410d380 0x7f9d8410f7a0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c009ca0 tx=0x7f9d7c02a8f0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.330+0000 7f9d88d64640 1 -- 172.21.15.90:0/1275768252 shutdown_connections 2024-04-25T11:43:10.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.330+0000 7f9d88d64640 1 --2- 172.21.15.90:0/1275768252 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d8410d380 0x7f9d8410f7a0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.330+0000 7f9d88d64640 1 --2- 172.21.15.90:0/1275768252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d84102af0 0x7f9d84102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.330+0000 7f9d88d64640 1 --2- 172.21.15.90:0/1275768252 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d84108a20 0x7f9d841025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.330+0000 7f9d88d64640 1 -- 172.21.15.90:0/1275768252 >> 172.21.15.90:0/1275768252 conn(0x7f9d84078110 msgr2=0x7f9d84078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:10.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.330+0000 7f9d88d64640 1 -- 172.21.15.90:0/1275768252 shutdown_connections 2024-04-25T11:43:10.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.331+0000 7f9d88d64640 1 -- 172.21.15.90:0/1275768252 wait complete. 2024-04-25T11:43:10.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.331+0000 7f9d88d64640 1 Processor -- start 2024-04-25T11:43:10.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.331+0000 7f9d88d64640 1 -- start start 2024-04-25T11:43:10.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.332+0000 7f9d88d64640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d84102af0 0x7f9d8419b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.332+0000 7f9d88d64640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d84108a20 0x7f9d8419bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.332+0000 7f9d88d64640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d8410d380 0x7f9d8419c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.332+0000 7f9d88d64640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9d84112ce0 con 0x7f9d84108a20 2024-04-25T11:43:10.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.332+0000 7f9d88d64640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9d84112b60 con 0x7f9d8410d380 2024-04-25T11:43:10.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.332+0000 7f9d88d64640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9d84112e60 con 0x7f9d84102af0 2024-04-25T11:43:10.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.332+0000 7f9d7bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d84108a20 0x7f9d8419bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.332+0000 7f9d7bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d84108a20 0x7f9d8419bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52630/0 (socket says 172.21.15.90:52630) 2024-04-25T11:43:10.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.332+0000 7f9d7bfff640 1 -- 172.21.15.90:0/1165227036 learned_addr learned my addr 172.21.15.90:0/1165227036 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:10.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.332+0000 7f9d8259c640 1 --2- 172.21.15.90:0/1165227036 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d84102af0 0x7f9d8419b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.332+0000 7f9d7bfff640 1 -- 172.21.15.90:0/1165227036 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d84102af0 msgr2=0x7f9d8419b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.332+0000 7f9d7bfff640 1 --2- 172.21.15.90:0/1165227036 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d84102af0 0x7f9d8419b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.332+0000 7f9d7bfff640 1 -- 172.21.15.90:0/1165227036 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d8410d380 msgr2=0x7f9d8419c0a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:43:10.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.332+0000 7f9d7bfff640 1 --2- 172.21.15.90:0/1165227036 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d8410d380 0x7f9d8419c0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.332+0000 7f9d7bfff640 1 -- 172.21.15.90:0/1165227036 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d841a6940 con 0x7f9d84108a20 2024-04-25T11:43:10.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.332+0000 7f9d8259c640 1 --2- 172.21.15.90:0/1165227036 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d84102af0 0x7f9d8419b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:43:10.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.332+0000 7f9d7bfff640 1 --2- 172.21.15.90:0/1165227036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d84108a20 0x7f9d8419bb40 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00b450 tx=0x7f9d6c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:10.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.332+0000 7f9d7b7fe640 1 -- 172.21.15.90:0/1165227036 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c015020 con 0x7f9d84108a20 2024-04-25T11:43:10.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.332+0000 7f9d7b7fe640 1 -- 172.21.15.90:0/1165227036 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d6c011d40 con 0x7f9d84108a20 2024-04-25T11:43:10.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.332+0000 7f9d88d64640 1 -- 172.21.15.90:0/1165227036 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d841a6c30 con 0x7f9d84108a20 2024-04-25T11:43:10.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.332+0000 7f9d7b7fe640 1 -- 172.21.15.90:0/1165227036 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c00f340 con 0x7f9d84108a20 2024-04-25T11:43:10.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.332+0000 7f9d88d64640 1 -- 172.21.15.90:0/1165227036 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d841a7130 con 0x7f9d84108a20 2024-04-25T11:43:10.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.333+0000 7f9d88d64640 1 -- 172.21.15.90:0/1165227036 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d84103cd0 con 0x7f9d84108a20 2024-04-25T11:43:10.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.335+0000 7f9d7b7fe640 1 -- 172.21.15.90:0/1165227036 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f9d6c018020 con 0x7f9d84108a20 2024-04-25T11:43:10.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.336+0000 7f9d7b7fe640 1 --2- 172.21.15.90:0/1165227036 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9d700afc60 0x7f9d700b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.336+0000 7f9d8259c640 1 --2- 172.21.15.90:0/1165227036 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9d700afc60 0x7f9d700b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.336+0000 7f9d7b7fe640 1 -- 172.21.15.90:0/1165227036 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7f9d6c0deff0 con 0x7f9d84108a20 2024-04-25T11:43:10.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.336+0000 7f9d8259c640 1 --2- 172.21.15.90:0/1165227036 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9d700afc60 0x7f9d700b2170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9d84103a20 tx=0x7f9d74008040 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:10.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.340+0000 7f9d7b7fe640 1 -- 172.21.15.90:0/1165227036 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9d6c01f020 con 0x7f9d84108a20 2024-04-25T11:43:10.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:10 smithi156 ceph-mon[33045]: pgmap v248: 97 pgs: 4 active+clean+snaptrim, 83 active+clean, 10 active+clean+snaptrim_wait; 60 GiB data, 185 GiB used, 888 GiB / 1.0 TiB avail; 13 KiB/s rd, 6.9 MiB/s wr, 99 op/s 2024-04-25T11:43:10.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:10 smithi090 ceph-mon[29575]: pgmap v248: 97 pgs: 4 active+clean+snaptrim, 83 active+clean, 10 active+clean+snaptrim_wait; 60 GiB data, 185 GiB used, 888 GiB / 1.0 TiB avail; 13 KiB/s rd, 6.9 MiB/s wr, 99 op/s 2024-04-25T11:43:10.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.587+0000 7f9d88d64640 1 -- 172.21.15.90:0/1165227036 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9d8406afa0 con 0x7f9d84108a20 2024-04-25T11:43:10.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.588+0000 7f9d7b7fe640 1 -- 172.21.15.90:0/1165227036 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11930 (secure 0 0 0) 0x7f9d8406afa0 con 0x7f9d84108a20 2024-04-25T11:43:10.574 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:43:10.574 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:39:20.469254+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24458,24431,24499,24475,24524]},"id":1}]} 2024-04-25T11:43:10.575 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 28 2024-04-25T11:43:10.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.590+0000 7f9d88d64640 1 -- 172.21.15.90:0/1165227036 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9d700afc60 msgr2=0x7f9d700b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.590+0000 7f9d88d64640 1 --2- 172.21.15.90:0/1165227036 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9d700afc60 0x7f9d700b2170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9d84103a20 tx=0x7f9d74008040 comp rx=0 tx=0).stop 2024-04-25T11:43:10.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.590+0000 7f9d88d64640 1 -- 172.21.15.90:0/1165227036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d84108a20 msgr2=0x7f9d8419bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.590+0000 7f9d88d64640 1 --2- 172.21.15.90:0/1165227036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d84108a20 0x7f9d8419bb40 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00b450 tx=0x7f9d6c00b910 comp rx=0 tx=0).stop 2024-04-25T11:43:10.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.590+0000 7f9d88d64640 1 -- 172.21.15.90:0/1165227036 shutdown_connections 2024-04-25T11:43:10.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.590+0000 7f9d88d64640 1 --2- 172.21.15.90:0/1165227036 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9d700afc60 0x7f9d700b2170 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.590+0000 7f9d88d64640 1 --2- 172.21.15.90:0/1165227036 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d8410d380 0x7f9d8419c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.590+0000 7f9d88d64640 1 --2- 172.21.15.90:0/1165227036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d84108a20 0x7f9d8419bb40 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.590+0000 7f9d88d64640 1 --2- 172.21.15.90:0/1165227036 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d84102af0 0x7f9d8419b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.590+0000 7f9d88d64640 1 -- 172.21.15.90:0/1165227036 >> 172.21.15.90:0/1165227036 conn(0x7f9d84078110 msgr2=0x7f9d841040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:10.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.591+0000 7f9d88d64640 1 -- 172.21.15.90:0/1165227036 shutdown_connections 2024-04-25T11:43:10.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.591+0000 7f9d88d64640 1 -- 172.21.15.90:0/1165227036 wait complete. 2024-04-25T11:43:10.590 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T11:43:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.722+0000 7f62ad09a640 1 -- 172.21.15.90:0/3111458399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62a8102af0 msgr2=0x7f62a8102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.722+0000 7f62ad09a640 1 --2- 172.21.15.90:0/3111458399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62a8102af0 0x7f62a8102f00 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f6290009900 tx=0x7f629002a410 comp rx=0 tx=0).stop 2024-04-25T11:43:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.722+0000 7f62ad09a640 1 -- 172.21.15.90:0/3111458399 shutdown_connections 2024-04-25T11:43:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.722+0000 7f62ad09a640 1 --2- 172.21.15.90:0/3111458399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62a810d380 0x7f62a810f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.722+0000 7f62ad09a640 1 --2- 172.21.15.90:0/3111458399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62a8102af0 0x7f62a8102f00 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.722+0000 7f62ad09a640 1 --2- 172.21.15.90:0/3111458399 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f62a8108a20 0x7f62a81025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.722+0000 7f62ad09a640 1 -- 172.21.15.90:0/3111458399 >> 172.21.15.90:0/3111458399 conn(0x7f62a8078100 msgr2=0x7f62a8078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.722+0000 7f62ad09a640 1 -- 172.21.15.90:0/3111458399 shutdown_connections 2024-04-25T11:43:10.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.722+0000 7f62ad09a640 1 -- 172.21.15.90:0/3111458399 wait complete. 2024-04-25T11:43:10.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.723+0000 7f62ad09a640 1 Processor -- start 2024-04-25T11:43:10.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.723+0000 7f62ad09a640 1 -- start start 2024-04-25T11:43:10.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.723+0000 7f62ad09a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f62a8102af0 0x7f62a819b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.723+0000 7f62ad09a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62a8108a20 0x7f62a819ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.723+0000 7f62ad09a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62a810d380 0x7f62a819bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.723+0000 7f62ad09a640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f62a8112be0 con 0x7f62a810d380 2024-04-25T11:43:10.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.723+0000 7f62ad09a640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f62a8112a60 con 0x7f62a8102af0 2024-04-25T11:43:10.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.723+0000 7f62ad09a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f62a8112d60 con 0x7f62a8108a20 2024-04-25T11:43:10.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.723+0000 7f62a759e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62a810d380 0x7f62a819bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.723+0000 7f62a759e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62a810d380 0x7f62a819bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52652/0 (socket says 172.21.15.90:52652) 2024-04-25T11:43:10.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.723+0000 7f62a759e640 1 -- 172.21.15.90:0/1634618827 learned_addr learned my addr 172.21.15.90:0/1634618827 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:10.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.723+0000 7f62a6d9d640 1 --2- 172.21.15.90:0/1634618827 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f62a8102af0 0x7f62a819b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.723+0000 7f629e59c640 1 --2- 172.21.15.90:0/1634618827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62a8108a20 0x7f62a819ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.724+0000 7f62a6d9d640 1 -- 172.21.15.90:0/1634618827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62a8108a20 msgr2=0x7f62a819ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.724+0000 7f62a6d9d640 1 --2- 172.21.15.90:0/1634618827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62a8108a20 0x7f62a819ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.724+0000 7f62a6d9d640 1 -- 172.21.15.90:0/1634618827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62a810d380 msgr2=0x7f62a819bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.724+0000 7f62a6d9d640 1 --2- 172.21.15.90:0/1634618827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62a810d380 0x7f62a819bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.724+0000 7f62a6d9d640 1 -- 172.21.15.90:0/1634618827 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62a81a6880 con 0x7f62a8102af0 2024-04-25T11:43:10.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.724+0000 7f62a759e640 1 --2- 172.21.15.90:0/1634618827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62a810d380 0x7f62a819bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:43:10.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.724+0000 7f62a6d9d640 1 --2- 172.21.15.90:0/1634618827 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f62a8102af0 0x7f62a819b540 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f629800e9c0 tx=0x7f629800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:10.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.724+0000 7f62a4d99640 1 -- 172.21.15.90:0/1634618827 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f629800ccc0 con 0x7f62a8102af0 2024-04-25T11:43:10.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.724+0000 7f62ad09a640 1 -- 172.21.15.90:0/1634618827 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62a81a6b70 con 0x7f62a8102af0 2024-04-25T11:43:10.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.724+0000 7f62a4d99640 1 -- 172.21.15.90:0/1634618827 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62980108e0 con 0x7f62a8102af0 2024-04-25T11:43:10.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.724+0000 7f62a4d99640 1 -- 172.21.15.90:0/1634618827 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62980187a0 con 0x7f62a8102af0 2024-04-25T11:43:10.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.724+0000 7f62ad09a640 1 -- 172.21.15.90:0/1634618827 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62a81a70e0 con 0x7f62a8102af0 2024-04-25T11:43:10.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.727+0000 7f62a4d99640 1 -- 172.21.15.90:0/1634618827 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f6298018940 con 0x7f62a8102af0 2024-04-25T11:43:10.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.727+0000 7f62b4020740 1 Processor -- start 2024-04-25T11:43:10.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.727+0000 7f62b4020740 1 -- start start 2024-04-25T11:43:10.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.727+0000 7f62b4020740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca8066390 0x560ca8407810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.727+0000 7f62b4020740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ca803d4d0 0x560ca8407d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.727+0000 7f62b4020740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ca80456b0 0x560ca84146a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.727+0000 7f62b4020740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560ca836cd80 con 0x560ca8066390 2024-04-25T11:43:10.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.727+0000 7f62b4020740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x560ca8326a20 con 0x560ca80456b0 2024-04-25T11:43:10.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.727+0000 7f62b4020740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560ca8368d40 con 0x560ca803d4d0 2024-04-25T11:43:10.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.727+0000 7f62a759e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca8066390 0x560ca8407810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.727+0000 7f62a759e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca8066390 0x560ca8407810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52660/0 (socket says 172.21.15.90:52660) 2024-04-25T11:43:10.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.727+0000 7f62a759e640 1 -- 172.21.15.90:0/2712852845 learned_addr learned my addr 172.21.15.90:0/2712852845 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:10.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.727+0000 7f629e59c640 1 --2- 172.21.15.90:0/2712852845 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ca80456b0 0x560ca84146a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.728+0000 7f62a6d9d640 1 --2- 172.21.15.90:0/2712852845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ca803d4d0 0x560ca8407d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.728+0000 7f62a4d99640 1 --2- 172.21.15.90:0/1634618827 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f62940afc60 0x7f62940b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.728+0000 7f62a759e640 1 -- 172.21.15.90:0/2712852845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ca803d4d0 msgr2=0x560ca8407d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:10.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.728+0000 7f62a759e640 1 --2- 172.21.15.90:0/2712852845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ca803d4d0 0x560ca8407d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.728+0000 7f62a759e640 1 -- 172.21.15.90:0/2712852845 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ca80456b0 msgr2=0x560ca84146a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:10.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.728+0000 7f62a759e640 1 --2- 172.21.15.90:0/2712852845 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ca80456b0 0x560ca84146a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.728+0000 7f62a759e640 1 -- 172.21.15.90:0/2712852845 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ca8331c30 con 0x560ca8066390 2024-04-25T11:43:10.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.728+0000 7f629e59c640 1 --2- 172.21.15.90:0/2712852845 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ca80456b0 0x560ca84146a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:43:10.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.728+0000 7f62a4d99640 1 -- 172.21.15.90:0/1634618827 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7f62980df4b0 con 0x7f62a8102af0 2024-04-25T11:43:10.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.728+0000 7f62a759e640 1 --2- 172.21.15.90:0/1634618827 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f62940afc60 0x7f62940b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.728+0000 7f62a759e640 1 --2- 172.21.15.90:0/2712852845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca8066390 0x560ca8407810 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f62a0011910 tx=0x7f62a0011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=845366006d4fcf46 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:10.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.728+0000 7f629d59a640 1 -- 172.21.15.90:0/2712852845 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a0017050 con 0x560ca8066390 2024-04-25T11:43:10.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.728+0000 7f629d59a640 1 -- 172.21.15.90:0/2712852845 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62a001bb90 con 0x560ca8066390 2024-04-25T11:43:10.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.728+0000 7f629d59a640 1 -- 172.21.15.90:0/2712852845 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a001a360 con 0x560ca8066390 2024-04-25T11:43:10.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.728+0000 7f62a759e640 1 --2- 172.21.15.90:0/1634618827 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f62940afc60 0x7f62940b2170 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f62a0023000 tx=0x7f62a0024000 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:10.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.729+0000 7f62b4020740 1 -- 172.21.15.90:0/2712852845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca8066390 msgr2=0x560ca8407810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.729+0000 7f62b4020740 1 --2- 172.21.15.90:0/2712852845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca8066390 0x560ca8407810 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f62a0011910 tx=0x7f62a0011dd0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.729+0000 7f62b4020740 1 -- 172.21.15.90:0/2712852845 shutdown_connections 2024-04-25T11:43:10.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.729+0000 7f62b4020740 1 --2- 172.21.15.90:0/2712852845 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ca80456b0 0x560ca84146a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.729+0000 7f62b4020740 1 --2- 172.21.15.90:0/2712852845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ca803d4d0 0x560ca8407d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.729+0000 7f62b4020740 1 --2- 172.21.15.90:0/2712852845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca8066390 0x560ca8407810 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.729+0000 7f62b4020740 1 -- 172.21.15.90:0/2712852845 >> 172.21.15.90:0/2712852845 conn(0x560ca805a4e0 msgr2=0x560ca8403fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:10.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.729+0000 7f62b4020740 1 -- 172.21.15.90:0/2712852845 shutdown_connections 2024-04-25T11:43:10.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.729+0000 7f62b4020740 1 -- 172.21.15.90:0/2712852845 wait complete. 2024-04-25T11:43:10.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.730+0000 7f62b4020740 1 -- start start 2024-04-25T11:43:10.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.730+0000 7f62b4020740 1 Processor -- start 2024-04-25T11:43:10.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.730+0000 7f62b4020740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca803d4d0 0x560ca84121b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.730+0000 7f62b4020740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ca80456b0 0x560ca84126f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.730+0000 7f62b4020740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ca8066390 0x560ca841e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.730+0000 7f62b4020740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560ca8414d90 con 0x560ca803d4d0 2024-04-25T11:43:10.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.730+0000 7f62b4020740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x560ca82f3dc0 con 0x560ca8066390 2024-04-25T11:43:10.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.730+0000 7f62b4020740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560ca8414f10 con 0x560ca80456b0 2024-04-25T11:43:10.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.730+0000 7f62b4020740 10 client.0 _ll_register_callbacks cb 0x560ca830f600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:10.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.730+0000 7f62a6d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca803d4d0 0x560ca84121b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.730+0000 7f62a6d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca803d4d0 0x560ca84121b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52676/0 (socket says 172.21.15.90:52676) 2024-04-25T11:43:10.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.730+0000 7f62a6d9d640 1 -- 172.21.15.90:0/1459983299 learned_addr learned my addr 172.21.15.90:0/1459983299 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:10.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.730+0000 7f62a759e640 1 --2- 172.21.15.90:0/1459983299 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ca8066390 0x560ca841e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.730+0000 7f629e59c640 1 --2- 172.21.15.90:0/1459983299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ca80456b0 0x560ca84126f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.730+0000 7f62a6d9d640 1 -- 172.21.15.90:0/1459983299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ca80456b0 msgr2=0x560ca84126f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:10.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.730+0000 7f62a6d9d640 1 --2- 172.21.15.90:0/1459983299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ca80456b0 0x560ca84126f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.730+0000 7f62a6d9d640 1 -- 172.21.15.90:0/1459983299 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ca8066390 msgr2=0x560ca841e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:10.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.730+0000 7f62a6d9d640 1 --2- 172.21.15.90:0/1459983299 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ca8066390 0x560ca841e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.730+0000 7f62a6d9d640 1 -- 172.21.15.90:0/1459983299 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ca807e3f0 con 0x560ca803d4d0 2024-04-25T11:43:10.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.731+0000 7f62a6d9d640 1 --2- 172.21.15.90:0/1459983299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca803d4d0 0x560ca84121b0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f62980dcbb0 tx=0x7f629800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d17b44a6a1c8c1e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:10.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.731+0000 7f628e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:10.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.731+0000 7f628e7fc640 1 -- 172.21.15.90:0/1459983299 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62980b0400 con 0x560ca803d4d0 2024-04-25T11:43:10.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.731+0000 7f628e7fc640 1 -- 172.21.15.90:0/1459983299 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7f62980900b0 con 0x560ca803d4d0 2024-04-25T11:43:10.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.731+0000 7f62b4020740 1 -- 172.21.15.90:0/1459983299 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62a001bb90 con 0x560ca803d4d0 2024-04-25T11:43:10.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.731+0000 7f628e7fc640 1 -- 172.21.15.90:0/1459983299 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f62980a7650 con 0x560ca803d4d0 2024-04-25T11:43:10.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.731+0000 7f62b4020740 10 client.18054 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:10.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.731+0000 7f62b4020740 1 -- 172.21.15.90:0/1459983299 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560ca81487d0 con 0x560ca803d4d0 2024-04-25T11:43:10.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.731+0000 7f628e7fc640 1 -- 172.21.15.90:0/1459983299 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62a001bb90 con 0x560ca803d4d0 2024-04-25T11:43:10.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.731+0000 7f628e7fc640 1 -- 172.21.15.90:0/1459983299 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f62980b0000 con 0x560ca803d4d0 2024-04-25T11:43:10.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.731+0000 7f62b4020740 10 client.18054 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:10.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.731+0000 7f62b4020740 10 client.18054 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:43:10.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.731+0000 7f62b4020740 20 client.18054 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:10.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.731+0000 7f62b4020740 1 --2- 172.21.15.90:0/1459983299 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x560ca8024f60 0x560ca8424650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.731+0000 7f62b4020740 4 client.18054 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:43:10.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.731+0000 7f62b4020740 1 -- 172.21.15.90:0/1459983299 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560ca837a990 con 0x560ca8024f60 2024-04-25T11:43:10.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.732+0000 7f629e59c640 1 --2- 172.21.15.90:0/1459983299 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x560ca8024f60 0x560ca8424650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.732+0000 7f628e7fc640 10 client.18054 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:43:10.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.732+0000 7f629e59c640 1 --2- 172.21.15.90:0/1459983299 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x560ca8024f60 0x560ca8424650 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3e5000e2f3952dd5 server_cookie=69cc521ea3170d15 in_seq=0 out_seq=0 2024-04-25T11:43:10.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.733+0000 7f628e7fc640 1 -- 172.21.15.90:0/1459983299 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560ca837a990 con 0x560ca8024f60 2024-04-25T11:43:10.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.733+0000 7f628e7fc640 10 client.18054 handle_command_reply: tid=0 2024-04-25T11:43:10.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.733+0000 7f62b4020740 1 client.18054 shutdown 2024-04-25T11:43:10.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.733+0000 7f62b4020740 1 -- 172.21.15.90:0/1459983299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca803d4d0 msgr2=0x560ca84121b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.733+0000 7f62b4020740 1 --2- 172.21.15.90:0/1459983299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca803d4d0 0x560ca84121b0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f62980dcbb0 tx=0x7f629800ce60 comp rx=0 tx=0).stop 2024-04-25T11:43:10.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.733+0000 7f62b4020740 1 -- 172.21.15.90:0/1459983299 shutdown_connections 2024-04-25T11:43:10.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.733+0000 7f62b4020740 1 --2- 172.21.15.90:0/1459983299 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x560ca8024f60 0x560ca8424650 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.733+0000 7f62b4020740 1 --2- 172.21.15.90:0/1459983299 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ca8066390 0x560ca841e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.733+0000 7f62b4020740 1 --2- 172.21.15.90:0/1459983299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ca80456b0 0x560ca84126f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.733+0000 7f62b4020740 1 --2- 172.21.15.90:0/1459983299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca803d4d0 0x560ca84121b0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.733+0000 7f628e7fc640 0 client.18054 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:43:10.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.733+0000 7f62b4020740 1 -- 172.21.15.90:0/1459983299 >> 172.21.15.90:0/1459983299 conn(0x560ca805a4e0 msgr2=0x560ca8409530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:10.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.733+0000 7f62b4020740 1 -- 172.21.15.90:0/1459983299 shutdown_connections 2024-04-25T11:43:10.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.733+0000 7f62b4020740 1 -- 172.21.15.90:0/1459983299 wait complete. 2024-04-25T11:43:10.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.733+0000 7f62b4020740 20 client.18054 trim_cache size 0 max 16384 2024-04-25T11:43:10.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.747+0000 7f62b4020740 1 Processor -- start 2024-04-25T11:43:10.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.747+0000 7f62b4020740 1 -- start start 2024-04-25T11:43:10.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.747+0000 7f62b4020740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ca803d4d0 0x560ca84112a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.747+0000 7f62b4020740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca80456b0 0x560ca8412840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.747+0000 7f62b4020740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ca8066390 0x560ca841de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.747+0000 7f62b4020740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f62a81afb90 con 0x560ca80456b0 2024-04-25T11:43:10.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.747+0000 7f62b4020740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f62a81b16d0 con 0x560ca8066390 2024-04-25T11:43:10.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.747+0000 7f62b4020740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f62a81b0260 con 0x560ca803d4d0 2024-04-25T11:43:10.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.747+0000 7f629e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca80456b0 0x560ca8412840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.748+0000 7f62a759e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ca8066390 0x560ca841de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.747+0000 7f629e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca80456b0 0x560ca8412840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52686/0 (socket says 172.21.15.90:52686) 2024-04-25T11:43:10.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.748+0000 7f629e59c640 1 -- 172.21.15.90:0/2884860399 learned_addr learned my addr 172.21.15.90:0/2884860399 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:10.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.748+0000 7f62a6d9d640 1 --2- 172.21.15.90:0/2884860399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ca803d4d0 0x560ca84112a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.748+0000 7f629e59c640 1 -- 172.21.15.90:0/2884860399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ca803d4d0 msgr2=0x560ca84112a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:10.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.748+0000 7f629e59c640 1 --2- 172.21.15.90:0/2884860399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ca803d4d0 0x560ca84112a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.748+0000 7f629e59c640 1 -- 172.21.15.90:0/2884860399 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ca8066390 msgr2=0x560ca841de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:10.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.748+0000 7f629e59c640 1 --2- 172.21.15.90:0/2884860399 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ca8066390 0x560ca841de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.748+0000 7f629e59c640 1 -- 172.21.15.90:0/2884860399 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ca8410520 con 0x560ca80456b0 2024-04-25T11:43:10.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.748+0000 7f62a6d9d640 1 --2- 172.21.15.90:0/2884860399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ca803d4d0 0x560ca84112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:43:10.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.748+0000 7f62a759e640 1 --2- 172.21.15.90:0/2884860399 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ca8066390 0x560ca841de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:43:10.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.748+0000 7f629e59c640 1 --2- 172.21.15.90:0/2884860399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca80456b0 0x560ca8412840 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f6290005940 tx=0x7f6290031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d23fb08cddc497e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:10.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.748+0000 7f629cd99640 1 -- 172.21.15.90:0/2884860399 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f629004b070 con 0x560ca80456b0 2024-04-25T11:43:10.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.748+0000 7f629cd99640 1 -- 172.21.15.90:0/2884860399 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6290004380 con 0x560ca80456b0 2024-04-25T11:43:10.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.748+0000 7f629cd99640 1 -- 172.21.15.90:0/2884860399 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6290035040 con 0x560ca80456b0 2024-04-25T11:43:10.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.749+0000 7f62b4020740 1 -- 172.21.15.90:0/2884860399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca80456b0 msgr2=0x560ca8412840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.749+0000 7f62b4020740 1 --2- 172.21.15.90:0/2884860399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca80456b0 0x560ca8412840 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f6290005940 tx=0x7f6290031400 comp rx=0 tx=0).stop 2024-04-25T11:43:10.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.749+0000 7f62b4020740 1 -- 172.21.15.90:0/2884860399 shutdown_connections 2024-04-25T11:43:10.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.749+0000 7f62b4020740 1 --2- 172.21.15.90:0/2884860399 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ca8066390 0x560ca841de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.749+0000 7f62b4020740 1 --2- 172.21.15.90:0/2884860399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca80456b0 0x560ca8412840 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.749+0000 7f62b4020740 1 --2- 172.21.15.90:0/2884860399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ca803d4d0 0x560ca84112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.749+0000 7f62b4020740 1 -- 172.21.15.90:0/2884860399 >> 172.21.15.90:0/2884860399 conn(0x560ca805a4e0 msgr2=0x560ca84094d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:10.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.749+0000 7f62b4020740 1 -- 172.21.15.90:0/2884860399 shutdown_connections 2024-04-25T11:43:10.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.749+0000 7f62b4020740 1 -- 172.21.15.90:0/2884860399 wait complete. 2024-04-25T11:43:10.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.749+0000 7f62b4020740 1 -- start start 2024-04-25T11:43:10.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.750+0000 7f62b4020740 1 Processor -- start 2024-04-25T11:43:10.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.750+0000 7f62b4020740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca803d4d0 0x560ca841a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.750+0000 7f62b4020740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ca80456b0 0x560ca8415b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.750+0000 7f62b4020740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ca8066390 0x560ca8416040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.750+0000 7f62b4020740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f62a81b1000 con 0x560ca803d4d0 2024-04-25T11:43:10.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.750+0000 7f62b4020740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f62a81b0930 con 0x560ca80456b0 2024-04-25T11:43:10.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.750+0000 7f62b4020740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f62a81adb90 con 0x560ca8066390 2024-04-25T11:43:10.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.750+0000 7f62a6d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca803d4d0 0x560ca841a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.750+0000 7f62a6d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca803d4d0 0x560ca841a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52700/0 (socket says 172.21.15.90:52700) 2024-04-25T11:43:10.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.750+0000 7f62a6d9d640 1 -- 172.21.15.90:0/492298898 learned_addr learned my addr 172.21.15.90:0/492298898 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:10.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.750+0000 7f62b4020740 10 client.0 _ll_register_callbacks cb 0x7f62980a6fa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:10.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.750+0000 7f629e59c640 1 --2- 172.21.15.90:0/492298898 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ca80456b0 0x560ca8415b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.750+0000 7f62a759e640 1 --2- 172.21.15.90:0/492298898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ca8066390 0x560ca8416040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.750+0000 7f62a6d9d640 1 -- 172.21.15.90:0/492298898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ca8066390 msgr2=0x560ca8416040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:10.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.750+0000 7f62a6d9d640 1 --2- 172.21.15.90:0/492298898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ca8066390 0x560ca8416040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.750+0000 7f62a6d9d640 1 -- 172.21.15.90:0/492298898 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ca80456b0 msgr2=0x560ca8415b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:10.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.750+0000 7f62a6d9d640 1 --2- 172.21.15.90:0/492298898 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ca80456b0 0x560ca8415b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.750+0000 7f62a6d9d640 1 -- 172.21.15.90:0/492298898 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ca8416780 con 0x560ca803d4d0 2024-04-25T11:43:10.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.750+0000 7f62a6d9d640 1 --2- 172.21.15.90:0/492298898 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca803d4d0 0x560ca841a760 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f62980b0370 tx=0x7f62980b01a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aac9b83097a6079 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:10.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.750+0000 7f628e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:10.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.751+0000 7f628e7fc640 1 -- 172.21.15.90:0/492298898 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62980b0400 con 0x560ca803d4d0 2024-04-25T11:43:10.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.751+0000 7f62b4020740 1 -- 172.21.15.90:0/492298898 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6290004380 con 0x560ca803d4d0 2024-04-25T11:43:10.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.751+0000 7f628e7fc640 1 -- 172.21.15.90:0/492298898 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7f62980a68b0 con 0x560ca803d4d0 2024-04-25T11:43:10.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.751+0000 7f628e7fc640 1 -- 172.21.15.90:0/492298898 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f62980ab950 con 0x560ca803d4d0 2024-04-25T11:43:10.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.751+0000 7f62b4020740 10 client.18060 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:10.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.751+0000 7f62b4020740 1 -- 172.21.15.90:0/492298898 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560ca848c6a0 con 0x560ca803d4d0 2024-04-25T11:43:10.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.751+0000 7f628e7fc640 1 -- 172.21.15.90:0/492298898 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6290004380 con 0x560ca803d4d0 2024-04-25T11:43:10.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.751+0000 7f628e7fc640 1 -- 172.21.15.90:0/492298898 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f6298088c90 con 0x560ca803d4d0 2024-04-25T11:43:10.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.751+0000 7f62b4020740 10 client.18060 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:10.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.751+0000 7f62b4020740 10 client.18060 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:43:10.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.751+0000 7f62b4020740 20 client.18060 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:10.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.751+0000 7f62b4020740 1 --2- 172.21.15.90:0/492298898 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x560ca7ffa3a0 0x560ca848ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.751+0000 7f62b4020740 4 client.18060 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:43:10.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.751+0000 7f62b4020740 1 -- 172.21.15.90:0/492298898 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560ca8374800 con 0x560ca7ffa3a0 2024-04-25T11:43:10.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.751+0000 7f629e59c640 1 --2- 172.21.15.90:0/492298898 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x560ca7ffa3a0 0x560ca848ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.752+0000 7f629e59c640 1 --2- 172.21.15.90:0/492298898 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x560ca7ffa3a0 0x560ca848ee30 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=47937707268a4d0a server_cookie=dae7d886ead51d0e in_seq=0 out_seq=0 2024-04-25T11:43:10.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.752+0000 7f628e7fc640 10 client.18060 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:43:10.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.752+0000 7f628e7fc640 1 -- 172.21.15.90:0/492298898 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560ca8374800 con 0x560ca7ffa3a0 2024-04-25T11:43:10.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.752+0000 7f628e7fc640 10 client.18060 handle_command_reply: tid=0 2024-04-25T11:43:10.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.752+0000 7f62b4020740 1 client.18060 shutdown 2024-04-25T11:43:10.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.752+0000 7f62b4020740 1 -- 172.21.15.90:0/492298898 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca803d4d0 msgr2=0x560ca841a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.752+0000 7f62b4020740 1 --2- 172.21.15.90:0/492298898 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca803d4d0 0x560ca841a760 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f62980b0370 tx=0x7f62980b01a0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.752+0000 7f62b4020740 1 -- 172.21.15.90:0/492298898 shutdown_connections 2024-04-25T11:43:10.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.752+0000 7f62b4020740 1 --2- 172.21.15.90:0/492298898 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x560ca7ffa3a0 0x560ca848ee30 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.752+0000 7f628e7fc640 0 client.18060 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:43:10.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.752+0000 7f62b4020740 1 --2- 172.21.15.90:0/492298898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ca8066390 0x560ca8416040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.752+0000 7f62b4020740 1 --2- 172.21.15.90:0/492298898 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ca80456b0 0x560ca8415b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.752+0000 7f62b4020740 1 --2- 172.21.15.90:0/492298898 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ca803d4d0 0x560ca841a760 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.752+0000 7f62b4020740 1 -- 172.21.15.90:0/492298898 >> 172.21.15.90:0/492298898 conn(0x560ca805a4e0 msgr2=0x560ca8411000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:10.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.753+0000 7f62b4020740 1 -- 172.21.15.90:0/492298898 shutdown_connections 2024-04-25T11:43:10.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.753+0000 7f62b4020740 1 -- 172.21.15.90:0/492298898 wait complete. 2024-04-25T11:43:10.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.753+0000 7f62b4020740 20 client.18060 trim_cache size 0 max 16384 2024-04-25T11:43:10.744 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:43:10.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.754+0000 7f62ad09a640 1 -- 172.21.15.90:0/1634618827 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f62940afc60 msgr2=0x7f62940b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.754+0000 7f62ad09a640 1 --2- 172.21.15.90:0/1634618827 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f62940afc60 0x7f62940b2170 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f62a0023000 tx=0x7f62a0024000 comp rx=0 tx=0).stop 2024-04-25T11:43:10.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.754+0000 7f62ad09a640 1 -- 172.21.15.90:0/1634618827 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f62a8102af0 msgr2=0x7f62a819b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.754+0000 7f62ad09a640 1 --2- 172.21.15.90:0/1634618827 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f62a8102af0 0x7f62a819b540 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f629800e9c0 tx=0x7f629800ee80 comp rx=0 tx=0).stop 2024-04-25T11:43:10.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.754+0000 7f62ad09a640 1 -- 172.21.15.90:0/1634618827 shutdown_connections 2024-04-25T11:43:10.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.754+0000 7f62ad09a640 1 --2- 172.21.15.90:0/1634618827 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f62940afc60 0x7f62940b2170 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.754+0000 7f62ad09a640 1 --2- 172.21.15.90:0/1634618827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62a810d380 0x7f62a819bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.754+0000 7f62ad09a640 1 --2- 172.21.15.90:0/1634618827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62a8108a20 0x7f62a819ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.754+0000 7f62ad09a640 1 --2- 172.21.15.90:0/1634618827 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f62a8102af0 0x7f62a819b540 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.754+0000 7f62ad09a640 1 -- 172.21.15.90:0/1634618827 >> 172.21.15.90:0/1634618827 conn(0x7f62a8078100 msgr2=0x7f62a81040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:10.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.754+0000 7f62ad09a640 1 -- 172.21.15.90:0/1634618827 shutdown_connections 2024-04-25T11:43:10.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.754+0000 7f62ad09a640 1 -- 172.21.15.90:0/1634618827 wait complete. 2024-04-25T11:43:10.752 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T11:43:10.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.890+0000 7f64bb449640 1 -- 172.21.15.90:0/362452330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64b4108a40 msgr2=0x7f64b41025d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.890+0000 7f64bb449640 1 --2- 172.21.15.90:0/362452330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64b4108a40 0x7f64b41025d0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f64b0009900 tx=0x7f64b002a410 comp rx=0 tx=0).stop 2024-04-25T11:43:10.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.890+0000 7f64bb449640 1 -- 172.21.15.90:0/362452330 shutdown_connections 2024-04-25T11:43:10.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.890+0000 7f64bb449640 1 --2- 172.21.15.90:0/362452330 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f64b410d3a0 0x7f64b410f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.890+0000 7f64bb449640 1 --2- 172.21.15.90:0/362452330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64b4102b10 0x7f64b4102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.890+0000 7f64bb449640 1 --2- 172.21.15.90:0/362452330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64b4108a40 0x7f64b41025d0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.890+0000 7f64bb449640 1 -- 172.21.15.90:0/362452330 >> 172.21.15.90:0/362452330 conn(0x7f64b4078180 msgr2=0x7f64b40785c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:10.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.890+0000 7f64bb449640 1 -- 172.21.15.90:0/362452330 shutdown_connections 2024-04-25T11:43:10.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.890+0000 7f64bb449640 1 -- 172.21.15.90:0/362452330 wait complete. 2024-04-25T11:43:10.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.890+0000 7f64bb449640 1 Processor -- start 2024-04-25T11:43:10.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.890+0000 7f64bb449640 1 -- start start 2024-04-25T11:43:10.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.891+0000 7f64bb449640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64b4102b10 0x7f64b4199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.891+0000 7f64bb449640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64b4108a40 0x7f64b4199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.891+0000 7f64bb449640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f64b410d3a0 0x7f64b419de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.891+0000 7f64bb449640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f64b4112c00 con 0x7f64b4102b10 2024-04-25T11:43:10.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.891+0000 7f64bb449640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f64b4112a80 con 0x7f64b410d3a0 2024-04-25T11:43:10.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.891+0000 7f64bb449640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f64b4112d80 con 0x7f64b4108a40 2024-04-25T11:43:10.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.891+0000 7f64b91e5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64b4102b10 0x7f64b4199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.891+0000 7f64b91e5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64b4102b10 0x7f64b4199640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52726/0 (socket says 172.21.15.90:52726) 2024-04-25T11:43:10.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.891+0000 7f64b91e5640 1 -- 172.21.15.90:0/72879772 learned_addr learned my addr 172.21.15.90:0/72879772 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:10.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.891+0000 7f64b99e6640 1 --2- 172.21.15.90:0/72879772 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f64b410d3a0 0x7f64b419de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.891+0000 7f64b89e4640 1 --2- 172.21.15.90:0/72879772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64b4108a40 0x7f64b4199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.891+0000 7f64b91e5640 1 -- 172.21.15.90:0/72879772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64b4108a40 msgr2=0x7f64b4199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.891+0000 7f64b91e5640 1 --2- 172.21.15.90:0/72879772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64b4108a40 0x7f64b4199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.891+0000 7f64b91e5640 1 -- 172.21.15.90:0/72879772 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f64b410d3a0 msgr2=0x7f64b419de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.891+0000 7f64b91e5640 1 --2- 172.21.15.90:0/72879772 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f64b410d3a0 0x7f64b419de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.891+0000 7f64b91e5640 1 -- 172.21.15.90:0/72879772 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64b419e4a0 con 0x7f64b4102b10 2024-04-25T11:43:10.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.891+0000 7f64b89e4640 1 --2- 172.21.15.90:0/72879772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64b4108a40 0x7f64b4199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:10.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.891+0000 7f64b91e5640 1 --2- 172.21.15.90:0/72879772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64b4102b10 0x7f64b4199640 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f64b0006010 tx=0x7f64b0004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:10.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.891+0000 7f64aa7fc640 1 -- 172.21.15.90:0/72879772 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b003f030 con 0x7f64b4102b10 2024-04-25T11:43:10.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.892+0000 7f64aa7fc640 1 -- 172.21.15.90:0/72879772 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64b0003aa0 con 0x7f64b4102b10 2024-04-25T11:43:10.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.892+0000 7f64bb449640 1 -- 172.21.15.90:0/72879772 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64b419e730 con 0x7f64b4102b10 2024-04-25T11:43:10.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.892+0000 7f64aa7fc640 1 -- 172.21.15.90:0/72879772 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b0035070 con 0x7f64b4102b10 2024-04-25T11:43:10.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.892+0000 7f64bb449640 1 -- 172.21.15.90:0/72879772 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64b419ec20 con 0x7f64b4102b10 2024-04-25T11:43:10.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.894+0000 7f64aa7fc640 1 -- 172.21.15.90:0/72879772 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f64b0042030 con 0x7f64b4102b10 2024-04-25T11:43:10.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.895+0000 7f64c23d3740 1 Processor -- start 2024-04-25T11:43:10.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.895+0000 7f64c23d3740 1 -- start start 2024-04-25T11:43:10.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.895+0000 7f64c23d3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5630dc847390 0x5630dcbeaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.895+0000 7f64c23d3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630dc81e4d0 0x5630dcbeaf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.895+0000 7f64c23d3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc8266b0 0x5630dcbf78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.895+0000 7f64c23d3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5630dcb4dd80 con 0x5630dc8266b0 2024-04-25T11:43:10.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.895+0000 7f64c23d3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5630dcb07a20 con 0x5630dc847390 2024-04-25T11:43:10.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.895+0000 7f64c23d3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5630dcb49d40 con 0x5630dc81e4d0 2024-04-25T11:43:10.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.895+0000 7f64b91e5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630dc81e4d0 0x5630dcbeaf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.895+0000 7f64aa7fc640 1 --2- 172.21.15.90:0/72879772 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f64a00afc60 0x7f64a00b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.895+0000 7f64b91e5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630dc81e4d0 0x5630dcbeaf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:44974/0 (socket says 172.21.15.90:44974) 2024-04-25T11:43:10.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.895+0000 7f64b89e4640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc8266b0 0x5630dcbf78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.895+0000 7f64b91e5640 1 -- 172.21.15.90:0/1524379760 learned_addr learned my addr 172.21.15.90:0/1524379760 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:10.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.895+0000 7f64b99e6640 1 --2- 172.21.15.90:0/72879772 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f64a00afc60 0x7f64a00b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.895+0000 7f64b99e6640 1 --2- 172.21.15.90:0/1524379760 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5630dc847390 0x5630dcbeaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.895+0000 7f64aa7fc640 1 -- 172.21.15.90:0/72879772 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7f64b00ff560 con 0x7f64b4102b10 2024-04-25T11:43:10.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.896+0000 7f64b89e4640 1 -- 172.21.15.90:0/1524379760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630dc81e4d0 msgr2=0x5630dcbeaf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:10.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.896+0000 7f64b89e4640 1 --2- 172.21.15.90:0/1524379760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630dc81e4d0 0x5630dcbeaf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.896+0000 7f64b89e4640 1 -- 172.21.15.90:0/1524379760 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5630dc847390 msgr2=0x5630dcbeaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:10.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.896+0000 7f64b89e4640 1 --2- 172.21.15.90:0/1524379760 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5630dc847390 0x5630dcbeaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.896+0000 7f64b89e4640 1 -- 172.21.15.90:0/1524379760 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630dcb12c30 con 0x5630dc8266b0 2024-04-25T11:43:10.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.896+0000 7f64b91e5640 1 --2- 172.21.15.90:0/1524379760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630dc81e4d0 0x5630dcbeaf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:43:10.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.896+0000 7f64b99e6640 1 --2- 172.21.15.90:0/72879772 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f64a00afc60 0x7f64a00b2170 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x5630dcbe7b90 tx=0x7f64ac009b60 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:10.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.896+0000 7f64b89e4640 1 --2- 172.21.15.90:0/1524379760 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc8266b0 0x5630dcbf78b0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f64b4075fd0 tx=0x7f64a40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25f80cffe85b6c3f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:10.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.896+0000 7f649f7fe640 1 -- 172.21.15.90:0/1524379760 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a4012450 con 0x5630dc8266b0 2024-04-25T11:43:10.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.896+0000 7f649f7fe640 1 -- 172.21.15.90:0/1524379760 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64a4012aa0 con 0x5630dc8266b0 2024-04-25T11:43:10.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.896+0000 7f649f7fe640 1 -- 172.21.15.90:0/1524379760 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a400f9a0 con 0x5630dc8266b0 2024-04-25T11:43:10.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.897+0000 7f64c23d3740 1 -- 172.21.15.90:0/1524379760 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc8266b0 msgr2=0x5630dcbf78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.897+0000 7f64c23d3740 1 --2- 172.21.15.90:0/1524379760 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc8266b0 0x5630dcbf78b0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f64b4075fd0 tx=0x7f64a40117b0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.897+0000 7f64c23d3740 1 -- 172.21.15.90:0/1524379760 shutdown_connections 2024-04-25T11:43:10.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.897+0000 7f64c23d3740 1 --2- 172.21.15.90:0/1524379760 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc8266b0 0x5630dcbf78b0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.897+0000 7f64c23d3740 1 --2- 172.21.15.90:0/1524379760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630dc81e4d0 0x5630dcbeaf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.897+0000 7f64c23d3740 1 --2- 172.21.15.90:0/1524379760 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5630dc847390 0x5630dcbeaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.897+0000 7f64c23d3740 1 -- 172.21.15.90:0/1524379760 >> 172.21.15.90:0/1524379760 conn(0x5630dc83b4e0 msgr2=0x5630dcbe71d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:10.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.897+0000 7f64c23d3740 1 -- 172.21.15.90:0/1524379760 shutdown_connections 2024-04-25T11:43:10.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.897+0000 7f64c23d3740 1 -- 172.21.15.90:0/1524379760 wait complete. 2024-04-25T11:43:10.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.898+0000 7f64c23d3740 1 -- start start 2024-04-25T11:43:10.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.898+0000 7f64c23d3740 1 Processor -- start 2024-04-25T11:43:10.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.899+0000 7f64c23d3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5630dc81e4d0 0x5630dcbf92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.899+0000 7f64c23d3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc8266b0 0x5630dcbf9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.899+0000 7f64c23d3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630dc847390 0x5630dcbf36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.899+0000 7f64c23d3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5630dcbe49e0 con 0x5630dc8266b0 2024-04-25T11:43:10.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.899+0000 7f64c23d3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5630dcad4dc0 con 0x5630dc81e4d0 2024-04-25T11:43:10.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.899+0000 7f64c23d3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5630dcbe4b60 con 0x5630dc847390 2024-04-25T11:43:10.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.899+0000 7f64c23d3740 10 client.0 _ll_register_callbacks cb 0x5630dcaf0600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:10.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.899+0000 7f64b89e4640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc8266b0 0x5630dcbf9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.899+0000 7f64b89e4640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc8266b0 0x5630dcbf9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52746/0 (socket says 172.21.15.90:52746) 2024-04-25T11:43:10.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.899+0000 7f64b89e4640 1 -- 172.21.15.90:0/1985049732 learned_addr learned my addr 172.21.15.90:0/1985049732 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:10.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.899+0000 7f64b91e5640 1 --2- 172.21.15.90:0/1985049732 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5630dc81e4d0 0x5630dcbf92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.899+0000 7f64b99e6640 1 --2- 172.21.15.90:0/1985049732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630dc847390 0x5630dcbf36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.899+0000 7f64b89e4640 1 -- 172.21.15.90:0/1985049732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630dc847390 msgr2=0x5630dcbf36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:10.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.899+0000 7f64b89e4640 1 --2- 172.21.15.90:0/1985049732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630dc847390 0x5630dcbf36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.899+0000 7f64b89e4640 1 -- 172.21.15.90:0/1985049732 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5630dc81e4d0 msgr2=0x5630dcbf92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:10.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.899+0000 7f64b89e4640 1 --2- 172.21.15.90:0/1985049732 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5630dc81e4d0 0x5630dcbf92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.899+0000 7f64b89e4640 1 -- 172.21.15.90:0/1985049732 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630dc85f3f0 con 0x5630dc8266b0 2024-04-25T11:43:10.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.899+0000 7f64b91e5640 1 --2- 172.21.15.90:0/1985049732 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5630dc81e4d0 0x5630dcbf92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:43:10.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.900+0000 7f64b89e4640 1 --2- 172.21.15.90:0/1985049732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc8266b0 0x5630dcbf9810 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f64a4012e80 tx=0x7f64a4012eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7334e1ff5d2627b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:10.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.900+0000 7f649cff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:10.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.900+0000 7f649cff9640 1 -- 172.21.15.90:0/1985049732 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a4029020 con 0x5630dc8266b0 2024-04-25T11:43:10.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.900+0000 7f649cff9640 1 -- 172.21.15.90:0/1985049732 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7f64a4012660 con 0x5630dc8266b0 2024-04-25T11:43:10.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.900+0000 7f64c23d3740 1 -- 172.21.15.90:0/1985049732 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64a4012aa0 con 0x5630dc8266b0 2024-04-25T11:43:10.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.900+0000 7f649cff9640 1 -- 172.21.15.90:0/1985049732 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f64a4034050 con 0x5630dc8266b0 2024-04-25T11:43:10.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.900+0000 7f64c23d3740 10 client.18072 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:10.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.900+0000 7f64c23d3740 1 -- 172.21.15.90:0/1985049732 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5630dc9297d0 con 0x5630dc8266b0 2024-04-25T11:43:10.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.900+0000 7f649cff9640 1 -- 172.21.15.90:0/1985049732 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64a4012aa0 con 0x5630dc8266b0 2024-04-25T11:43:10.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.901+0000 7f649cff9640 1 -- 172.21.15.90:0/1985049732 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f64a4004a60 con 0x5630dc8266b0 2024-04-25T11:43:10.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.901+0000 7f64c23d3740 10 client.18072 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:10.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.901+0000 7f64c23d3740 10 client.18072 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:43:10.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.901+0000 7f64c23d3740 20 client.18072 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:10.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.901+0000 7f64c23d3740 1 --2- 172.21.15.90:0/1985049732 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5630dc805f60 0x5630dcc055e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.901+0000 7f64c23d3740 4 client.18072 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:43:10.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.901+0000 7f64c23d3740 1 -- 172.21.15.90:0/1985049732 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5630dcb5b990 con 0x5630dc805f60 2024-04-25T11:43:10.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.901+0000 7f64b91e5640 1 --2- 172.21.15.90:0/1985049732 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5630dc805f60 0x5630dcc055e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.902+0000 7f64b91e5640 1 --2- 172.21.15.90:0/1985049732 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5630dc805f60 0x5630dcc055e0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d745d7737773cf31 server_cookie=b61b42a016fbea3c in_seq=0 out_seq=0 2024-04-25T11:43:10.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.902+0000 7f649cff9640 10 client.18072 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:43:10.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.902+0000 7f649cff9640 1 -- 172.21.15.90:0/1985049732 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5630dcb5b990 con 0x5630dc805f60 2024-04-25T11:43:10.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.902+0000 7f649cff9640 10 client.18072 handle_command_reply: tid=0 2024-04-25T11:43:10.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.903+0000 7f64c23d3740 1 client.18072 shutdown 2024-04-25T11:43:10.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.903+0000 7f64c23d3740 1 -- 172.21.15.90:0/1985049732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc8266b0 msgr2=0x5630dcbf9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.903+0000 7f64c23d3740 1 --2- 172.21.15.90:0/1985049732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc8266b0 0x5630dcbf9810 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f64a4012e80 tx=0x7f64a4012eb0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.903+0000 7f64c23d3740 1 -- 172.21.15.90:0/1985049732 shutdown_connections 2024-04-25T11:43:10.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.903+0000 7f64c23d3740 1 --2- 172.21.15.90:0/1985049732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630dc847390 0x5630dcbf36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.903+0000 7f64c23d3740 1 --2- 172.21.15.90:0/1985049732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc8266b0 0x5630dcbf9810 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.903+0000 7f64c23d3740 1 --2- 172.21.15.90:0/1985049732 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5630dc805f60 0x5630dcc055e0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.903+0000 7f64c23d3740 1 --2- 172.21.15.90:0/1985049732 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5630dc81e4d0 0x5630dcbf92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.903+0000 7f649cff9640 0 client.18072 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:43:10.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.903+0000 7f64c23d3740 1 -- 172.21.15.90:0/1985049732 >> 172.21.15.90:0/1985049732 conn(0x5630dc83b4e0 msgr2=0x5630dcbec740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:10.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.903+0000 7f64c23d3740 1 -- 172.21.15.90:0/1985049732 shutdown_connections 2024-04-25T11:43:10.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.903+0000 7f64c23d3740 1 -- 172.21.15.90:0/1985049732 wait complete. 2024-04-25T11:43:10.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.903+0000 7f64c23d3740 20 client.18072 trim_cache size 0 max 16384 2024-04-25T11:43:10.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.917+0000 7f64c23d3740 1 Processor -- start 2024-04-25T11:43:10.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.917+0000 7f64c23d3740 1 -- start start 2024-04-25T11:43:10.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.918+0000 7f64c23d3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc81e4d0 0x5630dcbecd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.918+0000 7f64c23d3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630dc8266b0 0x5630dcbf0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.918+0000 7f64c23d3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5630dc847390 0x5630dcbf0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.918+0000 7f64c23d3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f64b41ad8d0 con 0x5630dc81e4d0 2024-04-25T11:43:10.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.918+0000 7f64c23d3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f64b41af410 con 0x5630dc847390 2024-04-25T11:43:10.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.918+0000 7f64c23d3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f64b41adfa0 con 0x5630dc8266b0 2024-04-25T11:43:10.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.918+0000 7f64b91e5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc81e4d0 0x5630dcbecd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.918+0000 7f64b89e4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630dc8266b0 0x5630dcbf0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.918+0000 7f64b91e5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc81e4d0 0x5630dcbecd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52754/0 (socket says 172.21.15.90:52754) 2024-04-25T11:43:10.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.918+0000 7f64b99e6640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5630dc847390 0x5630dcbf0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.918+0000 7f64b91e5640 1 -- 172.21.15.90:0/1739061711 learned_addr learned my addr 172.21.15.90:0/1739061711 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:10.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.918+0000 7f64b89e4640 1 -- 172.21.15.90:0/1739061711 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5630dc847390 msgr2=0x5630dcbf0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:10.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.918+0000 7f64b89e4640 1 --2- 172.21.15.90:0/1739061711 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5630dc847390 0x5630dcbf0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.918+0000 7f64b89e4640 1 -- 172.21.15.90:0/1739061711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc81e4d0 msgr2=0x5630dcbecd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:10.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.918+0000 7f64b89e4640 1 --2- 172.21.15.90:0/1739061711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc81e4d0 0x5630dcbecd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.918+0000 7f64b89e4640 1 -- 172.21.15.90:0/1739061711 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630dcbed590 con 0x5630dc8266b0 2024-04-25T11:43:10.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.918+0000 7f64b91e5640 1 --2- 172.21.15.90:0/1739061711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc81e4d0 0x5630dcbecd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:43:10.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.918+0000 7f64b89e4640 1 --2- 172.21.15.90:0/1739061711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630dc8266b0 0x5630dcbf0900 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f64a4004e80 tx=0x7f64a4012f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d38088b96921db8f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:10.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.919+0000 7f649effd640 1 -- 172.21.15.90:0/1739061711 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a403d070 con 0x5630dc8266b0 2024-04-25T11:43:10.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.919+0000 7f649effd640 1 -- 172.21.15.90:0/1739061711 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64a400f8f0 con 0x5630dc8266b0 2024-04-25T11:43:10.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.919+0000 7f649effd640 1 -- 172.21.15.90:0/1739061711 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a4029070 con 0x5630dc8266b0 2024-04-25T11:43:10.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.919+0000 7f64c23d3740 1 -- 172.21.15.90:0/1739061711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630dc8266b0 msgr2=0x5630dcbf0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.919+0000 7f64c23d3740 1 --2- 172.21.15.90:0/1739061711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630dc8266b0 0x5630dcbf0900 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f64a4004e80 tx=0x7f64a4012f20 comp rx=0 tx=0).stop 2024-04-25T11:43:10.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.920+0000 7f64c23d3740 1 -- 172.21.15.90:0/1739061711 shutdown_connections 2024-04-25T11:43:10.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.920+0000 7f64c23d3740 1 --2- 172.21.15.90:0/1739061711 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5630dc847390 0x5630dcbf0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.920+0000 7f64c23d3740 1 --2- 172.21.15.90:0/1739061711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630dc8266b0 0x5630dcbf0900 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.920+0000 7f64c23d3740 1 --2- 172.21.15.90:0/1739061711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc81e4d0 0x5630dcbecd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.920+0000 7f64c23d3740 1 -- 172.21.15.90:0/1739061711 >> 172.21.15.90:0/1739061711 conn(0x5630dc83b4e0 msgr2=0x5630dcbec620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:10.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.920+0000 7f64c23d3740 1 -- 172.21.15.90:0/1739061711 shutdown_connections 2024-04-25T11:43:10.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.920+0000 7f64c23d3740 1 -- 172.21.15.90:0/1739061711 wait complete. 2024-04-25T11:43:10.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.920+0000 7f64c23d3740 1 -- start start 2024-04-25T11:43:10.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.920+0000 7f64c23d3740 1 Processor -- start 2024-04-25T11:43:10.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.920+0000 7f64c23d3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc81e4d0 0x5630dcbfe4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.920+0000 7f64c23d3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5630dc8266b0 0x5630dcbfea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.920+0000 7f64c23d3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630dc847390 0x5630dcbfef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.920+0000 7f64c23d3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f64b41aed40 con 0x5630dc81e4d0 2024-04-25T11:43:10.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.920+0000 7f64c23d3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f64b41ae670 con 0x5630dc8266b0 2024-04-25T11:43:10.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.920+0000 7f64c23d3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f64b41ab8d0 con 0x5630dc847390 2024-04-25T11:43:10.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.920+0000 7f64c23d3740 10 client.0 _ll_register_callbacks cb 0x7f64ac012730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:10.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.920+0000 7f64b91e5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc81e4d0 0x5630dcbfe4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.921+0000 7f64b91e5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc81e4d0 0x5630dcbfe4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52768/0 (socket says 172.21.15.90:52768) 2024-04-25T11:43:10.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.921+0000 7f64b91e5640 1 -- 172.21.15.90:0/3654148383 learned_addr learned my addr 172.21.15.90:0/3654148383 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:10.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.921+0000 7f64b89e4640 1 --2- 172.21.15.90:0/3654148383 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5630dc8266b0 0x5630dcbfea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.921+0000 7f64b99e6640 1 --2- 172.21.15.90:0/3654148383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630dc847390 0x5630dcbfef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.921+0000 7f64b91e5640 1 -- 172.21.15.90:0/3654148383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630dc847390 msgr2=0x5630dcbfef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:10.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.921+0000 7f64b91e5640 1 --2- 172.21.15.90:0/3654148383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630dc847390 0x5630dcbfef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.921+0000 7f64b91e5640 1 -- 172.21.15.90:0/3654148383 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5630dc8266b0 msgr2=0x5630dcbfea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:10.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.921+0000 7f64b91e5640 1 --2- 172.21.15.90:0/3654148383 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5630dc8266b0 0x5630dcbfea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.921+0000 7f64b91e5640 1 -- 172.21.15.90:0/3654148383 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630dcc6b940 con 0x5630dc81e4d0 2024-04-25T11:43:10.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.921+0000 7f64b91e5640 1 --2- 172.21.15.90:0/3654148383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc81e4d0 0x5630dcbfe4f0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f64b00fccb0 tx=0x7f64b00ea040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0d138a52c7a97d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:10.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.921+0000 7f649cff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:10.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.921+0000 7f649cff9640 1 -- 172.21.15.90:0/3654148383 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b0049040 con 0x5630dc81e4d0 2024-04-25T11:43:10.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.921+0000 7f649cff9640 1 -- 172.21.15.90:0/3654148383 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7f64b00fce80 con 0x5630dc81e4d0 2024-04-25T11:43:10.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.921+0000 7f64c23d3740 1 -- 172.21.15.90:0/3654148383 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64a400f8f0 con 0x5630dc81e4d0 2024-04-25T11:43:10.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.921+0000 7f649cff9640 1 -- 172.21.15.90:0/3654148383 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f64b003b460 con 0x5630dc81e4d0 2024-04-25T11:43:10.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.921+0000 7f64c23d3740 10 client.18078 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:10.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.921+0000 7f64c23d3740 1 -- 172.21.15.90:0/3654148383 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5630dcc6cc20 con 0x5630dc81e4d0 2024-04-25T11:43:10.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.922+0000 7f649cff9640 1 -- 172.21.15.90:0/3654148383 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64a400f8f0 con 0x5630dc81e4d0 2024-04-25T11:43:10.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.922+0000 7f649cff9640 1 -- 172.21.15.90:0/3654148383 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f64b00c38e0 con 0x5630dc81e4d0 2024-04-25T11:43:10.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.922+0000 7f64c23d3740 10 client.18078 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:10.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.922+0000 7f64c23d3740 10 client.18078 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:43:10.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.922+0000 7f64c23d3740 20 client.18078 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:10.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.922+0000 7f64c23d3740 1 --2- 172.21.15.90:0/3654148383 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5630dc7db3a0 0x5630dcc6f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:10.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.922+0000 7f64c23d3740 4 client.18078 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:43:10.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.922+0000 7f64c23d3740 1 -- 172.21.15.90:0/3654148383 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5630dcb55800 con 0x5630dc7db3a0 2024-04-25T11:43:10.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.922+0000 7f64b89e4640 1 --2- 172.21.15.90:0/3654148383 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5630dc7db3a0 0x5630dcc6f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:10.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.923+0000 7f649cff9640 10 client.18078 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:43:10.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.924+0000 7f64b89e4640 1 --2- 172.21.15.90:0/3654148383 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5630dc7db3a0 0x5630dcc6f390 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=29a37f44b363854f server_cookie=dab76fdb97475a6f in_seq=0 out_seq=0 2024-04-25T11:43:10.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.924+0000 7f649cff9640 1 -- 172.21.15.90:0/3654148383 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5630dcb55800 con 0x5630dc7db3a0 2024-04-25T11:43:10.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.924+0000 7f649cff9640 10 client.18078 handle_command_reply: tid=0 2024-04-25T11:43:10.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.924+0000 7f64c23d3740 1 client.18078 shutdown 2024-04-25T11:43:10.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.924+0000 7f64c23d3740 1 -- 172.21.15.90:0/3654148383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc81e4d0 msgr2=0x5630dcbfe4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.924+0000 7f64c23d3740 1 --2- 172.21.15.90:0/3654148383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc81e4d0 0x5630dcbfe4f0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f64b00fccb0 tx=0x7f64b00ea040 comp rx=0 tx=0).stop 2024-04-25T11:43:10.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.925+0000 7f64c23d3740 1 -- 172.21.15.90:0/3654148383 shutdown_connections 2024-04-25T11:43:10.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.925+0000 7f64c23d3740 1 --2- 172.21.15.90:0/3654148383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630dc847390 0x5630dcbfef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.925+0000 7f64c23d3740 1 --2- 172.21.15.90:0/3654148383 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5630dc7db3a0 0x5630dcc6f390 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.925+0000 7f649cff9640 0 client.18078 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:43:10.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.925+0000 7f64c23d3740 1 --2- 172.21.15.90:0/3654148383 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5630dc8266b0 0x5630dcbfea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.925+0000 7f64c23d3740 1 --2- 172.21.15.90:0/3654148383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5630dc81e4d0 0x5630dcbfe4f0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.925+0000 7f64c23d3740 1 -- 172.21.15.90:0/3654148383 >> 172.21.15.90:0/3654148383 conn(0x5630dc83b4e0 msgr2=0x5630dcbe3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:10.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.925+0000 7f64c23d3740 1 -- 172.21.15.90:0/3654148383 shutdown_connections 2024-04-25T11:43:10.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.925+0000 7f64c23d3740 1 -- 172.21.15.90:0/3654148383 wait complete. 2024-04-25T11:43:10.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.925+0000 7f64c23d3740 20 client.18078 trim_cache size 0 max 16384 2024-04-25T11:43:10.915 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:43:10.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.926+0000 7f64bb449640 1 -- 172.21.15.90:0/72879772 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f64a00afc60 msgr2=0x7f64a00b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.926+0000 7f64bb449640 1 --2- 172.21.15.90:0/72879772 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f64a00afc60 0x7f64a00b2170 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x5630dcbe7b90 tx=0x7f64ac009b60 comp rx=0 tx=0).stop 2024-04-25T11:43:10.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.926+0000 7f64bb449640 1 -- 172.21.15.90:0/72879772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64b4102b10 msgr2=0x7f64b4199640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:10.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.926+0000 7f64bb449640 1 --2- 172.21.15.90:0/72879772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64b4102b10 0x7f64b4199640 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f64b0006010 tx=0x7f64b0004060 comp rx=0 tx=0).stop 2024-04-25T11:43:10.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.926+0000 7f64bb449640 1 -- 172.21.15.90:0/72879772 shutdown_connections 2024-04-25T11:43:10.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.926+0000 7f64bb449640 1 --2- 172.21.15.90:0/72879772 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f64a00afc60 0x7f64a00b2170 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.926+0000 7f64bb449640 1 --2- 172.21.15.90:0/72879772 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f64b410d3a0 0x7f64b419de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.926+0000 7f64bb449640 1 --2- 172.21.15.90:0/72879772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64b4108a40 0x7f64b4199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:10.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.926+0000 7f64bb449640 1 --2- 172.21.15.90:0/72879772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64b4102b10 0x7f64b4199640 secure :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f64b0006010 tx=0x7f64b0004060 comp rx=0 tx=0).stop 2024-04-25T11:43:10.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.926+0000 7f64bb449640 1 -- 172.21.15.90:0/72879772 >> 172.21.15.90:0/72879772 conn(0x7f64b4078180 msgr2=0x7f64b41040d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:10.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.926+0000 7f64bb449640 1 -- 172.21.15.90:0/72879772 shutdown_connections 2024-04-25T11:43:10.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:10.926+0000 7f64bb449640 1 -- 172.21.15.90:0/72879772 wait complete. 2024-04-25T11:43:10.924 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T11:43:11.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.058+0000 7f204c3a8640 1 -- 172.21.15.90:0/3107033516 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2044102400 msgr2=0x7f2044102810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.058+0000 7f204c3a8640 1 --2- 172.21.15.90:0/3107033516 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2044102400 0x7f2044102810 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f2040009900 tx=0x7f204002a410 comp rx=0 tx=0).stop 2024-04-25T11:43:11.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.058+0000 7f204c3a8640 1 -- 172.21.15.90:0/3107033516 shutdown_connections 2024-04-25T11:43:11.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.058+0000 7f204c3a8640 1 --2- 172.21.15.90:0/3107033516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2044107450 0x7f204410ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.058+0000 7f204c3a8640 1 --2- 172.21.15.90:0/3107033516 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2044106990 0x7f2044106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.058+0000 7f204c3a8640 1 --2- 172.21.15.90:0/3107033516 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2044102400 0x7f2044102810 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.058+0000 7f204c3a8640 1 -- 172.21.15.90:0/3107033516 >> 172.21.15.90:0/3107033516 conn(0x7f2044077880 msgr2=0x7f2044077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:11.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.058+0000 7f204c3a8640 1 -- 172.21.15.90:0/3107033516 shutdown_connections 2024-04-25T11:43:11.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.058+0000 7f204c3a8640 1 -- 172.21.15.90:0/3107033516 wait complete. 2024-04-25T11:43:11.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.059+0000 7f204c3a8640 1 Processor -- start 2024-04-25T11:43:11.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.059+0000 7f204c3a8640 1 -- start start 2024-04-25T11:43:11.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.059+0000 7f204c3a8640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2044102400 0x7f2044199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.059+0000 7f204c3a8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2044106990 0x7f2044199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.059+0000 7f204c3a8640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2044107450 0x7f204419de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.059+0000 7f204c3a8640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2044112be0 con 0x7f2044102400 2024-04-25T11:43:11.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.059+0000 7f204c3a8640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2044112a60 con 0x7f2044107450 2024-04-25T11:43:11.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.059+0000 7f204c3a8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2044112d60 con 0x7f2044106990 2024-04-25T11:43:11.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.059+0000 7f204a144640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2044102400 0x7f2044199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.059+0000 7f204a144640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2044102400 0x7f2044199660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52778/0 (socket says 172.21.15.90:52778) 2024-04-25T11:43:11.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.059+0000 7f204a144640 1 -- 172.21.15.90:0/846785008 learned_addr learned my addr 172.21.15.90:0/846785008 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:11.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.059+0000 7f204a945640 1 --2- 172.21.15.90:0/846785008 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2044107450 0x7f204419de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.059+0000 7f2049943640 1 --2- 172.21.15.90:0/846785008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2044106990 0x7f2044199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.059+0000 7f204a144640 1 -- 172.21.15.90:0/846785008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2044106990 msgr2=0x7f2044199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.059+0000 7f204a144640 1 --2- 172.21.15.90:0/846785008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2044106990 0x7f2044199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.059+0000 7f204a144640 1 -- 172.21.15.90:0/846785008 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2044107450 msgr2=0x7f204419de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.059+0000 7f204a144640 1 --2- 172.21.15.90:0/846785008 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2044107450 0x7f204419de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.059+0000 7f204a144640 1 -- 172.21.15.90:0/846785008 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f204419e4c0 con 0x7f2044102400 2024-04-25T11:43:11.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.059+0000 7f204a144640 1 --2- 172.21.15.90:0/846785008 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2044102400 0x7f2044199660 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f2040005940 tx=0x7f20400040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:11.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.060+0000 7f203b7fe640 1 -- 172.21.15.90:0/846785008 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204003f030 con 0x7f2044102400 2024-04-25T11:43:11.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.060+0000 7f203b7fe640 1 -- 172.21.15.90:0/846785008 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20400044c0 con 0x7f2044102400 2024-04-25T11:43:11.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.060+0000 7f204c3a8640 1 -- 172.21.15.90:0/846785008 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f204419e750 con 0x7f2044102400 2024-04-25T11:43:11.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.060+0000 7f203b7fe640 1 -- 172.21.15.90:0/846785008 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2040035070 con 0x7f2044102400 2024-04-25T11:43:11.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.060+0000 7f204c3a8640 1 -- 172.21.15.90:0/846785008 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f204419ec90 con 0x7f2044102400 2024-04-25T11:43:11.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.063+0000 7f205325c740 1 Processor -- start 2024-04-25T11:43:11.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.063+0000 7f205325c740 1 -- start start 2024-04-25T11:43:11.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.063+0000 7f203b7fe640 1 -- 172.21.15.90:0/846785008 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f2040042030 con 0x7f2044102400 2024-04-25T11:43:11.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.063+0000 7f205325c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3b8390 0x55c9cc759810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.063+0000 7f205325c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc38f4d0 0x55c9cc759d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.063+0000 7f205325c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3976b0 0x55c9cc7666a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.063+0000 7f205325c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c9cc6bed80 con 0x55c9cc3976b0 2024-04-25T11:43:11.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.063+0000 7f205325c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c9cc678a20 con 0x55c9cc3b8390 2024-04-25T11:43:11.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.063+0000 7f205325c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c9cc6bad40 con 0x55c9cc38f4d0 2024-04-25T11:43:11.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.063+0000 7f2049943640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3976b0 0x55c9cc7666a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.063+0000 7f2049943640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3976b0 0x55c9cc7666a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52792/0 (socket says 172.21.15.90:52792) 2024-04-25T11:43:11.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.064+0000 7f2049943640 1 -- 172.21.15.90:0/1996247628 learned_addr learned my addr 172.21.15.90:0/1996247628 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:11.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.064+0000 7f204a945640 1 --2- 172.21.15.90:0/1996247628 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3b8390 0x55c9cc759810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.064+0000 7f204a144640 1 --2- 172.21.15.90:0/1996247628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc38f4d0 0x55c9cc759d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.064+0000 7f204a945640 1 -- 172.21.15.90:0/1996247628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc38f4d0 msgr2=0x55c9cc759d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:11.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.064+0000 7f204a945640 1 --2- 172.21.15.90:0/1996247628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc38f4d0 0x55c9cc759d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.064+0000 7f204a945640 1 -- 172.21.15.90:0/1996247628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3976b0 msgr2=0x55c9cc7666a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:11.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.064+0000 7f204a945640 1 --2- 172.21.15.90:0/1996247628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3976b0 0x55c9cc7666a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.064+0000 7f204a945640 1 -- 172.21.15.90:0/1996247628 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9cc683c30 con 0x55c9cc3b8390 2024-04-25T11:43:11.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.064+0000 7f204a945640 1 --2- 172.21.15.90:0/1996247628 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3b8390 0x55c9cc759810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f203c009f00 tx=0x7f203c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90d88f50aff1fee3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:11.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.065+0000 7f203b7fe640 1 --2- 172.21.15.90:0/846785008 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f20300afd60 0x7f20300b2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.065+0000 7f204a144640 1 --2- 172.21.15.90:0/846785008 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f20300afd60 0x7f20300b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.065+0000 7f203b7fe640 1 -- 172.21.15.90:0/846785008 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7f20400ff430 con 0x7f2044102400 2024-04-25T11:43:11.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.065+0000 7f2038bfe640 1 -- 172.21.15.90:0/1996247628 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f203c004d20 con 0x55c9cc3b8390 2024-04-25T11:43:11.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.065+0000 7f2038bfe640 1 -- 172.21.15.90:0/1996247628 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f203c004050 con 0x55c9cc3b8390 2024-04-25T11:43:11.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.065+0000 7f2038bfe640 1 -- 172.21.15.90:0/1996247628 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f203c005280 con 0x55c9cc3b8390 2024-04-25T11:43:11.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.066+0000 7f205325c740 1 -- 172.21.15.90:0/1996247628 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3b8390 msgr2=0x55c9cc759810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.066+0000 7f204a144640 1 --2- 172.21.15.90:0/846785008 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f20300afd60 0x7f20300b2270 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x55c9cc756f50 tx=0x7f204002a410 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:11.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.066+0000 7f205325c740 1 --2- 172.21.15.90:0/1996247628 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3b8390 0x55c9cc759810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f203c009f00 tx=0x7f203c011810 comp rx=0 tx=0).stop 2024-04-25T11:43:11.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.067+0000 7f205325c740 1 -- 172.21.15.90:0/1996247628 shutdown_connections 2024-04-25T11:43:11.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.067+0000 7f205325c740 1 --2- 172.21.15.90:0/1996247628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3976b0 0x55c9cc7666a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.067+0000 7f205325c740 1 --2- 172.21.15.90:0/1996247628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc38f4d0 0x55c9cc759d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.067+0000 7f205325c740 1 --2- 172.21.15.90:0/1996247628 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3b8390 0x55c9cc759810 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.067+0000 7f205325c740 1 -- 172.21.15.90:0/1996247628 >> 172.21.15.90:0/1996247628 conn(0x55c9cc3ac4e0 msgr2=0x55c9cc755fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:11.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.067+0000 7f205325c740 1 -- 172.21.15.90:0/1996247628 shutdown_connections 2024-04-25T11:43:11.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.067+0000 7f205325c740 1 -- 172.21.15.90:0/1996247628 wait complete. 2024-04-25T11:43:11.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.067+0000 7f205325c740 1 -- start start 2024-04-25T11:43:11.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.067+0000 7f205325c740 1 Processor -- start 2024-04-25T11:43:11.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.068+0000 7f205325c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc38f4d0 0x55c9cc7642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.068+0000 7f205325c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3976b0 0x55c9cc764800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.068+0000 7f205325c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc3b8390 0x55c9cc770260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.068+0000 7f205325c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c9cc766cd0 con 0x55c9cc3976b0 2024-04-25T11:43:11.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.068+0000 7f205325c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c9cc645dc0 con 0x55c9cc38f4d0 2024-04-25T11:43:11.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.068+0000 7f205325c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c9cc766e50 con 0x55c9cc3b8390 2024-04-25T11:43:11.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.068+0000 7f205325c740 10 client.0 _ll_register_callbacks cb 0x55c9cc661600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:11.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.068+0000 7f2049943640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3976b0 0x55c9cc764800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.068+0000 7f2049943640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3976b0 0x55c9cc764800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52806/0 (socket says 172.21.15.90:52806) 2024-04-25T11:43:11.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.068+0000 7f2049943640 1 -- 172.21.15.90:0/1722133724 learned_addr learned my addr 172.21.15.90:0/1722133724 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:11.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.068+0000 7f204a945640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc38f4d0 0x55c9cc7642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.068+0000 7f204a945640 1 --2- 172.21.15.90:0/1722133724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc3b8390 0x55c9cc770260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.068+0000 7f204a945640 1 -- 172.21.15.90:0/1722133724 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc38f4d0 msgr2=0x55c9cc7642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:11.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.068+0000 7f204a945640 1 --2- 172.21.15.90:0/1722133724 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc38f4d0 0x55c9cc7642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.068+0000 7f204a945640 1 -- 172.21.15.90:0/1722133724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3976b0 msgr2=0x55c9cc764800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:11.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.068+0000 7f204a945640 1 --2- 172.21.15.90:0/1722133724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3976b0 0x55c9cc764800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.068+0000 7f204a945640 1 -- 172.21.15.90:0/1722133724 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9cc3d03f0 con 0x55c9cc3b8390 2024-04-25T11:43:11.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.068+0000 7f2049943640 1 --2- 172.21.15.90:0/1722133724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3976b0 0x55c9cc764800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:43:11.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.069+0000 7f204a945640 1 --2- 172.21.15.90:0/1722133724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc3b8390 0x55c9cc770260 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f203c005fd0 tx=0x7f203c004990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=921e9fdfe8ac382d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:11.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.069+0000 7f202dffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:43:11.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.069+0000 7f202dffb640 1 -- 172.21.15.90:0/1722133724 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f203c02d060 con 0x55c9cc3b8390 2024-04-25T11:43:11.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.069+0000 7f202dffb640 1 -- 172.21.15.90:0/1722133724 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7f203c004a40 con 0x55c9cc3b8390 2024-04-25T11:43:11.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.069+0000 7f205325c740 1 -- 172.21.15.90:0/1722133724 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f203c004050 con 0x55c9cc3b8390 2024-04-25T11:43:11.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.069+0000 7f202dffb640 1 -- 172.21.15.90:0/1722133724 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f203c004c40 con 0x55c9cc3b8390 2024-04-25T11:43:11.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.069+0000 7f205325c740 10 client.26827 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:11.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.069+0000 7f205325c740 1 -- 172.21.15.90:0/1722133724 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c9cc49a7d0 con 0x55c9cc3b8390 2024-04-25T11:43:11.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.070+0000 7f202dffb640 1 -- 172.21.15.90:0/1722133724 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f203c004050 con 0x55c9cc3b8390 2024-04-25T11:43:11.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.070+0000 7f202dffb640 1 -- 172.21.15.90:0/1722133724 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f203c0288a0 con 0x55c9cc3b8390 2024-04-25T11:43:11.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.070+0000 7f205325c740 10 client.26827 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:11.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.070+0000 7f205325c740 10 client.26827 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:43:11.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.070+0000 7f205325c740 20 client.26827 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:11.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.070+0000 7f205325c740 1 --2- 172.21.15.90:0/1722133724 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55c9cc376f60 0x55c9cc7765e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.070+0000 7f205325c740 4 client.26827 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:43:11.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.070+0000 7f205325c740 1 -- 172.21.15.90:0/1722133724 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c9cc6cc990 con 0x55c9cc376f60 2024-04-25T11:43:11.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.070+0000 7f2049943640 1 --2- 172.21.15.90:0/1722133724 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55c9cc376f60 0x55c9cc7765e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.071+0000 7f202dffb640 10 client.26827 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:43:11.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.071+0000 7f2049943640 1 --2- 172.21.15.90:0/1722133724 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55c9cc376f60 0x55c9cc7765e0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=efb88220b5f10d9 server_cookie=8eb50cefe5f93a3 in_seq=0 out_seq=0 2024-04-25T11:43:11.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.072+0000 7f202dffb640 1 -- 172.21.15.90:0/1722133724 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c9cc6cc990 con 0x55c9cc376f60 2024-04-25T11:43:11.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.072+0000 7f202dffb640 10 client.26827 handle_command_reply: tid=0 2024-04-25T11:43:11.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.072+0000 7f205325c740 1 client.26827 shutdown 2024-04-25T11:43:11.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.072+0000 7f205325c740 1 -- 172.21.15.90:0/1722133724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc3b8390 msgr2=0x55c9cc770260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.072+0000 7f205325c740 1 --2- 172.21.15.90:0/1722133724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc3b8390 0x55c9cc770260 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f203c005fd0 tx=0x7f203c004990 comp rx=0 tx=0).stop 2024-04-25T11:43:11.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.072+0000 7f205325c740 1 -- 172.21.15.90:0/1722133724 shutdown_connections 2024-04-25T11:43:11.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.072+0000 7f205325c740 1 --2- 172.21.15.90:0/1722133724 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55c9cc376f60 0x55c9cc7765e0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.072+0000 7f205325c740 1 --2- 172.21.15.90:0/1722133724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc3b8390 0x55c9cc770260 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.072+0000 7f205325c740 1 --2- 172.21.15.90:0/1722133724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3976b0 0x55c9cc764800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.072+0000 7f202dffb640 0 client.26827 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:43:11.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.072+0000 7f205325c740 1 --2- 172.21.15.90:0/1722133724 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc38f4d0 0x55c9cc7642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.072+0000 7f205325c740 1 -- 172.21.15.90:0/1722133724 >> 172.21.15.90:0/1722133724 conn(0x55c9cc3ac4e0 msgr2=0x55c9cc75b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:11.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.072+0000 7f205325c740 1 -- 172.21.15.90:0/1722133724 shutdown_connections 2024-04-25T11:43:11.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.072+0000 7f205325c740 1 -- 172.21.15.90:0/1722133724 wait complete. 2024-04-25T11:43:11.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.072+0000 7f205325c740 20 client.26827 trim_cache size 0 max 16384 2024-04-25T11:43:11.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.090+0000 7f205325c740 1 Processor -- start 2024-04-25T11:43:11.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.090+0000 7f205325c740 1 -- start start 2024-04-25T11:43:11.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.090+0000 7f205325c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc38f4d0 0x55c9cc763860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.090+0000 7f205325c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3976b0 0x55c9cc75e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.090+0000 7f205325c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc3b8390 0x55c9cc75e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.090+0000 7f205325c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f20441b62b0 con 0x55c9cc38f4d0 2024-04-25T11:43:11.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.090+0000 7f205325c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f20441b7df0 con 0x55c9cc3976b0 2024-04-25T11:43:11.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.090+0000 7f205325c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f20441b6980 con 0x55c9cc3b8390 2024-04-25T11:43:11.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.090+0000 7f204a945640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc38f4d0 0x55c9cc763860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.090+0000 7f204a945640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc38f4d0 0x55c9cc763860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52812/0 (socket says 172.21.15.90:52812) 2024-04-25T11:43:11.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.090+0000 7f204a945640 1 -- 172.21.15.90:0/2258879198 learned_addr learned my addr 172.21.15.90:0/2258879198 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:11.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.090+0000 7f2049943640 1 --2- 172.21.15.90:0/2258879198 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3976b0 0x55c9cc75e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.090+0000 7f204a945640 1 --2- 172.21.15.90:0/2258879198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc3b8390 0x55c9cc75e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.090+0000 7f204a945640 1 -- 172.21.15.90:0/2258879198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc3b8390 msgr2=0x55c9cc75e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:11.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.090+0000 7f204a945640 1 --2- 172.21.15.90:0/2258879198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc3b8390 0x55c9cc75e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.090+0000 7f204a945640 1 -- 172.21.15.90:0/2258879198 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3976b0 msgr2=0x55c9cc75e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:11.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.090+0000 7f204a945640 1 --2- 172.21.15.90:0/2258879198 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3976b0 0x55c9cc75e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.091+0000 7f204a945640 1 -- 172.21.15.90:0/2258879198 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9cc764b90 con 0x55c9cc38f4d0 2024-04-25T11:43:11.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.091+0000 7f2049943640 1 --2- 172.21.15.90:0/2258879198 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3976b0 0x55c9cc75e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:11.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.091+0000 7f204a945640 1 --2- 172.21.15.90:0/2258879198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc38f4d0 0x55c9cc763860 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f203c009bb0 tx=0x7f203c004510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd682ade0e1015e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:11.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.091+0000 7f202ffff640 1 -- 172.21.15.90:0/2258879198 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f203c02d040 con 0x55c9cc38f4d0 2024-04-25T11:43:11.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.091+0000 7f202ffff640 1 -- 172.21.15.90:0/2258879198 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f203c028e50 con 0x55c9cc38f4d0 2024-04-25T11:43:11.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.091+0000 7f202ffff640 1 -- 172.21.15.90:0/2258879198 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f203c003e70 con 0x55c9cc38f4d0 2024-04-25T11:43:11.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.092+0000 7f205325c740 1 -- 172.21.15.90:0/2258879198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc38f4d0 msgr2=0x55c9cc763860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.092+0000 7f205325c740 1 --2- 172.21.15.90:0/2258879198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc38f4d0 0x55c9cc763860 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f203c009bb0 tx=0x7f203c004510 comp rx=0 tx=0).stop 2024-04-25T11:43:11.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.092+0000 7f205325c740 1 -- 172.21.15.90:0/2258879198 shutdown_connections 2024-04-25T11:43:11.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.092+0000 7f205325c740 1 --2- 172.21.15.90:0/2258879198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc3b8390 0x55c9cc75e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.092+0000 7f205325c740 1 --2- 172.21.15.90:0/2258879198 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3976b0 0x55c9cc75e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.092+0000 7f205325c740 1 --2- 172.21.15.90:0/2258879198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc38f4d0 0x55c9cc763860 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.092+0000 7f205325c740 1 -- 172.21.15.90:0/2258879198 >> 172.21.15.90:0/2258879198 conn(0x55c9cc3ac4e0 msgr2=0x55c9cc754610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:11.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.092+0000 7f205325c740 1 -- 172.21.15.90:0/2258879198 shutdown_connections 2024-04-25T11:43:11.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.092+0000 7f205325c740 1 -- 172.21.15.90:0/2258879198 wait complete. 2024-04-25T11:43:11.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.092+0000 7f205325c740 1 -- start start 2024-04-25T11:43:11.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.092+0000 7f205325c740 1 Processor -- start 2024-04-25T11:43:11.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.093+0000 7f205325c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc38f4d0 0x55c9cc767180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.093+0000 7f205325c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3976b0 0x55c9cc7676c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.093+0000 7f205325c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3b8390 0x55c9cc767c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.093+0000 7f205325c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f20441b7720 con 0x55c9cc3976b0 2024-04-25T11:43:11.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.093+0000 7f205325c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f20441b7050 con 0x55c9cc3b8390 2024-04-25T11:43:11.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.093+0000 7f205325c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f20441b42b0 con 0x55c9cc38f4d0 2024-04-25T11:43:11.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.093+0000 7f205325c740 10 client.0 _ll_register_callbacks cb 0x7f203c009e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:11.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.093+0000 7f2049943640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3976b0 0x55c9cc7676c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.093+0000 7f2049943640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3976b0 0x55c9cc7676c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52826/0 (socket says 172.21.15.90:52826) 2024-04-25T11:43:11.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.093+0000 7f2049943640 1 -- 172.21.15.90:0/3588050741 learned_addr learned my addr 172.21.15.90:0/3588050741 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:11.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.093+0000 7f204a945640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3b8390 0x55c9cc767c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.093+0000 7f204a945640 1 --2- 172.21.15.90:0/3588050741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc38f4d0 0x55c9cc767180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.093+0000 7f2049943640 1 -- 172.21.15.90:0/3588050741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc38f4d0 msgr2=0x55c9cc767180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:11.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.093+0000 7f2049943640 1 --2- 172.21.15.90:0/3588050741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc38f4d0 0x55c9cc767180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.093+0000 7f2049943640 1 -- 172.21.15.90:0/3588050741 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3b8390 msgr2=0x55c9cc767c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:11.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.093+0000 7f2049943640 1 --2- 172.21.15.90:0/3588050741 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3b8390 0x55c9cc767c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.093+0000 7f2049943640 1 -- 172.21.15.90:0/3588050741 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9cc7dc900 con 0x55c9cc3976b0 2024-04-25T11:43:11.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.093+0000 7f2049943640 1 --2- 172.21.15.90:0/3588050741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3976b0 0x55c9cc7676c0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x55c9cc776b80 tx=0x7f2034009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0868f1ac223a59a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:11.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.093+0000 7f202dffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:11.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.093+0000 7f202dffb640 1 -- 172.21.15.90:0/3588050741 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f203401e540 con 0x55c9cc3976b0 2024-04-25T11:43:11.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.094+0000 7f202dffb640 1 -- 172.21.15.90:0/3588050741 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7f203401e6e0 con 0x55c9cc3976b0 2024-04-25T11:43:11.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.094+0000 7f205325c740 1 -- 172.21.15.90:0/3588050741 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f203c028e50 con 0x55c9cc3976b0 2024-04-25T11:43:11.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.094+0000 7f202dffb640 1 -- 172.21.15.90:0/3588050741 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f203401e8e0 con 0x55c9cc3976b0 2024-04-25T11:43:11.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.094+0000 7f205325c740 10 client.18096 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:11.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.094+0000 7f205325c740 1 -- 172.21.15.90:0/3588050741 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c9cc7ddbe0 con 0x55c9cc3976b0 2024-04-25T11:43:11.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.094+0000 7f202dffb640 1 -- 172.21.15.90:0/3588050741 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f203c028e50 con 0x55c9cc3976b0 2024-04-25T11:43:11.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.094+0000 7f202dffb640 1 -- 172.21.15.90:0/3588050741 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f20340275e0 con 0x55c9cc3976b0 2024-04-25T11:43:11.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.094+0000 7f205325c740 10 client.18096 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:11.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.094+0000 7f205325c740 10 client.18096 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:43:11.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.094+0000 7f205325c740 20 client.18096 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:11.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.094+0000 7f205325c740 1 --2- 172.21.15.90:0/3588050741 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55c9cc34c3a0 0x55c9cc7e03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.094+0000 7f205325c740 4 client.18096 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:43:11.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.094+0000 7f205325c740 1 -- 172.21.15.90:0/3588050741 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c9cc6c6800 con 0x55c9cc34c3a0 2024-04-25T11:43:11.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.094+0000 7f204a945640 1 --2- 172.21.15.90:0/3588050741 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55c9cc34c3a0 0x55c9cc7e03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.095+0000 7f202dffb640 10 client.18096 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:43:11.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.095+0000 7f204a945640 1 --2- 172.21.15.90:0/3588050741 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55c9cc34c3a0 0x55c9cc7e03d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b7620297a9ab513c server_cookie=d8bb57a50404337f in_seq=0 out_seq=0 2024-04-25T11:43:11.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.095+0000 7f202dffb640 1 -- 172.21.15.90:0/3588050741 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c9cc6c6800 con 0x55c9cc34c3a0 2024-04-25T11:43:11.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.095+0000 7f202dffb640 10 client.18096 handle_command_reply: tid=0 2024-04-25T11:43:11.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.095+0000 7f205325c740 1 client.18096 shutdown 2024-04-25T11:43:11.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.095+0000 7f205325c740 1 -- 172.21.15.90:0/3588050741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3976b0 msgr2=0x55c9cc7676c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.095+0000 7f205325c740 1 --2- 172.21.15.90:0/3588050741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3976b0 0x55c9cc7676c0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x55c9cc776b80 tx=0x7f2034009310 comp rx=0 tx=0).stop 2024-04-25T11:43:11.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.095+0000 7f205325c740 1 -- 172.21.15.90:0/3588050741 shutdown_connections 2024-04-25T11:43:11.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.095+0000 7f205325c740 1 --2- 172.21.15.90:0/3588050741 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55c9cc34c3a0 0x55c9cc7e03d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.095+0000 7f205325c740 1 --2- 172.21.15.90:0/3588050741 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c9cc3b8390 0x55c9cc767c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.095+0000 7f205325c740 1 --2- 172.21.15.90:0/3588050741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9cc3976b0 0x55c9cc7676c0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.095+0000 7f202dffb640 0 client.18096 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:43:11.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.095+0000 7f205325c740 1 --2- 172.21.15.90:0/3588050741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c9cc38f4d0 0x55c9cc767180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.095+0000 7f205325c740 1 -- 172.21.15.90:0/3588050741 >> 172.21.15.90:0/3588050741 conn(0x55c9cc3ac4e0 msgr2=0x55c9cc755040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:11.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.095+0000 7f205325c740 1 -- 172.21.15.90:0/3588050741 shutdown_connections 2024-04-25T11:43:11.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.095+0000 7f205325c740 1 -- 172.21.15.90:0/3588050741 wait complete. 2024-04-25T11:43:11.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.095+0000 7f205325c740 20 client.18096 trim_cache size 0 max 16384 2024-04-25T11:43:11.087 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:43:11.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.096+0000 7f204c3a8640 1 -- 172.21.15.90:0/846785008 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f20300afd60 msgr2=0x7f20300b2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.096+0000 7f204c3a8640 1 --2- 172.21.15.90:0/846785008 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f20300afd60 0x7f20300b2270 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x55c9cc756f50 tx=0x7f204002a410 comp rx=0 tx=0).stop 2024-04-25T11:43:11.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.096+0000 7f204c3a8640 1 -- 172.21.15.90:0/846785008 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2044102400 msgr2=0x7f2044199660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.096+0000 7f204c3a8640 1 --2- 172.21.15.90:0/846785008 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2044102400 0x7f2044199660 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f2040005940 tx=0x7f20400040c0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.096+0000 7f204c3a8640 1 -- 172.21.15.90:0/846785008 shutdown_connections 2024-04-25T11:43:11.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.096+0000 7f204c3a8640 1 --2- 172.21.15.90:0/846785008 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f20300afd60 0x7f20300b2270 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.096+0000 7f204c3a8640 1 --2- 172.21.15.90:0/846785008 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2044107450 0x7f204419de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.096+0000 7f204c3a8640 1 --2- 172.21.15.90:0/846785008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2044106990 0x7f2044199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.096+0000 7f204c3a8640 1 --2- 172.21.15.90:0/846785008 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2044102400 0x7f2044199660 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.096+0000 7f204c3a8640 1 -- 172.21.15.90:0/846785008 >> 172.21.15.90:0/846785008 conn(0x7f2044077880 msgr2=0x7f20441008c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:11.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.096+0000 7f204c3a8640 1 -- 172.21.15.90:0/846785008 shutdown_connections 2024-04-25T11:43:11.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.096+0000 7f204c3a8640 1 -- 172.21.15.90:0/846785008 wait complete. 2024-04-25T11:43:11.095 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T11:43:11.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.230+0000 7fc796fee640 1 -- 172.21.15.90:0/2787098775 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc790108a20 msgr2=0x7fc7901025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.230+0000 7fc796fee640 1 --2- 172.21.15.90:0/2787098775 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc790108a20 0x7fc7901025b0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fc784009900 tx=0x7fc78402a410 comp rx=0 tx=0).stop 2024-04-25T11:43:11.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.230+0000 7fc796fee640 1 -- 172.21.15.90:0/2787098775 shutdown_connections 2024-04-25T11:43:11.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.230+0000 7fc796fee640 1 --2- 172.21.15.90:0/2787098775 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc79010d380 0x7fc79010f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.230+0000 7fc796fee640 1 --2- 172.21.15.90:0/2787098775 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc790102af0 0x7fc790102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.230+0000 7fc796fee640 1 --2- 172.21.15.90:0/2787098775 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc790108a20 0x7fc7901025b0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.230+0000 7fc796fee640 1 -- 172.21.15.90:0/2787098775 >> 172.21.15.90:0/2787098775 conn(0x7fc790078100 msgr2=0x7fc790078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:11.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.231+0000 7fc796fee640 1 -- 172.21.15.90:0/2787098775 shutdown_connections 2024-04-25T11:43:11.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.231+0000 7fc796fee640 1 -- 172.21.15.90:0/2787098775 wait complete. 2024-04-25T11:43:11.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.231+0000 7fc796fee640 1 Processor -- start 2024-04-25T11:43:11.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.231+0000 7fc796fee640 1 -- start start 2024-04-25T11:43:11.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.231+0000 7fc796fee640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc790102af0 0x7fc79019a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.232+0000 7fc796fee640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc790108a20 0x7fc79019acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.232+0000 7fc796fee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc79010d380 0x7fc79019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.232+0000 7fc796fee640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc790113d00 con 0x7fc790102af0 2024-04-25T11:43:11.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.232+0000 7fc796fee640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc790113b80 con 0x7fc790108a20 2024-04-25T11:43:11.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.232+0000 7fc796fee640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc790113e80 con 0x7fc79010d380 2024-04-25T11:43:11.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.232+0000 7fc794d8a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc790102af0 0x7fc79019a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.232+0000 7fc794d8a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc790102af0 0x7fc79019a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52848/0 (socket says 172.21.15.90:52848) 2024-04-25T11:43:11.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.232+0000 7fc794d8a640 1 -- 172.21.15.90:0/3830192272 learned_addr learned my addr 172.21.15.90:0/3830192272 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:11.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.232+0000 7fc78ffff640 1 --2- 172.21.15.90:0/3830192272 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc790108a20 0x7fc79019acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.232+0000 7fc79558b640 1 --2- 172.21.15.90:0/3830192272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc79010d380 0x7fc79019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.232+0000 7fc794d8a640 1 -- 172.21.15.90:0/3830192272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc79010d380 msgr2=0x7fc79019ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.232+0000 7fc794d8a640 1 --2- 172.21.15.90:0/3830192272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc79010d380 0x7fc79019ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.232+0000 7fc794d8a640 1 -- 172.21.15.90:0/3830192272 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc790108a20 msgr2=0x7fc79019acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.232+0000 7fc794d8a640 1 --2- 172.21.15.90:0/3830192272 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc790108a20 0x7fc79019acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.232+0000 7fc794d8a640 1 -- 172.21.15.90:0/3830192272 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc79019f5c0 con 0x7fc790102af0 2024-04-25T11:43:11.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.232+0000 7fc794d8a640 1 --2- 172.21.15.90:0/3830192272 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc790102af0 0x7fc79019a760 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fc784032fd0 tx=0x7fc784031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:11.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.232+0000 7fc78dffb640 1 -- 172.21.15.90:0/3830192272 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78403f030 con 0x7fc790102af0 2024-04-25T11:43:11.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.232+0000 7fc78dffb640 1 -- 172.21.15.90:0/3830192272 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc784004490 con 0x7fc790102af0 2024-04-25T11:43:11.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.232+0000 7fc796fee640 1 -- 172.21.15.90:0/3830192272 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc79019f850 con 0x7fc790102af0 2024-04-25T11:43:11.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.232+0000 7fc78dffb640 1 -- 172.21.15.90:0/3830192272 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc784035070 con 0x7fc790102af0 2024-04-25T11:43:11.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.233+0000 7fc796fee640 1 -- 172.21.15.90:0/3830192272 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc79019fd40 con 0x7fc790102af0 2024-04-25T11:43:11.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.235+0000 7fc78dffb640 1 -- 172.21.15.90:0/3830192272 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fc784042030 con 0x7fc790102af0 2024-04-25T11:43:11.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.235+0000 7fc79df74740 1 Processor -- start 2024-04-25T11:43:11.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.235+0000 7fc79df74740 1 -- start start 2024-04-25T11:43:11.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.235+0000 7fc79df74740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04ef390 0x560ec0890810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.235+0000 7fc79df74740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ec04c64d0 0x560ec0890d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.235+0000 7fc79df74740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ec04ce6b0 0x560ec089d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.235+0000 7fc79df74740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560ec07f5d80 con 0x560ec04ef390 2024-04-25T11:43:11.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.235+0000 7fc79df74740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x560ec07afa20 con 0x560ec04ce6b0 2024-04-25T11:43:11.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.235+0000 7fc79df74740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560ec07f1d40 con 0x560ec04c64d0 2024-04-25T11:43:11.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.235+0000 7fc79558b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04ef390 0x560ec0890810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.236+0000 7fc79558b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04ef390 0x560ec0890810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52862/0 (socket says 172.21.15.90:52862) 2024-04-25T11:43:11.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.236+0000 7fc79558b640 1 -- 172.21.15.90:0/3542866546 learned_addr learned my addr 172.21.15.90:0/3542866546 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:11.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.236+0000 7fc78ffff640 1 --2- 172.21.15.90:0/3542866546 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ec04ce6b0 0x560ec089d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.236+0000 7fc78dffb640 1 --2- 172.21.15.90:0/3830192272 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc7800afc60 0x7fc7800b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.236+0000 7fc794d8a640 1 --2- 172.21.15.90:0/3542866546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ec04c64d0 0x560ec0890d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.236+0000 7fc78ffff640 1 -- 172.21.15.90:0/3542866546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ec04c64d0 msgr2=0x560ec0890d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:11.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.236+0000 7fc78ffff640 1 --2- 172.21.15.90:0/3542866546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ec04c64d0 0x560ec0890d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.236+0000 7fc78ffff640 1 -- 172.21.15.90:0/3542866546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04ef390 msgr2=0x560ec0890810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:11.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.236+0000 7fc78ffff640 1 --2- 172.21.15.90:0/3542866546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04ef390 0x560ec0890810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.236+0000 7fc78ffff640 1 -- 172.21.15.90:0/3542866546 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ec07bac30 con 0x560ec04ce6b0 2024-04-25T11:43:11.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.236+0000 7fc794d8a640 1 --2- 172.21.15.90:0/3542866546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ec04c64d0 0x560ec0890d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:11.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.236+0000 7fc79558b640 1 --2- 172.21.15.90:0/3542866546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04ef390 0x560ec0890810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:43:11.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.236+0000 7fc78dffb640 1 -- 172.21.15.90:0/3830192272 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7fc7840ff4c0 con 0x7fc790102af0 2024-04-25T11:43:11.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.236+0000 7fc79558b640 1 --2- 172.21.15.90:0/3830192272 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc7800afc60 0x7fc7800b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.236+0000 7fc78ffff640 1 --2- 172.21.15.90:0/3542866546 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ec04ce6b0 0x560ec089d6a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc790075fd0 tx=0x7fc77c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f293633e0a6734e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:11.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.236+0000 7fc77affd640 1 -- 172.21.15.90:0/3542866546 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c009d70 con 0x560ec04ce6b0 2024-04-25T11:43:11.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.236+0000 7fc77affd640 1 -- 172.21.15.90:0/3542866546 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc77c005020 con 0x560ec04ce6b0 2024-04-25T11:43:11.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.236+0000 7fc77affd640 1 -- 172.21.15.90:0/3542866546 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c004a70 con 0x560ec04ce6b0 2024-04-25T11:43:11.225 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:11 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1165227036' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:43:11.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.237+0000 7fc79df74740 1 -- 172.21.15.90:0/3542866546 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ec04ce6b0 msgr2=0x560ec089d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.237+0000 7fc79558b640 1 --2- 172.21.15.90:0/3830192272 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc7800afc60 0x7fc7800b2170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc788009fd0 tx=0x7fc788004a40 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:11.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.237+0000 7fc79df74740 1 --2- 172.21.15.90:0/3542866546 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ec04ce6b0 0x560ec089d6a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc790075fd0 tx=0x7fc77c011810 comp rx=0 tx=0).stop 2024-04-25T11:43:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.237+0000 7fc79df74740 1 -- 172.21.15.90:0/3542866546 shutdown_connections 2024-04-25T11:43:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.237+0000 7fc79df74740 1 --2- 172.21.15.90:0/3542866546 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ec04ce6b0 0x560ec089d6a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.237+0000 7fc79df74740 1 --2- 172.21.15.90:0/3542866546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ec04c64d0 0x560ec0890d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.237+0000 7fc79df74740 1 --2- 172.21.15.90:0/3542866546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04ef390 0x560ec0890810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.237+0000 7fc79df74740 1 -- 172.21.15.90:0/3542866546 >> 172.21.15.90:0/3542866546 conn(0x560ec04e34e0 msgr2=0x560ec088cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.237+0000 7fc79df74740 1 -- 172.21.15.90:0/3542866546 shutdown_connections 2024-04-25T11:43:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.237+0000 7fc79df74740 1 -- 172.21.15.90:0/3542866546 wait complete. 2024-04-25T11:43:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.238+0000 7fc79df74740 1 -- start start 2024-04-25T11:43:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.238+0000 7fc79df74740 1 Processor -- start 2024-04-25T11:43:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.238+0000 7fc79df74740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ec04c64d0 0x560ec089b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.238+0000 7fc79df74740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ec04ce6b0 0x560ec089b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.238+0000 7fc79df74740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04ef390 0x560ec08a7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.238+0000 7fc79df74740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560ec089dd90 con 0x560ec04ef390 2024-04-25T11:43:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.238+0000 7fc79df74740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x560ec077cdc0 con 0x560ec04c64d0 2024-04-25T11:43:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.238+0000 7fc79df74740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560ec089df10 con 0x560ec04ce6b0 2024-04-25T11:43:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.238+0000 7fc79df74740 10 client.0 _ll_register_callbacks cb 0x560ec0798600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.238+0000 7fc79558b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04ef390 0x560ec08a7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.238+0000 7fc79558b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04ef390 0x560ec08a7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52876/0 (socket says 172.21.15.90:52876) 2024-04-25T11:43:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.238+0000 7fc79558b640 1 -- 172.21.15.90:0/3896777030 learned_addr learned my addr 172.21.15.90:0/3896777030 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.239+0000 7fc794d8a640 1 --2- 172.21.15.90:0/3896777030 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ec04c64d0 0x560ec089b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.239+0000 7fc78ffff640 1 --2- 172.21.15.90:0/3896777030 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ec04ce6b0 0x560ec089b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.239+0000 7fc79558b640 1 -- 172.21.15.90:0/3896777030 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ec04ce6b0 msgr2=0x560ec089b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.239+0000 7fc79558b640 1 --2- 172.21.15.90:0/3896777030 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ec04ce6b0 0x560ec089b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.239+0000 7fc79558b640 1 -- 172.21.15.90:0/3896777030 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ec04c64d0 msgr2=0x560ec089b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.239+0000 7fc79558b640 1 --2- 172.21.15.90:0/3896777030 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ec04c64d0 0x560ec089b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.239+0000 7fc79558b640 1 -- 172.21.15.90:0/3896777030 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ec05073f0 con 0x560ec04ef390 2024-04-25T11:43:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.239+0000 7fc79558b640 1 --2- 172.21.15.90:0/3896777030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04ef390 0x560ec08a7210 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fc788013a10 tx=0x7fc788004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15da12fee8c252c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.239+0000 7fc76ffff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.239+0000 7fc76ffff640 1 -- 172.21.15.90:0/3896777030 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc788011e10 con 0x560ec04ef390 2024-04-25T11:43:11.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.239+0000 7fc76ffff640 1 -- 172.21.15.90:0/3896777030 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7fc78801e420 con 0x560ec04ef390 2024-04-25T11:43:11.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.239+0000 7fc79df74740 1 -- 172.21.15.90:0/3896777030 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc77c005020 con 0x560ec04ef390 2024-04-25T11:43:11.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.239+0000 7fc76ffff640 1 -- 172.21.15.90:0/3896777030 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc78801e620 con 0x560ec04ef390 2024-04-25T11:43:11.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.239+0000 7fc79df74740 10 client.18108 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:11.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.239+0000 7fc79df74740 1 -- 172.21.15.90:0/3896777030 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560ec05d17d0 con 0x560ec04ef390 2024-04-25T11:43:11.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.240+0000 7fc76ffff640 1 -- 172.21.15.90:0/3896777030 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc77c005020 con 0x560ec04ef390 2024-04-25T11:43:11.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.240+0000 7fc76ffff640 1 -- 172.21.15.90:0/3896777030 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fc78802f880 con 0x560ec04ef390 2024-04-25T11:43:11.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.240+0000 7fc79df74740 10 client.18108 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:11.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.240+0000 7fc79df74740 10 client.18108 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:43:11.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.240+0000 7fc79df74740 20 client.18108 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:11.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.240+0000 7fc79df74740 1 --2- 172.21.15.90:0/3896777030 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x560ec04adf60 0x560ec08ad6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.240+0000 7fc79df74740 4 client.18108 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:43:11.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.240+0000 7fc79df74740 1 -- 172.21.15.90:0/3896777030 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560ec0803990 con 0x560ec04adf60 2024-04-25T11:43:11.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.240+0000 7fc794d8a640 1 --2- 172.21.15.90:0/3896777030 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x560ec04adf60 0x560ec08ad6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.241+0000 7fc76ffff640 10 client.18108 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:43:11.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.241+0000 7fc794d8a640 1 --2- 172.21.15.90:0/3896777030 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x560ec04adf60 0x560ec08ad6d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3661e0c94e52d1d4 server_cookie=117c1c2690482a37 in_seq=0 out_seq=0 2024-04-25T11:43:11.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.241+0000 7fc76ffff640 1 -- 172.21.15.90:0/3896777030 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560ec0803990 con 0x560ec04adf60 2024-04-25T11:43:11.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.241+0000 7fc76ffff640 10 client.18108 handle_command_reply: tid=0 2024-04-25T11:43:11.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.242+0000 7fc79df74740 1 client.18108 shutdown 2024-04-25T11:43:11.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.242+0000 7fc79df74740 1 -- 172.21.15.90:0/3896777030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04ef390 msgr2=0x560ec08a7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.242+0000 7fc79df74740 1 --2- 172.21.15.90:0/3896777030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04ef390 0x560ec08a7210 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fc788013a10 tx=0x7fc788004ab0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.242+0000 7fc79df74740 1 -- 172.21.15.90:0/3896777030 shutdown_connections 2024-04-25T11:43:11.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.242+0000 7fc79df74740 1 --2- 172.21.15.90:0/3896777030 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x560ec04adf60 0x560ec08ad6d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.242+0000 7fc79df74740 1 --2- 172.21.15.90:0/3896777030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04ef390 0x560ec08a7210 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.242+0000 7fc79df74740 1 --2- 172.21.15.90:0/3896777030 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ec04ce6b0 0x560ec089b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.242+0000 7fc79df74740 1 --2- 172.21.15.90:0/3896777030 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ec04c64d0 0x560ec089b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.242+0000 7fc79df74740 1 -- 172.21.15.90:0/3896777030 >> 172.21.15.90:0/3896777030 conn(0x560ec04e34e0 msgr2=0x560ec0892530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:11.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.242+0000 7fc76ffff640 0 client.18108 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:43:11.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.242+0000 7fc79df74740 1 -- 172.21.15.90:0/3896777030 shutdown_connections 2024-04-25T11:43:11.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.242+0000 7fc79df74740 1 -- 172.21.15.90:0/3896777030 wait complete. 2024-04-25T11:43:11.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.242+0000 7fc79df74740 20 client.18108 trim_cache size 0 max 16384 2024-04-25T11:43:11.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.257+0000 7fc79df74740 1 Processor -- start 2024-04-25T11:43:11.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.257+0000 7fc79df74740 1 -- start start 2024-04-25T11:43:11.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.257+0000 7fc79df74740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04c64d0 0x560ec08967f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.257+0000 7fc79df74740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ec04ce6b0 0x560ec088ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.257+0000 7fc79df74740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ec04ef390 0x560ec08a6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.257+0000 7fc79df74740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc7901afd60 con 0x560ec04c64d0 2024-04-25T11:43:11.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.257+0000 7fc79df74740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc7901b18a0 con 0x560ec04ef390 2024-04-25T11:43:11.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.257+0000 7fc79df74740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc7901b0430 con 0x560ec04ce6b0 2024-04-25T11:43:11.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.257+0000 7fc794d8a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04c64d0 0x560ec08967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.257+0000 7fc794d8a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04c64d0 0x560ec08967f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52884/0 (socket says 172.21.15.90:52884) 2024-04-25T11:43:11.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.257+0000 7fc794d8a640 1 -- 172.21.15.90:0/18336710 learned_addr learned my addr 172.21.15.90:0/18336710 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:11.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.257+0000 7fc79558b640 1 --2- 172.21.15.90:0/18336710 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ec04ef390 0x560ec08a6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.257+0000 7fc78ffff640 1 --2- 172.21.15.90:0/18336710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ec04ce6b0 0x560ec088ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.257+0000 7fc794d8a640 1 -- 172.21.15.90:0/18336710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ec04ce6b0 msgr2=0x560ec088ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:11.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.257+0000 7fc794d8a640 1 --2- 172.21.15.90:0/18336710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ec04ce6b0 0x560ec088ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.257+0000 7fc794d8a640 1 -- 172.21.15.90:0/18336710 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ec04ef390 msgr2=0x560ec08a6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:11.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.257+0000 7fc794d8a640 1 --2- 172.21.15.90:0/18336710 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ec04ef390 0x560ec08a6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.257+0000 7fc794d8a640 1 -- 172.21.15.90:0/18336710 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ec088d410 con 0x560ec04c64d0 2024-04-25T11:43:11.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.258+0000 7fc794d8a640 1 --2- 172.21.15.90:0/18336710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04c64d0 0x560ec08967f0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fc7840fcbc0 tx=0x7fc78402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6713b9cae3c847a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:11.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.258+0000 7fc77a7fc640 1 -- 172.21.15.90:0/18336710 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc784049040 con 0x560ec04c64d0 2024-04-25T11:43:11.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.258+0000 7fc77a7fc640 1 -- 172.21.15.90:0/18336710 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7840c70c0 con 0x560ec04c64d0 2024-04-25T11:43:11.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.258+0000 7fc77a7fc640 1 -- 172.21.15.90:0/18336710 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7840cb960 con 0x560ec04c64d0 2024-04-25T11:43:11.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.259+0000 7fc79df74740 1 -- 172.21.15.90:0/18336710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04c64d0 msgr2=0x560ec08967f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.259+0000 7fc79df74740 1 --2- 172.21.15.90:0/18336710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04c64d0 0x560ec08967f0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fc7840fcbc0 tx=0x7fc78402a410 comp rx=0 tx=0).stop 2024-04-25T11:43:11.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.259+0000 7fc79df74740 1 -- 172.21.15.90:0/18336710 shutdown_connections 2024-04-25T11:43:11.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.259+0000 7fc79df74740 1 --2- 172.21.15.90:0/18336710 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ec04ef390 0x560ec08a6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.259+0000 7fc79df74740 1 --2- 172.21.15.90:0/18336710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ec04ce6b0 0x560ec088ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.259+0000 7fc79df74740 1 --2- 172.21.15.90:0/18336710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04c64d0 0x560ec08967f0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.259+0000 7fc79df74740 1 -- 172.21.15.90:0/18336710 >> 172.21.15.90:0/18336710 conn(0x560ec04e34e0 msgr2=0x560ec088c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:11.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.259+0000 7fc79df74740 1 -- 172.21.15.90:0/18336710 shutdown_connections 2024-04-25T11:43:11.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.259+0000 7fc79df74740 1 -- 172.21.15.90:0/18336710 wait complete. 2024-04-25T11:43:11.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.259+0000 7fc79df74740 1 -- start start 2024-04-25T11:43:11.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.259+0000 7fc79df74740 1 Processor -- start 2024-04-25T11:43:11.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.260+0000 7fc79df74740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ec04c64d0 0x560ec089e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.260+0000 7fc79df74740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ec04ce6b0 0x560ec089e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.260+0000 7fc79df74740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04ef390 0x560ec0913a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.260+0000 7fc79df74740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc7901b11d0 con 0x560ec04ef390 2024-04-25T11:43:11.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.260+0000 7fc79df74740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc7901b0b00 con 0x560ec04c64d0 2024-04-25T11:43:11.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.260+0000 7fc79df74740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc7901add60 con 0x560ec04ce6b0 2024-04-25T11:43:11.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.260+0000 7fc79df74740 10 client.0 _ll_register_callbacks cb 0x7fc788009e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:11.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.260+0000 7fc79558b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04ef390 0x560ec0913a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.260+0000 7fc79558b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04ef390 0x560ec0913a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52894/0 (socket says 172.21.15.90:52894) 2024-04-25T11:43:11.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.260+0000 7fc79558b640 1 -- 172.21.15.90:0/2435063838 learned_addr learned my addr 172.21.15.90:0/2435063838 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:11.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.260+0000 7fc794d8a640 1 --2- 172.21.15.90:0/2435063838 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ec04c64d0 0x560ec089e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.260+0000 7fc78ffff640 1 --2- 172.21.15.90:0/2435063838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ec04ce6b0 0x560ec089e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.260+0000 7fc79558b640 1 -- 172.21.15.90:0/2435063838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ec04ce6b0 msgr2=0x560ec089e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:11.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.260+0000 7fc79558b640 1 --2- 172.21.15.90:0/2435063838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ec04ce6b0 0x560ec089e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.260+0000 7fc79558b640 1 -- 172.21.15.90:0/2435063838 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ec04c64d0 msgr2=0x560ec089e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:11.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.260+0000 7fc79558b640 1 --2- 172.21.15.90:0/2435063838 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ec04c64d0 0x560ec089e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.260+0000 7fc79558b640 1 -- 172.21.15.90:0/2435063838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ec0913f70 con 0x560ec04ef390 2024-04-25T11:43:11.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.260+0000 7fc78ffff640 1 --2- 172.21.15.90:0/2435063838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ec04ce6b0 0x560ec089e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:11.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.260+0000 7fc79558b640 1 --2- 172.21.15.90:0/2435063838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04ef390 0x560ec0913a30 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fc7880105c0 tx=0x7fc788009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f670214c9fef73 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:11.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.260+0000 7fc76ffff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:11.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.260+0000 7fc76ffff640 1 -- 172.21.15.90:0/2435063838 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78803d060 con 0x560ec04ef390 2024-04-25T11:43:11.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.261+0000 7fc76ffff640 1 -- 172.21.15.90:0/2435063838 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7fc78801e420 con 0x560ec04ef390 2024-04-25T11:43:11.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.261+0000 7fc79df74740 1 -- 172.21.15.90:0/2435063838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7840c70c0 con 0x560ec04ef390 2024-04-25T11:43:11.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.261+0000 7fc76ffff640 1 -- 172.21.15.90:0/2435063838 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc788009590 con 0x560ec04ef390 2024-04-25T11:43:11.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.261+0000 7fc79df74740 10 client.18114 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:11.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.261+0000 7fc79df74740 1 -- 172.21.15.90:0/2435063838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560ec0915070 con 0x560ec04ef390 2024-04-25T11:43:11.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.261+0000 7fc76ffff640 1 -- 172.21.15.90:0/2435063838 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7840c70c0 con 0x560ec04ef390 2024-04-25T11:43:11.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.261+0000 7fc76ffff640 1 -- 172.21.15.90:0/2435063838 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fc78801e7c0 con 0x560ec04ef390 2024-04-25T11:43:11.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.261+0000 7fc79df74740 10 client.18114 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:11.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.261+0000 7fc79df74740 10 client.18114 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:43:11.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.261+0000 7fc79df74740 20 client.18114 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:11.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.261+0000 7fc79df74740 1 --2- 172.21.15.90:0/2435063838 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x560ec04833a0 0x560ec0917800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.261+0000 7fc79df74740 4 client.18114 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:43:11.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.261+0000 7fc79df74740 1 -- 172.21.15.90:0/2435063838 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560ec07fd800 con 0x560ec04833a0 2024-04-25T11:43:11.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.261+0000 7fc794d8a640 1 --2- 172.21.15.90:0/2435063838 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x560ec04833a0 0x560ec0917800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.262+0000 7fc76ffff640 10 client.18114 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:43:11.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.262+0000 7fc794d8a640 1 --2- 172.21.15.90:0/2435063838 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x560ec04833a0 0x560ec0917800 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bae7c58c2118cd06 server_cookie=42353d36e1b7dfc3 in_seq=0 out_seq=0 2024-04-25T11:43:11.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.262+0000 7fc76ffff640 1 -- 172.21.15.90:0/2435063838 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560ec07fd800 con 0x560ec04833a0 2024-04-25T11:43:11.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.262+0000 7fc76ffff640 10 client.18114 handle_command_reply: tid=0 2024-04-25T11:43:11.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.262+0000 7fc79df74740 1 client.18114 shutdown 2024-04-25T11:43:11.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.262+0000 7fc79df74740 1 -- 172.21.15.90:0/2435063838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04ef390 msgr2=0x560ec0913a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.262+0000 7fc79df74740 1 --2- 172.21.15.90:0/2435063838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04ef390 0x560ec0913a30 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fc7880105c0 tx=0x7fc788009500 comp rx=0 tx=0).stop 2024-04-25T11:43:11.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.262+0000 7fc79df74740 1 -- 172.21.15.90:0/2435063838 shutdown_connections 2024-04-25T11:43:11.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.262+0000 7fc79df74740 1 --2- 172.21.15.90:0/2435063838 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x560ec04833a0 0x560ec0917800 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.262+0000 7fc79df74740 1 --2- 172.21.15.90:0/2435063838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560ec04ef390 0x560ec0913a30 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.262+0000 7fc79df74740 1 --2- 172.21.15.90:0/2435063838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ec04ce6b0 0x560ec089e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.262+0000 7fc76ffff640 0 client.18114 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:43:11.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.262+0000 7fc79df74740 1 --2- 172.21.15.90:0/2435063838 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560ec04c64d0 0x560ec089e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.262+0000 7fc79df74740 1 -- 172.21.15.90:0/2435063838 >> 172.21.15.90:0/2435063838 conn(0x560ec04e34e0 msgr2=0x560ec08998c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:11.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.262+0000 7fc79df74740 1 -- 172.21.15.90:0/2435063838 shutdown_connections 2024-04-25T11:43:11.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.262+0000 7fc79df74740 1 -- 172.21.15.90:0/2435063838 wait complete. 2024-04-25T11:43:11.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.263+0000 7fc79df74740 20 client.18114 trim_cache size 0 max 16384 2024-04-25T11:43:11.254 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:43:11.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.263+0000 7fc796fee640 1 -- 172.21.15.90:0/3830192272 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc7800afc60 msgr2=0x7fc7800b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.263+0000 7fc796fee640 1 --2- 172.21.15.90:0/3830192272 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc7800afc60 0x7fc7800b2170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc788009fd0 tx=0x7fc788004a40 comp rx=0 tx=0).stop 2024-04-25T11:43:11.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.263+0000 7fc796fee640 1 -- 172.21.15.90:0/3830192272 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc790102af0 msgr2=0x7fc79019a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.263+0000 7fc796fee640 1 --2- 172.21.15.90:0/3830192272 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc790102af0 0x7fc79019a760 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fc784032fd0 tx=0x7fc784031a20 comp rx=0 tx=0).stop 2024-04-25T11:43:11.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.263+0000 7fc796fee640 1 -- 172.21.15.90:0/3830192272 shutdown_connections 2024-04-25T11:43:11.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.263+0000 7fc796fee640 1 --2- 172.21.15.90:0/3830192272 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc7800afc60 0x7fc7800b2170 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.263+0000 7fc796fee640 1 --2- 172.21.15.90:0/3830192272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc79010d380 0x7fc79019ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.263+0000 7fc796fee640 1 --2- 172.21.15.90:0/3830192272 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc790108a20 0x7fc79019acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.263+0000 7fc796fee640 1 --2- 172.21.15.90:0/3830192272 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc790102af0 0x7fc79019a760 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.263+0000 7fc796fee640 1 -- 172.21.15.90:0/3830192272 >> 172.21.15.90:0/3830192272 conn(0x7fc790078100 msgr2=0x7fc7901040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:11.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.263+0000 7fc796fee640 1 -- 172.21.15.90:0/3830192272 shutdown_connections 2024-04-25T11:43:11.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.263+0000 7fc796fee640 1 -- 172.21.15.90:0/3830192272 wait complete. 2024-04-25T11:43:11.261 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T11:43:11.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:11 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1165227036' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:43:11.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:11 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1165227036' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:43:11.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.398+0000 7f3e087ee640 1 -- 172.21.15.90:0/3404461059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e0010d3a0 msgr2=0x7f3e0010f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.398+0000 7f3e087ee640 1 --2- 172.21.15.90:0/3404461059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e0010d3a0 0x7f3e0010f7c0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f3df8009ca0 tx=0x7f3df802a8c0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.398+0000 7f3e087ee640 1 -- 172.21.15.90:0/3404461059 shutdown_connections 2024-04-25T11:43:11.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.398+0000 7f3e087ee640 1 --2- 172.21.15.90:0/3404461059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e0010d3a0 0x7f3e0010f7c0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.398+0000 7f3e087ee640 1 --2- 172.21.15.90:0/3404461059 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3e00102b10 0x7f3e00102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.398+0000 7f3e087ee640 1 --2- 172.21.15.90:0/3404461059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e00108a40 0x7f3e001025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.398+0000 7f3e087ee640 1 -- 172.21.15.90:0/3404461059 >> 172.21.15.90:0/3404461059 conn(0x7f3e00078180 msgr2=0x7f3e000785c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:11.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.398+0000 7f3e087ee640 1 -- 172.21.15.90:0/3404461059 shutdown_connections 2024-04-25T11:43:11.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.399+0000 7f3e087ee640 1 -- 172.21.15.90:0/3404461059 wait complete. 2024-04-25T11:43:11.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.399+0000 7f3e087ee640 1 Processor -- start 2024-04-25T11:43:11.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.399+0000 7f3e087ee640 1 -- start start 2024-04-25T11:43:11.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.400+0000 7f3e087ee640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e00102b10 0x7f3e0019b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.400+0000 7f3e087ee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e00108a40 0x7f3e0019ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.400+0000 7f3e087ee640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3e0010d3a0 0x7f3e0019bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.400+0000 7f3e087ee640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3e00112c20 con 0x7f3e00102b10 2024-04-25T11:43:11.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.400+0000 7f3e087ee640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3e00112aa0 con 0x7f3e0010d3a0 2024-04-25T11:43:11.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.400+0000 7f3e087ee640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3e00112da0 con 0x7f3e00108a40 2024-04-25T11:43:11.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.400+0000 7f3e0658a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e00102b10 0x7f3e0019b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.400+0000 7f3e0658a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e00102b10 0x7f3e0019b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52908/0 (socket says 172.21.15.90:52908) 2024-04-25T11:43:11.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.400+0000 7f3e0658a640 1 -- 172.21.15.90:0/1184052452 learned_addr learned my addr 172.21.15.90:0/1184052452 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:11.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.400+0000 7f3e06d8b640 1 --2- 172.21.15.90:0/1184052452 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3e0010d3a0 0x7f3e0019bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.400+0000 7f3e05d89640 1 --2- 172.21.15.90:0/1184052452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e00108a40 0x7f3e0019ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.400+0000 7f3e0658a640 1 -- 172.21.15.90:0/1184052452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e00108a40 msgr2=0x7f3e0019ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.400+0000 7f3e0658a640 1 --2- 172.21.15.90:0/1184052452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e00108a40 0x7f3e0019ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.400+0000 7f3e0658a640 1 -- 172.21.15.90:0/1184052452 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3e0010d3a0 msgr2=0x7f3e0019bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.400+0000 7f3e0658a640 1 --2- 172.21.15.90:0/1184052452 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3e0010d3a0 0x7f3e0019bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.400+0000 7f3e0658a640 1 -- 172.21.15.90:0/1184052452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e001a67d0 con 0x7f3e00102b10 2024-04-25T11:43:11.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.400+0000 7f3e0658a640 1 --2- 172.21.15.90:0/1184052452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e00102b10 0x7f3e0019b4c0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc00d740 tx=0x7f3dfc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:11.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.400+0000 7f3df77fe640 1 -- 172.21.15.90:0/1184052452 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc017070 con 0x7f3e00102b10 2024-04-25T11:43:11.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.400+0000 7f3df77fe640 1 -- 172.21.15.90:0/1184052452 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3dfc00de50 con 0x7f3e00102b10 2024-04-25T11:43:11.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.400+0000 7f3e087ee640 1 -- 172.21.15.90:0/1184052452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e001a6ac0 con 0x7f3e00102b10 2024-04-25T11:43:11.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.400+0000 7f3df77fe640 1 -- 172.21.15.90:0/1184052452 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc004ce0 con 0x7f3e00102b10 2024-04-25T11:43:11.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.400+0000 7f3e087ee640 1 -- 172.21.15.90:0/1184052452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e001a6fc0 con 0x7f3e00102b10 2024-04-25T11:43:11.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.403+0000 7f3df77fe640 1 -- 172.21.15.90:0/1184052452 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f3dfc004e80 con 0x7f3e00102b10 2024-04-25T11:43:11.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.403+0000 7f3e0f773740 1 Processor -- start 2024-04-25T11:43:11.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.404+0000 7f3e0f773740 1 -- start start 2024-04-25T11:43:11.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.404+0000 7f3e0f773740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626489e0390 0x562648d816a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.404+0000 7f3e0f773740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626489b74d0 0x562648d81be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.404+0000 7f3df77fe640 1 --2- 172.21.15.90:0/1184052452 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3dec0afc60 0x7f3dec0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.404+0000 7f3e0f773740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626489bf6b0 0x562648d92ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.404+0000 7f3e0f773740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562648ce6d80 con 0x5626489bf6b0 2024-04-25T11:43:11.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.404+0000 7f3e0f773740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x562648ca0a20 con 0x5626489e0390 2024-04-25T11:43:11.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.404+0000 7f3e0f773740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562648ce2d40 con 0x5626489b74d0 2024-04-25T11:43:11.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.404+0000 7f3e06d8b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626489bf6b0 0x562648d92ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.404+0000 7f3e05d89640 1 --2- 172.21.15.90:0/1184052452 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3dec0afc60 0x7f3dec0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.404+0000 7f3e06d8b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626489bf6b0 0x562648d92ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52910/0 (socket says 172.21.15.90:52910) 2024-04-25T11:43:11.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.404+0000 7f3e06d8b640 1 -- 172.21.15.90:0/4065498431 learned_addr learned my addr 172.21.15.90:0/4065498431 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:11.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.404+0000 7f3e0658a640 1 --2- 172.21.15.90:0/4065498431 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626489b74d0 0x562648d81be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.404+0000 7f3e06d8b640 1 --2- 172.21.15.90:0/4065498431 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626489e0390 0x562648d816a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.404+0000 7f3df77fe640 1 -- 172.21.15.90:0/1184052452 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7f3dfc0de980 con 0x7f3e00102b10 2024-04-25T11:43:11.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.405+0000 7f3e0658a640 1 -- 172.21.15.90:0/4065498431 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626489e0390 msgr2=0x562648d816a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:11.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.405+0000 7f3e0658a640 1 --2- 172.21.15.90:0/4065498431 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626489e0390 0x562648d816a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.405+0000 7f3e0658a640 1 -- 172.21.15.90:0/4065498431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626489bf6b0 msgr2=0x562648d92ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:11.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.405+0000 7f3e0658a640 1 --2- 172.21.15.90:0/4065498431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626489bf6b0 0x562648d92ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.405+0000 7f3e0658a640 1 -- 172.21.15.90:0/4065498431 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562648cabc30 con 0x5626489b74d0 2024-04-25T11:43:11.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.405+0000 7f3e06d8b640 1 --2- 172.21.15.90:0/4065498431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626489bf6b0 0x562648d92ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:43:11.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.405+0000 7f3e05d89640 1 --2- 172.21.15.90:0/1184052452 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3dec0afc60 0x7f3dec0b2170 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3e0010e860 tx=0x7f3df000b390 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:11.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.405+0000 7f3e0658a640 1 --2- 172.21.15.90:0/4065498431 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626489b74d0 0x562648d81be0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc0dc020 tx=0x7f3dfc0a6970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba1d0847a7a408e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:11.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.405+0000 7f3df4ff9640 1 -- 172.21.15.90:0/4065498431 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc020070 con 0x5626489b74d0 2024-04-25T11:43:11.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.405+0000 7f3df4ff9640 1 -- 172.21.15.90:0/4065498431 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3dfc0af210 con 0x5626489b74d0 2024-04-25T11:43:11.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.405+0000 7f3df4ff9640 1 -- 172.21.15.90:0/4065498431 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc0a7da0 con 0x5626489b74d0 2024-04-25T11:43:11.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.406+0000 7f3e0f773740 1 -- 172.21.15.90:0/4065498431 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626489b74d0 msgr2=0x562648d81be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.406+0000 7f3e0f773740 1 --2- 172.21.15.90:0/4065498431 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626489b74d0 0x562648d81be0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc0dc020 tx=0x7f3dfc0a6970 comp rx=0 tx=0).stop 2024-04-25T11:43:11.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.406+0000 7f3e0f773740 1 -- 172.21.15.90:0/4065498431 shutdown_connections 2024-04-25T11:43:11.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.406+0000 7f3e0f773740 1 --2- 172.21.15.90:0/4065498431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626489bf6b0 0x562648d92ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.406+0000 7f3e0f773740 1 --2- 172.21.15.90:0/4065498431 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626489b74d0 0x562648d81be0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.406+0000 7f3e0f773740 1 --2- 172.21.15.90:0/4065498431 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626489e0390 0x562648d816a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.406+0000 7f3e0f773740 1 -- 172.21.15.90:0/4065498431 >> 172.21.15.90:0/4065498431 conn(0x5626489d44e0 msgr2=0x562648d7dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:11.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.406+0000 7f3e0f773740 1 -- 172.21.15.90:0/4065498431 shutdown_connections 2024-04-25T11:43:11.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.406+0000 7f3e0f773740 1 -- 172.21.15.90:0/4065498431 wait complete. 2024-04-25T11:43:11.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.407+0000 7f3e0f773740 1 -- start start 2024-04-25T11:43:11.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.407+0000 7f3e0f773740 1 Processor -- start 2024-04-25T11:43:11.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.407+0000 7f3e0f773740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626489b74d0 0x562648d90630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.407+0000 7f3e0f773740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626489bf6b0 0x562648d90b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.407+0000 7f3e0f773740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626489e0390 0x562648d9c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.407+0000 7f3e0f773740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562648d93140 con 0x5626489bf6b0 2024-04-25T11:43:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.407+0000 7f3e0f773740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x562648c6ddc0 con 0x5626489e0390 2024-04-25T11:43:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.407+0000 7f3e0f773740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562648d932c0 con 0x5626489b74d0 2024-04-25T11:43:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.407+0000 7f3e0f773740 10 client.0 _ll_register_callbacks cb 0x562648c89600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.407+0000 7f3e05d89640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626489bf6b0 0x562648d90b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.407+0000 7f3e05d89640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626489bf6b0 0x562648d90b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52916/0 (socket says 172.21.15.90:52916) 2024-04-25T11:43:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.407+0000 7f3e05d89640 1 -- 172.21.15.90:0/1409413957 learned_addr learned my addr 172.21.15.90:0/1409413957 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.407+0000 7f3e0658a640 1 --2- 172.21.15.90:0/1409413957 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626489b74d0 0x562648d90630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.407+0000 7f3e06d8b640 1 --2- 172.21.15.90:0/1409413957 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626489e0390 0x562648d9c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.408+0000 7f3e05d89640 1 -- 172.21.15.90:0/1409413957 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626489b74d0 msgr2=0x562648d90630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.408+0000 7f3e05d89640 1 --2- 172.21.15.90:0/1409413957 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626489b74d0 0x562648d90630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.408+0000 7f3e05d89640 1 -- 172.21.15.90:0/1409413957 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626489e0390 msgr2=0x562648d9c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.408+0000 7f3e05d89640 1 --2- 172.21.15.90:0/1409413957 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626489e0390 0x562648d9c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.408+0000 7f3e05d89640 1 -- 172.21.15.90:0/1409413957 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626489f83f0 con 0x5626489bf6b0 2024-04-25T11:43:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.408+0000 7f3e05d89640 1 --2- 172.21.15.90:0/1409413957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626489bf6b0 0x562648d90b70 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f3df000b6e0 tx=0x7f3df0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c2013ea4b7888da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:11.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.408+0000 7f3de9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.408+0000 7f3de9ffb640 1 -- 172.21.15.90:0/1409413957 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df001e8f0 con 0x5626489bf6b0 2024-04-25T11:43:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.408+0000 7f3de9ffb640 1 -- 172.21.15.90:0/1409413957 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7f3df001ea90 con 0x5626489bf6b0 2024-04-25T11:43:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.408+0000 7f3e0f773740 1 -- 172.21.15.90:0/1409413957 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562648cf4990 con 0x5626489bf6b0 2024-04-25T11:43:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.408+0000 7f3de9ffb640 1 -- 172.21.15.90:0/1409413957 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3df001ec90 con 0x5626489bf6b0 2024-04-25T11:43:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.408+0000 7f3e0f773740 10 client.18126 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.408+0000 7f3e0f773740 1 -- 172.21.15.90:0/1409413957 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562648ac27d0 con 0x5626489bf6b0 2024-04-25T11:43:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.408+0000 7f3de9ffb640 1 -- 172.21.15.90:0/1409413957 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x562648cf4990 con 0x5626489bf6b0 2024-04-25T11:43:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.408+0000 7f3de9ffb640 1 -- 172.21.15.90:0/1409413957 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f3df0026490 con 0x5626489bf6b0 2024-04-25T11:43:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.408+0000 7f3e0f773740 10 client.18126 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.408+0000 7f3e0f773740 10 client.18126 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:43:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.409+0000 7f3e0f773740 20 client.18126 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.409+0000 7f3e0f773740 1 --2- 172.21.15.90:0/1409413957 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56264899ef60 0x562648da2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.409+0000 7f3e0f773740 4 client.18126 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:43:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.409+0000 7f3e0f773740 1 -- 172.21.15.90:0/1409413957 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562648cee800 con 0x56264899ef60 2024-04-25T11:43:11.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.409+0000 7f3e0658a640 1 --2- 172.21.15.90:0/1409413957 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56264899ef60 0x562648da2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.409+0000 7f3e0658a640 1 --2- 172.21.15.90:0/1409413957 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56264899ef60 0x562648da2b80 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca4784a3118f2018 server_cookie=9d9e7aa6289514da in_seq=0 out_seq=0 2024-04-25T11:43:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.409+0000 7f3de9ffb640 10 client.18126 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.410+0000 7f3de9ffb640 1 -- 172.21.15.90:0/1409413957 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562648cee800 con 0x56264899ef60 2024-04-25T11:43:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.410+0000 7f3de9ffb640 10 client.18126 handle_command_reply: tid=0 2024-04-25T11:43:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.410+0000 7f3e0f773740 1 client.18126 shutdown 2024-04-25T11:43:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.410+0000 7f3e0f773740 1 -- 172.21.15.90:0/1409413957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626489bf6b0 msgr2=0x562648d90b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.410+0000 7f3e0f773740 1 --2- 172.21.15.90:0/1409413957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626489bf6b0 0x562648d90b70 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f3df000b6e0 tx=0x7f3df0004730 comp rx=0 tx=0).stop 2024-04-25T11:43:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.410+0000 7f3e0f773740 1 -- 172.21.15.90:0/1409413957 shutdown_connections 2024-04-25T11:43:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.410+0000 7f3e0f773740 1 --2- 172.21.15.90:0/1409413957 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56264899ef60 0x562648da2b80 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.410+0000 7f3e0f773740 1 --2- 172.21.15.90:0/1409413957 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626489e0390 0x562648d9c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.410+0000 7f3e0f773740 1 --2- 172.21.15.90:0/1409413957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626489bf6b0 0x562648d90b70 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.410+0000 7f3de9ffb640 0 client.18126 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.410+0000 7f3e0f773740 1 --2- 172.21.15.90:0/1409413957 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626489b74d0 0x562648d90630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.410+0000 7f3e0f773740 1 -- 172.21.15.90:0/1409413957 >> 172.21.15.90:0/1409413957 conn(0x5626489d44e0 msgr2=0x562648d83480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:11.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.410+0000 7f3e0f773740 1 -- 172.21.15.90:0/1409413957 shutdown_connections 2024-04-25T11:43:11.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.410+0000 7f3e0f773740 1 -- 172.21.15.90:0/1409413957 wait complete. 2024-04-25T11:43:11.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.410+0000 7f3e0f773740 20 client.18126 trim_cache size 0 max 16384 2024-04-25T11:43:11.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.425+0000 7f3e0f773740 1 Processor -- start 2024-04-25T11:43:11.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.425+0000 7f3e0f773740 1 -- start start 2024-04-25T11:43:11.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.425+0000 7f3e0f773740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56264899ef60 0x562648d9c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.425+0000 7f3e0f773740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626489b74d0 0x562648da8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.425+0000 7f3e0f773740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626489bf6b0 0x562648da8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.425+0000 7f3e0f773740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3e001adaf0 con 0x5626489b74d0 2024-04-25T11:43:11.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.425+0000 7f3e0f773740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3e001af630 con 0x56264899ef60 2024-04-25T11:43:11.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.425+0000 7f3e0f773740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3e001ae1c0 con 0x5626489bf6b0 2024-04-25T11:43:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.425+0000 7f3e05d89640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626489b74d0 0x562648da8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.425+0000 7f3e05d89640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626489b74d0 0x562648da8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52734/0 (socket says 172.21.15.90:52734) 2024-04-25T11:43:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.425+0000 7f3e05d89640 1 -- 172.21.15.90:0/1232550534 learned_addr learned my addr 172.21.15.90:0/1232550534 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.425+0000 7f3e0658a640 1 --2- 172.21.15.90:0/1232550534 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56264899ef60 0x562648d9c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.425+0000 7f3e06d8b640 1 --2- 172.21.15.90:0/1232550534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626489bf6b0 0x562648da8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.426+0000 7f3e05d89640 1 -- 172.21.15.90:0/1232550534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626489bf6b0 msgr2=0x562648da8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.426+0000 7f3e05d89640 1 --2- 172.21.15.90:0/1232550534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626489bf6b0 0x562648da8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.426+0000 7f3e05d89640 1 -- 172.21.15.90:0/1232550534 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56264899ef60 msgr2=0x562648d9c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.426+0000 7f3e05d89640 1 --2- 172.21.15.90:0/1232550534 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56264899ef60 0x562648d9c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.426+0000 7f3e05d89640 1 -- 172.21.15.90:0/1232550534 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562648d8e970 con 0x5626489b74d0 2024-04-25T11:43:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.426+0000 7f3e0658a640 1 --2- 172.21.15.90:0/1232550534 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56264899ef60 0x562648d9c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.426+0000 7f3e05d89640 1 --2- 172.21.15.90:0/1232550534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626489b74d0 0x562648da8340 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f3df0000c00 tx=0x7f3df001ffd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c972d8ae0020ca1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.426+0000 7f3debfff640 1 -- 172.21.15.90:0/1232550534 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df003a030 con 0x5626489b74d0 2024-04-25T11:43:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.426+0000 7f3debfff640 1 -- 172.21.15.90:0/1232550534 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3df0010e40 con 0x5626489b74d0 2024-04-25T11:43:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.426+0000 7f3debfff640 1 -- 172.21.15.90:0/1232550534 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df0004280 con 0x5626489b74d0 2024-04-25T11:43:11.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.427+0000 7f3e0f773740 1 -- 172.21.15.90:0/1232550534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626489b74d0 msgr2=0x562648da8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.427+0000 7f3e0f773740 1 --2- 172.21.15.90:0/1232550534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626489b74d0 0x562648da8340 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f3df0000c00 tx=0x7f3df001ffd0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.427+0000 7f3e0f773740 1 -- 172.21.15.90:0/1232550534 shutdown_connections 2024-04-25T11:43:11.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.427+0000 7f3e0f773740 1 --2- 172.21.15.90:0/1232550534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626489bf6b0 0x562648da8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.427+0000 7f3e0f773740 1 --2- 172.21.15.90:0/1232550534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626489b74d0 0x562648da8340 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.427+0000 7f3e0f773740 1 --2- 172.21.15.90:0/1232550534 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56264899ef60 0x562648d9c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.427+0000 7f3e0f773740 1 -- 172.21.15.90:0/1232550534 >> 172.21.15.90:0/1232550534 conn(0x5626489d44e0 msgr2=0x562648d815a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:11.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.427+0000 7f3e0f773740 1 -- 172.21.15.90:0/1232550534 shutdown_connections 2024-04-25T11:43:11.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.427+0000 7f3e0f773740 1 -- 172.21.15.90:0/1232550534 wait complete. 2024-04-25T11:43:11.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.427+0000 7f3e0f773740 1 -- start start 2024-04-25T11:43:11.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.427+0000 7f3e0f773740 1 Processor -- start 2024-04-25T11:43:11.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.428+0000 7f3e0f773740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56264899ef60 0x562648d93900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.428+0000 7f3e0f773740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626489b74d0 0x562648d93e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.428+0000 7f3e0f773740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626489bf6b0 0x562648e08ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.428+0000 7f3e0f773740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3e001aef60 con 0x56264899ef60 2024-04-25T11:43:11.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.428+0000 7f3e0f773740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3e001ae890 con 0x5626489bf6b0 2024-04-25T11:43:11.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.428+0000 7f3e0f773740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3e001abaf0 con 0x5626489b74d0 2024-04-25T11:43:11.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.428+0000 7f3e0f773740 10 client.0 _ll_register_callbacks cb 0x7f3df000b910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:11.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.428+0000 7f3e0658a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56264899ef60 0x562648d93900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.428+0000 7f3e0658a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56264899ef60 0x562648d93900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52748/0 (socket says 172.21.15.90:52748) 2024-04-25T11:43:11.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.428+0000 7f3e0658a640 1 -- 172.21.15.90:0/1688471363 learned_addr learned my addr 172.21.15.90:0/1688471363 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:11.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.428+0000 7f3e05d89640 1 --2- 172.21.15.90:0/1688471363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626489b74d0 0x562648d93e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.428+0000 7f3e06d8b640 1 --2- 172.21.15.90:0/1688471363 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626489bf6b0 0x562648e08ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.428+0000 7f3e0658a640 1 -- 172.21.15.90:0/1688471363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626489b74d0 msgr2=0x562648d93e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:11.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.428+0000 7f3e0658a640 1 --2- 172.21.15.90:0/1688471363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626489b74d0 0x562648d93e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.428+0000 7f3e0658a640 1 -- 172.21.15.90:0/1688471363 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626489bf6b0 msgr2=0x562648e08ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:11.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.428+0000 7f3e0658a640 1 --2- 172.21.15.90:0/1688471363 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626489bf6b0 0x562648e08ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.428+0000 7f3e0658a640 1 -- 172.21.15.90:0/1688471363 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562648e09420 con 0x56264899ef60 2024-04-25T11:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.428+0000 7f3e05d89640 1 --2- 172.21.15.90:0/1688471363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626489b74d0 0x562648d93e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.428+0000 7f3e0658a640 1 --2- 172.21.15.90:0/1688471363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56264899ef60 0x562648d93900 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc0a6820 tx=0x7f3dfc005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17112d5c752dcb8d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.428+0000 7f3de9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.428+0000 7f3de9ffb640 1 -- 172.21.15.90:0/1688471363 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc020040 con 0x56264899ef60 2024-04-25T11:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.428+0000 7f3de9ffb640 1 -- 172.21.15.90:0/1688471363 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7f3dfc0dc2a0 con 0x56264899ef60 2024-04-25T11:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.429+0000 7f3e0f773740 1 -- 172.21.15.90:0/1688471363 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3df0010e40 con 0x56264899ef60 2024-04-25T11:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.429+0000 7f3de9ffb640 1 -- 172.21.15.90:0/1688471363 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3dfc0aae20 con 0x56264899ef60 2024-04-25T11:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.429+0000 7f3e0f773740 10 client.18132 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.429+0000 7f3e0f773740 1 -- 172.21.15.90:0/1688471363 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562648e0a650 con 0x56264899ef60 2024-04-25T11:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.429+0000 7f3de9ffb640 1 -- 172.21.15.90:0/1688471363 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3df0010e40 con 0x56264899ef60 2024-04-25T11:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.429+0000 7f3de9ffb640 1 -- 172.21.15.90:0/1688471363 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f3dfc0af440 con 0x56264899ef60 2024-04-25T11:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.429+0000 7f3e0f773740 10 client.18132 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.429+0000 7f3e0f773740 10 client.18132 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.429+0000 7f3e0f773740 20 client.18132 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.429+0000 7f3e0f773740 1 --2- 172.21.15.90:0/1688471363 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5626489e0390 0x562648e0c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.429+0000 7f3e0f773740 4 client.18132 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:43:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.429+0000 7f3e0f773740 1 -- 172.21.15.90:0/1688471363 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562648c69610 con 0x5626489e0390 2024-04-25T11:43:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.429+0000 7f3e05d89640 1 --2- 172.21.15.90:0/1688471363 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5626489e0390 0x562648e0c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.430+0000 7f3de9ffb640 10 client.18132 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.430+0000 7f3e05d89640 1 --2- 172.21.15.90:0/1688471363 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5626489e0390 0x562648e0c940 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81711df8cd0b23a6 server_cookie=3c5f3b842efcc307 in_seq=0 out_seq=0 2024-04-25T11:43:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.430+0000 7f3de9ffb640 1 -- 172.21.15.90:0/1688471363 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562648c69610 con 0x5626489e0390 2024-04-25T11:43:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.430+0000 7f3de9ffb640 10 client.18132 handle_command_reply: tid=0 2024-04-25T11:43:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.430+0000 7f3e0f773740 1 client.18132 shutdown 2024-04-25T11:43:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.430+0000 7f3e0f773740 1 -- 172.21.15.90:0/1688471363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56264899ef60 msgr2=0x562648d93900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.430+0000 7f3e0f773740 1 --2- 172.21.15.90:0/1688471363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56264899ef60 0x562648d93900 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc0a6820 tx=0x7f3dfc005bf0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.430+0000 7f3e0f773740 1 -- 172.21.15.90:0/1688471363 shutdown_connections 2024-04-25T11:43:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.430+0000 7f3e0f773740 1 --2- 172.21.15.90:0/1688471363 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5626489e0390 0x562648e0c940 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.430+0000 7f3e0f773740 1 --2- 172.21.15.90:0/1688471363 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626489bf6b0 0x562648e08ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.430+0000 7f3e0f773740 1 --2- 172.21.15.90:0/1688471363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626489b74d0 0x562648d93e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.430+0000 7f3de9ffb640 0 client.18132 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:11.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.430+0000 7f3e0f773740 1 --2- 172.21.15.90:0/1688471363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56264899ef60 0x562648d93900 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.430+0000 7f3e0f773740 1 -- 172.21.15.90:0/1688471363 >> 172.21.15.90:0/1688471363 conn(0x5626489d44e0 msgr2=0x562648d7d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:11.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.430+0000 7f3e0f773740 1 -- 172.21.15.90:0/1688471363 shutdown_connections 2024-04-25T11:43:11.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.430+0000 7f3e0f773740 1 -- 172.21.15.90:0/1688471363 wait complete. 2024-04-25T11:43:11.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.430+0000 7f3e0f773740 20 client.18132 trim_cache size 0 max 16384 2024-04-25T11:43:11.423 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:43:11.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.431+0000 7f3e087ee640 1 -- 172.21.15.90:0/1184052452 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3dec0afc60 msgr2=0x7f3dec0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.431+0000 7f3e087ee640 1 --2- 172.21.15.90:0/1184052452 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3dec0afc60 0x7f3dec0b2170 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3e0010e860 tx=0x7f3df000b390 comp rx=0 tx=0).stop 2024-04-25T11:43:11.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.431+0000 7f3e087ee640 1 -- 172.21.15.90:0/1184052452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e00102b10 msgr2=0x7f3e0019b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.431+0000 7f3e087ee640 1 --2- 172.21.15.90:0/1184052452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e00102b10 0x7f3e0019b4c0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc00d740 tx=0x7f3dfc00dc00 comp rx=0 tx=0).stop 2024-04-25T11:43:11.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.431+0000 7f3e087ee640 1 -- 172.21.15.90:0/1184052452 shutdown_connections 2024-04-25T11:43:11.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.431+0000 7f3e087ee640 1 --2- 172.21.15.90:0/1184052452 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3dec0afc60 0x7f3dec0b2170 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.431+0000 7f3e087ee640 1 --2- 172.21.15.90:0/1184052452 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3e0010d3a0 0x7f3e0019bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.431+0000 7f3e087ee640 1 --2- 172.21.15.90:0/1184052452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e00108a40 0x7f3e0019ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.431+0000 7f3e087ee640 1 --2- 172.21.15.90:0/1184052452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e00102b10 0x7f3e0019b4c0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.431+0000 7f3e087ee640 1 -- 172.21.15.90:0/1184052452 >> 172.21.15.90:0/1184052452 conn(0x7f3e00078180 msgr2=0x7f3e001040d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:11.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.431+0000 7f3e087ee640 1 -- 172.21.15.90:0/1184052452 shutdown_connections 2024-04-25T11:43:11.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:11.431+0000 7f3e087ee640 1 -- 172.21.15.90:0/1184052452 wait complete. 2024-04-25T11:43:12.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:12 smithi138 ceph-mon[32328]: pgmap v249: 97 pgs: 24 active+clean+snaptrim, 33 active+clean, 40 active+clean+snaptrim_wait; 57 GiB data, 177 GiB used, 896 GiB / 1.0 TiB avail; 20 KiB/s rd, 1.6 MiB/s wr, 39 op/s 2024-04-25T11:43:12.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:12 smithi156 ceph-mon[33045]: pgmap v249: 97 pgs: 24 active+clean+snaptrim, 33 active+clean, 40 active+clean+snaptrim_wait; 57 GiB data, 177 GiB used, 896 GiB / 1.0 TiB avail; 20 KiB/s rd, 1.6 MiB/s wr, 39 op/s 2024-04-25T11:43:12.429 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T11:43:12.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:43:12.429 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T11:43:12.449 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:12 smithi090 ceph-mon[29575]: pgmap v249: 97 pgs: 24 active+clean+snaptrim, 33 active+clean, 40 active+clean+snaptrim_wait; 57 GiB data, 177 GiB used, 896 GiB / 1.0 TiB avail; 20 KiB/s rd, 1.6 MiB/s wr, 39 op/s 2024-04-25T11:43:12.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.561+0000 7fe753546640 1 -- 172.21.15.90:0/4066465799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe74c109ba0 msgr2=0x7fe74c101b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:12.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.561+0000 7fe753546640 1 --2- 172.21.15.90:0/4066465799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe74c109ba0 0x7fe74c101b10 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fe748009ca0 tx=0x7fe74802a8c0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.561+0000 7fe753546640 1 -- 172.21.15.90:0/4066465799 shutdown_connections 2024-04-25T11:43:12.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.561+0000 7fe753546640 1 --2- 172.21.15.90:0/4066465799 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe74c10e550 0x7fe74c110970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.561+0000 7fe753546640 1 --2- 172.21.15.90:0/4066465799 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe74c102050 0x7fe74c102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.561+0000 7fe753546640 1 --2- 172.21.15.90:0/4066465799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe74c109ba0 0x7fe74c101b10 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.561+0000 7fe753546640 1 -- 172.21.15.90:0/4066465799 >> 172.21.15.90:0/4066465799 conn(0x7fe74c077880 msgr2=0x7fe74c077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:12.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.561+0000 7fe753546640 1 -- 172.21.15.90:0/4066465799 shutdown_connections 2024-04-25T11:43:12.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.561+0000 7fe753546640 1 -- 172.21.15.90:0/4066465799 wait complete. 2024-04-25T11:43:12.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.562+0000 7fe753546640 1 Processor -- start 2024-04-25T11:43:12.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.562+0000 7fe753546640 1 -- start start 2024-04-25T11:43:12.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.562+0000 7fe753546640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe74c102050 0x7fe74c19a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.562+0000 7fe753546640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe74c109ba0 0x7fe74c19acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.562+0000 7fe753546640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe74c10e550 0x7fe74c19ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.562+0000 7fe753546640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c113d10 con 0x7fe74c102050 2024-04-25T11:43:12.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.562+0000 7fe753546640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c113b90 con 0x7fe74c109ba0 2024-04-25T11:43:12.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.562+0000 7fe753546640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c113e90 con 0x7fe74c10e550 2024-04-25T11:43:12.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.562+0000 7fe7512e2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe74c102050 0x7fe74c19a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.562+0000 7fe7512e2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe74c102050 0x7fe74c19a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52764/0 (socket says 172.21.15.90:52764) 2024-04-25T11:43:12.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.562+0000 7fe7512e2640 1 -- 172.21.15.90:0/2488489173 learned_addr learned my addr 172.21.15.90:0/2488489173 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:12.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.562+0000 7fe750ae1640 1 --2- 172.21.15.90:0/2488489173 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe74c109ba0 0x7fe74c19acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.563+0000 7fe751ae3640 1 --2- 172.21.15.90:0/2488489173 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe74c10e550 0x7fe74c19ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.563+0000 7fe7512e2640 1 -- 172.21.15.90:0/2488489173 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe74c10e550 msgr2=0x7fe74c19ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:12.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.563+0000 7fe7512e2640 1 --2- 172.21.15.90:0/2488489173 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe74c10e550 0x7fe74c19ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.563+0000 7fe7512e2640 1 -- 172.21.15.90:0/2488489173 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe74c109ba0 msgr2=0x7fe74c19acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:12.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.563+0000 7fe7512e2640 1 --2- 172.21.15.90:0/2488489173 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe74c109ba0 0x7fe74c19acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.563+0000 7fe7512e2640 1 -- 172.21.15.90:0/2488489173 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe74c19f5b0 con 0x7fe74c102050 2024-04-25T11:43:12.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.563+0000 7fe751ae3640 1 --2- 172.21.15.90:0/2488489173 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe74c10e550 0x7fe74c19ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:12.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.563+0000 7fe7512e2640 1 --2- 172.21.15.90:0/2488489173 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe74c102050 0x7fe74c19a750 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fe74802ade0 tx=0x7fe748031660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:12.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.563+0000 7fe7427fc640 1 -- 172.21.15.90:0/2488489173 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe748003b70 con 0x7fe74c102050 2024-04-25T11:43:12.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.563+0000 7fe753546640 1 -- 172.21.15.90:0/2488489173 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe74c19f840 con 0x7fe74c102050 2024-04-25T11:43:12.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.563+0000 7fe7427fc640 1 -- 172.21.15.90:0/2488489173 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe748031910 con 0x7fe74c102050 2024-04-25T11:43:12.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.563+0000 7fe7427fc640 1 -- 172.21.15.90:0/2488489173 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe74802f9e0 con 0x7fe74c102050 2024-04-25T11:43:12.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.563+0000 7fe753546640 1 -- 172.21.15.90:0/2488489173 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe74c1a7220 con 0x7fe74c102050 2024-04-25T11:43:12.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.566+0000 7fe75a3fc740 1 Processor -- start 2024-04-25T11:43:12.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.566+0000 7fe7427fc640 1 -- 172.21.15.90:0/2488489173 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fe74802fb80 con 0x7fe74c102050 2024-04-25T11:43:12.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.566+0000 7fe75a3fc740 1 -- start start 2024-04-25T11:43:12.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.566+0000 7fe75a3fc740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55625b44e410 0x55625b7ee620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.566+0000 7fe75a3fc740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55625b425550 0x55625b7eeb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.566+0000 7fe75a3fc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55625b42d730 0x55625b7ef0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.566+0000 7fe75a3fc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55625b754b00 con 0x55625b42d730 2024-04-25T11:43:12.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.566+0000 7fe75a3fc740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55625b6f0920 con 0x55625b44e410 2024-04-25T11:43:12.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.566+0000 7fe75a3fc740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55625b750ac0 con 0x55625b425550 2024-04-25T11:43:12.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.566+0000 7fe750ae1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55625b42d730 0x55625b7ef0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.566+0000 7fe751ae3640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55625b44e410 0x55625b7ee620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.566+0000 7fe750ae1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55625b42d730 0x55625b7ef0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52776/0 (socket says 172.21.15.90:52776) 2024-04-25T11:43:12.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.566+0000 7fe751ae3640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55625b44e410 0x55625b7ee620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:55116/0 (socket says 172.21.15.90:55116) 2024-04-25T11:43:12.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.566+0000 7fe750ae1640 1 -- 172.21.15.90:0/1709157597 learned_addr learned my addr 172.21.15.90:0/1709157597 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:12.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.566+0000 7fe7512e2640 1 --2- 172.21.15.90:0/1709157597 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55625b425550 0x55625b7eeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.566+0000 7fe7427fc640 1 --2- 172.21.15.90:0/2488489173 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe7380afc60 0x7fe7380b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.566+0000 7fe750ae1640 1 -- 172.21.15.90:0/1709157597 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55625b425550 msgr2=0x55625b7eeb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:12.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.566+0000 7fe750ae1640 1 --2- 172.21.15.90:0/1709157597 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55625b425550 0x55625b7eeb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.566+0000 7fe750ae1640 1 -- 172.21.15.90:0/1709157597 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55625b44e410 msgr2=0x55625b7ee620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:12.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.566+0000 7fe750ae1640 1 --2- 172.21.15.90:0/1709157597 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55625b44e410 0x55625b7ee620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.567+0000 7fe750ae1640 1 -- 172.21.15.90:0/1709157597 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55625b52e7e0 con 0x55625b42d730 2024-04-25T11:43:12.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.567+0000 7fe751ae3640 1 --2- 172.21.15.90:0/2488489173 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe7380afc60 0x7fe7380b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.567+0000 7fe7427fc640 1 -- 172.21.15.90:0/2488489173 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7fe7481002f0 con 0x7fe74c102050 2024-04-25T11:43:12.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.567+0000 7fe750ae1640 1 --2- 172.21.15.90:0/1709157597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55625b42d730 0x55625b7ef0a0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fe73c009c60 tx=0x7fe73c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6598f9d8a63e443 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:12.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.567+0000 7fe7377fe640 1 -- 172.21.15.90:0/1709157597 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73c004230 con 0x55625b42d730 2024-04-25T11:43:12.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.567+0000 7fe7377fe640 1 -- 172.21.15.90:0/1709157597 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe73c0043d0 con 0x55625b42d730 2024-04-25T11:43:12.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.567+0000 7fe7377fe640 1 -- 172.21.15.90:0/1709157597 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73c004cd0 con 0x55625b42d730 2024-04-25T11:43:12.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.568+0000 7fe75a3fc740 1 -- 172.21.15.90:0/1709157597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55625b42d730 msgr2=0x55625b7ef0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:12.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.568+0000 7fe751ae3640 1 --2- 172.21.15.90:0/2488489173 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe7380afc60 0x7fe7380b2170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x55625b7eb790 tx=0x7fe7440046c0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:12.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.568+0000 7fe75a3fc740 1 --2- 172.21.15.90:0/1709157597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55625b42d730 0x55625b7ef0a0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fe73c009c60 tx=0x7fe73c011810 comp rx=0 tx=0).stop 2024-04-25T11:43:12.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.568+0000 7fe75a3fc740 1 -- 172.21.15.90:0/1709157597 shutdown_connections 2024-04-25T11:43:12.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.568+0000 7fe75a3fc740 1 --2- 172.21.15.90:0/1709157597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55625b42d730 0x55625b7ef0a0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.568+0000 7fe75a3fc740 1 --2- 172.21.15.90:0/1709157597 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55625b425550 0x55625b7eeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.568+0000 7fe75a3fc740 1 --2- 172.21.15.90:0/1709157597 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55625b44e410 0x55625b7ee620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.568+0000 7fe75a3fc740 1 -- 172.21.15.90:0/1709157597 >> 172.21.15.90:0/1709157597 conn(0x55625b442560 msgr2=0x55625b7eadd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:12.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.568+0000 7fe75a3fc740 1 -- 172.21.15.90:0/1709157597 shutdown_connections 2024-04-25T11:43:12.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.568+0000 7fe75a3fc740 1 -- 172.21.15.90:0/1709157597 wait complete. 2024-04-25T11:43:12.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.569+0000 7fe75a3fc740 1 -- start start 2024-04-25T11:43:12.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.569+0000 7fe75a3fc740 1 Processor -- start 2024-04-25T11:43:12.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.569+0000 7fe75a3fc740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55625b425550 0x55625b7f8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.569+0000 7fe75a3fc740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55625b42d730 0x55625b7f90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.569+0000 7fe75a3fc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55625b44e410 0x55625b804e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.569+0000 7fe75a3fc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55625b7fb970 con 0x55625b44e410 2024-04-25T11:43:12.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.569+0000 7fe75a3fc740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55625b6ee5d0 con 0x55625b42d730 2024-04-25T11:43:12.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.569+0000 7fe75a3fc740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55625b7fbaf0 con 0x55625b425550 2024-04-25T11:43:12.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.569+0000 7fe75a3fc740 10 client.0 _ll_register_callbacks cb 0x55625b6dc080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:12.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.569+0000 7fe751ae3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55625b44e410 0x55625b804e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.569+0000 7fe750ae1640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55625b42d730 0x55625b7f90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.569+0000 7fe7512e2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55625b425550 0x55625b7f8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.569+0000 7fe750ae1640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55625b42d730 0x55625b7f90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:55130/0 (socket says 172.21.15.90:55130) 2024-04-25T11:43:12.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.569+0000 7fe751ae3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55625b44e410 0x55625b804e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52790/0 (socket says 172.21.15.90:52790) 2024-04-25T11:43:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.570+0000 7fe750ae1640 1 -- 172.21.15.90:0/2544337014 learned_addr learned my addr 172.21.15.90:0/2544337014 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.570+0000 7fe750ae1640 1 -- 172.21.15.90:0/2544337014 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55625b425550 msgr2=0x55625b7f8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.570+0000 7fe750ae1640 1 --2- 172.21.15.90:0/2544337014 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55625b425550 0x55625b7f8b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.570+0000 7fe750ae1640 1 -- 172.21.15.90:0/2544337014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55625b44e410 msgr2=0x55625b804e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.570+0000 7fe750ae1640 1 --2- 172.21.15.90:0/2544337014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55625b44e410 0x55625b804e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.570+0000 7fe750ae1640 1 -- 172.21.15.90:0/2544337014 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55625b58ef70 con 0x55625b42d730 2024-04-25T11:43:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.570+0000 7fe751ae3640 1 --2- 172.21.15.90:0/2544337014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55625b44e410 0x55625b804e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:43:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.570+0000 7fe7512e2640 1 --2- 172.21.15.90:0/2544337014 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55625b425550 0x55625b7f8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:43:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.570+0000 7fe750ae1640 1 --2- 172.21.15.90:0/2544337014 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55625b42d730 0x55625b7f90a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe73c0049b0 tx=0x7fe73c019ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb2b26af4f481c79 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.570+0000 7fe734ff9640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:43:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.570+0000 7fe734ff9640 1 -- 172.21.15.90:0/2544337014 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73c028020 con 0x55625b42d730 2024-04-25T11:43:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.570+0000 7fe75a3fc740 1 -- 172.21.15.90:0/2544337014 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe73c0043d0 con 0x55625b42d730 2024-04-25T11:43:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.570+0000 7fe734ff9640 1 -- 172.21.15.90:0/2544337014 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7fe73c024620 con 0x55625b42d730 2024-04-25T11:43:12.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.570+0000 7fe734ff9640 1 -- 172.21.15.90:0/2544337014 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe73c033050 con 0x55625b42d730 2024-04-25T11:43:12.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.571+0000 7fe75a3fc740 10 client.26984 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:12.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.571+0000 7fe75a3fc740 1 -- 172.21.15.90:0/2544337014 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55625b426170 con 0x55625b42d730 2024-04-25T11:43:12.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.571+0000 7fe734ff9640 1 -- 172.21.15.90:0/2544337014 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe73c0043d0 con 0x55625b42d730 2024-04-25T11:43:12.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.571+0000 7fe734ff9640 1 -- 172.21.15.90:0/2544337014 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fe73c018070 con 0x55625b42d730 2024-04-25T11:43:12.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.571+0000 7fe75a3fc740 10 client.26984 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:12.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.571+0000 7fe75a3fc740 10 client.26984 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:43:12.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.571+0000 7fe75a3fc740 20 client.26984 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:12.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.571+0000 7fe75a3fc740 1 --2- 172.21.15.90:0/2544337014 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55625b40d1f0 0x55625b80ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.571+0000 7fe75a3fc740 4 client.26984 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:43:12.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.571+0000 7fe75a3fc740 1 -- 172.21.15.90:0/2544337014 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55625b762710 con 0x55625b40d1f0 2024-04-25T11:43:12.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.571+0000 7fe7512e2640 1 --2- 172.21.15.90:0/2544337014 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55625b40d1f0 0x55625b80ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.572+0000 7fe734ff9640 10 client.26984 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.572+0000 7fe7512e2640 1 --2- 172.21.15.90:0/2544337014 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55625b40d1f0 0x55625b80ad80 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=767637093c578701 server_cookie=32c3b479d1ed50a3 in_seq=0 out_seq=0 2024-04-25T11:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.572+0000 7fe734ff9640 1 -- 172.21.15.90:0/2544337014 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55625b762710 con 0x55625b40d1f0 2024-04-25T11:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.572+0000 7fe734ff9640 10 client.26984 handle_command_reply: tid=0 2024-04-25T11:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.572+0000 7fe75a3fc740 1 client.26984 shutdown 2024-04-25T11:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.572+0000 7fe75a3fc740 1 -- 172.21.15.90:0/2544337014 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55625b42d730 msgr2=0x55625b7f90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.572+0000 7fe75a3fc740 1 --2- 172.21.15.90:0/2544337014 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55625b42d730 0x55625b7f90a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe73c0049b0 tx=0x7fe73c019ad0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.573+0000 7fe75a3fc740 1 -- 172.21.15.90:0/2544337014 shutdown_connections 2024-04-25T11:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.573+0000 7fe75a3fc740 1 --2- 172.21.15.90:0/2544337014 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55625b40d1f0 0x55625b80ad80 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.573+0000 7fe75a3fc740 1 --2- 172.21.15.90:0/2544337014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55625b44e410 0x55625b804e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.573+0000 7fe75a3fc740 1 --2- 172.21.15.90:0/2544337014 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55625b42d730 0x55625b7f90a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.573+0000 7fe734ff9640 0 client.26984 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.573+0000 7fe75a3fc740 1 --2- 172.21.15.90:0/2544337014 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55625b425550 0x55625b7f8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.573+0000 7fe75a3fc740 1 -- 172.21.15.90:0/2544337014 >> 172.21.15.90:0/2544337014 conn(0x55625b442560 msgr2=0x55625b7f0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:12.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.573+0000 7fe75a3fc740 1 -- 172.21.15.90:0/2544337014 shutdown_connections 2024-04-25T11:43:12.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.573+0000 7fe75a3fc740 1 -- 172.21.15.90:0/2544337014 wait complete. 2024-04-25T11:43:12.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.573+0000 7fe75a3fc740 20 client.26984 trim_cache size 0 max 16384 2024-04-25T11:43:12.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.593+0000 7fe75a3fc740 1 Processor -- start 2024-04-25T11:43:12.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.593+0000 7fe75a3fc740 1 -- start start 2024-04-25T11:43:12.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.593+0000 7fe75a3fc740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55625b425550 0x55625b7f8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.593+0000 7fe75a3fc740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55625b42d730 0x55625b7eebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.593+0000 7fe75a3fc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55625b44e410 0x55625b810410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.593+0000 7fe75a3fc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c1afd10 con 0x55625b44e410 2024-04-25T11:43:12.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.593+0000 7fe75a3fc740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c1b1850 con 0x55625b42d730 2024-04-25T11:43:12.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.593+0000 7fe75a3fc740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c1b03e0 con 0x55625b425550 2024-04-25T11:43:12.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.593+0000 7fe751ae3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55625b44e410 0x55625b810410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.593+0000 7fe750ae1640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55625b42d730 0x55625b7eebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.593+0000 7fe751ae3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55625b44e410 0x55625b810410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52800/0 (socket says 172.21.15.90:52800) 2024-04-25T11:43:12.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.593+0000 7fe751ae3640 1 -- 172.21.15.90:0/1460644398 learned_addr learned my addr 172.21.15.90:0/1460644398 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:12.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.593+0000 7fe7512e2640 1 --2- 172.21.15.90:0/1460644398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55625b425550 0x55625b7f8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.593+0000 7fe750ae1640 1 -- 172.21.15.90:0/1460644398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55625b425550 msgr2=0x55625b7f8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:12.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.593+0000 7fe750ae1640 1 --2- 172.21.15.90:0/1460644398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55625b425550 0x55625b7f8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.593+0000 7fe750ae1640 1 -- 172.21.15.90:0/1460644398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55625b44e410 msgr2=0x55625b810410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:12.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.593+0000 7fe750ae1640 1 --2- 172.21.15.90:0/1460644398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55625b44e410 0x55625b810410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.593+0000 7fe750ae1640 1 -- 172.21.15.90:0/1460644398 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55625b80b1e0 con 0x55625b42d730 2024-04-25T11:43:12.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.594+0000 7fe750ae1640 1 --2- 172.21.15.90:0/1460644398 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55625b42d730 0x55625b7eebd0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe73c009b80 tx=0x7fe73c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba1d43a3c6154581 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:12.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.594+0000 7fe736ffd640 1 -- 172.21.15.90:0/1460644398 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73c028040 con 0x55625b42d730 2024-04-25T11:43:12.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.594+0000 7fe736ffd640 1 -- 172.21.15.90:0/1460644398 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe73c01f040 con 0x55625b42d730 2024-04-25T11:43:12.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.594+0000 7fe736ffd640 1 -- 172.21.15.90:0/1460644398 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73c0042d0 con 0x55625b42d730 2024-04-25T11:43:12.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.595+0000 7fe75a3fc740 1 -- 172.21.15.90:0/1460644398 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55625b42d730 msgr2=0x55625b7eebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:12.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.595+0000 7fe75a3fc740 1 --2- 172.21.15.90:0/1460644398 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55625b42d730 0x55625b7eebd0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe73c009b80 tx=0x7fe73c011810 comp rx=0 tx=0).stop 2024-04-25T11:43:12.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.595+0000 7fe75a3fc740 1 -- 172.21.15.90:0/1460644398 shutdown_connections 2024-04-25T11:43:12.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.595+0000 7fe75a3fc740 1 --2- 172.21.15.90:0/1460644398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55625b44e410 0x55625b810410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.595+0000 7fe75a3fc740 1 --2- 172.21.15.90:0/1460644398 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55625b42d730 0x55625b7eebd0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.595+0000 7fe75a3fc740 1 --2- 172.21.15.90:0/1460644398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55625b425550 0x55625b7f8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.595+0000 7fe75a3fc740 1 -- 172.21.15.90:0/1460644398 >> 172.21.15.90:0/1460644398 conn(0x55625b442560 msgr2=0x55625b7ee4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:12.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.595+0000 7fe75a3fc740 1 -- 172.21.15.90:0/1460644398 shutdown_connections 2024-04-25T11:43:12.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.595+0000 7fe75a3fc740 1 -- 172.21.15.90:0/1460644398 wait complete. 2024-04-25T11:43:12.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.596+0000 7fe75a3fc740 1 -- start start 2024-04-25T11:43:12.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.596+0000 7fe75a3fc740 1 Processor -- start 2024-04-25T11:43:12.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.596+0000 7fe75a3fc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55625b425550 0x55625b7fbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.596+0000 7fe75a3fc740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55625b42d730 0x55625b7fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.596+0000 7fe75a3fc740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55625b44e410 0x55625b870fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.596+0000 7fe75a3fc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c1b1180 con 0x55625b425550 2024-04-25T11:43:12.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.596+0000 7fe75a3fc740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c1b0ab0 con 0x55625b44e410 2024-04-25T11:43:12.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.596+0000 7fe75a3fc740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c1add10 con 0x55625b42d730 2024-04-25T11:43:12.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.596+0000 7fe75a3fc740 10 client.0 _ll_register_callbacks cb 0x7fe73c009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:12.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.596+0000 7fe7512e2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55625b425550 0x55625b7fbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.596+0000 7fe750ae1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55625b42d730 0x55625b7fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.596+0000 7fe7512e2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55625b425550 0x55625b7fbf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52810/0 (socket says 172.21.15.90:52810) 2024-04-25T11:43:12.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.596+0000 7fe7512e2640 1 -- 172.21.15.90:0/301272970 learned_addr learned my addr 172.21.15.90:0/301272970 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:12.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.597+0000 7fe751ae3640 1 --2- 172.21.15.90:0/301272970 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55625b44e410 0x55625b870fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.597+0000 7fe750ae1640 1 -- 172.21.15.90:0/301272970 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55625b44e410 msgr2=0x55625b870fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:12.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.597+0000 7fe750ae1640 1 --2- 172.21.15.90:0/301272970 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55625b44e410 0x55625b870fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.597+0000 7fe750ae1640 1 -- 172.21.15.90:0/301272970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55625b425550 msgr2=0x55625b7fbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:12.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.597+0000 7fe750ae1640 1 --2- 172.21.15.90:0/301272970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55625b425550 0x55625b7fbf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.597+0000 7fe750ae1640 1 -- 172.21.15.90:0/301272970 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55625b8714f0 con 0x55625b42d730 2024-04-25T11:43:12.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.597+0000 7fe750ae1640 1 --2- 172.21.15.90:0/301272970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55625b42d730 0x55625b7fc4d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe73c013040 tx=0x7fe73c0189e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=640ab90567c8b94e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:12.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.597+0000 7fe734ff9640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:43:12.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.597+0000 7fe734ff9640 1 -- 172.21.15.90:0/301272970 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73c028030 con 0x55625b42d730 2024-04-25T11:43:12.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.597+0000 7fe734ff9640 1 -- 172.21.15.90:0/301272970 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7fe73c0097a0 con 0x55625b42d730 2024-04-25T11:43:12.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.597+0000 7fe75a3fc740 1 -- 172.21.15.90:0/301272970 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe73c01f040 con 0x55625b42d730 2024-04-25T11:43:12.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.597+0000 7fe734ff9640 1 -- 172.21.15.90:0/301272970 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe73c04d050 con 0x55625b42d730 2024-04-25T11:43:12.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.598+0000 7fe75a3fc740 10 client.26887 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:12.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.598+0000 7fe75a3fc740 1 -- 172.21.15.90:0/301272970 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55625b872650 con 0x55625b42d730 2024-04-25T11:43:12.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.598+0000 7fe734ff9640 1 -- 172.21.15.90:0/301272970 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe73c01f040 con 0x55625b42d730 2024-04-25T11:43:12.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.598+0000 7fe734ff9640 1 -- 172.21.15.90:0/301272970 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fe73c048030 con 0x55625b42d730 2024-04-25T11:43:12.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.598+0000 7fe75a3fc740 10 client.26887 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:12.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.598+0000 7fe75a3fc740 10 client.26887 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:43:12.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.598+0000 7fe75a3fc740 20 client.26887 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:12.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.598+0000 7fe75a3fc740 1 --2- 172.21.15.90:0/301272970 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55625b3e2630 0x55625b874d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.598+0000 7fe75a3fc740 4 client.26887 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T11:43:12.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.598+0000 7fe75a3fc740 1 -- 172.21.15.90:0/301272970 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55625b7f9430 con 0x55625b3e2630 2024-04-25T11:43:12.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.598+0000 7fe7512e2640 1 --2- 172.21.15.90:0/301272970 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55625b3e2630 0x55625b874d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.599+0000 7fe734ff9640 10 client.26887 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:12.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.599+0000 7fe7512e2640 1 --2- 172.21.15.90:0/301272970 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55625b3e2630 0x55625b874d40 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de81949ca087d4f server_cookie=7ffc430615397ee8 in_seq=0 out_seq=0 2024-04-25T11:43:12.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.600+0000 7fe734ff9640 1 -- 172.21.15.90:0/301272970 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55625b7f9430 con 0x55625b3e2630 2024-04-25T11:43:12.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.600+0000 7fe734ff9640 10 client.26887 handle_command_reply: tid=0 2024-04-25T11:43:12.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.600+0000 7fe75a3fc740 1 client.26887 shutdown 2024-04-25T11:43:12.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.600+0000 7fe75a3fc740 1 -- 172.21.15.90:0/301272970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55625b42d730 msgr2=0x55625b7fc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:12.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.600+0000 7fe75a3fc740 1 --2- 172.21.15.90:0/301272970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55625b42d730 0x55625b7fc4d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe73c013040 tx=0x7fe73c0189e0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.600+0000 7fe75a3fc740 1 -- 172.21.15.90:0/301272970 shutdown_connections 2024-04-25T11:43:12.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.600+0000 7fe75a3fc740 1 --2- 172.21.15.90:0/301272970 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55625b3e2630 0x55625b874d40 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.600+0000 7fe75a3fc740 1 --2- 172.21.15.90:0/301272970 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55625b44e410 0x55625b870fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.600+0000 7fe75a3fc740 1 --2- 172.21.15.90:0/301272970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55625b42d730 0x55625b7fc4d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.600+0000 7fe75a3fc740 1 --2- 172.21.15.90:0/301272970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55625b425550 0x55625b7fbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.600+0000 7fe734ff9640 0 client.26887 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:12.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.600+0000 7fe75a3fc740 1 -- 172.21.15.90:0/301272970 >> 172.21.15.90:0/301272970 conn(0x55625b442560 msgr2=0x55625b7e9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:12.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.600+0000 7fe75a3fc740 1 -- 172.21.15.90:0/301272970 shutdown_connections 2024-04-25T11:43:12.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.600+0000 7fe75a3fc740 1 -- 172.21.15.90:0/301272970 wait complete. 2024-04-25T11:43:12.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.600+0000 7fe75a3fc740 20 client.26887 trim_cache size 0 max 16384 2024-04-25T11:43:12.591 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:43:12.591 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-25T11:43:12.591 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "feb0ab91-4578-46d3-84b3-82b5dede08c1", 2024-04-25T11:43:12.591 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-25T11:43:12.591 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:43:12.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.601+0000 7fe753546640 1 -- 172.21.15.90:0/2488489173 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe7380afc60 msgr2=0x7fe7380b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:12.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.601+0000 7fe753546640 1 --2- 172.21.15.90:0/2488489173 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe7380afc60 0x7fe7380b2170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x55625b7eb790 tx=0x7fe7440046c0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.601+0000 7fe753546640 1 -- 172.21.15.90:0/2488489173 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe74c102050 msgr2=0x7fe74c19a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:12.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.601+0000 7fe753546640 1 --2- 172.21.15.90:0/2488489173 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe74c102050 0x7fe74c19a750 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fe74802ade0 tx=0x7fe748031660 comp rx=0 tx=0).stop 2024-04-25T11:43:12.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.601+0000 7fe753546640 1 -- 172.21.15.90:0/2488489173 shutdown_connections 2024-04-25T11:43:12.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.601+0000 7fe753546640 1 --2- 172.21.15.90:0/2488489173 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe7380afc60 0x7fe7380b2170 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.601+0000 7fe753546640 1 --2- 172.21.15.90:0/2488489173 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe74c10e550 0x7fe74c19ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.601+0000 7fe753546640 1 --2- 172.21.15.90:0/2488489173 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe74c109ba0 0x7fe74c19acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.601+0000 7fe753546640 1 --2- 172.21.15.90:0/2488489173 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe74c102050 0x7fe74c19a750 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.601+0000 7fe753546640 1 -- 172.21.15.90:0/2488489173 >> 172.21.15.90:0/2488489173 conn(0x7fe74c077880 msgr2=0x7fe74c10a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:12.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.601+0000 7fe753546640 1 -- 172.21.15.90:0/2488489173 shutdown_connections 2024-04-25T11:43:12.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.601+0000 7fe753546640 1 -- 172.21.15.90:0/2488489173 wait complete. 2024-04-25T11:43:12.599 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:43:12.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:43:12.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.772+0000 7f178fab1640 1 -- 172.21.15.90:0/3698568578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1788102b10 msgr2=0x7f1788102f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:12.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.772+0000 7f178fab1640 1 --2- 172.21.15.90:0/3698568578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1788102b10 0x7f1788102f20 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f1778009900 tx=0x7f177802a410 comp rx=0 tx=0).stop 2024-04-25T11:43:12.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.772+0000 7f178fab1640 1 -- 172.21.15.90:0/3698568578 shutdown_connections 2024-04-25T11:43:12.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.772+0000 7f178fab1640 1 --2- 172.21.15.90:0/3698568578 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f178810d3a0 0x7f178810f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.772+0000 7f178fab1640 1 --2- 172.21.15.90:0/3698568578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1788102b10 0x7f1788102f20 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.772+0000 7f178fab1640 1 --2- 172.21.15.90:0/3698568578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1788108a40 0x7f17881025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.772+0000 7f178fab1640 1 -- 172.21.15.90:0/3698568578 >> 172.21.15.90:0/3698568578 conn(0x7f1788078180 msgr2=0x7f17880785c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:12.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.773+0000 7f178fab1640 1 -- 172.21.15.90:0/3698568578 shutdown_connections 2024-04-25T11:43:12.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.773+0000 7f178fab1640 1 -- 172.21.15.90:0/3698568578 wait complete. 2024-04-25T11:43:12.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.774+0000 7f178fab1640 1 Processor -- start 2024-04-25T11:43:12.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.774+0000 7f178fab1640 1 -- start start 2024-04-25T11:43:12.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.774+0000 7f178fab1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1788102b10 0x7f178819a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.774+0000 7f178fab1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1788108a40 0x7f178819ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.774+0000 7f178fab1640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f178810d3a0 0x7f178819ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.774+0000 7f178fab1640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1788113d10 con 0x7f1788108a40 2024-04-25T11:43:12.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.774+0000 7f178fab1640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1788113b90 con 0x7f178810d3a0 2024-04-25T11:43:12.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.774+0000 7f178fab1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1788113e90 con 0x7f1788102b10 2024-04-25T11:43:12.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.774+0000 7f178d04c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1788108a40 0x7f178819ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.774+0000 7f178d84d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1788102b10 0x7f178819a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.774+0000 7f178d04c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1788108a40 0x7f178819ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52822/0 (socket says 172.21.15.90:52822) 2024-04-25T11:43:12.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.774+0000 7f178d04c640 1 -- 172.21.15.90:0/2137333156 learned_addr learned my addr 172.21.15.90:0/2137333156 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:12.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.774+0000 7f178e04e640 1 --2- 172.21.15.90:0/2137333156 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f178810d3a0 0x7f178819ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.775+0000 7f178e04e640 1 -- 172.21.15.90:0/2137333156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1788102b10 msgr2=0x7f178819a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:12.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.775+0000 7f178e04e640 1 --2- 172.21.15.90:0/2137333156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1788102b10 0x7f178819a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.775+0000 7f178e04e640 1 -- 172.21.15.90:0/2137333156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1788108a40 msgr2=0x7f178819ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:12.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.775+0000 7f178e04e640 1 --2- 172.21.15.90:0/2137333156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1788108a40 0x7f178819ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.775+0000 7f178e04e640 1 -- 172.21.15.90:0/2137333156 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f178819f550 con 0x7f178810d3a0 2024-04-25T11:43:12.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.775+0000 7f178e04e640 1 --2- 172.21.15.90:0/2137333156 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f178810d3a0 0x7f178819ef10 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1780009fd0 tx=0x7f178000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:12.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.775+0000 7f177effd640 1 -- 172.21.15.90:0/2137333156 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f178000ce20 con 0x7f178810d3a0 2024-04-25T11:43:12.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.775+0000 7f177effd640 1 -- 172.21.15.90:0/2137333156 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1780005020 con 0x7f178810d3a0 2024-04-25T11:43:12.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.775+0000 7f177effd640 1 -- 172.21.15.90:0/2137333156 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1780010690 con 0x7f178810d3a0 2024-04-25T11:43:12.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.775+0000 7f178fab1640 1 -- 172.21.15.90:0/2137333156 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f178819f840 con 0x7f178810d3a0 2024-04-25T11:43:12.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.775+0000 7f178fab1640 1 -- 172.21.15.90:0/2137333156 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17881a71c0 con 0x7f178810d3a0 2024-04-25T11:43:12.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.778+0000 7f177effd640 1 -- 172.21.15.90:0/2137333156 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f17800054d0 con 0x7f178810d3a0 2024-04-25T11:43:12.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.778+0000 7f17965fd740 1 Processor -- start 2024-04-25T11:43:12.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.778+0000 7f17965fd740 1 -- start start 2024-04-25T11:43:12.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.778+0000 7f17965fd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d37e04e390 0x55d37e3eea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.778+0000 7f17965fd740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d37e0254d0 0x55d37e3eefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.778+0000 7f17965fd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d37e02d6b0 0x55d37e3fb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.778+0000 7f17965fd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d37e354d80 con 0x55d37e02d6b0 2024-04-25T11:43:12.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.778+0000 7f17965fd740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55d37e30e8b0 con 0x55d37e0254d0 2024-04-25T11:43:12.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.778+0000 7f17965fd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d37e350d40 con 0x55d37e04e390 2024-04-25T11:43:12.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.778+0000 7f178d04c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d37e02d6b0 0x55d37e3fb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.779+0000 7f178d84d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d37e0254d0 0x55d37e3eefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.779+0000 7f178d84d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d37e0254d0 0x55d37e3eefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:55180/0 (socket says 172.21.15.90:55180) 2024-04-25T11:43:12.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.779+0000 7f178d84d640 1 -- 172.21.15.90:0/4285261665 learned_addr learned my addr 172.21.15.90:0/4285261665 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:12.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.779+0000 7f177effd640 1 --2- 172.21.15.90:0/2137333156 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f17740afce0 0x7f17740b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.779+0000 7f178d04c640 1 --2- 172.21.15.90:0/4285261665 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d37e04e390 0x55d37e3eea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.779+0000 7f178e04e640 1 --2- 172.21.15.90:0/2137333156 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f17740afce0 0x7f17740b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.779+0000 7f178d84d640 1 -- 172.21.15.90:0/4285261665 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d37e04e390 msgr2=0x55d37e3eea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:12.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.779+0000 7f178d84d640 1 --2- 172.21.15.90:0/4285261665 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d37e04e390 0x55d37e3eea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.779+0000 7f178d84d640 1 -- 172.21.15.90:0/4285261665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d37e02d6b0 msgr2=0x55d37e3fb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:12.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.779+0000 7f178d84d640 1 --2- 172.21.15.90:0/4285261665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d37e02d6b0 0x55d37e3fb8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.779+0000 7f178d84d640 1 -- 172.21.15.90:0/4285261665 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d37e2dfa40 con 0x55d37e0254d0 2024-04-25T11:43:12.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.779+0000 7f177effd640 1 -- 172.21.15.90:0/2137333156 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7f17800df710 con 0x7f178810d3a0 2024-04-25T11:43:12.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.779+0000 7f178d84d640 1 --2- 172.21.15.90:0/4285261665 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d37e0254d0 0x55d37e3eefa0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f178400bd10 tx=0x7f17840107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fa2bbcb6b25da9d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:12.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.779+0000 7f178e04e640 1 --2- 172.21.15.90:0/2137333156 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f17740afce0 0x7f17740b21f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f17800dce60 tx=0x7f1780075060 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:12.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.779+0000 7f1773fff640 1 -- 172.21.15.90:0/4285261665 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1784005220 con 0x55d37e0254d0 2024-04-25T11:43:12.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.779+0000 7f1773fff640 1 -- 172.21.15.90:0/4285261665 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1784004c50 con 0x55d37e0254d0 2024-04-25T11:43:12.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.779+0000 7f1773fff640 1 -- 172.21.15.90:0/4285261665 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1784011820 con 0x55d37e0254d0 2024-04-25T11:43:12.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.780+0000 7f17965fd740 1 -- 172.21.15.90:0/4285261665 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d37e0254d0 msgr2=0x55d37e3eefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:12.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.780+0000 7f17965fd740 1 --2- 172.21.15.90:0/4285261665 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d37e0254d0 0x55d37e3eefa0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f178400bd10 tx=0x7f17840107a0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.780+0000 7f17965fd740 1 -- 172.21.15.90:0/4285261665 shutdown_connections 2024-04-25T11:43:12.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.780+0000 7f17965fd740 1 --2- 172.21.15.90:0/4285261665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d37e02d6b0 0x55d37e3fb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.780+0000 7f17965fd740 1 --2- 172.21.15.90:0/4285261665 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d37e0254d0 0x55d37e3eefa0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.780+0000 7f17965fd740 1 --2- 172.21.15.90:0/4285261665 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d37e04e390 0x55d37e3eea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.780+0000 7f17965fd740 1 -- 172.21.15.90:0/4285261665 >> 172.21.15.90:0/4285261665 conn(0x55d37e0424e0 msgr2=0x55d37e3eb210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:12.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.780+0000 7f17965fd740 1 -- 172.21.15.90:0/4285261665 shutdown_connections 2024-04-25T11:43:12.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.780+0000 7f17965fd740 1 -- 172.21.15.90:0/4285261665 wait complete. 2024-04-25T11:43:12.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.781+0000 7f17965fd740 1 -- start start 2024-04-25T11:43:12.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.781+0000 7f17965fd740 1 Processor -- start 2024-04-25T11:43:12.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.781+0000 7f17965fd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d37e0254d0 0x55d37e3f95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.781+0000 7f17965fd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d37e02d6b0 0x55d37e3f9b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.781+0000 7f17965fd740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d37e04e390 0x55d37e3ff430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.781+0000 7f17965fd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d37e3fbfe0 con 0x55d37e0254d0 2024-04-25T11:43:12.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.781+0000 7f17965fd740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55d37e2dbd10 con 0x55d37e04e390 2024-04-25T11:43:12.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.781+0000 7f17965fd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d37e3fc160 con 0x55d37e02d6b0 2024-04-25T11:43:12.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.781+0000 7f17965fd740 10 client.0 _ll_register_callbacks cb 0x55d37e308210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:12.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.781+0000 7f178d04c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d37e0254d0 0x55d37e3f95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.781+0000 7f178d04c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d37e0254d0 0x55d37e3f95f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52840/0 (socket says 172.21.15.90:52840) 2024-04-25T11:43:12.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.781+0000 7f178d04c640 1 -- 172.21.15.90:0/3781055137 learned_addr learned my addr 172.21.15.90:0/3781055137 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:12.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.781+0000 7f178d84d640 1 --2- 172.21.15.90:0/3781055137 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d37e02d6b0 0x55d37e3f9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.781+0000 7f178d04c640 1 --2- 172.21.15.90:0/3781055137 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d37e04e390 0x55d37e3ff430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.782+0000 7f178d04c640 1 -- 172.21.15.90:0/3781055137 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d37e02d6b0 msgr2=0x55d37e3f9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:12.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.782+0000 7f178d04c640 1 --2- 172.21.15.90:0/3781055137 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d37e02d6b0 0x55d37e3f9b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.782+0000 7f178d04c640 1 -- 172.21.15.90:0/3781055137 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d37e04e390 msgr2=0x55d37e3ff430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:12.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.782+0000 7f178d04c640 1 --2- 172.21.15.90:0/3781055137 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d37e04e390 0x55d37e3ff430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.782+0000 7f178d04c640 1 -- 172.21.15.90:0/3781055137 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d37e1307d0 con 0x55d37e0254d0 2024-04-25T11:43:12.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.782+0000 7f178d04c640 1 --2- 172.21.15.90:0/3781055137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d37e0254d0 0x55d37e3f95f0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f1778006010 tx=0x7f177802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3387bed15c04ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:12.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.782+0000 7f17717fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:12.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.782+0000 7f17717fa640 1 -- 172.21.15.90:0/3781055137 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1778035040 con 0x55d37e0254d0 2024-04-25T11:43:12.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.782+0000 7f17717fa640 1 -- 172.21.15.90:0/3781055137 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7f1778004040 con 0x55d37e0254d0 2024-04-25T11:43:12.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.782+0000 7f17965fd740 1 -- 172.21.15.90:0/3781055137 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1784004c50 con 0x55d37e0254d0 2024-04-25T11:43:12.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.782+0000 7f17717fa640 1 -- 172.21.15.90:0/3781055137 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1778004240 con 0x55d37e0254d0 2024-04-25T11:43:12.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.782+0000 7f17965fd740 10 client.18162 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:12.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.782+0000 7f17965fd740 1 -- 172.21.15.90:0/3781055137 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d37e0663f0 con 0x55d37e0254d0 2024-04-25T11:43:12.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.782+0000 7f17717fa640 1 -- 172.21.15.90:0/3781055137 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1784004c50 con 0x55d37e0254d0 2024-04-25T11:43:12.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.783+0000 7f17717fa640 1 -- 172.21.15.90:0/3781055137 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f177803d450 con 0x55d37e0254d0 2024-04-25T11:43:12.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.783+0000 7f17965fd740 10 client.18162 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:12.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.783+0000 7f17965fd740 10 client.18162 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:43:12.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.783+0000 7f17965fd740 20 client.18162 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:12.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.783+0000 7f17965fd740 1 --2- 172.21.15.90:0/3781055137 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d37e00cf60 0x55d37e40b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.783+0000 7f17965fd740 4 client.18162 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:43:12.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.783+0000 7f17965fd740 1 -- 172.21.15.90:0/3781055137 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d37e362990 con 0x55d37e00cf60 2024-04-25T11:43:12.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.783+0000 7f178d84d640 1 --2- 172.21.15.90:0/3781055137 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d37e00cf60 0x55d37e40b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.783+0000 7f17717fa640 10 client.18162 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:12.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.783+0000 7f178d84d640 1 --2- 172.21.15.90:0/3781055137 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d37e00cf60 0x55d37e40b690 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8df254120951ada server_cookie=6602b86161249feb in_seq=0 out_seq=0 2024-04-25T11:43:12.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.784+0000 7f17717fa640 1 -- 172.21.15.90:0/3781055137 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d37e362990 con 0x55d37e00cf60 2024-04-25T11:43:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.784+0000 7f17717fa640 10 client.18162 handle_command_reply: tid=0 2024-04-25T11:43:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.784+0000 7f17965fd740 1 client.18162 shutdown 2024-04-25T11:43:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.784+0000 7f17965fd740 1 -- 172.21.15.90:0/3781055137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d37e0254d0 msgr2=0x55d37e3f95f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.784+0000 7f17965fd740 1 --2- 172.21.15.90:0/3781055137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d37e0254d0 0x55d37e3f95f0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f1778006010 tx=0x7f177802a410 comp rx=0 tx=0).stop 2024-04-25T11:43:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.784+0000 7f17965fd740 1 -- 172.21.15.90:0/3781055137 shutdown_connections 2024-04-25T11:43:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.784+0000 7f17965fd740 1 --2- 172.21.15.90:0/3781055137 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d37e00cf60 0x55d37e40b690 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.784+0000 7f17965fd740 1 --2- 172.21.15.90:0/3781055137 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d37e04e390 0x55d37e3ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.784+0000 7f17965fd740 1 --2- 172.21.15.90:0/3781055137 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d37e02d6b0 0x55d37e3f9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.785+0000 7f17965fd740 1 --2- 172.21.15.90:0/3781055137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d37e0254d0 0x55d37e3f95f0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.784+0000 7f17717fa640 0 client.18162 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.785+0000 7f17965fd740 1 -- 172.21.15.90:0/3781055137 >> 172.21.15.90:0/3781055137 conn(0x55d37e0424e0 msgr2=0x55d37e3f0780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.785+0000 7f17965fd740 1 -- 172.21.15.90:0/3781055137 shutdown_connections 2024-04-25T11:43:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.785+0000 7f17965fd740 1 -- 172.21.15.90:0/3781055137 wait complete. 2024-04-25T11:43:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.785+0000 7f17965fd740 20 client.18162 trim_cache size 0 max 16384 2024-04-25T11:43:12.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.804+0000 7f17965fd740 1 Processor -- start 2024-04-25T11:43:12.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.804+0000 7f17965fd740 1 -- start start 2024-04-25T11:43:12.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.805+0000 7f17965fd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d37e0424e0 0x55d37e3f8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.805+0000 7f17965fd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d37e00cf60 0x55d37e3ff420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.805+0000 7f17965fd740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d37e0254d0 0x55d37e3ff960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.805+0000 7f17965fd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f17881afc50 con 0x55d37e0424e0 2024-04-25T11:43:12.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.805+0000 7f17965fd740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f17881b1790 con 0x55d37e0254d0 2024-04-25T11:43:12.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.805+0000 7f17965fd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f17881b0320 con 0x55d37e00cf60 2024-04-25T11:43:12.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.805+0000 7f178d04c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d37e0424e0 0x55d37e3f8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.805+0000 7f178d04c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d37e0424e0 0x55d37e3f8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52856/0 (socket says 172.21.15.90:52856) 2024-04-25T11:43:12.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.805+0000 7f178d04c640 1 -- 172.21.15.90:0/4215095150 learned_addr learned my addr 172.21.15.90:0/4215095150 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:12.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.805+0000 7f178d84d640 1 --2- 172.21.15.90:0/4215095150 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d37e00cf60 0x55d37e3ff420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.805+0000 7f178d04c640 1 --2- 172.21.15.90:0/4215095150 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d37e0254d0 0x55d37e3ff960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.805+0000 7f178d04c640 1 -- 172.21.15.90:0/4215095150 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d37e00cf60 msgr2=0x55d37e3ff420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:12.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.805+0000 7f178d04c640 1 --2- 172.21.15.90:0/4215095150 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d37e00cf60 0x55d37e3ff420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.805+0000 7f178d04c640 1 -- 172.21.15.90:0/4215095150 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d37e0254d0 msgr2=0x55d37e3ff960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:12.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.805+0000 7f178d04c640 1 --2- 172.21.15.90:0/4215095150 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d37e0254d0 0x55d37e3ff960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.805+0000 7f178d04c640 1 -- 172.21.15.90:0/4215095150 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d37e0260f0 con 0x55d37e0424e0 2024-04-25T11:43:12.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.805+0000 7f178d04c640 1 --2- 172.21.15.90:0/4215095150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d37e0424e0 0x55d37e3f8700 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f1778001eb0 tx=0x7f1778009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e9084c269b69ac9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:12.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.805+0000 7f17737fe640 1 -- 172.21.15.90:0/4215095150 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1778056020 con 0x55d37e0424e0 2024-04-25T11:43:12.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.805+0000 7f17737fe640 1 -- 172.21.15.90:0/4215095150 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17780043e0 con 0x55d37e0424e0 2024-04-25T11:43:12.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.805+0000 7f17737fe640 1 -- 172.21.15.90:0/4215095150 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1778035070 con 0x55d37e0424e0 2024-04-25T11:43:12.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.806+0000 7f17965fd740 1 -- 172.21.15.90:0/4215095150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d37e0424e0 msgr2=0x55d37e3f8700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:12.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.806+0000 7f17965fd740 1 --2- 172.21.15.90:0/4215095150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d37e0424e0 0x55d37e3f8700 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f1778001eb0 tx=0x7f1778009520 comp rx=0 tx=0).stop 2024-04-25T11:43:12.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.806+0000 7f17965fd740 1 -- 172.21.15.90:0/4215095150 shutdown_connections 2024-04-25T11:43:12.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.806+0000 7f17965fd740 1 --2- 172.21.15.90:0/4215095150 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d37e0254d0 0x55d37e3ff960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.806+0000 7f17965fd740 1 --2- 172.21.15.90:0/4215095150 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d37e00cf60 0x55d37e3ff420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.806+0000 7f17965fd740 1 --2- 172.21.15.90:0/4215095150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d37e0424e0 0x55d37e3f8700 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.806+0000 7f17965fd740 1 -- 172.21.15.90:0/4215095150 >> 172.21.15.90:0/4215095150 conn(0x55d37e3eefc0 msgr2=0x55d37e3f0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:12.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.806+0000 7f17965fd740 1 -- 172.21.15.90:0/4215095150 shutdown_connections 2024-04-25T11:43:12.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.806+0000 7f17965fd740 1 -- 172.21.15.90:0/4215095150 wait complete. 2024-04-25T11:43:12.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.807+0000 7f17965fd740 1 -- start start 2024-04-25T11:43:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.807+0000 7f17965fd740 1 Processor -- start 2024-04-25T11:43:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.807+0000 7f17965fd740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d37e00cf60 0x55d37e405720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.807+0000 7f17965fd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d37e0254d0 0x55d37e405c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.807+0000 7f17965fd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d37e0424e0 0x55d37e400b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.807+0000 7f17965fd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f17881b10c0 con 0x55d37e0424e0 2024-04-25T11:43:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.807+0000 7f17965fd740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f17881b09f0 con 0x55d37e00cf60 2024-04-25T11:43:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.807+0000 7f17965fd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f17881adc50 con 0x55d37e0254d0 2024-04-25T11:43:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.807+0000 7f17965fd740 10 client.0 _ll_register_callbacks cb 0x7f1778032e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.807+0000 7f178d04c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d37e0424e0 0x55d37e400b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.807+0000 7f178d04c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d37e0424e0 0x55d37e400b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52864/0 (socket says 172.21.15.90:52864) 2024-04-25T11:43:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.807+0000 7f178d04c640 1 -- 172.21.15.90:0/1925501105 learned_addr learned my addr 172.21.15.90:0/1925501105 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.807+0000 7f178d04c640 1 --2- 172.21.15.90:0/1925501105 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d37e00cf60 0x55d37e405720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.808+0000 7f178d84d640 1 --2- 172.21.15.90:0/1925501105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d37e0254d0 0x55d37e405c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.808+0000 7f178d04c640 1 -- 172.21.15.90:0/1925501105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d37e0254d0 msgr2=0x55d37e405c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.808+0000 7f178d04c640 1 --2- 172.21.15.90:0/1925501105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d37e0254d0 0x55d37e405c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.808+0000 7f178d04c640 1 -- 172.21.15.90:0/1925501105 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d37e00cf60 msgr2=0x55d37e405720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.808+0000 7f178d04c640 1 --2- 172.21.15.90:0/1925501105 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d37e00cf60 0x55d37e405720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.808+0000 7f178d04c640 1 -- 172.21.15.90:0/1925501105 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d37e3154b0 con 0x55d37e0424e0 2024-04-25T11:43:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.808+0000 7f178d04c640 1 --2- 172.21.15.90:0/1925501105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d37e0424e0 0x55d37e400b70 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x55d37e4003d0 tx=0x7f1778047040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a66b46910033444 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.808+0000 7f17717fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.808+0000 7f17717fa640 1 -- 172.21.15.90:0/1925501105 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1778035070 con 0x55d37e0424e0 2024-04-25T11:43:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.808+0000 7f17717fa640 1 -- 172.21.15.90:0/1925501105 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7f1778005050 con 0x55d37e0424e0 2024-04-25T11:43:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.808+0000 7f17965fd740 1 -- 172.21.15.90:0/1925501105 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17780043e0 con 0x55d37e0424e0 2024-04-25T11:43:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.808+0000 7f17717fa640 1 -- 172.21.15.90:0/1925501105 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1778056070 con 0x55d37e0424e0 2024-04-25T11:43:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.808+0000 7f17965fd740 10 client.18168 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.808+0000 7f17965fd740 1 -- 172.21.15.90:0/1925501105 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d37e473190 con 0x55d37e0424e0 2024-04-25T11:43:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.808+0000 7f17717fa640 1 -- 172.21.15.90:0/1925501105 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17780043e0 con 0x55d37e0424e0 2024-04-25T11:43:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.809+0000 7f17717fa640 1 -- 172.21.15.90:0/1925501105 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f177805b070 con 0x55d37e0424e0 2024-04-25T11:43:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.809+0000 7f17965fd740 10 client.18168 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.809+0000 7f17965fd740 10 client.18168 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:43:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.809+0000 7f17965fd740 20 client.18168 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.809+0000 7f17965fd740 1 --2- 172.21.15.90:0/1925501105 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d37e02d6b0 0x55d37e475710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.809+0000 7f17965fd740 4 client.18168 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.809+0000 7f17965fd740 1 -- 172.21.15.90:0/1925501105 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d37e35c800 con 0x55d37e02d6b0 2024-04-25T11:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.809+0000 7f178d84d640 1 --2- 172.21.15.90:0/1925501105 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d37e02d6b0 0x55d37e475710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.809+0000 7f17717fa640 10 client.18168 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.809+0000 7f178d84d640 1 --2- 172.21.15.90:0/1925501105 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d37e02d6b0 0x55d37e475710 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfb28b2146d12e02 server_cookie=7866124388db95c1 in_seq=0 out_seq=0 2024-04-25T11:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.810+0000 7f17717fa640 1 -- 172.21.15.90:0/1925501105 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d37e35c800 con 0x55d37e02d6b0 2024-04-25T11:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.810+0000 7f17717fa640 10 client.18168 handle_command_reply: tid=0 2024-04-25T11:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.810+0000 7f17965fd740 1 client.18168 shutdown 2024-04-25T11:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.810+0000 7f17965fd740 1 -- 172.21.15.90:0/1925501105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d37e0424e0 msgr2=0x55d37e400b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.810+0000 7f17965fd740 1 --2- 172.21.15.90:0/1925501105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d37e0424e0 0x55d37e400b70 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x55d37e4003d0 tx=0x7f1778047040 comp rx=0 tx=0).stop 2024-04-25T11:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.810+0000 7f17965fd740 1 -- 172.21.15.90:0/1925501105 shutdown_connections 2024-04-25T11:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.810+0000 7f17965fd740 1 --2- 172.21.15.90:0/1925501105 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55d37e02d6b0 0x55d37e475710 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.810+0000 7f17965fd740 1 --2- 172.21.15.90:0/1925501105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d37e0424e0 0x55d37e400b70 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.810+0000 7f17965fd740 1 --2- 172.21.15.90:0/1925501105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d37e0254d0 0x55d37e405c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.810+0000 7f17717fa640 0 client.18168 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.810+0000 7f17965fd740 1 --2- 172.21.15.90:0/1925501105 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d37e00cf60 0x55d37e405720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.810+0000 7f17965fd740 1 -- 172.21.15.90:0/1925501105 >> 172.21.15.90:0/1925501105 conn(0x55d37e3eefc0 msgr2=0x55d37e3faa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.810+0000 7f17965fd740 1 -- 172.21.15.90:0/1925501105 shutdown_connections 2024-04-25T11:43:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.810+0000 7f17965fd740 1 -- 172.21.15.90:0/1925501105 wait complete. 2024-04-25T11:43:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.810+0000 7f17965fd740 20 client.18168 trim_cache size 0 max 16384 2024-04-25T11:43:12.801 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:43:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T11:43:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-25T11:43:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "feb0ab91-4578-46d3-84b3-82b5dede08c1": { 2024-04-25T11:43:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-25T11:43:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "feb0ab91-4578-46d3-84b3-82b5dede08c1", 2024-04-25T11:43:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-25T11:43:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:43:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:43:12.802 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:43:12.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.811+0000 7f178fab1640 1 -- 172.21.15.90:0/2137333156 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f17740afce0 msgr2=0x7f17740b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:12.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.811+0000 7f178fab1640 1 --2- 172.21.15.90:0/2137333156 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f17740afce0 0x7f17740b21f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f17800dce60 tx=0x7f1780075060 comp rx=0 tx=0).stop 2024-04-25T11:43:12.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.811+0000 7f178fab1640 1 -- 172.21.15.90:0/2137333156 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f178810d3a0 msgr2=0x7f178819ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:12.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.811+0000 7f178fab1640 1 --2- 172.21.15.90:0/2137333156 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f178810d3a0 0x7f178819ef10 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1780009fd0 tx=0x7f178000ef10 comp rx=0 tx=0).stop 2024-04-25T11:43:12.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.811+0000 7f178fab1640 1 -- 172.21.15.90:0/2137333156 shutdown_connections 2024-04-25T11:43:12.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.811+0000 7f178fab1640 1 --2- 172.21.15.90:0/2137333156 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f17740afce0 0x7f17740b21f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.811+0000 7f178fab1640 1 --2- 172.21.15.90:0/2137333156 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f178810d3a0 0x7f178819ef10 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.811+0000 7f178fab1640 1 --2- 172.21.15.90:0/2137333156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1788108a40 0x7f178819ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.811+0000 7f178fab1640 1 --2- 172.21.15.90:0/2137333156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1788102b10 0x7f178819a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.811+0000 7f178fab1640 1 -- 172.21.15.90:0/2137333156 >> 172.21.15.90:0/2137333156 conn(0x7f1788078180 msgr2=0x7f17881040d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:12.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.811+0000 7f178fab1640 1 -- 172.21.15.90:0/2137333156 shutdown_connections 2024-04-25T11:43:12.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:12.811+0000 7f178fab1640 1 -- 172.21.15.90:0/2137333156 wait complete. 2024-04-25T11:43:12.809 INFO:tasks.cephfs.filesystem:scrub status for tag:feb0ab91-4578-46d3-84b3-82b5dede08c1 - {'path': '/', 'tag': 'feb0ab91-4578-46d3-84b3-82b5dede08c1', 'options': 'recursive,force'} 2024-04-25T11:43:14.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:14 smithi138 ceph-mon[32328]: pgmap v250: 97 pgs: 24 active+clean+snaptrim, 33 active+clean, 40 active+clean+snaptrim_wait; 57 GiB data, 174 GiB used, 898 GiB / 1.0 TiB avail; 20 KiB/s rd, 67 KiB/s wr, 22 op/s 2024-04-25T11:43:14.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:14 smithi138 ceph-mon[32328]: scrub queued for path: / 2024-04-25T11:43:14.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:14 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:43:14.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:14 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:43:14.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:14 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:43:14.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:14 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:43:14.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:14 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:43:14.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:14 smithi156 ceph-mon[33045]: pgmap v250: 97 pgs: 24 active+clean+snaptrim, 33 active+clean, 40 active+clean+snaptrim_wait; 57 GiB data, 174 GiB used, 898 GiB / 1.0 TiB avail; 20 KiB/s rd, 67 KiB/s wr, 22 op/s 2024-04-25T11:43:14.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:14 smithi156 ceph-mon[33045]: scrub queued for path: / 2024-04-25T11:43:14.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:14 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:43:14.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:14 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:43:14.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:14 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:43:14.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:14 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:43:14.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:14 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:43:14.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:14 smithi090 ceph-mon[29575]: pgmap v250: 97 pgs: 24 active+clean+snaptrim, 33 active+clean, 40 active+clean+snaptrim_wait; 57 GiB data, 174 GiB used, 898 GiB / 1.0 TiB avail; 20 KiB/s rd, 67 KiB/s wr, 22 op/s 2024-04-25T11:43:14.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:14 smithi090 ceph-mon[29575]: scrub queued for path: / 2024-04-25T11:43:14.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:14 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:43:14.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:14 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:43:14.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:14 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:43:14.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:14 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:43:14.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:14 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:43:16.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:16 smithi138 ceph-mon[32328]: pgmap v251: 97 pgs: 22 active+clean+snaptrim, 42 active+clean, 33 active+clean+snaptrim_wait; 56 GiB data, 172 GiB used, 901 GiB / 1.0 TiB avail; 20 KiB/s rd, 67 KiB/s wr, 22 op/s 2024-04-25T11:43:16.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:16 smithi156 ceph-mon[33045]: pgmap v251: 97 pgs: 22 active+clean+snaptrim, 42 active+clean, 33 active+clean+snaptrim_wait; 56 GiB data, 172 GiB used, 901 GiB / 1.0 TiB avail; 20 KiB/s rd, 67 KiB/s wr, 22 op/s 2024-04-25T11:43:16.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:16 smithi090 ceph-mon[29575]: pgmap v251: 97 pgs: 22 active+clean+snaptrim, 42 active+clean, 33 active+clean+snaptrim_wait; 56 GiB data, 172 GiB used, 901 GiB / 1.0 TiB avail; 20 KiB/s rd, 67 KiB/s wr, 22 op/s 2024-04-25T11:43:18.066 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.080+0000 7f537126e640 1 -- 172.21.15.90:0/2512084078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f536c1031c0 msgr2=0x7f536c10f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:18.066 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.080+0000 7f537126e640 1 --2- 172.21.15.90:0/2512084078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f536c1031c0 0x7f536c10f4f0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f535c009900 tx=0x7f535c02a410 comp rx=0 tx=0).stop 2024-04-25T11:43:18.066 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.080+0000 7f537126e640 1 -- 172.21.15.90:0/2512084078 shutdown_connections 2024-04-25T11:43:18.067 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.080+0000 7f537126e640 1 --2- 172.21.15.90:0/2512084078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f536c1031c0 0x7f536c10f4f0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.067 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.080+0000 7f537126e640 1 --2- 172.21.15.90:0/2512084078 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f536c108f00 0x7f536c102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.067 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.080+0000 7f537126e640 1 --2- 172.21.15.90:0/2512084078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f536c108a60 0x7f536c102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.067 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.080+0000 7f537126e640 1 -- 172.21.15.90:0/2512084078 >> 172.21.15.90:0/2512084078 conn(0x7f536c0781b0 msgr2=0x7f536c0785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:18.067 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.080+0000 7f537126e640 1 -- 172.21.15.90:0/2512084078 shutdown_connections 2024-04-25T11:43:18.067 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.080+0000 7f537126e640 1 -- 172.21.15.90:0/2512084078 wait complete. 2024-04-25T11:43:18.067 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.080+0000 7f537126e640 1 Processor -- start 2024-04-25T11:43:18.067 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.081+0000 7f537126e640 1 -- start start 2024-04-25T11:43:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.081+0000 7f537126e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f536c1031c0 0x7f536c19b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.081+0000 7f537126e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f536c108a60 0x7f536c19baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.081+0000 7f537126e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f536c108f00 0x7f536c1a65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.081+0000 7f537126e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f536c112ca0 con 0x7f536c1031c0 2024-04-25T11:43:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.081+0000 7f537126e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f536c112b20 con 0x7f536c108a60 2024-04-25T11:43:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.081+0000 7f537126e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f536c112e20 con 0x7f536c108f00 2024-04-25T11:43:18.070 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.081+0000 7f536affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f536c1031c0 0x7f536c19b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:18.070 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.081+0000 7f536affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f536c1031c0 0x7f536c19b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52878/0 (socket says 172.21.15.90:52878) 2024-04-25T11:43:18.070 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.081+0000 7f536affd640 1 -- 172.21.15.90:0/3970623662 learned_addr learned my addr 172.21.15.90:0/3970623662 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:18.070 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.081+0000 7f536b7fe640 1 --2- 172.21.15.90:0/3970623662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f536c108f00 0x7f536c1a65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:18.070 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.081+0000 7f536a7fc640 1 --2- 172.21.15.90:0/3970623662 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f536c108a60 0x7f536c19baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:18.070 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.081+0000 7f536affd640 1 -- 172.21.15.90:0/3970623662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f536c108f00 msgr2=0x7f536c1a65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:18.070 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.081+0000 7f536affd640 1 --2- 172.21.15.90:0/3970623662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f536c108f00 0x7f536c1a65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.070 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.081+0000 7f536affd640 1 -- 172.21.15.90:0/3970623662 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f536c108a60 msgr2=0x7f536c19baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:18.070 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.081+0000 7f536affd640 1 --2- 172.21.15.90:0/3970623662 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f536c108a60 0x7f536c19baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.070 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.081+0000 7f536affd640 1 -- 172.21.15.90:0/3970623662 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f536c19c320 con 0x7f536c1031c0 2024-04-25T11:43:18.070 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.081+0000 7f536b7fe640 1 --2- 172.21.15.90:0/3970623662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f536c108f00 0x7f536c1a65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:43:18.070 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.082+0000 7f536affd640 1 --2- 172.21.15.90:0/3970623662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f536c1031c0 0x7f536c19b560 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f536400eb90 tx=0x7f536400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:18.070 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.082+0000 7f5353fff640 1 -- 172.21.15.90:0/3970623662 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5364004d20 con 0x7f536c1031c0 2024-04-25T11:43:18.071 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.082+0000 7f5353fff640 1 -- 172.21.15.90:0/3970623662 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f536400ee90 con 0x7f536c1031c0 2024-04-25T11:43:18.071 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.082+0000 7f537126e640 1 -- 172.21.15.90:0/3970623662 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f536c1a6be0 con 0x7f536c1031c0 2024-04-25T11:43:18.071 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.082+0000 7f5353fff640 1 -- 172.21.15.90:0/3970623662 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53640107f0 con 0x7f536c1031c0 2024-04-25T11:43:18.071 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.082+0000 7f537126e640 1 -- 172.21.15.90:0/3970623662 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f536c1a70e0 con 0x7f536c1031c0 2024-04-25T11:43:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.084+0000 7f5353fff640 1 -- 172.21.15.90:0/3970623662 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f5364004fc0 con 0x7f536c1031c0 2024-04-25T11:43:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.084+0000 7f5378202740 1 Processor -- start 2024-04-25T11:43:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.084+0000 7f5378202740 1 -- start start 2024-04-25T11:43:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.085+0000 7f5378202740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55899c6a31b0 0x55899ca42980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.085+0000 7f5378202740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c67a2f0 0x55899ca42ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.085+0000 7f5378202740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55899c6824d0 0x55899ca43400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.085+0000 7f5378202740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55899c9a9460 con 0x55899c67a2f0 2024-04-25T11:43:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.085+0000 7f5378202740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55899c963590 con 0x55899c6824d0 2024-04-25T11:43:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.085+0000 7f5378202740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55899c9a5420 con 0x55899c6a31b0 2024-04-25T11:43:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.085+0000 7f536affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c67a2f0 0x55899ca42ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.085+0000 7f536affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c67a2f0 0x55899ca42ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52886/0 (socket says 172.21.15.90:52886) 2024-04-25T11:43:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.085+0000 7f536affd640 1 -- 172.21.15.90:0/670985847 learned_addr learned my addr 172.21.15.90:0/670985847 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.085+0000 7f536b7fe640 1 --2- 172.21.15.90:0/670985847 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55899c6a31b0 0x55899ca42980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.085+0000 7f5353fff640 1 --2- 172.21.15.90:0/3970623662 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f53540afc50 0x7f53540b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.085+0000 7f536b7fe640 1 --2- 172.21.15.90:0/3970623662 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f53540afc50 0x7f53540b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.085+0000 7f536affd640 1 -- 172.21.15.90:0/670985847 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55899c6a31b0 msgr2=0x55899ca42980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.085+0000 7f536affd640 1 --2- 172.21.15.90:0/670985847 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55899c6a31b0 0x55899ca42980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.085+0000 7f536affd640 1 -- 172.21.15.90:0/670985847 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55899c6824d0 msgr2=0x55899ca43400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.085+0000 7f536affd640 1 --2- 172.21.15.90:0/670985847 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55899c6824d0 0x55899ca43400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.085+0000 7f5353fff640 1 -- 172.21.15.90:0/3970623662 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7f53640dfa20 con 0x7f536c1031c0 2024-04-25T11:43:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.085+0000 7f536affd640 1 -- 172.21.15.90:0/670985847 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55899c9700d0 con 0x55899c67a2f0 2024-04-25T11:43:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.085+0000 7f536b7fe640 1 --2- 172.21.15.90:0/670985847 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55899c6a31b0 0x55899ca42980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:43:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.085+0000 7f536affd640 1 --2- 172.21.15.90:0/670985847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c67a2f0 0x55899ca42ec0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f53640a7810 tx=0x7f5364005930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7f2df9494a0fc03 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.086+0000 7f53517fa640 1 -- 172.21.15.90:0/670985847 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53640a7280 con 0x55899c67a2f0 2024-04-25T11:43:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.086+0000 7f536b7fe640 1 --2- 172.21.15.90:0/3970623662 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f53540afc50 0x7f53540b2160 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x55899ca3fc90 tx=0x7f535c02a410 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.086+0000 7f53517fa640 1 -- 172.21.15.90:0/670985847 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53640b02c0 con 0x55899c67a2f0 2024-04-25T11:43:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.086+0000 7f53517fa640 1 -- 172.21.15.90:0/670985847 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53640abec0 con 0x55899c67a2f0 2024-04-25T11:43:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.086+0000 7f5378202740 1 -- 172.21.15.90:0/670985847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c67a2f0 msgr2=0x55899ca42ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.086+0000 7f5378202740 1 --2- 172.21.15.90:0/670985847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c67a2f0 0x55899ca42ec0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f53640a7810 tx=0x7f5364005930 comp rx=0 tx=0).stop 2024-04-25T11:43:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.086+0000 7f5378202740 1 -- 172.21.15.90:0/670985847 shutdown_connections 2024-04-25T11:43:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.086+0000 7f5378202740 1 --2- 172.21.15.90:0/670985847 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55899c6824d0 0x55899ca43400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.086+0000 7f5378202740 1 --2- 172.21.15.90:0/670985847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c67a2f0 0x55899ca42ec0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.086+0000 7f5378202740 1 --2- 172.21.15.90:0/670985847 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55899c6a31b0 0x55899ca42980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.086+0000 7f5378202740 1 -- 172.21.15.90:0/670985847 >> 172.21.15.90:0/670985847 conn(0x55899c697300 msgr2=0x55899ca3f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.087+0000 7f5378202740 1 -- 172.21.15.90:0/670985847 shutdown_connections 2024-04-25T11:43:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.087+0000 7f5378202740 1 -- 172.21.15.90:0/670985847 wait complete. 2024-04-25T11:43:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.087+0000 7f5378202740 1 -- start start 2024-04-25T11:43:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.087+0000 7f5378202740 1 Processor -- start 2024-04-25T11:43:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.087+0000 7f5378202740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c67a2f0 0x55899ca51590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.087+0000 7f5378202740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55899c6824d0 0x55899ca51ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:18.080 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.087+0000 7f5378202740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55899c6a31b0 0x55899ca57450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:18.080 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.087+0000 7f5378202740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55899ca53fd0 con 0x55899c67a2f0 2024-04-25T11:43:18.080 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.087+0000 7f5378202740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55899c930b60 con 0x55899c6a31b0 2024-04-25T11:43:18.080 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.087+0000 7f5378202740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55899ca54150 con 0x55899c6824d0 2024-04-25T11:43:18.080 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.087+0000 7f5378202740 10 client.0 _ll_register_callbacks cb 0x55899c9552b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:18.080 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.087+0000 7f536affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c67a2f0 0x55899ca51590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:18.080 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.087+0000 7f536affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c67a2f0 0x55899ca51590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52892/0 (socket says 172.21.15.90:52892) 2024-04-25T11:43:18.080 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.087+0000 7f536affd640 1 -- 172.21.15.90:0/3434968011 learned_addr learned my addr 172.21.15.90:0/3434968011 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:18.080 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.088+0000 7f536affd640 1 -- 172.21.15.90:0/3434968011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55899c6824d0 msgr2=0x55899ca51ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:18.080 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.088+0000 7f536affd640 1 --2- 172.21.15.90:0/3434968011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55899c6824d0 0x55899ca51ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.080 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.088+0000 7f536affd640 1 -- 172.21.15.90:0/3434968011 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55899c6a31b0 msgr2=0x55899ca57450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:18.080 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.088+0000 7f536affd640 1 --2- 172.21.15.90:0/3434968011 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55899c6a31b0 0x55899ca57450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.080 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.088+0000 7f536affd640 1 -- 172.21.15.90:0/3434968011 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55899c6bb210 con 0x55899c67a2f0 2024-04-25T11:43:18.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.088+0000 7f536affd640 1 --2- 172.21.15.90:0/3434968011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c67a2f0 0x55899ca51590 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f53640a7aa0 tx=0x7f536400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb883cf4ec95cb5f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:18.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.088+0000 7f534eff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:18.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.088+0000 7f534eff5640 1 -- 172.21.15.90:0/3434968011 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53640a7f70 con 0x55899c67a2f0 2024-04-25T11:43:18.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.088+0000 7f534eff5640 1 -- 172.21.15.90:0/3434968011 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7f53640b04e0 con 0x55899c67a2f0 2024-04-25T11:43:18.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.088+0000 7f5378202740 1 -- 172.21.15.90:0/3434968011 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53640b02c0 con 0x55899c67a2f0 2024-04-25T11:43:18.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.088+0000 7f534eff5640 1 -- 172.21.15.90:0/3434968011 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f53640a7620 con 0x55899c67a2f0 2024-04-25T11:43:18.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.088+0000 7f5378202740 10 client.18180 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:18.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.088+0000 7f5378202740 1 -- 172.21.15.90:0/3434968011 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55899c787620 con 0x55899c67a2f0 2024-04-25T11:43:18.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.088+0000 7f534eff5640 1 -- 172.21.15.90:0/3434968011 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53640b02c0 con 0x55899c67a2f0 2024-04-25T11:43:18.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.089+0000 7f534eff5640 1 -- 172.21.15.90:0/3434968011 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f53640a3480 con 0x55899c67a2f0 2024-04-25T11:43:18.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.089+0000 7f5378202740 10 client.18180 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:18.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.089+0000 7f5378202740 10 client.18180 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:43:18.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.089+0000 7f5378202740 20 client.18180 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:18.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.089+0000 7f5378202740 1 --2- 172.21.15.90:0/3434968011 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55899c661d80 0x55899ca638d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.089+0000 7f5378202740 4 client.18180 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:43:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.089+0000 7f5378202740 1 -- 172.21.15.90:0/3434968011 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55899c9b7070 con 0x55899c661d80 2024-04-25T11:43:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.089+0000 7f536a7fc640 1 --2- 172.21.15.90:0/3434968011 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55899c661d80 0x55899ca638d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.090+0000 7f534eff5640 10 client.18180 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.090+0000 7f536a7fc640 1 --2- 172.21.15.90:0/3434968011 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55899c661d80 0x55899ca638d0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=967fe153f1478433 server_cookie=cdcfd787d96248ea in_seq=0 out_seq=0 2024-04-25T11:43:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.091+0000 7f534eff5640 1 -- 172.21.15.90:0/3434968011 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55899c9b7070 con 0x55899c661d80 2024-04-25T11:43:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.091+0000 7f534eff5640 10 client.18180 handle_command_reply: tid=0 2024-04-25T11:43:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.091+0000 7f5378202740 1 client.18180 shutdown 2024-04-25T11:43:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.091+0000 7f5378202740 1 -- 172.21.15.90:0/3434968011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c67a2f0 msgr2=0x55899ca51590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.091+0000 7f5378202740 1 --2- 172.21.15.90:0/3434968011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c67a2f0 0x55899ca51590 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f53640a7aa0 tx=0x7f536400cf60 comp rx=0 tx=0).stop 2024-04-25T11:43:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.091+0000 7f5378202740 1 -- 172.21.15.90:0/3434968011 shutdown_connections 2024-04-25T11:43:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.091+0000 7f5378202740 1 --2- 172.21.15.90:0/3434968011 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55899c661d80 0x55899ca638d0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.091+0000 7f5378202740 1 --2- 172.21.15.90:0/3434968011 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55899c6a31b0 0x55899ca57450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.091+0000 7f534eff5640 0 client.18180 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.091+0000 7f5378202740 1 --2- 172.21.15.90:0/3434968011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55899c6824d0 0x55899ca51ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.091+0000 7f5378202740 1 --2- 172.21.15.90:0/3434968011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c67a2f0 0x55899ca51590 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.091+0000 7f5378202740 1 -- 172.21.15.90:0/3434968011 >> 172.21.15.90:0/3434968011 conn(0x55899c697300 msgr2=0x55899ca44810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.091+0000 7f5378202740 1 -- 172.21.15.90:0/3434968011 shutdown_connections 2024-04-25T11:43:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.091+0000 7f5378202740 1 -- 172.21.15.90:0/3434968011 wait complete. 2024-04-25T11:43:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.091+0000 7f5378202740 20 client.18180 trim_cache size 0 max 16384 2024-04-25T11:43:18.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.107+0000 7f5378202740 1 Processor -- start 2024-04-25T11:43:18.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.107+0000 7f5378202740 1 -- start start 2024-04-25T11:43:18.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.107+0000 7f5378202740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c661d80 0x55899ca57380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:18.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.107+0000 7f5378202740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55899c67a2f0 0x55899ca68ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:18.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.107+0000 7f5378202740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55899c6824d0 0x55899ca69410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:18.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.107+0000 7f5378202740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f536c1adb10 con 0x55899c661d80 2024-04-25T11:43:18.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.107+0000 7f5378202740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f536c1af650 con 0x55899c67a2f0 2024-04-25T11:43:18.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.107+0000 7f5378202740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f536c1ae1e0 con 0x55899c6824d0 2024-04-25T11:43:18.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.107+0000 7f536affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c661d80 0x55899ca57380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:18.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.107+0000 7f536affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c661d80 0x55899ca57380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52896/0 (socket says 172.21.15.90:52896) 2024-04-25T11:43:18.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.107+0000 7f536affd640 1 -- 172.21.15.90:0/1414904293 learned_addr learned my addr 172.21.15.90:0/1414904293 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:18.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.107+0000 7f536a7fc640 1 --2- 172.21.15.90:0/1414904293 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55899c67a2f0 0x55899ca68ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:18.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.107+0000 7f536affd640 1 -- 172.21.15.90:0/1414904293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55899c6824d0 msgr2=0x55899ca69410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:18.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.107+0000 7f536affd640 1 --2- 172.21.15.90:0/1414904293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55899c6824d0 0x55899ca69410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.107+0000 7f536affd640 1 -- 172.21.15.90:0/1414904293 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55899c67a2f0 msgr2=0x55899ca68ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:18.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.107+0000 7f536affd640 1 --2- 172.21.15.90:0/1414904293 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55899c67a2f0 0x55899ca68ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.107+0000 7f536affd640 1 -- 172.21.15.90:0/1414904293 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55899ca45740 con 0x55899c661d80 2024-04-25T11:43:18.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.108+0000 7f536affd640 1 --2- 172.21.15.90:0/1414904293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c661d80 0x55899ca57380 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f53640a75d0 tx=0x7f5364005930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=594184c7d3e41bf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:18.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.108+0000 7f5350ff9640 1 -- 172.21.15.90:0/1414904293 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53640a8210 con 0x55899c661d80 2024-04-25T11:43:18.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.108+0000 7f5350ff9640 1 -- 172.21.15.90:0/1414904293 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53640b0970 con 0x55899c661d80 2024-04-25T11:43:18.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.108+0000 7f5350ff9640 1 -- 172.21.15.90:0/1414904293 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53640b0740 con 0x55899c661d80 2024-04-25T11:43:18.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.108+0000 7f5378202740 1 -- 172.21.15.90:0/1414904293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c661d80 msgr2=0x55899ca57380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:18.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.108+0000 7f5378202740 1 --2- 172.21.15.90:0/1414904293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c661d80 0x55899ca57380 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f53640a75d0 tx=0x7f5364005930 comp rx=0 tx=0).stop 2024-04-25T11:43:18.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.108+0000 7f5378202740 1 -- 172.21.15.90:0/1414904293 shutdown_connections 2024-04-25T11:43:18.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.108+0000 7f5378202740 1 --2- 172.21.15.90:0/1414904293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55899c6824d0 0x55899ca69410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.100 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.109+0000 7f5378202740 1 --2- 172.21.15.90:0/1414904293 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55899c67a2f0 0x55899ca68ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.100 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.109+0000 7f5378202740 1 --2- 172.21.15.90:0/1414904293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c661d80 0x55899ca57380 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.100 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.109+0000 7f5378202740 1 -- 172.21.15.90:0/1414904293 >> 172.21.15.90:0/1414904293 conn(0x55899c697300 msgr2=0x55899ca432b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:18.100 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.109+0000 7f5378202740 1 -- 172.21.15.90:0/1414904293 shutdown_connections 2024-04-25T11:43:18.100 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.109+0000 7f5378202740 1 -- 172.21.15.90:0/1414904293 wait complete. 2024-04-25T11:43:18.100 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.109+0000 7f5378202740 1 -- start start 2024-04-25T11:43:18.100 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.109+0000 7f5378202740 1 Processor -- start 2024-04-25T11:43:18.100 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.109+0000 7f5378202740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55899c661d80 0x55899ca5d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:18.100 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.109+0000 7f5378202740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c67a2f0 0x55899ca5de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:18.100 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.109+0000 7f5378202740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55899c6824d0 0x55899ca58d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:18.100 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.109+0000 7f5378202740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f536c1aef80 con 0x55899c67a2f0 2024-04-25T11:43:18.100 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.109+0000 7f5378202740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f536c1ae8b0 con 0x55899c661d80 2024-04-25T11:43:18.100 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.109+0000 7f5378202740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f536c1abb10 con 0x55899c6824d0 2024-04-25T11:43:18.100 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.109+0000 7f5378202740 10 client.0 _ll_register_callbacks cb 0x7f53640093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:18.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.109+0000 7f536a7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c67a2f0 0x55899ca5de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:18.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.109+0000 7f536a7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c67a2f0 0x55899ca5de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52900/0 (socket says 172.21.15.90:52900) 2024-04-25T11:43:18.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.109+0000 7f536a7fc640 1 -- 172.21.15.90:0/4028125613 learned_addr learned my addr 172.21.15.90:0/4028125613 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:18.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.110+0000 7f536affd640 1 --2- 172.21.15.90:0/4028125613 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55899c661d80 0x55899ca5d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:18.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.110+0000 7f536a7fc640 1 -- 172.21.15.90:0/4028125613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55899c6824d0 msgr2=0x55899ca58d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:18.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.110+0000 7f536a7fc640 1 --2- 172.21.15.90:0/4028125613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55899c6824d0 0x55899ca58d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.110+0000 7f536a7fc640 1 -- 172.21.15.90:0/4028125613 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55899c661d80 msgr2=0x55899ca5d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:18.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.110+0000 7f536a7fc640 1 --2- 172.21.15.90:0/4028125613 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55899c661d80 0x55899ca5d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.110+0000 7f536a7fc640 1 -- 172.21.15.90:0/4028125613 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55899ca594c0 con 0x55899c67a2f0 2024-04-25T11:43:18.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.110+0000 7f536a7fc640 1 --2- 172.21.15.90:0/4028125613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c67a2f0 0x55899ca5de60 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f536c100370 tx=0x7f5360009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34cd97e178270525 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:18.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.110+0000 7f534eff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:18.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.110+0000 7f534eff5640 1 -- 172.21.15.90:0/4028125613 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5360005030 con 0x55899c67a2f0 2024-04-25T11:43:18.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.110+0000 7f534eff5640 1 -- 172.21.15.90:0/4028125613 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 8702+0+0 (secure 0 0 0) 0x7f53600051d0 con 0x55899c67a2f0 2024-04-25T11:43:18.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.110+0000 7f5378202740 1 -- 172.21.15.90:0/4028125613 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53640b0970 con 0x55899c67a2f0 2024-04-25T11:43:18.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.110+0000 7f534eff5640 1 -- 172.21.15.90:0/4028125613 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5360028050 con 0x55899c67a2f0 2024-04-25T11:43:18.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.110+0000 7f5378202740 10 client.18186 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:18.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.110+0000 7f5378202740 1 -- 172.21.15.90:0/4028125613 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55899cacb5b0 con 0x55899c67a2f0 2024-04-25T11:43:18.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.110+0000 7f534eff5640 1 -- 172.21.15.90:0/4028125613 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53640b0970 con 0x55899c67a2f0 2024-04-25T11:43:18.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.110+0000 7f534eff5640 1 -- 172.21.15.90:0/4028125613 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f5360018ce0 con 0x55899c67a2f0 2024-04-25T11:43:18.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.110+0000 7f5378202740 10 client.18186 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:18.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.110+0000 7f5378202740 10 client.18186 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:43:18.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.110+0000 7f5378202740 20 client.18186 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:18.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.111+0000 7f5378202740 1 --2- 172.21.15.90:0/4028125613 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55899c6a31b0 0x55899cacdda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:18.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.111+0000 7f5378202740 4 client.18186 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:43:18.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.111+0000 7f5378202740 1 -- 172.21.15.90:0/4028125613 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55899c9b0ee0 con 0x55899c6a31b0 2024-04-25T11:43:18.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.111+0000 7f536affd640 1 --2- 172.21.15.90:0/4028125613 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55899c6a31b0 0x55899cacdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:18.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.111+0000 7f534eff5640 10 client.18186 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:18.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.111+0000 7f536affd640 1 --2- 172.21.15.90:0/4028125613 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55899c6a31b0 0x55899cacdda0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d987f04171464d0 server_cookie=eb4ea93b4559df8e in_seq=0 out_seq=0 2024-04-25T11:43:18.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.121+0000 7f534eff5640 1 -- 172.21.15.90:0/4028125613 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55899c9b0ee0 con 0x55899c6a31b0 2024-04-25T11:43:18.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.121+0000 7f534eff5640 10 client.18186 handle_command_reply: tid=0 2024-04-25T11:43:18.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.121+0000 7f5378202740 1 client.18186 shutdown 2024-04-25T11:43:18.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.121+0000 7f5378202740 1 -- 172.21.15.90:0/4028125613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c67a2f0 msgr2=0x55899ca5de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:18.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.121+0000 7f5378202740 1 --2- 172.21.15.90:0/4028125613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c67a2f0 0x55899ca5de60 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f536c100370 tx=0x7f5360009f80 comp rx=0 tx=0).stop 2024-04-25T11:43:18.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.121+0000 7f5378202740 1 -- 172.21.15.90:0/4028125613 shutdown_connections 2024-04-25T11:43:18.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.121+0000 7f5378202740 1 --2- 172.21.15.90:0/4028125613 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55899c6a31b0 0x55899cacdda0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.121+0000 7f5378202740 1 --2- 172.21.15.90:0/4028125613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55899c6824d0 0x55899ca58d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.121+0000 7f5378202740 1 --2- 172.21.15.90:0/4028125613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55899c67a2f0 0x55899ca5de60 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.121+0000 7f534eff5640 0 client.18186 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:18.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.121+0000 7f5378202740 1 --2- 172.21.15.90:0/4028125613 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55899c661d80 0x55899ca5d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.121+0000 7f5378202740 1 -- 172.21.15.90:0/4028125613 >> 172.21.15.90:0/4028125613 conn(0x55899c697300 msgr2=0x55899ca44ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:18.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.121+0000 7f5378202740 1 -- 172.21.15.90:0/4028125613 shutdown_connections 2024-04-25T11:43:18.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.121+0000 7f5378202740 1 -- 172.21.15.90:0/4028125613 wait complete. 2024-04-25T11:43:18.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.121+0000 7f5378202740 20 client.18186 trim_cache size 0 max 16384 2024-04-25T11:43:18.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:43:18.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:43:18.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:43:18.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:43:18.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.122+0000 7f537126e640 1 -- 172.21.15.90:0/3970623662 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f53540afc50 msgr2=0x7f53540b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:18.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.122+0000 7f537126e640 1 --2- 172.21.15.90:0/3970623662 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f53540afc50 0x7f53540b2160 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x55899ca3fc90 tx=0x7f535c02a410 comp rx=0 tx=0).stop 2024-04-25T11:43:18.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.122+0000 7f537126e640 1 -- 172.21.15.90:0/3970623662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f536c1031c0 msgr2=0x7f536c19b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:18.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.122+0000 7f537126e640 1 --2- 172.21.15.90:0/3970623662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f536c1031c0 0x7f536c19b560 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f536400eb90 tx=0x7f536400c6a0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.122+0000 7f537126e640 1 -- 172.21.15.90:0/3970623662 shutdown_connections 2024-04-25T11:43:18.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.122+0000 7f537126e640 1 --2- 172.21.15.90:0/3970623662 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f53540afc50 0x7f53540b2160 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.122+0000 7f537126e640 1 --2- 172.21.15.90:0/3970623662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f536c108f00 0x7f536c1a65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.122+0000 7f537126e640 1 --2- 172.21.15.90:0/3970623662 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f536c108a60 0x7f536c19baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.122+0000 7f537126e640 1 --2- 172.21.15.90:0/3970623662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f536c1031c0 0x7f536c19b560 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.122+0000 7f537126e640 1 -- 172.21.15.90:0/3970623662 >> 172.21.15.90:0/3970623662 conn(0x7f536c0781b0 msgr2=0x7f536c100dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:18.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.122+0000 7f537126e640 1 -- 172.21.15.90:0/3970623662 shutdown_connections 2024-04-25T11:43:18.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:18.122+0000 7f537126e640 1 -- 172.21.15.90:0/3970623662 wait complete. 2024-04-25T11:43:18.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:18 smithi138 ceph-mon[32328]: pgmap v252: 97 pgs: 12 active+clean+snaptrim, 61 active+clean, 24 active+clean+snaptrim_wait; 53 GiB data, 164 GiB used, 909 GiB / 1.0 TiB avail; 14 KiB/s rd, 9.6 KiB/s wr, 15 op/s 2024-04-25T11:43:18.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:18 smithi156 ceph-mon[33045]: pgmap v252: 97 pgs: 12 active+clean+snaptrim, 61 active+clean, 24 active+clean+snaptrim_wait; 53 GiB data, 164 GiB used, 909 GiB / 1.0 TiB avail; 14 KiB/s rd, 9.6 KiB/s wr, 15 op/s 2024-04-25T11:43:18.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:18 smithi090 ceph-mon[29575]: pgmap v252: 97 pgs: 12 active+clean+snaptrim, 61 active+clean, 24 active+clean+snaptrim_wait; 53 GiB data, 164 GiB used, 909 GiB / 1.0 TiB avail; 14 KiB/s rd, 9.6 KiB/s wr, 15 op/s 2024-04-25T11:43:20.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:20 smithi138 ceph-mon[32328]: pgmap v253: 97 pgs: 11 active+clean+snaptrim, 66 active+clean, 20 active+clean+snaptrim_wait; 52 GiB data, 161 GiB used, 911 GiB / 1.0 TiB avail; 13 KiB/s rd, 8.3 KiB/s wr, 13 op/s 2024-04-25T11:43:20.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:20 smithi156 ceph-mon[33045]: pgmap v253: 97 pgs: 11 active+clean+snaptrim, 66 active+clean, 20 active+clean+snaptrim_wait; 52 GiB data, 161 GiB used, 911 GiB / 1.0 TiB avail; 13 KiB/s rd, 8.3 KiB/s wr, 13 op/s 2024-04-25T11:43:20.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:20 smithi090 ceph-mon[29575]: pgmap v253: 97 pgs: 11 active+clean+snaptrim, 66 active+clean, 20 active+clean+snaptrim_wait; 52 GiB data, 161 GiB used, 911 GiB / 1.0 TiB avail; 13 KiB/s rd, 8.3 KiB/s wr, 13 op/s 2024-04-25T11:43:21.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:21 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:43:21.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:21 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:43:21.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:21 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:43:22.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:22 smithi138 ceph-mon[32328]: pgmap v254: 97 pgs: 3 active+clean+snaptrim, 88 active+clean, 6 active+clean+snaptrim_wait; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 18 KiB/s rd, 11 KiB/s wr, 17 op/s 2024-04-25T11:43:22.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:22 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:43:22.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:22 smithi156 ceph-mon[33045]: pgmap v254: 97 pgs: 3 active+clean+snaptrim, 88 active+clean, 6 active+clean+snaptrim_wait; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 18 KiB/s rd, 11 KiB/s wr, 17 op/s 2024-04-25T11:43:22.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:22 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:43:22.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:22 smithi090 ceph-mon[29575]: pgmap v254: 97 pgs: 3 active+clean+snaptrim, 88 active+clean, 6 active+clean+snaptrim_wait; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 18 KiB/s rd, 11 KiB/s wr, 17 op/s 2024-04-25T11:43:22.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:22 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:43:23.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:23 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:43:23.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:23 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:43:23.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:23 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:43:24.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:24 smithi156 ceph-mon[33045]: pgmap v255: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 9.8 KiB/s rd, 2.8 KiB/s wr, 11 op/s 2024-04-25T11:43:24.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:24 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:43:24.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:24 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:43:24.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:24 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:43:24.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:24 smithi156 ceph-mon[33045]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T11:43:24.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:24 smithi090 ceph-mon[29575]: pgmap v255: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 9.8 KiB/s rd, 2.8 KiB/s wr, 11 op/s 2024-04-25T11:43:24.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:24 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:43:24.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:24 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:43:24.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:24 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:43:24.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:24 smithi090 ceph-mon[29575]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T11:43:24.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:24 smithi138 ceph-mon[32328]: pgmap v255: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 9.8 KiB/s rd, 2.8 KiB/s wr, 11 op/s 2024-04-25T11:43:24.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:24 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:43:24.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:24 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:43:24.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:24 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:43:24.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:24 smithi138 ceph-mon[32328]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T11:43:26.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:26 smithi156 ceph-mon[33045]: pgmap v257: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 11 KiB/s rd, 3.4 KiB/s wr, 13 op/s 2024-04-25T11:43:26.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:26 smithi090 ceph-mon[29575]: pgmap v257: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 11 KiB/s rd, 3.4 KiB/s wr, 13 op/s 2024-04-25T11:43:26.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:26 smithi138 ceph-mon[32328]: pgmap v257: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 11 KiB/s rd, 3.4 KiB/s wr, 13 op/s 2024-04-25T11:43:28.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:28 smithi156 ceph-mon[33045]: pgmap v258: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.3 KiB/s wr, 11 op/s 2024-04-25T11:43:28.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:28 smithi090 ceph-mon[29575]: pgmap v258: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.3 KiB/s wr, 11 op/s 2024-04-25T11:43:28.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:28 smithi138 ceph-mon[32328]: pgmap v258: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.3 KiB/s wr, 11 op/s 2024-04-25T11:43:30.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:30 smithi156 ceph-mon[33045]: pgmap v259: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 8.0 KiB/s rd, 3.3 KiB/s wr, 10 op/s 2024-04-25T11:43:30.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:30 smithi090 ceph-mon[29575]: pgmap v259: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 8.0 KiB/s rd, 3.3 KiB/s wr, 10 op/s 2024-04-25T11:43:30.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:30 smithi138 ceph-mon[32328]: pgmap v259: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 8.0 KiB/s rd, 3.3 KiB/s wr, 10 op/s 2024-04-25T11:43:31.215 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.228+0000 7f49fcf14640 1 -- 172.21.15.90:0/3081811053 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f810b6d0 msgr2=0x7f49f810daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:31.215 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.228+0000 7f49fcf14640 1 --2- 172.21.15.90:0/3081811053 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f810b6d0 0x7f49f810daf0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f49e0009900 tx=0x7f49e002a410 comp rx=0 tx=0).stop 2024-04-25T11:43:31.215 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.228+0000 7f49fcf14640 1 -- 172.21.15.90:0/3081811053 shutdown_connections 2024-04-25T11:43:31.215 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.228+0000 7f49fcf14640 1 --2- 172.21.15.90:0/3081811053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49f810e0c0 0x7f49f81105b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.215 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.228+0000 7f49fcf14640 1 --2- 172.21.15.90:0/3081811053 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f810b6d0 0x7f49f810daf0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.215 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.228+0000 7f49fcf14640 1 --2- 172.21.15.90:0/3081811053 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f49f8104c70 0x7f49f8105080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.215 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.228+0000 7f49fcf14640 1 -- 172.21.15.90:0/3081811053 >> 172.21.15.90:0/3081811053 conn(0x7f49f80781b0 msgr2=0x7f49f80785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:31.215 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.228+0000 7f49fcf14640 1 -- 172.21.15.90:0/3081811053 shutdown_connections 2024-04-25T11:43:31.215 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.228+0000 7f49fcf14640 1 -- 172.21.15.90:0/3081811053 wait complete. 2024-04-25T11:43:31.216 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.229+0000 7f49fcf14640 1 Processor -- start 2024-04-25T11:43:31.216 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.229+0000 7f49fcf14640 1 -- start start 2024-04-25T11:43:31.217 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.229+0000 7f49fcf14640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f8104c70 0x7f49f819a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:31.218 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.229+0000 7f49fcf14640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49f810b6d0 0x7f49f819ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:31.218 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.229+0000 7f49fcf14640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f49f810e0c0 0x7f49f819f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:31.218 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.229+0000 7f49fcf14640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f49f8113d60 con 0x7f49f8104c70 2024-04-25T11:43:31.218 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.229+0000 7f49fcf14640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f49f8113be0 con 0x7f49f810e0c0 2024-04-25T11:43:31.218 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.229+0000 7f49fcf14640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f49f8113ee0 con 0x7f49f810b6d0 2024-04-25T11:43:31.218 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.230+0000 7f49f659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f8104c70 0x7f49f819a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:31.218 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.230+0000 7f49f659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f8104c70 0x7f49f819a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46764/0 (socket says 172.21.15.90:46764) 2024-04-25T11:43:31.218 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.230+0000 7f49f659c640 1 -- 172.21.15.90:0/145610371 learned_addr learned my addr 172.21.15.90:0/145610371 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:31.218 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.230+0000 7f49f6d9d640 1 --2- 172.21.15.90:0/145610371 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f49f810e0c0 0x7f49f819f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:31.218 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.230+0000 7f49effff640 1 --2- 172.21.15.90:0/145610371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49f810b6d0 0x7f49f819ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:31.218 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.230+0000 7f49f659c640 1 -- 172.21.15.90:0/145610371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49f810b6d0 msgr2=0x7f49f819ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:31.218 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.230+0000 7f49f659c640 1 --2- 172.21.15.90:0/145610371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49f810b6d0 0x7f49f819ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.218 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.230+0000 7f49f659c640 1 -- 172.21.15.90:0/145610371 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f49f810e0c0 msgr2=0x7f49f819f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:31.219 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.230+0000 7f49f659c640 1 --2- 172.21.15.90:0/145610371 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f49f810e0c0 0x7f49f819f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.219 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.230+0000 7f49f659c640 1 -- 172.21.15.90:0/145610371 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49f819f640 con 0x7f49f8104c70 2024-04-25T11:43:31.219 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.230+0000 7f49f6d9d640 1 --2- 172.21.15.90:0/145610371 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f49f810e0c0 0x7f49f819f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:31.219 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.230+0000 7f49f659c640 1 --2- 172.21.15.90:0/145610371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f8104c70 0x7f49f819a7e0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f49e800c890 tx=0x7f49e800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:31.219 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.230+0000 7f49ef7fe640 1 -- 172.21.15.90:0/145610371 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e8004220 con 0x7f49f8104c70 2024-04-25T11:43:31.219 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.230+0000 7f49fcf14640 1 -- 172.21.15.90:0/145610371 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49f819f930 con 0x7f49f8104c70 2024-04-25T11:43:31.219 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.230+0000 7f49ef7fe640 1 -- 172.21.15.90:0/145610371 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49e80043c0 con 0x7f49f8104c70 2024-04-25T11:43:31.219 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.230+0000 7f49ef7fe640 1 -- 172.21.15.90:0/145610371 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e8004a00 con 0x7f49f8104c70 2024-04-25T11:43:31.219 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.230+0000 7f49fcf14640 1 -- 172.21.15.90:0/145610371 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49f81a72b0 con 0x7f49f8104c70 2024-04-25T11:43:31.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.233+0000 7f4a03dc7740 1 Processor -- start 2024-04-25T11:43:31.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.233+0000 7f49ef7fe640 1 -- 172.21.15.90:0/145610371 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f49e80079b0 con 0x7f49f8104c70 2024-04-25T11:43:31.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.233+0000 7f4a03dc7740 1 -- start start 2024-04-25T11:43:31.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.233+0000 7f4a03dc7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564426f8a1b0 0x564427329af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.233+0000 7f4a03dc7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f612f0 0x56442732a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.233+0000 7f4a03dc7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564426f694d0 0x56442732a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.233+0000 7f4a03dc7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564427290460 con 0x564426f612f0 2024-04-25T11:43:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.233+0000 7f4a03dc7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56442724a590 con 0x564426f8a1b0 2024-04-25T11:43:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.233+0000 7f4a03dc7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56442728c420 con 0x564426f694d0 2024-04-25T11:43:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.233+0000 7f49f659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f612f0 0x56442732a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.233+0000 7f49f659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f612f0 0x56442732a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46770/0 (socket says 172.21.15.90:46770) 2024-04-25T11:43:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.233+0000 7f49f659c640 1 -- 172.21.15.90:0/2651472716 learned_addr learned my addr 172.21.15.90:0/2651472716 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.233+0000 7f49effff640 1 --2- 172.21.15.90:0/2651472716 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564426f694d0 0x56442732a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.234+0000 7f49f6d9d640 1 --2- 172.21.15.90:0/2651472716 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564426f8a1b0 0x564427329af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.234+0000 7f49ef7fe640 1 --2- 172.21.15.90:0/145610371 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f49e40afc50 0x7f49e40b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.234+0000 7f49f6d9d640 1 --2- 172.21.15.90:0/145610371 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f49e40afc50 0x7f49e40b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.234+0000 7f49ef7fe640 1 -- 172.21.15.90:0/145610371 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f49e80def30 con 0x7f49f8104c70 2024-04-25T11:43:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.234+0000 7f49f659c640 1 -- 172.21.15.90:0/2651472716 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564426f694d0 msgr2=0x56442732a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.234+0000 7f49f659c640 1 --2- 172.21.15.90:0/2651472716 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564426f694d0 0x56442732a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.234+0000 7f49f659c640 1 -- 172.21.15.90:0/2651472716 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564426f8a1b0 msgr2=0x564427329af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.234+0000 7f49f659c640 1 --2- 172.21.15.90:0/2651472716 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564426f8a1b0 0x564427329af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.234+0000 7f49f659c640 1 -- 172.21.15.90:0/2651472716 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644272570d0 con 0x564426f612f0 2024-04-25T11:43:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.234+0000 7f49effff640 1 --2- 172.21.15.90:0/2651472716 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564426f694d0 0x56442732a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:43:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.234+0000 7f49f6d9d640 1 --2- 172.21.15.90:0/2651472716 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564426f8a1b0 0x564427329af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.234+0000 7f49f659c640 1 --2- 172.21.15.90:0/2651472716 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f612f0 0x56442732a030 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f49e80a6d40 tx=0x7f49e8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40a76b5edc57adbb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.234+0000 7f49ecff9640 1 -- 172.21.15.90:0/2651472716 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e801d030 con 0x564426f612f0 2024-04-25T11:43:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.234+0000 7f49f6d9d640 1 --2- 172.21.15.90:0/145610371 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f49e40afc50 0x7f49e40b2160 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x564427326c60 tx=0x7f49f0004a40 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.234+0000 7f49ecff9640 1 -- 172.21.15.90:0/2651472716 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49e8005290 con 0x564426f612f0 2024-04-25T11:43:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.234+0000 7f49ecff9640 1 -- 172.21.15.90:0/2651472716 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e80a7040 con 0x564426f612f0 2024-04-25T11:43:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.235+0000 7f4a03dc7740 1 -- 172.21.15.90:0/2651472716 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f612f0 msgr2=0x56442732a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.235+0000 7f4a03dc7740 1 --2- 172.21.15.90:0/2651472716 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f612f0 0x56442732a030 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f49e80a6d40 tx=0x7f49e8000bc0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.235+0000 7f4a03dc7740 1 -- 172.21.15.90:0/2651472716 shutdown_connections 2024-04-25T11:43:31.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.235+0000 7f4a03dc7740 1 --2- 172.21.15.90:0/2651472716 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564426f694d0 0x56442732a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.235+0000 7f4a03dc7740 1 --2- 172.21.15.90:0/2651472716 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f612f0 0x56442732a030 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.235+0000 7f4a03dc7740 1 --2- 172.21.15.90:0/2651472716 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564426f8a1b0 0x564427329af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.235+0000 7f4a03dc7740 1 -- 172.21.15.90:0/2651472716 >> 172.21.15.90:0/2651472716 conn(0x564426f7e300 msgr2=0x5644273262a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:31.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.235+0000 7f4a03dc7740 1 -- 172.21.15.90:0/2651472716 shutdown_connections 2024-04-25T11:43:31.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.235+0000 7f4a03dc7740 1 -- 172.21.15.90:0/2651472716 wait complete. 2024-04-25T11:43:31.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.236+0000 7f4a03dc7740 1 -- start start 2024-04-25T11:43:31.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.236+0000 7f4a03dc7740 1 Processor -- start 2024-04-25T11:43:31.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.236+0000 7f4a03dc7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564426f612f0 0x564427335b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:31.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.236+0000 7f4a03dc7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f694d0 0x564427336040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:31.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.236+0000 7f4a03dc7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564426f8a1b0 0x564427333ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:31.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.236+0000 7f4a03dc7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564427336bc0 con 0x564426f694d0 2024-04-25T11:43:31.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.236+0000 7f4a03dc7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x564427217b60 con 0x564426f8a1b0 2024-04-25T11:43:31.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.236+0000 7f4a03dc7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x564427336d40 con 0x564426f612f0 2024-04-25T11:43:31.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.236+0000 7f4a03dc7740 10 client.0 _ll_register_callbacks cb 0x56442723c2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:31.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.236+0000 7f49effff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f694d0 0x564427336040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:31.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.236+0000 7f49effff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f694d0 0x564427336040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46778/0 (socket says 172.21.15.90:46778) 2024-04-25T11:43:31.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.236+0000 7f49effff640 1 -- 172.21.15.90:0/710576363 learned_addr learned my addr 172.21.15.90:0/710576363 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:31.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.236+0000 7f49f6d9d640 1 --2- 172.21.15.90:0/710576363 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564426f8a1b0 0x564427333ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:31.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.236+0000 7f49effff640 1 -- 172.21.15.90:0/710576363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564426f612f0 msgr2=0x564427335b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:31.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.236+0000 7f49f659c640 1 --2- 172.21.15.90:0/710576363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564426f612f0 0x564427335b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:31.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.236+0000 7f49effff640 1 --2- 172.21.15.90:0/710576363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564426f612f0 0x564427335b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.236+0000 7f49effff640 1 -- 172.21.15.90:0/710576363 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564426f8a1b0 msgr2=0x564427333ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:31.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.236+0000 7f49effff640 1 --2- 172.21.15.90:0/710576363 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564426f8a1b0 0x564427333ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.236+0000 7f49effff640 1 -- 172.21.15.90:0/710576363 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564426fa2210 con 0x564426f694d0 2024-04-25T11:43:31.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.236+0000 7f49f6d9d640 1 --2- 172.21.15.90:0/710576363 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564426f8a1b0 0x564427333ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:43:31.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.236+0000 7f49f659c640 1 --2- 172.21.15.90:0/710576363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564426f612f0 0x564427335b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:43:31.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.237+0000 7f49effff640 1 --2- 172.21.15.90:0/710576363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f694d0 0x564427336040 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f49e002a930 tx=0x7f49e0031810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=236c8eaaca65735 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:31.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.237+0000 7f49ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:31.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.237+0000 7f49ddffb640 1 -- 172.21.15.90:0/710576363 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e0035040 con 0x564426f694d0 2024-04-25T11:43:31.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.237+0000 7f49ddffb640 1 -- 172.21.15.90:0/710576363 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f49e0004240 con 0x564426f694d0 2024-04-25T11:43:31.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.237+0000 7f4a03dc7740 1 -- 172.21.15.90:0/710576363 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56442729e070 con 0x564426f694d0 2024-04-25T11:43:31.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.237+0000 7f49ddffb640 1 -- 172.21.15.90:0/710576363 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f49e0004440 con 0x564426f694d0 2024-04-25T11:43:31.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.237+0000 7f4a03dc7740 10 client.18198 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:31.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.237+0000 7f4a03dc7740 1 -- 172.21.15.90:0/710576363 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56442706e620 con 0x564426f694d0 2024-04-25T11:43:31.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.237+0000 7f49ddffb640 1 -- 172.21.15.90:0/710576363 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56442729e070 con 0x564426f694d0 2024-04-25T11:43:31.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.237+0000 7f49ddffb640 1 -- 172.21.15.90:0/710576363 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f49e0004c60 con 0x564426f694d0 2024-04-25T11:43:31.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.237+0000 7f4a03dc7740 10 client.18198 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:31.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.237+0000 7f4a03dc7740 10 client.18198 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:43:31.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.237+0000 7f4a03dc7740 20 client.18198 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:31.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.237+0000 7f4a03dc7740 1 --2- 172.21.15.90:0/710576363 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564426f48d80 0x5644273466c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:31.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.237+0000 7f4a03dc7740 4 client.18198 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:43:31.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.237+0000 7f4a03dc7740 1 -- 172.21.15.90:0/710576363 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564427297ee0 con 0x564426f48d80 2024-04-25T11:43:31.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.238+0000 7f49f659c640 1 --2- 172.21.15.90:0/710576363 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564426f48d80 0x5644273466c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:31.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.238+0000 7f49ddffb640 10 client.18198 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:31.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.238+0000 7f49f659c640 1 --2- 172.21.15.90:0/710576363 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564426f48d80 0x5644273466c0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ace4b1fa01311ecd server_cookie=894fd05b88b3e8b in_seq=0 out_seq=0 2024-04-25T11:43:31.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.239+0000 7f49ddffb640 1 -- 172.21.15.90:0/710576363 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564427297ee0 con 0x564426f48d80 2024-04-25T11:43:31.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.239+0000 7f49ddffb640 10 client.18198 handle_command_reply: tid=0 2024-04-25T11:43:31.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.239+0000 7f4a03dc7740 1 client.18198 shutdown 2024-04-25T11:43:31.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.239+0000 7f4a03dc7740 1 -- 172.21.15.90:0/710576363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f694d0 msgr2=0x564427336040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:31.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.239+0000 7f4a03dc7740 1 --2- 172.21.15.90:0/710576363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f694d0 0x564427336040 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f49e002a930 tx=0x7f49e0031810 comp rx=0 tx=0).stop 2024-04-25T11:43:31.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.239+0000 7f4a03dc7740 1 -- 172.21.15.90:0/710576363 shutdown_connections 2024-04-25T11:43:31.232 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.239+0000 7f4a03dc7740 1 --2- 172.21.15.90:0/710576363 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564426f48d80 0x5644273466c0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.232 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.239+0000 7f4a03dc7740 1 --2- 172.21.15.90:0/710576363 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564426f8a1b0 0x564427333ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.232 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.239+0000 7f49ddffb640 0 client.18198 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:31.232 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.239+0000 7f4a03dc7740 1 --2- 172.21.15.90:0/710576363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f694d0 0x564427336040 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.232 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.239+0000 7f4a03dc7740 1 --2- 172.21.15.90:0/710576363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564426f612f0 0x564427335b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.232 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.239+0000 7f4a03dc7740 1 -- 172.21.15.90:0/710576363 >> 172.21.15.90:0/710576363 conn(0x564426f7e300 msgr2=0x56442732b810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:31.232 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.239+0000 7f4a03dc7740 1 -- 172.21.15.90:0/710576363 shutdown_connections 2024-04-25T11:43:31.232 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.239+0000 7f4a03dc7740 1 -- 172.21.15.90:0/710576363 wait complete. 2024-04-25T11:43:31.232 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.239+0000 7f4a03dc7740 20 client.18198 trim_cache size 0 max 16384 2024-04-25T11:43:31.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.255+0000 7f4a03dc7740 1 Processor -- start 2024-04-25T11:43:31.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.255+0000 7f4a03dc7740 1 -- start start 2024-04-25T11:43:31.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.255+0000 7f4a03dc7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564426f612f0 0x564427333f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:31.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.255+0000 7f4a03dc7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564426f694d0 0x56442732a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:31.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.255+0000 7f4a03dc7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f8a1b0 0x56442732d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:31.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.255+0000 7f4a03dc7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f49f81afd60 con 0x564426f8a1b0 2024-04-25T11:43:31.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.255+0000 7f4a03dc7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f49f81b18a0 con 0x564426f694d0 2024-04-25T11:43:31.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.255+0000 7f4a03dc7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f49f81b0430 con 0x564426f612f0 2024-04-25T11:43:31.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.255+0000 7f49f6d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f8a1b0 0x56442732d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:31.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.255+0000 7f49f6d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f8a1b0 0x56442732d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46780/0 (socket says 172.21.15.90:46780) 2024-04-25T11:43:31.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.255+0000 7f49f6d9d640 1 -- 172.21.15.90:0/2475483152 learned_addr learned my addr 172.21.15.90:0/2475483152 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:31.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.255+0000 7f49effff640 1 --2- 172.21.15.90:0/2475483152 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564426f694d0 0x56442732a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:31.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.255+0000 7f49f659c640 1 --2- 172.21.15.90:0/2475483152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564426f612f0 0x564427333f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:31.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.255+0000 7f49f6d9d640 1 -- 172.21.15.90:0/2475483152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564426f612f0 msgr2=0x564427333f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:31.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.255+0000 7f49f6d9d640 1 --2- 172.21.15.90:0/2475483152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564426f612f0 0x564427333f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.255+0000 7f49f6d9d640 1 -- 172.21.15.90:0/2475483152 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564426f694d0 msgr2=0x56442732a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:31.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.255+0000 7f49f6d9d640 1 --2- 172.21.15.90:0/2475483152 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564426f694d0 0x56442732a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.255+0000 7f49f6d9d640 1 -- 172.21.15.90:0/2475483152 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644273344d0 con 0x564426f8a1b0 2024-04-25T11:43:31.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.255+0000 7f49f6d9d640 1 --2- 172.21.15.90:0/2475483152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f8a1b0 0x56442732d7a0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f49f000c640 tx=0x7f49f000e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=240cc782f5ca5d25 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:31.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.256+0000 7f49dffff640 1 -- 172.21.15.90:0/2475483152 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f0021070 con 0x564426f8a1b0 2024-04-25T11:43:31.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.256+0000 7f49dffff640 1 -- 172.21.15.90:0/2475483152 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49f001ddd0 con 0x564426f8a1b0 2024-04-25T11:43:31.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.256+0000 7f49dffff640 1 -- 172.21.15.90:0/2475483152 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f000ae20 con 0x564426f8a1b0 2024-04-25T11:43:31.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.256+0000 7f4a03dc7740 1 -- 172.21.15.90:0/2475483152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f8a1b0 msgr2=0x56442732d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:31.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.256+0000 7f4a03dc7740 1 --2- 172.21.15.90:0/2475483152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f8a1b0 0x56442732d7a0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f49f000c640 tx=0x7f49f000e040 comp rx=0 tx=0).stop 2024-04-25T11:43:31.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.256+0000 7f4a03dc7740 1 -- 172.21.15.90:0/2475483152 shutdown_connections 2024-04-25T11:43:31.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.256+0000 7f4a03dc7740 1 --2- 172.21.15.90:0/2475483152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f8a1b0 0x56442732d7a0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.256+0000 7f4a03dc7740 1 --2- 172.21.15.90:0/2475483152 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564426f694d0 0x56442732a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.256+0000 7f4a03dc7740 1 --2- 172.21.15.90:0/2475483152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564426f612f0 0x564427333f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.256+0000 7f4a03dc7740 1 -- 172.21.15.90:0/2475483152 >> 172.21.15.90:0/2475483152 conn(0x564426f7e300 msgr2=0x564427333880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:31.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.256+0000 7f4a03dc7740 1 -- 172.21.15.90:0/2475483152 shutdown_connections 2024-04-25T11:43:31.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.256+0000 7f4a03dc7740 1 -- 172.21.15.90:0/2475483152 wait complete. 2024-04-25T11:43:31.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.257+0000 7f4a03dc7740 1 -- start start 2024-04-25T11:43:31.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.257+0000 7f4a03dc7740 1 Processor -- start 2024-04-25T11:43:31.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.257+0000 7f4a03dc7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564426f612f0 0x564427339bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:31.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.257+0000 7f4a03dc7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f694d0 0x56442733a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:31.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.257+0000 7f4a03dc7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564426f8a1b0 0x56442733a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:31.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.257+0000 7f4a03dc7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f49f81b11d0 con 0x564426f694d0 2024-04-25T11:43:31.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.257+0000 7f4a03dc7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f49f81b0b00 con 0x564426f8a1b0 2024-04-25T11:43:31.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.257+0000 7f4a03dc7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f49f81add60 con 0x564426f612f0 2024-04-25T11:43:31.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.257+0000 7f4a03dc7740 10 client.0 _ll_register_callbacks cb 0x7f49e0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:31.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.257+0000 7f49effff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f694d0 0x56442733a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:31.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.257+0000 7f49effff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f694d0 0x56442733a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46782/0 (socket says 172.21.15.90:46782) 2024-04-25T11:43:31.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.257+0000 7f49effff640 1 -- 172.21.15.90:0/2312543338 learned_addr learned my addr 172.21.15.90:0/2312543338 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:31.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.257+0000 7f49f6d9d640 1 --2- 172.21.15.90:0/2312543338 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564426f8a1b0 0x56442733a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:31.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.257+0000 7f49effff640 1 -- 172.21.15.90:0/2312543338 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564426f612f0 msgr2=0x564427339bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:31.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.257+0000 7f49effff640 1 --2- 172.21.15.90:0/2312543338 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564426f612f0 0x564427339bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.257+0000 7f49effff640 1 -- 172.21.15.90:0/2312543338 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564426f8a1b0 msgr2=0x56442733a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:31.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.257+0000 7f49effff640 1 --2- 172.21.15.90:0/2312543338 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564426f8a1b0 0x56442733a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.257+0000 7f49effff640 1 -- 172.21.15.90:0/2312543338 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644273ada40 con 0x564426f694d0 2024-04-25T11:43:31.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.257+0000 7f49f6d9d640 1 --2- 172.21.15.90:0/2312543338 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564426f8a1b0 0x56442733a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:31.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.258+0000 7f49effff640 1 --2- 172.21.15.90:0/2312543338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f694d0 0x56442733a100 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x564427320590 tx=0x7f49e0031570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dcbbe4f3f48c63b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:31.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.258+0000 7f49ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:31.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.258+0000 7f49ddffb640 1 -- 172.21.15.90:0/2312543338 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e0035040 con 0x564426f694d0 2024-04-25T11:43:31.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.258+0000 7f4a03dc7740 1 -- 172.21.15.90:0/2312543338 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49f001ddd0 con 0x564426f694d0 2024-04-25T11:43:31.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.258+0000 7f49ddffb640 1 -- 172.21.15.90:0/2312543338 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f49e00040d0 con 0x564426f694d0 2024-04-25T11:43:31.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.258+0000 7f49ddffb640 1 -- 172.21.15.90:0/2312543338 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f49e0055480 con 0x564426f694d0 2024-04-25T11:43:31.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.258+0000 7f4a03dc7740 10 client.18204 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:31.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.258+0000 7f4a03dc7740 1 -- 172.21.15.90:0/2312543338 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5644273aebe0 con 0x564426f694d0 2024-04-25T11:43:31.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.258+0000 7f49ddffb640 1 -- 172.21.15.90:0/2312543338 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49f001ddd0 con 0x564426f694d0 2024-04-25T11:43:31.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.258+0000 7f49ddffb640 1 -- 172.21.15.90:0/2312543338 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f49e005b030 con 0x564426f694d0 2024-04-25T11:43:31.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.258+0000 7f4a03dc7740 10 client.18204 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:31.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.258+0000 7f4a03dc7740 10 client.18204 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:43:31.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.258+0000 7f4a03dc7740 20 client.18204 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:31.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.258+0000 7f4a03dc7740 1 --2- 172.21.15.90:0/2312543338 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564426f1e1c0 0x5644273b1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:31.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.258+0000 7f4a03dc7740 4 client.18204 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:43:31.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.258+0000 7f4a03dc7740 1 -- 172.21.15.90:0/2312543338 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5644272135f0 con 0x564426f1e1c0 2024-04-25T11:43:31.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.259+0000 7f49f659c640 1 --2- 172.21.15.90:0/2312543338 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564426f1e1c0 0x5644273b1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:31.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.259+0000 7f49f659c640 1 --2- 172.21.15.90:0/2312543338 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564426f1e1c0 0x5644273b1370 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74a72638c0e1f761 server_cookie=e74042c6bac2a038 in_seq=0 out_seq=0 2024-04-25T11:43:31.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.259+0000 7f49ddffb640 10 client.18204 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:31.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.266+0000 7f49ddffb640 1 -- 172.21.15.90:0/2312543338 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5644272135f0 con 0x564426f1e1c0 2024-04-25T11:43:31.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.266+0000 7f49ddffb640 10 client.18204 handle_command_reply: tid=0 2024-04-25T11:43:31.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.266+0000 7f4a03dc7740 1 client.18204 shutdown 2024-04-25T11:43:31.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.266+0000 7f4a03dc7740 1 -- 172.21.15.90:0/2312543338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f694d0 msgr2=0x56442733a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:31.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.266+0000 7f4a03dc7740 1 --2- 172.21.15.90:0/2312543338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f694d0 0x56442733a100 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x564427320590 tx=0x7f49e0031570 comp rx=0 tx=0).stop 2024-04-25T11:43:31.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.266+0000 7f4a03dc7740 1 -- 172.21.15.90:0/2312543338 shutdown_connections 2024-04-25T11:43:31.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.266+0000 7f4a03dc7740 1 --2- 172.21.15.90:0/2312543338 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564426f1e1c0 0x5644273b1370 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.266+0000 7f4a03dc7740 1 --2- 172.21.15.90:0/2312543338 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564426f8a1b0 0x56442733a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.266+0000 7f4a03dc7740 1 --2- 172.21.15.90:0/2312543338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564426f694d0 0x56442733a100 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.266+0000 7f49ddffb640 0 client.18204 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:31.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.266+0000 7f4a03dc7740 1 --2- 172.21.15.90:0/2312543338 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564426f612f0 0x564427339bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.266+0000 7f4a03dc7740 1 -- 172.21.15.90:0/2312543338 >> 172.21.15.90:0/2312543338 conn(0x564426f7e300 msgr2=0x564427329970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:31.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.266+0000 7f4a03dc7740 1 -- 172.21.15.90:0/2312543338 shutdown_connections 2024-04-25T11:43:31.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.266+0000 7f4a03dc7740 1 -- 172.21.15.90:0/2312543338 wait complete. 2024-04-25T11:43:31.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.266+0000 7f4a03dc7740 20 client.18204 trim_cache size 0 max 16384 2024-04-25T11:43:31.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:43:31.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:43:31.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:43:31.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:43:31.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.267+0000 7f49fcf14640 1 -- 172.21.15.90:0/145610371 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f49e40afc50 msgr2=0x7f49e40b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:31.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.267+0000 7f49fcf14640 1 --2- 172.21.15.90:0/145610371 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f49e40afc50 0x7f49e40b2160 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x564427326c60 tx=0x7f49f0004a40 comp rx=0 tx=0).stop 2024-04-25T11:43:31.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.267+0000 7f49fcf14640 1 -- 172.21.15.90:0/145610371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f8104c70 msgr2=0x7f49f819a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:31.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.267+0000 7f49fcf14640 1 --2- 172.21.15.90:0/145610371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f8104c70 0x7f49f819a7e0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f49e800c890 tx=0x7f49e800cd50 comp rx=0 tx=0).stop 2024-04-25T11:43:31.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.267+0000 7f49fcf14640 1 -- 172.21.15.90:0/145610371 shutdown_connections 2024-04-25T11:43:31.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.267+0000 7f49fcf14640 1 --2- 172.21.15.90:0/145610371 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f49e40afc50 0x7f49e40b2160 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.267+0000 7f49fcf14640 1 --2- 172.21.15.90:0/145610371 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f49f810e0c0 0x7f49f819f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.267+0000 7f49fcf14640 1 --2- 172.21.15.90:0/145610371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49f810b6d0 0x7f49f819ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.267+0000 7f49fcf14640 1 --2- 172.21.15.90:0/145610371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f8104c70 0x7f49f819a7e0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:31.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.267+0000 7f49fcf14640 1 -- 172.21.15.90:0/145610371 >> 172.21.15.90:0/145610371 conn(0x7f49f80781b0 msgr2=0x7f49f8101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:31.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.267+0000 7f49fcf14640 1 -- 172.21.15.90:0/145610371 shutdown_connections 2024-04-25T11:43:31.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:31.267+0000 7f49fcf14640 1 -- 172.21.15.90:0/145610371 wait complete. 2024-04-25T11:43:32.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:32 smithi156 ceph-mon[33045]: pgmap v260: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T11:43:32.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:32 smithi090 ceph-mon[29575]: pgmap v260: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T11:43:32.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:32 smithi138 ceph-mon[32328]: pgmap v260: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T11:43:34.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:34 smithi156 ceph-mon[33045]: pgmap v261: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T11:43:34.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:34 smithi090 ceph-mon[29575]: pgmap v261: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T11:43:34.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:34 smithi138 ceph-mon[32328]: pgmap v261: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T11:43:36.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:36 smithi156 ceph-mon[33045]: pgmap v262: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-04-25T11:43:36.352 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:36 smithi138 ceph-mon[32328]: pgmap v262: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-04-25T11:43:36.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:36 smithi090 ceph-mon[29575]: pgmap v262: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-04-25T11:43:38.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:38 smithi156 ceph-mon[33045]: pgmap v263: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-04-25T11:43:38.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:38 smithi090 ceph-mon[29575]: pgmap v263: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-04-25T11:43:38.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:38 smithi138 ceph-mon[32328]: pgmap v263: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-04-25T11:43:40.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:40 smithi156 ceph-mon[33045]: pgmap v264: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:43:40.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:40 smithi090 ceph-mon[29575]: pgmap v264: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:43:40.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:40 smithi138 ceph-mon[32328]: pgmap v264: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:43:42.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:42 smithi156 ceph-mon[33045]: pgmap v265: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-04-25T11:43:42.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:42 smithi090 ceph-mon[29575]: pgmap v265: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-04-25T11:43:42.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:42 smithi138 ceph-mon[32328]: pgmap v265: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-04-25T11:43:42.810 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:43:42.810 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:43:42.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.949+0000 7f48c4106640 1 -- 172.21.15.90:0/1511922989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48bc109b10 msgr2=0x7f48bc1036b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:42.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.949+0000 7f48c4106640 1 --2- 172.21.15.90:0/1511922989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48bc109b10 0x7f48bc1036b0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f48b8009900 tx=0x7f48b802a410 comp rx=0 tx=0).stop 2024-04-25T11:43:42.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.949+0000 7f48c4106640 1 -- 172.21.15.90:0/1511922989 shutdown_connections 2024-04-25T11:43:42.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.949+0000 7f48c4106640 1 --2- 172.21.15.90:0/1511922989 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f48bc104260 0x7f48bc110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.949+0000 7f48c4106640 1 --2- 172.21.15.90:0/1511922989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48bc109fb0 0x7f48bc103bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.949+0000 7f48c4106640 1 --2- 172.21.15.90:0/1511922989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48bc109b10 0x7f48bc1036b0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.949+0000 7f48c4106640 1 -- 172.21.15.90:0/1511922989 >> 172.21.15.90:0/1511922989 conn(0x7f48bc0ff210 msgr2=0x7f48bc101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:42.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.950+0000 7f48c4106640 1 -- 172.21.15.90:0/1511922989 shutdown_connections 2024-04-25T11:43:42.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.950+0000 7f48c4106640 1 -- 172.21.15.90:0/1511922989 wait complete. 2024-04-25T11:43:42.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.950+0000 7f48c4106640 1 Processor -- start 2024-04-25T11:43:42.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.950+0000 7f48c4106640 1 -- start start 2024-04-25T11:43:42.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.950+0000 7f48c4106640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48bc104260 0x7f48bc19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:42.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.950+0000 7f48c4106640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f48bc109b10 0x7f48bc19ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.950+0000 7f48c4106640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48bc109fb0 0x7f48bc19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.950+0000 7f48c4106640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc113d00 con 0x7f48bc104260 2024-04-25T11:43:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.950+0000 7f48c4106640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc113b80 con 0x7f48bc109b10 2024-04-25T11:43:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.950+0000 7f48c4106640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc113e80 con 0x7f48bc109fb0 2024-04-25T11:43:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.951+0000 7f48c1ea2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48bc104260 0x7f48bc19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.951+0000 7f48c1ea2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48bc104260 0x7f48bc19a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41836/0 (socket says 172.21.15.90:41836) 2024-04-25T11:43:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.951+0000 7f48c1ea2640 1 -- 172.21.15.90:0/3610713414 learned_addr learned my addr 172.21.15.90:0/3610713414 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.951+0000 7f48c16a1640 1 --2- 172.21.15.90:0/3610713414 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f48bc109b10 0x7f48bc19ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.951+0000 7f48c26a3640 1 --2- 172.21.15.90:0/3610713414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48bc109fb0 0x7f48bc19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.951+0000 7f48c1ea2640 1 -- 172.21.15.90:0/3610713414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48bc109fb0 msgr2=0x7f48bc19efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.951+0000 7f48c1ea2640 1 --2- 172.21.15.90:0/3610713414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48bc109fb0 0x7f48bc19efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.951+0000 7f48c1ea2640 1 -- 172.21.15.90:0/3610713414 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f48bc109b10 msgr2=0x7f48bc19ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.951+0000 7f48c1ea2640 1 --2- 172.21.15.90:0/3610713414 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f48bc109b10 0x7f48bc19ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.951+0000 7f48c1ea2640 1 -- 172.21.15.90:0/3610713414 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48bc19f5e0 con 0x7f48bc104260 2024-04-25T11:43:42.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.951+0000 7f48c1ea2640 1 --2- 172.21.15.90:0/3610713414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48bc104260 0x7f48bc19a780 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f48b8006010 tx=0x7f48b8004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:42.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.951+0000 7f48b2ffd640 1 -- 172.21.15.90:0/3610713414 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b803f030 con 0x7f48bc104260 2024-04-25T11:43:42.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.951+0000 7f48c4106640 1 -- 172.21.15.90:0/3610713414 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48bc19f870 con 0x7f48bc104260 2024-04-25T11:43:42.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.951+0000 7f48b2ffd640 1 -- 172.21.15.90:0/3610713414 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48b8003aa0 con 0x7f48bc104260 2024-04-25T11:43:42.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.951+0000 7f48b2ffd640 1 -- 172.21.15.90:0/3610713414 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b8035070 con 0x7f48bc104260 2024-04-25T11:43:42.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.951+0000 7f48c4106640 1 -- 172.21.15.90:0/3610713414 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48bc19fdb0 con 0x7f48bc104260 2024-04-25T11:43:42.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.954+0000 7f48b2ffd640 1 -- 172.21.15.90:0/3610713414 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f48b8042030 con 0x7f48bc104260 2024-04-25T11:43:42.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.954+0000 7f48cafba740 1 Processor -- start 2024-04-25T11:43:42.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.954+0000 7f48cafba740 1 -- start start 2024-04-25T11:43:42.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.954+0000 7f48cafba740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f4f536390 0x564f4f8d6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:42.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.954+0000 7f48cafba740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f50d4d0 0x564f4f8d6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:42.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.954+0000 7f48cafba740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564f4f5156b0 0x564f4f8e38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:42.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.954+0000 7f48cafba740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564f4f83cd80 con 0x564f4f50d4d0 2024-04-25T11:43:42.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.954+0000 7f48cafba740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x564f4f7f68b0 con 0x564f4f5156b0 2024-04-25T11:43:42.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.954+0000 7f48cafba740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x564f4f838d40 con 0x564f4f536390 2024-04-25T11:43:42.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.954+0000 7f48b2ffd640 1 --2- 172.21.15.90:0/3610713414 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f48a80afc60 0x7f48a80b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:42.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.955+0000 7f48c1ea2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f50d4d0 0x564f4f8d6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:42.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.955+0000 7f48c1ea2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f50d4d0 0x564f4f8d6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41852/0 (socket says 172.21.15.90:41852) 2024-04-25T11:43:42.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.955+0000 7f48c1ea2640 1 -- 172.21.15.90:0/4212393319 learned_addr learned my addr 172.21.15.90:0/4212393319 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:42.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.955+0000 7f48c16a1640 1 --2- 172.21.15.90:0/4212393319 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564f4f5156b0 0x564f4f8e38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:42.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.955+0000 7f48c26a3640 1 --2- 172.21.15.90:0/3610713414 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f48a80afc60 0x7f48a80b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:42.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.955+0000 7f48c26a3640 1 --2- 172.21.15.90:0/4212393319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f4f536390 0x564f4f8d6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:42.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.955+0000 7f48b2ffd640 1 -- 172.21.15.90:0/3610713414 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f48b80ff280 con 0x7f48bc104260 2024-04-25T11:43:42.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.955+0000 7f48c1ea2640 1 -- 172.21.15.90:0/4212393319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f4f536390 msgr2=0x564f4f8d6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:42.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.955+0000 7f48c1ea2640 1 --2- 172.21.15.90:0/4212393319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f4f536390 0x564f4f8d6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.955+0000 7f48c1ea2640 1 -- 172.21.15.90:0/4212393319 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564f4f5156b0 msgr2=0x564f4f8e38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:42.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.955+0000 7f48c1ea2640 1 --2- 172.21.15.90:0/4212393319 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564f4f5156b0 0x564f4f8e38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.955+0000 7f48c1ea2640 1 -- 172.21.15.90:0/4212393319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f4f7c7a40 con 0x564f4f50d4d0 2024-04-25T11:43:42.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.955+0000 7f48c16a1640 1 --2- 172.21.15.90:0/4212393319 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564f4f5156b0 0x564f4f8e38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:43:42.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.955+0000 7f48c26a3640 1 --2- 172.21.15.90:0/3610713414 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f48a80afc60 0x7f48a80b2170 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x564f4f8d3bd0 tx=0x7f48b4009ac0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:42.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.955+0000 7f48c1ea2640 1 --2- 172.21.15.90:0/4212393319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f50d4d0 0x564f4f8d6fa0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f48b8031440 tx=0x7f48b8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f40fab969159697c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:42.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.955+0000 7f48a7fff640 1 -- 172.21.15.90:0/4212393319 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b8049070 con 0x564f4f50d4d0 2024-04-25T11:43:42.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.955+0000 7f48a7fff640 1 -- 172.21.15.90:0/4212393319 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48b80d3f70 con 0x564f4f50d4d0 2024-04-25T11:43:42.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.955+0000 7f48a7fff640 1 -- 172.21.15.90:0/4212393319 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b80cb740 con 0x564f4f50d4d0 2024-04-25T11:43:42.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.956+0000 7f48cafba740 1 -- 172.21.15.90:0/4212393319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f50d4d0 msgr2=0x564f4f8d6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:42.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.956+0000 7f48cafba740 1 --2- 172.21.15.90:0/4212393319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f50d4d0 0x564f4f8d6fa0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f48b8031440 tx=0x7f48b8009260 comp rx=0 tx=0).stop 2024-04-25T11:43:42.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.956+0000 7f48cafba740 1 -- 172.21.15.90:0/4212393319 shutdown_connections 2024-04-25T11:43:42.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.956+0000 7f48cafba740 1 --2- 172.21.15.90:0/4212393319 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564f4f5156b0 0x564f4f8e38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.956+0000 7f48cafba740 1 --2- 172.21.15.90:0/4212393319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f50d4d0 0x564f4f8d6fa0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.956+0000 7f48cafba740 1 --2- 172.21.15.90:0/4212393319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f4f536390 0x564f4f8d6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.956+0000 7f48cafba740 1 -- 172.21.15.90:0/4212393319 >> 172.21.15.90:0/4212393319 conn(0x564f4f52a4e0 msgr2=0x564f4f8d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:42.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.956+0000 7f48cafba740 1 -- 172.21.15.90:0/4212393319 shutdown_connections 2024-04-25T11:43:42.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.956+0000 7f48cafba740 1 -- 172.21.15.90:0/4212393319 wait complete. 2024-04-25T11:43:42.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.957+0000 7f48cafba740 1 -- start start 2024-04-25T11:43:42.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.957+0000 7f48cafba740 1 Processor -- start 2024-04-25T11:43:42.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.957+0000 7f48cafba740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f4f50d4d0 0x564f4f8e1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:42.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.957+0000 7f48cafba740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f5156b0 0x564f4f8e1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:42.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.957+0000 7f48cafba740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564f4f536390 0x564f4f8e7170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:42.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.957+0000 7f48cafba740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564f4f83cd80 con 0x564f4f5156b0 2024-04-25T11:43:42.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.957+0000 7f48cafba740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x564f4f7c3d10 con 0x564f4f536390 2024-04-25T11:43:42.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.957+0000 7f48cafba740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x564f4f8e3e30 con 0x564f4f50d4d0 2024-04-25T11:43:42.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.957+0000 7f48cafba740 10 client.0 _ll_register_callbacks cb 0x564f4f7f0210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.957+0000 7f48c16a1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f5156b0 0x564f4f8e1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.957+0000 7f48c16a1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f5156b0 0x564f4f8e1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41854/0 (socket says 172.21.15.90:41854) 2024-04-25T11:43:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.957+0000 7f48c16a1640 1 -- 172.21.15.90:0/1607143295 learned_addr learned my addr 172.21.15.90:0/1607143295 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.958+0000 7f48c26a3640 1 --2- 172.21.15.90:0/1607143295 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564f4f536390 0x564f4f8e7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.958+0000 7f48c1ea2640 1 --2- 172.21.15.90:0/1607143295 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f4f50d4d0 0x564f4f8e1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.958+0000 7f48c16a1640 1 -- 172.21.15.90:0/1607143295 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f4f50d4d0 msgr2=0x564f4f8e1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.958+0000 7f48c16a1640 1 --2- 172.21.15.90:0/1607143295 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f4f50d4d0 0x564f4f8e1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.958+0000 7f48c16a1640 1 -- 172.21.15.90:0/1607143295 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564f4f536390 msgr2=0x564f4f8e7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.958+0000 7f48c16a1640 1 --2- 172.21.15.90:0/1607143295 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564f4f536390 0x564f4f8e7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.958+0000 7f48c16a1640 1 -- 172.21.15.90:0/1607143295 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f4f6187d0 con 0x564f4f5156b0 2024-04-25T11:43:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.958+0000 7f48c1ea2640 1 --2- 172.21.15.90:0/1607143295 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f4f50d4d0 0x564f4f8e1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:43:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.958+0000 7f48c16a1640 1 --2- 172.21.15.90:0/1607143295 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f5156b0 0x564f4f8e1b60 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f48ac00be00 tx=0x7f48ac012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b72fe878f7d1e712 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.958+0000 7f48a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.958+0000 7f48a57fa640 1 -- 172.21.15.90:0/1607143295 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48ac0108a0 con 0x564f4f5156b0 2024-04-25T11:43:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.958+0000 7f48a57fa640 1 -- 172.21.15.90:0/1607143295 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f48ac010a40 con 0x564f4f5156b0 2024-04-25T11:43:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.958+0000 7f48cafba740 1 -- 172.21.15.90:0/1607143295 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48b80d3f70 con 0x564f4f5156b0 2024-04-25T11:43:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.958+0000 7f48a57fa640 1 -- 172.21.15.90:0/1607143295 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f48ac010c40 con 0x564f4f5156b0 2024-04-25T11:43:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.958+0000 7f48cafba740 10 client.18216 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.958+0000 7f48cafba740 1 -- 172.21.15.90:0/1607143295 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564f4f54e3f0 con 0x564f4f5156b0 2024-04-25T11:43:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.958+0000 7f48a57fa640 1 -- 172.21.15.90:0/1607143295 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48b80d3f70 con 0x564f4f5156b0 2024-04-25T11:43:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.959+0000 7f48a57fa640 1 -- 172.21.15.90:0/1607143295 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f48ac019070 con 0x564f4f5156b0 2024-04-25T11:43:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.959+0000 7f48cafba740 10 client.18216 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.959+0000 7f48cafba740 10 client.18216 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:43:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.959+0000 7f48cafba740 20 client.18216 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.959+0000 7f48cafba740 1 --2- 172.21.15.90:0/1607143295 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564f4f4f4f60 0x564f4f8f34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.959+0000 7f48cafba740 4 client.18216 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:43:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.959+0000 7f48cafba740 1 -- 172.21.15.90:0/1607143295 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f4f84a990 con 0x564f4f4f4f60 2024-04-25T11:43:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.959+0000 7f48c1ea2640 1 --2- 172.21.15.90:0/1607143295 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564f4f4f4f60 0x564f4f8f34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:42.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.959+0000 7f48a57fa640 10 client.18216 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:42.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.959+0000 7f48c1ea2640 1 --2- 172.21.15.90:0/1607143295 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564f4f4f4f60 0x564f4f8f34a0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d833fe811a68dc3 server_cookie=52a1b5352279dbad in_seq=0 out_seq=0 2024-04-25T11:43:42.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.960+0000 7f48a57fa640 1 -- 172.21.15.90:0/1607143295 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564f4f84a990 con 0x564f4f4f4f60 2024-04-25T11:43:42.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.960+0000 7f48a57fa640 10 client.18216 handle_command_reply: tid=0 2024-04-25T11:43:42.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.960+0000 7f48cafba740 1 client.18216 shutdown 2024-04-25T11:43:42.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.960+0000 7f48cafba740 1 -- 172.21.15.90:0/1607143295 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f5156b0 msgr2=0x564f4f8e1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:42.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.960+0000 7f48cafba740 1 --2- 172.21.15.90:0/1607143295 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f5156b0 0x564f4f8e1b60 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f48ac00be00 tx=0x7f48ac012820 comp rx=0 tx=0).stop 2024-04-25T11:43:42.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.960+0000 7f48cafba740 1 -- 172.21.15.90:0/1607143295 shutdown_connections 2024-04-25T11:43:42.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.960+0000 7f48cafba740 1 --2- 172.21.15.90:0/1607143295 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564f4f4f4f60 0x564f4f8f34a0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.960+0000 7f48cafba740 1 --2- 172.21.15.90:0/1607143295 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564f4f536390 0x564f4f8e7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.960+0000 7f48cafba740 1 --2- 172.21.15.90:0/1607143295 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f5156b0 0x564f4f8e1b60 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.960+0000 7f48cafba740 1 --2- 172.21.15.90:0/1607143295 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f4f50d4d0 0x564f4f8e1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.960+0000 7f48a57fa640 0 client.18216 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:42.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.960+0000 7f48cafba740 1 -- 172.21.15.90:0/1607143295 >> 172.21.15.90:0/1607143295 conn(0x564f4f52a4e0 msgr2=0x564f4f8d8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:42.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.960+0000 7f48cafba740 1 -- 172.21.15.90:0/1607143295 shutdown_connections 2024-04-25T11:43:42.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.960+0000 7f48cafba740 1 -- 172.21.15.90:0/1607143295 wait complete. 2024-04-25T11:43:42.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.961+0000 7f48cafba740 20 client.18216 trim_cache size 0 max 16384 2024-04-25T11:43:42.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.978+0000 7f48cafba740 1 Processor -- start 2024-04-25T11:43:42.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.978+0000 7f48cafba740 1 -- start start 2024-04-25T11:43:42.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.978+0000 7f48cafba740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564f4f50d4d0 0x564f4f8d6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:42.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.978+0000 7f48cafba740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f4f5156b0 0x564f4f8e70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:42.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.978+0000 7f48cafba740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f536390 0x564f4f8e7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:42.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.978+0000 7f48cafba740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc1afd40 con 0x564f4f536390 2024-04-25T11:43:42.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.978+0000 7f48cafba740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc1b1880 con 0x564f4f50d4d0 2024-04-25T11:43:42.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.978+0000 7f48cafba740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc1b0410 con 0x564f4f5156b0 2024-04-25T11:43:42.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.978+0000 7f48c26a3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f536390 0x564f4f8e7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:42.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.978+0000 7f48c26a3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f536390 0x564f4f8e7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41858/0 (socket says 172.21.15.90:41858) 2024-04-25T11:43:42.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.978+0000 7f48c26a3640 1 -- 172.21.15.90:0/4220334447 learned_addr learned my addr 172.21.15.90:0/4220334447 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:42.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.979+0000 7f48c1ea2640 1 --2- 172.21.15.90:0/4220334447 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564f4f50d4d0 0x564f4f8d6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:42.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.979+0000 7f48c26a3640 1 -- 172.21.15.90:0/4220334447 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f4f5156b0 msgr2=0x564f4f8e70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:42.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.979+0000 7f48c16a1640 1 --2- 172.21.15.90:0/4220334447 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f4f5156b0 0x564f4f8e70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:42.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.979+0000 7f48c26a3640 1 --2- 172.21.15.90:0/4220334447 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f4f5156b0 0x564f4f8e70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.979+0000 7f48c26a3640 1 -- 172.21.15.90:0/4220334447 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564f4f50d4d0 msgr2=0x564f4f8d6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:42.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.979+0000 7f48c26a3640 1 --2- 172.21.15.90:0/4220334447 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564f4f50d4d0 0x564f4f8d6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.979+0000 7f48c26a3640 1 -- 172.21.15.90:0/4220334447 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f4f50e0f0 con 0x564f4f536390 2024-04-25T11:43:42.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.979+0000 7f48c16a1640 1 --2- 172.21.15.90:0/4220334447 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f4f5156b0 0x564f4f8e70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:43:42.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.979+0000 7f48c26a3640 1 --2- 172.21.15.90:0/4220334447 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f536390 0x564f4f8e7630 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f48b40143c0 tx=0x7f48b4009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc5fb64bedf76afa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:42.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.979+0000 7f48a77fe640 1 -- 172.21.15.90:0/4220334447 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b402b670 con 0x564f4f536390 2024-04-25T11:43:42.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.979+0000 7f48a77fe640 1 -- 172.21.15.90:0/4220334447 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48b4026070 con 0x564f4f536390 2024-04-25T11:43:42.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.979+0000 7f48a77fe640 1 -- 172.21.15.90:0/4220334447 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b4012a80 con 0x564f4f536390 2024-04-25T11:43:42.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.980+0000 7f48cafba740 1 -- 172.21.15.90:0/4220334447 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f536390 msgr2=0x564f4f8e7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:42.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.980+0000 7f48cafba740 1 --2- 172.21.15.90:0/4220334447 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f536390 0x564f4f8e7630 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f48b40143c0 tx=0x7f48b4009500 comp rx=0 tx=0).stop 2024-04-25T11:43:42.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.980+0000 7f48cafba740 1 -- 172.21.15.90:0/4220334447 shutdown_connections 2024-04-25T11:43:42.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.980+0000 7f48cafba740 1 --2- 172.21.15.90:0/4220334447 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f536390 0x564f4f8e7630 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.980+0000 7f48cafba740 1 --2- 172.21.15.90:0/4220334447 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f4f5156b0 0x564f4f8e70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.980+0000 7f48cafba740 1 --2- 172.21.15.90:0/4220334447 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564f4f50d4d0 0x564f4f8d6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.980+0000 7f48cafba740 1 -- 172.21.15.90:0/4220334447 >> 172.21.15.90:0/4220334447 conn(0x564f4f52a4e0 msgr2=0x564f4f8d67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:42.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.980+0000 7f48cafba740 1 -- 172.21.15.90:0/4220334447 shutdown_connections 2024-04-25T11:43:42.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.980+0000 7f48cafba740 1 -- 172.21.15.90:0/4220334447 wait complete. 2024-04-25T11:43:42.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.980+0000 7f48cafba740 1 -- start start 2024-04-25T11:43:42.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.980+0000 7f48cafba740 1 Processor -- start 2024-04-25T11:43:42.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.981+0000 7f48cafba740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564f4f50d4d0 0x564f4f8d2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:42.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.981+0000 7f48cafba740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f4f5156b0 0x564f4f8d2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:42.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.981+0000 7f48cafba740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f536390 0x564f4f956d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:42.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.981+0000 7f48cafba740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc1b11b0 con 0x564f4f536390 2024-04-25T11:43:42.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.981+0000 7f48cafba740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc1b0ae0 con 0x564f4f50d4d0 2024-04-25T11:43:42.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.981+0000 7f48cafba740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f48bc1add40 con 0x564f4f5156b0 2024-04-25T11:43:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.981+0000 7f48cafba740 10 client.0 _ll_register_callbacks cb 0x7f48ac00b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.981+0000 7f48c26a3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f536390 0x564f4f956d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.981+0000 7f48c26a3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f536390 0x564f4f956d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41868/0 (socket says 172.21.15.90:41868) 2024-04-25T11:43:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.981+0000 7f48c26a3640 1 -- 172.21.15.90:0/3283434897 learned_addr learned my addr 172.21.15.90:0/3283434897 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.981+0000 7f48c1ea2640 1 --2- 172.21.15.90:0/3283434897 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564f4f50d4d0 0x564f4f8d2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.981+0000 7f48c26a3640 1 -- 172.21.15.90:0/3283434897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f4f5156b0 msgr2=0x564f4f8d2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.981+0000 7f48c26a3640 1 --2- 172.21.15.90:0/3283434897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f4f5156b0 0x564f4f8d2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.981+0000 7f48c26a3640 1 -- 172.21.15.90:0/3283434897 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564f4f50d4d0 msgr2=0x564f4f8d2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.981+0000 7f48c26a3640 1 --2- 172.21.15.90:0/3283434897 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564f4f50d4d0 0x564f4f8d2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.981+0000 7f48c26a3640 1 -- 172.21.15.90:0/3283434897 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f4f7fd4b0 con 0x564f4f536390 2024-04-25T11:43:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.981+0000 7f48c1ea2640 1 --2- 172.21.15.90:0/3283434897 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564f4f50d4d0 0x564f4f8d2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.981+0000 7f48c26a3640 1 --2- 172.21.15.90:0/3283434897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f536390 0x564f4f956d80 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f48b4005ae0 tx=0x7f48b40140d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93334a634be6cdcc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:42.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.981+0000 7f48a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:42.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.981+0000 7f48a57fa640 1 -- 172.21.15.90:0/3283434897 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b4012b10 con 0x564f4f536390 2024-04-25T11:43:42.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.982+0000 7f48a57fa640 1 -- 172.21.15.90:0/3283434897 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f48b4012cb0 con 0x564f4f536390 2024-04-25T11:43:42.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.982+0000 7f48cafba740 1 -- 172.21.15.90:0/3283434897 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48b4026070 con 0x564f4f536390 2024-04-25T11:43:42.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.982+0000 7f48a57fa640 1 -- 172.21.15.90:0/3283434897 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f48b402b470 con 0x564f4f536390 2024-04-25T11:43:42.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.982+0000 7f48cafba740 10 client.18222 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:42.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.982+0000 7f48cafba740 1 -- 172.21.15.90:0/3283434897 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564f4f95f610 con 0x564f4f536390 2024-04-25T11:43:42.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.982+0000 7f48a57fa640 1 -- 172.21.15.90:0/3283434897 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48b4026070 con 0x564f4f536390 2024-04-25T11:43:42.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.982+0000 7f48a57fa640 1 -- 172.21.15.90:0/3283434897 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f48b403a030 con 0x564f4f536390 2024-04-25T11:43:42.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.982+0000 7f48cafba740 10 client.18222 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:42.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.982+0000 7f48cafba740 10 client.18222 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:43:42.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.982+0000 7f48cafba740 20 client.18222 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:42.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.982+0000 7f48cafba740 1 --2- 172.21.15.90:0/3283434897 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564f4f4ca3a0 0x564f4f961ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:42.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.982+0000 7f48cafba740 4 client.18222 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:43:42.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.982+0000 7f48cafba740 1 -- 172.21.15.90:0/3283434897 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564f4f844800 con 0x564f4f4ca3a0 2024-04-25T11:43:42.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.982+0000 7f48c1ea2640 1 --2- 172.21.15.90:0/3283434897 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564f4f4ca3a0 0x564f4f961ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:42.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.983+0000 7f48a57fa640 10 client.18222 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:42.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.983+0000 7f48c1ea2640 1 --2- 172.21.15.90:0/3283434897 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564f4f4ca3a0 0x564f4f961ce0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3379c4fe3101a385 server_cookie=99946a4a574d421d in_seq=0 out_seq=0 2024-04-25T11:43:42.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.983+0000 7f48a57fa640 1 -- 172.21.15.90:0/3283434897 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564f4f844800 con 0x564f4f4ca3a0 2024-04-25T11:43:42.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.983+0000 7f48a57fa640 10 client.18222 handle_command_reply: tid=0 2024-04-25T11:43:42.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.983+0000 7f48cafba740 1 client.18222 shutdown 2024-04-25T11:43:42.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.983+0000 7f48cafba740 1 -- 172.21.15.90:0/3283434897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f536390 msgr2=0x564f4f956d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:42.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.983+0000 7f48cafba740 1 --2- 172.21.15.90:0/3283434897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f536390 0x564f4f956d80 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f48b4005ae0 tx=0x7f48b40140d0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.983+0000 7f48cafba740 1 -- 172.21.15.90:0/3283434897 shutdown_connections 2024-04-25T11:43:42.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.983+0000 7f48cafba740 1 --2- 172.21.15.90:0/3283434897 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x564f4f4ca3a0 0x564f4f961ce0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.983+0000 7f48cafba740 1 --2- 172.21.15.90:0/3283434897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564f4f536390 0x564f4f956d80 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.983+0000 7f48cafba740 1 --2- 172.21.15.90:0/3283434897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f4f5156b0 0x564f4f8d2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.983+0000 7f48cafba740 1 --2- 172.21.15.90:0/3283434897 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x564f4f50d4d0 0x564f4f8d2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.983+0000 7f48a57fa640 0 client.18222 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:42.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.983+0000 7f48cafba740 1 -- 172.21.15.90:0/3283434897 >> 172.21.15.90:0/3283434897 conn(0x564f4f52a4e0 msgr2=0x564f4f8d8660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:42.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.983+0000 7f48cafba740 1 -- 172.21.15.90:0/3283434897 shutdown_connections 2024-04-25T11:43:42.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.983+0000 7f48cafba740 1 -- 172.21.15.90:0/3283434897 wait complete. 2024-04-25T11:43:42.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.983+0000 7f48cafba740 20 client.18222 trim_cache size 0 max 16384 2024-04-25T11:43:42.977 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:43:42.977 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-25T11:43:42.977 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-25T11:43:42.977 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:43:42.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.984+0000 7f48c4106640 1 -- 172.21.15.90:0/3610713414 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f48a80afc60 msgr2=0x7f48a80b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:42.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.984+0000 7f48c4106640 1 --2- 172.21.15.90:0/3610713414 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f48a80afc60 0x7f48a80b2170 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x564f4f8d3bd0 tx=0x7f48b4009ac0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.984+0000 7f48c4106640 1 -- 172.21.15.90:0/3610713414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48bc104260 msgr2=0x7f48bc19a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:42.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.984+0000 7f48c4106640 1 --2- 172.21.15.90:0/3610713414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48bc104260 0x7f48bc19a780 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f48b8006010 tx=0x7f48b8004060 comp rx=0 tx=0).stop 2024-04-25T11:43:42.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.984+0000 7f48c4106640 1 -- 172.21.15.90:0/3610713414 shutdown_connections 2024-04-25T11:43:42.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.984+0000 7f48c4106640 1 --2- 172.21.15.90:0/3610713414 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f48a80afc60 0x7f48a80b2170 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.984+0000 7f48c4106640 1 --2- 172.21.15.90:0/3610713414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48bc109fb0 0x7f48bc19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.984+0000 7f48c4106640 1 --2- 172.21.15.90:0/3610713414 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f48bc109b10 0x7f48bc19ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.984+0000 7f48c4106640 1 --2- 172.21.15.90:0/3610713414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48bc104260 0x7f48bc19a780 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:42.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.984+0000 7f48c4106640 1 -- 172.21.15.90:0/3610713414 >> 172.21.15.90:0/3610713414 conn(0x7f48bc0ff210 msgr2=0x7f48bc0ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:42.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.984+0000 7f48c4106640 1 -- 172.21.15.90:0/3610713414 shutdown_connections 2024-04-25T11:43:42.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:42.984+0000 7f48c4106640 1 -- 172.21.15.90:0/3610713414 wait complete. 2024-04-25T11:43:42.982 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T11:43:42.982 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:43:43.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.104+0000 7fe75d815640 1 -- 172.21.15.90:0/1868970771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe758109b10 msgr2=0x7fe7581036b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.104+0000 7fe75d815640 1 --2- 172.21.15.90:0/1868970771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe758109b10 0x7fe7581036b0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fe74c009900 tx=0x7fe74c02a410 comp rx=0 tx=0).stop 2024-04-25T11:43:43.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.104+0000 7fe75d815640 1 -- 172.21.15.90:0/1868970771 shutdown_connections 2024-04-25T11:43:43.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.104+0000 7fe75d815640 1 --2- 172.21.15.90:0/1868970771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe758104260 0x7fe758110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.104+0000 7fe75d815640 1 --2- 172.21.15.90:0/1868970771 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe758109fb0 0x7fe758103bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.104+0000 7fe75d815640 1 --2- 172.21.15.90:0/1868970771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe758109b10 0x7fe7581036b0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.104+0000 7fe75d815640 1 -- 172.21.15.90:0/1868970771 >> 172.21.15.90:0/1868970771 conn(0x7fe758079270 msgr2=0x7fe7580796b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:43.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.104+0000 7fe75d815640 1 -- 172.21.15.90:0/1868970771 shutdown_connections 2024-04-25T11:43:43.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.104+0000 7fe75d815640 1 -- 172.21.15.90:0/1868970771 wait complete. 2024-04-25T11:43:43.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.104+0000 7fe75d815640 1 Processor -- start 2024-04-25T11:43:43.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.105+0000 7fe75d815640 1 -- start start 2024-04-25T11:43:43.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.105+0000 7fe75d815640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe758104260 0x7fe758074ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.105+0000 7fe75d815640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe758109b10 0x7fe7580753e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.105+0000 7fe75d815640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe758109fb0 0x7fe758075920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.105+0000 7fe75d815640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe758113d60 con 0x7fe758109b10 2024-04-25T11:43:43.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.105+0000 7fe75d815640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe758113be0 con 0x7fe758104260 2024-04-25T11:43:43.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.105+0000 7fe75d815640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe758113ee0 con 0x7fe758109fb0 2024-04-25T11:43:43.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.105+0000 7fe7567fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe758109b10 0x7fe7580753e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.105+0000 7fe7567fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe758109b10 0x7fe7580753e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41890/0 (socket says 172.21.15.90:41890) 2024-04-25T11:43:43.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.105+0000 7fe7567fc640 1 -- 172.21.15.90:0/4285065574 learned_addr learned my addr 172.21.15.90:0/4285065574 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:43.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.105+0000 7fe7577fe640 1 --2- 172.21.15.90:0/4285065574 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe758109fb0 0x7fe758075920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.105+0000 7fe756ffd640 1 --2- 172.21.15.90:0/4285065574 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe758104260 0x7fe758074ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.105+0000 7fe7567fc640 1 -- 172.21.15.90:0/4285065574 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe758109fb0 msgr2=0x7fe758075920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.105+0000 7fe7567fc640 1 --2- 172.21.15.90:0/4285065574 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe758109fb0 0x7fe758075920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.105+0000 7fe7567fc640 1 -- 172.21.15.90:0/4285065574 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe758104260 msgr2=0x7fe758074ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.105+0000 7fe7567fc640 1 --2- 172.21.15.90:0/4285065574 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe758104260 0x7fe758074ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.105+0000 7fe7567fc640 1 -- 172.21.15.90:0/4285065574 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7581a8b40 con 0x7fe758109b10 2024-04-25T11:43:43.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.105+0000 7fe7577fe640 1 --2- 172.21.15.90:0/4285065574 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe758109fb0 0x7fe758075920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:43.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.105+0000 7fe756ffd640 1 --2- 172.21.15.90:0/4285065574 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe758104260 0x7fe758074ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:43:43.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.105+0000 7fe7567fc640 1 --2- 172.21.15.90:0/4285065574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe758109b10 0x7fe7580753e0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fe74800d740 tx=0x7fe74800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:43.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.106+0000 7fe75c813640 1 -- 172.21.15.90:0/4285065574 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe748017070 con 0x7fe758109b10 2024-04-25T11:43:43.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.106+0000 7fe75c813640 1 -- 172.21.15.90:0/4285065574 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe74800de50 con 0x7fe758109b10 2024-04-25T11:43:43.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.106+0000 7fe75d815640 1 -- 172.21.15.90:0/4285065574 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7581a8e30 con 0x7fe758109b10 2024-04-25T11:43:43.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.106+0000 7fe75c813640 1 -- 172.21.15.90:0/4285065574 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe748003c20 con 0x7fe758109b10 2024-04-25T11:43:43.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.106+0000 7fe75d815640 1 -- 172.21.15.90:0/4285065574 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7581a92a0 con 0x7fe758109b10 2024-04-25T11:43:43.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.107+0000 7fe75d815640 1 -- 172.21.15.90:0/4285065574 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe758069310 con 0x7fe758109b10 2024-04-25T11:43:43.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.109+0000 7fe75c813640 1 -- 172.21.15.90:0/4285065574 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fe748003dc0 con 0x7fe758109b10 2024-04-25T11:43:43.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.109+0000 7fe75c813640 1 --2- 172.21.15.90:0/4285065574 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe7400afc60 0x7fe7400b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.110+0000 7fe756ffd640 1 --2- 172.21.15.90:0/4285065574 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe7400afc60 0x7fe7400b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.110+0000 7fe75c813640 1 -- 172.21.15.90:0/4285065574 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7fe7480df000 con 0x7fe758109b10 2024-04-25T11:43:43.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.110+0000 7fe756ffd640 1 --2- 172.21.15.90:0/4285065574 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe7400afc60 0x7fe7400b2170 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe74c0096c0 tx=0x7fe74c031400 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:43.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.113+0000 7fe75c813640 1 -- 172.21.15.90:0/4285065574 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe7480a7810 con 0x7fe758109b10 2024-04-25T11:43:43.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.332+0000 7fe75d815640 1 -- 172.21.15.90:0/4285065574 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7580763d0 con 0x7fe758109b10 2024-04-25T11:43:43.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.333+0000 7fe75c813640 1 -- 172.21.15.90:0/4285065574 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11930 (secure 0 0 0) 0x7fe7480a7110 con 0x7fe758109b10 2024-04-25T11:43:43.320 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:43:43.320 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:39:20.469254+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24458,24431,24499,24475,24524]},"id":1}]} 2024-04-25T11:43:43.320 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 28 2024-04-25T11:43:43.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.335+0000 7fe75d815640 1 -- 172.21.15.90:0/4285065574 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe7400afc60 msgr2=0x7fe7400b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.335+0000 7fe75d815640 1 --2- 172.21.15.90:0/4285065574 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe7400afc60 0x7fe7400b2170 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe74c0096c0 tx=0x7fe74c031400 comp rx=0 tx=0).stop 2024-04-25T11:43:43.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.335+0000 7fe75d815640 1 -- 172.21.15.90:0/4285065574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe758109b10 msgr2=0x7fe7580753e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.335+0000 7fe75d815640 1 --2- 172.21.15.90:0/4285065574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe758109b10 0x7fe7580753e0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fe74800d740 tx=0x7fe74800dc00 comp rx=0 tx=0).stop 2024-04-25T11:43:43.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.335+0000 7fe75d815640 1 -- 172.21.15.90:0/4285065574 shutdown_connections 2024-04-25T11:43:43.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.335+0000 7fe75d815640 1 --2- 172.21.15.90:0/4285065574 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe7400afc60 0x7fe7400b2170 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.335+0000 7fe75d815640 1 --2- 172.21.15.90:0/4285065574 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe758109fb0 0x7fe758075920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.335+0000 7fe75d815640 1 --2- 172.21.15.90:0/4285065574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe758109b10 0x7fe7580753e0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.335+0000 7fe75d815640 1 --2- 172.21.15.90:0/4285065574 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe758104260 0x7fe758074ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.335+0000 7fe75d815640 1 -- 172.21.15.90:0/4285065574 >> 172.21.15.90:0/4285065574 conn(0x7fe758079270 msgr2=0x7fe7581051b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:43.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.335+0000 7fe75d815640 1 -- 172.21.15.90:0/4285065574 shutdown_connections 2024-04-25T11:43:43.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.335+0000 7fe75d815640 1 -- 172.21.15.90:0/4285065574 wait complete. 2024-04-25T11:43:43.334 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T11:43:43.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.458+0000 7fd73230d640 1 -- 172.21.15.90:0/4220520365 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd72c10d3f0 msgr2=0x7fd72c10f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.458+0000 7fd73230d640 1 --2- 172.21.15.90:0/4220520365 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd72c10d3f0 0x7fd72c10f810 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fd724009cd0 tx=0x7fd72402a8e0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.458+0000 7fd73230d640 1 -- 172.21.15.90:0/4220520365 shutdown_connections 2024-04-25T11:43:43.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.458+0000 7fd73230d640 1 --2- 172.21.15.90:0/4220520365 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd72c10d3f0 0x7fd72c10f810 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.458+0000 7fd73230d640 1 --2- 172.21.15.90:0/4220520365 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd72c103280 0x7fd72c103690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.458+0000 7fd73230d640 1 --2- 172.21.15.90:0/4220520365 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd72c107950 0x7fd72c102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.458+0000 7fd73230d640 1 -- 172.21.15.90:0/4220520365 >> 172.21.15.90:0/4220520365 conn(0x7fd72c077880 msgr2=0x7fd72c077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:43.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.458+0000 7fd73230d640 1 -- 172.21.15.90:0/4220520365 shutdown_connections 2024-04-25T11:43:43.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.458+0000 7fd73230d640 1 -- 172.21.15.90:0/4220520365 wait complete. 2024-04-25T11:43:43.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.458+0000 7fd73230d640 1 Processor -- start 2024-04-25T11:43:43.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.458+0000 7fd73230d640 1 -- start start 2024-04-25T11:43:43.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.459+0000 7fd73230d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd72c103280 0x7fd72c199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.459+0000 7fd73230d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd72c107950 0x7fd72c199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.459+0000 7fd73230d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd72c10d3f0 0x7fd72c19de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.459+0000 7fd73230d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c112bf0 con 0x7fd72c107950 2024-04-25T11:43:43.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.459+0000 7fd73230d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c112a70 con 0x7fd72c103280 2024-04-25T11:43:43.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.459+0000 7fd73230d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c112d70 con 0x7fd72c10d3f0 2024-04-25T11:43:43.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.459+0000 7fd72b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd72c107950 0x7fd72c199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.459+0000 7fd72b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd72c107950 0x7fd72c199b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41910/0 (socket says 172.21.15.90:41910) 2024-04-25T11:43:43.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.459+0000 7fd72b7fe640 1 -- 172.21.15.90:0/438307750 learned_addr learned my addr 172.21.15.90:0/438307750 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:43.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.459+0000 7fd72bfff640 1 --2- 172.21.15.90:0/438307750 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd72c103280 0x7fd72c199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.459+0000 7fd7308aa640 1 --2- 172.21.15.90:0/438307750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd72c10d3f0 0x7fd72c19de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.459+0000 7fd72b7fe640 1 -- 172.21.15.90:0/438307750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd72c10d3f0 msgr2=0x7fd72c19de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.459+0000 7fd72b7fe640 1 --2- 172.21.15.90:0/438307750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd72c10d3f0 0x7fd72c19de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.459+0000 7fd72b7fe640 1 -- 172.21.15.90:0/438307750 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd72c103280 msgr2=0x7fd72c199610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.459+0000 7fd72b7fe640 1 --2- 172.21.15.90:0/438307750 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd72c103280 0x7fd72c199610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.459+0000 7fd72b7fe640 1 -- 172.21.15.90:0/438307750 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72c19e470 con 0x7fd72c107950 2024-04-25T11:43:43.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.459+0000 7fd7308aa640 1 --2- 172.21.15.90:0/438307750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd72c10d3f0 0x7fd72c19de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:43.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.459+0000 7fd72b7fe640 1 --2- 172.21.15.90:0/438307750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd72c107950 0x7fd72c199b70 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fd72000c970 tx=0x7fd72000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:43.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.460+0000 7fd7297fa640 1 -- 172.21.15.90:0/438307750 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd720007bb0 con 0x7fd72c107950 2024-04-25T11:43:43.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.460+0000 7fd73230d640 1 -- 172.21.15.90:0/438307750 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd72c19e760 con 0x7fd72c107950 2024-04-25T11:43:43.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.460+0000 7fd7297fa640 1 -- 172.21.15.90:0/438307750 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd720007d50 con 0x7fd72c107950 2024-04-25T11:43:43.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.460+0000 7fd7297fa640 1 -- 172.21.15.90:0/438307750 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd72000f450 con 0x7fd72c107950 2024-04-25T11:43:43.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.460+0000 7fd73230d640 1 -- 172.21.15.90:0/438307750 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd72c1a60e0 con 0x7fd72c107950 2024-04-25T11:43:43.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.462+0000 7fd7297fa640 1 -- 172.21.15.90:0/438307750 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fd720004ec0 con 0x7fd72c107950 2024-04-25T11:43:43.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.462+0000 7fd7391be740 1 Processor -- start 2024-04-25T11:43:43.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.462+0000 7fd7391be740 1 -- start start 2024-04-25T11:43:43.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.463+0000 7fd7391be740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f788b93390 0x55f788f34810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.463+0000 7fd7391be740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b6a4d0 0x55f788f34d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.463+0000 7fd7391be740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f788b726b0 0x55f788f416a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.463+0000 7fd7391be740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f788e99d80 con 0x55f788b6a4d0 2024-04-25T11:43:43.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.463+0000 7fd7391be740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f788e53a20 con 0x55f788b726b0 2024-04-25T11:43:43.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.463+0000 7fd7391be740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f788e95d40 con 0x55f788b93390 2024-04-25T11:43:43.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.463+0000 7fd72bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b6a4d0 0x55f788f34d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.463+0000 7fd72bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b6a4d0 0x55f788f34d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41920/0 (socket says 172.21.15.90:41920) 2024-04-25T11:43:43.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.463+0000 7fd72bfff640 1 -- 172.21.15.90:0/712785643 learned_addr learned my addr 172.21.15.90:0/712785643 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:43.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.463+0000 7fd7297fa640 1 --2- 172.21.15.90:0/438307750 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd7140afc60 0x7fd7140b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.463+0000 7fd7308aa640 1 --2- 172.21.15.90:0/712785643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f788b93390 0x55f788f34810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.463+0000 7fd7308aa640 1 --2- 172.21.15.90:0/438307750 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd7140afc60 0x7fd7140b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.463+0000 7fd7297fa640 1 -- 172.21.15.90:0/438307750 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7fd7200dee40 con 0x7fd72c107950 2024-04-25T11:43:43.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.463+0000 7fd72b7fe640 1 --2- 172.21.15.90:0/712785643 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f788b726b0 0x55f788f416a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.463+0000 7fd72bfff640 1 -- 172.21.15.90:0/712785643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f788b93390 msgr2=0x55f788f34810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.463+0000 7fd72bfff640 1 --2- 172.21.15.90:0/712785643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f788b93390 0x55f788f34810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.463+0000 7fd72bfff640 1 -- 172.21.15.90:0/712785643 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f788b726b0 msgr2=0x55f788f416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.463+0000 7fd72bfff640 1 --2- 172.21.15.90:0/712785643 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f788b726b0 0x55f788f416a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.463+0000 7fd72bfff640 1 -- 172.21.15.90:0/712785643 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f788e5ec30 con 0x55f788b6a4d0 2024-04-25T11:43:43.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.463+0000 7fd72b7fe640 1 --2- 172.21.15.90:0/712785643 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f788b726b0 0x55f788f416a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:43:43.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.463+0000 7fd72bfff640 1 --2- 172.21.15.90:0/712785643 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b6a4d0 0x55f788f34d50 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fd71c00bc30 tx=0x7fd71c010930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb08ad5fe13e00f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:43.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.463+0000 7fd7308aa640 1 --2- 172.21.15.90:0/438307750 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd7140afc60 0x7fd7140b2170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd7240094e0 tx=0x7fd7240095c0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:43.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.463+0000 7fd7127fc640 1 -- 172.21.15.90:0/712785643 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c01e070 con 0x55f788b6a4d0 2024-04-25T11:43:43.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.464+0000 7fd7127fc640 1 -- 172.21.15.90:0/712785643 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd71c0198c0 con 0x55f788b6a4d0 2024-04-25T11:43:43.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.464+0000 7fd7127fc640 1 -- 172.21.15.90:0/712785643 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c018770 con 0x55f788b6a4d0 2024-04-25T11:43:43.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.464+0000 7fd7391be740 1 -- 172.21.15.90:0/712785643 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b6a4d0 msgr2=0x55f788f34d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.464+0000 7fd7391be740 1 --2- 172.21.15.90:0/712785643 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b6a4d0 0x55f788f34d50 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fd71c00bc30 tx=0x7fd71c010930 comp rx=0 tx=0).stop 2024-04-25T11:43:43.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.464+0000 7fd7391be740 1 -- 172.21.15.90:0/712785643 shutdown_connections 2024-04-25T11:43:43.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.464+0000 7fd7391be740 1 --2- 172.21.15.90:0/712785643 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f788b726b0 0x55f788f416a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.464+0000 7fd7391be740 1 --2- 172.21.15.90:0/712785643 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b6a4d0 0x55f788f34d50 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.464+0000 7fd7391be740 1 --2- 172.21.15.90:0/712785643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f788b93390 0x55f788f34810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.464+0000 7fd7391be740 1 -- 172.21.15.90:0/712785643 >> 172.21.15.90:0/712785643 conn(0x55f788b874e0 msgr2=0x55f788f30fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:43.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.464+0000 7fd7391be740 1 -- 172.21.15.90:0/712785643 shutdown_connections 2024-04-25T11:43:43.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.464+0000 7fd7391be740 1 -- 172.21.15.90:0/712785643 wait complete. 2024-04-25T11:43:43.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.465+0000 7fd7391be740 1 -- start start 2024-04-25T11:43:43.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.465+0000 7fd7391be740 1 Processor -- start 2024-04-25T11:43:43.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.465+0000 7fd7391be740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f788b6a4d0 0x55f788f3f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.465+0000 7fd7391be740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b726b0 0x55f788f3f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.465+0000 7fd7391be740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f788b93390 0x55f788f4b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.465+0000 7fd7391be740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f788f41d90 con 0x55f788b726b0 2024-04-25T11:43:43.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.465+0000 7fd7391be740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f788e20dc0 con 0x55f788b6a4d0 2024-04-25T11:43:43.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.465+0000 7fd7391be740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f788f41f10 con 0x55f788b93390 2024-04-25T11:43:43.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.465+0000 7fd7391be740 10 client.0 _ll_register_callbacks cb 0x55f788e3c600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:43.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.465+0000 7fd72b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b726b0 0x55f788f3f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.465+0000 7fd72b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b726b0 0x55f788f3f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41936/0 (socket says 172.21.15.90:41936) 2024-04-25T11:43:43.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.465+0000 7fd72b7fe640 1 -- 172.21.15.90:0/1839393661 learned_addr learned my addr 172.21.15.90:0/1839393661 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:43.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.466+0000 7fd72bfff640 1 --2- 172.21.15.90:0/1839393661 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f788b6a4d0 0x55f788f3f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.466+0000 7fd72b7fe640 1 -- 172.21.15.90:0/1839393661 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f788b93390 msgr2=0x55f788f4b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.466+0000 7fd7308aa640 1 --2- 172.21.15.90:0/1839393661 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f788b93390 0x55f788f4b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.466+0000 7fd72b7fe640 1 --2- 172.21.15.90:0/1839393661 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f788b93390 0x55f788f4b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.466+0000 7fd72b7fe640 1 -- 172.21.15.90:0/1839393661 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f788b6a4d0 msgr2=0x55f788f3f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.466+0000 7fd72b7fe640 1 --2- 172.21.15.90:0/1839393661 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f788b6a4d0 0x55f788f3f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.466+0000 7fd72b7fe640 1 -- 172.21.15.90:0/1839393661 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f788bab3f0 con 0x55f788b726b0 2024-04-25T11:43:43.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.466+0000 7fd72bfff640 1 --2- 172.21.15.90:0/1839393661 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f788b6a4d0 0x55f788f3f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:43:43.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.466+0000 7fd72b7fe640 1 --2- 172.21.15.90:0/1839393661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b726b0 0x55f788f3f7c0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fd7200a2fd0 tx=0x7fd7200a7570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9bb384fddf1f26b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:43.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.466+0000 7fd70b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:43.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.466+0000 7fd70b7fe640 1 -- 172.21.15.90:0/1839393661 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd72001d040 con 0x55f788b726b0 2024-04-25T11:43:43.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.466+0000 7fd70b7fe640 1 -- 172.21.15.90:0/1839393661 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7fd7200a5c20 con 0x55f788b726b0 2024-04-25T11:43:43.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.466+0000 7fd7391be740 1 -- 172.21.15.90:0/1839393661 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd71c0198c0 con 0x55f788b726b0 2024-04-25T11:43:43.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.466+0000 7fd70b7fe640 1 -- 172.21.15.90:0/1839393661 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd7200046c0 con 0x55f788b726b0 2024-04-25T11:43:43.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.466+0000 7fd7391be740 10 client.18240 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:43.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.466+0000 7fd7391be740 1 -- 172.21.15.90:0/1839393661 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f788c757d0 con 0x55f788b726b0 2024-04-25T11:43:43.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.466+0000 7fd70b7fe640 1 -- 172.21.15.90:0/1839393661 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd71c0198c0 con 0x55f788b726b0 2024-04-25T11:43:43.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.466+0000 7fd70b7fe640 1 -- 172.21.15.90:0/1839393661 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fd7200a5e20 con 0x55f788b726b0 2024-04-25T11:43:43.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.466+0000 7fd7391be740 10 client.18240 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:43.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.467+0000 7fd7391be740 10 client.18240 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:43:43.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.467+0000 7fd7391be740 20 client.18240 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:43.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.467+0000 7fd7391be740 1 --2- 172.21.15.90:0/1839393661 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55f788b51f60 0x55f788f51700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.467+0000 7fd7391be740 4 client.18240 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:43:43.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.467+0000 7fd7391be740 1 -- 172.21.15.90:0/1839393661 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f788ea7990 con 0x55f788b51f60 2024-04-25T11:43:43.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.467+0000 7fd72bfff640 1 --2- 172.21.15.90:0/1839393661 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55f788b51f60 0x55f788f51700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.467+0000 7fd70b7fe640 10 client.18240 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:43:43.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.467+0000 7fd72bfff640 1 --2- 172.21.15.90:0/1839393661 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55f788b51f60 0x55f788f51700 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=840172dce6345c2f server_cookie=1ac4bd9ee9e78a12 in_seq=0 out_seq=0 2024-04-25T11:43:43.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.468+0000 7fd70b7fe640 1 -- 172.21.15.90:0/1839393661 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f788ea7990 con 0x55f788b51f60 2024-04-25T11:43:43.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.468+0000 7fd70b7fe640 10 client.18240 handle_command_reply: tid=0 2024-04-25T11:43:43.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.468+0000 7fd7391be740 1 client.18240 shutdown 2024-04-25T11:43:43.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.468+0000 7fd7391be740 1 -- 172.21.15.90:0/1839393661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b726b0 msgr2=0x55f788f3f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.468+0000 7fd7391be740 1 --2- 172.21.15.90:0/1839393661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b726b0 0x55f788f3f7c0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fd7200a2fd0 tx=0x7fd7200a7570 comp rx=0 tx=0).stop 2024-04-25T11:43:43.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.468+0000 7fd7391be740 1 -- 172.21.15.90:0/1839393661 shutdown_connections 2024-04-25T11:43:43.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.468+0000 7fd7391be740 1 --2- 172.21.15.90:0/1839393661 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55f788b51f60 0x55f788f51700 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.468+0000 7fd7391be740 1 --2- 172.21.15.90:0/1839393661 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f788b93390 0x55f788f4b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.468+0000 7fd70b7fe640 0 client.18240 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:43:43.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.468+0000 7fd7391be740 1 --2- 172.21.15.90:0/1839393661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b726b0 0x55f788f3f7c0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.468+0000 7fd7391be740 1 --2- 172.21.15.90:0/1839393661 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f788b6a4d0 0x55f788f3f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.468+0000 7fd7391be740 1 -- 172.21.15.90:0/1839393661 >> 172.21.15.90:0/1839393661 conn(0x55f788b874e0 msgr2=0x55f788f36530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:43.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.468+0000 7fd7391be740 1 -- 172.21.15.90:0/1839393661 shutdown_connections 2024-04-25T11:43:43.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.469+0000 7fd7391be740 1 -- 172.21.15.90:0/1839393661 wait complete. 2024-04-25T11:43:43.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.469+0000 7fd7391be740 20 client.18240 trim_cache size 0 max 16384 2024-04-25T11:43:43.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.482+0000 7fd7391be740 1 Processor -- start 2024-04-25T11:43:43.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.482+0000 7fd7391be740 1 -- start start 2024-04-25T11:43:43.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.482+0000 7fd7391be740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f788b6a4d0 0x55f788f3e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.482+0000 7fd7391be740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f788b726b0 0x55f788f4ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.482+0000 7fd7391be740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b93390 0x55f788f39430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.482+0000 7fd7391be740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c1ada70 con 0x55f788b93390 2024-04-25T11:43:43.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.482+0000 7fd7391be740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c1af5b0 con 0x55f788b726b0 2024-04-25T11:43:43.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.482+0000 7fd7391be740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c1ae140 con 0x55f788b6a4d0 2024-04-25T11:43:43.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.482+0000 7fd7308aa640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b93390 0x55f788f39430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.482+0000 7fd7308aa640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b93390 0x55f788f39430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41944/0 (socket says 172.21.15.90:41944) 2024-04-25T11:43:43.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.482+0000 7fd7308aa640 1 -- 172.21.15.90:0/2617549862 learned_addr learned my addr 172.21.15.90:0/2617549862 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:43.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.483+0000 7fd72bfff640 1 --2- 172.21.15.90:0/2617549862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f788b6a4d0 0x55f788f3e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.483+0000 7fd72b7fe640 1 --2- 172.21.15.90:0/2617549862 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f788b726b0 0x55f788f4ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.483+0000 7fd7308aa640 1 -- 172.21.15.90:0/2617549862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f788b6a4d0 msgr2=0x55f788f3e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.483+0000 7fd7308aa640 1 --2- 172.21.15.90:0/2617549862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f788b6a4d0 0x55f788f3e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.483+0000 7fd7308aa640 1 -- 172.21.15.90:0/2617549862 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f788b726b0 msgr2=0x55f788f4ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.483+0000 7fd7308aa640 1 --2- 172.21.15.90:0/2617549862 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f788b726b0 0x55f788f4ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.483+0000 7fd7308aa640 1 -- 172.21.15.90:0/2617549862 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f788f350c0 con 0x55f788b93390 2024-04-25T11:43:43.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.483+0000 7fd72bfff640 1 --2- 172.21.15.90:0/2617549862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f788b6a4d0 0x55f788f3e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:43.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.483+0000 7fd7308aa640 1 --2- 172.21.15.90:0/2617549862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b93390 0x55f788f39430 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fd72c103d20 tx=0x7fd72403a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30397d47dbbac394 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:43.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.483+0000 7fd711ffb640 1 -- 172.21.15.90:0/2617549862 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd724031970 con 0x55f788b93390 2024-04-25T11:43:43.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.483+0000 7fd711ffb640 1 -- 172.21.15.90:0/2617549862 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd724036040 con 0x55f788b93390 2024-04-25T11:43:43.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.483+0000 7fd711ffb640 1 -- 172.21.15.90:0/2617549862 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd724030690 con 0x55f788b93390 2024-04-25T11:43:43.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.484+0000 7fd7391be740 1 -- 172.21.15.90:0/2617549862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b93390 msgr2=0x55f788f39430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.484+0000 7fd7391be740 1 --2- 172.21.15.90:0/2617549862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b93390 0x55f788f39430 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fd72c103d20 tx=0x7fd72403a000 comp rx=0 tx=0).stop 2024-04-25T11:43:43.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.484+0000 7fd7391be740 1 -- 172.21.15.90:0/2617549862 shutdown_connections 2024-04-25T11:43:43.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.484+0000 7fd7391be740 1 --2- 172.21.15.90:0/2617549862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b93390 0x55f788f39430 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.484+0000 7fd7391be740 1 --2- 172.21.15.90:0/2617549862 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f788b726b0 0x55f788f4ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.484+0000 7fd7391be740 1 --2- 172.21.15.90:0/2617549862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f788b6a4d0 0x55f788f3e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.484+0000 7fd7391be740 1 -- 172.21.15.90:0/2617549862 >> 172.21.15.90:0/2617549862 conn(0x55f788b874e0 msgr2=0x55f788f3d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:43.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.484+0000 7fd7391be740 1 -- 172.21.15.90:0/2617549862 shutdown_connections 2024-04-25T11:43:43.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.484+0000 7fd7391be740 1 -- 172.21.15.90:0/2617549862 wait complete. 2024-04-25T11:43:43.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.484+0000 7fd7391be740 1 -- start start 2024-04-25T11:43:43.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.484+0000 7fd7391be740 1 Processor -- start 2024-04-25T11:43:43.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.485+0000 7fd7391be740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f788b6a4d0 0x55f788f473e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.485+0000 7fd7391be740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b726b0 0x55f788f42790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.485+0000 7fd7391be740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f788b93390 0x55f788f42cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.485+0000 7fd7391be740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c1aeee0 con 0x55f788b726b0 2024-04-25T11:43:43.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.485+0000 7fd7391be740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c1ae810 con 0x55f788b93390 2024-04-25T11:43:43.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.485+0000 7fd7391be740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c1aba70 con 0x55f788b6a4d0 2024-04-25T11:43:43.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.485+0000 7fd7391be740 10 client.0 _ll_register_callbacks cb 0x7fd7200a2e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:43.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.485+0000 7fd72b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b726b0 0x55f788f42790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.485+0000 7fd72b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b726b0 0x55f788f42790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41950/0 (socket says 172.21.15.90:41950) 2024-04-25T11:43:43.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.485+0000 7fd72b7fe640 1 -- 172.21.15.90:0/3277953871 learned_addr learned my addr 172.21.15.90:0/3277953871 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:43.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.485+0000 7fd7308aa640 1 --2- 172.21.15.90:0/3277953871 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f788b93390 0x55f788f42cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.485+0000 7fd72bfff640 1 --2- 172.21.15.90:0/3277953871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f788b6a4d0 0x55f788f473e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.485+0000 7fd72b7fe640 1 -- 172.21.15.90:0/3277953871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f788b6a4d0 msgr2=0x55f788f473e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.485+0000 7fd72b7fe640 1 --2- 172.21.15.90:0/3277953871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f788b6a4d0 0x55f788f473e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.485+0000 7fd72b7fe640 1 -- 172.21.15.90:0/3277953871 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f788b93390 msgr2=0x55f788f42cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.485+0000 7fd72b7fe640 1 --2- 172.21.15.90:0/3277953871 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f788b93390 0x55f788f42cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.485+0000 7fd72b7fe640 1 -- 172.21.15.90:0/3277953871 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f788f43210 con 0x55f788b726b0 2024-04-25T11:43:43.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.485+0000 7fd7308aa640 1 --2- 172.21.15.90:0/3277953871 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f788b93390 0x55f788f42cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:43.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.485+0000 7fd72b7fe640 1 --2- 172.21.15.90:0/3277953871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b726b0 0x55f788f42790 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fd7200df1f0 tx=0x7fd7200a5e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb1a575db0557a59 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:43.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.485+0000 7fd70b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:43.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.485+0000 7fd70b7fe640 1 -- 172.21.15.90:0/3277953871 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd72001d040 con 0x55f788b726b0 2024-04-25T11:43:43.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.485+0000 7fd70b7fe640 1 -- 172.21.15.90:0/3277953871 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7fd7200ac690 con 0x55f788b726b0 2024-04-25T11:43:43.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.485+0000 7fd7391be740 1 -- 172.21.15.90:0/3277953871 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd724036040 con 0x55f788b726b0 2024-04-25T11:43:43.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.485+0000 7fd70b7fe640 1 -- 172.21.15.90:0/3277953871 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd7200a6a60 con 0x55f788b726b0 2024-04-25T11:43:43.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.486+0000 7fd7391be740 10 client.18246 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:43.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.486+0000 7fd7391be740 1 -- 172.21.15.90:0/3277953871 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f788fb95d0 con 0x55f788b726b0 2024-04-25T11:43:43.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.486+0000 7fd70b7fe640 1 -- 172.21.15.90:0/3277953871 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd724036040 con 0x55f788b726b0 2024-04-25T11:43:43.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.486+0000 7fd70b7fe640 1 -- 172.21.15.90:0/3277953871 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fd72008a070 con 0x55f788b726b0 2024-04-25T11:43:43.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.486+0000 7fd7391be740 10 client.18246 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:43.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.486+0000 7fd7391be740 10 client.18246 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:43:43.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.486+0000 7fd7391be740 20 client.18246 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:43.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.486+0000 7fd7391be740 1 --2- 172.21.15.90:0/3277953871 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55f788b273a0 0x55f788fbbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.486+0000 7fd7391be740 4 client.18246 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:43:43.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.486+0000 7fd7391be740 1 -- 172.21.15.90:0/3277953871 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f788f3fb90 con 0x55f788b273a0 2024-04-25T11:43:43.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.486+0000 7fd72bfff640 1 --2- 172.21.15.90:0/3277953871 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55f788b273a0 0x55f788fbbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.487+0000 7fd70b7fe640 10 client.18246 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:43:43.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.487+0000 7fd72bfff640 1 --2- 172.21.15.90:0/3277953871 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55f788b273a0 0x55f788fbbd90 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e0476255dbf7e567 server_cookie=116a5dd776c4ba4f in_seq=0 out_seq=0 2024-04-25T11:43:43.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.487+0000 7fd70b7fe640 1 -- 172.21.15.90:0/3277953871 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f788f3fb90 con 0x55f788b273a0 2024-04-25T11:43:43.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.487+0000 7fd70b7fe640 10 client.18246 handle_command_reply: tid=0 2024-04-25T11:43:43.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.487+0000 7fd7391be740 1 client.18246 shutdown 2024-04-25T11:43:43.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.487+0000 7fd7391be740 1 -- 172.21.15.90:0/3277953871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b726b0 msgr2=0x55f788f42790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.487+0000 7fd7391be740 1 --2- 172.21.15.90:0/3277953871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b726b0 0x55f788f42790 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fd7200df1f0 tx=0x7fd7200a5e50 comp rx=0 tx=0).stop 2024-04-25T11:43:43.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.487+0000 7fd7391be740 1 -- 172.21.15.90:0/3277953871 shutdown_connections 2024-04-25T11:43:43.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.487+0000 7fd7391be740 1 --2- 172.21.15.90:0/3277953871 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55f788b273a0 0x55f788fbbd90 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.487+0000 7fd7391be740 1 --2- 172.21.15.90:0/3277953871 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f788b93390 0x55f788f42cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.487+0000 7fd7391be740 1 --2- 172.21.15.90:0/3277953871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f788b726b0 0x55f788f42790 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.487+0000 7fd70b7fe640 0 client.18246 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:43:43.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.487+0000 7fd7391be740 1 --2- 172.21.15.90:0/3277953871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f788b6a4d0 0x55f788f473e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.487+0000 7fd7391be740 1 -- 172.21.15.90:0/3277953871 >> 172.21.15.90:0/3277953871 conn(0x55f788b874e0 msgr2=0x55f788f2f610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:43.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.487+0000 7fd7391be740 1 -- 172.21.15.90:0/3277953871 shutdown_connections 2024-04-25T11:43:43.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.487+0000 7fd7391be740 1 -- 172.21.15.90:0/3277953871 wait complete. 2024-04-25T11:43:43.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.487+0000 7fd7391be740 20 client.18246 trim_cache size 0 max 16384 2024-04-25T11:43:43.482 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:43:43.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.488+0000 7fd73230d640 1 -- 172.21.15.90:0/438307750 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd7140afc60 msgr2=0x7fd7140b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.488+0000 7fd73230d640 1 --2- 172.21.15.90:0/438307750 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd7140afc60 0x7fd7140b2170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd7240094e0 tx=0x7fd7240095c0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.488+0000 7fd73230d640 1 -- 172.21.15.90:0/438307750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd72c107950 msgr2=0x7fd72c199b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.488+0000 7fd73230d640 1 --2- 172.21.15.90:0/438307750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd72c107950 0x7fd72c199b70 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fd72000c970 tx=0x7fd72000ce30 comp rx=0 tx=0).stop 2024-04-25T11:43:43.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.488+0000 7fd73230d640 1 -- 172.21.15.90:0/438307750 shutdown_connections 2024-04-25T11:43:43.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.488+0000 7fd73230d640 1 --2- 172.21.15.90:0/438307750 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd7140afc60 0x7fd7140b2170 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.488+0000 7fd73230d640 1 --2- 172.21.15.90:0/438307750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd72c10d3f0 0x7fd72c19de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.488+0000 7fd73230d640 1 --2- 172.21.15.90:0/438307750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd72c107950 0x7fd72c199b70 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.488+0000 7fd73230d640 1 --2- 172.21.15.90:0/438307750 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd72c103280 0x7fd72c199610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.488+0000 7fd73230d640 1 -- 172.21.15.90:0/438307750 >> 172.21.15.90:0/438307750 conn(0x7fd72c077880 msgr2=0x7fd72c1008c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:43.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.488+0000 7fd73230d640 1 -- 172.21.15.90:0/438307750 shutdown_connections 2024-04-25T11:43:43.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.488+0000 7fd73230d640 1 -- 172.21.15.90:0/438307750 wait complete. 2024-04-25T11:43:43.486 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T11:43:43.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.609+0000 7f5384e67640 1 -- 172.21.15.90:0/1430924507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5380109b20 msgr2=0x7f53801036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.609+0000 7f5384e67640 1 --2- 172.21.15.90:0/1430924507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5380109b20 0x7f53801036c0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f5370009900 tx=0x7f537002a410 comp rx=0 tx=0).stop 2024-04-25T11:43:43.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.609+0000 7f5384e67640 1 -- 172.21.15.90:0/1430924507 shutdown_connections 2024-04-25T11:43:43.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.609+0000 7f5384e67640 1 --2- 172.21.15.90:0/1430924507 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5380104270 0x7f5380110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.609+0000 7f5384e67640 1 --2- 172.21.15.90:0/1430924507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5380109fc0 0x7f5380103c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.609+0000 7f5384e67640 1 --2- 172.21.15.90:0/1430924507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5380109b20 0x7f53801036c0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.609+0000 7f5384e67640 1 -- 172.21.15.90:0/1430924507 >> 172.21.15.90:0/1430924507 conn(0x7f53800ff1e0 msgr2=0x7f5380101630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:43.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.609+0000 7f5384e67640 1 -- 172.21.15.90:0/1430924507 shutdown_connections 2024-04-25T11:43:43.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.609+0000 7f5384e67640 1 -- 172.21.15.90:0/1430924507 wait complete. 2024-04-25T11:43:43.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.610+0000 7f5384e67640 1 Processor -- start 2024-04-25T11:43:43.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.610+0000 7f5384e67640 1 -- start start 2024-04-25T11:43:43.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.610+0000 7f5384e67640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5380104270 0x7f538019a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.610+0000 7f5384e67640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5380109b20 0x7f538019ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.610+0000 7f5384e67640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5380109fc0 0x7f538019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.610+0000 7f5384e67640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5380113d10 con 0x7f5380109fc0 2024-04-25T11:43:43.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.610+0000 7f5384e67640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5380113b90 con 0x7f5380104270 2024-04-25T11:43:43.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.610+0000 7f5384e67640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5380113e90 con 0x7f5380109b20 2024-04-25T11:43:43.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.610+0000 7f537ed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5380109fc0 0x7f538019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.610+0000 7f537ed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5380109fc0 0x7f538019eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41976/0 (socket says 172.21.15.90:41976) 2024-04-25T11:43:43.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.611+0000 7f537ed9d640 1 -- 172.21.15.90:0/1368577238 learned_addr learned my addr 172.21.15.90:0/1368577238 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:43.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.611+0000 7f537e59c640 1 --2- 172.21.15.90:0/1368577238 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5380104270 0x7f538019a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.611+0000 7f537dd9b640 1 --2- 172.21.15.90:0/1368577238 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5380109b20 0x7f538019ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.611+0000 7f537ed9d640 1 -- 172.21.15.90:0/1368577238 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5380109b20 msgr2=0x7f538019ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.611+0000 7f537ed9d640 1 --2- 172.21.15.90:0/1368577238 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5380109b20 0x7f538019ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.611+0000 7f537ed9d640 1 -- 172.21.15.90:0/1368577238 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5380104270 msgr2=0x7f538019a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.611+0000 7f537ed9d640 1 --2- 172.21.15.90:0/1368577238 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5380104270 0x7f538019a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.611+0000 7f537ed9d640 1 -- 172.21.15.90:0/1368577238 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f538019f630 con 0x7f5380109fc0 2024-04-25T11:43:43.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.611+0000 7f537dd9b640 1 --2- 172.21.15.90:0/1368577238 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5380109b20 0x7f538019ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:43:43.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.611+0000 7f537ed9d640 1 --2- 172.21.15.90:0/1368577238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5380109fc0 0x7f538019eff0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f5378009fd0 tx=0x7f537800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:43.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.611+0000 7f536f7fe640 1 -- 172.21.15.90:0/1368577238 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5378017070 con 0x7f5380109fc0 2024-04-25T11:43:43.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.611+0000 7f536f7fe640 1 -- 172.21.15.90:0/1368577238 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5378003e20 con 0x7f5380109fc0 2024-04-25T11:43:43.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.611+0000 7f5384e67640 1 -- 172.21.15.90:0/1368577238 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f538019f920 con 0x7f5380109fc0 2024-04-25T11:43:43.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.611+0000 7f536f7fe640 1 -- 172.21.15.90:0/1368577238 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5378010460 con 0x7f5380109fc0 2024-04-25T11:43:43.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.611+0000 7f5384e67640 1 -- 172.21.15.90:0/1368577238 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f538019fe20 con 0x7f5380109fc0 2024-04-25T11:43:43.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.614+0000 7f536f7fe640 1 -- 172.21.15.90:0/1368577238 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f5378003970 con 0x7f5380109fc0 2024-04-25T11:43:43.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.614+0000 7f538bdfb740 1 Processor -- start 2024-04-25T11:43:43.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.614+0000 7f538bdfb740 1 -- start start 2024-04-25T11:43:43.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.614+0000 7f538bdfb740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c2066cc390 0x55c206a6d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.614+0000 7f538bdfb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c2066a34d0 0x55c206a6dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.614+0000 7f538bdfb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066ab6b0 0x55c206a7a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.614+0000 7f538bdfb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c2069d2d80 con 0x55c2066ab6b0 2024-04-25T11:43:43.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.614+0000 7f538bdfb740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c20698ca20 con 0x55c2066cc390 2024-04-25T11:43:43.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.614+0000 7f538bdfb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c2069ced40 con 0x55c2066a34d0 2024-04-25T11:43:43.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.614+0000 7f537dd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066ab6b0 0x55c206a7a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.615+0000 7f537dd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066ab6b0 0x55c206a7a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41986/0 (socket says 172.21.15.90:41986) 2024-04-25T11:43:43.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.615+0000 7f537dd9b640 1 -- 172.21.15.90:0/603137033 learned_addr learned my addr 172.21.15.90:0/603137033 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:43.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.615+0000 7f536f7fe640 1 --2- 172.21.15.90:0/1368577238 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f53680afc60 0x7f53680b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.615+0000 7f537e59c640 1 --2- 172.21.15.90:0/603137033 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c2066a34d0 0x55c206a6dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.615+0000 7f537dd9b640 1 --2- 172.21.15.90:0/603137033 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c2066cc390 0x55c206a6d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.615+0000 7f537ed9d640 1 --2- 172.21.15.90:0/1368577238 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f53680afc60 0x7f53680b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.615+0000 7f536f7fe640 1 -- 172.21.15.90:0/1368577238 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f53780dee70 con 0x7f5380109fc0 2024-04-25T11:43:43.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.615+0000 7f537dd9b640 1 -- 172.21.15.90:0/603137033 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c2066a34d0 msgr2=0x55c206a6dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.615+0000 7f537dd9b640 1 --2- 172.21.15.90:0/603137033 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c2066a34d0 0x55c206a6dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.615+0000 7f537dd9b640 1 -- 172.21.15.90:0/603137033 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c2066cc390 msgr2=0x55c206a6d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.615+0000 7f537dd9b640 1 --2- 172.21.15.90:0/603137033 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c2066cc390 0x55c206a6d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.615+0000 7f537dd9b640 1 -- 172.21.15.90:0/603137033 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c206997c30 con 0x55c2066ab6b0 2024-04-25T11:43:43.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.615+0000 7f537dd9b640 1 --2- 172.21.15.90:0/603137033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066ab6b0 0x55c206a7a6a0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f5374009fd0 tx=0x7f5374009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d285280cfe1a4ea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:43.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.615+0000 7f537ed9d640 1 --2- 172.21.15.90:0/1368577238 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f53680afc60 0x7f53680b2170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f53780dc5e0 tx=0x7f537804e040 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:43.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.615+0000 7f536cff9640 1 -- 172.21.15.90:0/603137033 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5374013240 con 0x55c2066ab6b0 2024-04-25T11:43:43.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.615+0000 7f536cff9640 1 -- 172.21.15.90:0/603137033 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5374011c60 con 0x55c2066ab6b0 2024-04-25T11:43:43.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.615+0000 7f536cff9640 1 -- 172.21.15.90:0/603137033 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5374012870 con 0x55c2066ab6b0 2024-04-25T11:43:43.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.616+0000 7f538bdfb740 1 -- 172.21.15.90:0/603137033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066ab6b0 msgr2=0x55c206a7a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.616+0000 7f538bdfb740 1 --2- 172.21.15.90:0/603137033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066ab6b0 0x55c206a7a6a0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f5374009fd0 tx=0x7f5374009680 comp rx=0 tx=0).stop 2024-04-25T11:43:43.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.616+0000 7f538bdfb740 1 -- 172.21.15.90:0/603137033 shutdown_connections 2024-04-25T11:43:43.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.616+0000 7f538bdfb740 1 --2- 172.21.15.90:0/603137033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066ab6b0 0x55c206a7a6a0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.616+0000 7f538bdfb740 1 --2- 172.21.15.90:0/603137033 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c2066a34d0 0x55c206a6dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.616+0000 7f538bdfb740 1 --2- 172.21.15.90:0/603137033 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c2066cc390 0x55c206a6d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.616+0000 7f538bdfb740 1 -- 172.21.15.90:0/603137033 >> 172.21.15.90:0/603137033 conn(0x55c2066c04e0 msgr2=0x55c206a69fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:43.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.616+0000 7f538bdfb740 1 -- 172.21.15.90:0/603137033 shutdown_connections 2024-04-25T11:43:43.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.616+0000 7f538bdfb740 1 -- 172.21.15.90:0/603137033 wait complete. 2024-04-25T11:43:43.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.617+0000 7f538bdfb740 1 -- start start 2024-04-25T11:43:43.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.617+0000 7f538bdfb740 1 Processor -- start 2024-04-25T11:43:43.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.617+0000 7f538bdfb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066a34d0 0x55c206a782a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.617+0000 7f538bdfb740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c2066ab6b0 0x55c206a787e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.617+0000 7f538bdfb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c2066cc390 0x55c206a84210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.617+0000 7f538bdfb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c206a7ad90 con 0x55c2066a34d0 2024-04-25T11:43:43.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.617+0000 7f538bdfb740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c206959dc0 con 0x55c2066ab6b0 2024-04-25T11:43:43.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.617+0000 7f538bdfb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c206a7af10 con 0x55c2066cc390 2024-04-25T11:43:43.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.617+0000 7f538bdfb740 10 client.0 _ll_register_callbacks cb 0x55c206975600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:43.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.617+0000 7f537dd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066a34d0 0x55c206a782a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.617+0000 7f537dd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066a34d0 0x55c206a782a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41990/0 (socket says 172.21.15.90:41990) 2024-04-25T11:43:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.617+0000 7f537dd9b640 1 -- 172.21.15.90:0/3837226581 learned_addr learned my addr 172.21.15.90:0/3837226581 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.617+0000 7f537e59c640 1 --2- 172.21.15.90:0/3837226581 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c2066ab6b0 0x55c206a787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.617+0000 7f537dd9b640 1 -- 172.21.15.90:0/3837226581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c2066cc390 msgr2=0x55c206a84210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.617+0000 7f537dd9b640 1 --2- 172.21.15.90:0/3837226581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c2066cc390 0x55c206a84210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.617+0000 7f537dd9b640 1 -- 172.21.15.90:0/3837226581 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c2066ab6b0 msgr2=0x55c206a787e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.617+0000 7f537dd9b640 1 --2- 172.21.15.90:0/3837226581 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c2066ab6b0 0x55c206a787e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.617+0000 7f537dd9b640 1 -- 172.21.15.90:0/3837226581 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2066e43f0 con 0x55c2066a34d0 2024-04-25T11:43:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.618+0000 7f537dd9b640 1 --2- 172.21.15.90:0/3837226581 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066a34d0 0x55c206a782a0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f5374005f10 tx=0x7f5374005f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea81094bd1121500 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.618+0000 7f5365ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.618+0000 7f5365ffb640 1 -- 172.21.15.90:0/3837226581 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5374005450 con 0x55c2066a34d0 2024-04-25T11:43:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.618+0000 7f5365ffb640 1 -- 172.21.15.90:0/3837226581 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f53740046c0 con 0x55c2066a34d0 2024-04-25T11:43:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.618+0000 7f538bdfb740 1 -- 172.21.15.90:0/3837226581 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5374011c60 con 0x55c2066a34d0 2024-04-25T11:43:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.618+0000 7f5365ffb640 1 -- 172.21.15.90:0/3837226581 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f53740055f0 con 0x55c2066a34d0 2024-04-25T11:43:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.618+0000 7f538bdfb740 10 client.18258 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:43.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.618+0000 7f538bdfb740 1 -- 172.21.15.90:0/3837226581 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c2067ae7d0 con 0x55c2066a34d0 2024-04-25T11:43:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.618+0000 7f5365ffb640 1 -- 172.21.15.90:0/3837226581 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5374011c60 con 0x55c2066a34d0 2024-04-25T11:43:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.618+0000 7f5365ffb640 1 -- 172.21.15.90:0/3837226581 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f5374031030 con 0x55c2066a34d0 2024-04-25T11:43:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.618+0000 7f538bdfb740 10 client.18258 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.618+0000 7f538bdfb740 10 client.18258 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:43:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.618+0000 7f538bdfb740 20 client.18258 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.618+0000 7f538bdfb740 1 --2- 172.21.15.90:0/3837226581 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55c20668af60 0x55c206a8a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.618+0000 7f538bdfb740 4 client.18258 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:43:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.618+0000 7f538bdfb740 1 -- 172.21.15.90:0/3837226581 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2069e0990 con 0x55c20668af60 2024-04-25T11:43:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.619+0000 7f537e59c640 1 --2- 172.21.15.90:0/3837226581 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55c20668af60 0x55c206a8a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.619+0000 7f537e59c640 1 --2- 172.21.15.90:0/3837226581 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55c20668af60 0x55c206a8a570 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=169618b85609e7a7 server_cookie=e3c36f37586b06a9 in_seq=0 out_seq=0 2024-04-25T11:43:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.619+0000 7f5365ffb640 10 client.18258 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:43:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.620+0000 7f5365ffb640 1 -- 172.21.15.90:0/3837226581 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c2069e0990 con 0x55c20668af60 2024-04-25T11:43:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.620+0000 7f5365ffb640 10 client.18258 handle_command_reply: tid=0 2024-04-25T11:43:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.620+0000 7f538bdfb740 1 client.18258 shutdown 2024-04-25T11:43:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.620+0000 7f538bdfb740 1 -- 172.21.15.90:0/3837226581 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066a34d0 msgr2=0x55c206a782a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.620+0000 7f538bdfb740 1 --2- 172.21.15.90:0/3837226581 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066a34d0 0x55c206a782a0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f5374005f10 tx=0x7f5374005f40 comp rx=0 tx=0).stop 2024-04-25T11:43:43.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.620+0000 7f538bdfb740 1 -- 172.21.15.90:0/3837226581 shutdown_connections 2024-04-25T11:43:43.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.620+0000 7f538bdfb740 1 --2- 172.21.15.90:0/3837226581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c2066cc390 0x55c206a84210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.620+0000 7f538bdfb740 1 --2- 172.21.15.90:0/3837226581 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55c20668af60 0x55c206a8a570 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.620+0000 7f538bdfb740 1 --2- 172.21.15.90:0/3837226581 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c2066ab6b0 0x55c206a787e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.620+0000 7f538bdfb740 1 --2- 172.21.15.90:0/3837226581 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066a34d0 0x55c206a782a0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.620+0000 7f5365ffb640 0 client.18258 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:43:43.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.620+0000 7f538bdfb740 1 -- 172.21.15.90:0/3837226581 >> 172.21.15.90:0/3837226581 conn(0x55c2066c04e0 msgr2=0x55c206a6f530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:43.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.620+0000 7f538bdfb740 1 -- 172.21.15.90:0/3837226581 shutdown_connections 2024-04-25T11:43:43.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.620+0000 7f538bdfb740 1 -- 172.21.15.90:0/3837226581 wait complete. 2024-04-25T11:43:43.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.620+0000 7f538bdfb740 20 client.18258 trim_cache size 0 max 16384 2024-04-25T11:43:43.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.634+0000 7f538bdfb740 1 Processor -- start 2024-04-25T11:43:43.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.634+0000 7f538bdfb740 1 -- start start 2024-04-25T11:43:43.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.634+0000 7f538bdfb740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c2066a34d0 0x55c206a73890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.634+0000 7f538bdfb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066ab6b0 0x55c206a83e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.634+0000 7f538bdfb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c2066cc390 0x55c206a76f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.634+0000 7f538bdfb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f53801afcb0 con 0x55c2066ab6b0 2024-04-25T11:43:43.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.634+0000 7f538bdfb740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f53801b17f0 con 0x55c2066a34d0 2024-04-25T11:43:43.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.634+0000 7f538bdfb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f53801b0380 con 0x55c2066cc390 2024-04-25T11:43:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.634+0000 7f537e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066ab6b0 0x55c206a83e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.634+0000 7f537e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066ab6b0 0x55c206a83e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41994/0 (socket says 172.21.15.90:41994) 2024-04-25T11:43:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.634+0000 7f537e59c640 1 -- 172.21.15.90:0/3828133370 learned_addr learned my addr 172.21.15.90:0/3828133370 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.634+0000 7f537dd9b640 1 --2- 172.21.15.90:0/3828133370 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c2066a34d0 0x55c206a73890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.634+0000 7f537e59c640 1 -- 172.21.15.90:0/3828133370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c2066cc390 msgr2=0x55c206a76f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.634+0000 7f537e59c640 1 --2- 172.21.15.90:0/3828133370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c2066cc390 0x55c206a76f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.634+0000 7f537e59c640 1 -- 172.21.15.90:0/3828133370 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c2066a34d0 msgr2=0x55c206a73890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.634+0000 7f537e59c640 1 --2- 172.21.15.90:0/3828133370 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c2066a34d0 0x55c206a73890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.634+0000 7f537e59c640 1 -- 172.21.15.90:0/3828133370 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c206a8a8e0 con 0x55c2066ab6b0 2024-04-25T11:43:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.635+0000 7f537e59c640 1 --2- 172.21.15.90:0/3828133370 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066ab6b0 0x55c206a83e70 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f5370009900 tx=0x7f5370031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb98e58e17ad6660 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.635+0000 7f5367fff640 1 -- 172.21.15.90:0/3828133370 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5370035040 con 0x55c2066ab6b0 2024-04-25T11:43:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.635+0000 7f5367fff640 1 -- 172.21.15.90:0/3828133370 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53700054f0 con 0x55c2066ab6b0 2024-04-25T11:43:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.635+0000 7f5367fff640 1 -- 172.21.15.90:0/3828133370 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5370004d20 con 0x55c2066ab6b0 2024-04-25T11:43:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.635+0000 7f538bdfb740 1 -- 172.21.15.90:0/3828133370 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066ab6b0 msgr2=0x55c206a83e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.635+0000 7f538bdfb740 1 --2- 172.21.15.90:0/3828133370 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066ab6b0 0x55c206a83e70 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f5370009900 tx=0x7f5370031400 comp rx=0 tx=0).stop 2024-04-25T11:43:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.635+0000 7f538bdfb740 1 -- 172.21.15.90:0/3828133370 shutdown_connections 2024-04-25T11:43:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.635+0000 7f538bdfb740 1 --2- 172.21.15.90:0/3828133370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c2066cc390 0x55c206a76f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.635+0000 7f538bdfb740 1 --2- 172.21.15.90:0/3828133370 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066ab6b0 0x55c206a83e70 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.635+0000 7f538bdfb740 1 --2- 172.21.15.90:0/3828133370 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c2066a34d0 0x55c206a73890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.635+0000 7f538bdfb740 1 -- 172.21.15.90:0/3828133370 >> 172.21.15.90:0/3828133370 conn(0x55c2066c04e0 msgr2=0x55c206a76830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.635+0000 7f538bdfb740 1 -- 172.21.15.90:0/3828133370 shutdown_connections 2024-04-25T11:43:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.636+0000 7f538bdfb740 1 -- 172.21.15.90:0/3828133370 wait complete. 2024-04-25T11:43:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.636+0000 7f538bdfb740 1 -- start start 2024-04-25T11:43:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.636+0000 7f538bdfb740 1 Processor -- start 2024-04-25T11:43:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.636+0000 7f538bdfb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c2066a34d0 0x55c206a7b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.636+0000 7f538bdfb740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c2066ab6b0 0x55c206a7b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.636+0000 7f538bdfb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066cc390 0x55c206af06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.636+0000 7f538bdfb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f53801b1120 con 0x55c2066cc390 2024-04-25T11:43:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.636+0000 7f538bdfb740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f53801b0a50 con 0x55c2066ab6b0 2024-04-25T11:43:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.636+0000 7f538bdfb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f53801adcb0 con 0x55c2066a34d0 2024-04-25T11:43:43.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.636+0000 7f538bdfb740 10 client.0 _ll_register_callbacks cb 0x7f5374009d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:43.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.636+0000 7f537dd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066cc390 0x55c206af06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.636+0000 7f537dd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066cc390 0x55c206af06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42008/0 (socket says 172.21.15.90:42008) 2024-04-25T11:43:43.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.636+0000 7f537dd9b640 1 -- 172.21.15.90:0/1542107018 learned_addr learned my addr 172.21.15.90:0/1542107018 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:43.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.636+0000 7f537e59c640 1 --2- 172.21.15.90:0/1542107018 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c2066ab6b0 0x55c206a7b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.636+0000 7f537dd9b640 1 --2- 172.21.15.90:0/1542107018 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c2066a34d0 0x55c206a7b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.637+0000 7f537dd9b640 1 -- 172.21.15.90:0/1542107018 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c2066a34d0 msgr2=0x55c206a7b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.637+0000 7f537dd9b640 1 --2- 172.21.15.90:0/1542107018 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c2066a34d0 0x55c206a7b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.637+0000 7f537dd9b640 1 -- 172.21.15.90:0/1542107018 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c2066ab6b0 msgr2=0x55c206a7b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.637+0000 7f537dd9b640 1 --2- 172.21.15.90:0/1542107018 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c2066ab6b0 0x55c206a7b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.637+0000 7f537dd9b640 1 -- 172.21.15.90:0/1542107018 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c206a7bf90 con 0x55c2066cc390 2024-04-25T11:43:43.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.637+0000 7f537dd9b640 1 --2- 172.21.15.90:0/1542107018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066cc390 0x55c206af06b0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f537402be50 tx=0x7f5374009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=600266c7fb4b76e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:43.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.637+0000 7f5365ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:43.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.637+0000 7f5365ffb640 1 -- 172.21.15.90:0/1542107018 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5374013240 con 0x55c2066cc390 2024-04-25T11:43:43.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.637+0000 7f5365ffb640 1 -- 172.21.15.90:0/1542107018 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f537402b470 con 0x55c2066cc390 2024-04-25T11:43:43.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.637+0000 7f538bdfb740 1 -- 172.21.15.90:0/1542107018 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53700054f0 con 0x55c2066cc390 2024-04-25T11:43:43.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.637+0000 7f5365ffb640 1 -- 172.21.15.90:0/1542107018 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f537402b670 con 0x55c2066cc390 2024-04-25T11:43:43.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.637+0000 7f538bdfb740 10 client.18264 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:43.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.637+0000 7f538bdfb740 1 -- 172.21.15.90:0/1542107018 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c206af1c40 con 0x55c2066cc390 2024-04-25T11:43:43.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.637+0000 7f5365ffb640 1 -- 172.21.15.90:0/1542107018 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53700054f0 con 0x55c2066cc390 2024-04-25T11:43:43.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.637+0000 7f5365ffb640 1 -- 172.21.15.90:0/1542107018 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f5374031040 con 0x55c2066cc390 2024-04-25T11:43:43.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.637+0000 7f538bdfb740 10 client.18264 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:43.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.637+0000 7f538bdfb740 10 client.18264 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:43:43.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.637+0000 7f538bdfb740 20 client.18264 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:43.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.637+0000 7f538bdfb740 1 --2- 172.21.15.90:0/1542107018 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55c2066603a0 0x55c206af42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.637+0000 7f538bdfb740 4 client.18264 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:43:43.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.637+0000 7f538bdfb740 1 -- 172.21.15.90:0/1542107018 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c2069da800 con 0x55c2066603a0 2024-04-25T11:43:43.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.638+0000 7f537e59c640 1 --2- 172.21.15.90:0/1542107018 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55c2066603a0 0x55c206af42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.638+0000 7f5365ffb640 10 client.18264 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:43:43.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.638+0000 7f537e59c640 1 --2- 172.21.15.90:0/1542107018 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55c2066603a0 0x55c206af42e0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=64f8e2368bb4f712 server_cookie=7fbed8ad160d35ef in_seq=0 out_seq=0 2024-04-25T11:43:43.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.638+0000 7f5365ffb640 1 -- 172.21.15.90:0/1542107018 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c2069da800 con 0x55c2066603a0 2024-04-25T11:43:43.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.638+0000 7f5365ffb640 10 client.18264 handle_command_reply: tid=0 2024-04-25T11:43:43.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.638+0000 7f538bdfb740 1 client.18264 shutdown 2024-04-25T11:43:43.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.638+0000 7f538bdfb740 1 -- 172.21.15.90:0/1542107018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066cc390 msgr2=0x55c206af06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.638+0000 7f538bdfb740 1 --2- 172.21.15.90:0/1542107018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066cc390 0x55c206af06b0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f537402be50 tx=0x7f5374009ec0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.638+0000 7f538bdfb740 1 -- 172.21.15.90:0/1542107018 shutdown_connections 2024-04-25T11:43:43.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.638+0000 7f538bdfb740 1 --2- 172.21.15.90:0/1542107018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c2066cc390 0x55c206af06b0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.638+0000 7f538bdfb740 1 --2- 172.21.15.90:0/1542107018 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55c2066603a0 0x55c206af42e0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.638+0000 7f538bdfb740 1 --2- 172.21.15.90:0/1542107018 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c2066ab6b0 0x55c206a7b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.638+0000 7f538bdfb740 1 --2- 172.21.15.90:0/1542107018 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c2066a34d0 0x55c206a7b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.638+0000 7f5365ffb640 0 client.18264 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:43:43.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.638+0000 7f538bdfb740 1 -- 172.21.15.90:0/1542107018 >> 172.21.15.90:0/1542107018 conn(0x55c2066c04e0 msgr2=0x55c206a79740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:43.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.638+0000 7f538bdfb740 1 -- 172.21.15.90:0/1542107018 shutdown_connections 2024-04-25T11:43:43.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.638+0000 7f538bdfb740 1 -- 172.21.15.90:0/1542107018 wait complete. 2024-04-25T11:43:43.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.639+0000 7f538bdfb740 20 client.18264 trim_cache size 0 max 16384 2024-04-25T11:43:43.632 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:43:43.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.639+0000 7f5384e67640 1 -- 172.21.15.90:0/1368577238 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f53680afc60 msgr2=0x7f53680b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.639+0000 7f5384e67640 1 --2- 172.21.15.90:0/1368577238 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f53680afc60 0x7f53680b2170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f53780dc5e0 tx=0x7f537804e040 comp rx=0 tx=0).stop 2024-04-25T11:43:43.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.639+0000 7f5384e67640 1 -- 172.21.15.90:0/1368577238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5380109fc0 msgr2=0x7f538019eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.639+0000 7f5384e67640 1 --2- 172.21.15.90:0/1368577238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5380109fc0 0x7f538019eff0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f5378009fd0 tx=0x7f537800b260 comp rx=0 tx=0).stop 2024-04-25T11:43:43.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.639+0000 7f5384e67640 1 -- 172.21.15.90:0/1368577238 shutdown_connections 2024-04-25T11:43:43.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.639+0000 7f5384e67640 1 --2- 172.21.15.90:0/1368577238 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f53680afc60 0x7f53680b2170 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.639+0000 7f5384e67640 1 --2- 172.21.15.90:0/1368577238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5380109fc0 0x7f538019eff0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.639+0000 7f5384e67640 1 --2- 172.21.15.90:0/1368577238 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5380109b20 0x7f538019ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.639+0000 7f5384e67640 1 --2- 172.21.15.90:0/1368577238 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5380104270 0x7f538019a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.639+0000 7f5384e67640 1 -- 172.21.15.90:0/1368577238 >> 172.21.15.90:0/1368577238 conn(0x7f53800ff1e0 msgr2=0x7f53800ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:43.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.639+0000 7f5384e67640 1 -- 172.21.15.90:0/1368577238 shutdown_connections 2024-04-25T11:43:43.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.639+0000 7f5384e67640 1 -- 172.21.15.90:0/1368577238 wait complete. 2024-04-25T11:43:43.637 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T11:43:43.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.760+0000 7fac33986640 1 -- 172.21.15.90:0/168659351 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac2c102440 msgr2=0x7fac2c102850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.760+0000 7fac33986640 1 --2- 172.21.15.90:0/168659351 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac2c102440 0x7fac2c102850 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fac24009ca0 tx=0x7fac2402a8c0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.760+0000 7fac33986640 1 -- 172.21.15.90:0/168659351 shutdown_connections 2024-04-25T11:43:43.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.760+0000 7fac33986640 1 --2- 172.21.15.90:0/168659351 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac2c107440 0x7fac2c10ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.760+0000 7fac33986640 1 --2- 172.21.15.90:0/168659351 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fac2c106980 0x7fac2c106db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.760+0000 7fac33986640 1 --2- 172.21.15.90:0/168659351 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac2c102440 0x7fac2c102850 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.760+0000 7fac33986640 1 -- 172.21.15.90:0/168659351 >> 172.21.15.90:0/168659351 conn(0x7fac2c077880 msgr2=0x7fac2c077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:43.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.760+0000 7fac33986640 1 -- 172.21.15.90:0/168659351 shutdown_connections 2024-04-25T11:43:43.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.760+0000 7fac33986640 1 -- 172.21.15.90:0/168659351 wait complete. 2024-04-25T11:43:43.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.761+0000 7fac33986640 1 Processor -- start 2024-04-25T11:43:43.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.761+0000 7fac33986640 1 -- start start 2024-04-25T11:43:43.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.761+0000 7fac33986640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fac2c102440 0x7fac2c19a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.761+0000 7fac33986640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac2c106980 0x7fac2c19ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.761+0000 7fac33986640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac2c107440 0x7fac2c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.761+0000 7fac33986640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fac2c113d60 con 0x7fac2c106980 2024-04-25T11:43:43.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.761+0000 7fac33986640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fac2c113be0 con 0x7fac2c102440 2024-04-25T11:43:43.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.761+0000 7fac33986640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fac2c113ee0 con 0x7fac2c107440 2024-04-25T11:43:43.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.761+0000 7fac30f21640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac2c106980 0x7fac2c19ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.761+0000 7fac30f21640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac2c106980 0x7fac2c19ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42018/0 (socket says 172.21.15.90:42018) 2024-04-25T11:43:43.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.761+0000 7fac30f21640 1 -- 172.21.15.90:0/3216833618 learned_addr learned my addr 172.21.15.90:0/3216833618 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:43.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.761+0000 7fac31722640 1 --2- 172.21.15.90:0/3216833618 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fac2c102440 0x7fac2c19a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.762+0000 7fac31f23640 1 --2- 172.21.15.90:0/3216833618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac2c107440 0x7fac2c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.762+0000 7fac30f21640 1 -- 172.21.15.90:0/3216833618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac2c107440 msgr2=0x7fac2c19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.762+0000 7fac30f21640 1 --2- 172.21.15.90:0/3216833618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac2c107440 0x7fac2c19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.762+0000 7fac30f21640 1 -- 172.21.15.90:0/3216833618 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fac2c102440 msgr2=0x7fac2c19a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.762+0000 7fac30f21640 1 --2- 172.21.15.90:0/3216833618 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fac2c102440 0x7fac2c19a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.762+0000 7fac30f21640 1 -- 172.21.15.90:0/3216833618 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac2c19f620 con 0x7fac2c106980 2024-04-25T11:43:43.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.762+0000 7fac31722640 1 --2- 172.21.15.90:0/3216833618 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fac2c102440 0x7fac2c19a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:43.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.762+0000 7fac31f23640 1 --2- 172.21.15.90:0/3216833618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac2c107440 0x7fac2c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:43:43.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.762+0000 7fac30f21640 1 --2- 172.21.15.90:0/3216833618 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac2c106980 0x7fac2c19ad20 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fac1c00b450 tx=0x7fac1c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:43.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.762+0000 7fac227fc640 1 -- 172.21.15.90:0/3216833618 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1c015020 con 0x7fac2c106980 2024-04-25T11:43:43.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.762+0000 7fac227fc640 1 -- 172.21.15.90:0/3216833618 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac1c011d40 con 0x7fac2c106980 2024-04-25T11:43:43.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.762+0000 7fac33986640 1 -- 172.21.15.90:0/3216833618 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac2c19f910 con 0x7fac2c106980 2024-04-25T11:43:43.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.762+0000 7fac227fc640 1 -- 172.21.15.90:0/3216833618 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1c00f340 con 0x7fac2c106980 2024-04-25T11:43:43.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.762+0000 7fac33986640 1 -- 172.21.15.90:0/3216833618 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac2c1a7290 con 0x7fac2c106980 2024-04-25T11:43:43.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.765+0000 7fac3a83f740 1 Processor -- start 2024-04-25T11:43:43.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.765+0000 7fac3a83f740 1 -- start start 2024-04-25T11:43:43.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.765+0000 7fac227fc640 1 -- 172.21.15.90:0/3216833618 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fac1c018020 con 0x7fac2c106980 2024-04-25T11:43:43.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.765+0000 7fac3a83f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56344ea5f390 0x56344ee006c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.765+0000 7fac3a83f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56344ea364d0 0x56344ee00c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.765+0000 7fac3a83f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea3e6b0 0x56344ee11ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.765+0000 7fac3a83f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56344ed65d80 con 0x56344ea3e6b0 2024-04-25T11:43:43.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.765+0000 7fac3a83f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56344ed1fa20 con 0x56344ea5f390 2024-04-25T11:43:43.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.765+0000 7fac3a83f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56344ed61d40 con 0x56344ea364d0 2024-04-25T11:43:43.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.765+0000 7fac30f21640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea3e6b0 0x56344ee11ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.765+0000 7fac30f21640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea3e6b0 0x56344ee11ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42030/0 (socket says 172.21.15.90:42030) 2024-04-25T11:43:43.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.765+0000 7fac30f21640 1 -- 172.21.15.90:0/541291158 learned_addr learned my addr 172.21.15.90:0/541291158 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:43.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.765+0000 7fac31f23640 1 --2- 172.21.15.90:0/541291158 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56344ea5f390 0x56344ee006c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.765+0000 7fac31722640 1 --2- 172.21.15.90:0/541291158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56344ea364d0 0x56344ee00c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.765+0000 7fac30f21640 1 -- 172.21.15.90:0/541291158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56344ea364d0 msgr2=0x56344ee00c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.765+0000 7fac30f21640 1 --2- 172.21.15.90:0/541291158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56344ea364d0 0x56344ee00c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.765+0000 7fac30f21640 1 -- 172.21.15.90:0/541291158 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56344ea5f390 msgr2=0x56344ee006c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.765+0000 7fac30f21640 1 --2- 172.21.15.90:0/541291158 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56344ea5f390 0x56344ee006c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.765+0000 7fac30f21640 1 -- 172.21.15.90:0/541291158 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56344ed2ac30 con 0x56344ea3e6b0 2024-04-25T11:43:43.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.765+0000 7fac31f23640 1 --2- 172.21.15.90:0/541291158 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56344ea5f390 0x56344ee006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:43.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.765+0000 7fac227fc640 1 --2- 172.21.15.90:0/3216833618 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fac180afc60 0x7fac180b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.766+0000 7fac30f21640 1 --2- 172.21.15.90:0/541291158 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea3e6b0 0x56344ee11ab0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fac1c0dc270 tx=0x7fac1c0e7670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bfa7e98653151fe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:43.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.766+0000 7fac31f23640 1 --2- 172.21.15.90:0/3216833618 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fac180afc60 0x7fac180b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.766+0000 7fac227fc640 1 -- 172.21.15.90:0/3216833618 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7fac1c0deb30 con 0x7fac2c106980 2024-04-25T11:43:43.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.766+0000 7fac177fe640 1 -- 172.21.15.90:0/541291158 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1c0a2c10 con 0x56344ea3e6b0 2024-04-25T11:43:43.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.766+0000 7fac177fe640 1 -- 172.21.15.90:0/541291158 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac1c0a6860 con 0x56344ea3e6b0 2024-04-25T11:43:43.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.766+0000 7fac177fe640 1 -- 172.21.15.90:0/541291158 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1c0a64c0 con 0x56344ea3e6b0 2024-04-25T11:43:43.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.767+0000 7fac3a83f740 1 -- 172.21.15.90:0/541291158 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea3e6b0 msgr2=0x56344ee11ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.767+0000 7fac3a83f740 1 --2- 172.21.15.90:0/541291158 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea3e6b0 0x56344ee11ab0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fac1c0dc270 tx=0x7fac1c0e7670 comp rx=0 tx=0).stop 2024-04-25T11:43:43.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.767+0000 7fac31f23640 1 --2- 172.21.15.90:0/3216833618 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fac180afc60 0x7fac180b2170 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fac28005fd0 tx=0x7fac28009350 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:43.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.767+0000 7fac3a83f740 1 -- 172.21.15.90:0/541291158 shutdown_connections 2024-04-25T11:43:43.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.767+0000 7fac3a83f740 1 --2- 172.21.15.90:0/541291158 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea3e6b0 0x56344ee11ab0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.767+0000 7fac3a83f740 1 --2- 172.21.15.90:0/541291158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56344ea364d0 0x56344ee00c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.767+0000 7fac3a83f740 1 --2- 172.21.15.90:0/541291158 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56344ea5f390 0x56344ee006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.767+0000 7fac3a83f740 1 -- 172.21.15.90:0/541291158 >> 172.21.15.90:0/541291158 conn(0x56344ea534e0 msgr2=0x56344edfcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:43.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.767+0000 7fac3a83f740 1 -- 172.21.15.90:0/541291158 shutdown_connections 2024-04-25T11:43:43.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.767+0000 7fac3a83f740 1 -- 172.21.15.90:0/541291158 wait complete. 2024-04-25T11:43:43.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.767+0000 7fac3a83f740 1 -- start start 2024-04-25T11:43:43.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.767+0000 7fac3a83f740 1 Processor -- start 2024-04-25T11:43:43.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.768+0000 7fac3a83f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56344ea364d0 0x56344ee0f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.768+0000 7fac3a83f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56344ea3e6b0 0x56344ee0fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.768+0000 7fac3a83f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea5f390 0x56344ee1b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.768+0000 7fac3a83f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56344ee121a0 con 0x56344ea5f390 2024-04-25T11:43:43.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.768+0000 7fac3a83f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56344ececdc0 con 0x56344ea364d0 2024-04-25T11:43:43.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.768+0000 7fac3a83f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56344ee12320 con 0x56344ea3e6b0 2024-04-25T11:43:43.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.768+0000 7fac3a83f740 10 client.0 _ll_register_callbacks cb 0x56344ed08600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:43.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.768+0000 7fac31f23640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea5f390 0x56344ee1b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.768+0000 7fac31f23640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea5f390 0x56344ee1b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42034/0 (socket says 172.21.15.90:42034) 2024-04-25T11:43:43.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.768+0000 7fac31f23640 1 -- 172.21.15.90:0/565729079 learned_addr learned my addr 172.21.15.90:0/565729079 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:43.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.768+0000 7fac31722640 1 --2- 172.21.15.90:0/565729079 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56344ea364d0 0x56344ee0f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.768+0000 7fac30f21640 1 --2- 172.21.15.90:0/565729079 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56344ea3e6b0 0x56344ee0fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.768+0000 7fac31f23640 1 -- 172.21.15.90:0/565729079 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56344ea3e6b0 msgr2=0x56344ee0fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.768+0000 7fac31f23640 1 --2- 172.21.15.90:0/565729079 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56344ea3e6b0 0x56344ee0fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.768+0000 7fac31f23640 1 -- 172.21.15.90:0/565729079 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56344ea364d0 msgr2=0x56344ee0f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.768+0000 7fac31f23640 1 --2- 172.21.15.90:0/565729079 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56344ea364d0 0x56344ee0f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.768+0000 7fac31f23640 1 -- 172.21.15.90:0/565729079 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56344ea773f0 con 0x56344ea5f390 2024-04-25T11:43:43.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.768+0000 7fac31f23640 1 --2- 172.21.15.90:0/565729079 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea5f390 0x56344ee1b6e0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fac280126d0 tx=0x7fac28004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2436fb8a9ad0c96 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:43.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.768+0000 7fac14ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:43.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.768+0000 7fac14ff9640 1 -- 172.21.15.90:0/565729079 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac28023070 con 0x56344ea5f390 2024-04-25T11:43:43.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.768+0000 7fac14ff9640 1 -- 172.21.15.90:0/565729079 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7fac280118b0 con 0x56344ea5f390 2024-04-25T11:43:43.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.768+0000 7fac3a83f740 1 -- 172.21.15.90:0/565729079 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac1c0a6860 con 0x56344ea5f390 2024-04-25T11:43:43.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.769+0000 7fac14ff9640 1 -- 172.21.15.90:0/565729079 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fac2802f050 con 0x56344ea5f390 2024-04-25T11:43:43.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.769+0000 7fac3a83f740 10 client.18276 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:43.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.769+0000 7fac3a83f740 1 -- 172.21.15.90:0/565729079 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56344eb417d0 con 0x56344ea5f390 2024-04-25T11:43:43.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.769+0000 7fac14ff9640 1 -- 172.21.15.90:0/565729079 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac1c0a6860 con 0x56344ea5f390 2024-04-25T11:43:43.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.769+0000 7fac14ff9640 1 -- 172.21.15.90:0/565729079 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fac2800f040 con 0x56344ea5f390 2024-04-25T11:43:43.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.769+0000 7fac3a83f740 10 client.18276 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:43.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.769+0000 7fac3a83f740 10 client.18276 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:43:43.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.769+0000 7fac3a83f740 20 client.18276 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:43.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.769+0000 7fac3a83f740 1 --2- 172.21.15.90:0/565729079 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x56344ea1df60 0x56344ee21b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.769+0000 7fac3a83f740 4 client.18276 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:43:43.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.769+0000 7fac3a83f740 1 -- 172.21.15.90:0/565729079 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56344ed73990 con 0x56344ea1df60 2024-04-25T11:43:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.769+0000 7fac31722640 1 --2- 172.21.15.90:0/565729079 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x56344ea1df60 0x56344ee21b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.770+0000 7fac31722640 1 --2- 172.21.15.90:0/565729079 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x56344ea1df60 0x56344ee21b40 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d0d8f61ee4ac68ad server_cookie=c53674b4df797474 in_seq=0 out_seq=0 2024-04-25T11:43:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.770+0000 7fac14ff9640 10 client.18276 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:43:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.771+0000 7fac14ff9640 1 -- 172.21.15.90:0/565729079 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56344ed73990 con 0x56344ea1df60 2024-04-25T11:43:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.771+0000 7fac14ff9640 10 client.18276 handle_command_reply: tid=0 2024-04-25T11:43:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.771+0000 7fac3a83f740 1 client.18276 shutdown 2024-04-25T11:43:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.771+0000 7fac3a83f740 1 -- 172.21.15.90:0/565729079 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea5f390 msgr2=0x56344ee1b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.771+0000 7fac3a83f740 1 --2- 172.21.15.90:0/565729079 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea5f390 0x56344ee1b6e0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fac280126d0 tx=0x7fac28004730 comp rx=0 tx=0).stop 2024-04-25T11:43:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.771+0000 7fac3a83f740 1 -- 172.21.15.90:0/565729079 shutdown_connections 2024-04-25T11:43:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.771+0000 7fac3a83f740 1 --2- 172.21.15.90:0/565729079 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x56344ea1df60 0x56344ee21b40 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.771+0000 7fac3a83f740 1 --2- 172.21.15.90:0/565729079 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea5f390 0x56344ee1b6e0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.771+0000 7fac3a83f740 1 --2- 172.21.15.90:0/565729079 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56344ea3e6b0 0x56344ee0fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.771+0000 7fac14ff9640 0 client.18276 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:43:43.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.771+0000 7fac3a83f740 1 --2- 172.21.15.90:0/565729079 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56344ea364d0 0x56344ee0f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.771+0000 7fac3a83f740 1 -- 172.21.15.90:0/565729079 >> 172.21.15.90:0/565729079 conn(0x56344ea534e0 msgr2=0x56344ee024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:43.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.771+0000 7fac3a83f740 1 -- 172.21.15.90:0/565729079 shutdown_connections 2024-04-25T11:43:43.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.771+0000 7fac3a83f740 1 -- 172.21.15.90:0/565729079 wait complete. 2024-04-25T11:43:43.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.771+0000 7fac3a83f740 20 client.18276 trim_cache size 0 max 16384 2024-04-25T11:43:43.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.784+0000 7fac3a83f740 1 Processor -- start 2024-04-25T11:43:43.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.784+0000 7fac3a83f740 1 -- start start 2024-04-25T11:43:43.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.785+0000 7fac3a83f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea364d0 0x56344ee1b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.785+0000 7fac3a83f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56344ea3e6b0 0x56344ee27300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.785+0000 7fac3a83f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56344ea5f390 0x56344ee27840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.785+0000 7fac3a83f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fac2c1adb10 con 0x56344ea364d0 2024-04-25T11:43:43.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.785+0000 7fac3a83f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fac2c1af650 con 0x56344ea5f390 2024-04-25T11:43:43.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.785+0000 7fac3a83f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fac2c1ae1e0 con 0x56344ea3e6b0 2024-04-25T11:43:43.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.785+0000 7fac31722640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea364d0 0x56344ee1b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.785+0000 7fac31722640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea364d0 0x56344ee1b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42048/0 (socket says 172.21.15.90:42048) 2024-04-25T11:43:43.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.785+0000 7fac31722640 1 -- 172.21.15.90:0/3128542625 learned_addr learned my addr 172.21.15.90:0/3128542625 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:43.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.785+0000 7fac31f23640 1 --2- 172.21.15.90:0/3128542625 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56344ea5f390 0x56344ee27840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.785+0000 7fac31722640 1 -- 172.21.15.90:0/3128542625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56344ea3e6b0 msgr2=0x56344ee27300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.785+0000 7fac31722640 1 --2- 172.21.15.90:0/3128542625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56344ea3e6b0 0x56344ee27300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.785+0000 7fac31722640 1 -- 172.21.15.90:0/3128542625 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56344ea5f390 msgr2=0x56344ee27840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.785+0000 7fac31722640 1 --2- 172.21.15.90:0/3128542625 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56344ea5f390 0x56344ee27840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.785+0000 7fac31722640 1 -- 172.21.15.90:0/3128542625 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56344ee0d9a0 con 0x56344ea364d0 2024-04-25T11:43:43.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.785+0000 7fac31722640 1 --2- 172.21.15.90:0/3128542625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea364d0 0x56344ee1b2c0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fac2402ade0 tx=0x7fac24031e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dad991ac987627d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:43.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.785+0000 7fac16ffd640 1 -- 172.21.15.90:0/3128542625 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac24031650 con 0x56344ea364d0 2024-04-25T11:43:43.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.785+0000 7fac16ffd640 1 -- 172.21.15.90:0/3128542625 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac24036040 con 0x56344ea364d0 2024-04-25T11:43:43.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.785+0000 7fac16ffd640 1 -- 172.21.15.90:0/3128542625 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac24004460 con 0x56344ea364d0 2024-04-25T11:43:43.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.786+0000 7fac3a83f740 1 -- 172.21.15.90:0/3128542625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea364d0 msgr2=0x56344ee1b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.786+0000 7fac3a83f740 1 --2- 172.21.15.90:0/3128542625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea364d0 0x56344ee1b2c0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fac2402ade0 tx=0x7fac24031e40 comp rx=0 tx=0).stop 2024-04-25T11:43:43.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.786+0000 7fac3a83f740 1 -- 172.21.15.90:0/3128542625 shutdown_connections 2024-04-25T11:43:43.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.786+0000 7fac3a83f740 1 --2- 172.21.15.90:0/3128542625 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56344ea5f390 0x56344ee27840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.786+0000 7fac3a83f740 1 --2- 172.21.15.90:0/3128542625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56344ea3e6b0 0x56344ee27300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.786+0000 7fac3a83f740 1 --2- 172.21.15.90:0/3128542625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea364d0 0x56344ee1b2c0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.786+0000 7fac3a83f740 1 -- 172.21.15.90:0/3128542625 >> 172.21.15.90:0/3128542625 conn(0x56344ea534e0 msgr2=0x56344ee0dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:43.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.786+0000 7fac3a83f740 1 -- 172.21.15.90:0/3128542625 shutdown_connections 2024-04-25T11:43:43.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.786+0000 7fac3a83f740 1 -- 172.21.15.90:0/3128542625 wait complete. 2024-04-25T11:43:43.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.787+0000 7fac3a83f740 1 -- start start 2024-04-25T11:43:43.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.787+0000 7fac3a83f740 1 Processor -- start 2024-04-25T11:43:43.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.787+0000 7fac3a83f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea364d0 0x56344ee13140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.787+0000 7fac3a83f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56344ea3e6b0 0x56344ee13680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.787+0000 7fac3a83f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56344ea5f390 0x56344ee13bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.787+0000 7fac3a83f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fac2c1aef80 con 0x56344ea364d0 2024-04-25T11:43:43.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.787+0000 7fac3a83f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fac2c1ae8b0 con 0x56344ea5f390 2024-04-25T11:43:43.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.787+0000 7fac3a83f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fac2c1abb30 con 0x56344ea3e6b0 2024-04-25T11:43:43.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.787+0000 7fac3a83f740 10 client.0 _ll_register_callbacks cb 0x7fac28016070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:43.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.787+0000 7fac31722640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea364d0 0x56344ee13140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.787+0000 7fac31722640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea364d0 0x56344ee13140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42058/0 (socket says 172.21.15.90:42058) 2024-04-25T11:43:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.787+0000 7fac31722640 1 -- 172.21.15.90:0/3501231746 learned_addr learned my addr 172.21.15.90:0/3501231746 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.787+0000 7fac31f23640 1 --2- 172.21.15.90:0/3501231746 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56344ea5f390 0x56344ee13bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.787+0000 7fac31722640 1 -- 172.21.15.90:0/3501231746 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56344ea3e6b0 msgr2=0x56344ee13680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.787+0000 7fac31722640 1 --2- 172.21.15.90:0/3501231746 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56344ea3e6b0 0x56344ee13680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.787+0000 7fac31722640 1 -- 172.21.15.90:0/3501231746 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56344ea5f390 msgr2=0x56344ee13bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.787+0000 7fac31722640 1 --2- 172.21.15.90:0/3501231746 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56344ea5f390 0x56344ee13bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.787+0000 7fac31722640 1 -- 172.21.15.90:0/3501231746 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56344ee88eb0 con 0x56344ea364d0 2024-04-25T11:43:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.788+0000 7fac31722640 1 --2- 172.21.15.90:0/3501231746 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea364d0 0x56344ee13140 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fac24009a60 tx=0x7fac24031500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfa4bd77729660ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.788+0000 7fac14ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.788+0000 7fac14ff9640 1 -- 172.21.15.90:0/3501231746 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac24003970 con 0x56344ea364d0 2024-04-25T11:43:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.788+0000 7fac14ff9640 1 -- 172.21.15.90:0/3501231746 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7fac24003b10 con 0x56344ea364d0 2024-04-25T11:43:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.788+0000 7fac3a83f740 1 -- 172.21.15.90:0/3501231746 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac24036040 con 0x56344ea364d0 2024-04-25T11:43:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.788+0000 7fac14ff9640 1 -- 172.21.15.90:0/3501231746 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fac24003d10 con 0x56344ea364d0 2024-04-25T11:43:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.788+0000 7fac3a83f740 10 client.18282 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.788+0000 7fac3a83f740 1 -- 172.21.15.90:0/3501231746 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56344ee89f70 con 0x56344ea364d0 2024-04-25T11:43:43.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.788+0000 7fac14ff9640 1 -- 172.21.15.90:0/3501231746 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac24036040 con 0x56344ea364d0 2024-04-25T11:43:43.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.788+0000 7fac14ff9640 1 -- 172.21.15.90:0/3501231746 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fac2402f8c0 con 0x56344ea364d0 2024-04-25T11:43:43.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.788+0000 7fac3a83f740 10 client.18282 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:43.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.788+0000 7fac3a83f740 10 client.18282 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:43:43.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.788+0000 7fac3a83f740 20 client.18282 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:43.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.788+0000 7fac3a83f740 1 --2- 172.21.15.90:0/3501231746 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x56344e9f33a0 0x56344ee8c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.788+0000 7fac3a83f740 4 client.18282 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:43:43.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.788+0000 7fac3a83f740 1 -- 172.21.15.90:0/3501231746 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56344ed6d800 con 0x56344e9f33a0 2024-04-25T11:43:43.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.789+0000 7fac30f21640 1 --2- 172.21.15.90:0/3501231746 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x56344e9f33a0 0x56344ee8c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.789+0000 7fac14ff9640 10 client.18282 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:43:43.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.789+0000 7fac30f21640 1 --2- 172.21.15.90:0/3501231746 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x56344e9f33a0 0x56344ee8c6c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9f27806cb825554e server_cookie=1733e8f345364a57 in_seq=0 out_seq=0 2024-04-25T11:43:43.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.789+0000 7fac14ff9640 1 -- 172.21.15.90:0/3501231746 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56344ed6d800 con 0x56344e9f33a0 2024-04-25T11:43:43.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.789+0000 7fac14ff9640 10 client.18282 handle_command_reply: tid=0 2024-04-25T11:43:43.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.789+0000 7fac3a83f740 1 client.18282 shutdown 2024-04-25T11:43:43.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.789+0000 7fac3a83f740 1 -- 172.21.15.90:0/3501231746 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea364d0 msgr2=0x56344ee13140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.789+0000 7fac3a83f740 1 --2- 172.21.15.90:0/3501231746 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea364d0 0x56344ee13140 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fac24009a60 tx=0x7fac24031500 comp rx=0 tx=0).stop 2024-04-25T11:43:43.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.789+0000 7fac3a83f740 1 -- 172.21.15.90:0/3501231746 shutdown_connections 2024-04-25T11:43:43.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.789+0000 7fac3a83f740 1 --2- 172.21.15.90:0/3501231746 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x56344e9f33a0 0x56344ee8c6c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.789+0000 7fac3a83f740 1 --2- 172.21.15.90:0/3501231746 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56344ea5f390 0x56344ee13bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.789+0000 7fac3a83f740 1 --2- 172.21.15.90:0/3501231746 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56344ea3e6b0 0x56344ee13680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.789+0000 7fac14ff9640 0 client.18282 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:43:43.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.789+0000 7fac3a83f740 1 --2- 172.21.15.90:0/3501231746 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56344ea364d0 0x56344ee13140 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.789+0000 7fac3a83f740 1 -- 172.21.15.90:0/3501231746 >> 172.21.15.90:0/3501231746 conn(0x56344ea534e0 msgr2=0x56344ee02410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:43.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.789+0000 7fac3a83f740 1 -- 172.21.15.90:0/3501231746 shutdown_connections 2024-04-25T11:43:43.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.790+0000 7fac3a83f740 1 -- 172.21.15.90:0/3501231746 wait complete. 2024-04-25T11:43:43.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.790+0000 7fac3a83f740 20 client.18282 trim_cache size 0 max 16384 2024-04-25T11:43:43.783 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:43:43.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.790+0000 7fac33986640 1 -- 172.21.15.90:0/3216833618 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fac180afc60 msgr2=0x7fac180b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.790+0000 7fac33986640 1 --2- 172.21.15.90:0/3216833618 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fac180afc60 0x7fac180b2170 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fac28005fd0 tx=0x7fac28009350 comp rx=0 tx=0).stop 2024-04-25T11:43:43.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.790+0000 7fac33986640 1 -- 172.21.15.90:0/3216833618 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac2c106980 msgr2=0x7fac2c19ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.790+0000 7fac33986640 1 --2- 172.21.15.90:0/3216833618 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac2c106980 0x7fac2c19ad20 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fac1c00b450 tx=0x7fac1c00b910 comp rx=0 tx=0).stop 2024-04-25T11:43:43.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.790+0000 7fac33986640 1 -- 172.21.15.90:0/3216833618 shutdown_connections 2024-04-25T11:43:43.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.790+0000 7fac33986640 1 --2- 172.21.15.90:0/3216833618 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fac180afc60 0x7fac180b2170 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.790+0000 7fac33986640 1 --2- 172.21.15.90:0/3216833618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac2c107440 0x7fac2c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.790+0000 7fac33986640 1 --2- 172.21.15.90:0/3216833618 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac2c106980 0x7fac2c19ad20 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.790+0000 7fac33986640 1 --2- 172.21.15.90:0/3216833618 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fac2c102440 0x7fac2c19a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.790+0000 7fac33986640 1 -- 172.21.15.90:0/3216833618 >> 172.21.15.90:0/3216833618 conn(0x7fac2c077880 msgr2=0x7fac2c100980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:43.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.790+0000 7fac33986640 1 -- 172.21.15.90:0/3216833618 shutdown_connections 2024-04-25T11:43:43.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.790+0000 7fac33986640 1 -- 172.21.15.90:0/3216833618 wait complete. 2024-04-25T11:43:43.788 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T11:43:43.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.921+0000 7f0f297ee640 1 -- 172.21.15.90:0/2516170764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f2410e570 msgr2=0x7f0f24110990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.921+0000 7f0f297ee640 1 --2- 172.21.15.90:0/2516170764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f2410e570 0x7f0f24110990 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c009ca0 tx=0x7f0f1c02a8c0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.921+0000 7f0f297ee640 1 -- 172.21.15.90:0/2516170764 shutdown_connections 2024-04-25T11:43:43.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.921+0000 7f0f297ee640 1 --2- 172.21.15.90:0/2516170764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f2410e570 0x7f0f24110990 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.921+0000 7f0f297ee640 1 --2- 172.21.15.90:0/2516170764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f24102070 0x7f0f24102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.922+0000 7f0f297ee640 1 --2- 172.21.15.90:0/2516170764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0f24109bc0 0x7f0f24101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.922+0000 7f0f297ee640 1 -- 172.21.15.90:0/2516170764 >> 172.21.15.90:0/2516170764 conn(0x7f0f24077920 msgr2=0x7f0f24077d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:43.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.922+0000 7f0f297ee640 1 -- 172.21.15.90:0/2516170764 shutdown_connections 2024-04-25T11:43:43.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.922+0000 7f0f297ee640 1 -- 172.21.15.90:0/2516170764 wait complete. 2024-04-25T11:43:43.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.922+0000 7f0f297ee640 1 Processor -- start 2024-04-25T11:43:43.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.922+0000 7f0f297ee640 1 -- start start 2024-04-25T11:43:43.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.923+0000 7f0f297ee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f24102070 0x7f0f24102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.923+0000 7f0f297ee640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f24109bc0 0x7f0f24103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.923+0000 7f0f297ee640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0f2410e570 0x7f0f24103790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.923+0000 7f0f297ee640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0f24113d30 con 0x7f0f24109bc0 2024-04-25T11:43:43.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.923+0000 7f0f297ee640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0f24113bb0 con 0x7f0f2410e570 2024-04-25T11:43:43.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.923+0000 7f0f297ee640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0f24113eb0 con 0x7f0f24102070 2024-04-25T11:43:43.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.923+0000 7f0f227fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f24109bc0 0x7f0f24103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.923+0000 7f0f227fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f24109bc0 0x7f0f24103250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42070/0 (socket says 172.21.15.90:42070) 2024-04-25T11:43:43.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.923+0000 7f0f227fc640 1 -- 172.21.15.90:0/636322988 learned_addr learned my addr 172.21.15.90:0/636322988 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:43.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.923+0000 7f0f237fe640 1 --2- 172.21.15.90:0/636322988 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0f2410e570 0x7f0f24103790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.923+0000 7f0f22ffd640 1 --2- 172.21.15.90:0/636322988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f24102070 0x7f0f24102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.923+0000 7f0f227fc640 1 -- 172.21.15.90:0/636322988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f24102070 msgr2=0x7f0f24102d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.923+0000 7f0f227fc640 1 --2- 172.21.15.90:0/636322988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f24102070 0x7f0f24102d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.923+0000 7f0f227fc640 1 -- 172.21.15.90:0/636322988 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0f2410e570 msgr2=0x7f0f24103790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.923+0000 7f0f227fc640 1 --2- 172.21.15.90:0/636322988 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0f2410e570 0x7f0f24103790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.923+0000 7f0f227fc640 1 -- 172.21.15.90:0/636322988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f241a8a50 con 0x7f0f24109bc0 2024-04-25T11:43:43.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.923+0000 7f0f227fc640 1 --2- 172.21.15.90:0/636322988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f24109bc0 0x7f0f24103250 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f0f18005fd0 tx=0x7f0f1800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:43.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.923+0000 7f0f0bfff640 1 -- 172.21.15.90:0/636322988 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f18017070 con 0x7f0f24109bc0 2024-04-25T11:43:43.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.923+0000 7f0f297ee640 1 -- 172.21.15.90:0/636322988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f241a8d40 con 0x7f0f24109bc0 2024-04-25T11:43:43.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.923+0000 7f0f0bfff640 1 -- 172.21.15.90:0/636322988 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f18009c30 con 0x7f0f24109bc0 2024-04-25T11:43:43.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.923+0000 7f0f0bfff640 1 -- 172.21.15.90:0/636322988 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f18003d30 con 0x7f0f24109bc0 2024-04-25T11:43:43.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.923+0000 7f0f297ee640 1 -- 172.21.15.90:0/636322988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f241a9220 con 0x7f0f24109bc0 2024-04-25T11:43:43.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.926+0000 7f0f0bfff640 1 -- 172.21.15.90:0/636322988 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f0f18003ed0 con 0x7f0f24109bc0 2024-04-25T11:43:43.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.926+0000 7f0f30773740 1 Processor -- start 2024-04-25T11:43:43.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.926+0000 7f0f30773740 1 -- start start 2024-04-25T11:43:43.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.926+0000 7f0f30773740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cab7884390 0x55cab7c25810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.926+0000 7f0f30773740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cab785b4d0 0x55cab7c25d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.926+0000 7f0f30773740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab78636b0 0x55cab7c326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.926+0000 7f0f30773740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55cab7b8ad80 con 0x55cab78636b0 2024-04-25T11:43:43.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.926+0000 7f0f30773740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55cab7b44a20 con 0x55cab7884390 2024-04-25T11:43:43.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.926+0000 7f0f30773740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55cab7b86d40 con 0x55cab785b4d0 2024-04-25T11:43:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.926+0000 7f0f227fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab78636b0 0x55cab7c326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.926+0000 7f0f227fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab78636b0 0x55cab7c326a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42084/0 (socket says 172.21.15.90:42084) 2024-04-25T11:43:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.926+0000 7f0f227fc640 1 -- 172.21.15.90:0/1640033869 learned_addr learned my addr 172.21.15.90:0/1640033869 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.927+0000 7f0f237fe640 1 --2- 172.21.15.90:0/1640033869 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cab7884390 0x55cab7c25810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.927+0000 7f0f22ffd640 1 --2- 172.21.15.90:0/1640033869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cab785b4d0 0x55cab7c25d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.927+0000 7f0f0bfff640 1 --2- 172.21.15.90:0/636322988 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0f0c0afc60 0x7f0f0c0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.927+0000 7f0f227fc640 1 -- 172.21.15.90:0/1640033869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cab785b4d0 msgr2=0x55cab7c25d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.927+0000 7f0f227fc640 1 --2- 172.21.15.90:0/1640033869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cab785b4d0 0x55cab7c25d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.927+0000 7f0f227fc640 1 -- 172.21.15.90:0/1640033869 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cab7884390 msgr2=0x55cab7c25810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.927+0000 7f0f227fc640 1 --2- 172.21.15.90:0/1640033869 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cab7884390 0x55cab7c25810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.927+0000 7f0f227fc640 1 -- 172.21.15.90:0/1640033869 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cab7b4fc30 con 0x55cab78636b0 2024-04-25T11:43:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.927+0000 7f0f22ffd640 1 --2- 172.21.15.90:0/1640033869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cab785b4d0 0x55cab7c25d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:43:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.927+0000 7f0f237fe640 1 --2- 172.21.15.90:0/636322988 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0f0c0afc60 0x7f0f0c0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.927+0000 7f0f0bfff640 1 -- 172.21.15.90:0/636322988 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f0f180ded90 con 0x7f0f24109bc0 2024-04-25T11:43:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.927+0000 7f0f227fc640 1 --2- 172.21.15.90:0/1640033869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab78636b0 0x55cab7c326a0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f0f180dc3c0 tx=0x7f0f180d55f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f59ea669af0826b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.927+0000 7f0f097fa640 1 -- 172.21.15.90:0/1640033869 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f180a6c70 con 0x55cab78636b0 2024-04-25T11:43:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.927+0000 7f0f097fa640 1 -- 172.21.15.90:0/1640033869 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f180b0d80 con 0x55cab78636b0 2024-04-25T11:43:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.927+0000 7f0f097fa640 1 -- 172.21.15.90:0/1640033869 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f180a7100 con 0x55cab78636b0 2024-04-25T11:43:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.928+0000 7f0f30773740 1 -- 172.21.15.90:0/1640033869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab78636b0 msgr2=0x55cab7c326a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.928+0000 7f0f237fe640 1 --2- 172.21.15.90:0/636322988 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0f0c0afc60 0x7f0f0c0b2170 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x55cab7c26910 tx=0x7f0f1c0095c0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.928+0000 7f0f30773740 1 --2- 172.21.15.90:0/1640033869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab78636b0 0x55cab7c326a0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f0f180dc3c0 tx=0x7f0f180d55f0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.928+0000 7f0f30773740 1 -- 172.21.15.90:0/1640033869 shutdown_connections 2024-04-25T11:43:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.928+0000 7f0f30773740 1 --2- 172.21.15.90:0/1640033869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab78636b0 0x55cab7c326a0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.928+0000 7f0f30773740 1 --2- 172.21.15.90:0/1640033869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cab785b4d0 0x55cab7c25d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.928+0000 7f0f30773740 1 --2- 172.21.15.90:0/1640033869 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cab7884390 0x55cab7c25810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.928+0000 7f0f30773740 1 -- 172.21.15.90:0/1640033869 >> 172.21.15.90:0/1640033869 conn(0x55cab78784e0 msgr2=0x55cab7c21fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:43.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.928+0000 7f0f30773740 1 -- 172.21.15.90:0/1640033869 shutdown_connections 2024-04-25T11:43:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.928+0000 7f0f30773740 1 -- 172.21.15.90:0/1640033869 wait complete. 2024-04-25T11:43:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.928+0000 7f0f30773740 1 -- start start 2024-04-25T11:43:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.929+0000 7f0f30773740 1 Processor -- start 2024-04-25T11:43:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.929+0000 7f0f30773740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cab785b4d0 0x55cab7c30280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.929+0000 7f0f30773740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab78636b0 0x55cab7c307c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.929+0000 7f0f30773740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cab7884390 0x55cab7c3c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.929+0000 7f0f30773740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55cab7c32cd0 con 0x55cab78636b0 2024-04-25T11:43:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.929+0000 7f0f30773740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55cab7b11dc0 con 0x55cab785b4d0 2024-04-25T11:43:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.929+0000 7f0f30773740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55cab7c32e50 con 0x55cab7884390 2024-04-25T11:43:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.929+0000 7f0f30773740 10 client.0 _ll_register_callbacks cb 0x55cab7b2d600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.929+0000 7f0f227fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab78636b0 0x55cab7c307c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.929+0000 7f0f227fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab78636b0 0x55cab7c307c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42096/0 (socket says 172.21.15.90:42096) 2024-04-25T11:43:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.929+0000 7f0f227fc640 1 -- 172.21.15.90:0/1567010133 learned_addr learned my addr 172.21.15.90:0/1567010133 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:43.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.929+0000 7f0f237fe640 1 --2- 172.21.15.90:0/1567010133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cab7884390 0x55cab7c3c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.929+0000 7f0f227fc640 1 -- 172.21.15.90:0/1567010133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cab7884390 msgr2=0x55cab7c3c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.929+0000 7f0f227fc640 1 --2- 172.21.15.90:0/1567010133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cab7884390 0x55cab7c3c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.929+0000 7f0f227fc640 1 -- 172.21.15.90:0/1567010133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cab785b4d0 msgr2=0x55cab7c30280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.929+0000 7f0f22ffd640 1 --2- 172.21.15.90:0/1567010133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cab785b4d0 0x55cab7c30280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.929+0000 7f0f227fc640 1 --2- 172.21.15.90:0/1567010133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cab785b4d0 0x55cab7c30280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.929+0000 7f0f227fc640 1 -- 172.21.15.90:0/1567010133 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cab789c3f0 con 0x55cab78636b0 2024-04-25T11:43:43.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.929+0000 7f0f22ffd640 1 --2- 172.21.15.90:0/1567010133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cab785b4d0 0x55cab7c30280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:43:43.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.929+0000 7f0f227fc640 1 --2- 172.21.15.90:0/1567010133 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab78636b0 0x55cab7c307c0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f0f180dca50 tx=0x7f0f180dc410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c11d5c21d8c6b4c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:43.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.929+0000 7f0f06ff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:43.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.929+0000 7f0f06ff5640 1 -- 172.21.15.90:0/1567010133 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f180a6610 con 0x55cab78636b0 2024-04-25T11:43:43.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.930+0000 7f0f06ff5640 1 -- 172.21.15.90:0/1567010133 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f0f180dc6d0 con 0x55cab78636b0 2024-04-25T11:43:43.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.930+0000 7f0f30773740 1 -- 172.21.15.90:0/1567010133 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f180b0d80 con 0x55cab78636b0 2024-04-25T11:43:43.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.930+0000 7f0f06ff5640 1 -- 172.21.15.90:0/1567010133 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0f180a67f0 con 0x55cab78636b0 2024-04-25T11:43:43.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.930+0000 7f0f30773740 10 client.18294 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:43.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.930+0000 7f0f30773740 1 -- 172.21.15.90:0/1567010133 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cab79667d0 con 0x55cab78636b0 2024-04-25T11:43:43.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.930+0000 7f0f06ff5640 1 -- 172.21.15.90:0/1567010133 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f180b0d80 con 0x55cab78636b0 2024-04-25T11:43:43.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.930+0000 7f0f06ff5640 1 -- 172.21.15.90:0/1567010133 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f0f180bd020 con 0x55cab78636b0 2024-04-25T11:43:43.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.930+0000 7f0f30773740 10 client.18294 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:43.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.930+0000 7f0f30773740 10 client.18294 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:43:43.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.930+0000 7f0f30773740 20 client.18294 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:43.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.930+0000 7f0f30773740 1 --2- 172.21.15.90:0/1567010133 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55cab7842f60 0x55cab7c42440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.930+0000 7f0f30773740 4 client.18294 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:43:43.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.930+0000 7f0f30773740 1 -- 172.21.15.90:0/1567010133 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cab7b98990 con 0x55cab7842f60 2024-04-25T11:43:43.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.931+0000 7f0f22ffd640 1 --2- 172.21.15.90:0/1567010133 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55cab7842f60 0x55cab7c42440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.931+0000 7f0f22ffd640 1 --2- 172.21.15.90:0/1567010133 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55cab7842f60 0x55cab7c42440 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3769097ac4376ab1 server_cookie=89ad4a4ff920c3e4 in_seq=0 out_seq=0 2024-04-25T11:43:43.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.931+0000 7f0f06ff5640 10 client.18294 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:43:43.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.932+0000 7f0f06ff5640 1 -- 172.21.15.90:0/1567010133 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55cab7b98990 con 0x55cab7842f60 2024-04-25T11:43:43.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.932+0000 7f0f06ff5640 10 client.18294 handle_command_reply: tid=0 2024-04-25T11:43:43.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.932+0000 7f0f30773740 1 client.18294 shutdown 2024-04-25T11:43:43.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.932+0000 7f0f30773740 1 -- 172.21.15.90:0/1567010133 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab78636b0 msgr2=0x55cab7c307c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.932+0000 7f0f30773740 1 --2- 172.21.15.90:0/1567010133 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab78636b0 0x55cab7c307c0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f0f180dca50 tx=0x7f0f180dc410 comp rx=0 tx=0).stop 2024-04-25T11:43:43.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.932+0000 7f0f30773740 1 -- 172.21.15.90:0/1567010133 shutdown_connections 2024-04-25T11:43:43.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.932+0000 7f0f30773740 1 --2- 172.21.15.90:0/1567010133 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55cab7842f60 0x55cab7c42440 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.932+0000 7f0f30773740 1 --2- 172.21.15.90:0/1567010133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cab7884390 0x55cab7c3c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.932+0000 7f0f06ff5640 0 client.18294 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:43:43.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.932+0000 7f0f30773740 1 --2- 172.21.15.90:0/1567010133 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab78636b0 0x55cab7c307c0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.932+0000 7f0f30773740 1 --2- 172.21.15.90:0/1567010133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cab785b4d0 0x55cab7c30280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.932+0000 7f0f30773740 1 -- 172.21.15.90:0/1567010133 >> 172.21.15.90:0/1567010133 conn(0x55cab78784e0 msgr2=0x55cab7c27530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:43.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.932+0000 7f0f30773740 1 -- 172.21.15.90:0/1567010133 shutdown_connections 2024-04-25T11:43:43.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.932+0000 7f0f30773740 1 -- 172.21.15.90:0/1567010133 wait complete. 2024-04-25T11:43:43.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.932+0000 7f0f30773740 20 client.18294 trim_cache size 0 max 16384 2024-04-25T11:43:43.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.946+0000 7f0f30773740 1 Processor -- start 2024-04-25T11:43:43.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.946+0000 7f0f30773740 1 -- start start 2024-04-25T11:43:43.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.946+0000 7f0f30773740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab785b4d0 0x55cab7c3c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.946+0000 7f0f30773740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cab78636b0 0x55cab7c2fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.946+0000 7f0f30773740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cab7884390 0x55cab7c21f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.946+0000 7f0f30773740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0f241b1ef0 con 0x55cab785b4d0 2024-04-25T11:43:43.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.946+0000 7f0f30773740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0f241b3a30 con 0x55cab78636b0 2024-04-25T11:43:43.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.946+0000 7f0f30773740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0f241b25c0 con 0x55cab7884390 2024-04-25T11:43:43.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.946+0000 7f0f22ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab785b4d0 0x55cab7c3c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.946+0000 7f0f22ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab785b4d0 0x55cab7c3c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42110/0 (socket says 172.21.15.90:42110) 2024-04-25T11:43:43.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.946+0000 7f0f22ffd640 1 -- 172.21.15.90:0/3972570650 learned_addr learned my addr 172.21.15.90:0/3972570650 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:43.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.946+0000 7f0f22ffd640 1 -- 172.21.15.90:0/3972570650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cab7884390 msgr2=0x55cab7c21f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.946+0000 7f0f22ffd640 1 --2- 172.21.15.90:0/3972570650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cab7884390 0x55cab7c21f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.946+0000 7f0f22ffd640 1 -- 172.21.15.90:0/3972570650 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cab78636b0 msgr2=0x55cab7c2fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.946+0000 7f0f22ffd640 1 --2- 172.21.15.90:0/3972570650 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cab78636b0 0x55cab7c2fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.946+0000 7f0f22ffd640 1 -- 172.21.15.90:0/3972570650 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cab7c30f80 con 0x55cab785b4d0 2024-04-25T11:43:43.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.946+0000 7f0f22ffd640 1 --2- 172.21.15.90:0/3972570650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab785b4d0 0x55cab7c3c140 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x55cab7c22f50 tx=0x7f0f1400f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=512cf29277206fb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:43.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.946+0000 7f0f08ff9640 1 -- 172.21.15.90:0/3972570650 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f1401c670 con 0x55cab785b4d0 2024-04-25T11:43:43.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.946+0000 7f0f08ff9640 1 -- 172.21.15.90:0/3972570650 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f14022070 con 0x55cab785b4d0 2024-04-25T11:43:43.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.946+0000 7f0f08ff9640 1 -- 172.21.15.90:0/3972570650 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f1400e3b0 con 0x55cab785b4d0 2024-04-25T11:43:43.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.947+0000 7f0f30773740 1 -- 172.21.15.90:0/3972570650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab785b4d0 msgr2=0x55cab7c3c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.947+0000 7f0f30773740 1 --2- 172.21.15.90:0/3972570650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab785b4d0 0x55cab7c3c140 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x55cab7c22f50 tx=0x7f0f1400f7f0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.947+0000 7f0f30773740 1 -- 172.21.15.90:0/3972570650 shutdown_connections 2024-04-25T11:43:43.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.947+0000 7f0f30773740 1 --2- 172.21.15.90:0/3972570650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cab7884390 0x55cab7c21f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.947+0000 7f0f30773740 1 --2- 172.21.15.90:0/3972570650 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cab78636b0 0x55cab7c2fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.947+0000 7f0f30773740 1 --2- 172.21.15.90:0/3972570650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab785b4d0 0x55cab7c3c140 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.947+0000 7f0f30773740 1 -- 172.21.15.90:0/3972570650 >> 172.21.15.90:0/3972570650 conn(0x55cab78784e0 msgr2=0x55cab7c21860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.947+0000 7f0f30773740 1 -- 172.21.15.90:0/3972570650 shutdown_connections 2024-04-25T11:43:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.947+0000 7f0f30773740 1 -- 172.21.15.90:0/3972570650 wait complete. 2024-04-25T11:43:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.948+0000 7f0f30773740 1 -- start start 2024-04-25T11:43:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.948+0000 7f0f30773740 1 Processor -- start 2024-04-25T11:43:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.948+0000 7f0f30773740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab785b4d0 0x55cab7c458a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.948+0000 7f0f30773740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cab78636b0 0x55cab7c42c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.948+0000 7f0f30773740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cab7884390 0x55cab7c43150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.948+0000 7f0f30773740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0f241b3360 con 0x55cab785b4d0 2024-04-25T11:43:43.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.948+0000 7f0f30773740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0f241b2c90 con 0x55cab7884390 2024-04-25T11:43:43.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.948+0000 7f0f30773740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0f241afef0 con 0x55cab78636b0 2024-04-25T11:43:43.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.948+0000 7f0f30773740 10 client.0 _ll_register_callbacks cb 0x7f0f1c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:43.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.948+0000 7f0f22ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab785b4d0 0x55cab7c458a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.948+0000 7f0f22ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab785b4d0 0x55cab7c458a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42122/0 (socket says 172.21.15.90:42122) 2024-04-25T11:43:43.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.948+0000 7f0f22ffd640 1 -- 172.21.15.90:0/3487779803 learned_addr learned my addr 172.21.15.90:0/3487779803 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:43.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.948+0000 7f0f237fe640 1 --2- 172.21.15.90:0/3487779803 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cab7884390 0x55cab7c43150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.948+0000 7f0f227fc640 1 --2- 172.21.15.90:0/3487779803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cab78636b0 0x55cab7c42c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.948+0000 7f0f22ffd640 1 -- 172.21.15.90:0/3487779803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cab78636b0 msgr2=0x55cab7c42c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.948+0000 7f0f22ffd640 1 --2- 172.21.15.90:0/3487779803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cab78636b0 0x55cab7c42c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.948+0000 7f0f22ffd640 1 -- 172.21.15.90:0/3487779803 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cab7884390 msgr2=0x55cab7c43150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:43.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.948+0000 7f0f22ffd640 1 --2- 172.21.15.90:0/3487779803 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cab7884390 0x55cab7c43150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.948+0000 7f0f22ffd640 1 -- 172.21.15.90:0/3487779803 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cab7c43740 con 0x55cab785b4d0 2024-04-25T11:43:43.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.948+0000 7f0f227fc640 1 --2- 172.21.15.90:0/3487779803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cab78636b0 0x55cab7c42c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:43.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.948+0000 7f0f22ffd640 1 --2- 172.21.15.90:0/3487779803 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab785b4d0 0x55cab7c458a0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f0f14009af0 tx=0x7f0f1401bb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d1167d2af52113 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:43.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.948+0000 7f0f06ff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:43.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.949+0000 7f0f06ff5640 1 -- 172.21.15.90:0/3487779803 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f14004250 con 0x55cab785b4d0 2024-04-25T11:43:43.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.949+0000 7f0f06ff5640 1 -- 172.21.15.90:0/3487779803 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f0f140043f0 con 0x55cab785b4d0 2024-04-25T11:43:43.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.949+0000 7f0f30773740 1 -- 172.21.15.90:0/3487779803 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f14022070 con 0x55cab785b4d0 2024-04-25T11:43:43.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.949+0000 7f0f06ff5640 1 -- 172.21.15.90:0/3487779803 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0f1401b3d0 con 0x55cab785b4d0 2024-04-25T11:43:43.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.949+0000 7f0f30773740 10 client.18300 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:43.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.949+0000 7f0f30773740 1 -- 172.21.15.90:0/3487779803 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cab7caa310 con 0x55cab785b4d0 2024-04-25T11:43:43.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.949+0000 7f0f06ff5640 1 -- 172.21.15.90:0/3487779803 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f14022070 con 0x55cab785b4d0 2024-04-25T11:43:43.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.949+0000 7f0f06ff5640 1 -- 172.21.15.90:0/3487779803 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f0f14005040 con 0x55cab785b4d0 2024-04-25T11:43:43.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.949+0000 7f0f30773740 10 client.18300 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:43.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.949+0000 7f0f30773740 10 client.18300 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:43:43.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.949+0000 7f0f30773740 20 client.18300 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:43.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.949+0000 7f0f30773740 1 --2- 172.21.15.90:0/3487779803 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55cab78183a0 0x55cab7cacac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:43.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.949+0000 7f0f30773740 4 client.18300 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:43:43.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.949+0000 7f0f30773740 1 -- 172.21.15.90:0/3487779803 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cab7c30b50 con 0x55cab78183a0 2024-04-25T11:43:43.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.949+0000 7f0f227fc640 1 --2- 172.21.15.90:0/3487779803 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55cab78183a0 0x55cab7cacac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:43.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.950+0000 7f0f06ff5640 10 client.18300 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:43:43.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.950+0000 7f0f227fc640 1 --2- 172.21.15.90:0/3487779803 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55cab78183a0 0x55cab7cacac0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c3c5636698fa0d server_cookie=3b8c222fec013241 in_seq=0 out_seq=0 2024-04-25T11:43:43.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.950+0000 7f0f06ff5640 1 -- 172.21.15.90:0/3487779803 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cab7c30b50 con 0x55cab78183a0 2024-04-25T11:43:43.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.950+0000 7f0f06ff5640 10 client.18300 handle_command_reply: tid=0 2024-04-25T11:43:43.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.950+0000 7f0f30773740 1 client.18300 shutdown 2024-04-25T11:43:43.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.950+0000 7f0f30773740 1 -- 172.21.15.90:0/3487779803 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab785b4d0 msgr2=0x55cab7c458a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.950+0000 7f0f30773740 1 --2- 172.21.15.90:0/3487779803 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab785b4d0 0x55cab7c458a0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f0f14009af0 tx=0x7f0f1401bb80 comp rx=0 tx=0).stop 2024-04-25T11:43:43.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.950+0000 7f0f30773740 1 -- 172.21.15.90:0/3487779803 shutdown_connections 2024-04-25T11:43:43.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.950+0000 7f0f30773740 1 --2- 172.21.15.90:0/3487779803 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55cab78183a0 0x55cab7cacac0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.950+0000 7f0f30773740 1 --2- 172.21.15.90:0/3487779803 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cab7884390 0x55cab7c43150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.950+0000 7f0f30773740 1 --2- 172.21.15.90:0/3487779803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cab78636b0 0x55cab7c42c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.950+0000 7f0f06ff5640 0 client.18300 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:43:43.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.950+0000 7f0f30773740 1 --2- 172.21.15.90:0/3487779803 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cab785b4d0 0x55cab7c458a0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.950+0000 7f0f30773740 1 -- 172.21.15.90:0/3487779803 >> 172.21.15.90:0/3487779803 conn(0x55cab78784e0 msgr2=0x55cab7c32070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:43.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.950+0000 7f0f30773740 1 -- 172.21.15.90:0/3487779803 shutdown_connections 2024-04-25T11:43:43.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.950+0000 7f0f30773740 1 -- 172.21.15.90:0/3487779803 wait complete. 2024-04-25T11:43:43.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.950+0000 7f0f30773740 20 client.18300 trim_cache size 0 max 16384 2024-04-25T11:43:43.944 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:43:43.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.951+0000 7f0f297ee640 1 -- 172.21.15.90:0/636322988 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0f0c0afc60 msgr2=0x7f0f0c0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.951+0000 7f0f297ee640 1 --2- 172.21.15.90:0/636322988 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0f0c0afc60 0x7f0f0c0b2170 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x55cab7c26910 tx=0x7f0f1c0095c0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.951+0000 7f0f297ee640 1 -- 172.21.15.90:0/636322988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f24109bc0 msgr2=0x7f0f24103250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:43.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.951+0000 7f0f297ee640 1 --2- 172.21.15.90:0/636322988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f24109bc0 0x7f0f24103250 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f0f18005fd0 tx=0x7f0f1800dde0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.951+0000 7f0f297ee640 1 -- 172.21.15.90:0/636322988 shutdown_connections 2024-04-25T11:43:43.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.951+0000 7f0f297ee640 1 --2- 172.21.15.90:0/636322988 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0f0c0afc60 0x7f0f0c0b2170 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.951+0000 7f0f297ee640 1 --2- 172.21.15.90:0/636322988 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0f2410e570 0x7f0f24103790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.951+0000 7f0f297ee640 1 --2- 172.21.15.90:0/636322988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f24109bc0 0x7f0f24103250 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.951+0000 7f0f297ee640 1 --2- 172.21.15.90:0/636322988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f24102070 0x7f0f24102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:43.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.951+0000 7f0f297ee640 1 -- 172.21.15.90:0/636322988 >> 172.21.15.90:0/636322988 conn(0x7f0f24077920 msgr2=0x7f0f24105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:43.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.951+0000 7f0f297ee640 1 -- 172.21.15.90:0/636322988 shutdown_connections 2024-04-25T11:43:43.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:43.951+0000 7f0f297ee640 1 -- 172.21.15.90:0/636322988 wait complete. 2024-04-25T11:43:43.949 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T11:43:44.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.073+0000 7f36dabee640 1 -- 172.21.15.90:0/2841237250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36d410e050 msgr2=0x7f36d4110560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:44.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.073+0000 7f36dabee640 1 --2- 172.21.15.90:0/2841237250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36d410e050 0x7f36d4110560 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f36cc009ca0 tx=0x7f36cc02a8c0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.073+0000 7f36dabee640 1 -- 172.21.15.90:0/2841237250 shutdown_connections 2024-04-25T11:43:44.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.073+0000 7f36dabee640 1 --2- 172.21.15.90:0/2841237250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36d410e050 0x7f36d4110560 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.073+0000 7f36dabee640 1 --2- 172.21.15.90:0/2841237250 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36d410b660 0x7f36d410da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.073+0000 7f36dabee640 1 --2- 172.21.15.90:0/2841237250 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f36d4104c00 0x7f36d4105010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.073+0000 7f36dabee640 1 -- 172.21.15.90:0/2841237250 >> 172.21.15.90:0/2841237250 conn(0x7f36d40fe260 msgr2=0x7f36d4100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:44.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.073+0000 7f36dabee640 1 -- 172.21.15.90:0/2841237250 shutdown_connections 2024-04-25T11:43:44.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.073+0000 7f36dabee640 1 -- 172.21.15.90:0/2841237250 wait complete. 2024-04-25T11:43:44.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.073+0000 7f36dabee640 1 Processor -- start 2024-04-25T11:43:44.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.073+0000 7f36dabee640 1 -- start start 2024-04-25T11:43:44.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.074+0000 7f36dabee640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f36d4104c00 0x7f36d41024c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.074+0000 7f36dabee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36d410b660 0x7f36d4102a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.074+0000 7f36dabee640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36d410e050 0x7f36d4102f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.074+0000 7f36dabee640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f36d4113cf0 con 0x7f36d410e050 2024-04-25T11:43:44.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.074+0000 7f36dabee640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f36d4113b70 con 0x7f36d4104c00 2024-04-25T11:43:44.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.074+0000 7f36dabee640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f36d4113e70 con 0x7f36d410b660 2024-04-25T11:43:44.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.074+0000 7f36d918b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36d410e050 0x7f36d4102f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.074+0000 7f36d918b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36d410e050 0x7f36d4102f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42132/0 (socket says 172.21.15.90:42132) 2024-04-25T11:43:44.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.074+0000 7f36d918b640 1 -- 172.21.15.90:0/3118069398 learned_addr learned my addr 172.21.15.90:0/3118069398 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:44.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.074+0000 7f36d3fff640 1 --2- 172.21.15.90:0/3118069398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36d410b660 0x7f36d4102a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.074+0000 7f36d898a640 1 --2- 172.21.15.90:0/3118069398 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f36d4104c00 0x7f36d41024c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.074+0000 7f36d918b640 1 -- 172.21.15.90:0/3118069398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36d410b660 msgr2=0x7f36d4102a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:44.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.074+0000 7f36d918b640 1 --2- 172.21.15.90:0/3118069398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36d410b660 0x7f36d4102a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.074+0000 7f36d918b640 1 -- 172.21.15.90:0/3118069398 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f36d4104c00 msgr2=0x7f36d41024c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:44.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.074+0000 7f36d918b640 1 --2- 172.21.15.90:0/3118069398 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f36d4104c00 0x7f36d41024c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.074+0000 7f36d918b640 1 -- 172.21.15.90:0/3118069398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36d41a8a90 con 0x7f36d410e050 2024-04-25T11:43:44.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.074+0000 7f36d918b640 1 --2- 172.21.15.90:0/3118069398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36d410e050 0x7f36d4102f40 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f36cc005cc0 tx=0x7f36cc02f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:44.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.074+0000 7f36d1ffb640 1 -- 172.21.15.90:0/3118069398 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36cc003bf0 con 0x7f36d410e050 2024-04-25T11:43:44.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.074+0000 7f36d1ffb640 1 -- 172.21.15.90:0/3118069398 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36cc003d90 con 0x7f36d410e050 2024-04-25T11:43:44.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.074+0000 7f36dabee640 1 -- 172.21.15.90:0/3118069398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36d41a8d20 con 0x7f36d410e050 2024-04-25T11:43:44.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.074+0000 7f36d1ffb640 1 -- 172.21.15.90:0/3118069398 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36cc02fbc0 con 0x7f36d410e050 2024-04-25T11:43:44.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.075+0000 7f36dabee640 1 -- 172.21.15.90:0/3118069398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36d41a9210 con 0x7f36d410e050 2024-04-25T11:43:44.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.078+0000 7f36e1b73740 1 Processor -- start 2024-04-25T11:43:44.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.078+0000 7f36d1ffb640 1 -- 172.21.15.90:0/3118069398 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f36cc02fd60 con 0x7f36d410e050 2024-04-25T11:43:44.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.078+0000 7f36e1b73740 1 -- start start 2024-04-25T11:43:44.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.078+0000 7f36e1b73740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec41eea390 0x55ec4228b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.078+0000 7f36e1b73740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41ec14d0 0x55ec4228bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.078+0000 7f36e1b73740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ec96b0 0x55ec422986a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.078+0000 7f36e1b73740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ec421f0d80 con 0x55ec41ec14d0 2024-04-25T11:43:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.078+0000 7f36e1b73740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ec421aaa20 con 0x55ec41eea390 2024-04-25T11:43:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.078+0000 7f36e1b73740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ec421ecd40 con 0x55ec41ec96b0 2024-04-25T11:43:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.078+0000 7f36d898a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41ec14d0 0x55ec4228bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.078+0000 7f36d898a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41ec14d0 0x55ec4228bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42146/0 (socket says 172.21.15.90:42146) 2024-04-25T11:43:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.078+0000 7f36d898a640 1 -- 172.21.15.90:0/2396037758 learned_addr learned my addr 172.21.15.90:0/2396037758 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.078+0000 7f36d3fff640 1 --2- 172.21.15.90:0/2396037758 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ec96b0 0x55ec422986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.078+0000 7f36d3fff640 1 --2- 172.21.15.90:0/2396037758 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec41eea390 0x55ec4228b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.079+0000 7f36d898a640 1 -- 172.21.15.90:0/2396037758 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ec96b0 msgr2=0x55ec422986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.079+0000 7f36d898a640 1 --2- 172.21.15.90:0/2396037758 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ec96b0 0x55ec422986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.079+0000 7f36d898a640 1 -- 172.21.15.90:0/2396037758 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec41eea390 msgr2=0x55ec4228b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.079+0000 7f36d898a640 1 --2- 172.21.15.90:0/2396037758 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec41eea390 0x55ec4228b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.079+0000 7f36d898a640 1 -- 172.21.15.90:0/2396037758 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec421b5c30 con 0x55ec41ec14d0 2024-04-25T11:43:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.079+0000 7f36d3fff640 1 --2- 172.21.15.90:0/2396037758 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec41eea390 0x55ec4228b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:43:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.079+0000 7f36d1ffb640 1 --2- 172.21.15.90:0/3118069398 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f36bc0afce0 0x7f36bc0b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.079+0000 7f36d3fff640 1 --2- 172.21.15.90:0/3118069398 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f36bc0afce0 0x7f36bc0b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.079+0000 7f36d898a640 1 --2- 172.21.15.90:0/2396037758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41ec14d0 0x55ec4228bd50 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f36c400bc90 tx=0x7f36c400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aea11cd56c565688 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.079+0000 7f36d1ffb640 1 -- 172.21.15.90:0/3118069398 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f36cc100780 con 0x7f36d410e050 2024-04-25T11:43:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.079+0000 7f36baffd640 1 -- 172.21.15.90:0/2396037758 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c4015070 con 0x55ec41ec14d0 2024-04-25T11:43:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.079+0000 7f36baffd640 1 -- 172.21.15.90:0/2396037758 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36c40035f0 con 0x55ec41ec14d0 2024-04-25T11:43:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.079+0000 7f36baffd640 1 -- 172.21.15.90:0/2396037758 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c40116a0 con 0x55ec41ec14d0 2024-04-25T11:43:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.080+0000 7f36e1b73740 1 -- 172.21.15.90:0/2396037758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41ec14d0 msgr2=0x55ec4228bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.080+0000 7f36d3fff640 1 --2- 172.21.15.90:0/3118069398 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f36bc0afce0 0x7f36bc0b21f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x55ec42288980 tx=0x7f36c800b390 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.080+0000 7f36e1b73740 1 --2- 172.21.15.90:0/2396037758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41ec14d0 0x55ec4228bd50 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f36c400bc90 tx=0x7f36c400f730 comp rx=0 tx=0).stop 2024-04-25T11:43:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.080+0000 7f36e1b73740 1 -- 172.21.15.90:0/2396037758 shutdown_connections 2024-04-25T11:43:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.080+0000 7f36e1b73740 1 --2- 172.21.15.90:0/2396037758 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ec96b0 0x55ec422986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.080+0000 7f36e1b73740 1 --2- 172.21.15.90:0/2396037758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41ec14d0 0x55ec4228bd50 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.080+0000 7f36e1b73740 1 --2- 172.21.15.90:0/2396037758 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec41eea390 0x55ec4228b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.080+0000 7f36e1b73740 1 -- 172.21.15.90:0/2396037758 >> 172.21.15.90:0/2396037758 conn(0x55ec41ede4e0 msgr2=0x55ec42287fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.080+0000 7f36e1b73740 1 -- 172.21.15.90:0/2396037758 shutdown_connections 2024-04-25T11:43:44.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.080+0000 7f36e1b73740 1 -- 172.21.15.90:0/2396037758 wait complete. 2024-04-25T11:43:44.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.080+0000 7f36e1b73740 1 -- start start 2024-04-25T11:43:44.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.080+0000 7f36e1b73740 1 Processor -- start 2024-04-25T11:43:44.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.081+0000 7f36e1b73740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec41ec14d0 0x55ec422962a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.081+0000 7f36e1b73740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ec96b0 0x55ec422967e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.081+0000 7f36e1b73740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41eea390 0x55ec422a2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.081+0000 7f36e1b73740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ec42298d90 con 0x55ec41eea390 2024-04-25T11:43:44.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.081+0000 7f36e1b73740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ec42177dc0 con 0x55ec41ec14d0 2024-04-25T11:43:44.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.081+0000 7f36e1b73740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ec42298f10 con 0x55ec41ec96b0 2024-04-25T11:43:44.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.081+0000 7f36e1b73740 10 client.0 _ll_register_callbacks cb 0x55ec42193600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:44.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.081+0000 7f36d918b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41eea390 0x55ec422a2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.081+0000 7f36d918b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41eea390 0x55ec422a2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42148/0 (socket says 172.21.15.90:42148) 2024-04-25T11:43:44.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.081+0000 7f36d918b640 1 -- 172.21.15.90:0/3321256429 learned_addr learned my addr 172.21.15.90:0/3321256429 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:44.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.081+0000 7f36d3fff640 1 --2- 172.21.15.90:0/3321256429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ec96b0 0x55ec422967e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.081+0000 7f36d898a640 1 --2- 172.21.15.90:0/3321256429 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec41ec14d0 0x55ec422962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.081+0000 7f36d918b640 1 -- 172.21.15.90:0/3321256429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ec96b0 msgr2=0x55ec422967e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:44.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.081+0000 7f36d918b640 1 --2- 172.21.15.90:0/3321256429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ec96b0 0x55ec422967e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.081+0000 7f36d918b640 1 -- 172.21.15.90:0/3321256429 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec41ec14d0 msgr2=0x55ec422962a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:44.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.081+0000 7f36d918b640 1 --2- 172.21.15.90:0/3321256429 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec41ec14d0 0x55ec422962a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.081+0000 7f36d918b640 1 -- 172.21.15.90:0/3321256429 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec41f023f0 con 0x55ec41eea390 2024-04-25T11:43:44.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.081+0000 7f36d898a640 1 --2- 172.21.15.90:0/3321256429 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec41ec14d0 0x55ec422962a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:43:44.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.081+0000 7f36d3fff640 1 --2- 172.21.15.90:0/3321256429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ec96b0 0x55ec422967e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:43:44.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.081+0000 7f36b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:44.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.081+0000 7f36d918b640 1 --2- 172.21.15.90:0/3321256429 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41eea390 0x55ec422a2210 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f36cc0c88d0 tx=0x7f36cc0c8510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4f7390631a74423 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:44.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.082+0000 7f36b3fff640 1 -- 172.21.15.90:0/3321256429 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36cc0c8940 con 0x55ec41eea390 2024-04-25T11:43:44.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.082+0000 7f36b3fff640 1 -- 172.21.15.90:0/3321256429 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f36cc0fe060 con 0x55ec41eea390 2024-04-25T11:43:44.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.082+0000 7f36e1b73740 1 -- 172.21.15.90:0/3321256429 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36c40035f0 con 0x55ec41eea390 2024-04-25T11:43:44.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.082+0000 7f36b3fff640 1 -- 172.21.15.90:0/3321256429 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f36cc0c4800 con 0x55ec41eea390 2024-04-25T11:43:44.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.082+0000 7f36e1b73740 10 client.18312 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:44.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.082+0000 7f36e1b73740 1 -- 172.21.15.90:0/3321256429 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ec41fcc7d0 con 0x55ec41eea390 2024-04-25T11:43:44.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.082+0000 7f36b3fff640 1 -- 172.21.15.90:0/3321256429 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36c40035f0 con 0x55ec41eea390 2024-04-25T11:43:44.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.082+0000 7f36b3fff640 1 -- 172.21.15.90:0/3321256429 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f36cc0e2020 con 0x55ec41eea390 2024-04-25T11:43:44.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.082+0000 7f36e1b73740 10 client.18312 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:44.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.082+0000 7f36e1b73740 10 client.18312 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:43:44.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.082+0000 7f36e1b73740 20 client.18312 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:44.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.082+0000 7f36e1b73740 1 --2- 172.21.15.90:0/3321256429 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ec41ea8f60 0x55ec422a86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.082+0000 7f36e1b73740 4 client.18312 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:43:44.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.082+0000 7f36e1b73740 1 -- 172.21.15.90:0/3321256429 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec421fe990 con 0x55ec41ea8f60 2024-04-25T11:43:44.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.082+0000 7f36d898a640 1 --2- 172.21.15.90:0/3321256429 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ec41ea8f60 0x55ec422a86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.083+0000 7f36d898a640 1 --2- 172.21.15.90:0/3321256429 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ec41ea8f60 0x55ec422a86b0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9618221fbcad5c4 server_cookie=1808356c25b6512b in_seq=0 out_seq=0 2024-04-25T11:43:44.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.083+0000 7f36b3fff640 10 client.18312 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:44.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.084+0000 7f36b3fff640 1 -- 172.21.15.90:0/3321256429 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ec421fe990 con 0x55ec41ea8f60 2024-04-25T11:43:44.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.084+0000 7f36b3fff640 10 client.18312 handle_command_reply: tid=0 2024-04-25T11:43:44.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.084+0000 7f36e1b73740 1 client.18312 shutdown 2024-04-25T11:43:44.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.084+0000 7f36e1b73740 1 -- 172.21.15.90:0/3321256429 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41eea390 msgr2=0x55ec422a2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:44.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.084+0000 7f36e1b73740 1 --2- 172.21.15.90:0/3321256429 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41eea390 0x55ec422a2210 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f36cc0c88d0 tx=0x7f36cc0c8510 comp rx=0 tx=0).stop 2024-04-25T11:43:44.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.084+0000 7f36e1b73740 1 -- 172.21.15.90:0/3321256429 shutdown_connections 2024-04-25T11:43:44.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.084+0000 7f36e1b73740 1 --2- 172.21.15.90:0/3321256429 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ec41ea8f60 0x55ec422a86b0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.084+0000 7f36e1b73740 1 --2- 172.21.15.90:0/3321256429 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41eea390 0x55ec422a2210 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.084+0000 7f36e1b73740 1 --2- 172.21.15.90:0/3321256429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ec96b0 0x55ec422967e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.084+0000 7f36b3fff640 0 client.18312 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:44.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.084+0000 7f36e1b73740 1 --2- 172.21.15.90:0/3321256429 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec41ec14d0 0x55ec422962a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.084+0000 7f36e1b73740 1 -- 172.21.15.90:0/3321256429 >> 172.21.15.90:0/3321256429 conn(0x55ec41ede4e0 msgr2=0x55ec4228d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:44.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.084+0000 7f36e1b73740 1 -- 172.21.15.90:0/3321256429 shutdown_connections 2024-04-25T11:43:44.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.084+0000 7f36e1b73740 1 -- 172.21.15.90:0/3321256429 wait complete. 2024-04-25T11:43:44.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.084+0000 7f36e1b73740 20 client.18312 trim_cache size 0 max 16384 2024-04-25T11:43:44.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.098+0000 7f36e1b73740 1 Processor -- start 2024-04-25T11:43:44.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.098+0000 7f36e1b73740 1 -- start start 2024-04-25T11:43:44.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.098+0000 7f36e1b73740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41ec14d0 0x55ec42294fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.098+0000 7f36e1b73740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ec96b0 0x55ec42287f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.098+0000 7f36e1b73740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec41eea390 0x55ec422a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.098+0000 7f36e1b73740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f36d41b1e80 con 0x55ec41ec14d0 2024-04-25T11:43:44.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.098+0000 7f36e1b73740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f36d41b39c0 con 0x55ec41eea390 2024-04-25T11:43:44.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.098+0000 7f36e1b73740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f36d41b2550 con 0x55ec41ec96b0 2024-04-25T11:43:44.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.098+0000 7f36d898a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41ec14d0 0x55ec42294fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.098+0000 7f36d898a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41ec14d0 0x55ec42294fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42150/0 (socket says 172.21.15.90:42150) 2024-04-25T11:43:44.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.098+0000 7f36d898a640 1 -- 172.21.15.90:0/1798803703 learned_addr learned my addr 172.21.15.90:0/1798803703 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:44.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.098+0000 7f36d3fff640 1 --2- 172.21.15.90:0/1798803703 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ec96b0 0x55ec42287f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.098+0000 7f36d918b640 1 --2- 172.21.15.90:0/1798803703 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec41eea390 0x55ec422a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.098+0000 7f36d898a640 1 -- 172.21.15.90:0/1798803703 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ec96b0 msgr2=0x55ec42287f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:44.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.098+0000 7f36d898a640 1 --2- 172.21.15.90:0/1798803703 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ec96b0 0x55ec42287f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.098+0000 7f36d898a640 1 -- 172.21.15.90:0/1798803703 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec41eea390 msgr2=0x55ec422a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:44.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.098+0000 7f36d898a640 1 --2- 172.21.15.90:0/1798803703 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec41eea390 0x55ec422a1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.098+0000 7f36d898a640 1 -- 172.21.15.90:0/1798803703 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec4228cd90 con 0x55ec41ec14d0 2024-04-25T11:43:44.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.099+0000 7f36d898a640 1 --2- 172.21.15.90:0/1798803703 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41ec14d0 0x55ec42294fd0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f36c4005d00 tx=0x7f36c4004160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=328b01253dc53c7c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:44.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.099+0000 7f36ba7fc640 1 -- 172.21.15.90:0/1798803703 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c402a070 con 0x55ec41ec14d0 2024-04-25T11:43:44.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.099+0000 7f36ba7fc640 1 -- 172.21.15.90:0/1798803703 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36c4004450 con 0x55ec41ec14d0 2024-04-25T11:43:44.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.099+0000 7f36ba7fc640 1 -- 172.21.15.90:0/1798803703 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c4015040 con 0x55ec41ec14d0 2024-04-25T11:43:44.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.100+0000 7f36e1b73740 1 -- 172.21.15.90:0/1798803703 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41ec14d0 msgr2=0x55ec42294fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:44.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.100+0000 7f36e1b73740 1 --2- 172.21.15.90:0/1798803703 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41ec14d0 0x55ec42294fd0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f36c4005d00 tx=0x7f36c4004160 comp rx=0 tx=0).stop 2024-04-25T11:43:44.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.100+0000 7f36e1b73740 1 -- 172.21.15.90:0/1798803703 shutdown_connections 2024-04-25T11:43:44.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.100+0000 7f36e1b73740 1 --2- 172.21.15.90:0/1798803703 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec41eea390 0x55ec422a1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.100+0000 7f36e1b73740 1 --2- 172.21.15.90:0/1798803703 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ec96b0 0x55ec42287f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.100+0000 7f36e1b73740 1 --2- 172.21.15.90:0/1798803703 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41ec14d0 0x55ec42294fd0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.100+0000 7f36e1b73740 1 -- 172.21.15.90:0/1798803703 >> 172.21.15.90:0/1798803703 conn(0x55ec41ede4e0 msgr2=0x55ec42287800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:44.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.100+0000 7f36e1b73740 1 -- 172.21.15.90:0/1798803703 shutdown_connections 2024-04-25T11:43:44.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.100+0000 7f36e1b73740 1 -- 172.21.15.90:0/1798803703 wait complete. 2024-04-25T11:43:44.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.100+0000 7f36e1b73740 1 -- start start 2024-04-25T11:43:44.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.100+0000 7f36e1b73740 1 Processor -- start 2024-04-25T11:43:44.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.100+0000 7f36e1b73740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ec14d0 0x55ec422991b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.100+0000 7f36e1b73740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec41ec96b0 0x55ec422996f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.100+0000 7f36e1b73740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41eea390 0x55ec42299c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.100+0000 7f36e1b73740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f36d41b32f0 con 0x55ec41eea390 2024-04-25T11:43:44.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.100+0000 7f36e1b73740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f36d41b2c20 con 0x55ec41ec96b0 2024-04-25T11:43:44.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.100+0000 7f36e1b73740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f36d41afe80 con 0x55ec41ec14d0 2024-04-25T11:43:44.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.101+0000 7f36e1b73740 10 client.0 _ll_register_callbacks cb 0x7f36cc009430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:44.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.101+0000 7f36d918b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41eea390 0x55ec42299c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.101+0000 7f36d918b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41eea390 0x55ec42299c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42152/0 (socket says 172.21.15.90:42152) 2024-04-25T11:43:44.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.101+0000 7f36d918b640 1 -- 172.21.15.90:0/1220019926 learned_addr learned my addr 172.21.15.90:0/1220019926 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:44.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.101+0000 7f36d898a640 1 --2- 172.21.15.90:0/1220019926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ec14d0 0x55ec422991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.101+0000 7f36d3fff640 1 --2- 172.21.15.90:0/1220019926 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec41ec96b0 0x55ec422996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.101+0000 7f36d918b640 1 -- 172.21.15.90:0/1220019926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ec14d0 msgr2=0x55ec422991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:44.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.101+0000 7f36d918b640 1 --2- 172.21.15.90:0/1220019926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ec14d0 0x55ec422991b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.101+0000 7f36d918b640 1 -- 172.21.15.90:0/1220019926 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec41ec96b0 msgr2=0x55ec422996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:44.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.101+0000 7f36d918b640 1 --2- 172.21.15.90:0/1220019926 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec41ec96b0 0x55ec422996f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.101+0000 7f36d918b640 1 -- 172.21.15.90:0/1220019926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec4230ea60 con 0x55ec41eea390 2024-04-25T11:43:44.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.101+0000 7f36d918b640 1 --2- 172.21.15.90:0/1220019926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41eea390 0x55ec42299c30 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f36cc0c8640 tx=0x7f36cc0fe310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc790ecd0d97a45 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:44.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.101+0000 7f36b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:44.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.101+0000 7f36b3fff640 1 -- 172.21.15.90:0/1220019926 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36cc0c8940 con 0x55ec41eea390 2024-04-25T11:43:44.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.101+0000 7f36b3fff640 1 -- 172.21.15.90:0/1220019926 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f36cc08f1a0 con 0x55ec41eea390 2024-04-25T11:43:44.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.101+0000 7f36e1b73740 1 -- 172.21.15.90:0/1220019926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36c4004450 con 0x55ec41eea390 2024-04-25T11:43:44.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.101+0000 7f36b3fff640 1 -- 172.21.15.90:0/1220019926 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f36cc0c8290 con 0x55ec41eea390 2024-04-25T11:43:44.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.101+0000 7f36e1b73740 10 client.18318 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:44.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.101+0000 7f36e1b73740 1 -- 172.21.15.90:0/1220019926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ec4230fce0 con 0x55ec41eea390 2024-04-25T11:43:44.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.102+0000 7f36b3fff640 1 -- 172.21.15.90:0/1220019926 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36c4004450 con 0x55ec41eea390 2024-04-25T11:43:44.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.102+0000 7f36b3fff640 1 -- 172.21.15.90:0/1220019926 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f36cc0c42b0 con 0x55ec41eea390 2024-04-25T11:43:44.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.102+0000 7f36e1b73740 10 client.18318 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:44.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.102+0000 7f36e1b73740 10 client.18318 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:43:44.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.102+0000 7f36e1b73740 20 client.18318 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:44.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.102+0000 7f36e1b73740 1 --2- 172.21.15.90:0/1220019926 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ec41e7e3a0 0x55ec42312270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.102+0000 7f36e1b73740 4 client.18318 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:43:44.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.102+0000 7f36e1b73740 1 -- 172.21.15.90:0/1220019926 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ec421f8800 con 0x55ec41e7e3a0 2024-04-25T11:43:44.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.102+0000 7f36d898a640 1 --2- 172.21.15.90:0/1220019926 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ec41e7e3a0 0x55ec42312270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.102+0000 7f36b3fff640 10 client.18318 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:44.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.102+0000 7f36d898a640 1 --2- 172.21.15.90:0/1220019926 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ec41e7e3a0 0x55ec42312270 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1b5d4144a194370 server_cookie=4d06be5237584f4f in_seq=0 out_seq=0 2024-04-25T11:43:44.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.103+0000 7f36b3fff640 1 -- 172.21.15.90:0/1220019926 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ec421f8800 con 0x55ec41e7e3a0 2024-04-25T11:43:44.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.103+0000 7f36b3fff640 10 client.18318 handle_command_reply: tid=0 2024-04-25T11:43:44.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.103+0000 7f36e1b73740 1 client.18318 shutdown 2024-04-25T11:43:44.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.103+0000 7f36e1b73740 1 -- 172.21.15.90:0/1220019926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41eea390 msgr2=0x55ec42299c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:44.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.103+0000 7f36e1b73740 1 --2- 172.21.15.90:0/1220019926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41eea390 0x55ec42299c30 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f36cc0c8640 tx=0x7f36cc0fe310 comp rx=0 tx=0).stop 2024-04-25T11:43:44.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.103+0000 7f36e1b73740 1 -- 172.21.15.90:0/1220019926 shutdown_connections 2024-04-25T11:43:44.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.103+0000 7f36e1b73740 1 --2- 172.21.15.90:0/1220019926 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ec41e7e3a0 0x55ec42312270 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.103+0000 7f36e1b73740 1 --2- 172.21.15.90:0/1220019926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ec41eea390 0x55ec42299c30 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.103+0000 7f36b3fff640 0 client.18318 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:44.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.103+0000 7f36e1b73740 1 --2- 172.21.15.90:0/1220019926 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ec41ec96b0 0x55ec422996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.103+0000 7f36e1b73740 1 --2- 172.21.15.90:0/1220019926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ec14d0 0x55ec422991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.103+0000 7f36e1b73740 1 -- 172.21.15.90:0/1220019926 >> 172.21.15.90:0/1220019926 conn(0x55ec41ede4e0 msgr2=0x55ec42286610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:44.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.103+0000 7f36e1b73740 1 -- 172.21.15.90:0/1220019926 shutdown_connections 2024-04-25T11:43:44.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.103+0000 7f36e1b73740 1 -- 172.21.15.90:0/1220019926 wait complete. 2024-04-25T11:43:44.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.103+0000 7f36e1b73740 20 client.18318 trim_cache size 0 max 16384 2024-04-25T11:43:44.097 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:43:44.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.104+0000 7f36dabee640 1 -- 172.21.15.90:0/3118069398 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f36bc0afce0 msgr2=0x7f36bc0b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:44.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.104+0000 7f36dabee640 1 --2- 172.21.15.90:0/3118069398 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f36bc0afce0 0x7f36bc0b21f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x55ec42288980 tx=0x7f36c800b390 comp rx=0 tx=0).stop 2024-04-25T11:43:44.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.104+0000 7f36dabee640 1 -- 172.21.15.90:0/3118069398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36d410e050 msgr2=0x7f36d4102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:44.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.104+0000 7f36dabee640 1 --2- 172.21.15.90:0/3118069398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36d410e050 0x7f36d4102f40 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f36cc005cc0 tx=0x7f36cc02f380 comp rx=0 tx=0).stop 2024-04-25T11:43:44.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.104+0000 7f36dabee640 1 -- 172.21.15.90:0/3118069398 shutdown_connections 2024-04-25T11:43:44.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.104+0000 7f36dabee640 1 --2- 172.21.15.90:0/3118069398 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f36bc0afce0 0x7f36bc0b21f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.104+0000 7f36dabee640 1 --2- 172.21.15.90:0/3118069398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36d410e050 0x7f36d4102f40 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.104+0000 7f36dabee640 1 --2- 172.21.15.90:0/3118069398 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36d410b660 0x7f36d4102a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.104+0000 7f36dabee640 1 --2- 172.21.15.90:0/3118069398 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f36d4104c00 0x7f36d41024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.104+0000 7f36dabee640 1 -- 172.21.15.90:0/3118069398 >> 172.21.15.90:0/3118069398 conn(0x7f36d40fe260 msgr2=0x7f36d410e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:44.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.104+0000 7f36dabee640 1 -- 172.21.15.90:0/3118069398 shutdown_connections 2024-04-25T11:43:44.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:44.104+0000 7f36dabee640 1 -- 172.21.15.90:0/3118069398 wait complete. 2024-04-25T11:43:44.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:44 smithi090 ceph-mon[29575]: pgmap v266: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:43:44.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:44 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/4285065574' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:43:44.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:44 smithi156 ceph-mon[33045]: pgmap v266: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:43:44.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:44 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/4285065574' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:43:44.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.362+0000 7f1c046e2640 1 -- 172.21.15.90:0/2436449605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1bfc1038e0 msgr2=0x7f1bfc10f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:44.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.362+0000 7f1c046e2640 1 --2- 172.21.15.90:0/2436449605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1bfc1038e0 0x7f1bfc10f580 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4009ca0 tx=0x7f1bf402a950 comp rx=0 tx=0).stop 2024-04-25T11:43:44.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.362+0000 7f1c046e2640 1 -- 172.21.15.90:0/2436449605 shutdown_connections 2024-04-25T11:43:44.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.363+0000 7f1c046e2640 1 --2- 172.21.15.90:0/2436449605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1bfc1038e0 0x7f1bfc10f580 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.363+0000 7f1c046e2640 1 --2- 172.21.15.90:0/2436449605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1bfc108ea0 0x7f1bfc103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.363+0000 7f1c046e2640 1 --2- 172.21.15.90:0/2436449605 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1bfc108a90 0x7f1bfc102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.363+0000 7f1c046e2640 1 -- 172.21.15.90:0/2436449605 >> 172.21.15.90:0/2436449605 conn(0x7f1bfc0fd8f0 msgr2=0x7f1bfc0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:44.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.363+0000 7f1c046e2640 1 -- 172.21.15.90:0/2436449605 shutdown_connections 2024-04-25T11:43:44.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.363+0000 7f1c046e2640 1 -- 172.21.15.90:0/2436449605 wait complete. 2024-04-25T11:43:44.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.363+0000 7f1c046e2640 1 Processor -- start 2024-04-25T11:43:44.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.363+0000 7f1c046e2640 1 -- start start 2024-04-25T11:43:44.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.363+0000 7f1c046e2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1bfc108a90 0x7f1bfc19a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.363+0000 7f1c046e2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1bfc108ea0 0x7f1bfc19a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.363+0000 7f1c046e2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1bfc19eb30 0x7f1bfc19ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.364+0000 7f1c046e2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1bfc113d70 con 0x7f1bfc108a90 2024-04-25T11:43:44.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.364+0000 7f1c046e2640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1bfc113bf0 con 0x7f1bfc19eb30 2024-04-25T11:43:44.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.364+0000 7f1c046e2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1bfc113ef0 con 0x7f1bfc108ea0 2024-04-25T11:43:44.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.364+0000 7f1c0247e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1bfc108a90 0x7f1bfc19a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.364+0000 7f1c0247e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1bfc108a90 0x7f1bfc19a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42168/0 (socket says 172.21.15.90:42168) 2024-04-25T11:43:44.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.364+0000 7f1c0247e640 1 -- 172.21.15.90:0/615846345 learned_addr learned my addr 172.21.15.90:0/615846345 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:44.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.364+0000 7f1c01c7d640 1 --2- 172.21.15.90:0/615846345 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1bfc108ea0 0x7f1bfc19a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.364+0000 7f1c02c7f640 1 --2- 172.21.15.90:0/615846345 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1bfc19eb30 0x7f1bfc19ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.364+0000 7f1c0247e640 1 -- 172.21.15.90:0/615846345 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1bfc108ea0 msgr2=0x7f1bfc19a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:44.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.364+0000 7f1c0247e640 1 --2- 172.21.15.90:0/615846345 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1bfc108ea0 0x7f1bfc19a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.364+0000 7f1c0247e640 1 -- 172.21.15.90:0/615846345 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1bfc19eb30 msgr2=0x7f1bfc19ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:44.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.364+0000 7f1c0247e640 1 --2- 172.21.15.90:0/615846345 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1bfc19eb30 0x7f1bfc19ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.364+0000 7f1c0247e640 1 -- 172.21.15.90:0/615846345 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bfc19f580 con 0x7f1bfc108a90 2024-04-25T11:43:44.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.364+0000 7f1c02c7f640 1 --2- 172.21.15.90:0/615846345 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1bfc19eb30 0x7f1bfc19ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:43:44.355 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.364+0000 7f1c01c7d640 1 --2- 172.21.15.90:0/615846345 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1bfc108ea0 0x7f1bfc19a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:44.355 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.364+0000 7f1c0247e640 1 --2- 172.21.15.90:0/615846345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1bfc108a90 0x7f1bfc19a310 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f1bf800d770 tx=0x7f1bf800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:44.355 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.364+0000 7f1bf37fe640 1 -- 172.21.15.90:0/615846345 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf8017070 con 0x7f1bfc108a90 2024-04-25T11:43:44.355 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.364+0000 7f1c046e2640 1 -- 172.21.15.90:0/615846345 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bfc19f870 con 0x7f1bfc108a90 2024-04-25T11:43:44.355 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.364+0000 7f1bf37fe640 1 -- 172.21.15.90:0/615846345 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bf8009c30 con 0x7f1bfc108a90 2024-04-25T11:43:44.355 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.364+0000 7f1bf37fe640 1 -- 172.21.15.90:0/615846345 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf800f620 con 0x7f1bfc108a90 2024-04-25T11:43:44.355 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.364+0000 7f1c046e2640 1 -- 172.21.15.90:0/615846345 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bfc1a70d0 con 0x7f1bfc108a90 2024-04-25T11:43:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.367+0000 7f1bf37fe640 1 -- 172.21.15.90:0/615846345 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f1bf801a070 con 0x7f1bfc108a90 2024-04-25T11:43:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.367+0000 7f1c0b58d740 1 Processor -- start 2024-04-25T11:43:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.367+0000 7f1c0b58d740 1 -- start start 2024-04-25T11:43:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.367+0000 7f1c0b58d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b3b1b0 0x5626e2edaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.367+0000 7f1c0b58d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626e2b122f0 0x5626e2edb030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.367+0000 7f1c0b58d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626e2b1a4d0 0x5626e2edb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.367+0000 7f1c0b58d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5626e2e41460 con 0x5626e2b3b1b0 2024-04-25T11:43:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.367+0000 7f1c0b58d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5626e2dfb590 con 0x5626e2b1a4d0 2024-04-25T11:43:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.367+0000 7f1c0b58d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5626e2e3d420 con 0x5626e2b122f0 2024-04-25T11:43:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.367+0000 7f1c02c7f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b3b1b0 0x5626e2edaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.367+0000 7f1c02c7f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b3b1b0 0x5626e2edaaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42174/0 (socket says 172.21.15.90:42174) 2024-04-25T11:43:44.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.367+0000 7f1c02c7f640 1 -- 172.21.15.90:0/1546050352 learned_addr learned my addr 172.21.15.90:0/1546050352 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:44.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.367+0000 7f1bf37fe640 1 --2- 172.21.15.90:0/615846345 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1be80afc50 0x7f1be80b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.367+0000 7f1c01c7d640 1 --2- 172.21.15.90:0/1546050352 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626e2b1a4d0 0x5626e2edb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.367+0000 7f1c0247e640 1 --2- 172.21.15.90:0/1546050352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626e2b122f0 0x5626e2edb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.367+0000 7f1c02c7f640 1 -- 172.21.15.90:0/1546050352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626e2b122f0 msgr2=0x5626e2edb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:44.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.368+0000 7f1c02c7f640 1 --2- 172.21.15.90:0/1546050352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626e2b122f0 0x5626e2edb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.368+0000 7f1c02c7f640 1 -- 172.21.15.90:0/1546050352 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626e2b1a4d0 msgr2=0x5626e2edb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:44.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.368+0000 7f1c02c7f640 1 --2- 172.21.15.90:0/1546050352 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626e2b1a4d0 0x5626e2edb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.368+0000 7f1c02c7f640 1 -- 172.21.15.90:0/1546050352 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626e2e080d0 con 0x5626e2b3b1b0 2024-04-25T11:43:44.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.368+0000 7f1c0247e640 1 --2- 172.21.15.90:0/1546050352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626e2b122f0 0x5626e2edb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:43:44.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.368+0000 7f1c01c7d640 1 --2- 172.21.15.90:0/1546050352 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626e2b1a4d0 0x5626e2edb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:44.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.368+0000 7f1bf37fe640 1 -- 172.21.15.90:0/615846345 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f1bf80decc0 con 0x7f1bfc108a90 2024-04-25T11:43:44.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.368+0000 7f1c02c7f640 1 --2- 172.21.15.90:0/615846345 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1be80afc50 0x7f1be80b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.368+0000 7f1c02c7f640 1 --2- 172.21.15.90:0/1546050352 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b3b1b0 0x5626e2edaaf0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4004af0 tx=0x7f1bf402feb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8aa4e5ee7590541 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.368+0000 7f1bf0ff9640 1 -- 172.21.15.90:0/1546050352 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf4040040 con 0x5626e2b3b1b0 2024-04-25T11:43:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.368+0000 7f1bf0ff9640 1 -- 172.21.15.90:0/1546050352 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bf40044f0 con 0x5626e2b3b1b0 2024-04-25T11:43:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.368+0000 7f1bf0ff9640 1 -- 172.21.15.90:0/1546050352 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf4034070 con 0x5626e2b3b1b0 2024-04-25T11:43:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.369+0000 7f1c0b58d740 1 -- 172.21.15.90:0/1546050352 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b3b1b0 msgr2=0x5626e2edaaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.369+0000 7f1c02c7f640 1 --2- 172.21.15.90:0/615846345 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1be80afc50 0x7f1be80b2160 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1bf404b000 tx=0x7f1bf404c000 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.369+0000 7f1c0b58d740 1 --2- 172.21.15.90:0/1546050352 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b3b1b0 0x5626e2edaaf0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4004af0 tx=0x7f1bf402feb0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.369+0000 7f1c0b58d740 1 -- 172.21.15.90:0/1546050352 shutdown_connections 2024-04-25T11:43:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.369+0000 7f1c0b58d740 1 --2- 172.21.15.90:0/1546050352 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626e2b1a4d0 0x5626e2edb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.369+0000 7f1c0b58d740 1 --2- 172.21.15.90:0/1546050352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626e2b122f0 0x5626e2edb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.369+0000 7f1c0b58d740 1 --2- 172.21.15.90:0/1546050352 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b3b1b0 0x5626e2edaaf0 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.369+0000 7f1c0b58d740 1 -- 172.21.15.90:0/1546050352 >> 172.21.15.90:0/1546050352 conn(0x5626e2b2f300 msgr2=0x5626e2ed72a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.369+0000 7f1c0b58d740 1 -- 172.21.15.90:0/1546050352 shutdown_connections 2024-04-25T11:43:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.369+0000 7f1c0b58d740 1 -- 172.21.15.90:0/1546050352 wait complete. 2024-04-25T11:43:44.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.369+0000 7f1c0b58d740 1 -- start start 2024-04-25T11:43:44.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.369+0000 7f1c0b58d740 1 Processor -- start 2024-04-25T11:43:44.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.370+0000 7f1c0b58d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626e2b122f0 0x5626e2ee6b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.370+0000 7f1c0b58d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626e2b1a4d0 0x5626e2ee7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.370+0000 7f1c0b58d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b3b1b0 0x5626e2ee4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.370+0000 7f1c0b58d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5626e2ee7c20 con 0x5626e2b3b1b0 2024-04-25T11:43:44.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.370+0000 7f1c0b58d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5626e2dc8b60 con 0x5626e2b122f0 2024-04-25T11:43:44.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.370+0000 7f1c0b58d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5626e2ee7da0 con 0x5626e2b1a4d0 2024-04-25T11:43:44.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.370+0000 7f1c0b58d740 10 client.0 _ll_register_callbacks cb 0x5626e2ded2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:44.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.370+0000 7f1c02c7f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b3b1b0 0x5626e2ee4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.370+0000 7f1c02c7f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b3b1b0 0x5626e2ee4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42190/0 (socket says 172.21.15.90:42190) 2024-04-25T11:43:44.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.370+0000 7f1c02c7f640 1 -- 172.21.15.90:0/3381595100 learned_addr learned my addr 172.21.15.90:0/3381595100 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:44.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.370+0000 7f1c01c7d640 1 --2- 172.21.15.90:0/3381595100 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626e2b1a4d0 0x5626e2ee7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.370+0000 7f1c0247e640 1 --2- 172.21.15.90:0/3381595100 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626e2b122f0 0x5626e2ee6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.370+0000 7f1c02c7f640 1 -- 172.21.15.90:0/3381595100 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626e2b1a4d0 msgr2=0x5626e2ee7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:44.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.370+0000 7f1c02c7f640 1 --2- 172.21.15.90:0/3381595100 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626e2b1a4d0 0x5626e2ee7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.370+0000 7f1c02c7f640 1 -- 172.21.15.90:0/3381595100 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626e2b122f0 msgr2=0x5626e2ee6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:44.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.370+0000 7f1c02c7f640 1 --2- 172.21.15.90:0/3381595100 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626e2b122f0 0x5626e2ee6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.370+0000 7f1c02c7f640 1 -- 172.21.15.90:0/3381595100 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626e2b53210 con 0x5626e2b3b1b0 2024-04-25T11:43:44.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.370+0000 7f1c0247e640 1 --2- 172.21.15.90:0/3381595100 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626e2b122f0 0x5626e2ee6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:44.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.370+0000 7f1c02c7f640 1 --2- 172.21.15.90:0/3381595100 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b3b1b0 0x5626e2ee4fb0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4009ca0 tx=0x7f1bf4030440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=531b8a511dbeca42 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:44.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.370+0000 7f1be5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:44.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.371+0000 7f1be5ffb640 1 -- 172.21.15.90:0/3381595100 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf4040030 con 0x5626e2b3b1b0 2024-04-25T11:43:44.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.371+0000 7f1c0b58d740 1 -- 172.21.15.90:0/3381595100 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bf40044f0 con 0x5626e2b3b1b0 2024-04-25T11:43:44.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.371+0000 7f1be5ffb640 1 -- 172.21.15.90:0/3381595100 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f1bf4004040 con 0x5626e2b3b1b0 2024-04-25T11:43:44.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.371+0000 7f1be5ffb640 1 -- 172.21.15.90:0/3381595100 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1bf4061050 con 0x5626e2b3b1b0 2024-04-25T11:43:44.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.371+0000 7f1c0b58d740 10 client.18330 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:44.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.371+0000 7f1c0b58d740 1 -- 172.21.15.90:0/3381595100 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5626e2c1f620 con 0x5626e2b3b1b0 2024-04-25T11:43:44.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.371+0000 7f1be5ffb640 1 -- 172.21.15.90:0/3381595100 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bf40044f0 con 0x5626e2b3b1b0 2024-04-25T11:43:44.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.371+0000 7f1be5ffb640 1 -- 172.21.15.90:0/3381595100 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f1bf404f560 con 0x5626e2b3b1b0 2024-04-25T11:43:44.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.371+0000 7f1c0b58d740 10 client.18330 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:44.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.371+0000 7f1c0b58d740 10 client.18330 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:43:44.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.371+0000 7f1c0b58d740 20 client.18330 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:44.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.371+0000 7f1c0b58d740 1 --2- 172.21.15.90:0/3381595100 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5626e2af9d80 0x5626e2ef7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.371+0000 7f1c0b58d740 4 client.18330 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:43:44.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.371+0000 7f1c0b58d740 1 -- 172.21.15.90:0/3381595100 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5626e2e4f070 con 0x5626e2af9d80 2024-04-25T11:43:44.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.371+0000 7f1c0247e640 1 --2- 172.21.15.90:0/3381595100 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5626e2af9d80 0x5626e2ef7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.372+0000 7f1be5ffb640 10 client.18330 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:44.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.372+0000 7f1c0247e640 1 --2- 172.21.15.90:0/3381595100 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5626e2af9d80 0x5626e2ef7540 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f36dfbe3a5b40618 server_cookie=dad75b947620e764 in_seq=0 out_seq=0 2024-04-25T11:43:44.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.373+0000 7f1be5ffb640 1 -- 172.21.15.90:0/3381595100 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5626e2e4f070 con 0x5626e2af9d80 2024-04-25T11:43:44.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.373+0000 7f1be5ffb640 10 client.18330 handle_command_reply: tid=0 2024-04-25T11:43:44.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.373+0000 7f1c0b58d740 1 client.18330 shutdown 2024-04-25T11:43:44.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.373+0000 7f1c0b58d740 1 -- 172.21.15.90:0/3381595100 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b3b1b0 msgr2=0x5626e2ee4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:44.368 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.373+0000 7f1c0b58d740 1 --2- 172.21.15.90:0/3381595100 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b3b1b0 0x5626e2ee4fb0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4009ca0 tx=0x7f1bf4030440 comp rx=0 tx=0).stop 2024-04-25T11:43:44.368 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.373+0000 7f1c0b58d740 1 -- 172.21.15.90:0/3381595100 shutdown_connections 2024-04-25T11:43:44.368 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.373+0000 7f1c0b58d740 1 --2- 172.21.15.90:0/3381595100 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5626e2af9d80 0x5626e2ef7540 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.368 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.373+0000 7f1c0b58d740 1 --2- 172.21.15.90:0/3381595100 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b3b1b0 0x5626e2ee4fb0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.368 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.373+0000 7f1be5ffb640 0 client.18330 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:44.368 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.373+0000 7f1c0b58d740 1 --2- 172.21.15.90:0/3381595100 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626e2b1a4d0 0x5626e2ee7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.368 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.373+0000 7f1c0b58d740 1 --2- 172.21.15.90:0/3381595100 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626e2b122f0 0x5626e2ee6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.368 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.373+0000 7f1c0b58d740 1 -- 172.21.15.90:0/3381595100 >> 172.21.15.90:0/3381595100 conn(0x5626e2b2f300 msgr2=0x5626e2eda920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:44.368 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.373+0000 7f1c0b58d740 1 -- 172.21.15.90:0/3381595100 shutdown_connections 2024-04-25T11:43:44.368 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.373+0000 7f1c0b58d740 1 -- 172.21.15.90:0/3381595100 wait complete. 2024-04-25T11:43:44.368 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.373+0000 7f1c0b58d740 20 client.18330 trim_cache size 0 max 16384 2024-04-25T11:43:44.379 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.387+0000 7f1c0b58d740 1 Processor -- start 2024-04-25T11:43:44.379 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.387+0000 7f1c0b58d740 1 -- start start 2024-04-25T11:43:44.379 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.388+0000 7f1c0b58d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626e2af9d80 0x5626e2ee07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.379 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.388+0000 7f1c0b58d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626e2b122f0 0x5626e2ededc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.379 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.388+0000 7f1c0b58d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b1a4d0 0x5626e2efa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.379 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.388+0000 7f1c0b58d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1bfc1afcf0 con 0x5626e2b1a4d0 2024-04-25T11:43:44.379 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.388+0000 7f1c0b58d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1bfc1b1830 con 0x5626e2af9d80 2024-04-25T11:43:44.379 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.388+0000 7f1c0b58d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1bfc1b03c0 con 0x5626e2b122f0 2024-04-25T11:43:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.388+0000 7f1c02c7f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b1a4d0 0x5626e2efa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.388+0000 7f1c02c7f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b1a4d0 0x5626e2efa570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42202/0 (socket says 172.21.15.90:42202) 2024-04-25T11:43:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.388+0000 7f1c02c7f640 1 -- 172.21.15.90:0/2967271 learned_addr learned my addr 172.21.15.90:0/2967271 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.388+0000 7f1c02c7f640 1 -- 172.21.15.90:0/2967271 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626e2b122f0 msgr2=0x5626e2ededc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.388+0000 7f1c02c7f640 1 --2- 172.21.15.90:0/2967271 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626e2b122f0 0x5626e2ededc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.388+0000 7f1c02c7f640 1 -- 172.21.15.90:0/2967271 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626e2af9d80 msgr2=0x5626e2ee07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.388+0000 7f1c02c7f640 1 --2- 172.21.15.90:0/2967271 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626e2af9d80 0x5626e2ee07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.388+0000 7f1c02c7f640 1 -- 172.21.15.90:0/2967271 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626e2ed7e90 con 0x5626e2b1a4d0 2024-04-25T11:43:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.388+0000 7f1c02c7f640 1 --2- 172.21.15.90:0/2967271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b1a4d0 0x5626e2efa570 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4004a80 tx=0x7f1bf4009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4573386f09306aec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.388+0000 7f1be7fff640 1 -- 172.21.15.90:0/2967271 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf4040070 con 0x5626e2b1a4d0 2024-04-25T11:43:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.388+0000 7f1be7fff640 1 -- 172.21.15.90:0/2967271 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bf4058070 con 0x5626e2b1a4d0 2024-04-25T11:43:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.388+0000 7f1be7fff640 1 -- 172.21.15.90:0/2967271 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf4034040 con 0x5626e2b1a4d0 2024-04-25T11:43:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.389+0000 7f1c0b58d740 1 -- 172.21.15.90:0/2967271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b1a4d0 msgr2=0x5626e2efa570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.389+0000 7f1c0b58d740 1 --2- 172.21.15.90:0/2967271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b1a4d0 0x5626e2efa570 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4004a80 tx=0x7f1bf4009680 comp rx=0 tx=0).stop 2024-04-25T11:43:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.389+0000 7f1c0b58d740 1 -- 172.21.15.90:0/2967271 shutdown_connections 2024-04-25T11:43:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.389+0000 7f1c0b58d740 1 --2- 172.21.15.90:0/2967271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b1a4d0 0x5626e2efa570 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.389+0000 7f1c0b58d740 1 --2- 172.21.15.90:0/2967271 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626e2b122f0 0x5626e2ededc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.389+0000 7f1c0b58d740 1 --2- 172.21.15.90:0/2967271 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626e2af9d80 0x5626e2ee07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.389+0000 7f1c0b58d740 1 -- 172.21.15.90:0/2967271 >> 172.21.15.90:0/2967271 conn(0x5626e2b2f300 msgr2=0x5626e2ede6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.389+0000 7f1c0b58d740 1 -- 172.21.15.90:0/2967271 shutdown_connections 2024-04-25T11:43:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.389+0000 7f1c0b58d740 1 -- 172.21.15.90:0/2967271 wait complete. 2024-04-25T11:43:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.389+0000 7f1c0b58d740 1 -- start start 2024-04-25T11:43:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.390+0000 7f1c0b58d740 1 Processor -- start 2024-04-25T11:43:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.390+0000 7f1c0b58d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626e2af9d80 0x5626e2f58460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.390+0000 7f1c0b58d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b122f0 0x5626e2f589a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.390+0000 7f1c0b58d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626e2b1a4d0 0x5626e2f52850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.390+0000 7f1c0b58d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1bfc1b1160 con 0x5626e2b122f0 2024-04-25T11:43:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.390+0000 7f1c0b58d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1bfc1b0a90 con 0x5626e2b1a4d0 2024-04-25T11:43:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.390+0000 7f1c0b58d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1bfc1adcf0 con 0x5626e2af9d80 2024-04-25T11:43:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.390+0000 7f1c0b58d740 10 client.0 _ll_register_callbacks cb 0x7f1bf402fd60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:44.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.390+0000 7f1c01c7d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b122f0 0x5626e2f589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.390+0000 7f1c01c7d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b122f0 0x5626e2f589a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42218/0 (socket says 172.21.15.90:42218) 2024-04-25T11:43:44.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.390+0000 7f1c01c7d640 1 -- 172.21.15.90:0/2101136897 learned_addr learned my addr 172.21.15.90:0/2101136897 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:44.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.390+0000 7f1c0247e640 1 --2- 172.21.15.90:0/2101136897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626e2af9d80 0x5626e2f58460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.390+0000 7f1c01c7d640 1 -- 172.21.15.90:0/2101136897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626e2af9d80 msgr2=0x5626e2f58460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:44.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.390+0000 7f1c02c7f640 1 --2- 172.21.15.90:0/2101136897 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626e2b1a4d0 0x5626e2f52850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.390+0000 7f1c01c7d640 1 --2- 172.21.15.90:0/2101136897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626e2af9d80 0x5626e2f58460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.390+0000 7f1c01c7d640 1 -- 172.21.15.90:0/2101136897 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626e2b1a4d0 msgr2=0x5626e2f52850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:44.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.390+0000 7f1c01c7d640 1 --2- 172.21.15.90:0/2101136897 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626e2b1a4d0 0x5626e2f52850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.390+0000 7f1c01c7d640 1 -- 172.21.15.90:0/2101136897 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626e2f52e60 con 0x5626e2b122f0 2024-04-25T11:43:44.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.390+0000 7f1c0247e640 1 --2- 172.21.15.90:0/2101136897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626e2af9d80 0x5626e2f58460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:44.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.390+0000 7f1c02c7f640 1 --2- 172.21.15.90:0/2101136897 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626e2b1a4d0 0x5626e2f52850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:43:44.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.390+0000 7f1c01c7d640 1 --2- 172.21.15.90:0/2101136897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b122f0 0x5626e2f589a0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc1092f0 tx=0x7f1bec011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ffc94bf7f216051 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:44.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.390+0000 7f1be5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:44.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.391+0000 7f1be5ffb640 1 -- 172.21.15.90:0/2101136897 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec010be0 con 0x5626e2b122f0 2024-04-25T11:43:44.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.391+0000 7f1be5ffb640 1 -- 172.21.15.90:0/2101136897 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f1bec010d80 con 0x5626e2b122f0 2024-04-25T11:43:44.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.391+0000 7f1c0b58d740 1 -- 172.21.15.90:0/2101136897 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bf4058070 con 0x5626e2b122f0 2024-04-25T11:43:44.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.391+0000 7f1be5ffb640 1 -- 172.21.15.90:0/2101136897 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1bec004020 con 0x5626e2b122f0 2024-04-25T11:43:44.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.391+0000 7f1c0b58d740 10 client.18336 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:44.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.391+0000 7f1c0b58d740 1 -- 172.21.15.90:0/2101136897 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5626e2f5fb10 con 0x5626e2b122f0 2024-04-25T11:43:44.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.391+0000 7f1be5ffb640 1 -- 172.21.15.90:0/2101136897 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bf4058070 con 0x5626e2b122f0 2024-04-25T11:43:44.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.391+0000 7f1be5ffb640 1 -- 172.21.15.90:0/2101136897 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f1bec004830 con 0x5626e2b122f0 2024-04-25T11:43:44.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.391+0000 7f1c0b58d740 10 client.18336 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:44.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.391+0000 7f1c0b58d740 10 client.18336 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:43:44.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.391+0000 7f1c0b58d740 20 client.18336 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:44.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.391+0000 7f1c0b58d740 1 --2- 172.21.15.90:0/2101136897 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5626e2b3b1b0 0x5626e2f62380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:44.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.391+0000 7f1c0b58d740 4 client.18336 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:43:44.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.391+0000 7f1c0b58d740 1 -- 172.21.15.90:0/2101136897 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5626e2e48ee0 con 0x5626e2b3b1b0 2024-04-25T11:43:44.384 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.391+0000 7f1c0247e640 1 --2- 172.21.15.90:0/2101136897 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5626e2b3b1b0 0x5626e2f62380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:44.384 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.392+0000 7f1c0247e640 1 --2- 172.21.15.90:0/2101136897 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5626e2b3b1b0 0x5626e2f62380 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8f9e162b693487d server_cookie=622dd33854566323 in_seq=0 out_seq=0 2024-04-25T11:43:44.384 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.392+0000 7f1be5ffb640 10 client.18336 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:44.386 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.398+0000 7f1be5ffb640 1 -- 172.21.15.90:0/2101136897 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5626e2e48ee0 con 0x5626e2b3b1b0 2024-04-25T11:43:44.386 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.398+0000 7f1be5ffb640 10 client.18336 handle_command_reply: tid=0 2024-04-25T11:43:44.386 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.398+0000 7f1c0b58d740 1 client.18336 shutdown 2024-04-25T11:43:44.386 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.399+0000 7f1c0b58d740 1 -- 172.21.15.90:0/2101136897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b122f0 msgr2=0x5626e2f589a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:44.386 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.399+0000 7f1c0b58d740 1 --2- 172.21.15.90:0/2101136897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b122f0 0x5626e2f589a0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc1092f0 tx=0x7f1bec011cb0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.399+0000 7f1c0b58d740 1 -- 172.21.15.90:0/2101136897 shutdown_connections 2024-04-25T11:43:44.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.399+0000 7f1c0b58d740 1 --2- 172.21.15.90:0/2101136897 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5626e2b3b1b0 0x5626e2f62380 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.399+0000 7f1c0b58d740 1 --2- 172.21.15.90:0/2101136897 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5626e2b1a4d0 0x5626e2f52850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.399+0000 7f1c0b58d740 1 --2- 172.21.15.90:0/2101136897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5626e2b122f0 0x5626e2f589a0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.399+0000 7f1c0b58d740 1 --2- 172.21.15.90:0/2101136897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626e2af9d80 0x5626e2f58460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.399+0000 7f1c0b58d740 1 -- 172.21.15.90:0/2101136897 >> 172.21.15.90:0/2101136897 conn(0x5626e2b2f300 msgr2=0x5626e2eda880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:44.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.399+0000 7f1be5ffb640 0 client.18336 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:44.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.399+0000 7f1c0b58d740 1 -- 172.21.15.90:0/2101136897 shutdown_connections 2024-04-25T11:43:44.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.399+0000 7f1c0b58d740 1 -- 172.21.15.90:0/2101136897 wait complete. 2024-04-25T11:43:44.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.399+0000 7f1c0b58d740 20 client.18336 trim_cache size 0 max 16384 2024-04-25T11:43:44.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:43:44.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:43:44.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:43:44.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:43:44.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.399+0000 7f1c046e2640 1 -- 172.21.15.90:0/615846345 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1be80afc50 msgr2=0x7f1be80b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:44.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.399+0000 7f1c046e2640 1 --2- 172.21.15.90:0/615846345 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1be80afc50 0x7f1be80b2160 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1bf404b000 tx=0x7f1bf404c000 comp rx=0 tx=0).stop 2024-04-25T11:43:44.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.399+0000 7f1c046e2640 1 -- 172.21.15.90:0/615846345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1bfc108a90 msgr2=0x7f1bfc19a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:44.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.399+0000 7f1c046e2640 1 --2- 172.21.15.90:0/615846345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1bfc108a90 0x7f1bfc19a310 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f1bf800d770 tx=0x7f1bf800dc30 comp rx=0 tx=0).stop 2024-04-25T11:43:44.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.399+0000 7f1c046e2640 1 -- 172.21.15.90:0/615846345 shutdown_connections 2024-04-25T11:43:44.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.399+0000 7f1c046e2640 1 --2- 172.21.15.90:0/615846345 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1be80afc50 0x7f1be80b2160 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.399+0000 7f1c046e2640 1 --2- 172.21.15.90:0/615846345 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1bfc19eb30 0x7f1bfc19ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.399+0000 7f1c046e2640 1 --2- 172.21.15.90:0/615846345 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1bfc108ea0 0x7f1bfc19a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.399+0000 7f1c046e2640 1 --2- 172.21.15.90:0/615846345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1bfc108a90 0x7f1bfc19a310 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:44.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.399+0000 7f1c046e2640 1 -- 172.21.15.90:0/615846345 >> 172.21.15.90:0/615846345 conn(0x7f1bfc0fd8f0 msgr2=0x7f1bfc0ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:44.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.400+0000 7f1c046e2640 1 -- 172.21.15.90:0/615846345 shutdown_connections 2024-04-25T11:43:44.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:44.400+0000 7f1c046e2640 1 -- 172.21.15.90:0/615846345 wait complete. 2024-04-25T11:43:44.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:44 smithi138 ceph-mon[32328]: pgmap v266: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:43:44.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:44 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/4285065574' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:43:45.103 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T11:43:45.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:43:45.103 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T11:43:45.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.239+0000 7f568e253640 1 -- 172.21.15.90:0/2017082883 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f568810e050 msgr2=0x7f5688110540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:45.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.239+0000 7f568e253640 1 --2- 172.21.15.90:0/2017082883 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f568810e050 0x7f5688110540 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f5680009ca0 tx=0x7f568002a950 comp rx=0 tx=0).stop 2024-04-25T11:43:45.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.239+0000 7f568e253640 1 -- 172.21.15.90:0/2017082883 shutdown_connections 2024-04-25T11:43:45.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.239+0000 7f568e253640 1 --2- 172.21.15.90:0/2017082883 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f568810e050 0x7f5688110540 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.239+0000 7f568e253640 1 --2- 172.21.15.90:0/2017082883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f568810b6f0 0x7f568810db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.239+0000 7f568e253640 1 --2- 172.21.15.90:0/2017082883 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5688104c90 0x7f56881050a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.239+0000 7f568e253640 1 -- 172.21.15.90:0/2017082883 >> 172.21.15.90:0/2017082883 conn(0x7f5688078100 msgr2=0x7f5688078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:45.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.240+0000 7f568e253640 1 -- 172.21.15.90:0/2017082883 shutdown_connections 2024-04-25T11:43:45.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.240+0000 7f568e253640 1 -- 172.21.15.90:0/2017082883 wait complete. 2024-04-25T11:43:45.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.240+0000 7f568e253640 1 Processor -- start 2024-04-25T11:43:45.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.240+0000 7f568e253640 1 -- start start 2024-04-25T11:43:45.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.240+0000 7f568e253640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5688104c90 0x7f568819a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.240+0000 7f568e253640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f568810b6f0 0x7f568819ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.241+0000 7f568e253640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f568810e050 0x7f568819eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.241+0000 7f568e253640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5688113cb0 con 0x7f5688104c90 2024-04-25T11:43:45.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.241+0000 7f568e253640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5688113b30 con 0x7f568810e050 2024-04-25T11:43:45.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.241+0000 7f568e253640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5688113e30 con 0x7f568810b6f0 2024-04-25T11:43:45.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.241+0000 7f56877fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5688104c90 0x7f568819a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.241+0000 7f56877fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5688104c90 0x7f568819a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42230/0 (socket says 172.21.15.90:42230) 2024-04-25T11:43:45.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.241+0000 7f56877fe640 1 -- 172.21.15.90:0/841046875 learned_addr learned my addr 172.21.15.90:0/841046875 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:45.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.241+0000 7f5686ffd640 1 --2- 172.21.15.90:0/841046875 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f568810b6f0 0x7f568819ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.241+0000 7f5687fff640 1 --2- 172.21.15.90:0/841046875 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f568810e050 0x7f568819eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.241+0000 7f56877fe640 1 -- 172.21.15.90:0/841046875 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f568810b6f0 msgr2=0x7f568819ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:45.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.241+0000 7f56877fe640 1 --2- 172.21.15.90:0/841046875 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f568810b6f0 0x7f568819ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.241+0000 7f56877fe640 1 -- 172.21.15.90:0/841046875 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f568810e050 msgr2=0x7f568819eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:45.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.241+0000 7f56877fe640 1 --2- 172.21.15.90:0/841046875 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f568810e050 0x7f568819eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.241+0000 7f56877fe640 1 -- 172.21.15.90:0/841046875 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f568819f530 con 0x7f5688104c90 2024-04-25T11:43:45.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.241+0000 7f56877fe640 1 --2- 172.21.15.90:0/841046875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5688104c90 0x7f568819a6d0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f567800d950 tx=0x7f567800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:45.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.241+0000 7f5684ff9640 1 -- 172.21.15.90:0/841046875 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5678017070 con 0x7f5688104c90 2024-04-25T11:43:45.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.241+0000 7f5684ff9640 1 -- 172.21.15.90:0/841046875 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5678009c30 con 0x7f5688104c90 2024-04-25T11:43:45.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.241+0000 7f568e253640 1 -- 172.21.15.90:0/841046875 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f568819f820 con 0x7f5688104c90 2024-04-25T11:43:45.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.241+0000 7f5684ff9640 1 -- 172.21.15.90:0/841046875 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567800f620 con 0x7f5688104c90 2024-04-25T11:43:45.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.241+0000 7f568e253640 1 -- 172.21.15.90:0/841046875 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56881a71a0 con 0x7f5688104c90 2024-04-25T11:43:45.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.244+0000 7f56951e0740 1 Processor -- start 2024-04-25T11:43:45.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.244+0000 7f5684ff9640 1 -- 172.21.15.90:0/841046875 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f5678010460 con 0x7f5688104c90 2024-04-25T11:43:45.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.244+0000 7f56951e0740 1 -- start start 2024-04-25T11:43:45.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.244+0000 7f56951e0740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a86f17d410 0x55a86f51d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.244+0000 7f56951e0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a86f154550 0x55a86f51db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.244+0000 7f56951e0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f15c730 0x55a86f51e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.244+0000 7f56951e0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a86f483b00 con 0x55a86f15c730 2024-04-25T11:43:45.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.244+0000 7f56951e0740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55a86f41f920 con 0x55a86f17d410 2024-04-25T11:43:45.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.244+0000 7f56951e0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55a86f47fac0 con 0x55a86f154550 2024-04-25T11:43:45.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.244+0000 7f5686ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f15c730 0x55a86f51e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.244+0000 7f5686ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f15c730 0x55a86f51e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42236/0 (socket says 172.21.15.90:42236) 2024-04-25T11:43:45.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.244+0000 7f5686ffd640 1 -- 172.21.15.90:0/2154167847 learned_addr learned my addr 172.21.15.90:0/2154167847 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:45.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.244+0000 7f56877fe640 1 --2- 172.21.15.90:0/2154167847 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a86f154550 0x55a86f51db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.244+0000 7f5687fff640 1 --2- 172.21.15.90:0/2154167847 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a86f17d410 0x55a86f51d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.244+0000 7f5686ffd640 1 -- 172.21.15.90:0/2154167847 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a86f154550 msgr2=0x55a86f51db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:45.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.244+0000 7f5686ffd640 1 --2- 172.21.15.90:0/2154167847 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a86f154550 0x55a86f51db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.245+0000 7f5686ffd640 1 -- 172.21.15.90:0/2154167847 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a86f17d410 msgr2=0x55a86f51d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:45.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.245+0000 7f5686ffd640 1 --2- 172.21.15.90:0/2154167847 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a86f17d410 0x55a86f51d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.245+0000 7f5686ffd640 1 -- 172.21.15.90:0/2154167847 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a86f25d7e0 con 0x55a86f15c730 2024-04-25T11:43:45.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.245+0000 7f5687fff640 1 --2- 172.21.15.90:0/2154167847 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a86f17d410 0x55a86f51d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:45.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.245+0000 7f5684ff9640 1 --2- 172.21.15.90:0/841046875 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f56700afc60 0x7f56700b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.245+0000 7f5686ffd640 1 --2- 172.21.15.90:0/2154167847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f15c730 0x55a86f51e0a0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f567c00ab70 tx=0x7f567c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94e867e7f85c2559 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:45.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.245+0000 7f5687fff640 1 --2- 172.21.15.90:0/841046875 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f56700afc60 0x7f56700b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.245+0000 7f5684ff9640 1 -- 172.21.15.90:0/841046875 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f56780dec70 con 0x7f5688104c90 2024-04-25T11:43:45.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.245+0000 7f566dffb640 1 -- 172.21.15.90:0/2154167847 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567c01d070 con 0x55a86f15c730 2024-04-25T11:43:45.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.245+0000 7f566dffb640 1 -- 172.21.15.90:0/2154167847 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f567c018af0 con 0x55a86f15c730 2024-04-25T11:43:45.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.245+0000 7f566dffb640 1 -- 172.21.15.90:0/2154167847 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567c0109c0 con 0x55a86f15c730 2024-04-25T11:43:45.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.246+0000 7f56951e0740 1 -- 172.21.15.90:0/2154167847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f15c730 msgr2=0x55a86f51e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:45.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.246+0000 7f5687fff640 1 --2- 172.21.15.90:0/841046875 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f56700afc60 0x7f56700b2170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x55a86f51a790 tx=0x7f5680005c50 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:45.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.246+0000 7f56951e0740 1 --2- 172.21.15.90:0/2154167847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f15c730 0x55a86f51e0a0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f567c00ab70 tx=0x7f567c00f790 comp rx=0 tx=0).stop 2024-04-25T11:43:45.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.246+0000 7f56951e0740 1 -- 172.21.15.90:0/2154167847 shutdown_connections 2024-04-25T11:43:45.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.246+0000 7f56951e0740 1 --2- 172.21.15.90:0/2154167847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f15c730 0x55a86f51e0a0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.246+0000 7f56951e0740 1 --2- 172.21.15.90:0/2154167847 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a86f154550 0x55a86f51db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.246+0000 7f56951e0740 1 --2- 172.21.15.90:0/2154167847 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a86f17d410 0x55a86f51d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.246+0000 7f56951e0740 1 -- 172.21.15.90:0/2154167847 >> 172.21.15.90:0/2154167847 conn(0x55a86f171560 msgr2=0x55a86f519dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:45.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.246+0000 7f56951e0740 1 -- 172.21.15.90:0/2154167847 shutdown_connections 2024-04-25T11:43:45.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.246+0000 7f56951e0740 1 -- 172.21.15.90:0/2154167847 wait complete. 2024-04-25T11:43:45.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.246+0000 7f56951e0740 1 -- start start 2024-04-25T11:43:45.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.247+0000 7f56951e0740 1 Processor -- start 2024-04-25T11:43:45.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.247+0000 7f56951e0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f154550 0x55a86f527c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.247+0000 7f56951e0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a86f15c730 0x55a86f528180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.247+0000 7f56951e0740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a86f17d410 0x55a86f533e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.247+0000 7f56951e0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a86f52a970 con 0x55a86f154550 2024-04-25T11:43:45.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.247+0000 7f56951e0740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55a86f41d5d0 con 0x55a86f17d410 2024-04-25T11:43:45.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.247+0000 7f56951e0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55a86f52aaf0 con 0x55a86f15c730 2024-04-25T11:43:45.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.247+0000 7f56951e0740 10 client.0 _ll_register_callbacks cb 0x55a86f40b080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:45.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.247+0000 7f56877fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f154550 0x55a86f527c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.247+0000 7f56877fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f154550 0x55a86f527c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42252/0 (socket says 172.21.15.90:42252) 2024-04-25T11:43:45.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.247+0000 7f56877fe640 1 -- 172.21.15.90:0/3400337200 learned_addr learned my addr 172.21.15.90:0/3400337200 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:45.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.247+0000 7f5686ffd640 1 --2- 172.21.15.90:0/3400337200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a86f15c730 0x55a86f528180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.247+0000 7f56877fe640 1 -- 172.21.15.90:0/3400337200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a86f15c730 msgr2=0x55a86f528180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:45.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.247+0000 7f5687fff640 1 --2- 172.21.15.90:0/3400337200 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a86f17d410 0x55a86f533e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.247+0000 7f56877fe640 1 --2- 172.21.15.90:0/3400337200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a86f15c730 0x55a86f528180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.247+0000 7f56877fe640 1 -- 172.21.15.90:0/3400337200 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a86f17d410 msgr2=0x55a86f533e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:45.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.247+0000 7f56877fe640 1 --2- 172.21.15.90:0/3400337200 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a86f17d410 0x55a86f533e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.247+0000 7f56877fe640 1 -- 172.21.15.90:0/3400337200 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a86f2bdf70 con 0x55a86f154550 2024-04-25T11:43:45.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.247+0000 7f5686ffd640 1 --2- 172.21.15.90:0/3400337200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a86f15c730 0x55a86f528180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:45.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.247+0000 7f56877fe640 1 --2- 172.21.15.90:0/3400337200 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f154550 0x55a86f527c40 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f56780a2eb0 tx=0x7f56780a6c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f885df5557ff2046 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:45.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.247+0000 7f5666ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:45.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.248+0000 7f5666ffd640 1 -- 172.21.15.90:0/3400337200 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5678020070 con 0x55a86f154550 2024-04-25T11:43:45.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.248+0000 7f5666ffd640 1 -- 172.21.15.90:0/3400337200 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f56780a6090 con 0x55a86f154550 2024-04-25T11:43:45.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.248+0000 7f56951e0740 1 -- 172.21.15.90:0/3400337200 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f567c018af0 con 0x55a86f154550 2024-04-25T11:43:45.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.248+0000 7f5666ffd640 1 -- 172.21.15.90:0/3400337200 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f56780a6940 con 0x55a86f154550 2024-04-25T11:43:45.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.248+0000 7f56951e0740 10 client.18348 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:45.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.248+0000 7f56951e0740 1 -- 172.21.15.90:0/3400337200 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a86f155170 con 0x55a86f154550 2024-04-25T11:43:45.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.248+0000 7f5666ffd640 1 -- 172.21.15.90:0/3400337200 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f567c018af0 con 0x55a86f154550 2024-04-25T11:43:45.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.248+0000 7f5666ffd640 1 -- 172.21.15.90:0/3400337200 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f56780c10d0 con 0x55a86f154550 2024-04-25T11:43:45.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.248+0000 7f56951e0740 10 client.18348 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:45.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.248+0000 7f56951e0740 10 client.18348 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:43:45.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.248+0000 7f56951e0740 20 client.18348 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:45.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.248+0000 7f56951e0740 1 --2- 172.21.15.90:0/3400337200 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55a86f13c1f0 0x55a86f539fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.248+0000 7f56951e0740 4 client.18348 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:43:45.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.248+0000 7f56951e0740 1 -- 172.21.15.90:0/3400337200 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a86f491710 con 0x55a86f13c1f0 2024-04-25T11:43:45.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.248+0000 7f5686ffd640 1 --2- 172.21.15.90:0/3400337200 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55a86f13c1f0 0x55a86f539fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.249+0000 7f5666ffd640 10 client.18348 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:45.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.249+0000 7f5686ffd640 1 --2- 172.21.15.90:0/3400337200 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55a86f13c1f0 0x55a86f539fd0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28aecbe934dfd961 server_cookie=7bc41b17208ea27e in_seq=0 out_seq=0 2024-04-25T11:43:45.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.250+0000 7f5666ffd640 1 -- 172.21.15.90:0/3400337200 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a86f491710 con 0x55a86f13c1f0 2024-04-25T11:43:45.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.250+0000 7f5666ffd640 10 client.18348 handle_command_reply: tid=0 2024-04-25T11:43:45.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.250+0000 7f56951e0740 1 client.18348 shutdown 2024-04-25T11:43:45.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.250+0000 7f56951e0740 1 -- 172.21.15.90:0/3400337200 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f154550 msgr2=0x55a86f527c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:45.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.250+0000 7f56951e0740 1 --2- 172.21.15.90:0/3400337200 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f154550 0x55a86f527c40 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f56780a2eb0 tx=0x7f56780a6c00 comp rx=0 tx=0).stop 2024-04-25T11:43:45.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.250+0000 7f56951e0740 1 -- 172.21.15.90:0/3400337200 shutdown_connections 2024-04-25T11:43:45.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.250+0000 7f56951e0740 1 --2- 172.21.15.90:0/3400337200 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55a86f13c1f0 0x55a86f539fd0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.250+0000 7f56951e0740 1 --2- 172.21.15.90:0/3400337200 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a86f17d410 0x55a86f533e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.250+0000 7f56951e0740 1 --2- 172.21.15.90:0/3400337200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a86f15c730 0x55a86f528180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.250+0000 7f5666ffd640 0 client.18348 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:45.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.250+0000 7f56951e0740 1 --2- 172.21.15.90:0/3400337200 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f154550 0x55a86f527c40 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.250+0000 7f56951e0740 1 -- 172.21.15.90:0/3400337200 >> 172.21.15.90:0/3400337200 conn(0x55a86f171560 msgr2=0x55a86f51f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:45.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.250+0000 7f56951e0740 1 -- 172.21.15.90:0/3400337200 shutdown_connections 2024-04-25T11:43:45.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.250+0000 7f56951e0740 1 -- 172.21.15.90:0/3400337200 wait complete. 2024-04-25T11:43:45.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.250+0000 7f56951e0740 20 client.18348 trim_cache size 0 max 16384 2024-04-25T11:43:45.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.268+0000 7f56951e0740 1 Processor -- start 2024-04-25T11:43:45.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.268+0000 7f56951e0740 1 -- start start 2024-04-25T11:43:45.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.268+0000 7f56951e0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a86f154550 0x55a86f5222f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.268+0000 7f56951e0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f15c730 0x55a86f529b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.268+0000 7f56951e0740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a86f17d410 0x55a86f53f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.268+0000 7f56951e0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f56881afcf0 con 0x55a86f15c730 2024-04-25T11:43:45.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.268+0000 7f56951e0740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f56881b1830 con 0x55a86f17d410 2024-04-25T11:43:45.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.268+0000 7f56951e0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f56881b03c0 con 0x55a86f154550 2024-04-25T11:43:45.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.268+0000 7f5686ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f15c730 0x55a86f529b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.268+0000 7f5686ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f15c730 0x55a86f529b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42254/0 (socket says 172.21.15.90:42254) 2024-04-25T11:43:45.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.268+0000 7f5686ffd640 1 -- 172.21.15.90:0/3340058313 learned_addr learned my addr 172.21.15.90:0/3340058313 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:45.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.268+0000 7f5686ffd640 1 -- 172.21.15.90:0/3340058313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a86f154550 msgr2=0x55a86f5222f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:45.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.268+0000 7f5686ffd640 1 --2- 172.21.15.90:0/3340058313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a86f154550 0x55a86f5222f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.268+0000 7f5686ffd640 1 -- 172.21.15.90:0/3340058313 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a86f17d410 msgr2=0x55a86f53f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:45.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.268+0000 7f5686ffd640 1 --2- 172.21.15.90:0/3340058313 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a86f17d410 0x55a86f53f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.268+0000 7f5686ffd640 1 -- 172.21.15.90:0/3340058313 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a86f525f80 con 0x55a86f15c730 2024-04-25T11:43:45.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.269+0000 7f5686ffd640 1 --2- 172.21.15.90:0/3340058313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f15c730 0x55a86f529b80 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f567c00a900 tx=0x7f567c00af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a98fd2159c3a9e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:45.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.269+0000 7f566d7fa640 1 -- 172.21.15.90:0/3340058313 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567c02e070 con 0x55a86f15c730 2024-04-25T11:43:45.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.269+0000 7f566d7fa640 1 -- 172.21.15.90:0/3340058313 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f567c0104d0 con 0x55a86f15c730 2024-04-25T11:43:45.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.269+0000 7f566d7fa640 1 -- 172.21.15.90:0/3340058313 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567c01d040 con 0x55a86f15c730 2024-04-25T11:43:45.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.269+0000 7f56951e0740 1 -- 172.21.15.90:0/3340058313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f15c730 msgr2=0x55a86f529b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:45.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.269+0000 7f56951e0740 1 --2- 172.21.15.90:0/3340058313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f15c730 0x55a86f529b80 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f567c00a900 tx=0x7f567c00af90 comp rx=0 tx=0).stop 2024-04-25T11:43:45.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.270+0000 7f56951e0740 1 -- 172.21.15.90:0/3340058313 shutdown_connections 2024-04-25T11:43:45.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.270+0000 7f56951e0740 1 --2- 172.21.15.90:0/3340058313 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a86f17d410 0x55a86f53f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.270+0000 7f56951e0740 1 --2- 172.21.15.90:0/3340058313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f15c730 0x55a86f529b80 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.270+0000 7f56951e0740 1 --2- 172.21.15.90:0/3340058313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a86f154550 0x55a86f5222f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.270+0000 7f56951e0740 1 -- 172.21.15.90:0/3340058313 >> 172.21.15.90:0/3340058313 conn(0x55a86f171560 msgr2=0x55a86f518b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:45.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.270+0000 7f56951e0740 1 -- 172.21.15.90:0/3340058313 shutdown_connections 2024-04-25T11:43:45.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.270+0000 7f56951e0740 1 -- 172.21.15.90:0/3340058313 wait complete. 2024-04-25T11:43:45.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.270+0000 7f56951e0740 1 -- start start 2024-04-25T11:43:45.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.270+0000 7f56951e0740 1 Processor -- start 2024-04-25T11:43:45.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.270+0000 7f56951e0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a86f154550 0x55a86f52bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.270+0000 7f56951e0740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a86f15c730 0x55a86f52c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.270+0000 7f56951e0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f17d410 0x55a86f52c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.270+0000 7f56951e0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f56881b1160 con 0x55a86f17d410 2024-04-25T11:43:45.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.270+0000 7f56951e0740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f56881b0a90 con 0x55a86f15c730 2024-04-25T11:43:45.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.271+0000 7f56951e0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f56881adcf0 con 0x55a86f154550 2024-04-25T11:43:45.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.271+0000 7f56951e0740 10 client.0 _ll_register_callbacks cb 0x7f56780a2cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:45.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.271+0000 7f5687fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f17d410 0x55a86f52c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.271+0000 7f5687fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f17d410 0x55a86f52c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42266/0 (socket says 172.21.15.90:42266) 2024-04-25T11:43:45.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.271+0000 7f5687fff640 1 -- 172.21.15.90:0/1454491272 learned_addr learned my addr 172.21.15.90:0/1454491272 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:45.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.271+0000 7f5686ffd640 1 --2- 172.21.15.90:0/1454491272 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a86f15c730 0x55a86f52c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.271+0000 7f56877fe640 1 --2- 172.21.15.90:0/1454491272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a86f154550 0x55a86f52bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.271+0000 7f5687fff640 1 -- 172.21.15.90:0/1454491272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a86f154550 msgr2=0x55a86f52bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:45.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.271+0000 7f5687fff640 1 --2- 172.21.15.90:0/1454491272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a86f154550 0x55a86f52bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.271+0000 7f5687fff640 1 -- 172.21.15.90:0/1454491272 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a86f15c730 msgr2=0x55a86f52c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:45.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.271+0000 7f5687fff640 1 --2- 172.21.15.90:0/1454491272 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a86f15c730 0x55a86f52c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.271+0000 7f5687fff640 1 -- 172.21.15.90:0/1454491272 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a86f5a14f0 con 0x55a86f17d410 2024-04-25T11:43:45.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.271+0000 7f5687fff640 1 --2- 172.21.15.90:0/1454491272 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f17d410 0x55a86f52c660 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f5680009ca0 tx=0x7f568002a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7a9461555d1723 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:45.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.271+0000 7f5666ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:45.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.271+0000 7f5666ffd640 1 -- 172.21.15.90:0/1454491272 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5680056060 con 0x55a86f17d410 2024-04-25T11:43:45.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.271+0000 7f5666ffd640 1 -- 172.21.15.90:0/1454491272 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f5680030440 con 0x55a86f17d410 2024-04-25T11:43:45.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.271+0000 7f56951e0740 1 -- 172.21.15.90:0/1454491272 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f567c0104d0 con 0x55a86f17d410 2024-04-25T11:43:45.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.271+0000 7f5666ffd640 1 -- 172.21.15.90:0/1454491272 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f568005d050 con 0x55a86f17d410 2024-04-25T11:43:45.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.271+0000 7f56951e0740 10 client.18354 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:45.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.271+0000 7f56951e0740 1 -- 172.21.15.90:0/1454491272 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a86f5a27b0 con 0x55a86f17d410 2024-04-25T11:43:45.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.272+0000 7f5666ffd640 1 -- 172.21.15.90:0/1454491272 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f567c0104d0 con 0x55a86f17d410 2024-04-25T11:43:45.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.272+0000 7f5666ffd640 1 -- 172.21.15.90:0/1454491272 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f568003a6d0 con 0x55a86f17d410 2024-04-25T11:43:45.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.272+0000 7f56951e0740 10 client.18354 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:45.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.272+0000 7f56951e0740 10 client.18354 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:43:45.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.272+0000 7f56951e0740 20 client.18354 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:45.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.272+0000 7f56951e0740 1 --2- 172.21.15.90:0/1454491272 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55a86f111630 0x55a86f5a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.272+0000 7f56951e0740 4 client.18354 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T11:43:45.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.272+0000 7f56951e0740 1 -- 172.21.15.90:0/1454491272 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a86f48b580 con 0x55a86f111630 2024-04-25T11:43:45.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.272+0000 7f56877fe640 1 --2- 172.21.15.90:0/1454491272 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55a86f111630 0x55a86f5a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.273+0000 7f5666ffd640 10 client.18354 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:45.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.273+0000 7f56877fe640 1 --2- 172.21.15.90:0/1454491272 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55a86f111630 0x55a86f5a4fa0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f569500bd47ab9f server_cookie=288b1edce902afc5 in_seq=0 out_seq=0 2024-04-25T11:43:45.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.274+0000 7f5666ffd640 1 -- 172.21.15.90:0/1454491272 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a86f48b580 con 0x55a86f111630 2024-04-25T11:43:45.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.274+0000 7f5666ffd640 10 client.18354 handle_command_reply: tid=0 2024-04-25T11:43:45.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.274+0000 7f56951e0740 1 client.18354 shutdown 2024-04-25T11:43:45.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.274+0000 7f56951e0740 1 -- 172.21.15.90:0/1454491272 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f17d410 msgr2=0x55a86f52c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:45.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.274+0000 7f56951e0740 1 --2- 172.21.15.90:0/1454491272 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f17d410 0x55a86f52c660 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f5680009ca0 tx=0x7f568002a950 comp rx=0 tx=0).stop 2024-04-25T11:43:45.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.274+0000 7f56951e0740 1 -- 172.21.15.90:0/1454491272 shutdown_connections 2024-04-25T11:43:45.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.274+0000 7f56951e0740 1 --2- 172.21.15.90:0/1454491272 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55a86f111630 0x55a86f5a4fa0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.274+0000 7f56951e0740 1 --2- 172.21.15.90:0/1454491272 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a86f17d410 0x55a86f52c660 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.274+0000 7f56951e0740 1 --2- 172.21.15.90:0/1454491272 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55a86f15c730 0x55a86f52c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.274+0000 7f5666ffd640 0 client.18354 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:45.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.274+0000 7f56951e0740 1 --2- 172.21.15.90:0/1454491272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a86f154550 0x55a86f52bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.274+0000 7f56951e0740 1 -- 172.21.15.90:0/1454491272 >> 172.21.15.90:0/1454491272 conn(0x55a86f171560 msgr2=0x55a86f518b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:45.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.275+0000 7f56951e0740 1 -- 172.21.15.90:0/1454491272 shutdown_connections 2024-04-25T11:43:45.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.275+0000 7f56951e0740 1 -- 172.21.15.90:0/1454491272 wait complete. 2024-04-25T11:43:45.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.275+0000 7f56951e0740 20 client.18354 trim_cache size 0 max 16384 2024-04-25T11:43:45.266 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:43:45.266 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-25T11:43:45.266 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "735b9e1a-02fd-4bb0-88bd-ac24a4415926", 2024-04-25T11:43:45.266 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-25T11:43:45.266 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:43:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.275+0000 7f568e253640 1 -- 172.21.15.90:0/841046875 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f56700afc60 msgr2=0x7f56700b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.275+0000 7f568e253640 1 --2- 172.21.15.90:0/841046875 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f56700afc60 0x7f56700b2170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x55a86f51a790 tx=0x7f5680005c50 comp rx=0 tx=0).stop 2024-04-25T11:43:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.275+0000 7f568e253640 1 -- 172.21.15.90:0/841046875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5688104c90 msgr2=0x7f568819a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.275+0000 7f568e253640 1 --2- 172.21.15.90:0/841046875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5688104c90 0x7f568819a6d0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f567800d950 tx=0x7f567800de10 comp rx=0 tx=0).stop 2024-04-25T11:43:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.275+0000 7f568e253640 1 -- 172.21.15.90:0/841046875 shutdown_connections 2024-04-25T11:43:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.275+0000 7f568e253640 1 --2- 172.21.15.90:0/841046875 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f56700afc60 0x7f56700b2170 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.275+0000 7f568e253640 1 --2- 172.21.15.90:0/841046875 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f568810e050 0x7f568819eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.275+0000 7f568e253640 1 --2- 172.21.15.90:0/841046875 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f568810b6f0 0x7f568819ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.275+0000 7f568e253640 1 --2- 172.21.15.90:0/841046875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5688104c90 0x7f568819a6d0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.275+0000 7f568e253640 1 -- 172.21.15.90:0/841046875 >> 172.21.15.90:0/841046875 conn(0x7f5688078100 msgr2=0x7f5688101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.275+0000 7f568e253640 1 -- 172.21.15.90:0/841046875 shutdown_connections 2024-04-25T11:43:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.275+0000 7f568e253640 1 -- 172.21.15.90:0/841046875 wait complete. 2024-04-25T11:43:45.273 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:43:45.274 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:43:45.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.397+0000 7f7bab796640 1 -- 172.21.15.90:0/3506028147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ba41042d0 msgr2=0x7f7ba4110650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:45.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.397+0000 7f7bab796640 1 --2- 172.21.15.90:0/3506028147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ba41042d0 0x7f7ba4110650 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c009ca0 tx=0x7f7b9c02a8c0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.397+0000 7f7bab796640 1 -- 172.21.15.90:0/3506028147 shutdown_connections 2024-04-25T11:43:45.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.397+0000 7f7bab796640 1 --2- 172.21.15.90:0/3506028147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ba41042d0 0x7f7ba4110650 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.397+0000 7f7bab796640 1 --2- 172.21.15.90:0/3506028147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ba4109fd0 0x7f7ba4103c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.397+0000 7f7bab796640 1 --2- 172.21.15.90:0/3506028147 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7ba4109b30 0x7f7ba4103720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.397+0000 7f7bab796640 1 -- 172.21.15.90:0/3506028147 >> 172.21.15.90:0/3506028147 conn(0x7f7ba4079280 msgr2=0x7f7ba40796c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:45.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.397+0000 7f7bab796640 1 -- 172.21.15.90:0/3506028147 shutdown_connections 2024-04-25T11:43:45.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.397+0000 7f7bab796640 1 -- 172.21.15.90:0/3506028147 wait complete. 2024-04-25T11:43:45.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.398+0000 7f7bab796640 1 Processor -- start 2024-04-25T11:43:45.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.398+0000 7f7bab796640 1 -- start start 2024-04-25T11:43:45.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.398+0000 7f7bab796640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7ba41042d0 0x7f7ba4074ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.398+0000 7f7bab796640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ba4109b30 0x7f7ba40753e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.398+0000 7f7bab796640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ba4109fd0 0x7f7ba4075920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.398+0000 7f7bab796640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba4113d40 con 0x7f7ba4109b30 2024-04-25T11:43:45.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.398+0000 7f7bab796640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba4113bc0 con 0x7f7ba41042d0 2024-04-25T11:43:45.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.398+0000 7f7bab796640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba4113ec0 con 0x7f7ba4109fd0 2024-04-25T11:43:45.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.398+0000 7f7ba8d31640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ba4109b30 0x7f7ba40753e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.398+0000 7f7ba8d31640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ba4109b30 0x7f7ba40753e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42280/0 (socket says 172.21.15.90:42280) 2024-04-25T11:43:45.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.398+0000 7f7ba8d31640 1 -- 172.21.15.90:0/273975268 learned_addr learned my addr 172.21.15.90:0/273975268 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:45.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.398+0000 7f7ba9532640 1 --2- 172.21.15.90:0/273975268 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7ba41042d0 0x7f7ba4074ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.398+0000 7f7ba9d33640 1 --2- 172.21.15.90:0/273975268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ba4109fd0 0x7f7ba4075920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.398+0000 7f7ba8d31640 1 -- 172.21.15.90:0/273975268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ba4109fd0 msgr2=0x7f7ba4075920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:45.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.398+0000 7f7ba8d31640 1 --2- 172.21.15.90:0/273975268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ba4109fd0 0x7f7ba4075920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.398+0000 7f7ba8d31640 1 -- 172.21.15.90:0/273975268 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7ba41042d0 msgr2=0x7f7ba4074ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:45.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.398+0000 7f7ba8d31640 1 --2- 172.21.15.90:0/273975268 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7ba41042d0 0x7f7ba4074ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.398+0000 7f7ba8d31640 1 -- 172.21.15.90:0/273975268 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ba41a8a60 con 0x7f7ba4109b30 2024-04-25T11:43:45.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.398+0000 7f7ba8d31640 1 --2- 172.21.15.90:0/273975268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ba4109b30 0x7f7ba40753e0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0005d00 tx=0x7f7ba000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:45.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.399+0000 7f7b9a7fc640 1 -- 172.21.15.90:0/273975268 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0017070 con 0x7f7ba4109b30 2024-04-25T11:43:45.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.399+0000 7f7b9a7fc640 1 -- 172.21.15.90:0/273975268 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ba0009c30 con 0x7f7ba4109b30 2024-04-25T11:43:45.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.399+0000 7f7bab796640 1 -- 172.21.15.90:0/273975268 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ba41a8d50 con 0x7f7ba4109b30 2024-04-25T11:43:45.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.399+0000 7f7b9a7fc640 1 -- 172.21.15.90:0/273975268 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba000f620 con 0x7f7ba4109b30 2024-04-25T11:43:45.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.399+0000 7f7bab796640 1 -- 172.21.15.90:0/273975268 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ba41a9190 con 0x7f7ba4109b30 2024-04-25T11:43:45.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.401+0000 7f7b9a7fc640 1 -- 172.21.15.90:0/273975268 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f7ba001a070 con 0x7f7ba4109b30 2024-04-25T11:43:45.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.401+0000 7f7bb2646740 1 Processor -- start 2024-04-25T11:43:45.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.401+0000 7f7bb2646740 1 -- start start 2024-04-25T11:43:45.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.402+0000 7f7bb2646740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55de7deb0390 0x55de7e250a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.402+0000 7f7bb2646740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de7de874d0 0x55de7e250fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.402+0000 7f7bb2646740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de8f6b0 0x55de7e25d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.402+0000 7f7bb2646740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55de7e1b6d80 con 0x55de7de8f6b0 2024-04-25T11:43:45.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.402+0000 7f7bb2646740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55de7e1708b0 con 0x55de7deb0390 2024-04-25T11:43:45.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.402+0000 7f7bb2646740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55de7e1b2d40 con 0x55de7de874d0 2024-04-25T11:43:45.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.402+0000 7f7ba8d31640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de8f6b0 0x55de7e25d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.402+0000 7f7ba8d31640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de8f6b0 0x55de7e25d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42286/0 (socket says 172.21.15.90:42286) 2024-04-25T11:43:45.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.402+0000 7f7ba8d31640 1 -- 172.21.15.90:0/3306674699 learned_addr learned my addr 172.21.15.90:0/3306674699 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:45.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.402+0000 7f7ba9d33640 1 --2- 172.21.15.90:0/3306674699 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55de7deb0390 0x55de7e250a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.402+0000 7f7b9a7fc640 1 --2- 172.21.15.90:0/273975268 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f7b900afc60 0x7f7b900b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.402+0000 7f7ba9532640 1 --2- 172.21.15.90:0/3306674699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de7de874d0 0x55de7e250fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.402+0000 7f7ba9d33640 1 --2- 172.21.15.90:0/273975268 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f7b900afc60 0x7f7b900b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.402+0000 7f7b9a7fc640 1 -- 172.21.15.90:0/273975268 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f7ba00deaf0 con 0x7f7ba4109b30 2024-04-25T11:43:45.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.402+0000 7f7ba8d31640 1 -- 172.21.15.90:0/3306674699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de7de874d0 msgr2=0x55de7e250fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:45.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.402+0000 7f7ba8d31640 1 --2- 172.21.15.90:0/3306674699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de7de874d0 0x55de7e250fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.402+0000 7f7ba8d31640 1 -- 172.21.15.90:0/3306674699 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55de7deb0390 msgr2=0x55de7e250a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:45.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.402+0000 7f7ba8d31640 1 --2- 172.21.15.90:0/3306674699 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55de7deb0390 0x55de7e250a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.402+0000 7f7ba8d31640 1 -- 172.21.15.90:0/3306674699 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de7e141a40 con 0x55de7de8f6b0 2024-04-25T11:43:45.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.402+0000 7f7ba8d31640 1 --2- 172.21.15.90:0/3306674699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de8f6b0 0x55de7e25d8f0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f7ba00a2c80 tx=0x7f7ba0010c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18d2b98101d6fd67 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:45.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.402+0000 7f7ba9d33640 1 --2- 172.21.15.90:0/273975268 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f7b900afc60 0x7f7b900b2170 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x55de7e24dbd0 tx=0x7f7b9c02a8c0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:45.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.403+0000 7f7b8f7fe640 1 -- 172.21.15.90:0/3306674699 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0021070 con 0x55de7de8f6b0 2024-04-25T11:43:45.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.403+0000 7f7b8f7fe640 1 -- 172.21.15.90:0/3306674699 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ba00af3b0 con 0x55de7de8f6b0 2024-04-25T11:43:45.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.403+0000 7f7b8f7fe640 1 -- 172.21.15.90:0/3306674699 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0009de0 con 0x55de7de8f6b0 2024-04-25T11:43:45.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.403+0000 7f7bb2646740 1 -- 172.21.15.90:0/3306674699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de8f6b0 msgr2=0x55de7e25d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:45.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.403+0000 7f7bb2646740 1 --2- 172.21.15.90:0/3306674699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de8f6b0 0x55de7e25d8f0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f7ba00a2c80 tx=0x7f7ba0010c00 comp rx=0 tx=0).stop 2024-04-25T11:43:45.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.403+0000 7f7bb2646740 1 -- 172.21.15.90:0/3306674699 shutdown_connections 2024-04-25T11:43:45.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.403+0000 7f7bb2646740 1 --2- 172.21.15.90:0/3306674699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de8f6b0 0x55de7e25d8f0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.403+0000 7f7bb2646740 1 --2- 172.21.15.90:0/3306674699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de7de874d0 0x55de7e250fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.403+0000 7f7bb2646740 1 --2- 172.21.15.90:0/3306674699 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55de7deb0390 0x55de7e250a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.403+0000 7f7bb2646740 1 -- 172.21.15.90:0/3306674699 >> 172.21.15.90:0/3306674699 conn(0x55de7dea44e0 msgr2=0x55de7e24d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:45.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.404+0000 7f7bb2646740 1 -- 172.21.15.90:0/3306674699 shutdown_connections 2024-04-25T11:43:45.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.404+0000 7f7bb2646740 1 -- 172.21.15.90:0/3306674699 wait complete. 2024-04-25T11:43:45.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.404+0000 7f7bb2646740 1 -- start start 2024-04-25T11:43:45.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.404+0000 7f7bb2646740 1 Processor -- start 2024-04-25T11:43:45.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.404+0000 7f7bb2646740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de874d0 0x55de7e25b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.404+0000 7f7bb2646740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55de7de8f6b0 0x55de7e25bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.404+0000 7f7bb2646740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de7deb0390 0x55de7e2613b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.404+0000 7f7bb2646740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55de7e25df20 con 0x55de7de874d0 2024-04-25T11:43:45.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.404+0000 7f7bb2646740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55de7e13dd10 con 0x55de7de8f6b0 2024-04-25T11:43:45.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.404+0000 7f7bb2646740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55de7e25e0a0 con 0x55de7deb0390 2024-04-25T11:43:45.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.404+0000 7f7bb2646740 10 client.0 _ll_register_callbacks cb 0x55de7e16a210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:45.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.405+0000 7f7ba9532640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de874d0 0x55de7e25b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.405+0000 7f7ba9532640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de874d0 0x55de7e25b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42302/0 (socket says 172.21.15.90:42302) 2024-04-25T11:43:45.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.405+0000 7f7ba9532640 1 -- 172.21.15.90:0/4290628704 learned_addr learned my addr 172.21.15.90:0/4290628704 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:45.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.405+0000 7f7ba8d31640 1 --2- 172.21.15.90:0/4290628704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55de7de8f6b0 0x55de7e25bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.405+0000 7f7ba9d33640 1 --2- 172.21.15.90:0/4290628704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de7deb0390 0x55de7e2613b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.405+0000 7f7ba9532640 1 -- 172.21.15.90:0/4290628704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de7deb0390 msgr2=0x55de7e2613b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:45.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.405+0000 7f7ba9532640 1 --2- 172.21.15.90:0/4290628704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de7deb0390 0x55de7e2613b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.405+0000 7f7ba9532640 1 -- 172.21.15.90:0/4290628704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55de7de8f6b0 msgr2=0x55de7e25bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:45.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.405+0000 7f7ba9532640 1 --2- 172.21.15.90:0/4290628704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55de7de8f6b0 0x55de7e25bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.405+0000 7f7ba9532640 1 -- 172.21.15.90:0/4290628704 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de7df927d0 con 0x55de7de874d0 2024-04-25T11:43:45.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.405+0000 7f7ba9532640 1 --2- 172.21.15.90:0/4290628704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de874d0 0x55de7e25b630 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f7b9400bad0 tx=0x7f7b9400bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d727fa57a6cf2f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:45.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.405+0000 7f7b8cff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:45.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.405+0000 7f7b8cff9640 1 -- 172.21.15.90:0/4290628704 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b94003e00 con 0x55de7de874d0 2024-04-25T11:43:45.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.405+0000 7f7b8cff9640 1 -- 172.21.15.90:0/4290628704 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f7b94011440 con 0x55de7de874d0 2024-04-25T11:43:45.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.405+0000 7f7bb2646740 1 -- 172.21.15.90:0/4290628704 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ba00af3b0 con 0x55de7de874d0 2024-04-25T11:43:45.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.405+0000 7f7b8cff9640 1 -- 172.21.15.90:0/4290628704 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7b94011680 con 0x55de7de874d0 2024-04-25T11:43:45.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.405+0000 7f7bb2646740 10 client.18366 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:45.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.405+0000 7f7bb2646740 1 -- 172.21.15.90:0/4290628704 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55de7dec83f0 con 0x55de7de874d0 2024-04-25T11:43:45.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.405+0000 7f7b8cff9640 1 -- 172.21.15.90:0/4290628704 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ba00af3b0 con 0x55de7de874d0 2024-04-25T11:43:45.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.406+0000 7f7b8cff9640 1 -- 172.21.15.90:0/4290628704 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f7b9401b9a0 con 0x55de7de874d0 2024-04-25T11:43:45.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.406+0000 7f7bb2646740 10 client.18366 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:45.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.406+0000 7f7bb2646740 10 client.18366 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:43:45.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.406+0000 7f7bb2646740 20 client.18366 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:45.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.406+0000 7f7bb2646740 1 --2- 172.21.15.90:0/4290628704 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55de7de6ef60 0x55de7e26d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.406+0000 7f7bb2646740 4 client.18366 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:43:45.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.406+0000 7f7bb2646740 1 -- 172.21.15.90:0/4290628704 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55de7e1c4990 con 0x55de7de6ef60 2024-04-25T11:43:45.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.406+0000 7f7ba8d31640 1 --2- 172.21.15.90:0/4290628704 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55de7de6ef60 0x55de7e26d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.406+0000 7f7b8cff9640 10 client.18366 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:45.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.406+0000 7f7ba8d31640 1 --2- 172.21.15.90:0/4290628704 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55de7de6ef60 0x55de7e26d6f0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df11a1080083199d server_cookie=85fed6f623ac5a19 in_seq=0 out_seq=0 2024-04-25T11:43:45.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.407+0000 7f7b8cff9640 1 -- 172.21.15.90:0/4290628704 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55de7e1c4990 con 0x55de7de6ef60 2024-04-25T11:43:45.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.407+0000 7f7b8cff9640 10 client.18366 handle_command_reply: tid=0 2024-04-25T11:43:45.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.407+0000 7f7bb2646740 1 client.18366 shutdown 2024-04-25T11:43:45.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.407+0000 7f7bb2646740 1 -- 172.21.15.90:0/4290628704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de874d0 msgr2=0x55de7e25b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:45.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.407+0000 7f7bb2646740 1 --2- 172.21.15.90:0/4290628704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de874d0 0x55de7e25b630 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f7b9400bad0 tx=0x7f7b9400bf90 comp rx=0 tx=0).stop 2024-04-25T11:43:45.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.407+0000 7f7bb2646740 1 -- 172.21.15.90:0/4290628704 shutdown_connections 2024-04-25T11:43:45.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.407+0000 7f7bb2646740 1 --2- 172.21.15.90:0/4290628704 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55de7de6ef60 0x55de7e26d6f0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.407+0000 7f7bb2646740 1 --2- 172.21.15.90:0/4290628704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de7deb0390 0x55de7e2613b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.407+0000 7f7bb2646740 1 --2- 172.21.15.90:0/4290628704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55de7de8f6b0 0x55de7e25bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.407+0000 7f7b8cff9640 0 client.18366 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:45.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.407+0000 7f7bb2646740 1 --2- 172.21.15.90:0/4290628704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de874d0 0x55de7e25b630 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.407+0000 7f7bb2646740 1 -- 172.21.15.90:0/4290628704 >> 172.21.15.90:0/4290628704 conn(0x55de7dea44e0 msgr2=0x55de7e252780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:45.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.408+0000 7f7bb2646740 1 -- 172.21.15.90:0/4290628704 shutdown_connections 2024-04-25T11:43:45.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.408+0000 7f7bb2646740 1 -- 172.21.15.90:0/4290628704 wait complete. 2024-04-25T11:43:45.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.408+0000 7f7bb2646740 20 client.18366 trim_cache size 0 max 16384 2024-04-25T11:43:45.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.425+0000 7f7bb2646740 1 Processor -- start 2024-04-25T11:43:45.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.425+0000 7f7bb2646740 1 -- start start 2024-04-25T11:43:45.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.426+0000 7f7bb2646740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de7de6ef60 0x55de7e25d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.426+0000 7f7bb2646740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de874d0 0x55de7e272eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.426+0000 7f7bb2646740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55de7de8f6b0 0x55de7e2733f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.426+0000 7f7bb2646740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba41b1e50 con 0x55de7de874d0 2024-04-25T11:43:45.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.426+0000 7f7bb2646740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba41b3990 con 0x55de7de8f6b0 2024-04-25T11:43:45.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.426+0000 7f7bb2646740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba41b2520 con 0x55de7de6ef60 2024-04-25T11:43:45.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.426+0000 7f7ba8d31640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de874d0 0x55de7e272eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.426+0000 7f7ba8d31640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de874d0 0x55de7e272eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42316/0 (socket says 172.21.15.90:42316) 2024-04-25T11:43:45.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.426+0000 7f7ba8d31640 1 -- 172.21.15.90:0/277665435 learned_addr learned my addr 172.21.15.90:0/277665435 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:45.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.426+0000 7f7ba9d33640 1 --2- 172.21.15.90:0/277665435 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55de7de8f6b0 0x55de7e2733f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.426+0000 7f7ba9532640 1 --2- 172.21.15.90:0/277665435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de7de6ef60 0x55de7e25d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.426+0000 7f7ba8d31640 1 -- 172.21.15.90:0/277665435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de7de6ef60 msgr2=0x55de7e25d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:45.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.426+0000 7f7ba8d31640 1 --2- 172.21.15.90:0/277665435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de7de6ef60 0x55de7e25d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.426+0000 7f7ba8d31640 1 -- 172.21.15.90:0/277665435 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55de7de8f6b0 msgr2=0x55de7e2733f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:45.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.426+0000 7f7ba8d31640 1 --2- 172.21.15.90:0/277665435 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55de7de8f6b0 0x55de7e2733f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.426+0000 7f7ba8d31640 1 -- 172.21.15.90:0/277665435 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de7de880f0 con 0x55de7de874d0 2024-04-25T11:43:45.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.426+0000 7f7ba8d31640 1 --2- 172.21.15.90:0/277665435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de874d0 0x55de7e272eb0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f7ba00a6b90 tx=0x7f7ba00a2b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac10b17847a6725f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:45.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.426+0000 7f7b8effd640 1 -- 172.21.15.90:0/277665435 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0021040 con 0x55de7de874d0 2024-04-25T11:43:45.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.426+0000 7f7b8effd640 1 -- 172.21.15.90:0/277665435 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ba0009de0 con 0x55de7de874d0 2024-04-25T11:43:45.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.426+0000 7f7b8effd640 1 -- 172.21.15.90:0/277665435 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba00dc430 con 0x55de7de874d0 2024-04-25T11:43:45.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.427+0000 7f7bb2646740 1 -- 172.21.15.90:0/277665435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de874d0 msgr2=0x55de7e272eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:45.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.427+0000 7f7bb2646740 1 --2- 172.21.15.90:0/277665435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de874d0 0x55de7e272eb0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f7ba00a6b90 tx=0x7f7ba00a2b70 comp rx=0 tx=0).stop 2024-04-25T11:43:45.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.427+0000 7f7bb2646740 1 -- 172.21.15.90:0/277665435 shutdown_connections 2024-04-25T11:43:45.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.427+0000 7f7bb2646740 1 --2- 172.21.15.90:0/277665435 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55de7de8f6b0 0x55de7e2733f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.427+0000 7f7bb2646740 1 --2- 172.21.15.90:0/277665435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de874d0 0x55de7e272eb0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.427+0000 7f7bb2646740 1 --2- 172.21.15.90:0/277665435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de7de6ef60 0x55de7e25d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.427+0000 7f7bb2646740 1 -- 172.21.15.90:0/277665435 >> 172.21.15.90:0/277665435 conn(0x55de7dea44e0 msgr2=0x55de7e25cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:45.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.427+0000 7f7bb2646740 1 -- 172.21.15.90:0/277665435 shutdown_connections 2024-04-25T11:43:45.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.427+0000 7f7bb2646740 1 -- 172.21.15.90:0/277665435 wait complete. 2024-04-25T11:43:45.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.428+0000 7f7bb2646740 1 -- start start 2024-04-25T11:43:45.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.428+0000 7f7bb2646740 1 Processor -- start 2024-04-25T11:43:45.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.428+0000 7f7bb2646740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55de7de6ef60 0x55de7e262df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.428+0000 7f7bb2646740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de874d0 0x55de7e263330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.428+0000 7f7bb2646740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de7de8f6b0 0x55de7e2cda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.428+0000 7f7bb2646740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba41b32c0 con 0x55de7de874d0 2024-04-25T11:43:45.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.428+0000 7f7bb2646740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba41b2bf0 con 0x55de7de6ef60 2024-04-25T11:43:45.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.428+0000 7f7bb2646740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba41afe50 con 0x55de7de8f6b0 2024-04-25T11:43:45.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.428+0000 7f7bb2646740 10 client.0 _ll_register_callbacks cb 0x7f7b94004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:45.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.428+0000 7f7ba8d31640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de874d0 0x55de7e263330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.428+0000 7f7ba8d31640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de874d0 0x55de7e263330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42320/0 (socket says 172.21.15.90:42320) 2024-04-25T11:43:45.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.428+0000 7f7ba8d31640 1 -- 172.21.15.90:0/582090303 learned_addr learned my addr 172.21.15.90:0/582090303 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:45.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.428+0000 7f7ba9532640 1 --2- 172.21.15.90:0/582090303 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55de7de6ef60 0x55de7e262df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.429+0000 7f7ba8d31640 1 -- 172.21.15.90:0/582090303 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de7de8f6b0 msgr2=0x55de7e2cda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:45.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.429+0000 7f7ba8d31640 1 --2- 172.21.15.90:0/582090303 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de7de8f6b0 0x55de7e2cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.429+0000 7f7ba8d31640 1 -- 172.21.15.90:0/582090303 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55de7de6ef60 msgr2=0x55de7e262df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:45.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.429+0000 7f7ba8d31640 1 --2- 172.21.15.90:0/582090303 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55de7de6ef60 0x55de7e262df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.429+0000 7f7ba8d31640 1 -- 172.21.15.90:0/582090303 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de7e1774b0 con 0x55de7de874d0 2024-04-25T11:43:45.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.429+0000 7f7ba8d31640 1 --2- 172.21.15.90:0/582090303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de874d0 0x55de7e263330 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f7ba00dc260 tx=0x7f7ba00a6400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f888f911ef26d90 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:45.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.429+0000 7f7b8cff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:45.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.429+0000 7f7b8cff9640 1 -- 172.21.15.90:0/582090303 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0021050 con 0x55de7de874d0 2024-04-25T11:43:45.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.429+0000 7f7b8cff9640 1 -- 172.21.15.90:0/582090303 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f7ba00a31f0 con 0x55de7de874d0 2024-04-25T11:43:45.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.429+0000 7f7bb2646740 1 -- 172.21.15.90:0/582090303 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ba0009de0 con 0x55de7de874d0 2024-04-25T11:43:45.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.429+0000 7f7b8cff9640 1 -- 172.21.15.90:0/582090303 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7ba00aafb0 con 0x55de7de874d0 2024-04-25T11:43:45.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.429+0000 7f7bb2646740 10 client.18372 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:45.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.429+0000 7f7bb2646740 1 -- 172.21.15.90:0/582090303 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55de7e2d50b0 con 0x55de7de874d0 2024-04-25T11:43:45.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.429+0000 7f7b8cff9640 1 -- 172.21.15.90:0/582090303 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ba0009de0 con 0x55de7de874d0 2024-04-25T11:43:45.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.429+0000 7f7b8cff9640 1 -- 172.21.15.90:0/582090303 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f7ba00a5e60 con 0x55de7de874d0 2024-04-25T11:43:45.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.429+0000 7f7bb2646740 10 client.18372 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:45.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.429+0000 7f7bb2646740 10 client.18372 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:43:45.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.429+0000 7f7bb2646740 20 client.18372 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:45.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.429+0000 7f7bb2646740 1 --2- 172.21.15.90:0/582090303 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55de7deb0390 0x55de7e2d7340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:45.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.429+0000 7f7bb2646740 4 client.18372 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:43:45.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.429+0000 7f7bb2646740 1 -- 172.21.15.90:0/582090303 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55de7e1be800 con 0x55de7deb0390 2024-04-25T11:43:45.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.430+0000 7f7ba9532640 1 --2- 172.21.15.90:0/582090303 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55de7deb0390 0x55de7e2d7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:45.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.430+0000 7f7b8cff9640 10 client.18372 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:45.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.430+0000 7f7ba9532640 1 --2- 172.21.15.90:0/582090303 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55de7deb0390 0x55de7e2d7340 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=764aa253e2a5a9dd server_cookie=ea3a73099dc6dfa4 in_seq=0 out_seq=0 2024-04-25T11:43:45.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.430+0000 7f7b8cff9640 1 -- 172.21.15.90:0/582090303 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55de7e1be800 con 0x55de7deb0390 2024-04-25T11:43:45.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.430+0000 7f7b8cff9640 10 client.18372 handle_command_reply: tid=0 2024-04-25T11:43:45.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.430+0000 7f7bb2646740 1 client.18372 shutdown 2024-04-25T11:43:45.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.430+0000 7f7bb2646740 1 -- 172.21.15.90:0/582090303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de874d0 msgr2=0x55de7e263330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:45.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.430+0000 7f7bb2646740 1 --2- 172.21.15.90:0/582090303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de874d0 0x55de7e263330 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f7ba00dc260 tx=0x7f7ba00a6400 comp rx=0 tx=0).stop 2024-04-25T11:43:45.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.430+0000 7f7bb2646740 1 -- 172.21.15.90:0/582090303 shutdown_connections 2024-04-25T11:43:45.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.430+0000 7f7bb2646740 1 --2- 172.21.15.90:0/582090303 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55de7deb0390 0x55de7e2d7340 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.430+0000 7f7bb2646740 1 --2- 172.21.15.90:0/582090303 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de7de8f6b0 0x55de7e2cda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.430+0000 7f7b8cff9640 0 client.18372 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:45.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.430+0000 7f7bb2646740 1 --2- 172.21.15.90:0/582090303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de7de874d0 0x55de7e263330 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.430+0000 7f7bb2646740 1 --2- 172.21.15.90:0/582090303 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55de7de6ef60 0x55de7e262df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.430+0000 7f7bb2646740 1 -- 172.21.15.90:0/582090303 >> 172.21.15.90:0/582090303 conn(0x55de7dea44e0 msgr2=0x55de7e25d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:45.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.431+0000 7f7bb2646740 1 -- 172.21.15.90:0/582090303 shutdown_connections 2024-04-25T11:43:45.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.431+0000 7f7bb2646740 1 -- 172.21.15.90:0/582090303 wait complete. 2024-04-25T11:43:45.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.431+0000 7f7bb2646740 20 client.18372 trim_cache size 0 max 16384 2024-04-25T11:43:45.425 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:43:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T11:43:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-25T11:43:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: "735b9e1a-02fd-4bb0-88bd-ac24a4415926": { 2024-04-25T11:43:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-25T11:43:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "735b9e1a-02fd-4bb0-88bd-ac24a4415926", 2024-04-25T11:43:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-25T11:43:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:43:45.425 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:43:45.425 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:43:45.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.431+0000 7f7bab796640 1 -- 172.21.15.90:0/273975268 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f7b900afc60 msgr2=0x7f7b900b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:45.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.431+0000 7f7bab796640 1 --2- 172.21.15.90:0/273975268 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f7b900afc60 0x7f7b900b2170 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x55de7e24dbd0 tx=0x7f7b9c02a8c0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.431+0000 7f7bab796640 1 -- 172.21.15.90:0/273975268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ba4109b30 msgr2=0x7f7ba40753e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:45.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.431+0000 7f7bab796640 1 --2- 172.21.15.90:0/273975268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ba4109b30 0x7f7ba40753e0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0005d00 tx=0x7f7ba000dde0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.431+0000 7f7bab796640 1 -- 172.21.15.90:0/273975268 shutdown_connections 2024-04-25T11:43:45.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.431+0000 7f7bab796640 1 --2- 172.21.15.90:0/273975268 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f7b900afc60 0x7f7b900b2170 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.431+0000 7f7bab796640 1 --2- 172.21.15.90:0/273975268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ba4109fd0 0x7f7ba4075920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.431+0000 7f7bab796640 1 --2- 172.21.15.90:0/273975268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ba4109b30 0x7f7ba40753e0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.431+0000 7f7bab796640 1 --2- 172.21.15.90:0/273975268 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7ba41042d0 0x7f7ba4074ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:45.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.431+0000 7f7bab796640 1 -- 172.21.15.90:0/273975268 >> 172.21.15.90:0/273975268 conn(0x7f7ba4079280 msgr2=0x7f7ba4100e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:45.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.431+0000 7f7bab796640 1 -- 172.21.15.90:0/273975268 shutdown_connections 2024-04-25T11:43:45.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:43:45.431+0000 7f7bab796640 1 -- 172.21.15.90:0/273975268 wait complete. 2024-04-25T11:43:45.429 INFO:tasks.cephfs.filesystem:scrub status for tag:735b9e1a-02fd-4bb0-88bd-ac24a4415926 - {'path': '/', 'tag': '735b9e1a-02fd-4bb0-88bd-ac24a4415926', 'options': 'recursive,force'} 2024-04-25T11:43:46.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:46 smithi156 ceph-mon[33045]: pgmap v267: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-04-25T11:43:46.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:46 smithi156 ceph-mon[33045]: scrub queued for path: / 2024-04-25T11:43:46.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:46 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:43:46.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:46 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:43:46.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:46 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:43:46.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:46 smithi090 ceph-mon[29575]: pgmap v267: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-04-25T11:43:46.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:46 smithi090 ceph-mon[29575]: scrub queued for path: / 2024-04-25T11:43:46.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:46 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:43:46.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:46 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:43:46.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:46 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:43:46.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:46 smithi138 ceph-mon[32328]: pgmap v267: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-04-25T11:43:46.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:46 smithi138 ceph-mon[32328]: scrub queued for path: / 2024-04-25T11:43:46.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:46 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:43:46.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:46 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:43:46.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:46 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:43:47.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:47 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:43:47.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:47 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:43:47.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:47 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:43:47.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:47 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:43:47.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:47 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:43:47.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:47 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:43:48.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:48 smithi090 ceph-mon[29575]: pgmap v268: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.3 KiB/s wr, 12 op/s 2024-04-25T11:43:48.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:48 smithi138 ceph-mon[32328]: pgmap v268: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.3 KiB/s wr, 12 op/s 2024-04-25T11:43:48.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:48 smithi156 ceph-mon[33045]: pgmap v268: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.3 KiB/s wr, 12 op/s 2024-04-25T11:43:50.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:50 smithi090 ceph-mon[29575]: pgmap v269: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-04-25T11:43:50.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:50 smithi138 ceph-mon[32328]: pgmap v269: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-04-25T11:43:50.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:50 smithi156 ceph-mon[33045]: pgmap v269: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-04-25T11:43:51.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:51 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:43:51.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:51 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:43:51.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:51 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:43:52.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:52 smithi090 ceph-mon[29575]: pgmap v270: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 12 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-04-25T11:43:52.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:52 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:43:52.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:52 smithi138 ceph-mon[32328]: pgmap v270: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 12 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-04-25T11:43:52.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:52 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:43:52.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:52 smithi156 ceph-mon[33045]: pgmap v270: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 12 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-04-25T11:43:52.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:52 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:43:54.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:54 smithi090 ceph-mon[29575]: pgmap v271: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:43:54.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:54 smithi138 ceph-mon[32328]: pgmap v271: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:43:54.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:54 smithi156 ceph-mon[33045]: pgmap v271: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:43:56.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:56 smithi090 ceph-mon[29575]: pgmap v272: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:43:56.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:56 smithi138 ceph-mon[32328]: pgmap v272: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:43:56.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:56 smithi156 ceph-mon[33045]: pgmap v272: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:43:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.507+0000 7f1272cd1640 1 -- 172.21.15.90:0/1651413031 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f126c10a010 msgr2=0x7f126c103c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:57.498 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.507+0000 7f1272cd1640 1 --2- 172.21.15.90:0/1651413031 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f126c10a010 0x7f126c103c50 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f1258009900 tx=0x7f125802a410 comp rx=0 tx=0).stop 2024-04-25T11:43:57.498 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.507+0000 7f1272cd1640 1 -- 172.21.15.90:0/1651413031 shutdown_connections 2024-04-25T11:43:57.498 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.507+0000 7f1272cd1640 1 --2- 172.21.15.90:0/1651413031 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f126c1042c0 0x7f126c110690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.498 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.507+0000 7f1272cd1640 1 --2- 172.21.15.90:0/1651413031 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f126c10a010 0x7f126c103c50 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.498 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.507+0000 7f1272cd1640 1 --2- 172.21.15.90:0/1651413031 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f126c109b70 0x7f126c103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.498 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.507+0000 7f1272cd1640 1 -- 172.21.15.90:0/1651413031 >> 172.21.15.90:0/1651413031 conn(0x7f126c0ff210 msgr2=0x7f126c101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:57.498 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.507+0000 7f1272cd1640 1 -- 172.21.15.90:0/1651413031 shutdown_connections 2024-04-25T11:43:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.507+0000 7f1272cd1640 1 -- 172.21.15.90:0/1651413031 wait complete. 2024-04-25T11:43:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.508+0000 7f1272cd1640 1 Processor -- start 2024-04-25T11:43:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.508+0000 7f1272cd1640 1 -- start start 2024-04-25T11:43:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.508+0000 7f1272cd1640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f126c1042c0 0x7f126c19a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.508+0000 7f1272cd1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f126c109b70 0x7f126c19ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.508+0000 7f1272cd1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f126c10a010 0x7f126c19f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.508+0000 7f1272cd1640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f126c113d60 con 0x7f126c10a010 2024-04-25T11:43:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.508+0000 7f1272cd1640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f126c113be0 con 0x7f126c1042c0 2024-04-25T11:43:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.508+0000 7f1272cd1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f126c113ee0 con 0x7f126c109b70 2024-04-25T11:43:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.508+0000 7f127126e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f126c10a010 0x7f126c19f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:57.500 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.508+0000 7f127126e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f126c10a010 0x7f126c19f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35436/0 (socket says 172.21.15.90:35436) 2024-04-25T11:43:57.500 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.508+0000 7f127126e640 1 -- 172.21.15.90:0/2360117674 learned_addr learned my addr 172.21.15.90:0/2360117674 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:57.500 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.509+0000 7f1270a6d640 1 --2- 172.21.15.90:0/2360117674 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f126c1042c0 0x7f126c19a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:57.500 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.509+0000 7f126bfff640 1 --2- 172.21.15.90:0/2360117674 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f126c109b70 0x7f126c19ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:57.500 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.509+0000 7f127126e640 1 -- 172.21.15.90:0/2360117674 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f126c109b70 msgr2=0x7f126c19ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:57.500 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.509+0000 7f127126e640 1 --2- 172.21.15.90:0/2360117674 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f126c109b70 0x7f126c19ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.500 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.509+0000 7f127126e640 1 -- 172.21.15.90:0/2360117674 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f126c1042c0 msgr2=0x7f126c19a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:57.500 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.509+0000 7f127126e640 1 --2- 172.21.15.90:0/2360117674 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f126c1042c0 0x7f126c19a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.500 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.509+0000 7f127126e640 1 -- 172.21.15.90:0/2360117674 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f126c19f5e0 con 0x7f126c10a010 2024-04-25T11:43:57.500 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.509+0000 7f1270a6d640 1 --2- 172.21.15.90:0/2360117674 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f126c1042c0 0x7f126c19a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:57.501 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.509+0000 7f126bfff640 1 --2- 172.21.15.90:0/2360117674 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f126c109b70 0x7f126c19ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:57.501 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.509+0000 7f127126e640 1 --2- 172.21.15.90:0/2360117674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f126c10a010 0x7f126c19f020 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f126400cb00 tx=0x7f126400a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:57.501 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.509+0000 7f1269ffb640 1 -- 172.21.15.90:0/2360117674 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1264004b20 con 0x7f126c10a010 2024-04-25T11:43:57.501 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.509+0000 7f1272cd1640 1 -- 172.21.15.90:0/2360117674 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f126c19f8d0 con 0x7f126c10a010 2024-04-25T11:43:57.501 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.509+0000 7f1269ffb640 1 -- 172.21.15.90:0/2360117674 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1264004cc0 con 0x7f126c10a010 2024-04-25T11:43:57.501 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.509+0000 7f1269ffb640 1 -- 172.21.15.90:0/2360117674 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f126400f6b0 con 0x7f126c10a010 2024-04-25T11:43:57.501 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.509+0000 7f1272cd1640 1 -- 172.21.15.90:0/2360117674 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f126c19fda0 con 0x7f126c10a010 2024-04-25T11:43:57.509 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.512+0000 7f1279b7f740 1 Processor -- start 2024-04-25T11:43:57.509 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.513+0000 7f1269ffb640 1 -- 172.21.15.90:0/2360117674 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f1264005020 con 0x7f126c10a010 2024-04-25T11:43:57.509 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.513+0000 7f1279b7f740 1 -- start start 2024-04-25T11:43:57.509 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.513+0000 7f1279b7f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c68461b0 0x5612c6be5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:57.509 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.513+0000 7f1279b7f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612c681d2f0 0x5612c6be6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:57.509 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.513+0000 7f1279b7f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612c68254d0 0x5612c6be6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:57.509 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.513+0000 7f1279b7f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5612c6b4c460 con 0x5612c68461b0 2024-04-25T11:43:57.509 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.513+0000 7f1279b7f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5612c6b06590 con 0x5612c68254d0 2024-04-25T11:43:57.510 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.513+0000 7f1279b7f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5612c6b48420 con 0x5612c681d2f0 2024-04-25T11:43:57.510 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.513+0000 7f126bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c68461b0 0x5612c6be5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:57.510 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.513+0000 7f126bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c68461b0 0x5612c6be5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35442/0 (socket says 172.21.15.90:35442) 2024-04-25T11:43:57.510 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.513+0000 7f126bfff640 1 -- 172.21.15.90:0/3615555005 learned_addr learned my addr 172.21.15.90:0/3615555005 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:57.510 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.513+0000 7f1270a6d640 1 --2- 172.21.15.90:0/3615555005 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612c681d2f0 0x5612c6be6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:57.510 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.513+0000 7f1269ffb640 1 --2- 172.21.15.90:0/2360117674 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f125c0afc50 0x7f125c0b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:57.510 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.513+0000 7f126bfff640 1 -- 172.21.15.90:0/3615555005 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612c681d2f0 msgr2=0x5612c6be6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:57.510 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.513+0000 7f126bfff640 1 --2- 172.21.15.90:0/3615555005 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612c681d2f0 0x5612c6be6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.510 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.513+0000 7f126bfff640 1 -- 172.21.15.90:0/3615555005 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612c68254d0 msgr2=0x5612c6be6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:57.510 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.513+0000 7f126bfff640 1 --2- 172.21.15.90:0/3615555005 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612c68254d0 0x5612c6be6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.511 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.513+0000 7f126bfff640 1 -- 172.21.15.90:0/3615555005 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612c6b130d0 con 0x5612c68461b0 2024-04-25T11:43:57.511 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.514+0000 7f127126e640 1 --2- 172.21.15.90:0/2360117674 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f125c0afc50 0x7f125c0b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:57.511 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.514+0000 7f126bfff640 1 --2- 172.21.15.90:0/3615555005 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c68461b0 0x5612c6be5af0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f1258031440 tx=0x7f1258031de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1a86d316a92562c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:57.511 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.514+0000 7f1269ffb640 1 -- 172.21.15.90:0/2360117674 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f12640df1a0 con 0x7f126c10a010 2024-04-25T11:43:57.511 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.514+0000 7f1256ffd640 1 -- 172.21.15.90:0/3615555005 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1258035050 con 0x5612c68461b0 2024-04-25T11:43:57.511 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.514+0000 7f1256ffd640 1 -- 172.21.15.90:0/3615555005 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12580316a0 con 0x5612c68461b0 2024-04-25T11:43:57.511 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.514+0000 7f1256ffd640 1 -- 172.21.15.90:0/3615555005 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1258003d10 con 0x5612c68461b0 2024-04-25T11:43:57.511 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.515+0000 7f1279b7f740 1 -- 172.21.15.90:0/3615555005 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c68461b0 msgr2=0x5612c6be5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:57.511 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.515+0000 7f127126e640 1 --2- 172.21.15.90:0/2360117674 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f125c0afc50 0x7f125c0b2160 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f12640dc910 tx=0x7f1264074b10 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:57.511 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.515+0000 7f1279b7f740 1 --2- 172.21.15.90:0/3615555005 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c68461b0 0x5612c6be5af0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f1258031440 tx=0x7f1258031de0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.512 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.515+0000 7f1279b7f740 1 -- 172.21.15.90:0/3615555005 shutdown_connections 2024-04-25T11:43:57.512 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.515+0000 7f1279b7f740 1 --2- 172.21.15.90:0/3615555005 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612c68254d0 0x5612c6be6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.512 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.515+0000 7f1279b7f740 1 --2- 172.21.15.90:0/3615555005 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612c681d2f0 0x5612c6be6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.512 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.515+0000 7f1279b7f740 1 --2- 172.21.15.90:0/3615555005 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c68461b0 0x5612c6be5af0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.512 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.515+0000 7f1279b7f740 1 -- 172.21.15.90:0/3615555005 >> 172.21.15.90:0/3615555005 conn(0x5612c683a300 msgr2=0x5612c6be22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:57.512 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.515+0000 7f1279b7f740 1 -- 172.21.15.90:0/3615555005 shutdown_connections 2024-04-25T11:43:57.512 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.515+0000 7f1279b7f740 1 -- 172.21.15.90:0/3615555005 wait complete. 2024-04-25T11:43:57.512 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.515+0000 7f1279b7f740 1 -- start start 2024-04-25T11:43:57.512 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.515+0000 7f1279b7f740 1 Processor -- start 2024-04-25T11:43:57.512 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.516+0000 7f1279b7f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c681d2f0 0x5612c6bf1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:57.512 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.516+0000 7f1279b7f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612c68254d0 0x5612c6bf2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.516+0000 7f1279b7f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612c68461b0 0x5612c6befff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.516+0000 7f1279b7f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5612c6bf2bc0 con 0x5612c681d2f0 2024-04-25T11:43:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.516+0000 7f1279b7f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5612c6ad3b60 con 0x5612c68461b0 2024-04-25T11:43:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.516+0000 7f1279b7f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5612c6bf2d40 con 0x5612c68254d0 2024-04-25T11:43:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.516+0000 7f1279b7f740 10 client.0 _ll_register_callbacks cb 0x5612c6af82b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.516+0000 7f126bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c681d2f0 0x5612c6bf1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.516+0000 7f126bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c681d2f0 0x5612c6bf1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35458/0 (socket says 172.21.15.90:35458) 2024-04-25T11:43:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.516+0000 7f126bfff640 1 -- 172.21.15.90:0/2004156389 learned_addr learned my addr 172.21.15.90:0/2004156389 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.516+0000 7f1270a6d640 1 --2- 172.21.15.90:0/2004156389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612c68254d0 0x5612c6bf2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.516+0000 7f126bfff640 1 --2- 172.21.15.90:0/2004156389 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612c68461b0 0x5612c6befff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.516+0000 7f126bfff640 1 -- 172.21.15.90:0/2004156389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612c68254d0 msgr2=0x5612c6bf2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.516+0000 7f126bfff640 1 --2- 172.21.15.90:0/2004156389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612c68254d0 0x5612c6bf2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.516+0000 7f126bfff640 1 -- 172.21.15.90:0/2004156389 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612c68461b0 msgr2=0x5612c6befff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.516+0000 7f126bfff640 1 --2- 172.21.15.90:0/2004156389 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612c68461b0 0x5612c6befff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.516+0000 7f126bfff640 1 -- 172.21.15.90:0/2004156389 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612c685e210 con 0x5612c681d2f0 2024-04-25T11:43:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.516+0000 7f1270a6d640 1 --2- 172.21.15.90:0/2004156389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612c68254d0 0x5612c6bf2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.516+0000 7f126bfff640 1 --2- 172.21.15.90:0/2004156389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c681d2f0 0x5612c6bf1b00 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f12580039b0 tx=0x7f1258003a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7737f37d17c71baa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.516+0000 7f124bfff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.517+0000 7f124bfff640 1 -- 172.21.15.90:0/2004156389 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1258049070 con 0x5612c681d2f0 2024-04-25T11:43:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.517+0000 7f124bfff640 1 -- 172.21.15.90:0/2004156389 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f12580042b0 con 0x5612c681d2f0 2024-04-25T11:43:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.517+0000 7f1279b7f740 1 -- 172.21.15.90:0/2004156389 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12580316a0 con 0x5612c681d2f0 2024-04-25T11:43:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.517+0000 7f124bfff640 1 -- 172.21.15.90:0/2004156389 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1258035040 con 0x5612c681d2f0 2024-04-25T11:43:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.517+0000 7f1279b7f740 10 client.18384 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.517+0000 7f1279b7f740 1 -- 172.21.15.90:0/2004156389 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5612c692a620 con 0x5612c681d2f0 2024-04-25T11:43:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.517+0000 7f124bfff640 1 -- 172.21.15.90:0/2004156389 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12580316a0 con 0x5612c681d2f0 2024-04-25T11:43:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.517+0000 7f124bfff640 1 -- 172.21.15.90:0/2004156389 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f1258004b70 con 0x5612c681d2f0 2024-04-25T11:43:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.517+0000 7f1279b7f740 10 client.18384 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.517+0000 7f1279b7f740 10 client.18384 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:43:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.517+0000 7f1279b7f740 20 client.18384 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.517+0000 7f1279b7f740 1 --2- 172.21.15.90:0/2004156389 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5612c6804d80 0x5612c6c02550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.517+0000 7f1279b7f740 4 client.18384 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:43:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.517+0000 7f1279b7f740 1 -- 172.21.15.90:0/2004156389 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612c6b5a070 con 0x5612c6804d80 2024-04-25T11:43:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.518+0000 7f1270a6d640 1 --2- 172.21.15.90:0/2004156389 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5612c6804d80 0x5612c6c02550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.518+0000 7f124bfff640 10 client.18384 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.518+0000 7f1270a6d640 1 --2- 172.21.15.90:0/2004156389 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5612c6804d80 0x5612c6c02550 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=490c9f8513487cf9 server_cookie=78972de8b833444b in_seq=0 out_seq=0 2024-04-25T11:43:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.519+0000 7f124bfff640 1 -- 172.21.15.90:0/2004156389 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5612c6b5a070 con 0x5612c6804d80 2024-04-25T11:43:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.519+0000 7f124bfff640 10 client.18384 handle_command_reply: tid=0 2024-04-25T11:43:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.519+0000 7f1279b7f740 1 client.18384 shutdown 2024-04-25T11:43:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.519+0000 7f1279b7f740 1 -- 172.21.15.90:0/2004156389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c681d2f0 msgr2=0x5612c6bf1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.519+0000 7f1279b7f740 1 --2- 172.21.15.90:0/2004156389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c681d2f0 0x5612c6bf1b00 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f12580039b0 tx=0x7f1258003a10 comp rx=0 tx=0).stop 2024-04-25T11:43:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.519+0000 7f1279b7f740 1 -- 172.21.15.90:0/2004156389 shutdown_connections 2024-04-25T11:43:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.519+0000 7f1279b7f740 1 --2- 172.21.15.90:0/2004156389 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5612c6804d80 0x5612c6c02550 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.519+0000 7f1279b7f740 1 --2- 172.21.15.90:0/2004156389 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612c68461b0 0x5612c6befff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.519+0000 7f1279b7f740 1 --2- 172.21.15.90:0/2004156389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612c68254d0 0x5612c6bf2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.519+0000 7f1279b7f740 1 --2- 172.21.15.90:0/2004156389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c681d2f0 0x5612c6bf1b00 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.519+0000 7f124bfff640 0 client.18384 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.519+0000 7f1279b7f740 1 -- 172.21.15.90:0/2004156389 >> 172.21.15.90:0/2004156389 conn(0x5612c683a300 msgr2=0x5612c6be58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.519+0000 7f1279b7f740 1 -- 172.21.15.90:0/2004156389 shutdown_connections 2024-04-25T11:43:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.519+0000 7f1279b7f740 1 -- 172.21.15.90:0/2004156389 wait complete. 2024-04-25T11:43:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.519+0000 7f1279b7f740 20 client.18384 trim_cache size 0 max 16384 2024-04-25T11:43:57.527 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.534+0000 7f1279b7f740 1 Processor -- start 2024-04-25T11:43:57.527 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.534+0000 7f1279b7f740 1 -- start start 2024-04-25T11:43:57.527 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.534+0000 7f1279b7f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c681d2f0 0x5612c6be5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:57.527 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.534+0000 7f1279b7f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612c68254d0 0x5612c6c05580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:57.527 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.534+0000 7f1279b7f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612c68461b0 0x5612c6c05ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:57.527 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.534+0000 7f1279b7f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f126c1afba0 con 0x5612c681d2f0 2024-04-25T11:43:57.527 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.534+0000 7f1279b7f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f126c1b16e0 con 0x5612c68461b0 2024-04-25T11:43:57.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.534+0000 7f1279b7f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f126c1b0270 con 0x5612c68254d0 2024-04-25T11:43:57.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.535+0000 7f126bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c681d2f0 0x5612c6be5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:57.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.535+0000 7f126bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c681d2f0 0x5612c6be5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35462/0 (socket says 172.21.15.90:35462) 2024-04-25T11:43:57.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.535+0000 7f126bfff640 1 -- 172.21.15.90:0/3662532322 learned_addr learned my addr 172.21.15.90:0/3662532322 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:57.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.535+0000 7f1270a6d640 1 --2- 172.21.15.90:0/3662532322 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612c68254d0 0x5612c6c05580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:57.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.535+0000 7f126bfff640 1 -- 172.21.15.90:0/3662532322 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612c68254d0 msgr2=0x5612c6c05580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:57.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.535+0000 7f126bfff640 1 --2- 172.21.15.90:0/3662532322 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612c68254d0 0x5612c6c05580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.535+0000 7f126bfff640 1 -- 172.21.15.90:0/3662532322 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612c68461b0 msgr2=0x5612c6c05ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:57.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.535+0000 7f126bfff640 1 --2- 172.21.15.90:0/3662532322 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612c68461b0 0x5612c6c05ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.535+0000 7f126bfff640 1 -- 172.21.15.90:0/3662532322 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612c6c06000 con 0x5612c681d2f0 2024-04-25T11:43:57.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.535+0000 7f1270a6d640 1 --2- 172.21.15.90:0/3662532322 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612c68254d0 0x5612c6c05580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:43:57.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.535+0000 7f126bfff640 1 --2- 172.21.15.90:0/3662532322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c681d2f0 0x5612c6be5f90 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f1258004f70 tx=0x7f1258003a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b80f7efda279796a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:57.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.535+0000 7f12567fc640 1 -- 172.21.15.90:0/3662532322 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1258049070 con 0x5612c681d2f0 2024-04-25T11:43:57.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.535+0000 7f12567fc640 1 -- 172.21.15.90:0/3662532322 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f125805a020 con 0x5612c681d2f0 2024-04-25T11:43:57.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.535+0000 7f12567fc640 1 -- 172.21.15.90:0/3662532322 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1258004310 con 0x5612c681d2f0 2024-04-25T11:43:57.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.536+0000 7f1279b7f740 1 -- 172.21.15.90:0/3662532322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c681d2f0 msgr2=0x5612c6be5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:57.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.536+0000 7f1279b7f740 1 --2- 172.21.15.90:0/3662532322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c681d2f0 0x5612c6be5f90 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f1258004f70 tx=0x7f1258003a80 comp rx=0 tx=0).stop 2024-04-25T11:43:57.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.536+0000 7f1279b7f740 1 -- 172.21.15.90:0/3662532322 shutdown_connections 2024-04-25T11:43:57.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.536+0000 7f1279b7f740 1 --2- 172.21.15.90:0/3662532322 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612c68461b0 0x5612c6c05ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.536+0000 7f1279b7f740 1 --2- 172.21.15.90:0/3662532322 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612c68254d0 0x5612c6c05580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.536+0000 7f1279b7f740 1 --2- 172.21.15.90:0/3662532322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c681d2f0 0x5612c6be5f90 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.536+0000 7f1279b7f740 1 -- 172.21.15.90:0/3662532322 >> 172.21.15.90:0/3662532322 conn(0x5612c683a300 msgr2=0x5612c6be5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.536+0000 7f1279b7f740 1 -- 172.21.15.90:0/3662532322 shutdown_connections 2024-04-25T11:43:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.536+0000 7f1279b7f740 1 -- 172.21.15.90:0/3662532322 wait complete. 2024-04-25T11:43:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.536+0000 7f1279b7f740 1 -- start start 2024-04-25T11:43:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.536+0000 7f1279b7f740 1 Processor -- start 2024-04-25T11:43:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.537+0000 7f1279b7f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c681d2f0 0x5612c6c62a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.537+0000 7f1279b7f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612c68254d0 0x5612c6c62f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.537+0000 7f1279b7f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612c68461b0 0x5612c6c5ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.537+0000 7f1279b7f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f126c1b1010 con 0x5612c681d2f0 2024-04-25T11:43:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.537+0000 7f1279b7f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f126c1b0940 con 0x5612c68461b0 2024-04-25T11:43:57.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.537+0000 7f1279b7f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f126c1adba0 con 0x5612c68254d0 2024-04-25T11:43:57.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.537+0000 7f1279b7f740 10 client.0 _ll_register_callbacks cb 0x7f1258031d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:43:57.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.537+0000 7f126bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c681d2f0 0x5612c6c62a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:57.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.537+0000 7f126bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c681d2f0 0x5612c6c62a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35472/0 (socket says 172.21.15.90:35472) 2024-04-25T11:43:57.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.537+0000 7f126bfff640 1 -- 172.21.15.90:0/1987057883 learned_addr learned my addr 172.21.15.90:0/1987057883 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:43:57.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.537+0000 7f1270a6d640 1 --2- 172.21.15.90:0/1987057883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612c68254d0 0x5612c6c62f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:57.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.537+0000 7f126bfff640 1 -- 172.21.15.90:0/1987057883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612c68254d0 msgr2=0x5612c6c62f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:57.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.537+0000 7f126bfff640 1 --2- 172.21.15.90:0/1987057883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612c68254d0 0x5612c6c62f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.537+0000 7f126bfff640 1 -- 172.21.15.90:0/1987057883 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612c68461b0 msgr2=0x5612c6c5ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:57.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.537+0000 7f126bfff640 1 --2- 172.21.15.90:0/1987057883 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612c68461b0 0x5612c6c5ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.537+0000 7f126bfff640 1 -- 172.21.15.90:0/1987057883 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612c6c5d450 con 0x5612c681d2f0 2024-04-25T11:43:57.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.537+0000 7f126bfff640 1 --2- 172.21.15.90:0/1987057883 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c681d2f0 0x5612c6c62a50 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f1258003a50 tx=0x7f125803c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=189c2cac0801489d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:57.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.537+0000 7f124bfff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:43:57.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.538+0000 7f124bfff640 1 -- 172.21.15.90:0/1987057883 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1258063020 con 0x5612c681d2f0 2024-04-25T11:43:57.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.538+0000 7f124bfff640 1 -- 172.21.15.90:0/1987057883 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8658+0+0 (secure 0 0 0) 0x7f12580042c0 con 0x5612c681d2f0 2024-04-25T11:43:57.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.538+0000 7f1279b7f740 1 -- 172.21.15.90:0/1987057883 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f125805a020 con 0x5612c681d2f0 2024-04-25T11:43:57.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.538+0000 7f124bfff640 1 -- 172.21.15.90:0/1987057883 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1258049070 con 0x5612c681d2f0 2024-04-25T11:43:57.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.538+0000 7f1279b7f740 10 client.18390 fetch_fsmap learned FSMap version 28 2024-04-25T11:43:57.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.538+0000 7f1279b7f740 1 -- 172.21.15.90:0/1987057883 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5612c6c6a070 con 0x5612c681d2f0 2024-04-25T11:43:57.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.538+0000 7f124bfff640 1 -- 172.21.15.90:0/1987057883 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f125805a020 con 0x5612c681d2f0 2024-04-25T11:43:57.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.538+0000 7f124bfff640 1 -- 172.21.15.90:0/1987057883 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f12580048c0 con 0x5612c681d2f0 2024-04-25T11:43:57.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.538+0000 7f1279b7f740 10 client.18390 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:43:57.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.538+0000 7f1279b7f740 10 client.18390 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:43:57.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.538+0000 7f1279b7f740 20 client.18390 populate_metadata read hostname 'smithi090' 2024-04-25T11:43:57.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.538+0000 7f1279b7f740 1 --2- 172.21.15.90:0/1987057883 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5612c67da1c0 0x5612c6c6c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:57.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.538+0000 7f1279b7f740 4 client.18390 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:43:57.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.538+0000 7f1279b7f740 1 -- 172.21.15.90:0/1987057883 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5612c6b53ee0 con 0x5612c67da1c0 2024-04-25T11:43:57.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.539+0000 7f1270a6d640 1 --2- 172.21.15.90:0/1987057883 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5612c67da1c0 0x5612c6c6c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:57.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.539+0000 7f124bfff640 10 client.18390 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:57.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.539+0000 7f1270a6d640 1 --2- 172.21.15.90:0/1987057883 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5612c67da1c0 0x5612c6c6c6e0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a3497da1cea2833 server_cookie=c9771405abadf93b in_seq=0 out_seq=0 2024-04-25T11:43:57.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.546+0000 7f124bfff640 1 -- 172.21.15.90:0/1987057883 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5612c6b53ee0 con 0x5612c67da1c0 2024-04-25T11:43:57.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.546+0000 7f124bfff640 10 client.18390 handle_command_reply: tid=0 2024-04-25T11:43:57.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.546+0000 7f1279b7f740 1 client.18390 shutdown 2024-04-25T11:43:57.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.546+0000 7f1279b7f740 1 -- 172.21.15.90:0/1987057883 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c681d2f0 msgr2=0x5612c6c62a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:57.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.546+0000 7f1279b7f740 1 --2- 172.21.15.90:0/1987057883 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c681d2f0 0x5612c6c62a50 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f1258003a50 tx=0x7f125803c560 comp rx=0 tx=0).stop 2024-04-25T11:43:57.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.546+0000 7f1279b7f740 1 -- 172.21.15.90:0/1987057883 shutdown_connections 2024-04-25T11:43:57.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.546+0000 7f1279b7f740 1 --2- 172.21.15.90:0/1987057883 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5612c67da1c0 0x5612c6c6c6e0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.546+0000 7f1279b7f740 1 --2- 172.21.15.90:0/1987057883 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5612c68461b0 0x5612c6c5ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.546+0000 7f1279b7f740 1 --2- 172.21.15.90:0/1987057883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5612c68254d0 0x5612c6c62f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.546+0000 7f124bfff640 0 client.18390 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:43:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.546+0000 7f1279b7f740 1 --2- 172.21.15.90:0/1987057883 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612c681d2f0 0x5612c6c62a50 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.546+0000 7f1279b7f740 1 -- 172.21.15.90:0/1987057883 >> 172.21.15.90:0/1987057883 conn(0x5612c683a300 msgr2=0x5612c6bef7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.546+0000 7f1279b7f740 1 -- 172.21.15.90:0/1987057883 shutdown_connections 2024-04-25T11:43:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.546+0000 7f1279b7f740 1 -- 172.21.15.90:0/1987057883 wait complete. 2024-04-25T11:43:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.546+0000 7f1279b7f740 20 client.18390 trim_cache size 0 max 16384 2024-04-25T11:43:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:43:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:43:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:43:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:43:57.538 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.547+0000 7f1272cd1640 1 -- 172.21.15.90:0/2360117674 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f125c0afc50 msgr2=0x7f125c0b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:57.538 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.547+0000 7f1272cd1640 1 --2- 172.21.15.90:0/2360117674 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f125c0afc50 0x7f125c0b2160 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f12640dc910 tx=0x7f1264074b10 comp rx=0 tx=0).stop 2024-04-25T11:43:57.538 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.547+0000 7f1272cd1640 1 -- 172.21.15.90:0/2360117674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f126c10a010 msgr2=0x7f126c19f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:57.538 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.547+0000 7f1272cd1640 1 --2- 172.21.15.90:0/2360117674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f126c10a010 0x7f126c19f020 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f126400cb00 tx=0x7f126400a250 comp rx=0 tx=0).stop 2024-04-25T11:43:57.538 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.547+0000 7f1272cd1640 1 -- 172.21.15.90:0/2360117674 shutdown_connections 2024-04-25T11:43:57.538 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.547+0000 7f1272cd1640 1 --2- 172.21.15.90:0/2360117674 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f125c0afc50 0x7f125c0b2160 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.538 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.547+0000 7f1272cd1640 1 --2- 172.21.15.90:0/2360117674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f126c10a010 0x7f126c19f020 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.538 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.547+0000 7f1272cd1640 1 --2- 172.21.15.90:0/2360117674 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f126c109b70 0x7f126c19ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.538 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.547+0000 7f1272cd1640 1 --2- 172.21.15.90:0/2360117674 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f126c1042c0 0x7f126c19a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:57.538 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.547+0000 7f1272cd1640 1 -- 172.21.15.90:0/2360117674 >> 172.21.15.90:0/2360117674 conn(0x7f126c0ff210 msgr2=0x7f126c10f740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:57.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.547+0000 7f1272cd1640 1 -- 172.21.15.90:0/2360117674 shutdown_connections 2024-04-25T11:43:57.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:43:57.547+0000 7f1272cd1640 1 -- 172.21.15.90:0/2360117674 wait complete. 2024-04-25T11:43:58.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:58 smithi090 ceph-mon[29575]: pgmap v273: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 12 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-04-25T11:43:58.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:58 smithi138 ceph-mon[32328]: pgmap v273: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 12 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-04-25T11:43:58.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:58 smithi156 ceph-mon[33045]: pgmap v273: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 12 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-04-25T11:43:59.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:43:59 smithi090 ceph-mon[29575]: pgmap v274: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 11 op/s 2024-04-25T11:43:59.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:43:59 smithi138 ceph-mon[32328]: pgmap v274: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 11 op/s 2024-04-25T11:43:59.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:43:59 smithi156 ceph-mon[33045]: pgmap v274: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 11 op/s 2024-04-25T11:44:01.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:01 smithi138 ceph-mon[32328]: pgmap v275: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 14 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-04-25T11:44:01.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:01 smithi156 ceph-mon[33045]: pgmap v275: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 14 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-04-25T11:44:01.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:01 smithi090 ceph-mon[29575]: pgmap v275: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 14 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-04-25T11:44:03.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:03 smithi138 ceph-mon[32328]: pgmap v276: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:44:03.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:03 smithi156 ceph-mon[33045]: pgmap v276: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:44:03.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:03 smithi090 ceph-mon[29575]: pgmap v276: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:44:05.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:05 smithi138 ceph-mon[32328]: pgmap v277: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-04-25T11:44:05.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:05 smithi156 ceph-mon[33045]: pgmap v277: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-04-25T11:44:05.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:05 smithi090 ceph-mon[29575]: pgmap v277: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-04-25T11:44:06.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:06 smithi138 ceph-mon[32328]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T11:44:06.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:06 smithi156 ceph-mon[33045]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T11:44:06.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:06 smithi090 ceph-mon[29575]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T11:44:07.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:07 smithi138 ceph-mon[32328]: pgmap v279: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 13 KiB/s rd, 57 KiB/s wr, 18 op/s 2024-04-25T11:44:07.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:07 smithi156 ceph-mon[33045]: pgmap v279: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 13 KiB/s rd, 57 KiB/s wr, 18 op/s 2024-04-25T11:44:07.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:07 smithi090 ceph-mon[29575]: pgmap v279: 97 pgs: 97 active+clean; 51 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 13 KiB/s rd, 57 KiB/s wr, 18 op/s 2024-04-25T11:44:09.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:09 smithi138 ceph-mon[32328]: pgmap v280: 97 pgs: 4 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 83 active+clean; 50 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 13 KiB/s rd, 57 KiB/s wr, 18 op/s 2024-04-25T11:44:09.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:09 smithi156 ceph-mon[33045]: pgmap v280: 97 pgs: 4 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 83 active+clean; 50 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 13 KiB/s rd, 57 KiB/s wr, 18 op/s 2024-04-25T11:44:09.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:09 smithi090 ceph-mon[29575]: pgmap v280: 97 pgs: 4 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 83 active+clean; 50 GiB data, 155 GiB used, 918 GiB / 1.0 TiB avail; 13 KiB/s rd, 57 KiB/s wr, 18 op/s 2024-04-25T11:44:10.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.651+0000 7fc11d651640 1 -- 172.21.15.90:0/2119294926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1181020a0 msgr2=0x7fc1181024b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:10.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.651+0000 7fc11d651640 1 --2- 172.21.15.90:0/2119294926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1181020a0 0x7fc1181024b0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fc10c009900 tx=0x7fc10c02a410 comp rx=0 tx=0).stop 2024-04-25T11:44:10.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.651+0000 7fc11d651640 1 -- 172.21.15.90:0/2119294926 shutdown_connections 2024-04-25T11:44:10.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.651+0000 7fc11d651640 1 --2- 172.21.15.90:0/2119294926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc11810e5a0 0x7fc1181109c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:10.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.651+0000 7fc11d651640 1 --2- 172.21.15.90:0/2119294926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1181020a0 0x7fc1181024b0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:10.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.651+0000 7fc11d651640 1 --2- 172.21.15.90:0/2119294926 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc118109bf0 0x7fc118101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:10.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.651+0000 7fc11d651640 1 -- 172.21.15.90:0/2119294926 >> 172.21.15.90:0/2119294926 conn(0x7fc1180778b0 msgr2=0x7fc118077cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:10.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.651+0000 7fc11d651640 1 -- 172.21.15.90:0/2119294926 shutdown_connections 2024-04-25T11:44:10.638 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.651+0000 7fc11d651640 1 -- 172.21.15.90:0/2119294926 wait complete. 2024-04-25T11:44:10.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.651+0000 7fc11d651640 1 Processor -- start 2024-04-25T11:44:10.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.651+0000 7fc11d651640 1 -- start start 2024-04-25T11:44:10.640 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.652+0000 7fc11d651640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc1181020a0 0x7fc118102d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:10.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.652+0000 7fc11d651640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc118109bf0 0x7fc118103280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:10.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.652+0000 7fc11d651640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc11810e5a0 0x7fc1181037c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:10.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.652+0000 7fc11d651640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc118113d60 con 0x7fc11810e5a0 2024-04-25T11:44:10.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.652+0000 7fc11d651640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc118113be0 con 0x7fc118109bf0 2024-04-25T11:44:10.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.652+0000 7fc11d651640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc118113ee0 con 0x7fc1181020a0 2024-04-25T11:44:10.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.652+0000 7fc1177fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc11810e5a0 0x7fc1181037c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:10.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.652+0000 7fc1177fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc11810e5a0 0x7fc1181037c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49064/0 (socket says 172.21.15.90:49064) 2024-04-25T11:44:10.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.652+0000 7fc116ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc1181020a0 0x7fc118102d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:10.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.652+0000 7fc1177fe640 1 -- 172.21.15.90:0/1735284025 learned_addr learned my addr 172.21.15.90:0/1735284025 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:10.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.652+0000 7fc1167fc640 1 --2- 172.21.15.90:0/1735284025 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc118109bf0 0x7fc118103280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:10.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.652+0000 7fc1177fe640 1 -- 172.21.15.90:0/1735284025 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc1181020a0 msgr2=0x7fc118102d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:10.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.652+0000 7fc1177fe640 1 --2- 172.21.15.90:0/1735284025 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc1181020a0 0x7fc118102d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:10.642 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.652+0000 7fc1177fe640 1 -- 172.21.15.90:0/1735284025 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc118109bf0 msgr2=0x7fc118103280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:10.642 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.652+0000 7fc1177fe640 1 --2- 172.21.15.90:0/1735284025 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc118109bf0 0x7fc118103280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:10.642 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.652+0000 7fc1177fe640 1 -- 172.21.15.90:0/1735284025 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1181a8aa0 con 0x7fc11810e5a0 2024-04-25T11:44:10.642 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.652+0000 7fc1177fe640 1 --2- 172.21.15.90:0/1735284025 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc11810e5a0 0x7fc1181037c0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fc11000eb40 tx=0x7fc11000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:10.642 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.653+0000 7fc0fffff640 1 -- 172.21.15.90:0/1735284025 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc110004b20 con 0x7fc11810e5a0 2024-04-25T11:44:10.642 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.653+0000 7fc0fffff640 1 -- 172.21.15.90:0/1735284025 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc110004cc0 con 0x7fc11810e5a0 2024-04-25T11:44:10.642 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.653+0000 7fc11d651640 1 -- 172.21.15.90:0/1735284025 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1181a8d90 con 0x7fc11810e5a0 2024-04-25T11:44:10.642 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.653+0000 7fc0fffff640 1 -- 172.21.15.90:0/1735284025 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc110010720 con 0x7fc11810e5a0 2024-04-25T11:44:10.642 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.653+0000 7fc11d651640 1 -- 172.21.15.90:0/1735284025 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1181a91f0 con 0x7fc11810e5a0 2024-04-25T11:44:10.649 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.655+0000 7fc0fffff640 1 -- 172.21.15.90:0/1735284025 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fc110005560 con 0x7fc11810e5a0 2024-04-25T11:44:10.649 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.655+0000 7fc1245e9740 1 Processor -- start 2024-04-25T11:44:10.649 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.656+0000 7fc1245e9740 1 -- start start 2024-04-25T11:44:10.649 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.656+0000 7fc0fffff640 1 --2- 172.21.15.90:0/1735284025 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc1000afc50 0x7fc1000b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:10.649 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.656+0000 7fc1245e9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56450a5a71b0 0x56450a946af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:10.649 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.656+0000 7fc1245e9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56450a57e2f0 0x56450a947030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:10.649 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.656+0000 7fc1245e9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5864d0 0x56450a947570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:10.649 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.656+0000 7fc1167fc640 1 --2- 172.21.15.90:0/1735284025 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc1000afc50 0x7fc1000b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.656+0000 7fc0fffff640 1 -- 172.21.15.90:0/1735284025 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7fc1100df7d0 con 0x7fc11810e5a0 2024-04-25T11:44:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.656+0000 7fc1245e9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56450a8ad460 con 0x56450a5864d0 2024-04-25T11:44:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.656+0000 7fc1245e9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56450a867590 con 0x56450a57e2f0 2024-04-25T11:44:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.656+0000 7fc1245e9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56450a8a9420 con 0x56450a5a71b0 2024-04-25T11:44:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.656+0000 7fc1177fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5864d0 0x56450a947570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.656+0000 7fc1177fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5864d0 0x56450a947570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49080/0 (socket says 172.21.15.90:49080) 2024-04-25T11:44:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.656+0000 7fc116ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56450a5a71b0 0x56450a946af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.656+0000 7fc1177fe640 1 -- 172.21.15.90:0/1652654791 learned_addr learned my addr 172.21.15.90:0/1652654791 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.656+0000 7fc1167fc640 1 --2- 172.21.15.90:0/1652654791 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56450a57e2f0 0x56450a947030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.656+0000 7fc1177fe640 1 -- 172.21.15.90:0/1652654791 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56450a5a71b0 msgr2=0x56450a946af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.656+0000 7fc1177fe640 1 --2- 172.21.15.90:0/1652654791 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56450a5a71b0 0x56450a946af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.656+0000 7fc1177fe640 1 -- 172.21.15.90:0/1652654791 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56450a57e2f0 msgr2=0x56450a947030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.656+0000 7fc1177fe640 1 --2- 172.21.15.90:0/1652654791 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56450a57e2f0 0x56450a947030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.656+0000 7fc1177fe640 1 -- 172.21.15.90:0/1652654791 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56450a8740d0 con 0x56450a5864d0 2024-04-25T11:44:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.656+0000 7fc116ffd640 1 --2- 172.21.15.90:0/1652654791 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56450a5a71b0 0x56450a946af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.656+0000 7fc1167fc640 1 --2- 172.21.15.90:0/1735284025 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc1000afc50 0x7fc1000b2160 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc10c000c00 tx=0x7fc10c032ee0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.657+0000 7fc1177fe640 1 --2- 172.21.15.90:0/1652654791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5864d0 0x56450a947570 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fc1100a7850 tx=0x7fc1100a7030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4828c595b8e5ee54 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.657+0000 7fc0fd7fa640 1 -- 172.21.15.90:0/1652654791 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1100a7fc0 con 0x56450a5864d0 2024-04-25T11:44:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.657+0000 7fc0fd7fa640 1 -- 172.21.15.90:0/1652654791 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1100a78c0 con 0x56450a5864d0 2024-04-25T11:44:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.657+0000 7fc0fd7fa640 1 -- 172.21.15.90:0/1652654791 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1100b0290 con 0x56450a5864d0 2024-04-25T11:44:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.658+0000 7fc1245e9740 1 -- 172.21.15.90:0/1652654791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5864d0 msgr2=0x56450a947570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.658+0000 7fc1245e9740 1 --2- 172.21.15.90:0/1652654791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5864d0 0x56450a947570 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fc1100a7850 tx=0x7fc1100a7030 comp rx=0 tx=0).stop 2024-04-25T11:44:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.658+0000 7fc1245e9740 1 -- 172.21.15.90:0/1652654791 shutdown_connections 2024-04-25T11:44:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.658+0000 7fc1245e9740 1 --2- 172.21.15.90:0/1652654791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5864d0 0x56450a947570 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.658+0000 7fc1245e9740 1 --2- 172.21.15.90:0/1652654791 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56450a57e2f0 0x56450a947030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.658+0000 7fc1245e9740 1 --2- 172.21.15.90:0/1652654791 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56450a5a71b0 0x56450a946af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.658+0000 7fc1245e9740 1 -- 172.21.15.90:0/1652654791 >> 172.21.15.90:0/1652654791 conn(0x56450a59b300 msgr2=0x56450a9432a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.658+0000 7fc1245e9740 1 -- 172.21.15.90:0/1652654791 shutdown_connections 2024-04-25T11:44:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.658+0000 7fc1245e9740 1 -- 172.21.15.90:0/1652654791 wait complete. 2024-04-25T11:44:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.658+0000 7fc1245e9740 1 -- start start 2024-04-25T11:44:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.659+0000 7fc1245e9740 1 Processor -- start 2024-04-25T11:44:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.659+0000 7fc1245e9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56450a57e2f0 0x56450a952b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.659+0000 7fc1245e9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56450a5864d0 0x56450a953040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.659+0000 7fc1245e9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5a71b0 0x56450a950ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.659+0000 7fc1245e9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56450a953c20 con 0x56450a5a71b0 2024-04-25T11:44:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.659+0000 7fc1245e9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56450a834b60 con 0x56450a57e2f0 2024-04-25T11:44:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.659+0000 7fc1245e9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56450a953da0 con 0x56450a5864d0 2024-04-25T11:44:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.659+0000 7fc1245e9740 10 client.0 _ll_register_callbacks cb 0x56450a8592b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.659+0000 7fc1177fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5a71b0 0x56450a950ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.659+0000 7fc1167fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56450a5864d0 0x56450a953040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.659+0000 7fc1177fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5a71b0 0x56450a950ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49092/0 (socket says 172.21.15.90:49092) 2024-04-25T11:44:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.659+0000 7fc1167fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56450a5864d0 0x56450a953040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:42550/0 (socket says 172.21.15.90:42550) 2024-04-25T11:44:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.659+0000 7fc1177fe640 1 -- 172.21.15.90:0/1066527283 learned_addr learned my addr 172.21.15.90:0/1066527283 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:10.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.659+0000 7fc116ffd640 1 --2- 172.21.15.90:0/1066527283 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56450a57e2f0 0x56450a952b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:10.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.659+0000 7fc1167fc640 1 -- 172.21.15.90:0/1066527283 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56450a57e2f0 msgr2=0x56450a952b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:10.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.659+0000 7fc1167fc640 1 --2- 172.21.15.90:0/1066527283 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56450a57e2f0 0x56450a952b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:10.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.659+0000 7fc1167fc640 1 -- 172.21.15.90:0/1066527283 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5a71b0 msgr2=0x56450a950ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:10.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.659+0000 7fc1167fc640 1 --2- 172.21.15.90:0/1066527283 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5a71b0 0x56450a950ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:10.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.659+0000 7fc1167fc640 1 -- 172.21.15.90:0/1066527283 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56450a5bf210 con 0x56450a5864d0 2024-04-25T11:44:10.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.660+0000 7fc1167fc640 1 --2- 172.21.15.90:0/1066527283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56450a5864d0 0x56450a953040 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc10c006010 tx=0x7fc10c02a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f58a83995b6564df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:10.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.660+0000 7fc0faff5640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:44:10.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.660+0000 7fc0faff5640 1 -- 172.21.15.90:0/1066527283 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc10c04e070 con 0x56450a5864d0 2024-04-25T11:44:10.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.660+0000 7fc0faff5640 1 -- 172.21.15.90:0/1066527283 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7fc10c004450 con 0x56450a5864d0 2024-04-25T11:44:10.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.660+0000 7fc1245e9740 1 -- 172.21.15.90:0/1066527283 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1100a78c0 con 0x56450a5864d0 2024-04-25T11:44:10.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.660+0000 7fc0faff5640 1 -- 172.21.15.90:0/1066527283 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc10c05a050 con 0x56450a5864d0 2024-04-25T11:44:10.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.660+0000 7fc1245e9740 10 client.27043 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:10.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.660+0000 7fc1245e9740 1 -- 172.21.15.90:0/1066527283 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56450a68b620 con 0x56450a5864d0 2024-04-25T11:44:10.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.660+0000 7fc0faff5640 1 -- 172.21.15.90:0/1066527283 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1100a78c0 con 0x56450a5864d0 2024-04-25T11:44:10.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.661+0000 7fc0faff5640 1 -- 172.21.15.90:0/1066527283 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fc10c004a80 con 0x56450a5864d0 2024-04-25T11:44:10.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.661+0000 7fc1245e9740 10 client.27043 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:10.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.661+0000 7fc1245e9740 10 client.27043 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:44:10.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.661+0000 7fc1245e9740 20 client.27043 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:10.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.661+0000 7fc1245e9740 1 --2- 172.21.15.90:0/1066527283 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56450a565d80 0x56450a963770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:10.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.661+0000 7fc1245e9740 4 client.27043 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:44:10.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.661+0000 7fc1245e9740 1 -- 172.21.15.90:0/1066527283 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56450a8bb070 con 0x56450a565d80 2024-04-25T11:44:10.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.661+0000 7fc116ffd640 1 --2- 172.21.15.90:0/1066527283 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56450a565d80 0x56450a963770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:10.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.661+0000 7fc0faff5640 10 client.27043 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:10.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.661+0000 7fc116ffd640 1 --2- 172.21.15.90:0/1066527283 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56450a565d80 0x56450a963770 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9b8b4fad403df50 server_cookie=e322f2348a0e9bcc in_seq=0 out_seq=0 2024-04-25T11:44:10.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.662+0000 7fc0faff5640 1 -- 172.21.15.90:0/1066527283 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56450a8bb070 con 0x56450a565d80 2024-04-25T11:44:10.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.662+0000 7fc0faff5640 10 client.27043 handle_command_reply: tid=0 2024-04-25T11:44:10.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.662+0000 7fc1245e9740 1 client.27043 shutdown 2024-04-25T11:44:10.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.662+0000 7fc1245e9740 1 -- 172.21.15.90:0/1066527283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56450a5864d0 msgr2=0x56450a953040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.662+0000 7fc1245e9740 1 --2- 172.21.15.90:0/1066527283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56450a5864d0 0x56450a953040 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc10c006010 tx=0x7fc10c02a410 comp rx=0 tx=0).stop 2024-04-25T11:44:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.662+0000 7fc1245e9740 1 -- 172.21.15.90:0/1066527283 shutdown_connections 2024-04-25T11:44:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.662+0000 7fc1245e9740 1 --2- 172.21.15.90:0/1066527283 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56450a565d80 0x56450a963770 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.662+0000 7fc1245e9740 1 --2- 172.21.15.90:0/1066527283 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5a71b0 0x56450a950ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.662+0000 7fc1245e9740 1 --2- 172.21.15.90:0/1066527283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56450a5864d0 0x56450a953040 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.662+0000 7fc1245e9740 1 --2- 172.21.15.90:0/1066527283 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56450a57e2f0 0x56450a952b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.662+0000 7fc0faff5640 0 client.27043 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.662+0000 7fc1245e9740 1 -- 172.21.15.90:0/1066527283 >> 172.21.15.90:0/1066527283 conn(0x56450a59b300 msgr2=0x56450a948810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.662+0000 7fc1245e9740 1 -- 172.21.15.90:0/1066527283 shutdown_connections 2024-04-25T11:44:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.662+0000 7fc1245e9740 1 -- 172.21.15.90:0/1066527283 wait complete. 2024-04-25T11:44:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.662+0000 7fc1245e9740 20 client.27043 trim_cache size 0 max 16384 2024-04-25T11:44:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.677+0000 7fc1245e9740 1 Processor -- start 2024-04-25T11:44:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.677+0000 7fc1245e9740 1 -- start start 2024-04-25T11:44:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.677+0000 7fc1245e9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56450a57e2f0 0x56450a943370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.678+0000 7fc1245e9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5864d0 0x56450a968f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.678+0000 7fc1245e9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56450a5a71b0 0x56450a9694b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.678+0000 7fc1245e9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc1181b1f20 con 0x56450a5864d0 2024-04-25T11:44:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.678+0000 7fc1245e9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc1181b3a60 con 0x56450a5a71b0 2024-04-25T11:44:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.678+0000 7fc1245e9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc1181b25f0 con 0x56450a57e2f0 2024-04-25T11:44:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.678+0000 7fc1167fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5864d0 0x56450a968f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.678+0000 7fc1167fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5864d0 0x56450a968f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49094/0 (socket says 172.21.15.90:49094) 2024-04-25T11:44:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.678+0000 7fc1167fc640 1 -- 172.21.15.90:0/2220911310 learned_addr learned my addr 172.21.15.90:0/2220911310 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.678+0000 7fc116ffd640 1 --2- 172.21.15.90:0/2220911310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56450a57e2f0 0x56450a943370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.678+0000 7fc1177fe640 1 --2- 172.21.15.90:0/2220911310 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56450a5a71b0 0x56450a9694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.678+0000 7fc1167fc640 1 -- 172.21.15.90:0/2220911310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56450a57e2f0 msgr2=0x56450a943370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.678+0000 7fc1167fc640 1 --2- 172.21.15.90:0/2220911310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56450a57e2f0 0x56450a943370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.678+0000 7fc1167fc640 1 -- 172.21.15.90:0/2220911310 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56450a5a71b0 msgr2=0x56450a9694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.678+0000 7fc1167fc640 1 --2- 172.21.15.90:0/2220911310 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56450a5a71b0 0x56450a9694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.678+0000 7fc1167fc640 1 -- 172.21.15.90:0/2220911310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56450a9502f0 con 0x56450a5864d0 2024-04-25T11:44:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.678+0000 7fc1177fe640 1 --2- 172.21.15.90:0/2220911310 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56450a5a71b0 0x56450a9694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.678+0000 7fc1167fc640 1 --2- 172.21.15.90:0/2220911310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5864d0 0x56450a968f70 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fc10c004ed0 tx=0x7fc10c003850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7c9ad2b53ec1ac9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.678+0000 7fc0fcff9640 1 -- 172.21.15.90:0/2220911310 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc10c04e070 con 0x56450a5864d0 2024-04-25T11:44:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.678+0000 7fc0fcff9640 1 -- 172.21.15.90:0/2220911310 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc10c004450 con 0x56450a5864d0 2024-04-25T11:44:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.678+0000 7fc0fcff9640 1 -- 172.21.15.90:0/2220911310 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc10c035040 con 0x56450a5864d0 2024-04-25T11:44:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.679+0000 7fc1245e9740 1 -- 172.21.15.90:0/2220911310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5864d0 msgr2=0x56450a968f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.679+0000 7fc1245e9740 1 --2- 172.21.15.90:0/2220911310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5864d0 0x56450a968f70 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fc10c004ed0 tx=0x7fc10c003850 comp rx=0 tx=0).stop 2024-04-25T11:44:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.679+0000 7fc1245e9740 1 -- 172.21.15.90:0/2220911310 shutdown_connections 2024-04-25T11:44:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.679+0000 7fc1245e9740 1 --2- 172.21.15.90:0/2220911310 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56450a5a71b0 0x56450a9694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.679+0000 7fc1245e9740 1 --2- 172.21.15.90:0/2220911310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5864d0 0x56450a968f70 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.679+0000 7fc1245e9740 1 --2- 172.21.15.90:0/2220911310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56450a57e2f0 0x56450a943370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.679+0000 7fc1245e9740 1 -- 172.21.15.90:0/2220911310 >> 172.21.15.90:0/2220911310 conn(0x56450a59b300 msgr2=0x56450a942c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.679+0000 7fc1245e9740 1 -- 172.21.15.90:0/2220911310 shutdown_connections 2024-04-25T11:44:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.679+0000 7fc1245e9740 1 -- 172.21.15.90:0/2220911310 wait complete. 2024-04-25T11:44:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.679+0000 7fc1245e9740 1 -- start start 2024-04-25T11:44:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.679+0000 7fc1245e9740 1 Processor -- start 2024-04-25T11:44:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.680+0000 7fc1245e9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56450a57e2f0 0x56450a9593c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.680+0000 7fc1245e9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56450a5864d0 0x56450a954710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.680+0000 7fc1245e9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5a71b0 0x56450a954c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.680+0000 7fc1245e9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc1181b3390 con 0x56450a5a71b0 2024-04-25T11:44:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.680+0000 7fc1245e9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc1181b2cc0 con 0x56450a5864d0 2024-04-25T11:44:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.680+0000 7fc1245e9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc1181aff20 con 0x56450a57e2f0 2024-04-25T11:44:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.680+0000 7fc1245e9740 10 client.0 _ll_register_callbacks cb 0x7fc10c02ab10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.680+0000 7fc1177fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5a71b0 0x56450a954c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.680+0000 7fc1167fc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56450a5864d0 0x56450a954710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.680+0000 7fc116ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56450a57e2f0 0x56450a9593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.680+0000 7fc1177fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5a71b0 0x56450a954c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49110/0 (socket says 172.21.15.90:49110) 2024-04-25T11:44:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.680+0000 7fc116ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56450a57e2f0 0x56450a9593c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:42570/0 (socket says 172.21.15.90:42570) 2024-04-25T11:44:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.680+0000 7fc1177fe640 1 -- 172.21.15.90:0/3061262396 learned_addr learned my addr 172.21.15.90:0/3061262396 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.680+0000 7fc1167fc640 1 -- 172.21.15.90:0/3061262396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56450a57e2f0 msgr2=0x56450a9593c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.680+0000 7fc1167fc640 1 --2- 172.21.15.90:0/3061262396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56450a57e2f0 0x56450a9593c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:10.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.680+0000 7fc1167fc640 1 -- 172.21.15.90:0/3061262396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5a71b0 msgr2=0x56450a954c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:10.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.680+0000 7fc1167fc640 1 --2- 172.21.15.90:0/3061262396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5a71b0 0x56450a954c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:10.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.680+0000 7fc1167fc640 1 -- 172.21.15.90:0/3061262396 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56450a955190 con 0x56450a5864d0 2024-04-25T11:44:10.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.680+0000 7fc116ffd640 1 --2- 172.21.15.90:0/3061262396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56450a57e2f0 0x56450a9593c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:10.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.680+0000 7fc1177fe640 1 --2- 172.21.15.90:0/3061262396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5a71b0 0x56450a954c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:10.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.680+0000 7fc1167fc640 1 --2- 172.21.15.90:0/3061262396 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56450a5864d0 0x56450a954710 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc10c003640 tx=0x7fc10c003c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc64b58ffe07fa0f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:10.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.680+0000 7fc0faff5640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:44:10.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.681+0000 7fc0faff5640 1 -- 172.21.15.90:0/3061262396 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc10c06b060 con 0x56450a5864d0 2024-04-25T11:44:10.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.681+0000 7fc1245e9740 1 -- 172.21.15.90:0/3061262396 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc10c004450 con 0x56450a5864d0 2024-04-25T11:44:10.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.681+0000 7fc0faff5640 1 -- 172.21.15.90:0/3061262396 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7fc10c004750 con 0x56450a5864d0 2024-04-25T11:44:10.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.681+0000 7fc0faff5640 1 -- 172.21.15.90:0/3061262396 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc10c072050 con 0x56450a5864d0 2024-04-25T11:44:10.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.681+0000 7fc1245e9740 10 client.27173 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:10.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.681+0000 7fc1245e9740 1 -- 172.21.15.90:0/3061262396 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56450a9cb790 con 0x56450a5864d0 2024-04-25T11:44:10.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.681+0000 7fc0faff5640 1 -- 172.21.15.90:0/3061262396 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc10c004450 con 0x56450a5864d0 2024-04-25T11:44:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.681+0000 7fc0faff5640 1 -- 172.21.15.90:0/3061262396 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fc10c005300 con 0x56450a5864d0 2024-04-25T11:44:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.681+0000 7fc1245e9740 10 client.27173 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.681+0000 7fc1245e9740 10 client.27173 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:44:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.681+0000 7fc1245e9740 20 client.27173 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.681+0000 7fc1245e9740 1 --2- 172.21.15.90:0/3061262396 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56450a53b1c0 0x56450a9cde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.681+0000 7fc1245e9740 4 client.27173 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:44:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.681+0000 7fc1245e9740 1 -- 172.21.15.90:0/3061262396 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56450a8b4ee0 con 0x56450a53b1c0 2024-04-25T11:44:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.681+0000 7fc116ffd640 1 --2- 172.21.15.90:0/3061262396 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56450a53b1c0 0x56450a9cde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.682+0000 7fc0faff5640 10 client.27173 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:10.682+0000 7fc116ffd640 1 --2- 172.21.15.90:0/3061262396 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56450a53b1c0 0x56450a9cde50 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e120201703b8c4b5 server_cookie=e9aa0d1b0fe05c9f in_seq=0 out_seq=0 2024-04-25T11:44:11.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.788+0000 7fc0faff5640 1 -- 172.21.15.90:0/3061262396 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56450a8b4ee0 con 0x56450a53b1c0 2024-04-25T11:44:11.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.788+0000 7fc0faff5640 10 client.27173 handle_command_reply: tid=0 2024-04-25T11:44:11.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.788+0000 7fc1245e9740 1 client.27173 shutdown 2024-04-25T11:44:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.788+0000 7fc1245e9740 1 -- 172.21.15.90:0/3061262396 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56450a5864d0 msgr2=0x56450a954710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.788+0000 7fc1245e9740 1 --2- 172.21.15.90:0/3061262396 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56450a5864d0 0x56450a954710 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc10c003640 tx=0x7fc10c003c50 comp rx=0 tx=0).stop 2024-04-25T11:44:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.788+0000 7fc1245e9740 1 -- 172.21.15.90:0/3061262396 shutdown_connections 2024-04-25T11:44:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.788+0000 7fc1245e9740 1 --2- 172.21.15.90:0/3061262396 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56450a53b1c0 0x56450a9cde50 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.788+0000 7fc1245e9740 1 --2- 172.21.15.90:0/3061262396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56450a5a71b0 0x56450a954c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.788+0000 7fc1245e9740 1 --2- 172.21.15.90:0/3061262396 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56450a5864d0 0x56450a954710 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.788+0000 7fc0faff5640 0 client.27173 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.788+0000 7fc1245e9740 1 --2- 172.21.15.90:0/3061262396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56450a57e2f0 0x56450a9593c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.788+0000 7fc1245e9740 1 -- 172.21.15.90:0/3061262396 >> 172.21.15.90:0/3061262396 conn(0x56450a59b300 msgr2=0x56450a946a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.788+0000 7fc1245e9740 1 -- 172.21.15.90:0/3061262396 shutdown_connections 2024-04-25T11:44:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.788+0000 7fc1245e9740 1 -- 172.21.15.90:0/3061262396 wait complete. 2024-04-25T11:44:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.788+0000 7fc1245e9740 20 client.27173 trim_cache size 0 max 16384 2024-04-25T11:44:11.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:44:11.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:44:11.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:44:11.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:44:11.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.789+0000 7fc11d651640 1 -- 172.21.15.90:0/1735284025 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc1000afc50 msgr2=0x7fc1000b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:11.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.789+0000 7fc11d651640 1 --2- 172.21.15.90:0/1735284025 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc1000afc50 0x7fc1000b2160 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc10c000c00 tx=0x7fc10c032ee0 comp rx=0 tx=0).stop 2024-04-25T11:44:11.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.789+0000 7fc11d651640 1 -- 172.21.15.90:0/1735284025 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc11810e5a0 msgr2=0x7fc1181037c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:11.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.789+0000 7fc11d651640 1 --2- 172.21.15.90:0/1735284025 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc11810e5a0 0x7fc1181037c0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fc11000eb40 tx=0x7fc11000c2d0 comp rx=0 tx=0).stop 2024-04-25T11:44:11.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.789+0000 7fc11d651640 1 -- 172.21.15.90:0/1735284025 shutdown_connections 2024-04-25T11:44:11.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.789+0000 7fc11d651640 1 --2- 172.21.15.90:0/1735284025 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc1000afc50 0x7fc1000b2160 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:11.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.789+0000 7fc11d651640 1 --2- 172.21.15.90:0/1735284025 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc11810e5a0 0x7fc1181037c0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:11.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.789+0000 7fc11d651640 1 --2- 172.21.15.90:0/1735284025 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc118109bf0 0x7fc118103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:11.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.789+0000 7fc11d651640 1 --2- 172.21.15.90:0/1735284025 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc1181020a0 0x7fc118102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:11.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.789+0000 7fc11d651640 1 -- 172.21.15.90:0/1735284025 >> 172.21.15.90:0/1735284025 conn(0x7fc1180778b0 msgr2=0x7fc118105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:11.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.789+0000 7fc11d651640 1 -- 172.21.15.90:0/1735284025 shutdown_connections 2024-04-25T11:44:11.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:11.789+0000 7fc11d651640 1 -- 172.21.15.90:0/1735284025 wait complete. 2024-04-25T11:44:11.799 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:11 smithi138 ceph-mon[32328]: pgmap v281: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 48 GiB data, 148 GiB used, 925 GiB / 1.0 TiB avail; 9.9 KiB/s rd, 54 KiB/s wr, 18 op/s 2024-04-25T11:44:11.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:11 smithi156 ceph-mon[33045]: pgmap v281: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 48 GiB data, 148 GiB used, 925 GiB / 1.0 TiB avail; 9.9 KiB/s rd, 54 KiB/s wr, 18 op/s 2024-04-25T11:44:11.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:11 smithi090 ceph-mon[29575]: pgmap v281: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 48 GiB data, 148 GiB used, 925 GiB / 1.0 TiB avail; 9.9 KiB/s rd, 54 KiB/s wr, 18 op/s 2024-04-25T11:44:13.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:13 smithi138 ceph-mon[32328]: pgmap v282: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 47 GiB data, 146 GiB used, 927 GiB / 1.0 TiB avail; 9.9 KiB/s rd, 54 KiB/s wr, 18 op/s 2024-04-25T11:44:13.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:13 smithi156 ceph-mon[33045]: pgmap v282: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 47 GiB data, 146 GiB used, 927 GiB / 1.0 TiB avail; 9.9 KiB/s rd, 54 KiB/s wr, 18 op/s 2024-04-25T11:44:13.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:13 smithi090 ceph-mon[29575]: pgmap v282: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 47 GiB data, 146 GiB used, 927 GiB / 1.0 TiB avail; 9.9 KiB/s rd, 54 KiB/s wr, 18 op/s 2024-04-25T11:44:15.431 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:44:15.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:44:15.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.563+0000 7f8e99ce7640 1 -- 172.21.15.90:0/2559519226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e94109b10 msgr2=0x7f8e941036b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:15.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.563+0000 7f8e99ce7640 1 --2- 172.21.15.90:0/2559519226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e94109b10 0x7f8e941036b0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f8e84009900 tx=0x7f8e8402a410 comp rx=0 tx=0).stop 2024-04-25T11:44:15.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.563+0000 7f8e99ce7640 1 -- 172.21.15.90:0/2559519226 shutdown_connections 2024-04-25T11:44:15.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.563+0000 7f8e99ce7640 1 --2- 172.21.15.90:0/2559519226 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8e94104260 0x7f8e94110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.563+0000 7f8e99ce7640 1 --2- 172.21.15.90:0/2559519226 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e94109fb0 0x7f8e94103bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.563+0000 7f8e99ce7640 1 --2- 172.21.15.90:0/2559519226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e94109b10 0x7f8e941036b0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.563+0000 7f8e99ce7640 1 -- 172.21.15.90:0/2559519226 >> 172.21.15.90:0/2559519226 conn(0x7f8e940ff1d0 msgr2=0x7f8e94101620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:15.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.563+0000 7f8e99ce7640 1 -- 172.21.15.90:0/2559519226 shutdown_connections 2024-04-25T11:44:15.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.563+0000 7f8e99ce7640 1 -- 172.21.15.90:0/2559519226 wait complete. 2024-04-25T11:44:15.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.564+0000 7f8e99ce7640 1 Processor -- start 2024-04-25T11:44:15.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.564+0000 7f8e99ce7640 1 -- start start 2024-04-25T11:44:15.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.564+0000 7f8e99ce7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e94104260 0x7f8e94075fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:15.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.564+0000 7f8e99ce7640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e94109b10 0x7f8e940764f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:15.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.564+0000 7f8e99ce7640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8e94109fb0 0x7f8e94076a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:15.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.564+0000 7f8e99ce7640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8e94113d00 con 0x7f8e94109b10 2024-04-25T11:44:15.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.564+0000 7f8e99ce7640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8e94113b80 con 0x7f8e94109fb0 2024-04-25T11:44:15.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.564+0000 7f8e99ce7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8e94113e80 con 0x7f8e94104260 2024-04-25T11:44:15.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.564+0000 7f8e92ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e94109b10 0x7f8e940764f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:15.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.564+0000 7f8e92ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e94109b10 0x7f8e940764f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47268/0 (socket says 172.21.15.90:47268) 2024-04-25T11:44:15.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.564+0000 7f8e92ffd640 1 -- 172.21.15.90:0/2345121378 learned_addr learned my addr 172.21.15.90:0/2345121378 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:15.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.564+0000 7f8e93fff640 1 --2- 172.21.15.90:0/2345121378 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8e94109fb0 0x7f8e94076a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:15.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.564+0000 7f8e937fe640 1 --2- 172.21.15.90:0/2345121378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e94104260 0x7f8e94075fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:15.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.565+0000 7f8e92ffd640 1 -- 172.21.15.90:0/2345121378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e94104260 msgr2=0x7f8e94075fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:15.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.565+0000 7f8e92ffd640 1 --2- 172.21.15.90:0/2345121378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e94104260 0x7f8e94075fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.565+0000 7f8e92ffd640 1 -- 172.21.15.90:0/2345121378 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8e94109fb0 msgr2=0x7f8e94076a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:15.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.565+0000 7f8e92ffd640 1 --2- 172.21.15.90:0/2345121378 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8e94109fb0 0x7f8e94076a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.565+0000 7f8e92ffd640 1 -- 172.21.15.90:0/2345121378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e941a8a90 con 0x7f8e94109b10 2024-04-25T11:44:15.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.565+0000 7f8e92ffd640 1 --2- 172.21.15.90:0/2345121378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e94109b10 0x7f8e940764f0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f8e88005d00 tx=0x7f8e8800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:15.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.565+0000 7f8e90ff9640 1 -- 172.21.15.90:0/2345121378 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e88017070 con 0x7f8e94109b10 2024-04-25T11:44:15.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.565+0000 7f8e99ce7640 1 -- 172.21.15.90:0/2345121378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e941a8d80 con 0x7f8e94109b10 2024-04-25T11:44:15.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.565+0000 7f8e90ff9640 1 -- 172.21.15.90:0/2345121378 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e88009c30 con 0x7f8e94109b10 2024-04-25T11:44:15.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.565+0000 7f8e99ce7640 1 -- 172.21.15.90:0/2345121378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e941a9210 con 0x7f8e94109b10 2024-04-25T11:44:15.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.565+0000 7f8e90ff9640 1 -- 172.21.15.90:0/2345121378 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8800f620 con 0x7f8e94109b10 2024-04-25T11:44:15.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.568+0000 7f8e90ff9640 1 -- 172.21.15.90:0/2345121378 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f8e8801a070 con 0x7f8e94109b10 2024-04-25T11:44:15.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.568+0000 7f8ea0b97740 1 Processor -- start 2024-04-25T11:44:15.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.568+0000 7f8ea0b97740 1 -- start start 2024-04-25T11:44:15.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.568+0000 7f8ea0b97740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e795054390 0x55e7953f4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:15.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.568+0000 7f8ea0b97740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e79502b4d0 0x55e7953f4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:15.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.568+0000 7f8ea0b97740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e7950336b0 0x55e7954018f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:15.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.568+0000 7f8ea0b97740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e79535ad80 con 0x55e79502b4d0 2024-04-25T11:44:15.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.568+0000 7f8ea0b97740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55e7953148b0 con 0x55e795054390 2024-04-25T11:44:15.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.568+0000 7f8ea0b97740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e795356d40 con 0x55e7950336b0 2024-04-25T11:44:15.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.568+0000 7f8e937fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e79502b4d0 0x55e7953f4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:15.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.569+0000 7f8e937fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e79502b4d0 0x55e7953f4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47282/0 (socket says 172.21.15.90:47282) 2024-04-25T11:44:15.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.569+0000 7f8e937fe640 1 -- 172.21.15.90:0/4268606314 learned_addr learned my addr 172.21.15.90:0/4268606314 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:15.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.569+0000 7f8e90ff9640 1 --2- 172.21.15.90:0/2345121378 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8e7c0afc60 0x7f8e7c0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:15.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.569+0000 7f8e93fff640 1 --2- 172.21.15.90:0/4268606314 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e795054390 0x55e7953f4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:15.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.569+0000 7f8e93fff640 1 --2- 172.21.15.90:0/2345121378 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8e7c0afc60 0x7f8e7c0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:15.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.569+0000 7f8e92ffd640 1 --2- 172.21.15.90:0/4268606314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e7950336b0 0x55e7954018f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:15.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.569+0000 7f8e937fe640 1 -- 172.21.15.90:0/4268606314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e7950336b0 msgr2=0x55e7954018f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:15.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.569+0000 7f8e937fe640 1 --2- 172.21.15.90:0/4268606314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e7950336b0 0x55e7954018f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.569+0000 7f8e937fe640 1 -- 172.21.15.90:0/4268606314 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e795054390 msgr2=0x55e7953f4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:15.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.569+0000 7f8e937fe640 1 --2- 172.21.15.90:0/4268606314 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e795054390 0x55e7953f4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.569+0000 7f8e937fe640 1 -- 172.21.15.90:0/4268606314 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7952e5a40 con 0x55e79502b4d0 2024-04-25T11:44:15.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.569+0000 7f8e90ff9640 1 -- 172.21.15.90:0/2345121378 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7f8e880dea80 con 0x7f8e94109b10 2024-04-25T11:44:15.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.569+0000 7f8e93fff640 1 --2- 172.21.15.90:0/4268606314 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e795054390 0x55e7953f4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:44:15.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.569+0000 7f8e937fe640 1 --2- 172.21.15.90:0/4268606314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e79502b4d0 0x55e7953f4fa0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f8e84009900 tx=0x7f8e84032df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ae53d8454c09dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:15.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.569+0000 7f8e93fff640 1 --2- 172.21.15.90:0/2345121378 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8e7c0afc60 0x7f8e7c0b2170 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x55e7953f1bd0 tx=0x7f8e8c007520 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:15.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.569+0000 7f8e79ffb640 1 -- 172.21.15.90:0/4268606314 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e84040070 con 0x55e79502b4d0 2024-04-25T11:44:15.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.569+0000 7f8e79ffb640 1 -- 172.21.15.90:0/4268606314 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e84005db0 con 0x55e79502b4d0 2024-04-25T11:44:15.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.569+0000 7f8e79ffb640 1 -- 172.21.15.90:0/4268606314 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e84035040 con 0x55e79502b4d0 2024-04-25T11:44:15.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.570+0000 7f8ea0b97740 1 -- 172.21.15.90:0/4268606314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e79502b4d0 msgr2=0x55e7953f4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:15.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.570+0000 7f8ea0b97740 1 --2- 172.21.15.90:0/4268606314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e79502b4d0 0x55e7953f4fa0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f8e84009900 tx=0x7f8e84032df0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.570+0000 7f8ea0b97740 1 -- 172.21.15.90:0/4268606314 shutdown_connections 2024-04-25T11:44:15.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.570+0000 7f8ea0b97740 1 --2- 172.21.15.90:0/4268606314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e7950336b0 0x55e7954018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.570+0000 7f8ea0b97740 1 --2- 172.21.15.90:0/4268606314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e79502b4d0 0x55e7953f4fa0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.570+0000 7f8ea0b97740 1 --2- 172.21.15.90:0/4268606314 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e795054390 0x55e7953f4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.570+0000 7f8ea0b97740 1 -- 172.21.15.90:0/4268606314 >> 172.21.15.90:0/4268606314 conn(0x55e7950484e0 msgr2=0x55e7953f1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:15.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.570+0000 7f8ea0b97740 1 -- 172.21.15.90:0/4268606314 shutdown_connections 2024-04-25T11:44:15.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.570+0000 7f8ea0b97740 1 -- 172.21.15.90:0/4268606314 wait complete. 2024-04-25T11:44:15.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.571+0000 7f8ea0b97740 1 -- start start 2024-04-25T11:44:15.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.571+0000 7f8ea0b97740 1 Processor -- start 2024-04-25T11:44:15.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.571+0000 7f8ea0b97740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e79502b4d0 0x55e7953ff660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:15.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.571+0000 7f8ea0b97740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7950336b0 0x55e7953ffba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:15.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.571+0000 7f8ea0b97740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e795054390 0x55e795405440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:15.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.571+0000 7f8ea0b97740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e795401f80 con 0x55e7950336b0 2024-04-25T11:44:15.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.571+0000 7f8ea0b97740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55e7952e1d10 con 0x55e795054390 2024-04-25T11:44:15.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.571+0000 7f8ea0b97740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e795402100 con 0x55e79502b4d0 2024-04-25T11:44:15.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.571+0000 7f8ea0b97740 10 client.0 _ll_register_callbacks cb 0x55e79530e210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:15.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.572+0000 7f8e92ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7950336b0 0x55e7953ffba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:15.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.572+0000 7f8e92ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7950336b0 0x55e7953ffba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47284/0 (socket says 172.21.15.90:47284) 2024-04-25T11:44:15.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.572+0000 7f8e92ffd640 1 -- 172.21.15.90:0/314518980 learned_addr learned my addr 172.21.15.90:0/314518980 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:15.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.572+0000 7f8e937fe640 1 --2- 172.21.15.90:0/314518980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e79502b4d0 0x55e7953ff660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:15.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.572+0000 7f8e93fff640 1 --2- 172.21.15.90:0/314518980 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e795054390 0x55e795405440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:15.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.572+0000 7f8e92ffd640 1 -- 172.21.15.90:0/314518980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e79502b4d0 msgr2=0x55e7953ff660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:15.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.572+0000 7f8e92ffd640 1 --2- 172.21.15.90:0/314518980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e79502b4d0 0x55e7953ff660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.572+0000 7f8e92ffd640 1 -- 172.21.15.90:0/314518980 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e795054390 msgr2=0x55e795405440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:15.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.572+0000 7f8e92ffd640 1 --2- 172.21.15.90:0/314518980 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e795054390 0x55e795405440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.572+0000 7f8e92ffd640 1 -- 172.21.15.90:0/314518980 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7951367d0 con 0x55e7950336b0 2024-04-25T11:44:15.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.572+0000 7f8e93fff640 1 --2- 172.21.15.90:0/314518980 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e795054390 0x55e795405440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:44:15.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.572+0000 7f8e92ffd640 1 --2- 172.21.15.90:0/314518980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7950336b0 0x55e7953ffba0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f8e880a6920 tx=0x7f8e88017600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b107766f7dc60d82 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:15.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.572+0000 7f8e72ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:15.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.572+0000 7f8e72ffd640 1 -- 172.21.15.90:0/314518980 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e880a6570 con 0x55e7950336b0 2024-04-25T11:44:15.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.572+0000 7f8e72ffd640 1 -- 172.21.15.90:0/314518980 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7f8e880af750 con 0x55e7950336b0 2024-04-25T11:44:15.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.572+0000 7f8ea0b97740 1 -- 172.21.15.90:0/314518980 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e84005db0 con 0x55e7950336b0 2024-04-25T11:44:15.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.572+0000 7f8e72ffd640 1 -- 172.21.15.90:0/314518980 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8e88015110 con 0x55e7950336b0 2024-04-25T11:44:15.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.572+0000 7f8ea0b97740 10 client.18420 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:15.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.572+0000 7f8ea0b97740 1 -- 172.21.15.90:0/314518980 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e79506c3f0 con 0x55e7950336b0 2024-04-25T11:44:15.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.572+0000 7f8e72ffd640 1 -- 172.21.15.90:0/314518980 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e84005db0 con 0x55e7950336b0 2024-04-25T11:44:15.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.573+0000 7f8e72ffd640 1 -- 172.21.15.90:0/314518980 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f8e880a2590 con 0x55e7950336b0 2024-04-25T11:44:15.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.573+0000 7f8ea0b97740 10 client.18420 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:15.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.573+0000 7f8ea0b97740 10 client.18420 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:44:15.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.573+0000 7f8ea0b97740 20 client.18420 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:15.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.573+0000 7f8ea0b97740 1 --2- 172.21.15.90:0/314518980 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e795012f60 0x55e795411770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:15.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.573+0000 7f8ea0b97740 4 client.18420 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:44:15.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.573+0000 7f8ea0b97740 1 -- 172.21.15.90:0/314518980 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e795368990 con 0x55e795012f60 2024-04-25T11:44:15.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.573+0000 7f8e937fe640 1 --2- 172.21.15.90:0/314518980 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e795012f60 0x55e795411770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:15.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.573+0000 7f8e937fe640 1 --2- 172.21.15.90:0/314518980 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e795012f60 0x55e795411770 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=797932ee221a894a server_cookie=94c14d08ea4e43bb in_seq=0 out_seq=0 2024-04-25T11:44:15.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.573+0000 7f8e72ffd640 10 client.18420 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:15.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.574+0000 7f8e72ffd640 1 -- 172.21.15.90:0/314518980 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e795368990 con 0x55e795012f60 2024-04-25T11:44:15.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.574+0000 7f8e72ffd640 10 client.18420 handle_command_reply: tid=0 2024-04-25T11:44:15.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.574+0000 7f8ea0b97740 1 client.18420 shutdown 2024-04-25T11:44:15.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.574+0000 7f8ea0b97740 1 -- 172.21.15.90:0/314518980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7950336b0 msgr2=0x55e7953ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:15.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.574+0000 7f8ea0b97740 1 --2- 172.21.15.90:0/314518980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7950336b0 0x55e7953ffba0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f8e880a6920 tx=0x7f8e88017600 comp rx=0 tx=0).stop 2024-04-25T11:44:15.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.574+0000 7f8ea0b97740 1 -- 172.21.15.90:0/314518980 shutdown_connections 2024-04-25T11:44:15.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.574+0000 7f8ea0b97740 1 --2- 172.21.15.90:0/314518980 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e795012f60 0x55e795411770 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.574+0000 7f8ea0b97740 1 --2- 172.21.15.90:0/314518980 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e795054390 0x55e795405440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.574+0000 7f8e72ffd640 0 client.18420 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:15.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.574+0000 7f8ea0b97740 1 --2- 172.21.15.90:0/314518980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7950336b0 0x55e7953ffba0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.574+0000 7f8ea0b97740 1 --2- 172.21.15.90:0/314518980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e79502b4d0 0x55e7953ff660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.574+0000 7f8ea0b97740 1 -- 172.21.15.90:0/314518980 >> 172.21.15.90:0/314518980 conn(0x55e7950484e0 msgr2=0x55e7953f6780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:15.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.574+0000 7f8ea0b97740 1 -- 172.21.15.90:0/314518980 shutdown_connections 2024-04-25T11:44:15.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.574+0000 7f8ea0b97740 1 -- 172.21.15.90:0/314518980 wait complete. 2024-04-25T11:44:15.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.575+0000 7f8ea0b97740 20 client.18420 trim_cache size 0 max 16384 2024-04-25T11:44:15.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.598+0000 7f8ea0b97740 1 Processor -- start 2024-04-25T11:44:15.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.598+0000 7f8ea0b97740 1 -- start start 2024-04-25T11:44:15.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.598+0000 7f8ea0b97740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e795012f60 0x55e7953fed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:15.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.598+0000 7f8ea0b97740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e79502b4d0 0x55e7954053c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:15.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.598+0000 7f8ea0b97740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7950336b0 0x55e795405900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:15.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.598+0000 7f8ea0b97740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8e941b1ea0 con 0x55e7950336b0 2024-04-25T11:44:15.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.598+0000 7f8ea0b97740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8e941b39e0 con 0x55e79502b4d0 2024-04-25T11:44:15.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.598+0000 7f8ea0b97740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8e941b2570 con 0x55e795012f60 2024-04-25T11:44:15.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.599+0000 7f8e93fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7950336b0 0x55e795405900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:15.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.599+0000 7f8e92ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e79502b4d0 0x55e7954053c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:15.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.599+0000 7f8e93fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7950336b0 0x55e795405900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47294/0 (socket says 172.21.15.90:47294) 2024-04-25T11:44:15.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.599+0000 7f8e92ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e79502b4d0 0x55e7954053c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:56810/0 (socket says 172.21.15.90:56810) 2024-04-25T11:44:15.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.599+0000 7f8e93fff640 1 -- 172.21.15.90:0/1376276537 learned_addr learned my addr 172.21.15.90:0/1376276537 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:15.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.599+0000 7f8e937fe640 1 --2- 172.21.15.90:0/1376276537 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e795012f60 0x55e7953fed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:15.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.599+0000 7f8e93fff640 1 -- 172.21.15.90:0/1376276537 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e795012f60 msgr2=0x55e7953fed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:15.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.599+0000 7f8e93fff640 1 --2- 172.21.15.90:0/1376276537 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e795012f60 0x55e7953fed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.599+0000 7f8e93fff640 1 -- 172.21.15.90:0/1376276537 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e79502b4d0 msgr2=0x55e7954053c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:15.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.599+0000 7f8e93fff640 1 --2- 172.21.15.90:0/1376276537 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e79502b4d0 0x55e7954053c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.599+0000 7f8e93fff640 1 -- 172.21.15.90:0/1376276537 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e79502c0f0 con 0x55e7950336b0 2024-04-25T11:44:15.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.599+0000 7f8e937fe640 1 --2- 172.21.15.90:0/1376276537 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e795012f60 0x55e7953fed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:15.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.599+0000 7f8e92ffd640 1 --2- 172.21.15.90:0/1376276537 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e79502b4d0 0x55e7954053c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:15.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.599+0000 7f8e93fff640 1 --2- 172.21.15.90:0/1376276537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7950336b0 0x55e795405900 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c003dc0 tx=0x7f8e8c007940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5268f293867a3c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:15.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.599+0000 7f8e797fa640 1 -- 172.21.15.90:0/1376276537 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c004b20 con 0x55e7950336b0 2024-04-25T11:44:15.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.599+0000 7f8e797fa640 1 -- 172.21.15.90:0/1376276537 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e8c01a070 con 0x55e7950336b0 2024-04-25T11:44:15.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.599+0000 7f8e797fa640 1 -- 172.21.15.90:0/1376276537 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c005350 con 0x55e7950336b0 2024-04-25T11:44:15.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.601+0000 7f8ea0b97740 1 -- 172.21.15.90:0/1376276537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7950336b0 msgr2=0x55e795405900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:15.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.601+0000 7f8ea0b97740 1 --2- 172.21.15.90:0/1376276537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7950336b0 0x55e795405900 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c003dc0 tx=0x7f8e8c007940 comp rx=0 tx=0).stop 2024-04-25T11:44:15.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.601+0000 7f8ea0b97740 1 -- 172.21.15.90:0/1376276537 shutdown_connections 2024-04-25T11:44:15.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.601+0000 7f8ea0b97740 1 --2- 172.21.15.90:0/1376276537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7950336b0 0x55e795405900 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.601+0000 7f8ea0b97740 1 --2- 172.21.15.90:0/1376276537 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e79502b4d0 0x55e7954053c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.601+0000 7f8ea0b97740 1 --2- 172.21.15.90:0/1376276537 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e795012f60 0x55e7953fed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.601+0000 7f8ea0b97740 1 -- 172.21.15.90:0/1376276537 >> 172.21.15.90:0/1376276537 conn(0x55e7950484e0 msgr2=0x55e795400a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:15.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.601+0000 7f8ea0b97740 1 -- 172.21.15.90:0/1376276537 shutdown_connections 2024-04-25T11:44:15.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.601+0000 7f8ea0b97740 1 -- 172.21.15.90:0/1376276537 wait complete. 2024-04-25T11:44:15.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.601+0000 7f8ea0b97740 1 -- start start 2024-04-25T11:44:15.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.601+0000 7f8ea0b97740 1 Processor -- start 2024-04-25T11:44:15.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.602+0000 7f8ea0b97740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e795012f60 0x55e79540b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:15.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.602+0000 7f8ea0b97740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e79502b4d0 0x55e79540b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:15.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.602+0000 7f8ea0b97740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e7950336b0 0x55e7954066f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:15.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.602+0000 7f8ea0b97740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8e941b3310 con 0x55e795012f60 2024-04-25T11:44:15.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.602+0000 7f8ea0b97740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8e941b2c40 con 0x55e79502b4d0 2024-04-25T11:44:15.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.602+0000 7f8ea0b97740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8e941afea0 con 0x55e7950336b0 2024-04-25T11:44:15.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.602+0000 7f8ea0b97740 10 client.0 _ll_register_callbacks cb 0x7f8e8800df10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:15.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.602+0000 7f8e937fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e795012f60 0x55e79540b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:15.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.602+0000 7f8e937fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e795012f60 0x55e79540b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47304/0 (socket says 172.21.15.90:47304) 2024-04-25T11:44:15.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.602+0000 7f8e937fe640 1 -- 172.21.15.90:0/1890026811 learned_addr learned my addr 172.21.15.90:0/1890026811 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:15.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.602+0000 7f8e92ffd640 1 --2- 172.21.15.90:0/1890026811 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e79502b4d0 0x55e79540b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:15.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.602+0000 7f8e93fff640 1 --2- 172.21.15.90:0/1890026811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e7950336b0 0x55e7954066f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:15.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.602+0000 7f8e92ffd640 1 -- 172.21.15.90:0/1890026811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e7950336b0 msgr2=0x55e7954066f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:15.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.602+0000 7f8e92ffd640 1 --2- 172.21.15.90:0/1890026811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e7950336b0 0x55e7954066f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.602+0000 7f8e92ffd640 1 -- 172.21.15.90:0/1890026811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e795012f60 msgr2=0x55e79540b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:15.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.602+0000 7f8e92ffd640 1 --2- 172.21.15.90:0/1890026811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e795012f60 0x55e79540b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.602+0000 7f8e92ffd640 1 -- 172.21.15.90:0/1890026811 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e79531b4b0 con 0x55e79502b4d0 2024-04-25T11:44:15.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.602+0000 7f8e937fe640 1 --2- 172.21.15.90:0/1890026811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e795012f60 0x55e79540b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:15.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.602+0000 7f8e92ffd640 1 --2- 172.21.15.90:0/1890026811 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e79502b4d0 0x55e79540b7f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8e880dc1d0 tx=0x7f8e880a6ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5814043d33f61339 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:15.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.602+0000 7f8e72ffd640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:44:15.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.602+0000 7f8e72ffd640 1 -- 172.21.15.90:0/1890026811 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e880a6390 con 0x55e79502b4d0 2024-04-25T11:44:15.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.603+0000 7f8ea0b97740 1 -- 172.21.15.90:0/1890026811 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e8c01a070 con 0x55e79502b4d0 2024-04-25T11:44:15.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.603+0000 7f8e72ffd640 1 -- 172.21.15.90:0/1890026811 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7f8e880a5970 con 0x55e79502b4d0 2024-04-25T11:44:15.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.603+0000 7f8e72ffd640 1 -- 172.21.15.90:0/1890026811 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8e880a6570 con 0x55e79502b4d0 2024-04-25T11:44:15.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.603+0000 7f8ea0b97740 10 client.27188 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:15.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.603+0000 7f8ea0b97740 1 -- 172.21.15.90:0/1890026811 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e7954793f0 con 0x55e79502b4d0 2024-04-25T11:44:15.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.603+0000 7f8e72ffd640 1 -- 172.21.15.90:0/1890026811 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e8c01a070 con 0x55e79502b4d0 2024-04-25T11:44:15.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.603+0000 7f8e72ffd640 1 -- 172.21.15.90:0/1890026811 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f8e88085030 con 0x55e79502b4d0 2024-04-25T11:44:15.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.603+0000 7f8ea0b97740 10 client.27188 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:15.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.603+0000 7f8ea0b97740 10 client.27188 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:44:15.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.603+0000 7f8ea0b97740 20 client.27188 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:15.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.603+0000 7f8ea0b97740 1 --2- 172.21.15.90:0/1890026811 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e795054390 0x55e79547bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:15.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.603+0000 7f8ea0b97740 4 client.27188 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:44:15.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.603+0000 7f8ea0b97740 1 -- 172.21.15.90:0/1890026811 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e7953f5330 con 0x55e795054390 2024-04-25T11:44:15.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.603+0000 7f8e937fe640 1 --2- 172.21.15.90:0/1890026811 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e795054390 0x55e79547bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:15.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.604+0000 7f8e72ffd640 10 client.27188 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:15.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.604+0000 7f8e937fe640 1 --2- 172.21.15.90:0/1890026811 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e795054390 0x55e79547bbb0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae59cb38d8f5a982 server_cookie=6bf4a49b64f4de6e in_seq=0 out_seq=0 2024-04-25T11:44:15.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.604+0000 7f8e72ffd640 1 -- 172.21.15.90:0/1890026811 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e7953f5330 con 0x55e795054390 2024-04-25T11:44:15.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.604+0000 7f8e72ffd640 10 client.27188 handle_command_reply: tid=0 2024-04-25T11:44:15.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.604+0000 7f8ea0b97740 1 client.27188 shutdown 2024-04-25T11:44:15.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.604+0000 7f8ea0b97740 1 -- 172.21.15.90:0/1890026811 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e79502b4d0 msgr2=0x55e79540b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:15.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.604+0000 7f8ea0b97740 1 --2- 172.21.15.90:0/1890026811 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e79502b4d0 0x55e79540b7f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8e880dc1d0 tx=0x7f8e880a6ad0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.604+0000 7f8ea0b97740 1 -- 172.21.15.90:0/1890026811 shutdown_connections 2024-04-25T11:44:15.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.604+0000 7f8ea0b97740 1 --2- 172.21.15.90:0/1890026811 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55e795054390 0x55e79547bbb0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.604+0000 7f8ea0b97740 1 --2- 172.21.15.90:0/1890026811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e7950336b0 0x55e7954066f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.604+0000 7f8ea0b97740 1 --2- 172.21.15.90:0/1890026811 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55e79502b4d0 0x55e79540b7f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.604+0000 7f8ea0b97740 1 --2- 172.21.15.90:0/1890026811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e795012f60 0x55e79540b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.604+0000 7f8e72ffd640 0 client.27188 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:15.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.604+0000 7f8ea0b97740 1 -- 172.21.15.90:0/1890026811 >> 172.21.15.90:0/1890026811 conn(0x55e7950484e0 msgr2=0x55e795401480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:15.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.604+0000 7f8ea0b97740 1 -- 172.21.15.90:0/1890026811 shutdown_connections 2024-04-25T11:44:15.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.604+0000 7f8ea0b97740 1 -- 172.21.15.90:0/1890026811 wait complete. 2024-04-25T11:44:15.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.604+0000 7f8ea0b97740 20 client.27188 trim_cache size 0 max 16384 2024-04-25T11:44:15.599 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:44:15.600 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-25T11:44:15.600 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-25T11:44:15.600 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:44:15.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.605+0000 7f8e99ce7640 1 -- 172.21.15.90:0/2345121378 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8e7c0afc60 msgr2=0x7f8e7c0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:15.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.605+0000 7f8e99ce7640 1 --2- 172.21.15.90:0/2345121378 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8e7c0afc60 0x7f8e7c0b2170 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x55e7953f1bd0 tx=0x7f8e8c007520 comp rx=0 tx=0).stop 2024-04-25T11:44:15.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.605+0000 7f8e99ce7640 1 -- 172.21.15.90:0/2345121378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e94109b10 msgr2=0x7f8e940764f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:15.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.605+0000 7f8e99ce7640 1 --2- 172.21.15.90:0/2345121378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e94109b10 0x7f8e940764f0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f8e88005d00 tx=0x7f8e8800dde0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.605+0000 7f8e99ce7640 1 -- 172.21.15.90:0/2345121378 shutdown_connections 2024-04-25T11:44:15.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.605+0000 7f8e99ce7640 1 --2- 172.21.15.90:0/2345121378 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8e7c0afc60 0x7f8e7c0b2170 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.605+0000 7f8e99ce7640 1 --2- 172.21.15.90:0/2345121378 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8e94109fb0 0x7f8e94076a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.605+0000 7f8e99ce7640 1 --2- 172.21.15.90:0/2345121378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e94109b10 0x7f8e940764f0 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.605+0000 7f8e99ce7640 1 --2- 172.21.15.90:0/2345121378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e94104260 0x7f8e94075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.605+0000 7f8e99ce7640 1 -- 172.21.15.90:0/2345121378 >> 172.21.15.90:0/2345121378 conn(0x7f8e940ff1d0 msgr2=0x7f8e940ff5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:15.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.605+0000 7f8e99ce7640 1 -- 172.21.15.90:0/2345121378 shutdown_connections 2024-04-25T11:44:15.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.605+0000 7f8e99ce7640 1 -- 172.21.15.90:0/2345121378 wait complete. 2024-04-25T11:44:15.603 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T11:44:15.603 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:44:15.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.732+0000 7f8286ce7640 1 -- 172.21.15.90:0/2390982666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f828010d380 msgr2=0x7f828010f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:15.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.732+0000 7f8286ce7640 1 --2- 172.21.15.90:0/2390982666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f828010d380 0x7f828010f7a0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f8270009900 tx=0x7f827002a410 comp rx=0 tx=0).stop 2024-04-25T11:44:15.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.732+0000 7f8286ce7640 1 -- 172.21.15.90:0/2390982666 shutdown_connections 2024-04-25T11:44:15.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.732+0000 7f8286ce7640 1 --2- 172.21.15.90:0/2390982666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f828010d380 0x7f828010f7a0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.732+0000 7f8286ce7640 1 --2- 172.21.15.90:0/2390982666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8280102af0 0x7f8280102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.732+0000 7f8286ce7640 1 --2- 172.21.15.90:0/2390982666 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8280108a20 0x7f82801025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.732+0000 7f8286ce7640 1 -- 172.21.15.90:0/2390982666 >> 172.21.15.90:0/2390982666 conn(0x7f8280078110 msgr2=0x7f8280078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:15.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.732+0000 7f8286ce7640 1 -- 172.21.15.90:0/2390982666 shutdown_connections 2024-04-25T11:44:15.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.732+0000 7f8286ce7640 1 -- 172.21.15.90:0/2390982666 wait complete. 2024-04-25T11:44:15.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.732+0000 7f8286ce7640 1 Processor -- start 2024-04-25T11:44:15.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.733+0000 7f8286ce7640 1 -- start start 2024-04-25T11:44:15.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.733+0000 7f8286ce7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8280102af0 0x7f8280199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:15.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.733+0000 7f8286ce7640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8280108a20 0x7f8280199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:15.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.733+0000 7f8286ce7640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f828010d380 0x7f828019dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:15.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.733+0000 7f8286ce7640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8280112c50 con 0x7f828010d380 2024-04-25T11:44:15.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.733+0000 7f8286ce7640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8280112ad0 con 0x7f8280108a20 2024-04-25T11:44:15.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.733+0000 7f8286ce7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8280112dd0 con 0x7f8280102af0 2024-04-25T11:44:15.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.733+0000 7f8285284640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f828010d380 0x7f828019dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:15.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.733+0000 7f8285284640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f828010d380 0x7f828019dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47314/0 (socket says 172.21.15.90:47314) 2024-04-25T11:44:15.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.733+0000 7f8285284640 1 -- 172.21.15.90:0/417597251 learned_addr learned my addr 172.21.15.90:0/417597251 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:15.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.733+0000 7f8284a83640 1 --2- 172.21.15.90:0/417597251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8280102af0 0x7f8280199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:15.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.733+0000 7f827ffff640 1 --2- 172.21.15.90:0/417597251 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8280108a20 0x7f8280199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:15.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.733+0000 7f8285284640 1 -- 172.21.15.90:0/417597251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8280102af0 msgr2=0x7f8280199570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:15.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.733+0000 7f8285284640 1 --2- 172.21.15.90:0/417597251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8280102af0 0x7f8280199570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.733+0000 7f8285284640 1 -- 172.21.15.90:0/417597251 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8280108a20 msgr2=0x7f8280199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:15.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.733+0000 7f8285284640 1 --2- 172.21.15.90:0/417597251 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8280108a20 0x7f8280199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.733+0000 7f8285284640 1 -- 172.21.15.90:0/417597251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f828019e3d0 con 0x7f828010d380 2024-04-25T11:44:15.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.733+0000 7f8285284640 1 --2- 172.21.15.90:0/417597251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f828010d380 0x7f828019dd90 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f827002a930 tx=0x7f8270003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:15.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.734+0000 7f827dffb640 1 -- 172.21.15.90:0/417597251 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827003f070 con 0x7f828010d380 2024-04-25T11:44:15.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.734+0000 7f8286ce7640 1 -- 172.21.15.90:0/417597251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f828019e660 con 0x7f828010d380 2024-04-25T11:44:15.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.734+0000 7f827dffb640 1 -- 172.21.15.90:0/417597251 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8270031b50 con 0x7f828010d380 2024-04-25T11:44:15.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.734+0000 7f827dffb640 1 -- 172.21.15.90:0/417597251 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8270035030 con 0x7f828010d380 2024-04-25T11:44:15.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.734+0000 7f8286ce7640 1 -- 172.21.15.90:0/417597251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f828019eba0 con 0x7f828010d380 2024-04-25T11:44:15.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.735+0000 7f8286ce7640 1 -- 172.21.15.90:0/417597251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f828006afa0 con 0x7f828010d380 2024-04-25T11:44:15.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.736+0000 7f827dffb640 1 -- 172.21.15.90:0/417597251 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f8270042070 con 0x7f828010d380 2024-04-25T11:44:15.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.737+0000 7f827dffb640 1 --2- 172.21.15.90:0/417597251 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f82680afc60 0x7f82680b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:15.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.737+0000 7f827dffb640 1 -- 172.21.15.90:0/417597251 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7f82700fffa0 con 0x7f828010d380 2024-04-25T11:44:15.723 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:15 smithi090 ceph-mon[29575]: pgmap v283: 97 pgs: 21 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 41 active+clean; 46 GiB data, 143 GiB used, 930 GiB / 1.0 TiB avail; 9.9 KiB/s rd, 54 KiB/s wr, 18 op/s 2024-04-25T11:44:15.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.737+0000 7f8284a83640 1 --2- 172.21.15.90:0/417597251 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f82680afc60 0x7f82680b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:15.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.738+0000 7f8284a83640 1 --2- 172.21.15.90:0/417597251 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f82680afc60 0x7f82680b2170 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8280103a20 tx=0x7f8278009500 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:15.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.740+0000 7f827dffb640 1 -- 172.21.15.90:0/417597251 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f82700c8790 con 0x7f828010d380 2024-04-25T11:44:15.800 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:15 smithi138 ceph-mon[32328]: pgmap v283: 97 pgs: 21 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 41 active+clean; 46 GiB data, 143 GiB used, 930 GiB / 1.0 TiB avail; 9.9 KiB/s rd, 54 KiB/s wr, 18 op/s 2024-04-25T11:44:15.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:15 smithi156 ceph-mon[33045]: pgmap v283: 97 pgs: 21 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 41 active+clean; 46 GiB data, 143 GiB used, 930 GiB / 1.0 TiB avail; 9.9 KiB/s rd, 54 KiB/s wr, 18 op/s 2024-04-25T11:44:15.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.983+0000 7f8286ce7640 1 -- 172.21.15.90:0/417597251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f828019a8c0 con 0x7f828010d380 2024-04-25T11:44:15.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.984+0000 7f827dffb640 1 -- 172.21.15.90:0/417597251 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11930 (secure 0 0 0) 0x7f82700c8090 con 0x7f828010d380 2024-04-25T11:44:15.970 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:44:15.970 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:39:20.469254+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24458,24431,24499,24475,24524]},"id":1}]} 2024-04-25T11:44:15.970 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 28 2024-04-25T11:44:15.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.986+0000 7f8286ce7640 1 -- 172.21.15.90:0/417597251 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f82680afc60 msgr2=0x7f82680b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:15.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.986+0000 7f8286ce7640 1 --2- 172.21.15.90:0/417597251 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f82680afc60 0x7f82680b2170 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8280103a20 tx=0x7f8278009500 comp rx=0 tx=0).stop 2024-04-25T11:44:15.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.986+0000 7f8286ce7640 1 -- 172.21.15.90:0/417597251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f828010d380 msgr2=0x7f828019dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:15.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.986+0000 7f8286ce7640 1 --2- 172.21.15.90:0/417597251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f828010d380 0x7f828019dd90 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f827002a930 tx=0x7f8270003b90 comp rx=0 tx=0).stop 2024-04-25T11:44:15.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.986+0000 7f8286ce7640 1 -- 172.21.15.90:0/417597251 shutdown_connections 2024-04-25T11:44:15.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.986+0000 7f8286ce7640 1 --2- 172.21.15.90:0/417597251 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f82680afc60 0x7f82680b2170 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.986+0000 7f8286ce7640 1 --2- 172.21.15.90:0/417597251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f828010d380 0x7f828019dd90 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.986+0000 7f8286ce7640 1 --2- 172.21.15.90:0/417597251 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8280108a20 0x7f8280199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.986+0000 7f8286ce7640 1 --2- 172.21.15.90:0/417597251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8280102af0 0x7f8280199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.986+0000 7f8286ce7640 1 -- 172.21.15.90:0/417597251 >> 172.21.15.90:0/417597251 conn(0x7f8280078110 msgr2=0x7f82801040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:15.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.986+0000 7f8286ce7640 1 -- 172.21.15.90:0/417597251 shutdown_connections 2024-04-25T11:44:15.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:15.986+0000 7f8286ce7640 1 -- 172.21.15.90:0/417597251 wait complete. 2024-04-25T11:44:15.984 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T11:44:16.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.114+0000 7f4626242640 1 -- 172.21.15.90:0/1927932667 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4620107450 msgr2=0x7f462010ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.114+0000 7f4626242640 1 --2- 172.21.15.90:0/1927932667 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4620107450 0x7f462010ebf0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f4618009ca0 tx=0x7f461802a8c0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.114+0000 7f4626242640 1 -- 172.21.15.90:0/1927932667 shutdown_connections 2024-04-25T11:44:16.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.114+0000 7f4626242640 1 --2- 172.21.15.90:0/1927932667 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4620107450 0x7f462010ebf0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.114+0000 7f4626242640 1 --2- 172.21.15.90:0/1927932667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4620106990 0x7f4620106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.114+0000 7f4626242640 1 --2- 172.21.15.90:0/1927932667 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4620102400 0x7f4620102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.114+0000 7f4626242640 1 -- 172.21.15.90:0/1927932667 >> 172.21.15.90:0/1927932667 conn(0x7f4620077880 msgr2=0x7f4620077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.115+0000 7f4626242640 1 -- 172.21.15.90:0/1927932667 shutdown_connections 2024-04-25T11:44:16.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.115+0000 7f4626242640 1 -- 172.21.15.90:0/1927932667 wait complete. 2024-04-25T11:44:16.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.115+0000 7f4626242640 1 Processor -- start 2024-04-25T11:44:16.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.115+0000 7f4626242640 1 -- start start 2024-04-25T11:44:16.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.115+0000 7f4626242640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4620102400 0x7f462019a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.115+0000 7f4626242640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4620106990 0x7f462019ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.116+0000 7f4626242640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4620107450 0x7f462019efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.116+0000 7f4626242640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4620113cf0 con 0x7f4620106990 2024-04-25T11:44:16.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.116+0000 7f4626242640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4620113b70 con 0x7f4620102400 2024-04-25T11:44:16.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.116+0000 7f4626242640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4620113e70 con 0x7f4620107450 2024-04-25T11:44:16.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.116+0000 7f461effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4620106990 0x7f462019ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.116+0000 7f461effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4620106990 0x7f462019ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47324/0 (socket says 172.21.15.90:47324) 2024-04-25T11:44:16.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.116+0000 7f461effd640 1 -- 172.21.15.90:0/945730744 learned_addr learned my addr 172.21.15.90:0/945730744 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:16.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.116+0000 7f461f7fe640 1 --2- 172.21.15.90:0/945730744 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4620102400 0x7f462019a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.116+0000 7f461ffff640 1 --2- 172.21.15.90:0/945730744 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4620107450 0x7f462019efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.116+0000 7f461effd640 1 -- 172.21.15.90:0/945730744 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4620107450 msgr2=0x7f462019efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.116+0000 7f461effd640 1 --2- 172.21.15.90:0/945730744 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4620107450 0x7f462019efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.116+0000 7f461effd640 1 -- 172.21.15.90:0/945730744 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4620102400 msgr2=0x7f462019a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.116+0000 7f461effd640 1 --2- 172.21.15.90:0/945730744 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4620102400 0x7f462019a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.116+0000 7f461effd640 1 -- 172.21.15.90:0/945730744 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f462019f610 con 0x7f4620106990 2024-04-25T11:44:16.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.116+0000 7f461effd640 1 --2- 172.21.15.90:0/945730744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4620106990 0x7f462019ad10 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f461400e9f0 tx=0x7f461400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.116+0000 7f461cff9640 1 -- 172.21.15.90:0/945730744 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461400ccc0 con 0x7f4620106990 2024-04-25T11:44:16.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.116+0000 7f461cff9640 1 -- 172.21.15.90:0/945730744 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46140054d0 con 0x7f4620106990 2024-04-25T11:44:16.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.116+0000 7f461cff9640 1 -- 172.21.15.90:0/945730744 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4614010430 con 0x7f4620106990 2024-04-25T11:44:16.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.116+0000 7f4626242640 1 -- 172.21.15.90:0/945730744 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f462019f900 con 0x7f4620106990 2024-04-25T11:44:16.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.116+0000 7f4626242640 1 -- 172.21.15.90:0/945730744 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46201a7280 con 0x7f4620106990 2024-04-25T11:44:16.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.119+0000 7f462d1ce740 1 Processor -- start 2024-04-25T11:44:16.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.119+0000 7f461cff9640 1 -- 172.21.15.90:0/945730744 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f46140046c0 con 0x7f4620106990 2024-04-25T11:44:16.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.119+0000 7f462d1ce740 1 -- start start 2024-04-25T11:44:16.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.119+0000 7f462d1ce740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3214cc390 0x55f32186d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.119+0000 7f462d1ce740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f3214a34d0 0x55f32186dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.119+0000 7f462d1ce740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214ab6b0 0x55f32187a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.119+0000 7f462d1ce740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f3217d2d80 con 0x55f3214ab6b0 2024-04-25T11:44:16.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.119+0000 7f462d1ce740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f32178ca20 con 0x55f3214a34d0 2024-04-25T11:44:16.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.119+0000 7f462d1ce740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f3217ced40 con 0x55f3214cc390 2024-04-25T11:44:16.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.119+0000 7f461effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214ab6b0 0x55f32187a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.119+0000 7f461f7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f3214a34d0 0x55f32186dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.119+0000 7f461effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214ab6b0 0x55f32187a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47332/0 (socket says 172.21.15.90:47332) 2024-04-25T11:44:16.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.119+0000 7f461effd640 1 -- 172.21.15.90:0/3642993282 learned_addr learned my addr 172.21.15.90:0/3642993282 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:16.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.119+0000 7f461ffff640 1 --2- 172.21.15.90:0/3642993282 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3214cc390 0x55f32186d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.119+0000 7f461cff9640 1 --2- 172.21.15.90:0/945730744 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f46080afce0 0x7f46080b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.119+0000 7f461effd640 1 -- 172.21.15.90:0/3642993282 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3214cc390 msgr2=0x55f32186d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.119+0000 7f461effd640 1 --2- 172.21.15.90:0/3642993282 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3214cc390 0x55f32186d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.119+0000 7f461effd640 1 -- 172.21.15.90:0/3642993282 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f3214a34d0 msgr2=0x55f32186dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.120+0000 7f461effd640 1 --2- 172.21.15.90:0/3642993282 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f3214a34d0 0x55f32186dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.120+0000 7f461effd640 1 -- 172.21.15.90:0/3642993282 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f321797c30 con 0x55f3214ab6b0 2024-04-25T11:44:16.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.120+0000 7f461f7fe640 1 --2- 172.21.15.90:0/3642993282 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f3214a34d0 0x55f32186dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:16.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.120+0000 7f461ffff640 1 --2- 172.21.15.90:0/945730744 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f46080afce0 0x7f46080b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.120+0000 7f461cff9640 1 -- 172.21.15.90:0/945730744 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7f46140df450 con 0x7f4620106990 2024-04-25T11:44:16.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.120+0000 7f461effd640 1 --2- 172.21.15.90:0/3642993282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214ab6b0 0x55f32187a6a0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f46140dca60 tx=0x7f46140d5c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bfc87a5d0698658 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.120+0000 7f4605ffb640 1 -- 172.21.15.90:0/3642993282 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46140a6e90 con 0x55f3214ab6b0 2024-04-25T11:44:16.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.120+0000 7f4605ffb640 1 -- 172.21.15.90:0/3642993282 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46140a7050 con 0x55f3214ab6b0 2024-04-25T11:44:16.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.120+0000 7f4605ffb640 1 -- 172.21.15.90:0/3642993282 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46140aeeb0 con 0x55f3214ab6b0 2024-04-25T11:44:16.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.121+0000 7f462d1ce740 1 -- 172.21.15.90:0/3642993282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214ab6b0 msgr2=0x55f32187a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.121+0000 7f461ffff640 1 --2- 172.21.15.90:0/945730744 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f46080afce0 0x7f46080b21f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x55f32186a980 tx=0x7f46180095c0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.121+0000 7f462d1ce740 1 --2- 172.21.15.90:0/3642993282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214ab6b0 0x55f32187a6a0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f46140dca60 tx=0x7f46140d5c90 comp rx=0 tx=0).stop 2024-04-25T11:44:16.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.121+0000 7f462d1ce740 1 -- 172.21.15.90:0/3642993282 shutdown_connections 2024-04-25T11:44:16.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.121+0000 7f462d1ce740 1 --2- 172.21.15.90:0/3642993282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214ab6b0 0x55f32187a6a0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.121+0000 7f462d1ce740 1 --2- 172.21.15.90:0/3642993282 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f3214a34d0 0x55f32186dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.121+0000 7f462d1ce740 1 --2- 172.21.15.90:0/3642993282 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3214cc390 0x55f32186d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.121+0000 7f462d1ce740 1 -- 172.21.15.90:0/3642993282 >> 172.21.15.90:0/3642993282 conn(0x55f3214c04e0 msgr2=0x55f321869fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.121+0000 7f462d1ce740 1 -- 172.21.15.90:0/3642993282 shutdown_connections 2024-04-25T11:44:16.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.121+0000 7f462d1ce740 1 -- 172.21.15.90:0/3642993282 wait complete. 2024-04-25T11:44:16.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.121+0000 7f462d1ce740 1 -- start start 2024-04-25T11:44:16.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.122+0000 7f462d1ce740 1 Processor -- start 2024-04-25T11:44:16.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.122+0000 7f462d1ce740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f3214a34d0 0x55f3218782a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.122+0000 7f462d1ce740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214ab6b0 0x55f3218787e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.122+0000 7f462d1ce740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3214cc390 0x55f3218841b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.122+0000 7f462d1ce740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f32187ad30 con 0x55f3214ab6b0 2024-04-25T11:44:16.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.122+0000 7f462d1ce740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f321759dc0 con 0x55f3214a34d0 2024-04-25T11:44:16.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.122+0000 7f462d1ce740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f32187aeb0 con 0x55f3214cc390 2024-04-25T11:44:16.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.122+0000 7f462d1ce740 10 client.0 _ll_register_callbacks cb 0x55f321775600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:16.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.122+0000 7f461effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214ab6b0 0x55f3218787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.122+0000 7f461effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214ab6b0 0x55f3218787e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47346/0 (socket says 172.21.15.90:47346) 2024-04-25T11:44:16.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.122+0000 7f461effd640 1 -- 172.21.15.90:0/2814737504 learned_addr learned my addr 172.21.15.90:0/2814737504 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:16.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.122+0000 7f461ffff640 1 --2- 172.21.15.90:0/2814737504 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3214cc390 0x55f3218841b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.122+0000 7f461f7fe640 1 --2- 172.21.15.90:0/2814737504 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f3214a34d0 0x55f3218782a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.122+0000 7f461effd640 1 -- 172.21.15.90:0/2814737504 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3214cc390 msgr2=0x55f3218841b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.122+0000 7f461effd640 1 --2- 172.21.15.90:0/2814737504 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3214cc390 0x55f3218841b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.122+0000 7f461effd640 1 -- 172.21.15.90:0/2814737504 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f3214a34d0 msgr2=0x55f3218782a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.122+0000 7f461effd640 1 --2- 172.21.15.90:0/2814737504 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f3214a34d0 0x55f3218782a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.122+0000 7f461effd640 1 -- 172.21.15.90:0/2814737504 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3214e43f0 con 0x55f3214ab6b0 2024-04-25T11:44:16.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.122+0000 7f461ffff640 1 --2- 172.21.15.90:0/2814737504 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3214cc390 0x55f3218841b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:44:16.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.122+0000 7f461f7fe640 1 --2- 172.21.15.90:0/2814737504 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f3214a34d0 0x55f3218782a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:44:16.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.123+0000 7f461effd640 1 --2- 172.21.15.90:0/2814737504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214ab6b0 0x55f3218787e0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f46140dcba0 tx=0x7f46140a73d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad370099375fba9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.123+0000 7f45feffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:16.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.123+0000 7f45feffd640 1 -- 172.21.15.90:0/2814737504 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46140afd00 con 0x55f3214ab6b0 2024-04-25T11:44:16.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.123+0000 7f45feffd640 1 -- 172.21.15.90:0/2814737504 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7f46140dcd70 con 0x55f3214ab6b0 2024-04-25T11:44:16.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.123+0000 7f462d1ce740 1 -- 172.21.15.90:0/2814737504 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46140a7050 con 0x55f3214ab6b0 2024-04-25T11:44:16.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.123+0000 7f45feffd640 1 -- 172.21.15.90:0/2814737504 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f46140a6cb0 con 0x55f3214ab6b0 2024-04-25T11:44:16.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.123+0000 7f462d1ce740 10 client.18444 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:16.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.123+0000 7f462d1ce740 1 -- 172.21.15.90:0/2814737504 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f3215ae7d0 con 0x55f3214ab6b0 2024-04-25T11:44:16.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.123+0000 7f45feffd640 1 -- 172.21.15.90:0/2814737504 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46140a7050 con 0x55f3214ab6b0 2024-04-25T11:44:16.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.123+0000 7f45feffd640 1 -- 172.21.15.90:0/2814737504 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f46140a37d0 con 0x55f3214ab6b0 2024-04-25T11:44:16.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.123+0000 7f462d1ce740 10 client.18444 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:16.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.123+0000 7f462d1ce740 10 client.18444 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:44:16.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.123+0000 7f462d1ce740 20 client.18444 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:16.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.123+0000 7f462d1ce740 1 --2- 172.21.15.90:0/2814737504 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55f32148af60 0x55f32188a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.123+0000 7f462d1ce740 4 client.18444 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:44:16.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.123+0000 7f462d1ce740 1 -- 172.21.15.90:0/2814737504 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3217e0990 con 0x55f32148af60 2024-04-25T11:44:16.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.124+0000 7f461f7fe640 1 --2- 172.21.15.90:0/2814737504 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55f32148af60 0x55f32188a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.124+0000 7f45feffd640 10 client.18444 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:44:16.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.124+0000 7f461f7fe640 1 --2- 172.21.15.90:0/2814737504 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55f32148af60 0x55f32188a4c0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=11cc6f50420eb2ae server_cookie=c0b8553408e59334 in_seq=0 out_seq=0 2024-04-25T11:44:16.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.125+0000 7f45feffd640 1 -- 172.21.15.90:0/2814737504 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f3217e0990 con 0x55f32148af60 2024-04-25T11:44:16.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.125+0000 7f45feffd640 10 client.18444 handle_command_reply: tid=0 2024-04-25T11:44:16.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.125+0000 7f462d1ce740 1 client.18444 shutdown 2024-04-25T11:44:16.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.125+0000 7f462d1ce740 1 -- 172.21.15.90:0/2814737504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214ab6b0 msgr2=0x55f3218787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.125+0000 7f462d1ce740 1 --2- 172.21.15.90:0/2814737504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214ab6b0 0x55f3218787e0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f46140dcba0 tx=0x7f46140a73d0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.125+0000 7f462d1ce740 1 -- 172.21.15.90:0/2814737504 shutdown_connections 2024-04-25T11:44:16.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.125+0000 7f462d1ce740 1 --2- 172.21.15.90:0/2814737504 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55f32148af60 0x55f32188a4c0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.125+0000 7f462d1ce740 1 --2- 172.21.15.90:0/2814737504 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3214cc390 0x55f3218841b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.125+0000 7f45feffd640 0 client.18444 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:44:16.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.125+0000 7f462d1ce740 1 --2- 172.21.15.90:0/2814737504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214ab6b0 0x55f3218787e0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.125+0000 7f462d1ce740 1 --2- 172.21.15.90:0/2814737504 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f3214a34d0 0x55f3218782a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.125+0000 7f462d1ce740 1 -- 172.21.15.90:0/2814737504 >> 172.21.15.90:0/2814737504 conn(0x55f3214c04e0 msgr2=0x55f32186f530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.125+0000 7f462d1ce740 1 -- 172.21.15.90:0/2814737504 shutdown_connections 2024-04-25T11:44:16.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.125+0000 7f462d1ce740 1 -- 172.21.15.90:0/2814737504 wait complete. 2024-04-25T11:44:16.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.125+0000 7f462d1ce740 20 client.18444 trim_cache size 0 max 16384 2024-04-25T11:44:16.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.139+0000 7f462d1ce740 1 Processor -- start 2024-04-25T11:44:16.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.139+0000 7f462d1ce740 1 -- start start 2024-04-25T11:44:16.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.140+0000 7f462d1ce740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214a34d0 0x55f3218841a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.140+0000 7f462d1ce740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3214ab6b0 0x55f321877c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.140+0000 7f462d1ce740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f3214cc390 0x55f32187a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.140+0000 7f462d1ce740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f46201afd90 con 0x55f3214a34d0 2024-04-25T11:44:16.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.140+0000 7f462d1ce740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f46201b18d0 con 0x55f3214cc390 2024-04-25T11:44:16.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.140+0000 7f462d1ce740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f46201b0460 con 0x55f3214ab6b0 2024-04-25T11:44:16.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.140+0000 7f461f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214a34d0 0x55f3218841a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.140+0000 7f461f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214a34d0 0x55f3218841a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47350/0 (socket says 172.21.15.90:47350) 2024-04-25T11:44:16.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.140+0000 7f461f7fe640 1 -- 172.21.15.90:0/2776063036 learned_addr learned my addr 172.21.15.90:0/2776063036 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:16.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.140+0000 7f461ffff640 1 --2- 172.21.15.90:0/2776063036 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f3214cc390 0x55f32187a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.140+0000 7f461effd640 1 --2- 172.21.15.90:0/2776063036 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3214ab6b0 0x55f321877c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.140+0000 7f461f7fe640 1 -- 172.21.15.90:0/2776063036 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3214ab6b0 msgr2=0x55f321877c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.140+0000 7f461f7fe640 1 --2- 172.21.15.90:0/2776063036 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3214ab6b0 0x55f321877c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.140+0000 7f461f7fe640 1 -- 172.21.15.90:0/2776063036 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f3214cc390 msgr2=0x55f32187a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.140+0000 7f461f7fe640 1 --2- 172.21.15.90:0/2776063036 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f3214cc390 0x55f32187a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.140+0000 7f461f7fe640 1 -- 172.21.15.90:0/2776063036 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f321878b70 con 0x55f3214a34d0 2024-04-25T11:44:16.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.140+0000 7f461effd640 1 --2- 172.21.15.90:0/2776063036 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3214ab6b0 0x55f321877c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:16.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.140+0000 7f461f7fe640 1 --2- 172.21.15.90:0/2776063036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214a34d0 0x55f3218841a0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x55f32186af50 tx=0x7f461000b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a4d9da19219f449 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.140+0000 7f46057fa640 1 -- 172.21.15.90:0/2776063036 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4610005220 con 0x55f3214a34d0 2024-04-25T11:44:16.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.140+0000 7f46057fa640 1 -- 172.21.15.90:0/2776063036 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4610023070 con 0x55f3214a34d0 2024-04-25T11:44:16.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.140+0000 7f46057fa640 1 -- 172.21.15.90:0/2776063036 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461001d860 con 0x55f3214a34d0 2024-04-25T11:44:16.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.141+0000 7f462d1ce740 1 -- 172.21.15.90:0/2776063036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214a34d0 msgr2=0x55f3218841a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.141+0000 7f462d1ce740 1 --2- 172.21.15.90:0/2776063036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214a34d0 0x55f3218841a0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x55f32186af50 tx=0x7f461000b410 comp rx=0 tx=0).stop 2024-04-25T11:44:16.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.141+0000 7f462d1ce740 1 -- 172.21.15.90:0/2776063036 shutdown_connections 2024-04-25T11:44:16.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.141+0000 7f462d1ce740 1 --2- 172.21.15.90:0/2776063036 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f3214cc390 0x55f32187a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.141+0000 7f462d1ce740 1 --2- 172.21.15.90:0/2776063036 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3214ab6b0 0x55f321877c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.141+0000 7f462d1ce740 1 --2- 172.21.15.90:0/2776063036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214a34d0 0x55f3218841a0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.141+0000 7f462d1ce740 1 -- 172.21.15.90:0/2776063036 >> 172.21.15.90:0/2776063036 conn(0x55f3214c04e0 msgr2=0x55f32186d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.141+0000 7f462d1ce740 1 -- 172.21.15.90:0/2776063036 shutdown_connections 2024-04-25T11:44:16.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.141+0000 7f462d1ce740 1 -- 172.21.15.90:0/2776063036 wait complete. 2024-04-25T11:44:16.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.142+0000 7f462d1ce740 1 -- start start 2024-04-25T11:44:16.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.142+0000 7f462d1ce740 1 Processor -- start 2024-04-25T11:44:16.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.142+0000 7f462d1ce740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f3214a34d0 0x55f32188a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.142+0000 7f462d1ce740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214ab6b0 0x55f32188acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.142+0000 7f462d1ce740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3214cc390 0x55f3218f05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.142+0000 7f462d1ce740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f46201b1200 con 0x55f3214ab6b0 2024-04-25T11:44:16.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.142+0000 7f462d1ce740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f46201b0b30 con 0x55f3214a34d0 2024-04-25T11:44:16.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.142+0000 7f462d1ce740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f46201add90 con 0x55f3214cc390 2024-04-25T11:44:16.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.142+0000 7f462d1ce740 10 client.0 _ll_register_callbacks cb 0x7f461000b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:16.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.142+0000 7f461effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214ab6b0 0x55f32188acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.142+0000 7f461effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214ab6b0 0x55f32188acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47364/0 (socket says 172.21.15.90:47364) 2024-04-25T11:44:16.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.142+0000 7f461effd640 1 -- 172.21.15.90:0/417575841 learned_addr learned my addr 172.21.15.90:0/417575841 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:16.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.142+0000 7f461f7fe640 1 --2- 172.21.15.90:0/417575841 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f3214a34d0 0x55f32188a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.142+0000 7f461ffff640 1 --2- 172.21.15.90:0/417575841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3214cc390 0x55f3218f05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.143+0000 7f461effd640 1 -- 172.21.15.90:0/417575841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3214cc390 msgr2=0x55f3218f05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.143+0000 7f461effd640 1 --2- 172.21.15.90:0/417575841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3214cc390 0x55f3218f05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.143+0000 7f461effd640 1 -- 172.21.15.90:0/417575841 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f3214a34d0 msgr2=0x55f32188a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.143+0000 7f461effd640 1 --2- 172.21.15.90:0/417575841 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f3214a34d0 0x55f32188a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.143+0000 7f461effd640 1 -- 172.21.15.90:0/417575841 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3218f0b20 con 0x55f3214ab6b0 2024-04-25T11:44:16.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.143+0000 7f461f7fe640 1 --2- 172.21.15.90:0/417575841 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f3214a34d0 0x55f32188a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:16.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.143+0000 7f461effd640 1 --2- 172.21.15.90:0/417575841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214ab6b0 0x55f32188acb0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f46140dd0f0 tx=0x7f46140a7540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2536ff35e67f947 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.143+0000 7f45feffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:16.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.143+0000 7f45feffd640 1 -- 172.21.15.90:0/417575841 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46140e2d70 con 0x55f3214ab6b0 2024-04-25T11:44:16.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.143+0000 7f45feffd640 1 -- 172.21.15.90:0/417575841 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7f46140a37d0 con 0x55f3214ab6b0 2024-04-25T11:44:16.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.143+0000 7f462d1ce740 1 -- 172.21.15.90:0/417575841 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4610023070 con 0x55f3214ab6b0 2024-04-25T11:44:16.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.143+0000 7f45feffd640 1 -- 172.21.15.90:0/417575841 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f46140ab8f0 con 0x55f3214ab6b0 2024-04-25T11:44:16.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.143+0000 7f462d1ce740 10 client.18450 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:16.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.143+0000 7f462d1ce740 1 -- 172.21.15.90:0/417575841 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f3218f0b20 con 0x55f3214ab6b0 2024-04-25T11:44:16.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.143+0000 7f45feffd640 1 -- 172.21.15.90:0/417575841 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4610023070 con 0x55f3214ab6b0 2024-04-25T11:44:16.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.143+0000 7f45feffd640 1 -- 172.21.15.90:0/417575841 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f46140ae0e0 con 0x55f3214ab6b0 2024-04-25T11:44:16.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.143+0000 7f462d1ce740 10 client.18450 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:16.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.143+0000 7f462d1ce740 10 client.18450 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:44:16.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.143+0000 7f462d1ce740 20 client.18450 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:16.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.143+0000 7f462d1ce740 1 --2- 172.21.15.90:0/417575841 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55f3214603a0 0x55f3218f3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.143+0000 7f462d1ce740 4 client.18450 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:44:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.143+0000 7f462d1ce740 1 -- 172.21.15.90:0/417575841 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f3217da800 con 0x55f3214603a0 2024-04-25T11:44:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.144+0000 7f461f7fe640 1 --2- 172.21.15.90:0/417575841 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55f3214603a0 0x55f3218f3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.144+0000 7f45feffd640 10 client.18450 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:44:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.144+0000 7f461f7fe640 1 --2- 172.21.15.90:0/417575841 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55f3214603a0 0x55f3218f3f90 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2972a1b923e5184a server_cookie=52e05502405a35d5 in_seq=0 out_seq=0 2024-04-25T11:44:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.144+0000 7f45feffd640 1 -- 172.21.15.90:0/417575841 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f3217da800 con 0x55f3214603a0 2024-04-25T11:44:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.144+0000 7f45feffd640 10 client.18450 handle_command_reply: tid=0 2024-04-25T11:44:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.144+0000 7f462d1ce740 1 client.18450 shutdown 2024-04-25T11:44:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.144+0000 7f462d1ce740 1 -- 172.21.15.90:0/417575841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214ab6b0 msgr2=0x55f32188acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.144+0000 7f462d1ce740 1 --2- 172.21.15.90:0/417575841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214ab6b0 0x55f32188acb0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f46140dd0f0 tx=0x7f46140a7540 comp rx=0 tx=0).stop 2024-04-25T11:44:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.145+0000 7f462d1ce740 1 -- 172.21.15.90:0/417575841 shutdown_connections 2024-04-25T11:44:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.145+0000 7f462d1ce740 1 --2- 172.21.15.90:0/417575841 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55f3214603a0 0x55f3218f3f90 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.145+0000 7f462d1ce740 1 --2- 172.21.15.90:0/417575841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f3214cc390 0x55f3218f05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.145+0000 7f462d1ce740 1 --2- 172.21.15.90:0/417575841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3214ab6b0 0x55f32188acb0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.145+0000 7f45feffd640 0 client.18450 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:44:16.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.145+0000 7f462d1ce740 1 --2- 172.21.15.90:0/417575841 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f3214a34d0 0x55f32188a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.145+0000 7f462d1ce740 1 -- 172.21.15.90:0/417575841 >> 172.21.15.90:0/417575841 conn(0x55f3214c04e0 msgr2=0x55f32186f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.145+0000 7f462d1ce740 1 -- 172.21.15.90:0/417575841 shutdown_connections 2024-04-25T11:44:16.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.145+0000 7f462d1ce740 1 -- 172.21.15.90:0/417575841 wait complete. 2024-04-25T11:44:16.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.145+0000 7f462d1ce740 20 client.18450 trim_cache size 0 max 16384 2024-04-25T11:44:16.137 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:44:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.145+0000 7f4626242640 1 -- 172.21.15.90:0/945730744 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f46080afce0 msgr2=0x7f46080b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.145+0000 7f4626242640 1 --2- 172.21.15.90:0/945730744 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f46080afce0 0x7f46080b21f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x55f32186a980 tx=0x7f46180095c0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.145+0000 7f4626242640 1 -- 172.21.15.90:0/945730744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4620106990 msgr2=0x7f462019ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.145+0000 7f4626242640 1 --2- 172.21.15.90:0/945730744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4620106990 0x7f462019ad10 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f461400e9f0 tx=0x7f461400eeb0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.145+0000 7f4626242640 1 -- 172.21.15.90:0/945730744 shutdown_connections 2024-04-25T11:44:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.145+0000 7f4626242640 1 --2- 172.21.15.90:0/945730744 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f46080afce0 0x7f46080b21f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.145+0000 7f4626242640 1 --2- 172.21.15.90:0/945730744 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4620107450 0x7f462019efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.145+0000 7f4626242640 1 --2- 172.21.15.90:0/945730744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4620106990 0x7f462019ad10 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.145+0000 7f4626242640 1 --2- 172.21.15.90:0/945730744 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4620102400 0x7f462019a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.145+0000 7f4626242640 1 -- 172.21.15.90:0/945730744 >> 172.21.15.90:0/945730744 conn(0x7f4620077880 msgr2=0x7f46200ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.146+0000 7f4626242640 1 -- 172.21.15.90:0/945730744 shutdown_connections 2024-04-25T11:44:16.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.146+0000 7f4626242640 1 -- 172.21.15.90:0/945730744 wait complete. 2024-04-25T11:44:16.143 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T11:44:16.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.271+0000 7f3710092640 1 -- 172.21.15.90:0/1277122118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f370810e080 msgr2=0x7f3708110570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.271+0000 7f3710092640 1 --2- 172.21.15.90:0/1277122118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f370810e080 0x7f3708110570 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f3700009ca0 tx=0x7f370002a8c0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.271+0000 7f3710092640 1 -- 172.21.15.90:0/1277122118 shutdown_connections 2024-04-25T11:44:16.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.271+0000 7f3710092640 1 --2- 172.21.15.90:0/1277122118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f370810e080 0x7f3708110570 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.271+0000 7f3710092640 1 --2- 172.21.15.90:0/1277122118 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f370810b690 0x7f370810dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.271+0000 7f3710092640 1 --2- 172.21.15.90:0/1277122118 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3708104c30 0x7f3708105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.271+0000 7f3710092640 1 -- 172.21.15.90:0/1277122118 >> 172.21.15.90:0/1277122118 conn(0x7f3708078110 msgr2=0x7f3708078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.271+0000 7f3710092640 1 -- 172.21.15.90:0/1277122118 shutdown_connections 2024-04-25T11:44:16.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.271+0000 7f3710092640 1 -- 172.21.15.90:0/1277122118 wait complete. 2024-04-25T11:44:16.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.272+0000 7f3710092640 1 Processor -- start 2024-04-25T11:44:16.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.272+0000 7f3710092640 1 -- start start 2024-04-25T11:44:16.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.272+0000 7f3710092640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3708104c30 0x7f370819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.272+0000 7f3710092640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f370810b690 0x7f370819aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.272+0000 7f3710092640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f370810e080 0x7f370819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.272+0000 7f3710092640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3708113ce0 con 0x7f370810e080 2024-04-25T11:44:16.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.272+0000 7f3710092640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3708113b60 con 0x7f370810b690 2024-04-25T11:44:16.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.272+0000 7f3710092640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3708113e60 con 0x7f3708104c30 2024-04-25T11:44:16.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.272+0000 7f370e62f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f370810e080 0x7f370819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.272+0000 7f370e62f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f370810e080 0x7f370819ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47380/0 (socket says 172.21.15.90:47380) 2024-04-25T11:44:16.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.272+0000 7f370e62f640 1 -- 172.21.15.90:0/4075365667 learned_addr learned my addr 172.21.15.90:0/4075365667 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:16.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.272+0000 7f370d62d640 1 --2- 172.21.15.90:0/4075365667 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f370810b690 0x7f370819aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.272+0000 7f370de2e640 1 --2- 172.21.15.90:0/4075365667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3708104c30 0x7f370819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.272+0000 7f370e62f640 1 -- 172.21.15.90:0/4075365667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3708104c30 msgr2=0x7f370819a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.272+0000 7f370e62f640 1 --2- 172.21.15.90:0/4075365667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3708104c30 0x7f370819a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.272+0000 7f370e62f640 1 -- 172.21.15.90:0/4075365667 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f370810b690 msgr2=0x7f370819aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.272+0000 7f370e62f640 1 --2- 172.21.15.90:0/4075365667 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f370810b690 0x7f370819aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.272+0000 7f370e62f640 1 -- 172.21.15.90:0/4075365667 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f370819f5a0 con 0x7f370810e080 2024-04-25T11:44:16.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.272+0000 7f370e62f640 1 --2- 172.21.15.90:0/4075365667 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f370810e080 0x7f370819ef60 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f3700005cc0 tx=0x7f3700031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.273+0000 7f36feffd640 1 -- 172.21.15.90:0/4075365667 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3700003b70 con 0x7f370810e080 2024-04-25T11:44:16.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.273+0000 7f3710092640 1 -- 172.21.15.90:0/4075365667 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f370819f830 con 0x7f370810e080 2024-04-25T11:44:16.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.273+0000 7f36feffd640 1 -- 172.21.15.90:0/4075365667 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37000318f0 con 0x7f370810e080 2024-04-25T11:44:16.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.273+0000 7f36feffd640 1 -- 172.21.15.90:0/4075365667 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370002f950 con 0x7f370810e080 2024-04-25T11:44:16.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.273+0000 7f3710092640 1 -- 172.21.15.90:0/4075365667 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f370819fd80 con 0x7f370810e080 2024-04-25T11:44:16.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.275+0000 7f36feffd640 1 -- 172.21.15.90:0/4075365667 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f370002fb50 con 0x7f370810e080 2024-04-25T11:44:16.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.275+0000 7f3717019740 1 Processor -- start 2024-04-25T11:44:16.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.276+0000 7f3717019740 1 -- start start 2024-04-25T11:44:16.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.276+0000 7f3717019740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe47610390 0x55fe479b1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.276+0000 7f36feffd640 1 --2- 172.21.15.90:0/4075365667 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f36f40afc60 0x7f36f40b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.276+0000 7f3717019740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe475e74d0 0x55fe479b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.276+0000 7f3717019740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475ef6b0 0x55fe479be6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.276+0000 7f3717019740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55fe47916d80 con 0x55fe475ef6b0 2024-04-25T11:44:16.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.276+0000 7f3717019740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55fe478d0a20 con 0x55fe475e74d0 2024-04-25T11:44:16.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.276+0000 7f3717019740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fe47912d40 con 0x55fe47610390 2024-04-25T11:44:16.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.276+0000 7f370e62f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475ef6b0 0x55fe479be6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.276+0000 7f370e62f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475ef6b0 0x55fe479be6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47392/0 (socket says 172.21.15.90:47392) 2024-04-25T11:44:16.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.276+0000 7f370e62f640 1 -- 172.21.15.90:0/1078472464 learned_addr learned my addr 172.21.15.90:0/1078472464 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:16.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.276+0000 7f370de2e640 1 --2- 172.21.15.90:0/4075365667 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f36f40afc60 0x7f36f40b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.276+0000 7f36feffd640 1 -- 172.21.15.90:0/4075365667 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7f37001003c0 con 0x7f370810e080 2024-04-25T11:44:16.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.276+0000 7f370d62d640 1 --2- 172.21.15.90:0/1078472464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe47610390 0x55fe479b1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.276+0000 7f370d62d640 1 --2- 172.21.15.90:0/1078472464 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe475e74d0 0x55fe479b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.276+0000 7f370e62f640 1 -- 172.21.15.90:0/1078472464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe47610390 msgr2=0x55fe479b1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.276+0000 7f370e62f640 1 --2- 172.21.15.90:0/1078472464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe47610390 0x55fe479b1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.276+0000 7f370e62f640 1 -- 172.21.15.90:0/1078472464 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe475e74d0 msgr2=0x55fe479b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.276+0000 7f370e62f640 1 --2- 172.21.15.90:0/1078472464 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe475e74d0 0x55fe479b1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.276+0000 7f370e62f640 1 -- 172.21.15.90:0/1078472464 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe478dbc30 con 0x55fe475ef6b0 2024-04-25T11:44:16.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.277+0000 7f370d62d640 1 --2- 172.21.15.90:0/1078472464 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe475e74d0 0x55fe479b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:44:16.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.277+0000 7f370de2e640 1 --2- 172.21.15.90:0/4075365667 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f36f40afc60 0x7f36f40b2170 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3704009b60 tx=0x7f3704005f90 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.277+0000 7f370e62f640 1 --2- 172.21.15.90:0/1078472464 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475ef6b0 0x55fe479be6a0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f37000c5830 tx=0x7f37000047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f40289e4554f95fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.277+0000 7f36f3fff640 1 -- 172.21.15.90:0/1078472464 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37000c7d30 con 0x55fe475ef6b0 2024-04-25T11:44:16.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.277+0000 7f36f3fff640 1 -- 172.21.15.90:0/1078472464 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3700036070 con 0x55fe475ef6b0 2024-04-25T11:44:16.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.277+0000 7f36f3fff640 1 -- 172.21.15.90:0/1078472464 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37000e2eb0 con 0x55fe475ef6b0 2024-04-25T11:44:16.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.278+0000 7f3717019740 1 -- 172.21.15.90:0/1078472464 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475ef6b0 msgr2=0x55fe479be6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.278+0000 7f3717019740 1 --2- 172.21.15.90:0/1078472464 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475ef6b0 0x55fe479be6a0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f37000c5830 tx=0x7f37000047b0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.278+0000 7f3717019740 1 -- 172.21.15.90:0/1078472464 shutdown_connections 2024-04-25T11:44:16.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.278+0000 7f3717019740 1 --2- 172.21.15.90:0/1078472464 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475ef6b0 0x55fe479be6a0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.278+0000 7f3717019740 1 --2- 172.21.15.90:0/1078472464 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe475e74d0 0x55fe479b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.278+0000 7f3717019740 1 --2- 172.21.15.90:0/1078472464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe47610390 0x55fe479b1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.278+0000 7f3717019740 1 -- 172.21.15.90:0/1078472464 >> 172.21.15.90:0/1078472464 conn(0x55fe476044e0 msgr2=0x55fe479adfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.278+0000 7f3717019740 1 -- 172.21.15.90:0/1078472464 shutdown_connections 2024-04-25T11:44:16.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.278+0000 7f3717019740 1 -- 172.21.15.90:0/1078472464 wait complete. 2024-04-25T11:44:16.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.278+0000 7f3717019740 1 -- start start 2024-04-25T11:44:16.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.278+0000 7f3717019740 1 Processor -- start 2024-04-25T11:44:16.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.279+0000 7f3717019740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe475e74d0 0x55fe479bc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.279+0000 7f3717019740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475ef6b0 0x55fe479bc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.279+0000 7f3717019740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe47610390 0x55fe479c7e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.279+0000 7f3717019740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55fe47916d80 con 0x55fe475ef6b0 2024-04-25T11:44:16.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.279+0000 7f3717019740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55fe4789ddc0 con 0x55fe475e74d0 2024-04-25T11:44:16.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.279+0000 7f3717019740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fe479bebe0 con 0x55fe47610390 2024-04-25T11:44:16.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.279+0000 7f3717019740 10 client.0 _ll_register_callbacks cb 0x55fe478b9600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:16.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.279+0000 7f370d62d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475ef6b0 0x55fe479bc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.279+0000 7f370d62d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475ef6b0 0x55fe479bc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47394/0 (socket says 172.21.15.90:47394) 2024-04-25T11:44:16.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.279+0000 7f370d62d640 1 -- 172.21.15.90:0/178537821 learned_addr learned my addr 172.21.15.90:0/178537821 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:16.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.279+0000 7f370de2e640 1 --2- 172.21.15.90:0/178537821 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe475e74d0 0x55fe479bc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.279+0000 7f370e62f640 1 --2- 172.21.15.90:0/178537821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe47610390 0x55fe479c7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.279+0000 7f370de2e640 1 -- 172.21.15.90:0/178537821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe47610390 msgr2=0x55fe479c7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.279+0000 7f370de2e640 1 --2- 172.21.15.90:0/178537821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe47610390 0x55fe479c7e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.279+0000 7f370de2e640 1 -- 172.21.15.90:0/178537821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475ef6b0 msgr2=0x55fe479bc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.279+0000 7f370de2e640 1 --2- 172.21.15.90:0/178537821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475ef6b0 0x55fe479bc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.279+0000 7f370de2e640 1 -- 172.21.15.90:0/178537821 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe476283f0 con 0x55fe475e74d0 2024-04-25T11:44:16.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.279+0000 7f370e62f640 1 --2- 172.21.15.90:0/178537821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe47610390 0x55fe479c7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:44:16.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.279+0000 7f370d62d640 1 --2- 172.21.15.90:0/178537821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475ef6b0 0x55fe479bc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:16.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.279+0000 7f370de2e640 1 --2- 172.21.15.90:0/178537821 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe475e74d0 0x55fe479bc260 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3704011870 tx=0x7f3704004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b04e7a351f3fd21 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.280+0000 7f36f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:44:16.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.280+0000 7f36f17fa640 1 -- 172.21.15.90:0/178537821 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370400fce0 con 0x55fe475e74d0 2024-04-25T11:44:16.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.280+0000 7f36f17fa640 1 -- 172.21.15.90:0/178537821 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7f370401f420 con 0x55fe475e74d0 2024-04-25T11:44:16.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.280+0000 7f3717019740 1 -- 172.21.15.90:0/178537821 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3700036070 con 0x55fe475e74d0 2024-04-25T11:44:16.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.280+0000 7f36f17fa640 1 -- 172.21.15.90:0/178537821 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f370400fe80 con 0x55fe475e74d0 2024-04-25T11:44:16.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.280+0000 7f3717019740 10 client.27224 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:16.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.280+0000 7f3717019740 1 -- 172.21.15.90:0/178537821 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fe476f27d0 con 0x55fe475e74d0 2024-04-25T11:44:16.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.280+0000 7f36f17fa640 1 -- 172.21.15.90:0/178537821 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3700036070 con 0x55fe475e74d0 2024-04-25T11:44:16.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.280+0000 7f36f17fa640 1 -- 172.21.15.90:0/178537821 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f370402e6e0 con 0x55fe475e74d0 2024-04-25T11:44:16.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.280+0000 7f3717019740 10 client.27224 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:16.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.280+0000 7f3717019740 10 client.27224 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:44:16.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.280+0000 7f3717019740 20 client.27224 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:16.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.280+0000 7f3717019740 1 --2- 172.21.15.90:0/178537821 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55fe475cef60 0x55fe479ce400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.280+0000 7f3717019740 4 client.27224 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:44:16.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.280+0000 7f3717019740 1 -- 172.21.15.90:0/178537821 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe47924990 con 0x55fe475cef60 2024-04-25T11:44:16.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.281+0000 7f370d62d640 1 --2- 172.21.15.90:0/178537821 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55fe475cef60 0x55fe479ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.281+0000 7f36f17fa640 10 client.27224 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:44:16.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.281+0000 7f370d62d640 1 --2- 172.21.15.90:0/178537821 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55fe475cef60 0x55fe479ce400 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ccf9f1b526f2e010 server_cookie=6aaf2ab67983091 in_seq=0 out_seq=0 2024-04-25T11:44:16.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.282+0000 7f36f17fa640 1 -- 172.21.15.90:0/178537821 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55fe47924990 con 0x55fe475cef60 2024-04-25T11:44:16.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.282+0000 7f36f17fa640 10 client.27224 handle_command_reply: tid=0 2024-04-25T11:44:16.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.282+0000 7f3717019740 1 client.27224 shutdown 2024-04-25T11:44:16.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.282+0000 7f3717019740 1 -- 172.21.15.90:0/178537821 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe475e74d0 msgr2=0x55fe479bc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.282+0000 7f3717019740 1 --2- 172.21.15.90:0/178537821 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe475e74d0 0x55fe479bc260 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3704011870 tx=0x7f3704004730 comp rx=0 tx=0).stop 2024-04-25T11:44:16.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.282+0000 7f3717019740 1 -- 172.21.15.90:0/178537821 shutdown_connections 2024-04-25T11:44:16.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.282+0000 7f3717019740 1 --2- 172.21.15.90:0/178537821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe47610390 0x55fe479c7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.282+0000 7f3717019740 1 --2- 172.21.15.90:0/178537821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475ef6b0 0x55fe479bc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.282+0000 7f3717019740 1 --2- 172.21.15.90:0/178537821 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55fe475cef60 0x55fe479ce400 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.282+0000 7f3717019740 1 --2- 172.21.15.90:0/178537821 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe475e74d0 0x55fe479bc260 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.282+0000 7f3717019740 1 -- 172.21.15.90:0/178537821 >> 172.21.15.90:0/178537821 conn(0x55fe476044e0 msgr2=0x55fe479b3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.282+0000 7f36f17fa640 0 client.27224 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:44:16.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.282+0000 7f3717019740 1 -- 172.21.15.90:0/178537821 shutdown_connections 2024-04-25T11:44:16.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.282+0000 7f3717019740 1 -- 172.21.15.90:0/178537821 wait complete. 2024-04-25T11:44:16.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.283+0000 7f3717019740 20 client.27224 trim_cache size 0 max 16384 2024-04-25T11:44:16.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.298+0000 7f3717019740 1 Processor -- start 2024-04-25T11:44:16.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.298+0000 7f3717019740 1 -- start start 2024-04-25T11:44:16.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.298+0000 7f3717019740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475e74d0 0x55fe479bb350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.298+0000 7f3717019740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe475ef6b0 0x55fe479c7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.298+0000 7f3717019740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe47610390 0x55fe479b6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.298+0000 7f3717019740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f37081afc60 con 0x55fe475e74d0 2024-04-25T11:44:16.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.298+0000 7f3717019740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f37081b17a0 con 0x55fe475ef6b0 2024-04-25T11:44:16.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.298+0000 7f3717019740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f37081b0330 con 0x55fe47610390 2024-04-25T11:44:16.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.298+0000 7f370de2e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475e74d0 0x55fe479bb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.298+0000 7f370de2e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475e74d0 0x55fe479bb350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47406/0 (socket says 172.21.15.90:47406) 2024-04-25T11:44:16.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.298+0000 7f370de2e640 1 -- 172.21.15.90:0/2158433344 learned_addr learned my addr 172.21.15.90:0/2158433344 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:16.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.298+0000 7f370d62d640 1 --2- 172.21.15.90:0/2158433344 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe475ef6b0 0x55fe479c7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.298+0000 7f370e62f640 1 --2- 172.21.15.90:0/2158433344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe47610390 0x55fe479b6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.299+0000 7f370de2e640 1 -- 172.21.15.90:0/2158433344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe47610390 msgr2=0x55fe479b6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.299+0000 7f370de2e640 1 --2- 172.21.15.90:0/2158433344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe47610390 0x55fe479b6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.299+0000 7f370de2e640 1 -- 172.21.15.90:0/2158433344 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe475ef6b0 msgr2=0x55fe479c7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.299+0000 7f370de2e640 1 --2- 172.21.15.90:0/2158433344 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe475ef6b0 0x55fe479c7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.299+0000 7f370de2e640 1 -- 172.21.15.90:0/2158433344 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe479b2d90 con 0x55fe475e74d0 2024-04-25T11:44:16.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.299+0000 7f370de2e640 1 --2- 172.21.15.90:0/2158433344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475e74d0 0x55fe479bb350 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f3704004700 tx=0x7f3704012f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecd9252e14cf0e61 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.299+0000 7f36f37fe640 1 -- 172.21.15.90:0/2158433344 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370402ede0 con 0x55fe475e74d0 2024-04-25T11:44:16.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.299+0000 7f36f37fe640 1 -- 172.21.15.90:0/2158433344 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f370402e8a0 con 0x55fe475e74d0 2024-04-25T11:44:16.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.299+0000 7f36f37fe640 1 -- 172.21.15.90:0/2158433344 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3704037c20 con 0x55fe475e74d0 2024-04-25T11:44:16.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.300+0000 7f3717019740 1 -- 172.21.15.90:0/2158433344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475e74d0 msgr2=0x55fe479bb350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.300+0000 7f3717019740 1 --2- 172.21.15.90:0/2158433344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475e74d0 0x55fe479bb350 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f3704004700 tx=0x7f3704012f20 comp rx=0 tx=0).stop 2024-04-25T11:44:16.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.300+0000 7f3717019740 1 -- 172.21.15.90:0/2158433344 shutdown_connections 2024-04-25T11:44:16.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.300+0000 7f3717019740 1 --2- 172.21.15.90:0/2158433344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe47610390 0x55fe479b6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.300+0000 7f3717019740 1 --2- 172.21.15.90:0/2158433344 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe475ef6b0 0x55fe479c7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.300+0000 7f3717019740 1 --2- 172.21.15.90:0/2158433344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475e74d0 0x55fe479bb350 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.300+0000 7f3717019740 1 -- 172.21.15.90:0/2158433344 >> 172.21.15.90:0/2158433344 conn(0x55fe476044e0 msgr2=0x55fe479ac610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.300+0000 7f3717019740 1 -- 172.21.15.90:0/2158433344 shutdown_connections 2024-04-25T11:44:16.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.300+0000 7f3717019740 1 -- 172.21.15.90:0/2158433344 wait complete. 2024-04-25T11:44:16.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.300+0000 7f3717019740 1 -- start start 2024-04-25T11:44:16.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.300+0000 7f3717019740 1 Processor -- start 2024-04-25T11:44:16.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.301+0000 7f3717019740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe475e74d0 0x55fe479bedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.301+0000 7f3717019740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475ef6b0 0x55fe479bf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.301+0000 7f3717019740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe47610390 0x55fe47a34760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.301+0000 7f3717019740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f37081b10d0 con 0x55fe475ef6b0 2024-04-25T11:44:16.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.301+0000 7f3717019740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f37081b0a00 con 0x55fe47610390 2024-04-25T11:44:16.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.301+0000 7f3717019740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f37081adc60 con 0x55fe475e74d0 2024-04-25T11:44:16.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.301+0000 7f3717019740 10 client.0 _ll_register_callbacks cb 0x7f3704009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:16.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.301+0000 7f370d62d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475ef6b0 0x55fe479bf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.301+0000 7f370de2e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe475e74d0 0x55fe479bedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.301+0000 7f370de2e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe475e74d0 0x55fe479bedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:54120/0 (socket says 172.21.15.90:54120) 2024-04-25T11:44:16.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.301+0000 7f370d62d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475ef6b0 0x55fe479bf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47412/0 (socket says 172.21.15.90:47412) 2024-04-25T11:44:16.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.301+0000 7f370de2e640 1 -- 172.21.15.90:0/1709377785 learned_addr learned my addr 172.21.15.90:0/1709377785 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:16.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.301+0000 7f370e62f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe47610390 0x55fe47a34760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.301+0000 7f370de2e640 1 -- 172.21.15.90:0/1709377785 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe47610390 msgr2=0x55fe47a34760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.301+0000 7f370de2e640 1 --2- 172.21.15.90:0/1709377785 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe47610390 0x55fe47a34760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.301+0000 7f370de2e640 1 -- 172.21.15.90:0/1709377785 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475ef6b0 msgr2=0x55fe479bf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.301+0000 7f370de2e640 1 --2- 172.21.15.90:0/1709377785 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475ef6b0 0x55fe479bf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.301+0000 7f370de2e640 1 -- 172.21.15.90:0/1709377785 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe479bfb80 con 0x55fe475e74d0 2024-04-25T11:44:16.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.301+0000 7f370d62d640 1 --2- 172.21.15.90:0/1709377785 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475ef6b0 0x55fe479bf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:16.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.301+0000 7f370de2e640 1 --2- 172.21.15.90:0/1709377785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe475e74d0 0x55fe479bedf0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3704011a90 tx=0x7f3704037840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28641a5fe66f6c7b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.301+0000 7f36f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:44:16.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.302+0000 7f36f17fa640 1 -- 172.21.15.90:0/1709377785 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370400fae0 con 0x55fe475e74d0 2024-04-25T11:44:16.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.302+0000 7f36f17fa640 1 -- 172.21.15.90:0/1709377785 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7f370400fc80 con 0x55fe475e74d0 2024-04-25T11:44:16.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.302+0000 7f3717019740 1 -- 172.21.15.90:0/1709377785 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fe4791e800 con 0x55fe475e74d0 2024-04-25T11:44:16.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.302+0000 7f36f17fa640 1 -- 172.21.15.90:0/1709377785 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f370404d050 con 0x55fe475e74d0 2024-04-25T11:44:16.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.302+0000 7f3717019740 10 client.27103 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:16.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.302+0000 7f3717019740 1 -- 172.21.15.90:0/1709377785 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fe47a35c80 con 0x55fe475e74d0 2024-04-25T11:44:16.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.302+0000 7f36f17fa640 1 -- 172.21.15.90:0/1709377785 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55fe4791e800 con 0x55fe475e74d0 2024-04-25T11:44:16.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.302+0000 7f36f17fa640 1 -- 172.21.15.90:0/1709377785 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f3704004730 con 0x55fe475e74d0 2024-04-25T11:44:16.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.302+0000 7f3717019740 10 client.27103 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:16.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.302+0000 7f3717019740 10 client.27103 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:44:16.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.302+0000 7f3717019740 20 client.27103 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:16.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.302+0000 7f3717019740 1 --2- 172.21.15.90:0/1709377785 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55fe475a43a0 0x55fe47a38340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.302+0000 7f3717019740 4 client.27103 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:44:16.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.302+0000 7f3717019740 1 -- 172.21.15.90:0/1709377785 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fe47899610 con 0x55fe475a43a0 2024-04-25T11:44:16.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.302+0000 7f370d62d640 1 --2- 172.21.15.90:0/1709377785 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55fe475a43a0 0x55fe47a38340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.303+0000 7f370d62d640 1 --2- 172.21.15.90:0/1709377785 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55fe475a43a0 0x55fe47a38340 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c396c641b10c2c8c server_cookie=bff10b2f54695c4e in_seq=0 out_seq=0 2024-04-25T11:44:16.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.303+0000 7f36f17fa640 10 client.27103 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:44:16.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.303+0000 7f36f17fa640 1 -- 172.21.15.90:0/1709377785 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fe47899610 con 0x55fe475a43a0 2024-04-25T11:44:16.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.303+0000 7f36f17fa640 10 client.27103 handle_command_reply: tid=0 2024-04-25T11:44:16.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.303+0000 7f3717019740 1 client.27103 shutdown 2024-04-25T11:44:16.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.303+0000 7f3717019740 1 -- 172.21.15.90:0/1709377785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe475e74d0 msgr2=0x55fe479bedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.303+0000 7f3717019740 1 --2- 172.21.15.90:0/1709377785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe475e74d0 0x55fe479bedf0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3704011a90 tx=0x7f3704037840 comp rx=0 tx=0).stop 2024-04-25T11:44:16.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.303+0000 7f3717019740 1 -- 172.21.15.90:0/1709377785 shutdown_connections 2024-04-25T11:44:16.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.303+0000 7f3717019740 1 --2- 172.21.15.90:0/1709377785 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55fe475a43a0 0x55fe47a38340 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.303+0000 7f3717019740 1 --2- 172.21.15.90:0/1709377785 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fe47610390 0x55fe47a34760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.303+0000 7f36f17fa640 0 client.27103 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:44:16.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.303+0000 7f3717019740 1 --2- 172.21.15.90:0/1709377785 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe475ef6b0 0x55fe479bf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.303+0000 7f3717019740 1 --2- 172.21.15.90:0/1709377785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fe475e74d0 0x55fe479bedf0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.303+0000 7f3717019740 1 -- 172.21.15.90:0/1709377785 >> 172.21.15.90:0/1709377785 conn(0x55fe476044e0 msgr2=0x55fe479bb0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.303+0000 7f3717019740 1 -- 172.21.15.90:0/1709377785 shutdown_connections 2024-04-25T11:44:16.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.303+0000 7f3717019740 1 -- 172.21.15.90:0/1709377785 wait complete. 2024-04-25T11:44:16.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.303+0000 7f3717019740 20 client.27103 trim_cache size 0 max 16384 2024-04-25T11:44:16.296 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:44:16.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.304+0000 7f3710092640 1 -- 172.21.15.90:0/4075365667 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f36f40afc60 msgr2=0x7f36f40b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.304+0000 7f3710092640 1 --2- 172.21.15.90:0/4075365667 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f36f40afc60 0x7f36f40b2170 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3704009b60 tx=0x7f3704005f90 comp rx=0 tx=0).stop 2024-04-25T11:44:16.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.304+0000 7f3710092640 1 -- 172.21.15.90:0/4075365667 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f370810e080 msgr2=0x7f370819ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.304+0000 7f3710092640 1 --2- 172.21.15.90:0/4075365667 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f370810e080 0x7f370819ef60 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f3700005cc0 tx=0x7f3700031700 comp rx=0 tx=0).stop 2024-04-25T11:44:16.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.304+0000 7f3710092640 1 -- 172.21.15.90:0/4075365667 shutdown_connections 2024-04-25T11:44:16.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.304+0000 7f3710092640 1 --2- 172.21.15.90:0/4075365667 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f36f40afc60 0x7f36f40b2170 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.304+0000 7f3710092640 1 --2- 172.21.15.90:0/4075365667 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f370810e080 0x7f370819ef60 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.304+0000 7f3710092640 1 --2- 172.21.15.90:0/4075365667 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f370810b690 0x7f370819aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.304+0000 7f3710092640 1 --2- 172.21.15.90:0/4075365667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3708104c30 0x7f370819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.304+0000 7f3710092640 1 -- 172.21.15.90:0/4075365667 >> 172.21.15.90:0/4075365667 conn(0x7f3708078110 msgr2=0x7f3708101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.304+0000 7f3710092640 1 -- 172.21.15.90:0/4075365667 shutdown_connections 2024-04-25T11:44:16.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.304+0000 7f3710092640 1 -- 172.21.15.90:0/4075365667 wait complete. 2024-04-25T11:44:16.301 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T11:44:16.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.431+0000 7f339deb8640 1 -- 172.21.15.90:0/3761532052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3398102400 msgr2=0x7f3398102810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.431+0000 7f339deb8640 1 --2- 172.21.15.90:0/3761532052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3398102400 0x7f3398102810 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f338800aa10 tx=0x7f338802b720 comp rx=0 tx=0).stop 2024-04-25T11:44:16.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.431+0000 7f339deb8640 1 -- 172.21.15.90:0/3761532052 shutdown_connections 2024-04-25T11:44:16.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.431+0000 7f339deb8640 1 --2- 172.21.15.90:0/3761532052 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3398107450 0x7f339810ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.431+0000 7f339deb8640 1 --2- 172.21.15.90:0/3761532052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3398106990 0x7f3398106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.431+0000 7f339deb8640 1 --2- 172.21.15.90:0/3761532052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3398102400 0x7f3398102810 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.431+0000 7f339deb8640 1 -- 172.21.15.90:0/3761532052 >> 172.21.15.90:0/3761532052 conn(0x7f3398077880 msgr2=0x7f3398077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.431+0000 7f339deb8640 1 -- 172.21.15.90:0/3761532052 shutdown_connections 2024-04-25T11:44:16.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.431+0000 7f339deb8640 1 -- 172.21.15.90:0/3761532052 wait complete. 2024-04-25T11:44:16.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.432+0000 7f339deb8640 1 Processor -- start 2024-04-25T11:44:16.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.432+0000 7f339deb8640 1 -- start start 2024-04-25T11:44:16.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.433+0000 7f339deb8640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3398102400 0x7f339819a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.433+0000 7f339deb8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3398106990 0x7f339819ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.433+0000 7f339deb8640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3398107450 0x7f339819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.433+0000 7f339deb8640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3398111b30 con 0x7f3398107450 2024-04-25T11:44:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.433+0000 7f339deb8640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f33981119b0 con 0x7f3398102400 2024-04-25T11:44:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.433+0000 7f339deb8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3398111cb0 con 0x7f3398106990 2024-04-25T11:44:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.433+0000 7f3397fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3398107450 0x7f339819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.433+0000 7f3397fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3398107450 0x7f339819efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47440/0 (socket says 172.21.15.90:47440) 2024-04-25T11:44:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.433+0000 7f3397fff640 1 -- 172.21.15.90:0/2981053511 learned_addr learned my addr 172.21.15.90:0/2981053511 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.433+0000 7f33977fe640 1 --2- 172.21.15.90:0/2981053511 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3398102400 0x7f339819a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.433+0000 7f3396ffd640 1 --2- 172.21.15.90:0/2981053511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3398106990 0x7f339819ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.433+0000 7f3397fff640 1 -- 172.21.15.90:0/2981053511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3398106990 msgr2=0x7f339819ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.433+0000 7f3397fff640 1 --2- 172.21.15.90:0/2981053511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3398106990 0x7f339819ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.433+0000 7f3397fff640 1 -- 172.21.15.90:0/2981053511 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3398102400 msgr2=0x7f339819a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.433+0000 7f3397fff640 1 --2- 172.21.15.90:0/2981053511 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3398102400 0x7f339819a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.433+0000 7f3397fff640 1 -- 172.21.15.90:0/2981053511 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f339819f600 con 0x7f3398107450 2024-04-25T11:44:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.433+0000 7f3397fff640 1 --2- 172.21.15.90:0/2981053511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3398107450 0x7f339819efc0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f339000eaa0 tx=0x7f339000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.433+0000 7f3394ff9640 1 -- 172.21.15.90:0/2981053511 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339000cdb0 con 0x7f3398107450 2024-04-25T11:44:16.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.433+0000 7f339deb8640 1 -- 172.21.15.90:0/2981053511 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f339819f8f0 con 0x7f3398107450 2024-04-25T11:44:16.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.433+0000 7f3394ff9640 1 -- 172.21.15.90:0/2981053511 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3390005020 con 0x7f3398107450 2024-04-25T11:44:16.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.434+0000 7f3394ff9640 1 -- 172.21.15.90:0/2981053511 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33900116d0 con 0x7f3398107450 2024-04-25T11:44:16.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.434+0000 7f339deb8640 1 -- 172.21.15.90:0/2981053511 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33981982c0 con 0x7f3398107450 2024-04-25T11:44:16.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.437+0000 7f3394ff9640 1 -- 172.21.15.90:0/2981053511 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f33900051d0 con 0x7f3398107450 2024-04-25T11:44:16.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.437+0000 7f33a4d68740 1 Processor -- start 2024-04-25T11:44:16.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.437+0000 7f33a4d68740 1 -- start start 2024-04-25T11:44:16.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.437+0000 7f33a4d68740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca8804b390 0x55ca883ec810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.437+0000 7f33a4d68740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ca880224d0 0x55ca883ecd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.437+0000 7f33a4d68740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca8802a6b0 0x55ca883f96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.437+0000 7f33a4d68740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ca88351d80 con 0x55ca8802a6b0 2024-04-25T11:44:16.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.437+0000 7f33a4d68740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ca8830ba20 con 0x55ca880224d0 2024-04-25T11:44:16.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.437+0000 7f33a4d68740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ca8834dd40 con 0x55ca8804b390 2024-04-25T11:44:16.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.437+0000 7f3394ff9640 1 --2- 172.21.15.90:0/2981053511 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f33800afc60 0x7f33800b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.437+0000 7f33977fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ca880224d0 0x55ca883ecd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.437+0000 7f33977fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ca880224d0 0x55ca883ecd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:56968/0 (socket says 172.21.15.90:56968) 2024-04-25T11:44:16.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.437+0000 7f33977fe640 1 -- 172.21.15.90:0/3153738154 learned_addr learned my addr 172.21.15.90:0/3153738154 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:16.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.437+0000 7f3396ffd640 1 --2- 172.21.15.90:0/3153738154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca8802a6b0 0x55ca883f96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.438+0000 7f3397fff640 1 --2- 172.21.15.90:0/2981053511 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f33800afc60 0x7f33800b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.438+0000 7f3396ffd640 1 --2- 172.21.15.90:0/3153738154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca8804b390 0x55ca883ec810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.438+0000 7f3394ff9640 1 -- 172.21.15.90:0/2981053511 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7f33900df2b0 con 0x7f3398107450 2024-04-25T11:44:16.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.438+0000 7f33977fe640 1 -- 172.21.15.90:0/3153738154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca8804b390 msgr2=0x55ca883ec810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.438+0000 7f33977fe640 1 --2- 172.21.15.90:0/3153738154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca8804b390 0x55ca883ec810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.438+0000 7f33977fe640 1 -- 172.21.15.90:0/3153738154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca8802a6b0 msgr2=0x55ca883f96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.438+0000 7f33977fe640 1 --2- 172.21.15.90:0/3153738154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca8802a6b0 0x55ca883f96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.438+0000 7f33977fe640 1 -- 172.21.15.90:0/3153738154 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca88316c30 con 0x55ca880224d0 2024-04-25T11:44:16.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.438+0000 7f3396ffd640 1 --2- 172.21.15.90:0/3153738154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca8802a6b0 0x55ca883f96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:16.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.438+0000 7f3397fff640 1 --2- 172.21.15.90:0/2981053511 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f33800afc60 0x7f33800b2170 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f33900dca00 tx=0x7f339006d040 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.438+0000 7f33977fe640 1 --2- 172.21.15.90:0/3153738154 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ca880224d0 0x55ca883ecd50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f338800a340 tx=0x7f3388004170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4af2e96877a9452f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.438+0000 7f337dffb640 1 -- 172.21.15.90:0/3153738154 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3388041070 con 0x55ca880224d0 2024-04-25T11:44:16.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.438+0000 7f337dffb640 1 -- 172.21.15.90:0/3153738154 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33880054f0 con 0x55ca880224d0 2024-04-25T11:44:16.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.439+0000 7f337dffb640 1 -- 172.21.15.90:0/3153738154 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3388036040 con 0x55ca880224d0 2024-04-25T11:44:16.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.439+0000 7f33a4d68740 1 -- 172.21.15.90:0/3153738154 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ca880224d0 msgr2=0x55ca883ecd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.439+0000 7f33a4d68740 1 --2- 172.21.15.90:0/3153738154 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ca880224d0 0x55ca883ecd50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f338800a340 tx=0x7f3388004170 comp rx=0 tx=0).stop 2024-04-25T11:44:16.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.439+0000 7f33a4d68740 1 -- 172.21.15.90:0/3153738154 shutdown_connections 2024-04-25T11:44:16.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.439+0000 7f33a4d68740 1 --2- 172.21.15.90:0/3153738154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca8802a6b0 0x55ca883f96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.439+0000 7f33a4d68740 1 --2- 172.21.15.90:0/3153738154 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ca880224d0 0x55ca883ecd50 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.439+0000 7f33a4d68740 1 --2- 172.21.15.90:0/3153738154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca8804b390 0x55ca883ec810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.439+0000 7f33a4d68740 1 -- 172.21.15.90:0/3153738154 >> 172.21.15.90:0/3153738154 conn(0x55ca8803f4e0 msgr2=0x55ca883e8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.439+0000 7f33a4d68740 1 -- 172.21.15.90:0/3153738154 shutdown_connections 2024-04-25T11:44:16.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.440+0000 7f33a4d68740 1 -- 172.21.15.90:0/3153738154 wait complete. 2024-04-25T11:44:16.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.440+0000 7f33a4d68740 1 -- start start 2024-04-25T11:44:16.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.440+0000 7f33a4d68740 1 Processor -- start 2024-04-25T11:44:16.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.440+0000 7f33a4d68740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca880224d0 0x55ca883f7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.440+0000 7f33a4d68740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca8802a6b0 0x55ca883f77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.440+0000 7f33a4d68740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ca8804b390 0x55ca884031b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.440+0000 7f33a4d68740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ca883f9d30 con 0x55ca8802a6b0 2024-04-25T11:44:16.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.440+0000 7f33a4d68740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ca882d8dc0 con 0x55ca8804b390 2024-04-25T11:44:16.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.440+0000 7f33a4d68740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ca883f9eb0 con 0x55ca880224d0 2024-04-25T11:44:16.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.440+0000 7f33a4d68740 10 client.0 _ll_register_callbacks cb 0x55ca882f4600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:16.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.441+0000 7f33977fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca8802a6b0 0x55ca883f77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.441+0000 7f3396ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ca8804b390 0x55ca884031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.441+0000 7f33977fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca8802a6b0 0x55ca883f77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47458/0 (socket says 172.21.15.90:47458) 2024-04-25T11:44:16.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.441+0000 7f33977fe640 1 -- 172.21.15.90:0/2992442889 learned_addr learned my addr 172.21.15.90:0/2992442889 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:16.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.441+0000 7f3396ffd640 1 --2- 172.21.15.90:0/2992442889 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca880224d0 0x55ca883f7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.441+0000 7f33977fe640 1 -- 172.21.15.90:0/2992442889 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca880224d0 msgr2=0x55ca883f7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.441+0000 7f33977fe640 1 --2- 172.21.15.90:0/2992442889 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca880224d0 0x55ca883f7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.441+0000 7f33977fe640 1 -- 172.21.15.90:0/2992442889 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ca8804b390 msgr2=0x55ca884031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.441+0000 7f33977fe640 1 --2- 172.21.15.90:0/2992442889 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ca8804b390 0x55ca884031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.441+0000 7f33977fe640 1 -- 172.21.15.90:0/2992442889 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca880633f0 con 0x55ca8802a6b0 2024-04-25T11:44:16.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.441+0000 7f3396ffd640 1 --2- 172.21.15.90:0/2992442889 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ca8804b390 0x55ca884031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:16.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.441+0000 7f33977fe640 1 --2- 172.21.15.90:0/2992442889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca8802a6b0 0x55ca883f77c0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f3388030550 tx=0x7f338802b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=577332be3a4586de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.441+0000 7f3376ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:16.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.441+0000 7f3376ffd640 1 -- 172.21.15.90:0/2992442889 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3388041040 con 0x55ca8802a6b0 2024-04-25T11:44:16.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.441+0000 7f3376ffd640 1 -- 172.21.15.90:0/2992442889 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7f3388004d00 con 0x55ca8802a6b0 2024-04-25T11:44:16.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.441+0000 7f33a4d68740 1 -- 172.21.15.90:0/2992442889 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33880054f0 con 0x55ca8802a6b0 2024-04-25T11:44:16.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.441+0000 7f3376ffd640 1 -- 172.21.15.90:0/2992442889 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f338800a630 con 0x55ca8802a6b0 2024-04-25T11:44:16.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.441+0000 7f33a4d68740 10 client.18480 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:16.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.441+0000 7f33a4d68740 1 -- 172.21.15.90:0/2992442889 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ca8812d7d0 con 0x55ca8802a6b0 2024-04-25T11:44:16.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.442+0000 7f3376ffd640 1 -- 172.21.15.90:0/2992442889 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33880054f0 con 0x55ca8802a6b0 2024-04-25T11:44:16.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.442+0000 7f3376ffd640 1 -- 172.21.15.90:0/2992442889 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f338803b040 con 0x55ca8802a6b0 2024-04-25T11:44:16.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.442+0000 7f33a4d68740 10 client.18480 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:16.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.442+0000 7f33a4d68740 10 client.18480 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:44:16.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.442+0000 7f33a4d68740 20 client.18480 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:16.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.442+0000 7f33a4d68740 1 --2- 172.21.15.90:0/2992442889 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55ca88009f60 0x55ca884094b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.442+0000 7f33a4d68740 4 client.18480 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:44:16.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.442+0000 7f33a4d68740 1 -- 172.21.15.90:0/2992442889 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca8835f990 con 0x55ca88009f60 2024-04-25T11:44:16.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.442+0000 7f3396ffd640 1 --2- 172.21.15.90:0/2992442889 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55ca88009f60 0x55ca884094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.443+0000 7f3376ffd640 10 client.18480 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:44:16.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.443+0000 7f3396ffd640 1 --2- 172.21.15.90:0/2992442889 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55ca88009f60 0x55ca884094b0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5a6b9b8833b3324e server_cookie=a378a111fb4d7a13 in_seq=0 out_seq=0 2024-04-25T11:44:16.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.443+0000 7f3376ffd640 1 -- 172.21.15.90:0/2992442889 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ca8835f990 con 0x55ca88009f60 2024-04-25T11:44:16.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.443+0000 7f3376ffd640 10 client.18480 handle_command_reply: tid=0 2024-04-25T11:44:16.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.443+0000 7f33a4d68740 1 client.18480 shutdown 2024-04-25T11:44:16.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.444+0000 7f33a4d68740 1 -- 172.21.15.90:0/2992442889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca8802a6b0 msgr2=0x55ca883f77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.444+0000 7f33a4d68740 1 --2- 172.21.15.90:0/2992442889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca8802a6b0 0x55ca883f77c0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f3388030550 tx=0x7f338802b720 comp rx=0 tx=0).stop 2024-04-25T11:44:16.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.444+0000 7f33a4d68740 1 -- 172.21.15.90:0/2992442889 shutdown_connections 2024-04-25T11:44:16.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.444+0000 7f33a4d68740 1 --2- 172.21.15.90:0/2992442889 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55ca88009f60 0x55ca884094b0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.444+0000 7f3376ffd640 0 client.18480 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:44:16.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.444+0000 7f33a4d68740 1 --2- 172.21.15.90:0/2992442889 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ca8804b390 0x55ca884031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.444+0000 7f33a4d68740 1 --2- 172.21.15.90:0/2992442889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca8802a6b0 0x55ca883f77c0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.444+0000 7f33a4d68740 1 --2- 172.21.15.90:0/2992442889 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca880224d0 0x55ca883f7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.444+0000 7f33a4d68740 1 -- 172.21.15.90:0/2992442889 >> 172.21.15.90:0/2992442889 conn(0x55ca8803f4e0 msgr2=0x55ca883ee530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.444+0000 7f33a4d68740 1 -- 172.21.15.90:0/2992442889 shutdown_connections 2024-04-25T11:44:16.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.444+0000 7f33a4d68740 1 -- 172.21.15.90:0/2992442889 wait complete. 2024-04-25T11:44:16.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.444+0000 7f33a4d68740 20 client.18480 trim_cache size 0 max 16384 2024-04-25T11:44:16.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.457+0000 7f33a4d68740 1 Processor -- start 2024-04-25T11:44:16.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.457+0000 7f33a4d68740 1 -- start start 2024-04-25T11:44:16.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.457+0000 7f33a4d68740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca88009f60 0x55ca884031a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.457+0000 7f33a4d68740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ca880224d0 0x55ca883e8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.457+0000 7f33a4d68740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca8802a6b0 0x55ca8840c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.457+0000 7f33a4d68740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f33981adb00 con 0x55ca8802a6b0 2024-04-25T11:44:16.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.457+0000 7f33a4d68740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f33981af640 con 0x55ca880224d0 2024-04-25T11:44:16.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.457+0000 7f33a4d68740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f33981ae1d0 con 0x55ca88009f60 2024-04-25T11:44:16.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.458+0000 7f3396ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca8802a6b0 0x55ca8840c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.458+0000 7f3396ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca8802a6b0 0x55ca8840c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47472/0 (socket says 172.21.15.90:47472) 2024-04-25T11:44:16.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.458+0000 7f3396ffd640 1 -- 172.21.15.90:0/3195281375 learned_addr learned my addr 172.21.15.90:0/3195281375 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:16.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.458+0000 7f33977fe640 1 --2- 172.21.15.90:0/3195281375 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ca880224d0 0x55ca883e8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.458+0000 7f3396ffd640 1 --2- 172.21.15.90:0/3195281375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca88009f60 0x55ca884031a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.458+0000 7f3396ffd640 1 -- 172.21.15.90:0/3195281375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca88009f60 msgr2=0x55ca884031a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.458+0000 7f3396ffd640 1 --2- 172.21.15.90:0/3195281375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca88009f60 0x55ca884031a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.458+0000 7f3396ffd640 1 -- 172.21.15.90:0/3195281375 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ca880224d0 msgr2=0x55ca883e8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.458+0000 7f3396ffd640 1 --2- 172.21.15.90:0/3195281375 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ca880224d0 0x55ca883e8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.458+0000 7f3396ffd640 1 -- 172.21.15.90:0/3195281375 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca883e8620 con 0x55ca8802a6b0 2024-04-25T11:44:16.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.458+0000 7f3396ffd640 1 --2- 172.21.15.90:0/3195281375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca8802a6b0 0x55ca8840c4d0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f338c0127a0 tx=0x7f338c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2682c9fe8f2c49de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.458+0000 7f337d7fa640 1 -- 172.21.15.90:0/3195281375 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f338c026e60 con 0x55ca8802a6b0 2024-04-25T11:44:16.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.458+0000 7f337d7fa640 1 -- 172.21.15.90:0/3195281375 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f338c0054d0 con 0x55ca8802a6b0 2024-04-25T11:44:16.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.458+0000 7f337d7fa640 1 -- 172.21.15.90:0/3195281375 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f338c0035f0 con 0x55ca8802a6b0 2024-04-25T11:44:16.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.459+0000 7f33a4d68740 1 -- 172.21.15.90:0/3195281375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca8802a6b0 msgr2=0x55ca8840c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.459+0000 7f33a4d68740 1 --2- 172.21.15.90:0/3195281375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca8802a6b0 0x55ca8840c4d0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f338c0127a0 tx=0x7f338c0046c0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.459+0000 7f33a4d68740 1 -- 172.21.15.90:0/3195281375 shutdown_connections 2024-04-25T11:44:16.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.459+0000 7f33a4d68740 1 --2- 172.21.15.90:0/3195281375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca8802a6b0 0x55ca8840c4d0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.459+0000 7f33a4d68740 1 --2- 172.21.15.90:0/3195281375 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ca880224d0 0x55ca883e8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.459+0000 7f33a4d68740 1 --2- 172.21.15.90:0/3195281375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca88009f60 0x55ca884031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.459+0000 7f33a4d68740 1 -- 172.21.15.90:0/3195281375 >> 172.21.15.90:0/3195281375 conn(0x55ca8803f4e0 msgr2=0x55ca883e8860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.459+0000 7f33a4d68740 1 -- 172.21.15.90:0/3195281375 shutdown_connections 2024-04-25T11:44:16.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.459+0000 7f33a4d68740 1 -- 172.21.15.90:0/3195281375 wait complete. 2024-04-25T11:44:16.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.459+0000 7f33a4d68740 1 -- start start 2024-04-25T11:44:16.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.460+0000 7f33a4d68740 1 Processor -- start 2024-04-25T11:44:16.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.460+0000 7f33a4d68740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca88009f60 0x55ca88409990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.460+0000 7f33a4d68740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca880224d0 0x55ca88409ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.460+0000 7f33a4d68740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ca8802a6b0 0x55ca88469680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.460+0000 7f33a4d68740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f33981aef70 con 0x55ca88009f60 2024-04-25T11:44:16.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.460+0000 7f33a4d68740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f33981ae8a0 con 0x55ca8802a6b0 2024-04-25T11:44:16.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.460+0000 7f33a4d68740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f33981abb00 con 0x55ca880224d0 2024-04-25T11:44:16.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.460+0000 7f33a4d68740 10 client.0 _ll_register_callbacks cb 0x7f338c009990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:16.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.460+0000 7f3396ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca88009f60 0x55ca88409990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.460+0000 7f3396ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca88009f60 0x55ca88409990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47484/0 (socket says 172.21.15.90:47484) 2024-04-25T11:44:16.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.460+0000 7f3396ffd640 1 -- 172.21.15.90:0/3326463441 learned_addr learned my addr 172.21.15.90:0/3326463441 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:16.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.460+0000 7f33977fe640 1 --2- 172.21.15.90:0/3326463441 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca880224d0 0x55ca88409ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.460+0000 7f3396ffd640 1 --2- 172.21.15.90:0/3326463441 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ca8802a6b0 0x55ca88469680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.460+0000 7f3396ffd640 1 -- 172.21.15.90:0/3326463441 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca880224d0 msgr2=0x55ca88409ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.460+0000 7f3396ffd640 1 --2- 172.21.15.90:0/3326463441 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca880224d0 0x55ca88409ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.460+0000 7f3396ffd640 1 -- 172.21.15.90:0/3326463441 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ca8802a6b0 msgr2=0x55ca88469680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.460+0000 7f3396ffd640 1 --2- 172.21.15.90:0/3326463441 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ca8802a6b0 0x55ca88469680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.460+0000 7f3396ffd640 1 -- 172.21.15.90:0/3326463441 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca88469bf0 con 0x55ca88009f60 2024-04-25T11:44:16.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.461+0000 7f3396ffd640 1 --2- 172.21.15.90:0/3326463441 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca88009f60 0x55ca88409990 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f338c004650 tx=0x7f338c004b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30abb081ecf440cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.461+0000 7f3376ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:16.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.461+0000 7f3376ffd640 1 -- 172.21.15.90:0/3326463441 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f338c026e60 con 0x55ca88009f60 2024-04-25T11:44:16.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.461+0000 7f3376ffd640 1 -- 172.21.15.90:0/3326463441 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7f338c004730 con 0x55ca88009f60 2024-04-25T11:44:16.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.461+0000 7f33a4d68740 1 -- 172.21.15.90:0/3326463441 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f338c0054d0 con 0x55ca88009f60 2024-04-25T11:44:16.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.461+0000 7f3376ffd640 1 -- 172.21.15.90:0/3326463441 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f338c004930 con 0x55ca88009f60 2024-04-25T11:44:16.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.461+0000 7f33a4d68740 10 client.18486 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:16.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.461+0000 7f33a4d68740 1 -- 172.21.15.90:0/3326463441 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ca88470ef0 con 0x55ca88009f60 2024-04-25T11:44:16.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.461+0000 7f3376ffd640 1 -- 172.21.15.90:0/3326463441 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f338c0054d0 con 0x55ca88009f60 2024-04-25T11:44:16.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.461+0000 7f3376ffd640 1 -- 172.21.15.90:0/3326463441 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f338c02e070 con 0x55ca88009f60 2024-04-25T11:44:16.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.461+0000 7f33a4d68740 10 client.18486 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:16.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.461+0000 7f33a4d68740 10 client.18486 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:44:16.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.461+0000 7f33a4d68740 20 client.18486 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:16.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.461+0000 7f33a4d68740 1 --2- 172.21.15.90:0/3326463441 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55ca8804b390 0x55ca884736b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.461+0000 7f33a4d68740 4 client.18486 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:44:16.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.461+0000 7f33a4d68740 1 -- 172.21.15.90:0/3326463441 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ca88403df0 con 0x55ca8804b390 2024-04-25T11:44:16.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.462+0000 7f33977fe640 1 --2- 172.21.15.90:0/3326463441 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55ca8804b390 0x55ca884736b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.462+0000 7f3376ffd640 10 client.18486 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:44:16.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.462+0000 7f33977fe640 1 --2- 172.21.15.90:0/3326463441 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55ca8804b390 0x55ca884736b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9c98aaf2e39e2f6a server_cookie=7fba5fa94ad52ec2 in_seq=0 out_seq=0 2024-04-25T11:44:16.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.462+0000 7f3376ffd640 1 -- 172.21.15.90:0/3326463441 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ca88403df0 con 0x55ca8804b390 2024-04-25T11:44:16.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.462+0000 7f3376ffd640 10 client.18486 handle_command_reply: tid=0 2024-04-25T11:44:16.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.462+0000 7f33a4d68740 1 client.18486 shutdown 2024-04-25T11:44:16.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.462+0000 7f33a4d68740 1 -- 172.21.15.90:0/3326463441 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca88009f60 msgr2=0x55ca88409990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.462+0000 7f33a4d68740 1 --2- 172.21.15.90:0/3326463441 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca88009f60 0x55ca88409990 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f338c004650 tx=0x7f338c004b70 comp rx=0 tx=0).stop 2024-04-25T11:44:16.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.462+0000 7f33a4d68740 1 -- 172.21.15.90:0/3326463441 shutdown_connections 2024-04-25T11:44:16.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.462+0000 7f33a4d68740 1 --2- 172.21.15.90:0/3326463441 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55ca8804b390 0x55ca884736b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.462+0000 7f33a4d68740 1 --2- 172.21.15.90:0/3326463441 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ca8802a6b0 0x55ca88469680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.462+0000 7f33a4d68740 1 --2- 172.21.15.90:0/3326463441 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca880224d0 0x55ca88409ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.462+0000 7f33a4d68740 1 --2- 172.21.15.90:0/3326463441 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca88009f60 0x55ca88409990 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.462+0000 7f3376ffd640 0 client.18486 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:44:16.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.462+0000 7f33a4d68740 1 -- 172.21.15.90:0/3326463441 >> 172.21.15.90:0/3326463441 conn(0x55ca8803f4e0 msgr2=0x55ca883f5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.463+0000 7f33a4d68740 1 -- 172.21.15.90:0/3326463441 shutdown_connections 2024-04-25T11:44:16.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.463+0000 7f33a4d68740 1 -- 172.21.15.90:0/3326463441 wait complete. 2024-04-25T11:44:16.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.463+0000 7f33a4d68740 20 client.18486 trim_cache size 0 max 16384 2024-04-25T11:44:16.455 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:44:16.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.463+0000 7f339deb8640 1 -- 172.21.15.90:0/2981053511 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f33800afc60 msgr2=0x7f33800b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.463+0000 7f339deb8640 1 --2- 172.21.15.90:0/2981053511 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f33800afc60 0x7f33800b2170 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f33900dca00 tx=0x7f339006d040 comp rx=0 tx=0).stop 2024-04-25T11:44:16.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.463+0000 7f339deb8640 1 -- 172.21.15.90:0/2981053511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3398107450 msgr2=0x7f339819efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.463+0000 7f339deb8640 1 --2- 172.21.15.90:0/2981053511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3398107450 0x7f339819efc0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f339000eaa0 tx=0x7f339000ef60 comp rx=0 tx=0).stop 2024-04-25T11:44:16.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.463+0000 7f339deb8640 1 -- 172.21.15.90:0/2981053511 shutdown_connections 2024-04-25T11:44:16.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.463+0000 7f339deb8640 1 --2- 172.21.15.90:0/2981053511 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f33800afc60 0x7f33800b2170 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.463+0000 7f339deb8640 1 --2- 172.21.15.90:0/2981053511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3398107450 0x7f339819efc0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.463+0000 7f339deb8640 1 --2- 172.21.15.90:0/2981053511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3398106990 0x7f339819ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.463+0000 7f339deb8640 1 --2- 172.21.15.90:0/2981053511 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3398102400 0x7f339819a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.463+0000 7f339deb8640 1 -- 172.21.15.90:0/2981053511 >> 172.21.15.90:0/2981053511 conn(0x7f3398077880 msgr2=0x7f33981008c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.463+0000 7f339deb8640 1 -- 172.21.15.90:0/2981053511 shutdown_connections 2024-04-25T11:44:16.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.463+0000 7f339deb8640 1 -- 172.21.15.90:0/2981053511 wait complete. 2024-04-25T11:44:16.461 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T11:44:16.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.591+0000 7ff227f5e640 1 -- 172.21.15.90:0/2163529375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff220109fe0 msgr2=0x7ff220102150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.591+0000 7ff227f5e640 1 --2- 172.21.15.90:0/2163529375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff220109fe0 0x7ff220102150 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7ff21c009ca0 tx=0x7ff21c02a8c0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.591+0000 7ff227f5e640 1 -- 172.21.15.90:0/2163529375 shutdown_connections 2024-04-25T11:44:16.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.591+0000 7ff227f5e640 1 --2- 172.21.15.90:0/2163529375 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff2201027e0 0x7ff220110660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.591+0000 7ff227f5e640 1 --2- 172.21.15.90:0/2163529375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff220109fe0 0x7ff220102150 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.591+0000 7ff227f5e640 1 --2- 172.21.15.90:0/2163529375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff220109b40 0x7ff220101bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.591+0000 7ff227f5e640 1 -- 172.21.15.90:0/2163529375 >> 172.21.15.90:0/2163529375 conn(0x7ff220077920 msgr2=0x7ff220077d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.591+0000 7ff227f5e640 1 -- 172.21.15.90:0/2163529375 shutdown_connections 2024-04-25T11:44:16.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.591+0000 7ff227f5e640 1 -- 172.21.15.90:0/2163529375 wait complete. 2024-04-25T11:44:16.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.592+0000 7ff227f5e640 1 Processor -- start 2024-04-25T11:44:16.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.592+0000 7ff227f5e640 1 -- start start 2024-04-25T11:44:16.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.592+0000 7ff227f5e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff220109b40 0x7ff220199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.592+0000 7ff227f5e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff22019a510 0x7ff22019e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.592+0000 7ff227f5e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff22019ed30 0x7ff22019f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.592+0000 7ff227f5e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff220113d30 con 0x7ff22019a510 2024-04-25T11:44:16.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.592+0000 7ff227f5e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff220113bb0 con 0x7ff220109b40 2024-04-25T11:44:16.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.592+0000 7ff227f5e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff220113eb0 con 0x7ff22019ed30 2024-04-25T11:44:16.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.592+0000 7ff2254f9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff22019a510 0x7ff22019e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.593+0000 7ff2254f9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff22019a510 0x7ff22019e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47506/0 (socket says 172.21.15.90:47506) 2024-04-25T11:44:16.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.593+0000 7ff2254f9640 1 -- 172.21.15.90:0/1974054253 learned_addr learned my addr 172.21.15.90:0/1974054253 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:16.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.593+0000 7ff2264fb640 1 --2- 172.21.15.90:0/1974054253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff22019ed30 0x7ff22019f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.593+0000 7ff225cfa640 1 --2- 172.21.15.90:0/1974054253 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff220109b40 0x7ff220199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.593+0000 7ff2254f9640 1 -- 172.21.15.90:0/1974054253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff22019ed30 msgr2=0x7ff22019f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.593+0000 7ff2254f9640 1 --2- 172.21.15.90:0/1974054253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff22019ed30 0x7ff22019f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.593+0000 7ff2254f9640 1 -- 172.21.15.90:0/1974054253 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff220109b40 msgr2=0x7ff220199fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.593+0000 7ff2254f9640 1 --2- 172.21.15.90:0/1974054253 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff220109b40 0x7ff220199fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.593+0000 7ff2254f9640 1 -- 172.21.15.90:0/1974054253 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2201a6aa0 con 0x7ff22019a510 2024-04-25T11:44:16.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.593+0000 7ff2254f9640 1 --2- 172.21.15.90:0/1974054253 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff22019a510 0x7ff22019e7f0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7ff21c033040 tx=0x7ff21c0316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.593+0000 7ff216ffd640 1 -- 172.21.15.90:0/1974054253 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21c003b70 con 0x7ff22019a510 2024-04-25T11:44:16.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.593+0000 7ff216ffd640 1 -- 172.21.15.90:0/1974054253 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff21c003d10 con 0x7ff22019a510 2024-04-25T11:44:16.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.593+0000 7ff216ffd640 1 -- 172.21.15.90:0/1974054253 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21c02fa80 con 0x7ff22019a510 2024-04-25T11:44:16.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.593+0000 7ff227f5e640 1 -- 172.21.15.90:0/1974054253 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2201a6c70 con 0x7ff22019a510 2024-04-25T11:44:16.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.593+0000 7ff227f5e640 1 -- 172.21.15.90:0/1974054253 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2201a70d0 con 0x7ff22019a510 2024-04-25T11:44:16.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.596+0000 7ff22ee0f740 1 Processor -- start 2024-04-25T11:44:16.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.596+0000 7ff22ee0f740 1 -- start start 2024-04-25T11:44:16.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.596+0000 7ff216ffd640 1 -- 172.21.15.90:0/1974054253 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7ff21c02fc20 con 0x7ff22019a510 2024-04-25T11:44:16.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.596+0000 7ff22ee0f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56415c84c390 0x56415cbed810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.596+0000 7ff22ee0f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56415c8234d0 0x56415cbedd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.596+0000 7ff22ee0f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c82b6b0 0x56415cbfa6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.596+0000 7ff22ee0f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56415cb52d80 con 0x56415c82b6b0 2024-04-25T11:44:16.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.596+0000 7ff22ee0f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56415cb0ca20 con 0x56415c8234d0 2024-04-25T11:44:16.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.596+0000 7ff22ee0f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56415cb4ed40 con 0x56415c84c390 2024-04-25T11:44:16.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.596+0000 7ff2254f9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c82b6b0 0x56415cbfa6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.596+0000 7ff225cfa640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56415c8234d0 0x56415cbedd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.596+0000 7ff2254f9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c82b6b0 0x56415cbfa6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47508/0 (socket says 172.21.15.90:47508) 2024-04-25T11:44:16.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.596+0000 7ff2254f9640 1 -- 172.21.15.90:0/1532630464 learned_addr learned my addr 172.21.15.90:0/1532630464 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:16.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.596+0000 7ff2264fb640 1 --2- 172.21.15.90:0/1532630464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56415c84c390 0x56415cbed810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.597+0000 7ff225cfa640 1 -- 172.21.15.90:0/1532630464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56415c84c390 msgr2=0x56415cbed810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.597+0000 7ff225cfa640 1 --2- 172.21.15.90:0/1532630464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56415c84c390 0x56415cbed810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.597+0000 7ff225cfa640 1 -- 172.21.15.90:0/1532630464 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c82b6b0 msgr2=0x56415cbfa6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.597+0000 7ff225cfa640 1 --2- 172.21.15.90:0/1532630464 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c82b6b0 0x56415cbfa6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.597+0000 7ff225cfa640 1 -- 172.21.15.90:0/1532630464 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56415cb17c30 con 0x56415c8234d0 2024-04-25T11:44:16.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.597+0000 7ff216ffd640 1 --2- 172.21.15.90:0/1974054253 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff2080afc60 0x7ff2080b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.597+0000 7ff225cfa640 1 --2- 172.21.15.90:0/1532630464 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56415c8234d0 0x56415cbedd50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff218009bd0 tx=0x7ff218011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a65826e531652503 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.597+0000 7ff2264fb640 1 --2- 172.21.15.90:0/1974054253 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff2080afc60 0x7ff2080b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.597+0000 7ff216ffd640 1 -- 172.21.15.90:0/1974054253 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7ff21c100340 con 0x7ff22019a510 2024-04-25T11:44:16.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.597+0000 7ff207fff640 1 -- 172.21.15.90:0/1532630464 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff218004050 con 0x56415c8234d0 2024-04-25T11:44:16.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.597+0000 7ff207fff640 1 -- 172.21.15.90:0/1532630464 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2180054d0 con 0x56415c8234d0 2024-04-25T11:44:16.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.597+0000 7ff207fff640 1 -- 172.21.15.90:0/1532630464 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff218004c00 con 0x56415c8234d0 2024-04-25T11:44:16.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.598+0000 7ff22ee0f740 1 -- 172.21.15.90:0/1532630464 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56415c8234d0 msgr2=0x56415cbedd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.598+0000 7ff2264fb640 1 --2- 172.21.15.90:0/1974054253 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff2080afc60 0x7ff2080b2170 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x56415cbea980 tx=0x7ff210005f90 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.598+0000 7ff22ee0f740 1 --2- 172.21.15.90:0/1532630464 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56415c8234d0 0x56415cbedd50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff218009bd0 tx=0x7ff218011810 comp rx=0 tx=0).stop 2024-04-25T11:44:16.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.598+0000 7ff22ee0f740 1 -- 172.21.15.90:0/1532630464 shutdown_connections 2024-04-25T11:44:16.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.598+0000 7ff22ee0f740 1 --2- 172.21.15.90:0/1532630464 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c82b6b0 0x56415cbfa6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.598+0000 7ff22ee0f740 1 --2- 172.21.15.90:0/1532630464 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56415c8234d0 0x56415cbedd50 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.598+0000 7ff22ee0f740 1 --2- 172.21.15.90:0/1532630464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56415c84c390 0x56415cbed810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.598+0000 7ff22ee0f740 1 -- 172.21.15.90:0/1532630464 >> 172.21.15.90:0/1532630464 conn(0x56415c8404e0 msgr2=0x56415cbe9fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.598+0000 7ff22ee0f740 1 -- 172.21.15.90:0/1532630464 shutdown_connections 2024-04-25T11:44:16.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.598+0000 7ff22ee0f740 1 -- 172.21.15.90:0/1532630464 wait complete. 2024-04-25T11:44:16.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.599+0000 7ff22ee0f740 1 -- start start 2024-04-25T11:44:16.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.599+0000 7ff22ee0f740 1 Processor -- start 2024-04-25T11:44:16.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.599+0000 7ff22ee0f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56415c8234d0 0x56415cbf8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.599+0000 7ff22ee0f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56415c82b6b0 0x56415cbf8750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.599+0000 7ff22ee0f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c84c390 0x56415cc04210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.599+0000 7ff22ee0f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56415cbfad90 con 0x56415c84c390 2024-04-25T11:44:16.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.599+0000 7ff22ee0f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56415cad9dc0 con 0x56415c8234d0 2024-04-25T11:44:16.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.599+0000 7ff22ee0f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56415cbfaf10 con 0x56415c82b6b0 2024-04-25T11:44:16.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.599+0000 7ff22ee0f740 10 client.0 _ll_register_callbacks cb 0x56415caf5600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:16.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.599+0000 7ff2264fb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c84c390 0x56415cc04210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.599+0000 7ff2264fb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c84c390 0x56415cc04210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47516/0 (socket says 172.21.15.90:47516) 2024-04-25T11:44:16.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.599+0000 7ff2264fb640 1 -- 172.21.15.90:0/49416479 learned_addr learned my addr 172.21.15.90:0/49416479 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:16.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.599+0000 7ff2254f9640 1 --2- 172.21.15.90:0/49416479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56415c82b6b0 0x56415cbf8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.599+0000 7ff225cfa640 1 --2- 172.21.15.90:0/49416479 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56415c8234d0 0x56415cbf8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.600+0000 7ff2264fb640 1 -- 172.21.15.90:0/49416479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56415c82b6b0 msgr2=0x56415cbf8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.600+0000 7ff2264fb640 1 --2- 172.21.15.90:0/49416479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56415c82b6b0 0x56415cbf8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.600+0000 7ff2264fb640 1 -- 172.21.15.90:0/49416479 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56415c8234d0 msgr2=0x56415cbf8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.600+0000 7ff2264fb640 1 --2- 172.21.15.90:0/49416479 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56415c8234d0 0x56415cbf8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.600+0000 7ff2264fb640 1 -- 172.21.15.90:0/49416479 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56415c8643f0 con 0x56415c84c390 2024-04-25T11:44:16.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.600+0000 7ff225cfa640 1 --2- 172.21.15.90:0/49416479 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56415c8234d0 0x56415cbf8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:44:16.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.600+0000 7ff2264fb640 1 --2- 172.21.15.90:0/49416479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c84c390 0x56415cc04210 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7ff210004610 tx=0x7ff210004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bdc5a0d6fc7a0b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.600+0000 7ff2057fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:16.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.600+0000 7ff2057fa640 1 -- 172.21.15.90:0/49416479 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21001e470 con 0x56415c84c390 2024-04-25T11:44:16.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.600+0000 7ff2057fa640 1 -- 172.21.15.90:0/49416479 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7ff21001e610 con 0x56415c84c390 2024-04-25T11:44:16.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.600+0000 7ff22ee0f740 1 -- 172.21.15.90:0/49416479 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2180054d0 con 0x56415c84c390 2024-04-25T11:44:16.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.600+0000 7ff2057fa640 1 -- 172.21.15.90:0/49416479 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff21001e810 con 0x56415c84c390 2024-04-25T11:44:16.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.600+0000 7ff22ee0f740 10 client.18498 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:16.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.600+0000 7ff22ee0f740 1 -- 172.21.15.90:0/49416479 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56415c92e7d0 con 0x56415c84c390 2024-04-25T11:44:16.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.600+0000 7ff2057fa640 1 -- 172.21.15.90:0/49416479 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2180054d0 con 0x56415c84c390 2024-04-25T11:44:16.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.601+0000 7ff2057fa640 1 -- 172.21.15.90:0/49416479 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7ff210027070 con 0x56415c84c390 2024-04-25T11:44:16.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.601+0000 7ff22ee0f740 10 client.18498 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:16.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.601+0000 7ff22ee0f740 10 client.18498 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:44:16.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.601+0000 7ff22ee0f740 20 client.18498 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:16.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.601+0000 7ff22ee0f740 1 --2- 172.21.15.90:0/49416479 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x56415c80af60 0x56415cc0a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.601+0000 7ff22ee0f740 4 client.18498 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:44:16.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.601+0000 7ff22ee0f740 1 -- 172.21.15.90:0/49416479 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56415cb60990 con 0x56415c80af60 2024-04-25T11:44:16.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.601+0000 7ff225cfa640 1 --2- 172.21.15.90:0/49416479 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x56415c80af60 0x56415cc0a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.601+0000 7ff2057fa640 10 client.18498 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:44:16.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.601+0000 7ff225cfa640 1 --2- 172.21.15.90:0/49416479 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x56415c80af60 0x56415cc0a790 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2bccc236f092f920 server_cookie=4cf62edd3925f86c in_seq=0 out_seq=0 2024-04-25T11:44:16.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.602+0000 7ff2057fa640 1 -- 172.21.15.90:0/49416479 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56415cb60990 con 0x56415c80af60 2024-04-25T11:44:16.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.602+0000 7ff2057fa640 10 client.18498 handle_command_reply: tid=0 2024-04-25T11:44:16.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.602+0000 7ff22ee0f740 1 client.18498 shutdown 2024-04-25T11:44:16.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.602+0000 7ff22ee0f740 1 -- 172.21.15.90:0/49416479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c84c390 msgr2=0x56415cc04210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.602+0000 7ff22ee0f740 1 --2- 172.21.15.90:0/49416479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c84c390 0x56415cc04210 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7ff210004610 tx=0x7ff210004730 comp rx=0 tx=0).stop 2024-04-25T11:44:16.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.602+0000 7ff22ee0f740 1 -- 172.21.15.90:0/49416479 shutdown_connections 2024-04-25T11:44:16.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.602+0000 7ff22ee0f740 1 --2- 172.21.15.90:0/49416479 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x56415c80af60 0x56415cc0a790 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.602+0000 7ff22ee0f740 1 --2- 172.21.15.90:0/49416479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c84c390 0x56415cc04210 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.602+0000 7ff2057fa640 0 client.18498 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:44:16.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.602+0000 7ff22ee0f740 1 --2- 172.21.15.90:0/49416479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56415c82b6b0 0x56415cbf8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.602+0000 7ff22ee0f740 1 --2- 172.21.15.90:0/49416479 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56415c8234d0 0x56415cbf8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.602+0000 7ff22ee0f740 1 -- 172.21.15.90:0/49416479 >> 172.21.15.90:0/49416479 conn(0x56415c8404e0 msgr2=0x56415cbef530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.602+0000 7ff22ee0f740 1 -- 172.21.15.90:0/49416479 shutdown_connections 2024-04-25T11:44:16.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.602+0000 7ff22ee0f740 1 -- 172.21.15.90:0/49416479 wait complete. 2024-04-25T11:44:16.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.602+0000 7ff22ee0f740 20 client.18498 trim_cache size 0 max 16384 2024-04-25T11:44:16.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.616+0000 7ff22ee0f740 1 Processor -- start 2024-04-25T11:44:16.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.616+0000 7ff22ee0f740 1 -- start start 2024-04-25T11:44:16.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.616+0000 7ff22ee0f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56415c8234d0 0x56415cbe9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.616+0000 7ff22ee0f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c82b6b0 0x56415cc03e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.616+0000 7ff22ee0f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56415c84c390 0x56415cbf2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.616+0000 7ff22ee0f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff2201afd40 con 0x56415c82b6b0 2024-04-25T11:44:16.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.616+0000 7ff22ee0f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff2201b1880 con 0x56415c84c390 2024-04-25T11:44:16.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.616+0000 7ff22ee0f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff2201b0410 con 0x56415c8234d0 2024-04-25T11:44:16.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.616+0000 7ff2254f9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c82b6b0 0x56415cc03e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.616+0000 7ff2264fb640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56415c84c390 0x56415cbf2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.616+0000 7ff2254f9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c82b6b0 0x56415cc03e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47524/0 (socket says 172.21.15.90:47524) 2024-04-25T11:44:16.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.616+0000 7ff225cfa640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56415c8234d0 0x56415cbe9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.616+0000 7ff2264fb640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56415c84c390 0x56415cbf2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:57032/0 (socket says 172.21.15.90:57032) 2024-04-25T11:44:16.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.616+0000 7ff2254f9640 1 -- 172.21.15.90:0/2005603700 learned_addr learned my addr 172.21.15.90:0/2005603700 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:16.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.617+0000 7ff2264fb640 1 -- 172.21.15.90:0/2005603700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56415c8234d0 msgr2=0x56415cbe9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.617+0000 7ff2264fb640 1 --2- 172.21.15.90:0/2005603700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56415c8234d0 0x56415cbe9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.617+0000 7ff2264fb640 1 -- 172.21.15.90:0/2005603700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c82b6b0 msgr2=0x56415cc03e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.617+0000 7ff2264fb640 1 --2- 172.21.15.90:0/2005603700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c82b6b0 0x56415cc03e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.617+0000 7ff2264fb640 1 -- 172.21.15.90:0/2005603700 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56415cbed5a0 con 0x56415c84c390 2024-04-25T11:44:16.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.617+0000 7ff2254f9640 1 --2- 172.21.15.90:0/2005603700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c82b6b0 0x56415cc03e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:16.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.617+0000 7ff2264fb640 1 --2- 172.21.15.90:0/2005603700 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56415c84c390 0x56415cbf2430 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff2100045e0 tx=0x7ff210004640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f8ae508115aa480 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.617+0000 7ff2077fe640 1 -- 172.21.15.90:0/2005603700 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21001e470 con 0x56415c84c390 2024-04-25T11:44:16.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.617+0000 7ff2077fe640 1 -- 172.21.15.90:0/2005603700 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff21001e610 con 0x56415c84c390 2024-04-25T11:44:16.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.617+0000 7ff2077fe640 1 -- 172.21.15.90:0/2005603700 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21001f720 con 0x56415c84c390 2024-04-25T11:44:16.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.618+0000 7ff22ee0f740 1 -- 172.21.15.90:0/2005603700 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56415c84c390 msgr2=0x56415cbf2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.618+0000 7ff22ee0f740 1 --2- 172.21.15.90:0/2005603700 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56415c84c390 0x56415cbf2430 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff2100045e0 tx=0x7ff210004640 comp rx=0 tx=0).stop 2024-04-25T11:44:16.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.618+0000 7ff22ee0f740 1 -- 172.21.15.90:0/2005603700 shutdown_connections 2024-04-25T11:44:16.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.618+0000 7ff22ee0f740 1 --2- 172.21.15.90:0/2005603700 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56415c84c390 0x56415cbf2430 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.618+0000 7ff22ee0f740 1 --2- 172.21.15.90:0/2005603700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c82b6b0 0x56415cc03e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.618+0000 7ff22ee0f740 1 --2- 172.21.15.90:0/2005603700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56415c8234d0 0x56415cbe9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.618+0000 7ff22ee0f740 1 -- 172.21.15.90:0/2005603700 >> 172.21.15.90:0/2005603700 conn(0x56415c8404e0 msgr2=0x56415cbe97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.618+0000 7ff22ee0f740 1 -- 172.21.15.90:0/2005603700 shutdown_connections 2024-04-25T11:44:16.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.618+0000 7ff22ee0f740 1 -- 172.21.15.90:0/2005603700 wait complete. 2024-04-25T11:44:16.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.618+0000 7ff22ee0f740 1 -- start start 2024-04-25T11:44:16.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.619+0000 7ff22ee0f740 1 Processor -- start 2024-04-25T11:44:16.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.619+0000 7ff22ee0f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56415c8234d0 0x56415cc00440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.619+0000 7ff22ee0f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56415c82b6b0 0x56415cbfb790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.619+0000 7ff22ee0f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c84c390 0x56415cbfbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.619+0000 7ff22ee0f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff2201b11b0 con 0x56415c84c390 2024-04-25T11:44:16.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.619+0000 7ff22ee0f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff2201b0ae0 con 0x56415c8234d0 2024-04-25T11:44:16.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.619+0000 7ff22ee0f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff2201add40 con 0x56415c82b6b0 2024-04-25T11:44:16.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.619+0000 7ff22ee0f740 10 client.0 _ll_register_callbacks cb 0x7ff210012f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:16.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.619+0000 7ff2264fb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c84c390 0x56415cbfbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.619+0000 7ff225cfa640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56415c8234d0 0x56415cc00440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.619+0000 7ff2254f9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56415c82b6b0 0x56415cbfb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.619+0000 7ff2264fb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c84c390 0x56415cbfbcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47540/0 (socket says 172.21.15.90:47540) 2024-04-25T11:44:16.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.619+0000 7ff225cfa640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56415c8234d0 0x56415cc00440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:57048/0 (socket says 172.21.15.90:57048) 2024-04-25T11:44:16.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.619+0000 7ff2254f9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56415c82b6b0 0x56415cbfb790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:54198/0 (socket says 172.21.15.90:54198) 2024-04-25T11:44:16.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.619+0000 7ff2264fb640 1 -- 172.21.15.90:0/3580359462 learned_addr learned my addr 172.21.15.90:0/3580359462 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:16.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.620+0000 7ff225cfa640 1 -- 172.21.15.90:0/3580359462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56415c82b6b0 msgr2=0x56415cbfb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.620+0000 7ff225cfa640 1 --2- 172.21.15.90:0/3580359462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56415c82b6b0 0x56415cbfb790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.620+0000 7ff225cfa640 1 -- 172.21.15.90:0/3580359462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c84c390 msgr2=0x56415cbfbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.620+0000 7ff225cfa640 1 --2- 172.21.15.90:0/3580359462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c84c390 0x56415cbfbcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.620+0000 7ff225cfa640 1 -- 172.21.15.90:0/3580359462 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56415cbfc210 con 0x56415c8234d0 2024-04-25T11:44:16.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.620+0000 7ff2264fb640 1 --2- 172.21.15.90:0/3580359462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c84c390 0x56415cbfbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:16.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.620+0000 7ff225cfa640 1 --2- 172.21.15.90:0/3580359462 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56415c8234d0 0x56415cc00440 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff218004650 tx=0x7ff218009da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5727e6081c3ff64f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.620+0000 7ff2057fa640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:44:16.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.620+0000 7ff2057fa640 1 -- 172.21.15.90:0/3580359462 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21802f070 con 0x56415c8234d0 2024-04-25T11:44:16.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.620+0000 7ff22ee0f740 1 -- 172.21.15.90:0/3580359462 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff21001e610 con 0x56415c8234d0 2024-04-25T11:44:16.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.620+0000 7ff2057fa640 1 -- 172.21.15.90:0/3580359462 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7ff21802b2b0 con 0x56415c8234d0 2024-04-25T11:44:16.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.621+0000 7ff2057fa640 1 -- 172.21.15.90:0/3580359462 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff21803b050 con 0x56415c8234d0 2024-04-25T11:44:16.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.621+0000 7ff22ee0f740 10 client.27266 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:16.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.621+0000 7ff22ee0f740 1 -- 172.21.15.90:0/3580359462 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56415cc72700 con 0x56415c8234d0 2024-04-25T11:44:16.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.621+0000 7ff2057fa640 1 -- 172.21.15.90:0/3580359462 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff21001e610 con 0x56415c8234d0 2024-04-25T11:44:16.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.621+0000 7ff2057fa640 1 -- 172.21.15.90:0/3580359462 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7ff21802bce0 con 0x56415c8234d0 2024-04-25T11:44:16.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.621+0000 7ff22ee0f740 10 client.27266 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:16.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.621+0000 7ff22ee0f740 10 client.27266 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:44:16.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.621+0000 7ff22ee0f740 20 client.27266 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:16.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.621+0000 7ff22ee0f740 1 --2- 172.21.15.90:0/3580359462 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x56415c7e03a0 0x56415cc74ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.621+0000 7ff22ee0f740 4 client.27266 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:44:16.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.621+0000 7ff22ee0f740 1 -- 172.21.15.90:0/3580359462 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56415cc04e50 con 0x56415c7e03a0 2024-04-25T11:44:16.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.621+0000 7ff2254f9640 1 --2- 172.21.15.90:0/3580359462 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x56415c7e03a0 0x56415cc74ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.622+0000 7ff2057fa640 10 client.27266 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:44:16.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.622+0000 7ff2254f9640 1 --2- 172.21.15.90:0/3580359462 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x56415c7e03a0 0x56415cc74ca0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2c42e8a888662ae8 server_cookie=bfbba20d702c9243 in_seq=0 out_seq=0 2024-04-25T11:44:16.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.622+0000 7ff2057fa640 1 -- 172.21.15.90:0/3580359462 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56415cc04e50 con 0x56415c7e03a0 2024-04-25T11:44:16.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.622+0000 7ff2057fa640 10 client.27266 handle_command_reply: tid=0 2024-04-25T11:44:16.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.622+0000 7ff22ee0f740 1 client.27266 shutdown 2024-04-25T11:44:16.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.622+0000 7ff22ee0f740 1 -- 172.21.15.90:0/3580359462 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56415c8234d0 msgr2=0x56415cc00440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.622+0000 7ff22ee0f740 1 --2- 172.21.15.90:0/3580359462 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56415c8234d0 0x56415cc00440 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff218004650 tx=0x7ff218009da0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.622+0000 7ff22ee0f740 1 -- 172.21.15.90:0/3580359462 shutdown_connections 2024-04-25T11:44:16.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.622+0000 7ff22ee0f740 1 --2- 172.21.15.90:0/3580359462 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x56415c7e03a0 0x56415cc74ca0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.622+0000 7ff22ee0f740 1 --2- 172.21.15.90:0/3580359462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56415c84c390 0x56415cbfbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.622+0000 7ff22ee0f740 1 --2- 172.21.15.90:0/3580359462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56415c82b6b0 0x56415cbfb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.622+0000 7ff22ee0f740 1 --2- 172.21.15.90:0/3580359462 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56415c8234d0 0x56415cc00440 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.622+0000 7ff2057fa640 0 client.27266 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:44:16.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.622+0000 7ff22ee0f740 1 -- 172.21.15.90:0/3580359462 >> 172.21.15.90:0/3580359462 conn(0x56415c8404e0 msgr2=0x56415cbe9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.622+0000 7ff22ee0f740 1 -- 172.21.15.90:0/3580359462 shutdown_connections 2024-04-25T11:44:16.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.622+0000 7ff22ee0f740 1 -- 172.21.15.90:0/3580359462 wait complete. 2024-04-25T11:44:16.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.622+0000 7ff22ee0f740 20 client.27266 trim_cache size 0 max 16384 2024-04-25T11:44:16.616 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:44:16.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.623+0000 7ff227f5e640 1 -- 172.21.15.90:0/1974054253 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff2080afc60 msgr2=0x7ff2080b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.623+0000 7ff227f5e640 1 --2- 172.21.15.90:0/1974054253 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff2080afc60 0x7ff2080b2170 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x56415cbea980 tx=0x7ff210005f90 comp rx=0 tx=0).stop 2024-04-25T11:44:16.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.623+0000 7ff227f5e640 1 -- 172.21.15.90:0/1974054253 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff22019a510 msgr2=0x7ff22019e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.623+0000 7ff227f5e640 1 --2- 172.21.15.90:0/1974054253 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff22019a510 0x7ff22019e7f0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7ff21c033040 tx=0x7ff21c0316f0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.623+0000 7ff227f5e640 1 -- 172.21.15.90:0/1974054253 shutdown_connections 2024-04-25T11:44:16.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.623+0000 7ff227f5e640 1 --2- 172.21.15.90:0/1974054253 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff2080afc60 0x7ff2080b2170 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.623+0000 7ff227f5e640 1 --2- 172.21.15.90:0/1974054253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff22019ed30 0x7ff22019f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.623+0000 7ff227f5e640 1 --2- 172.21.15.90:0/1974054253 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff22019a510 0x7ff22019e7f0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.623+0000 7ff227f5e640 1 --2- 172.21.15.90:0/1974054253 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff220109b40 0x7ff220199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.623+0000 7ff227f5e640 1 -- 172.21.15.90:0/1974054253 >> 172.21.15.90:0/1974054253 conn(0x7ff220077920 msgr2=0x7ff22010fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.623+0000 7ff227f5e640 1 -- 172.21.15.90:0/1974054253 shutdown_connections 2024-04-25T11:44:16.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.623+0000 7ff227f5e640 1 -- 172.21.15.90:0/1974054253 wait complete. 2024-04-25T11:44:16.619 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:16 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/417597251' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:44:16.620 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T11:44:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.750+0000 7f092db46640 1 -- 172.21.15.90:0/3953695568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f092810b650 msgr2=0x7f092810da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.750+0000 7f092db46640 1 --2- 172.21.15.90:0/3953695568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f092810b650 0x7f092810da70 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f091c009900 tx=0x7f091c02a410 comp rx=0 tx=0).stop 2024-04-25T11:44:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.750+0000 7f092db46640 1 -- 172.21.15.90:0/3953695568 shutdown_connections 2024-04-25T11:44:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.750+0000 7f092db46640 1 --2- 172.21.15.90:0/3953695568 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f092810e040 0x7f0928110530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.750+0000 7f092db46640 1 --2- 172.21.15.90:0/3953695568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f092810b650 0x7f092810da70 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.750+0000 7f092db46640 1 --2- 172.21.15.90:0/3953695568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0928104bf0 0x7f0928105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.750+0000 7f092db46640 1 -- 172.21.15.90:0/3953695568 >> 172.21.15.90:0/3953695568 conn(0x7f0928078100 msgr2=0x7f0928078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.750+0000 7f092db46640 1 -- 172.21.15.90:0/3953695568 shutdown_connections 2024-04-25T11:44:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.750+0000 7f092db46640 1 -- 172.21.15.90:0/3953695568 wait complete. 2024-04-25T11:44:16.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.751+0000 7f092db46640 1 Processor -- start 2024-04-25T11:44:16.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.751+0000 7f092db46640 1 -- start start 2024-04-25T11:44:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.751+0000 7f092db46640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0928104bf0 0x7f092819a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.751+0000 7f092db46640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f092810b650 0x7f092819ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.751+0000 7f092db46640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f092810e040 0x7f092819f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.751+0000 7f092db46640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0928113ca0 con 0x7f092810b650 2024-04-25T11:44:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.751+0000 7f092db46640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0928113b20 con 0x7f0928104bf0 2024-04-25T11:44:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.751+0000 7f092db46640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0928113e20 con 0x7f092810e040 2024-04-25T11:44:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.751+0000 7f0926ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f092810b650 0x7f092819ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.751+0000 7f0926ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f092810b650 0x7f092819ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47544/0 (socket says 172.21.15.90:47544) 2024-04-25T11:44:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.751+0000 7f0926ffd640 1 -- 172.21.15.90:0/398648289 learned_addr learned my addr 172.21.15.90:0/398648289 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.751+0000 7f09277fe640 1 --2- 172.21.15.90:0/398648289 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0928104bf0 0x7f092819a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.751+0000 7f0927fff640 1 --2- 172.21.15.90:0/398648289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f092810e040 0x7f092819f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.751+0000 7f0926ffd640 1 -- 172.21.15.90:0/398648289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f092810e040 msgr2=0x7f092819f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.751+0000 7f0926ffd640 1 --2- 172.21.15.90:0/398648289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f092810e040 0x7f092819f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.751+0000 7f0926ffd640 1 -- 172.21.15.90:0/398648289 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0928104bf0 msgr2=0x7f092819a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.751+0000 7f0926ffd640 1 --2- 172.21.15.90:0/398648289 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0928104bf0 0x7f092819a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.751+0000 7f0926ffd640 1 -- 172.21.15.90:0/398648289 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f092819f640 con 0x7f092810b650 2024-04-25T11:44:16.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.752+0000 7f0926ffd640 1 --2- 172.21.15.90:0/398648289 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f092810b650 0x7f092819ad40 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f091c005940 tx=0x7f091c0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.752+0000 7f0924ff9640 1 -- 172.21.15.90:0/398648289 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f091c03f030 con 0x7f092810b650 2024-04-25T11:44:16.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.752+0000 7f0924ff9640 1 -- 172.21.15.90:0/398648289 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f091c0044c0 con 0x7f092810b650 2024-04-25T11:44:16.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.752+0000 7f092db46640 1 -- 172.21.15.90:0/398648289 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f092819f8d0 con 0x7f092810b650 2024-04-25T11:44:16.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.752+0000 7f0924ff9640 1 -- 172.21.15.90:0/398648289 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f091c035070 con 0x7f092810b650 2024-04-25T11:44:16.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.752+0000 7f092db46640 1 -- 172.21.15.90:0/398648289 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f092819fdc0 con 0x7f092810b650 2024-04-25T11:44:16.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.755+0000 7f09349f8740 1 Processor -- start 2024-04-25T11:44:16.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.755+0000 7f09349f8740 1 -- start start 2024-04-25T11:44:16.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.756+0000 7f09349f8740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5571cd8b1390 0x5571cdc52810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.756+0000 7f09349f8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8884d0 0x5571cdc52d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.756+0000 7f09349f8740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5571cd8906b0 0x5571cdc5f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.756+0000 7f09349f8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5571cdbb7d80 con 0x5571cd8884d0 2024-04-25T11:44:16.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.756+0000 7f09349f8740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5571cdb71a20 con 0x5571cd8b1390 2024-04-25T11:44:16.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.756+0000 7f09349f8740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5571cdbb3d40 con 0x5571cd8906b0 2024-04-25T11:44:16.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.756+0000 7f0924ff9640 1 -- 172.21.15.90:0/398648289 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f091c042030 con 0x7f092810b650 2024-04-25T11:44:16.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.756+0000 7f0927fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5571cd8b1390 0x5571cdc52810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.756+0000 7f09277fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8884d0 0x5571cdc52d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.756+0000 7f0927fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5571cd8b1390 0x5571cdc52810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:57080/0 (socket says 172.21.15.90:57080) 2024-04-25T11:44:16.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.756+0000 7f0927fff640 1 -- 172.21.15.90:0/3688501287 learned_addr learned my addr 172.21.15.90:0/3688501287 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:16.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.756+0000 7f0927fff640 1 -- 172.21.15.90:0/3688501287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5571cd8906b0 msgr2=0x5571cdc5f6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:44:16.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.756+0000 7f0927fff640 1 --2- 172.21.15.90:0/3688501287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5571cd8906b0 0x5571cdc5f6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.756+0000 7f0927fff640 1 -- 172.21.15.90:0/3688501287 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8884d0 msgr2=0x5571cdc52d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.756+0000 7f0927fff640 1 --2- 172.21.15.90:0/3688501287 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8884d0 0x5571cdc52d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.756+0000 7f0927fff640 1 -- 172.21.15.90:0/3688501287 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571cdb7cc30 con 0x5571cd8b1390 2024-04-25T11:44:16.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.757+0000 7f09277fe640 1 --2- 172.21.15.90:0/3688501287 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8884d0 0x5571cdc52d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:16.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.757+0000 7f0927fff640 1 --2- 172.21.15.90:0/3688501287 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5571cd8b1390 0x5571cdc52810 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0920009f50 tx=0x7f09200118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2db6ece352199674 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.757+0000 7f090dffb640 1 -- 172.21.15.90:0/3688501287 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0920004d20 con 0x5571cd8b1390 2024-04-25T11:44:16.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.757+0000 7f0924ff9640 1 --2- 172.21.15.90:0/398648289 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f09100afc60 0x7f09100b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.757+0000 7f090dffb640 1 -- 172.21.15.90:0/3688501287 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0920016a80 con 0x5571cd8b1390 2024-04-25T11:44:16.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.757+0000 7f090dffb640 1 -- 172.21.15.90:0/3688501287 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0920005650 con 0x5571cd8b1390 2024-04-25T11:44:16.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.757+0000 7f09277fe640 1 --2- 172.21.15.90:0/398648289 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f09100afc60 0x7f09100b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.757+0000 7f0924ff9640 1 -- 172.21.15.90:0/398648289 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7f091c0ff450 con 0x7f092810b650 2024-04-25T11:44:16.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.758+0000 7f09349f8740 1 -- 172.21.15.90:0/3688501287 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5571cd8b1390 msgr2=0x5571cdc52810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.758+0000 7f09277fe640 1 --2- 172.21.15.90:0/398648289 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f09100afc60 0x7f09100b2170 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x5571cdc53fe0 tx=0x7f0918005f90 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.758+0000 7f09349f8740 1 --2- 172.21.15.90:0/3688501287 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5571cd8b1390 0x5571cdc52810 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0920009f50 tx=0x7f09200118d0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.759+0000 7f09349f8740 1 -- 172.21.15.90:0/3688501287 shutdown_connections 2024-04-25T11:44:16.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.759+0000 7f09349f8740 1 --2- 172.21.15.90:0/3688501287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5571cd8906b0 0x5571cdc5f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.759+0000 7f09349f8740 1 --2- 172.21.15.90:0/3688501287 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8884d0 0x5571cdc52d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.759+0000 7f09349f8740 1 --2- 172.21.15.90:0/3688501287 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5571cd8b1390 0x5571cdc52810 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.759+0000 7f09349f8740 1 -- 172.21.15.90:0/3688501287 >> 172.21.15.90:0/3688501287 conn(0x5571cd8a54e0 msgr2=0x5571cdc4efc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.759+0000 7f09349f8740 1 -- 172.21.15.90:0/3688501287 shutdown_connections 2024-04-25T11:44:16.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.759+0000 7f09349f8740 1 -- 172.21.15.90:0/3688501287 wait complete. 2024-04-25T11:44:16.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.759+0000 7f09349f8740 1 -- start start 2024-04-25T11:44:16.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.760+0000 7f09349f8740 1 Processor -- start 2024-04-25T11:44:16.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.760+0000 7f09349f8740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5571cd8884d0 0x5571cdc5d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.760+0000 7f09349f8740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5571cd8906b0 0x5571cdc5d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.760+0000 7f09349f8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8b1390 0x5571cdc69210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.760+0000 7f09349f8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5571cdc5fd90 con 0x5571cd8b1390 2024-04-25T11:44:16.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.760+0000 7f09349f8740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5571cdb3edc0 con 0x5571cd8906b0 2024-04-25T11:44:16.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.760+0000 7f09349f8740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5571cdc5ff10 con 0x5571cd8884d0 2024-04-25T11:44:16.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.760+0000 7f09349f8740 10 client.0 _ll_register_callbacks cb 0x5571cdb5a600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:16.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.760+0000 7f0927fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8b1390 0x5571cdc69210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.760+0000 7f0927fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8b1390 0x5571cdc69210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47562/0 (socket says 172.21.15.90:47562) 2024-04-25T11:44:16.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.760+0000 7f09277fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5571cd8884d0 0x5571cdc5d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.760+0000 7f0927fff640 1 -- 172.21.15.90:0/147684997 learned_addr learned my addr 172.21.15.90:0/147684997 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:16.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.760+0000 7f0926ffd640 1 --2- 172.21.15.90:0/147684997 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5571cd8906b0 0x5571cdc5d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.761+0000 7f0926ffd640 1 -- 172.21.15.90:0/147684997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5571cd8884d0 msgr2=0x5571cdc5d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.761+0000 7f0926ffd640 1 --2- 172.21.15.90:0/147684997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5571cd8884d0 0x5571cdc5d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.761+0000 7f0926ffd640 1 -- 172.21.15.90:0/147684997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8b1390 msgr2=0x5571cdc69210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.761+0000 7f0926ffd640 1 --2- 172.21.15.90:0/147684997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8b1390 0x5571cdc69210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.761+0000 7f0926ffd640 1 -- 172.21.15.90:0/147684997 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571cd8c93f0 con 0x5571cd8906b0 2024-04-25T11:44:16.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.761+0000 7f0927fff640 1 --2- 172.21.15.90:0/147684997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8b1390 0x5571cdc69210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:16.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.761+0000 7f09277fe640 1 --2- 172.21.15.90:0/147684997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5571cd8884d0 0x5571cdc5d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:16.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.761+0000 7f0926ffd640 1 --2- 172.21.15.90:0/147684997 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5571cd8906b0 0x5571cdc5d840 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f091c0fcaf0 tx=0x7f091c0c35e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54a21063e53d4651 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.761+0000 7f0906ffd640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:44:16.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.761+0000 7f0906ffd640 1 -- 172.21.15.90:0/147684997 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f091c049030 con 0x5571cd8906b0 2024-04-25T11:44:16.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.761+0000 7f0906ffd640 1 -- 172.21.15.90:0/147684997 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7f091c0c7850 con 0x5571cd8906b0 2024-04-25T11:44:16.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.761+0000 7f09349f8740 1 -- 172.21.15.90:0/147684997 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0920016a80 con 0x5571cd8906b0 2024-04-25T11:44:16.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.761+0000 7f0906ffd640 1 -- 172.21.15.90:0/147684997 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f091c0b00f0 con 0x5571cd8906b0 2024-04-25T11:44:16.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.761+0000 7f09349f8740 10 client.27278 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:16.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.761+0000 7f09349f8740 1 -- 172.21.15.90:0/147684997 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5571cd9937d0 con 0x5571cd8906b0 2024-04-25T11:44:16.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.761+0000 7f0906ffd640 1 -- 172.21.15.90:0/147684997 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0920016a80 con 0x5571cd8906b0 2024-04-25T11:44:16.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.762+0000 7f0906ffd640 1 -- 172.21.15.90:0/147684997 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f091c0cff10 con 0x5571cd8906b0 2024-04-25T11:44:16.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.762+0000 7f09349f8740 10 client.27278 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:16.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.762+0000 7f09349f8740 10 client.27278 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:44:16.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.762+0000 7f09349f8740 20 client.27278 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:16.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.762+0000 7f09349f8740 1 --2- 172.21.15.90:0/147684997 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5571cd86ff60 0x5571cdc6f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.762+0000 7f09349f8740 4 client.27278 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:44:16.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.762+0000 7f09349f8740 1 -- 172.21.15.90:0/147684997 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5571cdbc5990 con 0x5571cd86ff60 2024-04-25T11:44:16.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.762+0000 7f09277fe640 1 --2- 172.21.15.90:0/147684997 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5571cd86ff60 0x5571cdc6f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.762+0000 7f09277fe640 1 --2- 172.21.15.90:0/147684997 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5571cd86ff60 0x5571cdc6f570 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1dee9b3cc06976fd server_cookie=5f7e16ed202f91b4 in_seq=0 out_seq=0 2024-04-25T11:44:16.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.762+0000 7f0906ffd640 10 client.27278 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:16.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.763+0000 7f0906ffd640 1 -- 172.21.15.90:0/147684997 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5571cdbc5990 con 0x5571cd86ff60 2024-04-25T11:44:16.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.763+0000 7f0906ffd640 10 client.27278 handle_command_reply: tid=0 2024-04-25T11:44:16.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.763+0000 7f09349f8740 1 client.27278 shutdown 2024-04-25T11:44:16.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.764+0000 7f09349f8740 1 -- 172.21.15.90:0/147684997 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5571cd8906b0 msgr2=0x5571cdc5d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.764+0000 7f09349f8740 1 --2- 172.21.15.90:0/147684997 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5571cd8906b0 0x5571cdc5d840 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f091c0fcaf0 tx=0x7f091c0c35e0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.764+0000 7f09349f8740 1 -- 172.21.15.90:0/147684997 shutdown_connections 2024-04-25T11:44:16.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.764+0000 7f09349f8740 1 --2- 172.21.15.90:0/147684997 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5571cd86ff60 0x5571cdc6f570 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.764+0000 7f09349f8740 1 --2- 172.21.15.90:0/147684997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8b1390 0x5571cdc69210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.764+0000 7f0906ffd640 0 client.27278 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:16.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.764+0000 7f09349f8740 1 --2- 172.21.15.90:0/147684997 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5571cd8906b0 0x5571cdc5d840 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.764+0000 7f09349f8740 1 --2- 172.21.15.90:0/147684997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5571cd8884d0 0x5571cdc5d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.764+0000 7f09349f8740 1 -- 172.21.15.90:0/147684997 >> 172.21.15.90:0/147684997 conn(0x5571cd8a54e0 msgr2=0x5571cdc54530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.764+0000 7f09349f8740 1 -- 172.21.15.90:0/147684997 shutdown_connections 2024-04-25T11:44:16.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.764+0000 7f09349f8740 1 -- 172.21.15.90:0/147684997 wait complete. 2024-04-25T11:44:16.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.764+0000 7f09349f8740 20 client.27278 trim_cache size 0 max 16384 2024-04-25T11:44:16.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.777+0000 7f09349f8740 1 Processor -- start 2024-04-25T11:44:16.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.777+0000 7f09349f8740 1 -- start start 2024-04-25T11:44:16.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.777+0000 7f09349f8740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5571cd8884d0 0x5571cdc5c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.777+0000 7f09349f8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8906b0 0x5571cdc4f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.777+0000 7f09349f8740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5571cd8b1390 0x5571cdc68e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.777+0000 7f09349f8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f09281adb20 con 0x5571cd8906b0 2024-04-25T11:44:16.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.777+0000 7f09349f8740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f09281af660 con 0x5571cd8884d0 2024-04-25T11:44:16.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.777+0000 7f09349f8740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f09281ae1f0 con 0x5571cd8b1390 2024-04-25T11:44:16.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.778+0000 7f0926ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8906b0 0x5571cdc4f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.778+0000 7f09277fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5571cd8884d0 0x5571cdc5c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.778+0000 7f0926ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8906b0 0x5571cdc4f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47570/0 (socket says 172.21.15.90:47570) 2024-04-25T11:44:16.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.778+0000 7f09277fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5571cd8884d0 0x5571cdc5c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:57096/0 (socket says 172.21.15.90:57096) 2024-04-25T11:44:16.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.778+0000 7f0926ffd640 1 -- 172.21.15.90:0/350218097 learned_addr learned my addr 172.21.15.90:0/350218097 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:16.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.778+0000 7f0927fff640 1 --2- 172.21.15.90:0/350218097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5571cd8b1390 0x5571cdc68e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.778+0000 7f0926ffd640 1 -- 172.21.15.90:0/350218097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5571cd8b1390 msgr2=0x5571cdc68e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.778+0000 7f0926ffd640 1 --2- 172.21.15.90:0/350218097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5571cd8b1390 0x5571cdc68e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.778+0000 7f0926ffd640 1 -- 172.21.15.90:0/350218097 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5571cd8884d0 msgr2=0x5571cdc5c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.778+0000 7f0926ffd640 1 --2- 172.21.15.90:0/350218097 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5571cd8884d0 0x5571cdc5c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.778+0000 7f0926ffd640 1 -- 172.21.15.90:0/350218097 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571cdc5fc70 con 0x5571cd8906b0 2024-04-25T11:44:16.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.778+0000 7f09277fe640 1 --2- 172.21.15.90:0/350218097 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5571cd8884d0 0x5571cdc5c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:16.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.778+0000 7f0926ffd640 1 --2- 172.21.15.90:0/350218097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8906b0 0x5571cdc4f0c0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f091c0c3670 tx=0x7f091c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14122ec51b30aa2f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.778+0000 7f090d7fa640 1 -- 172.21.15.90:0/350218097 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f091c049030 con 0x5571cd8906b0 2024-04-25T11:44:16.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.778+0000 7f090d7fa640 1 -- 172.21.15.90:0/350218097 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f091c0c75f0 con 0x5571cd8906b0 2024-04-25T11:44:16.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.778+0000 7f090d7fa640 1 -- 172.21.15.90:0/350218097 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f091c0c7850 con 0x5571cd8906b0 2024-04-25T11:44:16.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.779+0000 7f09349f8740 1 -- 172.21.15.90:0/350218097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8906b0 msgr2=0x5571cdc4f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.779+0000 7f09349f8740 1 --2- 172.21.15.90:0/350218097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8906b0 0x5571cdc4f0c0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f091c0c3670 tx=0x7f091c02a410 comp rx=0 tx=0).stop 2024-04-25T11:44:16.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.779+0000 7f09349f8740 1 -- 172.21.15.90:0/350218097 shutdown_connections 2024-04-25T11:44:16.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.779+0000 7f09349f8740 1 --2- 172.21.15.90:0/350218097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5571cd8b1390 0x5571cdc68e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.779+0000 7f09349f8740 1 --2- 172.21.15.90:0/350218097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8906b0 0x5571cdc4f0c0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.779+0000 7f09349f8740 1 --2- 172.21.15.90:0/350218097 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5571cd8884d0 0x5571cdc5c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.779+0000 7f09349f8740 1 -- 172.21.15.90:0/350218097 >> 172.21.15.90:0/350218097 conn(0x5571cd8a54e0 msgr2=0x5571cdc5e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.779+0000 7f09349f8740 1 -- 172.21.15.90:0/350218097 shutdown_connections 2024-04-25T11:44:16.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.779+0000 7f09349f8740 1 -- 172.21.15.90:0/350218097 wait complete. 2024-04-25T11:44:16.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.779+0000 7f09349f8740 1 -- start start 2024-04-25T11:44:16.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.780+0000 7f09349f8740 1 Processor -- start 2024-04-25T11:44:16.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.780+0000 7f09349f8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8884d0 0x5571cdc601b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.780+0000 7f09349f8740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5571cd8906b0 0x5571cdc606f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.780+0000 7f09349f8740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5571cd8b1390 0x5571cdc60c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.780+0000 7f09349f8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f09281aef90 con 0x5571cd8884d0 2024-04-25T11:44:16.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.780+0000 7f09349f8740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f09281ae8c0 con 0x5571cd8906b0 2024-04-25T11:44:16.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.780+0000 7f09349f8740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f09281abb20 con 0x5571cd8b1390 2024-04-25T11:44:16.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.780+0000 7f09349f8740 10 client.0 _ll_register_callbacks cb 0x7f091c005ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:16.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.780+0000 7f09277fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8884d0 0x5571cdc601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.780+0000 7f09277fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8884d0 0x5571cdc601b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47576/0 (socket says 172.21.15.90:47576) 2024-04-25T11:44:16.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.780+0000 7f09277fe640 1 -- 172.21.15.90:0/3673395026 learned_addr learned my addr 172.21.15.90:0/3673395026 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:16.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.780+0000 7f0926ffd640 1 --2- 172.21.15.90:0/3673395026 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5571cd8906b0 0x5571cdc606f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.780+0000 7f0927fff640 1 --2- 172.21.15.90:0/3673395026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5571cd8b1390 0x5571cdc60c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.780+0000 7f09277fe640 1 -- 172.21.15.90:0/3673395026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5571cd8b1390 msgr2=0x5571cdc60c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.780+0000 7f09277fe640 1 --2- 172.21.15.90:0/3673395026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5571cd8b1390 0x5571cdc60c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.780+0000 7f09277fe640 1 -- 172.21.15.90:0/3673395026 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5571cd8906b0 msgr2=0x5571cdc606f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:16.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.780+0000 7f09277fe640 1 --2- 172.21.15.90:0/3673395026 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5571cd8906b0 0x5571cdc606f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.780+0000 7f09277fe640 1 -- 172.21.15.90:0/3673395026 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571cdcd57c0 con 0x5571cd8884d0 2024-04-25T11:44:16.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.780+0000 7f0927fff640 1 --2- 172.21.15.90:0/3673395026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5571cd8b1390 0x5571cdc60c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:44:16.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.780+0000 7f09277fe640 1 --2- 172.21.15.90:0/3673395026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8884d0 0x5571cdc601b0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f091801e8a0 tx=0x7f09180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fca899e64f81456e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:16.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.780+0000 7f0906ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:16.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.781+0000 7f0906ffd640 1 -- 172.21.15.90:0/3673395026 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f091801e950 con 0x5571cd8884d0 2024-04-25T11:44:16.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.781+0000 7f0906ffd640 1 -- 172.21.15.90:0/3673395026 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7f091801eaf0 con 0x5571cd8884d0 2024-04-25T11:44:16.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.781+0000 7f09349f8740 1 -- 172.21.15.90:0/3673395026 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f091c0c75f0 con 0x5571cd8884d0 2024-04-25T11:44:16.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.781+0000 7f0906ffd640 1 -- 172.21.15.90:0/3673395026 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f091801ecf0 con 0x5571cd8884d0 2024-04-25T11:44:16.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.781+0000 7f09349f8740 10 client.18522 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:16.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.781+0000 7f09349f8740 1 -- 172.21.15.90:0/3673395026 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5571cdcd69b0 con 0x5571cd8884d0 2024-04-25T11:44:16.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.781+0000 7f0906ffd640 1 -- 172.21.15.90:0/3673395026 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f091c0c75f0 con 0x5571cd8884d0 2024-04-25T11:44:16.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.781+0000 7f0906ffd640 1 -- 172.21.15.90:0/3673395026 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f0918037530 con 0x5571cd8884d0 2024-04-25T11:44:16.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.781+0000 7f09349f8740 10 client.18522 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:16.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.781+0000 7f09349f8740 10 client.18522 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:44:16.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.781+0000 7f09349f8740 20 client.18522 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:16.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.781+0000 7f09349f8740 1 --2- 172.21.15.90:0/3673395026 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5571cd8453a0 0x5571cdcd9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:16.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.781+0000 7f09349f8740 4 client.18522 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:44:16.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.781+0000 7f09349f8740 1 -- 172.21.15.90:0/3673395026 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5571cdbbf800 con 0x5571cd8453a0 2024-04-25T11:44:16.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.781+0000 7f0926ffd640 1 --2- 172.21.15.90:0/3673395026 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5571cd8453a0 0x5571cdcd9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:16.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.782+0000 7f0906ffd640 10 client.18522 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:16.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.782+0000 7f0926ffd640 1 --2- 172.21.15.90:0/3673395026 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5571cd8453a0 0x5571cdcd9170 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9fbc6d67989f90e server_cookie=e6a67368d92f946e in_seq=0 out_seq=0 2024-04-25T11:44:16.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.782+0000 7f0906ffd640 1 -- 172.21.15.90:0/3673395026 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5571cdbbf800 con 0x5571cd8453a0 2024-04-25T11:44:16.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.782+0000 7f0906ffd640 10 client.18522 handle_command_reply: tid=0 2024-04-25T11:44:16.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.782+0000 7f09349f8740 1 client.18522 shutdown 2024-04-25T11:44:16.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.782+0000 7f09349f8740 1 -- 172.21.15.90:0/3673395026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8884d0 msgr2=0x5571cdc601b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.782+0000 7f09349f8740 1 --2- 172.21.15.90:0/3673395026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8884d0 0x5571cdc601b0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f091801e8a0 tx=0x7f09180046c0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.782+0000 7f09349f8740 1 -- 172.21.15.90:0/3673395026 shutdown_connections 2024-04-25T11:44:16.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.782+0000 7f09349f8740 1 --2- 172.21.15.90:0/3673395026 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5571cd8453a0 0x5571cdcd9170 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.782+0000 7f09349f8740 1 --2- 172.21.15.90:0/3673395026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5571cd8b1390 0x5571cdc60c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.782+0000 7f09349f8740 1 --2- 172.21.15.90:0/3673395026 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5571cd8906b0 0x5571cdc606f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.782+0000 7f0906ffd640 0 client.18522 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:16.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.782+0000 7f09349f8740 1 --2- 172.21.15.90:0/3673395026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5571cd8884d0 0x5571cdc601b0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.782+0000 7f09349f8740 1 -- 172.21.15.90:0/3673395026 >> 172.21.15.90:0/3673395026 conn(0x5571cd8a54e0 msgr2=0x5571cdc5c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.782+0000 7f09349f8740 1 -- 172.21.15.90:0/3673395026 shutdown_connections 2024-04-25T11:44:16.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.782+0000 7f09349f8740 1 -- 172.21.15.90:0/3673395026 wait complete. 2024-04-25T11:44:16.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.782+0000 7f09349f8740 20 client.18522 trim_cache size 0 max 16384 2024-04-25T11:44:16.776 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:44:16.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.783+0000 7f092db46640 1 -- 172.21.15.90:0/398648289 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f09100afc60 msgr2=0x7f09100b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.783+0000 7f092db46640 1 --2- 172.21.15.90:0/398648289 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f09100afc60 0x7f09100b2170 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x5571cdc53fe0 tx=0x7f0918005f90 comp rx=0 tx=0).stop 2024-04-25T11:44:16.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.783+0000 7f092db46640 1 -- 172.21.15.90:0/398648289 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f092810b650 msgr2=0x7f092819ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:16.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.783+0000 7f092db46640 1 --2- 172.21.15.90:0/398648289 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f092810b650 0x7f092819ad40 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f091c005940 tx=0x7f091c0040c0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.783+0000 7f092db46640 1 -- 172.21.15.90:0/398648289 shutdown_connections 2024-04-25T11:44:16.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.783+0000 7f092db46640 1 --2- 172.21.15.90:0/398648289 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f09100afc60 0x7f09100b2170 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.783+0000 7f092db46640 1 --2- 172.21.15.90:0/398648289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f092810e040 0x7f092819f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.783+0000 7f092db46640 1 --2- 172.21.15.90:0/398648289 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f092810b650 0x7f092819ad40 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.783+0000 7f092db46640 1 --2- 172.21.15.90:0/398648289 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0928104bf0 0x7f092819a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:16.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.783+0000 7f092db46640 1 -- 172.21.15.90:0/398648289 >> 172.21.15.90:0/398648289 conn(0x7f0928078100 msgr2=0x7f0928101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:16.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.783+0000 7f092db46640 1 -- 172.21.15.90:0/398648289 shutdown_connections 2024-04-25T11:44:16.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:16.783+0000 7f092db46640 1 -- 172.21.15.90:0/398648289 wait complete. 2024-04-25T11:44:16.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:16 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/417597251' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:44:17.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:16 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/417597251' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:44:17.782 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T11:44:17.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:44:17.782 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T11:44:17.803 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:17 smithi090 ceph-mon[29575]: pgmap v284: 97 pgs: 18 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 48 active+clean; 44 GiB data, 138 GiB used, 935 GiB / 1.0 TiB avail; 7.4 KiB/s rd, 3.7 KiB/s wr, 10 op/s 2024-04-25T11:44:17.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:17 smithi156 ceph-mon[33045]: pgmap v284: 97 pgs: 18 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 48 active+clean; 44 GiB data, 138 GiB used, 935 GiB / 1.0 TiB avail; 7.4 KiB/s rd, 3.7 KiB/s wr, 10 op/s 2024-04-25T11:44:17.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.918+0000 7fb04d0bd640 1 -- 172.21.15.90:0/2395850509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb048109ba0 msgr2=0x7fb048101b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:17.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.918+0000 7fb04d0bd640 1 --2- 172.21.15.90:0/2395850509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb048109ba0 0x7fb048101b10 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fb038009900 tx=0x7fb03802a410 comp rx=0 tx=0).stop 2024-04-25T11:44:17.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.918+0000 7fb04d0bd640 1 -- 172.21.15.90:0/2395850509 shutdown_connections 2024-04-25T11:44:17.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.918+0000 7fb04d0bd640 1 --2- 172.21.15.90:0/2395850509 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb04810e550 0x7fb048110970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.918+0000 7fb04d0bd640 1 --2- 172.21.15.90:0/2395850509 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb048102050 0x7fb048102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.918+0000 7fb04d0bd640 1 --2- 172.21.15.90:0/2395850509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb048109ba0 0x7fb048101b10 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.918+0000 7fb04d0bd640 1 -- 172.21.15.90:0/2395850509 >> 172.21.15.90:0/2395850509 conn(0x7fb048077880 msgr2=0x7fb048077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:17.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.918+0000 7fb04d0bd640 1 -- 172.21.15.90:0/2395850509 shutdown_connections 2024-04-25T11:44:17.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.918+0000 7fb04d0bd640 1 -- 172.21.15.90:0/2395850509 wait complete. 2024-04-25T11:44:17.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.919+0000 7fb04d0bd640 1 Processor -- start 2024-04-25T11:44:17.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.919+0000 7fb04d0bd640 1 -- start start 2024-04-25T11:44:17.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.919+0000 7fb04d0bd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb048102050 0x7fb04819a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:17.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.919+0000 7fb04d0bd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb048109ba0 0x7fb04819ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:17.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.919+0000 7fb04d0bd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb04810e550 0x7fb04819ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:17.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.919+0000 7fb04d0bd640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb048113d10 con 0x7fb048109ba0 2024-04-25T11:44:17.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.919+0000 7fb04d0bd640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb048113b90 con 0x7fb048102050 2024-04-25T11:44:17.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.919+0000 7fb04d0bd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb048113e90 con 0x7fb04810e550 2024-04-25T11:44:17.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.919+0000 7fb04659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb048109ba0 0x7fb04819ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:17.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.919+0000 7fb046d9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb048102050 0x7fb04819a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:17.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.919+0000 7fb04659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb048109ba0 0x7fb04819ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47596/0 (socket says 172.21.15.90:47596) 2024-04-25T11:44:17.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.919+0000 7fb04759e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb04810e550 0x7fb04819ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:17.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.919+0000 7fb04659c640 1 -- 172.21.15.90:0/3113850275 learned_addr learned my addr 172.21.15.90:0/3113850275 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:17.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.920+0000 7fb04659c640 1 -- 172.21.15.90:0/3113850275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb04810e550 msgr2=0x7fb04819ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:17.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.920+0000 7fb04659c640 1 --2- 172.21.15.90:0/3113850275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb04810e550 0x7fb04819ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.920+0000 7fb04659c640 1 -- 172.21.15.90:0/3113850275 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb048102050 msgr2=0x7fb04819a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:17.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.920+0000 7fb04659c640 1 --2- 172.21.15.90:0/3113850275 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb048102050 0x7fb04819a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.920+0000 7fb04659c640 1 -- 172.21.15.90:0/3113850275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb04819f590 con 0x7fb048109ba0 2024-04-25T11:44:17.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.920+0000 7fb04659c640 1 --2- 172.21.15.90:0/3113850275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb048109ba0 0x7fb04819ac90 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fb03c00c970 tx=0x7fb03c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:17.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.920+0000 7fb02ffff640 1 -- 172.21.15.90:0/3113850275 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03c007bb0 con 0x7fb048109ba0 2024-04-25T11:44:17.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.920+0000 7fb04d0bd640 1 -- 172.21.15.90:0/3113850275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb04819f880 con 0x7fb048109ba0 2024-04-25T11:44:17.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.920+0000 7fb02ffff640 1 -- 172.21.15.90:0/3113850275 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb03c007d50 con 0x7fb048109ba0 2024-04-25T11:44:17.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.920+0000 7fb02ffff640 1 -- 172.21.15.90:0/3113850275 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03c00f450 con 0x7fb048109ba0 2024-04-25T11:44:17.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.920+0000 7fb04d0bd640 1 -- 172.21.15.90:0/3113850275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0481a7200 con 0x7fb048109ba0 2024-04-25T11:44:17.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.923+0000 7fb053f71740 1 Processor -- start 2024-04-25T11:44:17.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.923+0000 7fb053f71740 1 -- start start 2024-04-25T11:44:17.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.923+0000 7fb053f71740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c459f5410 0x563c45d954f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:17.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.923+0000 7fb053f71740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459cc550 0x563c45d95a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:17.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.923+0000 7fb053f71740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c459d4730 0x563c45d95f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:17.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.923+0000 7fb053f71740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563c45cfbb00 con 0x563c459cc550 2024-04-25T11:44:17.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.923+0000 7fb053f71740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x563c45c97920 con 0x563c459f5410 2024-04-25T11:44:17.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.923+0000 7fb053f71740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563c45cf7ac0 con 0x563c459d4730 2024-04-25T11:44:17.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.923+0000 7fb02ffff640 1 -- 172.21.15.90:0/3113850275 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fb03c005020 con 0x7fb048109ba0 2024-04-25T11:44:17.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.923+0000 7fb046d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459cc550 0x563c45d95a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:17.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.923+0000 7fb046d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459cc550 0x563c45d95a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47598/0 (socket says 172.21.15.90:47598) 2024-04-25T11:44:17.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.923+0000 7fb046d9d640 1 -- 172.21.15.90:0/89010544 learned_addr learned my addr 172.21.15.90:0/89010544 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:17.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.923+0000 7fb04759e640 1 --2- 172.21.15.90:0/89010544 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c459f5410 0x563c45d954f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:17.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.923+0000 7fb04659c640 1 --2- 172.21.15.90:0/89010544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c459d4730 0x563c45d95f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:17.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.924+0000 7fb046d9d640 1 -- 172.21.15.90:0/89010544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c459d4730 msgr2=0x563c45d95f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:17.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.924+0000 7fb046d9d640 1 --2- 172.21.15.90:0/89010544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c459d4730 0x563c45d95f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.924+0000 7fb046d9d640 1 -- 172.21.15.90:0/89010544 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c459f5410 msgr2=0x563c45d954f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:17.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.924+0000 7fb046d9d640 1 --2- 172.21.15.90:0/89010544 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c459f5410 0x563c45d954f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.924+0000 7fb046d9d640 1 -- 172.21.15.90:0/89010544 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c45ad57e0 con 0x563c459cc550 2024-04-25T11:44:17.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.924+0000 7fb046d9d640 1 --2- 172.21.15.90:0/89010544 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459cc550 0x563c45d95a30 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fb048069bf0 tx=0x7fb038000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a262dd71880f075 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:17.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.924+0000 7fb02ffff640 1 --2- 172.21.15.90:0/3113850275 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb0300afc60 0x7fb0300b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:17.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.924+0000 7fb04759e640 1 --2- 172.21.15.90:0/3113850275 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb0300afc60 0x7fb0300b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:17.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.924+0000 7fb02d7fa640 1 -- 172.21.15.90:0/89010544 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb038040070 con 0x563c459cc550 2024-04-25T11:44:17.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.924+0000 7fb02d7fa640 1 -- 172.21.15.90:0/89010544 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb038004030 con 0x563c459cc550 2024-04-25T11:44:17.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.924+0000 7fb02d7fa640 1 -- 172.21.15.90:0/89010544 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb038035040 con 0x563c459cc550 2024-04-25T11:44:17.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.924+0000 7fb02ffff640 1 -- 172.21.15.90:0/3113850275 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7fb03c0df0c0 con 0x7fb048109ba0 2024-04-25T11:44:17.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.925+0000 7fb053f71740 1 -- 172.21.15.90:0/89010544 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459cc550 msgr2=0x563c45d95a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:17.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.925+0000 7fb04759e640 1 --2- 172.21.15.90:0/3113850275 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb0300afc60 0x7fb0300b2170 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb048102af0 tx=0x7fb040009500 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:17.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.925+0000 7fb053f71740 1 --2- 172.21.15.90:0/89010544 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459cc550 0x563c45d95a30 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fb048069bf0 tx=0x7fb038000bc0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.925+0000 7fb053f71740 1 -- 172.21.15.90:0/89010544 shutdown_connections 2024-04-25T11:44:17.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.925+0000 7fb053f71740 1 --2- 172.21.15.90:0/89010544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c459d4730 0x563c45d95f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.925+0000 7fb053f71740 1 --2- 172.21.15.90:0/89010544 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459cc550 0x563c45d95a30 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.925+0000 7fb053f71740 1 --2- 172.21.15.90:0/89010544 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c459f5410 0x563c45d954f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.925+0000 7fb053f71740 1 -- 172.21.15.90:0/89010544 >> 172.21.15.90:0/89010544 conn(0x563c459e9560 msgr2=0x563c45d91dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:17.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.925+0000 7fb053f71740 1 -- 172.21.15.90:0/89010544 shutdown_connections 2024-04-25T11:44:17.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.925+0000 7fb053f71740 1 -- 172.21.15.90:0/89010544 wait complete. 2024-04-25T11:44:17.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.925+0000 7fb053f71740 1 -- start start 2024-04-25T11:44:17.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.926+0000 7fb053f71740 1 Processor -- start 2024-04-25T11:44:17.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.926+0000 7fb053f71740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c459cc550 0x563c45da3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:17.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.926+0000 7fb053f71740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c459d4730 0x563c45da4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:17.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.926+0000 7fb053f71740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459f5410 0x563c45db0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:17.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.926+0000 7fb053f71740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563c45da6da0 con 0x563c459f5410 2024-04-25T11:44:17.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.926+0000 7fb053f71740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x563c45c955d0 con 0x563c459d4730 2024-04-25T11:44:17.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.926+0000 7fb053f71740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563c45da6f20 con 0x563c459cc550 2024-04-25T11:44:17.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.926+0000 7fb053f71740 10 client.0 _ll_register_callbacks cb 0x563c45c83080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:17.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.926+0000 7fb04759e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459f5410 0x563c45db0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:17.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.926+0000 7fb04759e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459f5410 0x563c45db0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47612/0 (socket says 172.21.15.90:47612) 2024-04-25T11:44:17.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.926+0000 7fb04659c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c459d4730 0x563c45da4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:17.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.926+0000 7fb04759e640 1 -- 172.21.15.90:0/2050859759 learned_addr learned my addr 172.21.15.90:0/2050859759 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:17.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.926+0000 7fb046d9d640 1 --2- 172.21.15.90:0/2050859759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c459cc550 0x563c45da3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:17.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.926+0000 7fb04759e640 1 -- 172.21.15.90:0/2050859759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c459cc550 msgr2=0x563c45da3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:17.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.926+0000 7fb04759e640 1 --2- 172.21.15.90:0/2050859759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c459cc550 0x563c45da3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.926+0000 7fb04759e640 1 -- 172.21.15.90:0/2050859759 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c459d4730 msgr2=0x563c45da4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:17.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.926+0000 7fb04759e640 1 --2- 172.21.15.90:0/2050859759 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c459d4730 0x563c45da4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.926+0000 7fb04759e640 1 -- 172.21.15.90:0/2050859759 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c45b35f70 con 0x563c459f5410 2024-04-25T11:44:17.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.926+0000 7fb04759e640 1 --2- 172.21.15.90:0/2050859759 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459f5410 0x563c45db0250 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fb040004060 tx=0x7fb040004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=384e974cfe24e050 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:17.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.926+0000 7fb02aff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:17.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.927+0000 7fb02aff5640 1 -- 172.21.15.90:0/2050859759 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04000f8b0 con 0x563c459f5410 2024-04-25T11:44:17.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.927+0000 7fb02aff5640 1 -- 172.21.15.90:0/2050859759 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7fb04000fa50 con 0x563c459f5410 2024-04-25T11:44:17.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.927+0000 7fb053f71740 1 -- 172.21.15.90:0/2050859759 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb038004030 con 0x563c459f5410 2024-04-25T11:44:17.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.927+0000 7fb02aff5640 1 -- 172.21.15.90:0/2050859759 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb04000fc50 con 0x563c459f5410 2024-04-25T11:44:17.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.927+0000 7fb053f71740 10 client.18534 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:17.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.927+0000 7fb053f71740 1 -- 172.21.15.90:0/2050859759 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563c459cd170 con 0x563c459f5410 2024-04-25T11:44:17.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.927+0000 7fb02aff5640 1 -- 172.21.15.90:0/2050859759 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb038004030 con 0x563c459f5410 2024-04-25T11:44:17.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.927+0000 7fb02aff5640 1 -- 172.21.15.90:0/2050859759 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fb040026070 con 0x563c459f5410 2024-04-25T11:44:17.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.927+0000 7fb053f71740 10 client.18534 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:17.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.927+0000 7fb053f71740 10 client.18534 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:44:17.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.927+0000 7fb053f71740 20 client.18534 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:17.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.927+0000 7fb053f71740 1 --2- 172.21.15.90:0/2050859759 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563c459b41f0 0x563c45db64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:17.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.927+0000 7fb053f71740 4 client.18534 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:44:17.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.927+0000 7fb053f71740 1 -- 172.21.15.90:0/2050859759 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c45d09710 con 0x563c459b41f0 2024-04-25T11:44:17.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.927+0000 7fb046d9d640 1 --2- 172.21.15.90:0/2050859759 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563c459b41f0 0x563c45db64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:17.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.928+0000 7fb02aff5640 10 client.18534 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:17.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.928+0000 7fb046d9d640 1 --2- 172.21.15.90:0/2050859759 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563c459b41f0 0x563c45db64c0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59c2ec61314f9b5b server_cookie=4a8f903f0451424a in_seq=0 out_seq=0 2024-04-25T11:44:17.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.929+0000 7fb02aff5640 1 -- 172.21.15.90:0/2050859759 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563c45d09710 con 0x563c459b41f0 2024-04-25T11:44:17.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.929+0000 7fb02aff5640 10 client.18534 handle_command_reply: tid=0 2024-04-25T11:44:17.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.929+0000 7fb053f71740 1 client.18534 shutdown 2024-04-25T11:44:17.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.929+0000 7fb053f71740 1 -- 172.21.15.90:0/2050859759 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459f5410 msgr2=0x563c45db0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:17.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.929+0000 7fb053f71740 1 --2- 172.21.15.90:0/2050859759 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459f5410 0x563c45db0250 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fb040004060 tx=0x7fb040004ab0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.929+0000 7fb053f71740 1 -- 172.21.15.90:0/2050859759 shutdown_connections 2024-04-25T11:44:17.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.929+0000 7fb053f71740 1 --2- 172.21.15.90:0/2050859759 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563c459b41f0 0x563c45db64c0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.929+0000 7fb053f71740 1 --2- 172.21.15.90:0/2050859759 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459f5410 0x563c45db0250 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.929+0000 7fb053f71740 1 --2- 172.21.15.90:0/2050859759 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c459d4730 0x563c45da4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.929+0000 7fb02aff5640 0 client.18534 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:17.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.929+0000 7fb053f71740 1 --2- 172.21.15.90:0/2050859759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c459cc550 0x563c45da3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.929+0000 7fb053f71740 1 -- 172.21.15.90:0/2050859759 >> 172.21.15.90:0/2050859759 conn(0x563c459e9560 msgr2=0x563c45d95de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:17.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.929+0000 7fb053f71740 1 -- 172.21.15.90:0/2050859759 shutdown_connections 2024-04-25T11:44:17.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.929+0000 7fb053f71740 1 -- 172.21.15.90:0/2050859759 wait complete. 2024-04-25T11:44:17.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.929+0000 7fb053f71740 20 client.18534 trim_cache size 0 max 16384 2024-04-25T11:44:17.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.949+0000 7fb053f71740 1 Processor -- start 2024-04-25T11:44:17.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.949+0000 7fb053f71740 1 -- start start 2024-04-25T11:44:17.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.949+0000 7fb053f71740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c459cc550 0x563c45d9d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:17.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.949+0000 7fb053f71740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459d4730 0x563c45d97890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:17.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.949+0000 7fb053f71740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c459f5410 0x563c45d97dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:17.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.949+0000 7fb053f71740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb0481adb10 con 0x563c459d4730 2024-04-25T11:44:17.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.949+0000 7fb053f71740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb0481af650 con 0x563c459cc550 2024-04-25T11:44:17.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.949+0000 7fb053f71740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb0481ae1e0 con 0x563c459f5410 2024-04-25T11:44:17.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.949+0000 7fb04659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459d4730 0x563c45d97890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:17.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.949+0000 7fb04659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459d4730 0x563c45d97890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47620/0 (socket says 172.21.15.90:47620) 2024-04-25T11:44:17.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.949+0000 7fb04659c640 1 -- 172.21.15.90:0/4135223997 learned_addr learned my addr 172.21.15.90:0/4135223997 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:17.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.949+0000 7fb046d9d640 1 --2- 172.21.15.90:0/4135223997 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c459cc550 0x563c45d9d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:17.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.949+0000 7fb04759e640 1 --2- 172.21.15.90:0/4135223997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c459f5410 0x563c45d97dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:17.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.949+0000 7fb04659c640 1 -- 172.21.15.90:0/4135223997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c459f5410 msgr2=0x563c45d97dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:17.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.949+0000 7fb04659c640 1 --2- 172.21.15.90:0/4135223997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c459f5410 0x563c45d97dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.949+0000 7fb04659c640 1 -- 172.21.15.90:0/4135223997 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c459cc550 msgr2=0x563c45d9d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:17.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.949+0000 7fb04659c640 1 --2- 172.21.15.90:0/4135223997 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c459cc550 0x563c45d9d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.949+0000 7fb04659c640 1 -- 172.21.15.90:0/4135223997 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c45d963c0 con 0x563c459d4730 2024-04-25T11:44:17.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.950+0000 7fb04659c640 1 --2- 172.21.15.90:0/4135223997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459d4730 0x563c45d97890 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fb03c000c00 tx=0x7fb03c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e408b5a05f791c4a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:17.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.950+0000 7fb02cff9640 1 -- 172.21.15.90:0/4135223997 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03c01d040 con 0x563c459d4730 2024-04-25T11:44:17.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.950+0000 7fb02cff9640 1 -- 172.21.15.90:0/4135223997 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb03c0a6d70 con 0x563c459d4730 2024-04-25T11:44:17.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.950+0000 7fb02cff9640 1 -- 172.21.15.90:0/4135223997 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03c005310 con 0x563c459d4730 2024-04-25T11:44:17.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.951+0000 7fb053f71740 1 -- 172.21.15.90:0/4135223997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459d4730 msgr2=0x563c45d97890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:17.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.951+0000 7fb053f71740 1 --2- 172.21.15.90:0/4135223997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459d4730 0x563c45d97890 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fb03c000c00 tx=0x7fb03c00cf60 comp rx=0 tx=0).stop 2024-04-25T11:44:17.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.951+0000 7fb053f71740 1 -- 172.21.15.90:0/4135223997 shutdown_connections 2024-04-25T11:44:17.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.951+0000 7fb053f71740 1 --2- 172.21.15.90:0/4135223997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c459f5410 0x563c45d97dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.951+0000 7fb053f71740 1 --2- 172.21.15.90:0/4135223997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459d4730 0x563c45d97890 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.951+0000 7fb053f71740 1 --2- 172.21.15.90:0/4135223997 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c459cc550 0x563c45d9d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.951+0000 7fb053f71740 1 -- 172.21.15.90:0/4135223997 >> 172.21.15.90:0/4135223997 conn(0x563c459e9560 msgr2=0x563c45d97180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:17.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.951+0000 7fb053f71740 1 -- 172.21.15.90:0/4135223997 shutdown_connections 2024-04-25T11:44:17.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.951+0000 7fb053f71740 1 -- 172.21.15.90:0/4135223997 wait complete. 2024-04-25T11:44:17.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.951+0000 7fb053f71740 1 -- start start 2024-04-25T11:44:17.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.951+0000 7fb053f71740 1 Processor -- start 2024-04-25T11:44:17.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.952+0000 7fb053f71740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c459cc550 0x563c45dac720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:17.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.952+0000 7fb053f71740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c459d4730 0x563c45da0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:17.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.952+0000 7fb053f71740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459f5410 0x563c45da0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:17.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.952+0000 7fb053f71740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb0481aef80 con 0x563c459f5410 2024-04-25T11:44:17.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.952+0000 7fb053f71740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb0481ae8b0 con 0x563c459cc550 2024-04-25T11:44:17.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.952+0000 7fb053f71740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb0481abb10 con 0x563c459d4730 2024-04-25T11:44:17.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.952+0000 7fb053f71740 10 client.0 _ll_register_callbacks cb 0x7fb040003ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:17.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.952+0000 7fb04759e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459f5410 0x563c45da0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:17.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.952+0000 7fb046d9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c459cc550 0x563c45dac720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:17.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.952+0000 7fb04659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c459d4730 0x563c45da0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:17.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.952+0000 7fb04759e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459f5410 0x563c45da0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47626/0 (socket says 172.21.15.90:47626) 2024-04-25T11:44:17.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.952+0000 7fb04759e640 1 -- 172.21.15.90:0/3343579508 learned_addr learned my addr 172.21.15.90:0/3343579508 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:17.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.952+0000 7fb04759e640 1 -- 172.21.15.90:0/3343579508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c459d4730 msgr2=0x563c45da0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:17.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.952+0000 7fb04759e640 1 --2- 172.21.15.90:0/3343579508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c459d4730 0x563c45da0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.952+0000 7fb04759e640 1 -- 172.21.15.90:0/3343579508 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c459cc550 msgr2=0x563c45dac720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:17.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.952+0000 7fb04759e640 1 --2- 172.21.15.90:0/3343579508 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c459cc550 0x563c45dac720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.952+0000 7fb04759e640 1 -- 172.21.15.90:0/3343579508 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c45da1640 con 0x563c459f5410 2024-04-25T11:44:17.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.952+0000 7fb04759e640 1 --2- 172.21.15.90:0/3343579508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459f5410 0x563c45da0eb0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fb0400285b0 tx=0x7fb040011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63a2b30d0ea91734 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:17.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.952+0000 7fb02aff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:17.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.952+0000 7fb02aff5640 1 -- 172.21.15.90:0/3343579508 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb040028bf0 con 0x563c459f5410 2024-04-25T11:44:17.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.953+0000 7fb02aff5640 1 -- 172.21.15.90:0/3343579508 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7fb040028d90 con 0x563c459f5410 2024-04-25T11:44:17.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.953+0000 7fb053f71740 1 -- 172.21.15.90:0/3343579508 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb03c0a6d70 con 0x563c459f5410 2024-04-25T11:44:17.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.953+0000 7fb02aff5640 1 -- 172.21.15.90:0/3343579508 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb040004b20 con 0x563c459f5410 2024-04-25T11:44:17.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.953+0000 7fb053f71740 10 client.18540 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:17.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.953+0000 7fb053f71740 1 -- 172.21.15.90:0/3343579508 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563c45e1e830 con 0x563c459f5410 2024-04-25T11:44:17.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.953+0000 7fb02aff5640 1 -- 172.21.15.90:0/3343579508 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb03c0a6d70 con 0x563c459f5410 2024-04-25T11:44:17.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.953+0000 7fb02aff5640 1 -- 172.21.15.90:0/3343579508 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fb040026070 con 0x563c459f5410 2024-04-25T11:44:17.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.953+0000 7fb053f71740 10 client.18540 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:17.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.953+0000 7fb053f71740 10 client.18540 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:44:17.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.953+0000 7fb053f71740 20 client.18540 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:17.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.953+0000 7fb053f71740 1 --2- 172.21.15.90:0/3343579508 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563c45989630 0x563c45e20f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:17.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.953+0000 7fb053f71740 4 client.18540 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T11:44:17.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.953+0000 7fb053f71740 1 -- 172.21.15.90:0/3343579508 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563c45d03580 con 0x563c45989630 2024-04-25T11:44:17.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.953+0000 7fb046d9d640 1 --2- 172.21.15.90:0/3343579508 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563c45989630 0x563c45e20f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:17.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.954+0000 7fb02aff5640 10 client.18540 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:17.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.954+0000 7fb046d9d640 1 --2- 172.21.15.90:0/3343579508 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563c45989630 0x563c45e20f90 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d91db002f3aefa1 server_cookie=df6a9d32a345ff67 in_seq=0 out_seq=0 2024-04-25T11:44:17.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.955+0000 7fb02aff5640 1 -- 172.21.15.90:0/3343579508 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563c45d03580 con 0x563c45989630 2024-04-25T11:44:17.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.955+0000 7fb02aff5640 10 client.18540 handle_command_reply: tid=0 2024-04-25T11:44:17.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.955+0000 7fb053f71740 1 client.18540 shutdown 2024-04-25T11:44:17.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.956+0000 7fb053f71740 1 -- 172.21.15.90:0/3343579508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459f5410 msgr2=0x563c45da0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:17.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.956+0000 7fb053f71740 1 --2- 172.21.15.90:0/3343579508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459f5410 0x563c45da0eb0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fb0400285b0 tx=0x7fb040011f60 comp rx=0 tx=0).stop 2024-04-25T11:44:17.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.956+0000 7fb053f71740 1 -- 172.21.15.90:0/3343579508 shutdown_connections 2024-04-25T11:44:17.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.956+0000 7fb053f71740 1 --2- 172.21.15.90:0/3343579508 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x563c45989630 0x563c45e20f90 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.956+0000 7fb053f71740 1 --2- 172.21.15.90:0/3343579508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c459f5410 0x563c45da0eb0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.956+0000 7fb053f71740 1 --2- 172.21.15.90:0/3343579508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c459d4730 0x563c45da0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.956+0000 7fb053f71740 1 --2- 172.21.15.90:0/3343579508 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x563c459cc550 0x563c45dac720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.956+0000 7fb053f71740 1 -- 172.21.15.90:0/3343579508 >> 172.21.15.90:0/3343579508 conn(0x563c459e9560 msgr2=0x563c45d97180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:17.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.956+0000 7fb02aff5640 0 client.18540 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:17.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.956+0000 7fb053f71740 1 -- 172.21.15.90:0/3343579508 shutdown_connections 2024-04-25T11:44:17.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.956+0000 7fb053f71740 1 -- 172.21.15.90:0/3343579508 wait complete. 2024-04-25T11:44:17.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.956+0000 7fb053f71740 20 client.18540 trim_cache size 0 max 16384 2024-04-25T11:44:17.947 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:44:17.947 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-25T11:44:17.947 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "3628b186-9847-43ff-a836-387a6f10ef9d", 2024-04-25T11:44:17.947 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-25T11:44:17.947 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:44:17.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.956+0000 7fb04d0bd640 1 -- 172.21.15.90:0/3113850275 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb0300afc60 msgr2=0x7fb0300b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:17.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.956+0000 7fb04d0bd640 1 --2- 172.21.15.90:0/3113850275 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb0300afc60 0x7fb0300b2170 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb048102af0 tx=0x7fb040009500 comp rx=0 tx=0).stop 2024-04-25T11:44:17.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.956+0000 7fb04d0bd640 1 -- 172.21.15.90:0/3113850275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb048109ba0 msgr2=0x7fb04819ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:17.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.956+0000 7fb04d0bd640 1 --2- 172.21.15.90:0/3113850275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb048109ba0 0x7fb04819ac90 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fb03c00c970 tx=0x7fb03c00ce30 comp rx=0 tx=0).stop 2024-04-25T11:44:17.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.956+0000 7fb04d0bd640 1 -- 172.21.15.90:0/3113850275 shutdown_connections 2024-04-25T11:44:17.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.957+0000 7fb04d0bd640 1 --2- 172.21.15.90:0/3113850275 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb0300afc60 0x7fb0300b2170 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.957+0000 7fb04d0bd640 1 --2- 172.21.15.90:0/3113850275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb04810e550 0x7fb04819ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.957+0000 7fb04d0bd640 1 --2- 172.21.15.90:0/3113850275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb048109ba0 0x7fb04819ac90 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.957+0000 7fb04d0bd640 1 --2- 172.21.15.90:0/3113850275 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb048102050 0x7fb04819a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:17.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.957+0000 7fb04d0bd640 1 -- 172.21.15.90:0/3113850275 >> 172.21.15.90:0/3113850275 conn(0x7fb048077880 msgr2=0x7fb04810a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:17.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.957+0000 7fb04d0bd640 1 -- 172.21.15.90:0/3113850275 shutdown_connections 2024-04-25T11:44:17.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:17.957+0000 7fb04d0bd640 1 -- 172.21.15.90:0/3113850275 wait complete. 2024-04-25T11:44:17.954 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:44:17.954 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:44:18.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:17 smithi138 ceph-mon[32328]: pgmap v284: 97 pgs: 18 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 48 active+clean; 44 GiB data, 138 GiB used, 935 GiB / 1.0 TiB avail; 7.4 KiB/s rd, 3.7 KiB/s wr, 10 op/s 2024-04-25T11:44:18.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.134+0000 7f40945ba640 1 -- 172.21.15.90:0/875420199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f408c10d3a0 msgr2=0x7f408c10f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:18.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.134+0000 7f40945ba640 1 --2- 172.21.15.90:0/875420199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f408c10d3a0 0x7f408c10f7c0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f4084009ca0 tx=0x7f408402a8c0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.134+0000 7f40945ba640 1 -- 172.21.15.90:0/875420199 shutdown_connections 2024-04-25T11:44:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.134+0000 7f40945ba640 1 --2- 172.21.15.90:0/875420199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f408c10d3a0 0x7f408c10f7c0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.134+0000 7f40945ba640 1 --2- 172.21.15.90:0/875420199 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f408c102b10 0x7f408c102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.134+0000 7f40945ba640 1 --2- 172.21.15.90:0/875420199 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f408c108a40 0x7f408c1025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.134+0000 7f40945ba640 1 -- 172.21.15.90:0/875420199 >> 172.21.15.90:0/875420199 conn(0x7f408c078180 msgr2=0x7f408c0785c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.134+0000 7f40945ba640 1 -- 172.21.15.90:0/875420199 shutdown_connections 2024-04-25T11:44:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.134+0000 7f40945ba640 1 -- 172.21.15.90:0/875420199 wait complete. 2024-04-25T11:44:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.135+0000 7f40945ba640 1 Processor -- start 2024-04-25T11:44:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.135+0000 7f40945ba640 1 -- start start 2024-04-25T11:44:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.135+0000 7f40945ba640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f408c102b10 0x7f408c19b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.135+0000 7f40945ba640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f408c108a40 0x7f408c19ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.135+0000 7f40945ba640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f408c10d3a0 0x7f408c19bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.135+0000 7f40945ba640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f408c112c20 con 0x7f408c108a40 2024-04-25T11:44:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.135+0000 7f40945ba640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f408c112aa0 con 0x7f408c102b10 2024-04-25T11:44:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.135+0000 7f40945ba640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f408c112da0 con 0x7f408c10d3a0 2024-04-25T11:44:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.136+0000 7f4091b55640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f408c108a40 0x7f408c19ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.136+0000 7f4091b55640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f408c108a40 0x7f408c19ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47650/0 (socket says 172.21.15.90:47650) 2024-04-25T11:44:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.136+0000 7f4091b55640 1 -- 172.21.15.90:0/21956871 learned_addr learned my addr 172.21.15.90:0/21956871 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.136+0000 7f4092356640 1 --2- 172.21.15.90:0/21956871 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f408c102b10 0x7f408c19b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.136+0000 7f4092b57640 1 --2- 172.21.15.90:0/21956871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f408c10d3a0 0x7f408c19bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.136+0000 7f4091b55640 1 -- 172.21.15.90:0/21956871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f408c10d3a0 msgr2=0x7f408c19bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.136+0000 7f4091b55640 1 --2- 172.21.15.90:0/21956871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f408c10d3a0 0x7f408c19bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.136+0000 7f4091b55640 1 -- 172.21.15.90:0/21956871 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f408c102b10 msgr2=0x7f408c19b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.136+0000 7f4091b55640 1 --2- 172.21.15.90:0/21956871 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f408c102b10 0x7f408c19b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.136+0000 7f4091b55640 1 -- 172.21.15.90:0/21956871 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f408c1a67f0 con 0x7f408c108a40 2024-04-25T11:44:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.136+0000 7f4091b55640 1 --2- 172.21.15.90:0/21956871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f408c108a40 0x7f408c19ba20 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f407c00e9f0 tx=0x7f407c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.136+0000 7f40837fe640 1 -- 172.21.15.90:0/21956871 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407c00ccc0 con 0x7f408c108a40 2024-04-25T11:44:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.136+0000 7f40837fe640 1 -- 172.21.15.90:0/21956871 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f407c0035f0 con 0x7f408c108a40 2024-04-25T11:44:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.136+0000 7f40837fe640 1 -- 172.21.15.90:0/21956871 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407c00ce60 con 0x7f408c108a40 2024-04-25T11:44:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.136+0000 7f40945ba640 1 -- 172.21.15.90:0/21956871 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f408c1a6ae0 con 0x7f408c108a40 2024-04-25T11:44:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.136+0000 7f40945ba640 1 -- 172.21.15.90:0/21956871 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f408c1a7030 con 0x7f408c108a40 2024-04-25T11:44:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.140+0000 7f409b45f740 1 Processor -- start 2024-04-25T11:44:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.140+0000 7f409b45f740 1 -- start start 2024-04-25T11:44:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.140+0000 7f40837fe640 1 -- 172.21.15.90:0/21956871 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f407c003aa0 con 0x7f408c108a40 2024-04-25T11:44:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.140+0000 7f409b45f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323821a390 0x5632385baa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.140+0000 7f409b45f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5632381f14d0 0x5632385bafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.140+0000 7f409b45f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5632381f96b0 0x5632385c78f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.140+0000 7f409b45f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563238520d80 con 0x56323821a390 2024-04-25T11:44:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.140+0000 7f409b45f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5632384da8b0 con 0x5632381f96b0 2024-04-25T11:44:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.140+0000 7f409b45f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56323851cd40 con 0x5632381f14d0 2024-04-25T11:44:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.140+0000 7f4092b57640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323821a390 0x5632385baa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.140+0000 7f4092b57640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323821a390 0x5632385baa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47652/0 (socket says 172.21.15.90:47652) 2024-04-25T11:44:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.140+0000 7f4092b57640 1 -- 172.21.15.90:0/2958109953 learned_addr learned my addr 172.21.15.90:0/2958109953 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.140+0000 7f4091b55640 1 --2- 172.21.15.90:0/2958109953 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5632381f96b0 0x5632385c78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.140+0000 7f4092356640 1 --2- 172.21.15.90:0/2958109953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5632381f14d0 0x5632385bafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.141+0000 7f4092b57640 1 -- 172.21.15.90:0/2958109953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5632381f14d0 msgr2=0x5632385bafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.141+0000 7f4092b57640 1 --2- 172.21.15.90:0/2958109953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5632381f14d0 0x5632385bafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.141+0000 7f4092b57640 1 -- 172.21.15.90:0/2958109953 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5632381f96b0 msgr2=0x5632385c78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.141+0000 7f4092b57640 1 --2- 172.21.15.90:0/2958109953 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5632381f96b0 0x5632385c78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.141+0000 7f4092b57640 1 -- 172.21.15.90:0/2958109953 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632384aba40 con 0x56323821a390 2024-04-25T11:44:18.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.141+0000 7f4091b55640 1 --2- 172.21.15.90:0/2958109953 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5632381f96b0 0x5632385c78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:18.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.141+0000 7f40837fe640 1 --2- 172.21.15.90:0/21956871 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f40780afce0 0x7f40780b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:18.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.141+0000 7f4092b57640 1 --2- 172.21.15.90:0/2958109953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323821a390 0x5632385baa60 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f4084035000 tx=0x7f4084009460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86a574f196adb7f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:18.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.141+0000 7f4092356640 1 --2- 172.21.15.90:0/21956871 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f40780afce0 0x7f40780b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:18.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.141+0000 7f4080bfe640 1 -- 172.21.15.90:0/2958109953 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4084031af0 con 0x56323821a390 2024-04-25T11:44:18.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.141+0000 7f40837fe640 1 -- 172.21.15.90:0/21956871 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7f407c0df3c0 con 0x7f408c108a40 2024-04-25T11:44:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.141+0000 7f4080bfe640 1 -- 172.21.15.90:0/2958109953 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4084036070 con 0x56323821a390 2024-04-25T11:44:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.141+0000 7f4080bfe640 1 -- 172.21.15.90:0/2958109953 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4084004040 con 0x56323821a390 2024-04-25T11:44:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.142+0000 7f409b45f740 1 -- 172.21.15.90:0/2958109953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323821a390 msgr2=0x5632385baa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.142+0000 7f409b45f740 1 --2- 172.21.15.90:0/2958109953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323821a390 0x5632385baa60 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f4084035000 tx=0x7f4084009460 comp rx=0 tx=0).stop 2024-04-25T11:44:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.142+0000 7f4092356640 1 --2- 172.21.15.90:0/21956871 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f40780afce0 0x7f40780b21f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x5632385bc230 tx=0x7f4088005f90 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.142+0000 7f409b45f740 1 -- 172.21.15.90:0/2958109953 shutdown_connections 2024-04-25T11:44:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.142+0000 7f409b45f740 1 --2- 172.21.15.90:0/2958109953 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5632381f96b0 0x5632385c78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.142+0000 7f409b45f740 1 --2- 172.21.15.90:0/2958109953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5632381f14d0 0x5632385bafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.142+0000 7f409b45f740 1 --2- 172.21.15.90:0/2958109953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323821a390 0x5632385baa60 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.142+0000 7f409b45f740 1 -- 172.21.15.90:0/2958109953 >> 172.21.15.90:0/2958109953 conn(0x56323820e4e0 msgr2=0x5632385b7210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.142+0000 7f409b45f740 1 -- 172.21.15.90:0/2958109953 shutdown_connections 2024-04-25T11:44:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.142+0000 7f409b45f740 1 -- 172.21.15.90:0/2958109953 wait complete. 2024-04-25T11:44:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.142+0000 7f409b45f740 1 -- start start 2024-04-25T11:44:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.143+0000 7f409b45f740 1 Processor -- start 2024-04-25T11:44:18.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.143+0000 7f409b45f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5632381f14d0 0x5632385c6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:18.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.143+0000 7f409b45f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5632381f96b0 0x5632385c7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:18.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.143+0000 7f409b45f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56323821a390 0x5632385c54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:18.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.143+0000 7f409b45f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5632385c7f80 con 0x5632381f96b0 2024-04-25T11:44:18.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.143+0000 7f409b45f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5632384a7d10 con 0x56323821a390 2024-04-25T11:44:18.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.143+0000 7f409b45f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5632385c8100 con 0x5632381f14d0 2024-04-25T11:44:18.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.143+0000 7f409b45f740 10 client.0 _ll_register_callbacks cb 0x5632384d4210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:18.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.143+0000 7f4091b55640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5632381f96b0 0x5632385c7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:18.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.143+0000 7f4091b55640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5632381f96b0 0x5632385c7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47666/0 (socket says 172.21.15.90:47666) 2024-04-25T11:44:18.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.143+0000 7f4091b55640 1 -- 172.21.15.90:0/3499846820 learned_addr learned my addr 172.21.15.90:0/3499846820 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:18.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.143+0000 7f4092b57640 1 --2- 172.21.15.90:0/3499846820 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56323821a390 0x5632385c54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:18.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.143+0000 7f4092356640 1 --2- 172.21.15.90:0/3499846820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5632381f14d0 0x5632385c6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:18.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.143+0000 7f4091b55640 1 -- 172.21.15.90:0/3499846820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5632381f14d0 msgr2=0x5632385c6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:18.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.143+0000 7f4091b55640 1 --2- 172.21.15.90:0/3499846820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5632381f14d0 0x5632385c6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.143+0000 7f4091b55640 1 -- 172.21.15.90:0/3499846820 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56323821a390 msgr2=0x5632385c54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:18.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.143+0000 7f4091b55640 1 --2- 172.21.15.90:0/3499846820 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56323821a390 0x5632385c54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.143+0000 7f4091b55640 1 -- 172.21.15.90:0/3499846820 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632382fc7d0 con 0x5632381f96b0 2024-04-25T11:44:18.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.143+0000 7f4092356640 1 --2- 172.21.15.90:0/3499846820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5632381f14d0 0x5632385c6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:44:18.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.144+0000 7f4091b55640 1 --2- 172.21.15.90:0/3499846820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5632381f96b0 0x5632385c7520 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f407c0a71b0 tx=0x7f407c0a39c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=defe6172a1697978 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:18.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.144+0000 7f4075ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:18.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.144+0000 7f4075ffb640 1 -- 172.21.15.90:0/3499846820 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407c0a6e00 con 0x5632381f96b0 2024-04-25T11:44:18.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.144+0000 7f4075ffb640 1 -- 172.21.15.90:0/3499846820 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7f407c0a7920 con 0x5632381f96b0 2024-04-25T11:44:18.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.144+0000 7f409b45f740 1 -- 172.21.15.90:0/3499846820 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4084036070 con 0x5632381f96b0 2024-04-25T11:44:18.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.144+0000 7f4075ffb640 1 -- 172.21.15.90:0/3499846820 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f407c0a6c20 con 0x5632381f96b0 2024-04-25T11:44:18.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.144+0000 7f409b45f740 10 client.18552 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:18.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.144+0000 7f409b45f740 1 -- 172.21.15.90:0/3499846820 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5632382323f0 con 0x5632381f96b0 2024-04-25T11:44:18.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.144+0000 7f4075ffb640 1 -- 172.21.15.90:0/3499846820 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4084036070 con 0x5632381f96b0 2024-04-25T11:44:18.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.144+0000 7f4075ffb640 1 -- 172.21.15.90:0/3499846820 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f407c0afe80 con 0x5632381f96b0 2024-04-25T11:44:18.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.144+0000 7f409b45f740 10 client.18552 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:18.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.145+0000 7f409b45f740 10 client.18552 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:44:18.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.145+0000 7f409b45f740 20 client.18552 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:18.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.145+0000 7f409b45f740 1 --2- 172.21.15.90:0/3499846820 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5632381d8f60 0x5632385d7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:18.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.145+0000 7f409b45f740 4 client.18552 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:44:18.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.145+0000 7f409b45f740 1 -- 172.21.15.90:0/3499846820 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56323852e990 con 0x5632381d8f60 2024-04-25T11:44:18.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.145+0000 7f4092356640 1 --2- 172.21.15.90:0/3499846820 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5632381d8f60 0x5632385d7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:18.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.145+0000 7f4075ffb640 10 client.18552 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:18.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.145+0000 7f4092356640 1 --2- 172.21.15.90:0/3499846820 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5632381d8f60 0x5632385d7760 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9d1811acba29c8a server_cookie=ed972819073aed1 in_seq=0 out_seq=0 2024-04-25T11:44:18.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.146+0000 7f4075ffb640 1 -- 172.21.15.90:0/3499846820 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56323852e990 con 0x5632381d8f60 2024-04-25T11:44:18.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.146+0000 7f4075ffb640 10 client.18552 handle_command_reply: tid=0 2024-04-25T11:44:18.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.146+0000 7f409b45f740 1 client.18552 shutdown 2024-04-25T11:44:18.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.146+0000 7f409b45f740 1 -- 172.21.15.90:0/3499846820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5632381f96b0 msgr2=0x5632385c7520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:18.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.146+0000 7f409b45f740 1 --2- 172.21.15.90:0/3499846820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5632381f96b0 0x5632385c7520 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f407c0a71b0 tx=0x7f407c0a39c0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.146+0000 7f409b45f740 1 -- 172.21.15.90:0/3499846820 shutdown_connections 2024-04-25T11:44:18.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.146+0000 7f409b45f740 1 --2- 172.21.15.90:0/3499846820 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5632381d8f60 0x5632385d7760 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.146+0000 7f409b45f740 1 --2- 172.21.15.90:0/3499846820 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56323821a390 0x5632385c54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.146+0000 7f4075ffb640 0 client.18552 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:18.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.146+0000 7f409b45f740 1 --2- 172.21.15.90:0/3499846820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5632381f96b0 0x5632385c7520 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.146+0000 7f409b45f740 1 --2- 172.21.15.90:0/3499846820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5632381f14d0 0x5632385c6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.146+0000 7f409b45f740 1 -- 172.21.15.90:0/3499846820 >> 172.21.15.90:0/3499846820 conn(0x56323820e4e0 msgr2=0x5632385b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:18.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.146+0000 7f409b45f740 1 -- 172.21.15.90:0/3499846820 shutdown_connections 2024-04-25T11:44:18.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.147+0000 7f409b45f740 1 -- 172.21.15.90:0/3499846820 wait complete. 2024-04-25T11:44:18.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.147+0000 7f409b45f740 20 client.18552 trim_cache size 0 max 16384 2024-04-25T11:44:18.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.165+0000 7f409b45f740 1 Processor -- start 2024-04-25T11:44:18.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.165+0000 7f409b45f740 1 -- start start 2024-04-25T11:44:18.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.165+0000 7f409b45f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5632381f14d0 0x5632385baf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:18.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.165+0000 7f409b45f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5632381f96b0 0x5632385cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:18.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.165+0000 7f409b45f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56323821a390 0x5632385cd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:18.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.165+0000 7f409b45f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f408c1afae0 con 0x5632381f96b0 2024-04-25T11:44:18.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.165+0000 7f409b45f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f408c1b1620 con 0x56323821a390 2024-04-25T11:44:18.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.165+0000 7f409b45f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f408c1b01b0 con 0x5632381f14d0 2024-04-25T11:44:18.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.166+0000 7f4091b55640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5632381f96b0 0x5632385cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:18.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.166+0000 7f4091b55640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5632381f96b0 0x5632385cd3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47676/0 (socket says 172.21.15.90:47676) 2024-04-25T11:44:18.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.166+0000 7f4091b55640 1 -- 172.21.15.90:0/3276481363 learned_addr learned my addr 172.21.15.90:0/3276481363 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:18.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.166+0000 7f4092356640 1 --2- 172.21.15.90:0/3276481363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5632381f14d0 0x5632385baf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:18.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.166+0000 7f4092b57640 1 --2- 172.21.15.90:0/3276481363 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56323821a390 0x5632385cd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:18.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.166+0000 7f4091b55640 1 -- 172.21.15.90:0/3276481363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5632381f14d0 msgr2=0x5632385baf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:18.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.166+0000 7f4091b55640 1 --2- 172.21.15.90:0/3276481363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5632381f14d0 0x5632385baf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.166+0000 7f4091b55640 1 -- 172.21.15.90:0/3276481363 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56323821a390 msgr2=0x5632385cd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:18.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.166+0000 7f4091b55640 1 --2- 172.21.15.90:0/3276481363 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56323821a390 0x5632385cd920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.166+0000 7f4091b55640 1 -- 172.21.15.90:0/3276481363 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632381f20f0 con 0x5632381f96b0 2024-04-25T11:44:18.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.166+0000 7f4091b55640 1 --2- 172.21.15.90:0/3276481363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5632381f96b0 0x5632385cd3e0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f407c0a7440 tx=0x7f407c0dcb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f5fec21479120ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:18.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.166+0000 7f4077fff640 1 -- 172.21.15.90:0/3276481363 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407c0a6e00 con 0x5632381f96b0 2024-04-25T11:44:18.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.166+0000 7f4077fff640 1 -- 172.21.15.90:0/3276481363 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f407c0a6fc0 con 0x5632381f96b0 2024-04-25T11:44:18.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.166+0000 7f4077fff640 1 -- 172.21.15.90:0/3276481363 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407c0a2f60 con 0x5632381f96b0 2024-04-25T11:44:18.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.167+0000 7f409b45f740 1 -- 172.21.15.90:0/3276481363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5632381f96b0 msgr2=0x5632385cd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:18.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.167+0000 7f409b45f740 1 --2- 172.21.15.90:0/3276481363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5632381f96b0 0x5632385cd3e0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f407c0a7440 tx=0x7f407c0dcb30 comp rx=0 tx=0).stop 2024-04-25T11:44:18.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.167+0000 7f409b45f740 1 -- 172.21.15.90:0/3276481363 shutdown_connections 2024-04-25T11:44:18.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.167+0000 7f409b45f740 1 --2- 172.21.15.90:0/3276481363 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56323821a390 0x5632385cd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.167+0000 7f409b45f740 1 --2- 172.21.15.90:0/3276481363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5632381f96b0 0x5632385cd3e0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.167+0000 7f409b45f740 1 --2- 172.21.15.90:0/3276481363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5632381f14d0 0x5632385baf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.167+0000 7f409b45f740 1 -- 172.21.15.90:0/3276481363 >> 172.21.15.90:0/3276481363 conn(0x56323820e4e0 msgr2=0x5632385ba7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:18.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.167+0000 7f409b45f740 1 -- 172.21.15.90:0/3276481363 shutdown_connections 2024-04-25T11:44:18.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.167+0000 7f409b45f740 1 -- 172.21.15.90:0/3276481363 wait complete. 2024-04-25T11:44:18.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.168+0000 7f409b45f740 1 -- start start 2024-04-25T11:44:18.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.168+0000 7f409b45f740 1 Processor -- start 2024-04-25T11:44:18.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.168+0000 7f409b45f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5632381f14d0 0x5632385cd580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:18.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.168+0000 7f409b45f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5632381f96b0 0x5632385cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:18.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.168+0000 7f409b45f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323821a390 0x5632385ca9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:18.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.168+0000 7f409b45f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f408c1b0f50 con 0x56323821a390 2024-04-25T11:44:18.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.168+0000 7f409b45f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f408c1b0880 con 0x5632381f96b0 2024-04-25T11:44:18.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.168+0000 7f409b45f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f408c1adae0 con 0x5632381f14d0 2024-04-25T11:44:18.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.168+0000 7f409b45f740 10 client.0 _ll_register_callbacks cb 0x7f407c0045b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:18.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.168+0000 7f4092b57640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323821a390 0x5632385ca9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:18.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.168+0000 7f4092b57640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323821a390 0x5632385ca9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47678/0 (socket says 172.21.15.90:47678) 2024-04-25T11:44:18.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.168+0000 7f4092b57640 1 -- 172.21.15.90:0/3153790499 learned_addr learned my addr 172.21.15.90:0/3153790499 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:18.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.168+0000 7f4092356640 1 --2- 172.21.15.90:0/3153790499 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5632381f14d0 0x5632385cd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:18.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.168+0000 7f4091b55640 1 --2- 172.21.15.90:0/3153790499 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5632381f96b0 0x5632385cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:18.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.168+0000 7f4092b57640 1 -- 172.21.15.90:0/3153790499 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5632381f14d0 msgr2=0x5632385cd580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:18.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.168+0000 7f4092b57640 1 --2- 172.21.15.90:0/3153790499 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5632381f14d0 0x5632385cd580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.168+0000 7f4092b57640 1 -- 172.21.15.90:0/3153790499 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5632381f96b0 msgr2=0x5632385cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:18.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.168+0000 7f4092b57640 1 --2- 172.21.15.90:0/3153790499 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5632381f96b0 0x5632385cdac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.168+0000 7f4092b57640 1 -- 172.21.15.90:0/3153790499 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632384e14b0 con 0x56323821a390 2024-04-25T11:44:18.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.168+0000 7f4092356640 1 --2- 172.21.15.90:0/3153790499 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5632381f14d0 0x5632385cd580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:18.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.169+0000 7f4092b57640 1 --2- 172.21.15.90:0/3153790499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323821a390 0x5632385ca9e0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f4084003af0 tx=0x7f4084031f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52f541657b5d3388 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:18.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.169+0000 7f4075ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:18.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.169+0000 7f4075ffb640 1 -- 172.21.15.90:0/3153790499 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4084031860 con 0x56323821a390 2024-04-25T11:44:18.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.169+0000 7f4075ffb640 1 -- 172.21.15.90:0/3153790499 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7f408402f660 con 0x56323821a390 2024-04-25T11:44:18.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.169+0000 7f409b45f740 1 -- 172.21.15.90:0/3153790499 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f407c0a6fc0 con 0x56323821a390 2024-04-25T11:44:18.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.169+0000 7f4075ffb640 1 -- 172.21.15.90:0/3153790499 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f408402f860 con 0x56323821a390 2024-04-25T11:44:18.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.169+0000 7f409b45f740 10 client.18558 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:18.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.169+0000 7f409b45f740 1 -- 172.21.15.90:0/3153790499 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56323863f3c0 con 0x56323821a390 2024-04-25T11:44:18.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.169+0000 7f4075ffb640 1 -- 172.21.15.90:0/3153790499 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f407c0a6fc0 con 0x56323821a390 2024-04-25T11:44:18.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.169+0000 7f4075ffb640 1 -- 172.21.15.90:0/3153790499 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f4084043020 con 0x56323821a390 2024-04-25T11:44:18.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.169+0000 7f409b45f740 10 client.18558 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:18.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.169+0000 7f409b45f740 10 client.18558 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:44:18.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.169+0000 7f409b45f740 20 client.18558 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:18.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.169+0000 7f409b45f740 1 --2- 172.21.15.90:0/3153790499 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5632381ae3a0 0x563238641710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:18.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.169+0000 7f409b45f740 4 client.18558 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:44:18.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.169+0000 7f409b45f740 1 -- 172.21.15.90:0/3153790499 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563238528800 con 0x5632381ae3a0 2024-04-25T11:44:18.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.170+0000 7f4092356640 1 --2- 172.21.15.90:0/3153790499 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5632381ae3a0 0x563238641710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:18.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.170+0000 7f4075ffb640 10 client.18558 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:18.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.170+0000 7f4092356640 1 --2- 172.21.15.90:0/3153790499 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5632381ae3a0 0x563238641710 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ad849a890383a0c server_cookie=5567ccffcde9a034 in_seq=0 out_seq=0 2024-04-25T11:44:18.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.170+0000 7f4075ffb640 1 -- 172.21.15.90:0/3153790499 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563238528800 con 0x5632381ae3a0 2024-04-25T11:44:18.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.170+0000 7f4075ffb640 10 client.18558 handle_command_reply: tid=0 2024-04-25T11:44:18.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.170+0000 7f409b45f740 1 client.18558 shutdown 2024-04-25T11:44:18.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.170+0000 7f409b45f740 1 -- 172.21.15.90:0/3153790499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323821a390 msgr2=0x5632385ca9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:18.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.171+0000 7f409b45f740 1 --2- 172.21.15.90:0/3153790499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323821a390 0x5632385ca9e0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f4084003af0 tx=0x7f4084031f80 comp rx=0 tx=0).stop 2024-04-25T11:44:18.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.171+0000 7f409b45f740 1 -- 172.21.15.90:0/3153790499 shutdown_connections 2024-04-25T11:44:18.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.171+0000 7f409b45f740 1 --2- 172.21.15.90:0/3153790499 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5632381ae3a0 0x563238641710 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.171+0000 7f409b45f740 1 --2- 172.21.15.90:0/3153790499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323821a390 0x5632385ca9e0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.171+0000 7f409b45f740 1 --2- 172.21.15.90:0/3153790499 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5632381f96b0 0x5632385cdac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.171+0000 7f4075ffb640 0 client.18558 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:18.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.171+0000 7f409b45f740 1 --2- 172.21.15.90:0/3153790499 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5632381f14d0 0x5632385cd580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.171+0000 7f409b45f740 1 -- 172.21.15.90:0/3153790499 >> 172.21.15.90:0/3153790499 conn(0x56323820e4e0 msgr2=0x5632385b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:18.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.171+0000 7f409b45f740 1 -- 172.21.15.90:0/3153790499 shutdown_connections 2024-04-25T11:44:18.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.171+0000 7f409b45f740 1 -- 172.21.15.90:0/3153790499 wait complete. 2024-04-25T11:44:18.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.171+0000 7f409b45f740 20 client.18558 trim_cache size 0 max 16384 2024-04-25T11:44:18.165 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:44:18.166 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T11:44:18.166 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-25T11:44:18.166 INFO:teuthology.orchestra.run.smithi090.stdout: "3628b186-9847-43ff-a836-387a6f10ef9d": { 2024-04-25T11:44:18.166 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-25T11:44:18.166 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "3628b186-9847-43ff-a836-387a6f10ef9d", 2024-04-25T11:44:18.166 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-25T11:44:18.166 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:44:18.166 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:44:18.166 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:44:18.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.171+0000 7f40945ba640 1 -- 172.21.15.90:0/21956871 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f40780afce0 msgr2=0x7f40780b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:18.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.171+0000 7f40945ba640 1 --2- 172.21.15.90:0/21956871 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f40780afce0 0x7f40780b21f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x5632385bc230 tx=0x7f4088005f90 comp rx=0 tx=0).stop 2024-04-25T11:44:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.171+0000 7f40945ba640 1 -- 172.21.15.90:0/21956871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f408c108a40 msgr2=0x7f408c19ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.171+0000 7f40945ba640 1 --2- 172.21.15.90:0/21956871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f408c108a40 0x7f408c19ba20 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f407c00e9f0 tx=0x7f407c00eeb0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.171+0000 7f40945ba640 1 -- 172.21.15.90:0/21956871 shutdown_connections 2024-04-25T11:44:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.171+0000 7f40945ba640 1 --2- 172.21.15.90:0/21956871 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f40780afce0 0x7f40780b21f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.171+0000 7f40945ba640 1 --2- 172.21.15.90:0/21956871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f408c10d3a0 0x7f408c19bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.171+0000 7f40945ba640 1 --2- 172.21.15.90:0/21956871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f408c108a40 0x7f408c19ba20 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.171+0000 7f40945ba640 1 --2- 172.21.15.90:0/21956871 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f408c102b10 0x7f408c19b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.171+0000 7f40945ba640 1 -- 172.21.15.90:0/21956871 >> 172.21.15.90:0/21956871 conn(0x7f408c078180 msgr2=0x7f408c1040d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:18.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.171+0000 7f40945ba640 1 -- 172.21.15.90:0/21956871 shutdown_connections 2024-04-25T11:44:18.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:18.171+0000 7f40945ba640 1 -- 172.21.15.90:0/21956871 wait complete. 2024-04-25T11:44:18.170 INFO:tasks.cephfs.filesystem:scrub status for tag:3628b186-9847-43ff-a836-387a6f10ef9d - {'path': '/', 'tag': '3628b186-9847-43ff-a836-387a6f10ef9d', 'options': 'recursive,force'} 2024-04-25T11:44:18.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:18 smithi156 ceph-mon[33045]: scrub queued for path: / 2024-04-25T11:44:18.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:18 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:44:18.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:18 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:44:18.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:18 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:44:18.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:18 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:44:18.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:18 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:44:18.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:18 smithi090 ceph-mon[29575]: scrub queued for path: / 2024-04-25T11:44:18.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:18 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:44:18.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:18 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:44:18.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:18 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:44:18.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:18 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:44:18.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:18 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:44:19.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:18 smithi138 ceph-mon[32328]: scrub queued for path: / 2024-04-25T11:44:19.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:18 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:44:19.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:18 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:44:19.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:18 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:44:19.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:18 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:44:19.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:18 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:44:19.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:19 smithi156 ceph-mon[33045]: pgmap v285: 97 pgs: 18 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 48 active+clean; 44 GiB data, 136 GiB used, 937 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.3 KiB/s wr, 9 op/s 2024-04-25T11:44:19.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:19 smithi090 ceph-mon[29575]: pgmap v285: 97 pgs: 18 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 48 active+clean; 44 GiB data, 136 GiB used, 937 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.3 KiB/s wr, 9 op/s 2024-04-25T11:44:20.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:19 smithi138 ceph-mon[32328]: pgmap v285: 97 pgs: 18 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 48 active+clean; 44 GiB data, 136 GiB used, 937 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.3 KiB/s wr, 9 op/s 2024-04-25T11:44:21.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:21 smithi156 ceph-mon[33045]: pgmap v286: 97 pgs: 7 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 75 active+clean; 42 GiB data, 129 GiB used, 944 GiB / 1.0 TiB avail; 11 KiB/s rd, 3.3 KiB/s wr, 15 op/s 2024-04-25T11:44:21.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:21 smithi090 ceph-mon[29575]: pgmap v286: 97 pgs: 7 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 75 active+clean; 42 GiB data, 129 GiB used, 944 GiB / 1.0 TiB avail; 11 KiB/s rd, 3.3 KiB/s wr, 15 op/s 2024-04-25T11:44:22.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:21 smithi138 ceph-mon[32328]: pgmap v286: 97 pgs: 7 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 75 active+clean; 42 GiB data, 129 GiB used, 944 GiB / 1.0 TiB avail; 11 KiB/s rd, 3.3 KiB/s wr, 15 op/s 2024-04-25T11:44:23.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:23 smithi156 ceph-mon[33045]: pgmap v287: 97 pgs: 3 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 89 active+clean; 41 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-04-25T11:44:23.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:23 smithi090 ceph-mon[29575]: pgmap v287: 97 pgs: 3 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 89 active+clean; 41 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-04-25T11:44:24.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:23 smithi138 ceph-mon[32328]: pgmap v287: 97 pgs: 3 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 89 active+clean; 41 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-04-25T11:44:24.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:24 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:44:24.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:24 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:44:24.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:24 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:44:24.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:24 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:44:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.884+0000 7fe0b5067640 1 -- 172.21.15.90:0/3582317197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0b010e5b0 msgr2=0x7fe0b01109d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.884+0000 7fe0b5067640 1 --2- 172.21.15.90:0/3582317197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0b010e5b0 0x7fe0b01109d0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8009ca0 tx=0x7fe0a802a8c0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.885+0000 7fe0b5067640 1 -- 172.21.15.90:0/3582317197 shutdown_connections 2024-04-25T11:44:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.885+0000 7fe0b5067640 1 --2- 172.21.15.90:0/3582317197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0b010e5b0 0x7fe0b01109d0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.885+0000 7fe0b5067640 1 --2- 172.21.15.90:0/3582317197 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe0b01020b0 0x7fe0b01024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.885+0000 7fe0b5067640 1 --2- 172.21.15.90:0/3582317197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe0b0109c00 0x7fe0b0101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.885+0000 7fe0b5067640 1 -- 172.21.15.90:0/3582317197 >> 172.21.15.90:0/3582317197 conn(0x7fe0b00fd8a0 msgr2=0x7fe0b00ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.885+0000 7fe0b5067640 1 -- 172.21.15.90:0/3582317197 shutdown_connections 2024-04-25T11:44:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.885+0000 7fe0b5067640 1 -- 172.21.15.90:0/3582317197 wait complete. 2024-04-25T11:44:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.885+0000 7fe0b5067640 1 Processor -- start 2024-04-25T11:44:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.885+0000 7fe0b5067640 1 -- start start 2024-04-25T11:44:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.886+0000 7fe0b5067640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe0b01020b0 0x7fe0b019a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.886+0000 7fe0b5067640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0b0109c00 0x7fe0b019ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.886+0000 7fe0b5067640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe0b010e5b0 0x7fe0b019ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.886+0000 7fe0b5067640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b0113d70 con 0x7fe0b0109c00 2024-04-25T11:44:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.886+0000 7fe0b5067640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b0113bf0 con 0x7fe0b01020b0 2024-04-25T11:44:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.886+0000 7fe0b5067640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b0113ef0 con 0x7fe0b010e5b0 2024-04-25T11:44:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.886+0000 7fe0a7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0b0109c00 0x7fe0b019ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.886+0000 7fe0a7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0b0109c00 0x7fe0b019ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51414/0 (socket says 172.21.15.90:51414) 2024-04-25T11:44:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.886+0000 7fe0a7fff640 1 -- 172.21.15.90:0/3767348097 learned_addr learned my addr 172.21.15.90:0/3767348097 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.886+0000 7fe0af59e640 1 --2- 172.21.15.90:0/3767348097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe0b010e5b0 0x7fe0b019ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.886+0000 7fe0aed9d640 1 --2- 172.21.15.90:0/3767348097 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe0b01020b0 0x7fe0b019a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.886+0000 7fe0a7fff640 1 -- 172.21.15.90:0/3767348097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe0b010e5b0 msgr2=0x7fe0b019ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.886+0000 7fe0a7fff640 1 --2- 172.21.15.90:0/3767348097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe0b010e5b0 0x7fe0b019ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.875 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.886+0000 7fe0a7fff640 1 -- 172.21.15.90:0/3767348097 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe0b01020b0 msgr2=0x7fe0b019a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:24.875 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.886+0000 7fe0a7fff640 1 --2- 172.21.15.90:0/3767348097 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe0b01020b0 0x7fe0b019a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.875 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.886+0000 7fe0a7fff640 1 -- 172.21.15.90:0/3767348097 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0b019f590 con 0x7fe0b0109c00 2024-04-25T11:44:24.875 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.886+0000 7fe0a7fff640 1 --2- 172.21.15.90:0/3767348097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0b0109c00 0x7fe0b019ac90 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fe09800d770 tx=0x7fe09800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:24.875 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.886+0000 7fe0acd99640 1 -- 172.21.15.90:0/3767348097 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe098017070 con 0x7fe0b0109c00 2024-04-25T11:44:24.875 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.886+0000 7fe0acd99640 1 -- 172.21.15.90:0/3767348097 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe098009c30 con 0x7fe0b0109c00 2024-04-25T11:44:24.875 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.887+0000 7fe0b5067640 1 -- 172.21.15.90:0/3767348097 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0b019f880 con 0x7fe0b0109c00 2024-04-25T11:44:24.875 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.887+0000 7fe0acd99640 1 -- 172.21.15.90:0/3767348097 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09800f620 con 0x7fe0b0109c00 2024-04-25T11:44:24.875 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.887+0000 7fe0b5067640 1 -- 172.21.15.90:0/3767348097 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0b01a7200 con 0x7fe0b0109c00 2024-04-25T11:44:24.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.889+0000 7fe0bbffa740 1 Processor -- start 2024-04-25T11:44:24.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.889+0000 7fe0acd99640 1 -- 172.21.15.90:0/3767348097 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fe09801a070 con 0x7fe0b0109c00 2024-04-25T11:44:24.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.889+0000 7fe0bbffa740 1 -- start start 2024-04-25T11:44:24.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.890+0000 7fe0bbffa740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562787fcf1b0 0x56278836eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:24.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.890+0000 7fe0bbffa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fa62f0 0x56278836f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:24.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.890+0000 7fe0bbffa740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562787fae4d0 0x56278836f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:24.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.890+0000 7fe0bbffa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5627882d5460 con 0x562787fa62f0 2024-04-25T11:44:24.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.890+0000 7fe0bbffa740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56278828f590 con 0x562787fae4d0 2024-04-25T11:44:24.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.890+0000 7fe0bbffa740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5627882d1420 con 0x562787fcf1b0 2024-04-25T11:44:24.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.890+0000 7fe0aed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fa62f0 0x56278836f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:24.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.890+0000 7fe0a7fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562787fae4d0 0x56278836f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:24.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.890+0000 7fe0aed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fa62f0 0x56278836f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51428/0 (socket says 172.21.15.90:51428) 2024-04-25T11:44:24.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.890+0000 7fe0aed9d640 1 -- 172.21.15.90:0/526623962 learned_addr learned my addr 172.21.15.90:0/526623962 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:24.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.890+0000 7fe0af59e640 1 --2- 172.21.15.90:0/526623962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562787fcf1b0 0x56278836eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:24.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.890+0000 7fe0a7fff640 1 -- 172.21.15.90:0/526623962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562787fcf1b0 msgr2=0x56278836eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:24.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.890+0000 7fe0a7fff640 1 --2- 172.21.15.90:0/526623962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562787fcf1b0 0x56278836eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.890+0000 7fe0a7fff640 1 -- 172.21.15.90:0/526623962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fa62f0 msgr2=0x56278836f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:24.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.890+0000 7fe0a7fff640 1 --2- 172.21.15.90:0/526623962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fa62f0 0x56278836f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.890+0000 7fe0acd99640 1 --2- 172.21.15.90:0/3767348097 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe09c0afc50 0x7fe09c0b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:24.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.890+0000 7fe0a7fff640 1 -- 172.21.15.90:0/526623962 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56278829c0d0 con 0x562787fae4d0 2024-04-25T11:44:24.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.890+0000 7fe0aed9d640 1 --2- 172.21.15.90:0/526623962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fa62f0 0x56278836f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:24.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.890+0000 7fe0af59e640 1 --2- 172.21.15.90:0/526623962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562787fcf1b0 0x56278836eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:44:24.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.891+0000 7fe0af59e640 1 --2- 172.21.15.90:0/3767348097 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe09c0afc50 0x7fe09c0b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:24.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.891+0000 7fe0acd99640 1 -- 172.21.15.90:0/3767348097 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7fe0980deb80 con 0x7fe0b0109c00 2024-04-25T11:44:24.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.891+0000 7fe0a7fff640 1 --2- 172.21.15.90:0/526623962 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562787fae4d0 0x56278836f570 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe0980dc220 tx=0x7fe0980e76c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=109dddc3161d3610 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:24.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.891+0000 7fe0af59e640 1 --2- 172.21.15.90:0/3767348097 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe09c0afc50 0x7fe09c0b2160 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x56278836bc60 tx=0x7fe0a8009480 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:24.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.891+0000 7fe0a57fa640 1 -- 172.21.15.90:0/526623962 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe098021070 con 0x562787fae4d0 2024-04-25T11:44:24.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.891+0000 7fe0a57fa640 1 -- 172.21.15.90:0/526623962 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0980af420 con 0x562787fae4d0 2024-04-25T11:44:24.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.891+0000 7fe0a57fa640 1 -- 172.21.15.90:0/526623962 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09808f070 con 0x562787fae4d0 2024-04-25T11:44:24.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.892+0000 7fe0bbffa740 1 -- 172.21.15.90:0/526623962 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562787fae4d0 msgr2=0x56278836f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:24.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.892+0000 7fe0bbffa740 1 --2- 172.21.15.90:0/526623962 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562787fae4d0 0x56278836f570 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe0980dc220 tx=0x7fe0980e76c0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.892+0000 7fe0bbffa740 1 -- 172.21.15.90:0/526623962 shutdown_connections 2024-04-25T11:44:24.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.892+0000 7fe0bbffa740 1 --2- 172.21.15.90:0/526623962 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562787fae4d0 0x56278836f570 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.892+0000 7fe0bbffa740 1 --2- 172.21.15.90:0/526623962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fa62f0 0x56278836f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.892+0000 7fe0bbffa740 1 --2- 172.21.15.90:0/526623962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562787fcf1b0 0x56278836eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.892+0000 7fe0bbffa740 1 -- 172.21.15.90:0/526623962 >> 172.21.15.90:0/526623962 conn(0x562787fc3300 msgr2=0x56278836b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.892+0000 7fe0bbffa740 1 -- 172.21.15.90:0/526623962 shutdown_connections 2024-04-25T11:44:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.892+0000 7fe0bbffa740 1 -- 172.21.15.90:0/526623962 wait complete. 2024-04-25T11:44:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.893+0000 7fe0bbffa740 1 -- start start 2024-04-25T11:44:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.893+0000 7fe0bbffa740 1 Processor -- start 2024-04-25T11:44:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.893+0000 7fe0bbffa740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562787fa62f0 0x56278837ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.893+0000 7fe0bbffa740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562787fae4d0 0x56278837b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.893+0000 7fe0bbffa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fcf1b0 0x562788378ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.893+0000 7fe0bbffa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56278837bbc0 con 0x562787fcf1b0 2024-04-25T11:44:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.893+0000 7fe0bbffa740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56278825cb60 con 0x562787fa62f0 2024-04-25T11:44:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.893+0000 7fe0bbffa740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56278837bd40 con 0x562787fae4d0 2024-04-25T11:44:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.893+0000 7fe0bbffa740 10 client.0 _ll_register_callbacks cb 0x5627882812b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.893+0000 7fe0a7fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562787fae4d0 0x56278837b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.893+0000 7fe0af59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fcf1b0 0x562788378ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.893+0000 7fe0a7fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562787fae4d0 0x56278837b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:51230/0 (socket says 172.21.15.90:51230) 2024-04-25T11:44:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.893+0000 7fe0a7fff640 1 -- 172.21.15.90:0/1561284206 learned_addr learned my addr 172.21.15.90:0/1561284206 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.893+0000 7fe0aed9d640 1 --2- 172.21.15.90:0/1561284206 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562787fa62f0 0x56278837ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.893+0000 7fe0af59e640 1 -- 172.21.15.90:0/1561284206 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562787fae4d0 msgr2=0x56278837b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.893+0000 7fe0af59e640 1 --2- 172.21.15.90:0/1561284206 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562787fae4d0 0x56278837b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.894+0000 7fe0af59e640 1 -- 172.21.15.90:0/1561284206 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562787fa62f0 msgr2=0x56278837ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.894+0000 7fe0af59e640 1 --2- 172.21.15.90:0/1561284206 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562787fa62f0 0x56278837ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.894+0000 7fe0af59e640 1 -- 172.21.15.90:0/1561284206 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562787fe7210 con 0x562787fcf1b0 2024-04-25T11:44:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.894+0000 7fe0a7fff640 1 --2- 172.21.15.90:0/1561284206 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562787fae4d0 0x56278837b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.894+0000 7fe0af59e640 1 --2- 172.21.15.90:0/1561284206 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fcf1b0 0x562788378ff0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8005e30 tx=0x7fe0a802a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb3b95791cba3576 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.894+0000 7fe0967fc640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.894+0000 7fe0967fc640 1 -- 172.21.15.90:0/1561284206 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8003d00 con 0x562787fcf1b0 2024-04-25T11:44:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.894+0000 7fe0967fc640 1 -- 172.21.15.90:0/1561284206 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7fe0a8004040 con 0x562787fcf1b0 2024-04-25T11:44:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.894+0000 7fe0bbffa740 1 -- 172.21.15.90:0/1561284206 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0980af420 con 0x562787fcf1b0 2024-04-25T11:44:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.894+0000 7fe0967fc640 1 -- 172.21.15.90:0/1561284206 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe0a8004240 con 0x562787fcf1b0 2024-04-25T11:44:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.894+0000 7fe0bbffa740 10 client.18570 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.894+0000 7fe0bbffa740 1 -- 172.21.15.90:0/1561284206 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5627880b3620 con 0x562787fcf1b0 2024-04-25T11:44:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.894+0000 7fe0967fc640 1 -- 172.21.15.90:0/1561284206 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0980af420 con 0x562787fcf1b0 2024-04-25T11:44:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.894+0000 7fe0967fc640 1 -- 172.21.15.90:0/1561284206 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fe0a8051070 con 0x562787fcf1b0 2024-04-25T11:44:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.894+0000 7fe0bbffa740 10 client.18570 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.894+0000 7fe0bbffa740 10 client.18570 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:44:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.894+0000 7fe0bbffa740 20 client.18570 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.894+0000 7fe0bbffa740 1 --2- 172.21.15.90:0/1561284206 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562787f8dd80 0x56278838b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.894+0000 7fe0bbffa740 4 client.18570 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:44:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.894+0000 7fe0bbffa740 1 -- 172.21.15.90:0/1561284206 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627882e3070 con 0x562787f8dd80 2024-04-25T11:44:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.895+0000 7fe0aed9d640 1 --2- 172.21.15.90:0/1561284206 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562787f8dd80 0x56278838b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.895+0000 7fe0967fc640 10 client.18570 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.895+0000 7fe0aed9d640 1 --2- 172.21.15.90:0/1561284206 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562787f8dd80 0x56278838b700 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4e3fa4072f81ee2 server_cookie=c524da14d842e222 in_seq=0 out_seq=0 2024-04-25T11:44:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.896+0000 7fe0967fc640 1 -- 172.21.15.90:0/1561284206 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5627882e3070 con 0x562787f8dd80 2024-04-25T11:44:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.896+0000 7fe0967fc640 10 client.18570 handle_command_reply: tid=0 2024-04-25T11:44:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.896+0000 7fe0bbffa740 1 client.18570 shutdown 2024-04-25T11:44:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.896+0000 7fe0bbffa740 1 -- 172.21.15.90:0/1561284206 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fcf1b0 msgr2=0x562788378ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.896+0000 7fe0bbffa740 1 --2- 172.21.15.90:0/1561284206 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fcf1b0 0x562788378ff0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8005e30 tx=0x7fe0a802a8c0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.896+0000 7fe0bbffa740 1 -- 172.21.15.90:0/1561284206 shutdown_connections 2024-04-25T11:44:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.896+0000 7fe0bbffa740 1 --2- 172.21.15.90:0/1561284206 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562787f8dd80 0x56278838b700 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.896+0000 7fe0bbffa740 1 --2- 172.21.15.90:0/1561284206 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fcf1b0 0x562788378ff0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.896+0000 7fe0bbffa740 1 --2- 172.21.15.90:0/1561284206 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562787fae4d0 0x56278837b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.896+0000 7fe0967fc640 0 client.18570 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.896+0000 7fe0bbffa740 1 --2- 172.21.15.90:0/1561284206 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562787fa62f0 0x56278837ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.896+0000 7fe0bbffa740 1 -- 172.21.15.90:0/1561284206 >> 172.21.15.90:0/1561284206 conn(0x562787fc3300 msgr2=0x562788370810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.896+0000 7fe0bbffa740 1 -- 172.21.15.90:0/1561284206 shutdown_connections 2024-04-25T11:44:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.896+0000 7fe0bbffa740 1 -- 172.21.15.90:0/1561284206 wait complete. 2024-04-25T11:44:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.896+0000 7fe0bbffa740 20 client.18570 trim_cache size 0 max 16384 2024-04-25T11:44:24.896 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.911+0000 7fe0bbffa740 1 Processor -- start 2024-04-25T11:44:24.896 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.911+0000 7fe0bbffa740 1 -- start start 2024-04-25T11:44:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.911+0000 7fe0bbffa740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562787f8dd80 0x562788370eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.911+0000 7fe0bbffa740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562787fa62f0 0x5627883727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.911+0000 7fe0bbffa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fae4d0 0x562788372ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.911+0000 7fe0bbffa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b01afd00 con 0x562787fae4d0 2024-04-25T11:44:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.911+0000 7fe0bbffa740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b01b1840 con 0x562787fa62f0 2024-04-25T11:44:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.911+0000 7fe0bbffa740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b01b03d0 con 0x562787f8dd80 2024-04-25T11:44:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.911+0000 7fe0af59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fae4d0 0x562788372ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.911+0000 7fe0af59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fae4d0 0x562788372ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51440/0 (socket says 172.21.15.90:51440) 2024-04-25T11:44:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.911+0000 7fe0af59e640 1 -- 172.21.15.90:0/3083756 learned_addr learned my addr 172.21.15.90:0/3083756 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.912+0000 7fe0a7fff640 1 --2- 172.21.15.90:0/3083756 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562787fa62f0 0x5627883727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:24.902 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.912+0000 7fe0aed9d640 1 --2- 172.21.15.90:0/3083756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562787f8dd80 0x562788370eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.912+0000 7fe0af59e640 1 -- 172.21.15.90:0/3083756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562787f8dd80 msgr2=0x562788370eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.912+0000 7fe0af59e640 1 --2- 172.21.15.90:0/3083756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562787f8dd80 0x562788370eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.912+0000 7fe0af59e640 1 -- 172.21.15.90:0/3083756 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562787fa62f0 msgr2=0x5627883727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.912+0000 7fe0af59e640 1 --2- 172.21.15.90:0/3083756 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562787fa62f0 0x5627883727a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.912+0000 7fe0af59e640 1 -- 172.21.15.90:0/3083756 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56278836bc90 con 0x562787fae4d0 2024-04-25T11:44:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.912+0000 7fe0aed9d640 1 --2- 172.21.15.90:0/3083756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562787f8dd80 0x562788370eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.912+0000 7fe0a7fff640 1 --2- 172.21.15.90:0/3083756 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562787fa62f0 0x5627883727a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.912+0000 7fe0af59e640 1 --2- 172.21.15.90:0/3083756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fae4d0 0x562788372ce0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fe0a802ade0 tx=0x7fe0a8009960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd76e2071a0d935 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.912+0000 7fe0a4ff9640 1 -- 172.21.15.90:0/3083756 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a80318e0 con 0x562787fae4d0 2024-04-25T11:44:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.912+0000 7fe0a4ff9640 1 -- 172.21.15.90:0/3083756 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0a8036050 con 0x562787fae4d0 2024-04-25T11:44:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.912+0000 7fe0a4ff9640 1 -- 172.21.15.90:0/3083756 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a80043e0 con 0x562787fae4d0 2024-04-25T11:44:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.913+0000 7fe0bbffa740 1 -- 172.21.15.90:0/3083756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fae4d0 msgr2=0x562788372ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.913+0000 7fe0bbffa740 1 --2- 172.21.15.90:0/3083756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fae4d0 0x562788372ce0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fe0a802ade0 tx=0x7fe0a8009960 comp rx=0 tx=0).stop 2024-04-25T11:44:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.913+0000 7fe0bbffa740 1 -- 172.21.15.90:0/3083756 shutdown_connections 2024-04-25T11:44:24.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.913+0000 7fe0bbffa740 1 --2- 172.21.15.90:0/3083756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fae4d0 0x562788372ce0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.913+0000 7fe0bbffa740 1 --2- 172.21.15.90:0/3083756 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562787fa62f0 0x5627883727a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.913+0000 7fe0bbffa740 1 --2- 172.21.15.90:0/3083756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562787f8dd80 0x562788370eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.913+0000 7fe0bbffa740 1 -- 172.21.15.90:0/3083756 >> 172.21.15.90:0/3083756 conn(0x562787fc3300 msgr2=0x5627883707a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.913+0000 7fe0bbffa740 1 -- 172.21.15.90:0/3083756 shutdown_connections 2024-04-25T11:44:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.913+0000 7fe0bbffa740 1 -- 172.21.15.90:0/3083756 wait complete. 2024-04-25T11:44:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.913+0000 7fe0bbffa740 1 -- start start 2024-04-25T11:44:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.913+0000 7fe0bbffa740 1 Processor -- start 2024-04-25T11:44:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.913+0000 7fe0bbffa740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562787f8dd80 0x562788381340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.913+0000 7fe0bbffa740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562787fa62f0 0x56278837e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.913+0000 7fe0bbffa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fae4d0 0x56278837ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.913+0000 7fe0bbffa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b01b1170 con 0x562787fae4d0 2024-04-25T11:44:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.914+0000 7fe0bbffa740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b01b0aa0 con 0x562787fa62f0 2024-04-25T11:44:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.914+0000 7fe0bbffa740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b01add00 con 0x562787f8dd80 2024-04-25T11:44:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.914+0000 7fe0bbffa740 10 client.0 _ll_register_callbacks cb 0x7fe0a8009360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.914+0000 7fe0af59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fae4d0 0x56278837ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.914+0000 7fe0af59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fae4d0 0x56278837ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51450/0 (socket says 172.21.15.90:51450) 2024-04-25T11:44:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.914+0000 7fe0af59e640 1 -- 172.21.15.90:0/2007339344 learned_addr learned my addr 172.21.15.90:0/2007339344 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.914+0000 7fe0aed9d640 1 --2- 172.21.15.90:0/2007339344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562787f8dd80 0x562788381340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.914+0000 7fe0a7fff640 1 --2- 172.21.15.90:0/2007339344 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562787fa62f0 0x56278837e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.914+0000 7fe0af59e640 1 -- 172.21.15.90:0/2007339344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562787f8dd80 msgr2=0x562788381340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.914+0000 7fe0af59e640 1 --2- 172.21.15.90:0/2007339344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562787f8dd80 0x562788381340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.914+0000 7fe0af59e640 1 -- 172.21.15.90:0/2007339344 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562787fa62f0 msgr2=0x56278837e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.914+0000 7fe0af59e640 1 --2- 172.21.15.90:0/2007339344 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562787fa62f0 0x56278837e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.914+0000 7fe0af59e640 1 -- 172.21.15.90:0/2007339344 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56278837f130 con 0x562787fae4d0 2024-04-25T11:44:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.914+0000 7fe0af59e640 1 --2- 172.21.15.90:0/2007339344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fae4d0 0x56278837ebf0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fe0a802a900 tx=0x7fe0a8031960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bd4f86266ab885b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.914+0000 7fe0967fc640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.914+0000 7fe0967fc640 1 -- 172.21.15.90:0/2007339344 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8004b40 con 0x562787fae4d0 2024-04-25T11:44:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.914+0000 7fe0967fc640 1 -- 172.21.15.90:0/2007339344 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8702+0+0 (secure 0 0 0) 0x7fe0a8004ce0 con 0x562787fae4d0 2024-04-25T11:44:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.914+0000 7fe0bbffa740 1 -- 172.21.15.90:0/2007339344 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0a8036050 con 0x562787fae4d0 2024-04-25T11:44:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.914+0000 7fe0967fc640 1 -- 172.21.15.90:0/2007339344 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe0a8070050 con 0x562787fae4d0 2024-04-25T11:44:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.914+0000 7fe0bbffa740 10 client.18576 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.914+0000 7fe0bbffa740 1 -- 172.21.15.90:0/2007339344 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5627883f3680 con 0x562787fae4d0 2024-04-25T11:44:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.915+0000 7fe0967fc640 1 -- 172.21.15.90:0/2007339344 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0a8036050 con 0x562787fae4d0 2024-04-25T11:44:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.915+0000 7fe0967fc640 1 -- 172.21.15.90:0/2007339344 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fe0a806b030 con 0x562787fae4d0 2024-04-25T11:44:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.915+0000 7fe0bbffa740 10 client.18576 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.915+0000 7fe0bbffa740 10 client.18576 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:44:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.915+0000 7fe0bbffa740 20 client.18576 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.915+0000 7fe0bbffa740 1 --2- 172.21.15.90:0/2007339344 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562787fcf1b0 0x5627883f5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.915+0000 7fe0bbffa740 4 client.18576 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:44:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.915+0000 7fe0bbffa740 1 -- 172.21.15.90:0/2007339344 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5627882dcee0 con 0x562787fcf1b0 2024-04-25T11:44:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.915+0000 7fe0aed9d640 1 --2- 172.21.15.90:0/2007339344 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562787fcf1b0 0x5627883f5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:24.907 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.915+0000 7fe0967fc640 10 client.18576 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:24.907 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.915+0000 7fe0aed9d640 1 --2- 172.21.15.90:0/2007339344 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562787fcf1b0 0x5627883f5d10 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7870691e564ac21a server_cookie=e288a208e66cea75 in_seq=0 out_seq=0 2024-04-25T11:44:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.921+0000 7fe0967fc640 1 -- 172.21.15.90:0/2007339344 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5627882dcee0 con 0x562787fcf1b0 2024-04-25T11:44:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.921+0000 7fe0967fc640 10 client.18576 handle_command_reply: tid=0 2024-04-25T11:44:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.921+0000 7fe0bbffa740 1 client.18576 shutdown 2024-04-25T11:44:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.921+0000 7fe0bbffa740 1 -- 172.21.15.90:0/2007339344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fae4d0 msgr2=0x56278837ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.921+0000 7fe0bbffa740 1 --2- 172.21.15.90:0/2007339344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fae4d0 0x56278837ebf0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fe0a802a900 tx=0x7fe0a8031960 comp rx=0 tx=0).stop 2024-04-25T11:44:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.921+0000 7fe0bbffa740 1 -- 172.21.15.90:0/2007339344 shutdown_connections 2024-04-25T11:44:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.921+0000 7fe0bbffa740 1 --2- 172.21.15.90:0/2007339344 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562787fcf1b0 0x5627883f5d10 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.921+0000 7fe0bbffa740 1 --2- 172.21.15.90:0/2007339344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562787fae4d0 0x56278837ebf0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.921+0000 7fe0bbffa740 1 --2- 172.21.15.90:0/2007339344 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562787fa62f0 0x56278837e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.921+0000 7fe0967fc640 0 client.18576 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.921+0000 7fe0bbffa740 1 --2- 172.21.15.90:0/2007339344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562787f8dd80 0x562788381340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.921+0000 7fe0bbffa740 1 -- 172.21.15.90:0/2007339344 >> 172.21.15.90:0/2007339344 conn(0x562787fc3300 msgr2=0x562788378880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.921+0000 7fe0bbffa740 1 -- 172.21.15.90:0/2007339344 shutdown_connections 2024-04-25T11:44:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.921+0000 7fe0bbffa740 1 -- 172.21.15.90:0/2007339344 wait complete. 2024-04-25T11:44:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.921+0000 7fe0bbffa740 20 client.18576 trim_cache size 0 max 16384 2024-04-25T11:44:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:44:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:44:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:44:24.910 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:44:24.910 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.922+0000 7fe0b5067640 1 -- 172.21.15.90:0/3767348097 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe09c0afc50 msgr2=0x7fe09c0b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.922+0000 7fe0b5067640 1 --2- 172.21.15.90:0/3767348097 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe09c0afc50 0x7fe09c0b2160 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x56278836bc60 tx=0x7fe0a8009480 comp rx=0 tx=0).stop 2024-04-25T11:44:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.922+0000 7fe0b5067640 1 -- 172.21.15.90:0/3767348097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0b0109c00 msgr2=0x7fe0b019ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.922+0000 7fe0b5067640 1 --2- 172.21.15.90:0/3767348097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0b0109c00 0x7fe0b019ac90 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fe09800d770 tx=0x7fe09800dc30 comp rx=0 tx=0).stop 2024-04-25T11:44:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.922+0000 7fe0b5067640 1 -- 172.21.15.90:0/3767348097 shutdown_connections 2024-04-25T11:44:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.922+0000 7fe0b5067640 1 --2- 172.21.15.90:0/3767348097 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe09c0afc50 0x7fe09c0b2160 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.922+0000 7fe0b5067640 1 --2- 172.21.15.90:0/3767348097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe0b010e5b0 0x7fe0b019ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.922+0000 7fe0b5067640 1 --2- 172.21.15.90:0/3767348097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0b0109c00 0x7fe0b019ac90 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.922+0000 7fe0b5067640 1 --2- 172.21.15.90:0/3767348097 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe0b01020b0 0x7fe0b019a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.922+0000 7fe0b5067640 1 -- 172.21.15.90:0/3767348097 >> 172.21.15.90:0/3767348097 conn(0x7fe0b00fd8a0 msgr2=0x7fe0b00fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.922+0000 7fe0b5067640 1 -- 172.21.15.90:0/3767348097 shutdown_connections 2024-04-25T11:44:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:24.922+0000 7fe0b5067640 1 -- 172.21.15.90:0/3767348097 wait complete. 2024-04-25T11:44:24.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:24 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:44:24.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:24 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:44:24.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:24 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:44:24.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:24 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:44:25.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:24 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:44:25.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:24 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:44:25.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:24 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:44:25.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:24 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:44:25.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:25 smithi156 ceph-mon[33045]: pgmap v288: 97 pgs: 3 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 89 active+clean; 41 GiB data, 126 GiB used, 946 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 3.2 KiB/s wr, 12 op/s 2024-04-25T11:44:25.836 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:25 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:44:25.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:25 smithi090 ceph-mon[29575]: pgmap v288: 97 pgs: 3 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 89 active+clean; 41 GiB data, 126 GiB used, 946 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 3.2 KiB/s wr, 12 op/s 2024-04-25T11:44:25.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:25 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:44:26.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:25 smithi138 ceph-mon[32328]: pgmap v288: 97 pgs: 3 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 89 active+clean; 41 GiB data, 126 GiB used, 946 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 3.2 KiB/s wr, 12 op/s 2024-04-25T11:44:26.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:25 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:44:26.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:26 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:44:26.874 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:26 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:44:27.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:26 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:44:27.973 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:27 smithi090 ceph-mon[29575]: pgmap v289: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 12 KiB/s rd, 3.2 KiB/s wr, 15 op/s 2024-04-25T11:44:28.050 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:27 smithi138 ceph-mon[32328]: pgmap v289: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 12 KiB/s rd, 3.2 KiB/s wr, 15 op/s 2024-04-25T11:44:28.085 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:27 smithi156 ceph-mon[33045]: pgmap v289: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 12 KiB/s rd, 3.2 KiB/s wr, 15 op/s 2024-04-25T11:44:30.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:29 smithi090 ceph-mon[29575]: pgmap v290: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 7.2 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-25T11:44:30.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:29 smithi090 ceph-mon[29575]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T11:44:30.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:29 smithi138 ceph-mon[32328]: pgmap v290: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 7.2 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-25T11:44:30.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:29 smithi138 ceph-mon[32328]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T11:44:30.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:29 smithi156 ceph-mon[33045]: pgmap v290: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 7.2 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-25T11:44:30.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:29 smithi156 ceph-mon[33045]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T11:44:32.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:31 smithi090 ceph-mon[29575]: pgmap v292: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:44:32.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:31 smithi138 ceph-mon[32328]: pgmap v292: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:44:32.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:31 smithi156 ceph-mon[33045]: pgmap v292: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:44:34.138 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:33 smithi156 ceph-mon[33045]: pgmap v293: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 8.0 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:44:34.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:33 smithi090 ceph-mon[29575]: pgmap v293: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 8.0 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:44:34.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:33 smithi138 ceph-mon[32328]: pgmap v293: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 8.0 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:44:36.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:35 smithi090 ceph-mon[29575]: pgmap v294: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 8.0 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:44:36.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:35 smithi138 ceph-mon[32328]: pgmap v294: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 8.0 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:44:36.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:35 smithi156 ceph-mon[33045]: pgmap v294: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 8.0 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:44:38.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.031+0000 7f5bdd13f640 1 -- 172.21.15.90:0/917774337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bd8108a90 msgr2=0x7f5bd8102630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:38.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.031+0000 7f5bdd13f640 1 --2- 172.21.15.90:0/917774337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bd8108a90 0x7f5bd8102630 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8009900 tx=0x7f5bc802a410 comp rx=0 tx=0).stop 2024-04-25T11:44:38.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.031+0000 7f5bdd13f640 1 -- 172.21.15.90:0/917774337 shutdown_connections 2024-04-25T11:44:38.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.031+0000 7f5bdd13f640 1 --2- 172.21.15.90:0/917774337 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5bd81031e0 0x7f5bd810f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.031+0000 7f5bdd13f640 1 --2- 172.21.15.90:0/917774337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bd8108f30 0x7f5bd8102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.031+0000 7f5bdd13f640 1 --2- 172.21.15.90:0/917774337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bd8108a90 0x7f5bd8102630 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.031+0000 7f5bdd13f640 1 -- 172.21.15.90:0/917774337 >> 172.21.15.90:0/917774337 conn(0x7f5bd80fe260 msgr2=0x7f5bd8100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:38.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.031+0000 7f5bdd13f640 1 -- 172.21.15.90:0/917774337 shutdown_connections 2024-04-25T11:44:38.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.031+0000 7f5bdd13f640 1 -- 172.21.15.90:0/917774337 wait complete. 2024-04-25T11:44:38.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.032+0000 7f5bdd13f640 1 Processor -- start 2024-04-25T11:44:38.021 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.032+0000 7f5bdd13f640 1 -- start start 2024-04-25T11:44:38.021 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.033+0000 7f5bdd13f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5bd81031e0 0x7f5bd819b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:38.021 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.033+0000 7f5bdd13f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bd8108a90 0x7f5bd819bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:38.021 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.033+0000 7f5bdd13f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bd8108f30 0x7f5bd819c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:38.021 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.033+0000 7f5bdd13f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd8112cd0 con 0x7f5bd8108f30 2024-04-25T11:44:38.021 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.033+0000 7f5bdd13f640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd8112b50 con 0x7f5bd81031e0 2024-04-25T11:44:38.021 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.033+0000 7f5bdd13f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd8112e50 con 0x7f5bd8108a90 2024-04-25T11:44:38.021 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.033+0000 7f5bd759e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bd8108f30 0x7f5bd819c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:38.021 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.033+0000 7f5bd759e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bd8108f30 0x7f5bd819c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59632/0 (socket says 172.21.15.90:59632) 2024-04-25T11:44:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.033+0000 7f5bd759e640 1 -- 172.21.15.90:0/3065401953 learned_addr learned my addr 172.21.15.90:0/3065401953 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.033+0000 7f5bd6d9d640 1 --2- 172.21.15.90:0/3065401953 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5bd81031e0 0x7f5bd819b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.033+0000 7f5bd659c640 1 --2- 172.21.15.90:0/3065401953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bd8108a90 0x7f5bd819bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.033+0000 7f5bd759e640 1 -- 172.21.15.90:0/3065401953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bd8108a90 msgr2=0x7f5bd819bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.033+0000 7f5bd759e640 1 --2- 172.21.15.90:0/3065401953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bd8108a90 0x7f5bd819bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.033+0000 7f5bd759e640 1 -- 172.21.15.90:0/3065401953 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5bd81031e0 msgr2=0x7f5bd819b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.033+0000 7f5bd759e640 1 --2- 172.21.15.90:0/3065401953 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5bd81031e0 0x7f5bd819b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.033+0000 7f5bd759e640 1 -- 172.21.15.90:0/3065401953 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bd81a68b0 con 0x7f5bd8108f30 2024-04-25T11:44:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.033+0000 7f5bd659c640 1 --2- 172.21.15.90:0/3065401953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bd8108a90 0x7f5bd819bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:44:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.033+0000 7f5bd759e640 1 --2- 172.21.15.90:0/3065401953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bd8108f30 0x7f5bd819c0b0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f5bd000cce0 tx=0x7f5bd000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.033+0000 7f5bbffff640 1 -- 172.21.15.90:0/3065401953 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd000f450 con 0x7f5bd8108f30 2024-04-25T11:44:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.033+0000 7f5bdd13f640 1 -- 172.21.15.90:0/3065401953 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bd81a6ba0 con 0x7f5bd8108f30 2024-04-25T11:44:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.034+0000 7f5bbffff640 1 -- 172.21.15.90:0/3065401953 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bd000faa0 con 0x7f5bd8108f30 2024-04-25T11:44:38.023 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.034+0000 7f5bbffff640 1 -- 172.21.15.90:0/3065401953 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0018b70 con 0x7f5bd8108f30 2024-04-25T11:44:38.023 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.034+0000 7f5bdd13f640 1 -- 172.21.15.90:0/3065401953 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bd81a70f0 con 0x7f5bd8108f30 2024-04-25T11:44:38.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.036+0000 7f5bbffff640 1 -- 172.21.15.90:0/3065401953 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f5bd0018d10 con 0x7f5bd8108f30 2024-04-25T11:44:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.036+0000 7f5be3ff4740 1 Processor -- start 2024-04-25T11:44:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.036+0000 7f5be3ff4740 1 -- start start 2024-04-25T11:44:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.037+0000 7f5be3ff4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc907191b0 0x55cc90ab8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.037+0000 7f5be3ff4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc906f02f0 0x55cc90ab9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.037+0000 7f5be3ff4740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc906f84d0 0x55cc90ab9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.037+0000 7f5be3ff4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55cc90a1f460 con 0x55cc907191b0 2024-04-25T11:44:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.037+0000 7f5be3ff4740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55cc909d9590 con 0x55cc906f84d0 2024-04-25T11:44:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.037+0000 7f5be3ff4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55cc90a1b420 con 0x55cc906f02f0 2024-04-25T11:44:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.037+0000 7f5bd659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc907191b0 0x55cc90ab8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.037+0000 7f5bd659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc907191b0 0x55cc90ab8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59640/0 (socket says 172.21.15.90:59640) 2024-04-25T11:44:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.037+0000 7f5bd659c640 1 -- 172.21.15.90:0/583685390 learned_addr learned my addr 172.21.15.90:0/583685390 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.037+0000 7f5bbffff640 1 --2- 172.21.15.90:0/3065401953 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5bc00afc50 0x7f5bc00b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.037+0000 7f5bd6d9d640 1 --2- 172.21.15.90:0/583685390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc906f02f0 0x55cc90ab9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.037+0000 7f5bd659c640 1 --2- 172.21.15.90:0/583685390 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc906f84d0 0x55cc90ab9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.037+0000 7f5bd659c640 1 -- 172.21.15.90:0/583685390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc906f02f0 msgr2=0x55cc90ab9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.037+0000 7f5bd659c640 1 --2- 172.21.15.90:0/583685390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc906f02f0 0x55cc90ab9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.037+0000 7f5bd659c640 1 -- 172.21.15.90:0/583685390 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc906f84d0 msgr2=0x55cc90ab9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.037+0000 7f5bd659c640 1 --2- 172.21.15.90:0/583685390 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc906f84d0 0x55cc90ab9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.037+0000 7f5bd659c640 1 -- 172.21.15.90:0/583685390 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc909e60d0 con 0x55cc907191b0 2024-04-25T11:44:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.037+0000 7f5bd759e640 1 --2- 172.21.15.90:0/3065401953 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5bc00afc50 0x7f5bc00b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.037+0000 7f5bd6d9d640 1 --2- 172.21.15.90:0/583685390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc906f02f0 0x55cc90ab9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:44:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.037+0000 7f5bbffff640 1 -- 172.21.15.90:0/3065401953 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f5bd00df570 con 0x7f5bd8108f30 2024-04-25T11:44:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.037+0000 7f5bd659c640 1 --2- 172.21.15.90:0/583685390 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc907191b0 0x55cc90ab8af0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc00bfc0 tx=0x7f5bcc00b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b69bc90c84b354 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.037+0000 7f5bbd7fa640 1 -- 172.21.15.90:0/583685390 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc019030 con 0x55cc907191b0 2024-04-25T11:44:38.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.038+0000 7f5bbd7fa640 1 -- 172.21.15.90:0/583685390 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bcc0035f0 con 0x55cc907191b0 2024-04-25T11:44:38.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.038+0000 7f5bbd7fa640 1 -- 172.21.15.90:0/583685390 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc011720 con 0x55cc907191b0 2024-04-25T11:44:38.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.038+0000 7f5be3ff4740 1 -- 172.21.15.90:0/583685390 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc907191b0 msgr2=0x55cc90ab8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:38.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.038+0000 7f5bd759e640 1 --2- 172.21.15.90:0/3065401953 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5bc00afc50 0x7f5bc00b2160 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5bd00dd230 tx=0x7f5bd0076040 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:38.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.038+0000 7f5be3ff4740 1 --2- 172.21.15.90:0/583685390 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc907191b0 0x55cc90ab8af0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc00bfc0 tx=0x7f5bcc00b670 comp rx=0 tx=0).stop 2024-04-25T11:44:38.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.038+0000 7f5be3ff4740 1 -- 172.21.15.90:0/583685390 shutdown_connections 2024-04-25T11:44:38.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.038+0000 7f5be3ff4740 1 --2- 172.21.15.90:0/583685390 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc906f84d0 0x55cc90ab9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.038+0000 7f5be3ff4740 1 --2- 172.21.15.90:0/583685390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc906f02f0 0x55cc90ab9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.038+0000 7f5be3ff4740 1 --2- 172.21.15.90:0/583685390 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc907191b0 0x55cc90ab8af0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.038+0000 7f5be3ff4740 1 -- 172.21.15.90:0/583685390 >> 172.21.15.90:0/583685390 conn(0x55cc9070d300 msgr2=0x55cc90ab52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:38.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.039+0000 7f5be3ff4740 1 -- 172.21.15.90:0/583685390 shutdown_connections 2024-04-25T11:44:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.039+0000 7f5be3ff4740 1 -- 172.21.15.90:0/583685390 wait complete. 2024-04-25T11:44:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.039+0000 7f5be3ff4740 1 -- start start 2024-04-25T11:44:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.039+0000 7f5be3ff4740 1 Processor -- start 2024-04-25T11:44:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.039+0000 7f5be3ff4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc906f02f0 0x55cc90ac31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.039+0000 7f5be3ff4740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc906f84d0 0x55cc90ac3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.039+0000 7f5be3ff4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc907191b0 0x55cc90ac9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.039+0000 7f5be3ff4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55cc90ac5c20 con 0x55cc906f02f0 2024-04-25T11:44:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.039+0000 7f5be3ff4740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55cc909a6b60 con 0x55cc906f84d0 2024-04-25T11:44:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.039+0000 7f5be3ff4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55cc90ac5da0 con 0x55cc907191b0 2024-04-25T11:44:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.039+0000 7f5be3ff4740 10 client.0 _ll_register_callbacks cb 0x55cc909cb2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.039+0000 7f5bd659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc906f02f0 0x55cc90ac31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.039+0000 7f5bd659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc906f02f0 0x55cc90ac31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59644/0 (socket says 172.21.15.90:59644) 2024-04-25T11:44:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.039+0000 7f5bd659c640 1 -- 172.21.15.90:0/621488767 learned_addr learned my addr 172.21.15.90:0/621488767 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.040+0000 7f5bd6d9d640 1 --2- 172.21.15.90:0/621488767 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc906f84d0 0x55cc90ac3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.040+0000 7f5bd659c640 1 -- 172.21.15.90:0/621488767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc907191b0 msgr2=0x55cc90ac9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.040+0000 7f5bd659c640 1 --2- 172.21.15.90:0/621488767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc907191b0 0x55cc90ac9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.040+0000 7f5bd659c640 1 -- 172.21.15.90:0/621488767 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc906f84d0 msgr2=0x55cc90ac3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.040+0000 7f5bd659c640 1 --2- 172.21.15.90:0/621488767 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc906f84d0 0x55cc90ac3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.040+0000 7f5bd659c640 1 -- 172.21.15.90:0/621488767 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc90731210 con 0x55cc906f02f0 2024-04-25T11:44:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.040+0000 7f5bd659c640 1 --2- 172.21.15.90:0/621488767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc906f02f0 0x55cc90ac31e0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc0057a0 tx=0x7f5bcc00be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a6bc85d8ab082c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.040+0000 7f5bbaff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.040+0000 7f5bbaff5640 1 -- 172.21.15.90:0/621488767 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc019050 con 0x55cc906f02f0 2024-04-25T11:44:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.040+0000 7f5bbaff5640 1 -- 172.21.15.90:0/621488767 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f5bcc0042e0 con 0x55cc906f02f0 2024-04-25T11:44:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.040+0000 7f5be3ff4740 1 -- 172.21.15.90:0/621488767 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bcc0035f0 con 0x55cc906f02f0 2024-04-25T11:44:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.040+0000 7f5bbaff5640 1 -- 172.21.15.90:0/621488767 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5bcc005020 con 0x55cc906f02f0 2024-04-25T11:44:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.040+0000 7f5be3ff4740 10 client.18588 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.040+0000 7f5be3ff4740 1 -- 172.21.15.90:0/621488767 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cc907fd620 con 0x55cc906f02f0 2024-04-25T11:44:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.040+0000 7f5bbaff5640 1 -- 172.21.15.90:0/621488767 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bcc0035f0 con 0x55cc906f02f0 2024-04-25T11:44:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.041+0000 7f5bbaff5640 1 -- 172.21.15.90:0/621488767 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f5bcc0114c0 con 0x55cc906f02f0 2024-04-25T11:44:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.041+0000 7f5be3ff4740 10 client.18588 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.041+0000 7f5be3ff4740 10 client.18588 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:44:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.041+0000 7f5be3ff4740 20 client.18588 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.041+0000 7f5be3ff4740 1 --2- 172.21.15.90:0/621488767 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55cc906d7d80 0x55cc90ad5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.041+0000 7f5be3ff4740 4 client.18588 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:44:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.041+0000 7f5be3ff4740 1 -- 172.21.15.90:0/621488767 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc90a2d070 con 0x55cc906d7d80 2024-04-25T11:44:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.041+0000 7f5bd6d9d640 1 --2- 172.21.15.90:0/621488767 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55cc906d7d80 0x55cc90ad5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.042+0000 7f5bbaff5640 10 client.18588 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.042+0000 7f5bd6d9d640 1 --2- 172.21.15.90:0/621488767 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55cc906d7d80 0x55cc90ad5660 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9be2bfea181a16 server_cookie=d5e050a29c1de258 in_seq=0 out_seq=0 2024-04-25T11:44:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.042+0000 7f5bbaff5640 1 -- 172.21.15.90:0/621488767 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55cc90a2d070 con 0x55cc906d7d80 2024-04-25T11:44:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.042+0000 7f5bbaff5640 10 client.18588 handle_command_reply: tid=0 2024-04-25T11:44:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.042+0000 7f5be3ff4740 1 client.18588 shutdown 2024-04-25T11:44:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.043+0000 7f5be3ff4740 1 -- 172.21.15.90:0/621488767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc906f02f0 msgr2=0x55cc90ac31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.043+0000 7f5be3ff4740 1 --2- 172.21.15.90:0/621488767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc906f02f0 0x55cc90ac31e0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc0057a0 tx=0x7f5bcc00be10 comp rx=0 tx=0).stop 2024-04-25T11:44:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.043+0000 7f5be3ff4740 1 -- 172.21.15.90:0/621488767 shutdown_connections 2024-04-25T11:44:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.043+0000 7f5be3ff4740 1 --2- 172.21.15.90:0/621488767 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55cc906d7d80 0x55cc90ad5660 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.043+0000 7f5be3ff4740 1 --2- 172.21.15.90:0/621488767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc907191b0 0x55cc90ac9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.043+0000 7f5be3ff4740 1 --2- 172.21.15.90:0/621488767 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc906f84d0 0x55cc90ac3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.043+0000 7f5be3ff4740 1 --2- 172.21.15.90:0/621488767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc906f02f0 0x55cc90ac31e0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.043+0000 7f5bbaff5640 0 client.18588 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.043+0000 7f5be3ff4740 1 -- 172.21.15.90:0/621488767 >> 172.21.15.90:0/621488767 conn(0x55cc9070d300 msgr2=0x55cc90ab88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.043+0000 7f5be3ff4740 1 -- 172.21.15.90:0/621488767 shutdown_connections 2024-04-25T11:44:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.043+0000 7f5be3ff4740 1 -- 172.21.15.90:0/621488767 wait complete. 2024-04-25T11:44:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.043+0000 7f5be3ff4740 20 client.18588 trim_cache size 0 max 16384 2024-04-25T11:44:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.058+0000 7f5be3ff4740 1 Processor -- start 2024-04-25T11:44:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.058+0000 7f5be3ff4740 1 -- start start 2024-04-25T11:44:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.059+0000 7f5be3ff4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc906f02f0 0x55cc90abce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.059+0000 7f5be3ff4740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc906f84d0 0x55cc90ab8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.059+0000 7f5be3ff4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc907191b0 0x55cc90ab8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.059+0000 7f5be3ff4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd81afbe0 con 0x55cc907191b0 2024-04-25T11:44:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.059+0000 7f5be3ff4740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd81b1720 con 0x55cc906f84d0 2024-04-25T11:44:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.059+0000 7f5be3ff4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd81b02b0 con 0x55cc906f02f0 2024-04-25T11:44:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.059+0000 7f5bd659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc907191b0 0x55cc90ab8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.059+0000 7f5bd659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc907191b0 0x55cc90ab8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59660/0 (socket says 172.21.15.90:59660) 2024-04-25T11:44:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.059+0000 7f5bd659c640 1 -- 172.21.15.90:0/884228208 learned_addr learned my addr 172.21.15.90:0/884228208 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.059+0000 7f5bd659c640 1 --2- 172.21.15.90:0/884228208 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc906f02f0 0x55cc90abce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.059+0000 7f5bd6d9d640 1 --2- 172.21.15.90:0/884228208 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc906f84d0 0x55cc90ab8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.059+0000 7f5bd659c640 1 -- 172.21.15.90:0/884228208 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc906f02f0 msgr2=0x55cc90abce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.059+0000 7f5bd659c640 1 --2- 172.21.15.90:0/884228208 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc906f02f0 0x55cc90abce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.059+0000 7f5bd659c640 1 -- 172.21.15.90:0/884228208 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc906f84d0 msgr2=0x55cc90ab8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.059+0000 7f5bd659c640 1 --2- 172.21.15.90:0/884228208 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc906f84d0 0x55cc90ab8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.059+0000 7f5bd659c640 1 -- 172.21.15.90:0/884228208 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc90ac3ec0 con 0x55cc907191b0 2024-04-25T11:44:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.059+0000 7f5bd659c640 1 --2- 172.21.15.90:0/884228208 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc907191b0 0x55cc90ab8dc0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc003e80 tx=0x7f5bcc00b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac61b30652dcd164 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.059+0000 7f5bbcff9640 1 -- 172.21.15.90:0/884228208 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc019070 con 0x55cc907191b0 2024-04-25T11:44:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.060+0000 7f5bbcff9640 1 -- 172.21.15.90:0/884228208 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bcc022070 con 0x55cc907191b0 2024-04-25T11:44:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.060+0000 7f5bbcff9640 1 -- 172.21.15.90:0/884228208 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc0116a0 con 0x55cc907191b0 2024-04-25T11:44:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.060+0000 7f5be3ff4740 1 -- 172.21.15.90:0/884228208 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc907191b0 msgr2=0x55cc90ab8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.060+0000 7f5be3ff4740 1 --2- 172.21.15.90:0/884228208 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc907191b0 0x55cc90ab8dc0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc003e80 tx=0x7f5bcc00b7e0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.060+0000 7f5be3ff4740 1 -- 172.21.15.90:0/884228208 shutdown_connections 2024-04-25T11:44:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.060+0000 7f5be3ff4740 1 --2- 172.21.15.90:0/884228208 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc907191b0 0x55cc90ab8dc0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.060+0000 7f5be3ff4740 1 --2- 172.21.15.90:0/884228208 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc906f84d0 0x55cc90ab8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.060+0000 7f5be3ff4740 1 --2- 172.21.15.90:0/884228208 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc906f02f0 0x55cc90abce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.060+0000 7f5be3ff4740 1 -- 172.21.15.90:0/884228208 >> 172.21.15.90:0/884228208 conn(0x55cc9070d300 msgr2=0x55cc90abc740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.060+0000 7f5be3ff4740 1 -- 172.21.15.90:0/884228208 shutdown_connections 2024-04-25T11:44:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.060+0000 7f5be3ff4740 1 -- 172.21.15.90:0/884228208 wait complete. 2024-04-25T11:44:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.061+0000 7f5be3ff4740 1 -- start start 2024-04-25T11:44:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.061+0000 7f5be3ff4740 1 Processor -- start 2024-04-25T11:44:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.061+0000 7f5be3ff4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc906f02f0 0x55cc90acec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.061+0000 7f5be3ff4740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc906f84d0 0x55cc90acf190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.061+0000 7f5be3ff4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc907191b0 0x55cc90b347c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.061+0000 7f5be3ff4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd81b1050 con 0x55cc907191b0 2024-04-25T11:44:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.061+0000 7f5be3ff4740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd81b0980 con 0x55cc906f84d0 2024-04-25T11:44:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.061+0000 7f5be3ff4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd81adbe0 con 0x55cc906f02f0 2024-04-25T11:44:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.061+0000 7f5be3ff4740 10 client.0 _ll_register_callbacks cb 0x7f5bcc013400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.061+0000 7f5bd659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc907191b0 0x55cc90b347c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.061+0000 7f5bd659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc907191b0 0x55cc90b347c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59676/0 (socket says 172.21.15.90:59676) 2024-04-25T11:44:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.061+0000 7f5bd659c640 1 -- 172.21.15.90:0/1049187957 learned_addr learned my addr 172.21.15.90:0/1049187957 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.061+0000 7f5bd659c640 1 --2- 172.21.15.90:0/1049187957 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc906f02f0 0x55cc90acec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.061+0000 7f5bd6d9d640 1 --2- 172.21.15.90:0/1049187957 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc906f84d0 0x55cc90acf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.061+0000 7f5bd659c640 1 -- 172.21.15.90:0/1049187957 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc906f02f0 msgr2=0x55cc90acec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.061+0000 7f5bd659c640 1 --2- 172.21.15.90:0/1049187957 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc906f02f0 0x55cc90acec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.061+0000 7f5bd659c640 1 -- 172.21.15.90:0/1049187957 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc906f84d0 msgr2=0x55cc90acf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.061+0000 7f5bd659c640 1 --2- 172.21.15.90:0/1049187957 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc906f84d0 0x55cc90acf190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.061+0000 7f5bd659c640 1 -- 172.21.15.90:0/1049187957 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc90b34d80 con 0x55cc907191b0 2024-04-25T11:44:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.062+0000 7f5bd659c640 1 --2- 172.21.15.90:0/1049187957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc907191b0 0x55cc90b347c0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc0059f0 tx=0x7f5bcc00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cc86b95e0fd3a89 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.062+0000 7f5bbaff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.062+0000 7f5bbaff5640 1 -- 172.21.15.90:0/1049187957 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc044060 con 0x55cc907191b0 2024-04-25T11:44:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.062+0000 7f5bbaff5640 1 -- 172.21.15.90:0/1049187957 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f5bcc011d00 con 0x55cc907191b0 2024-04-25T11:44:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.062+0000 7f5be3ff4740 1 -- 172.21.15.90:0/1049187957 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bcc022070 con 0x55cc907191b0 2024-04-25T11:44:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.062+0000 7f5bbaff5640 1 -- 172.21.15.90:0/1049187957 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5bcc037040 con 0x55cc907191b0 2024-04-25T11:44:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.062+0000 7f5be3ff4740 10 client.18594 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.062+0000 7f5be3ff4740 1 -- 172.21.15.90:0/1049187957 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cc90b3d230 con 0x55cc907191b0 2024-04-25T11:44:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.062+0000 7f5bbaff5640 1 -- 172.21.15.90:0/1049187957 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bcc022070 con 0x55cc907191b0 2024-04-25T11:44:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.062+0000 7f5bbaff5640 1 -- 172.21.15.90:0/1049187957 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f5bcc0049d0 con 0x55cc907191b0 2024-04-25T11:44:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.062+0000 7f5be3ff4740 10 client.18594 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.062+0000 7f5be3ff4740 10 client.18594 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:44:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.062+0000 7f5be3ff4740 20 client.18594 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.062+0000 7f5be3ff4740 1 --2- 172.21.15.90:0/1049187957 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55cc906ad1c0 0x55cc90b3f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.062+0000 7f5be3ff4740 4 client.18594 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:44:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.062+0000 7f5be3ff4740 1 -- 172.21.15.90:0/1049187957 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cc90a26ee0 con 0x55cc906ad1c0 2024-04-25T11:44:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.063+0000 7f5bd6d9d640 1 --2- 172.21.15.90:0/1049187957 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55cc906ad1c0 0x55cc90b3f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.063+0000 7f5bbaff5640 10 client.18594 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.063+0000 7f5bd6d9d640 1 --2- 172.21.15.90:0/1049187957 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55cc906ad1c0 0x55cc90b3f960 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d9f0d2033b22f67 server_cookie=daac8bf701834b31 in_seq=0 out_seq=0 2024-04-25T11:44:38.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.069+0000 7f5bbaff5640 1 -- 172.21.15.90:0/1049187957 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cc90a26ee0 con 0x55cc906ad1c0 2024-04-25T11:44:38.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.069+0000 7f5bbaff5640 10 client.18594 handle_command_reply: tid=0 2024-04-25T11:44:38.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.069+0000 7f5be3ff4740 1 client.18594 shutdown 2024-04-25T11:44:38.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.070+0000 7f5be3ff4740 1 -- 172.21.15.90:0/1049187957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc907191b0 msgr2=0x55cc90b347c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:38.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.070+0000 7f5be3ff4740 1 --2- 172.21.15.90:0/1049187957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc907191b0 0x55cc90b347c0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc0059f0 tx=0x7f5bcc00b8b0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.070+0000 7f5be3ff4740 1 -- 172.21.15.90:0/1049187957 shutdown_connections 2024-04-25T11:44:38.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.070+0000 7f5be3ff4740 1 --2- 172.21.15.90:0/1049187957 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55cc906ad1c0 0x55cc90b3f960 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.070+0000 7f5be3ff4740 1 --2- 172.21.15.90:0/1049187957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cc907191b0 0x55cc90b347c0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.070+0000 7f5bbaff5640 0 client.18594 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:38.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.070+0000 7f5be3ff4740 1 --2- 172.21.15.90:0/1049187957 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cc906f84d0 0x55cc90acf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.070+0000 7f5be3ff4740 1 --2- 172.21.15.90:0/1049187957 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cc906f02f0 0x55cc90acec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.070+0000 7f5be3ff4740 1 -- 172.21.15.90:0/1049187957 >> 172.21.15.90:0/1049187957 conn(0x55cc9070d300 msgr2=0x55cc90ab8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:38.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.070+0000 7f5be3ff4740 1 -- 172.21.15.90:0/1049187957 shutdown_connections 2024-04-25T11:44:38.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.070+0000 7f5be3ff4740 1 -- 172.21.15.90:0/1049187957 wait complete. 2024-04-25T11:44:38.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.070+0000 7f5be3ff4740 20 client.18594 trim_cache size 0 max 16384 2024-04-25T11:44:38.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:44:38.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:44:38.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:44:38.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:44:38.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.070+0000 7f5bdd13f640 1 -- 172.21.15.90:0/3065401953 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5bc00afc50 msgr2=0x7f5bc00b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.070+0000 7f5bdd13f640 1 --2- 172.21.15.90:0/3065401953 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5bc00afc50 0x7f5bc00b2160 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5bd00dd230 tx=0x7f5bd0076040 comp rx=0 tx=0).stop 2024-04-25T11:44:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.070+0000 7f5bdd13f640 1 -- 172.21.15.90:0/3065401953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bd8108f30 msgr2=0x7f5bd819c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.070+0000 7f5bdd13f640 1 --2- 172.21.15.90:0/3065401953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bd8108f30 0x7f5bd819c0b0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f5bd000cce0 tx=0x7f5bd000a620 comp rx=0 tx=0).stop 2024-04-25T11:44:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.070+0000 7f5bdd13f640 1 -- 172.21.15.90:0/3065401953 shutdown_connections 2024-04-25T11:44:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.070+0000 7f5bdd13f640 1 --2- 172.21.15.90:0/3065401953 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5bc00afc50 0x7f5bc00b2160 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.070+0000 7f5bdd13f640 1 --2- 172.21.15.90:0/3065401953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bd8108f30 0x7f5bd819c0b0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.070+0000 7f5bdd13f640 1 --2- 172.21.15.90:0/3065401953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bd8108a90 0x7f5bd819bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.070+0000 7f5bdd13f640 1 --2- 172.21.15.90:0/3065401953 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5bd81031e0 0x7f5bd819b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.070+0000 7f5bdd13f640 1 -- 172.21.15.90:0/3065401953 >> 172.21.15.90:0/3065401953 conn(0x7f5bd80fe260 msgr2=0x7f5bd80fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.071+0000 7f5bdd13f640 1 -- 172.21.15.90:0/3065401953 shutdown_connections 2024-04-25T11:44:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:38.071+0000 7f5bdd13f640 1 -- 172.21.15.90:0/3065401953 wait complete. 2024-04-25T11:44:38.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:37 smithi090 ceph-mon[29575]: pgmap v295: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 8.0 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:44:38.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:37 smithi138 ceph-mon[32328]: pgmap v295: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 8.0 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:44:38.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:37 smithi156 ceph-mon[33045]: pgmap v295: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 8.0 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:44:40.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:39 smithi090 ceph-mon[29575]: pgmap v296: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 8.0 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:44:40.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:39 smithi138 ceph-mon[32328]: pgmap v296: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 8.0 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:44:40.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:39 smithi156 ceph-mon[33045]: pgmap v296: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 8.0 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:44:42.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:41 smithi090 ceph-mon[29575]: pgmap v297: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 12 KiB/s rd, 5.5 KiB/s wr, 13 op/s 2024-04-25T11:44:42.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:41 smithi138 ceph-mon[32328]: pgmap v297: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 12 KiB/s rd, 5.5 KiB/s wr, 13 op/s 2024-04-25T11:44:42.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:41 smithi156 ceph-mon[33045]: pgmap v297: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 12 KiB/s rd, 5.5 KiB/s wr, 13 op/s 2024-04-25T11:44:44.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:43 smithi090 ceph-mon[29575]: pgmap v298: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:44:44.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:43 smithi138 ceph-mon[32328]: pgmap v298: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:44:44.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:43 smithi156 ceph-mon[33045]: pgmap v298: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:44:46.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:45 smithi090 ceph-mon[29575]: pgmap v299: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-04-25T11:44:46.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:45 smithi138 ceph-mon[32328]: pgmap v299: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-04-25T11:44:46.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:45 smithi156 ceph-mon[33045]: pgmap v299: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-04-25T11:44:48.171 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:44:48.172 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:44:48.198 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:47 smithi090 ceph-mon[29575]: pgmap v300: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-04-25T11:44:48.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.309+0000 7f7286649640 1 -- 172.21.15.90:0/1189206069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7280108a20 msgr2=0x7f72801025d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.309+0000 7f7286649640 1 --2- 172.21.15.90:0/1189206069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7280108a20 0x7f72801025d0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f7270009900 tx=0x7f727002a410 comp rx=0 tx=0).stop 2024-04-25T11:44:48.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.309+0000 7f7286649640 1 -- 172.21.15.90:0/1189206069 shutdown_connections 2024-04-25T11:44:48.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.309+0000 7f7286649640 1 --2- 172.21.15.90:0/1189206069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7280103180 0x7f728010f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.309+0000 7f7286649640 1 --2- 172.21.15.90:0/1189206069 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7280108ec0 0x7f7280102b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.309+0000 7f7286649640 1 --2- 172.21.15.90:0/1189206069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7280108a20 0x7f72801025d0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.309+0000 7f7286649640 1 -- 172.21.15.90:0/1189206069 >> 172.21.15.90:0/1189206069 conn(0x7f7280078110 msgr2=0x7f7280078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:48.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.309+0000 7f7286649640 1 -- 172.21.15.90:0/1189206069 shutdown_connections 2024-04-25T11:44:48.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.309+0000 7f7286649640 1 -- 172.21.15.90:0/1189206069 wait complete. 2024-04-25T11:44:48.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.310+0000 7f7286649640 1 Processor -- start 2024-04-25T11:44:48.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.310+0000 7f7286649640 1 -- start start 2024-04-25T11:44:48.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.310+0000 7f7286649640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7280103180 0x7f72801a25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.310+0000 7f7286649640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7280108a20 0x7f72801a2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.310+0000 7f7286649640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7280108ec0 0x7f728019c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.310+0000 7f7286649640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7280115f10 con 0x7f7280108a20 2024-04-25T11:44:48.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.310+0000 7f7286649640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7280115d90 con 0x7f7280108ec0 2024-04-25T11:44:48.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.310+0000 7f7286649640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7280116090 con 0x7f7280103180 2024-04-25T11:44:48.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.310+0000 7f727f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7280108a20 0x7f72801a2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.310+0000 7f727f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7280108a20 0x7f72801a2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48378/0 (socket says 172.21.15.90:48378) 2024-04-25T11:44:48.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.310+0000 7f727f7fe640 1 -- 172.21.15.90:0/1361067964 learned_addr learned my addr 172.21.15.90:0/1361067964 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:48.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.311+0000 7f727ffff640 1 --2- 172.21.15.90:0/1361067964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7280103180 0x7f72801a25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.311+0000 7f7284be6640 1 --2- 172.21.15.90:0/1361067964 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7280108ec0 0x7f728019c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.311+0000 7f727f7fe640 1 -- 172.21.15.90:0/1361067964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7280103180 msgr2=0x7f72801a25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.311+0000 7f727f7fe640 1 --2- 172.21.15.90:0/1361067964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7280103180 0x7f72801a25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.311+0000 7f727f7fe640 1 -- 172.21.15.90:0/1361067964 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7280108ec0 msgr2=0x7f728019c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.311+0000 7f727f7fe640 1 --2- 172.21.15.90:0/1361067964 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7280108ec0 0x7f728019c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.311+0000 7f727f7fe640 1 -- 172.21.15.90:0/1361067964 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f728019ce40 con 0x7f7280108a20 2024-04-25T11:44:48.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.311+0000 7f727f7fe640 1 --2- 172.21.15.90:0/1361067964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7280108a20 0x7f72801a2ae0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f727400c970 tx=0x7f727400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:48.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.311+0000 7f727d7fa640 1 -- 172.21.15.90:0/1361067964 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7274007c70 con 0x7f7280108a20 2024-04-25T11:44:48.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.311+0000 7f7286649640 1 -- 172.21.15.90:0/1361067964 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f728019d130 con 0x7f7280108a20 2024-04-25T11:44:48.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.311+0000 7f7286649640 1 -- 172.21.15.90:0/1361067964 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72801a93f0 con 0x7f7280108a20 2024-04-25T11:44:48.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.311+0000 7f727d7fa640 1 -- 172.21.15.90:0/1361067964 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7274007e10 con 0x7f7280108a20 2024-04-25T11:44:48.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.311+0000 7f727d7fa640 1 -- 172.21.15.90:0/1361067964 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7274018760 con 0x7f7280108a20 2024-04-25T11:44:48.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:47 smithi138 ceph-mon[32328]: pgmap v300: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-04-25T11:44:48.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.314+0000 7f728d5d4740 1 Processor -- start 2024-04-25T11:44:48.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.314+0000 7f728d5d4740 1 -- start start 2024-04-25T11:44:48.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.314+0000 7f727d7fa640 1 -- 172.21.15.90:0/1361067964 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f727400fab0 con 0x7f7280108a20 2024-04-25T11:44:48.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.314+0000 7f728d5d4740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5601432b7390 0x560143657a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.314+0000 7f728d5d4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56014328e4d0 0x560143657fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.314+0000 7f728d5d4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601432966b0 0x5601436648f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.314+0000 7f728d5d4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5601435bdd80 con 0x5601432966b0 2024-04-25T11:44:48.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.314+0000 7f728d5d4740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5601435778b0 con 0x5601432b7390 2024-04-25T11:44:48.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.314+0000 7f728d5d4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5601435b9d40 con 0x56014328e4d0 2024-04-25T11:44:48.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.314+0000 7f727f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601432966b0 0x5601436648f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.314+0000 7f727f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601432966b0 0x5601436648f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48382/0 (socket says 172.21.15.90:48382) 2024-04-25T11:44:48.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.314+0000 7f727f7fe640 1 -- 172.21.15.90:0/3927344439 learned_addr learned my addr 172.21.15.90:0/3927344439 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:48.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.314+0000 7f7284be6640 1 --2- 172.21.15.90:0/3927344439 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5601432b7390 0x560143657a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.314+0000 7f727ffff640 1 --2- 172.21.15.90:0/3927344439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56014328e4d0 0x560143657fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.315+0000 7f727f7fe640 1 -- 172.21.15.90:0/3927344439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56014328e4d0 msgr2=0x560143657fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:48.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.315+0000 7f727f7fe640 1 --2- 172.21.15.90:0/3927344439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56014328e4d0 0x560143657fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.315+0000 7f727f7fe640 1 -- 172.21.15.90:0/3927344439 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5601432b7390 msgr2=0x560143657a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:48.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.315+0000 7f727f7fe640 1 --2- 172.21.15.90:0/3927344439 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5601432b7390 0x560143657a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.315+0000 7f727f7fe640 1 -- 172.21.15.90:0/3927344439 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560143548a40 con 0x5601432966b0 2024-04-25T11:44:48.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.315+0000 7f727ffff640 1 --2- 172.21.15.90:0/3927344439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56014328e4d0 0x560143657fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:48.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.315+0000 7f727d7fa640 1 --2- 172.21.15.90:0/1361067964 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f72680afc40 0x7f72680b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.315+0000 7f727f7fe640 1 --2- 172.21.15.90:0/3927344439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601432966b0 0x5601436648f0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f72740dc880 tx=0x7f72740e7580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64ca48bf25b473d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:48.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.315+0000 7f7284be6640 1 --2- 172.21.15.90:0/1361067964 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f72680afc40 0x7f72680b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.315+0000 7f727d7fa640 1 -- 172.21.15.90:0/1361067964 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f72740df250 con 0x7f7280108a20 2024-04-25T11:44:48.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.315+0000 7f72667fc640 1 -- 172.21.15.90:0/3927344439 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72740a3220 con 0x5601432966b0 2024-04-25T11:44:48.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.315+0000 7f72667fc640 1 -- 172.21.15.90:0/3927344439 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72740a6e70 con 0x5601432966b0 2024-04-25T11:44:48.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.315+0000 7f72667fc640 1 -- 172.21.15.90:0/3927344439 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72740a6ad0 con 0x5601432966b0 2024-04-25T11:44:48.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.316+0000 7f728d5d4740 1 -- 172.21.15.90:0/3927344439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601432966b0 msgr2=0x5601436648f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.316+0000 7f7284be6640 1 --2- 172.21.15.90:0/1361067964 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f72680afc40 0x7f72680b2150 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x560143654bd0 tx=0x7f72780085f0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:48.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.316+0000 7f728d5d4740 1 --2- 172.21.15.90:0/3927344439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601432966b0 0x5601436648f0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f72740dc880 tx=0x7f72740e7580 comp rx=0 tx=0).stop 2024-04-25T11:44:48.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.316+0000 7f728d5d4740 1 -- 172.21.15.90:0/3927344439 shutdown_connections 2024-04-25T11:44:48.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.316+0000 7f728d5d4740 1 --2- 172.21.15.90:0/3927344439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601432966b0 0x5601436648f0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.316+0000 7f728d5d4740 1 --2- 172.21.15.90:0/3927344439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56014328e4d0 0x560143657fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.316+0000 7f728d5d4740 1 --2- 172.21.15.90:0/3927344439 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5601432b7390 0x560143657a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.316+0000 7f728d5d4740 1 -- 172.21.15.90:0/3927344439 >> 172.21.15.90:0/3927344439 conn(0x5601432ab4e0 msgr2=0x560143654210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:48.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.316+0000 7f728d5d4740 1 -- 172.21.15.90:0/3927344439 shutdown_connections 2024-04-25T11:44:48.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.316+0000 7f728d5d4740 1 -- 172.21.15.90:0/3927344439 wait complete. 2024-04-25T11:44:48.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.316+0000 7f728d5d4740 1 -- start start 2024-04-25T11:44:48.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.317+0000 7f728d5d4740 1 Processor -- start 2024-04-25T11:44:48.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.317+0000 7f728d5d4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56014328e4d0 0x560143662570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.317+0000 7f728d5d4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601432966b0 0x560143662ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.317+0000 7f728d5d4740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5601432b7390 0x56014366e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.317+0000 7f728d5d4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560143664fe0 con 0x5601432966b0 2024-04-25T11:44:48.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.317+0000 7f728d5d4740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x560143544d10 con 0x5601432b7390 2024-04-25T11:44:48.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.317+0000 7f728d5d4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560143665160 con 0x56014328e4d0 2024-04-25T11:44:48.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.317+0000 7f728d5d4740 10 client.0 _ll_register_callbacks cb 0x560143571210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:48.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.317+0000 7f727f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601432966b0 0x560143662ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.317+0000 7f727f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601432966b0 0x560143662ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48390/0 (socket says 172.21.15.90:48390) 2024-04-25T11:44:48.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.317+0000 7f727f7fe640 1 -- 172.21.15.90:0/2290379497 learned_addr learned my addr 172.21.15.90:0/2290379497 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:48.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.317+0000 7f7284be6640 1 --2- 172.21.15.90:0/2290379497 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5601432b7390 0x56014366e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.317+0000 7f727ffff640 1 --2- 172.21.15.90:0/2290379497 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56014328e4d0 0x560143662570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.317+0000 7f727f7fe640 1 -- 172.21.15.90:0/2290379497 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56014328e4d0 msgr2=0x560143662570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:48.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.317+0000 7f727f7fe640 1 --2- 172.21.15.90:0/2290379497 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56014328e4d0 0x560143662570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.317+0000 7f727f7fe640 1 -- 172.21.15.90:0/2290379497 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5601432b7390 msgr2=0x56014366e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:48.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.317+0000 7f727f7fe640 1 --2- 172.21.15.90:0/2290379497 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5601432b7390 0x56014366e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.317+0000 7f727f7fe640 1 -- 172.21.15.90:0/2290379497 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601433997d0 con 0x5601432966b0 2024-04-25T11:44:48.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.318+0000 7f727f7fe640 1 --2- 172.21.15.90:0/2290379497 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601432966b0 0x560143662ab0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f72740dc9c0 tx=0x7f72740df550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=749843d49ebb18c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:48.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.318+0000 7f725f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:48.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.318+0000 7f725f7fe640 1 -- 172.21.15.90:0/2290379497 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72740a6cb0 con 0x5601432966b0 2024-04-25T11:44:48.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.318+0000 7f725f7fe640 1 -- 172.21.15.90:0/2290379497 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f72740a35f0 con 0x5601432966b0 2024-04-25T11:44:48.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.318+0000 7f728d5d4740 1 -- 172.21.15.90:0/2290379497 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72740a6e70 con 0x5601432966b0 2024-04-25T11:44:48.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.318+0000 7f725f7fe640 1 -- 172.21.15.90:0/2290379497 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f72740a7130 con 0x5601432966b0 2024-04-25T11:44:48.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.318+0000 7f728d5d4740 10 client.18606 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:48.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.318+0000 7f728d5d4740 1 -- 172.21.15.90:0/2290379497 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5601432cf3f0 con 0x5601432966b0 2024-04-25T11:44:48.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.318+0000 7f725f7fe640 1 -- 172.21.15.90:0/2290379497 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72740a6e70 con 0x5601432966b0 2024-04-25T11:44:48.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.318+0000 7f725f7fe640 1 -- 172.21.15.90:0/2290379497 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f72740a75c0 con 0x5601432966b0 2024-04-25T11:44:48.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.318+0000 7f728d5d4740 10 client.18606 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:48.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.318+0000 7f728d5d4740 10 client.18606 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:44:48.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.318+0000 7f728d5d4740 20 client.18606 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:48.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.318+0000 7f728d5d4740 1 --2- 172.21.15.90:0/2290379497 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x560143275f60 0x560143674460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.318+0000 7f728d5d4740 4 client.18606 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:44:48.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.318+0000 7f728d5d4740 1 -- 172.21.15.90:0/2290379497 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5601435cb990 con 0x560143275f60 2024-04-25T11:44:48.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.319+0000 7f727ffff640 1 --2- 172.21.15.90:0/2290379497 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x560143275f60 0x560143674460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.319+0000 7f725f7fe640 10 client.18606 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:48.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.319+0000 7f727ffff640 1 --2- 172.21.15.90:0/2290379497 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x560143275f60 0x560143674460 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b928fe2dffedce6a server_cookie=40af2b92d5efd8cb in_seq=0 out_seq=0 2024-04-25T11:44:48.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.320+0000 7f725f7fe640 1 -- 172.21.15.90:0/2290379497 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5601435cb990 con 0x560143275f60 2024-04-25T11:44:48.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.320+0000 7f725f7fe640 10 client.18606 handle_command_reply: tid=0 2024-04-25T11:44:48.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.320+0000 7f728d5d4740 1 client.18606 shutdown 2024-04-25T11:44:48.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.320+0000 7f728d5d4740 1 -- 172.21.15.90:0/2290379497 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601432966b0 msgr2=0x560143662ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.320+0000 7f728d5d4740 1 --2- 172.21.15.90:0/2290379497 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601432966b0 0x560143662ab0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f72740dc9c0 tx=0x7f72740df550 comp rx=0 tx=0).stop 2024-04-25T11:44:48.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.320+0000 7f728d5d4740 1 -- 172.21.15.90:0/2290379497 shutdown_connections 2024-04-25T11:44:48.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.320+0000 7f728d5d4740 1 --2- 172.21.15.90:0/2290379497 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x560143275f60 0x560143674460 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.320+0000 7f728d5d4740 1 --2- 172.21.15.90:0/2290379497 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5601432b7390 0x56014366e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.320+0000 7f725f7fe640 0 client.18606 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:48.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.320+0000 7f728d5d4740 1 --2- 172.21.15.90:0/2290379497 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601432966b0 0x560143662ab0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.320+0000 7f728d5d4740 1 --2- 172.21.15.90:0/2290379497 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56014328e4d0 0x560143662570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.320+0000 7f728d5d4740 1 -- 172.21.15.90:0/2290379497 >> 172.21.15.90:0/2290379497 conn(0x5601432ab4e0 msgr2=0x560143659780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:48.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.320+0000 7f728d5d4740 1 -- 172.21.15.90:0/2290379497 shutdown_connections 2024-04-25T11:44:48.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.320+0000 7f728d5d4740 1 -- 172.21.15.90:0/2290379497 wait complete. 2024-04-25T11:44:48.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.320+0000 7f728d5d4740 20 client.18606 trim_cache size 0 max 16384 2024-04-25T11:44:48.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.338+0000 7f728d5d4740 1 Processor -- start 2024-04-25T11:44:48.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.338+0000 7f728d5d4740 1 -- start start 2024-04-25T11:44:48.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.338+0000 7f728d5d4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560143275f60 0x560143654240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.338+0000 7f728d5d4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56014328e4d0 0x560143664190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.338+0000 7f728d5d4740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5601432966b0 0x56014366e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.338+0000 7f728d5d4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f72801b1f40 con 0x56014328e4d0 2024-04-25T11:44:48.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.338+0000 7f728d5d4740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f72801b3a80 con 0x5601432966b0 2024-04-25T11:44:48.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.338+0000 7f728d5d4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f72801b2610 con 0x560143275f60 2024-04-25T11:44:48.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.338+0000 7f727f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56014328e4d0 0x560143664190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.338+0000 7f727f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56014328e4d0 0x560143664190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48398/0 (socket says 172.21.15.90:48398) 2024-04-25T11:44:48.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.338+0000 7f727f7fe640 1 -- 172.21.15.90:0/576325556 learned_addr learned my addr 172.21.15.90:0/576325556 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:48.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.339+0000 7f7284be6640 1 --2- 172.21.15.90:0/576325556 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5601432966b0 0x56014366e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.339+0000 7f727f7fe640 1 -- 172.21.15.90:0/576325556 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560143275f60 msgr2=0x560143654240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:48.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.339+0000 7f727f7fe640 1 --2- 172.21.15.90:0/576325556 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560143275f60 0x560143654240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.339+0000 7f727f7fe640 1 -- 172.21.15.90:0/576325556 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5601432966b0 msgr2=0x56014366e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:48.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.339+0000 7f727f7fe640 1 --2- 172.21.15.90:0/576325556 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5601432966b0 0x56014366e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.339+0000 7f727f7fe640 1 -- 172.21.15.90:0/576325556 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56014328f0f0 con 0x56014328e4d0 2024-04-25T11:44:48.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.339+0000 7f727f7fe640 1 --2- 172.21.15.90:0/576325556 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56014328e4d0 0x560143664190 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f72740a7060 tx=0x7f72740df5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c692b9b8fb76b6a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:48.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.339+0000 7f7265ffb640 1 -- 172.21.15.90:0/576325556 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72740a6cb0 con 0x56014328e4d0 2024-04-25T11:44:48.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.339+0000 7f7265ffb640 1 -- 172.21.15.90:0/576325556 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72740a6e70 con 0x56014328e4d0 2024-04-25T11:44:48.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.339+0000 7f7265ffb640 1 -- 172.21.15.90:0/576325556 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72740a6ad0 con 0x56014328e4d0 2024-04-25T11:44:48.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.340+0000 7f728d5d4740 1 -- 172.21.15.90:0/576325556 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56014328e4d0 msgr2=0x560143664190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.340+0000 7f728d5d4740 1 --2- 172.21.15.90:0/576325556 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56014328e4d0 0x560143664190 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f72740a7060 tx=0x7f72740df5c0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.340+0000 7f728d5d4740 1 -- 172.21.15.90:0/576325556 shutdown_connections 2024-04-25T11:44:48.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.340+0000 7f728d5d4740 1 --2- 172.21.15.90:0/576325556 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5601432966b0 0x56014366e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.340+0000 7f728d5d4740 1 --2- 172.21.15.90:0/576325556 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56014328e4d0 0x560143664190 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.340+0000 7f728d5d4740 1 --2- 172.21.15.90:0/576325556 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560143275f60 0x560143654240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.340+0000 7f728d5d4740 1 -- 172.21.15.90:0/576325556 >> 172.21.15.90:0/576325556 conn(0x5601432ab4e0 msgr2=0x560143663a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:48.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.340+0000 7f728d5d4740 1 -- 172.21.15.90:0/576325556 shutdown_connections 2024-04-25T11:44:48.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.340+0000 7f728d5d4740 1 -- 172.21.15.90:0/576325556 wait complete. 2024-04-25T11:44:48.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.340+0000 7f728d5d4740 1 -- start start 2024-04-25T11:44:48.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.340+0000 7f728d5d4740 1 Processor -- start 2024-04-25T11:44:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.341+0000 7f728d5d4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560143275f60 0x560143674870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.341+0000 7f728d5d4740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56014328e4d0 0x560143674db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.341+0000 7f728d5d4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601432966b0 0x5601436da580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.341+0000 7f728d5d4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f72801b33b0 con 0x5601432966b0 2024-04-25T11:44:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.341+0000 7f728d5d4740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f72801b2ce0 con 0x56014328e4d0 2024-04-25T11:44:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.341+0000 7f728d5d4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f72801aff40 con 0x560143275f60 2024-04-25T11:44:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.341+0000 7f728d5d4740 10 client.0 _ll_register_callbacks cb 0x7f7278008490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.341+0000 7f7284be6640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601432966b0 0x5601436da580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.341+0000 7f7284be6640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601432966b0 0x5601436da580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48408/0 (socket says 172.21.15.90:48408) 2024-04-25T11:44:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.341+0000 7f7284be6640 1 -- 172.21.15.90:0/2993975203 learned_addr learned my addr 172.21.15.90:0/2993975203 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.341+0000 7f727ffff640 1 --2- 172.21.15.90:0/2993975203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560143275f60 0x560143674870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.341+0000 7f727f7fe640 1 --2- 172.21.15.90:0/2993975203 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56014328e4d0 0x560143674db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.341+0000 7f7284be6640 1 -- 172.21.15.90:0/2993975203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560143275f60 msgr2=0x560143674870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.341+0000 7f7284be6640 1 --2- 172.21.15.90:0/2993975203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560143275f60 0x560143674870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.341+0000 7f7284be6640 1 -- 172.21.15.90:0/2993975203 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56014328e4d0 msgr2=0x560143674db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.341+0000 7f7284be6640 1 --2- 172.21.15.90:0/2993975203 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56014328e4d0 0x560143674db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.341+0000 7f7284be6640 1 -- 172.21.15.90:0/2993975203 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56014357e4b0 con 0x5601432966b0 2024-04-25T11:44:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.341+0000 7f7284be6640 1 --2- 172.21.15.90:0/2993975203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601432966b0 0x5601436da580 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f727801bdf0 tx=0x7f7278013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efb52f2e818836f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.341+0000 7f725f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.341+0000 7f725f7fe640 1 -- 172.21.15.90:0/2993975203 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727802a060 con 0x5601432966b0 2024-04-25T11:44:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.341+0000 7f725f7fe640 1 -- 172.21.15.90:0/2993975203 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f727801a660 con 0x5601432966b0 2024-04-25T11:44:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.341+0000 7f728d5d4740 1 -- 172.21.15.90:0/2993975203 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72740a6e70 con 0x5601432966b0 2024-04-25T11:44:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.342+0000 7f725f7fe640 1 -- 172.21.15.90:0/2993975203 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f727801a860 con 0x5601432966b0 2024-04-25T11:44:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.342+0000 7f728d5d4740 10 client.18612 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.342+0000 7f728d5d4740 1 -- 172.21.15.90:0/2993975203 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5601436dbbf0 con 0x5601432966b0 2024-04-25T11:44:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.342+0000 7f725f7fe640 1 -- 172.21.15.90:0/2993975203 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72740a6e70 con 0x5601432966b0 2024-04-25T11:44:48.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.342+0000 7f725f7fe640 1 -- 172.21.15.90:0/2993975203 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f7278018040 con 0x5601432966b0 2024-04-25T11:44:48.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.342+0000 7f728d5d4740 10 client.18612 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:48.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.342+0000 7f728d5d4740 10 client.18612 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:44:48.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.342+0000 7f728d5d4740 20 client.18612 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:48.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.342+0000 7f728d5d4740 1 --2- 172.21.15.90:0/2993975203 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5601432b7390 0x5601436de3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.342+0000 7f728d5d4740 4 client.18612 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:44:48.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.342+0000 7f728d5d4740 1 -- 172.21.15.90:0/2993975203 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5601435c5800 con 0x5601432b7390 2024-04-25T11:44:48.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.342+0000 7f727ffff640 1 --2- 172.21.15.90:0/2993975203 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5601432b7390 0x5601436de3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.343+0000 7f725f7fe640 10 client.18612 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:48.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.343+0000 7f727ffff640 1 --2- 172.21.15.90:0/2993975203 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5601432b7390 0x5601436de3e0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6340d0b2dbc2472 server_cookie=4dcb695713b54e8b in_seq=0 out_seq=0 2024-04-25T11:44:48.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.343+0000 7f725f7fe640 1 -- 172.21.15.90:0/2993975203 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5601435c5800 con 0x5601432b7390 2024-04-25T11:44:48.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.343+0000 7f725f7fe640 10 client.18612 handle_command_reply: tid=0 2024-04-25T11:44:48.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.343+0000 7f728d5d4740 1 client.18612 shutdown 2024-04-25T11:44:48.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.343+0000 7f728d5d4740 1 -- 172.21.15.90:0/2993975203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601432966b0 msgr2=0x5601436da580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.343+0000 7f728d5d4740 1 --2- 172.21.15.90:0/2993975203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601432966b0 0x5601436da580 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f727801bdf0 tx=0x7f7278013040 comp rx=0 tx=0).stop 2024-04-25T11:44:48.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.343+0000 7f728d5d4740 1 -- 172.21.15.90:0/2993975203 shutdown_connections 2024-04-25T11:44:48.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.343+0000 7f728d5d4740 1 --2- 172.21.15.90:0/2993975203 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5601432b7390 0x5601436de3e0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.343+0000 7f728d5d4740 1 --2- 172.21.15.90:0/2993975203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601432966b0 0x5601436da580 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.343+0000 7f728d5d4740 1 --2- 172.21.15.90:0/2993975203 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56014328e4d0 0x560143674db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.343+0000 7f725f7fe640 0 client.18612 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:48.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.343+0000 7f728d5d4740 1 --2- 172.21.15.90:0/2993975203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560143275f60 0x560143674870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.343+0000 7f728d5d4740 1 -- 172.21.15.90:0/2993975203 >> 172.21.15.90:0/2993975203 conn(0x5601432ab4e0 msgr2=0x5601436533f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:48.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.343+0000 7f728d5d4740 1 -- 172.21.15.90:0/2993975203 shutdown_connections 2024-04-25T11:44:48.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.343+0000 7f728d5d4740 1 -- 172.21.15.90:0/2993975203 wait complete. 2024-04-25T11:44:48.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.344+0000 7f728d5d4740 20 client.18612 trim_cache size 0 max 16384 2024-04-25T11:44:48.337 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:44:48.337 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-25T11:44:48.337 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-25T11:44:48.337 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:44:48.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.344+0000 7f7286649640 1 -- 172.21.15.90:0/1361067964 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f72680afc40 msgr2=0x7f72680b2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.344+0000 7f7286649640 1 --2- 172.21.15.90:0/1361067964 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f72680afc40 0x7f72680b2150 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x560143654bd0 tx=0x7f72780085f0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.344+0000 7f7286649640 1 -- 172.21.15.90:0/1361067964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7280108a20 msgr2=0x7f72801a2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.344+0000 7f7286649640 1 --2- 172.21.15.90:0/1361067964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7280108a20 0x7f72801a2ae0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f727400c970 tx=0x7f727400ce30 comp rx=0 tx=0).stop 2024-04-25T11:44:48.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.344+0000 7f7286649640 1 -- 172.21.15.90:0/1361067964 shutdown_connections 2024-04-25T11:44:48.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.344+0000 7f7286649640 1 --2- 172.21.15.90:0/1361067964 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f72680afc40 0x7f72680b2150 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.344+0000 7f7286649640 1 --2- 172.21.15.90:0/1361067964 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7280108ec0 0x7f728019c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.344+0000 7f7286649640 1 --2- 172.21.15.90:0/1361067964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7280108a20 0x7f72801a2ae0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.344+0000 7f7286649640 1 --2- 172.21.15.90:0/1361067964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7280103180 0x7f72801a25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.344+0000 7f7286649640 1 -- 172.21.15.90:0/1361067964 >> 172.21.15.90:0/1361067964 conn(0x7f7280078110 msgr2=0x7f7280104100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:48.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.344+0000 7f7286649640 1 -- 172.21.15.90:0/1361067964 shutdown_connections 2024-04-25T11:44:48.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.344+0000 7f7286649640 1 -- 172.21.15.90:0/1361067964 wait complete. 2024-04-25T11:44:48.339 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:47 smithi156 ceph-mon[33045]: pgmap v300: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-04-25T11:44:48.342 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T11:44:48.342 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:44:48.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.504+0000 7fe437772640 1 -- 172.21.15.90:0/3354268501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe43010e080 msgr2=0x7fe430110570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.504+0000 7fe437772640 1 --2- 172.21.15.90:0/3354268501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe43010e080 0x7fe430110570 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fe428009ca0 tx=0x7fe42802a950 comp rx=0 tx=0).stop 2024-04-25T11:44:48.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.504+0000 7fe437772640 1 -- 172.21.15.90:0/3354268501 shutdown_connections 2024-04-25T11:44:48.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.504+0000 7fe437772640 1 --2- 172.21.15.90:0/3354268501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe43010e080 0x7fe430110570 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.504+0000 7fe437772640 1 --2- 172.21.15.90:0/3354268501 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe43010b720 0x7fe43010db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.504+0000 7fe437772640 1 --2- 172.21.15.90:0/3354268501 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe430104cc0 0x7fe4301050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.504+0000 7fe437772640 1 -- 172.21.15.90:0/3354268501 >> 172.21.15.90:0/3354268501 conn(0x7fe4300fe260 msgr2=0x7fe430100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:48.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.504+0000 7fe437772640 1 -- 172.21.15.90:0/3354268501 shutdown_connections 2024-04-25T11:44:48.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.504+0000 7fe437772640 1 -- 172.21.15.90:0/3354268501 wait complete. 2024-04-25T11:44:48.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.505+0000 7fe437772640 1 Processor -- start 2024-04-25T11:44:48.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.505+0000 7fe437772640 1 -- start start 2024-04-25T11:44:48.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.505+0000 7fe437772640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe430104cc0 0x7fe43019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.505+0000 7fe437772640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe43010b720 0x7fe43019ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.505+0000 7fe437772640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe43010e080 0x7fe43019ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.505+0000 7fe437772640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe430113d10 con 0x7fe43010e080 2024-04-25T11:44:48.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.505+0000 7fe437772640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe430113b90 con 0x7fe430104cc0 2024-04-25T11:44:48.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.505+0000 7fe437772640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe430113e90 con 0x7fe43010b720 2024-04-25T11:44:48.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.505+0000 7fe435d0f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe43010e080 0x7fe43019ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.505+0000 7fe435d0f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe43010e080 0x7fe43019ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48424/0 (socket says 172.21.15.90:48424) 2024-04-25T11:44:48.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.505+0000 7fe435d0f640 1 -- 172.21.15.90:0/1917969616 learned_addr learned my addr 172.21.15.90:0/1917969616 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:48.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.506+0000 7fe43550e640 1 --2- 172.21.15.90:0/1917969616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe430104cc0 0x7fe43019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.506+0000 7fe435d0f640 1 -- 172.21.15.90:0/1917969616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe43010b720 msgr2=0x7fe43019ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.506+0000 7fe435d0f640 1 --2- 172.21.15.90:0/1917969616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe43010b720 0x7fe43019ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.506+0000 7fe435d0f640 1 -- 172.21.15.90:0/1917969616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe430104cc0 msgr2=0x7fe43019a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.506+0000 7fe435d0f640 1 --2- 172.21.15.90:0/1917969616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe430104cc0 0x7fe43019a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.506+0000 7fe435d0f640 1 -- 172.21.15.90:0/1917969616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe43019f570 con 0x7fe43010e080 2024-04-25T11:44:48.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.506+0000 7fe43550e640 1 --2- 172.21.15.90:0/1917969616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe430104cc0 0x7fe43019a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:48.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.506+0000 7fe435d0f640 1 --2- 172.21.15.90:0/1917969616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe43010e080 0x7fe43019ef30 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fe42802ae70 tx=0x7fe42802f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:48.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.506+0000 7fe4267fc640 1 -- 172.21.15.90:0/1917969616 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe428034040 con 0x7fe43010e080 2024-04-25T11:44:48.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.506+0000 7fe4267fc640 1 -- 172.21.15.90:0/1917969616 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4280308f0 con 0x7fe43010e080 2024-04-25T11:44:48.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.506+0000 7fe437772640 1 -- 172.21.15.90:0/1917969616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe43019f800 con 0x7fe43010e080 2024-04-25T11:44:48.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.506+0000 7fe4267fc640 1 -- 172.21.15.90:0/1917969616 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4280047f0 con 0x7fe43010e080 2024-04-25T11:44:48.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.506+0000 7fe437772640 1 -- 172.21.15.90:0/1917969616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe43019fcf0 con 0x7fe43010e080 2024-04-25T11:44:48.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.507+0000 7fe437772640 1 -- 172.21.15.90:0/1917969616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe43006a190 con 0x7fe43010e080 2024-04-25T11:44:48.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.509+0000 7fe4267fc640 1 -- 172.21.15.90:0/1917969616 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fe428037020 con 0x7fe43010e080 2024-04-25T11:44:48.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.509+0000 7fe4267fc640 1 --2- 172.21.15.90:0/1917969616 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe41c0afc60 0x7fe41c0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.509+0000 7fe43550e640 1 --2- 172.21.15.90:0/1917969616 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe41c0afc60 0x7fe41c0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.509+0000 7fe4267fc640 1 -- 172.21.15.90:0/1917969616 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7fe428100250 con 0x7fe43010e080 2024-04-25T11:44:48.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.510+0000 7fe43550e640 1 --2- 172.21.15.90:0/1917969616 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe41c0afc60 0x7fe41c0b2170 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe42c009b40 tx=0x7fe42c005e50 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:48.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.513+0000 7fe4267fc640 1 -- 172.21.15.90:0/1917969616 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe428080ff0 con 0x7fe43010e080 2024-04-25T11:44:48.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.727+0000 7fe437772640 1 -- 172.21.15.90:0/1917969616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe4301a7310 con 0x7fe43010e080 2024-04-25T11:44:48.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.728+0000 7fe4267fc640 1 -- 172.21.15.90:0/1917969616 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11930 (secure 0 0 0) 0x7fe4280c8b70 con 0x7fe43010e080 2024-04-25T11:44:48.715 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:44:48.715 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:39:20.469254+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24458,24431,24499,24475,24524]},"id":1}]} 2024-04-25T11:44:48.715 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 28 2024-04-25T11:44:48.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.730+0000 7fe437772640 1 -- 172.21.15.90:0/1917969616 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe41c0afc60 msgr2=0x7fe41c0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.730+0000 7fe437772640 1 --2- 172.21.15.90:0/1917969616 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe41c0afc60 0x7fe41c0b2170 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe42c009b40 tx=0x7fe42c005e50 comp rx=0 tx=0).stop 2024-04-25T11:44:48.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.730+0000 7fe437772640 1 -- 172.21.15.90:0/1917969616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe43010e080 msgr2=0x7fe43019ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.730+0000 7fe437772640 1 --2- 172.21.15.90:0/1917969616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe43010e080 0x7fe43019ef30 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fe42802ae70 tx=0x7fe42802f5d0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.730+0000 7fe437772640 1 -- 172.21.15.90:0/1917969616 shutdown_connections 2024-04-25T11:44:48.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.730+0000 7fe437772640 1 --2- 172.21.15.90:0/1917969616 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe41c0afc60 0x7fe41c0b2170 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.730+0000 7fe437772640 1 --2- 172.21.15.90:0/1917969616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe43010e080 0x7fe43019ef30 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.730+0000 7fe437772640 1 --2- 172.21.15.90:0/1917969616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe43010b720 0x7fe43019ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.730+0000 7fe437772640 1 --2- 172.21.15.90:0/1917969616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe430104cc0 0x7fe43019a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.730+0000 7fe437772640 1 -- 172.21.15.90:0/1917969616 >> 172.21.15.90:0/1917969616 conn(0x7fe4300fe260 msgr2=0x7fe43010e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:48.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.730+0000 7fe437772640 1 -- 172.21.15.90:0/1917969616 shutdown_connections 2024-04-25T11:44:48.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.730+0000 7fe437772640 1 -- 172.21.15.90:0/1917969616 wait complete. 2024-04-25T11:44:48.728 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T11:44:48.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.852+0000 7fad35d3f640 1 -- 172.21.15.90:0/3011425423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad30104c50 msgr2=0x7fad30105060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.852+0000 7fad35d3f640 1 --2- 172.21.15.90:0/3011425423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad30104c50 0x7fad30105060 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fad20009900 tx=0x7fad2002a510 comp rx=0 tx=0).stop 2024-04-25T11:44:48.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.852+0000 7fad35d3f640 1 -- 172.21.15.90:0/3011425423 shutdown_connections 2024-04-25T11:44:48.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.852+0000 7fad35d3f640 1 --2- 172.21.15.90:0/3011425423 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fad3010e060 0x7fad30110550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.852+0000 7fad35d3f640 1 --2- 172.21.15.90:0/3011425423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad3010b6b0 0x7fad3010dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.852+0000 7fad35d3f640 1 --2- 172.21.15.90:0/3011425423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad30104c50 0x7fad30105060 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.852+0000 7fad35d3f640 1 -- 172.21.15.90:0/3011425423 >> 172.21.15.90:0/3011425423 conn(0x7fad30078100 msgr2=0x7fad30078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:48.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.852+0000 7fad35d3f640 1 -- 172.21.15.90:0/3011425423 shutdown_connections 2024-04-25T11:44:48.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.852+0000 7fad35d3f640 1 -- 172.21.15.90:0/3011425423 wait complete. 2024-04-25T11:44:48.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.853+0000 7fad35d3f640 1 Processor -- start 2024-04-25T11:44:48.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.853+0000 7fad35d3f640 1 -- start start 2024-04-25T11:44:48.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.853+0000 7fad35d3f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad30104c50 0x7fad3019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.853+0000 7fad35d3f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fad3010b6b0 0x7fad3019aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.853+0000 7fad35d3f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad3010e060 0x7fad3019ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.853+0000 7fad35d3f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fad30113cc0 con 0x7fad3010e060 2024-04-25T11:44:48.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.853+0000 7fad35d3f640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fad30113b40 con 0x7fad3010b6b0 2024-04-25T11:44:48.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.853+0000 7fad35d3f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fad30113e40 con 0x7fad30104c50 2024-04-25T11:44:48.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.853+0000 7fad2ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad3010e060 0x7fad3019ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.853+0000 7fad2ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad3010e060 0x7fad3019ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48432/0 (socket says 172.21.15.90:48432) 2024-04-25T11:44:48.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.853+0000 7fad2ffff640 1 -- 172.21.15.90:0/3223412867 learned_addr learned my addr 172.21.15.90:0/3223412867 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:48.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.853+0000 7fad2f7fe640 1 --2- 172.21.15.90:0/3223412867 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad30104c50 0x7fad3019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.853+0000 7fad2effd640 1 --2- 172.21.15.90:0/3223412867 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fad3010b6b0 0x7fad3019aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.853+0000 7fad2ffff640 1 -- 172.21.15.90:0/3223412867 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad30104c50 msgr2=0x7fad3019a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.853+0000 7fad2ffff640 1 --2- 172.21.15.90:0/3223412867 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad30104c50 0x7fad3019a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.853+0000 7fad2ffff640 1 -- 172.21.15.90:0/3223412867 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fad3010b6b0 msgr2=0x7fad3019aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.853+0000 7fad2ffff640 1 --2- 172.21.15.90:0/3223412867 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fad3010b6b0 0x7fad3019aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.853+0000 7fad2ffff640 1 -- 172.21.15.90:0/3223412867 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad3019f5a0 con 0x7fad3010e060 2024-04-25T11:44:48.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.854+0000 7fad2ffff640 1 --2- 172.21.15.90:0/3223412867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad3010e060 0x7fad3019ef60 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fad2800eb10 tx=0x7fad2800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:48.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.854+0000 7fad2cff9640 1 -- 172.21.15.90:0/3223412867 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad28004b20 con 0x7fad3010e060 2024-04-25T11:44:48.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.854+0000 7fad2cff9640 1 -- 172.21.15.90:0/3223412867 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad28004cc0 con 0x7fad3010e060 2024-04-25T11:44:48.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.854+0000 7fad35d3f640 1 -- 172.21.15.90:0/3223412867 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad3019f890 con 0x7fad3010e060 2024-04-25T11:44:48.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.854+0000 7fad2cff9640 1 -- 172.21.15.90:0/3223412867 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad28010670 con 0x7fad3010e060 2024-04-25T11:44:48.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.854+0000 7fad35d3f640 1 -- 172.21.15.90:0/3223412867 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad3019fd90 con 0x7fad3010e060 2024-04-25T11:44:48.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.856+0000 7fad3cbf3740 1 Processor -- start 2024-04-25T11:44:48.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.856+0000 7fad2cff9640 1 -- 172.21.15.90:0/3223412867 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fad28005020 con 0x7fad3010e060 2024-04-25T11:44:48.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.856+0000 7fad3cbf3740 1 -- start start 2024-04-25T11:44:48.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.856+0000 7fad3cbf3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572754a5390 0x557275846810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.856+0000 7fad3cbf3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55727547c4d0 0x557275846d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.856+0000 7fad3cbf3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572754846b0 0x5572758536a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.856+0000 7fad3cbf3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5572757abd80 con 0x5572754846b0 2024-04-25T11:44:48.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.857+0000 7fad3cbf3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x557275765a20 con 0x55727547c4d0 2024-04-25T11:44:48.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.857+0000 7fad3cbf3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5572757a7d40 con 0x5572754a5390 2024-04-25T11:44:48.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.857+0000 7fad2effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572754846b0 0x5572758536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.857+0000 7fad2effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572754846b0 0x5572758536a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48444/0 (socket says 172.21.15.90:48444) 2024-04-25T11:44:48.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.857+0000 7fad2effd640 1 -- 172.21.15.90:0/1522886651 learned_addr learned my addr 172.21.15.90:0/1522886651 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:48.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.857+0000 7fad2f7fe640 1 --2- 172.21.15.90:0/1522886651 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55727547c4d0 0x557275846d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.857+0000 7fad2effd640 1 --2- 172.21.15.90:0/1522886651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572754a5390 0x557275846810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.857+0000 7fad2cff9640 1 --2- 172.21.15.90:0/3223412867 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fad180afc60 0x7fad180b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.857+0000 7fad2effd640 1 -- 172.21.15.90:0/1522886651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572754a5390 msgr2=0x557275846810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:48.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.857+0000 7fad2effd640 1 --2- 172.21.15.90:0/1522886651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572754a5390 0x557275846810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.857+0000 7fad2effd640 1 -- 172.21.15.90:0/1522886651 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55727547c4d0 msgr2=0x557275846d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:48.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.857+0000 7fad2effd640 1 --2- 172.21.15.90:0/1522886651 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55727547c4d0 0x557275846d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.857+0000 7fad2effd640 1 -- 172.21.15.90:0/1522886651 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557275770c30 con 0x5572754846b0 2024-04-25T11:44:48.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.857+0000 7fad2f7fe640 1 --2- 172.21.15.90:0/1522886651 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55727547c4d0 0x557275846d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:48.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.857+0000 7fad2ffff640 1 --2- 172.21.15.90:0/3223412867 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fad180afc60 0x7fad180b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.857+0000 7fad2effd640 1 --2- 172.21.15.90:0/1522886651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572754846b0 0x5572758536a0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fad24009da0 tx=0x7fad240096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19b5d825fc2c4902 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:48.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.857+0000 7fad2cff9640 1 -- 172.21.15.90:0/3223412867 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7fad280df760 con 0x7fad3010e060 2024-04-25T11:44:48.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.857+0000 7fad15ffb640 1 -- 172.21.15.90:0/1522886651 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad24012070 con 0x5572754846b0 2024-04-25T11:44:48.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.857+0000 7fad15ffb640 1 -- 172.21.15.90:0/1522886651 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad2401a070 con 0x5572754846b0 2024-04-25T11:44:48.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.857+0000 7fad15ffb640 1 -- 172.21.15.90:0/1522886651 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad24003d90 con 0x5572754846b0 2024-04-25T11:44:48.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.858+0000 7fad3cbf3740 1 -- 172.21.15.90:0/1522886651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572754846b0 msgr2=0x5572758536a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.858+0000 7fad2ffff640 1 --2- 172.21.15.90:0/3223412867 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fad180afc60 0x7fad180b2170 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fad280a4ae0 tx=0x7fad280750d0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:48.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.858+0000 7fad3cbf3740 1 --2- 172.21.15.90:0/1522886651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572754846b0 0x5572758536a0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fad24009da0 tx=0x7fad240096c0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.858+0000 7fad3cbf3740 1 -- 172.21.15.90:0/1522886651 shutdown_connections 2024-04-25T11:44:48.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.858+0000 7fad3cbf3740 1 --2- 172.21.15.90:0/1522886651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572754846b0 0x5572758536a0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.858+0000 7fad3cbf3740 1 --2- 172.21.15.90:0/1522886651 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55727547c4d0 0x557275846d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.858+0000 7fad3cbf3740 1 --2- 172.21.15.90:0/1522886651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572754a5390 0x557275846810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.858+0000 7fad3cbf3740 1 -- 172.21.15.90:0/1522886651 >> 172.21.15.90:0/1522886651 conn(0x5572754994e0 msgr2=0x557275842fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:48.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.858+0000 7fad3cbf3740 1 -- 172.21.15.90:0/1522886651 shutdown_connections 2024-04-25T11:44:48.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.858+0000 7fad3cbf3740 1 -- 172.21.15.90:0/1522886651 wait complete. 2024-04-25T11:44:48.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.859+0000 7fad3cbf3740 1 -- start start 2024-04-25T11:44:48.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.859+0000 7fad3cbf3740 1 Processor -- start 2024-04-25T11:44:48.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.859+0000 7fad3cbf3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55727547c4d0 0x5572758512b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.859+0000 7fad3cbf3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572754846b0 0x5572758517f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.859+0000 7fad3cbf3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572754a5390 0x55727585d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.859+0000 7fad3cbf3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557275853d90 con 0x5572754a5390 2024-04-25T11:44:48.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.859+0000 7fad3cbf3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x557275732dc0 con 0x55727547c4d0 2024-04-25T11:44:48.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.859+0000 7fad3cbf3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x557275853f10 con 0x5572754846b0 2024-04-25T11:44:48.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.859+0000 7fad3cbf3740 10 client.0 _ll_register_callbacks cb 0x55727574e600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:48.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.859+0000 7fad2effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572754a5390 0x55727585d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.859+0000 7fad2effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572754a5390 0x55727585d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48446/0 (socket says 172.21.15.90:48446) 2024-04-25T11:44:48.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.859+0000 7fad2effd640 1 -- 172.21.15.90:0/2853048326 learned_addr learned my addr 172.21.15.90:0/2853048326 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:48.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.859+0000 7fad2f7fe640 1 --2- 172.21.15.90:0/2853048326 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572754846b0 0x5572758517f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.860+0000 7fad2effd640 1 --2- 172.21.15.90:0/2853048326 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55727547c4d0 0x5572758512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.860+0000 7fad2effd640 1 -- 172.21.15.90:0/2853048326 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572754846b0 msgr2=0x5572758517f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:48.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.860+0000 7fad2effd640 1 --2- 172.21.15.90:0/2853048326 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572754846b0 0x5572758517f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.860+0000 7fad2effd640 1 -- 172.21.15.90:0/2853048326 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55727547c4d0 msgr2=0x5572758512b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:48.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.860+0000 7fad2effd640 1 --2- 172.21.15.90:0/2853048326 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55727547c4d0 0x5572758512b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.860+0000 7fad2effd640 1 -- 172.21.15.90:0/2853048326 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572754bd3f0 con 0x5572754a5390 2024-04-25T11:44:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.860+0000 7fad2effd640 1 --2- 172.21.15.90:0/2853048326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572754a5390 0x55727585d210 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fad24009a70 tx=0x7fad24003a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fca9ceccedec8390 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.860+0000 7fad0effd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.860+0000 7fad0effd640 1 -- 172.21.15.90:0/2853048326 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2402a060 con 0x5572754a5390 2024-04-25T11:44:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.860+0000 7fad0effd640 1 -- 172.21.15.90:0/2853048326 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7fad24004020 con 0x5572754a5390 2024-04-25T11:44:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.860+0000 7fad3cbf3740 1 -- 172.21.15.90:0/2853048326 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad2401a070 con 0x5572754a5390 2024-04-25T11:44:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.860+0000 7fad0effd640 1 -- 172.21.15.90:0/2853048326 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fad24004220 con 0x5572754a5390 2024-04-25T11:44:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.860+0000 7fad3cbf3740 10 client.18630 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.860+0000 7fad3cbf3740 1 -- 172.21.15.90:0/2853048326 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5572755877d0 con 0x5572754a5390 2024-04-25T11:44:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.860+0000 7fad0effd640 1 -- 172.21.15.90:0/2853048326 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad2401a070 con 0x5572754a5390 2024-04-25T11:44:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.860+0000 7fad0effd640 1 -- 172.21.15.90:0/2853048326 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fad2400f040 con 0x5572754a5390 2024-04-25T11:44:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.860+0000 7fad3cbf3740 10 client.18630 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.860+0000 7fad3cbf3740 10 client.18630 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:44:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.860+0000 7fad3cbf3740 20 client.18630 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.860+0000 7fad3cbf3740 1 --2- 172.21.15.90:0/2853048326 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x557275463f60 0x5572758635c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.860+0000 7fad3cbf3740 4 client.18630 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:44:48.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.860+0000 7fad3cbf3740 1 -- 172.21.15.90:0/2853048326 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5572757b9990 con 0x557275463f60 2024-04-25T11:44:48.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.861+0000 7fad2f7fe640 1 --2- 172.21.15.90:0/2853048326 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x557275463f60 0x5572758635c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.861+0000 7fad0effd640 10 client.18630 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:44:48.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.861+0000 7fad2f7fe640 1 --2- 172.21.15.90:0/2853048326 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x557275463f60 0x5572758635c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a578ab7c94a4fe69 server_cookie=96b541a050fc8007 in_seq=0 out_seq=0 2024-04-25T11:44:48.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.862+0000 7fad0effd640 1 -- 172.21.15.90:0/2853048326 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5572757b9990 con 0x557275463f60 2024-04-25T11:44:48.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.862+0000 7fad0effd640 10 client.18630 handle_command_reply: tid=0 2024-04-25T11:44:48.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.862+0000 7fad3cbf3740 1 client.18630 shutdown 2024-04-25T11:44:48.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.862+0000 7fad3cbf3740 1 -- 172.21.15.90:0/2853048326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572754a5390 msgr2=0x55727585d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.862+0000 7fad3cbf3740 1 --2- 172.21.15.90:0/2853048326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572754a5390 0x55727585d210 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fad24009a70 tx=0x7fad24003a10 comp rx=0 tx=0).stop 2024-04-25T11:44:48.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.862+0000 7fad3cbf3740 1 -- 172.21.15.90:0/2853048326 shutdown_connections 2024-04-25T11:44:48.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.862+0000 7fad3cbf3740 1 --2- 172.21.15.90:0/2853048326 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x557275463f60 0x5572758635c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.862+0000 7fad3cbf3740 1 --2- 172.21.15.90:0/2853048326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572754a5390 0x55727585d210 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.862+0000 7fad3cbf3740 1 --2- 172.21.15.90:0/2853048326 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572754846b0 0x5572758517f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.862+0000 7fad0effd640 0 client.18630 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:44:48.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.862+0000 7fad3cbf3740 1 --2- 172.21.15.90:0/2853048326 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55727547c4d0 0x5572758512b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.862+0000 7fad3cbf3740 1 -- 172.21.15.90:0/2853048326 >> 172.21.15.90:0/2853048326 conn(0x5572754994e0 msgr2=0x557275848530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:48.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.862+0000 7fad3cbf3740 1 -- 172.21.15.90:0/2853048326 shutdown_connections 2024-04-25T11:44:48.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.862+0000 7fad3cbf3740 1 -- 172.21.15.90:0/2853048326 wait complete. 2024-04-25T11:44:48.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.862+0000 7fad3cbf3740 20 client.18630 trim_cache size 0 max 16384 2024-04-25T11:44:48.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.876+0000 7fad3cbf3740 1 Processor -- start 2024-04-25T11:44:48.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.876+0000 7fad3cbf3740 1 -- start start 2024-04-25T11:44:48.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.876+0000 7fad3cbf3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557275463f60 0x55727585d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.876+0000 7fad3cbf3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55727547c4d0 0x5572758503a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.876+0000 7fad3cbf3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572754846b0 0x557275841d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.876+0000 7fad3cbf3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fad301afd10 con 0x55727547c4d0 2024-04-25T11:44:48.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.876+0000 7fad3cbf3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fad301b1850 con 0x557275463f60 2024-04-25T11:44:48.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.876+0000 7fad3cbf3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fad301b03e0 con 0x5572754846b0 2024-04-25T11:44:48.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.876+0000 7fad2f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55727547c4d0 0x5572758503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.876+0000 7fad2f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55727547c4d0 0x5572758503a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48462/0 (socket says 172.21.15.90:48462) 2024-04-25T11:44:48.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.876+0000 7fad2f7fe640 1 -- 172.21.15.90:0/448265280 learned_addr learned my addr 172.21.15.90:0/448265280 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:48.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.876+0000 7fad2f7fe640 1 -- 172.21.15.90:0/448265280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572754846b0 msgr2=0x557275841d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:48.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.876+0000 7fad2f7fe640 1 --2- 172.21.15.90:0/448265280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572754846b0 0x557275841d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.876+0000 7fad2f7fe640 1 -- 172.21.15.90:0/448265280 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557275463f60 msgr2=0x55727585d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:48.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.876+0000 7fad2f7fe640 1 --2- 172.21.15.90:0/448265280 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557275463f60 0x55727585d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.876+0000 7fad2f7fe640 1 -- 172.21.15.90:0/448265280 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572758465a0 con 0x55727547c4d0 2024-04-25T11:44:48.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.876+0000 7fad2f7fe640 1 --2- 172.21.15.90:0/448265280 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55727547c4d0 0x5572758503a0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fad20005ab0 tx=0x7fad20032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f8fab60e43bf821 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:48.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.877+0000 7fad157fa640 1 -- 172.21.15.90:0/448265280 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20035040 con 0x55727547c4d0 2024-04-25T11:44:48.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.877+0000 7fad157fa640 1 -- 172.21.15.90:0/448265280 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad20005040 con 0x55727547c4d0 2024-04-25T11:44:48.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.877+0000 7fad157fa640 1 -- 172.21.15.90:0/448265280 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20004a40 con 0x55727547c4d0 2024-04-25T11:44:48.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.877+0000 7fad3cbf3740 1 -- 172.21.15.90:0/448265280 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55727547c4d0 msgr2=0x5572758503a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.877+0000 7fad3cbf3740 1 --2- 172.21.15.90:0/448265280 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55727547c4d0 0x5572758503a0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fad20005ab0 tx=0x7fad20032ea0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.877+0000 7fad3cbf3740 1 -- 172.21.15.90:0/448265280 shutdown_connections 2024-04-25T11:44:48.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.877+0000 7fad3cbf3740 1 --2- 172.21.15.90:0/448265280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572754846b0 0x557275841d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.877+0000 7fad3cbf3740 1 --2- 172.21.15.90:0/448265280 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55727547c4d0 0x5572758503a0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.877+0000 7fad3cbf3740 1 --2- 172.21.15.90:0/448265280 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557275463f60 0x55727585d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.877+0000 7fad3cbf3740 1 -- 172.21.15.90:0/448265280 >> 172.21.15.90:0/448265280 conn(0x5572754994e0 msgr2=0x557275841610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:48.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.877+0000 7fad3cbf3740 1 -- 172.21.15.90:0/448265280 shutdown_connections 2024-04-25T11:44:48.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.877+0000 7fad3cbf3740 1 -- 172.21.15.90:0/448265280 wait complete. 2024-04-25T11:44:48.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.878+0000 7fad3cbf3740 1 -- start start 2024-04-25T11:44:48.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.878+0000 7fad3cbf3740 1 Processor -- start 2024-04-25T11:44:48.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.878+0000 7fad3cbf3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557275463f60 0x557275863900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.878+0000 7fad3cbf3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55727547c4d0 0x557275863e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.878+0000 7fad3cbf3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5572754846b0 0x5572758c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.878+0000 7fad3cbf3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fad301b1180 con 0x55727547c4d0 2024-04-25T11:44:48.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.878+0000 7fad3cbf3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fad301b0ab0 con 0x5572754846b0 2024-04-25T11:44:48.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.878+0000 7fad3cbf3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fad301add10 con 0x557275463f60 2024-04-25T11:44:48.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.878+0000 7fad3cbf3740 10 client.0 _ll_register_callbacks cb 0x7fad24009c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:48.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.878+0000 7fad2f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55727547c4d0 0x557275863e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.878+0000 7fad2f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55727547c4d0 0x557275863e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48476/0 (socket says 172.21.15.90:48476) 2024-04-25T11:44:48.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.878+0000 7fad2f7fe640 1 -- 172.21.15.90:0/1170851416 learned_addr learned my addr 172.21.15.90:0/1170851416 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:48.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.879+0000 7fad2effd640 1 --2- 172.21.15.90:0/1170851416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557275463f60 0x557275863900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.879+0000 7fad2effd640 1 --2- 172.21.15.90:0/1170851416 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5572754846b0 0x5572758c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.879+0000 7fad2f7fe640 1 -- 172.21.15.90:0/1170851416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557275463f60 msgr2=0x557275863900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:48.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.879+0000 7fad2f7fe640 1 --2- 172.21.15.90:0/1170851416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557275463f60 0x557275863900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.879+0000 7fad2f7fe640 1 -- 172.21.15.90:0/1170851416 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5572754846b0 msgr2=0x5572758c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:48.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.879+0000 7fad2f7fe640 1 --2- 172.21.15.90:0/1170851416 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5572754846b0 0x5572758c96e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.879+0000 7fad2f7fe640 1 -- 172.21.15.90:0/1170851416 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572758c9c20 con 0x55727547c4d0 2024-04-25T11:44:48.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.879+0000 7fad2effd640 1 --2- 172.21.15.90:0/1170851416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557275463f60 0x557275863900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:48.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.879+0000 7fad2f7fe640 1 --2- 172.21.15.90:0/1170851416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55727547c4d0 0x557275863e40 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fad20006010 tx=0x7fad20032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=312cd45d27a1af9f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:48.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.879+0000 7fad0effd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:48.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.879+0000 7fad0effd640 1 -- 172.21.15.90:0/1170851416 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20035040 con 0x55727547c4d0 2024-04-25T11:44:48.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.879+0000 7fad0effd640 1 -- 172.21.15.90:0/1170851416 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7fad2004a040 con 0x55727547c4d0 2024-04-25T11:44:48.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.879+0000 7fad3cbf3740 1 -- 172.21.15.90:0/1170851416 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad20005040 con 0x55727547c4d0 2024-04-25T11:44:48.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.879+0000 7fad0effd640 1 -- 172.21.15.90:0/1170851416 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fad200051f0 con 0x55727547c4d0 2024-04-25T11:44:48.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.879+0000 7fad3cbf3740 10 client.18636 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:48.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.879+0000 7fad3cbf3740 1 -- 172.21.15.90:0/1170851416 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5572758cade0 con 0x55727547c4d0 2024-04-25T11:44:48.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.879+0000 7fad0effd640 1 -- 172.21.15.90:0/1170851416 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad20005040 con 0x55727547c4d0 2024-04-25T11:44:48.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.880+0000 7fad0effd640 1 -- 172.21.15.90:0/1170851416 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fad200037b0 con 0x55727547c4d0 2024-04-25T11:44:48.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.880+0000 7fad3cbf3740 10 client.18636 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:48.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.880+0000 7fad3cbf3740 10 client.18636 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:44:48.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.880+0000 7fad3cbf3740 20 client.18636 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:48.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.880+0000 7fad3cbf3740 1 --2- 172.21.15.90:0/1170851416 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x5572754a5390 0x5572758cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.880+0000 7fad3cbf3740 4 client.18636 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:44:48.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.880+0000 7fad3cbf3740 1 -- 172.21.15.90:0/1170851416 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5572757b3800 con 0x5572754a5390 2024-04-25T11:44:48.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.880+0000 7fad2effd640 1 --2- 172.21.15.90:0/1170851416 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x5572754a5390 0x5572758cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.880+0000 7fad0effd640 10 client.18636 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:44:48.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.880+0000 7fad2effd640 1 --2- 172.21.15.90:0/1170851416 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x5572754a5390 0x5572758cd560 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a492983696496b7 server_cookie=a5db07e1865b6300 in_seq=0 out_seq=0 2024-04-25T11:44:48.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.880+0000 7fad0effd640 1 -- 172.21.15.90:0/1170851416 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5572757b3800 con 0x5572754a5390 2024-04-25T11:44:48.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.880+0000 7fad0effd640 10 client.18636 handle_command_reply: tid=0 2024-04-25T11:44:48.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.880+0000 7fad3cbf3740 1 client.18636 shutdown 2024-04-25T11:44:48.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.881+0000 7fad3cbf3740 1 -- 172.21.15.90:0/1170851416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55727547c4d0 msgr2=0x557275863e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.881+0000 7fad3cbf3740 1 --2- 172.21.15.90:0/1170851416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55727547c4d0 0x557275863e40 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fad20006010 tx=0x7fad20032f90 comp rx=0 tx=0).stop 2024-04-25T11:44:48.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.881+0000 7fad3cbf3740 1 -- 172.21.15.90:0/1170851416 shutdown_connections 2024-04-25T11:44:48.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.881+0000 7fad3cbf3740 1 --2- 172.21.15.90:0/1170851416 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x5572754a5390 0x5572758cd560 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.881+0000 7fad3cbf3740 1 --2- 172.21.15.90:0/1170851416 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5572754846b0 0x5572758c96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.881+0000 7fad0effd640 0 client.18636 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:44:48.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.881+0000 7fad3cbf3740 1 --2- 172.21.15.90:0/1170851416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55727547c4d0 0x557275863e40 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.881+0000 7fad3cbf3740 1 --2- 172.21.15.90:0/1170851416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557275463f60 0x557275863900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.881+0000 7fad3cbf3740 1 -- 172.21.15.90:0/1170851416 >> 172.21.15.90:0/1170851416 conn(0x5572754994e0 msgr2=0x557275850100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:48.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.881+0000 7fad3cbf3740 1 -- 172.21.15.90:0/1170851416 shutdown_connections 2024-04-25T11:44:48.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.881+0000 7fad3cbf3740 1 -- 172.21.15.90:0/1170851416 wait complete. 2024-04-25T11:44:48.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.881+0000 7fad3cbf3740 20 client.18636 trim_cache size 0 max 16384 2024-04-25T11:44:48.875 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:44:48.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.881+0000 7fad35d3f640 1 -- 172.21.15.90:0/3223412867 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fad180afc60 msgr2=0x7fad180b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.881+0000 7fad35d3f640 1 --2- 172.21.15.90:0/3223412867 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fad180afc60 0x7fad180b2170 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fad280a4ae0 tx=0x7fad280750d0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.881+0000 7fad35d3f640 1 -- 172.21.15.90:0/3223412867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad3010e060 msgr2=0x7fad3019ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.881+0000 7fad35d3f640 1 --2- 172.21.15.90:0/3223412867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad3010e060 0x7fad3019ef60 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fad2800eb10 tx=0x7fad2800c2d0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.881+0000 7fad35d3f640 1 -- 172.21.15.90:0/3223412867 shutdown_connections 2024-04-25T11:44:48.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.881+0000 7fad35d3f640 1 --2- 172.21.15.90:0/3223412867 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fad180afc60 0x7fad180b2170 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.881+0000 7fad35d3f640 1 --2- 172.21.15.90:0/3223412867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad3010e060 0x7fad3019ef60 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.881+0000 7fad35d3f640 1 --2- 172.21.15.90:0/3223412867 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fad3010b6b0 0x7fad3019aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.881+0000 7fad35d3f640 1 --2- 172.21.15.90:0/3223412867 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad30104c50 0x7fad3019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.881+0000 7fad35d3f640 1 -- 172.21.15.90:0/3223412867 >> 172.21.15.90:0/3223412867 conn(0x7fad30078100 msgr2=0x7fad30101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:48.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.881+0000 7fad35d3f640 1 -- 172.21.15.90:0/3223412867 shutdown_connections 2024-04-25T11:44:48.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:48.881+0000 7fad35d3f640 1 -- 172.21.15.90:0/3223412867 wait complete. 2024-04-25T11:44:48.879 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T11:44:48.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.003+0000 7f80bc99d640 1 -- 172.21.15.90:0/3487693469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80b8103a80 msgr2=0x7f80b8103e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.003+0000 7f80bc99d640 1 --2- 172.21.15.90:0/3487693469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80b8103a80 0x7f80b8103e90 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f80ac009900 tx=0x7f80ac02a410 comp rx=0 tx=0).stop 2024-04-25T11:44:48.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.003+0000 7f80bc99d640 1 -- 172.21.15.90:0/3487693469 shutdown_connections 2024-04-25T11:44:48.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.003+0000 7f80bc99d640 1 --2- 172.21.15.90:0/3487693469 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f80b810e310 0x7f80b8110730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.003+0000 7f80bc99d640 1 --2- 172.21.15.90:0/3487693469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80b8103a80 0x7f80b8103e90 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.003+0000 7f80bc99d640 1 --2- 172.21.15.90:0/3487693469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80b81099b0 0x7f80b8103540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.003+0000 7f80bc99d640 1 -- 172.21.15.90:0/3487693469 >> 172.21.15.90:0/3487693469 conn(0x7f80b80ff1a0 msgr2=0x7f80b81015c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:48.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.003+0000 7f80bc99d640 1 -- 172.21.15.90:0/3487693469 shutdown_connections 2024-04-25T11:44:48.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.003+0000 7f80bc99d640 1 -- 172.21.15.90:0/3487693469 wait complete. 2024-04-25T11:44:48.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.003+0000 7f80bc99d640 1 Processor -- start 2024-04-25T11:44:48.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.004+0000 7f80bc99d640 1 -- start start 2024-04-25T11:44:48.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.004+0000 7f80bc99d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f80b8103a80 0x7f80b819a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.004+0000 7f80bc99d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80b81099b0 0x7f80b819aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.004+0000 7f80bc99d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80b810e310 0x7f80b819edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.004+0000 7f80bc99d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f80b8113b70 con 0x7f80b810e310 2024-04-25T11:44:48.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.004+0000 7f80bc99d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f80b81139f0 con 0x7f80b8103a80 2024-04-25T11:44:48.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.004+0000 7f80bc99d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f80b8113cf0 con 0x7f80b81099b0 2024-04-25T11:44:48.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.004+0000 7f80b7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80b810e310 0x7f80b819edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.004+0000 7f80b7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80b810e310 0x7f80b819edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48482/0 (socket says 172.21.15.90:48482) 2024-04-25T11:44:48.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.004+0000 7f80b7fff640 1 -- 172.21.15.90:0/2875130231 learned_addr learned my addr 172.21.15.90:0/2875130231 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:48.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.004+0000 7f80b6ffd640 1 --2- 172.21.15.90:0/2875130231 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80b81099b0 0x7f80b819aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.004+0000 7f80b77fe640 1 --2- 172.21.15.90:0/2875130231 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f80b8103a80 0x7f80b819a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.004+0000 7f80b7fff640 1 -- 172.21.15.90:0/2875130231 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80b81099b0 msgr2=0x7f80b819aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.004+0000 7f80b7fff640 1 --2- 172.21.15.90:0/2875130231 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80b81099b0 0x7f80b819aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.004+0000 7f80b7fff640 1 -- 172.21.15.90:0/2875130231 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f80b8103a80 msgr2=0x7f80b819a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.004+0000 7f80b7fff640 1 --2- 172.21.15.90:0/2875130231 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f80b8103a80 0x7f80b819a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.004+0000 7f80b7fff640 1 -- 172.21.15.90:0/2875130231 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80b819f370 con 0x7f80b810e310 2024-04-25T11:44:48.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.004+0000 7f80b7fff640 1 --2- 172.21.15.90:0/2875130231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80b810e310 0x7f80b819edb0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f80b000cd10 tx=0x7f80b000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:48.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.005+0000 7f80b4ff9640 1 -- 172.21.15.90:0/2875130231 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b000f450 con 0x7f80b810e310 2024-04-25T11:44:48.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.005+0000 7f80b4ff9640 1 -- 172.21.15.90:0/2875130231 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80b000faa0 con 0x7f80b810e310 2024-04-25T11:44:48.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.005+0000 7f80bc99d640 1 -- 172.21.15.90:0/2875130231 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80b819f660 con 0x7f80b810e310 2024-04-25T11:44:48.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.005+0000 7f80b4ff9640 1 -- 172.21.15.90:0/2875130231 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b0018b50 con 0x7f80b810e310 2024-04-25T11:44:48.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.005+0000 7f80bc99d640 1 -- 172.21.15.90:0/2875130231 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80b819fb30 con 0x7f80b810e310 2024-04-25T11:44:49.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.007+0000 7f80c4b8c740 1 Processor -- start 2024-04-25T11:44:49.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.007+0000 7f80b4ff9640 1 -- 172.21.15.90:0/2875130231 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f80b000fc50 con 0x7f80b810e310 2024-04-25T11:44:49.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.008+0000 7f80c4b8c740 1 -- start start 2024-04-25T11:44:49.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.008+0000 7f80c4b8c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a2c390 0x556266dcd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.008+0000 7f80c4b8c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556266a034d0 0x556266dcdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.008+0000 7f80c4b8c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556266a0b6b0 0x556266dda870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.008+0000 7f80c4b8c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556266d32d80 con 0x556266a2c390 2024-04-25T11:44:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.008+0000 7f80c4b8c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x556266ceca20 con 0x556266a034d0 2024-04-25T11:44:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.008+0000 7f80c4b8c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x556266d2ed40 con 0x556266a0b6b0 2024-04-25T11:44:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.008+0000 7f80b6ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a2c390 0x556266dcd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.008+0000 7f80b6ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a2c390 0x556266dcd9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48498/0 (socket says 172.21.15.90:48498) 2024-04-25T11:44:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.008+0000 7f80b6ffd640 1 -- 172.21.15.90:0/1131324896 learned_addr learned my addr 172.21.15.90:0/1131324896 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.008+0000 7f80b77fe640 1 --2- 172.21.15.90:0/1131324896 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556266a034d0 0x556266dcdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.008+0000 7f80b6ffd640 1 --2- 172.21.15.90:0/1131324896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556266a0b6b0 0x556266dda870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.008+0000 7f80b4ff9640 1 --2- 172.21.15.90:0/2875130231 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f80a00afc50 0x7f80a00b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.008+0000 7f80b6ffd640 1 -- 172.21.15.90:0/1131324896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556266a0b6b0 msgr2=0x556266dda870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.008+0000 7f80b6ffd640 1 --2- 172.21.15.90:0/1131324896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556266a0b6b0 0x556266dda870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.008+0000 7f80b6ffd640 1 -- 172.21.15.90:0/1131324896 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556266a034d0 msgr2=0x556266dcdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.008+0000 7f80b6ffd640 1 --2- 172.21.15.90:0/1131324896 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556266a034d0 0x556266dcdf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.008+0000 7f80b6ffd640 1 -- 172.21.15.90:0/1131324896 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556266cf7c30 con 0x556266a2c390 2024-04-25T11:44:49.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.008+0000 7f80b7fff640 1 --2- 172.21.15.90:0/2875130231 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f80a00afc50 0x7f80a00b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.008+0000 7f80b6ffd640 1 --2- 172.21.15.90:0/1131324896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a2c390 0x556266dcd9e0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f80b806c250 tx=0x7f80ac000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=119dc871d52e9aad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:49.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.008+0000 7f80b4ff9640 1 -- 172.21.15.90:0/2875130231 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f80b00df4d0 con 0x7f80b810e310 2024-04-25T11:44:49.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.009+0000 7f809dffb640 1 -- 172.21.15.90:0/1131324896 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ac044020 con 0x556266a2c390 2024-04-25T11:44:49.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.009+0000 7f809dffb640 1 -- 172.21.15.90:0/1131324896 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80ac0316a0 con 0x556266a2c390 2024-04-25T11:44:49.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.009+0000 7f809dffb640 1 -- 172.21.15.90:0/1131324896 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ac035040 con 0x556266a2c390 2024-04-25T11:44:49.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.009+0000 7f80c4b8c740 1 -- 172.21.15.90:0/1131324896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a2c390 msgr2=0x556266dcd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.009+0000 7f80c4b8c740 1 --2- 172.21.15.90:0/1131324896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a2c390 0x556266dcd9e0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f80b806c250 tx=0x7f80ac000bc0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.009+0000 7f80b7fff640 1 --2- 172.21.15.90:0/2875130231 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f80a00afc50 0x7f80a00b2160 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f80b00dcc40 tx=0x7f80b000ff70 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:49.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.010+0000 7f80c4b8c740 1 -- 172.21.15.90:0/1131324896 shutdown_connections 2024-04-25T11:44:49.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.010+0000 7f80c4b8c740 1 --2- 172.21.15.90:0/1131324896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556266a0b6b0 0x556266dda870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.010+0000 7f80c4b8c740 1 --2- 172.21.15.90:0/1131324896 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556266a034d0 0x556266dcdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.010+0000 7f80c4b8c740 1 --2- 172.21.15.90:0/1131324896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a2c390 0x556266dcd9e0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.010+0000 7f80c4b8c740 1 -- 172.21.15.90:0/1131324896 >> 172.21.15.90:0/1131324896 conn(0x556266a204e0 msgr2=0x556266dca190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:49.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.010+0000 7f80c4b8c740 1 -- 172.21.15.90:0/1131324896 shutdown_connections 2024-04-25T11:44:49.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.010+0000 7f80c4b8c740 1 -- 172.21.15.90:0/1131324896 wait complete. 2024-04-25T11:44:49.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.010+0000 7f80c4b8c740 1 -- start start 2024-04-25T11:44:49.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.010+0000 7f80c4b8c740 1 Processor -- start 2024-04-25T11:44:49.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.010+0000 7f80c4b8c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556266a034d0 0x556266dd8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.010+0000 7f80c4b8c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a0b6b0 0x556266dd8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.010+0000 7f80c4b8c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556266a2c390 0x556266de4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.010+0000 7f80c4b8c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556266ddaf00 con 0x556266a0b6b0 2024-04-25T11:44:49.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.010+0000 7f80c4b8c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x556266cb9dc0 con 0x556266a2c390 2024-04-25T11:44:49.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.010+0000 7f80c4b8c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x556266ddb080 con 0x556266a034d0 2024-04-25T11:44:49.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.011+0000 7f80c4b8c740 10 client.0 _ll_register_callbacks cb 0x556266cd5600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:49.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.011+0000 7f80b77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a0b6b0 0x556266dd8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.011+0000 7f80b77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a0b6b0 0x556266dd8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48504/0 (socket says 172.21.15.90:48504) 2024-04-25T11:44:49.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.011+0000 7f80b77fe640 1 -- 172.21.15.90:0/2181221747 learned_addr learned my addr 172.21.15.90:0/2181221747 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:49.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.011+0000 7f80b6ffd640 1 --2- 172.21.15.90:0/2181221747 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556266a2c390 0x556266de4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.011+0000 7f80b6ffd640 1 --2- 172.21.15.90:0/2181221747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556266a034d0 0x556266dd8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.011+0000 7f80b77fe640 1 -- 172.21.15.90:0/2181221747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556266a034d0 msgr2=0x556266dd8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.011+0000 7f80b77fe640 1 --2- 172.21.15.90:0/2181221747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556266a034d0 0x556266dd8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.011+0000 7f80b77fe640 1 -- 172.21.15.90:0/2181221747 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556266a2c390 msgr2=0x556266de4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.011+0000 7f80b77fe640 1 --2- 172.21.15.90:0/2181221747 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556266a2c390 0x556266de4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.011+0000 7f80b77fe640 1 -- 172.21.15.90:0/2181221747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556266a443f0 con 0x556266a0b6b0 2024-04-25T11:44:49.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.011+0000 7f80b6ffd640 1 --2- 172.21.15.90:0/2181221747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556266a034d0 0x556266dd8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:49.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.011+0000 7f80b77fe640 1 --2- 172.21.15.90:0/2181221747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a0b6b0 0x556266dd8990 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f80a800bf00 tx=0x7f80a8012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad2e6ec3d21d0f7d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:49.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.011+0000 7f8096ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:49.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.011+0000 7f8096ffd640 1 -- 172.21.15.90:0/2181221747 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a8005220 con 0x556266a0b6b0 2024-04-25T11:44:49.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.011+0000 7f8096ffd640 1 -- 172.21.15.90:0/2181221747 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f80a80053c0 con 0x556266a0b6b0 2024-04-25T11:44:49.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.011+0000 7f80c4b8c740 1 -- 172.21.15.90:0/2181221747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80ac0316a0 con 0x556266a0b6b0 2024-04-25T11:44:49.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.011+0000 7f8096ffd640 1 -- 172.21.15.90:0/2181221747 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f80a80055c0 con 0x556266a0b6b0 2024-04-25T11:44:49.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.011+0000 7f80c4b8c740 10 client.18648 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:49.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.011+0000 7f80c4b8c740 1 -- 172.21.15.90:0/2181221747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556266b0e7d0 con 0x556266a0b6b0 2024-04-25T11:44:49.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.011+0000 7f8096ffd640 1 -- 172.21.15.90:0/2181221747 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80ac0316a0 con 0x556266a0b6b0 2024-04-25T11:44:49.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.012+0000 7f8096ffd640 1 -- 172.21.15.90:0/2181221747 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f80a8019070 con 0x556266a0b6b0 2024-04-25T11:44:49.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.012+0000 7f80c4b8c740 10 client.18648 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:49.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.012+0000 7f80c4b8c740 10 client.18648 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:44:49.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.012+0000 7f80c4b8c740 20 client.18648 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:49.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.012+0000 7f80c4b8c740 1 --2- 172.21.15.90:0/2181221747 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5562669eaf60 0x556266dea910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.012+0000 7f80c4b8c740 4 client.18648 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:44:49.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.012+0000 7f80c4b8c740 1 -- 172.21.15.90:0/2181221747 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556266d40990 con 0x5562669eaf60 2024-04-25T11:44:49.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.012+0000 7f80b6ffd640 1 --2- 172.21.15.90:0/2181221747 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5562669eaf60 0x556266dea910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.012+0000 7f8096ffd640 10 client.18648 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:44:49.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.012+0000 7f80b6ffd640 1 --2- 172.21.15.90:0/2181221747 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5562669eaf60 0x556266dea910 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7894f4bf944e81c9 server_cookie=1da1269787550e56 in_seq=0 out_seq=0 2024-04-25T11:44:49.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.013+0000 7f8096ffd640 1 -- 172.21.15.90:0/2181221747 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556266d40990 con 0x5562669eaf60 2024-04-25T11:44:49.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.013+0000 7f8096ffd640 10 client.18648 handle_command_reply: tid=0 2024-04-25T11:44:49.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.013+0000 7f80c4b8c740 1 client.18648 shutdown 2024-04-25T11:44:49.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.013+0000 7f80c4b8c740 1 -- 172.21.15.90:0/2181221747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a0b6b0 msgr2=0x556266dd8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.013+0000 7f80c4b8c740 1 --2- 172.21.15.90:0/2181221747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a0b6b0 0x556266dd8990 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f80a800bf00 tx=0x7f80a8012820 comp rx=0 tx=0).stop 2024-04-25T11:44:49.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.013+0000 7f80c4b8c740 1 -- 172.21.15.90:0/2181221747 shutdown_connections 2024-04-25T11:44:49.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.013+0000 7f80c4b8c740 1 --2- 172.21.15.90:0/2181221747 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5562669eaf60 0x556266dea910 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.013+0000 7f80c4b8c740 1 --2- 172.21.15.90:0/2181221747 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556266a2c390 0x556266de4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.013+0000 7f8096ffd640 0 client.18648 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:44:49.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.013+0000 7f80c4b8c740 1 --2- 172.21.15.90:0/2181221747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a0b6b0 0x556266dd8990 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.013+0000 7f80c4b8c740 1 --2- 172.21.15.90:0/2181221747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556266a034d0 0x556266dd8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.013+0000 7f80c4b8c740 1 -- 172.21.15.90:0/2181221747 >> 172.21.15.90:0/2181221747 conn(0x556266a204e0 msgr2=0x556266dcf700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:49.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.014+0000 7f80c4b8c740 1 -- 172.21.15.90:0/2181221747 shutdown_connections 2024-04-25T11:44:49.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.014+0000 7f80c4b8c740 1 -- 172.21.15.90:0/2181221747 wait complete. 2024-04-25T11:44:49.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.014+0000 7f80c4b8c740 20 client.18648 trim_cache size 0 max 16384 2024-04-25T11:44:49.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.027+0000 7f80c4b8c740 1 Processor -- start 2024-04-25T11:44:49.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.027+0000 7f80c4b8c740 1 -- start start 2024-04-25T11:44:49.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.027+0000 7f80c4b8c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556266a034d0 0x556266dcde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.027+0000 7f80c4b8c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a0b6b0 0x556266de3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.027+0000 7f80c4b8c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556266a2c390 0x556266dd2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.027+0000 7f80c4b8c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f80b81af750 con 0x556266a0b6b0 2024-04-25T11:44:49.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.027+0000 7f80c4b8c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f80b81b1290 con 0x556266a034d0 2024-04-25T11:44:49.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.027+0000 7f80c4b8c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f80b81afe20 con 0x556266a2c390 2024-04-25T11:44:49.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.027+0000 7f80b77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a0b6b0 0x556266de3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.027+0000 7f80b77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a0b6b0 0x556266de3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48514/0 (socket says 172.21.15.90:48514) 2024-04-25T11:44:49.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.027+0000 7f80b77fe640 1 -- 172.21.15.90:0/2978809619 learned_addr learned my addr 172.21.15.90:0/2978809619 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:49.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.028+0000 7f80b6ffd640 1 --2- 172.21.15.90:0/2978809619 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556266a034d0 0x556266dcde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.028+0000 7f80b77fe640 1 -- 172.21.15.90:0/2978809619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556266a2c390 msgr2=0x556266dd2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.028+0000 7f80b77fe640 1 --2- 172.21.15.90:0/2978809619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556266a2c390 0x556266dd2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.028+0000 7f80b77fe640 1 -- 172.21.15.90:0/2978809619 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556266a034d0 msgr2=0x556266dcde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.028+0000 7f80b77fe640 1 --2- 172.21.15.90:0/2978809619 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556266a034d0 0x556266dcde80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.028+0000 7f80b77fe640 1 -- 172.21.15.90:0/2978809619 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556266dbbba0 con 0x556266a0b6b0 2024-04-25T11:44:49.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.028+0000 7f80b77fe640 1 --2- 172.21.15.90:0/2978809619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a0b6b0 0x556266de3fe0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f80a800bed0 tx=0x7f80a800bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34700774fc33bfe8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:49.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.028+0000 7f809d7fa640 1 -- 172.21.15.90:0/2978809619 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a8010960 con 0x556266a0b6b0 2024-04-25T11:44:49.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.028+0000 7f809d7fa640 1 -- 172.21.15.90:0/2978809619 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80a801a960 con 0x556266a0b6b0 2024-04-25T11:44:49.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.028+0000 7f809d7fa640 1 -- 172.21.15.90:0/2978809619 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a8021910 con 0x556266a0b6b0 2024-04-25T11:44:49.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.029+0000 7f80c4b8c740 1 -- 172.21.15.90:0/2978809619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a0b6b0 msgr2=0x556266de3fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.029+0000 7f80c4b8c740 1 --2- 172.21.15.90:0/2978809619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a0b6b0 0x556266de3fe0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f80a800bed0 tx=0x7f80a800bb00 comp rx=0 tx=0).stop 2024-04-25T11:44:49.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.029+0000 7f80c4b8c740 1 -- 172.21.15.90:0/2978809619 shutdown_connections 2024-04-25T11:44:49.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.029+0000 7f80c4b8c740 1 --2- 172.21.15.90:0/2978809619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556266a2c390 0x556266dd2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.029+0000 7f80c4b8c740 1 --2- 172.21.15.90:0/2978809619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a0b6b0 0x556266de3fe0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.029+0000 7f80c4b8c740 1 --2- 172.21.15.90:0/2978809619 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556266a034d0 0x556266dcde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.029+0000 7f80c4b8c740 1 -- 172.21.15.90:0/2978809619 >> 172.21.15.90:0/2978809619 conn(0x556266a204e0 msgr2=0x556266dcd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:49.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.029+0000 7f80c4b8c740 1 -- 172.21.15.90:0/2978809619 shutdown_connections 2024-04-25T11:44:49.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.029+0000 7f80c4b8c740 1 -- 172.21.15.90:0/2978809619 wait complete. 2024-04-25T11:44:49.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.029+0000 7f80c4b8c740 1 -- start start 2024-04-25T11:44:49.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.029+0000 7f80c4b8c740 1 Processor -- start 2024-04-25T11:44:49.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.030+0000 7f80c4b8c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556266a034d0 0x556266de0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.030+0000 7f80c4b8c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a0b6b0 0x556266ddb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.030+0000 7f80c4b8c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556266a2c390 0x556266ddbf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.030+0000 7f80c4b8c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f80b81b0bc0 con 0x556266a0b6b0 2024-04-25T11:44:49.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.030+0000 7f80c4b8c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f80b81b04f0 con 0x556266a2c390 2024-04-25T11:44:49.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.030+0000 7f80c4b8c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f80b81ad750 con 0x556266a034d0 2024-04-25T11:44:49.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.030+0000 7f80c4b8c740 10 client.0 _ll_register_callbacks cb 0x7f80a800b730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:49.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.030+0000 7f80b77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a0b6b0 0x556266ddb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.030+0000 7f80b77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a0b6b0 0x556266ddb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48530/0 (socket says 172.21.15.90:48530) 2024-04-25T11:44:49.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.030+0000 7f80b77fe640 1 -- 172.21.15.90:0/1311040127 learned_addr learned my addr 172.21.15.90:0/1311040127 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:49.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.030+0000 7f80b6ffd640 1 --2- 172.21.15.90:0/1311040127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556266a034d0 0x556266de0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.030+0000 7f80b6ffd640 1 --2- 172.21.15.90:0/1311040127 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556266a2c390 0x556266ddbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.030+0000 7f80b77fe640 1 -- 172.21.15.90:0/1311040127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556266a034d0 msgr2=0x556266de0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.030+0000 7f80b77fe640 1 --2- 172.21.15.90:0/1311040127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556266a034d0 0x556266de0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.030+0000 7f80b77fe640 1 -- 172.21.15.90:0/1311040127 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556266a2c390 msgr2=0x556266ddbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.030+0000 7f80b77fe640 1 --2- 172.21.15.90:0/1311040127 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556266a2c390 0x556266ddbf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.030+0000 7f80b77fe640 1 -- 172.21.15.90:0/1311040127 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556266ddc460 con 0x556266a0b6b0 2024-04-25T11:44:49.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.030+0000 7f80b77fe640 1 --2- 172.21.15.90:0/1311040127 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a0b6b0 0x556266ddb9e0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f80a8010cf0 tx=0x7f80a8012da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e1fa0af24600ad9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:49.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.030+0000 7f8096ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:49.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.030+0000 7f8096ffd640 1 -- 172.21.15.90:0/1311040127 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a8010760 con 0x556266a0b6b0 2024-04-25T11:44:49.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.031+0000 7f8096ffd640 1 -- 172.21.15.90:0/1311040127 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f80a8010900 con 0x556266a0b6b0 2024-04-25T11:44:49.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.031+0000 7f80c4b8c740 1 -- 172.21.15.90:0/1311040127 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80a801a960 con 0x556266a0b6b0 2024-04-25T11:44:49.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.031+0000 7f8096ffd640 1 -- 172.21.15.90:0/1311040127 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f80a8040050 con 0x556266a0b6b0 2024-04-25T11:44:49.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.031+0000 7f80c4b8c740 10 client.18654 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:49.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.031+0000 7f80c4b8c740 1 -- 172.21.15.90:0/1311040127 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556266e528c0 con 0x556266a0b6b0 2024-04-25T11:44:49.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.031+0000 7f8096ffd640 1 -- 172.21.15.90:0/1311040127 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80a801a960 con 0x556266a0b6b0 2024-04-25T11:44:49.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.031+0000 7f8096ffd640 1 -- 172.21.15.90:0/1311040127 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f80a803b030 con 0x556266a0b6b0 2024-04-25T11:44:49.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.031+0000 7f80c4b8c740 10 client.18654 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:49.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.031+0000 7f80c4b8c740 10 client.18654 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:44:49.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.031+0000 7f80c4b8c740 20 client.18654 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:49.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.031+0000 7f80c4b8c740 1 --2- 172.21.15.90:0/1311040127 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5562669c03a0 0x556266e54f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.031+0000 7f80c4b8c740 4 client.18654 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:44:49.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.031+0000 7f80c4b8c740 1 -- 172.21.15.90:0/1311040127 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556266d3a800 con 0x5562669c03a0 2024-04-25T11:44:49.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.031+0000 7f80b6ffd640 1 --2- 172.21.15.90:0/1311040127 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5562669c03a0 0x556266e54f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.032+0000 7f8096ffd640 10 client.18654 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:44:49.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.032+0000 7f80b6ffd640 1 --2- 172.21.15.90:0/1311040127 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5562669c03a0 0x556266e54f00 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5c808fcc06c05a73 server_cookie=d093382ba04fe582 in_seq=0 out_seq=0 2024-04-25T11:44:49.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.032+0000 7f8096ffd640 1 -- 172.21.15.90:0/1311040127 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556266d3a800 con 0x5562669c03a0 2024-04-25T11:44:49.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.032+0000 7f8096ffd640 10 client.18654 handle_command_reply: tid=0 2024-04-25T11:44:49.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.032+0000 7f80c4b8c740 1 client.18654 shutdown 2024-04-25T11:44:49.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.032+0000 7f80c4b8c740 1 -- 172.21.15.90:0/1311040127 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a0b6b0 msgr2=0x556266ddb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.032+0000 7f80c4b8c740 1 --2- 172.21.15.90:0/1311040127 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a0b6b0 0x556266ddb9e0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f80a8010cf0 tx=0x7f80a8012da0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.032+0000 7f80c4b8c740 1 -- 172.21.15.90:0/1311040127 shutdown_connections 2024-04-25T11:44:49.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.032+0000 7f80c4b8c740 1 --2- 172.21.15.90:0/1311040127 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5562669c03a0 0x556266e54f00 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.032+0000 7f80c4b8c740 1 --2- 172.21.15.90:0/1311040127 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556266a2c390 0x556266ddbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.032+0000 7f80c4b8c740 1 --2- 172.21.15.90:0/1311040127 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556266a0b6b0 0x556266ddb9e0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.032+0000 7f8096ffd640 0 client.18654 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:44:49.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.032+0000 7f80c4b8c740 1 --2- 172.21.15.90:0/1311040127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556266a034d0 0x556266de0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.032+0000 7f80c4b8c740 1 -- 172.21.15.90:0/1311040127 >> 172.21.15.90:0/1311040127 conn(0x556266a204e0 msgr2=0x556266dd6b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:49.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.032+0000 7f80c4b8c740 1 -- 172.21.15.90:0/1311040127 shutdown_connections 2024-04-25T11:44:49.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.032+0000 7f80c4b8c740 1 -- 172.21.15.90:0/1311040127 wait complete. 2024-04-25T11:44:49.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.032+0000 7f80c4b8c740 20 client.18654 trim_cache size 0 max 16384 2024-04-25T11:44:49.025 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:44:49.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.033+0000 7f80bc99d640 1 -- 172.21.15.90:0/2875130231 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f80a00afc50 msgr2=0x7f80a00b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.033+0000 7f80bc99d640 1 --2- 172.21.15.90:0/2875130231 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f80a00afc50 0x7f80a00b2160 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f80b00dcc40 tx=0x7f80b000ff70 comp rx=0 tx=0).stop 2024-04-25T11:44:49.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.033+0000 7f80bc99d640 1 -- 172.21.15.90:0/2875130231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80b810e310 msgr2=0x7f80b819edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.033+0000 7f80bc99d640 1 --2- 172.21.15.90:0/2875130231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80b810e310 0x7f80b819edb0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f80b000cd10 tx=0x7f80b000a620 comp rx=0 tx=0).stop 2024-04-25T11:44:49.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.033+0000 7f80bc99d640 1 -- 172.21.15.90:0/2875130231 shutdown_connections 2024-04-25T11:44:49.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.033+0000 7f80bc99d640 1 --2- 172.21.15.90:0/2875130231 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f80a00afc50 0x7f80a00b2160 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.033+0000 7f80bc99d640 1 --2- 172.21.15.90:0/2875130231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80b810e310 0x7f80b819edb0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.033+0000 7f80bc99d640 1 --2- 172.21.15.90:0/2875130231 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80b81099b0 0x7f80b819aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.033+0000 7f80bc99d640 1 --2- 172.21.15.90:0/2875130231 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f80b8103a80 0x7f80b819a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.033+0000 7f80bc99d640 1 -- 172.21.15.90:0/2875130231 >> 172.21.15.90:0/2875130231 conn(0x7f80b80ff1a0 msgr2=0x7f80b80ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:49.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.033+0000 7f80bc99d640 1 -- 172.21.15.90:0/2875130231 shutdown_connections 2024-04-25T11:44:49.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.033+0000 7f80bc99d640 1 -- 172.21.15.90:0/2875130231 wait complete. 2024-04-25T11:44:49.029 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:48 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1917969616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:44:49.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T11:44:49.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.153+0000 7f09d82f2640 1 -- 172.21.15.90:0/2753238098 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09d01069b0 msgr2=0x7f09d0106de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.153+0000 7f09d82f2640 1 --2- 172.21.15.90:0/2753238098 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09d01069b0 0x7f09d0106de0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f09cc009900 tx=0x7f09cc02a410 comp rx=0 tx=0).stop 2024-04-25T11:44:49.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.153+0000 7f09d82f2640 1 -- 172.21.15.90:0/2753238098 shutdown_connections 2024-04-25T11:44:49.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.153+0000 7f09d82f2640 1 --2- 172.21.15.90:0/2753238098 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f09d0107470 0x7f09d010ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.153+0000 7f09d82f2640 1 --2- 172.21.15.90:0/2753238098 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09d01069b0 0x7f09d0106de0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.153+0000 7f09d82f2640 1 --2- 172.21.15.90:0/2753238098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09d0102420 0x7f09d0102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.153+0000 7f09d82f2640 1 -- 172.21.15.90:0/2753238098 >> 172.21.15.90:0/2753238098 conn(0x7f09d0077920 msgr2=0x7f09d0077d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:49.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.153+0000 7f09d82f2640 1 -- 172.21.15.90:0/2753238098 shutdown_connections 2024-04-25T11:44:49.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.153+0000 7f09d82f2640 1 -- 172.21.15.90:0/2753238098 wait complete. 2024-04-25T11:44:49.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.154+0000 7f09d82f2640 1 Processor -- start 2024-04-25T11:44:49.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.154+0000 7f09d82f2640 1 -- start start 2024-04-25T11:44:49.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.154+0000 7f09d82f2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09d0102420 0x7f09d019b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.154+0000 7f09d82f2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09d01069b0 0x7f09d019bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.154+0000 7f09d82f2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f09d0107470 0x7f09d019c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.154+0000 7f09d82f2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f09d0112350 con 0x7f09d01069b0 2024-04-25T11:44:49.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.154+0000 7f09d82f2640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f09d01121d0 con 0x7f09d0107470 2024-04-25T11:44:49.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.154+0000 7f09d82f2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f09d01124d0 con 0x7f09d0102420 2024-04-25T11:44:49.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.154+0000 7f09d588d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09d01069b0 0x7f09d019bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.154+0000 7f09d588d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09d01069b0 0x7f09d019bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48552/0 (socket says 172.21.15.90:48552) 2024-04-25T11:44:49.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.154+0000 7f09d588d640 1 -- 172.21.15.90:0/2745184779 learned_addr learned my addr 172.21.15.90:0/2745184779 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:49.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.155+0000 7f09d688f640 1 --2- 172.21.15.90:0/2745184779 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f09d0107470 0x7f09d019c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.155+0000 7f09d608e640 1 --2- 172.21.15.90:0/2745184779 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09d0102420 0x7f09d019b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.155+0000 7f09d588d640 1 -- 172.21.15.90:0/2745184779 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09d0102420 msgr2=0x7f09d019b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.155+0000 7f09d588d640 1 --2- 172.21.15.90:0/2745184779 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09d0102420 0x7f09d019b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.155+0000 7f09d588d640 1 -- 172.21.15.90:0/2745184779 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f09d0107470 msgr2=0x7f09d019c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.155+0000 7f09d588d640 1 --2- 172.21.15.90:0/2745184779 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f09d0107470 0x7f09d019c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.155+0000 7f09d588d640 1 -- 172.21.15.90:0/2745184779 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09d01a6880 con 0x7f09d01069b0 2024-04-25T11:44:49.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.155+0000 7f09d608e640 1 --2- 172.21.15.90:0/2745184779 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09d0102420 0x7f09d019b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:49.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.155+0000 7f09d588d640 1 --2- 172.21.15.90:0/2745184779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09d01069b0 0x7f09d019bb40 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f09cc031a90 tx=0x7f09cc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:49.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.155+0000 7f09c37fe640 1 -- 172.21.15.90:0/2745184779 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc03f030 con 0x7f09d01069b0 2024-04-25T11:44:49.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.155+0000 7f09d82f2640 1 -- 172.21.15.90:0/2745184779 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09d01a6b10 con 0x7f09d01069b0 2024-04-25T11:44:49.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.155+0000 7f09c37fe640 1 -- 172.21.15.90:0/2745184779 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09cc003cb0 con 0x7f09d01069b0 2024-04-25T11:44:49.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.155+0000 7f09c37fe640 1 -- 172.21.15.90:0/2745184779 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc035070 con 0x7f09d01069b0 2024-04-25T11:44:49.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.155+0000 7f09d82f2640 1 -- 172.21.15.90:0/2745184779 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09d01a7000 con 0x7f09d01069b0 2024-04-25T11:44:49.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.157+0000 7f09c37fe640 1 -- 172.21.15.90:0/2745184779 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f09cc042030 con 0x7f09d01069b0 2024-04-25T11:44:49.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.158+0000 7f09df1a1740 1 Processor -- start 2024-04-25T11:44:49.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.158+0000 7f09df1a1740 1 -- start start 2024-04-25T11:44:49.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.158+0000 7f09df1a1740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561742e8a390 0x56174322b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.158+0000 7f09df1a1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561742e614d0 0x56174322bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.158+0000 7f09df1a1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e696b0 0x56174323cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.158+0000 7f09df1a1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561743190d80 con 0x561742e696b0 2024-04-25T11:44:49.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.158+0000 7f09df1a1740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56174314aa20 con 0x561742e8a390 2024-04-25T11:44:49.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.158+0000 7f09df1a1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56174318cd40 con 0x561742e614d0 2024-04-25T11:44:49.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.158+0000 7f09c37fe640 1 --2- 172.21.15.90:0/2745184779 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f09bc0afc60 0x7f09bc0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.159+0000 7f09d588d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e696b0 0x56174323cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.159+0000 7f09d688f640 1 --2- 172.21.15.90:0/2745184779 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f09bc0afc60 0x7f09bc0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.159+0000 7f09d588d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e696b0 0x56174323cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48564/0 (socket says 172.21.15.90:48564) 2024-04-25T11:44:49.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.159+0000 7f09d588d640 1 -- 172.21.15.90:0/2361405935 learned_addr learned my addr 172.21.15.90:0/2361405935 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:49.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.159+0000 7f09d688f640 1 --2- 172.21.15.90:0/2361405935 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561742e8a390 0x56174322b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.159+0000 7f09d608e640 1 --2- 172.21.15.90:0/2361405935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561742e614d0 0x56174322bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.159+0000 7f09c37fe640 1 -- 172.21.15.90:0/2745184779 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f09cc0ff770 con 0x7f09d01069b0 2024-04-25T11:44:49.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.159+0000 7f09d588d640 1 -- 172.21.15.90:0/2361405935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561742e614d0 msgr2=0x56174322bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.159+0000 7f09d588d640 1 --2- 172.21.15.90:0/2361405935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561742e614d0 0x56174322bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.159+0000 7f09d588d640 1 -- 172.21.15.90:0/2361405935 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561742e8a390 msgr2=0x56174322b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.159+0000 7f09d588d640 1 --2- 172.21.15.90:0/2361405935 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561742e8a390 0x56174322b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.159+0000 7f09d588d640 1 -- 172.21.15.90:0/2361405935 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561743155c30 con 0x561742e696b0 2024-04-25T11:44:49.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.159+0000 7f09d688f640 1 --2- 172.21.15.90:0/2361405935 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561742e8a390 0x56174322b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:49.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.159+0000 7f09d588d640 1 --2- 172.21.15.90:0/2361405935 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e696b0 0x56174323cab0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f09cc003fd0 tx=0x7f09cc096380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bdca73277599784 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:49.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.159+0000 7f09d688f640 1 --2- 172.21.15.90:0/2745184779 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f09bc0afc60 0x7f09bc0b2170 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x5617432289b0 tx=0x7f09c8005c40 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:49.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.159+0000 7f09c0bfe640 1 -- 172.21.15.90:0/2361405935 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc049070 con 0x561742e696b0 2024-04-25T11:44:49.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.159+0000 7f09c0bfe640 1 -- 172.21.15.90:0/2361405935 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09cc0c7390 con 0x561742e696b0 2024-04-25T11:44:49.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.159+0000 7f09c0bfe640 1 -- 172.21.15.90:0/2361405935 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc0d06e0 con 0x561742e696b0 2024-04-25T11:44:49.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.160+0000 7f09df1a1740 1 -- 172.21.15.90:0/2361405935 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e696b0 msgr2=0x56174323cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.160+0000 7f09df1a1740 1 --2- 172.21.15.90:0/2361405935 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e696b0 0x56174323cab0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f09cc003fd0 tx=0x7f09cc096380 comp rx=0 tx=0).stop 2024-04-25T11:44:49.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.160+0000 7f09df1a1740 1 -- 172.21.15.90:0/2361405935 shutdown_connections 2024-04-25T11:44:49.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.160+0000 7f09df1a1740 1 --2- 172.21.15.90:0/2361405935 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e696b0 0x56174323cab0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.160+0000 7f09df1a1740 1 --2- 172.21.15.90:0/2361405935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561742e614d0 0x56174322bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.160+0000 7f09df1a1740 1 --2- 172.21.15.90:0/2361405935 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561742e8a390 0x56174322b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.160+0000 7f09df1a1740 1 -- 172.21.15.90:0/2361405935 >> 172.21.15.90:0/2361405935 conn(0x561742e7e4e0 msgr2=0x561743227fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:49.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.160+0000 7f09df1a1740 1 -- 172.21.15.90:0/2361405935 shutdown_connections 2024-04-25T11:44:49.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.160+0000 7f09df1a1740 1 -- 172.21.15.90:0/2361405935 wait complete. 2024-04-25T11:44:49.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.161+0000 7f09df1a1740 1 -- start start 2024-04-25T11:44:49.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.161+0000 7f09df1a1740 1 Processor -- start 2024-04-25T11:44:49.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.161+0000 7f09df1a1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561742e614d0 0x56174323a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.161+0000 7f09df1a1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e696b0 0x56174323ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.161+0000 7f09df1a1740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561742e8a390 0x561743246420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.161+0000 7f09df1a1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56174323cff0 con 0x561742e696b0 2024-04-25T11:44:49.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.161+0000 7f09df1a1740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x561743117dc0 con 0x561742e8a390 2024-04-25T11:44:49.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.161+0000 7f09df1a1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56174323d170 con 0x561742e614d0 2024-04-25T11:44:49.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.161+0000 7f09df1a1740 10 client.0 _ll_register_callbacks cb 0x561743133600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:49.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.161+0000 7f09d588d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e696b0 0x56174323ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.161+0000 7f09d588d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e696b0 0x56174323ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48572/0 (socket says 172.21.15.90:48572) 2024-04-25T11:44:49.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.161+0000 7f09d588d640 1 -- 172.21.15.90:0/2179392806 learned_addr learned my addr 172.21.15.90:0/2179392806 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:49.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.161+0000 7f09d588d640 1 -- 172.21.15.90:0/2179392806 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561742e614d0 msgr2=0x56174323a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.161+0000 7f09d688f640 1 --2- 172.21.15.90:0/2179392806 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561742e8a390 0x561743246420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.161+0000 7f09d588d640 1 --2- 172.21.15.90:0/2179392806 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561742e614d0 0x56174323a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.161+0000 7f09d588d640 1 -- 172.21.15.90:0/2179392806 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561742e8a390 msgr2=0x561743246420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.161+0000 7f09d588d640 1 --2- 172.21.15.90:0/2179392806 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561742e8a390 0x561743246420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.161+0000 7f09d588d640 1 -- 172.21.15.90:0/2179392806 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561742ea23f0 con 0x561742e696b0 2024-04-25T11:44:49.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.161+0000 7f09d688f640 1 --2- 172.21.15.90:0/2179392806 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561742e8a390 0x561743246420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:49.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.162+0000 7f09d588d640 1 --2- 172.21.15.90:0/2179392806 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e696b0 0x56174323ab10 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f09cc0ffb20 tx=0x7f09cc03b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e41fea38462ad0b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:49.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.162+0000 7f09b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:49.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.162+0000 7f09b9ffb640 1 -- 172.21.15.90:0/2179392806 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc049070 con 0x561742e696b0 2024-04-25T11:44:49.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.162+0000 7f09b9ffb640 1 -- 172.21.15.90:0/2179392806 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f09cc0c3b10 con 0x561742e696b0 2024-04-25T11:44:49.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.162+0000 7f09df1a1740 1 -- 172.21.15.90:0/2179392806 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09cc0c7390 con 0x561742e696b0 2024-04-25T11:44:49.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.162+0000 7f09b9ffb640 1 -- 172.21.15.90:0/2179392806 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f09cc0d06e0 con 0x561742e696b0 2024-04-25T11:44:49.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.162+0000 7f09df1a1740 10 client.18666 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:49.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.162+0000 7f09df1a1740 1 -- 172.21.15.90:0/2179392806 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561742f6c7d0 con 0x561742e696b0 2024-04-25T11:44:49.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.162+0000 7f09b9ffb640 1 -- 172.21.15.90:0/2179392806 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09cc0c7390 con 0x561742e696b0 2024-04-25T11:44:49.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.162+0000 7f09b9ffb640 1 -- 172.21.15.90:0/2179392806 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f09cc03bce0 con 0x561742e696b0 2024-04-25T11:44:49.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.162+0000 7f09df1a1740 10 client.18666 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:49.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.162+0000 7f09df1a1740 10 client.18666 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:44:49.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.162+0000 7f09df1a1740 20 client.18666 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:49.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.162+0000 7f09df1a1740 1 --2- 172.21.15.90:0/2179392806 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x561742e48f60 0x56174324c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.162+0000 7f09df1a1740 4 client.18666 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:44:49.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.162+0000 7f09df1a1740 1 -- 172.21.15.90:0/2179392806 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56174319e990 con 0x561742e48f60 2024-04-25T11:44:49.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.163+0000 7f09d608e640 1 --2- 172.21.15.90:0/2179392806 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x561742e48f60 0x56174324c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.163+0000 7f09b9ffb640 10 client.18666 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:44:49.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.163+0000 7f09d608e640 1 --2- 172.21.15.90:0/2179392806 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x561742e48f60 0x56174324c650 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bedc84e083c01b5c server_cookie=1b9b97d099ad88e in_seq=0 out_seq=0 2024-04-25T11:44:49.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.164+0000 7f09b9ffb640 1 -- 172.21.15.90:0/2179392806 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56174319e990 con 0x561742e48f60 2024-04-25T11:44:49.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.164+0000 7f09b9ffb640 10 client.18666 handle_command_reply: tid=0 2024-04-25T11:44:49.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.164+0000 7f09df1a1740 1 client.18666 shutdown 2024-04-25T11:44:49.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.164+0000 7f09df1a1740 1 -- 172.21.15.90:0/2179392806 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e696b0 msgr2=0x56174323ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.164+0000 7f09df1a1740 1 --2- 172.21.15.90:0/2179392806 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e696b0 0x56174323ab10 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f09cc0ffb20 tx=0x7f09cc03b720 comp rx=0 tx=0).stop 2024-04-25T11:44:49.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.164+0000 7f09df1a1740 1 -- 172.21.15.90:0/2179392806 shutdown_connections 2024-04-25T11:44:49.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.164+0000 7f09df1a1740 1 --2- 172.21.15.90:0/2179392806 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x561742e48f60 0x56174324c650 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.164+0000 7f09df1a1740 1 --2- 172.21.15.90:0/2179392806 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561742e8a390 0x561743246420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.164+0000 7f09df1a1740 1 --2- 172.21.15.90:0/2179392806 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e696b0 0x56174323ab10 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.164+0000 7f09df1a1740 1 --2- 172.21.15.90:0/2179392806 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561742e614d0 0x56174323a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.164+0000 7f09b9ffb640 0 client.18666 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:44:49.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.164+0000 7f09df1a1740 1 -- 172.21.15.90:0/2179392806 >> 172.21.15.90:0/2179392806 conn(0x561742e7e4e0 msgr2=0x56174322d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:49.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.164+0000 7f09df1a1740 1 -- 172.21.15.90:0/2179392806 shutdown_connections 2024-04-25T11:44:49.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.164+0000 7f09df1a1740 1 -- 172.21.15.90:0/2179392806 wait complete. 2024-04-25T11:44:49.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.164+0000 7f09df1a1740 20 client.18666 trim_cache size 0 max 16384 2024-04-25T11:44:49.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.178+0000 7f09df1a1740 1 Processor -- start 2024-04-25T11:44:49.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.178+0000 7f09df1a1740 1 -- start start 2024-04-25T11:44:49.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.178+0000 7f09df1a1740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561742e614d0 0x56174323a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.178+0000 7f09df1a1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561742e696b0 0x561743226d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.178+0000 7f09df1a1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e8a390 0x561743246040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.178+0000 7f09df1a1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f09d01ada60 con 0x561742e8a390 2024-04-25T11:44:49.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.178+0000 7f09df1a1740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f09d01af5a0 con 0x561742e614d0 2024-04-25T11:44:49.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.178+0000 7f09df1a1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f09d01ae130 con 0x561742e696b0 2024-04-25T11:44:49.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.178+0000 7f09d688f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e8a390 0x561743246040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.178+0000 7f09d688f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e8a390 0x561743246040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48576/0 (socket says 172.21.15.90:48576) 2024-04-25T11:44:49.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.178+0000 7f09d688f640 1 -- 172.21.15.90:0/1790004096 learned_addr learned my addr 172.21.15.90:0/1790004096 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:49.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.178+0000 7f09d688f640 1 -- 172.21.15.90:0/1790004096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561742e696b0 msgr2=0x561743226d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.178+0000 7f09d588d640 1 --2- 172.21.15.90:0/1790004096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561742e696b0 0x561743226d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.178+0000 7f09d688f640 1 --2- 172.21.15.90:0/1790004096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561742e696b0 0x561743226d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.178+0000 7f09d688f640 1 -- 172.21.15.90:0/1790004096 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561742e614d0 msgr2=0x56174323a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.178+0000 7f09d608e640 1 --2- 172.21.15.90:0/1790004096 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561742e614d0 0x56174323a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.178+0000 7f09d688f640 1 --2- 172.21.15.90:0/1790004096 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561742e614d0 0x56174323a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.178+0000 7f09d688f640 1 -- 172.21.15.90:0/1790004096 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56174322d020 con 0x561742e8a390 2024-04-25T11:44:49.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.178+0000 7f09d588d640 1 --2- 172.21.15.90:0/1790004096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561742e696b0 0x561743226d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:49.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.178+0000 7f09d608e640 1 --2- 172.21.15.90:0/1790004096 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561742e614d0 0x56174323a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:49.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.178+0000 7f09d688f640 1 --2- 172.21.15.90:0/1790004096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e8a390 0x561743246040 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f09c8015010 tx=0x7f09c8015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a46213efb4935d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:49.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.179+0000 7f09bbfff640 1 -- 172.21.15.90:0/1790004096 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c800adc0 con 0x561742e8a390 2024-04-25T11:44:49.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.179+0000 7f09bbfff640 1 -- 172.21.15.90:0/1790004096 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09c800f9d0 con 0x561742e8a390 2024-04-25T11:44:49.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.179+0000 7f09bbfff640 1 -- 172.21.15.90:0/1790004096 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c8005550 con 0x561742e8a390 2024-04-25T11:44:49.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.179+0000 7f09df1a1740 1 -- 172.21.15.90:0/1790004096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e8a390 msgr2=0x561743246040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.179+0000 7f09df1a1740 1 --2- 172.21.15.90:0/1790004096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e8a390 0x561743246040 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f09c8015010 tx=0x7f09c8015040 comp rx=0 tx=0).stop 2024-04-25T11:44:49.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.179+0000 7f09df1a1740 1 -- 172.21.15.90:0/1790004096 shutdown_connections 2024-04-25T11:44:49.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.179+0000 7f09df1a1740 1 --2- 172.21.15.90:0/1790004096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e8a390 0x561743246040 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.179+0000 7f09df1a1740 1 --2- 172.21.15.90:0/1790004096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561742e696b0 0x561743226d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.179+0000 7f09df1a1740 1 --2- 172.21.15.90:0/1790004096 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561742e614d0 0x56174323a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.179+0000 7f09df1a1740 1 -- 172.21.15.90:0/1790004096 >> 172.21.15.90:0/1790004096 conn(0x561742e7e4e0 msgr2=0x561743226610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:49.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.179+0000 7f09df1a1740 1 -- 172.21.15.90:0/1790004096 shutdown_connections 2024-04-25T11:44:49.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.179+0000 7f09df1a1740 1 -- 172.21.15.90:0/1790004096 wait complete. 2024-04-25T11:44:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.180+0000 7f09df1a1740 1 -- start start 2024-04-25T11:44:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.180+0000 7f09df1a1740 1 Processor -- start 2024-04-25T11:44:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.180+0000 7f09df1a1740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561742e614d0 0x56174323d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.180+0000 7f09df1a1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e696b0 0x56174323dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.180+0000 7f09df1a1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561742e8a390 0x5617432b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.180+0000 7f09df1a1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f09d01aeed0 con 0x561742e696b0 2024-04-25T11:44:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.180+0000 7f09df1a1740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f09d01ae800 con 0x561742e614d0 2024-04-25T11:44:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.180+0000 7f09df1a1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f09d01aba60 con 0x561742e8a390 2024-04-25T11:44:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.180+0000 7f09df1a1740 10 client.0 _ll_register_callbacks cb 0x7f09c800dc80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.180+0000 7f09d588d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e696b0 0x56174323dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.180+0000 7f09d588d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e696b0 0x56174323dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48588/0 (socket says 172.21.15.90:48588) 2024-04-25T11:44:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.180+0000 7f09d588d640 1 -- 172.21.15.90:0/380745736 learned_addr learned my addr 172.21.15.90:0/380745736 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.180+0000 7f09d608e640 1 --2- 172.21.15.90:0/380745736 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561742e614d0 0x56174323d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.180+0000 7f09d688f640 1 --2- 172.21.15.90:0/380745736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561742e8a390 0x5617432b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.180+0000 7f09d588d640 1 -- 172.21.15.90:0/380745736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561742e8a390 msgr2=0x5617432b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.180+0000 7f09d588d640 1 --2- 172.21.15.90:0/380745736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561742e8a390 0x5617432b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.180+0000 7f09d588d640 1 -- 172.21.15.90:0/380745736 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561742e614d0 msgr2=0x56174323d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.180+0000 7f09d588d640 1 --2- 172.21.15.90:0/380745736 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561742e614d0 0x56174323d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.180+0000 7f09d588d640 1 -- 172.21.15.90:0/380745736 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617432b2e50 con 0x561742e696b0 2024-04-25T11:44:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.180+0000 7f09d688f640 1 --2- 172.21.15.90:0/380745736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561742e8a390 0x5617432b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.181+0000 7f09d588d640 1 --2- 172.21.15.90:0/380745736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e696b0 0x56174323dc30 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f09cc005ef0 tx=0x7f09cc0d02c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcec5c2c0f813ff0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.181+0000 7f09b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.181+0000 7f09b9ffb640 1 -- 172.21.15.90:0/380745736 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc049070 con 0x561742e696b0 2024-04-25T11:44:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.181+0000 7f09b9ffb640 1 -- 172.21.15.90:0/380745736 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f09cc0c3b10 con 0x561742e696b0 2024-04-25T11:44:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.181+0000 7f09df1a1740 1 -- 172.21.15.90:0/380745736 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09c800f9d0 con 0x561742e696b0 2024-04-25T11:44:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.181+0000 7f09b9ffb640 1 -- 172.21.15.90:0/380745736 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f09cc0e3030 con 0x561742e696b0 2024-04-25T11:44:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.181+0000 7f09df1a1740 10 client.18672 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:49.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.181+0000 7f09df1a1740 1 -- 172.21.15.90:0/380745736 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5617432b2e50 con 0x561742e696b0 2024-04-25T11:44:49.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.181+0000 7f09b9ffb640 1 -- 172.21.15.90:0/380745736 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09c800f9d0 con 0x561742e696b0 2024-04-25T11:44:49.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.181+0000 7f09b9ffb640 1 -- 172.21.15.90:0/380745736 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f09cc0b0110 con 0x561742e696b0 2024-04-25T11:44:49.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.181+0000 7f09df1a1740 10 client.18672 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:49.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.181+0000 7f09df1a1740 10 client.18672 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:44:49.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.181+0000 7f09df1a1740 20 client.18672 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:49.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.181+0000 7f09df1a1740 1 --2- 172.21.15.90:0/380745736 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x561742e1e3a0 0x5617432b63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.181+0000 7f09df1a1740 4 client.18672 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:44:49.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.181+0000 7f09df1a1740 1 -- 172.21.15.90:0/380745736 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561743198800 con 0x561742e1e3a0 2024-04-25T11:44:49.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.182+0000 7f09d608e640 1 --2- 172.21.15.90:0/380745736 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x561742e1e3a0 0x5617432b63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.182+0000 7f09b9ffb640 10 client.18672 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:44:49.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.182+0000 7f09d608e640 1 --2- 172.21.15.90:0/380745736 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x561742e1e3a0 0x5617432b63c0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b1c4f059a9ee09e8 server_cookie=a912d2560115897a in_seq=0 out_seq=0 2024-04-25T11:44:49.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.182+0000 7f09b9ffb640 1 -- 172.21.15.90:0/380745736 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561743198800 con 0x561742e1e3a0 2024-04-25T11:44:49.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.182+0000 7f09b9ffb640 10 client.18672 handle_command_reply: tid=0 2024-04-25T11:44:49.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.182+0000 7f09df1a1740 1 client.18672 shutdown 2024-04-25T11:44:49.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.182+0000 7f09df1a1740 1 -- 172.21.15.90:0/380745736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e696b0 msgr2=0x56174323dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.182+0000 7f09df1a1740 1 --2- 172.21.15.90:0/380745736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e696b0 0x56174323dc30 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f09cc005ef0 tx=0x7f09cc0d02c0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.182+0000 7f09df1a1740 1 -- 172.21.15.90:0/380745736 shutdown_connections 2024-04-25T11:44:49.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.182+0000 7f09df1a1740 1 --2- 172.21.15.90:0/380745736 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x561742e1e3a0 0x5617432b63c0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.182+0000 7f09df1a1740 1 --2- 172.21.15.90:0/380745736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561742e8a390 0x5617432b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.182+0000 7f09df1a1740 1 --2- 172.21.15.90:0/380745736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561742e696b0 0x56174323dc30 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.182+0000 7f09b9ffb640 0 client.18672 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:44:49.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.182+0000 7f09df1a1740 1 --2- 172.21.15.90:0/380745736 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561742e614d0 0x56174323d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.182+0000 7f09df1a1740 1 -- 172.21.15.90:0/380745736 >> 172.21.15.90:0/380745736 conn(0x561742e7e4e0 msgr2=0x561743226da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:49.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.182+0000 7f09df1a1740 1 -- 172.21.15.90:0/380745736 shutdown_connections 2024-04-25T11:44:49.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.182+0000 7f09df1a1740 1 -- 172.21.15.90:0/380745736 wait complete. 2024-04-25T11:44:49.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.182+0000 7f09df1a1740 20 client.18672 trim_cache size 0 max 16384 2024-04-25T11:44:49.178 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:44:49.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.183+0000 7f09d82f2640 1 -- 172.21.15.90:0/2745184779 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f09bc0afc60 msgr2=0x7f09bc0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.183+0000 7f09d82f2640 1 --2- 172.21.15.90:0/2745184779 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f09bc0afc60 0x7f09bc0b2170 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x5617432289b0 tx=0x7f09c8005c40 comp rx=0 tx=0).stop 2024-04-25T11:44:49.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.183+0000 7f09d82f2640 1 -- 172.21.15.90:0/2745184779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09d01069b0 msgr2=0x7f09d019bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.183+0000 7f09d82f2640 1 --2- 172.21.15.90:0/2745184779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09d01069b0 0x7f09d019bb40 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f09cc031a90 tx=0x7f09cc004270 comp rx=0 tx=0).stop 2024-04-25T11:44:49.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.183+0000 7f09d82f2640 1 -- 172.21.15.90:0/2745184779 shutdown_connections 2024-04-25T11:44:49.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.183+0000 7f09d82f2640 1 --2- 172.21.15.90:0/2745184779 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f09bc0afc60 0x7f09bc0b2170 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.183+0000 7f09d82f2640 1 --2- 172.21.15.90:0/2745184779 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f09d0107470 0x7f09d019c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.183+0000 7f09d82f2640 1 --2- 172.21.15.90:0/2745184779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09d01069b0 0x7f09d019bb40 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.183+0000 7f09d82f2640 1 --2- 172.21.15.90:0/2745184779 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09d0102420 0x7f09d019b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.183+0000 7f09d82f2640 1 -- 172.21.15.90:0/2745184779 >> 172.21.15.90:0/2745184779 conn(0x7f09d0077920 msgr2=0x7f09d01038f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:49.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.183+0000 7f09d82f2640 1 -- 172.21.15.90:0/2745184779 shutdown_connections 2024-04-25T11:44:49.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.183+0000 7f09d82f2640 1 -- 172.21.15.90:0/2745184779 wait complete. 2024-04-25T11:44:49.181 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T11:44:49.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.305+0000 7f95a6e34640 1 -- 172.21.15.90:0/1889848004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95a01032a0 msgr2=0x7f95a01036b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.305+0000 7f95a6e34640 1 --2- 172.21.15.90:0/1889848004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95a01032a0 0x7f95a01036b0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f9594009900 tx=0x7f959402a410 comp rx=0 tx=0).stop 2024-04-25T11:44:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.305+0000 7f95a6e34640 1 -- 172.21.15.90:0/1889848004 shutdown_connections 2024-04-25T11:44:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.305+0000 7f95a6e34640 1 --2- 172.21.15.90:0/1889848004 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95a010d420 0x7f95a010f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.305+0000 7f95a6e34640 1 --2- 172.21.15.90:0/1889848004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95a01032a0 0x7f95a01036b0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.305+0000 7f95a6e34640 1 --2- 172.21.15.90:0/1889848004 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f95a0107930 0x7f95a0102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.305+0000 7f95a6e34640 1 -- 172.21.15.90:0/1889848004 >> 172.21.15.90:0/1889848004 conn(0x7f95a0077880 msgr2=0x7f95a0077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.305+0000 7f95a6e34640 1 -- 172.21.15.90:0/1889848004 shutdown_connections 2024-04-25T11:44:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.305+0000 7f95a6e34640 1 -- 172.21.15.90:0/1889848004 wait complete. 2024-04-25T11:44:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.305+0000 7f95a6e34640 1 Processor -- start 2024-04-25T11:44:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.305+0000 7f95a6e34640 1 -- start start 2024-04-25T11:44:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.306+0000 7f95a6e34640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95a01032a0 0x7f95a019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.306+0000 7f95a6e34640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95a0107930 0x7f95a019acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.306+0000 7f95a6e34640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f95a010d420 0x7f95a019ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.306+0000 7f95a6e34640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f95a0112be0 con 0x7f95a01032a0 2024-04-25T11:44:49.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.306+0000 7f95a6e34640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f95a0112a60 con 0x7f95a010d420 2024-04-25T11:44:49.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.306+0000 7f95a6e34640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f95a0112d60 con 0x7f95a0107930 2024-04-25T11:44:49.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.306+0000 7f95a4bd0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95a01032a0 0x7f95a019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.306+0000 7f95a4bd0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95a01032a0 0x7f95a019a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48600/0 (socket says 172.21.15.90:48600) 2024-04-25T11:44:49.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.306+0000 7f95a4bd0640 1 -- 172.21.15.90:0/3640509801 learned_addr learned my addr 172.21.15.90:0/3640509801 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:49.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.306+0000 7f959ffff640 1 --2- 172.21.15.90:0/3640509801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95a0107930 0x7f95a019acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.306+0000 7f95a53d1640 1 --2- 172.21.15.90:0/3640509801 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f95a010d420 0x7f95a019ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.306+0000 7f95a4bd0640 1 -- 172.21.15.90:0/3640509801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95a0107930 msgr2=0x7f95a019acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.306+0000 7f95a4bd0640 1 --2- 172.21.15.90:0/3640509801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95a0107930 0x7f95a019acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.306+0000 7f95a4bd0640 1 -- 172.21.15.90:0/3640509801 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f95a010d420 msgr2=0x7f95a019ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.306+0000 7f95a4bd0640 1 --2- 172.21.15.90:0/3640509801 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f95a010d420 0x7f95a019ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.306+0000 7f95a4bd0640 1 -- 172.21.15.90:0/3640509801 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95a019f530 con 0x7f95a01032a0 2024-04-25T11:44:49.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.306+0000 7f95a53d1640 1 --2- 172.21.15.90:0/3640509801 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f95a010d420 0x7f95a019ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:49.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.306+0000 7f95a4bd0640 1 --2- 172.21.15.90:0/3640509801 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95a01032a0 0x7f95a019a750 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f959000c950 tx=0x7f959000ce10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:49.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.306+0000 7f959dffb640 1 -- 172.21.15.90:0/3640509801 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9590007c70 con 0x7f95a01032a0 2024-04-25T11:44:49.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.306+0000 7f959dffb640 1 -- 172.21.15.90:0/3640509801 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9590007e10 con 0x7f95a01032a0 2024-04-25T11:44:49.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.306+0000 7f959dffb640 1 -- 172.21.15.90:0/3640509801 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95900187f0 con 0x7f95a01032a0 2024-04-25T11:44:49.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.306+0000 7f95a6e34640 1 -- 172.21.15.90:0/3640509801 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95a019f820 con 0x7f95a01032a0 2024-04-25T11:44:49.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.307+0000 7f95a6e34640 1 -- 172.21.15.90:0/3640509801 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95a019fd40 con 0x7f95a01032a0 2024-04-25T11:44:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.309+0000 7f959dffb640 1 -- 172.21.15.90:0/3640509801 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f9590018990 con 0x7f95a01032a0 2024-04-25T11:44:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.309+0000 7f95addbe740 1 Processor -- start 2024-04-25T11:44:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.309+0000 7f95addbe740 1 -- start start 2024-04-25T11:44:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.310+0000 7f959dffb640 1 --2- 172.21.15.90:0/3640509801 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f95880afc60 0x7f95880b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.310+0000 7f95addbe740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48fa390 0x55fdd4c9b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.310+0000 7f95addbe740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdd48d14d0 0x55fdd4c9bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.310+0000 7f95addbe740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fdd48d96b0 0x55fdd4cacab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.310+0000 7f95addbe740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55fdd4c00d80 con 0x55fdd48fa390 2024-04-25T11:44:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.310+0000 7f95addbe740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55fdd4bbaa20 con 0x55fdd48d96b0 2024-04-25T11:44:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.310+0000 7f95addbe740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fdd4bfcd40 con 0x55fdd48d14d0 2024-04-25T11:44:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.310+0000 7f95a53d1640 1 --2- 172.21.15.90:0/3640509801 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f95880afc60 0x7f95880b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.310+0000 7f95a4bd0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48fa390 0x55fdd4c9b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.310+0000 7f95a4bd0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48fa390 0x55fdd4c9b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48614/0 (socket says 172.21.15.90:48614) 2024-04-25T11:44:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.310+0000 7f95a4bd0640 1 -- 172.21.15.90:0/2421733435 learned_addr learned my addr 172.21.15.90:0/2421733435 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.310+0000 7f959dffb640 1 -- 172.21.15.90:0/3640509801 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f95900df170 con 0x7f95a01032a0 2024-04-25T11:44:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.310+0000 7f959ffff640 1 --2- 172.21.15.90:0/2421733435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdd48d14d0 0x55fdd4c9bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.310+0000 7f95a53d1640 1 --2- 172.21.15.90:0/2421733435 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fdd48d96b0 0x55fdd4cacab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.310+0000 7f95a4bd0640 1 -- 172.21.15.90:0/2421733435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdd48d14d0 msgr2=0x55fdd4c9bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.310+0000 7f95a4bd0640 1 --2- 172.21.15.90:0/2421733435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdd48d14d0 0x55fdd4c9bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.310+0000 7f95a4bd0640 1 -- 172.21.15.90:0/2421733435 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fdd48d96b0 msgr2=0x55fdd4cacab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.310+0000 7f95a4bd0640 1 --2- 172.21.15.90:0/2421733435 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fdd48d96b0 0x55fdd4cacab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.310+0000 7f95a4bd0640 1 -- 172.21.15.90:0/2421733435 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdd4bc5c30 con 0x55fdd48fa390 2024-04-25T11:44:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.310+0000 7f95a53d1640 1 --2- 172.21.15.90:0/3640509801 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f95880afc60 0x7f95880b2170 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9598009270 tx=0x7f9598013a80 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.310+0000 7f95a4bd0640 1 --2- 172.21.15.90:0/2421733435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48fa390 0x55fdd4c9b6a0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f9590007590 tx=0x7f9590020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c30d30dc21ea19f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.310+0000 7f9586ffd640 1 -- 172.21.15.90:0/2421733435 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95900d56e0 con 0x55fdd48fa390 2024-04-25T11:44:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.311+0000 7f9586ffd640 1 -- 172.21.15.90:0/2421733435 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95900a6e30 con 0x55fdd48fa390 2024-04-25T11:44:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.311+0000 7f9586ffd640 1 -- 172.21.15.90:0/2421733435 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95900cb0b0 con 0x55fdd48fa390 2024-04-25T11:44:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.311+0000 7f95addbe740 1 -- 172.21.15.90:0/2421733435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48fa390 msgr2=0x55fdd4c9b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.311+0000 7f95addbe740 1 --2- 172.21.15.90:0/2421733435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48fa390 0x55fdd4c9b6a0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f9590007590 tx=0x7f9590020000 comp rx=0 tx=0).stop 2024-04-25T11:44:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.311+0000 7f95addbe740 1 -- 172.21.15.90:0/2421733435 shutdown_connections 2024-04-25T11:44:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.311+0000 7f95addbe740 1 --2- 172.21.15.90:0/2421733435 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fdd48d96b0 0x55fdd4cacab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.311+0000 7f95addbe740 1 --2- 172.21.15.90:0/2421733435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdd48d14d0 0x55fdd4c9bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.311+0000 7f95addbe740 1 --2- 172.21.15.90:0/2421733435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48fa390 0x55fdd4c9b6a0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.311+0000 7f95addbe740 1 -- 172.21.15.90:0/2421733435 >> 172.21.15.90:0/2421733435 conn(0x55fdd48ee4e0 msgr2=0x55fdd4c97fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.311+0000 7f95addbe740 1 -- 172.21.15.90:0/2421733435 shutdown_connections 2024-04-25T11:44:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.311+0000 7f95addbe740 1 -- 172.21.15.90:0/2421733435 wait complete. 2024-04-25T11:44:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.312+0000 7f95addbe740 1 -- start start 2024-04-25T11:44:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.312+0000 7f95addbe740 1 Processor -- start 2024-04-25T11:44:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.312+0000 7f95addbe740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48d14d0 0x55fdd4caa670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.312+0000 7f95addbe740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fdd48d96b0 0x55fdd4caabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.312+0000 7f95addbe740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdd48fa390 0x55fdd4cb66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.312+0000 7f95addbe740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55fdd4cad1a0 con 0x55fdd48d14d0 2024-04-25T11:44:49.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.312+0000 7f95addbe740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55fdd4b87dc0 con 0x55fdd48d96b0 2024-04-25T11:44:49.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.312+0000 7f95addbe740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fdd4cad320 con 0x55fdd48fa390 2024-04-25T11:44:49.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.312+0000 7f95addbe740 10 client.0 _ll_register_callbacks cb 0x55fdd4ba3600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:49.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.312+0000 7f95a4bd0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48d14d0 0x55fdd4caa670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.312+0000 7f95a4bd0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48d14d0 0x55fdd4caa670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48624/0 (socket says 172.21.15.90:48624) 2024-04-25T11:44:49.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.312+0000 7f95a4bd0640 1 -- 172.21.15.90:0/2543184173 learned_addr learned my addr 172.21.15.90:0/2543184173 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:49.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.313+0000 7f959ffff640 1 --2- 172.21.15.90:0/2543184173 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fdd48d96b0 0x55fdd4caabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.313+0000 7f95a4bd0640 1 -- 172.21.15.90:0/2543184173 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdd48fa390 msgr2=0x55fdd4cb66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.313+0000 7f95a4bd0640 1 --2- 172.21.15.90:0/2543184173 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdd48fa390 0x55fdd4cb66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.313+0000 7f95a4bd0640 1 -- 172.21.15.90:0/2543184173 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fdd48d96b0 msgr2=0x55fdd4caabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.313+0000 7f95a4bd0640 1 --2- 172.21.15.90:0/2543184173 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fdd48d96b0 0x55fdd4caabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.313+0000 7f95a4bd0640 1 -- 172.21.15.90:0/2543184173 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdd49123f0 con 0x55fdd48d14d0 2024-04-25T11:44:49.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.313+0000 7f959ffff640 1 --2- 172.21.15.90:0/2543184173 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fdd48d96b0 0x55fdd4caabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.313+0000 7f95a4bd0640 1 --2- 172.21.15.90:0/2543184173 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48d14d0 0x55fdd4caa670 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f95900dc980 tx=0x7f95900a3860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=869f5f7677ed9163 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.313+0000 7f957ffff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.313+0000 7f957ffff640 1 -- 172.21.15.90:0/2543184173 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95900a7320 con 0x55fdd48d14d0 2024-04-25T11:44:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.313+0000 7f957ffff640 1 -- 172.21.15.90:0/2543184173 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f95900a7580 con 0x55fdd48d14d0 2024-04-25T11:44:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.313+0000 7f95addbe740 1 -- 172.21.15.90:0/2543184173 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95900a6e30 con 0x55fdd48d14d0 2024-04-25T11:44:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.313+0000 7f957ffff640 1 -- 172.21.15.90:0/2543184173 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f95900ab6d0 con 0x55fdd48d14d0 2024-04-25T11:44:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.313+0000 7f95addbe740 10 client.18684 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.313+0000 7f95addbe740 1 -- 172.21.15.90:0/2543184173 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fdd49dc7d0 con 0x55fdd48d14d0 2024-04-25T11:44:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.313+0000 7f957ffff640 1 -- 172.21.15.90:0/2543184173 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95900a6e30 con 0x55fdd48d14d0 2024-04-25T11:44:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.314+0000 7f957ffff640 1 -- 172.21.15.90:0/2543184173 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f9590067ce0 con 0x55fdd48d14d0 2024-04-25T11:44:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.314+0000 7f95addbe740 10 client.18684 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.314+0000 7f95addbe740 10 client.18684 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:44:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.314+0000 7f95addbe740 20 client.18684 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.314+0000 7f95addbe740 1 --2- 172.21.15.90:0/2543184173 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55fdd48b8f60 0x55fdd4cbc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.314+0000 7f95addbe740 4 client.18684 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:44:49.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.314+0000 7f95addbe740 1 -- 172.21.15.90:0/2543184173 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fdd4c0e990 con 0x55fdd48b8f60 2024-04-25T11:44:49.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.314+0000 7f959ffff640 1 --2- 172.21.15.90:0/2543184173 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55fdd48b8f60 0x55fdd4cbc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.314+0000 7f957ffff640 10 client.18684 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:44:49.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.314+0000 7f959ffff640 1 --2- 172.21.15.90:0/2543184173 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55fdd48b8f60 0x55fdd4cbc9c0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=283f262672c76736 server_cookie=d2527d880218498a in_seq=0 out_seq=0 2024-04-25T11:44:49.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.315+0000 7f957ffff640 1 -- 172.21.15.90:0/2543184173 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55fdd4c0e990 con 0x55fdd48b8f60 2024-04-25T11:44:49.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.315+0000 7f957ffff640 10 client.18684 handle_command_reply: tid=0 2024-04-25T11:44:49.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.315+0000 7f95addbe740 1 client.18684 shutdown 2024-04-25T11:44:49.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.315+0000 7f95addbe740 1 -- 172.21.15.90:0/2543184173 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48d14d0 msgr2=0x55fdd4caa670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.315+0000 7f95addbe740 1 --2- 172.21.15.90:0/2543184173 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48d14d0 0x55fdd4caa670 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f95900dc980 tx=0x7f95900a3860 comp rx=0 tx=0).stop 2024-04-25T11:44:49.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.315+0000 7f95addbe740 1 -- 172.21.15.90:0/2543184173 shutdown_connections 2024-04-25T11:44:49.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.315+0000 7f95addbe740 1 --2- 172.21.15.90:0/2543184173 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55fdd48b8f60 0x55fdd4cbc9c0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.315+0000 7f95addbe740 1 --2- 172.21.15.90:0/2543184173 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdd48fa390 0x55fdd4cb66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.316+0000 7f95addbe740 1 --2- 172.21.15.90:0/2543184173 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fdd48d96b0 0x55fdd4caabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.315+0000 7f957ffff640 0 client.18684 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:44:49.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.316+0000 7f95addbe740 1 --2- 172.21.15.90:0/2543184173 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48d14d0 0x55fdd4caa670 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.316+0000 7f95addbe740 1 -- 172.21.15.90:0/2543184173 >> 172.21.15.90:0/2543184173 conn(0x55fdd48ee4e0 msgr2=0x55fdd4c9d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:49.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.316+0000 7f95addbe740 1 -- 172.21.15.90:0/2543184173 shutdown_connections 2024-04-25T11:44:49.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.316+0000 7f95addbe740 1 -- 172.21.15.90:0/2543184173 wait complete. 2024-04-25T11:44:49.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.316+0000 7f95addbe740 20 client.18684 trim_cache size 0 max 16384 2024-04-25T11:44:49.309 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:48 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1917969616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:44:49.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.329+0000 7f95addbe740 1 Processor -- start 2024-04-25T11:44:49.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.329+0000 7f95addbe740 1 -- start start 2024-04-25T11:44:49.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.329+0000 7f95addbe740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdd48d14d0 0x55fdd4caa400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.329+0000 7f95addbe740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fdd48d96b0 0x55fdd4caa940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.330+0000 7f95addbe740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48fa390 0x55fdd4ca76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.330+0000 7f95addbe740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f95a01ad9b0 con 0x55fdd48fa390 2024-04-25T11:44:49.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.330+0000 7f95addbe740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f95a01af4f0 con 0x55fdd48d96b0 2024-04-25T11:44:49.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.330+0000 7f95addbe740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f95a01ae080 con 0x55fdd48d14d0 2024-04-25T11:44:49.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.330+0000 7f95a53d1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48fa390 0x55fdd4ca76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.330+0000 7f95a53d1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48fa390 0x55fdd4ca76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48630/0 (socket says 172.21.15.90:48630) 2024-04-25T11:44:49.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.330+0000 7f95a53d1640 1 -- 172.21.15.90:0/1996651871 learned_addr learned my addr 172.21.15.90:0/1996651871 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:49.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.330+0000 7f959ffff640 1 --2- 172.21.15.90:0/1996651871 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fdd48d96b0 0x55fdd4caa940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.330+0000 7f95a53d1640 1 -- 172.21.15.90:0/1996651871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdd48d14d0 msgr2=0x55fdd4caa400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.330+0000 7f95a53d1640 1 --2- 172.21.15.90:0/1996651871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdd48d14d0 0x55fdd4caa400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.330+0000 7f95a53d1640 1 -- 172.21.15.90:0/1996651871 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fdd48d96b0 msgr2=0x55fdd4caa940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.330+0000 7f95a53d1640 1 --2- 172.21.15.90:0/1996651871 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fdd48d96b0 0x55fdd4caa940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.330+0000 7f95a53d1640 1 -- 172.21.15.90:0/1996651871 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdd4c98af0 con 0x55fdd48fa390 2024-04-25T11:44:49.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.330+0000 7f959ffff640 1 --2- 172.21.15.90:0/1996651871 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fdd48d96b0 0x55fdd4caa940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:44:49.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.330+0000 7f95a53d1640 1 --2- 172.21.15.90:0/1996651871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48fa390 0x55fdd4ca76a0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f9598011780 tx=0x7f9598019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=258f09448d176520 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:49.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.330+0000 7f95867fc640 1 -- 172.21.15.90:0/1996651871 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9598024070 con 0x55fdd48fa390 2024-04-25T11:44:49.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.330+0000 7f95867fc640 1 -- 172.21.15.90:0/1996651871 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f959800e7d0 con 0x55fdd48fa390 2024-04-25T11:44:49.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.330+0000 7f95867fc640 1 -- 172.21.15.90:0/1996651871 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95980053a0 con 0x55fdd48fa390 2024-04-25T11:44:49.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.331+0000 7f95addbe740 1 -- 172.21.15.90:0/1996651871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48fa390 msgr2=0x55fdd4ca76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.331+0000 7f95addbe740 1 --2- 172.21.15.90:0/1996651871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48fa390 0x55fdd4ca76a0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f9598011780 tx=0x7f9598019000 comp rx=0 tx=0).stop 2024-04-25T11:44:49.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.331+0000 7f95addbe740 1 -- 172.21.15.90:0/1996651871 shutdown_connections 2024-04-25T11:44:49.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.331+0000 7f95addbe740 1 --2- 172.21.15.90:0/1996651871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48fa390 0x55fdd4ca76a0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.331+0000 7f95addbe740 1 --2- 172.21.15.90:0/1996651871 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fdd48d96b0 0x55fdd4caa940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.331+0000 7f95addbe740 1 --2- 172.21.15.90:0/1996651871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdd48d14d0 0x55fdd4caa400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.331+0000 7f95addbe740 1 -- 172.21.15.90:0/1996651871 >> 172.21.15.90:0/1996651871 conn(0x55fdd48ee4e0 msgr2=0x55fdd4ca9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:49.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.331+0000 7f95addbe740 1 -- 172.21.15.90:0/1996651871 shutdown_connections 2024-04-25T11:44:49.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.331+0000 7f95addbe740 1 -- 172.21.15.90:0/1996651871 wait complete. 2024-04-25T11:44:49.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.331+0000 7f95addbe740 1 -- start start 2024-04-25T11:44:49.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.331+0000 7f95addbe740 1 Processor -- start 2024-04-25T11:44:49.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.332+0000 7f95addbe740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48d14d0 0x55fdd4d1cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.332+0000 7f95addbe740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fdd48d96b0 0x55fdd4d1d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.332+0000 7f95addbe740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdd48fa390 0x55fdd4cbce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.332+0000 7f95addbe740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f95a01aee20 con 0x55fdd48d14d0 2024-04-25T11:44:49.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.332+0000 7f95addbe740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f95a01ae750 con 0x55fdd48d96b0 2024-04-25T11:44:49.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.332+0000 7f95addbe740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f95a01ab9b0 con 0x55fdd48fa390 2024-04-25T11:44:49.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.332+0000 7f95addbe740 10 client.0 _ll_register_callbacks cb 0x7f9594009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:49.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.332+0000 7f95a4bd0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48d14d0 0x55fdd4d1cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.332+0000 7f95a4bd0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48d14d0 0x55fdd4d1cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48634/0 (socket says 172.21.15.90:48634) 2024-04-25T11:44:49.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.332+0000 7f95a4bd0640 1 -- 172.21.15.90:0/3789533370 learned_addr learned my addr 172.21.15.90:0/3789533370 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:49.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.332+0000 7f95a4bd0640 1 -- 172.21.15.90:0/3789533370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdd48fa390 msgr2=0x55fdd4cbce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.332+0000 7f95a53d1640 1 --2- 172.21.15.90:0/3789533370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdd48fa390 0x55fdd4cbce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.332+0000 7f95a4bd0640 1 --2- 172.21.15.90:0/3789533370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdd48fa390 0x55fdd4cbce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.332+0000 7f95a4bd0640 1 -- 172.21.15.90:0/3789533370 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fdd48d96b0 msgr2=0x55fdd4d1d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.332+0000 7f959ffff640 1 --2- 172.21.15.90:0/3789533370 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fdd48d96b0 0x55fdd4d1d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.332+0000 7f95a4bd0640 1 --2- 172.21.15.90:0/3789533370 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fdd48d96b0 0x55fdd4d1d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.332+0000 7f95a4bd0640 1 -- 172.21.15.90:0/3789533370 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdd4cbd460 con 0x55fdd48d14d0 2024-04-25T11:44:49.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.332+0000 7f95a4bd0640 1 --2- 172.21.15.90:0/3789533370 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48d14d0 0x55fdd4d1cf10 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f95900a4590 tx=0x7f95900b02b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16268dc4a1b0e0ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:49.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.332+0000 7f957ffff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:49.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.333+0000 7f957ffff640 1 -- 172.21.15.90:0/3789533370 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95900a6a90 con 0x55fdd48d14d0 2024-04-25T11:44:49.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.333+0000 7f957ffff640 1 -- 172.21.15.90:0/3789533370 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f95900a7790 con 0x55fdd48d14d0 2024-04-25T11:44:49.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.333+0000 7f95addbe740 1 -- 172.21.15.90:0/3789533370 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f959800e7d0 con 0x55fdd48d14d0 2024-04-25T11:44:49.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.333+0000 7f957ffff640 1 -- 172.21.15.90:0/3789533370 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f95900a70f0 con 0x55fdd48d14d0 2024-04-25T11:44:49.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.333+0000 7f95addbe740 10 client.18690 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:49.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.333+0000 7f95addbe740 1 -- 172.21.15.90:0/3789533370 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fdd4cbd460 con 0x55fdd48d14d0 2024-04-25T11:44:49.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.333+0000 7f957ffff640 1 -- 172.21.15.90:0/3789533370 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f959800e7d0 con 0x55fdd48d14d0 2024-04-25T11:44:49.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.333+0000 7f957ffff640 1 -- 172.21.15.90:0/3789533370 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f9590067ce0 con 0x55fdd48d14d0 2024-04-25T11:44:49.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.333+0000 7f95addbe740 10 client.18690 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:49.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.333+0000 7f95addbe740 10 client.18690 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:44:49.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.333+0000 7f95addbe740 20 client.18690 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:49.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.333+0000 7f95addbe740 1 --2- 172.21.15.90:0/3789533370 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55fdd488e3a0 0x55fdd4d26860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.333+0000 7f95addbe740 4 client.18690 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:44:49.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.333+0000 7f95addbe740 1 -- 172.21.15.90:0/3789533370 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fdd4c08800 con 0x55fdd488e3a0 2024-04-25T11:44:49.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.333+0000 7f959ffff640 1 --2- 172.21.15.90:0/3789533370 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55fdd488e3a0 0x55fdd4d26860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.334+0000 7f959ffff640 1 --2- 172.21.15.90:0/3789533370 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55fdd488e3a0 0x55fdd4d26860 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7fcf7eb71fd452e4 server_cookie=72fdd49fa12ed2cd in_seq=0 out_seq=0 2024-04-25T11:44:49.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.334+0000 7f957ffff640 10 client.18690 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:44:49.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.334+0000 7f957ffff640 1 -- 172.21.15.90:0/3789533370 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fdd4c08800 con 0x55fdd488e3a0 2024-04-25T11:44:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.334+0000 7f957ffff640 10 client.18690 handle_command_reply: tid=0 2024-04-25T11:44:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.334+0000 7f95addbe740 1 client.18690 shutdown 2024-04-25T11:44:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.334+0000 7f95addbe740 1 -- 172.21.15.90:0/3789533370 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48d14d0 msgr2=0x55fdd4d1cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.334+0000 7f95addbe740 1 --2- 172.21.15.90:0/3789533370 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48d14d0 0x55fdd4d1cf10 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f95900a4590 tx=0x7f95900b02b0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.334+0000 7f95addbe740 1 -- 172.21.15.90:0/3789533370 shutdown_connections 2024-04-25T11:44:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.334+0000 7f95addbe740 1 --2- 172.21.15.90:0/3789533370 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55fdd488e3a0 0x55fdd4d26860 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.334+0000 7f95addbe740 1 --2- 172.21.15.90:0/3789533370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdd48fa390 0x55fdd4cbce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.334+0000 7f957ffff640 0 client.18690 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:44:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.334+0000 7f95addbe740 1 --2- 172.21.15.90:0/3789533370 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fdd48d96b0 0x55fdd4d1d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.334+0000 7f95addbe740 1 --2- 172.21.15.90:0/3789533370 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fdd48d14d0 0x55fdd4d1cf10 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.334+0000 7f95addbe740 1 -- 172.21.15.90:0/3789533370 >> 172.21.15.90:0/3789533370 conn(0x55fdd48ee4e0 msgr2=0x55fdd4ca9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.334+0000 7f95addbe740 1 -- 172.21.15.90:0/3789533370 shutdown_connections 2024-04-25T11:44:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.334+0000 7f95addbe740 1 -- 172.21.15.90:0/3789533370 wait complete. 2024-04-25T11:44:49.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.334+0000 7f95addbe740 20 client.18690 trim_cache size 0 max 16384 2024-04-25T11:44:49.328 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:44:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.335+0000 7f95a6e34640 1 -- 172.21.15.90:0/3640509801 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f95880afc60 msgr2=0x7f95880b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.335+0000 7f95a6e34640 1 --2- 172.21.15.90:0/3640509801 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f95880afc60 0x7f95880b2170 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9598009270 tx=0x7f9598013a80 comp rx=0 tx=0).stop 2024-04-25T11:44:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.335+0000 7f95a6e34640 1 -- 172.21.15.90:0/3640509801 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95a01032a0 msgr2=0x7f95a019a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.335+0000 7f95a6e34640 1 --2- 172.21.15.90:0/3640509801 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95a01032a0 0x7f95a019a750 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f959000c950 tx=0x7f959000ce10 comp rx=0 tx=0).stop 2024-04-25T11:44:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.335+0000 7f95a6e34640 1 -- 172.21.15.90:0/3640509801 shutdown_connections 2024-04-25T11:44:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.335+0000 7f95a6e34640 1 --2- 172.21.15.90:0/3640509801 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f95880afc60 0x7f95880b2170 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.335+0000 7f95a6e34640 1 --2- 172.21.15.90:0/3640509801 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f95a010d420 0x7f95a019ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.335+0000 7f95a6e34640 1 --2- 172.21.15.90:0/3640509801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95a0107930 0x7f95a019acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.335+0000 7f95a6e34640 1 --2- 172.21.15.90:0/3640509801 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95a01032a0 0x7f95a019a750 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.335+0000 7f95a6e34640 1 -- 172.21.15.90:0/3640509801 >> 172.21.15.90:0/3640509801 conn(0x7f95a0077880 msgr2=0x7f95a0100ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.335+0000 7f95a6e34640 1 -- 172.21.15.90:0/3640509801 shutdown_connections 2024-04-25T11:44:49.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.335+0000 7f95a6e34640 1 -- 172.21.15.90:0/3640509801 wait complete. 2024-04-25T11:44:49.335 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T11:44:49.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:48 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1917969616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:44:49.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.458+0000 7f86f8892640 1 -- 172.21.15.90:0/3525631459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f4102440 msgr2=0x7f86f4102850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.458+0000 7f86f8892640 1 --2- 172.21.15.90:0/3525631459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f4102440 0x7f86f4102850 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f86e4009900 tx=0x7f86e402a410 comp rx=0 tx=0).stop 2024-04-25T11:44:49.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.459+0000 7f86f8892640 1 -- 172.21.15.90:0/3525631459 shutdown_connections 2024-04-25T11:44:49.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.459+0000 7f86f8892640 1 --2- 172.21.15.90:0/3525631459 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f86f4107440 0x7f86f410ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.459+0000 7f86f8892640 1 --2- 172.21.15.90:0/3525631459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86f4106980 0x7f86f4106db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.459+0000 7f86f8892640 1 --2- 172.21.15.90:0/3525631459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f4102440 0x7f86f4102850 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.459+0000 7f86f8892640 1 -- 172.21.15.90:0/3525631459 >> 172.21.15.90:0/3525631459 conn(0x7f86f4077880 msgr2=0x7f86f4077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:49.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.459+0000 7f86f8892640 1 -- 172.21.15.90:0/3525631459 shutdown_connections 2024-04-25T11:44:49.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.459+0000 7f86f8892640 1 -- 172.21.15.90:0/3525631459 wait complete. 2024-04-25T11:44:49.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.459+0000 7f86f8892640 1 Processor -- start 2024-04-25T11:44:49.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.460+0000 7f86f8892640 1 -- start start 2024-04-25T11:44:49.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.460+0000 7f86f8892640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86f4102440 0x7f86f419a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.460+0000 7f86f8892640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f86f4106980 0x7f86f419ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.460+0000 7f86f8892640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f4107440 0x7f86f419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.460+0000 7f86f8892640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f86f4112370 con 0x7f86f4107440 2024-04-25T11:44:49.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.460+0000 7f86f8892640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f86f41121f0 con 0x7f86f4106980 2024-04-25T11:44:49.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.460+0000 7f86f8892640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f86f41124f0 con 0x7f86f4102440 2024-04-25T11:44:49.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.460+0000 7f86f2d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f4107440 0x7f86f419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.460+0000 7f86f2d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f4107440 0x7f86f419efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48654/0 (socket says 172.21.15.90:48654) 2024-04-25T11:44:49.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.460+0000 7f86f2d9d640 1 -- 172.21.15.90:0/492638827 learned_addr learned my addr 172.21.15.90:0/492638827 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:49.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.460+0000 7f86f1d9b640 1 --2- 172.21.15.90:0/492638827 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f86f4106980 0x7f86f419ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.460+0000 7f86f259c640 1 --2- 172.21.15.90:0/492638827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86f4102440 0x7f86f419a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.460+0000 7f86f2d9d640 1 -- 172.21.15.90:0/492638827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86f4102440 msgr2=0x7f86f419a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.460+0000 7f86f2d9d640 1 --2- 172.21.15.90:0/492638827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86f4102440 0x7f86f419a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.460+0000 7f86f2d9d640 1 -- 172.21.15.90:0/492638827 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f86f4106980 msgr2=0x7f86f419ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.460+0000 7f86f2d9d640 1 --2- 172.21.15.90:0/492638827 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f86f4106980 0x7f86f419ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.460+0000 7f86f2d9d640 1 -- 172.21.15.90:0/492638827 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86f419f610 con 0x7f86f4107440 2024-04-25T11:44:49.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.460+0000 7f86f2d9d640 1 --2- 172.21.15.90:0/492638827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f4107440 0x7f86f419efd0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f86ec00eaa0 tx=0x7f86ec00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:49.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.460+0000 7f86e37fe640 1 -- 172.21.15.90:0/492638827 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ec00cdb0 con 0x7f86f4107440 2024-04-25T11:44:49.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.461+0000 7f86f8892640 1 -- 172.21.15.90:0/492638827 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86f419f900 con 0x7f86f4107440 2024-04-25T11:44:49.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.461+0000 7f86e37fe640 1 -- 172.21.15.90:0/492638827 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86ec005020 con 0x7f86f4107440 2024-04-25T11:44:49.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.461+0000 7f86e37fe640 1 -- 172.21.15.90:0/492638827 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ec0116d0 con 0x7f86f4107440 2024-04-25T11:44:49.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.461+0000 7f86f8892640 1 -- 172.21.15.90:0/492638827 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86f419fe00 con 0x7f86f4107440 2024-04-25T11:44:49.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.463+0000 7f86ff81a740 1 Processor -- start 2024-04-25T11:44:49.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.463+0000 7f86e37fe640 1 -- 172.21.15.90:0/492638827 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f86ec0051d0 con 0x7f86f4107440 2024-04-25T11:44:49.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.463+0000 7f86ff81a740 1 -- start start 2024-04-25T11:44:49.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.463+0000 7f86ff81a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e678390 0x55c64ea19810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.463+0000 7f86ff81a740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c64e64f4d0 0x55c64ea19d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.463+0000 7f86ff81a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c64e6576b0 0x55c64ea266a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.463+0000 7f86ff81a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c64e97ed80 con 0x55c64e678390 2024-04-25T11:44:49.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.463+0000 7f86ff81a740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c64e938a20 con 0x55c64e64f4d0 2024-04-25T11:44:49.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.463+0000 7f86ff81a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c64e97ad40 con 0x55c64e6576b0 2024-04-25T11:44:49.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.464+0000 7f86f1d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e678390 0x55c64ea19810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.464+0000 7f86f1d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e678390 0x55c64ea19810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48670/0 (socket says 172.21.15.90:48670) 2024-04-25T11:44:49.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.464+0000 7f86f1d9b640 1 -- 172.21.15.90:0/3968970971 learned_addr learned my addr 172.21.15.90:0/3968970971 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:49.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.464+0000 7f86f259c640 1 --2- 172.21.15.90:0/3968970971 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c64e64f4d0 0x55c64ea19d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.464+0000 7f86e37fe640 1 --2- 172.21.15.90:0/492638827 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f86dc0afc60 0x7f86dc0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.464+0000 7f86f1d9b640 1 -- 172.21.15.90:0/3968970971 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c64e6576b0 msgr2=0x55c64ea266a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.464+0000 7f86f1d9b640 1 --2- 172.21.15.90:0/3968970971 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c64e6576b0 0x55c64ea266a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.464+0000 7f86f1d9b640 1 -- 172.21.15.90:0/3968970971 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c64e64f4d0 msgr2=0x55c64ea19d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.464+0000 7f86f1d9b640 1 --2- 172.21.15.90:0/3968970971 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c64e64f4d0 0x55c64ea19d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.464+0000 7f86f2d9d640 1 --2- 172.21.15.90:0/492638827 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f86dc0afc60 0x7f86dc0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.464+0000 7f86f1d9b640 1 -- 172.21.15.90:0/3968970971 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c64e943c30 con 0x55c64e678390 2024-04-25T11:44:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.464+0000 7f86e37fe640 1 -- 172.21.15.90:0/492638827 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f86ec0df290 con 0x7f86f4107440 2024-04-25T11:44:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.464+0000 7f86f1d9b640 1 --2- 172.21.15.90:0/3968970971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e678390 0x55c64ea19810 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f86e8004650 tx=0x7f86e800bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6318bc6d4d93ec66 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.464+0000 7f86e0ff9640 1 -- 172.21.15.90:0/3968970971 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e8003bd0 con 0x55c64e678390 2024-04-25T11:44:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.464+0000 7f86e0ff9640 1 -- 172.21.15.90:0/3968970971 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86e8016030 con 0x55c64e678390 2024-04-25T11:44:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.464+0000 7f86e0ff9640 1 -- 172.21.15.90:0/3968970971 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e8004c70 con 0x55c64e678390 2024-04-25T11:44:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.464+0000 7f86f2d9d640 1 --2- 172.21.15.90:0/492638827 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f86dc0afc60 0x7f86dc0b2170 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f86ec0dca00 tx=0x7f86ec005890 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.465+0000 7f86ff81a740 1 -- 172.21.15.90:0/3968970971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e678390 msgr2=0x55c64ea19810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.465+0000 7f86ff81a740 1 --2- 172.21.15.90:0/3968970971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e678390 0x55c64ea19810 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f86e8004650 tx=0x7f86e800bc20 comp rx=0 tx=0).stop 2024-04-25T11:44:49.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.465+0000 7f86ff81a740 1 -- 172.21.15.90:0/3968970971 shutdown_connections 2024-04-25T11:44:49.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.465+0000 7f86ff81a740 1 --2- 172.21.15.90:0/3968970971 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c64e6576b0 0x55c64ea266a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.465+0000 7f86ff81a740 1 --2- 172.21.15.90:0/3968970971 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c64e64f4d0 0x55c64ea19d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.465+0000 7f86ff81a740 1 --2- 172.21.15.90:0/3968970971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e678390 0x55c64ea19810 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.465+0000 7f86ff81a740 1 -- 172.21.15.90:0/3968970971 >> 172.21.15.90:0/3968970971 conn(0x55c64e66c4e0 msgr2=0x55c64ea15fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:49.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.465+0000 7f86ff81a740 1 -- 172.21.15.90:0/3968970971 shutdown_connections 2024-04-25T11:44:49.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.465+0000 7f86ff81a740 1 -- 172.21.15.90:0/3968970971 wait complete. 2024-04-25T11:44:49.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.466+0000 7f86ff81a740 1 -- start start 2024-04-25T11:44:49.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.466+0000 7f86ff81a740 1 Processor -- start 2024-04-25T11:44:49.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.466+0000 7f86ff81a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e64f4d0 0x55c64ea241a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.466+0000 7f86ff81a740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c64e6576b0 0x55c64ea246e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.466+0000 7f86ff81a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c64e678390 0x55c64ea301b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.466+0000 7f86ff81a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c64ea26d30 con 0x55c64e64f4d0 2024-04-25T11:44:49.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.466+0000 7f86ff81a740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c64e905dc0 con 0x55c64e6576b0 2024-04-25T11:44:49.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.466+0000 7f86ff81a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c64ea26eb0 con 0x55c64e678390 2024-04-25T11:44:49.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.466+0000 7f86ff81a740 10 client.0 _ll_register_callbacks cb 0x55c64e921600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:49.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.466+0000 7f86f1d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e64f4d0 0x55c64ea241a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.466+0000 7f86f1d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e64f4d0 0x55c64ea241a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48672/0 (socket says 172.21.15.90:48672) 2024-04-25T11:44:49.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.466+0000 7f86f1d9b640 1 -- 172.21.15.90:0/2797672106 learned_addr learned my addr 172.21.15.90:0/2797672106 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:49.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.466+0000 7f86f259c640 1 --2- 172.21.15.90:0/2797672106 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c64e6576b0 0x55c64ea246e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.466+0000 7f86f1d9b640 1 -- 172.21.15.90:0/2797672106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c64e678390 msgr2=0x55c64ea301b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.466+0000 7f86f1d9b640 1 --2- 172.21.15.90:0/2797672106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c64e678390 0x55c64ea301b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.466+0000 7f86f1d9b640 1 -- 172.21.15.90:0/2797672106 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c64e6576b0 msgr2=0x55c64ea246e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.466+0000 7f86f1d9b640 1 --2- 172.21.15.90:0/2797672106 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c64e6576b0 0x55c64ea246e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.466+0000 7f86f1d9b640 1 -- 172.21.15.90:0/2797672106 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c64e6903f0 con 0x55c64e64f4d0 2024-04-25T11:44:49.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.467+0000 7f86f1d9b640 1 --2- 172.21.15.90:0/2797672106 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e64f4d0 0x55c64ea241a0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f86e8004850 tx=0x7f86e8004420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae19e1d912101380 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:49.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.467+0000 7f86d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:49.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.467+0000 7f86d9ffb640 1 -- 172.21.15.90:0/2797672106 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e801b670 con 0x55c64e64f4d0 2024-04-25T11:44:49.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.467+0000 7f86d9ffb640 1 -- 172.21.15.90:0/2797672106 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f86e801b810 con 0x55c64e64f4d0 2024-04-25T11:44:49.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.467+0000 7f86ff81a740 1 -- 172.21.15.90:0/2797672106 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86e8016030 con 0x55c64e64f4d0 2024-04-25T11:44:49.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.467+0000 7f86d9ffb640 1 -- 172.21.15.90:0/2797672106 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f86e801ba10 con 0x55c64e64f4d0 2024-04-25T11:44:49.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.467+0000 7f86ff81a740 10 client.18702 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:49.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.467+0000 7f86ff81a740 1 -- 172.21.15.90:0/2797672106 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c64e75a7d0 con 0x55c64e64f4d0 2024-04-25T11:44:49.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.467+0000 7f86d9ffb640 1 -- 172.21.15.90:0/2797672106 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86e8016030 con 0x55c64e64f4d0 2024-04-25T11:44:49.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.467+0000 7f86d9ffb640 1 -- 172.21.15.90:0/2797672106 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f86e802a4e0 con 0x55c64e64f4d0 2024-04-25T11:44:49.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.467+0000 7f86ff81a740 10 client.18702 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:49.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.467+0000 7f86ff81a740 10 client.18702 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:44:49.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.467+0000 7f86ff81a740 20 client.18702 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:49.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.467+0000 7f86ff81a740 1 --2- 172.21.15.90:0/2797672106 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c64e636f60 0x55c64ea36430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.467+0000 7f86ff81a740 4 client.18702 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:44:49.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.467+0000 7f86ff81a740 1 -- 172.21.15.90:0/2797672106 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c64e98c990 con 0x55c64e636f60 2024-04-25T11:44:49.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.468+0000 7f86f259c640 1 --2- 172.21.15.90:0/2797672106 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c64e636f60 0x55c64ea36430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.468+0000 7f86d9ffb640 10 client.18702 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:49.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.468+0000 7f86f259c640 1 --2- 172.21.15.90:0/2797672106 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c64e636f60 0x55c64ea36430 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=330cb92ac46f3f18 server_cookie=6115658c8327ebf1 in_seq=0 out_seq=0 2024-04-25T11:44:49.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.469+0000 7f86d9ffb640 1 -- 172.21.15.90:0/2797672106 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c64e98c990 con 0x55c64e636f60 2024-04-25T11:44:49.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.469+0000 7f86d9ffb640 10 client.18702 handle_command_reply: tid=0 2024-04-25T11:44:49.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.469+0000 7f86ff81a740 1 client.18702 shutdown 2024-04-25T11:44:49.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.469+0000 7f86ff81a740 1 -- 172.21.15.90:0/2797672106 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e64f4d0 msgr2=0x55c64ea241a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.469+0000 7f86ff81a740 1 --2- 172.21.15.90:0/2797672106 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e64f4d0 0x55c64ea241a0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f86e8004850 tx=0x7f86e8004420 comp rx=0 tx=0).stop 2024-04-25T11:44:49.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.469+0000 7f86ff81a740 1 -- 172.21.15.90:0/2797672106 shutdown_connections 2024-04-25T11:44:49.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.469+0000 7f86ff81a740 1 --2- 172.21.15.90:0/2797672106 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c64e636f60 0x55c64ea36430 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.469+0000 7f86ff81a740 1 --2- 172.21.15.90:0/2797672106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c64e678390 0x55c64ea301b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.469+0000 7f86d9ffb640 0 client.18702 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:49.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.469+0000 7f86ff81a740 1 --2- 172.21.15.90:0/2797672106 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c64e6576b0 0x55c64ea246e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.469+0000 7f86ff81a740 1 --2- 172.21.15.90:0/2797672106 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e64f4d0 0x55c64ea241a0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.469+0000 7f86ff81a740 1 -- 172.21.15.90:0/2797672106 >> 172.21.15.90:0/2797672106 conn(0x55c64e66c4e0 msgr2=0x55c64ea1b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:49.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.469+0000 7f86ff81a740 1 -- 172.21.15.90:0/2797672106 shutdown_connections 2024-04-25T11:44:49.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.469+0000 7f86ff81a740 1 -- 172.21.15.90:0/2797672106 wait complete. 2024-04-25T11:44:49.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.469+0000 7f86ff81a740 20 client.18702 trim_cache size 0 max 16384 2024-04-25T11:44:49.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.483+0000 7f86ff81a740 1 Processor -- start 2024-04-25T11:44:49.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.483+0000 7f86ff81a740 1 -- start start 2024-04-25T11:44:49.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.483+0000 7f86ff81a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c64e64f4d0 0x55c64ea16150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.483+0000 7f86ff81a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e6576b0 0x55c64ea2fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.483+0000 7f86ff81a740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c64e678390 0x55c64ea22e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.483+0000 7f86ff81a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f86f41afce0 con 0x55c64e6576b0 2024-04-25T11:44:49.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.483+0000 7f86ff81a740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f86f41b1820 con 0x55c64e678390 2024-04-25T11:44:49.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.483+0000 7f86ff81a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f86f41b03b0 con 0x55c64e64f4d0 2024-04-25T11:44:49.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.483+0000 7f86f259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e6576b0 0x55c64ea2fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.484+0000 7f86f259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e6576b0 0x55c64ea2fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48678/0 (socket says 172.21.15.90:48678) 2024-04-25T11:44:49.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.484+0000 7f86f259c640 1 -- 172.21.15.90:0/1061144543 learned_addr learned my addr 172.21.15.90:0/1061144543 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:49.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.484+0000 7f86f1d9b640 1 --2- 172.21.15.90:0/1061144543 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c64e678390 0x55c64ea22e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.484+0000 7f86f1d9b640 1 --2- 172.21.15.90:0/1061144543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c64e64f4d0 0x55c64ea16150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.484+0000 7f86f259c640 1 -- 172.21.15.90:0/1061144543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c64e64f4d0 msgr2=0x55c64ea16150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.484+0000 7f86f259c640 1 --2- 172.21.15.90:0/1061144543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c64e64f4d0 0x55c64ea16150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.484+0000 7f86f259c640 1 -- 172.21.15.90:0/1061144543 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c64e678390 msgr2=0x55c64ea22e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.484+0000 7f86f259c640 1 --2- 172.21.15.90:0/1061144543 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c64e678390 0x55c64ea22e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.484+0000 7f86f259c640 1 -- 172.21.15.90:0/1061144543 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c64ea224e0 con 0x55c64e6576b0 2024-04-25T11:44:49.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.484+0000 7f86f259c640 1 --2- 172.21.15.90:0/1061144543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e6576b0 0x55c64ea2fe10 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f86e4009340 tx=0x7f86e40092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29d5a6ba6c144f9f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:49.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.484+0000 7f86dbfff640 1 -- 172.21.15.90:0/1061144543 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e4035040 con 0x55c64e6576b0 2024-04-25T11:44:49.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.484+0000 7f86dbfff640 1 -- 172.21.15.90:0/1061144543 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86e4049070 con 0x55c64e6576b0 2024-04-25T11:44:49.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.484+0000 7f86dbfff640 1 -- 172.21.15.90:0/1061144543 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e4004920 con 0x55c64e6576b0 2024-04-25T11:44:49.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.485+0000 7f86ff81a740 1 -- 172.21.15.90:0/1061144543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e6576b0 msgr2=0x55c64ea2fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.485+0000 7f86ff81a740 1 --2- 172.21.15.90:0/1061144543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e6576b0 0x55c64ea2fe10 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f86e4009340 tx=0x7f86e40092d0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.485+0000 7f86ff81a740 1 -- 172.21.15.90:0/1061144543 shutdown_connections 2024-04-25T11:44:49.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.485+0000 7f86ff81a740 1 --2- 172.21.15.90:0/1061144543 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c64e678390 0x55c64ea22e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.485+0000 7f86ff81a740 1 --2- 172.21.15.90:0/1061144543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e6576b0 0x55c64ea2fe10 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.485+0000 7f86ff81a740 1 --2- 172.21.15.90:0/1061144543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c64e64f4d0 0x55c64ea16150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.485+0000 7f86ff81a740 1 -- 172.21.15.90:0/1061144543 >> 172.21.15.90:0/1061144543 conn(0x55c64e66c4e0 msgr2=0x55c64ea22730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:49.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.485+0000 7f86ff81a740 1 -- 172.21.15.90:0/1061144543 shutdown_connections 2024-04-25T11:44:49.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.485+0000 7f86ff81a740 1 -- 172.21.15.90:0/1061144543 wait complete. 2024-04-25T11:44:49.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.485+0000 7f86ff81a740 1 -- start start 2024-04-25T11:44:49.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.485+0000 7f86ff81a740 1 Processor -- start 2024-04-25T11:44:49.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.486+0000 7f86ff81a740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c64e64f4d0 0x55c64ea27150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.486+0000 7f86ff81a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c64e6576b0 0x55c64ea27690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.486+0000 7f86ff81a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e678390 0x55c64ea9c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.486+0000 7f86ff81a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f86f41b1150 con 0x55c64e678390 2024-04-25T11:44:49.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.486+0000 7f86ff81a740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f86f41b0a80 con 0x55c64e64f4d0 2024-04-25T11:44:49.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.486+0000 7f86ff81a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f86f41adce0 con 0x55c64e6576b0 2024-04-25T11:44:49.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.486+0000 7f86ff81a740 10 client.0 _ll_register_callbacks cb 0x7f86e4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:49.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.486+0000 7f86f1d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e678390 0x55c64ea9c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.486+0000 7f86f1d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e678390 0x55c64ea9c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48690/0 (socket says 172.21.15.90:48690) 2024-04-25T11:44:49.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.486+0000 7f86f1d9b640 1 -- 172.21.15.90:0/1616975880 learned_addr learned my addr 172.21.15.90:0/1616975880 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:49.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.486+0000 7f86f1d9b640 1 --2- 172.21.15.90:0/1616975880 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c64e64f4d0 0x55c64ea27150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.486+0000 7f86f259c640 1 --2- 172.21.15.90:0/1616975880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c64e6576b0 0x55c64ea27690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.486+0000 7f86f1d9b640 1 -- 172.21.15.90:0/1616975880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c64e6576b0 msgr2=0x55c64ea27690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.486+0000 7f86f1d9b640 1 --2- 172.21.15.90:0/1616975880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c64e6576b0 0x55c64ea27690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.486+0000 7f86f1d9b640 1 -- 172.21.15.90:0/1616975880 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c64e64f4d0 msgr2=0x55c64ea27150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.486+0000 7f86f1d9b640 1 --2- 172.21.15.90:0/1616975880 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c64e64f4d0 0x55c64ea27150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.486+0000 7f86f1d9b640 1 -- 172.21.15.90:0/1616975880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c64ea27ea0 con 0x55c64e678390 2024-04-25T11:44:49.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.486+0000 7f86f1d9b640 1 --2- 172.21.15.90:0/1616975880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e678390 0x55c64ea9c600 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f86e8001eb0 tx=0x7f86e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a580c9e7be8e8a5f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:49.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.486+0000 7f86d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:49.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.486+0000 7f86d9ffb640 1 -- 172.21.15.90:0/1616975880 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e801bdc0 con 0x55c64e678390 2024-04-25T11:44:49.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.486+0000 7f86d9ffb640 1 -- 172.21.15.90:0/1616975880 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f86e801b470 con 0x55c64e678390 2024-04-25T11:44:49.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.486+0000 7f86ff81a740 1 -- 172.21.15.90:0/1616975880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86e4049070 con 0x55c64e678390 2024-04-25T11:44:49.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.486+0000 7f86d9ffb640 1 -- 172.21.15.90:0/1616975880 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f86e8040050 con 0x55c64e678390 2024-04-25T11:44:49.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.486+0000 7f86ff81a740 10 client.18708 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:49.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.486+0000 7f86ff81a740 1 -- 172.21.15.90:0/1616975880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c64ea9db00 con 0x55c64e678390 2024-04-25T11:44:49.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.487+0000 7f86d9ffb640 1 -- 172.21.15.90:0/1616975880 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86e4049070 con 0x55c64e678390 2024-04-25T11:44:49.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.487+0000 7f86d9ffb640 1 -- 172.21.15.90:0/1616975880 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f86e802a340 con 0x55c64e678390 2024-04-25T11:44:49.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.487+0000 7f86ff81a740 10 client.18708 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:49.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.487+0000 7f86ff81a740 10 client.18708 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:44:49.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.487+0000 7f86ff81a740 20 client.18708 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:49.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.487+0000 7f86ff81a740 1 --2- 172.21.15.90:0/1616975880 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c64e60c3a0 0x55c64eaa0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.487+0000 7f86ff81a740 4 client.18708 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:44:49.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.487+0000 7f86ff81a740 1 -- 172.21.15.90:0/1616975880 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c64ea1a0e0 con 0x55c64e60c3a0 2024-04-25T11:44:49.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.487+0000 7f86f259c640 1 --2- 172.21.15.90:0/1616975880 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c64e60c3a0 0x55c64eaa0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.488+0000 7f86d9ffb640 10 client.18708 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:49.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.488+0000 7f86f259c640 1 --2- 172.21.15.90:0/1616975880 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c64e60c3a0 0x55c64eaa0210 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1c513e79c2af123 server_cookie=adf3860de600a9ab in_seq=0 out_seq=0 2024-04-25T11:44:49.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.488+0000 7f86d9ffb640 1 -- 172.21.15.90:0/1616975880 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c64ea1a0e0 con 0x55c64e60c3a0 2024-04-25T11:44:49.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.488+0000 7f86d9ffb640 10 client.18708 handle_command_reply: tid=0 2024-04-25T11:44:49.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.488+0000 7f86ff81a740 1 client.18708 shutdown 2024-04-25T11:44:49.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.488+0000 7f86ff81a740 1 -- 172.21.15.90:0/1616975880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e678390 msgr2=0x55c64ea9c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.488+0000 7f86ff81a740 1 --2- 172.21.15.90:0/1616975880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e678390 0x55c64ea9c600 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f86e8001eb0 tx=0x7f86e80046c0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.488+0000 7f86ff81a740 1 -- 172.21.15.90:0/1616975880 shutdown_connections 2024-04-25T11:44:49.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.488+0000 7f86ff81a740 1 --2- 172.21.15.90:0/1616975880 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c64e60c3a0 0x55c64eaa0210 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.488+0000 7f86ff81a740 1 --2- 172.21.15.90:0/1616975880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c64e678390 0x55c64ea9c600 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.488+0000 7f86d9ffb640 0 client.18708 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:49.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.488+0000 7f86ff81a740 1 --2- 172.21.15.90:0/1616975880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c64e6576b0 0x55c64ea27690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.488+0000 7f86ff81a740 1 --2- 172.21.15.90:0/1616975880 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c64e64f4d0 0x55c64ea27150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.488+0000 7f86ff81a740 1 -- 172.21.15.90:0/1616975880 >> 172.21.15.90:0/1616975880 conn(0x55c64e66c4e0 msgr2=0x55c64ea22ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:49.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.488+0000 7f86ff81a740 1 -- 172.21.15.90:0/1616975880 shutdown_connections 2024-04-25T11:44:49.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.488+0000 7f86ff81a740 1 -- 172.21.15.90:0/1616975880 wait complete. 2024-04-25T11:44:49.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.488+0000 7f86ff81a740 20 client.18708 trim_cache size 0 max 16384 2024-04-25T11:44:49.482 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:44:49.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.489+0000 7f86f8892640 1 -- 172.21.15.90:0/492638827 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f86dc0afc60 msgr2=0x7f86dc0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.489+0000 7f86f8892640 1 --2- 172.21.15.90:0/492638827 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f86dc0afc60 0x7f86dc0b2170 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f86ec0dca00 tx=0x7f86ec005890 comp rx=0 tx=0).stop 2024-04-25T11:44:49.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.489+0000 7f86f8892640 1 -- 172.21.15.90:0/492638827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f4107440 msgr2=0x7f86f419efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.489+0000 7f86f8892640 1 --2- 172.21.15.90:0/492638827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f4107440 0x7f86f419efd0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f86ec00eaa0 tx=0x7f86ec00ef60 comp rx=0 tx=0).stop 2024-04-25T11:44:49.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.489+0000 7f86f8892640 1 -- 172.21.15.90:0/492638827 shutdown_connections 2024-04-25T11:44:49.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.489+0000 7f86f8892640 1 --2- 172.21.15.90:0/492638827 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f86dc0afc60 0x7f86dc0b2170 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.489+0000 7f86f8892640 1 --2- 172.21.15.90:0/492638827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f4107440 0x7f86f419efd0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.489+0000 7f86f8892640 1 --2- 172.21.15.90:0/492638827 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f86f4106980 0x7f86f419ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.489+0000 7f86f8892640 1 --2- 172.21.15.90:0/492638827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86f4102440 0x7f86f419a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.489+0000 7f86f8892640 1 -- 172.21.15.90:0/492638827 >> 172.21.15.90:0/492638827 conn(0x7f86f4077880 msgr2=0x7f86f410ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:49.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.489+0000 7f86f8892640 1 -- 172.21.15.90:0/492638827 shutdown_connections 2024-04-25T11:44:49.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:49.489+0000 7f86f8892640 1 -- 172.21.15.90:0/492638827 wait complete. 2024-04-25T11:44:50.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:49 smithi090 ceph-mon[29575]: pgmap v301: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:44:50.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:49 smithi138 ceph-mon[32328]: pgmap v301: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:44:50.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:49 smithi156 ceph-mon[33045]: pgmap v301: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:44:50.487 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T11:44:50.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:44:50.488 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T11:44:50.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.627+0000 7f6675a27640 1 -- 172.21.15.90:0/4050183189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f667010d380 msgr2=0x7f667010f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.627+0000 7f6675a27640 1 --2- 172.21.15.90:0/4050183189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f667010d380 0x7f667010f7a0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f6668009900 tx=0x7f666802a410 comp rx=0 tx=0).stop 2024-04-25T11:44:50.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.627+0000 7f6675a27640 1 -- 172.21.15.90:0/4050183189 shutdown_connections 2024-04-25T11:44:50.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.627+0000 7f6675a27640 1 --2- 172.21.15.90:0/4050183189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f667010d380 0x7f667010f7a0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.627+0000 7f6675a27640 1 --2- 172.21.15.90:0/4050183189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6670102af0 0x7f6670102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.627+0000 7f6675a27640 1 --2- 172.21.15.90:0/4050183189 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6670108a20 0x7f66701025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.627+0000 7f6675a27640 1 -- 172.21.15.90:0/4050183189 >> 172.21.15.90:0/4050183189 conn(0x7f6670078100 msgr2=0x7f6670078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:50.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.627+0000 7f6675a27640 1 -- 172.21.15.90:0/4050183189 shutdown_connections 2024-04-25T11:44:50.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.627+0000 7f6675a27640 1 -- 172.21.15.90:0/4050183189 wait complete. 2024-04-25T11:44:50.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.628+0000 7f6675a27640 1 Processor -- start 2024-04-25T11:44:50.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.628+0000 7f6675a27640 1 -- start start 2024-04-25T11:44:50.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.628+0000 7f6675a27640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6670102af0 0x7f667019b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.628+0000 7f6675a27640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6670108a20 0x7f667019ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.628+0000 7f6675a27640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f667010d380 0x7f667019bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.628+0000 7f6675a27640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6670112c00 con 0x7f6670102af0 2024-04-25T11:44:50.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.628+0000 7f6675a27640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6670112a80 con 0x7f6670108a20 2024-04-25T11:44:50.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.628+0000 7f6675a27640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6670112d80 con 0x7f667010d380 2024-04-25T11:44:50.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.628+0000 7f666effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6670102af0 0x7f667019b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.628+0000 7f666effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6670102af0 0x7f667019b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48712/0 (socket says 172.21.15.90:48712) 2024-04-25T11:44:50.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.628+0000 7f666effd640 1 -- 172.21.15.90:0/1969669715 learned_addr learned my addr 172.21.15.90:0/1969669715 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:50.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.628+0000 7f666f7fe640 1 --2- 172.21.15.90:0/1969669715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f667010d380 0x7f667019bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.628+0000 7f666e7fc640 1 --2- 172.21.15.90:0/1969669715 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6670108a20 0x7f667019ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.629+0000 7f666effd640 1 -- 172.21.15.90:0/1969669715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f667010d380 msgr2=0x7f667019bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.629+0000 7f666effd640 1 --2- 172.21.15.90:0/1969669715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f667010d380 0x7f667019bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.629+0000 7f666effd640 1 -- 172.21.15.90:0/1969669715 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6670108a20 msgr2=0x7f667019ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.629+0000 7f666effd640 1 --2- 172.21.15.90:0/1969669715 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6670108a20 0x7f667019ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.629+0000 7f666effd640 1 -- 172.21.15.90:0/1969669715 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66701a6810 con 0x7f6670102af0 2024-04-25T11:44:50.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.629+0000 7f666e7fc640 1 --2- 172.21.15.90:0/1969669715 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6670108a20 0x7f667019ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:50.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.629+0000 7f666effd640 1 --2- 172.21.15.90:0/1969669715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6670102af0 0x7f667019b500 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f666000eb90 tx=0x7f666000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:50.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.629+0000 7f6674a25640 1 -- 172.21.15.90:0/1969669715 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6660010630 con 0x7f6670102af0 2024-04-25T11:44:50.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.629+0000 7f6674a25640 1 -- 172.21.15.90:0/1969669715 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f666000ee90 con 0x7f6670102af0 2024-04-25T11:44:50.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.629+0000 7f6675a27640 1 -- 172.21.15.90:0/1969669715 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66701a6b00 con 0x7f6670102af0 2024-04-25T11:44:50.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.629+0000 7f6674a25640 1 -- 172.21.15.90:0/1969669715 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6660018ac0 con 0x7f6670102af0 2024-04-25T11:44:50.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.629+0000 7f6675a27640 1 -- 172.21.15.90:0/1969669715 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66701a7050 con 0x7f6670102af0 2024-04-25T11:44:50.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.631+0000 7f6674a25640 1 -- 172.21.15.90:0/1969669715 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f6660018c60 con 0x7f6670102af0 2024-04-25T11:44:50.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.631+0000 7f667c9bb740 1 Processor -- start 2024-04-25T11:44:50.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.631+0000 7f667c9bb740 1 -- start start 2024-04-25T11:44:50.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.632+0000 7f667c9bb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1833410 0x5618d1bd3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.632+0000 7f667c9bb740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5618d180a550 0x5618d1bd3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.632+0000 7f667c9bb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618d1812730 0x5618d1bd40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.632+0000 7f667c9bb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5618d1b39b00 con 0x5618d1833410 2024-04-25T11:44:50.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.632+0000 7f667c9bb740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5618d1ad5920 con 0x5618d180a550 2024-04-25T11:44:50.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.632+0000 7f667c9bb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5618d1b35ac0 con 0x5618d1812730 2024-04-25T11:44:50.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.632+0000 7f666f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1833410 0x5618d1bd3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.632+0000 7f666f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1833410 0x5618d1bd3620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48726/0 (socket says 172.21.15.90:48726) 2024-04-25T11:44:50.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.632+0000 7f666f7fe640 1 -- 172.21.15.90:0/3794021214 learned_addr learned my addr 172.21.15.90:0/3794021214 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:50.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.632+0000 7f6674a25640 1 --2- 172.21.15.90:0/1969669715 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f66580afc60 0x7f66580b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.632+0000 7f666e7fc640 1 --2- 172.21.15.90:0/3794021214 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618d1812730 0x5618d1bd40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.632+0000 7f666effd640 1 --2- 172.21.15.90:0/3794021214 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5618d180a550 0x5618d1bd3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.632+0000 7f666f7fe640 1 --2- 172.21.15.90:0/1969669715 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f66580afc60 0x7f66580b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.632+0000 7f666f7fe640 1 -- 172.21.15.90:0/3794021214 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618d1812730 msgr2=0x5618d1bd40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:50.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.632+0000 7f666f7fe640 1 --2- 172.21.15.90:0/3794021214 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618d1812730 0x5618d1bd40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.632+0000 7f666f7fe640 1 -- 172.21.15.90:0/3794021214 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5618d180a550 msgr2=0x5618d1bd3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:50.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.632+0000 7f666f7fe640 1 --2- 172.21.15.90:0/3794021214 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5618d180a550 0x5618d1bd3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.632+0000 7f6674a25640 1 -- 172.21.15.90:0/1969669715 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f66600df760 con 0x7f6670102af0 2024-04-25T11:44:50.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.632+0000 7f666f7fe640 1 -- 172.21.15.90:0/3794021214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618d19137e0 con 0x5618d1833410 2024-04-25T11:44:50.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.632+0000 7f666f7fe640 1 --2- 172.21.15.90:0/3794021214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1833410 0x5618d1bd3620 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f6668005ab0 tx=0x7f66680314a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7839c28d4f4268f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:50.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.633+0000 7f6655ffb640 1 -- 172.21.15.90:0/3794021214 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666804c070 con 0x5618d1833410 2024-04-25T11:44:50.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.633+0000 7f6655ffb640 1 -- 172.21.15.90:0/3794021214 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66680054f0 con 0x5618d1833410 2024-04-25T11:44:50.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.633+0000 7f666f7fe640 1 --2- 172.21.15.90:0/1969669715 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f66580afc60 0x7f66580b2170 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6668049000 tx=0x7f6668046000 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:50.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.633+0000 7f6655ffb640 1 -- 172.21.15.90:0/3794021214 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6668035070 con 0x5618d1833410 2024-04-25T11:44:50.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.633+0000 7f667c9bb740 1 -- 172.21.15.90:0/3794021214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1833410 msgr2=0x5618d1bd3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.633+0000 7f667c9bb740 1 --2- 172.21.15.90:0/3794021214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1833410 0x5618d1bd3620 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f6668005ab0 tx=0x7f66680314a0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.634+0000 7f667c9bb740 1 -- 172.21.15.90:0/3794021214 shutdown_connections 2024-04-25T11:44:50.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.634+0000 7f667c9bb740 1 --2- 172.21.15.90:0/3794021214 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618d1812730 0x5618d1bd40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.634+0000 7f667c9bb740 1 --2- 172.21.15.90:0/3794021214 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5618d180a550 0x5618d1bd3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.634+0000 7f667c9bb740 1 --2- 172.21.15.90:0/3794021214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1833410 0x5618d1bd3620 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.634+0000 7f667c9bb740 1 -- 172.21.15.90:0/3794021214 >> 172.21.15.90:0/3794021214 conn(0x5618d1827560 msgr2=0x5618d1bcfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:50.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.634+0000 7f667c9bb740 1 -- 172.21.15.90:0/3794021214 shutdown_connections 2024-04-25T11:44:50.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.634+0000 7f667c9bb740 1 -- 172.21.15.90:0/3794021214 wait complete. 2024-04-25T11:44:50.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.634+0000 7f667c9bb740 1 -- start start 2024-04-25T11:44:50.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.634+0000 7f667c9bb740 1 Processor -- start 2024-04-25T11:44:50.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.635+0000 7f667c9bb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618d180a550 0x5618d1bddb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.635+0000 7f667c9bb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1812730 0x5618d1bde0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.635+0000 7f667c9bb740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5618d1833410 0x5618d1be9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.635+0000 7f667c9bb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5618d1be0970 con 0x5618d1812730 2024-04-25T11:44:50.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.635+0000 7f667c9bb740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5618d1ad35d0 con 0x5618d1833410 2024-04-25T11:44:50.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.635+0000 7f667c9bb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5618d1be0af0 con 0x5618d180a550 2024-04-25T11:44:50.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.635+0000 7f667c9bb740 10 client.0 _ll_register_callbacks cb 0x5618d1ac1080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:50.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.635+0000 7f666e7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1812730 0x5618d1bde0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.635+0000 7f666e7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1812730 0x5618d1bde0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48732/0 (socket says 172.21.15.90:48732) 2024-04-25T11:44:50.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.635+0000 7f666e7fc640 1 -- 172.21.15.90:0/1041918863 learned_addr learned my addr 172.21.15.90:0/1041918863 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:50.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.635+0000 7f666effd640 1 --2- 172.21.15.90:0/1041918863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618d180a550 0x5618d1bddb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.635+0000 7f666f7fe640 1 --2- 172.21.15.90:0/1041918863 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5618d1833410 0x5618d1be9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.635+0000 7f666e7fc640 1 -- 172.21.15.90:0/1041918863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618d180a550 msgr2=0x5618d1bddb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:50.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.635+0000 7f666e7fc640 1 --2- 172.21.15.90:0/1041918863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618d180a550 0x5618d1bddb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.635+0000 7f666e7fc640 1 -- 172.21.15.90:0/1041918863 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5618d1833410 msgr2=0x5618d1be9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:50.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.635+0000 7f666e7fc640 1 --2- 172.21.15.90:0/1041918863 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5618d1833410 0x5618d1be9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.635+0000 7f666e7fc640 1 -- 172.21.15.90:0/1041918863 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618d1973f70 con 0x5618d1812730 2024-04-25T11:44:50.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.635+0000 7f666e7fc640 1 --2- 172.21.15.90:0/1041918863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1812730 0x5618d1bde0b0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f6664009f80 tx=0x7f66640117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f41cb0e33ed7f68 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:50.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.635+0000 7f664effd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:50.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.635+0000 7f664effd640 1 -- 172.21.15.90:0/1041918863 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6664005220 con 0x5618d1812730 2024-04-25T11:44:50.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.635+0000 7f664effd640 1 -- 172.21.15.90:0/1041918863 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f66640053c0 con 0x5618d1812730 2024-04-25T11:44:50.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.635+0000 7f667c9bb740 1 -- 172.21.15.90:0/1041918863 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66680054f0 con 0x5618d1812730 2024-04-25T11:44:50.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.636+0000 7f664effd640 1 -- 172.21.15.90:0/1041918863 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f66640055c0 con 0x5618d1812730 2024-04-25T11:44:50.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.636+0000 7f667c9bb740 10 client.18720 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:50.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.636+0000 7f667c9bb740 1 -- 172.21.15.90:0/1041918863 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5618d180b170 con 0x5618d1812730 2024-04-25T11:44:50.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.636+0000 7f664effd640 1 -- 172.21.15.90:0/1041918863 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66680054f0 con 0x5618d1812730 2024-04-25T11:44:50.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.636+0000 7f664effd640 1 -- 172.21.15.90:0/1041918863 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f666400f040 con 0x5618d1812730 2024-04-25T11:44:50.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.636+0000 7f667c9bb740 10 client.18720 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:50.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.636+0000 7f667c9bb740 10 client.18720 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:44:50.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.636+0000 7f667c9bb740 20 client.18720 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:50.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.636+0000 7f667c9bb740 1 --2- 172.21.15.90:0/1041918863 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5618d17f21f0 0x5618d1bf0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.636+0000 7f667c9bb740 4 client.18720 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:44:50.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.636+0000 7f667c9bb740 1 -- 172.21.15.90:0/1041918863 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618d1b47710 con 0x5618d17f21f0 2024-04-25T11:44:50.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.636+0000 7f666effd640 1 --2- 172.21.15.90:0/1041918863 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5618d17f21f0 0x5618d1bf0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.637+0000 7f664effd640 10 client.18720 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:50.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.637+0000 7f666effd640 1 --2- 172.21.15.90:0/1041918863 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5618d17f21f0 0x5618d1bf0050 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f38667aeabec0fdf server_cookie=b89fe2f4af5dc464 in_seq=0 out_seq=0 2024-04-25T11:44:50.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.637+0000 7f664effd640 1 -- 172.21.15.90:0/1041918863 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5618d1b47710 con 0x5618d17f21f0 2024-04-25T11:44:50.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.637+0000 7f664effd640 10 client.18720 handle_command_reply: tid=0 2024-04-25T11:44:50.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.638+0000 7f667c9bb740 1 client.18720 shutdown 2024-04-25T11:44:50.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.638+0000 7f667c9bb740 1 -- 172.21.15.90:0/1041918863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1812730 msgr2=0x5618d1bde0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.638+0000 7f667c9bb740 1 --2- 172.21.15.90:0/1041918863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1812730 0x5618d1bde0b0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f6664009f80 tx=0x7f66640117b0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.638+0000 7f667c9bb740 1 -- 172.21.15.90:0/1041918863 shutdown_connections 2024-04-25T11:44:50.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.638+0000 7f667c9bb740 1 --2- 172.21.15.90:0/1041918863 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5618d17f21f0 0x5618d1bf0050 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.638+0000 7f667c9bb740 1 --2- 172.21.15.90:0/1041918863 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5618d1833410 0x5618d1be9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.638+0000 7f667c9bb740 1 --2- 172.21.15.90:0/1041918863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1812730 0x5618d1bde0b0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.638+0000 7f667c9bb740 1 --2- 172.21.15.90:0/1041918863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618d180a550 0x5618d1bddb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.638+0000 7f664effd640 0 client.18720 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:50.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.638+0000 7f667c9bb740 1 -- 172.21.15.90:0/1041918863 >> 172.21.15.90:0/1041918863 conn(0x5618d1827560 msgr2=0x5618d1bce4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:50.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.638+0000 7f667c9bb740 1 -- 172.21.15.90:0/1041918863 shutdown_connections 2024-04-25T11:44:50.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.638+0000 7f667c9bb740 1 -- 172.21.15.90:0/1041918863 wait complete. 2024-04-25T11:44:50.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.638+0000 7f667c9bb740 20 client.18720 trim_cache size 0 max 16384 2024-04-25T11:44:50.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.656+0000 7f667c9bb740 1 Processor -- start 2024-04-25T11:44:50.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.656+0000 7f667c9bb740 1 -- start start 2024-04-25T11:44:50.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.656+0000 7f667c9bb740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5618d180a550 0x5618d1bdcc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.656+0000 7f667c9bb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618d1812730 0x5618d1bd3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.656+0000 7f667c9bb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1833410 0x5618d1be9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.656+0000 7f667c9bb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f66701afaf0 con 0x5618d1833410 2024-04-25T11:44:50.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.656+0000 7f667c9bb740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f66701b1630 con 0x5618d180a550 2024-04-25T11:44:50.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.656+0000 7f667c9bb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f66701b01c0 con 0x5618d1812730 2024-04-25T11:44:50.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.656+0000 7f666f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1833410 0x5618d1be9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.656+0000 7f666f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1833410 0x5618d1be9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48748/0 (socket says 172.21.15.90:48748) 2024-04-25T11:44:50.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.656+0000 7f666f7fe640 1 -- 172.21.15.90:0/3531678064 learned_addr learned my addr 172.21.15.90:0/3531678064 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:50.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.656+0000 7f666effd640 1 --2- 172.21.15.90:0/3531678064 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5618d180a550 0x5618d1bdcc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.656+0000 7f666e7fc640 1 --2- 172.21.15.90:0/3531678064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618d1812730 0x5618d1bd3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.656+0000 7f666f7fe640 1 -- 172.21.15.90:0/3531678064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618d1812730 msgr2=0x5618d1bd3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:50.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.656+0000 7f666f7fe640 1 --2- 172.21.15.90:0/3531678064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618d1812730 0x5618d1bd3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.656+0000 7f666f7fe640 1 -- 172.21.15.90:0/3531678064 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5618d180a550 msgr2=0x5618d1bdcc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:50.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.656+0000 7f666f7fe640 1 --2- 172.21.15.90:0/3531678064 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5618d180a550 0x5618d1bdcc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.656+0000 7f666f7fe640 1 -- 172.21.15.90:0/3531678064 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618d1bdbeb0 con 0x5618d1833410 2024-04-25T11:44:50.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.657+0000 7f666f7fe640 1 --2- 172.21.15.90:0/3531678064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1833410 0x5618d1be9a00 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f666802a930 tx=0x7f6668004450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=644c5b3910c2389a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:50.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.657+0000 7f66557fa640 1 -- 172.21.15.90:0/3531678064 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666804c070 con 0x5618d1833410 2024-04-25T11:44:50.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.657+0000 7f66557fa640 1 -- 172.21.15.90:0/3531678064 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6668055030 con 0x5618d1833410 2024-04-25T11:44:50.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.657+0000 7f66557fa640 1 -- 172.21.15.90:0/3531678064 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6668035040 con 0x5618d1833410 2024-04-25T11:44:50.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.658+0000 7f667c9bb740 1 -- 172.21.15.90:0/3531678064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1833410 msgr2=0x5618d1be9a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.658+0000 7f667c9bb740 1 --2- 172.21.15.90:0/3531678064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1833410 0x5618d1be9a00 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f666802a930 tx=0x7f6668004450 comp rx=0 tx=0).stop 2024-04-25T11:44:50.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.658+0000 7f667c9bb740 1 -- 172.21.15.90:0/3531678064 shutdown_connections 2024-04-25T11:44:50.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.658+0000 7f667c9bb740 1 --2- 172.21.15.90:0/3531678064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1833410 0x5618d1be9a00 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.658+0000 7f667c9bb740 1 --2- 172.21.15.90:0/3531678064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618d1812730 0x5618d1bd3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.658+0000 7f667c9bb740 1 --2- 172.21.15.90:0/3531678064 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5618d180a550 0x5618d1bdcc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.658+0000 7f667c9bb740 1 -- 172.21.15.90:0/3531678064 >> 172.21.15.90:0/3531678064 conn(0x5618d1827560 msgr2=0x5618d1bd33b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:50.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.658+0000 7f667c9bb740 1 -- 172.21.15.90:0/3531678064 shutdown_connections 2024-04-25T11:44:50.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.658+0000 7f667c9bb740 1 -- 172.21.15.90:0/3531678064 wait complete. 2024-04-25T11:44:50.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.658+0000 7f667c9bb740 1 -- start start 2024-04-25T11:44:50.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.658+0000 7f667c9bb740 1 Processor -- start 2024-04-25T11:44:50.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.658+0000 7f667c9bb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618d180a550 0x5618d1be63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.659+0000 7f667c9bb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1812730 0x5618d1be3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.659+0000 7f667c9bb740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5618d1833410 0x5618d1be3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.659+0000 7f667c9bb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f66701b0f60 con 0x5618d1812730 2024-04-25T11:44:50.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.659+0000 7f667c9bb740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f66701b0890 con 0x5618d1833410 2024-04-25T11:44:50.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.659+0000 7f667c9bb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f66701adaf0 con 0x5618d180a550 2024-04-25T11:44:50.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.659+0000 7f667c9bb740 10 client.0 _ll_register_callbacks cb 0x7f6664009b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:50.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.659+0000 7f666e7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1812730 0x5618d1be3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.659+0000 7f666e7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1812730 0x5618d1be3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48754/0 (socket says 172.21.15.90:48754) 2024-04-25T11:44:50.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.659+0000 7f666e7fc640 1 -- 172.21.15.90:0/3988022094 learned_addr learned my addr 172.21.15.90:0/3988022094 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:50.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.659+0000 7f666f7fe640 1 --2- 172.21.15.90:0/3988022094 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5618d1833410 0x5618d1be3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.659+0000 7f666effd640 1 --2- 172.21.15.90:0/3988022094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618d180a550 0x5618d1be63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.659+0000 7f666e7fc640 1 -- 172.21.15.90:0/3988022094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618d180a550 msgr2=0x5618d1be63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:50.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.659+0000 7f666e7fc640 1 --2- 172.21.15.90:0/3988022094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618d180a550 0x5618d1be63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.659+0000 7f666e7fc640 1 -- 172.21.15.90:0/3988022094 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5618d1833410 msgr2=0x5618d1be3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:50.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.659+0000 7f666e7fc640 1 --2- 172.21.15.90:0/3988022094 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5618d1833410 0x5618d1be3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.659+0000 7f666e7fc640 1 -- 172.21.15.90:0/3988022094 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618d1be44a0 con 0x5618d1812730 2024-04-25T11:44:50.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.659+0000 7f666e7fc640 1 --2- 172.21.15.90:0/3988022094 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1812730 0x5618d1be3750 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f666400a010 tx=0x7f6664022700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f837aa2e3c444b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:50.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.659+0000 7f664effd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:50.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.659+0000 7f664effd640 1 -- 172.21.15.90:0/3988022094 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666402f060 con 0x5618d1812730 2024-04-25T11:44:50.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.659+0000 7f664effd640 1 -- 172.21.15.90:0/3988022094 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f6664022d00 con 0x5618d1812730 2024-04-25T11:44:50.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.659+0000 7f667c9bb740 1 -- 172.21.15.90:0/3988022094 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6668055030 con 0x5618d1812730 2024-04-25T11:44:50.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.659+0000 7f664effd640 1 -- 172.21.15.90:0/3988022094 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6664036050 con 0x5618d1812730 2024-04-25T11:44:50.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.659+0000 7f667c9bb740 10 client.18726 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:50.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.659+0000 7f667c9bb740 1 -- 172.21.15.90:0/3988022094 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5618d1c58430 con 0x5618d1812730 2024-04-25T11:44:50.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.660+0000 7f664effd640 1 -- 172.21.15.90:0/3988022094 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6668055030 con 0x5618d1812730 2024-04-25T11:44:50.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.660+0000 7f664effd640 1 -- 172.21.15.90:0/3988022094 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f666401a400 con 0x5618d1812730 2024-04-25T11:44:50.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.660+0000 7f667c9bb740 10 client.18726 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:50.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.660+0000 7f667c9bb740 10 client.18726 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:44:50.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.660+0000 7f667c9bb740 20 client.18726 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:50.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.660+0000 7f667c9bb740 1 --2- 172.21.15.90:0/3988022094 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5618d17c7630 0x5618d1c5abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.660+0000 7f667c9bb740 4 client.18726 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T11:44:50.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.660+0000 7f667c9bb740 1 -- 172.21.15.90:0/3988022094 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5618d1b41580 con 0x5618d17c7630 2024-04-25T11:44:50.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.660+0000 7f666effd640 1 --2- 172.21.15.90:0/3988022094 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5618d17c7630 0x5618d1c5abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.661+0000 7f664effd640 10 client.18726 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:50.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.661+0000 7f666effd640 1 --2- 172.21.15.90:0/3988022094 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5618d17c7630 0x5618d1c5abf0 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1449185d449939fd server_cookie=d786892c2518d9ee in_seq=0 out_seq=0 2024-04-25T11:44:50.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.662+0000 7f664effd640 1 -- 172.21.15.90:0/3988022094 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5618d1b41580 con 0x5618d17c7630 2024-04-25T11:44:50.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.662+0000 7f664effd640 10 client.18726 handle_command_reply: tid=0 2024-04-25T11:44:50.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.662+0000 7f667c9bb740 1 client.18726 shutdown 2024-04-25T11:44:50.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.662+0000 7f667c9bb740 1 -- 172.21.15.90:0/3988022094 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1812730 msgr2=0x5618d1be3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.662+0000 7f667c9bb740 1 --2- 172.21.15.90:0/3988022094 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1812730 0x5618d1be3750 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f666400a010 tx=0x7f6664022700 comp rx=0 tx=0).stop 2024-04-25T11:44:50.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.662+0000 7f667c9bb740 1 -- 172.21.15.90:0/3988022094 shutdown_connections 2024-04-25T11:44:50.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.662+0000 7f667c9bb740 1 --2- 172.21.15.90:0/3988022094 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5618d17c7630 0x5618d1c5abf0 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.662+0000 7f667c9bb740 1 --2- 172.21.15.90:0/3988022094 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5618d1833410 0x5618d1be3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.662+0000 7f667c9bb740 1 --2- 172.21.15.90:0/3988022094 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618d1812730 0x5618d1be3750 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.662+0000 7f667c9bb740 1 --2- 172.21.15.90:0/3988022094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618d180a550 0x5618d1be63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.662+0000 7f667c9bb740 1 -- 172.21.15.90:0/3988022094 >> 172.21.15.90:0/3988022094 conn(0x5618d1827560 msgr2=0x5618d1bdc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:50.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.662+0000 7f664effd640 0 client.18726 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:50.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.662+0000 7f667c9bb740 1 -- 172.21.15.90:0/3988022094 shutdown_connections 2024-04-25T11:44:50.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.662+0000 7f667c9bb740 1 -- 172.21.15.90:0/3988022094 wait complete. 2024-04-25T11:44:50.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.662+0000 7f667c9bb740 20 client.18726 trim_cache size 0 max 16384 2024-04-25T11:44:50.655 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:44:50.655 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-25T11:44:50.655 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "b5782ea6-7d5b-4b55-af57-07dab3a85149", 2024-04-25T11:44:50.655 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-25T11:44:50.655 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:44:50.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.663+0000 7f6675a27640 1 -- 172.21.15.90:0/1969669715 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f66580afc60 msgr2=0x7f66580b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.663+0000 7f6675a27640 1 --2- 172.21.15.90:0/1969669715 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f66580afc60 0x7f66580b2170 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6668049000 tx=0x7f6668046000 comp rx=0 tx=0).stop 2024-04-25T11:44:50.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.663+0000 7f6675a27640 1 -- 172.21.15.90:0/1969669715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6670102af0 msgr2=0x7f667019b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.663+0000 7f6675a27640 1 --2- 172.21.15.90:0/1969669715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6670102af0 0x7f667019b500 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f666000eb90 tx=0x7f666000c6a0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.663+0000 7f6675a27640 1 -- 172.21.15.90:0/1969669715 shutdown_connections 2024-04-25T11:44:50.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.663+0000 7f6675a27640 1 --2- 172.21.15.90:0/1969669715 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f66580afc60 0x7f66580b2170 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.663+0000 7f6675a27640 1 --2- 172.21.15.90:0/1969669715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f667010d380 0x7f667019bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.663+0000 7f6675a27640 1 --2- 172.21.15.90:0/1969669715 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6670108a20 0x7f667019ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.663+0000 7f6675a27640 1 --2- 172.21.15.90:0/1969669715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6670102af0 0x7f667019b500 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.663+0000 7f6675a27640 1 -- 172.21.15.90:0/1969669715 >> 172.21.15.90:0/1969669715 conn(0x7f6670078100 msgr2=0x7f66701040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:50.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.663+0000 7f6675a27640 1 -- 172.21.15.90:0/1969669715 shutdown_connections 2024-04-25T11:44:50.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.663+0000 7f6675a27640 1 -- 172.21.15.90:0/1969669715 wait complete. 2024-04-25T11:44:50.661 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:44:50.661 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:44:50.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.785+0000 7f54da1a8640 1 -- 172.21.15.90:0/2525200171 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54d4102af0 msgr2=0x7f54d4102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.785+0000 7f54da1a8640 1 --2- 172.21.15.90:0/2525200171 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54d4102af0 0x7f54d4102f00 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f54bc009900 tx=0x7f54bc02a410 comp rx=0 tx=0).stop 2024-04-25T11:44:50.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.785+0000 7f54da1a8640 1 -- 172.21.15.90:0/2525200171 shutdown_connections 2024-04-25T11:44:50.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.785+0000 7f54da1a8640 1 --2- 172.21.15.90:0/2525200171 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54d410d380 0x7f54d410f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.785+0000 7f54da1a8640 1 --2- 172.21.15.90:0/2525200171 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54d4102af0 0x7f54d4102f00 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.785+0000 7f54da1a8640 1 --2- 172.21.15.90:0/2525200171 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f54d4108a20 0x7f54d41025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.785+0000 7f54da1a8640 1 -- 172.21.15.90:0/2525200171 >> 172.21.15.90:0/2525200171 conn(0x7f54d4078100 msgr2=0x7f54d4078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:50.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.785+0000 7f54da1a8640 1 -- 172.21.15.90:0/2525200171 shutdown_connections 2024-04-25T11:44:50.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.785+0000 7f54da1a8640 1 -- 172.21.15.90:0/2525200171 wait complete. 2024-04-25T11:44:50.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.786+0000 7f54da1a8640 1 Processor -- start 2024-04-25T11:44:50.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.786+0000 7f54da1a8640 1 -- start start 2024-04-25T11:44:50.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.786+0000 7f54da1a8640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f54d4102af0 0x7f54d419b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.786+0000 7f54da1a8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54d4108a20 0x7f54d419baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.786+0000 7f54da1a8640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54d410d380 0x7f54d419bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.786+0000 7f54da1a8640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f54d4112be0 con 0x7f54d410d380 2024-04-25T11:44:50.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.786+0000 7f54da1a8640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f54d4112a60 con 0x7f54d4102af0 2024-04-25T11:44:50.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.786+0000 7f54da1a8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f54d4112d60 con 0x7f54d4108a20 2024-04-25T11:44:50.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.786+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54d410d380 0x7f54d419bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.786+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54d410d380 0x7f54d419bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48774/0 (socket says 172.21.15.90:48774) 2024-04-25T11:44:50.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.786+0000 7f54d3fff640 1 -- 172.21.15.90:0/3754110952 learned_addr learned my addr 172.21.15.90:0/3754110952 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:50.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.786+0000 7f54d37fe640 1 --2- 172.21.15.90:0/3754110952 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f54d4102af0 0x7f54d419b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.786+0000 7f54cbfff640 1 --2- 172.21.15.90:0/3754110952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54d4108a20 0x7f54d419baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.786+0000 7f54d3fff640 1 -- 172.21.15.90:0/3754110952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54d4108a20 msgr2=0x7f54d419baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.786+0000 7f54d3fff640 1 --2- 172.21.15.90:0/3754110952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54d4108a20 0x7f54d419baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.787+0000 7f54d3fff640 1 -- 172.21.15.90:0/3754110952 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f54d4102af0 msgr2=0x7f54d419b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.787+0000 7f54d3fff640 1 --2- 172.21.15.90:0/3754110952 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f54d4102af0 0x7f54d419b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.787+0000 7f54d3fff640 1 -- 172.21.15.90:0/3754110952 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54d41a67e0 con 0x7f54d410d380 2024-04-25T11:44:50.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.787+0000 7f54d37fe640 1 --2- 172.21.15.90:0/3754110952 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f54d4102af0 0x7f54d419b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:44:50.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.787+0000 7f54d3fff640 1 --2- 172.21.15.90:0/3754110952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54d410d380 0x7f54d419bfe0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f54cc00eaa0 tx=0x7f54cc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:50.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.787+0000 7f54d17fa640 1 -- 172.21.15.90:0/3754110952 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc00cdb0 con 0x7f54d410d380 2024-04-25T11:44:50.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.787+0000 7f54da1a8640 1 -- 172.21.15.90:0/3754110952 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54d41a6ad0 con 0x7f54d410d380 2024-04-25T11:44:50.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.787+0000 7f54d17fa640 1 -- 172.21.15.90:0/3754110952 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54cc005020 con 0x7f54d410d380 2024-04-25T11:44:50.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.787+0000 7f54d17fa640 1 -- 172.21.15.90:0/3754110952 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc0116d0 con 0x7f54d410d380 2024-04-25T11:44:50.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.787+0000 7f54da1a8640 1 -- 172.21.15.90:0/3754110952 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54d41a6ff0 con 0x7f54d410d380 2024-04-25T11:44:50.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.789+0000 7f54d17fa640 1 -- 172.21.15.90:0/3754110952 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f54cc0051d0 con 0x7f54d410d380 2024-04-25T11:44:50.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.789+0000 7f54e105c740 1 Processor -- start 2024-04-25T11:44:50.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.790+0000 7f54e105c740 1 -- start start 2024-04-25T11:44:50.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.790+0000 7f54e105c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562043afd390 0x562043e9da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.790+0000 7f54e105c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043ad44d0 0x562043e9dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.790+0000 7f54e105c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562043adc6b0 0x562043eaa8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.790+0000 7f54e105c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562043e03d80 con 0x562043ad44d0 2024-04-25T11:44:50.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.790+0000 7f54e105c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x562043dbd8b0 con 0x562043afd390 2024-04-25T11:44:50.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.790+0000 7f54e105c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562043dffd40 con 0x562043adc6b0 2024-04-25T11:44:50.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.790+0000 7f54d17fa640 1 --2- 172.21.15.90:0/3754110952 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f54c00afc60 0x7f54c00b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.790+0000 7f54d37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043ad44d0 0x562043e9dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.790+0000 7f54d37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043ad44d0 0x562043e9dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48788/0 (socket says 172.21.15.90:48788) 2024-04-25T11:44:50.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.790+0000 7f54d37fe640 1 -- 172.21.15.90:0/3101017037 learned_addr learned my addr 172.21.15.90:0/3101017037 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:50.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.790+0000 7f54d3fff640 1 --2- 172.21.15.90:0/3754110952 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f54c00afc60 0x7f54c00b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.790+0000 7f54cbfff640 1 --2- 172.21.15.90:0/3101017037 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562043afd390 0x562043e9da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.790+0000 7f54d17fa640 1 -- 172.21.15.90:0/3754110952 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f54cc0df2f0 con 0x7f54d410d380 2024-04-25T11:44:50.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.790+0000 7f54cbfff640 1 --2- 172.21.15.90:0/3101017037 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562043adc6b0 0x562043eaa8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.790+0000 7f54d37fe640 1 -- 172.21.15.90:0/3101017037 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562043adc6b0 msgr2=0x562043eaa8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:50.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.790+0000 7f54d37fe640 1 --2- 172.21.15.90:0/3101017037 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562043adc6b0 0x562043eaa8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.790+0000 7f54d37fe640 1 -- 172.21.15.90:0/3101017037 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562043afd390 msgr2=0x562043e9da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:50.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.790+0000 7f54d37fe640 1 --2- 172.21.15.90:0/3101017037 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562043afd390 0x562043e9da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.790+0000 7f54d37fe640 1 -- 172.21.15.90:0/3101017037 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562043d8ea40 con 0x562043ad44d0 2024-04-25T11:44:50.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.790+0000 7f54cbfff640 1 --2- 172.21.15.90:0/3101017037 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562043adc6b0 0x562043eaa8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:50.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.790+0000 7f54d37fe640 1 --2- 172.21.15.90:0/3101017037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043ad44d0 0x562043e9dfa0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f54c4008c10 tx=0x7f54c400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9819472ce6bd0191 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:50.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.791+0000 7f54d3fff640 1 --2- 172.21.15.90:0/3754110952 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f54c00afc60 0x7f54c00b2170 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f54cc0dca60 tx=0x7f54cc06dcc0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:50.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.791+0000 7f54c9ffb640 1 -- 172.21.15.90:0/3101017037 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c4014070 con 0x562043ad44d0 2024-04-25T11:44:50.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.791+0000 7f54c9ffb640 1 -- 172.21.15.90:0/3101017037 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54c40047a0 con 0x562043ad44d0 2024-04-25T11:44:50.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.791+0000 7f54c9ffb640 1 -- 172.21.15.90:0/3101017037 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c4010690 con 0x562043ad44d0 2024-04-25T11:44:50.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.791+0000 7f54e105c740 1 -- 172.21.15.90:0/3101017037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043ad44d0 msgr2=0x562043e9dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.791+0000 7f54e105c740 1 --2- 172.21.15.90:0/3101017037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043ad44d0 0x562043e9dfa0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f54c4008c10 tx=0x7f54c400e720 comp rx=0 tx=0).stop 2024-04-25T11:44:50.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.792+0000 7f54e105c740 1 -- 172.21.15.90:0/3101017037 shutdown_connections 2024-04-25T11:44:50.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.792+0000 7f54e105c740 1 --2- 172.21.15.90:0/3101017037 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562043adc6b0 0x562043eaa8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.792+0000 7f54e105c740 1 --2- 172.21.15.90:0/3101017037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043ad44d0 0x562043e9dfa0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.792+0000 7f54e105c740 1 --2- 172.21.15.90:0/3101017037 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562043afd390 0x562043e9da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.792+0000 7f54e105c740 1 -- 172.21.15.90:0/3101017037 >> 172.21.15.90:0/3101017037 conn(0x562043af14e0 msgr2=0x562043e9a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:50.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.792+0000 7f54e105c740 1 -- 172.21.15.90:0/3101017037 shutdown_connections 2024-04-25T11:44:50.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.792+0000 7f54e105c740 1 -- 172.21.15.90:0/3101017037 wait complete. 2024-04-25T11:44:50.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.792+0000 7f54e105c740 1 -- start start 2024-04-25T11:44:50.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.792+0000 7f54e105c740 1 Processor -- start 2024-04-25T11:44:50.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.792+0000 7f54e105c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562043ad44d0 0x562043ea8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.792+0000 7f54e105c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043adc6b0 0x562043ea8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.792+0000 7f54e105c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562043afd390 0x562043eb4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.792+0000 7f54e105c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562043eaafe0 con 0x562043adc6b0 2024-04-25T11:44:50.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.792+0000 7f54e105c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x562043d8ad10 con 0x562043afd390 2024-04-25T11:44:50.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.792+0000 7f54e105c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562043eab160 con 0x562043ad44d0 2024-04-25T11:44:50.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.792+0000 7f54e105c740 10 client.0 _ll_register_callbacks cb 0x562043db7210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:50.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.793+0000 7f54d37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043adc6b0 0x562043ea8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.793+0000 7f54d37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043adc6b0 0x562043ea8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48796/0 (socket says 172.21.15.90:48796) 2024-04-25T11:44:50.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.793+0000 7f54d37fe640 1 -- 172.21.15.90:0/2379103320 learned_addr learned my addr 172.21.15.90:0/2379103320 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:50.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.793+0000 7f54cbfff640 1 --2- 172.21.15.90:0/2379103320 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562043afd390 0x562043eb4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.793+0000 7f54cbfff640 1 --2- 172.21.15.90:0/2379103320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562043ad44d0 0x562043ea8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.793+0000 7f54d37fe640 1 -- 172.21.15.90:0/2379103320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562043ad44d0 msgr2=0x562043ea8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:50.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.793+0000 7f54d37fe640 1 --2- 172.21.15.90:0/2379103320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562043ad44d0 0x562043ea8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.793+0000 7f54d37fe640 1 -- 172.21.15.90:0/2379103320 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562043afd390 msgr2=0x562043eb4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:50.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.793+0000 7f54d37fe640 1 --2- 172.21.15.90:0/2379103320 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562043afd390 0x562043eb4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.793+0000 7f54d37fe640 1 -- 172.21.15.90:0/2379103320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562043bdf7d0 con 0x562043adc6b0 2024-04-25T11:44:50.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.793+0000 7f54d37fe640 1 --2- 172.21.15.90:0/2379103320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043adc6b0 0x562043ea8a40 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f54c4000c00 tx=0x7f54c4005210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4cf97997e982587 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:50.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.793+0000 7f54b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:50.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.793+0000 7f54b2ffd640 1 -- 172.21.15.90:0/2379103320 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c4014030 con 0x562043adc6b0 2024-04-25T11:44:50.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.793+0000 7f54b2ffd640 1 -- 172.21.15.90:0/2379103320 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f54c4004050 con 0x562043adc6b0 2024-04-25T11:44:50.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.793+0000 7f54e105c740 1 -- 172.21.15.90:0/2379103320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54c40047a0 con 0x562043adc6b0 2024-04-25T11:44:50.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.793+0000 7f54b2ffd640 1 -- 172.21.15.90:0/2379103320 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f54c4004df0 con 0x562043adc6b0 2024-04-25T11:44:50.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.793+0000 7f54e105c740 10 client.18738 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:50.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.793+0000 7f54e105c740 1 -- 172.21.15.90:0/2379103320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562043b153f0 con 0x562043adc6b0 2024-04-25T11:44:50.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.794+0000 7f54b2ffd640 1 -- 172.21.15.90:0/2379103320 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54c40047a0 con 0x562043adc6b0 2024-04-25T11:44:50.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.794+0000 7f54b2ffd640 1 -- 172.21.15.90:0/2379103320 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f54c402a440 con 0x562043adc6b0 2024-04-25T11:44:50.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.794+0000 7f54e105c740 10 client.18738 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:50.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.794+0000 7f54e105c740 10 client.18738 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:44:50.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.794+0000 7f54e105c740 20 client.18738 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:50.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.794+0000 7f54e105c740 1 --2- 172.21.15.90:0/2379103320 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562043abbf60 0x562043eba4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.794+0000 7f54e105c740 4 client.18738 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:44:50.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.794+0000 7f54e105c740 1 -- 172.21.15.90:0/2379103320 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562043e11990 con 0x562043abbf60 2024-04-25T11:44:50.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.794+0000 7f54cbfff640 1 --2- 172.21.15.90:0/2379103320 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562043abbf60 0x562043eba4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.794+0000 7f54b2ffd640 10 client.18738 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:50.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.794+0000 7f54cbfff640 1 --2- 172.21.15.90:0/2379103320 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562043abbf60 0x562043eba4b0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8caff84936ed7bf4 server_cookie=1a74393d1ee36111 in_seq=0 out_seq=0 2024-04-25T11:44:50.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.795+0000 7f54b2ffd640 1 -- 172.21.15.90:0/2379103320 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562043e11990 con 0x562043abbf60 2024-04-25T11:44:50.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.795+0000 7f54b2ffd640 10 client.18738 handle_command_reply: tid=0 2024-04-25T11:44:50.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.795+0000 7f54e105c740 1 client.18738 shutdown 2024-04-25T11:44:50.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.795+0000 7f54e105c740 1 -- 172.21.15.90:0/2379103320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043adc6b0 msgr2=0x562043ea8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.795+0000 7f54e105c740 1 --2- 172.21.15.90:0/2379103320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043adc6b0 0x562043ea8a40 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f54c4000c00 tx=0x7f54c4005210 comp rx=0 tx=0).stop 2024-04-25T11:44:50.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.795+0000 7f54e105c740 1 -- 172.21.15.90:0/2379103320 shutdown_connections 2024-04-25T11:44:50.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.795+0000 7f54e105c740 1 --2- 172.21.15.90:0/2379103320 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562043abbf60 0x562043eba4b0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.795+0000 7f54e105c740 1 --2- 172.21.15.90:0/2379103320 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562043afd390 0x562043eb4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.795+0000 7f54b2ffd640 0 client.18738 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:50.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.795+0000 7f54e105c740 1 --2- 172.21.15.90:0/2379103320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043adc6b0 0x562043ea8a40 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.795+0000 7f54e105c740 1 --2- 172.21.15.90:0/2379103320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562043ad44d0 0x562043ea8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.795+0000 7f54e105c740 1 -- 172.21.15.90:0/2379103320 >> 172.21.15.90:0/2379103320 conn(0x562043af14e0 msgr2=0x562043e9f780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:50.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.795+0000 7f54e105c740 1 -- 172.21.15.90:0/2379103320 shutdown_connections 2024-04-25T11:44:50.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.795+0000 7f54e105c740 1 -- 172.21.15.90:0/2379103320 wait complete. 2024-04-25T11:44:50.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.796+0000 7f54e105c740 20 client.18738 trim_cache size 0 max 16384 2024-04-25T11:44:50.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.813+0000 7f54e105c740 1 Processor -- start 2024-04-25T11:44:50.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.813+0000 7f54e105c740 1 -- start start 2024-04-25T11:44:50.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.813+0000 7f54e105c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562043ad44d0 0x562043ea27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.813+0000 7f54e105c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562043adc6b0 0x562043ebfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.813+0000 7f54e105c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043afd390 0x562043ec0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.813+0000 7f54e105c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f54d41afc70 con 0x562043afd390 2024-04-25T11:44:50.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.813+0000 7f54e105c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f54d41b17b0 con 0x562043ad44d0 2024-04-25T11:44:50.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.813+0000 7f54e105c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f54d41b0340 con 0x562043adc6b0 2024-04-25T11:44:50.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.813+0000 7f54cbfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043afd390 0x562043ec0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.813+0000 7f54cbfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043afd390 0x562043ec0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48798/0 (socket says 172.21.15.90:48798) 2024-04-25T11:44:50.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.813+0000 7f54cbfff640 1 -- 172.21.15.90:0/1697036504 learned_addr learned my addr 172.21.15.90:0/1697036504 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:50.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.814+0000 7f54cbfff640 1 --2- 172.21.15.90:0/1697036504 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562043ad44d0 0x562043ea27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.814+0000 7f54cbfff640 1 -- 172.21.15.90:0/1697036504 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562043adc6b0 msgr2=0x562043ebfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:50.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.814+0000 7f54cbfff640 1 --2- 172.21.15.90:0/1697036504 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562043adc6b0 0x562043ebfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.814+0000 7f54cbfff640 1 -- 172.21.15.90:0/1697036504 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562043ad44d0 msgr2=0x562043ea27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:50.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.814+0000 7f54cbfff640 1 --2- 172.21.15.90:0/1697036504 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562043ad44d0 0x562043ea27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.814+0000 7f54cbfff640 1 -- 172.21.15.90:0/1697036504 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562043ad50f0 con 0x562043afd390 2024-04-25T11:44:50.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.814+0000 7f54cbfff640 1 --2- 172.21.15.90:0/1697036504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043afd390 0x562043ec0080 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f54bc000c00 tx=0x7f54bc0095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d0c005969d18c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:50.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.814+0000 7f54c97fa640 1 -- 172.21.15.90:0/1697036504 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54bc043070 con 0x562043afd390 2024-04-25T11:44:50.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.814+0000 7f54c97fa640 1 -- 172.21.15.90:0/1697036504 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54bc04b030 con 0x562043afd390 2024-04-25T11:44:50.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.814+0000 7f54c97fa640 1 -- 172.21.15.90:0/1697036504 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54bc046040 con 0x562043afd390 2024-04-25T11:44:50.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.815+0000 7f54e105c740 1 -- 172.21.15.90:0/1697036504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043afd390 msgr2=0x562043ec0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.815+0000 7f54e105c740 1 --2- 172.21.15.90:0/1697036504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043afd390 0x562043ec0080 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f54bc000c00 tx=0x7f54bc0095e0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.815+0000 7f54e105c740 1 -- 172.21.15.90:0/1697036504 shutdown_connections 2024-04-25T11:44:50.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.815+0000 7f54e105c740 1 --2- 172.21.15.90:0/1697036504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043afd390 0x562043ec0080 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.815+0000 7f54e105c740 1 --2- 172.21.15.90:0/1697036504 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562043adc6b0 0x562043ebfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.815+0000 7f54e105c740 1 --2- 172.21.15.90:0/1697036504 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562043ad44d0 0x562043ea27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.815+0000 7f54e105c740 1 -- 172.21.15.90:0/1697036504 >> 172.21.15.90:0/1697036504 conn(0x562043af14e0 msgr2=0x562043ea20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:50.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.815+0000 7f54e105c740 1 -- 172.21.15.90:0/1697036504 shutdown_connections 2024-04-25T11:44:50.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.815+0000 7f54e105c740 1 -- 172.21.15.90:0/1697036504 wait complete. 2024-04-25T11:44:50.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.815+0000 7f54e105c740 1 -- start start 2024-04-25T11:44:50.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.815+0000 7f54e105c740 1 Processor -- start 2024-04-25T11:44:50.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.816+0000 7f54e105c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562043ad44d0 0x562043eb0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.816+0000 7f54e105c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043adc6b0 0x562043eb0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.816+0000 7f54e105c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562043afd390 0x562043eaba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.816+0000 7f54e105c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f54d41b10e0 con 0x562043adc6b0 2024-04-25T11:44:50.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.816+0000 7f54e105c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f54d41b0a10 con 0x562043ad44d0 2024-04-25T11:44:50.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.816+0000 7f54e105c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f54d41adc70 con 0x562043afd390 2024-04-25T11:44:50.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.816+0000 7f54e105c740 10 client.0 _ll_register_callbacks cb 0x7f54bc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:50.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.816+0000 7f54d37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043adc6b0 0x562043eb0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.816+0000 7f54d37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043adc6b0 0x562043eb0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48802/0 (socket says 172.21.15.90:48802) 2024-04-25T11:44:50.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.816+0000 7f54d37fe640 1 -- 172.21.15.90:0/4016743937 learned_addr learned my addr 172.21.15.90:0/4016743937 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:50.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.816+0000 7f54cbfff640 1 --2- 172.21.15.90:0/4016743937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562043afd390 0x562043eaba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.816+0000 7f54cbfff640 1 --2- 172.21.15.90:0/4016743937 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562043ad44d0 0x562043eb0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.816+0000 7f54d37fe640 1 -- 172.21.15.90:0/4016743937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562043afd390 msgr2=0x562043eaba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:50.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.816+0000 7f54d37fe640 1 --2- 172.21.15.90:0/4016743937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562043afd390 0x562043eaba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.816+0000 7f54d37fe640 1 -- 172.21.15.90:0/4016743937 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562043ad44d0 msgr2=0x562043eb0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:50.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.816+0000 7f54d37fe640 1 --2- 172.21.15.90:0/4016743937 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562043ad44d0 0x562043eb0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.816+0000 7f54d37fe640 1 -- 172.21.15.90:0/4016743937 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562043dc44b0 con 0x562043adc6b0 2024-04-25T11:44:50.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.816+0000 7f54cbfff640 1 --2- 172.21.15.90:0/4016743937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562043afd390 0x562043eaba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:50.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.816+0000 7f54d37fe640 1 --2- 172.21.15.90:0/4016743937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043adc6b0 0x562043eb0a70 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f54c400f010 tx=0x7f54c400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f892a9f6676215dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:50.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.816+0000 7f54b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:50.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.816+0000 7f54b2ffd640 1 -- 172.21.15.90:0/4016743937 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c4014030 con 0x562043adc6b0 2024-04-25T11:44:50.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.816+0000 7f54b2ffd640 1 -- 172.21.15.90:0/4016743937 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f54c402ac60 con 0x562043adc6b0 2024-04-25T11:44:50.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.816+0000 7f54e105c740 1 -- 172.21.15.90:0/4016743937 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54bc04b030 con 0x562043adc6b0 2024-04-25T11:44:50.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.817+0000 7f54b2ffd640 1 -- 172.21.15.90:0/4016743937 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f54c4008a90 con 0x562043adc6b0 2024-04-25T11:44:50.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.817+0000 7f54e105c740 10 client.18744 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:50.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.817+0000 7f54e105c740 1 -- 172.21.15.90:0/4016743937 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562043f21fa0 con 0x562043adc6b0 2024-04-25T11:44:50.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.817+0000 7f54b2ffd640 1 -- 172.21.15.90:0/4016743937 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54bc04b030 con 0x562043adc6b0 2024-04-25T11:44:50.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.817+0000 7f54b2ffd640 1 -- 172.21.15.90:0/4016743937 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f54c40037f0 con 0x562043adc6b0 2024-04-25T11:44:50.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.817+0000 7f54e105c740 10 client.18744 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:50.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.817+0000 7f54e105c740 10 client.18744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:44:50.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.817+0000 7f54e105c740 20 client.18744 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:50.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.817+0000 7f54e105c740 1 --2- 172.21.15.90:0/4016743937 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562043a913a0 0x562043f246a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.817+0000 7f54e105c740 4 client.18744 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:44:50.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.817+0000 7f54e105c740 1 -- 172.21.15.90:0/4016743937 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562043e0b800 con 0x562043a913a0 2024-04-25T11:44:50.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.817+0000 7f54cbfff640 1 --2- 172.21.15.90:0/4016743937 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562043a913a0 0x562043f246a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.818+0000 7f54b2ffd640 10 client.18744 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:50.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.818+0000 7f54cbfff640 1 --2- 172.21.15.90:0/4016743937 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562043a913a0 0x562043f246a0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dabc490d13d2359d server_cookie=bee6684a15fb160b in_seq=0 out_seq=0 2024-04-25T11:44:50.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.818+0000 7f54b2ffd640 1 -- 172.21.15.90:0/4016743937 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x562043e0b800 con 0x562043a913a0 2024-04-25T11:44:50.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.818+0000 7f54b2ffd640 10 client.18744 handle_command_reply: tid=0 2024-04-25T11:44:50.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.818+0000 7f54e105c740 1 client.18744 shutdown 2024-04-25T11:44:50.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.818+0000 7f54e105c740 1 -- 172.21.15.90:0/4016743937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043adc6b0 msgr2=0x562043eb0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.818+0000 7f54e105c740 1 --2- 172.21.15.90:0/4016743937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043adc6b0 0x562043eb0a70 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f54c400f010 tx=0x7f54c400eca0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.818+0000 7f54e105c740 1 -- 172.21.15.90:0/4016743937 shutdown_connections 2024-04-25T11:44:50.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.818+0000 7f54e105c740 1 --2- 172.21.15.90:0/4016743937 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562043a913a0 0x562043f246a0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.818+0000 7f54e105c740 1 --2- 172.21.15.90:0/4016743937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562043afd390 0x562043eaba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.818+0000 7f54e105c740 1 --2- 172.21.15.90:0/4016743937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562043adc6b0 0x562043eb0a70 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.818+0000 7f54b2ffd640 0 client.18744 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:50.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.818+0000 7f54e105c740 1 --2- 172.21.15.90:0/4016743937 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562043ad44d0 0x562043eb0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.818+0000 7f54e105c740 1 -- 172.21.15.90:0/4016743937 >> 172.21.15.90:0/4016743937 conn(0x562043af14e0 msgr2=0x562043ebfcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:50.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.818+0000 7f54e105c740 1 -- 172.21.15.90:0/4016743937 shutdown_connections 2024-04-25T11:44:50.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.818+0000 7f54e105c740 1 -- 172.21.15.90:0/4016743937 wait complete. 2024-04-25T11:44:50.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.818+0000 7f54e105c740 20 client.18744 trim_cache size 0 max 16384 2024-04-25T11:44:50.812 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:44:50.812 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T11:44:50.812 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-25T11:44:50.812 INFO:teuthology.orchestra.run.smithi090.stdout: "b5782ea6-7d5b-4b55-af57-07dab3a85149": { 2024-04-25T11:44:50.812 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-25T11:44:50.812 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "b5782ea6-7d5b-4b55-af57-07dab3a85149", 2024-04-25T11:44:50.812 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-25T11:44:50.813 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:44:50.813 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:44:50.813 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:44:50.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.819+0000 7f54da1a8640 1 -- 172.21.15.90:0/3754110952 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f54c00afc60 msgr2=0x7f54c00b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.819+0000 7f54da1a8640 1 --2- 172.21.15.90:0/3754110952 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f54c00afc60 0x7f54c00b2170 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f54cc0dca60 tx=0x7f54cc06dcc0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.819+0000 7f54da1a8640 1 -- 172.21.15.90:0/3754110952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54d410d380 msgr2=0x7f54d419bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.819+0000 7f54da1a8640 1 --2- 172.21.15.90:0/3754110952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54d410d380 0x7f54d419bfe0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f54cc00eaa0 tx=0x7f54cc00ef60 comp rx=0 tx=0).stop 2024-04-25T11:44:50.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.819+0000 7f54da1a8640 1 -- 172.21.15.90:0/3754110952 shutdown_connections 2024-04-25T11:44:50.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.819+0000 7f54da1a8640 1 --2- 172.21.15.90:0/3754110952 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f54c00afc60 0x7f54c00b2170 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.819+0000 7f54da1a8640 1 --2- 172.21.15.90:0/3754110952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54d410d380 0x7f54d419bfe0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.819+0000 7f54da1a8640 1 --2- 172.21.15.90:0/3754110952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54d4108a20 0x7f54d419baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.819+0000 7f54da1a8640 1 --2- 172.21.15.90:0/3754110952 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f54d4102af0 0x7f54d419b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.819+0000 7f54da1a8640 1 -- 172.21.15.90:0/3754110952 >> 172.21.15.90:0/3754110952 conn(0x7f54d4078100 msgr2=0x7f54d41040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:50.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.819+0000 7f54da1a8640 1 -- 172.21.15.90:0/3754110952 shutdown_connections 2024-04-25T11:44:50.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:44:50.819+0000 7f54da1a8640 1 -- 172.21.15.90:0/3754110952 wait complete. 2024-04-25T11:44:50.817 INFO:tasks.cephfs.filesystem:scrub status for tag:b5782ea6-7d5b-4b55-af57-07dab3a85149 - {'path': '/', 'tag': 'b5782ea6-7d5b-4b55-af57-07dab3a85149', 'options': 'recursive,force'} 2024-04-25T11:44:51.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.165+0000 7f8bdd79e640 1 -- 172.21.15.90:0/815147130 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8bd8102430 msgr2=0x7f8bd8102840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:51.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.165+0000 7f8bdd79e640 1 --2- 172.21.15.90:0/815147130 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8bd8102430 0x7f8bd8102840 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f8bc800adb0 tx=0x7f8bc802b7a0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.165+0000 7f8bdd79e640 1 -- 172.21.15.90:0/815147130 shutdown_connections 2024-04-25T11:44:51.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.165+0000 7f8bdd79e640 1 --2- 172.21.15.90:0/815147130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bd8107480 0x7f8bd810ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.165+0000 7f8bdd79e640 1 --2- 172.21.15.90:0/815147130 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8bd81069c0 0x7f8bd8106df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.165+0000 7f8bdd79e640 1 --2- 172.21.15.90:0/815147130 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8bd8102430 0x7f8bd8102840 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.165+0000 7f8bdd79e640 1 -- 172.21.15.90:0/815147130 >> 172.21.15.90:0/815147130 conn(0x7f8bd80fd8f0 msgr2=0x7f8bd80ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:51.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.165+0000 7f8bdd79e640 1 -- 172.21.15.90:0/815147130 shutdown_connections 2024-04-25T11:44:51.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.165+0000 7f8bdd79e640 1 -- 172.21.15.90:0/815147130 wait complete. 2024-04-25T11:44:51.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.166+0000 7f8bdd79e640 1 Processor -- start 2024-04-25T11:44:51.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.166+0000 7f8bdd79e640 1 -- start start 2024-04-25T11:44:51.155 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.166+0000 7f8bdd79e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8bd8102430 0x7f8bd819a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:51.155 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.167+0000 7f8bdd79e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8bd81069c0 0x7f8bd819acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:51.155 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.167+0000 7f8bdd79e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bd8107480 0x7f8bd819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.167+0000 7f8bdd79e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd8111b70 con 0x7f8bd81069c0 2024-04-25T11:44:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.167+0000 7f8bdd79e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd81119f0 con 0x7f8bd8102430 2024-04-25T11:44:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.167+0000 7f8bdd79e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd8111cf0 con 0x7f8bd8107480 2024-04-25T11:44:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.167+0000 7f8bd67fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8bd81069c0 0x7f8bd819acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.167+0000 7f8bd67fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8bd81069c0 0x7f8bd819acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48822/0 (socket says 172.21.15.90:48822) 2024-04-25T11:44:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.167+0000 7f8bd67fc640 1 -- 172.21.15.90:0/2767668326 learned_addr learned my addr 172.21.15.90:0/2767668326 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.167+0000 7f8bd6ffd640 1 --2- 172.21.15.90:0/2767668326 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8bd8102430 0x7f8bd819a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.167+0000 7f8bd77fe640 1 --2- 172.21.15.90:0/2767668326 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bd8107480 0x7f8bd819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.167+0000 7f8bd67fc640 1 -- 172.21.15.90:0/2767668326 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bd8107480 msgr2=0x7f8bd819efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.167+0000 7f8bd67fc640 1 --2- 172.21.15.90:0/2767668326 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bd8107480 0x7f8bd819efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.167+0000 7f8bd67fc640 1 -- 172.21.15.90:0/2767668326 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8bd8102430 msgr2=0x7f8bd819a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.167+0000 7f8bd67fc640 1 --2- 172.21.15.90:0/2767668326 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8bd8102430 0x7f8bd819a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.167+0000 7f8bd67fc640 1 -- 172.21.15.90:0/2767668326 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd819f5f0 con 0x7f8bd81069c0 2024-04-25T11:44:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.167+0000 7f8bd77fe640 1 --2- 172.21.15.90:0/2767668326 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bd8107480 0x7f8bd819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:44:51.157 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.167+0000 7f8bd67fc640 1 --2- 172.21.15.90:0/2767668326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8bd81069c0 0x7f8bd819acf0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc00d770 tx=0x7f8bcc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:51.157 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.167+0000 7f8bbffff640 1 -- 172.21.15.90:0/2767668326 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc017070 con 0x7f8bd81069c0 2024-04-25T11:44:51.157 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.167+0000 7f8bbffff640 1 -- 172.21.15.90:0/2767668326 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8bcc009c30 con 0x7f8bd81069c0 2024-04-25T11:44:51.157 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.167+0000 7f8bbffff640 1 -- 172.21.15.90:0/2767668326 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc00f620 con 0x7f8bd81069c0 2024-04-25T11:44:51.157 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.167+0000 7f8bdd79e640 1 -- 172.21.15.90:0/2767668326 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bd819f8e0 con 0x7f8bd81069c0 2024-04-25T11:44:51.157 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.167+0000 7f8bdd79e640 1 -- 172.21.15.90:0/2767668326 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bd81a7260 con 0x7f8bd81069c0 2024-04-25T11:44:51.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.170+0000 7f8bbffff640 1 -- 172.21.15.90:0/2767668326 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f8bcc01a070 con 0x7f8bd81069c0 2024-04-25T11:44:51.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.170+0000 7f8be464b740 1 Processor -- start 2024-04-25T11:44:51.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.170+0000 7f8be464b740 1 -- start start 2024-04-25T11:44:51.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.170+0000 7f8be464b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55652af521b0 0x55652b2f1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:51.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.170+0000 7f8be464b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55652af292f0 0x55652b2f2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:51.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.170+0000 7f8be464b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af314d0 0x55652b2f2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:51.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.170+0000 7f8be464b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55652b258460 con 0x55652af314d0 2024-04-25T11:44:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.170+0000 7f8be464b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55652b212590 con 0x55652af521b0 2024-04-25T11:44:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.170+0000 7f8be464b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55652b254420 con 0x55652af292f0 2024-04-25T11:44:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.171+0000 7f8bd67fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af314d0 0x55652b2f2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.171+0000 7f8bd67fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af314d0 0x55652b2f2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48836/0 (socket says 172.21.15.90:48836) 2024-04-25T11:44:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.171+0000 7f8bd67fc640 1 -- 172.21.15.90:0/1352649157 learned_addr learned my addr 172.21.15.90:0/1352649157 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.171+0000 7f8bd77fe640 1 --2- 172.21.15.90:0/1352649157 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55652af521b0 0x55652b2f1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.171+0000 7f8bbffff640 1 --2- 172.21.15.90:0/2767668326 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8bc00afc50 0x7f8bc00b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.171+0000 7f8bd6ffd640 1 --2- 172.21.15.90:0/1352649157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55652af292f0 0x55652b2f2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.171+0000 7f8bd67fc640 1 -- 172.21.15.90:0/1352649157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55652af292f0 msgr2=0x55652b2f2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.171+0000 7f8bd67fc640 1 --2- 172.21.15.90:0/1352649157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55652af292f0 0x55652b2f2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.171+0000 7f8bd67fc640 1 -- 172.21.15.90:0/1352649157 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55652af521b0 msgr2=0x55652b2f1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.171+0000 7f8bd67fc640 1 --2- 172.21.15.90:0/1352649157 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55652af521b0 0x55652b2f1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.171+0000 7f8bd67fc640 1 -- 172.21.15.90:0/1352649157 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55652b21f0d0 con 0x55652af314d0 2024-04-25T11:44:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.171+0000 7f8bd77fe640 1 --2- 172.21.15.90:0/1352649157 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55652af521b0 0x55652b2f1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:44:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.171+0000 7f8bd77fe640 1 --2- 172.21.15.90:0/2767668326 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8bc00afc50 0x7f8bc00b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.171+0000 7f8bbffff640 1 -- 172.21.15.90:0/2767668326 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f8bcc0dec10 con 0x7f8bd81069c0 2024-04-25T11:44:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.171+0000 7f8bd67fc640 1 --2- 172.21.15.90:0/1352649157 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af314d0 0x55652b2f2570 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc0dc240 tx=0x7f8bcc09b610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cd7c87d81f8ee7c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.171+0000 7f8bbd7fa640 1 -- 172.21.15.90:0/1352649157 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc08f070 con 0x55652af314d0 2024-04-25T11:44:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.171+0000 7f8bd77fe640 1 --2- 172.21.15.90:0/2767668326 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8bc00afc50 0x7f8bc00b2160 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55652b2eec60 tx=0x7f8bd00047d0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.171+0000 7f8bbd7fa640 1 -- 172.21.15.90:0/1352649157 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8bcc0a6830 con 0x55652af314d0 2024-04-25T11:44:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.171+0000 7f8bbd7fa640 1 -- 172.21.15.90:0/1352649157 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc0745f0 con 0x55652af314d0 2024-04-25T11:44:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.172+0000 7f8be464b740 1 -- 172.21.15.90:0/1352649157 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af314d0 msgr2=0x55652b2f2570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.172+0000 7f8be464b740 1 --2- 172.21.15.90:0/1352649157 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af314d0 0x55652b2f2570 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc0dc240 tx=0x7f8bcc09b610 comp rx=0 tx=0).stop 2024-04-25T11:44:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.172+0000 7f8be464b740 1 -- 172.21.15.90:0/1352649157 shutdown_connections 2024-04-25T11:44:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.172+0000 7f8be464b740 1 --2- 172.21.15.90:0/1352649157 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af314d0 0x55652b2f2570 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.172+0000 7f8be464b740 1 --2- 172.21.15.90:0/1352649157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55652af292f0 0x55652b2f2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.172+0000 7f8be464b740 1 --2- 172.21.15.90:0/1352649157 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55652af521b0 0x55652b2f1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.172+0000 7f8be464b740 1 -- 172.21.15.90:0/1352649157 >> 172.21.15.90:0/1352649157 conn(0x55652af46300 msgr2=0x55652b2ee2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.172+0000 7f8be464b740 1 -- 172.21.15.90:0/1352649157 shutdown_connections 2024-04-25T11:44:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.172+0000 7f8be464b740 1 -- 172.21.15.90:0/1352649157 wait complete. 2024-04-25T11:44:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.173+0000 7f8be464b740 1 -- start start 2024-04-25T11:44:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.173+0000 7f8be464b740 1 Processor -- start 2024-04-25T11:44:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.173+0000 7f8be464b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55652af292f0 0x55652b2fc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.173+0000 7f8be464b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af314d0 0x55652b2fc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.173+0000 7f8be464b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55652af521b0 0x55652b3020b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.173+0000 7f8be464b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55652b2febc0 con 0x55652af314d0 2024-04-25T11:44:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.173+0000 7f8be464b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55652b1dfb60 con 0x55652af521b0 2024-04-25T11:44:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.173+0000 7f8be464b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55652b2fed40 con 0x55652af292f0 2024-04-25T11:44:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.173+0000 7f8be464b740 10 client.0 _ll_register_callbacks cb 0x55652b2042b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:51.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.173+0000 7f8bd67fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af314d0 0x55652b2fc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:51.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.173+0000 7f8bd67fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af314d0 0x55652b2fc6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48848/0 (socket says 172.21.15.90:48848) 2024-04-25T11:44:51.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.173+0000 7f8bd67fc640 1 -- 172.21.15.90:0/625727753 learned_addr learned my addr 172.21.15.90:0/625727753 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:51.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.173+0000 7f8bd6ffd640 1 --2- 172.21.15.90:0/625727753 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55652af292f0 0x55652b2fc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:51.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.173+0000 7f8bd67fc640 1 -- 172.21.15.90:0/625727753 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55652af292f0 msgr2=0x55652b2fc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:51.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.173+0000 7f8bd67fc640 1 --2- 172.21.15.90:0/625727753 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55652af292f0 0x55652b2fc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.173+0000 7f8bd67fc640 1 -- 172.21.15.90:0/625727753 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55652af521b0 msgr2=0x55652b3020b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:51.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.173+0000 7f8bd67fc640 1 --2- 172.21.15.90:0/625727753 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55652af521b0 0x55652b3020b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.173+0000 7f8bd67fc640 1 -- 172.21.15.90:0/625727753 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55652af6a210 con 0x55652af314d0 2024-04-25T11:44:51.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.173+0000 7f8bd67fc640 1 --2- 172.21.15.90:0/625727753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af314d0 0x55652b2fc6f0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc0dc8d0 tx=0x7f8bcc00df10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f59bb97d2ef2a4d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:51.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.174+0000 7f8bbaff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:51.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.174+0000 7f8bbaff5640 1 -- 172.21.15.90:0/625727753 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc021030 con 0x55652af314d0 2024-04-25T11:44:51.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.174+0000 7f8bbaff5640 1 -- 172.21.15.90:0/625727753 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f8bcc0a2fb0 con 0x55652af314d0 2024-04-25T11:44:51.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.174+0000 7f8be464b740 1 -- 172.21.15.90:0/625727753 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8bcc0a6830 con 0x55652af314d0 2024-04-25T11:44:51.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.174+0000 7f8bbaff5640 1 -- 172.21.15.90:0/625727753 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8bcc08f040 con 0x55652af314d0 2024-04-25T11:44:51.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.174+0000 7f8be464b740 10 client.18756 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:51.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.174+0000 7f8be464b740 1 -- 172.21.15.90:0/625727753 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55652b036620 con 0x55652af314d0 2024-04-25T11:44:51.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.174+0000 7f8bbaff5640 1 -- 172.21.15.90:0/625727753 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8bcc0a6830 con 0x55652af314d0 2024-04-25T11:44:51.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.174+0000 7f8bbaff5640 1 -- 172.21.15.90:0/625727753 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f8bcc0a6f80 con 0x55652af314d0 2024-04-25T11:44:51.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.174+0000 7f8be464b740 10 client.18756 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:51.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.174+0000 7f8be464b740 10 client.18756 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:44:51.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.174+0000 7f8be464b740 20 client.18756 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:51.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.174+0000 7f8be464b740 1 --2- 172.21.15.90:0/625727753 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55652af10d80 0x55652b30e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:51.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.174+0000 7f8be464b740 4 client.18756 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:44:51.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.174+0000 7f8be464b740 1 -- 172.21.15.90:0/625727753 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55652b266070 con 0x55652af10d80 2024-04-25T11:44:51.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.175+0000 7f8bd6ffd640 1 --2- 172.21.15.90:0/625727753 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55652af10d80 0x55652b30e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:51.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.175+0000 7f8bbaff5640 10 client.18756 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:51.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.175+0000 7f8bd6ffd640 1 --2- 172.21.15.90:0/625727753 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55652af10d80 0x55652b30e550 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8bc22f54c45c46d3 server_cookie=afbf65f3be81d459 in_seq=0 out_seq=0 2024-04-25T11:44:51.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.176+0000 7f8bbaff5640 1 -- 172.21.15.90:0/625727753 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55652b266070 con 0x55652af10d80 2024-04-25T11:44:51.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.176+0000 7f8bbaff5640 10 client.18756 handle_command_reply: tid=0 2024-04-25T11:44:51.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.176+0000 7f8be464b740 1 client.18756 shutdown 2024-04-25T11:44:51.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.176+0000 7f8be464b740 1 -- 172.21.15.90:0/625727753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af314d0 msgr2=0x55652b2fc6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:51.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.176+0000 7f8be464b740 1 --2- 172.21.15.90:0/625727753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af314d0 0x55652b2fc6f0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc0dc8d0 tx=0x7f8bcc00df10 comp rx=0 tx=0).stop 2024-04-25T11:44:51.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.176+0000 7f8be464b740 1 -- 172.21.15.90:0/625727753 shutdown_connections 2024-04-25T11:44:51.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.176+0000 7f8be464b740 1 --2- 172.21.15.90:0/625727753 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55652af10d80 0x55652b30e550 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.176+0000 7f8be464b740 1 --2- 172.21.15.90:0/625727753 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55652af521b0 0x55652b3020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.176+0000 7f8be464b740 1 --2- 172.21.15.90:0/625727753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af314d0 0x55652b2fc6f0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.176+0000 7f8be464b740 1 --2- 172.21.15.90:0/625727753 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55652af292f0 0x55652b2fc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.176+0000 7f8bbaff5640 0 client.18756 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:51.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.176+0000 7f8be464b740 1 -- 172.21.15.90:0/625727753 >> 172.21.15.90:0/625727753 conn(0x55652af46300 msgr2=0x55652b2f3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:51.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.176+0000 7f8be464b740 1 -- 172.21.15.90:0/625727753 shutdown_connections 2024-04-25T11:44:51.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.176+0000 7f8be464b740 1 -- 172.21.15.90:0/625727753 wait complete. 2024-04-25T11:44:51.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.176+0000 7f8be464b740 20 client.18756 trim_cache size 0 max 16384 2024-04-25T11:44:51.182 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.191+0000 7f8be464b740 1 Processor -- start 2024-04-25T11:44:51.182 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.191+0000 7f8be464b740 1 -- start start 2024-04-25T11:44:51.182 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.191+0000 7f8be464b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af292f0 0x55652b301fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:51.182 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.191+0000 7f8be464b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55652af314d0 0x55652b2f3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:51.182 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.191+0000 7f8be464b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55652af521b0 0x55652b3084a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:51.182 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.191+0000 7f8be464b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd81afda0 con 0x55652af292f0 2024-04-25T11:44:51.182 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.191+0000 7f8be464b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd81b18e0 con 0x55652af314d0 2024-04-25T11:44:51.182 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.191+0000 7f8be464b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd81b0470 con 0x55652af521b0 2024-04-25T11:44:51.183 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.191+0000 7f8bd6ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af292f0 0x55652b301fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:51.183 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.191+0000 7f8bd6ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af292f0 0x55652b301fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48858/0 (socket says 172.21.15.90:48858) 2024-04-25T11:44:51.183 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.191+0000 7f8bd6ffd640 1 -- 172.21.15.90:0/1789851730 learned_addr learned my addr 172.21.15.90:0/1789851730 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:51.183 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.192+0000 7f8bd67fc640 1 --2- 172.21.15.90:0/1789851730 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55652af314d0 0x55652b2f3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:51.183 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.192+0000 7f8bd6ffd640 1 -- 172.21.15.90:0/1789851730 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55652af521b0 msgr2=0x55652b3084a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:51.183 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.192+0000 7f8bd6ffd640 1 --2- 172.21.15.90:0/1789851730 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55652af521b0 0x55652b3084a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.183 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.192+0000 7f8bd6ffd640 1 -- 172.21.15.90:0/1789851730 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55652af314d0 msgr2=0x55652b2f3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:51.183 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.192+0000 7f8bd6ffd640 1 --2- 172.21.15.90:0/1789851730 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55652af314d0 0x55652b2f3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.183 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.192+0000 7f8bd6ffd640 1 -- 172.21.15.90:0/1789851730 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55652b2fec80 con 0x55652af292f0 2024-04-25T11:44:51.183 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.192+0000 7f8bd67fc640 1 --2- 172.21.15.90:0/1789851730 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55652af314d0 0x55652b2f3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:51.183 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.192+0000 7f8bd6ffd640 1 --2- 172.21.15.90:0/1789851730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af292f0 0x55652b301fb0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8000f80 tx=0x7f8bc8032de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86ed9d6d18409e74 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:51.183 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.192+0000 7f8bbcff9640 1 -- 172.21.15.90:0/1789851730 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc8003b70 con 0x55652af292f0 2024-04-25T11:44:51.183 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.192+0000 7f8bbcff9640 1 -- 172.21.15.90:0/1789851730 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8bc8032e50 con 0x55652af292f0 2024-04-25T11:44:51.184 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.192+0000 7f8bbcff9640 1 -- 172.21.15.90:0/1789851730 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc80477b0 con 0x55652af292f0 2024-04-25T11:44:51.184 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.193+0000 7f8be464b740 1 -- 172.21.15.90:0/1789851730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af292f0 msgr2=0x55652b301fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:51.184 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.193+0000 7f8be464b740 1 --2- 172.21.15.90:0/1789851730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af292f0 0x55652b301fb0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8000f80 tx=0x7f8bc8032de0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.184 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.193+0000 7f8be464b740 1 -- 172.21.15.90:0/1789851730 shutdown_connections 2024-04-25T11:44:51.184 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.193+0000 7f8be464b740 1 --2- 172.21.15.90:0/1789851730 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55652af521b0 0x55652b3084a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.184 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.193+0000 7f8be464b740 1 --2- 172.21.15.90:0/1789851730 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55652af314d0 0x55652b2f3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.184 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.193+0000 7f8be464b740 1 --2- 172.21.15.90:0/1789851730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af292f0 0x55652b301fb0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.184 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.193+0000 7f8be464b740 1 -- 172.21.15.90:0/1789851730 >> 172.21.15.90:0/1789851730 conn(0x55652af46300 msgr2=0x55652b2f37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:51.184 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.193+0000 7f8be464b740 1 -- 172.21.15.90:0/1789851730 shutdown_connections 2024-04-25T11:44:51.184 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.193+0000 7f8be464b740 1 -- 172.21.15.90:0/1789851730 wait complete. 2024-04-25T11:44:51.184 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.193+0000 7f8be464b740 1 -- start start 2024-04-25T11:44:51.184 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.193+0000 7f8be464b740 1 Processor -- start 2024-04-25T11:44:51.184 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.193+0000 7f8be464b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af292f0 0x55652b373520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:51.184 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.193+0000 7f8be464b740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55652af314d0 0x55652b373a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:51.184 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.193+0000 7f8be464b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55652af521b0 0x55652b36d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:51.185 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.193+0000 7f8be464b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd81b1210 con 0x55652af292f0 2024-04-25T11:44:51.185 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.193+0000 7f8be464b740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd81b0b40 con 0x55652af314d0 2024-04-25T11:44:51.185 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.193+0000 7f8be464b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd81adda0 con 0x55652af521b0 2024-04-25T11:44:51.185 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.193+0000 7f8be464b740 10 client.0 _ll_register_callbacks cb 0x7f8bcc0a2c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:44:51.185 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.193+0000 7f8bd6ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af292f0 0x55652b373520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:51.185 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.193+0000 7f8bd6ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af292f0 0x55652b373520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48864/0 (socket says 172.21.15.90:48864) 2024-04-25T11:44:51.185 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.193+0000 7f8bd6ffd640 1 -- 172.21.15.90:0/3423852766 learned_addr learned my addr 172.21.15.90:0/3423852766 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:44:51.185 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.194+0000 7f8bd77fe640 1 --2- 172.21.15.90:0/3423852766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55652af521b0 0x55652b36d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:51.185 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.194+0000 7f8bd67fc640 1 --2- 172.21.15.90:0/3423852766 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55652af314d0 0x55652b373a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:51.185 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.194+0000 7f8bd6ffd640 1 -- 172.21.15.90:0/3423852766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55652af521b0 msgr2=0x55652b36d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:51.185 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.194+0000 7f8bd6ffd640 1 --2- 172.21.15.90:0/3423852766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55652af521b0 0x55652b36d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.186 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.194+0000 7f8bd6ffd640 1 -- 172.21.15.90:0/3423852766 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55652af314d0 msgr2=0x55652b373a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:51.186 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.194+0000 7f8bd6ffd640 1 --2- 172.21.15.90:0/3423852766 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55652af314d0 0x55652b373a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.186 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.194+0000 7f8bd6ffd640 1 -- 172.21.15.90:0/3423852766 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55652b36df20 con 0x55652af292f0 2024-04-25T11:44:51.186 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.194+0000 7f8bd6ffd640 1 --2- 172.21.15.90:0/3423852766 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af292f0 0x55652b373520 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8032c70 tx=0x7f8bc8003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dde30110cd2d7601 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:51.186 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.194+0000 7f8bbaff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:44:51.186 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.194+0000 7f8bbaff5640 1 -- 172.21.15.90:0/3423852766 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc8053020 con 0x55652af292f0 2024-04-25T11:44:51.186 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.194+0000 7f8bbaff5640 1 -- 172.21.15.90:0/3423852766 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7f8bc8032600 con 0x55652af292f0 2024-04-25T11:44:51.186 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.194+0000 7f8be464b740 1 -- 172.21.15.90:0/3423852766 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8bc8032e50 con 0x55652af292f0 2024-04-25T11:44:51.186 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.194+0000 7f8bbaff5640 1 -- 172.21.15.90:0/3423852766 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8bc805e050 con 0x55652af292f0 2024-04-25T11:44:51.186 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.194+0000 7f8be464b740 10 client.18762 fetch_fsmap learned FSMap version 28 2024-04-25T11:44:51.186 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.194+0000 7f8be464b740 1 -- 172.21.15.90:0/3423852766 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55652b37ab40 con 0x55652af292f0 2024-04-25T11:44:51.186 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.194+0000 7f8bbaff5640 1 -- 172.21.15.90:0/3423852766 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8bc8032e50 con 0x55652af292f0 2024-04-25T11:44:51.186 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.195+0000 7f8bbaff5640 1 -- 172.21.15.90:0/3423852766 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f8bc80041c0 con 0x55652af292f0 2024-04-25T11:44:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.195+0000 7f8be464b740 10 client.18762 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:44:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.195+0000 7f8be464b740 10 client.18762 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:44:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.195+0000 7f8be464b740 20 client.18762 populate_metadata read hostname 'smithi090' 2024-04-25T11:44:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.195+0000 7f8be464b740 1 --2- 172.21.15.90:0/3423852766 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55652aee61c0 0x55652b37d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.195+0000 7f8be464b740 4 client.18762 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:44:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.195+0000 7f8be464b740 1 -- 172.21.15.90:0/3423852766 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55652b25fee0 con 0x55652aee61c0 2024-04-25T11:44:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.195+0000 7f8bd67fc640 1 --2- 172.21.15.90:0/3423852766 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55652aee61c0 0x55652b37d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.195+0000 7f8bbaff5640 10 client.18762 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.195+0000 7f8bd67fc640 1 --2- 172.21.15.90:0/3423852766 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55652aee61c0 0x55652b37d1f0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87de7f77218d144f server_cookie=8a77f63869ae6f69 in_seq=0 out_seq=0 2024-04-25T11:44:51.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.202+0000 7f8bbaff5640 1 -- 172.21.15.90:0/3423852766 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55652b25fee0 con 0x55652aee61c0 2024-04-25T11:44:51.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.202+0000 7f8bbaff5640 10 client.18762 handle_command_reply: tid=0 2024-04-25T11:44:51.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.202+0000 7f8be464b740 1 client.18762 shutdown 2024-04-25T11:44:51.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.202+0000 7f8be464b740 1 -- 172.21.15.90:0/3423852766 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af292f0 msgr2=0x55652b373520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:51.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.202+0000 7f8be464b740 1 --2- 172.21.15.90:0/3423852766 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af292f0 0x55652b373520 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8032c70 tx=0x7f8bc8003970 comp rx=0 tx=0).stop 2024-04-25T11:44:51.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.202+0000 7f8be464b740 1 -- 172.21.15.90:0/3423852766 shutdown_connections 2024-04-25T11:44:51.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.202+0000 7f8be464b740 1 --2- 172.21.15.90:0/3423852766 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55652aee61c0 0x55652b37d1f0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.202+0000 7f8be464b740 1 --2- 172.21.15.90:0/3423852766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55652af521b0 0x55652b36d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.202+0000 7f8be464b740 1 --2- 172.21.15.90:0/3423852766 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55652af314d0 0x55652b373a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.202+0000 7f8bbaff5640 0 client.18762 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:44:51.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.202+0000 7f8be464b740 1 --2- 172.21.15.90:0/3423852766 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55652af292f0 0x55652b373520 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.191 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.202+0000 7f8be464b740 1 -- 172.21.15.90:0/3423852766 >> 172.21.15.90:0/3423852766 conn(0x55652af46300 msgr2=0x55652b2f1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:51.191 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.202+0000 7f8be464b740 1 -- 172.21.15.90:0/3423852766 shutdown_connections 2024-04-25T11:44:51.191 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.202+0000 7f8be464b740 1 -- 172.21.15.90:0/3423852766 wait complete. 2024-04-25T11:44:51.191 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.203+0000 7f8be464b740 20 client.18762 trim_cache size 0 max 16384 2024-04-25T11:44:51.191 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:44:51.191 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:44:51.191 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:44:51.191 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:44:51.192 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.203+0000 7f8bdd79e640 1 -- 172.21.15.90:0/2767668326 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8bc00afc50 msgr2=0x7f8bc00b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:51.192 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.203+0000 7f8bdd79e640 1 --2- 172.21.15.90:0/2767668326 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8bc00afc50 0x7f8bc00b2160 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55652b2eec60 tx=0x7f8bd00047d0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.192 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.203+0000 7f8bdd79e640 1 -- 172.21.15.90:0/2767668326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8bd81069c0 msgr2=0x7f8bd819acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:51.192 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.203+0000 7f8bdd79e640 1 --2- 172.21.15.90:0/2767668326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8bd81069c0 0x7f8bd819acf0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc00d770 tx=0x7f8bcc00dc30 comp rx=0 tx=0).stop 2024-04-25T11:44:51.192 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.203+0000 7f8bdd79e640 1 -- 172.21.15.90:0/2767668326 shutdown_connections 2024-04-25T11:44:51.193 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.203+0000 7f8bdd79e640 1 --2- 172.21.15.90:0/2767668326 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8bc00afc50 0x7f8bc00b2160 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.193 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.203+0000 7f8bdd79e640 1 --2- 172.21.15.90:0/2767668326 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bd8107480 0x7f8bd819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.193 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.203+0000 7f8bdd79e640 1 --2- 172.21.15.90:0/2767668326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8bd81069c0 0x7f8bd819acf0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.193 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.203+0000 7f8bdd79e640 1 --2- 172.21.15.90:0/2767668326 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8bd8102430 0x7f8bd819a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.193 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.203+0000 7f8bdd79e640 1 -- 172.21.15.90:0/2767668326 >> 172.21.15.90:0/2767668326 conn(0x7f8bd80fd8f0 msgr2=0x7f8bd8105150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:51.193 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.203+0000 7f8bdd79e640 1 -- 172.21.15.90:0/2767668326 shutdown_connections 2024-04-25T11:44:51.193 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:44:51.203+0000 7f8bdd79e640 1 -- 172.21.15.90:0/2767668326 wait complete. 2024-04-25T11:44:52.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:51 smithi090 ceph-mon[29575]: pgmap v302: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-04-25T11:44:52.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:51 smithi090 ceph-mon[29575]: scrub queued for path: / 2024-04-25T11:44:52.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:51 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:44:52.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:51 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:44:52.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:51 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:44:52.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:51 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:44:52.224 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:51 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:44:52.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:51 smithi138 ceph-mon[32328]: pgmap v302: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-04-25T11:44:52.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:51 smithi138 ceph-mon[32328]: scrub queued for path: / 2024-04-25T11:44:52.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:51 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:44:52.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:51 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:44:52.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:51 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:44:52.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:51 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:44:52.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:51 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:44:52.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:51 smithi156 ceph-mon[33045]: pgmap v302: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-04-25T11:44:52.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:51 smithi156 ceph-mon[33045]: scrub queued for path: / 2024-04-25T11:44:52.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:51 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:44:52.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:51 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:44:52.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:51 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:44:52.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:51 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:44:52.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:51 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:44:54.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:53 smithi090 ceph-mon[29575]: pgmap v303: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T11:44:54.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:53 smithi138 ceph-mon[32328]: pgmap v303: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T11:44:54.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:53 smithi156 ceph-mon[33045]: pgmap v303: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T11:44:56.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:55 smithi090 ceph-mon[29575]: pgmap v304: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-04-25T11:44:56.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:55 smithi138 ceph-mon[32328]: pgmap v304: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-04-25T11:44:56.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:55 smithi156 ceph-mon[33045]: pgmap v304: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-04-25T11:44:58.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:57 smithi090 ceph-mon[29575]: pgmap v305: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 12 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-04-25T11:44:58.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:57 smithi138 ceph-mon[32328]: pgmap v305: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 12 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-04-25T11:44:58.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:57 smithi156 ceph-mon[33045]: pgmap v305: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 12 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-04-25T11:45:00.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:44:59 smithi090 ceph-mon[29575]: pgmap v306: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:45:00.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:44:59 smithi138 ceph-mon[32328]: pgmap v306: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:45:00.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:44:59 smithi156 ceph-mon[33045]: pgmap v306: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:45:01.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:00 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:45:01.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:00 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:45:01.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:00 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:45:02.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:01 smithi090 ceph-mon[29575]: pgmap v307: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 12 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-04-25T11:45:02.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:01 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:45:02.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:01 smithi138 ceph-mon[32328]: pgmap v307: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 12 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-04-25T11:45:02.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:01 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:45:02.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:01 smithi156 ceph-mon[33045]: pgmap v307: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 12 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-04-25T11:45:02.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:01 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:45:04.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:03 smithi090 ceph-mon[29575]: pgmap v308: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:45:04.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.313+0000 7ff161415640 1 -- 172.21.15.90:0/3136521751 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff15c104ce0 msgr2=0x7ff15c1050f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:04.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.313+0000 7ff161415640 1 --2- 172.21.15.90:0/3136521751 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff15c104ce0 0x7ff15c1050f0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7ff14c009900 tx=0x7ff14c02a480 comp rx=0 tx=0).stop 2024-04-25T11:45:04.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.313+0000 7ff161415640 1 -- 172.21.15.90:0/3136521751 shutdown_connections 2024-04-25T11:45:04.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.313+0000 7ff161415640 1 --2- 172.21.15.90:0/3136521751 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff15c10e0f0 0x7ff15c1105c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.313+0000 7ff161415640 1 --2- 172.21.15.90:0/3136521751 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff15c10b740 0x7ff15c10db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.313+0000 7ff161415640 1 --2- 172.21.15.90:0/3136521751 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff15c104ce0 0x7ff15c1050f0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.313+0000 7ff161415640 1 -- 172.21.15.90:0/3136521751 >> 172.21.15.90:0/3136521751 conn(0x7ff15c0fe260 msgr2=0x7ff15c100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:04.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.313+0000 7ff161415640 1 -- 172.21.15.90:0/3136521751 shutdown_connections 2024-04-25T11:45:04.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.313+0000 7ff161415640 1 -- 172.21.15.90:0/3136521751 wait complete. 2024-04-25T11:45:04.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.313+0000 7ff161415640 1 Processor -- start 2024-04-25T11:45:04.301 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:03 smithi138 ceph-mon[32328]: pgmap v308: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:45:04.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.314+0000 7ff161415640 1 -- start start 2024-04-25T11:45:04.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.314+0000 7ff161415640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff15c104ce0 0x7ff15c19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.314+0000 7ff161415640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff15c10b740 0x7ff15c19ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.314+0000 7ff161415640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff15c10e0f0 0x7ff15c19ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.314+0000 7ff161415640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c113d30 con 0x7ff15c10b740 2024-04-25T11:45:04.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.314+0000 7ff161415640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c113bb0 con 0x7ff15c104ce0 2024-04-25T11:45:04.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.314+0000 7ff161415640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c113eb0 con 0x7ff15c10e0f0 2024-04-25T11:45:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.314+0000 7ff1527fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff15c10b740 0x7ff15c19ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.314+0000 7ff1527fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff15c10b740 0x7ff15c19ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53536/0 (socket says 172.21.15.90:53536) 2024-04-25T11:45:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.314+0000 7ff1527fc640 1 -- 172.21.15.90:0/1937040879 learned_addr learned my addr 172.21.15.90:0/1937040879 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.314+0000 7ff15b7fe640 1 --2- 172.21.15.90:0/1937040879 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff15c10e0f0 0x7ff15c19ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.314+0000 7ff15affd640 1 --2- 172.21.15.90:0/1937040879 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff15c104ce0 0x7ff15c19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.314+0000 7ff1527fc640 1 -- 172.21.15.90:0/1937040879 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff15c10e0f0 msgr2=0x7ff15c19ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.314+0000 7ff1527fc640 1 --2- 172.21.15.90:0/1937040879 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff15c10e0f0 0x7ff15c19ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.314+0000 7ff1527fc640 1 -- 172.21.15.90:0/1937040879 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff15c104ce0 msgr2=0x7ff15c19a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.314+0000 7ff1527fc640 1 --2- 172.21.15.90:0/1937040879 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff15c104ce0 0x7ff15c19a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.314+0000 7ff1527fc640 1 -- 172.21.15.90:0/1937040879 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff15c19f590 con 0x7ff15c10b740 2024-04-25T11:45:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.314+0000 7ff15affd640 1 --2- 172.21.15.90:0/1937040879 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff15c104ce0 0x7ff15c19a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.315+0000 7ff1527fc640 1 --2- 172.21.15.90:0/1937040879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff15c10b740 0x7ff15c19ac90 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7ff14400e9f0 tx=0x7ff14400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.315+0000 7ff158ff9640 1 -- 172.21.15.90:0/1937040879 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff14400cd80 con 0x7ff15c10b740 2024-04-25T11:45:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.315+0000 7ff158ff9640 1 -- 172.21.15.90:0/1937040879 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1440108e0 con 0x7ff15c10b740 2024-04-25T11:45:04.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.315+0000 7ff158ff9640 1 -- 172.21.15.90:0/1937040879 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff144018980 con 0x7ff15c10b740 2024-04-25T11:45:04.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.315+0000 7ff161415640 1 -- 172.21.15.90:0/1937040879 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff15c19f880 con 0x7ff15c10b740 2024-04-25T11:45:04.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.315+0000 7ff161415640 1 -- 172.21.15.90:0/1937040879 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff15c1a7200 con 0x7ff15c10b740 2024-04-25T11:45:04.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.318+0000 7ff158ff9640 1 -- 172.21.15.90:0/1937040879 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7ff144010430 con 0x7ff15c10b740 2024-04-25T11:45:04.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.318+0000 7ff1683a5740 1 Processor -- start 2024-04-25T11:45:04.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.318+0000 7ff1683a5740 1 -- start start 2024-04-25T11:45:04.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.318+0000 7ff1683a5740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad056a51b0 0x55ad05a44af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.318+0000 7ff1683a5740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad0567c2f0 0x55ad05a45030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.318+0000 7ff1683a5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad056844d0 0x55ad05a45570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.318+0000 7ff1683a5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ad059ab460 con 0x55ad056844d0 2024-04-25T11:45:04.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.318+0000 7ff1683a5740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ad05965590 con 0x55ad056a51b0 2024-04-25T11:45:04.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.318+0000 7ff1683a5740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ad059a7420 con 0x55ad0567c2f0 2024-04-25T11:45:04.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.318+0000 7ff1527fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad056844d0 0x55ad05a45570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.318+0000 7ff1527fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad056844d0 0x55ad05a45570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53544/0 (socket says 172.21.15.90:53544) 2024-04-25T11:45:04.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.318+0000 7ff1527fc640 1 -- 172.21.15.90:0/935672915 learned_addr learned my addr 172.21.15.90:0/935672915 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:04.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.318+0000 7ff15b7fe640 1 --2- 172.21.15.90:0/935672915 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad056a51b0 0x55ad05a44af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.319+0000 7ff15affd640 1 --2- 172.21.15.90:0/935672915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad0567c2f0 0x55ad05a45030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.319+0000 7ff158ff9640 1 --2- 172.21.15.90:0/1937040879 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff1480afcd0 0x7ff1480b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.319+0000 7ff1527fc640 1 -- 172.21.15.90:0/935672915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad0567c2f0 msgr2=0x55ad05a45030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:04.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.319+0000 7ff1527fc640 1 --2- 172.21.15.90:0/935672915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad0567c2f0 0x55ad05a45030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.319+0000 7ff1527fc640 1 -- 172.21.15.90:0/935672915 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad056a51b0 msgr2=0x55ad05a44af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:04.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.319+0000 7ff1527fc640 1 --2- 172.21.15.90:0/935672915 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad056a51b0 0x55ad05a44af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.319+0000 7ff1527fc640 1 -- 172.21.15.90:0/935672915 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad059720d0 con 0x55ad056844d0 2024-04-25T11:45:04.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.319+0000 7ff15b7fe640 1 --2- 172.21.15.90:0/1937040879 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff1480afcd0 0x7ff1480b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.319+0000 7ff1527fc640 1 --2- 172.21.15.90:0/935672915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad056844d0 0x55ad05a45570 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7ff1440dca70 tx=0x7ff1440b8bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f362ec73f9c0b06 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:04.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.319+0000 7ff158ff9640 1 -- 172.21.15.90:0/1937040879 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7ff1440df410 con 0x7ff15c10b740 2024-04-25T11:45:04.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.319+0000 7ff1517fa640 1 -- 172.21.15.90:0/935672915 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1440a6e70 con 0x55ad056844d0 2024-04-25T11:45:04.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.319+0000 7ff1517fa640 1 -- 172.21.15.90:0/935672915 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff144010720 con 0x55ad056844d0 2024-04-25T11:45:04.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.319+0000 7ff1517fa640 1 -- 172.21.15.90:0/935672915 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1440a6c90 con 0x55ad056844d0 2024-04-25T11:45:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.320+0000 7ff1683a5740 1 -- 172.21.15.90:0/935672915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad056844d0 msgr2=0x55ad05a45570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.320+0000 7ff1683a5740 1 --2- 172.21.15.90:0/935672915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad056844d0 0x55ad05a45570 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7ff1440dca70 tx=0x7ff1440b8bd0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.320+0000 7ff15b7fe640 1 --2- 172.21.15.90:0/1937040879 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff1480afcd0 0x7ff1480b21e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff154005c40 tx=0x7ff154004a40 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.320+0000 7ff1683a5740 1 -- 172.21.15.90:0/935672915 shutdown_connections 2024-04-25T11:45:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.320+0000 7ff1683a5740 1 --2- 172.21.15.90:0/935672915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad056844d0 0x55ad05a45570 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.320+0000 7ff1683a5740 1 --2- 172.21.15.90:0/935672915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad0567c2f0 0x55ad05a45030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.320+0000 7ff1683a5740 1 --2- 172.21.15.90:0/935672915 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad056a51b0 0x55ad05a44af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.320+0000 7ff1683a5740 1 -- 172.21.15.90:0/935672915 >> 172.21.15.90:0/935672915 conn(0x55ad05699300 msgr2=0x55ad05a412a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.320+0000 7ff1683a5740 1 -- 172.21.15.90:0/935672915 shutdown_connections 2024-04-25T11:45:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.320+0000 7ff1683a5740 1 -- 172.21.15.90:0/935672915 wait complete. 2024-04-25T11:45:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.321+0000 7ff1683a5740 1 -- start start 2024-04-25T11:45:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.321+0000 7ff1683a5740 1 Processor -- start 2024-04-25T11:45:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.321+0000 7ff1683a5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad0567c2f0 0x55ad05a50b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.321+0000 7ff1683a5740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad056844d0 0x55ad05a51070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.321+0000 7ff1683a5740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad056a51b0 0x55ad05a4f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.321+0000 7ff1683a5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ad05a51c20 con 0x55ad0567c2f0 2024-04-25T11:45:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.321+0000 7ff1683a5740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ad05932b60 con 0x55ad056844d0 2024-04-25T11:45:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.321+0000 7ff1683a5740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ad05a51da0 con 0x55ad056a51b0 2024-04-25T11:45:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.321+0000 7ff1683a5740 10 client.0 _ll_register_callbacks cb 0x55ad059572b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.321+0000 7ff15affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad0567c2f0 0x55ad05a50b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.321+0000 7ff15affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad0567c2f0 0x55ad05a50b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53550/0 (socket says 172.21.15.90:53550) 2024-04-25T11:45:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.321+0000 7ff15affd640 1 -- 172.21.15.90:0/2353819019 learned_addr learned my addr 172.21.15.90:0/2353819019 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.321+0000 7ff15b7fe640 1 --2- 172.21.15.90:0/2353819019 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad056a51b0 0x55ad05a4f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.321+0000 7ff1527fc640 1 --2- 172.21.15.90:0/2353819019 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad056844d0 0x55ad05a51070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.321+0000 7ff15affd640 1 -- 172.21.15.90:0/2353819019 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad056a51b0 msgr2=0x55ad05a4f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.321+0000 7ff15affd640 1 --2- 172.21.15.90:0/2353819019 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad056a51b0 0x55ad05a4f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.321+0000 7ff15affd640 1 -- 172.21.15.90:0/2353819019 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad056844d0 msgr2=0x55ad05a51070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.321+0000 7ff15affd640 1 --2- 172.21.15.90:0/2353819019 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad056844d0 0x55ad05a51070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.321+0000 7ff15affd640 1 -- 172.21.15.90:0/2353819019 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad056bd210 con 0x55ad0567c2f0 2024-04-25T11:45:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.321+0000 7ff1527fc640 1 --2- 172.21.15.90:0/2353819019 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad056844d0 0x55ad05a51070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.321+0000 7ff15b7fe640 1 --2- 172.21.15.90:0/2353819019 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad056a51b0 0x55ad05a4f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.322+0000 7ff15affd640 1 --2- 172.21.15.90:0/2353819019 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad0567c2f0 0x55ad05a50b30 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7ff14c02a9a0 tx=0x7ff14c031750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf47719ff9cf1d5b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.322+0000 7ff1427fc640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.322+0000 7ff1427fc640 1 -- 172.21.15.90:0/2353819019 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff14c035070 con 0x55ad0567c2f0 2024-04-25T11:45:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.322+0000 7ff1427fc640 1 -- 172.21.15.90:0/2353819019 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7ff14c004040 con 0x55ad0567c2f0 2024-04-25T11:45:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.322+0000 7ff1683a5740 1 -- 172.21.15.90:0/2353819019 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ad059b9070 con 0x55ad0567c2f0 2024-04-25T11:45:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.322+0000 7ff1427fc640 1 -- 172.21.15.90:0/2353819019 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff14c004240 con 0x55ad0567c2f0 2024-04-25T11:45:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.322+0000 7ff1683a5740 10 client.18774 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.322+0000 7ff1683a5740 1 -- 172.21.15.90:0/2353819019 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ad05789620 con 0x55ad0567c2f0 2024-04-25T11:45:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.322+0000 7ff1427fc640 1 -- 172.21.15.90:0/2353819019 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55ad059b9070 con 0x55ad0567c2f0 2024-04-25T11:45:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.322+0000 7ff1427fc640 1 -- 172.21.15.90:0/2353819019 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7ff14c04a030 con 0x55ad0567c2f0 2024-04-25T11:45:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.322+0000 7ff1683a5740 10 client.18774 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.322+0000 7ff1683a5740 10 client.18774 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:45:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.322+0000 7ff1683a5740 20 client.18774 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.322+0000 7ff1683a5740 1 --2- 172.21.15.90:0/2353819019 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ad05663d80 0x55ad05a61720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.322+0000 7ff1683a5740 4 client.18774 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:45:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.322+0000 7ff1683a5740 1 -- 172.21.15.90:0/2353819019 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad059b2ee0 con 0x55ad05663d80 2024-04-25T11:45:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.323+0000 7ff1527fc640 1 --2- 172.21.15.90:0/2353819019 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ad05663d80 0x55ad05a61720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.323+0000 7ff1427fc640 10 client.18774 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.323+0000 7ff1527fc640 1 --2- 172.21.15.90:0/2353819019 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ad05663d80 0x55ad05a61720 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27c827a06c234200 server_cookie=e6a4b40073bc085d in_seq=0 out_seq=0 2024-04-25T11:45:04.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.324+0000 7ff1427fc640 1 -- 172.21.15.90:0/2353819019 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ad059b2ee0 con 0x55ad05663d80 2024-04-25T11:45:04.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.324+0000 7ff1427fc640 10 client.18774 handle_command_reply: tid=0 2024-04-25T11:45:04.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.324+0000 7ff1683a5740 1 client.18774 shutdown 2024-04-25T11:45:04.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.324+0000 7ff1683a5740 1 -- 172.21.15.90:0/2353819019 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad0567c2f0 msgr2=0x55ad05a50b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:04.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.324+0000 7ff1683a5740 1 --2- 172.21.15.90:0/2353819019 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad0567c2f0 0x55ad05a50b30 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7ff14c02a9a0 tx=0x7ff14c031750 comp rx=0 tx=0).stop 2024-04-25T11:45:04.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.324+0000 7ff1683a5740 1 -- 172.21.15.90:0/2353819019 shutdown_connections 2024-04-25T11:45:04.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.324+0000 7ff1683a5740 1 --2- 172.21.15.90:0/2353819019 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ad05663d80 0x55ad05a61720 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.324+0000 7ff1683a5740 1 --2- 172.21.15.90:0/2353819019 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad056a51b0 0x55ad05a4f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.324+0000 7ff1683a5740 1 --2- 172.21.15.90:0/2353819019 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad056844d0 0x55ad05a51070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.324+0000 7ff1683a5740 1 --2- 172.21.15.90:0/2353819019 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad0567c2f0 0x55ad05a50b30 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.324+0000 7ff1427fc640 0 client.18774 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:04.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.324+0000 7ff1683a5740 1 -- 172.21.15.90:0/2353819019 >> 172.21.15.90:0/2353819019 conn(0x55ad05699300 msgr2=0x55ad05a46810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:04.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.324+0000 7ff1683a5740 1 -- 172.21.15.90:0/2353819019 shutdown_connections 2024-04-25T11:45:04.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.324+0000 7ff1683a5740 1 -- 172.21.15.90:0/2353819019 wait complete. 2024-04-25T11:45:04.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.324+0000 7ff1683a5740 20 client.18774 trim_cache size 0 max 16384 2024-04-25T11:45:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.339+0000 7ff1683a5740 1 Processor -- start 2024-04-25T11:45:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.339+0000 7ff1683a5740 1 -- start start 2024-04-25T11:45:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.340+0000 7ff1683a5740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad0567c2f0 0x55ad05a46e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.340+0000 7ff1683a5740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad056844d0 0x55ad05a66f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.340+0000 7ff1683a5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad056a51b0 0x55ad05a67440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.340+0000 7ff1683a5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c1afcd0 con 0x55ad056a51b0 2024-04-25T11:45:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.340+0000 7ff1683a5740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c1b1810 con 0x55ad0567c2f0 2024-04-25T11:45:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.340+0000 7ff1683a5740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c1b03a0 con 0x55ad056844d0 2024-04-25T11:45:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.340+0000 7ff15b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad056a51b0 0x55ad05a67440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.340+0000 7ff15b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad056a51b0 0x55ad05a67440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53556/0 (socket says 172.21.15.90:53556) 2024-04-25T11:45:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.340+0000 7ff15b7fe640 1 -- 172.21.15.90:0/1565690371 learned_addr learned my addr 172.21.15.90:0/1565690371 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.340+0000 7ff15affd640 1 --2- 172.21.15.90:0/1565690371 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad0567c2f0 0x55ad05a46e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.340+0000 7ff1527fc640 1 --2- 172.21.15.90:0/1565690371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad056844d0 0x55ad05a66f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.340+0000 7ff15b7fe640 1 -- 172.21.15.90:0/1565690371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad056844d0 msgr2=0x55ad05a66f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.340+0000 7ff15b7fe640 1 --2- 172.21.15.90:0/1565690371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad056844d0 0x55ad05a66f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.340+0000 7ff15b7fe640 1 -- 172.21.15.90:0/1565690371 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad0567c2f0 msgr2=0x55ad05a46e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.340+0000 7ff15b7fe640 1 --2- 172.21.15.90:0/1565690371 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad0567c2f0 0x55ad05a46e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.340+0000 7ff15b7fe640 1 -- 172.21.15.90:0/1565690371 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad05a47380 con 0x55ad056a51b0 2024-04-25T11:45:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.340+0000 7ff15b7fe640 1 --2- 172.21.15.90:0/1565690371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad056a51b0 0x55ad05a67440 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7ff15400f8c0 tx=0x7ff154019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa9bbf2523d9448a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.340+0000 7ff150ff9640 1 -- 172.21.15.90:0/1565690371 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff15402c070 con 0x55ad056a51b0 2024-04-25T11:45:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.341+0000 7ff150ff9640 1 -- 172.21.15.90:0/1565690371 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff15401bac0 con 0x55ad056a51b0 2024-04-25T11:45:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.341+0000 7ff150ff9640 1 -- 172.21.15.90:0/1565690371 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff154027ad0 con 0x55ad056a51b0 2024-04-25T11:45:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.341+0000 7ff1683a5740 1 -- 172.21.15.90:0/1565690371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad056a51b0 msgr2=0x55ad05a67440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.341+0000 7ff1683a5740 1 --2- 172.21.15.90:0/1565690371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad056a51b0 0x55ad05a67440 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7ff15400f8c0 tx=0x7ff154019000 comp rx=0 tx=0).stop 2024-04-25T11:45:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.341+0000 7ff1683a5740 1 -- 172.21.15.90:0/1565690371 shutdown_connections 2024-04-25T11:45:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.341+0000 7ff1683a5740 1 --2- 172.21.15.90:0/1565690371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad056a51b0 0x55ad05a67440 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.341+0000 7ff1683a5740 1 --2- 172.21.15.90:0/1565690371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad056844d0 0x55ad05a66f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.341+0000 7ff1683a5740 1 --2- 172.21.15.90:0/1565690371 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad0567c2f0 0x55ad05a46e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.341+0000 7ff1683a5740 1 -- 172.21.15.90:0/1565690371 >> 172.21.15.90:0/1565690371 conn(0x55ad05699300 msgr2=0x55ad05a46730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:04.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.341+0000 7ff1683a5740 1 -- 172.21.15.90:0/1565690371 shutdown_connections 2024-04-25T11:45:04.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.341+0000 7ff1683a5740 1 -- 172.21.15.90:0/1565690371 wait complete. 2024-04-25T11:45:04.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.342+0000 7ff1683a5740 1 -- start start 2024-04-25T11:45:04.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.342+0000 7ff1683a5740 1 Processor -- start 2024-04-25T11:45:04.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.342+0000 7ff1683a5740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad0567c2f0 0x55ad05a52360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.342+0000 7ff1683a5740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad056844d0 0x55ad05a528a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.342+0000 7ff1683a5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad056a51b0 0x55ad05ac7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.342+0000 7ff1683a5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c1b1140 con 0x55ad056a51b0 2024-04-25T11:45:04.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.342+0000 7ff1683a5740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c1b0a70 con 0x55ad0567c2f0 2024-04-25T11:45:04.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.342+0000 7ff1683a5740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c1adcd0 con 0x55ad056844d0 2024-04-25T11:45:04.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.342+0000 7ff1683a5740 10 client.0 _ll_register_callbacks cb 0x7ff144005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:04.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.342+0000 7ff15b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad056a51b0 0x55ad05ac7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.342+0000 7ff15b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad056a51b0 0x55ad05ac7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53564/0 (socket says 172.21.15.90:53564) 2024-04-25T11:45:04.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.342+0000 7ff15b7fe640 1 -- 172.21.15.90:0/1558870119 learned_addr learned my addr 172.21.15.90:0/1558870119 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:04.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.342+0000 7ff15affd640 1 --2- 172.21.15.90:0/1558870119 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad0567c2f0 0x55ad05a52360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.342+0000 7ff1527fc640 1 --2- 172.21.15.90:0/1558870119 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad056844d0 0x55ad05a528a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.343+0000 7ff15b7fe640 1 -- 172.21.15.90:0/1558870119 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad056844d0 msgr2=0x55ad05a528a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:04.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.343+0000 7ff15b7fe640 1 --2- 172.21.15.90:0/1558870119 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad056844d0 0x55ad05a528a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.343+0000 7ff15b7fe640 1 -- 172.21.15.90:0/1558870119 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad0567c2f0 msgr2=0x55ad05a52360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:04.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.343+0000 7ff15b7fe640 1 --2- 172.21.15.90:0/1558870119 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad0567c2f0 0x55ad05a52360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.335 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.343+0000 7ff15b7fe640 1 -- 172.21.15.90:0/1558870119 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad05ac7fe0 con 0x55ad056a51b0 2024-04-25T11:45:04.335 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.343+0000 7ff15b7fe640 1 --2- 172.21.15.90:0/1558870119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad056a51b0 0x55ad05ac7aa0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7ff154027410 tx=0x7ff154027440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9059750b00c2d5ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:04.335 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.343+0000 7ff1427fc640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:04.335 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.343+0000 7ff1427fc640 1 -- 172.21.15.90:0/1558870119 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff15402c040 con 0x55ad056a51b0 2024-04-25T11:45:04.335 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.343+0000 7ff1427fc640 1 -- 172.21.15.90:0/1558870119 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8658+0+0 (secure 0 0 0) 0x7ff154004cb0 con 0x55ad056a51b0 2024-04-25T11:45:04.335 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.343+0000 7ff1683a5740 1 -- 172.21.15.90:0/1558870119 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff15401bac0 con 0x55ad056a51b0 2024-04-25T11:45:04.335 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.343+0000 7ff1427fc640 1 -- 172.21.15.90:0/1558870119 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff154003970 con 0x55ad056a51b0 2024-04-25T11:45:04.335 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.343+0000 7ff1683a5740 10 client.18780 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.343+0000 7ff1683a5740 1 -- 172.21.15.90:0/1558870119 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ad05ac9250 con 0x55ad056a51b0 2024-04-25T11:45:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.343+0000 7ff1427fc640 1 -- 172.21.15.90:0/1558870119 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff15401bac0 con 0x55ad056a51b0 2024-04-25T11:45:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.343+0000 7ff1427fc640 1 -- 172.21.15.90:0/1558870119 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7ff154026070 con 0x55ad056a51b0 2024-04-25T11:45:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.344+0000 7ff1683a5740 10 client.18780 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.344+0000 7ff1683a5740 10 client.18780 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:45:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.344+0000 7ff1683a5740 20 client.18780 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.344+0000 7ff1683a5740 1 --2- 172.21.15.90:0/1558870119 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ad056391c0 0x55ad05acb950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.344+0000 7ff1683a5740 4 client.18780 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:45:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.344+0000 7ff1683a5740 1 -- 172.21.15.90:0/1558870119 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ad0592e5f0 con 0x55ad056391c0 2024-04-25T11:45:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.344+0000 7ff15affd640 1 --2- 172.21.15.90:0/1558870119 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ad056391c0 0x55ad05acb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.344+0000 7ff1427fc640 10 client.18780 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.344+0000 7ff15affd640 1 --2- 172.21.15.90:0/1558870119 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ad056391c0 0x55ad05acb950 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2f16d4584b78f74 server_cookie=c9211f031674deaa in_seq=0 out_seq=0 2024-04-25T11:45:04.342 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:03 smithi156 ceph-mon[33045]: pgmap v308: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:45:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.354+0000 7ff1427fc640 1 -- 172.21.15.90:0/1558870119 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ad0592e5f0 con 0x55ad056391c0 2024-04-25T11:45:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.354+0000 7ff1427fc640 10 client.18780 handle_command_reply: tid=0 2024-04-25T11:45:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.354+0000 7ff1683a5740 1 client.18780 shutdown 2024-04-25T11:45:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.354+0000 7ff1683a5740 1 -- 172.21.15.90:0/1558870119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad056a51b0 msgr2=0x55ad05ac7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.355+0000 7ff1683a5740 1 --2- 172.21.15.90:0/1558870119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad056a51b0 0x55ad05ac7aa0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7ff154027410 tx=0x7ff154027440 comp rx=0 tx=0).stop 2024-04-25T11:45:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.355+0000 7ff1683a5740 1 -- 172.21.15.90:0/1558870119 shutdown_connections 2024-04-25T11:45:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.355+0000 7ff1683a5740 1 --2- 172.21.15.90:0/1558870119 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ad056391c0 0x55ad05acb950 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.355+0000 7ff1683a5740 1 --2- 172.21.15.90:0/1558870119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad056a51b0 0x55ad05ac7aa0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.355+0000 7ff1683a5740 1 --2- 172.21.15.90:0/1558870119 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad056844d0 0x55ad05a528a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.355+0000 7ff1683a5740 1 --2- 172.21.15.90:0/1558870119 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad0567c2f0 0x55ad05a52360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.355+0000 7ff1427fc640 0 client.18780 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.355+0000 7ff1683a5740 1 -- 172.21.15.90:0/1558870119 >> 172.21.15.90:0/1558870119 conn(0x55ad05699300 msgr2=0x55ad05a4e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.355+0000 7ff1683a5740 1 -- 172.21.15.90:0/1558870119 shutdown_connections 2024-04-25T11:45:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.355+0000 7ff1683a5740 1 -- 172.21.15.90:0/1558870119 wait complete. 2024-04-25T11:45:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.355+0000 7ff1683a5740 20 client.18780 trim_cache size 0 max 16384 2024-04-25T11:45:04.346 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:45:04.346 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:45:04.346 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:45:04.346 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:45:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.355+0000 7ff161415640 1 -- 172.21.15.90:0/1937040879 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff1480afcd0 msgr2=0x7ff1480b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.355+0000 7ff161415640 1 --2- 172.21.15.90:0/1937040879 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff1480afcd0 0x7ff1480b21e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff154005c40 tx=0x7ff154004a40 comp rx=0 tx=0).stop 2024-04-25T11:45:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.355+0000 7ff161415640 1 -- 172.21.15.90:0/1937040879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff15c10b740 msgr2=0x7ff15c19ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.355+0000 7ff161415640 1 --2- 172.21.15.90:0/1937040879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff15c10b740 0x7ff15c19ac90 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7ff14400e9f0 tx=0x7ff14400eeb0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.355+0000 7ff161415640 1 -- 172.21.15.90:0/1937040879 shutdown_connections 2024-04-25T11:45:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.355+0000 7ff161415640 1 --2- 172.21.15.90:0/1937040879 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff1480afcd0 0x7ff1480b21e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.355+0000 7ff161415640 1 --2- 172.21.15.90:0/1937040879 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff15c10e0f0 0x7ff15c19ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.348 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.355+0000 7ff161415640 1 --2- 172.21.15.90:0/1937040879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff15c10b740 0x7ff15c19ac90 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.348 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.355+0000 7ff161415640 1 --2- 172.21.15.90:0/1937040879 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff15c104ce0 0x7ff15c19a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.348 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.355+0000 7ff161415640 1 -- 172.21.15.90:0/1937040879 >> 172.21.15.90:0/1937040879 conn(0x7ff15c0fe260 msgr2=0x7ff15c10e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:04.348 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.355+0000 7ff161415640 1 -- 172.21.15.90:0/1937040879 shutdown_connections 2024-04-25T11:45:04.348 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:04.355+0000 7ff161415640 1 -- 172.21.15.90:0/1937040879 wait complete. 2024-04-25T11:45:06.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:05 smithi090 ceph-mon[29575]: pgmap v309: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 9.3 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:45:06.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:05 smithi138 ceph-mon[32328]: pgmap v309: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 9.3 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:45:06.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:05 smithi156 ceph-mon[33045]: pgmap v309: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 9.3 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:45:07.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:06 smithi090 ceph-mon[29575]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T11:45:07.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:06 smithi138 ceph-mon[32328]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T11:45:07.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:06 smithi156 ceph-mon[33045]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T11:45:08.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:07 smithi090 ceph-mon[29575]: pgmap v311: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 9.8 KiB/s rd, 57 KiB/s wr, 19 op/s 2024-04-25T11:45:08.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:07 smithi138 ceph-mon[32328]: pgmap v311: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 9.8 KiB/s rd, 57 KiB/s wr, 19 op/s 2024-04-25T11:45:08.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:07 smithi156 ceph-mon[33045]: pgmap v311: 97 pgs: 97 active+clean; 41 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 9.8 KiB/s rd, 57 KiB/s wr, 19 op/s 2024-04-25T11:45:10.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:09 smithi090 ceph-mon[29575]: pgmap v312: 97 pgs: 4 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 83 active+clean; 40 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 9.8 KiB/s rd, 57 KiB/s wr, 19 op/s 2024-04-25T11:45:10.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:09 smithi138 ceph-mon[32328]: pgmap v312: 97 pgs: 4 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 83 active+clean; 40 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 9.8 KiB/s rd, 57 KiB/s wr, 19 op/s 2024-04-25T11:45:10.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:09 smithi156 ceph-mon[33045]: pgmap v312: 97 pgs: 4 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 83 active+clean; 40 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 9.8 KiB/s rd, 57 KiB/s wr, 19 op/s 2024-04-25T11:45:12.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:11 smithi090 ceph-mon[29575]: pgmap v313: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 38 GiB data, 118 GiB used, 955 GiB / 1.0 TiB avail; 13 KiB/s rd, 57 KiB/s wr, 19 op/s 2024-04-25T11:45:12.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:11 smithi138 ceph-mon[32328]: pgmap v313: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 38 GiB data, 118 GiB used, 955 GiB / 1.0 TiB avail; 13 KiB/s rd, 57 KiB/s wr, 19 op/s 2024-04-25T11:45:12.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:11 smithi156 ceph-mon[33045]: pgmap v313: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 38 GiB data, 118 GiB used, 955 GiB / 1.0 TiB avail; 13 KiB/s rd, 57 KiB/s wr, 19 op/s 2024-04-25T11:45:14.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:13 smithi090 ceph-mon[29575]: pgmap v314: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 37 GiB data, 116 GiB used, 957 GiB / 1.0 TiB avail; 13 KiB/s rd, 57 KiB/s wr, 19 op/s 2024-04-25T11:45:14.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:13 smithi138 ceph-mon[32328]: pgmap v314: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 37 GiB data, 116 GiB used, 957 GiB / 1.0 TiB avail; 13 KiB/s rd, 57 KiB/s wr, 19 op/s 2024-04-25T11:45:14.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:13 smithi156 ceph-mon[33045]: pgmap v314: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 37 GiB data, 116 GiB used, 957 GiB / 1.0 TiB avail; 13 KiB/s rd, 57 KiB/s wr, 19 op/s 2024-04-25T11:45:16.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:15 smithi090 ceph-mon[29575]: pgmap v315: 97 pgs: 20 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 41 active+clean; 36 GiB data, 112 GiB used, 961 GiB / 1.0 TiB avail; 13 KiB/s rd, 57 KiB/s wr, 19 op/s 2024-04-25T11:45:16.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:15 smithi138 ceph-mon[32328]: pgmap v315: 97 pgs: 20 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 41 active+clean; 36 GiB data, 112 GiB used, 961 GiB / 1.0 TiB avail; 13 KiB/s rd, 57 KiB/s wr, 19 op/s 2024-04-25T11:45:16.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:15 smithi156 ceph-mon[33045]: pgmap v315: 97 pgs: 20 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 41 active+clean; 36 GiB data, 112 GiB used, 961 GiB / 1.0 TiB avail; 13 KiB/s rd, 57 KiB/s wr, 19 op/s 2024-04-25T11:45:17.449 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.462+0000 7fe3cf7b7640 1 -- 172.21.15.90:0/954586378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3c81031c0 msgr2=0x7fe3c810f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:17.449 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.462+0000 7fe3cf7b7640 1 --2- 172.21.15.90:0/954586378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3c81031c0 0x7fe3c810f4f0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fe3c0009ca0 tx=0x7fe3c002a8c0 comp rx=0 tx=0).stop 2024-04-25T11:45:17.449 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.462+0000 7fe3cf7b7640 1 -- 172.21.15.90:0/954586378 shutdown_connections 2024-04-25T11:45:17.449 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.462+0000 7fe3cf7b7640 1 --2- 172.21.15.90:0/954586378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3c81031c0 0x7fe3c810f4f0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:17.449 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.462+0000 7fe3cf7b7640 1 --2- 172.21.15.90:0/954586378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3c8108f00 0x7fe3c8102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:17.449 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.462+0000 7fe3cf7b7640 1 --2- 172.21.15.90:0/954586378 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe3c8108a60 0x7fe3c8102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:17.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.462+0000 7fe3cf7b7640 1 -- 172.21.15.90:0/954586378 >> 172.21.15.90:0/954586378 conn(0x7fe3c80781b0 msgr2=0x7fe3c80785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:17.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.462+0000 7fe3cf7b7640 1 -- 172.21.15.90:0/954586378 shutdown_connections 2024-04-25T11:45:17.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.462+0000 7fe3cf7b7640 1 -- 172.21.15.90:0/954586378 wait complete. 2024-04-25T11:45:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.463+0000 7fe3cf7b7640 1 Processor -- start 2024-04-25T11:45:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.463+0000 7fe3cf7b7640 1 -- start start 2024-04-25T11:45:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.463+0000 7fe3cf7b7640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3c81031c0 0x7fe3c819a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.463+0000 7fe3cf7b7640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe3c8108a60 0x7fe3c819ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.463+0000 7fe3cf7b7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3c8108f00 0x7fe3c819ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.463+0000 7fe3cf7b7640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c8112ca0 con 0x7fe3c81031c0 2024-04-25T11:45:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.463+0000 7fe3cf7b7640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c8112b20 con 0x7fe3c8108a60 2024-04-25T11:45:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.463+0000 7fe3cf7b7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c8112e20 con 0x7fe3c8108f00 2024-04-25T11:45:17.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.463+0000 7fe3cd553640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3c81031c0 0x7fe3c819a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:17.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.464+0000 7fe3cd553640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3c81031c0 0x7fe3c819a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59848/0 (socket says 172.21.15.90:59848) 2024-04-25T11:45:17.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.464+0000 7fe3cd553640 1 -- 172.21.15.90:0/382354848 learned_addr learned my addr 172.21.15.90:0/382354848 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:17.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.464+0000 7fe3ccd52640 1 --2- 172.21.15.90:0/382354848 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe3c8108a60 0x7fe3c819ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:17.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.464+0000 7fe3cdd54640 1 --2- 172.21.15.90:0/382354848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3c8108f00 0x7fe3c819ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:17.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.464+0000 7fe3cd553640 1 -- 172.21.15.90:0/382354848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3c8108f00 msgr2=0x7fe3c819ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:17.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.464+0000 7fe3cd553640 1 --2- 172.21.15.90:0/382354848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3c8108f00 0x7fe3c819ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:17.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.464+0000 7fe3cd553640 1 -- 172.21.15.90:0/382354848 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe3c8108a60 msgr2=0x7fe3c819ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:17.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.464+0000 7fe3cd553640 1 --2- 172.21.15.90:0/382354848 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe3c8108a60 0x7fe3c819ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:17.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.464+0000 7fe3cd553640 1 -- 172.21.15.90:0/382354848 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c819f590 con 0x7fe3c81031c0 2024-04-25T11:45:17.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.464+0000 7fe3ccd52640 1 --2- 172.21.15.90:0/382354848 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe3c8108a60 0x7fe3c819ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:45:17.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.464+0000 7fe3cd553640 1 --2- 172.21.15.90:0/382354848 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3c81031c0 0x7fe3c819a730 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fe3c400e9f0 tx=0x7fe3c400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:17.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.464+0000 7fe3be7fc640 1 -- 172.21.15.90:0/382354848 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c400ccc0 con 0x7fe3c81031c0 2024-04-25T11:45:17.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.464+0000 7fe3cf7b7640 1 -- 172.21.15.90:0/382354848 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3c819f880 con 0x7fe3c81031c0 2024-04-25T11:45:17.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.464+0000 7fe3cf7b7640 1 -- 172.21.15.90:0/382354848 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3c81a7200 con 0x7fe3c81031c0 2024-04-25T11:45:17.454 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.464+0000 7fe3be7fc640 1 -- 172.21.15.90:0/382354848 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3c40108e0 con 0x7fe3c81031c0 2024-04-25T11:45:17.454 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.464+0000 7fe3be7fc640 1 -- 172.21.15.90:0/382354848 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c4018860 con 0x7fe3c81031c0 2024-04-25T11:45:17.460 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.467+0000 7fe3d666d740 1 Processor -- start 2024-04-25T11:45:17.460 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.467+0000 7fe3d666d740 1 -- start start 2024-04-25T11:45:17.460 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.467+0000 7fe3d666d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e95441b0 0x55c8e98e3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.467+0000 7fe3be7fc640 1 -- 172.21.15.90:0/382354848 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fe3c4010430 con 0x7fe3c81031c0 2024-04-25T11:45:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.467+0000 7fe3d666d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e951b2f0 0x55c8e98e4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.467+0000 7fe3d666d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e95234d0 0x55c8e98e4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.467+0000 7fe3d666d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c8e984a460 con 0x55c8e95234d0 2024-04-25T11:45:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.467+0000 7fe3d666d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c8e9804590 con 0x55c8e95441b0 2024-04-25T11:45:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.467+0000 7fe3d666d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c8e9846420 con 0x55c8e951b2f0 2024-04-25T11:45:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.467+0000 7fe3ccd52640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e95234d0 0x55c8e98e4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.468+0000 7fe3ccd52640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e95234d0 0x55c8e98e4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59860/0 (socket says 172.21.15.90:59860) 2024-04-25T11:45:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.468+0000 7fe3cdd54640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e95441b0 0x55c8e98e3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.468+0000 7fe3ccd52640 1 -- 172.21.15.90:0/2493723224 learned_addr learned my addr 172.21.15.90:0/2493723224 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.468+0000 7fe3cd553640 1 --2- 172.21.15.90:0/2493723224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e951b2f0 0x55c8e98e4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.468+0000 7fe3ccd52640 1 -- 172.21.15.90:0/2493723224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e951b2f0 msgr2=0x55c8e98e4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.468+0000 7fe3ccd52640 1 --2- 172.21.15.90:0/2493723224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e951b2f0 0x55c8e98e4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.468+0000 7fe3ccd52640 1 -- 172.21.15.90:0/2493723224 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e95441b0 msgr2=0x55c8e98e3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.468+0000 7fe3ccd52640 1 --2- 172.21.15.90:0/2493723224 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e95441b0 0x55c8e98e3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.468+0000 7fe3ccd52640 1 -- 172.21.15.90:0/2493723224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8e98110d0 con 0x55c8e95234d0 2024-04-25T11:45:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.468+0000 7fe3cd553640 1 --2- 172.21.15.90:0/2493723224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e951b2f0 0x55c8e98e4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:45:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.468+0000 7fe3cdd54640 1 --2- 172.21.15.90:0/2493723224 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e95441b0 0x55c8e98e3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:45:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.468+0000 7fe3ccd52640 1 --2- 172.21.15.90:0/2493723224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e95234d0 0x55c8e98e4570 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8010400 tx=0x7fe3b80108c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f44bb6135e5574 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.468+0000 7fe3be7fc640 1 --2- 172.21.15.90:0/382354848 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe3b40afc50 0x7fe3b40b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.468+0000 7fe3b37fe640 1 -- 172.21.15.90:0/2493723224 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b8011760 con 0x55c8e95234d0 2024-04-25T11:45:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.468+0000 7fe3cdd54640 1 --2- 172.21.15.90:0/382354848 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe3b40afc50 0x7fe3b40b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.468+0000 7fe3b37fe640 1 -- 172.21.15.90:0/2493723224 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3b8011db0 con 0x55c8e95234d0 2024-04-25T11:45:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.468+0000 7fe3b37fe640 1 -- 172.21.15.90:0/2493723224 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b801a380 con 0x55c8e95234d0 2024-04-25T11:45:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.468+0000 7fe3be7fc640 1 -- 172.21.15.90:0/382354848 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7fe3c40df320 con 0x7fe3c81031c0 2024-04-25T11:45:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.469+0000 7fe3d666d740 1 -- 172.21.15.90:0/2493723224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e95234d0 msgr2=0x55c8e98e4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.469+0000 7fe3cdd54640 1 --2- 172.21.15.90:0/382354848 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe3b40afc50 0x7fe3b40b2160 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x55c8e98e0c60 tx=0x7fe3c0031c10 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.469+0000 7fe3d666d740 1 --2- 172.21.15.90:0/2493723224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e95234d0 0x55c8e98e4570 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8010400 tx=0x7fe3b80108c0 comp rx=0 tx=0).stop 2024-04-25T11:45:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.469+0000 7fe3d666d740 1 -- 172.21.15.90:0/2493723224 shutdown_connections 2024-04-25T11:45:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.469+0000 7fe3d666d740 1 --2- 172.21.15.90:0/2493723224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e95234d0 0x55c8e98e4570 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.469+0000 7fe3d666d740 1 --2- 172.21.15.90:0/2493723224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e951b2f0 0x55c8e98e4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.469+0000 7fe3d666d740 1 --2- 172.21.15.90:0/2493723224 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e95441b0 0x55c8e98e3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.469+0000 7fe3d666d740 1 -- 172.21.15.90:0/2493723224 >> 172.21.15.90:0/2493723224 conn(0x55c8e9538300 msgr2=0x55c8e98e02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.469+0000 7fe3d666d740 1 -- 172.21.15.90:0/2493723224 shutdown_connections 2024-04-25T11:45:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.469+0000 7fe3d666d740 1 -- 172.21.15.90:0/2493723224 wait complete. 2024-04-25T11:45:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.470+0000 7fe3d666d740 1 -- start start 2024-04-25T11:45:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.470+0000 7fe3d666d740 1 Processor -- start 2024-04-25T11:45:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.470+0000 7fe3d666d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e951b2f0 0x55c8e98ee040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.470+0000 7fe3d666d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e95234d0 0x55c8e98ee580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.470+0000 7fe3d666d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e95441b0 0x55c8e98fa0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.470+0000 7fe3d666d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c8e98f0c20 con 0x55c8e95234d0 2024-04-25T11:45:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.470+0000 7fe3d666d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c8e97d1b60 con 0x55c8e951b2f0 2024-04-25T11:45:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.470+0000 7fe3d666d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c8e98f0da0 con 0x55c8e95441b0 2024-04-25T11:45:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.470+0000 7fe3d666d740 10 client.0 _ll_register_callbacks cb 0x55c8e97f62b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.470+0000 7fe3ccd52640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e95234d0 0x55c8e98ee580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.470+0000 7fe3cd553640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e951b2f0 0x55c8e98ee040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.470+0000 7fe3ccd52640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e95234d0 0x55c8e98ee580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59876/0 (socket says 172.21.15.90:59876) 2024-04-25T11:45:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.470+0000 7fe3ccd52640 1 -- 172.21.15.90:0/1244904424 learned_addr learned my addr 172.21.15.90:0/1244904424 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.471+0000 7fe3cdd54640 1 --2- 172.21.15.90:0/1244904424 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e95441b0 0x55c8e98fa0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.471+0000 7fe3ccd52640 1 -- 172.21.15.90:0/1244904424 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e95441b0 msgr2=0x55c8e98fa0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.471+0000 7fe3ccd52640 1 --2- 172.21.15.90:0/1244904424 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e95441b0 0x55c8e98fa0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.471+0000 7fe3ccd52640 1 -- 172.21.15.90:0/1244904424 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e951b2f0 msgr2=0x55c8e98ee040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.471+0000 7fe3ccd52640 1 --2- 172.21.15.90:0/1244904424 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e951b2f0 0x55c8e98ee040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.471+0000 7fe3ccd52640 1 -- 172.21.15.90:0/1244904424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8e955c210 con 0x55c8e95234d0 2024-04-25T11:45:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.471+0000 7fe3cdd54640 1 --2- 172.21.15.90:0/1244904424 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e95441b0 0x55c8e98fa0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.471+0000 7fe3cd553640 1 --2- 172.21.15.90:0/1244904424 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e951b2f0 0x55c8e98ee040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:45:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.471+0000 7fe3ccd52640 1 --2- 172.21.15.90:0/1244904424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e95234d0 0x55c8e98ee580 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fe3b80103d0 tx=0x7fe3b801c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc2486079328ab16 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.471+0000 7fe3b0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.471+0000 7fe3b0ff9640 1 -- 172.21.15.90:0/1244904424 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b8005450 con 0x55c8e95234d0 2024-04-25T11:45:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.471+0000 7fe3b0ff9640 1 -- 172.21.15.90:0/1244904424 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7fe3b80055f0 con 0x55c8e95234d0 2024-04-25T11:45:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.471+0000 7fe3d666d740 1 -- 172.21.15.90:0/1244904424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3b8011db0 con 0x55c8e95234d0 2024-04-25T11:45:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.471+0000 7fe3b0ff9640 1 -- 172.21.15.90:0/1244904424 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe3b8032050 con 0x55c8e95234d0 2024-04-25T11:45:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.471+0000 7fe3d666d740 10 client.18792 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.471+0000 7fe3d666d740 1 -- 172.21.15.90:0/1244904424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c8e9628620 con 0x55c8e95234d0 2024-04-25T11:45:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.471+0000 7fe3b0ff9640 1 -- 172.21.15.90:0/1244904424 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3b8011db0 con 0x55c8e95234d0 2024-04-25T11:45:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.472+0000 7fe3b0ff9640 1 -- 172.21.15.90:0/1244904424 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fe3b802d030 con 0x55c8e95234d0 2024-04-25T11:45:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.472+0000 7fe3d666d740 10 client.18792 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.472+0000 7fe3d666d740 10 client.18792 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:45:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.472+0000 7fe3d666d740 20 client.18792 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.472+0000 7fe3d666d740 1 --2- 172.21.15.90:0/1244904424 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c8e9502d80 0x55c8e9900440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.472+0000 7fe3d666d740 4 client.18792 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:45:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.472+0000 7fe3d666d740 1 -- 172.21.15.90:0/1244904424 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8e9858070 con 0x55c8e9502d80 2024-04-25T11:45:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.472+0000 7fe3cd553640 1 --2- 172.21.15.90:0/1244904424 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c8e9502d80 0x55c8e9900440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:17.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.472+0000 7fe3b0ff9640 10 client.18792 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:17.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.472+0000 7fe3cd553640 1 --2- 172.21.15.90:0/1244904424 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c8e9502d80 0x55c8e9900440 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8bfed1a45cf22232 server_cookie=849e21fec11d62a4 in_seq=0 out_seq=0 2024-04-25T11:45:17.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.473+0000 7fe3b0ff9640 1 -- 172.21.15.90:0/1244904424 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c8e9858070 con 0x55c8e9502d80 2024-04-25T11:45:17.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.473+0000 7fe3b0ff9640 10 client.18792 handle_command_reply: tid=0 2024-04-25T11:45:17.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.473+0000 7fe3d666d740 1 client.18792 shutdown 2024-04-25T11:45:17.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.473+0000 7fe3d666d740 1 -- 172.21.15.90:0/1244904424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e95234d0 msgr2=0x55c8e98ee580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:17.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.473+0000 7fe3d666d740 1 --2- 172.21.15.90:0/1244904424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e95234d0 0x55c8e98ee580 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fe3b80103d0 tx=0x7fe3b801c6b0 comp rx=0 tx=0).stop 2024-04-25T11:45:17.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.473+0000 7fe3d666d740 1 -- 172.21.15.90:0/1244904424 shutdown_connections 2024-04-25T11:45:17.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.473+0000 7fe3d666d740 1 --2- 172.21.15.90:0/1244904424 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c8e9502d80 0x55c8e9900440 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:17.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.473+0000 7fe3d666d740 1 --2- 172.21.15.90:0/1244904424 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e95441b0 0x55c8e98fa0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:17.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.473+0000 7fe3b0ff9640 0 client.18792 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:17.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.473+0000 7fe3d666d740 1 --2- 172.21.15.90:0/1244904424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e95234d0 0x55c8e98ee580 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:17.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.473+0000 7fe3d666d740 1 --2- 172.21.15.90:0/1244904424 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e951b2f0 0x55c8e98ee040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:17.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.473+0000 7fe3d666d740 1 -- 172.21.15.90:0/1244904424 >> 172.21.15.90:0/1244904424 conn(0x55c8e9538300 msgr2=0x55c8e98e5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:17.467 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.473+0000 7fe3d666d740 1 -- 172.21.15.90:0/1244904424 shutdown_connections 2024-04-25T11:45:17.467 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.473+0000 7fe3d666d740 1 -- 172.21.15.90:0/1244904424 wait complete. 2024-04-25T11:45:17.467 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.473+0000 7fe3d666d740 20 client.18792 trim_cache size 0 max 16384 2024-04-25T11:45:17.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.489+0000 7fe3d666d740 1 Processor -- start 2024-04-25T11:45:17.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.489+0000 7fe3d666d740 1 -- start start 2024-04-25T11:45:17.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.489+0000 7fe3d666d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e9502d80 0x55c8e98e0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:17.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.489+0000 7fe3d666d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e951b2f0 0x55c8e98f9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:17.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.489+0000 7fe3d666d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e95234d0 0x55c8e9905a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:17.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.489+0000 7fe3d666d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ac00ea70 con 0x55c8e95234d0 2024-04-25T11:45:17.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.489+0000 7fe3d666d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ac0105b0 con 0x55c8e9502d80 2024-04-25T11:45:17.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.489+0000 7fe3d666d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ac00f140 con 0x55c8e951b2f0 2024-04-25T11:45:17.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.489+0000 7fe3cdd54640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e95234d0 0x55c8e9905a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:17.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.489+0000 7fe3cd553640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e9502d80 0x55c8e98e0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:17.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.489+0000 7fe3cd553640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e9502d80 0x55c8e98e0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:60496/0 (socket says 172.21.15.90:60496) 2024-04-25T11:45:17.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.489+0000 7fe3cdd54640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e95234d0 0x55c8e9905a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59886/0 (socket says 172.21.15.90:59886) 2024-04-25T11:45:17.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.489+0000 7fe3cd553640 1 -- 172.21.15.90:0/1857380776 learned_addr learned my addr 172.21.15.90:0/1857380776 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:17.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.489+0000 7fe3ccd52640 1 --2- 172.21.15.90:0/1857380776 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e951b2f0 0x55c8e98f9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:17.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.489+0000 7fe3cd553640 1 -- 172.21.15.90:0/1857380776 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e951b2f0 msgr2=0x55c8e98f9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:17.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.489+0000 7fe3cd553640 1 --2- 172.21.15.90:0/1857380776 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e951b2f0 0x55c8e98f9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:17.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.489+0000 7fe3cd553640 1 -- 172.21.15.90:0/1857380776 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e95234d0 msgr2=0x55c8e9905a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:17.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.489+0000 7fe3cd553640 1 --2- 172.21.15.90:0/1857380776 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e95234d0 0x55c8e9905a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:17.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.489+0000 7fe3cd553640 1 -- 172.21.15.90:0/1857380776 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8e98e34b0 con 0x55c8e9502d80 2024-04-25T11:45:17.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.489+0000 7fe3cdd54640 1 --2- 172.21.15.90:0/1857380776 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e95234d0 0x55c8e9905a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:45:17.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.490+0000 7fe3cd553640 1 --2- 172.21.15.90:0/1857380776 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e9502d80 0x55c8e98e0220 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe3c40dcfc0 tx=0x7fe3c40e1830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6341856f383d62a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:17.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.490+0000 7fe3b2ffd640 1 -- 172.21.15.90:0/1857380776 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c40df070 con 0x55c8e9502d80 2024-04-25T11:45:17.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.490+0000 7fe3b2ffd640 1 -- 172.21.15.90:0/1857380776 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3c4010720 con 0x55c8e9502d80 2024-04-25T11:45:17.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.490+0000 7fe3b2ffd640 1 -- 172.21.15.90:0/1857380776 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c40b0270 con 0x55c8e9502d80 2024-04-25T11:45:17.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.490+0000 7fe3d666d740 1 -- 172.21.15.90:0/1857380776 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e9502d80 msgr2=0x55c8e98e0220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:17.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.490+0000 7fe3d666d740 1 --2- 172.21.15.90:0/1857380776 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e9502d80 0x55c8e98e0220 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe3c40dcfc0 tx=0x7fe3c40e1830 comp rx=0 tx=0).stop 2024-04-25T11:45:17.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.491+0000 7fe3d666d740 1 -- 172.21.15.90:0/1857380776 shutdown_connections 2024-04-25T11:45:17.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.491+0000 7fe3d666d740 1 --2- 172.21.15.90:0/1857380776 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e95234d0 0x55c8e9905a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:17.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.491+0000 7fe3d666d740 1 --2- 172.21.15.90:0/1857380776 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e951b2f0 0x55c8e98f9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:17.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.491+0000 7fe3d666d740 1 --2- 172.21.15.90:0/1857380776 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e9502d80 0x55c8e98e0220 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:17.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.491+0000 7fe3d666d740 1 -- 172.21.15.90:0/1857380776 >> 172.21.15.90:0/1857380776 conn(0x55c8e9538300 msgr2=0x55c8e98dfb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:17.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.491+0000 7fe3d666d740 1 -- 172.21.15.90:0/1857380776 shutdown_connections 2024-04-25T11:45:17.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.491+0000 7fe3d666d740 1 -- 172.21.15.90:0/1857380776 wait complete. 2024-04-25T11:45:17.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.491+0000 7fe3d666d740 1 -- start start 2024-04-25T11:45:17.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.491+0000 7fe3d666d740 1 Processor -- start 2024-04-25T11:45:17.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.491+0000 7fe3d666d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e9502d80 0x55c8e98f1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:17.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.491+0000 7fe3d666d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e951b2f0 0x55c8e98f2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:17.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.491+0000 7fe3d666d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e95234d0 0x55c8e98f25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:17.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.491+0000 7fe3d666d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ac00fee0 con 0x55c8e9502d80 2024-04-25T11:45:17.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.491+0000 7fe3d666d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ac00f810 con 0x55c8e95234d0 2024-04-25T11:45:17.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.491+0000 7fe3d666d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ac00ca70 con 0x55c8e951b2f0 2024-04-25T11:45:17.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.492+0000 7fe3d666d740 10 client.0 _ll_register_callbacks cb 0x7fe3b800be80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:17.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.492+0000 7fe3cd553640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e9502d80 0x55c8e98f1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:17.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.492+0000 7fe3cd553640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e9502d80 0x55c8e98f1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59898/0 (socket says 172.21.15.90:59898) 2024-04-25T11:45:17.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.492+0000 7fe3cd553640 1 -- 172.21.15.90:0/3052701423 learned_addr learned my addr 172.21.15.90:0/3052701423 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:17.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.492+0000 7fe3cdd54640 1 --2- 172.21.15.90:0/3052701423 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e95234d0 0x55c8e98f25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.492+0000 7fe3ccd52640 1 --2- 172.21.15.90:0/3052701423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e951b2f0 0x55c8e98f2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.492+0000 7fe3cd553640 1 -- 172.21.15.90:0/3052701423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e951b2f0 msgr2=0x55c8e98f2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.492+0000 7fe3cd553640 1 --2- 172.21.15.90:0/3052701423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e951b2f0 0x55c8e98f2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.492+0000 7fe3cd553640 1 -- 172.21.15.90:0/3052701423 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e95234d0 msgr2=0x55c8e98f25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.492+0000 7fe3cd553640 1 --2- 172.21.15.90:0/3052701423 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e95234d0 0x55c8e98f25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.492+0000 7fe3cd553640 1 -- 172.21.15.90:0/3052701423 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8e996b9d0 con 0x55c8e9502d80 2024-04-25T11:45:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.492+0000 7fe3ccd52640 1 --2- 172.21.15.90:0/3052701423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e951b2f0 0x55c8e98f2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.492+0000 7fe3cd553640 1 --2- 172.21.15.90:0/3052701423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e9502d80 0x55c8e98f1b30 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fe3c40df540 tx=0x7fe3c40a39e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec06d45438a42085 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.492+0000 7fe3b0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.492+0000 7fe3b0ff9640 1 -- 172.21.15.90:0/3052701423 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c40df040 con 0x55c8e9502d80 2024-04-25T11:45:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.492+0000 7fe3b0ff9640 1 -- 172.21.15.90:0/3052701423 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7fe3c40aed50 con 0x55c8e9502d80 2024-04-25T11:45:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.492+0000 7fe3d666d740 1 -- 172.21.15.90:0/3052701423 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c8e98ee910 con 0x55c8e9502d80 2024-04-25T11:45:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.492+0000 7fe3b0ff9640 1 -- 172.21.15.90:0/3052701423 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe3c40a6c30 con 0x55c8e9502d80 2024-04-25T11:45:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.492+0000 7fe3d666d740 10 client.18798 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.492+0000 7fe3d666d740 1 -- 172.21.15.90:0/3052701423 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c8e996caa0 con 0x55c8e9502d80 2024-04-25T11:45:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.493+0000 7fe3b0ff9640 1 -- 172.21.15.90:0/3052701423 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55c8e98ee910 con 0x55c8e9502d80 2024-04-25T11:45:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.493+0000 7fe3b0ff9640 1 -- 172.21.15.90:0/3052701423 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fe3c40a6190 con 0x55c8e9502d80 2024-04-25T11:45:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.493+0000 7fe3d666d740 10 client.18798 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.493+0000 7fe3d666d740 10 client.18798 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:45:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.493+0000 7fe3d666d740 20 client.18798 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.493+0000 7fe3d666d740 1 --2- 172.21.15.90:0/3052701423 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c8e95441b0 0x55c8e996f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.493+0000 7fe3d666d740 4 client.18798 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:45:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.493+0000 7fe3d666d740 1 -- 172.21.15.90:0/3052701423 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c8e9851ee0 con 0x55c8e95441b0 2024-04-25T11:45:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.493+0000 7fe3ccd52640 1 --2- 172.21.15.90:0/3052701423 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c8e95441b0 0x55c8e996f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.494+0000 7fe3ccd52640 1 --2- 172.21.15.90:0/3052701423 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c8e95441b0 0x55c8e996f1d0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec36b786c7bea4cf server_cookie=1c87125d5d718671 in_seq=0 out_seq=0 2024-04-25T11:45:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:17.494+0000 7fe3b0ff9640 10 client.18798 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:18.008 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.021+0000 7fe3b0ff9640 1 -- 172.21.15.90:0/3052701423 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c8e9851ee0 con 0x55c8e95441b0 2024-04-25T11:45:18.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.021+0000 7fe3b0ff9640 10 client.18798 handle_command_reply: tid=0 2024-04-25T11:45:18.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.021+0000 7fe3d666d740 1 client.18798 shutdown 2024-04-25T11:45:18.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.021+0000 7fe3d666d740 1 -- 172.21.15.90:0/3052701423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e9502d80 msgr2=0x55c8e98f1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:18.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.021+0000 7fe3d666d740 1 --2- 172.21.15.90:0/3052701423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e9502d80 0x55c8e98f1b30 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fe3c40df540 tx=0x7fe3c40a39e0 comp rx=0 tx=0).stop 2024-04-25T11:45:18.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.021+0000 7fe3d666d740 1 -- 172.21.15.90:0/3052701423 shutdown_connections 2024-04-25T11:45:18.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.021+0000 7fe3d666d740 1 --2- 172.21.15.90:0/3052701423 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c8e95441b0 0x55c8e996f1d0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:18.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.021+0000 7fe3d666d740 1 --2- 172.21.15.90:0/3052701423 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c8e95234d0 0x55c8e98f25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:18.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.021+0000 7fe3d666d740 1 --2- 172.21.15.90:0/3052701423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c8e951b2f0 0x55c8e98f2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:18.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.021+0000 7fe3b0ff9640 0 client.18798 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:18.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.021+0000 7fe3d666d740 1 --2- 172.21.15.90:0/3052701423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c8e9502d80 0x55c8e98f1b30 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:18.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.021+0000 7fe3d666d740 1 -- 172.21.15.90:0/3052701423 >> 172.21.15.90:0/3052701423 conn(0x55c8e9538300 msgr2=0x55c8e98e3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:18.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.022+0000 7fe3d666d740 1 -- 172.21.15.90:0/3052701423 shutdown_connections 2024-04-25T11:45:18.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.022+0000 7fe3d666d740 1 -- 172.21.15.90:0/3052701423 wait complete. 2024-04-25T11:45:18.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.022+0000 7fe3d666d740 20 client.18798 trim_cache size 0 max 16384 2024-04-25T11:45:18.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:45:18.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:45:18.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:45:18.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:45:18.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.022+0000 7fe3cf7b7640 1 -- 172.21.15.90:0/382354848 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe3b40afc50 msgr2=0x7fe3b40b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:18.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.022+0000 7fe3cf7b7640 1 --2- 172.21.15.90:0/382354848 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe3b40afc50 0x7fe3b40b2160 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x55c8e98e0c60 tx=0x7fe3c0031c10 comp rx=0 tx=0).stop 2024-04-25T11:45:18.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.022+0000 7fe3cf7b7640 1 -- 172.21.15.90:0/382354848 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3c81031c0 msgr2=0x7fe3c819a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:18.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.022+0000 7fe3cf7b7640 1 --2- 172.21.15.90:0/382354848 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3c81031c0 0x7fe3c819a730 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fe3c400e9f0 tx=0x7fe3c400eeb0 comp rx=0 tx=0).stop 2024-04-25T11:45:18.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.022+0000 7fe3cf7b7640 1 -- 172.21.15.90:0/382354848 shutdown_connections 2024-04-25T11:45:18.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.022+0000 7fe3cf7b7640 1 --2- 172.21.15.90:0/382354848 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fe3b40afc50 0x7fe3b40b2160 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:18.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.022+0000 7fe3cf7b7640 1 --2- 172.21.15.90:0/382354848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3c8108f00 0x7fe3c819ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:18.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.022+0000 7fe3cf7b7640 1 --2- 172.21.15.90:0/382354848 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe3c8108a60 0x7fe3c819ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:18.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.022+0000 7fe3cf7b7640 1 --2- 172.21.15.90:0/382354848 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3c81031c0 0x7fe3c819a730 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:18.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.022+0000 7fe3cf7b7640 1 -- 172.21.15.90:0/382354848 >> 172.21.15.90:0/382354848 conn(0x7fe3c80781b0 msgr2=0x7fe3c806a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:18.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.022+0000 7fe3cf7b7640 1 -- 172.21.15.90:0/382354848 shutdown_connections 2024-04-25T11:45:18.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:18.022+0000 7fe3cf7b7640 1 -- 172.21.15.90:0/382354848 wait complete. 2024-04-25T11:45:18.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:17 smithi138 ceph-mon[32328]: pgmap v316: 97 pgs: 20 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 41 active+clean; 34 GiB data, 107 GiB used, 966 GiB / 1.0 TiB avail; 7.7 KiB/s rd, 97 B/s wr, 9 op/s 2024-04-25T11:45:18.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:17 smithi156 ceph-mon[33045]: pgmap v316: 97 pgs: 20 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 41 active+clean; 34 GiB data, 107 GiB used, 966 GiB / 1.0 TiB avail; 7.7 KiB/s rd, 97 B/s wr, 9 op/s 2024-04-25T11:45:18.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:17 smithi090 ceph-mon[29575]: pgmap v316: 97 pgs: 20 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 41 active+clean; 34 GiB data, 107 GiB used, 966 GiB / 1.0 TiB avail; 7.7 KiB/s rd, 97 B/s wr, 9 op/s 2024-04-25T11:45:20.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:19 smithi138 ceph-mon[32328]: pgmap v317: 97 pgs: 20 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 41 active+clean; 34 GiB data, 105 GiB used, 968 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T11:45:20.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:19 smithi156 ceph-mon[33045]: pgmap v317: 97 pgs: 20 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 41 active+clean; 34 GiB data, 105 GiB used, 968 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T11:45:20.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:19 smithi090 ceph-mon[29575]: pgmap v317: 97 pgs: 20 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 41 active+clean; 34 GiB data, 105 GiB used, 968 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T11:45:20.817 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:45:20.818 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:45:20.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.943+0000 7fd912fc5640 1 -- 172.21.15.90:0/2214721399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd90c102b00 msgr2=0x7fd90c102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:20.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.943+0000 7fd912fc5640 1 --2- 172.21.15.90:0/2214721399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd90c102b00 0x7fd90c102f10 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8009900 tx=0x7fd8f802a410 comp rx=0 tx=0).stop 2024-04-25T11:45:20.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.943+0000 7fd912fc5640 1 -- 172.21.15.90:0/2214721399 shutdown_connections 2024-04-25T11:45:20.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.943+0000 7fd912fc5640 1 --2- 172.21.15.90:0/2214721399 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd90c10d390 0x7fd90c10f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.943+0000 7fd912fc5640 1 --2- 172.21.15.90:0/2214721399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd90c102b00 0x7fd90c102f10 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.943+0000 7fd912fc5640 1 --2- 172.21.15.90:0/2214721399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd90c108a30 0x7fd90c1025c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.943+0000 7fd912fc5640 1 -- 172.21.15.90:0/2214721399 >> 172.21.15.90:0/2214721399 conn(0x7fd90c0fe260 msgr2=0x7fd90c100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:20.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.943+0000 7fd912fc5640 1 -- 172.21.15.90:0/2214721399 shutdown_connections 2024-04-25T11:45:20.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.943+0000 7fd912fc5640 1 -- 172.21.15.90:0/2214721399 wait complete. 2024-04-25T11:45:20.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.944+0000 7fd912fc5640 1 Processor -- start 2024-04-25T11:45:20.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.944+0000 7fd912fc5640 1 -- start start 2024-04-25T11:45:20.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.944+0000 7fd912fc5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd90c102b00 0x7fd90c199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:20.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.944+0000 7fd912fc5640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd90c108a30 0x7fd90c199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:20.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.944+0000 7fd912fc5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd90c10d390 0x7fd90c19deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:20.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.944+0000 7fd912fc5640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd90c112bf0 con 0x7fd90c10d390 2024-04-25T11:45:20.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.944+0000 7fd912fc5640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd90c112a70 con 0x7fd90c108a30 2024-04-25T11:45:20.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.944+0000 7fd912fc5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd90c112d70 con 0x7fd90c102b00 2024-04-25T11:45:20.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.944+0000 7fd911562640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd90c10d390 0x7fd90c19deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:20.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.944+0000 7fd911562640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd90c10d390 0x7fd90c19deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59928/0 (socket says 172.21.15.90:59928) 2024-04-25T11:45:20.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.944+0000 7fd911562640 1 -- 172.21.15.90:0/4143676882 learned_addr learned my addr 172.21.15.90:0/4143676882 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:20.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.945+0000 7fd90bfff640 1 --2- 172.21.15.90:0/4143676882 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd90c108a30 0x7fd90c199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:20.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.945+0000 7fd911562640 1 -- 172.21.15.90:0/4143676882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd90c102b00 msgr2=0x7fd90c199690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:20.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.945+0000 7fd911562640 1 --2- 172.21.15.90:0/4143676882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd90c102b00 0x7fd90c199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.945+0000 7fd911562640 1 -- 172.21.15.90:0/4143676882 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd90c108a30 msgr2=0x7fd90c199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:20.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.945+0000 7fd911562640 1 --2- 172.21.15.90:0/4143676882 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd90c108a30 0x7fd90c199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.945+0000 7fd911562640 1 -- 172.21.15.90:0/4143676882 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd90c19e4f0 con 0x7fd90c10d390 2024-04-25T11:45:20.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.945+0000 7fd90bfff640 1 --2- 172.21.15.90:0/4143676882 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd90c108a30 0x7fd90c199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:20.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.945+0000 7fd911562640 1 --2- 172.21.15.90:0/4143676882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd90c10d390 0x7fd90c19deb0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fd904009fd0 tx=0x7fd90400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:20.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.945+0000 7fd909ffb640 1 -- 172.21.15.90:0/4143676882 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd904017070 con 0x7fd90c10d390 2024-04-25T11:45:20.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.945+0000 7fd909ffb640 1 -- 172.21.15.90:0/4143676882 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd904003e20 con 0x7fd90c10d390 2024-04-25T11:45:20.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.945+0000 7fd912fc5640 1 -- 172.21.15.90:0/4143676882 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd90c19e7e0 con 0x7fd90c10d390 2024-04-25T11:45:20.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.945+0000 7fd909ffb640 1 -- 172.21.15.90:0/4143676882 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd904010460 con 0x7fd90c10d390 2024-04-25T11:45:20.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.945+0000 7fd912fc5640 1 -- 172.21.15.90:0/4143676882 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd90c19ece0 con 0x7fd90c10d390 2024-04-25T11:45:20.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.947+0000 7fd909ffb640 1 -- 172.21.15.90:0/4143676882 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fd904003970 con 0x7fd90c10d390 2024-04-25T11:45:20.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.948+0000 7fd919e78740 1 Processor -- start 2024-04-25T11:45:20.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.948+0000 7fd919e78740 1 -- start start 2024-04-25T11:45:20.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.948+0000 7fd919e78740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ae2390 0x565354e84b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:20.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.948+0000 7fd919e78740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x565354ab94d0 0x565354e85080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:20.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.948+0000 7fd909ffb640 1 --2- 172.21.15.90:0/4143676882 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd8fc0afc60 0x7fd8fc0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:20.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.948+0000 7fd919e78740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565354ac16b0 0x565354e855c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:20.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.948+0000 7fd919e78740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x565354de8d80 con 0x565354ae2390 2024-04-25T11:45:20.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.948+0000 7fd919e78740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x565354da28b0 con 0x565354ab94d0 2024-04-25T11:45:20.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.948+0000 7fd919e78740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x565354de4d40 con 0x565354ac16b0 2024-04-25T11:45:20.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.948+0000 7fd90bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ae2390 0x565354e84b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:20.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.948+0000 7fd910d61640 1 --2- 172.21.15.90:0/4143676882 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd8fc0afc60 0x7fd8fc0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:20.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.948+0000 7fd90bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ae2390 0x565354e84b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59930/0 (socket says 172.21.15.90:59930) 2024-04-25T11:45:20.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.948+0000 7fd90bfff640 1 -- 172.21.15.90:0/2177805147 learned_addr learned my addr 172.21.15.90:0/2177805147 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:20.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.948+0000 7fd909ffb640 1 -- 172.21.15.90:0/4143676882 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7fd9040def70 con 0x7fd90c10d390 2024-04-25T11:45:20.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.948+0000 7fd90bfff640 1 --2- 172.21.15.90:0/2177805147 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x565354ab94d0 0x565354e85080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:20.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.948+0000 7fd911562640 1 --2- 172.21.15.90:0/2177805147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565354ac16b0 0x565354e855c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:20.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.949+0000 7fd90bfff640 1 -- 172.21.15.90:0/2177805147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565354ac16b0 msgr2=0x565354e855c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:20.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.949+0000 7fd90bfff640 1 --2- 172.21.15.90:0/2177805147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565354ac16b0 0x565354e855c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.949+0000 7fd90bfff640 1 -- 172.21.15.90:0/2177805147 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x565354ab94d0 msgr2=0x565354e85080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:20.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.949+0000 7fd90bfff640 1 --2- 172.21.15.90:0/2177805147 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x565354ab94d0 0x565354e85080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.949+0000 7fd90bfff640 1 -- 172.21.15.90:0/2177805147 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565354d73a40 con 0x565354ae2390 2024-04-25T11:45:20.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.949+0000 7fd910d61640 1 --2- 172.21.15.90:0/4143676882 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd8fc0afc60 0x7fd8fc0b2170 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd900005fd0 tx=0x7fd900005850 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:20.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.949+0000 7fd90bfff640 1 --2- 172.21.15.90:0/2177805147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ae2390 0x565354e84b40 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8005ab0 tx=0x7fd8f8009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=327a9f6f5708dbcf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:20.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.949+0000 7fd8f6ffd640 1 -- 172.21.15.90:0/2177805147 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f8035070 con 0x565354ae2390 2024-04-25T11:45:20.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.949+0000 7fd8f6ffd640 1 -- 172.21.15.90:0/2177805147 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8f8031910 con 0x565354ae2390 2024-04-25T11:45:20.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.949+0000 7fd8f6ffd640 1 -- 172.21.15.90:0/2177805147 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f8003e00 con 0x565354ae2390 2024-04-25T11:45:20.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.950+0000 7fd919e78740 1 -- 172.21.15.90:0/2177805147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ae2390 msgr2=0x565354e84b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:20.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.950+0000 7fd919e78740 1 --2- 172.21.15.90:0/2177805147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ae2390 0x565354e84b40 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8005ab0 tx=0x7fd8f8009560 comp rx=0 tx=0).stop 2024-04-25T11:45:20.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.950+0000 7fd919e78740 1 -- 172.21.15.90:0/2177805147 shutdown_connections 2024-04-25T11:45:20.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.950+0000 7fd919e78740 1 --2- 172.21.15.90:0/2177805147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565354ac16b0 0x565354e855c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.950+0000 7fd919e78740 1 --2- 172.21.15.90:0/2177805147 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x565354ab94d0 0x565354e85080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.950+0000 7fd919e78740 1 --2- 172.21.15.90:0/2177805147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ae2390 0x565354e84b40 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.950+0000 7fd919e78740 1 -- 172.21.15.90:0/2177805147 >> 172.21.15.90:0/2177805147 conn(0x565354ad64e0 msgr2=0x565354e81420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:20.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.950+0000 7fd919e78740 1 -- 172.21.15.90:0/2177805147 shutdown_connections 2024-04-25T11:45:20.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.950+0000 7fd919e78740 1 -- 172.21.15.90:0/2177805147 wait complete. 2024-04-25T11:45:20.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.950+0000 7fd919e78740 1 -- start start 2024-04-25T11:45:20.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.950+0000 7fd919e78740 1 Processor -- start 2024-04-25T11:45:20.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.951+0000 7fd919e78740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x565354ab94d0 0x565354e91770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:20.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.951+0000 7fd919e78740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ac16b0 0x565354e91cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:20.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.951+0000 7fd919e78740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565354ae2390 0x565354e978c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:20.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.951+0000 7fd919e78740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x565354e943d0 con 0x565354ac16b0 2024-04-25T11:45:20.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.951+0000 7fd919e78740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x565354d6fd10 con 0x565354ab94d0 2024-04-25T11:45:20.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.951+0000 7fd919e78740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x565354e94550 con 0x565354ae2390 2024-04-25T11:45:20.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.951+0000 7fd919e78740 10 client.0 _ll_register_callbacks cb 0x565354d9c210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:20.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.951+0000 7fd90bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ac16b0 0x565354e91cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:20.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.951+0000 7fd90bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ac16b0 0x565354e91cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59936/0 (socket says 172.21.15.90:59936) 2024-04-25T11:45:20.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.951+0000 7fd90bfff640 1 -- 172.21.15.90:0/999177236 learned_addr learned my addr 172.21.15.90:0/999177236 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:20.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.951+0000 7fd910d61640 1 --2- 172.21.15.90:0/999177236 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x565354ab94d0 0x565354e91770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:20.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.951+0000 7fd911562640 1 --2- 172.21.15.90:0/999177236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565354ae2390 0x565354e978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:20.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.951+0000 7fd90bfff640 1 -- 172.21.15.90:0/999177236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565354ae2390 msgr2=0x565354e978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:20.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.951+0000 7fd90bfff640 1 --2- 172.21.15.90:0/999177236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565354ae2390 0x565354e978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.951+0000 7fd90bfff640 1 -- 172.21.15.90:0/999177236 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x565354ab94d0 msgr2=0x565354e91770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:20.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.951+0000 7fd90bfff640 1 --2- 172.21.15.90:0/999177236 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x565354ab94d0 0x565354e91770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.951+0000 7fd90bfff640 1 -- 172.21.15.90:0/999177236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565354bc47d0 con 0x565354ac16b0 2024-04-25T11:45:20.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.951+0000 7fd911562640 1 --2- 172.21.15.90:0/999177236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565354ae2390 0x565354e978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:20.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.951+0000 7fd910d61640 1 --2- 172.21.15.90:0/999177236 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x565354ab94d0 0x565354e91770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:45:20.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.951+0000 7fd90bfff640 1 --2- 172.21.15.90:0/999177236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ac16b0 0x565354e91cb0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8003db0 tx=0x7fd8f8003de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3142e3ca77e72ddd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:20.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.951+0000 7fd8ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:20.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.952+0000 7fd8ebfff640 1 -- 172.21.15.90:0/999177236 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f8035070 con 0x565354ac16b0 2024-04-25T11:45:20.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.952+0000 7fd8ebfff640 1 -- 172.21.15.90:0/999177236 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7fd8f8004230 con 0x565354ac16b0 2024-04-25T11:45:20.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.952+0000 7fd919e78740 1 -- 172.21.15.90:0/999177236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8f8031910 con 0x565354ac16b0 2024-04-25T11:45:20.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.952+0000 7fd8ebfff640 1 -- 172.21.15.90:0/999177236 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd8f8004430 con 0x565354ac16b0 2024-04-25T11:45:20.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.952+0000 7fd919e78740 10 client.18810 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:20.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.952+0000 7fd919e78740 1 -- 172.21.15.90:0/999177236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x565354afa3f0 con 0x565354ac16b0 2024-04-25T11:45:20.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.952+0000 7fd8ebfff640 1 -- 172.21.15.90:0/999177236 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8f8031910 con 0x565354ac16b0 2024-04-25T11:45:20.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.952+0000 7fd8ebfff640 1 -- 172.21.15.90:0/999177236 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fd8f8004b40 con 0x565354ac16b0 2024-04-25T11:45:20.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.952+0000 7fd919e78740 10 client.18810 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:20.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.952+0000 7fd919e78740 10 client.18810 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:45:20.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.952+0000 7fd919e78740 20 client.18810 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:20.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.952+0000 7fd919e78740 1 --2- 172.21.15.90:0/999177236 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x565354aa0f60 0x565354ea3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:20.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.952+0000 7fd919e78740 4 client.18810 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:45:20.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.952+0000 7fd919e78740 1 -- 172.21.15.90:0/999177236 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565354df6990 con 0x565354aa0f60 2024-04-25T11:45:20.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.952+0000 7fd910d61640 1 --2- 172.21.15.90:0/999177236 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x565354aa0f60 0x565354ea3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:20.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.953+0000 7fd910d61640 1 --2- 172.21.15.90:0/999177236 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x565354aa0f60 0x565354ea3980 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c8fcb97d6c000e3 server_cookie=55e95c163f9580ba in_seq=0 out_seq=0 2024-04-25T11:45:20.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.953+0000 7fd8ebfff640 10 client.18810 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:20.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.953+0000 7fd8ebfff640 1 -- 172.21.15.90:0/999177236 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x565354df6990 con 0x565354aa0f60 2024-04-25T11:45:20.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.953+0000 7fd8ebfff640 10 client.18810 handle_command_reply: tid=0 2024-04-25T11:45:20.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.954+0000 7fd919e78740 1 client.18810 shutdown 2024-04-25T11:45:20.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.954+0000 7fd919e78740 1 -- 172.21.15.90:0/999177236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ac16b0 msgr2=0x565354e91cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:20.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.954+0000 7fd919e78740 1 --2- 172.21.15.90:0/999177236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ac16b0 0x565354e91cb0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8003db0 tx=0x7fd8f8003de0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.954+0000 7fd919e78740 1 -- 172.21.15.90:0/999177236 shutdown_connections 2024-04-25T11:45:20.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.954+0000 7fd919e78740 1 --2- 172.21.15.90:0/999177236 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x565354aa0f60 0x565354ea3980 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.954+0000 7fd919e78740 1 --2- 172.21.15.90:0/999177236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565354ae2390 0x565354e978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.954+0000 7fd919e78740 1 --2- 172.21.15.90:0/999177236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ac16b0 0x565354e91cb0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.954+0000 7fd919e78740 1 --2- 172.21.15.90:0/999177236 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x565354ab94d0 0x565354e91770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.954+0000 7fd8ebfff640 0 client.18810 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:20.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.954+0000 7fd919e78740 1 -- 172.21.15.90:0/999177236 >> 172.21.15.90:0/999177236 conn(0x565354ad64e0 msgr2=0x565354e85430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:20.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.954+0000 7fd919e78740 1 -- 172.21.15.90:0/999177236 shutdown_connections 2024-04-25T11:45:20.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.954+0000 7fd919e78740 1 -- 172.21.15.90:0/999177236 wait complete. 2024-04-25T11:45:20.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.954+0000 7fd919e78740 20 client.18810 trim_cache size 0 max 16384 2024-04-25T11:45:20.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.971+0000 7fd919e78740 1 Processor -- start 2024-04-25T11:45:20.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.971+0000 7fd919e78740 1 -- start start 2024-04-25T11:45:20.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.972+0000 7fd919e78740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x565354ab94d0 0x565354e81430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:20.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.972+0000 7fd919e78740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565354ac16b0 0x565354e90430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:20.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.972+0000 7fd919e78740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ae2390 0x565354e977c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:20.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.972+0000 7fd919e78740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd90c1ad900 con 0x565354ae2390 2024-04-25T11:45:20.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.972+0000 7fd919e78740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd90c1af440 con 0x565354ab94d0 2024-04-25T11:45:20.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.972+0000 7fd919e78740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd90c1adfd0 con 0x565354ac16b0 2024-04-25T11:45:20.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.972+0000 7fd911562640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ae2390 0x565354e977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:20.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.972+0000 7fd911562640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ae2390 0x565354e977c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59942/0 (socket says 172.21.15.90:59942) 2024-04-25T11:45:20.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.972+0000 7fd911562640 1 -- 172.21.15.90:0/96660064 learned_addr learned my addr 172.21.15.90:0/96660064 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:20.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.972+0000 7fd90bfff640 1 --2- 172.21.15.90:0/96660064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565354ac16b0 0x565354e90430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:20.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.972+0000 7fd910d61640 1 --2- 172.21.15.90:0/96660064 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x565354ab94d0 0x565354e81430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:20.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.972+0000 7fd911562640 1 -- 172.21.15.90:0/96660064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565354ac16b0 msgr2=0x565354e90430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:20.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.972+0000 7fd911562640 1 --2- 172.21.15.90:0/96660064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565354ac16b0 0x565354e90430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.972+0000 7fd911562640 1 -- 172.21.15.90:0/96660064 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x565354ab94d0 msgr2=0x565354e81430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:20.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.972+0000 7fd911562640 1 --2- 172.21.15.90:0/96660064 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x565354ab94d0 0x565354e81430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.972+0000 7fd911562640 1 -- 172.21.15.90:0/96660064 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565354aba0f0 con 0x565354ae2390 2024-04-25T11:45:20.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.972+0000 7fd910d61640 1 --2- 172.21.15.90:0/96660064 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x565354ab94d0 0x565354e81430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:20.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.972+0000 7fd911562640 1 --2- 172.21.15.90:0/96660064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ae2390 0x565354e977c0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fd9040dcc10 tx=0x7fd9040a6ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67ccbff6c76c5378 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:20.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.972+0000 7fd8f67fc640 1 -- 172.21.15.90:0/96660064 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd904020040 con 0x565354ae2390 2024-04-25T11:45:20.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.972+0000 7fd8f67fc640 1 -- 172.21.15.90:0/96660064 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9040ab410 con 0x565354ae2390 2024-04-25T11:45:20.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.972+0000 7fd8f67fc640 1 -- 172.21.15.90:0/96660064 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9040dc890 con 0x565354ae2390 2024-04-25T11:45:20.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.973+0000 7fd919e78740 1 -- 172.21.15.90:0/96660064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ae2390 msgr2=0x565354e977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:20.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.973+0000 7fd919e78740 1 --2- 172.21.15.90:0/96660064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ae2390 0x565354e977c0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fd9040dcc10 tx=0x7fd9040a6ae0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.973+0000 7fd919e78740 1 -- 172.21.15.90:0/96660064 shutdown_connections 2024-04-25T11:45:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.973+0000 7fd919e78740 1 --2- 172.21.15.90:0/96660064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ae2390 0x565354e977c0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.973+0000 7fd919e78740 1 --2- 172.21.15.90:0/96660064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565354ac16b0 0x565354e90430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.973+0000 7fd919e78740 1 --2- 172.21.15.90:0/96660064 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x565354ab94d0 0x565354e81430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.973+0000 7fd919e78740 1 -- 172.21.15.90:0/96660064 >> 172.21.15.90:0/96660064 conn(0x565354ad64e0 msgr2=0x565354e8fd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.973+0000 7fd919e78740 1 -- 172.21.15.90:0/96660064 shutdown_connections 2024-04-25T11:45:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.973+0000 7fd919e78740 1 -- 172.21.15.90:0/96660064 wait complete. 2024-04-25T11:45:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.973+0000 7fd919e78740 1 -- start start 2024-04-25T11:45:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.974+0000 7fd919e78740 1 Processor -- start 2024-04-25T11:45:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.974+0000 7fd919e78740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ab94d0 0x565354e9aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.974+0000 7fd919e78740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x565354ac16b0 0x565354e9af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.974+0000 7fd919e78740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565354ae2390 0x565354e9b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.974+0000 7fd919e78740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd90c1aed70 con 0x565354ab94d0 2024-04-25T11:45:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.974+0000 7fd919e78740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd90c1ae6a0 con 0x565354ac16b0 2024-04-25T11:45:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.974+0000 7fd919e78740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd90c1ab900 con 0x565354ae2390 2024-04-25T11:45:20.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.974+0000 7fd919e78740 10 client.0 _ll_register_callbacks cb 0x7fd8f8032d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:20.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.974+0000 7fd910d61640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ab94d0 0x565354e9aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:20.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.974+0000 7fd910d61640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ab94d0 0x565354e9aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59948/0 (socket says 172.21.15.90:59948) 2024-04-25T11:45:20.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.974+0000 7fd910d61640 1 -- 172.21.15.90:0/2199207779 learned_addr learned my addr 172.21.15.90:0/2199207779 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:20.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.974+0000 7fd911562640 1 --2- 172.21.15.90:0/2199207779 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565354ae2390 0x565354e9b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:20.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.974+0000 7fd90bfff640 1 --2- 172.21.15.90:0/2199207779 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x565354ac16b0 0x565354e9af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:20.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.974+0000 7fd910d61640 1 -- 172.21.15.90:0/2199207779 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565354ae2390 msgr2=0x565354e9b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:20.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.974+0000 7fd910d61640 1 --2- 172.21.15.90:0/2199207779 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565354ae2390 0x565354e9b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.974+0000 7fd910d61640 1 -- 172.21.15.90:0/2199207779 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x565354ac16b0 msgr2=0x565354e9af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:20.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.974+0000 7fd910d61640 1 --2- 172.21.15.90:0/2199207779 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x565354ac16b0 0x565354e9af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.974+0000 7fd910d61640 1 -- 172.21.15.90:0/2199207779 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565354da94b0 con 0x565354ab94d0 2024-04-25T11:45:20.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.974+0000 7fd911562640 1 --2- 172.21.15.90:0/2199207779 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565354ae2390 0x565354e9b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:20.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.975+0000 7fd910d61640 1 --2- 172.21.15.90:0/2199207779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ab94d0 0x565354e9aa20 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fd90000e010 tx=0x7fd900005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d2a355da783fdbf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:20.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.975+0000 7fd8ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:20.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.975+0000 7fd8ebfff640 1 -- 172.21.15.90:0/2199207779 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd900010e00 con 0x565354ab94d0 2024-04-25T11:45:20.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.975+0000 7fd8ebfff640 1 -- 172.21.15.90:0/2199207779 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7fd900026460 con 0x565354ab94d0 2024-04-25T11:45:20.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.975+0000 7fd919e78740 1 -- 172.21.15.90:0/2199207779 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9040ab410 con 0x565354ab94d0 2024-04-25T11:45:20.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.975+0000 7fd8ebfff640 1 -- 172.21.15.90:0/2199207779 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd900035050 con 0x565354ab94d0 2024-04-25T11:45:20.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.975+0000 7fd919e78740 10 client.18816 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:20.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.975+0000 7fd919e78740 1 -- 172.21.15.90:0/2199207779 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x565354f0a980 con 0x565354ab94d0 2024-04-25T11:45:20.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.975+0000 7fd8ebfff640 1 -- 172.21.15.90:0/2199207779 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9040ab410 con 0x565354ab94d0 2024-04-25T11:45:20.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.975+0000 7fd8ebfff640 1 -- 172.21.15.90:0/2199207779 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fd900026ce0 con 0x565354ab94d0 2024-04-25T11:45:20.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.975+0000 7fd919e78740 10 client.18816 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:20.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.975+0000 7fd919e78740 10 client.18816 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:45:20.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.975+0000 7fd919e78740 20 client.18816 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:20.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.975+0000 7fd919e78740 1 --2- 172.21.15.90:0/2199207779 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x565354a763a0 0x565354f0d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:20.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.975+0000 7fd919e78740 4 client.18816 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:45:20.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.975+0000 7fd919e78740 1 -- 172.21.15.90:0/2199207779 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x565354df0800 con 0x565354a763a0 2024-04-25T11:45:20.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.975+0000 7fd90bfff640 1 --2- 172.21.15.90:0/2199207779 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x565354a763a0 0x565354f0d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:20.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.976+0000 7fd8ebfff640 10 client.18816 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:20.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.976+0000 7fd90bfff640 1 --2- 172.21.15.90:0/2199207779 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x565354a763a0 0x565354f0d140 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe712c77f67c19d1 server_cookie=631d53792832b9e3 in_seq=0 out_seq=0 2024-04-25T11:45:20.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.976+0000 7fd8ebfff640 1 -- 172.21.15.90:0/2199207779 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x565354df0800 con 0x565354a763a0 2024-04-25T11:45:20.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.976+0000 7fd8ebfff640 10 client.18816 handle_command_reply: tid=0 2024-04-25T11:45:20.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.976+0000 7fd919e78740 1 client.18816 shutdown 2024-04-25T11:45:20.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.976+0000 7fd919e78740 1 -- 172.21.15.90:0/2199207779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ab94d0 msgr2=0x565354e9aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:20.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.976+0000 7fd919e78740 1 --2- 172.21.15.90:0/2199207779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ab94d0 0x565354e9aa20 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fd90000e010 tx=0x7fd900005c90 comp rx=0 tx=0).stop 2024-04-25T11:45:20.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.976+0000 7fd919e78740 1 -- 172.21.15.90:0/2199207779 shutdown_connections 2024-04-25T11:45:20.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.976+0000 7fd919e78740 1 --2- 172.21.15.90:0/2199207779 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x565354a763a0 0x565354f0d140 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.976+0000 7fd919e78740 1 --2- 172.21.15.90:0/2199207779 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x565354ae2390 0x565354e9b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.976+0000 7fd8ebfff640 0 client.18816 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:20.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.976+0000 7fd919e78740 1 --2- 172.21.15.90:0/2199207779 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x565354ac16b0 0x565354e9af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.976+0000 7fd919e78740 1 --2- 172.21.15.90:0/2199207779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565354ab94d0 0x565354e9aa20 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.976+0000 7fd919e78740 1 -- 172.21.15.90:0/2199207779 >> 172.21.15.90:0/2199207779 conn(0x565354ad64e0 msgr2=0x565354e8fd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:20.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.976+0000 7fd919e78740 1 -- 172.21.15.90:0/2199207779 shutdown_connections 2024-04-25T11:45:20.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.976+0000 7fd919e78740 1 -- 172.21.15.90:0/2199207779 wait complete. 2024-04-25T11:45:20.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.976+0000 7fd919e78740 20 client.18816 trim_cache size 0 max 16384 2024-04-25T11:45:20.969 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:45:20.969 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-25T11:45:20.969 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-25T11:45:20.969 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:45:20.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.977+0000 7fd912fc5640 1 -- 172.21.15.90:0/4143676882 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd8fc0afc60 msgr2=0x7fd8fc0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:20.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.977+0000 7fd912fc5640 1 --2- 172.21.15.90:0/4143676882 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd8fc0afc60 0x7fd8fc0b2170 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd900005fd0 tx=0x7fd900005850 comp rx=0 tx=0).stop 2024-04-25T11:45:20.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.977+0000 7fd912fc5640 1 -- 172.21.15.90:0/4143676882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd90c10d390 msgr2=0x7fd90c19deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:20.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.977+0000 7fd912fc5640 1 --2- 172.21.15.90:0/4143676882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd90c10d390 0x7fd90c19deb0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fd904009fd0 tx=0x7fd90400b260 comp rx=0 tx=0).stop 2024-04-25T11:45:20.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.977+0000 7fd912fc5640 1 -- 172.21.15.90:0/4143676882 shutdown_connections 2024-04-25T11:45:20.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.977+0000 7fd912fc5640 1 --2- 172.21.15.90:0/4143676882 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd8fc0afc60 0x7fd8fc0b2170 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.977+0000 7fd912fc5640 1 --2- 172.21.15.90:0/4143676882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd90c10d390 0x7fd90c19deb0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.977+0000 7fd912fc5640 1 --2- 172.21.15.90:0/4143676882 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd90c108a30 0x7fd90c199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.977+0000 7fd912fc5640 1 --2- 172.21.15.90:0/4143676882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd90c102b00 0x7fd90c199690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:20.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.977+0000 7fd912fc5640 1 -- 172.21.15.90:0/4143676882 >> 172.21.15.90:0/4143676882 conn(0x7fd90c0fe260 msgr2=0x7fd90c0fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:20.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.977+0000 7fd912fc5640 1 -- 172.21.15.90:0/4143676882 shutdown_connections 2024-04-25T11:45:20.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:20.977+0000 7fd912fc5640 1 -- 172.21.15.90:0/4143676882 wait complete. 2024-04-25T11:45:20.973 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T11:45:20.974 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:45:21.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.097+0000 7f53cbebd640 1 -- 172.21.15.90:0/3615258422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53c4108a40 msgr2=0x7f53c4102620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.097+0000 7f53cbebd640 1 --2- 172.21.15.90:0/3615258422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53c4108a40 0x7f53c4102620 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f53b4009900 tx=0x7f53b402a410 comp rx=0 tx=0).stop 2024-04-25T11:45:21.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.097+0000 7f53cbebd640 1 -- 172.21.15.90:0/3615258422 shutdown_connections 2024-04-25T11:45:21.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.097+0000 7f53cbebd640 1 --2- 172.21.15.90:0/3615258422 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f53c410d3a0 0x7f53c410f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.097+0000 7f53cbebd640 1 --2- 172.21.15.90:0/3615258422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53c4102b60 0x7f53c4102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.097+0000 7f53cbebd640 1 --2- 172.21.15.90:0/3615258422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53c4108a40 0x7f53c4102620 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.097+0000 7f53cbebd640 1 -- 172.21.15.90:0/3615258422 >> 172.21.15.90:0/3615258422 conn(0x7f53c4078040 msgr2=0x7f53c4078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:21.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.097+0000 7f53cbebd640 1 -- 172.21.15.90:0/3615258422 shutdown_connections 2024-04-25T11:45:21.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.097+0000 7f53cbebd640 1 -- 172.21.15.90:0/3615258422 wait complete. 2024-04-25T11:45:21.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.097+0000 7f53cbebd640 1 Processor -- start 2024-04-25T11:45:21.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.098+0000 7f53cbebd640 1 -- start start 2024-04-25T11:45:21.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.098+0000 7f53cbebd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53c4102b60 0x7f53c419b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.098+0000 7f53cbebd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f53c4108a40 0x7f53c419bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.098+0000 7f53cbebd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53c410d3a0 0x7f53c419c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.098+0000 7f53cbebd640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f53c4112c60 con 0x7f53c410d3a0 2024-04-25T11:45:21.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.098+0000 7f53cbebd640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f53c4112ae0 con 0x7f53c4108a40 2024-04-25T11:45:21.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.098+0000 7f53cbebd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f53c4112de0 con 0x7f53c4102b60 2024-04-25T11:45:21.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.098+0000 7f53ca45a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53c410d3a0 0x7f53c419c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.098+0000 7f53ca45a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53c410d3a0 0x7f53c419c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59968/0 (socket says 172.21.15.90:59968) 2024-04-25T11:45:21.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.098+0000 7f53ca45a640 1 -- 172.21.15.90:0/2710997881 learned_addr learned my addr 172.21.15.90:0/2710997881 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:21.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.098+0000 7f53c9c59640 1 --2- 172.21.15.90:0/2710997881 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53c4102b60 0x7f53c419b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.098+0000 7f53c9458640 1 --2- 172.21.15.90:0/2710997881 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f53c4108a40 0x7f53c419bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.098+0000 7f53ca45a640 1 -- 172.21.15.90:0/2710997881 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53c4102b60 msgr2=0x7f53c419b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.098+0000 7f53ca45a640 1 --2- 172.21.15.90:0/2710997881 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53c4102b60 0x7f53c419b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.098+0000 7f53ca45a640 1 -- 172.21.15.90:0/2710997881 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f53c4108a40 msgr2=0x7f53c419bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.098+0000 7f53ca45a640 1 --2- 172.21.15.90:0/2710997881 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f53c4108a40 0x7f53c419bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.098+0000 7f53ca45a640 1 -- 172.21.15.90:0/2710997881 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53c41a68c0 con 0x7f53c410d3a0 2024-04-25T11:45:21.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.098+0000 7f53c9458640 1 --2- 172.21.15.90:0/2710997881 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f53c4108a40 0x7f53c419bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:21.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.098+0000 7f53ca45a640 1 --2- 172.21.15.90:0/2710997881 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53c410d3a0 0x7f53c419c020 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f53bc00eaa0 tx=0x7f53bc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:21.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.098+0000 7f53baffd640 1 -- 172.21.15.90:0/2710997881 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53bc00cdb0 con 0x7f53c410d3a0 2024-04-25T11:45:21.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.099+0000 7f53baffd640 1 -- 172.21.15.90:0/2710997881 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53bc005020 con 0x7f53c410d3a0 2024-04-25T11:45:21.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.099+0000 7f53cbebd640 1 -- 172.21.15.90:0/2710997881 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53c41a6bb0 con 0x7f53c410d3a0 2024-04-25T11:45:21.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.099+0000 7f53baffd640 1 -- 172.21.15.90:0/2710997881 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53bc0116d0 con 0x7f53c410d3a0 2024-04-25T11:45:21.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.099+0000 7f53cbebd640 1 -- 172.21.15.90:0/2710997881 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53c41a70b0 con 0x7f53c410d3a0 2024-04-25T11:45:21.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.100+0000 7f53cbebd640 1 -- 172.21.15.90:0/2710997881 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53c406a190 con 0x7f53c410d3a0 2024-04-25T11:45:21.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.101+0000 7f53baffd640 1 -- 172.21.15.90:0/2710997881 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f53bc0051d0 con 0x7f53c410d3a0 2024-04-25T11:45:21.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.102+0000 7f53baffd640 1 --2- 172.21.15.90:0/2710997881 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f53b00afc60 0x7f53b00b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.102+0000 7f53c9c59640 1 --2- 172.21.15.90:0/2710997881 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f53b00afc60 0x7f53b00b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.102+0000 7f53baffd640 1 -- 172.21.15.90:0/2710997881 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7f53bc0df3f0 con 0x7f53c410d3a0 2024-04-25T11:45:21.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.102+0000 7f53c9c59640 1 --2- 172.21.15.90:0/2710997881 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f53b00afc60 0x7f53b00b2170 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f53b4000c00 tx=0x7f53b4031400 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:21.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.106+0000 7f53baffd640 1 -- 172.21.15.90:0/2710997881 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f53bc016030 con 0x7f53c410d3a0 2024-04-25T11:45:21.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.322+0000 7f53cbebd640 1 -- 172.21.15.90:0/2710997881 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53c40753e0 con 0x7f53c410d3a0 2024-04-25T11:45:21.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.323+0000 7f53baffd640 1 -- 172.21.15.90:0/2710997881 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11930 (secure 0 0 0) 0x7f53bc0a7be0 con 0x7f53c410d3a0 2024-04-25T11:45:21.309 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:45:21.309 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:39:20.469254+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24458,24431,24499,24475,24524]},"id":1}]} 2024-04-25T11:45:21.309 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 28 2024-04-25T11:45:21.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.325+0000 7f53cbebd640 1 -- 172.21.15.90:0/2710997881 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f53b00afc60 msgr2=0x7f53b00b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.325+0000 7f53cbebd640 1 --2- 172.21.15.90:0/2710997881 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f53b00afc60 0x7f53b00b2170 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f53b4000c00 tx=0x7f53b4031400 comp rx=0 tx=0).stop 2024-04-25T11:45:21.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.325+0000 7f53cbebd640 1 -- 172.21.15.90:0/2710997881 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53c410d3a0 msgr2=0x7f53c419c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.325+0000 7f53cbebd640 1 --2- 172.21.15.90:0/2710997881 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53c410d3a0 0x7f53c419c020 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f53bc00eaa0 tx=0x7f53bc00ef60 comp rx=0 tx=0).stop 2024-04-25T11:45:21.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.325+0000 7f53cbebd640 1 -- 172.21.15.90:0/2710997881 shutdown_connections 2024-04-25T11:45:21.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.325+0000 7f53cbebd640 1 --2- 172.21.15.90:0/2710997881 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f53b00afc60 0x7f53b00b2170 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.325+0000 7f53cbebd640 1 --2- 172.21.15.90:0/2710997881 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53c410d3a0 0x7f53c419c020 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.325+0000 7f53cbebd640 1 --2- 172.21.15.90:0/2710997881 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f53c4108a40 0x7f53c419bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.325+0000 7f53cbebd640 1 --2- 172.21.15.90:0/2710997881 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53c4102b60 0x7f53c419b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.325+0000 7f53cbebd640 1 -- 172.21.15.90:0/2710997881 >> 172.21.15.90:0/2710997881 conn(0x7f53c4078040 msgr2=0x7f53c41040d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:21.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.325+0000 7f53cbebd640 1 -- 172.21.15.90:0/2710997881 shutdown_connections 2024-04-25T11:45:21.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.325+0000 7f53cbebd640 1 -- 172.21.15.90:0/2710997881 wait complete. 2024-04-25T11:45:21.322 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T11:45:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.447+0000 7f431e364640 1 -- 172.21.15.90:0/3929875330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4318109f90 msgr2=0x7f43181020b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.447+0000 7f431e364640 1 --2- 172.21.15.90:0/3929875330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4318109f90 0x7f43181020b0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f4308009900 tx=0x7f430802a410 comp rx=0 tx=0).stop 2024-04-25T11:45:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.447+0000 7f431e364640 1 -- 172.21.15.90:0/3929875330 shutdown_connections 2024-04-25T11:45:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.447+0000 7f431e364640 1 --2- 172.21.15.90:0/3929875330 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4318102720 0x7f4318110570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.447+0000 7f431e364640 1 --2- 172.21.15.90:0/3929875330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4318109f90 0x7f43181020b0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.447+0000 7f431e364640 1 --2- 172.21.15.90:0/3929875330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4318109af0 0x7f4318101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.447+0000 7f431e364640 1 -- 172.21.15.90:0/3929875330 >> 172.21.15.90:0/3929875330 conn(0x7f4318077880 msgr2=0x7f4318077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:21.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.447+0000 7f431e364640 1 -- 172.21.15.90:0/3929875330 shutdown_connections 2024-04-25T11:45:21.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.447+0000 7f431e364640 1 -- 172.21.15.90:0/3929875330 wait complete. 2024-04-25T11:45:21.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.448+0000 7f431e364640 1 Processor -- start 2024-04-25T11:45:21.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.448+0000 7f431e364640 1 -- start start 2024-04-25T11:45:21.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.448+0000 7f431e364640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4318102720 0x7f431819a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.448+0000 7f431e364640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4318109af0 0x7f431819ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.448+0000 7f431e364640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4318109f90 0x7f431819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.448+0000 7f431e364640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4318113ce0 con 0x7f4318109f90 2024-04-25T11:45:21.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.448+0000 7f431e364640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4318113b60 con 0x7f4318102720 2024-04-25T11:45:21.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.448+0000 7f431e364640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4318113e60 con 0x7f4318109af0 2024-04-25T11:45:21.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.448+0000 7f431c901640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4318109f90 0x7f431819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.448+0000 7f431c901640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4318109f90 0x7f431819ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41942/0 (socket says 172.21.15.90:41942) 2024-04-25T11:45:21.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.448+0000 7f431c901640 1 -- 172.21.15.90:0/1238846164 learned_addr learned my addr 172.21.15.90:0/1238846164 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:21.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.448+0000 7f43177fe640 1 --2- 172.21.15.90:0/1238846164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4318109af0 0x7f431819ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.448+0000 7f4317fff640 1 --2- 172.21.15.90:0/1238846164 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4318102720 0x7f431819a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.449+0000 7f431c901640 1 -- 172.21.15.90:0/1238846164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4318109af0 msgr2=0x7f431819ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.449+0000 7f431c901640 1 --2- 172.21.15.90:0/1238846164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4318109af0 0x7f431819ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.449+0000 7f431c901640 1 -- 172.21.15.90:0/1238846164 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4318102720 msgr2=0x7f431819a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.449+0000 7f431c901640 1 --2- 172.21.15.90:0/1238846164 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4318102720 0x7f431819a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.449+0000 7f431c901640 1 -- 172.21.15.90:0/1238846164 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f431819f560 con 0x7f4318109f90 2024-04-25T11:45:21.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.449+0000 7f4317fff640 1 --2- 172.21.15.90:0/1238846164 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4318102720 0x7f431819a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:21.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.449+0000 7f431c901640 1 --2- 172.21.15.90:0/1238846164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4318109f90 0x7f431819ef20 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f431000cab0 tx=0x7f431000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:21.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.449+0000 7f43157fa640 1 -- 172.21.15.90:0/1238846164 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f431000ae20 con 0x7f4318109f90 2024-04-25T11:45:21.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.449+0000 7f431e364640 1 -- 172.21.15.90:0/1238846164 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f431819f850 con 0x7f4318109f90 2024-04-25T11:45:21.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.449+0000 7f43157fa640 1 -- 172.21.15.90:0/1238846164 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f431000f900 con 0x7f4318109f90 2024-04-25T11:45:21.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.449+0000 7f43157fa640 1 -- 172.21.15.90:0/1238846164 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4310018a50 con 0x7f4318109f90 2024-04-25T11:45:21.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.449+0000 7f431e364640 1 -- 172.21.15.90:0/1238846164 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43181a71d0 con 0x7f4318109f90 2024-04-25T11:45:21.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.451+0000 7f4325211740 1 Processor -- start 2024-04-25T11:45:21.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.451+0000 7f43157fa640 1 -- 172.21.15.90:0/1238846164 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f431000f450 con 0x7f4318109f90 2024-04-25T11:45:21.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.451+0000 7f4325211740 1 -- start start 2024-04-25T11:45:21.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.452+0000 7f4325211740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595eaf5e390 0x5595eb2ff810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.452+0000 7f4325211740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf354d0 0x5595eb2ffd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.452+0000 7f4325211740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595eaf3d6b0 0x5595eb30c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.452+0000 7f4325211740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5595eb264d80 con 0x5595eaf354d0 2024-04-25T11:45:21.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.452+0000 7f4325211740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5595eb21ea20 con 0x5595eaf3d6b0 2024-04-25T11:45:21.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.452+0000 7f4325211740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5595eb260d40 con 0x5595eaf5e390 2024-04-25T11:45:21.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.452+0000 7f4317fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf354d0 0x5595eb2ffd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.452+0000 7f4317fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf354d0 0x5595eb2ffd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41952/0 (socket says 172.21.15.90:41952) 2024-04-25T11:45:21.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.452+0000 7f4317fff640 1 -- 172.21.15.90:0/1853849488 learned_addr learned my addr 172.21.15.90:0/1853849488 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:21.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.452+0000 7f43177fe640 1 --2- 172.21.15.90:0/1853849488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595eaf5e390 0x5595eb2ff810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.452+0000 7f43177fe640 1 --2- 172.21.15.90:0/1853849488 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595eaf3d6b0 0x5595eb30c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.452+0000 7f4317fff640 1 -- 172.21.15.90:0/1853849488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595eaf5e390 msgr2=0x5595eb2ff810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.452+0000 7f4317fff640 1 --2- 172.21.15.90:0/1853849488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595eaf5e390 0x5595eb2ff810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.452+0000 7f4317fff640 1 -- 172.21.15.90:0/1853849488 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595eaf3d6b0 msgr2=0x5595eb30c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.452+0000 7f4317fff640 1 --2- 172.21.15.90:0/1853849488 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595eaf3d6b0 0x5595eb30c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.452+0000 7f4317fff640 1 -- 172.21.15.90:0/1853849488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595eb229c30 con 0x5595eaf354d0 2024-04-25T11:45:21.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.452+0000 7f43157fa640 1 --2- 172.21.15.90:0/1238846164 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f43000afc60 0x7f43000b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.452+0000 7f43177fe640 1 --2- 172.21.15.90:0/1853849488 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595eaf3d6b0 0x5595eb30c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:21.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.452+0000 7f43177fe640 1 --2- 172.21.15.90:0/1238846164 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f43000afc60 0x7f43000b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.452+0000 7f4317fff640 1 --2- 172.21.15.90:0/1853849488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf354d0 0x5595eb2ffd50 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f430c00bb70 tx=0x7f430c00d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=647422a1f74c981d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:21.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.452+0000 7f43157fa640 1 -- 172.21.15.90:0/1238846164 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7f43100df9e0 con 0x7f4318109f90 2024-04-25T11:45:21.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.452+0000 7f42fe7fc640 1 -- 172.21.15.90:0/1853849488 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430c017620 con 0x5595eaf354d0 2024-04-25T11:45:21.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.453+0000 7f42fe7fc640 1 -- 172.21.15.90:0/1853849488 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f430c017c70 con 0x5595eaf354d0 2024-04-25T11:45:21.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.453+0000 7f42fe7fc640 1 -- 172.21.15.90:0/1853849488 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430c00fc00 con 0x5595eaf354d0 2024-04-25T11:45:21.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.453+0000 7f4325211740 1 -- 172.21.15.90:0/1853849488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf354d0 msgr2=0x5595eb2ffd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.453+0000 7f4325211740 1 --2- 172.21.15.90:0/1853849488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf354d0 0x5595eb2ffd50 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f430c00bb70 tx=0x7f430c00d710 comp rx=0 tx=0).stop 2024-04-25T11:45:21.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.453+0000 7f43177fe640 1 --2- 172.21.15.90:0/1238846164 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f43000afc60 0x7f43000b2170 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4308000c00 tx=0x7f4308031400 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:21.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.453+0000 7f4325211740 1 -- 172.21.15.90:0/1853849488 shutdown_connections 2024-04-25T11:45:21.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.453+0000 7f4325211740 1 --2- 172.21.15.90:0/1853849488 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595eaf3d6b0 0x5595eb30c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.453+0000 7f4325211740 1 --2- 172.21.15.90:0/1853849488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf354d0 0x5595eb2ffd50 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.453+0000 7f4325211740 1 --2- 172.21.15.90:0/1853849488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595eaf5e390 0x5595eb2ff810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.453+0000 7f4325211740 1 -- 172.21.15.90:0/1853849488 >> 172.21.15.90:0/1853849488 conn(0x5595eaf524e0 msgr2=0x5595eb2fbfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:21.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.453+0000 7f4325211740 1 -- 172.21.15.90:0/1853849488 shutdown_connections 2024-04-25T11:45:21.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.454+0000 7f4325211740 1 -- 172.21.15.90:0/1853849488 wait complete. 2024-04-25T11:45:21.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.454+0000 7f4325211740 1 -- start start 2024-04-25T11:45:21.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.454+0000 7f4325211740 1 Processor -- start 2024-04-25T11:45:21.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.454+0000 7f4325211740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595eaf354d0 0x5595eb30a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.454+0000 7f4325211740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595eaf3d6b0 0x5595eb30a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.454+0000 7f4325211740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf5e390 0x5595eb3162b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.454+0000 7f4325211740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5595eb30ce30 con 0x5595eaf5e390 2024-04-25T11:45:21.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.454+0000 7f4325211740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5595eb1ebdc0 con 0x5595eaf3d6b0 2024-04-25T11:45:21.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.454+0000 7f4325211740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5595eb30cfb0 con 0x5595eaf354d0 2024-04-25T11:45:21.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.454+0000 7f4325211740 10 client.0 _ll_register_callbacks cb 0x5595eb207600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:21.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.454+0000 7f431c901640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf5e390 0x5595eb3162b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.455+0000 7f431c901640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf5e390 0x5595eb3162b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41958/0 (socket says 172.21.15.90:41958) 2024-04-25T11:45:21.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.455+0000 7f431c901640 1 -- 172.21.15.90:0/943356809 learned_addr learned my addr 172.21.15.90:0/943356809 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:21.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.455+0000 7f4317fff640 1 --2- 172.21.15.90:0/943356809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595eaf354d0 0x5595eb30a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.455+0000 7f43177fe640 1 --2- 172.21.15.90:0/943356809 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595eaf3d6b0 0x5595eb30a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.455+0000 7f431c901640 1 -- 172.21.15.90:0/943356809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595eaf354d0 msgr2=0x5595eb30a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.455+0000 7f431c901640 1 --2- 172.21.15.90:0/943356809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595eaf354d0 0x5595eb30a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.455+0000 7f431c901640 1 -- 172.21.15.90:0/943356809 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595eaf3d6b0 msgr2=0x5595eb30a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.455+0000 7f431c901640 1 --2- 172.21.15.90:0/943356809 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595eaf3d6b0 0x5595eb30a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.455+0000 7f431c901640 1 -- 172.21.15.90:0/943356809 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595eaf763f0 con 0x5595eaf5e390 2024-04-25T11:45:21.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.455+0000 7f43177fe640 1 --2- 172.21.15.90:0/943356809 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595eaf3d6b0 0x5595eb30a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:21.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.455+0000 7f431c901640 1 --2- 172.21.15.90:0/943356809 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf5e390 0x5595eb3162b0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f43100a77d0 tx=0x7f43100a75e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58a5ccbbf7b2c3df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:21.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.455+0000 7f42f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:21.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.455+0000 7f42f77fe640 1 -- 172.21.15.90:0/943356809 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43100c1060 con 0x5595eaf5e390 2024-04-25T11:45:21.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.455+0000 7f42f77fe640 1 -- 172.21.15.90:0/943356809 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7f43100a40c0 con 0x5595eaf5e390 2024-04-25T11:45:21.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.455+0000 7f4325211740 1 -- 172.21.15.90:0/943356809 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f430c017c70 con 0x5595eaf5e390 2024-04-25T11:45:21.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.455+0000 7f42f77fe640 1 -- 172.21.15.90:0/943356809 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f43100abe80 con 0x5595eaf5e390 2024-04-25T11:45:21.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.455+0000 7f4325211740 10 client.18834 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:21.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.455+0000 7f4325211740 1 -- 172.21.15.90:0/943356809 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5595eb0407d0 con 0x5595eaf5e390 2024-04-25T11:45:21.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.455+0000 7f42f77fe640 1 -- 172.21.15.90:0/943356809 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f430c017c70 con 0x5595eaf5e390 2024-04-25T11:45:21.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.455+0000 7f42f77fe640 1 -- 172.21.15.90:0/943356809 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f43100b0550 con 0x5595eaf5e390 2024-04-25T11:45:21.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.456+0000 7f4325211740 10 client.18834 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:21.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.456+0000 7f4325211740 10 client.18834 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:45:21.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.456+0000 7f4325211740 20 client.18834 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:21.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.456+0000 7f4325211740 1 --2- 172.21.15.90:0/943356809 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x5595eaf1cf60 0x5595eb31c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.456+0000 7f4325211740 4 client.18834 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:45:21.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.456+0000 7f4325211740 1 -- 172.21.15.90:0/943356809 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5595eb272990 con 0x5595eaf1cf60 2024-04-25T11:45:21.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.456+0000 7f4317fff640 1 --2- 172.21.15.90:0/943356809 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x5595eaf1cf60 0x5595eb31c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.456+0000 7f42f77fe640 10 client.18834 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:45:21.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.456+0000 7f4317fff640 1 --2- 172.21.15.90:0/943356809 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x5595eaf1cf60 0x5595eb31c5c0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ed36a3150742a887 server_cookie=3258a891ebe42725 in_seq=0 out_seq=0 2024-04-25T11:45:21.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.457+0000 7f42f77fe640 1 -- 172.21.15.90:0/943356809 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5595eb272990 con 0x5595eaf1cf60 2024-04-25T11:45:21.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.457+0000 7f42f77fe640 10 client.18834 handle_command_reply: tid=0 2024-04-25T11:45:21.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.457+0000 7f4325211740 1 client.18834 shutdown 2024-04-25T11:45:21.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.457+0000 7f4325211740 1 -- 172.21.15.90:0/943356809 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf5e390 msgr2=0x5595eb3162b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.457+0000 7f4325211740 1 --2- 172.21.15.90:0/943356809 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf5e390 0x5595eb3162b0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f43100a77d0 tx=0x7f43100a75e0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.457+0000 7f4325211740 1 -- 172.21.15.90:0/943356809 shutdown_connections 2024-04-25T11:45:21.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.457+0000 7f4325211740 1 --2- 172.21.15.90:0/943356809 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x5595eaf1cf60 0x5595eb31c5c0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.457+0000 7f4325211740 1 --2- 172.21.15.90:0/943356809 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf5e390 0x5595eb3162b0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.457+0000 7f4325211740 1 --2- 172.21.15.90:0/943356809 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595eaf3d6b0 0x5595eb30a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.457+0000 7f42f77fe640 0 client.18834 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:45:21.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.457+0000 7f4325211740 1 --2- 172.21.15.90:0/943356809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595eaf354d0 0x5595eb30a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.457+0000 7f4325211740 1 -- 172.21.15.90:0/943356809 >> 172.21.15.90:0/943356809 conn(0x5595eaf524e0 msgr2=0x5595eb301530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:21.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.457+0000 7f4325211740 1 -- 172.21.15.90:0/943356809 shutdown_connections 2024-04-25T11:45:21.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.457+0000 7f4325211740 1 -- 172.21.15.90:0/943356809 wait complete. 2024-04-25T11:45:21.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.457+0000 7f4325211740 20 client.18834 trim_cache size 0 max 16384 2024-04-25T11:45:21.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.471+0000 7f4325211740 1 Processor -- start 2024-04-25T11:45:21.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.471+0000 7f4325211740 1 -- start start 2024-04-25T11:45:21.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.471+0000 7f4325211740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595eaf354d0 0x5595eb308f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.471+0000 7f4325211740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595eaf3d6b0 0x5595eb304430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.471+0000 7f4325211740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf5e390 0x5595eb321d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.471+0000 7f4325211740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f43181afb40 con 0x5595eaf5e390 2024-04-25T11:45:21.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.471+0000 7f4325211740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f43181b1680 con 0x5595eaf354d0 2024-04-25T11:45:21.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.471+0000 7f4325211740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f43181b0210 con 0x5595eaf3d6b0 2024-04-25T11:45:21.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.471+0000 7f431c901640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf5e390 0x5595eb321d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.471+0000 7f431c901640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf5e390 0x5595eb321d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41960/0 (socket says 172.21.15.90:41960) 2024-04-25T11:45:21.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.471+0000 7f431c901640 1 -- 172.21.15.90:0/687379820 learned_addr learned my addr 172.21.15.90:0/687379820 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:21.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.471+0000 7f4317fff640 1 --2- 172.21.15.90:0/687379820 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595eaf354d0 0x5595eb308f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.471+0000 7f43177fe640 1 --2- 172.21.15.90:0/687379820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595eaf3d6b0 0x5595eb304430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.471+0000 7f431c901640 1 -- 172.21.15.90:0/687379820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595eaf3d6b0 msgr2=0x5595eb304430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.471+0000 7f431c901640 1 --2- 172.21.15.90:0/687379820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595eaf3d6b0 0x5595eb304430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.472+0000 7f431c901640 1 -- 172.21.15.90:0/687379820 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595eaf354d0 msgr2=0x5595eb308f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.472+0000 7f431c901640 1 --2- 172.21.15.90:0/687379820 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595eaf354d0 0x5595eb308f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.472+0000 7f431c901640 1 -- 172.21.15.90:0/687379820 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595eb301010 con 0x5595eaf5e390 2024-04-25T11:45:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.472+0000 7f43177fe640 1 --2- 172.21.15.90:0/687379820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595eaf3d6b0 0x5595eb304430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.472+0000 7f4317fff640 1 --2- 172.21.15.90:0/687379820 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595eaf354d0 0x5595eb308f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.472+0000 7f431c901640 1 --2- 172.21.15.90:0/687379820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf5e390 0x5595eb321d30 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f431000a6e0 tx=0x7f43100a7c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99bec47f8fb4749a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.472+0000 7f42fdffb640 1 -- 172.21.15.90:0/687379820 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f431006e430 con 0x5595eaf5e390 2024-04-25T11:45:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.472+0000 7f42fdffb640 1 -- 172.21.15.90:0/687379820 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43100b0930 con 0x5595eaf5e390 2024-04-25T11:45:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.472+0000 7f42fdffb640 1 -- 172.21.15.90:0/687379820 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43100b04a0 con 0x5595eaf5e390 2024-04-25T11:45:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.473+0000 7f4325211740 1 -- 172.21.15.90:0/687379820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf5e390 msgr2=0x5595eb321d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.473+0000 7f4325211740 1 --2- 172.21.15.90:0/687379820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf5e390 0x5595eb321d30 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f431000a6e0 tx=0x7f43100a7c50 comp rx=0 tx=0).stop 2024-04-25T11:45:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.473+0000 7f4325211740 1 -- 172.21.15.90:0/687379820 shutdown_connections 2024-04-25T11:45:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.473+0000 7f4325211740 1 --2- 172.21.15.90:0/687379820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf5e390 0x5595eb321d30 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.473+0000 7f4325211740 1 --2- 172.21.15.90:0/687379820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595eaf3d6b0 0x5595eb304430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.473+0000 7f4325211740 1 --2- 172.21.15.90:0/687379820 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595eaf354d0 0x5595eb308f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.473+0000 7f4325211740 1 -- 172.21.15.90:0/687379820 >> 172.21.15.90:0/687379820 conn(0x5595eaf524e0 msgr2=0x5595eb308860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.473+0000 7f4325211740 1 -- 172.21.15.90:0/687379820 shutdown_connections 2024-04-25T11:45:21.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.473+0000 7f4325211740 1 -- 172.21.15.90:0/687379820 wait complete. 2024-04-25T11:45:21.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.473+0000 7f4325211740 1 -- start start 2024-04-25T11:45:21.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.473+0000 7f4325211740 1 Processor -- start 2024-04-25T11:45:21.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.473+0000 7f4325211740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595eaf354d0 0x5595eb312110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.473+0000 7f4325211740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf3d6b0 0x5595eb312670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.473+0000 7f4325211740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595eaf5e390 0x5595eb30f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.473+0000 7f4325211740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f43181b0fb0 con 0x5595eaf3d6b0 2024-04-25T11:45:21.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.473+0000 7f4325211740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f43181b08e0 con 0x5595eaf354d0 2024-04-25T11:45:21.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.473+0000 7f4325211740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f43181adb40 con 0x5595eaf5e390 2024-04-25T11:45:21.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.473+0000 7f4325211740 10 client.0 _ll_register_callbacks cb 0x7f43100a32d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:21.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.474+0000 7f43177fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf3d6b0 0x5595eb312670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.474+0000 7f43177fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf3d6b0 0x5595eb312670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41974/0 (socket says 172.21.15.90:41974) 2024-04-25T11:45:21.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.474+0000 7f43177fe640 1 -- 172.21.15.90:0/3828940655 learned_addr learned my addr 172.21.15.90:0/3828940655 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:21.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.474+0000 7f4317fff640 1 --2- 172.21.15.90:0/3828940655 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595eaf354d0 0x5595eb312110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.474+0000 7f431c901640 1 --2- 172.21.15.90:0/3828940655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595eaf5e390 0x5595eb30f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.474+0000 7f43177fe640 1 -- 172.21.15.90:0/3828940655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595eaf5e390 msgr2=0x5595eb30f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.474+0000 7f43177fe640 1 --2- 172.21.15.90:0/3828940655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595eaf5e390 0x5595eb30f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.474+0000 7f43177fe640 1 -- 172.21.15.90:0/3828940655 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595eaf354d0 msgr2=0x5595eb312110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.474+0000 7f43177fe640 1 --2- 172.21.15.90:0/3828940655 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595eaf354d0 0x5595eb312110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.474+0000 7f43177fe640 1 -- 172.21.15.90:0/3828940655 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595eb30fc70 con 0x5595eaf3d6b0 2024-04-25T11:45:21.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.474+0000 7f43177fe640 1 --2- 172.21.15.90:0/3828940655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf3d6b0 0x5595eb312670 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f43180754b0 tx=0x7f4308005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec7ad20734ffdb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:21.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.474+0000 7f42f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:21.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.474+0000 7f42f77fe640 1 -- 172.21.15.90:0/3828940655 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4308035040 con 0x5595eaf3d6b0 2024-04-25T11:45:21.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.474+0000 7f42f77fe640 1 -- 172.21.15.90:0/3828940655 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7f43080035f0 con 0x5595eaf3d6b0 2024-04-25T11:45:21.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.474+0000 7f4325211740 1 -- 172.21.15.90:0/3828940655 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43100b0930 con 0x5595eaf3d6b0 2024-04-25T11:45:21.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.474+0000 7f42f77fe640 1 -- 172.21.15.90:0/3828940655 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f43080037f0 con 0x5595eaf3d6b0 2024-04-25T11:45:21.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.474+0000 7f4325211740 10 client.18840 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:21.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.474+0000 7f4325211740 1 -- 172.21.15.90:0/3828940655 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5595eb384110 con 0x5595eaf3d6b0 2024-04-25T11:45:21.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.475+0000 7f42f77fe640 1 -- 172.21.15.90:0/3828940655 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43100b0930 con 0x5595eaf3d6b0 2024-04-25T11:45:21.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.475+0000 7f42f77fe640 1 -- 172.21.15.90:0/3828940655 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f430803a040 con 0x5595eaf3d6b0 2024-04-25T11:45:21.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.475+0000 7f4325211740 10 client.18840 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:21.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.475+0000 7f4325211740 10 client.18840 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:45:21.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.475+0000 7f4325211740 20 client.18840 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:21.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.475+0000 7f4325211740 1 --2- 172.21.15.90:0/3828940655 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x5595eaef23a0 0x5595eb386950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.475+0000 7f4325211740 4 client.18840 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:45:21.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.475+0000 7f4325211740 1 -- 172.21.15.90:0/3828940655 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5595eb26c800 con 0x5595eaef23a0 2024-04-25T11:45:21.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.475+0000 7f4317fff640 1 --2- 172.21.15.90:0/3828940655 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x5595eaef23a0 0x5595eb386950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.475+0000 7f42f77fe640 10 client.18840 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:45:21.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.475+0000 7f4317fff640 1 --2- 172.21.15.90:0/3828940655 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x5595eaef23a0 0x5595eb386950 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=623a56503f0c625 server_cookie=858965c2e8cc463d in_seq=0 out_seq=0 2024-04-25T11:45:21.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.476+0000 7f42f77fe640 1 -- 172.21.15.90:0/3828940655 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5595eb26c800 con 0x5595eaef23a0 2024-04-25T11:45:21.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.476+0000 7f42f77fe640 10 client.18840 handle_command_reply: tid=0 2024-04-25T11:45:21.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.476+0000 7f4325211740 1 client.18840 shutdown 2024-04-25T11:45:21.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.476+0000 7f4325211740 1 -- 172.21.15.90:0/3828940655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf3d6b0 msgr2=0x5595eb312670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.476+0000 7f4325211740 1 --2- 172.21.15.90:0/3828940655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf3d6b0 0x5595eb312670 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f43180754b0 tx=0x7f4308005a20 comp rx=0 tx=0).stop 2024-04-25T11:45:21.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.476+0000 7f4325211740 1 -- 172.21.15.90:0/3828940655 shutdown_connections 2024-04-25T11:45:21.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.476+0000 7f4325211740 1 --2- 172.21.15.90:0/3828940655 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x5595eaef23a0 0x5595eb386950 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.476+0000 7f4325211740 1 --2- 172.21.15.90:0/3828940655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5595eaf5e390 0x5595eb30f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.476+0000 7f4325211740 1 --2- 172.21.15.90:0/3828940655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595eaf3d6b0 0x5595eb312670 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.476+0000 7f42f77fe640 0 client.18840 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:45:21.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.476+0000 7f4325211740 1 --2- 172.21.15.90:0/3828940655 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5595eaf354d0 0x5595eb312110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.476+0000 7f4325211740 1 -- 172.21.15.90:0/3828940655 >> 172.21.15.90:0/3828940655 conn(0x5595eaf524e0 msgr2=0x5595eb30b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:21.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.476+0000 7f4325211740 1 -- 172.21.15.90:0/3828940655 shutdown_connections 2024-04-25T11:45:21.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.476+0000 7f4325211740 1 -- 172.21.15.90:0/3828940655 wait complete. 2024-04-25T11:45:21.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.476+0000 7f4325211740 20 client.18840 trim_cache size 0 max 16384 2024-04-25T11:45:21.468 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:45:21.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.476+0000 7f431e364640 1 -- 172.21.15.90:0/1238846164 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f43000afc60 msgr2=0x7f43000b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.476+0000 7f431e364640 1 --2- 172.21.15.90:0/1238846164 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f43000afc60 0x7f43000b2170 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4308000c00 tx=0x7f4308031400 comp rx=0 tx=0).stop 2024-04-25T11:45:21.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.476+0000 7f431e364640 1 -- 172.21.15.90:0/1238846164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4318109f90 msgr2=0x7f431819ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.476+0000 7f431e364640 1 --2- 172.21.15.90:0/1238846164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4318109f90 0x7f431819ef20 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f431000cab0 tx=0x7f431000cf70 comp rx=0 tx=0).stop 2024-04-25T11:45:21.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.477+0000 7f431e364640 1 -- 172.21.15.90:0/1238846164 shutdown_connections 2024-04-25T11:45:21.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.477+0000 7f431e364640 1 --2- 172.21.15.90:0/1238846164 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f43000afc60 0x7f43000b2170 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.477+0000 7f431e364640 1 --2- 172.21.15.90:0/1238846164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4318109f90 0x7f431819ef20 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.477+0000 7f431e364640 1 --2- 172.21.15.90:0/1238846164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4318109af0 0x7f431819ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.477+0000 7f431e364640 1 --2- 172.21.15.90:0/1238846164 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4318102720 0x7f431819a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.477+0000 7f431e364640 1 -- 172.21.15.90:0/1238846164 >> 172.21.15.90:0/1238846164 conn(0x7f4318077880 msgr2=0x7f43181051f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:21.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.477+0000 7f431e364640 1 -- 172.21.15.90:0/1238846164 shutdown_connections 2024-04-25T11:45:21.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.477+0000 7f431e364640 1 -- 172.21.15.90:0/1238846164 wait complete. 2024-04-25T11:45:21.473 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T11:45:21.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.598+0000 7f89766aa640 1 -- 172.21.15.90:0/10064981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8970108a20 msgr2=0x7f89701025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.598+0000 7f89766aa640 1 --2- 172.21.15.90:0/10064981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8970108a20 0x7f89701025b0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f8958009900 tx=0x7f895802a410 comp rx=0 tx=0).stop 2024-04-25T11:45:21.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.598+0000 7f89766aa640 1 -- 172.21.15.90:0/10064981 shutdown_connections 2024-04-25T11:45:21.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.598+0000 7f89766aa640 1 --2- 172.21.15.90:0/10064981 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f897010d380 0x7f897010f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.598+0000 7f89766aa640 1 --2- 172.21.15.90:0/10064981 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8970102af0 0x7f8970102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.598+0000 7f89766aa640 1 --2- 172.21.15.90:0/10064981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8970108a20 0x7f89701025b0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.598+0000 7f89766aa640 1 -- 172.21.15.90:0/10064981 >> 172.21.15.90:0/10064981 conn(0x7f8970078100 msgr2=0x7f8970078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:21.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.598+0000 7f89766aa640 1 -- 172.21.15.90:0/10064981 shutdown_connections 2024-04-25T11:45:21.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.598+0000 7f89766aa640 1 -- 172.21.15.90:0/10064981 wait complete. 2024-04-25T11:45:21.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.598+0000 7f89766aa640 1 Processor -- start 2024-04-25T11:45:21.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.599+0000 7f89766aa640 1 -- start start 2024-04-25T11:45:21.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.599+0000 7f89766aa640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8970102af0 0x7f89701995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.599+0000 7f89766aa640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8970108a20 0x7f8970199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.599+0000 7f89766aa640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897010d380 0x7f897019de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.599+0000 7f89766aa640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8970112be0 con 0x7f897010d380 2024-04-25T11:45:21.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.599+0000 7f89766aa640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8970112a60 con 0x7f8970108a20 2024-04-25T11:45:21.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.599+0000 7f89766aa640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8970112d60 con 0x7f8970102af0 2024-04-25T11:45:21.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.599+0000 7f8974c47640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897010d380 0x7f897019de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.599+0000 7f8974c47640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897010d380 0x7f897019de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41998/0 (socket says 172.21.15.90:41998) 2024-04-25T11:45:21.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.599+0000 7f8974c47640 1 -- 172.21.15.90:0/3126277853 learned_addr learned my addr 172.21.15.90:0/3126277853 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:21.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.599+0000 7f896f7fe640 1 --2- 172.21.15.90:0/3126277853 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8970108a20 0x7f8970199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.599+0000 7f896ffff640 1 --2- 172.21.15.90:0/3126277853 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8970102af0 0x7f89701995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.599+0000 7f8974c47640 1 -- 172.21.15.90:0/3126277853 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8970102af0 msgr2=0x7f89701995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.599+0000 7f8974c47640 1 --2- 172.21.15.90:0/3126277853 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8970102af0 0x7f89701995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.599+0000 7f8974c47640 1 -- 172.21.15.90:0/3126277853 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8970108a20 msgr2=0x7f8970199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.599+0000 7f8974c47640 1 --2- 172.21.15.90:0/3126277853 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8970108a20 0x7f8970199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.599+0000 7f8974c47640 1 -- 172.21.15.90:0/3126277853 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f897019e440 con 0x7f897010d380 2024-04-25T11:45:21.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.599+0000 7f896f7fe640 1 --2- 172.21.15.90:0/3126277853 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8970108a20 0x7f8970199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:21.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.599+0000 7f8974c47640 1 --2- 172.21.15.90:0/3126277853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897010d380 0x7f897019de00 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f896800eaa0 tx=0x7f896800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:21.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.600+0000 7f896d7fa640 1 -- 172.21.15.90:0/3126277853 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896800cdb0 con 0x7f897010d380 2024-04-25T11:45:21.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.600+0000 7f89766aa640 1 -- 172.21.15.90:0/3126277853 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f897019e730 con 0x7f897010d380 2024-04-25T11:45:21.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.600+0000 7f896d7fa640 1 -- 172.21.15.90:0/3126277853 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8968005020 con 0x7f897010d380 2024-04-25T11:45:21.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.600+0000 7f896d7fa640 1 -- 172.21.15.90:0/3126277853 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89680116d0 con 0x7f897010d380 2024-04-25T11:45:21.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.600+0000 7f89766aa640 1 -- 172.21.15.90:0/3126277853 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f897019ec30 con 0x7f897010d380 2024-04-25T11:45:21.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.602+0000 7f896d7fa640 1 -- 172.21.15.90:0/3126277853 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f89680051d0 con 0x7f897010d380 2024-04-25T11:45:21.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.602+0000 7f897d633740 1 Processor -- start 2024-04-25T11:45:21.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.602+0000 7f897d633740 1 -- start start 2024-04-25T11:45:21.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.603+0000 7f897d633740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555e3079a390 0x555e30b3b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.603+0000 7f897d633740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e307714d0 0x555e30b3bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.603+0000 7f897d633740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e307796b0 0x555e30b4a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.603+0000 7f897d633740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555e30aa0d80 con 0x555e307796b0 2024-04-25T11:45:21.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.603+0000 7f897d633740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x555e30a5aa20 con 0x555e3079a390 2024-04-25T11:45:21.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.603+0000 7f897d633740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x555e30a9cd40 con 0x555e307714d0 2024-04-25T11:45:21.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.603+0000 7f896f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e307796b0 0x555e30b4a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.603+0000 7f896f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e307796b0 0x555e30b4a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42014/0 (socket says 172.21.15.90:42014) 2024-04-25T11:45:21.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.603+0000 7f896f7fe640 1 -- 172.21.15.90:0/1366420237 learned_addr learned my addr 172.21.15.90:0/1366420237 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:21.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.603+0000 7f896ffff640 1 --2- 172.21.15.90:0/1366420237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e307714d0 0x555e30b3bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.603+0000 7f896d7fa640 1 --2- 172.21.15.90:0/3126277853 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f895c0afc60 0x7f895c0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.603+0000 7f896f7fe640 1 --2- 172.21.15.90:0/1366420237 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555e3079a390 0x555e30b3b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.603+0000 7f896f7fe640 1 -- 172.21.15.90:0/1366420237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e307714d0 msgr2=0x555e30b3bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.603+0000 7f8974c47640 1 --2- 172.21.15.90:0/3126277853 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f895c0afc60 0x7f895c0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.603+0000 7f896f7fe640 1 --2- 172.21.15.90:0/1366420237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e307714d0 0x555e30b3bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.603+0000 7f896f7fe640 1 -- 172.21.15.90:0/1366420237 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555e3079a390 msgr2=0x555e30b3b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.603+0000 7f896f7fe640 1 --2- 172.21.15.90:0/1366420237 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555e3079a390 0x555e30b3b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.603+0000 7f896f7fe640 1 -- 172.21.15.90:0/1366420237 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e30a65c30 con 0x555e307796b0 2024-04-25T11:45:21.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.603+0000 7f896d7fa640 1 -- 172.21.15.90:0/3126277853 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7f89680df360 con 0x7f897010d380 2024-04-25T11:45:21.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.603+0000 7f896f7fe640 1 --2- 172.21.15.90:0/1366420237 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e307796b0 0x555e30b4a8b0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f8960012470 tx=0x7f896000b5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ba4adf3229ffc06 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:21.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.604+0000 7f89667fc640 1 -- 172.21.15.90:0/1366420237 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8960018030 con 0x555e307796b0 2024-04-25T11:45:21.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.604+0000 7f89667fc640 1 -- 172.21.15.90:0/1366420237 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89600054d0 con 0x555e307796b0 2024-04-25T11:45:21.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.604+0000 7f8974c47640 1 --2- 172.21.15.90:0/3126277853 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f895c0afc60 0x7f895c0b2170 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f89680a7150 tx=0x7f8968074cb0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:21.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.604+0000 7f89667fc640 1 -- 172.21.15.90:0/1366420237 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896000e3f0 con 0x555e307796b0 2024-04-25T11:45:21.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.604+0000 7f897d633740 1 -- 172.21.15.90:0/1366420237 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e307796b0 msgr2=0x555e30b4a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.604+0000 7f897d633740 1 --2- 172.21.15.90:0/1366420237 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e307796b0 0x555e30b4a8b0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f8960012470 tx=0x7f896000b5c0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.604+0000 7f897d633740 1 -- 172.21.15.90:0/1366420237 shutdown_connections 2024-04-25T11:45:21.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.604+0000 7f897d633740 1 --2- 172.21.15.90:0/1366420237 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e307796b0 0x555e30b4a8b0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.604+0000 7f897d633740 1 --2- 172.21.15.90:0/1366420237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e307714d0 0x555e30b3bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.604+0000 7f897d633740 1 --2- 172.21.15.90:0/1366420237 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555e3079a390 0x555e30b3b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.604+0000 7f897d633740 1 -- 172.21.15.90:0/1366420237 >> 172.21.15.90:0/1366420237 conn(0x555e3078e4e0 msgr2=0x555e30b37fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:21.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.605+0000 7f897d633740 1 -- 172.21.15.90:0/1366420237 shutdown_connections 2024-04-25T11:45:21.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.605+0000 7f897d633740 1 -- 172.21.15.90:0/1366420237 wait complete. 2024-04-25T11:45:21.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.605+0000 7f897d633740 1 -- start start 2024-04-25T11:45:21.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.605+0000 7f897d633740 1 Processor -- start 2024-04-25T11:45:21.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.605+0000 7f897d633740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555e307714d0 0x555e30b48400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.605+0000 7f897d633740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e307796b0 0x555e30b48940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.605+0000 7f897d633740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e3079a390 0x555e30b54450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.605+0000 7f897d633740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555e30b4afa0 con 0x555e3079a390 2024-04-25T11:45:21.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.605+0000 7f897d633740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x555e30a27dc0 con 0x555e307714d0 2024-04-25T11:45:21.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.605+0000 7f897d633740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x555e30b4b120 con 0x555e307796b0 2024-04-25T11:45:21.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.605+0000 7f897d633740 10 client.0 _ll_register_callbacks cb 0x555e30a43600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:21.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.606+0000 7f896f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e3079a390 0x555e30b54450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.606+0000 7f896f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e3079a390 0x555e30b54450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42018/0 (socket says 172.21.15.90:42018) 2024-04-25T11:45:21.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.606+0000 7f896f7fe640 1 -- 172.21.15.90:0/210708592 learned_addr learned my addr 172.21.15.90:0/210708592 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:21.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.606+0000 7f896ffff640 1 --2- 172.21.15.90:0/210708592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e307796b0 0x555e30b48940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.606+0000 7f896f7fe640 1 --2- 172.21.15.90:0/210708592 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555e307714d0 0x555e30b48400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.606+0000 7f896f7fe640 1 -- 172.21.15.90:0/210708592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e307796b0 msgr2=0x555e30b48940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.606+0000 7f896f7fe640 1 --2- 172.21.15.90:0/210708592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e307796b0 0x555e30b48940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.606+0000 7f896f7fe640 1 -- 172.21.15.90:0/210708592 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555e307714d0 msgr2=0x555e30b48400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.606+0000 7f896f7fe640 1 --2- 172.21.15.90:0/210708592 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555e307714d0 0x555e30b48400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.606+0000 7f896f7fe640 1 -- 172.21.15.90:0/210708592 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e307b23f0 con 0x555e3079a390 2024-04-25T11:45:21.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.606+0000 7f896f7fe640 1 --2- 172.21.15.90:0/210708592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e3079a390 0x555e30b54450 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x555e30b47a50 tx=0x7f8960005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd265201490e4ebd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:21.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.606+0000 7f894f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:21.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.606+0000 7f894f7fe640 1 -- 172.21.15.90:0/210708592 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8960018030 con 0x555e3079a390 2024-04-25T11:45:21.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.606+0000 7f894f7fe640 1 -- 172.21.15.90:0/210708592 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7f8960019070 con 0x555e3079a390 2024-04-25T11:45:21.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.606+0000 7f897d633740 1 -- 172.21.15.90:0/210708592 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89600054d0 con 0x555e3079a390 2024-04-25T11:45:21.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.606+0000 7f894f7fe640 1 -- 172.21.15.90:0/210708592 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f89600046c0 con 0x555e3079a390 2024-04-25T11:45:21.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.606+0000 7f897d633740 10 client.18852 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:21.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.606+0000 7f897d633740 1 -- 172.21.15.90:0/210708592 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555e3087c7d0 con 0x555e3079a390 2024-04-25T11:45:21.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.607+0000 7f894f7fe640 1 -- 172.21.15.90:0/210708592 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89600054d0 con 0x555e3079a390 2024-04-25T11:45:21.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.607+0000 7f894f7fe640 1 -- 172.21.15.90:0/210708592 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f896000f040 con 0x555e3079a390 2024-04-25T11:45:21.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.607+0000 7f897d633740 10 client.18852 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:21.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.607+0000 7f897d633740 10 client.18852 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:45:21.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.607+0000 7f897d633740 20 client.18852 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:21.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.607+0000 7f897d633740 1 --2- 172.21.15.90:0/210708592 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x555e30758f60 0x555e30b5a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.607+0000 7f897d633740 4 client.18852 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:45:21.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.607+0000 7f897d633740 1 -- 172.21.15.90:0/210708592 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e30aae990 con 0x555e30758f60 2024-04-25T11:45:21.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.607+0000 7f896ffff640 1 --2- 172.21.15.90:0/210708592 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x555e30758f60 0x555e30b5a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.608+0000 7f894f7fe640 10 client.18852 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:45:21.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.608+0000 7f896ffff640 1 --2- 172.21.15.90:0/210708592 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x555e30758f60 0x555e30b5a6c0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6a2d6627ec67b4b8 server_cookie=6ed06d60b039b94e in_seq=0 out_seq=0 2024-04-25T11:45:21.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.608+0000 7f894f7fe640 1 -- 172.21.15.90:0/210708592 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x555e30aae990 con 0x555e30758f60 2024-04-25T11:45:21.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.608+0000 7f894f7fe640 10 client.18852 handle_command_reply: tid=0 2024-04-25T11:45:21.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.608+0000 7f897d633740 1 client.18852 shutdown 2024-04-25T11:45:21.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.609+0000 7f897d633740 1 -- 172.21.15.90:0/210708592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e3079a390 msgr2=0x555e30b54450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.609+0000 7f897d633740 1 --2- 172.21.15.90:0/210708592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e3079a390 0x555e30b54450 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x555e30b47a50 tx=0x7f8960005850 comp rx=0 tx=0).stop 2024-04-25T11:45:21.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.609+0000 7f897d633740 1 -- 172.21.15.90:0/210708592 shutdown_connections 2024-04-25T11:45:21.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.609+0000 7f897d633740 1 --2- 172.21.15.90:0/210708592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e3079a390 0x555e30b54450 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.609+0000 7f897d633740 1 --2- 172.21.15.90:0/210708592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e307796b0 0x555e30b48940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.609+0000 7f897d633740 1 --2- 172.21.15.90:0/210708592 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x555e30758f60 0x555e30b5a6c0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.609+0000 7f897d633740 1 --2- 172.21.15.90:0/210708592 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555e307714d0 0x555e30b48400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.609+0000 7f897d633740 1 -- 172.21.15.90:0/210708592 >> 172.21.15.90:0/210708592 conn(0x555e3078e4e0 msgr2=0x555e30b3d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:21.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.609+0000 7f894f7fe640 0 client.18852 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:45:21.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.609+0000 7f897d633740 1 -- 172.21.15.90:0/210708592 shutdown_connections 2024-04-25T11:45:21.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.609+0000 7f897d633740 1 -- 172.21.15.90:0/210708592 wait complete. 2024-04-25T11:45:21.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.609+0000 7f897d633740 20 client.18852 trim_cache size 0 max 16384 2024-04-25T11:45:21.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.622+0000 7f897d633740 1 Processor -- start 2024-04-25T11:45:21.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.622+0000 7f897d633740 1 -- start start 2024-04-25T11:45:21.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.623+0000 7f897d633740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555e307714d0 0x555e30b54030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.623+0000 7f897d633740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e307796b0 0x555e30b5fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.623+0000 7f897d633740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e3079a390 0x555e30b60290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.623+0000 7f897d633740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f89701adb00 con 0x555e307796b0 2024-04-25T11:45:21.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.623+0000 7f897d633740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f89701af640 con 0x555e307714d0 2024-04-25T11:45:21.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.623+0000 7f897d633740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f89701ae1d0 con 0x555e3079a390 2024-04-25T11:45:21.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.623+0000 7f896ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e307796b0 0x555e30b5fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.623+0000 7f896ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e307796b0 0x555e30b5fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42024/0 (socket says 172.21.15.90:42024) 2024-04-25T11:45:21.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.623+0000 7f896ffff640 1 -- 172.21.15.90:0/4276431465 learned_addr learned my addr 172.21.15.90:0/4276431465 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:21.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.623+0000 7f896ffff640 1 -- 172.21.15.90:0/4276431465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e3079a390 msgr2=0x555e30b60290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.623+0000 7f896ffff640 1 --2- 172.21.15.90:0/4276431465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e3079a390 0x555e30b60290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.623+0000 7f896ffff640 1 -- 172.21.15.90:0/4276431465 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555e307714d0 msgr2=0x555e30b54030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.623+0000 7f896ffff640 1 --2- 172.21.15.90:0/4276431465 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555e307714d0 0x555e30b54030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.623+0000 7f896ffff640 1 -- 172.21.15.90:0/4276431465 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e30b46740 con 0x555e307796b0 2024-04-25T11:45:21.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.623+0000 7f896ffff640 1 --2- 172.21.15.90:0/4276431465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e307796b0 0x555e30b5fd50 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f8970103a20 tx=0x7f8958031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feb6a02194bf873d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:21.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.623+0000 7f8965ffb640 1 -- 172.21.15.90:0/4276431465 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8958035040 con 0x555e307796b0 2024-04-25T11:45:21.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.623+0000 7f8965ffb640 1 -- 172.21.15.90:0/4276431465 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8958048070 con 0x555e307796b0 2024-04-25T11:45:21.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.623+0000 7f8965ffb640 1 -- 172.21.15.90:0/4276431465 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8958004920 con 0x555e307796b0 2024-04-25T11:45:21.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.624+0000 7f897d633740 1 -- 172.21.15.90:0/4276431465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e307796b0 msgr2=0x555e30b5fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.624+0000 7f897d633740 1 --2- 172.21.15.90:0/4276431465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e307796b0 0x555e30b5fd50 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f8970103a20 tx=0x7f8958031400 comp rx=0 tx=0).stop 2024-04-25T11:45:21.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.624+0000 7f897d633740 1 -- 172.21.15.90:0/4276431465 shutdown_connections 2024-04-25T11:45:21.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.624+0000 7f897d633740 1 --2- 172.21.15.90:0/4276431465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e3079a390 0x555e30b60290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.624+0000 7f897d633740 1 --2- 172.21.15.90:0/4276431465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e307796b0 0x555e30b5fd50 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.624+0000 7f897d633740 1 --2- 172.21.15.90:0/4276431465 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555e307714d0 0x555e30b54030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.624+0000 7f897d633740 1 -- 172.21.15.90:0/4276431465 >> 172.21.15.90:0/4276431465 conn(0x555e3078e4e0 msgr2=0x555e30b433b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:21.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.624+0000 7f897d633740 1 -- 172.21.15.90:0/4276431465 shutdown_connections 2024-04-25T11:45:21.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.624+0000 7f897d633740 1 -- 172.21.15.90:0/4276431465 wait complete. 2024-04-25T11:45:21.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.624+0000 7f897d633740 1 -- start start 2024-04-25T11:45:21.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.625+0000 7f897d633740 1 Processor -- start 2024-04-25T11:45:21.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.625+0000 7f897d633740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555e307714d0 0x555e30b4b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.625+0000 7f897d633740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e307796b0 0x555e30b4b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.625+0000 7f897d633740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e3079a390 0x555e30b4bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.625+0000 7f897d633740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f89701aef70 con 0x555e3079a390 2024-04-25T11:45:21.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.625+0000 7f897d633740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f89701ae8a0 con 0x555e307714d0 2024-04-25T11:45:21.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.625+0000 7f897d633740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f89701abb00 con 0x555e307796b0 2024-04-25T11:45:21.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.625+0000 7f897d633740 10 client.0 _ll_register_callbacks cb 0x7f8958009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:21.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.625+0000 7f896f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e3079a390 0x555e30b4bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.625+0000 7f896f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e3079a390 0x555e30b4bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42034/0 (socket says 172.21.15.90:42034) 2024-04-25T11:45:21.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.625+0000 7f896f7fe640 1 -- 172.21.15.90:0/3579996341 learned_addr learned my addr 172.21.15.90:0/3579996341 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:21.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.625+0000 7f896f7fe640 1 --2- 172.21.15.90:0/3579996341 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555e307714d0 0x555e30b4b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.625+0000 7f896ffff640 1 --2- 172.21.15.90:0/3579996341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e307796b0 0x555e30b4b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.625+0000 7f896f7fe640 1 -- 172.21.15.90:0/3579996341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e307796b0 msgr2=0x555e30b4b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.625+0000 7f896f7fe640 1 --2- 172.21.15.90:0/3579996341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e307796b0 0x555e30b4b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.625+0000 7f896f7fe640 1 -- 172.21.15.90:0/3579996341 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555e307714d0 msgr2=0x555e30b4b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.625+0000 7f896f7fe640 1 --2- 172.21.15.90:0/3579996341 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555e307714d0 0x555e30b4b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.625+0000 7f896f7fe640 1 -- 172.21.15.90:0/3579996341 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e30bc08f0 con 0x555e3079a390 2024-04-25T11:45:21.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.625+0000 7f896ffff640 1 --2- 172.21.15.90:0/3579996341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e307796b0 0x555e30b4b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:21.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.626+0000 7f896f7fe640 1 --2- 172.21.15.90:0/3579996341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e3079a390 0x555e30b4bdf0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x555e30b384a0 tx=0x7f8960014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecb274243d07c355 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:21.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.626+0000 7f894f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:21.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.626+0000 7f894f7fe640 1 -- 172.21.15.90:0/3579996341 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8960021070 con 0x555e3079a390 2024-04-25T11:45:21.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.626+0000 7f894f7fe640 1 -- 172.21.15.90:0/3579996341 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7f896002d250 con 0x555e3079a390 2024-04-25T11:45:21.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.626+0000 7f897d633740 1 -- 172.21.15.90:0/3579996341 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8958048070 con 0x555e3079a390 2024-04-25T11:45:21.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.626+0000 7f894f7fe640 1 -- 172.21.15.90:0/3579996341 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8960018040 con 0x555e3079a390 2024-04-25T11:45:21.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.626+0000 7f897d633740 10 client.18858 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:21.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.626+0000 7f897d633740 1 -- 172.21.15.90:0/3579996341 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555e30bc1a80 con 0x555e3079a390 2024-04-25T11:45:21.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.626+0000 7f894f7fe640 1 -- 172.21.15.90:0/3579996341 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8958048070 con 0x555e3079a390 2024-04-25T11:45:21.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.626+0000 7f894f7fe640 1 -- 172.21.15.90:0/3579996341 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f896002da40 con 0x555e3079a390 2024-04-25T11:45:21.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.626+0000 7f897d633740 10 client.18858 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:21.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.626+0000 7f897d633740 10 client.18858 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:45:21.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.626+0000 7f897d633740 20 client.18858 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:21.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.626+0000 7f897d633740 1 --2- 172.21.15.90:0/3579996341 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x555e3072e3a0 0x555e30bc4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.626+0000 7f897d633740 4 client.18858 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:45:21.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.626+0000 7f897d633740 1 -- 172.21.15.90:0/3579996341 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555e30aa8800 con 0x555e3072e3a0 2024-04-25T11:45:21.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.627+0000 7f896ffff640 1 --2- 172.21.15.90:0/3579996341 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x555e3072e3a0 0x555e30bc4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.627+0000 7f894f7fe640 10 client.18858 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:45:21.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.627+0000 7f896ffff640 1 --2- 172.21.15.90:0/3579996341 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x555e3072e3a0 0x555e30bc4210 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a91b44a3d6043c69 server_cookie=fb061603ad0a3cc8 in_seq=0 out_seq=0 2024-04-25T11:45:21.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.627+0000 7f894f7fe640 1 -- 172.21.15.90:0/3579996341 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555e30aa8800 con 0x555e3072e3a0 2024-04-25T11:45:21.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.627+0000 7f894f7fe640 10 client.18858 handle_command_reply: tid=0 2024-04-25T11:45:21.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.627+0000 7f897d633740 1 client.18858 shutdown 2024-04-25T11:45:21.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.627+0000 7f897d633740 1 -- 172.21.15.90:0/3579996341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e3079a390 msgr2=0x555e30b4bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.627+0000 7f897d633740 1 --2- 172.21.15.90:0/3579996341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e3079a390 0x555e30b4bdf0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x555e30b384a0 tx=0x7f8960014040 comp rx=0 tx=0).stop 2024-04-25T11:45:21.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.627+0000 7f897d633740 1 -- 172.21.15.90:0/3579996341 shutdown_connections 2024-04-25T11:45:21.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.627+0000 7f897d633740 1 --2- 172.21.15.90:0/3579996341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e3079a390 0x555e30b4bdf0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.627+0000 7f897d633740 1 --2- 172.21.15.90:0/3579996341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e307796b0 0x555e30b4b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.627+0000 7f897d633740 1 --2- 172.21.15.90:0/3579996341 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x555e3072e3a0 0x555e30bc4210 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.627+0000 7f897d633740 1 --2- 172.21.15.90:0/3579996341 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555e307714d0 0x555e30b4b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.627+0000 7f894f7fe640 0 client.18858 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:45:21.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.627+0000 7f897d633740 1 -- 172.21.15.90:0/3579996341 >> 172.21.15.90:0/3579996341 conn(0x555e3078e4e0 msgr2=0x555e30b3d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:21.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.628+0000 7f897d633740 1 -- 172.21.15.90:0/3579996341 shutdown_connections 2024-04-25T11:45:21.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.628+0000 7f897d633740 1 -- 172.21.15.90:0/3579996341 wait complete. 2024-04-25T11:45:21.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.628+0000 7f897d633740 20 client.18858 trim_cache size 0 max 16384 2024-04-25T11:45:21.620 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:45:21.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.628+0000 7f89766aa640 1 -- 172.21.15.90:0/3126277853 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f895c0afc60 msgr2=0x7f895c0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.628+0000 7f89766aa640 1 --2- 172.21.15.90:0/3126277853 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f895c0afc60 0x7f895c0b2170 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f89680a7150 tx=0x7f8968074cb0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.628+0000 7f89766aa640 1 -- 172.21.15.90:0/3126277853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897010d380 msgr2=0x7f897019de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.628+0000 7f89766aa640 1 --2- 172.21.15.90:0/3126277853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897010d380 0x7f897019de00 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f896800eaa0 tx=0x7f896800ef60 comp rx=0 tx=0).stop 2024-04-25T11:45:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.628+0000 7f89766aa640 1 -- 172.21.15.90:0/3126277853 shutdown_connections 2024-04-25T11:45:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.628+0000 7f89766aa640 1 --2- 172.21.15.90:0/3126277853 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f895c0afc60 0x7f895c0b2170 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.628+0000 7f89766aa640 1 --2- 172.21.15.90:0/3126277853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897010d380 0x7f897019de00 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.628+0000 7f89766aa640 1 --2- 172.21.15.90:0/3126277853 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8970108a20 0x7f8970199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.628+0000 7f89766aa640 1 --2- 172.21.15.90:0/3126277853 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8970102af0 0x7f89701995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.628+0000 7f89766aa640 1 -- 172.21.15.90:0/3126277853 >> 172.21.15.90:0/3126277853 conn(0x7f8970078100 msgr2=0x7f89701040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.628+0000 7f89766aa640 1 -- 172.21.15.90:0/3126277853 shutdown_connections 2024-04-25T11:45:21.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.628+0000 7f89766aa640 1 -- 172.21.15.90:0/3126277853 wait complete. 2024-04-25T11:45:21.625 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T11:45:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.749+0000 7f11f36b6640 1 -- 172.21.15.90:0/1925046824 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11ec102af0 msgr2=0x7f11ec102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.749+0000 7f11f36b6640 1 --2- 172.21.15.90:0/1925046824 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11ec102af0 0x7f11ec102f00 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f11dc009900 tx=0x7f11dc02a410 comp rx=0 tx=0).stop 2024-04-25T11:45:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.749+0000 7f11f36b6640 1 -- 172.21.15.90:0/1925046824 shutdown_connections 2024-04-25T11:45:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.749+0000 7f11f36b6640 1 --2- 172.21.15.90:0/1925046824 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11ec10d380 0x7f11ec10f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.749+0000 7f11f36b6640 1 --2- 172.21.15.90:0/1925046824 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11ec102af0 0x7f11ec102f00 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.749+0000 7f11f36b6640 1 --2- 172.21.15.90:0/1925046824 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11ec108a20 0x7f11ec1025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.749+0000 7f11f36b6640 1 -- 172.21.15.90:0/1925046824 >> 172.21.15.90:0/1925046824 conn(0x7f11ec078100 msgr2=0x7f11ec078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.749+0000 7f11f36b6640 1 -- 172.21.15.90:0/1925046824 shutdown_connections 2024-04-25T11:45:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.749+0000 7f11f36b6640 1 -- 172.21.15.90:0/1925046824 wait complete. 2024-04-25T11:45:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.750+0000 7f11f36b6640 1 Processor -- start 2024-04-25T11:45:21.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.750+0000 7f11f36b6640 1 -- start start 2024-04-25T11:45:21.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.750+0000 7f11f36b6640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11ec102af0 0x7f11ec199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.750+0000 7f11f36b6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11ec108a20 0x7f11ec199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.750+0000 7f11f36b6640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11ec10d380 0x7f11ec19de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.750+0000 7f11f36b6640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec112be0 con 0x7f11ec10d380 2024-04-25T11:45:21.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.750+0000 7f11f36b6640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec112a60 con 0x7f11ec102af0 2024-04-25T11:45:21.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.750+0000 7f11f36b6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec112d60 con 0x7f11ec108a20 2024-04-25T11:45:21.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.750+0000 7f11f1c53640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11ec10d380 0x7f11ec19de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.750+0000 7f11f1c53640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11ec10d380 0x7f11ec19de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42060/0 (socket says 172.21.15.90:42060) 2024-04-25T11:45:21.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.750+0000 7f11f1c53640 1 -- 172.21.15.90:0/237449239 learned_addr learned my addr 172.21.15.90:0/237449239 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:21.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.750+0000 7f11f1452640 1 --2- 172.21.15.90:0/237449239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11ec102af0 0x7f11ec199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.750+0000 7f11f1c53640 1 -- 172.21.15.90:0/237449239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11ec108a20 msgr2=0x7f11ec199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.750+0000 7f11f1c53640 1 --2- 172.21.15.90:0/237449239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11ec108a20 0x7f11ec199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.750+0000 7f11f1c53640 1 -- 172.21.15.90:0/237449239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11ec102af0 msgr2=0x7f11ec199660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.750+0000 7f11f1c53640 1 --2- 172.21.15.90:0/237449239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11ec102af0 0x7f11ec199660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.750+0000 7f11f1c53640 1 -- 172.21.15.90:0/237449239 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11ec19e4c0 con 0x7f11ec10d380 2024-04-25T11:45:21.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.750+0000 7f11f1452640 1 --2- 172.21.15.90:0/237449239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11ec102af0 0x7f11ec199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:21.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.750+0000 7f11f1c53640 1 --2- 172.21.15.90:0/237449239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11ec10d380 0x7f11ec19de80 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f11e400d750 tx=0x7f11e400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:21.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.751+0000 7f11e27fc640 1 -- 172.21.15.90:0/237449239 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e4017070 con 0x7f11ec10d380 2024-04-25T11:45:21.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.751+0000 7f11f36b6640 1 -- 172.21.15.90:0/237449239 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11ec19e7b0 con 0x7f11ec10d380 2024-04-25T11:45:21.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.751+0000 7f11e27fc640 1 -- 172.21.15.90:0/237449239 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11e40054d0 con 0x7f11ec10d380 2024-04-25T11:45:21.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.751+0000 7f11e27fc640 1 -- 172.21.15.90:0/237449239 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e4004c60 con 0x7f11ec10d380 2024-04-25T11:45:21.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.751+0000 7f11f36b6640 1 -- 172.21.15.90:0/237449239 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11ec1a6130 con 0x7f11ec10d380 2024-04-25T11:45:21.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.753+0000 7f11e27fc640 1 -- 172.21.15.90:0/237449239 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f11e4009bd0 con 0x7f11ec10d380 2024-04-25T11:45:21.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.753+0000 7f11fa569740 1 Processor -- start 2024-04-25T11:45:21.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.753+0000 7f11fa569740 1 -- start start 2024-04-25T11:45:21.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.754+0000 7f11e27fc640 1 --2- 172.21.15.90:0/237449239 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f11d80afc60 0x7f11d80b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.754+0000 7f11fa569740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f573eaf390 0x55f574252a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.754+0000 7f11fa569740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f573e864d0 0x55f574252f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.754+0000 7f11fa569740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573e8e6b0 0x55f57425f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.754+0000 7f11fa569740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f5741b5d80 con 0x55f573e8e6b0 2024-04-25T11:45:21.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.754+0000 7f11fa569740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f57416fa20 con 0x55f573eaf390 2024-04-25T11:45:21.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.754+0000 7f11fa569740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f5741b1d40 con 0x55f573e864d0 2024-04-25T11:45:21.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.754+0000 7f11f0c51640 1 --2- 172.21.15.90:0/237449239 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f11d80afc60 0x7f11d80b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.754+0000 7f11f1c53640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573e8e6b0 0x55f57425f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.754+0000 7f11f1c53640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573e8e6b0 0x55f57425f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42066/0 (socket says 172.21.15.90:42066) 2024-04-25T11:45:21.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.754+0000 7f11e27fc640 1 -- 172.21.15.90:0/237449239 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7f11e40debc0 con 0x7f11ec10d380 2024-04-25T11:45:21.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.754+0000 7f11f1c53640 1 -- 172.21.15.90:0/239685090 learned_addr learned my addr 172.21.15.90:0/239685090 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:21.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.754+0000 7f11f1452640 1 --2- 172.21.15.90:0/239685090 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f573eaf390 0x55f574252a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.754+0000 7f11f0c51640 1 --2- 172.21.15.90:0/239685090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f573e864d0 0x55f574252f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.754+0000 7f11f1c53640 1 -- 172.21.15.90:0/239685090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f573e864d0 msgr2=0x55f574252f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.754+0000 7f11f1c53640 1 --2- 172.21.15.90:0/239685090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f573e864d0 0x55f574252f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.754+0000 7f11f1c53640 1 -- 172.21.15.90:0/239685090 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f573eaf390 msgr2=0x55f574252a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.754+0000 7f11f1c53640 1 --2- 172.21.15.90:0/239685090 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f573eaf390 0x55f574252a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.754+0000 7f11f1c53640 1 -- 172.21.15.90:0/239685090 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f57417ac30 con 0x55f573e8e6b0 2024-04-25T11:45:21.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.754+0000 7f11f0c51640 1 --2- 172.21.15.90:0/237449239 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f11d80afc60 0x7f11d80b2170 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f11dc02a930 tx=0x7f11dc032d10 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:21.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.754+0000 7f11f1c53640 1 --2- 172.21.15.90:0/239685090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573e8e6b0 0x55f57425f8b0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f11e40a3f20 tx=0x7f11e40a2ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b60826b8ca47c5f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:21.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.755+0000 7f11d77fe640 1 -- 172.21.15.90:0/239685090 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e4020070 con 0x55f573e8e6b0 2024-04-25T11:45:21.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.755+0000 7f11d77fe640 1 -- 172.21.15.90:0/239685090 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11e40a73b0 con 0x55f573e8e6b0 2024-04-25T11:45:21.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.755+0000 7f11d77fe640 1 -- 172.21.15.90:0/239685090 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e40a2f40 con 0x55f573e8e6b0 2024-04-25T11:45:21.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.755+0000 7f11fa569740 1 -- 172.21.15.90:0/239685090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573e8e6b0 msgr2=0x55f57425f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.755+0000 7f11fa569740 1 --2- 172.21.15.90:0/239685090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573e8e6b0 0x55f57425f8b0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f11e40a3f20 tx=0x7f11e40a2ca0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.755+0000 7f11fa569740 1 -- 172.21.15.90:0/239685090 shutdown_connections 2024-04-25T11:45:21.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.755+0000 7f11fa569740 1 --2- 172.21.15.90:0/239685090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573e8e6b0 0x55f57425f8b0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.755+0000 7f11fa569740 1 --2- 172.21.15.90:0/239685090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f573e864d0 0x55f574252f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.755+0000 7f11fa569740 1 --2- 172.21.15.90:0/239685090 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f573eaf390 0x55f574252a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.755+0000 7f11fa569740 1 -- 172.21.15.90:0/239685090 >> 172.21.15.90:0/239685090 conn(0x55f573ea34e0 msgr2=0x55f57424f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:21.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.755+0000 7f11fa569740 1 -- 172.21.15.90:0/239685090 shutdown_connections 2024-04-25T11:45:21.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.756+0000 7f11fa569740 1 -- 172.21.15.90:0/239685090 wait complete. 2024-04-25T11:45:21.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.756+0000 7f11fa569740 1 -- start start 2024-04-25T11:45:21.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.756+0000 7f11fa569740 1 Processor -- start 2024-04-25T11:45:21.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.756+0000 7f11fa569740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f573e864d0 0x55f57425d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.756+0000 7f11fa569740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f573e8e6b0 0x55f57425da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.756+0000 7f11fa569740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573eaf390 0x55f5742694e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.756+0000 7f11fa569740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f57425ffa0 con 0x55f573eaf390 2024-04-25T11:45:21.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.756+0000 7f11fa569740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55f57413cdc0 con 0x55f573e864d0 2024-04-25T11:45:21.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.756+0000 7f11fa569740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f574260120 con 0x55f573e8e6b0 2024-04-25T11:45:21.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.756+0000 7f11fa569740 10 client.0 _ll_register_callbacks cb 0x55f574158600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:21.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.756+0000 7f11f1c53640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573eaf390 0x55f5742694e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.757+0000 7f11f1c53640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573eaf390 0x55f5742694e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42076/0 (socket says 172.21.15.90:42076) 2024-04-25T11:45:21.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.757+0000 7f11f1c53640 1 -- 172.21.15.90:0/1416212292 learned_addr learned my addr 172.21.15.90:0/1416212292 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:21.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.757+0000 7f11f1452640 1 --2- 172.21.15.90:0/1416212292 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f573e864d0 0x55f57425d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.757+0000 7f11f0c51640 1 --2- 172.21.15.90:0/1416212292 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f573e8e6b0 0x55f57425da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.757+0000 7f11f1c53640 1 -- 172.21.15.90:0/1416212292 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f573e8e6b0 msgr2=0x55f57425da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.757+0000 7f11f1c53640 1 --2- 172.21.15.90:0/1416212292 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f573e8e6b0 0x55f57425da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.757+0000 7f11f1c53640 1 -- 172.21.15.90:0/1416212292 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f573e864d0 msgr2=0x55f57425d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.757+0000 7f11f1c53640 1 --2- 172.21.15.90:0/1416212292 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f573e864d0 0x55f57425d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.757+0000 7f11f1c53640 1 -- 172.21.15.90:0/1416212292 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f573ec73f0 con 0x55f573eaf390 2024-04-25T11:45:21.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.757+0000 7f11f1452640 1 --2- 172.21.15.90:0/1416212292 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f573e864d0 0x55f57425d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:21.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.757+0000 7f11f1c53640 1 --2- 172.21.15.90:0/1416212292 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573eaf390 0x55f5742694e0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f11e40a69b0 tx=0x7f11e401cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27ee31126df9a20 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:21.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.757+0000 7f11d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:21.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.757+0000 7f11d4ff9640 1 -- 172.21.15.90:0/1416212292 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e4020050 con 0x55f573eaf390 2024-04-25T11:45:21.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.757+0000 7f11d4ff9640 1 -- 172.21.15.90:0/1416212292 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7f11e4011030 con 0x55f573eaf390 2024-04-25T11:45:21.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.757+0000 7f11fa569740 1 -- 172.21.15.90:0/1416212292 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11e40a73b0 con 0x55f573eaf390 2024-04-25T11:45:21.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.757+0000 7f11d4ff9640 1 -- 172.21.15.90:0/1416212292 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f11e40afb10 con 0x55f573eaf390 2024-04-25T11:45:21.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.757+0000 7f11fa569740 10 client.18870 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:21.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.757+0000 7f11fa569740 1 -- 172.21.15.90:0/1416212292 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f573f917d0 con 0x55f573eaf390 2024-04-25T11:45:21.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.757+0000 7f11d4ff9640 1 -- 172.21.15.90:0/1416212292 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11e40a73b0 con 0x55f573eaf390 2024-04-25T11:45:21.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.757+0000 7f11d4ff9640 1 -- 172.21.15.90:0/1416212292 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f11e4073320 con 0x55f573eaf390 2024-04-25T11:45:21.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.758+0000 7f11fa569740 10 client.18870 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:21.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.758+0000 7f11fa569740 10 client.18870 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:45:21.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.758+0000 7f11fa569740 20 client.18870 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:21.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.758+0000 7f11fa569740 1 --2- 172.21.15.90:0/1416212292 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f573e6df60 0x55f57426f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.758+0000 7f11fa569740 4 client.18870 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:45:21.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.758+0000 7f11fa569740 1 -- 172.21.15.90:0/1416212292 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f5741c3990 con 0x55f573e6df60 2024-04-25T11:45:21.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.758+0000 7f11f1452640 1 --2- 172.21.15.90:0/1416212292 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f573e6df60 0x55f57426f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.758+0000 7f11d4ff9640 10 client.18870 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:45:21.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.758+0000 7f11f1452640 1 --2- 172.21.15.90:0/1416212292 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f573e6df60 0x55f57426f810 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=23ccd8feac419d16 server_cookie=153ad30cbd010030 in_seq=0 out_seq=0 2024-04-25T11:45:21.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.759+0000 7f11d4ff9640 1 -- 172.21.15.90:0/1416212292 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f5741c3990 con 0x55f573e6df60 2024-04-25T11:45:21.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.759+0000 7f11d4ff9640 10 client.18870 handle_command_reply: tid=0 2024-04-25T11:45:21.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.759+0000 7f11fa569740 1 client.18870 shutdown 2024-04-25T11:45:21.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.759+0000 7f11fa569740 1 -- 172.21.15.90:0/1416212292 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573eaf390 msgr2=0x55f5742694e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.759+0000 7f11fa569740 1 --2- 172.21.15.90:0/1416212292 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573eaf390 0x55f5742694e0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f11e40a69b0 tx=0x7f11e401cec0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.759+0000 7f11fa569740 1 -- 172.21.15.90:0/1416212292 shutdown_connections 2024-04-25T11:45:21.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.759+0000 7f11fa569740 1 --2- 172.21.15.90:0/1416212292 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f573e6df60 0x55f57426f810 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.759+0000 7f11fa569740 1 --2- 172.21.15.90:0/1416212292 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573eaf390 0x55f5742694e0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.759+0000 7f11d4ff9640 0 client.18870 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:45:21.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.759+0000 7f11fa569740 1 --2- 172.21.15.90:0/1416212292 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f573e8e6b0 0x55f57425da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.759+0000 7f11fa569740 1 --2- 172.21.15.90:0/1416212292 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f573e864d0 0x55f57425d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.759+0000 7f11fa569740 1 -- 172.21.15.90:0/1416212292 >> 172.21.15.90:0/1416212292 conn(0x55f573ea34e0 msgr2=0x55f574254740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:21.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.760+0000 7f11fa569740 1 -- 172.21.15.90:0/1416212292 shutdown_connections 2024-04-25T11:45:21.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.760+0000 7f11fa569740 1 -- 172.21.15.90:0/1416212292 wait complete. 2024-04-25T11:45:21.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.760+0000 7f11fa569740 20 client.18870 trim_cache size 0 max 16384 2024-04-25T11:45:21.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.773+0000 7f11fa569740 1 Processor -- start 2024-04-25T11:45:21.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.773+0000 7f11fa569740 1 -- start start 2024-04-25T11:45:21.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.773+0000 7f11fa569740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f573e864d0 0x55f574252e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.773+0000 7f11fa569740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573e8e6b0 0x55f57425cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.773+0000 7f11fa569740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f573eaf390 0x55f574257640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.773+0000 7f11fa569740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1adc10 con 0x55f573e8e6b0 2024-04-25T11:45:21.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.773+0000 7f11fa569740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1af750 con 0x55f573e864d0 2024-04-25T11:45:21.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.773+0000 7f11fa569740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1ae2e0 con 0x55f573eaf390 2024-04-25T11:45:21.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.773+0000 7f11f0c51640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573e8e6b0 0x55f57425cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.773+0000 7f11f0c51640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573e8e6b0 0x55f57425cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42092/0 (socket says 172.21.15.90:42092) 2024-04-25T11:45:21.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.773+0000 7f11f0c51640 1 -- 172.21.15.90:0/1320169811 learned_addr learned my addr 172.21.15.90:0/1320169811 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:21.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.774+0000 7f11f1452640 1 --2- 172.21.15.90:0/1320169811 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f573e864d0 0x55f574252e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.774+0000 7f11f0c51640 1 -- 172.21.15.90:0/1320169811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f573eaf390 msgr2=0x55f574257640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.774+0000 7f11f0c51640 1 --2- 172.21.15.90:0/1320169811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f573eaf390 0x55f574257640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.774+0000 7f11f0c51640 1 -- 172.21.15.90:0/1320169811 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f573e864d0 msgr2=0x55f574252e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.774+0000 7f11f0c51640 1 --2- 172.21.15.90:0/1320169811 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f573e864d0 0x55f574252e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.774+0000 7f11f0c51640 1 -- 172.21.15.90:0/1320169811 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f57425ddb0 con 0x55f573e8e6b0 2024-04-25T11:45:21.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.774+0000 7f11f1452640 1 --2- 172.21.15.90:0/1320169811 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f573e864d0 0x55f574252e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:21.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.774+0000 7f11f0c51640 1 --2- 172.21.15.90:0/1320169811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573e8e6b0 0x55f57425cbb0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f11ec06c4d0 tx=0x7f11dc041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3c9e55e48be6e6d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:21.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.774+0000 7f11d6ffd640 1 -- 172.21.15.90:0/1320169811 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc045070 con 0x55f573e8e6b0 2024-04-25T11:45:21.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.774+0000 7f11d6ffd640 1 -- 172.21.15.90:0/1320169811 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11dc004030 con 0x55f573e8e6b0 2024-04-25T11:45:21.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.774+0000 7f11d6ffd640 1 -- 172.21.15.90:0/1320169811 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc035040 con 0x55f573e8e6b0 2024-04-25T11:45:21.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.775+0000 7f11fa569740 1 -- 172.21.15.90:0/1320169811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573e8e6b0 msgr2=0x55f57425cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.775+0000 7f11fa569740 1 --2- 172.21.15.90:0/1320169811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573e8e6b0 0x55f57425cbb0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f11ec06c4d0 tx=0x7f11dc041000 comp rx=0 tx=0).stop 2024-04-25T11:45:21.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.775+0000 7f11fa569740 1 -- 172.21.15.90:0/1320169811 shutdown_connections 2024-04-25T11:45:21.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.775+0000 7f11fa569740 1 --2- 172.21.15.90:0/1320169811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f573eaf390 0x55f574257640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.775+0000 7f11fa569740 1 --2- 172.21.15.90:0/1320169811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573e8e6b0 0x55f57425cbb0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.775+0000 7f11fa569740 1 --2- 172.21.15.90:0/1320169811 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f573e864d0 0x55f574252e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.775+0000 7f11fa569740 1 -- 172.21.15.90:0/1320169811 >> 172.21.15.90:0/1320169811 conn(0x55f573ea34e0 msgr2=0x55f57424c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:21.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.775+0000 7f11fa569740 1 -- 172.21.15.90:0/1320169811 shutdown_connections 2024-04-25T11:45:21.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.775+0000 7f11fa569740 1 -- 172.21.15.90:0/1320169811 wait complete. 2024-04-25T11:45:21.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.775+0000 7f11fa569740 1 -- start start 2024-04-25T11:45:21.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.775+0000 7f11fa569740 1 Processor -- start 2024-04-25T11:45:21.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.776+0000 7f11fa569740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573e864d0 0x55f57426fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.776+0000 7f11fa569740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f573e8e6b0 0x55f574270210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.776+0000 7f11fa569740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f573eaf390 0x55f5742cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.776+0000 7f11fa569740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1af080 con 0x55f573e864d0 2024-04-25T11:45:21.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.776+0000 7f11fa569740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1ae9b0 con 0x55f573e8e6b0 2024-04-25T11:45:21.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.776+0000 7f11fa569740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1abc10 con 0x55f573eaf390 2024-04-25T11:45:21.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.776+0000 7f11fa569740 10 client.0 _ll_register_callbacks cb 0x7f11e40a6420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:21.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.776+0000 7f11f1452640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573e864d0 0x55f57426fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.776+0000 7f11f1452640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573e864d0 0x55f57426fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42104/0 (socket says 172.21.15.90:42104) 2024-04-25T11:45:21.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.776+0000 7f11f1452640 1 -- 172.21.15.90:0/2426456829 learned_addr learned my addr 172.21.15.90:0/2426456829 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:21.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.776+0000 7f11f0c51640 1 --2- 172.21.15.90:0/2426456829 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f573e8e6b0 0x55f574270210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.776+0000 7f11f1c53640 1 --2- 172.21.15.90:0/2426456829 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f573eaf390 0x55f5742cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.776+0000 7f11f1452640 1 -- 172.21.15.90:0/2426456829 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f573eaf390 msgr2=0x55f5742cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.776+0000 7f11f1452640 1 --2- 172.21.15.90:0/2426456829 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f573eaf390 0x55f5742cf970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.776+0000 7f11f1452640 1 -- 172.21.15.90:0/2426456829 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f573e8e6b0 msgr2=0x55f574270210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.776+0000 7f11f1452640 1 --2- 172.21.15.90:0/2426456829 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f573e8e6b0 0x55f574270210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.776+0000 7f11f1452640 1 -- 172.21.15.90:0/2426456829 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5742cfee0 con 0x55f573e864d0 2024-04-25T11:45:21.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.776+0000 7f11f1452640 1 --2- 172.21.15.90:0/2426456829 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573e864d0 0x55f57426fcd0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f11e8009f70 tx=0x7f11e8013980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42bb7a1239a9ca40 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:21.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.776+0000 7f11d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:21.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.776+0000 7f11d4ff9640 1 -- 172.21.15.90:0/2426456829 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e801b640 con 0x55f573e864d0 2024-04-25T11:45:21.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.776+0000 7f11d4ff9640 1 -- 172.21.15.90:0/2426456829 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7f11e801b7e0 con 0x55f573e864d0 2024-04-25T11:45:21.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.776+0000 7f11fa569740 1 -- 172.21.15.90:0/2426456829 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11dc004030 con 0x55f573e864d0 2024-04-25T11:45:21.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.777+0000 7f11d4ff9640 1 -- 172.21.15.90:0/2426456829 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f11e801b9e0 con 0x55f573e864d0 2024-04-25T11:45:21.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.777+0000 7f11fa569740 10 client.18876 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:21.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.777+0000 7f11fa569740 1 -- 172.21.15.90:0/2426456829 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f5742d7190 con 0x55f573e864d0 2024-04-25T11:45:21.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.777+0000 7f11d4ff9640 1 -- 172.21.15.90:0/2426456829 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11dc004030 con 0x55f573e864d0 2024-04-25T11:45:21.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.777+0000 7f11d4ff9640 1 -- 172.21.15.90:0/2426456829 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f11e8023470 con 0x55f573e864d0 2024-04-25T11:45:21.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.777+0000 7f11fa569740 10 client.18876 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:21.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.777+0000 7f11fa569740 10 client.18876 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:45:21.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.777+0000 7f11fa569740 20 client.18876 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:21.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.777+0000 7f11fa569740 1 --2- 172.21.15.90:0/2426456829 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f573e433a0 0x55f5742d9930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.777+0000 7f11fa569740 4 client.18876 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:45:21.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.777+0000 7f11fa569740 1 -- 172.21.15.90:0/2426456829 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f5741bd800 con 0x55f573e433a0 2024-04-25T11:45:21.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.777+0000 7f11f0c51640 1 --2- 172.21.15.90:0/2426456829 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f573e433a0 0x55f5742d9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.778+0000 7f11d4ff9640 10 client.18876 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:45:21.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.778+0000 7f11f0c51640 1 --2- 172.21.15.90:0/2426456829 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f573e433a0 0x55f5742d9930 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3d8be995e2fa09ca server_cookie=435b0d7e103aac1c in_seq=0 out_seq=0 2024-04-25T11:45:21.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.778+0000 7f11d4ff9640 1 -- 172.21.15.90:0/2426456829 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f5741bd800 con 0x55f573e433a0 2024-04-25T11:45:21.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.778+0000 7f11d4ff9640 10 client.18876 handle_command_reply: tid=0 2024-04-25T11:45:21.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.778+0000 7f11fa569740 1 client.18876 shutdown 2024-04-25T11:45:21.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.778+0000 7f11fa569740 1 -- 172.21.15.90:0/2426456829 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573e864d0 msgr2=0x55f57426fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.778+0000 7f11fa569740 1 --2- 172.21.15.90:0/2426456829 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573e864d0 0x55f57426fcd0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f11e8009f70 tx=0x7f11e8013980 comp rx=0 tx=0).stop 2024-04-25T11:45:21.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.778+0000 7f11fa569740 1 -- 172.21.15.90:0/2426456829 shutdown_connections 2024-04-25T11:45:21.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.778+0000 7f11fa569740 1 --2- 172.21.15.90:0/2426456829 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55f573e433a0 0x55f5742d9930 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.778+0000 7f11fa569740 1 --2- 172.21.15.90:0/2426456829 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f573eaf390 0x55f5742cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.778+0000 7f11fa569740 1 --2- 172.21.15.90:0/2426456829 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55f573e8e6b0 0x55f574270210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.778+0000 7f11fa569740 1 --2- 172.21.15.90:0/2426456829 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f573e864d0 0x55f57426fcd0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.778+0000 7f11d4ff9640 0 client.18876 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:45:21.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.778+0000 7f11fa569740 1 -- 172.21.15.90:0/2426456829 >> 172.21.15.90:0/2426456829 conn(0x55f573ea34e0 msgr2=0x55f57424bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:21.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.778+0000 7f11fa569740 1 -- 172.21.15.90:0/2426456829 shutdown_connections 2024-04-25T11:45:21.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.778+0000 7f11fa569740 1 -- 172.21.15.90:0/2426456829 wait complete. 2024-04-25T11:45:21.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.778+0000 7f11fa569740 20 client.18876 trim_cache size 0 max 16384 2024-04-25T11:45:21.771 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:45:21.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.779+0000 7f11f36b6640 1 -- 172.21.15.90:0/237449239 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f11d80afc60 msgr2=0x7f11d80b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.779+0000 7f11f36b6640 1 --2- 172.21.15.90:0/237449239 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f11d80afc60 0x7f11d80b2170 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f11dc02a930 tx=0x7f11dc032d10 comp rx=0 tx=0).stop 2024-04-25T11:45:21.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.779+0000 7f11f36b6640 1 -- 172.21.15.90:0/237449239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11ec10d380 msgr2=0x7f11ec19de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.779+0000 7f11f36b6640 1 --2- 172.21.15.90:0/237449239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11ec10d380 0x7f11ec19de80 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f11e400d750 tx=0x7f11e400dc10 comp rx=0 tx=0).stop 2024-04-25T11:45:21.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.779+0000 7f11f36b6640 1 -- 172.21.15.90:0/237449239 shutdown_connections 2024-04-25T11:45:21.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.779+0000 7f11f36b6640 1 --2- 172.21.15.90:0/237449239 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f11d80afc60 0x7f11d80b2170 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.779+0000 7f11f36b6640 1 --2- 172.21.15.90:0/237449239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11ec10d380 0x7f11ec19de80 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.779+0000 7f11f36b6640 1 --2- 172.21.15.90:0/237449239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11ec108a20 0x7f11ec199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.779+0000 7f11f36b6640 1 --2- 172.21.15.90:0/237449239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11ec102af0 0x7f11ec199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.779+0000 7f11f36b6640 1 -- 172.21.15.90:0/237449239 >> 172.21.15.90:0/237449239 conn(0x7f11ec078100 msgr2=0x7f11ec1040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:21.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.779+0000 7f11f36b6640 1 -- 172.21.15.90:0/237449239 shutdown_connections 2024-04-25T11:45:21.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.779+0000 7f11f36b6640 1 -- 172.21.15.90:0/237449239 wait complete. 2024-04-25T11:45:21.775 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T11:45:21.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.899+0000 7fbecc98c640 1 -- 172.21.15.90:0/1645571613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbec8108a20 msgr2=0x7fbec81025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.899+0000 7fbecc98c640 1 --2- 172.21.15.90:0/1645571613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbec8108a20 0x7fbec81025b0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8009900 tx=0x7fbeb802a410 comp rx=0 tx=0).stop 2024-04-25T11:45:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.899+0000 7fbecc98c640 1 -- 172.21.15.90:0/1645571613 shutdown_connections 2024-04-25T11:45:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.899+0000 7fbecc98c640 1 --2- 172.21.15.90:0/1645571613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbec810d380 0x7fbec810f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.899+0000 7fbecc98c640 1 --2- 172.21.15.90:0/1645571613 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbec8102af0 0x7fbec8102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.899+0000 7fbecc98c640 1 --2- 172.21.15.90:0/1645571613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbec8108a20 0x7fbec81025b0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.899+0000 7fbecc98c640 1 -- 172.21.15.90:0/1645571613 >> 172.21.15.90:0/1645571613 conn(0x7fbec8078100 msgr2=0x7fbec8078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.899+0000 7fbecc98c640 1 -- 172.21.15.90:0/1645571613 shutdown_connections 2024-04-25T11:45:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.899+0000 7fbecc98c640 1 -- 172.21.15.90:0/1645571613 wait complete. 2024-04-25T11:45:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.900+0000 7fbecc98c640 1 Processor -- start 2024-04-25T11:45:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.900+0000 7fbecc98c640 1 -- start start 2024-04-25T11:45:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.900+0000 7fbecc98c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbec8102af0 0x7fbec819a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.900+0000 7fbecc98c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbec8108a20 0x7fbec819ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.900+0000 7fbecc98c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbec810d380 0x7fbec819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.900+0000 7fbecc98c640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbec8113d00 con 0x7fbec810d380 2024-04-25T11:45:21.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.900+0000 7fbecc98c640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbec8113b80 con 0x7fbec8102af0 2024-04-25T11:45:21.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.900+0000 7fbecc98c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbec8113e80 con 0x7fbec8108a20 2024-04-25T11:45:21.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.900+0000 7fbec6d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbec810d380 0x7fbec819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.900+0000 7fbec6d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbec810d380 0x7fbec819ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42128/0 (socket says 172.21.15.90:42128) 2024-04-25T11:45:21.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.900+0000 7fbec6d9d640 1 -- 172.21.15.90:0/1077576577 learned_addr learned my addr 172.21.15.90:0/1077576577 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:21.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.900+0000 7fbebffff640 1 --2- 172.21.15.90:0/1077576577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbec8108a20 0x7fbec819ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.901+0000 7fbec659c640 1 --2- 172.21.15.90:0/1077576577 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbec8102af0 0x7fbec819a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.901+0000 7fbec6d9d640 1 -- 172.21.15.90:0/1077576577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbec8108a20 msgr2=0x7fbec819ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.901+0000 7fbec6d9d640 1 --2- 172.21.15.90:0/1077576577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbec8108a20 0x7fbec819ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.901+0000 7fbec6d9d640 1 -- 172.21.15.90:0/1077576577 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbec8102af0 msgr2=0x7fbec819a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.901+0000 7fbec6d9d640 1 --2- 172.21.15.90:0/1077576577 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbec8102af0 0x7fbec819a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.901+0000 7fbec6d9d640 1 -- 172.21.15.90:0/1077576577 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbec819f560 con 0x7fbec810d380 2024-04-25T11:45:21.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.901+0000 7fbec659c640 1 --2- 172.21.15.90:0/1077576577 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbec8102af0 0x7fbec819a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:21.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.901+0000 7fbec6d9d640 1 --2- 172.21.15.90:0/1077576577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbec810d380 0x7fbec819ef20 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fbec000cad0 tx=0x7fbec000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:21.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.901+0000 7fbebf7fe640 1 -- 172.21.15.90:0/1077576577 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec000ae20 con 0x7fbec810d380 2024-04-25T11:45:21.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.901+0000 7fbebf7fe640 1 -- 172.21.15.90:0/1077576577 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbec0005020 con 0x7fbec810d380 2024-04-25T11:45:21.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.901+0000 7fbecc98c640 1 -- 172.21.15.90:0/1077576577 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbec819f850 con 0x7fbec810d380 2024-04-25T11:45:21.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.901+0000 7fbebf7fe640 1 -- 172.21.15.90:0/1077576577 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec000f6b0 con 0x7fbec810d380 2024-04-25T11:45:21.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.901+0000 7fbecc98c640 1 -- 172.21.15.90:0/1077576577 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbec81a71d0 con 0x7fbec810d380 2024-04-25T11:45:21.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.903+0000 7fbebf7fe640 1 -- 172.21.15.90:0/1077576577 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fbec000f850 con 0x7fbec810d380 2024-04-25T11:45:21.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.904+0000 7fbed383d740 1 Processor -- start 2024-04-25T11:45:21.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.904+0000 7fbed383d740 1 -- start start 2024-04-25T11:45:21.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.904+0000 7fbed383d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560745512390 0x5607458b3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.904+0000 7fbed383d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5607454e94d0 0x5607458b3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.904+0000 7fbed383d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5607454f16b0 0x5607458c06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.904+0000 7fbed383d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560745818d80 con 0x5607454e94d0 2024-04-25T11:45:21.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.904+0000 7fbed383d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5607457d2a20 con 0x5607454f16b0 2024-04-25T11:45:21.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.904+0000 7fbed383d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560745814d40 con 0x560745512390 2024-04-25T11:45:21.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.904+0000 7fbebf7fe640 1 --2- 172.21.15.90:0/1077576577 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbeb40afc60 0x7fbeb40b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.904+0000 7fbebffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560745512390 0x5607458b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.904+0000 7fbec659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5607454e94d0 0x5607458b3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.904+0000 7fbebffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560745512390 0x5607458b3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:47554/0 (socket says 172.21.15.90:47554) 2024-04-25T11:45:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.904+0000 7fbebffff640 1 -- 172.21.15.90:0/112645436 learned_addr learned my addr 172.21.15.90:0/112645436 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.904+0000 7fbebffff640 1 --2- 172.21.15.90:0/112645436 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5607454f16b0 0x5607458c06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.904+0000 7fbec6d9d640 1 --2- 172.21.15.90:0/1077576577 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbeb40afc60 0x7fbeb40b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.905+0000 7fbebffff640 1 -- 172.21.15.90:0/112645436 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5607454f16b0 msgr2=0x5607458c06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.905+0000 7fbebffff640 1 --2- 172.21.15.90:0/112645436 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5607454f16b0 0x5607458c06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.905+0000 7fbebf7fe640 1 -- 172.21.15.90:0/1077576577 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7fbec00df060 con 0x7fbec810d380 2024-04-25T11:45:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.905+0000 7fbebffff640 1 -- 172.21.15.90:0/112645436 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5607454e94d0 msgr2=0x5607458b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.905+0000 7fbebffff640 1 --2- 172.21.15.90:0/112645436 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5607454e94d0 0x5607458b3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.905+0000 7fbebffff640 1 -- 172.21.15.90:0/112645436 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607457ddc30 con 0x560745512390 2024-04-25T11:45:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.905+0000 7fbebffff640 1 --2- 172.21.15.90:0/112645436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560745512390 0x5607458b3810 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0012740 tx=0x7fbeb0005c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ecf36a270dc5a52 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.905+0000 7fbec6d9d640 1 --2- 172.21.15.90:0/1077576577 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbeb40afc60 0x7fbeb40b2170 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fbec000caa0 tx=0x7fbec00ab040 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.905+0000 7fbebcff9640 1 -- 172.21.15.90:0/112645436 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0012d80 con 0x560745512390 2024-04-25T11:45:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.905+0000 7fbebcff9640 1 -- 172.21.15.90:0/112645436 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbeb00054d0 con 0x560745512390 2024-04-25T11:45:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.905+0000 7fbebcff9640 1 -- 172.21.15.90:0/112645436 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0010e90 con 0x560745512390 2024-04-25T11:45:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.906+0000 7fbed383d740 1 -- 172.21.15.90:0/112645436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560745512390 msgr2=0x5607458b3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.906+0000 7fbed383d740 1 --2- 172.21.15.90:0/112645436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560745512390 0x5607458b3810 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0012740 tx=0x7fbeb0005c50 comp rx=0 tx=0).stop 2024-04-25T11:45:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.906+0000 7fbed383d740 1 -- 172.21.15.90:0/112645436 shutdown_connections 2024-04-25T11:45:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.906+0000 7fbed383d740 1 --2- 172.21.15.90:0/112645436 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5607454f16b0 0x5607458c06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.906+0000 7fbed383d740 1 --2- 172.21.15.90:0/112645436 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5607454e94d0 0x5607458b3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.906+0000 7fbed383d740 1 --2- 172.21.15.90:0/112645436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560745512390 0x5607458b3810 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.906+0000 7fbed383d740 1 -- 172.21.15.90:0/112645436 >> 172.21.15.90:0/112645436 conn(0x5607455064e0 msgr2=0x5607458affc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.906+0000 7fbed383d740 1 -- 172.21.15.90:0/112645436 shutdown_connections 2024-04-25T11:45:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.906+0000 7fbed383d740 1 -- 172.21.15.90:0/112645436 wait complete. 2024-04-25T11:45:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.907+0000 7fbed383d740 1 -- start start 2024-04-25T11:45:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.907+0000 7fbed383d740 1 Processor -- start 2024-04-25T11:45:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.907+0000 7fbed383d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607454e94d0 0x5607458be290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.907+0000 7fbed383d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5607454f16b0 0x5607458be7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.907+0000 7fbed383d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560745512390 0x5607458ca210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.907+0000 7fbed383d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5607458c0d90 con 0x560745512390 2024-04-25T11:45:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.907+0000 7fbed383d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56074579fdc0 con 0x5607454f16b0 2024-04-25T11:45:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.907+0000 7fbed383d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5607458c0f10 con 0x5607454e94d0 2024-04-25T11:45:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.907+0000 7fbed383d740 10 client.0 _ll_register_callbacks cb 0x5607457bb600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.907+0000 7fbebffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560745512390 0x5607458ca210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.907+0000 7fbebffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560745512390 0x5607458ca210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42156/0 (socket says 172.21.15.90:42156) 2024-04-25T11:45:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.907+0000 7fbebffff640 1 -- 172.21.15.90:0/921158949 learned_addr learned my addr 172.21.15.90:0/921158949 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.907+0000 7fbebffff640 1 --2- 172.21.15.90:0/921158949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607454e94d0 0x5607458be290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.907+0000 7fbec659c640 1 --2- 172.21.15.90:0/921158949 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5607454f16b0 0x5607458be7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.907+0000 7fbebffff640 1 -- 172.21.15.90:0/921158949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607454e94d0 msgr2=0x5607458be290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.907+0000 7fbebffff640 1 --2- 172.21.15.90:0/921158949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607454e94d0 0x5607458be290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.907+0000 7fbebffff640 1 -- 172.21.15.90:0/921158949 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5607454f16b0 msgr2=0x5607458be7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.907+0000 7fbebffff640 1 --2- 172.21.15.90:0/921158949 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5607454f16b0 0x5607458be7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.907+0000 7fbebffff640 1 -- 172.21.15.90:0/921158949 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56074552a3f0 con 0x560745512390 2024-04-25T11:45:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.908+0000 7fbebffff640 1 --2- 172.21.15.90:0/921158949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560745512390 0x5607458ca210 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fbeb00057c0 tx=0x7fbeb0017040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66792d38212c6b8b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.908+0000 7fbeadffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.908+0000 7fbeadffb640 1 -- 172.21.15.90:0/921158949 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0010c30 con 0x560745512390 2024-04-25T11:45:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.908+0000 7fbeadffb640 1 -- 172.21.15.90:0/921158949 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7fbeb0010dd0 con 0x560745512390 2024-04-25T11:45:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.908+0000 7fbed383d740 1 -- 172.21.15.90:0/921158949 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbeb00054d0 con 0x560745512390 2024-04-25T11:45:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.908+0000 7fbeadffb640 1 -- 172.21.15.90:0/921158949 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbeb0033050 con 0x560745512390 2024-04-25T11:45:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.908+0000 7fbed383d740 10 client.18888 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.908+0000 7fbed383d740 1 -- 172.21.15.90:0/921158949 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5607455f47d0 con 0x560745512390 2024-04-25T11:45:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.908+0000 7fbeadffb640 1 -- 172.21.15.90:0/921158949 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbeb00054d0 con 0x560745512390 2024-04-25T11:45:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.908+0000 7fbeadffb640 1 -- 172.21.15.90:0/921158949 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fbeb002b020 con 0x560745512390 2024-04-25T11:45:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.908+0000 7fbed383d740 10 client.18888 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:21.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.908+0000 7fbed383d740 10 client.18888 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:45:21.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.908+0000 7fbed383d740 20 client.18888 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:21.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.908+0000 7fbed383d740 1 --2- 172.21.15.90:0/921158949 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5607454d0f60 0x5607458d0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.908+0000 7fbed383d740 4 client.18888 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:45:21.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.908+0000 7fbed383d740 1 -- 172.21.15.90:0/921158949 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560745826990 con 0x5607454d0f60 2024-04-25T11:45:21.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.909+0000 7fbec659c640 1 --2- 172.21.15.90:0/921158949 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5607454d0f60 0x5607458d0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.909+0000 7fbeadffb640 10 client.18888 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:45:21.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.909+0000 7fbec659c640 1 --2- 172.21.15.90:0/921158949 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5607454d0f60 0x5607458d0500 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a8b2aed933224ff5 server_cookie=dab776f31aea6dba in_seq=0 out_seq=0 2024-04-25T11:45:21.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.910+0000 7fbeadffb640 1 -- 172.21.15.90:0/921158949 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560745826990 con 0x5607454d0f60 2024-04-25T11:45:21.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.910+0000 7fbeadffb640 10 client.18888 handle_command_reply: tid=0 2024-04-25T11:45:21.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.910+0000 7fbed383d740 1 client.18888 shutdown 2024-04-25T11:45:21.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.910+0000 7fbed383d740 1 -- 172.21.15.90:0/921158949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560745512390 msgr2=0x5607458ca210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.910+0000 7fbed383d740 1 --2- 172.21.15.90:0/921158949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560745512390 0x5607458ca210 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fbeb00057c0 tx=0x7fbeb0017040 comp rx=0 tx=0).stop 2024-04-25T11:45:21.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.910+0000 7fbed383d740 1 -- 172.21.15.90:0/921158949 shutdown_connections 2024-04-25T11:45:21.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.910+0000 7fbed383d740 1 --2- 172.21.15.90:0/921158949 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5607454d0f60 0x5607458d0500 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.910+0000 7fbed383d740 1 --2- 172.21.15.90:0/921158949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560745512390 0x5607458ca210 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.910+0000 7fbed383d740 1 --2- 172.21.15.90:0/921158949 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5607454f16b0 0x5607458be7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.910+0000 7fbed383d740 1 --2- 172.21.15.90:0/921158949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607454e94d0 0x5607458be290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.910+0000 7fbeadffb640 0 client.18888 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:45:21.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.910+0000 7fbed383d740 1 -- 172.21.15.90:0/921158949 >> 172.21.15.90:0/921158949 conn(0x5607455064e0 msgr2=0x5607458b5530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:21.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.910+0000 7fbed383d740 1 -- 172.21.15.90:0/921158949 shutdown_connections 2024-04-25T11:45:21.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.910+0000 7fbed383d740 1 -- 172.21.15.90:0/921158949 wait complete. 2024-04-25T11:45:21.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.910+0000 7fbed383d740 20 client.18888 trim_cache size 0 max 16384 2024-04-25T11:45:21.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.924+0000 7fbed383d740 1 Processor -- start 2024-04-25T11:45:21.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.924+0000 7fbed383d740 1 -- start start 2024-04-25T11:45:21.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.924+0000 7fbed383d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607454e94d0 0x5607458aff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.924+0000 7fbed383d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5607454f16b0 0x5607458c9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.924+0000 7fbed383d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560745512390 0x5607458d5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.924+0000 7fbed383d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbec81afd70 con 0x560745512390 2024-04-25T11:45:21.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.924+0000 7fbed383d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbec81b18b0 con 0x5607454f16b0 2024-04-25T11:45:21.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.924+0000 7fbed383d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbec81b0440 con 0x5607454e94d0 2024-04-25T11:45:21.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.924+0000 7fbebffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560745512390 0x5607458d5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.925+0000 7fbebffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560745512390 0x5607458d5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42164/0 (socket says 172.21.15.90:42164) 2024-04-25T11:45:21.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.925+0000 7fbebffff640 1 -- 172.21.15.90:0/1495460068 learned_addr learned my addr 172.21.15.90:0/1495460068 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:21.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.925+0000 7fbebffff640 1 --2- 172.21.15.90:0/1495460068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607454e94d0 0x5607458aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.925+0000 7fbec659c640 1 --2- 172.21.15.90:0/1495460068 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5607454f16b0 0x5607458c9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.925+0000 7fbebffff640 1 -- 172.21.15.90:0/1495460068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607454e94d0 msgr2=0x5607458aff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.925+0000 7fbebffff640 1 --2- 172.21.15.90:0/1495460068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607454e94d0 0x5607458aff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.925+0000 7fbebffff640 1 -- 172.21.15.90:0/1495460068 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5607454f16b0 msgr2=0x5607458c9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.925+0000 7fbebffff640 1 --2- 172.21.15.90:0/1495460068 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5607454f16b0 0x5607458c9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.925+0000 7fbebffff640 1 -- 172.21.15.90:0/1495460068 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607458af620 con 0x560745512390 2024-04-25T11:45:21.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.925+0000 7fbebffff640 1 --2- 172.21.15.90:0/1495460068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560745512390 0x5607458d5b30 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0005890 tx=0x7fbeb0012f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4afe256b3a9de573 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:21.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.925+0000 7fbeaffff640 1 -- 172.21.15.90:0/1495460068 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb000bd20 con 0x560745512390 2024-04-25T11:45:21.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.925+0000 7fbeaffff640 1 -- 172.21.15.90:0/1495460068 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbeb002c670 con 0x560745512390 2024-04-25T11:45:21.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.925+0000 7fbeaffff640 1 -- 172.21.15.90:0/1495460068 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb002c950 con 0x560745512390 2024-04-25T11:45:21.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.926+0000 7fbed383d740 1 -- 172.21.15.90:0/1495460068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560745512390 msgr2=0x5607458d5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.926+0000 7fbed383d740 1 --2- 172.21.15.90:0/1495460068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560745512390 0x5607458d5b30 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0005890 tx=0x7fbeb0012f90 comp rx=0 tx=0).stop 2024-04-25T11:45:21.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.926+0000 7fbed383d740 1 -- 172.21.15.90:0/1495460068 shutdown_connections 2024-04-25T11:45:21.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.926+0000 7fbed383d740 1 --2- 172.21.15.90:0/1495460068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560745512390 0x5607458d5b30 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.926+0000 7fbed383d740 1 --2- 172.21.15.90:0/1495460068 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5607454f16b0 0x5607458c9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.926+0000 7fbed383d740 1 --2- 172.21.15.90:0/1495460068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607454e94d0 0x5607458aff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.926+0000 7fbed383d740 1 -- 172.21.15.90:0/1495460068 >> 172.21.15.90:0/1495460068 conn(0x5607455064e0 msgr2=0x5607458af860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:21.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.926+0000 7fbed383d740 1 -- 172.21.15.90:0/1495460068 shutdown_connections 2024-04-25T11:45:21.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.926+0000 7fbed383d740 1 -- 172.21.15.90:0/1495460068 wait complete. 2024-04-25T11:45:21.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.926+0000 7fbed383d740 1 -- start start 2024-04-25T11:45:21.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.926+0000 7fbed383d740 1 Processor -- start 2024-04-25T11:45:21.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.927+0000 7fbed383d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607454e94d0 0x5607458c2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.927+0000 7fbed383d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5607454f16b0 0x5607458c33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.927+0000 7fbed383d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560745512390 0x5607458c3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.927+0000 7fbed383d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbec81b11e0 con 0x5607454f16b0 2024-04-25T11:45:21.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.927+0000 7fbed383d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbec81b0b10 con 0x560745512390 2024-04-25T11:45:21.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.927+0000 7fbed383d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbec81add70 con 0x5607454e94d0 2024-04-25T11:45:21.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.927+0000 7fbed383d740 10 client.0 _ll_register_callbacks cb 0x7fbeb8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:21.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.927+0000 7fbec659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5607454f16b0 0x5607458c33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.927+0000 7fbec659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5607454f16b0 0x5607458c33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42178/0 (socket says 172.21.15.90:42178) 2024-04-25T11:45:21.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.927+0000 7fbec659c640 1 -- 172.21.15.90:0/3721416704 learned_addr learned my addr 172.21.15.90:0/3721416704 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:21.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.927+0000 7fbebffff640 1 --2- 172.21.15.90:0/3721416704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607454e94d0 0x5607458c2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.927+0000 7fbec659c640 1 -- 172.21.15.90:0/3721416704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607454e94d0 msgr2=0x5607458c2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.927+0000 7fbec659c640 1 --2- 172.21.15.90:0/3721416704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607454e94d0 0x5607458c2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.927+0000 7fbec659c640 1 -- 172.21.15.90:0/3721416704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560745512390 msgr2=0x5607458c3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:21.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.927+0000 7fbec659c640 1 --2- 172.21.15.90:0/3721416704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560745512390 0x5607458c3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.927+0000 7fbec659c640 1 -- 172.21.15.90:0/3721416704 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56074593ab20 con 0x5607454f16b0 2024-04-25T11:45:21.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.927+0000 7fbebffff640 1 --2- 172.21.15.90:0/3721416704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607454e94d0 0x5607458c2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:21.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.927+0000 7fbec659c640 1 --2- 172.21.15.90:0/3721416704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5607454f16b0 0x5607458c33f0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8032f10 tx=0x7fbeb8004320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de9eed99ad89f15c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:21.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.927+0000 7fbeadffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:21.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.927+0000 7fbeadffb640 1 -- 172.21.15.90:0/3721416704 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb8035040 con 0x5607454f16b0 2024-04-25T11:45:21.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.927+0000 7fbeadffb640 1 -- 172.21.15.90:0/3721416704 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7fbeb8005240 con 0x5607454f16b0 2024-04-25T11:45:21.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.927+0000 7fbed383d740 1 -- 172.21.15.90:0/3721416704 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbeb002c670 con 0x5607454f16b0 2024-04-25T11:45:21.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.928+0000 7fbeadffb640 1 -- 172.21.15.90:0/3721416704 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbeb8053050 con 0x5607454f16b0 2024-04-25T11:45:21.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.928+0000 7fbed383d740 10 client.18894 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:21.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.928+0000 7fbed383d740 1 -- 172.21.15.90:0/3721416704 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56074593bdc0 con 0x5607454f16b0 2024-04-25T11:45:21.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.928+0000 7fbeadffb640 1 -- 172.21.15.90:0/3721416704 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbeb002c670 con 0x5607454f16b0 2024-04-25T11:45:21.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.928+0000 7fbeadffb640 1 -- 172.21.15.90:0/3721416704 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fbeb8044ce0 con 0x5607454f16b0 2024-04-25T11:45:21.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.928+0000 7fbed383d740 10 client.18894 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:21.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.928+0000 7fbed383d740 10 client.18894 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:45:21.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.928+0000 7fbed383d740 20 client.18894 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:21.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.928+0000 7fbed383d740 1 --2- 172.21.15.90:0/3721416704 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5607454a63a0 0x56074593e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:21.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.928+0000 7fbed383d740 4 client.18894 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:45:21.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.928+0000 7fbed383d740 1 -- 172.21.15.90:0/3721416704 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560745820800 con 0x5607454a63a0 2024-04-25T11:45:21.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.928+0000 7fbebffff640 1 --2- 172.21.15.90:0/3721416704 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5607454a63a0 0x56074593e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:21.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.929+0000 7fbeadffb640 10 client.18894 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:45:21.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.929+0000 7fbebffff640 1 --2- 172.21.15.90:0/3721416704 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5607454a63a0 0x56074593e580 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6c282e11d3323e64 server_cookie=4a59c182ebf94bbb in_seq=0 out_seq=0 2024-04-25T11:45:21.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.929+0000 7fbeadffb640 1 -- 172.21.15.90:0/3721416704 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560745820800 con 0x5607454a63a0 2024-04-25T11:45:21.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.929+0000 7fbeadffb640 10 client.18894 handle_command_reply: tid=0 2024-04-25T11:45:21.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.929+0000 7fbed383d740 1 client.18894 shutdown 2024-04-25T11:45:21.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.929+0000 7fbed383d740 1 -- 172.21.15.90:0/3721416704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5607454f16b0 msgr2=0x5607458c33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.929+0000 7fbed383d740 1 --2- 172.21.15.90:0/3721416704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5607454f16b0 0x5607458c33f0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8032f10 tx=0x7fbeb8004320 comp rx=0 tx=0).stop 2024-04-25T11:45:21.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.929+0000 7fbed383d740 1 -- 172.21.15.90:0/3721416704 shutdown_connections 2024-04-25T11:45:21.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.929+0000 7fbed383d740 1 --2- 172.21.15.90:0/3721416704 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x5607454a63a0 0x56074593e580 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.929+0000 7fbed383d740 1 --2- 172.21.15.90:0/3721416704 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x560745512390 0x5607458c3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.929+0000 7fbed383d740 1 --2- 172.21.15.90:0/3721416704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5607454f16b0 0x5607458c33f0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.929+0000 7fbed383d740 1 --2- 172.21.15.90:0/3721416704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607454e94d0 0x5607458c2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.929+0000 7fbeadffb640 0 client.18894 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:45:21.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.929+0000 7fbed383d740 1 -- 172.21.15.90:0/3721416704 >> 172.21.15.90:0/3721416704 conn(0x5607455064e0 msgr2=0x5607458b5410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:21.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.929+0000 7fbed383d740 1 -- 172.21.15.90:0/3721416704 shutdown_connections 2024-04-25T11:45:21.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.929+0000 7fbed383d740 1 -- 172.21.15.90:0/3721416704 wait complete. 2024-04-25T11:45:21.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.929+0000 7fbed383d740 20 client.18894 trim_cache size 0 max 16384 2024-04-25T11:45:21.923 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:45:21.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.930+0000 7fbecc98c640 1 -- 172.21.15.90:0/1077576577 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbeb40afc60 msgr2=0x7fbeb40b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.930+0000 7fbecc98c640 1 --2- 172.21.15.90:0/1077576577 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbeb40afc60 0x7fbeb40b2170 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fbec000caa0 tx=0x7fbec00ab040 comp rx=0 tx=0).stop 2024-04-25T11:45:21.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.930+0000 7fbecc98c640 1 -- 172.21.15.90:0/1077576577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbec810d380 msgr2=0x7fbec819ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:21.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.930+0000 7fbecc98c640 1 --2- 172.21.15.90:0/1077576577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbec810d380 0x7fbec819ef20 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fbec000cad0 tx=0x7fbec000cf90 comp rx=0 tx=0).stop 2024-04-25T11:45:21.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.930+0000 7fbecc98c640 1 -- 172.21.15.90:0/1077576577 shutdown_connections 2024-04-25T11:45:21.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.930+0000 7fbecc98c640 1 --2- 172.21.15.90:0/1077576577 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbeb40afc60 0x7fbeb40b2170 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.930+0000 7fbecc98c640 1 --2- 172.21.15.90:0/1077576577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbec810d380 0x7fbec819ef20 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.930+0000 7fbecc98c640 1 --2- 172.21.15.90:0/1077576577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbec8108a20 0x7fbec819ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.930+0000 7fbecc98c640 1 --2- 172.21.15.90:0/1077576577 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbec8102af0 0x7fbec819a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:21.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.930+0000 7fbecc98c640 1 -- 172.21.15.90:0/1077576577 >> 172.21.15.90:0/1077576577 conn(0x7fbec8078100 msgr2=0x7fbec81040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:21.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.930+0000 7fbecc98c640 1 -- 172.21.15.90:0/1077576577 shutdown_connections 2024-04-25T11:45:21.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:21.930+0000 7fbecc98c640 1 -- 172.21.15.90:0/1077576577 wait complete. 2024-04-25T11:45:21.927 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T11:45:22.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.052+0000 7f1c79d84640 1 -- 172.21.15.90:0/4209694982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c74104c60 msgr2=0x7f1c74105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:22.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.052+0000 7f1c79d84640 1 --2- 172.21.15.90:0/4209694982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c74104c60 0x7f1c74105070 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f1c64009900 tx=0x7f1c6402a410 comp rx=0 tx=0).stop 2024-04-25T11:45:22.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.052+0000 7f1c79d84640 1 -- 172.21.15.90:0/4209694982 shutdown_connections 2024-04-25T11:45:22.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.052+0000 7f1c79d84640 1 --2- 172.21.15.90:0/4209694982 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c7410e0b0 0x7f1c74110590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.052+0000 7f1c79d84640 1 --2- 172.21.15.90:0/4209694982 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c7410b6c0 0x7f1c7410dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.052+0000 7f1c79d84640 1 --2- 172.21.15.90:0/4209694982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c74104c60 0x7f1c74105070 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.052+0000 7f1c79d84640 1 -- 172.21.15.90:0/4209694982 >> 172.21.15.90:0/4209694982 conn(0x7f1c740fe260 msgr2=0x7f1c74100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:22.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.052+0000 7f1c79d84640 1 -- 172.21.15.90:0/4209694982 shutdown_connections 2024-04-25T11:45:22.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.052+0000 7f1c79d84640 1 -- 172.21.15.90:0/4209694982 wait complete. 2024-04-25T11:45:22.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.053+0000 7f1c79d84640 1 Processor -- start 2024-04-25T11:45:22.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.053+0000 7f1c79d84640 1 -- start start 2024-04-25T11:45:22.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.053+0000 7f1c79d84640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c74104c60 0x7f1c741a0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:22.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.053+0000 7f1c79d84640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c7410b6c0 0x7f1c741a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:22.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.053+0000 7f1c79d84640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c7410e0b0 0x7f1c7419a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:22.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.053+0000 7f1c79d84640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1c74113d30 con 0x7f1c74104c60 2024-04-25T11:45:22.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.054+0000 7f1c79d84640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1c74113bb0 con 0x7f1c7410b6c0 2024-04-25T11:45:22.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.054+0000 7f1c79d84640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1c74113eb0 con 0x7f1c7410e0b0 2024-04-25T11:45:22.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.054+0000 7f1c737fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c74104c60 0x7f1c741a0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:22.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.054+0000 7f1c737fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c74104c60 0x7f1c741a0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42184/0 (socket says 172.21.15.90:42184) 2024-04-25T11:45:22.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.054+0000 7f1c737fe640 1 -- 172.21.15.90:0/2565176333 learned_addr learned my addr 172.21.15.90:0/2565176333 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:22.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.054+0000 7f1c72ffd640 1 --2- 172.21.15.90:0/2565176333 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c7410b6c0 0x7f1c741a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:22.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.054+0000 7f1c737fe640 1 -- 172.21.15.90:0/2565176333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c7410e0b0 msgr2=0x7f1c7419a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:22.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.054+0000 7f1c737fe640 1 --2- 172.21.15.90:0/2565176333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c7410e0b0 0x7f1c7419a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.054+0000 7f1c737fe640 1 -- 172.21.15.90:0/2565176333 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c7410b6c0 msgr2=0x7f1c741a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:22.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.054+0000 7f1c737fe640 1 --2- 172.21.15.90:0/2565176333 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c7410b6c0 0x7f1c741a0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.054+0000 7f1c737fe640 1 -- 172.21.15.90:0/2565176333 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c7419acf0 con 0x7f1c74104c60 2024-04-25T11:45:22.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.054+0000 7f1c737fe640 1 --2- 172.21.15.90:0/2565176333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c74104c60 0x7f1c741a0530 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f1c64031a90 tx=0x7f1c64004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:22.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.054+0000 7f1c70ff9640 1 -- 172.21.15.90:0/2565176333 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c6403f030 con 0x7f1c74104c60 2024-04-25T11:45:22.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.054+0000 7f1c70ff9640 1 -- 172.21.15.90:0/2565176333 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c64003cb0 con 0x7f1c74104c60 2024-04-25T11:45:22.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.054+0000 7f1c79d84640 1 -- 172.21.15.90:0/2565176333 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c7419af80 con 0x7f1c74104c60 2024-04-25T11:45:22.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.054+0000 7f1c70ff9640 1 -- 172.21.15.90:0/2565176333 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c64035070 con 0x7f1c74104c60 2024-04-25T11:45:22.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.054+0000 7f1c79d84640 1 -- 172.21.15.90:0/2565176333 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c7419b470 con 0x7f1c74104c60 2024-04-25T11:45:22.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.056+0000 7f1c70ff9640 1 -- 172.21.15.90:0/2565176333 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f1c64042030 con 0x7f1c74104c60 2024-04-25T11:45:22.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.057+0000 7f1c80c38740 1 Processor -- start 2024-04-25T11:45:22.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.057+0000 7f1c80c38740 1 -- start start 2024-04-25T11:45:22.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.057+0000 7f1c80c38740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a7d390 0x55ad56e1e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:22.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.057+0000 7f1c80c38740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad56a544d0 0x55ad56e1ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:22.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.057+0000 7f1c80c38740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad56a5c6b0 0x55ad56e2fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:22.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.057+0000 7f1c80c38740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ad56d83d80 con 0x55ad56a7d390 2024-04-25T11:45:22.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.057+0000 7f1c80c38740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ad56d3da20 con 0x55ad56a5c6b0 2024-04-25T11:45:22.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.057+0000 7f1c80c38740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ad56d7fd40 con 0x55ad56a544d0 2024-04-25T11:45:22.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.057+0000 7f1c73fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a7d390 0x55ad56e1e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:22.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.057+0000 7f1c73fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a7d390 0x55ad56e1e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42190/0 (socket says 172.21.15.90:42190) 2024-04-25T11:45:22.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.057+0000 7f1c73fff640 1 -- 172.21.15.90:0/834990475 learned_addr learned my addr 172.21.15.90:0/834990475 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:22.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.057+0000 7f1c737fe640 1 --2- 172.21.15.90:0/834990475 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad56a544d0 0x55ad56e1ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:22.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.057+0000 7f1c70ff9640 1 --2- 172.21.15.90:0/2565176333 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1c5c0afc60 0x7f1c5c0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:22.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.057+0000 7f1c72ffd640 1 --2- 172.21.15.90:0/834990475 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad56a5c6b0 0x55ad56e2fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:22.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.057+0000 7f1c73fff640 1 -- 172.21.15.90:0/834990475 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad56a544d0 msgr2=0x55ad56e1ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:22.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.057+0000 7f1c73fff640 1 --2- 172.21.15.90:0/834990475 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad56a544d0 0x55ad56e1ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.057+0000 7f1c73fff640 1 -- 172.21.15.90:0/834990475 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad56a5c6b0 msgr2=0x55ad56e2fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:22.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.057+0000 7f1c73fff640 1 --2- 172.21.15.90:0/834990475 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad56a5c6b0 0x55ad56e2fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.057+0000 7f1c73fff640 1 -- 172.21.15.90:0/834990475 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad56d48c30 con 0x55ad56a7d390 2024-04-25T11:45:22.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.057+0000 7f1c737fe640 1 --2- 172.21.15.90:0/834990475 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad56a544d0 0x55ad56e1ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:22.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.057+0000 7f1c73fff640 1 --2- 172.21.15.90:0/2565176333 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1c5c0afc60 0x7f1c5c0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:22.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.058+0000 7f1c70ff9640 1 -- 172.21.15.90:0/2565176333 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7f1c640ff740 con 0x7f1c74104c60 2024-04-25T11:45:22.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.058+0000 7f1c73fff640 1 --2- 172.21.15.90:0/834990475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a7d390 0x55ad56e1e6a0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c007e70 tx=0x7f1c6c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef161e2c6dd945be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:22.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.058+0000 7f1c59ffb640 1 -- 172.21.15.90:0/834990475 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c016070 con 0x55ad56a7d390 2024-04-25T11:45:22.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.058+0000 7f1c59ffb640 1 -- 172.21.15.90:0/834990475 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c6c004710 con 0x55ad56a7d390 2024-04-25T11:45:22.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.058+0000 7f1c59ffb640 1 -- 172.21.15.90:0/834990475 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c00f6a0 con 0x55ad56a7d390 2024-04-25T11:45:22.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.058+0000 7f1c73fff640 1 --2- 172.21.15.90:0/2565176333 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1c5c0afc60 0x7f1c5c0b2170 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c01d000 tx=0x7f1c6c01e000 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:22.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.059+0000 7f1c80c38740 1 -- 172.21.15.90:0/834990475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a7d390 msgr2=0x55ad56e1e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:22.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.059+0000 7f1c80c38740 1 --2- 172.21.15.90:0/834990475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a7d390 0x55ad56e1e6a0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c007e70 tx=0x7f1c6c00c6a0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.059+0000 7f1c80c38740 1 -- 172.21.15.90:0/834990475 shutdown_connections 2024-04-25T11:45:22.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.059+0000 7f1c80c38740 1 --2- 172.21.15.90:0/834990475 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad56a5c6b0 0x55ad56e2fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.059+0000 7f1c80c38740 1 --2- 172.21.15.90:0/834990475 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad56a544d0 0x55ad56e1ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.059+0000 7f1c80c38740 1 --2- 172.21.15.90:0/834990475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a7d390 0x55ad56e1e6a0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.059+0000 7f1c80c38740 1 -- 172.21.15.90:0/834990475 >> 172.21.15.90:0/834990475 conn(0x55ad56a714e0 msgr2=0x55ad56e1afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:22.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.059+0000 7f1c80c38740 1 -- 172.21.15.90:0/834990475 shutdown_connections 2024-04-25T11:45:22.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.059+0000 7f1c80c38740 1 -- 172.21.15.90:0/834990475 wait complete. 2024-04-25T11:45:22.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.059+0000 7f1c80c38740 1 -- start start 2024-04-25T11:45:22.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.059+0000 7f1c80c38740 1 Processor -- start 2024-04-25T11:45:22.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.060+0000 7f1c80c38740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad56a544d0 0x55ad56e2d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:22.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.060+0000 7f1c80c38740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a5c6b0 0x55ad56e2dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:22.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.060+0000 7f1c80c38740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad56a7d390 0x55ad56e396e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:22.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.060+0000 7f1c80c38740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ad56e301a0 con 0x55ad56a5c6b0 2024-04-25T11:45:22.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.060+0000 7f1c80c38740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ad56d0adc0 con 0x55ad56a7d390 2024-04-25T11:45:22.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.060+0000 7f1c80c38740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ad56e30320 con 0x55ad56a544d0 2024-04-25T11:45:22.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.060+0000 7f1c80c38740 10 client.0 _ll_register_callbacks cb 0x55ad56d26600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:22.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.060+0000 7f1c72ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a5c6b0 0x55ad56e2dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:22.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.060+0000 7f1c72ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a5c6b0 0x55ad56e2dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42206/0 (socket says 172.21.15.90:42206) 2024-04-25T11:45:22.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.060+0000 7f1c72ffd640 1 -- 172.21.15.90:0/2440730102 learned_addr learned my addr 172.21.15.90:0/2440730102 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:22.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.060+0000 7f1c73fff640 1 --2- 172.21.15.90:0/2440730102 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad56a7d390 0x55ad56e396e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:22.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.060+0000 7f1c737fe640 1 --2- 172.21.15.90:0/2440730102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad56a544d0 0x55ad56e2d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:22.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.060+0000 7f1c72ffd640 1 -- 172.21.15.90:0/2440730102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad56a544d0 msgr2=0x55ad56e2d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:22.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.060+0000 7f1c72ffd640 1 --2- 172.21.15.90:0/2440730102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad56a544d0 0x55ad56e2d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.060+0000 7f1c72ffd640 1 -- 172.21.15.90:0/2440730102 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad56a7d390 msgr2=0x55ad56e396e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:22.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.060+0000 7f1c72ffd640 1 --2- 172.21.15.90:0/2440730102 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad56a7d390 0x55ad56e396e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.060+0000 7f1c72ffd640 1 -- 172.21.15.90:0/2440730102 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad56a953f0 con 0x55ad56a5c6b0 2024-04-25T11:45:22.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.060+0000 7f1c737fe640 1 --2- 172.21.15.90:0/2440730102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad56a544d0 0x55ad56e2d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:22.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.060+0000 7f1c72ffd640 1 --2- 172.21.15.90:0/2440730102 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a5c6b0 0x55ad56e2dbf0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f1c6800bef0 tx=0x7f1c6800d590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a435db7206df7a95 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:22.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.060+0000 7f1c52ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:22.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.060+0000 7f1c52ffd640 1 -- 172.21.15.90:0/2440730102 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c68016070 con 0x55ad56a5c6b0 2024-04-25T11:45:22.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.061+0000 7f1c52ffd640 1 -- 172.21.15.90:0/2440730102 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7f1c68012660 con 0x55ad56a5c6b0 2024-04-25T11:45:22.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.061+0000 7f1c80c38740 1 -- 172.21.15.90:0/2440730102 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c6c004710 con 0x55ad56a5c6b0 2024-04-25T11:45:22.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.061+0000 7f1c52ffd640 1 -- 172.21.15.90:0/2440730102 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1c68022050 con 0x55ad56a5c6b0 2024-04-25T11:45:22.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.061+0000 7f1c80c38740 10 client.18906 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:22.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.061+0000 7f1c80c38740 1 -- 172.21.15.90:0/2440730102 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ad56b5f7d0 con 0x55ad56a5c6b0 2024-04-25T11:45:22.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.061+0000 7f1c52ffd640 1 -- 172.21.15.90:0/2440730102 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c6c004710 con 0x55ad56a5c6b0 2024-04-25T11:45:22.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.061+0000 7f1c52ffd640 1 -- 172.21.15.90:0/2440730102 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f1c68011a00 con 0x55ad56a5c6b0 2024-04-25T11:45:22.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.061+0000 7f1c80c38740 10 client.18906 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:22.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.061+0000 7f1c80c38740 10 client.18906 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:45:22.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.061+0000 7f1c80c38740 20 client.18906 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:22.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.061+0000 7f1c80c38740 1 --2- 172.21.15.90:0/2440730102 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ad56a3bf60 0x55ad56e3fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:22.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.061+0000 7f1c80c38740 4 client.18906 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:45:22.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.061+0000 7f1c80c38740 1 -- 172.21.15.90:0/2440730102 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad56d91990 con 0x55ad56a3bf60 2024-04-25T11:45:22.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.061+0000 7f1c737fe640 1 --2- 172.21.15.90:0/2440730102 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ad56a3bf60 0x55ad56e3fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:22.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.062+0000 7f1c737fe640 1 --2- 172.21.15.90:0/2440730102 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ad56a3bf60 0x55ad56e3fb70 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bfd32d7e835fb45 server_cookie=1f7611228315b1 in_seq=0 out_seq=0 2024-04-25T11:45:22.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.062+0000 7f1c52ffd640 10 client.18906 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:22.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.062+0000 7f1c52ffd640 1 -- 172.21.15.90:0/2440730102 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ad56d91990 con 0x55ad56a3bf60 2024-04-25T11:45:22.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.063+0000 7f1c52ffd640 10 client.18906 handle_command_reply: tid=0 2024-04-25T11:45:22.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.063+0000 7f1c80c38740 1 client.18906 shutdown 2024-04-25T11:45:22.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.063+0000 7f1c80c38740 1 -- 172.21.15.90:0/2440730102 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a5c6b0 msgr2=0x55ad56e2dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:22.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.063+0000 7f1c80c38740 1 --2- 172.21.15.90:0/2440730102 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a5c6b0 0x55ad56e2dbf0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f1c6800bef0 tx=0x7f1c6800d590 comp rx=0 tx=0).stop 2024-04-25T11:45:22.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.063+0000 7f1c80c38740 1 -- 172.21.15.90:0/2440730102 shutdown_connections 2024-04-25T11:45:22.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.063+0000 7f1c80c38740 1 --2- 172.21.15.90:0/2440730102 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ad56a3bf60 0x55ad56e3fb70 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.063+0000 7f1c80c38740 1 --2- 172.21.15.90:0/2440730102 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad56a7d390 0x55ad56e396e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.063+0000 7f1c80c38740 1 --2- 172.21.15.90:0/2440730102 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a5c6b0 0x55ad56e2dbf0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.063+0000 7f1c80c38740 1 --2- 172.21.15.90:0/2440730102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad56a544d0 0x55ad56e2d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.063+0000 7f1c80c38740 1 -- 172.21.15.90:0/2440730102 >> 172.21.15.90:0/2440730102 conn(0x55ad56a714e0 msgr2=0x55ad56e20480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:22.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.063+0000 7f1c52ffd640 0 client.18906 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:22.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.063+0000 7f1c80c38740 1 -- 172.21.15.90:0/2440730102 shutdown_connections 2024-04-25T11:45:22.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.063+0000 7f1c80c38740 1 -- 172.21.15.90:0/2440730102 wait complete. 2024-04-25T11:45:22.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.063+0000 7f1c80c38740 20 client.18906 trim_cache size 0 max 16384 2024-04-25T11:45:22.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.076+0000 7f1c80c38740 1 Processor -- start 2024-04-25T11:45:22.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.077+0000 7f1c80c38740 1 -- start start 2024-04-25T11:45:22.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.077+0000 7f1c80c38740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad56a544d0 0x55ad56e392c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:22.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.077+0000 7f1c80c38740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a5c6b0 0x55ad56e2bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:22.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.077+0000 7f1c80c38740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad56a7d390 0x55ad56e2c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:22.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.077+0000 7f1c80c38740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1c741adb30 con 0x55ad56a5c6b0 2024-04-25T11:45:22.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.077+0000 7f1c80c38740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1c741af670 con 0x55ad56a544d0 2024-04-25T11:45:22.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.077+0000 7f1c80c38740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1c741ae200 con 0x55ad56a7d390 2024-04-25T11:45:22.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.077+0000 7f1c72ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a5c6b0 0x55ad56e2bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:22.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.077+0000 7f1c72ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a5c6b0 0x55ad56e2bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42220/0 (socket says 172.21.15.90:42220) 2024-04-25T11:45:22.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.077+0000 7f1c72ffd640 1 -- 172.21.15.90:0/3693142386 learned_addr learned my addr 172.21.15.90:0/3693142386 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:22.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.077+0000 7f1c737fe640 1 --2- 172.21.15.90:0/3693142386 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad56a544d0 0x55ad56e392c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:22.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.077+0000 7f1c72ffd640 1 -- 172.21.15.90:0/3693142386 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad56a7d390 msgr2=0x55ad56e2c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:22.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.077+0000 7f1c72ffd640 1 --2- 172.21.15.90:0/3693142386 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad56a7d390 0x55ad56e2c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.077+0000 7f1c72ffd640 1 -- 172.21.15.90:0/3693142386 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad56a544d0 msgr2=0x55ad56e392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:22.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.077+0000 7f1c72ffd640 1 --2- 172.21.15.90:0/3693142386 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad56a544d0 0x55ad56e392c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.077+0000 7f1c72ffd640 1 -- 172.21.15.90:0/3693142386 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad56e2e3c0 con 0x55ad56a5c6b0 2024-04-25T11:45:22.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.077+0000 7f1c737fe640 1 --2- 172.21.15.90:0/3693142386 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad56a544d0 0x55ad56e392c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:22.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.077+0000 7f1c72ffd640 1 --2- 172.21.15.90:0/3693142386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a5c6b0 0x55ad56e2bd70 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f1c6800bec0 tx=0x7f1c6800bae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=362ee9b2ab26e19f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:22.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.077+0000 7f1c597fa640 1 -- 172.21.15.90:0/3693142386 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c6802a070 con 0x55ad56a5c6b0 2024-04-25T11:45:22.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.077+0000 7f1c597fa640 1 -- 172.21.15.90:0/3693142386 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c6801a600 con 0x55ad56a5c6b0 2024-04-25T11:45:22.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.077+0000 7f1c597fa640 1 -- 172.21.15.90:0/3693142386 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c68016040 con 0x55ad56a5c6b0 2024-04-25T11:45:22.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.078+0000 7f1c80c38740 1 -- 172.21.15.90:0/3693142386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a5c6b0 msgr2=0x55ad56e2bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:22.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.078+0000 7f1c80c38740 1 --2- 172.21.15.90:0/3693142386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a5c6b0 0x55ad56e2bd70 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f1c6800bec0 tx=0x7f1c6800bae0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.078+0000 7f1c80c38740 1 -- 172.21.15.90:0/3693142386 shutdown_connections 2024-04-25T11:45:22.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.078+0000 7f1c80c38740 1 --2- 172.21.15.90:0/3693142386 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad56a7d390 0x55ad56e2c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.078+0000 7f1c80c38740 1 --2- 172.21.15.90:0/3693142386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a5c6b0 0x55ad56e2bd70 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.078+0000 7f1c80c38740 1 --2- 172.21.15.90:0/3693142386 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad56a544d0 0x55ad56e392c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.078+0000 7f1c80c38740 1 -- 172.21.15.90:0/3693142386 >> 172.21.15.90:0/3693142386 conn(0x55ad56a714e0 msgr2=0x55ad56e28630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:22.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.078+0000 7f1c80c38740 1 -- 172.21.15.90:0/3693142386 shutdown_connections 2024-04-25T11:45:22.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.078+0000 7f1c80c38740 1 -- 172.21.15.90:0/3693142386 wait complete. 2024-04-25T11:45:22.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.078+0000 7f1c80c38740 1 -- start start 2024-04-25T11:45:22.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.079+0000 7f1c80c38740 1 Processor -- start 2024-04-25T11:45:22.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.079+0000 7f1c80c38740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad56a544d0 0x55ad56e309f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:22.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.079+0000 7f1c80c38740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad56a5c6b0 0x55ad56e30f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:22.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.079+0000 7f1c80c38740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a7d390 0x55ad56ea5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:22.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.079+0000 7f1c80c38740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1c741aefa0 con 0x55ad56a7d390 2024-04-25T11:45:22.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.079+0000 7f1c80c38740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1c741ae8d0 con 0x55ad56a5c6b0 2024-04-25T11:45:22.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.079+0000 7f1c80c38740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1c741abb30 con 0x55ad56a544d0 2024-04-25T11:45:22.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.079+0000 7f1c80c38740 10 client.0 _ll_register_callbacks cb 0x7f1c68006e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:22.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.079+0000 7f1c73fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a7d390 0x55ad56ea5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:22.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.079+0000 7f1c73fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a7d390 0x55ad56ea5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42224/0 (socket says 172.21.15.90:42224) 2024-04-25T11:45:22.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.079+0000 7f1c73fff640 1 -- 172.21.15.90:0/994805662 learned_addr learned my addr 172.21.15.90:0/994805662 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:22.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.079+0000 7f1c72ffd640 1 --2- 172.21.15.90:0/994805662 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad56a5c6b0 0x55ad56e30f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:22.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.079+0000 7f1c737fe640 1 --2- 172.21.15.90:0/994805662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad56a544d0 0x55ad56e309f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:22.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.079+0000 7f1c73fff640 1 -- 172.21.15.90:0/994805662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad56a544d0 msgr2=0x55ad56e309f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:22.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.079+0000 7f1c73fff640 1 --2- 172.21.15.90:0/994805662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad56a544d0 0x55ad56e309f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.079+0000 7f1c73fff640 1 -- 172.21.15.90:0/994805662 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad56a5c6b0 msgr2=0x55ad56e30f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:22.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.079+0000 7f1c73fff640 1 --2- 172.21.15.90:0/994805662 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad56a5c6b0 0x55ad56e30f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.079+0000 7f1c73fff640 1 -- 172.21.15.90:0/994805662 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad56e152b0 con 0x55ad56a7d390 2024-04-25T11:45:22.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.079+0000 7f1c72ffd640 1 --2- 172.21.15.90:0/994805662 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad56a5c6b0 0x55ad56e30f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:22.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.080+0000 7f1c73fff640 1 --2- 172.21.15.90:0/994805662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a7d390 0x55ad56ea5ed0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c004960 tx=0x7f1c6c005e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa342d9e0670dc74 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:22.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.080+0000 7f1c52ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:22.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.080+0000 7f1c52ffd640 1 -- 172.21.15.90:0/994805662 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c016030 con 0x55ad56a7d390 2024-04-25T11:45:22.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.080+0000 7f1c52ffd640 1 -- 172.21.15.90:0/994805662 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7f1c6c011e00 con 0x55ad56a7d390 2024-04-25T11:45:22.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.080+0000 7f1c80c38740 1 -- 172.21.15.90:0/994805662 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c6801a600 con 0x55ad56a7d390 2024-04-25T11:45:22.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.080+0000 7f1c52ffd640 1 -- 172.21.15.90:0/994805662 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1c6c007200 con 0x55ad56a7d390 2024-04-25T11:45:22.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.080+0000 7f1c80c38740 10 client.18912 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:22.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.080+0000 7f1c80c38740 1 -- 172.21.15.90:0/994805662 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ad56ea73d0 con 0x55ad56a7d390 2024-04-25T11:45:22.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.080+0000 7f1c52ffd640 1 -- 172.21.15.90:0/994805662 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c6801a600 con 0x55ad56a7d390 2024-04-25T11:45:22.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.080+0000 7f1c52ffd640 1 -- 172.21.15.90:0/994805662 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f1c6c01a070 con 0x55ad56a7d390 2024-04-25T11:45:22.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.080+0000 7f1c80c38740 10 client.18912 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:22.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.080+0000 7f1c80c38740 10 client.18912 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:45:22.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.080+0000 7f1c80c38740 20 client.18912 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:22.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.080+0000 7f1c80c38740 1 --2- 172.21.15.90:0/994805662 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ad56a113a0 0x55ad56ea9820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:22.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.080+0000 7f1c80c38740 4 client.18912 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:45:22.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.080+0000 7f1c80c38740 1 -- 172.21.15.90:0/994805662 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ad56d8b800 con 0x55ad56a113a0 2024-04-25T11:45:22.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.081+0000 7f1c737fe640 1 --2- 172.21.15.90:0/994805662 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ad56a113a0 0x55ad56ea9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:22.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.081+0000 7f1c52ffd640 10 client.18912 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:22.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.081+0000 7f1c737fe640 1 --2- 172.21.15.90:0/994805662 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ad56a113a0 0x55ad56ea9820 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de3ce10de4c6ac7f server_cookie=34991c868ddbcba1 in_seq=0 out_seq=0 2024-04-25T11:45:22.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.081+0000 7f1c52ffd640 1 -- 172.21.15.90:0/994805662 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ad56d8b800 con 0x55ad56a113a0 2024-04-25T11:45:22.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.081+0000 7f1c52ffd640 10 client.18912 handle_command_reply: tid=0 2024-04-25T11:45:22.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.081+0000 7f1c80c38740 1 client.18912 shutdown 2024-04-25T11:45:22.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.081+0000 7f1c80c38740 1 -- 172.21.15.90:0/994805662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a7d390 msgr2=0x55ad56ea5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:22.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.081+0000 7f1c80c38740 1 --2- 172.21.15.90:0/994805662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a7d390 0x55ad56ea5ed0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c004960 tx=0x7f1c6c005e90 comp rx=0 tx=0).stop 2024-04-25T11:45:22.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.081+0000 7f1c80c38740 1 -- 172.21.15.90:0/994805662 shutdown_connections 2024-04-25T11:45:22.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.081+0000 7f1c80c38740 1 --2- 172.21.15.90:0/994805662 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ad56a113a0 0x55ad56ea9820 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.081+0000 7f1c80c38740 1 --2- 172.21.15.90:0/994805662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad56a7d390 0x55ad56ea5ed0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.081+0000 7f1c52ffd640 0 client.18912 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:22.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.081+0000 7f1c80c38740 1 --2- 172.21.15.90:0/994805662 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad56a5c6b0 0x55ad56e30f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.081+0000 7f1c80c38740 1 --2- 172.21.15.90:0/994805662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad56a544d0 0x55ad56e309f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.081+0000 7f1c80c38740 1 -- 172.21.15.90:0/994805662 >> 172.21.15.90:0/994805662 conn(0x55ad56a714e0 msgr2=0x55ad56e28630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:22.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.081+0000 7f1c80c38740 1 -- 172.21.15.90:0/994805662 shutdown_connections 2024-04-25T11:45:22.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.081+0000 7f1c80c38740 1 -- 172.21.15.90:0/994805662 wait complete. 2024-04-25T11:45:22.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.082+0000 7f1c80c38740 20 client.18912 trim_cache size 0 max 16384 2024-04-25T11:45:22.075 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:45:22.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.082+0000 7f1c79d84640 1 -- 172.21.15.90:0/2565176333 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1c5c0afc60 msgr2=0x7f1c5c0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:22.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.082+0000 7f1c79d84640 1 --2- 172.21.15.90:0/2565176333 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1c5c0afc60 0x7f1c5c0b2170 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c01d000 tx=0x7f1c6c01e000 comp rx=0 tx=0).stop 2024-04-25T11:45:22.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.082+0000 7f1c79d84640 1 -- 172.21.15.90:0/2565176333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c74104c60 msgr2=0x7f1c741a0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:22.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.082+0000 7f1c79d84640 1 --2- 172.21.15.90:0/2565176333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c74104c60 0x7f1c741a0530 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f1c64031a90 tx=0x7f1c64004270 comp rx=0 tx=0).stop 2024-04-25T11:45:22.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.082+0000 7f1c79d84640 1 -- 172.21.15.90:0/2565176333 shutdown_connections 2024-04-25T11:45:22.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.082+0000 7f1c79d84640 1 --2- 172.21.15.90:0/2565176333 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1c5c0afc60 0x7f1c5c0b2170 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.082+0000 7f1c79d84640 1 --2- 172.21.15.90:0/2565176333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c7410e0b0 0x7f1c7419a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.082+0000 7f1c79d84640 1 --2- 172.21.15.90:0/2565176333 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c7410b6c0 0x7f1c741a0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.082+0000 7f1c79d84640 1 --2- 172.21.15.90:0/2565176333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c74104c60 0x7f1c741a0530 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:22.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.082+0000 7f1c79d84640 1 -- 172.21.15.90:0/2565176333 >> 172.21.15.90:0/2565176333 conn(0x7f1c740fe260 msgr2=0x7f1c7410e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:22.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.082+0000 7f1c79d84640 1 -- 172.21.15.90:0/2565176333 shutdown_connections 2024-04-25T11:45:22.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:22.082+0000 7f1c79d84640 1 -- 172.21.15.90:0/2565176333 wait complete. 2024-04-25T11:45:22.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:22 smithi090 ceph-mon[29575]: pgmap v318: 97 pgs: 7 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 74 active+clean; 31 GiB data, 98 GiB used, 975 GiB / 1.0 TiB avail; 11 KiB/s rd, 2.8 KiB/s wr, 13 op/s 2024-04-25T11:45:22.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:22 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2710997881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:45:22.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:22 smithi138 ceph-mon[32328]: pgmap v318: 97 pgs: 7 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 74 active+clean; 31 GiB data, 98 GiB used, 975 GiB / 1.0 TiB avail; 11 KiB/s rd, 2.8 KiB/s wr, 13 op/s 2024-04-25T11:45:22.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:22 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2710997881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:45:22.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:22 smithi156 ceph-mon[33045]: pgmap v318: 97 pgs: 7 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 74 active+clean; 31 GiB data, 98 GiB used, 975 GiB / 1.0 TiB avail; 11 KiB/s rd, 2.8 KiB/s wr, 13 op/s 2024-04-25T11:45:22.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:22 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2710997881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:45:23.079 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T11:45:23.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:45:23.080 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T11:45:23.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.220+0000 7fda651ee640 1 -- 172.21.15.90:0/1847161526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda60103170 msgr2=0x7fda6010f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:23.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.220+0000 7fda651ee640 1 --2- 172.21.15.90:0/1847161526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda60103170 0x7fda6010f4a0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7fda58009ca0 tx=0x7fda5802a8c0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.220+0000 7fda651ee640 1 -- 172.21.15.90:0/1847161526 shutdown_connections 2024-04-25T11:45:23.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.220+0000 7fda651ee640 1 --2- 172.21.15.90:0/1847161526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda60103170 0x7fda6010f4a0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.220+0000 7fda651ee640 1 --2- 172.21.15.90:0/1847161526 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fda60108eb0 0x7fda60102b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.220+0000 7fda651ee640 1 --2- 172.21.15.90:0/1847161526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda60108a10 0x7fda601025c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.220+0000 7fda651ee640 1 -- 172.21.15.90:0/1847161526 >> 172.21.15.90:0/1847161526 conn(0x7fda60078150 msgr2=0x7fda60078590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:23.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.220+0000 7fda651ee640 1 -- 172.21.15.90:0/1847161526 shutdown_connections 2024-04-25T11:45:23.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.220+0000 7fda651ee640 1 -- 172.21.15.90:0/1847161526 wait complete. 2024-04-25T11:45:23.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.221+0000 7fda651ee640 1 Processor -- start 2024-04-25T11:45:23.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.221+0000 7fda651ee640 1 -- start start 2024-04-25T11:45:23.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.221+0000 7fda651ee640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fda60103170 0x7fda6019a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.221+0000 7fda651ee640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda60108a10 0x7fda6019ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.221+0000 7fda651ee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda60108eb0 0x7fda6019efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.221+0000 7fda651ee640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fda60112c30 con 0x7fda60108a10 2024-04-25T11:45:23.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.221+0000 7fda651ee640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fda60112ab0 con 0x7fda60103170 2024-04-25T11:45:23.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.221+0000 7fda651ee640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fda60112db0 con 0x7fda60108eb0 2024-04-25T11:45:23.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.222+0000 7fda5e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda60108a10 0x7fda6019ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.222+0000 7fda5e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda60108a10 0x7fda6019ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42242/0 (socket says 172.21.15.90:42242) 2024-04-25T11:45:23.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.222+0000 7fda5e59c640 1 -- 172.21.15.90:0/2726355539 learned_addr learned my addr 172.21.15.90:0/2726355539 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:23.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.222+0000 7fda5ed9d640 1 --2- 172.21.15.90:0/2726355539 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fda60103170 0x7fda6019a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.222+0000 7fda5f59e640 1 --2- 172.21.15.90:0/2726355539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda60108eb0 0x7fda6019efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.222+0000 7fda5e59c640 1 -- 172.21.15.90:0/2726355539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda60108eb0 msgr2=0x7fda6019efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:23.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.222+0000 7fda5e59c640 1 --2- 172.21.15.90:0/2726355539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda60108eb0 0x7fda6019efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.222+0000 7fda5e59c640 1 -- 172.21.15.90:0/2726355539 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fda60103170 msgr2=0x7fda6019a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:23.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.222+0000 7fda5e59c640 1 --2- 172.21.15.90:0/2726355539 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fda60103170 0x7fda6019a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.222+0000 7fda5e59c640 1 -- 172.21.15.90:0/2726355539 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda6019f560 con 0x7fda60108a10 2024-04-25T11:45:23.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.222+0000 7fda5e59c640 1 --2- 172.21.15.90:0/2726355539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda60108a10 0x7fda6019ace0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fda5400c9a0 tx=0x7fda5400cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:23.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.222+0000 7fda47fff640 1 -- 172.21.15.90:0/2726355539 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda54004220 con 0x7fda60108a10 2024-04-25T11:45:23.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.222+0000 7fda651ee640 1 -- 172.21.15.90:0/2726355539 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda6019f850 con 0x7fda60108a10 2024-04-25T11:45:23.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.222+0000 7fda47fff640 1 -- 172.21.15.90:0/2726355539 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda540043c0 con 0x7fda60108a10 2024-04-25T11:45:23.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.222+0000 7fda47fff640 1 -- 172.21.15.90:0/2726355539 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda54003df0 con 0x7fda60108a10 2024-04-25T11:45:23.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.222+0000 7fda651ee640 1 -- 172.21.15.90:0/2726355539 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda6019fda0 con 0x7fda60108a10 2024-04-25T11:45:23.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.225+0000 7fda47fff640 1 -- 172.21.15.90:0/2726355539 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fda540079b0 con 0x7fda60108a10 2024-04-25T11:45:23.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.225+0000 7fda6c177740 1 Processor -- start 2024-04-25T11:45:23.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.225+0000 7fda6c177740 1 -- start start 2024-04-25T11:45:23.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.225+0000 7fda6c177740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e702410 0x56124eaa2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.225+0000 7fda6c177740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56124e6d9550 0x56124eaa2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.225+0000 7fda6c177740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56124e6e1730 0x56124eaa2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.225+0000 7fda6c177740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56124ea08b00 con 0x56124e702410 2024-04-25T11:45:23.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.225+0000 7fda6c177740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56124e9a4920 con 0x56124e6d9550 2024-04-25T11:45:23.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.225+0000 7fda6c177740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56124ea04ac0 con 0x56124e6e1730 2024-04-25T11:45:23.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.225+0000 7fda5f59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e702410 0x56124eaa2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.225+0000 7fda5f59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e702410 0x56124eaa2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42256/0 (socket says 172.21.15.90:42256) 2024-04-25T11:45:23.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.225+0000 7fda5f59e640 1 -- 172.21.15.90:0/2899337299 learned_addr learned my addr 172.21.15.90:0/2899337299 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:23.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.225+0000 7fda5ed9d640 1 --2- 172.21.15.90:0/2899337299 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56124e6d9550 0x56124eaa2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.226+0000 7fda5e59c640 1 --2- 172.21.15.90:0/2899337299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56124e6e1730 0x56124eaa2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.226+0000 7fda5f59e640 1 -- 172.21.15.90:0/2899337299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56124e6e1730 msgr2=0x56124eaa2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:23.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.226+0000 7fda5f59e640 1 --2- 172.21.15.90:0/2899337299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56124e6e1730 0x56124eaa2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.226+0000 7fda5f59e640 1 -- 172.21.15.90:0/2899337299 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56124e6d9550 msgr2=0x56124eaa2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:23.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.226+0000 7fda5f59e640 1 --2- 172.21.15.90:0/2899337299 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56124e6d9550 0x56124eaa2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.226+0000 7fda5f59e640 1 -- 172.21.15.90:0/2899337299 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56124e7e27e0 con 0x56124e702410 2024-04-25T11:45:23.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.226+0000 7fda47fff640 1 --2- 172.21.15.90:0/2726355539 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fda480afc60 0x7fda480b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.226+0000 7fda5e59c640 1 --2- 172.21.15.90:0/2899337299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56124e6e1730 0x56124eaa2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:23.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.226+0000 7fda5e59c640 1 --2- 172.21.15.90:0/2726355539 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fda480afc60 0x7fda480b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.226+0000 7fda5f59e640 1 --2- 172.21.15.90:0/2899337299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e702410 0x56124eaa2510 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fda58009930 tx=0x7fda58009960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78616ca3a6c31722 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:23.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.226+0000 7fda47fff640 1 -- 172.21.15.90:0/2726355539 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7fda540dedb0 con 0x7fda60108a10 2024-04-25T11:45:23.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.226+0000 7fda457fa640 1 -- 172.21.15.90:0/2899337299 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda58031df0 con 0x56124e702410 2024-04-25T11:45:23.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.226+0000 7fda457fa640 1 -- 172.21.15.90:0/2899337299 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda58036040 con 0x56124e702410 2024-04-25T11:45:23.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.226+0000 7fda457fa640 1 -- 172.21.15.90:0/2899337299 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda580042d0 con 0x56124e702410 2024-04-25T11:45:23.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.227+0000 7fda6c177740 1 -- 172.21.15.90:0/2899337299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e702410 msgr2=0x56124eaa2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:23.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.227+0000 7fda5e59c640 1 --2- 172.21.15.90:0/2726355539 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fda480afc60 0x7fda480b2170 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fda540dc500 tx=0x7fda54004570 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:23.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.227+0000 7fda6c177740 1 --2- 172.21.15.90:0/2899337299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e702410 0x56124eaa2510 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fda58009930 tx=0x7fda58009960 comp rx=0 tx=0).stop 2024-04-25T11:45:23.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.227+0000 7fda6c177740 1 -- 172.21.15.90:0/2899337299 shutdown_connections 2024-04-25T11:45:23.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.227+0000 7fda6c177740 1 --2- 172.21.15.90:0/2899337299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56124e6e1730 0x56124eaa2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.227+0000 7fda6c177740 1 --2- 172.21.15.90:0/2899337299 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56124e6d9550 0x56124eaa2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.227+0000 7fda6c177740 1 --2- 172.21.15.90:0/2899337299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e702410 0x56124eaa2510 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.227+0000 7fda6c177740 1 -- 172.21.15.90:0/2899337299 >> 172.21.15.90:0/2899337299 conn(0x56124e6f6560 msgr2=0x56124ea9edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:23.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.227+0000 7fda6c177740 1 -- 172.21.15.90:0/2899337299 shutdown_connections 2024-04-25T11:45:23.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.227+0000 7fda6c177740 1 -- 172.21.15.90:0/2899337299 wait complete. 2024-04-25T11:45:23.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.227+0000 7fda6c177740 1 -- start start 2024-04-25T11:45:23.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.228+0000 7fda6c177740 1 Processor -- start 2024-04-25T11:45:23.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.228+0000 7fda6c177740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56124e6d9550 0x56124eab0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.228+0000 7fda6c177740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56124e6e1730 0x56124eab14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.228+0000 7fda6c177740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e702410 0x56124eabd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.228+0000 7fda6c177740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56124eab3d20 con 0x56124e702410 2024-04-25T11:45:23.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.228+0000 7fda6c177740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56124e9a25d0 con 0x56124e6e1730 2024-04-25T11:45:23.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.228+0000 7fda6c177740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56124eab3ea0 con 0x56124e6d9550 2024-04-25T11:45:23.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.228+0000 7fda6c177740 10 client.0 _ll_register_callbacks cb 0x56124e990080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:23.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.228+0000 7fda5f59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e702410 0x56124eabd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.228+0000 7fda5f59e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e702410 0x56124eabd200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42266/0 (socket says 172.21.15.90:42266) 2024-04-25T11:45:23.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.228+0000 7fda5f59e640 1 -- 172.21.15.90:0/970150959 learned_addr learned my addr 172.21.15.90:0/970150959 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:23.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.228+0000 7fda5ed9d640 1 --2- 172.21.15.90:0/970150959 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56124e6e1730 0x56124eab14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.228+0000 7fda5f59e640 1 --2- 172.21.15.90:0/970150959 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56124e6d9550 0x56124eab0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.228+0000 7fda5f59e640 1 -- 172.21.15.90:0/970150959 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56124e6d9550 msgr2=0x56124eab0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:23.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.228+0000 7fda5f59e640 1 --2- 172.21.15.90:0/970150959 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56124e6d9550 0x56124eab0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.228+0000 7fda5f59e640 1 -- 172.21.15.90:0/970150959 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56124e6e1730 msgr2=0x56124eab14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:23.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.228+0000 7fda5f59e640 1 --2- 172.21.15.90:0/970150959 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56124e6e1730 0x56124eab14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.228+0000 7fda5f59e640 1 -- 172.21.15.90:0/970150959 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56124e842f70 con 0x56124e702410 2024-04-25T11:45:23.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.228+0000 7fda5f59e640 1 --2- 172.21.15.90:0/970150959 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e702410 0x56124eabd200 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fda58009ca0 tx=0x7fda5803e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ee56d2ad52ef0c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:23.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.228+0000 7fda42ff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:23.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.229+0000 7fda42ff5640 1 -- 172.21.15.90:0/970150959 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda58058060 con 0x56124e702410 2024-04-25T11:45:23.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.229+0000 7fda42ff5640 1 -- 172.21.15.90:0/970150959 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7fda58031df0 con 0x56124e702410 2024-04-25T11:45:23.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.229+0000 7fda6c177740 1 -- 172.21.15.90:0/970150959 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda58036040 con 0x56124e702410 2024-04-25T11:45:23.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.229+0000 7fda42ff5640 1 -- 172.21.15.90:0/970150959 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fda58004440 con 0x56124e702410 2024-04-25T11:45:23.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.229+0000 7fda6c177740 10 client.18924 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:23.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.229+0000 7fda6c177740 1 -- 172.21.15.90:0/970150959 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56124e6da170 con 0x56124e702410 2024-04-25T11:45:23.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.229+0000 7fda42ff5640 1 -- 172.21.15.90:0/970150959 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda58036040 con 0x56124e702410 2024-04-25T11:45:23.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.229+0000 7fda42ff5640 1 -- 172.21.15.90:0/970150959 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fda58050070 con 0x56124e702410 2024-04-25T11:45:23.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.229+0000 7fda6c177740 10 client.18924 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:23.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.229+0000 7fda6c177740 10 client.18924 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:45:23.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.229+0000 7fda6c177740 20 client.18924 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:23.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.229+0000 7fda6c177740 1 --2- 172.21.15.90:0/970150959 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56124e6c11f0 0x56124eac3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.229+0000 7fda6c177740 4 client.18924 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:45:23.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.229+0000 7fda6c177740 1 -- 172.21.15.90:0/970150959 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56124ea16710 con 0x56124e6c11f0 2024-04-25T11:45:23.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.229+0000 7fda5ed9d640 1 --2- 172.21.15.90:0/970150959 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56124e6c11f0 0x56124eac3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.230+0000 7fda42ff5640 10 client.18924 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:23.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.230+0000 7fda5ed9d640 1 --2- 172.21.15.90:0/970150959 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56124e6c11f0 0x56124eac3110 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81d56ce4be6cb845 server_cookie=118f082e44887c9b in_seq=0 out_seq=0 2024-04-25T11:45:23.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.231+0000 7fda42ff5640 1 -- 172.21.15.90:0/970150959 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56124ea16710 con 0x56124e6c11f0 2024-04-25T11:45:23.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.231+0000 7fda42ff5640 10 client.18924 handle_command_reply: tid=0 2024-04-25T11:45:23.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.231+0000 7fda6c177740 1 client.18924 shutdown 2024-04-25T11:45:23.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.231+0000 7fda6c177740 1 -- 172.21.15.90:0/970150959 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e702410 msgr2=0x56124eabd200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:23.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.231+0000 7fda6c177740 1 --2- 172.21.15.90:0/970150959 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e702410 0x56124eabd200 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fda58009ca0 tx=0x7fda5803e040 comp rx=0 tx=0).stop 2024-04-25T11:45:23.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.231+0000 7fda6c177740 1 -- 172.21.15.90:0/970150959 shutdown_connections 2024-04-25T11:45:23.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.231+0000 7fda6c177740 1 --2- 172.21.15.90:0/970150959 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56124e6c11f0 0x56124eac3110 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.231+0000 7fda6c177740 1 --2- 172.21.15.90:0/970150959 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e702410 0x56124eabd200 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.231+0000 7fda6c177740 1 --2- 172.21.15.90:0/970150959 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56124e6e1730 0x56124eab14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.231+0000 7fda42ff5640 0 client.18924 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:23.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.231+0000 7fda6c177740 1 --2- 172.21.15.90:0/970150959 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56124e6d9550 0x56124eab0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.231+0000 7fda6c177740 1 -- 172.21.15.90:0/970150959 >> 172.21.15.90:0/970150959 conn(0x56124e6f6560 msgr2=0x56124eaa23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:23.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.231+0000 7fda6c177740 1 -- 172.21.15.90:0/970150959 shutdown_connections 2024-04-25T11:45:23.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.231+0000 7fda6c177740 1 -- 172.21.15.90:0/970150959 wait complete. 2024-04-25T11:45:23.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.231+0000 7fda6c177740 20 client.18924 trim_cache size 0 max 16384 2024-04-25T11:45:23.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.249+0000 7fda6c177740 1 Processor -- start 2024-04-25T11:45:23.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.249+0000 7fda6c177740 1 -- start start 2024-04-25T11:45:23.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.249+0000 7fda6c177740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56124e6d9550 0x56124eab00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.249+0000 7fda6c177740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e6e1730 0x56124ea9dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.249+0000 7fda6c177740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56124e702410 0x56124eabd170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.249+0000 7fda6c177740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fda601adb60 con 0x56124e6e1730 2024-04-25T11:45:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.249+0000 7fda6c177740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fda601af6a0 con 0x56124e702410 2024-04-25T11:45:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.249+0000 7fda6c177740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fda601ae230 con 0x56124e6d9550 2024-04-25T11:45:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.249+0000 7fda5ed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e6e1730 0x56124ea9dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.249+0000 7fda5ed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e6e1730 0x56124ea9dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42282/0 (socket says 172.21.15.90:42282) 2024-04-25T11:45:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.249+0000 7fda5ed9d640 1 -- 172.21.15.90:0/2649269356 learned_addr learned my addr 172.21.15.90:0/2649269356 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.249+0000 7fda5f59e640 1 --2- 172.21.15.90:0/2649269356 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56124e6d9550 0x56124eab00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.249+0000 7fda5f59e640 1 --2- 172.21.15.90:0/2649269356 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56124e702410 0x56124eabd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.249+0000 7fda5ed9d640 1 -- 172.21.15.90:0/2649269356 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56124e6d9550 msgr2=0x56124eab00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.249+0000 7fda5ed9d640 1 --2- 172.21.15.90:0/2649269356 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56124e6d9550 0x56124eab00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.249+0000 7fda5ed9d640 1 -- 172.21.15.90:0/2649269356 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56124e702410 msgr2=0x56124eabd170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.249+0000 7fda5ed9d640 1 --2- 172.21.15.90:0/2649269356 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56124e702410 0x56124eabd170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.249+0000 7fda5ed9d640 1 -- 172.21.15.90:0/2649269356 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56124eaaba90 con 0x56124e6e1730 2024-04-25T11:45:23.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.250+0000 7fda5ed9d640 1 --2- 172.21.15.90:0/2649269356 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e6e1730 0x56124ea9dbd0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fda50000c00 tx=0x7fda5000a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=641ed538b809cb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:23.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.250+0000 7fda44ff9640 1 -- 172.21.15.90:0/2649269356 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda50005220 con 0x56124e6e1730 2024-04-25T11:45:23.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.250+0000 7fda44ff9640 1 -- 172.21.15.90:0/2649269356 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda50024070 con 0x56124e6e1730 2024-04-25T11:45:23.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.250+0000 7fda44ff9640 1 -- 172.21.15.90:0/2649269356 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda50020470 con 0x56124e6e1730 2024-04-25T11:45:23.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.250+0000 7fda6c177740 1 -- 172.21.15.90:0/2649269356 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e6e1730 msgr2=0x56124ea9dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:23.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.250+0000 7fda6c177740 1 --2- 172.21.15.90:0/2649269356 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e6e1730 0x56124ea9dbd0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fda50000c00 tx=0x7fda5000a2d0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.250+0000 7fda6c177740 1 -- 172.21.15.90:0/2649269356 shutdown_connections 2024-04-25T11:45:23.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.250+0000 7fda6c177740 1 --2- 172.21.15.90:0/2649269356 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56124e702410 0x56124eabd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.250+0000 7fda6c177740 1 --2- 172.21.15.90:0/2649269356 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e6e1730 0x56124ea9dbd0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.250+0000 7fda6c177740 1 --2- 172.21.15.90:0/2649269356 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56124e6d9550 0x56124eab00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.250+0000 7fda6c177740 1 -- 172.21.15.90:0/2649269356 >> 172.21.15.90:0/2649269356 conn(0x56124e6f6560 msgr2=0x56124ea9d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:23.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.250+0000 7fda6c177740 1 -- 172.21.15.90:0/2649269356 shutdown_connections 2024-04-25T11:45:23.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.250+0000 7fda6c177740 1 -- 172.21.15.90:0/2649269356 wait complete. 2024-04-25T11:45:23.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.251+0000 7fda6c177740 1 -- start start 2024-04-25T11:45:23.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.251+0000 7fda6c177740 1 Processor -- start 2024-04-25T11:45:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.251+0000 7fda6c177740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56124e6d9550 0x56124eb23680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.251+0000 7fda6c177740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e6e1730 0x56124eb23bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.251+0000 7fda6c177740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56124e702410 0x56124eac3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.251+0000 7fda6c177740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fda601aefd0 con 0x56124e6e1730 2024-04-25T11:45:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.251+0000 7fda6c177740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fda601ae900 con 0x56124e702410 2024-04-25T11:45:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.251+0000 7fda6c177740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fda601abb60 con 0x56124e6d9550 2024-04-25T11:45:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.251+0000 7fda6c177740 10 client.0 _ll_register_callbacks cb 0x7fda58003a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.251+0000 7fda5ed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e6e1730 0x56124eb23bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.251+0000 7fda5ed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e6e1730 0x56124eb23bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42290/0 (socket says 172.21.15.90:42290) 2024-04-25T11:45:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.251+0000 7fda5ed9d640 1 -- 172.21.15.90:0/3743780571 learned_addr learned my addr 172.21.15.90:0/3743780571 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.251+0000 7fda5f59e640 1 --2- 172.21.15.90:0/3743780571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56124e6d9550 0x56124eb23680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.251+0000 7fda5f59e640 1 --2- 172.21.15.90:0/3743780571 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56124e702410 0x56124eac3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.251+0000 7fda5ed9d640 1 -- 172.21.15.90:0/3743780571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56124e6d9550 msgr2=0x56124eb23680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:23.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.251+0000 7fda5ed9d640 1 --2- 172.21.15.90:0/3743780571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56124e6d9550 0x56124eb23680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.252+0000 7fda5ed9d640 1 -- 172.21.15.90:0/3743780571 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56124e702410 msgr2=0x56124eac3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:23.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.252+0000 7fda5ed9d640 1 --2- 172.21.15.90:0/3743780571 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56124e702410 0x56124eac3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.252+0000 7fda5ed9d640 1 -- 172.21.15.90:0/3743780571 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56124eac3b40 con 0x56124e6e1730 2024-04-25T11:45:23.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.252+0000 7fda5f59e640 1 --2- 172.21.15.90:0/3743780571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56124e6d9550 0x56124eb23680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:23.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.252+0000 7fda5ed9d640 1 --2- 172.21.15.90:0/3743780571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e6e1730 0x56124eb23bc0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fda5000ad70 tx=0x7fda50005360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53301959a8e3cb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:23.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.252+0000 7fda42ff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:23.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.252+0000 7fda42ff5640 1 -- 172.21.15.90:0/3743780571 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda500205b0 con 0x56124e6e1730 2024-04-25T11:45:23.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.252+0000 7fda42ff5640 1 -- 172.21.15.90:0/3743780571 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7fda50020750 con 0x56124e6e1730 2024-04-25T11:45:23.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.252+0000 7fda6c177740 1 -- 172.21.15.90:0/3743780571 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda50024070 con 0x56124e6e1730 2024-04-25T11:45:23.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.252+0000 7fda42ff5640 1 -- 172.21.15.90:0/3743780571 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fda50020950 con 0x56124e6e1730 2024-04-25T11:45:23.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.252+0000 7fda6c177740 10 client.18930 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:23.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.252+0000 7fda6c177740 1 -- 172.21.15.90:0/3743780571 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56124eb2ac70 con 0x56124e6e1730 2024-04-25T11:45:23.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.252+0000 7fda42ff5640 1 -- 172.21.15.90:0/3743780571 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda50024070 con 0x56124e6e1730 2024-04-25T11:45:23.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.252+0000 7fda42ff5640 1 -- 172.21.15.90:0/3743780571 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fda50031760 con 0x56124e6e1730 2024-04-25T11:45:23.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.252+0000 7fda6c177740 10 client.18930 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:23.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.252+0000 7fda6c177740 10 client.18930 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:45:23.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.252+0000 7fda6c177740 20 client.18930 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:23.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.252+0000 7fda6c177740 1 --2- 172.21.15.90:0/3743780571 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56124e696630 0x56124eb2d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.252+0000 7fda6c177740 4 client.18930 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T11:45:23.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.252+0000 7fda6c177740 1 -- 172.21.15.90:0/3743780571 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56124ea10580 con 0x56124e696630 2024-04-25T11:45:23.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.253+0000 7fda5f59e640 1 --2- 172.21.15.90:0/3743780571 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56124e696630 0x56124eb2d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.253+0000 7fda42ff5640 10 client.18930 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:23.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.253+0000 7fda5f59e640 1 --2- 172.21.15.90:0/3743780571 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56124e696630 0x56124eb2d3c0 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4743ab78ba4f8a07 server_cookie=7fd58f20bbfb0de9 in_seq=0 out_seq=0 2024-04-25T11:45:23.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.255+0000 7fda42ff5640 1 -- 172.21.15.90:0/3743780571 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56124ea10580 con 0x56124e696630 2024-04-25T11:45:23.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.255+0000 7fda42ff5640 10 client.18930 handle_command_reply: tid=0 2024-04-25T11:45:23.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.255+0000 7fda6c177740 1 client.18930 shutdown 2024-04-25T11:45:23.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.255+0000 7fda6c177740 1 -- 172.21.15.90:0/3743780571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e6e1730 msgr2=0x56124eb23bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:23.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.255+0000 7fda6c177740 1 --2- 172.21.15.90:0/3743780571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e6e1730 0x56124eb23bc0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fda5000ad70 tx=0x7fda50005360 comp rx=0 tx=0).stop 2024-04-25T11:45:23.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.255+0000 7fda6c177740 1 -- 172.21.15.90:0/3743780571 shutdown_connections 2024-04-25T11:45:23.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.255+0000 7fda6c177740 1 --2- 172.21.15.90:0/3743780571 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56124e696630 0x56124eb2d3c0 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.255+0000 7fda6c177740 1 --2- 172.21.15.90:0/3743780571 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56124e702410 0x56124eac3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.255+0000 7fda6c177740 1 --2- 172.21.15.90:0/3743780571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56124e6e1730 0x56124eb23bc0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.255+0000 7fda6c177740 1 --2- 172.21.15.90:0/3743780571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56124e6d9550 0x56124eb23680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.255+0000 7fda6c177740 1 -- 172.21.15.90:0/3743780571 >> 172.21.15.90:0/3743780571 conn(0x56124e6f6560 msgr2=0x56124ea9dc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:23.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.255+0000 7fda42ff5640 0 client.18930 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:23.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.255+0000 7fda6c177740 1 -- 172.21.15.90:0/3743780571 shutdown_connections 2024-04-25T11:45:23.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.255+0000 7fda6c177740 1 -- 172.21.15.90:0/3743780571 wait complete. 2024-04-25T11:45:23.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.255+0000 7fda6c177740 20 client.18930 trim_cache size 0 max 16384 2024-04-25T11:45:23.247 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:45:23.247 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-25T11:45:23.247 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "12f39fd7-897e-4d8a-b5dc-a006d3fbd56a", 2024-04-25T11:45:23.247 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-25T11:45:23.247 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:45:23.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.255+0000 7fda651ee640 1 -- 172.21.15.90:0/2726355539 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fda480afc60 msgr2=0x7fda480b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:23.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.255+0000 7fda651ee640 1 --2- 172.21.15.90:0/2726355539 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fda480afc60 0x7fda480b2170 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fda540dc500 tx=0x7fda54004570 comp rx=0 tx=0).stop 2024-04-25T11:45:23.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.255+0000 7fda651ee640 1 -- 172.21.15.90:0/2726355539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda60108a10 msgr2=0x7fda6019ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:23.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.255+0000 7fda651ee640 1 --2- 172.21.15.90:0/2726355539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda60108a10 0x7fda6019ace0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fda5400c9a0 tx=0x7fda5400cda0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.255+0000 7fda651ee640 1 -- 172.21.15.90:0/2726355539 shutdown_connections 2024-04-25T11:45:23.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.255+0000 7fda651ee640 1 --2- 172.21.15.90:0/2726355539 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fda480afc60 0x7fda480b2170 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.256+0000 7fda651ee640 1 --2- 172.21.15.90:0/2726355539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda60108eb0 0x7fda6019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.256+0000 7fda651ee640 1 --2- 172.21.15.90:0/2726355539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda60108a10 0x7fda6019ace0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.256+0000 7fda651ee640 1 --2- 172.21.15.90:0/2726355539 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fda60103170 0x7fda6019a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.256+0000 7fda651ee640 1 -- 172.21.15.90:0/2726355539 >> 172.21.15.90:0/2726355539 conn(0x7fda60078150 msgr2=0x7fda60106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:23.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.256+0000 7fda651ee640 1 -- 172.21.15.90:0/2726355539 shutdown_connections 2024-04-25T11:45:23.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.256+0000 7fda651ee640 1 -- 172.21.15.90:0/2726355539 wait complete. 2024-04-25T11:45:23.252 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:45:23.252 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:45:23.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.377+0000 7fb18e677640 1 -- 172.21.15.90:0/86632504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb18810e060 msgr2=0x7fb188110550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:23.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.377+0000 7fb18e677640 1 --2- 172.21.15.90:0/86632504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb18810e060 0x7fb188110550 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fb180009ca0 tx=0x7fb18002a9e0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.377+0000 7fb18e677640 1 -- 172.21.15.90:0/86632504 shutdown_connections 2024-04-25T11:45:23.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.377+0000 7fb18e677640 1 --2- 172.21.15.90:0/86632504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb18810e060 0x7fb188110550 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.377+0000 7fb18e677640 1 --2- 172.21.15.90:0/86632504 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb18810b6b0 0x7fb18810dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.377+0000 7fb18e677640 1 --2- 172.21.15.90:0/86632504 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb188104c50 0x7fb188105060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.377+0000 7fb18e677640 1 -- 172.21.15.90:0/86632504 >> 172.21.15.90:0/86632504 conn(0x7fb188078100 msgr2=0x7fb188078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:23.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.377+0000 7fb18e677640 1 -- 172.21.15.90:0/86632504 shutdown_connections 2024-04-25T11:45:23.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.377+0000 7fb18e677640 1 -- 172.21.15.90:0/86632504 wait complete. 2024-04-25T11:45:23.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.378+0000 7fb18e677640 1 Processor -- start 2024-04-25T11:45:23.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.378+0000 7fb18e677640 1 -- start start 2024-04-25T11:45:23.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.378+0000 7fb18e677640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb188104c50 0x7fb18819a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.378+0000 7fb18e677640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb18810b6b0 0x7fb18819ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.378+0000 7fb18e677640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb18810e060 0x7fb18819eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.378+0000 7fb18e677640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb188113cc0 con 0x7fb18810b6b0 2024-04-25T11:45:23.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.378+0000 7fb18e677640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb188113b40 con 0x7fb18810e060 2024-04-25T11:45:23.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.378+0000 7fb18e677640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb188113e40 con 0x7fb188104c50 2024-04-25T11:45:23.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.378+0000 7fb1877fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb18810b6b0 0x7fb18819ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.378+0000 7fb1877fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb18810b6b0 0x7fb18819ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42310/0 (socket says 172.21.15.90:42310) 2024-04-25T11:45:23.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.378+0000 7fb1877fe640 1 -- 172.21.15.90:0/2881659313 learned_addr learned my addr 172.21.15.90:0/2881659313 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:23.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.378+0000 7fb18cc14640 1 --2- 172.21.15.90:0/2881659313 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb18810e060 0x7fb18819eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.378+0000 7fb187fff640 1 --2- 172.21.15.90:0/2881659313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb188104c50 0x7fb18819a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.378+0000 7fb1877fe640 1 -- 172.21.15.90:0/2881659313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb188104c50 msgr2=0x7fb18819a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:23.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.378+0000 7fb1877fe640 1 --2- 172.21.15.90:0/2881659313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb188104c50 0x7fb18819a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.378+0000 7fb1877fe640 1 -- 172.21.15.90:0/2881659313 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb18810e060 msgr2=0x7fb18819eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:23.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.378+0000 7fb1877fe640 1 --2- 172.21.15.90:0/2881659313 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb18810e060 0x7fb18819eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.378+0000 7fb1877fe640 1 -- 172.21.15.90:0/2881659313 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb18819f520 con 0x7fb18810b6b0 2024-04-25T11:45:23.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.378+0000 7fb187fff640 1 --2- 172.21.15.90:0/2881659313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb188104c50 0x7fb18819a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:23.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.378+0000 7fb18cc14640 1 --2- 172.21.15.90:0/2881659313 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb18810e060 0x7fb18819eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:23.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.379+0000 7fb1877fe640 1 --2- 172.21.15.90:0/2881659313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb18810b6b0 0x7fb18819ac20 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fb17c00d950 tx=0x7fb17c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:23.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.379+0000 7fb1857fa640 1 -- 172.21.15.90:0/2881659313 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c017070 con 0x7fb18810b6b0 2024-04-25T11:45:23.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.379+0000 7fb18e677640 1 -- 172.21.15.90:0/2881659313 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb18819f810 con 0x7fb18810b6b0 2024-04-25T11:45:23.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.379+0000 7fb1857fa640 1 -- 172.21.15.90:0/2881659313 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb17c009c30 con 0x7fb18810b6b0 2024-04-25T11:45:23.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.379+0000 7fb1857fa640 1 -- 172.21.15.90:0/2881659313 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c00f620 con 0x7fb18810b6b0 2024-04-25T11:45:23.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.379+0000 7fb18e677640 1 -- 172.21.15.90:0/2881659313 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1881a7190 con 0x7fb18810b6b0 2024-04-25T11:45:23.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.381+0000 7fb1857fa640 1 -- 172.21.15.90:0/2881659313 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fb17c010460 con 0x7fb18810b6b0 2024-04-25T11:45:23.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.381+0000 7fb195605740 1 Processor -- start 2024-04-25T11:45:23.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.381+0000 7fb195605740 1 -- start start 2024-04-25T11:45:23.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.382+0000 7fb195605740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5609cf6e9390 0x5609cfa89a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.382+0000 7fb195605740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cf6c04d0 0x5609cfa89fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.382+0000 7fb195605740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c86b0 0x5609cfa968f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.382+0000 7fb195605740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5609cf9efd80 con 0x5609cf6c86b0 2024-04-25T11:45:23.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.382+0000 7fb195605740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5609cf9a98b0 con 0x5609cf6e9390 2024-04-25T11:45:23.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.382+0000 7fb195605740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5609cf9ebd40 con 0x5609cf6c04d0 2024-04-25T11:45:23.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.382+0000 7fb1877fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c86b0 0x5609cfa968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.382+0000 7fb1877fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c86b0 0x5609cfa968f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42312/0 (socket says 172.21.15.90:42312) 2024-04-25T11:45:23.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.382+0000 7fb1877fe640 1 -- 172.21.15.90:0/3003494958 learned_addr learned my addr 172.21.15.90:0/3003494958 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:23.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.382+0000 7fb18cc14640 1 --2- 172.21.15.90:0/3003494958 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5609cf6e9390 0x5609cfa89a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.382+0000 7fb187fff640 1 --2- 172.21.15.90:0/3003494958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cf6c04d0 0x5609cfa89fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.382+0000 7fb1857fa640 1 --2- 172.21.15.90:0/2881659313 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb1700afc60 0x7fb1700b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.382+0000 7fb1877fe640 1 -- 172.21.15.90:0/3003494958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cf6c04d0 msgr2=0x5609cfa89fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:23.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.382+0000 7fb1877fe640 1 --2- 172.21.15.90:0/3003494958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cf6c04d0 0x5609cfa89fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.382+0000 7fb1877fe640 1 -- 172.21.15.90:0/3003494958 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5609cf6e9390 msgr2=0x5609cfa89a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:23.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.382+0000 7fb1877fe640 1 --2- 172.21.15.90:0/3003494958 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5609cf6e9390 0x5609cfa89a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.382+0000 7fb1877fe640 1 -- 172.21.15.90:0/3003494958 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609cf97aa40 con 0x5609cf6c86b0 2024-04-25T11:45:23.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.382+0000 7fb187fff640 1 --2- 172.21.15.90:0/3003494958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cf6c04d0 0x5609cfa89fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:23.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.382+0000 7fb18cc14640 1 --2- 172.21.15.90:0/3003494958 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5609cf6e9390 0x5609cfa89a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:23.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.382+0000 7fb18cc14640 1 --2- 172.21.15.90:0/2881659313 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb1700afc60 0x7fb1700b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.382+0000 7fb1857fa640 1 -- 172.21.15.90:0/2881659313 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7fb17c0deda0 con 0x7fb18810b6b0 2024-04-25T11:45:23.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.382+0000 7fb1877fe640 1 --2- 172.21.15.90:0/3003494958 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c86b0 0x5609cfa968f0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fb17c0dc3b0 tx=0x7fb17c06d2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5b8d8833e3cfa2b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:23.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.383+0000 7fb18cc14640 1 --2- 172.21.15.90:0/2881659313 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb1700afc60 0x7fb1700b2170 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb180009880 tx=0x7fb1800095c0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:23.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.383+0000 7fb16e7fc640 1 -- 172.21.15.90:0/3003494958 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c0a6c60 con 0x5609cf6c86b0 2024-04-25T11:45:23.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.383+0000 7fb16e7fc640 1 -- 172.21.15.90:0/3003494958 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb17c0a69a0 con 0x5609cf6c86b0 2024-04-25T11:45:23.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.383+0000 7fb16e7fc640 1 -- 172.21.15.90:0/3003494958 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c06da50 con 0x5609cf6c86b0 2024-04-25T11:45:23.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.383+0000 7fb195605740 1 -- 172.21.15.90:0/3003494958 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c86b0 msgr2=0x5609cfa968f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:23.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.383+0000 7fb195605740 1 --2- 172.21.15.90:0/3003494958 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c86b0 0x5609cfa968f0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fb17c0dc3b0 tx=0x7fb17c06d2b0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.383+0000 7fb195605740 1 -- 172.21.15.90:0/3003494958 shutdown_connections 2024-04-25T11:45:23.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.383+0000 7fb195605740 1 --2- 172.21.15.90:0/3003494958 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c86b0 0x5609cfa968f0 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.383+0000 7fb195605740 1 --2- 172.21.15.90:0/3003494958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cf6c04d0 0x5609cfa89fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.383+0000 7fb195605740 1 --2- 172.21.15.90:0/3003494958 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5609cf6e9390 0x5609cfa89a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.383+0000 7fb195605740 1 -- 172.21.15.90:0/3003494958 >> 172.21.15.90:0/3003494958 conn(0x5609cf6dd4e0 msgr2=0x5609cfa86210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:23.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.384+0000 7fb195605740 1 -- 172.21.15.90:0/3003494958 shutdown_connections 2024-04-25T11:45:23.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.384+0000 7fb195605740 1 -- 172.21.15.90:0/3003494958 wait complete. 2024-04-25T11:45:23.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.384+0000 7fb195605740 1 -- start start 2024-04-25T11:45:23.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.384+0000 7fb195605740 1 Processor -- start 2024-04-25T11:45:23.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.384+0000 7fb195605740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cf6c04d0 0x5609cfa94620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.384+0000 7fb195605740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c86b0 0x5609cfa94b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.384+0000 7fb195605740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5609cf6e9390 0x5609cfa9a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.384+0000 7fb195605740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5609cfa96f80 con 0x5609cf6c86b0 2024-04-25T11:45:23.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.384+0000 7fb195605740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5609cf976d10 con 0x5609cf6e9390 2024-04-25T11:45:23.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.384+0000 7fb195605740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5609cfa97100 con 0x5609cf6c04d0 2024-04-25T11:45:23.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.384+0000 7fb195605740 10 client.0 _ll_register_callbacks cb 0x5609cf9a3210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:23.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.385+0000 7fb1877fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c86b0 0x5609cfa94b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.385+0000 7fb1877fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c86b0 0x5609cfa94b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42324/0 (socket says 172.21.15.90:42324) 2024-04-25T11:45:23.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.385+0000 7fb1877fe640 1 -- 172.21.15.90:0/1873192607 learned_addr learned my addr 172.21.15.90:0/1873192607 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:23.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.385+0000 7fb18cc14640 1 --2- 172.21.15.90:0/1873192607 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5609cf6e9390 0x5609cfa9a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.385+0000 7fb187fff640 1 --2- 172.21.15.90:0/1873192607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cf6c04d0 0x5609cfa94620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.385+0000 7fb1877fe640 1 -- 172.21.15.90:0/1873192607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cf6c04d0 msgr2=0x5609cfa94620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:23.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.385+0000 7fb1877fe640 1 --2- 172.21.15.90:0/1873192607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cf6c04d0 0x5609cfa94620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.385+0000 7fb1877fe640 1 -- 172.21.15.90:0/1873192607 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5609cf6e9390 msgr2=0x5609cfa9a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:23.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.385+0000 7fb1877fe640 1 --2- 172.21.15.90:0/1873192607 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5609cf6e9390 0x5609cfa9a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.385+0000 7fb1877fe640 1 -- 172.21.15.90:0/1873192607 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609cf7cb7d0 con 0x5609cf6c86b0 2024-04-25T11:45:23.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.385+0000 7fb1877fe640 1 --2- 172.21.15.90:0/1873192607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c86b0 0x5609cfa94b60 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fb17c0a2b50 tx=0x7fb17c009740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f48003a22eec49f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:23.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.385+0000 7fb1677fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:23.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.385+0000 7fb1677fe640 1 -- 172.21.15.90:0/1873192607 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c020070 con 0x5609cf6c86b0 2024-04-25T11:45:23.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.385+0000 7fb1677fe640 1 -- 172.21.15.90:0/1873192607 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7fb17c0af860 con 0x5609cf6c86b0 2024-04-25T11:45:23.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.385+0000 7fb195605740 1 -- 172.21.15.90:0/1873192607 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb17c0a69a0 con 0x5609cf6c86b0 2024-04-25T11:45:23.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.385+0000 7fb1677fe640 1 -- 172.21.15.90:0/1873192607 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb17c0a6600 con 0x5609cf6c86b0 2024-04-25T11:45:23.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.385+0000 7fb195605740 10 client.18942 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:23.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.385+0000 7fb195605740 1 -- 172.21.15.90:0/1873192607 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5609cf7013f0 con 0x5609cf6c86b0 2024-04-25T11:45:23.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.385+0000 7fb1677fe640 1 -- 172.21.15.90:0/1873192607 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb17c0a69a0 con 0x5609cf6c86b0 2024-04-25T11:45:23.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.386+0000 7fb1677fe640 1 -- 172.21.15.90:0/1873192607 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fb17c0710d0 con 0x5609cf6c86b0 2024-04-25T11:45:23.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.386+0000 7fb195605740 10 client.18942 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:23.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.386+0000 7fb195605740 10 client.18942 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:45:23.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.386+0000 7fb195605740 20 client.18942 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:23.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.386+0000 7fb195605740 1 --2- 172.21.15.90:0/1873192607 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5609cf6a7f60 0x5609cfaa6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.386+0000 7fb195605740 4 client.18942 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:45:23.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.386+0000 7fb195605740 1 -- 172.21.15.90:0/1873192607 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609cf9fd990 con 0x5609cf6a7f60 2024-04-25T11:45:23.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.386+0000 7fb187fff640 1 --2- 172.21.15.90:0/1873192607 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5609cf6a7f60 0x5609cfaa6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.386+0000 7fb1677fe640 10 client.18942 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:23.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.386+0000 7fb187fff640 1 --2- 172.21.15.90:0/1873192607 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5609cf6a7f60 0x5609cfaa6510 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ceee3351b520eece server_cookie=db2a4a5e7e4b4c1d in_seq=0 out_seq=0 2024-04-25T11:45:23.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.387+0000 7fb1677fe640 1 -- 172.21.15.90:0/1873192607 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5609cf9fd990 con 0x5609cf6a7f60 2024-04-25T11:45:23.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.387+0000 7fb1677fe640 10 client.18942 handle_command_reply: tid=0 2024-04-25T11:45:23.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.387+0000 7fb195605740 1 client.18942 shutdown 2024-04-25T11:45:23.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.387+0000 7fb195605740 1 -- 172.21.15.90:0/1873192607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c86b0 msgr2=0x5609cfa94b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:23.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.387+0000 7fb195605740 1 --2- 172.21.15.90:0/1873192607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c86b0 0x5609cfa94b60 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fb17c0a2b50 tx=0x7fb17c009740 comp rx=0 tx=0).stop 2024-04-25T11:45:23.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.387+0000 7fb195605740 1 -- 172.21.15.90:0/1873192607 shutdown_connections 2024-04-25T11:45:23.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.387+0000 7fb195605740 1 --2- 172.21.15.90:0/1873192607 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5609cf6a7f60 0x5609cfaa6510 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.387+0000 7fb195605740 1 --2- 172.21.15.90:0/1873192607 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5609cf6e9390 0x5609cfa9a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.387+0000 7fb195605740 1 --2- 172.21.15.90:0/1873192607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c86b0 0x5609cfa94b60 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.387+0000 7fb1677fe640 0 client.18942 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:23.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.387+0000 7fb195605740 1 --2- 172.21.15.90:0/1873192607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cf6c04d0 0x5609cfa94620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.387+0000 7fb195605740 1 -- 172.21.15.90:0/1873192607 >> 172.21.15.90:0/1873192607 conn(0x5609cf6dd4e0 msgr2=0x5609cfa8b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:23.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.387+0000 7fb195605740 1 -- 172.21.15.90:0/1873192607 shutdown_connections 2024-04-25T11:45:23.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.387+0000 7fb195605740 1 -- 172.21.15.90:0/1873192607 wait complete. 2024-04-25T11:45:23.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.388+0000 7fb195605740 20 client.18942 trim_cache size 0 max 16384 2024-04-25T11:45:23.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.405+0000 7fb195605740 1 Processor -- start 2024-04-25T11:45:23.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.405+0000 7fb195605740 1 -- start start 2024-04-25T11:45:23.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.405+0000 7fb195605740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5609cf6a7f60 0x5609cfa859d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.405+0000 7fb195605740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cf6c04d0 0x5609cfa85f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.405+0000 7fb195605740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c86b0 0x5609cfaa0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.405+0000 7fb195605740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb1881afcb0 con 0x5609cf6c86b0 2024-04-25T11:45:23.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.405+0000 7fb195605740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb1881b17f0 con 0x5609cf6a7f60 2024-04-25T11:45:23.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.405+0000 7fb195605740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb1881b0380 con 0x5609cf6c04d0 2024-04-25T11:45:23.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.406+0000 7fb18cc14640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c86b0 0x5609cfaa0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.406+0000 7fb18cc14640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c86b0 0x5609cfaa0100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42336/0 (socket says 172.21.15.90:42336) 2024-04-25T11:45:23.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.406+0000 7fb18cc14640 1 -- 172.21.15.90:0/169164358 learned_addr learned my addr 172.21.15.90:0/169164358 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:23.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.406+0000 7fb18cc14640 1 -- 172.21.15.90:0/169164358 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cf6c04d0 msgr2=0x5609cfa85f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:23.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.406+0000 7fb1877fe640 1 --2- 172.21.15.90:0/169164358 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cf6c04d0 0x5609cfa85f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.406+0000 7fb187fff640 1 --2- 172.21.15.90:0/169164358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5609cf6a7f60 0x5609cfa859d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.406+0000 7fb18cc14640 1 --2- 172.21.15.90:0/169164358 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cf6c04d0 0x5609cfa85f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.406+0000 7fb18cc14640 1 -- 172.21.15.90:0/169164358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5609cf6a7f60 msgr2=0x5609cfa859d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:23.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.406+0000 7fb18cc14640 1 --2- 172.21.15.90:0/169164358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5609cf6a7f60 0x5609cfa859d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.406+0000 7fb18cc14640 1 -- 172.21.15.90:0/169164358 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609cf6c10f0 con 0x5609cf6c86b0 2024-04-25T11:45:23.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.406+0000 7fb18cc14640 1 --2- 172.21.15.90:0/169164358 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c86b0 0x5609cfaa0100 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fb180035000 tx=0x7fb180041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bc3f67dedc86a9d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:23.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.406+0000 7fb16dffb640 1 -- 172.21.15.90:0/169164358 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb180033040 con 0x5609cf6c86b0 2024-04-25T11:45:23.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.406+0000 7fb16dffb640 1 -- 172.21.15.90:0/169164358 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb18002e650 con 0x5609cf6c86b0 2024-04-25T11:45:23.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.406+0000 7fb16dffb640 1 -- 172.21.15.90:0/169164358 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb18002f9c0 con 0x5609cf6c86b0 2024-04-25T11:45:23.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.407+0000 7fb195605740 1 -- 172.21.15.90:0/169164358 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c86b0 msgr2=0x5609cfaa0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:23.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.407+0000 7fb195605740 1 --2- 172.21.15.90:0/169164358 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c86b0 0x5609cfaa0100 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fb180035000 tx=0x7fb180041000 comp rx=0 tx=0).stop 2024-04-25T11:45:23.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.407+0000 7fb195605740 1 -- 172.21.15.90:0/169164358 shutdown_connections 2024-04-25T11:45:23.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.407+0000 7fb195605740 1 --2- 172.21.15.90:0/169164358 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c86b0 0x5609cfaa0100 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.407+0000 7fb195605740 1 --2- 172.21.15.90:0/169164358 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cf6c04d0 0x5609cfa85f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.407+0000 7fb195605740 1 --2- 172.21.15.90:0/169164358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5609cf6a7f60 0x5609cfa859d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.407+0000 7fb195605740 1 -- 172.21.15.90:0/169164358 >> 172.21.15.90:0/169164358 conn(0x5609cf6dd4e0 msgr2=0x5609cfa993b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:23.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.407+0000 7fb195605740 1 -- 172.21.15.90:0/169164358 shutdown_connections 2024-04-25T11:45:23.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.407+0000 7fb195605740 1 -- 172.21.15.90:0/169164358 wait complete. 2024-04-25T11:45:23.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.407+0000 7fb195605740 1 -- start start 2024-04-25T11:45:23.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.407+0000 7fb195605740 1 Processor -- start 2024-04-25T11:45:23.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.408+0000 7fb195605740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5609cf6a7f60 0x5609cfb06dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.408+0000 7fb195605740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c04d0 0x5609cfb07310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.408+0000 7fb195605740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cf6c86b0 0x5609cfaa02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.408+0000 7fb195605740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb1881b1120 con 0x5609cf6c04d0 2024-04-25T11:45:23.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.408+0000 7fb195605740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb1881b0a50 con 0x5609cf6a7f60 2024-04-25T11:45:23.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.408+0000 7fb195605740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb1881adcb0 con 0x5609cf6c86b0 2024-04-25T11:45:23.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.408+0000 7fb195605740 10 client.0 _ll_register_callbacks cb 0x7fb17c0a2de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:23.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.408+0000 7fb1877fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c04d0 0x5609cfb07310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.408+0000 7fb1877fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c04d0 0x5609cfb07310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42348/0 (socket says 172.21.15.90:42348) 2024-04-25T11:45:23.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.408+0000 7fb1877fe640 1 -- 172.21.15.90:0/3767307160 learned_addr learned my addr 172.21.15.90:0/3767307160 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:23.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.408+0000 7fb18cc14640 1 --2- 172.21.15.90:0/3767307160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cf6c86b0 0x5609cfaa02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.408+0000 7fb187fff640 1 --2- 172.21.15.90:0/3767307160 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5609cf6a7f60 0x5609cfb06dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.408+0000 7fb1877fe640 1 -- 172.21.15.90:0/3767307160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cf6c86b0 msgr2=0x5609cfaa02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:23.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.408+0000 7fb1877fe640 1 --2- 172.21.15.90:0/3767307160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cf6c86b0 0x5609cfaa02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.408+0000 7fb1877fe640 1 -- 172.21.15.90:0/3767307160 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5609cf6a7f60 msgr2=0x5609cfb06dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:23.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.408+0000 7fb1877fe640 1 --2- 172.21.15.90:0/3767307160 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5609cf6a7f60 0x5609cfb06dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.408+0000 7fb1877fe640 1 -- 172.21.15.90:0/3767307160 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609cf9b04b0 con 0x5609cf6c04d0 2024-04-25T11:45:23.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.408+0000 7fb1877fe640 1 --2- 172.21.15.90:0/3767307160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c04d0 0x5609cfb07310 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fb17c0dca40 tx=0x7fb17c0e2450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51b2fd10353db106 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:23.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.408+0000 7fb1677fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:23.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.408+0000 7fb1677fe640 1 -- 172.21.15.90:0/3767307160 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c020070 con 0x5609cf6c04d0 2024-04-25T11:45:23.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.408+0000 7fb1677fe640 1 -- 172.21.15.90:0/3767307160 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8702+0+0 (secure 0 0 0) 0x7fb17c0a70f0 con 0x5609cf6c04d0 2024-04-25T11:45:23.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.408+0000 7fb195605740 1 -- 172.21.15.90:0/3767307160 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb18002e650 con 0x5609cf6c04d0 2024-04-25T11:45:23.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.409+0000 7fb1677fe640 1 -- 172.21.15.90:0/3767307160 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb17c0a67e0 con 0x5609cf6c04d0 2024-04-25T11:45:23.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.409+0000 7fb195605740 10 client.18948 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:23.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.409+0000 7fb195605740 1 -- 172.21.15.90:0/3767307160 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5609cfb0e2c0 con 0x5609cf6c04d0 2024-04-25T11:45:23.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.409+0000 7fb1677fe640 1 -- 172.21.15.90:0/3767307160 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb18002e650 con 0x5609cf6c04d0 2024-04-25T11:45:23.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.409+0000 7fb1677fe640 1 -- 172.21.15.90:0/3767307160 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fb17c0af860 con 0x5609cf6c04d0 2024-04-25T11:45:23.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.409+0000 7fb195605740 10 client.18948 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:23.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.409+0000 7fb195605740 10 client.18948 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:45:23.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.409+0000 7fb195605740 20 client.18948 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:23.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.409+0000 7fb195605740 1 --2- 172.21.15.90:0/3767307160 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5609cf6e9390 0x5609cfb10800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:23.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.409+0000 7fb195605740 4 client.18948 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:45:23.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.409+0000 7fb195605740 1 -- 172.21.15.90:0/3767307160 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5609cf9f7800 con 0x5609cf6e9390 2024-04-25T11:45:23.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.409+0000 7fb187fff640 1 --2- 172.21.15.90:0/3767307160 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5609cf6e9390 0x5609cfb10800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:23.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.410+0000 7fb187fff640 1 --2- 172.21.15.90:0/3767307160 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5609cf6e9390 0x5609cfb10800 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9776528954729f27 server_cookie=d38e9e87a25c9329 in_seq=0 out_seq=0 2024-04-25T11:45:23.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.410+0000 7fb1677fe640 10 client.18948 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:23.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.410+0000 7fb1677fe640 1 -- 172.21.15.90:0/3767307160 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5609cf9f7800 con 0x5609cf6e9390 2024-04-25T11:45:23.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.410+0000 7fb1677fe640 10 client.18948 handle_command_reply: tid=0 2024-04-25T11:45:23.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.410+0000 7fb195605740 1 client.18948 shutdown 2024-04-25T11:45:23.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.410+0000 7fb195605740 1 -- 172.21.15.90:0/3767307160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c04d0 msgr2=0x5609cfb07310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:23.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.410+0000 7fb195605740 1 --2- 172.21.15.90:0/3767307160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c04d0 0x5609cfb07310 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fb17c0dca40 tx=0x7fb17c0e2450 comp rx=0 tx=0).stop 2024-04-25T11:45:23.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.410+0000 7fb195605740 1 -- 172.21.15.90:0/3767307160 shutdown_connections 2024-04-25T11:45:23.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.410+0000 7fb195605740 1 --2- 172.21.15.90:0/3767307160 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5609cf6e9390 0x5609cfb10800 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.410+0000 7fb195605740 1 --2- 172.21.15.90:0/3767307160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cf6c86b0 0x5609cfaa02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.410+0000 7fb195605740 1 --2- 172.21.15.90:0/3767307160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609cf6c04d0 0x5609cfb07310 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.410+0000 7fb195605740 1 --2- 172.21.15.90:0/3767307160 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5609cf6a7f60 0x5609cfb06dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.410+0000 7fb1677fe640 0 client.18948 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:23.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.410+0000 7fb195605740 1 -- 172.21.15.90:0/3767307160 >> 172.21.15.90:0/3767307160 conn(0x5609cf6dd4e0 msgr2=0x5609cfa96130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:23.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.410+0000 7fb195605740 1 -- 172.21.15.90:0/3767307160 shutdown_connections 2024-04-25T11:45:23.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.410+0000 7fb195605740 1 -- 172.21.15.90:0/3767307160 wait complete. 2024-04-25T11:45:23.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.410+0000 7fb195605740 20 client.18948 trim_cache size 0 max 16384 2024-04-25T11:45:23.403 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:45:23.403 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T11:45:23.403 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-25T11:45:23.403 INFO:teuthology.orchestra.run.smithi090.stdout: "12f39fd7-897e-4d8a-b5dc-a006d3fbd56a": { 2024-04-25T11:45:23.403 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-25T11:45:23.403 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "12f39fd7-897e-4d8a-b5dc-a006d3fbd56a", 2024-04-25T11:45:23.403 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-25T11:45:23.403 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:45:23.403 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:45:23.403 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:45:23.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.411+0000 7fb18e677640 1 -- 172.21.15.90:0/2881659313 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb1700afc60 msgr2=0x7fb1700b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:23.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.411+0000 7fb18e677640 1 --2- 172.21.15.90:0/2881659313 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb1700afc60 0x7fb1700b2170 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb180009880 tx=0x7fb1800095c0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.411+0000 7fb18e677640 1 -- 172.21.15.90:0/2881659313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb18810b6b0 msgr2=0x7fb18819ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:23.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.411+0000 7fb18e677640 1 --2- 172.21.15.90:0/2881659313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb18810b6b0 0x7fb18819ac20 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fb17c00d950 tx=0x7fb17c00de10 comp rx=0 tx=0).stop 2024-04-25T11:45:23.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.411+0000 7fb18e677640 1 -- 172.21.15.90:0/2881659313 shutdown_connections 2024-04-25T11:45:23.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.411+0000 7fb18e677640 1 --2- 172.21.15.90:0/2881659313 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb1700afc60 0x7fb1700b2170 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.411+0000 7fb18e677640 1 --2- 172.21.15.90:0/2881659313 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb18810e060 0x7fb18819eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.411+0000 7fb18e677640 1 --2- 172.21.15.90:0/2881659313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb18810b6b0 0x7fb18819ac20 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.411+0000 7fb18e677640 1 --2- 172.21.15.90:0/2881659313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb188104c50 0x7fb18819a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:23.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.411+0000 7fb18e677640 1 -- 172.21.15.90:0/2881659313 >> 172.21.15.90:0/2881659313 conn(0x7fb188078100 msgr2=0x7fb188101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:23.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.411+0000 7fb18e677640 1 -- 172.21.15.90:0/2881659313 shutdown_connections 2024-04-25T11:45:23.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:23.411+0000 7fb18e677640 1 -- 172.21.15.90:0/2881659313 wait complete. 2024-04-25T11:45:23.407 INFO:tasks.cephfs.filesystem:scrub status for tag:12f39fd7-897e-4d8a-b5dc-a006d3fbd56a - {'path': '/', 'tag': '12f39fd7-897e-4d8a-b5dc-a006d3fbd56a', 'options': 'recursive,force'} 2024-04-25T11:45:24.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:24 smithi138 ceph-mon[32328]: pgmap v319: 97 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 88 active+clean; 31 GiB data, 97 GiB used, 976 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-04-25T11:45:24.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:24 smithi138 ceph-mon[32328]: scrub queued for path: / 2024-04-25T11:45:24.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:24 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:45:24.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:24 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:45:24.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:24 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:45:24.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:24 smithi156 ceph-mon[33045]: pgmap v319: 97 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 88 active+clean; 31 GiB data, 97 GiB used, 976 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-04-25T11:45:24.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:24 smithi156 ceph-mon[33045]: scrub queued for path: / 2024-04-25T11:45:24.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:24 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:45:24.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:24 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:45:24.336 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:24 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:45:24.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:24 smithi090 ceph-mon[29575]: pgmap v319: 97 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 88 active+clean; 31 GiB data, 97 GiB used, 976 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-04-25T11:45:24.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:24 smithi090 ceph-mon[29575]: scrub queued for path: / 2024-04-25T11:45:24.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:24 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:45:24.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:24 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:45:24.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:24 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:45:25.270 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:25 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:45:25.271 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:25 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:45:25.271 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:25 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:45:25.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:25 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:45:25.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:25 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:45:25.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:25 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:45:25.314 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:25 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:45:25.314 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:25 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:45:25.314 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:25 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:45:26.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:26 smithi138 ceph-mon[32328]: pgmap v320: 97 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 88 active+clean; 31 GiB data, 96 GiB used, 977 GiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.8 KiB/s wr, 9 op/s 2024-04-25T11:45:26.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:26 smithi156 ceph-mon[33045]: pgmap v320: 97 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 88 active+clean; 31 GiB data, 96 GiB used, 977 GiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.8 KiB/s wr, 9 op/s 2024-04-25T11:45:26.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:26 smithi090 ceph-mon[29575]: pgmap v320: 97 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 88 active+clean; 31 GiB data, 96 GiB used, 977 GiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.8 KiB/s wr, 9 op/s 2024-04-25T11:45:27.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:45:27.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:45:27.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:45:27.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:45:27.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:45:27.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:45:27.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: pgmap v321: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 11 KiB/s rd, 2.8 KiB/s wr, 15 op/s 2024-04-25T11:45:27.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: Adjusting osd_memory_target on smithi138 to 176.2M 2024-04-25T11:45:27.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: Adjusting osd_memory_target on smithi090 to 176.2M 2024-04-25T11:45:27.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: Unable to set osd_memory_target on smithi138 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-04-25T11:45:27.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: Unable to set osd_memory_target on smithi090 to 184765952: error parsing value: Value '184765952' is below minimum 939524096 2024-04-25T11:45:27.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: Adjusting osd_memory_target on smithi156 to 176.2M 2024-04-25T11:45:27.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: Unable to set osd_memory_target on smithi156 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-25T11:45:27.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:45:27.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:45:27.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:27 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:45:27.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:45:27.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:45:27.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:45:27.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:45:27.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:45:27.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:45:27.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: pgmap v321: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 11 KiB/s rd, 2.8 KiB/s wr, 15 op/s 2024-04-25T11:45:27.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: Adjusting osd_memory_target on smithi138 to 176.2M 2024-04-25T11:45:27.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: Adjusting osd_memory_target on smithi090 to 176.2M 2024-04-25T11:45:27.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: Unable to set osd_memory_target on smithi138 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-04-25T11:45:27.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: Unable to set osd_memory_target on smithi090 to 184765952: error parsing value: Value '184765952' is below minimum 939524096 2024-04-25T11:45:27.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: Adjusting osd_memory_target on smithi156 to 176.2M 2024-04-25T11:45:27.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: Unable to set osd_memory_target on smithi156 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-25T11:45:27.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:45:27.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:45:27.552 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:27 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:45:27.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:45:27.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:45:27.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:45:27.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:45:27.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:45:27.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:45:27.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: pgmap v321: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 11 KiB/s rd, 2.8 KiB/s wr, 15 op/s 2024-04-25T11:45:27.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: Adjusting osd_memory_target on smithi138 to 176.2M 2024-04-25T11:45:27.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: Adjusting osd_memory_target on smithi090 to 176.2M 2024-04-25T11:45:27.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: Unable to set osd_memory_target on smithi138 to 184768819: error parsing value: Value '184768819' is below minimum 939524096 2024-04-25T11:45:27.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: Unable to set osd_memory_target on smithi090 to 184765952: error parsing value: Value '184765952' is below minimum 939524096 2024-04-25T11:45:27.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T11:45:27.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: Adjusting osd_memory_target on smithi156 to 176.2M 2024-04-25T11:45:27.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: Unable to set osd_memory_target on smithi156 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-25T11:45:27.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:45:27.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:45:27.588 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:27 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:45:30.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:29 smithi090 ceph-mon[29575]: pgmap v322: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:45:30.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:29 smithi090 ceph-mon[29575]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T11:45:30.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:29 smithi138 ceph-mon[32328]: pgmap v322: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:45:30.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:29 smithi138 ceph-mon[32328]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T11:45:30.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:29 smithi156 ceph-mon[33045]: pgmap v322: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:45:30.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:29 smithi156 ceph-mon[33045]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T11:45:31.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.131+0000 7fd37c649640 1 -- 172.21.15.90:0/3508737063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd37410e0c0 msgr2=0x7fd3741105b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:31.119 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.131+0000 7fd37c649640 1 --2- 172.21.15.90:0/3508737063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd37410e0c0 0x7fd3741105b0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fd36c009ca0 tx=0x7fd36c02a8c0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.119 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.131+0000 7fd37c649640 1 -- 172.21.15.90:0/3508737063 shutdown_connections 2024-04-25T11:45:31.119 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.131+0000 7fd37c649640 1 --2- 172.21.15.90:0/3508737063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd37410e0c0 0x7fd3741105b0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.119 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.131+0000 7fd37c649640 1 --2- 172.21.15.90:0/3508737063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd37410b6d0 0x7fd37410daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.131+0000 7fd37c649640 1 --2- 172.21.15.90:0/3508737063 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd374104c70 0x7fd374105080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.131+0000 7fd37c649640 1 -- 172.21.15.90:0/3508737063 >> 172.21.15.90:0/3508737063 conn(0x7fd3740781b0 msgr2=0x7fd3740785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:31.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.131+0000 7fd37c649640 1 -- 172.21.15.90:0/3508737063 shutdown_connections 2024-04-25T11:45:31.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.131+0000 7fd37c649640 1 -- 172.21.15.90:0/3508737063 wait complete. 2024-04-25T11:45:31.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.132+0000 7fd37c649640 1 Processor -- start 2024-04-25T11:45:31.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.132+0000 7fd37c649640 1 -- start start 2024-04-25T11:45:31.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.132+0000 7fd37c649640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd374104c70 0x7fd37419a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:31.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.132+0000 7fd37c649640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd37410b6d0 0x7fd37419ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:31.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.132+0000 7fd37c649640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd37410e0c0 0x7fd37419ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:31.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.132+0000 7fd37c649640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd374113d20 con 0x7fd374104c70 2024-04-25T11:45:31.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.132+0000 7fd37c649640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd374113ba0 con 0x7fd37410e0c0 2024-04-25T11:45:31.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.132+0000 7fd37c649640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd374113ea0 con 0x7fd37410b6d0 2024-04-25T11:45:31.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.132+0000 7fd37a3e5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd374104c70 0x7fd37419a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:31.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.132+0000 7fd37a3e5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd374104c70 0x7fd37419a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42362/0 (socket says 172.21.15.90:42362) 2024-04-25T11:45:31.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.132+0000 7fd37a3e5640 1 -- 172.21.15.90:0/3070292541 learned_addr learned my addr 172.21.15.90:0/3070292541 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:31.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.132+0000 7fd37abe6640 1 --2- 172.21.15.90:0/3070292541 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd37410e0c0 0x7fd37419ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:31.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.132+0000 7fd37a3e5640 1 -- 172.21.15.90:0/3070292541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd37410b6d0 msgr2=0x7fd37419ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:31.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.132+0000 7fd37a3e5640 1 --2- 172.21.15.90:0/3070292541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd37410b6d0 0x7fd37419ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.132+0000 7fd37a3e5640 1 -- 172.21.15.90:0/3070292541 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd37410e0c0 msgr2=0x7fd37419ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:31.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.132+0000 7fd37a3e5640 1 --2- 172.21.15.90:0/3070292541 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd37410e0c0 0x7fd37419ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.133+0000 7fd37a3e5640 1 -- 172.21.15.90:0/3070292541 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd37419f540 con 0x7fd374104c70 2024-04-25T11:45:31.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.133+0000 7fd37a3e5640 1 --2- 172.21.15.90:0/3070292541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd374104c70 0x7fd37419a6e0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fd36400d950 tx=0x7fd36400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:31.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.133+0000 7fd36b7fe640 1 -- 172.21.15.90:0/3070292541 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd364017070 con 0x7fd374104c70 2024-04-25T11:45:31.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.133+0000 7fd36b7fe640 1 -- 172.21.15.90:0/3070292541 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd364009c30 con 0x7fd374104c70 2024-04-25T11:45:31.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.133+0000 7fd36b7fe640 1 -- 172.21.15.90:0/3070292541 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd36400f620 con 0x7fd374104c70 2024-04-25T11:45:31.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.133+0000 7fd37c649640 1 -- 172.21.15.90:0/3070292541 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd37419f830 con 0x7fd374104c70 2024-04-25T11:45:31.122 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.133+0000 7fd37c649640 1 -- 172.21.15.90:0/3070292541 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3741a71b0 con 0x7fd374104c70 2024-04-25T11:45:31.128 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.135+0000 7fd36b7fe640 1 -- 172.21.15.90:0/3070292541 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fd364010460 con 0x7fd374104c70 2024-04-25T11:45:31.128 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.136+0000 7fd3835d1740 1 Processor -- start 2024-04-25T11:45:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.136+0000 7fd3835d1740 1 -- start start 2024-04-25T11:45:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.136+0000 7fd3835d1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 0x561c98673980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.136+0000 7fd3835d1740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c982ab2f0 0x561c98673ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.136+0000 7fd3835d1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c982b34d0 0x561c98674400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.136+0000 7fd3835d1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561c985da460 con 0x561c982d41b0 2024-04-25T11:45:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.136+0000 7fd3835d1740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x561c98594590 con 0x561c982ab2f0 2024-04-25T11:45:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.136+0000 7fd3835d1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561c985d6420 con 0x561c982b34d0 2024-04-25T11:45:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.136+0000 7fd37abe6640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 0x561c98673980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.136+0000 7fd37abe6640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 0x561c98673980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42366/0 (socket says 172.21.15.90:42366) 2024-04-25T11:45:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.136+0000 7fd37abe6640 1 -- 172.21.15.90:0/756319009 learned_addr learned my addr 172.21.15.90:0/756319009 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.136+0000 7fd379be4640 1 --2- 172.21.15.90:0/756319009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c982b34d0 0x561c98674400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.136+0000 7fd36b7fe640 1 --2- 172.21.15.90:0/3070292541 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd3600afcd0 0x7fd3600b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.136+0000 7fd37a3e5640 1 --2- 172.21.15.90:0/756319009 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c982ab2f0 0x561c98673ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.137+0000 7fd37abe6640 1 -- 172.21.15.90:0/756319009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c982b34d0 msgr2=0x561c98674400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.137+0000 7fd37abe6640 1 --2- 172.21.15.90:0/756319009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c982b34d0 0x561c98674400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.137+0000 7fd37abe6640 1 -- 172.21.15.90:0/756319009 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c982ab2f0 msgr2=0x561c98673ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.137+0000 7fd37abe6640 1 --2- 172.21.15.90:0/756319009 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c982ab2f0 0x561c98673ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.137+0000 7fd37abe6640 1 -- 172.21.15.90:0/756319009 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c985a10d0 con 0x561c982d41b0 2024-04-25T11:45:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.137+0000 7fd36b7fe640 1 -- 172.21.15.90:0/3070292541 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7fd3640ded50 con 0x7fd374104c70 2024-04-25T11:45:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.137+0000 7fd37a3e5640 1 --2- 172.21.15.90:0/756319009 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c982ab2f0 0x561c98673ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.137+0000 7fd37abe6640 1 --2- 172.21.15.90:0/3070292541 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd3600afcd0 0x7fd3600b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.137+0000 7fd37abe6640 1 --2- 172.21.15.90:0/756319009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 0x561c98673980 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fd36c005e30 tx=0x7fd36c009950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a250f452d86f8b08 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.137+0000 7fd368bfe640 1 -- 172.21.15.90:0/756319009 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd36c003e40 con 0x561c982d41b0 2024-04-25T11:45:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.137+0000 7fd368bfe640 1 -- 172.21.15.90:0/756319009 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd36c036070 con 0x561c982d41b0 2024-04-25T11:45:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.137+0000 7fd368bfe640 1 -- 172.21.15.90:0/756319009 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd36c02faa0 con 0x561c982d41b0 2024-04-25T11:45:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.138+0000 7fd3835d1740 1 -- 172.21.15.90:0/756319009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 msgr2=0x561c98673980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.138+0000 7fd37abe6640 1 --2- 172.21.15.90:0/3070292541 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd3600afcd0 0x7fd3600b21e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x561c98675620 tx=0x7fd36c048000 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.138+0000 7fd3835d1740 1 --2- 172.21.15.90:0/756319009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 0x561c98673980 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fd36c005e30 tx=0x7fd36c009950 comp rx=0 tx=0).stop 2024-04-25T11:45:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.138+0000 7fd3835d1740 1 -- 172.21.15.90:0/756319009 shutdown_connections 2024-04-25T11:45:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.138+0000 7fd3835d1740 1 --2- 172.21.15.90:0/756319009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c982b34d0 0x561c98674400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.138+0000 7fd3835d1740 1 --2- 172.21.15.90:0/756319009 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c982ab2f0 0x561c98673ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.138+0000 7fd3835d1740 1 --2- 172.21.15.90:0/756319009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 0x561c98673980 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.138+0000 7fd3835d1740 1 -- 172.21.15.90:0/756319009 >> 172.21.15.90:0/756319009 conn(0x561c982c8300 msgr2=0x561c986702a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.138+0000 7fd3835d1740 1 -- 172.21.15.90:0/756319009 shutdown_connections 2024-04-25T11:45:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.138+0000 7fd3835d1740 1 -- 172.21.15.90:0/756319009 wait complete. 2024-04-25T11:45:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.139+0000 7fd3835d1740 1 -- start start 2024-04-25T11:45:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.139+0000 7fd3835d1740 1 Processor -- start 2024-04-25T11:45:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.139+0000 7fd3835d1740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c982ab2f0 0x561c98683f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.139+0000 7fd3835d1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c982b34d0 0x561c98684450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:31.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.139+0000 7fd3835d1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 0x561c98682400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.139+0000 7fd3835d1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561c98684f10 con 0x561c982d41b0 2024-04-25T11:45:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.139+0000 7fd3835d1740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x561c98561b60 con 0x561c982ab2f0 2024-04-25T11:45:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.139+0000 7fd3835d1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561c98685090 con 0x561c982b34d0 2024-04-25T11:45:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.139+0000 7fd3835d1740 10 client.0 _ll_register_callbacks cb 0x561c985862b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.139+0000 7fd37abe6640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 0x561c98682400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.139+0000 7fd37abe6640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 0x561c98682400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42376/0 (socket says 172.21.15.90:42376) 2024-04-25T11:45:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.139+0000 7fd37abe6640 1 -- 172.21.15.90:0/921136994 learned_addr learned my addr 172.21.15.90:0/921136994 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.139+0000 7fd379be4640 1 --2- 172.21.15.90:0/921136994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c982b34d0 0x561c98684450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.139+0000 7fd37a3e5640 1 --2- 172.21.15.90:0/921136994 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c982ab2f0 0x561c98683f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.139+0000 7fd37abe6640 1 -- 172.21.15.90:0/921136994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c982b34d0 msgr2=0x561c98684450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.139+0000 7fd37abe6640 1 --2- 172.21.15.90:0/921136994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c982b34d0 0x561c98684450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.139+0000 7fd37abe6640 1 -- 172.21.15.90:0/921136994 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c982ab2f0 msgr2=0x561c98683f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.139+0000 7fd37abe6640 1 --2- 172.21.15.90:0/921136994 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c982ab2f0 0x561c98683f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.139+0000 7fd37abe6640 1 -- 172.21.15.90:0/921136994 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c982ec210 con 0x561c982d41b0 2024-04-25T11:45:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.139+0000 7fd37a3e5640 1 --2- 172.21.15.90:0/921136994 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c982ab2f0 0x561c98683f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.140+0000 7fd37abe6640 1 --2- 172.21.15.90:0/921136994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 0x561c98682400 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fd36c02f6a0 tx=0x7fd36c0095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92e1f33d94cb2a7c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.140+0000 7fd35dffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.140+0000 7fd35dffb640 1 -- 172.21.15.90:0/921136994 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd36c004040 con 0x561c982d41b0 2024-04-25T11:45:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.140+0000 7fd35dffb640 1 -- 172.21.15.90:0/921136994 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7fd36c04d070 con 0x561c982d41b0 2024-04-25T11:45:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.140+0000 7fd3835d1740 1 -- 172.21.15.90:0/921136994 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd36c036070 con 0x561c982d41b0 2024-04-25T11:45:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.140+0000 7fd35dffb640 1 -- 172.21.15.90:0/921136994 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd36c0041e0 con 0x561c982d41b0 2024-04-25T11:45:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.140+0000 7fd3835d1740 10 client.18960 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.140+0000 7fd3835d1740 1 -- 172.21.15.90:0/921136994 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561c983b8620 con 0x561c982d41b0 2024-04-25T11:45:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.140+0000 7fd35dffb640 1 -- 172.21.15.90:0/921136994 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd36c036070 con 0x561c982d41b0 2024-04-25T11:45:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.140+0000 7fd35dffb640 1 -- 172.21.15.90:0/921136994 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fd36c0308f0 con 0x561c982d41b0 2024-04-25T11:45:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.140+0000 7fd3835d1740 10 client.18960 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.140+0000 7fd3835d1740 10 client.18960 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:45:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.140+0000 7fd3835d1740 20 client.18960 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.140+0000 7fd3835d1740 1 --2- 172.21.15.90:0/921136994 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x561c98292d80 0x561c98694810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.140+0000 7fd3835d1740 4 client.18960 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:45:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.140+0000 7fd3835d1740 1 -- 172.21.15.90:0/921136994 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c985e8070 con 0x561c98292d80 2024-04-25T11:45:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.141+0000 7fd37a3e5640 1 --2- 172.21.15.90:0/921136994 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x561c98292d80 0x561c98694810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.141+0000 7fd35dffb640 10 client.18960 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.141+0000 7fd37a3e5640 1 --2- 172.21.15.90:0/921136994 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x561c98292d80 0x561c98694810 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=908df1cc6bc5ea6f server_cookie=ccb4d591b5929306 in_seq=0 out_seq=0 2024-04-25T11:45:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.142+0000 7fd35dffb640 1 -- 172.21.15.90:0/921136994 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561c985e8070 con 0x561c98292d80 2024-04-25T11:45:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.142+0000 7fd35dffb640 10 client.18960 handle_command_reply: tid=0 2024-04-25T11:45:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.142+0000 7fd3835d1740 1 client.18960 shutdown 2024-04-25T11:45:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.142+0000 7fd3835d1740 1 -- 172.21.15.90:0/921136994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 msgr2=0x561c98682400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.142+0000 7fd3835d1740 1 --2- 172.21.15.90:0/921136994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 0x561c98682400 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fd36c02f6a0 tx=0x7fd36c0095a0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.142+0000 7fd3835d1740 1 -- 172.21.15.90:0/921136994 shutdown_connections 2024-04-25T11:45:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.142+0000 7fd3835d1740 1 --2- 172.21.15.90:0/921136994 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x561c98292d80 0x561c98694810 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.142+0000 7fd3835d1740 1 --2- 172.21.15.90:0/921136994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 0x561c98682400 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.142+0000 7fd3835d1740 1 --2- 172.21.15.90:0/921136994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c982b34d0 0x561c98684450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.142+0000 7fd35dffb640 0 client.18960 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.142+0000 7fd3835d1740 1 --2- 172.21.15.90:0/921136994 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c982ab2f0 0x561c98683f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.142+0000 7fd3835d1740 1 -- 172.21.15.90:0/921136994 >> 172.21.15.90:0/921136994 conn(0x561c982c8300 msgr2=0x561c98674330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.142+0000 7fd3835d1740 1 -- 172.21.15.90:0/921136994 shutdown_connections 2024-04-25T11:45:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.142+0000 7fd3835d1740 1 -- 172.21.15.90:0/921136994 wait complete. 2024-04-25T11:45:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.142+0000 7fd3835d1740 20 client.18960 trim_cache size 0 max 16384 2024-04-25T11:45:31.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.157+0000 7fd3835d1740 1 Processor -- start 2024-04-25T11:45:31.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.157+0000 7fd3835d1740 1 -- start start 2024-04-25T11:45:31.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.158+0000 7fd3835d1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c982ab2f0 0x561c986823a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:31.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.158+0000 7fd3835d1740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c982b34d0 0x561c9867bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:31.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.158+0000 7fd3835d1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 0x561c9867c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:31.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.158+0000 7fd3835d1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd3741ada70 con 0x561c982d41b0 2024-04-25T11:45:31.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.158+0000 7fd3835d1740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd3741af5b0 con 0x561c982b34d0 2024-04-25T11:45:31.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.158+0000 7fd3835d1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd3741ae140 con 0x561c982ab2f0 2024-04-25T11:45:31.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.158+0000 7fd37abe6640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 0x561c9867c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:31.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.158+0000 7fd37abe6640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 0x561c9867c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42386/0 (socket says 172.21.15.90:42386) 2024-04-25T11:45:31.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.158+0000 7fd37abe6640 1 -- 172.21.15.90:0/4016478242 learned_addr learned my addr 172.21.15.90:0/4016478242 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:31.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.158+0000 7fd379be4640 1 --2- 172.21.15.90:0/4016478242 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c982b34d0 0x561c9867bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:31.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.158+0000 7fd37abe6640 1 -- 172.21.15.90:0/4016478242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c982ab2f0 msgr2=0x561c986823a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:31.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.158+0000 7fd37abe6640 1 --2- 172.21.15.90:0/4016478242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c982ab2f0 0x561c986823a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.158+0000 7fd37abe6640 1 -- 172.21.15.90:0/4016478242 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c982b34d0 msgr2=0x561c9867bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:31.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.158+0000 7fd37abe6640 1 --2- 172.21.15.90:0/4016478242 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c982b34d0 0x561c9867bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.158+0000 7fd37abe6640 1 -- 172.21.15.90:0/4016478242 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c986828e0 con 0x561c982d41b0 2024-04-25T11:45:31.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.158+0000 7fd37abe6640 1 --2- 172.21.15.90:0/4016478242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 0x561c9867c270 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fd36c02ade0 tx=0x7fd36c02bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=409846a09829d4c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:31.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.158+0000 7fd35ffff640 1 -- 172.21.15.90:0/4016478242 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd36c02f6d0 con 0x561c982d41b0 2024-04-25T11:45:31.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.158+0000 7fd35ffff640 1 -- 172.21.15.90:0/4016478242 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd36c036040 con 0x561c982d41b0 2024-04-25T11:45:31.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.158+0000 7fd35ffff640 1 -- 172.21.15.90:0/4016478242 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd36c030930 con 0x561c982d41b0 2024-04-25T11:45:31.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.159+0000 7fd3835d1740 1 -- 172.21.15.90:0/4016478242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 msgr2=0x561c9867c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.159+0000 7fd3835d1740 1 --2- 172.21.15.90:0/4016478242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 0x561c9867c270 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fd36c02ade0 tx=0x7fd36c02bfd0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.159+0000 7fd3835d1740 1 -- 172.21.15.90:0/4016478242 shutdown_connections 2024-04-25T11:45:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.159+0000 7fd3835d1740 1 --2- 172.21.15.90:0/4016478242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 0x561c9867c270 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.159+0000 7fd3835d1740 1 --2- 172.21.15.90:0/4016478242 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c982b34d0 0x561c9867bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.159+0000 7fd3835d1740 1 --2- 172.21.15.90:0/4016478242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c982ab2f0 0x561c986823a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.159+0000 7fd3835d1740 1 -- 172.21.15.90:0/4016478242 >> 172.21.15.90:0/4016478242 conn(0x561c982c8300 msgr2=0x561c98681c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.159+0000 7fd3835d1740 1 -- 172.21.15.90:0/4016478242 shutdown_connections 2024-04-25T11:45:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.159+0000 7fd3835d1740 1 -- 172.21.15.90:0/4016478242 wait complete. 2024-04-25T11:45:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.159+0000 7fd3835d1740 1 -- start start 2024-04-25T11:45:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.160+0000 7fd3835d1740 1 Processor -- start 2024-04-25T11:45:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.160+0000 7fd3835d1740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c982ab2f0 0x561c986f4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.160+0000 7fd3835d1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c982b34d0 0x561c986f52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.160+0000 7fd3835d1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 0x561c986856b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.160+0000 7fd3835d1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd3741aeee0 con 0x561c982d41b0 2024-04-25T11:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.160+0000 7fd3835d1740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd3741ae810 con 0x561c982ab2f0 2024-04-25T11:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.160+0000 7fd3835d1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd3741aba70 con 0x561c982b34d0 2024-04-25T11:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.160+0000 7fd3835d1740 10 client.0 _ll_register_callbacks cb 0x7fd36c04a020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.160+0000 7fd37abe6640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 0x561c986856b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.160+0000 7fd37abe6640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 0x561c986856b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42394/0 (socket says 172.21.15.90:42394) 2024-04-25T11:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.160+0000 7fd37abe6640 1 -- 172.21.15.90:0/98924876 learned_addr learned my addr 172.21.15.90:0/98924876 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.160+0000 7fd379be4640 1 --2- 172.21.15.90:0/98924876 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c982b34d0 0x561c986f52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.160+0000 7fd37a3e5640 1 --2- 172.21.15.90:0/98924876 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c982ab2f0 0x561c986f4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.160+0000 7fd37abe6640 1 -- 172.21.15.90:0/98924876 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c982b34d0 msgr2=0x561c986f52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.160+0000 7fd37abe6640 1 --2- 172.21.15.90:0/98924876 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c982b34d0 0x561c986f52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.160+0000 7fd37abe6640 1 -- 172.21.15.90:0/98924876 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c982ab2f0 msgr2=0x561c986f4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.160+0000 7fd37abe6640 1 --2- 172.21.15.90:0/98924876 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c982ab2f0 0x561c986f4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.160+0000 7fd37abe6640 1 -- 172.21.15.90:0/98924876 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c98685ca0 con 0x561c982d41b0 2024-04-25T11:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.160+0000 7fd37a3e5640 1 --2- 172.21.15.90:0/98924876 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c982ab2f0 0x561c986f4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.160+0000 7fd37abe6640 1 --2- 172.21.15.90:0/98924876 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 0x561c986856b0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fd36c006010 tx=0x7fd36c030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3c830a2c8c4f241 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.160+0000 7fd35dffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.161+0000 7fd35dffb640 1 -- 172.21.15.90:0/98924876 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd36c070020 con 0x561c982d41b0 2024-04-25T11:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.161+0000 7fd35dffb640 1 -- 172.21.15.90:0/98924876 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7fd36c04d040 con 0x561c982d41b0 2024-04-25T11:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.161+0000 7fd3835d1740 1 -- 172.21.15.90:0/98924876 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd36c036040 con 0x561c982d41b0 2024-04-25T11:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.161+0000 7fd35dffb640 1 -- 172.21.15.90:0/98924876 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd36c07b050 con 0x561c982d41b0 2024-04-25T11:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.161+0000 7fd3835d1740 10 client.18966 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.161+0000 7fd3835d1740 1 -- 172.21.15.90:0/98924876 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561c986fc3d0 con 0x561c982d41b0 2024-04-25T11:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.161+0000 7fd35dffb640 1 -- 172.21.15.90:0/98924876 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd36c036040 con 0x561c982d41b0 2024-04-25T11:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.161+0000 7fd35dffb640 1 -- 172.21.15.90:0/98924876 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fd36c06c930 con 0x561c982d41b0 2024-04-25T11:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.161+0000 7fd3835d1740 10 client.18966 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.161+0000 7fd3835d1740 10 client.18966 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:45:31.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.161+0000 7fd3835d1740 20 client.18966 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:31.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.161+0000 7fd3835d1740 1 --2- 172.21.15.90:0/98924876 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x561c982681c0 0x561c986feb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:31.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.161+0000 7fd3835d1740 4 client.18966 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:45:31.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.161+0000 7fd3835d1740 1 -- 172.21.15.90:0/98924876 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561c985e1ee0 con 0x561c982681c0 2024-04-25T11:45:31.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.161+0000 7fd37a3e5640 1 --2- 172.21.15.90:0/98924876 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x561c982681c0 0x561c986feb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:31.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.162+0000 7fd35dffb640 10 client.18966 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:31.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.162+0000 7fd37a3e5640 1 --2- 172.21.15.90:0/98924876 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x561c982681c0 0x561c986feb20 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53c255b028073163 server_cookie=1612019998096f0 in_seq=0 out_seq=0 2024-04-25T11:45:31.155 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.169+0000 7fd35dffb640 1 -- 172.21.15.90:0/98924876 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561c985e1ee0 con 0x561c982681c0 2024-04-25T11:45:31.155 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.169+0000 7fd35dffb640 10 client.18966 handle_command_reply: tid=0 2024-04-25T11:45:31.155 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.169+0000 7fd3835d1740 1 client.18966 shutdown 2024-04-25T11:45:31.155 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.169+0000 7fd3835d1740 1 -- 172.21.15.90:0/98924876 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 msgr2=0x561c986856b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:31.155 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.169+0000 7fd3835d1740 1 --2- 172.21.15.90:0/98924876 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 0x561c986856b0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fd36c006010 tx=0x7fd36c030000 comp rx=0 tx=0).stop 2024-04-25T11:45:31.155 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.169+0000 7fd3835d1740 1 -- 172.21.15.90:0/98924876 shutdown_connections 2024-04-25T11:45:31.155 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.169+0000 7fd3835d1740 1 --2- 172.21.15.90:0/98924876 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x561c982681c0 0x561c986feb20 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.155 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.169+0000 7fd3835d1740 1 --2- 172.21.15.90:0/98924876 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561c982d41b0 0x561c986856b0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.169+0000 7fd35dffb640 0 client.18966 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:31.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.169+0000 7fd3835d1740 1 --2- 172.21.15.90:0/98924876 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c982b34d0 0x561c986f52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.169+0000 7fd3835d1740 1 --2- 172.21.15.90:0/98924876 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561c982ab2f0 0x561c986f4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.169+0000 7fd3835d1740 1 -- 172.21.15.90:0/98924876 >> 172.21.15.90:0/98924876 conn(0x561c982c8300 msgr2=0x561c98673880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:31.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.169+0000 7fd3835d1740 1 -- 172.21.15.90:0/98924876 shutdown_connections 2024-04-25T11:45:31.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.169+0000 7fd3835d1740 1 -- 172.21.15.90:0/98924876 wait complete. 2024-04-25T11:45:31.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.169+0000 7fd3835d1740 20 client.18966 trim_cache size 0 max 16384 2024-04-25T11:45:31.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:45:31.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:45:31.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:45:31.157 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:45:31.157 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.170+0000 7fd37c649640 1 -- 172.21.15.90:0/3070292541 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd3600afcd0 msgr2=0x7fd3600b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:31.157 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.170+0000 7fd37c649640 1 --2- 172.21.15.90:0/3070292541 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd3600afcd0 0x7fd3600b21e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x561c98675620 tx=0x7fd36c048000 comp rx=0 tx=0).stop 2024-04-25T11:45:31.158 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.170+0000 7fd37c649640 1 -- 172.21.15.90:0/3070292541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd374104c70 msgr2=0x7fd37419a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:31.158 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.170+0000 7fd37c649640 1 --2- 172.21.15.90:0/3070292541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd374104c70 0x7fd37419a6e0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fd36400d950 tx=0x7fd36400de10 comp rx=0 tx=0).stop 2024-04-25T11:45:31.158 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.170+0000 7fd37c649640 1 -- 172.21.15.90:0/3070292541 shutdown_connections 2024-04-25T11:45:31.158 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.170+0000 7fd37c649640 1 --2- 172.21.15.90:0/3070292541 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fd3600afcd0 0x7fd3600b21e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.158 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.170+0000 7fd37c649640 1 --2- 172.21.15.90:0/3070292541 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd37410e0c0 0x7fd37419ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.158 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.170+0000 7fd37c649640 1 --2- 172.21.15.90:0/3070292541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd37410b6d0 0x7fd37419ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.158 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.170+0000 7fd37c649640 1 --2- 172.21.15.90:0/3070292541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd374104c70 0x7fd37419a6e0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:31.158 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.170+0000 7fd37c649640 1 -- 172.21.15.90:0/3070292541 >> 172.21.15.90:0/3070292541 conn(0x7fd3740781b0 msgr2=0x7fd374101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:31.158 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.170+0000 7fd37c649640 1 -- 172.21.15.90:0/3070292541 shutdown_connections 2024-04-25T11:45:31.158 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:31.170+0000 7fd37c649640 1 -- 172.21.15.90:0/3070292541 wait complete. 2024-04-25T11:45:31.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:30 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:45:31.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:30 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:45:31.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:30 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:45:32.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:31 smithi090 ceph-mon[29575]: pgmap v324: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 8.6 KiB/s rd, 13 op/s 2024-04-25T11:45:32.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:31 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:45:32.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:31 smithi138 ceph-mon[32328]: pgmap v324: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 8.6 KiB/s rd, 13 op/s 2024-04-25T11:45:32.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:31 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:45:32.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:31 smithi156 ceph-mon[33045]: pgmap v324: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 8.6 KiB/s rd, 13 op/s 2024-04-25T11:45:32.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:31 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:45:34.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:33 smithi090 ceph-mon[29575]: pgmap v325: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 8.6 KiB/s rd, 13 op/s 2024-04-25T11:45:34.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:33 smithi138 ceph-mon[32328]: pgmap v325: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 8.6 KiB/s rd, 13 op/s 2024-04-25T11:45:34.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:33 smithi156 ceph-mon[33045]: pgmap v325: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 8.6 KiB/s rd, 13 op/s 2024-04-25T11:45:36.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:35 smithi090 ceph-mon[29575]: pgmap v326: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 7.9 KiB/s rd, 12 op/s 2024-04-25T11:45:36.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:35 smithi138 ceph-mon[32328]: pgmap v326: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 7.9 KiB/s rd, 12 op/s 2024-04-25T11:45:36.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:35 smithi156 ceph-mon[33045]: pgmap v326: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 7.9 KiB/s rd, 12 op/s 2024-04-25T11:45:38.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:37 smithi090 ceph-mon[29575]: pgmap v327: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:45:38.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:37 smithi138 ceph-mon[32328]: pgmap v327: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:45:38.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:37 smithi156 ceph-mon[33045]: pgmap v327: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:45:40.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:39 smithi090 ceph-mon[29575]: pgmap v328: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 8.0 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:45:40.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:39 smithi138 ceph-mon[32328]: pgmap v328: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 8.0 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:45:40.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:39 smithi156 ceph-mon[33045]: pgmap v328: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 8.0 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:45:42.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:41 smithi090 ceph-mon[29575]: pgmap v329: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 9.1 KiB/s rd, 2.8 KiB/s wr, 13 op/s 2024-04-25T11:45:42.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:41 smithi138 ceph-mon[32328]: pgmap v329: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 9.1 KiB/s rd, 2.8 KiB/s wr, 13 op/s 2024-04-25T11:45:42.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:41 smithi156 ceph-mon[33045]: pgmap v329: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 9.1 KiB/s rd, 2.8 KiB/s wr, 13 op/s 2024-04-25T11:45:44.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:43 smithi090 ceph-mon[29575]: pgmap v330: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:45:44.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.278+0000 7fadb08bd640 1 -- 172.21.15.90:0/4154760053 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fadac108a60 msgr2=0x7fadac102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:44.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.278+0000 7fadb08bd640 1 --2- 172.21.15.90:0/4154760053 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fadac108a60 0x7fadac102610 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fad9c009ca0 tx=0x7fad9c02a8e0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.278+0000 7fadb08bd640 1 -- 172.21.15.90:0/4154760053 shutdown_connections 2024-04-25T11:45:44.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.278+0000 7fadb08bd640 1 --2- 172.21.15.90:0/4154760053 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fadac1031c0 0x7fadac10f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.278+0000 7fadb08bd640 1 --2- 172.21.15.90:0/4154760053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fadac108f00 0x7fadac102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.278+0000 7fadb08bd640 1 --2- 172.21.15.90:0/4154760053 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fadac108a60 0x7fadac102610 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.278+0000 7fadb08bd640 1 -- 172.21.15.90:0/4154760053 >> 172.21.15.90:0/4154760053 conn(0x7fadac0781b0 msgr2=0x7fadac0785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:44.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.278+0000 7fadb08bd640 1 -- 172.21.15.90:0/4154760053 shutdown_connections 2024-04-25T11:45:44.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.278+0000 7fadb08bd640 1 -- 172.21.15.90:0/4154760053 wait complete. 2024-04-25T11:45:44.279 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.279+0000 7fadb08bd640 1 Processor -- start 2024-04-25T11:45:44.279 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.279+0000 7fadb08bd640 1 -- start start 2024-04-25T11:45:44.279 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.279+0000 7fadb08bd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fadac1031c0 0x7fadac0ff250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:44.279 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.279+0000 7fadb08bd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fadac108a60 0x7fadac0ff790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:44.279 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.279+0000 7fadb08bd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fadac108f00 0x7fadac0ffcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:44.279 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.279+0000 7fadb08bd640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fadac112d40 con 0x7fadac1031c0 2024-04-25T11:45:44.279 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.279+0000 7fadb08bd640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fadac112bc0 con 0x7fadac108a60 2024-04-25T11:45:44.279 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.279+0000 7fadb08bd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fadac112ec0 con 0x7fadac108f00 2024-04-25T11:45:44.279 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.279+0000 7fadaa59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fadac1031c0 0x7fadac0ff250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.279+0000 7fadaa59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fadac1031c0 0x7fadac0ff250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33592/0 (socket says 172.21.15.90:33592) 2024-04-25T11:45:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.279+0000 7fadaa59c640 1 -- 172.21.15.90:0/3103706044 learned_addr learned my addr 172.21.15.90:0/3103706044 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.279+0000 7fada9d9b640 1 --2- 172.21.15.90:0/3103706044 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fadac108a60 0x7fadac0ff790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.280+0000 7fadaad9d640 1 --2- 172.21.15.90:0/3103706044 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fadac108f00 0x7fadac0ffcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.280+0000 7fadaa59c640 1 -- 172.21.15.90:0/3103706044 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fadac108f00 msgr2=0x7fadac0ffcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.280+0000 7fadaa59c640 1 --2- 172.21.15.90:0/3103706044 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fadac108f00 0x7fadac0ffcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.280+0000 7fadaa59c640 1 -- 172.21.15.90:0/3103706044 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fadac108a60 msgr2=0x7fadac0ff790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.280+0000 7fadaa59c640 1 --2- 172.21.15.90:0/3103706044 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fadac108a60 0x7fadac0ff790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.280+0000 7fadaa59c640 1 -- 172.21.15.90:0/3103706044 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadac1a8bf0 con 0x7fadac1031c0 2024-04-25T11:45:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.280+0000 7fadaad9d640 1 --2- 172.21.15.90:0/3103706044 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fadac108f00 0x7fadac0ffcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.280+0000 7fadaa59c640 1 --2- 172.21.15.90:0/3103706044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fadac1031c0 0x7fadac0ff250 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fad9c031ec0 tx=0x7fad9c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.280+0000 7fad9b7fe640 1 -- 172.21.15.90:0/3103706044 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad9c004040 con 0x7fadac1031c0 2024-04-25T11:45:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.280+0000 7fad9b7fe640 1 -- 172.21.15.90:0/3103706044 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad9c004690 con 0x7fadac1031c0 2024-04-25T11:45:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.280+0000 7fadb08bd640 1 -- 172.21.15.90:0/3103706044 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadac1a8e80 con 0x7fadac1031c0 2024-04-25T11:45:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.280+0000 7fad9b7fe640 1 -- 172.21.15.90:0/3103706044 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad9c030e20 con 0x7fadac1031c0 2024-04-25T11:45:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.280+0000 7fadb08bd640 1 -- 172.21.15.90:0/3103706044 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadac1a92e0 con 0x7fadac1031c0 2024-04-25T11:45:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.283+0000 7fadb7775740 1 Processor -- start 2024-04-25T11:45:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.283+0000 7fad9b7fe640 1 -- 172.21.15.90:0/3103706044 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fad9c0305e0 con 0x7fadac1031c0 2024-04-25T11:45:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.283+0000 7fadb7775740 1 -- start start 2024-04-25T11:45:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.283+0000 7fadb7775740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616b9ba61b0 0x5616b9f45af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.284+0000 7fadb7775740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5616b9b7d2f0 0x5616b9f46030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.284+0000 7fadb7775740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9b854d0 0x5616b9f46570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.284+0000 7fadb7775740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5616b9eac460 con 0x5616b9b854d0 2024-04-25T11:45:44.282 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.284+0000 7fadb7775740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5616b9e66590 con 0x5616b9b7d2f0 2024-04-25T11:45:44.282 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.284+0000 7fadb7775740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5616b9ea8420 con 0x5616b9ba61b0 2024-04-25T11:45:44.282 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.284+0000 7fada9d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9b854d0 0x5616b9f46570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:44.282 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.284+0000 7fada9d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9b854d0 0x5616b9f46570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33598/0 (socket says 172.21.15.90:33598) 2024-04-25T11:45:44.282 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.284+0000 7fada9d9b640 1 -- 172.21.15.90:0/570051527 learned_addr learned my addr 172.21.15.90:0/570051527 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:44.282 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.284+0000 7fadaad9d640 1 --2- 172.21.15.90:0/570051527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616b9ba61b0 0x5616b9f45af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:44.282 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.284+0000 7fadaa59c640 1 --2- 172.21.15.90:0/570051527 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5616b9b7d2f0 0x5616b9f46030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:44.282 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.284+0000 7fad9b7fe640 1 --2- 172.21.15.90:0/3103706044 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fad940afc50 0x7fad940b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:44.282 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.284+0000 7fada9d9b640 1 -- 172.21.15.90:0/570051527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616b9ba61b0 msgr2=0x5616b9f45af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:44.282 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.284+0000 7fada9d9b640 1 --2- 172.21.15.90:0/570051527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616b9ba61b0 0x5616b9f45af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.282 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.284+0000 7fada9d9b640 1 -- 172.21.15.90:0/570051527 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5616b9b7d2f0 msgr2=0x5616b9f46030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.284+0000 7fada9d9b640 1 --2- 172.21.15.90:0/570051527 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5616b9b7d2f0 0x5616b9f46030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.284+0000 7fada9d9b640 1 -- 172.21.15.90:0/570051527 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616b9e730d0 con 0x5616b9b854d0 2024-04-25T11:45:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.284+0000 7fadaa59c640 1 --2- 172.21.15.90:0/570051527 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5616b9b7d2f0 0x5616b9f46030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.284+0000 7fadaad9d640 1 --2- 172.21.15.90:0/570051527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616b9ba61b0 0x5616b9f45af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.284+0000 7fadaad9d640 1 --2- 172.21.15.90:0/3103706044 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fad940afc50 0x7fad940b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.284+0000 7fad9b7fe640 1 -- 172.21.15.90:0/3103706044 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7fad9c1004b0 con 0x7fadac1031c0 2024-04-25T11:45:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.284+0000 7fada9d9b640 1 --2- 172.21.15.90:0/570051527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9b854d0 0x5616b9f46570 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fada00099d0 tx=0x7fada0009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac9bed18af68d0d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.284+0000 7fad98ff9640 1 -- 172.21.15.90:0/570051527 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada0017070 con 0x5616b9b854d0 2024-04-25T11:45:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.285+0000 7fad98ff9640 1 -- 172.21.15.90:0/570051527 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fada0013470 con 0x5616b9b854d0 2024-04-25T11:45:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.285+0000 7fad98ff9640 1 -- 172.21.15.90:0/570051527 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada0012610 con 0x5616b9b854d0 2024-04-25T11:45:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.285+0000 7fadb7775740 1 -- 172.21.15.90:0/570051527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9b854d0 msgr2=0x5616b9f46570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.285+0000 7fadaad9d640 1 --2- 172.21.15.90:0/3103706044 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fad940afc50 0x7fad940b2160 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x5616b9f42c60 tx=0x7fada40047d0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.285+0000 7fadb7775740 1 --2- 172.21.15.90:0/570051527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9b854d0 0x5616b9f46570 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fada00099d0 tx=0x7fada0009e90 comp rx=0 tx=0).stop 2024-04-25T11:45:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.285+0000 7fadb7775740 1 -- 172.21.15.90:0/570051527 shutdown_connections 2024-04-25T11:45:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.285+0000 7fadb7775740 1 --2- 172.21.15.90:0/570051527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9b854d0 0x5616b9f46570 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.285+0000 7fadb7775740 1 --2- 172.21.15.90:0/570051527 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5616b9b7d2f0 0x5616b9f46030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.285+0000 7fadb7775740 1 --2- 172.21.15.90:0/570051527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616b9ba61b0 0x5616b9f45af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.286+0000 7fadb7775740 1 -- 172.21.15.90:0/570051527 >> 172.21.15.90:0/570051527 conn(0x5616b9b9a300 msgr2=0x5616b9f422a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.286+0000 7fadb7775740 1 -- 172.21.15.90:0/570051527 shutdown_connections 2024-04-25T11:45:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.286+0000 7fadb7775740 1 -- 172.21.15.90:0/570051527 wait complete. 2024-04-25T11:45:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.286+0000 7fadb7775740 1 -- start start 2024-04-25T11:45:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.286+0000 7fadb7775740 1 Processor -- start 2024-04-25T11:45:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.286+0000 7fadb7775740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9b7d2f0 0x5616b9f50140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.286+0000 7fadb7775740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616b9b854d0 0x5616b9f50680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.286+0000 7fadb7775740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5616b9ba61b0 0x5616b9f560a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.286+0000 7fadb7775740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5616b9f52c20 con 0x5616b9b7d2f0 2024-04-25T11:45:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.286+0000 7fadb7775740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5616b9e33b60 con 0x5616b9ba61b0 2024-04-25T11:45:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.286+0000 7fadb7775740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5616b9f52da0 con 0x5616b9b854d0 2024-04-25T11:45:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.286+0000 7fadb7775740 10 client.0 _ll_register_callbacks cb 0x5616b9e582b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.286+0000 7fadaa59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9b7d2f0 0x5616b9f50140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.287+0000 7fadaa59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9b7d2f0 0x5616b9f50140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33612/0 (socket says 172.21.15.90:33612) 2024-04-25T11:45:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.287+0000 7fadaa59c640 1 -- 172.21.15.90:0/4184541696 learned_addr learned my addr 172.21.15.90:0/4184541696 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.287+0000 7fada9d9b640 1 --2- 172.21.15.90:0/4184541696 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616b9b854d0 0x5616b9f50680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.287+0000 7fadaad9d640 1 --2- 172.21.15.90:0/4184541696 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5616b9ba61b0 0x5616b9f560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.287+0000 7fadaa59c640 1 -- 172.21.15.90:0/4184541696 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616b9b854d0 msgr2=0x5616b9f50680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.287+0000 7fadaa59c640 1 --2- 172.21.15.90:0/4184541696 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616b9b854d0 0x5616b9f50680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.287+0000 7fadaa59c640 1 -- 172.21.15.90:0/4184541696 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5616b9ba61b0 msgr2=0x5616b9f560a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.287+0000 7fadaa59c640 1 --2- 172.21.15.90:0/4184541696 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5616b9ba61b0 0x5616b9f560a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.287+0000 7fadaa59c640 1 -- 172.21.15.90:0/4184541696 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616b9bbe210 con 0x5616b9b7d2f0 2024-04-25T11:45:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.287+0000 7fadaa59c640 1 --2- 172.21.15.90:0/4184541696 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9b7d2f0 0x5616b9f50140 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fad9c0c8600 tx=0x7fad9c0c3ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=530f9768fcfddb26 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.287+0000 7fad91ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.287+0000 7fad91ffb640 1 -- 172.21.15.90:0/4184541696 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad9c0c7fc0 con 0x5616b9b7d2f0 2024-04-25T11:45:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.287+0000 7fad91ffb640 1 -- 172.21.15.90:0/4184541696 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7fad9c0ccc80 con 0x5616b9b7d2f0 2024-04-25T11:45:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.287+0000 7fadb7775740 1 -- 172.21.15.90:0/4184541696 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fada0013470 con 0x5616b9b7d2f0 2024-04-25T11:45:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.287+0000 7fad91ffb640 1 -- 172.21.15.90:0/4184541696 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fad9c0c7de0 con 0x5616b9b7d2f0 2024-04-25T11:45:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.287+0000 7fadb7775740 10 client.18978 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.287+0000 7fadb7775740 1 -- 172.21.15.90:0/4184541696 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5616b9c8a620 con 0x5616b9b7d2f0 2024-04-25T11:45:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.287+0000 7fad91ffb640 1 -- 172.21.15.90:0/4184541696 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fada0013470 con 0x5616b9b7d2f0 2024-04-25T11:45:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.288+0000 7fad91ffb640 1 -- 172.21.15.90:0/4184541696 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fad9c0d8020 con 0x5616b9b7d2f0 2024-04-25T11:45:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.288+0000 7fadb7775740 10 client.18978 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.288+0000 7fadb7775740 10 client.18978 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:45:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.288+0000 7fadb7775740 20 client.18978 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.288+0000 7fadb7775740 1 --2- 172.21.15.90:0/4184541696 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5616b9b64d80 0x5616b9f626d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.288+0000 7fadb7775740 4 client.18978 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:45:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.288+0000 7fadb7775740 1 -- 172.21.15.90:0/4184541696 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616b9eba070 con 0x5616b9b64d80 2024-04-25T11:45:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.288+0000 7fada9d9b640 1 --2- 172.21.15.90:0/4184541696 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5616b9b64d80 0x5616b9f626d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.289+0000 7fad91ffb640 10 client.18978 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.289+0000 7fada9d9b640 1 --2- 172.21.15.90:0/4184541696 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5616b9b64d80 0x5616b9f626d0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c54744fb7ee9fe0c server_cookie=208680d53acee13b in_seq=0 out_seq=0 2024-04-25T11:45:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.289+0000 7fad91ffb640 1 -- 172.21.15.90:0/4184541696 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5616b9eba070 con 0x5616b9b64d80 2024-04-25T11:45:44.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.289+0000 7fad91ffb640 10 client.18978 handle_command_reply: tid=0 2024-04-25T11:45:44.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.289+0000 7fadb7775740 1 client.18978 shutdown 2024-04-25T11:45:44.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.290+0000 7fadb7775740 1 -- 172.21.15.90:0/4184541696 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9b7d2f0 msgr2=0x5616b9f50140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:44.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.290+0000 7fadb7775740 1 --2- 172.21.15.90:0/4184541696 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9b7d2f0 0x5616b9f50140 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fad9c0c8600 tx=0x7fad9c0c3ed0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.290+0000 7fadb7775740 1 -- 172.21.15.90:0/4184541696 shutdown_connections 2024-04-25T11:45:44.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.290+0000 7fadb7775740 1 --2- 172.21.15.90:0/4184541696 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5616b9b64d80 0x5616b9f626d0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.290+0000 7fadb7775740 1 --2- 172.21.15.90:0/4184541696 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5616b9ba61b0 0x5616b9f560a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.290+0000 7fad91ffb640 0 client.18978 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:44.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.290+0000 7fadb7775740 1 --2- 172.21.15.90:0/4184541696 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616b9b854d0 0x5616b9f50680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.290+0000 7fadb7775740 1 --2- 172.21.15.90:0/4184541696 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9b7d2f0 0x5616b9f50140 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.290+0000 7fadb7775740 1 -- 172.21.15.90:0/4184541696 >> 172.21.15.90:0/4184541696 conn(0x5616b9b9a300 msgr2=0x5616b9f47810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:44.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.290+0000 7fadb7775740 1 -- 172.21.15.90:0/4184541696 shutdown_connections 2024-04-25T11:45:44.289 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.290+0000 7fadb7775740 1 -- 172.21.15.90:0/4184541696 wait complete. 2024-04-25T11:45:44.289 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.290+0000 7fadb7775740 20 client.18978 trim_cache size 0 max 16384 2024-04-25T11:45:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.305+0000 7fadb7775740 1 Processor -- start 2024-04-25T11:45:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.305+0000 7fadb7775740 1 -- start start 2024-04-25T11:45:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.305+0000 7fadb7775740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5616b9b7d2f0 0x5616b9f46950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.305+0000 7fadb7775740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616b9b854d0 0x5616b9f51d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.305+0000 7fadb7775740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9ba61b0 0x5616b9f49850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.305+0000 7fadb7775740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fadac1b1e60 con 0x5616b9ba61b0 2024-04-25T11:45:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.305+0000 7fadb7775740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fadac1b39a0 con 0x5616b9b7d2f0 2024-04-25T11:45:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.305+0000 7fadb7775740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fadac1b2530 con 0x5616b9b854d0 2024-04-25T11:45:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.305+0000 7fadaad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9ba61b0 0x5616b9f49850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.305+0000 7fadaad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9ba61b0 0x5616b9f49850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33624/0 (socket says 172.21.15.90:33624) 2024-04-25T11:45:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.305+0000 7fadaad9d640 1 -- 172.21.15.90:0/3054171643 learned_addr learned my addr 172.21.15.90:0/3054171643 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.306+0000 7fada9d9b640 1 --2- 172.21.15.90:0/3054171643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616b9b854d0 0x5616b9f51d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.306+0000 7fadaa59c640 1 --2- 172.21.15.90:0/3054171643 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5616b9b7d2f0 0x5616b9f46950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.306+0000 7fadaad9d640 1 -- 172.21.15.90:0/3054171643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616b9b854d0 msgr2=0x5616b9f51d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.306+0000 7fadaad9d640 1 --2- 172.21.15.90:0/3054171643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616b9b854d0 0x5616b9f51d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.306+0000 7fadaad9d640 1 -- 172.21.15.90:0/3054171643 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5616b9b7d2f0 msgr2=0x5616b9f46950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.306+0000 7fadaad9d640 1 --2- 172.21.15.90:0/3054171643 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5616b9b7d2f0 0x5616b9f46950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.306+0000 7fadaad9d640 1 -- 172.21.15.90:0/3054171643 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616b9f51410 con 0x5616b9ba61b0 2024-04-25T11:45:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.306+0000 7fadaa59c640 1 --2- 172.21.15.90:0/3054171643 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5616b9b7d2f0 0x5616b9f46950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.306+0000 7fadaad9d640 1 --2- 172.21.15.90:0/3054171643 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9ba61b0 0x5616b9f49850 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fada400fc90 tx=0x7fada4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4dd7501bae6f15b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.306+0000 7fad93fff640 1 -- 172.21.15.90:0/3054171643 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada401a030 con 0x5616b9ba61b0 2024-04-25T11:45:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.306+0000 7fad93fff640 1 -- 172.21.15.90:0/3054171643 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fada4025070 con 0x5616b9ba61b0 2024-04-25T11:45:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.306+0000 7fad93fff640 1 -- 172.21.15.90:0/3054171643 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada400f040 con 0x5616b9ba61b0 2024-04-25T11:45:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.307+0000 7fadb7775740 1 -- 172.21.15.90:0/3054171643 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9ba61b0 msgr2=0x5616b9f49850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.307+0000 7fadb7775740 1 --2- 172.21.15.90:0/3054171643 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9ba61b0 0x5616b9f49850 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fada400fc90 tx=0x7fada4012000 comp rx=0 tx=0).stop 2024-04-25T11:45:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.307+0000 7fadb7775740 1 -- 172.21.15.90:0/3054171643 shutdown_connections 2024-04-25T11:45:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.307+0000 7fadb7775740 1 --2- 172.21.15.90:0/3054171643 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9ba61b0 0x5616b9f49850 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.307+0000 7fadb7775740 1 --2- 172.21.15.90:0/3054171643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616b9b854d0 0x5616b9f51d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.307+0000 7fadb7775740 1 --2- 172.21.15.90:0/3054171643 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5616b9b7d2f0 0x5616b9f46950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.307+0000 7fadb7775740 1 -- 172.21.15.90:0/3054171643 >> 172.21.15.90:0/3054171643 conn(0x5616b9b9a300 msgr2=0x5616b9f51670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.307+0000 7fadb7775740 1 -- 172.21.15.90:0/3054171643 shutdown_connections 2024-04-25T11:45:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.307+0000 7fadb7775740 1 -- 172.21.15.90:0/3054171643 wait complete. 2024-04-25T11:45:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.307+0000 7fadb7775740 1 -- start start 2024-04-25T11:45:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.307+0000 7fadb7775740 1 Processor -- start 2024-04-25T11:45:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.307+0000 7fadb7775740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5616b9b7d2f0 0x5616b9f57750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.307+0000 7fadb7775740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9b854d0 0x5616b9f57c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.307+0000 7fadb7775740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616b9ba61b0 0x5616b9fc8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.307+0000 7fadb7775740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fadac1b32d0 con 0x5616b9b854d0 2024-04-25T11:45:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.307+0000 7fadb7775740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fadac1b2c00 con 0x5616b9b7d2f0 2024-04-25T11:45:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.307+0000 7fadb7775740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fadac1afe60 con 0x5616b9ba61b0 2024-04-25T11:45:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.308+0000 7fadb7775740 10 client.0 _ll_register_callbacks cb 0x7fad9c030fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.308+0000 7fada9d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9b854d0 0x5616b9f57c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.308+0000 7fada9d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9b854d0 0x5616b9f57c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33634/0 (socket says 172.21.15.90:33634) 2024-04-25T11:45:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.308+0000 7fada9d9b640 1 -- 172.21.15.90:0/4216060868 learned_addr learned my addr 172.21.15.90:0/4216060868 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.308+0000 7fadaa59c640 1 --2- 172.21.15.90:0/4216060868 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5616b9b7d2f0 0x5616b9f57750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.308+0000 7fadaad9d640 1 --2- 172.21.15.90:0/4216060868 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616b9ba61b0 0x5616b9fc8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.308+0000 7fada9d9b640 1 -- 172.21.15.90:0/4216060868 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616b9ba61b0 msgr2=0x5616b9fc8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.308+0000 7fada9d9b640 1 --2- 172.21.15.90:0/4216060868 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616b9ba61b0 0x5616b9fc8a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.308+0000 7fada9d9b640 1 -- 172.21.15.90:0/4216060868 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5616b9b7d2f0 msgr2=0x5616b9f57750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.308+0000 7fada9d9b640 1 --2- 172.21.15.90:0/4216060868 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5616b9b7d2f0 0x5616b9f57750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.308+0000 7fada9d9b640 1 -- 172.21.15.90:0/4216060868 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616b9fc8fb0 con 0x5616b9b854d0 2024-04-25T11:45:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.308+0000 7fada9d9b640 1 --2- 172.21.15.90:0/4216060868 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9b854d0 0x5616b9f57c90 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fada00099d0 tx=0x7fada0013dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7a57d0024cfdb27 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.308+0000 7fad91ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.308+0000 7fad91ffb640 1 -- 172.21.15.90:0/4216060868 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada0017040 con 0x5616b9b854d0 2024-04-25T11:45:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.308+0000 7fad91ffb640 1 -- 172.21.15.90:0/4216060868 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7fada00275d0 con 0x5616b9b854d0 2024-04-25T11:45:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.308+0000 7fadb7775740 1 -- 172.21.15.90:0/4216060868 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fada4025070 con 0x5616b9b854d0 2024-04-25T11:45:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.308+0000 7fad91ffb640 1 -- 172.21.15.90:0/4216060868 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fada0033050 con 0x5616b9b854d0 2024-04-25T11:45:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.308+0000 7fadb7775740 10 client.18984 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.308+0000 7fadb7775740 1 -- 172.21.15.90:0/4216060868 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5616b9fca110 con 0x5616b9b854d0 2024-04-25T11:45:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.309+0000 7fad91ffb640 1 -- 172.21.15.90:0/4216060868 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fada4025070 con 0x5616b9b854d0 2024-04-25T11:45:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.309+0000 7fad91ffb640 1 -- 172.21.15.90:0/4216060868 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fada001c450 con 0x5616b9b854d0 2024-04-25T11:45:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.309+0000 7fadb7775740 10 client.18984 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.309+0000 7fadb7775740 10 client.18984 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:45:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.309+0000 7fadb7775740 20 client.18984 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.309+0000 7fadb7775740 1 --2- 172.21.15.90:0/4216060868 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5616b9b3a1c0 0x5616b9fcc670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.309+0000 7fadb7775740 4 client.18984 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:45:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.309+0000 7fadb7775740 1 -- 172.21.15.90:0/4216060868 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5616b9eb3ee0 con 0x5616b9b3a1c0 2024-04-25T11:45:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.309+0000 7fadaa59c640 1 --2- 172.21.15.90:0/4216060868 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5616b9b3a1c0 0x5616b9fcc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.310+0000 7fad91ffb640 10 client.18984 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.310+0000 7fadaa59c640 1 --2- 172.21.15.90:0/4216060868 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5616b9b3a1c0 0x5616b9fcc670 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e331fb62402515b server_cookie=e2208a37842614a1 in_seq=0 out_seq=0 2024-04-25T11:45:44.302 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:43 smithi138 ceph-mon[32328]: pgmap v330: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:45:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.316+0000 7fad91ffb640 1 -- 172.21.15.90:0/4216060868 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5616b9eb3ee0 con 0x5616b9b3a1c0 2024-04-25T11:45:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.316+0000 7fad91ffb640 10 client.18984 handle_command_reply: tid=0 2024-04-25T11:45:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.316+0000 7fadb7775740 1 client.18984 shutdown 2024-04-25T11:45:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.316+0000 7fadb7775740 1 -- 172.21.15.90:0/4216060868 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9b854d0 msgr2=0x5616b9f57c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.316+0000 7fadb7775740 1 --2- 172.21.15.90:0/4216060868 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9b854d0 0x5616b9f57c90 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fada00099d0 tx=0x7fada0013dd0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.316+0000 7fadb7775740 1 -- 172.21.15.90:0/4216060868 shutdown_connections 2024-04-25T11:45:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.316+0000 7fadb7775740 1 --2- 172.21.15.90:0/4216060868 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5616b9b3a1c0 0x5616b9fcc670 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.316+0000 7fadb7775740 1 --2- 172.21.15.90:0/4216060868 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616b9ba61b0 0x5616b9fc8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.316+0000 7fadb7775740 1 --2- 172.21.15.90:0/4216060868 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5616b9b854d0 0x5616b9f57c90 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.316+0000 7fadb7775740 1 --2- 172.21.15.90:0/4216060868 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5616b9b7d2f0 0x5616b9f57750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.316+0000 7fad91ffb640 0 client.18984 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.316+0000 7fadb7775740 1 -- 172.21.15.90:0/4216060868 >> 172.21.15.90:0/4216060868 conn(0x5616b9b9a300 msgr2=0x5616b9f47d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.316+0000 7fadb7775740 1 -- 172.21.15.90:0/4216060868 shutdown_connections 2024-04-25T11:45:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.316+0000 7fadb7775740 1 -- 172.21.15.90:0/4216060868 wait complete. 2024-04-25T11:45:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.317+0000 7fadb7775740 20 client.18984 trim_cache size 0 max 16384 2024-04-25T11:45:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:45:44.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:45:44.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:45:44.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:45:44.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.317+0000 7fadb08bd640 1 -- 172.21.15.90:0/3103706044 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fad940afc50 msgr2=0x7fad940b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:44.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.317+0000 7fadb08bd640 1 --2- 172.21.15.90:0/3103706044 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fad940afc50 0x7fad940b2160 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x5616b9f42c60 tx=0x7fada40047d0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.317+0000 7fadb08bd640 1 -- 172.21.15.90:0/3103706044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fadac1031c0 msgr2=0x7fadac0ff250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:44.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.317+0000 7fadb08bd640 1 --2- 172.21.15.90:0/3103706044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fadac1031c0 0x7fadac0ff250 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fad9c031ec0 tx=0x7fad9c003ba0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.317+0000 7fadb08bd640 1 -- 172.21.15.90:0/3103706044 shutdown_connections 2024-04-25T11:45:44.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.317+0000 7fadb08bd640 1 --2- 172.21.15.90:0/3103706044 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fad940afc50 0x7fad940b2160 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.317+0000 7fadb08bd640 1 --2- 172.21.15.90:0/3103706044 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fadac108f00 0x7fadac0ffcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.317+0000 7fadb08bd640 1 --2- 172.21.15.90:0/3103706044 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fadac108a60 0x7fadac0ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.317+0000 7fadb08bd640 1 --2- 172.21.15.90:0/3103706044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fadac1031c0 0x7fadac0ff250 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:44.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.317+0000 7fadb08bd640 1 -- 172.21.15.90:0/3103706044 >> 172.21.15.90:0/3103706044 conn(0x7fadac0781b0 msgr2=0x7fadac06a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:44.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.317+0000 7fadb08bd640 1 -- 172.21.15.90:0/3103706044 shutdown_connections 2024-04-25T11:45:44.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:44.317+0000 7fadb08bd640 1 -- 172.21.15.90:0/3103706044 wait complete. 2024-04-25T11:45:44.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:43 smithi156 ceph-mon[33045]: pgmap v330: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:45:46.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:45 smithi090 ceph-mon[29575]: pgmap v331: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-04-25T11:45:46.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:45 smithi138 ceph-mon[32328]: pgmap v331: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-04-25T11:45:46.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:45 smithi156 ceph-mon[33045]: pgmap v331: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-04-25T11:45:48.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:47 smithi090 ceph-mon[29575]: pgmap v332: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.3 KiB/s wr, 12 op/s 2024-04-25T11:45:48.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:47 smithi138 ceph-mon[32328]: pgmap v332: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.3 KiB/s wr, 12 op/s 2024-04-25T11:45:48.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:47 smithi156 ceph-mon[33045]: pgmap v332: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.3 KiB/s wr, 12 op/s 2024-04-25T11:45:50.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:49 smithi090 ceph-mon[29575]: pgmap v333: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:45:50.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:49 smithi138 ceph-mon[32328]: pgmap v333: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:45:50.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:49 smithi156 ceph-mon[33045]: pgmap v333: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:45:52.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:51 smithi090 ceph-mon[29575]: pgmap v334: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-04-25T11:45:52.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:51 smithi138 ceph-mon[32328]: pgmap v334: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-04-25T11:45:52.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:51 smithi156 ceph-mon[33045]: pgmap v334: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-04-25T11:45:53.408 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:45:53.409 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:45:53.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.549+0000 7f1ca1be3640 1 -- 172.21.15.90:0/1984920323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c9c107470 msgr2=0x7f1c9c10ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:53.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.549+0000 7f1ca1be3640 1 --2- 172.21.15.90:0/1984920323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c9c107470 0x7f1c9c10ec10 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f1c94009ca0 tx=0x7f1c9402a8c0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.549+0000 7f1ca1be3640 1 -- 172.21.15.90:0/1984920323 shutdown_connections 2024-04-25T11:45:53.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.549+0000 7f1ca1be3640 1 --2- 172.21.15.90:0/1984920323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c9c107470 0x7f1c9c10ec10 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.549+0000 7f1ca1be3640 1 --2- 172.21.15.90:0/1984920323 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c9c1069b0 0x7f1c9c106de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.549+0000 7f1ca1be3640 1 --2- 172.21.15.90:0/1984920323 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c9c102420 0x7f1c9c102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.549+0000 7f1ca1be3640 1 -- 172.21.15.90:0/1984920323 >> 172.21.15.90:0/1984920323 conn(0x7f1c9c077920 msgr2=0x7f1c9c077d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:53.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.549+0000 7f1ca1be3640 1 -- 172.21.15.90:0/1984920323 shutdown_connections 2024-04-25T11:45:53.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.549+0000 7f1ca1be3640 1 -- 172.21.15.90:0/1984920323 wait complete. 2024-04-25T11:45:53.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.550+0000 7f1ca1be3640 1 Processor -- start 2024-04-25T11:45:53.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.550+0000 7f1ca1be3640 1 -- start start 2024-04-25T11:45:53.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.550+0000 7f1ca1be3640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c9c102420 0x7f1c9c199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.550+0000 7f1ca1be3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c9c1069b0 0x7f1c9c199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.550+0000 7f1ca1be3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c9c107470 0x7f1c9c19de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.551+0000 7f1ca1be3640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c112c00 con 0x7f1c9c107470 2024-04-25T11:45:53.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.551+0000 7f1ca1be3640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c112a80 con 0x7f1c9c102420 2024-04-25T11:45:53.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.551+0000 7f1ca1be3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c112d80 con 0x7f1c9c1069b0 2024-04-25T11:45:53.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.551+0000 7f1c9bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c9c107470 0x7f1c9c19de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:53.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.551+0000 7f1c9bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c9c107470 0x7f1c9c19de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37766/0 (socket says 172.21.15.90:37766) 2024-04-25T11:45:53.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.551+0000 7f1c9bfff640 1 -- 172.21.15.90:0/4037754342 learned_addr learned my addr 172.21.15.90:0/4037754342 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:53.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.551+0000 7f1c9b7fe640 1 --2- 172.21.15.90:0/4037754342 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c9c102420 0x7f1c9c199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:53.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.551+0000 7f1c9affd640 1 --2- 172.21.15.90:0/4037754342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c9c1069b0 0x7f1c9c199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:53.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.551+0000 7f1c9bfff640 1 -- 172.21.15.90:0/4037754342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c9c1069b0 msgr2=0x7f1c9c199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:53.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.551+0000 7f1c9bfff640 1 --2- 172.21.15.90:0/4037754342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c9c1069b0 0x7f1c9c199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.551+0000 7f1c9bfff640 1 -- 172.21.15.90:0/4037754342 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c9c102420 msgr2=0x7f1c9c199620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:53.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.551+0000 7f1c9bfff640 1 --2- 172.21.15.90:0/4037754342 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c9c102420 0x7f1c9c199620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.551+0000 7f1c9bfff640 1 -- 172.21.15.90:0/4037754342 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c9c19e400 con 0x7f1c9c107470 2024-04-25T11:45:53.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.551+0000 7f1c9bfff640 1 --2- 172.21.15.90:0/4037754342 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c9c107470 0x7f1c9c19de40 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f1c94006010 tx=0x7f1c9402f470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:53.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.551+0000 7f1c98ff9640 1 -- 172.21.15.90:0/4037754342 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c94003cf0 con 0x7f1c9c107470 2024-04-25T11:45:53.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.551+0000 7f1c98ff9640 1 -- 172.21.15.90:0/4037754342 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c940044f0 con 0x7f1c9c107470 2024-04-25T11:45:53.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.551+0000 7f1ca1be3640 1 -- 172.21.15.90:0/4037754342 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c9c19e690 con 0x7f1c9c107470 2024-04-25T11:45:53.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.551+0000 7f1c98ff9640 1 -- 172.21.15.90:0/4037754342 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9402fd40 con 0x7f1c9c107470 2024-04-25T11:45:53.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.551+0000 7f1ca1be3640 1 -- 172.21.15.90:0/4037754342 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c9c19ebe0 con 0x7f1c9c107470 2024-04-25T11:45:53.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.554+0000 7f1c98ff9640 1 -- 172.21.15.90:0/4037754342 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f1c94004040 con 0x7f1c9c107470 2024-04-25T11:45:53.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.554+0000 7f1ca8b6f740 1 Processor -- start 2024-04-25T11:45:53.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.554+0000 7f1ca8b6f740 1 -- start start 2024-04-25T11:45:53.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.554+0000 7f1ca8b6f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f562d390 0x5585f59cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.554+0000 7f1ca8b6f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5585f56044d0 0x5585f59cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.554+0000 7f1ca8b6f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f560c6b0 0x5585f59da8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.554+0000 7f1ca8b6f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5585f5933d80 con 0x5585f560c6b0 2024-04-25T11:45:53.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.554+0000 7f1ca8b6f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5585f58ed8b0 con 0x5585f56044d0 2024-04-25T11:45:53.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.554+0000 7f1ca8b6f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5585f592fd40 con 0x5585f562d390 2024-04-25T11:45:53.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.555+0000 7f1c9affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f560c6b0 0x5585f59da8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:53.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.555+0000 7f1c9affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f560c6b0 0x5585f59da8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37778/0 (socket says 172.21.15.90:37778) 2024-04-25T11:45:53.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.555+0000 7f1c9affd640 1 -- 172.21.15.90:0/2597939106 learned_addr learned my addr 172.21.15.90:0/2597939106 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:53.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.555+0000 7f1c9b7fe640 1 --2- 172.21.15.90:0/2597939106 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5585f56044d0 0x5585f59cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:53.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.555+0000 7f1c98ff9640 1 --2- 172.21.15.90:0/4037754342 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1c840afc60 0x7f1c840b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.555+0000 7f1c9affd640 1 --2- 172.21.15.90:0/2597939106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f562d390 0x5585f59cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:53.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.555+0000 7f1c9affd640 1 -- 172.21.15.90:0/2597939106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f562d390 msgr2=0x5585f59cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:53.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.555+0000 7f1c9bfff640 1 --2- 172.21.15.90:0/4037754342 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1c840afc60 0x7f1c840b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:53.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.555+0000 7f1c9affd640 1 --2- 172.21.15.90:0/2597939106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f562d390 0x5585f59cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.555+0000 7f1c9affd640 1 -- 172.21.15.90:0/2597939106 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5585f56044d0 msgr2=0x5585f59cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:53.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.555+0000 7f1c9affd640 1 --2- 172.21.15.90:0/2597939106 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5585f56044d0 0x5585f59cdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.555+0000 7f1c9affd640 1 -- 172.21.15.90:0/2597939106 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585f58bea40 con 0x5585f560c6b0 2024-04-25T11:45:53.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.555+0000 7f1c98ff9640 1 -- 172.21.15.90:0/4037754342 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7f1c94100850 con 0x7f1c9c107470 2024-04-25T11:45:53.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.555+0000 7f1c9affd640 1 --2- 172.21.15.90:0/2597939106 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f560c6b0 0x5585f59da8f0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f1c90005e10 tx=0x7f1c9000e610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54819eb2c1aed606 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:53.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.555+0000 7f1c81ffb640 1 -- 172.21.15.90:0/2597939106 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c90018070 con 0x5585f560c6b0 2024-04-25T11:45:53.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.555+0000 7f1c81ffb640 1 -- 172.21.15.90:0/2597939106 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c90004220 con 0x5585f560c6b0 2024-04-25T11:45:53.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.555+0000 7f1c81ffb640 1 -- 172.21.15.90:0/2597939106 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c90003950 con 0x5585f560c6b0 2024-04-25T11:45:53.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.556+0000 7f1ca8b6f740 1 -- 172.21.15.90:0/2597939106 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f560c6b0 msgr2=0x5585f59da8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:53.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.556+0000 7f1c9bfff640 1 --2- 172.21.15.90:0/4037754342 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1c840afc60 0x7f1c840b2170 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1c940c8770 tx=0x7f1c9408ee90 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:53.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.556+0000 7f1ca8b6f740 1 --2- 172.21.15.90:0/2597939106 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f560c6b0 0x5585f59da8f0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f1c90005e10 tx=0x7f1c9000e610 comp rx=0 tx=0).stop 2024-04-25T11:45:53.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.556+0000 7f1ca8b6f740 1 -- 172.21.15.90:0/2597939106 shutdown_connections 2024-04-25T11:45:53.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.556+0000 7f1ca8b6f740 1 --2- 172.21.15.90:0/2597939106 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f560c6b0 0x5585f59da8f0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.556+0000 7f1ca8b6f740 1 --2- 172.21.15.90:0/2597939106 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5585f56044d0 0x5585f59cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.556+0000 7f1ca8b6f740 1 --2- 172.21.15.90:0/2597939106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f562d390 0x5585f59cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.556+0000 7f1ca8b6f740 1 -- 172.21.15.90:0/2597939106 >> 172.21.15.90:0/2597939106 conn(0x5585f56214e0 msgr2=0x5585f59ca210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:53.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.556+0000 7f1ca8b6f740 1 -- 172.21.15.90:0/2597939106 shutdown_connections 2024-04-25T11:45:53.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.556+0000 7f1ca8b6f740 1 -- 172.21.15.90:0/2597939106 wait complete. 2024-04-25T11:45:53.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.557+0000 7f1ca8b6f740 1 -- start start 2024-04-25T11:45:53.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.557+0000 7f1ca8b6f740 1 Processor -- start 2024-04-25T11:45:53.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.557+0000 7f1ca8b6f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f56044d0 0x5585f59da050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.557+0000 7f1ca8b6f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5585f560c6b0 0x5585f59da590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.557+0000 7f1ca8b6f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f562d390 0x5585f59d8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.557+0000 7f1ca8b6f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5585f59dafe0 con 0x5585f56044d0 2024-04-25T11:45:53.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.557+0000 7f1ca8b6f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5585f58bad10 con 0x5585f560c6b0 2024-04-25T11:45:53.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.557+0000 7f1ca8b6f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5585f59db160 con 0x5585f562d390 2024-04-25T11:45:53.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.557+0000 7f1ca8b6f740 10 client.0 _ll_register_callbacks cb 0x5585f58e7210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:53.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.557+0000 7f1c9affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f56044d0 0x5585f59da050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:53.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.557+0000 7f1c9affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f56044d0 0x5585f59da050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37780/0 (socket says 172.21.15.90:37780) 2024-04-25T11:45:53.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.557+0000 7f1c9affd640 1 -- 172.21.15.90:0/4172494863 learned_addr learned my addr 172.21.15.90:0/4172494863 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:53.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.557+0000 7f1c9b7fe640 1 --2- 172.21.15.90:0/4172494863 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5585f560c6b0 0x5585f59da590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:53.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.558+0000 7f1c9affd640 1 -- 172.21.15.90:0/4172494863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f562d390 msgr2=0x5585f59d8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:53.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.558+0000 7f1c9affd640 1 --2- 172.21.15.90:0/4172494863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f562d390 0x5585f59d8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.558+0000 7f1c9affd640 1 -- 172.21.15.90:0/4172494863 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5585f560c6b0 msgr2=0x5585f59da590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:53.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.558+0000 7f1c9affd640 1 --2- 172.21.15.90:0/4172494863 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5585f560c6b0 0x5585f59da590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.558+0000 7f1c9affd640 1 -- 172.21.15.90:0/4172494863 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585f570f7d0 con 0x5585f56044d0 2024-04-25T11:45:53.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.558+0000 7f1c9b7fe640 1 --2- 172.21.15.90:0/4172494863 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5585f560c6b0 0x5585f59da590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:53.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.558+0000 7f1c9affd640 1 --2- 172.21.15.90:0/4172494863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f56044d0 0x5585f59da050 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f1c90001eb0 tx=0x7f1c90004160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=667721b394b049bc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:53.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.558+0000 7f1c7affd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:53.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.558+0000 7f1c7affd640 1 -- 172.21.15.90:0/4172494863 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9002a060 con 0x5585f56044d0 2024-04-25T11:45:53.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.558+0000 7f1c7affd640 1 -- 172.21.15.90:0/4172494863 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7f1c90010920 con 0x5585f56044d0 2024-04-25T11:45:53.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.558+0000 7f1ca8b6f740 1 -- 172.21.15.90:0/4172494863 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c90004220 con 0x5585f56044d0 2024-04-25T11:45:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.558+0000 7f1c7affd640 1 -- 172.21.15.90:0/4172494863 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1c90018070 con 0x5585f56044d0 2024-04-25T11:45:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.558+0000 7f1ca8b6f740 10 client.18996 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.558+0000 7f1ca8b6f740 1 -- 172.21.15.90:0/4172494863 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5585f56453f0 con 0x5585f56044d0 2024-04-25T11:45:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.558+0000 7f1c7affd640 1 -- 172.21.15.90:0/4172494863 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c90004220 con 0x5585f56044d0 2024-04-25T11:45:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.559+0000 7f1c7affd640 1 -- 172.21.15.90:0/4172494863 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f1c900036d0 con 0x5585f56044d0 2024-04-25T11:45:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.559+0000 7f1ca8b6f740 10 client.18996 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.559+0000 7f1ca8b6f740 10 client.18996 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:45:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.559+0000 7f1ca8b6f740 20 client.18996 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.559+0000 7f1ca8b6f740 1 --2- 172.21.15.90:0/4172494863 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5585f55ebf60 0x5585f59ea6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.559+0000 7f1ca8b6f740 4 client.18996 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:45:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.559+0000 7f1ca8b6f740 1 -- 172.21.15.90:0/4172494863 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585f5941990 con 0x5585f55ebf60 2024-04-25T11:45:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.559+0000 7f1c9b7fe640 1 --2- 172.21.15.90:0/4172494863 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5585f55ebf60 0x5585f59ea6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.560+0000 7f1c9b7fe640 1 --2- 172.21.15.90:0/4172494863 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5585f55ebf60 0x5585f59ea6a0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c190fd92a5ae9ae3 server_cookie=797dec0d0e32a79f in_seq=0 out_seq=0 2024-04-25T11:45:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.560+0000 7f1c7affd640 10 client.18996 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:53.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.560+0000 7f1c7affd640 1 -- 172.21.15.90:0/4172494863 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5585f5941990 con 0x5585f55ebf60 2024-04-25T11:45:53.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.560+0000 7f1c7affd640 10 client.18996 handle_command_reply: tid=0 2024-04-25T11:45:53.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.560+0000 7f1ca8b6f740 1 client.18996 shutdown 2024-04-25T11:45:53.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.561+0000 7f1ca8b6f740 1 -- 172.21.15.90:0/4172494863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f56044d0 msgr2=0x5585f59da050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:53.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.561+0000 7f1ca8b6f740 1 --2- 172.21.15.90:0/4172494863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f56044d0 0x5585f59da050 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f1c90001eb0 tx=0x7f1c90004160 comp rx=0 tx=0).stop 2024-04-25T11:45:53.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.561+0000 7f1ca8b6f740 1 -- 172.21.15.90:0/4172494863 shutdown_connections 2024-04-25T11:45:53.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.561+0000 7f1ca8b6f740 1 --2- 172.21.15.90:0/4172494863 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5585f55ebf60 0x5585f59ea6a0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.561+0000 7f1ca8b6f740 1 --2- 172.21.15.90:0/4172494863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f562d390 0x5585f59d8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.561+0000 7f1ca8b6f740 1 --2- 172.21.15.90:0/4172494863 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5585f560c6b0 0x5585f59da590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.561+0000 7f1ca8b6f740 1 --2- 172.21.15.90:0/4172494863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f56044d0 0x5585f59da050 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.561+0000 7f1c7affd640 0 client.18996 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:53.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.561+0000 7f1ca8b6f740 1 -- 172.21.15.90:0/4172494863 >> 172.21.15.90:0/4172494863 conn(0x5585f56214e0 msgr2=0x5585f59cf780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:53.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.561+0000 7f1ca8b6f740 1 -- 172.21.15.90:0/4172494863 shutdown_connections 2024-04-25T11:45:53.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.561+0000 7f1ca8b6f740 1 -- 172.21.15.90:0/4172494863 wait complete. 2024-04-25T11:45:53.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.561+0000 7f1ca8b6f740 20 client.18996 trim_cache size 0 max 16384 2024-04-25T11:45:53.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.578+0000 7f1ca8b6f740 1 Processor -- start 2024-04-25T11:45:53.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.578+0000 7f1ca8b6f740 1 -- start start 2024-04-25T11:45:53.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.579+0000 7f1ca8b6f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f56044d0 0x5585f59da390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.579+0000 7f1ca8b6f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f560c6b0 0x5585f59efcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.579+0000 7f1ca8b6f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5585f562d390 0x5585f59f0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.579+0000 7f1ca8b6f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c1aeae0 con 0x5585f56044d0 2024-04-25T11:45:53.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.579+0000 7f1ca8b6f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c1b0620 con 0x5585f562d390 2024-04-25T11:45:53.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.579+0000 7f1ca8b6f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c1af1b0 con 0x5585f560c6b0 2024-04-25T11:45:53.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.579+0000 7f1c9affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f56044d0 0x5585f59da390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:53.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.579+0000 7f1c9affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f56044d0 0x5585f59da390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37794/0 (socket says 172.21.15.90:37794) 2024-04-25T11:45:53.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.579+0000 7f1c9affd640 1 -- 172.21.15.90:0/1897882998 learned_addr learned my addr 172.21.15.90:0/1897882998 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:53.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.579+0000 7f1c9affd640 1 -- 172.21.15.90:0/1897882998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f560c6b0 msgr2=0x5585f59efcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:53.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.579+0000 7f1c9affd640 1 --2- 172.21.15.90:0/1897882998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f560c6b0 0x5585f59efcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.579+0000 7f1c9affd640 1 -- 172.21.15.90:0/1897882998 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5585f562d390 msgr2=0x5585f59f0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:53.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.579+0000 7f1c9affd640 1 --2- 172.21.15.90:0/1897882998 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5585f562d390 0x5585f59f0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.579+0000 7f1c9affd640 1 -- 172.21.15.90:0/1897882998 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585f56050f0 con 0x5585f56044d0 2024-04-25T11:45:53.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.579+0000 7f1c9affd640 1 --2- 172.21.15.90:0/1897882998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f56044d0 0x5585f59da390 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f1c90003dc0 tx=0x7f1c90012f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80ba092668776691 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:53.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.579+0000 7f1c817fa640 1 -- 172.21.15.90:0/1897882998 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c90039070 con 0x5585f56044d0 2024-04-25T11:45:53.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.579+0000 7f1c817fa640 1 -- 172.21.15.90:0/1897882998 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c90003e40 con 0x5585f56044d0 2024-04-25T11:45:53.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.579+0000 7f1c817fa640 1 -- 172.21.15.90:0/1897882998 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9002a040 con 0x5585f56044d0 2024-04-25T11:45:53.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.580+0000 7f1ca8b6f740 1 -- 172.21.15.90:0/1897882998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f56044d0 msgr2=0x5585f59da390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:53.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.580+0000 7f1ca8b6f740 1 --2- 172.21.15.90:0/1897882998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f56044d0 0x5585f59da390 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f1c90003dc0 tx=0x7f1c90012f60 comp rx=0 tx=0).stop 2024-04-25T11:45:53.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.580+0000 7f1ca8b6f740 1 -- 172.21.15.90:0/1897882998 shutdown_connections 2024-04-25T11:45:53.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.580+0000 7f1ca8b6f740 1 --2- 172.21.15.90:0/1897882998 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5585f562d390 0x5585f59f0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.580+0000 7f1ca8b6f740 1 --2- 172.21.15.90:0/1897882998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f560c6b0 0x5585f59efcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.580+0000 7f1ca8b6f740 1 --2- 172.21.15.90:0/1897882998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f56044d0 0x5585f59da390 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.580+0000 7f1ca8b6f740 1 -- 172.21.15.90:0/1897882998 >> 172.21.15.90:0/1897882998 conn(0x5585f56214e0 msgr2=0x5585f59d9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:53.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.580+0000 7f1ca8b6f740 1 -- 172.21.15.90:0/1897882998 shutdown_connections 2024-04-25T11:45:53.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.580+0000 7f1ca8b6f740 1 -- 172.21.15.90:0/1897882998 wait complete. 2024-04-25T11:45:53.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.580+0000 7f1ca8b6f740 1 -- start start 2024-04-25T11:45:53.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.581+0000 7f1ca8b6f740 1 Processor -- start 2024-04-25T11:45:53.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.581+0000 7f1ca8b6f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f56044d0 0x5585f59e07a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.581+0000 7f1ca8b6f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f560c6b0 0x5585f59ddb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.581+0000 7f1ca8b6f740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5585f562d390 0x5585f59de050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.581+0000 7f1ca8b6f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c1aff50 con 0x5585f56044d0 2024-04-25T11:45:53.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.581+0000 7f1ca8b6f740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c1af880 con 0x5585f562d390 2024-04-25T11:45:53.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.581+0000 7f1ca8b6f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c1acae0 con 0x5585f560c6b0 2024-04-25T11:45:53.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.581+0000 7f1ca8b6f740 10 client.0 _ll_register_callbacks cb 0x7f1c8c00a590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:53.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.581+0000 7f1c9affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f56044d0 0x5585f59e07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:53.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.581+0000 7f1c9affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f56044d0 0x5585f59e07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37804/0 (socket says 172.21.15.90:37804) 2024-04-25T11:45:53.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.581+0000 7f1c9affd640 1 -- 172.21.15.90:0/3561888683 learned_addr learned my addr 172.21.15.90:0/3561888683 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:53.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.581+0000 7f1c9b7fe640 1 --2- 172.21.15.90:0/3561888683 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f560c6b0 0x5585f59ddb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:53.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.581+0000 7f1c9affd640 1 --2- 172.21.15.90:0/3561888683 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5585f562d390 0x5585f59de050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:53.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.581+0000 7f1c9affd640 1 -- 172.21.15.90:0/3561888683 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f560c6b0 msgr2=0x5585f59ddb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:53.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.581+0000 7f1c9affd640 1 --2- 172.21.15.90:0/3561888683 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f560c6b0 0x5585f59ddb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.581+0000 7f1c9affd640 1 -- 172.21.15.90:0/3561888683 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5585f562d390 msgr2=0x5585f59de050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:53.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.581+0000 7f1c9affd640 1 --2- 172.21.15.90:0/3561888683 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5585f562d390 0x5585f59de050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.581+0000 7f1c9affd640 1 -- 172.21.15.90:0/3561888683 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585f58f44b0 con 0x5585f56044d0 2024-04-25T11:45:53.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.582+0000 7f1c9affd640 1 --2- 172.21.15.90:0/3561888683 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f56044d0 0x5585f59e07a0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x5585f58d6f70 tx=0x7f1c9000e430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da4d16250ca0c61b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:53.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.582+0000 7f1c7affd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:53.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.582+0000 7f1c7affd640 1 -- 172.21.15.90:0/3561888683 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9002a070 con 0x5585f56044d0 2024-04-25T11:45:53.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.582+0000 7f1c7affd640 1 -- 172.21.15.90:0/3561888683 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7f1c90012230 con 0x5585f56044d0 2024-04-25T11:45:53.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.582+0000 7f1ca8b6f740 1 -- 172.21.15.90:0/3561888683 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c90003e40 con 0x5585f56044d0 2024-04-25T11:45:53.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.582+0000 7f1c7affd640 1 -- 172.21.15.90:0/3561888683 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1c90039040 con 0x5585f56044d0 2024-04-25T11:45:53.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.582+0000 7f1ca8b6f740 10 client.19002 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:53.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.582+0000 7f1ca8b6f740 1 -- 172.21.15.90:0/3561888683 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5585f5a523a0 con 0x5585f56044d0 2024-04-25T11:45:53.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.582+0000 7f1c7affd640 1 -- 172.21.15.90:0/3561888683 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c90003e40 con 0x5585f56044d0 2024-04-25T11:45:53.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.582+0000 7f1c7affd640 1 -- 172.21.15.90:0/3561888683 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f1c90046030 con 0x5585f56044d0 2024-04-25T11:45:53.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.582+0000 7f1ca8b6f740 10 client.19002 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:53.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.582+0000 7f1ca8b6f740 10 client.19002 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:45:53.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.582+0000 7f1ca8b6f740 20 client.19002 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:53.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.582+0000 7f1ca8b6f740 1 --2- 172.21.15.90:0/3561888683 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5585f55c13a0 0x5585f5a54a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.582+0000 7f1ca8b6f740 4 client.19002 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:45:53.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.582+0000 7f1ca8b6f740 1 -- 172.21.15.90:0/3561888683 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5585f593b800 con 0x5585f55c13a0 2024-04-25T11:45:53.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.582+0000 7f1c9b7fe640 1 --2- 172.21.15.90:0/3561888683 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5585f55c13a0 0x5585f5a54a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:53.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.583+0000 7f1c7affd640 10 client.19002 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:53.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.583+0000 7f1c9b7fe640 1 --2- 172.21.15.90:0/3561888683 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5585f55c13a0 0x5585f5a54a70 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7d7c0466bc4613c server_cookie=138ca4669ec8d23 in_seq=0 out_seq=0 2024-04-25T11:45:53.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.583+0000 7f1c7affd640 1 -- 172.21.15.90:0/3561888683 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5585f593b800 con 0x5585f55c13a0 2024-04-25T11:45:53.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.583+0000 7f1c7affd640 10 client.19002 handle_command_reply: tid=0 2024-04-25T11:45:53.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.583+0000 7f1ca8b6f740 1 client.19002 shutdown 2024-04-25T11:45:53.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.583+0000 7f1ca8b6f740 1 -- 172.21.15.90:0/3561888683 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f56044d0 msgr2=0x5585f59e07a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:53.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.583+0000 7f1ca8b6f740 1 --2- 172.21.15.90:0/3561888683 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f56044d0 0x5585f59e07a0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x5585f58d6f70 tx=0x7f1c9000e430 comp rx=0 tx=0).stop 2024-04-25T11:45:53.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.583+0000 7f1ca8b6f740 1 -- 172.21.15.90:0/3561888683 shutdown_connections 2024-04-25T11:45:53.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.583+0000 7f1ca8b6f740 1 --2- 172.21.15.90:0/3561888683 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5585f55c13a0 0x5585f5a54a70 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.583+0000 7f1ca8b6f740 1 --2- 172.21.15.90:0/3561888683 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5585f562d390 0x5585f59de050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.583+0000 7f1c7affd640 0 client.19002 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:53.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.583+0000 7f1ca8b6f740 1 --2- 172.21.15.90:0/3561888683 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f560c6b0 0x5585f59ddb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.583+0000 7f1ca8b6f740 1 --2- 172.21.15.90:0/3561888683 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5585f56044d0 0x5585f59e07a0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.583+0000 7f1ca8b6f740 1 -- 172.21.15.90:0/3561888683 >> 172.21.15.90:0/3561888683 conn(0x5585f56214e0 msgr2=0x5585f59da3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:53.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.583+0000 7f1ca8b6f740 1 -- 172.21.15.90:0/3561888683 shutdown_connections 2024-04-25T11:45:53.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.583+0000 7f1ca8b6f740 1 -- 172.21.15.90:0/3561888683 wait complete. 2024-04-25T11:45:53.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.583+0000 7f1ca8b6f740 20 client.19002 trim_cache size 0 max 16384 2024-04-25T11:45:53.576 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:45:53.576 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-25T11:45:53.576 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-25T11:45:53.576 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:45:53.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.584+0000 7f1ca1be3640 1 -- 172.21.15.90:0/4037754342 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1c840afc60 msgr2=0x7f1c840b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:53.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.584+0000 7f1ca1be3640 1 --2- 172.21.15.90:0/4037754342 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1c840afc60 0x7f1c840b2170 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1c940c8770 tx=0x7f1c9408ee90 comp rx=0 tx=0).stop 2024-04-25T11:45:53.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.584+0000 7f1ca1be3640 1 -- 172.21.15.90:0/4037754342 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c9c107470 msgr2=0x7f1c9c19de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:53.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.584+0000 7f1ca1be3640 1 --2- 172.21.15.90:0/4037754342 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c9c107470 0x7f1c9c19de40 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f1c94006010 tx=0x7f1c9402f470 comp rx=0 tx=0).stop 2024-04-25T11:45:53.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.584+0000 7f1ca1be3640 1 -- 172.21.15.90:0/4037754342 shutdown_connections 2024-04-25T11:45:53.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.584+0000 7f1ca1be3640 1 --2- 172.21.15.90:0/4037754342 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1c840afc60 0x7f1c840b2170 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.584+0000 7f1ca1be3640 1 --2- 172.21.15.90:0/4037754342 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c9c107470 0x7f1c9c19de40 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.584+0000 7f1ca1be3640 1 --2- 172.21.15.90:0/4037754342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c9c1069b0 0x7f1c9c199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.584+0000 7f1ca1be3640 1 --2- 172.21.15.90:0/4037754342 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c9c102420 0x7f1c9c199620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.584+0000 7f1ca1be3640 1 -- 172.21.15.90:0/4037754342 >> 172.21.15.90:0/4037754342 conn(0x7f1c9c077920 msgr2=0x7f1c9c0ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:53.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.584+0000 7f1ca1be3640 1 -- 172.21.15.90:0/4037754342 shutdown_connections 2024-04-25T11:45:53.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.584+0000 7f1ca1be3640 1 -- 172.21.15.90:0/4037754342 wait complete. 2024-04-25T11:45:53.581 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T11:45:53.582 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:45:53.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.703+0000 7f89ad5a8640 1 -- 172.21.15.90:0/76790147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89a810b650 msgr2=0x7f89a810da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:53.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.703+0000 7f89ad5a8640 1 --2- 172.21.15.90:0/76790147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89a810b650 0x7f89a810da70 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f899c009900 tx=0x7f899c02a410 comp rx=0 tx=0).stop 2024-04-25T11:45:53.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.703+0000 7f89ad5a8640 1 -- 172.21.15.90:0/76790147 shutdown_connections 2024-04-25T11:45:53.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.703+0000 7f89ad5a8640 1 --2- 172.21.15.90:0/76790147 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f89a810e040 0x7f89a8110530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.703+0000 7f89ad5a8640 1 --2- 172.21.15.90:0/76790147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89a810b650 0x7f89a810da70 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.703+0000 7f89ad5a8640 1 --2- 172.21.15.90:0/76790147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89a8104bf0 0x7f89a8105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.703+0000 7f89ad5a8640 1 -- 172.21.15.90:0/76790147 >> 172.21.15.90:0/76790147 conn(0x7f89a8078110 msgr2=0x7f89a8078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:53.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.703+0000 7f89ad5a8640 1 -- 172.21.15.90:0/76790147 shutdown_connections 2024-04-25T11:45:53.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.703+0000 7f89ad5a8640 1 -- 172.21.15.90:0/76790147 wait complete. 2024-04-25T11:45:53.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.704+0000 7f89ad5a8640 1 Processor -- start 2024-04-25T11:45:53.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.704+0000 7f89ad5a8640 1 -- start start 2024-04-25T11:45:53.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.704+0000 7f89ad5a8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89a8104bf0 0x7f89a819a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.704+0000 7f89ad5a8640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f89a810b650 0x7f89a819ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.704+0000 7f89ad5a8640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89a810e040 0x7f89a819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.704+0000 7f89ad5a8640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f89a8113ca0 con 0x7f89a810e040 2024-04-25T11:45:53.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.704+0000 7f89ad5a8640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f89a8113b20 con 0x7f89a810b650 2024-04-25T11:45:53.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.704+0000 7f89ad5a8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f89a8113e20 con 0x7f89a8104bf0 2024-04-25T11:45:53.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.704+0000 7f89a77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89a810e040 0x7f89a819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:53.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.705+0000 7f89a77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89a810e040 0x7f89a819efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37818/0 (socket says 172.21.15.90:37818) 2024-04-25T11:45:53.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.705+0000 7f89a77fe640 1 -- 172.21.15.90:0/20902846 learned_addr learned my addr 172.21.15.90:0/20902846 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:53.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.705+0000 7f89a67fc640 1 --2- 172.21.15.90:0/20902846 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f89a810b650 0x7f89a819ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:53.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.705+0000 7f89a6ffd640 1 --2- 172.21.15.90:0/20902846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89a8104bf0 0x7f89a819a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:53.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.705+0000 7f89a77fe640 1 -- 172.21.15.90:0/20902846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89a8104bf0 msgr2=0x7f89a819a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:53.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.705+0000 7f89a77fe640 1 --2- 172.21.15.90:0/20902846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89a8104bf0 0x7f89a819a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.705+0000 7f89a77fe640 1 -- 172.21.15.90:0/20902846 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f89a810b650 msgr2=0x7f89a819ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:53.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.705+0000 7f89a77fe640 1 --2- 172.21.15.90:0/20902846 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f89a810b650 0x7f89a819ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.705+0000 7f89a77fe640 1 -- 172.21.15.90:0/20902846 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89a819f560 con 0x7f89a810e040 2024-04-25T11:45:53.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.705+0000 7f89a6ffd640 1 --2- 172.21.15.90:0/20902846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89a8104bf0 0x7f89a819a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:53.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.705+0000 7f89a77fe640 1 --2- 172.21.15.90:0/20902846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89a810e040 0x7f89a819efa0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f89a000cd10 tx=0x7f89a000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:53.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.705+0000 7f898ffff640 1 -- 172.21.15.90:0/20902846 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a000f450 con 0x7f89a810e040 2024-04-25T11:45:53.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.705+0000 7f898ffff640 1 -- 172.21.15.90:0/20902846 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89a000faa0 con 0x7f89a810e040 2024-04-25T11:45:53.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.705+0000 7f89ad5a8640 1 -- 172.21.15.90:0/20902846 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89a819f850 con 0x7f89a810e040 2024-04-25T11:45:53.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.705+0000 7f898ffff640 1 -- 172.21.15.90:0/20902846 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a0018b50 con 0x7f89a810e040 2024-04-25T11:45:53.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.705+0000 7f89ad5a8640 1 -- 172.21.15.90:0/20902846 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89a819fd20 con 0x7f89a810e040 2024-04-25T11:45:53.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.706+0000 7f89ad5a8640 1 -- 172.21.15.90:0/20902846 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89a806afa0 con 0x7f89a810e040 2024-04-25T11:45:53.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.708+0000 7f898ffff640 1 -- 172.21.15.90:0/20902846 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f89a000fc50 con 0x7f89a810e040 2024-04-25T11:45:53.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.708+0000 7f898ffff640 1 --2- 172.21.15.90:0/20902846 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f89900afc60 0x7f89900b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.708+0000 7f89a6ffd640 1 --2- 172.21.15.90:0/20902846 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f89900afc60 0x7f89900b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:53.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.708+0000 7f898ffff640 1 -- 172.21.15.90:0/20902846 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7f89a00df8a0 con 0x7f89a810e040 2024-04-25T11:45:53.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.709+0000 7f89a6ffd640 1 --2- 172.21.15.90:0/20902846 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f89900afc60 0x7f89900b2170 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8998009b40 tx=0x7f8998005e50 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:53.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.712+0000 7f898ffff640 1 -- 172.21.15.90:0/20902846 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f89a0014030 con 0x7f89a810e040 2024-04-25T11:45:53.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.928+0000 7f89ad5a8640 1 -- 172.21.15.90:0/20902846 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89a819bad0 con 0x7f89a810e040 2024-04-25T11:45:53.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.929+0000 7f898ffff640 1 -- 172.21.15.90:0/20902846 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11930 (secure 0 0 0) 0x7f89a819bad0 con 0x7f89a810e040 2024-04-25T11:45:53.916 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:45:53.916 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:39:20.469254+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24458,24431,24499,24475,24524]},"id":1}]} 2024-04-25T11:45:53.916 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 28 2024-04-25T11:45:53.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.931+0000 7f89ad5a8640 1 -- 172.21.15.90:0/20902846 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f89900afc60 msgr2=0x7f89900b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:53.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.931+0000 7f89ad5a8640 1 --2- 172.21.15.90:0/20902846 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f89900afc60 0x7f89900b2170 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8998009b40 tx=0x7f8998005e50 comp rx=0 tx=0).stop 2024-04-25T11:45:53.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.931+0000 7f89ad5a8640 1 -- 172.21.15.90:0/20902846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89a810e040 msgr2=0x7f89a819efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:53.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.931+0000 7f89ad5a8640 1 --2- 172.21.15.90:0/20902846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89a810e040 0x7f89a819efa0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f89a000cd10 tx=0x7f89a000a620 comp rx=0 tx=0).stop 2024-04-25T11:45:53.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.931+0000 7f89ad5a8640 1 -- 172.21.15.90:0/20902846 shutdown_connections 2024-04-25T11:45:53.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.931+0000 7f89ad5a8640 1 --2- 172.21.15.90:0/20902846 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f89900afc60 0x7f89900b2170 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.931+0000 7f89ad5a8640 1 --2- 172.21.15.90:0/20902846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89a810e040 0x7f89a819efa0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.931+0000 7f89ad5a8640 1 --2- 172.21.15.90:0/20902846 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f89a810b650 0x7f89a819ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.931+0000 7f89ad5a8640 1 --2- 172.21.15.90:0/20902846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89a8104bf0 0x7f89a819a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.931+0000 7f89ad5a8640 1 -- 172.21.15.90:0/20902846 >> 172.21.15.90:0/20902846 conn(0x7f89a8078110 msgr2=0x7f89a8101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:53.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.931+0000 7f89ad5a8640 1 -- 172.21.15.90:0/20902846 shutdown_connections 2024-04-25T11:45:53.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:53.931+0000 7f89ad5a8640 1 -- 172.21.15.90:0/20902846 wait complete. 2024-04-25T11:45:53.930 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T11:45:54.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.053+0000 7f1e981d6640 1 -- 172.21.15.90:0/995039266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e90108a10 msgr2=0x7f1e901025c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.053+0000 7f1e981d6640 1 --2- 172.21.15.90:0/995039266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e90108a10 0x7f1e901025c0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c009900 tx=0x7f1e8c02a410 comp rx=0 tx=0).stop 2024-04-25T11:45:54.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.054+0000 7f1e981d6640 1 -- 172.21.15.90:0/995039266 shutdown_connections 2024-04-25T11:45:54.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.054+0000 7f1e981d6640 1 --2- 172.21.15.90:0/995039266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e90103170 0x7f1e9010f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.054+0000 7f1e981d6640 1 --2- 172.21.15.90:0/995039266 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1e90108eb0 0x7f1e90102b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.054+0000 7f1e981d6640 1 --2- 172.21.15.90:0/995039266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e90108a10 0x7f1e901025c0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.054+0000 7f1e981d6640 1 -- 172.21.15.90:0/995039266 >> 172.21.15.90:0/995039266 conn(0x7f1e90078150 msgr2=0x7f1e90078590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.054+0000 7f1e981d6640 1 -- 172.21.15.90:0/995039266 shutdown_connections 2024-04-25T11:45:54.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.054+0000 7f1e981d6640 1 -- 172.21.15.90:0/995039266 wait complete. 2024-04-25T11:45:54.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.054+0000 7f1e981d6640 1 Processor -- start 2024-04-25T11:45:54.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.054+0000 7f1e981d6640 1 -- start start 2024-04-25T11:45:54.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.055+0000 7f1e981d6640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e90103170 0x7f1e9019a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.055+0000 7f1e981d6640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1e90108a10 0x7f1e9019ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.055+0000 7f1e981d6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e90108eb0 0x7f1e9019f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.055+0000 7f1e981d6640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1e90112c70 con 0x7f1e90103170 2024-04-25T11:45:54.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.055+0000 7f1e981d6640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1e90112af0 con 0x7f1e90108a10 2024-04-25T11:45:54.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.055+0000 7f1e981d6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1e90112df0 con 0x7f1e90108eb0 2024-04-25T11:45:54.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.055+0000 7f1e95f72640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e90103170 0x7f1e9019a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.055+0000 7f1e95f72640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e90103170 0x7f1e9019a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37832/0 (socket says 172.21.15.90:37832) 2024-04-25T11:45:54.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.055+0000 7f1e95f72640 1 -- 172.21.15.90:0/3589365870 learned_addr learned my addr 172.21.15.90:0/3589365870 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:54.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.055+0000 7f1e95771640 1 --2- 172.21.15.90:0/3589365870 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1e90108a10 0x7f1e9019ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.055+0000 7f1e95f72640 1 -- 172.21.15.90:0/3589365870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e90108eb0 msgr2=0x7f1e9019f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.055+0000 7f1e95f72640 1 --2- 172.21.15.90:0/3589365870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e90108eb0 0x7f1e9019f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.055+0000 7f1e95f72640 1 -- 172.21.15.90:0/3589365870 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1e90108a10 msgr2=0x7f1e9019ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.055+0000 7f1e95f72640 1 --2- 172.21.15.90:0/3589365870 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1e90108a10 0x7f1e9019ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.055+0000 7f1e95f72640 1 -- 172.21.15.90:0/3589365870 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e9019f660 con 0x7f1e90103170 2024-04-25T11:45:54.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.055+0000 7f1e95f72640 1 --2- 172.21.15.90:0/3589365870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e90103170 0x7f1e9019a800 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c02a930 tx=0x7f1e8c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.055+0000 7f1e86ffd640 1 -- 172.21.15.90:0/3589365870 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c03f030 con 0x7f1e90103170 2024-04-25T11:45:54.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.055+0000 7f1e86ffd640 1 -- 172.21.15.90:0/3589365870 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e8c003aa0 con 0x7f1e90103170 2024-04-25T11:45:54.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.055+0000 7f1e981d6640 1 -- 172.21.15.90:0/3589365870 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e9019f8f0 con 0x7f1e90103170 2024-04-25T11:45:54.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.055+0000 7f1e86ffd640 1 -- 172.21.15.90:0/3589365870 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c035070 con 0x7f1e90103170 2024-04-25T11:45:54.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.055+0000 7f1e981d6640 1 -- 172.21.15.90:0/3589365870 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e9019fe30 con 0x7f1e90103170 2024-04-25T11:45:54.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.058+0000 7f1e86ffd640 1 -- 172.21.15.90:0/3589365870 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f1e8c042030 con 0x7f1e90103170 2024-04-25T11:45:54.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.058+0000 7f1e9f085740 1 Processor -- start 2024-04-25T11:45:54.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.058+0000 7f1e9f085740 1 -- start start 2024-04-25T11:45:54.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.058+0000 7f1e9f085740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de346390 0x55b7de6e7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.058+0000 7f1e9f085740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7de31d4d0 0x55b7de6e7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.058+0000 7f1e9f085740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7de3256b0 0x55b7de6f46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.058+0000 7f1e9f085740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b7de64cd80 con 0x55b7de346390 2024-04-25T11:45:54.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.058+0000 7f1e9f085740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55b7de606a20 con 0x55b7de31d4d0 2024-04-25T11:45:54.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.058+0000 7f1e9f085740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b7de648d40 con 0x55b7de3256b0 2024-04-25T11:45:54.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.059+0000 7f1e96773640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de346390 0x55b7de6e7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.059+0000 7f1e96773640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de346390 0x55b7de6e7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37848/0 (socket says 172.21.15.90:37848) 2024-04-25T11:45:54.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.059+0000 7f1e96773640 1 -- 172.21.15.90:0/2666276957 learned_addr learned my addr 172.21.15.90:0/2666276957 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:54.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.059+0000 7f1e86ffd640 1 --2- 172.21.15.90:0/3589365870 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1e7c0afce0 0x7f1e7c0b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.059+0000 7f1e95f72640 1 --2- 172.21.15.90:0/2666276957 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7de31d4d0 0x55b7de6e7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.059+0000 7f1e95771640 1 --2- 172.21.15.90:0/2666276957 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7de3256b0 0x55b7de6f46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.059+0000 7f1e96773640 1 --2- 172.21.15.90:0/3589365870 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1e7c0afce0 0x7f1e7c0b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.059+0000 7f1e86ffd640 1 -- 172.21.15.90:0/3589365870 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7f1e8c0ff750 con 0x7f1e90103170 2024-04-25T11:45:54.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.059+0000 7f1e96773640 1 -- 172.21.15.90:0/2666276957 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7de3256b0 msgr2=0x55b7de6f46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.059+0000 7f1e96773640 1 --2- 172.21.15.90:0/2666276957 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7de3256b0 0x55b7de6f46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.059+0000 7f1e96773640 1 -- 172.21.15.90:0/2666276957 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7de31d4d0 msgr2=0x55b7de6e7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.059+0000 7f1e96773640 1 --2- 172.21.15.90:0/2666276957 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7de31d4d0 0x55b7de6e7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.059+0000 7f1e96773640 1 -- 172.21.15.90:0/2666276957 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7de611c30 con 0x55b7de346390 2024-04-25T11:45:54.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.059+0000 7f1e96773640 1 --2- 172.21.15.90:0/2666276957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de346390 0x55b7de6e7810 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f1e880126c0 tx=0x7f1e88012b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26db86e2917b9021 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.059+0000 7f1e7bfff640 1 -- 172.21.15.90:0/2666276957 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e88012050 con 0x55b7de346390 2024-04-25T11:45:54.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.059+0000 7f1e96773640 1 --2- 172.21.15.90:0/3589365870 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1e7c0afce0 0x7f1e7c0b21f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1e8801d000 tx=0x7f1e8801b000 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.060+0000 7f1e7bfff640 1 -- 172.21.15.90:0/2666276957 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e88013040 con 0x55b7de346390 2024-04-25T11:45:54.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.060+0000 7f1e7bfff640 1 -- 172.21.15.90:0/2666276957 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8801b070 con 0x55b7de346390 2024-04-25T11:45:54.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.060+0000 7f1e9f085740 1 -- 172.21.15.90:0/2666276957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de346390 msgr2=0x55b7de6e7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.060+0000 7f1e9f085740 1 --2- 172.21.15.90:0/2666276957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de346390 0x55b7de6e7810 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f1e880126c0 tx=0x7f1e88012b80 comp rx=0 tx=0).stop 2024-04-25T11:45:54.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.060+0000 7f1e9f085740 1 -- 172.21.15.90:0/2666276957 shutdown_connections 2024-04-25T11:45:54.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.060+0000 7f1e9f085740 1 --2- 172.21.15.90:0/2666276957 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7de3256b0 0x55b7de6f46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.060+0000 7f1e9f085740 1 --2- 172.21.15.90:0/2666276957 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7de31d4d0 0x55b7de6e7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.060+0000 7f1e9f085740 1 --2- 172.21.15.90:0/2666276957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de346390 0x55b7de6e7810 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.060+0000 7f1e9f085740 1 -- 172.21.15.90:0/2666276957 >> 172.21.15.90:0/2666276957 conn(0x55b7de33a4e0 msgr2=0x55b7de6e3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.060+0000 7f1e9f085740 1 -- 172.21.15.90:0/2666276957 shutdown_connections 2024-04-25T11:45:54.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.061+0000 7f1e9f085740 1 -- 172.21.15.90:0/2666276957 wait complete. 2024-04-25T11:45:54.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.061+0000 7f1e9f085740 1 -- start start 2024-04-25T11:45:54.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.061+0000 7f1e9f085740 1 Processor -- start 2024-04-25T11:45:54.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.061+0000 7f1e9f085740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7de31d4d0 0x55b7de6f2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.061+0000 7f1e9f085740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de3256b0 0x55b7de6f2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.061+0000 7f1e9f085740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7de346390 0x55b7de6fe210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.061+0000 7f1e9f085740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b7de6f4d90 con 0x55b7de3256b0 2024-04-25T11:45:54.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.061+0000 7f1e9f085740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55b7de5d3dc0 con 0x55b7de346390 2024-04-25T11:45:54.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.061+0000 7f1e9f085740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b7de6f4f10 con 0x55b7de31d4d0 2024-04-25T11:45:54.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.061+0000 7f1e9f085740 10 client.0 _ll_register_callbacks cb 0x55b7de5ef600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:54.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.061+0000 7f1e95771640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de3256b0 0x55b7de6f2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.062+0000 7f1e95771640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de3256b0 0x55b7de6f2790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37864/0 (socket says 172.21.15.90:37864) 2024-04-25T11:45:54.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.062+0000 7f1e95771640 1 -- 172.21.15.90:0/3482243138 learned_addr learned my addr 172.21.15.90:0/3482243138 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:54.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.062+0000 7f1e95771640 1 -- 172.21.15.90:0/3482243138 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7de31d4d0 msgr2=0x55b7de6f2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.062+0000 7f1e95771640 1 --2- 172.21.15.90:0/3482243138 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7de31d4d0 0x55b7de6f2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.062+0000 7f1e95771640 1 -- 172.21.15.90:0/3482243138 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7de346390 msgr2=0x55b7de6fe210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.062+0000 7f1e95771640 1 --2- 172.21.15.90:0/3482243138 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7de346390 0x55b7de6fe210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.062+0000 7f1e95771640 1 -- 172.21.15.90:0/3482243138 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7de35e3f0 con 0x55b7de3256b0 2024-04-25T11:45:54.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.062+0000 7f1e95771640 1 --2- 172.21.15.90:0/3482243138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de3256b0 0x55b7de6f2790 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f1e8000bad0 tx=0x7f1e8000bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abd7db3fa1999c50 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.062+0000 7f1e797fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:54.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.062+0000 7f1e797fa640 1 -- 172.21.15.90:0/3482243138 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e80003e00 con 0x55b7de3256b0 2024-04-25T11:45:54.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.062+0000 7f1e797fa640 1 -- 172.21.15.90:0/3482243138 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7f1e80005020 con 0x55b7de3256b0 2024-04-25T11:45:54.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.062+0000 7f1e9f085740 1 -- 172.21.15.90:0/3482243138 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e88013040 con 0x55b7de3256b0 2024-04-25T11:45:54.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.062+0000 7f1e797fa640 1 -- 172.21.15.90:0/3482243138 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1e80005220 con 0x55b7de3256b0 2024-04-25T11:45:54.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.062+0000 7f1e9f085740 10 client.19020 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:54.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.062+0000 7f1e9f085740 1 -- 172.21.15.90:0/3482243138 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b7de4287d0 con 0x55b7de3256b0 2024-04-25T11:45:54.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.062+0000 7f1e797fa640 1 -- 172.21.15.90:0/3482243138 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e88013040 con 0x55b7de3256b0 2024-04-25T11:45:54.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.063+0000 7f1e797fa640 1 -- 172.21.15.90:0/3482243138 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f1e80021030 con 0x55b7de3256b0 2024-04-25T11:45:54.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.063+0000 7f1e9f085740 10 client.19020 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:54.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.063+0000 7f1e9f085740 10 client.19020 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:45:54.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.063+0000 7f1e9f085740 20 client.19020 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:54.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.063+0000 7f1e9f085740 1 --2- 172.21.15.90:0/3482243138 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55b7de304f60 0x55b7de7047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.063+0000 7f1e9f085740 4 client.19020 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:45:54.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.063+0000 7f1e9f085740 1 -- 172.21.15.90:0/3482243138 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b7de65a990 con 0x55b7de304f60 2024-04-25T11:45:54.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.063+0000 7f1e95f72640 1 --2- 172.21.15.90:0/3482243138 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55b7de304f60 0x55b7de7047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.063+0000 7f1e797fa640 10 client.19020 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:45:54.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.063+0000 7f1e95f72640 1 --2- 172.21.15.90:0/3482243138 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55b7de304f60 0x55b7de7047c0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e3356065fdc8199 server_cookie=a748f7e8b1f65b7d in_seq=0 out_seq=0 2024-04-25T11:45:54.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.064+0000 7f1e797fa640 1 -- 172.21.15.90:0/3482243138 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b7de65a990 con 0x55b7de304f60 2024-04-25T11:45:54.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.064+0000 7f1e797fa640 10 client.19020 handle_command_reply: tid=0 2024-04-25T11:45:54.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.064+0000 7f1e9f085740 1 client.19020 shutdown 2024-04-25T11:45:54.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.065+0000 7f1e9f085740 1 -- 172.21.15.90:0/3482243138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de3256b0 msgr2=0x55b7de6f2790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.065+0000 7f1e9f085740 1 --2- 172.21.15.90:0/3482243138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de3256b0 0x55b7de6f2790 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f1e8000bad0 tx=0x7f1e8000bf90 comp rx=0 tx=0).stop 2024-04-25T11:45:54.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.065+0000 7f1e9f085740 1 -- 172.21.15.90:0/3482243138 shutdown_connections 2024-04-25T11:45:54.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.065+0000 7f1e9f085740 1 --2- 172.21.15.90:0/3482243138 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55b7de304f60 0x55b7de7047c0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.065+0000 7f1e9f085740 1 --2- 172.21.15.90:0/3482243138 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7de346390 0x55b7de6fe210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.065+0000 7f1e9f085740 1 --2- 172.21.15.90:0/3482243138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de3256b0 0x55b7de6f2790 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.065+0000 7f1e9f085740 1 --2- 172.21.15.90:0/3482243138 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7de31d4d0 0x55b7de6f2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.065+0000 7f1e797fa640 0 client.19020 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:45:54.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.065+0000 7f1e9f085740 1 -- 172.21.15.90:0/3482243138 >> 172.21.15.90:0/3482243138 conn(0x55b7de33a4e0 msgr2=0x55b7de6e9530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.065+0000 7f1e9f085740 1 -- 172.21.15.90:0/3482243138 shutdown_connections 2024-04-25T11:45:54.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.065+0000 7f1e9f085740 1 -- 172.21.15.90:0/3482243138 wait complete. 2024-04-25T11:45:54.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.065+0000 7f1e9f085740 20 client.19020 trim_cache size 0 max 16384 2024-04-25T11:45:54.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.078+0000 7f1e9f085740 1 Processor -- start 2024-04-25T11:45:54.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.078+0000 7f1e9f085740 1 -- start start 2024-04-25T11:45:54.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.078+0000 7f1e9f085740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7de31d4d0 0x55b7de6fde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.078+0000 7f1e9f085740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de3256b0 0x55b7de709f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.078+0000 7f1e9f085740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7de346390 0x55b7de70a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.078+0000 7f1e9f085740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1e901afd80 con 0x55b7de3256b0 2024-04-25T11:45:54.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.078+0000 7f1e9f085740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1e901b18c0 con 0x55b7de31d4d0 2024-04-25T11:45:54.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.078+0000 7f1e9f085740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1e901b0450 con 0x55b7de346390 2024-04-25T11:45:54.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.078+0000 7f1e95771640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de3256b0 0x55b7de709f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.078+0000 7f1e95771640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de3256b0 0x55b7de709f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37866/0 (socket says 172.21.15.90:37866) 2024-04-25T11:45:54.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.078+0000 7f1e95771640 1 -- 172.21.15.90:0/1538246018 learned_addr learned my addr 172.21.15.90:0/1538246018 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:54.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.079+0000 7f1e95f72640 1 --2- 172.21.15.90:0/1538246018 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7de31d4d0 0x55b7de6fde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.079+0000 7f1e95771640 1 -- 172.21.15.90:0/1538246018 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7de346390 msgr2=0x55b7de70a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.079+0000 7f1e95771640 1 --2- 172.21.15.90:0/1538246018 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7de346390 0x55b7de70a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.079+0000 7f1e95771640 1 -- 172.21.15.90:0/1538246018 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7de31d4d0 msgr2=0x55b7de6fde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.079+0000 7f1e95771640 1 --2- 172.21.15.90:0/1538246018 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7de31d4d0 0x55b7de6fde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.079+0000 7f1e95771640 1 -- 172.21.15.90:0/1538246018 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7de6f2e10 con 0x55b7de3256b0 2024-04-25T11:45:54.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.079+0000 7f1e95f72640 1 --2- 172.21.15.90:0/1538246018 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7de31d4d0 0x55b7de6fde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:54.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.079+0000 7f1e95771640 1 --2- 172.21.15.90:0/1538246018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de3256b0 0x55b7de709f80 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f1e80001eb0 tx=0x7f1e800040f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c903385cc0e3ec0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.079+0000 7f1e7b7fe640 1 -- 172.21.15.90:0/1538246018 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e80000e60 con 0x55b7de3256b0 2024-04-25T11:45:54.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.079+0000 7f1e7b7fe640 1 -- 172.21.15.90:0/1538246018 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e8001b810 con 0x55b7de3256b0 2024-04-25T11:45:54.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.079+0000 7f1e7b7fe640 1 -- 172.21.15.90:0/1538246018 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e80011440 con 0x55b7de3256b0 2024-04-25T11:45:54.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.080+0000 7f1e9f085740 1 -- 172.21.15.90:0/1538246018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de3256b0 msgr2=0x55b7de709f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.080+0000 7f1e9f085740 1 --2- 172.21.15.90:0/1538246018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de3256b0 0x55b7de709f80 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f1e80001eb0 tx=0x7f1e800040f0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.080+0000 7f1e9f085740 1 -- 172.21.15.90:0/1538246018 shutdown_connections 2024-04-25T11:45:54.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.080+0000 7f1e9f085740 1 --2- 172.21.15.90:0/1538246018 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7de346390 0x55b7de70a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.080+0000 7f1e9f085740 1 --2- 172.21.15.90:0/1538246018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de3256b0 0x55b7de709f80 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.080+0000 7f1e9f085740 1 --2- 172.21.15.90:0/1538246018 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7de31d4d0 0x55b7de6fde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.080+0000 7f1e9f085740 1 -- 172.21.15.90:0/1538246018 >> 172.21.15.90:0/1538246018 conn(0x55b7de33a4e0 msgr2=0x55b7de6f0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.080+0000 7f1e9f085740 1 -- 172.21.15.90:0/1538246018 shutdown_connections 2024-04-25T11:45:54.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.080+0000 7f1e9f085740 1 -- 172.21.15.90:0/1538246018 wait complete. 2024-04-25T11:45:54.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.080+0000 7f1e9f085740 1 -- start start 2024-04-25T11:45:54.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.080+0000 7f1e9f085740 1 Processor -- start 2024-04-25T11:45:54.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.081+0000 7f1e9f085740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7de31d4d0 0x55b7de6f5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.081+0000 7f1e9f085740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7de3256b0 0x55b7de6f6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.081+0000 7f1e9f085740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de346390 0x55b7de6f6790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.081+0000 7f1e9f085740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1e901b11f0 con 0x55b7de346390 2024-04-25T11:45:54.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.081+0000 7f1e9f085740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1e901b0b20 con 0x55b7de31d4d0 2024-04-25T11:45:54.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.081+0000 7f1e9f085740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1e901add80 con 0x55b7de3256b0 2024-04-25T11:45:54.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.081+0000 7f1e9f085740 10 client.0 _ll_register_callbacks cb 0x7f1e80004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:54.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.081+0000 7f1e96773640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de346390 0x55b7de6f6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.081+0000 7f1e96773640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de346390 0x55b7de6f6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37874/0 (socket says 172.21.15.90:37874) 2024-04-25T11:45:54.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.081+0000 7f1e96773640 1 -- 172.21.15.90:0/1673535977 learned_addr learned my addr 172.21.15.90:0/1673535977 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:54.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.081+0000 7f1e95f72640 1 --2- 172.21.15.90:0/1673535977 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7de31d4d0 0x55b7de6f5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.081+0000 7f1e95771640 1 --2- 172.21.15.90:0/1673535977 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7de3256b0 0x55b7de6f6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.081+0000 7f1e96773640 1 -- 172.21.15.90:0/1673535977 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7de3256b0 msgr2=0x55b7de6f6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.081+0000 7f1e96773640 1 --2- 172.21.15.90:0/1673535977 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7de3256b0 0x55b7de6f6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.081+0000 7f1e96773640 1 -- 172.21.15.90:0/1673535977 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7de31d4d0 msgr2=0x55b7de6f5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.081+0000 7f1e96773640 1 --2- 172.21.15.90:0/1673535977 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7de31d4d0 0x55b7de6f5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.081+0000 7f1e96773640 1 -- 172.21.15.90:0/1673535977 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7de76bb30 con 0x55b7de346390 2024-04-25T11:45:54.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.081+0000 7f1e95f72640 1 --2- 172.21.15.90:0/1673535977 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7de31d4d0 0x55b7de6f5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:54.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.081+0000 7f1e95771640 1 --2- 172.21.15.90:0/1673535977 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7de3256b0 0x55b7de6f6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:54.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.081+0000 7f1e96773640 1 --2- 172.21.15.90:0/1673535977 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de346390 0x55b7de6f6790 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f1e88012590 tx=0x7f1e880123b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36705ba03b5e5909 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.081+0000 7f1e797fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:54.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.081+0000 7f1e797fa640 1 -- 172.21.15.90:0/1673535977 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8802f060 con 0x55b7de346390 2024-04-25T11:45:54.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.081+0000 7f1e797fa640 1 -- 172.21.15.90:0/1673535977 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7f1e880174f0 con 0x55b7de346390 2024-04-25T11:45:54.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.081+0000 7f1e9f085740 1 -- 172.21.15.90:0/1673535977 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e8001b810 con 0x55b7de346390 2024-04-25T11:45:54.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.082+0000 7f1e797fa640 1 -- 172.21.15.90:0/1673535977 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1e880176f0 con 0x55b7de346390 2024-04-25T11:45:54.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.082+0000 7f1e9f085740 10 client.19026 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:54.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.082+0000 7f1e9f085740 1 -- 172.21.15.90:0/1673535977 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b7de76cc60 con 0x55b7de346390 2024-04-25T11:45:54.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.082+0000 7f1e797fa640 1 -- 172.21.15.90:0/1673535977 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e8001b810 con 0x55b7de346390 2024-04-25T11:45:54.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.082+0000 7f1e797fa640 1 -- 172.21.15.90:0/1673535977 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f1e880207a0 con 0x55b7de346390 2024-04-25T11:45:54.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.082+0000 7f1e9f085740 10 client.19026 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:54.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.082+0000 7f1e9f085740 10 client.19026 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:45:54.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.082+0000 7f1e9f085740 20 client.19026 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:54.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.082+0000 7f1e9f085740 1 --2- 172.21.15.90:0/1673535977 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55b7de2da3a0 0x55b7de76f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.082+0000 7f1e9f085740 4 client.19026 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:45:54.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.082+0000 7f1e9f085740 1 -- 172.21.15.90:0/1673535977 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b7de654800 con 0x55b7de2da3a0 2024-04-25T11:45:54.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.082+0000 7f1e95f72640 1 --2- 172.21.15.90:0/1673535977 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55b7de2da3a0 0x55b7de76f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.083+0000 7f1e797fa640 10 client.19026 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:45:54.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.083+0000 7f1e95f72640 1 --2- 172.21.15.90:0/1673535977 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55b7de2da3a0 0x55b7de76f100 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b10c810113a39016 server_cookie=bbe3539d238291f2 in_seq=0 out_seq=0 2024-04-25T11:45:54.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.083+0000 7f1e797fa640 1 -- 172.21.15.90:0/1673535977 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b7de654800 con 0x55b7de2da3a0 2024-04-25T11:45:54.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.083+0000 7f1e797fa640 10 client.19026 handle_command_reply: tid=0 2024-04-25T11:45:54.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.083+0000 7f1e9f085740 1 client.19026 shutdown 2024-04-25T11:45:54.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.083+0000 7f1e9f085740 1 -- 172.21.15.90:0/1673535977 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de346390 msgr2=0x55b7de6f6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.083+0000 7f1e9f085740 1 --2- 172.21.15.90:0/1673535977 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de346390 0x55b7de6f6790 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f1e88012590 tx=0x7f1e880123b0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.083+0000 7f1e9f085740 1 -- 172.21.15.90:0/1673535977 shutdown_connections 2024-04-25T11:45:54.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.083+0000 7f1e9f085740 1 --2- 172.21.15.90:0/1673535977 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55b7de2da3a0 0x55b7de76f100 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.083+0000 7f1e9f085740 1 --2- 172.21.15.90:0/1673535977 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7de346390 0x55b7de6f6790 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.083+0000 7f1e9f085740 1 --2- 172.21.15.90:0/1673535977 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7de3256b0 0x55b7de6f6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.083+0000 7f1e9f085740 1 --2- 172.21.15.90:0/1673535977 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7de31d4d0 0x55b7de6f5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.083+0000 7f1e797fa640 0 client.19026 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:45:54.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.083+0000 7f1e9f085740 1 -- 172.21.15.90:0/1673535977 >> 172.21.15.90:0/1673535977 conn(0x55b7de33a4e0 msgr2=0x55b7de6e3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.083+0000 7f1e9f085740 1 -- 172.21.15.90:0/1673535977 shutdown_connections 2024-04-25T11:45:54.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.083+0000 7f1e9f085740 1 -- 172.21.15.90:0/1673535977 wait complete. 2024-04-25T11:45:54.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.083+0000 7f1e9f085740 20 client.19026 trim_cache size 0 max 16384 2024-04-25T11:45:54.077 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:45:54.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.084+0000 7f1e981d6640 1 -- 172.21.15.90:0/3589365870 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1e7c0afce0 msgr2=0x7f1e7c0b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.084+0000 7f1e981d6640 1 --2- 172.21.15.90:0/3589365870 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1e7c0afce0 0x7f1e7c0b21f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1e8801d000 tx=0x7f1e8801b000 comp rx=0 tx=0).stop 2024-04-25T11:45:54.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.084+0000 7f1e981d6640 1 -- 172.21.15.90:0/3589365870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e90103170 msgr2=0x7f1e9019a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.084+0000 7f1e981d6640 1 --2- 172.21.15.90:0/3589365870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e90103170 0x7f1e9019a800 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c02a930 tx=0x7f1e8c004270 comp rx=0 tx=0).stop 2024-04-25T11:45:54.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.084+0000 7f1e981d6640 1 -- 172.21.15.90:0/3589365870 shutdown_connections 2024-04-25T11:45:54.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.084+0000 7f1e981d6640 1 --2- 172.21.15.90:0/3589365870 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1e7c0afce0 0x7f1e7c0b21f0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.084+0000 7f1e981d6640 1 --2- 172.21.15.90:0/3589365870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e90108eb0 0x7f1e9019f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.084+0000 7f1e981d6640 1 --2- 172.21.15.90:0/3589365870 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1e90108a10 0x7f1e9019ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.084+0000 7f1e981d6640 1 --2- 172.21.15.90:0/3589365870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e90103170 0x7f1e9019a800 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.084+0000 7f1e981d6640 1 -- 172.21.15.90:0/3589365870 >> 172.21.15.90:0/3589365870 conn(0x7f1e90078150 msgr2=0x7f1e90074dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.084+0000 7f1e981d6640 1 -- 172.21.15.90:0/3589365870 shutdown_connections 2024-04-25T11:45:54.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.084+0000 7f1e981d6640 1 -- 172.21.15.90:0/3589365870 wait complete. 2024-04-25T11:45:54.081 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T11:45:54.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.205+0000 7f9e495e0640 1 -- 172.21.15.90:0/1545964736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e44102af0 msgr2=0x7f9e44102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.205+0000 7f9e495e0640 1 --2- 172.21.15.90:0/1545964736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e44102af0 0x7f9e44102f00 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c009900 tx=0x7f9e2c02a410 comp rx=0 tx=0).stop 2024-04-25T11:45:54.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.205+0000 7f9e495e0640 1 -- 172.21.15.90:0/1545964736 shutdown_connections 2024-04-25T11:45:54.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.205+0000 7f9e495e0640 1 --2- 172.21.15.90:0/1545964736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9e4410d380 0x7f9e4410f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.205+0000 7f9e495e0640 1 --2- 172.21.15.90:0/1545964736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e44102af0 0x7f9e44102f00 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.205+0000 7f9e495e0640 1 --2- 172.21.15.90:0/1545964736 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9e44108a20 0x7f9e441025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.205+0000 7f9e495e0640 1 -- 172.21.15.90:0/1545964736 >> 172.21.15.90:0/1545964736 conn(0x7f9e44078100 msgr2=0x7f9e44078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.205+0000 7f9e495e0640 1 -- 172.21.15.90:0/1545964736 shutdown_connections 2024-04-25T11:45:54.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.205+0000 7f9e495e0640 1 -- 172.21.15.90:0/1545964736 wait complete. 2024-04-25T11:45:54.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.206+0000 7f9e495e0640 1 Processor -- start 2024-04-25T11:45:54.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.206+0000 7f9e495e0640 1 -- start start 2024-04-25T11:45:54.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.206+0000 7f9e495e0640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9e44102af0 0x7f9e44199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.206+0000 7f9e495e0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e44108a20 0x7f9e44199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.206+0000 7f9e495e0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9e4410d380 0x7f9e4419de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.206+0000 7f9e495e0640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9e44112be0 con 0x7f9e44108a20 2024-04-25T11:45:54.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.206+0000 7f9e495e0640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9e44112a60 con 0x7f9e44102af0 2024-04-25T11:45:54.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.206+0000 7f9e495e0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9e44112d60 con 0x7f9e4410d380 2024-04-25T11:45:54.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.206+0000 7f9e3a7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e44108a20 0x7f9e44199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.206+0000 7f9e3a7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e44108a20 0x7f9e44199b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37898/0 (socket says 172.21.15.90:37898) 2024-04-25T11:45:54.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.206+0000 7f9e3a7fc640 1 -- 172.21.15.90:0/2390921650 learned_addr learned my addr 172.21.15.90:0/2390921650 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:54.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.207+0000 7f9e42ffd640 1 --2- 172.21.15.90:0/2390921650 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9e44102af0 0x7f9e44199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.207+0000 7f9e437fe640 1 --2- 172.21.15.90:0/2390921650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9e4410d380 0x7f9e4419de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.207+0000 7f9e3a7fc640 1 -- 172.21.15.90:0/2390921650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9e4410d380 msgr2=0x7f9e4419de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.207+0000 7f9e3a7fc640 1 --2- 172.21.15.90:0/2390921650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9e4410d380 0x7f9e4419de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.207+0000 7f9e3a7fc640 1 -- 172.21.15.90:0/2390921650 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9e44102af0 msgr2=0x7f9e44199600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.207+0000 7f9e3a7fc640 1 --2- 172.21.15.90:0/2390921650 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9e44102af0 0x7f9e44199600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.207+0000 7f9e3a7fc640 1 -- 172.21.15.90:0/2390921650 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e4419e460 con 0x7f9e44108a20 2024-04-25T11:45:54.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.207+0000 7f9e437fe640 1 --2- 172.21.15.90:0/2390921650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9e4410d380 0x7f9e4419de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:54.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.207+0000 7f9e3a7fc640 1 --2- 172.21.15.90:0/2390921650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e44108a20 0x7f9e44199b60 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c02a930 tx=0x7f9e2c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.207+0000 7f9e40ff9640 1 -- 172.21.15.90:0/2390921650 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c03f030 con 0x7f9e44108a20 2024-04-25T11:45:54.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.207+0000 7f9e40ff9640 1 -- 172.21.15.90:0/2390921650 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e2c003aa0 con 0x7f9e44108a20 2024-04-25T11:45:54.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.207+0000 7f9e495e0640 1 -- 172.21.15.90:0/2390921650 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e4419e6f0 con 0x7f9e44108a20 2024-04-25T11:45:54.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.207+0000 7f9e40ff9640 1 -- 172.21.15.90:0/2390921650 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c035070 con 0x7f9e44108a20 2024-04-25T11:45:54.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.207+0000 7f9e495e0640 1 -- 172.21.15.90:0/2390921650 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e4419ebe0 con 0x7f9e44108a20 2024-04-25T11:45:54.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.210+0000 7f9e40ff9640 1 -- 172.21.15.90:0/2390921650 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f9e2c042030 con 0x7f9e44108a20 2024-04-25T11:45:54.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.210+0000 7f9e50568740 1 Processor -- start 2024-04-25T11:45:54.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.210+0000 7f9e50568740 1 -- start start 2024-04-25T11:45:54.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.210+0000 7f9e50568740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56178510d390 0x5617854ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.210+0000 7f9e50568740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617850e44d0 0x5617854aec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.210+0000 7f9e50568740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5617850ec6b0 0x5617854bd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.210+0000 7f9e50568740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561785413d80 con 0x5617850e44d0 2024-04-25T11:45:54.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.210+0000 7f9e50568740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5617853cda20 con 0x56178510d390 2024-04-25T11:45:54.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.210+0000 7f9e50568740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56178540fd40 con 0x5617850ec6b0 2024-04-25T11:45:54.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.210+0000 7f9e42ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617850e44d0 0x5617854aec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.210+0000 7f9e42ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617850e44d0 0x5617854aec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37904/0 (socket says 172.21.15.90:37904) 2024-04-25T11:45:54.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.210+0000 7f9e42ffd640 1 -- 172.21.15.90:0/3016432433 learned_addr learned my addr 172.21.15.90:0/3016432433 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:54.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.210+0000 7f9e40ff9640 1 --2- 172.21.15.90:0/2390921650 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9e300afc60 0x7f9e300b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.211+0000 7f9e437fe640 1 --2- 172.21.15.90:0/3016432433 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56178510d390 0x5617854ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.211+0000 7f9e3a7fc640 1 --2- 172.21.15.90:0/3016432433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5617850ec6b0 0x5617854bd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.211+0000 7f9e42ffd640 1 -- 172.21.15.90:0/3016432433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5617850ec6b0 msgr2=0x5617854bd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.211+0000 7f9e42ffd640 1 --2- 172.21.15.90:0/3016432433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5617850ec6b0 0x5617854bd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.211+0000 7f9e42ffd640 1 -- 172.21.15.90:0/3016432433 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56178510d390 msgr2=0x5617854ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.211+0000 7f9e42ffd640 1 --2- 172.21.15.90:0/3016432433 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56178510d390 0x5617854ae700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.211+0000 7f9e40ff9640 1 -- 172.21.15.90:0/2390921650 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7f9e2c0ff450 con 0x7f9e44108a20 2024-04-25T11:45:54.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.211+0000 7f9e42ffd640 1 -- 172.21.15.90:0/3016432433 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617853d8c30 con 0x5617850e44d0 2024-04-25T11:45:54.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.211+0000 7f9e437fe640 1 --2- 172.21.15.90:0/3016432433 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56178510d390 0x5617854ae700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:54.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.211+0000 7f9e3a7fc640 1 --2- 172.21.15.90:0/3016432433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5617850ec6b0 0x5617854bd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:54.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.211+0000 7f9e437fe640 1 --2- 172.21.15.90:0/2390921650 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9e300afc60 0x7f9e300b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.211+0000 7f9e42ffd640 1 --2- 172.21.15.90:0/3016432433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617850e44d0 0x5617854aec40 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f9e3400bcb0 tx=0x7f9e340107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53f7d69c187739d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.211+0000 7f9e397fa640 1 -- 172.21.15.90:0/3016432433 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e34004030 con 0x5617850e44d0 2024-04-25T11:45:54.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.211+0000 7f9e397fa640 1 -- 172.21.15.90:0/3016432433 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e340041d0 con 0x5617850e44d0 2024-04-25T11:45:54.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.211+0000 7f9e397fa640 1 -- 172.21.15.90:0/3016432433 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e34011770 con 0x5617850e44d0 2024-04-25T11:45:54.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.211+0000 7f9e437fe640 1 --2- 172.21.15.90:0/2390921650 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9e300afc60 0x7f9e300b2170 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x5617854b0500 tx=0x7f9e3c004a40 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.212+0000 7f9e50568740 1 -- 172.21.15.90:0/3016432433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617850e44d0 msgr2=0x5617854aec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.212+0000 7f9e50568740 1 --2- 172.21.15.90:0/3016432433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617850e44d0 0x5617854aec40 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f9e3400bcb0 tx=0x7f9e340107a0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.212+0000 7f9e50568740 1 -- 172.21.15.90:0/3016432433 shutdown_connections 2024-04-25T11:45:54.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.212+0000 7f9e50568740 1 --2- 172.21.15.90:0/3016432433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5617850ec6b0 0x5617854bd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.212+0000 7f9e50568740 1 --2- 172.21.15.90:0/3016432433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617850e44d0 0x5617854aec40 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.212+0000 7f9e50568740 1 --2- 172.21.15.90:0/3016432433 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56178510d390 0x5617854ae700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.212+0000 7f9e50568740 1 -- 172.21.15.90:0/3016432433 >> 172.21.15.90:0/3016432433 conn(0x5617851014e0 msgr2=0x5617854aafc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.212+0000 7f9e50568740 1 -- 172.21.15.90:0/3016432433 shutdown_connections 2024-04-25T11:45:54.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.212+0000 7f9e50568740 1 -- 172.21.15.90:0/3016432433 wait complete. 2024-04-25T11:45:54.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.213+0000 7f9e50568740 1 -- start start 2024-04-25T11:45:54.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.213+0000 7f9e50568740 1 Processor -- start 2024-04-25T11:45:54.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.213+0000 7f9e50568740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617850e44d0 0x5617854bb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.213+0000 7f9e50568740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5617850ec6b0 0x5617854bb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.213+0000 7f9e50568740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56178510d390 0x5617854c7510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.213+0000 7f9e50568740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5617854bdfa0 con 0x5617850e44d0 2024-04-25T11:45:54.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.213+0000 7f9e50568740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56178539adc0 con 0x56178510d390 2024-04-25T11:45:54.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.213+0000 7f9e50568740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5617854be120 con 0x5617850ec6b0 2024-04-25T11:45:54.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.213+0000 7f9e50568740 10 client.0 _ll_register_callbacks cb 0x5617853b6600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:54.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.213+0000 7f9e42ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617850e44d0 0x5617854bb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.213+0000 7f9e42ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617850e44d0 0x5617854bb150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37914/0 (socket says 172.21.15.90:37914) 2024-04-25T11:45:54.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.213+0000 7f9e42ffd640 1 -- 172.21.15.90:0/383045943 learned_addr learned my addr 172.21.15.90:0/383045943 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:54.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.213+0000 7f9e42ffd640 1 -- 172.21.15.90:0/383045943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5617850ec6b0 msgr2=0x5617854bb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.213+0000 7f9e42ffd640 1 --2- 172.21.15.90:0/383045943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5617850ec6b0 0x5617854bb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.213+0000 7f9e42ffd640 1 -- 172.21.15.90:0/383045943 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56178510d390 msgr2=0x5617854c7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.213+0000 7f9e437fe640 1 --2- 172.21.15.90:0/383045943 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56178510d390 0x5617854c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.213+0000 7f9e42ffd640 1 --2- 172.21.15.90:0/383045943 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56178510d390 0x5617854c7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.213+0000 7f9e42ffd640 1 -- 172.21.15.90:0/383045943 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617851253f0 con 0x5617850e44d0 2024-04-25T11:45:54.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.213+0000 7f9e437fe640 1 --2- 172.21.15.90:0/383045943 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56178510d390 0x5617854c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:54.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.214+0000 7f9e42ffd640 1 --2- 172.21.15.90:0/383045943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617850e44d0 0x5617854bb150 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f9e34000c00 tx=0x7f9e3400b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=878ed85d480436e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.214+0000 7f9e2a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:54.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.214+0000 7f9e2a7fc640 1 -- 172.21.15.90:0/383045943 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e3402a060 con 0x5617850e44d0 2024-04-25T11:45:54.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.214+0000 7f9e2a7fc640 1 -- 172.21.15.90:0/383045943 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7f9e340035f0 con 0x5617850e44d0 2024-04-25T11:45:54.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.214+0000 7f9e50568740 1 -- 172.21.15.90:0/383045943 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e340041d0 con 0x5617850e44d0 2024-04-25T11:45:54.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.214+0000 7f9e2a7fc640 1 -- 172.21.15.90:0/383045943 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9e34031050 con 0x5617850e44d0 2024-04-25T11:45:54.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.214+0000 7f9e50568740 10 client.19038 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:54.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.214+0000 7f9e50568740 1 -- 172.21.15.90:0/383045943 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5617851ef7d0 con 0x5617850e44d0 2024-04-25T11:45:54.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.214+0000 7f9e2a7fc640 1 -- 172.21.15.90:0/383045943 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e340041d0 con 0x5617850e44d0 2024-04-25T11:45:54.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.214+0000 7f9e2a7fc640 1 -- 172.21.15.90:0/383045943 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f9e340046c0 con 0x5617850e44d0 2024-04-25T11:45:54.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.214+0000 7f9e50568740 10 client.19038 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:54.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.214+0000 7f9e50568740 10 client.19038 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:45:54.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.214+0000 7f9e50568740 20 client.19038 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:54.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.214+0000 7f9e50568740 1 --2- 172.21.15.90:0/383045943 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5617850cbf60 0x5617854cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.214+0000 7f9e50568740 4 client.19038 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:45:54.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.214+0000 7f9e50568740 1 -- 172.21.15.90:0/383045943 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561785421990 con 0x5617850cbf60 2024-04-25T11:45:54.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.215+0000 7f9e3a7fc640 1 --2- 172.21.15.90:0/383045943 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5617850cbf60 0x5617854cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.215+0000 7f9e3a7fc640 1 --2- 172.21.15.90:0/383045943 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5617850cbf60 0x5617854cd700 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3d8b3c648a27e6a0 server_cookie=6d6e5640180966fe in_seq=0 out_seq=0 2024-04-25T11:45:54.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.215+0000 7f9e2a7fc640 10 client.19038 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:45:54.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.216+0000 7f9e2a7fc640 1 -- 172.21.15.90:0/383045943 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561785421990 con 0x5617850cbf60 2024-04-25T11:45:54.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.216+0000 7f9e2a7fc640 10 client.19038 handle_command_reply: tid=0 2024-04-25T11:45:54.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.216+0000 7f9e50568740 1 client.19038 shutdown 2024-04-25T11:45:54.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.216+0000 7f9e50568740 1 -- 172.21.15.90:0/383045943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617850e44d0 msgr2=0x5617854bb150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.216+0000 7f9e50568740 1 --2- 172.21.15.90:0/383045943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617850e44d0 0x5617854bb150 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f9e34000c00 tx=0x7f9e3400b5b0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.216+0000 7f9e50568740 1 -- 172.21.15.90:0/383045943 shutdown_connections 2024-04-25T11:45:54.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.216+0000 7f9e50568740 1 --2- 172.21.15.90:0/383045943 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5617850cbf60 0x5617854cd700 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.216+0000 7f9e50568740 1 --2- 172.21.15.90:0/383045943 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56178510d390 0x5617854c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.216+0000 7f9e2a7fc640 0 client.19038 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:45:54.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.216+0000 7f9e50568740 1 --2- 172.21.15.90:0/383045943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5617850ec6b0 0x5617854bb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.216+0000 7f9e50568740 1 --2- 172.21.15.90:0/383045943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617850e44d0 0x5617854bb150 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.216+0000 7f9e50568740 1 -- 172.21.15.90:0/383045943 >> 172.21.15.90:0/383045943 conn(0x5617851014e0 msgr2=0x5617854b0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.216+0000 7f9e50568740 1 -- 172.21.15.90:0/383045943 shutdown_connections 2024-04-25T11:45:54.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.216+0000 7f9e50568740 1 -- 172.21.15.90:0/383045943 wait complete. 2024-04-25T11:45:54.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.216+0000 7f9e50568740 20 client.19038 trim_cache size 0 max 16384 2024-04-25T11:45:54.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.229+0000 7f9e50568740 1 Processor -- start 2024-04-25T11:45:54.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.229+0000 7f9e50568740 1 -- start start 2024-04-25T11:45:54.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.230+0000 7f9e50568740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617851014e0 0x5617854bd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.230+0000 7f9e50568740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5617850cbf60 0x5617854b0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.230+0000 7f9e50568740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5617850e44d0 0x5617854bb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.230+0000 7f9e50568740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9e441ada40 con 0x5617851014e0 2024-04-25T11:45:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.230+0000 7f9e50568740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9e441af580 con 0x5617850e44d0 2024-04-25T11:45:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.230+0000 7f9e50568740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9e441ae110 con 0x5617850cbf60 2024-04-25T11:45:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.230+0000 7f9e42ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617851014e0 0x5617854bd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.230+0000 7f9e42ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617851014e0 0x5617854bd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37920/0 (socket says 172.21.15.90:37920) 2024-04-25T11:45:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.230+0000 7f9e42ffd640 1 -- 172.21.15.90:0/854674285 learned_addr learned my addr 172.21.15.90:0/854674285 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.230+0000 7f9e3a7fc640 1 --2- 172.21.15.90:0/854674285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5617850cbf60 0x5617854b0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.230+0000 7f9e437fe640 1 --2- 172.21.15.90:0/854674285 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5617850e44d0 0x5617854bb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.230+0000 7f9e42ffd640 1 -- 172.21.15.90:0/854674285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5617850cbf60 msgr2=0x5617854b0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.230+0000 7f9e42ffd640 1 --2- 172.21.15.90:0/854674285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5617850cbf60 0x5617854b0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.230+0000 7f9e42ffd640 1 -- 172.21.15.90:0/854674285 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5617850e44d0 msgr2=0x5617854bb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.230+0000 7f9e42ffd640 1 --2- 172.21.15.90:0/854674285 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5617850e44d0 0x5617854bb0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.230+0000 7f9e42ffd640 1 -- 172.21.15.90:0/854674285 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617854b96a0 con 0x5617851014e0 2024-04-25T11:45:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.230+0000 7f9e437fe640 1 --2- 172.21.15.90:0/854674285 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5617850e44d0 0x5617854bb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:54.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.230+0000 7f9e42ffd640 1 --2- 172.21.15.90:0/854674285 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617851014e0 0x5617854bd880 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f9e3400bbd0 tx=0x7f9e34004cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53bc94c8b4a73998 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.231+0000 7f9e38ff9640 1 -- 172.21.15.90:0/854674285 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e3402a070 con 0x5617851014e0 2024-04-25T11:45:54.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.231+0000 7f9e38ff9640 1 -- 172.21.15.90:0/854674285 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e34004180 con 0x5617851014e0 2024-04-25T11:45:54.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.231+0000 7f9e38ff9640 1 -- 172.21.15.90:0/854674285 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e34005010 con 0x5617851014e0 2024-04-25T11:45:54.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.231+0000 7f9e50568740 1 -- 172.21.15.90:0/854674285 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617851014e0 msgr2=0x5617854bd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.231+0000 7f9e50568740 1 --2- 172.21.15.90:0/854674285 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617851014e0 0x5617854bd880 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f9e3400bbd0 tx=0x7f9e34004cb0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.231+0000 7f9e50568740 1 -- 172.21.15.90:0/854674285 shutdown_connections 2024-04-25T11:45:54.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.231+0000 7f9e50568740 1 --2- 172.21.15.90:0/854674285 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5617850e44d0 0x5617854bb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.231+0000 7f9e50568740 1 --2- 172.21.15.90:0/854674285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5617850cbf60 0x5617854b0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.231+0000 7f9e50568740 1 --2- 172.21.15.90:0/854674285 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617851014e0 0x5617854bd880 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.231+0000 7f9e50568740 1 -- 172.21.15.90:0/854674285 >> 172.21.15.90:0/854674285 conn(0x5617854b5bc0 msgr2=0x5617854ba9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.232+0000 7f9e50568740 1 -- 172.21.15.90:0/854674285 shutdown_connections 2024-04-25T11:45:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.232+0000 7f9e50568740 1 -- 172.21.15.90:0/854674285 wait complete. 2024-04-25T11:45:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.232+0000 7f9e50568740 1 -- start start 2024-04-25T11:45:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.232+0000 7f9e50568740 1 Processor -- start 2024-04-25T11:45:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.232+0000 7f9e50568740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617850cbf60 0x5617854ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.232+0000 7f9e50568740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5617850e44d0 0x5617854ce890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.232+0000 7f9e50568740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5617851014e0 0x5617854cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.232+0000 7f9e50568740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9e441aeeb0 con 0x5617850cbf60 2024-04-25T11:45:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.232+0000 7f9e50568740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9e441ae7e0 con 0x5617851014e0 2024-04-25T11:45:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.232+0000 7f9e50568740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9e441aba40 con 0x5617850e44d0 2024-04-25T11:45:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.232+0000 7f9e50568740 10 client.0 _ll_register_callbacks cb 0x7f9e3400bb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.232+0000 7f9e42ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617850cbf60 0x5617854ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.232+0000 7f9e42ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617850cbf60 0x5617854ce350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37926/0 (socket says 172.21.15.90:37926) 2024-04-25T11:45:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.232+0000 7f9e42ffd640 1 -- 172.21.15.90:0/1955842529 learned_addr learned my addr 172.21.15.90:0/1955842529 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.232+0000 7f9e3a7fc640 1 --2- 172.21.15.90:0/1955842529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5617850e44d0 0x5617854ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.232+0000 7f9e437fe640 1 --2- 172.21.15.90:0/1955842529 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5617851014e0 0x5617854cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.233+0000 7f9e42ffd640 1 -- 172.21.15.90:0/1955842529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5617850e44d0 msgr2=0x5617854ce890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.233+0000 7f9e42ffd640 1 --2- 172.21.15.90:0/1955842529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5617850e44d0 0x5617854ce890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.233+0000 7f9e42ffd640 1 -- 172.21.15.90:0/1955842529 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5617851014e0 msgr2=0x5617854cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.233+0000 7f9e42ffd640 1 --2- 172.21.15.90:0/1955842529 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5617851014e0 0x5617854cedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.233+0000 7f9e42ffd640 1 -- 172.21.15.90:0/1955842529 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561785534830 con 0x5617850cbf60 2024-04-25T11:45:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.233+0000 7f9e437fe640 1 --2- 172.21.15.90:0/1955842529 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5617851014e0 0x5617854cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.233+0000 7f9e42ffd640 1 --2- 172.21.15.90:0/1955842529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617850cbf60 0x5617854ce350 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f9e3400ba40 tx=0x7f9e3400bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c520d9154a97d9c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.233+0000 7f9e2a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.233+0000 7f9e2a7fc640 1 -- 172.21.15.90:0/1955842529 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e34042060 con 0x5617850cbf60 2024-04-25T11:45:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.233+0000 7f9e2a7fc640 1 -- 172.21.15.90:0/1955842529 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7f9e3403a040 con 0x5617850cbf60 2024-04-25T11:45:54.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.233+0000 7f9e50568740 1 -- 172.21.15.90:0/1955842529 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e34004180 con 0x5617850cbf60 2024-04-25T11:45:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.233+0000 7f9e2a7fc640 1 -- 172.21.15.90:0/1955842529 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9e34049050 con 0x5617850cbf60 2024-04-25T11:45:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.233+0000 7f9e50568740 10 client.19044 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.233+0000 7f9e50568740 1 -- 172.21.15.90:0/1955842529 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5617855359b0 con 0x5617850cbf60 2024-04-25T11:45:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.233+0000 7f9e2a7fc640 1 -- 172.21.15.90:0/1955842529 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e34004180 con 0x5617850cbf60 2024-04-25T11:45:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.233+0000 7f9e2a7fc640 1 -- 172.21.15.90:0/1955842529 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f9e34005360 con 0x5617850cbf60 2024-04-25T11:45:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.233+0000 7f9e50568740 10 client.19044 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.233+0000 7f9e50568740 10 client.19044 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:45:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.233+0000 7f9e50568740 20 client.19044 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.233+0000 7f9e50568740 1 --2- 172.21.15.90:0/1955842529 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5617850ec6b0 0x561785538030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.234+0000 7f9e50568740 4 client.19044 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:45:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.234+0000 7f9e50568740 1 -- 172.21.15.90:0/1955842529 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56178541b800 con 0x5617850ec6b0 2024-04-25T11:45:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.234+0000 7f9e3a7fc640 1 --2- 172.21.15.90:0/1955842529 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5617850ec6b0 0x561785538030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.234+0000 7f9e2a7fc640 10 client.19044 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:45:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.234+0000 7f9e3a7fc640 1 --2- 172.21.15.90:0/1955842529 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5617850ec6b0 0x561785538030 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=930577e9cbf3c9fe server_cookie=50b514937de0c685 in_seq=0 out_seq=0 2024-04-25T11:45:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.234+0000 7f9e2a7fc640 1 -- 172.21.15.90:0/1955842529 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56178541b800 con 0x5617850ec6b0 2024-04-25T11:45:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.234+0000 7f9e2a7fc640 10 client.19044 handle_command_reply: tid=0 2024-04-25T11:45:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.234+0000 7f9e50568740 1 client.19044 shutdown 2024-04-25T11:45:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.235+0000 7f9e50568740 1 -- 172.21.15.90:0/1955842529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617850cbf60 msgr2=0x5617854ce350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.235+0000 7f9e50568740 1 --2- 172.21.15.90:0/1955842529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617850cbf60 0x5617854ce350 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f9e3400ba40 tx=0x7f9e3400bfd0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.235+0000 7f9e50568740 1 -- 172.21.15.90:0/1955842529 shutdown_connections 2024-04-25T11:45:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.235+0000 7f9e50568740 1 --2- 172.21.15.90:0/1955842529 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x5617850ec6b0 0x561785538030 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.235+0000 7f9e50568740 1 --2- 172.21.15.90:0/1955842529 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5617851014e0 0x5617854cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.235+0000 7f9e2a7fc640 0 client.19044 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:45:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.235+0000 7f9e50568740 1 --2- 172.21.15.90:0/1955842529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5617850e44d0 0x5617854ce890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.235+0000 7f9e50568740 1 --2- 172.21.15.90:0/1955842529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617850cbf60 0x5617854ce350 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.235+0000 7f9e50568740 1 -- 172.21.15.90:0/1955842529 >> 172.21.15.90:0/1955842529 conn(0x5617854b5bc0 msgr2=0x5617854b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.235+0000 7f9e50568740 1 -- 172.21.15.90:0/1955842529 shutdown_connections 2024-04-25T11:45:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.235+0000 7f9e50568740 1 -- 172.21.15.90:0/1955842529 wait complete. 2024-04-25T11:45:54.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.235+0000 7f9e50568740 20 client.19044 trim_cache size 0 max 16384 2024-04-25T11:45:54.228 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:45:54.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.235+0000 7f9e495e0640 1 -- 172.21.15.90:0/2390921650 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9e300afc60 msgr2=0x7f9e300b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.235+0000 7f9e495e0640 1 --2- 172.21.15.90:0/2390921650 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9e300afc60 0x7f9e300b2170 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x5617854b0500 tx=0x7f9e3c004a40 comp rx=0 tx=0).stop 2024-04-25T11:45:54.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.235+0000 7f9e495e0640 1 -- 172.21.15.90:0/2390921650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e44108a20 msgr2=0x7f9e44199b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.235+0000 7f9e495e0640 1 --2- 172.21.15.90:0/2390921650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e44108a20 0x7f9e44199b60 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c02a930 tx=0x7f9e2c004270 comp rx=0 tx=0).stop 2024-04-25T11:45:54.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.235+0000 7f9e495e0640 1 -- 172.21.15.90:0/2390921650 shutdown_connections 2024-04-25T11:45:54.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.235+0000 7f9e495e0640 1 --2- 172.21.15.90:0/2390921650 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f9e300afc60 0x7f9e300b2170 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.235+0000 7f9e495e0640 1 --2- 172.21.15.90:0/2390921650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9e4410d380 0x7f9e4419de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.235+0000 7f9e495e0640 1 --2- 172.21.15.90:0/2390921650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e44108a20 0x7f9e44199b60 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.235+0000 7f9e495e0640 1 --2- 172.21.15.90:0/2390921650 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9e44102af0 0x7f9e44199600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.235+0000 7f9e495e0640 1 -- 172.21.15.90:0/2390921650 >> 172.21.15.90:0/2390921650 conn(0x7f9e44078100 msgr2=0x7f9e441040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.235+0000 7f9e495e0640 1 -- 172.21.15.90:0/2390921650 shutdown_connections 2024-04-25T11:45:54.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.235+0000 7f9e495e0640 1 -- 172.21.15.90:0/2390921650 wait complete. 2024-04-25T11:45:54.230 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:53 smithi090 ceph-mon[29575]: pgmap v335: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:45:54.230 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:53 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/20902846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:45:54.233 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T11:45:54.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:53 smithi138 ceph-mon[32328]: pgmap v335: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:45:54.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:53 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/20902846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:45:54.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:53 smithi156 ceph-mon[33045]: pgmap v335: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:45:54.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:53 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/20902846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:45:54.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.357+0000 7f8414e27640 1 -- 172.21.15.90:0/3668164442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8410103990 msgr2=0x7f841010f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.357+0000 7f8414e27640 1 --2- 172.21.15.90:0/3668164442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8410103990 0x7f841010f4a0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f8408009ca0 tx=0x7f840802a950 comp rx=0 tx=0).stop 2024-04-25T11:45:54.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.357+0000 7f8414e27640 1 -- 172.21.15.90:0/3668164442 shutdown_connections 2024-04-25T11:45:54.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.357+0000 7f8414e27640 1 --2- 172.21.15.90:0/3668164442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8410103990 0x7f841010f4a0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.357+0000 7f8414e27640 1 --2- 172.21.15.90:0/3668164442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8410108e50 0x7f8410103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.357+0000 7f8414e27640 1 --2- 172.21.15.90:0/3668164442 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8410108a40 0x7f8410102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.357+0000 7f8414e27640 1 -- 172.21.15.90:0/3668164442 >> 172.21.15.90:0/3668164442 conn(0x7f8410077880 msgr2=0x7f8410077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.357+0000 7f8414e27640 1 -- 172.21.15.90:0/3668164442 shutdown_connections 2024-04-25T11:45:54.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.357+0000 7f8414e27640 1 -- 172.21.15.90:0/3668164442 wait complete. 2024-04-25T11:45:54.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.358+0000 7f8414e27640 1 Processor -- start 2024-04-25T11:45:54.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.358+0000 7f8414e27640 1 -- start start 2024-04-25T11:45:54.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.358+0000 7f8414e27640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8410108a40 0x7f841019b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.358+0000 7f8414e27640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8410108e50 0x7f841019b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.358+0000 7f8414e27640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f841019bc70 0x7f84101a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.358+0000 7f8414e27640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8410112c90 con 0x7f8410108e50 2024-04-25T11:45:54.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.358+0000 7f8414e27640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8410112b10 con 0x7f841019bc70 2024-04-25T11:45:54.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.358+0000 7f8414e27640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8410112e10 con 0x7f8410108a40 2024-04-25T11:45:54.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.359+0000 7f8407fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8410108e50 0x7f841019b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.359+0000 7f8407fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8410108e50 0x7f841019b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37936/0 (socket says 172.21.15.90:37936) 2024-04-25T11:45:54.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.359+0000 7f8407fff640 1 -- 172.21.15.90:0/2743306482 learned_addr learned my addr 172.21.15.90:0/2743306482 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:54.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.359+0000 7f840ed9d640 1 --2- 172.21.15.90:0/2743306482 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f841019bc70 0x7f84101a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.359+0000 7f840e59c640 1 --2- 172.21.15.90:0/2743306482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8410108a40 0x7f841019b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.359+0000 7f8407fff640 1 -- 172.21.15.90:0/2743306482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8410108a40 msgr2=0x7f841019b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.359+0000 7f8407fff640 1 --2- 172.21.15.90:0/2743306482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8410108a40 0x7f841019b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.359+0000 7f8407fff640 1 -- 172.21.15.90:0/2743306482 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f841019bc70 msgr2=0x7f84101a6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.359+0000 7f8407fff640 1 --2- 172.21.15.90:0/2743306482 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f841019bc70 0x7f84101a6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.359+0000 7f8407fff640 1 -- 172.21.15.90:0/2743306482 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84101a6910 con 0x7f8410108e50 2024-04-25T11:45:54.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.359+0000 7f840e59c640 1 --2- 172.21.15.90:0/2743306482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8410108a40 0x7f841019b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:54.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.359+0000 7f8407fff640 1 --2- 172.21.15.90:0/2743306482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8410108e50 0x7f841019b730 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f83f800b450 tx=0x7f83f800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.359+0000 7f84077fe640 1 -- 172.21.15.90:0/2743306482 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f8015020 con 0x7f8410108e50 2024-04-25T11:45:54.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.359+0000 7f84077fe640 1 -- 172.21.15.90:0/2743306482 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83f8011d40 con 0x7f8410108e50 2024-04-25T11:45:54.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.359+0000 7f8414e27640 1 -- 172.21.15.90:0/2743306482 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84101a6c00 con 0x7f8410108e50 2024-04-25T11:45:54.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.359+0000 7f84077fe640 1 -- 172.21.15.90:0/2743306482 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f800f340 con 0x7f8410108e50 2024-04-25T11:45:54.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.359+0000 7f8414e27640 1 -- 172.21.15.90:0/2743306482 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84101a7100 con 0x7f8410108e50 2024-04-25T11:45:54.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.362+0000 7f841bdbb740 1 Processor -- start 2024-04-25T11:45:54.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.362+0000 7f841bdbb740 1 -- start start 2024-04-25T11:45:54.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.362+0000 7f84077fe640 1 -- 172.21.15.90:0/2743306482 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f83f8018020 con 0x7f8410108e50 2024-04-25T11:45:54.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.362+0000 7f841bdbb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa879ef390 0x55fa87d90810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.362+0000 7f841bdbb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879c64d0 0x55fa87d90d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.362+0000 7f841bdbb740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fa879ce6b0 0x55fa87d9d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.362+0000 7f841bdbb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55fa87cf5d80 con 0x55fa879c64d0 2024-04-25T11:45:54.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.362+0000 7f841bdbb740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55fa87cafa20 con 0x55fa879ce6b0 2024-04-25T11:45:54.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.362+0000 7f841bdbb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fa87cf1d40 con 0x55fa879ef390 2024-04-25T11:45:54.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.362+0000 7f840e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879c64d0 0x55fa87d90d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.362+0000 7f840e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879c64d0 0x55fa87d90d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37948/0 (socket says 172.21.15.90:37948) 2024-04-25T11:45:54.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.362+0000 7f840e59c640 1 -- 172.21.15.90:0/3859106634 learned_addr learned my addr 172.21.15.90:0/3859106634 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:54.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.362+0000 7f8407fff640 1 --2- 172.21.15.90:0/3859106634 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fa879ce6b0 0x55fa87d9d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.363+0000 7f840ed9d640 1 --2- 172.21.15.90:0/3859106634 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa879ef390 0x55fa87d90810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.363+0000 7f840e59c640 1 -- 172.21.15.90:0/3859106634 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa879ef390 msgr2=0x55fa87d90810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.363+0000 7f840e59c640 1 --2- 172.21.15.90:0/3859106634 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa879ef390 0x55fa87d90810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.363+0000 7f840e59c640 1 -- 172.21.15.90:0/3859106634 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fa879ce6b0 msgr2=0x55fa87d9d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.363+0000 7f840e59c640 1 --2- 172.21.15.90:0/3859106634 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fa879ce6b0 0x55fa87d9d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.363+0000 7f840e59c640 1 -- 172.21.15.90:0/3859106634 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa87cbac30 con 0x55fa879c64d0 2024-04-25T11:45:54.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.363+0000 7f84077fe640 1 --2- 172.21.15.90:0/2743306482 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f83fc0afc60 0x7f83fc0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.363+0000 7f840e59c640 1 --2- 172.21.15.90:0/3859106634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879c64d0 0x55fa87d90d50 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f840000bb40 tx=0x7f840000d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2af4d8a3b5dc28c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.363+0000 7f840ed9d640 1 --2- 172.21.15.90:0/2743306482 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f83fc0afc60 0x7f83fc0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.363+0000 7f8404ff9640 1 -- 172.21.15.90:0/3859106634 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8400017420 con 0x55fa879c64d0 2024-04-25T11:45:54.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.363+0000 7f84077fe640 1 -- 172.21.15.90:0/2743306482 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7f83f80deb00 con 0x7f8410108e50 2024-04-25T11:45:54.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.363+0000 7f8404ff9640 1 -- 172.21.15.90:0/3859106634 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8400017a70 con 0x55fa879c64d0 2024-04-25T11:45:54.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.363+0000 7f8404ff9640 1 -- 172.21.15.90:0/3859106634 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f840000fa80 con 0x55fa879c64d0 2024-04-25T11:45:54.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.364+0000 7f841bdbb740 1 -- 172.21.15.90:0/3859106634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879c64d0 msgr2=0x55fa87d90d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.364+0000 7f841bdbb740 1 --2- 172.21.15.90:0/3859106634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879c64d0 0x55fa87d90d50 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f840000bb40 tx=0x7f840000d340 comp rx=0 tx=0).stop 2024-04-25T11:45:54.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.364+0000 7f840ed9d640 1 --2- 172.21.15.90:0/2743306482 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f83fc0afc60 0x7f83fc0b2170 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8410102d70 tx=0x7f8408005c50 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.364+0000 7f841bdbb740 1 -- 172.21.15.90:0/3859106634 shutdown_connections 2024-04-25T11:45:54.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.364+0000 7f841bdbb740 1 --2- 172.21.15.90:0/3859106634 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fa879ce6b0 0x55fa87d9d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.364+0000 7f841bdbb740 1 --2- 172.21.15.90:0/3859106634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879c64d0 0x55fa87d90d50 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.364+0000 7f841bdbb740 1 --2- 172.21.15.90:0/3859106634 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa879ef390 0x55fa87d90810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.364+0000 7f841bdbb740 1 -- 172.21.15.90:0/3859106634 >> 172.21.15.90:0/3859106634 conn(0x55fa879e34e0 msgr2=0x55fa87d8cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.364+0000 7f841bdbb740 1 -- 172.21.15.90:0/3859106634 shutdown_connections 2024-04-25T11:45:54.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.364+0000 7f841bdbb740 1 -- 172.21.15.90:0/3859106634 wait complete. 2024-04-25T11:45:54.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.364+0000 7f841bdbb740 1 -- start start 2024-04-25T11:45:54.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.365+0000 7f841bdbb740 1 Processor -- start 2024-04-25T11:45:54.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.365+0000 7f841bdbb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa879c64d0 0x55fa87d9b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.365+0000 7f841bdbb740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fa879ce6b0 0x55fa87d9b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.365+0000 7f841bdbb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879ef390 0x55fa87da7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.365+0000 7f841bdbb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55fa87d9dd90 con 0x55fa879ef390 2024-04-25T11:45:54.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.365+0000 7f841bdbb740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55fa87c7cdc0 con 0x55fa879ce6b0 2024-04-25T11:45:54.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.365+0000 7f841bdbb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fa87d9df10 con 0x55fa879c64d0 2024-04-25T11:45:54.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.365+0000 7f841bdbb740 10 client.0 _ll_register_callbacks cb 0x55fa87c98600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:54.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.365+0000 7f840ed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879ef390 0x55fa87da7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.365+0000 7f840ed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879ef390 0x55fa87da7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37958/0 (socket says 172.21.15.90:37958) 2024-04-25T11:45:54.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.365+0000 7f840ed9d640 1 -- 172.21.15.90:0/2747325050 learned_addr learned my addr 172.21.15.90:0/2747325050 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:54.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.365+0000 7f8407fff640 1 --2- 172.21.15.90:0/2747325050 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fa879ce6b0 0x55fa87d9b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.365+0000 7f840e59c640 1 --2- 172.21.15.90:0/2747325050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa879c64d0 0x55fa87d9b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.365+0000 7f840ed9d640 1 -- 172.21.15.90:0/2747325050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa879c64d0 msgr2=0x55fa87d9b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.365+0000 7f840ed9d640 1 --2- 172.21.15.90:0/2747325050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa879c64d0 0x55fa87d9b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.365+0000 7f840ed9d640 1 -- 172.21.15.90:0/2747325050 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fa879ce6b0 msgr2=0x55fa87d9b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.365+0000 7f840ed9d640 1 --2- 172.21.15.90:0/2747325050 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fa879ce6b0 0x55fa87d9b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.365+0000 7f840ed9d640 1 -- 172.21.15.90:0/2747325050 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa87a073f0 con 0x55fa879ef390 2024-04-25T11:45:54.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.365+0000 7f840ed9d640 1 --2- 172.21.15.90:0/2747325050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879ef390 0x55fa87da7210 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f8408009ca0 tx=0x7f84080095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10af9d0bb0209a97 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.365+0000 7f83f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:54.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.366+0000 7f83f5ffb640 1 -- 172.21.15.90:0/2747325050 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8408034040 con 0x55fa879ef390 2024-04-25T11:45:54.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.366+0000 7f83f5ffb640 1 -- 172.21.15.90:0/2747325050 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7f8408004510 con 0x55fa879ef390 2024-04-25T11:45:54.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.366+0000 7f841bdbb740 1 -- 172.21.15.90:0/2747325050 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8400017a70 con 0x55fa879ef390 2024-04-25T11:45:54.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.366+0000 7f83f5ffb640 1 -- 172.21.15.90:0/2747325050 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8408004710 con 0x55fa879ef390 2024-04-25T11:45:54.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.366+0000 7f841bdbb740 10 client.19056 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:54.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.366+0000 7f841bdbb740 1 -- 172.21.15.90:0/2747325050 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fa87ad17d0 con 0x55fa879ef390 2024-04-25T11:45:54.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.366+0000 7f83f5ffb640 1 -- 172.21.15.90:0/2747325050 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8400017a70 con 0x55fa879ef390 2024-04-25T11:45:54.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.366+0000 7f83f5ffb640 1 -- 172.21.15.90:0/2747325050 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f8408050070 con 0x55fa879ef390 2024-04-25T11:45:54.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.366+0000 7f841bdbb740 10 client.19056 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:54.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.366+0000 7f841bdbb740 10 client.19056 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:45:54.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.366+0000 7f841bdbb740 20 client.19056 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:54.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.366+0000 7f841bdbb740 1 --2- 172.21.15.90:0/2747325050 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55fa879adf60 0x55fa87dad730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.366+0000 7f841bdbb740 4 client.19056 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:45:54.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.366+0000 7f841bdbb740 1 -- 172.21.15.90:0/2747325050 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa87d03990 con 0x55fa879adf60 2024-04-25T11:45:54.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.367+0000 7f840e59c640 1 --2- 172.21.15.90:0/2747325050 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55fa879adf60 0x55fa87dad730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.367+0000 7f83f5ffb640 10 client.19056 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:45:54.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.367+0000 7f840e59c640 1 --2- 172.21.15.90:0/2747325050 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55fa879adf60 0x55fa87dad730 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1b926b8c655c5a0c server_cookie=9ea562bd7d57f18d in_seq=0 out_seq=0 2024-04-25T11:45:54.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.368+0000 7f83f5ffb640 1 -- 172.21.15.90:0/2747325050 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55fa87d03990 con 0x55fa879adf60 2024-04-25T11:45:54.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.368+0000 7f83f5ffb640 10 client.19056 handle_command_reply: tid=0 2024-04-25T11:45:54.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.368+0000 7f841bdbb740 1 client.19056 shutdown 2024-04-25T11:45:54.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.368+0000 7f841bdbb740 1 -- 172.21.15.90:0/2747325050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879ef390 msgr2=0x55fa87da7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.368+0000 7f841bdbb740 1 --2- 172.21.15.90:0/2747325050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879ef390 0x55fa87da7210 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f8408009ca0 tx=0x7f84080095c0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.368+0000 7f841bdbb740 1 -- 172.21.15.90:0/2747325050 shutdown_connections 2024-04-25T11:45:54.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.368+0000 7f841bdbb740 1 --2- 172.21.15.90:0/2747325050 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55fa879adf60 0x55fa87dad730 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.368+0000 7f841bdbb740 1 --2- 172.21.15.90:0/2747325050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879ef390 0x55fa87da7210 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.368+0000 7f841bdbb740 1 --2- 172.21.15.90:0/2747325050 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fa879ce6b0 0x55fa87d9b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.368+0000 7f83f5ffb640 0 client.19056 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:45:54.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.368+0000 7f841bdbb740 1 --2- 172.21.15.90:0/2747325050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa879c64d0 0x55fa87d9b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.368+0000 7f841bdbb740 1 -- 172.21.15.90:0/2747325050 >> 172.21.15.90:0/2747325050 conn(0x55fa879e34e0 msgr2=0x55fa87d92530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.368+0000 7f841bdbb740 1 -- 172.21.15.90:0/2747325050 shutdown_connections 2024-04-25T11:45:54.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.368+0000 7f841bdbb740 1 -- 172.21.15.90:0/2747325050 wait complete. 2024-04-25T11:45:54.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.368+0000 7f841bdbb740 20 client.19056 trim_cache size 0 max 16384 2024-04-25T11:45:54.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.382+0000 7f841bdbb740 1 Processor -- start 2024-04-25T11:45:54.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.382+0000 7f841bdbb740 1 -- start start 2024-04-25T11:45:54.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.382+0000 7f841bdbb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879c64d0 0x55fa87d99fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.382+0000 7f841bdbb740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fa879ce6b0 0x55fa87da6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.382+0000 7f841bdbb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa879ef390 0x55fa87d95510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.382+0000 7f841bdbb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f84101afca0 con 0x55fa879c64d0 2024-04-25T11:45:54.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.382+0000 7f841bdbb740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f84101b17e0 con 0x55fa879ce6b0 2024-04-25T11:45:54.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.382+0000 7f841bdbb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f84101b0370 con 0x55fa879ef390 2024-04-25T11:45:54.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.382+0000 7f840e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879c64d0 0x55fa87d99fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.382+0000 7f840e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879c64d0 0x55fa87d99fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37960/0 (socket says 172.21.15.90:37960) 2024-04-25T11:45:54.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.382+0000 7f840e59c640 1 -- 172.21.15.90:0/2708537 learned_addr learned my addr 172.21.15.90:0/2708537 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:54.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.382+0000 7f840e59c640 1 -- 172.21.15.90:0/2708537 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa879ef390 msgr2=0x55fa87d95510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.382+0000 7f840e59c640 1 --2- 172.21.15.90:0/2708537 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa879ef390 0x55fa87d95510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.382+0000 7f8407fff640 1 --2- 172.21.15.90:0/2708537 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fa879ce6b0 0x55fa87da6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.382+0000 7f840e59c640 1 -- 172.21.15.90:0/2708537 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fa879ce6b0 msgr2=0x55fa87da6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.382+0000 7f840e59c640 1 --2- 172.21.15.90:0/2708537 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fa879ce6b0 0x55fa87da6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.382+0000 7f840e59c640 1 -- 172.21.15.90:0/2708537 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa87d8c620 con 0x55fa879c64d0 2024-04-25T11:45:54.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.382+0000 7f8407fff640 1 --2- 172.21.15.90:0/2708537 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fa879ce6b0 0x55fa87da6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:54.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.382+0000 7f840e59c640 1 --2- 172.21.15.90:0/2708537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879c64d0 0x55fa87d99fc0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f8400019040 tx=0x7f840000df10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=315199ab0339a251 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.383+0000 7f83f7fff640 1 -- 172.21.15.90:0/2708537 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f840000f830 con 0x55fa879c64d0 2024-04-25T11:45:54.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.383+0000 7f83f7fff640 1 -- 172.21.15.90:0/2708537 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8400017420 con 0x55fa879c64d0 2024-04-25T11:45:54.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.383+0000 7f83f7fff640 1 -- 172.21.15.90:0/2708537 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8400022660 con 0x55fa879c64d0 2024-04-25T11:45:54.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.383+0000 7f841bdbb740 1 -- 172.21.15.90:0/2708537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879c64d0 msgr2=0x55fa87d99fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.383+0000 7f841bdbb740 1 --2- 172.21.15.90:0/2708537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879c64d0 0x55fa87d99fc0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f8400019040 tx=0x7f840000df10 comp rx=0 tx=0).stop 2024-04-25T11:45:54.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.383+0000 7f841bdbb740 1 -- 172.21.15.90:0/2708537 shutdown_connections 2024-04-25T11:45:54.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.383+0000 7f841bdbb740 1 --2- 172.21.15.90:0/2708537 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa879ef390 0x55fa87d95510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.383+0000 7f841bdbb740 1 --2- 172.21.15.90:0/2708537 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fa879ce6b0 0x55fa87da6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.383+0000 7f841bdbb740 1 --2- 172.21.15.90:0/2708537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879c64d0 0x55fa87d99fc0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.383+0000 7f841bdbb740 1 -- 172.21.15.90:0/2708537 >> 172.21.15.90:0/2708537 conn(0x55fa879e34e0 msgr2=0x55fa87d998b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.383+0000 7f841bdbb740 1 -- 172.21.15.90:0/2708537 shutdown_connections 2024-04-25T11:45:54.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.383+0000 7f841bdbb740 1 -- 172.21.15.90:0/2708537 wait complete. 2024-04-25T11:45:54.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.384+0000 7f841bdbb740 1 -- start start 2024-04-25T11:45:54.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.384+0000 7f841bdbb740 1 Processor -- start 2024-04-25T11:45:54.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.384+0000 7f841bdbb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879c64d0 0x55fa87da3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.384+0000 7f841bdbb740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fa879ce6b0 0x55fa87d9e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.384+0000 7f841bdbb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa879ef390 0x55fa87d9ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.384+0000 7f841bdbb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f84101b1110 con 0x55fa879c64d0 2024-04-25T11:45:54.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.384+0000 7f841bdbb740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f84101b0a40 con 0x55fa879ce6b0 2024-04-25T11:45:54.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.384+0000 7f841bdbb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f84101adca0 con 0x55fa879ef390 2024-04-25T11:45:54.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.384+0000 7f841bdbb740 10 client.0 _ll_register_callbacks cb 0x7f840802f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:54.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.384+0000 7f840e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879c64d0 0x55fa87da3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.384+0000 7f840e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879c64d0 0x55fa87da3600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37972/0 (socket says 172.21.15.90:37972) 2024-04-25T11:45:54.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.384+0000 7f840e59c640 1 -- 172.21.15.90:0/990744867 learned_addr learned my addr 172.21.15.90:0/990744867 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:54.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.384+0000 7f8407fff640 1 --2- 172.21.15.90:0/990744867 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fa879ce6b0 0x55fa87d9e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.384+0000 7f840ed9d640 1 --2- 172.21.15.90:0/990744867 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa879ef390 0x55fa87d9ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.384+0000 7f840e59c640 1 -- 172.21.15.90:0/990744867 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa879ef390 msgr2=0x55fa87d9ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.384+0000 7f840e59c640 1 --2- 172.21.15.90:0/990744867 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa879ef390 0x55fa87d9ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.384+0000 7f840e59c640 1 -- 172.21.15.90:0/990744867 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fa879ce6b0 msgr2=0x55fa87d9e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.384+0000 7f840e59c640 1 --2- 172.21.15.90:0/990744867 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fa879ce6b0 0x55fa87d9e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.384+0000 7f840e59c640 1 -- 172.21.15.90:0/990744867 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa87d9f450 con 0x55fa879c64d0 2024-04-25T11:45:54.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.384+0000 7f8407fff640 1 --2- 172.21.15.90:0/990744867 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fa879ce6b0 0x55fa87d9e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:54.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.384+0000 7f840e59c640 1 --2- 172.21.15.90:0/990744867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879c64d0 0x55fa87da3600 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f840000d9c0 tx=0x7f840000b6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=797104196fc600af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.384+0000 7f83f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:54.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.385+0000 7f83f5ffb640 1 -- 172.21.15.90:0/990744867 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84000177d0 con 0x55fa879c64d0 2024-04-25T11:45:54.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.385+0000 7f83f5ffb640 1 -- 172.21.15.90:0/990744867 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7f8400017970 con 0x55fa879c64d0 2024-04-25T11:45:54.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.385+0000 7f841bdbb740 1 -- 172.21.15.90:0/990744867 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8400017420 con 0x55fa879c64d0 2024-04-25T11:45:54.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.385+0000 7f83f5ffb640 1 -- 172.21.15.90:0/990744867 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8400017b70 con 0x55fa879c64d0 2024-04-25T11:45:54.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.385+0000 7f841bdbb740 10 client.19062 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:54.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.385+0000 7f841bdbb740 1 -- 172.21.15.90:0/990744867 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fa87e15780 con 0x55fa879c64d0 2024-04-25T11:45:54.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.385+0000 7f83f5ffb640 1 -- 172.21.15.90:0/990744867 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8400017420 con 0x55fa879c64d0 2024-04-25T11:45:54.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.385+0000 7f83f5ffb640 1 -- 172.21.15.90:0/990744867 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f8400035600 con 0x55fa879c64d0 2024-04-25T11:45:54.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.385+0000 7f841bdbb740 10 client.19062 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:54.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.385+0000 7f841bdbb740 10 client.19062 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:45:54.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.385+0000 7f841bdbb740 20 client.19062 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:54.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.385+0000 7f841bdbb740 1 --2- 172.21.15.90:0/990744867 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55fa879833a0 0x55fa87e17c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.385+0000 7f841bdbb740 4 client.19062 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:45:54.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.385+0000 7f841bdbb740 1 -- 172.21.15.90:0/990744867 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fa87cfd800 con 0x55fa879833a0 2024-04-25T11:45:54.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.385+0000 7f8407fff640 1 --2- 172.21.15.90:0/990744867 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55fa879833a0 0x55fa87e17c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.386+0000 7f83f5ffb640 10 client.19062 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:45:54.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.386+0000 7f8407fff640 1 --2- 172.21.15.90:0/990744867 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55fa879833a0 0x55fa87e17c80 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=82e3cd6c67f9f705 server_cookie=86112589872e154 in_seq=0 out_seq=0 2024-04-25T11:45:54.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.386+0000 7f83f5ffb640 1 -- 172.21.15.90:0/990744867 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fa87cfd800 con 0x55fa879833a0 2024-04-25T11:45:54.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.386+0000 7f83f5ffb640 10 client.19062 handle_command_reply: tid=0 2024-04-25T11:45:54.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.386+0000 7f841bdbb740 1 client.19062 shutdown 2024-04-25T11:45:54.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.386+0000 7f841bdbb740 1 -- 172.21.15.90:0/990744867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879c64d0 msgr2=0x55fa87da3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.386+0000 7f841bdbb740 1 --2- 172.21.15.90:0/990744867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879c64d0 0x55fa87da3600 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f840000d9c0 tx=0x7f840000b6f0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.386+0000 7f841bdbb740 1 -- 172.21.15.90:0/990744867 shutdown_connections 2024-04-25T11:45:54.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.386+0000 7f841bdbb740 1 --2- 172.21.15.90:0/990744867 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x55fa879833a0 0x55fa87e17c80 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.386+0000 7f841bdbb740 1 --2- 172.21.15.90:0/990744867 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa879ef390 0x55fa87d9ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.386+0000 7f83f5ffb640 0 client.19062 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:45:54.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.386+0000 7f841bdbb740 1 --2- 172.21.15.90:0/990744867 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55fa879ce6b0 0x55fa87d9e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.386+0000 7f841bdbb740 1 --2- 172.21.15.90:0/990744867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fa879c64d0 0x55fa87da3600 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.386+0000 7f841bdbb740 1 -- 172.21.15.90:0/990744867 >> 172.21.15.90:0/990744867 conn(0x55fa879e34e0 msgr2=0x55fa87d8b610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.386+0000 7f841bdbb740 1 -- 172.21.15.90:0/990744867 shutdown_connections 2024-04-25T11:45:54.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.386+0000 7f841bdbb740 1 -- 172.21.15.90:0/990744867 wait complete. 2024-04-25T11:45:54.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.386+0000 7f841bdbb740 20 client.19062 trim_cache size 0 max 16384 2024-04-25T11:45:54.380 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:45:54.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.387+0000 7f8414e27640 1 -- 172.21.15.90:0/2743306482 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f83fc0afc60 msgr2=0x7f83fc0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.387+0000 7f8414e27640 1 --2- 172.21.15.90:0/2743306482 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f83fc0afc60 0x7f83fc0b2170 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8410102d70 tx=0x7f8408005c50 comp rx=0 tx=0).stop 2024-04-25T11:45:54.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.387+0000 7f8414e27640 1 -- 172.21.15.90:0/2743306482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8410108e50 msgr2=0x7f841019b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.387+0000 7f8414e27640 1 --2- 172.21.15.90:0/2743306482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8410108e50 0x7f841019b730 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f83f800b450 tx=0x7f83f800b910 comp rx=0 tx=0).stop 2024-04-25T11:45:54.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.387+0000 7f8414e27640 1 -- 172.21.15.90:0/2743306482 shutdown_connections 2024-04-25T11:45:54.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.387+0000 7f8414e27640 1 --2- 172.21.15.90:0/2743306482 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f83fc0afc60 0x7f83fc0b2170 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.387+0000 7f8414e27640 1 --2- 172.21.15.90:0/2743306482 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f841019bc70 0x7f84101a6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.387+0000 7f8414e27640 1 --2- 172.21.15.90:0/2743306482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8410108e50 0x7f841019b730 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.387+0000 7f8414e27640 1 --2- 172.21.15.90:0/2743306482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8410108a40 0x7f841019b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.387+0000 7f8414e27640 1 -- 172.21.15.90:0/2743306482 >> 172.21.15.90:0/2743306482 conn(0x7f8410077880 msgr2=0x7f84101092f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.387+0000 7f8414e27640 1 -- 172.21.15.90:0/2743306482 shutdown_connections 2024-04-25T11:45:54.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.387+0000 7f8414e27640 1 -- 172.21.15.90:0/2743306482 wait complete. 2024-04-25T11:45:54.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T11:45:54.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.509+0000 7f6a62413640 1 -- 172.21.15.90:0/1763960173 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a5c104c70 msgr2=0x7f6a5c105080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.509+0000 7f6a62413640 1 --2- 172.21.15.90:0/1763960173 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a5c104c70 0x7f6a5c105080 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f6a50009900 tx=0x7f6a5002a410 comp rx=0 tx=0).stop 2024-04-25T11:45:54.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.509+0000 7f6a62413640 1 -- 172.21.15.90:0/1763960173 shutdown_connections 2024-04-25T11:45:54.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.509+0000 7f6a62413640 1 --2- 172.21.15.90:0/1763960173 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a5c10e0c0 0x7f6a5c1105a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.509+0000 7f6a62413640 1 --2- 172.21.15.90:0/1763960173 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6a5c10b6d0 0x7f6a5c10daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.509+0000 7f6a62413640 1 --2- 172.21.15.90:0/1763960173 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a5c104c70 0x7f6a5c105080 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.509+0000 7f6a62413640 1 -- 172.21.15.90:0/1763960173 >> 172.21.15.90:0/1763960173 conn(0x7f6a5c078110 msgr2=0x7f6a5c078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.509+0000 7f6a62413640 1 -- 172.21.15.90:0/1763960173 shutdown_connections 2024-04-25T11:45:54.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.509+0000 7f6a62413640 1 -- 172.21.15.90:0/1763960173 wait complete. 2024-04-25T11:45:54.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.510+0000 7f6a62413640 1 Processor -- start 2024-04-25T11:45:54.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.510+0000 7f6a62413640 1 -- start start 2024-04-25T11:45:54.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.510+0000 7f6a62413640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a5c104c70 0x7f6a5c19a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.510+0000 7f6a62413640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a5c10b6d0 0x7f6a5c19acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.510+0000 7f6a62413640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6a5c10e0c0 0x7f6a5c19ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.510+0000 7f6a62413640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5c113ce0 con 0x7f6a5c104c70 2024-04-25T11:45:54.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.510+0000 7f6a62413640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5c113b60 con 0x7f6a5c10e0c0 2024-04-25T11:45:54.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.510+0000 7f6a62413640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5c113e60 con 0x7f6a5c10b6d0 2024-04-25T11:45:54.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.510+0000 7f6a5bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a5c104c70 0x7f6a5c19a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.510+0000 7f6a5bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a5c104c70 0x7f6a5c19a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37996/0 (socket says 172.21.15.90:37996) 2024-04-25T11:45:54.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.510+0000 7f6a5bfff640 1 -- 172.21.15.90:0/602148060 learned_addr learned my addr 172.21.15.90:0/602148060 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:54.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.511+0000 7f6a609b0640 1 --2- 172.21.15.90:0/602148060 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6a5c10e0c0 0x7f6a5c19ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.511+0000 7f6a5bfff640 1 -- 172.21.15.90:0/602148060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a5c10b6d0 msgr2=0x7f6a5c19acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.511+0000 7f6a5b7fe640 1 --2- 172.21.15.90:0/602148060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a5c10b6d0 0x7f6a5c19acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.511+0000 7f6a5bfff640 1 --2- 172.21.15.90:0/602148060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a5c10b6d0 0x7f6a5c19acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.511+0000 7f6a5bfff640 1 -- 172.21.15.90:0/602148060 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6a5c10e0c0 msgr2=0x7f6a5c19ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.511+0000 7f6a5bfff640 1 --2- 172.21.15.90:0/602148060 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6a5c10e0c0 0x7f6a5c19ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.511+0000 7f6a5bfff640 1 -- 172.21.15.90:0/602148060 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a5c19f5c0 con 0x7f6a5c104c70 2024-04-25T11:45:54.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.511+0000 7f6a609b0640 1 --2- 172.21.15.90:0/602148060 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6a5c10e0c0 0x7f6a5c19ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:54.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.511+0000 7f6a5b7fe640 1 --2- 172.21.15.90:0/602148060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a5c10b6d0 0x7f6a5c19acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:54.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.511+0000 7f6a5bfff640 1 --2- 172.21.15.90:0/602148060 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a5c104c70 0x7f6a5c19a760 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f6a50005940 tx=0x7f6a500042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.511+0000 7f6a597fa640 1 -- 172.21.15.90:0/602148060 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5003f030 con 0x7f6a5c104c70 2024-04-25T11:45:54.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.511+0000 7f6a597fa640 1 -- 172.21.15.90:0/602148060 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a50003cb0 con 0x7f6a5c104c70 2024-04-25T11:45:54.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.511+0000 7f6a62413640 1 -- 172.21.15.90:0/602148060 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a5c19f850 con 0x7f6a5c104c70 2024-04-25T11:45:54.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.511+0000 7f6a597fa640 1 -- 172.21.15.90:0/602148060 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a50035070 con 0x7f6a5c104c70 2024-04-25T11:45:54.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.511+0000 7f6a62413640 1 -- 172.21.15.90:0/602148060 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a5c19fd40 con 0x7f6a5c104c70 2024-04-25T11:45:54.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.513+0000 7f6a597fa640 1 -- 172.21.15.90:0/602148060 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f6a50042030 con 0x7f6a5c104c70 2024-04-25T11:45:54.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.514+0000 7f6a693a0740 1 Processor -- start 2024-04-25T11:45:54.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.514+0000 7f6a693a0740 1 -- start start 2024-04-25T11:45:54.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.514+0000 7f6a597fa640 1 --2- 172.21.15.90:0/602148060 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f6a440afc60 0x7f6a440b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.514+0000 7f6a693a0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fef390 0x55d026390810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.514+0000 7f6a693a0740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d025fc64d0 0x55d026390d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.514+0000 7f6a693a0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d025fce6b0 0x55d02639d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.514+0000 7f6a693a0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d0262f5d80 con 0x55d025fef390 2024-04-25T11:45:54.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.514+0000 7f6a693a0740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55d0262afa20 con 0x55d025fc64d0 2024-04-25T11:45:54.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.514+0000 7f6a693a0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d0262f1d40 con 0x55d025fce6b0 2024-04-25T11:45:54.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.514+0000 7f6a5bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fef390 0x55d026390810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.514+0000 7f6a609b0640 1 --2- 172.21.15.90:0/602148060 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f6a440afc60 0x7f6a440b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.514+0000 7f6a5bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fef390 0x55d026390810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38012/0 (socket says 172.21.15.90:38012) 2024-04-25T11:45:54.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.514+0000 7f6a5bfff640 1 -- 172.21.15.90:0/778860856 learned_addr learned my addr 172.21.15.90:0/778860856 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:54.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.514+0000 7f6a597fa640 1 -- 172.21.15.90:0/602148060 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7f6a500ffa70 con 0x7f6a5c104c70 2024-04-25T11:45:54.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.514+0000 7f6a5b7fe640 1 --2- 172.21.15.90:0/778860856 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d025fc64d0 0x55d026390d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.514+0000 7f6a5bfff640 1 -- 172.21.15.90:0/778860856 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d025fce6b0 msgr2=0x55d02639d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.514+0000 7f6a5bfff640 1 --2- 172.21.15.90:0/778860856 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d025fce6b0 0x55d02639d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.514+0000 7f6a5bfff640 1 -- 172.21.15.90:0/778860856 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d025fc64d0 msgr2=0x55d026390d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.514+0000 7f6a5bfff640 1 --2- 172.21.15.90:0/778860856 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d025fc64d0 0x55d026390d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.514+0000 7f6a5bfff640 1 -- 172.21.15.90:0/778860856 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0262bac30 con 0x55d025fef390 2024-04-25T11:45:54.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.514+0000 7f6a5b7fe640 1 --2- 172.21.15.90:0/778860856 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d025fc64d0 0x55d026390d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:54.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.515+0000 7f6a5bfff640 1 --2- 172.21.15.90:0/778860856 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fef390 0x55d026390810 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f6a500c7880 tx=0x7f6a50036e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf0e8491480c2b70 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.515+0000 7f6a609b0640 1 --2- 172.21.15.90:0/602148060 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f6a440afc60 0x7f6a440b2170 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6a54005c40 tx=0x7f6a540097a0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.515+0000 7f6a427fc640 1 -- 172.21.15.90:0/778860856 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a50049040 con 0x55d025fef390 2024-04-25T11:45:54.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.515+0000 7f6a427fc640 1 -- 172.21.15.90:0/778860856 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a500d0330 con 0x55d025fef390 2024-04-25T11:45:54.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.515+0000 7f6a427fc640 1 -- 172.21.15.90:0/778860856 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a500c8280 con 0x55d025fef390 2024-04-25T11:45:54.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.516+0000 7f6a693a0740 1 -- 172.21.15.90:0/778860856 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fef390 msgr2=0x55d026390810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.516+0000 7f6a693a0740 1 --2- 172.21.15.90:0/778860856 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fef390 0x55d026390810 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f6a500c7880 tx=0x7f6a50036e20 comp rx=0 tx=0).stop 2024-04-25T11:45:54.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.516+0000 7f6a693a0740 1 -- 172.21.15.90:0/778860856 shutdown_connections 2024-04-25T11:45:54.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.516+0000 7f6a693a0740 1 --2- 172.21.15.90:0/778860856 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d025fce6b0 0x55d02639d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.516+0000 7f6a693a0740 1 --2- 172.21.15.90:0/778860856 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d025fc64d0 0x55d026390d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.516+0000 7f6a693a0740 1 --2- 172.21.15.90:0/778860856 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fef390 0x55d026390810 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.516+0000 7f6a693a0740 1 -- 172.21.15.90:0/778860856 >> 172.21.15.90:0/778860856 conn(0x55d025fe34e0 msgr2=0x55d02638cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.516+0000 7f6a693a0740 1 -- 172.21.15.90:0/778860856 shutdown_connections 2024-04-25T11:45:54.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.516+0000 7f6a693a0740 1 -- 172.21.15.90:0/778860856 wait complete. 2024-04-25T11:45:54.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.516+0000 7f6a693a0740 1 -- start start 2024-04-25T11:45:54.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.516+0000 7f6a693a0740 1 Processor -- start 2024-04-25T11:45:54.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.517+0000 7f6a693a0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d025fc64d0 0x55d02639b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.517+0000 7f6a693a0740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d025fce6b0 0x55d02639b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.517+0000 7f6a693a0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fef390 0x55d0263a7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.517+0000 7f6a693a0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d02639dbe0 con 0x55d025fef390 2024-04-25T11:45:54.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.517+0000 7f6a693a0740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55d02627cdc0 con 0x55d025fce6b0 2024-04-25T11:45:54.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.517+0000 7f6a693a0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d02639dd60 con 0x55d025fc64d0 2024-04-25T11:45:54.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.517+0000 7f6a693a0740 10 client.0 _ll_register_callbacks cb 0x55d026298600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:54.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.517+0000 7f6a609b0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fef390 0x55d0263a7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.517+0000 7f6a609b0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fef390 0x55d0263a7010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38028/0 (socket says 172.21.15.90:38028) 2024-04-25T11:45:54.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.517+0000 7f6a609b0640 1 -- 172.21.15.90:0/997516609 learned_addr learned my addr 172.21.15.90:0/997516609 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:54.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.517+0000 7f6a5b7fe640 1 --2- 172.21.15.90:0/997516609 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d025fce6b0 0x55d02639b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.517+0000 7f6a5bfff640 1 --2- 172.21.15.90:0/997516609 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d025fc64d0 0x55d02639b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.517+0000 7f6a609b0640 1 -- 172.21.15.90:0/997516609 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d025fc64d0 msgr2=0x55d02639b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.517+0000 7f6a609b0640 1 --2- 172.21.15.90:0/997516609 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d025fc64d0 0x55d02639b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.517+0000 7f6a609b0640 1 -- 172.21.15.90:0/997516609 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d025fce6b0 msgr2=0x55d02639b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.517+0000 7f6a609b0640 1 --2- 172.21.15.90:0/997516609 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d025fce6b0 0x55d02639b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.517+0000 7f6a609b0640 1 -- 172.21.15.90:0/997516609 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0260073f0 con 0x55d025fef390 2024-04-25T11:45:54.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.517+0000 7f6a609b0640 1 --2- 172.21.15.90:0/997516609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fef390 0x55d0263a7010 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f6a54012710 tx=0x7f6a54004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42da4b22878daaa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.517+0000 7f6a3b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:54.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.517+0000 7f6a3b7fe640 1 -- 172.21.15.90:0/997516609 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a54023070 con 0x55d025fef390 2024-04-25T11:45:54.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.517+0000 7f6a3b7fe640 1 -- 172.21.15.90:0/997516609 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7f6a5400ea60 con 0x55d025fef390 2024-04-25T11:45:54.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.517+0000 7f6a693a0740 1 -- 172.21.15.90:0/997516609 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a500d0330 con 0x55d025fef390 2024-04-25T11:45:54.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.518+0000 7f6a3b7fe640 1 -- 172.21.15.90:0/997516609 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6a5402f050 con 0x55d025fef390 2024-04-25T11:45:54.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.518+0000 7f6a693a0740 10 client.19074 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:54.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.518+0000 7f6a693a0740 1 -- 172.21.15.90:0/997516609 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d0260d17d0 con 0x55d025fef390 2024-04-25T11:45:54.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.518+0000 7f6a3b7fe640 1 -- 172.21.15.90:0/997516609 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a500d0330 con 0x55d025fef390 2024-04-25T11:45:54.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.518+0000 7f6a3b7fe640 1 -- 172.21.15.90:0/997516609 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f6a540273b0 con 0x55d025fef390 2024-04-25T11:45:54.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.518+0000 7f6a693a0740 10 client.19074 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:54.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.518+0000 7f6a693a0740 10 client.19074 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:45:54.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.518+0000 7f6a693a0740 20 client.19074 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:54.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.518+0000 7f6a693a0740 1 --2- 172.21.15.90:0/997516609 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55d025fadf60 0x55d0263ad430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.518+0000 7f6a693a0740 4 client.19074 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:45:54.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.518+0000 7f6a693a0740 1 -- 172.21.15.90:0/997516609 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d026303990 con 0x55d025fadf60 2024-04-25T11:45:54.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.518+0000 7f6a5bfff640 1 --2- 172.21.15.90:0/997516609 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55d025fadf60 0x55d0263ad430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.519+0000 7f6a3b7fe640 10 client.19074 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:45:54.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.519+0000 7f6a5bfff640 1 --2- 172.21.15.90:0/997516609 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55d025fadf60 0x55d0263ad430 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5f1d0228d67f0303 server_cookie=ee2b7dbd99ee17b7 in_seq=0 out_seq=0 2024-04-25T11:45:54.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.519+0000 7f6a3b7fe640 1 -- 172.21.15.90:0/997516609 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d026303990 con 0x55d025fadf60 2024-04-25T11:45:54.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.519+0000 7f6a3b7fe640 10 client.19074 handle_command_reply: tid=0 2024-04-25T11:45:54.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.519+0000 7f6a693a0740 1 client.19074 shutdown 2024-04-25T11:45:54.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.520+0000 7f6a693a0740 1 -- 172.21.15.90:0/997516609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fef390 msgr2=0x55d0263a7010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.520+0000 7f6a693a0740 1 --2- 172.21.15.90:0/997516609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fef390 0x55d0263a7010 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f6a54012710 tx=0x7f6a54004ab0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.520+0000 7f6a693a0740 1 -- 172.21.15.90:0/997516609 shutdown_connections 2024-04-25T11:45:54.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.520+0000 7f6a693a0740 1 --2- 172.21.15.90:0/997516609 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55d025fadf60 0x55d0263ad430 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.520+0000 7f6a693a0740 1 --2- 172.21.15.90:0/997516609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fef390 0x55d0263a7010 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.520+0000 7f6a693a0740 1 --2- 172.21.15.90:0/997516609 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d025fce6b0 0x55d02639b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.520+0000 7f6a693a0740 1 --2- 172.21.15.90:0/997516609 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d025fc64d0 0x55d02639b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.520+0000 7f6a3b7fe640 0 client.19074 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:45:54.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.520+0000 7f6a693a0740 1 -- 172.21.15.90:0/997516609 >> 172.21.15.90:0/997516609 conn(0x55d025fe34e0 msgr2=0x55d026392530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.520+0000 7f6a693a0740 1 -- 172.21.15.90:0/997516609 shutdown_connections 2024-04-25T11:45:54.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.520+0000 7f6a693a0740 1 -- 172.21.15.90:0/997516609 wait complete. 2024-04-25T11:45:54.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.520+0000 7f6a693a0740 20 client.19074 trim_cache size 0 max 16384 2024-04-25T11:45:54.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.533+0000 7f6a693a0740 1 Processor -- start 2024-04-25T11:45:54.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.533+0000 7f6a693a0740 1 -- start start 2024-04-25T11:45:54.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.533+0000 7f6a693a0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fc64d0 0x55d02638d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.533+0000 7f6a693a0740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d025fce6b0 0x55d0263a6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.533+0000 7f6a693a0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d025fef390 0x55d0263b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.533+0000 7f6a693a0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5c1afcb0 con 0x55d025fc64d0 2024-04-25T11:45:54.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.533+0000 7f6a693a0740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5c1b17f0 con 0x55d025fce6b0 2024-04-25T11:45:54.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.533+0000 7f6a693a0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5c1b0380 con 0x55d025fef390 2024-04-25T11:45:54.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.533+0000 7f6a5bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fc64d0 0x55d02638d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.534+0000 7f6a5bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fc64d0 0x55d02638d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38044/0 (socket says 172.21.15.90:38044) 2024-04-25T11:45:54.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.534+0000 7f6a5bfff640 1 -- 172.21.15.90:0/4260930771 learned_addr learned my addr 172.21.15.90:0/4260930771 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:54.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.534+0000 7f6a5b7fe640 1 --2- 172.21.15.90:0/4260930771 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d025fce6b0 0x55d0263a6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.534+0000 7f6a5bfff640 1 -- 172.21.15.90:0/4260930771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d025fef390 msgr2=0x55d0263b2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.534+0000 7f6a5bfff640 1 --2- 172.21.15.90:0/4260930771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d025fef390 0x55d0263b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.534+0000 7f6a5bfff640 1 -- 172.21.15.90:0/4260930771 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d025fce6b0 msgr2=0x55d0263a6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.534+0000 7f6a5bfff640 1 --2- 172.21.15.90:0/4260930771 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d025fce6b0 0x55d0263a6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.534+0000 7f6a5bfff640 1 -- 172.21.15.90:0/4260930771 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d026392010 con 0x55d025fc64d0 2024-04-25T11:45:54.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.534+0000 7f6a5bfff640 1 --2- 172.21.15.90:0/4260930771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fc64d0 0x55d02638d090 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x55d026391590 tx=0x7f6a500ea040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5857bffcf09d5759 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.534+0000 7f6a41ffb640 1 -- 172.21.15.90:0/4260930771 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a500a9c70 con 0x55d025fc64d0 2024-04-25T11:45:54.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.534+0000 7f6a41ffb640 1 -- 172.21.15.90:0/4260930771 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a500c7b80 con 0x55d025fc64d0 2024-04-25T11:45:54.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.534+0000 7f6a41ffb640 1 -- 172.21.15.90:0/4260930771 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a50049040 con 0x55d025fc64d0 2024-04-25T11:45:54.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.535+0000 7f6a693a0740 1 -- 172.21.15.90:0/4260930771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fc64d0 msgr2=0x55d02638d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.535+0000 7f6a693a0740 1 --2- 172.21.15.90:0/4260930771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fc64d0 0x55d02638d090 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x55d026391590 tx=0x7f6a500ea040 comp rx=0 tx=0).stop 2024-04-25T11:45:54.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.535+0000 7f6a693a0740 1 -- 172.21.15.90:0/4260930771 shutdown_connections 2024-04-25T11:45:54.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.535+0000 7f6a693a0740 1 --2- 172.21.15.90:0/4260930771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d025fef390 0x55d0263b2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.535+0000 7f6a693a0740 1 --2- 172.21.15.90:0/4260930771 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d025fce6b0 0x55d0263a6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.535+0000 7f6a693a0740 1 --2- 172.21.15.90:0/4260930771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fc64d0 0x55d02638d090 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.535+0000 7f6a693a0740 1 -- 172.21.15.90:0/4260930771 >> 172.21.15.90:0/4260930771 conn(0x55d025fe34e0 msgr2=0x55d02638c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.535+0000 7f6a693a0740 1 -- 172.21.15.90:0/4260930771 shutdown_connections 2024-04-25T11:45:54.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.535+0000 7f6a693a0740 1 -- 172.21.15.90:0/4260930771 wait complete. 2024-04-25T11:45:54.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.535+0000 7f6a693a0740 1 -- start start 2024-04-25T11:45:54.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.535+0000 7f6a693a0740 1 Processor -- start 2024-04-25T11:45:54.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.536+0000 7f6a693a0740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d025fc64d0 0x55d0263a34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.536+0000 7f6a693a0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fce6b0 0x55d02639e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.536+0000 7f6a693a0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d025fef390 0x55d02639edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.536+0000 7f6a693a0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5c1b1120 con 0x55d025fce6b0 2024-04-25T11:45:54.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.536+0000 7f6a693a0740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5c1b0a50 con 0x55d025fc64d0 2024-04-25T11:45:54.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.536+0000 7f6a693a0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5c1adcb0 con 0x55d025fef390 2024-04-25T11:45:54.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.536+0000 7f6a693a0740 10 client.0 _ll_register_callbacks cb 0x7f6a54009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:54.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.536+0000 7f6a5b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fce6b0 0x55d02639e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.536+0000 7f6a5b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fce6b0 0x55d02639e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38050/0 (socket says 172.21.15.90:38050) 2024-04-25T11:45:54.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.536+0000 7f6a5b7fe640 1 -- 172.21.15.90:0/4257388889 learned_addr learned my addr 172.21.15.90:0/4257388889 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:54.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.536+0000 7f6a5bfff640 1 --2- 172.21.15.90:0/4257388889 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d025fc64d0 0x55d0263a34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.536+0000 7f6a609b0640 1 --2- 172.21.15.90:0/4257388889 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d025fef390 0x55d02639edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.536+0000 7f6a5b7fe640 1 -- 172.21.15.90:0/4257388889 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d025fef390 msgr2=0x55d02639edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.536+0000 7f6a5b7fe640 1 --2- 172.21.15.90:0/4257388889 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d025fef390 0x55d02639edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.536+0000 7f6a5b7fe640 1 -- 172.21.15.90:0/4257388889 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d025fc64d0 msgr2=0x55d0263a34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.536+0000 7f6a5b7fe640 1 --2- 172.21.15.90:0/4257388889 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d025fc64d0 0x55d0263a34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.536+0000 7f6a5b7fe640 1 -- 172.21.15.90:0/4257388889 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d02639f4a0 con 0x55d025fce6b0 2024-04-25T11:45:54.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.536+0000 7f6a609b0640 1 --2- 172.21.15.90:0/4257388889 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d025fef390 0x55d02639edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:54.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.536+0000 7f6a5b7fe640 1 --2- 172.21.15.90:0/4257388889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fce6b0 0x55d02639e870 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c0058c0 tx=0x7f6a4c00f8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b67501f99dd806f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.536+0000 7f6a3b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:54.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.536+0000 7f6a3b7fe640 1 -- 172.21.15.90:0/4257388889 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c01f060 con 0x55d025fce6b0 2024-04-25T11:45:54.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.536+0000 7f6a3b7fe640 1 -- 172.21.15.90:0/4257388889 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7f6a4c00e8e0 con 0x55d025fce6b0 2024-04-25T11:45:54.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.536+0000 7f6a693a0740 1 -- 172.21.15.90:0/4257388889 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a500c7b80 con 0x55d025fce6b0 2024-04-25T11:45:54.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.537+0000 7f6a3b7fe640 1 -- 172.21.15.90:0/4257388889 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6a4c026050 con 0x55d025fce6b0 2024-04-25T11:45:54.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.537+0000 7f6a693a0740 10 client.19080 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:54.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.537+0000 7f6a693a0740 1 -- 172.21.15.90:0/4257388889 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d026415690 con 0x55d025fce6b0 2024-04-25T11:45:54.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.537+0000 7f6a3b7fe640 1 -- 172.21.15.90:0/4257388889 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a500c7b80 con 0x55d025fce6b0 2024-04-25T11:45:54.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.537+0000 7f6a3b7fe640 1 -- 172.21.15.90:0/4257388889 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f6a4c019bf0 con 0x55d025fce6b0 2024-04-25T11:45:54.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.537+0000 7f6a693a0740 10 client.19080 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:54.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.537+0000 7f6a693a0740 10 client.19080 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:45:54.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.537+0000 7f6a693a0740 20 client.19080 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:54.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.537+0000 7f6a693a0740 1 --2- 172.21.15.90:0/4257388889 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55d025f833a0 0x55d026417eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.537+0000 7f6a693a0740 4 client.19080 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:45:54.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.537+0000 7f6a693a0740 1 -- 172.21.15.90:0/4257388889 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d0262fd800 con 0x55d025f833a0 2024-04-25T11:45:54.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.537+0000 7f6a5bfff640 1 --2- 172.21.15.90:0/4257388889 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55d025f833a0 0x55d026417eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.538+0000 7f6a3b7fe640 10 client.19080 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:45:54.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.538+0000 7f6a5bfff640 1 --2- 172.21.15.90:0/4257388889 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55d025f833a0 0x55d026417eb0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=396aa7e52cbfb709 server_cookie=15b80069ef66b492 in_seq=0 out_seq=0 2024-04-25T11:45:54.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.538+0000 7f6a3b7fe640 1 -- 172.21.15.90:0/4257388889 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d0262fd800 con 0x55d025f833a0 2024-04-25T11:45:54.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.538+0000 7f6a3b7fe640 10 client.19080 handle_command_reply: tid=0 2024-04-25T11:45:54.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.538+0000 7f6a693a0740 1 client.19080 shutdown 2024-04-25T11:45:54.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.538+0000 7f6a693a0740 1 -- 172.21.15.90:0/4257388889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fce6b0 msgr2=0x55d02639e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.538+0000 7f6a693a0740 1 --2- 172.21.15.90:0/4257388889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fce6b0 0x55d02639e870 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c0058c0 tx=0x7f6a4c00f8a0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.538+0000 7f6a693a0740 1 -- 172.21.15.90:0/4257388889 shutdown_connections 2024-04-25T11:45:54.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.538+0000 7f6a693a0740 1 --2- 172.21.15.90:0/4257388889 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55d025f833a0 0x55d026417eb0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.538+0000 7f6a693a0740 1 --2- 172.21.15.90:0/4257388889 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d025fef390 0x55d02639edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.538+0000 7f6a693a0740 1 --2- 172.21.15.90:0/4257388889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d025fce6b0 0x55d02639e870 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.538+0000 7f6a693a0740 1 --2- 172.21.15.90:0/4257388889 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d025fc64d0 0x55d0263a34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.538+0000 7f6a3b7fe640 0 client.19080 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:45:54.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.538+0000 7f6a693a0740 1 -- 172.21.15.90:0/4257388889 >> 172.21.15.90:0/4257388889 conn(0x55d025fe34e0 msgr2=0x55d026399940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.538+0000 7f6a693a0740 1 -- 172.21.15.90:0/4257388889 shutdown_connections 2024-04-25T11:45:54.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.538+0000 7f6a693a0740 1 -- 172.21.15.90:0/4257388889 wait complete. 2024-04-25T11:45:54.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.538+0000 7f6a693a0740 20 client.19080 trim_cache size 0 max 16384 2024-04-25T11:45:54.531 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:45:54.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.539+0000 7f6a62413640 1 -- 172.21.15.90:0/602148060 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f6a440afc60 msgr2=0x7f6a440b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.539+0000 7f6a62413640 1 --2- 172.21.15.90:0/602148060 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f6a440afc60 0x7f6a440b2170 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6a54005c40 tx=0x7f6a540097a0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.539+0000 7f6a62413640 1 -- 172.21.15.90:0/602148060 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a5c104c70 msgr2=0x7f6a5c19a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.539+0000 7f6a62413640 1 --2- 172.21.15.90:0/602148060 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a5c104c70 0x7f6a5c19a760 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f6a50005940 tx=0x7f6a500042f0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.539+0000 7f6a62413640 1 -- 172.21.15.90:0/602148060 shutdown_connections 2024-04-25T11:45:54.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.539+0000 7f6a62413640 1 --2- 172.21.15.90:0/602148060 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f6a440afc60 0x7f6a440b2170 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.539+0000 7f6a62413640 1 --2- 172.21.15.90:0/602148060 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6a5c10e0c0 0x7f6a5c19ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.539+0000 7f6a62413640 1 --2- 172.21.15.90:0/602148060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a5c10b6d0 0x7f6a5c19acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.539+0000 7f6a62413640 1 --2- 172.21.15.90:0/602148060 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a5c104c70 0x7f6a5c19a760 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.539+0000 7f6a62413640 1 -- 172.21.15.90:0/602148060 >> 172.21.15.90:0/602148060 conn(0x7f6a5c078110 msgr2=0x7f6a5c101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.539+0000 7f6a62413640 1 -- 172.21.15.90:0/602148060 shutdown_connections 2024-04-25T11:45:54.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.539+0000 7f6a62413640 1 -- 172.21.15.90:0/602148060 wait complete. 2024-04-25T11:45:54.536 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T11:45:54.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.660+0000 7f673a572640 1 -- 172.21.15.90:0/422924260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67341033a0 msgr2=0x7f67341037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.660+0000 7f673a572640 1 --2- 172.21.15.90:0/422924260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67341033a0 0x7f67341037b0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f6724009900 tx=0x7f672402a410 comp rx=0 tx=0).stop 2024-04-25T11:45:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.660+0000 7f673a572640 1 -- 172.21.15.90:0/422924260 shutdown_connections 2024-04-25T11:45:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.660+0000 7f673a572640 1 --2- 172.21.15.90:0/422924260 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6734107ce0 0x7f673410f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.660+0000 7f673a572640 1 --2- 172.21.15.90:0/422924260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6734107260 0x7f6734107670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.660+0000 7f673a572640 1 --2- 172.21.15.90:0/422924260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67341033a0 0x7f67341037b0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.660+0000 7f673a572640 1 -- 172.21.15.90:0/422924260 >> 172.21.15.90:0/422924260 conn(0x7f6734077880 msgr2=0x7f6734077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.660+0000 7f673a572640 1 -- 172.21.15.90:0/422924260 shutdown_connections 2024-04-25T11:45:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.660+0000 7f673a572640 1 -- 172.21.15.90:0/422924260 wait complete. 2024-04-25T11:45:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.660+0000 7f673a572640 1 Processor -- start 2024-04-25T11:45:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.660+0000 7f673a572640 1 -- start start 2024-04-25T11:45:54.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.661+0000 7f673a572640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f67341033a0 0x7f673419a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.661+0000 7f673a572640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6734107260 0x7f673419ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.661+0000 7f673a572640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6734107ce0 0x7f673419ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.661+0000 7f673a572640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6734112be0 con 0x7f6734107ce0 2024-04-25T11:45:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.661+0000 7f673a572640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6734112a60 con 0x7f67341033a0 2024-04-25T11:45:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.661+0000 7f673a572640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6734112d60 con 0x7f6734107260 2024-04-25T11:45:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.661+0000 7f6738b0f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6734107ce0 0x7f673419ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.661+0000 7f6738b0f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6734107ce0 0x7f673419ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38066/0 (socket says 172.21.15.90:38066) 2024-04-25T11:45:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.661+0000 7f6738b0f640 1 -- 172.21.15.90:0/4216303675 learned_addr learned my addr 172.21.15.90:0/4216303675 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.661+0000 7f6733fff640 1 --2- 172.21.15.90:0/4216303675 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f67341033a0 0x7f673419a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.661+0000 7f67337fe640 1 --2- 172.21.15.90:0/4216303675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6734107260 0x7f673419ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.661+0000 7f6738b0f640 1 -- 172.21.15.90:0/4216303675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6734107260 msgr2=0x7f673419ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.661+0000 7f6738b0f640 1 --2- 172.21.15.90:0/4216303675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6734107260 0x7f673419ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.661+0000 7f6738b0f640 1 -- 172.21.15.90:0/4216303675 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f67341033a0 msgr2=0x7f673419a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.661+0000 7f6738b0f640 1 --2- 172.21.15.90:0/4216303675 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f67341033a0 0x7f673419a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.661+0000 7f6738b0f640 1 -- 172.21.15.90:0/4216303675 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f673419f4d0 con 0x7f6734107ce0 2024-04-25T11:45:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.661+0000 7f6733fff640 1 --2- 172.21.15.90:0/4216303675 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f67341033a0 0x7f673419a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.661+0000 7f6738b0f640 1 --2- 172.21.15.90:0/4216303675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6734107ce0 0x7f673419ef10 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f672c00ead0 tx=0x7f672c00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.662+0000 7f67317fa640 1 -- 172.21.15.90:0/4216303675 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672c00cde0 con 0x7f6734107ce0 2024-04-25T11:45:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.662+0000 7f67317fa640 1 -- 172.21.15.90:0/4216303675 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f672c005020 con 0x7f6734107ce0 2024-04-25T11:45:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.662+0000 7f673a572640 1 -- 172.21.15.90:0/4216303675 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f673419f7c0 con 0x7f6734107ce0 2024-04-25T11:45:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.662+0000 7f67317fa640 1 -- 172.21.15.90:0/4216303675 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672c0116d0 con 0x7f6734107ce0 2024-04-25T11:45:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.662+0000 7f673a572640 1 -- 172.21.15.90:0/4216303675 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f673419fd10 con 0x7f6734107ce0 2024-04-25T11:45:54.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.664+0000 7f67317fa640 1 -- 172.21.15.90:0/4216303675 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f672c0051d0 con 0x7f6734107ce0 2024-04-25T11:45:54.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.664+0000 7f6741429740 1 Processor -- start 2024-04-25T11:45:54.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.664+0000 7f6741429740 1 -- start start 2024-04-25T11:45:54.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.664+0000 7f6741429740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c733fff390 0x55c7343a0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.664+0000 7f6741429740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c733fd64d0 0x55c7343a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.664+0000 7f6741429740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fde6b0 0x55c7343ad6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.664+0000 7f6741429740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c734305d80 con 0x55c733fde6b0 2024-04-25T11:45:54.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.664+0000 7f6741429740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c7342bfa20 con 0x55c733fff390 2024-04-25T11:45:54.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.664+0000 7f6741429740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c734301d40 con 0x55c733fd64d0 2024-04-25T11:45:54.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.664+0000 7f67337fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fde6b0 0x55c7343ad6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.664+0000 7f67337fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fde6b0 0x55c7343ad6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38068/0 (socket says 172.21.15.90:38068) 2024-04-25T11:45:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.664+0000 7f67337fe640 1 -- 172.21.15.90:0/4083652926 learned_addr learned my addr 172.21.15.90:0/4083652926 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.665+0000 7f67337fe640 1 --2- 172.21.15.90:0/4083652926 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c733fff390 0x55c7343a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.665+0000 7f67317fa640 1 --2- 172.21.15.90:0/4216303675 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f671c0afce0 0x7f671c0b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.665+0000 7f6733fff640 1 --2- 172.21.15.90:0/4083652926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c733fd64d0 0x55c7343a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.665+0000 7f67337fe640 1 -- 172.21.15.90:0/4083652926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c733fd64d0 msgr2=0x55c7343a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.665+0000 7f67337fe640 1 --2- 172.21.15.90:0/4083652926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c733fd64d0 0x55c7343a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.665+0000 7f6738b0f640 1 --2- 172.21.15.90:0/4216303675 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f671c0afce0 0x7f671c0b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.665+0000 7f67337fe640 1 -- 172.21.15.90:0/4083652926 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c733fff390 msgr2=0x55c7343a0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.665+0000 7f67337fe640 1 --2- 172.21.15.90:0/4083652926 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c733fff390 0x55c7343a0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.665+0000 7f67337fe640 1 -- 172.21.15.90:0/4083652926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7342cac30 con 0x55c733fde6b0 2024-04-25T11:45:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.665+0000 7f67317fa640 1 -- 172.21.15.90:0/4216303675 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7f672c0df2d0 con 0x7f6734107ce0 2024-04-25T11:45:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.665+0000 7f67337fe640 1 --2- 172.21.15.90:0/4083652926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fde6b0 0x55c7343ad6a0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f6728005fd0 tx=0x7f672800d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c175298ee025459e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.665+0000 7f671a7fc640 1 -- 172.21.15.90:0/4083652926 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67280117e0 con 0x55c733fde6b0 2024-04-25T11:45:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.665+0000 7f671a7fc640 1 -- 172.21.15.90:0/4083652926 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6728011e30 con 0x55c733fde6b0 2024-04-25T11:45:54.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.665+0000 7f671a7fc640 1 -- 172.21.15.90:0/4083652926 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6728012b30 con 0x55c733fde6b0 2024-04-25T11:45:54.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.666+0000 7f6741429740 1 -- 172.21.15.90:0/4083652926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fde6b0 msgr2=0x55c7343ad6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.666+0000 7f6741429740 1 --2- 172.21.15.90:0/4083652926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fde6b0 0x55c7343ad6a0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f6728005fd0 tx=0x7f672800d360 comp rx=0 tx=0).stop 2024-04-25T11:45:54.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.666+0000 7f6738b0f640 1 --2- 172.21.15.90:0/4216303675 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f671c0afce0 0x7f671c0b21f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f672c0a7370 tx=0x7f672c0a2b80 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.666+0000 7f6741429740 1 -- 172.21.15.90:0/4083652926 shutdown_connections 2024-04-25T11:45:54.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.666+0000 7f6741429740 1 --2- 172.21.15.90:0/4083652926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fde6b0 0x55c7343ad6a0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.666+0000 7f6741429740 1 --2- 172.21.15.90:0/4083652926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c733fd64d0 0x55c7343a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.666+0000 7f6741429740 1 --2- 172.21.15.90:0/4083652926 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c733fff390 0x55c7343a0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.666+0000 7f6741429740 1 -- 172.21.15.90:0/4083652926 >> 172.21.15.90:0/4083652926 conn(0x55c733ff34e0 msgr2=0x55c73439cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.666+0000 7f6741429740 1 -- 172.21.15.90:0/4083652926 shutdown_connections 2024-04-25T11:45:54.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.666+0000 7f6741429740 1 -- 172.21.15.90:0/4083652926 wait complete. 2024-04-25T11:45:54.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.666+0000 7f6741429740 1 -- start start 2024-04-25T11:45:54.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.667+0000 7f6741429740 1 Processor -- start 2024-04-25T11:45:54.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.667+0000 7f6741429740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fd64d0 0x55c7343ab280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.667+0000 7f6741429740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c733fde6b0 0x55c7343ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.667+0000 7f6741429740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c733fff390 0x55c7343b7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.667+0000 7f6741429740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c7343add90 con 0x55c733fd64d0 2024-04-25T11:45:54.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.667+0000 7f6741429740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c73428cdc0 con 0x55c733fde6b0 2024-04-25T11:45:54.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.667+0000 7f6741429740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c7343adf10 con 0x55c733fff390 2024-04-25T11:45:54.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.667+0000 7f6741429740 10 client.0 _ll_register_callbacks cb 0x55c7342a8600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:54.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.667+0000 7f67337fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fd64d0 0x55c7343ab280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.667+0000 7f67337fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fd64d0 0x55c7343ab280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38078/0 (socket says 172.21.15.90:38078) 2024-04-25T11:45:54.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.667+0000 7f67337fe640 1 -- 172.21.15.90:0/1847907310 learned_addr learned my addr 172.21.15.90:0/1847907310 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:54.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.667+0000 7f67337fe640 1 --2- 172.21.15.90:0/1847907310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c733fff390 0x55c7343b7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.667+0000 7f6733fff640 1 --2- 172.21.15.90:0/1847907310 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c733fde6b0 0x55c7343ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.667+0000 7f67337fe640 1 -- 172.21.15.90:0/1847907310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c733fff390 msgr2=0x55c7343b7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.667+0000 7f67337fe640 1 --2- 172.21.15.90:0/1847907310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c733fff390 0x55c7343b7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.667+0000 7f67337fe640 1 -- 172.21.15.90:0/1847907310 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c733fde6b0 msgr2=0x55c7343ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.667+0000 7f67337fe640 1 --2- 172.21.15.90:0/1847907310 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c733fde6b0 0x55c7343ab7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.667+0000 7f67337fe640 1 -- 172.21.15.90:0/1847907310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7340173f0 con 0x55c733fd64d0 2024-04-25T11:45:54.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.668+0000 7f67337fe640 1 --2- 172.21.15.90:0/1847907310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fd64d0 0x55c7343ab280 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f6728012810 tx=0x7f6728005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11fbd27f0f8229cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.668+0000 7f67137fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:54.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.668+0000 7f67137fe640 1 -- 172.21.15.90:0/1847907310 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67280119f0 con 0x55c733fd64d0 2024-04-25T11:45:54.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.668+0000 7f67137fe640 1 -- 172.21.15.90:0/1847907310 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7f6728011b90 con 0x55c733fd64d0 2024-04-25T11:45:54.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.668+0000 7f6741429740 1 -- 172.21.15.90:0/1847907310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6728011e30 con 0x55c733fd64d0 2024-04-25T11:45:54.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.668+0000 7f67137fe640 1 -- 172.21.15.90:0/1847907310 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f67280128b0 con 0x55c733fd64d0 2024-04-25T11:45:54.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.668+0000 7f6741429740 10 client.19092 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:54.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.668+0000 7f6741429740 1 -- 172.21.15.90:0/1847907310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c7340e17d0 con 0x55c733fd64d0 2024-04-25T11:45:54.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.668+0000 7f67137fe640 1 -- 172.21.15.90:0/1847907310 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6728011e30 con 0x55c733fd64d0 2024-04-25T11:45:54.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.668+0000 7f67137fe640 1 -- 172.21.15.90:0/1847907310 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f6728030030 con 0x55c733fd64d0 2024-04-25T11:45:54.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.668+0000 7f6741429740 10 client.19092 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:54.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.668+0000 7f6741429740 10 client.19092 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:45:54.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.668+0000 7f6741429740 20 client.19092 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:54.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.668+0000 7f6741429740 1 --2- 172.21.15.90:0/1847907310 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c733fbdf60 0x55c7343bd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.668+0000 7f6741429740 4 client.19092 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:45:54.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.668+0000 7f6741429740 1 -- 172.21.15.90:0/1847907310 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c734313990 con 0x55c733fbdf60 2024-04-25T11:45:54.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.669+0000 7f6733fff640 1 --2- 172.21.15.90:0/1847907310 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c733fbdf60 0x55c7343bd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.669+0000 7f67137fe640 10 client.19092 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:54.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.669+0000 7f6733fff640 1 --2- 172.21.15.90:0/1847907310 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c733fbdf60 0x55c7343bd550 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6116757a8e9d50be server_cookie=3a5cac4d4af63611 in_seq=0 out_seq=0 2024-04-25T11:45:54.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.670+0000 7f67137fe640 1 -- 172.21.15.90:0/1847907310 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c734313990 con 0x55c733fbdf60 2024-04-25T11:45:54.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.670+0000 7f67137fe640 10 client.19092 handle_command_reply: tid=0 2024-04-25T11:45:54.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.670+0000 7f6741429740 1 client.19092 shutdown 2024-04-25T11:45:54.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.670+0000 7f6741429740 1 -- 172.21.15.90:0/1847907310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fd64d0 msgr2=0x55c7343ab280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.670+0000 7f6741429740 1 --2- 172.21.15.90:0/1847907310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fd64d0 0x55c7343ab280 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f6728012810 tx=0x7f6728005850 comp rx=0 tx=0).stop 2024-04-25T11:45:54.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.670+0000 7f6741429740 1 -- 172.21.15.90:0/1847907310 shutdown_connections 2024-04-25T11:45:54.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.670+0000 7f6741429740 1 --2- 172.21.15.90:0/1847907310 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c733fbdf60 0x55c7343bd550 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.670+0000 7f6741429740 1 --2- 172.21.15.90:0/1847907310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c733fff390 0x55c7343b7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.670+0000 7f6741429740 1 --2- 172.21.15.90:0/1847907310 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c733fde6b0 0x55c7343ab7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.670+0000 7f67137fe640 0 client.19092 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:54.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.670+0000 7f6741429740 1 --2- 172.21.15.90:0/1847907310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fd64d0 0x55c7343ab280 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.670+0000 7f6741429740 1 -- 172.21.15.90:0/1847907310 >> 172.21.15.90:0/1847907310 conn(0x55c733ff34e0 msgr2=0x55c7343a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.670+0000 7f6741429740 1 -- 172.21.15.90:0/1847907310 shutdown_connections 2024-04-25T11:45:54.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.670+0000 7f6741429740 1 -- 172.21.15.90:0/1847907310 wait complete. 2024-04-25T11:45:54.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.670+0000 7f6741429740 20 client.19092 trim_cache size 0 max 16384 2024-04-25T11:45:54.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.683+0000 7f6741429740 1 Processor -- start 2024-04-25T11:45:54.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.683+0000 7f6741429740 1 -- start start 2024-04-25T11:45:54.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.684+0000 7f6741429740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fd64d0 0x55c7343b6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.684+0000 7f6741429740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c733fde6b0 0x55c73439bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.684+0000 7f6741429740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c733fff390 0x55c73439c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.684+0000 7f6741429740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f67341afc00 con 0x55c733fd64d0 2024-04-25T11:45:54.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.684+0000 7f6741429740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f67341b1740 con 0x55c733fde6b0 2024-04-25T11:45:54.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.684+0000 7f6741429740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f67341b02d0 con 0x55c733fff390 2024-04-25T11:45:54.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.684+0000 7f67337fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fd64d0 0x55c7343b6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.684+0000 7f67337fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fd64d0 0x55c7343b6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38088/0 (socket says 172.21.15.90:38088) 2024-04-25T11:45:54.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.684+0000 7f67337fe640 1 -- 172.21.15.90:0/101895110 learned_addr learned my addr 172.21.15.90:0/101895110 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:54.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.684+0000 7f6733fff640 1 --2- 172.21.15.90:0/101895110 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c733fde6b0 0x55c73439bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.684+0000 7f67337fe640 1 -- 172.21.15.90:0/101895110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c733fff390 msgr2=0x55c73439c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.684+0000 7f67337fe640 1 --2- 172.21.15.90:0/101895110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c733fff390 0x55c73439c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.684+0000 7f67337fe640 1 -- 172.21.15.90:0/101895110 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c733fde6b0 msgr2=0x55c73439bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.684+0000 7f67337fe640 1 --2- 172.21.15.90:0/101895110 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c733fde6b0 0x55c73439bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.684+0000 7f67337fe640 1 -- 172.21.15.90:0/101895110 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7343abf80 con 0x55c733fd64d0 2024-04-25T11:45:54.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.684+0000 7f6733fff640 1 --2- 172.21.15.90:0/101895110 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c733fde6b0 0x55c73439bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:54.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.684+0000 7f67337fe640 1 --2- 172.21.15.90:0/101895110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fd64d0 0x55c7343b6e70 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f672802ace0 tx=0x7f672802ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62ea132fdf7c87ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.684+0000 7f6719ffb640 1 -- 172.21.15.90:0/101895110 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6728029b50 con 0x55c733fd64d0 2024-04-25T11:45:54.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.684+0000 7f6719ffb640 1 -- 172.21.15.90:0/101895110 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6728029cf0 con 0x55c733fd64d0 2024-04-25T11:45:54.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.684+0000 7f6719ffb640 1 -- 172.21.15.90:0/101895110 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6728012cb0 con 0x55c733fd64d0 2024-04-25T11:45:54.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.685+0000 7f6741429740 1 -- 172.21.15.90:0/101895110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fd64d0 msgr2=0x55c7343b6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.685+0000 7f6741429740 1 --2- 172.21.15.90:0/101895110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fd64d0 0x55c7343b6e70 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f672802ace0 tx=0x7f672802ad10 comp rx=0 tx=0).stop 2024-04-25T11:45:54.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.685+0000 7f6741429740 1 -- 172.21.15.90:0/101895110 shutdown_connections 2024-04-25T11:45:54.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.685+0000 7f6741429740 1 --2- 172.21.15.90:0/101895110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c733fff390 0x55c73439c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.685+0000 7f6741429740 1 --2- 172.21.15.90:0/101895110 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c733fde6b0 0x55c73439bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.685+0000 7f6741429740 1 --2- 172.21.15.90:0/101895110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fd64d0 0x55c7343b6e70 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.685+0000 7f6741429740 1 -- 172.21.15.90:0/101895110 >> 172.21.15.90:0/101895110 conn(0x55c733ff34e0 msgr2=0x55c73439b610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.685+0000 7f6741429740 1 -- 172.21.15.90:0/101895110 shutdown_connections 2024-04-25T11:45:54.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.685+0000 7f6741429740 1 -- 172.21.15.90:0/101895110 wait complete. 2024-04-25T11:45:54.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.685+0000 7f6741429740 1 -- start start 2024-04-25T11:45:54.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.686+0000 7f6741429740 1 Processor -- start 2024-04-25T11:45:54.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.686+0000 7f6741429740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fd64d0 0x55c7343ae1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.686+0000 7f6741429740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c733fde6b0 0x55c7343ae6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.686+0000 7f6741429740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c733fff390 0x55c734423690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.686+0000 7f6741429740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f67341b1070 con 0x55c733fd64d0 2024-04-25T11:45:54.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.686+0000 7f6741429740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f67341b09a0 con 0x55c733fff390 2024-04-25T11:45:54.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.686+0000 7f6741429740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f67341adc00 con 0x55c733fde6b0 2024-04-25T11:45:54.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.686+0000 7f6741429740 10 client.0 _ll_register_callbacks cb 0x7f672800dd20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:54.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.686+0000 7f67337fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fd64d0 0x55c7343ae1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.686+0000 7f67337fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fd64d0 0x55c7343ae1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38104/0 (socket says 172.21.15.90:38104) 2024-04-25T11:45:54.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.686+0000 7f67337fe640 1 -- 172.21.15.90:0/718408473 learned_addr learned my addr 172.21.15.90:0/718408473 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:54.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.686+0000 7f67337fe640 1 --2- 172.21.15.90:0/718408473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c733fff390 0x55c734423690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.686+0000 7f6733fff640 1 --2- 172.21.15.90:0/718408473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c733fde6b0 0x55c7343ae6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.686+0000 7f67337fe640 1 -- 172.21.15.90:0/718408473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c733fde6b0 msgr2=0x55c7343ae6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.686+0000 7f67337fe640 1 --2- 172.21.15.90:0/718408473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c733fde6b0 0x55c7343ae6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.686+0000 7f67337fe640 1 -- 172.21.15.90:0/718408473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c733fff390 msgr2=0x55c734423690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.686+0000 7f67337fe640 1 --2- 172.21.15.90:0/718408473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c733fff390 0x55c734423690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.686+0000 7f67337fe640 1 -- 172.21.15.90:0/718408473 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7343aef40 con 0x55c733fd64d0 2024-04-25T11:45:54.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.687+0000 7f67337fe640 1 --2- 172.21.15.90:0/718408473 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fd64d0 0x55c7343ae1b0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f6728012880 tx=0x7f6728004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99cc10fc61b087ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.687+0000 7f67137fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:54.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.687+0000 7f67137fe640 1 -- 172.21.15.90:0/718408473 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6728029b50 con 0x55c733fd64d0 2024-04-25T11:45:54.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.687+0000 7f67137fe640 1 -- 172.21.15.90:0/718408473 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7f672802a440 con 0x55c733fd64d0 2024-04-25T11:45:54.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.687+0000 7f6741429740 1 -- 172.21.15.90:0/718408473 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6728029cf0 con 0x55c733fd64d0 2024-04-25T11:45:54.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.687+0000 7f67137fe640 1 -- 172.21.15.90:0/718408473 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f672800d430 con 0x55c733fd64d0 2024-04-25T11:45:54.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.687+0000 7f6741429740 10 client.19098 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:54.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.687+0000 7f6741429740 1 -- 172.21.15.90:0/718408473 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c734424b20 con 0x55c733fd64d0 2024-04-25T11:45:54.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.687+0000 7f67137fe640 1 -- 172.21.15.90:0/718408473 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6728029cf0 con 0x55c733fd64d0 2024-04-25T11:45:54.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.687+0000 7f67137fe640 1 -- 172.21.15.90:0/718408473 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f6728039070 con 0x55c733fd64d0 2024-04-25T11:45:54.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.687+0000 7f6741429740 10 client.19098 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:54.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.687+0000 7f6741429740 10 client.19098 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:45:54.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.687+0000 7f6741429740 20 client.19098 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:54.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.687+0000 7f6741429740 1 --2- 172.21.15.90:0/718408473 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c733f933a0 0x55c7344271e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.687+0000 7f6741429740 4 client.19098 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:45:54.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.687+0000 7f6741429740 1 -- 172.21.15.90:0/718408473 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c7343a10e0 con 0x55c733f933a0 2024-04-25T11:45:54.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.688+0000 7f6733fff640 1 --2- 172.21.15.90:0/718408473 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c733f933a0 0x55c7344271e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.688+0000 7f67137fe640 10 client.19098 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:54.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.688+0000 7f6733fff640 1 --2- 172.21.15.90:0/718408473 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c733f933a0 0x55c7344271e0 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48b0bc4773705454 server_cookie=c4898d0d32f3c6d8 in_seq=0 out_seq=0 2024-04-25T11:45:54.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.688+0000 7f67137fe640 1 -- 172.21.15.90:0/718408473 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c7343a10e0 con 0x55c733f933a0 2024-04-25T11:45:54.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.688+0000 7f67137fe640 10 client.19098 handle_command_reply: tid=0 2024-04-25T11:45:54.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.688+0000 7f6741429740 1 client.19098 shutdown 2024-04-25T11:45:54.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.688+0000 7f6741429740 1 -- 172.21.15.90:0/718408473 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fd64d0 msgr2=0x55c7343ae1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.688+0000 7f6741429740 1 --2- 172.21.15.90:0/718408473 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fd64d0 0x55c7343ae1b0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f6728012880 tx=0x7f6728004730 comp rx=0 tx=0).stop 2024-04-25T11:45:54.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.688+0000 7f6741429740 1 -- 172.21.15.90:0/718408473 shutdown_connections 2024-04-25T11:45:54.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.688+0000 7f6741429740 1 --2- 172.21.15.90:0/718408473 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c733f933a0 0x55c7344271e0 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.688+0000 7f6741429740 1 --2- 172.21.15.90:0/718408473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c733fff390 0x55c734423690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.688+0000 7f6741429740 1 --2- 172.21.15.90:0/718408473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c733fde6b0 0x55c7343ae6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.688+0000 7f67137fe640 0 client.19098 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:54.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.688+0000 7f6741429740 1 --2- 172.21.15.90:0/718408473 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c733fd64d0 0x55c7343ae1b0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.688+0000 7f6741429740 1 -- 172.21.15.90:0/718408473 >> 172.21.15.90:0/718408473 conn(0x55c733ff34e0 msgr2=0x55c7343ac720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.689+0000 7f6741429740 1 -- 172.21.15.90:0/718408473 shutdown_connections 2024-04-25T11:45:54.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.689+0000 7f6741429740 1 -- 172.21.15.90:0/718408473 wait complete. 2024-04-25T11:45:54.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.689+0000 7f6741429740 20 client.19098 trim_cache size 0 max 16384 2024-04-25T11:45:54.681 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:45:54.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.689+0000 7f673a572640 1 -- 172.21.15.90:0/4216303675 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f671c0afce0 msgr2=0x7f671c0b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.689+0000 7f673a572640 1 --2- 172.21.15.90:0/4216303675 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f671c0afce0 0x7f671c0b21f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f672c0a7370 tx=0x7f672c0a2b80 comp rx=0 tx=0).stop 2024-04-25T11:45:54.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.689+0000 7f673a572640 1 -- 172.21.15.90:0/4216303675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6734107ce0 msgr2=0x7f673419ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.689+0000 7f673a572640 1 --2- 172.21.15.90:0/4216303675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6734107ce0 0x7f673419ef10 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f672c00ead0 tx=0x7f672c00ef90 comp rx=0 tx=0).stop 2024-04-25T11:45:54.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.689+0000 7f673a572640 1 -- 172.21.15.90:0/4216303675 shutdown_connections 2024-04-25T11:45:54.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.689+0000 7f673a572640 1 --2- 172.21.15.90:0/4216303675 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f671c0afce0 0x7f671c0b21f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.689+0000 7f673a572640 1 --2- 172.21.15.90:0/4216303675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6734107ce0 0x7f673419ef10 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.689+0000 7f673a572640 1 --2- 172.21.15.90:0/4216303675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6734107260 0x7f673419ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.689+0000 7f673a572640 1 --2- 172.21.15.90:0/4216303675 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f67341033a0 0x7f673419a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.689+0000 7f673a572640 1 -- 172.21.15.90:0/4216303675 >> 172.21.15.90:0/4216303675 conn(0x7f6734077880 msgr2=0x7f67341066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.689+0000 7f673a572640 1 -- 172.21.15.90:0/4216303675 shutdown_connections 2024-04-25T11:45:54.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:54.689+0000 7f673a572640 1 -- 172.21.15.90:0/4216303675 wait complete. 2024-04-25T11:45:55.688 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T11:45:55.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:45:55.688 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T11:45:55.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.827+0000 7fc79d46e640 1 -- 172.21.15.90:0/2253323005 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc798104bf0 msgr2=0x7fc798105000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:55.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.827+0000 7fc79d46e640 1 --2- 172.21.15.90:0/2253323005 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc798104bf0 0x7fc798105000 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fc788009900 tx=0x7fc78802a410 comp rx=0 tx=0).stop 2024-04-25T11:45:55.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.827+0000 7fc79d46e640 1 -- 172.21.15.90:0/2253323005 shutdown_connections 2024-04-25T11:45:55.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.827+0000 7fc79d46e640 1 --2- 172.21.15.90:0/2253323005 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc79810e040 0x7fc798110530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.827+0000 7fc79d46e640 1 --2- 172.21.15.90:0/2253323005 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc79810b650 0x7fc79810da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.827+0000 7fc79d46e640 1 --2- 172.21.15.90:0/2253323005 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc798104bf0 0x7fc798105000 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.827+0000 7fc79d46e640 1 -- 172.21.15.90:0/2253323005 >> 172.21.15.90:0/2253323005 conn(0x7fc798078100 msgr2=0x7fc798078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:55.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.827+0000 7fc79d46e640 1 -- 172.21.15.90:0/2253323005 shutdown_connections 2024-04-25T11:45:55.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.827+0000 7fc79d46e640 1 -- 172.21.15.90:0/2253323005 wait complete. 2024-04-25T11:45:55.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.828+0000 7fc79d46e640 1 Processor -- start 2024-04-25T11:45:55.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.828+0000 7fc79d46e640 1 -- start start 2024-04-25T11:45:55.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.829+0000 7fc79d46e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc798104bf0 0x7fc79819a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.829+0000 7fc79d46e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc79810b650 0x7fc79819ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.829+0000 7fc79d46e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc79810e040 0x7fc79819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.829+0000 7fc79d46e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc798113ca0 con 0x7fc798104bf0 2024-04-25T11:45:55.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.829+0000 7fc79d46e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc798113b20 con 0x7fc79810e040 2024-04-25T11:45:55.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.829+0000 7fc79d46e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc798113e20 con 0x7fc79810b650 2024-04-25T11:45:55.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.829+0000 7fc796ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc798104bf0 0x7fc79819a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.829+0000 7fc796ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc798104bf0 0x7fc79819a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38122/0 (socket says 172.21.15.90:38122) 2024-04-25T11:45:55.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.829+0000 7fc796ffd640 1 -- 172.21.15.90:0/1897105037 learned_addr learned my addr 172.21.15.90:0/1897105037 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:55.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.829+0000 7fc7967fc640 1 --2- 172.21.15.90:0/1897105037 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc79810b650 0x7fc79819ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.829+0000 7fc7977fe640 1 --2- 172.21.15.90:0/1897105037 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc79810e040 0x7fc79819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.829+0000 7fc796ffd640 1 -- 172.21.15.90:0/1897105037 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc79810b650 msgr2=0x7fc79819ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:55.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.829+0000 7fc796ffd640 1 --2- 172.21.15.90:0/1897105037 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc79810b650 0x7fc79819ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.829+0000 7fc796ffd640 1 -- 172.21.15.90:0/1897105037 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc79810e040 msgr2=0x7fc79819efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:55.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.829+0000 7fc796ffd640 1 --2- 172.21.15.90:0/1897105037 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc79810e040 0x7fc79819efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.829+0000 7fc796ffd640 1 -- 172.21.15.90:0/1897105037 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc79819f600 con 0x7fc798104bf0 2024-04-25T11:45:55.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.829+0000 7fc796ffd640 1 --2- 172.21.15.90:0/1897105037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc798104bf0 0x7fc79819a7a0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fc788005940 tx=0x7fc788004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:55.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.829+0000 7fc77ffff640 1 -- 172.21.15.90:0/1897105037 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78803f030 con 0x7fc798104bf0 2024-04-25T11:45:55.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.829+0000 7fc79d46e640 1 -- 172.21.15.90:0/1897105037 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc79819f890 con 0x7fc798104bf0 2024-04-25T11:45:55.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.829+0000 7fc77ffff640 1 -- 172.21.15.90:0/1897105037 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc788003aa0 con 0x7fc798104bf0 2024-04-25T11:45:55.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.829+0000 7fc77ffff640 1 -- 172.21.15.90:0/1897105037 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc788035070 con 0x7fc798104bf0 2024-04-25T11:45:55.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.829+0000 7fc79d46e640 1 -- 172.21.15.90:0/1897105037 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc79819fdd0 con 0x7fc798104bf0 2024-04-25T11:45:55.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.832+0000 7fc77ffff640 1 -- 172.21.15.90:0/1897105037 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fc788042030 con 0x7fc798104bf0 2024-04-25T11:45:55.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.832+0000 7fc7a4402740 1 Processor -- start 2024-04-25T11:45:55.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.832+0000 7fc7a4402740 1 -- start start 2024-04-25T11:45:55.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.832+0000 7fc7a4402740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa9095410 0x562aa9435620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.832+0000 7fc7a4402740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562aa906c550 0x562aa9435b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.832+0000 7fc7a4402740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa9074730 0x562aa94360a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.832+0000 7fc7a4402740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562aa939bb00 con 0x562aa9095410 2024-04-25T11:45:55.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.832+0000 7fc7a4402740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x562aa9337920 con 0x562aa906c550 2024-04-25T11:45:55.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.832+0000 7fc7a4402740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562aa9397ac0 con 0x562aa9074730 2024-04-25T11:45:55.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.833+0000 7fc7977fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa9095410 0x562aa9435620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.833+0000 7fc7977fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa9095410 0x562aa9435620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38138/0 (socket says 172.21.15.90:38138) 2024-04-25T11:45:55.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.833+0000 7fc7977fe640 1 -- 172.21.15.90:0/927199736 learned_addr learned my addr 172.21.15.90:0/927199736 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:55.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.833+0000 7fc796ffd640 1 --2- 172.21.15.90:0/927199736 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562aa906c550 0x562aa9435b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.833+0000 7fc77ffff640 1 --2- 172.21.15.90:0/1897105037 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc7800afc60 0x7fc7800b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.833+0000 7fc7967fc640 1 --2- 172.21.15.90:0/927199736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa9074730 0x562aa94360a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.833+0000 7fc7977fe640 1 -- 172.21.15.90:0/927199736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa9074730 msgr2=0x562aa94360a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:55.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.833+0000 7fc7977fe640 1 --2- 172.21.15.90:0/927199736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa9074730 0x562aa94360a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.833+0000 7fc7977fe640 1 -- 172.21.15.90:0/927199736 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562aa906c550 msgr2=0x562aa9435b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:55.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.833+0000 7fc7977fe640 1 --2- 172.21.15.90:0/927199736 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562aa906c550 0x562aa9435b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.833+0000 7fc7977fe640 1 -- 172.21.15.90:0/927199736 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562aa91757e0 con 0x562aa9095410 2024-04-25T11:45:55.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.833+0000 7fc7967fc640 1 --2- 172.21.15.90:0/927199736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa9074730 0x562aa94360a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:55.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.833+0000 7fc77ffff640 1 -- 172.21.15.90:0/1897105037 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7fc7880ff670 con 0x7fc798104bf0 2024-04-25T11:45:55.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.833+0000 7fc7977fe640 1 --2- 172.21.15.90:0/1897105037 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc7800afc60 0x7fc7800b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.833+0000 7fc7977fe640 1 --2- 172.21.15.90:0/927199736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa9095410 0x562aa9435620 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fc790010520 tx=0x7fc7900109e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca8d5855e112eb1e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:55.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.833+0000 7fc77d7fa640 1 -- 172.21.15.90:0/927199736 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc790019070 con 0x562aa9095410 2024-04-25T11:45:55.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.833+0000 7fc77d7fa640 1 -- 172.21.15.90:0/927199736 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc790003e10 con 0x562aa9095410 2024-04-25T11:45:55.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.833+0000 7fc77d7fa640 1 -- 172.21.15.90:0/927199736 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc79001d4f0 con 0x562aa9095410 2024-04-25T11:45:55.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.834+0000 7fc7a4402740 1 -- 172.21.15.90:0/927199736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa9095410 msgr2=0x562aa9435620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:55.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.834+0000 7fc7977fe640 1 --2- 172.21.15.90:0/1897105037 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc7800afc60 0x7fc7800b2170 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc790021000 tx=0x7fc79001e000 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:55.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.834+0000 7fc7a4402740 1 --2- 172.21.15.90:0/927199736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa9095410 0x562aa9435620 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fc790010520 tx=0x7fc7900109e0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.834+0000 7fc7a4402740 1 -- 172.21.15.90:0/927199736 shutdown_connections 2024-04-25T11:45:55.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.834+0000 7fc7a4402740 1 --2- 172.21.15.90:0/927199736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa9074730 0x562aa94360a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.834+0000 7fc7a4402740 1 --2- 172.21.15.90:0/927199736 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562aa906c550 0x562aa9435b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.834+0000 7fc7a4402740 1 --2- 172.21.15.90:0/927199736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa9095410 0x562aa9435620 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.834+0000 7fc7a4402740 1 -- 172.21.15.90:0/927199736 >> 172.21.15.90:0/927199736 conn(0x562aa9089560 msgr2=0x562aa9431dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:55.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.834+0000 7fc7a4402740 1 -- 172.21.15.90:0/927199736 shutdown_connections 2024-04-25T11:45:55.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.834+0000 7fc7a4402740 1 -- 172.21.15.90:0/927199736 wait complete. 2024-04-25T11:45:55.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.835+0000 7fc7a4402740 1 -- start start 2024-04-25T11:45:55.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.835+0000 7fc7a4402740 1 Processor -- start 2024-04-25T11:45:55.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.835+0000 7fc7a4402740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562aa906c550 0x562aa943fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.835+0000 7fc7a4402740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa9074730 0x562aa9440190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.835+0000 7fc7a4402740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa9095410 0x562aa9445df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.835+0000 7fc7a4402740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562aa9442970 con 0x562aa9095410 2024-04-25T11:45:55.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.835+0000 7fc7a4402740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x562aa93355d0 con 0x562aa906c550 2024-04-25T11:45:55.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.835+0000 7fc7a4402740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562aa9442af0 con 0x562aa9074730 2024-04-25T11:45:55.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.835+0000 7fc7a4402740 10 client.0 _ll_register_callbacks cb 0x562aa9323080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:55.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.835+0000 7fc7977fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa9095410 0x562aa9445df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.835+0000 7fc7977fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa9095410 0x562aa9445df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38140/0 (socket says 172.21.15.90:38140) 2024-04-25T11:45:55.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.835+0000 7fc7977fe640 1 -- 172.21.15.90:0/1081954731 learned_addr learned my addr 172.21.15.90:0/1081954731 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:55.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.835+0000 7fc796ffd640 1 --2- 172.21.15.90:0/1081954731 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562aa906c550 0x562aa943fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.835+0000 7fc7967fc640 1 --2- 172.21.15.90:0/1081954731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa9074730 0x562aa9440190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.835+0000 7fc7977fe640 1 -- 172.21.15.90:0/1081954731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa9074730 msgr2=0x562aa9440190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:55.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.835+0000 7fc7977fe640 1 --2- 172.21.15.90:0/1081954731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa9074730 0x562aa9440190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.836+0000 7fc7977fe640 1 -- 172.21.15.90:0/1081954731 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562aa906c550 msgr2=0x562aa943fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:55.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.836+0000 7fc7977fe640 1 --2- 172.21.15.90:0/1081954731 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562aa906c550 0x562aa943fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.836+0000 7fc7977fe640 1 -- 172.21.15.90:0/1081954731 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562aa91d5f70 con 0x562aa9095410 2024-04-25T11:45:55.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.836+0000 7fc7977fe640 1 --2- 172.21.15.90:0/1081954731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa9095410 0x562aa9445df0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fc790013f40 tx=0x7fc79001da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce26493b2e8dc000 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:55.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.836+0000 7fc77aff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:55.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.836+0000 7fc77aff5640 1 -- 172.21.15.90:0/1081954731 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc790019050 con 0x562aa9095410 2024-04-25T11:45:55.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.836+0000 7fc77aff5640 1 -- 172.21.15.90:0/1081954731 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7fc790013520 con 0x562aa9095410 2024-04-25T11:45:55.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.836+0000 7fc7a4402740 1 -- 172.21.15.90:0/1081954731 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc790003e10 con 0x562aa9095410 2024-04-25T11:45:55.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.836+0000 7fc77aff5640 1 -- 172.21.15.90:0/1081954731 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc79003d050 con 0x562aa9095410 2024-04-25T11:45:55.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.836+0000 7fc7a4402740 10 client.19110 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:55.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.836+0000 7fc7a4402740 1 -- 172.21.15.90:0/1081954731 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562aa906d170 con 0x562aa9095410 2024-04-25T11:45:55.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.836+0000 7fc77aff5640 1 -- 172.21.15.90:0/1081954731 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc790003e10 con 0x562aa9095410 2024-04-25T11:45:55.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.836+0000 7fc77aff5640 1 -- 172.21.15.90:0/1081954731 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fc790038030 con 0x562aa9095410 2024-04-25T11:45:55.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.836+0000 7fc7a4402740 10 client.19110 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:55.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.836+0000 7fc7a4402740 10 client.19110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:45:55.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.836+0000 7fc7a4402740 20 client.19110 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:55.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.836+0000 7fc7a4402740 1 --2- 172.21.15.90:0/1081954731 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562aa90541f0 0x562aa9451e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.836+0000 7fc7a4402740 4 client.19110 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:45:55.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.836+0000 7fc7a4402740 1 -- 172.21.15.90:0/1081954731 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562aa93a9710 con 0x562aa90541f0 2024-04-25T11:45:55.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.837+0000 7fc796ffd640 1 --2- 172.21.15.90:0/1081954731 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562aa90541f0 0x562aa9451e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.837+0000 7fc796ffd640 1 --2- 172.21.15.90:0/1081954731 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562aa90541f0 0x562aa9451e50 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f32c2eee66cd6803 server_cookie=6fbfdc1afb192bad in_seq=0 out_seq=0 2024-04-25T11:45:55.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.837+0000 7fc77aff5640 10 client.19110 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:55.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.838+0000 7fc77aff5640 1 -- 172.21.15.90:0/1081954731 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562aa93a9710 con 0x562aa90541f0 2024-04-25T11:45:55.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.838+0000 7fc77aff5640 10 client.19110 handle_command_reply: tid=0 2024-04-25T11:45:55.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.838+0000 7fc7a4402740 1 client.19110 shutdown 2024-04-25T11:45:55.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.838+0000 7fc7a4402740 1 -- 172.21.15.90:0/1081954731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa9095410 msgr2=0x562aa9445df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:55.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.838+0000 7fc7a4402740 1 --2- 172.21.15.90:0/1081954731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa9095410 0x562aa9445df0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fc790013f40 tx=0x7fc79001da80 comp rx=0 tx=0).stop 2024-04-25T11:45:55.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.838+0000 7fc7a4402740 1 -- 172.21.15.90:0/1081954731 shutdown_connections 2024-04-25T11:45:55.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.838+0000 7fc7a4402740 1 --2- 172.21.15.90:0/1081954731 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562aa90541f0 0x562aa9451e50 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.838+0000 7fc7a4402740 1 --2- 172.21.15.90:0/1081954731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa9095410 0x562aa9445df0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.838+0000 7fc77aff5640 0 client.19110 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:55.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.838+0000 7fc7a4402740 1 --2- 172.21.15.90:0/1081954731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa9074730 0x562aa9440190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.838+0000 7fc7a4402740 1 --2- 172.21.15.90:0/1081954731 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562aa906c550 0x562aa943fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.838+0000 7fc7a4402740 1 -- 172.21.15.90:0/1081954731 >> 172.21.15.90:0/1081954731 conn(0x562aa9089560 msgr2=0x562aa9435450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:55.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.838+0000 7fc7a4402740 1 -- 172.21.15.90:0/1081954731 shutdown_connections 2024-04-25T11:45:55.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.838+0000 7fc7a4402740 1 -- 172.21.15.90:0/1081954731 wait complete. 2024-04-25T11:45:55.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.838+0000 7fc7a4402740 20 client.19110 trim_cache size 0 max 16384 2024-04-25T11:45:55.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.856+0000 7fc7a4402740 1 Processor -- start 2024-04-25T11:45:55.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.856+0000 7fc7a4402740 1 -- start start 2024-04-25T11:45:55.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.856+0000 7fc7a4402740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562aa90541f0 0x562aa9430bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.856+0000 7fc7a4402740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa906c550 0x562aa9445d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.856+0000 7fc7a4402740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa9074730 0x562aa94462a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.856+0000 7fc7a4402740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc7981afca0 con 0x562aa906c550 2024-04-25T11:45:55.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.856+0000 7fc7a4402740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc7981b17e0 con 0x562aa90541f0 2024-04-25T11:45:55.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.856+0000 7fc7a4402740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc7981b0370 con 0x562aa9074730 2024-04-25T11:45:55.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.856+0000 7fc7967fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa906c550 0x562aa9445d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.856+0000 7fc7967fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa906c550 0x562aa9445d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38154/0 (socket says 172.21.15.90:38154) 2024-04-25T11:45:55.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.856+0000 7fc7967fc640 1 -- 172.21.15.90:0/3305451917 learned_addr learned my addr 172.21.15.90:0/3305451917 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:55.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.857+0000 7fc796ffd640 1 --2- 172.21.15.90:0/3305451917 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562aa90541f0 0x562aa9430bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.857+0000 7fc7977fe640 1 --2- 172.21.15.90:0/3305451917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa9074730 0x562aa94462a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.857+0000 7fc7967fc640 1 -- 172.21.15.90:0/3305451917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa9074730 msgr2=0x562aa94462a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:55.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.857+0000 7fc7967fc640 1 --2- 172.21.15.90:0/3305451917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa9074730 0x562aa94462a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.857+0000 7fc7967fc640 1 -- 172.21.15.90:0/3305451917 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562aa90541f0 msgr2=0x562aa9430bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:55.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.857+0000 7fc7967fc640 1 --2- 172.21.15.90:0/3305451917 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562aa90541f0 0x562aa9430bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.857+0000 7fc7967fc640 1 -- 172.21.15.90:0/3305451917 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562aa94427c0 con 0x562aa906c550 2024-04-25T11:45:55.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.857+0000 7fc796ffd640 1 --2- 172.21.15.90:0/3305451917 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562aa90541f0 0x562aa9430bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:55.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.857+0000 7fc7977fe640 1 --2- 172.21.15.90:0/3305451917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa9074730 0x562aa94462a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:55.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.857+0000 7fc7967fc640 1 --2- 172.21.15.90:0/3305451917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa906c550 0x562aa9445d60 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00be80 tx=0x7fc78c012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc42191b3df82957 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:55.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.857+0000 7fc77cff9640 1 -- 172.21.15.90:0/3305451917 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78c005020 con 0x562aa906c550 2024-04-25T11:45:55.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.857+0000 7fc77cff9640 1 -- 172.21.15.90:0/3305451917 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc78c0047a0 con 0x562aa906c550 2024-04-25T11:45:55.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.857+0000 7fc77cff9640 1 -- 172.21.15.90:0/3305451917 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78c01b7b0 con 0x562aa906c550 2024-04-25T11:45:55.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.858+0000 7fc7a4402740 1 -- 172.21.15.90:0/3305451917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa906c550 msgr2=0x562aa9445d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:55.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.858+0000 7fc7a4402740 1 --2- 172.21.15.90:0/3305451917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa906c550 0x562aa9445d60 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00be80 tx=0x7fc78c012820 comp rx=0 tx=0).stop 2024-04-25T11:45:55.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.858+0000 7fc7a4402740 1 -- 172.21.15.90:0/3305451917 shutdown_connections 2024-04-25T11:45:55.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.858+0000 7fc7a4402740 1 --2- 172.21.15.90:0/3305451917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa9074730 0x562aa94462a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.858+0000 7fc7a4402740 1 --2- 172.21.15.90:0/3305451917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa906c550 0x562aa9445d60 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.858+0000 7fc7a4402740 1 --2- 172.21.15.90:0/3305451917 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562aa90541f0 0x562aa9430bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.858+0000 7fc7a4402740 1 -- 172.21.15.90:0/3305451917 >> 172.21.15.90:0/3305451917 conn(0x562aa9089560 msgr2=0x562aa94304c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:55.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.858+0000 7fc7a4402740 1 -- 172.21.15.90:0/3305451917 shutdown_connections 2024-04-25T11:45:55.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.858+0000 7fc7a4402740 1 -- 172.21.15.90:0/3305451917 wait complete. 2024-04-25T11:45:55.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.858+0000 7fc7a4402740 1 -- start start 2024-04-25T11:45:55.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.858+0000 7fc7a4402740 1 Processor -- start 2024-04-25T11:45:55.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.858+0000 7fc7a4402740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562aa90541f0 0x562aa944c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.858+0000 7fc7a4402740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa906c550 0x562aa9447b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.858+0000 7fc7a4402740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa9074730 0x562aa94480e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.858+0000 7fc7a4402740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc7981b1110 con 0x562aa906c550 2024-04-25T11:45:55.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.858+0000 7fc7a4402740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc7981b0a40 con 0x562aa90541f0 2024-04-25T11:45:55.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.858+0000 7fc7a4402740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc7981adca0 con 0x562aa9074730 2024-04-25T11:45:55.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.858+0000 7fc7a4402740 10 client.0 _ll_register_callbacks cb 0x7fc7880c3e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:55.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.858+0000 7fc7967fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa906c550 0x562aa9447b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.859+0000 7fc7967fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa906c550 0x562aa9447b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38168/0 (socket says 172.21.15.90:38168) 2024-04-25T11:45:55.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.859+0000 7fc7967fc640 1 -- 172.21.15.90:0/3480098530 learned_addr learned my addr 172.21.15.90:0/3480098530 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:55.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.859+0000 7fc7977fe640 1 --2- 172.21.15.90:0/3480098530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa9074730 0x562aa94480e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.859+0000 7fc796ffd640 1 --2- 172.21.15.90:0/3480098530 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562aa90541f0 0x562aa944c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.859+0000 7fc7967fc640 1 -- 172.21.15.90:0/3480098530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa9074730 msgr2=0x562aa94480e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:55.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.859+0000 7fc7967fc640 1 --2- 172.21.15.90:0/3480098530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa9074730 0x562aa94480e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.859+0000 7fc7967fc640 1 -- 172.21.15.90:0/3480098530 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562aa90541f0 msgr2=0x562aa944c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:55.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.859+0000 7fc7967fc640 1 --2- 172.21.15.90:0/3480098530 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562aa90541f0 0x562aa944c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.859+0000 7fc7967fc640 1 -- 172.21.15.90:0/3480098530 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562aa9448870 con 0x562aa906c550 2024-04-25T11:45:55.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.859+0000 7fc7977fe640 1 --2- 172.21.15.90:0/3480098530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa9074730 0x562aa94480e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:55.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.859+0000 7fc7967fc640 1 --2- 172.21.15.90:0/3480098530 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa906c550 0x562aa9447b60 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fc78c005750 tx=0x7fc78c004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=583be290a6411205 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:55.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.859+0000 7fc77aff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:55.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.859+0000 7fc77aff5640 1 -- 172.21.15.90:0/3480098530 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78c029030 con 0x562aa906c550 2024-04-25T11:45:55.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.859+0000 7fc77aff5640 1 -- 172.21.15.90:0/3480098530 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7fc78c018040 con 0x562aa906c550 2024-04-25T11:45:55.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.859+0000 7fc7a4402740 1 -- 172.21.15.90:0/3480098530 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc78c0047a0 con 0x562aa906c550 2024-04-25T11:45:55.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.859+0000 7fc77aff5640 1 -- 172.21.15.90:0/3480098530 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc78c033050 con 0x562aa906c550 2024-04-25T11:45:55.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.859+0000 7fc7a4402740 10 client.19116 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:55.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.859+0000 7fc7a4402740 1 -- 172.21.15.90:0/3480098530 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562aa94b9fe0 con 0x562aa906c550 2024-04-25T11:45:55.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.859+0000 7fc77aff5640 1 -- 172.21.15.90:0/3480098530 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc78c0047a0 con 0x562aa906c550 2024-04-25T11:45:55.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.860+0000 7fc77aff5640 1 -- 172.21.15.90:0/3480098530 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fc78c01b410 con 0x562aa906c550 2024-04-25T11:45:55.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.860+0000 7fc7a4402740 10 client.19116 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:55.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.860+0000 7fc7a4402740 10 client.19116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:45:55.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.860+0000 7fc7a4402740 20 client.19116 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:55.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.860+0000 7fc7a4402740 1 --2- 172.21.15.90:0/3480098530 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562aa9095410 0x562aa94bc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.860+0000 7fc7a4402740 4 client.19116 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T11:45:55.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.860+0000 7fc7a4402740 1 -- 172.21.15.90:0/3480098530 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562aa93a3580 con 0x562aa9095410 2024-04-25T11:45:55.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.860+0000 7fc796ffd640 1 --2- 172.21.15.90:0/3480098530 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562aa9095410 0x562aa94bc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.860+0000 7fc796ffd640 1 --2- 172.21.15.90:0/3480098530 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562aa9095410 0x562aa94bc730 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5829d1946256f375 server_cookie=5f0c2749319f5fd7 in_seq=0 out_seq=0 2024-04-25T11:45:55.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.860+0000 7fc77aff5640 10 client.19116 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:55.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.862+0000 7fc77aff5640 1 -- 172.21.15.90:0/3480098530 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562aa93a3580 con 0x562aa9095410 2024-04-25T11:45:55.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.862+0000 7fc77aff5640 10 client.19116 handle_command_reply: tid=0 2024-04-25T11:45:55.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.862+0000 7fc7a4402740 1 client.19116 shutdown 2024-04-25T11:45:55.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.862+0000 7fc7a4402740 1 -- 172.21.15.90:0/3480098530 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa906c550 msgr2=0x562aa9447b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:55.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.862+0000 7fc7a4402740 1 --2- 172.21.15.90:0/3480098530 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa906c550 0x562aa9447b60 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fc78c005750 tx=0x7fc78c004e20 comp rx=0 tx=0).stop 2024-04-25T11:45:55.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.862+0000 7fc7a4402740 1 -- 172.21.15.90:0/3480098530 shutdown_connections 2024-04-25T11:45:55.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.862+0000 7fc7a4402740 1 --2- 172.21.15.90:0/3480098530 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x562aa9095410 0x562aa94bc730 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.862+0000 7fc7a4402740 1 --2- 172.21.15.90:0/3480098530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa9074730 0x562aa94480e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.862+0000 7fc7a4402740 1 --2- 172.21.15.90:0/3480098530 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562aa906c550 0x562aa9447b60 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.862+0000 7fc77aff5640 0 client.19116 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:55.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.862+0000 7fc7a4402740 1 --2- 172.21.15.90:0/3480098530 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x562aa90541f0 0x562aa944c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.862+0000 7fc7a4402740 1 -- 172.21.15.90:0/3480098530 >> 172.21.15.90:0/3480098530 conn(0x562aa9089560 msgr2=0x562aa943e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:55.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.862+0000 7fc7a4402740 1 -- 172.21.15.90:0/3480098530 shutdown_connections 2024-04-25T11:45:55.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.862+0000 7fc7a4402740 1 -- 172.21.15.90:0/3480098530 wait complete. 2024-04-25T11:45:55.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.862+0000 7fc7a4402740 20 client.19116 trim_cache size 0 max 16384 2024-04-25T11:45:55.854 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:45:55.855 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-25T11:45:55.855 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "c8d96337-582b-45f4-bc0b-8e43c2938be8", 2024-04-25T11:45:55.855 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-25T11:45:55.855 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:45:55.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.862+0000 7fc79d46e640 1 -- 172.21.15.90:0/1897105037 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc7800afc60 msgr2=0x7fc7800b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:55.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.862+0000 7fc79d46e640 1 --2- 172.21.15.90:0/1897105037 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc7800afc60 0x7fc7800b2170 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc790021000 tx=0x7fc79001e000 comp rx=0 tx=0).stop 2024-04-25T11:45:55.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.862+0000 7fc79d46e640 1 -- 172.21.15.90:0/1897105037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc798104bf0 msgr2=0x7fc79819a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:55.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.862+0000 7fc79d46e640 1 --2- 172.21.15.90:0/1897105037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc798104bf0 0x7fc79819a7a0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fc788005940 tx=0x7fc788004270 comp rx=0 tx=0).stop 2024-04-25T11:45:55.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.862+0000 7fc79d46e640 1 -- 172.21.15.90:0/1897105037 shutdown_connections 2024-04-25T11:45:55.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.862+0000 7fc79d46e640 1 --2- 172.21.15.90:0/1897105037 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc7800afc60 0x7fc7800b2170 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.862+0000 7fc79d46e640 1 --2- 172.21.15.90:0/1897105037 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc79810e040 0x7fc79819efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.862+0000 7fc79d46e640 1 --2- 172.21.15.90:0/1897105037 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc79810b650 0x7fc79819ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.862+0000 7fc79d46e640 1 --2- 172.21.15.90:0/1897105037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc798104bf0 0x7fc79819a7a0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.862+0000 7fc79d46e640 1 -- 172.21.15.90:0/1897105037 >> 172.21.15.90:0/1897105037 conn(0x7fc798078100 msgr2=0x7fc798101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:55.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.863+0000 7fc79d46e640 1 -- 172.21.15.90:0/1897105037 shutdown_connections 2024-04-25T11:45:55.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.863+0000 7fc79d46e640 1 -- 172.21.15.90:0/1897105037 wait complete. 2024-04-25T11:45:55.860 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:45:55.860 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:45:55.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.984+0000 7fcfc6625640 1 -- 172.21.15.90:0/216338764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfc0102b00 msgr2=0x7fcfc0102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:55.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.984+0000 7fcfc6625640 1 --2- 172.21.15.90:0/216338764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfc0102b00 0x7fcfc0102f10 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4009900 tx=0x7fcfb402a410 comp rx=0 tx=0).stop 2024-04-25T11:45:55.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.984+0000 7fcfc6625640 1 -- 172.21.15.90:0/216338764 shutdown_connections 2024-04-25T11:45:55.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.985+0000 7fcfc6625640 1 --2- 172.21.15.90:0/216338764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfc010d390 0x7fcfc010f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.985+0000 7fcfc6625640 1 --2- 172.21.15.90:0/216338764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfc0102b00 0x7fcfc0102f10 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.985+0000 7fcfc6625640 1 --2- 172.21.15.90:0/216338764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcfc0108a30 0x7fcfc01025c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.985+0000 7fcfc6625640 1 -- 172.21.15.90:0/216338764 >> 172.21.15.90:0/216338764 conn(0x7fcfc00fe260 msgr2=0x7fcfc0100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:55.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.985+0000 7fcfc6625640 1 -- 172.21.15.90:0/216338764 shutdown_connections 2024-04-25T11:45:55.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.985+0000 7fcfc6625640 1 -- 172.21.15.90:0/216338764 wait complete. 2024-04-25T11:45:55.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.985+0000 7fcfc6625640 1 Processor -- start 2024-04-25T11:45:55.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.985+0000 7fcfc6625640 1 -- start start 2024-04-25T11:45:55.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.986+0000 7fcfc6625640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfc0102b00 0x7fcfc019a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.986+0000 7fcfc6625640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfc0108a30 0x7fcfc019ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.986+0000 7fcfc6625640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcfc010d390 0x7fcfc019f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.986+0000 7fcfc6625640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc0112c50 con 0x7fcfc0108a30 2024-04-25T11:45:55.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.986+0000 7fcfc6625640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc0112ad0 con 0x7fcfc010d390 2024-04-25T11:45:55.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.986+0000 7fcfc6625640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc0112dd0 con 0x7fcfc0102b00 2024-04-25T11:45:55.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.986+0000 7fcfbf7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfc0108a30 0x7fcfc019ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.986+0000 7fcfbf7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfc0108a30 0x7fcfc019ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38176/0 (socket says 172.21.15.90:38176) 2024-04-25T11:45:55.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.986+0000 7fcfbf7fe640 1 -- 172.21.15.90:0/1449653172 learned_addr learned my addr 172.21.15.90:0/1449653172 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:55.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.986+0000 7fcfbffff640 1 --2- 172.21.15.90:0/1449653172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfc0102b00 0x7fcfc019a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.986+0000 7fcfc4bc2640 1 --2- 172.21.15.90:0/1449653172 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcfc010d390 0x7fcfc019f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.975 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:55 smithi090 ceph-mon[29575]: pgmap v336: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-04-25T11:45:55.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.986+0000 7fcfbf7fe640 1 -- 172.21.15.90:0/1449653172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfc0102b00 msgr2=0x7fcfc019a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:55.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.986+0000 7fcfbf7fe640 1 --2- 172.21.15.90:0/1449653172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfc0102b00 0x7fcfc019a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.986+0000 7fcfbf7fe640 1 -- 172.21.15.90:0/1449653172 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcfc010d390 msgr2=0x7fcfc019f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:55.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.986+0000 7fcfbf7fe640 1 --2- 172.21.15.90:0/1449653172 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcfc010d390 0x7fcfc019f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.986+0000 7fcfbf7fe640 1 -- 172.21.15.90:0/1449653172 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfc019f640 con 0x7fcfc0108a30 2024-04-25T11:45:55.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.986+0000 7fcfc4bc2640 1 --2- 172.21.15.90:0/1449653172 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcfc010d390 0x7fcfc019f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:55.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.986+0000 7fcfbffff640 1 --2- 172.21.15.90:0/1449653172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfc0102b00 0x7fcfc019a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:55.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.986+0000 7fcfbf7fe640 1 --2- 172.21.15.90:0/1449653172 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfc0108a30 0x7fcfc019ad40 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fcfb402a930 tx=0x7fcfb4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:55.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.986+0000 7fcfbd7fa640 1 -- 172.21.15.90:0/1449653172 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb403f030 con 0x7fcfc0108a30 2024-04-25T11:45:55.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.986+0000 7fcfc6625640 1 -- 172.21.15.90:0/1449653172 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfc019f8d0 con 0x7fcfc0108a30 2024-04-25T11:45:55.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.987+0000 7fcfbd7fa640 1 -- 172.21.15.90:0/1449653172 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfb4003cb0 con 0x7fcfc0108a30 2024-04-25T11:45:55.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.987+0000 7fcfbd7fa640 1 -- 172.21.15.90:0/1449653172 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4035070 con 0x7fcfc0108a30 2024-04-25T11:45:55.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.987+0000 7fcfc6625640 1 -- 172.21.15.90:0/1449653172 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfc01a72b0 con 0x7fcfc0108a30 2024-04-25T11:45:55.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.989+0000 7fcfbd7fa640 1 -- 172.21.15.90:0/1449653172 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fcfb4042030 con 0x7fcfc0108a30 2024-04-25T11:45:55.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.989+0000 7fcfcd5a9740 1 Processor -- start 2024-04-25T11:45:55.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.989+0000 7fcfcd5a9740 1 -- start start 2024-04-25T11:45:55.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.989+0000 7fcfcd5a9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f35e390 0x55723f6fea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.989+0000 7fcfcd5a9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55723f3354d0 0x55723f6fefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.989+0000 7fcfcd5a9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55723f33d6b0 0x55723f70b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.989+0000 7fcfcd5a9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55723f664d80 con 0x55723f35e390 2024-04-25T11:45:55.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.989+0000 7fcfbd7fa640 1 --2- 172.21.15.90:0/1449653172 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcfa80afc60 0x7fcfa80b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.990+0000 7fcfcd5a9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55723f61e8b0 con 0x55723f3354d0 2024-04-25T11:45:55.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.990+0000 7fcfcd5a9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55723f660d40 con 0x55723f33d6b0 2024-04-25T11:45:55.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.990+0000 7fcfc4bc2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f35e390 0x55723f6fea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.990+0000 7fcfc4bc2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f35e390 0x55723f6fea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38190/0 (socket says 172.21.15.90:38190) 2024-04-25T11:45:55.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.990+0000 7fcfc4bc2640 1 -- 172.21.15.90:0/2222721020 learned_addr learned my addr 172.21.15.90:0/2222721020 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:55.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.990+0000 7fcfc4bc2640 1 --2- 172.21.15.90:0/1449653172 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcfa80afc60 0x7fcfa80b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.990+0000 7fcfbffff640 1 --2- 172.21.15.90:0/2222721020 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55723f3354d0 0x55723f6fefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.990+0000 7fcfbd7fa640 1 -- 172.21.15.90:0/1449653172 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7fcfb40ff7b0 con 0x7fcfc0108a30 2024-04-25T11:45:55.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.990+0000 7fcfbf7fe640 1 --2- 172.21.15.90:0/2222721020 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55723f33d6b0 0x55723f70b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.990+0000 7fcfc4bc2640 1 -- 172.21.15.90:0/2222721020 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55723f33d6b0 msgr2=0x55723f70b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:55.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.990+0000 7fcfc4bc2640 1 --2- 172.21.15.90:0/2222721020 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55723f33d6b0 0x55723f70b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.990+0000 7fcfc4bc2640 1 -- 172.21.15.90:0/2222721020 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55723f3354d0 msgr2=0x55723f6fefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:55.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.990+0000 7fcfc4bc2640 1 --2- 172.21.15.90:0/2222721020 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55723f3354d0 0x55723f6fefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.990+0000 7fcfc4bc2640 1 -- 172.21.15.90:0/2222721020 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55723f5efa40 con 0x55723f35e390 2024-04-25T11:45:55.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.990+0000 7fcfc4bc2640 1 --2- 172.21.15.90:0/2222721020 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f35e390 0x55723f6fea60 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fcfb800ed70 tx=0x7fcfb800c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a88cb19ab7b6d93 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:55.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.990+0000 7fcfc4bc2640 1 --2- 172.21.15.90:0/1449653172 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcfa80afc60 0x7fcfa80b2170 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fcfb800cc40 tx=0x7fcfb8019000 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:55.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.990+0000 7fcfa67fc640 1 -- 172.21.15.90:0/2222721020 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8018b70 con 0x55723f35e390 2024-04-25T11:45:55.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.990+0000 7fcfa67fc640 1 -- 172.21.15.90:0/2222721020 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfb8018d10 con 0x55723f35e390 2024-04-25T11:45:55.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.990+0000 7fcfa67fc640 1 -- 172.21.15.90:0/2222721020 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb80233a0 con 0x55723f35e390 2024-04-25T11:45:55.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.991+0000 7fcfcd5a9740 1 -- 172.21.15.90:0/2222721020 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f35e390 msgr2=0x55723f6fea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:55.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.991+0000 7fcfcd5a9740 1 --2- 172.21.15.90:0/2222721020 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f35e390 0x55723f6fea60 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fcfb800ed70 tx=0x7fcfb800c740 comp rx=0 tx=0).stop 2024-04-25T11:45:55.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.991+0000 7fcfcd5a9740 1 -- 172.21.15.90:0/2222721020 shutdown_connections 2024-04-25T11:45:55.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.991+0000 7fcfcd5a9740 1 --2- 172.21.15.90:0/2222721020 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55723f33d6b0 0x55723f70b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.991+0000 7fcfcd5a9740 1 --2- 172.21.15.90:0/2222721020 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55723f3354d0 0x55723f6fefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.991+0000 7fcfcd5a9740 1 --2- 172.21.15.90:0/2222721020 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f35e390 0x55723f6fea60 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.991+0000 7fcfcd5a9740 1 -- 172.21.15.90:0/2222721020 >> 172.21.15.90:0/2222721020 conn(0x55723f3524e0 msgr2=0x55723f6fb210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:55.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.991+0000 7fcfcd5a9740 1 -- 172.21.15.90:0/2222721020 shutdown_connections 2024-04-25T11:45:55.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.991+0000 7fcfcd5a9740 1 -- 172.21.15.90:0/2222721020 wait complete. 2024-04-25T11:45:55.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.992+0000 7fcfcd5a9740 1 -- start start 2024-04-25T11:45:55.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.992+0000 7fcfcd5a9740 1 Processor -- start 2024-04-25T11:45:55.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.992+0000 7fcfcd5a9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55723f3354d0 0x55723f70b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.992+0000 7fcfcd5a9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55723f33d6b0 0x55723f70b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.992+0000 7fcfcd5a9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f35e390 0x55723f709570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.992+0000 7fcfcd5a9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55723f70bf80 con 0x55723f35e390 2024-04-25T11:45:55.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.992+0000 7fcfcd5a9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55723f5ebd10 con 0x55723f33d6b0 2024-04-25T11:45:55.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.992+0000 7fcfcd5a9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55723f70c100 con 0x55723f3354d0 2024-04-25T11:45:55.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.992+0000 7fcfcd5a9740 10 client.0 _ll_register_callbacks cb 0x55723f618210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:55.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.992+0000 7fcfc4bc2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f35e390 0x55723f709570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.992+0000 7fcfc4bc2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f35e390 0x55723f709570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38202/0 (socket says 172.21.15.90:38202) 2024-04-25T11:45:55.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.992+0000 7fcfc4bc2640 1 -- 172.21.15.90:0/2528738541 learned_addr learned my addr 172.21.15.90:0/2528738541 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:55.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.992+0000 7fcfbffff640 1 --2- 172.21.15.90:0/2528738541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55723f3354d0 0x55723f70b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.993+0000 7fcfbf7fe640 1 --2- 172.21.15.90:0/2528738541 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55723f33d6b0 0x55723f70b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.993+0000 7fcfc4bc2640 1 -- 172.21.15.90:0/2528738541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55723f3354d0 msgr2=0x55723f70b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:55.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.993+0000 7fcfc4bc2640 1 --2- 172.21.15.90:0/2528738541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55723f3354d0 0x55723f70b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.993+0000 7fcfc4bc2640 1 -- 172.21.15.90:0/2528738541 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55723f33d6b0 msgr2=0x55723f70b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:55.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.993+0000 7fcfc4bc2640 1 --2- 172.21.15.90:0/2528738541 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55723f33d6b0 0x55723f70b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.993+0000 7fcfc4bc2640 1 -- 172.21.15.90:0/2528738541 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55723f4407d0 con 0x55723f35e390 2024-04-25T11:45:55.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.993+0000 7fcfc4bc2640 1 --2- 172.21.15.90:0/2528738541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f35e390 0x55723f709570 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8009500 tx=0x7fcfb801aa50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=165d01060e68b653 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:55.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.993+0000 7fcf9f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:55.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.993+0000 7fcf9f7fe640 1 -- 172.21.15.90:0/2528738541 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb802a070 con 0x55723f35e390 2024-04-25T11:45:55.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.993+0000 7fcf9f7fe640 1 -- 172.21.15.90:0/2528738541 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7fcfb8024800 con 0x55723f35e390 2024-04-25T11:45:55.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.993+0000 7fcfcd5a9740 1 -- 172.21.15.90:0/2528738541 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfb8018d10 con 0x55723f35e390 2024-04-25T11:45:55.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.993+0000 7fcf9f7fe640 1 -- 172.21.15.90:0/2528738541 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcfb8036050 con 0x55723f35e390 2024-04-25T11:45:55.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.993+0000 7fcfcd5a9740 10 client.19128 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:55.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.993+0000 7fcfcd5a9740 1 -- 172.21.15.90:0/2528738541 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55723f3763f0 con 0x55723f35e390 2024-04-25T11:45:55.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.993+0000 7fcf9f7fe640 1 -- 172.21.15.90:0/2528738541 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfb8018d10 con 0x55723f35e390 2024-04-25T11:45:55.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.993+0000 7fcf9f7fe640 1 -- 172.21.15.90:0/2528738541 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fcfb8023430 con 0x55723f35e390 2024-04-25T11:45:55.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.993+0000 7fcfcd5a9740 10 client.19128 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:55.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.993+0000 7fcfcd5a9740 10 client.19128 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:45:55.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.993+0000 7fcfcd5a9740 20 client.19128 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:55.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.994+0000 7fcfcd5a9740 1 --2- 172.21.15.90:0/2528738541 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55723f31cf60 0x55723f71b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.994+0000 7fcfcd5a9740 4 client.19128 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:45:55.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.994+0000 7fcfcd5a9740 1 -- 172.21.15.90:0/2528738541 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55723f672990 con 0x55723f31cf60 2024-04-25T11:45:55.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.994+0000 7fcfbffff640 1 --2- 172.21.15.90:0/2528738541 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55723f31cf60 0x55723f71b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.994+0000 7fcfbffff640 1 --2- 172.21.15.90:0/2528738541 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55723f31cf60 0x55723f71b660 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85accfc69248f78a server_cookie=6a7e0a2f47a6b410 in_seq=0 out_seq=0 2024-04-25T11:45:55.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.994+0000 7fcf9f7fe640 10 client.19128 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:55.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.995+0000 7fcf9f7fe640 1 -- 172.21.15.90:0/2528738541 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55723f672990 con 0x55723f31cf60 2024-04-25T11:45:55.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.995+0000 7fcf9f7fe640 10 client.19128 handle_command_reply: tid=0 2024-04-25T11:45:55.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.995+0000 7fcfcd5a9740 1 client.19128 shutdown 2024-04-25T11:45:55.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.995+0000 7fcfcd5a9740 1 -- 172.21.15.90:0/2528738541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f35e390 msgr2=0x55723f709570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:55.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.995+0000 7fcfcd5a9740 1 --2- 172.21.15.90:0/2528738541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f35e390 0x55723f709570 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8009500 tx=0x7fcfb801aa50 comp rx=0 tx=0).stop 2024-04-25T11:45:55.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.995+0000 7fcfcd5a9740 1 -- 172.21.15.90:0/2528738541 shutdown_connections 2024-04-25T11:45:55.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.995+0000 7fcfcd5a9740 1 --2- 172.21.15.90:0/2528738541 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55723f31cf60 0x55723f71b660 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.995+0000 7fcfcd5a9740 1 --2- 172.21.15.90:0/2528738541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f35e390 0x55723f709570 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.995+0000 7fcfcd5a9740 1 --2- 172.21.15.90:0/2528738541 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55723f33d6b0 0x55723f70b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.995+0000 7fcf9f7fe640 0 client.19128 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:55.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.995+0000 7fcfcd5a9740 1 --2- 172.21.15.90:0/2528738541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55723f3354d0 0x55723f70b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.995+0000 7fcfcd5a9740 1 -- 172.21.15.90:0/2528738541 >> 172.21.15.90:0/2528738541 conn(0x55723f3524e0 msgr2=0x55723f700780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:55.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.995+0000 7fcfcd5a9740 1 -- 172.21.15.90:0/2528738541 shutdown_connections 2024-04-25T11:45:55.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.995+0000 7fcfcd5a9740 1 -- 172.21.15.90:0/2528738541 wait complete. 2024-04-25T11:45:55.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:55.996+0000 7fcfcd5a9740 20 client.19128 trim_cache size 0 max 16384 2024-04-25T11:45:56.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.013+0000 7fcfcd5a9740 1 Processor -- start 2024-04-25T11:45:56.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.013+0000 7fcfcd5a9740 1 -- start start 2024-04-25T11:45:56.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.013+0000 7fcfcd5a9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55723f3354d0 0x55723f700e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:56.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.013+0000 7fcfcd5a9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f33d6b0 0x55723f6fee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:56.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.013+0000 7fcfcd5a9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55723f35e390 0x55723f7113e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:56.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.013+0000 7fcfcd5a9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc01afd40 con 0x55723f33d6b0 2024-04-25T11:45:56.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.013+0000 7fcfcd5a9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc01b1880 con 0x55723f3354d0 2024-04-25T11:45:56.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.013+0000 7fcfcd5a9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc01b0410 con 0x55723f35e390 2024-04-25T11:45:56.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.013+0000 7fcfbf7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f33d6b0 0x55723f6fee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:56.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.013+0000 7fcfbf7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f33d6b0 0x55723f6fee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38212/0 (socket says 172.21.15.90:38212) 2024-04-25T11:45:56.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.013+0000 7fcfbf7fe640 1 -- 172.21.15.90:0/946672494 learned_addr learned my addr 172.21.15.90:0/946672494 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:56.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.013+0000 7fcfbffff640 1 --2- 172.21.15.90:0/946672494 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55723f3354d0 0x55723f700e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:56.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.014+0000 7fcfbf7fe640 1 -- 172.21.15.90:0/946672494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55723f35e390 msgr2=0x55723f7113e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:56.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.014+0000 7fcfc4bc2640 1 --2- 172.21.15.90:0/946672494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55723f35e390 0x55723f7113e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:56.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.014+0000 7fcfbf7fe640 1 --2- 172.21.15.90:0/946672494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55723f35e390 0x55723f7113e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.014+0000 7fcfbf7fe640 1 -- 172.21.15.90:0/946672494 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55723f3354d0 msgr2=0x55723f700e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:56.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.014+0000 7fcfbf7fe640 1 --2- 172.21.15.90:0/946672494 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55723f3354d0 0x55723f700e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.014+0000 7fcfbf7fe640 1 -- 172.21.15.90:0/946672494 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55723f3360f0 con 0x55723f33d6b0 2024-04-25T11:45:56.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.014+0000 7fcfbf7fe640 1 --2- 172.21.15.90:0/946672494 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f33d6b0 0x55723f6fee60 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fcfb40c75c0 tx=0x7fcfb4009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2164d5bac084dcf6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:56.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.014+0000 7fcfa5ffb640 1 -- 172.21.15.90:0/946672494 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4049070 con 0x55723f33d6b0 2024-04-25T11:45:56.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.014+0000 7fcfa5ffb640 1 -- 172.21.15.90:0/946672494 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfb40d0060 con 0x55723f33d6b0 2024-04-25T11:45:56.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.014+0000 7fcfa5ffb640 1 -- 172.21.15.90:0/946672494 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb40c78c0 con 0x55723f33d6b0 2024-04-25T11:45:56.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.015+0000 7fcfcd5a9740 1 -- 172.21.15.90:0/946672494 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f33d6b0 msgr2=0x55723f6fee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:56.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.015+0000 7fcfcd5a9740 1 --2- 172.21.15.90:0/946672494 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f33d6b0 0x55723f6fee60 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fcfb40c75c0 tx=0x7fcfb4009560 comp rx=0 tx=0).stop 2024-04-25T11:45:56.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.015+0000 7fcfcd5a9740 1 -- 172.21.15.90:0/946672494 shutdown_connections 2024-04-25T11:45:56.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.015+0000 7fcfcd5a9740 1 --2- 172.21.15.90:0/946672494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55723f35e390 0x55723f7113e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.015+0000 7fcfcd5a9740 1 --2- 172.21.15.90:0/946672494 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f33d6b0 0x55723f6fee60 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.015+0000 7fcfcd5a9740 1 --2- 172.21.15.90:0/946672494 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55723f3354d0 0x55723f700e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.015+0000 7fcfcd5a9740 1 -- 172.21.15.90:0/946672494 >> 172.21.15.90:0/946672494 conn(0x55723f3524e0 msgr2=0x55723f700740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:56.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.015+0000 7fcfcd5a9740 1 -- 172.21.15.90:0/946672494 shutdown_connections 2024-04-25T11:45:56.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.015+0000 7fcfcd5a9740 1 -- 172.21.15.90:0/946672494 wait complete. 2024-04-25T11:45:56.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.015+0000 7fcfcd5a9740 1 -- start start 2024-04-25T11:45:56.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.015+0000 7fcfcd5a9740 1 Processor -- start 2024-04-25T11:45:56.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.016+0000 7fcfcd5a9740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55723f3354d0 0x55723f77ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:56.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.016+0000 7fcfcd5a9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f33d6b0 0x55723f77b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:56.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.016+0000 7fcfcd5a9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55723f35e390 0x55723f710910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:56.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.016+0000 7fcfcd5a9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc01b11b0 con 0x55723f33d6b0 2024-04-25T11:45:56.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.016+0000 7fcfcd5a9740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc01b0ae0 con 0x55723f3354d0 2024-04-25T11:45:56.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.016+0000 7fcfcd5a9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc01add40 con 0x55723f35e390 2024-04-25T11:45:56.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.016+0000 7fcfcd5a9740 10 client.0 _ll_register_callbacks cb 0x7fcfb000c390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:56.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.016+0000 7fcfbf7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f33d6b0 0x55723f77b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:56.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.016+0000 7fcfbf7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f33d6b0 0x55723f77b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38224/0 (socket says 172.21.15.90:38224) 2024-04-25T11:45:56.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.016+0000 7fcfbf7fe640 1 -- 172.21.15.90:0/662313448 learned_addr learned my addr 172.21.15.90:0/662313448 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:56.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.016+0000 7fcfbffff640 1 --2- 172.21.15.90:0/662313448 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55723f3354d0 0x55723f77ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:56.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.016+0000 7fcfc4bc2640 1 --2- 172.21.15.90:0/662313448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55723f35e390 0x55723f710910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:56.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.016+0000 7fcfbf7fe640 1 -- 172.21.15.90:0/662313448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55723f35e390 msgr2=0x55723f710910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:56.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.016+0000 7fcfbf7fe640 1 --2- 172.21.15.90:0/662313448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55723f35e390 0x55723f710910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.016+0000 7fcfbf7fe640 1 -- 172.21.15.90:0/662313448 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55723f3354d0 msgr2=0x55723f77ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:56.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.016+0000 7fcfbf7fe640 1 --2- 172.21.15.90:0/662313448 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55723f3354d0 0x55723f77ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.016+0000 7fcfbf7fe640 1 -- 172.21.15.90:0/662313448 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55723f6254b0 con 0x55723f33d6b0 2024-04-25T11:45:56.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.016+0000 7fcfbf7fe640 1 --2- 172.21.15.90:0/662313448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f33d6b0 0x55723f77b090 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4006010 tx=0x7fcfb40fd3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=407d7e08dbc610b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:56.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.016+0000 7fcf9f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:56.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.016+0000 7fcf9f7fe640 1 -- 172.21.15.90:0/662313448 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb40e0070 con 0x55723f33d6b0 2024-04-25T11:45:56.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.016+0000 7fcfcd5a9740 1 -- 172.21.15.90:0/662313448 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55723f66c800 con 0x55723f33d6b0 2024-04-25T11:45:56.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.016+0000 7fcf9f7fe640 1 -- 172.21.15.90:0/662313448 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7fcfb40c3b50 con 0x55723f33d6b0 2024-04-25T11:45:56.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.017+0000 7fcf9f7fe640 1 -- 172.21.15.90:0/662313448 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcfb4049030 con 0x55723f33d6b0 2024-04-25T11:45:56.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.017+0000 7fcfcd5a9740 10 client.19134 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:56.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.017+0000 7fcfcd5a9740 1 -- 172.21.15.90:0/662313448 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55723f7830d0 con 0x55723f33d6b0 2024-04-25T11:45:56.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.017+0000 7fcf9f7fe640 1 -- 172.21.15.90:0/662313448 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55723f66c800 con 0x55723f33d6b0 2024-04-25T11:45:56.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.017+0000 7fcf9f7fe640 1 -- 172.21.15.90:0/662313448 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fcfb408e080 con 0x55723f33d6b0 2024-04-25T11:45:56.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.017+0000 7fcfcd5a9740 10 client.19134 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:56.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.017+0000 7fcfcd5a9740 10 client.19134 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:45:56.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.017+0000 7fcfcd5a9740 20 client.19134 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:56.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.017+0000 7fcfcd5a9740 1 --2- 172.21.15.90:0/662313448 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55723f2f23a0 0x55723f7857b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:56.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.017+0000 7fcfcd5a9740 4 client.19134 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:45:56.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.017+0000 7fcfcd5a9740 1 -- 172.21.15.90:0/662313448 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55723f5e74a0 con 0x55723f2f23a0 2024-04-25T11:45:56.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.017+0000 7fcfbffff640 1 --2- 172.21.15.90:0/662313448 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55723f2f23a0 0x55723f7857b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:56.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.018+0000 7fcf9f7fe640 10 client.19134 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:56.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.018+0000 7fcfbffff640 1 --2- 172.21.15.90:0/662313448 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55723f2f23a0 0x55723f7857b0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29ec8ae729750a13 server_cookie=24fa17012a10d284 in_seq=0 out_seq=0 2024-04-25T11:45:56.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.018+0000 7fcf9f7fe640 1 -- 172.21.15.90:0/662313448 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55723f5e74a0 con 0x55723f2f23a0 2024-04-25T11:45:56.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.018+0000 7fcf9f7fe640 10 client.19134 handle_command_reply: tid=0 2024-04-25T11:45:56.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.018+0000 7fcfcd5a9740 1 client.19134 shutdown 2024-04-25T11:45:56.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.018+0000 7fcfcd5a9740 1 -- 172.21.15.90:0/662313448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f33d6b0 msgr2=0x55723f77b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:56.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.018+0000 7fcfcd5a9740 1 --2- 172.21.15.90:0/662313448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f33d6b0 0x55723f77b090 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4006010 tx=0x7fcfb40fd3a0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.018+0000 7fcfcd5a9740 1 -- 172.21.15.90:0/662313448 shutdown_connections 2024-04-25T11:45:56.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.018+0000 7fcfcd5a9740 1 --2- 172.21.15.90:0/662313448 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55723f2f23a0 0x55723f7857b0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.018+0000 7fcfcd5a9740 1 --2- 172.21.15.90:0/662313448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55723f35e390 0x55723f710910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.018+0000 7fcfcd5a9740 1 --2- 172.21.15.90:0/662313448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55723f33d6b0 0x55723f77b090 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.018+0000 7fcf9f7fe640 0 client.19134 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:56.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.018+0000 7fcfcd5a9740 1 --2- 172.21.15.90:0/662313448 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55723f3354d0 0x55723f77ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.018+0000 7fcfcd5a9740 1 -- 172.21.15.90:0/662313448 >> 172.21.15.90:0/662313448 conn(0x55723f3524e0 msgr2=0x55723f720650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:56.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.018+0000 7fcfcd5a9740 1 -- 172.21.15.90:0/662313448 shutdown_connections 2024-04-25T11:45:56.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.018+0000 7fcfcd5a9740 1 -- 172.21.15.90:0/662313448 wait complete. 2024-04-25T11:45:56.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.018+0000 7fcfcd5a9740 20 client.19134 trim_cache size 0 max 16384 2024-04-25T11:45:56.012 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:45:56.012 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T11:45:56.012 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-25T11:45:56.012 INFO:teuthology.orchestra.run.smithi090.stdout: "c8d96337-582b-45f4-bc0b-8e43c2938be8": { 2024-04-25T11:45:56.012 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-25T11:45:56.012 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "c8d96337-582b-45f4-bc0b-8e43c2938be8", 2024-04-25T11:45:56.012 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-25T11:45:56.012 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:45:56.012 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:45:56.013 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:45:56.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.019+0000 7fcfc6625640 1 -- 172.21.15.90:0/1449653172 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcfa80afc60 msgr2=0x7fcfa80b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:56.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.019+0000 7fcfc6625640 1 --2- 172.21.15.90:0/1449653172 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcfa80afc60 0x7fcfa80b2170 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fcfb800cc40 tx=0x7fcfb8019000 comp rx=0 tx=0).stop 2024-04-25T11:45:56.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.019+0000 7fcfc6625640 1 -- 172.21.15.90:0/1449653172 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfc0108a30 msgr2=0x7fcfc019ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:56.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.019+0000 7fcfc6625640 1 --2- 172.21.15.90:0/1449653172 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfc0108a30 0x7fcfc019ad40 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fcfb402a930 tx=0x7fcfb4004270 comp rx=0 tx=0).stop 2024-04-25T11:45:56.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.019+0000 7fcfc6625640 1 -- 172.21.15.90:0/1449653172 shutdown_connections 2024-04-25T11:45:56.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.019+0000 7fcfc6625640 1 --2- 172.21.15.90:0/1449653172 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fcfa80afc60 0x7fcfa80b2170 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.019+0000 7fcfc6625640 1 --2- 172.21.15.90:0/1449653172 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcfc010d390 0x7fcfc019f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.019+0000 7fcfc6625640 1 --2- 172.21.15.90:0/1449653172 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfc0108a30 0x7fcfc019ad40 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.019+0000 7fcfc6625640 1 --2- 172.21.15.90:0/1449653172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfc0102b00 0x7fcfc019a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.019+0000 7fcfc6625640 1 -- 172.21.15.90:0/1449653172 >> 172.21.15.90:0/1449653172 conn(0x7fcfc00fe260 msgr2=0x7fcfc0100ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:56.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.019+0000 7fcfc6625640 1 -- 172.21.15.90:0/1449653172 shutdown_connections 2024-04-25T11:45:56.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:45:56.019+0000 7fcfc6625640 1 -- 172.21.15.90:0/1449653172 wait complete. 2024-04-25T11:45:56.016 INFO:tasks.cephfs.filesystem:scrub status for tag:c8d96337-582b-45f4-bc0b-8e43c2938be8 - {'path': '/', 'tag': 'c8d96337-582b-45f4-bc0b-8e43c2938be8', 'options': 'recursive,force'} 2024-04-25T11:45:56.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:55 smithi138 ceph-mon[32328]: pgmap v336: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-04-25T11:45:56.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:55 smithi156 ceph-mon[33045]: pgmap v336: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-04-25T11:45:57.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:56 smithi090 ceph-mon[29575]: scrub queued for path: / 2024-04-25T11:45:57.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:56 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:45:57.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:56 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:45:57.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:56 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:45:57.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:56 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:45:57.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:56 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:45:57.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:56 smithi138 ceph-mon[32328]: scrub queued for path: / 2024-04-25T11:45:57.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:56 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:45:57.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:56 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:45:57.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:56 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:45:57.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:56 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:45:57.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:56 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:45:57.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:56 smithi156 ceph-mon[33045]: scrub queued for path: / 2024-04-25T11:45:57.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:56 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:45:57.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:56 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:45:57.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:56 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:45:57.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:56 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:45:57.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:56 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:45:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.423+0000 7f31909e3640 1 -- 172.21.15.90:0/921514577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f318c1042b0 msgr2=0x7f318c110680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.423+0000 7f31909e3640 1 --2- 172.21.15.90:0/921514577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f318c1042b0 0x7f318c110680 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f3184009ca0 tx=0x7f318402a8c0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.423+0000 7f31909e3640 1 -- 172.21.15.90:0/921514577 shutdown_connections 2024-04-25T11:45:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.423+0000 7f31909e3640 1 --2- 172.21.15.90:0/921514577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f318c1042b0 0x7f318c110680 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.423+0000 7f31909e3640 1 --2- 172.21.15.90:0/921514577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f318c10a000 0x7f318c103c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.423+0000 7f31909e3640 1 --2- 172.21.15.90:0/921514577 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f318c109b60 0x7f318c103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.424+0000 7f31909e3640 1 -- 172.21.15.90:0/921514577 >> 172.21.15.90:0/921514577 conn(0x7f318c079170 msgr2=0x7f318c0795b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:57.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.424+0000 7f31909e3640 1 -- 172.21.15.90:0/921514577 shutdown_connections 2024-04-25T11:45:57.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.424+0000 7f31909e3640 1 -- 172.21.15.90:0/921514577 wait complete. 2024-04-25T11:45:57.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.424+0000 7f31909e3640 1 Processor -- start 2024-04-25T11:45:57.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.424+0000 7f31909e3640 1 -- start start 2024-04-25T11:45:57.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.425+0000 7f31909e3640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f318c1042b0 0x7f318c19a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:57.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.425+0000 7f31909e3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f318c109b60 0x7f318c19acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:57.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.425+0000 7f31909e3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f318c10a000 0x7f318c19ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:57.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.425+0000 7f31909e3640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f318c113d70 con 0x7f318c10a000 2024-04-25T11:45:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.425+0000 7f31909e3640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f318c113bf0 con 0x7f318c1042b0 2024-04-25T11:45:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.425+0000 7f31909e3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f318c113ef0 con 0x7f318c109b60 2024-04-25T11:45:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.425+0000 7f318ad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f318c10a000 0x7f318c19ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.425+0000 7f318ad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f318c10a000 0x7f318c19ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38238/0 (socket says 172.21.15.90:38238) 2024-04-25T11:45:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.425+0000 7f318ad9d640 1 -- 172.21.15.90:0/179267953 learned_addr learned my addr 172.21.15.90:0/179267953 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.425+0000 7f3183fff640 1 --2- 172.21.15.90:0/179267953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f318c109b60 0x7f318c19acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.425+0000 7f318a59c640 1 --2- 172.21.15.90:0/179267953 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f318c1042b0 0x7f318c19a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.425+0000 7f318ad9d640 1 -- 172.21.15.90:0/179267953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f318c109b60 msgr2=0x7f318c19acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.425+0000 7f318ad9d640 1 --2- 172.21.15.90:0/179267953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f318c109b60 0x7f318c19acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.425+0000 7f318ad9d640 1 -- 172.21.15.90:0/179267953 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f318c1042b0 msgr2=0x7f318c19a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.425+0000 7f318ad9d640 1 --2- 172.21.15.90:0/179267953 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f318c1042b0 0x7f318c19a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.425+0000 7f318ad9d640 1 -- 172.21.15.90:0/179267953 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f318c19f560 con 0x7f318c10a000 2024-04-25T11:45:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.425+0000 7f318a59c640 1 --2- 172.21.15.90:0/179267953 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f318c1042b0 0x7f318c19a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.425+0000 7f318ad9d640 1 --2- 172.21.15.90:0/179267953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f318c10a000 0x7f318c19ef20 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f31840098d0 tx=0x7f31840317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.425+0000 7f31837fe640 1 -- 172.21.15.90:0/179267953 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3184003b70 con 0x7f318c10a000 2024-04-25T11:45:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.426+0000 7f31909e3640 1 -- 172.21.15.90:0/179267953 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f318c19f7f0 con 0x7f318c10a000 2024-04-25T11:45:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.426+0000 7f31837fe640 1 -- 172.21.15.90:0/179267953 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3184003d10 con 0x7f318c10a000 2024-04-25T11:45:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.426+0000 7f31837fe640 1 -- 172.21.15.90:0/179267953 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318402fa20 con 0x7f318c10a000 2024-04-25T11:45:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.426+0000 7f31909e3640 1 -- 172.21.15.90:0/179267953 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f318c1a71d0 con 0x7f318c10a000 2024-04-25T11:45:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.428+0000 7f31837fe640 1 -- 172.21.15.90:0/179267953 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f318402fc40 con 0x7f318c10a000 2024-04-25T11:45:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.428+0000 7f3197969740 1 Processor -- start 2024-04-25T11:45:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.429+0000 7f3197969740 1 -- start start 2024-04-25T11:45:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.429+0000 7f3197969740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd391b0 0x557d0e0d8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.429+0000 7f3197969740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d0dd102f0 0x557d0e0d9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.429+0000 7f3197969740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557d0dd184d0 0x557d0e0d9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.429+0000 7f3197969740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557d0e03f460 con 0x557d0dd391b0 2024-04-25T11:45:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.429+0000 7f3197969740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x557d0dff9590 con 0x557d0dd184d0 2024-04-25T11:45:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.429+0000 7f3197969740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x557d0e03b420 con 0x557d0dd102f0 2024-04-25T11:45:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.429+0000 7f3183fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd391b0 0x557d0e0d8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.429+0000 7f3183fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd391b0 0x557d0e0d8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38246/0 (socket says 172.21.15.90:38246) 2024-04-25T11:45:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.429+0000 7f3183fff640 1 -- 172.21.15.90:0/3444369820 learned_addr learned my addr 172.21.15.90:0/3444369820 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.429+0000 7f31837fe640 1 --2- 172.21.15.90:0/179267953 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f31780afc50 0x7f31780b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.429+0000 7f3183fff640 1 --2- 172.21.15.90:0/3444369820 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557d0dd184d0 0x557d0e0d9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.429+0000 7f318a59c640 1 --2- 172.21.15.90:0/3444369820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d0dd102f0 0x557d0e0d9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.429+0000 7f3183fff640 1 -- 172.21.15.90:0/3444369820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d0dd102f0 msgr2=0x557d0e0d9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.429+0000 7f3183fff640 1 --2- 172.21.15.90:0/3444369820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d0dd102f0 0x557d0e0d9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.429+0000 7f3183fff640 1 -- 172.21.15.90:0/3444369820 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557d0dd184d0 msgr2=0x557d0e0d9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.429+0000 7f3183fff640 1 --2- 172.21.15.90:0/3444369820 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557d0dd184d0 0x557d0e0d9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.429+0000 7f3183fff640 1 -- 172.21.15.90:0/3444369820 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d0e0060d0 con 0x557d0dd391b0 2024-04-25T11:45:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.429+0000 7f318a59c640 1 --2- 172.21.15.90:0/3444369820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d0dd102f0 0x557d0e0d9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.429+0000 7f318ad9d640 1 --2- 172.21.15.90:0/179267953 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f31780afc50 0x7f31780b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.430+0000 7f31837fe640 1 -- 172.21.15.90:0/179267953 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7f3184100550 con 0x7f318c10a000 2024-04-25T11:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.430+0000 7f3183fff640 1 --2- 172.21.15.90:0/3444369820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd391b0 0x557d0e0d8af0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f31740124c0 tx=0x7f317400b740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70529f22bf53ece server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.430+0000 7f3180ff9640 1 -- 172.21.15.90:0/3444369820 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3174005020 con 0x557d0dd391b0 2024-04-25T11:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.430+0000 7f3180ff9640 1 -- 172.21.15.90:0/3444369820 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31740047a0 con 0x557d0dd391b0 2024-04-25T11:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.430+0000 7f3180ff9640 1 -- 172.21.15.90:0/3444369820 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31740106c0 con 0x557d0dd391b0 2024-04-25T11:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.431+0000 7f3197969740 1 -- 172.21.15.90:0/3444369820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd391b0 msgr2=0x557d0e0d8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.431+0000 7f3197969740 1 --2- 172.21.15.90:0/3444369820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd391b0 0x557d0e0d8af0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f31740124c0 tx=0x7f317400b740 comp rx=0 tx=0).stop 2024-04-25T11:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.431+0000 7f318ad9d640 1 --2- 172.21.15.90:0/179267953 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f31780afc50 0x7f31780b2160 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f31840fddd0 tx=0x7f31840fb040 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.431+0000 7f3197969740 1 -- 172.21.15.90:0/3444369820 shutdown_connections 2024-04-25T11:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.431+0000 7f3197969740 1 --2- 172.21.15.90:0/3444369820 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557d0dd184d0 0x557d0e0d9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.431+0000 7f3197969740 1 --2- 172.21.15.90:0/3444369820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d0dd102f0 0x557d0e0d9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.431+0000 7f3197969740 1 --2- 172.21.15.90:0/3444369820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd391b0 0x557d0e0d8af0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.431+0000 7f3197969740 1 -- 172.21.15.90:0/3444369820 >> 172.21.15.90:0/3444369820 conn(0x557d0dd2d300 msgr2=0x557d0e0d52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.431+0000 7f3197969740 1 -- 172.21.15.90:0/3444369820 shutdown_connections 2024-04-25T11:45:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.431+0000 7f3197969740 1 -- 172.21.15.90:0/3444369820 wait complete. 2024-04-25T11:45:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.431+0000 7f3197969740 1 -- start start 2024-04-25T11:45:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.431+0000 7f3197969740 1 Processor -- start 2024-04-25T11:45:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.432+0000 7f3197969740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd102f0 0x557d0e0e4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.432+0000 7f3197969740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d0dd184d0 0x557d0e0e5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.432+0000 7f3197969740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557d0dd391b0 0x557d0e0e3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.432+0000 7f3197969740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557d0e0e5c20 con 0x557d0dd102f0 2024-04-25T11:45:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.432+0000 7f3197969740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x557d0dfc6b60 con 0x557d0dd391b0 2024-04-25T11:45:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.432+0000 7f3197969740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x557d0e0e5da0 con 0x557d0dd184d0 2024-04-25T11:45:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.432+0000 7f3197969740 10 client.0 _ll_register_callbacks cb 0x557d0dfeb2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.432+0000 7f3183fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd102f0 0x557d0e0e4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:57.426 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.432+0000 7f3183fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd102f0 0x557d0e0e4be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38248/0 (socket says 172.21.15.90:38248) 2024-04-25T11:45:57.426 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.432+0000 7f3183fff640 1 -- 172.21.15.90:0/3291415420 learned_addr learned my addr 172.21.15.90:0/3291415420 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:57.427 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.432+0000 7f318a59c640 1 --2- 172.21.15.90:0/3291415420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d0dd184d0 0x557d0e0e5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:57.427 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.432+0000 7f3183fff640 1 --2- 172.21.15.90:0/3291415420 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557d0dd391b0 0x557d0e0e3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:57.427 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.432+0000 7f3183fff640 1 -- 172.21.15.90:0/3291415420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d0dd184d0 msgr2=0x557d0e0e5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:57.429 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.432+0000 7f3183fff640 1 --2- 172.21.15.90:0/3291415420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d0dd184d0 0x557d0e0e5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.429 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.432+0000 7f3183fff640 1 -- 172.21.15.90:0/3291415420 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557d0dd391b0 msgr2=0x557d0e0e3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:57.429 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.432+0000 7f3183fff640 1 --2- 172.21.15.90:0/3291415420 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557d0dd391b0 0x557d0e0e3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.429 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.432+0000 7f3183fff640 1 -- 172.21.15.90:0/3291415420 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d0dd51210 con 0x557d0dd102f0 2024-04-25T11:45:57.430 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.432+0000 7f318a59c640 1 --2- 172.21.15.90:0/3291415420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d0dd184d0 0x557d0e0e5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:57.430 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.432+0000 7f3183fff640 1 --2- 172.21.15.90:0/3291415420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd102f0 0x557d0e0e4be0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f317400bdb0 tx=0x7f31740058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2749ad120edce8cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:57.430 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.432+0000 7f3171ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:57.430 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.432+0000 7f3171ffb640 1 -- 172.21.15.90:0/3291415420 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3174005200 con 0x557d0dd102f0 2024-04-25T11:45:57.430 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.433+0000 7f3171ffb640 1 -- 172.21.15.90:0/3291415420 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7f31740053a0 con 0x557d0dd102f0 2024-04-25T11:45:57.430 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.433+0000 7f3197969740 1 -- 172.21.15.90:0/3291415420 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31740047a0 con 0x557d0dd102f0 2024-04-25T11:45:57.430 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.433+0000 7f3171ffb640 1 -- 172.21.15.90:0/3291415420 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f31740055a0 con 0x557d0dd102f0 2024-04-25T11:45:57.430 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.433+0000 7f3197969740 10 client.19146 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:57.430 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.433+0000 7f3197969740 1 -- 172.21.15.90:0/3291415420 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557d0de1d620 con 0x557d0dd102f0 2024-04-25T11:45:57.430 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.433+0000 7f3171ffb640 1 -- 172.21.15.90:0/3291415420 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31740047a0 con 0x557d0dd102f0 2024-04-25T11:45:57.430 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.433+0000 7f3171ffb640 1 -- 172.21.15.90:0/3291415420 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f31740314f0 con 0x557d0dd102f0 2024-04-25T11:45:57.430 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.433+0000 7f3197969740 10 client.19146 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:57.431 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.433+0000 7f3197969740 10 client.19146 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:45:57.431 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.433+0000 7f3197969740 20 client.19146 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:57.431 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.433+0000 7f3197969740 1 --2- 172.21.15.90:0/3291415420 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x557d0dcf7d80 0x557d0e0f55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:57.431 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.433+0000 7f3197969740 4 client.19146 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:45:57.431 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.433+0000 7f3197969740 1 -- 172.21.15.90:0/3291415420 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557d0e04d070 con 0x557d0dcf7d80 2024-04-25T11:45:57.431 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.433+0000 7f318a59c640 1 --2- 172.21.15.90:0/3291415420 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x557d0dcf7d80 0x557d0e0f55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:57.431 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.434+0000 7f3171ffb640 10 client.19146 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:57.431 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.434+0000 7f318a59c640 1 --2- 172.21.15.90:0/3291415420 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x557d0dcf7d80 0x557d0e0f55e0 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88e4efe9b8f42df5 server_cookie=6dd1796befb1161c in_seq=0 out_seq=0 2024-04-25T11:45:57.431 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.435+0000 7f3171ffb640 1 -- 172.21.15.90:0/3291415420 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557d0e04d070 con 0x557d0dcf7d80 2024-04-25T11:45:57.431 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.435+0000 7f3171ffb640 10 client.19146 handle_command_reply: tid=0 2024-04-25T11:45:57.431 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.435+0000 7f3197969740 1 client.19146 shutdown 2024-04-25T11:45:57.431 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.435+0000 7f3197969740 1 -- 172.21.15.90:0/3291415420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd102f0 msgr2=0x557d0e0e4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:57.431 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.435+0000 7f3197969740 1 --2- 172.21.15.90:0/3291415420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd102f0 0x557d0e0e4be0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f317400bdb0 tx=0x7f31740058d0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.431 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.435+0000 7f3197969740 1 -- 172.21.15.90:0/3291415420 shutdown_connections 2024-04-25T11:45:57.432 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.435+0000 7f3197969740 1 --2- 172.21.15.90:0/3291415420 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x557d0dcf7d80 0x557d0e0f55e0 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.432 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.435+0000 7f3197969740 1 --2- 172.21.15.90:0/3291415420 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557d0dd391b0 0x557d0e0e3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.432 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.435+0000 7f3197969740 1 --2- 172.21.15.90:0/3291415420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d0dd184d0 0x557d0e0e5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.432 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.435+0000 7f3171ffb640 0 client.19146 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:57.432 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.435+0000 7f3197969740 1 --2- 172.21.15.90:0/3291415420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd102f0 0x557d0e0e4be0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.432 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.435+0000 7f3197969740 1 -- 172.21.15.90:0/3291415420 >> 172.21.15.90:0/3291415420 conn(0x557d0dd2d300 msgr2=0x557d0e0d88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:57.432 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.435+0000 7f3197969740 1 -- 172.21.15.90:0/3291415420 shutdown_connections 2024-04-25T11:45:57.432 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.435+0000 7f3197969740 1 -- 172.21.15.90:0/3291415420 wait complete. 2024-04-25T11:45:57.432 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.435+0000 7f3197969740 20 client.19146 trim_cache size 0 max 16384 2024-04-25T11:45:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.450+0000 7f3197969740 1 Processor -- start 2024-04-25T11:45:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.450+0000 7f3197969740 1 -- start start 2024-04-25T11:45:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.450+0000 7f3197969740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd102f0 0x557d0e0d8f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.450+0000 7f3197969740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557d0dd184d0 0x557d0e0fabe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.450+0000 7f3197969740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d0dd391b0 0x557d0e0fb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.450+0000 7f3197969740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f318c1afc70 con 0x557d0dd102f0 2024-04-25T11:45:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.450+0000 7f3197969740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f318c1b17b0 con 0x557d0dd184d0 2024-04-25T11:45:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.450+0000 7f3197969740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f318c1b0340 con 0x557d0dd391b0 2024-04-25T11:45:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.450+0000 7f3183fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd102f0 0x557d0e0d8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.451+0000 7f3183fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd102f0 0x557d0e0d8f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38254/0 (socket says 172.21.15.90:38254) 2024-04-25T11:45:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.451+0000 7f3183fff640 1 -- 172.21.15.90:0/3377392160 learned_addr learned my addr 172.21.15.90:0/3377392160 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.451+0000 7f3183fff640 1 -- 172.21.15.90:0/3377392160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d0dd391b0 msgr2=0x557d0e0fb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.451+0000 7f3183fff640 1 --2- 172.21.15.90:0/3377392160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d0dd391b0 0x557d0e0fb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.451+0000 7f3183fff640 1 -- 172.21.15.90:0/3377392160 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557d0dd184d0 msgr2=0x557d0e0fabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.451+0000 7f318a59c640 1 --2- 172.21.15.90:0/3377392160 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557d0dd184d0 0x557d0e0fabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.451+0000 7f3183fff640 1 --2- 172.21.15.90:0/3377392160 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557d0dd184d0 0x557d0e0fabe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.451+0000 7f3183fff640 1 -- 172.21.15.90:0/3377392160 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d0e0d9ed0 con 0x557d0dd102f0 2024-04-25T11:45:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.451+0000 7f318a59c640 1 --2- 172.21.15.90:0/3377392160 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557d0dd184d0 0x557d0e0fabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.451+0000 7f3183fff640 1 --2- 172.21.15.90:0/3377392160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd102f0 0x557d0e0d8f90 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f31740318c0 tx=0x7f317400b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99fd9d4a4acf0089 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.451+0000 7f3173fff640 1 -- 172.21.15.90:0/3377392160 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3174004bd0 con 0x557d0dd102f0 2024-04-25T11:45:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.451+0000 7f3173fff640 1 -- 172.21.15.90:0/3377392160 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3174024030 con 0x557d0dd102f0 2024-04-25T11:45:57.444 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.451+0000 7f3173fff640 1 -- 172.21.15.90:0/3377392160 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3174031040 con 0x557d0dd102f0 2024-04-25T11:45:57.444 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.452+0000 7f3197969740 1 -- 172.21.15.90:0/3377392160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd102f0 msgr2=0x557d0e0d8f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:57.444 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.452+0000 7f3197969740 1 --2- 172.21.15.90:0/3377392160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd102f0 0x557d0e0d8f90 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f31740318c0 tx=0x7f317400b670 comp rx=0 tx=0).stop 2024-04-25T11:45:57.444 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.452+0000 7f3197969740 1 -- 172.21.15.90:0/3377392160 shutdown_connections 2024-04-25T11:45:57.444 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.452+0000 7f3197969740 1 --2- 172.21.15.90:0/3377392160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d0dd391b0 0x557d0e0fb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.444 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.452+0000 7f3197969740 1 --2- 172.21.15.90:0/3377392160 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557d0dd184d0 0x557d0e0fabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.444 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.452+0000 7f3197969740 1 --2- 172.21.15.90:0/3377392160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd102f0 0x557d0e0d8f90 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.444 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.452+0000 7f3197969740 1 -- 172.21.15.90:0/3377392160 >> 172.21.15.90:0/3377392160 conn(0x557d0dd2d300 msgr2=0x557d0e0d8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:57.444 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.452+0000 7f3197969740 1 -- 172.21.15.90:0/3377392160 shutdown_connections 2024-04-25T11:45:57.444 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.452+0000 7f3197969740 1 -- 172.21.15.90:0/3377392160 wait complete. 2024-04-25T11:45:57.444 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.452+0000 7f3197969740 1 -- start start 2024-04-25T11:45:57.444 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.452+0000 7f3197969740 1 Processor -- start 2024-04-25T11:45:57.444 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.453+0000 7f3197969740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d0dd102f0 0x557d0e0ea450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:57.444 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.453+0000 7f3197969740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557d0dd184d0 0x557d0e0ea990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:57.444 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.453+0000 7f3197969740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd391b0 0x557d0e155770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:57.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.453+0000 7f3197969740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f318c1b10e0 con 0x557d0dd391b0 2024-04-25T11:45:57.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.453+0000 7f3197969740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f318c1b0a10 con 0x557d0dd184d0 2024-04-25T11:45:57.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.453+0000 7f3197969740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f318c1adc70 con 0x557d0dd102f0 2024-04-25T11:45:57.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.453+0000 7f3197969740 10 client.0 _ll_register_callbacks cb 0x7f317400bf30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:45:57.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.453+0000 7f3183fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd391b0 0x557d0e155770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:57.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.453+0000 7f3183fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd391b0 0x557d0e155770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38262/0 (socket says 172.21.15.90:38262) 2024-04-25T11:45:57.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.453+0000 7f3183fff640 1 -- 172.21.15.90:0/2032683389 learned_addr learned my addr 172.21.15.90:0/2032683389 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:45:57.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.453+0000 7f3183fff640 1 --2- 172.21.15.90:0/2032683389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d0dd102f0 0x557d0e0ea450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:57.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.453+0000 7f318a59c640 1 --2- 172.21.15.90:0/2032683389 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557d0dd184d0 0x557d0e0ea990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:57.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.453+0000 7f3183fff640 1 -- 172.21.15.90:0/2032683389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d0dd102f0 msgr2=0x557d0e0ea450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:57.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.453+0000 7f3183fff640 1 --2- 172.21.15.90:0/2032683389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d0dd102f0 0x557d0e0ea450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.453+0000 7f3183fff640 1 -- 172.21.15.90:0/2032683389 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557d0dd184d0 msgr2=0x557d0e0ea990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:57.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.453+0000 7f3183fff640 1 --2- 172.21.15.90:0/2032683389 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557d0dd184d0 0x557d0e0ea990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.453+0000 7f3183fff640 1 -- 172.21.15.90:0/2032683389 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d0e155cb0 con 0x557d0dd391b0 2024-04-25T11:45:57.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.453+0000 7f3183fff640 1 --2- 172.21.15.90:0/2032683389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd391b0 0x557d0e155770 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f3174004f50 tx=0x7f3174020040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae6736f13af77eb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:57.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.453+0000 7f3171ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:45:57.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.453+0000 7f3171ffb640 1 -- 172.21.15.90:0/2032683389 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3174005200 con 0x557d0dd391b0 2024-04-25T11:45:57.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.453+0000 7f3171ffb640 1 -- 172.21.15.90:0/2032683389 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7f31740053a0 con 0x557d0dd391b0 2024-04-25T11:45:57.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.453+0000 7f3197969740 1 -- 172.21.15.90:0/2032683389 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3174024030 con 0x557d0dd391b0 2024-04-25T11:45:57.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.454+0000 7f3171ffb640 1 -- 172.21.15.90:0/2032683389 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3174031040 con 0x557d0dd391b0 2024-04-25T11:45:57.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.454+0000 7f3197969740 10 client.19152 fetch_fsmap learned FSMap version 28 2024-04-25T11:45:57.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.454+0000 7f3197969740 1 -- 172.21.15.90:0/2032683389 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557d0e15cf20 con 0x557d0dd391b0 2024-04-25T11:45:57.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.454+0000 7f3171ffb640 1 -- 172.21.15.90:0/2032683389 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3174024030 con 0x557d0dd391b0 2024-04-25T11:45:57.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.454+0000 7f3171ffb640 1 -- 172.21.15.90:0/2032683389 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f3174010760 con 0x557d0dd391b0 2024-04-25T11:45:57.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.454+0000 7f3197969740 10 client.19152 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:45:57.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.454+0000 7f3197969740 10 client.19152 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:45:57.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.454+0000 7f3197969740 20 client.19152 populate_metadata read hostname 'smithi090' 2024-04-25T11:45:57.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.454+0000 7f3197969740 1 --2- 172.21.15.90:0/2032683389 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x557d0dccd1c0 0x557d0e15f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:57.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.454+0000 7f3197969740 4 client.19152 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:45:57.447 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.454+0000 7f3197969740 1 -- 172.21.15.90:0/2032683389 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557d0e046ee0 con 0x557d0dccd1c0 2024-04-25T11:45:57.447 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.454+0000 7f318a59c640 1 --2- 172.21.15.90:0/2032683389 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x557d0dccd1c0 0x557d0e15f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:57.447 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.455+0000 7f3171ffb640 10 client.19152 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:57.447 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.455+0000 7f318a59c640 1 --2- 172.21.15.90:0/2032683389 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x557d0dccd1c0 0x557d0e15f590 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50b2ac5841d87475 server_cookie=651e5d8244a23d8b in_seq=0 out_seq=0 2024-04-25T11:45:57.449 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.462+0000 7f3171ffb640 1 -- 172.21.15.90:0/2032683389 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557d0e046ee0 con 0x557d0dccd1c0 2024-04-25T11:45:57.449 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.462+0000 7f3171ffb640 10 client.19152 handle_command_reply: tid=0 2024-04-25T11:45:57.449 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.462+0000 7f3197969740 1 client.19152 shutdown 2024-04-25T11:45:57.449 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.462+0000 7f3197969740 1 -- 172.21.15.90:0/2032683389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd391b0 msgr2=0x557d0e155770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:57.449 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.462+0000 7f3197969740 1 --2- 172.21.15.90:0/2032683389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd391b0 0x557d0e155770 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f3174004f50 tx=0x7f3174020040 comp rx=0 tx=0).stop 2024-04-25T11:45:57.449 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.462+0000 7f3197969740 1 -- 172.21.15.90:0/2032683389 shutdown_connections 2024-04-25T11:45:57.449 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.462+0000 7f3197969740 1 --2- 172.21.15.90:0/2032683389 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x557d0dccd1c0 0x557d0e15f590 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.449 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.462+0000 7f3197969740 1 --2- 172.21.15.90:0/2032683389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d0dd391b0 0x557d0e155770 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.449 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.462+0000 7f3197969740 1 --2- 172.21.15.90:0/2032683389 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x557d0dd184d0 0x557d0e0ea990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.449 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.462+0000 7f3197969740 1 --2- 172.21.15.90:0/2032683389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d0dd102f0 0x557d0e0ea450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.449 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.462+0000 7f3171ffb640 0 client.19152 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:45:57.449 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.462+0000 7f3197969740 1 -- 172.21.15.90:0/2032683389 >> 172.21.15.90:0/2032683389 conn(0x557d0dd2d300 msgr2=0x557d0e0e2890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:57.449 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.462+0000 7f3197969740 1 -- 172.21.15.90:0/2032683389 shutdown_connections 2024-04-25T11:45:57.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.462+0000 7f3197969740 1 -- 172.21.15.90:0/2032683389 wait complete. 2024-04-25T11:45:57.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.462+0000 7f3197969740 20 client.19152 trim_cache size 0 max 16384 2024-04-25T11:45:57.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:45:57.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:45:57.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:45:57.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:45:57.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.462+0000 7f31909e3640 1 -- 172.21.15.90:0/179267953 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f31780afc50 msgr2=0x7f31780b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:57.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.462+0000 7f31909e3640 1 --2- 172.21.15.90:0/179267953 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f31780afc50 0x7f31780b2160 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f31840fddd0 tx=0x7f31840fb040 comp rx=0 tx=0).stop 2024-04-25T11:45:57.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.462+0000 7f31909e3640 1 -- 172.21.15.90:0/179267953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f318c10a000 msgr2=0x7f318c19ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:57.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.462+0000 7f31909e3640 1 --2- 172.21.15.90:0/179267953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f318c10a000 0x7f318c19ef20 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f31840098d0 tx=0x7f31840317b0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.462+0000 7f31909e3640 1 -- 172.21.15.90:0/179267953 shutdown_connections 2024-04-25T11:45:57.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.462+0000 7f31909e3640 1 --2- 172.21.15.90:0/179267953 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f31780afc50 0x7f31780b2160 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.462+0000 7f31909e3640 1 --2- 172.21.15.90:0/179267953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f318c10a000 0x7f318c19ef20 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.462+0000 7f31909e3640 1 --2- 172.21.15.90:0/179267953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f318c109b60 0x7f318c19acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.462+0000 7f31909e3640 1 --2- 172.21.15.90:0/179267953 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f318c1042b0 0x7f318c19a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.462+0000 7f31909e3640 1 -- 172.21.15.90:0/179267953 >> 172.21.15.90:0/179267953 conn(0x7f318c079170 msgr2=0x7f318c101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:57.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.463+0000 7f31909e3640 1 -- 172.21.15.90:0/179267953 shutdown_connections 2024-04-25T11:45:57.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:45:57.463+0000 7f31909e3640 1 -- 172.21.15.90:0/179267953 wait complete. 2024-04-25T11:45:58.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:57 smithi090 ceph-mon[29575]: pgmap v337: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:45:58.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:57 smithi138 ceph-mon[32328]: pgmap v337: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:45:58.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:57 smithi156 ceph-mon[33045]: pgmap v337: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:46:00.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:45:59 smithi090 ceph-mon[29575]: pgmap v338: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 7.1 KiB/s rd, 8 op/s 2024-04-25T11:46:00.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:45:59 smithi138 ceph-mon[32328]: pgmap v338: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 7.1 KiB/s rd, 8 op/s 2024-04-25T11:46:00.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:45:59 smithi156 ceph-mon[33045]: pgmap v338: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 7.1 KiB/s rd, 8 op/s 2024-04-25T11:46:02.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:01 smithi090 ceph-mon[29575]: pgmap v339: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 14 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-04-25T11:46:02.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:01 smithi138 ceph-mon[32328]: pgmap v339: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 14 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-04-25T11:46:02.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:01 smithi156 ceph-mon[33045]: pgmap v339: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 14 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-04-25T11:46:04.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:03 smithi090 ceph-mon[29575]: pgmap v340: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:46:04.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:03 smithi138 ceph-mon[32328]: pgmap v340: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:46:04.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:03 smithi156 ceph-mon[33045]: pgmap v340: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:46:06.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:05 smithi090 ceph-mon[29575]: pgmap v341: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 10 KiB/s rd, 25 KiB/s wr, 14 op/s 2024-04-25T11:46:06.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:05 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:46:06.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:05 smithi138 ceph-mon[32328]: pgmap v341: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 10 KiB/s rd, 25 KiB/s wr, 14 op/s 2024-04-25T11:46:06.300 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:05 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:46:06.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:05 smithi156 ceph-mon[33045]: pgmap v341: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 10 KiB/s rd, 25 KiB/s wr, 14 op/s 2024-04-25T11:46:06.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:05 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:46:07.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:06 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:46:07.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:06 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:46:07.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:06 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:46:08.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:07 smithi090 ceph-mon[29575]: pgmap v342: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 16 KiB/s rd, 47 KiB/s wr, 21 op/s 2024-04-25T11:46:08.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:07 smithi138 ceph-mon[32328]: pgmap v342: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 16 KiB/s rd, 47 KiB/s wr, 21 op/s 2024-04-25T11:46:08.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:07 smithi156 ceph-mon[33045]: pgmap v342: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 16 KiB/s rd, 47 KiB/s wr, 21 op/s 2024-04-25T11:46:10.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:09 smithi090 ceph-mon[29575]: pgmap v343: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 14 KiB/s rd, 47 KiB/s wr, 18 op/s 2024-04-25T11:46:10.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:09 smithi138 ceph-mon[32328]: pgmap v343: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 14 KiB/s rd, 47 KiB/s wr, 18 op/s 2024-04-25T11:46:10.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:09 smithi156 ceph-mon[33045]: pgmap v343: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 14 KiB/s rd, 47 KiB/s wr, 18 op/s 2024-04-25T11:46:10.558 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.571+0000 7f4c09014640 1 -- 172.21.15.90:0/7236130 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0410e0d0 msgr2=0x7f4c041105c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:10.558 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.571+0000 7f4c09014640 1 --2- 172.21.15.90:0/7236130 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0410e0d0 0x7f4c041105c0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc009ca0 tx=0x7f4bfc02a950 comp rx=0 tx=0).stop 2024-04-25T11:46:10.558 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.571+0000 7f4c09014640 1 -- 172.21.15.90:0/7236130 shutdown_connections 2024-04-25T11:46:10.558 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.571+0000 7f4c09014640 1 --2- 172.21.15.90:0/7236130 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0410e0d0 0x7f4c041105c0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.559 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.571+0000 7f4c09014640 1 --2- 172.21.15.90:0/7236130 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4c0410b770 0x7f4c0410db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.559 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.571+0000 7f4c09014640 1 --2- 172.21.15.90:0/7236130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c04104d10 0x7f4c04105120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.559 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.571+0000 7f4c09014640 1 -- 172.21.15.90:0/7236130 >> 172.21.15.90:0/7236130 conn(0x7f4c040781b0 msgr2=0x7f4c040785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:10.559 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.571+0000 7f4c09014640 1 -- 172.21.15.90:0/7236130 shutdown_connections 2024-04-25T11:46:10.559 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.571+0000 7f4c09014640 1 -- 172.21.15.90:0/7236130 wait complete. 2024-04-25T11:46:10.559 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.572+0000 7f4c09014640 1 Processor -- start 2024-04-25T11:46:10.559 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.572+0000 7f4c09014640 1 -- start start 2024-04-25T11:46:10.561 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.572+0000 7f4c09014640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4c04104d10 0x7f4c0419a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:10.561 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.572+0000 7f4c09014640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0410b770 0x7f4c0419acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:10.561 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.572+0000 7f4c09014640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c0410e0d0 0x7f4c0419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:10.561 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.572+0000 7f4c09014640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4c04113d30 con 0x7f4c0410b770 2024-04-25T11:46:10.561 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.572+0000 7f4c09014640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4c04113bb0 con 0x7f4c04104d10 2024-04-25T11:46:10.561 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.572+0000 7f4c09014640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4c04113eb0 con 0x7f4c0410e0d0 2024-04-25T11:46:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.572+0000 7f4bfbfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0410b770 0x7f4c0419acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.572+0000 7f4bfbfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0410b770 0x7f4c0419acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53982/0 (socket says 172.21.15.90:53982) 2024-04-25T11:46:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.572+0000 7f4bfbfff640 1 -- 172.21.15.90:0/1158883638 learned_addr learned my addr 172.21.15.90:0/1158883638 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.572+0000 7f4c02d9d640 1 --2- 172.21.15.90:0/1158883638 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4c04104d10 0x7f4c0419a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.573+0000 7f4c0359e640 1 --2- 172.21.15.90:0/1158883638 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c0410e0d0 0x7f4c0419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.573+0000 7f4bfbfff640 1 -- 172.21.15.90:0/1158883638 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c0410e0d0 msgr2=0x7f4c0419efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.573+0000 7f4bfbfff640 1 --2- 172.21.15.90:0/1158883638 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c0410e0d0 0x7f4c0419efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.573+0000 7f4bfbfff640 1 -- 172.21.15.90:0/1158883638 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4c04104d10 msgr2=0x7f4c0419a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.573+0000 7f4bfbfff640 1 --2- 172.21.15.90:0/1158883638 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4c04104d10 0x7f4c0419a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.573+0000 7f4bfbfff640 1 -- 172.21.15.90:0/1158883638 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c0419f5f0 con 0x7f4c0410b770 2024-04-25T11:46:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.573+0000 7f4bfbfff640 1 --2- 172.21.15.90:0/1158883638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0410b770 0x7f4c0419acf0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f4bec00d950 tx=0x7f4bec00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.573+0000 7f4c00d99640 1 -- 172.21.15.90:0/1158883638 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bec017070 con 0x7f4c0410b770 2024-04-25T11:46:10.563 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.573+0000 7f4c00d99640 1 -- 172.21.15.90:0/1158883638 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bec009c30 con 0x7f4c0410b770 2024-04-25T11:46:10.563 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.573+0000 7f4c09014640 1 -- 172.21.15.90:0/1158883638 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c0419f8e0 con 0x7f4c0410b770 2024-04-25T11:46:10.563 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.573+0000 7f4c00d99640 1 -- 172.21.15.90:0/1158883638 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bec00f620 con 0x7f4c0410b770 2024-04-25T11:46:10.563 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.573+0000 7f4c09014640 1 -- 172.21.15.90:0/1158883638 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c041a7260 con 0x7f4c0410b770 2024-04-25T11:46:10.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.576+0000 7f4c0ff9c740 1 Processor -- start 2024-04-25T11:46:10.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.576+0000 7f4c00d99640 1 -- 172.21.15.90:0/1158883638 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f4bec010460 con 0x7f4c0410b770 2024-04-25T11:46:10.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.576+0000 7f4c0ff9c740 1 -- start start 2024-04-25T11:46:10.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.576+0000 7f4c0ff9c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118d0b1b0 0x55c1190aaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:10.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.576+0000 7f4c0ff9c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c118ce22f0 0x55c1190ab030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.576+0000 7f4c0ff9c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c118cea4d0 0x55c1190ab570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.576+0000 7f4c0ff9c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c119011460 con 0x55c118d0b1b0 2024-04-25T11:46:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.576+0000 7f4c0ff9c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c118fcb590 con 0x55c118cea4d0 2024-04-25T11:46:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.576+0000 7f4c0ff9c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c11900d420 con 0x55c118ce22f0 2024-04-25T11:46:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.577+0000 7f4c0359e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118d0b1b0 0x55c1190aaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.577+0000 7f4c0359e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118d0b1b0 0x55c1190aaaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53992/0 (socket says 172.21.15.90:53992) 2024-04-25T11:46:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.577+0000 7f4c0359e640 1 -- 172.21.15.90:0/3694502050 learned_addr learned my addr 172.21.15.90:0/3694502050 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.577+0000 7f4bfbfff640 1 --2- 172.21.15.90:0/3694502050 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c118cea4d0 0x55c1190ab570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.577+0000 7f4c02d9d640 1 --2- 172.21.15.90:0/3694502050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c118ce22f0 0x55c1190ab030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.577+0000 7f4c00d99640 1 --2- 172.21.15.90:0/1158883638 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4bf00afcd0 0x7f4bf00b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.577+0000 7f4c0359e640 1 -- 172.21.15.90:0/3694502050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c118ce22f0 msgr2=0x55c1190ab030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:10.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.577+0000 7f4c0359e640 1 --2- 172.21.15.90:0/3694502050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c118ce22f0 0x55c1190ab030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.577+0000 7f4c0359e640 1 -- 172.21.15.90:0/3694502050 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c118cea4d0 msgr2=0x55c1190ab570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.577+0000 7f4c0359e640 1 --2- 172.21.15.90:0/3694502050 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c118cea4d0 0x55c1190ab570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.577+0000 7f4c0359e640 1 -- 172.21.15.90:0/3694502050 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c118fd80d0 con 0x55c118d0b1b0 2024-04-25T11:46:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.577+0000 7f4bfbfff640 1 --2- 172.21.15.90:0/3694502050 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c118cea4d0 0x55c1190ab570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:46:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.577+0000 7f4c0359e640 1 --2- 172.21.15.90:0/1158883638 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4bf00afcd0 0x7f4bf00b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.577+0000 7f4c00d99640 1 -- 172.21.15.90:0/1158883638 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7f4bec0deb60 con 0x7f4c0410b770 2024-04-25T11:46:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.577+0000 7f4c0359e640 1 --2- 172.21.15.90:0/3694502050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118d0b1b0 0x55c1190aaaf0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc035000 tx=0x7f4bfc02fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f00aae2cda3b285e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.577+0000 7f4bf97fa640 1 -- 172.21.15.90:0/3694502050 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc034040 con 0x55c118d0b1b0 2024-04-25T11:46:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.578+0000 7f4bf97fa640 1 -- 172.21.15.90:0/3694502050 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bfc004040 con 0x55c118d0b1b0 2024-04-25T11:46:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.578+0000 7f4bf97fa640 1 -- 172.21.15.90:0/3694502050 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc030640 con 0x55c118d0b1b0 2024-04-25T11:46:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.578+0000 7f4c0ff9c740 1 -- 172.21.15.90:0/3694502050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118d0b1b0 msgr2=0x55c1190aaaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.578+0000 7f4c0359e640 1 --2- 172.21.15.90:0/1158883638 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4bf00afcd0 0x7f4bf00b21e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc030040 tx=0x7f4bfc04d000 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:10.572 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.578+0000 7f4c0ff9c740 1 --2- 172.21.15.90:0/3694502050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118d0b1b0 0x55c1190aaaf0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc035000 tx=0x7f4bfc02fd40 comp rx=0 tx=0).stop 2024-04-25T11:46:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.579+0000 7f4c0ff9c740 1 -- 172.21.15.90:0/3694502050 shutdown_connections 2024-04-25T11:46:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.579+0000 7f4c0ff9c740 1 --2- 172.21.15.90:0/3694502050 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c118cea4d0 0x55c1190ab570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.579+0000 7f4c0ff9c740 1 --2- 172.21.15.90:0/3694502050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c118ce22f0 0x55c1190ab030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.579+0000 7f4c0ff9c740 1 --2- 172.21.15.90:0/3694502050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118d0b1b0 0x55c1190aaaf0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.579+0000 7f4c0ff9c740 1 -- 172.21.15.90:0/3694502050 >> 172.21.15.90:0/3694502050 conn(0x55c118cff300 msgr2=0x55c1190a72a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.579+0000 7f4c0ff9c740 1 -- 172.21.15.90:0/3694502050 shutdown_connections 2024-04-25T11:46:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.579+0000 7f4c0ff9c740 1 -- 172.21.15.90:0/3694502050 wait complete. 2024-04-25T11:46:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.579+0000 7f4c0ff9c740 1 -- start start 2024-04-25T11:46:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.579+0000 7f4c0ff9c740 1 Processor -- start 2024-04-25T11:46:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.579+0000 7f4c0ff9c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118ce22f0 0x55c1190b51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.579+0000 7f4c0ff9c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c118cea4d0 0x55c1190b5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.579+0000 7f4c0ff9c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c118d0b1b0 0x55c1190bb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.579+0000 7f4c0ff9c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c1190b7bc0 con 0x55c118ce22f0 2024-04-25T11:46:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.579+0000 7f4c0ff9c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c118f98b60 con 0x55c118cea4d0 2024-04-25T11:46:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.579+0000 7f4c0ff9c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c1190b7d40 con 0x55c118d0b1b0 2024-04-25T11:46:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.579+0000 7f4c0ff9c740 10 client.0 _ll_register_callbacks cb 0x55c118fbd2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.579+0000 7f4c02d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118ce22f0 0x55c1190b51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.580+0000 7f4c02d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118ce22f0 0x55c1190b51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53996/0 (socket says 172.21.15.90:53996) 2024-04-25T11:46:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.580+0000 7f4c02d9d640 1 -- 172.21.15.90:0/1135594120 learned_addr learned my addr 172.21.15.90:0/1135594120 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.580+0000 7f4bfbfff640 1 --2- 172.21.15.90:0/1135594120 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c118cea4d0 0x55c1190b5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.580+0000 7f4c0359e640 1 --2- 172.21.15.90:0/1135594120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c118d0b1b0 0x55c1190bb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.580+0000 7f4c02d9d640 1 -- 172.21.15.90:0/1135594120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c118d0b1b0 msgr2=0x55c1190bb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.580+0000 7f4c02d9d640 1 --2- 172.21.15.90:0/1135594120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c118d0b1b0 0x55c1190bb0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.580+0000 7f4c02d9d640 1 -- 172.21.15.90:0/1135594120 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c118cea4d0 msgr2=0x55c1190b5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.580+0000 7f4c02d9d640 1 --2- 172.21.15.90:0/1135594120 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c118cea4d0 0x55c1190b5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.580+0000 7f4c02d9d640 1 -- 172.21.15.90:0/1135594120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c118d23210 con 0x55c118ce22f0 2024-04-25T11:46:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.580+0000 7f4c02d9d640 1 --2- 172.21.15.90:0/1135594120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118ce22f0 0x55c1190b51f0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f4bf400b950 tx=0x7f4bf400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2346ec137d4edf9f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.580+0000 7f4bea7fc640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:46:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.580+0000 7f4bea7fc640 1 -- 172.21.15.90:0/1135594120 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf401b440 con 0x55c118ce22f0 2024-04-25T11:46:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.580+0000 7f4bea7fc640 1 -- 172.21.15.90:0/1135594120 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7f4bf4011440 con 0x55c118ce22f0 2024-04-25T11:46:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.580+0000 7f4c0ff9c740 1 -- 172.21.15.90:0/1135594120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bfc004040 con 0x55c118ce22f0 2024-04-25T11:46:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.580+0000 7f4bea7fc640 1 -- 172.21.15.90:0/1135594120 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4bf4011680 con 0x55c118ce22f0 2024-04-25T11:46:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.580+0000 7f4c0ff9c740 10 client.19164 fetch_fsmap learned FSMap version 28 2024-04-25T11:46:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.580+0000 7f4c0ff9c740 1 -- 172.21.15.90:0/1135594120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c118def620 con 0x55c118ce22f0 2024-04-25T11:46:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.581+0000 7f4bea7fc640 1 -- 172.21.15.90:0/1135594120 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bfc004040 con 0x55c118ce22f0 2024-04-25T11:46:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.581+0000 7f4bea7fc640 1 -- 172.21.15.90:0/1135594120 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f4bf4019020 con 0x55c118ce22f0 2024-04-25T11:46:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.581+0000 7f4c0ff9c740 10 client.19164 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:46:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.581+0000 7f4c0ff9c740 10 client.19164 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:46:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.581+0000 7f4c0ff9c740 20 client.19164 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.581+0000 7f4c0ff9c740 1 --2- 172.21.15.90:0/1135594120 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c118cc9d80 0x55c1190c7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.581+0000 7f4c0ff9c740 4 client.19164 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:46:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.581+0000 7f4c0ff9c740 1 -- 172.21.15.90:0/1135594120 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c11901f070 con 0x55c118cc9d80 2024-04-25T11:46:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.581+0000 7f4bfbfff640 1 --2- 172.21.15.90:0/1135594120 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c118cc9d80 0x55c1190c7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.582+0000 7f4bea7fc640 10 client.19164 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.582+0000 7f4bfbfff640 1 --2- 172.21.15.90:0/1135594120 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c118cc9d80 0x55c1190c7800 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a06d85e16b2d87c9 server_cookie=73b1ab66dc12ffb1 in_seq=0 out_seq=0 2024-04-25T11:46:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.582+0000 7f4bea7fc640 1 -- 172.21.15.90:0/1135594120 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c11901f070 con 0x55c118cc9d80 2024-04-25T11:46:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.582+0000 7f4bea7fc640 10 client.19164 handle_command_reply: tid=0 2024-04-25T11:46:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.582+0000 7f4c0ff9c740 1 client.19164 shutdown 2024-04-25T11:46:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.583+0000 7f4c0ff9c740 1 -- 172.21.15.90:0/1135594120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118ce22f0 msgr2=0x55c1190b51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.583+0000 7f4c0ff9c740 1 --2- 172.21.15.90:0/1135594120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118ce22f0 0x55c1190b51f0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f4bf400b950 tx=0x7f4bf400bd50 comp rx=0 tx=0).stop 2024-04-25T11:46:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.583+0000 7f4c0ff9c740 1 -- 172.21.15.90:0/1135594120 shutdown_connections 2024-04-25T11:46:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.583+0000 7f4c0ff9c740 1 --2- 172.21.15.90:0/1135594120 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c118cc9d80 0x55c1190c7800 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.583+0000 7f4c0ff9c740 1 --2- 172.21.15.90:0/1135594120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c118d0b1b0 0x55c1190bb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.583+0000 7f4bea7fc640 0 client.19164 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:10.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.583+0000 7f4c0ff9c740 1 --2- 172.21.15.90:0/1135594120 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c118cea4d0 0x55c1190b5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.583+0000 7f4c0ff9c740 1 --2- 172.21.15.90:0/1135594120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118ce22f0 0x55c1190b51f0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.583+0000 7f4c0ff9c740 1 -- 172.21.15.90:0/1135594120 >> 172.21.15.90:0/1135594120 conn(0x55c118cff300 msgr2=0x55c1190aa920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:10.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.583+0000 7f4c0ff9c740 1 -- 172.21.15.90:0/1135594120 shutdown_connections 2024-04-25T11:46:10.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.583+0000 7f4c0ff9c740 1 -- 172.21.15.90:0/1135594120 wait complete. 2024-04-25T11:46:10.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.583+0000 7f4c0ff9c740 20 client.19164 trim_cache size 0 max 16384 2024-04-25T11:46:10.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.598+0000 7f4c0ff9c740 1 Processor -- start 2024-04-25T11:46:10.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.598+0000 7f4c0ff9c740 1 -- start start 2024-04-25T11:46:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.598+0000 7f4c0ff9c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118ce22f0 0x55c1190b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.598+0000 7f4c0ff9c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c118cea4d0 0x55c1190aaf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.598+0000 7f4c0ff9c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c118d0b1b0 0x55c1190cd000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.598+0000 7f4c0ff9c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041afcb0 con 0x55c118ce22f0 2024-04-25T11:46:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.598+0000 7f4c0ff9c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041b17f0 con 0x55c118cea4d0 2024-04-25T11:46:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.598+0000 7f4c0ff9c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041b0380 con 0x55c118d0b1b0 2024-04-25T11:46:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.598+0000 7f4c02d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118ce22f0 0x55c1190b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.598+0000 7f4c02d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118ce22f0 0x55c1190b3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54008/0 (socket says 172.21.15.90:54008) 2024-04-25T11:46:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.598+0000 7f4c02d9d640 1 -- 172.21.15.90:0/2892468745 learned_addr learned my addr 172.21.15.90:0/2892468745 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.599+0000 7f4c0359e640 1 --2- 172.21.15.90:0/2892468745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c118d0b1b0 0x55c1190cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.599+0000 7f4bfbfff640 1 --2- 172.21.15.90:0/2892468745 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c118cea4d0 0x55c1190aaf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.599+0000 7f4c02d9d640 1 -- 172.21.15.90:0/2892468745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c118d0b1b0 msgr2=0x55c1190cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.599+0000 7f4c02d9d640 1 --2- 172.21.15.90:0/2892468745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c118d0b1b0 0x55c1190cd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.599+0000 7f4c02d9d640 1 -- 172.21.15.90:0/2892468745 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c118cea4d0 msgr2=0x55c1190aaf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.599+0000 7f4c02d9d640 1 --2- 172.21.15.90:0/2892468745 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c118cea4d0 0x55c1190aaf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.599+0000 7f4c02d9d640 1 -- 172.21.15.90:0/2892468745 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1190abe80 con 0x55c118ce22f0 2024-04-25T11:46:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.599+0000 7f4bfbfff640 1 --2- 172.21.15.90:0/2892468745 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c118cea4d0 0x55c1190aaf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:46:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.599+0000 7f4c02d9d640 1 --2- 172.21.15.90:0/2892468745 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118ce22f0 0x55c1190b3f20 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f4bf401bd50 tx=0x7f4bf4003af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42a64c455d1aa5e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.599+0000 7f4bf8ff9640 1 -- 172.21.15.90:0/2892468745 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf4003de0 con 0x55c118ce22f0 2024-04-25T11:46:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.599+0000 7f4bf8ff9640 1 -- 172.21.15.90:0/2892468745 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bf401b8f0 con 0x55c118ce22f0 2024-04-25T11:46:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.599+0000 7f4bf8ff9640 1 -- 172.21.15.90:0/2892468745 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf4011aa0 con 0x55c118ce22f0 2024-04-25T11:46:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.600+0000 7f4c0ff9c740 1 -- 172.21.15.90:0/2892468745 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118ce22f0 msgr2=0x55c1190b3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.600+0000 7f4c0ff9c740 1 --2- 172.21.15.90:0/2892468745 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118ce22f0 0x55c1190b3f20 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f4bf401bd50 tx=0x7f4bf4003af0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.600+0000 7f4c0ff9c740 1 -- 172.21.15.90:0/2892468745 shutdown_connections 2024-04-25T11:46:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.600+0000 7f4c0ff9c740 1 --2- 172.21.15.90:0/2892468745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c118d0b1b0 0x55c1190cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.600+0000 7f4c0ff9c740 1 --2- 172.21.15.90:0/2892468745 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c118cea4d0 0x55c1190aaf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.600+0000 7f4c0ff9c740 1 --2- 172.21.15.90:0/2892468745 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118ce22f0 0x55c1190b3f20 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.600+0000 7f4c0ff9c740 1 -- 172.21.15.90:0/2892468745 >> 172.21.15.90:0/2892468745 conn(0x55c118cff300 msgr2=0x55c1190aa880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.600+0000 7f4c0ff9c740 1 -- 172.21.15.90:0/2892468745 shutdown_connections 2024-04-25T11:46:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.600+0000 7f4c0ff9c740 1 -- 172.21.15.90:0/2892468745 wait complete. 2024-04-25T11:46:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.600+0000 7f4c0ff9c740 1 -- start start 2024-04-25T11:46:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.600+0000 7f4c0ff9c740 1 Processor -- start 2024-04-25T11:46:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.601+0000 7f4c0ff9c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118ce22f0 0x55c1190be7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.601+0000 7f4c0ff9c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c118cea4d0 0x55c1190bed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.601+0000 7f4c0ff9c740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c118d0b1b0 0x55c11912dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.601+0000 7f4c0ff9c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041b1120 con 0x55c118ce22f0 2024-04-25T11:46:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.601+0000 7f4c0ff9c740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041b0a50 con 0x55c118d0b1b0 2024-04-25T11:46:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.601+0000 7f4c0ff9c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041adcb0 con 0x55c118cea4d0 2024-04-25T11:46:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.601+0000 7f4c0ff9c740 10 client.0 _ll_register_callbacks cb 0x7f4bf4004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.601+0000 7f4c02d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118ce22f0 0x55c1190be7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.601+0000 7f4c02d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118ce22f0 0x55c1190be7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54022/0 (socket says 172.21.15.90:54022) 2024-04-25T11:46:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.601+0000 7f4c02d9d640 1 -- 172.21.15.90:0/963651470 learned_addr learned my addr 172.21.15.90:0/963651470 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.601+0000 7f4bfbfff640 1 --2- 172.21.15.90:0/963651470 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c118cea4d0 0x55c1190bed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.601+0000 7f4c0359e640 1 --2- 172.21.15.90:0/963651470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c118d0b1b0 0x55c11912dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.601+0000 7f4c02d9d640 1 -- 172.21.15.90:0/963651470 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c118cea4d0 msgr2=0x55c1190bed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.601+0000 7f4c02d9d640 1 --2- 172.21.15.90:0/963651470 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c118cea4d0 0x55c1190bed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.601+0000 7f4c02d9d640 1 -- 172.21.15.90:0/963651470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c118d0b1b0 msgr2=0x55c11912dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.601+0000 7f4c02d9d640 1 --2- 172.21.15.90:0/963651470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c118d0b1b0 0x55c11912dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.601+0000 7f4c02d9d640 1 -- 172.21.15.90:0/963651470 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c11912e0f0 con 0x55c118ce22f0 2024-04-25T11:46:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.601+0000 7f4c0359e640 1 --2- 172.21.15.90:0/963651470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c118d0b1b0 0x55c11912dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:46:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.601+0000 7f4c02d9d640 1 --2- 172.21.15.90:0/963651470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118ce22f0 0x55c1190be7e0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f4bf400b950 tx=0x7f4bf401bc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b21ffdc1dc2dd42 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.601+0000 7f4bea7fc640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:46:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.601+0000 7f4bea7fc640 1 -- 172.21.15.90:0/963651470 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf4039060 con 0x55c118ce22f0 2024-04-25T11:46:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.601+0000 7f4bea7fc640 1 -- 172.21.15.90:0/963651470 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8658+0+0 (secure 0 0 0) 0x7f4bf4003de0 con 0x55c118ce22f0 2024-04-25T11:46:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.601+0000 7f4c0ff9c740 1 -- 172.21.15.90:0/963651470 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bf401b8f0 con 0x55c118ce22f0 2024-04-25T11:46:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.602+0000 7f4bea7fc640 1 -- 172.21.15.90:0/963651470 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4bf4040050 con 0x55c118ce22f0 2024-04-25T11:46:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.602+0000 7f4c0ff9c740 10 client.19170 fetch_fsmap learned FSMap version 28 2024-04-25T11:46:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.602+0000 7f4c0ff9c740 1 -- 172.21.15.90:0/963651470 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c11912f240 con 0x55c118ce22f0 2024-04-25T11:46:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.602+0000 7f4bea7fc640 1 -- 172.21.15.90:0/963651470 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bf401b8f0 con 0x55c118ce22f0 2024-04-25T11:46:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.602+0000 7f4bea7fc640 1 -- 172.21.15.90:0/963651470 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f4bf4029070 con 0x55c118ce22f0 2024-04-25T11:46:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.602+0000 7f4c0ff9c740 10 client.19170 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:46:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.602+0000 7f4c0ff9c740 10 client.19170 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:46:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.602+0000 7f4c0ff9c740 20 client.19170 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.602+0000 7f4c0ff9c740 1 --2- 172.21.15.90:0/963651470 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c118c9f1c0 0x55c1191318e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.602+0000 7f4c0ff9c740 4 client.19170 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:46:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.602+0000 7f4c0ff9c740 1 -- 172.21.15.90:0/963651470 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c119018ee0 con 0x55c118c9f1c0 2024-04-25T11:46:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.602+0000 7f4bfbfff640 1 --2- 172.21.15.90:0/963651470 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c118c9f1c0 0x55c1191318e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.603+0000 7f4bea7fc640 10 client.19170 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.603+0000 7f4bfbfff640 1 --2- 172.21.15.90:0/963651470 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c118c9f1c0 0x55c1191318e0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c1429888f158b78 server_cookie=a09af3363741a18a in_seq=0 out_seq=0 2024-04-25T11:46:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.612+0000 7f4bea7fc640 1 -- 172.21.15.90:0/963651470 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c119018ee0 con 0x55c118c9f1c0 2024-04-25T11:46:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.612+0000 7f4bea7fc640 10 client.19170 handle_command_reply: tid=0 2024-04-25T11:46:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.612+0000 7f4c0ff9c740 1 client.19170 shutdown 2024-04-25T11:46:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.612+0000 7f4c0ff9c740 1 -- 172.21.15.90:0/963651470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118ce22f0 msgr2=0x55c1190be7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.612+0000 7f4c0ff9c740 1 --2- 172.21.15.90:0/963651470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118ce22f0 0x55c1190be7e0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f4bf400b950 tx=0x7f4bf401bc40 comp rx=0 tx=0).stop 2024-04-25T11:46:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.612+0000 7f4c0ff9c740 1 -- 172.21.15.90:0/963651470 shutdown_connections 2024-04-25T11:46:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.612+0000 7f4c0ff9c740 1 --2- 172.21.15.90:0/963651470 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c118c9f1c0 0x55c1191318e0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.612+0000 7f4c0ff9c740 1 --2- 172.21.15.90:0/963651470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c118d0b1b0 0x55c11912dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.612+0000 7f4bea7fc640 0 client.19170 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.612+0000 7f4c0ff9c740 1 --2- 172.21.15.90:0/963651470 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c118cea4d0 0x55c1190bed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.612+0000 7f4c0ff9c740 1 --2- 172.21.15.90:0/963651470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c118ce22f0 0x55c1190be7e0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.612+0000 7f4c0ff9c740 1 -- 172.21.15.90:0/963651470 >> 172.21.15.90:0/963651470 conn(0x55c118cff300 msgr2=0x55c1190b4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.613+0000 7f4c0ff9c740 1 -- 172.21.15.90:0/963651470 shutdown_connections 2024-04-25T11:46:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.613+0000 7f4c0ff9c740 1 -- 172.21.15.90:0/963651470 wait complete. 2024-04-25T11:46:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.613+0000 7f4c0ff9c740 20 client.19170 trim_cache size 0 max 16384 2024-04-25T11:46:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:46:10.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:46:10.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:46:10.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:46:10.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.613+0000 7f4c09014640 1 -- 172.21.15.90:0/1158883638 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4bf00afcd0 msgr2=0x7f4bf00b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:10.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.613+0000 7f4c09014640 1 --2- 172.21.15.90:0/1158883638 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4bf00afcd0 0x7f4bf00b21e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc030040 tx=0x7f4bfc04d000 comp rx=0 tx=0).stop 2024-04-25T11:46:10.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.613+0000 7f4c09014640 1 -- 172.21.15.90:0/1158883638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0410b770 msgr2=0x7f4c0419acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:10.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.613+0000 7f4c09014640 1 --2- 172.21.15.90:0/1158883638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0410b770 0x7f4c0419acf0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f4bec00d950 tx=0x7f4bec00de10 comp rx=0 tx=0).stop 2024-04-25T11:46:10.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.613+0000 7f4c09014640 1 -- 172.21.15.90:0/1158883638 shutdown_connections 2024-04-25T11:46:10.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.613+0000 7f4c09014640 1 --2- 172.21.15.90:0/1158883638 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f4bf00afcd0 0x7f4bf00b21e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.613+0000 7f4c09014640 1 --2- 172.21.15.90:0/1158883638 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c0410e0d0 0x7f4c0419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.613+0000 7f4c09014640 1 --2- 172.21.15.90:0/1158883638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0410b770 0x7f4c0419acf0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.613+0000 7f4c09014640 1 --2- 172.21.15.90:0/1158883638 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4c04104d10 0x7f4c0419a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:10.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.613+0000 7f4c09014640 1 -- 172.21.15.90:0/1158883638 >> 172.21.15.90:0/1158883638 conn(0x7f4c040781b0 msgr2=0x7f4c04101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:10.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.613+0000 7f4c09014640 1 -- 172.21.15.90:0/1158883638 shutdown_connections 2024-04-25T11:46:10.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:10.613+0000 7f4c09014640 1 -- 172.21.15.90:0/1158883638 wait complete. 2024-04-25T11:46:12.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:11 smithi090 ceph-mon[29575]: pgmap v344: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 15 KiB/s rd, 47 KiB/s wr, 22 op/s 2024-04-25T11:46:12.223 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:11 smithi090 ceph-mon[29575]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T11:46:12.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:11 smithi138 ceph-mon[32328]: pgmap v344: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 15 KiB/s rd, 47 KiB/s wr, 22 op/s 2024-04-25T11:46:12.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:11 smithi138 ceph-mon[32328]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T11:46:12.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:11 smithi156 ceph-mon[33045]: pgmap v344: 97 pgs: 97 active+clean; 31 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 15 KiB/s rd, 47 KiB/s wr, 22 op/s 2024-04-25T11:46:12.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:11 smithi156 ceph-mon[33045]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T11:46:14.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:13 smithi138 ceph-mon[32328]: pgmap v346: 97 pgs: 4 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 83 active+clean; 30 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 9.8 KiB/s rd, 53 KiB/s wr, 18 op/s 2024-04-25T11:46:14.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:13 smithi156 ceph-mon[33045]: pgmap v346: 97 pgs: 4 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 83 active+clean; 30 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 9.8 KiB/s rd, 53 KiB/s wr, 18 op/s 2024-04-25T11:46:14.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:13 smithi090 ceph-mon[29575]: pgmap v346: 97 pgs: 4 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 83 active+clean; 30 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 9.8 KiB/s rd, 53 KiB/s wr, 18 op/s 2024-04-25T11:46:16.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:15 smithi138 ceph-mon[32328]: pgmap v347: 97 pgs: 16 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 57 active+clean; 29 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 9.4 KiB/s rd, 27 KiB/s wr, 16 op/s 2024-04-25T11:46:16.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:15 smithi156 ceph-mon[33045]: pgmap v347: 97 pgs: 16 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 57 active+clean; 29 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 9.4 KiB/s rd, 27 KiB/s wr, 16 op/s 2024-04-25T11:46:16.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:15 smithi090 ceph-mon[29575]: pgmap v347: 97 pgs: 16 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 57 active+clean; 29 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 9.4 KiB/s rd, 27 KiB/s wr, 16 op/s 2024-04-25T11:46:18.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:18 smithi138 ceph-mon[32328]: pgmap v348: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 29 GiB data, 90 GiB used, 983 GiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.9 KiB/s wr, 11 op/s 2024-04-25T11:46:18.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:18 smithi156 ceph-mon[33045]: pgmap v348: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 29 GiB data, 90 GiB used, 983 GiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.9 KiB/s wr, 11 op/s 2024-04-25T11:46:18.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:18 smithi090 ceph-mon[29575]: pgmap v348: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 29 GiB data, 90 GiB used, 983 GiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.9 KiB/s wr, 11 op/s 2024-04-25T11:46:20.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:20 smithi138 ceph-mon[32328]: pgmap v349: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 28 GiB data, 89 GiB used, 984 GiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.9 KiB/s wr, 11 op/s 2024-04-25T11:46:20.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:20 smithi156 ceph-mon[33045]: pgmap v349: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 28 GiB data, 89 GiB used, 984 GiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.9 KiB/s wr, 11 op/s 2024-04-25T11:46:20.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:20 smithi090 ceph-mon[29575]: pgmap v349: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 28 GiB data, 89 GiB used, 984 GiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.9 KiB/s wr, 11 op/s 2024-04-25T11:46:22.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:22 smithi138 ceph-mon[32328]: pgmap v350: 97 pgs: 21 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 38 active+clean; 26 GiB data, 81 GiB used, 991 GiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.9 KiB/s wr, 11 op/s 2024-04-25T11:46:22.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:22 smithi156 ceph-mon[33045]: pgmap v350: 97 pgs: 21 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 38 active+clean; 26 GiB data, 81 GiB used, 991 GiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.9 KiB/s wr, 11 op/s 2024-04-25T11:46:22.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:22 smithi090 ceph-mon[29575]: pgmap v350: 97 pgs: 21 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 38 active+clean; 26 GiB data, 81 GiB used, 991 GiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.9 KiB/s wr, 11 op/s 2024-04-25T11:46:23.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.714+0000 7fb484661640 1 -- 172.21.15.90:0/4159312506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb47c1033f0 msgr2=0x7fb47c103800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:23.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.714+0000 7fb484661640 1 --2- 172.21.15.90:0/4159312506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb47c1033f0 0x7fb47c103800 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fb478009900 tx=0x7fb47802a410 comp rx=0 tx=0).stop 2024-04-25T11:46:23.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.714+0000 7fb484661640 1 -- 172.21.15.90:0/4159312506 shutdown_connections 2024-04-25T11:46:23.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.714+0000 7fb484661640 1 --2- 172.21.15.90:0/4159312506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb47c107d30 0x7fb47c10f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:23.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.714+0000 7fb484661640 1 --2- 172.21.15.90:0/4159312506 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb47c1072b0 0x7fb47c1076c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:23.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.714+0000 7fb484661640 1 --2- 172.21.15.90:0/4159312506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb47c1033f0 0x7fb47c103800 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:23.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.714+0000 7fb484661640 1 -- 172.21.15.90:0/4159312506 >> 172.21.15.90:0/4159312506 conn(0x7fb47c0778b0 msgr2=0x7fb47c077cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:23.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.714+0000 7fb484661640 1 -- 172.21.15.90:0/4159312506 shutdown_connections 2024-04-25T11:46:23.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.714+0000 7fb484661640 1 -- 172.21.15.90:0/4159312506 wait complete. 2024-04-25T11:46:23.702 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.715+0000 7fb484661640 1 Processor -- start 2024-04-25T11:46:23.702 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.715+0000 7fb484661640 1 -- start start 2024-04-25T11:46:23.703 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.715+0000 7fb484661640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb47c1033f0 0x7fb47c199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:23.703 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.715+0000 7fb484661640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb47c1072b0 0x7fb47c199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:23.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.715+0000 7fb484661640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb47c107d30 0x7fb47c19deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:23.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.715+0000 7fb484661640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb47c112c30 con 0x7fb47c1033f0 2024-04-25T11:46:23.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.715+0000 7fb484661640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb47c112ab0 con 0x7fb47c107d30 2024-04-25T11:46:23.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.715+0000 7fb484661640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb47c112db0 con 0x7fb47c1072b0 2024-04-25T11:46:23.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.715+0000 7fb4823fd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb47c1033f0 0x7fb47c199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:23.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.715+0000 7fb4823fd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb47c1033f0 0x7fb47c199690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48044/0 (socket says 172.21.15.90:48044) 2024-04-25T11:46:23.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.715+0000 7fb4823fd640 1 -- 172.21.15.90:0/2066522800 learned_addr learned my addr 172.21.15.90:0/2066522800 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:23.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.715+0000 7fb482bfe640 1 --2- 172.21.15.90:0/2066522800 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb47c107d30 0x7fb47c19deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:23.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.715+0000 7fb481bfc640 1 --2- 172.21.15.90:0/2066522800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb47c1072b0 0x7fb47c199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:23.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.715+0000 7fb4823fd640 1 -- 172.21.15.90:0/2066522800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb47c1072b0 msgr2=0x7fb47c199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:23.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.715+0000 7fb4823fd640 1 --2- 172.21.15.90:0/2066522800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb47c1072b0 0x7fb47c199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:23.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.715+0000 7fb4823fd640 1 -- 172.21.15.90:0/2066522800 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb47c107d30 msgr2=0x7fb47c19deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:23.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.715+0000 7fb4823fd640 1 --2- 172.21.15.90:0/2066522800 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb47c107d30 0x7fb47c19deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:23.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.715+0000 7fb4823fd640 1 -- 172.21.15.90:0/2066522800 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb47c19e4f0 con 0x7fb47c1033f0 2024-04-25T11:46:23.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.715+0000 7fb482bfe640 1 --2- 172.21.15.90:0/2066522800 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb47c107d30 0x7fb47c19deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:46:23.705 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.716+0000 7fb4823fd640 1 --2- 172.21.15.90:0/2066522800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb47c1033f0 0x7fb47c199690 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fb478006010 tx=0x7fb478004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:23.705 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.716+0000 7fb4737fe640 1 -- 172.21.15.90:0/2066522800 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb47803f030 con 0x7fb47c1033f0 2024-04-25T11:46:23.705 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.716+0000 7fb4737fe640 1 -- 172.21.15.90:0/2066522800 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb478003aa0 con 0x7fb47c1033f0 2024-04-25T11:46:23.705 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.716+0000 7fb484661640 1 -- 172.21.15.90:0/2066522800 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb47c19e780 con 0x7fb47c1033f0 2024-04-25T11:46:23.705 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.716+0000 7fb4737fe640 1 -- 172.21.15.90:0/2066522800 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb478035070 con 0x7fb47c1033f0 2024-04-25T11:46:23.705 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.716+0000 7fb484661640 1 -- 172.21.15.90:0/2066522800 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb47c19ec70 con 0x7fb47c1033f0 2024-04-25T11:46:23.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.719+0000 7fb48b5ef740 1 Processor -- start 2024-04-25T11:46:23.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.719+0000 7fb48b5ef740 1 -- start start 2024-04-25T11:46:23.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.719+0000 7fb4737fe640 1 -- 172.21.15.90:0/2066522800 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fb478042030 con 0x7fb47c1033f0 2024-04-25T11:46:23.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.719+0000 7fb48b5ef740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c408c1b1b0 0x55c408fbaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:23.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.719+0000 7fb48b5ef740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c408bf22f0 0x55c408fbb030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:23.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.719+0000 7fb48b5ef740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c408bfa4d0 0x55c408fbb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:23.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.719+0000 7fb48b5ef740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c408f21460 con 0x55c408bf22f0 2024-04-25T11:46:23.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.719+0000 7fb48b5ef740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c408edb590 con 0x55c408bfa4d0 2024-04-25T11:46:23.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.719+0000 7fb48b5ef740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c408f1d420 con 0x55c408c1b1b0 2024-04-25T11:46:23.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.719+0000 7fb4823fd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c408bf22f0 0x55c408fbb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:23.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.719+0000 7fb481bfc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c408bfa4d0 0x55c408fbb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.719+0000 7fb481bfc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c408bfa4d0 0x55c408fbb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:33782/0 (socket says 172.21.15.90:33782) 2024-04-25T11:46:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.719+0000 7fb481bfc640 1 -- 172.21.15.90:0/1747775154 learned_addr learned my addr 172.21.15.90:0/1747775154 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.719+0000 7fb4823fd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c408bf22f0 0x55c408fbb030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48052/0 (socket says 172.21.15.90:48052) 2024-04-25T11:46:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.719+0000 7fb482bfe640 1 --2- 172.21.15.90:0/1747775154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c408c1b1b0 0x55c408fbaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.720+0000 7fb481bfc640 1 -- 172.21.15.90:0/1747775154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c408c1b1b0 msgr2=0x55c408fbaaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.720+0000 7fb481bfc640 1 --2- 172.21.15.90:0/1747775154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c408c1b1b0 0x55c408fbaaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.720+0000 7fb481bfc640 1 -- 172.21.15.90:0/1747775154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c408bf22f0 msgr2=0x55c408fbb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.720+0000 7fb481bfc640 1 --2- 172.21.15.90:0/1747775154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c408bf22f0 0x55c408fbb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.720+0000 7fb481bfc640 1 -- 172.21.15.90:0/1747775154 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c408ee80d0 con 0x55c408bfa4d0 2024-04-25T11:46:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.720+0000 7fb4737fe640 1 --2- 172.21.15.90:0/2066522800 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb4680afc50 0x7fb4680b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.720+0000 7fb481bfc640 1 --2- 172.21.15.90:0/1747775154 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c408bfa4d0 0x55c408fbb570 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb46c009bb0 tx=0x7fb46c0117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c56793eb10b0a67 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.720+0000 7fb482bfe640 1 --2- 172.21.15.90:0/2066522800 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb4680afc50 0x7fb4680b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.720+0000 7fb4737fe640 1 -- 172.21.15.90:0/2066522800 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7fb4780ff640 con 0x7fb47c1033f0 2024-04-25T11:46:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.720+0000 7fb470bfe640 1 -- 172.21.15.90:0/1747775154 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46c005020 con 0x55c408bfa4d0 2024-04-25T11:46:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.720+0000 7fb470bfe640 1 -- 172.21.15.90:0/1747775154 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb46c0047a0 con 0x55c408bfa4d0 2024-04-25T11:46:23.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.720+0000 7fb470bfe640 1 -- 172.21.15.90:0/1747775154 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46c0126b0 con 0x55c408bfa4d0 2024-04-25T11:46:23.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.721+0000 7fb48b5ef740 1 -- 172.21.15.90:0/1747775154 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c408bfa4d0 msgr2=0x55c408fbb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:23.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.721+0000 7fb482bfe640 1 --2- 172.21.15.90:0/2066522800 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb4680afc50 0x7fb4680b2160 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x55c408fb7c60 tx=0x7fb474009820 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:23.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.721+0000 7fb48b5ef740 1 --2- 172.21.15.90:0/1747775154 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c408bfa4d0 0x55c408fbb570 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb46c009bb0 tx=0x7fb46c0117b0 comp rx=0 tx=0).stop 2024-04-25T11:46:23.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.721+0000 7fb48b5ef740 1 -- 172.21.15.90:0/1747775154 shutdown_connections 2024-04-25T11:46:23.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.721+0000 7fb48b5ef740 1 --2- 172.21.15.90:0/1747775154 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c408bfa4d0 0x55c408fbb570 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:23.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.721+0000 7fb48b5ef740 1 --2- 172.21.15.90:0/1747775154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c408bf22f0 0x55c408fbb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:23.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.721+0000 7fb48b5ef740 1 --2- 172.21.15.90:0/1747775154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c408c1b1b0 0x55c408fbaaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:23.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.721+0000 7fb48b5ef740 1 -- 172.21.15.90:0/1747775154 >> 172.21.15.90:0/1747775154 conn(0x55c408c0f300 msgr2=0x55c408fb72a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:23.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.721+0000 7fb48b5ef740 1 -- 172.21.15.90:0/1747775154 shutdown_connections 2024-04-25T11:46:23.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.721+0000 7fb48b5ef740 1 -- 172.21.15.90:0/1747775154 wait complete. 2024-04-25T11:46:23.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.722+0000 7fb48b5ef740 1 -- start start 2024-04-25T11:46:23.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.722+0000 7fb48b5ef740 1 Processor -- start 2024-04-25T11:46:23.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.722+0000 7fb48b5ef740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c408bf22f0 0x55c408fcd360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:23.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.722+0000 7fb48b5ef740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c408bfa4d0 0x55c408fcd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:23.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.722+0000 7fb48b5ef740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c408c1b1b0 0x55c408fcc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:23.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.722+0000 7fb48b5ef740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c408fc7c20 con 0x55c408c1b1b0 2024-04-25T11:46:23.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.722+0000 7fb48b5ef740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c408ea8b60 con 0x55c408bfa4d0 2024-04-25T11:46:23.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.722+0000 7fb48b5ef740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c408fc7da0 con 0x55c408bf22f0 2024-04-25T11:46:23.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.722+0000 7fb48b5ef740 10 client.0 _ll_register_callbacks cb 0x55c408ecd2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:23.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.722+0000 7fb4823fd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c408bf22f0 0x55c408fcd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:23.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.722+0000 7fb482bfe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c408c1b1b0 0x55c408fcc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:23.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.722+0000 7fb4823fd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c408bf22f0 0x55c408fcd360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:48986/0 (socket says 172.21.15.90:48986) 2024-04-25T11:46:23.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.722+0000 7fb4823fd640 1 -- 172.21.15.90:0/3975922473 learned_addr learned my addr 172.21.15.90:0/3975922473 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:23.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.722+0000 7fb481bfc640 1 --2- 172.21.15.90:0/3975922473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c408bfa4d0 0x55c408fcd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:23.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.722+0000 7fb482bfe640 1 -- 172.21.15.90:0/3975922473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c408bf22f0 msgr2=0x55c408fcd360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:23.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.722+0000 7fb482bfe640 1 --2- 172.21.15.90:0/3975922473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c408bf22f0 0x55c408fcd360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:23.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.722+0000 7fb482bfe640 1 -- 172.21.15.90:0/3975922473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c408bfa4d0 msgr2=0x55c408fcd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:23.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.722+0000 7fb482bfe640 1 --2- 172.21.15.90:0/3975922473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c408bfa4d0 0x55c408fcd8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:23.715 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.722+0000 7fb482bfe640 1 -- 172.21.15.90:0/3975922473 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c408c33210 con 0x55c408c1b1b0 2024-04-25T11:46:23.715 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.722+0000 7fb4823fd640 1 --2- 172.21.15.90:0/3975922473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c408bf22f0 0x55c408fcd360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:46:23.715 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.723+0000 7fb482bfe640 1 --2- 172.21.15.90:0/3975922473 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c408c1b1b0 0x55c408fcc0b0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fb4740109a0 tx=0x7fb474004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e40aaf6762dc24d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:23.715 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.723+0000 7fb465ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:46:23.715 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.723+0000 7fb465ffb640 1 -- 172.21.15.90:0/3975922473 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb47401e6d0 con 0x55c408c1b1b0 2024-04-25T11:46:23.715 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.723+0000 7fb465ffb640 1 -- 172.21.15.90:0/3975922473 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7fb47401e870 con 0x55c408c1b1b0 2024-04-25T11:46:23.715 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.723+0000 7fb48b5ef740 1 -- 172.21.15.90:0/3975922473 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb46c0047a0 con 0x55c408c1b1b0 2024-04-25T11:46:23.715 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.723+0000 7fb465ffb640 1 -- 172.21.15.90:0/3975922473 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb47401eab0 con 0x55c408c1b1b0 2024-04-25T11:46:23.715 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.723+0000 7fb48b5ef740 10 client.19182 fetch_fsmap learned FSMap version 28 2024-04-25T11:46:23.715 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.723+0000 7fb48b5ef740 1 -- 172.21.15.90:0/3975922473 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c408cff620 con 0x55c408c1b1b0 2024-04-25T11:46:23.715 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.723+0000 7fb465ffb640 1 -- 172.21.15.90:0/3975922473 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb46c0047a0 con 0x55c408c1b1b0 2024-04-25T11:46:23.715 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.723+0000 7fb465ffb640 1 -- 172.21.15.90:0/3975922473 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fb4740263d0 con 0x55c408c1b1b0 2024-04-25T11:46:23.715 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.724+0000 7fb48b5ef740 10 client.19182 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:46:23.715 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.724+0000 7fb48b5ef740 10 client.19182 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:46:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.724+0000 7fb48b5ef740 20 client.19182 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.724+0000 7fb48b5ef740 1 --2- 172.21.15.90:0/3975922473 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c408bd9d80 0x55c408fd9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.724+0000 7fb48b5ef740 4 client.19182 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:46:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.724+0000 7fb48b5ef740 1 -- 172.21.15.90:0/3975922473 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c408f2f070 con 0x55c408bd9d80 2024-04-25T11:46:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.724+0000 7fb4823fd640 1 --2- 172.21.15.90:0/3975922473 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c408bd9d80 0x55c408fd9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.724+0000 7fb465ffb640 10 client.19182 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.725+0000 7fb4823fd640 1 --2- 172.21.15.90:0/3975922473 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c408bd9d80 0x55c408fd9950 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3deb42c7c92e68a0 server_cookie=5693d0fbaaacbc40 in_seq=0 out_seq=0 2024-04-25T11:46:23.717 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.725+0000 7fb465ffb640 1 -- 172.21.15.90:0/3975922473 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c408f2f070 con 0x55c408bd9d80 2024-04-25T11:46:23.717 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.725+0000 7fb465ffb640 10 client.19182 handle_command_reply: tid=0 2024-04-25T11:46:23.717 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.725+0000 7fb48b5ef740 1 client.19182 shutdown 2024-04-25T11:46:23.717 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.725+0000 7fb48b5ef740 1 -- 172.21.15.90:0/3975922473 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c408c1b1b0 msgr2=0x55c408fcc0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:23.717 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.725+0000 7fb48b5ef740 1 --2- 172.21.15.90:0/3975922473 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c408c1b1b0 0x55c408fcc0b0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fb4740109a0 tx=0x7fb474004a40 comp rx=0 tx=0).stop 2024-04-25T11:46:23.717 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.725+0000 7fb48b5ef740 1 -- 172.21.15.90:0/3975922473 shutdown_connections 2024-04-25T11:46:23.717 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.725+0000 7fb48b5ef740 1 --2- 172.21.15.90:0/3975922473 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c408bd9d80 0x55c408fd9950 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:23.717 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.725+0000 7fb48b5ef740 1 --2- 172.21.15.90:0/3975922473 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c408c1b1b0 0x55c408fcc0b0 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:23.718 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.725+0000 7fb48b5ef740 1 --2- 172.21.15.90:0/3975922473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c408bfa4d0 0x55c408fcd8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:23.718 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.725+0000 7fb48b5ef740 1 --2- 172.21.15.90:0/3975922473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c408bf22f0 0x55c408fcd360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:23.718 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.726+0000 7fb48b5ef740 1 -- 172.21.15.90:0/3975922473 >> 172.21.15.90:0/3975922473 conn(0x55c408c0f300 msgr2=0x55c408fbc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:23.718 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.726+0000 7fb465ffb640 0 client.19182 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:23.718 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.726+0000 7fb48b5ef740 1 -- 172.21.15.90:0/3975922473 shutdown_connections 2024-04-25T11:46:23.718 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.726+0000 7fb48b5ef740 1 -- 172.21.15.90:0/3975922473 wait complete. 2024-04-25T11:46:23.718 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.726+0000 7fb48b5ef740 20 client.19182 trim_cache size 0 max 16384 2024-04-25T11:46:23.729 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.742+0000 7fb48b5ef740 1 Processor -- start 2024-04-25T11:46:23.729 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.742+0000 7fb48b5ef740 1 -- start start 2024-04-25T11:46:23.729 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.742+0000 7fb48b5ef740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c408bf22f0 0x55c408fbe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:23.730 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.742+0000 7fb48b5ef740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c408bfa4d0 0x55c408fbece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:23.730 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.742+0000 7fb48b5ef740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c408c1b1b0 0x55c408fdf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:23.730 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.742+0000 7fb48b5ef740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb47c1adb30 con 0x55c408c1b1b0 2024-04-25T11:46:23.730 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.742+0000 7fb48b5ef740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb47c1af670 con 0x55c408bfa4d0 2024-04-25T11:46:23.730 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.742+0000 7fb48b5ef740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb47c1ae200 con 0x55c408bf22f0 2024-04-25T11:46:23.730 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.742+0000 7fb482bfe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c408c1b1b0 0x55c408fdf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:23.730 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.742+0000 7fb482bfe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c408c1b1b0 0x55c408fdf1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48070/0 (socket says 172.21.15.90:48070) 2024-04-25T11:46:23.730 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.742+0000 7fb482bfe640 1 -- 172.21.15.90:0/2504770026 learned_addr learned my addr 172.21.15.90:0/2504770026 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:23.730 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.742+0000 7fb4823fd640 1 --2- 172.21.15.90:0/2504770026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c408bf22f0 0x55c408fbe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:23.730 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.742+0000 7fb481bfc640 1 --2- 172.21.15.90:0/2504770026 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c408bfa4d0 0x55c408fbece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:23.730 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.742+0000 7fb482bfe640 1 -- 172.21.15.90:0/2504770026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c408bf22f0 msgr2=0x55c408fbe7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:23.730 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.742+0000 7fb482bfe640 1 --2- 172.21.15.90:0/2504770026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c408bf22f0 0x55c408fbe7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:23.730 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.742+0000 7fb482bfe640 1 -- 172.21.15.90:0/2504770026 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c408bfa4d0 msgr2=0x55c408fbece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:23.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.742+0000 7fb482bfe640 1 --2- 172.21.15.90:0/2504770026 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c408bfa4d0 0x55c408fbece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:23.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.742+0000 7fb482bfe640 1 -- 172.21.15.90:0/2504770026 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c408bd9fb0 con 0x55c408c1b1b0 2024-04-25T11:46:23.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.742+0000 7fb481bfc640 1 --2- 172.21.15.90:0/2504770026 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c408bfa4d0 0x55c408fbece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:46:23.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.743+0000 7fb482bfe640 1 --2- 172.21.15.90:0/2504770026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c408c1b1b0 0x55c408fdf1a0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fb474012e30 tx=0x7fb474009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e34fe67798239b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:23.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.743+0000 7fb467fff640 1 -- 172.21.15.90:0/2504770026 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb474026e90 con 0x55c408c1b1b0 2024-04-25T11:46:23.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.743+0000 7fb467fff640 1 -- 172.21.15.90:0/2504770026 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb474026a00 con 0x55c408c1b1b0 2024-04-25T11:46:23.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.743+0000 7fb467fff640 1 -- 172.21.15.90:0/2504770026 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4740052a0 con 0x55c408c1b1b0 2024-04-25T11:46:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.744+0000 7fb48b5ef740 1 -- 172.21.15.90:0/2504770026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c408c1b1b0 msgr2=0x55c408fdf1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.744+0000 7fb48b5ef740 1 --2- 172.21.15.90:0/2504770026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c408c1b1b0 0x55c408fdf1a0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fb474012e30 tx=0x7fb474009f60 comp rx=0 tx=0).stop 2024-04-25T11:46:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.744+0000 7fb48b5ef740 1 -- 172.21.15.90:0/2504770026 shutdown_connections 2024-04-25T11:46:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.744+0000 7fb48b5ef740 1 --2- 172.21.15.90:0/2504770026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c408c1b1b0 0x55c408fdf1a0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.744+0000 7fb48b5ef740 1 --2- 172.21.15.90:0/2504770026 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c408bfa4d0 0x55c408fbece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.744+0000 7fb48b5ef740 1 --2- 172.21.15.90:0/2504770026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c408bf22f0 0x55c408fbe7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.744+0000 7fb48b5ef740 1 -- 172.21.15.90:0/2504770026 >> 172.21.15.90:0/2504770026 conn(0x55c408c0f300 msgr2=0x55c408fc3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.744+0000 7fb48b5ef740 1 -- 172.21.15.90:0/2504770026 shutdown_connections 2024-04-25T11:46:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.744+0000 7fb48b5ef740 1 -- 172.21.15.90:0/2504770026 wait complete. 2024-04-25T11:46:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.744+0000 7fb48b5ef740 1 -- start start 2024-04-25T11:46:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.744+0000 7fb48b5ef740 1 Processor -- start 2024-04-25T11:46:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.744+0000 7fb48b5ef740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c408bf22f0 0x55c408fce690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:23.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.744+0000 7fb48b5ef740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c408bfa4d0 0x55c408fcebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:23.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.744+0000 7fb48b5ef740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c408c1b1b0 0x55c408fcf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:23.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.745+0000 7fb48b5ef740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb47c1aefa0 con 0x55c408bf22f0 2024-04-25T11:46:23.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.745+0000 7fb48b5ef740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb47c1ae8d0 con 0x55c408bfa4d0 2024-04-25T11:46:23.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.745+0000 7fb48b5ef740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb47c1abb30 con 0x55c408c1b1b0 2024-04-25T11:46:23.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.745+0000 7fb48b5ef740 10 client.0 _ll_register_callbacks cb 0x7fb474010650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:23.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.745+0000 7fb4823fd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c408bf22f0 0x55c408fce690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:23.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.745+0000 7fb4823fd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c408bf22f0 0x55c408fce690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48080/0 (socket says 172.21.15.90:48080) 2024-04-25T11:46:23.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.745+0000 7fb4823fd640 1 -- 172.21.15.90:0/4217955392 learned_addr learned my addr 172.21.15.90:0/4217955392 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:23.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.745+0000 7fb481bfc640 1 --2- 172.21.15.90:0/4217955392 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c408bfa4d0 0x55c408fcebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:23.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.745+0000 7fb482bfe640 1 --2- 172.21.15.90:0/4217955392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c408c1b1b0 0x55c408fcf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:23.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.745+0000 7fb4823fd640 1 -- 172.21.15.90:0/4217955392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c408c1b1b0 msgr2=0x55c408fcf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:23.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.745+0000 7fb4823fd640 1 --2- 172.21.15.90:0/4217955392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c408c1b1b0 0x55c408fcf110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:23.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.745+0000 7fb4823fd640 1 -- 172.21.15.90:0/4217955392 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c408bfa4d0 msgr2=0x55c408fcebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:23.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.745+0000 7fb4823fd640 1 --2- 172.21.15.90:0/4217955392 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c408bfa4d0 0x55c408fcebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:23.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.745+0000 7fb4823fd640 1 -- 172.21.15.90:0/4217955392 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c40903fef0 con 0x55c408bf22f0 2024-04-25T11:46:23.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.745+0000 7fb4823fd640 1 --2- 172.21.15.90:0/4217955392 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c408bf22f0 0x55c408fce690 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fb4780c7430 tx=0x7fb478003f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bab015f292243c48 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:23.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.745+0000 7fb465ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:46:23.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.745+0000 7fb465ffb640 1 -- 172.21.15.90:0/4217955392 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb478049040 con 0x55c408bf22f0 2024-04-25T11:46:23.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.745+0000 7fb465ffb640 1 -- 172.21.15.90:0/4217955392 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7fb4780fcf60 con 0x55c408bf22f0 2024-04-25T11:46:23.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.745+0000 7fb48b5ef740 1 -- 172.21.15.90:0/4217955392 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb474026a00 con 0x55c408bf22f0 2024-04-25T11:46:23.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.745+0000 7fb465ffb640 1 -- 172.21.15.90:0/4217955392 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb4780cee60 con 0x55c408bf22f0 2024-04-25T11:46:23.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.745+0000 7fb48b5ef740 10 client.19188 fetch_fsmap learned FSMap version 28 2024-04-25T11:46:23.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.745+0000 7fb48b5ef740 1 -- 172.21.15.90:0/4217955392 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c4090411e0 con 0x55c408bf22f0 2024-04-25T11:46:23.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.745+0000 7fb465ffb640 1 -- 172.21.15.90:0/4217955392 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb474026a00 con 0x55c408bf22f0 2024-04-25T11:46:23.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.746+0000 7fb465ffb640 1 -- 172.21.15.90:0/4217955392 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fb4780c39c0 con 0x55c408bf22f0 2024-04-25T11:46:23.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.746+0000 7fb48b5ef740 10 client.19188 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:46:23.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.746+0000 7fb48b5ef740 10 client.19188 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:46:23.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.746+0000 7fb48b5ef740 20 client.19188 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:23.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.746+0000 7fb48b5ef740 1 --2- 172.21.15.90:0/4217955392 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c408baf1c0 0x55c409043920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:23.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.746+0000 7fb48b5ef740 4 client.19188 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:46:23.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.746+0000 7fb48b5ef740 1 -- 172.21.15.90:0/4217955392 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c408f28ee0 con 0x55c408baf1c0 2024-04-25T11:46:23.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.746+0000 7fb481bfc640 1 --2- 172.21.15.90:0/4217955392 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c408baf1c0 0x55c409043920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:23.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.746+0000 7fb465ffb640 10 client.19188 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:23.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:23.746+0000 7fb481bfc640 1 --2- 172.21.15.90:0/4217955392 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c408baf1c0 0x55c409043920 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ba137a79c157cf8 server_cookie=9ab3e84647dfa21c in_seq=0 out_seq=0 2024-04-25T11:46:24.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:24 smithi138 ceph-mon[32328]: pgmap v351: 97 pgs: 21 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 38 active+clean; 25 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 7.0 KiB/s rd, 3.4 KiB/s wr, 9 op/s 2024-04-25T11:46:24.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:24 smithi156 ceph-mon[33045]: pgmap v351: 97 pgs: 21 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 38 active+clean; 25 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 7.0 KiB/s rd, 3.4 KiB/s wr, 9 op/s 2024-04-25T11:46:24.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:24 smithi090 ceph-mon[29575]: pgmap v351: 97 pgs: 21 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 38 active+clean; 25 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 7.0 KiB/s rd, 3.4 KiB/s wr, 9 op/s 2024-04-25T11:46:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.021+0000 7fb465ffb640 1 -- 172.21.15.90:0/4217955392 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c408f28ee0 con 0x55c408baf1c0 2024-04-25T11:46:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.021+0000 7fb465ffb640 10 client.19188 handle_command_reply: tid=0 2024-04-25T11:46:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.021+0000 7fb48b5ef740 1 client.19188 shutdown 2024-04-25T11:46:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.022+0000 7fb48b5ef740 1 -- 172.21.15.90:0/4217955392 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c408bf22f0 msgr2=0x55c408fce690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.022+0000 7fb48b5ef740 1 --2- 172.21.15.90:0/4217955392 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c408bf22f0 0x55c408fce690 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fb4780c7430 tx=0x7fb478003f50 comp rx=0 tx=0).stop 2024-04-25T11:46:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.022+0000 7fb48b5ef740 1 -- 172.21.15.90:0/4217955392 shutdown_connections 2024-04-25T11:46:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.022+0000 7fb48b5ef740 1 --2- 172.21.15.90:0/4217955392 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c408baf1c0 0x55c409043920 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.022+0000 7fb48b5ef740 1 --2- 172.21.15.90:0/4217955392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c408c1b1b0 0x55c408fcf110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.022+0000 7fb465ffb640 0 client.19188 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.022+0000 7fb48b5ef740 1 --2- 172.21.15.90:0/4217955392 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c408bfa4d0 0x55c408fcebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.022+0000 7fb48b5ef740 1 --2- 172.21.15.90:0/4217955392 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c408bf22f0 0x55c408fce690 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.022+0000 7fb48b5ef740 1 -- 172.21.15.90:0/4217955392 >> 172.21.15.90:0/4217955392 conn(0x55c408c0f300 msgr2=0x55c408fc3500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.022+0000 7fb48b5ef740 1 -- 172.21.15.90:0/4217955392 shutdown_connections 2024-04-25T11:46:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.022+0000 7fb48b5ef740 1 -- 172.21.15.90:0/4217955392 wait complete. 2024-04-25T11:46:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.022+0000 7fb48b5ef740 20 client.19188 trim_cache size 0 max 16384 2024-04-25T11:46:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:46:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:46:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:46:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:46:25.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.022+0000 7fb484661640 1 -- 172.21.15.90:0/2066522800 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb4680afc50 msgr2=0x7fb4680b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:25.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.022+0000 7fb484661640 1 --2- 172.21.15.90:0/2066522800 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb4680afc50 0x7fb4680b2160 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x55c408fb7c60 tx=0x7fb474009820 comp rx=0 tx=0).stop 2024-04-25T11:46:25.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.022+0000 7fb484661640 1 -- 172.21.15.90:0/2066522800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb47c1033f0 msgr2=0x7fb47c199690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:25.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.022+0000 7fb484661640 1 --2- 172.21.15.90:0/2066522800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb47c1033f0 0x7fb47c199690 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fb478006010 tx=0x7fb478004060 comp rx=0 tx=0).stop 2024-04-25T11:46:25.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.022+0000 7fb484661640 1 -- 172.21.15.90:0/2066522800 shutdown_connections 2024-04-25T11:46:25.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.022+0000 7fb484661640 1 --2- 172.21.15.90:0/2066522800 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb4680afc50 0x7fb4680b2160 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:25.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.022+0000 7fb484661640 1 --2- 172.21.15.90:0/2066522800 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb47c107d30 0x7fb47c19deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:25.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.022+0000 7fb484661640 1 --2- 172.21.15.90:0/2066522800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb47c1072b0 0x7fb47c199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:25.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.022+0000 7fb484661640 1 --2- 172.21.15.90:0/2066522800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb47c1033f0 0x7fb47c199690 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:25.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.022+0000 7fb484661640 1 -- 172.21.15.90:0/2066522800 >> 172.21.15.90:0/2066522800 conn(0x7fb47c0778b0 msgr2=0x7fb47c0ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:25.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.023+0000 7fb484661640 1 -- 172.21.15.90:0/2066522800 shutdown_connections 2024-04-25T11:46:25.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:25.023+0000 7fb484661640 1 -- 172.21.15.90:0/2066522800 wait complete. 2024-04-25T11:46:26.017 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:46:26.017 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:46:26.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.146+0000 7ff9e3cf2640 1 -- 172.21.15.90:0/3978646748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9dc102af0 msgr2=0x7ff9dc102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.146+0000 7ff9e3cf2640 1 --2- 172.21.15.90:0/3978646748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9dc102af0 0x7ff9dc102f00 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc009900 tx=0x7ff9cc02a410 comp rx=0 tx=0).stop 2024-04-25T11:46:26.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.146+0000 7ff9e3cf2640 1 -- 172.21.15.90:0/3978646748 shutdown_connections 2024-04-25T11:46:26.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.146+0000 7ff9e3cf2640 1 --2- 172.21.15.90:0/3978646748 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9dc10d380 0x7ff9dc10f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.146+0000 7ff9e3cf2640 1 --2- 172.21.15.90:0/3978646748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9dc102af0 0x7ff9dc102f00 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.146+0000 7ff9e3cf2640 1 --2- 172.21.15.90:0/3978646748 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff9dc108a20 0x7ff9dc1025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.146+0000 7ff9e3cf2640 1 -- 172.21.15.90:0/3978646748 >> 172.21.15.90:0/3978646748 conn(0x7ff9dc078100 msgr2=0x7ff9dc078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:26.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.146+0000 7ff9e3cf2640 1 -- 172.21.15.90:0/3978646748 shutdown_connections 2024-04-25T11:46:26.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.146+0000 7ff9e3cf2640 1 -- 172.21.15.90:0/3978646748 wait complete. 2024-04-25T11:46:26.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.147+0000 7ff9e3cf2640 1 Processor -- start 2024-04-25T11:46:26.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.147+0000 7ff9e3cf2640 1 -- start start 2024-04-25T11:46:26.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.147+0000 7ff9e3cf2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff9dc102af0 0x7ff9dc19b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.147+0000 7ff9e3cf2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9dc108a20 0x7ff9dc19bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.147+0000 7ff9e3cf2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9dc10d380 0x7ff9dc19c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.147+0000 7ff9e3cf2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc112be0 con 0x7ff9dc108a20 2024-04-25T11:46:26.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.147+0000 7ff9e3cf2640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc112a60 con 0x7ff9dc102af0 2024-04-25T11:46:26.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.147+0000 7ff9e3cf2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc112d60 con 0x7ff9dc10d380 2024-04-25T11:46:26.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.147+0000 7ff9e128d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9dc108a20 0x7ff9dc19bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.147+0000 7ff9e128d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9dc108a20 0x7ff9dc19bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48104/0 (socket says 172.21.15.90:48104) 2024-04-25T11:46:26.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.147+0000 7ff9e128d640 1 -- 172.21.15.90:0/1244171375 learned_addr learned my addr 172.21.15.90:0/1244171375 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:26.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.147+0000 7ff9e1a8e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff9dc102af0 0x7ff9dc19b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.147+0000 7ff9e228f640 1 --2- 172.21.15.90:0/1244171375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9dc10d380 0x7ff9dc19c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.147+0000 7ff9e128d640 1 -- 172.21.15.90:0/1244171375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9dc10d380 msgr2=0x7ff9dc19c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.147+0000 7ff9e128d640 1 --2- 172.21.15.90:0/1244171375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9dc10d380 0x7ff9dc19c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.147+0000 7ff9e128d640 1 -- 172.21.15.90:0/1244171375 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff9dc102af0 msgr2=0x7ff9dc19b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.147+0000 7ff9e128d640 1 --2- 172.21.15.90:0/1244171375 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff9dc102af0 0x7ff9dc19b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.147+0000 7ff9e128d640 1 -- 172.21.15.90:0/1244171375 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9dc1a68c0 con 0x7ff9dc108a20 2024-04-25T11:46:26.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.148+0000 7ff9e128d640 1 --2- 172.21.15.90:0/1244171375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9dc108a20 0x7ff9dc19bac0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc006010 tx=0x7ff9cc004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:26.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.148+0000 7ff9d2ffd640 1 -- 172.21.15.90:0/1244171375 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc03f030 con 0x7ff9dc108a20 2024-04-25T11:46:26.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.148+0000 7ff9d2ffd640 1 -- 172.21.15.90:0/1244171375 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9cc003aa0 con 0x7ff9dc108a20 2024-04-25T11:46:26.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.148+0000 7ff9e3cf2640 1 -- 172.21.15.90:0/1244171375 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9dc1a6b50 con 0x7ff9dc108a20 2024-04-25T11:46:26.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.148+0000 7ff9d2ffd640 1 -- 172.21.15.90:0/1244171375 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc035070 con 0x7ff9dc108a20 2024-04-25T11:46:26.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.148+0000 7ff9e3cf2640 1 -- 172.21.15.90:0/1244171375 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9dc1a7040 con 0x7ff9dc108a20 2024-04-25T11:46:26.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.150+0000 7ff9d2ffd640 1 -- 172.21.15.90:0/1244171375 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7ff9cc042030 con 0x7ff9dc108a20 2024-04-25T11:46:26.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.151+0000 7ff9eaba7740 1 Processor -- start 2024-04-25T11:46:26.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.151+0000 7ff9eaba7740 1 -- start start 2024-04-25T11:46:26.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.151+0000 7ff9eaba7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56030f18e390 0x56030f52ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.151+0000 7ff9eaba7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f1654d0 0x56030f52efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.151+0000 7ff9eaba7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56030f16d6b0 0x56030f53b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.151+0000 7ff9eaba7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56030f494d80 con 0x56030f1654d0 2024-04-25T11:46:26.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.151+0000 7ff9eaba7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56030f44e8b0 con 0x56030f16d6b0 2024-04-25T11:46:26.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.151+0000 7ff9eaba7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56030f490d40 con 0x56030f18e390 2024-04-25T11:46:26.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.151+0000 7ff9e1a8e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f1654d0 0x56030f52efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.151+0000 7ff9d2ffd640 1 --2- 172.21.15.90:0/1244171375 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff9c80afc60 0x7ff9c80b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.151+0000 7ff9e1a8e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f1654d0 0x56030f52efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48108/0 (socket says 172.21.15.90:48108) 2024-04-25T11:46:26.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.151+0000 7ff9e1a8e640 1 -- 172.21.15.90:0/3793359991 learned_addr learned my addr 172.21.15.90:0/3793359991 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:26.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.151+0000 7ff9e228f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56030f18e390 0x56030f52ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.151+0000 7ff9e128d640 1 --2- 172.21.15.90:0/3793359991 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56030f16d6b0 0x56030f53b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.152+0000 7ff9e228f640 1 --2- 172.21.15.90:0/1244171375 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff9c80afc60 0x7ff9c80b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.152+0000 7ff9e1a8e640 1 -- 172.21.15.90:0/3793359991 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56030f18e390 msgr2=0x56030f52ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.152+0000 7ff9e1a8e640 1 --2- 172.21.15.90:0/3793359991 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56030f18e390 0x56030f52ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.152+0000 7ff9e1a8e640 1 -- 172.21.15.90:0/3793359991 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56030f16d6b0 msgr2=0x56030f53b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.152+0000 7ff9e1a8e640 1 --2- 172.21.15.90:0/3793359991 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56030f16d6b0 0x56030f53b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.152+0000 7ff9e1a8e640 1 -- 172.21.15.90:0/3793359991 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56030f41fa40 con 0x56030f1654d0 2024-04-25T11:46:26.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.152+0000 7ff9d2ffd640 1 -- 172.21.15.90:0/1244171375 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7ff9cc0ff620 con 0x7ff9dc108a20 2024-04-25T11:46:26.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.152+0000 7ff9e1a8e640 1 --2- 172.21.15.90:0/3793359991 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f1654d0 0x56030f52efa0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8009c90 tx=0x7ff9d8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eef25b6596d72313 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:26.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.152+0000 7ff9e228f640 1 --2- 172.21.15.90:0/1244171375 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff9c80afc60 0x7ff9c80b2170 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x56030f52bbd0 tx=0x7ff9d4004ab0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:26.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.152+0000 7ff9c7fff640 1 -- 172.21.15.90:0/3793359991 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d801e070 con 0x56030f1654d0 2024-04-25T11:46:26.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.152+0000 7ff9c7fff640 1 -- 172.21.15.90:0/3793359991 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9d8019ae0 con 0x56030f1654d0 2024-04-25T11:46:26.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.152+0000 7ff9c7fff640 1 -- 172.21.15.90:0/3793359991 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d800ea40 con 0x56030f1654d0 2024-04-25T11:46:26.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.153+0000 7ff9eaba7740 1 -- 172.21.15.90:0/3793359991 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f1654d0 msgr2=0x56030f52efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.153+0000 7ff9eaba7740 1 --2- 172.21.15.90:0/3793359991 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f1654d0 0x56030f52efa0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8009c90 tx=0x7ff9d8011810 comp rx=0 tx=0).stop 2024-04-25T11:46:26.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.153+0000 7ff9eaba7740 1 -- 172.21.15.90:0/3793359991 shutdown_connections 2024-04-25T11:46:26.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.153+0000 7ff9eaba7740 1 --2- 172.21.15.90:0/3793359991 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56030f16d6b0 0x56030f53b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.153+0000 7ff9eaba7740 1 --2- 172.21.15.90:0/3793359991 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f1654d0 0x56030f52efa0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.153+0000 7ff9eaba7740 1 --2- 172.21.15.90:0/3793359991 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56030f18e390 0x56030f52ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.153+0000 7ff9eaba7740 1 -- 172.21.15.90:0/3793359991 >> 172.21.15.90:0/3793359991 conn(0x56030f1824e0 msgr2=0x56030f52b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:26.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.153+0000 7ff9eaba7740 1 -- 172.21.15.90:0/3793359991 shutdown_connections 2024-04-25T11:46:26.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.153+0000 7ff9eaba7740 1 -- 172.21.15.90:0/3793359991 wait complete. 2024-04-25T11:46:26.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.154+0000 7ff9eaba7740 1 -- start start 2024-04-25T11:46:26.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.154+0000 7ff9eaba7740 1 Processor -- start 2024-04-25T11:46:26.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.154+0000 7ff9eaba7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56030f1654d0 0x56030f539640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.154+0000 7ff9eaba7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f16d6b0 0x56030f539b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.154+0000 7ff9eaba7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56030f18e390 0x56030f53f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.155+0000 7ff9eaba7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56030f53bfe0 con 0x56030f16d6b0 2024-04-25T11:46:26.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.155+0000 7ff9eaba7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56030f41bd10 con 0x56030f1654d0 2024-04-25T11:46:26.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.155+0000 7ff9eaba7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56030f53c160 con 0x56030f18e390 2024-04-25T11:46:26.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.155+0000 7ff9eaba7740 10 client.0 _ll_register_callbacks cb 0x56030f448210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:26.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.155+0000 7ff9e128d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f16d6b0 0x56030f539b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.155+0000 7ff9e128d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f16d6b0 0x56030f539b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48110/0 (socket says 172.21.15.90:48110) 2024-04-25T11:46:26.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.155+0000 7ff9e1a8e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56030f1654d0 0x56030f539640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.155+0000 7ff9e128d640 1 -- 172.21.15.90:0/116998115 learned_addr learned my addr 172.21.15.90:0/116998115 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:26.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.155+0000 7ff9e228f640 1 --2- 172.21.15.90:0/116998115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56030f18e390 0x56030f53f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.155+0000 7ff9e128d640 1 -- 172.21.15.90:0/116998115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56030f18e390 msgr2=0x56030f53f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.155+0000 7ff9e128d640 1 --2- 172.21.15.90:0/116998115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56030f18e390 0x56030f53f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.155+0000 7ff9e128d640 1 -- 172.21.15.90:0/116998115 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56030f1654d0 msgr2=0x56030f539640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.155+0000 7ff9e128d640 1 --2- 172.21.15.90:0/116998115 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56030f1654d0 0x56030f539640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.155+0000 7ff9e128d640 1 -- 172.21.15.90:0/116998115 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56030f2707d0 con 0x56030f16d6b0 2024-04-25T11:46:26.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.155+0000 7ff9e1a8e640 1 --2- 172.21.15.90:0/116998115 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56030f1654d0 0x56030f539640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:46:26.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.155+0000 7ff9e128d640 1 --2- 172.21.15.90:0/116998115 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f16d6b0 0x56030f539b80 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc0c3730 tx=0x7ff9cc03b5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86081e1d65c3e068 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:26.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.155+0000 7ff9c57fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:46:26.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.155+0000 7ff9c57fa640 1 -- 172.21.15.90:0/116998115 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc049070 con 0x56030f16d6b0 2024-04-25T11:46:26.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.155+0000 7ff9c57fa640 1 -- 172.21.15.90:0/116998115 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7ff9cc0fcee0 con 0x56030f16d6b0 2024-04-25T11:46:26.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.155+0000 7ff9eaba7740 1 -- 172.21.15.90:0/116998115 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9d8019ae0 con 0x56030f16d6b0 2024-04-25T11:46:26.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.155+0000 7ff9c57fa640 1 -- 172.21.15.90:0/116998115 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff9cc0e1050 con 0x56030f16d6b0 2024-04-25T11:46:26.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.155+0000 7ff9eaba7740 10 client.19200 fetch_fsmap learned FSMap version 28 2024-04-25T11:46:26.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.155+0000 7ff9eaba7740 1 -- 172.21.15.90:0/116998115 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56030f1a63f0 con 0x56030f16d6b0 2024-04-25T11:46:26.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.156+0000 7ff9c57fa640 1 -- 172.21.15.90:0/116998115 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9d8019ae0 con 0x56030f16d6b0 2024-04-25T11:46:26.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.156+0000 7ff9c57fa640 1 -- 172.21.15.90:0/116998115 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7ff9cc0c7910 con 0x56030f16d6b0 2024-04-25T11:46:26.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.156+0000 7ff9eaba7740 10 client.19200 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:46:26.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.156+0000 7ff9eaba7740 10 client.19200 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:46:26.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.156+0000 7ff9eaba7740 20 client.19200 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:26.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.156+0000 7ff9eaba7740 1 --2- 172.21.15.90:0/116998115 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56030f14cf60 0x56030f54b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.156+0000 7ff9eaba7740 4 client.19200 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:46:26.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.156+0000 7ff9eaba7740 1 -- 172.21.15.90:0/116998115 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56030f4a2990 con 0x56030f14cf60 2024-04-25T11:46:26.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.156+0000 7ff9e1a8e640 1 --2- 172.21.15.90:0/116998115 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56030f14cf60 0x56030f54b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.156+0000 7ff9e1a8e640 1 --2- 172.21.15.90:0/116998115 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56030f14cf60 0x56030f54b780 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c04595d52fd0beb server_cookie=45c8b101de9a8c0 in_seq=0 out_seq=0 2024-04-25T11:46:26.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.156+0000 7ff9c57fa640 10 client.19200 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:26.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.157+0000 7ff9c57fa640 1 -- 172.21.15.90:0/116998115 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56030f4a2990 con 0x56030f14cf60 2024-04-25T11:46:26.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.157+0000 7ff9c57fa640 10 client.19200 handle_command_reply: tid=0 2024-04-25T11:46:26.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.157+0000 7ff9eaba7740 1 client.19200 shutdown 2024-04-25T11:46:26.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.157+0000 7ff9eaba7740 1 -- 172.21.15.90:0/116998115 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f16d6b0 msgr2=0x56030f539b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.157+0000 7ff9eaba7740 1 --2- 172.21.15.90:0/116998115 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f16d6b0 0x56030f539b80 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc0c3730 tx=0x7ff9cc03b5d0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.157+0000 7ff9eaba7740 1 -- 172.21.15.90:0/116998115 shutdown_connections 2024-04-25T11:46:26.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.157+0000 7ff9eaba7740 1 --2- 172.21.15.90:0/116998115 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56030f14cf60 0x56030f54b780 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.157+0000 7ff9eaba7740 1 --2- 172.21.15.90:0/116998115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56030f18e390 0x56030f53f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.157+0000 7ff9c57fa640 0 client.19200 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:26.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.157+0000 7ff9eaba7740 1 --2- 172.21.15.90:0/116998115 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f16d6b0 0x56030f539b80 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.157+0000 7ff9eaba7740 1 --2- 172.21.15.90:0/116998115 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56030f1654d0 0x56030f539640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.157+0000 7ff9eaba7740 1 -- 172.21.15.90:0/116998115 >> 172.21.15.90:0/116998115 conn(0x56030f1824e0 msgr2=0x56030f530780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:26.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.158+0000 7ff9eaba7740 1 -- 172.21.15.90:0/116998115 shutdown_connections 2024-04-25T11:46:26.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.158+0000 7ff9eaba7740 1 -- 172.21.15.90:0/116998115 wait complete. 2024-04-25T11:46:26.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.158+0000 7ff9eaba7740 20 client.19200 trim_cache size 0 max 16384 2024-04-25T11:46:26.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.175+0000 7ff9eaba7740 1 Processor -- start 2024-04-25T11:46:26.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.176+0000 7ff9eaba7740 1 -- start start 2024-04-25T11:46:26.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.176+0000 7ff9eaba7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f14cf60 0x56030f533910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.176+0000 7ff9eaba7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56030f1654d0 0x56030f550f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.176+0000 7ff9eaba7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56030f16d6b0 0x56030f551480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.176+0000 7ff9eaba7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc1afda0 con 0x56030f14cf60 2024-04-25T11:46:26.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.176+0000 7ff9eaba7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc1b18e0 con 0x56030f16d6b0 2024-04-25T11:46:26.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.176+0000 7ff9eaba7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc1b0470 con 0x56030f1654d0 2024-04-25T11:46:26.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.176+0000 7ff9e1a8e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f14cf60 0x56030f533910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.176+0000 7ff9e1a8e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f14cf60 0x56030f533910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48124/0 (socket says 172.21.15.90:48124) 2024-04-25T11:46:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.176+0000 7ff9e1a8e640 1 -- 172.21.15.90:0/3870435635 learned_addr learned my addr 172.21.15.90:0/3870435635 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.176+0000 7ff9e228f640 1 --2- 172.21.15.90:0/3870435635 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56030f16d6b0 0x56030f551480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.176+0000 7ff9e128d640 1 --2- 172.21.15.90:0/3870435635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56030f1654d0 0x56030f550f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.176+0000 7ff9e1a8e640 1 -- 172.21.15.90:0/3870435635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56030f1654d0 msgr2=0x56030f550f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.176+0000 7ff9e1a8e640 1 --2- 172.21.15.90:0/3870435635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56030f1654d0 0x56030f550f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.176+0000 7ff9e1a8e640 1 -- 172.21.15.90:0/3870435635 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56030f16d6b0 msgr2=0x56030f551480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.176+0000 7ff9e1a8e640 1 --2- 172.21.15.90:0/3870435635 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56030f16d6b0 0x56030f551480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.176+0000 7ff9e1a8e640 1 -- 172.21.15.90:0/3870435635 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56030f534d10 con 0x56030f14cf60 2024-04-25T11:46:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.176+0000 7ff9e1a8e640 1 --2- 172.21.15.90:0/3870435635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f14cf60 0x56030f533910 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7ff9d801afd0 tx=0x7ff9d800e5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9007468b5b7b64 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.176+0000 7ff9c77fe640 1 -- 172.21.15.90:0/3870435635 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8032070 con 0x56030f14cf60 2024-04-25T11:46:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.177+0000 7ff9c77fe640 1 -- 172.21.15.90:0/3870435635 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9d802c2f0 con 0x56030f14cf60 2024-04-25T11:46:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.177+0000 7ff9c77fe640 1 -- 172.21.15.90:0/3870435635 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d801e040 con 0x56030f14cf60 2024-04-25T11:46:26.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.177+0000 7ff9eaba7740 1 -- 172.21.15.90:0/3870435635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f14cf60 msgr2=0x56030f533910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.177+0000 7ff9eaba7740 1 --2- 172.21.15.90:0/3870435635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f14cf60 0x56030f533910 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7ff9d801afd0 tx=0x7ff9d800e5a0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.177+0000 7ff9eaba7740 1 -- 172.21.15.90:0/3870435635 shutdown_connections 2024-04-25T11:46:26.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.177+0000 7ff9eaba7740 1 --2- 172.21.15.90:0/3870435635 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56030f16d6b0 0x56030f551480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.177+0000 7ff9eaba7740 1 --2- 172.21.15.90:0/3870435635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56030f1654d0 0x56030f550f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.177+0000 7ff9eaba7740 1 --2- 172.21.15.90:0/3870435635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f14cf60 0x56030f533910 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.177+0000 7ff9eaba7740 1 -- 172.21.15.90:0/3870435635 >> 172.21.15.90:0/3870435635 conn(0x56030f1824e0 msgr2=0x56030f537d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:26.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.177+0000 7ff9eaba7740 1 -- 172.21.15.90:0/3870435635 shutdown_connections 2024-04-25T11:46:26.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.177+0000 7ff9eaba7740 1 -- 172.21.15.90:0/3870435635 wait complete. 2024-04-25T11:46:26.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.178+0000 7ff9eaba7740 1 -- start start 2024-04-25T11:46:26.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.178+0000 7ff9eaba7740 1 Processor -- start 2024-04-25T11:46:26.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.178+0000 7ff9eaba7740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56030f14cf60 0x56030f540110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.178+0000 7ff9eaba7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56030f1654d0 0x56030f540670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.178+0000 7ff9eaba7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f16d6b0 0x56030f5b1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.178+0000 7ff9eaba7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc1b1210 con 0x56030f16d6b0 2024-04-25T11:46:26.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.178+0000 7ff9eaba7740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc1b0b40 con 0x56030f14cf60 2024-04-25T11:46:26.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.178+0000 7ff9eaba7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc1adda0 con 0x56030f1654d0 2024-04-25T11:46:26.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.178+0000 7ff9eaba7740 10 client.0 _ll_register_callbacks cb 0x7ff9cc005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:26.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.178+0000 7ff9e228f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f16d6b0 0x56030f5b1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.178+0000 7ff9e1a8e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56030f14cf60 0x56030f540110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.178+0000 7ff9e228f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f16d6b0 0x56030f5b1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48134/0 (socket says 172.21.15.90:48134) 2024-04-25T11:46:26.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.178+0000 7ff9e228f640 1 -- 172.21.15.90:0/1523631035 learned_addr learned my addr 172.21.15.90:0/1523631035 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:26.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.178+0000 7ff9e128d640 1 --2- 172.21.15.90:0/1523631035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56030f1654d0 0x56030f540670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.179+0000 7ff9e1a8e640 1 -- 172.21.15.90:0/1523631035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56030f1654d0 msgr2=0x56030f540670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.179+0000 7ff9e1a8e640 1 --2- 172.21.15.90:0/1523631035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56030f1654d0 0x56030f540670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.179+0000 7ff9e1a8e640 1 -- 172.21.15.90:0/1523631035 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f16d6b0 msgr2=0x56030f5b1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.179+0000 7ff9e1a8e640 1 --2- 172.21.15.90:0/1523631035 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f16d6b0 0x56030f5b1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.179+0000 7ff9e1a8e640 1 -- 172.21.15.90:0/1523631035 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56030f1660f0 con 0x56030f14cf60 2024-04-25T11:46:26.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.179+0000 7ff9e128d640 1 --2- 172.21.15.90:0/1523631035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56030f1654d0 0x56030f540670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:46:26.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.179+0000 7ff9e228f640 1 --2- 172.21.15.90:0/1523631035 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f16d6b0 0x56030f5b1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:46:26.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.179+0000 7ff9e1a8e640 1 --2- 172.21.15.90:0/1523631035 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56030f14cf60 0x56030f540110 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff9d801afd0 tx=0x7ff9d8005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0ea458e6552774b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:26.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.179+0000 7ff9c57fa640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:46:26.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.179+0000 7ff9c57fa640 1 -- 172.21.15.90:0/1523631035 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d801e050 con 0x56030f14cf60 2024-04-25T11:46:26.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.179+0000 7ff9eaba7740 1 -- 172.21.15.90:0/1523631035 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9d802c2f0 con 0x56030f14cf60 2024-04-25T11:46:26.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.179+0000 7ff9c57fa640 1 -- 172.21.15.90:0/1523631035 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7ff9d802c6b0 con 0x56030f14cf60 2024-04-25T11:46:26.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.179+0000 7ff9c57fa640 1 -- 172.21.15.90:0/1523631035 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff9d8032070 con 0x56030f14cf60 2024-04-25T11:46:26.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.179+0000 7ff9eaba7740 10 client.27752 fetch_fsmap learned FSMap version 28 2024-04-25T11:46:26.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.179+0000 7ff9eaba7740 1 -- 172.21.15.90:0/1523631035 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56030f4554b0 con 0x56030f14cf60 2024-04-25T11:46:26.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.179+0000 7ff9c57fa640 1 -- 172.21.15.90:0/1523631035 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9d802c2f0 con 0x56030f14cf60 2024-04-25T11:46:26.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.180+0000 7ff9c57fa640 1 -- 172.21.15.90:0/1523631035 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7ff9d800e810 con 0x56030f14cf60 2024-04-25T11:46:26.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.180+0000 7ff9eaba7740 10 client.27752 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:46:26.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.180+0000 7ff9eaba7740 10 client.27752 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:46:26.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.180+0000 7ff9eaba7740 20 client.27752 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:26.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.180+0000 7ff9eaba7740 1 --2- 172.21.15.90:0/1523631035 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56030f18e390 0x56030f5b5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.180+0000 7ff9eaba7740 4 client.27752 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:46:26.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.180+0000 7ff9eaba7740 1 -- 172.21.15.90:0/1523631035 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56030f49c800 con 0x56030f18e390 2024-04-25T11:46:26.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.180+0000 7ff9e128d640 1 --2- 172.21.15.90:0/1523631035 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56030f18e390 0x56030f5b5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.180+0000 7ff9c57fa640 10 client.27752 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:26.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.180+0000 7ff9e128d640 1 --2- 172.21.15.90:0/1523631035 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56030f18e390 0x56030f5b5230 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fa14abb4ddf4773 server_cookie=bbf82175cedbdc1c in_seq=0 out_seq=0 2024-04-25T11:46:26.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.180+0000 7ff9c57fa640 1 -- 172.21.15.90:0/1523631035 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56030f49c800 con 0x56030f18e390 2024-04-25T11:46:26.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.180+0000 7ff9c57fa640 10 client.27752 handle_command_reply: tid=0 2024-04-25T11:46:26.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.181+0000 7ff9eaba7740 1 client.27752 shutdown 2024-04-25T11:46:26.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.181+0000 7ff9eaba7740 1 -- 172.21.15.90:0/1523631035 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56030f14cf60 msgr2=0x56030f540110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.181+0000 7ff9eaba7740 1 --2- 172.21.15.90:0/1523631035 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56030f14cf60 0x56030f540110 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff9d801afd0 tx=0x7ff9d8005fd0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.181+0000 7ff9eaba7740 1 -- 172.21.15.90:0/1523631035 shutdown_connections 2024-04-25T11:46:26.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.181+0000 7ff9eaba7740 1 --2- 172.21.15.90:0/1523631035 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x56030f18e390 0x56030f5b5230 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.181+0000 7ff9eaba7740 1 --2- 172.21.15.90:0/1523631035 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56030f16d6b0 0x56030f5b1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.181+0000 7ff9eaba7740 1 --2- 172.21.15.90:0/1523631035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56030f1654d0 0x56030f540670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.181+0000 7ff9eaba7740 1 --2- 172.21.15.90:0/1523631035 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56030f14cf60 0x56030f540110 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.181+0000 7ff9c57fa640 0 client.27752 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:26.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.181+0000 7ff9eaba7740 1 -- 172.21.15.90:0/1523631035 >> 172.21.15.90:0/1523631035 conn(0x56030f1824e0 msgr2=0x56030f52e7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:26.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.181+0000 7ff9eaba7740 1 -- 172.21.15.90:0/1523631035 shutdown_connections 2024-04-25T11:46:26.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.181+0000 7ff9eaba7740 1 -- 172.21.15.90:0/1523631035 wait complete. 2024-04-25T11:46:26.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.181+0000 7ff9eaba7740 20 client.27752 trim_cache size 0 max 16384 2024-04-25T11:46:26.173 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:46:26.173 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-25T11:46:26.173 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-25T11:46:26.173 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:46:26.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.181+0000 7ff9e3cf2640 1 -- 172.21.15.90:0/1244171375 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff9c80afc60 msgr2=0x7ff9c80b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.181+0000 7ff9e3cf2640 1 --2- 172.21.15.90:0/1244171375 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff9c80afc60 0x7ff9c80b2170 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x56030f52bbd0 tx=0x7ff9d4004ab0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.181+0000 7ff9e3cf2640 1 -- 172.21.15.90:0/1244171375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9dc108a20 msgr2=0x7ff9dc19bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.181+0000 7ff9e3cf2640 1 --2- 172.21.15.90:0/1244171375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9dc108a20 0x7ff9dc19bac0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc006010 tx=0x7ff9cc004060 comp rx=0 tx=0).stop 2024-04-25T11:46:26.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.181+0000 7ff9e3cf2640 1 -- 172.21.15.90:0/1244171375 shutdown_connections 2024-04-25T11:46:26.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.181+0000 7ff9e3cf2640 1 --2- 172.21.15.90:0/1244171375 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff9c80afc60 0x7ff9c80b2170 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.182+0000 7ff9e3cf2640 1 --2- 172.21.15.90:0/1244171375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9dc10d380 0x7ff9dc19c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.182+0000 7ff9e3cf2640 1 --2- 172.21.15.90:0/1244171375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9dc108a20 0x7ff9dc19bac0 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.182+0000 7ff9e3cf2640 1 --2- 172.21.15.90:0/1244171375 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff9dc102af0 0x7ff9dc19b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.182+0000 7ff9e3cf2640 1 -- 172.21.15.90:0/1244171375 >> 172.21.15.90:0/1244171375 conn(0x7ff9dc078100 msgr2=0x7ff9dc1040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:26.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.182+0000 7ff9e3cf2640 1 -- 172.21.15.90:0/1244171375 shutdown_connections 2024-04-25T11:46:26.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.182+0000 7ff9e3cf2640 1 -- 172.21.15.90:0/1244171375 wait complete. 2024-04-25T11:46:26.179 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T11:46:26.180 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:46:26.274 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:26 smithi090 ceph-mon[29575]: pgmap v352: 97 pgs: 15 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 53 active+clean; 24 GiB data, 76 GiB used, 997 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:46:26.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.304+0000 7f3c481ba640 1 -- 172.21.15.90:0/1328763741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c40107450 msgr2=0x7f3c4010ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.304+0000 7f3c481ba640 1 --2- 172.21.15.90:0/1328763741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c40107450 0x7f3c4010ebf0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f3c38009ca0 tx=0x7f3c3802a8c0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.304+0000 7f3c481ba640 1 -- 172.21.15.90:0/1328763741 shutdown_connections 2024-04-25T11:46:26.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.304+0000 7f3c481ba640 1 --2- 172.21.15.90:0/1328763741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c40107450 0x7f3c4010ebf0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.304+0000 7f3c481ba640 1 --2- 172.21.15.90:0/1328763741 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3c40106990 0x7f3c40106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.304+0000 7f3c481ba640 1 --2- 172.21.15.90:0/1328763741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c40102400 0x7f3c40102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.304+0000 7f3c481ba640 1 -- 172.21.15.90:0/1328763741 >> 172.21.15.90:0/1328763741 conn(0x7f3c400fd860 msgr2=0x7f3c400ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:26.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.305+0000 7f3c481ba640 1 -- 172.21.15.90:0/1328763741 shutdown_connections 2024-04-25T11:46:26.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.305+0000 7f3c481ba640 1 -- 172.21.15.90:0/1328763741 wait complete. 2024-04-25T11:46:26.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.305+0000 7f3c481ba640 1 Processor -- start 2024-04-25T11:46:26.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.305+0000 7f3c481ba640 1 -- start start 2024-04-25T11:46:26.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.305+0000 7f3c481ba640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c40102400 0x7f3c40199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.305+0000 7f3c481ba640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c40106990 0x7f3c40199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.306+0000 7f3c481ba640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3c40107450 0x7f3c4019dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.306+0000 7f3c481ba640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3c40112be0 con 0x7f3c40106990 2024-04-25T11:46:26.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.306+0000 7f3c481ba640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3c40112a60 con 0x7f3c40107450 2024-04-25T11:46:26.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.306+0000 7f3c481ba640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3c40112d60 con 0x7f3c40102400 2024-04-25T11:46:26.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.306+0000 7f3c45755640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c40106990 0x7f3c40199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.306+0000 7f3c45755640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c40106990 0x7f3c40199ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48144/0 (socket says 172.21.15.90:48144) 2024-04-25T11:46:26.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.306+0000 7f3c45755640 1 -- 172.21.15.90:0/4151648309 learned_addr learned my addr 172.21.15.90:0/4151648309 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:26.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.306+0000 7f3c45f56640 1 --2- 172.21.15.90:0/4151648309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c40102400 0x7f3c40199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.306+0000 7f3c46757640 1 --2- 172.21.15.90:0/4151648309 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3c40107450 0x7f3c4019dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.306+0000 7f3c45755640 1 -- 172.21.15.90:0/4151648309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c40102400 msgr2=0x7f3c40199580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.306+0000 7f3c45755640 1 --2- 172.21.15.90:0/4151648309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c40102400 0x7f3c40199580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.306+0000 7f3c45755640 1 -- 172.21.15.90:0/4151648309 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3c40107450 msgr2=0x7f3c4019dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.306+0000 7f3c45755640 1 --2- 172.21.15.90:0/4151648309 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3c40107450 0x7f3c4019dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.306+0000 7f3c45755640 1 -- 172.21.15.90:0/4151648309 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c4019e3e0 con 0x7f3c40106990 2024-04-25T11:46:26.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.306+0000 7f3c45f56640 1 --2- 172.21.15.90:0/4151648309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c40102400 0x7f3c40199580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:46:26.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.306+0000 7f3c46757640 1 --2- 172.21.15.90:0/4151648309 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3c40107450 0x7f3c4019dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:46:26.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.306+0000 7f3c45755640 1 --2- 172.21.15.90:0/4151648309 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c40106990 0x7f3c40199ae0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f3c3000e9a0 tx=0x7f3c3000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:26.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.306+0000 7f3c36ffd640 1 -- 172.21.15.90:0/4151648309 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3000cd80 con 0x7f3c40106990 2024-04-25T11:46:26.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.306+0000 7f3c481ba640 1 -- 172.21.15.90:0/4151648309 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c4019e6d0 con 0x7f3c40106990 2024-04-25T11:46:26.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.306+0000 7f3c36ffd640 1 -- 172.21.15.90:0/4151648309 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c300035f0 con 0x7f3c40106990 2024-04-25T11:46:26.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.306+0000 7f3c36ffd640 1 -- 172.21.15.90:0/4151648309 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c30010690 con 0x7f3c40106990 2024-04-25T11:46:26.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.307+0000 7f3c481ba640 1 -- 172.21.15.90:0/4151648309 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c401a6050 con 0x7f3c40106990 2024-04-25T11:46:26.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.308+0000 7f3c481ba640 1 -- 172.21.15.90:0/4151648309 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c4006a190 con 0x7f3c40106990 2024-04-25T11:46:26.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.310+0000 7f3c36ffd640 1 -- 172.21.15.90:0/4151648309 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f3c30003fb0 con 0x7f3c40106990 2024-04-25T11:46:26.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.310+0000 7f3c36ffd640 1 --2- 172.21.15.90:0/4151648309 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3c2c0afc60 0x7f3c2c0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.310+0000 7f3c45f56640 1 --2- 172.21.15.90:0/4151648309 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3c2c0afc60 0x7f3c2c0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.310+0000 7f3c36ffd640 1 -- 172.21.15.90:0/4151648309 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7f3c300df3b0 con 0x7f3c40106990 2024-04-25T11:46:26.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.311+0000 7f3c45f56640 1 --2- 172.21.15.90:0/4151648309 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3c2c0afc60 0x7f3c2c0b2170 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009fc0 tx=0x7f3c3c005f90 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:26.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:26 smithi138 ceph-mon[32328]: pgmap v352: 97 pgs: 15 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 53 active+clean; 24 GiB data, 76 GiB used, 997 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:46:26.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.313+0000 7f3c36ffd640 1 -- 172.21.15.90:0/4151648309 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3c300a7c30 con 0x7f3c40106990 2024-04-25T11:46:26.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:26 smithi156 ceph-mon[33045]: pgmap v352: 97 pgs: 15 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 53 active+clean; 24 GiB data, 76 GiB used, 997 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:46:26.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.544+0000 7f3c481ba640 1 -- 172.21.15.90:0/4151648309 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c4010a210 con 0x7f3c40106990 2024-04-25T11:46:26.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.545+0000 7f3c36ffd640 1 -- 172.21.15.90:0/4151648309 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11930 (secure 0 0 0) 0x7f3c300a7530 con 0x7f3c40106990 2024-04-25T11:46:26.532 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:46:26.532 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:39:20.469254+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24458,24431,24499,24475,24524]},"id":1}]} 2024-04-25T11:46:26.532 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 28 2024-04-25T11:46:26.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.547+0000 7f3c481ba640 1 -- 172.21.15.90:0/4151648309 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3c2c0afc60 msgr2=0x7f3c2c0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.547+0000 7f3c481ba640 1 --2- 172.21.15.90:0/4151648309 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3c2c0afc60 0x7f3c2c0b2170 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009fc0 tx=0x7f3c3c005f90 comp rx=0 tx=0).stop 2024-04-25T11:46:26.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.547+0000 7f3c481ba640 1 -- 172.21.15.90:0/4151648309 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c40106990 msgr2=0x7f3c40199ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.547+0000 7f3c481ba640 1 --2- 172.21.15.90:0/4151648309 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c40106990 0x7f3c40199ae0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f3c3000e9a0 tx=0x7f3c3000ee60 comp rx=0 tx=0).stop 2024-04-25T11:46:26.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.547+0000 7f3c481ba640 1 -- 172.21.15.90:0/4151648309 shutdown_connections 2024-04-25T11:46:26.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.547+0000 7f3c481ba640 1 --2- 172.21.15.90:0/4151648309 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3c2c0afc60 0x7f3c2c0b2170 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.547+0000 7f3c481ba640 1 --2- 172.21.15.90:0/4151648309 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3c40107450 0x7f3c4019dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.547+0000 7f3c481ba640 1 --2- 172.21.15.90:0/4151648309 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c40106990 0x7f3c40199ae0 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.547+0000 7f3c481ba640 1 --2- 172.21.15.90:0/4151648309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c40102400 0x7f3c40199580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.547+0000 7f3c481ba640 1 -- 172.21.15.90:0/4151648309 >> 172.21.15.90:0/4151648309 conn(0x7f3c400fd860 msgr2=0x7f3c40104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:26.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.547+0000 7f3c481ba640 1 -- 172.21.15.90:0/4151648309 shutdown_connections 2024-04-25T11:46:26.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.547+0000 7f3c481ba640 1 -- 172.21.15.90:0/4151648309 wait complete. 2024-04-25T11:46:26.545 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T11:46:26.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.675+0000 7f068ec70640 1 -- 172.21.15.90:0/892239701 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0688108a20 msgr2=0x7f06881025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.675+0000 7f068ec70640 1 --2- 172.21.15.90:0/892239701 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0688108a20 0x7f06881025b0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f067c009900 tx=0x7f067c02a410 comp rx=0 tx=0).stop 2024-04-25T11:46:26.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.675+0000 7f068ec70640 1 -- 172.21.15.90:0/892239701 shutdown_connections 2024-04-25T11:46:26.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.675+0000 7f068ec70640 1 --2- 172.21.15.90:0/892239701 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f068810d380 0x7f068810f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.675+0000 7f068ec70640 1 --2- 172.21.15.90:0/892239701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0688102af0 0x7f0688102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.675+0000 7f068ec70640 1 --2- 172.21.15.90:0/892239701 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0688108a20 0x7f06881025b0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.675+0000 7f068ec70640 1 -- 172.21.15.90:0/892239701 >> 172.21.15.90:0/892239701 conn(0x7f0688078100 msgr2=0x7f0688078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:26.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.675+0000 7f068ec70640 1 -- 172.21.15.90:0/892239701 shutdown_connections 2024-04-25T11:46:26.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.675+0000 7f068ec70640 1 -- 172.21.15.90:0/892239701 wait complete. 2024-04-25T11:46:26.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.676+0000 7f068ec70640 1 Processor -- start 2024-04-25T11:46:26.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.676+0000 7f068ec70640 1 -- start start 2024-04-25T11:46:26.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.676+0000 7f068ec70640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0688102af0 0x7f068819b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.676+0000 7f068ec70640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0688108a20 0x7f068819baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.676+0000 7f068ec70640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f068810d380 0x7f068819bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.676+0000 7f068ec70640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0688112be0 con 0x7f0688108a20 2024-04-25T11:46:26.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.676+0000 7f068ec70640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0688112a60 con 0x7f068810d380 2024-04-25T11:46:26.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.676+0000 7f068ec70640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0688112d60 con 0x7f0688102af0 2024-04-25T11:46:26.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.676+0000 7f0687fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0688108a20 0x7f068819baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.676+0000 7f068ca0c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0688102af0 0x7f068819b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.676+0000 7f0687fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0688108a20 0x7f068819baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48166/0 (socket says 172.21.15.90:48166) 2024-04-25T11:46:26.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.676+0000 7f0687fff640 1 -- 172.21.15.90:0/1471575623 learned_addr learned my addr 172.21.15.90:0/1471575623 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:26.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.676+0000 7f068d20d640 1 --2- 172.21.15.90:0/1471575623 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f068810d380 0x7f068819bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.677+0000 7f0687fff640 1 -- 172.21.15.90:0/1471575623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0688102af0 msgr2=0x7f068819b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.677+0000 7f0687fff640 1 --2- 172.21.15.90:0/1471575623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0688102af0 0x7f068819b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.677+0000 7f0687fff640 1 -- 172.21.15.90:0/1471575623 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f068810d380 msgr2=0x7f068819bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.677+0000 7f0687fff640 1 --2- 172.21.15.90:0/1471575623 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f068810d380 0x7f068819bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.677+0000 7f0687fff640 1 -- 172.21.15.90:0/1471575623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06881a6870 con 0x7f0688108a20 2024-04-25T11:46:26.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.677+0000 7f0687fff640 1 --2- 172.21.15.90:0/1471575623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0688108a20 0x7f068819baa0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f0674005d00 tx=0x7f067400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:26.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.677+0000 7f0685ffb640 1 -- 172.21.15.90:0/1471575623 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0674017070 con 0x7f0688108a20 2024-04-25T11:46:26.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.677+0000 7f0685ffb640 1 -- 172.21.15.90:0/1471575623 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0674009c30 con 0x7f0688108a20 2024-04-25T11:46:26.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.677+0000 7f068ec70640 1 -- 172.21.15.90:0/1471575623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06881a6b60 con 0x7f0688108a20 2024-04-25T11:46:26.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.677+0000 7f0685ffb640 1 -- 172.21.15.90:0/1471575623 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067400f620 con 0x7f0688108a20 2024-04-25T11:46:26.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.677+0000 7f068ec70640 1 -- 172.21.15.90:0/1471575623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06881a70b0 con 0x7f0688108a20 2024-04-25T11:46:26.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.680+0000 7f0695bf3740 1 Processor -- start 2024-04-25T11:46:26.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.680+0000 7f0695bf3740 1 -- start start 2024-04-25T11:46:26.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.680+0000 7f0685ffb640 1 -- 172.21.15.90:0/1471575623 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f067401a070 con 0x7f0688108a20 2024-04-25T11:46:26.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.680+0000 7f0695bf3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc064390 0x55b6fc405810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.680+0000 7f0695bf3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b6fc03b4d0 0x55b6fc405d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.680+0000 7f0695bf3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b6fc0436b0 0x55b6fc4126a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.680+0000 7f0695bf3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b6fc36ad80 con 0x55b6fc064390 2024-04-25T11:46:26.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.680+0000 7f0695bf3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55b6fc324a20 con 0x55b6fc03b4d0 2024-04-25T11:46:26.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.680+0000 7f0695bf3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b6fc366d40 con 0x55b6fc0436b0 2024-04-25T11:46:26.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.680+0000 7f068d20d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc064390 0x55b6fc405810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.680+0000 7f068d20d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc064390 0x55b6fc405810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48172/0 (socket says 172.21.15.90:48172) 2024-04-25T11:46:26.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.680+0000 7f068d20d640 1 -- 172.21.15.90:0/1793620278 learned_addr learned my addr 172.21.15.90:0/1793620278 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:26.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.680+0000 7f0687fff640 1 --2- 172.21.15.90:0/1793620278 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b6fc0436b0 0x55b6fc4126a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.680+0000 7f068ca0c640 1 --2- 172.21.15.90:0/1793620278 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b6fc03b4d0 0x55b6fc405d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.680+0000 7f068d20d640 1 -- 172.21.15.90:0/1793620278 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b6fc0436b0 msgr2=0x55b6fc4126a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.680+0000 7f068d20d640 1 --2- 172.21.15.90:0/1793620278 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b6fc0436b0 0x55b6fc4126a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.680+0000 7f068d20d640 1 -- 172.21.15.90:0/1793620278 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b6fc03b4d0 msgr2=0x55b6fc405d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.680+0000 7f068d20d640 1 --2- 172.21.15.90:0/1793620278 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b6fc03b4d0 0x55b6fc405d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.680+0000 7f068d20d640 1 -- 172.21.15.90:0/1793620278 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6fc32fc30 con 0x55b6fc064390 2024-04-25T11:46:26.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.681+0000 7f0685ffb640 1 --2- 172.21.15.90:0/1471575623 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f06780afce0 0x7f06780b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.681+0000 7f068d20d640 1 --2- 172.21.15.90:0/1793620278 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc064390 0x55b6fc405810 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f0680009f40 tx=0x7f0680010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b807e41c3c054fb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:26.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.681+0000 7f068ca0c640 1 --2- 172.21.15.90:0/1471575623 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f06780afce0 0x7f06780b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.681+0000 7f0685ffb640 1 -- 172.21.15.90:0/1471575623 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7f06740dea20 con 0x7f0688108a20 2024-04-25T11:46:26.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.681+0000 7f0672ffd640 1 -- 172.21.15.90:0/1793620278 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0680016070 con 0x55b6fc064390 2024-04-25T11:46:26.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.681+0000 7f0672ffd640 1 -- 172.21.15.90:0/1793620278 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0680004d20 con 0x55b6fc064390 2024-04-25T11:46:26.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.681+0000 7f0672ffd640 1 -- 172.21.15.90:0/1793620278 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06800126d0 con 0x55b6fc064390 2024-04-25T11:46:26.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.682+0000 7f0695bf3740 1 -- 172.21.15.90:0/1793620278 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc064390 msgr2=0x55b6fc405810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.682+0000 7f0695bf3740 1 --2- 172.21.15.90:0/1793620278 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc064390 0x55b6fc405810 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f0680009f40 tx=0x7f0680010800 comp rx=0 tx=0).stop 2024-04-25T11:46:26.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.682+0000 7f068ca0c640 1 --2- 172.21.15.90:0/1471575623 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f06780afce0 0x7f06780b21f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f067c000c00 tx=0x7f067c031400 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:26.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.682+0000 7f0695bf3740 1 -- 172.21.15.90:0/1793620278 shutdown_connections 2024-04-25T11:46:26.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.682+0000 7f0695bf3740 1 --2- 172.21.15.90:0/1793620278 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b6fc0436b0 0x55b6fc4126a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.682+0000 7f0695bf3740 1 --2- 172.21.15.90:0/1793620278 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b6fc03b4d0 0x55b6fc405d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.682+0000 7f0695bf3740 1 --2- 172.21.15.90:0/1793620278 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc064390 0x55b6fc405810 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.682+0000 7f0695bf3740 1 -- 172.21.15.90:0/1793620278 >> 172.21.15.90:0/1793620278 conn(0x55b6fc0584e0 msgr2=0x55b6fc401fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:26.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.682+0000 7f0695bf3740 1 -- 172.21.15.90:0/1793620278 shutdown_connections 2024-04-25T11:46:26.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.682+0000 7f0695bf3740 1 -- 172.21.15.90:0/1793620278 wait complete. 2024-04-25T11:46:26.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.682+0000 7f0695bf3740 1 -- start start 2024-04-25T11:46:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.683+0000 7f0695bf3740 1 Processor -- start 2024-04-25T11:46:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.683+0000 7f0695bf3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc03b4d0 0x55b6fc410270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.683+0000 7f0695bf3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b6fc0436b0 0x55b6fc4107b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.683+0000 7f0695bf3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b6fc064390 0x55b6fc41c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.683+0000 7f0695bf3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b6fc412d90 con 0x55b6fc03b4d0 2024-04-25T11:46:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.683+0000 7f0695bf3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55b6fc2f1dc0 con 0x55b6fc064390 2024-04-25T11:46:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.683+0000 7f0695bf3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b6fc412f10 con 0x55b6fc0436b0 2024-04-25T11:46:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.683+0000 7f0695bf3740 10 client.0 _ll_register_callbacks cb 0x55b6fc30d600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.683+0000 7f068ca0c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc03b4d0 0x55b6fc410270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.683+0000 7f068ca0c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc03b4d0 0x55b6fc410270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48180/0 (socket says 172.21.15.90:48180) 2024-04-25T11:46:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.683+0000 7f068ca0c640 1 -- 172.21.15.90:0/3264396638 learned_addr learned my addr 172.21.15.90:0/3264396638 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.683+0000 7f068d20d640 1 --2- 172.21.15.90:0/3264396638 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b6fc064390 0x55b6fc41c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.683+0000 7f0687fff640 1 --2- 172.21.15.90:0/3264396638 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b6fc0436b0 0x55b6fc4107b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.683+0000 7f068d20d640 1 -- 172.21.15.90:0/3264396638 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b6fc0436b0 msgr2=0x55b6fc4107b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.683+0000 7f068d20d640 1 --2- 172.21.15.90:0/3264396638 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b6fc0436b0 0x55b6fc4107b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.683+0000 7f068d20d640 1 -- 172.21.15.90:0/3264396638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc03b4d0 msgr2=0x55b6fc410270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.684+0000 7f068d20d640 1 --2- 172.21.15.90:0/3264396638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc03b4d0 0x55b6fc410270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.684+0000 7f068d20d640 1 -- 172.21.15.90:0/3264396638 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6fc07c3f0 con 0x55b6fc064390 2024-04-25T11:46:26.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.684+0000 7f068ca0c640 1 --2- 172.21.15.90:0/3264396638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc03b4d0 0x55b6fc410270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:46:26.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.684+0000 7f068d20d640 1 --2- 172.21.15.90:0/3264396638 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b6fc064390 0x55b6fc41c210 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0680009f10 tx=0x7f0680009c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7c7b369908d0bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:26.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.684+0000 7f0667fff640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:46:26.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.684+0000 7f0667fff640 1 -- 172.21.15.90:0/3264396638 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0680029060 con 0x55b6fc064390 2024-04-25T11:46:26.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.684+0000 7f0695bf3740 1 -- 172.21.15.90:0/3264396638 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0680004d20 con 0x55b6fc064390 2024-04-25T11:46:26.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.684+0000 7f0667fff640 1 -- 172.21.15.90:0/3264396638 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7f0680005290 con 0x55b6fc064390 2024-04-25T11:46:26.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.684+0000 7f0667fff640 1 -- 172.21.15.90:0/3264396638 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0680030050 con 0x55b6fc064390 2024-04-25T11:46:26.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.684+0000 7f0695bf3740 10 client.27770 fetch_fsmap learned FSMap version 28 2024-04-25T11:46:26.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.684+0000 7f0695bf3740 1 -- 172.21.15.90:0/3264396638 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b6fc1467d0 con 0x55b6fc064390 2024-04-25T11:46:26.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.684+0000 7f0667fff640 1 -- 172.21.15.90:0/3264396638 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0680004d20 con 0x55b6fc064390 2024-04-25T11:46:26.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.684+0000 7f0667fff640 1 -- 172.21.15.90:0/3264396638 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f068000f040 con 0x55b6fc064390 2024-04-25T11:46:26.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.684+0000 7f0695bf3740 10 client.27770 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:46:26.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.684+0000 7f0695bf3740 10 client.27770 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:46:26.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.684+0000 7f0695bf3740 20 client.27770 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:26.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.685+0000 7f0695bf3740 1 --2- 172.21.15.90:0/3264396638 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55b6fc022f60 0x55b6fc422a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.685+0000 7f0695bf3740 4 client.27770 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:46:26.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.685+0000 7f0695bf3740 1 -- 172.21.15.90:0/3264396638 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6fc378990 con 0x55b6fc022f60 2024-04-25T11:46:26.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.685+0000 7f068ca0c640 1 --2- 172.21.15.90:0/3264396638 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55b6fc022f60 0x55b6fc422a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.685+0000 7f068ca0c640 1 --2- 172.21.15.90:0/3264396638 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55b6fc022f60 0x55b6fc422a50 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4b7d0d7cd00def84 server_cookie=363c2953f68aa936 in_seq=0 out_seq=0 2024-04-25T11:46:26.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.685+0000 7f0667fff640 10 client.27770 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:46:26.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.686+0000 7f0667fff640 1 -- 172.21.15.90:0/3264396638 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b6fc378990 con 0x55b6fc022f60 2024-04-25T11:46:26.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.686+0000 7f0667fff640 10 client.27770 handle_command_reply: tid=0 2024-04-25T11:46:26.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.686+0000 7f0695bf3740 1 client.27770 shutdown 2024-04-25T11:46:26.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.686+0000 7f0695bf3740 1 -- 172.21.15.90:0/3264396638 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b6fc064390 msgr2=0x55b6fc41c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.686+0000 7f0695bf3740 1 --2- 172.21.15.90:0/3264396638 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b6fc064390 0x55b6fc41c210 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0680009f10 tx=0x7f0680009c90 comp rx=0 tx=0).stop 2024-04-25T11:46:26.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.686+0000 7f0695bf3740 1 -- 172.21.15.90:0/3264396638 shutdown_connections 2024-04-25T11:46:26.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.686+0000 7f0695bf3740 1 --2- 172.21.15.90:0/3264396638 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55b6fc022f60 0x55b6fc422a50 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.686+0000 7f0695bf3740 1 --2- 172.21.15.90:0/3264396638 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b6fc064390 0x55b6fc41c210 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.686+0000 7f0695bf3740 1 --2- 172.21.15.90:0/3264396638 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b6fc0436b0 0x55b6fc4107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.686+0000 7f0667fff640 0 client.27770 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:46:26.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.686+0000 7f0695bf3740 1 --2- 172.21.15.90:0/3264396638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc03b4d0 0x55b6fc410270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.686+0000 7f0695bf3740 1 -- 172.21.15.90:0/3264396638 >> 172.21.15.90:0/3264396638 conn(0x55b6fc0584e0 msgr2=0x55b6fc407530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:26.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.686+0000 7f0695bf3740 1 -- 172.21.15.90:0/3264396638 shutdown_connections 2024-04-25T11:46:26.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.686+0000 7f0695bf3740 1 -- 172.21.15.90:0/3264396638 wait complete. 2024-04-25T11:46:26.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.686+0000 7f0695bf3740 20 client.27770 trim_cache size 0 max 16384 2024-04-25T11:46:26.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.700+0000 7f0695bf3740 1 Processor -- start 2024-04-25T11:46:26.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.700+0000 7f0695bf3740 1 -- start start 2024-04-25T11:46:26.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.700+0000 7f0695bf3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc022f60 0x55b6fc41be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.700+0000 7f0695bf3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b6fc03b4d0 0x55b6fc428210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.700+0000 7f0695bf3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b6fc0436b0 0x55b6fc428750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.700+0000 7f0695bf3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f06881afc50 con 0x55b6fc022f60 2024-04-25T11:46:26.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.700+0000 7f0695bf3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f06881b1790 con 0x55b6fc0436b0 2024-04-25T11:46:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.700+0000 7f0695bf3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f06881b0320 con 0x55b6fc03b4d0 2024-04-25T11:46:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.700+0000 7f068ca0c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc022f60 0x55b6fc41be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.700+0000 7f068ca0c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc022f60 0x55b6fc41be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48190/0 (socket says 172.21.15.90:48190) 2024-04-25T11:46:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.700+0000 7f068ca0c640 1 -- 172.21.15.90:0/727327149 learned_addr learned my addr 172.21.15.90:0/727327149 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.700+0000 7f0687fff640 1 --2- 172.21.15.90:0/727327149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b6fc03b4d0 0x55b6fc428210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.700+0000 7f068d20d640 1 --2- 172.21.15.90:0/727327149 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b6fc0436b0 0x55b6fc428750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.701+0000 7f068ca0c640 1 -- 172.21.15.90:0/727327149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b6fc03b4d0 msgr2=0x55b6fc428210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.701+0000 7f068ca0c640 1 --2- 172.21.15.90:0/727327149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b6fc03b4d0 0x55b6fc428210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.701+0000 7f068ca0c640 1 -- 172.21.15.90:0/727327149 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b6fc0436b0 msgr2=0x55b6fc428750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.701+0000 7f068ca0c640 1 --2- 172.21.15.90:0/727327149 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b6fc0436b0 0x55b6fc428750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.701+0000 7f068ca0c640 1 -- 172.21.15.90:0/727327149 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6fc422f80 con 0x55b6fc022f60 2024-04-25T11:46:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.701+0000 7f0687fff640 1 --2- 172.21.15.90:0/727327149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b6fc03b4d0 0x55b6fc428210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:46:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.701+0000 7f068ca0c640 1 --2- 172.21.15.90:0/727327149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc022f60 0x55b6fc41be70 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f067c006010 tx=0x7f067c03f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eaea9a7df4fbc57 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.701+0000 7f06727fc640 1 -- 172.21.15.90:0/727327149 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067c057070 con 0x55b6fc022f60 2024-04-25T11:46:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.701+0000 7f06727fc640 1 -- 172.21.15.90:0/727327149 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f067c0054d0 con 0x55b6fc022f60 2024-04-25T11:46:26.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.701+0000 7f06727fc640 1 -- 172.21.15.90:0/727327149 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067c035040 con 0x55b6fc022f60 2024-04-25T11:46:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.702+0000 7f0695bf3740 1 -- 172.21.15.90:0/727327149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc022f60 msgr2=0x55b6fc41be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.702+0000 7f0695bf3740 1 --2- 172.21.15.90:0/727327149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc022f60 0x55b6fc41be70 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f067c006010 tx=0x7f067c03f000 comp rx=0 tx=0).stop 2024-04-25T11:46:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.702+0000 7f0695bf3740 1 -- 172.21.15.90:0/727327149 shutdown_connections 2024-04-25T11:46:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.702+0000 7f0695bf3740 1 --2- 172.21.15.90:0/727327149 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b6fc0436b0 0x55b6fc428750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.702+0000 7f0695bf3740 1 --2- 172.21.15.90:0/727327149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b6fc03b4d0 0x55b6fc428210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.702+0000 7f0695bf3740 1 --2- 172.21.15.90:0/727327149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc022f60 0x55b6fc41be70 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.702+0000 7f0695bf3740 1 -- 172.21.15.90:0/727327149 >> 172.21.15.90:0/727327149 conn(0x55b6fc0584e0 msgr2=0x55b6fc40e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.702+0000 7f0695bf3740 1 -- 172.21.15.90:0/727327149 shutdown_connections 2024-04-25T11:46:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.702+0000 7f0695bf3740 1 -- 172.21.15.90:0/727327149 wait complete. 2024-04-25T11:46:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.702+0000 7f0695bf3740 1 -- start start 2024-04-25T11:46:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.702+0000 7f0695bf3740 1 Processor -- start 2024-04-25T11:46:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.703+0000 7f0695bf3740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b6fc022f60 0x55b6fc418760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.703+0000 7f0695bf3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc03b4d0 0x55b6fc4131b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.703+0000 7f0695bf3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b6fc0436b0 0x55b6fc4136f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.703+0000 7f0695bf3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f06881b10c0 con 0x55b6fc03b4d0 2024-04-25T11:46:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.703+0000 7f0695bf3740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f06881b09f0 con 0x55b6fc022f60 2024-04-25T11:46:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.703+0000 7f0695bf3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f06881adc50 con 0x55b6fc0436b0 2024-04-25T11:46:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.703+0000 7f0695bf3740 10 client.0 _ll_register_callbacks cb 0x7f067c009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.703+0000 7f0687fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc03b4d0 0x55b6fc4131b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.703+0000 7f0687fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc03b4d0 0x55b6fc4131b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48196/0 (socket says 172.21.15.90:48196) 2024-04-25T11:46:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.703+0000 7f0687fff640 1 -- 172.21.15.90:0/440849874 learned_addr learned my addr 172.21.15.90:0/440849874 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.703+0000 7f068ca0c640 1 --2- 172.21.15.90:0/440849874 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b6fc022f60 0x55b6fc418760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.703+0000 7f068d20d640 1 --2- 172.21.15.90:0/440849874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b6fc0436b0 0x55b6fc4136f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.703+0000 7f0687fff640 1 -- 172.21.15.90:0/440849874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b6fc0436b0 msgr2=0x55b6fc4136f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.703+0000 7f0687fff640 1 --2- 172.21.15.90:0/440849874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b6fc0436b0 0x55b6fc4136f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.703+0000 7f0687fff640 1 -- 172.21.15.90:0/440849874 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b6fc022f60 msgr2=0x55b6fc418760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.703+0000 7f0687fff640 1 --2- 172.21.15.90:0/440849874 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b6fc022f60 0x55b6fc418760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.703+0000 7f0687fff640 1 -- 172.21.15.90:0/440849874 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6fc413e30 con 0x55b6fc03b4d0 2024-04-25T11:46:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.703+0000 7f068d20d640 1 --2- 172.21.15.90:0/440849874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b6fc0436b0 0x55b6fc4136f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:46:26.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.703+0000 7f068ca0c640 1 --2- 172.21.15.90:0/440849874 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b6fc022f60 0x55b6fc418760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:46:26.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.703+0000 7f0687fff640 1 --2- 172.21.15.90:0/440849874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc03b4d0 0x55b6fc4131b0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f06740a2c40 tx=0x7f06740a23c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12e95da762621df8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:26.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.703+0000 7f0667fff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:46:26.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.704+0000 7f0667fff640 1 -- 172.21.15.90:0/440849874 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0674021040 con 0x55b6fc03b4d0 2024-04-25T11:46:26.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.704+0000 7f0667fff640 1 -- 172.21.15.90:0/440849874 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7f0674010670 con 0x55b6fc03b4d0 2024-04-25T11:46:26.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.704+0000 7f0695bf3740 1 -- 172.21.15.90:0/440849874 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f067c0054d0 con 0x55b6fc03b4d0 2024-04-25T11:46:26.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.704+0000 7f0667fff640 1 -- 172.21.15.90:0/440849874 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f06740a6510 con 0x55b6fc03b4d0 2024-04-25T11:46:26.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.704+0000 7f0695bf3740 10 client.19230 fetch_fsmap learned FSMap version 28 2024-04-25T11:46:26.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.704+0000 7f0695bf3740 1 -- 172.21.15.90:0/440849874 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b6fc489c60 con 0x55b6fc03b4d0 2024-04-25T11:46:26.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.704+0000 7f0667fff640 1 -- 172.21.15.90:0/440849874 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f067c0054d0 con 0x55b6fc03b4d0 2024-04-25T11:46:26.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.704+0000 7f0667fff640 1 -- 172.21.15.90:0/440849874 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f06740a2530 con 0x55b6fc03b4d0 2024-04-25T11:46:26.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.704+0000 7f0695bf3740 10 client.19230 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:46:26.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.704+0000 7f0695bf3740 10 client.19230 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:46:26.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.704+0000 7f0695bf3740 20 client.19230 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:26.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.704+0000 7f0695bf3740 1 --2- 172.21.15.90:0/440849874 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55b6fc064390 0x55b6fc48c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.704+0000 7f0695bf3740 4 client.19230 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:46:26.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.704+0000 7f0695bf3740 1 -- 172.21.15.90:0/440849874 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b6fc372800 con 0x55b6fc064390 2024-04-25T11:46:26.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.704+0000 7f068ca0c640 1 --2- 172.21.15.90:0/440849874 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55b6fc064390 0x55b6fc48c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.705+0000 7f0667fff640 10 client.19230 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:46:26.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.705+0000 7f068ca0c640 1 --2- 172.21.15.90:0/440849874 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55b6fc064390 0x55b6fc48c4a0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6d2ae0ac7f5eef96 server_cookie=7982174931871ca1 in_seq=0 out_seq=0 2024-04-25T11:46:26.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.705+0000 7f0667fff640 1 -- 172.21.15.90:0/440849874 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b6fc372800 con 0x55b6fc064390 2024-04-25T11:46:26.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.705+0000 7f0667fff640 10 client.19230 handle_command_reply: tid=0 2024-04-25T11:46:26.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.705+0000 7f0695bf3740 1 client.19230 shutdown 2024-04-25T11:46:26.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.705+0000 7f0695bf3740 1 -- 172.21.15.90:0/440849874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc03b4d0 msgr2=0x55b6fc4131b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.705+0000 7f0695bf3740 1 --2- 172.21.15.90:0/440849874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc03b4d0 0x55b6fc4131b0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f06740a2c40 tx=0x7f06740a23c0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.705+0000 7f0695bf3740 1 -- 172.21.15.90:0/440849874 shutdown_connections 2024-04-25T11:46:26.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.705+0000 7f0695bf3740 1 --2- 172.21.15.90:0/440849874 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x55b6fc064390 0x55b6fc48c4a0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.705+0000 7f0695bf3740 1 --2- 172.21.15.90:0/440849874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b6fc0436b0 0x55b6fc4136f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.705+0000 7f0667fff640 0 client.19230 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:46:26.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.705+0000 7f0695bf3740 1 --2- 172.21.15.90:0/440849874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6fc03b4d0 0x55b6fc4131b0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.705+0000 7f0695bf3740 1 --2- 172.21.15.90:0/440849874 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b6fc022f60 0x55b6fc418760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.705+0000 7f0695bf3740 1 -- 172.21.15.90:0/440849874 >> 172.21.15.90:0/440849874 conn(0x55b6fc0584e0 msgr2=0x55b6fc4055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:26.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.705+0000 7f0695bf3740 1 -- 172.21.15.90:0/440849874 shutdown_connections 2024-04-25T11:46:26.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.705+0000 7f0695bf3740 1 -- 172.21.15.90:0/440849874 wait complete. 2024-04-25T11:46:26.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.705+0000 7f0695bf3740 20 client.19230 trim_cache size 0 max 16384 2024-04-25T11:46:26.697 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:46:26.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.706+0000 7f068ec70640 1 -- 172.21.15.90:0/1471575623 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f06780afce0 msgr2=0x7f06780b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.706+0000 7f068ec70640 1 --2- 172.21.15.90:0/1471575623 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f06780afce0 0x7f06780b21f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f067c000c00 tx=0x7f067c031400 comp rx=0 tx=0).stop 2024-04-25T11:46:26.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.706+0000 7f068ec70640 1 -- 172.21.15.90:0/1471575623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0688108a20 msgr2=0x7f068819baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.706+0000 7f068ec70640 1 --2- 172.21.15.90:0/1471575623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0688108a20 0x7f068819baa0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f0674005d00 tx=0x7f067400dde0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.706+0000 7f068ec70640 1 -- 172.21.15.90:0/1471575623 shutdown_connections 2024-04-25T11:46:26.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.706+0000 7f068ec70640 1 --2- 172.21.15.90:0/1471575623 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f06780afce0 0x7f06780b21f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.706+0000 7f068ec70640 1 --2- 172.21.15.90:0/1471575623 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f068810d380 0x7f068819bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.706+0000 7f068ec70640 1 --2- 172.21.15.90:0/1471575623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0688108a20 0x7f068819baa0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.706+0000 7f068ec70640 1 --2- 172.21.15.90:0/1471575623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0688102af0 0x7f068819b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.706+0000 7f068ec70640 1 -- 172.21.15.90:0/1471575623 >> 172.21.15.90:0/1471575623 conn(0x7f0688078100 msgr2=0x7f06881040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:26.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.706+0000 7f068ec70640 1 -- 172.21.15.90:0/1471575623 shutdown_connections 2024-04-25T11:46:26.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.706+0000 7f068ec70640 1 -- 172.21.15.90:0/1471575623 wait complete. 2024-04-25T11:46:26.703 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T11:46:26.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.833+0000 7f1cb98e9640 1 -- 172.21.15.90:0/3081350364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cb4108a10 msgr2=0x7f1cb4102c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.833+0000 7f1cb98e9640 1 --2- 172.21.15.90:0/3081350364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cb4108a10 0x7f1cb4102c80 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f1ca4009900 tx=0x7f1ca402a410 comp rx=0 tx=0).stop 2024-04-25T11:46:26.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.833+0000 7f1cb98e9640 1 -- 172.21.15.90:0/3081350364 shutdown_connections 2024-04-25T11:46:26.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.833+0000 7f1cb98e9640 1 --2- 172.21.15.90:0/3081350364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1cb4103830 0x7f1cb410f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.833+0000 7f1cb98e9640 1 --2- 172.21.15.90:0/3081350364 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1cb4108eb0 0x7f1cb41031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.833+0000 7f1cb98e9640 1 --2- 172.21.15.90:0/3081350364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cb4108a10 0x7f1cb4102c80 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.833+0000 7f1cb98e9640 1 -- 172.21.15.90:0/3081350364 >> 172.21.15.90:0/3081350364 conn(0x7f1cb4077880 msgr2=0x7f1cb4077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:26.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.833+0000 7f1cb98e9640 1 -- 172.21.15.90:0/3081350364 shutdown_connections 2024-04-25T11:46:26.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.833+0000 7f1cb98e9640 1 -- 172.21.15.90:0/3081350364 wait complete. 2024-04-25T11:46:26.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.834+0000 7f1cb98e9640 1 Processor -- start 2024-04-25T11:46:26.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.834+0000 7f1cb98e9640 1 -- start start 2024-04-25T11:46:26.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.834+0000 7f1cb98e9640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1cb4103830 0x7f1cb419b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.834+0000 7f1cb98e9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1cb4108a10 0x7f1cb419bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.834+0000 7f1cb98e9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cb4108eb0 0x7f1cb419c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.834+0000 7f1cb98e9640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb4112c00 con 0x7f1cb4108eb0 2024-04-25T11:46:26.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.834+0000 7f1cb98e9640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb4112a80 con 0x7f1cb4103830 2024-04-25T11:46:26.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.834+0000 7f1cb98e9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb4112d80 con 0x7f1cb4108a10 2024-04-25T11:46:26.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.834+0000 7f1cb37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cb4108eb0 0x7f1cb419c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.834+0000 7f1cb37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cb4108eb0 0x7f1cb419c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48218/0 (socket says 172.21.15.90:48218) 2024-04-25T11:46:26.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.834+0000 7f1cb37fe640 1 -- 172.21.15.90:0/2736633708 learned_addr learned my addr 172.21.15.90:0/2736633708 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:26.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.834+0000 7f1cb2ffd640 1 --2- 172.21.15.90:0/2736633708 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1cb4103830 0x7f1cb419b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.834+0000 7f1cb27fc640 1 --2- 172.21.15.90:0/2736633708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1cb4108a10 0x7f1cb419bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.834+0000 7f1cb37fe640 1 -- 172.21.15.90:0/2736633708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1cb4108a10 msgr2=0x7f1cb419bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.834+0000 7f1cb37fe640 1 --2- 172.21.15.90:0/2736633708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1cb4108a10 0x7f1cb419bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.834+0000 7f1cb37fe640 1 -- 172.21.15.90:0/2736633708 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1cb4103830 msgr2=0x7f1cb419b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.834+0000 7f1cb37fe640 1 --2- 172.21.15.90:0/2736633708 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1cb4103830 0x7f1cb419b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.834+0000 7f1cb37fe640 1 -- 172.21.15.90:0/2736633708 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cb41a6900 con 0x7f1cb4108eb0 2024-04-25T11:46:26.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.834+0000 7f1cb2ffd640 1 --2- 172.21.15.90:0/2736633708 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1cb4103830 0x7f1cb419b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:46:26.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.835+0000 7f1cb37fe640 1 --2- 172.21.15.90:0/2736633708 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cb4108eb0 0x7f1cb419c060 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f1cac00eb10 tx=0x7f1cac00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:26.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.835+0000 7f1cb88e7640 1 -- 172.21.15.90:0/2736633708 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cac004b20 con 0x7f1cb4108eb0 2024-04-25T11:46:26.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.835+0000 7f1cb98e9640 1 -- 172.21.15.90:0/2736633708 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cb41a6bf0 con 0x7f1cb4108eb0 2024-04-25T11:46:26.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.835+0000 7f1cb88e7640 1 -- 172.21.15.90:0/2736633708 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1cac004cc0 con 0x7f1cb4108eb0 2024-04-25T11:46:26.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.835+0000 7f1cb88e7640 1 -- 172.21.15.90:0/2736633708 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cac010690 con 0x7f1cb4108eb0 2024-04-25T11:46:26.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.835+0000 7f1cb98e9640 1 -- 172.21.15.90:0/2736633708 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cb41a7140 con 0x7f1cb4108eb0 2024-04-25T11:46:26.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.837+0000 7f1cc0795740 1 Processor -- start 2024-04-25T11:46:26.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.837+0000 7f1cb88e7640 1 -- 172.21.15.90:0/2736633708 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f1cac005020 con 0x7f1cb4108eb0 2024-04-25T11:46:26.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.837+0000 7f1cc0795740 1 -- start start 2024-04-25T11:46:26.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.837+0000 7f1cc0795740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffa839e390 0x55ffa873f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.837+0000 7f1cc0795740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffa83754d0 0x55ffa873fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.838+0000 7f1cc0795740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffa837d6b0 0x55ffa874c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.838+0000 7f1cc0795740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ffa86a4d80 con 0x55ffa837d6b0 2024-04-25T11:46:26.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.838+0000 7f1cc0795740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ffa865ea20 con 0x55ffa839e390 2024-04-25T11:46:26.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.838+0000 7f1cc0795740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ffa86a0d40 con 0x55ffa83754d0 2024-04-25T11:46:26.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.838+0000 7f1cb27fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffa837d6b0 0x55ffa874c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.838+0000 7f1cb27fc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffa839e390 0x55ffa873f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.838+0000 7f1cb27fc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffa839e390 0x55ffa873f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:33924/0 (socket says 172.21.15.90:33924) 2024-04-25T11:46:26.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.838+0000 7f1cb27fc640 1 -- 172.21.15.90:0/235431523 learned_addr learned my addr 172.21.15.90:0/235431523 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:26.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.838+0000 7f1cb88e7640 1 --2- 172.21.15.90:0/2736633708 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1c9c0afc60 0x7f1c9c0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.838+0000 7f1cb2ffd640 1 --2- 172.21.15.90:0/235431523 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffa83754d0 0x55ffa873fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.838+0000 7f1cb37fe640 1 --2- 172.21.15.90:0/2736633708 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1c9c0afc60 0x7f1c9c0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.838+0000 7f1cb27fc640 1 -- 172.21.15.90:0/235431523 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffa83754d0 msgr2=0x55ffa873fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.838+0000 7f1cb27fc640 1 --2- 172.21.15.90:0/235431523 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffa83754d0 0x55ffa873fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.838+0000 7f1cb27fc640 1 -- 172.21.15.90:0/235431523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffa837d6b0 msgr2=0x55ffa874c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.838+0000 7f1cb27fc640 1 --2- 172.21.15.90:0/235431523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffa837d6b0 0x55ffa874c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.838+0000 7f1cb27fc640 1 -- 172.21.15.90:0/235431523 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffa8669c30 con 0x55ffa839e390 2024-04-25T11:46:26.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.838+0000 7f1cb88e7640 1 -- 172.21.15.90:0/2736633708 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7f1cac0df7a0 con 0x7f1cb4108eb0 2024-04-25T11:46:26.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.838+0000 7f1cb27fc640 1 --2- 172.21.15.90:0/235431523 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffa839e390 0x55ffa873f810 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1ca80126a0 tx=0x7f1ca8005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3ddc69b3ac06543 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:26.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.838+0000 7f1cb37fe640 1 --2- 172.21.15.90:0/2736633708 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1c9c0afc60 0x7f1c9c0b2170 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1cac0a4b00 tx=0x7f1cac072f90 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:26.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.838+0000 7f1c99ffb640 1 -- 172.21.15.90:0/235431523 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca8012110 con 0x55ffa839e390 2024-04-25T11:46:26.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.838+0000 7f1c99ffb640 1 -- 172.21.15.90:0/235431523 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ca80122b0 con 0x55ffa839e390 2024-04-25T11:46:26.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.838+0000 7f1c99ffb640 1 -- 172.21.15.90:0/235431523 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca8004d90 con 0x55ffa839e390 2024-04-25T11:46:26.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.839+0000 7f1cc0795740 1 -- 172.21.15.90:0/235431523 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffa839e390 msgr2=0x55ffa873f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.839+0000 7f1cc0795740 1 --2- 172.21.15.90:0/235431523 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffa839e390 0x55ffa873f810 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1ca80126a0 tx=0x7f1ca8005f90 comp rx=0 tx=0).stop 2024-04-25T11:46:26.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.839+0000 7f1cc0795740 1 -- 172.21.15.90:0/235431523 shutdown_connections 2024-04-25T11:46:26.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.839+0000 7f1cc0795740 1 --2- 172.21.15.90:0/235431523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffa837d6b0 0x55ffa874c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.839+0000 7f1cc0795740 1 --2- 172.21.15.90:0/235431523 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffa83754d0 0x55ffa873fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.839+0000 7f1cc0795740 1 --2- 172.21.15.90:0/235431523 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffa839e390 0x55ffa873f810 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.839+0000 7f1cc0795740 1 -- 172.21.15.90:0/235431523 >> 172.21.15.90:0/235431523 conn(0x55ffa83924e0 msgr2=0x55ffa873bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:26.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.839+0000 7f1cc0795740 1 -- 172.21.15.90:0/235431523 shutdown_connections 2024-04-25T11:46:26.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.839+0000 7f1cc0795740 1 -- 172.21.15.90:0/235431523 wait complete. 2024-04-25T11:46:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.840+0000 7f1cc0795740 1 -- start start 2024-04-25T11:46:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.840+0000 7f1cc0795740 1 Processor -- start 2024-04-25T11:46:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.840+0000 7f1cc0795740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffa83754d0 0x55ffa874a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.840+0000 7f1cc0795740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffa837d6b0 0x55ffa874a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.840+0000 7f1cc0795740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffa839e390 0x55ffa8756210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.840+0000 7f1cc0795740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ffa874cd90 con 0x55ffa837d6b0 2024-04-25T11:46:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.840+0000 7f1cc0795740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ffa862bdc0 con 0x55ffa83754d0 2024-04-25T11:46:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.840+0000 7f1cc0795740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ffa874cf10 con 0x55ffa839e390 2024-04-25T11:46:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.840+0000 7f1cc0795740 10 client.0 _ll_register_callbacks cb 0x55ffa8647600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.840+0000 7f1cb2ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffa837d6b0 0x55ffa874a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.840+0000 7f1cb2ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffa837d6b0 0x55ffa874a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48244/0 (socket says 172.21.15.90:48244) 2024-04-25T11:46:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.840+0000 7f1cb2ffd640 1 -- 172.21.15.90:0/391863136 learned_addr learned my addr 172.21.15.90:0/391863136 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.840+0000 7f1cb27fc640 1 --2- 172.21.15.90:0/391863136 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffa83754d0 0x55ffa874a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.841+0000 7f1cb27fc640 1 --2- 172.21.15.90:0/391863136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffa839e390 0x55ffa8756210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.841+0000 7f1cb2ffd640 1 -- 172.21.15.90:0/391863136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffa839e390 msgr2=0x55ffa8756210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.841+0000 7f1cb2ffd640 1 --2- 172.21.15.90:0/391863136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffa839e390 0x55ffa8756210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.841+0000 7f1cb2ffd640 1 -- 172.21.15.90:0/391863136 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffa83754d0 msgr2=0x55ffa874a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.841+0000 7f1cb2ffd640 1 --2- 172.21.15.90:0/391863136 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffa83754d0 0x55ffa874a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.841+0000 7f1cb2ffd640 1 -- 172.21.15.90:0/391863136 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffa83b63f0 con 0x55ffa837d6b0 2024-04-25T11:46:26.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.841+0000 7f1cb27fc640 1 --2- 172.21.15.90:0/391863136 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffa83754d0 0x55ffa874a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:46:26.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.841+0000 7f1cb2ffd640 1 --2- 172.21.15.90:0/391863136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffa837d6b0 0x55ffa874a780 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f1ca4031440 tx=0x7f1ca4031750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cf673d65c85950c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:26.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.841+0000 7f1c92ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:46:26.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.841+0000 7f1c92ffd640 1 -- 172.21.15.90:0/391863136 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4035070 con 0x55ffa837d6b0 2024-04-25T11:46:26.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.841+0000 7f1c92ffd640 1 -- 172.21.15.90:0/391863136 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7f1ca4004040 con 0x55ffa837d6b0 2024-04-25T11:46:26.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.841+0000 7f1cc0795740 1 -- 172.21.15.90:0/391863136 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ca80122b0 con 0x55ffa837d6b0 2024-04-25T11:46:26.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.841+0000 7f1c92ffd640 1 -- 172.21.15.90:0/391863136 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1ca4004240 con 0x55ffa837d6b0 2024-04-25T11:46:26.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.841+0000 7f1cc0795740 10 client.19242 fetch_fsmap learned FSMap version 28 2024-04-25T11:46:26.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.841+0000 7f1cc0795740 1 -- 172.21.15.90:0/391863136 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ffa84807d0 con 0x55ffa837d6b0 2024-04-25T11:46:26.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.841+0000 7f1c92ffd640 1 -- 172.21.15.90:0/391863136 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ca80122b0 con 0x55ffa837d6b0 2024-04-25T11:46:26.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.841+0000 7f1c92ffd640 1 -- 172.21.15.90:0/391863136 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f1ca4004960 con 0x55ffa837d6b0 2024-04-25T11:46:26.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.842+0000 7f1cc0795740 10 client.19242 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:46:26.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.842+0000 7f1cc0795740 10 client.19242 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:46:26.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.842+0000 7f1cc0795740 20 client.19242 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:26.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.842+0000 7f1cc0795740 1 --2- 172.21.15.90:0/391863136 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55ffa835cf60 0x55ffa875c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.842+0000 7f1cc0795740 4 client.19242 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:46:26.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.842+0000 7f1cc0795740 1 -- 172.21.15.90:0/391863136 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ffa86b2990 con 0x55ffa835cf60 2024-04-25T11:46:26.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.842+0000 7f1cb27fc640 1 --2- 172.21.15.90:0/391863136 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55ffa835cf60 0x55ffa875c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.842+0000 7f1c92ffd640 10 client.19242 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:46:26.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.842+0000 7f1cb27fc640 1 --2- 172.21.15.90:0/391863136 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55ffa835cf60 0x55ffa875c650 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a5430e2c15a5536d server_cookie=572d0406bc141f0b in_seq=0 out_seq=0 2024-04-25T11:46:26.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.843+0000 7f1c92ffd640 1 -- 172.21.15.90:0/391863136 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ffa86b2990 con 0x55ffa835cf60 2024-04-25T11:46:26.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.843+0000 7f1c92ffd640 10 client.19242 handle_command_reply: tid=0 2024-04-25T11:46:26.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.843+0000 7f1cc0795740 1 client.19242 shutdown 2024-04-25T11:46:26.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.843+0000 7f1cc0795740 1 -- 172.21.15.90:0/391863136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffa837d6b0 msgr2=0x55ffa874a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.843+0000 7f1cc0795740 1 --2- 172.21.15.90:0/391863136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffa837d6b0 0x55ffa874a780 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f1ca4031440 tx=0x7f1ca4031750 comp rx=0 tx=0).stop 2024-04-25T11:46:26.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.843+0000 7f1cc0795740 1 -- 172.21.15.90:0/391863136 shutdown_connections 2024-04-25T11:46:26.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.843+0000 7f1cc0795740 1 --2- 172.21.15.90:0/391863136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffa839e390 0x55ffa8756210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.843+0000 7f1cc0795740 1 --2- 172.21.15.90:0/391863136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffa837d6b0 0x55ffa874a780 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.843+0000 7f1cc0795740 1 --2- 172.21.15.90:0/391863136 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55ffa835cf60 0x55ffa875c650 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.843+0000 7f1c92ffd640 0 client.19242 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:46:26.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.843+0000 7f1cc0795740 1 --2- 172.21.15.90:0/391863136 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffa83754d0 0x55ffa874a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.843+0000 7f1cc0795740 1 -- 172.21.15.90:0/391863136 >> 172.21.15.90:0/391863136 conn(0x55ffa83924e0 msgr2=0x55ffa8741530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:26.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.843+0000 7f1cc0795740 1 -- 172.21.15.90:0/391863136 shutdown_connections 2024-04-25T11:46:26.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.843+0000 7f1cc0795740 1 -- 172.21.15.90:0/391863136 wait complete. 2024-04-25T11:46:26.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.843+0000 7f1cc0795740 20 client.19242 trim_cache size 0 max 16384 2024-04-25T11:46:26.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.857+0000 7f1cc0795740 1 Processor -- start 2024-04-25T11:46:26.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.857+0000 7f1cc0795740 1 -- start start 2024-04-25T11:46:26.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.857+0000 7f1cc0795740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffa835cf60 0x55ffa873fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.857+0000 7f1cc0795740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffa83754d0 0x55ffa8755e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.857+0000 7f1cc0795740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffa837d6b0 0x55ffa8744430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.857+0000 7f1cc0795740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb41afd00 con 0x55ffa83754d0 2024-04-25T11:46:26.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.857+0000 7f1cc0795740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb41b1840 con 0x55ffa835cf60 2024-04-25T11:46:26.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.857+0000 7f1cc0795740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb41b03d0 con 0x55ffa837d6b0 2024-04-25T11:46:26.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.857+0000 7f1cb2ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffa83754d0 0x55ffa8755e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.857+0000 7f1cb2ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffa83754d0 0x55ffa8755e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48256/0 (socket says 172.21.15.90:48256) 2024-04-25T11:46:26.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.857+0000 7f1cb2ffd640 1 -- 172.21.15.90:0/811814966 learned_addr learned my addr 172.21.15.90:0/811814966 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:26.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.857+0000 7f1cb27fc640 1 --2- 172.21.15.90:0/811814966 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffa835cf60 0x55ffa873fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.857+0000 7f1cb27fc640 1 --2- 172.21.15.90:0/811814966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffa837d6b0 0x55ffa8744430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.857+0000 7f1cb2ffd640 1 -- 172.21.15.90:0/811814966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffa837d6b0 msgr2=0x55ffa8744430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.857+0000 7f1cb2ffd640 1 --2- 172.21.15.90:0/811814966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffa837d6b0 0x55ffa8744430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.857+0000 7f1cb2ffd640 1 -- 172.21.15.90:0/811814966 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffa835cf60 msgr2=0x55ffa873fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.857+0000 7f1cb2ffd640 1 --2- 172.21.15.90:0/811814966 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffa835cf60 0x55ffa873fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.857+0000 7f1cb2ffd640 1 -- 172.21.15.90:0/811814966 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffa873c9b0 con 0x55ffa83754d0 2024-04-25T11:46:26.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.858+0000 7f1cb2ffd640 1 --2- 172.21.15.90:0/811814966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffa83754d0 0x55ffa8755e70 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f1ca4006010 tx=0x7f1ca402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2211f82b8b30a552 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:26.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.858+0000 7f1c997fa640 1 -- 172.21.15.90:0/811814966 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4035050 con 0x55ffa83754d0 2024-04-25T11:46:26.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.858+0000 7f1c997fa640 1 -- 172.21.15.90:0/811814966 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ca40035f0 con 0x55ffa83754d0 2024-04-25T11:46:26.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.858+0000 7f1c997fa640 1 -- 172.21.15.90:0/811814966 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca40055b0 con 0x55ffa83754d0 2024-04-25T11:46:26.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.859+0000 7f1cc0795740 1 -- 172.21.15.90:0/811814966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffa83754d0 msgr2=0x55ffa8755e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.859+0000 7f1cc0795740 1 --2- 172.21.15.90:0/811814966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffa83754d0 0x55ffa8755e70 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f1ca4006010 tx=0x7f1ca402a410 comp rx=0 tx=0).stop 2024-04-25T11:46:26.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.859+0000 7f1cc0795740 1 -- 172.21.15.90:0/811814966 shutdown_connections 2024-04-25T11:46:26.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.859+0000 7f1cc0795740 1 --2- 172.21.15.90:0/811814966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffa837d6b0 0x55ffa8744430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.859+0000 7f1cc0795740 1 --2- 172.21.15.90:0/811814966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffa83754d0 0x55ffa8755e70 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.859+0000 7f1cc0795740 1 --2- 172.21.15.90:0/811814966 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffa835cf60 0x55ffa873fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.859+0000 7f1cc0795740 1 -- 172.21.15.90:0/811814966 >> 172.21.15.90:0/811814966 conn(0x55ffa83924e0 msgr2=0x55ffa873f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:26.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.859+0000 7f1cc0795740 1 -- 172.21.15.90:0/811814966 shutdown_connections 2024-04-25T11:46:26.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.859+0000 7f1cc0795740 1 -- 172.21.15.90:0/811814966 wait complete. 2024-04-25T11:46:26.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.859+0000 7f1cc0795740 1 -- start start 2024-04-25T11:46:26.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.859+0000 7f1cc0795740 1 Processor -- start 2024-04-25T11:46:26.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.859+0000 7f1cc0795740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffa835cf60 0x55ffa87bcd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.860+0000 7f1cc0795740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffa83754d0 0x55ffa87bd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.860+0000 7f1cc0795740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffa837d6b0 0x55ffa874d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.860+0000 7f1cc0795740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb41b1170 con 0x55ffa837d6b0 2024-04-25T11:46:26.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.860+0000 7f1cc0795740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb41b0aa0 con 0x55ffa83754d0 2024-04-25T11:46:26.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.860+0000 7f1cc0795740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb41add00 con 0x55ffa835cf60 2024-04-25T11:46:26.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.860+0000 7f1cc0795740 10 client.0 _ll_register_callbacks cb 0x7f1ca8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:26.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.860+0000 7f1cb27fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffa837d6b0 0x55ffa874d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.860+0000 7f1cb2ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffa83754d0 0x55ffa87bd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.860+0000 7f1cb2ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffa83754d0 0x55ffa87bd2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:33954/0 (socket says 172.21.15.90:33954) 2024-04-25T11:46:26.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.860+0000 7f1cb2ffd640 1 -- 172.21.15.90:0/2558887028 learned_addr learned my addr 172.21.15.90:0/2558887028 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:26.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.860+0000 7f1cb27fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffa837d6b0 0x55ffa874d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48266/0 (socket says 172.21.15.90:48266) 2024-04-25T11:46:26.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.860+0000 7f1cb27fc640 1 --2- 172.21.15.90:0/2558887028 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffa835cf60 0x55ffa87bcd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.860+0000 7f1cb2ffd640 1 -- 172.21.15.90:0/2558887028 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffa835cf60 msgr2=0x55ffa87bcd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.860+0000 7f1cb2ffd640 1 --2- 172.21.15.90:0/2558887028 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffa835cf60 0x55ffa87bcd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.860+0000 7f1cb2ffd640 1 -- 172.21.15.90:0/2558887028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffa837d6b0 msgr2=0x55ffa874d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.860+0000 7f1cb2ffd640 1 --2- 172.21.15.90:0/2558887028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffa837d6b0 0x55ffa874d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.860+0000 7f1cb2ffd640 1 -- 172.21.15.90:0/2558887028 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffa874dbb0 con 0x55ffa83754d0 2024-04-25T11:46:26.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.860+0000 7f1cb2ffd640 1 --2- 172.21.15.90:0/2558887028 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffa83754d0 0x55ffa87bd2b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1ca4004e50 tx=0x7f1ca4031750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21e089203113a94c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:26.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.860+0000 7f1c92ffd640 10 client.0 ms_handle_connect on v2:172.21.15.138:3300/0 2024-04-25T11:46:26.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.860+0000 7f1c92ffd640 1 -- 172.21.15.90:0/2558887028 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4035040 con 0x55ffa83754d0 2024-04-25T11:46:26.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.860+0000 7f1c92ffd640 1 -- 172.21.15.90:0/2558887028 <== mon.2 v2:172.21.15.138:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7f1ca4005440 con 0x55ffa83754d0 2024-04-25T11:46:26.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.860+0000 7f1cc0795740 1 -- 172.21.15.90:0/2558887028 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ca40035f0 con 0x55ffa83754d0 2024-04-25T11:46:26.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.861+0000 7f1c92ffd640 1 -- 172.21.15.90:0/2558887028 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1ca40046c0 con 0x55ffa83754d0 2024-04-25T11:46:26.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.861+0000 7f1cc0795740 10 client.27794 fetch_fsmap learned FSMap version 28 2024-04-25T11:46:26.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.861+0000 7f1cc0795740 1 -- 172.21.15.90:0/2558887028 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ffa87c4390 con 0x55ffa83754d0 2024-04-25T11:46:26.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.861+0000 7f1c92ffd640 1 -- 172.21.15.90:0/2558887028 <== mon.2 v2:172.21.15.138:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ca40035f0 con 0x55ffa83754d0 2024-04-25T11:46:26.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.861+0000 7f1c92ffd640 1 -- 172.21.15.90:0/2558887028 <== mon.2 v2:172.21.15.138:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f1ca4003e80 con 0x55ffa83754d0 2024-04-25T11:46:26.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.861+0000 7f1cc0795740 10 client.27794 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:46:26.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.861+0000 7f1cc0795740 10 client.27794 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:46:26.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.861+0000 7f1cc0795740 20 client.27794 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:26.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.861+0000 7f1cc0795740 1 --2- 172.21.15.90:0/2558887028 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55ffa839e390 0x55ffa87c6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.861+0000 7f1cc0795740 4 client.27794 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:46:26.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.861+0000 7f1cc0795740 1 -- 172.21.15.90:0/2558887028 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ffa86ac800 con 0x55ffa839e390 2024-04-25T11:46:26.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.861+0000 7f1cb27fc640 1 --2- 172.21.15.90:0/2558887028 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55ffa839e390 0x55ffa87c6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.862+0000 7f1c92ffd640 10 client.27794 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:46:26.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.862+0000 7f1cb27fc640 1 --2- 172.21.15.90:0/2558887028 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55ffa839e390 0x55ffa87c6ab0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ddad4fcfc3f57ff server_cookie=547f4ad11015f0af in_seq=0 out_seq=0 2024-04-25T11:46:26.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.862+0000 7f1c92ffd640 1 -- 172.21.15.90:0/2558887028 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ffa86ac800 con 0x55ffa839e390 2024-04-25T11:46:26.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.862+0000 7f1c92ffd640 10 client.27794 handle_command_reply: tid=0 2024-04-25T11:46:26.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.862+0000 7f1cc0795740 1 client.27794 shutdown 2024-04-25T11:46:26.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.862+0000 7f1cc0795740 1 -- 172.21.15.90:0/2558887028 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffa83754d0 msgr2=0x55ffa87bd2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.862+0000 7f1cc0795740 1 --2- 172.21.15.90:0/2558887028 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffa83754d0 0x55ffa87bd2b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1ca4004e50 tx=0x7f1ca4031750 comp rx=0 tx=0).stop 2024-04-25T11:46:26.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.862+0000 7f1cc0795740 1 -- 172.21.15.90:0/2558887028 shutdown_connections 2024-04-25T11:46:26.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.862+0000 7f1cc0795740 1 --2- 172.21.15.90:0/2558887028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ffa837d6b0 0x55ffa874d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.862+0000 7f1cc0795740 1 --2- 172.21.15.90:0/2558887028 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55ffa839e390 0x55ffa87c6ab0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.862+0000 7f1cc0795740 1 --2- 172.21.15.90:0/2558887028 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ffa83754d0 0x55ffa87bd2b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.862+0000 7f1cc0795740 1 --2- 172.21.15.90:0/2558887028 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ffa835cf60 0x55ffa87bcd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.862+0000 7f1c92ffd640 0 client.27794 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:46:26.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.862+0000 7f1cc0795740 1 -- 172.21.15.90:0/2558887028 >> 172.21.15.90:0/2558887028 conn(0x55ffa83924e0 msgr2=0x55ffa873b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:26.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.862+0000 7f1cc0795740 1 -- 172.21.15.90:0/2558887028 shutdown_connections 2024-04-25T11:46:26.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.862+0000 7f1cc0795740 1 -- 172.21.15.90:0/2558887028 wait complete. 2024-04-25T11:46:26.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.862+0000 7f1cc0795740 20 client.27794 trim_cache size 0 max 16384 2024-04-25T11:46:26.855 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:46:26.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.863+0000 7f1cb98e9640 1 -- 172.21.15.90:0/2736633708 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1c9c0afc60 msgr2=0x7f1c9c0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.863+0000 7f1cb98e9640 1 --2- 172.21.15.90:0/2736633708 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1c9c0afc60 0x7f1c9c0b2170 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1cac0a4b00 tx=0x7f1cac072f90 comp rx=0 tx=0).stop 2024-04-25T11:46:26.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.863+0000 7f1cb98e9640 1 -- 172.21.15.90:0/2736633708 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cb4108eb0 msgr2=0x7f1cb419c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.863+0000 7f1cb98e9640 1 --2- 172.21.15.90:0/2736633708 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cb4108eb0 0x7f1cb419c060 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f1cac00eb10 tx=0x7f1cac00c2d0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.863+0000 7f1cb98e9640 1 -- 172.21.15.90:0/2736633708 shutdown_connections 2024-04-25T11:46:26.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.863+0000 7f1cb98e9640 1 --2- 172.21.15.90:0/2736633708 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f1c9c0afc60 0x7f1c9c0b2170 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.863+0000 7f1cb98e9640 1 --2- 172.21.15.90:0/2736633708 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cb4108eb0 0x7f1cb419c060 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.863+0000 7f1cb98e9640 1 --2- 172.21.15.90:0/2736633708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1cb4108a10 0x7f1cb419bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.863+0000 7f1cb98e9640 1 --2- 172.21.15.90:0/2736633708 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1cb4103830 0x7f1cb419b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.863+0000 7f1cb98e9640 1 -- 172.21.15.90:0/2736633708 >> 172.21.15.90:0/2736633708 conn(0x7f1cb4077880 msgr2=0x7f1cb41040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:26.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.863+0000 7f1cb98e9640 1 -- 172.21.15.90:0/2736633708 shutdown_connections 2024-04-25T11:46:26.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.863+0000 7f1cb98e9640 1 -- 172.21.15.90:0/2736633708 wait complete. 2024-04-25T11:46:26.860 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T11:46:26.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.988+0000 7f5fab0bd640 1 -- 172.21.15.90:0/3502744276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5fa4107ce0 msgr2=0x7f5fa410f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.988+0000 7f5fab0bd640 1 --2- 172.21.15.90:0/3502744276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5fa4107ce0 0x7f5fa410f510 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c009ca0 tx=0x7f5f9c02a8c0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.988+0000 7f5fab0bd640 1 -- 172.21.15.90:0/3502744276 shutdown_connections 2024-04-25T11:46:26.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.988+0000 7f5fab0bd640 1 --2- 172.21.15.90:0/3502744276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5fa4107ce0 0x7f5fa410f510 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.988+0000 7f5fab0bd640 1 --2- 172.21.15.90:0/3502744276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5fa4107260 0x7f5fa4107670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.988+0000 7f5fab0bd640 1 --2- 172.21.15.90:0/3502744276 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5fa41033a0 0x7f5fa41037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.988+0000 7f5fab0bd640 1 -- 172.21.15.90:0/3502744276 >> 172.21.15.90:0/3502744276 conn(0x7f5fa4077880 msgr2=0x7f5fa4077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:26.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.988+0000 7f5fab0bd640 1 -- 172.21.15.90:0/3502744276 shutdown_connections 2024-04-25T11:46:26.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.988+0000 7f5fab0bd640 1 -- 172.21.15.90:0/3502744276 wait complete. 2024-04-25T11:46:26.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.989+0000 7f5fab0bd640 1 Processor -- start 2024-04-25T11:46:26.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.989+0000 7f5fab0bd640 1 -- start start 2024-04-25T11:46:26.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.989+0000 7f5fab0bd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5fa41033a0 0x7f5fa4199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.989+0000 7f5fab0bd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5fa4107260 0x7f5fa4199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.989+0000 7f5fab0bd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5fa4107ce0 0x7f5fa419de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.989+0000 7f5fab0bd640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa4112c00 con 0x7f5fa4107ce0 2024-04-25T11:46:26.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.989+0000 7f5fab0bd640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa4112a80 con 0x7f5fa41033a0 2024-04-25T11:46:26.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.989+0000 7f5fab0bd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa4112d80 con 0x7f5fa4107260 2024-04-25T11:46:26.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.989+0000 7f5fa965a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5fa4107ce0 0x7f5fa419de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.989+0000 7f5fa8e59640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5fa41033a0 0x7f5fa4199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.989+0000 7f5fa965a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5fa4107ce0 0x7f5fa419de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48274/0 (socket says 172.21.15.90:48274) 2024-04-25T11:46:26.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.989+0000 7f5fa965a640 1 -- 172.21.15.90:0/3554759495 learned_addr learned my addr 172.21.15.90:0/3554759495 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:26.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.989+0000 7f5fa3fff640 1 --2- 172.21.15.90:0/3554759495 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5fa4107260 0x7f5fa4199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.989+0000 7f5fa965a640 1 -- 172.21.15.90:0/3554759495 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5fa4107260 msgr2=0x7f5fa4199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.989+0000 7f5fa965a640 1 --2- 172.21.15.90:0/3554759495 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5fa4107260 0x7f5fa4199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.989+0000 7f5fa965a640 1 -- 172.21.15.90:0/3554759495 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5fa41033a0 msgr2=0x7f5fa4199620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.989+0000 7f5fa965a640 1 --2- 172.21.15.90:0/3554759495 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5fa41033a0 0x7f5fa4199620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.989+0000 7f5fa965a640 1 -- 172.21.15.90:0/3554759495 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa419e480 con 0x7f5fa4107ce0 2024-04-25T11:46:26.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.990+0000 7f5fa8e59640 1 --2- 172.21.15.90:0/3554759495 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5fa41033a0 0x7f5fa4199620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:46:26.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.990+0000 7f5fa3fff640 1 --2- 172.21.15.90:0/3554759495 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5fa4107260 0x7f5fa4199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:46:26.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.990+0000 7f5fa965a640 1 --2- 172.21.15.90:0/3554759495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5fa4107ce0 0x7f5fa419de40 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c005cc0 tx=0x7f5f9c031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:26.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.990+0000 7f5fa1ffb640 1 -- 172.21.15.90:0/3554759495 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c003b70 con 0x7f5fa4107ce0 2024-04-25T11:46:26.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.990+0000 7f5fab0bd640 1 -- 172.21.15.90:0/3554759495 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fa419e710 con 0x7f5fa4107ce0 2024-04-25T11:46:26.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.990+0000 7f5fa1ffb640 1 -- 172.21.15.90:0/3554759495 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f9c0318f0 con 0x7f5fa4107ce0 2024-04-25T11:46:26.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.990+0000 7f5fa1ffb640 1 -- 172.21.15.90:0/3554759495 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c02f950 con 0x7f5fa4107ce0 2024-04-25T11:46:26.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.990+0000 7f5fab0bd640 1 -- 172.21.15.90:0/3554759495 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fa419ec60 con 0x7f5fa4107ce0 2024-04-25T11:46:26.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.993+0000 7f5fb1f75740 1 Processor -- start 2024-04-25T11:46:26.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.993+0000 7f5fa1ffb640 1 -- 172.21.15.90:0/3554759495 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f5f9c02fb50 con 0x7f5fa4107ce0 2024-04-25T11:46:26.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.993+0000 7f5fb1f75740 1 -- start start 2024-04-25T11:46:26.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.993+0000 7f5fb1f75740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee0384390 0x556ee0725660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.994+0000 7f5fb1f75740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ee035b4d0 0x556ee0725ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.994+0000 7f5fb1f75740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556ee03636b0 0x556ee07348b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.994+0000 7f5fb1f75740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556ee068ad80 con 0x556ee0384390 2024-04-25T11:46:26.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.994+0000 7f5fb1f75740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x556ee0644a20 con 0x556ee03636b0 2024-04-25T11:46:26.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.994+0000 7f5fb1f75740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x556ee0686d40 con 0x556ee035b4d0 2024-04-25T11:46:26.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.994+0000 7f5fa3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee0384390 0x556ee0725660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.994+0000 7f5fa3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee0384390 0x556ee0725660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48280/0 (socket says 172.21.15.90:48280) 2024-04-25T11:46:26.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.994+0000 7f5fa3fff640 1 -- 172.21.15.90:0/2518926161 learned_addr learned my addr 172.21.15.90:0/2518926161 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:26.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.994+0000 7f5fa3fff640 1 --2- 172.21.15.90:0/2518926161 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556ee03636b0 0x556ee07348b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.994+0000 7f5fa3fff640 1 -- 172.21.15.90:0/2518926161 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ee035b4d0 msgr2=0x556ee0725ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.994+0000 7f5fa8e59640 1 --2- 172.21.15.90:0/2518926161 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ee035b4d0 0x556ee0725ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.994+0000 7f5fa3fff640 1 --2- 172.21.15.90:0/2518926161 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ee035b4d0 0x556ee0725ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.994+0000 7f5fa3fff640 1 -- 172.21.15.90:0/2518926161 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556ee03636b0 msgr2=0x556ee07348b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.994+0000 7f5fa3fff640 1 --2- 172.21.15.90:0/2518926161 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556ee03636b0 0x556ee07348b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.994+0000 7f5fa1ffb640 1 --2- 172.21.15.90:0/3554759495 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5f940afc60 0x7f5f940b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.994+0000 7f5fa3fff640 1 -- 172.21.15.90:0/2518926161 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ee064fc30 con 0x556ee0384390 2024-04-25T11:46:26.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.994+0000 7f5fa8e59640 1 --2- 172.21.15.90:0/2518926161 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ee035b4d0 0x556ee0725ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:46:26.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.995+0000 7f5fa1ffb640 1 -- 172.21.15.90:0/3554759495 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7f5f9c100460 con 0x7f5fa4107ce0 2024-04-25T11:46:26.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.995+0000 7f5fa3fff640 1 --2- 172.21.15.90:0/2518926161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee0384390 0x556ee0725660 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f5f900155d0 tx=0x7f5f900098f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f70706aa007a1d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:26.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.995+0000 7f5fa8e59640 1 --2- 172.21.15.90:0/3554759495 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5f940afc60 0x7f5f940b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.995+0000 7f5f8effd640 1 -- 172.21.15.90:0/2518926161 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f9001b020 con 0x556ee0384390 2024-04-25T11:46:26.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.995+0000 7f5f8effd640 1 -- 172.21.15.90:0/2518926161 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f900035f0 con 0x556ee0384390 2024-04-25T11:46:26.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.995+0000 7f5f8effd640 1 -- 172.21.15.90:0/2518926161 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f900044e0 con 0x556ee0384390 2024-04-25T11:46:26.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.997+0000 7f5fb1f75740 1 -- 172.21.15.90:0/2518926161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee0384390 msgr2=0x556ee0725660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.997+0000 7f5fa8e59640 1 --2- 172.21.15.90:0/3554759495 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5f940afc60 0x7f5f940b2170 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4069e60 tx=0x7f5f98005f90 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:26.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.997+0000 7f5fb1f75740 1 --2- 172.21.15.90:0/2518926161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee0384390 0x556ee0725660 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f5f900155d0 tx=0x7f5f900098f0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.997+0000 7f5fb1f75740 1 -- 172.21.15.90:0/2518926161 shutdown_connections 2024-04-25T11:46:26.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.997+0000 7f5fb1f75740 1 --2- 172.21.15.90:0/2518926161 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556ee03636b0 0x556ee07348b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.997+0000 7f5fb1f75740 1 --2- 172.21.15.90:0/2518926161 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ee035b4d0 0x556ee0725ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.997+0000 7f5fb1f75740 1 --2- 172.21.15.90:0/2518926161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee0384390 0x556ee0725660 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.997+0000 7f5fb1f75740 1 -- 172.21.15.90:0/2518926161 >> 172.21.15.90:0/2518926161 conn(0x556ee03784e0 msgr2=0x556ee0721fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:26.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.997+0000 7f5fb1f75740 1 -- 172.21.15.90:0/2518926161 shutdown_connections 2024-04-25T11:46:26.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.997+0000 7f5fb1f75740 1 -- 172.21.15.90:0/2518926161 wait complete. 2024-04-25T11:46:26.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.998+0000 7f5fb1f75740 1 -- start start 2024-04-25T11:46:26.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.998+0000 7f5fb1f75740 1 Processor -- start 2024-04-25T11:46:26.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.999+0000 7f5fb1f75740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ee035b4d0 0x556ee0734310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.999+0000 7f5fb1f75740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee03636b0 0x556ee0734850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.999+0000 7f5fa8e59640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ee035b4d0 0x556ee0734310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.999+0000 7f5fa8e59640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ee035b4d0 0x556ee0734310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:49210/0 (socket says 172.21.15.90:49210) 2024-04-25T11:46:26.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.999+0000 7f5fa8e59640 1 -- 172.21.15.90:0/1532312515 learned_addr learned my addr 172.21.15.90:0/1532312515 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:26.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.999+0000 7f5fb1f75740 1 --2- 172.21.15.90:0/1532312515 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556ee0384390 0x556ee0730780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.999+0000 7f5fa3fff640 1 --2- 172.21.15.90:0/1532312515 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee03636b0 0x556ee0734850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.999+0000 7f5fb1f75740 1 -- 172.21.15.90:0/1532312515 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556ee07287e0 con 0x556ee03636b0 2024-04-25T11:46:26.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.999+0000 7f5fb1f75740 1 -- 172.21.15.90:0/1532312515 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x556ee0611dc0 con 0x556ee0384390 2024-04-25T11:46:26.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.999+0000 7f5fa965a640 1 --2- 172.21.15.90:0/1532312515 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556ee0384390 0x556ee0730780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:26.999+0000 7f5fb1f75740 1 -- 172.21.15.90:0/1532312515 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x556ee0728960 con 0x556ee035b4d0 2024-04-25T11:46:26.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.000+0000 7f5fb1f75740 10 client.0 _ll_register_callbacks cb 0x556ee062d600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:26.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.000+0000 7f5fa8e59640 1 -- 172.21.15.90:0/1532312515 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556ee0384390 msgr2=0x556ee0730780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.000+0000 7f5fa8e59640 1 --2- 172.21.15.90:0/1532312515 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556ee0384390 0x556ee0730780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.000+0000 7f5fa8e59640 1 -- 172.21.15.90:0/1532312515 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee03636b0 msgr2=0x556ee0734850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:26.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.000+0000 7f5fa8e59640 1 --2- 172.21.15.90:0/1532312515 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee03636b0 0x556ee0734850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.000+0000 7f5fa8e59640 1 -- 172.21.15.90:0/1532312515 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ee039c3f0 con 0x556ee035b4d0 2024-04-25T11:46:26.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.000+0000 7f5fa965a640 1 --2- 172.21.15.90:0/1532312515 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556ee0384390 0x556ee0730780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:46:26.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.000+0000 7f5fa3fff640 1 --2- 172.21.15.90:0/1532312515 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee03636b0 0x556ee0734850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:46:26.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.000+0000 7f5fa8e59640 1 --2- 172.21.15.90:0/1532312515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ee035b4d0 0x556ee0734310 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5f980058c0 tx=0x7f5f980046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c5dea1441ef8cc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:26.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.000+0000 7f5f83fff640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:46:26.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.000+0000 7f5f83fff640 1 -- 172.21.15.90:0/1532312515 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f9801fb20 con 0x556ee035b4d0 2024-04-25T11:46:26.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.000+0000 7f5fb1f75740 1 -- 172.21.15.90:0/1532312515 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f900035f0 con 0x556ee035b4d0 2024-04-25T11:46:26.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.000+0000 7f5f83fff640 1 -- 172.21.15.90:0/1532312515 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7f5f9801fcc0 con 0x556ee035b4d0 2024-04-25T11:46:26.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.001+0000 7f5f83fff640 1 -- 172.21.15.90:0/1532312515 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5f98005020 con 0x556ee035b4d0 2024-04-25T11:46:26.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.001+0000 7f5fb1f75740 10 client.27622 fetch_fsmap learned FSMap version 28 2024-04-25T11:46:26.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.001+0000 7f5fb1f75740 1 -- 172.21.15.90:0/1532312515 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556ee04667d0 con 0x556ee035b4d0 2024-04-25T11:46:26.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.001+0000 7f5f83fff640 1 -- 172.21.15.90:0/1532312515 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f900035f0 con 0x556ee035b4d0 2024-04-25T11:46:26.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.001+0000 7f5f83fff640 1 -- 172.21.15.90:0/1532312515 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f5f9801d410 con 0x556ee035b4d0 2024-04-25T11:46:26.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.001+0000 7f5fb1f75740 10 client.27622 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:46:26.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.001+0000 7f5fb1f75740 10 client.27622 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:46:26.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.001+0000 7f5fb1f75740 20 client.27622 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:26.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.001+0000 7f5fb1f75740 1 --2- 172.21.15.90:0/1532312515 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x556ee0342f60 0x556ee0746900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:26.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.001+0000 7f5fb1f75740 4 client.27622 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:46:26.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.001+0000 7f5fb1f75740 1 -- 172.21.15.90:0/1532312515 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ee0698990 con 0x556ee0342f60 2024-04-25T11:46:26.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.001+0000 7f5fa3fff640 1 --2- 172.21.15.90:0/1532312515 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x556ee0342f60 0x556ee0746900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:26.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.002+0000 7f5f83fff640 10 client.27622 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:46:26.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.002+0000 7f5fa3fff640 1 --2- 172.21.15.90:0/1532312515 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x556ee0342f60 0x556ee0746900 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a8049f496edb6148 server_cookie=16902d4bccdb54c4 in_seq=0 out_seq=0 2024-04-25T11:46:26.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.003+0000 7f5f83fff640 1 -- 172.21.15.90:0/1532312515 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556ee0698990 con 0x556ee0342f60 2024-04-25T11:46:26.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.003+0000 7f5f83fff640 10 client.27622 handle_command_reply: tid=0 2024-04-25T11:46:26.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.003+0000 7f5fb1f75740 1 client.27622 shutdown 2024-04-25T11:46:26.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.003+0000 7f5fb1f75740 1 -- 172.21.15.90:0/1532312515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ee035b4d0 msgr2=0x556ee0734310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:26.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.003+0000 7f5fb1f75740 1 --2- 172.21.15.90:0/1532312515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ee035b4d0 0x556ee0734310 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5f980058c0 tx=0x7f5f980046c0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.003+0000 7f5fb1f75740 1 -- 172.21.15.90:0/1532312515 shutdown_connections 2024-04-25T11:46:26.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.003+0000 7f5fb1f75740 1 --2- 172.21.15.90:0/1532312515 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x556ee0342f60 0x556ee0746900 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.003+0000 7f5fb1f75740 1 --2- 172.21.15.90:0/1532312515 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556ee0384390 0x556ee0730780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.003+0000 7f5fb1f75740 1 --2- 172.21.15.90:0/1532312515 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee03636b0 0x556ee0734850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.003+0000 7f5f83fff640 0 client.27622 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:46:26.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.003+0000 7f5fb1f75740 1 --2- 172.21.15.90:0/1532312515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ee035b4d0 0x556ee0734310 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:26.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.003+0000 7f5fb1f75740 1 -- 172.21.15.90:0/1532312515 >> 172.21.15.90:0/1532312515 conn(0x556ee03784e0 msgr2=0x556ee07274c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:26.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.003+0000 7f5fb1f75740 1 -- 172.21.15.90:0/1532312515 shutdown_connections 2024-04-25T11:46:26.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.003+0000 7f5fb1f75740 1 -- 172.21.15.90:0/1532312515 wait complete. 2024-04-25T11:46:26.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.003+0000 7f5fb1f75740 20 client.27622 trim_cache size 0 max 16384 2024-04-25T11:46:27.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.027+0000 7f5fb1f75740 1 Processor -- start 2024-04-25T11:46:27.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.027+0000 7f5fb1f75740 1 -- start start 2024-04-25T11:46:27.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.027+0000 7f5fb1f75740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee035b4d0 0x556ee072ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.027+0000 7f5fb1f75740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556ee03636b0 0x556ee072f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.027+0000 7f5fb1f75740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ee0384390 0x556ee074c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.027+0000 7f5fb1f75740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa41adb30 con 0x556ee035b4d0 2024-04-25T11:46:27.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.027+0000 7f5fb1f75740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa41af670 con 0x556ee03636b0 2024-04-25T11:46:27.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.027+0000 7f5fb1f75740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa41ae200 con 0x556ee0384390 2024-04-25T11:46:27.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.027+0000 7f5fa8e59640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee035b4d0 0x556ee072ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.027+0000 7f5fa3fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556ee03636b0 0x556ee072f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.027+0000 7f5fa8e59640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee035b4d0 0x556ee072ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48302/0 (socket says 172.21.15.90:48302) 2024-04-25T11:46:27.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.027+0000 7f5fa8e59640 1 -- 172.21.15.90:0/667479651 learned_addr learned my addr 172.21.15.90:0/667479651 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:27.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.027+0000 7f5fa965a640 1 --2- 172.21.15.90:0/667479651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ee0384390 0x556ee074c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.028+0000 7f5fa8e59640 1 -- 172.21.15.90:0/667479651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ee0384390 msgr2=0x556ee074c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:27.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.028+0000 7f5fa8e59640 1 --2- 172.21.15.90:0/667479651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ee0384390 0x556ee074c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.028+0000 7f5fa8e59640 1 -- 172.21.15.90:0/667479651 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556ee03636b0 msgr2=0x556ee072f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:27.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.028+0000 7f5fa8e59640 1 --2- 172.21.15.90:0/667479651 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556ee03636b0 0x556ee072f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.028+0000 7f5fa8e59640 1 -- 172.21.15.90:0/667479651 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ee07433f0 con 0x556ee035b4d0 2024-04-25T11:46:27.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.028+0000 7f5fa3fff640 1 --2- 172.21.15.90:0/667479651 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556ee03636b0 0x556ee072f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:46:27.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.028+0000 7f5fa965a640 1 --2- 172.21.15.90:0/667479651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ee0384390 0x556ee074c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:46:27.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.028+0000 7f5fa8e59640 1 --2- 172.21.15.90:0/667479651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee035b4d0 0x556ee072ef40 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f5f98018040 tx=0x7f5f9801de70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2976880f23e3af2a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:27.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.030+0000 7f5f8e7fc640 1 -- 172.21.15.90:0/667479651 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f9801d610 con 0x556ee035b4d0 2024-04-25T11:46:27.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.030+0000 7f5f8e7fc640 1 -- 172.21.15.90:0/667479651 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f9801daa0 con 0x556ee035b4d0 2024-04-25T11:46:27.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.030+0000 7f5f8e7fc640 1 -- 172.21.15.90:0/667479651 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f98004730 con 0x556ee035b4d0 2024-04-25T11:46:27.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.030+0000 7f5fb1f75740 1 -- 172.21.15.90:0/667479651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee035b4d0 msgr2=0x556ee072ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:27.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.030+0000 7f5fb1f75740 1 --2- 172.21.15.90:0/667479651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee035b4d0 0x556ee072ef40 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f5f98018040 tx=0x7f5f9801de70 comp rx=0 tx=0).stop 2024-04-25T11:46:27.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.030+0000 7f5fb1f75740 1 -- 172.21.15.90:0/667479651 shutdown_connections 2024-04-25T11:46:27.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.030+0000 7f5fb1f75740 1 --2- 172.21.15.90:0/667479651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ee0384390 0x556ee074c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.031+0000 7f5fb1f75740 1 --2- 172.21.15.90:0/667479651 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556ee03636b0 0x556ee072f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.031+0000 7f5fb1f75740 1 --2- 172.21.15.90:0/667479651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee035b4d0 0x556ee072ef40 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.031+0000 7f5fb1f75740 1 -- 172.21.15.90:0/667479651 >> 172.21.15.90:0/667479651 conn(0x556ee03784e0 msgr2=0x556ee072b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:27.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.031+0000 7f5fb1f75740 1 -- 172.21.15.90:0/667479651 shutdown_connections 2024-04-25T11:46:27.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.031+0000 7f5fb1f75740 1 -- 172.21.15.90:0/667479651 wait complete. 2024-04-25T11:46:27.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.031+0000 7f5fb1f75740 1 -- start start 2024-04-25T11:46:27.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.031+0000 7f5fb1f75740 1 Processor -- start 2024-04-25T11:46:27.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.031+0000 7f5fb1f75740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ee035b4d0 0x556ee073e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.031+0000 7f5fb1f75740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556ee03636b0 0x556ee073aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.031+0000 7f5fb1f75740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee0384390 0x556ee073afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.031+0000 7f5fb1f75740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa41aefa0 con 0x556ee0384390 2024-04-25T11:46:27.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.031+0000 7f5fb1f75740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa41ae8d0 con 0x556ee03636b0 2024-04-25T11:46:27.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.031+0000 7f5fb1f75740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa41abb30 con 0x556ee035b4d0 2024-04-25T11:46:27.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.032+0000 7f5fb1f75740 10 client.0 _ll_register_callbacks cb 0x7f5f98009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:27.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.032+0000 7f5fa965a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee0384390 0x556ee073afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.032+0000 7f5fa3fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556ee03636b0 0x556ee073aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.032+0000 7f5fa8e59640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ee035b4d0 0x556ee073e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.032+0000 7f5fa965a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee0384390 0x556ee073afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48308/0 (socket says 172.21.15.90:48308) 2024-04-25T11:46:27.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.032+0000 7f5fa965a640 1 -- 172.21.15.90:0/1926730318 learned_addr learned my addr 172.21.15.90:0/1926730318 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:27.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.032+0000 7f5fa965a640 1 -- 172.21.15.90:0/1926730318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ee035b4d0 msgr2=0x556ee073e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:27.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.032+0000 7f5fa965a640 1 --2- 172.21.15.90:0/1926730318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ee035b4d0 0x556ee073e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.032+0000 7f5fa965a640 1 -- 172.21.15.90:0/1926730318 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556ee03636b0 msgr2=0x556ee073aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:27.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.032+0000 7f5fa965a640 1 --2- 172.21.15.90:0/1926730318 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556ee03636b0 0x556ee073aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.032+0000 7f5fa965a640 1 -- 172.21.15.90:0/1926730318 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ee073b650 con 0x556ee0384390 2024-04-25T11:46:27.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.032+0000 7f5fa3fff640 1 --2- 172.21.15.90:0/1926730318 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556ee03636b0 0x556ee073aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:46:27.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.032+0000 7f5fa965a640 1 --2- 172.21.15.90:0/1926730318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee0384390 0x556ee073afb0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c0097b0 tx=0x7f5f9c003f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a90687ac3352f649 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:27.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.032+0000 7f5f83fff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:46:27.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.032+0000 7f5f83fff640 1 -- 172.21.15.90:0/1926730318 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c0e1020 con 0x556ee0384390 2024-04-25T11:46:27.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.032+0000 7f5f83fff640 1 -- 172.21.15.90:0/1926730318 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7f5f9c0c8860 con 0x556ee0384390 2024-04-25T11:46:27.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.032+0000 7f5fb1f75740 1 -- 172.21.15.90:0/1926730318 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f9801daa0 con 0x556ee0384390 2024-04-25T11:46:27.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.033+0000 7f5f83fff640 1 -- 172.21.15.90:0/1926730318 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5f9c0c8600 con 0x556ee0384390 2024-04-25T11:46:27.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.033+0000 7f5fb1f75740 10 client.19266 fetch_fsmap learned FSMap version 28 2024-04-25T11:46:27.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.033+0000 7f5fb1f75740 1 -- 172.21.15.90:0/1926730318 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556ee07adb00 con 0x556ee0384390 2024-04-25T11:46:27.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.038+0000 7f5f83fff640 1 -- 172.21.15.90:0/1926730318 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f9801daa0 con 0x556ee0384390 2024-04-25T11:46:27.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.039+0000 7f5f83fff640 1 -- 172.21.15.90:0/1926730318 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f5f9c0d0fd0 con 0x556ee0384390 2024-04-25T11:46:27.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.039+0000 7f5fb1f75740 10 client.19266 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:46:27.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.039+0000 7f5fb1f75740 10 client.19266 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:46:27.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.039+0000 7f5fb1f75740 20 client.19266 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:27.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.039+0000 7f5fb1f75740 1 --2- 172.21.15.90:0/1926730318 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x556ee03183a0 0x556ee07b0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.039+0000 7f5fb1f75740 4 client.19266 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:46:27.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.039+0000 7f5fb1f75740 1 -- 172.21.15.90:0/1926730318 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556ee0692800 con 0x556ee03183a0 2024-04-25T11:46:27.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.039+0000 7f5fa8e59640 1 --2- 172.21.15.90:0/1926730318 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x556ee03183a0 0x556ee07b0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.039+0000 7f5f83fff640 10 client.19266 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:46:27.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.039+0000 7f5fa8e59640 1 --2- 172.21.15.90:0/1926730318 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x556ee03183a0 0x556ee07b0260 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cdec12a9d3cdc6db server_cookie=6161ce67b2e64c8d in_seq=0 out_seq=0 2024-04-25T11:46:27.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.040+0000 7f5f83fff640 1 -- 172.21.15.90:0/1926730318 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556ee0692800 con 0x556ee03183a0 2024-04-25T11:46:27.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.040+0000 7f5f83fff640 10 client.19266 handle_command_reply: tid=0 2024-04-25T11:46:27.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.040+0000 7f5fb1f75740 1 client.19266 shutdown 2024-04-25T11:46:27.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.040+0000 7f5fb1f75740 1 -- 172.21.15.90:0/1926730318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee0384390 msgr2=0x556ee073afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:27.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.040+0000 7f5fb1f75740 1 --2- 172.21.15.90:0/1926730318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee0384390 0x556ee073afb0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c0097b0 tx=0x7f5f9c003f40 comp rx=0 tx=0).stop 2024-04-25T11:46:27.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.040+0000 7f5fb1f75740 1 -- 172.21.15.90:0/1926730318 shutdown_connections 2024-04-25T11:46:27.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.040+0000 7f5fb1f75740 1 --2- 172.21.15.90:0/1926730318 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x556ee03183a0 0x556ee07b0260 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.040+0000 7f5fb1f75740 1 --2- 172.21.15.90:0/1926730318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ee0384390 0x556ee073afb0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.040+0000 7f5fb1f75740 1 --2- 172.21.15.90:0/1926730318 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x556ee03636b0 0x556ee073aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.040+0000 7f5fb1f75740 1 --2- 172.21.15.90:0/1926730318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ee035b4d0 0x556ee073e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.040+0000 7f5f83fff640 0 client.19266 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:46:27.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.040+0000 7f5fb1f75740 1 -- 172.21.15.90:0/1926730318 >> 172.21.15.90:0/1926730318 conn(0x556ee03784e0 msgr2=0x556ee072b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:27.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.040+0000 7f5fb1f75740 1 -- 172.21.15.90:0/1926730318 shutdown_connections 2024-04-25T11:46:27.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.040+0000 7f5fb1f75740 1 -- 172.21.15.90:0/1926730318 wait complete. 2024-04-25T11:46:27.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.040+0000 7f5fb1f75740 20 client.19266 trim_cache size 0 max 16384 2024-04-25T11:46:27.028 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:46:27.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.041+0000 7f5fab0bd640 1 -- 172.21.15.90:0/3554759495 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5f940afc60 msgr2=0x7f5f940b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:27.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.041+0000 7f5fab0bd640 1 --2- 172.21.15.90:0/3554759495 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5f940afc60 0x7f5f940b2170 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4069e60 tx=0x7f5f98005f90 comp rx=0 tx=0).stop 2024-04-25T11:46:27.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.041+0000 7f5fab0bd640 1 -- 172.21.15.90:0/3554759495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5fa4107ce0 msgr2=0x7f5fa419de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:27.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.041+0000 7f5fab0bd640 1 --2- 172.21.15.90:0/3554759495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5fa4107ce0 0x7f5fa419de40 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c005cc0 tx=0x7f5f9c031700 comp rx=0 tx=0).stop 2024-04-25T11:46:27.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.041+0000 7f5fab0bd640 1 -- 172.21.15.90:0/3554759495 shutdown_connections 2024-04-25T11:46:27.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.041+0000 7f5fab0bd640 1 --2- 172.21.15.90:0/3554759495 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f5f940afc60 0x7f5f940b2170 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.041+0000 7f5fab0bd640 1 --2- 172.21.15.90:0/3554759495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5fa4107ce0 0x7f5fa419de40 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.041+0000 7f5fab0bd640 1 --2- 172.21.15.90:0/3554759495 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5fa4107260 0x7f5fa4199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.041+0000 7f5fab0bd640 1 --2- 172.21.15.90:0/3554759495 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5fa41033a0 0x7f5fa4199620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.041+0000 7f5fab0bd640 1 -- 172.21.15.90:0/3554759495 >> 172.21.15.90:0/3554759495 conn(0x7f5fa4077880 msgr2=0x7f5fa40ff4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:27.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.041+0000 7f5fab0bd640 1 -- 172.21.15.90:0/3554759495 shutdown_connections 2024-04-25T11:46:27.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.041+0000 7f5fab0bd640 1 -- 172.21.15.90:0/3554759495 wait complete. 2024-04-25T11:46:27.040 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T11:46:27.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.173+0000 7fb747157640 1 -- 172.21.15.90:0/2606005032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb740104c60 msgr2=0x7fb740105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:27.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.173+0000 7fb747157640 1 --2- 172.21.15.90:0/2606005032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb740104c60 0x7fb740105070 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fb734009900 tx=0x7fb73402a480 comp rx=0 tx=0).stop 2024-04-25T11:46:27.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.173+0000 7fb747157640 1 -- 172.21.15.90:0/2606005032 shutdown_connections 2024-04-25T11:46:27.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.173+0000 7fb747157640 1 --2- 172.21.15.90:0/2606005032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb74010e070 0x7fb740110560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.173+0000 7fb747157640 1 --2- 172.21.15.90:0/2606005032 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb74010b6c0 0x7fb74010dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.173+0000 7fb747157640 1 --2- 172.21.15.90:0/2606005032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb740104c60 0x7fb740105070 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.173+0000 7fb747157640 1 -- 172.21.15.90:0/2606005032 >> 172.21.15.90:0/2606005032 conn(0x7fb7400fe260 msgr2=0x7fb740100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:27.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.173+0000 7fb747157640 1 -- 172.21.15.90:0/2606005032 shutdown_connections 2024-04-25T11:46:27.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.173+0000 7fb747157640 1 -- 172.21.15.90:0/2606005032 wait complete. 2024-04-25T11:46:27.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.175+0000 7fb747157640 1 Processor -- start 2024-04-25T11:46:27.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.175+0000 7fb747157640 1 -- start start 2024-04-25T11:46:27.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.175+0000 7fb747157640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb740104c60 0x7fb74019a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.175+0000 7fb747157640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb74010b6c0 0x7fb74019ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.175+0000 7fb747157640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb74010e070 0x7fb74019eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.175+0000 7fb747157640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb740113cd0 con 0x7fb740104c60 2024-04-25T11:46:27.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.175+0000 7fb747157640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb740113b50 con 0x7fb74010b6c0 2024-04-25T11:46:27.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.175+0000 7fb747157640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb740113e50 con 0x7fb74010e070 2024-04-25T11:46:27.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.175+0000 7fb744ef3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb740104c60 0x7fb74019a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.175+0000 7fb7456f4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb74010e070 0x7fb74019eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.175+0000 7fb73ffff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb74010b6c0 0x7fb74019ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.175+0000 7fb744ef3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb740104c60 0x7fb74019a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48314/0 (socket says 172.21.15.90:48314) 2024-04-25T11:46:27.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.175+0000 7fb744ef3640 1 -- 172.21.15.90:0/1672833961 learned_addr learned my addr 172.21.15.90:0/1672833961 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:27.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.175+0000 7fb7456f4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb74010e070 0x7fb74019eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:49232/0 (socket says 172.21.15.90:49232) 2024-04-25T11:46:27.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.175+0000 7fb73ffff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb74010b6c0 0x7fb74019ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:34020/0 (socket says 172.21.15.90:34020) 2024-04-25T11:46:27.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.175+0000 7fb744ef3640 1 -- 172.21.15.90:0/1672833961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb74010e070 msgr2=0x7fb74019eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:27.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.175+0000 7fb744ef3640 1 --2- 172.21.15.90:0/1672833961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb74010e070 0x7fb74019eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.175+0000 7fb744ef3640 1 -- 172.21.15.90:0/1672833961 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb74010b6c0 msgr2=0x7fb74019ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:27.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.175+0000 7fb744ef3640 1 --2- 172.21.15.90:0/1672833961 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb74010b6c0 0x7fb74019ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.175+0000 7fb744ef3640 1 -- 172.21.15.90:0/1672833961 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb74019f530 con 0x7fb740104c60 2024-04-25T11:46:27.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.175+0000 7fb744ef3640 1 --2- 172.21.15.90:0/1672833961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb740104c60 0x7fb74019a6d0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fb734005940 tx=0x7fb7340040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:27.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.175+0000 7fb73dffb640 1 -- 172.21.15.90:0/1672833961 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73403f030 con 0x7fb740104c60 2024-04-25T11:46:27.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.175+0000 7fb73dffb640 1 -- 172.21.15.90:0/1672833961 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7340044c0 con 0x7fb740104c60 2024-04-25T11:46:27.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.175+0000 7fb747157640 1 -- 172.21.15.90:0/1672833961 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb74019f7c0 con 0x7fb740104c60 2024-04-25T11:46:27.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.175+0000 7fb73dffb640 1 -- 172.21.15.90:0/1672833961 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb734035070 con 0x7fb740104c60 2024-04-25T11:46:27.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.175+0000 7fb747157640 1 -- 172.21.15.90:0/1672833961 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb74019fcb0 con 0x7fb740104c60 2024-04-25T11:46:27.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.178+0000 7fb73dffb640 1 -- 172.21.15.90:0/1672833961 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fb734042030 con 0x7fb740104c60 2024-04-25T11:46:27.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.178+0000 7fb74e009740 1 Processor -- start 2024-04-25T11:46:27.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.178+0000 7fb74e009740 1 -- start start 2024-04-25T11:46:27.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.178+0000 7fb74e009740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cde038f390 0x55cde0730810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.178+0000 7fb74e009740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cde03664d0 0x55cde0730d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.178+0000 7fb74e009740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cde036e6b0 0x55cde073d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.178+0000 7fb74e009740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55cde0695d80 con 0x55cde038f390 2024-04-25T11:46:27.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.178+0000 7fb74e009740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55cde064fa20 con 0x55cde036e6b0 2024-04-25T11:46:27.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.178+0000 7fb74e009740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55cde0691d40 con 0x55cde03664d0 2024-04-25T11:46:27.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.178+0000 7fb7456f4640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cde038f390 0x55cde0730810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.178+0000 7fb7456f4640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cde038f390 0x55cde0730810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48324/0 (socket says 172.21.15.90:48324) 2024-04-25T11:46:27.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.178+0000 7fb7456f4640 1 -- 172.21.15.90:0/23647118 learned_addr learned my addr 172.21.15.90:0/23647118 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:27.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.178+0000 7fb73ffff640 1 --2- 172.21.15.90:0/23647118 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cde036e6b0 0x55cde073d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.178+0000 7fb744ef3640 1 --2- 172.21.15.90:0/23647118 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cde03664d0 0x55cde0730d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.178+0000 7fb73dffb640 1 --2- 172.21.15.90:0/1672833961 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb7300afc60 0x7fb7300b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.178+0000 7fb73ffff640 1 -- 172.21.15.90:0/23647118 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cde03664d0 msgr2=0x55cde0730d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:27.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.178+0000 7fb73ffff640 1 --2- 172.21.15.90:0/23647118 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cde03664d0 0x55cde0730d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.178+0000 7fb73ffff640 1 -- 172.21.15.90:0/23647118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cde038f390 msgr2=0x55cde0730810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:27.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.178+0000 7fb73ffff640 1 --2- 172.21.15.90:0/23647118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cde038f390 0x55cde0730810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.178+0000 7fb73ffff640 1 -- 172.21.15.90:0/23647118 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cde065ac30 con 0x55cde036e6b0 2024-04-25T11:46:27.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.179+0000 7fb7456f4640 1 --2- 172.21.15.90:0/1672833961 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb7300afc60 0x7fb7300b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.179+0000 7fb73dffb640 1 -- 172.21.15.90:0/1672833961 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7fb7340ff450 con 0x7fb740104c60 2024-04-25T11:46:27.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.179+0000 7fb73ffff640 1 --2- 172.21.15.90:0/23647118 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cde036e6b0 0x55cde073d6a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb72c010d20 tx=0x7fb72c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e50a7d979350915e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:27.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.179+0000 7fb72affd640 1 -- 172.21.15.90:0/23647118 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb72c01f070 con 0x55cde036e6b0 2024-04-25T11:46:27.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.179+0000 7fb72affd640 1 -- 172.21.15.90:0/23647118 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb72c004250 con 0x55cde036e6b0 2024-04-25T11:46:27.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.179+0000 7fb72affd640 1 -- 172.21.15.90:0/23647118 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb72c01a760 con 0x55cde036e6b0 2024-04-25T11:46:27.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.181+0000 7fb74e009740 1 -- 172.21.15.90:0/23647118 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cde036e6b0 msgr2=0x55cde073d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:27.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.181+0000 7fb7456f4640 1 --2- 172.21.15.90:0/1672833961 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb7300afc60 0x7fb7300b2170 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x55cde072d980 tx=0x7fb738004ab0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:27.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.181+0000 7fb74e009740 1 --2- 172.21.15.90:0/23647118 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cde036e6b0 0x55cde073d6a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb72c010d20 tx=0x7fb72c005f90 comp rx=0 tx=0).stop 2024-04-25T11:46:27.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.181+0000 7fb74e009740 1 -- 172.21.15.90:0/23647118 shutdown_connections 2024-04-25T11:46:27.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.181+0000 7fb74e009740 1 --2- 172.21.15.90:0/23647118 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cde036e6b0 0x55cde073d6a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.181+0000 7fb74e009740 1 --2- 172.21.15.90:0/23647118 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cde03664d0 0x55cde0730d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.181+0000 7fb74e009740 1 --2- 172.21.15.90:0/23647118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cde038f390 0x55cde0730810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.181+0000 7fb74e009740 1 -- 172.21.15.90:0/23647118 >> 172.21.15.90:0/23647118 conn(0x55cde03834e0 msgr2=0x55cde072cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:27.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.181+0000 7fb74e009740 1 -- 172.21.15.90:0/23647118 shutdown_connections 2024-04-25T11:46:27.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.181+0000 7fb74e009740 1 -- 172.21.15.90:0/23647118 wait complete. 2024-04-25T11:46:27.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.181+0000 7fb74e009740 1 -- start start 2024-04-25T11:46:27.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.181+0000 7fb74e009740 1 Processor -- start 2024-04-25T11:46:27.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.181+0000 7fb74e009740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cde03664d0 0x55cde073b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.181+0000 7fb74e009740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cde036e6b0 0x55cde073b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.181+0000 7fb74e009740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cde038f390 0x55cde0747210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.181+0000 7fb74e009740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55cde073dd90 con 0x55cde036e6b0 2024-04-25T11:46:27.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.181+0000 7fb74e009740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55cde061cdc0 con 0x55cde03664d0 2024-04-25T11:46:27.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.181+0000 7fb74e009740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55cde073df10 con 0x55cde038f390 2024-04-25T11:46:27.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.181+0000 7fb74e009740 10 client.0 _ll_register_callbacks cb 0x55cde0638600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:27.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.181+0000 7fb73ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cde036e6b0 0x55cde073b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.181+0000 7fb744ef3640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cde03664d0 0x55cde073b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.181+0000 7fb7456f4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cde038f390 0x55cde0747210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.181+0000 7fb744ef3640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cde03664d0 0x55cde073b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:34042/0 (socket says 172.21.15.90:34042) 2024-04-25T11:46:27.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.181+0000 7fb744ef3640 1 -- 172.21.15.90:0/3659662876 learned_addr learned my addr 172.21.15.90:0/3659662876 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:27.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.181+0000 7fb7456f4640 1 -- 172.21.15.90:0/3659662876 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cde03664d0 msgr2=0x55cde073b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:27.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.182+0000 7fb7456f4640 1 --2- 172.21.15.90:0/3659662876 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cde03664d0 0x55cde073b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.182+0000 7fb7456f4640 1 -- 172.21.15.90:0/3659662876 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cde036e6b0 msgr2=0x55cde073b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:27.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.182+0000 7fb7456f4640 1 --2- 172.21.15.90:0/3659662876 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cde036e6b0 0x55cde073b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.182+0000 7fb7456f4640 1 -- 172.21.15.90:0/3659662876 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cde03a73f0 con 0x55cde038f390 2024-04-25T11:46:27.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.182+0000 7fb744ef3640 1 --2- 172.21.15.90:0/3659662876 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cde03664d0 0x55cde073b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:46:27.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.182+0000 7fb7456f4640 1 --2- 172.21.15.90:0/3659662876 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cde038f390 0x55cde0747210 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb738003e70 tx=0x7fb738004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=987bf0b3800259b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:27.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.182+0000 7fb71ffff640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:46:27.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.182+0000 7fb71ffff640 1 -- 172.21.15.90:0/3659662876 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73801a040 con 0x55cde038f390 2024-04-25T11:46:27.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.182+0000 7fb74e009740 1 -- 172.21.15.90:0/3659662876 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb72c004250 con 0x55cde038f390 2024-04-25T11:46:27.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.182+0000 7fb71ffff640 1 -- 172.21.15.90:0/3659662876 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7fb738004d20 con 0x55cde038f390 2024-04-25T11:46:27.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.182+0000 7fb71ffff640 1 -- 172.21.15.90:0/3659662876 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb738004310 con 0x55cde038f390 2024-04-25T11:46:27.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.182+0000 7fb74e009740 10 client.27640 fetch_fsmap learned FSMap version 28 2024-04-25T11:46:27.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.182+0000 7fb74e009740 1 -- 172.21.15.90:0/3659662876 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cde04717d0 con 0x55cde038f390 2024-04-25T11:46:27.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.182+0000 7fb71ffff640 1 -- 172.21.15.90:0/3659662876 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb72c004250 con 0x55cde038f390 2024-04-25T11:46:27.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.183+0000 7fb71ffff640 1 -- 172.21.15.90:0/3659662876 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fb738005720 con 0x55cde038f390 2024-04-25T11:46:27.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.183+0000 7fb74e009740 10 client.27640 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:46:27.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.183+0000 7fb74e009740 10 client.27640 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:46:27.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.183+0000 7fb74e009740 20 client.27640 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:27.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.183+0000 7fb74e009740 1 --2- 172.21.15.90:0/3659662876 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55cde034df60 0x55cde074dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.183+0000 7fb74e009740 4 client.27640 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:46:27.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.183+0000 7fb74e009740 1 -- 172.21.15.90:0/3659662876 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cde06a3990 con 0x55cde034df60 2024-04-25T11:46:27.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.183+0000 7fb744ef3640 1 --2- 172.21.15.90:0/3659662876 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55cde034df60 0x55cde074dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.185+0000 7fb71ffff640 10 client.27640 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:46:27.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.185+0000 7fb744ef3640 1 --2- 172.21.15.90:0/3659662876 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55cde034df60 0x55cde074dc40 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=60c71ad9dc4ec82 server_cookie=95ca92f675c503f2 in_seq=0 out_seq=0 2024-04-25T11:46:27.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.185+0000 7fb71ffff640 1 -- 172.21.15.90:0/3659662876 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55cde06a3990 con 0x55cde034df60 2024-04-25T11:46:27.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.185+0000 7fb71ffff640 10 client.27640 handle_command_reply: tid=0 2024-04-25T11:46:27.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.185+0000 7fb74e009740 1 client.27640 shutdown 2024-04-25T11:46:27.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.185+0000 7fb74e009740 1 -- 172.21.15.90:0/3659662876 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cde038f390 msgr2=0x55cde0747210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:27.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.185+0000 7fb74e009740 1 --2- 172.21.15.90:0/3659662876 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cde038f390 0x55cde0747210 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb738003e70 tx=0x7fb738004a40 comp rx=0 tx=0).stop 2024-04-25T11:46:27.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.185+0000 7fb74e009740 1 -- 172.21.15.90:0/3659662876 shutdown_connections 2024-04-25T11:46:27.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.185+0000 7fb74e009740 1 --2- 172.21.15.90:0/3659662876 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55cde034df60 0x55cde074dc40 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.185+0000 7fb74e009740 1 --2- 172.21.15.90:0/3659662876 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cde038f390 0x55cde0747210 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.185+0000 7fb74e009740 1 --2- 172.21.15.90:0/3659662876 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cde036e6b0 0x55cde073b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.185+0000 7fb74e009740 1 --2- 172.21.15.90:0/3659662876 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cde03664d0 0x55cde073b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.185+0000 7fb74e009740 1 -- 172.21.15.90:0/3659662876 >> 172.21.15.90:0/3659662876 conn(0x55cde03834e0 msgr2=0x55cde0732530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:27.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.185+0000 7fb71ffff640 0 client.27640 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:46:27.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.185+0000 7fb74e009740 1 -- 172.21.15.90:0/3659662876 shutdown_connections 2024-04-25T11:46:27.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.185+0000 7fb74e009740 1 -- 172.21.15.90:0/3659662876 wait complete. 2024-04-25T11:46:27.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.185+0000 7fb74e009740 20 client.27640 trim_cache size 0 max 16384 2024-04-25T11:46:27.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.214+0000 7fb74e009740 1 Processor -- start 2024-04-25T11:46:27.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.214+0000 7fb74e009740 1 -- start start 2024-04-25T11:46:27.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.214+0000 7fb74e009740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cde03664d0 0x55cde0728190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.215+0000 7fb74e009740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cde036e6b0 0x55cde0746e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.215+0000 7fb74e009740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cde038f390 0x55cde0735510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.215+0000 7fb74e009740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb7401adae0 con 0x55cde036e6b0 2024-04-25T11:46:27.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.215+0000 7fb74e009740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb7401af620 con 0x55cde03664d0 2024-04-25T11:46:27.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.215+0000 7fb74e009740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb7401ae1b0 con 0x55cde038f390 2024-04-25T11:46:27.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.215+0000 7fb73ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cde036e6b0 0x55cde0746e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.215+0000 7fb73ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cde036e6b0 0x55cde0746e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48344/0 (socket says 172.21.15.90:48344) 2024-04-25T11:46:27.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.215+0000 7fb73ffff640 1 -- 172.21.15.90:0/2745270019 learned_addr learned my addr 172.21.15.90:0/2745270019 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:27.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.215+0000 7fb744ef3640 1 --2- 172.21.15.90:0/2745270019 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cde03664d0 0x55cde0728190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.215+0000 7fb7456f4640 1 --2- 172.21.15.90:0/2745270019 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cde038f390 0x55cde0735510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.215+0000 7fb73ffff640 1 -- 172.21.15.90:0/2745270019 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cde038f390 msgr2=0x55cde0735510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:27.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.215+0000 7fb73ffff640 1 --2- 172.21.15.90:0/2745270019 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cde038f390 0x55cde0735510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.215+0000 7fb73ffff640 1 -- 172.21.15.90:0/2745270019 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cde03664d0 msgr2=0x55cde0728190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:27.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.215+0000 7fb73ffff640 1 --2- 172.21.15.90:0/2745270019 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cde03664d0 0x55cde0728190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.215+0000 7fb73ffff640 1 -- 172.21.15.90:0/2745270019 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cde074e210 con 0x55cde036e6b0 2024-04-25T11:46:27.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.215+0000 7fb744ef3640 1 --2- 172.21.15.90:0/2745270019 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cde03664d0 0x55cde0728190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:46:27.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.215+0000 7fb73ffff640 1 --2- 172.21.15.90:0/2745270019 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cde036e6b0 0x55cde0746e70 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fb72c009380 tx=0x7fb72c010ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=746be5f65ddf5c6b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:27.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.216+0000 7fb72a7fc640 1 -- 172.21.15.90:0/2745270019 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb72c02b070 con 0x55cde036e6b0 2024-04-25T11:46:27.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.216+0000 7fb72a7fc640 1 -- 172.21.15.90:0/2745270019 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb72c020040 con 0x55cde036e6b0 2024-04-25T11:46:27.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.216+0000 7fb72a7fc640 1 -- 172.21.15.90:0/2745270019 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb72c01f030 con 0x55cde036e6b0 2024-04-25T11:46:27.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.216+0000 7fb74e009740 1 -- 172.21.15.90:0/2745270019 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cde036e6b0 msgr2=0x55cde0746e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:27.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.216+0000 7fb74e009740 1 --2- 172.21.15.90:0/2745270019 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cde036e6b0 0x55cde0746e70 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fb72c009380 tx=0x7fb72c010ae0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.217+0000 7fb74e009740 1 -- 172.21.15.90:0/2745270019 shutdown_connections 2024-04-25T11:46:27.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.217+0000 7fb74e009740 1 --2- 172.21.15.90:0/2745270019 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cde038f390 0x55cde0735510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.217+0000 7fb74e009740 1 --2- 172.21.15.90:0/2745270019 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cde036e6b0 0x55cde0746e70 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.217+0000 7fb74e009740 1 --2- 172.21.15.90:0/2745270019 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cde03664d0 0x55cde0728190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.217+0000 7fb74e009740 1 -- 172.21.15.90:0/2745270019 >> 172.21.15.90:0/2745270019 conn(0x55cde03834e0 msgr2=0x55cde0736280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.217+0000 7fb74e009740 1 -- 172.21.15.90:0/2745270019 shutdown_connections 2024-04-25T11:46:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.217+0000 7fb74e009740 1 -- 172.21.15.90:0/2745270019 wait complete. 2024-04-25T11:46:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.217+0000 7fb74e009740 1 -- start start 2024-04-25T11:46:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.217+0000 7fb74e009740 1 Processor -- start 2024-04-25T11:46:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.217+0000 7fb74e009740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cde03664d0 0x55cde0743610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.217+0000 7fb74e009740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cde036e6b0 0x55cde073e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.217+0000 7fb74e009740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cde038f390 0x55cde073e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.217+0000 7fb74e009740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb7401aef50 con 0x55cde03664d0 2024-04-25T11:46:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.217+0000 7fb74e009740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb7401ae880 con 0x55cde038f390 2024-04-25T11:46:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.217+0000 7fb74e009740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb7401abae0 con 0x55cde036e6b0 2024-04-25T11:46:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.218+0000 7fb74e009740 10 client.0 _ll_register_callbacks cb 0x7fb738011f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.218+0000 7fb744ef3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cde03664d0 0x55cde0743610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.218+0000 7fb744ef3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cde03664d0 0x55cde0743610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48350/0 (socket says 172.21.15.90:48350) 2024-04-25T11:46:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.218+0000 7fb744ef3640 1 -- 172.21.15.90:0/4095683919 learned_addr learned my addr 172.21.15.90:0/4095683919 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.218+0000 7fb7456f4640 1 --2- 172.21.15.90:0/4095683919 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cde038f390 0x55cde073e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.218+0000 7fb73ffff640 1 --2- 172.21.15.90:0/4095683919 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cde036e6b0 0x55cde073e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.218+0000 7fb744ef3640 1 -- 172.21.15.90:0/4095683919 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cde036e6b0 msgr2=0x55cde073e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.218+0000 7fb744ef3640 1 --2- 172.21.15.90:0/4095683919 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cde036e6b0 0x55cde073e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.218+0000 7fb744ef3640 1 -- 172.21.15.90:0/4095683919 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cde038f390 msgr2=0x55cde073e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.218+0000 7fb744ef3640 1 --2- 172.21.15.90:0/4095683919 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cde038f390 0x55cde073e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.218+0000 7fb744ef3640 1 -- 172.21.15.90:0/4095683919 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cde073ed30 con 0x55cde03664d0 2024-04-25T11:46:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.218+0000 7fb744ef3640 1 --2- 172.21.15.90:0/4095683919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cde03664d0 0x55cde0743610 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fb734032af0 tx=0x7fb734009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c091b32a38f0f48 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.218+0000 7fb71ffff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:46:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.218+0000 7fb71ffff640 1 -- 172.21.15.90:0/4095683919 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb734049040 con 0x55cde03664d0 2024-04-25T11:46:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.218+0000 7fb71ffff640 1 -- 172.21.15.90:0/4095683919 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7fb734102c30 con 0x55cde03664d0 2024-04-25T11:46:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.218+0000 7fb74e009740 1 -- 172.21.15.90:0/4095683919 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb72c020040 con 0x55cde03664d0 2024-04-25T11:46:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.218+0000 7fb71ffff640 1 -- 172.21.15.90:0/4095683919 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb7340c6d60 con 0x55cde03664d0 2024-04-25T11:46:27.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.219+0000 7fb74e009740 10 client.19284 fetch_fsmap learned FSMap version 28 2024-04-25T11:46:27.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.219+0000 7fb74e009740 1 -- 172.21.15.90:0/4095683919 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cde07b4dd0 con 0x55cde03664d0 2024-04-25T11:46:27.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.219+0000 7fb71ffff640 1 -- 172.21.15.90:0/4095683919 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb72c020040 con 0x55cde03664d0 2024-04-25T11:46:27.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.220+0000 7fb71ffff640 1 -- 172.21.15.90:0/4095683919 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fb7340cff10 con 0x55cde03664d0 2024-04-25T11:46:27.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.220+0000 7fb74e009740 10 client.19284 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:46:27.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.220+0000 7fb74e009740 10 client.19284 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:46:27.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.220+0000 7fb74e009740 20 client.19284 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:27.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.220+0000 7fb74e009740 1 --2- 172.21.15.90:0/4095683919 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55cde03233a0 0x55cde07b7360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.220+0000 7fb74e009740 4 client.19284 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:46:27.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.220+0000 7fb74e009740 1 -- 172.21.15.90:0/4095683919 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cde069d800 con 0x55cde03233a0 2024-04-25T11:46:27.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.220+0000 7fb73ffff640 1 --2- 172.21.15.90:0/4095683919 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55cde03233a0 0x55cde07b7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.220+0000 7fb71ffff640 10 client.19284 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:46:27.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.220+0000 7fb73ffff640 1 --2- 172.21.15.90:0/4095683919 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55cde03233a0 0x55cde07b7360 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cf758adc9e0c5e87 server_cookie=b997254ffdcb84af in_seq=0 out_seq=0 2024-04-25T11:46:27.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.221+0000 7fb71ffff640 1 -- 172.21.15.90:0/4095683919 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cde069d800 con 0x55cde03233a0 2024-04-25T11:46:27.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.221+0000 7fb71ffff640 10 client.19284 handle_command_reply: tid=0 2024-04-25T11:46:27.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.221+0000 7fb74e009740 1 client.19284 shutdown 2024-04-25T11:46:27.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.221+0000 7fb74e009740 1 -- 172.21.15.90:0/4095683919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cde03664d0 msgr2=0x55cde0743610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:27.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.221+0000 7fb74e009740 1 --2- 172.21.15.90:0/4095683919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cde03664d0 0x55cde0743610 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fb734032af0 tx=0x7fb734009680 comp rx=0 tx=0).stop 2024-04-25T11:46:27.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.221+0000 7fb74e009740 1 -- 172.21.15.90:0/4095683919 shutdown_connections 2024-04-25T11:46:27.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.221+0000 7fb74e009740 1 --2- 172.21.15.90:0/4095683919 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x55cde03233a0 0x55cde07b7360 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.221+0000 7fb74e009740 1 --2- 172.21.15.90:0/4095683919 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55cde038f390 0x55cde073e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.221+0000 7fb71ffff640 0 client.19284 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:46:27.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.221+0000 7fb74e009740 1 --2- 172.21.15.90:0/4095683919 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cde036e6b0 0x55cde073e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.221+0000 7fb74e009740 1 --2- 172.21.15.90:0/4095683919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cde03664d0 0x55cde0743610 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.221+0000 7fb74e009740 1 -- 172.21.15.90:0/4095683919 >> 172.21.15.90:0/4095683919 conn(0x55cde03834e0 msgr2=0x55cde073c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:27.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.221+0000 7fb74e009740 1 -- 172.21.15.90:0/4095683919 shutdown_connections 2024-04-25T11:46:27.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.221+0000 7fb74e009740 1 -- 172.21.15.90:0/4095683919 wait complete. 2024-04-25T11:46:27.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.221+0000 7fb74e009740 20 client.19284 trim_cache size 0 max 16384 2024-04-25T11:46:27.211 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:46:27.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.222+0000 7fb747157640 1 -- 172.21.15.90:0/1672833961 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb7300afc60 msgr2=0x7fb7300b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:27.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.222+0000 7fb747157640 1 --2- 172.21.15.90:0/1672833961 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb7300afc60 0x7fb7300b2170 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x55cde072d980 tx=0x7fb738004ab0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.222+0000 7fb747157640 1 -- 172.21.15.90:0/1672833961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb740104c60 msgr2=0x7fb74019a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:27.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.222+0000 7fb747157640 1 --2- 172.21.15.90:0/1672833961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb740104c60 0x7fb74019a6d0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fb734005940 tx=0x7fb7340040c0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.222+0000 7fb747157640 1 -- 172.21.15.90:0/1672833961 shutdown_connections 2024-04-25T11:46:27.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.222+0000 7fb747157640 1 --2- 172.21.15.90:0/1672833961 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb7300afc60 0x7fb7300b2170 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.222+0000 7fb747157640 1 --2- 172.21.15.90:0/1672833961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb74010e070 0x7fb74019eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.222+0000 7fb747157640 1 --2- 172.21.15.90:0/1672833961 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb74010b6c0 0x7fb74019ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.222+0000 7fb747157640 1 --2- 172.21.15.90:0/1672833961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb740104c60 0x7fb74019a6d0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.222+0000 7fb747157640 1 -- 172.21.15.90:0/1672833961 >> 172.21.15.90:0/1672833961 conn(0x7fb7400fe260 msgr2=0x7fb74010e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:27.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.222+0000 7fb747157640 1 -- 172.21.15.90:0/1672833961 shutdown_connections 2024-04-25T11:46:27.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.222+0000 7fb747157640 1 -- 172.21.15.90:0/1672833961 wait complete. 2024-04-25T11:46:27.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:27 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/4151648309' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:46:27.218 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:27 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:46:27.219 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T11:46:27.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:27 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/4151648309' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:46:27.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:27 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:46:27.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:27 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/4151648309' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:46:27.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:27 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:46:27.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.355+0000 7f995a1ce640 1 -- 172.21.15.90:0/561147936 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f995410d3a0 msgr2=0x7f995410f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:27.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.355+0000 7f995a1ce640 1 --2- 172.21.15.90:0/561147936 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f995410d3a0 0x7f995410f7c0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f994c009ca0 tx=0x7f994c02a8c0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.355+0000 7f995a1ce640 1 -- 172.21.15.90:0/561147936 shutdown_connections 2024-04-25T11:46:27.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.355+0000 7f995a1ce640 1 --2- 172.21.15.90:0/561147936 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f995410d3a0 0x7f995410f7c0 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.355+0000 7f995a1ce640 1 --2- 172.21.15.90:0/561147936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9954102b60 0x7f9954102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.355+0000 7f995a1ce640 1 --2- 172.21.15.90:0/561147936 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9954108a40 0x7f9954102620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.355+0000 7f995a1ce640 1 -- 172.21.15.90:0/561147936 >> 172.21.15.90:0/561147936 conn(0x7f99540fe260 msgr2=0x7f9954100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:27.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.355+0000 7f995a1ce640 1 -- 172.21.15.90:0/561147936 shutdown_connections 2024-04-25T11:46:27.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.355+0000 7f995a1ce640 1 -- 172.21.15.90:0/561147936 wait complete. 2024-04-25T11:46:27.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.356+0000 7f995a1ce640 1 Processor -- start 2024-04-25T11:46:27.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.356+0000 7f995a1ce640 1 -- start start 2024-04-25T11:46:27.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.356+0000 7f995a1ce640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9954102b60 0x7f995419a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.356+0000 7f995a1ce640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9954108a40 0x7f995419ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.356+0000 7f995a1ce640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f995410d3a0 0x7f995419ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.356+0000 7f995a1ce640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9954112c00 con 0x7f9954108a40 2024-04-25T11:46:27.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.356+0000 7f995a1ce640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9954112a80 con 0x7f995410d3a0 2024-04-25T11:46:27.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.356+0000 7f995a1ce640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9954112d80 con 0x7f9954102b60 2024-04-25T11:46:27.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.356+0000 7f9953fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f995410d3a0 0x7f995419ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.356+0000 7f99537fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9954102b60 0x7f995419a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.356+0000 7f9952ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9954108a40 0x7f995419ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.356+0000 7f9953fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f995410d3a0 0x7f995419ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:34076/0 (socket says 172.21.15.90:34076) 2024-04-25T11:46:27.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.356+0000 7f9953fff640 1 -- 172.21.15.90:0/1436020902 learned_addr learned my addr 172.21.15.90:0/1436020902 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:27.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.357+0000 7f9953fff640 1 -- 172.21.15.90:0/1436020902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9954102b60 msgr2=0x7f995419a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:27.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.357+0000 7f9953fff640 1 --2- 172.21.15.90:0/1436020902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9954102b60 0x7f995419a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.357+0000 7f9953fff640 1 -- 172.21.15.90:0/1436020902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9954108a40 msgr2=0x7f995419ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:27.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.357+0000 7f9953fff640 1 --2- 172.21.15.90:0/1436020902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9954108a40 0x7f995419ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.357+0000 7f9953fff640 1 -- 172.21.15.90:0/1436020902 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f995419f550 con 0x7f995410d3a0 2024-04-25T11:46:27.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.357+0000 7f9952ffd640 1 --2- 172.21.15.90:0/1436020902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9954108a40 0x7f995419ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:46:27.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.357+0000 7f9953fff640 1 --2- 172.21.15.90:0/1436020902 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f995410d3a0 0x7f995419ef10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f994c005cc0 tx=0x7f994c031700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:27.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.357+0000 7f9950ff9640 1 -- 172.21.15.90:0/1436020902 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f994c003b70 con 0x7f995410d3a0 2024-04-25T11:46:27.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.358+0000 7f9950ff9640 1 -- 172.21.15.90:0/1436020902 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f994c0318f0 con 0x7f995410d3a0 2024-04-25T11:46:27.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.358+0000 7f9950ff9640 1 -- 172.21.15.90:0/1436020902 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f994c02f950 con 0x7f995410d3a0 2024-04-25T11:46:27.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.358+0000 7f995a1ce640 1 -- 172.21.15.90:0/1436020902 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f995419f7e0 con 0x7f995410d3a0 2024-04-25T11:46:27.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.358+0000 7f995a1ce640 1 -- 172.21.15.90:0/1436020902 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f995419fce0 con 0x7f995410d3a0 2024-04-25T11:46:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.360+0000 7f9961081740 1 Processor -- start 2024-04-25T11:46:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.360+0000 7f9950ff9640 1 -- 172.21.15.90:0/1436020902 <== mon.2 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f994c02fb50 con 0x7f995410d3a0 2024-04-25T11:46:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.360+0000 7f9961081740 1 -- start start 2024-04-25T11:46:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.360+0000 7f9961081740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c198dcf390 0x55c199170810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.360+0000 7f9961081740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c198da64d0 0x55c199170d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.360+0000 7f9961081740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c198dae6b0 0x55c19917d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.361+0000 7f9961081740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c1990d5d80 con 0x55c198dae6b0 2024-04-25T11:46:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.361+0000 7f9961081740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c19908fa20 con 0x55c198dcf390 2024-04-25T11:46:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.361+0000 7f9961081740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c1990d1d40 con 0x55c198da64d0 2024-04-25T11:46:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.361+0000 7f9952ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c198dae6b0 0x55c19917d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.361+0000 7f9952ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c198dae6b0 0x55c19917d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48386/0 (socket says 172.21.15.90:48386) 2024-04-25T11:46:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.361+0000 7f99537fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c198da64d0 0x55c199170d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.361+0000 7f9952ffd640 1 -- 172.21.15.90:0/1772886039 learned_addr learned my addr 172.21.15.90:0/1772886039 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.361+0000 7f9952ffd640 1 --2- 172.21.15.90:0/1772886039 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c198dcf390 0x55c199170810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.361+0000 7f9950ff9640 1 --2- 172.21.15.90:0/1436020902 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f993c0afc60 0x7f993c0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.361+0000 7f99537fe640 1 -- 172.21.15.90:0/1772886039 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c198dcf390 msgr2=0x55c199170810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.361+0000 7f99537fe640 1 --2- 172.21.15.90:0/1772886039 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c198dcf390 0x55c199170810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.361+0000 7f99537fe640 1 -- 172.21.15.90:0/1772886039 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c198dae6b0 msgr2=0x55c19917d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.361+0000 7f99537fe640 1 --2- 172.21.15.90:0/1772886039 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c198dae6b0 0x55c19917d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.361+0000 7f99537fe640 1 -- 172.21.15.90:0/1772886039 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c19909ac30 con 0x55c198da64d0 2024-04-25T11:46:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.361+0000 7f9953fff640 1 --2- 172.21.15.90:0/1436020902 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f993c0afc60 0x7f993c0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.361+0000 7f9950ff9640 1 -- 172.21.15.90:0/1436020902 <== mon.2 v2:172.21.15.138:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7f994c1003b0 con 0x7f995410d3a0 2024-04-25T11:46:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.361+0000 7f99537fe640 1 --2- 172.21.15.90:0/1772886039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c198da64d0 0x55c199170d50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9948009ac0 tx=0x7f9948009f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f4be050bfba674c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.361+0000 7f9939ffb640 1 -- 172.21.15.90:0/1772886039 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9948011da0 con 0x55c198da64d0 2024-04-25T11:46:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.362+0000 7f9939ffb640 1 -- 172.21.15.90:0/1772886039 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9948012900 con 0x55c198da64d0 2024-04-25T11:46:27.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.362+0000 7f9939ffb640 1 -- 172.21.15.90:0/1772886039 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f994800f8e0 con 0x55c198da64d0 2024-04-25T11:46:27.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.362+0000 7f9961081740 1 -- 172.21.15.90:0/1772886039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c198da64d0 msgr2=0x55c199170d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:27.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.362+0000 7f9953fff640 1 --2- 172.21.15.90:0/1436020902 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f993c0afc60 0x7f993c0b2170 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f994c0c5820 tx=0x7f994c0970b0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:27.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.362+0000 7f9961081740 1 --2- 172.21.15.90:0/1772886039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c198da64d0 0x55c199170d50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9948009ac0 tx=0x7f9948009f80 comp rx=0 tx=0).stop 2024-04-25T11:46:27.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.363+0000 7f9961081740 1 -- 172.21.15.90:0/1772886039 shutdown_connections 2024-04-25T11:46:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.363+0000 7f9961081740 1 --2- 172.21.15.90:0/1772886039 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c198dae6b0 0x55c19917d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.363+0000 7f9961081740 1 --2- 172.21.15.90:0/1772886039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c198da64d0 0x55c199170d50 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.363+0000 7f9961081740 1 --2- 172.21.15.90:0/1772886039 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c198dcf390 0x55c199170810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.363+0000 7f9961081740 1 -- 172.21.15.90:0/1772886039 >> 172.21.15.90:0/1772886039 conn(0x55c198dc34e0 msgr2=0x55c19916cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.363+0000 7f9961081740 1 -- 172.21.15.90:0/1772886039 shutdown_connections 2024-04-25T11:46:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.363+0000 7f9961081740 1 -- 172.21.15.90:0/1772886039 wait complete. 2024-04-25T11:46:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.363+0000 7f9961081740 1 -- start start 2024-04-25T11:46:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.363+0000 7f9961081740 1 Processor -- start 2024-04-25T11:46:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.363+0000 7f9961081740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c198da64d0 0x55c19917b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.363+0000 7f9961081740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c198dae6b0 0x55c19917b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.364+0000 7f9961081740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c198dcf390 0x55c199187210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.364+0000 7f9961081740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c19917dd90 con 0x55c198dae6b0 2024-04-25T11:46:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.364+0000 7f9961081740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55c19905cdc0 con 0x55c198da64d0 2024-04-25T11:46:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.364+0000 7f9961081740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c19917df10 con 0x55c198dcf390 2024-04-25T11:46:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.364+0000 7f9961081740 10 client.0 _ll_register_callbacks cb 0x55c199078600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.364+0000 7f99537fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c198dae6b0 0x55c19917b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.364+0000 7f9952ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c198da64d0 0x55c19917b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.364+0000 7f99537fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c198dae6b0 0x55c19917b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48392/0 (socket says 172.21.15.90:48392) 2024-04-25T11:46:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.364+0000 7f99537fe640 1 -- 172.21.15.90:0/2009281551 learned_addr learned my addr 172.21.15.90:0/2009281551 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.364+0000 7f9952ffd640 1 --2- 172.21.15.90:0/2009281551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c198dcf390 0x55c199187210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.364+0000 7f99537fe640 1 -- 172.21.15.90:0/2009281551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c198dcf390 msgr2=0x55c199187210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.364+0000 7f99537fe640 1 --2- 172.21.15.90:0/2009281551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c198dcf390 0x55c199187210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.364+0000 7f99537fe640 1 -- 172.21.15.90:0/2009281551 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c198da64d0 msgr2=0x55c19917b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.364+0000 7f99537fe640 1 --2- 172.21.15.90:0/2009281551 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c198da64d0 0x55c19917b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.364+0000 7f99537fe640 1 -- 172.21.15.90:0/2009281551 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c198de73f0 con 0x55c198dae6b0 2024-04-25T11:46:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.364+0000 7f9952ffd640 1 --2- 172.21.15.90:0/2009281551 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c198da64d0 0x55c19917b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:46:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.364+0000 7f99537fe640 1 --2- 172.21.15.90:0/2009281551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c198dae6b0 0x55c19917b800 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f994801b000 tx=0x7f9948012680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61a70f2b370f0fd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.364+0000 7f9932ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:46:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.364+0000 7f9932ffd640 1 -- 172.21.15.90:0/2009281551 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9948028070 con 0x55c198dae6b0 2024-04-25T11:46:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.364+0000 7f9932ffd640 1 -- 172.21.15.90:0/2009281551 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7f9948005040 con 0x55c198dae6b0 2024-04-25T11:46:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.364+0000 7f9961081740 1 -- 172.21.15.90:0/2009281551 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9948012900 con 0x55c198dae6b0 2024-04-25T11:46:27.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.365+0000 7f9932ffd640 1 -- 172.21.15.90:0/2009281551 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9948034050 con 0x55c198dae6b0 2024-04-25T11:46:27.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.365+0000 7f9961081740 10 client.19296 fetch_fsmap learned FSMap version 28 2024-04-25T11:46:27.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.365+0000 7f9961081740 1 -- 172.21.15.90:0/2009281551 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c198eb17d0 con 0x55c198dae6b0 2024-04-25T11:46:27.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.365+0000 7f9932ffd640 1 -- 172.21.15.90:0/2009281551 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9948012900 con 0x55c198dae6b0 2024-04-25T11:46:27.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.365+0000 7f9932ffd640 1 -- 172.21.15.90:0/2009281551 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f9948004aa0 con 0x55c198dae6b0 2024-04-25T11:46:27.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.365+0000 7f9961081740 10 client.19296 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:46:27.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.365+0000 7f9961081740 10 client.19296 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:46:27.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.365+0000 7f9961081740 20 client.19296 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:27.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.365+0000 7f9961081740 1 --2- 172.21.15.90:0/2009281551 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c198d8df60 0x55c19918d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.365+0000 7f9961081740 4 client.19296 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:46:27.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.365+0000 7f9961081740 1 -- 172.21.15.90:0/2009281551 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1990e3990 con 0x55c198d8df60 2024-04-25T11:46:27.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.365+0000 7f9952ffd640 1 --2- 172.21.15.90:0/2009281551 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c198d8df60 0x55c19918d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.366+0000 7f9932ffd640 10 client.19296 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:27.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.366+0000 7f9952ffd640 1 --2- 172.21.15.90:0/2009281551 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c198d8df60 0x55c19918d590 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4cad6f10ea341e5 server_cookie=88102c161b3e5778 in_seq=0 out_seq=0 2024-04-25T11:46:27.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.367+0000 7f9932ffd640 1 -- 172.21.15.90:0/2009281551 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c1990e3990 con 0x55c198d8df60 2024-04-25T11:46:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.367+0000 7f9932ffd640 10 client.19296 handle_command_reply: tid=0 2024-04-25T11:46:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.367+0000 7f9961081740 1 client.19296 shutdown 2024-04-25T11:46:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.367+0000 7f9961081740 1 -- 172.21.15.90:0/2009281551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c198dae6b0 msgr2=0x55c19917b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.367+0000 7f9961081740 1 --2- 172.21.15.90:0/2009281551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c198dae6b0 0x55c19917b800 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f994801b000 tx=0x7f9948012680 comp rx=0 tx=0).stop 2024-04-25T11:46:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.367+0000 7f9961081740 1 -- 172.21.15.90:0/2009281551 shutdown_connections 2024-04-25T11:46:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.367+0000 7f9961081740 1 --2- 172.21.15.90:0/2009281551 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c198d8df60 0x55c19918d590 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.367+0000 7f9961081740 1 --2- 172.21.15.90:0/2009281551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c198dcf390 0x55c199187210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.367+0000 7f9961081740 1 --2- 172.21.15.90:0/2009281551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c198dae6b0 0x55c19917b800 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.367+0000 7f9932ffd640 0 client.19296 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.367+0000 7f9961081740 1 --2- 172.21.15.90:0/2009281551 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c198da64d0 0x55c19917b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.367+0000 7f9961081740 1 -- 172.21.15.90:0/2009281551 >> 172.21.15.90:0/2009281551 conn(0x55c198dc34e0 msgr2=0x55c199172530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.367+0000 7f9961081740 1 -- 172.21.15.90:0/2009281551 shutdown_connections 2024-04-25T11:46:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.367+0000 7f9961081740 1 -- 172.21.15.90:0/2009281551 wait complete. 2024-04-25T11:46:27.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.367+0000 7f9961081740 20 client.19296 trim_cache size 0 max 16384 2024-04-25T11:46:27.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.381+0000 7f9961081740 1 Processor -- start 2024-04-25T11:46:27.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.381+0000 7f9961081740 1 -- start start 2024-04-25T11:46:27.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.381+0000 7f9961081740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c198da64d0 0x55c199187200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.381+0000 7f9961081740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c198dae6b0 0x55c199170d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.381+0000 7f9961081740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c198dcf390 0x55c199175430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.381+0000 7f9961081740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f99541afc80 con 0x55c198dcf390 2024-04-25T11:46:27.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.381+0000 7f9961081740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f99541b17c0 con 0x55c198da64d0 2024-04-25T11:46:27.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.381+0000 7f9961081740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f99541b0350 con 0x55c198dae6b0 2024-04-25T11:46:27.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.382+0000 7f9952ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c198da64d0 0x55c199187200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.382+0000 7f9952ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c198da64d0 0x55c199187200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:34088/0 (socket says 172.21.15.90:34088) 2024-04-25T11:46:27.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.382+0000 7f9952ffd640 1 -- 172.21.15.90:0/1097808428 learned_addr learned my addr 172.21.15.90:0/1097808428 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:27.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.382+0000 7f99537fe640 1 --2- 172.21.15.90:0/1097808428 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c198dae6b0 0x55c199170d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.382+0000 7f9952ffd640 1 --2- 172.21.15.90:0/1097808428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c198dcf390 0x55c199175430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.382+0000 7f9952ffd640 1 -- 172.21.15.90:0/1097808428 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c198dae6b0 msgr2=0x55c199170d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:27.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.382+0000 7f9952ffd640 1 --2- 172.21.15.90:0/1097808428 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c198dae6b0 0x55c199170d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.382+0000 7f9952ffd640 1 -- 172.21.15.90:0/1097808428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c198dcf390 msgr2=0x55c199175430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:27.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.382+0000 7f9952ffd640 1 --2- 172.21.15.90:0/1097808428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c198dcf390 0x55c199175430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.382+0000 7f9952ffd640 1 -- 172.21.15.90:0/1097808428 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c19916db60 con 0x55c198da64d0 2024-04-25T11:46:27.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.382+0000 7f9952ffd640 1 --2- 172.21.15.90:0/1097808428 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c198da64d0 0x55c199187200 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x55c19917e020 tx=0x7f9944009ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd25fe8b9e40496 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:27.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.382+0000 7f99397fa640 1 -- 172.21.15.90:0/1097808428 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9944005020 con 0x55c198da64d0 2024-04-25T11:46:27.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.382+0000 7f99397fa640 1 -- 172.21.15.90:0/1097808428 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9944005670 con 0x55c198da64d0 2024-04-25T11:46:27.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.382+0000 7f99397fa640 1 -- 172.21.15.90:0/1097808428 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9944004dd0 con 0x55c198da64d0 2024-04-25T11:46:27.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.383+0000 7f9961081740 1 -- 172.21.15.90:0/1097808428 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c198da64d0 msgr2=0x55c199187200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:27.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.383+0000 7f9961081740 1 --2- 172.21.15.90:0/1097808428 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c198da64d0 0x55c199187200 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x55c19917e020 tx=0x7f9944009ed0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.383+0000 7f9961081740 1 -- 172.21.15.90:0/1097808428 shutdown_connections 2024-04-25T11:46:27.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.383+0000 7f9961081740 1 --2- 172.21.15.90:0/1097808428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c198dcf390 0x55c199175430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.383+0000 7f9961081740 1 --2- 172.21.15.90:0/1097808428 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c198dae6b0 0x55c199170d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.383+0000 7f9961081740 1 --2- 172.21.15.90:0/1097808428 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c198da64d0 0x55c199187200 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.383+0000 7f9961081740 1 -- 172.21.15.90:0/1097808428 >> 172.21.15.90:0/1097808428 conn(0x55c198dc34e0 msgr2=0x55c199170600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:27.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.383+0000 7f9961081740 1 -- 172.21.15.90:0/1097808428 shutdown_connections 2024-04-25T11:46:27.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.383+0000 7f9961081740 1 -- 172.21.15.90:0/1097808428 wait complete. 2024-04-25T11:46:27.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.384+0000 7f9961081740 1 -- start start 2024-04-25T11:46:27.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.384+0000 7f9961081740 1 Processor -- start 2024-04-25T11:46:27.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.384+0000 7f9961081740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c198da64d0 0x55c19918d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.384+0000 7f9961081740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c198dae6b0 0x55c19918da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.384+0000 7f9961081740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c198dcf390 0x55c1991f36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.384+0000 7f9961081740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f99541b10f0 con 0x55c198dae6b0 2024-04-25T11:46:27.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.384+0000 7f9961081740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f99541b0a20 con 0x55c198da64d0 2024-04-25T11:46:27.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.384+0000 7f9961081740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f99541adc80 con 0x55c198dcf390 2024-04-25T11:46:27.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.384+0000 7f9961081740 10 client.0 _ll_register_callbacks cb 0x7f9944009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:27.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.384+0000 7f99537fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c198dae6b0 0x55c19918da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.384+0000 7f99537fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c198dae6b0 0x55c19918da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48398/0 (socket says 172.21.15.90:48398) 2024-04-25T11:46:27.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.384+0000 7f99537fe640 1 -- 172.21.15.90:0/2559279998 learned_addr learned my addr 172.21.15.90:0/2559279998 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:27.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.384+0000 7f9952ffd640 1 --2- 172.21.15.90:0/2559279998 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c198da64d0 0x55c19918d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.384+0000 7f9952ffd640 1 --2- 172.21.15.90:0/2559279998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c198dcf390 0x55c1991f36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.385+0000 7f99537fe640 1 -- 172.21.15.90:0/2559279998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c198dcf390 msgr2=0x55c1991f36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:27.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.385+0000 7f99537fe640 1 --2- 172.21.15.90:0/2559279998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c198dcf390 0x55c1991f36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.385+0000 7f99537fe640 1 -- 172.21.15.90:0/2559279998 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c198da64d0 msgr2=0x55c19918d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:27.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.385+0000 7f99537fe640 1 --2- 172.21.15.90:0/2559279998 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c198da64d0 0x55c19918d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.385+0000 7f99537fe640 1 -- 172.21.15.90:0/2559279998 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c19918e2d0 con 0x55c198dae6b0 2024-04-25T11:46:27.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.385+0000 7f99537fe640 1 --2- 172.21.15.90:0/2559279998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c198dae6b0 0x55c19918da80 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f994800f6b0 tx=0x7f994800fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=921a59f6b7a9c7cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:27.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.385+0000 7f9932ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:46:27.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.385+0000 7f9932ffd640 1 -- 172.21.15.90:0/2559279998 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f994803d060 con 0x55c198dae6b0 2024-04-25T11:46:27.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.385+0000 7f9961081740 1 -- 172.21.15.90:0/2559279998 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9944005670 con 0x55c198dae6b0 2024-04-25T11:46:27.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.385+0000 7f9932ffd640 1 -- 172.21.15.90:0/2559279998 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7f994800fdd0 con 0x55c198dae6b0 2024-04-25T11:46:27.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.385+0000 7f9932ffd640 1 -- 172.21.15.90:0/2559279998 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9948044050 con 0x55c198dae6b0 2024-04-25T11:46:27.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.385+0000 7f9961081740 10 client.19302 fetch_fsmap learned FSMap version 28 2024-04-25T11:46:27.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.385+0000 7f9961081740 1 -- 172.21.15.90:0/2559279998 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c1991f4c00 con 0x55c198dae6b0 2024-04-25T11:46:27.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.385+0000 7f9932ffd640 1 -- 172.21.15.90:0/2559279998 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9944005670 con 0x55c198dae6b0 2024-04-25T11:46:27.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.386+0000 7f9932ffd640 1 -- 172.21.15.90:0/2559279998 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f9948004fc0 con 0x55c198dae6b0 2024-04-25T11:46:27.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.386+0000 7f9961081740 10 client.19302 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:46:27.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.386+0000 7f9961081740 10 client.19302 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:46:27.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.386+0000 7f9961081740 20 client.19302 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:27.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.386+0000 7f9961081740 1 --2- 172.21.15.90:0/2559279998 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c198d633a0 0x55c1991f7360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:27.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.386+0000 7f9961081740 4 client.19302 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:46:27.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.386+0000 7f9961081740 1 -- 172.21.15.90:0/2559279998 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c1990dd800 con 0x55c198d633a0 2024-04-25T11:46:27.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.386+0000 7f9952ffd640 1 --2- 172.21.15.90:0/2559279998 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c198d633a0 0x55c1991f7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:27.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.386+0000 7f9932ffd640 10 client.19302 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:27.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.386+0000 7f9952ffd640 1 --2- 172.21.15.90:0/2559279998 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c198d633a0 0x55c1991f7360 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aaf29e0f110619e0 server_cookie=3060638b4ecdddbf in_seq=0 out_seq=0 2024-04-25T11:46:27.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.387+0000 7f9932ffd640 1 -- 172.21.15.90:0/2559279998 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c1990dd800 con 0x55c198d633a0 2024-04-25T11:46:27.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.387+0000 7f9932ffd640 10 client.19302 handle_command_reply: tid=0 2024-04-25T11:46:27.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.387+0000 7f9961081740 1 client.19302 shutdown 2024-04-25T11:46:27.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.387+0000 7f9961081740 1 -- 172.21.15.90:0/2559279998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c198dae6b0 msgr2=0x55c19918da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:27.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.387+0000 7f9961081740 1 --2- 172.21.15.90:0/2559279998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c198dae6b0 0x55c19918da80 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f994800f6b0 tx=0x7f994800fae0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.387+0000 7f9961081740 1 -- 172.21.15.90:0/2559279998 shutdown_connections 2024-04-25T11:46:27.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.387+0000 7f9961081740 1 --2- 172.21.15.90:0/2559279998 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55c198d633a0 0x55c1991f7360 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.387+0000 7f9961081740 1 --2- 172.21.15.90:0/2559279998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c198dcf390 0x55c1991f36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.387+0000 7f9961081740 1 --2- 172.21.15.90:0/2559279998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c198dae6b0 0x55c19918da80 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.387+0000 7f9932ffd640 0 client.19302 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:27.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.387+0000 7f9961081740 1 --2- 172.21.15.90:0/2559279998 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55c198da64d0 0x55c19918d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.387+0000 7f9961081740 1 -- 172.21.15.90:0/2559279998 >> 172.21.15.90:0/2559279998 conn(0x55c198dc34e0 msgr2=0x55c199179540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:27.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.387+0000 7f9961081740 1 -- 172.21.15.90:0/2559279998 shutdown_connections 2024-04-25T11:46:27.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.387+0000 7f9961081740 1 -- 172.21.15.90:0/2559279998 wait complete. 2024-04-25T11:46:27.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.387+0000 7f9961081740 20 client.19302 trim_cache size 0 max 16384 2024-04-25T11:46:27.377 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:46:27.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.387+0000 7f995a1ce640 1 -- 172.21.15.90:0/1436020902 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f993c0afc60 msgr2=0x7f993c0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:27.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.387+0000 7f995a1ce640 1 --2- 172.21.15.90:0/1436020902 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f993c0afc60 0x7f993c0b2170 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f994c0c5820 tx=0x7f994c0970b0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.387+0000 7f995a1ce640 1 -- 172.21.15.90:0/1436020902 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f995410d3a0 msgr2=0x7f995419ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:27.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.387+0000 7f995a1ce640 1 --2- 172.21.15.90:0/1436020902 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f995410d3a0 0x7f995419ef10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f994c005cc0 tx=0x7f994c031700 comp rx=0 tx=0).stop 2024-04-25T11:46:27.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.388+0000 7f995a1ce640 1 -- 172.21.15.90:0/1436020902 shutdown_connections 2024-04-25T11:46:27.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.388+0000 7f995a1ce640 1 --2- 172.21.15.90:0/1436020902 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f993c0afc60 0x7f993c0b2170 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.388+0000 7f995a1ce640 1 --2- 172.21.15.90:0/1436020902 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f995410d3a0 0x7f995419ef10 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.388+0000 7f995a1ce640 1 --2- 172.21.15.90:0/1436020902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9954108a40 0x7f995419ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.388+0000 7f995a1ce640 1 --2- 172.21.15.90:0/1436020902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9954102b60 0x7f995419a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:27.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.388+0000 7f995a1ce640 1 -- 172.21.15.90:0/1436020902 >> 172.21.15.90:0/1436020902 conn(0x7f99540fe260 msgr2=0x7f99541009d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:27.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.388+0000 7f995a1ce640 1 -- 172.21.15.90:0/1436020902 shutdown_connections 2024-04-25T11:46:27.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:27.388+0000 7f995a1ce640 1 -- 172.21.15.90:0/1436020902 wait complete. 2024-04-25T11:46:28.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:28 smithi138 ceph-mon[32328]: pgmap v353: 97 pgs: 14 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 57 active+clean; 23 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 KiB/s wr, 12 op/s 2024-04-25T11:46:28.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:28 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:46:28.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:28 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:46:28.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:28 smithi156 ceph-mon[33045]: pgmap v353: 97 pgs: 14 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 57 active+clean; 23 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 KiB/s wr, 12 op/s 2024-04-25T11:46:28.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:28 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:46:28.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:28 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:46:28.387 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T11:46:28.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:46:28.388 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T11:46:28.408 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:28 smithi090 ceph-mon[29575]: pgmap v353: 97 pgs: 14 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 57 active+clean; 23 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 KiB/s wr, 12 op/s 2024-04-25T11:46:28.408 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:28 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:46:28.408 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:28 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:46:28.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.523+0000 7fbe7faf9640 1 -- 172.21.15.90:0/337091764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe78102af0 msgr2=0x7fbe78102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:28.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.523+0000 7fbe7faf9640 1 --2- 172.21.15.90:0/337091764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe78102af0 0x7fbe78102f00 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fbe68009900 tx=0x7fbe6802a410 comp rx=0 tx=0).stop 2024-04-25T11:46:28.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.523+0000 7fbe7faf9640 1 -- 172.21.15.90:0/337091764 shutdown_connections 2024-04-25T11:46:28.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.523+0000 7fbe7faf9640 1 --2- 172.21.15.90:0/337091764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe7810d380 0x7fbe7810f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.523+0000 7fbe7faf9640 1 --2- 172.21.15.90:0/337091764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe78102af0 0x7fbe78102f00 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.523+0000 7fbe7faf9640 1 --2- 172.21.15.90:0/337091764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe78108a20 0x7fbe781025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.523+0000 7fbe7faf9640 1 -- 172.21.15.90:0/337091764 >> 172.21.15.90:0/337091764 conn(0x7fbe78078100 msgr2=0x7fbe78078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:28.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.523+0000 7fbe7faf9640 1 -- 172.21.15.90:0/337091764 shutdown_connections 2024-04-25T11:46:28.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.523+0000 7fbe7faf9640 1 -- 172.21.15.90:0/337091764 wait complete. 2024-04-25T11:46:28.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.524+0000 7fbe7faf9640 1 Processor -- start 2024-04-25T11:46:28.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.524+0000 7fbe7faf9640 1 -- start start 2024-04-25T11:46:28.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.524+0000 7fbe7faf9640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe78102af0 0x7fbe7819b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.524+0000 7fbe7faf9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe78108a20 0x7fbe7819bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.524+0000 7fbe7faf9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe7810d380 0x7fbe7819c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.524+0000 7fbe7faf9640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbe78112be0 con 0x7fbe78108a20 2024-04-25T11:46:28.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.524+0000 7fbe7faf9640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbe78112a60 con 0x7fbe78102af0 2024-04-25T11:46:28.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.524+0000 7fbe7faf9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbe78112d60 con 0x7fbe7810d380 2024-04-25T11:46:28.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.524+0000 7fbe7d094640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe78108a20 0x7fbe7819bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.524+0000 7fbe7d094640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe78108a20 0x7fbe7819bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48418/0 (socket says 172.21.15.90:48418) 2024-04-25T11:46:28.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.524+0000 7fbe7d094640 1 -- 172.21.15.90:0/3082286424 learned_addr learned my addr 172.21.15.90:0/3082286424 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:28.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.525+0000 7fbe7d895640 1 --2- 172.21.15.90:0/3082286424 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe78102af0 0x7fbe7819b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.525+0000 7fbe7e096640 1 --2- 172.21.15.90:0/3082286424 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe7810d380 0x7fbe7819c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.525+0000 7fbe7d094640 1 -- 172.21.15.90:0/3082286424 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe7810d380 msgr2=0x7fbe7819c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:28.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.525+0000 7fbe7d094640 1 --2- 172.21.15.90:0/3082286424 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe7810d380 0x7fbe7819c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.525+0000 7fbe7d094640 1 -- 172.21.15.90:0/3082286424 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe78102af0 msgr2=0x7fbe7819b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:28.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.525+0000 7fbe7d094640 1 --2- 172.21.15.90:0/3082286424 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe78102af0 0x7fbe7819b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.525+0000 7fbe7d094640 1 -- 172.21.15.90:0/3082286424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe781a68d0 con 0x7fbe78108a20 2024-04-25T11:46:28.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.525+0000 7fbe7d094640 1 --2- 172.21.15.90:0/3082286424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe78108a20 0x7fbe7819bb00 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fbe6802a930 tx=0x7fbe68031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:28.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.525+0000 7fbe6effd640 1 -- 172.21.15.90:0/3082286424 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe6803f030 con 0x7fbe78108a20 2024-04-25T11:46:28.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.525+0000 7fbe6effd640 1 -- 172.21.15.90:0/3082286424 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe68004490 con 0x7fbe78108a20 2024-04-25T11:46:28.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.525+0000 7fbe7faf9640 1 -- 172.21.15.90:0/3082286424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe781a6b60 con 0x7fbe78108a20 2024-04-25T11:46:28.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.525+0000 7fbe6effd640 1 -- 172.21.15.90:0/3082286424 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe68035070 con 0x7fbe78108a20 2024-04-25T11:46:28.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.525+0000 7fbe7faf9640 1 -- 172.21.15.90:0/3082286424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe781a70a0 con 0x7fbe78108a20 2024-04-25T11:46:28.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.529+0000 7fbe869ad740 1 Processor -- start 2024-04-25T11:46:28.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.529+0000 7fbe6effd640 1 -- 172.21.15.90:0/3082286424 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fbe68042030 con 0x7fbe78108a20 2024-04-25T11:46:28.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.529+0000 7fbe869ad740 1 -- start start 2024-04-25T11:46:28.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.529+0000 7fbe869ad740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5641e9a36410 0x5641e9dd6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.529+0000 7fbe869ad740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641e9a0d550 0x5641e9dd6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.529+0000 7fbe869ad740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641e9a15730 0x5641e9dd70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.529+0000 7fbe869ad740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5641e9d3cb00 con 0x5641e9a15730 2024-04-25T11:46:28.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.529+0000 7fbe869ad740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5641e9cd8920 con 0x5641e9a36410 2024-04-25T11:46:28.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.529+0000 7fbe869ad740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5641e9d38ac0 con 0x5641e9a0d550 2024-04-25T11:46:28.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.529+0000 7fbe7e096640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5641e9a36410 0x5641e9dd6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.529+0000 7fbe7d895640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641e9a0d550 0x5641e9dd6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.529+0000 7fbe7e096640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5641e9a36410 0x5641e9dd6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.90:34124/0 (socket says 172.21.15.90:34124) 2024-04-25T11:46:28.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.530+0000 7fbe7e096640 1 -- 172.21.15.90:0/1777117013 learned_addr learned my addr 172.21.15.90:0/1777117013 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:28.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.530+0000 7fbe7d895640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641e9a0d550 0x5641e9dd6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.90:49352/0 (socket says 172.21.15.90:49352) 2024-04-25T11:46:28.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.530+0000 7fbe7d094640 1 --2- 172.21.15.90:0/1777117013 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641e9a15730 0x5641e9dd70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.530+0000 7fbe7e096640 1 -- 172.21.15.90:0/1777117013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641e9a0d550 msgr2=0x5641e9dd6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:28.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.530+0000 7fbe7e096640 1 --2- 172.21.15.90:0/1777117013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641e9a0d550 0x5641e9dd6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.530+0000 7fbe7e096640 1 -- 172.21.15.90:0/1777117013 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641e9a15730 msgr2=0x5641e9dd70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:28.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.530+0000 7fbe6effd640 1 --2- 172.21.15.90:0/3082286424 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbe640afce0 0x7fbe640b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.530+0000 7fbe7e096640 1 --2- 172.21.15.90:0/1777117013 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641e9a15730 0x5641e9dd70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.530+0000 7fbe7e096640 1 -- 172.21.15.90:0/1777117013 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641e9b167e0 con 0x5641e9a36410 2024-04-25T11:46:28.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.530+0000 7fbe6effd640 1 -- 172.21.15.90:0/3082286424 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7fbe680ff420 con 0x7fbe78108a20 2024-04-25T11:46:28.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.530+0000 7fbe7e096640 1 --2- 172.21.15.90:0/3082286424 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbe640afce0 0x7fbe640b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.530+0000 7fbe7e096640 1 --2- 172.21.15.90:0/1777117013 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5641e9a36410 0x5641e9dd6620 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbe74009e20 tx=0x7fbe7400f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db91b210063f1a56 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:28.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.531+0000 7fbe63fff640 1 -- 172.21.15.90:0/1777117013 <== mon.2 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe74021070 con 0x5641e9a36410 2024-04-25T11:46:28.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.531+0000 7fbe63fff640 1 -- 172.21.15.90:0/1777117013 <== mon.2 v2:172.21.15.138:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe7401be20 con 0x5641e9a36410 2024-04-25T11:46:28.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.531+0000 7fbe63fff640 1 -- 172.21.15.90:0/1777117013 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe7401cde0 con 0x5641e9a36410 2024-04-25T11:46:28.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.532+0000 7fbe869ad740 1 -- 172.21.15.90:0/1777117013 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5641e9a36410 msgr2=0x5641e9dd6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:28.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.532+0000 7fbe7e096640 1 --2- 172.21.15.90:0/3082286424 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbe640afce0 0x7fbe640b21f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbe7401e000 tx=0x7fbe74021000 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:28.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.532+0000 7fbe869ad740 1 --2- 172.21.15.90:0/1777117013 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5641e9a36410 0x5641e9dd6620 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbe74009e20 tx=0x7fbe7400f730 comp rx=0 tx=0).stop 2024-04-25T11:46:28.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.532+0000 7fbe869ad740 1 -- 172.21.15.90:0/1777117013 shutdown_connections 2024-04-25T11:46:28.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.532+0000 7fbe869ad740 1 --2- 172.21.15.90:0/1777117013 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641e9a15730 0x5641e9dd70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.532+0000 7fbe869ad740 1 --2- 172.21.15.90:0/1777117013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641e9a0d550 0x5641e9dd6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.532+0000 7fbe869ad740 1 --2- 172.21.15.90:0/1777117013 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5641e9a36410 0x5641e9dd6620 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.532+0000 7fbe869ad740 1 -- 172.21.15.90:0/1777117013 >> 172.21.15.90:0/1777117013 conn(0x5641e9a2a560 msgr2=0x5641e9dd2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:28.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.532+0000 7fbe869ad740 1 -- 172.21.15.90:0/1777117013 shutdown_connections 2024-04-25T11:46:28.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.532+0000 7fbe869ad740 1 -- 172.21.15.90:0/1777117013 wait complete. 2024-04-25T11:46:28.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.533+0000 7fbe869ad740 1 -- start start 2024-04-25T11:46:28.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.533+0000 7fbe869ad740 1 Processor -- start 2024-04-25T11:46:28.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.533+0000 7fbe869ad740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641e9a0d550 0x5641e9de0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.534+0000 7fbe869ad740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5641e9a15730 0x5641e9de10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.534+0000 7fbe869ad740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641e9a36410 0x5641e9dece80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.534+0000 7fbe869ad740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5641e9de3970 con 0x5641e9a0d550 2024-04-25T11:46:28.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.534+0000 7fbe869ad740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5641e9cd65d0 con 0x5641e9a15730 2024-04-25T11:46:28.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.534+0000 7fbe869ad740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5641e9de3af0 con 0x5641e9a36410 2024-04-25T11:46:28.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.534+0000 7fbe869ad740 10 client.0 _ll_register_callbacks cb 0x5641e9cc4080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:28.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.534+0000 7fbe7d895640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641e9a0d550 0x5641e9de0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.534+0000 7fbe7d895640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641e9a0d550 0x5641e9de0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48436/0 (socket says 172.21.15.90:48436) 2024-04-25T11:46:28.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.534+0000 7fbe7d895640 1 -- 172.21.15.90:0/1877934744 learned_addr learned my addr 172.21.15.90:0/1877934744 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:28.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.534+0000 7fbe7e096640 1 --2- 172.21.15.90:0/1877934744 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641e9a36410 0x5641e9dece80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.534+0000 7fbe7d094640 1 --2- 172.21.15.90:0/1877934744 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5641e9a15730 0x5641e9de10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.534+0000 7fbe7d895640 1 -- 172.21.15.90:0/1877934744 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641e9a36410 msgr2=0x5641e9dece80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:28.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.534+0000 7fbe7d895640 1 --2- 172.21.15.90:0/1877934744 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641e9a36410 0x5641e9dece80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.534+0000 7fbe7d895640 1 -- 172.21.15.90:0/1877934744 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5641e9a15730 msgr2=0x5641e9de10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.534+0000 7fbe7d895640 1 --2- 172.21.15.90:0/1877934744 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5641e9a15730 0x5641e9de10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.534+0000 7fbe7d895640 1 -- 172.21.15.90:0/1877934744 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641e9b76f70 con 0x5641e9a0d550 2024-04-25T11:46:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.534+0000 7fbe7e096640 1 --2- 172.21.15.90:0/1877934744 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641e9a36410 0x5641e9dece80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:46:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.534+0000 7fbe7d094640 1 --2- 172.21.15.90:0/1877934744 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5641e9a15730 0x5641e9de10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:46:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.534+0000 7fbe7d895640 1 --2- 172.21.15.90:0/1877934744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641e9a0d550 0x5641e9de0bb0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fbe70012700 tx=0x7fbe70012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ffea37b6d44ccda server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.534+0000 7fbe617fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:46:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.534+0000 7fbe617fa640 1 -- 172.21.15.90:0/1877934744 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe70004050 con 0x5641e9a0d550 2024-04-25T11:46:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.534+0000 7fbe617fa640 1 -- 172.21.15.90:0/1877934744 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7fbe700041f0 con 0x5641e9a0d550 2024-04-25T11:46:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.534+0000 7fbe869ad740 1 -- 172.21.15.90:0/1877934744 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe7401be20 con 0x5641e9a0d550 2024-04-25T11:46:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.535+0000 7fbe617fa640 1 -- 172.21.15.90:0/1877934744 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbe700043f0 con 0x5641e9a0d550 2024-04-25T11:46:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.535+0000 7fbe869ad740 10 client.19314 fetch_fsmap learned FSMap version 28 2024-04-25T11:46:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.535+0000 7fbe869ad740 1 -- 172.21.15.90:0/1877934744 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5641e9a0e170 con 0x5641e9a0d550 2024-04-25T11:46:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.535+0000 7fbe617fa640 1 -- 172.21.15.90:0/1877934744 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe7401be20 con 0x5641e9a0d550 2024-04-25T11:46:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.535+0000 7fbe617fa640 1 -- 172.21.15.90:0/1877934744 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fbe70003b20 con 0x5641e9a0d550 2024-04-25T11:46:28.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.535+0000 7fbe869ad740 10 client.19314 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:46:28.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.535+0000 7fbe869ad740 10 client.19314 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:46:28.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.535+0000 7fbe869ad740 20 client.19314 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:28.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.535+0000 7fbe869ad740 1 --2- 172.21.15.90:0/1877934744 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5641e99f51f0 0x5641e9df3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.535+0000 7fbe869ad740 4 client.19314 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:46:28.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.535+0000 7fbe869ad740 1 -- 172.21.15.90:0/1877934744 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5641e9d4a710 con 0x5641e99f51f0 2024-04-25T11:46:28.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.535+0000 7fbe7d094640 1 --2- 172.21.15.90:0/1877934744 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5641e99f51f0 0x5641e9df3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.536+0000 7fbe617fa640 10 client.19314 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:28.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.536+0000 7fbe7d094640 1 --2- 172.21.15.90:0/1877934744 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5641e99f51f0 0x5641e9df3120 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8e0292905dfa1dc server_cookie=c293fddd79b3129b in_seq=0 out_seq=0 2024-04-25T11:46:28.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.536+0000 7fbe617fa640 1 -- 172.21.15.90:0/1877934744 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5641e9d4a710 con 0x5641e99f51f0 2024-04-25T11:46:28.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.536+0000 7fbe617fa640 10 client.19314 handle_command_reply: tid=0 2024-04-25T11:46:28.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.536+0000 7fbe869ad740 1 client.19314 shutdown 2024-04-25T11:46:28.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.537+0000 7fbe869ad740 1 -- 172.21.15.90:0/1877934744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641e9a0d550 msgr2=0x5641e9de0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:28.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.537+0000 7fbe869ad740 1 --2- 172.21.15.90:0/1877934744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641e9a0d550 0x5641e9de0bb0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fbe70012700 tx=0x7fbe70012bc0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.537+0000 7fbe869ad740 1 -- 172.21.15.90:0/1877934744 shutdown_connections 2024-04-25T11:46:28.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.537+0000 7fbe869ad740 1 --2- 172.21.15.90:0/1877934744 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5641e99f51f0 0x5641e9df3120 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.537+0000 7fbe869ad740 1 --2- 172.21.15.90:0/1877934744 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641e9a36410 0x5641e9dece80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.537+0000 7fbe869ad740 1 --2- 172.21.15.90:0/1877934744 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5641e9a15730 0x5641e9de10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.537+0000 7fbe869ad740 1 --2- 172.21.15.90:0/1877934744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641e9a0d550 0x5641e9de0bb0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.537+0000 7fbe617fa640 0 client.19314 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:28.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.537+0000 7fbe869ad740 1 -- 172.21.15.90:0/1877934744 >> 172.21.15.90:0/1877934744 conn(0x5641e9a2a560 msgr2=0x5641e9dd63f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:28.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.537+0000 7fbe869ad740 1 -- 172.21.15.90:0/1877934744 shutdown_connections 2024-04-25T11:46:28.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.537+0000 7fbe869ad740 1 -- 172.21.15.90:0/1877934744 wait complete. 2024-04-25T11:46:28.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.537+0000 7fbe869ad740 20 client.19314 trim_cache size 0 max 16384 2024-04-25T11:46:28.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.556+0000 7fbe869ad740 1 Processor -- start 2024-04-25T11:46:28.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.556+0000 7fbe869ad740 1 -- start start 2024-04-25T11:46:28.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.556+0000 7fbe869ad740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5641e9a0d550 0x5641e9dd1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.556+0000 7fbe869ad740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641e9a15730 0x5641e9dd2d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.556+0000 7fbe869ad740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641e9a36410 0x5641e9ddb130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.556+0000 7fbe869ad740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbe781afce0 con 0x5641e9a36410 2024-04-25T11:46:28.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.556+0000 7fbe869ad740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbe781b1820 con 0x5641e9a0d550 2024-04-25T11:46:28.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.556+0000 7fbe869ad740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbe781b03b0 con 0x5641e9a15730 2024-04-25T11:46:28.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.556+0000 7fbe7e096640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641e9a36410 0x5641e9ddb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.556+0000 7fbe7e096640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641e9a36410 0x5641e9ddb130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48438/0 (socket says 172.21.15.90:48438) 2024-04-25T11:46:28.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.556+0000 7fbe7d895640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5641e9a0d550 0x5641e9dd1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.556+0000 7fbe7e096640 1 -- 172.21.15.90:0/1648699033 learned_addr learned my addr 172.21.15.90:0/1648699033 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:28.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.556+0000 7fbe7d094640 1 --2- 172.21.15.90:0/1648699033 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641e9a15730 0x5641e9dd2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.556+0000 7fbe7e096640 1 -- 172.21.15.90:0/1648699033 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641e9a15730 msgr2=0x5641e9dd2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:28.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.556+0000 7fbe7e096640 1 --2- 172.21.15.90:0/1648699033 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641e9a15730 0x5641e9dd2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.556+0000 7fbe7e096640 1 -- 172.21.15.90:0/1648699033 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5641e9a0d550 msgr2=0x5641e9dd1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:28.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.556+0000 7fbe7e096640 1 --2- 172.21.15.90:0/1648699033 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5641e9a0d550 0x5641e9dd1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.556+0000 7fbe7e096640 1 -- 172.21.15.90:0/1648699033 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641e9dd39c0 con 0x5641e9a36410 2024-04-25T11:46:28.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.556+0000 7fbe7d094640 1 --2- 172.21.15.90:0/1648699033 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641e9a15730 0x5641e9dd2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:46:28.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.556+0000 7fbe7d895640 1 --2- 172.21.15.90:0/1648699033 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5641e9a0d550 0x5641e9dd1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:46:28.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.557+0000 7fbe7e096640 1 --2- 172.21.15.90:0/1648699033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641e9a36410 0x5641e9ddb130 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fbe74009dc0 tx=0x7fbe74025000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3e981c9deed5ca9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:28.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.557+0000 7fbe637fe640 1 -- 172.21.15.90:0/1648699033 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe74035070 con 0x5641e9a36410 2024-04-25T11:46:28.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.557+0000 7fbe637fe640 1 -- 172.21.15.90:0/1648699033 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe7401be20 con 0x5641e9a36410 2024-04-25T11:46:28.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.557+0000 7fbe637fe640 1 -- 172.21.15.90:0/1648699033 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe74010c20 con 0x5641e9a36410 2024-04-25T11:46:28.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.558+0000 7fbe869ad740 1 -- 172.21.15.90:0/1648699033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641e9a36410 msgr2=0x5641e9ddb130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:28.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.558+0000 7fbe869ad740 1 --2- 172.21.15.90:0/1648699033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641e9a36410 0x5641e9ddb130 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fbe74009dc0 tx=0x7fbe74025000 comp rx=0 tx=0).stop 2024-04-25T11:46:28.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.558+0000 7fbe869ad740 1 -- 172.21.15.90:0/1648699033 shutdown_connections 2024-04-25T11:46:28.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.558+0000 7fbe869ad740 1 --2- 172.21.15.90:0/1648699033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641e9a36410 0x5641e9ddb130 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.558+0000 7fbe869ad740 1 --2- 172.21.15.90:0/1648699033 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641e9a15730 0x5641e9dd2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.558+0000 7fbe869ad740 1 --2- 172.21.15.90:0/1648699033 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5641e9a0d550 0x5641e9dd1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.558+0000 7fbe869ad740 1 -- 172.21.15.90:0/1648699033 >> 172.21.15.90:0/1648699033 conn(0x5641e9a2a560 msgr2=0x5641e9dd63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:28.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.558+0000 7fbe869ad740 1 -- 172.21.15.90:0/1648699033 shutdown_connections 2024-04-25T11:46:28.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.558+0000 7fbe869ad740 1 -- 172.21.15.90:0/1648699033 wait complete. 2024-04-25T11:46:28.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.558+0000 7fbe869ad740 1 -- start start 2024-04-25T11:46:28.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.558+0000 7fbe869ad740 1 Processor -- start 2024-04-25T11:46:28.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.559+0000 7fbe869ad740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641e9a0d550 0x5641e9de3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.559+0000 7fbe869ad740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641e9a15730 0x5641e9de42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.559+0000 7fbe869ad740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5641e9a36410 0x5641e9de4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.559+0000 7fbe869ad740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbe781b1150 con 0x5641e9a15730 2024-04-25T11:46:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.559+0000 7fbe869ad740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbe781b0a80 con 0x5641e9a36410 2024-04-25T11:46:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.559+0000 7fbe869ad740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbe781adce0 con 0x5641e9a0d550 2024-04-25T11:46:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.559+0000 7fbe869ad740 10 client.0 _ll_register_callbacks cb 0x7fbe70009d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.559+0000 7fbe7d094640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641e9a15730 0x5641e9de42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.559+0000 7fbe7d895640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641e9a0d550 0x5641e9de3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.559+0000 7fbe7d094640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641e9a15730 0x5641e9de42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48440/0 (socket says 172.21.15.90:48440) 2024-04-25T11:46:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.559+0000 7fbe7d094640 1 -- 172.21.15.90:0/450403628 learned_addr learned my addr 172.21.15.90:0/450403628 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.559+0000 7fbe7e096640 1 --2- 172.21.15.90:0/450403628 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5641e9a36410 0x5641e9de4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.559+0000 7fbe7d094640 1 -- 172.21.15.90:0/450403628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641e9a0d550 msgr2=0x5641e9de3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.559+0000 7fbe7d094640 1 --2- 172.21.15.90:0/450403628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641e9a0d550 0x5641e9de3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.559+0000 7fbe7d094640 1 -- 172.21.15.90:0/450403628 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5641e9a36410 msgr2=0x5641e9de4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.559+0000 7fbe7d094640 1 --2- 172.21.15.90:0/450403628 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5641e9a36410 0x5641e9de4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.559+0000 7fbe7d094640 1 -- 172.21.15.90:0/450403628 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641e9de97f0 con 0x5641e9a15730 2024-04-25T11:46:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.559+0000 7fbe7d895640 1 --2- 172.21.15.90:0/450403628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641e9a0d550 0x5641e9de3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:46:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.559+0000 7fbe7d094640 1 --2- 172.21.15.90:0/450403628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641e9a15730 0x5641e9de42e0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fbe680c7210 tx=0x7fbe6803b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a759fb80ebb86f24 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.559+0000 7fbe617fa640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:46:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.559+0000 7fbe617fa640 1 -- 172.21.15.90:0/450403628 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe68049070 con 0x5641e9a15730 2024-04-25T11:46:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.559+0000 7fbe617fa640 1 -- 172.21.15.90:0/450403628 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7fbe680c37a0 con 0x5641e9a15730 2024-04-25T11:46:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.559+0000 7fbe869ad740 1 -- 172.21.15.90:0/450403628 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe7401be20 con 0x5641e9a15730 2024-04-25T11:46:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.560+0000 7fbe617fa640 1 -- 172.21.15.90:0/450403628 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbe680c6c80 con 0x5641e9a15730 2024-04-25T11:46:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.560+0000 7fbe869ad740 10 client.19320 fetch_fsmap learned FSMap version 28 2024-04-25T11:46:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.560+0000 7fbe869ad740 1 -- 172.21.15.90:0/450403628 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5641e9e5a5a0 con 0x5641e9a15730 2024-04-25T11:46:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.560+0000 7fbe617fa640 1 -- 172.21.15.90:0/450403628 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe7401be20 con 0x5641e9a15730 2024-04-25T11:46:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.560+0000 7fbe617fa640 1 -- 172.21.15.90:0/450403628 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fbe680c7770 con 0x5641e9a15730 2024-04-25T11:46:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.560+0000 7fbe869ad740 10 client.19320 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:46:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.560+0000 7fbe869ad740 10 client.19320 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:46:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.560+0000 7fbe869ad740 20 client.19320 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.560+0000 7fbe869ad740 1 --2- 172.21.15.90:0/450403628 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5641e99ca630 0x5641e9e5cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.560+0000 7fbe869ad740 4 client.19320 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T11:46:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.560+0000 7fbe869ad740 1 -- 172.21.15.90:0/450403628 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5641e9d44580 con 0x5641e99ca630 2024-04-25T11:46:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.560+0000 7fbe7d895640 1 --2- 172.21.15.90:0/450403628 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5641e99ca630 0x5641e9e5cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.561+0000 7fbe617fa640 10 client.19320 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.561+0000 7fbe7d895640 1 --2- 172.21.15.90:0/450403628 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5641e99ca630 0x5641e9e5cd10 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff20651d531f2adc server_cookie=ddf617b81749b900 in_seq=0 out_seq=0 2024-04-25T11:46:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.562+0000 7fbe617fa640 1 -- 172.21.15.90:0/450403628 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5641e9d44580 con 0x5641e99ca630 2024-04-25T11:46:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.562+0000 7fbe617fa640 10 client.19320 handle_command_reply: tid=0 2024-04-25T11:46:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.562+0000 7fbe869ad740 1 client.19320 shutdown 2024-04-25T11:46:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.562+0000 7fbe869ad740 1 -- 172.21.15.90:0/450403628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641e9a15730 msgr2=0x5641e9de42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.562+0000 7fbe869ad740 1 --2- 172.21.15.90:0/450403628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641e9a15730 0x5641e9de42e0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fbe680c7210 tx=0x7fbe6803b560 comp rx=0 tx=0).stop 2024-04-25T11:46:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.562+0000 7fbe869ad740 1 -- 172.21.15.90:0/450403628 shutdown_connections 2024-04-25T11:46:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.562+0000 7fbe869ad740 1 --2- 172.21.15.90:0/450403628 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5641e99ca630 0x5641e9e5cd10 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.562+0000 7fbe869ad740 1 --2- 172.21.15.90:0/450403628 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5641e9a36410 0x5641e9de4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.562+0000 7fbe869ad740 1 --2- 172.21.15.90:0/450403628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641e9a15730 0x5641e9de42e0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.562+0000 7fbe869ad740 1 --2- 172.21.15.90:0/450403628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641e9a0d550 0x5641e9de3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.562+0000 7fbe869ad740 1 -- 172.21.15.90:0/450403628 >> 172.21.15.90:0/450403628 conn(0x5641e9a2a560 msgr2=0x5641e9dd1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:28.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.562+0000 7fbe617fa640 0 client.19320 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:28.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.562+0000 7fbe869ad740 1 -- 172.21.15.90:0/450403628 shutdown_connections 2024-04-25T11:46:28.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.562+0000 7fbe869ad740 1 -- 172.21.15.90:0/450403628 wait complete. 2024-04-25T11:46:28.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.562+0000 7fbe869ad740 20 client.19320 trim_cache size 0 max 16384 2024-04-25T11:46:28.552 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:46:28.552 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-25T11:46:28.552 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "68445a2a-81ad-41b3-8065-1660a34d0e82", 2024-04-25T11:46:28.552 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-25T11:46:28.552 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:46:28.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.563+0000 7fbe7faf9640 1 -- 172.21.15.90:0/3082286424 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbe640afce0 msgr2=0x7fbe640b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:28.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.563+0000 7fbe7faf9640 1 --2- 172.21.15.90:0/3082286424 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbe640afce0 0x7fbe640b21f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbe7401e000 tx=0x7fbe74021000 comp rx=0 tx=0).stop 2024-04-25T11:46:28.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.563+0000 7fbe7faf9640 1 -- 172.21.15.90:0/3082286424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe78108a20 msgr2=0x7fbe7819bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:28.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.563+0000 7fbe7faf9640 1 --2- 172.21.15.90:0/3082286424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe78108a20 0x7fbe7819bb00 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fbe6802a930 tx=0x7fbe68031a20 comp rx=0 tx=0).stop 2024-04-25T11:46:28.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.563+0000 7fbe7faf9640 1 -- 172.21.15.90:0/3082286424 shutdown_connections 2024-04-25T11:46:28.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.563+0000 7fbe7faf9640 1 --2- 172.21.15.90:0/3082286424 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbe640afce0 0x7fbe640b21f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.563+0000 7fbe7faf9640 1 --2- 172.21.15.90:0/3082286424 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe7810d380 0x7fbe7819c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.563+0000 7fbe7faf9640 1 --2- 172.21.15.90:0/3082286424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe78108a20 0x7fbe7819bb00 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.563+0000 7fbe7faf9640 1 --2- 172.21.15.90:0/3082286424 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbe78102af0 0x7fbe7819b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.563+0000 7fbe7faf9640 1 -- 172.21.15.90:0/3082286424 >> 172.21.15.90:0/3082286424 conn(0x7fbe78078100 msgr2=0x7fbe781040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:28.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.563+0000 7fbe7faf9640 1 -- 172.21.15.90:0/3082286424 shutdown_connections 2024-04-25T11:46:28.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.563+0000 7fbe7faf9640 1 -- 172.21.15.90:0/3082286424 wait complete. 2024-04-25T11:46:28.559 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:46:28.559 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:46:28.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.730+0000 7f35fd461640 1 -- 172.21.15.90:0/673753258 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f81042b0 msgr2=0x7f35f8110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:28.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.730+0000 7f35fd461640 1 --2- 172.21.15.90:0/673753258 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f81042b0 0x7f35f8110630 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f35f0009ca0 tx=0x7f35f002a8c0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.730+0000 7f35fd461640 1 -- 172.21.15.90:0/673753258 shutdown_connections 2024-04-25T11:46:28.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.730+0000 7f35fd461640 1 --2- 172.21.15.90:0/673753258 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f81042b0 0x7f35f8110630 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.730+0000 7f35fd461640 1 --2- 172.21.15.90:0/673753258 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35f8109fb0 0x7f35f8103c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.730+0000 7f35fd461640 1 --2- 172.21.15.90:0/673753258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f8109b10 0x7f35f8103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.730+0000 7f35fd461640 1 -- 172.21.15.90:0/673753258 >> 172.21.15.90:0/673753258 conn(0x7f35f80ff360 msgr2=0x7f35f8101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:28.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.730+0000 7f35fd461640 1 -- 172.21.15.90:0/673753258 shutdown_connections 2024-04-25T11:46:28.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.730+0000 7f35fd461640 1 -- 172.21.15.90:0/673753258 wait complete. 2024-04-25T11:46:28.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.730+0000 7f35fd461640 1 Processor -- start 2024-04-25T11:46:28.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.731+0000 7f35fd461640 1 -- start start 2024-04-25T11:46:28.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.731+0000 7f35fd461640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f81042b0 0x7f35f819a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.731+0000 7f35fd461640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35f8109b10 0x7f35f819ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.731+0000 7f35fd461640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f8109fb0 0x7f35f819ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.731+0000 7f35fd461640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f35f8113d00 con 0x7f35f8109fb0 2024-04-25T11:46:28.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.731+0000 7f35fd461640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f35f8113b80 con 0x7f35f8109b10 2024-04-25T11:46:28.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.731+0000 7f35fd461640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f35f8113e80 con 0x7f35f81042b0 2024-04-25T11:46:28.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.731+0000 7f35f77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f8109fb0 0x7f35f819ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.731+0000 7f35f77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f8109fb0 0x7f35f819ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48448/0 (socket says 172.21.15.90:48448) 2024-04-25T11:46:28.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.731+0000 7f35f77fe640 1 -- 172.21.15.90:0/171439437 learned_addr learned my addr 172.21.15.90:0/171439437 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:28.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.731+0000 7f35f6ffd640 1 --2- 172.21.15.90:0/171439437 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f81042b0 0x7f35f819a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.731+0000 7f35ee7fc640 1 --2- 172.21.15.90:0/171439437 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35f8109b10 0x7f35f819ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.731+0000 7f35f77fe640 1 -- 172.21.15.90:0/171439437 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f81042b0 msgr2=0x7f35f819a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:28.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.731+0000 7f35f77fe640 1 --2- 172.21.15.90:0/171439437 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f81042b0 0x7f35f819a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.731+0000 7f35f77fe640 1 -- 172.21.15.90:0/171439437 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35f8109b10 msgr2=0x7f35f819ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:28.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.731+0000 7f35f77fe640 1 --2- 172.21.15.90:0/171439437 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35f8109b10 0x7f35f819ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.731+0000 7f35f77fe640 1 -- 172.21.15.90:0/171439437 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35f819f4c0 con 0x7f35f8109fb0 2024-04-25T11:46:28.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.731+0000 7f35f77fe640 1 --2- 172.21.15.90:0/171439437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f8109fb0 0x7f35f819ef00 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f35f0006010 tx=0x7f35f002f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:28.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.732+0000 7f35f4ff9640 1 -- 172.21.15.90:0/171439437 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f0003bf0 con 0x7f35f8109fb0 2024-04-25T11:46:28.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.732+0000 7f35f4ff9640 1 -- 172.21.15.90:0/171439437 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35f0003d90 con 0x7f35f8109fb0 2024-04-25T11:46:28.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.732+0000 7f35fd461640 1 -- 172.21.15.90:0/171439437 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35f819f750 con 0x7f35f8109fb0 2024-04-25T11:46:28.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.732+0000 7f35f4ff9640 1 -- 172.21.15.90:0/171439437 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f002fb20 con 0x7f35f8109fb0 2024-04-25T11:46:28.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.732+0000 7f35fd461640 1 -- 172.21.15.90:0/171439437 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35f819fca0 con 0x7f35f8109fb0 2024-04-25T11:46:28.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.734+0000 7f36043ed740 1 Processor -- start 2024-04-25T11:46:28.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.734+0000 7f35f4ff9640 1 -- 172.21.15.90:0/171439437 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f35f002fcc0 con 0x7f35f8109fb0 2024-04-25T11:46:28.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.734+0000 7f36043ed740 1 -- start start 2024-04-25T11:46:28.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.735+0000 7f36043ed740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5584000b3390 0x558400453a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.735+0000 7f36043ed740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55840008a4d0 0x558400453fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.735+0000 7f36043ed740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584000926b0 0x5584004608f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.735+0000 7f36043ed740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5584003b9d80 con 0x55840008a4d0 2024-04-25T11:46:28.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.735+0000 7f36043ed740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5584003738b0 con 0x5584000b3390 2024-04-25T11:46:28.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.735+0000 7f36043ed740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5584003b5d40 con 0x5584000926b0 2024-04-25T11:46:28.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.735+0000 7f35f6ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55840008a4d0 0x558400453fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.735+0000 7f35f6ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55840008a4d0 0x558400453fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48458/0 (socket says 172.21.15.90:48458) 2024-04-25T11:46:28.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.735+0000 7f35f6ffd640 1 -- 172.21.15.90:0/2867755758 learned_addr learned my addr 172.21.15.90:0/2867755758 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:28.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.735+0000 7f35ee7fc640 1 --2- 172.21.15.90:0/2867755758 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5584000b3390 0x558400453a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.735+0000 7f35ee7fc640 1 --2- 172.21.15.90:0/2867755758 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584000926b0 0x5584004608f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.735+0000 7f35f4ff9640 1 --2- 172.21.15.90:0/171439437 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f35e40afc60 0x7f35e40b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.735+0000 7f35f6ffd640 1 -- 172.21.15.90:0/2867755758 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584000926b0 msgr2=0x5584004608f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:28.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.735+0000 7f35f6ffd640 1 --2- 172.21.15.90:0/2867755758 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584000926b0 0x5584004608f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.735+0000 7f35f6ffd640 1 -- 172.21.15.90:0/2867755758 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5584000b3390 msgr2=0x558400453a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:28.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.735+0000 7f35f6ffd640 1 --2- 172.21.15.90:0/2867755758 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5584000b3390 0x558400453a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.735+0000 7f35f6ffd640 1 -- 172.21.15.90:0/2867755758 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558400344a40 con 0x55840008a4d0 2024-04-25T11:46:28.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.735+0000 7f35f77fe640 1 --2- 172.21.15.90:0/171439437 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f35e40afc60 0x7f35e40b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.735+0000 7f35f6ffd640 1 --2- 172.21.15.90:0/2867755758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55840008a4d0 0x558400453fa0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f35e800aca0 tx=0x7f35e800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4fdb0554293d21f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:28.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.735+0000 7f35f4ff9640 1 -- 172.21.15.90:0/171439437 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7f35f0100760 con 0x7f35f8109fb0 2024-04-25T11:46:28.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.736+0000 7f35ed7fa640 1 -- 172.21.15.90:0/2867755758 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e801d070 con 0x55840008a4d0 2024-04-25T11:46:28.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.736+0000 7f35ed7fa640 1 -- 172.21.15.90:0/2867755758 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35e8018af0 con 0x55840008a4d0 2024-04-25T11:46:28.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.736+0000 7f35ed7fa640 1 -- 172.21.15.90:0/2867755758 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e8010b70 con 0x55840008a4d0 2024-04-25T11:46:28.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.737+0000 7f36043ed740 1 -- 172.21.15.90:0/2867755758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55840008a4d0 msgr2=0x558400453fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:28.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.737+0000 7f35f77fe640 1 --2- 172.21.15.90:0/171439437 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f35e40afc60 0x7f35e40b2170 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f35f00fe400 tx=0x7f35f00961c0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:28.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.737+0000 7f36043ed740 1 --2- 172.21.15.90:0/2867755758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55840008a4d0 0x558400453fa0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f35e800aca0 tx=0x7f35e800f790 comp rx=0 tx=0).stop 2024-04-25T11:46:28.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.737+0000 7f36043ed740 1 -- 172.21.15.90:0/2867755758 shutdown_connections 2024-04-25T11:46:28.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.737+0000 7f36043ed740 1 --2- 172.21.15.90:0/2867755758 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584000926b0 0x5584004608f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.737+0000 7f36043ed740 1 --2- 172.21.15.90:0/2867755758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55840008a4d0 0x558400453fa0 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.737+0000 7f36043ed740 1 --2- 172.21.15.90:0/2867755758 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5584000b3390 0x558400453a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.737+0000 7f36043ed740 1 -- 172.21.15.90:0/2867755758 >> 172.21.15.90:0/2867755758 conn(0x5584000a74e0 msgr2=0x558400450210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:28.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.737+0000 7f36043ed740 1 -- 172.21.15.90:0/2867755758 shutdown_connections 2024-04-25T11:46:28.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.737+0000 7f36043ed740 1 -- 172.21.15.90:0/2867755758 wait complete. 2024-04-25T11:46:28.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.737+0000 7f36043ed740 1 -- start start 2024-04-25T11:46:28.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.738+0000 7f36043ed740 1 Processor -- start 2024-04-25T11:46:28.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.738+0000 7f36043ed740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55840008a4d0 0x55840045e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.738+0000 7f36043ed740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584000926b0 0x55840045eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.738+0000 7f36043ed740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584000b3390 0x5584004644a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.738+0000 7f36043ed740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558400460fe0 con 0x5584000b3390 2024-04-25T11:46:28.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.738+0000 7f36043ed740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x558400340d10 con 0x55840008a4d0 2024-04-25T11:46:28.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.738+0000 7f36043ed740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558400461160 con 0x5584000926b0 2024-04-25T11:46:28.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.738+0000 7f36043ed740 10 client.0 _ll_register_callbacks cb 0x55840036d210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:28.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.738+0000 7f35ee7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584000b3390 0x5584004644a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.738+0000 7f35ee7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584000b3390 0x5584004644a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48470/0 (socket says 172.21.15.90:48470) 2024-04-25T11:46:28.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.738+0000 7f35ee7fc640 1 -- 172.21.15.90:0/2075800234 learned_addr learned my addr 172.21.15.90:0/2075800234 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:28.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.738+0000 7f35f6ffd640 1 --2- 172.21.15.90:0/2075800234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584000926b0 0x55840045eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.738+0000 7f35ee7fc640 1 --2- 172.21.15.90:0/2075800234 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55840008a4d0 0x55840045e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.739+0000 7f35f6ffd640 1 -- 172.21.15.90:0/2075800234 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55840008a4d0 msgr2=0x55840045e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:28.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.739+0000 7f35f6ffd640 1 --2- 172.21.15.90:0/2075800234 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55840008a4d0 0x55840045e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.739+0000 7f35f6ffd640 1 -- 172.21.15.90:0/2075800234 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584000b3390 msgr2=0x5584004644a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:28.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.739+0000 7f35f6ffd640 1 --2- 172.21.15.90:0/2075800234 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584000b3390 0x5584004644a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.739+0000 7f35f6ffd640 1 -- 172.21.15.90:0/2075800234 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584001957d0 con 0x5584000926b0 2024-04-25T11:46:28.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.739+0000 7f35ee7fc640 1 --2- 172.21.15.90:0/2075800234 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584000b3390 0x5584004644a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:46:28.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.739+0000 7f35f6ffd640 1 --2- 172.21.15.90:0/2075800234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584000926b0 0x55840045eba0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f35e8000c00 tx=0x7f35e8005eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=212513068d7478ad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:28.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.739+0000 7f35de7fc640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-04-25T11:46:28.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.739+0000 7f35de7fc640 1 -- 172.21.15.90:0/2075800234 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e8027020 con 0x5584000926b0 2024-04-25T11:46:28.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.739+0000 7f35de7fc640 1 -- 172.21.15.90:0/2075800234 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7f35e8018780 con 0x5584000926b0 2024-04-25T11:46:28.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.739+0000 7f36043ed740 1 -- 172.21.15.90:0/2075800234 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35e8018af0 con 0x5584000926b0 2024-04-25T11:46:28.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.739+0000 7f35de7fc640 1 -- 172.21.15.90:0/2075800234 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f35e801d040 con 0x5584000926b0 2024-04-25T11:46:28.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.740+0000 7f36043ed740 10 client.27688 fetch_fsmap learned FSMap version 28 2024-04-25T11:46:28.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.740+0000 7f36043ed740 1 -- 172.21.15.90:0/2075800234 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5584000cb3f0 con 0x5584000926b0 2024-04-25T11:46:28.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.740+0000 7f35de7fc640 1 -- 172.21.15.90:0/2075800234 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35e8018af0 con 0x5584000926b0 2024-04-25T11:46:28.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.740+0000 7f35de7fc640 1 -- 172.21.15.90:0/2075800234 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f35e8005320 con 0x5584000926b0 2024-04-25T11:46:28.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.740+0000 7f36043ed740 10 client.27688 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:46:28.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.740+0000 7f36043ed740 10 client.27688 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:46:28.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.740+0000 7f36043ed740 20 client.27688 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:28.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.740+0000 7f36043ed740 1 --2- 172.21.15.90:0/2075800234 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x558400071f60 0x558400470560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.741+0000 7f36043ed740 4 client.27688 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:46:28.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.741+0000 7f36043ed740 1 -- 172.21.15.90:0/2075800234 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584003c7990 con 0x558400071f60 2024-04-25T11:46:28.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.741+0000 7f35ee7fc640 1 --2- 172.21.15.90:0/2075800234 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x558400071f60 0x558400470560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.741+0000 7f35ee7fc640 1 --2- 172.21.15.90:0/2075800234 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x558400071f60 0x558400470560 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32b7fe20503ace38 server_cookie=2dbb05006b9f65da in_seq=0 out_seq=0 2024-04-25T11:46:28.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.741+0000 7f35de7fc640 10 client.27688 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:28.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.742+0000 7f35de7fc640 1 -- 172.21.15.90:0/2075800234 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5584003c7990 con 0x558400071f60 2024-04-25T11:46:28.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.742+0000 7f35de7fc640 10 client.27688 handle_command_reply: tid=0 2024-04-25T11:46:28.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.742+0000 7f36043ed740 1 client.27688 shutdown 2024-04-25T11:46:28.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.742+0000 7f36043ed740 1 -- 172.21.15.90:0/2075800234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584000926b0 msgr2=0x55840045eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:28.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.742+0000 7f36043ed740 1 --2- 172.21.15.90:0/2075800234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584000926b0 0x55840045eba0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f35e8000c00 tx=0x7f35e8005eb0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.742+0000 7f36043ed740 1 -- 172.21.15.90:0/2075800234 shutdown_connections 2024-04-25T11:46:28.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.742+0000 7f36043ed740 1 --2- 172.21.15.90:0/2075800234 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x558400071f60 0x558400470560 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.742+0000 7f36043ed740 1 --2- 172.21.15.90:0/2075800234 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584000b3390 0x5584004644a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.742+0000 7f36043ed740 1 --2- 172.21.15.90:0/2075800234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584000926b0 0x55840045eba0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.742+0000 7f36043ed740 1 --2- 172.21.15.90:0/2075800234 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55840008a4d0 0x55840045e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.742+0000 7f36043ed740 1 -- 172.21.15.90:0/2075800234 >> 172.21.15.90:0/2075800234 conn(0x5584000a74e0 msgr2=0x558400455780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:28.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.742+0000 7f35de7fc640 0 client.27688 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:28.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.742+0000 7f36043ed740 1 -- 172.21.15.90:0/2075800234 shutdown_connections 2024-04-25T11:46:28.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.742+0000 7f36043ed740 1 -- 172.21.15.90:0/2075800234 wait complete. 2024-04-25T11:46:28.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.743+0000 7f36043ed740 20 client.27688 trim_cache size 0 max 16384 2024-04-25T11:46:28.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.761+0000 7f36043ed740 1 Processor -- start 2024-04-25T11:46:28.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.761+0000 7f36043ed740 1 -- start start 2024-04-25T11:46:28.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.762+0000 7f36043ed740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55840008a4d0 0x558400455d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.762+0000 7f36043ed740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584000926b0 0x558400464420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.762+0000 7f36043ed740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584000b3390 0x558400464960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.762+0000 7f36043ed740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f35f81afcc0 con 0x5584000b3390 2024-04-25T11:46:28.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.762+0000 7f36043ed740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f35f81b1800 con 0x55840008a4d0 2024-04-25T11:46:28.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.762+0000 7f36043ed740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f35f81b0390 con 0x5584000926b0 2024-04-25T11:46:28.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.762+0000 7f35ee7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584000b3390 0x558400464960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.762+0000 7f35ee7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584000b3390 0x558400464960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48478/0 (socket says 172.21.15.90:48478) 2024-04-25T11:46:28.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.762+0000 7f35ee7fc640 1 -- 172.21.15.90:0/97970052 learned_addr learned my addr 172.21.15.90:0/97970052 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:28.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.762+0000 7f35f6ffd640 1 --2- 172.21.15.90:0/97970052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584000926b0 0x558400464420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.762+0000 7f35ee7fc640 1 --2- 172.21.15.90:0/97970052 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55840008a4d0 0x558400455d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.762+0000 7f35ee7fc640 1 -- 172.21.15.90:0/97970052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584000926b0 msgr2=0x558400464420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:28.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.762+0000 7f35ee7fc640 1 --2- 172.21.15.90:0/97970052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584000926b0 0x558400464420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.762+0000 7f35ee7fc640 1 -- 172.21.15.90:0/97970052 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55840008a4d0 msgr2=0x558400455d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:28.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.762+0000 7f35ee7fc640 1 --2- 172.21.15.90:0/97970052 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55840008a4d0 0x558400455d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.762+0000 7f35ee7fc640 1 -- 172.21.15.90:0/97970052 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55840008b0f0 con 0x5584000b3390 2024-04-25T11:46:28.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.762+0000 7f35f6ffd640 1 --2- 172.21.15.90:0/97970052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584000926b0 0x558400464420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:46:28.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.763+0000 7f35ee7fc640 1 --2- 172.21.15.90:0/97970052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584000b3390 0x558400464960 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f35e000cbb0 tx=0x7f35e0016040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eff72b3bd895fba0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:28.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.763+0000 7f35ecff9640 1 -- 172.21.15.90:0/97970052 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e0024ab0 con 0x5584000b3390 2024-04-25T11:46:28.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.763+0000 7f35ecff9640 1 -- 172.21.15.90:0/97970052 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35e0024c50 con 0x5584000b3390 2024-04-25T11:46:28.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.763+0000 7f35ecff9640 1 -- 172.21.15.90:0/97970052 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e0011530 con 0x5584000b3390 2024-04-25T11:46:28.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.764+0000 7f36043ed740 1 -- 172.21.15.90:0/97970052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584000b3390 msgr2=0x558400464960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:28.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.764+0000 7f36043ed740 1 --2- 172.21.15.90:0/97970052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584000b3390 0x558400464960 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f35e000cbb0 tx=0x7f35e0016040 comp rx=0 tx=0).stop 2024-04-25T11:46:28.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.764+0000 7f36043ed740 1 -- 172.21.15.90:0/97970052 shutdown_connections 2024-04-25T11:46:28.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.764+0000 7f36043ed740 1 --2- 172.21.15.90:0/97970052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584000b3390 0x558400464960 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.764+0000 7f36043ed740 1 --2- 172.21.15.90:0/97970052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584000926b0 0x558400464420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.764+0000 7f36043ed740 1 --2- 172.21.15.90:0/97970052 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55840008a4d0 0x558400455d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.764+0000 7f36043ed740 1 -- 172.21.15.90:0/97970052 >> 172.21.15.90:0/97970052 conn(0x5584000a74e0 msgr2=0x558400455660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:28.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.764+0000 7f36043ed740 1 -- 172.21.15.90:0/97970052 shutdown_connections 2024-04-25T11:46:28.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.764+0000 7f36043ed740 1 -- 172.21.15.90:0/97970052 wait complete. 2024-04-25T11:46:28.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.764+0000 7f36043ed740 1 -- start start 2024-04-25T11:46:28.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.764+0000 7f36043ed740 1 Processor -- start 2024-04-25T11:46:28.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.765+0000 7f36043ed740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55840008a4d0 0x558400465520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.765+0000 7f36043ed740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584000926b0 0x558400465a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.765+0000 7f36043ed740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584000b3390 0x558400465fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.765+0000 7f36043ed740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f35f81b1130 con 0x5584000b3390 2024-04-25T11:46:28.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.765+0000 7f36043ed740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f35f81b0a60 con 0x55840008a4d0 2024-04-25T11:46:28.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.765+0000 7f36043ed740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f35f81adcc0 con 0x5584000926b0 2024-04-25T11:46:28.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.765+0000 7f36043ed740 10 client.0 _ll_register_callbacks cb 0x7f35e801b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:28.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.765+0000 7f35ee7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584000b3390 0x558400465fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.765+0000 7f35ee7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584000b3390 0x558400465fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48492/0 (socket says 172.21.15.90:48492) 2024-04-25T11:46:28.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.765+0000 7f35ee7fc640 1 -- 172.21.15.90:0/3490722195 learned_addr learned my addr 172.21.15.90:0/3490722195 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:28.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.765+0000 7f35ee7fc640 1 --2- 172.21.15.90:0/3490722195 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55840008a4d0 0x558400465520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.765+0000 7f35f6ffd640 1 --2- 172.21.15.90:0/3490722195 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584000926b0 0x558400465a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.765+0000 7f35ee7fc640 1 -- 172.21.15.90:0/3490722195 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584000926b0 msgr2=0x558400465a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:28.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.765+0000 7f35ee7fc640 1 --2- 172.21.15.90:0/3490722195 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584000926b0 0x558400465a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.765+0000 7f35ee7fc640 1 -- 172.21.15.90:0/3490722195 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55840008a4d0 msgr2=0x558400465520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:28.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.765+0000 7f35ee7fc640 1 --2- 172.21.15.90:0/3490722195 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55840008a4d0 0x558400465520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.765+0000 7f35ee7fc640 1 -- 172.21.15.90:0/3490722195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55840037a4b0 con 0x5584000b3390 2024-04-25T11:46:28.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.765+0000 7f35ee7fc640 1 --2- 172.21.15.90:0/3490722195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584000b3390 0x558400465fa0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f35e0014390 tx=0x7f35e000c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f968eef8caa250f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:28.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.765+0000 7f35de7fc640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:46:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.765+0000 7f35de7fc640 1 -- 172.21.15.90:0/3490722195 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e0005050 con 0x5584000b3390 2024-04-25T11:46:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.765+0000 7f35de7fc640 1 -- 172.21.15.90:0/3490722195 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8702+0+0 (secure 0 0 0) 0x7f35e00051f0 con 0x5584000b3390 2024-04-25T11:46:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.765+0000 7f36043ed740 1 -- 172.21.15.90:0/3490722195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35e0024c50 con 0x5584000b3390 2024-04-25T11:46:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.765+0000 7f35de7fc640 1 -- 172.21.15.90:0/3490722195 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f35e0043050 con 0x5584000b3390 2024-04-25T11:46:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.765+0000 7f36043ed740 10 client.19338 fetch_fsmap learned FSMap version 28 2024-04-25T11:46:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.765+0000 7f36043ed740 1 -- 172.21.15.90:0/3490722195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5584004d7710 con 0x5584000b3390 2024-04-25T11:46:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.766+0000 7f35de7fc640 1 -- 172.21.15.90:0/3490722195 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35e0024c50 con 0x5584000b3390 2024-04-25T11:46:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.766+0000 7f35de7fc640 1 -- 172.21.15.90:0/3490722195 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f35e00047e0 con 0x5584000b3390 2024-04-25T11:46:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.766+0000 7f36043ed740 10 client.19338 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:46:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.766+0000 7f36043ed740 10 client.19338 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:46:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.766+0000 7f36043ed740 20 client.19338 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.766+0000 7f36043ed740 1 --2- 172.21.15.90:0/3490722195 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5584000473a0 0x5584004d9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.766+0000 7f36043ed740 4 client.19338 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:46:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.766+0000 7f36043ed740 1 -- 172.21.15.90:0/3490722195 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5584003c1800 con 0x5584000473a0 2024-04-25T11:46:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.766+0000 7f35f6ffd640 1 --2- 172.21.15.90:0/3490722195 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5584000473a0 0x5584004d9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:28.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.767+0000 7f35f6ffd640 1 --2- 172.21.15.90:0/3490722195 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5584000473a0 0x5584004d9cd0 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=502acb9d12918b36 server_cookie=76c14ed33f38c5b2 in_seq=0 out_seq=0 2024-04-25T11:46:28.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.767+0000 7f35de7fc640 10 client.19338 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:28.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.767+0000 7f35de7fc640 1 -- 172.21.15.90:0/3490722195 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5584003c1800 con 0x5584000473a0 2024-04-25T11:46:28.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.767+0000 7f35de7fc640 10 client.19338 handle_command_reply: tid=0 2024-04-25T11:46:28.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.767+0000 7f36043ed740 1 client.19338 shutdown 2024-04-25T11:46:28.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.767+0000 7f36043ed740 1 -- 172.21.15.90:0/3490722195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584000b3390 msgr2=0x558400465fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:28.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.767+0000 7f36043ed740 1 --2- 172.21.15.90:0/3490722195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584000b3390 0x558400465fa0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f35e0014390 tx=0x7f35e000c3b0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.767+0000 7f36043ed740 1 -- 172.21.15.90:0/3490722195 shutdown_connections 2024-04-25T11:46:28.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.767+0000 7f36043ed740 1 --2- 172.21.15.90:0/3490722195 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5584000473a0 0x5584004d9cd0 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.767+0000 7f36043ed740 1 --2- 172.21.15.90:0/3490722195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584000b3390 0x558400465fa0 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.767+0000 7f36043ed740 1 --2- 172.21.15.90:0/3490722195 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5584000926b0 0x558400465a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.767+0000 7f35de7fc640 0 client.19338 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:28.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.767+0000 7f36043ed740 1 --2- 172.21.15.90:0/3490722195 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55840008a4d0 0x558400465520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.767+0000 7f36043ed740 1 -- 172.21.15.90:0/3490722195 >> 172.21.15.90:0/3490722195 conn(0x5584000a74e0 msgr2=0x55840045c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:28.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.767+0000 7f36043ed740 1 -- 172.21.15.90:0/3490722195 shutdown_connections 2024-04-25T11:46:28.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.767+0000 7f36043ed740 1 -- 172.21.15.90:0/3490722195 wait complete. 2024-04-25T11:46:28.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.767+0000 7f36043ed740 20 client.19338 trim_cache size 0 max 16384 2024-04-25T11:46:28.758 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:46:28.758 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T11:46:28.758 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-25T11:46:28.758 INFO:teuthology.orchestra.run.smithi090.stdout: "68445a2a-81ad-41b3-8065-1660a34d0e82": { 2024-04-25T11:46:28.758 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-25T11:46:28.758 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "68445a2a-81ad-41b3-8065-1660a34d0e82", 2024-04-25T11:46:28.758 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-25T11:46:28.758 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:46:28.758 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:46:28.758 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:46:28.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.768+0000 7f35fd461640 1 -- 172.21.15.90:0/171439437 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f35e40afc60 msgr2=0x7f35e40b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:28.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.768+0000 7f35fd461640 1 --2- 172.21.15.90:0/171439437 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f35e40afc60 0x7f35e40b2170 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f35f00fe400 tx=0x7f35f00961c0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.768+0000 7f35fd461640 1 -- 172.21.15.90:0/171439437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f8109fb0 msgr2=0x7f35f819ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:28.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.768+0000 7f35fd461640 1 --2- 172.21.15.90:0/171439437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f8109fb0 0x7f35f819ef00 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f35f0006010 tx=0x7f35f002f380 comp rx=0 tx=0).stop 2024-04-25T11:46:28.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.768+0000 7f35fd461640 1 -- 172.21.15.90:0/171439437 shutdown_connections 2024-04-25T11:46:28.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.768+0000 7f35fd461640 1 --2- 172.21.15.90:0/171439437 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f35e40afc60 0x7f35e40b2170 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.768+0000 7f35fd461640 1 --2- 172.21.15.90:0/171439437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f8109fb0 0x7f35f819ef00 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.768+0000 7f35fd461640 1 --2- 172.21.15.90:0/171439437 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35f8109b10 0x7f35f819ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.768+0000 7f35fd461640 1 --2- 172.21.15.90:0/171439437 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f81042b0 0x7f35f819a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:28.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.768+0000 7f35fd461640 1 -- 172.21.15.90:0/171439437 >> 172.21.15.90:0/171439437 conn(0x7f35f80ff360 msgr2=0x7f35f8106420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:28.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.768+0000 7f35fd461640 1 -- 172.21.15.90:0/171439437 shutdown_connections 2024-04-25T11:46:28.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:28.768+0000 7f35fd461640 1 -- 172.21.15.90:0/171439437 wait complete. 2024-04-25T11:46:28.764 INFO:tasks.cephfs.filesystem:scrub status for tag:68445a2a-81ad-41b3-8065-1660a34d0e82 - {'path': '/', 'tag': '68445a2a-81ad-41b3-8065-1660a34d0e82', 'options': 'recursive,force'} 2024-04-25T11:46:29.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:29 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:46:29.391 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:29 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:46:29.391 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:29 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:46:29.391 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:29 smithi138 ceph-mon[32328]: scrub queued for path: / 2024-04-25T11:46:29.391 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:29 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:46:29.391 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:29 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:46:29.391 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:29 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:46:29.391 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:29 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:46:29.391 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:29 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:46:29.392 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:29 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:46:29.392 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:29 smithi156 ceph-mon[33045]: scrub queued for path: / 2024-04-25T11:46:29.392 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:29 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:46:29.392 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:29 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:46:29.392 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:29 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:46:29.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:29 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:46:29.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:29 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:46:29.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:29 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:46:29.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:29 smithi090 ceph-mon[29575]: scrub queued for path: / 2024-04-25T11:46:29.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:29 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:46:29.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:29 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:46:29.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:29 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:46:30.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:30 smithi138 ceph-mon[32328]: pgmap v354: 97 pgs: 14 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 57 active+clean; 22 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.6 KiB/s wr, 8 op/s 2024-04-25T11:46:30.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:30 smithi138 ceph-mon[32328]: scrub summary: active paths [/] 2024-04-25T11:46:30.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:30 smithi138 ceph-mon[32328]: scrub summary: idle+waiting paths [/] 2024-04-25T11:46:30.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:30 smithi156 ceph-mon[33045]: pgmap v354: 97 pgs: 14 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 57 active+clean; 22 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.6 KiB/s wr, 8 op/s 2024-04-25T11:46:30.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:30 smithi156 ceph-mon[33045]: scrub summary: active paths [/] 2024-04-25T11:46:30.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:30 smithi156 ceph-mon[33045]: scrub summary: idle+waiting paths [/] 2024-04-25T11:46:30.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:30 smithi090 ceph-mon[29575]: pgmap v354: 97 pgs: 14 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 57 active+clean; 22 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.6 KiB/s wr, 8 op/s 2024-04-25T11:46:30.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:30 smithi090 ceph-mon[29575]: scrub summary: active paths [/] 2024-04-25T11:46:30.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:30 smithi090 ceph-mon[29575]: scrub summary: idle+waiting paths [/] 2024-04-25T11:46:32.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:32 smithi138 ceph-mon[32328]: pgmap v355: 97 pgs: 4 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 83 active+clean; 21 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 11 KiB/s rd, 2.8 KiB/s wr, 15 op/s 2024-04-25T11:46:32.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:32 smithi156 ceph-mon[33045]: pgmap v355: 97 pgs: 4 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 83 active+clean; 21 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 11 KiB/s rd, 2.8 KiB/s wr, 15 op/s 2024-04-25T11:46:32.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:32 smithi090 ceph-mon[29575]: pgmap v355: 97 pgs: 4 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 83 active+clean; 21 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 11 KiB/s rd, 2.8 KiB/s wr, 15 op/s 2024-04-25T11:46:34.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:34 smithi138 ceph-mon[32328]: pgmap v356: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1007 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:46:34.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:34 smithi138 ceph-mon[32328]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T11:46:34.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:34 smithi156 ceph-mon[33045]: pgmap v356: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1007 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:46:34.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:34 smithi156 ceph-mon[33045]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T11:46:34.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:34 smithi090 ceph-mon[29575]: pgmap v356: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1007 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:46:34.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:34 smithi090 ceph-mon[29575]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T11:46:36.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:36 smithi138 ceph-mon[32328]: pgmap v358: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 12 KiB/s rd, 3.3 KiB/s wr, 13 op/s 2024-04-25T11:46:36.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:36 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:46:36.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:36 smithi156 ceph-mon[33045]: pgmap v358: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 12 KiB/s rd, 3.3 KiB/s wr, 13 op/s 2024-04-25T11:46:36.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:36 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:46:36.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:36 smithi090 ceph-mon[29575]: pgmap v358: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 12 KiB/s rd, 3.3 KiB/s wr, 13 op/s 2024-04-25T11:46:36.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:36 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:46:37.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:37 smithi138 ceph-mon[32328]: scrub summary: idle 2024-04-25T11:46:37.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:37 smithi156 ceph-mon[33045]: scrub summary: idle 2024-04-25T11:46:37.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:37 smithi090 ceph-mon[29575]: scrub summary: idle 2024-04-25T11:46:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.132+0000 7f0e16a6e640 1 -- 172.21.15.90:0/1789035542 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e101042c0 msgr2=0x7f0e10110690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.132+0000 7f0e16a6e640 1 --2- 172.21.15.90:0/1789035542 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e101042c0 0x7f0e10110690 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f0e08009ca0 tx=0x7f0e0802a8c0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.132+0000 7f0e16a6e640 1 -- 172.21.15.90:0/1789035542 shutdown_connections 2024-04-25T11:46:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.132+0000 7f0e16a6e640 1 --2- 172.21.15.90:0/1789035542 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e101042c0 0x7f0e10110690 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.132+0000 7f0e16a6e640 1 --2- 172.21.15.90:0/1789035542 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0e1010a010 0x7f0e10103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.132+0000 7f0e16a6e640 1 --2- 172.21.15.90:0/1789035542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e10109b70 0x7f0e10103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.132+0000 7f0e16a6e640 1 -- 172.21.15.90:0/1789035542 >> 172.21.15.90:0/1789035542 conn(0x7f0e100ff210 msgr2=0x7f0e10101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.132+0000 7f0e16a6e640 1 -- 172.21.15.90:0/1789035542 shutdown_connections 2024-04-25T11:46:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.132+0000 7f0e16a6e640 1 -- 172.21.15.90:0/1789035542 wait complete. 2024-04-25T11:46:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.133+0000 7f0e16a6e640 1 Processor -- start 2024-04-25T11:46:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.133+0000 7f0e16a6e640 1 -- start start 2024-04-25T11:46:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.133+0000 7f0e16a6e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0e101042c0 0x7f0e1019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.133+0000 7f0e16a6e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e10109b70 0x7f0e1019ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.133+0000 7f0e16a6e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e1010a010 0x7f0e1019ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.133+0000 7f0e16a6e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0e10113d60 con 0x7f0e10109b70 2024-04-25T11:46:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.133+0000 7f0e16a6e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0e10113be0 con 0x7f0e101042c0 2024-04-25T11:46:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.133+0000 7f0e16a6e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0e10113ee0 con 0x7f0e1010a010 2024-04-25T11:46:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.133+0000 7f0e0ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e10109b70 0x7f0e1019ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.133+0000 7f0e0ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e10109b70 0x7f0e1019ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51062/0 (socket says 172.21.15.90:51062) 2024-04-25T11:46:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.133+0000 7f0e0ffff640 1 -- 172.21.15.90:0/4003897223 learned_addr learned my addr 172.21.15.90:0/4003897223 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.133+0000 7f0e1480a640 1 --2- 172.21.15.90:0/4003897223 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0e101042c0 0x7f0e1019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:38.122 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.134+0000 7f0e1500b640 1 --2- 172.21.15.90:0/4003897223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e1010a010 0x7f0e1019ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:38.122 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.134+0000 7f0e0ffff640 1 -- 172.21.15.90:0/4003897223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e1010a010 msgr2=0x7f0e1019ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:38.122 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.134+0000 7f0e0ffff640 1 --2- 172.21.15.90:0/4003897223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e1010a010 0x7f0e1019ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.122 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.134+0000 7f0e0ffff640 1 -- 172.21.15.90:0/4003897223 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0e101042c0 msgr2=0x7f0e1019a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:38.122 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.134+0000 7f0e0ffff640 1 --2- 172.21.15.90:0/4003897223 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0e101042c0 0x7f0e1019a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.122 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.134+0000 7f0e0ffff640 1 -- 172.21.15.90:0/4003897223 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e1019f580 con 0x7f0e10109b70 2024-04-25T11:46:38.122 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.134+0000 7f0e1480a640 1 --2- 172.21.15.90:0/4003897223 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0e101042c0 0x7f0e1019a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:46:38.122 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.134+0000 7f0e0ffff640 1 --2- 172.21.15.90:0/4003897223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e10109b70 0x7f0e1019ac80 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc00d740 tx=0x7f0dfc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:38.122 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.134+0000 7f0e0dffb640 1 -- 172.21.15.90:0/4003897223 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc017070 con 0x7f0e10109b70 2024-04-25T11:46:38.122 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.134+0000 7f0e16a6e640 1 -- 172.21.15.90:0/4003897223 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e1019f870 con 0x7f0e10109b70 2024-04-25T11:46:38.122 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.134+0000 7f0e0dffb640 1 -- 172.21.15.90:0/4003897223 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0dfc00de50 con 0x7f0e10109b70 2024-04-25T11:46:38.122 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.134+0000 7f0e0dffb640 1 -- 172.21.15.90:0/4003897223 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc003c30 con 0x7f0e10109b70 2024-04-25T11:46:38.122 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.134+0000 7f0e16a6e640 1 -- 172.21.15.90:0/4003897223 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e101a71f0 con 0x7f0e10109b70 2024-04-25T11:46:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.136+0000 7f0e0dffb640 1 -- 172.21.15.90:0/4003897223 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f0dfc003dd0 con 0x7f0e10109b70 2024-04-25T11:46:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.137+0000 7f0e1da04740 1 Processor -- start 2024-04-25T11:46:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.137+0000 7f0e1da04740 1 -- start start 2024-04-25T11:46:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.137+0000 7f0e0dffb640 1 --2- 172.21.15.90:0/4003897223 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0e000afc50 0x7f0e000b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.137+0000 7f0e1da04740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71409b1b0 0x55b71443a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.137+0000 7f0e1da04740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7140722f0 0x55b71443aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.137+0000 7f0e1da04740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b71407a4d0 0x55b71443b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.137+0000 7f0e1da04740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b7143a1460 con 0x55b71409b1b0 2024-04-25T11:46:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.137+0000 7f0e1da04740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55b71435b590 con 0x55b71407a4d0 2024-04-25T11:46:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.137+0000 7f0e1da04740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b71439d420 con 0x55b7140722f0 2024-04-25T11:46:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.137+0000 7f0e1500b640 1 --2- 172.21.15.90:0/4003897223 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0e000afc50 0x7f0e000b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.137+0000 7f0e0dffb640 1 -- 172.21.15.90:0/4003897223 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7f0dfc0deb30 con 0x7f0e10109b70 2024-04-25T11:46:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.137+0000 7f0e1480a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71409b1b0 0x55b71443a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.137+0000 7f0e1480a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71409b1b0 0x55b71443a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51078/0 (socket says 172.21.15.90:51078) 2024-04-25T11:46:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.137+0000 7f0e1480a640 1 -- 172.21.15.90:0/3413907230 learned_addr learned my addr 172.21.15.90:0/3413907230 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.137+0000 7f0e1500b640 1 --2- 172.21.15.90:0/3413907230 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b71407a4d0 0x55b71443b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.138+0000 7f0e0ffff640 1 --2- 172.21.15.90:0/3413907230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7140722f0 0x55b71443aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.138+0000 7f0e1480a640 1 -- 172.21.15.90:0/3413907230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7140722f0 msgr2=0x55b71443aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.138+0000 7f0e1480a640 1 --2- 172.21.15.90:0/3413907230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7140722f0 0x55b71443aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.138+0000 7f0e1480a640 1 -- 172.21.15.90:0/3413907230 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b71407a4d0 msgr2=0x55b71443b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.138+0000 7f0e1480a640 1 --2- 172.21.15.90:0/3413907230 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b71407a4d0 0x55b71443b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.138+0000 7f0e1480a640 1 -- 172.21.15.90:0/3413907230 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7143680d0 con 0x55b71409b1b0 2024-04-25T11:46:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.138+0000 7f0e0ffff640 1 --2- 172.21.15.90:0/3413907230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7140722f0 0x55b71443aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:46:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.138+0000 7f0e1500b640 1 --2- 172.21.15.90:0/4003897223 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0e000afc50 0x7f0e000b2160 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0e080094e0 tx=0x7f0e08009360 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.138+0000 7f0e1480a640 1 --2- 172.21.15.90:0/3413907230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71409b1b0 0x55b71443a9a0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f0e0400bd30 tx=0x7f0e0400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c4f22754d838c29 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.138+0000 7f0dfaffd640 1 -- 172.21.15.90:0/3413907230 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e04015070 con 0x55b71409b1b0 2024-04-25T11:46:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.138+0000 7f0dfaffd640 1 -- 172.21.15.90:0/3413907230 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e0400be20 con 0x55b71409b1b0 2024-04-25T11:46:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.138+0000 7f0dfaffd640 1 -- 172.21.15.90:0/3413907230 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e040118d0 con 0x55b71409b1b0 2024-04-25T11:46:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.139+0000 7f0e1da04740 1 -- 172.21.15.90:0/3413907230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71409b1b0 msgr2=0x55b71443a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.139+0000 7f0e1da04740 1 --2- 172.21.15.90:0/3413907230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71409b1b0 0x55b71443a9a0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f0e0400bd30 tx=0x7f0e0400f730 comp rx=0 tx=0).stop 2024-04-25T11:46:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.139+0000 7f0e1da04740 1 -- 172.21.15.90:0/3413907230 shutdown_connections 2024-04-25T11:46:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.139+0000 7f0e1da04740 1 --2- 172.21.15.90:0/3413907230 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b71407a4d0 0x55b71443b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.139+0000 7f0e1da04740 1 --2- 172.21.15.90:0/3413907230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b7140722f0 0x55b71443aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.139+0000 7f0e1da04740 1 --2- 172.21.15.90:0/3413907230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71409b1b0 0x55b71443a9a0 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.139+0000 7f0e1da04740 1 -- 172.21.15.90:0/3413907230 >> 172.21.15.90:0/3413907230 conn(0x55b71408f300 msgr2=0x55b7144372a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.139+0000 7f0e1da04740 1 -- 172.21.15.90:0/3413907230 shutdown_connections 2024-04-25T11:46:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.139+0000 7f0e1da04740 1 -- 172.21.15.90:0/3413907230 wait complete. 2024-04-25T11:46:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.139+0000 7f0e1da04740 1 -- start start 2024-04-25T11:46:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.139+0000 7f0e1da04740 1 Processor -- start 2024-04-25T11:46:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.140+0000 7f0e1da04740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7140722f0 0x55b71444afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.140+0000 7f0e1da04740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b71407a4d0 0x55b71444b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.140+0000 7f0e1da04740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71409b1b0 0x55b714449430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.140+0000 7f0e1da04740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b71444c030 con 0x55b71409b1b0 2024-04-25T11:46:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.140+0000 7f0e1da04740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55b714328b60 con 0x55b7140722f0 2024-04-25T11:46:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.140+0000 7f0e1da04740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b71444c1b0 con 0x55b71407a4d0 2024-04-25T11:46:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.140+0000 7f0e1da04740 10 client.0 _ll_register_callbacks cb 0x55b71434d2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.140+0000 7f0e1500b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71409b1b0 0x55b714449430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.140+0000 7f0e1500b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71409b1b0 0x55b714449430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51086/0 (socket says 172.21.15.90:51086) 2024-04-25T11:46:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.140+0000 7f0e1500b640 1 -- 172.21.15.90:0/384112132 learned_addr learned my addr 172.21.15.90:0/384112132 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.140+0000 7f0e0ffff640 1 --2- 172.21.15.90:0/384112132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b71407a4d0 0x55b71444b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.140+0000 7f0e1480a640 1 --2- 172.21.15.90:0/384112132 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7140722f0 0x55b71444afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.140+0000 7f0e1500b640 1 -- 172.21.15.90:0/384112132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b71407a4d0 msgr2=0x55b71444b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.140+0000 7f0e1500b640 1 --2- 172.21.15.90:0/384112132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b71407a4d0 0x55b71444b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.140+0000 7f0e1500b640 1 -- 172.21.15.90:0/384112132 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7140722f0 msgr2=0x55b71444afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.140+0000 7f0e1500b640 1 --2- 172.21.15.90:0/384112132 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7140722f0 0x55b71444afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.140+0000 7f0e1500b640 1 -- 172.21.15.90:0/384112132 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7140b3210 con 0x55b71409b1b0 2024-04-25T11:46:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.141+0000 7f0e1500b640 1 --2- 172.21.15.90:0/384112132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71409b1b0 0x55b714449430 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f0e08003bf0 tx=0x7f0e080095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eba197b5cd653d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.141+0000 7f0deffff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:46:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.141+0000 7f0deffff640 1 -- 172.21.15.90:0/384112132 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e08050020 con 0x55b71409b1b0 2024-04-25T11:46:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.141+0000 7f0deffff640 1 -- 172.21.15.90:0/384112132 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7f0e080045c0 con 0x55b71409b1b0 2024-04-25T11:46:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.141+0000 7f0e1da04740 1 -- 172.21.15.90:0/384112132 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e0400be20 con 0x55b71409b1b0 2024-04-25T11:46:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.141+0000 7f0deffff640 1 -- 172.21.15.90:0/384112132 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0e0805b050 con 0x55b71409b1b0 2024-04-25T11:46:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.141+0000 7f0e1da04740 10 client.19350 fetch_fsmap learned FSMap version 28 2024-04-25T11:46:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.141+0000 7f0e1da04740 1 -- 172.21.15.90:0/384112132 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b71417f620 con 0x55b71409b1b0 2024-04-25T11:46:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.141+0000 7f0deffff640 1 -- 172.21.15.90:0/384112132 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e0400be20 con 0x55b71409b1b0 2024-04-25T11:46:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.141+0000 7f0deffff640 1 -- 172.21.15.90:0/384112132 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f0e080305c0 con 0x55b71409b1b0 2024-04-25T11:46:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.141+0000 7f0e1da04740 10 client.19350 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:46:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.141+0000 7f0e1da04740 10 client.19350 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:46:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.141+0000 7f0e1da04740 20 client.19350 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.141+0000 7f0e1da04740 1 --2- 172.21.15.90:0/384112132 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b714059d80 0x55b71445bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.141+0000 7f0e1da04740 4 client.19350 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:46:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.141+0000 7f0e1da04740 1 -- 172.21.15.90:0/384112132 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b7143af070 con 0x55b714059d80 2024-04-25T11:46:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.142+0000 7f0e1480a640 1 --2- 172.21.15.90:0/384112132 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b714059d80 0x55b71445bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.142+0000 7f0deffff640 10 client.19350 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.142+0000 7f0e1480a640 1 --2- 172.21.15.90:0/384112132 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b714059d80 0x55b71445bbd0 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=226df41837b514b server_cookie=ff88ff3b783911f3 in_seq=0 out_seq=0 2024-04-25T11:46:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.143+0000 7f0deffff640 1 -- 172.21.15.90:0/384112132 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b7143af070 con 0x55b714059d80 2024-04-25T11:46:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.143+0000 7f0deffff640 10 client.19350 handle_command_reply: tid=0 2024-04-25T11:46:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.143+0000 7f0e1da04740 1 client.19350 shutdown 2024-04-25T11:46:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.143+0000 7f0e1da04740 1 -- 172.21.15.90:0/384112132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71409b1b0 msgr2=0x55b714449430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.143+0000 7f0e1da04740 1 --2- 172.21.15.90:0/384112132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71409b1b0 0x55b714449430 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f0e08003bf0 tx=0x7f0e080095c0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.143+0000 7f0e1da04740 1 -- 172.21.15.90:0/384112132 shutdown_connections 2024-04-25T11:46:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.143+0000 7f0e1da04740 1 --2- 172.21.15.90:0/384112132 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b714059d80 0x55b71445bbd0 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.143+0000 7f0e1da04740 1 --2- 172.21.15.90:0/384112132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71409b1b0 0x55b714449430 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.143+0000 7f0deffff640 0 client.19350 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.143+0000 7f0e1da04740 1 --2- 172.21.15.90:0/384112132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b71407a4d0 0x55b71444b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.143+0000 7f0e1da04740 1 --2- 172.21.15.90:0/384112132 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7140722f0 0x55b71444afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.143+0000 7f0e1da04740 1 -- 172.21.15.90:0/384112132 >> 172.21.15.90:0/384112132 conn(0x55b71408f300 msgr2=0x55b71443c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.143+0000 7f0e1da04740 1 -- 172.21.15.90:0/384112132 shutdown_connections 2024-04-25T11:46:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.143+0000 7f0e1da04740 1 -- 172.21.15.90:0/384112132 wait complete. 2024-04-25T11:46:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.143+0000 7f0e1da04740 20 client.19350 trim_cache size 0 max 16384 2024-04-25T11:46:38.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.158+0000 7f0e1da04740 1 Processor -- start 2024-04-25T11:46:38.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.158+0000 7f0e1da04740 1 -- start start 2024-04-25T11:46:38.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.159+0000 7f0e1da04740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7140722f0 0x55b7144613d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:38.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.159+0000 7f0e1da04740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71407a4d0 0x55b714461910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:38.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.159+0000 7f0e1da04740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b71409b1b0 0x55b714448cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:38.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.159+0000 7f0e1da04740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0e101ada90 con 0x55b71407a4d0 2024-04-25T11:46:38.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.159+0000 7f0e1da04740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0e101af5d0 con 0x55b7140722f0 2024-04-25T11:46:38.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.159+0000 7f0e1da04740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0e101ae160 con 0x55b71409b1b0 2024-04-25T11:46:38.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.159+0000 7f0e0ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71407a4d0 0x55b714461910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:38.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.159+0000 7f0e0ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71407a4d0 0x55b714461910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51088/0 (socket says 172.21.15.90:51088) 2024-04-25T11:46:38.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.159+0000 7f0e0ffff640 1 -- 172.21.15.90:0/1605610153 learned_addr learned my addr 172.21.15.90:0/1605610153 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:38.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.159+0000 7f0e1500b640 1 --2- 172.21.15.90:0/1605610153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b71409b1b0 0x55b714448cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:38.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.159+0000 7f0e0ffff640 1 -- 172.21.15.90:0/1605610153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b71409b1b0 msgr2=0x55b714448cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:38.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.159+0000 7f0e1480a640 1 --2- 172.21.15.90:0/1605610153 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7140722f0 0x55b7144613d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:38.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.159+0000 7f0e0ffff640 1 --2- 172.21.15.90:0/1605610153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b71409b1b0 0x55b714448cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.159+0000 7f0e0ffff640 1 -- 172.21.15.90:0/1605610153 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7140722f0 msgr2=0x55b7144613d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:38.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.159+0000 7f0e0ffff640 1 --2- 172.21.15.90:0/1605610153 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7140722f0 0x55b7144613d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.159+0000 7f0e0ffff640 1 -- 172.21.15.90:0/1605610153 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7144314c0 con 0x55b71407a4d0 2024-04-25T11:46:38.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.159+0000 7f0e0ffff640 1 --2- 172.21.15.90:0/1605610153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71407a4d0 0x55b714461910 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc0dc7f0 tx=0x7f0dfc0a6480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e21afde32bc74d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:38.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.159+0000 7f0dfa7fc640 1 -- 172.21.15.90:0/1605610153 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc020040 con 0x55b71407a4d0 2024-04-25T11:46:38.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.159+0000 7f0dfa7fc640 1 -- 172.21.15.90:0/1605610153 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0dfc0af3e0 con 0x55b71407a4d0 2024-04-25T11:46:38.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.159+0000 7f0dfa7fc640 1 -- 172.21.15.90:0/1605610153 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc0a2ed0 con 0x55b71407a4d0 2024-04-25T11:46:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.160+0000 7f0e1da04740 1 -- 172.21.15.90:0/1605610153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71407a4d0 msgr2=0x55b714461910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.160+0000 7f0e1da04740 1 --2- 172.21.15.90:0/1605610153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71407a4d0 0x55b714461910 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc0dc7f0 tx=0x7f0dfc0a6480 comp rx=0 tx=0).stop 2024-04-25T11:46:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.160+0000 7f0e1da04740 1 -- 172.21.15.90:0/1605610153 shutdown_connections 2024-04-25T11:46:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.160+0000 7f0e1da04740 1 --2- 172.21.15.90:0/1605610153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b71409b1b0 0x55b714448cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.160+0000 7f0e1da04740 1 --2- 172.21.15.90:0/1605610153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71407a4d0 0x55b714461910 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.160+0000 7f0e1da04740 1 --2- 172.21.15.90:0/1605610153 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7140722f0 0x55b7144613d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.160+0000 7f0e1da04740 1 -- 172.21.15.90:0/1605610153 >> 172.21.15.90:0/1605610153 conn(0x55b71408f300 msgr2=0x55b714447d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.160+0000 7f0e1da04740 1 -- 172.21.15.90:0/1605610153 shutdown_connections 2024-04-25T11:46:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.160+0000 7f0e1da04740 1 -- 172.21.15.90:0/1605610153 wait complete. 2024-04-25T11:46:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.160+0000 7f0e1da04740 1 -- start start 2024-04-25T11:46:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.161+0000 7f0e1da04740 1 Processor -- start 2024-04-25T11:46:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.161+0000 7f0e1da04740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7140722f0 0x55b7144512b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.161+0000 7f0e1da04740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71407a4d0 0x55b7144517f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.161+0000 7f0e1da04740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b71409b1b0 0x55b71444f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.161+0000 7f0e1da04740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0e101aef00 con 0x55b71407a4d0 2024-04-25T11:46:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.161+0000 7f0e1da04740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0e101ae830 con 0x55b7140722f0 2024-04-25T11:46:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.161+0000 7f0e1da04740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0e101aba90 con 0x55b71409b1b0 2024-04-25T11:46:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.161+0000 7f0e1da04740 10 client.0 _ll_register_callbacks cb 0x7f0e08031f90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.161+0000 7f0e0ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71407a4d0 0x55b7144517f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.161+0000 7f0e0ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71407a4d0 0x55b7144517f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51096/0 (socket says 172.21.15.90:51096) 2024-04-25T11:46:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.161+0000 7f0e0ffff640 1 -- 172.21.15.90:0/2481846120 learned_addr learned my addr 172.21.15.90:0/2481846120 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.161+0000 7f0e1480a640 1 --2- 172.21.15.90:0/2481846120 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7140722f0 0x55b7144512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.161+0000 7f0e1500b640 1 --2- 172.21.15.90:0/2481846120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b71409b1b0 0x55b71444f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.161+0000 7f0e0ffff640 1 -- 172.21.15.90:0/2481846120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b71409b1b0 msgr2=0x55b71444f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.161+0000 7f0e0ffff640 1 --2- 172.21.15.90:0/2481846120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b71409b1b0 0x55b71444f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.161+0000 7f0e0ffff640 1 -- 172.21.15.90:0/2481846120 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7140722f0 msgr2=0x55b7144512b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.161+0000 7f0e0ffff640 1 --2- 172.21.15.90:0/2481846120 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7140722f0 0x55b7144512b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.161+0000 7f0e0ffff640 1 -- 172.21.15.90:0/2481846120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b71444fe40 con 0x55b71407a4d0 2024-04-25T11:46:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.161+0000 7f0e1480a640 1 --2- 172.21.15.90:0/2481846120 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7140722f0 0x55b7144512b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:46:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.162+0000 7f0e0ffff640 1 --2- 172.21.15.90:0/2481846120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71407a4d0 0x55b7144517f0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc022ff0 tx=0x7f0dfc0ab080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18ed6bdab2d29987 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.162+0000 7f0deffff640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:46:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.162+0000 7f0deffff640 1 -- 172.21.15.90:0/2481846120 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc06b030 con 0x55b71407a4d0 2024-04-25T11:46:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.162+0000 7f0deffff640 1 -- 172.21.15.90:0/2481846120 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7f0dfc0a3230 con 0x55b71407a4d0 2024-04-25T11:46:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.162+0000 7f0e1da04740 1 -- 172.21.15.90:0/2481846120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b7143a8ee0 con 0x55b71407a4d0 2024-04-25T11:46:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.162+0000 7f0deffff640 1 -- 172.21.15.90:0/2481846120 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0dfc020070 con 0x55b71407a4d0 2024-04-25T11:46:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.162+0000 7f0e1da04740 10 client.19356 fetch_fsmap learned FSMap version 28 2024-04-25T11:46:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.162+0000 7f0e1da04740 1 -- 172.21.15.90:0/2481846120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b7144c4890 con 0x55b71407a4d0 2024-04-25T11:46:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.162+0000 7f0deffff640 1 -- 172.21.15.90:0/2481846120 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55b7143a8ee0 con 0x55b71407a4d0 2024-04-25T11:46:38.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.162+0000 7f0deffff640 1 -- 172.21.15.90:0/2481846120 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f0dfc0a2ed0 con 0x55b71407a4d0 2024-04-25T11:46:38.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.162+0000 7f0e1da04740 10 client.19356 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T11:46:38.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.162+0000 7f0e1da04740 10 client.19356 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T11:46:38.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.162+0000 7f0e1da04740 20 client.19356 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:38.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.162+0000 7f0e1da04740 1 --2- 172.21.15.90:0/2481846120 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b71402f1c0 0x55b7144c6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:38.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.162+0000 7f0e1da04740 4 client.19356 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T11:46:38.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.162+0000 7f0e1da04740 1 -- 172.21.15.90:0/2481846120 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b7143245f0 con 0x55b71402f1c0 2024-04-25T11:46:38.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.163+0000 7f0e1480a640 1 --2- 172.21.15.90:0/2481846120 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b71402f1c0 0x55b7144c6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:38.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.163+0000 7f0deffff640 10 client.19356 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:38.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.163+0000 7f0e1480a640 1 --2- 172.21.15.90:0/2481846120 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b71402f1c0 0x55b7144c6fa0 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ac1d4dc4bf56edf server_cookie=ff2c78718480fed6 in_seq=0 out_seq=0 2024-04-25T11:46:38.155 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.169+0000 7f0deffff640 1 -- 172.21.15.90:0/2481846120 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b7143245f0 con 0x55b71402f1c0 2024-04-25T11:46:38.155 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.169+0000 7f0deffff640 10 client.19356 handle_command_reply: tid=0 2024-04-25T11:46:38.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.169+0000 7f0e1da04740 1 client.19356 shutdown 2024-04-25T11:46:38.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.170+0000 7f0e1da04740 1 -- 172.21.15.90:0/2481846120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71407a4d0 msgr2=0x55b7144517f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:38.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.170+0000 7f0e1da04740 1 --2- 172.21.15.90:0/2481846120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71407a4d0 0x55b7144517f0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc022ff0 tx=0x7f0dfc0ab080 comp rx=0 tx=0).stop 2024-04-25T11:46:38.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.170+0000 7f0e1da04740 1 -- 172.21.15.90:0/2481846120 shutdown_connections 2024-04-25T11:46:38.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.170+0000 7f0e1da04740 1 --2- 172.21.15.90:0/2481846120 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55b71402f1c0 0x55b7144c6fa0 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.170+0000 7f0e1da04740 1 --2- 172.21.15.90:0/2481846120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b71409b1b0 0x55b71444f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.170+0000 7f0e1da04740 1 --2- 172.21.15.90:0/2481846120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b71407a4d0 0x55b7144517f0 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.170+0000 7f0deffff640 0 client.19356 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:38.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.170+0000 7f0e1da04740 1 --2- 172.21.15.90:0/2481846120 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55b7140722f0 0x55b7144512b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.170+0000 7f0e1da04740 1 -- 172.21.15.90:0/2481846120 >> 172.21.15.90:0/2481846120 conn(0x55b71408f300 msgr2=0x55b714448cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:38.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.170+0000 7f0e1da04740 1 -- 172.21.15.90:0/2481846120 shutdown_connections 2024-04-25T11:46:38.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.170+0000 7f0e1da04740 1 -- 172.21.15.90:0/2481846120 wait complete. 2024-04-25T11:46:38.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.170+0000 7f0e1da04740 20 client.19356 trim_cache size 0 max 16384 2024-04-25T11:46:38.157 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-25T11:46:38.157 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-25T11:46:38.157 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-25T11:46:38.157 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-25T11:46:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.170+0000 7f0e16a6e640 1 -- 172.21.15.90:0/4003897223 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0e000afc50 msgr2=0x7f0e000b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.170+0000 7f0e16a6e640 1 --2- 172.21.15.90:0/4003897223 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0e000afc50 0x7f0e000b2160 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0e080094e0 tx=0x7f0e08009360 comp rx=0 tx=0).stop 2024-04-25T11:46:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.171+0000 7f0e16a6e640 1 -- 172.21.15.90:0/4003897223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e10109b70 msgr2=0x7f0e1019ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.171+0000 7f0e16a6e640 1 --2- 172.21.15.90:0/4003897223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e10109b70 0x7f0e1019ac80 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc00d740 tx=0x7f0dfc00dc00 comp rx=0 tx=0).stop 2024-04-25T11:46:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.171+0000 7f0e16a6e640 1 -- 172.21.15.90:0/4003897223 shutdown_connections 2024-04-25T11:46:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.171+0000 7f0e16a6e640 1 --2- 172.21.15.90:0/4003897223 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f0e000afc50 0x7f0e000b2160 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.171+0000 7f0e16a6e640 1 --2- 172.21.15.90:0/4003897223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e1010a010 0x7f0e1019ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.171+0000 7f0e16a6e640 1 --2- 172.21.15.90:0/4003897223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e10109b70 0x7f0e1019ac80 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.171+0000 7f0e16a6e640 1 --2- 172.21.15.90:0/4003897223 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0e101042c0 0x7f0e1019a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.171+0000 7f0e16a6e640 1 -- 172.21.15.90:0/4003897223 >> 172.21.15.90:0/4003897223 conn(0x7f0e100ff210 msgr2=0x7f0e101065b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.171+0000 7f0e16a6e640 1 -- 172.21.15.90:0/4003897223 shutdown_connections 2024-04-25T11:46:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-25T11:46:38.171+0000 7f0e16a6e640 1 -- 172.21.15.90:0/4003897223 wait complete. 2024-04-25T11:46:38.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:38 smithi138 ceph-mon[32328]: pgmap v359: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 8.6 KiB/s rd, 307 B/s wr, 13 op/s 2024-04-25T11:46:38.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:38 smithi156 ceph-mon[33045]: pgmap v359: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 8.6 KiB/s rd, 307 B/s wr, 13 op/s 2024-04-25T11:46:38.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:38 smithi090 ceph-mon[29575]: pgmap v359: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 8.6 KiB/s rd, 307 B/s wr, 13 op/s 2024-04-25T11:46:40.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:40 smithi138 ceph-mon[32328]: pgmap v360: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 8.6 KiB/s rd, 307 B/s wr, 13 op/s 2024-04-25T11:46:40.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:40 smithi156 ceph-mon[33045]: pgmap v360: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 8.6 KiB/s rd, 307 B/s wr, 13 op/s 2024-04-25T11:46:40.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:40 smithi090 ceph-mon[29575]: pgmap v360: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 8.6 KiB/s rd, 307 B/s wr, 13 op/s 2024-04-25T11:46:42.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:42 smithi138 ceph-mon[32328]: pgmap v361: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 8.4 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:46:42.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:42 smithi156 ceph-mon[33045]: pgmap v361: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 8.4 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:46:42.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:42 smithi090 ceph-mon[29575]: pgmap v361: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 8.4 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:46:44.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:44 smithi156 ceph-mon[33045]: pgmap v362: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 7.9 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:46:44.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:44 smithi090 ceph-mon[29575]: pgmap v362: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 7.9 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:46:44.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:44 smithi138 ceph-mon[32328]: pgmap v362: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 7.9 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-25T11:46:46.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:46 smithi156 ceph-mon[33045]: pgmap v363: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 7.5 KiB/s rd, 3.0 KiB/s wr, 11 op/s 2024-04-25T11:46:46.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:46 smithi090 ceph-mon[29575]: pgmap v363: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 7.5 KiB/s rd, 3.0 KiB/s wr, 11 op/s 2024-04-25T11:46:46.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:46 smithi138 ceph-mon[32328]: pgmap v363: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 7.5 KiB/s rd, 3.0 KiB/s wr, 11 op/s 2024-04-25T11:46:47.094 INFO:tasks.workunit.client.0.smithi090.stdout:stopping iogen 2024-04-25T11:46:47.095 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'stopping iogen' 2024-04-25T11:46:47.095 INFO:tasks.workunit.client.0.smithi090.stderr:+ ./iogen -k 2024-04-25T11:46:47.096 INFO:tasks.workunit.client.0.smithi090.stdout:OK 2024-04-25T11:46:47.096 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo OK 2024-04-25T11:46:48.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:48 smithi156 ceph-mon[33045]: pgmap v364: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:46:48.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:48 smithi090 ceph-mon[29575]: pgmap v364: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:46:48.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:48 smithi138 ceph-mon[32328]: pgmap v364: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T11:46:49.113 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T11:46:49.113 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T11:46:49.789 INFO:tasks.workunit:Stopping ['suites/iogen.sh'] on client.0... 2024-04-25T11:46:49.789 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-25T11:46:50.056 DEBUG:teuthology.parallel:result is None 2024-04-25T11:46:50.056 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T11:46:50.091 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T11:46:50.091 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-25T11:46:50.102 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T11:46:50.140 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-25T11:46:50.152 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-25T11:46:50.152 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T11:46:50.152 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T11:46:50.152 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-25T11:46:50.205 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-25T11:46:50.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-25T11:46:50.331 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:50 smithi090 ceph-mon[29575]: pgmap v365: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:46:50.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:50 smithi156 ceph-mon[33045]: pgmap v365: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:46:50.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.435+0000 7f22edfc0640 1 -- 172.21.15.90:0/2499607395 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22e810b6e0 msgr2=0x7f22e810db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:50.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.435+0000 7f22edfc0640 1 --2- 172.21.15.90:0/2499607395 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22e810b6e0 0x7f22e810db00 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f22d0009900 tx=0x7f22d002a410 comp rx=0 tx=0).stop 2024-04-25T11:46:50.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.435+0000 7f22edfc0640 1 -- 172.21.15.90:0/2499607395 shutdown_connections 2024-04-25T11:46:50.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.435+0000 7f22edfc0640 1 --2- 172.21.15.90:0/2499607395 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f22e810e0d0 0x7f22e81105c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:50.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.435+0000 7f22edfc0640 1 --2- 172.21.15.90:0/2499607395 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22e810b6e0 0x7f22e810db00 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:50.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.435+0000 7f22edfc0640 1 --2- 172.21.15.90:0/2499607395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22e8104c80 0x7f22e8105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:50.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.435+0000 7f22edfc0640 1 -- 172.21.15.90:0/2499607395 >> 172.21.15.90:0/2499607395 conn(0x7f22e8078150 msgr2=0x7f22e8078590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:50.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.435+0000 7f22edfc0640 1 -- 172.21.15.90:0/2499607395 shutdown_connections 2024-04-25T11:46:50.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.435+0000 7f22edfc0640 1 -- 172.21.15.90:0/2499607395 wait complete. 2024-04-25T11:46:50.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.436+0000 7f22edfc0640 1 Processor -- start 2024-04-25T11:46:50.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.436+0000 7f22edfc0640 1 -- start start 2024-04-25T11:46:50.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.436+0000 7f22edfc0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22e8104c80 0x7f22e8102540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:50.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.436+0000 7f22edfc0640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f22e810b6e0 0x7f22e8102a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:50.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.436+0000 7f22edfc0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22e810e0d0 0x7f22e8102fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:50.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.436+0000 7f22edfc0640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f22e8113d70 con 0x7f22e8104c80 2024-04-25T11:46:50.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.436+0000 7f22edfc0640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f22e8113bf0 con 0x7f22e810b6e0 2024-04-25T11:46:50.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.436+0000 7f22edfc0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f22e8113ef0 con 0x7f22e810e0d0 2024-04-25T11:46:50.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.436+0000 7f22e77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22e8104c80 0x7f22e8102540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:50.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.436+0000 7f22e77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22e8104c80 0x7f22e8102540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51708/0 (socket says 172.21.15.90:51708) 2024-04-25T11:46:50.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.436+0000 7f22e77fe640 1 -- 172.21.15.90:0/1082352097 learned_addr learned my addr 172.21.15.90:0/1082352097 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:50.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.436+0000 7f22dffff640 1 --2- 172.21.15.90:0/1082352097 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f22e810b6e0 0x7f22e8102a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:50.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.436+0000 7f22e77fe640 1 -- 172.21.15.90:0/1082352097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22e810e0d0 msgr2=0x7f22e8102fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:50.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.436+0000 7f22e77fe640 1 --2- 172.21.15.90:0/1082352097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22e810e0d0 0x7f22e8102fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:50.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.436+0000 7f22e77fe640 1 -- 172.21.15.90:0/1082352097 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f22e810b6e0 msgr2=0x7f22e8102a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:50.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.436+0000 7f22e77fe640 1 --2- 172.21.15.90:0/1082352097 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f22e810b6e0 0x7f22e8102a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:50.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.436+0000 7f22e77fe640 1 -- 172.21.15.90:0/1082352097 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22e81a8ae0 con 0x7f22e8104c80 2024-04-25T11:46:50.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.436+0000 7f22dffff640 1 --2- 172.21.15.90:0/1082352097 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f22e810b6e0 0x7f22e8102a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:46:50.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.436+0000 7f22e77fe640 1 --2- 172.21.15.90:0/1082352097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22e8104c80 0x7f22e8102540 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f22d800b4a0 tx=0x7f22d800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:50.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.437+0000 7f22e57fa640 1 -- 172.21.15.90:0/1082352097 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d8014070 con 0x7f22e8104c80 2024-04-25T11:46:50.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.437+0000 7f22e57fa640 1 -- 172.21.15.90:0/1082352097 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22d8010ac0 con 0x7f22e8104c80 2024-04-25T11:46:50.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.437+0000 7f22e57fa640 1 -- 172.21.15.90:0/1082352097 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d8017c30 con 0x7f22e8104c80 2024-04-25T11:46:50.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.437+0000 7f22edfc0640 1 -- 172.21.15.90:0/1082352097 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22e81a8dd0 con 0x7f22e8104c80 2024-04-25T11:46:50.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.437+0000 7f22edfc0640 1 -- 172.21.15.90:0/1082352097 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22e81a9260 con 0x7f22e8104c80 2024-04-25T11:46:50.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.438+0000 7f22edfc0640 1 -- 172.21.15.90:0/1082352097 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22e8050740 con 0x7f22e8104c80 2024-04-25T11:46:50.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.439+0000 7f22e57fa640 1 -- 172.21.15.90:0/1082352097 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f22d8017070 con 0x7f22e8104c80 2024-04-25T11:46:50.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.440+0000 7f22e57fa640 1 --2- 172.21.15.90:0/1082352097 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f22d40afcd0 0x7f22d40b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:50.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.440+0000 7f22dffff640 1 --2- 172.21.15.90:0/1082352097 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f22d40afcd0 0x7f22d40b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:50.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.440+0000 7f22e57fa640 1 -- 172.21.15.90:0/1082352097 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7f22d80de7b0 con 0x7f22e8104c80 2024-04-25T11:46:50.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.440+0000 7f22dffff640 1 --2- 172.21.15.90:0/1082352097 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f22d40afcd0 0x7f22d40b21e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f22d00096c0 tx=0x7f22d0031400 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:50.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.443+0000 7f22e57fa640 1 -- 172.21.15.90:0/1082352097 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f22d80a70d0 con 0x7f22e8104c80 2024-04-25T11:46:50.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:50 smithi138 ceph-mon[32328]: pgmap v365: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-04-25T11:46:50.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.646+0000 7f22edfc0640 1 -- 172.21.15.90:0/1082352097 --> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f22e8101950 con 0x7f22d40afcd0 2024-04-25T11:46:50.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.654+0000 7f22e57fa640 1 -- 172.21.15.90:0/1082352097 <== mgr.15366 v2:172.21.15.90:6800/4078377640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7f22e8101950 con 0x7f22d40afcd0 2024-04-25T11:46:50.639 INFO:teuthology.orchestra.run.smithi090.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-25T00:00:00", "created": "2024-04-25T11:35:13", "first": "2024-04-25T11:36:00", "last": "2024-04-25T11:46:00", "last_pruned": "2024-04-25T11:46:00", "created_count": 11, "pruned_count": 5, "active": true} 2024-04-25T11:46:50.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.656+0000 7f22edfc0640 1 -- 172.21.15.90:0/1082352097 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f22d40afcd0 msgr2=0x7f22d40b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:50.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.656+0000 7f22edfc0640 1 --2- 172.21.15.90:0/1082352097 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f22d40afcd0 0x7f22d40b21e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f22d00096c0 tx=0x7f22d0031400 comp rx=0 tx=0).stop 2024-04-25T11:46:50.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.656+0000 7f22edfc0640 1 -- 172.21.15.90:0/1082352097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22e8104c80 msgr2=0x7f22e8102540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:50.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.656+0000 7f22edfc0640 1 --2- 172.21.15.90:0/1082352097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22e8104c80 0x7f22e8102540 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f22d800b4a0 tx=0x7f22d800b960 comp rx=0 tx=0).stop 2024-04-25T11:46:50.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.656+0000 7f22edfc0640 1 -- 172.21.15.90:0/1082352097 shutdown_connections 2024-04-25T11:46:50.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.656+0000 7f22edfc0640 1 --2- 172.21.15.90:0/1082352097 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f22d40afcd0 0x7f22d40b21e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:50.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.657+0000 7f22edfc0640 1 --2- 172.21.15.90:0/1082352097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22e810e0d0 0x7f22e8102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:50.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.657+0000 7f22edfc0640 1 --2- 172.21.15.90:0/1082352097 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f22e810b6e0 0x7f22e8102a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:50.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.657+0000 7f22edfc0640 1 --2- 172.21.15.90:0/1082352097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22e8104c80 0x7f22e8102540 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:50.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.657+0000 7f22edfc0640 1 -- 172.21.15.90:0/1082352097 >> 172.21.15.90:0/1082352097 conn(0x7f22e8078150 msgr2=0x7f22e8101240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:50.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.657+0000 7f22edfc0640 1 -- 172.21.15.90:0/1082352097 shutdown_connections 2024-04-25T11:46:50.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.657+0000 7f22edfc0640 1 -- 172.21.15.90:0/1082352097 wait complete. 2024-04-25T11:46:50.653 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-25T11:46:50.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.819+0000 7fa64a8e0640 1 -- 172.21.15.90:0/963767317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa644104c90 msgr2=0x7fa6441050a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:50.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.819+0000 7fa64a8e0640 1 --2- 172.21.15.90:0/963767317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa644104c90 0x7fa6441050a0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fa634009900 tx=0x7fa63402a410 comp rx=0 tx=0).stop 2024-04-25T11:46:50.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.819+0000 7fa64a8e0640 1 -- 172.21.15.90:0/963767317 shutdown_connections 2024-04-25T11:46:50.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.819+0000 7fa64a8e0640 1 --2- 172.21.15.90:0/963767317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa64410e0e0 0x7fa6441105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:50.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.819+0000 7fa64a8e0640 1 --2- 172.21.15.90:0/963767317 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa64410b6f0 0x7fa64410db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:50.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.819+0000 7fa64a8e0640 1 --2- 172.21.15.90:0/963767317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa644104c90 0x7fa6441050a0 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:50.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.819+0000 7fa64a8e0640 1 -- 172.21.15.90:0/963767317 >> 172.21.15.90:0/963767317 conn(0x7fa644078080 msgr2=0x7fa6440784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:50.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.819+0000 7fa64a8e0640 1 -- 172.21.15.90:0/963767317 shutdown_connections 2024-04-25T11:46:50.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.819+0000 7fa64a8e0640 1 -- 172.21.15.90:0/963767317 wait complete. 2024-04-25T11:46:50.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.819+0000 7fa64a8e0640 1 Processor -- start 2024-04-25T11:46:50.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.819+0000 7fa64a8e0640 1 -- start start 2024-04-25T11:46:50.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.820+0000 7fa64a8e0640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa644104c90 0x7fa64419a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:50.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.820+0000 7fa64a8e0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa64410b6f0 0x7fa64419ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:50.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.820+0000 7fa64a8e0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa64410e0e0 0x7fa64419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:50.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.820+0000 7fa64a8e0640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa644113d40 con 0x7fa64410b6f0 2024-04-25T11:46:50.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.820+0000 7fa64a8e0640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa644113bc0 con 0x7fa644104c90 2024-04-25T11:46:50.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.820+0000 7fa64a8e0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa644113ec0 con 0x7fa64410e0e0 2024-04-25T11:46:50.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.820+0000 7fa6437fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa64410b6f0 0x7fa64419ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:50.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.820+0000 7fa6437fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa64410b6f0 0x7fa64419ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51728/0 (socket says 172.21.15.90:51728) 2024-04-25T11:46:50.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.820+0000 7fa6437fe640 1 -- 172.21.15.90:0/2753889717 learned_addr learned my addr 172.21.15.90:0/2753889717 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:50.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.820+0000 7fa643fff640 1 --2- 172.21.15.90:0/2753889717 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa644104c90 0x7fa64419a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:50.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.820+0000 7fa648e7d640 1 --2- 172.21.15.90:0/2753889717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa64410e0e0 0x7fa64419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:50.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.820+0000 7fa6437fe640 1 -- 172.21.15.90:0/2753889717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa64410e0e0 msgr2=0x7fa64419efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:50.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.820+0000 7fa6437fe640 1 --2- 172.21.15.90:0/2753889717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa64410e0e0 0x7fa64419efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:50.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.820+0000 7fa6437fe640 1 -- 172.21.15.90:0/2753889717 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa644104c90 msgr2=0x7fa64419a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:50.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.820+0000 7fa6437fe640 1 --2- 172.21.15.90:0/2753889717 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa644104c90 0x7fa64419a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:50.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.820+0000 7fa6437fe640 1 -- 172.21.15.90:0/2753889717 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa64419f5e0 con 0x7fa64410b6f0 2024-04-25T11:46:50.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.821+0000 7fa6437fe640 1 --2- 172.21.15.90:0/2753889717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa64410b6f0 0x7fa64419ace0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fa63800d950 tx=0x7fa63800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:50.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.821+0000 7fa6417fa640 1 -- 172.21.15.90:0/2753889717 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa638017070 con 0x7fa64410b6f0 2024-04-25T11:46:50.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.821+0000 7fa6417fa640 1 -- 172.21.15.90:0/2753889717 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa638009c30 con 0x7fa64410b6f0 2024-04-25T11:46:50.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.821+0000 7fa64a8e0640 1 -- 172.21.15.90:0/2753889717 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa64419f8d0 con 0x7fa64410b6f0 2024-04-25T11:46:50.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.821+0000 7fa6417fa640 1 -- 172.21.15.90:0/2753889717 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63800f620 con 0x7fa64410b6f0 2024-04-25T11:46:50.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.821+0000 7fa64a8e0640 1 -- 172.21.15.90:0/2753889717 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6441a7250 con 0x7fa64410b6f0 2024-04-25T11:46:50.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.822+0000 7fa64a8e0640 1 -- 172.21.15.90:0/2753889717 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa644050740 con 0x7fa64410b6f0 2024-04-25T11:46:50.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.823+0000 7fa6417fa640 1 -- 172.21.15.90:0/2753889717 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fa638010460 con 0x7fa64410b6f0 2024-04-25T11:46:50.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.824+0000 7fa6417fa640 1 --2- 172.21.15.90:0/2753889717 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa62c0afc50 0x7fa62c0b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:50.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.824+0000 7fa643fff640 1 --2- 172.21.15.90:0/2753889717 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa62c0afc50 0x7fa62c0b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:50.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.824+0000 7fa6417fa640 1 -- 172.21.15.90:0/2753889717 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7fa6380deda0 con 0x7fa64410b6f0 2024-04-25T11:46:50.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.824+0000 7fa643fff640 1 --2- 172.21.15.90:0/2753889717 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa62c0afc50 0x7fa62c0b2160 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa634005f70 tx=0x7fa634000bc0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:50.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:50.827+0000 7fa6417fa640 1 -- 172.21.15.90:0/2753889717 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa6380a75b0 con 0x7fa64410b6f0 2024-04-25T11:46:51.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:51.025+0000 7fa64a8e0640 1 -- 172.21.15.90:0/2753889717 --> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fa6441018c0 con 0x7fa62c0afc50 2024-04-25T11:46:51.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:51.032+0000 7fa6417fa640 1 -- 172.21.15.90:0/2753889717 <== mgr.15366 v2:172.21.15.90:6800/4078377640 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fa6441018c0 con 0x7fa62c0afc50 2024-04-25T11:46:51.018 INFO:teuthology.orchestra.run.smithi090.stdout:/ 1m 6m3h 2024-04-25T11:46:51.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:51.034+0000 7fa64a8e0640 1 -- 172.21.15.90:0/2753889717 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa62c0afc50 msgr2=0x7fa62c0b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:51.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:51.034+0000 7fa64a8e0640 1 --2- 172.21.15.90:0/2753889717 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa62c0afc50 0x7fa62c0b2160 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa634005f70 tx=0x7fa634000bc0 comp rx=0 tx=0).stop 2024-04-25T11:46:51.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:51.034+0000 7fa64a8e0640 1 -- 172.21.15.90:0/2753889717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa64410b6f0 msgr2=0x7fa64419ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:51.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:51.034+0000 7fa64a8e0640 1 --2- 172.21.15.90:0/2753889717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa64410b6f0 0x7fa64419ace0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fa63800d950 tx=0x7fa63800de10 comp rx=0 tx=0).stop 2024-04-25T11:46:51.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:51.034+0000 7fa64a8e0640 1 -- 172.21.15.90:0/2753889717 shutdown_connections 2024-04-25T11:46:51.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:51.035+0000 7fa64a8e0640 1 --2- 172.21.15.90:0/2753889717 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa62c0afc50 0x7fa62c0b2160 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:51.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:51.035+0000 7fa64a8e0640 1 --2- 172.21.15.90:0/2753889717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa64410e0e0 0x7fa64419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:51.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:51.035+0000 7fa64a8e0640 1 --2- 172.21.15.90:0/2753889717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa64410b6f0 0x7fa64419ace0 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:51.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:51.035+0000 7fa64a8e0640 1 --2- 172.21.15.90:0/2753889717 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa644104c90 0x7fa64419a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:51.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:51.035+0000 7fa64a8e0640 1 -- 172.21.15.90:0/2753889717 >> 172.21.15.90:0/2753889717 conn(0x7fa644078080 msgr2=0x7fa6441011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:51.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:51.035+0000 7fa64a8e0640 1 -- 172.21.15.90:0/2753889717 shutdown_connections 2024-04-25T11:46:51.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:51.035+0000 7fa64a8e0640 1 -- 172.21.15.90:0/2753889717 wait complete. 2024-04-25T11:46:51.031 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-25T11:46:51.042 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-25T11:46:52.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:52 smithi156 ceph-mon[33045]: pgmap v366: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 13 op/s 2024-04-25T11:46:52.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:52 smithi156 ceph-mon[33045]: from='client.19362 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:46:52.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:52 smithi156 ceph-mon[33045]: from='client.19368 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:46:52.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:52 smithi090 ceph-mon[29575]: pgmap v366: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 13 op/s 2024-04-25T11:46:52.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:52 smithi090 ceph-mon[29575]: from='client.19362 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:46:52.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:52 smithi090 ceph-mon[29575]: from='client.19368 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:46:52.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:52 smithi138 ceph-mon[32328]: pgmap v366: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 13 op/s 2024-04-25T11:46:52.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:52 smithi138 ceph-mon[32328]: from='client.19362 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:46:52.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:52 smithi138 ceph-mon[32328]: from='client.19368 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T11:46:53.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:53 smithi156 ceph-mon[33045]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T11:46:53.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:53 smithi090 ceph-mon[29575]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T11:46:53.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:53 smithi138 ceph-mon[32328]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T11:46:54.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:54 smithi156 ceph-mon[33045]: pgmap v367: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 13 op/s 2024-04-25T11:46:54.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:54 smithi090 ceph-mon[29575]: pgmap v367: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 13 op/s 2024-04-25T11:46:54.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:54 smithi138 ceph-mon[32328]: pgmap v367: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 13 op/s 2024-04-25T11:46:56.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:56 smithi156 ceph-mon[33045]: pgmap v368: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 101 op/s 2024-04-25T11:46:56.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:56 smithi090 ceph-mon[29575]: pgmap v368: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 101 op/s 2024-04-25T11:46:56.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:56 smithi138 ceph-mon[32328]: pgmap v368: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 101 op/s 2024-04-25T11:46:58.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:46:58 smithi090 ceph-mon[29575]: pgmap v369: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 28 KiB/s rd, 21 KiB/s wr, 188 op/s 2024-04-25T11:46:58.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:46:58 smithi138 ceph-mon[32328]: pgmap v369: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 28 KiB/s rd, 21 KiB/s wr, 188 op/s 2024-04-25T11:46:58.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:46:58 smithi156 ceph-mon[33045]: pgmap v369: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 28 KiB/s rd, 21 KiB/s wr, 188 op/s 2024-04-25T11:46:58.765 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T11:46:58.765 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T11:46:58.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.904+0000 7fb8a93b4640 1 -- 172.21.15.90:0/292296159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8a4102af0 msgr2=0x7fb8a4102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:58.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.904+0000 7fb8a93b4640 1 --2- 172.21.15.90:0/292296159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8a4102af0 0x7fb8a4102f00 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fb88c009900 tx=0x7fb88c02a410 comp rx=0 tx=0).stop 2024-04-25T11:46:58.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.904+0000 7fb8a93b4640 1 -- 172.21.15.90:0/292296159 shutdown_connections 2024-04-25T11:46:58.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.904+0000 7fb8a93b4640 1 --2- 172.21.15.90:0/292296159 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8a410d380 0x7fb8a410f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.904+0000 7fb8a93b4640 1 --2- 172.21.15.90:0/292296159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8a4102af0 0x7fb8a4102f00 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.904+0000 7fb8a93b4640 1 --2- 172.21.15.90:0/292296159 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb8a4108a20 0x7fb8a41025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.904+0000 7fb8a93b4640 1 -- 172.21.15.90:0/292296159 >> 172.21.15.90:0/292296159 conn(0x7fb8a4078100 msgr2=0x7fb8a4078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:58.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.904+0000 7fb8a93b4640 1 -- 172.21.15.90:0/292296159 shutdown_connections 2024-04-25T11:46:58.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.904+0000 7fb8a93b4640 1 -- 172.21.15.90:0/292296159 wait complete. 2024-04-25T11:46:58.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.904+0000 7fb8a93b4640 1 Processor -- start 2024-04-25T11:46:58.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.904+0000 7fb8a93b4640 1 -- start start 2024-04-25T11:46:58.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.905+0000 7fb8a93b4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8a4102af0 0x7fb8a419b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:58.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.905+0000 7fb8a93b4640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8a4108a20 0x7fb8a419baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:58.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.905+0000 7fb8a93b4640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb8a410d380 0x7fb8a419c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:58.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.905+0000 7fb8a93b4640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a4112be0 con 0x7fb8a4108a20 2024-04-25T11:46:58.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.905+0000 7fb8a93b4640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a4112a60 con 0x7fb8a410d380 2024-04-25T11:46:58.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.905+0000 7fb8a93b4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a4112d60 con 0x7fb8a4102af0 2024-04-25T11:46:58.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.905+0000 7fb89bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8a4108a20 0x7fb8a419baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:58.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.905+0000 7fb89bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8a4108a20 0x7fb8a419baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59646/0 (socket says 172.21.15.90:59646) 2024-04-25T11:46:58.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.905+0000 7fb89bfff640 1 -- 172.21.15.90:0/4002767204 learned_addr learned my addr 172.21.15.90:0/4002767204 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:58.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.905+0000 7fb8a2ffd640 1 --2- 172.21.15.90:0/4002767204 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8a4102af0 0x7fb8a419b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:58.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.905+0000 7fb8a37fe640 1 --2- 172.21.15.90:0/4002767204 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb8a410d380 0x7fb8a419c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:58.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.905+0000 7fb89bfff640 1 -- 172.21.15.90:0/4002767204 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8a4102af0 msgr2=0x7fb8a419b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:58.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.905+0000 7fb89bfff640 1 --2- 172.21.15.90:0/4002767204 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8a4102af0 0x7fb8a419b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.905+0000 7fb89bfff640 1 -- 172.21.15.90:0/4002767204 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb8a410d380 msgr2=0x7fb8a419c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:58.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.905+0000 7fb89bfff640 1 --2- 172.21.15.90:0/4002767204 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb8a410d380 0x7fb8a419c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.905+0000 7fb89bfff640 1 -- 172.21.15.90:0/4002767204 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8a41a68a0 con 0x7fb8a4108a20 2024-04-25T11:46:58.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.905+0000 7fb8a2ffd640 1 --2- 172.21.15.90:0/4002767204 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8a4102af0 0x7fb8a419b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:46:58.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.905+0000 7fb89bfff640 1 --2- 172.21.15.90:0/4002767204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8a4108a20 0x7fb8a419baa0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fb88c006010 tx=0x7fb88c004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:58.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.905+0000 7fb8a0ff9640 1 -- 172.21.15.90:0/4002767204 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c03f030 con 0x7fb8a4108a20 2024-04-25T11:46:58.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.905+0000 7fb8a0ff9640 1 -- 172.21.15.90:0/4002767204 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb88c003aa0 con 0x7fb8a4108a20 2024-04-25T11:46:58.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.905+0000 7fb8a0ff9640 1 -- 172.21.15.90:0/4002767204 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c035070 con 0x7fb8a4108a20 2024-04-25T11:46:58.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.905+0000 7fb8a93b4640 1 -- 172.21.15.90:0/4002767204 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8a41a6b30 con 0x7fb8a4108a20 2024-04-25T11:46:58.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.906+0000 7fb8a93b4640 1 -- 172.21.15.90:0/4002767204 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8a41a7020 con 0x7fb8a4108a20 2024-04-25T11:46:58.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.908+0000 7fb8a0ff9640 1 -- 172.21.15.90:0/4002767204 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fb88c042030 con 0x7fb8a4108a20 2024-04-25T11:46:58.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.908+0000 7fb8b0261740 1 Processor -- start 2024-04-25T11:46:58.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.908+0000 7fb8b0261740 1 -- start start 2024-04-25T11:46:58.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.909+0000 7fb8b0261740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5605dc80e390 0x5605dcbaea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:58.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.909+0000 7fb8b0261740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dc7e54d0 0x5605dcbaefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:58.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.909+0000 7fb8b0261740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc7ed6b0 0x5605dcbbb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:58.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.909+0000 7fb8b0261740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5605dcb14d80 con 0x5605dc7ed6b0 2024-04-25T11:46:58.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.909+0000 7fb8b0261740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5605dcace8b0 con 0x5605dc80e390 2024-04-25T11:46:58.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.909+0000 7fb8b0261740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5605dcb10d40 con 0x5605dc7e54d0 2024-04-25T11:46:58.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.909+0000 7fb89bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc7ed6b0 0x5605dcbbb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:58.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.909+0000 7fb89bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc7ed6b0 0x5605dcbbb8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59652/0 (socket says 172.21.15.90:59652) 2024-04-25T11:46:58.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.909+0000 7fb89bfff640 1 -- 172.21.15.90:0/3662689595 learned_addr learned my addr 172.21.15.90:0/3662689595 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:58.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.909+0000 7fb8a37fe640 1 --2- 172.21.15.90:0/3662689595 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5605dc80e390 0x5605dcbaea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:58.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.909+0000 7fb8a2ffd640 1 --2- 172.21.15.90:0/3662689595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dc7e54d0 0x5605dcbaefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:58.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.909+0000 7fb8a0ff9640 1 --2- 172.21.15.90:0/4002767204 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8900afc60 0x7fb8900b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:58.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.909+0000 7fb8a37fe640 1 --2- 172.21.15.90:0/4002767204 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8900afc60 0x7fb8900b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:58.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.909+0000 7fb89bfff640 1 -- 172.21.15.90:0/3662689595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dc7e54d0 msgr2=0x5605dcbaefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:58.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.909+0000 7fb89bfff640 1 --2- 172.21.15.90:0/3662689595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dc7e54d0 0x5605dcbaefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.909+0000 7fb89bfff640 1 -- 172.21.15.90:0/3662689595 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5605dc80e390 msgr2=0x5605dcbaea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:58.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.909+0000 7fb89bfff640 1 --2- 172.21.15.90:0/3662689595 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5605dc80e390 0x5605dcbaea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.909+0000 7fb8a0ff9640 1 -- 172.21.15.90:0/4002767204 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7fb88c0ff6c0 con 0x7fb8a4108a20 2024-04-25T11:46:58.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.909+0000 7fb89bfff640 1 -- 172.21.15.90:0/3662689595 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605dca9fa40 con 0x5605dc7ed6b0 2024-04-25T11:46:58.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.910+0000 7fb89bfff640 1 --2- 172.21.15.90:0/3662689595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc7ed6b0 0x5605dcbbb8f0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fb88c0c7700 tx=0x7fb88c105530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22a1d05550625e64 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:58.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.910+0000 7fb8a37fe640 1 --2- 172.21.15.90:0/4002767204 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8900afc60 0x7fb8900b2170 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x5605dcbabbd0 tx=0x7fb89c004ab0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:58.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.910+0000 7fb8997fa640 1 -- 172.21.15.90:0/3662689595 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c049070 con 0x5605dc7ed6b0 2024-04-25T11:46:58.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.910+0000 7fb8997fa640 1 -- 172.21.15.90:0/3662689595 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb88c0c7280 con 0x5605dc7ed6b0 2024-04-25T11:46:58.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.910+0000 7fb8997fa640 1 -- 172.21.15.90:0/3662689595 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c0d05d0 con 0x5605dc7ed6b0 2024-04-25T11:46:58.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.911+0000 7fb8b0261740 1 -- 172.21.15.90:0/3662689595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc7ed6b0 msgr2=0x5605dcbbb8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:58.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.911+0000 7fb8b0261740 1 --2- 172.21.15.90:0/3662689595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc7ed6b0 0x5605dcbbb8f0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fb88c0c7700 tx=0x7fb88c105530 comp rx=0 tx=0).stop 2024-04-25T11:46:58.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.911+0000 7fb8b0261740 1 -- 172.21.15.90:0/3662689595 shutdown_connections 2024-04-25T11:46:58.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.911+0000 7fb8b0261740 1 --2- 172.21.15.90:0/3662689595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc7ed6b0 0x5605dcbbb8f0 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.911+0000 7fb8b0261740 1 --2- 172.21.15.90:0/3662689595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dc7e54d0 0x5605dcbaefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.911+0000 7fb8b0261740 1 --2- 172.21.15.90:0/3662689595 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5605dc80e390 0x5605dcbaea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.911+0000 7fb8b0261740 1 -- 172.21.15.90:0/3662689595 >> 172.21.15.90:0/3662689595 conn(0x5605dc8024e0 msgr2=0x5605dcbab210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:58.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.911+0000 7fb8b0261740 1 -- 172.21.15.90:0/3662689595 shutdown_connections 2024-04-25T11:46:58.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.911+0000 7fb8b0261740 1 -- 172.21.15.90:0/3662689595 wait complete. 2024-04-25T11:46:58.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.911+0000 7fb8b0261740 1 -- start start 2024-04-25T11:46:58.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.911+0000 7fb8b0261740 1 Processor -- start 2024-04-25T11:46:58.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.911+0000 7fb8b0261740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5605dc7e54d0 0x5605dcbb95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:58.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.911+0000 7fb8b0261740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dc7ed6b0 0x5605dcbb9b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:58.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.911+0000 7fb8b0261740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc80e390 0x5605dcbbf300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:58.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.911+0000 7fb8b0261740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5605dcbbbe30 con 0x5605dc80e390 2024-04-25T11:46:58.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.911+0000 7fb8b0261740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5605dca9bd10 con 0x5605dc7e54d0 2024-04-25T11:46:58.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.911+0000 7fb8b0261740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5605dcbbbfb0 con 0x5605dc7ed6b0 2024-04-25T11:46:58.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.911+0000 7fb8b0261740 10 client.0 _ll_register_callbacks cb 0x5605dcac8210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:58.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.912+0000 7fb8a37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc80e390 0x5605dcbbf300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:58.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.912+0000 7fb8a37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc80e390 0x5605dcbbf300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59658/0 (socket says 172.21.15.90:59658) 2024-04-25T11:46:58.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.912+0000 7fb8a37fe640 1 -- 172.21.15.90:0/2569015222 learned_addr learned my addr 172.21.15.90:0/2569015222 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:58.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.912+0000 7fb8a2ffd640 1 --2- 172.21.15.90:0/2569015222 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5605dc7e54d0 0x5605dcbb95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:58.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.912+0000 7fb89bfff640 1 --2- 172.21.15.90:0/2569015222 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dc7ed6b0 0x5605dcbb9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:58.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.912+0000 7fb8a37fe640 1 -- 172.21.15.90:0/2569015222 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dc7ed6b0 msgr2=0x5605dcbb9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:58.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.912+0000 7fb8a37fe640 1 --2- 172.21.15.90:0/2569015222 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dc7ed6b0 0x5605dcbb9b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.912+0000 7fb8a37fe640 1 -- 172.21.15.90:0/2569015222 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5605dc7e54d0 msgr2=0x5605dcbb95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:58.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.912+0000 7fb8a37fe640 1 --2- 172.21.15.90:0/2569015222 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5605dc7e54d0 0x5605dcbb95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.912+0000 7fb8a37fe640 1 -- 172.21.15.90:0/2569015222 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605dc8f07d0 con 0x5605dc80e390 2024-04-25T11:46:58.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.912+0000 7fb8a2ffd640 1 --2- 172.21.15.90:0/2569015222 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5605dc7e54d0 0x5605dcbb95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:46:58.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.912+0000 7fb8a37fe640 1 --2- 172.21.15.90:0/2569015222 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc80e390 0x5605dcbbf300 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fb89c014400 tx=0x7fb89c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ea7fa0cd1ab92b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:58.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.912+0000 7fb88a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:46:58.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.912+0000 7fb88a7fc640 1 -- 172.21.15.90:0/2569015222 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89c026020 con 0x5605dc80e390 2024-04-25T11:46:58.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.912+0000 7fb88a7fc640 1 -- 172.21.15.90:0/2569015222 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7fb89c021720 con 0x5605dc80e390 2024-04-25T11:46:58.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.912+0000 7fb8b0261740 1 -- 172.21.15.90:0/2569015222 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb88c0c7280 con 0x5605dc80e390 2024-04-25T11:46:58.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.912+0000 7fb88a7fc640 1 -- 172.21.15.90:0/2569015222 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb89c031050 con 0x5605dc80e390 2024-04-25T11:46:58.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.912+0000 7fb8b0261740 10 client.19380 fetch_fsmap learned FSMap version 29 2024-04-25T11:46:58.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.912+0000 7fb8b0261740 1 -- 172.21.15.90:0/2569015222 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5605dc8263f0 con 0x5605dc80e390 2024-04-25T11:46:58.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.913+0000 7fb88a7fc640 1 -- 172.21.15.90:0/2569015222 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb88c0c7280 con 0x5605dc80e390 2024-04-25T11:46:58.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.913+0000 7fb88a7fc640 1 -- 172.21.15.90:0/2569015222 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 29) ==== 8358+0+0 (secure 0 0 0) 0x7fb89c00f040 con 0x5605dc80e390 2024-04-25T11:46:58.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.913+0000 7fb8b0261740 10 client.19380 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T11:46:58.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.913+0000 7fb8b0261740 10 client.19380 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:46:58.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.913+0000 7fb8b0261740 20 client.19380 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:58.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.913+0000 7fb8b0261740 1 --2- 172.21.15.90:0/2569015222 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5605dc7ccf60 0x5605dcbcb700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:58.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.913+0000 7fb8b0261740 4 client.19380 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:46:58.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.913+0000 7fb8b0261740 1 -- 172.21.15.90:0/2569015222 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5605dcb22990 con 0x5605dc7ccf60 2024-04-25T11:46:58.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.913+0000 7fb8a2ffd640 1 --2- 172.21.15.90:0/2569015222 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5605dc7ccf60 0x5605dcbcb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:58.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.914+0000 7fb88a7fc640 10 client.19380 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:58.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.914+0000 7fb8a2ffd640 1 --2- 172.21.15.90:0/2569015222 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5605dc7ccf60 0x5605dcbcb700 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfc8c2ad4baa7447 server_cookie=9bde6bca5f763376 in_seq=0 out_seq=0 2024-04-25T11:46:58.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.914+0000 7fb88a7fc640 1 -- 172.21.15.90:0/2569015222 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5605dcb22990 con 0x5605dc7ccf60 2024-04-25T11:46:58.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.914+0000 7fb88a7fc640 10 client.19380 handle_command_reply: tid=0 2024-04-25T11:46:58.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.914+0000 7fb8b0261740 1 client.19380 shutdown 2024-04-25T11:46:58.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.914+0000 7fb8b0261740 1 -- 172.21.15.90:0/2569015222 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc80e390 msgr2=0x5605dcbbf300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:58.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.914+0000 7fb8b0261740 1 --2- 172.21.15.90:0/2569015222 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc80e390 0x5605dcbbf300 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fb89c014400 tx=0x7fb89c004a40 comp rx=0 tx=0).stop 2024-04-25T11:46:58.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.914+0000 7fb8b0261740 1 -- 172.21.15.90:0/2569015222 shutdown_connections 2024-04-25T11:46:58.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.914+0000 7fb8b0261740 1 --2- 172.21.15.90:0/2569015222 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5605dc7ccf60 0x5605dcbcb700 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.914+0000 7fb8b0261740 1 --2- 172.21.15.90:0/2569015222 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc80e390 0x5605dcbbf300 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.915+0000 7fb8b0261740 1 --2- 172.21.15.90:0/2569015222 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dc7ed6b0 0x5605dcbb9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.915+0000 7fb8b0261740 1 --2- 172.21.15.90:0/2569015222 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5605dc7e54d0 0x5605dcbb95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.915+0000 7fb8b0261740 1 -- 172.21.15.90:0/2569015222 >> 172.21.15.90:0/2569015222 conn(0x5605dc8024e0 msgr2=0x5605dcbb0780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:58.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.915+0000 7fb88a7fc640 0 client.19380 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:58.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.915+0000 7fb8b0261740 1 -- 172.21.15.90:0/2569015222 shutdown_connections 2024-04-25T11:46:58.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.915+0000 7fb8b0261740 1 -- 172.21.15.90:0/2569015222 wait complete. 2024-04-25T11:46:58.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.915+0000 7fb8b0261740 20 client.19380 trim_cache size 0 max 16384 2024-04-25T11:46:58.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.932+0000 7fb8b0261740 1 Processor -- start 2024-04-25T11:46:58.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.933+0000 7fb8b0261740 1 -- start start 2024-04-25T11:46:58.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.933+0000 7fb8b0261740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dc7e54d0 0x5605dcbaee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:58.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.933+0000 7fb8b0261740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5605dc7ed6b0 0x5605dcbbb550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:58.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.933+0000 7fb8b0261740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc80e390 0x5605dcbbf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:58.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.933+0000 7fb8b0261740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a41afb70 con 0x5605dc80e390 2024-04-25T11:46:58.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.933+0000 7fb8b0261740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a41b16e0 con 0x5605dc7ed6b0 2024-04-25T11:46:58.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.933+0000 7fb8b0261740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a41b0240 con 0x5605dc7e54d0 2024-04-25T11:46:58.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.933+0000 7fb8a37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc80e390 0x5605dcbbf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:58.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.933+0000 7fb8a37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc80e390 0x5605dcbbf270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59672/0 (socket says 172.21.15.90:59672) 2024-04-25T11:46:58.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.933+0000 7fb8a37fe640 1 -- 172.21.15.90:0/2596683507 learned_addr learned my addr 172.21.15.90:0/2596683507 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:58.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.933+0000 7fb8a2ffd640 1 --2- 172.21.15.90:0/2596683507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dc7e54d0 0x5605dcbaee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:58.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.933+0000 7fb89bfff640 1 --2- 172.21.15.90:0/2596683507 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5605dc7ed6b0 0x5605dcbbb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:58.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.933+0000 7fb8a37fe640 1 -- 172.21.15.90:0/2596683507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dc7e54d0 msgr2=0x5605dcbaee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:58.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.933+0000 7fb8a37fe640 1 --2- 172.21.15.90:0/2596683507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dc7e54d0 0x5605dcbaee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.933+0000 7fb8a37fe640 1 -- 172.21.15.90:0/2596683507 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5605dc7ed6b0 msgr2=0x5605dcbbb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:58.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.933+0000 7fb8a37fe640 1 --2- 172.21.15.90:0/2596683507 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5605dc7ed6b0 0x5605dcbbb550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.933+0000 7fb8a37fe640 1 -- 172.21.15.90:0/2596683507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605dc7e60f0 con 0x5605dc80e390 2024-04-25T11:46:58.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.934+0000 7fb8a37fe640 1 --2- 172.21.15.90:0/2596683507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc80e390 0x5605dcbbf270 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fb89c009610 tx=0x7fb89c009c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f34c52be627da90 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:58.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.934+0000 7fb898ff9640 1 -- 172.21.15.90:0/2596683507 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89c026040 con 0x5605dc80e390 2024-04-25T11:46:58.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.934+0000 7fb898ff9640 1 -- 172.21.15.90:0/2596683507 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb89c03a070 con 0x5605dc80e390 2024-04-25T11:46:58.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.934+0000 7fb898ff9640 1 -- 172.21.15.90:0/2596683507 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89c035430 con 0x5605dc80e390 2024-04-25T11:46:58.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.934+0000 7fb8b0261740 1 -- 172.21.15.90:0/2596683507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc80e390 msgr2=0x5605dcbbf270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:58.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.934+0000 7fb8b0261740 1 --2- 172.21.15.90:0/2596683507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc80e390 0x5605dcbbf270 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fb89c009610 tx=0x7fb89c009c50 comp rx=0 tx=0).stop 2024-04-25T11:46:58.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.935+0000 7fb8b0261740 1 -- 172.21.15.90:0/2596683507 shutdown_connections 2024-04-25T11:46:58.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.935+0000 7fb8b0261740 1 --2- 172.21.15.90:0/2596683507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc80e390 0x5605dcbbf270 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.935+0000 7fb8b0261740 1 --2- 172.21.15.90:0/2596683507 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5605dc7ed6b0 0x5605dcbbb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.935+0000 7fb8b0261740 1 --2- 172.21.15.90:0/2596683507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dc7e54d0 0x5605dcbaee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.935+0000 7fb8b0261740 1 -- 172.21.15.90:0/2596683507 >> 172.21.15.90:0/2596683507 conn(0x5605dc8024e0 msgr2=0x5605dcbb0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:58.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.935+0000 7fb8b0261740 1 -- 172.21.15.90:0/2596683507 shutdown_connections 2024-04-25T11:46:58.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.935+0000 7fb8b0261740 1 -- 172.21.15.90:0/2596683507 wait complete. 2024-04-25T11:46:58.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.935+0000 7fb8b0261740 1 -- start start 2024-04-25T11:46:58.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.935+0000 7fb8b0261740 1 Processor -- start 2024-04-25T11:46:58.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.935+0000 7fb8b0261740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dc7e54d0 0x5605dcbc22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:58.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.935+0000 7fb8b0261740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5605dc7ed6b0 0x5605dcbc2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:58.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.935+0000 7fb8b0261740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc80e390 0x5605dcbc2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:58.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.935+0000 7fb8b0261740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a41b1010 con 0x5605dc80e390 2024-04-25T11:46:58.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.935+0000 7fb8b0261740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a41b0910 con 0x5605dc7ed6b0 2024-04-25T11:46:58.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.935+0000 7fb8b0261740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a41adb70 con 0x5605dc7e54d0 2024-04-25T11:46:58.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.935+0000 7fb8b0261740 10 client.0 _ll_register_callbacks cb 0x7fb89c014710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:58.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.935+0000 7fb8a37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc80e390 0x5605dcbc2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:58.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.935+0000 7fb8a37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc80e390 0x5605dcbc2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59676/0 (socket says 172.21.15.90:59676) 2024-04-25T11:46:58.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.935+0000 7fb8a37fe640 1 -- 172.21.15.90:0/2983307049 learned_addr learned my addr 172.21.15.90:0/2983307049 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:58.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.936+0000 7fb89bfff640 1 --2- 172.21.15.90:0/2983307049 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5605dc7ed6b0 0x5605dcbc2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:58.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.936+0000 7fb8a2ffd640 1 --2- 172.21.15.90:0/2983307049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dc7e54d0 0x5605dcbc22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:58.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.936+0000 7fb8a37fe640 1 -- 172.21.15.90:0/2983307049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dc7e54d0 msgr2=0x5605dcbc22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:58.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.936+0000 7fb8a37fe640 1 --2- 172.21.15.90:0/2983307049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dc7e54d0 0x5605dcbc22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.936+0000 7fb8a37fe640 1 -- 172.21.15.90:0/2983307049 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5605dc7ed6b0 msgr2=0x5605dcbc2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:58.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.936+0000 7fb8a37fe640 1 --2- 172.21.15.90:0/2983307049 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5605dc7ed6b0 0x5605dcbc2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.936+0000 7fb8a37fe640 1 -- 172.21.15.90:0/2983307049 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605dcad54b0 con 0x5605dc80e390 2024-04-25T11:46:58.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.936+0000 7fb89bfff640 1 --2- 172.21.15.90:0/2983307049 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5605dc7ed6b0 0x5605dcbc2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:46:58.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.936+0000 7fb8a2ffd640 1 --2- 172.21.15.90:0/2983307049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dc7e54d0 0x5605dcbc22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:46:58.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.936+0000 7fb8a37fe640 1 --2- 172.21.15.90:0/2983307049 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc80e390 0x5605dcbc2d70 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fb89c0148e0 tx=0x7fb89c010e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=498f00ea70cf9556 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:58.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.936+0000 7fb88a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:46:58.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.936+0000 7fb88a7fc640 1 -- 172.21.15.90:0/2983307049 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89c026040 con 0x5605dc80e390 2024-04-25T11:46:58.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.936+0000 7fb88a7fc640 1 -- 172.21.15.90:0/2983307049 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7fb89c021930 con 0x5605dc80e390 2024-04-25T11:46:58.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.936+0000 7fb8b0261740 1 -- 172.21.15.90:0/2983307049 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb89c03a070 con 0x5605dc80e390 2024-04-25T11:46:58.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.936+0000 7fb88a7fc640 1 -- 172.21.15.90:0/2983307049 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb89c04c050 con 0x5605dc80e390 2024-04-25T11:46:58.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.936+0000 7fb8b0261740 10 client.19386 fetch_fsmap learned FSMap version 29 2024-04-25T11:46:58.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.936+0000 7fb8b0261740 1 -- 172.21.15.90:0/2983307049 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5605dcc32ad0 con 0x5605dc80e390 2024-04-25T11:46:58.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.936+0000 7fb88a7fc640 1 -- 172.21.15.90:0/2983307049 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb89c03a070 con 0x5605dc80e390 2024-04-25T11:46:58.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.937+0000 7fb88a7fc640 1 -- 172.21.15.90:0/2983307049 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 29) ==== 8358+0+0 (secure 0 0 0) 0x7fb89c03ea20 con 0x5605dc80e390 2024-04-25T11:46:58.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.937+0000 7fb8b0261740 10 client.19386 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T11:46:58.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.937+0000 7fb8b0261740 10 client.19386 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:46:58.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.937+0000 7fb8b0261740 20 client.19386 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:58.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.937+0000 7fb8b0261740 1 --2- 172.21.15.90:0/2983307049 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5605dc7a23a0 0x5605dcc35200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:58.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.937+0000 7fb8b0261740 4 client.19386 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T11:46:58.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.937+0000 7fb8b0261740 1 -- 172.21.15.90:0/2983307049 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5605dcb1c800 con 0x5605dc7a23a0 2024-04-25T11:46:58.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.937+0000 7fb8a2ffd640 1 --2- 172.21.15.90:0/2983307049 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5605dc7a23a0 0x5605dcc35200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:58.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.938+0000 7fb88a7fc640 10 client.19386 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:58.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.938+0000 7fb8a2ffd640 1 --2- 172.21.15.90:0/2983307049 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5605dc7a23a0 0x5605dcc35200 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be9732ec76c68e8d server_cookie=d37bb476ef1e78d1 in_seq=0 out_seq=0 2024-04-25T11:46:58.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.938+0000 7fb88a7fc640 1 -- 172.21.15.90:0/2983307049 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5605dcb1c800 con 0x5605dc7a23a0 2024-04-25T11:46:58.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.938+0000 7fb88a7fc640 10 client.19386 handle_command_reply: tid=0 2024-04-25T11:46:58.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.938+0000 7fb8b0261740 1 client.19386 shutdown 2024-04-25T11:46:58.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.938+0000 7fb8b0261740 1 -- 172.21.15.90:0/2983307049 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc80e390 msgr2=0x5605dcbc2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:58.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.938+0000 7fb8b0261740 1 --2- 172.21.15.90:0/2983307049 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc80e390 0x5605dcbc2d70 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fb89c0148e0 tx=0x7fb89c010e00 comp rx=0 tx=0).stop 2024-04-25T11:46:58.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.938+0000 7fb8b0261740 1 -- 172.21.15.90:0/2983307049 shutdown_connections 2024-04-25T11:46:58.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.938+0000 7fb8b0261740 1 --2- 172.21.15.90:0/2983307049 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x5605dc7a23a0 0x5605dcc35200 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.938+0000 7fb8b0261740 1 --2- 172.21.15.90:0/2983307049 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dc80e390 0x5605dcbc2d70 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.938+0000 7fb8b0261740 1 --2- 172.21.15.90:0/2983307049 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x5605dc7ed6b0 0x5605dcbc2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.938+0000 7fb8b0261740 1 --2- 172.21.15.90:0/2983307049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dc7e54d0 0x5605dcbc22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.938+0000 7fb88a7fc640 0 client.19386 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:46:58.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.938+0000 7fb8b0261740 1 -- 172.21.15.90:0/2983307049 >> 172.21.15.90:0/2983307049 conn(0x5605dc8024e0 msgr2=0x5605dcbcfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:58.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.938+0000 7fb8b0261740 1 -- 172.21.15.90:0/2983307049 shutdown_connections 2024-04-25T11:46:58.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.938+0000 7fb8b0261740 1 -- 172.21.15.90:0/2983307049 wait complete. 2024-04-25T11:46:58.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.938+0000 7fb8b0261740 20 client.19386 trim_cache size 0 max 16384 2024-04-25T11:46:58.931 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:46:58.931 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-25T11:46:58.931 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-25T11:46:58.931 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:46:58.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.938+0000 7fb8a93b4640 1 -- 172.21.15.90:0/4002767204 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8900afc60 msgr2=0x7fb8900b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:58.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.938+0000 7fb8a93b4640 1 --2- 172.21.15.90:0/4002767204 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8900afc60 0x7fb8900b2170 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x5605dcbabbd0 tx=0x7fb89c004ab0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.938+0000 7fb8a93b4640 1 -- 172.21.15.90:0/4002767204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8a4108a20 msgr2=0x7fb8a419baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:58.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.938+0000 7fb8a93b4640 1 --2- 172.21.15.90:0/4002767204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8a4108a20 0x7fb8a419baa0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fb88c006010 tx=0x7fb88c004060 comp rx=0 tx=0).stop 2024-04-25T11:46:58.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.939+0000 7fb8a93b4640 1 -- 172.21.15.90:0/4002767204 shutdown_connections 2024-04-25T11:46:58.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.939+0000 7fb8a93b4640 1 --2- 172.21.15.90:0/4002767204 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8900afc60 0x7fb8900b2170 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.939+0000 7fb8a93b4640 1 --2- 172.21.15.90:0/4002767204 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb8a410d380 0x7fb8a419c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.939+0000 7fb8a93b4640 1 --2- 172.21.15.90:0/4002767204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8a4108a20 0x7fb8a419baa0 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.939+0000 7fb8a93b4640 1 --2- 172.21.15.90:0/4002767204 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8a4102af0 0x7fb8a419b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:58.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.939+0000 7fb8a93b4640 1 -- 172.21.15.90:0/4002767204 >> 172.21.15.90:0/4002767204 conn(0x7fb8a4078100 msgr2=0x7fb8a41040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:58.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.939+0000 7fb8a93b4640 1 -- 172.21.15.90:0/4002767204 shutdown_connections 2024-04-25T11:46:58.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:58.939+0000 7fb8a93b4640 1 -- 172.21.15.90:0/4002767204 wait complete. 2024-04-25T11:46:58.936 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T11:46:58.936 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:46:59.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.058+0000 7f692e0a7640 1 -- 172.21.15.90:0/2055338574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f692810e040 msgr2=0x7f6928110530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.058+0000 7f692e0a7640 1 --2- 172.21.15.90:0/2055338574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f692810e040 0x7f6928110530 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f6920009ca0 tx=0x7f692002a8c0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.058+0000 7f692e0a7640 1 -- 172.21.15.90:0/2055338574 shutdown_connections 2024-04-25T11:46:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.058+0000 7f692e0a7640 1 --2- 172.21.15.90:0/2055338574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f692810e040 0x7f6928110530 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.058+0000 7f692e0a7640 1 --2- 172.21.15.90:0/2055338574 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f692810b650 0x7f692810da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.058+0000 7f692e0a7640 1 --2- 172.21.15.90:0/2055338574 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6928104bf0 0x7f6928105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.058+0000 7f692e0a7640 1 -- 172.21.15.90:0/2055338574 >> 172.21.15.90:0/2055338574 conn(0x7f6928078110 msgr2=0x7f6928078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.058+0000 7f692e0a7640 1 -- 172.21.15.90:0/2055338574 shutdown_connections 2024-04-25T11:46:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.058+0000 7f692e0a7640 1 -- 172.21.15.90:0/2055338574 wait complete. 2024-04-25T11:46:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.059+0000 7f692e0a7640 1 Processor -- start 2024-04-25T11:46:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.059+0000 7f692e0a7640 1 -- start start 2024-04-25T11:46:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.059+0000 7f692e0a7640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6928104bf0 0x7f692819a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.059+0000 7f692e0a7640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f692810b650 0x7f692819abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.059+0000 7f692e0a7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f692810e040 0x7f692819ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.059+0000 7f692e0a7640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6928113ca0 con 0x7f692810b650 2024-04-25T11:46:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.059+0000 7f692e0a7640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6928113b20 con 0x7f6928104bf0 2024-04-25T11:46:59.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.059+0000 7f692e0a7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6928113e20 con 0x7f692810e040 2024-04-25T11:46:59.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.059+0000 7f691ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f692810b650 0x7f692819abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.059+0000 7f691ffff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f692810b650 0x7f692819abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59684/0 (socket says 172.21.15.90:59684) 2024-04-25T11:46:59.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.059+0000 7f691ffff640 1 -- 172.21.15.90:0/2491798680 learned_addr learned my addr 172.21.15.90:0/2491798680 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:59.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.059+0000 7f69277fe640 1 --2- 172.21.15.90:0/2491798680 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6928104bf0 0x7f692819a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.060+0000 7f691ffff640 1 -- 172.21.15.90:0/2491798680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f692810e040 msgr2=0x7f692819ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.060+0000 7f691ffff640 1 --2- 172.21.15.90:0/2491798680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f692810e040 0x7f692819ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.060+0000 7f691ffff640 1 -- 172.21.15.90:0/2491798680 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6928104bf0 msgr2=0x7f692819a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.060+0000 7f691ffff640 1 --2- 172.21.15.90:0/2491798680 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6928104bf0 0x7f692819a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.060+0000 7f691ffff640 1 -- 172.21.15.90:0/2491798680 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f692819f4c0 con 0x7f692810b650 2024-04-25T11:46:59.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.060+0000 7f691ffff640 1 --2- 172.21.15.90:0/2491798680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f692810b650 0x7f692819abc0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f691000d950 tx=0x7f691000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:59.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.060+0000 7f69257fa640 1 -- 172.21.15.90:0/2491798680 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6910017070 con 0x7f692810b650 2024-04-25T11:46:59.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.060+0000 7f69257fa640 1 -- 172.21.15.90:0/2491798680 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6910009c30 con 0x7f692810b650 2024-04-25T11:46:59.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.060+0000 7f69257fa640 1 -- 172.21.15.90:0/2491798680 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f691000f620 con 0x7f692810b650 2024-04-25T11:46:59.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.060+0000 7f692e0a7640 1 -- 172.21.15.90:0/2491798680 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f692819f7b0 con 0x7f692810b650 2024-04-25T11:46:59.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.060+0000 7f692e0a7640 1 -- 172.21.15.90:0/2491798680 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69281a7130 con 0x7f692810b650 2024-04-25T11:46:59.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.061+0000 7f692e0a7640 1 -- 172.21.15.90:0/2491798680 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f692806a770 con 0x7f692810b650 2024-04-25T11:46:59.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.063+0000 7f69257fa640 1 -- 172.21.15.90:0/2491798680 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f6910010460 con 0x7f692810b650 2024-04-25T11:46:59.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.063+0000 7f69257fa640 1 --2- 172.21.15.90:0/2491798680 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f69140afc60 0x7f69140b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.063+0000 7f69277fe640 1 --2- 172.21.15.90:0/2491798680 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f69140afc60 0x7f69140b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.063+0000 7f69257fa640 1 -- 172.21.15.90:0/2491798680 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7f69100dec60 con 0x7f692810b650 2024-04-25T11:46:59.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.064+0000 7f69277fe640 1 --2- 172.21.15.90:0/2491798680 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f69140afc60 0x7f69140b2170 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f69180058c0 tx=0x7f691800a250 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:59.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.067+0000 7f69257fa640 1 -- 172.21.15.90:0/2491798680 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f69100a7470 con 0x7f692810b650 2024-04-25T11:46:59.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.287+0000 7f692e0a7640 1 -- 172.21.15.90:0/2491798680 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f692806afa0 con 0x7f692810b650 2024-04-25T11:46:59.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.288+0000 7f69257fa640 1 -- 172.21.15.90:0/2491798680 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11931 (secure 0 0 0) 0x7f692806afa0 con 0x7f692810b650 2024-04-25T11:46:59.274 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:46:59.274 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:46:52.065160+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524,24475,24499,24458,24431]},"id":1}]} 2024-04-25T11:46:59.275 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 29 2024-04-25T11:46:59.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.290+0000 7f692e0a7640 1 -- 172.21.15.90:0/2491798680 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f69140afc60 msgr2=0x7f69140b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.290+0000 7f692e0a7640 1 --2- 172.21.15.90:0/2491798680 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f69140afc60 0x7f69140b2170 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f69180058c0 tx=0x7f691800a250 comp rx=0 tx=0).stop 2024-04-25T11:46:59.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.290+0000 7f692e0a7640 1 -- 172.21.15.90:0/2491798680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f692810b650 msgr2=0x7f692819abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.290+0000 7f692e0a7640 1 --2- 172.21.15.90:0/2491798680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f692810b650 0x7f692819abc0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f691000d950 tx=0x7f691000de10 comp rx=0 tx=0).stop 2024-04-25T11:46:59.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.290+0000 7f692e0a7640 1 -- 172.21.15.90:0/2491798680 shutdown_connections 2024-04-25T11:46:59.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.290+0000 7f692e0a7640 1 --2- 172.21.15.90:0/2491798680 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f69140afc60 0x7f69140b2170 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.290+0000 7f692e0a7640 1 --2- 172.21.15.90:0/2491798680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f692810e040 0x7f692819ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.290+0000 7f692e0a7640 1 --2- 172.21.15.90:0/2491798680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f692810b650 0x7f692819abc0 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.290+0000 7f692e0a7640 1 --2- 172.21.15.90:0/2491798680 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6928104bf0 0x7f692819a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.290+0000 7f692e0a7640 1 -- 172.21.15.90:0/2491798680 >> 172.21.15.90:0/2491798680 conn(0x7f6928078110 msgr2=0x7f6928101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:59.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.290+0000 7f692e0a7640 1 -- 172.21.15.90:0/2491798680 shutdown_connections 2024-04-25T11:46:59.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.290+0000 7f692e0a7640 1 -- 172.21.15.90:0/2491798680 wait complete. 2024-04-25T11:46:59.288 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T11:46:59.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.412+0000 7fc070db7640 1 -- 172.21.15.90:0/1756967737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc06c108a10 msgr2=0x7fc06c102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.412+0000 7fc070db7640 1 --2- 172.21.15.90:0/1756967737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc06c108a10 0x7fc06c102610 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fc060009900 tx=0x7fc06002a410 comp rx=0 tx=0).stop 2024-04-25T11:46:59.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.412+0000 7fc070db7640 1 -- 172.21.15.90:0/1756967737 shutdown_connections 2024-04-25T11:46:59.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.412+0000 7fc070db7640 1 --2- 172.21.15.90:0/1756967737 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc06c1031c0 0x7fc06c10f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.412+0000 7fc070db7640 1 --2- 172.21.15.90:0/1756967737 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc06c108eb0 0x7fc06c102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.412+0000 7fc070db7640 1 --2- 172.21.15.90:0/1756967737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc06c108a10 0x7fc06c102610 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.412+0000 7fc070db7640 1 -- 172.21.15.90:0/1756967737 >> 172.21.15.90:0/1756967737 conn(0x7fc06c078110 msgr2=0x7fc06c078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:59.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.412+0000 7fc070db7640 1 -- 172.21.15.90:0/1756967737 shutdown_connections 2024-04-25T11:46:59.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.412+0000 7fc070db7640 1 -- 172.21.15.90:0/1756967737 wait complete. 2024-04-25T11:46:59.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.413+0000 7fc070db7640 1 Processor -- start 2024-04-25T11:46:59.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.413+0000 7fc070db7640 1 -- start start 2024-04-25T11:46:59.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.413+0000 7fc070db7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc06c1031c0 0x7fc06c199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.413+0000 7fc070db7640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc06c108a10 0x7fc06c199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.413+0000 7fc070db7640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc06c108eb0 0x7fc06c19de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.413+0000 7fc070db7640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c112c10 con 0x7fc06c108a10 2024-04-25T11:46:59.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.413+0000 7fc070db7640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c112a90 con 0x7fc06c108eb0 2024-04-25T11:46:59.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.413+0000 7fc070db7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c112d90 con 0x7fc06c1031c0 2024-04-25T11:46:59.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.413+0000 7fc069d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc06c108a10 0x7fc06c199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.413+0000 7fc069d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc06c108a10 0x7fc06c199b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59696/0 (socket says 172.21.15.90:59696) 2024-04-25T11:46:59.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.413+0000 7fc069d9b640 1 -- 172.21.15.90:0/1530997255 learned_addr learned my addr 172.21.15.90:0/1530997255 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:59.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.413+0000 7fc06ad9d640 1 --2- 172.21.15.90:0/1530997255 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc06c108eb0 0x7fc06c19de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.413+0000 7fc06a59c640 1 --2- 172.21.15.90:0/1530997255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc06c1031c0 0x7fc06c199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.414+0000 7fc069d9b640 1 -- 172.21.15.90:0/1530997255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc06c1031c0 msgr2=0x7fc06c199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.414+0000 7fc069d9b640 1 --2- 172.21.15.90:0/1530997255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc06c1031c0 0x7fc06c199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.414+0000 7fc069d9b640 1 -- 172.21.15.90:0/1530997255 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc06c108eb0 msgr2=0x7fc06c19de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.414+0000 7fc069d9b640 1 --2- 172.21.15.90:0/1530997255 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc06c108eb0 0x7fc06c19de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.414+0000 7fc069d9b640 1 -- 172.21.15.90:0/1530997255 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc06c19e490 con 0x7fc06c108a10 2024-04-25T11:46:59.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.414+0000 7fc06a59c640 1 --2- 172.21.15.90:0/1530997255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc06c1031c0 0x7fc06c199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:46:59.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.414+0000 7fc069d9b640 1 --2- 172.21.15.90:0/1530997255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc06c108a10 0x7fc06c199b90 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fc05c005d00 tx=0x7fc05c00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:59.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.414+0000 7fc05b7fe640 1 -- 172.21.15.90:0/1530997255 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c017070 con 0x7fc06c108a10 2024-04-25T11:46:59.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.414+0000 7fc05b7fe640 1 -- 172.21.15.90:0/1530997255 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc05c009c30 con 0x7fc06c108a10 2024-04-25T11:46:59.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.414+0000 7fc070db7640 1 -- 172.21.15.90:0/1530997255 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc06c19e780 con 0x7fc06c108a10 2024-04-25T11:46:59.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.414+0000 7fc05b7fe640 1 -- 172.21.15.90:0/1530997255 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c00f620 con 0x7fc06c108a10 2024-04-25T11:46:59.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.414+0000 7fc070db7640 1 -- 172.21.15.90:0/1530997255 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc06c19ec80 con 0x7fc06c108a10 2024-04-25T11:46:59.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.417+0000 7fc05b7fe640 1 -- 172.21.15.90:0/1530997255 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fc05c01a070 con 0x7fc06c108a10 2024-04-25T11:46:59.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.417+0000 7fc077c6d740 1 Processor -- start 2024-04-25T11:46:59.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.417+0000 7fc077c6d740 1 -- start start 2024-04-25T11:46:59.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.417+0000 7fc077c6d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc0c6a390 0x555cc100b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.417+0000 7fc077c6d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc0c414d0 0x555cc100bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.417+0000 7fc077c6d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c496b0 0x555cc10186a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.417+0000 7fc077c6d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555cc0f70d80 con 0x555cc0c496b0 2024-04-25T11:46:59.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.417+0000 7fc077c6d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x555cc0f2aa20 con 0x555cc0c414d0 2024-04-25T11:46:59.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.417+0000 7fc077c6d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x555cc0f6cd40 con 0x555cc0c6a390 2024-04-25T11:46:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.417+0000 7fc069d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c496b0 0x555cc10186a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.417+0000 7fc069d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c496b0 0x555cc10186a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59704/0 (socket says 172.21.15.90:59704) 2024-04-25T11:46:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.417+0000 7fc069d9b640 1 -- 172.21.15.90:0/1862040766 learned_addr learned my addr 172.21.15.90:0/1862040766 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.417+0000 7fc06ad9d640 1 --2- 172.21.15.90:0/1862040766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc0c6a390 0x555cc100b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.417+0000 7fc06a59c640 1 --2- 172.21.15.90:0/1862040766 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc0c414d0 0x555cc100bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.417+0000 7fc05b7fe640 1 --2- 172.21.15.90:0/1530997255 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc0540afc60 0x7fc0540b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.418+0000 7fc069d9b640 1 -- 172.21.15.90:0/1862040766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc0c6a390 msgr2=0x555cc100b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.418+0000 7fc069d9b640 1 --2- 172.21.15.90:0/1862040766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc0c6a390 0x555cc100b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.418+0000 7fc069d9b640 1 -- 172.21.15.90:0/1862040766 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc0c414d0 msgr2=0x555cc100bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.418+0000 7fc069d9b640 1 --2- 172.21.15.90:0/1862040766 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc0c414d0 0x555cc100bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.418+0000 7fc069d9b640 1 -- 172.21.15.90:0/1862040766 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555cc0f35c30 con 0x555cc0c496b0 2024-04-25T11:46:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.418+0000 7fc06ad9d640 1 --2- 172.21.15.90:0/1862040766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc0c6a390 0x555cc100b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:46:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.418+0000 7fc06ad9d640 1 --2- 172.21.15.90:0/1530997255 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc0540afc60 0x7fc0540b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.418+0000 7fc05b7fe640 1 -- 172.21.15.90:0/1530997255 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7fc05c0deb60 con 0x7fc06c108a10 2024-04-25T11:46:59.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.418+0000 7fc069d9b640 1 --2- 172.21.15.90:0/1862040766 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c496b0 0x555cc10186a0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fc05c0dc190 tx=0x7fc05c023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64a703da8f419c26 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:59.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.418+0000 7fc058ff9640 1 -- 172.21.15.90:0/1862040766 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c0a6a40 con 0x555cc0c496b0 2024-04-25T11:46:59.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.418+0000 7fc06ad9d640 1 --2- 172.21.15.90:0/1530997255 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc0540afc60 0x7fc0540b2170 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x555cc1008980 tx=0x7fc064014040 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:59.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.418+0000 7fc058ff9640 1 -- 172.21.15.90:0/1862040766 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc05c0a6780 con 0x555cc0c496b0 2024-04-25T11:46:59.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.418+0000 7fc058ff9640 1 -- 172.21.15.90:0/1862040766 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c007820 con 0x555cc0c496b0 2024-04-25T11:46:59.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.419+0000 7fc077c6d740 1 -- 172.21.15.90:0/1862040766 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c496b0 msgr2=0x555cc10186a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.419+0000 7fc077c6d740 1 --2- 172.21.15.90:0/1862040766 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c496b0 0x555cc10186a0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fc05c0dc190 tx=0x7fc05c023000 comp rx=0 tx=0).stop 2024-04-25T11:46:59.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.419+0000 7fc077c6d740 1 -- 172.21.15.90:0/1862040766 shutdown_connections 2024-04-25T11:46:59.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.419+0000 7fc077c6d740 1 --2- 172.21.15.90:0/1862040766 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c496b0 0x555cc10186a0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.419+0000 7fc077c6d740 1 --2- 172.21.15.90:0/1862040766 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc0c414d0 0x555cc100bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.419+0000 7fc077c6d740 1 --2- 172.21.15.90:0/1862040766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc0c6a390 0x555cc100b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.419+0000 7fc077c6d740 1 -- 172.21.15.90:0/1862040766 >> 172.21.15.90:0/1862040766 conn(0x555cc0c5e4e0 msgr2=0x555cc1007fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:59.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.419+0000 7fc077c6d740 1 -- 172.21.15.90:0/1862040766 shutdown_connections 2024-04-25T11:46:59.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.419+0000 7fc077c6d740 1 -- 172.21.15.90:0/1862040766 wait complete. 2024-04-25T11:46:59.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.420+0000 7fc077c6d740 1 -- start start 2024-04-25T11:46:59.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.420+0000 7fc077c6d740 1 Processor -- start 2024-04-25T11:46:59.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.420+0000 7fc077c6d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc0c414d0 0x555cc10162f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.420+0000 7fc077c6d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc0c496b0 0x555cc1016830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.420+0000 7fc077c6d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c6a390 0x555cc1022150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.420+0000 7fc077c6d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555cc1018cd0 con 0x555cc0c6a390 2024-04-25T11:46:59.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.420+0000 7fc077c6d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x555cc0ef7dc0 con 0x555cc0c496b0 2024-04-25T11:46:59.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.420+0000 7fc077c6d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x555cc1018e50 con 0x555cc0c414d0 2024-04-25T11:46:59.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.420+0000 7fc077c6d740 10 client.0 _ll_register_callbacks cb 0x555cc0f13600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:59.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.420+0000 7fc06ad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c6a390 0x555cc1022150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.420+0000 7fc06ad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c6a390 0x555cc1022150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59718/0 (socket says 172.21.15.90:59718) 2024-04-25T11:46:59.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.420+0000 7fc06ad9d640 1 -- 172.21.15.90:0/1335859234 learned_addr learned my addr 172.21.15.90:0/1335859234 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:59.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.420+0000 7fc06a59c640 1 --2- 172.21.15.90:0/1335859234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc0c414d0 0x555cc10162f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.420+0000 7fc069d9b640 1 --2- 172.21.15.90:0/1335859234 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc0c496b0 0x555cc1016830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.420+0000 7fc06ad9d640 1 -- 172.21.15.90:0/1335859234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc0c414d0 msgr2=0x555cc10162f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.420+0000 7fc06ad9d640 1 --2- 172.21.15.90:0/1335859234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc0c414d0 0x555cc10162f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.420+0000 7fc06ad9d640 1 -- 172.21.15.90:0/1335859234 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc0c496b0 msgr2=0x555cc1016830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.420+0000 7fc06ad9d640 1 --2- 172.21.15.90:0/1335859234 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc0c496b0 0x555cc1016830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.420+0000 7fc06ad9d640 1 -- 172.21.15.90:0/1335859234 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555cc0c823f0 con 0x555cc0c6a390 2024-04-25T11:46:59.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.420+0000 7fc06ad9d640 1 --2- 172.21.15.90:0/1335859234 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c6a390 0x555cc1022150 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fc064010910 tx=0x7fc064009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7b46886eba615a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:59.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.421+0000 7fc051ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:46:59.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.421+0000 7fc051ffb640 1 -- 172.21.15.90:0/1335859234 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc06401e9b0 con 0x555cc0c6a390 2024-04-25T11:46:59.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.421+0000 7fc051ffb640 1 -- 172.21.15.90:0/1335859234 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7fc06401eb50 con 0x555cc0c6a390 2024-04-25T11:46:59.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.421+0000 7fc077c6d740 1 -- 172.21.15.90:0/1335859234 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc05c0a6780 con 0x555cc0c6a390 2024-04-25T11:46:59.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.421+0000 7fc051ffb640 1 -- 172.21.15.90:0/1335859234 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc06401ed50 con 0x555cc0c6a390 2024-04-25T11:46:59.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.421+0000 7fc077c6d740 10 client.19404 fetch_fsmap learned FSMap version 29 2024-04-25T11:46:59.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.421+0000 7fc077c6d740 1 -- 172.21.15.90:0/1335859234 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555cc0d4c7d0 con 0x555cc0c6a390 2024-04-25T11:46:59.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.421+0000 7fc051ffb640 1 -- 172.21.15.90:0/1335859234 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc05c0a6780 con 0x555cc0c6a390 2024-04-25T11:46:59.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.421+0000 7fc051ffb640 1 -- 172.21.15.90:0/1335859234 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 29) ==== 8358+0+0 (secure 0 0 0) 0x7fc064025070 con 0x555cc0c6a390 2024-04-25T11:46:59.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.421+0000 7fc077c6d740 10 client.19404 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T11:46:59.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.421+0000 7fc077c6d740 10 client.19404 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:46:59.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.421+0000 7fc077c6d740 20 client.19404 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:59.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.421+0000 7fc077c6d740 1 --2- 172.21.15.90:0/1335859234 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x555cc0c28f60 0x555cc1028780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.421+0000 7fc077c6d740 4 client.19404 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:46:59.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.421+0000 7fc077c6d740 1 -- 172.21.15.90:0/1335859234 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555cc0f7e990 con 0x555cc0c28f60 2024-04-25T11:46:59.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.421+0000 7fc06a59c640 1 --2- 172.21.15.90:0/1335859234 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x555cc0c28f60 0x555cc1028780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.422+0000 7fc051ffb640 10 client.19404 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:46:59.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.422+0000 7fc06a59c640 1 --2- 172.21.15.90:0/1335859234 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x555cc0c28f60 0x555cc1028780 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=86e528f842371599 server_cookie=aa10bcf2b989713b in_seq=0 out_seq=0 2024-04-25T11:46:59.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.423+0000 7fc051ffb640 1 -- 172.21.15.90:0/1335859234 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x555cc0f7e990 con 0x555cc0c28f60 2024-04-25T11:46:59.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.423+0000 7fc051ffb640 10 client.19404 handle_command_reply: tid=0 2024-04-25T11:46:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.423+0000 7fc077c6d740 1 client.19404 shutdown 2024-04-25T11:46:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.423+0000 7fc077c6d740 1 -- 172.21.15.90:0/1335859234 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c6a390 msgr2=0x555cc1022150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.423+0000 7fc077c6d740 1 --2- 172.21.15.90:0/1335859234 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c6a390 0x555cc1022150 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fc064010910 tx=0x7fc064009500 comp rx=0 tx=0).stop 2024-04-25T11:46:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.423+0000 7fc077c6d740 1 -- 172.21.15.90:0/1335859234 shutdown_connections 2024-04-25T11:46:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.423+0000 7fc077c6d740 1 --2- 172.21.15.90:0/1335859234 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x555cc0c28f60 0x555cc1028780 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.423+0000 7fc077c6d740 1 --2- 172.21.15.90:0/1335859234 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c6a390 0x555cc1022150 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.423+0000 7fc077c6d740 1 --2- 172.21.15.90:0/1335859234 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc0c496b0 0x555cc1016830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.423+0000 7fc051ffb640 0 client.19404 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:46:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.423+0000 7fc077c6d740 1 --2- 172.21.15.90:0/1335859234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc0c414d0 0x555cc10162f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.423+0000 7fc077c6d740 1 -- 172.21.15.90:0/1335859234 >> 172.21.15.90:0/1335859234 conn(0x555cc0c5e4e0 msgr2=0x555cc100d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.423+0000 7fc077c6d740 1 -- 172.21.15.90:0/1335859234 shutdown_connections 2024-04-25T11:46:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.423+0000 7fc077c6d740 1 -- 172.21.15.90:0/1335859234 wait complete. 2024-04-25T11:46:59.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.423+0000 7fc077c6d740 20 client.19404 trim_cache size 0 max 16384 2024-04-25T11:46:59.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.437+0000 7fc077c6d740 1 Processor -- start 2024-04-25T11:46:59.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.437+0000 7fc077c6d740 1 -- start start 2024-04-25T11:46:59.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.437+0000 7fc077c6d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c28f60 0x555cc1008010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.437+0000 7fc077c6d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc0c414d0 0x555cc1021db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.437+0000 7fc077c6d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc0c496b0 0x555cc1010430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.437+0000 7fc077c6d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c1aeae0 con 0x555cc0c28f60 2024-04-25T11:46:59.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.437+0000 7fc077c6d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c1b0650 con 0x555cc0c496b0 2024-04-25T11:46:59.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.437+0000 7fc077c6d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c1af1b0 con 0x555cc0c414d0 2024-04-25T11:46:59.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.437+0000 7fc06a59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c28f60 0x555cc1008010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.437+0000 7fc06a59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c28f60 0x555cc1008010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59734/0 (socket says 172.21.15.90:59734) 2024-04-25T11:46:59.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.437+0000 7fc06a59c640 1 -- 172.21.15.90:0/1341195955 learned_addr learned my addr 172.21.15.90:0/1341195955 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:59.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.437+0000 7fc069d9b640 1 --2- 172.21.15.90:0/1341195955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc0c414d0 0x555cc1021db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.437+0000 7fc06ad9d640 1 --2- 172.21.15.90:0/1341195955 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc0c496b0 0x555cc1010430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.437+0000 7fc06a59c640 1 -- 172.21.15.90:0/1341195955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc0c414d0 msgr2=0x555cc1021db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.437+0000 7fc06a59c640 1 --2- 172.21.15.90:0/1341195955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc0c414d0 0x555cc1021db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.437+0000 7fc06a59c640 1 -- 172.21.15.90:0/1341195955 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc0c496b0 msgr2=0x555cc1010430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.437+0000 7fc06a59c640 1 --2- 172.21.15.90:0/1341195955 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc0c496b0 0x555cc1010430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.437+0000 7fc06a59c640 1 -- 172.21.15.90:0/1341195955 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555cc1018980 con 0x555cc0c28f60 2024-04-25T11:46:59.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.437+0000 7fc06ad9d640 1 --2- 172.21.15.90:0/1341195955 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc0c496b0 0x555cc1010430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:46:59.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.437+0000 7fc06a59c640 1 --2- 172.21.15.90:0/1341195955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c28f60 0x555cc1008010 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fc06c0791a0 tx=0x7fc060005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a75c5910a247e921 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:59.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.437+0000 7fc053fff640 1 -- 172.21.15.90:0/1341195955 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc060049070 con 0x555cc0c28f60 2024-04-25T11:46:59.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.437+0000 7fc053fff640 1 -- 172.21.15.90:0/1341195955 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0600054f0 con 0x555cc0c28f60 2024-04-25T11:46:59.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.437+0000 7fc053fff640 1 -- 172.21.15.90:0/1341195955 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc060035040 con 0x555cc0c28f60 2024-04-25T11:46:59.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.438+0000 7fc077c6d740 1 -- 172.21.15.90:0/1341195955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c28f60 msgr2=0x555cc1008010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.438+0000 7fc077c6d740 1 --2- 172.21.15.90:0/1341195955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c28f60 0x555cc1008010 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fc06c0791a0 tx=0x7fc060005a20 comp rx=0 tx=0).stop 2024-04-25T11:46:59.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.438+0000 7fc077c6d740 1 -- 172.21.15.90:0/1341195955 shutdown_connections 2024-04-25T11:46:59.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.438+0000 7fc077c6d740 1 --2- 172.21.15.90:0/1341195955 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc0c496b0 0x555cc1010430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.438+0000 7fc077c6d740 1 --2- 172.21.15.90:0/1341195955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc0c414d0 0x555cc1021db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.438+0000 7fc077c6d740 1 --2- 172.21.15.90:0/1341195955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c28f60 0x555cc1008010 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.438+0000 7fc077c6d740 1 -- 172.21.15.90:0/1341195955 >> 172.21.15.90:0/1341195955 conn(0x555cc0c5e4e0 msgr2=0x555cc1007900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:59.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.438+0000 7fc077c6d740 1 -- 172.21.15.90:0/1341195955 shutdown_connections 2024-04-25T11:46:59.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.438+0000 7fc077c6d740 1 -- 172.21.15.90:0/1341195955 wait complete. 2024-04-25T11:46:59.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.439+0000 7fc077c6d740 1 -- start start 2024-04-25T11:46:59.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.439+0000 7fc077c6d740 1 Processor -- start 2024-04-25T11:46:59.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.439+0000 7fc077c6d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc0c28f60 0x555cc101df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.439+0000 7fc077c6d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c414d0 0x555cc101e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.439+0000 7fc077c6d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc0c496b0 0x555cc1019500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.439+0000 7fc077c6d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c1aff80 con 0x555cc0c414d0 2024-04-25T11:46:59.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.439+0000 7fc077c6d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c1af880 con 0x555cc0c496b0 2024-04-25T11:46:59.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.439+0000 7fc077c6d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c1acae0 con 0x555cc0c28f60 2024-04-25T11:46:59.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.439+0000 7fc077c6d740 10 client.0 _ll_register_callbacks cb 0x7fc064009e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.439+0000 7fc069d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c414d0 0x555cc101e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.439+0000 7fc069d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c414d0 0x555cc101e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59740/0 (socket says 172.21.15.90:59740) 2024-04-25T11:46:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.439+0000 7fc069d9b640 1 -- 172.21.15.90:0/843513854 learned_addr learned my addr 172.21.15.90:0/843513854 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.439+0000 7fc06a59c640 1 --2- 172.21.15.90:0/843513854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc0c28f60 0x555cc101df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.439+0000 7fc069d9b640 1 -- 172.21.15.90:0/843513854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc0c28f60 msgr2=0x555cc101df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.439+0000 7fc069d9b640 1 --2- 172.21.15.90:0/843513854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc0c28f60 0x555cc101df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.439+0000 7fc069d9b640 1 -- 172.21.15.90:0/843513854 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc0c496b0 msgr2=0x555cc1019500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.439+0000 7fc069d9b640 1 --2- 172.21.15.90:0/843513854 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc0c496b0 0x555cc1019500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.439+0000 7fc069d9b640 1 -- 172.21.15.90:0/843513854 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555cc1019bf0 con 0x555cc0c414d0 2024-04-25T11:46:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.439+0000 7fc069d9b640 1 --2- 172.21.15.90:0/843513854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c414d0 0x555cc101e4e0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fc05c005fd0 tx=0x7fc05c0e2570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=868825f98ecbfc29 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.439+0000 7fc051ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:46:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.440+0000 7fc051ffb640 1 -- 172.21.15.90:0/843513854 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c015110 con 0x555cc0c414d0 2024-04-25T11:46:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.440+0000 7fc051ffb640 1 -- 172.21.15.90:0/843513854 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7fc05c0a25e0 con 0x555cc0c414d0 2024-04-25T11:46:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.440+0000 7fc077c6d740 1 -- 172.21.15.90:0/843513854 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0600054f0 con 0x555cc0c414d0 2024-04-25T11:46:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.440+0000 7fc051ffb640 1 -- 172.21.15.90:0/843513854 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc05c0a63e0 con 0x555cc0c414d0 2024-04-25T11:46:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.440+0000 7fc077c6d740 10 client.19410 fetch_fsmap learned FSMap version 29 2024-04-25T11:46:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.440+0000 7fc077c6d740 1 -- 172.21.15.90:0/843513854 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555cc1090380 con 0x555cc0c414d0 2024-04-25T11:46:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.440+0000 7fc051ffb640 1 -- 172.21.15.90:0/843513854 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0600054f0 con 0x555cc0c414d0 2024-04-25T11:46:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.440+0000 7fc051ffb640 1 -- 172.21.15.90:0/843513854 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 29) ==== 8358+0+0 (secure 0 0 0) 0x7fc05c0a6ed0 con 0x555cc0c414d0 2024-04-25T11:46:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.440+0000 7fc077c6d740 10 client.19410 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T11:46:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.440+0000 7fc077c6d740 10 client.19410 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-04-25T11:46:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.440+0000 7fc077c6d740 20 client.19410 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.440+0000 7fc077c6d740 1 --2- 172.21.15.90:0/843513854 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x555cc0c6a390 0x555cc1092720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.440+0000 7fc077c6d740 4 client.19410 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:46:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.440+0000 7fc077c6d740 1 -- 172.21.15.90:0/843513854 --> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555cc0f78800 con 0x555cc0c6a390 2024-04-25T11:46:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.440+0000 7fc06a59c640 1 --2- 172.21.15.90:0/843513854 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x555cc0c6a390 0x555cc1092720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.441+0000 7fc06a59c640 1 --2- 172.21.15.90:0/843513854 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x555cc0c6a390 0x555cc1092720 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=453e3335d8307b96 server_cookie=6bdbe23e3256d6cd in_seq=0 out_seq=0 2024-04-25T11:46:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.441+0000 7fc051ffb640 10 client.19410 ms_handle_connect on v2:172.21.15.138:6832/51785544 2024-04-25T11:46:59.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.441+0000 7fc051ffb640 1 -- 172.21.15.90:0/843513854 <== mds.4 v2:172.21.15.138:6832/51785544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555cc0f78800 con 0x555cc0c6a390 2024-04-25T11:46:59.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.441+0000 7fc051ffb640 10 client.19410 handle_command_reply: tid=0 2024-04-25T11:46:59.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.441+0000 7fc077c6d740 1 client.19410 shutdown 2024-04-25T11:46:59.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.441+0000 7fc077c6d740 1 -- 172.21.15.90:0/843513854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c414d0 msgr2=0x555cc101e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.441+0000 7fc077c6d740 1 --2- 172.21.15.90:0/843513854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c414d0 0x555cc101e4e0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fc05c005fd0 tx=0x7fc05c0e2570 comp rx=0 tx=0).stop 2024-04-25T11:46:59.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.441+0000 7fc077c6d740 1 -- 172.21.15.90:0/843513854 shutdown_connections 2024-04-25T11:46:59.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.441+0000 7fc077c6d740 1 --2- 172.21.15.90:0/843513854 >> [v2:172.21.15.138:6832/51785544,v1:172.21.15.138:6833/51785544] conn(0x555cc0c6a390 0x555cc1092720 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.441+0000 7fc077c6d740 1 --2- 172.21.15.90:0/843513854 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555cc0c496b0 0x555cc1019500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.441+0000 7fc051ffb640 0 client.19410 ms_handle_reset on v2:172.21.15.138:6832/51785544 2024-04-25T11:46:59.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.441+0000 7fc077c6d740 1 --2- 172.21.15.90:0/843513854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555cc0c414d0 0x555cc101e4e0 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.441+0000 7fc077c6d740 1 --2- 172.21.15.90:0/843513854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555cc0c28f60 0x555cc101df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.441+0000 7fc077c6d740 1 -- 172.21.15.90:0/843513854 >> 172.21.15.90:0/843513854 conn(0x555cc0c5e4e0 msgr2=0x555cc10176b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:59.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.441+0000 7fc077c6d740 1 -- 172.21.15.90:0/843513854 shutdown_connections 2024-04-25T11:46:59.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.441+0000 7fc077c6d740 1 -- 172.21.15.90:0/843513854 wait complete. 2024-04-25T11:46:59.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.441+0000 7fc077c6d740 20 client.19410 trim_cache size 0 max 16384 2024-04-25T11:46:59.435 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:46:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.442+0000 7fc070db7640 1 -- 172.21.15.90:0/1530997255 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc0540afc60 msgr2=0x7fc0540b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.442+0000 7fc070db7640 1 --2- 172.21.15.90:0/1530997255 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc0540afc60 0x7fc0540b2170 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x555cc1008980 tx=0x7fc064014040 comp rx=0 tx=0).stop 2024-04-25T11:46:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.442+0000 7fc070db7640 1 -- 172.21.15.90:0/1530997255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc06c108a10 msgr2=0x7fc06c199b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.442+0000 7fc070db7640 1 --2- 172.21.15.90:0/1530997255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc06c108a10 0x7fc06c199b90 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fc05c005d00 tx=0x7fc05c00dde0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.442+0000 7fc070db7640 1 -- 172.21.15.90:0/1530997255 shutdown_connections 2024-04-25T11:46:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.442+0000 7fc070db7640 1 --2- 172.21.15.90:0/1530997255 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fc0540afc60 0x7fc0540b2170 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.442+0000 7fc070db7640 1 --2- 172.21.15.90:0/1530997255 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc06c108eb0 0x7fc06c19de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.442+0000 7fc070db7640 1 --2- 172.21.15.90:0/1530997255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc06c108a10 0x7fc06c199b90 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.442+0000 7fc070db7640 1 --2- 172.21.15.90:0/1530997255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc06c1031c0 0x7fc06c199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.442+0000 7fc070db7640 1 -- 172.21.15.90:0/1530997255 >> 172.21.15.90:0/1530997255 conn(0x7fc06c078110 msgr2=0x7fc06c1040f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:59.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.442+0000 7fc070db7640 1 -- 172.21.15.90:0/1530997255 shutdown_connections 2024-04-25T11:46:59.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.442+0000 7fc070db7640 1 -- 172.21.15.90:0/1530997255 wait complete. 2024-04-25T11:46:59.439 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T11:46:59.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.563+0000 7fa8c1d64640 1 -- 172.21.15.90:0/348028469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8bc102720 msgr2=0x7fa8bc110570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.563+0000 7fa8c1d64640 1 --2- 172.21.15.90:0/348028469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8bc102720 0x7fa8bc110570 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009ca0 tx=0x7fa8b402a8c0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.563+0000 7fa8c1d64640 1 -- 172.21.15.90:0/348028469 shutdown_connections 2024-04-25T11:46:59.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.563+0000 7fa8c1d64640 1 --2- 172.21.15.90:0/348028469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8bc102720 0x7fa8bc110570 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.563+0000 7fa8c1d64640 1 --2- 172.21.15.90:0/348028469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8bc109f90 0x7fa8bc1020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.563+0000 7fa8c1d64640 1 --2- 172.21.15.90:0/348028469 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa8bc109af0 0x7fa8bc101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.563+0000 7fa8c1d64640 1 -- 172.21.15.90:0/348028469 >> 172.21.15.90:0/348028469 conn(0x7fa8bc077880 msgr2=0x7fa8bc077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:59.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.563+0000 7fa8c1d64640 1 -- 172.21.15.90:0/348028469 shutdown_connections 2024-04-25T11:46:59.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.563+0000 7fa8c1d64640 1 -- 172.21.15.90:0/348028469 wait complete. 2024-04-25T11:46:59.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.564+0000 7fa8c1d64640 1 Processor -- start 2024-04-25T11:46:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.564+0000 7fa8c1d64640 1 -- start start 2024-04-25T11:46:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.564+0000 7fa8c1d64640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa8bc102720 0x7fa8bc19a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.564+0000 7fa8c1d64640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8bc109af0 0x7fa8bc19abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.564+0000 7fa8c1d64640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8bc109f90 0x7fa8bc19eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.564+0000 7fa8c1d64640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc113ce0 con 0x7fa8bc109af0 2024-04-25T11:46:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.564+0000 7fa8c1d64640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc113b60 con 0x7fa8bc102720 2024-04-25T11:46:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.564+0000 7fa8c1d64640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc113e60 con 0x7fa8bc109f90 2024-04-25T11:46:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.565+0000 7fa8baffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8bc109af0 0x7fa8bc19abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.565+0000 7fa8baffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8bc109af0 0x7fa8bc19abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59746/0 (socket says 172.21.15.90:59746) 2024-04-25T11:46:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.565+0000 7fa8baffd640 1 -- 172.21.15.90:0/3642527007 learned_addr learned my addr 172.21.15.90:0/3642527007 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.565+0000 7fa8bb7fe640 1 --2- 172.21.15.90:0/3642527007 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa8bc102720 0x7fa8bc19a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.565+0000 7fa8bbfff640 1 --2- 172.21.15.90:0/3642527007 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8bc109f90 0x7fa8bc19eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.565+0000 7fa8baffd640 1 -- 172.21.15.90:0/3642527007 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8bc109f90 msgr2=0x7fa8bc19eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.565+0000 7fa8baffd640 1 --2- 172.21.15.90:0/3642527007 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8bc109f90 0x7fa8bc19eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.565+0000 7fa8baffd640 1 -- 172.21.15.90:0/3642527007 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa8bc102720 msgr2=0x7fa8bc19a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.565+0000 7fa8baffd640 1 --2- 172.21.15.90:0/3642527007 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa8bc102720 0x7fa8bc19a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.565+0000 7fa8baffd640 1 -- 172.21.15.90:0/3642527007 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8bc19f4e0 con 0x7fa8bc109af0 2024-04-25T11:46:59.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.565+0000 7fa8bbfff640 1 --2- 172.21.15.90:0/3642527007 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8bc109f90 0x7fa8bc19eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:46:59.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.565+0000 7fa8baffd640 1 --2- 172.21.15.90:0/3642527007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8bc109af0 0x7fa8bc19abe0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac00d740 tx=0x7fa8ac00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:59.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.565+0000 7fa8b8ff9640 1 -- 172.21.15.90:0/3642527007 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac017070 con 0x7fa8bc109af0 2024-04-25T11:46:59.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.565+0000 7fa8b8ff9640 1 -- 172.21.15.90:0/3642527007 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8ac00de50 con 0x7fa8bc109af0 2024-04-25T11:46:59.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.565+0000 7fa8c1d64640 1 -- 172.21.15.90:0/3642527007 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8bc19f7d0 con 0x7fa8bc109af0 2024-04-25T11:46:59.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.565+0000 7fa8b8ff9640 1 -- 172.21.15.90:0/3642527007 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac004ce0 con 0x7fa8bc109af0 2024-04-25T11:46:59.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.565+0000 7fa8c1d64640 1 -- 172.21.15.90:0/3642527007 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8bc19fcd0 con 0x7fa8bc109af0 2024-04-25T11:46:59.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.568+0000 7fa8b8ff9640 1 -- 172.21.15.90:0/3642527007 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fa8ac004e80 con 0x7fa8bc109af0 2024-04-25T11:46:59.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.568+0000 7fa8c8c15740 1 Processor -- start 2024-04-25T11:46:59.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.568+0000 7fa8c8c15740 1 -- start start 2024-04-25T11:46:59.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.568+0000 7fa8c8c15740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55992bdee390 0x55992c18f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.568+0000 7fa8c8c15740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55992bdc54d0 0x55992c18fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.568+0000 7fa8c8c15740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdcd6b0 0x55992c19c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.568+0000 7fa8c8c15740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55992c0f4d80 con 0x55992bdcd6b0 2024-04-25T11:46:59.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.568+0000 7fa8c8c15740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55992c0aea20 con 0x55992bdc54d0 2024-04-25T11:46:59.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.568+0000 7fa8c8c15740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55992c0f0d40 con 0x55992bdee390 2024-04-25T11:46:59.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.568+0000 7fa8b8ff9640 1 --2- 172.21.15.90:0/3642527007 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa8a40afc60 0x7fa8a40b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.568+0000 7fa8baffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdcd6b0 0x55992c19c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.568+0000 7fa8baffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdcd6b0 0x55992c19c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59754/0 (socket says 172.21.15.90:59754) 2024-04-25T11:46:59.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.568+0000 7fa8bbfff640 1 --2- 172.21.15.90:0/3642527007 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa8a40afc60 0x7fa8a40b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.568+0000 7fa8baffd640 1 -- 172.21.15.90:0/1812258968 learned_addr learned my addr 172.21.15.90:0/1812258968 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:59.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.568+0000 7fa8bb7fe640 1 --2- 172.21.15.90:0/1812258968 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55992bdc54d0 0x55992c18fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.569+0000 7fa8bbfff640 1 --2- 172.21.15.90:0/1812258968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55992bdee390 0x55992c18f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.569+0000 7fa8b8ff9640 1 -- 172.21.15.90:0/3642527007 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7fa8ac0deaa0 con 0x7fa8bc109af0 2024-04-25T11:46:59.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.569+0000 7fa8baffd640 1 -- 172.21.15.90:0/1812258968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55992bdee390 msgr2=0x55992c18f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.569+0000 7fa8baffd640 1 --2- 172.21.15.90:0/1812258968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55992bdee390 0x55992c18f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.569+0000 7fa8baffd640 1 -- 172.21.15.90:0/1812258968 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55992bdc54d0 msgr2=0x55992c18fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.569+0000 7fa8baffd640 1 --2- 172.21.15.90:0/1812258968 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55992bdc54d0 0x55992c18fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.569+0000 7fa8baffd640 1 -- 172.21.15.90:0/1812258968 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55992c0b9c30 con 0x55992bdcd6b0 2024-04-25T11:46:59.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.569+0000 7fa8bbfff640 1 --2- 172.21.15.90:0/3642527007 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa8a40afc60 0x7fa8a40b2170 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa8b402f500 tx=0x7fa8b4032b00 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:59.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.569+0000 7fa8baffd640 1 --2- 172.21.15.90:0/1812258968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdcd6b0 0x55992c19c6a0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac0dc0d0 tx=0x7fa8ac01e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df5ac01d008dd4fa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:59.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.569+0000 7fa8a1ffb640 1 -- 172.21.15.90:0/1812258968 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac0c7e60 con 0x55992bdcd6b0 2024-04-25T11:46:59.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.569+0000 7fa8a1ffb640 1 -- 172.21.15.90:0/1812258968 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8ac0a66c0 con 0x55992bdcd6b0 2024-04-25T11:46:59.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.569+0000 7fa8a1ffb640 1 -- 172.21.15.90:0/1812258968 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac020070 con 0x55992bdcd6b0 2024-04-25T11:46:59.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.570+0000 7fa8c8c15740 1 -- 172.21.15.90:0/1812258968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdcd6b0 msgr2=0x55992c19c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.570+0000 7fa8c8c15740 1 --2- 172.21.15.90:0/1812258968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdcd6b0 0x55992c19c6a0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac0dc0d0 tx=0x7fa8ac01e000 comp rx=0 tx=0).stop 2024-04-25T11:46:59.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.570+0000 7fa8c8c15740 1 -- 172.21.15.90:0/1812258968 shutdown_connections 2024-04-25T11:46:59.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.570+0000 7fa8c8c15740 1 --2- 172.21.15.90:0/1812258968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdcd6b0 0x55992c19c6a0 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.570+0000 7fa8c8c15740 1 --2- 172.21.15.90:0/1812258968 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55992bdc54d0 0x55992c18fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.570+0000 7fa8c8c15740 1 --2- 172.21.15.90:0/1812258968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55992bdee390 0x55992c18f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.570+0000 7fa8c8c15740 1 -- 172.21.15.90:0/1812258968 >> 172.21.15.90:0/1812258968 conn(0x55992bde24e0 msgr2=0x55992c18bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:59.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.570+0000 7fa8c8c15740 1 -- 172.21.15.90:0/1812258968 shutdown_connections 2024-04-25T11:46:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.570+0000 7fa8c8c15740 1 -- 172.21.15.90:0/1812258968 wait complete. 2024-04-25T11:46:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.570+0000 7fa8c8c15740 1 -- start start 2024-04-25T11:46:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.570+0000 7fa8c8c15740 1 Processor -- start 2024-04-25T11:46:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.571+0000 7fa8c8c15740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdc54d0 0x55992c19a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.571+0000 7fa8c8c15740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55992bdcd6b0 0x55992c19a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.571+0000 7fa8c8c15740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55992bdee390 0x55992c1a6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.571+0000 7fa8c8c15740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55992c19ccd0 con 0x55992bdc54d0 2024-04-25T11:46:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.571+0000 7fa8c8c15740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55992c07bdc0 con 0x55992bdee390 2024-04-25T11:46:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.571+0000 7fa8c8c15740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55992c19ce50 con 0x55992bdcd6b0 2024-04-25T11:46:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.571+0000 7fa8c8c15740 10 client.0 _ll_register_callbacks cb 0x55992c097600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.571+0000 7fa8bb7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdc54d0 0x55992c19a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.571+0000 7fa8bb7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdc54d0 0x55992c19a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59770/0 (socket says 172.21.15.90:59770) 2024-04-25T11:46:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.571+0000 7fa8bb7fe640 1 -- 172.21.15.90:0/2468881313 learned_addr learned my addr 172.21.15.90:0/2468881313 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.571+0000 7fa8baffd640 1 --2- 172.21.15.90:0/2468881313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55992bdcd6b0 0x55992c19a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.571+0000 7fa8bbfff640 1 --2- 172.21.15.90:0/2468881313 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55992bdee390 0x55992c1a6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.571+0000 7fa8bb7fe640 1 -- 172.21.15.90:0/2468881313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55992bdcd6b0 msgr2=0x55992c19a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.571+0000 7fa8bb7fe640 1 --2- 172.21.15.90:0/2468881313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55992bdcd6b0 0x55992c19a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.571+0000 7fa8bb7fe640 1 -- 172.21.15.90:0/2468881313 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55992bdee390 msgr2=0x55992c1a6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.571+0000 7fa8bb7fe640 1 --2- 172.21.15.90:0/2468881313 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55992bdee390 0x55992c1a6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.571+0000 7fa8bb7fe640 1 -- 172.21.15.90:0/2468881313 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55992be063f0 con 0x55992bdc54d0 2024-04-25T11:46:59.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.571+0000 7fa8bbfff640 1 --2- 172.21.15.90:0/2468881313 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55992bdee390 0x55992c1a6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:46:59.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.571+0000 7fa8bb7fe640 1 --2- 172.21.15.90:0/2468881313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdc54d0 0x55992c19a2f0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0012720 tx=0x7fa8b0012be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a790d1fe8b54a823 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:59.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.571+0000 7fa89affd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:46:59.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.571+0000 7fa89affd640 1 -- 172.21.15.90:0/2468881313 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b000f920 con 0x55992bdc54d0 2024-04-25T11:46:59.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.572+0000 7fa89affd640 1 -- 172.21.15.90:0/2468881313 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7fa8b000fac0 con 0x55992bdc54d0 2024-04-25T11:46:59.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.572+0000 7fa8c8c15740 1 -- 172.21.15.90:0/2468881313 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8ac0a66c0 con 0x55992bdc54d0 2024-04-25T11:46:59.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.572+0000 7fa89affd640 1 -- 172.21.15.90:0/2468881313 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa8b000fd00 con 0x55992bdc54d0 2024-04-25T11:46:59.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.572+0000 7fa8c8c15740 10 client.19422 fetch_fsmap learned FSMap version 29 2024-04-25T11:46:59.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.572+0000 7fa8c8c15740 1 -- 172.21.15.90:0/2468881313 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55992bed07d0 con 0x55992bdc54d0 2024-04-25T11:46:59.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.572+0000 7fa89affd640 1 -- 172.21.15.90:0/2468881313 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8ac0a66c0 con 0x55992bdc54d0 2024-04-25T11:46:59.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.572+0000 7fa89affd640 1 -- 172.21.15.90:0/2468881313 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 29) ==== 8358+0+0 (secure 0 0 0) 0x7fa8b0010430 con 0x55992bdc54d0 2024-04-25T11:46:59.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.572+0000 7fa8c8c15740 10 client.19422 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T11:46:59.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.572+0000 7fa8c8c15740 10 client.19422 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:46:59.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.572+0000 7fa8c8c15740 20 client.19422 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:59.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.572+0000 7fa8c8c15740 1 --2- 172.21.15.90:0/2468881313 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55992bdacf60 0x55992c1ac6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.572+0000 7fa8c8c15740 4 client.19422 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:46:59.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.572+0000 7fa8c8c15740 1 -- 172.21.15.90:0/2468881313 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55992c102990 con 0x55992bdacf60 2024-04-25T11:46:59.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.572+0000 7fa8baffd640 1 --2- 172.21.15.90:0/2468881313 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55992bdacf60 0x55992c1ac6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.573+0000 7fa89affd640 10 client.19422 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:46:59.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.573+0000 7fa8baffd640 1 --2- 172.21.15.90:0/2468881313 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55992bdacf60 0x55992c1ac6d0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7036de9903c65e93 server_cookie=c2d9e90ad54114e4 in_seq=0 out_seq=0 2024-04-25T11:46:59.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.573+0000 7fa89affd640 1 -- 172.21.15.90:0/2468881313 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55992c102990 con 0x55992bdacf60 2024-04-25T11:46:59.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.573+0000 7fa89affd640 10 client.19422 handle_command_reply: tid=0 2024-04-25T11:46:59.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.573+0000 7fa8c8c15740 1 client.19422 shutdown 2024-04-25T11:46:59.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.574+0000 7fa8c8c15740 1 -- 172.21.15.90:0/2468881313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdc54d0 msgr2=0x55992c19a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.574+0000 7fa8c8c15740 1 --2- 172.21.15.90:0/2468881313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdc54d0 0x55992c19a2f0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0012720 tx=0x7fa8b0012be0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.574+0000 7fa8c8c15740 1 -- 172.21.15.90:0/2468881313 shutdown_connections 2024-04-25T11:46:59.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.574+0000 7fa8c8c15740 1 --2- 172.21.15.90:0/2468881313 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55992bdacf60 0x55992c1ac6d0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.574+0000 7fa8c8c15740 1 --2- 172.21.15.90:0/2468881313 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55992bdee390 0x55992c1a6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.574+0000 7fa8c8c15740 1 --2- 172.21.15.90:0/2468881313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55992bdcd6b0 0x55992c19a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.574+0000 7fa8c8c15740 1 --2- 172.21.15.90:0/2468881313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdc54d0 0x55992c19a2f0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.574+0000 7fa89affd640 0 client.19422 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:46:59.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.574+0000 7fa8c8c15740 1 -- 172.21.15.90:0/2468881313 >> 172.21.15.90:0/2468881313 conn(0x55992bde24e0 msgr2=0x55992c191530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:59.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.574+0000 7fa8c8c15740 1 -- 172.21.15.90:0/2468881313 shutdown_connections 2024-04-25T11:46:59.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.574+0000 7fa8c8c15740 1 -- 172.21.15.90:0/2468881313 wait complete. 2024-04-25T11:46:59.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.574+0000 7fa8c8c15740 20 client.19422 trim_cache size 0 max 16384 2024-04-25T11:46:59.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.587+0000 7fa8c8c15740 1 Processor -- start 2024-04-25T11:46:59.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.587+0000 7fa8c8c15740 1 -- start start 2024-04-25T11:46:59.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.587+0000 7fa8c8c15740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdc54d0 0x55992c1a5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.587+0000 7fa8c8c15740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55992bdcd6b0 0x55992c19bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.587+0000 7fa8c8c15740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55992bdee390 0x55992c194430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.588+0000 7fa8c8c15740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc1afca0 con 0x55992bdc54d0 2024-04-25T11:46:59.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.588+0000 7fa8c8c15740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc1b1810 con 0x55992bdcd6b0 2024-04-25T11:46:59.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.588+0000 7fa8c8c15740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc1b0370 con 0x55992bdee390 2024-04-25T11:46:59.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.588+0000 7fa8bb7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdc54d0 0x55992c1a5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.588+0000 7fa8bb7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdc54d0 0x55992c1a5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59782/0 (socket says 172.21.15.90:59782) 2024-04-25T11:46:59.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.588+0000 7fa8bb7fe640 1 -- 172.21.15.90:0/358195440 learned_addr learned my addr 172.21.15.90:0/358195440 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:59.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.588+0000 7fa8baffd640 1 --2- 172.21.15.90:0/358195440 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55992bdcd6b0 0x55992c19bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.588+0000 7fa8bb7fe640 1 -- 172.21.15.90:0/358195440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55992bdee390 msgr2=0x55992c194430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.588+0000 7fa8bb7fe640 1 --2- 172.21.15.90:0/358195440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55992bdee390 0x55992c194430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.588+0000 7fa8bb7fe640 1 -- 172.21.15.90:0/358195440 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55992bdcd6b0 msgr2=0x55992c19bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.588+0000 7fa8bb7fe640 1 --2- 172.21.15.90:0/358195440 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55992bdcd6b0 0x55992c19bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.588+0000 7fa8bb7fe640 1 -- 172.21.15.90:0/358195440 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55992c198630 con 0x55992bdc54d0 2024-04-25T11:46:59.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.588+0000 7fa8bb7fe640 1 --2- 172.21.15.90:0/358195440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdc54d0 0x55992c1a5db0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fa8b00059f0 tx=0x7fa8b00096f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52b9da148452e2e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:59.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.588+0000 7fa8a17fa640 1 -- 172.21.15.90:0/358195440 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b0010430 con 0x55992bdc54d0 2024-04-25T11:46:59.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.588+0000 7fa8a17fa640 1 -- 172.21.15.90:0/358195440 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8b00105d0 con 0x55992bdc54d0 2024-04-25T11:46:59.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.588+0000 7fa8a17fa640 1 -- 172.21.15.90:0/358195440 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b00056e0 con 0x55992bdc54d0 2024-04-25T11:46:59.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.589+0000 7fa8c8c15740 1 -- 172.21.15.90:0/358195440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdc54d0 msgr2=0x55992c1a5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.589+0000 7fa8c8c15740 1 --2- 172.21.15.90:0/358195440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdc54d0 0x55992c1a5db0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fa8b00059f0 tx=0x7fa8b00096f0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.589+0000 7fa8c8c15740 1 -- 172.21.15.90:0/358195440 shutdown_connections 2024-04-25T11:46:59.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.589+0000 7fa8c8c15740 1 --2- 172.21.15.90:0/358195440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55992bdee390 0x55992c194430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.589+0000 7fa8c8c15740 1 --2- 172.21.15.90:0/358195440 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55992bdcd6b0 0x55992c19bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.589+0000 7fa8c8c15740 1 --2- 172.21.15.90:0/358195440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdc54d0 0x55992c1a5db0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.589+0000 7fa8c8c15740 1 -- 172.21.15.90:0/358195440 >> 172.21.15.90:0/358195440 conn(0x55992bde24e0 msgr2=0x55992c19b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:59.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.589+0000 7fa8c8c15740 1 -- 172.21.15.90:0/358195440 shutdown_connections 2024-04-25T11:46:59.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.589+0000 7fa8c8c15740 1 -- 172.21.15.90:0/358195440 wait complete. 2024-04-25T11:46:59.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.589+0000 7fa8c8c15740 1 -- start start 2024-04-25T11:46:59.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.589+0000 7fa8c8c15740 1 Processor -- start 2024-04-25T11:46:59.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.590+0000 7fa8c8c15740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55992bdc54d0 0x55992c20cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.590+0000 7fa8c8c15740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55992bdcd6b0 0x55992c20d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.590+0000 7fa8c8c15740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdee390 0x55992c19d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.590+0000 7fa8c8c15740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc1b1140 con 0x55992bdee390 2024-04-25T11:46:59.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.590+0000 7fa8c8c15740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc1b0a40 con 0x55992bdcd6b0 2024-04-25T11:46:59.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.590+0000 7fa8c8c15740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc1adca0 con 0x55992bdc54d0 2024-04-25T11:46:59.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.590+0000 7fa8c8c15740 10 client.0 _ll_register_callbacks cb 0x7fa8b0009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:59.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.590+0000 7fa8bbfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdee390 0x55992c19d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.590+0000 7fa8bbfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdee390 0x55992c19d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59786/0 (socket says 172.21.15.90:59786) 2024-04-25T11:46:59.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.590+0000 7fa8bbfff640 1 -- 172.21.15.90:0/58245135 learned_addr learned my addr 172.21.15.90:0/58245135 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:59.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.590+0000 7fa8baffd640 1 --2- 172.21.15.90:0/58245135 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55992bdcd6b0 0x55992c20d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.590+0000 7fa8bb7fe640 1 --2- 172.21.15.90:0/58245135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55992bdc54d0 0x55992c20cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.590+0000 7fa8bbfff640 1 -- 172.21.15.90:0/58245135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55992bdc54d0 msgr2=0x55992c20cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.590+0000 7fa8bbfff640 1 --2- 172.21.15.90:0/58245135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55992bdc54d0 0x55992c20cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.590+0000 7fa8bbfff640 1 -- 172.21.15.90:0/58245135 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55992bdcd6b0 msgr2=0x55992c20d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.590+0000 7fa8bbfff640 1 --2- 172.21.15.90:0/58245135 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55992bdcd6b0 0x55992c20d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.590+0000 7fa8bbfff640 1 -- 172.21.15.90:0/58245135 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55992c19dd30 con 0x55992bdee390 2024-04-25T11:46:59.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.590+0000 7fa8bb7fe640 1 --2- 172.21.15.90:0/58245135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55992bdc54d0 0x55992c20cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:46:59.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.590+0000 7fa8bbfff640 1 --2- 172.21.15.90:0/58245135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdee390 0x55992c19d6f0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009ca0 tx=0x7fa8b40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ef9727819b0285a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:59.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.590+0000 7fa89affd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:46:59.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.590+0000 7fa89affd640 1 -- 172.21.15.90:0/58245135 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b404f030 con 0x55992bdee390 2024-04-25T11:46:59.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.590+0000 7fa89affd640 1 -- 172.21.15.90:0/58245135 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7fa8b4004040 con 0x55992bdee390 2024-04-25T11:46:59.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.590+0000 7fa8c8c15740 1 -- 172.21.15.90:0/58245135 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8b00105d0 con 0x55992bdee390 2024-04-25T11:46:59.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.591+0000 7fa89affd640 1 -- 172.21.15.90:0/58245135 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa8b4004240 con 0x55992bdee390 2024-04-25T11:46:59.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.591+0000 7fa8c8c15740 10 client.19428 fetch_fsmap learned FSMap version 29 2024-04-25T11:46:59.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.591+0000 7fa8c8c15740 1 -- 172.21.15.90:0/58245135 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55992c214450 con 0x55992bdee390 2024-04-25T11:46:59.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.591+0000 7fa89affd640 1 -- 172.21.15.90:0/58245135 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8b00105d0 con 0x55992bdee390 2024-04-25T11:46:59.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.591+0000 7fa89affd640 1 -- 172.21.15.90:0/58245135 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 29) ==== 8358+0+0 (secure 0 0 0) 0x7fa8b4049040 con 0x55992bdee390 2024-04-25T11:46:59.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.591+0000 7fa8c8c15740 10 client.19428 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T11:46:59.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.591+0000 7fa8c8c15740 10 client.19428 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T11:46:59.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.591+0000 7fa8c8c15740 20 client.19428 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:59.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.591+0000 7fa8c8c15740 1 --2- 172.21.15.90:0/58245135 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55992bd823a0 0x55992c216c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.591+0000 7fa8c8c15740 4 client.19428 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:46:59.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.591+0000 7fa8c8c15740 1 -- 172.21.15.90:0/58245135 --> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55992c0fc800 con 0x55992bd823a0 2024-04-25T11:46:59.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.591+0000 7fa8bb7fe640 1 --2- 172.21.15.90:0/58245135 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55992bd823a0 0x55992c216c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.592+0000 7fa89affd640 10 client.19428 ms_handle_connect on v2:172.21.15.138:6834/162690028 2024-04-25T11:46:59.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.592+0000 7fa8bb7fe640 1 --2- 172.21.15.90:0/58245135 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55992bd823a0 0x55992c216c60 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=311d4bc8303ff57d server_cookie=d09c8193e38f9741 in_seq=0 out_seq=0 2024-04-25T11:46:59.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.592+0000 7fa89affd640 1 -- 172.21.15.90:0/58245135 <== mds.3 v2:172.21.15.138:6834/162690028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55992c0fc800 con 0x55992bd823a0 2024-04-25T11:46:59.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.592+0000 7fa89affd640 10 client.19428 handle_command_reply: tid=0 2024-04-25T11:46:59.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.592+0000 7fa8c8c15740 1 client.19428 shutdown 2024-04-25T11:46:59.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.592+0000 7fa8c8c15740 1 -- 172.21.15.90:0/58245135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdee390 msgr2=0x55992c19d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.592+0000 7fa8c8c15740 1 --2- 172.21.15.90:0/58245135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdee390 0x55992c19d6f0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009ca0 tx=0x7fa8b40095c0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.592+0000 7fa8c8c15740 1 -- 172.21.15.90:0/58245135 shutdown_connections 2024-04-25T11:46:59.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.592+0000 7fa8c8c15740 1 --2- 172.21.15.90:0/58245135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55992bdee390 0x55992c19d6f0 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.592+0000 7fa8c8c15740 1 --2- 172.21.15.90:0/58245135 >> [v2:172.21.15.138:6834/162690028,v1:172.21.15.138:6835/162690028] conn(0x55992bd823a0 0x55992c216c60 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.592+0000 7fa8c8c15740 1 --2- 172.21.15.90:0/58245135 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55992bdcd6b0 0x55992c20d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.592+0000 7fa8c8c15740 1 --2- 172.21.15.90:0/58245135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55992bdc54d0 0x55992c20cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.592+0000 7fa89affd640 0 client.19428 ms_handle_reset on v2:172.21.15.138:6834/162690028 2024-04-25T11:46:59.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.592+0000 7fa8c8c15740 1 -- 172.21.15.90:0/58245135 >> 172.21.15.90:0/58245135 conn(0x55992bde24e0 msgr2=0x55992c1998d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:59.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.592+0000 7fa8c8c15740 1 -- 172.21.15.90:0/58245135 shutdown_connections 2024-04-25T11:46:59.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.592+0000 7fa8c8c15740 1 -- 172.21.15.90:0/58245135 wait complete. 2024-04-25T11:46:59.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.592+0000 7fa8c8c15740 20 client.19428 trim_cache size 0 max 16384 2024-04-25T11:46:59.585 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:46:59.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.593+0000 7fa8c1d64640 1 -- 172.21.15.90:0/3642527007 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa8a40afc60 msgr2=0x7fa8a40b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.593+0000 7fa8c1d64640 1 --2- 172.21.15.90:0/3642527007 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa8a40afc60 0x7fa8a40b2170 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa8b402f500 tx=0x7fa8b4032b00 comp rx=0 tx=0).stop 2024-04-25T11:46:59.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.593+0000 7fa8c1d64640 1 -- 172.21.15.90:0/3642527007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8bc109af0 msgr2=0x7fa8bc19abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.593+0000 7fa8c1d64640 1 --2- 172.21.15.90:0/3642527007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8bc109af0 0x7fa8bc19abe0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac00d740 tx=0x7fa8ac00dc00 comp rx=0 tx=0).stop 2024-04-25T11:46:59.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.593+0000 7fa8c1d64640 1 -- 172.21.15.90:0/3642527007 shutdown_connections 2024-04-25T11:46:59.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.593+0000 7fa8c1d64640 1 --2- 172.21.15.90:0/3642527007 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fa8a40afc60 0x7fa8a40b2170 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.593+0000 7fa8c1d64640 1 --2- 172.21.15.90:0/3642527007 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8bc109f90 0x7fa8bc19eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.593+0000 7fa8c1d64640 1 --2- 172.21.15.90:0/3642527007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8bc109af0 0x7fa8bc19abe0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.593+0000 7fa8c1d64640 1 --2- 172.21.15.90:0/3642527007 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa8bc102720 0x7fa8bc19a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.593+0000 7fa8c1d64640 1 -- 172.21.15.90:0/3642527007 >> 172.21.15.90:0/3642527007 conn(0x7fa8bc077880 msgr2=0x7fa8bc100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:59.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.593+0000 7fa8c1d64640 1 -- 172.21.15.90:0/3642527007 shutdown_connections 2024-04-25T11:46:59.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.593+0000 7fa8c1d64640 1 -- 172.21.15.90:0/3642527007 wait complete. 2024-04-25T11:46:59.591 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T11:46:59.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.715+0000 7faed5772640 1 -- 172.21.15.90:0/1319547955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faed0109b40 msgr2=0x7faed0101bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.715+0000 7faed5772640 1 --2- 172.21.15.90:0/1319547955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faed0109b40 0x7faed0101bf0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7faec0009900 tx=0x7faec002a410 comp rx=0 tx=0).stop 2024-04-25T11:46:59.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.715+0000 7faed5772640 1 -- 172.21.15.90:0/1319547955 shutdown_connections 2024-04-25T11:46:59.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.715+0000 7faed5772640 1 --2- 172.21.15.90:0/1319547955 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faed01027e0 0x7faed0110660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.715+0000 7faed5772640 1 --2- 172.21.15.90:0/1319547955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faed0109fe0 0x7faed0102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.715+0000 7faed5772640 1 --2- 172.21.15.90:0/1319547955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faed0109b40 0x7faed0101bf0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.715+0000 7faed5772640 1 -- 172.21.15.90:0/1319547955 >> 172.21.15.90:0/1319547955 conn(0x7faed0077920 msgr2=0x7faed0077d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:59.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.715+0000 7faed5772640 1 -- 172.21.15.90:0/1319547955 shutdown_connections 2024-04-25T11:46:59.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.715+0000 7faed5772640 1 -- 172.21.15.90:0/1319547955 wait complete. 2024-04-25T11:46:59.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.715+0000 7faed5772640 1 Processor -- start 2024-04-25T11:46:59.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.716+0000 7faed5772640 1 -- start start 2024-04-25T11:46:59.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.716+0000 7faed5772640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faed0109b40 0x7faed0199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.716+0000 7faed5772640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faed019a4b0 0x7faed019e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.716+0000 7faed5772640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faed019ecd0 0x7faed019f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.716+0000 7faed5772640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7faed0113d30 con 0x7faed0109b40 2024-04-25T11:46:59.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.716+0000 7faed5772640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7faed0113bb0 con 0x7faed019a4b0 2024-04-25T11:46:59.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.716+0000 7faed5772640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7faed0113eb0 con 0x7faed019ecd0 2024-04-25T11:46:59.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.716+0000 7faeceffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faed0109b40 0x7faed0199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.716+0000 7faeceffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faed0109b40 0x7faed0199f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59798/0 (socket says 172.21.15.90:59798) 2024-04-25T11:46:59.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.716+0000 7faeceffd640 1 -- 172.21.15.90:0/3334382652 learned_addr learned my addr 172.21.15.90:0/3334382652 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:59.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.716+0000 7faecf7fe640 1 --2- 172.21.15.90:0/3334382652 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faed019ecd0 0x7faed019f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.716+0000 7faeceffd640 1 -- 172.21.15.90:0/3334382652 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faed019ecd0 msgr2=0x7faed019f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.716+0000 7faece7fc640 1 --2- 172.21.15.90:0/3334382652 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faed019a4b0 0x7faed019e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.716+0000 7faeceffd640 1 --2- 172.21.15.90:0/3334382652 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faed019ecd0 0x7faed019f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.716+0000 7faeceffd640 1 -- 172.21.15.90:0/3334382652 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faed019a4b0 msgr2=0x7faed019e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.716+0000 7faeceffd640 1 --2- 172.21.15.90:0/3334382652 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faed019a4b0 0x7faed019e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.716+0000 7faeceffd640 1 -- 172.21.15.90:0/3334382652 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed01a6a40 con 0x7faed0109b40 2024-04-25T11:46:59.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.716+0000 7faece7fc640 1 --2- 172.21.15.90:0/3334382652 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faed019a4b0 0x7faed019e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:46:59.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.716+0000 7faeceffd640 1 --2- 172.21.15.90:0/3334382652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faed0109b40 0x7faed0199f70 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7faec0032fd0 tx=0x7faec0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:59.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.717+0000 7faeb7fff640 1 -- 172.21.15.90:0/3334382652 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec003f030 con 0x7faed0109b40 2024-04-25T11:46:59.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.717+0000 7faed5772640 1 -- 172.21.15.90:0/3334382652 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faed01a6c10 con 0x7faed0109b40 2024-04-25T11:46:59.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.717+0000 7faeb7fff640 1 -- 172.21.15.90:0/3334382652 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faec0004490 con 0x7faed0109b40 2024-04-25T11:46:59.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.717+0000 7faeb7fff640 1 -- 172.21.15.90:0/3334382652 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec0035070 con 0x7faed0109b40 2024-04-25T11:46:59.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.717+0000 7faed5772640 1 -- 172.21.15.90:0/3334382652 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faed01a7010 con 0x7faed0109b40 2024-04-25T11:46:59.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.719+0000 7faeb7fff640 1 -- 172.21.15.90:0/3334382652 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7faec0042030 con 0x7faed0109b40 2024-04-25T11:46:59.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.719+0000 7faedc61d740 1 Processor -- start 2024-04-25T11:46:59.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.719+0000 7faedc61d740 1 -- start start 2024-04-25T11:46:59.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.720+0000 7faedc61d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56174901e390 0x5617493bf810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.720+0000 7faedc61d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ff54d0 0x5617493bfd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.720+0000 7faedc61d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561748ffd6b0 0x5617493cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.720+0000 7faedc61d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561749324d80 con 0x561748ff54d0 2024-04-25T11:46:59.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.720+0000 7faedc61d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5617492dea20 con 0x56174901e390 2024-04-25T11:46:59.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.720+0000 7faedc61d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561749320d40 con 0x561748ffd6b0 2024-04-25T11:46:59.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.720+0000 7faeb7fff640 1 --2- 172.21.15.90:0/3334382652 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7faeb80afce0 0x7faeb80b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.720+0000 7faeceffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ff54d0 0x5617493bfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.720+0000 7faeceffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ff54d0 0x5617493bfd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59802/0 (socket says 172.21.15.90:59802) 2024-04-25T11:46:59.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.720+0000 7faeceffd640 1 -- 172.21.15.90:0/1944260509 learned_addr learned my addr 172.21.15.90:0/1944260509 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:59.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.720+0000 7faecf7fe640 1 --2- 172.21.15.90:0/3334382652 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7faeb80afce0 0x7faeb80b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.720+0000 7faece7fc640 1 --2- 172.21.15.90:0/1944260509 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561748ffd6b0 0x5617493cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.720+0000 7faecf7fe640 1 --2- 172.21.15.90:0/1944260509 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56174901e390 0x5617493bf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.720+0000 7faeb7fff640 1 -- 172.21.15.90:0/3334382652 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7faec00ff6e0 con 0x7faed0109b40 2024-04-25T11:46:59.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.720+0000 7faeceffd640 1 -- 172.21.15.90:0/1944260509 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561748ffd6b0 msgr2=0x5617493cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.720+0000 7faeceffd640 1 --2- 172.21.15.90:0/1944260509 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561748ffd6b0 0x5617493cc6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.720+0000 7faeceffd640 1 -- 172.21.15.90:0/1944260509 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56174901e390 msgr2=0x5617493bf810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.720+0000 7faeceffd640 1 --2- 172.21.15.90:0/1944260509 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56174901e390 0x5617493bf810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.720+0000 7faeceffd640 1 -- 172.21.15.90:0/1944260509 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617492e9c30 con 0x561748ff54d0 2024-04-25T11:46:59.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.720+0000 7faeceffd640 1 --2- 172.21.15.90:0/1944260509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ff54d0 0x5617493bfd50 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7faec00fccb0 tx=0x7faec0031a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc6e84e445f6ca26 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:59.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.720+0000 7faecf7fe640 1 --2- 172.21.15.90:0/3334382652 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7faeb80afce0 0x7faeb80b21f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7faec8005fd0 tx=0x7faec8009d10 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:59.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.720+0000 7faeb57fa640 1 -- 172.21.15.90:0/1944260509 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec0049070 con 0x561748ff54d0 2024-04-25T11:46:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.720+0000 7faeb57fa640 1 -- 172.21.15.90:0/1944260509 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faec00c72a0 con 0x561748ff54d0 2024-04-25T11:46:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.720+0000 7faeb57fa640 1 -- 172.21.15.90:0/1944260509 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec00cbb40 con 0x561748ff54d0 2024-04-25T11:46:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.721+0000 7faedc61d740 1 -- 172.21.15.90:0/1944260509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ff54d0 msgr2=0x5617493bfd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.721+0000 7faedc61d740 1 --2- 172.21.15.90:0/1944260509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ff54d0 0x5617493bfd50 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7faec00fccb0 tx=0x7faec0031a10 comp rx=0 tx=0).stop 2024-04-25T11:46:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.721+0000 7faedc61d740 1 -- 172.21.15.90:0/1944260509 shutdown_connections 2024-04-25T11:46:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.721+0000 7faedc61d740 1 --2- 172.21.15.90:0/1944260509 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561748ffd6b0 0x5617493cc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.721+0000 7faedc61d740 1 --2- 172.21.15.90:0/1944260509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ff54d0 0x5617493bfd50 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.721+0000 7faedc61d740 1 --2- 172.21.15.90:0/1944260509 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56174901e390 0x5617493bf810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.721+0000 7faedc61d740 1 -- 172.21.15.90:0/1944260509 >> 172.21.15.90:0/1944260509 conn(0x5617490124e0 msgr2=0x5617493bbfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.721+0000 7faedc61d740 1 -- 172.21.15.90:0/1944260509 shutdown_connections 2024-04-25T11:46:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.721+0000 7faedc61d740 1 -- 172.21.15.90:0/1944260509 wait complete. 2024-04-25T11:46:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.722+0000 7faedc61d740 1 -- start start 2024-04-25T11:46:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.722+0000 7faedc61d740 1 Processor -- start 2024-04-25T11:46:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.722+0000 7faedc61d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561748ff54d0 0x5617493ca230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.722+0000 7faedc61d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ffd6b0 0x5617493ca770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.722+0000 7faedc61d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56174901e390 0x5617493d5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.722+0000 7faedc61d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561749324d80 con 0x561748ffd6b0 2024-04-25T11:46:59.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.722+0000 7faedc61d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x5617492abdc0 con 0x561748ff54d0 2024-04-25T11:46:59.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.722+0000 7faedc61d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5617493ccbe0 con 0x56174901e390 2024-04-25T11:46:59.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.722+0000 7faedc61d740 10 client.0 _ll_register_callbacks cb 0x5617492c7600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:59.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.722+0000 7faece7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ffd6b0 0x5617493ca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.722+0000 7faece7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ffd6b0 0x5617493ca770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59804/0 (socket says 172.21.15.90:59804) 2024-04-25T11:46:59.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.722+0000 7faece7fc640 1 -- 172.21.15.90:0/1718829525 learned_addr learned my addr 172.21.15.90:0/1718829525 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:59.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.722+0000 7faeceffd640 1 --2- 172.21.15.90:0/1718829525 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561748ff54d0 0x5617493ca230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.722+0000 7faece7fc640 1 -- 172.21.15.90:0/1718829525 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56174901e390 msgr2=0x5617493d5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.722+0000 7faece7fc640 1 --2- 172.21.15.90:0/1718829525 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56174901e390 0x5617493d5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.722+0000 7faece7fc640 1 -- 172.21.15.90:0/1718829525 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561748ff54d0 msgr2=0x5617493ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.722+0000 7faece7fc640 1 --2- 172.21.15.90:0/1718829525 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561748ff54d0 0x5617493ca230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.723+0000 7faece7fc640 1 -- 172.21.15.90:0/1718829525 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617490363f0 con 0x561748ffd6b0 2024-04-25T11:46:59.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.723+0000 7faece7fc640 1 --2- 172.21.15.90:0/1718829525 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ffd6b0 0x5617493ca770 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7faec40106e0 tx=0x7faec4010ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6efba570da685fc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.723+0000 7faeb2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:46:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.723+0000 7faeb2ff5640 1 -- 172.21.15.90:0/1718829525 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec401d060 con 0x561748ffd6b0 2024-04-25T11:46:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.723+0000 7faeb2ff5640 1 -- 172.21.15.90:0/1718829525 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7faec400eb90 con 0x561748ffd6b0 2024-04-25T11:46:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.723+0000 7faedc61d740 1 -- 172.21.15.90:0/1718829525 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faec00c72a0 con 0x561748ffd6b0 2024-04-25T11:46:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.723+0000 7faeb2ff5640 1 -- 172.21.15.90:0/1718829525 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faec4024050 con 0x561748ffd6b0 2024-04-25T11:46:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.723+0000 7faedc61d740 10 client.19440 fetch_fsmap learned FSMap version 29 2024-04-25T11:46:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.723+0000 7faedc61d740 1 -- 172.21.15.90:0/1718829525 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5617491007d0 con 0x561748ffd6b0 2024-04-25T11:46:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.723+0000 7faeb2ff5640 1 -- 172.21.15.90:0/1718829525 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faec00c72a0 con 0x561748ffd6b0 2024-04-25T11:46:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.723+0000 7faeb2ff5640 1 -- 172.21.15.90:0/1718829525 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 29) ==== 8358+0+0 (secure 0 0 0) 0x7faec4016330 con 0x561748ffd6b0 2024-04-25T11:46:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.723+0000 7faedc61d740 10 client.19440 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T11:46:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.723+0000 7faedc61d740 10 client.19440 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:46:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.723+0000 7faedc61d740 20 client.19440 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.723+0000 7faedc61d740 1 --2- 172.21.15.90:0/1718829525 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x561748fdcf60 0x5617493dc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.723+0000 7faedc61d740 4 client.19440 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:46:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.723+0000 7faedc61d740 1 -- 172.21.15.90:0/1718829525 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561749332990 con 0x561748fdcf60 2024-04-25T11:46:59.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.724+0000 7faeceffd640 1 --2- 172.21.15.90:0/1718829525 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x561748fdcf60 0x5617493dc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.724+0000 7faeb2ff5640 10 client.19440 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:46:59.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.724+0000 7faeceffd640 1 --2- 172.21.15.90:0/1718829525 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x561748fdcf60 0x5617493dc560 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ab43a86d20c1335f server_cookie=59f247fcfae0a8fe in_seq=0 out_seq=0 2024-04-25T11:46:59.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.725+0000 7faeb2ff5640 1 -- 172.21.15.90:0/1718829525 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561749332990 con 0x561748fdcf60 2024-04-25T11:46:59.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.725+0000 7faeb2ff5640 10 client.19440 handle_command_reply: tid=0 2024-04-25T11:46:59.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.725+0000 7faedc61d740 1 client.19440 shutdown 2024-04-25T11:46:59.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.725+0000 7faedc61d740 1 -- 172.21.15.90:0/1718829525 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ffd6b0 msgr2=0x5617493ca770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.725+0000 7faedc61d740 1 --2- 172.21.15.90:0/1718829525 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ffd6b0 0x5617493ca770 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7faec40106e0 tx=0x7faec4010ba0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.725+0000 7faedc61d740 1 -- 172.21.15.90:0/1718829525 shutdown_connections 2024-04-25T11:46:59.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.725+0000 7faedc61d740 1 --2- 172.21.15.90:0/1718829525 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x561748fdcf60 0x5617493dc560 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.725+0000 7faedc61d740 1 --2- 172.21.15.90:0/1718829525 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56174901e390 0x5617493d5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.725+0000 7faedc61d740 1 --2- 172.21.15.90:0/1718829525 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ffd6b0 0x5617493ca770 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.725+0000 7faeb2ff5640 0 client.19440 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:46:59.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.725+0000 7faedc61d740 1 --2- 172.21.15.90:0/1718829525 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561748ff54d0 0x5617493ca230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.725+0000 7faedc61d740 1 -- 172.21.15.90:0/1718829525 >> 172.21.15.90:0/1718829525 conn(0x5617490124e0 msgr2=0x5617493c1530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:59.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.725+0000 7faedc61d740 1 -- 172.21.15.90:0/1718829525 shutdown_connections 2024-04-25T11:46:59.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.725+0000 7faedc61d740 1 -- 172.21.15.90:0/1718829525 wait complete. 2024-04-25T11:46:59.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.725+0000 7faedc61d740 20 client.19440 trim_cache size 0 max 16384 2024-04-25T11:46:59.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.738+0000 7faedc61d740 1 Processor -- start 2024-04-25T11:46:59.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.739+0000 7faedc61d740 1 -- start start 2024-04-25T11:46:59.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.739+0000 7faedc61d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561748ff54d0 0x5617493d5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.739+0000 7faedc61d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ffd6b0 0x5617493c4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.739+0000 7faedc61d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56174901e390 0x5617493e1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.739+0000 7faedc61d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7faed01afc40 con 0x561748ffd6b0 2024-04-25T11:46:59.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.739+0000 7faedc61d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7faed01b17b0 con 0x56174901e390 2024-04-25T11:46:59.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.739+0000 7faedc61d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7faed01b0310 con 0x561748ff54d0 2024-04-25T11:46:59.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.739+0000 7faece7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ffd6b0 0x5617493c4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.739+0000 7faece7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ffd6b0 0x5617493c4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59818/0 (socket says 172.21.15.90:59818) 2024-04-25T11:46:59.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.739+0000 7faece7fc640 1 -- 172.21.15.90:0/3751060514 learned_addr learned my addr 172.21.15.90:0/3751060514 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:59.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.739+0000 7faeceffd640 1 --2- 172.21.15.90:0/3751060514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561748ff54d0 0x5617493d5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.739+0000 7faece7fc640 1 -- 172.21.15.90:0/3751060514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561748ff54d0 msgr2=0x5617493d5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.739+0000 7faece7fc640 1 --2- 172.21.15.90:0/3751060514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561748ff54d0 0x5617493d5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.739+0000 7faece7fc640 1 -- 172.21.15.90:0/3751060514 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56174901e390 msgr2=0x5617493e1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.739+0000 7faece7fc640 1 --2- 172.21.15.90:0/3751060514 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56174901e390 0x5617493e1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.739+0000 7faece7fc640 1 -- 172.21.15.90:0/3751060514 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617493caf00 con 0x561748ffd6b0 2024-04-25T11:46:59.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.739+0000 7faece7fc640 1 --2- 172.21.15.90:0/3751060514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ffd6b0 0x5617493c4450 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7faec40058e0 tx=0x7faec4005ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a7f9ae875292422 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:59.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.739+0000 7faeb4ff9640 1 -- 172.21.15.90:0/3751060514 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec402c070 con 0x561748ffd6b0 2024-04-25T11:46:59.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.739+0000 7faeb4ff9640 1 -- 172.21.15.90:0/3751060514 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faec400ee40 con 0x561748ffd6b0 2024-04-25T11:46:59.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.739+0000 7faeb4ff9640 1 -- 172.21.15.90:0/3751060514 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec4018720 con 0x561748ffd6b0 2024-04-25T11:46:59.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.740+0000 7faedc61d740 1 -- 172.21.15.90:0/3751060514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ffd6b0 msgr2=0x5617493c4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.740+0000 7faedc61d740 1 --2- 172.21.15.90:0/3751060514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ffd6b0 0x5617493c4450 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7faec40058e0 tx=0x7faec4005ba0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.740+0000 7faedc61d740 1 -- 172.21.15.90:0/3751060514 shutdown_connections 2024-04-25T11:46:59.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.740+0000 7faedc61d740 1 --2- 172.21.15.90:0/3751060514 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56174901e390 0x5617493e1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.740+0000 7faedc61d740 1 --2- 172.21.15.90:0/3751060514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ffd6b0 0x5617493c4450 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.740+0000 7faedc61d740 1 --2- 172.21.15.90:0/3751060514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561748ff54d0 0x5617493d5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.740+0000 7faedc61d740 1 -- 172.21.15.90:0/3751060514 >> 172.21.15.90:0/3751060514 conn(0x5617490124e0 msgr2=0x5617493c88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:59.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.740+0000 7faedc61d740 1 -- 172.21.15.90:0/3751060514 shutdown_connections 2024-04-25T11:46:59.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.740+0000 7faedc61d740 1 -- 172.21.15.90:0/3751060514 wait complete. 2024-04-25T11:46:59.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.741+0000 7faedc61d740 1 -- start start 2024-04-25T11:46:59.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.741+0000 7faedc61d740 1 Processor -- start 2024-04-25T11:46:59.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.741+0000 7faedc61d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ff54d0 0x5617493cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.741+0000 7faedc61d740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561748ffd6b0 0x5617493cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.741+0000 7faedc61d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56174901e390 0x5617493ce520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.741+0000 7faedc61d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7faed01b10e0 con 0x561748ff54d0 2024-04-25T11:46:59.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.741+0000 7faedc61d740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7faed01b09e0 con 0x561748ffd6b0 2024-04-25T11:46:59.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.741+0000 7faedc61d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7faed01adc40 con 0x56174901e390 2024-04-25T11:46:59.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.741+0000 7faedc61d740 10 client.0 _ll_register_callbacks cb 0x7faec400b5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:59.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.741+0000 7faeceffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ff54d0 0x5617493cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.741+0000 7faeceffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ff54d0 0x5617493cdaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59834/0 (socket says 172.21.15.90:59834) 2024-04-25T11:46:59.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.741+0000 7faeceffd640 1 -- 172.21.15.90:0/3646392281 learned_addr learned my addr 172.21.15.90:0/3646392281 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:59.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.741+0000 7faecf7fe640 1 --2- 172.21.15.90:0/3646392281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56174901e390 0x5617493ce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.741+0000 7faeceffd640 1 -- 172.21.15.90:0/3646392281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56174901e390 msgr2=0x5617493ce520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.741+0000 7faeceffd640 1 --2- 172.21.15.90:0/3646392281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56174901e390 0x5617493ce520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.741+0000 7faeceffd640 1 -- 172.21.15.90:0/3646392281 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561748ffd6b0 msgr2=0x5617493cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.741+0000 7faeceffd640 1 --2- 172.21.15.90:0/3646392281 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561748ffd6b0 0x5617493cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.741+0000 7faeceffd640 1 -- 172.21.15.90:0/3646392281 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561749443910 con 0x561748ff54d0 2024-04-25T11:46:59.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.741+0000 7faecf7fe640 1 --2- 172.21.15.90:0/3646392281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56174901e390 0x5617493ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:46:59.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.741+0000 7faeceffd640 1 --2- 172.21.15.90:0/3646392281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ff54d0 0x5617493cdaa0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7faec0031440 tx=0x7faec003b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de71cb0087c9a5f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:59.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.741+0000 7faeb2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:46:59.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.742+0000 7faeb2ff5640 1 -- 172.21.15.90:0/3646392281 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec0049040 con 0x561748ff54d0 2024-04-25T11:46:59.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.742+0000 7faeb2ff5640 1 -- 172.21.15.90:0/3646392281 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7faec00fcfc0 con 0x561748ff54d0 2024-04-25T11:46:59.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.742+0000 7faedc61d740 1 -- 172.21.15.90:0/3646392281 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faec400ee40 con 0x561748ff54d0 2024-04-25T11:46:59.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.742+0000 7faeb2ff5640 1 -- 172.21.15.90:0/3646392281 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faec00c3de0 con 0x561748ff54d0 2024-04-25T11:46:59.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.742+0000 7faedc61d740 10 client.19446 fetch_fsmap learned FSMap version 29 2024-04-25T11:46:59.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.742+0000 7faedc61d740 1 -- 172.21.15.90:0/3646392281 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561749444a90 con 0x561748ff54d0 2024-04-25T11:46:59.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.742+0000 7faeb2ff5640 1 -- 172.21.15.90:0/3646392281 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faec400ee40 con 0x561748ff54d0 2024-04-25T11:46:59.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.742+0000 7faeb2ff5640 1 -- 172.21.15.90:0/3646392281 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 29) ==== 8358+0+0 (secure 0 0 0) 0x7faec003bc20 con 0x561748ff54d0 2024-04-25T11:46:59.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.742+0000 7faedc61d740 10 client.19446 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T11:46:59.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.742+0000 7faedc61d740 10 client.19446 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-25T11:46:59.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.742+0000 7faedc61d740 20 client.19446 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:59.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.742+0000 7faedc61d740 1 --2- 172.21.15.90:0/3646392281 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x561748fb23a0 0x561749446ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.742+0000 7faedc61d740 4 client.19446 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:46:59.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.742+0000 7faedc61d740 1 -- 172.21.15.90:0/3646392281 --> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56174932c800 con 0x561748fb23a0 2024-04-25T11:46:59.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.742+0000 7faece7fc640 1 --2- 172.21.15.90:0/3646392281 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x561748fb23a0 0x561749446ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.743+0000 7faeb2ff5640 10 client.19446 ms_handle_connect on v2:172.21.15.156:6834/1641860251 2024-04-25T11:46:59.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.743+0000 7faece7fc640 1 --2- 172.21.15.90:0/3646392281 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x561748fb23a0 0x561749446ef0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6442ef983bf5b906 server_cookie=43c7544517bef711 in_seq=0 out_seq=0 2024-04-25T11:46:59.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.743+0000 7faeb2ff5640 1 -- 172.21.15.90:0/3646392281 <== mds.2 v2:172.21.15.156:6834/1641860251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56174932c800 con 0x561748fb23a0 2024-04-25T11:46:59.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.743+0000 7faeb2ff5640 10 client.19446 handle_command_reply: tid=0 2024-04-25T11:46:59.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.743+0000 7faedc61d740 1 client.19446 shutdown 2024-04-25T11:46:59.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.743+0000 7faedc61d740 1 -- 172.21.15.90:0/3646392281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ff54d0 msgr2=0x5617493cdaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.743+0000 7faedc61d740 1 --2- 172.21.15.90:0/3646392281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ff54d0 0x5617493cdaa0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7faec0031440 tx=0x7faec003b720 comp rx=0 tx=0).stop 2024-04-25T11:46:59.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.743+0000 7faedc61d740 1 -- 172.21.15.90:0/3646392281 shutdown_connections 2024-04-25T11:46:59.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.743+0000 7faedc61d740 1 --2- 172.21.15.90:0/3646392281 >> [v2:172.21.15.156:6834/1641860251,v1:172.21.15.156:6835/1641860251] conn(0x561748fb23a0 0x561749446ef0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.743+0000 7faedc61d740 1 --2- 172.21.15.90:0/3646392281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56174901e390 0x5617493ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.743+0000 7faedc61d740 1 --2- 172.21.15.90:0/3646392281 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x561748ffd6b0 0x5617493cdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.743+0000 7faedc61d740 1 --2- 172.21.15.90:0/3646392281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561748ff54d0 0x5617493cdaa0 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.743+0000 7faeb2ff5640 0 client.19446 ms_handle_reset on v2:172.21.15.156:6834/1641860251 2024-04-25T11:46:59.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.743+0000 7faedc61d740 1 -- 172.21.15.90:0/3646392281 >> 172.21.15.90:0/3646392281 conn(0x5617490124e0 msgr2=0x5617493c8890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:59.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.743+0000 7faedc61d740 1 -- 172.21.15.90:0/3646392281 shutdown_connections 2024-04-25T11:46:59.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.743+0000 7faedc61d740 1 -- 172.21.15.90:0/3646392281 wait complete. 2024-04-25T11:46:59.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.744+0000 7faedc61d740 20 client.19446 trim_cache size 0 max 16384 2024-04-25T11:46:59.736 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:46:59.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.744+0000 7faed5772640 1 -- 172.21.15.90:0/3334382652 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7faeb80afce0 msgr2=0x7faeb80b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.744+0000 7faed5772640 1 --2- 172.21.15.90:0/3334382652 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7faeb80afce0 0x7faeb80b21f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7faec8005fd0 tx=0x7faec8009d10 comp rx=0 tx=0).stop 2024-04-25T11:46:59.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.744+0000 7faed5772640 1 -- 172.21.15.90:0/3334382652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faed0109b40 msgr2=0x7faed0199f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.744+0000 7faed5772640 1 --2- 172.21.15.90:0/3334382652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faed0109b40 0x7faed0199f70 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7faec0032fd0 tx=0x7faec0004270 comp rx=0 tx=0).stop 2024-04-25T11:46:59.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.744+0000 7faed5772640 1 -- 172.21.15.90:0/3334382652 shutdown_connections 2024-04-25T11:46:59.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.744+0000 7faed5772640 1 --2- 172.21.15.90:0/3334382652 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7faeb80afce0 0x7faeb80b21f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.744+0000 7faed5772640 1 --2- 172.21.15.90:0/3334382652 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faed019ecd0 0x7faed019f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.744+0000 7faed5772640 1 --2- 172.21.15.90:0/3334382652 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faed019a4b0 0x7faed019e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.744+0000 7faed5772640 1 --2- 172.21.15.90:0/3334382652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faed0109b40 0x7faed0199f70 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.744+0000 7faed5772640 1 -- 172.21.15.90:0/3334382652 >> 172.21.15.90:0/3334382652 conn(0x7faed0077920 msgr2=0x7faed0069710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:59.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.744+0000 7faed5772640 1 -- 172.21.15.90:0/3334382652 shutdown_connections 2024-04-25T11:46:59.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.744+0000 7faed5772640 1 -- 172.21.15.90:0/3334382652 wait complete. 2024-04-25T11:46:59.741 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T11:46:59.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.866+0000 7f281496c640 1 -- 172.21.15.90:0/3287020764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f281010d380 msgr2=0x7f281010f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.866+0000 7f281496c640 1 --2- 172.21.15.90:0/3287020764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f281010d380 0x7f281010f7a0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f2800009900 tx=0x7f280002a410 comp rx=0 tx=0).stop 2024-04-25T11:46:59.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.866+0000 7f281496c640 1 -- 172.21.15.90:0/3287020764 shutdown_connections 2024-04-25T11:46:59.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.866+0000 7f281496c640 1 --2- 172.21.15.90:0/3287020764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f281010d380 0x7f281010f7a0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.866+0000 7f281496c640 1 --2- 172.21.15.90:0/3287020764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2810102af0 0x7f2810102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.866+0000 7f281496c640 1 --2- 172.21.15.90:0/3287020764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2810108a20 0x7f28101025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.866+0000 7f281496c640 1 -- 172.21.15.90:0/3287020764 >> 172.21.15.90:0/3287020764 conn(0x7f2810078100 msgr2=0x7f2810078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:59.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.866+0000 7f281496c640 1 -- 172.21.15.90:0/3287020764 shutdown_connections 2024-04-25T11:46:59.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.866+0000 7f281496c640 1 -- 172.21.15.90:0/3287020764 wait complete. 2024-04-25T11:46:59.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.867+0000 7f281496c640 1 Processor -- start 2024-04-25T11:46:59.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.867+0000 7f281496c640 1 -- start start 2024-04-25T11:46:59.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.867+0000 7f281496c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2810102af0 0x7f281019b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.867+0000 7f281496c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2810108a20 0x7f281019ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.867+0000 7f281496c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f281010d380 0x7f281019bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.867+0000 7f281496c640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2810112c00 con 0x7f281010d380 2024-04-25T11:46:59.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.867+0000 7f281496c640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2810112a80 con 0x7f2810102af0 2024-04-25T11:46:59.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.867+0000 7f281496c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2810112d80 con 0x7f2810108a20 2024-04-25T11:46:59.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.867+0000 7f280ed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f281010d380 0x7f281019bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.867+0000 7f280ed9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f281010d380 0x7f281019bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59848/0 (socket says 172.21.15.90:59848) 2024-04-25T11:46:59.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.867+0000 7f280ed9d640 1 -- 172.21.15.90:0/236018006 learned_addr learned my addr 172.21.15.90:0/236018006 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:59.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.867+0000 7f280dd9b640 1 --2- 172.21.15.90:0/236018006 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2810108a20 0x7f281019ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.867+0000 7f280e59c640 1 --2- 172.21.15.90:0/236018006 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2810102af0 0x7f281019b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.867+0000 7f280ed9d640 1 -- 172.21.15.90:0/236018006 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2810108a20 msgr2=0x7f281019ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.867+0000 7f280ed9d640 1 --2- 172.21.15.90:0/236018006 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2810108a20 0x7f281019ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.867+0000 7f280ed9d640 1 -- 172.21.15.90:0/236018006 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2810102af0 msgr2=0x7f281019b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.867+0000 7f280ed9d640 1 --2- 172.21.15.90:0/236018006 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2810102af0 0x7f281019b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.867+0000 7f280ed9d640 1 -- 172.21.15.90:0/236018006 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28101a6850 con 0x7f281010d380 2024-04-25T11:46:59.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.867+0000 7f280ed9d640 1 --2- 172.21.15.90:0/236018006 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f281010d380 0x7f281019bfc0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f2800004770 tx=0x7f2800004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:59.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.868+0000 7f27ff7fe640 1 -- 172.21.15.90:0/236018006 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280003f030 con 0x7f281010d380 2024-04-25T11:46:59.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.868+0000 7f281496c640 1 -- 172.21.15.90:0/236018006 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28101a6ae0 con 0x7f281010d380 2024-04-25T11:46:59.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.868+0000 7f27ff7fe640 1 -- 172.21.15.90:0/236018006 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2800003aa0 con 0x7f281010d380 2024-04-25T11:46:59.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.868+0000 7f27ff7fe640 1 -- 172.21.15.90:0/236018006 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2800035070 con 0x7f281010d380 2024-04-25T11:46:59.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.868+0000 7f281496c640 1 -- 172.21.15.90:0/236018006 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28101a7020 con 0x7f281010d380 2024-04-25T11:46:59.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.870+0000 7f27ff7fe640 1 -- 172.21.15.90:0/236018006 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f2800042030 con 0x7f281010d380 2024-04-25T11:46:59.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.870+0000 7f281b819740 1 Processor -- start 2024-04-25T11:46:59.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.871+0000 7f281b819740 1 -- start start 2024-04-25T11:46:59.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.871+0000 7f281b819740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aacc390 0x56403ae6d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.871+0000 7f281b819740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56403aaa34d0 0x56403ae6dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.871+0000 7f281b819740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56403aaab6b0 0x56403ae7a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.871+0000 7f281b819740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56403add2d80 con 0x56403aacc390 2024-04-25T11:46:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.871+0000 7f281b819740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56403ad8ca20 con 0x56403aaa34d0 2024-04-25T11:46:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.871+0000 7f281b819740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56403adced40 con 0x56403aaab6b0 2024-04-25T11:46:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.871+0000 7f280dd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aacc390 0x56403ae6d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.871+0000 7f280dd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aacc390 0x56403ae6d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59858/0 (socket says 172.21.15.90:59858) 2024-04-25T11:46:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.871+0000 7f280dd9b640 1 -- 172.21.15.90:0/3844739175 learned_addr learned my addr 172.21.15.90:0/3844739175 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.871+0000 7f27ff7fe640 1 --2- 172.21.15.90:0/236018006 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f27f80afc60 0x7f27f80b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.871+0000 7f280e59c640 1 --2- 172.21.15.90:0/3844739175 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56403aaa34d0 0x56403ae6dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.871+0000 7f280dd9b640 1 --2- 172.21.15.90:0/3844739175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56403aaab6b0 0x56403ae7a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.871+0000 7f280dd9b640 1 -- 172.21.15.90:0/3844739175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56403aaab6b0 msgr2=0x56403ae7a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.871+0000 7f280dd9b640 1 --2- 172.21.15.90:0/3844739175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56403aaab6b0 0x56403ae7a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.871+0000 7f280dd9b640 1 -- 172.21.15.90:0/3844739175 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56403aaa34d0 msgr2=0x56403ae6dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.871+0000 7f280dd9b640 1 --2- 172.21.15.90:0/3844739175 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56403aaa34d0 0x56403ae6dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.871+0000 7f280ed9d640 1 --2- 172.21.15.90:0/236018006 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f27f80afc60 0x7f27f80b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.871+0000 7f280dd9b640 1 -- 172.21.15.90:0/3844739175 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56403ad97c30 con 0x56403aacc390 2024-04-25T11:46:59.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.871+0000 7f27ff7fe640 1 -- 172.21.15.90:0/236018006 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7f28000ff5c0 con 0x7f281010d380 2024-04-25T11:46:59.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.871+0000 7f280dd9b640 1 --2- 172.21.15.90:0/3844739175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aacc390 0x56403ae6d810 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f2804013530 tx=0x7f2804005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2efb1bfe93ea068e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:59.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.872+0000 7f27fcff9640 1 -- 172.21.15.90:0/3844739175 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2804005020 con 0x56403aacc390 2024-04-25T11:46:59.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.872+0000 7f27fcff9640 1 -- 172.21.15.90:0/3844739175 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28040047a0 con 0x56403aacc390 2024-04-25T11:46:59.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.872+0000 7f27fcff9640 1 -- 172.21.15.90:0/3844739175 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280400f760 con 0x56403aacc390 2024-04-25T11:46:59.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.873+0000 7f281b819740 1 -- 172.21.15.90:0/3844739175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aacc390 msgr2=0x56403ae6d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.873+0000 7f280ed9d640 1 --2- 172.21.15.90:0/236018006 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f27f80afc60 0x7f27f80b2170 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f28000fcd30 tx=0x7f28000b0040 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:59.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.873+0000 7f281b819740 1 --2- 172.21.15.90:0/3844739175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aacc390 0x56403ae6d810 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f2804013530 tx=0x7f2804005850 comp rx=0 tx=0).stop 2024-04-25T11:46:59.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.873+0000 7f281b819740 1 -- 172.21.15.90:0/3844739175 shutdown_connections 2024-04-25T11:46:59.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.873+0000 7f281b819740 1 --2- 172.21.15.90:0/3844739175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56403aaab6b0 0x56403ae7a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.873+0000 7f281b819740 1 --2- 172.21.15.90:0/3844739175 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56403aaa34d0 0x56403ae6dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.873+0000 7f281b819740 1 --2- 172.21.15.90:0/3844739175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aacc390 0x56403ae6d810 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.873+0000 7f281b819740 1 -- 172.21.15.90:0/3844739175 >> 172.21.15.90:0/3844739175 conn(0x56403aac04e0 msgr2=0x56403ae69fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:59.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.873+0000 7f281b819740 1 -- 172.21.15.90:0/3844739175 shutdown_connections 2024-04-25T11:46:59.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.873+0000 7f281b819740 1 -- 172.21.15.90:0/3844739175 wait complete. 2024-04-25T11:46:59.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.873+0000 7f281b819740 1 -- start start 2024-04-25T11:46:59.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.873+0000 7f281b819740 1 Processor -- start 2024-04-25T11:46:59.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.873+0000 7f281b819740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aaa34d0 0x56403ae78280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.873+0000 7f281b819740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56403aaab6b0 0x56403ae787c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.873+0000 7f281b819740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56403aacc390 0x56403ae84210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.873+0000 7f281b819740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56403ae7ad90 con 0x56403aaa34d0 2024-04-25T11:46:59.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.873+0000 7f281b819740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x56403ad59dc0 con 0x56403aaab6b0 2024-04-25T11:46:59.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.873+0000 7f281b819740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56403ae7af10 con 0x56403aacc390 2024-04-25T11:46:59.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.874+0000 7f281b819740 10 client.0 _ll_register_callbacks cb 0x56403ad75600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:59.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.874+0000 7f280dd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aaa34d0 0x56403ae78280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.874+0000 7f280dd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aaa34d0 0x56403ae78280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59864/0 (socket says 172.21.15.90:59864) 2024-04-25T11:46:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.874+0000 7f280dd9b640 1 -- 172.21.15.90:0/3753074297 learned_addr learned my addr 172.21.15.90:0/3753074297 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.874+0000 7f280e59c640 1 --2- 172.21.15.90:0/3753074297 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56403aaab6b0 0x56403ae787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.874+0000 7f280dd9b640 1 -- 172.21.15.90:0/3753074297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56403aacc390 msgr2=0x56403ae84210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.874+0000 7f280dd9b640 1 --2- 172.21.15.90:0/3753074297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56403aacc390 0x56403ae84210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.874+0000 7f280dd9b640 1 -- 172.21.15.90:0/3753074297 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56403aaab6b0 msgr2=0x56403ae787c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.874+0000 7f280dd9b640 1 --2- 172.21.15.90:0/3753074297 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56403aaab6b0 0x56403ae787c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.874+0000 7f280dd9b640 1 -- 172.21.15.90:0/3753074297 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56403aae43f0 con 0x56403aaa34d0 2024-04-25T11:46:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.874+0000 7f280dd9b640 1 --2- 172.21.15.90:0/3753074297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aaa34d0 0x56403ae78280 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f28040058c0 tx=0x7f2804009df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee4be4c73805bd17 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.874+0000 7f27f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:46:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.874+0000 7f27f5ffb640 1 -- 172.21.15.90:0/3753074297 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280402e060 con 0x56403aaa34d0 2024-04-25T11:46:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.874+0000 7f27f5ffb640 1 -- 172.21.15.90:0/3753074297 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7f2804005210 con 0x56403aaa34d0 2024-04-25T11:46:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.874+0000 7f281b819740 1 -- 172.21.15.90:0/3753074297 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28040047a0 con 0x56403aaa34d0 2024-04-25T11:46:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.874+0000 7f27f5ffb640 1 -- 172.21.15.90:0/3753074297 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2804005410 con 0x56403aaa34d0 2024-04-25T11:46:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.874+0000 7f281b819740 10 client.19458 fetch_fsmap learned FSMap version 29 2024-04-25T11:46:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.874+0000 7f281b819740 1 -- 172.21.15.90:0/3753074297 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56403abae7d0 con 0x56403aaa34d0 2024-04-25T11:46:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.875+0000 7f27f5ffb640 1 -- 172.21.15.90:0/3753074297 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28040047a0 con 0x56403aaa34d0 2024-04-25T11:46:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.875+0000 7f27f5ffb640 1 -- 172.21.15.90:0/3753074297 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 29) ==== 8358+0+0 (secure 0 0 0) 0x7f2804029470 con 0x56403aaa34d0 2024-04-25T11:46:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.875+0000 7f281b819740 10 client.19458 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T11:46:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.875+0000 7f281b819740 10 client.19458 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:46:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.875+0000 7f281b819740 20 client.19458 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.875+0000 7f281b819740 1 --2- 172.21.15.90:0/3753074297 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x56403aa8af60 0x56403ae8a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.875+0000 7f281b819740 4 client.19458 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:46:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.875+0000 7f281b819740 1 -- 172.21.15.90:0/3753074297 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56403ade0990 con 0x56403aa8af60 2024-04-25T11:46:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.875+0000 7f280e59c640 1 --2- 172.21.15.90:0/3753074297 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x56403aa8af60 0x56403ae8a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.875+0000 7f27f5ffb640 10 client.19458 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:46:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.875+0000 7f280e59c640 1 --2- 172.21.15.90:0/3753074297 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x56403aa8af60 0x56403ae8a620 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=801e51ee0cb2c7c2 server_cookie=197ac0e82e33187f in_seq=0 out_seq=0 2024-04-25T11:46:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.876+0000 7f27f5ffb640 1 -- 172.21.15.90:0/3753074297 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56403ade0990 con 0x56403aa8af60 2024-04-25T11:46:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.876+0000 7f27f5ffb640 10 client.19458 handle_command_reply: tid=0 2024-04-25T11:46:59.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.876+0000 7f281b819740 1 client.19458 shutdown 2024-04-25T11:46:59.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.876+0000 7f281b819740 1 -- 172.21.15.90:0/3753074297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aaa34d0 msgr2=0x56403ae78280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.876+0000 7f281b819740 1 --2- 172.21.15.90:0/3753074297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aaa34d0 0x56403ae78280 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f28040058c0 tx=0x7f2804009df0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.876+0000 7f281b819740 1 -- 172.21.15.90:0/3753074297 shutdown_connections 2024-04-25T11:46:59.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.876+0000 7f281b819740 1 --2- 172.21.15.90:0/3753074297 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x56403aa8af60 0x56403ae8a620 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.876+0000 7f281b819740 1 --2- 172.21.15.90:0/3753074297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56403aacc390 0x56403ae84210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.876+0000 7f281b819740 1 --2- 172.21.15.90:0/3753074297 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56403aaab6b0 0x56403ae787c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.876+0000 7f281b819740 1 --2- 172.21.15.90:0/3753074297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aaa34d0 0x56403ae78280 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.876+0000 7f27f5ffb640 0 client.19458 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:46:59.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.876+0000 7f281b819740 1 -- 172.21.15.90:0/3753074297 >> 172.21.15.90:0/3753074297 conn(0x56403aac04e0 msgr2=0x56403ae6f530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:59.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.876+0000 7f281b819740 1 -- 172.21.15.90:0/3753074297 shutdown_connections 2024-04-25T11:46:59.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.876+0000 7f281b819740 1 -- 172.21.15.90:0/3753074297 wait complete. 2024-04-25T11:46:59.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.876+0000 7f281b819740 20 client.19458 trim_cache size 0 max 16384 2024-04-25T11:46:59.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.890+0000 7f281b819740 1 Processor -- start 2024-04-25T11:46:59.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.890+0000 7f281b819740 1 -- start start 2024-04-25T11:46:59.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.890+0000 7f281b819740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56403aaa34d0 0x56403ae77370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.890+0000 7f281b819740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aaab6b0 0x56403ae83e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.890+0000 7f281b819740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56403aacc390 0x56403ae79ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.890+0000 7f281b819740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f28101afc40 con 0x56403aaab6b0 2024-04-25T11:46:59.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.890+0000 7f281b819740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f28101b17b0 con 0x56403aacc390 2024-04-25T11:46:59.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.890+0000 7f281b819740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f28101b0310 con 0x56403aaa34d0 2024-04-25T11:46:59.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.890+0000 7f280e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aaab6b0 0x56403ae83e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.890+0000 7f280e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aaab6b0 0x56403ae83e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59870/0 (socket says 172.21.15.90:59870) 2024-04-25T11:46:59.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.890+0000 7f280e59c640 1 -- 172.21.15.90:0/2399481355 learned_addr learned my addr 172.21.15.90:0/2399481355 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:59.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.890+0000 7f280dd9b640 1 --2- 172.21.15.90:0/2399481355 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56403aaa34d0 0x56403ae77370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.890+0000 7f280dd9b640 1 --2- 172.21.15.90:0/2399481355 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56403aacc390 0x56403ae79ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.890+0000 7f280e59c640 1 -- 172.21.15.90:0/2399481355 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56403aaa34d0 msgr2=0x56403ae77370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.890+0000 7f280e59c640 1 --2- 172.21.15.90:0/2399481355 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56403aaa34d0 0x56403ae77370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.890+0000 7f280e59c640 1 -- 172.21.15.90:0/2399481355 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56403aacc390 msgr2=0x56403ae79ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.890+0000 7f280e59c640 1 --2- 172.21.15.90:0/2399481355 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56403aacc390 0x56403ae79ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.890+0000 7f280e59c640 1 -- 172.21.15.90:0/2399481355 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56403ae657f0 con 0x56403aaab6b0 2024-04-25T11:46:59.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.890+0000 7f280dd9b640 1 --2- 172.21.15.90:0/2399481355 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56403aaa34d0 0x56403ae77370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:46:59.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.890+0000 7f280e59c640 1 --2- 172.21.15.90:0/2399481355 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aaab6b0 0x56403ae83e70 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f2808000f80 tx=0x7f2808014b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b2e29fdfad6818c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:59.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.891+0000 7f27f7fff640 1 -- 172.21.15.90:0/2399481355 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2808014390 con 0x56403aaab6b0 2024-04-25T11:46:59.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.891+0000 7f27f7fff640 1 -- 172.21.15.90:0/2399481355 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2808003e20 con 0x56403aaab6b0 2024-04-25T11:46:59.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.891+0000 7f27f7fff640 1 -- 172.21.15.90:0/2399481355 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28080115e0 con 0x56403aaab6b0 2024-04-25T11:46:59.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.891+0000 7f281b819740 1 -- 172.21.15.90:0/2399481355 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aaab6b0 msgr2=0x56403ae83e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.891+0000 7f281b819740 1 --2- 172.21.15.90:0/2399481355 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aaab6b0 0x56403ae83e70 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f2808000f80 tx=0x7f2808014b20 comp rx=0 tx=0).stop 2024-04-25T11:46:59.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.891+0000 7f281b819740 1 -- 172.21.15.90:0/2399481355 shutdown_connections 2024-04-25T11:46:59.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.891+0000 7f281b819740 1 --2- 172.21.15.90:0/2399481355 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56403aacc390 0x56403ae79ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.891+0000 7f281b819740 1 --2- 172.21.15.90:0/2399481355 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aaab6b0 0x56403ae83e70 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.891+0000 7f281b819740 1 --2- 172.21.15.90:0/2399481355 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56403aaa34d0 0x56403ae77370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.891+0000 7f281b819740 1 -- 172.21.15.90:0/2399481355 >> 172.21.15.90:0/2399481355 conn(0x56403aac04e0 msgr2=0x56403ae79790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:59.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.891+0000 7f281b819740 1 -- 172.21.15.90:0/2399481355 shutdown_connections 2024-04-25T11:46:59.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.891+0000 7f281b819740 1 -- 172.21.15.90:0/2399481355 wait complete. 2024-04-25T11:46:59.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.892+0000 7f281b819740 1 -- start start 2024-04-25T11:46:59.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.892+0000 7f281b819740 1 Processor -- start 2024-04-25T11:46:59.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.892+0000 7f281b819740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56403aaa34d0 0x56403aeeaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.892+0000 7f281b819740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aaab6b0 0x56403aeeaf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.892+0000 7f281b819740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56403aacc390 0x56403ae7fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.892+0000 7f281b819740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f28101b10e0 con 0x56403aaab6b0 2024-04-25T11:46:59.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.892+0000 7f281b819740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f28101b09e0 con 0x56403aaa34d0 2024-04-25T11:46:59.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.892+0000 7f281b819740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f28101adc40 con 0x56403aacc390 2024-04-25T11:46:59.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.892+0000 7f281b819740 10 client.0 _ll_register_callbacks cb 0x7f280800a4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:46:59.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.892+0000 7f280e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aaab6b0 0x56403aeeaf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.892+0000 7f280e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aaab6b0 0x56403aeeaf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59880/0 (socket says 172.21.15.90:59880) 2024-04-25T11:46:59.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.892+0000 7f280e59c640 1 -- 172.21.15.90:0/2720039665 learned_addr learned my addr 172.21.15.90:0/2720039665 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:46:59.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.892+0000 7f280dd9b640 1 --2- 172.21.15.90:0/2720039665 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56403aaa34d0 0x56403aeeaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.892+0000 7f280dd9b640 1 --2- 172.21.15.90:0/2720039665 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56403aacc390 0x56403ae7fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.892+0000 7f280e59c640 1 -- 172.21.15.90:0/2720039665 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56403aacc390 msgr2=0x56403ae7fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.892+0000 7f280e59c640 1 --2- 172.21.15.90:0/2720039665 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56403aacc390 0x56403ae7fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.892+0000 7f280e59c640 1 -- 172.21.15.90:0/2720039665 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56403aaa34d0 msgr2=0x56403aeeaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:46:59.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.892+0000 7f280e59c640 1 --2- 172.21.15.90:0/2720039665 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56403aaa34d0 0x56403aeeaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.892+0000 7f280e59c640 1 -- 172.21.15.90:0/2720039665 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56403ae80390 con 0x56403aaab6b0 2024-04-25T11:46:59.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.892+0000 7f280dd9b640 1 --2- 172.21.15.90:0/2720039665 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56403aaa34d0 0x56403aeeaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:46:59.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.893+0000 7f280e59c640 1 --2- 172.21.15.90:0/2720039665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aaab6b0 0x56403aeeaf60 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f28080143c0 tx=0x7f28080143f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2c4633b8b8fdc71 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:46:59.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.893+0000 7f27f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:46:59.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.893+0000 7f27f5ffb640 1 -- 172.21.15.90:0/2720039665 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2808034060 con 0x56403aaab6b0 2024-04-25T11:46:59.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.893+0000 7f27f5ffb640 1 -- 172.21.15.90:0/2720039665 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7f2808004000 con 0x56403aaab6b0 2024-04-25T11:46:59.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.893+0000 7f281b819740 1 -- 172.21.15.90:0/2720039665 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2808003e20 con 0x56403aaab6b0 2024-04-25T11:46:59.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.893+0000 7f27f5ffb640 1 -- 172.21.15.90:0/2720039665 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2808004200 con 0x56403aaab6b0 2024-04-25T11:46:59.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.893+0000 7f281b819740 10 client.19464 fetch_fsmap learned FSMap version 29 2024-04-25T11:46:59.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.893+0000 7f281b819740 1 -- 172.21.15.90:0/2720039665 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56403aef2180 con 0x56403aaab6b0 2024-04-25T11:46:59.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.893+0000 7f27f5ffb640 1 -- 172.21.15.90:0/2720039665 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2808003e20 con 0x56403aaab6b0 2024-04-25T11:46:59.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.893+0000 7f27f5ffb640 1 -- 172.21.15.90:0/2720039665 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 29) ==== 8358+0+0 (secure 0 0 0) 0x7f2808011c40 con 0x56403aaab6b0 2024-04-25T11:46:59.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.893+0000 7f281b819740 10 client.19464 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T11:46:59.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.893+0000 7f281b819740 10 client.19464 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T11:46:59.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.893+0000 7f281b819740 20 client.19464 populate_metadata read hostname 'smithi090' 2024-04-25T11:46:59.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.893+0000 7f281b819740 1 --2- 172.21.15.90:0/2720039665 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x56403aa603a0 0x56403aef48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:46:59.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.893+0000 7f281b819740 4 client.19464 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:46:59.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.893+0000 7f281b819740 1 -- 172.21.15.90:0/2720039665 --> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56403ae6e0e0 con 0x56403aa603a0 2024-04-25T11:46:59.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.894+0000 7f280dd9b640 1 --2- 172.21.15.90:0/2720039665 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x56403aa603a0 0x56403aef48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:46:59.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.894+0000 7f27f5ffb640 10 client.19464 ms_handle_connect on v2:172.21.15.156:6838/4179744597 2024-04-25T11:46:59.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.894+0000 7f280dd9b640 1 --2- 172.21.15.90:0/2720039665 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x56403aa603a0 0x56403aef48d0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ee4cb53722eb4d78 server_cookie=ac79b212a14faef4 in_seq=0 out_seq=0 2024-04-25T11:46:59.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.894+0000 7f27f5ffb640 1 -- 172.21.15.90:0/2720039665 <== mds.1 v2:172.21.15.156:6838/4179744597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56403ae6e0e0 con 0x56403aa603a0 2024-04-25T11:46:59.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.894+0000 7f27f5ffb640 10 client.19464 handle_command_reply: tid=0 2024-04-25T11:46:59.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.894+0000 7f281b819740 1 client.19464 shutdown 2024-04-25T11:46:59.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.894+0000 7f281b819740 1 -- 172.21.15.90:0/2720039665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aaab6b0 msgr2=0x56403aeeaf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.894+0000 7f281b819740 1 --2- 172.21.15.90:0/2720039665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aaab6b0 0x56403aeeaf60 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f28080143c0 tx=0x7f28080143f0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.894+0000 7f281b819740 1 -- 172.21.15.90:0/2720039665 shutdown_connections 2024-04-25T11:46:59.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.895+0000 7f281b819740 1 --2- 172.21.15.90:0/2720039665 >> [v2:172.21.15.156:6838/4179744597,v1:172.21.15.156:6839/4179744597] conn(0x56403aa603a0 0x56403aef48d0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.895+0000 7f281b819740 1 --2- 172.21.15.90:0/2720039665 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56403aacc390 0x56403ae7fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.895+0000 7f27f5ffb640 0 client.19464 ms_handle_reset on v2:172.21.15.156:6838/4179744597 2024-04-25T11:46:59.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.895+0000 7f281b819740 1 --2- 172.21.15.90:0/2720039665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56403aaab6b0 0x56403aeeaf60 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.895+0000 7f281b819740 1 --2- 172.21.15.90:0/2720039665 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x56403aaa34d0 0x56403aeeaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.895+0000 7f281b819740 1 -- 172.21.15.90:0/2720039665 >> 172.21.15.90:0/2720039665 conn(0x56403aac04e0 msgr2=0x56403ae770d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:59.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.895+0000 7f281b819740 1 -- 172.21.15.90:0/2720039665 shutdown_connections 2024-04-25T11:46:59.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.895+0000 7f281b819740 1 -- 172.21.15.90:0/2720039665 wait complete. 2024-04-25T11:46:59.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.895+0000 7f281b819740 20 client.19464 trim_cache size 0 max 16384 2024-04-25T11:46:59.888 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:46:59.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.895+0000 7f281496c640 1 -- 172.21.15.90:0/236018006 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f27f80afc60 msgr2=0x7f27f80b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.895+0000 7f281496c640 1 --2- 172.21.15.90:0/236018006 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f27f80afc60 0x7f27f80b2170 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f28000fcd30 tx=0x7f28000b0040 comp rx=0 tx=0).stop 2024-04-25T11:46:59.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.895+0000 7f281496c640 1 -- 172.21.15.90:0/236018006 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f281010d380 msgr2=0x7f281019bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:46:59.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.895+0000 7f281496c640 1 --2- 172.21.15.90:0/236018006 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f281010d380 0x7f281019bfc0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f2800004770 tx=0x7f2800004270 comp rx=0 tx=0).stop 2024-04-25T11:46:59.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.895+0000 7f281496c640 1 -- 172.21.15.90:0/236018006 shutdown_connections 2024-04-25T11:46:59.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.895+0000 7f281496c640 1 --2- 172.21.15.90:0/236018006 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f27f80afc60 0x7f27f80b2170 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.895+0000 7f281496c640 1 --2- 172.21.15.90:0/236018006 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f281010d380 0x7f281019bfc0 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.895+0000 7f281496c640 1 --2- 172.21.15.90:0/236018006 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2810108a20 0x7f281019ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.895+0000 7f281496c640 1 --2- 172.21.15.90:0/236018006 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2810102af0 0x7f281019b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:46:59.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.895+0000 7f281496c640 1 -- 172.21.15.90:0/236018006 >> 172.21.15.90:0/236018006 conn(0x7f2810078100 msgr2=0x7f28101040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:46:59.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.895+0000 7f281496c640 1 -- 172.21.15.90:0/236018006 shutdown_connections 2024-04-25T11:46:59.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:46:59.895+0000 7f281496c640 1 -- 172.21.15.90:0/236018006 wait complete. 2024-04-25T11:46:59.893 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T11:47:00.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.016+0000 7f835a71f640 1 -- 172.21.15.90:0/3255002570 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83541031c0 msgr2=0x7f835410f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:00.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.016+0000 7f835a71f640 1 --2- 172.21.15.90:0/3255002570 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83541031c0 0x7f835410f4a0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f834c009ca0 tx=0x7f834c02a8c0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.016+0000 7f835a71f640 1 -- 172.21.15.90:0/3255002570 shutdown_connections 2024-04-25T11:47:00.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.016+0000 7f835a71f640 1 --2- 172.21.15.90:0/3255002570 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83541031c0 0x7f835410f4a0 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.016+0000 7f835a71f640 1 --2- 172.21.15.90:0/3255002570 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8354108eb0 0x7f8354102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.016+0000 7f835a71f640 1 --2- 172.21.15.90:0/3255002570 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8354108a10 0x7f8354102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.016+0000 7f835a71f640 1 -- 172.21.15.90:0/3255002570 >> 172.21.15.90:0/3255002570 conn(0x7f8354078110 msgr2=0x7f8354078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:00.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.016+0000 7f835a71f640 1 -- 172.21.15.90:0/3255002570 shutdown_connections 2024-04-25T11:47:00.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.016+0000 7f835a71f640 1 -- 172.21.15.90:0/3255002570 wait complete. 2024-04-25T11:47:00.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.017+0000 7f835a71f640 1 Processor -- start 2024-04-25T11:47:00.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.017+0000 7f835a71f640 1 -- start start 2024-04-25T11:47:00.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.017+0000 7f835a71f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83541031c0 0x7f8354199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:00.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.017+0000 7f835a71f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8354108a10 0x7f8354199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:00.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.017+0000 7f835a71f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8354108eb0 0x7f835419deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:00.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.017+0000 7f835a71f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8354112c50 con 0x7f8354108a10 2024-04-25T11:47:00.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.017+0000 7f835a71f640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8354112ad0 con 0x7f8354108eb0 2024-04-25T11:47:00.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.017+0000 7f835a71f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8354112dd0 con 0x7f83541031c0 2024-04-25T11:47:00.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.018+0000 7f83537fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8354108a10 0x7f8354199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:00.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.018+0000 7f83537fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8354108a10 0x7f8354199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59898/0 (socket says 172.21.15.90:59898) 2024-04-25T11:47:00.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.018+0000 7f83537fe640 1 -- 172.21.15.90:0/2875164187 learned_addr learned my addr 172.21.15.90:0/2875164187 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:47:00.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.018+0000 7f8353fff640 1 --2- 172.21.15.90:0/2875164187 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83541031c0 0x7f8354199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:00.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.018+0000 7f8358cbc640 1 --2- 172.21.15.90:0/2875164187 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8354108eb0 0x7f835419deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:00.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.018+0000 7f83537fe640 1 -- 172.21.15.90:0/2875164187 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83541031c0 msgr2=0x7f8354199690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:00.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.018+0000 7f83537fe640 1 --2- 172.21.15.90:0/2875164187 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83541031c0 0x7f8354199690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.018+0000 7f83537fe640 1 -- 172.21.15.90:0/2875164187 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8354108eb0 msgr2=0x7f835419deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:00.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.018+0000 7f83537fe640 1 --2- 172.21.15.90:0/2875164187 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8354108eb0 0x7f835419deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.018+0000 7f83537fe640 1 -- 172.21.15.90:0/2875164187 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f835419e4f0 con 0x7f8354108a10 2024-04-25T11:47:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.018+0000 7f83537fe640 1 --2- 172.21.15.90:0/2875164187 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8354108a10 0x7f8354199bf0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f834800e9f0 tx=0x7f834800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.018+0000 7f83517fa640 1 -- 172.21.15.90:0/2875164187 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834800ccc0 con 0x7f8354108a10 2024-04-25T11:47:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.018+0000 7f835a71f640 1 -- 172.21.15.90:0/2875164187 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f835419e7e0 con 0x7f8354108a10 2024-04-25T11:47:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.018+0000 7f83517fa640 1 -- 172.21.15.90:0/2875164187 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83480035f0 con 0x7f8354108a10 2024-04-25T11:47:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.018+0000 7f83517fa640 1 -- 172.21.15.90:0/2875164187 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834800ce60 con 0x7f8354108a10 2024-04-25T11:47:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.018+0000 7f835a71f640 1 -- 172.21.15.90:0/2875164187 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f835419ece0 con 0x7f8354108a10 2024-04-25T11:47:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.021+0000 7f83517fa640 1 -- 172.21.15.90:0/2875164187 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f8348003aa0 con 0x7f8354108a10 2024-04-25T11:47:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.021+0000 7f83615d5740 1 Processor -- start 2024-04-25T11:47:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.021+0000 7f83615d5740 1 -- start start 2024-04-25T11:47:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.021+0000 7f83615d5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13ea6390 0x55ad14249a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.021+0000 7f83615d5740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad13e7d4d0 0x55ad14249f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.021+0000 7f83615d5740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad13e856b0 0x55ad142568b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.021+0000 7f83615d5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ad141acd80 con 0x55ad13ea6390 2024-04-25T11:47:00.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.021+0000 7f83615d5740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ad14166a20 con 0x55ad13e856b0 2024-04-25T11:47:00.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.021+0000 7f83615d5740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ad141a8d40 con 0x55ad13e7d4d0 2024-04-25T11:47:00.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.021+0000 7f8358cbc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13ea6390 0x55ad14249a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:00.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.021+0000 7f8358cbc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13ea6390 0x55ad14249a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59904/0 (socket says 172.21.15.90:59904) 2024-04-25T11:47:00.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.021+0000 7f8358cbc640 1 -- 172.21.15.90:0/1481974340 learned_addr learned my addr 172.21.15.90:0/1481974340 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:47:00.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.021+0000 7f83517fa640 1 --2- 172.21.15.90:0/2875164187 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f833c0afc60 0x7f833c0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:00.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.021+0000 7f8353fff640 1 --2- 172.21.15.90:0/1481974340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad13e7d4d0 0x55ad14249f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:00.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.021+0000 7f83537fe640 1 --2- 172.21.15.90:0/1481974340 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad13e856b0 0x55ad142568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:00.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.021+0000 7f8358cbc640 1 -- 172.21.15.90:0/1481974340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad13e7d4d0 msgr2=0x55ad14249f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:47:00.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.021+0000 7f8358cbc640 1 --2- 172.21.15.90:0/1481974340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad13e7d4d0 0x55ad14249f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.021+0000 7f8358cbc640 1 -- 172.21.15.90:0/1481974340 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad13e856b0 msgr2=0x55ad142568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:47:00.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.021+0000 7f8358cbc640 1 --2- 172.21.15.90:0/1481974340 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad13e856b0 0x55ad142568b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.021+0000 7f8358cbc640 1 -- 172.21.15.90:0/1481974340 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad14171c30 con 0x55ad13ea6390 2024-04-25T11:47:00.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.021+0000 7f83537fe640 1 --2- 172.21.15.90:0/1481974340 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad13e856b0 0x55ad142568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:47:00.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.022+0000 7f83517fa640 1 -- 172.21.15.90:0/2875164187 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7f83480df3d0 con 0x7f8354108a10 2024-04-25T11:47:00.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.022+0000 7f8358cbc640 1 --2- 172.21.15.90:0/2875164187 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f833c0afc60 0x7f833c0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:00.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.022+0000 7f8358cbc640 1 --2- 172.21.15.90:0/1481974340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13ea6390 0x55ad14249a20 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f834c009480 tx=0x7f834c009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c4edf9a4b3a79ea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:00.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.022+0000 7f833a7fc640 1 -- 172.21.15.90:0/1481974340 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834c042040 con 0x55ad13ea6390 2024-04-25T11:47:00.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.022+0000 7f833a7fc640 1 -- 172.21.15.90:0/1481974340 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f834c036070 con 0x55ad13ea6390 2024-04-25T11:47:00.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.022+0000 7f833a7fc640 1 -- 172.21.15.90:0/1481974340 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834c004700 con 0x55ad13ea6390 2024-04-25T11:47:00.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.023+0000 7f83615d5740 1 -- 172.21.15.90:0/1481974340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13ea6390 msgr2=0x55ad14249a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:00.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.023+0000 7f8358cbc640 1 --2- 172.21.15.90:0/2875164187 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f833c0afc60 0x7f833c0b2170 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f834c04d000 tx=0x7f834c04e000 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:00.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.023+0000 7f83615d5740 1 --2- 172.21.15.90:0/1481974340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13ea6390 0x55ad14249a20 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f834c009480 tx=0x7f834c009800 comp rx=0 tx=0).stop 2024-04-25T11:47:00.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.023+0000 7f83615d5740 1 -- 172.21.15.90:0/1481974340 shutdown_connections 2024-04-25T11:47:00.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.023+0000 7f83615d5740 1 --2- 172.21.15.90:0/1481974340 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad13e856b0 0x55ad142568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.023+0000 7f83615d5740 1 --2- 172.21.15.90:0/1481974340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad13e7d4d0 0x55ad14249f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.023+0000 7f83615d5740 1 --2- 172.21.15.90:0/1481974340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13ea6390 0x55ad14249a20 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.023+0000 7f83615d5740 1 -- 172.21.15.90:0/1481974340 >> 172.21.15.90:0/1481974340 conn(0x55ad13e9a4e0 msgr2=0x55ad142461d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:00.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.023+0000 7f83615d5740 1 -- 172.21.15.90:0/1481974340 shutdown_connections 2024-04-25T11:47:00.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.023+0000 7f83615d5740 1 -- 172.21.15.90:0/1481974340 wait complete. 2024-04-25T11:47:00.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.023+0000 7f83615d5740 1 -- start start 2024-04-25T11:47:00.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.023+0000 7f83615d5740 1 Processor -- start 2024-04-25T11:47:00.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.024+0000 7f83615d5740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad13e7d4d0 0x55ad14254410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:00.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.024+0000 7f83615d5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13e856b0 0x55ad14254950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:00.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.024+0000 7f83615d5740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad13ea6390 0x55ad142604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:00.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.024+0000 7f83615d5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ad14256fa0 con 0x55ad13e856b0 2024-04-25T11:47:00.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.024+0000 7f83615d5740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55ad14133dc0 con 0x55ad13e7d4d0 2024-04-25T11:47:00.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.024+0000 7f83615d5740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ad14257120 con 0x55ad13ea6390 2024-04-25T11:47:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.024+0000 7f83615d5740 10 client.0 _ll_register_callbacks cb 0x55ad1414f600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:47:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.024+0000 7f83537fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13e856b0 0x55ad14254950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.024+0000 7f83537fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13e856b0 0x55ad14254950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59912/0 (socket says 172.21.15.90:59912) 2024-04-25T11:47:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.024+0000 7f83537fe640 1 -- 172.21.15.90:0/749070602 learned_addr learned my addr 172.21.15.90:0/749070602 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:47:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.024+0000 7f8358cbc640 1 --2- 172.21.15.90:0/749070602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad13ea6390 0x55ad142604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.024+0000 7f83537fe640 1 -- 172.21.15.90:0/749070602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad13ea6390 msgr2=0x55ad142604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:47:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.024+0000 7f83537fe640 1 --2- 172.21.15.90:0/749070602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad13ea6390 0x55ad142604e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.024+0000 7f8353fff640 1 --2- 172.21.15.90:0/749070602 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad13e7d4d0 0x55ad14254410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.024+0000 7f83537fe640 1 -- 172.21.15.90:0/749070602 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad13e7d4d0 msgr2=0x55ad14254410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:47:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.024+0000 7f83537fe640 1 --2- 172.21.15.90:0/749070602 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad13e7d4d0 0x55ad14254410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.024+0000 7f83537fe640 1 -- 172.21.15.90:0/749070602 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad13ebe3f0 con 0x55ad13e856b0 2024-04-25T11:47:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.024+0000 7f8358cbc640 1 --2- 172.21.15.90:0/749070602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad13ea6390 0x55ad142604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:47:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.024+0000 7f8353fff640 1 --2- 172.21.15.90:0/749070602 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad13e7d4d0 0x55ad14254410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:47:00.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.024+0000 7f83537fe640 1 --2- 172.21.15.90:0/749070602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13e856b0 0x55ad14254950 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f83480dcaf0 tx=0x7f83480a6c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c04d3d091ca9267 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:00.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.024+0000 7f83337fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:47:00.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.024+0000 7f83337fe640 1 -- 172.21.15.90:0/749070602 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83480c1060 con 0x55ad13e856b0 2024-04-25T11:47:00.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.024+0000 7f83337fe640 1 -- 172.21.15.90:0/749070602 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7f834801fd50 con 0x55ad13e856b0 2024-04-25T11:47:00.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.024+0000 7f83615d5740 1 -- 172.21.15.90:0/749070602 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f834c036070 con 0x55ad13e856b0 2024-04-25T11:47:00.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.025+0000 7f83337fe640 1 -- 172.21.15.90:0/749070602 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f83480a7be0 con 0x55ad13e856b0 2024-04-25T11:47:00.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.025+0000 7f83615d5740 10 client.19476 fetch_fsmap learned FSMap version 29 2024-04-25T11:47:00.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.025+0000 7f83615d5740 1 -- 172.21.15.90:0/749070602 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ad13f887d0 con 0x55ad13e856b0 2024-04-25T11:47:00.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.025+0000 7f83337fe640 1 -- 172.21.15.90:0/749070602 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f834c036070 con 0x55ad13e856b0 2024-04-25T11:47:00.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.025+0000 7f83337fe640 1 -- 172.21.15.90:0/749070602 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 29) ==== 8358+0+0 (secure 0 0 0) 0x7f83480a3770 con 0x55ad13e856b0 2024-04-25T11:47:00.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.025+0000 7f83615d5740 10 client.19476 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T11:47:00.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.025+0000 7f83615d5740 10 client.19476 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:47:00.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.025+0000 7f83615d5740 20 client.19476 populate_metadata read hostname 'smithi090' 2024-04-25T11:47:00.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.025+0000 7f83615d5740 1 --2- 172.21.15.90:0/749070602 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ad13e64f60 0x55ad14266850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.025+0000 7f83615d5740 4 client.19476 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:47:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.025+0000 7f83615d5740 1 -- 172.21.15.90:0/749070602 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad141ba990 con 0x55ad13e64f60 2024-04-25T11:47:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.025+0000 7f8353fff640 1 --2- 172.21.15.90:0/749070602 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ad13e64f60 0x55ad14266850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.026+0000 7f83337fe640 10 client.19476 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:47:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.026+0000 7f8353fff640 1 --2- 172.21.15.90:0/749070602 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ad13e64f60 0x55ad14266850 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65316718080ac4fb server_cookie=858f8ce6909b02ac in_seq=0 out_seq=0 2024-04-25T11:47:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.026+0000 7f83337fe640 1 -- 172.21.15.90:0/749070602 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ad141ba990 con 0x55ad13e64f60 2024-04-25T11:47:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.026+0000 7f83337fe640 10 client.19476 handle_command_reply: tid=0 2024-04-25T11:47:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.026+0000 7f83615d5740 1 client.19476 shutdown 2024-04-25T11:47:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.027+0000 7f83615d5740 1 -- 172.21.15.90:0/749070602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13e856b0 msgr2=0x55ad14254950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.027+0000 7f83615d5740 1 --2- 172.21.15.90:0/749070602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13e856b0 0x55ad14254950 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f83480dcaf0 tx=0x7f83480a6c50 comp rx=0 tx=0).stop 2024-04-25T11:47:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.027+0000 7f83615d5740 1 -- 172.21.15.90:0/749070602 shutdown_connections 2024-04-25T11:47:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.027+0000 7f83615d5740 1 --2- 172.21.15.90:0/749070602 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ad13e64f60 0x55ad14266850 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.027+0000 7f83615d5740 1 --2- 172.21.15.90:0/749070602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad13ea6390 0x55ad142604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.027+0000 7f83615d5740 1 --2- 172.21.15.90:0/749070602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13e856b0 0x55ad14254950 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.027+0000 7f83337fe640 0 client.19476 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:47:00.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.027+0000 7f83615d5740 1 --2- 172.21.15.90:0/749070602 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad13e7d4d0 0x55ad14254410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.027+0000 7f83615d5740 1 -- 172.21.15.90:0/749070602 >> 172.21.15.90:0/749070602 conn(0x55ad13e9a4e0 msgr2=0x55ad1424b740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:00.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.027+0000 7f83615d5740 1 -- 172.21.15.90:0/749070602 shutdown_connections 2024-04-25T11:47:00.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.027+0000 7f83615d5740 1 -- 172.21.15.90:0/749070602 wait complete. 2024-04-25T11:47:00.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.027+0000 7f83615d5740 20 client.19476 trim_cache size 0 max 16384 2024-04-25T11:47:00.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.040+0000 7f83615d5740 1 Processor -- start 2024-04-25T11:47:00.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.040+0000 7f83615d5740 1 -- start start 2024-04-25T11:47:00.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.041+0000 7f83615d5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13e7d4d0 0x55ad14253140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:00.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.041+0000 7f83615d5740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad13e856b0 0x55ad14249ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:00.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.041+0000 7f83615d5740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad13ea6390 0x55ad142600c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:00.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.041+0000 7f83615d5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f83541ada10 con 0x55ad13e7d4d0 2024-04-25T11:47:00.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.041+0000 7f83615d5740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f83541af580 con 0x55ad13ea6390 2024-04-25T11:47:00.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.041+0000 7f83615d5740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f83541ae0e0 con 0x55ad13e856b0 2024-04-25T11:47:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.041+0000 7f8353fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13e7d4d0 0x55ad14253140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.041+0000 7f8353fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13e7d4d0 0x55ad14253140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59922/0 (socket says 172.21.15.90:59922) 2024-04-25T11:47:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.041+0000 7f8353fff640 1 -- 172.21.15.90:0/508862731 learned_addr learned my addr 172.21.15.90:0/508862731 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:47:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.041+0000 7f8358cbc640 1 --2- 172.21.15.90:0/508862731 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad13ea6390 0x55ad142600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.041+0000 7f8353fff640 1 -- 172.21.15.90:0/508862731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad13e856b0 msgr2=0x55ad14249ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:47:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.041+0000 7f8353fff640 1 --2- 172.21.15.90:0/508862731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad13e856b0 0x55ad14249ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.041+0000 7f8353fff640 1 -- 172.21.15.90:0/508862731 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad13ea6390 msgr2=0x55ad142600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:47:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.041+0000 7f8353fff640 1 --2- 172.21.15.90:0/508862731 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad13ea6390 0x55ad142600c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.041+0000 7f8353fff640 1 -- 172.21.15.90:0/508862731 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad1424afa0 con 0x55ad13e7d4d0 2024-04-25T11:47:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.041+0000 7f8358cbc640 1 --2- 172.21.15.90:0/508862731 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad13ea6390 0x55ad142600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:47:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.041+0000 7f8353fff640 1 --2- 172.21.15.90:0/508862731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13e7d4d0 0x55ad14253140 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f83540695c0 tx=0x7f8344012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c72c5782c39c7003 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.041+0000 7f8339ffb640 1 -- 172.21.15.90:0/508862731 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8344005020 con 0x55ad13e7d4d0 2024-04-25T11:47:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.041+0000 7f8339ffb640 1 -- 172.21.15.90:0/508862731 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83440047a0 con 0x55ad13e7d4d0 2024-04-25T11:47:00.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.041+0000 7f8339ffb640 1 -- 172.21.15.90:0/508862731 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834401f350 con 0x55ad13e7d4d0 2024-04-25T11:47:00.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.042+0000 7f83615d5740 1 -- 172.21.15.90:0/508862731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13e7d4d0 msgr2=0x55ad14253140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:00.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.042+0000 7f83615d5740 1 --2- 172.21.15.90:0/508862731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13e7d4d0 0x55ad14253140 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f83540695c0 tx=0x7f8344012820 comp rx=0 tx=0).stop 2024-04-25T11:47:00.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.042+0000 7f83615d5740 1 -- 172.21.15.90:0/508862731 shutdown_connections 2024-04-25T11:47:00.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.042+0000 7f83615d5740 1 --2- 172.21.15.90:0/508862731 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad13ea6390 0x55ad142600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.042+0000 7f83615d5740 1 --2- 172.21.15.90:0/508862731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad13e856b0 0x55ad14249ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.042+0000 7f83615d5740 1 --2- 172.21.15.90:0/508862731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13e7d4d0 0x55ad14253140 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.042+0000 7f83615d5740 1 -- 172.21.15.90:0/508862731 >> 172.21.15.90:0/508862731 conn(0x55ad13e9a4e0 msgr2=0x55ad142497b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:00.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.042+0000 7f83615d5740 1 -- 172.21.15.90:0/508862731 shutdown_connections 2024-04-25T11:47:00.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.042+0000 7f83615d5740 1 -- 172.21.15.90:0/508862731 wait complete. 2024-04-25T11:47:00.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.043+0000 7f83615d5740 1 -- start start 2024-04-25T11:47:00.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.043+0000 7f83615d5740 1 Processor -- start 2024-04-25T11:47:00.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.043+0000 7f83615d5740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad13e7d4d0 0x55ad142c6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:00.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.043+0000 7f83615d5740 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad13e856b0 0x55ad142c7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:00.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.043+0000 7f83615d5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13ea6390 0x55ad142577d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:00.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.043+0000 7f83615d5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f83541aeeb0 con 0x55ad13ea6390 2024-04-25T11:47:00.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.043+0000 7f83615d5740 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f83541ae7b0 con 0x55ad13e856b0 2024-04-25T11:47:00.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.043+0000 7f83615d5740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f83541aba10 con 0x55ad13e7d4d0 2024-04-25T11:47:00.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.043+0000 7f83615d5740 10 client.0 _ll_register_callbacks cb 0x7f83480045b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:47:00.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.043+0000 7f8358cbc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13ea6390 0x55ad142577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:00.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.043+0000 7f8358cbc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13ea6390 0x55ad142577d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59936/0 (socket says 172.21.15.90:59936) 2024-04-25T11:47:00.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.043+0000 7f8358cbc640 1 -- 172.21.15.90:0/497840429 learned_addr learned my addr 172.21.15.90:0/497840429 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:47:00.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.043+0000 7f8353fff640 1 --2- 172.21.15.90:0/497840429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad13e7d4d0 0x55ad142c6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:00.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.044+0000 7f83537fe640 1 --2- 172.21.15.90:0/497840429 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad13e856b0 0x55ad142c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:00.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.044+0000 7f8358cbc640 1 -- 172.21.15.90:0/497840429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad13e7d4d0 msgr2=0x55ad142c6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:47:00.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.044+0000 7f8358cbc640 1 --2- 172.21.15.90:0/497840429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad13e7d4d0 0x55ad142c6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.044+0000 7f8358cbc640 1 -- 172.21.15.90:0/497840429 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad13e856b0 msgr2=0x55ad142c7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:47:00.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.044+0000 7f8358cbc640 1 --2- 172.21.15.90:0/497840429 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad13e856b0 0x55ad142c7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.044+0000 7f8358cbc640 1 -- 172.21.15.90:0/497840429 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad14257e10 con 0x55ad13ea6390 2024-04-25T11:47:00.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.044+0000 7f8353fff640 1 --2- 172.21.15.90:0/497840429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad13e7d4d0 0x55ad142c6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:47:00.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.044+0000 7f8358cbc640 1 --2- 172.21.15.90:0/497840429 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13ea6390 0x55ad142577d0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f834c009480 tx=0x7f834c009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf04dbd353e7e8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:00.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.044+0000 7f83337fe640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-25T11:47:00.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.044+0000 7f83337fe640 1 -- 172.21.15.90:0/497840429 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834c042040 con 0x55ad13ea6390 2024-04-25T11:47:00.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.044+0000 7f83337fe640 1 -- 172.21.15.90:0/497840429 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7f834c004040 con 0x55ad13ea6390 2024-04-25T11:47:00.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.044+0000 7f83615d5740 1 -- 172.21.15.90:0/497840429 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83440047a0 con 0x55ad13ea6390 2024-04-25T11:47:00.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.044+0000 7f83337fe640 1 -- 172.21.15.90:0/497840429 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f834c062050 con 0x55ad13ea6390 2024-04-25T11:47:00.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.044+0000 7f83615d5740 10 client.19482 fetch_fsmap learned FSMap version 29 2024-04-25T11:47:00.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.044+0000 7f83615d5740 1 -- 172.21.15.90:0/497840429 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ad142ce560 con 0x55ad13ea6390 2024-04-25T11:47:00.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.044+0000 7f83337fe640 1 -- 172.21.15.90:0/497840429 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83440047a0 con 0x55ad13ea6390 2024-04-25T11:47:00.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.044+0000 7f83337fe640 1 -- 172.21.15.90:0/497840429 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 29) ==== 8358+0+0 (secure 0 0 0) 0x7f834c02fac0 con 0x55ad13ea6390 2024-04-25T11:47:00.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.045+0000 7f83615d5740 10 client.19482 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T11:47:00.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.045+0000 7f83615d5740 10 client.19482 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T11:47:00.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.045+0000 7f83615d5740 20 client.19482 populate_metadata read hostname 'smithi090' 2024-04-25T11:47:00.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.045+0000 7f83615d5740 1 --2- 172.21.15.90:0/497840429 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ad13e3a3a0 0x55ad142d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:00.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.045+0000 7f83615d5740 4 client.19482 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T11:47:00.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.045+0000 7f83615d5740 1 -- 172.21.15.90:0/497840429 --> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ad141b4800 con 0x55ad13e3a3a0 2024-04-25T11:47:00.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.045+0000 7f8353fff640 1 --2- 172.21.15.90:0/497840429 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ad13e3a3a0 0x55ad142d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:00.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.045+0000 7f83337fe640 10 client.19482 ms_handle_connect on v2:172.21.15.138:6838/1849836338 2024-04-25T11:47:00.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.045+0000 7f8353fff640 1 --2- 172.21.15.90:0/497840429 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ad13e3a3a0 0x55ad142d08c0 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17c0cbdc060cdb9c server_cookie=1892f81ecf73f0d2 in_seq=0 out_seq=0 2024-04-25T11:47:00.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.045+0000 7f83337fe640 1 -- 172.21.15.90:0/497840429 <== mds.0 v2:172.21.15.138:6838/1849836338 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ad141b4800 con 0x55ad13e3a3a0 2024-04-25T11:47:00.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.045+0000 7f83337fe640 10 client.19482 handle_command_reply: tid=0 2024-04-25T11:47:00.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.045+0000 7f83615d5740 1 client.19482 shutdown 2024-04-25T11:47:00.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.046+0000 7f83615d5740 1 -- 172.21.15.90:0/497840429 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13ea6390 msgr2=0x55ad142577d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:00.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.046+0000 7f83615d5740 1 --2- 172.21.15.90:0/497840429 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13ea6390 0x55ad142577d0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f834c009480 tx=0x7f834c009360 comp rx=0 tx=0).stop 2024-04-25T11:47:00.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.046+0000 7f83615d5740 1 -- 172.21.15.90:0/497840429 shutdown_connections 2024-04-25T11:47:00.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.046+0000 7f83615d5740 1 --2- 172.21.15.90:0/497840429 >> [v2:172.21.15.138:6838/1849836338,v1:172.21.15.138:6839/1849836338] conn(0x55ad13e3a3a0 0x55ad142d08c0 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.046+0000 7f83615d5740 1 --2- 172.21.15.90:0/497840429 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad13ea6390 0x55ad142577d0 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.046+0000 7f83337fe640 0 client.19482 ms_handle_reset on v2:172.21.15.138:6838/1849836338 2024-04-25T11:47:00.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.046+0000 7f83615d5740 1 --2- 172.21.15.90:0/497840429 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55ad13e856b0 0x55ad142c7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.046+0000 7f83615d5740 1 --2- 172.21.15.90:0/497840429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad13e7d4d0 0x55ad142c6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.046+0000 7f83615d5740 1 -- 172.21.15.90:0/497840429 >> 172.21.15.90:0/497840429 conn(0x55ad13e9a4e0 msgr2=0x55ad14242610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:00.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.046+0000 7f83615d5740 1 -- 172.21.15.90:0/497840429 shutdown_connections 2024-04-25T11:47:00.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.046+0000 7f83615d5740 1 -- 172.21.15.90:0/497840429 wait complete. 2024-04-25T11:47:00.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.046+0000 7f83615d5740 20 client.19482 trim_cache size 0 max 16384 2024-04-25T11:47:00.087 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-25T11:47:00.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.046+0000 7f835a71f640 1 -- 172.21.15.90:0/2875164187 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f833c0afc60 msgr2=0x7f833c0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:00.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.046+0000 7f835a71f640 1 --2- 172.21.15.90:0/2875164187 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f833c0afc60 0x7f833c0b2170 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f834c04d000 tx=0x7f834c04e000 comp rx=0 tx=0).stop 2024-04-25T11:47:00.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.046+0000 7f835a71f640 1 -- 172.21.15.90:0/2875164187 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8354108a10 msgr2=0x7f8354199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:00.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.046+0000 7f835a71f640 1 --2- 172.21.15.90:0/2875164187 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8354108a10 0x7f8354199bf0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f834800e9f0 tx=0x7f834800eeb0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.046+0000 7f835a71f640 1 -- 172.21.15.90:0/2875164187 shutdown_connections 2024-04-25T11:47:00.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.046+0000 7f835a71f640 1 --2- 172.21.15.90:0/2875164187 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f833c0afc60 0x7f833c0b2170 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.046+0000 7f835a71f640 1 --2- 172.21.15.90:0/2875164187 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8354108eb0 0x7f835419deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.046+0000 7f835a71f640 1 --2- 172.21.15.90:0/2875164187 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8354108a10 0x7f8354199bf0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.046+0000 7f835a71f640 1 --2- 172.21.15.90:0/2875164187 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83541031c0 0x7f8354199690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:00.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.046+0000 7f835a71f640 1 -- 172.21.15.90:0/2875164187 >> 172.21.15.90:0/2875164187 conn(0x7f8354078110 msgr2=0x7f8354100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:00.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.046+0000 7f835a71f640 1 -- 172.21.15.90:0/2875164187 shutdown_connections 2024-04-25T11:47:00.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:00.046+0000 7f835a71f640 1 -- 172.21.15.90:0/2875164187 wait complete. 2024-04-25T11:47:00.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:00 smithi090 ceph-mon[29575]: pgmap v370: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 26 KiB/s rd, 21 KiB/s wr, 228 op/s 2024-04-25T11:47:00.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:00 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2491798680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:47:00.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:00 smithi138 ceph-mon[32328]: pgmap v370: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 26 KiB/s rd, 21 KiB/s wr, 228 op/s 2024-04-25T11:47:00.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:00 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2491798680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:47:00.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:00 smithi156 ceph-mon[33045]: pgmap v370: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 26 KiB/s rd, 21 KiB/s wr, 228 op/s 2024-04-25T11:47:00.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:00 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2491798680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:47:01.091 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-04-25T11:47:01.091 INFO:tasks.fwd_scrub:done joining 2024-04-25T11:47:01.092 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-25T11:47:01.103 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:47:01.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.228+0000 7ff8ef7c5640 1 -- 172.21.15.90:0/1005828851 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8e81038c0 msgr2=0x7ff8e810f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:01.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.228+0000 7ff8ef7c5640 1 --2- 172.21.15.90:0/1005828851 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8e81038c0 0x7ff8e810f6d0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0009ca0 tx=0x7ff8e002a8c0 comp rx=0 tx=0).stop 2024-04-25T11:47:01.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.228+0000 7ff8ef7c5640 1 -- 172.21.15.90:0/1005828851 shutdown_connections 2024-04-25T11:47:01.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.228+0000 7ff8ef7c5640 1 --2- 172.21.15.90:0/1005828851 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8e81038c0 0x7ff8e810f6d0 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:01.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.228+0000 7ff8ef7c5640 1 --2- 172.21.15.90:0/1005828851 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8e81090e0 0x7ff8e8103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:01.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.228+0000 7ff8ef7c5640 1 --2- 172.21.15.90:0/1005828851 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff8e8108c40 0x7ff8e8102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:01.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.228+0000 7ff8ef7c5640 1 -- 172.21.15.90:0/1005828851 >> 172.21.15.90:0/1005828851 conn(0x7ff8e80fd850 msgr2=0x7ff8e80ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:01.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.228+0000 7ff8ef7c5640 1 -- 172.21.15.90:0/1005828851 shutdown_connections 2024-04-25T11:47:01.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.228+0000 7ff8ef7c5640 1 -- 172.21.15.90:0/1005828851 wait complete. 2024-04-25T11:47:01.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.229+0000 7ff8ef7c5640 1 Processor -- start 2024-04-25T11:47:01.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.229+0000 7ff8ef7c5640 1 -- start start 2024-04-25T11:47:01.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.229+0000 7ff8ef7c5640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff8e81038c0 0x7ff8e81997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:01.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.229+0000 7ff8ef7c5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8e8108c40 0x7ff8e8199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:01.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.229+0000 7ff8ef7c5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8e81090e0 0x7ff8e819dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:01.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.229+0000 7ff8ef7c5640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e8112e60 con 0x7ff8e81090e0 2024-04-25T11:47:01.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.229+0000 7ff8ef7c5640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e8112ce0 con 0x7ff8e81038c0 2024-04-25T11:47:01.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.229+0000 7ff8ef7c5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e8112fe0 con 0x7ff8e8108c40 2024-04-25T11:47:01.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.229+0000 7ff8edd62640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8e81090e0 0x7ff8e819dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:01.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.229+0000 7ff8edd62640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8e81090e0 0x7ff8e819dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59962/0 (socket says 172.21.15.90:59962) 2024-04-25T11:47:01.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.229+0000 7ff8edd62640 1 -- 172.21.15.90:0/451070098 learned_addr learned my addr 172.21.15.90:0/451070098 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:47:01.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.230+0000 7ff8ecd60640 1 --2- 172.21.15.90:0/451070098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8e8108c40 0x7ff8e8199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:01.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.230+0000 7ff8ed561640 1 --2- 172.21.15.90:0/451070098 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff8e81038c0 0x7ff8e81997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:01.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.230+0000 7ff8edd62640 1 -- 172.21.15.90:0/451070098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8e8108c40 msgr2=0x7ff8e8199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:01.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.230+0000 7ff8edd62640 1 --2- 172.21.15.90:0/451070098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8e8108c40 0x7ff8e8199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:01.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.230+0000 7ff8edd62640 1 -- 172.21.15.90:0/451070098 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff8e81038c0 msgr2=0x7ff8e81997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:01.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.230+0000 7ff8edd62640 1 --2- 172.21.15.90:0/451070098 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff8e81038c0 0x7ff8e81997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:01.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.230+0000 7ff8edd62640 1 -- 172.21.15.90:0/451070098 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8e819e580 con 0x7ff8e81090e0 2024-04-25T11:47:01.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.230+0000 7ff8ed561640 1 --2- 172.21.15.90:0/451070098 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff8e81038c0 0x7ff8e81997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:47:01.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.230+0000 7ff8edd62640 1 --2- 172.21.15.90:0/451070098 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8e81090e0 0x7ff8e819dfc0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0006010 tx=0x7ff8e00319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:01.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.230+0000 7ff8de7fc640 1 -- 172.21.15.90:0/451070098 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e0049070 con 0x7ff8e81090e0 2024-04-25T11:47:01.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.230+0000 7ff8de7fc640 1 -- 172.21.15.90:0/451070098 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8e0003b80 con 0x7ff8e81090e0 2024-04-25T11:47:01.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.230+0000 7ff8de7fc640 1 -- 172.21.15.90:0/451070098 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e002f990 con 0x7ff8e81090e0 2024-04-25T11:47:01.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.230+0000 7ff8ef7c5640 1 -- 172.21.15.90:0/451070098 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8e819e810 con 0x7ff8e81090e0 2024-04-25T11:47:01.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.230+0000 7ff8ef7c5640 1 -- 172.21.15.90:0/451070098 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8e819ed00 con 0x7ff8e81090e0 2024-04-25T11:47:01.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.231+0000 7ff8ef7c5640 1 -- 172.21.15.90:0/451070098 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8e806a190 con 0x7ff8e81090e0 2024-04-25T11:47:01.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.232+0000 7ff8de7fc640 1 -- 172.21.15.90:0/451070098 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7ff8e004c070 con 0x7ff8e81090e0 2024-04-25T11:47:01.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.233+0000 7ff8de7fc640 1 --2- 172.21.15.90:0/451070098 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff8d40afc60 0x7ff8d40b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:01.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.233+0000 7ff8ed561640 1 --2- 172.21.15.90:0/451070098 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff8d40afc60 0x7ff8d40b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:01.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.233+0000 7ff8de7fc640 1 -- 172.21.15.90:0/451070098 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7ff8e01015f0 con 0x7ff8e81090e0 2024-04-25T11:47:01.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.234+0000 7ff8ed561640 1 --2- 172.21.15.90:0/451070098 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff8d40afc60 0x7ff8d40b2170 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff8e40058c0 tx=0x7ff8e4005850 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:01.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.237+0000 7ff8de7fc640 1 -- 172.21.15.90:0/451070098 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff8e00c9f10 con 0x7ff8e81090e0 2024-04-25T11:47:01.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.416+0000 7ff8ef7c5640 1 -- 172.21.15.90:0/451070098 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff8e806b3a0 con 0x7ff8e81090e0 2024-04-25T11:47:01.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.418+0000 7ff8de7fc640 1 -- 172.21.15.90:0/451070098 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+23366 (secure 0 0 0) 0x7ff8e00c9810 con 0x7ff8e81090e0 2024-04-25T11:47:01.405 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:47:01.405 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":92,"fsid":"58bd4e7a-02f6-11ef-bc93-c7b262605968","created":"2024-04-25T11:25:23.574550+0000","modified":"2024-04-25T11:46:33.909282+0000","last_up_change":"2024-04-25T11:32:23.852211+0000","last_in_change":"2024-04-25T11:32:04.188507+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:28:42.746649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:33:08.442925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:33:09.352886+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":91,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e54d888-e67b-4951-8ae7-e355fb6b10a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6803","nonce":1201629159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6805","nonce":1201629159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6809","nonce":1201629159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6807","nonce":1201629159}]},"public_addr":"172.21.15.90:6803/1201629159","cluster_addr":"172.21.15.90:6805/1201629159","heartbeat_back_addr":"172.21.15.90:6809/1201629159","heartbeat_front_addr":"172.21.15.90:6807/1201629159","state":["exists","up"]},{"osd":1,"uuid":"91a71c78-dae9-4bc7-9f6f-9e55cfefbe28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6801","nonce":3976851933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6803","nonce":3976851933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6807","nonce":3976851933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6805","nonce":3976851933}]},"public_addr":"172.21.15.138:6801/3976851933","cluster_addr":"172.21.15.138:6803/3976851933","heartbeat_back_addr":"172.21.15.138:6807/3976851933","heartbeat_front_addr":"172.21.15.138:6805/3976851933","state":["exists","up"]},{"osd":2,"uuid":"daff6c7f-80fa-4859-9339-ef7a5e840832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6801","nonce":1505807670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6803","nonce":1505807670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6807","nonce":1505807670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6805","nonce":1505807670}]},"public_addr":"172.21.15.156:6801/1505807670","cluster_addr":"172.21.15.156:6803/1505807670","heartbeat_back_addr":"172.21.15.156:6807/1505807670","heartbeat_front_addr":"172.21.15.156:6805/1505807670","state":["exists","up"]},{"osd":3,"uuid":"33fd94ca-7465-401a-bc1e-706b6b31b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6811","nonce":2520012383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6813","nonce":2520012383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6817","nonce":2520012383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6815","nonce":2520012383}]},"public_addr":"172.21.15.90:6811/2520012383","cluster_addr":"172.21.15.90:6813/2520012383","heartbeat_back_addr":"172.21.15.90:6817/2520012383","heartbeat_front_addr":"172.21.15.90:6815/2520012383","state":["exists","up"]},{"osd":4,"uuid":"c5326212-750c-49fd-96e3-d1e62bb4b7f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6809","nonce":3752678279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6811","nonce":3752678279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6815","nonce":3752678279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6813","nonce":3752678279}]},"public_addr":"172.21.15.138:6809/3752678279","cluster_addr":"172.21.15.138:6811/3752678279","heartbeat_back_addr":"172.21.15.138:6815/3752678279","heartbeat_front_addr":"172.21.15.138:6813/3752678279","state":["exists","up"]},{"osd":5,"uuid":"d04a1156-820b-4b43-be89-e16a8910c3aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6809","nonce":3591210191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6811","nonce":3591210191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6815","nonce":3591210191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6813","nonce":3591210191}]},"public_addr":"172.21.15.156:6809/3591210191","cluster_addr":"172.21.15.156:6811/3591210191","heartbeat_back_addr":"172.21.15.156:6815/3591210191","heartbeat_front_addr":"172.21.15.156:6813/3591210191","state":["exists","up"]},{"osd":6,"uuid":"75675aa3-e978-490c-8205-13d9868a3d6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6819","nonce":2364955456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6821","nonce":2364955456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6825","nonce":2364955456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6823","nonce":2364955456}]},"public_addr":"172.21.15.90:6819/2364955456","cluster_addr":"172.21.15.90:6821/2364955456","heartbeat_back_addr":"172.21.15.90:6825/2364955456","heartbeat_front_addr":"172.21.15.90:6823/2364955456","state":["exists","up"]},{"osd":7,"uuid":"1a95fdf5-f548-4271-bbcd-e330f7955eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6817","nonce":4012476387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6819","nonce":4012476387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6823","nonce":4012476387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6821","nonce":4012476387}]},"public_addr":"172.21.15.138:6817/4012476387","cluster_addr":"172.21.15.138:6819/4012476387","heartbeat_back_addr":"172.21.15.138:6823/4012476387","heartbeat_front_addr":"172.21.15.138:6821/4012476387","state":["exists","up"]},{"osd":8,"uuid":"605e10b5-5806-4c1f-8e28-a7994d34ddce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6817","nonce":101092224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6819","nonce":101092224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6823","nonce":101092224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6821","nonce":101092224}]},"public_addr":"172.21.15.156:6817/101092224","cluster_addr":"172.21.15.156:6819/101092224","heartbeat_back_addr":"172.21.15.156:6823/101092224","heartbeat_front_addr":"172.21.15.156:6821/101092224","state":["exists","up"]},{"osd":9,"uuid":"d70fdf5e-9d0f-483d-a7d8-505b79b8a418","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6827","nonce":1171265966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6829","nonce":1171265966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6833","nonce":1171265966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6831","nonce":1171265966}]},"public_addr":"172.21.15.90:6827/1171265966","cluster_addr":"172.21.15.90:6829/1171265966","heartbeat_back_addr":"172.21.15.90:6833/1171265966","heartbeat_front_addr":"172.21.15.90:6831/1171265966","state":["exists","up"]},{"osd":10,"uuid":"1ec86c7d-c3a4-432d-9664-8e40836e1e20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6825","nonce":4264666232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6827","nonce":4264666232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6831","nonce":4264666232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6829","nonce":4264666232}]},"public_addr":"172.21.15.138:6825/4264666232","cluster_addr":"172.21.15.138:6827/4264666232","heartbeat_back_addr":"172.21.15.138:6831/4264666232","heartbeat_front_addr":"172.21.15.138:6829/4264666232","state":["exists","up"]},{"osd":11,"uuid":"51cbe729-292e-4d7b-b8e9-e1e2821ffe76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6825","nonce":1224251744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6827","nonce":1224251744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6831","nonce":1224251744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6829","nonce":1224251744}]},"public_addr":"172.21.15.156:6825/1224251744","cluster_addr":"172.21.15.156:6827/1224251744","heartbeat_back_addr":"172.21.15.156:6831/1224251744","heartbeat_front_addr":"172.21.15.156:6829/1224251744","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:27:50.794226+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:15.030278+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:38.518163+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:03.107351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:27.500707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:51.463545+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:16.015076+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:41.182071+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:05.362309+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:31.048936+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:56.488509+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:32:22.129178+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":7}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/813599224":"2024-04-26T11:35:08.201598+0000","172.21.15.90:6800/219449516":"2024-04-26T11:35:08.201598+0000","172.21.15.90:0/478282854":"2024-04-26T11:35:08.201598+0000","172.21.15.90:0/2202319376":"2024-04-26T11:35:08.201598+0000","172.21.15.90:0/2381633953":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6800/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/2190846900":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/888963322":"2024-04-26T11:35:08.201598+0000","172.21.15.90:6801/219449516":"2024-04-26T11:35:08.201598+0000","172.21.15.90:0/1707792941":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6800/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6801/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/6061052":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1849633392":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6801/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/2277490702":"2024-04-26T11:26:16.655278+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:47:01.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.420+0000 7ff8ef7c5640 1 -- 172.21.15.90:0/451070098 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff8d40afc60 msgr2=0x7ff8d40b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:01.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.420+0000 7ff8ef7c5640 1 --2- 172.21.15.90:0/451070098 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff8d40afc60 0x7ff8d40b2170 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff8e40058c0 tx=0x7ff8e4005850 comp rx=0 tx=0).stop 2024-04-25T11:47:01.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.420+0000 7ff8ef7c5640 1 -- 172.21.15.90:0/451070098 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8e81090e0 msgr2=0x7ff8e819dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:01.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.420+0000 7ff8ef7c5640 1 --2- 172.21.15.90:0/451070098 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8e81090e0 0x7ff8e819dfc0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0006010 tx=0x7ff8e00319f0 comp rx=0 tx=0).stop 2024-04-25T11:47:01.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.420+0000 7ff8ef7c5640 1 -- 172.21.15.90:0/451070098 shutdown_connections 2024-04-25T11:47:01.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.420+0000 7ff8ef7c5640 1 --2- 172.21.15.90:0/451070098 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff8d40afc60 0x7ff8d40b2170 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:01.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.420+0000 7ff8ef7c5640 1 --2- 172.21.15.90:0/451070098 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8e81090e0 0x7ff8e819dfc0 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:01.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.420+0000 7ff8ef7c5640 1 --2- 172.21.15.90:0/451070098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8e8108c40 0x7ff8e8199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:01.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.420+0000 7ff8ef7c5640 1 --2- 172.21.15.90:0/451070098 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff8e81038c0 0x7ff8e81997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:01.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.420+0000 7ff8ef7c5640 1 -- 172.21.15.90:0/451070098 >> 172.21.15.90:0/451070098 conn(0x7ff8e80fd850 msgr2=0x7ff8e80ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:01.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.420+0000 7ff8ef7c5640 1 -- 172.21.15.90:0/451070098 shutdown_connections 2024-04-25T11:47:01.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.420+0000 7ff8ef7c5640 1 -- 172.21.15.90:0/451070098 wait complete. 2024-04-25T11:47:01.418 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T11:28:42.746649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T11:33:08.442925+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T11:33:09.352886+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 91, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T11:47:01.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T11:47:01.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.544+0000 7ff22dbce640 1 -- 172.21.15.90:0/1229825157 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff228102af0 msgr2=0x7ff228102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:01.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.544+0000 7ff22dbce640 1 --2- 172.21.15.90:0/1229825157 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff228102af0 0x7ff228102f00 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7ff21c009900 tx=0x7ff21c02a410 comp rx=0 tx=0).stop 2024-04-25T11:47:01.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.544+0000 7ff22dbce640 1 -- 172.21.15.90:0/1229825157 shutdown_connections 2024-04-25T11:47:01.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.544+0000 7ff22dbce640 1 --2- 172.21.15.90:0/1229825157 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff22810d380 0x7ff22810f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:01.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.544+0000 7ff22dbce640 1 --2- 172.21.15.90:0/1229825157 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff228102af0 0x7ff228102f00 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:01.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.544+0000 7ff22dbce640 1 --2- 172.21.15.90:0/1229825157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff228108a20 0x7ff2281025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:01.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.544+0000 7ff22dbce640 1 -- 172.21.15.90:0/1229825157 >> 172.21.15.90:0/1229825157 conn(0x7ff228078100 msgr2=0x7ff228078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:01.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.544+0000 7ff22dbce640 1 -- 172.21.15.90:0/1229825157 shutdown_connections 2024-04-25T11:47:01.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.544+0000 7ff22dbce640 1 -- 172.21.15.90:0/1229825157 wait complete. 2024-04-25T11:47:01.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.544+0000 7ff22dbce640 1 Processor -- start 2024-04-25T11:47:01.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.545+0000 7ff22dbce640 1 -- start start 2024-04-25T11:47:01.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.545+0000 7ff22dbce640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff228102af0 0x7ff228199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:01.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.545+0000 7ff22dbce640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff228108a20 0x7ff228199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:01.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.545+0000 7ff22dbce640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff22810d380 0x7ff22819dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:01.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.545+0000 7ff22dbce640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff228112be0 con 0x7ff228102af0 2024-04-25T11:47:01.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.545+0000 7ff22dbce640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff228112a60 con 0x7ff228108a20 2024-04-25T11:47:01.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.545+0000 7ff22dbce640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff228112d60 con 0x7ff22810d380 2024-04-25T11:47:01.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.545+0000 7ff2277fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff228102af0 0x7ff228199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:01.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.545+0000 7ff2277fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff228102af0 0x7ff228199680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44670/0 (socket says 172.21.15.90:44670) 2024-04-25T11:47:01.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.545+0000 7ff2277fe640 1 -- 172.21.15.90:0/1523221036 learned_addr learned my addr 172.21.15.90:0/1523221036 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:47:01.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.545+0000 7ff227fff640 1 --2- 172.21.15.90:0/1523221036 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff22810d380 0x7ff22819dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:01.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.545+0000 7ff226ffd640 1 --2- 172.21.15.90:0/1523221036 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff228108a20 0x7ff228199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:01.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.545+0000 7ff2277fe640 1 -- 172.21.15.90:0/1523221036 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff22810d380 msgr2=0x7ff22819dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:01.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.545+0000 7ff2277fe640 1 --2- 172.21.15.90:0/1523221036 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff22810d380 0x7ff22819dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:01.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.545+0000 7ff2277fe640 1 -- 172.21.15.90:0/1523221036 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff228108a20 msgr2=0x7ff228199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:01.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.545+0000 7ff2277fe640 1 --2- 172.21.15.90:0/1523221036 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff228108a20 0x7ff228199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:01.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.545+0000 7ff2277fe640 1 -- 172.21.15.90:0/1523221036 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff22819e4e0 con 0x7ff228102af0 2024-04-25T11:47:01.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.545+0000 7ff226ffd640 1 --2- 172.21.15.90:0/1523221036 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff228108a20 0x7ff228199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:47:01.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.546+0000 7ff2277fe640 1 --2- 172.21.15.90:0/1523221036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff228102af0 0x7ff228199680 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7ff21800ebe0 tx=0x7ff218007590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:01.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.546+0000 7ff224ff9640 1 -- 172.21.15.90:0/1523221036 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21800e0d0 con 0x7ff228102af0 2024-04-25T11:47:01.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.546+0000 7ff22dbce640 1 -- 172.21.15.90:0/1523221036 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff22819e770 con 0x7ff228102af0 2024-04-25T11:47:01.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.546+0000 7ff224ff9640 1 -- 172.21.15.90:0/1523221036 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff21800e270 con 0x7ff228102af0 2024-04-25T11:47:01.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.546+0000 7ff22dbce640 1 -- 172.21.15.90:0/1523221036 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff22819ecc0 con 0x7ff228102af0 2024-04-25T11:47:01.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.546+0000 7ff224ff9640 1 -- 172.21.15.90:0/1523221036 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff218005880 con 0x7ff228102af0 2024-04-25T11:47:01.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.547+0000 7ff22dbce640 1 -- 172.21.15.90:0/1523221036 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff22819a9d0 con 0x7ff228102af0 2024-04-25T11:47:01.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.548+0000 7ff224ff9640 1 -- 172.21.15.90:0/1523221036 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7ff218004a40 con 0x7ff228102af0 2024-04-25T11:47:01.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.549+0000 7ff224ff9640 1 --2- 172.21.15.90:0/1523221036 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff2100afc60 0x7ff2100b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:01.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.549+0000 7ff226ffd640 1 --2- 172.21.15.90:0/1523221036 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff2100afc60 0x7ff2100b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:01.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.549+0000 7ff224ff9640 1 -- 172.21.15.90:0/1523221036 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7ff2180e0c50 con 0x7ff228102af0 2024-04-25T11:47:01.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.549+0000 7ff226ffd640 1 --2- 172.21.15.90:0/1523221036 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff2100afc60 0x7ff2100b2170 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff21c000c00 tx=0x7ff21c031400 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:01.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.552+0000 7ff224ff9640 1 -- 172.21.15.90:0/1523221036 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff2180a9460 con 0x7ff228102af0 2024-04-25T11:47:01.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.730+0000 7ff22dbce640 1 -- 172.21.15.90:0/1523221036 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff228106f00 con 0x7ff228102af0 2024-04-25T11:47:01.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.730+0000 7ff224ff9640 1 -- 172.21.15.90:0/1523221036 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7ff2180910a0 con 0x7ff228102af0 2024-04-25T11:47:01.717 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T11:47:01.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.732+0000 7ff22dbce640 1 -- 172.21.15.90:0/1523221036 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff2100afc60 msgr2=0x7ff2100b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:01.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.732+0000 7ff22dbce640 1 --2- 172.21.15.90:0/1523221036 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff2100afc60 0x7ff2100b2170 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff21c000c00 tx=0x7ff21c031400 comp rx=0 tx=0).stop 2024-04-25T11:47:01.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.732+0000 7ff22dbce640 1 -- 172.21.15.90:0/1523221036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff228102af0 msgr2=0x7ff228199680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:01.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.732+0000 7ff22dbce640 1 --2- 172.21.15.90:0/1523221036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff228102af0 0x7ff228199680 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7ff21800ebe0 tx=0x7ff218007590 comp rx=0 tx=0).stop 2024-04-25T11:47:01.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.732+0000 7ff22dbce640 1 -- 172.21.15.90:0/1523221036 shutdown_connections 2024-04-25T11:47:01.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.732+0000 7ff22dbce640 1 --2- 172.21.15.90:0/1523221036 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff2100afc60 0x7ff2100b2170 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:01.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.732+0000 7ff22dbce640 1 --2- 172.21.15.90:0/1523221036 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff22810d380 0x7ff22819dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:01.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.732+0000 7ff22dbce640 1 --2- 172.21.15.90:0/1523221036 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff228108a20 0x7ff228199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:01.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.732+0000 7ff22dbce640 1 --2- 172.21.15.90:0/1523221036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff228102af0 0x7ff228199680 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:01.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.732+0000 7ff22dbce640 1 -- 172.21.15.90:0/1523221036 >> 172.21.15.90:0/1523221036 conn(0x7ff228078100 msgr2=0x7ff2281040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:01.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.733+0000 7ff22dbce640 1 -- 172.21.15.90:0/1523221036 shutdown_connections 2024-04-25T11:47:01.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.733+0000 7ff22dbce640 1 -- 172.21.15.90:0/1523221036 wait complete. 2024-04-25T11:47:01.730 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T11:47:01.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.855+0000 7f2508642640 1 -- 172.21.15.90:0/210004821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f250010d380 msgr2=0x7f250010f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:01.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.855+0000 7f2508642640 1 --2- 172.21.15.90:0/210004821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f250010d380 0x7f250010f7a0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f24f8009ca0 tx=0x7f24f802a8c0 comp rx=0 tx=0).stop 2024-04-25T11:47:01.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.855+0000 7f2508642640 1 -- 172.21.15.90:0/210004821 shutdown_connections 2024-04-25T11:47:01.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.855+0000 7f2508642640 1 --2- 172.21.15.90:0/210004821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f250010d380 0x7f250010f7a0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:01.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.855+0000 7f2508642640 1 --2- 172.21.15.90:0/210004821 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2500102af0 0x7f2500102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:01.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.855+0000 7f2508642640 1 --2- 172.21.15.90:0/210004821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2500108a20 0x7f25001025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:01.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.855+0000 7f2508642640 1 -- 172.21.15.90:0/210004821 >> 172.21.15.90:0/210004821 conn(0x7f2500078100 msgr2=0x7f2500078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:01.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.855+0000 7f2508642640 1 -- 172.21.15.90:0/210004821 shutdown_connections 2024-04-25T11:47:01.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.855+0000 7f2508642640 1 -- 172.21.15.90:0/210004821 wait complete. 2024-04-25T11:47:01.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.856+0000 7f2508642640 1 Processor -- start 2024-04-25T11:47:01.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.856+0000 7f2508642640 1 -- start start 2024-04-25T11:47:01.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.856+0000 7f2508642640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2500102af0 0x7f25001995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:01.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.856+0000 7f2508642640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2500108a20 0x7f2500199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:01.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.856+0000 7f2508642640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f250010d380 0x7f250019de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:01.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.856+0000 7f2508642640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2500112c00 con 0x7f2500102af0 2024-04-25T11:47:01.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.856+0000 7f2508642640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2500112a80 con 0x7f2500108a20 2024-04-25T11:47:01.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.856+0000 7f2508642640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2500112d80 con 0x7f250010d380 2024-04-25T11:47:01.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.856+0000 7f25063de640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2500102af0 0x7f25001995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:01.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.856+0000 7f25063de640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2500102af0 0x7f25001995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44676/0 (socket says 172.21.15.90:44676) 2024-04-25T11:47:01.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.856+0000 7f25063de640 1 -- 172.21.15.90:0/3508488740 learned_addr learned my addr 172.21.15.90:0/3508488740 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:47:01.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.856+0000 7f2505bdd640 1 --2- 172.21.15.90:0/3508488740 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2500108a20 0x7f2500199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:01.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.856+0000 7f2506bdf640 1 --2- 172.21.15.90:0/3508488740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f250010d380 0x7f250019de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:01.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.856+0000 7f25063de640 1 -- 172.21.15.90:0/3508488740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f250010d380 msgr2=0x7f250019de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:01.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.856+0000 7f25063de640 1 --2- 172.21.15.90:0/3508488740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f250010d380 0x7f250019de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:01.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.856+0000 7f25063de640 1 -- 172.21.15.90:0/3508488740 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2500108a20 msgr2=0x7f2500199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:01.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.856+0000 7f25063de640 1 --2- 172.21.15.90:0/3508488740 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2500108a20 0x7f2500199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:01.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.856+0000 7f25063de640 1 -- 172.21.15.90:0/3508488740 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f250019e440 con 0x7f2500102af0 2024-04-25T11:47:01.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.856+0000 7f2505bdd640 1 --2- 172.21.15.90:0/3508488740 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2500108a20 0x7f2500199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:47:01.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.856+0000 7f2506bdf640 1 --2- 172.21.15.90:0/3508488740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f250010d380 0x7f250019de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:47:01.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.856+0000 7f25063de640 1 --2- 172.21.15.90:0/3508488740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2500102af0 0x7f25001995e0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f24fc00d740 tx=0x7f24fc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:01.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.857+0000 7f24f77fe640 1 -- 172.21.15.90:0/3508488740 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24fc017070 con 0x7f2500102af0 2024-04-25T11:47:01.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.857+0000 7f24f77fe640 1 -- 172.21.15.90:0/3508488740 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24fc00de50 con 0x7f2500102af0 2024-04-25T11:47:01.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.857+0000 7f2508642640 1 -- 172.21.15.90:0/3508488740 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f250019e730 con 0x7f2500102af0 2024-04-25T11:47:01.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.857+0000 7f24f77fe640 1 -- 172.21.15.90:0/3508488740 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24fc004ce0 con 0x7f2500102af0 2024-04-25T11:47:01.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.857+0000 7f2508642640 1 -- 172.21.15.90:0/3508488740 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f250019ec30 con 0x7f2500102af0 2024-04-25T11:47:01.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.858+0000 7f2508642640 1 -- 172.21.15.90:0/3508488740 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f250006afa0 con 0x7f2500102af0 2024-04-25T11:47:01.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.859+0000 7f24f77fe640 1 -- 172.21.15.90:0/3508488740 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f24fc004e80 con 0x7f2500102af0 2024-04-25T11:47:01.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.859+0000 7f24f77fe640 1 --2- 172.21.15.90:0/3508488740 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f24ec0afc60 0x7f24ec0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:01.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.860+0000 7f2505bdd640 1 --2- 172.21.15.90:0/3508488740 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f24ec0afc60 0x7f24ec0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:01.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.860+0000 7f24f77fe640 1 -- 172.21.15.90:0/3508488740 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7f24fc0def60 con 0x7f2500102af0 2024-04-25T11:47:01.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.860+0000 7f2505bdd640 1 --2- 172.21.15.90:0/3508488740 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f24ec0afc60 0x7f24ec0b2170 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f250010e660 tx=0x7f24f0005f90 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:01.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:01.863+0000 7f24f77fe640 1 -- 172.21.15.90:0/3508488740 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f250019ec30 con 0x7f2500102af0 2024-04-25T11:47:02.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.041+0000 7f2508642640 1 -- 172.21.15.90:0/3508488740 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f250002db50 con 0x7f2500102af0 2024-04-25T11:47:02.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.041+0000 7f24f77fe640 1 -- 172.21.15.90:0/3508488740 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f24fc0a7770 con 0x7f2500102af0 2024-04-25T11:47:02.028 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T11:47:02.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.043+0000 7f2508642640 1 -- 172.21.15.90:0/3508488740 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f24ec0afc60 msgr2=0x7f24ec0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:02.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.043+0000 7f2508642640 1 --2- 172.21.15.90:0/3508488740 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f24ec0afc60 0x7f24ec0b2170 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f250010e660 tx=0x7f24f0005f90 comp rx=0 tx=0).stop 2024-04-25T11:47:02.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.043+0000 7f2508642640 1 -- 172.21.15.90:0/3508488740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2500102af0 msgr2=0x7f25001995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:02.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.043+0000 7f2508642640 1 --2- 172.21.15.90:0/3508488740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2500102af0 0x7f25001995e0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f24fc00d740 tx=0x7f24fc00dc00 comp rx=0 tx=0).stop 2024-04-25T11:47:02.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.043+0000 7f2508642640 1 -- 172.21.15.90:0/3508488740 shutdown_connections 2024-04-25T11:47:02.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.043+0000 7f2508642640 1 --2- 172.21.15.90:0/3508488740 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f24ec0afc60 0x7f24ec0b2170 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:02.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.043+0000 7f2508642640 1 --2- 172.21.15.90:0/3508488740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f250010d380 0x7f250019de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:02.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.043+0000 7f2508642640 1 --2- 172.21.15.90:0/3508488740 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2500108a20 0x7f2500199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:02.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.043+0000 7f2508642640 1 --2- 172.21.15.90:0/3508488740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2500102af0 0x7f25001995e0 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:02.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.043+0000 7f2508642640 1 -- 172.21.15.90:0/3508488740 >> 172.21.15.90:0/3508488740 conn(0x7f2500078100 msgr2=0x7f25001040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:02.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.043+0000 7f2508642640 1 -- 172.21.15.90:0/3508488740 shutdown_connections 2024-04-25T11:47:02.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.043+0000 7f2508642640 1 -- 172.21.15.90:0/3508488740 wait complete. 2024-04-25T11:47:02.041 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T11:47:02.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.168+0000 7f720e95e640 1 -- 172.21.15.90:0/527468801 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7208108ed0 msgr2=0x7f7208102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:02.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.168+0000 7f720e95e640 1 --2- 172.21.15.90:0/527468801 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7208108ed0 0x7f7208102b20 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f71fc009900 tx=0x7f71fc02a410 comp rx=0 tx=0).stop 2024-04-25T11:47:02.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.168+0000 7f720e95e640 1 -- 172.21.15.90:0/527468801 shutdown_connections 2024-04-25T11:47:02.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.168+0000 7f720e95e640 1 --2- 172.21.15.90:0/527468801 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7208103190 0x7f720810f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:02.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.168+0000 7f720e95e640 1 --2- 172.21.15.90:0/527468801 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7208108ed0 0x7f7208102b20 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:02.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.168+0000 7f720e95e640 1 --2- 172.21.15.90:0/527468801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7208108a30 0x7f72081025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:02.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.168+0000 7f720e95e640 1 -- 172.21.15.90:0/527468801 >> 172.21.15.90:0/527468801 conn(0x7f72080fe260 msgr2=0x7f7208100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:02.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.168+0000 7f720e95e640 1 -- 172.21.15.90:0/527468801 shutdown_connections 2024-04-25T11:47:02.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.168+0000 7f720e95e640 1 -- 172.21.15.90:0/527468801 wait complete. 2024-04-25T11:47:02.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.169+0000 7f720e95e640 1 Processor -- start 2024-04-25T11:47:02.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.169+0000 7f720e95e640 1 -- start start 2024-04-25T11:47:02.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.169+0000 7f720e95e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7208103190 0x7f720819b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:02.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.169+0000 7f720e95e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7208108a30 0x7f720819bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:02.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.169+0000 7f720e95e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7208108ed0 0x7f720819c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:02.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.169+0000 7f720e95e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7208112ca0 con 0x7f7208108a30 2024-04-25T11:47:02.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.169+0000 7f720e95e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7208112b20 con 0x7f7208108ed0 2024-04-25T11:47:02.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.169+0000 7f720e95e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7208112e20 con 0x7f7208103190 2024-04-25T11:47:02.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.170+0000 7f72077fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7208108a30 0x7f720819bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:02.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.170+0000 7f72077fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7208108a30 0x7f720819bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44682/0 (socket says 172.21.15.90:44682) 2024-04-25T11:47:02.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.170+0000 7f72077fe640 1 -- 172.21.15.90:0/3900307528 learned_addr learned my addr 172.21.15.90:0/3900307528 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:47:02.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.170+0000 7f7207fff640 1 --2- 172.21.15.90:0/3900307528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7208103190 0x7f720819b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:02.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.170+0000 7f720cefb640 1 --2- 172.21.15.90:0/3900307528 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7208108ed0 0x7f720819c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:02.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.170+0000 7f72077fe640 1 -- 172.21.15.90:0/3900307528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7208103190 msgr2=0x7f720819b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:02.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.170+0000 7f72077fe640 1 --2- 172.21.15.90:0/3900307528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7208103190 0x7f720819b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:02.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.170+0000 7f72077fe640 1 -- 172.21.15.90:0/3900307528 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7208108ed0 msgr2=0x7f720819c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:02.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.170+0000 7f72077fe640 1 --2- 172.21.15.90:0/3900307528 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7208108ed0 0x7f720819c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:02.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.170+0000 7f72077fe640 1 -- 172.21.15.90:0/3900307528 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72081a6900 con 0x7f7208108a30 2024-04-25T11:47:02.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.170+0000 7f72077fe640 1 --2- 172.21.15.90:0/3900307528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7208108a30 0x7f720819bb00 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f71fc005ab0 tx=0x7f71fc003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:02.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.170+0000 7f72057fa640 1 -- 172.21.15.90:0/3900307528 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71fc03f070 con 0x7f7208108a30 2024-04-25T11:47:02.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.170+0000 7f72057fa640 1 -- 172.21.15.90:0/3900307528 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71fc031d60 con 0x7f7208108a30 2024-04-25T11:47:02.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.170+0000 7f720e95e640 1 -- 172.21.15.90:0/3900307528 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72081a6b90 con 0x7f7208108a30 2024-04-25T11:47:02.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.170+0000 7f72057fa640 1 -- 172.21.15.90:0/3900307528 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71fc035030 con 0x7f7208108a30 2024-04-25T11:47:02.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.170+0000 7f720e95e640 1 -- 172.21.15.90:0/3900307528 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72081a7080 con 0x7f7208108a30 2024-04-25T11:47:02.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.171+0000 7f720e95e640 1 -- 172.21.15.90:0/3900307528 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f720806a160 con 0x7f7208108a30 2024-04-25T11:47:02.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.173+0000 7f72057fa640 1 -- 172.21.15.90:0/3900307528 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f71fc042070 con 0x7f7208108a30 2024-04-25T11:47:02.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.173+0000 7f72057fa640 1 --2- 172.21.15.90:0/3900307528 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f71f00afc60 0x7f71f00b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:02.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.173+0000 7f7207fff640 1 --2- 172.21.15.90:0/3900307528 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f71f00afc60 0x7f71f00b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:02.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.174+0000 7f72057fa640 1 -- 172.21.15.90:0/3900307528 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7f71fc0fff00 con 0x7f7208108a30 2024-04-25T11:47:02.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.174+0000 7f7207fff640 1 --2- 172.21.15.90:0/3900307528 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f71f00afc60 0x7f71f00b2170 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f71f80058c0 tx=0x7f71f800a250 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:02.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.177+0000 7f72057fa640 1 -- 172.21.15.90:0/3900307528 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f71fc0c8710 con 0x7f7208108a30 2024-04-25T11:47:02.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.354+0000 7f720e95e640 1 -- 172.21.15.90:0/3900307528 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f72081a7320 con 0x7f7208108a30 2024-04-25T11:47:02.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.355+0000 7f72057fa640 1 -- 172.21.15.90:0/3900307528 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7f71fc0c8010 con 0x7f7208108a30 2024-04-25T11:47:02.341 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T11:47:02.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.356+0000 7f720e95e640 1 -- 172.21.15.90:0/3900307528 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f71f00afc60 msgr2=0x7f71f00b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:02.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.356+0000 7f720e95e640 1 --2- 172.21.15.90:0/3900307528 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f71f00afc60 0x7f71f00b2170 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f71f80058c0 tx=0x7f71f800a250 comp rx=0 tx=0).stop 2024-04-25T11:47:02.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.356+0000 7f720e95e640 1 -- 172.21.15.90:0/3900307528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7208108a30 msgr2=0x7f720819bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:02.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.356+0000 7f720e95e640 1 --2- 172.21.15.90:0/3900307528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7208108a30 0x7f720819bb00 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f71fc005ab0 tx=0x7f71fc003cd0 comp rx=0 tx=0).stop 2024-04-25T11:47:02.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.357+0000 7f720e95e640 1 -- 172.21.15.90:0/3900307528 shutdown_connections 2024-04-25T11:47:02.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.357+0000 7f720e95e640 1 --2- 172.21.15.90:0/3900307528 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f71f00afc60 0x7f71f00b2170 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:02.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.357+0000 7f720e95e640 1 --2- 172.21.15.90:0/3900307528 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7208108ed0 0x7f720819c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:02.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.357+0000 7f720e95e640 1 --2- 172.21.15.90:0/3900307528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7208108a30 0x7f720819bb00 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:02.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.357+0000 7f720e95e640 1 --2- 172.21.15.90:0/3900307528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7208103190 0x7f720819b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:02.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.357+0000 7f720e95e640 1 -- 172.21.15.90:0/3900307528 >> 172.21.15.90:0/3900307528 conn(0x7f72080fe260 msgr2=0x7f72080ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:02.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.357+0000 7f720e95e640 1 -- 172.21.15.90:0/3900307528 shutdown_connections 2024-04-25T11:47:02.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.357+0000 7f720e95e640 1 -- 172.21.15.90:0/3900307528 wait complete. 2024-04-25T11:47:02.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:02 smithi090 ceph-mon[29575]: pgmap v371: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 32 KiB/s rd, 26 KiB/s wr, 233 op/s 2024-04-25T11:47:02.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:02 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/451070098' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:47:02.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:02 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1523221036' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:47:02.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:02 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3508488740' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:47:02.354 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-25T11:47:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.476+0000 7fb40d7b2640 1 -- 172.21.15.90:0/1707125156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb408104c50 msgr2=0x7fb408105060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.476+0000 7fb40d7b2640 1 --2- 172.21.15.90:0/1707125156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb408104c50 0x7fb408105060 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc009900 tx=0x7fb3fc02a510 comp rx=0 tx=0).stop 2024-04-25T11:47:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.476+0000 7fb40d7b2640 1 -- 172.21.15.90:0/1707125156 shutdown_connections 2024-04-25T11:47:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.476+0000 7fb40d7b2640 1 --2- 172.21.15.90:0/1707125156 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb40810e060 0x7fb408110550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.476+0000 7fb40d7b2640 1 --2- 172.21.15.90:0/1707125156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb40810b6b0 0x7fb40810dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.476+0000 7fb40d7b2640 1 --2- 172.21.15.90:0/1707125156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb408104c50 0x7fb408105060 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.476+0000 7fb40d7b2640 1 -- 172.21.15.90:0/1707125156 >> 172.21.15.90:0/1707125156 conn(0x7fb408078110 msgr2=0x7fb408078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.476+0000 7fb40d7b2640 1 -- 172.21.15.90:0/1707125156 shutdown_connections 2024-04-25T11:47:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.476+0000 7fb40d7b2640 1 -- 172.21.15.90:0/1707125156 wait complete. 2024-04-25T11:47:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.477+0000 7fb40d7b2640 1 Processor -- start 2024-04-25T11:47:02.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.477+0000 7fb40d7b2640 1 -- start start 2024-04-25T11:47:02.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.477+0000 7fb40d7b2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb408104c50 0x7fb4081a2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:02.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.477+0000 7fb40d7b2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb40810b6b0 0x7fb4081a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:02.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.477+0000 7fb40d7b2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb40810e060 0x7fb40819c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:02.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.477+0000 7fb40d7b2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb408113d10 con 0x7fb40810b6b0 2024-04-25T11:47:02.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.477+0000 7fb40d7b2640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb408113b90 con 0x7fb408104c50 2024-04-25T11:47:02.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.477+0000 7fb40d7b2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb408113e90 con 0x7fb40810e060 2024-04-25T11:47:02.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.477+0000 7fb4067fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb40810b6b0 0x7fb4081a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:02.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.477+0000 7fb4067fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb40810b6b0 0x7fb4081a2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44696/0 (socket says 172.21.15.90:44696) 2024-04-25T11:47:02.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.477+0000 7fb4067fc640 1 -- 172.21.15.90:0/4183624255 learned_addr learned my addr 172.21.15.90:0/4183624255 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:47:02.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.477+0000 7fb406ffd640 1 --2- 172.21.15.90:0/4183624255 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb408104c50 0x7fb4081a2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:02.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.478+0000 7fb4077fe640 1 --2- 172.21.15.90:0/4183624255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb40810e060 0x7fb40819c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:02.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.478+0000 7fb4067fc640 1 -- 172.21.15.90:0/4183624255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb40810e060 msgr2=0x7fb40819c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:02.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.478+0000 7fb4067fc640 1 --2- 172.21.15.90:0/4183624255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb40810e060 0x7fb40819c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:02.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.478+0000 7fb4067fc640 1 -- 172.21.15.90:0/4183624255 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb408104c50 msgr2=0x7fb4081a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:02.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.478+0000 7fb4067fc640 1 --2- 172.21.15.90:0/4183624255 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb408104c50 0x7fb4081a2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:02.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.478+0000 7fb4067fc640 1 -- 172.21.15.90:0/4183624255 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb40819cf50 con 0x7fb40810b6b0 2024-04-25T11:47:02.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.478+0000 7fb4067fc640 1 --2- 172.21.15.90:0/4183624255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb40810b6b0 0x7fb4081a2b80 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fb3f800d770 tx=0x7fb3f800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:02.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.478+0000 7fb3effff640 1 -- 172.21.15.90:0/4183624255 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8017070 con 0x7fb40810b6b0 2024-04-25T11:47:02.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.478+0000 7fb3effff640 1 -- 172.21.15.90:0/4183624255 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3f80054d0 con 0x7fb40810b6b0 2024-04-25T11:47:02.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.478+0000 7fb40d7b2640 1 -- 172.21.15.90:0/4183624255 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb40819d240 con 0x7fb40810b6b0 2024-04-25T11:47:02.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.478+0000 7fb3effff640 1 -- 172.21.15.90:0/4183624255 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8004c30 con 0x7fb40810b6b0 2024-04-25T11:47:02.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.478+0000 7fb40d7b2640 1 -- 172.21.15.90:0/4183624255 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4081a9490 con 0x7fb40810b6b0 2024-04-25T11:47:02.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.479+0000 7fb40d7b2640 1 -- 172.21.15.90:0/4183624255 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb40806a710 con 0x7fb40810b6b0 2024-04-25T11:47:02.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.480+0000 7fb3effff640 1 -- 172.21.15.90:0/4183624255 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fb3f8004dd0 con 0x7fb40810b6b0 2024-04-25T11:47:02.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.481+0000 7fb3effff640 1 --2- 172.21.15.90:0/4183624255 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb3f00afc60 0x7fb3f00b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:02.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.481+0000 7fb406ffd640 1 --2- 172.21.15.90:0/4183624255 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb3f00afc60 0x7fb3f00b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:02.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.481+0000 7fb3effff640 1 -- 172.21.15.90:0/4183624255 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8658+0+0 (secure 0 0 0) 0x7fb3f80defd0 con 0x7fb40810b6b0 2024-04-25T11:47:02.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.482+0000 7fb406ffd640 1 --2- 172.21.15.90:0/4183624255 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb3f00afc60 0x7fb3f00b2170 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc005ab0 tx=0x7fb3fc005a20 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:02.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.485+0000 7fb3effff640 1 -- 172.21.15.90:0/4183624255 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb3f80a77e0 con 0x7fb40810b6b0 2024-04-25T11:47:02.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:02 smithi138 ceph-mon[32328]: pgmap v371: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 32 KiB/s rd, 26 KiB/s wr, 233 op/s 2024-04-25T11:47:02.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:02 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/451070098' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:47:02.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:02 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1523221036' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:47:02.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:02 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3508488740' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:47:02.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:02 smithi156 ceph-mon[33045]: pgmap v371: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 32 KiB/s rd, 26 KiB/s wr, 233 op/s 2024-04-25T11:47:02.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:02 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/451070098' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:47:02.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:02 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1523221036' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:47:02.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:02 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3508488740' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:47:02.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.712+0000 7fb40d7b2640 1 -- 172.21.15.90:0/4183624255 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fb40819dd80 con 0x7fb40810b6b0 2024-04-25T11:47:02.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.720+0000 7fb3effff640 1 -- 172.21.15.90:0/4183624255 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) ==== 81+0+803656 (secure 0 0 0) 0x7fb3f80a70e0 con 0x7fb40810b6b0 2024-04-25T11:47:02.717 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:47:02.718 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:47:02.718 INFO:teuthology.orchestra.run.smithi090.stdout: "epoch": 25, 2024-04-25T11:47:02.718 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.718 INFO:teuthology.orchestra.run.smithi090.stdout: "active_gid": 15366, 2024-04-25T11:47:02.718 INFO:teuthology.orchestra.run.smithi090.stdout: "active_name": "x", 2024-04-25T11:47:02.718 INFO:teuthology.orchestra.run.smithi090.stdout: "active_addrs": { 2024-04-25T11:47:02.718 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-25T11:47:02.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:02.718 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-25T11:47:02.718 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6800", 2024-04-25T11:47:02.718 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4078377640 2024-04-25T11:47:02.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:02.719 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-25T11:47:02.719 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6801", 2024-04-25T11:47:02.719 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4078377640 2024-04-25T11:47:02.719 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.719 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-25T11:47:02.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.719 INFO:teuthology.orchestra.run.smithi090.stdout: "active_addr": "172.21.15.90:6801/4078377640", 2024-04-25T11:47:02.719 INFO:teuthology.orchestra.run.smithi090.stdout: "active_change": "2024-04-25T11:35:08.201837+0000", 2024-04-25T11:47:02.719 INFO:teuthology.orchestra.run.smithi090.stdout: "active_mgr_features": 4540701547738038271, 2024-04-25T11:47:02.719 INFO:teuthology.orchestra.run.smithi090.stdout: "available": true, 2024-04-25T11:47:02.719 INFO:teuthology.orchestra.run.smithi090.stdout: "standbys": [ 2024-04-25T11:47:02.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:02.719 INFO:teuthology.orchestra.run.smithi090.stdout: "gid": 24946, 2024-04-25T11:47:02.720 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "z", 2024-04-25T11:47:02.720 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_features": 4540701547738038271, 2024-04-25T11:47:02.720 INFO:teuthology.orchestra.run.smithi090.stdout: "available_modules": [ 2024-04-25T11:47:02.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:02.720 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "alerts", 2024-04-25T11:47:02.720 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:02.720 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:02.720 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:02.720 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-25T11:47:02.720 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-25T11:47:02.720 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:02.720 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.720 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.720 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-25T11:47:02.721 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.721 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.721 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.721 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T11:47:02.721 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.721 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.721 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.721 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:02.721 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:02.721 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.721 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.721 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.721 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.722 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.722 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.722 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.722 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.722 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.722 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.722 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.722 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.722 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.722 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.722 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.722 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.722 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.723 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.723 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:02.723 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:02.723 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.723 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.723 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.723 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.723 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.723 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.723 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.723 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.723 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.723 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.723 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.724 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:02.724 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:02.724 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.724 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.724 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.724 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:02.724 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.724 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.724 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.724 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.724 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.724 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.724 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.724 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.724 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.724 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.725 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.725 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.725 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.725 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.725 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:02.725 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:02.725 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.725 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.725 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.725 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.725 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.725 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.725 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.725 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.725 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.726 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.726 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.726 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_destination": { 2024-04-25T11:47:02.726 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_destination", 2024-04-25T11:47:02.726 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.726 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.726 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.726 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.726 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.726 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.726 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.726 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Email address to send alerts to", 2024-04-25T11:47:02.726 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.727 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.727 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.727 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_from_name": { 2024-04-25T11:47:02.727 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_from_name", 2024-04-25T11:47:02.727 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.727 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.727 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.727 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "Ceph", 2024-04-25T11:47:02.727 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.727 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.727 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.727 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Email From: name", 2024-04-25T11:47:02.727 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.728 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.728 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.728 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_host": { 2024-04-25T11:47:02.728 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_host", 2024-04-25T11:47:02.728 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.728 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.728 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.728 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.728 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.728 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.728 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.728 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "SMTP server", 2024-04-25T11:47:02.728 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.728 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.728 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.729 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_password": { 2024-04-25T11:47:02.729 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_password", 2024-04-25T11:47:02.729 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.729 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.729 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.729 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.729 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.729 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.729 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.729 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Password to authenticate with", 2024-04-25T11:47:02.729 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.729 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.729 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.729 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_port": { 2024-04-25T11:47:02.730 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_port", 2024-04-25T11:47:02.730 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.730 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.730 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.730 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "465", 2024-04-25T11:47:02.730 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.730 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.730 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.730 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "SMTP port", 2024-04-25T11:47:02.730 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.730 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.730 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.730 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_sender": { 2024-04-25T11:47:02.731 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_sender", 2024-04-25T11:47:02.731 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.731 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.731 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.731 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.731 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.731 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.731 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.731 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "SMTP envelope sender", 2024-04-25T11:47:02.731 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.731 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.731 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.732 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_ssl": { 2024-04-25T11:47:02.732 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_ssl", 2024-04-25T11:47:02.732 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.732 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.732 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.732 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.732 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.732 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.732 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.732 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T11:47:02.732 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.732 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.732 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.732 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_user": { 2024-04-25T11:47:02.733 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_user", 2024-04-25T11:47:02.733 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.733 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.733 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.733 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.733 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.733 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.733 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.733 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "User to authenticate as", 2024-04-25T11:47:02.733 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.733 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.733 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.733 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:02.733 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:02.734 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.734 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:02.734 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.734 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:02.734 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.734 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.734 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.734 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.734 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.734 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.734 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.734 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.734 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:02.735 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "balancer", 2024-04-25T11:47:02.735 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:02.735 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:02.735 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:02.735 INFO:teuthology.orchestra.run.smithi090.stdout: "active": { 2024-04-25T11:47:02.735 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "active", 2024-04-25T11:47:02.735 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.735 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.735 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.735 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.735 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.735 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.735 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.735 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T11:47:02.736 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.736 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.736 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.736 INFO:teuthology.orchestra.run.smithi090.stdout: "begin_time": { 2024-04-25T11:47:02.736 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "begin_time", 2024-04-25T11:47:02.736 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.736 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.736 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.736 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0000", 2024-04-25T11:47:02.736 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.736 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.736 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.736 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T11:47:02.736 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T11:47:02.737 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.737 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.737 INFO:teuthology.orchestra.run.smithi090.stdout: "begin_weekday": { 2024-04-25T11:47:02.737 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "begin_weekday", 2024-04-25T11:47:02.737 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-25T11:47:02.737 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.737 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.737 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:02.737 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "0", 2024-04-25T11:47:02.737 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "6", 2024-04-25T11:47:02.737 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.737 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T11:47:02.737 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T11:47:02.738 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.738 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.738 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_compat_max_iterations": { 2024-04-25T11:47:02.738 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crush_compat_max_iterations", 2024-04-25T11:47:02.738 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-25T11:47:02.738 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.738 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.738 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "25", 2024-04-25T11:47:02.738 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1", 2024-04-25T11:47:02.738 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "250", 2024-04-25T11:47:02.738 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.738 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T11:47:02.738 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.739 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.739 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.739 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_compat_metrics": { 2024-04-25T11:47:02.739 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crush_compat_metrics", 2024-04-25T11:47:02.739 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.739 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.739 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.739 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T11:47:02.739 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.739 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.739 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.739 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T11:47:02.739 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T11:47:02.739 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.739 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.740 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_compat_step": { 2024-04-25T11:47:02.740 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crush_compat_step", 2024-04-25T11:47:02.740 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-25T11:47:02.740 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.740 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.740 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0.5", 2024-04-25T11:47:02.740 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "0.001", 2024-04-25T11:47:02.740 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "0.999", 2024-04-25T11:47:02.740 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.740 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "aggressiveness of optimization", 2024-04-25T11:47:02.740 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T11:47:02.740 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.740 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.741 INFO:teuthology.orchestra.run.smithi090.stdout: "end_time": { 2024-04-25T11:47:02.741 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "end_time", 2024-04-25T11:47:02.741 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.741 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.741 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.741 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "2359", 2024-04-25T11:47:02.741 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.741 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.741 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.741 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T11:47:02.741 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T11:47:02.741 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.741 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.742 INFO:teuthology.orchestra.run.smithi090.stdout: "end_weekday": { 2024-04-25T11:47:02.742 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "end_weekday", 2024-04-25T11:47:02.742 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-25T11:47:02.742 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.742 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.742 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:02.742 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "0", 2024-04-25T11:47:02.742 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "6", 2024-04-25T11:47:02.742 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.742 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T11:47:02.742 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T11:47:02.742 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.742 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.743 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:02.743 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:02.743 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.743 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.743 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.743 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.743 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.743 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.743 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.743 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.743 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.743 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.743 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.743 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.744 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.744 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.744 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.744 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.744 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.744 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.744 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:02.744 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:02.744 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.744 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.744 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.744 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.744 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.745 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.745 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.745 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.745 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.745 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.745 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.745 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:02.745 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:02.745 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.745 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.745 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.745 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:02.745 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.746 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.746 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.746 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.746 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.746 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.746 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.746 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.746 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.746 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.746 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.746 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.746 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.746 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.747 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:02.747 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:02.747 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.747 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.747 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.747 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.747 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.747 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.747 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.747 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.747 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.747 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.747 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.747 INFO:teuthology.orchestra.run.smithi090.stdout: "min_score": { 2024-04-25T11:47:02.748 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "min_score", 2024-04-25T11:47:02.748 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-25T11:47:02.748 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.748 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.748 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:02.748 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.748 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.748 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.748 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T11:47:02.748 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.748 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.748 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.748 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": { 2024-04-25T11:47:02.749 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mode", 2024-04-25T11:47:02.749 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.749 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.749 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.749 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "upmap", 2024-04-25T11:47:02.749 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.749 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.749 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.749 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-compat", 2024-04-25T11:47:02.749 INFO:teuthology.orchestra.run.smithi090.stdout: "none", 2024-04-25T11:47:02.749 INFO:teuthology.orchestra.run.smithi090.stdout: "read", 2024-04-25T11:47:02.749 INFO:teuthology.orchestra.run.smithi090.stdout: "upmap", 2024-04-25T11:47:02.749 INFO:teuthology.orchestra.run.smithi090.stdout: "upmap-read" 2024-04-25T11:47:02.749 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.749 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Balancer mode", 2024-04-25T11:47:02.750 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.750 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.750 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.750 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_ids": { 2024-04-25T11:47:02.750 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "pool_ids", 2024-04-25T11:47:02.750 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.750 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.750 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.750 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.750 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.750 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.750 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.750 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T11:47:02.751 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.751 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.751 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.751 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-25T11:47:02.751 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-25T11:47:02.751 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:02.751 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.751 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.751 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-25T11:47:02.751 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.751 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.751 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.751 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T11:47:02.751 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.752 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.752 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.752 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:02.752 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:02.752 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.752 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:02.752 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.752 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:02.752 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.752 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.752 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.752 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.752 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.753 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.753 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.753 INFO:teuthology.orchestra.run.smithi090.stdout: "upmap_max_deviation": { 2024-04-25T11:47:02.753 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "upmap_max_deviation", 2024-04-25T11:47:02.753 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.753 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.753 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.753 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5", 2024-04-25T11:47:02.753 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1", 2024-04-25T11:47:02.753 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.753 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.753 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T11:47:02.753 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T11:47:02.753 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.754 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.754 INFO:teuthology.orchestra.run.smithi090.stdout: "upmap_max_optimizations": { 2024-04-25T11:47:02.754 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "upmap_max_optimizations", 2024-04-25T11:47:02.754 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-25T11:47:02.754 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.754 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.754 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-25T11:47:02.754 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.754 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.754 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.754 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T11:47:02.754 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.754 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.755 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.755 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.755 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:02.755 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cephadm", 2024-04-25T11:47:02.755 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:02.755 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:02.755 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:02.755 INFO:teuthology.orchestra.run.smithi090.stdout: "agent_down_multiplier": { 2024-04-25T11:47:02.755 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "agent_down_multiplier", 2024-04-25T11:47:02.755 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-25T11:47:02.755 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.755 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.755 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "3.0", 2024-04-25T11:47:02.756 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.756 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.756 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.756 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T11:47:02.756 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.756 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.756 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.756 INFO:teuthology.orchestra.run.smithi090.stdout: "agent_refresh_rate": { 2024-04-25T11:47:02.756 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "agent_refresh_rate", 2024-04-25T11:47:02.756 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:02.756 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.756 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.756 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "20", 2024-04-25T11:47:02.757 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.757 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.757 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.757 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T11:47:02.757 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.757 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.757 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.757 INFO:teuthology.orchestra.run.smithi090.stdout: "agent_starting_port": { 2024-04-25T11:47:02.757 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "agent_starting_port", 2024-04-25T11:47:02.757 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.757 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.757 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.757 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "4721", 2024-04-25T11:47:02.758 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.758 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.758 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.758 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T11:47:02.758 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.758 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.758 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.758 INFO:teuthology.orchestra.run.smithi090.stdout: "allow_ptrace": { 2024-04-25T11:47:02.758 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "allow_ptrace", 2024-04-25T11:47:02.758 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.758 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.758 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.758 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.759 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.759 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.759 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.759 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T11:47:02.759 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T11:47:02.759 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.759 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.759 INFO:teuthology.orchestra.run.smithi090.stdout: "autotune_interval": { 2024-04-25T11:47:02.759 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "autotune_interval", 2024-04-25T11:47:02.759 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:02.759 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.759 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.759 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-25T11:47:02.760 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.760 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.760 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.760 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T11:47:02.760 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.760 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.760 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.760 INFO:teuthology.orchestra.run.smithi090.stdout: "autotune_memory_target_ratio": { 2024-04-25T11:47:02.760 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T11:47:02.760 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-25T11:47:02.760 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.760 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.760 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0.7", 2024-04-25T11:47:02.760 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.761 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.761 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.761 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T11:47:02.761 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.761 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.761 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.761 INFO:teuthology.orchestra.run.smithi090.stdout: "cephadm_log_destination": { 2024-04-25T11:47:02.761 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cephadm_log_destination", 2024-04-25T11:47:02.761 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.761 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.761 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.761 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.761 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.761 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.762 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.762 INFO:teuthology.orchestra.run.smithi090.stdout: "file", 2024-04-25T11:47:02.762 INFO:teuthology.orchestra.run.smithi090.stdout: "file,syslog", 2024-04-25T11:47:02.762 INFO:teuthology.orchestra.run.smithi090.stdout: "syslog" 2024-04-25T11:47:02.762 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.762 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T11:47:02.762 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.762 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.762 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.762 INFO:teuthology.orchestra.run.smithi090.stdout: "cgroups_split": { 2024-04-25T11:47:02.762 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cgroups_split", 2024-04-25T11:47:02.762 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.762 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.763 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.763 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.763 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.763 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.763 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.763 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T11:47:02.763 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.763 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.763 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.763 INFO:teuthology.orchestra.run.smithi090.stdout: "config_checks_enabled": { 2024-04-25T11:47:02.763 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "config_checks_enabled", 2024-04-25T11:47:02.763 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.763 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.763 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.764 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.764 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.764 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.764 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.764 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T11:47:02.764 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.764 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.764 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.764 INFO:teuthology.orchestra.run.smithi090.stdout: "config_dashboard": { 2024-04-25T11:47:02.764 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "config_dashboard", 2024-04-25T11:47:02.764 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.764 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.764 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.764 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.765 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.765 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.765 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.765 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T11:47:02.765 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.765 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.765 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.765 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_alertmanager": { 2024-04-25T11:47:02.765 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_alertmanager", 2024-04-25T11:47:02.765 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.765 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.765 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.765 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T11:47:02.765 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.765 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.766 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.766 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Prometheus container image", 2024-04-25T11:47:02.766 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.766 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.766 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.766 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_base": { 2024-04-25T11:47:02.766 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_base", 2024-04-25T11:47:02.766 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.766 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.766 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.766 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T11:47:02.766 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.767 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.767 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.767 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Container image name, without the tag", 2024-04-25T11:47:02.767 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.767 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.767 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.767 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_elasticsearch": { 2024-04-25T11:47:02.767 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_elasticsearch", 2024-04-25T11:47:02.767 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.767 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.767 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.767 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T11:47:02.767 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.767 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.768 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.768 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "elasticsearch container image", 2024-04-25T11:47:02.768 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.768 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.768 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.768 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_grafana": { 2024-04-25T11:47:02.768 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_grafana", 2024-04-25T11:47:02.768 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.768 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.768 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.768 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T11:47:02.768 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.769 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.769 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.769 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Prometheus container image", 2024-04-25T11:47:02.769 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.769 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.769 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.769 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_haproxy": { 2024-04-25T11:47:02.769 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_haproxy", 2024-04-25T11:47:02.769 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.769 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.769 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.769 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T11:47:02.769 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.770 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.770 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.770 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "HAproxy container image", 2024-04-25T11:47:02.770 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.770 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.770 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.770 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_jaeger_agent": { 2024-04-25T11:47:02.770 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_jaeger_agent", 2024-04-25T11:47:02.770 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.770 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.770 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.770 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T11:47:02.770 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.771 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.771 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.771 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Jaeger agent container image", 2024-04-25T11:47:02.771 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.771 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.771 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.771 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_jaeger_collector": { 2024-04-25T11:47:02.771 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_jaeger_collector", 2024-04-25T11:47:02.771 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.771 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.771 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.771 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T11:47:02.771 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.771 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.772 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.772 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Jaeger collector container image", 2024-04-25T11:47:02.772 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.772 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.772 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.772 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_jaeger_query": { 2024-04-25T11:47:02.772 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_jaeger_query", 2024-04-25T11:47:02.772 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.772 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.772 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.772 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T11:47:02.772 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.772 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Jaeger query container image", 2024-04-25T11:47:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_keepalived": { 2024-04-25T11:47:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_keepalived", 2024-04-25T11:47:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T11:47:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.774 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Keepalived container image", 2024-04-25T11:47:02.774 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.774 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.774 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.774 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_loki": { 2024-04-25T11:47:02.774 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_loki", 2024-04-25T11:47:02.774 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.774 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.774 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.774 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T11:47:02.774 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.774 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.774 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.775 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Loki container image", 2024-04-25T11:47:02.775 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.775 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.775 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.775 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_node_exporter": { 2024-04-25T11:47:02.775 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_node_exporter", 2024-04-25T11:47:02.775 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.775 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.775 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.775 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T11:47:02.775 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.775 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.775 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.776 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Prometheus container image", 2024-04-25T11:47:02.776 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.776 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.776 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.776 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_nvmeof": { 2024-04-25T11:47:02.776 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_nvmeof", 2024-04-25T11:47:02.776 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.776 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.776 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.776 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T11:47:02.776 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.776 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.776 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.776 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Nvme-of container image", 2024-04-25T11:47:02.777 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.777 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.777 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.777 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_prometheus": { 2024-04-25T11:47:02.777 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_prometheus", 2024-04-25T11:47:02.777 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.777 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.777 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.777 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T11:47:02.777 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.777 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.777 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.777 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Prometheus container image", 2024-04-25T11:47:02.777 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.778 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.778 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.778 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_promtail": { 2024-04-25T11:47:02.778 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_promtail", 2024-04-25T11:47:02.778 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.778 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.778 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.778 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T11:47:02.778 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.778 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.778 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.778 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Promtail container image", 2024-04-25T11:47:02.778 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.779 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.779 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.779 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_samba": { 2024-04-25T11:47:02.779 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_samba", 2024-04-25T11:47:02.779 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.779 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.779 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.779 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T11:47:02.779 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.779 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.779 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.779 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Samba/SMB container image", 2024-04-25T11:47:02.779 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.780 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.780 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.780 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_snmp_gateway": { 2024-04-25T11:47:02.780 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_snmp_gateway", 2024-04-25T11:47:02.780 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.780 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.780 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.780 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T11:47:02.780 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.780 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.780 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.780 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "SNMP Gateway container image", 2024-04-25T11:47:02.780 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.781 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.781 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.781 INFO:teuthology.orchestra.run.smithi090.stdout: "container_init": { 2024-04-25T11:47:02.781 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_init", 2024-04-25T11:47:02.781 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.781 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.781 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.781 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.781 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.781 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.781 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.781 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T11:47:02.781 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.781 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.782 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.782 INFO:teuthology.orchestra.run.smithi090.stdout: "daemon_cache_timeout": { 2024-04-25T11:47:02.782 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "daemon_cache_timeout", 2024-04-25T11:47:02.782 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:02.782 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.782 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.782 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-25T11:47:02.782 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.782 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.782 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.782 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T11:47:02.782 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.783 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.783 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.783 INFO:teuthology.orchestra.run.smithi090.stdout: "default_cephadm_command_timeout": { 2024-04-25T11:47:02.783 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T11:47:02.783 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.783 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.783 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.783 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "900", 2024-04-25T11:47:02.783 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.783 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.783 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.784 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T11:47:02.784 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.784 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.784 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.784 INFO:teuthology.orchestra.run.smithi090.stdout: "default_registry": { 2024-04-25T11:47:02.784 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "default_registry", 2024-04-25T11:47:02.784 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.784 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.784 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.784 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "docker.io", 2024-04-25T11:47:02.784 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.784 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.785 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.785 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T11:47:02.785 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.785 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.785 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.785 INFO:teuthology.orchestra.run.smithi090.stdout: "device_cache_timeout": { 2024-04-25T11:47:02.785 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "device_cache_timeout", 2024-04-25T11:47:02.785 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:02.785 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.785 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.785 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "1800", 2024-04-25T11:47:02.785 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.785 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.785 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.786 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "seconds to cache device inventory", 2024-04-25T11:47:02.786 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.786 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.786 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.786 INFO:teuthology.orchestra.run.smithi090.stdout: "device_enhanced_scan": { 2024-04-25T11:47:02.786 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "device_enhanced_scan", 2024-04-25T11:47:02.786 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.786 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.786 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.786 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.786 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.786 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.786 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.787 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T11:47:02.787 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.787 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.787 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.787 INFO:teuthology.orchestra.run.smithi090.stdout: "facts_cache_timeout": { 2024-04-25T11:47:02.787 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "facts_cache_timeout", 2024-04-25T11:47:02.787 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:02.787 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.787 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.787 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-25T11:47:02.787 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.787 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.787 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.788 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "seconds to cache host facts data", 2024-04-25T11:47:02.788 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.788 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.788 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.788 INFO:teuthology.orchestra.run.smithi090.stdout: "grafana_dashboards_path": { 2024-04-25T11:47:02.788 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "grafana_dashboards_path", 2024-04-25T11:47:02.788 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.788 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.788 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.788 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T11:47:02.788 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.788 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.788 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.789 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T11:47:02.789 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.789 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.789 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.789 INFO:teuthology.orchestra.run.smithi090.stdout: "host_check_interval": { 2024-04-25T11:47:02.789 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "host_check_interval", 2024-04-25T11:47:02.789 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:02.789 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.789 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.789 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-25T11:47:02.789 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.789 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.789 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.790 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to perform a host check", 2024-04-25T11:47:02.790 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.790 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.790 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.790 INFO:teuthology.orchestra.run.smithi090.stdout: "hw_monitoring": { 2024-04-25T11:47:02.790 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "hw_monitoring", 2024-04-25T11:47:02.790 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.790 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.790 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.790 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.790 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.790 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.790 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.791 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T11:47:02.791 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.791 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.791 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.791 INFO:teuthology.orchestra.run.smithi090.stdout: "inventory_list_all": { 2024-04-25T11:47:02.793 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "inventory_list_all", 2024-04-25T11:47:02.793 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.793 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.793 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.793 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.793 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.793 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.793 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.793 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T11:47:02.793 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.794 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.794 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.794 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:02.794 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:02.794 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.794 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.794 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.794 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.794 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.794 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.794 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.794 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.795 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.795 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.795 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.795 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.795 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.795 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.795 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.795 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.795 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.795 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.795 INFO:teuthology.orchestra.run.smithi090.stdout: "log_refresh_metadata": { 2024-04-25T11:47:02.796 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_refresh_metadata", 2024-04-25T11:47:02.796 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.796 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.796 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.796 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.796 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.796 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.796 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.796 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T11:47:02.796 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.796 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.796 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.796 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:02.797 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:02.797 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.797 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.797 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.797 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.797 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.797 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.797 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.797 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T11:47:02.797 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.797 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.797 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.797 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:02.798 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:02.798 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.798 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.798 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.798 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:02.798 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.798 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.798 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.798 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.798 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.798 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.798 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.798 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.798 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.799 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.799 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.799 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.799 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.799 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.799 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:02.799 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:02.799 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.799 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.799 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.799 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.799 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.799 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.800 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.800 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.800 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.800 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.800 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.800 INFO:teuthology.orchestra.run.smithi090.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T11:47:02.800 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T11:47:02.800 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.800 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.800 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.800 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.800 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.800 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.800 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.801 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T11:47:02.801 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.801 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.801 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.801 INFO:teuthology.orchestra.run.smithi090.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T11:47:02.801 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T11:47:02.801 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.801 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.801 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.801 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "*", 2024-04-25T11:47:02.801 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.801 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.801 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.801 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T11:47:02.802 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.802 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.802 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.802 INFO:teuthology.orchestra.run.smithi090.stdout: "max_count_per_host": { 2024-04-25T11:47:02.802 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_count_per_host", 2024-04-25T11:47:02.802 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.802 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.802 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.802 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-25T11:47:02.802 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.802 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.803 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.803 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "max number of daemons per service per host", 2024-04-25T11:47:02.803 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.803 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.803 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.803 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd_draining_count": { 2024-04-25T11:47:02.803 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_osd_draining_count", 2024-04-25T11:47:02.803 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.803 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.803 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.803 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-25T11:47:02.803 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.803 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.804 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.804 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T11:47:02.804 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.804 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.804 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.804 INFO:teuthology.orchestra.run.smithi090.stdout: "migration_current": { 2024-04-25T11:47:02.804 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "migration_current", 2024-04-25T11:47:02.804 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.804 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.804 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.804 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.804 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.804 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.805 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.805 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "internal - do not modify", 2024-04-25T11:47:02.805 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.805 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.805 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.805 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": { 2024-04-25T11:47:02.805 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mode", 2024-04-25T11:47:02.805 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.805 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.805 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.805 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "root", 2024-04-25T11:47:02.805 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.805 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.805 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.806 INFO:teuthology.orchestra.run.smithi090.stdout: "cephadm-package", 2024-04-25T11:47:02.806 INFO:teuthology.orchestra.run.smithi090.stdout: "root" 2024-04-25T11:47:02.806 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.806 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T11:47:02.806 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.806 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.806 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.806 INFO:teuthology.orchestra.run.smithi090.stdout: "oob_default_addr": { 2024-04-25T11:47:02.806 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "oob_default_addr", 2024-04-25T11:47:02.806 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.806 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.806 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.807 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "169.254.1.1", 2024-04-25T11:47:02.807 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.807 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.807 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.807 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T11:47:02.807 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.807 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.807 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.807 INFO:teuthology.orchestra.run.smithi090.stdout: "prometheus_alerts_path": { 2024-04-25T11:47:02.807 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "prometheus_alerts_path", 2024-04-25T11:47:02.807 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.807 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.807 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.808 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T11:47:02.808 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.808 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.808 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.808 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T11:47:02.808 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.808 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.808 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.808 INFO:teuthology.orchestra.run.smithi090.stdout: "registry_insecure": { 2024-04-25T11:47:02.808 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "registry_insecure", 2024-04-25T11:47:02.808 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.808 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.808 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.809 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.809 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.809 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.809 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.809 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T11:47:02.809 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.809 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.809 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.809 INFO:teuthology.orchestra.run.smithi090.stdout: "registry_password": { 2024-04-25T11:47:02.809 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "registry_password", 2024-04-25T11:47:02.809 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.809 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.809 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.809 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.810 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.810 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.810 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.810 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T11:47:02.810 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.810 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.810 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.810 INFO:teuthology.orchestra.run.smithi090.stdout: "registry_url": { 2024-04-25T11:47:02.810 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "registry_url", 2024-04-25T11:47:02.810 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.810 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.810 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.811 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.811 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.811 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.811 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.811 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T11:47:02.811 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.811 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.811 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.811 INFO:teuthology.orchestra.run.smithi090.stdout: "registry_username": { 2024-04-25T11:47:02.811 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "registry_username", 2024-04-25T11:47:02.811 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.811 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.811 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.811 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.812 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.812 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.812 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.812 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T11:47:02.812 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.812 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.812 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.812 INFO:teuthology.orchestra.run.smithi090.stdout: "secure_monitoring_stack": { 2024-04-25T11:47:02.812 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "secure_monitoring_stack", 2024-04-25T11:47:02.812 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.812 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.812 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.812 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.812 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.813 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.813 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.813 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T11:47:02.813 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.813 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.813 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.813 INFO:teuthology.orchestra.run.smithi090.stdout: "service_discovery_port": { 2024-04-25T11:47:02.813 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "service_discovery_port", 2024-04-25T11:47:02.813 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.813 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.813 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.813 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8765", 2024-04-25T11:47:02.813 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.813 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.814 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.814 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "cephadm service discovery port", 2024-04-25T11:47:02.814 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.814 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.814 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.814 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:02.814 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:02.814 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.814 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:02.814 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.814 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:02.814 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.814 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.815 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.815 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.815 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.815 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.815 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.815 INFO:teuthology.orchestra.run.smithi090.stdout: "ssh_config_file": { 2024-04-25T11:47:02.815 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ssh_config_file", 2024-04-25T11:47:02.815 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.815 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.815 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.815 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.815 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.815 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.815 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.816 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T11:47:02.816 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.816 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.816 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.816 INFO:teuthology.orchestra.run.smithi090.stdout: "use_agent": { 2024-04-25T11:47:02.816 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "use_agent", 2024-04-25T11:47:02.816 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.816 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.816 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.816 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.816 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.816 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.816 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.816 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T11:47:02.817 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.817 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.817 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.817 INFO:teuthology.orchestra.run.smithi090.stdout: "use_repo_digest": { 2024-04-25T11:47:02.817 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "use_repo_digest", 2024-04-25T11:47:02.817 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.817 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.817 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.817 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.817 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.817 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.817 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.817 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T11:47:02.818 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.818 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.818 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.818 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_on_failed_host_check": { 2024-04-25T11:47:02.818 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_on_failed_host_check", 2024-04-25T11:47:02.818 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.818 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.818 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.818 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.818 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.818 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.818 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.819 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T11:47:02.819 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.819 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.819 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.819 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_on_stray_daemons": { 2024-04-25T11:47:02.819 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_on_stray_daemons", 2024-04-25T11:47:02.819 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.819 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.819 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.819 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.819 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.819 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.820 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.820 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T11:47:02.820 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.820 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.820 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.820 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_on_stray_hosts": { 2024-04-25T11:47:02.820 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_on_stray_hosts", 2024-04-25T11:47:02.820 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.820 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.820 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.820 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.820 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.820 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.820 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T11:47:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crash", 2024-04-25T11:47:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.825 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.825 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.825 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.825 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.825 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.825 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.825 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:02.825 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:02.825 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.825 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.825 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.825 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.825 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.825 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.825 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.826 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.826 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.826 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.826 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.826 INFO:teuthology.orchestra.run.smithi090.stdout: "retain_interval": { 2024-04-25T11:47:02.826 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "retain_interval", 2024-04-25T11:47:02.826 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:02.826 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.826 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.826 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "31536000", 2024-04-25T11:47:02.826 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.826 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.826 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.826 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T11:47:02.827 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.827 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.827 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.827 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:02.827 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:02.827 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.827 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:02.827 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.827 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:02.827 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.827 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.827 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.827 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.827 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.828 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.828 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.828 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_recent_interval": { 2024-04-25T11:47:02.828 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_recent_interval", 2024-04-25T11:47:02.828 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:02.828 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.828 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.828 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "1209600", 2024-04-25T11:47:02.828 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.828 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.828 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.828 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T11:47:02.828 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.828 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "dashboard", 2024-04-25T11:47:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T11:47:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T11:47:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-25T11:47:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T11:47:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T11:47:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.831 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.831 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.831 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.831 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.831 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.831 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.831 INFO:teuthology.orchestra.run.smithi090.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T11:47:02.831 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T11:47:02.831 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.831 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.831 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.831 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.832 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.832 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.832 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.832 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.832 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.832 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.832 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.832 INFO:teuthology.orchestra.run.smithi090.stdout: "AUDIT_API_ENABLED": { 2024-04-25T11:47:02.832 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T11:47:02.832 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.832 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.832 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.832 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.832 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.833 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.833 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.833 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.833 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.833 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.833 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.833 INFO:teuthology.orchestra.run.smithi090.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T11:47:02.833 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T11:47:02.833 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.833 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.833 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.833 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.833 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.833 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.833 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.834 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.834 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.834 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.834 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.834 INFO:teuthology.orchestra.run.smithi090.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T11:47:02.834 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T11:47:02.834 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.834 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.834 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.834 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.834 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.834 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.834 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.835 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.835 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.835 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.835 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.835 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T11:47:02.835 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T11:47:02.835 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.835 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.835 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.835 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.835 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.835 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.835 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.835 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.836 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.836 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.836 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.836 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T11:47:02.836 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T11:47:02.836 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.836 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.836 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.836 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.836 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.836 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.836 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.836 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.836 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.836 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.837 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.837 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T11:47:02.837 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T11:47:02.837 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.837 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.837 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.837 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.837 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.837 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.837 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.837 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.837 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.837 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.837 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.838 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T11:47:02.838 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T11:47:02.838 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.838 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.838 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.838 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.838 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.838 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.838 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.838 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.838 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.838 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.839 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.839 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T11:47:02.839 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T11:47:02.839 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.839 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.839 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.839 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.839 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.839 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.839 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.839 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.839 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.839 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.840 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.840 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T11:47:02.840 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T11:47:02.840 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.840 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.840 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.840 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.840 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.840 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.840 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.840 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.840 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.840 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.841 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.841 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T11:47:02.841 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T11:47:02.841 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.841 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.841 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.841 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.841 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.841 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.841 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.841 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.841 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.841 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T11:47:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T11:47:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T11:47:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T11:47:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "admin", 2024-04-25T11:47:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.844 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.844 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T11:47:02.844 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T11:47:02.844 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.844 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.844 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.844 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.844 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.844 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.844 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.844 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.844 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.844 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.845 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.845 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_API_URL": { 2024-04-25T11:47:02.845 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_API_URL", 2024-04-25T11:47:02.845 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.845 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.845 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.845 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.845 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.845 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.845 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.845 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.845 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.846 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.846 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.846 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T11:47:02.846 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T11:47:02.846 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.846 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.846 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.846 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "admin", 2024-04-25T11:47:02.846 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.846 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.846 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.846 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.846 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.847 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.847 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.847 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T11:47:02.847 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T11:47:02.847 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.847 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.847 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.847 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.847 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.847 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.847 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.847 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.847 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.848 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.848 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.848 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T11:47:02.848 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T11:47:02.848 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.848 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.848 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.848 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.848 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.848 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.848 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.848 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.848 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.849 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.849 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.849 INFO:teuthology.orchestra.run.smithi090.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T11:47:02.849 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T11:47:02.849 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.849 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.849 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.849 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.849 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.849 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.849 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.849 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.849 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.850 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.850 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.850 INFO:teuthology.orchestra.run.smithi090.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T11:47:02.850 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T11:47:02.850 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.850 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.850 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.850 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.850 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.850 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.850 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.850 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.851 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.851 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.851 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.851 INFO:teuthology.orchestra.run.smithi090.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-25T11:47:02.851 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-25T11:47:02.851 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.851 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.851 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.851 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "[]", 2024-04-25T11:47:02.851 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.851 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.851 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.851 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.852 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.852 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.852 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.852 INFO:teuthology.orchestra.run.smithi090.stdout: "MULTICLUSTER_CONFIG": { 2024-04-25T11:47:02.852 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-25T11:47:02.852 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.852 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.852 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.852 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "{}", 2024-04-25T11:47:02.852 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.852 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.852 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.852 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.852 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.853 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.853 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.853 INFO:teuthology.orchestra.run.smithi090.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T11:47:02.853 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T11:47:02.853 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.853 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.853 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.853 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.853 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.853 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.853 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.853 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.854 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.854 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.854 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.854 INFO:teuthology.orchestra.run.smithi090.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T11:47:02.854 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T11:47:02.854 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.854 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.854 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.854 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.854 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.854 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.854 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.854 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.855 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.855 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.855 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.855 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T11:47:02.855 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T11:47:02.855 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.855 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.855 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.855 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.855 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.855 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.855 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.855 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.856 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.856 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.856 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.856 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T11:47:02.856 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T11:47:02.856 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.856 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.856 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.856 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.856 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.856 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.856 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.856 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.856 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.857 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.857 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.857 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T11:47:02.857 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T11:47:02.857 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.857 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.857 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.857 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.857 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.857 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.857 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.857 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.857 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.857 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.858 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.858 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T11:47:02.858 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T11:47:02.858 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.858 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.858 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.858 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.858 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.858 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.858 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.858 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.858 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.858 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.859 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.859 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T11:47:02.859 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T11:47:02.859 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.859 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.859 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.859 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.859 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.859 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.859 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.859 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.859 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.859 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.860 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.860 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T11:47:02.860 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T11:47:02.860 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.860 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.860 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.860 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.860 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.860 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.860 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.860 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.860 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.860 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.860 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.861 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T11:47:02.861 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T11:47:02.861 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.861 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.861 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.861 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.861 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.861 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.861 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.861 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.861 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.861 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.861 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.861 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T11:47:02.861 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T11:47:02.862 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.862 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.862 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.862 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.862 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.862 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.862 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.862 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.862 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.862 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.862 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.862 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T11:47:02.862 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T11:47:02.862 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.863 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.863 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.863 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T11:47:02.863 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.863 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.863 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.863 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.863 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.863 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.863 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.863 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T11:47:02.863 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T11:47:02.863 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.863 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.864 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.864 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-25T11:47:02.864 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.864 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.864 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.864 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.864 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.864 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.864 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.864 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T11:47:02.864 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T11:47:02.864 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.864 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.864 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.865 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8", 2024-04-25T11:47:02.865 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.865 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.865 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.865 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.865 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.865 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.865 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.865 INFO:teuthology.orchestra.run.smithi090.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T11:47:02.865 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T11:47:02.865 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.865 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.865 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.865 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "45", 2024-04-25T11:47:02.866 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.866 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.866 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.866 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.866 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.866 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.866 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.866 INFO:teuthology.orchestra.run.smithi090.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T11:47:02.866 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T11:47:02.866 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.866 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.866 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.866 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.866 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.867 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.867 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.867 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.867 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.867 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.867 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.867 INFO:teuthology.orchestra.run.smithi090.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T11:47:02.867 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T11:47:02.867 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.867 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.867 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.867 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "admin", 2024-04-25T11:47:02.867 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.867 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.868 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.868 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.868 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.868 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.868 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.868 INFO:teuthology.orchestra.run.smithi090.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T11:47:02.868 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T11:47:02.868 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.868 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.868 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.868 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.868 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.868 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.868 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.869 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.869 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.869 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.869 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.869 INFO:teuthology.orchestra.run.smithi090.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T11:47:02.869 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T11:47:02.869 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.869 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.869 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.869 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.869 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.869 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.869 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.870 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.870 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.870 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.870 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.870 INFO:teuthology.orchestra.run.smithi090.stdout: "UNSAFE_TLS_v1_2": { 2024-04-25T11:47:02.870 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-25T11:47:02.870 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.870 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.870 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.870 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.870 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.870 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.870 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.870 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.871 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.871 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.871 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.871 INFO:teuthology.orchestra.run.smithi090.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T11:47:02.871 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T11:47:02.871 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.871 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.871 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.871 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:02.871 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.871 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.871 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.871 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.871 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.871 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.872 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.872 INFO:teuthology.orchestra.run.smithi090.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T11:47:02.872 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T11:47:02.872 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.872 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.872 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.872 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-25T11:47:02.872 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.872 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.872 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.872 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.872 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.872 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.872 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.873 INFO:teuthology.orchestra.run.smithi090.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T11:47:02.873 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T11:47:02.873 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.873 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.873 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.873 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5", 2024-04-25T11:47:02.873 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.873 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.873 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.873 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.873 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.873 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.873 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.874 INFO:teuthology.orchestra.run.smithi090.stdout: "cross_origin_url": { 2024-04-25T11:47:02.874 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cross_origin_url", 2024-04-25T11:47:02.874 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.874 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.874 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.874 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.874 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.874 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.874 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.874 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.874 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.874 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.874 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.875 INFO:teuthology.orchestra.run.smithi090.stdout: "crt_file": { 2024-04-25T11:47:02.875 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crt_file", 2024-04-25T11:47:02.875 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.875 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.875 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.875 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.875 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.875 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.875 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.875 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.875 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.875 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.875 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.875 INFO:teuthology.orchestra.run.smithi090.stdout: "debug": { 2024-04-25T11:47:02.876 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "debug", 2024-04-25T11:47:02.876 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.876 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.876 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.876 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.876 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.876 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.876 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.876 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Enable/disable debug options", 2024-04-25T11:47:02.876 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.876 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.876 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.876 INFO:teuthology.orchestra.run.smithi090.stdout: "jwt_token_ttl": { 2024-04-25T11:47:02.877 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "jwt_token_ttl", 2024-04-25T11:47:02.877 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.877 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.877 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.877 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "28800", 2024-04-25T11:47:02.877 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.877 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.877 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.877 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.877 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.877 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.877 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.877 INFO:teuthology.orchestra.run.smithi090.stdout: "key_file": { 2024-04-25T11:47:02.877 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "key_file", 2024-04-25T11:47:02.877 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.878 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.878 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.878 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.878 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.878 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.878 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.878 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.878 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.878 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.878 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.878 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:02.878 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:02.878 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.878 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.879 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.879 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.879 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.879 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.879 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.879 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.879 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.879 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.879 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.879 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.879 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.879 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.879 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.879 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.879 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.880 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.880 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:02.880 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:02.880 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.880 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.880 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.880 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.880 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.880 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.880 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.880 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.880 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.880 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.881 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.881 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:02.881 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:02.881 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.881 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.881 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.881 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:02.881 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.881 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.881 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.881 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.881 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.881 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.881 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.882 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.882 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.882 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.882 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.882 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.882 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.882 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.882 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:02.882 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:02.882 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.882 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.882 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.882 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.883 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.883 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.883 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.883 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.883 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.883 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.883 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.883 INFO:teuthology.orchestra.run.smithi090.stdout: "motd": { 2024-04-25T11:47:02.883 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "motd", 2024-04-25T11:47:02.883 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.883 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.883 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.883 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.884 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.884 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.884 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.884 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "The message of the day", 2024-04-25T11:47:02.884 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.884 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.884 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.884 INFO:teuthology.orchestra.run.smithi090.stdout: "redirect_resolve_ip_addr": { 2024-04-25T11:47:02.884 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T11:47:02.884 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.884 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.885 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.885 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.885 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.885 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.885 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.885 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.885 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.885 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.885 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.885 INFO:teuthology.orchestra.run.smithi090.stdout: "server_addr": { 2024-04-25T11:47:02.885 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_addr", 2024-04-25T11:47:02.885 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.885 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.886 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.886 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "::", 2024-04-25T11:47:02.886 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.886 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.886 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.886 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.886 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.886 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.886 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.886 INFO:teuthology.orchestra.run.smithi090.stdout: "server_port": { 2024-04-25T11:47:02.886 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_port", 2024-04-25T11:47:02.886 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.887 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.887 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.887 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8080", 2024-04-25T11:47:02.887 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.887 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.887 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.887 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.887 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.887 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.887 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.887 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:02.887 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:02.887 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.887 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:02.888 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.888 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:02.888 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.888 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.888 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.888 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.888 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.888 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.888 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.888 INFO:teuthology.orchestra.run.smithi090.stdout: "ssl": { 2024-04-25T11:47:02.888 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ssl", 2024-04-25T11:47:02.888 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.888 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.888 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.889 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.889 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.889 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.889 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.889 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.889 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.889 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.889 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.889 INFO:teuthology.orchestra.run.smithi090.stdout: "ssl_server_port": { 2024-04-25T11:47:02.889 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ssl_server_port", 2024-04-25T11:47:02.889 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.889 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.889 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.889 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8443", 2024-04-25T11:47:02.889 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.890 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.890 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.890 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.890 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.890 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.890 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.890 INFO:teuthology.orchestra.run.smithi090.stdout: "standby_behaviour": { 2024-04-25T11:47:02.890 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "standby_behaviour", 2024-04-25T11:47:02.890 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.890 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.890 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.890 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "redirect", 2024-04-25T11:47:02.891 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.891 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.891 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.891 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.891 INFO:teuthology.orchestra.run.smithi090.stdout: "redirect" 2024-04-25T11:47:02.891 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.891 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.891 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.891 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.891 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.891 INFO:teuthology.orchestra.run.smithi090.stdout: "standby_error_status_code": { 2024-04-25T11:47:02.891 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "standby_error_status_code", 2024-04-25T11:47:02.891 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.891 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.892 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.892 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "500", 2024-04-25T11:47:02.892 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "400", 2024-04-25T11:47:02.892 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "599", 2024-04-25T11:47:02.892 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.892 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.892 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.892 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.892 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.892 INFO:teuthology.orchestra.run.smithi090.stdout: "url_prefix": { 2024-04-25T11:47:02.892 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "url_prefix", 2024-04-25T11:47:02.892 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.892 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.893 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.893 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.893 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.893 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.893 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.893 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.893 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.893 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.893 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.893 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.893 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:02.893 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "devicehealth", 2024-04-25T11:47:02.894 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:02.894 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:02.894 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:02.894 INFO:teuthology.orchestra.run.smithi090.stdout: "enable_monitoring": { 2024-04-25T11:47:02.894 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "enable_monitoring", 2024-04-25T11:47:02.894 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.894 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.894 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.894 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.894 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.894 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.894 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.894 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "monitor device health metrics", 2024-04-25T11:47:02.894 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.895 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.895 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.895 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:02.895 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:02.895 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.895 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.895 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.895 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.895 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.895 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.895 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.895 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.895 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.896 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.896 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.896 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.896 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.896 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.896 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.896 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.896 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.896 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.896 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:02.896 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:02.896 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.896 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.897 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.897 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.897 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.897 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.897 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.897 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.897 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.897 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.897 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.897 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:02.897 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:02.897 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.897 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.898 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.898 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:02.898 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.898 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.898 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.898 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.898 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.898 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.898 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.898 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.898 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.898 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.898 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.898 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.899 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.899 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.899 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:02.899 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:02.899 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.899 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.899 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.899 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.899 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.899 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.899 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.899 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.899 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.900 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.900 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.900 INFO:teuthology.orchestra.run.smithi090.stdout: "mark_out_threshold": { 2024-04-25T11:47:02.900 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mark_out_threshold", 2024-04-25T11:47:02.900 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:02.900 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.900 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.900 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "2419200", 2024-04-25T11:47:02.900 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.900 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.900 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.900 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T11:47:02.900 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.900 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.901 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.901 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": { 2024-04-25T11:47:02.901 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "pool_name", 2024-04-25T11:47:02.901 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.901 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.901 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.901 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "device_health_metrics", 2024-04-25T11:47:02.901 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.901 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.901 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.901 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T11:47:02.901 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.901 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.902 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.902 INFO:teuthology.orchestra.run.smithi090.stdout: "retention_period": { 2024-04-25T11:47:02.902 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "retention_period", 2024-04-25T11:47:02.902 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:02.902 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.902 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.902 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "15552000", 2024-04-25T11:47:02.902 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.902 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.902 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.902 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how long to retain device health metrics", 2024-04-25T11:47:02.902 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.902 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.903 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.903 INFO:teuthology.orchestra.run.smithi090.stdout: "scrape_frequency": { 2024-04-25T11:47:02.903 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "scrape_frequency", 2024-04-25T11:47:02.903 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:02.903 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.903 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.903 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "86400", 2024-04-25T11:47:02.903 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.903 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.903 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.903 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T11:47:02.903 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.903 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.903 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.904 INFO:teuthology.orchestra.run.smithi090.stdout: "self_heal": { 2024-04-25T11:47:02.904 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "self_heal", 2024-04-25T11:47:02.904 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.904 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.904 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.904 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:02.904 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.904 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.904 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.904 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T11:47:02.904 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.904 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.904 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.904 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-25T11:47:02.905 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-25T11:47:02.905 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:02.905 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.905 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.905 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-25T11:47:02.905 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.905 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.905 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.905 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T11:47:02.905 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.905 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.905 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.905 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:02.905 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:02.905 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.906 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:02.906 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.906 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:02.906 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.906 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.906 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.906 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.906 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.906 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.906 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.906 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_threshold": { 2024-04-25T11:47:02.906 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_threshold", 2024-04-25T11:47:02.906 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:02.907 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.907 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.907 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "7257600", 2024-04-25T11:47:02.907 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.907 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.907 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.907 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T11:47:02.907 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.907 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.907 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.907 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.907 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:02.907 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "diskprediction_local", 2024-04-25T11:47:02.907 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:02.907 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:02.908 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:02.908 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:02.908 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:02.908 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.908 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.908 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.908 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.908 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.908 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.908 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.908 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.908 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.908 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.908 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.908 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.909 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.909 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.909 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.909 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.909 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.909 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.909 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:02.909 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:02.909 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.909 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.909 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.909 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.909 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.909 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.910 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.910 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.910 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.910 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.910 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.910 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:02.910 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:02.910 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.910 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.910 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.910 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:02.910 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.910 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.912 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.912 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.912 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.912 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.912 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.912 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.912 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.912 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.912 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.912 INFO:teuthology.orchestra.run.smithi090.stdout: "predict_interval": { 2024-04-25T11:47:02.912 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "predict_interval", 2024-04-25T11:47:02.912 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.912 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.912 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.913 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "86400", 2024-04-25T11:47:02.913 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.913 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.913 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.913 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.913 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.913 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.913 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.913 INFO:teuthology.orchestra.run.smithi090.stdout: "predictor_model": { 2024-04-25T11:47:02.913 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "predictor_model", 2024-04-25T11:47:02.913 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.913 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.913 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.913 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "prophetstor", 2024-04-25T11:47:02.913 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.914 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.914 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.914 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.914 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.914 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.914 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.914 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-25T11:47:02.914 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-25T11:47:02.914 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.914 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.914 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.914 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-25T11:47:02.914 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "influx", 2024-04-25T11:47:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": false, 2024-04-25T11:47:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "influxdb python module not found", 2024-04-25T11:47:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: "batch_size": { 2024-04-25T11:47:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "batch_size", 2024-04-25T11:47:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5000", 2024-04-25T11:47:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T11:47:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: "database": { 2024-04-25T11:47:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "database", 2024-04-25T11:47:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.918 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.918 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "ceph", 2024-04-25T11:47:02.918 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.918 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.918 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.918 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T11:47:02.918 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.918 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.918 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.918 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": { 2024-04-25T11:47:02.918 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "hostname", 2024-04-25T11:47:02.918 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.918 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "InfluxDB server hostname", 2024-04-25T11:47:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-25T11:47:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-25T11:47:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.920 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.920 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "30", 2024-04-25T11:47:02.920 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "5", 2024-04-25T11:47:02.920 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.920 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.920 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T11:47:02.920 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.920 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.920 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.920 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:02.920 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:02.920 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.920 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.921 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.921 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.921 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.921 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.921 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.921 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.921 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.921 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.921 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.921 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.921 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.921 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.921 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.921 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.921 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.922 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.922 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:02.922 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:02.922 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.922 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.922 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.922 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.922 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.922 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.922 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.922 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.922 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.922 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.922 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.923 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:02.923 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:02.923 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.923 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.923 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.923 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:02.923 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.923 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.923 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.923 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.923 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.923 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.923 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.923 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.924 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.924 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.924 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.924 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.924 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.924 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.924 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:02.924 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:02.924 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.924 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.924 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.924 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.924 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.925 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.925 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.925 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.925 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.925 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.925 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.925 INFO:teuthology.orchestra.run.smithi090.stdout: "password": { 2024-04-25T11:47:02.925 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "password", 2024-04-25T11:47:02.925 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.925 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.925 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.925 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.925 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.925 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.925 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.925 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "password of InfluxDB server user", 2024-04-25T11:47:02.926 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.926 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.926 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.926 INFO:teuthology.orchestra.run.smithi090.stdout: "port": { 2024-04-25T11:47:02.926 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "port", 2024-04-25T11:47:02.926 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.926 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.926 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.926 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8086", 2024-04-25T11:47:02.926 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.926 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.926 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.926 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "InfluxDB server port", 2024-04-25T11:47:02.927 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.927 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.927 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.927 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:02.927 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:02.927 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.927 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:02.927 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.927 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:02.927 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.927 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.927 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.927 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.927 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.928 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.928 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.928 INFO:teuthology.orchestra.run.smithi090.stdout: "ssl": { 2024-04-25T11:47:02.928 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ssl", 2024-04-25T11:47:02.928 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.928 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.928 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.928 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "false", 2024-04-25T11:47:02.928 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.928 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.928 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.928 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T11:47:02.928 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.928 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.928 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.929 INFO:teuthology.orchestra.run.smithi090.stdout: "threads": { 2024-04-25T11:47:02.929 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "threads", 2024-04-25T11:47:02.929 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.929 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.929 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.929 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5", 2024-04-25T11:47:02.929 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1", 2024-04-25T11:47:02.929 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "32", 2024-04-25T11:47:02.929 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.929 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T11:47:02.929 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.929 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.929 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.929 INFO:teuthology.orchestra.run.smithi090.stdout: "username": { 2024-04-25T11:47:02.930 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "username", 2024-04-25T11:47:02.930 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.930 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.930 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.930 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.930 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.930 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.930 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.930 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "username of InfluxDB server user", 2024-04-25T11:47:02.930 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.930 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.930 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.930 INFO:teuthology.orchestra.run.smithi090.stdout: "verify_ssl": { 2024-04-25T11:47:02.930 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "verify_ssl", 2024-04-25T11:47:02.930 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.931 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.931 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.931 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "true", 2024-04-25T11:47:02.931 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.931 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.931 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.931 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T11:47:02.931 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.931 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.931 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.931 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.931 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:02.932 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "insights", 2024-04-25T11:47:02.932 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:02.932 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:02.932 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:02.932 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:02.932 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:02.932 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.932 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.932 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.932 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.932 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.932 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.932 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.932 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.932 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.933 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.933 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.933 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.933 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.933 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.933 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.933 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.933 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.933 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.933 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:02.933 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:02.933 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.933 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.933 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.934 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.934 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.934 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.934 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.934 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.934 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.934 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.934 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.934 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:02.934 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:02.934 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.934 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.934 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.935 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:02.935 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.935 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.935 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.935 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.935 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.935 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.935 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.935 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.935 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.935 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.935 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.935 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.935 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.935 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.936 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:02.936 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:02.936 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.936 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.936 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.936 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.936 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.936 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.936 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.936 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.936 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.936 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.936 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.936 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:02.938 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "iostat", 2024-04-25T11:47:02.938 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:02.938 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:02.938 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:02.938 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:02.938 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:02.938 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.938 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.938 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.938 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.938 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.938 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.938 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.938 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:02.941 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.941 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.941 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.941 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.941 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.941 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.941 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.941 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.941 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.941 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.941 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.941 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.941 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.941 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.942 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:02.942 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:02.942 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.942 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.942 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.942 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.942 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.942 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.942 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.942 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.942 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.942 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.942 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:02.944 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "k8sevents", 2024-04-25T11:47:02.944 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:02.944 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:02.944 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:02.944 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_event_retention_days": { 2024-04-25T11:47:02.944 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ceph_event_retention_days", 2024-04-25T11:47:02.944 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.944 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.944 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.944 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "7", 2024-04-25T11:47:02.944 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.944 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.944 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.944 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T11:47:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: "config_check_secs": { 2024-04-25T11:47:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "config_check_secs", 2024-04-25T11:47:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-25T11:47:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "10", 2024-04-25T11:47:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T11:47:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:02.949 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.949 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.949 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.949 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.949 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.949 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.949 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.949 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.949 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.949 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.949 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.949 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.949 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.949 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.950 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:02.950 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:02.950 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.950 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.950 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.950 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.950 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.950 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.950 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.950 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.950 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.950 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.950 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.951 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:02.951 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:02.951 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.951 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:02.951 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.951 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:02.951 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.951 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.951 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.951 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.951 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.951 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.951 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.951 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "localpool", 2024-04-25T11:47:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: "failure_domain": { 2024-04-25T11:47:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "failure_domain", 2024-04-25T11:47:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "host", 2024-04-25T11:47:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.953 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.953 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "failure domain for any created local pool", 2024-04-25T11:47:02.953 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T11:47:02.953 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.953 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.953 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:02.953 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:02.953 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.953 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.953 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.953 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.953 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.953 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.953 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.954 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.954 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.954 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.954 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.954 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.954 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.954 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.954 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.954 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.954 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.954 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.954 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:02.954 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:02.955 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.955 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.955 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.955 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.955 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.955 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.955 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.955 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.955 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.955 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.955 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.955 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:02.955 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:02.956 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.956 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.956 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.956 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:02.956 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.956 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.956 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.956 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.956 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.956 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.956 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.956 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.956 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.956 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.956 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.957 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.957 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.957 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.957 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:02.957 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:02.957 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.957 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.957 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.957 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.957 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.957 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.957 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.957 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.957 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.958 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.958 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.958 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": { 2024-04-25T11:47:02.958 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "min_size", 2024-04-25T11:47:02.958 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.958 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.958 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.958 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.958 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.958 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.958 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.958 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "default min_size for any created local pool", 2024-04-25T11:47:02.958 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T11:47:02.958 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.959 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.959 INFO:teuthology.orchestra.run.smithi090.stdout: "num_rep": { 2024-04-25T11:47:02.959 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "num_rep", 2024-04-25T11:47:02.959 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.959 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.959 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.959 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "3", 2024-04-25T11:47:02.959 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.959 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.959 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.959 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "default replica count for any created local pool", 2024-04-25T11:47:02.959 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.959 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.960 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.960 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": { 2024-04-25T11:47:02.960 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "pg_num", 2024-04-25T11:47:02.960 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.960 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.960 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.960 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "128", 2024-04-25T11:47:02.960 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.960 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.960 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.960 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T11:47:02.960 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.960 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.960 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.961 INFO:teuthology.orchestra.run.smithi090.stdout: "prefix": { 2024-04-25T11:47:02.961 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "prefix", 2024-04-25T11:47:02.961 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.961 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.961 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.961 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.961 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.961 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.961 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.961 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "name prefix for any created local pool", 2024-04-25T11:47:02.961 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.961 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.961 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.962 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:02.962 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:02.962 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.962 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:02.962 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.962 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:02.962 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.962 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.962 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.962 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.962 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.962 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.962 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.962 INFO:teuthology.orchestra.run.smithi090.stdout: "subtree": { 2024-04-25T11:47:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "subtree", 2024-04-25T11:47:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "rack", 2024-04-25T11:47:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T11:47:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T11:47:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mds_autoscaler", 2024-04-25T11:47:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mirroring", 2024-04-25T11:47:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "nfs", 2024-04-25T11:47:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.977 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.977 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.977 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.977 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.977 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.977 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.977 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.977 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.977 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.977 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.977 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.977 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:02.977 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.980 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.980 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.980 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.980 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.980 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:02.980 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:02.980 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.980 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.980 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.980 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.980 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.980 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.980 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "orchestrator", 2024-04-25T11:47:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: "fail_fs": { 2024-04-25T11:47:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "fail_fs", 2024-04-25T11:47:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T11:47:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:02.984 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.984 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.984 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.984 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.984 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.984 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.984 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.984 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.984 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.984 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.984 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.984 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.984 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.984 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.987 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.987 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.987 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.987 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.987 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.987 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.987 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.987 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.987 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.987 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.987 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.987 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.987 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.988 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:02.988 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:02.988 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.988 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.988 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.988 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.988 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.988 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.988 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.988 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.988 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.988 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.988 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: "orchestrator": { 2024-04-25T11:47:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "orchestrator", 2024-04-25T11:47:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: "cephadm", 2024-04-25T11:47:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: "rook", 2024-04-25T11:47:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: "test_orchestrator" 2024-04-25T11:47:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Orchestrator backend", 2024-04-25T11:47:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.991 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.991 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.991 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.991 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.991 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:02.991 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "osd_perf_query", 2024-04-25T11:47:02.991 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:02.991 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:02.991 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:02.991 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:02.991 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:02.991 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "osd_support", 2024-04-25T11:47:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "pg_autoscaler", 2024-04-25T11:47:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.007 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.007 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.007 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-25T11:47:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-25T11:47:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-25T11:47:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "threshold": { 2024-04-25T11:47:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "threshold", 2024-04-25T11:47:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-25T11:47:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "3.0", 2024-04-25T11:47:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1.0", 2024-04-25T11:47:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "scaling threshold", 2024-04-25T11:47:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T11:47:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "progress", 2024-04-25T11:47:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "allow_pg_recovery_event": { 2024-04-25T11:47:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "allow_pg_recovery_event", 2024-04-25T11:47:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T11:47:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "enabled": { 2024-04-25T11:47:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "enabled", 2024-04-25T11:47:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "max_completed_events": { 2024-04-25T11:47:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_completed_events", 2024-04-25T11:47:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "50", 2024-04-25T11:47:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "number of past completed events to remember", 2024-04-25T11:47:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-25T11:47:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-25T11:47:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5", 2024-04-25T11:47:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how long the module is going to sleep", 2024-04-25T11:47:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "prometheus", 2024-04-25T11:47:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "cache": { 2024-04-25T11:47:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cache", 2024-04-25T11:47:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "exclude_perf_counters": { 2024-04-25T11:47:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "exclude_perf_counters", 2024-04-25T11:47:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T11:47:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T11:47:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_stats_pools": { 2024-04-25T11:47:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rbd_stats_pools", 2024-04-25T11:47:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T11:47:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T11:47:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "300", 2024-04-25T11:47:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "scrape_interval": { 2024-04-25T11:47:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "scrape_interval", 2024-04-25T11:47:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-25T11:47:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "15.0", 2024-04-25T11:47:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "server_addr": { 2024-04-25T11:47:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_addr", 2024-04-25T11:47:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "::", 2024-04-25T11:47:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T11:47:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "server_port": { 2024-04-25T11:47:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_port", 2024-04-25T11:47:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "9283", 2024-04-25T11:47:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T11:47:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "stale_cache_strategy": { 2024-04-25T11:47:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "stale_cache_strategy", 2024-04-25T11:47:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "log", 2024-04-25T11:47:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "standby_behaviour": { 2024-04-25T11:47:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "standby_behaviour", 2024-04-25T11:47:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "default", 2024-04-25T11:47:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: "default", 2024-04-25T11:47:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: "error" 2024-04-25T11:47:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: "standby_error_status_code": { 2024-04-25T11:47:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "standby_error_status_code", 2024-04-25T11:47:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "500", 2024-04-25T11:47:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "400", 2024-04-25T11:47:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "599", 2024-04-25T11:47:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rbd_support", 2024-04-25T11:47:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "max_concurrent_snap_create": { 2024-04-25T11:47:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_concurrent_snap_create", 2024-04-25T11:47:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-25T11:47:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "mirror_snapshot_schedule": { 2024-04-25T11:47:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T11:47:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "trash_purge_schedule": { 2024-04-25T11:47:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "trash_purge_schedule", 2024-04-25T11:47:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "restful", 2024-04-25T11:47:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "enable_auth": { 2024-04-25T11:47:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "enable_auth", 2024-04-25T11:47:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "key_file": { 2024-04-25T11:47:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "key_file", 2024-04-25T11:47:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "server_addr": { 2024-04-25T11:47:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_addr", 2024-04-25T11:47:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "server_port": { 2024-04-25T11:47:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_port", 2024-04-25T11:47:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rgw", 2024-04-25T11:47:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rook", 2024-04-25T11:47:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "storage_class": { 2024-04-25T11:47:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "storage_class", 2024-04-25T11:47:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "local", 2024-04-25T11:47:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T11:47:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "selftest", 2024-04-25T11:47:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "roption1": { 2024-04-25T11:47:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "roption1", 2024-04-25T11:47:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "roption2": { 2024-04-25T11:47:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "roption2", 2024-04-25T11:47:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "xyz", 2024-04-25T11:47:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption1": { 2024-04-25T11:47:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption1", 2024-04-25T11:47:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption2": { 2024-04-25T11:47:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption2", 2024-04-25T11:47:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption3": { 2024-04-25T11:47:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption3", 2024-04-25T11:47:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-25T11:47:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption4": { 2024-04-25T11:47:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption4", 2024-04-25T11:47:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption5": { 2024-04-25T11:47:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption5", 2024-04-25T11:47:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.084 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption6": { 2024-04-25T11:47:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption6", 2024-04-25T11:47:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.085 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption7": { 2024-04-25T11:47:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption7", 2024-04-25T11:47:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1", 2024-04-25T11:47:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "42", 2024-04-25T11:47:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.086 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: "testkey": { 2024-04-25T11:47:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "testkey", 2024-04-25T11:47:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: "testlkey": { 2024-04-25T11:47:03.088 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "testlkey", 2024-04-25T11:47:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: "testnewline": { 2024-04-25T11:47:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "testnewline", 2024-04-25T11:47:03.089 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.090 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "snap_schedule", 2024-04-25T11:47:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: "allow_m_granularity": { 2024-04-25T11:47:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "allow_m_granularity", 2024-04-25T11:47:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T11:47:03.091 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: "dump_on_update": { 2024-04-25T11:47:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "dump_on_update", 2024-04-25T11:47:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "dump database to debug log on update", 2024-04-25T11:47:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.092 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.093 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.094 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.095 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.096 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.098 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "stats", 2024-04-25T11:47:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.099 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.100 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.101 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.101 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.101 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.101 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.101 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.101 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.101 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.101 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.101 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.101 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.101 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.101 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.101 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.102 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.102 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.102 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.102 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.102 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.102 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.102 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.102 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.102 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.102 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.102 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.102 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.102 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.102 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.103 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.104 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "status", 2024-04-25T11:47:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.105 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.106 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.107 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.108 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.109 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.110 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.110 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.110 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.110 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.110 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.110 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.110 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.110 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.110 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.110 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.110 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.110 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "telegraf", 2024-04-25T11:47:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: "address": { 2024-04-25T11:47:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "address", 2024-04-25T11:47:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T11:47:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.112 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-25T11:47:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-25T11:47:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "15", 2024-04-25T11:47:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.113 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.114 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.115 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.116 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.117 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.117 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.117 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.117 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.117 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.117 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.117 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.117 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.117 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.117 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.117 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.117 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.117 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.117 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.118 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.119 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "telemetry", 2024-04-25T11:47:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_basic": { 2024-04-25T11:47:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_basic", 2024-04-25T11:47:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.120 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T11:47:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_crash": { 2024-04-25T11:47:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_crash", 2024-04-25T11:47:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.121 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T11:47:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_device": { 2024-04-25T11:47:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_device", 2024-04-25T11:47:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.122 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T11:47:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_ident": { 2024-04-25T11:47:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_ident", 2024-04-25T11:47:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.123 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T11:47:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_perf": { 2024-04-25T11:47:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_perf", 2024-04-25T11:47:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.124 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T11:47:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: "contact": { 2024-04-25T11:47:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "contact", 2024-04-25T11:47:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.125 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: "description": { 2024-04-25T11:47:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "description", 2024-04-25T11:47:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.126 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: "device_url": { 2024-04-25T11:47:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "device_url", 2024-04-25T11:47:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T11:47:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.127 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: "enabled": { 2024-04-25T11:47:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "enabled", 2024-04-25T11:47:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.128 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-25T11:47:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-25T11:47:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "24", 2024-04-25T11:47:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "8", 2024-04-25T11:47:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.129 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: "last_opt_revision": { 2024-04-25T11:47:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "last_opt_revision", 2024-04-25T11:47:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "1", 2024-04-25T11:47:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.130 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: "leaderboard": { 2024-04-25T11:47:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "leaderboard", 2024-04-25T11:47:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.131 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: "leaderboard_description": { 2024-04-25T11:47:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "leaderboard_description", 2024-04-25T11:47:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.133 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.134 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.135 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.136 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: "organization": { 2024-04-25T11:47:03.137 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "organization", 2024-04-25T11:47:03.138 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.138 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.138 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.138 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.138 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.138 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.138 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.138 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.138 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.138 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.138 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.138 INFO:teuthology.orchestra.run.smithi090.stdout: "proxy": { 2024-04-25T11:47:03.138 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "proxy", 2024-04-25T11:47:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.139 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: "url": { 2024-04-25T11:47:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "url", 2024-04-25T11:47:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.140 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T11:47:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.142 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "test_orchestrator", 2024-04-25T11:47:03.142 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.142 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.142 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.142 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.142 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.142 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.142 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.142 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.142 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.142 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.142 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.142 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.143 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.143 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.143 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.143 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.143 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.143 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.143 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.143 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.143 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.143 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.143 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.143 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.143 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.144 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.144 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.144 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.144 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.144 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.144 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.144 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.144 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.144 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.144 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.144 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.144 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.144 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.145 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.145 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.145 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.145 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.145 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.145 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.145 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.145 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.145 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.145 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.145 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.145 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.145 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.145 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.146 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.146 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.146 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.146 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.146 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.146 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.146 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.146 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.146 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.146 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.146 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.146 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.146 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.147 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.147 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.147 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.147 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.147 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.147 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.147 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.147 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.147 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.147 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.147 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.147 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.148 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.148 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.148 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.148 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.148 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.148 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.148 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.148 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "volumes", 2024-04-25T11:47:03.148 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.148 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.148 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.148 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.148 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.149 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.149 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.149 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.149 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.149 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.149 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.149 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.149 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.149 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.149 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.149 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.149 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.149 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.149 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.150 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.150 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.150 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.150 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.150 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.150 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.150 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.150 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.150 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.150 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.150 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.150 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.150 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.151 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.151 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.151 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.151 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.151 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.151 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.151 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.151 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.151 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.151 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.151 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.151 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.151 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.152 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.152 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.152 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.152 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.152 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.152 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.152 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.152 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.152 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.152 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.152 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.152 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.152 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.153 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.153 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.153 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.153 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.153 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.153 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.153 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.153 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.153 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.153 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.153 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.153 INFO:teuthology.orchestra.run.smithi090.stdout: "max_concurrent_clones": { 2024-04-25T11:47:03.153 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_concurrent_clones", 2024-04-25T11:47:03.153 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.153 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.154 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.154 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "4", 2024-04-25T11:47:03.154 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.154 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.154 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.154 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T11:47:03.154 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.154 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.154 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.154 INFO:teuthology.orchestra.run.smithi090.stdout: "periodic_async_work": { 2024-04-25T11:47:03.154 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "periodic_async_work", 2024-04-25T11:47:03.154 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.155 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.155 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.155 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.155 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.155 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.155 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.155 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Periodically check for async work", 2024-04-25T11:47:03.155 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.155 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.155 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.155 INFO:teuthology.orchestra.run.smithi090.stdout: "snapshot_clone_delay": { 2024-04-25T11:47:03.155 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "snapshot_clone_delay", 2024-04-25T11:47:03.155 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.155 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.155 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.156 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.156 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.156 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.156 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.156 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T11:47:03.156 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.156 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.156 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.156 INFO:teuthology.orchestra.run.smithi090.stdout: "snapshot_clone_no_wait": { 2024-04-25T11:47:03.156 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T11:47:03.156 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.156 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.156 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.156 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.157 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.157 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.157 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.157 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T11:47:03.157 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.157 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.157 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.157 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.157 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.157 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.157 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.157 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.157 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.157 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.157 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.158 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.158 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.158 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.158 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.158 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.158 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.158 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.158 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "zabbix", 2024-04-25T11:47:03.158 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.158 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.158 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.158 INFO:teuthology.orchestra.run.smithi090.stdout: "discovery_interval": { 2024-04-25T11:47:03.159 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "discovery_interval", 2024-04-25T11:47:03.159 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-25T11:47:03.159 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.159 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.159 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "100", 2024-04-25T11:47:03.159 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.159 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.159 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.159 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.159 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.159 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.159 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.159 INFO:teuthology.orchestra.run.smithi090.stdout: "identifier": { 2024-04-25T11:47:03.160 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "identifier", 2024-04-25T11:47:03.160 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.160 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.160 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.160 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.160 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.160 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.160 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.160 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.160 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.160 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.160 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.160 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-25T11:47:03.161 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-25T11:47:03.161 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.161 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.161 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.161 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-25T11:47:03.161 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.161 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.161 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.161 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.161 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.161 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.161 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.161 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.161 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.161 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.162 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.162 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.162 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.162 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.162 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.162 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.162 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.162 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.162 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.162 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.162 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.162 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.162 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.162 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.163 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.163 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.163 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.163 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.163 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.163 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.163 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.163 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.163 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.163 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.163 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.163 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.163 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.164 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.164 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.164 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.164 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.164 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.164 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.164 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.164 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.164 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.164 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.164 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.164 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.164 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.165 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.165 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.165 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.165 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.165 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.165 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.165 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.165 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.165 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.165 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.165 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.165 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.165 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.165 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.166 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.166 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.166 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.166 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.166 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.166 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.166 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.166 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.166 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.166 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.166 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.166 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.166 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.167 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.167 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.167 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.167 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.167 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.167 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.167 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.167 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.167 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.167 INFO:teuthology.orchestra.run.smithi090.stdout: "zabbix_host": { 2024-04-25T11:47:03.167 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "zabbix_host", 2024-04-25T11:47:03.167 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.167 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.168 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.168 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.168 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.168 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.168 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.168 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.168 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.168 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.168 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.168 INFO:teuthology.orchestra.run.smithi090.stdout: "zabbix_port": { 2024-04-25T11:47:03.168 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "zabbix_port", 2024-04-25T11:47:03.168 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.168 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.168 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.169 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10051", 2024-04-25T11:47:03.169 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.169 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.169 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.169 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.169 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.169 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.169 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.169 INFO:teuthology.orchestra.run.smithi090.stdout: "zabbix_sender": { 2024-04-25T11:47:03.169 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "zabbix_sender", 2024-04-25T11:47:03.169 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.169 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.170 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.170 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T11:47:03.170 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.170 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.170 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.170 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.170 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.170 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.170 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.170 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.170 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.170 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.170 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-25T11:47:03.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.171 INFO:teuthology.orchestra.run.smithi090.stdout: "gid": 24959, 2024-04-25T11:47:03.171 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "y", 2024-04-25T11:47:03.171 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_features": 4540701547738038271, 2024-04-25T11:47:03.171 INFO:teuthology.orchestra.run.smithi090.stdout: "available_modules": [ 2024-04-25T11:47:03.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.171 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "alerts", 2024-04-25T11:47:03.171 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.171 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.171 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.171 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-25T11:47:03.171 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-25T11:47:03.171 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.171 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.171 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.172 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-25T11:47:03.172 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.172 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.172 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.172 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T11:47:03.172 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.172 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.172 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.172 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.172 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.172 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.172 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.172 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.173 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.173 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.173 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.173 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.173 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.173 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.173 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.173 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.173 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.173 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.173 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.173 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.173 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.173 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.173 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.174 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.174 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.174 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.174 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.174 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.174 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.174 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.174 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.174 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.174 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.174 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.174 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.174 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.175 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.175 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.175 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.175 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.175 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.175 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.175 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.175 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.175 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.175 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.175 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.175 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.175 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.175 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.176 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.176 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.176 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.176 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.176 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.176 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.176 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.176 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.176 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.176 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.176 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.176 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.176 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_destination": { 2024-04-25T11:47:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_destination", 2024-04-25T11:47:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.178 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.178 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Email address to send alerts to", 2024-04-25T11:47:03.178 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.178 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.178 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.178 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_from_name": { 2024-04-25T11:47:03.178 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_from_name", 2024-04-25T11:47:03.178 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.178 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.178 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.178 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "Ceph", 2024-04-25T11:47:03.178 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.178 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.179 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.179 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Email From: name", 2024-04-25T11:47:03.179 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.179 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.179 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.179 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_host": { 2024-04-25T11:47:03.179 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_host", 2024-04-25T11:47:03.179 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.179 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.179 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.179 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.179 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.179 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.180 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.180 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "SMTP server", 2024-04-25T11:47:03.180 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.180 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.180 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.180 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_password": { 2024-04-25T11:47:03.180 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_password", 2024-04-25T11:47:03.180 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.180 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.180 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.180 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.180 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.180 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.181 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.181 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Password to authenticate with", 2024-04-25T11:47:03.181 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.181 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.181 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.181 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_port": { 2024-04-25T11:47:03.181 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_port", 2024-04-25T11:47:03.181 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.181 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.181 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.181 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "465", 2024-04-25T11:47:03.181 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.181 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.182 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.182 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "SMTP port", 2024-04-25T11:47:03.182 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.182 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.182 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.182 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_sender": { 2024-04-25T11:47:03.182 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_sender", 2024-04-25T11:47:03.182 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.182 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.182 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.182 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.182 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.182 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.183 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.183 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "SMTP envelope sender", 2024-04-25T11:47:03.183 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.183 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.183 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.183 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_ssl": { 2024-04-25T11:47:03.183 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_ssl", 2024-04-25T11:47:03.183 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.183 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.183 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.183 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.183 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.183 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.183 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.183 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T11:47:03.183 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.184 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.184 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.184 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_user": { 2024-04-25T11:47:03.184 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_user", 2024-04-25T11:47:03.184 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.184 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.184 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.184 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.184 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.184 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.184 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.184 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "User to authenticate as", 2024-04-25T11:47:03.185 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.185 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.185 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.185 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.185 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.185 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.185 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.185 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.185 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.185 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.185 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.185 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.185 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.185 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.186 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.186 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.186 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.186 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.186 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "balancer", 2024-04-25T11:47:03.186 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.186 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.186 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.186 INFO:teuthology.orchestra.run.smithi090.stdout: "active": { 2024-04-25T11:47:03.186 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "active", 2024-04-25T11:47:03.186 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.186 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.187 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.187 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.187 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.187 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.187 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.187 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T11:47:03.187 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.187 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.187 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.187 INFO:teuthology.orchestra.run.smithi090.stdout: "begin_time": { 2024-04-25T11:47:03.187 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "begin_time", 2024-04-25T11:47:03.187 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.188 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.188 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.188 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0000", 2024-04-25T11:47:03.188 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.188 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.188 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.188 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T11:47:03.188 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T11:47:03.188 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.188 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.188 INFO:teuthology.orchestra.run.smithi090.stdout: "begin_weekday": { 2024-04-25T11:47:03.188 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "begin_weekday", 2024-04-25T11:47:03.188 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-25T11:47:03.189 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.189 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.189 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.189 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "0", 2024-04-25T11:47:03.189 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "6", 2024-04-25T11:47:03.189 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.189 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T11:47:03.189 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T11:47:03.189 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.189 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.189 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_compat_max_iterations": { 2024-04-25T11:47:03.189 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crush_compat_max_iterations", 2024-04-25T11:47:03.189 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-25T11:47:03.189 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.190 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.190 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "25", 2024-04-25T11:47:03.190 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1", 2024-04-25T11:47:03.190 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "250", 2024-04-25T11:47:03.190 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.190 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T11:47:03.190 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.190 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.190 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.190 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_compat_metrics": { 2024-04-25T11:47:03.190 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crush_compat_metrics", 2024-04-25T11:47:03.190 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.190 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.191 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.191 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T11:47:03.191 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.191 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.191 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.191 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T11:47:03.191 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T11:47:03.191 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.191 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.191 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_compat_step": { 2024-04-25T11:47:03.191 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crush_compat_step", 2024-04-25T11:47:03.191 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-25T11:47:03.191 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.192 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.192 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0.5", 2024-04-25T11:47:03.192 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "0.001", 2024-04-25T11:47:03.192 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "0.999", 2024-04-25T11:47:03.192 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.192 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "aggressiveness of optimization", 2024-04-25T11:47:03.192 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T11:47:03.192 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.192 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.192 INFO:teuthology.orchestra.run.smithi090.stdout: "end_time": { 2024-04-25T11:47:03.192 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "end_time", 2024-04-25T11:47:03.192 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.193 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.193 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.193 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "2359", 2024-04-25T11:47:03.193 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.193 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.193 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.193 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T11:47:03.193 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T11:47:03.193 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.193 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.193 INFO:teuthology.orchestra.run.smithi090.stdout: "end_weekday": { 2024-04-25T11:47:03.193 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "end_weekday", 2024-04-25T11:47:03.193 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-25T11:47:03.193 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.194 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.194 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.194 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "0", 2024-04-25T11:47:03.194 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "6", 2024-04-25T11:47:03.194 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.194 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T11:47:03.194 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T11:47:03.194 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.194 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.194 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.194 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.194 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.194 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.195 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.195 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.195 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.195 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.195 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.195 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.195 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.195 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.195 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.195 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.195 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.195 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.195 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.195 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.196 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.196 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.196 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.196 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.196 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.196 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.196 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.196 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.196 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.196 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.196 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.196 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.196 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.197 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.197 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.197 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.197 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.197 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.197 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.197 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.197 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.197 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.197 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.197 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.197 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.197 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.198 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.198 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.198 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.198 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.198 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.198 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.198 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.198 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.198 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.198 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.198 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.198 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.198 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.199 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.199 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.199 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.199 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.199 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.199 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.199 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.199 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.199 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.199 INFO:teuthology.orchestra.run.smithi090.stdout: "min_score": { 2024-04-25T11:47:03.199 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "min_score", 2024-04-25T11:47:03.199 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-25T11:47:03.199 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.199 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.200 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.200 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.200 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.200 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.200 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T11:47:03.200 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.200 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.200 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.200 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": { 2024-04-25T11:47:03.200 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mode", 2024-04-25T11:47:03.200 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.200 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.200 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.201 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "upmap", 2024-04-25T11:47:03.201 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.201 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.201 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.201 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-compat", 2024-04-25T11:47:03.201 INFO:teuthology.orchestra.run.smithi090.stdout: "none", 2024-04-25T11:47:03.201 INFO:teuthology.orchestra.run.smithi090.stdout: "read", 2024-04-25T11:47:03.201 INFO:teuthology.orchestra.run.smithi090.stdout: "upmap", 2024-04-25T11:47:03.201 INFO:teuthology.orchestra.run.smithi090.stdout: "upmap-read" 2024-04-25T11:47:03.201 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.201 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Balancer mode", 2024-04-25T11:47:03.201 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.201 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.201 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.202 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_ids": { 2024-04-25T11:47:03.202 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "pool_ids", 2024-04-25T11:47:03.202 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.202 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.202 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.202 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.202 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.202 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.202 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.202 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T11:47:03.202 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.202 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.202 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.203 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-25T11:47:03.203 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-25T11:47:03.203 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.203 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.203 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.203 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-25T11:47:03.203 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.203 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.203 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.203 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T11:47:03.203 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.203 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.203 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.204 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.204 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.204 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.204 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.204 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.204 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.204 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.204 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.204 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.204 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.204 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.204 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.204 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.204 INFO:teuthology.orchestra.run.smithi090.stdout: "upmap_max_deviation": { 2024-04-25T11:47:03.205 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "upmap_max_deviation", 2024-04-25T11:47:03.205 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.205 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.205 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.205 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5", 2024-04-25T11:47:03.205 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1", 2024-04-25T11:47:03.205 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.205 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.205 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T11:47:03.205 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T11:47:03.205 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.205 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.205 INFO:teuthology.orchestra.run.smithi090.stdout: "upmap_max_optimizations": { 2024-04-25T11:47:03.205 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "upmap_max_optimizations", 2024-04-25T11:47:03.205 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-25T11:47:03.206 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.206 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.206 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-25T11:47:03.206 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.206 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.206 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.206 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T11:47:03.206 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.206 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.206 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.206 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.206 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.207 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cephadm", 2024-04-25T11:47:03.207 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.207 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.207 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.207 INFO:teuthology.orchestra.run.smithi090.stdout: "agent_down_multiplier": { 2024-04-25T11:47:03.207 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "agent_down_multiplier", 2024-04-25T11:47:03.207 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-25T11:47:03.207 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.207 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.207 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "3.0", 2024-04-25T11:47:03.207 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.207 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.207 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.207 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T11:47:03.207 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.208 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.208 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.208 INFO:teuthology.orchestra.run.smithi090.stdout: "agent_refresh_rate": { 2024-04-25T11:47:03.208 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "agent_refresh_rate", 2024-04-25T11:47:03.208 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.208 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.208 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.208 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "20", 2024-04-25T11:47:03.208 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.208 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.208 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.208 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T11:47:03.208 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.208 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.208 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.209 INFO:teuthology.orchestra.run.smithi090.stdout: "agent_starting_port": { 2024-04-25T11:47:03.209 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "agent_starting_port", 2024-04-25T11:47:03.209 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.209 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.209 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.209 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "4721", 2024-04-25T11:47:03.209 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.209 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.209 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.209 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T11:47:03.209 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.209 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.209 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.209 INFO:teuthology.orchestra.run.smithi090.stdout: "allow_ptrace": { 2024-04-25T11:47:03.210 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "allow_ptrace", 2024-04-25T11:47:03.210 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.210 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.210 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.210 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.210 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.210 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.210 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.210 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T11:47:03.210 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T11:47:03.210 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.210 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.210 INFO:teuthology.orchestra.run.smithi090.stdout: "autotune_interval": { 2024-04-25T11:47:03.211 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "autotune_interval", 2024-04-25T11:47:03.211 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.211 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.211 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.211 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-25T11:47:03.211 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.211 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.211 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.211 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T11:47:03.211 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.211 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.211 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.211 INFO:teuthology.orchestra.run.smithi090.stdout: "autotune_memory_target_ratio": { 2024-04-25T11:47:03.212 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T11:47:03.212 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-25T11:47:03.212 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.212 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.212 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0.7", 2024-04-25T11:47:03.212 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.212 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.212 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.212 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T11:47:03.212 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.212 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.212 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.212 INFO:teuthology.orchestra.run.smithi090.stdout: "cephadm_log_destination": { 2024-04-25T11:47:03.212 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cephadm_log_destination", 2024-04-25T11:47:03.212 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.213 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.213 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.213 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.213 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.213 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.213 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.213 INFO:teuthology.orchestra.run.smithi090.stdout: "file", 2024-04-25T11:47:03.213 INFO:teuthology.orchestra.run.smithi090.stdout: "file,syslog", 2024-04-25T11:47:03.213 INFO:teuthology.orchestra.run.smithi090.stdout: "syslog" 2024-04-25T11:47:03.213 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.213 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T11:47:03.213 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.213 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.213 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.214 INFO:teuthology.orchestra.run.smithi090.stdout: "cgroups_split": { 2024-04-25T11:47:03.214 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cgroups_split", 2024-04-25T11:47:03.214 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.214 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.214 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.214 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.214 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.214 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.214 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.214 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T11:47:03.214 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.214 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.214 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: "config_checks_enabled": { 2024-04-25T11:47:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "config_checks_enabled", 2024-04-25T11:47:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T11:47:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: "config_dashboard": { 2024-04-25T11:47:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "config_dashboard", 2024-04-25T11:47:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T11:47:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.217 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.217 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_alertmanager": { 2024-04-25T11:47:03.217 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_alertmanager", 2024-04-25T11:47:03.217 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.217 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.217 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.217 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T11:47:03.217 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.217 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.217 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.217 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Prometheus container image", 2024-04-25T11:47:03.217 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.217 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.217 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.218 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_base": { 2024-04-25T11:47:03.218 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_base", 2024-04-25T11:47:03.218 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.218 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.218 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.218 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T11:47:03.218 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.218 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.218 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.218 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Container image name, without the tag", 2024-04-25T11:47:03.218 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.218 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.219 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.219 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_elasticsearch": { 2024-04-25T11:47:03.219 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_elasticsearch", 2024-04-25T11:47:03.219 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.219 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.219 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.219 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T11:47:03.219 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.219 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.219 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.219 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "elasticsearch container image", 2024-04-25T11:47:03.219 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.219 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.220 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.220 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_grafana": { 2024-04-25T11:47:03.220 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_grafana", 2024-04-25T11:47:03.220 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.220 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.220 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.220 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T11:47:03.220 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.220 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.220 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.220 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Prometheus container image", 2024-04-25T11:47:03.220 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.220 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.221 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.221 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_haproxy": { 2024-04-25T11:47:03.221 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_haproxy", 2024-04-25T11:47:03.221 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.221 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.221 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.221 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T11:47:03.221 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.221 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.221 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.221 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "HAproxy container image", 2024-04-25T11:47:03.221 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.221 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.222 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.222 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_jaeger_agent": { 2024-04-25T11:47:03.222 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_jaeger_agent", 2024-04-25T11:47:03.222 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.222 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.222 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.222 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T11:47:03.222 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.222 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.222 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.222 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Jaeger agent container image", 2024-04-25T11:47:03.222 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.222 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.222 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.223 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_jaeger_collector": { 2024-04-25T11:47:03.223 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_jaeger_collector", 2024-04-25T11:47:03.223 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.223 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.223 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.223 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T11:47:03.223 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.223 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.223 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.223 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Jaeger collector container image", 2024-04-25T11:47:03.223 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.223 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.224 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.224 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_jaeger_query": { 2024-04-25T11:47:03.224 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_jaeger_query", 2024-04-25T11:47:03.224 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.224 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.224 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.224 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T11:47:03.224 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.224 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.224 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.224 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Jaeger query container image", 2024-04-25T11:47:03.224 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.224 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.225 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.225 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_keepalived": { 2024-04-25T11:47:03.225 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_keepalived", 2024-04-25T11:47:03.225 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.225 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.225 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.225 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T11:47:03.225 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.225 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.225 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.225 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Keepalived container image", 2024-04-25T11:47:03.225 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.226 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.226 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.226 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_loki": { 2024-04-25T11:47:03.226 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_loki", 2024-04-25T11:47:03.226 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.226 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.226 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.226 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T11:47:03.226 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.226 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.226 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.226 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Loki container image", 2024-04-25T11:47:03.226 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.227 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.227 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.227 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_node_exporter": { 2024-04-25T11:47:03.227 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_node_exporter", 2024-04-25T11:47:03.227 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.227 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.227 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.227 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T11:47:03.227 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.227 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.227 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.227 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Prometheus container image", 2024-04-25T11:47:03.227 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.227 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.228 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.228 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_nvmeof": { 2024-04-25T11:47:03.228 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_nvmeof", 2024-04-25T11:47:03.228 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.228 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.228 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.228 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T11:47:03.228 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.228 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.228 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.228 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Nvme-of container image", 2024-04-25T11:47:03.228 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.228 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.229 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.229 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_prometheus": { 2024-04-25T11:47:03.229 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_prometheus", 2024-04-25T11:47:03.229 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.229 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.229 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.229 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T11:47:03.229 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.229 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.229 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.229 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Prometheus container image", 2024-04-25T11:47:03.229 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.229 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.229 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.230 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_promtail": { 2024-04-25T11:47:03.230 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_promtail", 2024-04-25T11:47:03.230 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.230 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.230 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.230 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T11:47:03.230 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.230 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.230 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.230 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Promtail container image", 2024-04-25T11:47:03.230 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.230 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.230 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.231 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_samba": { 2024-04-25T11:47:03.231 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_samba", 2024-04-25T11:47:03.231 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.231 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.231 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.231 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T11:47:03.231 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.231 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.231 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.231 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Samba/SMB container image", 2024-04-25T11:47:03.231 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.231 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.231 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.231 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_snmp_gateway": { 2024-04-25T11:47:03.232 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_snmp_gateway", 2024-04-25T11:47:03.232 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.232 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.232 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.232 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T11:47:03.232 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.232 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.232 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.232 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "SNMP Gateway container image", 2024-04-25T11:47:03.232 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.232 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.232 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.232 INFO:teuthology.orchestra.run.smithi090.stdout: "container_init": { 2024-04-25T11:47:03.232 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_init", 2024-04-25T11:47:03.232 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T11:47:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: "daemon_cache_timeout": { 2024-04-25T11:47:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "daemon_cache_timeout", 2024-04-25T11:47:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.234 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-25T11:47:03.234 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.234 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.234 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.234 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T11:47:03.234 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.234 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.234 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.234 INFO:teuthology.orchestra.run.smithi090.stdout: "default_cephadm_command_timeout": { 2024-04-25T11:47:03.234 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T11:47:03.234 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.234 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.235 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.235 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "900", 2024-04-25T11:47:03.235 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.235 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.235 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.235 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T11:47:03.235 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.235 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.235 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.235 INFO:teuthology.orchestra.run.smithi090.stdout: "default_registry": { 2024-04-25T11:47:03.235 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "default_registry", 2024-04-25T11:47:03.235 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.235 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.235 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.235 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "docker.io", 2024-04-25T11:47:03.236 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.236 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.236 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.236 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T11:47:03.236 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.236 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.236 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.236 INFO:teuthology.orchestra.run.smithi090.stdout: "device_cache_timeout": { 2024-04-25T11:47:03.236 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "device_cache_timeout", 2024-04-25T11:47:03.236 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.236 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.236 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.236 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "1800", 2024-04-25T11:47:03.236 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.237 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.237 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.237 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "seconds to cache device inventory", 2024-04-25T11:47:03.237 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.237 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.237 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.237 INFO:teuthology.orchestra.run.smithi090.stdout: "device_enhanced_scan": { 2024-04-25T11:47:03.237 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "device_enhanced_scan", 2024-04-25T11:47:03.237 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.237 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.237 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.237 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.237 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.237 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.237 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.238 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T11:47:03.238 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.238 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.238 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.238 INFO:teuthology.orchestra.run.smithi090.stdout: "facts_cache_timeout": { 2024-04-25T11:47:03.238 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "facts_cache_timeout", 2024-04-25T11:47:03.238 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.238 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.238 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.238 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-25T11:47:03.238 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.238 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.238 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.239 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "seconds to cache host facts data", 2024-04-25T11:47:03.239 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.239 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.239 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.239 INFO:teuthology.orchestra.run.smithi090.stdout: "grafana_dashboards_path": { 2024-04-25T11:47:03.239 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "grafana_dashboards_path", 2024-04-25T11:47:03.239 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.239 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.239 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.239 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T11:47:03.239 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.239 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.239 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.239 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T11:47:03.240 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.240 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.240 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.240 INFO:teuthology.orchestra.run.smithi090.stdout: "host_check_interval": { 2024-04-25T11:47:03.240 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "host_check_interval", 2024-04-25T11:47:03.240 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.240 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.240 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.240 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-25T11:47:03.240 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.240 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.240 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.240 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to perform a host check", 2024-04-25T11:47:03.240 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.240 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.241 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.241 INFO:teuthology.orchestra.run.smithi090.stdout: "hw_monitoring": { 2024-04-25T11:47:03.241 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "hw_monitoring", 2024-04-25T11:47:03.241 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.241 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.241 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.241 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.241 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.241 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.241 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.241 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T11:47:03.241 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.241 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.241 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.242 INFO:teuthology.orchestra.run.smithi090.stdout: "inventory_list_all": { 2024-04-25T11:47:03.242 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "inventory_list_all", 2024-04-25T11:47:03.242 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.242 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.242 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.242 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.242 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.242 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.242 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.242 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T11:47:03.242 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.242 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.242 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.243 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.243 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.243 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.243 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.243 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.243 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.243 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.243 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.243 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.243 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.243 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.243 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.243 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.243 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.243 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.243 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.244 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.244 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.244 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.244 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.244 INFO:teuthology.orchestra.run.smithi090.stdout: "log_refresh_metadata": { 2024-04-25T11:47:03.244 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_refresh_metadata", 2024-04-25T11:47:03.244 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.244 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.244 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.244 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.244 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.244 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.244 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.244 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T11:47:03.245 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.245 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.245 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.245 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.245 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.245 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.245 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.245 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.245 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.245 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.245 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.245 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.245 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T11:47:03.246 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.246 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.246 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.246 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.246 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.246 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.246 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.246 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.246 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.246 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.246 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.246 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.247 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.247 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.247 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.247 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.247 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.247 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.247 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.247 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.247 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.247 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.247 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.247 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.247 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.247 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.248 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.248 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.248 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.248 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.248 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.248 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.248 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.248 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.248 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.248 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.248 INFO:teuthology.orchestra.run.smithi090.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T11:47:03.248 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T11:47:03.248 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.248 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.248 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.249 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.249 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.249 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.249 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.249 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T11:47:03.249 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.249 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.249 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.249 INFO:teuthology.orchestra.run.smithi090.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T11:47:03.249 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T11:47:03.249 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.249 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.249 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.249 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "*", 2024-04-25T11:47:03.250 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.250 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.250 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.250 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T11:47:03.250 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.250 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.250 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.250 INFO:teuthology.orchestra.run.smithi090.stdout: "max_count_per_host": { 2024-04-25T11:47:03.250 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_count_per_host", 2024-04-25T11:47:03.250 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.250 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.250 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.250 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-25T11:47:03.250 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.251 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.251 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.251 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "max number of daemons per service per host", 2024-04-25T11:47:03.251 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.251 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.251 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.251 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd_draining_count": { 2024-04-25T11:47:03.251 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_osd_draining_count", 2024-04-25T11:47:03.251 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.251 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.251 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.251 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-25T11:47:03.251 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.251 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.252 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.252 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T11:47:03.252 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.252 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.252 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.252 INFO:teuthology.orchestra.run.smithi090.stdout: "migration_current": { 2024-04-25T11:47:03.252 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "migration_current", 2024-04-25T11:47:03.252 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.252 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.252 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.252 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.252 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.252 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.252 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.253 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "internal - do not modify", 2024-04-25T11:47:03.253 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.253 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.253 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.253 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": { 2024-04-25T11:47:03.253 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mode", 2024-04-25T11:47:03.253 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.253 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.253 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.253 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "root", 2024-04-25T11:47:03.253 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.253 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.253 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.254 INFO:teuthology.orchestra.run.smithi090.stdout: "cephadm-package", 2024-04-25T11:47:03.254 INFO:teuthology.orchestra.run.smithi090.stdout: "root" 2024-04-25T11:47:03.254 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.254 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T11:47:03.254 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.254 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.254 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.254 INFO:teuthology.orchestra.run.smithi090.stdout: "oob_default_addr": { 2024-04-25T11:47:03.254 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "oob_default_addr", 2024-04-25T11:47:03.254 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.254 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.254 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.254 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "169.254.1.1", 2024-04-25T11:47:03.254 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.255 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.255 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.255 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T11:47:03.255 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.255 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.255 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.255 INFO:teuthology.orchestra.run.smithi090.stdout: "prometheus_alerts_path": { 2024-04-25T11:47:03.255 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "prometheus_alerts_path", 2024-04-25T11:47:03.255 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.255 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.255 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.255 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T11:47:03.255 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.255 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.255 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.256 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T11:47:03.256 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.256 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.256 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.256 INFO:teuthology.orchestra.run.smithi090.stdout: "registry_insecure": { 2024-04-25T11:47:03.256 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "registry_insecure", 2024-04-25T11:47:03.256 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.256 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.256 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.256 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.256 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.256 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.256 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.256 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T11:47:03.256 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.257 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.257 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.257 INFO:teuthology.orchestra.run.smithi090.stdout: "registry_password": { 2024-04-25T11:47:03.257 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "registry_password", 2024-04-25T11:47:03.257 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.257 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.257 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.257 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.257 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.257 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.257 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.257 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T11:47:03.257 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.257 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.258 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.258 INFO:teuthology.orchestra.run.smithi090.stdout: "registry_url": { 2024-04-25T11:47:03.258 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "registry_url", 2024-04-25T11:47:03.258 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.258 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.258 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.258 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.258 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.258 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.258 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.258 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T11:47:03.258 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.259 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.259 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.259 INFO:teuthology.orchestra.run.smithi090.stdout: "registry_username": { 2024-04-25T11:47:03.259 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "registry_username", 2024-04-25T11:47:03.259 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.259 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.259 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.259 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.259 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.259 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.259 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.259 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T11:47:03.259 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.259 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.260 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.260 INFO:teuthology.orchestra.run.smithi090.stdout: "secure_monitoring_stack": { 2024-04-25T11:47:03.260 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "secure_monitoring_stack", 2024-04-25T11:47:03.260 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.260 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.260 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.260 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.260 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.260 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.260 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.260 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T11:47:03.260 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.260 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.260 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.261 INFO:teuthology.orchestra.run.smithi090.stdout: "service_discovery_port": { 2024-04-25T11:47:03.261 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "service_discovery_port", 2024-04-25T11:47:03.261 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.261 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.261 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.261 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8765", 2024-04-25T11:47:03.261 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.261 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.261 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.261 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "cephadm service discovery port", 2024-04-25T11:47:03.261 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.261 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.261 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.262 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.262 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.262 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.262 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.262 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.262 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.262 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.262 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.262 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.262 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.262 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.262 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.262 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.263 INFO:teuthology.orchestra.run.smithi090.stdout: "ssh_config_file": { 2024-04-25T11:47:03.263 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ssh_config_file", 2024-04-25T11:47:03.263 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.263 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.263 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.263 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.263 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.263 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.263 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.263 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T11:47:03.263 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.263 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.263 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.263 INFO:teuthology.orchestra.run.smithi090.stdout: "use_agent": { 2024-04-25T11:47:03.264 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "use_agent", 2024-04-25T11:47:03.264 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.264 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.264 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.264 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.264 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.264 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.264 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.264 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T11:47:03.264 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.264 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.264 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.264 INFO:teuthology.orchestra.run.smithi090.stdout: "use_repo_digest": { 2024-04-25T11:47:03.264 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "use_repo_digest", 2024-04-25T11:47:03.265 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.265 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.265 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.265 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.265 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.265 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.265 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.265 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T11:47:03.265 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.265 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.265 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.265 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_on_failed_host_check": { 2024-04-25T11:47:03.265 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_on_failed_host_check", 2024-04-25T11:47:03.266 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.266 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.266 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.266 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.266 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.266 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.266 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.266 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T11:47:03.266 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.266 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.266 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.266 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_on_stray_daemons": { 2024-04-25T11:47:03.267 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_on_stray_daemons", 2024-04-25T11:47:03.267 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.267 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.267 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.267 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.267 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.267 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.267 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.267 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T11:47:03.267 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.267 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.267 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.267 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_on_stray_hosts": { 2024-04-25T11:47:03.267 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_on_stray_hosts", 2024-04-25T11:47:03.268 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.268 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.268 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.268 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.268 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.268 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.268 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.268 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T11:47:03.268 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.268 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.268 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.268 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.268 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.269 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crash", 2024-04-25T11:47:03.269 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.269 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.269 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.269 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.269 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.269 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.269 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.269 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.269 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.269 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.269 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.269 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.269 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.272 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.272 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.272 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.272 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.272 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.272 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.272 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.272 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.272 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.272 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.272 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.272 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.272 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.272 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.272 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.273 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.273 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.273 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.273 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.273 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.273 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.273 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.273 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.273 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.273 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.273 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.273 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.273 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.274 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.274 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.274 INFO:teuthology.orchestra.run.smithi090.stdout: "retain_interval": { 2024-04-25T11:47:03.274 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "retain_interval", 2024-04-25T11:47:03.274 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.274 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.274 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.274 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "31536000", 2024-04-25T11:47:03.274 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.274 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.274 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.274 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T11:47:03.274 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.275 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.275 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.275 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.275 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.275 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.275 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.275 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.275 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.275 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.275 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.275 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.275 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.275 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.275 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.276 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.276 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_recent_interval": { 2024-04-25T11:47:03.276 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_recent_interval", 2024-04-25T11:47:03.276 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.276 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.276 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.276 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "1209600", 2024-04-25T11:47:03.276 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.276 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.276 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.276 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T11:47:03.276 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.276 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.277 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.277 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.277 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.277 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "dashboard", 2024-04-25T11:47:03.277 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.277 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.277 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.277 INFO:teuthology.orchestra.run.smithi090.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T11:47:03.277 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T11:47:03.277 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.277 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.277 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.277 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-25T11:47:03.278 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.278 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.278 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.278 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.278 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.278 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.278 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.278 INFO:teuthology.orchestra.run.smithi090.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T11:47:03.278 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T11:47:03.278 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.278 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.278 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.278 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T11:47:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T11:47:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.280 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.280 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.280 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.280 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.280 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.280 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.280 INFO:teuthology.orchestra.run.smithi090.stdout: "AUDIT_API_ENABLED": { 2024-04-25T11:47:03.280 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T11:47:03.280 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.280 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.280 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.280 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.280 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T11:47:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T11:47:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T11:47:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T11:47:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.283 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.283 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.283 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.283 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.283 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.283 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.283 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T11:47:03.283 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T11:47:03.283 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.283 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.283 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.283 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.283 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.284 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.284 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.284 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.284 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.284 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.284 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.284 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T11:47:03.284 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T11:47:03.284 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.284 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.284 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.284 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.284 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T11:47:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T11:47:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.286 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.286 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.286 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.286 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.286 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.286 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T11:47:03.286 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T11:47:03.286 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.286 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.286 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.286 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.286 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T11:47:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T11:47:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T11:47:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T11:47:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.289 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.289 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.289 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.289 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.289 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T11:47:03.289 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T11:47:03.289 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.289 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.289 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.289 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.289 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.289 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.289 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.290 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.290 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.290 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.290 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.290 INFO:teuthology.orchestra.run.smithi090.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T11:47:03.290 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T11:47:03.290 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.290 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.290 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.290 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.290 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.290 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.290 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.291 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.291 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.291 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.291 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.291 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T11:47:03.291 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T11:47:03.291 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.291 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.291 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.291 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "admin", 2024-04-25T11:47:03.291 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.291 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.291 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.292 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.292 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.292 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.292 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.292 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T11:47:03.292 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T11:47:03.292 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.292 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.292 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.292 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.292 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.292 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.292 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.293 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.293 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.293 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.293 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.293 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_API_URL": { 2024-04-25T11:47:03.293 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_API_URL", 2024-04-25T11:47:03.293 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.293 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.293 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.293 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.293 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.293 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.293 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.293 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.294 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.294 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.294 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.294 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T11:47:03.294 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T11:47:03.294 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.294 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.294 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.294 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "admin", 2024-04-25T11:47:03.294 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.294 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.294 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.294 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.295 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.295 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.295 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.295 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T11:47:03.295 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T11:47:03.295 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.295 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.295 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.295 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.295 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.295 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.295 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.295 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.295 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.296 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.296 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.296 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T11:47:03.296 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T11:47:03.296 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.296 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.296 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.296 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.296 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.296 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.296 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.296 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.296 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.297 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.297 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.297 INFO:teuthology.orchestra.run.smithi090.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T11:47:03.297 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T11:47:03.297 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.297 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.297 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.297 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.297 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.297 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.297 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.297 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.297 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.298 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.298 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.298 INFO:teuthology.orchestra.run.smithi090.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T11:47:03.298 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T11:47:03.298 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.298 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.298 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.298 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.298 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.298 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.298 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.298 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.298 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.299 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.299 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.299 INFO:teuthology.orchestra.run.smithi090.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-25T11:47:03.299 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-25T11:47:03.299 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.299 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.299 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.299 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "[]", 2024-04-25T11:47:03.299 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.299 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.299 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.299 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.299 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.300 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.300 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.300 INFO:teuthology.orchestra.run.smithi090.stdout: "MULTICLUSTER_CONFIG": { 2024-04-25T11:47:03.300 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-25T11:47:03.300 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.300 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.300 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.300 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "{}", 2024-04-25T11:47:03.300 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.300 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.300 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.300 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.300 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.300 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.301 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.301 INFO:teuthology.orchestra.run.smithi090.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T11:47:03.301 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T11:47:03.301 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.301 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.301 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.301 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.301 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.301 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.301 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.301 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.301 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.301 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.302 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.302 INFO:teuthology.orchestra.run.smithi090.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T11:47:03.302 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T11:47:03.302 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.302 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.302 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.302 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.302 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.302 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.302 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.302 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.302 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.302 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.303 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.303 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T11:47:03.303 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T11:47:03.303 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.303 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.303 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.303 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.303 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.303 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.303 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.303 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.303 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.303 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.304 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.304 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T11:47:03.304 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T11:47:03.304 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.304 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.304 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.304 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.304 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.304 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.304 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.304 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.304 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.304 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.305 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.305 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T11:47:03.305 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T11:47:03.305 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.305 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.305 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.305 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.305 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.305 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.305 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.305 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.305 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.305 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.305 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.306 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T11:47:03.306 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T11:47:03.306 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.306 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.306 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.306 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.306 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.306 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.306 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.306 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.306 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.306 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.306 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.307 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T11:47:03.307 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T11:47:03.307 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.307 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.307 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.307 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.307 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.307 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.307 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.307 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.307 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.307 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.307 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.307 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T11:47:03.307 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T11:47:03.308 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.308 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.308 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.308 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.308 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.308 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.308 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.308 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.308 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.308 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.308 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.308 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T11:47:03.308 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T11:47:03.309 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.309 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.309 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.309 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.309 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.309 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.309 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.309 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.309 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.309 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.309 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.309 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T11:47:03.309 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T11:47:03.310 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.310 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.310 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.310 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.310 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.310 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.310 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.310 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.310 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.310 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.310 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.310 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T11:47:03.311 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T11:47:03.311 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.311 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.311 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.311 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T11:47:03.311 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.311 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.311 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.311 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.311 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.311 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.311 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.311 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T11:47:03.312 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T11:47:03.312 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.312 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.312 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.312 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-25T11:47:03.312 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.312 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.312 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.312 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.312 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.312 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.312 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.312 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T11:47:03.313 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T11:47:03.313 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.313 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.313 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.313 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8", 2024-04-25T11:47:03.313 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.313 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.313 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.313 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.313 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.313 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.313 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.313 INFO:teuthology.orchestra.run.smithi090.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T11:47:03.313 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T11:47:03.314 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.314 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.314 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.314 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "45", 2024-04-25T11:47:03.314 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.314 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.314 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.314 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.314 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.314 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.314 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.314 INFO:teuthology.orchestra.run.smithi090.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T11:47:03.314 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T11:47:03.315 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.315 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.315 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.315 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.315 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.315 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.315 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.315 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.315 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.315 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.315 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.315 INFO:teuthology.orchestra.run.smithi090.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T11:47:03.315 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T11:47:03.315 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.316 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.316 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.316 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "admin", 2024-04-25T11:47:03.316 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.316 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.316 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.316 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.316 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.316 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.316 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.316 INFO:teuthology.orchestra.run.smithi090.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T11:47:03.316 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T11:47:03.316 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.316 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.317 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.317 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.317 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.317 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.317 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.317 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.317 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.317 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.317 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.317 INFO:teuthology.orchestra.run.smithi090.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T11:47:03.317 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T11:47:03.317 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.317 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.317 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.318 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.318 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.318 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.318 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.318 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.318 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.318 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.318 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.318 INFO:teuthology.orchestra.run.smithi090.stdout: "UNSAFE_TLS_v1_2": { 2024-04-25T11:47:03.318 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-25T11:47:03.318 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.318 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.318 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.319 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.319 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.319 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.319 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.319 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.319 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.319 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.319 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.319 INFO:teuthology.orchestra.run.smithi090.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T11:47:03.319 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T11:47:03.319 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.319 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.319 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.319 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.320 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.320 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.320 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.320 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.320 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.320 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.320 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.320 INFO:teuthology.orchestra.run.smithi090.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T11:47:03.320 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T11:47:03.320 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.320 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.320 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.320 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-25T11:47:03.320 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.321 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.321 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.321 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.321 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.321 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.321 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.321 INFO:teuthology.orchestra.run.smithi090.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T11:47:03.321 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T11:47:03.321 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.321 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.321 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.321 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5", 2024-04-25T11:47:03.321 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.321 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.321 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.322 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.322 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.322 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.322 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.322 INFO:teuthology.orchestra.run.smithi090.stdout: "cross_origin_url": { 2024-04-25T11:47:03.322 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cross_origin_url", 2024-04-25T11:47:03.322 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.322 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.322 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.322 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.322 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.322 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.323 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.323 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.323 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.323 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.323 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.323 INFO:teuthology.orchestra.run.smithi090.stdout: "crt_file": { 2024-04-25T11:47:03.323 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crt_file", 2024-04-25T11:47:03.323 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.323 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.323 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.323 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.323 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.323 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.323 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.324 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.324 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.324 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.324 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.324 INFO:teuthology.orchestra.run.smithi090.stdout: "debug": { 2024-04-25T11:47:03.324 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "debug", 2024-04-25T11:47:03.324 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.324 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.324 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.324 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.324 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.324 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.324 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.325 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Enable/disable debug options", 2024-04-25T11:47:03.325 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.325 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.325 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.325 INFO:teuthology.orchestra.run.smithi090.stdout: "jwt_token_ttl": { 2024-04-25T11:47:03.325 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "jwt_token_ttl", 2024-04-25T11:47:03.325 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.325 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.325 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.325 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "28800", 2024-04-25T11:47:03.325 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.325 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.325 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.325 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.325 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.326 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.326 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.326 INFO:teuthology.orchestra.run.smithi090.stdout: "key_file": { 2024-04-25T11:47:03.326 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "key_file", 2024-04-25T11:47:03.326 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.326 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.326 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.326 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.326 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.326 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.326 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.326 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.326 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.327 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.327 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.327 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.327 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.327 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.327 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.327 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.327 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.327 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.327 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.327 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.327 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.327 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.327 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.328 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.328 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.328 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.328 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.328 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.328 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.328 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.328 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.328 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.328 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.328 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.328 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.328 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.328 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.329 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.329 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.329 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.329 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.329 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.329 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.329 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.329 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.329 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.329 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.329 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.329 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.329 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.330 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.330 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.330 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.330 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.330 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.330 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.330 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.330 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.330 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.330 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.330 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.330 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.330 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.330 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.331 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.331 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.331 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.331 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.331 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.331 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.331 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.331 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.331 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.331 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.331 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.331 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.331 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.332 INFO:teuthology.orchestra.run.smithi090.stdout: "motd": { 2024-04-25T11:47:03.332 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "motd", 2024-04-25T11:47:03.332 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.332 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.332 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.332 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.332 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.332 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.332 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.332 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "The message of the day", 2024-04-25T11:47:03.332 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.332 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.332 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.333 INFO:teuthology.orchestra.run.smithi090.stdout: "redirect_resolve_ip_addr": { 2024-04-25T11:47:03.333 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T11:47:03.333 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.333 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.333 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.333 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.333 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.333 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.333 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.333 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.333 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.333 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.333 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.334 INFO:teuthology.orchestra.run.smithi090.stdout: "server_addr": { 2024-04-25T11:47:03.334 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_addr", 2024-04-25T11:47:03.334 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.334 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.334 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.334 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "::", 2024-04-25T11:47:03.334 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.334 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.334 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.334 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.334 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.334 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.334 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.335 INFO:teuthology.orchestra.run.smithi090.stdout: "server_port": { 2024-04-25T11:47:03.335 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_port", 2024-04-25T11:47:03.335 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.335 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.335 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.335 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8080", 2024-04-25T11:47:03.335 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.335 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.335 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.335 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.335 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.335 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.335 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.336 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.336 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.336 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.336 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.336 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.336 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.336 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.336 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.336 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.336 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.336 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.336 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.336 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.337 INFO:teuthology.orchestra.run.smithi090.stdout: "ssl": { 2024-04-25T11:47:03.337 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ssl", 2024-04-25T11:47:03.337 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.337 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.337 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.337 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.337 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.337 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.337 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.337 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.337 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.337 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.337 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.337 INFO:teuthology.orchestra.run.smithi090.stdout: "ssl_server_port": { 2024-04-25T11:47:03.338 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ssl_server_port", 2024-04-25T11:47:03.338 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.338 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.338 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.338 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8443", 2024-04-25T11:47:03.338 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.338 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.338 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.338 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.338 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.338 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.338 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.338 INFO:teuthology.orchestra.run.smithi090.stdout: "standby_behaviour": { 2024-04-25T11:47:03.339 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "standby_behaviour", 2024-04-25T11:47:03.339 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.339 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.339 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.339 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "redirect", 2024-04-25T11:47:03.339 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.339 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.339 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.339 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.339 INFO:teuthology.orchestra.run.smithi090.stdout: "redirect" 2024-04-25T11:47:03.339 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.339 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.339 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.339 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.340 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.340 INFO:teuthology.orchestra.run.smithi090.stdout: "standby_error_status_code": { 2024-04-25T11:47:03.340 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "standby_error_status_code", 2024-04-25T11:47:03.340 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.340 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.340 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.340 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "500", 2024-04-25T11:47:03.340 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "400", 2024-04-25T11:47:03.340 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "599", 2024-04-25T11:47:03.340 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.340 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.340 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.340 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.341 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.341 INFO:teuthology.orchestra.run.smithi090.stdout: "url_prefix": { 2024-04-25T11:47:03.341 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "url_prefix", 2024-04-25T11:47:03.341 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.341 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.341 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.341 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.341 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.341 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.341 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.341 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.341 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.341 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.342 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.342 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.342 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.342 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "devicehealth", 2024-04-25T11:47:03.342 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.342 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.342 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.342 INFO:teuthology.orchestra.run.smithi090.stdout: "enable_monitoring": { 2024-04-25T11:47:03.342 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "enable_monitoring", 2024-04-25T11:47:03.342 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.342 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.343 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.343 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.343 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.343 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.343 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.343 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "monitor device health metrics", 2024-04-25T11:47:03.343 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.343 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.343 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.343 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.343 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.343 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.343 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.344 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.344 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.344 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.344 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.344 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.344 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.344 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.344 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.344 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.344 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.344 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.344 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.344 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.344 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.344 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.344 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.345 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.345 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.345 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.345 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.345 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.345 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.345 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.345 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.345 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.345 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.345 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.345 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.345 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.346 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.346 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.346 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.346 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.346 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.346 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.346 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.346 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.346 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.346 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.346 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.346 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.346 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.346 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.347 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.347 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.347 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.347 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.347 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.347 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.347 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.347 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.347 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.347 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.347 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.347 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.347 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.348 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.348 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.348 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.348 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.348 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.348 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.348 INFO:teuthology.orchestra.run.smithi090.stdout: "mark_out_threshold": { 2024-04-25T11:47:03.348 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mark_out_threshold", 2024-04-25T11:47:03.348 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.348 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.348 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.348 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "2419200", 2024-04-25T11:47:03.348 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.348 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.349 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.349 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T11:47:03.349 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.349 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.349 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.349 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": { 2024-04-25T11:47:03.349 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "pool_name", 2024-04-25T11:47:03.349 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.349 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.349 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.349 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "device_health_metrics", 2024-04-25T11:47:03.349 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.349 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.350 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.350 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T11:47:03.350 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.350 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.350 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.350 INFO:teuthology.orchestra.run.smithi090.stdout: "retention_period": { 2024-04-25T11:47:03.350 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "retention_period", 2024-04-25T11:47:03.350 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.350 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.350 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.350 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "15552000", 2024-04-25T11:47:03.350 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.350 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.351 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.351 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how long to retain device health metrics", 2024-04-25T11:47:03.351 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.351 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.351 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.351 INFO:teuthology.orchestra.run.smithi090.stdout: "scrape_frequency": { 2024-04-25T11:47:03.351 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "scrape_frequency", 2024-04-25T11:47:03.351 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.351 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.351 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.351 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "86400", 2024-04-25T11:47:03.351 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.351 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.351 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.352 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T11:47:03.352 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.352 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.352 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.352 INFO:teuthology.orchestra.run.smithi090.stdout: "self_heal": { 2024-04-25T11:47:03.352 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "self_heal", 2024-04-25T11:47:03.352 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.352 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.352 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.352 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.352 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.352 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.352 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.353 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T11:47:03.353 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.353 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.353 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.353 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-25T11:47:03.353 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-25T11:47:03.353 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.353 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.353 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.353 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-25T11:47:03.353 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.353 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.353 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.353 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T11:47:03.354 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.354 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.354 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.354 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.354 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.354 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.354 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.354 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.354 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.354 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.354 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.354 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_threshold": { 2024-04-25T11:47:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_threshold", 2024-04-25T11:47:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "7257600", 2024-04-25T11:47:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T11:47:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.356 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.356 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.356 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.356 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.356 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "diskprediction_local", 2024-04-25T11:47:03.356 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.356 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.356 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.356 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.356 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.356 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.356 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.356 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.357 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.357 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.357 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.357 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.357 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.357 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.357 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.357 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.357 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.357 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.357 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.357 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.357 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.357 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.357 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.358 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.358 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.358 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.358 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.358 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.358 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.358 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.358 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.358 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.358 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.358 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.358 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.358 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.359 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.359 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.359 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.359 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.359 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.359 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.359 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.359 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.359 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.359 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.359 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.359 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.359 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.360 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.360 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.360 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.360 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.360 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.360 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.360 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.360 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.360 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.360 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.360 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.360 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.360 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.360 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.361 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.361 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.361 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.361 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.361 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.361 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.361 INFO:teuthology.orchestra.run.smithi090.stdout: "predict_interval": { 2024-04-25T11:47:03.361 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "predict_interval", 2024-04-25T11:47:03.361 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.361 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.361 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.362 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "86400", 2024-04-25T11:47:03.362 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.362 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.362 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.362 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.362 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.362 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.362 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.362 INFO:teuthology.orchestra.run.smithi090.stdout: "predictor_model": { 2024-04-25T11:47:03.362 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "predictor_model", 2024-04-25T11:47:03.362 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.362 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "prophetstor", 2024-04-25T11:47:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-25T11:47:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-25T11:47:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-25T11:47:03.364 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.364 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.364 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.364 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.364 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.364 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.364 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.364 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.364 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.364 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.364 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.364 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.364 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.364 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.365 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.365 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.365 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.365 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.365 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.365 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.365 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.365 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.365 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "influx", 2024-04-25T11:47:03.365 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": false, 2024-04-25T11:47:03.365 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "influxdb python module not found", 2024-04-25T11:47:03.365 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.365 INFO:teuthology.orchestra.run.smithi090.stdout: "batch_size": { 2024-04-25T11:47:03.366 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "batch_size", 2024-04-25T11:47:03.366 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.366 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.366 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.366 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5000", 2024-04-25T11:47:03.366 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.366 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.366 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.366 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T11:47:03.366 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.366 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.366 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.366 INFO:teuthology.orchestra.run.smithi090.stdout: "database": { 2024-04-25T11:47:03.366 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "database", 2024-04-25T11:47:03.367 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.367 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.367 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.367 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "ceph", 2024-04-25T11:47:03.367 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.367 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.367 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.367 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T11:47:03.367 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.367 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.367 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.367 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": { 2024-04-25T11:47:03.367 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "hostname", 2024-04-25T11:47:03.368 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.368 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.368 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.368 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.368 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.368 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.368 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.368 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "InfluxDB server hostname", 2024-04-25T11:47:03.368 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.368 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.368 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.368 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-25T11:47:03.368 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-25T11:47:03.368 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.369 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.369 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.369 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "30", 2024-04-25T11:47:03.369 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "5", 2024-04-25T11:47:03.369 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.369 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.369 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T11:47:03.369 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.369 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.369 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.369 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.369 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.369 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.370 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.370 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.370 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.370 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.370 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.370 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.370 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.370 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.370 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.370 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.370 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.370 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.370 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.371 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.371 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.371 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.371 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.371 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.371 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.371 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.371 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.371 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.371 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.371 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.371 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.371 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.371 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.372 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.372 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.372 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.372 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.372 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.372 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.372 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.372 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.372 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.372 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.372 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.372 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.372 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.372 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.372 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.373 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.373 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.373 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.373 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.373 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.373 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.373 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.373 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.373 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.373 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.373 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.373 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.373 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.374 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.374 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.374 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.374 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.374 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.374 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.374 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.374 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.374 INFO:teuthology.orchestra.run.smithi090.stdout: "password": { 2024-04-25T11:47:03.374 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "password", 2024-04-25T11:47:03.374 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.374 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.375 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.375 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.375 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.375 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.375 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.375 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "password of InfluxDB server user", 2024-04-25T11:47:03.375 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.375 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.375 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.375 INFO:teuthology.orchestra.run.smithi090.stdout: "port": { 2024-04-25T11:47:03.375 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "port", 2024-04-25T11:47:03.375 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.375 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.376 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.376 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8086", 2024-04-25T11:47:03.376 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.376 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.376 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.376 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "InfluxDB server port", 2024-04-25T11:47:03.376 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.376 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.376 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.376 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.376 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.376 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.376 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.377 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.377 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.377 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.377 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.377 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.377 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.377 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.377 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.377 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.377 INFO:teuthology.orchestra.run.smithi090.stdout: "ssl": { 2024-04-25T11:47:03.377 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ssl", 2024-04-25T11:47:03.377 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.377 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.377 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.378 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "false", 2024-04-25T11:47:03.378 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.378 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.378 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.378 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T11:47:03.378 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.378 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.378 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.378 INFO:teuthology.orchestra.run.smithi090.stdout: "threads": { 2024-04-25T11:47:03.378 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "threads", 2024-04-25T11:47:03.378 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.378 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.378 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.379 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5", 2024-04-25T11:47:03.379 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1", 2024-04-25T11:47:03.379 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "32", 2024-04-25T11:47:03.379 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.379 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T11:47:03.379 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.379 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.379 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.379 INFO:teuthology.orchestra.run.smithi090.stdout: "username": { 2024-04-25T11:47:03.379 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "username", 2024-04-25T11:47:03.379 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.379 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.379 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.380 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.380 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.380 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.380 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.380 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "username of InfluxDB server user", 2024-04-25T11:47:03.380 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.380 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.380 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.380 INFO:teuthology.orchestra.run.smithi090.stdout: "verify_ssl": { 2024-04-25T11:47:03.380 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "verify_ssl", 2024-04-25T11:47:03.380 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.380 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.380 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.380 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "true", 2024-04-25T11:47:03.381 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.381 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.381 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.381 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T11:47:03.381 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.381 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.381 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.381 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.381 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.381 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "insights", 2024-04-25T11:47:03.381 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.381 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.382 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.382 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.382 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.382 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.382 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.382 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.382 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.382 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.382 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.382 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.382 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.382 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.382 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.382 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.383 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.383 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.383 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.383 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.383 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.383 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.383 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.383 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.383 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.383 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.383 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.383 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.383 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.384 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.384 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.384 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.384 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.384 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.384 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.384 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.384 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.384 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.384 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.384 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.384 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.384 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.385 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.385 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.385 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.385 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.385 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.385 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.385 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.385 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.385 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.385 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.385 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.385 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.385 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.385 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.386 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.386 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.386 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.386 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.386 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.386 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.386 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.386 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.386 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.386 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.386 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.386 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.386 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.387 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.387 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.387 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.387 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.387 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.387 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.387 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.387 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.387 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.387 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.387 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.387 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.387 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.388 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.388 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.388 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "iostat", 2024-04-25T11:47:03.388 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.388 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.388 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.388 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.388 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.388 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.388 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.388 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.389 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.389 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.389 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.389 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.389 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.389 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.389 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.389 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.389 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.389 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.389 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.389 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.389 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.389 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.389 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.390 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.390 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.390 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.390 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.390 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.390 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.390 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.390 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.390 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.390 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.390 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.390 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.390 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.391 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.391 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.391 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.391 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.391 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.391 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.391 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.391 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.391 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.391 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.391 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.391 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.391 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.391 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.392 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.392 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.392 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.392 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.392 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.392 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.392 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.392 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.392 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.392 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.392 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.392 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.392 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.393 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.393 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.393 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.393 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.393 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.393 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.393 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.393 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.393 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.393 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.393 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.393 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.393 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.394 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.394 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.394 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.394 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.394 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.394 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.394 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.394 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.394 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "k8sevents", 2024-04-25T11:47:03.394 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.394 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.394 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.395 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_event_retention_days": { 2024-04-25T11:47:03.395 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ceph_event_retention_days", 2024-04-25T11:47:03.395 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.395 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.395 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.395 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "7", 2024-04-25T11:47:03.395 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.395 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.395 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.395 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T11:47:03.395 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.395 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.395 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.396 INFO:teuthology.orchestra.run.smithi090.stdout: "config_check_secs": { 2024-04-25T11:47:03.396 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "config_check_secs", 2024-04-25T11:47:03.396 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.396 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.396 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.396 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-25T11:47:03.396 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "10", 2024-04-25T11:47:03.396 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.396 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.396 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T11:47:03.396 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.396 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.396 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.396 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.397 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.397 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.397 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.397 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.397 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.397 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.397 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.397 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.397 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.397 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.397 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.397 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.397 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.397 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.398 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.398 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.398 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.398 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.398 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.398 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.398 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.398 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.398 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.398 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.398 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.398 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.398 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.399 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.399 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.399 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.399 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.399 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.399 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.399 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.399 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.399 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.399 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.399 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.399 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.399 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.399 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.400 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.400 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.400 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.400 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.400 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.400 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.400 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.400 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.400 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.400 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.400 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.400 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.400 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.400 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.401 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.401 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.401 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.401 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.401 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.401 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.401 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.401 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.401 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.401 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.401 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.401 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.401 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.402 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.402 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.402 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.402 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.402 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.402 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.402 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.402 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.402 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.402 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.402 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.402 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.403 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "localpool", 2024-04-25T11:47:03.403 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.403 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.403 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.403 INFO:teuthology.orchestra.run.smithi090.stdout: "failure_domain": { 2024-04-25T11:47:03.403 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "failure_domain", 2024-04-25T11:47:03.403 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.403 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.403 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.403 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "host", 2024-04-25T11:47:03.403 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.403 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.403 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.403 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "failure domain for any created local pool", 2024-04-25T11:47:03.404 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T11:47:03.404 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.404 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.404 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.404 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.404 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.404 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.404 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.404 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.404 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.404 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.404 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.404 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.405 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.405 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.405 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.405 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.405 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.405 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.405 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.405 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.405 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.405 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.405 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.405 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.405 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.405 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.406 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.406 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.406 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.406 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.406 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.406 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.406 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.406 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.406 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.406 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.406 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.406 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.406 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.407 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.407 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.407 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.407 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.407 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.407 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.407 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.407 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.407 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.407 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.407 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.407 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.407 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.407 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.408 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.408 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.408 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.408 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.408 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.408 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.408 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.408 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.408 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.408 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.408 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.408 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.408 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.408 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.408 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.409 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": { 2024-04-25T11:47:03.409 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "min_size", 2024-04-25T11:47:03.409 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.409 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.409 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.409 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.409 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.409 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.409 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.409 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "default min_size for any created local pool", 2024-04-25T11:47:03.409 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T11:47:03.409 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.409 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.410 INFO:teuthology.orchestra.run.smithi090.stdout: "num_rep": { 2024-04-25T11:47:03.410 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "num_rep", 2024-04-25T11:47:03.410 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.410 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.410 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.410 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "3", 2024-04-25T11:47:03.410 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.410 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.410 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.410 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "default replica count for any created local pool", 2024-04-25T11:47:03.410 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.410 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.410 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.411 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": { 2024-04-25T11:47:03.411 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "pg_num", 2024-04-25T11:47:03.411 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.411 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.411 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.411 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "128", 2024-04-25T11:47:03.411 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.411 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.411 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.411 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T11:47:03.411 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.411 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.411 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.412 INFO:teuthology.orchestra.run.smithi090.stdout: "prefix": { 2024-04-25T11:47:03.412 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "prefix", 2024-04-25T11:47:03.412 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.412 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.412 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.412 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.412 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.412 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.412 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.412 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "name prefix for any created local pool", 2024-04-25T11:47:03.412 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.412 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.412 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.413 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.413 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.413 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.413 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.413 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.413 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.413 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.413 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.413 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.413 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.413 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.413 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.413 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.414 INFO:teuthology.orchestra.run.smithi090.stdout: "subtree": { 2024-04-25T11:47:03.414 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "subtree", 2024-04-25T11:47:03.414 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.414 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.414 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.414 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "rack", 2024-04-25T11:47:03.414 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.414 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.414 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.414 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T11:47:03.414 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T11:47:03.414 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.414 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.415 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.415 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.415 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mds_autoscaler", 2024-04-25T11:47:03.415 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.415 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.415 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.415 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.415 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.415 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.415 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.415 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.415 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.415 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.416 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.416 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.416 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.416 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.416 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.416 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.416 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.416 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.416 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.416 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.416 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.416 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.416 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.417 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.417 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.417 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.417 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.417 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.417 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.417 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.417 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.417 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.417 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.417 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.417 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.417 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.417 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.418 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.418 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.418 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.418 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.418 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.418 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.418 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.418 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.418 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.418 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.418 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.418 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.418 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.418 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.419 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.419 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.419 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.419 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.419 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.419 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.419 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.419 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.419 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.419 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.419 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.419 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.420 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.420 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.420 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.420 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.420 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.420 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.420 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.420 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.420 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.420 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.420 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.420 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.420 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.420 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.421 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.421 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.421 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.421 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.421 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.421 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.421 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.421 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mirroring", 2024-04-25T11:47:03.421 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.421 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.421 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.421 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.422 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.422 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.422 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.422 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.422 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.422 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.422 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.422 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.422 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.422 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.422 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.422 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.422 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.422 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.423 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.423 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.423 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.423 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.423 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.423 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.423 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.423 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.423 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.423 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.423 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.423 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.423 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.423 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.424 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.424 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.424 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.424 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.424 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.424 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.424 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.424 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.424 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.424 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.424 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.424 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.424 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.424 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.425 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.425 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.425 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.425 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.425 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.425 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.425 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.425 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.425 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.425 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.425 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.425 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.425 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.425 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.426 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.426 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.426 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.426 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.426 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.426 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.426 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.426 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.426 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.426 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.426 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.426 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.426 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.427 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.427 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.427 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.427 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.427 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.427 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.427 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.427 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.427 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.427 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.427 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.427 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "nfs", 2024-04-25T11:47:03.427 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.428 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.428 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.428 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.428 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.428 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.428 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.428 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.428 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.428 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.428 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.428 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.428 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.428 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.428 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.429 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.429 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.429 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.429 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.429 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.429 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.429 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.429 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.429 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.429 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.429 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.429 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.429 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.430 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.430 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.430 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.430 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.430 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.430 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.430 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.430 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.430 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.430 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.430 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.430 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.430 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.431 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.431 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.431 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.431 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.431 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.431 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.431 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.431 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.431 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.431 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.431 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.431 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.431 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.431 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.432 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.432 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.432 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.432 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.432 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.432 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.432 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.432 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.432 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.432 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.432 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.432 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.432 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.432 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.433 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.433 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.433 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.433 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.433 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.433 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.433 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.433 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.433 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.433 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.433 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.433 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.433 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.433 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.434 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.434 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "orchestrator", 2024-04-25T11:47:03.434 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.434 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.434 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.434 INFO:teuthology.orchestra.run.smithi090.stdout: "fail_fs": { 2024-04-25T11:47:03.434 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "fail_fs", 2024-04-25T11:47:03.434 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.434 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.434 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.434 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.434 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.435 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.435 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.435 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T11:47:03.435 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.435 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.435 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.435 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.435 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.435 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.435 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.435 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.435 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.435 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.436 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.436 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.436 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.436 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.436 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.436 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.436 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.436 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.436 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.436 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.436 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.436 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.436 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.437 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.437 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.437 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.437 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.437 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.437 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.437 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.437 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.437 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.437 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.437 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.437 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.437 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.437 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.438 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.438 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.438 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.438 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.438 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.438 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.438 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.438 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.438 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.438 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.438 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.438 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.438 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.438 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.438 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.439 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.439 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.439 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.439 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.439 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.439 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.439 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.439 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.439 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.439 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.439 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.439 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.439 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.440 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.440 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.440 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.440 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.440 INFO:teuthology.orchestra.run.smithi090.stdout: "orchestrator": { 2024-04-25T11:47:03.440 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "orchestrator", 2024-04-25T11:47:03.440 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.440 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.440 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.440 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.440 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.440 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.440 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.440 INFO:teuthology.orchestra.run.smithi090.stdout: "cephadm", 2024-04-25T11:47:03.441 INFO:teuthology.orchestra.run.smithi090.stdout: "rook", 2024-04-25T11:47:03.441 INFO:teuthology.orchestra.run.smithi090.stdout: "test_orchestrator" 2024-04-25T11:47:03.441 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.441 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Orchestrator backend", 2024-04-25T11:47:03.441 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.441 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.441 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.441 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.441 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.441 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.441 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.441 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.442 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.442 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.442 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.442 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.442 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.442 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.442 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.442 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.442 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.442 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.442 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "osd_perf_query", 2024-04-25T11:47:03.442 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.443 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.443 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.443 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.443 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.443 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.443 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.443 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.443 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.443 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.443 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.443 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.443 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.443 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.443 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.444 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.444 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.444 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.444 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.444 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.444 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.444 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.444 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.444 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.444 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.444 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.444 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.445 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.445 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.445 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.445 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.445 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.445 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.445 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.445 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.445 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.445 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.445 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.445 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.446 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.446 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.446 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.446 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.446 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.446 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.446 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.446 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.446 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.446 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.446 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.446 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.446 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.447 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.447 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.447 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.447 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.447 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.447 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.447 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.447 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.447 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.447 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.447 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.447 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.447 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.447 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.448 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.448 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.448 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.448 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.448 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.448 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.448 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.448 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.448 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.448 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.448 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.448 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.448 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.449 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.449 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.449 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.449 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.449 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.449 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "osd_support", 2024-04-25T11:47:03.449 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.449 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.449 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.449 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.449 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.449 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.449 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.450 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.450 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.450 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.450 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.450 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.450 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.450 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.450 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.450 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.450 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.450 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.450 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.450 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.450 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.451 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.451 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.451 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.451 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.451 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.451 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.451 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.451 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.451 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.451 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.451 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.451 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.451 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.451 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.452 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.452 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.452 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.452 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.452 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.452 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.452 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.452 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.452 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.452 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.452 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.452 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.452 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.452 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.453 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.453 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.453 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.453 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.453 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.453 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.453 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.453 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.453 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.453 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.453 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.453 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.453 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.454 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.454 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.454 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.454 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.454 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.454 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.454 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.454 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.454 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.454 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.454 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.454 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.454 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.455 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.455 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.455 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.455 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.455 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.455 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.455 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.455 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.455 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.455 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "pg_autoscaler", 2024-04-25T11:47:03.455 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.455 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.456 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.456 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.456 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.456 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.456 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.456 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.456 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.456 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.456 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.456 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.456 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.456 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.456 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.456 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.457 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.457 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.457 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.457 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.457 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.457 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.457 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.457 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.457 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.457 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.457 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.457 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.457 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.458 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.458 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.458 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.458 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.458 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.458 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.458 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.458 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.458 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.458 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.458 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.458 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.459 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.459 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.459 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.459 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.459 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.459 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.459 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.459 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.459 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.459 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.459 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.459 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.459 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.459 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.460 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.460 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.460 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.460 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.460 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.460 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.460 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.460 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.460 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.460 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.460 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.460 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.460 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.461 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.461 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-25T11:47:03.461 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-25T11:47:03.461 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.461 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.461 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.461 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-25T11:47:03.461 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.461 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.461 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.461 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.461 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.461 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.462 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.462 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.462 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.462 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.462 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.462 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.462 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.462 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.462 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.462 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.462 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.462 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.462 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.462 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.463 INFO:teuthology.orchestra.run.smithi090.stdout: "threshold": { 2024-04-25T11:47:03.463 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "threshold", 2024-04-25T11:47:03.463 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-25T11:47:03.463 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.463 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.463 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "3.0", 2024-04-25T11:47:03.463 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1.0", 2024-04-25T11:47:03.463 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.463 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.463 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "scaling threshold", 2024-04-25T11:47:03.463 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T11:47:03.463 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.463 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.464 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.464 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.464 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "progress", 2024-04-25T11:47:03.464 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.464 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.464 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.464 INFO:teuthology.orchestra.run.smithi090.stdout: "allow_pg_recovery_event": { 2024-04-25T11:47:03.464 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "allow_pg_recovery_event", 2024-04-25T11:47:03.464 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.464 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.464 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.464 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.464 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.465 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.465 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.465 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T11:47:03.465 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.465 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.465 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.465 INFO:teuthology.orchestra.run.smithi090.stdout: "enabled": { 2024-04-25T11:47:03.465 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "enabled", 2024-04-25T11:47:03.465 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.465 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.465 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.465 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.465 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.465 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.466 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.466 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.466 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.466 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.466 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.466 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.466 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.466 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.466 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.466 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.466 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.466 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.466 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.467 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.467 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.467 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.467 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.467 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.467 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.467 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.467 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.467 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.467 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.467 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.467 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.467 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.467 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.468 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.468 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.468 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.468 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.468 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.468 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.468 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.468 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.468 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.468 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.468 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.468 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.468 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.469 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.469 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.469 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.469 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.469 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.469 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.469 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.469 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.469 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.469 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.469 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.469 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.469 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.469 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.470 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.470 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.470 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.470 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.470 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.470 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.470 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.470 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.470 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.470 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.470 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.470 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.470 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.471 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.471 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.471 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.471 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.471 INFO:teuthology.orchestra.run.smithi090.stdout: "max_completed_events": { 2024-04-25T11:47:03.471 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_completed_events", 2024-04-25T11:47:03.471 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.471 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.471 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.471 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "50", 2024-04-25T11:47:03.471 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.471 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.471 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.472 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "number of past completed events to remember", 2024-04-25T11:47:03.472 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.472 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.472 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.472 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-25T11:47:03.472 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-25T11:47:03.472 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.472 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.472 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.472 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5", 2024-04-25T11:47:03.472 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.472 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.472 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.472 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how long the module is going to sleep", 2024-04-25T11:47:03.473 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.473 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.473 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.473 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.473 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.473 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.473 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.473 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.473 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.473 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.473 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.473 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.473 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.474 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.474 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.474 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.474 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.474 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.474 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "prometheus", 2024-04-25T11:47:03.474 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.474 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.474 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.474 INFO:teuthology.orchestra.run.smithi090.stdout: "cache": { 2024-04-25T11:47:03.474 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cache", 2024-04-25T11:47:03.475 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.475 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.475 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.475 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.475 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.475 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.475 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.475 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.475 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.475 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.475 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.475 INFO:teuthology.orchestra.run.smithi090.stdout: "exclude_perf_counters": { 2024-04-25T11:47:03.475 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "exclude_perf_counters", 2024-04-25T11:47:03.475 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.476 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.476 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.476 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.476 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.476 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.476 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.476 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T11:47:03.476 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T11:47:03.476 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.476 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.476 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.476 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.476 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.477 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.477 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.477 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.477 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.477 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.477 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.477 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.477 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.477 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.477 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.477 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.477 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.477 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.477 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.478 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.478 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.478 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.478 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.478 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.478 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.478 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.478 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.478 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.478 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.478 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.478 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.478 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.478 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.479 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.479 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.479 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.479 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.479 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.479 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.479 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.479 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.479 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.479 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.479 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.479 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.479 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.479 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.480 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.480 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.480 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.480 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.480 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.480 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.480 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.480 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.480 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.480 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.480 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.480 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.480 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.481 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.481 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.481 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.481 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.481 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.481 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.481 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.481 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.481 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_stats_pools": { 2024-04-25T11:47:03.481 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rbd_stats_pools", 2024-04-25T11:47:03.481 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.481 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.481 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.482 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.482 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.482 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.482 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.482 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.482 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.482 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.482 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.482 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T11:47:03.482 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T11:47:03.482 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.482 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.482 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.483 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "300", 2024-04-25T11:47:03.483 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.483 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.483 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.483 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.483 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.483 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.483 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.483 INFO:teuthology.orchestra.run.smithi090.stdout: "scrape_interval": { 2024-04-25T11:47:03.483 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "scrape_interval", 2024-04-25T11:47:03.483 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-25T11:47:03.483 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.483 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.484 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "15.0", 2024-04-25T11:47:03.484 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.484 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.484 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.484 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.484 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.484 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.484 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.484 INFO:teuthology.orchestra.run.smithi090.stdout: "server_addr": { 2024-04-25T11:47:03.484 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_addr", 2024-04-25T11:47:03.484 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.484 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.484 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.484 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "::", 2024-04-25T11:47:03.485 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.485 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.485 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.485 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T11:47:03.485 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.485 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.485 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.485 INFO:teuthology.orchestra.run.smithi090.stdout: "server_port": { 2024-04-25T11:47:03.485 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_port", 2024-04-25T11:47:03.485 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.485 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.485 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.485 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "9283", 2024-04-25T11:47:03.485 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.486 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.486 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.486 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T11:47:03.486 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.486 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.486 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.486 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.486 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.486 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.486 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.486 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.486 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.486 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.487 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.487 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.487 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.487 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.487 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.487 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.487 INFO:teuthology.orchestra.run.smithi090.stdout: "stale_cache_strategy": { 2024-04-25T11:47:03.487 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "stale_cache_strategy", 2024-04-25T11:47:03.487 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.487 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.487 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.487 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "log", 2024-04-25T11:47:03.487 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.487 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.488 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.488 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.488 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.488 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.488 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.488 INFO:teuthology.orchestra.run.smithi090.stdout: "standby_behaviour": { 2024-04-25T11:47:03.488 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "standby_behaviour", 2024-04-25T11:47:03.488 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.488 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.488 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.488 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "default", 2024-04-25T11:47:03.488 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.488 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.488 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.488 INFO:teuthology.orchestra.run.smithi090.stdout: "default", 2024-04-25T11:47:03.489 INFO:teuthology.orchestra.run.smithi090.stdout: "error" 2024-04-25T11:47:03.489 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.489 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.489 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.489 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.489 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.489 INFO:teuthology.orchestra.run.smithi090.stdout: "standby_error_status_code": { 2024-04-25T11:47:03.489 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "standby_error_status_code", 2024-04-25T11:47:03.489 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.489 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.489 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.489 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "500", 2024-04-25T11:47:03.489 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "400", 2024-04-25T11:47:03.490 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "599", 2024-04-25T11:47:03.490 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.490 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.490 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.490 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.490 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.490 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.490 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.490 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rbd_support", 2024-04-25T11:47:03.490 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.490 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.490 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.491 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.491 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.491 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.491 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.491 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.491 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.491 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.491 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.491 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.491 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.491 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.491 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.491 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.491 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.492 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.492 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.492 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.492 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.492 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.492 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.492 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.492 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.492 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.492 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.492 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.492 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.492 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.492 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.493 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.493 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.493 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.493 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.493 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.493 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.493 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.493 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.493 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.493 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.493 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.493 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.493 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.494 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.494 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.494 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.494 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.494 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.494 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.494 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.494 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.494 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.494 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.494 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.494 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.494 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.494 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.495 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.495 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.495 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.495 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.495 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.495 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.495 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.495 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.495 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.495 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.495 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.495 INFO:teuthology.orchestra.run.smithi090.stdout: "max_concurrent_snap_create": { 2024-04-25T11:47:03.495 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_concurrent_snap_create", 2024-04-25T11:47:03.496 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.496 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.496 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.496 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-25T11:47:03.496 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.496 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.496 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.496 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.496 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.496 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.496 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.496 INFO:teuthology.orchestra.run.smithi090.stdout: "mirror_snapshot_schedule": { 2024-04-25T11:47:03.496 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T11:47:03.496 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.497 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.497 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.497 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.497 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.497 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.497 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.497 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.497 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.497 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.497 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.497 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.497 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.497 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.497 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.498 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.498 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.498 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.498 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.498 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.498 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.498 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.498 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.498 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.498 INFO:teuthology.orchestra.run.smithi090.stdout: "trash_purge_schedule": { 2024-04-25T11:47:03.498 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "trash_purge_schedule", 2024-04-25T11:47:03.498 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.498 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.498 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.499 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.499 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.499 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.499 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.499 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.499 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.499 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.499 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.499 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.499 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.499 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "restful", 2024-04-25T11:47:03.499 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.499 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.500 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.500 INFO:teuthology.orchestra.run.smithi090.stdout: "enable_auth": { 2024-04-25T11:47:03.500 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "enable_auth", 2024-04-25T11:47:03.500 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.500 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.500 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.500 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.500 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.500 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.500 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.500 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.500 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.500 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.500 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.501 INFO:teuthology.orchestra.run.smithi090.stdout: "key_file": { 2024-04-25T11:47:03.501 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "key_file", 2024-04-25T11:47:03.501 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.501 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.501 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.501 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.501 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.501 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.501 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.501 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.501 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.501 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.501 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.501 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.502 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.502 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.502 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.502 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.502 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.502 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.502 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.502 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.502 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.502 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.502 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.502 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.502 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.502 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.502 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.503 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.503 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.503 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.503 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.503 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.503 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.503 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.503 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.503 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.503 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.503 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.503 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.503 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.503 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.504 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.504 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.504 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.504 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.504 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.504 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.504 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.504 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.504 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.504 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.504 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.504 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.505 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.505 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.505 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.505 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.505 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.505 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.505 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.505 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.505 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.505 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.505 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.505 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.505 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.505 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.506 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.506 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.506 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.506 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.506 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.506 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.506 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.506 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.506 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.506 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.506 INFO:teuthology.orchestra.run.smithi090.stdout: "server_addr": { 2024-04-25T11:47:03.506 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_addr", 2024-04-25T11:47:03.506 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.507 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.507 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.507 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.507 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.507 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.507 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.507 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.507 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.507 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.507 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.507 INFO:teuthology.orchestra.run.smithi090.stdout: "server_port": { 2024-04-25T11:47:03.507 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_port", 2024-04-25T11:47:03.507 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.508 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.508 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.508 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.508 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.508 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.508 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.508 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.508 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.508 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.508 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.508 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.508 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.508 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.509 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.509 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.509 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.509 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.509 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.509 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.509 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.509 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.509 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.509 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.509 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.509 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.510 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rgw", 2024-04-25T11:47:03.510 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.510 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.510 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.510 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.510 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.510 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.510 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.510 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.510 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.510 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.510 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.510 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.510 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.511 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.511 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.511 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.511 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.511 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.511 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.511 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.511 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.511 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.511 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.511 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.511 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.512 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.512 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.512 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.512 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.512 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.512 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.512 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.512 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.512 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.512 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.512 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.512 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.512 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.513 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.513 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.513 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.513 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.513 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.513 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.513 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.513 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.513 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.513 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.513 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.513 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.513 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.513 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.514 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.514 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.514 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.514 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.514 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.514 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.514 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.514 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.514 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.514 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.514 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.514 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.515 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.515 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.515 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.515 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.515 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.515 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.515 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.515 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.515 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.515 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.515 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.515 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.515 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.515 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.516 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.516 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.516 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.516 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.516 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.516 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.516 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rook", 2024-04-25T11:47:03.516 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.516 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.516 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.516 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.516 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.517 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.517 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.517 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.517 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.517 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.517 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.517 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.517 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.517 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.517 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.517 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.517 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.517 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.517 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.517 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.518 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.518 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.518 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.518 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.518 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.518 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.518 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.518 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.518 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.518 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.518 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.518 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.518 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.519 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.519 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.519 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.519 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.519 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.519 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.519 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.519 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.519 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.519 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.519 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.519 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.519 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.519 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.520 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.520 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.520 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.520 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.520 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.520 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.520 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.520 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.520 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.520 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.520 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.520 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.520 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.520 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.521 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.521 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.521 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.521 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.521 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.521 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.521 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.521 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.521 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.521 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.521 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.521 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.521 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.522 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.522 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.522 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.522 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.522 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.522 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.522 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.522 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.522 INFO:teuthology.orchestra.run.smithi090.stdout: "storage_class": { 2024-04-25T11:47:03.522 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "storage_class", 2024-04-25T11:47:03.522 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.522 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.523 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.523 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "local", 2024-04-25T11:47:03.523 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.523 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.523 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.523 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T11:47:03.523 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.523 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.523 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.523 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.523 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.523 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "selftest", 2024-04-25T11:47:03.523 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.524 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.524 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.524 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.524 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.524 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.524 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.524 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.524 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.524 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.524 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.524 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.524 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.524 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.524 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.524 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.526 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.526 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.526 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.526 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.526 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.526 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.526 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.526 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.526 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.526 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.526 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.526 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.526 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.528 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.528 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.528 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.528 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.528 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.528 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.528 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.528 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.528 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.528 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.528 INFO:teuthology.orchestra.run.smithi090.stdout: "roption1": { 2024-04-25T11:47:03.528 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "roption1", 2024-04-25T11:47:03.528 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.528 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: "roption2": { 2024-04-25T11:47:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "roption2", 2024-04-25T11:47:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "xyz", 2024-04-25T11:47:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.530 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.530 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.530 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.530 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.530 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.530 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.530 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption1": { 2024-04-25T11:47:03.530 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption1", 2024-04-25T11:47:03.530 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.530 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.530 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.530 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.530 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.531 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.531 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.531 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.531 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.531 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.531 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.531 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption2": { 2024-04-25T11:47:03.531 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption2", 2024-04-25T11:47:03.531 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.531 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.531 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.531 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.531 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.531 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.532 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.532 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.532 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.532 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.532 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.532 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption3": { 2024-04-25T11:47:03.532 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption3", 2024-04-25T11:47:03.532 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-25T11:47:03.532 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.532 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.532 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.532 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption4": { 2024-04-25T11:47:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption4", 2024-04-25T11:47:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.534 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.534 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.534 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.534 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.534 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption5": { 2024-04-25T11:47:03.534 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption5", 2024-04-25T11:47:03.534 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.534 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.534 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.534 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.534 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.534 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.534 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption6": { 2024-04-25T11:47:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption6", 2024-04-25T11:47:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.536 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.536 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.536 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.536 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.536 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption7": { 2024-04-25T11:47:03.536 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption7", 2024-04-25T11:47:03.536 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.536 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.536 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.536 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.536 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1", 2024-04-25T11:47:03.536 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "42", 2024-04-25T11:47:03.536 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.537 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.537 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.537 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.537 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.537 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.537 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.537 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.537 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.537 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.537 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.537 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.537 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.537 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.537 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.538 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.538 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.538 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.538 INFO:teuthology.orchestra.run.smithi090.stdout: "testkey": { 2024-04-25T11:47:03.538 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "testkey", 2024-04-25T11:47:03.538 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.538 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.538 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.538 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.538 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.538 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.538 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.538 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.539 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.539 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.539 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.539 INFO:teuthology.orchestra.run.smithi090.stdout: "testlkey": { 2024-04-25T11:47:03.539 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "testlkey", 2024-04-25T11:47:03.539 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.539 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.539 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.539 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.539 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.539 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.539 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.539 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.539 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.540 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.540 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.540 INFO:teuthology.orchestra.run.smithi090.stdout: "testnewline": { 2024-04-25T11:47:03.540 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "testnewline", 2024-04-25T11:47:03.540 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.540 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.540 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.540 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.540 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.540 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.540 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.540 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.540 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.540 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.541 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.541 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.541 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.541 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "snap_schedule", 2024-04-25T11:47:03.541 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.541 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.541 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.541 INFO:teuthology.orchestra.run.smithi090.stdout: "allow_m_granularity": { 2024-04-25T11:47:03.541 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "allow_m_granularity", 2024-04-25T11:47:03.541 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.541 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.541 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.541 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.542 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.542 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.542 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.542 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T11:47:03.542 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.542 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.542 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.542 INFO:teuthology.orchestra.run.smithi090.stdout: "dump_on_update": { 2024-04-25T11:47:03.542 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "dump_on_update", 2024-04-25T11:47:03.542 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.542 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.542 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.542 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.543 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.543 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.543 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.543 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "dump database to debug log on update", 2024-04-25T11:47:03.543 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.543 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.543 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.543 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.543 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.543 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.543 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.543 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.543 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.544 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.544 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.544 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.544 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.544 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.544 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.544 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.544 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.544 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.544 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.544 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.544 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.544 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.544 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.545 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.545 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.545 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.545 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.545 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.545 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.545 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.545 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.545 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.545 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.545 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.545 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.545 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.546 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.546 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.546 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.546 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.546 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.546 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.546 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.546 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.546 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.546 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.546 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.546 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.546 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.546 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.547 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.547 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.547 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.547 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.547 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.547 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.547 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.547 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.547 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.547 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.547 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.547 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.547 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.547 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.547 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.548 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.548 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.548 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.548 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.548 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.548 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.548 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.548 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.548 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.548 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.548 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.548 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.548 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.548 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.549 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.549 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.549 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.549 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.549 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.549 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "stats", 2024-04-25T11:47:03.549 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.549 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.549 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.549 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.549 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.549 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.549 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.549 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.550 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.550 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.550 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.550 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.550 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.550 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.550 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.550 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.550 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.550 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.550 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.550 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.550 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.550 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.550 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.551 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.551 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.551 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.551 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.551 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.551 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.551 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.551 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.551 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.551 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.551 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.551 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.551 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.552 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.552 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.552 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.552 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.552 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.552 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.552 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.552 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.552 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.552 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.552 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.552 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.552 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.552 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.553 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.553 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.553 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.553 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.553 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.553 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.553 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.553 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.553 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.553 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.553 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.553 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.553 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.554 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.554 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.554 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.554 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.554 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.554 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.554 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.554 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.554 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.554 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.554 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.554 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.554 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.555 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.555 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.555 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.555 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.555 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.555 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.555 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.555 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.555 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "status", 2024-04-25T11:47:03.555 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.555 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.555 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.555 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.556 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.556 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.556 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.556 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.556 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.556 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.556 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.556 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.556 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.556 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.556 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.556 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.556 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.556 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.556 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.557 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.557 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.557 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.557 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.557 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.557 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.557 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.557 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.557 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.557 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.557 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.557 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.557 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.557 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.558 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.558 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.558 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.558 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.558 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.558 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.558 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.558 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.558 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.558 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.558 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.558 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.559 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.559 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.559 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.559 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.559 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.559 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.559 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.559 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.559 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.559 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.559 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.559 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.559 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.559 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.559 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.560 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.560 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.560 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.560 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.560 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.560 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.560 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.560 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.560 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.560 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.560 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.560 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.560 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.560 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.560 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.561 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.561 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.561 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.561 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.561 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.561 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.561 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.561 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.561 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.561 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "telegraf", 2024-04-25T11:47:03.561 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.561 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.562 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.562 INFO:teuthology.orchestra.run.smithi090.stdout: "address": { 2024-04-25T11:47:03.562 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "address", 2024-04-25T11:47:03.562 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.562 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.562 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.562 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T11:47:03.562 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.562 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.562 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.562 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.562 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.562 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.562 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.563 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-25T11:47:03.563 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-25T11:47:03.563 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.563 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.563 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.563 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "15", 2024-04-25T11:47:03.563 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.563 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.563 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.563 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.563 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.563 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.563 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.564 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.564 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.564 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.564 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.564 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.564 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.564 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.564 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.564 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.564 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.564 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.564 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.564 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.564 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.565 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.565 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.565 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.565 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.565 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.565 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.565 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.565 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.565 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.565 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.565 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.565 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.565 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.565 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.566 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.566 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.566 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.566 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.566 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.566 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.566 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.566 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.566 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.566 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.566 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.566 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.567 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.567 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.567 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.567 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.567 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.567 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.567 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.567 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.567 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.567 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.567 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.567 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.567 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.567 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.568 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.568 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.568 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.568 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.568 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.568 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.568 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.568 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.568 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.568 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.568 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.568 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.569 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.569 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.569 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.569 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.569 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.569 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.569 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.569 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.569 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.569 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.569 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.569 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.569 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.569 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.569 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.570 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "telemetry", 2024-04-25T11:47:03.570 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.570 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.570 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.570 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_basic": { 2024-04-25T11:47:03.570 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_basic", 2024-04-25T11:47:03.570 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.570 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.570 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.570 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.570 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.571 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.571 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.571 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T11:47:03.571 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.571 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.571 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.571 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_crash": { 2024-04-25T11:47:03.571 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_crash", 2024-04-25T11:47:03.571 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.571 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.571 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.571 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.571 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.572 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.572 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.572 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T11:47:03.572 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.572 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.572 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.572 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_device": { 2024-04-25T11:47:03.572 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_device", 2024-04-25T11:47:03.572 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.572 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.572 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.572 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.572 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.572 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.573 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.573 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T11:47:03.573 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.573 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.573 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.573 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_ident": { 2024-04-25T11:47:03.573 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_ident", 2024-04-25T11:47:03.573 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.573 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.573 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.573 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.573 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.573 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.574 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.574 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T11:47:03.574 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.574 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.574 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.574 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_perf": { 2024-04-25T11:47:03.574 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_perf", 2024-04-25T11:47:03.574 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.574 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.574 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.574 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.574 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.575 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.575 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.575 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T11:47:03.575 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.575 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.575 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.575 INFO:teuthology.orchestra.run.smithi090.stdout: "contact": { 2024-04-25T11:47:03.575 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "contact", 2024-04-25T11:47:03.575 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.575 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.575 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.575 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.575 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.576 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.576 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.576 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.576 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.576 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.576 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.576 INFO:teuthology.orchestra.run.smithi090.stdout: "description": { 2024-04-25T11:47:03.576 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "description", 2024-04-25T11:47:03.576 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.576 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.576 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.576 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.576 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.576 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.577 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.577 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.577 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.577 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.577 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.577 INFO:teuthology.orchestra.run.smithi090.stdout: "device_url": { 2024-04-25T11:47:03.577 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "device_url", 2024-04-25T11:47:03.577 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.577 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.577 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.577 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T11:47:03.577 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.577 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.578 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.578 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.578 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.578 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.578 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.578 INFO:teuthology.orchestra.run.smithi090.stdout: "enabled": { 2024-04-25T11:47:03.578 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "enabled", 2024-04-25T11:47:03.578 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.578 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.578 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.578 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.578 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.579 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.579 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.579 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.579 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.579 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.579 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.579 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-25T11:47:03.579 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-25T11:47:03.579 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.579 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.579 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.579 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "24", 2024-04-25T11:47:03.579 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "8", 2024-04-25T11:47:03.579 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.580 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.580 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.580 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.580 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.580 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.580 INFO:teuthology.orchestra.run.smithi090.stdout: "last_opt_revision": { 2024-04-25T11:47:03.580 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "last_opt_revision", 2024-04-25T11:47:03.580 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.580 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.580 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.580 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "1", 2024-04-25T11:47:03.580 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.580 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.581 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.581 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.581 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.581 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.581 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.581 INFO:teuthology.orchestra.run.smithi090.stdout: "leaderboard": { 2024-04-25T11:47:03.581 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "leaderboard", 2024-04-25T11:47:03.581 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.581 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.581 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.581 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.581 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.581 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.582 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.582 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.582 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.582 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.582 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.582 INFO:teuthology.orchestra.run.smithi090.stdout: "leaderboard_description": { 2024-04-25T11:47:03.582 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "leaderboard_description", 2024-04-25T11:47:03.582 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.582 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.582 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.582 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.582 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.582 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.583 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.583 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.583 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.583 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.583 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.583 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.583 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.583 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.583 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.583 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.583 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.583 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.583 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.583 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.583 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.584 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.584 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.584 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.584 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.584 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.584 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.584 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.584 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.584 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.584 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.584 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.584 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.584 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.584 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.585 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.585 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.585 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.585 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.585 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.585 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.585 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.585 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.585 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.585 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.585 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.585 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.586 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.586 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.586 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.586 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.586 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.586 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.586 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.586 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.586 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.586 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.586 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.586 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.586 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.586 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.587 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.587 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.587 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.587 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.587 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.587 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.587 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.587 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.587 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.587 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.587 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.587 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.587 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.587 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.588 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.588 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.588 INFO:teuthology.orchestra.run.smithi090.stdout: "organization": { 2024-04-25T11:47:03.588 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "organization", 2024-04-25T11:47:03.588 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.588 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.588 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.588 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.588 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.588 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.588 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.588 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.588 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.589 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.589 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.589 INFO:teuthology.orchestra.run.smithi090.stdout: "proxy": { 2024-04-25T11:47:03.589 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "proxy", 2024-04-25T11:47:03.589 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.589 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.589 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.589 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.589 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.589 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.589 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.589 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.589 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.590 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.590 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.590 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.590 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.590 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.590 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.590 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.590 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.590 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.590 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.590 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.590 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.590 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.591 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.591 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.591 INFO:teuthology.orchestra.run.smithi090.stdout: "url": { 2024-04-25T11:47:03.591 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "url", 2024-04-25T11:47:03.591 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.591 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.591 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.591 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T11:47:03.591 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.591 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.591 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.591 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.591 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.591 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.592 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.592 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.592 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.592 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "test_orchestrator", 2024-04-25T11:47:03.592 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.592 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.592 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.592 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.592 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.592 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.592 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.592 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.593 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.593 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.593 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.593 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.593 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.593 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.593 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.593 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.593 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.593 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.593 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.593 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.593 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.593 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.593 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.594 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.594 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.594 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.594 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.594 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.594 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.594 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.594 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.594 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.594 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.594 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.594 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.595 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.595 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.595 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.595 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.595 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.595 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.595 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.595 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.595 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.595 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.595 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.595 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.595 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.595 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.596 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.596 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.596 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.596 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.596 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.596 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.596 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.596 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.596 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.596 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.596 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.596 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.596 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.596 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.597 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.597 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.597 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.597 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.597 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.597 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.597 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.597 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.597 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.597 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.597 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.597 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.597 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.597 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.598 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.598 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.598 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.598 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.598 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.598 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.598 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.598 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "volumes", 2024-04-25T11:47:03.598 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.598 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.598 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.600 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.600 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.600 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.600 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.600 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.600 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.600 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.600 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.600 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.600 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.600 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.600 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.600 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.601 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.601 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.601 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.601 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.601 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.601 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.601 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.601 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.601 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.601 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.601 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.601 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.601 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.602 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.602 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.602 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.602 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.602 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.602 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.602 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.602 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.602 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.602 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.602 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.602 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.603 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.603 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.603 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.603 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.603 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.603 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.603 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.603 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.603 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.603 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.603 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.603 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.603 INFO:teuthology.orchestra.run.smithi090.stdout: "max_concurrent_clones": { 2024-04-25T11:47:03.603 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_concurrent_clones", 2024-04-25T11:47:03.604 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.604 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.604 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.604 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "4", 2024-04-25T11:47:03.604 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.604 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.604 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.604 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T11:47:03.604 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.604 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.604 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.604 INFO:teuthology.orchestra.run.smithi090.stdout: "periodic_async_work": { 2024-04-25T11:47:03.604 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "periodic_async_work", 2024-04-25T11:47:03.605 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.605 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.605 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.605 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.605 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.605 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.605 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.605 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Periodically check for async work", 2024-04-25T11:47:03.605 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.605 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.605 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.605 INFO:teuthology.orchestra.run.smithi090.stdout: "snapshot_clone_delay": { 2024-04-25T11:47:03.605 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "snapshot_clone_delay", 2024-04-25T11:47:03.605 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.606 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.606 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.606 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.606 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.606 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.606 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.606 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T11:47:03.606 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.606 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.606 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.606 INFO:teuthology.orchestra.run.smithi090.stdout: "snapshot_clone_no_wait": { 2024-04-25T11:47:03.606 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T11:47:03.607 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.607 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.607 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.607 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.607 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.607 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.607 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.607 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T11:47:03.607 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.607 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.607 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.607 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.607 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.607 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.608 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.608 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.608 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.608 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.608 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.608 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.608 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.608 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.608 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.608 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.608 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.608 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.608 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "zabbix", 2024-04-25T11:47:03.609 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.609 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.609 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.609 INFO:teuthology.orchestra.run.smithi090.stdout: "discovery_interval": { 2024-04-25T11:47:03.609 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "discovery_interval", 2024-04-25T11:47:03.609 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-25T11:47:03.609 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.609 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.609 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "100", 2024-04-25T11:47:03.609 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.609 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.609 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.609 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.609 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.609 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.610 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.610 INFO:teuthology.orchestra.run.smithi090.stdout: "identifier": { 2024-04-25T11:47:03.610 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "identifier", 2024-04-25T11:47:03.610 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.610 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.610 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.610 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.610 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.610 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.610 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.610 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.610 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.611 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.611 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.611 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-25T11:47:03.611 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-25T11:47:03.611 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.611 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.611 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.611 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-25T11:47:03.611 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.611 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.611 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.611 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.611 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.612 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.612 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.612 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.612 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.612 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.612 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.612 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.612 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.612 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.612 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.612 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.612 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.612 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.612 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.613 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.613 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.613 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.613 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.613 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.613 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.613 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.613 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.613 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.613 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.613 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.613 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.613 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.614 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.614 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.614 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.614 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.614 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.614 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.614 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.614 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.614 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.614 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.614 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.614 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.614 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.615 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.615 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.615 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.615 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.615 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.615 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.615 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.615 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.615 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.615 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.615 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.615 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.615 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.615 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.615 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.616 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.616 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.616 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.616 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.616 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.616 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.616 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.616 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.616 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.616 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.616 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.616 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.616 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.617 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.617 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.617 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.617 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.617 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.617 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.617 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.617 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.617 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.617 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.617 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.617 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.617 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.618 INFO:teuthology.orchestra.run.smithi090.stdout: "zabbix_host": { 2024-04-25T11:47:03.618 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "zabbix_host", 2024-04-25T11:47:03.618 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.618 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.618 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.618 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.618 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.618 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.618 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.618 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.618 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.618 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.618 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.619 INFO:teuthology.orchestra.run.smithi090.stdout: "zabbix_port": { 2024-04-25T11:47:03.619 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "zabbix_port", 2024-04-25T11:47:03.619 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.619 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.619 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.619 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10051", 2024-04-25T11:47:03.619 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.619 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.619 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.619 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.619 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.619 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.619 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.619 INFO:teuthology.orchestra.run.smithi090.stdout: "zabbix_sender": { 2024-04-25T11:47:03.620 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "zabbix_sender", 2024-04-25T11:47:03.620 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.620 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.620 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.620 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T11:47:03.620 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.620 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.620 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.620 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.620 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.620 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.620 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.620 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.620 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.620 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.621 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-25T11:47:03.621 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.621 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.621 INFO:teuthology.orchestra.run.smithi090.stdout: "modules": [ 2024-04-25T11:47:03.621 INFO:teuthology.orchestra.run.smithi090.stdout: "cephadm", 2024-04-25T11:47:03.621 INFO:teuthology.orchestra.run.smithi090.stdout: "dashboard", 2024-04-25T11:47:03.621 INFO:teuthology.orchestra.run.smithi090.stdout: "iostat", 2024-04-25T11:47:03.621 INFO:teuthology.orchestra.run.smithi090.stdout: "nfs", 2024-04-25T11:47:03.621 INFO:teuthology.orchestra.run.smithi090.stdout: "restful", 2024-04-25T11:47:03.621 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_schedule" 2024-04-25T11:47:03.621 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.621 INFO:teuthology.orchestra.run.smithi090.stdout: "available_modules": [ 2024-04-25T11:47:03.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.621 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "alerts", 2024-04-25T11:47:03.621 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.622 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.622 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.622 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-25T11:47:03.622 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-25T11:47:03.622 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.622 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.622 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.622 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-25T11:47:03.622 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.622 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.622 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.622 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T11:47:03.622 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.622 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.625 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.625 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.625 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.625 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.625 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.625 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.625 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.625 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.625 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.625 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.626 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.626 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.626 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.626 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.626 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.626 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.626 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.626 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.626 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.626 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.626 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.626 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.627 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.627 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.627 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.627 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.627 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.627 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.627 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.627 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.627 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.627 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.627 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.627 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.627 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.627 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.628 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.628 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.628 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_destination": { 2024-04-25T11:47:03.628 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_destination", 2024-04-25T11:47:03.628 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.628 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.628 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.628 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.628 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.628 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.628 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.628 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Email address to send alerts to", 2024-04-25T11:47:03.628 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.628 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.628 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.629 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_from_name": { 2024-04-25T11:47:03.629 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_from_name", 2024-04-25T11:47:03.629 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.629 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.629 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.629 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "Ceph", 2024-04-25T11:47:03.629 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.629 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.629 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.629 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Email From: name", 2024-04-25T11:47:03.629 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.629 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.629 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.629 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_host": { 2024-04-25T11:47:03.630 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_host", 2024-04-25T11:47:03.630 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.630 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.630 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.630 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.630 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.630 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.630 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.630 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "SMTP server", 2024-04-25T11:47:03.630 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.630 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.630 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.630 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_password": { 2024-04-25T11:47:03.631 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_password", 2024-04-25T11:47:03.631 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.631 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.631 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.631 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.631 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.631 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.631 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.631 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Password to authenticate with", 2024-04-25T11:47:03.631 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.631 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.631 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.631 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_port": { 2024-04-25T11:47:03.631 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_port", 2024-04-25T11:47:03.632 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.632 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.632 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.632 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "465", 2024-04-25T11:47:03.632 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.632 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.632 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.632 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "SMTP port", 2024-04-25T11:47:03.632 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.632 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.632 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.632 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_sender": { 2024-04-25T11:47:03.632 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_sender", 2024-04-25T11:47:03.633 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.633 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.633 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.633 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.633 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.633 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.633 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.633 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "SMTP envelope sender", 2024-04-25T11:47:03.633 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.633 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.633 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.633 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_ssl": { 2024-04-25T11:47:03.633 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_ssl", 2024-04-25T11:47:03.634 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.634 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.634 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.634 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.634 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.634 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.634 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.634 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T11:47:03.634 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.634 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.634 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.634 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_user": { 2024-04-25T11:47:03.635 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_user", 2024-04-25T11:47:03.635 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.635 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.635 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.635 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.635 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.635 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.635 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.635 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "User to authenticate as", 2024-04-25T11:47:03.635 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.635 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.635 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.635 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.636 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.636 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.636 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.636 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.636 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.636 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.636 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.636 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.636 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.636 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.636 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.636 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.636 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.636 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.637 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "balancer", 2024-04-25T11:47:03.637 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.637 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.637 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.637 INFO:teuthology.orchestra.run.smithi090.stdout: "active": { 2024-04-25T11:47:03.637 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "active", 2024-04-25T11:47:03.637 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.637 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.637 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.637 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.637 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.637 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.638 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.638 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T11:47:03.638 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.638 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.638 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.638 INFO:teuthology.orchestra.run.smithi090.stdout: "begin_time": { 2024-04-25T11:47:03.638 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "begin_time", 2024-04-25T11:47:03.638 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.638 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.638 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.638 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0000", 2024-04-25T11:47:03.638 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.638 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.638 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.639 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T11:47:03.639 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T11:47:03.639 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.639 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.639 INFO:teuthology.orchestra.run.smithi090.stdout: "begin_weekday": { 2024-04-25T11:47:03.639 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "begin_weekday", 2024-04-25T11:47:03.639 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-25T11:47:03.639 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.639 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.639 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.639 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "0", 2024-04-25T11:47:03.639 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "6", 2024-04-25T11:47:03.639 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.640 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T11:47:03.640 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T11:47:03.640 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.640 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.640 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_compat_max_iterations": { 2024-04-25T11:47:03.640 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crush_compat_max_iterations", 2024-04-25T11:47:03.640 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-25T11:47:03.640 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.640 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.640 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "25", 2024-04-25T11:47:03.640 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1", 2024-04-25T11:47:03.640 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "250", 2024-04-25T11:47:03.640 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.641 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T11:47:03.641 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.641 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.641 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.641 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_compat_metrics": { 2024-04-25T11:47:03.641 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crush_compat_metrics", 2024-04-25T11:47:03.641 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.641 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.641 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.641 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T11:47:03.641 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.641 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.641 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.642 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T11:47:03.642 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T11:47:03.642 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.642 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.642 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_compat_step": { 2024-04-25T11:47:03.642 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crush_compat_step", 2024-04-25T11:47:03.642 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-25T11:47:03.642 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.642 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.642 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0.5", 2024-04-25T11:47:03.642 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "0.001", 2024-04-25T11:47:03.642 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "0.999", 2024-04-25T11:47:03.642 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.642 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "aggressiveness of optimization", 2024-04-25T11:47:03.643 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T11:47:03.643 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.643 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.643 INFO:teuthology.orchestra.run.smithi090.stdout: "end_time": { 2024-04-25T11:47:03.643 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "end_time", 2024-04-25T11:47:03.643 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.643 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.643 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.643 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "2359", 2024-04-25T11:47:03.643 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.643 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.643 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.643 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T11:47:03.644 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T11:47:03.644 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.644 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.644 INFO:teuthology.orchestra.run.smithi090.stdout: "end_weekday": { 2024-04-25T11:47:03.644 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "end_weekday", 2024-04-25T11:47:03.644 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-25T11:47:03.644 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.644 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.644 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.644 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "0", 2024-04-25T11:47:03.644 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "6", 2024-04-25T11:47:03.644 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.644 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T11:47:03.644 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T11:47:03.645 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.645 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.645 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.645 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.645 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.645 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.645 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.645 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.645 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.645 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.645 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.645 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.645 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.646 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.646 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.646 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.646 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.646 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.646 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.646 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.646 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.646 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.646 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.646 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.646 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.646 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.647 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.647 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.647 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.647 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.647 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.647 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.647 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.647 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.647 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.647 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.647 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.647 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.647 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.648 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.648 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.648 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.648 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.648 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.648 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.648 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.648 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.648 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.648 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.648 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.648 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.648 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.648 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.649 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.649 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.649 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.649 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.649 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.649 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.649 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.649 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.649 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.649 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.649 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.649 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.649 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.649 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.650 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.650 INFO:teuthology.orchestra.run.smithi090.stdout: "min_score": { 2024-04-25T11:47:03.650 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "min_score", 2024-04-25T11:47:03.650 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-25T11:47:03.650 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.650 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.650 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.650 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.650 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.650 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.650 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T11:47:03.650 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.650 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.651 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.651 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": { 2024-04-25T11:47:03.651 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mode", 2024-04-25T11:47:03.651 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.651 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.651 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.651 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "upmap", 2024-04-25T11:47:03.651 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.651 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.651 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.651 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-compat", 2024-04-25T11:47:03.651 INFO:teuthology.orchestra.run.smithi090.stdout: "none", 2024-04-25T11:47:03.652 INFO:teuthology.orchestra.run.smithi090.stdout: "read", 2024-04-25T11:47:03.652 INFO:teuthology.orchestra.run.smithi090.stdout: "upmap", 2024-04-25T11:47:03.652 INFO:teuthology.orchestra.run.smithi090.stdout: "upmap-read" 2024-04-25T11:47:03.652 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.652 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Balancer mode", 2024-04-25T11:47:03.652 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.652 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.652 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.652 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_ids": { 2024-04-25T11:47:03.652 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "pool_ids", 2024-04-25T11:47:03.652 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.652 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.652 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.652 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.653 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.653 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.653 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.653 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T11:47:03.653 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.653 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.653 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.653 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-25T11:47:03.653 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-25T11:47:03.653 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.653 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.653 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.653 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-25T11:47:03.653 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.654 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.654 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.654 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T11:47:03.654 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.654 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.654 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.654 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.654 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.654 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.654 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.654 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.654 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.654 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.655 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.655 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.655 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.655 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.655 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.655 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.655 INFO:teuthology.orchestra.run.smithi090.stdout: "upmap_max_deviation": { 2024-04-25T11:47:03.655 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "upmap_max_deviation", 2024-04-25T11:47:03.655 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.655 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.655 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.655 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5", 2024-04-25T11:47:03.655 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1", 2024-04-25T11:47:03.655 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.656 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.656 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T11:47:03.656 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T11:47:03.656 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.656 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.656 INFO:teuthology.orchestra.run.smithi090.stdout: "upmap_max_optimizations": { 2024-04-25T11:47:03.656 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "upmap_max_optimizations", 2024-04-25T11:47:03.656 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-25T11:47:03.656 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.656 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.656 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-25T11:47:03.656 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.656 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.656 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.657 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T11:47:03.657 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.657 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.657 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.657 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.657 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.657 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cephadm", 2024-04-25T11:47:03.657 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.657 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.657 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.657 INFO:teuthology.orchestra.run.smithi090.stdout: "agent_down_multiplier": { 2024-04-25T11:47:03.657 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "agent_down_multiplier", 2024-04-25T11:47:03.657 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-25T11:47:03.658 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.658 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.658 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "3.0", 2024-04-25T11:47:03.658 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.658 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.658 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.658 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T11:47:03.658 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.658 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.658 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.658 INFO:teuthology.orchestra.run.smithi090.stdout: "agent_refresh_rate": { 2024-04-25T11:47:03.658 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "agent_refresh_rate", 2024-04-25T11:47:03.658 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.659 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.659 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.659 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "20", 2024-04-25T11:47:03.659 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.659 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.659 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.659 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T11:47:03.659 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.659 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.659 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.659 INFO:teuthology.orchestra.run.smithi090.stdout: "agent_starting_port": { 2024-04-25T11:47:03.659 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "agent_starting_port", 2024-04-25T11:47:03.659 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.660 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.660 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.660 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "4721", 2024-04-25T11:47:03.660 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.660 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.660 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.660 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T11:47:03.660 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.660 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.660 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.660 INFO:teuthology.orchestra.run.smithi090.stdout: "allow_ptrace": { 2024-04-25T11:47:03.660 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "allow_ptrace", 2024-04-25T11:47:03.660 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.660 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.661 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.661 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.661 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.661 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.661 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.661 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T11:47:03.661 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T11:47:03.661 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.661 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.661 INFO:teuthology.orchestra.run.smithi090.stdout: "autotune_interval": { 2024-04-25T11:47:03.661 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "autotune_interval", 2024-04-25T11:47:03.661 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.661 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.662 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.662 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-25T11:47:03.662 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.662 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.662 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.662 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T11:47:03.662 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.662 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.662 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.662 INFO:teuthology.orchestra.run.smithi090.stdout: "autotune_memory_target_ratio": { 2024-04-25T11:47:03.662 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T11:47:03.662 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-25T11:47:03.662 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.663 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.663 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0.7", 2024-04-25T11:47:03.663 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.663 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.663 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.663 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T11:47:03.663 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.663 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.663 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.663 INFO:teuthology.orchestra.run.smithi090.stdout: "cephadm_log_destination": { 2024-04-25T11:47:03.663 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cephadm_log_destination", 2024-04-25T11:47:03.663 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.663 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.663 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.664 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.664 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.664 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.664 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.664 INFO:teuthology.orchestra.run.smithi090.stdout: "file", 2024-04-25T11:47:03.664 INFO:teuthology.orchestra.run.smithi090.stdout: "file,syslog", 2024-04-25T11:47:03.664 INFO:teuthology.orchestra.run.smithi090.stdout: "syslog" 2024-04-25T11:47:03.664 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.664 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T11:47:03.664 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.664 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.664 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.664 INFO:teuthology.orchestra.run.smithi090.stdout: "cgroups_split": { 2024-04-25T11:47:03.665 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cgroups_split", 2024-04-25T11:47:03.665 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.665 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.665 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.665 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.665 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.665 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.665 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.665 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T11:47:03.665 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.665 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.665 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.665 INFO:teuthology.orchestra.run.smithi090.stdout: "config_checks_enabled": { 2024-04-25T11:47:03.666 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "config_checks_enabled", 2024-04-25T11:47:03.666 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.666 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.666 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.666 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.666 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.666 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.666 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.666 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T11:47:03.666 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.666 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.666 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.666 INFO:teuthology.orchestra.run.smithi090.stdout: "config_dashboard": { 2024-04-25T11:47:03.666 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "config_dashboard", 2024-04-25T11:47:03.667 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.667 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.667 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.667 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.667 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.667 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.667 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.667 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T11:47:03.667 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.667 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.667 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.667 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_alertmanager": { 2024-04-25T11:47:03.667 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_alertmanager", 2024-04-25T11:47:03.668 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.668 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.668 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.668 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T11:47:03.668 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.668 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.668 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.668 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Prometheus container image", 2024-04-25T11:47:03.668 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.668 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.668 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.668 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_base": { 2024-04-25T11:47:03.668 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_base", 2024-04-25T11:47:03.669 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.669 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.669 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.669 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T11:47:03.669 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.669 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.669 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.669 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Container image name, without the tag", 2024-04-25T11:47:03.669 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.669 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.669 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.669 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_elasticsearch": { 2024-04-25T11:47:03.669 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_elasticsearch", 2024-04-25T11:47:03.669 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.670 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.670 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.670 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T11:47:03.670 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.670 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.670 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.670 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "elasticsearch container image", 2024-04-25T11:47:03.670 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.670 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.670 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.670 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_grafana": { 2024-04-25T11:47:03.670 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_grafana", 2024-04-25T11:47:03.671 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.671 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.671 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.671 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T11:47:03.671 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.671 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.671 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.671 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Prometheus container image", 2024-04-25T11:47:03.671 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.671 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.671 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.671 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_haproxy": { 2024-04-25T11:47:03.671 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_haproxy", 2024-04-25T11:47:03.671 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.672 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.672 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.672 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T11:47:03.672 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.672 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.672 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.672 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "HAproxy container image", 2024-04-25T11:47:03.672 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.672 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.672 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.672 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_jaeger_agent": { 2024-04-25T11:47:03.672 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_jaeger_agent", 2024-04-25T11:47:03.672 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.672 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.673 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.673 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T11:47:03.673 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.673 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.673 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.673 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Jaeger agent container image", 2024-04-25T11:47:03.673 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.673 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.673 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.673 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_jaeger_collector": { 2024-04-25T11:47:03.673 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_jaeger_collector", 2024-04-25T11:47:03.673 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.673 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.674 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.674 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T11:47:03.674 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.674 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.674 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.674 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Jaeger collector container image", 2024-04-25T11:47:03.674 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.674 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.674 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.674 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_jaeger_query": { 2024-04-25T11:47:03.674 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_jaeger_query", 2024-04-25T11:47:03.674 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.674 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.675 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.675 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T11:47:03.675 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.675 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.675 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.675 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Jaeger query container image", 2024-04-25T11:47:03.675 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.675 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.675 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.675 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_keepalived": { 2024-04-25T11:47:03.675 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_keepalived", 2024-04-25T11:47:03.675 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.675 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.675 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.676 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T11:47:03.676 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.676 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.676 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.676 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Keepalived container image", 2024-04-25T11:47:03.676 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.676 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.676 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.676 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_loki": { 2024-04-25T11:47:03.676 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_loki", 2024-04-25T11:47:03.676 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.676 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.676 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.677 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T11:47:03.677 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.677 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.677 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.677 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Loki container image", 2024-04-25T11:47:03.677 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.677 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.677 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.677 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_node_exporter": { 2024-04-25T11:47:03.677 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_node_exporter", 2024-04-25T11:47:03.677 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.677 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.677 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.678 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T11:47:03.678 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.678 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.678 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.678 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Prometheus container image", 2024-04-25T11:47:03.678 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.678 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.678 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.678 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_nvmeof": { 2024-04-25T11:47:03.678 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_nvmeof", 2024-04-25T11:47:03.678 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.678 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.679 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.679 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T11:47:03.679 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.679 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.679 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.679 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Nvme-of container image", 2024-04-25T11:47:03.679 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.679 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.679 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.679 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_prometheus": { 2024-04-25T11:47:03.679 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_prometheus", 2024-04-25T11:47:03.679 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.679 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.679 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.680 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T11:47:03.680 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.680 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.680 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.680 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Prometheus container image", 2024-04-25T11:47:03.680 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.680 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.680 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.680 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_promtail": { 2024-04-25T11:47:03.680 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_promtail", 2024-04-25T11:47:03.680 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.680 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.680 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.680 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T11:47:03.681 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.681 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.681 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.681 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Promtail container image", 2024-04-25T11:47:03.681 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.681 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.681 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.681 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_samba": { 2024-04-25T11:47:03.681 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_samba", 2024-04-25T11:47:03.681 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.681 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.681 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.682 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T11:47:03.682 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.682 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.682 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.682 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Samba/SMB container image", 2024-04-25T11:47:03.682 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.682 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.682 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.682 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_snmp_gateway": { 2024-04-25T11:47:03.682 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_snmp_gateway", 2024-04-25T11:47:03.682 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.682 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.682 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.683 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T11:47:03.683 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.683 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.683 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.683 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "SNMP Gateway container image", 2024-04-25T11:47:03.683 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.683 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.683 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.683 INFO:teuthology.orchestra.run.smithi090.stdout: "container_init": { 2024-04-25T11:47:03.683 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_init", 2024-04-25T11:47:03.683 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.683 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.683 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.683 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.684 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.684 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.684 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.684 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T11:47:03.684 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.684 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.684 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.684 INFO:teuthology.orchestra.run.smithi090.stdout: "daemon_cache_timeout": { 2024-04-25T11:47:03.684 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "daemon_cache_timeout", 2024-04-25T11:47:03.684 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.684 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.684 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.684 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-25T11:47:03.684 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.685 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.685 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.685 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T11:47:03.685 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.685 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.685 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.685 INFO:teuthology.orchestra.run.smithi090.stdout: "default_cephadm_command_timeout": { 2024-04-25T11:47:03.685 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T11:47:03.685 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.685 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.685 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.685 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "900", 2024-04-25T11:47:03.685 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.686 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.686 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.686 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T11:47:03.686 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.686 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.686 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.686 INFO:teuthology.orchestra.run.smithi090.stdout: "default_registry": { 2024-04-25T11:47:03.686 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "default_registry", 2024-04-25T11:47:03.686 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.686 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.686 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.686 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "docker.io", 2024-04-25T11:47:03.686 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.687 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.687 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.687 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T11:47:03.687 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.687 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.687 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.687 INFO:teuthology.orchestra.run.smithi090.stdout: "device_cache_timeout": { 2024-04-25T11:47:03.687 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "device_cache_timeout", 2024-04-25T11:47:03.687 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.687 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.687 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.687 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "1800", 2024-04-25T11:47:03.687 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.688 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.688 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.688 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "seconds to cache device inventory", 2024-04-25T11:47:03.688 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.688 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.688 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.688 INFO:teuthology.orchestra.run.smithi090.stdout: "device_enhanced_scan": { 2024-04-25T11:47:03.688 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "device_enhanced_scan", 2024-04-25T11:47:03.688 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.688 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.688 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.688 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.688 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.688 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.689 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.689 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T11:47:03.689 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.689 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.689 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.689 INFO:teuthology.orchestra.run.smithi090.stdout: "facts_cache_timeout": { 2024-04-25T11:47:03.689 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "facts_cache_timeout", 2024-04-25T11:47:03.689 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.689 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.689 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.689 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-25T11:47:03.689 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.689 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.690 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.690 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "seconds to cache host facts data", 2024-04-25T11:47:03.690 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.690 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.690 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.690 INFO:teuthology.orchestra.run.smithi090.stdout: "grafana_dashboards_path": { 2024-04-25T11:47:03.690 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "grafana_dashboards_path", 2024-04-25T11:47:03.690 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.690 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.690 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.690 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T11:47:03.690 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.690 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.691 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.691 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T11:47:03.691 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.691 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.691 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.691 INFO:teuthology.orchestra.run.smithi090.stdout: "host_check_interval": { 2024-04-25T11:47:03.691 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "host_check_interval", 2024-04-25T11:47:03.691 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.691 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.691 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.691 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-25T11:47:03.691 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.691 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.692 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.692 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to perform a host check", 2024-04-25T11:47:03.692 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.692 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.692 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.692 INFO:teuthology.orchestra.run.smithi090.stdout: "hw_monitoring": { 2024-04-25T11:47:03.692 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "hw_monitoring", 2024-04-25T11:47:03.692 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.692 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.692 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.692 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.692 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.692 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.693 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.693 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T11:47:03.693 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.693 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.693 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.693 INFO:teuthology.orchestra.run.smithi090.stdout: "inventory_list_all": { 2024-04-25T11:47:03.693 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "inventory_list_all", 2024-04-25T11:47:03.693 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.693 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.693 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.693 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.693 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.693 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.694 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.694 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T11:47:03.694 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.694 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.694 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.694 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.694 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.694 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.694 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.694 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.694 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.694 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.694 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.695 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.695 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.695 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.695 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.695 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.695 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.695 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.695 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.695 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.695 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.695 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.695 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.695 INFO:teuthology.orchestra.run.smithi090.stdout: "log_refresh_metadata": { 2024-04-25T11:47:03.696 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_refresh_metadata", 2024-04-25T11:47:03.696 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.696 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.696 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.696 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.696 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.696 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.696 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.696 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T11:47:03.696 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.696 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.696 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.696 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.697 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.697 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.697 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.697 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.697 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.697 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.697 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.697 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.697 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T11:47:03.697 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.697 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.697 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.697 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.697 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.698 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.698 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.698 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.698 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.698 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.698 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.698 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.698 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.698 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.698 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.698 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.698 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.698 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.698 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.699 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.699 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.699 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.699 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.699 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.699 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.699 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.699 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.699 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.699 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.699 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.699 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.699 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.700 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.700 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.700 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.700 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.700 INFO:teuthology.orchestra.run.smithi090.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T11:47:03.700 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T11:47:03.700 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.700 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.700 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.700 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.700 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.700 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.700 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.700 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T11:47:03.701 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.701 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.701 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.701 INFO:teuthology.orchestra.run.smithi090.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T11:47:03.701 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T11:47:03.701 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.701 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.701 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.701 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "*", 2024-04-25T11:47:03.701 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.701 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.701 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.701 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T11:47:03.702 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.702 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.702 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.702 INFO:teuthology.orchestra.run.smithi090.stdout: "max_count_per_host": { 2024-04-25T11:47:03.702 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_count_per_host", 2024-04-25T11:47:03.702 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.702 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.702 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.702 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-25T11:47:03.702 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.702 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.702 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.702 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "max number of daemons per service per host", 2024-04-25T11:47:03.703 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.703 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.703 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.703 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd_draining_count": { 2024-04-25T11:47:03.703 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_osd_draining_count", 2024-04-25T11:47:03.703 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.703 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.703 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.703 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-25T11:47:03.703 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.703 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.703 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.703 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T11:47:03.703 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.704 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.704 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.704 INFO:teuthology.orchestra.run.smithi090.stdout: "migration_current": { 2024-04-25T11:47:03.704 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "migration_current", 2024-04-25T11:47:03.704 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.704 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.704 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.704 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.704 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.704 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.704 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.704 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "internal - do not modify", 2024-04-25T11:47:03.704 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.705 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.705 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.705 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": { 2024-04-25T11:47:03.705 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mode", 2024-04-25T11:47:03.705 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.705 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.705 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.705 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "root", 2024-04-25T11:47:03.705 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.705 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.705 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.705 INFO:teuthology.orchestra.run.smithi090.stdout: "cephadm-package", 2024-04-25T11:47:03.705 INFO:teuthology.orchestra.run.smithi090.stdout: "root" 2024-04-25T11:47:03.706 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.706 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T11:47:03.706 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.706 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.706 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.706 INFO:teuthology.orchestra.run.smithi090.stdout: "oob_default_addr": { 2024-04-25T11:47:03.706 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "oob_default_addr", 2024-04-25T11:47:03.706 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.706 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.706 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.706 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "169.254.1.1", 2024-04-25T11:47:03.706 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.707 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.707 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.707 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T11:47:03.707 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.707 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.707 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.707 INFO:teuthology.orchestra.run.smithi090.stdout: "prometheus_alerts_path": { 2024-04-25T11:47:03.707 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "prometheus_alerts_path", 2024-04-25T11:47:03.707 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.707 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.707 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.707 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T11:47:03.707 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.708 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.708 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.708 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T11:47:03.708 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.708 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.708 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.708 INFO:teuthology.orchestra.run.smithi090.stdout: "registry_insecure": { 2024-04-25T11:47:03.708 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "registry_insecure", 2024-04-25T11:47:03.708 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.708 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.708 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.708 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.708 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.708 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.709 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.709 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T11:47:03.709 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.709 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.709 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.709 INFO:teuthology.orchestra.run.smithi090.stdout: "registry_password": { 2024-04-25T11:47:03.709 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "registry_password", 2024-04-25T11:47:03.709 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.709 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.709 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.709 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.709 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.709 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.710 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.710 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T11:47:03.710 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.710 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.710 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.710 INFO:teuthology.orchestra.run.smithi090.stdout: "registry_url": { 2024-04-25T11:47:03.710 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "registry_url", 2024-04-25T11:47:03.710 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.710 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.710 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.710 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.710 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.710 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.711 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.711 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T11:47:03.711 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.711 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.711 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.711 INFO:teuthology.orchestra.run.smithi090.stdout: "registry_username": { 2024-04-25T11:47:03.711 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "registry_username", 2024-04-25T11:47:03.711 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.711 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.711 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.711 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.711 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.711 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.712 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.712 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T11:47:03.712 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.712 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.712 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.712 INFO:teuthology.orchestra.run.smithi090.stdout: "secure_monitoring_stack": { 2024-04-25T11:47:03.712 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "secure_monitoring_stack", 2024-04-25T11:47:03.712 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.712 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.712 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.712 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.712 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.712 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.713 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.713 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T11:47:03.713 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.713 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.713 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.713 INFO:teuthology.orchestra.run.smithi090.stdout: "service_discovery_port": { 2024-04-25T11:47:03.713 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "service_discovery_port", 2024-04-25T11:47:03.713 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.713 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.713 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.713 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8765", 2024-04-25T11:47:03.713 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.713 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.714 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.714 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "cephadm service discovery port", 2024-04-25T11:47:03.714 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.714 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.714 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.714 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.714 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.714 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.714 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.714 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.714 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.714 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.715 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.715 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.715 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.715 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.715 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.715 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.715 INFO:teuthology.orchestra.run.smithi090.stdout: "ssh_config_file": { 2024-04-25T11:47:03.715 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ssh_config_file", 2024-04-25T11:47:03.715 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.715 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.715 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.715 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.715 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.715 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.716 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.716 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T11:47:03.716 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.716 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.716 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.716 INFO:teuthology.orchestra.run.smithi090.stdout: "use_agent": { 2024-04-25T11:47:03.716 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "use_agent", 2024-04-25T11:47:03.716 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.716 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.716 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.716 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.716 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.716 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.716 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.717 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T11:47:03.717 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.717 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.717 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.717 INFO:teuthology.orchestra.run.smithi090.stdout: "use_repo_digest": { 2024-04-25T11:47:03.717 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "use_repo_digest", 2024-04-25T11:47:03.717 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.717 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.717 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.717 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.717 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.717 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.717 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.717 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T11:47:03.718 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.718 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.718 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.718 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_on_failed_host_check": { 2024-04-25T11:47:03.718 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_on_failed_host_check", 2024-04-25T11:47:03.718 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.718 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.718 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.718 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.718 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.718 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.718 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.718 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T11:47:03.719 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.719 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.719 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.719 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_on_stray_daemons": { 2024-04-25T11:47:03.719 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_on_stray_daemons", 2024-04-25T11:47:03.719 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.719 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.719 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.719 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.719 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.720 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.720 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.720 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T11:47:03.720 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.720 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.720 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.720 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_on_stray_hosts": { 2024-04-25T11:47:03.720 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_on_stray_hosts", 2024-04-25T11:47:03.720 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.720 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.720 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.720 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.720 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.720 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.721 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.721 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T11:47:03.721 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.721 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.721 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.721 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.721 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.721 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crash", 2024-04-25T11:47:03.721 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.721 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.721 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.721 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.721 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.721 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.722 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.722 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.722 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.722 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.722 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.722 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.722 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.722 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.722 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.722 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.722 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.722 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.722 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.722 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.723 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.723 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.723 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.723 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.723 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.723 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.723 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.723 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.723 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.723 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.723 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.723 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.723 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.723 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.724 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.724 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.724 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.724 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.724 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.724 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.724 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.724 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.724 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.724 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.724 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.724 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.724 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.724 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.724 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.725 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.725 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.725 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.725 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.725 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.725 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.725 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.725 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.725 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.725 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.725 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.725 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.725 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.725 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.725 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.726 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.726 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.726 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.726 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.726 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.726 INFO:teuthology.orchestra.run.smithi090.stdout: "retain_interval": { 2024-04-25T11:47:03.726 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "retain_interval", 2024-04-25T11:47:03.726 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.726 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.726 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.726 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "31536000", 2024-04-25T11:47:03.726 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.726 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.726 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.727 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T11:47:03.727 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.727 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.727 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.727 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.727 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.727 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.727 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.727 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.727 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.727 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.727 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.727 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.727 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.727 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.728 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.728 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.728 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_recent_interval": { 2024-04-25T11:47:03.728 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_recent_interval", 2024-04-25T11:47:03.728 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.728 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.728 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.728 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "1209600", 2024-04-25T11:47:03.728 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.728 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.728 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.728 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T11:47:03.728 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.728 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.728 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.728 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.729 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.729 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "dashboard", 2024-04-25T11:47:03.729 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.729 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.729 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.729 INFO:teuthology.orchestra.run.smithi090.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T11:47:03.729 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T11:47:03.729 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.729 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.729 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.729 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-25T11:47:03.729 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.729 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.730 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.730 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.730 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.730 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.730 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.730 INFO:teuthology.orchestra.run.smithi090.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T11:47:03.730 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T11:47:03.730 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.730 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.730 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.730 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.730 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.730 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.731 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.731 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.731 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.731 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.731 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.731 INFO:teuthology.orchestra.run.smithi090.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T11:47:03.731 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T11:47:03.731 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.731 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.731 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.731 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.731 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.731 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.731 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.731 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.732 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.732 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.732 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.732 INFO:teuthology.orchestra.run.smithi090.stdout: "AUDIT_API_ENABLED": { 2024-04-25T11:47:03.732 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T11:47:03.732 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.732 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.732 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.732 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.732 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.732 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.732 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.732 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.732 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.732 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.733 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.733 INFO:teuthology.orchestra.run.smithi090.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T11:47:03.733 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T11:47:03.733 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.733 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.733 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.733 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.733 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.733 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.733 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.733 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.733 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.733 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.733 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.734 INFO:teuthology.orchestra.run.smithi090.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T11:47:03.734 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T11:47:03.734 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.734 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.734 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.734 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.734 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.734 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.734 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.734 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.734 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.734 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.734 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.735 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T11:47:03.735 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T11:47:03.735 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.735 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.735 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.735 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.735 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.735 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.735 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.735 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.735 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.735 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.735 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.736 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T11:47:03.736 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T11:47:03.736 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.736 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.736 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.736 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.736 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.736 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.736 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.736 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.736 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.736 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.736 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.737 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T11:47:03.737 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T11:47:03.737 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.737 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.737 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.737 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.737 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.737 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.737 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.737 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.737 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.737 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.737 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.737 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T11:47:03.738 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T11:47:03.738 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.738 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.738 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.738 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.738 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.738 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.738 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.738 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.738 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.738 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.738 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.739 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T11:47:03.739 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T11:47:03.739 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.739 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.739 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.739 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.739 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.739 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.739 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.739 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.739 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.739 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.739 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.739 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T11:47:03.740 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T11:47:03.740 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.740 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.740 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.740 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.740 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.740 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.740 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.740 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.740 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.740 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.740 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.740 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T11:47:03.740 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T11:47:03.741 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.741 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.741 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.741 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.741 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.741 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.741 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.741 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.741 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.741 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.741 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.741 INFO:teuthology.orchestra.run.smithi090.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T11:47:03.741 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T11:47:03.741 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.742 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.742 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.742 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.742 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.742 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.742 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.742 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.742 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.742 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.742 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.742 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T11:47:03.742 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T11:47:03.742 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.742 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.743 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.743 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "admin", 2024-04-25T11:47:03.743 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.743 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.743 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.743 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.743 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.743 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.743 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.743 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T11:47:03.743 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T11:47:03.743 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.743 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.743 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.743 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.744 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.744 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.744 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.744 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.744 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.744 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.744 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.744 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_API_URL": { 2024-04-25T11:47:03.744 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_API_URL", 2024-04-25T11:47:03.744 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.744 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.744 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.744 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.744 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.745 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.745 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.745 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.745 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.745 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.745 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.745 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T11:47:03.745 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T11:47:03.745 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.745 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.745 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.745 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "admin", 2024-04-25T11:47:03.745 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.745 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.746 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.746 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.746 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.746 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.746 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.746 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T11:47:03.746 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T11:47:03.746 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.746 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.746 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.746 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.746 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.746 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.746 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.747 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.747 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.747 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.747 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.747 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T11:47:03.747 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T11:47:03.747 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.747 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.747 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.747 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.747 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.747 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.747 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.747 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.747 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.748 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.748 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.748 INFO:teuthology.orchestra.run.smithi090.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T11:47:03.748 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T11:47:03.748 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.748 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.748 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.748 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.748 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.748 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.748 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.748 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.748 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.748 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.748 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.749 INFO:teuthology.orchestra.run.smithi090.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T11:47:03.749 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T11:47:03.749 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.749 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.749 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.749 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.749 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.749 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.749 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.749 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.749 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.749 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.749 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.749 INFO:teuthology.orchestra.run.smithi090.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-25T11:47:03.750 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-25T11:47:03.750 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.750 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.750 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.750 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "[]", 2024-04-25T11:47:03.750 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.750 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.750 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.750 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.750 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.750 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.750 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.750 INFO:teuthology.orchestra.run.smithi090.stdout: "MULTICLUSTER_CONFIG": { 2024-04-25T11:47:03.750 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-25T11:47:03.751 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.751 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.751 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.751 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "{}", 2024-04-25T11:47:03.751 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.751 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.751 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.751 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.751 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.751 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.751 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.751 INFO:teuthology.orchestra.run.smithi090.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T11:47:03.751 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T11:47:03.752 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.752 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.752 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.752 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.752 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.752 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.752 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.752 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.752 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.752 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.752 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.752 INFO:teuthology.orchestra.run.smithi090.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T11:47:03.752 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T11:47:03.752 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.753 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.753 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.753 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.753 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.753 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.753 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.753 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.753 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.753 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.753 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.753 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T11:47:03.753 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T11:47:03.753 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.754 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.754 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.754 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.754 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.754 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.754 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.754 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.754 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.754 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.754 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.754 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T11:47:03.754 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T11:47:03.754 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.755 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.755 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.755 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.755 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.755 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.755 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.755 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.755 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.755 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.755 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.755 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T11:47:03.755 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T11:47:03.755 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.756 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.756 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.756 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.756 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.756 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.756 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.756 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.756 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.756 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.756 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.756 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T11:47:03.756 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T11:47:03.756 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.756 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.756 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.756 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.757 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.757 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.757 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.757 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.757 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.757 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.757 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.757 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T11:47:03.757 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T11:47:03.757 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.757 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.757 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.757 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.757 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.757 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.758 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.758 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.758 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.758 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.758 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.758 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T11:47:03.758 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T11:47:03.758 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.758 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.758 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.758 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.758 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.758 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.759 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.759 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.759 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.759 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.759 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.759 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T11:47:03.759 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T11:47:03.759 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.759 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.759 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.759 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.759 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.759 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.760 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.760 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.760 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.760 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.760 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.760 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T11:47:03.760 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T11:47:03.760 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.760 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.760 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.760 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.760 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.760 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.760 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.760 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.761 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.761 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.761 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.761 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T11:47:03.761 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T11:47:03.761 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.761 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.761 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.761 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T11:47:03.761 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.761 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.761 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.761 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.761 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.762 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.762 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.762 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T11:47:03.762 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T11:47:03.762 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.762 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.762 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.762 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-25T11:47:03.762 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.762 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.762 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.762 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.762 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.763 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.763 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.763 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T11:47:03.763 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T11:47:03.763 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.763 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.763 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.763 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8", 2024-04-25T11:47:03.763 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.763 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.763 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.763 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.763 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.764 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.764 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.764 INFO:teuthology.orchestra.run.smithi090.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T11:47:03.764 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T11:47:03.764 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.764 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.764 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.764 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "45", 2024-04-25T11:47:03.764 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.764 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.764 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.764 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.764 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.764 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.765 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.765 INFO:teuthology.orchestra.run.smithi090.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T11:47:03.765 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T11:47:03.765 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.765 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.765 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.765 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.765 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.765 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.765 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.765 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.765 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.765 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.766 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.766 INFO:teuthology.orchestra.run.smithi090.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T11:47:03.766 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T11:47:03.766 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.766 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.766 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.766 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "admin", 2024-04-25T11:47:03.766 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.766 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.766 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.766 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.766 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.767 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.767 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.767 INFO:teuthology.orchestra.run.smithi090.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T11:47:03.767 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T11:47:03.767 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.767 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.767 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.767 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.767 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.767 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.767 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.767 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.767 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.767 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.768 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.768 INFO:teuthology.orchestra.run.smithi090.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T11:47:03.768 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T11:47:03.768 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.768 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.768 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.768 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.768 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.768 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.768 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.768 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.768 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.768 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.768 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.769 INFO:teuthology.orchestra.run.smithi090.stdout: "UNSAFE_TLS_v1_2": { 2024-04-25T11:47:03.769 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-25T11:47:03.769 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.769 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.769 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.769 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.769 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.769 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.769 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.769 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.769 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.769 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.769 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.770 INFO:teuthology.orchestra.run.smithi090.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T11:47:03.770 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T11:47:03.770 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.770 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.770 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.770 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.770 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.770 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.770 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.770 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.770 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.770 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.770 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.771 INFO:teuthology.orchestra.run.smithi090.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T11:47:03.771 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T11:47:03.771 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.771 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.771 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.771 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-25T11:47:03.771 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.771 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.771 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.771 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.771 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.771 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.771 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.772 INFO:teuthology.orchestra.run.smithi090.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T11:47:03.772 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T11:47:03.772 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.772 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.772 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.772 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5", 2024-04-25T11:47:03.772 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.772 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.772 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.772 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.772 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.772 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.772 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.772 INFO:teuthology.orchestra.run.smithi090.stdout: "cross_origin_url": { 2024-04-25T11:47:03.773 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cross_origin_url", 2024-04-25T11:47:03.773 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.773 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.773 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.773 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.773 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.773 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.773 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.773 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.773 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.773 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.773 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.773 INFO:teuthology.orchestra.run.smithi090.stdout: "crt_file": { 2024-04-25T11:47:03.773 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crt_file", 2024-04-25T11:47:03.773 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.774 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.774 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.774 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.774 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.774 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.774 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.774 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.774 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.774 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.774 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.774 INFO:teuthology.orchestra.run.smithi090.stdout: "debug": { 2024-04-25T11:47:03.774 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "debug", 2024-04-25T11:47:03.774 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.775 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.775 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.775 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.775 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.775 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.775 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.775 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Enable/disable debug options", 2024-04-25T11:47:03.775 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.775 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.775 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.775 INFO:teuthology.orchestra.run.smithi090.stdout: "jwt_token_ttl": { 2024-04-25T11:47:03.775 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "jwt_token_ttl", 2024-04-25T11:47:03.775 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.775 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.775 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.776 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "28800", 2024-04-25T11:47:03.776 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.776 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.776 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.776 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.776 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.776 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.776 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.776 INFO:teuthology.orchestra.run.smithi090.stdout: "key_file": { 2024-04-25T11:47:03.776 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "key_file", 2024-04-25T11:47:03.776 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.776 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.776 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.777 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.777 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.777 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.777 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.777 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.777 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.777 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.777 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.777 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.777 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.777 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.777 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.777 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.778 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.778 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.778 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.778 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.778 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.778 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.778 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.778 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.778 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.778 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.778 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.778 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.778 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.778 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.779 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.779 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.779 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.779 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.779 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.779 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.779 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.779 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.779 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.779 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.779 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.779 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.779 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.779 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.780 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.780 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.780 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.780 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.780 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.780 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.780 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.780 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.780 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.780 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.780 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.780 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.780 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.780 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.780 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.781 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.781 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.781 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.781 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.781 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.781 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.781 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.781 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.781 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.781 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.781 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.781 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.781 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.782 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.782 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.782 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.782 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.782 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.782 INFO:teuthology.orchestra.run.smithi090.stdout: "motd": { 2024-04-25T11:47:03.782 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "motd", 2024-04-25T11:47:03.782 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.782 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.782 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.782 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.782 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.782 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.783 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.783 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "The message of the day", 2024-04-25T11:47:03.783 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.783 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.783 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.783 INFO:teuthology.orchestra.run.smithi090.stdout: "redirect_resolve_ip_addr": { 2024-04-25T11:47:03.783 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T11:47:03.783 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.783 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.783 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.783 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.783 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.783 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.783 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.784 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.784 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.784 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.784 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.784 INFO:teuthology.orchestra.run.smithi090.stdout: "server_addr": { 2024-04-25T11:47:03.784 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_addr", 2024-04-25T11:47:03.784 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.784 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.784 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.784 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "::", 2024-04-25T11:47:03.784 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.784 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.784 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.784 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.784 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.785 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.785 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.785 INFO:teuthology.orchestra.run.smithi090.stdout: "server_port": { 2024-04-25T11:47:03.785 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_port", 2024-04-25T11:47:03.785 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.785 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.785 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.785 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8080", 2024-04-25T11:47:03.785 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.785 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.785 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.785 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.785 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.785 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.786 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.786 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.786 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.786 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.786 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.786 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.786 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.786 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.786 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.786 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.786 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.786 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.786 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.787 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.787 INFO:teuthology.orchestra.run.smithi090.stdout: "ssl": { 2024-04-25T11:47:03.787 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ssl", 2024-04-25T11:47:03.787 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.787 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.787 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.787 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.787 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.787 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.787 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.787 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.787 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.787 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.787 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.788 INFO:teuthology.orchestra.run.smithi090.stdout: "ssl_server_port": { 2024-04-25T11:47:03.788 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ssl_server_port", 2024-04-25T11:47:03.788 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.788 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.788 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.788 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8443", 2024-04-25T11:47:03.788 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.788 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.788 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.788 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.788 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.788 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.788 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.788 INFO:teuthology.orchestra.run.smithi090.stdout: "standby_behaviour": { 2024-04-25T11:47:03.788 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "standby_behaviour", 2024-04-25T11:47:03.789 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.789 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.789 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.789 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "redirect", 2024-04-25T11:47:03.789 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.789 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.789 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.789 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.789 INFO:teuthology.orchestra.run.smithi090.stdout: "redirect" 2024-04-25T11:47:03.789 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.789 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.789 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.789 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.789 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.790 INFO:teuthology.orchestra.run.smithi090.stdout: "standby_error_status_code": { 2024-04-25T11:47:03.790 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "standby_error_status_code", 2024-04-25T11:47:03.790 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.790 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.790 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.790 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "500", 2024-04-25T11:47:03.790 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "400", 2024-04-25T11:47:03.790 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "599", 2024-04-25T11:47:03.790 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.790 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.790 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.790 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.790 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.791 INFO:teuthology.orchestra.run.smithi090.stdout: "url_prefix": { 2024-04-25T11:47:03.791 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "url_prefix", 2024-04-25T11:47:03.791 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.791 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.791 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.791 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.791 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.791 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.791 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.791 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.791 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.791 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.791 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.791 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.791 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.792 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "devicehealth", 2024-04-25T11:47:03.792 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.792 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.792 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.792 INFO:teuthology.orchestra.run.smithi090.stdout: "enable_monitoring": { 2024-04-25T11:47:03.792 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "enable_monitoring", 2024-04-25T11:47:03.792 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.792 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.792 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.792 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.792 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.792 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.792 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.792 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "monitor device health metrics", 2024-04-25T11:47:03.792 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.793 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.793 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.793 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.793 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.793 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.793 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.793 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.793 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.793 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.793 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.793 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.793 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.793 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.794 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.794 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.794 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.794 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.794 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.794 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.794 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.794 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.794 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.794 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.794 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.794 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.794 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.795 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.795 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.795 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.795 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.795 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.795 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.795 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.795 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.795 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.795 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.795 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.795 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.795 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.795 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.796 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.796 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.796 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.796 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.796 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.796 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.796 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.796 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.796 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.796 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.796 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.796 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.796 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.796 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.796 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.797 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.797 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.797 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.797 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.797 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.797 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.797 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.797 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.797 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.797 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.797 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.797 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.797 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.798 INFO:teuthology.orchestra.run.smithi090.stdout: "mark_out_threshold": { 2024-04-25T11:47:03.798 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mark_out_threshold", 2024-04-25T11:47:03.798 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.798 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.798 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.798 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "2419200", 2024-04-25T11:47:03.798 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.798 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.798 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.798 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T11:47:03.798 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.798 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.798 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.799 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": { 2024-04-25T11:47:03.799 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "pool_name", 2024-04-25T11:47:03.799 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.799 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.799 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.799 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "device_health_metrics", 2024-04-25T11:47:03.799 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.799 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.799 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.799 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T11:47:03.799 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.799 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.799 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.800 INFO:teuthology.orchestra.run.smithi090.stdout: "retention_period": { 2024-04-25T11:47:03.800 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "retention_period", 2024-04-25T11:47:03.800 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.800 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.800 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.800 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "15552000", 2024-04-25T11:47:03.800 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.800 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.800 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.800 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how long to retain device health metrics", 2024-04-25T11:47:03.800 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.800 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.800 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.801 INFO:teuthology.orchestra.run.smithi090.stdout: "scrape_frequency": { 2024-04-25T11:47:03.801 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "scrape_frequency", 2024-04-25T11:47:03.801 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.801 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.801 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.801 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "86400", 2024-04-25T11:47:03.801 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.801 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.801 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.801 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T11:47:03.801 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.801 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.801 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.801 INFO:teuthology.orchestra.run.smithi090.stdout: "self_heal": { 2024-04-25T11:47:03.802 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "self_heal", 2024-04-25T11:47:03.802 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.802 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.802 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.802 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.802 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.802 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.802 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.802 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T11:47:03.802 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.802 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.802 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.803 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-25T11:47:03.803 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-25T11:47:03.803 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.803 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.803 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.803 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-25T11:47:03.803 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.803 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.803 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.803 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T11:47:03.803 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.803 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.803 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.804 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.804 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.804 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.804 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.804 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.804 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.804 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.804 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.804 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.804 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.804 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.804 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.804 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.804 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_threshold": { 2024-04-25T11:47:03.805 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_threshold", 2024-04-25T11:47:03.805 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.805 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.805 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.805 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "7257600", 2024-04-25T11:47:03.805 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.805 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.805 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.805 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T11:47:03.805 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.805 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.805 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.805 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.805 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.806 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "diskprediction_local", 2024-04-25T11:47:03.806 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.806 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.806 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.806 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.806 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.806 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.806 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.806 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.806 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.806 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.806 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.806 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.807 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.807 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.807 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.807 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.807 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.807 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.807 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.807 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.807 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.807 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.807 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.807 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.807 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.808 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.808 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.808 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.808 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.808 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.808 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.808 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.808 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.808 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.808 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.808 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.808 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.808 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.808 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.809 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.809 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.809 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.809 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.809 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.809 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.809 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.809 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.809 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.809 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.809 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.809 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.809 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.809 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.809 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.810 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.810 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.810 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.810 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.810 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.810 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.810 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.810 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.810 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.810 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.810 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.810 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.810 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.811 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.811 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.811 INFO:teuthology.orchestra.run.smithi090.stdout: "predict_interval": { 2024-04-25T11:47:03.811 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "predict_interval", 2024-04-25T11:47:03.811 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.811 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.811 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.811 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "86400", 2024-04-25T11:47:03.811 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.811 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.811 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.811 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.811 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.812 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.812 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.812 INFO:teuthology.orchestra.run.smithi090.stdout: "predictor_model": { 2024-04-25T11:47:03.812 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "predictor_model", 2024-04-25T11:47:03.812 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.812 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.812 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.812 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "prophetstor", 2024-04-25T11:47:03.812 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.812 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.812 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.812 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.812 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.813 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.813 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.813 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-25T11:47:03.813 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-25T11:47:03.813 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.813 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.813 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.813 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-25T11:47:03.813 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.813 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.813 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.813 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.814 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.814 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.814 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.814 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.814 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.814 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.814 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.814 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.814 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.814 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.814 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.814 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.815 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.815 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.815 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.815 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.815 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.815 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.815 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "influx", 2024-04-25T11:47:03.815 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": false, 2024-04-25T11:47:03.815 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "influxdb python module not found", 2024-04-25T11:47:03.815 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.815 INFO:teuthology.orchestra.run.smithi090.stdout: "batch_size": { 2024-04-25T11:47:03.815 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "batch_size", 2024-04-25T11:47:03.816 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.817 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.817 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.817 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5000", 2024-04-25T11:47:03.817 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.817 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.817 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.817 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T11:47:03.817 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.818 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.818 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.818 INFO:teuthology.orchestra.run.smithi090.stdout: "database": { 2024-04-25T11:47:03.818 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "database", 2024-04-25T11:47:03.818 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.818 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.818 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.818 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "ceph", 2024-04-25T11:47:03.818 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.818 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.818 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.818 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T11:47:03.819 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.819 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.819 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.819 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": { 2024-04-25T11:47:03.819 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "hostname", 2024-04-25T11:47:03.819 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.819 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.819 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.819 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.819 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.819 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.819 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.819 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "InfluxDB server hostname", 2024-04-25T11:47:03.819 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.820 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.820 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.820 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-25T11:47:03.820 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-25T11:47:03.820 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.820 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.820 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.820 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "30", 2024-04-25T11:47:03.820 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "5", 2024-04-25T11:47:03.820 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.820 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.820 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T11:47:03.820 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.821 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.821 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.821 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.821 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.821 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.821 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.821 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.821 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.821 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.821 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.821 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.821 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.821 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.821 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.822 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.822 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.822 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.822 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.822 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.822 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.822 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.822 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.822 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.822 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.822 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.822 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.823 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.823 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.823 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.823 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.823 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.823 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.823 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.823 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.823 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.823 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.823 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.823 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.823 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.824 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.824 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.824 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.824 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.824 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.824 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.824 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.824 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.824 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.824 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.824 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.824 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.824 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.824 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.825 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.825 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.825 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.825 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.825 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.825 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.825 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.825 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.825 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.825 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.825 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.825 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.825 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.826 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.826 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.826 INFO:teuthology.orchestra.run.smithi090.stdout: "password": { 2024-04-25T11:47:03.826 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "password", 2024-04-25T11:47:03.826 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.826 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.826 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.826 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.826 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.826 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.826 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.826 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "password of InfluxDB server user", 2024-04-25T11:47:03.827 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.827 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.827 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.827 INFO:teuthology.orchestra.run.smithi090.stdout: "port": { 2024-04-25T11:47:03.827 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "port", 2024-04-25T11:47:03.827 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.827 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.827 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.827 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8086", 2024-04-25T11:47:03.827 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.827 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.827 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.827 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "InfluxDB server port", 2024-04-25T11:47:03.827 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.828 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.828 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.828 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.828 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.828 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.828 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.828 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.828 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.828 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.828 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.828 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.828 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.828 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.829 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.829 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.829 INFO:teuthology.orchestra.run.smithi090.stdout: "ssl": { 2024-04-25T11:47:03.829 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ssl", 2024-04-25T11:47:03.829 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.829 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.829 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.829 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "false", 2024-04-25T11:47:03.829 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.829 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.829 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.829 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T11:47:03.829 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: "threads": { 2024-04-25T11:47:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "threads", 2024-04-25T11:47:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5", 2024-04-25T11:47:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1", 2024-04-25T11:47:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "32", 2024-04-25T11:47:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T11:47:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: "username": { 2024-04-25T11:47:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "username", 2024-04-25T11:47:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "username of InfluxDB server user", 2024-04-25T11:47:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: "verify_ssl": { 2024-04-25T11:47:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "verify_ssl", 2024-04-25T11:47:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "true", 2024-04-25T11:47:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T11:47:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "insights", 2024-04-25T11:47:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.835 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.835 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.835 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.835 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.835 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.835 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.835 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.835 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.835 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.835 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.835 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.835 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.835 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.836 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.836 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.836 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.836 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.836 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.836 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.836 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.836 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.836 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.836 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.836 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.836 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.836 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.836 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.837 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.837 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.837 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.837 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.837 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.837 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.837 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.837 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.837 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.837 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.837 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.837 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.837 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.837 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.838 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.838 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.838 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.838 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.838 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.838 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.838 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.838 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.838 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.838 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.838 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.838 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.838 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.839 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.839 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.839 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.839 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.839 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.839 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.839 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.839 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.839 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.839 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.839 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.839 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "iostat", 2024-04-25T11:47:03.839 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.840 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.840 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.840 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.840 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.840 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.840 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.840 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.840 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.840 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.840 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.840 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.840 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.840 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.840 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.840 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.840 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.841 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.841 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.841 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.841 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.841 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.841 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.841 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.841 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.841 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.841 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.841 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.841 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.841 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.841 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "k8sevents", 2024-04-25T11:47:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_event_retention_days": { 2024-04-25T11:47:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ceph_event_retention_days", 2024-04-25T11:47:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "7", 2024-04-25T11:47:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T11:47:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "config_check_secs": { 2024-04-25T11:47:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "config_check_secs", 2024-04-25T11:47:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-25T11:47:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "10", 2024-04-25T11:47:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T11:47:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "localpool", 2024-04-25T11:47:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "failure_domain": { 2024-04-25T11:47:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "failure_domain", 2024-04-25T11:47:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "host", 2024-04-25T11:47:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "failure domain for any created local pool", 2024-04-25T11:47:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T11:47:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": { 2024-04-25T11:47:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "min_size", 2024-04-25T11:47:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "default min_size for any created local pool", 2024-04-25T11:47:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T11:47:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: "num_rep": { 2024-04-25T11:47:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "num_rep", 2024-04-25T11:47:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "3", 2024-04-25T11:47:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "default replica count for any created local pool", 2024-04-25T11:47:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": { 2024-04-25T11:47:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "pg_num", 2024-04-25T11:47:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "128", 2024-04-25T11:47:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T11:47:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: "prefix": { 2024-04-25T11:47:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "prefix", 2024-04-25T11:47:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "name prefix for any created local pool", 2024-04-25T11:47:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "subtree": { 2024-04-25T11:47:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "subtree", 2024-04-25T11:47:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "rack", 2024-04-25T11:47:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T11:47:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T11:47:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mds_autoscaler", 2024-04-25T11:47:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mirroring", 2024-04-25T11:47:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "nfs", 2024-04-25T11:47:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "orchestrator", 2024-04-25T11:47:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "fail_fs": { 2024-04-25T11:47:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "fail_fs", 2024-04-25T11:47:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T11:47:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "orchestrator": { 2024-04-25T11:47:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "orchestrator", 2024-04-25T11:47:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "cephadm", 2024-04-25T11:47:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "rook", 2024-04-25T11:47:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "test_orchestrator" 2024-04-25T11:47:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Orchestrator backend", 2024-04-25T11:47:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "osd_perf_query", 2024-04-25T11:47:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "osd_support", 2024-04-25T11:47:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "pg_autoscaler", 2024-04-25T11:47:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-25T11:47:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-25T11:47:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-25T11:47:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "threshold": { 2024-04-25T11:47:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "threshold", 2024-04-25T11:47:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-25T11:47:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "3.0", 2024-04-25T11:47:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1.0", 2024-04-25T11:47:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "scaling threshold", 2024-04-25T11:47:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T11:47:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "progress", 2024-04-25T11:47:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: "allow_pg_recovery_event": { 2024-04-25T11:47:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "allow_pg_recovery_event", 2024-04-25T11:47:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T11:47:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "enabled": { 2024-04-25T11:47:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "enabled", 2024-04-25T11:47:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "max_completed_events": { 2024-04-25T11:47:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_completed_events", 2024-04-25T11:47:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "50", 2024-04-25T11:47:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "number of past completed events to remember", 2024-04-25T11:47:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-25T11:47:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-25T11:47:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5", 2024-04-25T11:47:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how long the module is going to sleep", 2024-04-25T11:47:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "prometheus", 2024-04-25T11:47:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "cache": { 2024-04-25T11:47:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cache", 2024-04-25T11:47:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "exclude_perf_counters": { 2024-04-25T11:47:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "exclude_perf_counters", 2024-04-25T11:47:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T11:47:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T11:47:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_stats_pools": { 2024-04-25T11:47:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rbd_stats_pools", 2024-04-25T11:47:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T11:47:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T11:47:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "300", 2024-04-25T11:47:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "scrape_interval": { 2024-04-25T11:47:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "scrape_interval", 2024-04-25T11:47:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-25T11:47:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "15.0", 2024-04-25T11:47:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "server_addr": { 2024-04-25T11:47:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_addr", 2024-04-25T11:47:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "::", 2024-04-25T11:47:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T11:47:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "server_port": { 2024-04-25T11:47:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_port", 2024-04-25T11:47:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "9283", 2024-04-25T11:47:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T11:47:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.936 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.936 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.936 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.936 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.936 INFO:teuthology.orchestra.run.smithi090.stdout: "stale_cache_strategy": { 2024-04-25T11:47:03.936 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "stale_cache_strategy", 2024-04-25T11:47:03.936 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.936 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.936 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.936 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "log", 2024-04-25T11:47:03.936 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.936 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.936 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.937 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.937 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.937 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.937 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.937 INFO:teuthology.orchestra.run.smithi090.stdout: "standby_behaviour": { 2024-04-25T11:47:03.937 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "standby_behaviour", 2024-04-25T11:47:03.937 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.937 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.937 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.937 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "default", 2024-04-25T11:47:03.937 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.937 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.937 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.938 INFO:teuthology.orchestra.run.smithi090.stdout: "default", 2024-04-25T11:47:03.938 INFO:teuthology.orchestra.run.smithi090.stdout: "error" 2024-04-25T11:47:03.938 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.938 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.938 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.938 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.938 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.938 INFO:teuthology.orchestra.run.smithi090.stdout: "standby_error_status_code": { 2024-04-25T11:47:03.938 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "standby_error_status_code", 2024-04-25T11:47:03.938 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.938 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.938 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.938 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "500", 2024-04-25T11:47:03.939 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "400", 2024-04-25T11:47:03.939 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "599", 2024-04-25T11:47:03.939 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.939 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.939 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.939 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.939 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.939 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.939 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.939 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rbd_support", 2024-04-25T11:47:03.939 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.940 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.940 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.940 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.940 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.940 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.940 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.940 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.940 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.940 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.940 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.940 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.940 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.940 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.940 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.940 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.941 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.941 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.941 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.941 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.941 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.941 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.941 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.941 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.941 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.941 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.941 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.941 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.941 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.941 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.941 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.942 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.942 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.942 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.942 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.942 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.942 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.942 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.942 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.942 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.942 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.942 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.942 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.942 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.943 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.943 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.943 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.943 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.943 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.943 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.943 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.943 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.943 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.943 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.943 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.943 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.943 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.944 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.944 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.944 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.944 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.944 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.944 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.944 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.944 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.944 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.944 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.944 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.944 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.944 INFO:teuthology.orchestra.run.smithi090.stdout: "max_concurrent_snap_create": { 2024-04-25T11:47:03.945 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_concurrent_snap_create", 2024-04-25T11:47:03.945 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.945 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.945 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.945 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-25T11:47:03.945 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.945 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.945 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.945 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.945 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.945 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.945 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.945 INFO:teuthology.orchestra.run.smithi090.stdout: "mirror_snapshot_schedule": { 2024-04-25T11:47:03.946 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T11:47:03.946 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.946 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.946 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.946 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.946 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.946 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.946 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.946 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.946 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.946 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.946 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: "trash_purge_schedule": { 2024-04-25T11:47:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "trash_purge_schedule", 2024-04-25T11:47:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "restful", 2024-04-25T11:47:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: "enable_auth": { 2024-04-25T11:47:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "enable_auth", 2024-04-25T11:47:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: "key_file": { 2024-04-25T11:47:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "key_file", 2024-04-25T11:47:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: "server_addr": { 2024-04-25T11:47:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_addr", 2024-04-25T11:47:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: "server_port": { 2024-04-25T11:47:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_port", 2024-04-25T11:47:03.957 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.957 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.957 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.957 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.957 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.957 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.957 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.957 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.957 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.957 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.957 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.957 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.957 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.957 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.958 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.958 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.958 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.958 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.958 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.958 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.958 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.958 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.958 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.958 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.958 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.958 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.958 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rgw", 2024-04-25T11:47:03.959 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.959 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.959 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.959 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.959 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.959 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.959 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.959 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.959 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.959 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.959 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.959 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.959 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.959 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.960 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.960 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.960 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.960 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.960 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.960 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.960 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.960 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.960 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.960 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.960 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.960 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.960 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.960 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.961 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.961 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.961 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.961 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.961 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.961 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.961 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.961 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.961 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.961 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.961 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.961 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.961 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.962 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.962 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.962 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.962 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.962 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.962 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.962 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.962 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.962 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.962 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.962 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.962 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.962 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.962 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.963 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.963 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.963 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.963 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.963 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.963 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.963 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.963 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.963 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.963 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.963 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.963 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.963 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.964 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.964 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.964 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.964 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.964 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.964 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.964 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.964 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.964 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.964 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.964 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.964 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.964 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.964 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.965 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.965 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.965 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rook", 2024-04-25T11:47:03.965 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.965 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.965 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.965 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.965 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.965 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.965 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.965 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.965 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.965 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.966 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.966 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.966 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.966 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.966 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.966 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.966 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.966 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.966 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.966 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.966 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.966 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.966 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.967 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.967 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.967 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.967 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.967 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.967 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.967 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.967 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.967 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.967 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.967 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.967 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.967 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.967 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.968 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.968 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.968 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.968 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.968 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.968 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.968 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.968 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.968 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.968 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.968 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.968 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.968 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.968 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.968 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.969 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.969 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.969 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.969 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.969 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.969 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.969 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.969 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.969 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.969 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.969 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.969 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.969 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.970 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.970 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.970 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.970 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.970 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.970 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.970 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.970 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.970 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.970 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.970 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.970 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.970 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.971 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.971 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.971 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.971 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.971 INFO:teuthology.orchestra.run.smithi090.stdout: "storage_class": { 2024-04-25T11:47:03.971 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "storage_class", 2024-04-25T11:47:03.971 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.971 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.971 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.971 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "local", 2024-04-25T11:47:03.971 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.971 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.971 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.972 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T11:47:03.972 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.972 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.972 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.972 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.972 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.972 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "selftest", 2024-04-25T11:47:03.972 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.972 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.972 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.972 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.973 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.973 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.973 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.973 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.973 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.973 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.973 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.973 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.973 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.973 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.973 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.973 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.973 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.973 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.973 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.974 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.974 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.974 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.974 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.974 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.974 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.974 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.974 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.974 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.974 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.974 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.974 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.974 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.975 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.975 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.975 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.975 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.975 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.975 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.975 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.975 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.975 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.975 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.975 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.726+0000 7fb40d7b2640 1 -- 172.21.15.90:0/4183624255 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb3f00afc60 msgr2=0x7fb3f00b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:03.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.726+0000 7fb40d7b2640 1 --2- 172.21.15.90:0/4183624255 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb3f00afc60 0x7fb3f00b2170 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc005ab0 tx=0x7fb3fc005a20 comp rx=0 tx=0).stop 2024-04-25T11:47:03.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.726+0000 7fb40d7b2640 1 -- 172.21.15.90:0/4183624255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb40810b6b0 msgr2=0x7fb4081a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:03.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.726+0000 7fb40d7b2640 1 --2- 172.21.15.90:0/4183624255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb40810b6b0 0x7fb4081a2b80 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fb3f800d770 tx=0x7fb3f800dc30 comp rx=0 tx=0).stop 2024-04-25T11:47:03.976 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.977 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.977 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.977 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.977 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.977 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.977 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.977 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.977 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.977 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.977 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.977 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.977 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.977 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.977 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.978 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.978 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.978 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.978 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.978 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.978 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.978 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.978 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.978 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.978 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.978 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.978 INFO:teuthology.orchestra.run.smithi090.stdout: "roption1": { 2024-04-25T11:47:03.978 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "roption1", 2024-04-25T11:47:03.979 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.979 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.979 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.979 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.979 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.979 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.979 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.979 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.979 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.979 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.979 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.979 INFO:teuthology.orchestra.run.smithi090.stdout: "roption2": { 2024-04-25T11:47:03.979 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "roption2", 2024-04-25T11:47:03.979 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.979 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.980 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.980 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "xyz", 2024-04-25T11:47:03.980 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.980 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.980 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.980 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.980 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.980 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.980 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.980 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption1": { 2024-04-25T11:47:03.980 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption1", 2024-04-25T11:47:03.980 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.980 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.980 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.981 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.981 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.981 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.981 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.981 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.981 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.981 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.981 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.981 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption2": { 2024-04-25T11:47:03.981 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption2", 2024-04-25T11:47:03.981 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.981 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.981 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.981 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.981 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.982 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.982 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.982 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.982 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.982 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.982 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.982 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption3": { 2024-04-25T11:47:03.982 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption3", 2024-04-25T11:47:03.982 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-25T11:47:03.982 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.982 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.982 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.982 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.983 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.983 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.983 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.983 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.983 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.983 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.983 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption4": { 2024-04-25T11:47:03.983 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption4", 2024-04-25T11:47:03.983 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.983 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.983 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.983 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.983 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.984 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.984 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.984 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.984 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.984 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.984 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.984 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption5": { 2024-04-25T11:47:03.984 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption5", 2024-04-25T11:47:03.984 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.984 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.984 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.984 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.984 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.984 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.984 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.985 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.985 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.985 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.985 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.985 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption6": { 2024-04-25T11:47:03.985 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption6", 2024-04-25T11:47:03.985 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.985 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.985 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.985 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:03.985 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.985 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.985 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.985 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.986 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.986 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.986 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.986 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption7": { 2024-04-25T11:47:03.986 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption7", 2024-04-25T11:47:03.986 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.986 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.986 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.986 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.986 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1", 2024-04-25T11:47:03.986 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "42", 2024-04-25T11:47:03.986 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.987 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.987 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.987 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.987 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.987 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.987 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.987 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.987 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.987 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.987 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.987 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.987 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.987 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.987 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.988 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.988 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.988 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.988 INFO:teuthology.orchestra.run.smithi090.stdout: "testkey": { 2024-04-25T11:47:03.988 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "testkey", 2024-04-25T11:47:03.988 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.988 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.988 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.988 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.988 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.988 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.988 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.988 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.988 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.989 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.989 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.989 INFO:teuthology.orchestra.run.smithi090.stdout: "testlkey": { 2024-04-25T11:47:03.989 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "testlkey", 2024-04-25T11:47:03.989 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.989 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.989 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.989 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.989 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.989 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.989 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.989 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.989 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.989 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.990 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.990 INFO:teuthology.orchestra.run.smithi090.stdout: "testnewline": { 2024-04-25T11:47:03.990 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "testnewline", 2024-04-25T11:47:03.990 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.990 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.990 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:03.990 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.990 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.990 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.990 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.990 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.990 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.991 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.991 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.991 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.991 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.991 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "snap_schedule", 2024-04-25T11:47:03.991 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.991 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.991 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.991 INFO:teuthology.orchestra.run.smithi090.stdout: "allow_m_granularity": { 2024-04-25T11:47:03.991 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "allow_m_granularity", 2024-04-25T11:47:03.991 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.991 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.991 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.991 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.992 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.992 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.992 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.992 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T11:47:03.992 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.992 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.992 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.992 INFO:teuthology.orchestra.run.smithi090.stdout: "dump_on_update": { 2024-04-25T11:47:03.992 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "dump_on_update", 2024-04-25T11:47:03.992 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.992 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.992 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.992 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.992 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.993 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.993 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.993 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "dump database to debug log on update", 2024-04-25T11:47:03.993 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.993 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.993 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.993 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.993 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.993 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.993 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.993 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.993 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:03.993 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.993 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.994 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.994 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.994 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.994 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.994 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.994 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.994 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.994 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.994 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.994 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.994 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.994 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.994 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:03.995 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:03.995 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.995 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.995 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.995 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.995 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.995 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.995 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.995 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.995 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.995 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.995 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.995 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:03.996 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:03.996 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.996 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.996 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.996 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:03.996 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.996 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.996 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:03.996 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:03.996 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:03.996 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:03.996 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:03.996 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:03.996 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:03.996 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:03.996 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.997 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.997 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.997 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.997 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:03.997 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:03.997 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:03.997 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:03.997 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.997 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:03.997 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.997 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.997 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.997 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.997 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.998 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.998 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.998 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:03.998 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:03.998 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:03.998 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:03.998 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:03.998 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:03.998 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:03.998 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:03.998 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:03.998 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:03.998 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:03.999 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:03.999 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:03.999 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.999 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:03.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:03.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:03.999 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "stats", 2024-04-25T11:47:03.999 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:03.999 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:03.999 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:03.999 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:03.999 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:03.999 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:03.999 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.000 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.000 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:04.000 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.000 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.000 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:04.000 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:04.000 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:04.000 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:04.000 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:04.000 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:04.000 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:04.000 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:04.000 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.000 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.001 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.001 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.001 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:04.001 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:04.001 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:04.001 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.001 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.001 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:04.001 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.001 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.001 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.001 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.001 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.001 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.001 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.002 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:04.002 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:04.002 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:04.002 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.002 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.002 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:04.002 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.002 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.002 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:04.002 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:04.002 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:04.002 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:04.002 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:04.003 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:04.003 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:04.003 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:04.003 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.003 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.003 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.003 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.003 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:04.003 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:04.003 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:04.003 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.003 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.003 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:04.004 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.004 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.004 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.004 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.004 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.004 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.004 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.004 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:04.004 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:04.004 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:04.004 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:04.004 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.004 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:04.005 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.005 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.005 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.005 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.005 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.005 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.005 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.005 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:04.005 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:04.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:04.005 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "status", 2024-04-25T11:47:04.005 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:04.005 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:04.005 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:04.005 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:04.006 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:04.006 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:04.006 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.006 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.006 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:04.006 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.006 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.006 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:04.006 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:04.006 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:04.006 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:04.006 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:04.006 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:04.006 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:04.007 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:04.007 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.007 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.007 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.007 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.007 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:04.007 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:04.007 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:04.007 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.007 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.007 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:04.007 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.007 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.008 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.008 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.008 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.008 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.008 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.008 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:04.008 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:04.008 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:04.008 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.008 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.008 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:04.008 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.008 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.008 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:04.008 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:04.009 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:04.009 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:04.009 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:04.009 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:04.009 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:04.009 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:04.009 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.009 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.009 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.009 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.009 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:04.009 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:04.009 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:04.009 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.010 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.010 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:04.010 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.010 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.010 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.010 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.010 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.010 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.010 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.010 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:04.010 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:04.010 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:04.010 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:04.011 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.011 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:04.011 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.011 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.011 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.011 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.011 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.011 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.011 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.011 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:04.011 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:04.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:04.011 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "telegraf", 2024-04-25T11:47:04.011 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:04.012 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:04.012 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:04.012 INFO:teuthology.orchestra.run.smithi090.stdout: "address": { 2024-04-25T11:47:04.012 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "address", 2024-04-25T11:47:04.012 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:04.012 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.012 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.012 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T11:47:04.012 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.012 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.012 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.012 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.012 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.012 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.013 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.013 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-25T11:47:04.013 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-25T11:47:04.013 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:04.013 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.013 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.013 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "15", 2024-04-25T11:47:04.013 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.013 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.013 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.013 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.013 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.013 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.013 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.014 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:04.014 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:04.014 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:04.014 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.014 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.014 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:04.014 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.014 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.014 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:04.014 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:04.014 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:04.014 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:04.015 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:04.015 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:04.015 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:04.015 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:04.015 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.015 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.015 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.015 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.015 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:04.015 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:04.015 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:04.015 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.015 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.015 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:04.016 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.016 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.016 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.016 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.016 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.016 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.016 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.016 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:04.016 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:04.016 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:04.016 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.016 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.016 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:04.016 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.016 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.017 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:04.017 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:04.017 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:04.017 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:04.017 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:04.017 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:04.017 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:04.017 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:04.017 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.017 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.017 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.017 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.017 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:04.017 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:04.017 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:04.018 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.018 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.018 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:04.018 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.018 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.018 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.018 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.018 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.018 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.018 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.018 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:04.018 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:04.019 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:04.019 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:04.019 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.019 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:04.019 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.019 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.019 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.019 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.019 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.019 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.019 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.019 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:04.019 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:04.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:04.020 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "telemetry", 2024-04-25T11:47:04.020 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:04.020 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:04.020 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:04.020 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_basic": { 2024-04-25T11:47:04.020 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_basic", 2024-04-25T11:47:04.020 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:04.020 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.020 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.020 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:04.020 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.020 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.020 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.020 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T11:47:04.020 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.020 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.021 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.021 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_crash": { 2024-04-25T11:47:04.021 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_crash", 2024-04-25T11:47:04.021 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:04.021 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.021 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.021 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:04.021 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.021 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.021 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.021 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T11:47:04.021 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.021 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.021 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.022 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_device": { 2024-04-25T11:47:04.022 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_device", 2024-04-25T11:47:04.022 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:04.022 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.022 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.022 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:04.022 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.022 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.022 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.022 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T11:47:04.022 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.022 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.022 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.023 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_ident": { 2024-04-25T11:47:04.023 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_ident", 2024-04-25T11:47:04.023 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:04.023 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.023 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.023 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:04.023 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.023 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.023 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.023 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T11:47:04.023 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.023 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.023 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.024 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_perf": { 2024-04-25T11:47:04.024 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_perf", 2024-04-25T11:47:04.024 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:04.024 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.024 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.024 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:04.024 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.024 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.024 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.024 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T11:47:04.024 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.024 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.024 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.025 INFO:teuthology.orchestra.run.smithi090.stdout: "contact": { 2024-04-25T11:47:04.025 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "contact", 2024-04-25T11:47:04.025 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:04.025 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.025 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.025 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:04.025 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.025 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.025 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.025 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.025 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.025 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.025 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.026 INFO:teuthology.orchestra.run.smithi090.stdout: "description": { 2024-04-25T11:47:04.026 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "description", 2024-04-25T11:47:04.026 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:04.026 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.026 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.026 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:04.026 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.026 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.026 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.026 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.026 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.026 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.026 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.027 INFO:teuthology.orchestra.run.smithi090.stdout: "device_url": { 2024-04-25T11:47:04.027 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "device_url", 2024-04-25T11:47:04.027 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:04.027 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.027 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.027 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T11:47:04.027 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.027 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.027 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.027 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.027 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.027 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.027 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.028 INFO:teuthology.orchestra.run.smithi090.stdout: "enabled": { 2024-04-25T11:47:04.028 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "enabled", 2024-04-25T11:47:04.028 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:04.028 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.028 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.028 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:04.028 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.028 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.028 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.028 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.028 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.028 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.028 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.028 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-25T11:47:04.029 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-25T11:47:04.029 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:04.029 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.029 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.029 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "24", 2024-04-25T11:47:04.029 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "8", 2024-04-25T11:47:04.029 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.029 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.029 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.029 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.029 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.029 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.029 INFO:teuthology.orchestra.run.smithi090.stdout: "last_opt_revision": { 2024-04-25T11:47:04.029 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "last_opt_revision", 2024-04-25T11:47:04.030 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:04.030 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.030 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.030 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "1", 2024-04-25T11:47:04.030 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.030 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.030 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.030 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.030 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.030 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.030 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.030 INFO:teuthology.orchestra.run.smithi090.stdout: "leaderboard": { 2024-04-25T11:47:04.030 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "leaderboard", 2024-04-25T11:47:04.031 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:04.031 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.031 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.031 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:04.031 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.031 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.031 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.031 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.031 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.031 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.031 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.031 INFO:teuthology.orchestra.run.smithi090.stdout: "leaderboard_description": { 2024-04-25T11:47:04.031 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "leaderboard_description", 2024-04-25T11:47:04.031 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:04.032 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.032 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.032 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:04.032 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.032 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.032 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.032 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.032 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.032 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.032 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.032 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:04.032 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:04.032 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:04.032 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.033 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.033 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:04.033 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.033 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.033 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:04.033 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:04.033 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:04.033 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:04.033 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:04.033 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:04.033 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:04.033 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:04.033 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.033 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.034 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.034 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.034 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:04.034 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:04.034 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:04.034 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.034 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.034 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:04.034 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.034 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.034 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.034 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.035 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.035 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.035 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.035 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:04.035 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:04.035 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:04.035 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.035 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.035 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:04.035 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.035 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.035 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:04.035 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:04.035 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:04.035 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:04.036 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:04.036 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:04.036 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:04.036 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:04.036 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.036 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.036 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.036 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.036 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:04.036 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:04.036 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:04.036 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.036 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.036 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:04.037 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.037 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.037 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.037 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.037 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.037 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.037 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.037 INFO:teuthology.orchestra.run.smithi090.stdout: "organization": { 2024-04-25T11:47:04.037 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "organization", 2024-04-25T11:47:04.037 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:04.037 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.037 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.037 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:04.037 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.038 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.038 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.038 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.038 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.038 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.038 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.038 INFO:teuthology.orchestra.run.smithi090.stdout: "proxy": { 2024-04-25T11:47:04.038 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "proxy", 2024-04-25T11:47:04.038 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:04.038 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.038 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.038 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:04.038 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.039 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.039 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.039 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.039 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.039 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.039 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.039 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:04.039 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:04.039 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:04.039 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:04.039 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.039 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:04.039 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.039 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.040 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.040 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.040 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.040 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.040 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.040 INFO:teuthology.orchestra.run.smithi090.stdout: "url": { 2024-04-25T11:47:04.040 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "url", 2024-04-25T11:47:04.040 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:04.040 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.040 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.040 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T11:47:04.040 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.040 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.041 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.041 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.041 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.041 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.041 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.041 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:04.041 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:04.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:04.041 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "test_orchestrator", 2024-04-25T11:47:04.041 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:04.041 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:04.041 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:04.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.726+0000 7fb40d7b2640 1 -- 172.21.15.90:0/4183624255 shutdown_connections 2024-04-25T11:47:04.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.726+0000 7fb40d7b2640 1 --2- 172.21.15.90:0/4183624255 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb3f00afc60 0x7fb3f00b2170 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:04.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.726+0000 7fb40d7b2640 1 --2- 172.21.15.90:0/4183624255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb40810e060 0x7fb40819c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:04.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.726+0000 7fb40d7b2640 1 --2- 172.21.15.90:0/4183624255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb40810b6b0 0x7fb4081a2b80 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:04.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.726+0000 7fb40d7b2640 1 --2- 172.21.15.90:0/4183624255 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb408104c50 0x7fb4081a2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:04.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.726+0000 7fb40d7b2640 1 -- 172.21.15.90:0/4183624255 >> 172.21.15.90:0/4183624255 conn(0x7fb408078110 msgr2=0x7fb408109a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "volumes", 2024-04-25T11:47:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: "max_concurrent_clones": { 2024-04-25T11:47:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_concurrent_clones", 2024-04-25T11:47:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "4", 2024-04-25T11:47:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T11:47:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "periodic_async_work": { 2024-04-25T11:47:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "periodic_async_work", 2024-04-25T11:47:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Periodically check for async work", 2024-04-25T11:47:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: "snapshot_clone_delay": { 2024-04-25T11:47:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "snapshot_clone_delay", 2024-04-25T11:47:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T11:47:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "snapshot_clone_no_wait": { 2024-04-25T11:47:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T11:47:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-25T11:47:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T11:47:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "zabbix", 2024-04-25T11:47:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-25T11:47:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-25T11:47:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-25T11:47:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "discovery_interval": { 2024-04-25T11:47:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "discovery_interval", 2024-04-25T11:47:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-25T11:47:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "100", 2024-04-25T11:47:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: "identifier": { 2024-04-25T11:47:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "identifier", 2024-04-25T11:47:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-25T11:47:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-25T11:47:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-25T11:47:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-25T11:47:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-25T11:47:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-25T11:47:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-25T11:47:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-25T11:47:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.064 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.064 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:04.064 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.064 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.064 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.064 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.064 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.064 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.064 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.064 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-25T11:47:04.064 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-25T11:47:04.064 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:04.064 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.064 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-25T11:47:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-25T11:47:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-25T11:47:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-25T11:47:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-25T11:47:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-25T11:47:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-25T11:47:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-25T11:47:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-25T11:47:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-25T11:47:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-25T11:47:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-25T11:47:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.067 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.067 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.067 INFO:teuthology.orchestra.run.smithi090.stdout: "sqlite3_killpoint": { 2024-04-25T11:47:04.067 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sqlite3_killpoint", 2024-04-25T11:47:04.067 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:04.067 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "dev", 2024-04-25T11:47:04.067 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-25T11:47:04.067 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-25T11:47:04.067 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.067 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.067 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.067 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.067 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.067 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.068 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.068 INFO:teuthology.orchestra.run.smithi090.stdout: "zabbix_host": { 2024-04-25T11:47:04.068 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "zabbix_host", 2024-04-25T11:47:04.068 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:04.068 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.068 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.068 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-25T11:47:04.068 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.068 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.068 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.068 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.068 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.068 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.068 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.069 INFO:teuthology.orchestra.run.smithi090.stdout: "zabbix_port": { 2024-04-25T11:47:04.069 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "zabbix_port", 2024-04-25T11:47:04.069 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-25T11:47:04.069 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.069 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.069 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10051", 2024-04-25T11:47:04.069 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.069 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.069 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.069 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.069 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.069 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.069 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.070 INFO:teuthology.orchestra.run.smithi090.stdout: "zabbix_sender": { 2024-04-25T11:47:04.070 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "zabbix_sender", 2024-04-25T11:47:04.070 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-25T11:47:04.070 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-25T11:47:04.070 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-25T11:47:04.070 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T11:47:04.070 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-25T11:47:04.070 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-25T11:47:04.070 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-25T11:47:04.070 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-25T11:47:04.070 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-25T11:47:04.070 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-25T11:47:04.071 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-25T11:47:04.071 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:04.071 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:04.071 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:04.071 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:04.071 INFO:teuthology.orchestra.run.smithi090.stdout: "services": { 2024-04-25T11:47:04.071 INFO:teuthology.orchestra.run.smithi090.stdout: "dashboard": "https://172.21.15.90:8443/" 2024-04-25T11:47:04.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.071 INFO:teuthology.orchestra.run.smithi090.stdout: "always_on_modules": { 2024-04-25T11:47:04.071 INFO:teuthology.orchestra.run.smithi090.stdout: "octopus": [ 2024-04-25T11:47:04.071 INFO:teuthology.orchestra.run.smithi090.stdout: "balancer", 2024-04-25T11:47:04.071 INFO:teuthology.orchestra.run.smithi090.stdout: "crash", 2024-04-25T11:47:04.071 INFO:teuthology.orchestra.run.smithi090.stdout: "devicehealth", 2024-04-25T11:47:04.071 INFO:teuthology.orchestra.run.smithi090.stdout: "orchestrator", 2024-04-25T11:47:04.071 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscaler", 2024-04-25T11:47:04.072 INFO:teuthology.orchestra.run.smithi090.stdout: "progress", 2024-04-25T11:47:04.072 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_support", 2024-04-25T11:47:04.072 INFO:teuthology.orchestra.run.smithi090.stdout: "status", 2024-04-25T11:47:04.072 INFO:teuthology.orchestra.run.smithi090.stdout: "telemetry", 2024-04-25T11:47:04.072 INFO:teuthology.orchestra.run.smithi090.stdout: "volumes" 2024-04-25T11:47:04.072 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:04.072 INFO:teuthology.orchestra.run.smithi090.stdout: "pacific": [ 2024-04-25T11:47:04.072 INFO:teuthology.orchestra.run.smithi090.stdout: "balancer", 2024-04-25T11:47:04.072 INFO:teuthology.orchestra.run.smithi090.stdout: "crash", 2024-04-25T11:47:04.072 INFO:teuthology.orchestra.run.smithi090.stdout: "devicehealth", 2024-04-25T11:47:04.072 INFO:teuthology.orchestra.run.smithi090.stdout: "orchestrator", 2024-04-25T11:47:04.072 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscaler", 2024-04-25T11:47:04.072 INFO:teuthology.orchestra.run.smithi090.stdout: "progress", 2024-04-25T11:47:04.072 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_support", 2024-04-25T11:47:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: "status", 2024-04-25T11:47:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: "telemetry", 2024-04-25T11:47:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: "volumes" 2024-04-25T11:47:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: "quincy": [ 2024-04-25T11:47:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: "balancer", 2024-04-25T11:47:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: "crash", 2024-04-25T11:47:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: "devicehealth", 2024-04-25T11:47:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: "orchestrator", 2024-04-25T11:47:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscaler", 2024-04-25T11:47:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: "progress", 2024-04-25T11:47:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_support", 2024-04-25T11:47:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: "status", 2024-04-25T11:47:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: "telemetry", 2024-04-25T11:47:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "volumes" 2024-04-25T11:47:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "reef": [ 2024-04-25T11:47:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "balancer", 2024-04-25T11:47:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "crash", 2024-04-25T11:47:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "devicehealth", 2024-04-25T11:47:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "orchestrator", 2024-04-25T11:47:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscaler", 2024-04-25T11:47:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "progress", 2024-04-25T11:47:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_support", 2024-04-25T11:47:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "status", 2024-04-25T11:47:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "telemetry", 2024-04-25T11:47:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "volumes" 2024-04-25T11:47:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-25T11:47:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "squid": [ 2024-04-25T11:47:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "balancer", 2024-04-25T11:47:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "crash", 2024-04-25T11:47:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "devicehealth", 2024-04-25T11:47:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "orchestrator", 2024-04-25T11:47:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscaler", 2024-04-25T11:47:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "progress", 2024-04-25T11:47:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_support", 2024-04-25T11:47:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "status", 2024-04-25T11:47:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "telemetry", 2024-04-25T11:47:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "volumes" 2024-04-25T11:47:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-25T11:47:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "last_failure_osd_epoch": 80, 2024-04-25T11:47:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "active_clients": [ 2024-04-25T11:47:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "devicehealth", 2024-04-25T11:47:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-25T11:47:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-25T11:47:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:0", 2024-04-25T11:47:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1338716598 2024-04-25T11:47:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-25T11:47:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "libcephsqlite", 2024-04-25T11:47:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-25T11:47:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-25T11:47:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:0", 2024-04-25T11:47:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2543750937 2024-04-25T11:47:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-25T11:47:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rbd_support", 2024-04-25T11:47:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-25T11:47:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-25T11:47:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:0", 2024-04-25T11:47:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4248337631 2024-04-25T11:47:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-25T11:47:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "snap_schedule", 2024-04-25T11:47:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-25T11:47:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-25T11:47:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:0", 2024-04-25T11:47:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 360786711 2024-04-25T11:47:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-25T11:47:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "snap_schedule", 2024-04-25T11:47:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-25T11:47:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-25T11:47:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:0", 2024-04-25T11:47:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3562749241 2024-04-25T11:47:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-25T11:47:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "volumes", 2024-04-25T11:47:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-25T11:47:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-25T11:47:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-25T11:47:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:0", 2024-04-25T11:47:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 39821726 2024-04-25T11:47:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-25T11:47:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-25T11:47:04.080 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:47:04.080 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:03 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3900307528' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:47:04.080 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:03 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/4183624255' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T11:47:04.080 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:03 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3900307528' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:47:04.080 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:03 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/4183624255' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T11:47:04.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.726+0000 7fb40d7b2640 1 -- 172.21.15.90:0/4183624255 shutdown_connections 2024-04-25T11:47:04.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:02.726+0000 7fb40d7b2640 1 -- 172.21.15.90:0/4183624255 wait complete. 2024-04-25T11:47:04.087 DEBUG:tasks.check_counter:Getting stats from a 2024-04-25T11:47:04.087 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-25T11:47:04.088 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:03 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3900307528' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:47:04.088 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:03 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/4183624255' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T11:47:04.350 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:47:04.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:04 smithi090 ceph-mon[29575]: pgmap v372: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 15 KiB/s rd, 8.9 KiB/s wr, 228 op/s 2024-04-25T11:47:04.486 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:04 smithi156 ceph-mon[33045]: pgmap v372: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 15 KiB/s rd, 8.9 KiB/s wr, 228 op/s 2024-04-25T11:47:04.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:04 smithi138 ceph-mon[32328]: pgmap v372: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 15 KiB/s rd, 8.9 KiB/s wr, 228 op/s 2024-04-25T11:47:06.216 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:47:06.216 INFO:teuthology.orchestra.run.smithi090.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T11:47:06.216 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_messages": 993, 2024-04-25T11:47:06.216 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages": 995, 2024-04-25T11:47:06.216 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_bytes": 322055, 2024-04-25T11:47:06.217 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_bytes": 315775, 2024-04-25T11:47:06.217 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_created_connections": 6, 2024-04-25T11:47:06.217 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_active_connections": 2, 2024-04-25T11:47:06.217 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_total_time": 0.122162897, 2024-04-25T11:47:06.217 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_send_time": 0.035872279, 2024-04-25T11:47:06.217 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_recv_time": 0.036172635, 2024-04-25T11:47:06.217 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_fast_dispatch_time": 0.040116848, 2024-04-25T11:47:06.217 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:06.217 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 993, 2024-04-25T11:47:06.217 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.061176773, 2024-04-25T11:47:06.217 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000061608 2024-04-25T11:47:06.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.217 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:06.217 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.218 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.218 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.218 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T11:47:06.218 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T11:47:06.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.218 INFO:teuthology.orchestra.run.smithi090.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T11:47:06.218 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_messages": 969, 2024-04-25T11:47:06.218 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages": 955, 2024-04-25T11:47:06.218 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_bytes": 1770602, 2024-04-25T11:47:06.218 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_bytes": 367946, 2024-04-25T11:47:06.218 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_created_connections": 4, 2024-04-25T11:47:06.218 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_active_connections": 3, 2024-04-25T11:47:06.219 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_total_time": 0.130698808, 2024-04-25T11:47:06.219 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_send_time": 0.033833080, 2024-04-25T11:47:06.219 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_recv_time": 0.064306524, 2024-04-25T11:47:06.219 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_fast_dispatch_time": 0.038297626, 2024-04-25T11:47:06.219 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:06.219 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 954, 2024-04-25T11:47:06.219 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.081772316, 2024-04-25T11:47:06.219 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000085715 2024-04-25T11:47:06.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.219 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:06.219 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.219 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.219 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.220 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_encrypted_bytes": 1595536, 2024-04-25T11:47:06.220 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_encrypted_bytes": 142416 2024-04-25T11:47:06.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.220 INFO:teuthology.orchestra.run.smithi090.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T11:47:06.220 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_messages": 655, 2024-04-25T11:47:06.220 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages": 823, 2024-04-25T11:47:06.220 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_bytes": 173754, 2024-04-25T11:47:06.220 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_bytes": 679264, 2024-04-25T11:47:06.220 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_created_connections": 7, 2024-04-25T11:47:06.220 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_active_connections": 5, 2024-04-25T11:47:06.220 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_total_time": 0.088288103, 2024-04-25T11:47:06.220 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_send_time": 0.029780110, 2024-04-25T11:47:06.220 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_recv_time": 0.022959634, 2024-04-25T11:47:06.220 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_fast_dispatch_time": 0.023648039, 2024-04-25T11:47:06.221 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:06.221 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 821, 2024-04-25T11:47:06.221 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.051138756, 2024-04-25T11:47:06.221 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000062288 2024-04-25T11:47:06.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.221 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:06.221 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.221 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.221 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.221 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-25T11:47:06.221 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_encrypted_bytes": 473200 2024-04-25T11:47:06.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.221 INFO:teuthology.orchestra.run.smithi090.stdout: "cct": { 2024-04-25T11:47:06.221 INFO:teuthology.orchestra.run.smithi090.stdout: "total_workers": 1, 2024-04-25T11:47:06.222 INFO:teuthology.orchestra.run.smithi090.stdout: "unhealthy_workers": 0 2024-04-25T11:47:06.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.222 INFO:teuthology.orchestra.run.smithi090.stdout: "finisher-MDSRank": { 2024-04-25T11:47:06.222 INFO:teuthology.orchestra.run.smithi090.stdout: "queue_len": 0, 2024-04-25T11:47:06.222 INFO:teuthology.orchestra.run.smithi090.stdout: "complete_latency": { 2024-04-25T11:47:06.222 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 3884, 2024-04-25T11:47:06.222 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.145073345, 2024-04-25T11:47:06.222 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000037351 2024-04-25T11:47:06.222 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:06.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.222 INFO:teuthology.orchestra.run.smithi090.stdout: "finisher-PurgeQueue": { 2024-04-25T11:47:06.222 INFO:teuthology.orchestra.run.smithi090.stdout: "queue_len": 0, 2024-04-25T11:47:06.222 INFO:teuthology.orchestra.run.smithi090.stdout: "complete_latency": { 2024-04-25T11:47:06.223 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.223 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.223 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.223 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:06.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.223 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": { 2024-04-25T11:47:06.223 INFO:teuthology.orchestra.run.smithi090.stdout: "request": 0, 2024-04-25T11:47:06.223 INFO:teuthology.orchestra.run.smithi090.stdout: "reply": 0, 2024-04-25T11:47:06.223 INFO:teuthology.orchestra.run.smithi090.stdout: "reply_latency": { 2024-04-25T11:47:06.223 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.223 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.223 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.223 INFO:teuthology.orchestra.run.smithi090.stdout: "slow_reply": 0, 2024-04-25T11:47:06.223 INFO:teuthology.orchestra.run.smithi090.stdout: "forward": 0, 2024-04-25T11:47:06.224 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_fetch_complete": 0, 2024-04-25T11:47:06.224 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_fetch_keys": 0, 2024-04-25T11:47:06.224 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_commit": 0, 2024-04-25T11:47:06.224 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_split": 0, 2024-04-25T11:47:06.224 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_merge": 0, 2024-04-25T11:47:06.224 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes": 0, 2024-04-25T11:47:06.224 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes_top": 0, 2024-04-25T11:47:06.224 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes_bottom": 0, 2024-04-25T11:47:06.224 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes_pin_tail": 0, 2024-04-25T11:47:06.224 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes_pinned": 0, 2024-04-25T11:47:06.224 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes_expired": 0, 2024-04-25T11:47:06.224 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes_with_caps": 0, 2024-04-25T11:47:06.224 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": 0, 2024-04-25T11:47:06.224 INFO:teuthology.orchestra.run.smithi090.stdout: "subtrees": 1, 2024-04-25T11:47:06.225 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse": 0, 2024-04-25T11:47:06.225 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse_hit": 0, 2024-04-25T11:47:06.225 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse_forward": 0, 2024-04-25T11:47:06.225 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse_discover": 0, 2024-04-25T11:47:06.225 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse_dir_fetch": 0, 2024-04-25T11:47:06.225 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse_remote_ino": 0, 2024-04-25T11:47:06.225 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse_lock": 0, 2024-04-25T11:47:06.225 INFO:teuthology.orchestra.run.smithi090.stdout: "load_cent": 0, 2024-04-25T11:47:06.225 INFO:teuthology.orchestra.run.smithi090.stdout: "q": 0, 2024-04-25T11:47:06.225 INFO:teuthology.orchestra.run.smithi090.stdout: "exported": 0, 2024-04-25T11:47:06.225 INFO:teuthology.orchestra.run.smithi090.stdout: "exported_inodes": 0, 2024-04-25T11:47:06.225 INFO:teuthology.orchestra.run.smithi090.stdout: "imported": 0, 2024-04-25T11:47:06.225 INFO:teuthology.orchestra.run.smithi090.stdout: "imported_inodes": 0, 2024-04-25T11:47:06.225 INFO:teuthology.orchestra.run.smithi090.stdout: "openino_dir_fetch": 0, 2024-04-25T11:47:06.225 INFO:teuthology.orchestra.run.smithi090.stdout: "openino_backtrace_fetch": 0, 2024-04-25T11:47:06.226 INFO:teuthology.orchestra.run.smithi090.stdout: "openino_peer_discover": 0, 2024-04-25T11:47:06.226 INFO:teuthology.orchestra.run.smithi090.stdout: "root_rfiles": 0, 2024-04-25T11:47:06.226 INFO:teuthology.orchestra.run.smithi090.stdout: "root_rbytes": 0, 2024-04-25T11:47:06.226 INFO:teuthology.orchestra.run.smithi090.stdout: "root_rsnaps": 0, 2024-04-25T11:47:06.226 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T11:47:06.226 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_set_tag": 0, 2024-04-25T11:47:06.226 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T11:47:06.226 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_inotable_repaired": 0, 2024-04-25T11:47:06.226 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_dir_inodes": 0, 2024-04-25T11:47:06.226 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T11:47:06.226 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T11:47:06.226 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_file_inodes": 0, 2024-04-25T11:47:06.226 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_inode_file_caps": 0, 2024-04-25T11:47:06.226 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T11:47:06.227 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_cap_op_grant": 0, 2024-04-25T11:47:06.227 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T11:47:06.227 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T11:47:06.227 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T11:47:06.227 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_client_caps": 0, 2024-04-25T11:47:06.227 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_client_caps_dirty": 0, 2024-04-25T11:47:06.227 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_client_cap_release": 0, 2024-04-25T11:47:06.227 INFO:teuthology.orchestra.run.smithi090.stdout: "process_request_cap_release": 0 2024-04-25T11:47:06.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.227 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache": { 2024-04-25T11:47:06.227 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_update": 0, 2024-04-25T11:47:06.227 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_update_receipt": 0, 2024-04-25T11:47:06.227 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_try_discover": 0, 2024-04-25T11:47:06.228 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_send_discover": 0, 2024-04-25T11:47:06.228 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_handle_discover": 0, 2024-04-25T11:47:06.228 INFO:teuthology.orchestra.run.smithi090.stdout: "num_strays": 0, 2024-04-25T11:47:06.228 INFO:teuthology.orchestra.run.smithi090.stdout: "num_strays_delayed": 0, 2024-04-25T11:47:06.228 INFO:teuthology.orchestra.run.smithi090.stdout: "num_strays_enqueuing": 0, 2024-04-25T11:47:06.228 INFO:teuthology.orchestra.run.smithi090.stdout: "strays_created": 0, 2024-04-25T11:47:06.228 INFO:teuthology.orchestra.run.smithi090.stdout: "strays_enqueued": 0, 2024-04-25T11:47:06.228 INFO:teuthology.orchestra.run.smithi090.stdout: "strays_reintegrated": 0, 2024-04-25T11:47:06.228 INFO:teuthology.orchestra.run.smithi090.stdout: "strays_migrated": 0, 2024-04-25T11:47:06.228 INFO:teuthology.orchestra.run.smithi090.stdout: "num_recovering_processing": 0, 2024-04-25T11:47:06.228 INFO:teuthology.orchestra.run.smithi090.stdout: "num_recovering_enqueued": 0, 2024-04-25T11:47:06.228 INFO:teuthology.orchestra.run.smithi090.stdout: "num_recovering_prioritized": 0, 2024-04-25T11:47:06.228 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_started": 0, 2024-04-25T11:47:06.228 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_completed": 0, 2024-04-25T11:47:06.229 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_quiesce_path": 0, 2024-04-25T11:47:06.229 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_quiesce_inode": 0, 2024-04-25T11:47:06.229 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T11:47:06.229 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_exportdir": 0, 2024-04-25T11:47:06.229 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_flush": 0, 2024-04-25T11:47:06.229 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_fragmentdir": 0, 2024-04-25T11:47:06.229 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_fragstats": 0, 2024-04-25T11:47:06.229 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_inodestats": 0 2024-04-25T11:47:06.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.229 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log": { 2024-04-25T11:47:06.229 INFO:teuthology.orchestra.run.smithi090.stdout: "evlrg": 0, 2024-04-25T11:47:06.229 INFO:teuthology.orchestra.run.smithi090.stdout: "evadd": 0, 2024-04-25T11:47:06.229 INFO:teuthology.orchestra.run.smithi090.stdout: "evex": 0, 2024-04-25T11:47:06.229 INFO:teuthology.orchestra.run.smithi090.stdout: "evtrm": 0, 2024-04-25T11:47:06.230 INFO:teuthology.orchestra.run.smithi090.stdout: "ev": 6, 2024-04-25T11:47:06.230 INFO:teuthology.orchestra.run.smithi090.stdout: "evexg": 0, 2024-04-25T11:47:06.230 INFO:teuthology.orchestra.run.smithi090.stdout: "evexd": 0, 2024-04-25T11:47:06.230 INFO:teuthology.orchestra.run.smithi090.stdout: "segadd": 0, 2024-04-25T11:47:06.230 INFO:teuthology.orchestra.run.smithi090.stdout: "segex": 0, 2024-04-25T11:47:06.230 INFO:teuthology.orchestra.run.smithi090.stdout: "segtrm": 0, 2024-04-25T11:47:06.230 INFO:teuthology.orchestra.run.smithi090.stdout: "seg": 2, 2024-04-25T11:47:06.230 INFO:teuthology.orchestra.run.smithi090.stdout: "segmjr": 2, 2024-04-25T11:47:06.230 INFO:teuthology.orchestra.run.smithi090.stdout: "segexg": 0, 2024-04-25T11:47:06.230 INFO:teuthology.orchestra.run.smithi090.stdout: "segexd": 0, 2024-04-25T11:47:06.230 INFO:teuthology.orchestra.run.smithi090.stdout: "expos": 4194304, 2024-04-25T11:47:06.230 INFO:teuthology.orchestra.run.smithi090.stdout: "wrpos": 4195956, 2024-04-25T11:47:06.230 INFO:teuthology.orchestra.run.smithi090.stdout: "rdpos": 4195815, 2024-04-25T11:47:06.230 INFO:teuthology.orchestra.run.smithi090.stdout: "jlat": { 2024-04-25T11:47:06.231 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.231 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.231 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.231 INFO:teuthology.orchestra.run.smithi090.stdout: "replayed": 6 2024-04-25T11:47:06.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.231 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_mem": { 2024-04-25T11:47:06.231 INFO:teuthology.orchestra.run.smithi090.stdout: "ino": 3, 2024-04-25T11:47:06.231 INFO:teuthology.orchestra.run.smithi090.stdout: "ino+": 3, 2024-04-25T11:47:06.231 INFO:teuthology.orchestra.run.smithi090.stdout: "ino-": 0, 2024-04-25T11:47:06.231 INFO:teuthology.orchestra.run.smithi090.stdout: "dir": 1, 2024-04-25T11:47:06.231 INFO:teuthology.orchestra.run.smithi090.stdout: "dir+": 1, 2024-04-25T11:47:06.231 INFO:teuthology.orchestra.run.smithi090.stdout: "dir-": 0, 2024-04-25T11:47:06.231 INFO:teuthology.orchestra.run.smithi090.stdout: "dn": 0, 2024-04-25T11:47:06.232 INFO:teuthology.orchestra.run.smithi090.stdout: "dn+": 0, 2024-04-25T11:47:06.232 INFO:teuthology.orchestra.run.smithi090.stdout: "dn-": 0, 2024-04-25T11:47:06.232 INFO:teuthology.orchestra.run.smithi090.stdout: "cap": 0, 2024-04-25T11:47:06.232 INFO:teuthology.orchestra.run.smithi090.stdout: "cap+": 0, 2024-04-25T11:47:06.232 INFO:teuthology.orchestra.run.smithi090.stdout: "cap-": 0, 2024-04-25T11:47:06.232 INFO:teuthology.orchestra.run.smithi090.stdout: "rss": 119888, 2024-04-25T11:47:06.232 INFO:teuthology.orchestra.run.smithi090.stdout: "heap": 215440 2024-04-25T11:47:06.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.232 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_server": { 2024-04-25T11:47:06.232 INFO:teuthology.orchestra.run.smithi090.stdout: "dispatch_client_request": 0, 2024-04-25T11:47:06.232 INFO:teuthology.orchestra.run.smithi090.stdout: "dispatch_server_request": 0, 2024-04-25T11:47:06.232 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_client_request": 0, 2024-04-25T11:47:06.232 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_client_session": 0, 2024-04-25T11:47:06.232 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_peer_request": 0, 2024-04-25T11:47:06.233 INFO:teuthology.orchestra.run.smithi090.stdout: "req_create_latency": { 2024-04-25T11:47:06.233 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.233 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.233 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.233 INFO:teuthology.orchestra.run.smithi090.stdout: "req_getattr_latency": { 2024-04-25T11:47:06.233 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.233 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.233 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.233 INFO:teuthology.orchestra.run.smithi090.stdout: "req_getfilelock_latency": { 2024-04-25T11:47:06.233 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.233 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.233 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.234 INFO:teuthology.orchestra.run.smithi090.stdout: "req_link_latency": { 2024-04-25T11:47:06.234 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.234 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.234 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.234 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lookup_latency": { 2024-04-25T11:47:06.234 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.234 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.234 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.234 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lookuphash_latency": { 2024-04-25T11:47:06.234 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.234 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.235 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.235 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lookupino_latency": { 2024-04-25T11:47:06.235 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.235 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.235 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.235 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lookupname_latency": { 2024-04-25T11:47:06.235 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.235 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.235 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.235 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lookupparent_latency": { 2024-04-25T11:47:06.235 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.236 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.236 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.236 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lookupsnap_latency": { 2024-04-25T11:47:06.236 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.236 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.236 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.236 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lssnap_latency": { 2024-04-25T11:47:06.236 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.236 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.236 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.236 INFO:teuthology.orchestra.run.smithi090.stdout: "req_mkdir_latency": { 2024-04-25T11:47:06.236 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.237 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.237 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.237 INFO:teuthology.orchestra.run.smithi090.stdout: "req_mknod_latency": { 2024-04-25T11:47:06.237 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.237 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.238 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.238 INFO:teuthology.orchestra.run.smithi090.stdout: "req_mksnap_latency": { 2024-04-25T11:47:06.238 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.238 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.238 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.238 INFO:teuthology.orchestra.run.smithi090.stdout: "req_open_latency": { 2024-04-25T11:47:06.238 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.238 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.238 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.239 INFO:teuthology.orchestra.run.smithi090.stdout: "req_readdir_latency": { 2024-04-25T11:47:06.239 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.239 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.239 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.239 INFO:teuthology.orchestra.run.smithi090.stdout: "req_rename_latency": { 2024-04-25T11:47:06.239 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.239 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.239 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.239 INFO:teuthology.orchestra.run.smithi090.stdout: "req_renamesnap_latency": { 2024-04-25T11:47:06.239 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.239 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.239 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.240 INFO:teuthology.orchestra.run.smithi090.stdout: "req_snapdiff_latency": { 2024-04-25T11:47:06.240 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.240 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.240 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.240 INFO:teuthology.orchestra.run.smithi090.stdout: "req_rmdir_latency": { 2024-04-25T11:47:06.240 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.240 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.240 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.240 INFO:teuthology.orchestra.run.smithi090.stdout: "req_rmsnap_latency": { 2024-04-25T11:47:06.240 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.240 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.240 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.241 INFO:teuthology.orchestra.run.smithi090.stdout: "req_rmxattr_latency": { 2024-04-25T11:47:06.241 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.241 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.241 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.241 INFO:teuthology.orchestra.run.smithi090.stdout: "req_setattr_latency": { 2024-04-25T11:47:06.241 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.241 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.241 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.241 INFO:teuthology.orchestra.run.smithi090.stdout: "req_setdirlayout_latency": { 2024-04-25T11:47:06.241 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.241 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.242 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.242 INFO:teuthology.orchestra.run.smithi090.stdout: "req_setfilelock_latency": { 2024-04-25T11:47:06.242 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.242 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.242 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.242 INFO:teuthology.orchestra.run.smithi090.stdout: "req_setlayout_latency": { 2024-04-25T11:47:06.242 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.242 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.242 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.242 INFO:teuthology.orchestra.run.smithi090.stdout: "req_setxattr_latency": { 2024-04-25T11:47:06.243 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.243 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.243 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.243 INFO:teuthology.orchestra.run.smithi090.stdout: "req_symlink_latency": { 2024-04-25T11:47:06.243 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.243 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.243 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.243 INFO:teuthology.orchestra.run.smithi090.stdout: "req_unlink_latency": { 2024-04-25T11:47:06.243 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.243 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.243 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.244 INFO:teuthology.orchestra.run.smithi090.stdout: "cap_revoke_eviction": 0, 2024-04-25T11:47:06.244 INFO:teuthology.orchestra.run.smithi090.stdout: "cap_acquisition_throttle": 0, 2024-04-25T11:47:06.244 INFO:teuthology.orchestra.run.smithi090.stdout: "req_getvxattr_latency": { 2024-04-25T11:47:06.244 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.244 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.244 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.244 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:06.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.244 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_sessions": { 2024-04-25T11:47:06.244 INFO:teuthology.orchestra.run.smithi090.stdout: "session_count": 0, 2024-04-25T11:47:06.244 INFO:teuthology.orchestra.run.smithi090.stdout: "session_add": 2, 2024-04-25T11:47:06.244 INFO:teuthology.orchestra.run.smithi090.stdout: "session_remove": 2, 2024-04-25T11:47:06.244 INFO:teuthology.orchestra.run.smithi090.stdout: "sessions_open": 2, 2024-04-25T11:47:06.244 INFO:teuthology.orchestra.run.smithi090.stdout: "sessions_stale": 0, 2024-04-25T11:47:06.245 INFO:teuthology.orchestra.run.smithi090.stdout: "total_load": 0, 2024-04-25T11:47:06.245 INFO:teuthology.orchestra.run.smithi090.stdout: "average_load": 0, 2024-04-25T11:47:06.245 INFO:teuthology.orchestra.run.smithi090.stdout: "avg_session_uptime": 1248, 2024-04-25T11:47:06.245 INFO:teuthology.orchestra.run.smithi090.stdout: "mdthresh_evicted": 0 2024-04-25T11:47:06.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.245 INFO:teuthology.orchestra.run.smithi090.stdout: "mempool": { 2024-04-25T11:47:06.245 INFO:teuthology.orchestra.run.smithi090.stdout: "bloom_filter_bytes": 0, 2024-04-25T11:47:06.245 INFO:teuthology.orchestra.run.smithi090.stdout: "bloom_filter_items": 0, 2024-04-25T11:47:06.245 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T11:47:06.245 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_alloc_items": 0, 2024-04-25T11:47:06.245 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T11:47:06.245 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_data_items": 0, 2024-04-25T11:47:06.245 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T11:47:06.245 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T11:47:06.245 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T11:47:06.246 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T11:47:06.246 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T11:47:06.246 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_other_items": 0, 2024-04-25T11:47:06.246 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T11:47:06.246 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T11:47:06.246 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_bytes": 0, 2024-04-25T11:47:06.246 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_items": 0, 2024-04-25T11:47:06.246 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_blob_bytes": 0, 2024-04-25T11:47:06.246 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_blob_items": 0, 2024-04-25T11:47:06.246 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T11:47:06.246 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T11:47:06.246 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T11:47:06.247 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T11:47:06.247 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T11:47:06.247 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_items": 0, 2024-04-25T11:47:06.247 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_txc_bytes": 0, 2024-04-25T11:47:06.247 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_txc_items": 0, 2024-04-25T11:47:06.247 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T11:47:06.247 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T11:47:06.247 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_bytes": 0, 2024-04-25T11:47:06.247 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_items": 0, 2024-04-25T11:47:06.247 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_bytes": 0, 2024-04-25T11:47:06.247 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_items": 0, 2024-04-25T11:47:06.247 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T11:47:06.247 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_reader_items": 0, 2024-04-25T11:47:06.247 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T11:47:06.247 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_writer_items": 0, 2024-04-25T11:47:06.248 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_anon_bytes": 198257, 2024-04-25T11:47:06.248 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_anon_items": 57, 2024-04-25T11:47:06.248 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_meta_bytes": 0, 2024-04-25T11:47:06.248 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_meta_items": 0, 2024-04-25T11:47:06.248 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_bytes": 0, 2024-04-25T11:47:06.248 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_items": 0, 2024-04-25T11:47:06.248 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mapbl_bytes": 0, 2024-04-25T11:47:06.248 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mapbl_items": 0, 2024-04-25T11:47:06.248 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pglog_bytes": 0, 2024-04-25T11:47:06.248 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pglog_items": 0, 2024-04-25T11:47:06.248 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_bytes": 8512, 2024-04-25T11:47:06.248 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_items": 177, 2024-04-25T11:47:06.248 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T11:47:06.248 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_mapping_items": 0, 2024-04-25T11:47:06.249 INFO:teuthology.orchestra.run.smithi090.stdout: "pgmap_bytes": 0, 2024-04-25T11:47:06.249 INFO:teuthology.orchestra.run.smithi090.stdout: "pgmap_items": 0, 2024-04-25T11:47:06.249 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_co_bytes": 8512, 2024-04-25T11:47:06.249 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_co_items": 13, 2024-04-25T11:47:06.249 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_1_bytes": 0, 2024-04-25T11:47:06.249 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_1_items": 0, 2024-04-25T11:47:06.249 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_2_bytes": 0, 2024-04-25T11:47:06.249 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_2_items": 0 2024-04-25T11:47:06.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.249 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter": { 2024-04-25T11:47:06.249 INFO:teuthology.orchestra.run.smithi090.stdout: "op_active": 0, 2024-04-25T11:47:06.249 INFO:teuthology.orchestra.run.smithi090.stdout: "op_laggy": 0, 2024-04-25T11:47:06.249 INFO:teuthology.orchestra.run.smithi090.stdout: "op_send": 2350, 2024-04-25T11:47:06.249 INFO:teuthology.orchestra.run.smithi090.stdout: "op_send_bytes": 21, 2024-04-25T11:47:06.249 INFO:teuthology.orchestra.run.smithi090.stdout: "op_resend": 3, 2024-04-25T11:47:06.250 INFO:teuthology.orchestra.run.smithi090.stdout: "op_reply": 2350, 2024-04-25T11:47:06.250 INFO:teuthology.orchestra.run.smithi090.stdout: "op_latency": { 2024-04-25T11:47:06.250 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 2350, 2024-04-25T11:47:06.250 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 12.579812396, 2024-04-25T11:47:06.250 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.005353111 2024-04-25T11:47:06.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.250 INFO:teuthology.orchestra.run.smithi090.stdout: "op_inflight": 0, 2024-04-25T11:47:06.250 INFO:teuthology.orchestra.run.smithi090.stdout: "oplen_avg": { 2024-04-25T11:47:06.250 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 2350, 2024-04-25T11:47:06.250 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 2351 2024-04-25T11:47:06.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.250 INFO:teuthology.orchestra.run.smithi090.stdout: "op": 2350, 2024-04-25T11:47:06.250 INFO:teuthology.orchestra.run.smithi090.stdout: "op_r": 2350, 2024-04-25T11:47:06.251 INFO:teuthology.orchestra.run.smithi090.stdout: "op_w": 0, 2024-04-25T11:47:06.251 INFO:teuthology.orchestra.run.smithi090.stdout: "op_rmw": 0, 2024-04-25T11:47:06.251 INFO:teuthology.orchestra.run.smithi090.stdout: "op_pg": 0, 2024-04-25T11:47:06.251 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_stat": 1524, 2024-04-25T11:47:06.251 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_create": 0, 2024-04-25T11:47:06.251 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_read": 824, 2024-04-25T11:47:06.251 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_write": 0, 2024-04-25T11:47:06.251 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_writefull": 0, 2024-04-25T11:47:06.251 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_writesame": 0, 2024-04-25T11:47:06.251 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_append": 0, 2024-04-25T11:47:06.251 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_zero": 0, 2024-04-25T11:47:06.251 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_truncate": 0, 2024-04-25T11:47:06.251 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_delete": 0, 2024-04-25T11:47:06.251 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_mapext": 0, 2024-04-25T11:47:06.252 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_sparse_read": 0, 2024-04-25T11:47:06.252 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_clonerange": 0, 2024-04-25T11:47:06.252 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_getxattr": 1, 2024-04-25T11:47:06.252 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_setxattr": 0, 2024-04-25T11:47:06.252 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_cmpxattr": 0, 2024-04-25T11:47:06.252 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_rmxattr": 0, 2024-04-25T11:47:06.252 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_resetxattrs": 0, 2024-04-25T11:47:06.252 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_call": 0, 2024-04-25T11:47:06.252 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_watch": 0, 2024-04-25T11:47:06.252 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_notify": 0, 2024-04-25T11:47:06.252 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T11:47:06.252 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_pgls": 0, 2024-04-25T11:47:06.252 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_pgls_filter": 0, 2024-04-25T11:47:06.252 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_other": 0, 2024-04-25T11:47:06.253 INFO:teuthology.orchestra.run.smithi090.stdout: "linger_active": 0, 2024-04-25T11:47:06.253 INFO:teuthology.orchestra.run.smithi090.stdout: "linger_send": 0, 2024-04-25T11:47:06.253 INFO:teuthology.orchestra.run.smithi090.stdout: "linger_resend": 0, 2024-04-25T11:47:06.253 INFO:teuthology.orchestra.run.smithi090.stdout: "linger_ping": 0, 2024-04-25T11:47:06.253 INFO:teuthology.orchestra.run.smithi090.stdout: "poolop_active": 0, 2024-04-25T11:47:06.253 INFO:teuthology.orchestra.run.smithi090.stdout: "poolop_send": 0, 2024-04-25T11:47:06.253 INFO:teuthology.orchestra.run.smithi090.stdout: "poolop_resend": 0, 2024-04-25T11:47:06.253 INFO:teuthology.orchestra.run.smithi090.stdout: "poolstat_active": 0, 2024-04-25T11:47:06.253 INFO:teuthology.orchestra.run.smithi090.stdout: "poolstat_send": 0, 2024-04-25T11:47:06.253 INFO:teuthology.orchestra.run.smithi090.stdout: "poolstat_resend": 0, 2024-04-25T11:47:06.253 INFO:teuthology.orchestra.run.smithi090.stdout: "statfs_active": 0, 2024-04-25T11:47:06.253 INFO:teuthology.orchestra.run.smithi090.stdout: "statfs_send": 0, 2024-04-25T11:47:06.253 INFO:teuthology.orchestra.run.smithi090.stdout: "statfs_resend": 0, 2024-04-25T11:47:06.253 INFO:teuthology.orchestra.run.smithi090.stdout: "command_active": 0, 2024-04-25T11:47:06.254 INFO:teuthology.orchestra.run.smithi090.stdout: "command_send": 0, 2024-04-25T11:47:06.254 INFO:teuthology.orchestra.run.smithi090.stdout: "command_resend": 0, 2024-04-25T11:47:06.254 INFO:teuthology.orchestra.run.smithi090.stdout: "map_epoch": 92, 2024-04-25T11:47:06.254 INFO:teuthology.orchestra.run.smithi090.stdout: "map_full": 0, 2024-04-25T11:47:06.254 INFO:teuthology.orchestra.run.smithi090.stdout: "map_inc": 15, 2024-04-25T11:47:06.254 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_sessions": 6, 2024-04-25T11:47:06.254 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_session_open": 6, 2024-04-25T11:47:06.254 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_session_close": 0, 2024-04-25T11:47:06.254 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_laggy": 0, 2024-04-25T11:47:06.254 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_wr": 0, 2024-04-25T11:47:06.254 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_rd": 2, 2024-04-25T11:47:06.254 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_del": 0 2024-04-25T11:47:06.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.254 INFO:teuthology.orchestra.run.smithi090.stdout: "oft": { 2024-04-25T11:47:06.255 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_total_objs": 0, 2024-04-25T11:47:06.255 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_total_kv_pairs": 0, 2024-04-25T11:47:06.255 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_total_updates": 0, 2024-04-25T11:47:06.255 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_total_removes": 0 2024-04-25T11:47:06.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.255 INFO:teuthology.orchestra.run.smithi090.stdout: "osd-slow-ops": { 2024-04-25T11:47:06.255 INFO:teuthology.orchestra.run.smithi090.stdout: "slow_ops_count": 0 2024-04-25T11:47:06.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.255 INFO:teuthology.orchestra.run.smithi090.stdout: "purge_queue": { 2024-04-25T11:47:06.255 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_executing_ops": 0, 2024-04-25T11:47:06.255 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T11:47:06.255 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_executing": 0, 2024-04-25T11:47:06.255 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_executing_high_water": 0, 2024-04-25T11:47:06.255 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_executed_ops": 0, 2024-04-25T11:47:06.256 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_executed": 0, 2024-04-25T11:47:06.256 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_item_in_journal": 0 2024-04-25T11:47:06.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.256 INFO:teuthology.orchestra.run.smithi090.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T11:47:06.256 INFO:teuthology.orchestra.run.smithi090.stdout: "val": 0, 2024-04-25T11:47:06.256 INFO:teuthology.orchestra.run.smithi090.stdout: "max": 104857600, 2024-04-25T11:47:06.256 INFO:teuthology.orchestra.run.smithi090.stdout: "get_started": 0, 2024-04-25T11:47:06.256 INFO:teuthology.orchestra.run.smithi090.stdout: "get": 2613, 2024-04-25T11:47:06.256 INFO:teuthology.orchestra.run.smithi090.stdout: "get_sum": 2014933, 2024-04-25T11:47:06.256 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:06.256 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_success": 2613, 2024-04-25T11:47:06.256 INFO:teuthology.orchestra.run.smithi090.stdout: "take": 0, 2024-04-25T11:47:06.256 INFO:teuthology.orchestra.run.smithi090.stdout: "take_sum": 0, 2024-04-25T11:47:06.256 INFO:teuthology.orchestra.run.smithi090.stdout: "put": 2613, 2024-04-25T11:47:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: "put_sum": 2014933, 2024-04-25T11:47:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: "wait": { 2024-04-25T11:47:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: "throttle-objecter_bytes": { 2024-04-25T11:47:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: "val": 0, 2024-04-25T11:47:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: "max": 104857600, 2024-04-25T11:47:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: "get_started": 0, 2024-04-25T11:47:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: "get": 0, 2024-04-25T11:47:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: "get_sum": 0, 2024-04-25T11:47:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:06.258 INFO:teuthology.orchestra.run.smithi090.stdout: "take": 2350, 2024-04-25T11:47:06.258 INFO:teuthology.orchestra.run.smithi090.stdout: "take_sum": 1657, 2024-04-25T11:47:06.258 INFO:teuthology.orchestra.run.smithi090.stdout: "put": 2350, 2024-04-25T11:47:06.258 INFO:teuthology.orchestra.run.smithi090.stdout: "put_sum": 1657, 2024-04-25T11:47:06.258 INFO:teuthology.orchestra.run.smithi090.stdout: "wait": { 2024-04-25T11:47:06.258 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.258 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.258 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.258 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:06.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.258 INFO:teuthology.orchestra.run.smithi090.stdout: "throttle-objecter_ops": { 2024-04-25T11:47:06.258 INFO:teuthology.orchestra.run.smithi090.stdout: "val": 0, 2024-04-25T11:47:06.258 INFO:teuthology.orchestra.run.smithi090.stdout: "max": 1024, 2024-04-25T11:47:06.259 INFO:teuthology.orchestra.run.smithi090.stdout: "get_started": 0, 2024-04-25T11:47:06.259 INFO:teuthology.orchestra.run.smithi090.stdout: "get": 0, 2024-04-25T11:47:06.259 INFO:teuthology.orchestra.run.smithi090.stdout: "get_sum": 0, 2024-04-25T11:47:06.259 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:06.259 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:06.259 INFO:teuthology.orchestra.run.smithi090.stdout: "take": 2350, 2024-04-25T11:47:06.259 INFO:teuthology.orchestra.run.smithi090.stdout: "take_sum": 2350, 2024-04-25T11:47:06.259 INFO:teuthology.orchestra.run.smithi090.stdout: "put": 2350, 2024-04-25T11:47:06.259 INFO:teuthology.orchestra.run.smithi090.stdout: "put_sum": 2350, 2024-04-25T11:47:06.259 INFO:teuthology.orchestra.run.smithi090.stdout: "wait": { 2024-04-25T11:47:06.259 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.259 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.259 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.259 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:06.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.260 INFO:teuthology.orchestra.run.smithi090.stdout: "throttle-write_buf_throttle": { 2024-04-25T11:47:06.260 INFO:teuthology.orchestra.run.smithi090.stdout: "val": 0, 2024-04-25T11:47:06.260 INFO:teuthology.orchestra.run.smithi090.stdout: "max": 3758096384, 2024-04-25T11:47:06.260 INFO:teuthology.orchestra.run.smithi090.stdout: "get_started": 0, 2024-04-25T11:47:06.260 INFO:teuthology.orchestra.run.smithi090.stdout: "get": 0, 2024-04-25T11:47:06.260 INFO:teuthology.orchestra.run.smithi090.stdout: "get_sum": 0, 2024-04-25T11:47:06.260 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:06.260 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:06.260 INFO:teuthology.orchestra.run.smithi090.stdout: "take": 0, 2024-04-25T11:47:06.260 INFO:teuthology.orchestra.run.smithi090.stdout: "take_sum": 0, 2024-04-25T11:47:06.260 INFO:teuthology.orchestra.run.smithi090.stdout: "put": 0, 2024-04-25T11:47:06.260 INFO:teuthology.orchestra.run.smithi090.stdout: "put_sum": 0, 2024-04-25T11:47:06.260 INFO:teuthology.orchestra.run.smithi090.stdout: "wait": { 2024-04-25T11:47:06.261 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.261 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.261 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.261 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:06.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:06.261 INFO:teuthology.orchestra.run.smithi090.stdout: "throttle-write_buf_throttle-0x55b9ce8ecc30": { 2024-04-25T11:47:06.261 INFO:teuthology.orchestra.run.smithi090.stdout: "val": 0, 2024-04-25T11:47:06.261 INFO:teuthology.orchestra.run.smithi090.stdout: "max": 3758096384, 2024-04-25T11:47:06.261 INFO:teuthology.orchestra.run.smithi090.stdout: "get_started": 0, 2024-04-25T11:47:06.261 INFO:teuthology.orchestra.run.smithi090.stdout: "get": 0, 2024-04-25T11:47:06.261 INFO:teuthology.orchestra.run.smithi090.stdout: "get_sum": 0, 2024-04-25T11:47:06.261 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:06.261 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:06.262 INFO:teuthology.orchestra.run.smithi090.stdout: "take": 0, 2024-04-25T11:47:06.262 INFO:teuthology.orchestra.run.smithi090.stdout: "take_sum": 0, 2024-04-25T11:47:06.262 INFO:teuthology.orchestra.run.smithi090.stdout: "put": 0, 2024-04-25T11:47:06.262 INFO:teuthology.orchestra.run.smithi090.stdout: "put_sum": 0, 2024-04-25T11:47:06.262 INFO:teuthology.orchestra.run.smithi090.stdout: "wait": { 2024-04-25T11:47:06.262 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:06.262 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:06.262 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:06.262 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:06.262 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:06.262 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:47:06.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:06 smithi090 ceph-mon[29575]: pgmap v373: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 15 KiB/s rd, 31 KiB/s wr, 227 op/s 2024-04-25T11:47:06.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:06 smithi138 ceph-mon[32328]: pgmap v373: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 15 KiB/s rd, 31 KiB/s wr, 227 op/s 2024-04-25T11:47:06.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:06 smithi156 ceph-mon[33045]: pgmap v373: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 15 KiB/s rd, 31 KiB/s wr, 227 op/s 2024-04-25T11:47:07.203 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-04-25T11:47:07.204 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-04-25T11:47:07.204 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update=0 2024-04-25T11:47:07.204 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update_receipt=0 2024-04-25T11:47:07.204 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-04-25T11:47:07.204 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-04-25T11:47:07.204 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T11:47:07.204 DEBUG:tasks.check_counter:Getting stats from d 2024-04-25T11:47:07.204 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-25T11:47:07.450 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:47:08.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:08 smithi090 ceph-mon[29575]: pgmap v374: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 19 KiB/s rd, 53 KiB/s wr, 147 op/s 2024-04-25T11:47:08.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:08 smithi138 ceph-mon[32328]: pgmap v374: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 19 KiB/s rd, 53 KiB/s wr, 147 op/s 2024-04-25T11:47:08.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:08 smithi156 ceph-mon[33045]: pgmap v374: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 19 KiB/s rd, 53 KiB/s wr, 147 op/s 2024-04-25T11:47:09.111 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:47:09.111 INFO:teuthology.orchestra.run.smithi090.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T11:47:09.111 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_messages": 0, 2024-04-25T11:47:09.111 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages": 2, 2024-04-25T11:47:09.111 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_bytes": 0, 2024-04-25T11:47:09.112 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_bytes": 0, 2024-04-25T11:47:09.112 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_created_connections": 4, 2024-04-25T11:47:09.112 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_active_connections": 0, 2024-04-25T11:47:09.112 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_total_time": 0.001122034, 2024-04-25T11:47:09.112 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T11:47:09.112 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_recv_time": 0.000242989, 2024-04-25T11:47:09.112 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T11:47:09.112 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:09.112 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:09.112 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:09.112 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:09.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:09.112 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:09.113 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:09.113 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:09.113 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:09.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:09.113 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T11:47:09.113 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T11:47:09.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:09.113 INFO:teuthology.orchestra.run.smithi090.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T11:47:09.113 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_messages": 242, 2024-04-25T11:47:09.113 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages": 232, 2024-04-25T11:47:09.113 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_bytes": 1545424, 2024-04-25T11:47:09.113 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_bytes": 142784, 2024-04-25T11:47:09.113 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_created_connections": 2, 2024-04-25T11:47:09.113 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_active_connections": 1, 2024-04-25T11:47:09.113 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_total_time": 0.052675892, 2024-04-25T11:47:09.114 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_send_time": 0.016052998, 2024-04-25T11:47:09.114 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_recv_time": 0.041743502, 2024-04-25T11:47:09.114 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_fast_dispatch_time": 0.005457151, 2024-04-25T11:47:09.114 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:09.114 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 231, 2024-04-25T11:47:09.114 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.017242856, 2024-04-25T11:47:09.114 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000074644 2024-04-25T11:47:09.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:09.114 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:09.114 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:09.114 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:09.114 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:09.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:09.115 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_encrypted_bytes": 1545424, 2024-04-25T11:47:09.115 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_encrypted_bytes": 142784 2024-04-25T11:47:09.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:09.115 INFO:teuthology.orchestra.run.smithi090.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T11:47:09.115 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_messages": 6, 2024-04-25T11:47:09.115 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages": 179, 2024-04-25T11:47:09.115 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_bytes": 3312, 2024-04-25T11:47:09.115 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_bytes": 188704, 2024-04-25T11:47:09.115 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_created_connections": 5, 2024-04-25T11:47:09.115 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_active_connections": 3, 2024-04-25T11:47:09.115 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_total_time": 0.016197688, 2024-04-25T11:47:09.115 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_send_time": 0.011142005, 2024-04-25T11:47:09.115 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_recv_time": 0.002353877, 2024-04-25T11:47:09.115 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T11:47:09.115 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 177, 2024-04-25T11:47:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.028497465, 2024-04-25T11:47:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000161002 2024-04-25T11:47:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-25T11:47:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_encrypted_bytes": 188704 2024-04-25T11:47:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: "cct": { 2024-04-25T11:47:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: "total_workers": 0, 2024-04-25T11:47:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: "unhealthy_workers": 0 2024-04-25T11:47:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:09.117 INFO:teuthology.orchestra.run.smithi090.stdout: "mempool": { 2024-04-25T11:47:09.117 INFO:teuthology.orchestra.run.smithi090.stdout: "bloom_filter_bytes": 0, 2024-04-25T11:47:09.117 INFO:teuthology.orchestra.run.smithi090.stdout: "bloom_filter_items": 0, 2024-04-25T11:47:09.117 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T11:47:09.117 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_alloc_items": 0, 2024-04-25T11:47:09.117 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T11:47:09.117 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_data_items": 0, 2024-04-25T11:47:09.117 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T11:47:09.117 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T11:47:09.117 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T11:47:09.117 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T11:47:09.117 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T11:47:09.117 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_other_items": 0, 2024-04-25T11:47:09.117 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T11:47:09.117 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T11:47:09.118 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_bytes": 0, 2024-04-25T11:47:09.118 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_items": 0, 2024-04-25T11:47:09.118 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_blob_bytes": 0, 2024-04-25T11:47:09.118 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_blob_items": 0, 2024-04-25T11:47:09.118 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T11:47:09.118 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T11:47:09.118 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T11:47:09.118 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T11:47:09.118 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T11:47:09.118 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_items": 0, 2024-04-25T11:47:09.118 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_txc_bytes": 0, 2024-04-25T11:47:09.118 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_txc_items": 0, 2024-04-25T11:47:09.118 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T11:47:09.118 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T11:47:09.119 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_bytes": 0, 2024-04-25T11:47:09.119 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_items": 0, 2024-04-25T11:47:09.119 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_bytes": 0, 2024-04-25T11:47:09.119 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_items": 0, 2024-04-25T11:47:09.119 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T11:47:09.119 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_reader_items": 0, 2024-04-25T11:47:09.119 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T11:47:09.119 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_writer_items": 0, 2024-04-25T11:47:09.119 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_anon_bytes": 148977, 2024-04-25T11:47:09.119 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_anon_items": 30, 2024-04-25T11:47:09.119 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_meta_bytes": 0, 2024-04-25T11:47:09.119 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_meta_items": 0, 2024-04-25T11:47:09.119 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_bytes": 0, 2024-04-25T11:47:09.119 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_items": 0, 2024-04-25T11:47:09.119 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mapbl_bytes": 0, 2024-04-25T11:47:09.120 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mapbl_items": 0, 2024-04-25T11:47:09.120 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pglog_bytes": 0, 2024-04-25T11:47:09.120 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pglog_items": 0, 2024-04-25T11:47:09.120 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_bytes": 0, 2024-04-25T11:47:09.120 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_items": 0, 2024-04-25T11:47:09.120 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T11:47:09.120 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_mapping_items": 0, 2024-04-25T11:47:09.120 INFO:teuthology.orchestra.run.smithi090.stdout: "pgmap_bytes": 0, 2024-04-25T11:47:09.120 INFO:teuthology.orchestra.run.smithi090.stdout: "pgmap_items": 0, 2024-04-25T11:47:09.120 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_co_bytes": 928, 2024-04-25T11:47:09.120 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_co_items": 2, 2024-04-25T11:47:09.120 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_1_bytes": 0, 2024-04-25T11:47:09.120 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_1_items": 0, 2024-04-25T11:47:09.120 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_2_bytes": 0, 2024-04-25T11:47:09.120 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_2_items": 0 2024-04-25T11:47:09.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:09.121 INFO:teuthology.orchestra.run.smithi090.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T11:47:09.121 INFO:teuthology.orchestra.run.smithi090.stdout: "val": 0, 2024-04-25T11:47:09.121 INFO:teuthology.orchestra.run.smithi090.stdout: "max": 104857600, 2024-04-25T11:47:09.121 INFO:teuthology.orchestra.run.smithi090.stdout: "get_started": 0, 2024-04-25T11:47:09.121 INFO:teuthology.orchestra.run.smithi090.stdout: "get": 244, 2024-04-25T11:47:09.121 INFO:teuthology.orchestra.run.smithi090.stdout: "get_sum": 1511274, 2024-04-25T11:47:09.121 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:09.121 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_success": 244, 2024-04-25T11:47:09.121 INFO:teuthology.orchestra.run.smithi090.stdout: "take": 0, 2024-04-25T11:47:09.121 INFO:teuthology.orchestra.run.smithi090.stdout: "take_sum": 0, 2024-04-25T11:47:09.121 INFO:teuthology.orchestra.run.smithi090.stdout: "put": 244, 2024-04-25T11:47:09.121 INFO:teuthology.orchestra.run.smithi090.stdout: "put_sum": 1511274, 2024-04-25T11:47:09.121 INFO:teuthology.orchestra.run.smithi090.stdout: "wait": { 2024-04-25T11:47:09.121 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:09.121 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:09.121 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:09.121 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:09.122 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:09.122 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:47:09.731 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-25T11:47:09.731 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-25T11:47:09.731 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-04-25T11:47:09.731 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-04-25T11:47:09.731 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-25T11:47:09.731 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-25T11:47:09.731 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-25T11:47:09.731 DEBUG:tasks.check_counter:Getting stats from g 2024-04-25T11:47:09.732 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-25T11:47:09.970 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:47:10.417 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:10 smithi090 ceph-mon[29575]: pgmap v375: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 13 KiB/s rd, 50 KiB/s wr, 59 op/s 2024-04-25T11:47:10.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:10 smithi138 ceph-mon[32328]: pgmap v375: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 13 KiB/s rd, 50 KiB/s wr, 59 op/s 2024-04-25T11:47:10.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:10 smithi156 ceph-mon[33045]: pgmap v375: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 13 KiB/s rd, 50 KiB/s wr, 59 op/s 2024-04-25T11:47:11.755 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:47:11.755 INFO:teuthology.orchestra.run.smithi090.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T11:47:11.755 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_messages": 828, 2024-04-25T11:47:11.755 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages": 830, 2024-04-25T11:47:11.755 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_bytes": 278153, 2024-04-25T11:47:11.755 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_bytes": 263305, 2024-04-25T11:47:11.755 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_created_connections": 6, 2024-04-25T11:47:11.755 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_active_connections": 2, 2024-04-25T11:47:11.755 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_total_time": 0.106442488, 2024-04-25T11:47:11.755 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_send_time": 0.040868552, 2024-04-25T11:47:11.755 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_recv_time": 0.025413170, 2024-04-25T11:47:11.756 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_fast_dispatch_time": 0.030734541, 2024-04-25T11:47:11.756 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:11.756 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 828, 2024-04-25T11:47:11.756 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.051912010, 2024-04-25T11:47:11.756 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000062695 2024-04-25T11:47:11.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.756 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:11.756 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.756 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.756 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.756 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T11:47:11.756 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T11:47:11.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.757 INFO:teuthology.orchestra.run.smithi090.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T11:47:11.757 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_messages": 1499, 2024-04-25T11:47:11.757 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages": 1670, 2024-04-25T11:47:11.757 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_bytes": 384865, 2024-04-25T11:47:11.757 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_bytes": 949999, 2024-04-25T11:47:11.757 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_created_connections": 7, 2024-04-25T11:47:11.757 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_active_connections": 5, 2024-04-25T11:47:11.757 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_total_time": 0.132349341, 2024-04-25T11:47:11.757 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_send_time": 0.040343052, 2024-04-25T11:47:11.757 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_recv_time": 0.033087054, 2024-04-25T11:47:11.757 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_fast_dispatch_time": 0.048836140, 2024-04-25T11:47:11.757 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:11.757 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 1668, 2024-04-25T11:47:11.757 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.078839913, 2024-04-25T11:47:11.757 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000047266 2024-04-25T11:47:11.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.758 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:11.758 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.758 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.758 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.758 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-25T11:47:11.758 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_encrypted_bytes": 475168 2024-04-25T11:47:11.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.758 INFO:teuthology.orchestra.run.smithi090.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T11:47:11.758 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_messages": 266, 2024-04-25T11:47:11.758 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages": 252, 2024-04-25T11:47:11.758 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_bytes": 1598292, 2024-04-25T11:47:11.759 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_bytes": 144837, 2024-04-25T11:47:11.759 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_created_connections": 4, 2024-04-25T11:47:11.759 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_active_connections": 3, 2024-04-25T11:47:11.759 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_total_time": 0.051454482, 2024-04-25T11:47:11.759 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_send_time": 0.015985973, 2024-04-25T11:47:11.759 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_recv_time": 0.041419828, 2024-04-25T11:47:11.759 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_fast_dispatch_time": 0.005488657, 2024-04-25T11:47:11.759 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:11.759 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 251, 2024-04-25T11:47:11.759 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.022736497, 2024-04-25T11:47:11.759 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000090583 2024-04-25T11:47:11.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.759 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:11.759 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.760 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.760 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.760 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_encrypted_bytes": 1596944, 2024-04-25T11:47:11.760 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_encrypted_bytes": 143872 2024-04-25T11:47:11.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.760 INFO:teuthology.orchestra.run.smithi090.stdout: "cct": { 2024-04-25T11:47:11.760 INFO:teuthology.orchestra.run.smithi090.stdout: "total_workers": 1, 2024-04-25T11:47:11.760 INFO:teuthology.orchestra.run.smithi090.stdout: "unhealthy_workers": 0 2024-04-25T11:47:11.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.760 INFO:teuthology.orchestra.run.smithi090.stdout: "finisher-MDSRank": { 2024-04-25T11:47:11.760 INFO:teuthology.orchestra.run.smithi090.stdout: "queue_len": 0, 2024-04-25T11:47:11.760 INFO:teuthology.orchestra.run.smithi090.stdout: "complete_latency": { 2024-04-25T11:47:11.760 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 3929, 2024-04-25T11:47:11.760 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.119781508, 2024-04-25T11:47:11.761 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000030486 2024-04-25T11:47:11.761 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:11.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.761 INFO:teuthology.orchestra.run.smithi090.stdout: "finisher-PurgeQueue": { 2024-04-25T11:47:11.761 INFO:teuthology.orchestra.run.smithi090.stdout: "queue_len": 0, 2024-04-25T11:47:11.761 INFO:teuthology.orchestra.run.smithi090.stdout: "complete_latency": { 2024-04-25T11:47:11.761 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.761 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.761 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.761 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:11.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.761 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": { 2024-04-25T11:47:11.761 INFO:teuthology.orchestra.run.smithi090.stdout: "request": 0, 2024-04-25T11:47:11.761 INFO:teuthology.orchestra.run.smithi090.stdout: "reply": 0, 2024-04-25T11:47:11.761 INFO:teuthology.orchestra.run.smithi090.stdout: "reply_latency": { 2024-04-25T11:47:11.762 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.762 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.762 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.762 INFO:teuthology.orchestra.run.smithi090.stdout: "slow_reply": 0, 2024-04-25T11:47:11.762 INFO:teuthology.orchestra.run.smithi090.stdout: "forward": 0, 2024-04-25T11:47:11.762 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_fetch_complete": 0, 2024-04-25T11:47:11.762 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_fetch_keys": 0, 2024-04-25T11:47:11.762 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_commit": 0, 2024-04-25T11:47:11.762 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_split": 0, 2024-04-25T11:47:11.762 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_merge": 0, 2024-04-25T11:47:11.762 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes": 0, 2024-04-25T11:47:11.762 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes_top": 0, 2024-04-25T11:47:11.762 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes_bottom": 0, 2024-04-25T11:47:11.763 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes_pin_tail": 0, 2024-04-25T11:47:11.763 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes_pinned": 0, 2024-04-25T11:47:11.763 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes_expired": 0, 2024-04-25T11:47:11.763 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes_with_caps": 0, 2024-04-25T11:47:11.763 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": 0, 2024-04-25T11:47:11.763 INFO:teuthology.orchestra.run.smithi090.stdout: "subtrees": 1, 2024-04-25T11:47:11.763 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse": 0, 2024-04-25T11:47:11.763 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse_hit": 0, 2024-04-25T11:47:11.763 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse_forward": 0, 2024-04-25T11:47:11.763 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse_discover": 0, 2024-04-25T11:47:11.763 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse_dir_fetch": 0, 2024-04-25T11:47:11.763 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse_remote_ino": 0, 2024-04-25T11:47:11.763 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse_lock": 0, 2024-04-25T11:47:11.763 INFO:teuthology.orchestra.run.smithi090.stdout: "load_cent": 0, 2024-04-25T11:47:11.764 INFO:teuthology.orchestra.run.smithi090.stdout: "q": 0, 2024-04-25T11:47:11.764 INFO:teuthology.orchestra.run.smithi090.stdout: "exported": 0, 2024-04-25T11:47:11.764 INFO:teuthology.orchestra.run.smithi090.stdout: "exported_inodes": 0, 2024-04-25T11:47:11.764 INFO:teuthology.orchestra.run.smithi090.stdout: "imported": 0, 2024-04-25T11:47:11.764 INFO:teuthology.orchestra.run.smithi090.stdout: "imported_inodes": 0, 2024-04-25T11:47:11.764 INFO:teuthology.orchestra.run.smithi090.stdout: "openino_dir_fetch": 0, 2024-04-25T11:47:11.764 INFO:teuthology.orchestra.run.smithi090.stdout: "openino_backtrace_fetch": 0, 2024-04-25T11:47:11.764 INFO:teuthology.orchestra.run.smithi090.stdout: "openino_peer_discover": 0, 2024-04-25T11:47:11.764 INFO:teuthology.orchestra.run.smithi090.stdout: "root_rfiles": 0, 2024-04-25T11:47:11.764 INFO:teuthology.orchestra.run.smithi090.stdout: "root_rbytes": 0, 2024-04-25T11:47:11.764 INFO:teuthology.orchestra.run.smithi090.stdout: "root_rsnaps": 0, 2024-04-25T11:47:11.764 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T11:47:11.764 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_set_tag": 0, 2024-04-25T11:47:11.764 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T11:47:11.765 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_inotable_repaired": 0, 2024-04-25T11:47:11.765 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_dir_inodes": 0, 2024-04-25T11:47:11.765 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T11:47:11.765 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T11:47:11.765 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_file_inodes": 0, 2024-04-25T11:47:11.765 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_inode_file_caps": 0, 2024-04-25T11:47:11.765 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T11:47:11.765 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_cap_op_grant": 0, 2024-04-25T11:47:11.765 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T11:47:11.765 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T11:47:11.765 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T11:47:11.765 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_client_caps": 0, 2024-04-25T11:47:11.765 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_client_caps_dirty": 0, 2024-04-25T11:47:11.765 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_client_cap_release": 0, 2024-04-25T11:47:11.765 INFO:teuthology.orchestra.run.smithi090.stdout: "process_request_cap_release": 0 2024-04-25T11:47:11.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.766 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache": { 2024-04-25T11:47:11.766 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_update": 0, 2024-04-25T11:47:11.766 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_update_receipt": 0, 2024-04-25T11:47:11.766 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_try_discover": 0, 2024-04-25T11:47:11.766 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_send_discover": 0, 2024-04-25T11:47:11.766 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_handle_discover": 0, 2024-04-25T11:47:11.767 INFO:teuthology.orchestra.run.smithi090.stdout: "num_strays": 0, 2024-04-25T11:47:11.767 INFO:teuthology.orchestra.run.smithi090.stdout: "num_strays_delayed": 0, 2024-04-25T11:47:11.767 INFO:teuthology.orchestra.run.smithi090.stdout: "num_strays_enqueuing": 0, 2024-04-25T11:47:11.767 INFO:teuthology.orchestra.run.smithi090.stdout: "strays_created": 0, 2024-04-25T11:47:11.767 INFO:teuthology.orchestra.run.smithi090.stdout: "strays_enqueued": 0, 2024-04-25T11:47:11.767 INFO:teuthology.orchestra.run.smithi090.stdout: "strays_reintegrated": 0, 2024-04-25T11:47:11.767 INFO:teuthology.orchestra.run.smithi090.stdout: "strays_migrated": 0, 2024-04-25T11:47:11.767 INFO:teuthology.orchestra.run.smithi090.stdout: "num_recovering_processing": 0, 2024-04-25T11:47:11.767 INFO:teuthology.orchestra.run.smithi090.stdout: "num_recovering_enqueued": 0, 2024-04-25T11:47:11.768 INFO:teuthology.orchestra.run.smithi090.stdout: "num_recovering_prioritized": 0, 2024-04-25T11:47:11.768 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_started": 0, 2024-04-25T11:47:11.768 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_completed": 0, 2024-04-25T11:47:11.768 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_quiesce_path": 0, 2024-04-25T11:47:11.768 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_quiesce_inode": 0, 2024-04-25T11:47:11.768 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T11:47:11.768 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_exportdir": 0, 2024-04-25T11:47:11.768 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_flush": 0, 2024-04-25T11:47:11.768 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_fragmentdir": 0, 2024-04-25T11:47:11.768 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_fragstats": 0, 2024-04-25T11:47:11.768 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_inodestats": 0 2024-04-25T11:47:11.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.768 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log": { 2024-04-25T11:47:11.769 INFO:teuthology.orchestra.run.smithi090.stdout: "evlrg": 0, 2024-04-25T11:47:11.769 INFO:teuthology.orchestra.run.smithi090.stdout: "evadd": 0, 2024-04-25T11:47:11.769 INFO:teuthology.orchestra.run.smithi090.stdout: "evex": 0, 2024-04-25T11:47:11.769 INFO:teuthology.orchestra.run.smithi090.stdout: "evtrm": 0, 2024-04-25T11:47:11.769 INFO:teuthology.orchestra.run.smithi090.stdout: "ev": 3, 2024-04-25T11:47:11.769 INFO:teuthology.orchestra.run.smithi090.stdout: "evexg": 0, 2024-04-25T11:47:11.769 INFO:teuthology.orchestra.run.smithi090.stdout: "evexd": 0, 2024-04-25T11:47:11.769 INFO:teuthology.orchestra.run.smithi090.stdout: "segadd": 0, 2024-04-25T11:47:11.769 INFO:teuthology.orchestra.run.smithi090.stdout: "segex": 0, 2024-04-25T11:47:11.769 INFO:teuthology.orchestra.run.smithi090.stdout: "segtrm": 0, 2024-04-25T11:47:11.769 INFO:teuthology.orchestra.run.smithi090.stdout: "seg": 2, 2024-04-25T11:47:11.769 INFO:teuthology.orchestra.run.smithi090.stdout: "segmjr": 2, 2024-04-25T11:47:11.769 INFO:teuthology.orchestra.run.smithi090.stdout: "segexg": 0, 2024-04-25T11:47:11.770 INFO:teuthology.orchestra.run.smithi090.stdout: "segexd": 0, 2024-04-25T11:47:11.770 INFO:teuthology.orchestra.run.smithi090.stdout: "expos": 4194304, 2024-04-25T11:47:11.770 INFO:teuthology.orchestra.run.smithi090.stdout: "wrpos": 4195283, 2024-04-25T11:47:11.770 INFO:teuthology.orchestra.run.smithi090.stdout: "rdpos": 4194892, 2024-04-25T11:47:11.770 INFO:teuthology.orchestra.run.smithi090.stdout: "jlat": { 2024-04-25T11:47:11.770 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.770 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.770 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.770 INFO:teuthology.orchestra.run.smithi090.stdout: "replayed": 3 2024-04-25T11:47:11.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.770 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_mem": { 2024-04-25T11:47:11.771 INFO:teuthology.orchestra.run.smithi090.stdout: "ino": 3, 2024-04-25T11:47:11.771 INFO:teuthology.orchestra.run.smithi090.stdout: "ino+": 3, 2024-04-25T11:47:11.771 INFO:teuthology.orchestra.run.smithi090.stdout: "ino-": 0, 2024-04-25T11:47:11.771 INFO:teuthology.orchestra.run.smithi090.stdout: "dir": 1, 2024-04-25T11:47:11.771 INFO:teuthology.orchestra.run.smithi090.stdout: "dir+": 1, 2024-04-25T11:47:11.771 INFO:teuthology.orchestra.run.smithi090.stdout: "dir-": 0, 2024-04-25T11:47:11.771 INFO:teuthology.orchestra.run.smithi090.stdout: "dn": 0, 2024-04-25T11:47:11.771 INFO:teuthology.orchestra.run.smithi090.stdout: "dn+": 0, 2024-04-25T11:47:11.771 INFO:teuthology.orchestra.run.smithi090.stdout: "dn-": 0, 2024-04-25T11:47:11.771 INFO:teuthology.orchestra.run.smithi090.stdout: "cap": 0, 2024-04-25T11:47:11.771 INFO:teuthology.orchestra.run.smithi090.stdout: "cap+": 0, 2024-04-25T11:47:11.771 INFO:teuthology.orchestra.run.smithi090.stdout: "cap-": 0, 2024-04-25T11:47:11.771 INFO:teuthology.orchestra.run.smithi090.stdout: "rss": 118536, 2024-04-25T11:47:11.772 INFO:teuthology.orchestra.run.smithi090.stdout: "heap": 215440 2024-04-25T11:47:11.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.772 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_server": { 2024-04-25T11:47:11.772 INFO:teuthology.orchestra.run.smithi090.stdout: "dispatch_client_request": 0, 2024-04-25T11:47:11.772 INFO:teuthology.orchestra.run.smithi090.stdout: "dispatch_server_request": 0, 2024-04-25T11:47:11.772 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_client_request": 0, 2024-04-25T11:47:11.772 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_client_session": 0, 2024-04-25T11:47:11.772 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_peer_request": 0, 2024-04-25T11:47:11.772 INFO:teuthology.orchestra.run.smithi090.stdout: "req_create_latency": { 2024-04-25T11:47:11.772 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.772 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.773 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.773 INFO:teuthology.orchestra.run.smithi090.stdout: "req_getattr_latency": { 2024-04-25T11:47:11.773 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.773 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.773 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.773 INFO:teuthology.orchestra.run.smithi090.stdout: "req_getfilelock_latency": { 2024-04-25T11:47:11.773 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.773 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.773 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.774 INFO:teuthology.orchestra.run.smithi090.stdout: "req_link_latency": { 2024-04-25T11:47:11.774 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.774 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.774 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.774 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lookup_latency": { 2024-04-25T11:47:11.774 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.774 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.774 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.774 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lookuphash_latency": { 2024-04-25T11:47:11.774 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.775 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.775 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.775 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lookupino_latency": { 2024-04-25T11:47:11.775 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.775 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.775 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.775 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lookupname_latency": { 2024-04-25T11:47:11.775 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.775 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.775 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.776 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lookupparent_latency": { 2024-04-25T11:47:11.776 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.776 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.776 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.776 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lookupsnap_latency": { 2024-04-25T11:47:11.776 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.776 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.776 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.776 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lssnap_latency": { 2024-04-25T11:47:11.776 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.776 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.777 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.777 INFO:teuthology.orchestra.run.smithi090.stdout: "req_mkdir_latency": { 2024-04-25T11:47:11.777 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.778 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.778 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.778 INFO:teuthology.orchestra.run.smithi090.stdout: "req_mknod_latency": { 2024-04-25T11:47:11.778 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.778 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.778 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.778 INFO:teuthology.orchestra.run.smithi090.stdout: "req_mksnap_latency": { 2024-04-25T11:47:11.778 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.779 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.779 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.779 INFO:teuthology.orchestra.run.smithi090.stdout: "req_open_latency": { 2024-04-25T11:47:11.779 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.779 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.779 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.779 INFO:teuthology.orchestra.run.smithi090.stdout: "req_readdir_latency": { 2024-04-25T11:47:11.779 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.779 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.779 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.780 INFO:teuthology.orchestra.run.smithi090.stdout: "req_rename_latency": { 2024-04-25T11:47:11.780 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.780 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.780 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.780 INFO:teuthology.orchestra.run.smithi090.stdout: "req_renamesnap_latency": { 2024-04-25T11:47:11.780 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.780 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.780 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.780 INFO:teuthology.orchestra.run.smithi090.stdout: "req_snapdiff_latency": { 2024-04-25T11:47:11.780 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.780 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.781 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.781 INFO:teuthology.orchestra.run.smithi090.stdout: "req_rmdir_latency": { 2024-04-25T11:47:11.781 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.781 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.781 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.781 INFO:teuthology.orchestra.run.smithi090.stdout: "req_rmsnap_latency": { 2024-04-25T11:47:11.781 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.781 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.781 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.782 INFO:teuthology.orchestra.run.smithi090.stdout: "req_rmxattr_latency": { 2024-04-25T11:47:11.782 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.782 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.782 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.782 INFO:teuthology.orchestra.run.smithi090.stdout: "req_setattr_latency": { 2024-04-25T11:47:11.782 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.782 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.782 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.783 INFO:teuthology.orchestra.run.smithi090.stdout: "req_setdirlayout_latency": { 2024-04-25T11:47:11.783 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.783 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.783 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.783 INFO:teuthology.orchestra.run.smithi090.stdout: "req_setfilelock_latency": { 2024-04-25T11:47:11.783 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.783 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.783 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.783 INFO:teuthology.orchestra.run.smithi090.stdout: "req_setlayout_latency": { 2024-04-25T11:47:11.783 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.784 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.784 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.784 INFO:teuthology.orchestra.run.smithi090.stdout: "req_setxattr_latency": { 2024-04-25T11:47:11.784 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.784 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.784 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.784 INFO:teuthology.orchestra.run.smithi090.stdout: "req_symlink_latency": { 2024-04-25T11:47:11.784 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.784 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.784 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.785 INFO:teuthology.orchestra.run.smithi090.stdout: "req_unlink_latency": { 2024-04-25T11:47:11.785 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.785 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.785 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.785 INFO:teuthology.orchestra.run.smithi090.stdout: "cap_revoke_eviction": 0, 2024-04-25T11:47:11.785 INFO:teuthology.orchestra.run.smithi090.stdout: "cap_acquisition_throttle": 0, 2024-04-25T11:47:11.785 INFO:teuthology.orchestra.run.smithi090.stdout: "req_getvxattr_latency": { 2024-04-25T11:47:11.785 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.785 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.785 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.785 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:11.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.786 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_sessions": { 2024-04-25T11:47:11.786 INFO:teuthology.orchestra.run.smithi090.stdout: "session_count": 1, 2024-04-25T11:47:11.786 INFO:teuthology.orchestra.run.smithi090.stdout: "session_add": 1, 2024-04-25T11:47:11.786 INFO:teuthology.orchestra.run.smithi090.stdout: "session_remove": 0, 2024-04-25T11:47:11.786 INFO:teuthology.orchestra.run.smithi090.stdout: "sessions_open": 1, 2024-04-25T11:47:11.786 INFO:teuthology.orchestra.run.smithi090.stdout: "sessions_stale": 0, 2024-04-25T11:47:11.786 INFO:teuthology.orchestra.run.smithi090.stdout: "total_load": 0, 2024-04-25T11:47:11.786 INFO:teuthology.orchestra.run.smithi090.stdout: "average_load": 0, 2024-04-25T11:47:11.786 INFO:teuthology.orchestra.run.smithi090.stdout: "avg_session_uptime": 1881, 2024-04-25T11:47:11.786 INFO:teuthology.orchestra.run.smithi090.stdout: "mdthresh_evicted": 0 2024-04-25T11:47:11.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.786 INFO:teuthology.orchestra.run.smithi090.stdout: "mempool": { 2024-04-25T11:47:11.787 INFO:teuthology.orchestra.run.smithi090.stdout: "bloom_filter_bytes": 0, 2024-04-25T11:47:11.787 INFO:teuthology.orchestra.run.smithi090.stdout: "bloom_filter_items": 0, 2024-04-25T11:47:11.787 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T11:47:11.787 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_alloc_items": 0, 2024-04-25T11:47:11.787 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T11:47:11.787 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_data_items": 0, 2024-04-25T11:47:11.787 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T11:47:11.787 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T11:47:11.787 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T11:47:11.787 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T11:47:11.787 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T11:47:11.788 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_other_items": 0, 2024-04-25T11:47:11.788 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T11:47:11.788 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T11:47:11.788 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_bytes": 0, 2024-04-25T11:47:11.788 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_items": 0, 2024-04-25T11:47:11.788 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_blob_bytes": 0, 2024-04-25T11:47:11.788 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_blob_items": 0, 2024-04-25T11:47:11.788 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T11:47:11.788 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T11:47:11.788 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T11:47:11.788 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T11:47:11.788 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T11:47:11.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_items": 0, 2024-04-25T11:47:11.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_txc_bytes": 0, 2024-04-25T11:47:11.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_txc_items": 0, 2024-04-25T11:47:11.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T11:47:11.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T11:47:11.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_bytes": 0, 2024-04-25T11:47:11.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_items": 0, 2024-04-25T11:47:11.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_bytes": 0, 2024-04-25T11:47:11.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_items": 0, 2024-04-25T11:47:11.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T11:47:11.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_reader_items": 0, 2024-04-25T11:47:11.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T11:47:11.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_writer_items": 0, 2024-04-25T11:47:11.790 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_anon_bytes": 197473, 2024-04-25T11:47:11.790 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_anon_items": 55, 2024-04-25T11:47:11.790 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_meta_bytes": 0, 2024-04-25T11:47:11.790 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_meta_items": 0, 2024-04-25T11:47:11.790 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_bytes": 0, 2024-04-25T11:47:11.790 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_items": 0, 2024-04-25T11:47:11.790 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mapbl_bytes": 0, 2024-04-25T11:47:11.790 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mapbl_items": 0, 2024-04-25T11:47:11.790 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pglog_bytes": 0, 2024-04-25T11:47:11.790 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pglog_items": 0, 2024-04-25T11:47:11.790 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_bytes": 8512, 2024-04-25T11:47:11.791 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_items": 177, 2024-04-25T11:47:11.791 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T11:47:11.791 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_mapping_items": 0, 2024-04-25T11:47:11.791 INFO:teuthology.orchestra.run.smithi090.stdout: "pgmap_bytes": 0, 2024-04-25T11:47:11.791 INFO:teuthology.orchestra.run.smithi090.stdout: "pgmap_items": 0, 2024-04-25T11:47:11.791 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_co_bytes": 8512, 2024-04-25T11:47:11.791 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_co_items": 13, 2024-04-25T11:47:11.791 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_1_bytes": 0, 2024-04-25T11:47:11.791 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_1_items": 0, 2024-04-25T11:47:11.791 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_2_bytes": 0, 2024-04-25T11:47:11.792 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_2_items": 0 2024-04-25T11:47:11.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.792 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter": { 2024-04-25T11:47:11.792 INFO:teuthology.orchestra.run.smithi090.stdout: "op_active": 0, 2024-04-25T11:47:11.792 INFO:teuthology.orchestra.run.smithi090.stdout: "op_laggy": 0, 2024-04-25T11:47:11.792 INFO:teuthology.orchestra.run.smithi090.stdout: "op_send": 2324, 2024-04-25T11:47:11.792 INFO:teuthology.orchestra.run.smithi090.stdout: "op_send_bytes": 21, 2024-04-25T11:47:11.792 INFO:teuthology.orchestra.run.smithi090.stdout: "op_resend": 3, 2024-04-25T11:47:11.792 INFO:teuthology.orchestra.run.smithi090.stdout: "op_reply": 2324, 2024-04-25T11:47:11.792 INFO:teuthology.orchestra.run.smithi090.stdout: "op_latency": { 2024-04-25T11:47:11.792 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 2324, 2024-04-25T11:47:11.792 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 9.083864187, 2024-04-25T11:47:11.793 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.003908719 2024-04-25T11:47:11.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.793 INFO:teuthology.orchestra.run.smithi090.stdout: "op_inflight": 0, 2024-04-25T11:47:11.793 INFO:teuthology.orchestra.run.smithi090.stdout: "oplen_avg": { 2024-04-25T11:47:11.793 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 2324, 2024-04-25T11:47:11.793 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 2325 2024-04-25T11:47:11.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.793 INFO:teuthology.orchestra.run.smithi090.stdout: "op": 2324, 2024-04-25T11:47:11.793 INFO:teuthology.orchestra.run.smithi090.stdout: "op_r": 2324, 2024-04-25T11:47:11.793 INFO:teuthology.orchestra.run.smithi090.stdout: "op_w": 0, 2024-04-25T11:47:11.793 INFO:teuthology.orchestra.run.smithi090.stdout: "op_rmw": 0, 2024-04-25T11:47:11.793 INFO:teuthology.orchestra.run.smithi090.stdout: "op_pg": 0, 2024-04-25T11:47:11.794 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_stat": 1490, 2024-04-25T11:47:11.794 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_create": 0, 2024-04-25T11:47:11.794 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_read": 832, 2024-04-25T11:47:11.794 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_write": 0, 2024-04-25T11:47:11.794 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_writefull": 0, 2024-04-25T11:47:11.794 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_writesame": 0, 2024-04-25T11:47:11.794 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_append": 0, 2024-04-25T11:47:11.794 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_zero": 0, 2024-04-25T11:47:11.794 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_truncate": 0, 2024-04-25T11:47:11.794 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_delete": 0, 2024-04-25T11:47:11.794 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_mapext": 0, 2024-04-25T11:47:11.795 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_sparse_read": 0, 2024-04-25T11:47:11.795 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_clonerange": 0, 2024-04-25T11:47:11.795 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_getxattr": 1, 2024-04-25T11:47:11.795 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_setxattr": 0, 2024-04-25T11:47:11.795 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_cmpxattr": 0, 2024-04-25T11:47:11.795 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_rmxattr": 0, 2024-04-25T11:47:11.795 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_resetxattrs": 0, 2024-04-25T11:47:11.795 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_call": 0, 2024-04-25T11:47:11.795 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_watch": 0, 2024-04-25T11:47:11.795 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_notify": 0, 2024-04-25T11:47:11.795 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T11:47:11.795 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_pgls": 0, 2024-04-25T11:47:11.796 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_pgls_filter": 0, 2024-04-25T11:47:11.796 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_other": 0, 2024-04-25T11:47:11.796 INFO:teuthology.orchestra.run.smithi090.stdout: "linger_active": 0, 2024-04-25T11:47:11.796 INFO:teuthology.orchestra.run.smithi090.stdout: "linger_send": 0, 2024-04-25T11:47:11.796 INFO:teuthology.orchestra.run.smithi090.stdout: "linger_resend": 0, 2024-04-25T11:47:11.796 INFO:teuthology.orchestra.run.smithi090.stdout: "linger_ping": 0, 2024-04-25T11:47:11.796 INFO:teuthology.orchestra.run.smithi090.stdout: "poolop_active": 0, 2024-04-25T11:47:11.796 INFO:teuthology.orchestra.run.smithi090.stdout: "poolop_send": 0, 2024-04-25T11:47:11.796 INFO:teuthology.orchestra.run.smithi090.stdout: "poolop_resend": 0, 2024-04-25T11:47:11.796 INFO:teuthology.orchestra.run.smithi090.stdout: "poolstat_active": 0, 2024-04-25T11:47:11.796 INFO:teuthology.orchestra.run.smithi090.stdout: "poolstat_send": 0, 2024-04-25T11:47:11.797 INFO:teuthology.orchestra.run.smithi090.stdout: "poolstat_resend": 0, 2024-04-25T11:47:11.797 INFO:teuthology.orchestra.run.smithi090.stdout: "statfs_active": 0, 2024-04-25T11:47:11.797 INFO:teuthology.orchestra.run.smithi090.stdout: "statfs_send": 0, 2024-04-25T11:47:11.797 INFO:teuthology.orchestra.run.smithi090.stdout: "statfs_resend": 0, 2024-04-25T11:47:11.797 INFO:teuthology.orchestra.run.smithi090.stdout: "command_active": 0, 2024-04-25T11:47:11.797 INFO:teuthology.orchestra.run.smithi090.stdout: "command_send": 0, 2024-04-25T11:47:11.797 INFO:teuthology.orchestra.run.smithi090.stdout: "command_resend": 0, 2024-04-25T11:47:11.797 INFO:teuthology.orchestra.run.smithi090.stdout: "map_epoch": 93, 2024-04-25T11:47:11.797 INFO:teuthology.orchestra.run.smithi090.stdout: "map_full": 0, 2024-04-25T11:47:11.797 INFO:teuthology.orchestra.run.smithi090.stdout: "map_inc": 16, 2024-04-25T11:47:11.797 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_sessions": 6, 2024-04-25T11:47:11.797 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_session_open": 6, 2024-04-25T11:47:11.798 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_session_close": 0, 2024-04-25T11:47:11.798 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_laggy": 0, 2024-04-25T11:47:11.798 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_wr": 0, 2024-04-25T11:47:11.798 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_rd": 2, 2024-04-25T11:47:11.798 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_del": 0 2024-04-25T11:47:11.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.798 INFO:teuthology.orchestra.run.smithi090.stdout: "oft": { 2024-04-25T11:47:11.798 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_total_objs": 0, 2024-04-25T11:47:11.799 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_total_kv_pairs": 0, 2024-04-25T11:47:11.799 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_total_updates": 0, 2024-04-25T11:47:11.799 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_total_removes": 0 2024-04-25T11:47:11.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.799 INFO:teuthology.orchestra.run.smithi090.stdout: "osd-slow-ops": { 2024-04-25T11:47:11.799 INFO:teuthology.orchestra.run.smithi090.stdout: "slow_ops_count": 0 2024-04-25T11:47:11.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.799 INFO:teuthology.orchestra.run.smithi090.stdout: "purge_queue": { 2024-04-25T11:47:11.799 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_executing_ops": 0, 2024-04-25T11:47:11.799 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T11:47:11.799 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_executing": 0, 2024-04-25T11:47:11.799 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_executing_high_water": 0, 2024-04-25T11:47:11.800 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_executed_ops": 0, 2024-04-25T11:47:11.800 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_executed": 0, 2024-04-25T11:47:11.800 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_item_in_journal": 0 2024-04-25T11:47:11.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.800 INFO:teuthology.orchestra.run.smithi090.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T11:47:11.800 INFO:teuthology.orchestra.run.smithi090.stdout: "val": 0, 2024-04-25T11:47:11.800 INFO:teuthology.orchestra.run.smithi090.stdout: "max": 104857600, 2024-04-25T11:47:11.800 INFO:teuthology.orchestra.run.smithi090.stdout: "get_started": 0, 2024-04-25T11:47:11.800 INFO:teuthology.orchestra.run.smithi090.stdout: "get": 2589, 2024-04-25T11:47:11.800 INFO:teuthology.orchestra.run.smithi090.stdout: "get_sum": 2011837, 2024-04-25T11:47:11.800 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:11.800 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_success": 2589, 2024-04-25T11:47:11.800 INFO:teuthology.orchestra.run.smithi090.stdout: "take": 0, 2024-04-25T11:47:11.801 INFO:teuthology.orchestra.run.smithi090.stdout: "take_sum": 0, 2024-04-25T11:47:11.801 INFO:teuthology.orchestra.run.smithi090.stdout: "put": 2589, 2024-04-25T11:47:11.801 INFO:teuthology.orchestra.run.smithi090.stdout: "put_sum": 2011837, 2024-04-25T11:47:11.801 INFO:teuthology.orchestra.run.smithi090.stdout: "wait": { 2024-04-25T11:47:11.801 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.801 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.801 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.801 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:11.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.801 INFO:teuthology.orchestra.run.smithi090.stdout: "throttle-objecter_bytes": { 2024-04-25T11:47:11.801 INFO:teuthology.orchestra.run.smithi090.stdout: "val": 0, 2024-04-25T11:47:11.801 INFO:teuthology.orchestra.run.smithi090.stdout: "max": 104857600, 2024-04-25T11:47:11.802 INFO:teuthology.orchestra.run.smithi090.stdout: "get_started": 0, 2024-04-25T11:47:11.802 INFO:teuthology.orchestra.run.smithi090.stdout: "get": 0, 2024-04-25T11:47:11.802 INFO:teuthology.orchestra.run.smithi090.stdout: "get_sum": 0, 2024-04-25T11:47:11.802 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:11.802 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:11.802 INFO:teuthology.orchestra.run.smithi090.stdout: "take": 2324, 2024-04-25T11:47:11.802 INFO:teuthology.orchestra.run.smithi090.stdout: "take_sum": 984, 2024-04-25T11:47:11.802 INFO:teuthology.orchestra.run.smithi090.stdout: "put": 2324, 2024-04-25T11:47:11.802 INFO:teuthology.orchestra.run.smithi090.stdout: "put_sum": 984, 2024-04-25T11:47:11.802 INFO:teuthology.orchestra.run.smithi090.stdout: "wait": { 2024-04-25T11:47:11.802 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.802 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.803 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.803 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:11.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.803 INFO:teuthology.orchestra.run.smithi090.stdout: "throttle-objecter_ops": { 2024-04-25T11:47:11.803 INFO:teuthology.orchestra.run.smithi090.stdout: "val": 0, 2024-04-25T11:47:11.803 INFO:teuthology.orchestra.run.smithi090.stdout: "max": 1024, 2024-04-25T11:47:11.803 INFO:teuthology.orchestra.run.smithi090.stdout: "get_started": 0, 2024-04-25T11:47:11.803 INFO:teuthology.orchestra.run.smithi090.stdout: "get": 0, 2024-04-25T11:47:11.803 INFO:teuthology.orchestra.run.smithi090.stdout: "get_sum": 0, 2024-04-25T11:47:11.803 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:11.803 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:11.804 INFO:teuthology.orchestra.run.smithi090.stdout: "take": 2324, 2024-04-25T11:47:11.804 INFO:teuthology.orchestra.run.smithi090.stdout: "take_sum": 2324, 2024-04-25T11:47:11.804 INFO:teuthology.orchestra.run.smithi090.stdout: "put": 2324, 2024-04-25T11:47:11.804 INFO:teuthology.orchestra.run.smithi090.stdout: "put_sum": 2324, 2024-04-25T11:47:11.804 INFO:teuthology.orchestra.run.smithi090.stdout: "wait": { 2024-04-25T11:47:11.804 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.804 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.804 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.804 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:11.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.804 INFO:teuthology.orchestra.run.smithi090.stdout: "throttle-write_buf_throttle": { 2024-04-25T11:47:11.805 INFO:teuthology.orchestra.run.smithi090.stdout: "val": 0, 2024-04-25T11:47:11.805 INFO:teuthology.orchestra.run.smithi090.stdout: "max": 3758096384, 2024-04-25T11:47:11.805 INFO:teuthology.orchestra.run.smithi090.stdout: "get_started": 0, 2024-04-25T11:47:11.805 INFO:teuthology.orchestra.run.smithi090.stdout: "get": 0, 2024-04-25T11:47:11.805 INFO:teuthology.orchestra.run.smithi090.stdout: "get_sum": 0, 2024-04-25T11:47:11.805 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:11.805 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:11.805 INFO:teuthology.orchestra.run.smithi090.stdout: "take": 0, 2024-04-25T11:47:11.805 INFO:teuthology.orchestra.run.smithi090.stdout: "take_sum": 0, 2024-04-25T11:47:11.805 INFO:teuthology.orchestra.run.smithi090.stdout: "put": 0, 2024-04-25T11:47:11.805 INFO:teuthology.orchestra.run.smithi090.stdout: "put_sum": 0, 2024-04-25T11:47:11.805 INFO:teuthology.orchestra.run.smithi090.stdout: "wait": { 2024-04-25T11:47:11.805 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.806 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.806 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.806 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:11.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:11.806 INFO:teuthology.orchestra.run.smithi090.stdout: "throttle-write_buf_throttle-0x55f20eeb2c30": { 2024-04-25T11:47:11.806 INFO:teuthology.orchestra.run.smithi090.stdout: "val": 0, 2024-04-25T11:47:11.806 INFO:teuthology.orchestra.run.smithi090.stdout: "max": 3758096384, 2024-04-25T11:47:11.806 INFO:teuthology.orchestra.run.smithi090.stdout: "get_started": 0, 2024-04-25T11:47:11.806 INFO:teuthology.orchestra.run.smithi090.stdout: "get": 0, 2024-04-25T11:47:11.806 INFO:teuthology.orchestra.run.smithi090.stdout: "get_sum": 0, 2024-04-25T11:47:11.806 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:11.807 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:11.807 INFO:teuthology.orchestra.run.smithi090.stdout: "take": 0, 2024-04-25T11:47:11.807 INFO:teuthology.orchestra.run.smithi090.stdout: "take_sum": 0, 2024-04-25T11:47:11.807 INFO:teuthology.orchestra.run.smithi090.stdout: "put": 0, 2024-04-25T11:47:11.807 INFO:teuthology.orchestra.run.smithi090.stdout: "put_sum": 0, 2024-04-25T11:47:11.807 INFO:teuthology.orchestra.run.smithi090.stdout: "wait": { 2024-04-25T11:47:11.807 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:11.807 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:11.807 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:11.807 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:11.807 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:11.807 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:47:12.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:12 smithi090 ceph-mon[29575]: pgmap v376: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 15 KiB/s rd, 50 KiB/s wr, 20 op/s 2024-04-25T11:47:12.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:12 smithi090 ceph-mon[29575]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T11:47:12.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:12 smithi138 ceph-mon[32328]: pgmap v376: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 15 KiB/s rd, 50 KiB/s wr, 20 op/s 2024-04-25T11:47:12.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:12 smithi138 ceph-mon[32328]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T11:47:12.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:12 smithi156 ceph-mon[33045]: pgmap v376: 97 pgs: 97 active+clean; 21 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 15 KiB/s rd, 50 KiB/s wr, 20 op/s 2024-04-25T11:47:12.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:12 smithi156 ceph-mon[33045]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T11:47:12.685 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-04-25T11:47:12.685 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-04-25T11:47:12.685 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update=0 2024-04-25T11:47:12.685 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update_receipt=0 2024-04-25T11:47:12.685 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=0 2024-04-25T11:47:12.685 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-04-25T11:47:12.685 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T11:47:12.685 DEBUG:tasks.check_counter:Getting stats from j 2024-04-25T11:47:12.686 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-25T11:47:12.944 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T11:47:14.355 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:14 smithi090 ceph-mon[29575]: pgmap v378: 97 pgs: 4 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 83 active+clean; 20 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 9.9 KiB/s rd, 54 KiB/s wr, 18 op/s 2024-04-25T11:47:14.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:14 smithi138 ceph-mon[32328]: pgmap v378: 97 pgs: 4 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 83 active+clean; 20 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 9.9 KiB/s rd, 54 KiB/s wr, 18 op/s 2024-04-25T11:47:14.584 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:14 smithi156 ceph-mon[33045]: pgmap v378: 97 pgs: 4 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 83 active+clean; 20 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 9.9 KiB/s rd, 54 KiB/s wr, 18 op/s 2024-04-25T11:47:14.636 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-25T11:47:14.636 INFO:teuthology.orchestra.run.smithi090.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T11:47:14.637 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_messages": 248, 2024-04-25T11:47:14.637 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages": 236, 2024-04-25T11:47:14.637 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_bytes": 1548960, 2024-04-25T11:47:14.637 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_bytes": 144512, 2024-04-25T11:47:14.637 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_created_connections": 4, 2024-04-25T11:47:14.637 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_active_connections": 0, 2024-04-25T11:47:14.637 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_total_time": 0.055105733, 2024-04-25T11:47:14.637 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_send_time": 0.015493067, 2024-04-25T11:47:14.637 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_recv_time": 0.043690735, 2024-04-25T11:47:14.637 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_fast_dispatch_time": 0.005462747, 2024-04-25T11:47:14.637 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:14.637 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 236, 2024-04-25T11:47:14.638 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.020368920, 2024-04-25T11:47:14.638 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000086308 2024-04-25T11:47:14.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:14.638 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:14.638 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:14.638 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:14.638 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:14.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:14.638 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_encrypted_bytes": 1548960, 2024-04-25T11:47:14.638 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_encrypted_bytes": 144512 2024-04-25T11:47:14.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:14.638 INFO:teuthology.orchestra.run.smithi090.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T11:47:14.638 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_messages": 2, 2024-04-25T11:47:14.639 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages": 180, 2024-04-25T11:47:14.639 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_bytes": 320, 2024-04-25T11:47:14.639 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_bytes": 188784, 2024-04-25T11:47:14.639 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_created_connections": 5, 2024-04-25T11:47:14.639 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_active_connections": 3, 2024-04-25T11:47:14.639 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_total_time": 0.015702779, 2024-04-25T11:47:14.639 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_send_time": 0.010694041, 2024-04-25T11:47:14.639 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_recv_time": 0.002055972, 2024-04-25T11:47:14.639 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T11:47:14.639 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:14.639 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 177, 2024-04-25T11:47:14.639 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.020850751, 2024-04-25T11:47:14.639 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000117800 2024-04-25T11:47:14.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:14.640 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:14.640 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:14.640 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:14.640 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:14.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:14.640 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T11:47:14.640 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_encrypted_bytes": 188784 2024-04-25T11:47:14.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:14.640 INFO:teuthology.orchestra.run.smithi090.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T11:47:14.640 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_messages": 0, 2024-04-25T11:47:14.640 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages": 2, 2024-04-25T11:47:14.640 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_bytes": 0, 2024-04-25T11:47:14.640 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_bytes": 0, 2024-04-25T11:47:14.641 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_created_connections": 2, 2024-04-25T11:47:14.641 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_active_connections": 1, 2024-04-25T11:47:14.641 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_total_time": 0.000288038, 2024-04-25T11:47:14.641 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T11:47:14.641 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_recv_time": 0.000216937, 2024-04-25T11:47:14.641 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T11:47:14.641 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:14.641 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:14.641 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:14.641 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:14.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:14.641 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:14.641 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:14.642 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:14.642 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:14.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:14.642 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T11:47:14.642 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T11:47:14.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:14.642 INFO:teuthology.orchestra.run.smithi090.stdout: "cct": { 2024-04-25T11:47:14.642 INFO:teuthology.orchestra.run.smithi090.stdout: "total_workers": 0, 2024-04-25T11:47:14.642 INFO:teuthology.orchestra.run.smithi090.stdout: "unhealthy_workers": 0 2024-04-25T11:47:14.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:14.642 INFO:teuthology.orchestra.run.smithi090.stdout: "mempool": { 2024-04-25T11:47:14.642 INFO:teuthology.orchestra.run.smithi090.stdout: "bloom_filter_bytes": 0, 2024-04-25T11:47:14.642 INFO:teuthology.orchestra.run.smithi090.stdout: "bloom_filter_items": 0, 2024-04-25T11:47:14.643 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T11:47:14.643 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_alloc_items": 0, 2024-04-25T11:47:14.643 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T11:47:14.643 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_data_items": 0, 2024-04-25T11:47:14.643 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T11:47:14.643 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T11:47:14.643 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T11:47:14.643 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T11:47:14.643 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T11:47:14.643 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_other_items": 0, 2024-04-25T11:47:14.643 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T11:47:14.643 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T11:47:14.643 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_bytes": 0, 2024-04-25T11:47:14.643 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_items": 0, 2024-04-25T11:47:14.644 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_blob_bytes": 0, 2024-04-25T11:47:14.644 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_blob_items": 0, 2024-04-25T11:47:14.644 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T11:47:14.644 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T11:47:14.644 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T11:47:14.644 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T11:47:14.644 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T11:47:14.644 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_items": 0, 2024-04-25T11:47:14.644 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_txc_bytes": 0, 2024-04-25T11:47:14.644 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_txc_items": 0, 2024-04-25T11:47:14.644 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T11:47:14.644 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T11:47:14.644 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_bytes": 0, 2024-04-25T11:47:14.645 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_items": 0, 2024-04-25T11:47:14.645 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_bytes": 0, 2024-04-25T11:47:14.645 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_items": 0, 2024-04-25T11:47:14.645 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T11:47:14.645 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_reader_items": 0, 2024-04-25T11:47:14.645 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T11:47:14.645 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_writer_items": 0, 2024-04-25T11:47:14.645 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_anon_bytes": 148977, 2024-04-25T11:47:14.645 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_anon_items": 30, 2024-04-25T11:47:14.645 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_meta_bytes": 0, 2024-04-25T11:47:14.645 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_meta_items": 0, 2024-04-25T11:47:14.645 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_bytes": 0, 2024-04-25T11:47:14.645 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_items": 0, 2024-04-25T11:47:14.645 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mapbl_bytes": 0, 2024-04-25T11:47:14.646 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mapbl_items": 0, 2024-04-25T11:47:14.646 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pglog_bytes": 0, 2024-04-25T11:47:14.646 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pglog_items": 0, 2024-04-25T11:47:14.646 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_bytes": 0, 2024-04-25T11:47:14.646 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_items": 0, 2024-04-25T11:47:14.646 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T11:47:14.646 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_mapping_items": 0, 2024-04-25T11:47:14.646 INFO:teuthology.orchestra.run.smithi090.stdout: "pgmap_bytes": 0, 2024-04-25T11:47:14.646 INFO:teuthology.orchestra.run.smithi090.stdout: "pgmap_items": 0, 2024-04-25T11:47:14.646 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_co_bytes": 928, 2024-04-25T11:47:14.646 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_co_items": 2, 2024-04-25T11:47:14.646 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_1_bytes": 0, 2024-04-25T11:47:14.646 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_1_items": 0, 2024-04-25T11:47:14.647 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_2_bytes": 0, 2024-04-25T11:47:14.647 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_2_items": 0 2024-04-25T11:47:14.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-25T11:47:14.647 INFO:teuthology.orchestra.run.smithi090.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T11:47:14.647 INFO:teuthology.orchestra.run.smithi090.stdout: "val": 0, 2024-04-25T11:47:14.647 INFO:teuthology.orchestra.run.smithi090.stdout: "max": 104857600, 2024-04-25T11:47:14.647 INFO:teuthology.orchestra.run.smithi090.stdout: "get_started": 0, 2024-04-25T11:47:14.647 INFO:teuthology.orchestra.run.smithi090.stdout: "get": 246, 2024-04-25T11:47:14.647 INFO:teuthology.orchestra.run.smithi090.stdout: "get_sum": 1511534, 2024-04-25T11:47:14.647 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:14.647 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_success": 246, 2024-04-25T11:47:14.647 INFO:teuthology.orchestra.run.smithi090.stdout: "take": 0, 2024-04-25T11:47:14.647 INFO:teuthology.orchestra.run.smithi090.stdout: "take_sum": 0, 2024-04-25T11:47:14.647 INFO:teuthology.orchestra.run.smithi090.stdout: "put": 246, 2024-04-25T11:47:14.648 INFO:teuthology.orchestra.run.smithi090.stdout: "put_sum": 1511534, 2024-04-25T11:47:14.648 INFO:teuthology.orchestra.run.smithi090.stdout: "wait": { 2024-04-25T11:47:14.648 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-25T11:47:14.648 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-25T11:47:14.648 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-25T11:47:14.648 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:14.648 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-25T11:47:14.648 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-25T11:47:15.366 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-25T11:47:15.366 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-25T11:47:15.366 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-04-25T11:47:15.366 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-04-25T11:47:15.366 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-25T11:47:15.366 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-25T11:47:15.366 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-25T11:47:15.366 DEBUG:tasks.check_counter:Getting stats from b 2024-04-25T11:47:15.367 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-25T11:47:15.627 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T11:47:16.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:16 smithi090 ceph-mon[29575]: pgmap v379: 97 pgs: 16 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 57 active+clean; 19 GiB data, 61 GiB used, 1011 GiB / 1.0 TiB avail; 9.4 KiB/s rd, 27 KiB/s wr, 16 op/s 2024-04-25T11:47:16.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:16 smithi138 ceph-mon[32328]: pgmap v379: 97 pgs: 16 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 57 active+clean; 19 GiB data, 61 GiB used, 1011 GiB / 1.0 TiB avail; 9.4 KiB/s rd, 27 KiB/s wr, 16 op/s 2024-04-25T11:47:16.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:16 smithi156 ceph-mon[33045]: pgmap v379: 97 pgs: 16 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 57 active+clean; 19 GiB data, 61 GiB used, 1011 GiB / 1.0 TiB avail; 9.4 KiB/s rd, 27 KiB/s wr, 16 op/s 2024-04-25T11:47:17.607 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-25T11:47:17.607 INFO:teuthology.orchestra.run.smithi138.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T11:47:17.607 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_messages": 6992, 2024-04-25T11:47:17.607 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_messages": 7909, 2024-04-25T11:47:17.607 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_bytes": 1945546, 2024-04-25T11:47:17.608 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_bytes": 184072021, 2024-04-25T11:47:17.608 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_created_connections": 12, 2024-04-25T11:47:17.608 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_active_connections": 5, 2024-04-25T11:47:17.608 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_total_time": 1.380009088, 2024-04-25T11:47:17.608 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_send_time": 0.823579902, 2024-04-25T11:47:17.608 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_recv_time": 0.195143457, 2024-04-25T11:47:17.608 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_fast_dispatch_time": 0.315102896, 2024-04-25T11:47:17.608 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:17.608 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 7908, 2024-04-25T11:47:17.608 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 1.582555872, 2024-04-25T11:47:17.608 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000200120 2024-04-25T11:47:17.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.609 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:17.609 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 4255, 2024-04-25T11:47:17.609 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.001042250, 2024-04-25T11:47:17.609 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000244 2024-04-25T11:47:17.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.609 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-04-25T11:47:17.609 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_encrypted_bytes": 354288 2024-04-25T11:47:17.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.609 INFO:teuthology.orchestra.run.smithi138.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T11:47:17.609 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_messages": 196992, 2024-04-25T11:47:17.609 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_messages": 198927, 2024-04-25T11:47:17.609 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_bytes": 64199363, 2024-04-25T11:47:17.609 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_bytes": 276028960, 2024-04-25T11:47:17.610 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_created_connections": 42, 2024-04-25T11:47:17.610 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_active_connections": 5, 2024-04-25T11:47:17.610 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_total_time": 10.739894176, 2024-04-25T11:47:17.610 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_send_time": 4.678485221, 2024-04-25T11:47:17.610 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_recv_time": 12.561495769, 2024-04-25T11:47:17.610 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_fast_dispatch_time": 0.362540009, 2024-04-25T11:47:17.610 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:17.610 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 198924, 2024-04-25T11:47:17.610 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 19.299845664, 2024-04-25T11:47:17.610 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000097021 2024-04-25T11:47:17.610 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.610 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:17.610 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 324837, 2024-04-25T11:47:17.610 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.087766014, 2024-04-25T11:47:17.611 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000270 2024-04-25T11:47:17.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.611 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_encrypted_bytes": 460352, 2024-04-25T11:47:17.611 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_encrypted_bytes": 268800 2024-04-25T11:47:17.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.611 INFO:teuthology.orchestra.run.smithi138.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T11:47:17.611 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_messages": 4576, 2024-04-25T11:47:17.611 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_messages": 5241, 2024-04-25T11:47:17.611 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_bytes": 3042055, 2024-04-25T11:47:17.611 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_bytes": 133973724, 2024-04-25T11:47:17.611 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_created_connections": 246, 2024-04-25T11:47:17.611 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_active_connections": 9, 2024-04-25T11:47:17.611 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_total_time": 1.124433673, 2024-04-25T11:47:17.611 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_send_time": 0.520155916, 2024-04-25T11:47:17.612 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_recv_time": 0.305677245, 2024-04-25T11:47:17.612 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_fast_dispatch_time": 0.298288778, 2024-04-25T11:47:17.612 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:17.612 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 5240, 2024-04-25T11:47:17.612 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 1.012465534, 2024-04-25T11:47:17.612 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000193218 2024-04-25T11:47:17.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.612 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:17.612 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 4808, 2024-04-25T11:47:17.612 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.001263477, 2024-04-25T11:47:17.612 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000262 2024-04-25T11:47:17.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.612 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_encrypted_bytes": 1690560, 2024-04-25T11:47:17.612 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_encrypted_bytes": 179184 2024-04-25T11:47:17.613 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.613 INFO:teuthology.orchestra.run.smithi138.stdout: "cct": { 2024-04-25T11:47:17.613 INFO:teuthology.orchestra.run.smithi138.stdout: "total_workers": 1, 2024-04-25T11:47:17.613 INFO:teuthology.orchestra.run.smithi138.stdout: "unhealthy_workers": 0 2024-04-25T11:47:17.613 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.613 INFO:teuthology.orchestra.run.smithi138.stdout: "finisher-MDSRank": { 2024-04-25T11:47:17.613 INFO:teuthology.orchestra.run.smithi138.stdout: "queue_len": 0, 2024-04-25T11:47:17.613 INFO:teuthology.orchestra.run.smithi138.stdout: "complete_latency": { 2024-04-25T11:47:17.613 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 2950, 2024-04-25T11:47:17.613 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 115.425789897, 2024-04-25T11:47:17.613 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.039127386 2024-04-25T11:47:17.613 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:17.613 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.613 INFO:teuthology.orchestra.run.smithi138.stdout: "finisher-PurgeQueue": { 2024-04-25T11:47:17.613 INFO:teuthology.orchestra.run.smithi138.stdout: "queue_len": 0, 2024-04-25T11:47:17.614 INFO:teuthology.orchestra.run.smithi138.stdout: "complete_latency": { 2024-04-25T11:47:17.614 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 2473, 2024-04-25T11:47:17.614 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.081385162, 2024-04-25T11:47:17.614 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000032909 2024-04-25T11:47:17.614 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:17.614 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.614 INFO:teuthology.orchestra.run.smithi138.stdout: "mds": { 2024-04-25T11:47:17.614 INFO:teuthology.orchestra.run.smithi138.stdout: "request": 237, 2024-04-25T11:47:17.614 INFO:teuthology.orchestra.run.smithi138.stdout: "reply": 236, 2024-04-25T11:47:17.614 INFO:teuthology.orchestra.run.smithi138.stdout: "reply_latency": { 2024-04-25T11:47:17.614 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 236, 2024-04-25T11:47:17.614 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 7.298083656, 2024-04-25T11:47:17.614 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.030924083 2024-04-25T11:47:17.614 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.615 INFO:teuthology.orchestra.run.smithi138.stdout: "slow_reply": 0, 2024-04-25T11:47:17.615 INFO:teuthology.orchestra.run.smithi138.stdout: "forward": 1, 2024-04-25T11:47:17.615 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_fetch_complete": 0, 2024-04-25T11:47:17.615 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_fetch_keys": 0, 2024-04-25T11:47:17.615 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_commit": 106, 2024-04-25T11:47:17.615 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_split": 0, 2024-04-25T11:47:17.615 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_merge": 0, 2024-04-25T11:47:17.615 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes": 124, 2024-04-25T11:47:17.615 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes_top": 45, 2024-04-25T11:47:17.615 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes_bottom": 79, 2024-04-25T11:47:17.615 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes_pin_tail": 0, 2024-04-25T11:47:17.615 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes_pinned": 59, 2024-04-25T11:47:17.615 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes_expired": 2, 2024-04-25T11:47:17.615 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes_with_caps": 2, 2024-04-25T11:47:17.616 INFO:teuthology.orchestra.run.smithi138.stdout: "caps": 3, 2024-04-25T11:47:17.616 INFO:teuthology.orchestra.run.smithi138.stdout: "subtrees": 3, 2024-04-25T11:47:17.616 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse": 319, 2024-04-25T11:47:17.616 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse_hit": 303, 2024-04-25T11:47:17.616 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse_forward": 0, 2024-04-25T11:47:17.616 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse_discover": 0, 2024-04-25T11:47:17.616 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse_dir_fetch": 0, 2024-04-25T11:47:17.616 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse_remote_ino": 0, 2024-04-25T11:47:17.616 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse_lock": 0, 2024-04-25T11:47:17.616 INFO:teuthology.orchestra.run.smithi138.stdout: "load_cent": 105, 2024-04-25T11:47:17.616 INFO:teuthology.orchestra.run.smithi138.stdout: "q": 0, 2024-04-25T11:47:17.616 INFO:teuthology.orchestra.run.smithi138.stdout: "exported": 1, 2024-04-25T11:47:17.616 INFO:teuthology.orchestra.run.smithi138.stdout: "exported_inodes": 2, 2024-04-25T11:47:17.617 INFO:teuthology.orchestra.run.smithi138.stdout: "imported": 0, 2024-04-25T11:47:17.617 INFO:teuthology.orchestra.run.smithi138.stdout: "imported_inodes": 0, 2024-04-25T11:47:17.617 INFO:teuthology.orchestra.run.smithi138.stdout: "openino_dir_fetch": 0, 2024-04-25T11:47:17.617 INFO:teuthology.orchestra.run.smithi138.stdout: "openino_backtrace_fetch": 595, 2024-04-25T11:47:17.617 INFO:teuthology.orchestra.run.smithi138.stdout: "openino_peer_discover": 0, 2024-04-25T11:47:17.617 INFO:teuthology.orchestra.run.smithi138.stdout: "root_rfiles": 4, 2024-04-25T11:47:17.617 INFO:teuthology.orchestra.run.smithi138.stdout: "root_rbytes": 234, 2024-04-25T11:47:17.617 INFO:teuthology.orchestra.run.smithi138.stdout: "root_rsnaps": 6, 2024-04-25T11:47:17.617 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_backtrace_fetch": 595, 2024-04-25T11:47:17.617 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_set_tag": 0, 2024-04-25T11:47:17.617 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T11:47:17.617 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_inotable_repaired": 0, 2024-04-25T11:47:17.617 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_dir_inodes": 221, 2024-04-25T11:47:17.617 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_dir_base_inodes": 21, 2024-04-25T11:47:17.618 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_dirfrag_rstats": 200, 2024-04-25T11:47:17.618 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_file_inodes": 374, 2024-04-25T11:47:17.618 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_inode_file_caps": 2, 2024-04-25T11:47:17.618 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_cap_op_revoke": 48, 2024-04-25T11:47:17.618 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_cap_op_grant": 3385, 2024-04-25T11:47:17.618 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T11:47:17.618 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T11:47:17.618 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T11:47:17.618 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_client_caps": 191430, 2024-04-25T11:47:17.618 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_client_caps_dirty": 191369, 2024-04-25T11:47:17.618 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_client_cap_release": 1, 2024-04-25T11:47:17.618 INFO:teuthology.orchestra.run.smithi138.stdout: "process_request_cap_release": 47 2024-04-25T11:47:17.619 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.619 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_cache": { 2024-04-25T11:47:17.619 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_update": 13, 2024-04-25T11:47:17.619 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_update_receipt": 0, 2024-04-25T11:47:17.619 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_try_discover": 0, 2024-04-25T11:47:17.619 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_send_discover": 0, 2024-04-25T11:47:17.619 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_handle_discover": 35, 2024-04-25T11:47:17.619 INFO:teuthology.orchestra.run.smithi138.stdout: "num_strays": 4, 2024-04-25T11:47:17.619 INFO:teuthology.orchestra.run.smithi138.stdout: "num_strays_delayed": 0, 2024-04-25T11:47:17.619 INFO:teuthology.orchestra.run.smithi138.stdout: "num_strays_enqueuing": 0, 2024-04-25T11:47:17.619 INFO:teuthology.orchestra.run.smithi138.stdout: "strays_created": 20, 2024-04-25T11:47:17.619 INFO:teuthology.orchestra.run.smithi138.stdout: "strays_enqueued": 16, 2024-04-25T11:47:17.619 INFO:teuthology.orchestra.run.smithi138.stdout: "strays_reintegrated": 0, 2024-04-25T11:47:17.620 INFO:teuthology.orchestra.run.smithi138.stdout: "strays_migrated": 0, 2024-04-25T11:47:17.620 INFO:teuthology.orchestra.run.smithi138.stdout: "num_recovering_processing": 0, 2024-04-25T11:47:17.620 INFO:teuthology.orchestra.run.smithi138.stdout: "num_recovering_enqueued": 0, 2024-04-25T11:47:17.620 INFO:teuthology.orchestra.run.smithi138.stdout: "num_recovering_prioritized": 0, 2024-04-25T11:47:17.620 INFO:teuthology.orchestra.run.smithi138.stdout: "recovery_started": 0, 2024-04-25T11:47:17.620 INFO:teuthology.orchestra.run.smithi138.stdout: "recovery_completed": 0, 2024-04-25T11:47:17.620 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_quiesce_path": 0, 2024-04-25T11:47:17.620 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_quiesce_inode": 0, 2024-04-25T11:47:17.620 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_enqueue_scrub": 21, 2024-04-25T11:47:17.620 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_exportdir": 1, 2024-04-25T11:47:17.620 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_flush": 0, 2024-04-25T11:47:17.620 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_fragmentdir": 0, 2024-04-25T11:47:17.620 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_fragstats": 0, 2024-04-25T11:47:17.620 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_inodestats": 0 2024-04-25T11:47:17.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.621 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_log": { 2024-04-25T11:47:17.621 INFO:teuthology.orchestra.run.smithi138.stdout: "evlrg": 0, 2024-04-25T11:47:17.621 INFO:teuthology.orchestra.run.smithi138.stdout: "evadd": 192169, 2024-04-25T11:47:17.621 INFO:teuthology.orchestra.run.smithi138.stdout: "evex": 192099, 2024-04-25T11:47:17.621 INFO:teuthology.orchestra.run.smithi138.stdout: "evtrm": 192099, 2024-04-25T11:47:17.621 INFO:teuthology.orchestra.run.smithi138.stdout: "ev": 70, 2024-04-25T11:47:17.621 INFO:teuthology.orchestra.run.smithi138.stdout: "evexg": 0, 2024-04-25T11:47:17.621 INFO:teuthology.orchestra.run.smithi138.stdout: "evexd": 1, 2024-04-25T11:47:17.621 INFO:teuthology.orchestra.run.smithi138.stdout: "segadd": 295, 2024-04-25T11:47:17.621 INFO:teuthology.orchestra.run.smithi138.stdout: "segex": 294, 2024-04-25T11:47:17.621 INFO:teuthology.orchestra.run.smithi138.stdout: "segtrm": 294, 2024-04-25T11:47:17.621 INFO:teuthology.orchestra.run.smithi138.stdout: "seg": 1, 2024-04-25T11:47:17.621 INFO:teuthology.orchestra.run.smithi138.stdout: "segmjr": 1, 2024-04-25T11:47:17.621 INFO:teuthology.orchestra.run.smithi138.stdout: "segexg": 0, 2024-04-25T11:47:17.621 INFO:teuthology.orchestra.run.smithi138.stdout: "segexd": 1, 2024-04-25T11:47:17.622 INFO:teuthology.orchestra.run.smithi138.stdout: "expos": 646513167, 2024-04-25T11:47:17.622 INFO:teuthology.orchestra.run.smithi138.stdout: "wrpos": 646827462, 2024-04-25T11:47:17.622 INFO:teuthology.orchestra.run.smithi138.stdout: "rdpos": 0, 2024-04-25T11:47:17.622 INFO:teuthology.orchestra.run.smithi138.stdout: "jlat": { 2024-04-25T11:47:17.622 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 6021, 2024-04-25T11:47:17.622 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 3987.776070478, 2024-04-25T11:47:17.622 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.662311255 2024-04-25T11:47:17.622 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.622 INFO:teuthology.orchestra.run.smithi138.stdout: "replayed": 0 2024-04-25T11:47:17.622 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.622 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_mem": { 2024-04-25T11:47:17.622 INFO:teuthology.orchestra.run.smithi138.stdout: "ino": 98, 2024-04-25T11:47:17.622 INFO:teuthology.orchestra.run.smithi138.stdout: "ino+": 137, 2024-04-25T11:47:17.622 INFO:teuthology.orchestra.run.smithi138.stdout: "ino-": 39, 2024-04-25T11:47:17.622 INFO:teuthology.orchestra.run.smithi138.stdout: "dir": 22, 2024-04-25T11:47:17.623 INFO:teuthology.orchestra.run.smithi138.stdout: "dir+": 22, 2024-04-25T11:47:17.623 INFO:teuthology.orchestra.run.smithi138.stdout: "dir-": 0, 2024-04-25T11:47:17.623 INFO:teuthology.orchestra.run.smithi138.stdout: "dn": 124, 2024-04-25T11:47:17.623 INFO:teuthology.orchestra.run.smithi138.stdout: "dn+": 142, 2024-04-25T11:47:17.623 INFO:teuthology.orchestra.run.smithi138.stdout: "dn-": 18, 2024-04-25T11:47:17.623 INFO:teuthology.orchestra.run.smithi138.stdout: "cap": 3, 2024-04-25T11:47:17.623 INFO:teuthology.orchestra.run.smithi138.stdout: "cap+": 36, 2024-04-25T11:47:17.623 INFO:teuthology.orchestra.run.smithi138.stdout: "cap-": 33, 2024-04-25T11:47:17.623 INFO:teuthology.orchestra.run.smithi138.stdout: "rss": 144048, 2024-04-25T11:47:17.623 INFO:teuthology.orchestra.run.smithi138.stdout: "heap": 231824 2024-04-25T11:47:17.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.623 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_server": { 2024-04-25T11:47:17.623 INFO:teuthology.orchestra.run.smithi138.stdout: "dispatch_client_request": 312, 2024-04-25T11:47:17.623 INFO:teuthology.orchestra.run.smithi138.stdout: "dispatch_server_request": 30, 2024-04-25T11:47:17.623 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_client_request": 237, 2024-04-25T11:47:17.624 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_client_session": 34, 2024-04-25T11:47:17.624 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_peer_request": 68, 2024-04-25T11:47:17.624 INFO:teuthology.orchestra.run.smithi138.stdout: "req_create_latency": { 2024-04-25T11:47:17.624 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 22, 2024-04-25T11:47:17.624 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.029763441, 2024-04-25T11:47:17.624 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.001352883 2024-04-25T11:47:17.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.624 INFO:teuthology.orchestra.run.smithi138.stdout: "req_getattr_latency": { 2024-04-25T11:47:17.624 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 120, 2024-04-25T11:47:17.624 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.028055960, 2024-04-25T11:47:17.624 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000233799 2024-04-25T11:47:17.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.624 INFO:teuthology.orchestra.run.smithi138.stdout: "req_getfilelock_latency": { 2024-04-25T11:47:17.625 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:17.625 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:17.625 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:17.625 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.625 INFO:teuthology.orchestra.run.smithi138.stdout: "req_link_latency": { 2024-04-25T11:47:17.625 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:17.625 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:17.625 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:17.625 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.625 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lookup_latency": { 2024-04-25T11:47:17.625 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 18, 2024-04-25T11:47:17.625 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.005803931, 2024-04-25T11:47:17.625 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000322440 2024-04-25T11:47:17.625 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.625 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lookuphash_latency": { 2024-04-25T11:47:17.626 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:17.626 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:17.626 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:17.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.626 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lookupino_latency": { 2024-04-25T11:47:17.626 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 1, 2024-04-25T11:47:17.626 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000208488, 2024-04-25T11:47:17.626 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000208488 2024-04-25T11:47:17.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.626 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lookupname_latency": { 2024-04-25T11:47:17.626 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:17.626 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:17.626 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:17.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.627 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lookupparent_latency": { 2024-04-25T11:47:17.627 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:17.627 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:17.627 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:17.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.627 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lookupsnap_latency": { 2024-04-25T11:47:17.627 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:17.627 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:17.627 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:17.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.627 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lssnap_latency": { 2024-04-25T11:47:17.628 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 12, 2024-04-25T11:47:17.628 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.002443481, 2024-04-25T11:47:17.628 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000203623 2024-04-25T11:47:17.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.628 INFO:teuthology.orchestra.run.smithi138.stdout: "req_mkdir_latency": { 2024-04-25T11:47:17.628 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 10, 2024-04-25T11:47:17.628 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.016236411, 2024-04-25T11:47:17.628 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.001623641 2024-04-25T11:47:17.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.628 INFO:teuthology.orchestra.run.smithi138.stdout: "req_mknod_latency": { 2024-04-25T11:47:17.628 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:17.629 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:17.629 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:17.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.629 INFO:teuthology.orchestra.run.smithi138.stdout: "req_mksnap_latency": { 2024-04-25T11:47:17.629 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 12, 2024-04-25T11:47:17.629 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 6.426306309, 2024-04-25T11:47:17.629 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.535525525 2024-04-25T11:47:17.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.629 INFO:teuthology.orchestra.run.smithi138.stdout: "req_open_latency": { 2024-04-25T11:47:17.629 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 1, 2024-04-25T11:47:17.629 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000235357, 2024-04-25T11:47:17.629 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000235357 2024-04-25T11:47:17.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.629 INFO:teuthology.orchestra.run.smithi138.stdout: "req_readdir_latency": { 2024-04-25T11:47:17.630 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 4, 2024-04-25T11:47:17.630 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.013219827, 2024-04-25T11:47:17.630 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.003304956 2024-04-25T11:47:17.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.630 INFO:teuthology.orchestra.run.smithi138.stdout: "req_rename_latency": { 2024-04-25T11:47:17.630 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 2, 2024-04-25T11:47:17.630 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.002175792, 2024-04-25T11:47:17.630 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.001087896 2024-04-25T11:47:17.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.630 INFO:teuthology.orchestra.run.smithi138.stdout: "req_renamesnap_latency": { 2024-04-25T11:47:17.630 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:17.630 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:17.631 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:17.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.631 INFO:teuthology.orchestra.run.smithi138.stdout: "req_snapdiff_latency": { 2024-04-25T11:47:17.631 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:17.631 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:17.631 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:17.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.631 INFO:teuthology.orchestra.run.smithi138.stdout: "req_rmdir_latency": { 2024-04-25T11:47:17.631 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 4, 2024-04-25T11:47:17.631 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.606045450, 2024-04-25T11:47:17.631 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.151511362 2024-04-25T11:47:17.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.631 INFO:teuthology.orchestra.run.smithi138.stdout: "req_rmsnap_latency": { 2024-04-25T11:47:17.631 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 6, 2024-04-25T11:47:17.632 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.120813773, 2024-04-25T11:47:17.632 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.020135628 2024-04-25T11:47:17.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.632 INFO:teuthology.orchestra.run.smithi138.stdout: "req_rmxattr_latency": { 2024-04-25T11:47:17.632 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:17.632 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:17.632 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:17.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.632 INFO:teuthology.orchestra.run.smithi138.stdout: "req_setattr_latency": { 2024-04-25T11:47:17.632 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 1, 2024-04-25T11:47:17.632 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000973896, 2024-04-25T11:47:17.632 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000973896 2024-04-25T11:47:17.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.632 INFO:teuthology.orchestra.run.smithi138.stdout: "req_setdirlayout_latency": { 2024-04-25T11:47:17.633 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:17.633 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:17.633 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:17.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.633 INFO:teuthology.orchestra.run.smithi138.stdout: "req_setfilelock_latency": { 2024-04-25T11:47:17.633 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:17.633 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:17.633 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:17.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.633 INFO:teuthology.orchestra.run.smithi138.stdout: "req_setlayout_latency": { 2024-04-25T11:47:17.633 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:17.633 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:17.633 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:17.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.633 INFO:teuthology.orchestra.run.smithi138.stdout: "req_setxattr_latency": { 2024-04-25T11:47:17.634 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 7, 2024-04-25T11:47:17.634 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.021814338, 2024-04-25T11:47:17.634 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.003116334 2024-04-25T11:47:17.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.634 INFO:teuthology.orchestra.run.smithi138.stdout: "req_symlink_latency": { 2024-04-25T11:47:17.634 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:17.634 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:17.634 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:17.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.634 INFO:teuthology.orchestra.run.smithi138.stdout: "req_unlink_latency": { 2024-04-25T11:47:17.634 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 16, 2024-04-25T11:47:17.634 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.023987202, 2024-04-25T11:47:17.634 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.001499200 2024-04-25T11:47:17.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.635 INFO:teuthology.orchestra.run.smithi138.stdout: "cap_revoke_eviction": 0, 2024-04-25T11:47:17.635 INFO:teuthology.orchestra.run.smithi138.stdout: "cap_acquisition_throttle": 0, 2024-04-25T11:47:17.635 INFO:teuthology.orchestra.run.smithi138.stdout: "req_getvxattr_latency": { 2024-04-25T11:47:17.635 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:17.635 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:17.635 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:17.635 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:17.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.635 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_sessions": { 2024-04-25T11:47:17.635 INFO:teuthology.orchestra.run.smithi138.stdout: "session_count": 2, 2024-04-25T11:47:17.635 INFO:teuthology.orchestra.run.smithi138.stdout: "session_add": 5, 2024-04-25T11:47:17.635 INFO:teuthology.orchestra.run.smithi138.stdout: "session_remove": 3, 2024-04-25T11:47:17.635 INFO:teuthology.orchestra.run.smithi138.stdout: "sessions_open": 2, 2024-04-25T11:47:17.636 INFO:teuthology.orchestra.run.smithi138.stdout: "sessions_stale": 0, 2024-04-25T11:47:17.636 INFO:teuthology.orchestra.run.smithi138.stdout: "total_load": 62, 2024-04-25T11:47:17.636 INFO:teuthology.orchestra.run.smithi138.stdout: "average_load": 31, 2024-04-25T11:47:17.636 INFO:teuthology.orchestra.run.smithi138.stdout: "avg_session_uptime": 702, 2024-04-25T11:47:17.636 INFO:teuthology.orchestra.run.smithi138.stdout: "mdthresh_evicted": 0 2024-04-25T11:47:17.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.636 INFO:teuthology.orchestra.run.smithi138.stdout: "mempool": { 2024-04-25T11:47:17.636 INFO:teuthology.orchestra.run.smithi138.stdout: "bloom_filter_bytes": 0, 2024-04-25T11:47:17.636 INFO:teuthology.orchestra.run.smithi138.stdout: "bloom_filter_items": 0, 2024-04-25T11:47:17.636 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T11:47:17.636 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_alloc_items": 0, 2024-04-25T11:47:17.636 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T11:47:17.636 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_data_items": 0, 2024-04-25T11:47:17.637 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T11:47:17.637 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T11:47:17.637 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T11:47:17.637 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T11:47:17.637 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T11:47:17.637 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_other_items": 0, 2024-04-25T11:47:17.637 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T11:47:17.637 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T11:47:17.637 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_extent_bytes": 0, 2024-04-25T11:47:17.637 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_extent_items": 0, 2024-04-25T11:47:17.637 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_blob_bytes": 0, 2024-04-25T11:47:17.637 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_blob_items": 0, 2024-04-25T11:47:17.637 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T11:47:17.637 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T11:47:17.638 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T11:47:17.638 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T11:47:17.638 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T11:47:17.638 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_fsck_items": 0, 2024-04-25T11:47:17.638 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_txc_bytes": 0, 2024-04-25T11:47:17.638 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_txc_items": 0, 2024-04-25T11:47:17.638 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T11:47:17.638 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T11:47:17.638 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_writing_bytes": 0, 2024-04-25T11:47:17.638 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_writing_items": 0, 2024-04-25T11:47:17.638 INFO:teuthology.orchestra.run.smithi138.stdout: "bluefs_bytes": 0, 2024-04-25T11:47:17.638 INFO:teuthology.orchestra.run.smithi138.stdout: "bluefs_items": 0, 2024-04-25T11:47:17.639 INFO:teuthology.orchestra.run.smithi138.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T11:47:17.639 INFO:teuthology.orchestra.run.smithi138.stdout: "bluefs_file_reader_items": 0, 2024-04-25T11:47:17.639 INFO:teuthology.orchestra.run.smithi138.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T11:47:17.639 INFO:teuthology.orchestra.run.smithi138.stdout: "bluefs_file_writer_items": 0, 2024-04-25T11:47:17.639 INFO:teuthology.orchestra.run.smithi138.stdout: "buffer_anon_bytes": 394488, 2024-04-25T11:47:17.639 INFO:teuthology.orchestra.run.smithi138.stdout: "buffer_anon_items": 173, 2024-04-25T11:47:17.639 INFO:teuthology.orchestra.run.smithi138.stdout: "buffer_meta_bytes": 528, 2024-04-25T11:47:17.639 INFO:teuthology.orchestra.run.smithi138.stdout: "buffer_meta_items": 6, 2024-04-25T11:47:17.639 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_bytes": 0, 2024-04-25T11:47:17.639 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_items": 0, 2024-04-25T11:47:17.639 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_mapbl_bytes": 0, 2024-04-25T11:47:17.639 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_mapbl_items": 0, 2024-04-25T11:47:17.639 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_pglog_bytes": 0, 2024-04-25T11:47:17.639 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_pglog_items": 0, 2024-04-25T11:47:17.639 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap_bytes": 8632, 2024-04-25T11:47:17.640 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap_items": 180, 2024-04-25T11:47:17.640 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T11:47:17.640 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap_mapping_items": 0, 2024-04-25T11:47:17.640 INFO:teuthology.orchestra.run.smithi138.stdout: "pgmap_bytes": 0, 2024-04-25T11:47:17.640 INFO:teuthology.orchestra.run.smithi138.stdout: "pgmap_items": 0, 2024-04-25T11:47:17.640 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_co_bytes": 374607, 2024-04-25T11:47:17.640 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_co_items": 2831, 2024-04-25T11:47:17.640 INFO:teuthology.orchestra.run.smithi138.stdout: "unittest_1_bytes": 0, 2024-04-25T11:47:17.640 INFO:teuthology.orchestra.run.smithi138.stdout: "unittest_1_items": 0, 2024-04-25T11:47:17.640 INFO:teuthology.orchestra.run.smithi138.stdout: "unittest_2_bytes": 0, 2024-04-25T11:47:17.640 INFO:teuthology.orchestra.run.smithi138.stdout: "unittest_2_items": 0 2024-04-25T11:47:17.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.640 INFO:teuthology.orchestra.run.smithi138.stdout: "objecter": { 2024-04-25T11:47:17.640 INFO:teuthology.orchestra.run.smithi138.stdout: "op_active": 0, 2024-04-25T11:47:17.641 INFO:teuthology.orchestra.run.smithi138.stdout: "op_laggy": 0, 2024-04-25T11:47:17.641 INFO:teuthology.orchestra.run.smithi138.stdout: "op_send": 10655, 2024-04-25T11:47:17.641 INFO:teuthology.orchestra.run.smithi138.stdout: "op_send_bytes": 643122632, 2024-04-25T11:47:17.641 INFO:teuthology.orchestra.run.smithi138.stdout: "op_resend": 37, 2024-04-25T11:47:17.641 INFO:teuthology.orchestra.run.smithi138.stdout: "op_reply": 10655, 2024-04-25T11:47:17.641 INFO:teuthology.orchestra.run.smithi138.stdout: "op_latency": { 2024-04-25T11:47:17.641 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 10655, 2024-04-25T11:47:17.641 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 3684.744322052, 2024-04-25T11:47:17.641 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.345823024 2024-04-25T11:47:17.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.641 INFO:teuthology.orchestra.run.smithi138.stdout: "op_inflight": 0, 2024-04-25T11:47:17.641 INFO:teuthology.orchestra.run.smithi138.stdout: "oplen_avg": { 2024-04-25T11:47:17.642 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 10655, 2024-04-25T11:47:17.642 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 10953 2024-04-25T11:47:17.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.642 INFO:teuthology.orchestra.run.smithi138.stdout: "op": 10655, 2024-04-25T11:47:17.642 INFO:teuthology.orchestra.run.smithi138.stdout: "op_r": 638, 2024-04-25T11:47:17.642 INFO:teuthology.orchestra.run.smithi138.stdout: "op_w": 10017, 2024-04-25T11:47:17.642 INFO:teuthology.orchestra.run.smithi138.stdout: "op_rmw": 0, 2024-04-25T11:47:17.642 INFO:teuthology.orchestra.run.smithi138.stdout: "op_pg": 0, 2024-04-25T11:47:17.642 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_stat": 84, 2024-04-25T11:47:17.642 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_create": 44, 2024-04-25T11:47:17.642 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_read": 22, 2024-04-25T11:47:17.642 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_write": 6022, 2024-04-25T11:47:17.642 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_writefull": 164, 2024-04-25T11:47:17.642 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_writesame": 0, 2024-04-25T11:47:17.643 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_append": 0, 2024-04-25T11:47:17.643 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_zero": 0, 2024-04-25T11:47:17.643 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_truncate": 0, 2024-04-25T11:47:17.643 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_delete": 3394, 2024-04-25T11:47:17.643 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_mapext": 0, 2024-04-25T11:47:17.643 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_sparse_read": 0, 2024-04-25T11:47:17.643 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_clonerange": 0, 2024-04-25T11:47:17.643 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_getxattr": 616, 2024-04-25T11:47:17.643 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_setxattr": 88, 2024-04-25T11:47:17.643 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_cmpxattr": 0, 2024-04-25T11:47:17.643 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_rmxattr": 0, 2024-04-25T11:47:17.643 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_resetxattrs": 0, 2024-04-25T11:47:17.643 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_call": 0, 2024-04-25T11:47:17.643 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_watch": 0, 2024-04-25T11:47:17.643 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_notify": 0, 2024-04-25T11:47:17.644 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T11:47:17.644 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_pgls": 0, 2024-04-25T11:47:17.644 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_pgls_filter": 0, 2024-04-25T11:47:17.644 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_other": 0, 2024-04-25T11:47:17.644 INFO:teuthology.orchestra.run.smithi138.stdout: "linger_active": 0, 2024-04-25T11:47:17.644 INFO:teuthology.orchestra.run.smithi138.stdout: "linger_send": 0, 2024-04-25T11:47:17.644 INFO:teuthology.orchestra.run.smithi138.stdout: "linger_resend": 0, 2024-04-25T11:47:17.644 INFO:teuthology.orchestra.run.smithi138.stdout: "linger_ping": 0, 2024-04-25T11:47:17.644 INFO:teuthology.orchestra.run.smithi138.stdout: "poolop_active": 0, 2024-04-25T11:47:17.644 INFO:teuthology.orchestra.run.smithi138.stdout: "poolop_send": 0, 2024-04-25T11:47:17.644 INFO:teuthology.orchestra.run.smithi138.stdout: "poolop_resend": 0, 2024-04-25T11:47:17.644 INFO:teuthology.orchestra.run.smithi138.stdout: "poolstat_active": 0, 2024-04-25T11:47:17.644 INFO:teuthology.orchestra.run.smithi138.stdout: "poolstat_send": 0, 2024-04-25T11:47:17.644 INFO:teuthology.orchestra.run.smithi138.stdout: "poolstat_resend": 0, 2024-04-25T11:47:17.644 INFO:teuthology.orchestra.run.smithi138.stdout: "statfs_active": 0, 2024-04-25T11:47:17.644 INFO:teuthology.orchestra.run.smithi138.stdout: "statfs_send": 0, 2024-04-25T11:47:17.645 INFO:teuthology.orchestra.run.smithi138.stdout: "statfs_resend": 0, 2024-04-25T11:47:17.645 INFO:teuthology.orchestra.run.smithi138.stdout: "command_active": 0, 2024-04-25T11:47:17.645 INFO:teuthology.orchestra.run.smithi138.stdout: "command_send": 0, 2024-04-25T11:47:17.645 INFO:teuthology.orchestra.run.smithi138.stdout: "command_resend": 0, 2024-04-25T11:47:17.645 INFO:teuthology.orchestra.run.smithi138.stdout: "map_epoch": 93, 2024-04-25T11:47:17.645 INFO:teuthology.orchestra.run.smithi138.stdout: "map_full": 0, 2024-04-25T11:47:17.645 INFO:teuthology.orchestra.run.smithi138.stdout: "map_inc": 17, 2024-04-25T11:47:17.645 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_sessions": 12, 2024-04-25T11:47:17.645 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_session_open": 12, 2024-04-25T11:47:17.645 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_session_close": 0, 2024-04-25T11:47:17.645 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_laggy": 0, 2024-04-25T11:47:17.645 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_wr": 504, 2024-04-25T11:47:17.645 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_rd": 0, 2024-04-25T11:47:17.645 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_del": 15 2024-04-25T11:47:17.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.646 INFO:teuthology.orchestra.run.smithi138.stdout: "oft": { 2024-04-25T11:47:17.646 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_total_objs": 1, 2024-04-25T11:47:17.646 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_total_kv_pairs": 0, 2024-04-25T11:47:17.646 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_total_updates": 0, 2024-04-25T11:47:17.646 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_total_removes": 0 2024-04-25T11:47:17.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.646 INFO:teuthology.orchestra.run.smithi138.stdout: "osd-slow-ops": { 2024-04-25T11:47:17.646 INFO:teuthology.orchestra.run.smithi138.stdout: "slow_ops_count": 0 2024-04-25T11:47:17.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.646 INFO:teuthology.orchestra.run.smithi138.stdout: "purge_queue": { 2024-04-25T11:47:17.646 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_executing_ops": 0, 2024-04-25T11:47:17.646 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_executing_ops_high_water": 518, 2024-04-25T11:47:17.646 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_executing": 0, 2024-04-25T11:47:17.647 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_executing_high_water": 6, 2024-04-25T11:47:17.647 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_executed_ops": 2576, 2024-04-25T11:47:17.647 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_executed": 16, 2024-04-25T11:47:17.647 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_item_in_journal": 0 2024-04-25T11:47:17.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.647 INFO:teuthology.orchestra.run.smithi138.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T11:47:17.647 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 0, 2024-04-25T11:47:17.647 INFO:teuthology.orchestra.run.smithi138.stdout: "max": 104857600, 2024-04-25T11:47:17.647 INFO:teuthology.orchestra.run.smithi138.stdout: "get_started": 0, 2024-04-25T11:47:17.647 INFO:teuthology.orchestra.run.smithi138.stdout: "get": 208556, 2024-04-25T11:47:17.647 INFO:teuthology.orchestra.run.smithi138.stdout: "get_sum": 50343223, 2024-04-25T11:47:17.647 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:17.647 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_success": 208556, 2024-04-25T11:47:17.648 INFO:teuthology.orchestra.run.smithi138.stdout: "take": 0, 2024-04-25T11:47:17.648 INFO:teuthology.orchestra.run.smithi138.stdout: "take_sum": 0, 2024-04-25T11:47:17.648 INFO:teuthology.orchestra.run.smithi138.stdout: "put": 208556, 2024-04-25T11:47:17.648 INFO:teuthology.orchestra.run.smithi138.stdout: "put_sum": 50343223, 2024-04-25T11:47:17.648 INFO:teuthology.orchestra.run.smithi138.stdout: "wait": { 2024-04-25T11:47:17.648 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:17.648 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:17.648 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:17.648 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:17.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.648 INFO:teuthology.orchestra.run.smithi138.stdout: "throttle-objecter_bytes": { 2024-04-25T11:47:17.648 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 0, 2024-04-25T11:47:17.648 INFO:teuthology.orchestra.run.smithi138.stdout: "max": 104857600, 2024-04-25T11:47:17.648 INFO:teuthology.orchestra.run.smithi138.stdout: "get_started": 0, 2024-04-25T11:47:17.648 INFO:teuthology.orchestra.run.smithi138.stdout: "get": 0, 2024-04-25T11:47:17.649 INFO:teuthology.orchestra.run.smithi138.stdout: "get_sum": 0, 2024-04-25T11:47:17.649 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:17.649 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:17.649 INFO:teuthology.orchestra.run.smithi138.stdout: "take": 10655, 2024-04-25T11:47:17.649 INFO:teuthology.orchestra.run.smithi138.stdout: "take_sum": 643125016, 2024-04-25T11:47:17.649 INFO:teuthology.orchestra.run.smithi138.stdout: "put": 10655, 2024-04-25T11:47:17.649 INFO:teuthology.orchestra.run.smithi138.stdout: "put_sum": 643125016, 2024-04-25T11:47:17.649 INFO:teuthology.orchestra.run.smithi138.stdout: "wait": { 2024-04-25T11:47:17.649 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:17.649 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:17.649 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:17.649 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:17.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.650 INFO:teuthology.orchestra.run.smithi138.stdout: "throttle-objecter_ops": { 2024-04-25T11:47:17.650 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 0, 2024-04-25T11:47:17.650 INFO:teuthology.orchestra.run.smithi138.stdout: "max": 1024, 2024-04-25T11:47:17.650 INFO:teuthology.orchestra.run.smithi138.stdout: "get_started": 0, 2024-04-25T11:47:17.650 INFO:teuthology.orchestra.run.smithi138.stdout: "get": 0, 2024-04-25T11:47:17.650 INFO:teuthology.orchestra.run.smithi138.stdout: "get_sum": 0, 2024-04-25T11:47:17.650 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:17.650 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:17.650 INFO:teuthology.orchestra.run.smithi138.stdout: "take": 10655, 2024-04-25T11:47:17.650 INFO:teuthology.orchestra.run.smithi138.stdout: "take_sum": 10655, 2024-04-25T11:47:17.650 INFO:teuthology.orchestra.run.smithi138.stdout: "put": 10655, 2024-04-25T11:47:17.650 INFO:teuthology.orchestra.run.smithi138.stdout: "put_sum": 10655, 2024-04-25T11:47:17.651 INFO:teuthology.orchestra.run.smithi138.stdout: "wait": { 2024-04-25T11:47:17.651 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:17.651 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:17.651 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:17.651 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:17.651 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.651 INFO:teuthology.orchestra.run.smithi138.stdout: "throttle-write_buf_throttle": { 2024-04-25T11:47:17.651 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 0, 2024-04-25T11:47:17.651 INFO:teuthology.orchestra.run.smithi138.stdout: "max": 3758096384, 2024-04-25T11:47:17.651 INFO:teuthology.orchestra.run.smithi138.stdout: "get_started": 0, 2024-04-25T11:47:17.651 INFO:teuthology.orchestra.run.smithi138.stdout: "get": 16, 2024-04-25T11:47:17.651 INFO:teuthology.orchestra.run.smithi138.stdout: "get_sum": 2384, 2024-04-25T11:47:17.651 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:17.652 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_success": 16, 2024-04-25T11:47:17.652 INFO:teuthology.orchestra.run.smithi138.stdout: "take": 0, 2024-04-25T11:47:17.652 INFO:teuthology.orchestra.run.smithi138.stdout: "take_sum": 0, 2024-04-25T11:47:17.652 INFO:teuthology.orchestra.run.smithi138.stdout: "put": 1, 2024-04-25T11:47:17.652 INFO:teuthology.orchestra.run.smithi138.stdout: "put_sum": 2384, 2024-04-25T11:47:17.652 INFO:teuthology.orchestra.run.smithi138.stdout: "wait": { 2024-04-25T11:47:17.652 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:17.652 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:17.652 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:17.652 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:17.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:17.652 INFO:teuthology.orchestra.run.smithi138.stdout: "throttle-write_buf_throttle-0x560756af4750": { 2024-04-25T11:47:17.652 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 0, 2024-04-25T11:47:17.652 INFO:teuthology.orchestra.run.smithi138.stdout: "max": 3758096384, 2024-04-25T11:47:17.652 INFO:teuthology.orchestra.run.smithi138.stdout: "get_started": 0, 2024-04-25T11:47:17.653 INFO:teuthology.orchestra.run.smithi138.stdout: "get": 192169, 2024-04-25T11:47:17.653 INFO:teuthology.orchestra.run.smithi138.stdout: "get_sum": 642633158, 2024-04-25T11:47:17.653 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:17.653 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_success": 192169, 2024-04-25T11:47:17.653 INFO:teuthology.orchestra.run.smithi138.stdout: "take": 0, 2024-04-25T11:47:17.653 INFO:teuthology.orchestra.run.smithi138.stdout: "take_sum": 0, 2024-04-25T11:47:17.653 INFO:teuthology.orchestra.run.smithi138.stdout: "put": 6021, 2024-04-25T11:47:17.653 INFO:teuthology.orchestra.run.smithi138.stdout: "put_sum": 642633158, 2024-04-25T11:47:17.653 INFO:teuthology.orchestra.run.smithi138.stdout: "wait": { 2024-04-25T11:47:17.653 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:17.653 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:17.653 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:17.653 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:17.653 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:17.654 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-25T11:47:18.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:18 smithi138 ceph-mon[32328]: pgmap v380: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 18 GiB data, 59 GiB used, 1014 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-04-25T11:47:18.360 INFO:tasks.check_counter:Daemon mds.b mds.exported=1 2024-04-25T11:47:18.360 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-04-25T11:47:18.360 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=13 2024-04-25T11:47:18.360 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=0 2024-04-25T11:47:18.360 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-25T11:47:18.360 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=12 2024-04-25T11:47:18.361 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=6 2024-04-25T11:47:18.361 DEBUG:tasks.check_counter:Getting stats from e 2024-04-25T11:47:18.361 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-25T11:47:18.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:18 smithi090 ceph-mon[29575]: pgmap v380: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 18 GiB data, 59 GiB used, 1014 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-04-25T11:47:18.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:18 smithi156 ceph-mon[33045]: pgmap v380: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 18 GiB data, 59 GiB used, 1014 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-04-25T11:47:18.627 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T11:47:20.400 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:20 smithi138 ceph-mon[32328]: pgmap v381: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 18 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-04-25T11:47:20.446 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-25T11:47:20.446 INFO:teuthology.orchestra.run.smithi138.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T11:47:20.446 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_messages": 1413, 2024-04-25T11:47:20.446 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_messages": 1027, 2024-04-25T11:47:20.446 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_bytes": 442553, 2024-04-25T11:47:20.446 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_bytes": 288096, 2024-04-25T11:47:20.446 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_created_connections": 10, 2024-04-25T11:47:20.447 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_active_connections": 3, 2024-04-25T11:47:20.447 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_total_time": 0.123866957, 2024-04-25T11:47:20.447 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_send_time": 0.055006072, 2024-04-25T11:47:20.447 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_recv_time": 0.075337962, 2024-04-25T11:47:20.447 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_fast_dispatch_time": 0.004834638, 2024-04-25T11:47:20.447 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:20.447 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 1029, 2024-04-25T11:47:20.447 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.068495415, 2024-04-25T11:47:20.447 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000066565 2024-04-25T11:47:20.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.447 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:20.447 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 2250, 2024-04-25T11:47:20.447 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000545892, 2024-04-25T11:47:20.447 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000242 2024-04-25T11:47:20.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.448 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T11:47:20.448 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T11:47:20.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.448 INFO:teuthology.orchestra.run.smithi138.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T11:47:20.448 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_messages": 124, 2024-04-25T11:47:20.448 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_messages": 303, 2024-04-25T11:47:20.448 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_bytes": 55732, 2024-04-25T11:47:20.448 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_bytes": 984295, 2024-04-25T11:47:20.448 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_created_connections": 49, 2024-04-25T11:47:20.448 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_active_connections": 5, 2024-04-25T11:47:20.448 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_total_time": 0.036241174, 2024-04-25T11:47:20.448 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_send_time": 0.010202212, 2024-04-25T11:47:20.449 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_recv_time": 0.023061156, 2024-04-25T11:47:20.449 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_fast_dispatch_time": 0.000231545, 2024-04-25T11:47:20.449 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:20.449 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 300, 2024-04-25T11:47:20.449 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.045386232, 2024-04-25T11:47:20.449 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000151287 2024-04-25T11:47:20.449 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.449 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:20.449 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 167, 2024-04-25T11:47:20.449 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000036936, 2024-04-25T11:47:20.449 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000221 2024-04-25T11:47:20.449 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.449 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_encrypted_bytes": 2160, 2024-04-25T11:47:20.450 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_encrypted_bytes": 481520 2024-04-25T11:47:20.450 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.450 INFO:teuthology.orchestra.run.smithi138.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T11:47:20.450 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_messages": 391, 2024-04-25T11:47:20.450 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_messages": 376, 2024-04-25T11:47:20.450 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_bytes": 1673286, 2024-04-25T11:47:20.450 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_bytes": 303829, 2024-04-25T11:47:20.450 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_created_connections": 17, 2024-04-25T11:47:20.450 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_active_connections": 6, 2024-04-25T11:47:20.450 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_total_time": 0.065238723, 2024-04-25T11:47:20.450 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_send_time": 0.017848249, 2024-04-25T11:47:20.450 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_recv_time": 0.052347655, 2024-04-25T11:47:20.450 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_fast_dispatch_time": 0.006834467, 2024-04-25T11:47:20.451 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:20.451 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 372, 2024-04-25T11:47:20.451 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.028849242, 2024-04-25T11:47:20.451 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000077551 2024-04-25T11:47:20.451 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.451 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:20.451 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 132, 2024-04-25T11:47:20.451 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000048376, 2024-04-25T11:47:20.451 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000366 2024-04-25T11:47:20.451 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.451 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_encrypted_bytes": 1626288, 2024-04-25T11:47:20.451 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_encrypted_bytes": 154576 2024-04-25T11:47:20.451 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.452 INFO:teuthology.orchestra.run.smithi138.stdout: "cct": { 2024-04-25T11:47:20.452 INFO:teuthology.orchestra.run.smithi138.stdout: "total_workers": 1, 2024-04-25T11:47:20.452 INFO:teuthology.orchestra.run.smithi138.stdout: "unhealthy_workers": 0 2024-04-25T11:47:20.452 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.452 INFO:teuthology.orchestra.run.smithi138.stdout: "finisher-MDSRank": { 2024-04-25T11:47:20.452 INFO:teuthology.orchestra.run.smithi138.stdout: "queue_len": 0, 2024-04-25T11:47:20.452 INFO:teuthology.orchestra.run.smithi138.stdout: "complete_latency": { 2024-04-25T11:47:20.452 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 51, 2024-04-25T11:47:20.452 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.001642363, 2024-04-25T11:47:20.452 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000032203 2024-04-25T11:47:20.452 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:20.452 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.452 INFO:teuthology.orchestra.run.smithi138.stdout: "finisher-PurgeQueue": { 2024-04-25T11:47:20.453 INFO:teuthology.orchestra.run.smithi138.stdout: "queue_len": 0, 2024-04-25T11:47:20.453 INFO:teuthology.orchestra.run.smithi138.stdout: "complete_latency": { 2024-04-25T11:47:20.453 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 2, 2024-04-25T11:47:20.453 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000007106, 2024-04-25T11:47:20.453 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000003553 2024-04-25T11:47:20.453 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:20.453 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.453 INFO:teuthology.orchestra.run.smithi138.stdout: "mds": { 2024-04-25T11:47:20.453 INFO:teuthology.orchestra.run.smithi138.stdout: "request": 3, 2024-04-25T11:47:20.453 INFO:teuthology.orchestra.run.smithi138.stdout: "reply": 2, 2024-04-25T11:47:20.453 INFO:teuthology.orchestra.run.smithi138.stdout: "reply_latency": { 2024-04-25T11:47:20.453 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 2, 2024-04-25T11:47:20.453 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.001539030, 2024-04-25T11:47:20.454 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000769515 2024-04-25T11:47:20.454 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.454 INFO:teuthology.orchestra.run.smithi138.stdout: "slow_reply": 0, 2024-04-25T11:47:20.454 INFO:teuthology.orchestra.run.smithi138.stdout: "forward": 1, 2024-04-25T11:47:20.454 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_fetch_complete": 0, 2024-04-25T11:47:20.454 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_fetch_keys": 0, 2024-04-25T11:47:20.454 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_commit": 11, 2024-04-25T11:47:20.454 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_split": 0, 2024-04-25T11:47:20.454 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_merge": 0, 2024-04-25T11:47:20.454 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes": 16, 2024-04-25T11:47:20.454 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes_top": 0, 2024-04-25T11:47:20.454 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes_bottom": 16, 2024-04-25T11:47:20.454 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes_pin_tail": 0, 2024-04-25T11:47:20.455 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes_pinned": 13, 2024-04-25T11:47:20.455 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes_expired": 8, 2024-04-25T11:47:20.455 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes_with_caps": 0, 2024-04-25T11:47:20.455 INFO:teuthology.orchestra.run.smithi138.stdout: "caps": 0, 2024-04-25T11:47:20.455 INFO:teuthology.orchestra.run.smithi138.stdout: "subtrees": 3, 2024-04-25T11:47:20.455 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse": 14, 2024-04-25T11:47:20.455 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse_hit": 9, 2024-04-25T11:47:20.455 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse_forward": 1, 2024-04-25T11:47:20.455 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse_discover": 4, 2024-04-25T11:47:20.455 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse_dir_fetch": 0, 2024-04-25T11:47:20.455 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse_remote_ino": 0, 2024-04-25T11:47:20.455 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse_lock": 0, 2024-04-25T11:47:20.455 INFO:teuthology.orchestra.run.smithi138.stdout: "load_cent": 0, 2024-04-25T11:47:20.456 INFO:teuthology.orchestra.run.smithi138.stdout: "q": 0, 2024-04-25T11:47:20.456 INFO:teuthology.orchestra.run.smithi138.stdout: "exported": 0, 2024-04-25T11:47:20.456 INFO:teuthology.orchestra.run.smithi138.stdout: "exported_inodes": 0, 2024-04-25T11:47:20.456 INFO:teuthology.orchestra.run.smithi138.stdout: "imported": 0, 2024-04-25T11:47:20.456 INFO:teuthology.orchestra.run.smithi138.stdout: "imported_inodes": 0, 2024-04-25T11:47:20.456 INFO:teuthology.orchestra.run.smithi138.stdout: "openino_dir_fetch": 0, 2024-04-25T11:47:20.456 INFO:teuthology.orchestra.run.smithi138.stdout: "openino_backtrace_fetch": 0, 2024-04-25T11:47:20.456 INFO:teuthology.orchestra.run.smithi138.stdout: "openino_peer_discover": 0, 2024-04-25T11:47:20.456 INFO:teuthology.orchestra.run.smithi138.stdout: "root_rfiles": 22, 2024-04-25T11:47:20.456 INFO:teuthology.orchestra.run.smithi138.stdout: "root_rbytes": 10738142561, 2024-04-25T11:47:20.456 INFO:teuthology.orchestra.run.smithi138.stdout: "root_rsnaps": 6, 2024-04-25T11:47:20.456 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T11:47:20.456 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_set_tag": 0, 2024-04-25T11:47:20.456 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T11:47:20.457 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_inotable_repaired": 0, 2024-04-25T11:47:20.457 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_dir_inodes": 0, 2024-04-25T11:47:20.457 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T11:47:20.457 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T11:47:20.457 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_file_inodes": 0, 2024-04-25T11:47:20.457 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_inode_file_caps": 0, 2024-04-25T11:47:20.457 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T11:47:20.457 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_cap_op_grant": 0, 2024-04-25T11:47:20.457 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T11:47:20.457 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T11:47:20.457 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T11:47:20.457 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_client_caps": 0, 2024-04-25T11:47:20.457 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_client_caps_dirty": 0, 2024-04-25T11:47:20.458 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_client_cap_release": 0, 2024-04-25T11:47:20.458 INFO:teuthology.orchestra.run.smithi138.stdout: "process_request_cap_release": 0 2024-04-25T11:47:20.458 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.458 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_cache": { 2024-04-25T11:47:20.458 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_update": 0, 2024-04-25T11:47:20.458 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_update_receipt": 12, 2024-04-25T11:47:20.458 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_try_discover": 8, 2024-04-25T11:47:20.458 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_send_discover": 10, 2024-04-25T11:47:20.458 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_handle_discover": 0, 2024-04-25T11:47:20.458 INFO:teuthology.orchestra.run.smithi138.stdout: "num_strays": 0, 2024-04-25T11:47:20.458 INFO:teuthology.orchestra.run.smithi138.stdout: "num_strays_delayed": 0, 2024-04-25T11:47:20.458 INFO:teuthology.orchestra.run.smithi138.stdout: "num_strays_enqueuing": 0, 2024-04-25T11:47:20.458 INFO:teuthology.orchestra.run.smithi138.stdout: "strays_created": 0, 2024-04-25T11:47:20.458 INFO:teuthology.orchestra.run.smithi138.stdout: "strays_enqueued": 0, 2024-04-25T11:47:20.459 INFO:teuthology.orchestra.run.smithi138.stdout: "strays_reintegrated": 0, 2024-04-25T11:47:20.459 INFO:teuthology.orchestra.run.smithi138.stdout: "strays_migrated": 0, 2024-04-25T11:47:20.459 INFO:teuthology.orchestra.run.smithi138.stdout: "num_recovering_processing": 0, 2024-04-25T11:47:20.459 INFO:teuthology.orchestra.run.smithi138.stdout: "num_recovering_enqueued": 0, 2024-04-25T11:47:20.459 INFO:teuthology.orchestra.run.smithi138.stdout: "num_recovering_prioritized": 0, 2024-04-25T11:47:20.459 INFO:teuthology.orchestra.run.smithi138.stdout: "recovery_started": 0, 2024-04-25T11:47:20.459 INFO:teuthology.orchestra.run.smithi138.stdout: "recovery_completed": 0, 2024-04-25T11:47:20.459 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_quiesce_path": 0, 2024-04-25T11:47:20.459 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_quiesce_inode": 0, 2024-04-25T11:47:20.459 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T11:47:20.459 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_exportdir": 0, 2024-04-25T11:47:20.459 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_flush": 0, 2024-04-25T11:47:20.459 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_fragmentdir": 0, 2024-04-25T11:47:20.459 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_fragstats": 0, 2024-04-25T11:47:20.460 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_inodestats": 0 2024-04-25T11:47:20.460 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.460 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_log": { 2024-04-25T11:47:20.460 INFO:teuthology.orchestra.run.smithi138.stdout: "evlrg": 0, 2024-04-25T11:47:20.460 INFO:teuthology.orchestra.run.smithi138.stdout: "evadd": 6, 2024-04-25T11:47:20.460 INFO:teuthology.orchestra.run.smithi138.stdout: "evex": 0, 2024-04-25T11:47:20.460 INFO:teuthology.orchestra.run.smithi138.stdout: "evtrm": 0, 2024-04-25T11:47:20.460 INFO:teuthology.orchestra.run.smithi138.stdout: "ev": 6, 2024-04-25T11:47:20.460 INFO:teuthology.orchestra.run.smithi138.stdout: "evexg": 0, 2024-04-25T11:47:20.460 INFO:teuthology.orchestra.run.smithi138.stdout: "evexd": 0, 2024-04-25T11:47:20.460 INFO:teuthology.orchestra.run.smithi138.stdout: "segadd": 2, 2024-04-25T11:47:20.460 INFO:teuthology.orchestra.run.smithi138.stdout: "segex": 0, 2024-04-25T11:47:20.460 INFO:teuthology.orchestra.run.smithi138.stdout: "segtrm": 0, 2024-04-25T11:47:20.460 INFO:teuthology.orchestra.run.smithi138.stdout: "seg": 2, 2024-04-25T11:47:20.460 INFO:teuthology.orchestra.run.smithi138.stdout: "segmjr": 2, 2024-04-25T11:47:20.461 INFO:teuthology.orchestra.run.smithi138.stdout: "segexg": 0, 2024-04-25T11:47:20.461 INFO:teuthology.orchestra.run.smithi138.stdout: "segexd": 0, 2024-04-25T11:47:20.461 INFO:teuthology.orchestra.run.smithi138.stdout: "expos": 4194304, 2024-04-25T11:47:20.461 INFO:teuthology.orchestra.run.smithi138.stdout: "wrpos": 4195956, 2024-04-25T11:47:20.461 INFO:teuthology.orchestra.run.smithi138.stdout: "rdpos": 0, 2024-04-25T11:47:20.461 INFO:teuthology.orchestra.run.smithi138.stdout: "jlat": { 2024-04-25T11:47:20.461 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 5, 2024-04-25T11:47:20.461 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.010029190, 2024-04-25T11:47:20.461 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.002005838 2024-04-25T11:47:20.461 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.461 INFO:teuthology.orchestra.run.smithi138.stdout: "replayed": 0 2024-04-25T11:47:20.461 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.461 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_mem": { 2024-04-25T11:47:20.461 INFO:teuthology.orchestra.run.smithi138.stdout: "ino": 20, 2024-04-25T11:47:20.462 INFO:teuthology.orchestra.run.smithi138.stdout: "ino+": 24, 2024-04-25T11:47:20.462 INFO:teuthology.orchestra.run.smithi138.stdout: "ino-": 4, 2024-04-25T11:47:20.462 INFO:teuthology.orchestra.run.smithi138.stdout: "dir": 19, 2024-04-25T11:47:20.462 INFO:teuthology.orchestra.run.smithi138.stdout: "dir+": 23, 2024-04-25T11:47:20.462 INFO:teuthology.orchestra.run.smithi138.stdout: "dir-": 4, 2024-04-25T11:47:20.462 INFO:teuthology.orchestra.run.smithi138.stdout: "dn": 16, 2024-04-25T11:47:20.462 INFO:teuthology.orchestra.run.smithi138.stdout: "dn+": 24, 2024-04-25T11:47:20.462 INFO:teuthology.orchestra.run.smithi138.stdout: "dn-": 8, 2024-04-25T11:47:20.462 INFO:teuthology.orchestra.run.smithi138.stdout: "cap": 0, 2024-04-25T11:47:20.462 INFO:teuthology.orchestra.run.smithi138.stdout: "cap+": 2, 2024-04-25T11:47:20.462 INFO:teuthology.orchestra.run.smithi138.stdout: "cap-": 2, 2024-04-25T11:47:20.462 INFO:teuthology.orchestra.run.smithi138.stdout: "rss": 121608, 2024-04-25T11:47:20.462 INFO:teuthology.orchestra.run.smithi138.stdout: "heap": 223632 2024-04-25T11:47:20.462 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.463 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_server": { 2024-04-25T11:47:20.463 INFO:teuthology.orchestra.run.smithi138.stdout: "dispatch_client_request": 6, 2024-04-25T11:47:20.463 INFO:teuthology.orchestra.run.smithi138.stdout: "dispatch_server_request": 1, 2024-04-25T11:47:20.463 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_client_request": 3, 2024-04-25T11:47:20.463 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_client_session": 8, 2024-04-25T11:47:20.463 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_peer_request": 3, 2024-04-25T11:47:20.463 INFO:teuthology.orchestra.run.smithi138.stdout: "req_create_latency": { 2024-04-25T11:47:20.463 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.463 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.463 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.463 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.463 INFO:teuthology.orchestra.run.smithi138.stdout: "req_getattr_latency": { 2024-04-25T11:47:20.463 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 1, 2024-04-25T11:47:20.464 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.001384462, 2024-04-25T11:47:20.464 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.001384462 2024-04-25T11:47:20.464 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.464 INFO:teuthology.orchestra.run.smithi138.stdout: "req_getfilelock_latency": { 2024-04-25T11:47:20.464 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.464 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.464 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.464 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.464 INFO:teuthology.orchestra.run.smithi138.stdout: "req_link_latency": { 2024-04-25T11:47:20.464 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.464 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.464 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.464 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.465 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lookup_latency": { 2024-04-25T11:47:20.465 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 1, 2024-04-25T11:47:20.465 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000154568, 2024-04-25T11:47:20.465 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000154568 2024-04-25T11:47:20.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.465 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lookuphash_latency": { 2024-04-25T11:47:20.465 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.465 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.465 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.465 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lookupino_latency": { 2024-04-25T11:47:20.465 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.465 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.465 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.466 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lookupname_latency": { 2024-04-25T11:47:20.466 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.466 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.466 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.466 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.466 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lookupparent_latency": { 2024-04-25T11:47:20.466 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.466 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.466 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.466 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.466 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lookupsnap_latency": { 2024-04-25T11:47:20.466 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.466 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.466 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.467 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.467 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lssnap_latency": { 2024-04-25T11:47:20.467 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.467 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.467 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.467 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.467 INFO:teuthology.orchestra.run.smithi138.stdout: "req_mkdir_latency": { 2024-04-25T11:47:20.467 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.468 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.468 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.468 INFO:teuthology.orchestra.run.smithi138.stdout: "req_mknod_latency": { 2024-04-25T11:47:20.468 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.468 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.468 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.468 INFO:teuthology.orchestra.run.smithi138.stdout: "req_mksnap_latency": { 2024-04-25T11:47:20.468 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.468 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.468 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.468 INFO:teuthology.orchestra.run.smithi138.stdout: "req_open_latency": { 2024-04-25T11:47:20.468 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.469 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.469 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.469 INFO:teuthology.orchestra.run.smithi138.stdout: "req_readdir_latency": { 2024-04-25T11:47:20.469 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.469 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.469 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.469 INFO:teuthology.orchestra.run.smithi138.stdout: "req_rename_latency": { 2024-04-25T11:47:20.469 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.469 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.469 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.469 INFO:teuthology.orchestra.run.smithi138.stdout: "req_renamesnap_latency": { 2024-04-25T11:47:20.469 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.470 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.470 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.470 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.470 INFO:teuthology.orchestra.run.smithi138.stdout: "req_snapdiff_latency": { 2024-04-25T11:47:20.470 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.470 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.470 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.470 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.470 INFO:teuthology.orchestra.run.smithi138.stdout: "req_rmdir_latency": { 2024-04-25T11:47:20.470 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.470 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.470 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.470 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.471 INFO:teuthology.orchestra.run.smithi138.stdout: "req_rmsnap_latency": { 2024-04-25T11:47:20.471 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.471 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.471 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.471 INFO:teuthology.orchestra.run.smithi138.stdout: "req_rmxattr_latency": { 2024-04-25T11:47:20.471 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.471 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.471 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.471 INFO:teuthology.orchestra.run.smithi138.stdout: "req_setattr_latency": { 2024-04-25T11:47:20.471 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.471 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.471 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.472 INFO:teuthology.orchestra.run.smithi138.stdout: "req_setdirlayout_latency": { 2024-04-25T11:47:20.472 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.472 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.472 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.472 INFO:teuthology.orchestra.run.smithi138.stdout: "req_setfilelock_latency": { 2024-04-25T11:47:20.472 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.472 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.472 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.472 INFO:teuthology.orchestra.run.smithi138.stdout: "req_setlayout_latency": { 2024-04-25T11:47:20.472 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.472 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.473 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.473 INFO:teuthology.orchestra.run.smithi138.stdout: "req_setxattr_latency": { 2024-04-25T11:47:20.473 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.473 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.473 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.473 INFO:teuthology.orchestra.run.smithi138.stdout: "req_symlink_latency": { 2024-04-25T11:47:20.473 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.473 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.473 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.473 INFO:teuthology.orchestra.run.smithi138.stdout: "req_unlink_latency": { 2024-04-25T11:47:20.473 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.473 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.474 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.474 INFO:teuthology.orchestra.run.smithi138.stdout: "cap_revoke_eviction": 0, 2024-04-25T11:47:20.474 INFO:teuthology.orchestra.run.smithi138.stdout: "cap_acquisition_throttle": 0, 2024-04-25T11:47:20.474 INFO:teuthology.orchestra.run.smithi138.stdout: "req_getvxattr_latency": { 2024-04-25T11:47:20.474 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.474 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.474 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.474 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:20.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.474 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_sessions": { 2024-04-25T11:47:20.474 INFO:teuthology.orchestra.run.smithi138.stdout: "session_count": 0, 2024-04-25T11:47:20.474 INFO:teuthology.orchestra.run.smithi138.stdout: "session_add": 2, 2024-04-25T11:47:20.475 INFO:teuthology.orchestra.run.smithi138.stdout: "session_remove": 2, 2024-04-25T11:47:20.475 INFO:teuthology.orchestra.run.smithi138.stdout: "sessions_open": 0, 2024-04-25T11:47:20.475 INFO:teuthology.orchestra.run.smithi138.stdout: "sessions_stale": 0, 2024-04-25T11:47:20.475 INFO:teuthology.orchestra.run.smithi138.stdout: "total_load": 1, 2024-04-25T11:47:20.475 INFO:teuthology.orchestra.run.smithi138.stdout: "average_load": 0, 2024-04-25T11:47:20.475 INFO:teuthology.orchestra.run.smithi138.stdout: "avg_session_uptime": 87, 2024-04-25T11:47:20.475 INFO:teuthology.orchestra.run.smithi138.stdout: "mdthresh_evicted": 0 2024-04-25T11:47:20.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.475 INFO:teuthology.orchestra.run.smithi138.stdout: "mempool": { 2024-04-25T11:47:20.475 INFO:teuthology.orchestra.run.smithi138.stdout: "bloom_filter_bytes": 0, 2024-04-25T11:47:20.475 INFO:teuthology.orchestra.run.smithi138.stdout: "bloom_filter_items": 0, 2024-04-25T11:47:20.475 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T11:47:20.475 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_alloc_items": 0, 2024-04-25T11:47:20.475 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T11:47:20.475 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_data_items": 0, 2024-04-25T11:47:20.476 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T11:47:20.476 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T11:47:20.476 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T11:47:20.476 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T11:47:20.476 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T11:47:20.476 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_other_items": 0, 2024-04-25T11:47:20.476 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T11:47:20.476 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T11:47:20.476 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_extent_bytes": 0, 2024-04-25T11:47:20.476 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_extent_items": 0, 2024-04-25T11:47:20.476 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_blob_bytes": 0, 2024-04-25T11:47:20.476 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_blob_items": 0, 2024-04-25T11:47:20.476 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T11:47:20.476 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T11:47:20.476 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T11:47:20.477 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T11:47:20.477 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T11:47:20.477 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_fsck_items": 0, 2024-04-25T11:47:20.477 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_txc_bytes": 0, 2024-04-25T11:47:20.477 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_txc_items": 0, 2024-04-25T11:47:20.477 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T11:47:20.477 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T11:47:20.477 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_writing_bytes": 0, 2024-04-25T11:47:20.477 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_writing_items": 0, 2024-04-25T11:47:20.477 INFO:teuthology.orchestra.run.smithi138.stdout: "bluefs_bytes": 0, 2024-04-25T11:47:20.477 INFO:teuthology.orchestra.run.smithi138.stdout: "bluefs_items": 0, 2024-04-25T11:47:20.477 INFO:teuthology.orchestra.run.smithi138.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T11:47:20.477 INFO:teuthology.orchestra.run.smithi138.stdout: "bluefs_file_reader_items": 0, 2024-04-25T11:47:20.477 INFO:teuthology.orchestra.run.smithi138.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T11:47:20.478 INFO:teuthology.orchestra.run.smithi138.stdout: "bluefs_file_writer_items": 0, 2024-04-25T11:47:20.478 INFO:teuthology.orchestra.run.smithi138.stdout: "buffer_anon_bytes": 254631, 2024-04-25T11:47:20.478 INFO:teuthology.orchestra.run.smithi138.stdout: "buffer_anon_items": 79, 2024-04-25T11:47:20.478 INFO:teuthology.orchestra.run.smithi138.stdout: "buffer_meta_bytes": 0, 2024-04-25T11:47:20.478 INFO:teuthology.orchestra.run.smithi138.stdout: "buffer_meta_items": 0, 2024-04-25T11:47:20.478 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_bytes": 0, 2024-04-25T11:47:20.478 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_items": 0, 2024-04-25T11:47:20.478 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_mapbl_bytes": 0, 2024-04-25T11:47:20.478 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_mapbl_items": 0, 2024-04-25T11:47:20.478 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_pglog_bytes": 0, 2024-04-25T11:47:20.478 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_pglog_items": 0, 2024-04-25T11:47:20.478 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap_bytes": 8632, 2024-04-25T11:47:20.478 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap_items": 180, 2024-04-25T11:47:20.478 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T11:47:20.479 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap_mapping_items": 0, 2024-04-25T11:47:20.479 INFO:teuthology.orchestra.run.smithi138.stdout: "pgmap_bytes": 0, 2024-04-25T11:47:20.479 INFO:teuthology.orchestra.run.smithi138.stdout: "pgmap_items": 0, 2024-04-25T11:47:20.479 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_co_bytes": 82645, 2024-04-25T11:47:20.479 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_co_items": 331, 2024-04-25T11:47:20.479 INFO:teuthology.orchestra.run.smithi138.stdout: "unittest_1_bytes": 0, 2024-04-25T11:47:20.479 INFO:teuthology.orchestra.run.smithi138.stdout: "unittest_1_items": 0, 2024-04-25T11:47:20.479 INFO:teuthology.orchestra.run.smithi138.stdout: "unittest_2_bytes": 0, 2024-04-25T11:47:20.479 INFO:teuthology.orchestra.run.smithi138.stdout: "unittest_2_items": 0 2024-04-25T11:47:20.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.479 INFO:teuthology.orchestra.run.smithi138.stdout: "objecter": { 2024-04-25T11:47:20.479 INFO:teuthology.orchestra.run.smithi138.stdout: "op_active": 0, 2024-04-25T11:47:20.479 INFO:teuthology.orchestra.run.smithi138.stdout: "op_laggy": 0, 2024-04-25T11:47:20.479 INFO:teuthology.orchestra.run.smithi138.stdout: "op_send": 43, 2024-04-25T11:47:20.480 INFO:teuthology.orchestra.run.smithi138.stdout: "op_send_bytes": 12099, 2024-04-25T11:47:20.480 INFO:teuthology.orchestra.run.smithi138.stdout: "op_resend": 33, 2024-04-25T11:47:20.480 INFO:teuthology.orchestra.run.smithi138.stdout: "op_reply": 43, 2024-04-25T11:47:20.480 INFO:teuthology.orchestra.run.smithi138.stdout: "op_latency": { 2024-04-25T11:47:20.480 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 43, 2024-04-25T11:47:20.480 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.210995308, 2024-04-25T11:47:20.480 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.004906867 2024-04-25T11:47:20.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.480 INFO:teuthology.orchestra.run.smithi138.stdout: "op_inflight": 0, 2024-04-25T11:47:20.480 INFO:teuthology.orchestra.run.smithi138.stdout: "oplen_avg": { 2024-04-25T11:47:20.480 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 43, 2024-04-25T11:47:20.480 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 67 2024-04-25T11:47:20.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.480 INFO:teuthology.orchestra.run.smithi138.stdout: "op": 43, 2024-04-25T11:47:20.480 INFO:teuthology.orchestra.run.smithi138.stdout: "op_r": 0, 2024-04-25T11:47:20.481 INFO:teuthology.orchestra.run.smithi138.stdout: "op_w": 43, 2024-04-25T11:47:20.481 INFO:teuthology.orchestra.run.smithi138.stdout: "op_rmw": 0, 2024-04-25T11:47:20.481 INFO:teuthology.orchestra.run.smithi138.stdout: "op_pg": 0, 2024-04-25T11:47:20.481 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_stat": 0, 2024-04-25T11:47:20.481 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_create": 10, 2024-04-25T11:47:20.481 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_read": 0, 2024-04-25T11:47:20.481 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_write": 5, 2024-04-25T11:47:20.481 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_writefull": 7, 2024-04-25T11:47:20.481 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_writesame": 0, 2024-04-25T11:47:20.481 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_append": 0, 2024-04-25T11:47:20.481 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_zero": 0, 2024-04-25T11:47:20.481 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_truncate": 0, 2024-04-25T11:47:20.481 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_delete": 6, 2024-04-25T11:47:20.481 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_mapext": 0, 2024-04-25T11:47:20.481 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_sparse_read": 0, 2024-04-25T11:47:20.482 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_clonerange": 0, 2024-04-25T11:47:20.482 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_getxattr": 0, 2024-04-25T11:47:20.482 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_setxattr": 20, 2024-04-25T11:47:20.482 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_cmpxattr": 0, 2024-04-25T11:47:20.482 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_rmxattr": 0, 2024-04-25T11:47:20.482 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_resetxattrs": 0, 2024-04-25T11:47:20.482 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_call": 0, 2024-04-25T11:47:20.482 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_watch": 0, 2024-04-25T11:47:20.482 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_notify": 0, 2024-04-25T11:47:20.482 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T11:47:20.482 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_pgls": 0, 2024-04-25T11:47:20.482 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_pgls_filter": 0, 2024-04-25T11:47:20.482 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_other": 0, 2024-04-25T11:47:20.482 INFO:teuthology.orchestra.run.smithi138.stdout: "linger_active": 0, 2024-04-25T11:47:20.483 INFO:teuthology.orchestra.run.smithi138.stdout: "linger_send": 0, 2024-04-25T11:47:20.483 INFO:teuthology.orchestra.run.smithi138.stdout: "linger_resend": 0, 2024-04-25T11:47:20.483 INFO:teuthology.orchestra.run.smithi138.stdout: "linger_ping": 0, 2024-04-25T11:47:20.483 INFO:teuthology.orchestra.run.smithi138.stdout: "poolop_active": 0, 2024-04-25T11:47:20.483 INFO:teuthology.orchestra.run.smithi138.stdout: "poolop_send": 0, 2024-04-25T11:47:20.483 INFO:teuthology.orchestra.run.smithi138.stdout: "poolop_resend": 0, 2024-04-25T11:47:20.483 INFO:teuthology.orchestra.run.smithi138.stdout: "poolstat_active": 0, 2024-04-25T11:47:20.483 INFO:teuthology.orchestra.run.smithi138.stdout: "poolstat_send": 0, 2024-04-25T11:47:20.483 INFO:teuthology.orchestra.run.smithi138.stdout: "poolstat_resend": 0, 2024-04-25T11:47:20.483 INFO:teuthology.orchestra.run.smithi138.stdout: "statfs_active": 0, 2024-04-25T11:47:20.483 INFO:teuthology.orchestra.run.smithi138.stdout: "statfs_send": 0, 2024-04-25T11:47:20.483 INFO:teuthology.orchestra.run.smithi138.stdout: "statfs_resend": 0, 2024-04-25T11:47:20.484 INFO:teuthology.orchestra.run.smithi138.stdout: "command_active": 0, 2024-04-25T11:47:20.484 INFO:teuthology.orchestra.run.smithi138.stdout: "command_send": 0, 2024-04-25T11:47:20.484 INFO:teuthology.orchestra.run.smithi138.stdout: "command_resend": 0, 2024-04-25T11:47:20.484 INFO:teuthology.orchestra.run.smithi138.stdout: "map_epoch": 93, 2024-04-25T11:47:20.484 INFO:teuthology.orchestra.run.smithi138.stdout: "map_full": 0, 2024-04-25T11:47:20.484 INFO:teuthology.orchestra.run.smithi138.stdout: "map_inc": 16, 2024-04-25T11:47:20.484 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_sessions": 9, 2024-04-25T11:47:20.484 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_session_open": 9, 2024-04-25T11:47:20.484 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_session_close": 0, 2024-04-25T11:47:20.484 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_laggy": 0, 2024-04-25T11:47:20.484 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_wr": 17, 2024-04-25T11:47:20.484 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_rd": 0, 2024-04-25T11:47:20.484 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_del": 2 2024-04-25T11:47:20.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.484 INFO:teuthology.orchestra.run.smithi138.stdout: "oft": { 2024-04-25T11:47:20.485 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_total_objs": 1, 2024-04-25T11:47:20.485 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_total_kv_pairs": 0, 2024-04-25T11:47:20.485 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_total_updates": 0, 2024-04-25T11:47:20.485 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_total_removes": 0 2024-04-25T11:47:20.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.485 INFO:teuthology.orchestra.run.smithi138.stdout: "osd-slow-ops": { 2024-04-25T11:47:20.485 INFO:teuthology.orchestra.run.smithi138.stdout: "slow_ops_count": 0 2024-04-25T11:47:20.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.485 INFO:teuthology.orchestra.run.smithi138.stdout: "purge_queue": { 2024-04-25T11:47:20.485 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_executing_ops": 0, 2024-04-25T11:47:20.485 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T11:47:20.485 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_executing": 0, 2024-04-25T11:47:20.485 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_executing_high_water": 0, 2024-04-25T11:47:20.485 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_executed_ops": 0, 2024-04-25T11:47:20.485 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_executed": 0, 2024-04-25T11:47:20.486 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_item_in_journal": 0 2024-04-25T11:47:20.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.486 INFO:teuthology.orchestra.run.smithi138.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T11:47:20.486 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 0, 2024-04-25T11:47:20.486 INFO:teuthology.orchestra.run.smithi138.stdout: "max": 104857600, 2024-04-25T11:47:20.486 INFO:teuthology.orchestra.run.smithi138.stdout: "get_started": 0, 2024-04-25T11:47:20.486 INFO:teuthology.orchestra.run.smithi138.stdout: "get": 1924, 2024-04-25T11:47:20.486 INFO:teuthology.orchestra.run.smithi138.stdout: "get_sum": 1979549, 2024-04-25T11:47:20.486 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:20.486 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_success": 1924, 2024-04-25T11:47:20.486 INFO:teuthology.orchestra.run.smithi138.stdout: "take": 0, 2024-04-25T11:47:20.486 INFO:teuthology.orchestra.run.smithi138.stdout: "take_sum": 0, 2024-04-25T11:47:20.486 INFO:teuthology.orchestra.run.smithi138.stdout: "put": 1924, 2024-04-25T11:47:20.486 INFO:teuthology.orchestra.run.smithi138.stdout: "put_sum": 1979549, 2024-04-25T11:47:20.487 INFO:teuthology.orchestra.run.smithi138.stdout: "wait": { 2024-04-25T11:47:20.487 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.487 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.487 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.487 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:20.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.487 INFO:teuthology.orchestra.run.smithi138.stdout: "throttle-objecter_bytes": { 2024-04-25T11:47:20.487 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 0, 2024-04-25T11:47:20.487 INFO:teuthology.orchestra.run.smithi138.stdout: "max": 104857600, 2024-04-25T11:47:20.487 INFO:teuthology.orchestra.run.smithi138.stdout: "get_started": 0, 2024-04-25T11:47:20.487 INFO:teuthology.orchestra.run.smithi138.stdout: "get": 0, 2024-04-25T11:47:20.487 INFO:teuthology.orchestra.run.smithi138.stdout: "get_sum": 0, 2024-04-25T11:47:20.487 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:20.487 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:20.488 INFO:teuthology.orchestra.run.smithi138.stdout: "take": 43, 2024-04-25T11:47:20.488 INFO:teuthology.orchestra.run.smithi138.stdout: "take_sum": 12099, 2024-04-25T11:47:20.488 INFO:teuthology.orchestra.run.smithi138.stdout: "put": 43, 2024-04-25T11:47:20.488 INFO:teuthology.orchestra.run.smithi138.stdout: "put_sum": 12099, 2024-04-25T11:47:20.488 INFO:teuthology.orchestra.run.smithi138.stdout: "wait": { 2024-04-25T11:47:20.488 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.488 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.488 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.488 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:20.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.488 INFO:teuthology.orchestra.run.smithi138.stdout: "throttle-objecter_ops": { 2024-04-25T11:47:20.488 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 0, 2024-04-25T11:47:20.488 INFO:teuthology.orchestra.run.smithi138.stdout: "max": 1024, 2024-04-25T11:47:20.488 INFO:teuthology.orchestra.run.smithi138.stdout: "get_started": 0, 2024-04-25T11:47:20.489 INFO:teuthology.orchestra.run.smithi138.stdout: "get": 0, 2024-04-25T11:47:20.489 INFO:teuthology.orchestra.run.smithi138.stdout: "get_sum": 0, 2024-04-25T11:47:20.489 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:20.489 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:20.489 INFO:teuthology.orchestra.run.smithi138.stdout: "take": 43, 2024-04-25T11:47:20.489 INFO:teuthology.orchestra.run.smithi138.stdout: "take_sum": 43, 2024-04-25T11:47:20.489 INFO:teuthology.orchestra.run.smithi138.stdout: "put": 43, 2024-04-25T11:47:20.489 INFO:teuthology.orchestra.run.smithi138.stdout: "put_sum": 43, 2024-04-25T11:47:20.489 INFO:teuthology.orchestra.run.smithi138.stdout: "wait": { 2024-04-25T11:47:20.489 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.489 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.489 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.489 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:20.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.490 INFO:teuthology.orchestra.run.smithi138.stdout: "throttle-write_buf_throttle": { 2024-04-25T11:47:20.490 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 0, 2024-04-25T11:47:20.490 INFO:teuthology.orchestra.run.smithi138.stdout: "max": 3758096384, 2024-04-25T11:47:20.490 INFO:teuthology.orchestra.run.smithi138.stdout: "get_started": 0, 2024-04-25T11:47:20.490 INFO:teuthology.orchestra.run.smithi138.stdout: "get": 0, 2024-04-25T11:47:20.490 INFO:teuthology.orchestra.run.smithi138.stdout: "get_sum": 0, 2024-04-25T11:47:20.490 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:20.490 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:20.490 INFO:teuthology.orchestra.run.smithi138.stdout: "take": 0, 2024-04-25T11:47:20.490 INFO:teuthology.orchestra.run.smithi138.stdout: "take_sum": 0, 2024-04-25T11:47:20.490 INFO:teuthology.orchestra.run.smithi138.stdout: "put": 0, 2024-04-25T11:47:20.490 INFO:teuthology.orchestra.run.smithi138.stdout: "put_sum": 0, 2024-04-25T11:47:20.490 INFO:teuthology.orchestra.run.smithi138.stdout: "wait": { 2024-04-25T11:47:20.490 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.491 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.491 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.491 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:20.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:20.491 INFO:teuthology.orchestra.run.smithi138.stdout: "throttle-write_buf_throttle-0x55d0b31e4750": { 2024-04-25T11:47:20.491 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 0, 2024-04-25T11:47:20.491 INFO:teuthology.orchestra.run.smithi138.stdout: "max": 3758096384, 2024-04-25T11:47:20.491 INFO:teuthology.orchestra.run.smithi138.stdout: "get_started": 0, 2024-04-25T11:47:20.491 INFO:teuthology.orchestra.run.smithi138.stdout: "get": 6, 2024-04-25T11:47:20.491 INFO:teuthology.orchestra.run.smithi138.stdout: "get_sum": 1652, 2024-04-25T11:47:20.491 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:20.491 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_success": 6, 2024-04-25T11:47:20.491 INFO:teuthology.orchestra.run.smithi138.stdout: "take": 0, 2024-04-25T11:47:20.491 INFO:teuthology.orchestra.run.smithi138.stdout: "take_sum": 0, 2024-04-25T11:47:20.491 INFO:teuthology.orchestra.run.smithi138.stdout: "put": 5, 2024-04-25T11:47:20.492 INFO:teuthology.orchestra.run.smithi138.stdout: "put_sum": 1652, 2024-04-25T11:47:20.492 INFO:teuthology.orchestra.run.smithi138.stdout: "wait": { 2024-04-25T11:47:20.492 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:20.492 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:20.492 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:20.492 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:20.492 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:20.492 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-25T11:47:20.492 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:20 smithi090 ceph-mon[29575]: pgmap v381: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 18 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-04-25T11:47:20.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:20 smithi156 ceph-mon[33045]: pgmap v381: 97 pgs: 24 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 33 active+clean; 18 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-04-25T11:47:21.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:21 smithi138 ceph-mon[32328]: pgmap v382: 97 pgs: 16 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 53 active+clean; 14 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-04-25T11:47:21.469 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-25T11:47:21.469 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-25T11:47:21.469 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-04-25T11:47:21.469 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=12 2024-04-25T11:47:21.470 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=6 2024-04-25T11:47:21.470 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-04-25T11:47:21.470 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T11:47:21.470 DEBUG:tasks.check_counter:Getting stats from h 2024-04-25T11:47:21.470 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-25T11:47:21.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:21 smithi090 ceph-mon[29575]: pgmap v382: 97 pgs: 16 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 53 active+clean; 14 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-04-25T11:47:21.584 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:21 smithi156 ceph-mon[33045]: pgmap v382: 97 pgs: 16 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 53 active+clean; 14 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-04-25T11:47:21.732 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T11:47:23.573 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-25T11:47:23.574 INFO:teuthology.orchestra.run.smithi138.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T11:47:23.574 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_messages": 2093, 2024-04-25T11:47:23.574 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_messages": 2069, 2024-04-25T11:47:23.574 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_bytes": 187676520, 2024-04-25T11:47:23.574 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_bytes": 723900, 2024-04-25T11:47:23.574 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_created_connections": 7, 2024-04-25T11:47:23.574 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_active_connections": 3, 2024-04-25T11:47:23.574 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_total_time": 0.329084781, 2024-04-25T11:47:23.574 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_send_time": 0.074142296, 2024-04-25T11:47:23.574 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_recv_time": 0.191620235, 2024-04-25T11:47:23.574 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_fast_dispatch_time": 0.059172999, 2024-04-25T11:47:23.574 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:23.574 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 2069, 2024-04-25T11:47:23.575 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.081006763, 2024-04-25T11:47:23.575 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000039152 2024-04-25T11:47:23.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.575 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:23.575 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.575 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.575 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.575 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_encrypted_bytes": 1630848, 2024-04-25T11:47:23.575 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_encrypted_bytes": 147680 2024-04-25T11:47:23.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.575 INFO:teuthology.orchestra.run.smithi138.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T11:47:23.575 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_messages": 884, 2024-04-25T11:47:23.575 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_messages": 1060, 2024-04-25T11:47:23.576 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_bytes": 222601937, 2024-04-25T11:47:23.576 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_bytes": 760638, 2024-04-25T11:47:23.576 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_created_connections": 9, 2024-04-25T11:47:23.576 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_active_connections": 7, 2024-04-25T11:47:23.576 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_total_time": 0.218974799, 2024-04-25T11:47:23.576 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_send_time": 0.027450634, 2024-04-25T11:47:23.576 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_recv_time": 0.152786242, 2024-04-25T11:47:23.576 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_fast_dispatch_time": 0.029415720, 2024-04-25T11:47:23.576 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:23.576 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 1057, 2024-04-25T11:47:23.576 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.053487131, 2024-04-25T11:47:23.576 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000050602 2024-04-25T11:47:23.576 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.577 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:23.577 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.577 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.577 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.577 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T11:47:23.577 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_encrypted_bytes": 480096 2024-04-25T11:47:23.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.577 INFO:teuthology.orchestra.run.smithi138.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T11:47:23.577 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_messages": 777, 2024-04-25T11:47:23.577 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_messages": 779, 2024-04-25T11:47:23.577 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_bytes": 235151471, 2024-04-25T11:47:23.577 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_bytes": 247093, 2024-04-25T11:47:23.577 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_created_connections": 6, 2024-04-25T11:47:23.577 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_active_connections": 5, 2024-04-25T11:47:23.577 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_total_time": 0.219845821, 2024-04-25T11:47:23.578 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_send_time": 0.017089227, 2024-04-25T11:47:23.578 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_recv_time": 0.169356554, 2024-04-25T11:47:23.578 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_fast_dispatch_time": 0.026039706, 2024-04-25T11:47:23.578 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:23.578 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 777, 2024-04-25T11:47:23.578 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.060529372, 2024-04-25T11:47:23.578 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000077901 2024-04-25T11:47:23.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.578 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:23.578 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.578 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.578 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.578 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T11:47:23.579 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T11:47:23.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.579 INFO:teuthology.orchestra.run.smithi138.stdout: "cct": { 2024-04-25T11:47:23.579 INFO:teuthology.orchestra.run.smithi138.stdout: "total_workers": 1, 2024-04-25T11:47:23.579 INFO:teuthology.orchestra.run.smithi138.stdout: "unhealthy_workers": 0 2024-04-25T11:47:23.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.579 INFO:teuthology.orchestra.run.smithi138.stdout: "finisher-MDSRank": { 2024-04-25T11:47:23.579 INFO:teuthology.orchestra.run.smithi138.stdout: "queue_len": 0, 2024-04-25T11:47:23.579 INFO:teuthology.orchestra.run.smithi138.stdout: "complete_latency": { 2024-04-25T11:47:23.579 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 5314, 2024-04-25T11:47:23.579 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.152208339, 2024-04-25T11:47:23.579 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000028642 2024-04-25T11:47:23.579 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:23.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.579 INFO:teuthology.orchestra.run.smithi138.stdout: "finisher-PurgeQueue": { 2024-04-25T11:47:23.580 INFO:teuthology.orchestra.run.smithi138.stdout: "queue_len": 0, 2024-04-25T11:47:23.580 INFO:teuthology.orchestra.run.smithi138.stdout: "complete_latency": { 2024-04-25T11:47:23.580 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.580 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.580 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.580 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:23.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.580 INFO:teuthology.orchestra.run.smithi138.stdout: "mds": { 2024-04-25T11:47:23.580 INFO:teuthology.orchestra.run.smithi138.stdout: "request": 0, 2024-04-25T11:47:23.580 INFO:teuthology.orchestra.run.smithi138.stdout: "reply": 0, 2024-04-25T11:47:23.580 INFO:teuthology.orchestra.run.smithi138.stdout: "reply_latency": { 2024-04-25T11:47:23.580 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.580 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.580 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.580 INFO:teuthology.orchestra.run.smithi138.stdout: "slow_reply": 0, 2024-04-25T11:47:23.581 INFO:teuthology.orchestra.run.smithi138.stdout: "forward": 0, 2024-04-25T11:47:23.581 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_fetch_complete": 0, 2024-04-25T11:47:23.581 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_fetch_keys": 0, 2024-04-25T11:47:23.581 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_commit": 0, 2024-04-25T11:47:23.581 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_split": 0, 2024-04-25T11:47:23.581 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_merge": 0, 2024-04-25T11:47:23.581 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes": 128, 2024-04-25T11:47:23.581 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes_top": 44, 2024-04-25T11:47:23.581 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes_bottom": 84, 2024-04-25T11:47:23.581 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes_pin_tail": 0, 2024-04-25T11:47:23.581 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes_pinned": 66, 2024-04-25T11:47:23.581 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes_expired": 0, 2024-04-25T11:47:23.581 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes_with_caps": 0, 2024-04-25T11:47:23.581 INFO:teuthology.orchestra.run.smithi138.stdout: "caps": 0, 2024-04-25T11:47:23.581 INFO:teuthology.orchestra.run.smithi138.stdout: "subtrees": 3, 2024-04-25T11:47:23.581 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse": 0, 2024-04-25T11:47:23.582 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse_hit": 0, 2024-04-25T11:47:23.582 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse_forward": 0, 2024-04-25T11:47:23.582 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse_discover": 0, 2024-04-25T11:47:23.582 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse_dir_fetch": 0, 2024-04-25T11:47:23.582 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse_remote_ino": 0, 2024-04-25T11:47:23.582 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse_lock": 0, 2024-04-25T11:47:23.582 INFO:teuthology.orchestra.run.smithi138.stdout: "load_cent": 0, 2024-04-25T11:47:23.582 INFO:teuthology.orchestra.run.smithi138.stdout: "q": 0, 2024-04-25T11:47:23.582 INFO:teuthology.orchestra.run.smithi138.stdout: "exported": 0, 2024-04-25T11:47:23.582 INFO:teuthology.orchestra.run.smithi138.stdout: "exported_inodes": 0, 2024-04-25T11:47:23.582 INFO:teuthology.orchestra.run.smithi138.stdout: "imported": 0, 2024-04-25T11:47:23.582 INFO:teuthology.orchestra.run.smithi138.stdout: "imported_inodes": 0, 2024-04-25T11:47:23.582 INFO:teuthology.orchestra.run.smithi138.stdout: "openino_dir_fetch": 0, 2024-04-25T11:47:23.582 INFO:teuthology.orchestra.run.smithi138.stdout: "openino_backtrace_fetch": 0, 2024-04-25T11:47:23.583 INFO:teuthology.orchestra.run.smithi138.stdout: "openino_peer_discover": 0, 2024-04-25T11:47:23.583 INFO:teuthology.orchestra.run.smithi138.stdout: "root_rfiles": 4, 2024-04-25T11:47:23.583 INFO:teuthology.orchestra.run.smithi138.stdout: "root_rbytes": 234, 2024-04-25T11:47:23.583 INFO:teuthology.orchestra.run.smithi138.stdout: "root_rsnaps": 6, 2024-04-25T11:47:23.583 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T11:47:23.583 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_set_tag": 0, 2024-04-25T11:47:23.583 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T11:47:23.583 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_inotable_repaired": 0, 2024-04-25T11:47:23.583 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_dir_inodes": 0, 2024-04-25T11:47:23.583 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T11:47:23.583 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T11:47:23.583 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_file_inodes": 0, 2024-04-25T11:47:23.583 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_inode_file_caps": 0, 2024-04-25T11:47:23.583 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T11:47:23.583 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_cap_op_grant": 0, 2024-04-25T11:47:23.583 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T11:47:23.584 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T11:47:23.584 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T11:47:23.584 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_client_caps": 0, 2024-04-25T11:47:23.584 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_client_caps_dirty": 0, 2024-04-25T11:47:23.584 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_client_cap_release": 0, 2024-04-25T11:47:23.584 INFO:teuthology.orchestra.run.smithi138.stdout: "process_request_cap_release": 0 2024-04-25T11:47:23.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.584 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_cache": { 2024-04-25T11:47:23.584 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_update": 0, 2024-04-25T11:47:23.584 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_update_receipt": 0, 2024-04-25T11:47:23.584 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_try_discover": 0, 2024-04-25T11:47:23.584 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_send_discover": 0, 2024-04-25T11:47:23.584 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_handle_discover": 0, 2024-04-25T11:47:23.584 INFO:teuthology.orchestra.run.smithi138.stdout: "num_strays": 0, 2024-04-25T11:47:23.584 INFO:teuthology.orchestra.run.smithi138.stdout: "num_strays_delayed": 0, 2024-04-25T11:47:23.584 INFO:teuthology.orchestra.run.smithi138.stdout: "num_strays_enqueuing": 0, 2024-04-25T11:47:23.584 INFO:teuthology.orchestra.run.smithi138.stdout: "strays_created": 0, 2024-04-25T11:47:23.585 INFO:teuthology.orchestra.run.smithi138.stdout: "strays_enqueued": 0, 2024-04-25T11:47:23.585 INFO:teuthology.orchestra.run.smithi138.stdout: "strays_reintegrated": 0, 2024-04-25T11:47:23.585 INFO:teuthology.orchestra.run.smithi138.stdout: "strays_migrated": 0, 2024-04-25T11:47:23.585 INFO:teuthology.orchestra.run.smithi138.stdout: "num_recovering_processing": 0, 2024-04-25T11:47:23.585 INFO:teuthology.orchestra.run.smithi138.stdout: "num_recovering_enqueued": 0, 2024-04-25T11:47:23.585 INFO:teuthology.orchestra.run.smithi138.stdout: "num_recovering_prioritized": 0, 2024-04-25T11:47:23.585 INFO:teuthology.orchestra.run.smithi138.stdout: "recovery_started": 0, 2024-04-25T11:47:23.585 INFO:teuthology.orchestra.run.smithi138.stdout: "recovery_completed": 0, 2024-04-25T11:47:23.585 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_quiesce_path": 0, 2024-04-25T11:47:23.585 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_quiesce_inode": 0, 2024-04-25T11:47:23.585 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T11:47:23.585 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_exportdir": 0, 2024-04-25T11:47:23.585 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_flush": 0, 2024-04-25T11:47:23.585 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_fragmentdir": 0, 2024-04-25T11:47:23.585 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_fragstats": 0, 2024-04-25T11:47:23.585 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_inodestats": 0 2024-04-25T11:47:23.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.586 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_log": { 2024-04-25T11:47:23.586 INFO:teuthology.orchestra.run.smithi138.stdout: "evlrg": 0, 2024-04-25T11:47:23.586 INFO:teuthology.orchestra.run.smithi138.stdout: "evadd": 0, 2024-04-25T11:47:23.586 INFO:teuthology.orchestra.run.smithi138.stdout: "evex": 0, 2024-04-25T11:47:23.586 INFO:teuthology.orchestra.run.smithi138.stdout: "evtrm": 0, 2024-04-25T11:47:23.586 INFO:teuthology.orchestra.run.smithi138.stdout: "ev": 192169, 2024-04-25T11:47:23.586 INFO:teuthology.orchestra.run.smithi138.stdout: "evexg": 0, 2024-04-25T11:47:23.586 INFO:teuthology.orchestra.run.smithi138.stdout: "evexd": 0, 2024-04-25T11:47:23.586 INFO:teuthology.orchestra.run.smithi138.stdout: "segadd": 0, 2024-04-25T11:47:23.586 INFO:teuthology.orchestra.run.smithi138.stdout: "segex": 0, 2024-04-25T11:47:23.586 INFO:teuthology.orchestra.run.smithi138.stdout: "segtrm": 0, 2024-04-25T11:47:23.586 INFO:teuthology.orchestra.run.smithi138.stdout: "seg": 2, 2024-04-25T11:47:23.587 INFO:teuthology.orchestra.run.smithi138.stdout: "segmjr": 52, 2024-04-25T11:47:23.587 INFO:teuthology.orchestra.run.smithi138.stdout: "segexg": 0, 2024-04-25T11:47:23.587 INFO:teuthology.orchestra.run.smithi138.stdout: "segexd": 0, 2024-04-25T11:47:23.587 INFO:teuthology.orchestra.run.smithi138.stdout: "expos": 646513167, 2024-04-25T11:47:23.587 INFO:teuthology.orchestra.run.smithi138.stdout: "wrpos": 646827462, 2024-04-25T11:47:23.587 INFO:teuthology.orchestra.run.smithi138.stdout: "rdpos": 646827350, 2024-04-25T11:47:23.587 INFO:teuthology.orchestra.run.smithi138.stdout: "jlat": { 2024-04-25T11:47:23.587 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.587 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.587 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.587 INFO:teuthology.orchestra.run.smithi138.stdout: "replayed": 192169 2024-04-25T11:47:23.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.587 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_mem": { 2024-04-25T11:47:23.587 INFO:teuthology.orchestra.run.smithi138.stdout: "ino": 89, 2024-04-25T11:47:23.588 INFO:teuthology.orchestra.run.smithi138.stdout: "ino+": 107, 2024-04-25T11:47:23.588 INFO:teuthology.orchestra.run.smithi138.stdout: "ino-": 18, 2024-04-25T11:47:23.588 INFO:teuthology.orchestra.run.smithi138.stdout: "dir": 13, 2024-04-25T11:47:23.588 INFO:teuthology.orchestra.run.smithi138.stdout: "dir+": 13, 2024-04-25T11:47:23.588 INFO:teuthology.orchestra.run.smithi138.stdout: "dir-": 0, 2024-04-25T11:47:23.588 INFO:teuthology.orchestra.run.smithi138.stdout: "dn": 128, 2024-04-25T11:47:23.588 INFO:teuthology.orchestra.run.smithi138.stdout: "dn+": 130, 2024-04-25T11:47:23.588 INFO:teuthology.orchestra.run.smithi138.stdout: "dn-": 2, 2024-04-25T11:47:23.588 INFO:teuthology.orchestra.run.smithi138.stdout: "cap": 0, 2024-04-25T11:47:23.588 INFO:teuthology.orchestra.run.smithi138.stdout: "cap+": 0, 2024-04-25T11:47:23.588 INFO:teuthology.orchestra.run.smithi138.stdout: "cap-": 0, 2024-04-25T11:47:23.588 INFO:teuthology.orchestra.run.smithi138.stdout: "rss": 123528, 2024-04-25T11:47:23.588 INFO:teuthology.orchestra.run.smithi138.stdout: "heap": 215440 2024-04-25T11:47:23.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.589 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_server": { 2024-04-25T11:47:23.589 INFO:teuthology.orchestra.run.smithi138.stdout: "dispatch_client_request": 0, 2024-04-25T11:47:23.589 INFO:teuthology.orchestra.run.smithi138.stdout: "dispatch_server_request": 0, 2024-04-25T11:47:23.589 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_client_request": 0, 2024-04-25T11:47:23.589 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_client_session": 0, 2024-04-25T11:47:23.589 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_peer_request": 0, 2024-04-25T11:47:23.589 INFO:teuthology.orchestra.run.smithi138.stdout: "req_create_latency": { 2024-04-25T11:47:23.589 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.589 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.589 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.589 INFO:teuthology.orchestra.run.smithi138.stdout: "req_getattr_latency": { 2024-04-25T11:47:23.589 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.589 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.589 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.590 INFO:teuthology.orchestra.run.smithi138.stdout: "req_getfilelock_latency": { 2024-04-25T11:47:23.590 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.590 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.590 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.590 INFO:teuthology.orchestra.run.smithi138.stdout: "req_link_latency": { 2024-04-25T11:47:23.590 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.590 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.590 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.590 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lookup_latency": { 2024-04-25T11:47:23.590 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.590 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.590 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.591 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lookuphash_latency": { 2024-04-25T11:47:23.591 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.591 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.591 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.591 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lookupino_latency": { 2024-04-25T11:47:23.591 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.591 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.591 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.591 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lookupname_latency": { 2024-04-25T11:47:23.591 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.591 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.591 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.592 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lookupparent_latency": { 2024-04-25T11:47:23.592 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.592 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.592 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.592 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lookupsnap_latency": { 2024-04-25T11:47:23.592 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.592 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.592 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.592 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lssnap_latency": { 2024-04-25T11:47:23.592 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.592 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.592 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.592 INFO:teuthology.orchestra.run.smithi138.stdout: "req_mkdir_latency": { 2024-04-25T11:47:23.593 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.593 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.593 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.593 INFO:teuthology.orchestra.run.smithi138.stdout: "req_mknod_latency": { 2024-04-25T11:47:23.593 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.593 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.593 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.594 INFO:teuthology.orchestra.run.smithi138.stdout: "req_mksnap_latency": { 2024-04-25T11:47:23.594 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.594 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.594 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.594 INFO:teuthology.orchestra.run.smithi138.stdout: "req_open_latency": { 2024-04-25T11:47:23.594 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.594 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.594 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.594 INFO:teuthology.orchestra.run.smithi138.stdout: "req_readdir_latency": { 2024-04-25T11:47:23.594 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.594 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.595 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.595 INFO:teuthology.orchestra.run.smithi138.stdout: "req_rename_latency": { 2024-04-25T11:47:23.595 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.595 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.595 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.595 INFO:teuthology.orchestra.run.smithi138.stdout: "req_renamesnap_latency": { 2024-04-25T11:47:23.595 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.595 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.595 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.595 INFO:teuthology.orchestra.run.smithi138.stdout: "req_snapdiff_latency": { 2024-04-25T11:47:23.595 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.595 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.596 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.596 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.596 INFO:teuthology.orchestra.run.smithi138.stdout: "req_rmdir_latency": { 2024-04-25T11:47:23.596 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.596 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.596 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.596 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.596 INFO:teuthology.orchestra.run.smithi138.stdout: "req_rmsnap_latency": { 2024-04-25T11:47:23.596 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.596 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.596 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.596 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.596 INFO:teuthology.orchestra.run.smithi138.stdout: "req_rmxattr_latency": { 2024-04-25T11:47:23.596 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.597 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.597 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.597 INFO:teuthology.orchestra.run.smithi138.stdout: "req_setattr_latency": { 2024-04-25T11:47:23.597 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.597 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.597 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.597 INFO:teuthology.orchestra.run.smithi138.stdout: "req_setdirlayout_latency": { 2024-04-25T11:47:23.597 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.597 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.597 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.597 INFO:teuthology.orchestra.run.smithi138.stdout: "req_setfilelock_latency": { 2024-04-25T11:47:23.597 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.597 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.598 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.598 INFO:teuthology.orchestra.run.smithi138.stdout: "req_setlayout_latency": { 2024-04-25T11:47:23.598 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.598 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.598 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.598 INFO:teuthology.orchestra.run.smithi138.stdout: "req_setxattr_latency": { 2024-04-25T11:47:23.598 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.598 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.598 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.598 INFO:teuthology.orchestra.run.smithi138.stdout: "req_symlink_latency": { 2024-04-25T11:47:23.598 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.599 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.599 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.599 INFO:teuthology.orchestra.run.smithi138.stdout: "req_unlink_latency": { 2024-04-25T11:47:23.599 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.599 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.599 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.599 INFO:teuthology.orchestra.run.smithi138.stdout: "cap_revoke_eviction": 0, 2024-04-25T11:47:23.599 INFO:teuthology.orchestra.run.smithi138.stdout: "cap_acquisition_throttle": 0, 2024-04-25T11:47:23.599 INFO:teuthology.orchestra.run.smithi138.stdout: "req_getvxattr_latency": { 2024-04-25T11:47:23.599 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.599 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.599 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.599 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:23.600 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.600 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_sessions": { 2024-04-25T11:47:23.600 INFO:teuthology.orchestra.run.smithi138.stdout: "session_count": 2, 2024-04-25T11:47:23.600 INFO:teuthology.orchestra.run.smithi138.stdout: "session_add": 5, 2024-04-25T11:47:23.600 INFO:teuthology.orchestra.run.smithi138.stdout: "session_remove": 3, 2024-04-25T11:47:23.600 INFO:teuthology.orchestra.run.smithi138.stdout: "sessions_open": 4, 2024-04-25T11:47:23.600 INFO:teuthology.orchestra.run.smithi138.stdout: "sessions_stale": 0, 2024-04-25T11:47:23.600 INFO:teuthology.orchestra.run.smithi138.stdout: "total_load": 0, 2024-04-25T11:47:23.600 INFO:teuthology.orchestra.run.smithi138.stdout: "average_load": 0, 2024-04-25T11:47:23.600 INFO:teuthology.orchestra.run.smithi138.stdout: "avg_session_uptime": 3141, 2024-04-25T11:47:23.600 INFO:teuthology.orchestra.run.smithi138.stdout: "mdthresh_evicted": 0 2024-04-25T11:47:23.600 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.600 INFO:teuthology.orchestra.run.smithi138.stdout: "mempool": { 2024-04-25T11:47:23.601 INFO:teuthology.orchestra.run.smithi138.stdout: "bloom_filter_bytes": 0, 2024-04-25T11:47:23.601 INFO:teuthology.orchestra.run.smithi138.stdout: "bloom_filter_items": 0, 2024-04-25T11:47:23.601 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T11:47:23.601 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_alloc_items": 0, 2024-04-25T11:47:23.601 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T11:47:23.601 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_data_items": 0, 2024-04-25T11:47:23.601 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T11:47:23.601 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T11:47:23.601 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T11:47:23.601 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T11:47:23.601 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T11:47:23.601 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_other_items": 0, 2024-04-25T11:47:23.601 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T11:47:23.601 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T11:47:23.601 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_extent_bytes": 0, 2024-04-25T11:47:23.601 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_extent_items": 0, 2024-04-25T11:47:23.601 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_blob_bytes": 0, 2024-04-25T11:47:23.602 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_blob_items": 0, 2024-04-25T11:47:23.602 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T11:47:23.602 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T11:47:23.602 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T11:47:23.602 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T11:47:23.602 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T11:47:23.602 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_fsck_items": 0, 2024-04-25T11:47:23.602 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_txc_bytes": 0, 2024-04-25T11:47:23.602 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_txc_items": 0, 2024-04-25T11:47:23.602 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T11:47:23.602 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T11:47:23.602 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_writing_bytes": 0, 2024-04-25T11:47:23.602 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_writing_items": 0, 2024-04-25T11:47:23.602 INFO:teuthology.orchestra.run.smithi138.stdout: "bluefs_bytes": 0, 2024-04-25T11:47:23.603 INFO:teuthology.orchestra.run.smithi138.stdout: "bluefs_items": 0, 2024-04-25T11:47:23.603 INFO:teuthology.orchestra.run.smithi138.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T11:47:23.603 INFO:teuthology.orchestra.run.smithi138.stdout: "bluefs_file_reader_items": 0, 2024-04-25T11:47:23.603 INFO:teuthology.orchestra.run.smithi138.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T11:47:23.603 INFO:teuthology.orchestra.run.smithi138.stdout: "bluefs_file_writer_items": 0, 2024-04-25T11:47:23.603 INFO:teuthology.orchestra.run.smithi138.stdout: "buffer_anon_bytes": 242014, 2024-04-25T11:47:23.603 INFO:teuthology.orchestra.run.smithi138.stdout: "buffer_anon_items": 167, 2024-04-25T11:47:23.603 INFO:teuthology.orchestra.run.smithi138.stdout: "buffer_meta_bytes": 0, 2024-04-25T11:47:23.603 INFO:teuthology.orchestra.run.smithi138.stdout: "buffer_meta_items": 0, 2024-04-25T11:47:23.603 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_bytes": 0, 2024-04-25T11:47:23.603 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_items": 0, 2024-04-25T11:47:23.603 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_mapbl_bytes": 0, 2024-04-25T11:47:23.603 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_mapbl_items": 0, 2024-04-25T11:47:23.603 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_pglog_bytes": 0, 2024-04-25T11:47:23.603 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_pglog_items": 0, 2024-04-25T11:47:23.604 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap_bytes": 8632, 2024-04-25T11:47:23.604 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap_items": 180, 2024-04-25T11:47:23.604 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T11:47:23.604 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap_mapping_items": 0, 2024-04-25T11:47:23.604 INFO:teuthology.orchestra.run.smithi138.stdout: "pgmap_bytes": 0, 2024-04-25T11:47:23.604 INFO:teuthology.orchestra.run.smithi138.stdout: "pgmap_items": 0, 2024-04-25T11:47:23.604 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_co_bytes": 349817, 2024-04-25T11:47:23.604 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_co_items": 3029, 2024-04-25T11:47:23.604 INFO:teuthology.orchestra.run.smithi138.stdout: "unittest_1_bytes": 0, 2024-04-25T11:47:23.604 INFO:teuthology.orchestra.run.smithi138.stdout: "unittest_1_items": 0, 2024-04-25T11:47:23.604 INFO:teuthology.orchestra.run.smithi138.stdout: "unittest_2_bytes": 0, 2024-04-25T11:47:23.604 INFO:teuthology.orchestra.run.smithi138.stdout: "unittest_2_items": 0 2024-04-25T11:47:23.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.604 INFO:teuthology.orchestra.run.smithi138.stdout: "objecter": { 2024-04-25T11:47:23.604 INFO:teuthology.orchestra.run.smithi138.stdout: "op_active": 0, 2024-04-25T11:47:23.604 INFO:teuthology.orchestra.run.smithi138.stdout: "op_laggy": 0, 2024-04-25T11:47:23.604 INFO:teuthology.orchestra.run.smithi138.stdout: "op_send": 3471, 2024-04-25T11:47:23.605 INFO:teuthology.orchestra.run.smithi138.stdout: "op_send_bytes": 26, 2024-04-25T11:47:23.605 INFO:teuthology.orchestra.run.smithi138.stdout: "op_resend": 4, 2024-04-25T11:47:23.605 INFO:teuthology.orchestra.run.smithi138.stdout: "op_reply": 3471, 2024-04-25T11:47:23.605 INFO:teuthology.orchestra.run.smithi138.stdout: "op_latency": { 2024-04-25T11:47:23.605 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 3471, 2024-04-25T11:47:23.605 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 57.558718620, 2024-04-25T11:47:23.605 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.016582748 2024-04-25T11:47:23.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.605 INFO:teuthology.orchestra.run.smithi138.stdout: "op_inflight": 0, 2024-04-25T11:47:23.605 INFO:teuthology.orchestra.run.smithi138.stdout: "oplen_avg": { 2024-04-25T11:47:23.605 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 3471, 2024-04-25T11:47:23.605 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 3472 2024-04-25T11:47:23.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.605 INFO:teuthology.orchestra.run.smithi138.stdout: "op": 3471, 2024-04-25T11:47:23.605 INFO:teuthology.orchestra.run.smithi138.stdout: "op_r": 3471, 2024-04-25T11:47:23.605 INFO:teuthology.orchestra.run.smithi138.stdout: "op_w": 0, 2024-04-25T11:47:23.606 INFO:teuthology.orchestra.run.smithi138.stdout: "op_rmw": 0, 2024-04-25T11:47:23.606 INFO:teuthology.orchestra.run.smithi138.stdout: "op_pg": 0, 2024-04-25T11:47:23.606 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_stat": 2236, 2024-04-25T11:47:23.606 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_create": 0, 2024-04-25T11:47:23.606 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_read": 1232, 2024-04-25T11:47:23.606 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_write": 0, 2024-04-25T11:47:23.606 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_writefull": 0, 2024-04-25T11:47:23.606 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_writesame": 0, 2024-04-25T11:47:23.606 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_append": 0, 2024-04-25T11:47:23.606 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_zero": 0, 2024-04-25T11:47:23.606 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_truncate": 0, 2024-04-25T11:47:23.606 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_delete": 0, 2024-04-25T11:47:23.606 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_mapext": 0, 2024-04-25T11:47:23.606 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_sparse_read": 0, 2024-04-25T11:47:23.607 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_clonerange": 0, 2024-04-25T11:47:23.607 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_getxattr": 2, 2024-04-25T11:47:23.607 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_setxattr": 0, 2024-04-25T11:47:23.607 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_cmpxattr": 0, 2024-04-25T11:47:23.607 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_rmxattr": 0, 2024-04-25T11:47:23.607 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_resetxattrs": 0, 2024-04-25T11:47:23.607 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_call": 0, 2024-04-25T11:47:23.607 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_watch": 0, 2024-04-25T11:47:23.607 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_notify": 0, 2024-04-25T11:47:23.607 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T11:47:23.607 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_pgls": 0, 2024-04-25T11:47:23.607 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_pgls_filter": 0, 2024-04-25T11:47:23.607 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_other": 0, 2024-04-25T11:47:23.607 INFO:teuthology.orchestra.run.smithi138.stdout: "linger_active": 0, 2024-04-25T11:47:23.607 INFO:teuthology.orchestra.run.smithi138.stdout: "linger_send": 0, 2024-04-25T11:47:23.607 INFO:teuthology.orchestra.run.smithi138.stdout: "linger_resend": 0, 2024-04-25T11:47:23.608 INFO:teuthology.orchestra.run.smithi138.stdout: "linger_ping": 0, 2024-04-25T11:47:23.608 INFO:teuthology.orchestra.run.smithi138.stdout: "poolop_active": 0, 2024-04-25T11:47:23.608 INFO:teuthology.orchestra.run.smithi138.stdout: "poolop_send": 0, 2024-04-25T11:47:23.608 INFO:teuthology.orchestra.run.smithi138.stdout: "poolop_resend": 0, 2024-04-25T11:47:23.608 INFO:teuthology.orchestra.run.smithi138.stdout: "poolstat_active": 0, 2024-04-25T11:47:23.608 INFO:teuthology.orchestra.run.smithi138.stdout: "poolstat_send": 0, 2024-04-25T11:47:23.608 INFO:teuthology.orchestra.run.smithi138.stdout: "poolstat_resend": 0, 2024-04-25T11:47:23.608 INFO:teuthology.orchestra.run.smithi138.stdout: "statfs_active": 0, 2024-04-25T11:47:23.608 INFO:teuthology.orchestra.run.smithi138.stdout: "statfs_send": 0, 2024-04-25T11:47:23.608 INFO:teuthology.orchestra.run.smithi138.stdout: "statfs_resend": 0, 2024-04-25T11:47:23.608 INFO:teuthology.orchestra.run.smithi138.stdout: "command_active": 0, 2024-04-25T11:47:23.608 INFO:teuthology.orchestra.run.smithi138.stdout: "command_send": 0, 2024-04-25T11:47:23.608 INFO:teuthology.orchestra.run.smithi138.stdout: "command_resend": 0, 2024-04-25T11:47:23.608 INFO:teuthology.orchestra.run.smithi138.stdout: "map_epoch": 93, 2024-04-25T11:47:23.608 INFO:teuthology.orchestra.run.smithi138.stdout: "map_full": 0, 2024-04-25T11:47:23.608 INFO:teuthology.orchestra.run.smithi138.stdout: "map_inc": 16, 2024-04-25T11:47:23.608 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_sessions": 11, 2024-04-25T11:47:23.609 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_session_open": 11, 2024-04-25T11:47:23.609 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_session_close": 0, 2024-04-25T11:47:23.609 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_laggy": 0, 2024-04-25T11:47:23.609 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_wr": 0, 2024-04-25T11:47:23.609 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_rd": 2, 2024-04-25T11:47:23.609 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_del": 0 2024-04-25T11:47:23.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.609 INFO:teuthology.orchestra.run.smithi138.stdout: "oft": { 2024-04-25T11:47:23.609 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_total_objs": 0, 2024-04-25T11:47:23.609 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_total_kv_pairs": 0, 2024-04-25T11:47:23.609 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_total_updates": 0, 2024-04-25T11:47:23.609 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_total_removes": 0 2024-04-25T11:47:23.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.609 INFO:teuthology.orchestra.run.smithi138.stdout: "osd-slow-ops": { 2024-04-25T11:47:23.609 INFO:teuthology.orchestra.run.smithi138.stdout: "slow_ops_count": 0 2024-04-25T11:47:23.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.610 INFO:teuthology.orchestra.run.smithi138.stdout: "purge_queue": { 2024-04-25T11:47:23.610 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_executing_ops": 0, 2024-04-25T11:47:23.610 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T11:47:23.610 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_executing": 0, 2024-04-25T11:47:23.610 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_executing_high_water": 0, 2024-04-25T11:47:23.610 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_executed_ops": 0, 2024-04-25T11:47:23.610 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_executed": 0, 2024-04-25T11:47:23.610 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_item_in_journal": 0 2024-04-25T11:47:23.610 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.610 INFO:teuthology.orchestra.run.smithi138.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T11:47:23.610 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 0, 2024-04-25T11:47:23.610 INFO:teuthology.orchestra.run.smithi138.stdout: "max": 104857600, 2024-04-25T11:47:23.610 INFO:teuthology.orchestra.run.smithi138.stdout: "get_started": 0, 2024-04-25T11:47:23.610 INFO:teuthology.orchestra.run.smithi138.stdout: "get": 3750, 2024-04-25T11:47:23.610 INFO:teuthology.orchestra.run.smithi138.stdout: "get_sum": 645075334, 2024-04-25T11:47:23.610 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:23.611 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_success": 3750, 2024-04-25T11:47:23.611 INFO:teuthology.orchestra.run.smithi138.stdout: "take": 0, 2024-04-25T11:47:23.611 INFO:teuthology.orchestra.run.smithi138.stdout: "take_sum": 0, 2024-04-25T11:47:23.611 INFO:teuthology.orchestra.run.smithi138.stdout: "put": 3750, 2024-04-25T11:47:23.611 INFO:teuthology.orchestra.run.smithi138.stdout: "put_sum": 645075334, 2024-04-25T11:47:23.611 INFO:teuthology.orchestra.run.smithi138.stdout: "wait": { 2024-04-25T11:47:23.611 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.611 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.611 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.611 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:23.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.611 INFO:teuthology.orchestra.run.smithi138.stdout: "throttle-objecter_bytes": { 2024-04-25T11:47:23.611 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 0, 2024-04-25T11:47:23.611 INFO:teuthology.orchestra.run.smithi138.stdout: "max": 104857600, 2024-04-25T11:47:23.611 INFO:teuthology.orchestra.run.smithi138.stdout: "get_started": 0, 2024-04-25T11:47:23.611 INFO:teuthology.orchestra.run.smithi138.stdout: "get": 0, 2024-04-25T11:47:23.611 INFO:teuthology.orchestra.run.smithi138.stdout: "get_sum": 0, 2024-04-25T11:47:23.612 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:23.612 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:23.612 INFO:teuthology.orchestra.run.smithi138.stdout: "take": 3471, 2024-04-25T11:47:23.612 INFO:teuthology.orchestra.run.smithi138.stdout: "take_sum": 642847602, 2024-04-25T11:47:23.612 INFO:teuthology.orchestra.run.smithi138.stdout: "put": 3471, 2024-04-25T11:47:23.612 INFO:teuthology.orchestra.run.smithi138.stdout: "put_sum": 642847602, 2024-04-25T11:47:23.612 INFO:teuthology.orchestra.run.smithi138.stdout: "wait": { 2024-04-25T11:47:23.612 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.612 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.612 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.612 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:23.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.612 INFO:teuthology.orchestra.run.smithi138.stdout: "throttle-objecter_ops": { 2024-04-25T11:47:23.612 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 0, 2024-04-25T11:47:23.612 INFO:teuthology.orchestra.run.smithi138.stdout: "max": 1024, 2024-04-25T11:47:23.613 INFO:teuthology.orchestra.run.smithi138.stdout: "get_started": 0, 2024-04-25T11:47:23.613 INFO:teuthology.orchestra.run.smithi138.stdout: "get": 0, 2024-04-25T11:47:23.613 INFO:teuthology.orchestra.run.smithi138.stdout: "get_sum": 0, 2024-04-25T11:47:23.613 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:23.613 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:23.613 INFO:teuthology.orchestra.run.smithi138.stdout: "take": 3471, 2024-04-25T11:47:23.613 INFO:teuthology.orchestra.run.smithi138.stdout: "take_sum": 3471, 2024-04-25T11:47:23.613 INFO:teuthology.orchestra.run.smithi138.stdout: "put": 3471, 2024-04-25T11:47:23.613 INFO:teuthology.orchestra.run.smithi138.stdout: "put_sum": 3471, 2024-04-25T11:47:23.613 INFO:teuthology.orchestra.run.smithi138.stdout: "wait": { 2024-04-25T11:47:23.613 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.613 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.613 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.613 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:23.613 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.613 INFO:teuthology.orchestra.run.smithi138.stdout: "throttle-write_buf_throttle": { 2024-04-25T11:47:23.614 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 0, 2024-04-25T11:47:23.614 INFO:teuthology.orchestra.run.smithi138.stdout: "max": 3758096384, 2024-04-25T11:47:23.614 INFO:teuthology.orchestra.run.smithi138.stdout: "get_started": 0, 2024-04-25T11:47:23.614 INFO:teuthology.orchestra.run.smithi138.stdout: "get": 0, 2024-04-25T11:47:23.614 INFO:teuthology.orchestra.run.smithi138.stdout: "get_sum": 0, 2024-04-25T11:47:23.614 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:23.614 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:23.614 INFO:teuthology.orchestra.run.smithi138.stdout: "take": 0, 2024-04-25T11:47:23.614 INFO:teuthology.orchestra.run.smithi138.stdout: "take_sum": 0, 2024-04-25T11:47:23.614 INFO:teuthology.orchestra.run.smithi138.stdout: "put": 0, 2024-04-25T11:47:23.614 INFO:teuthology.orchestra.run.smithi138.stdout: "put_sum": 0, 2024-04-25T11:47:23.614 INFO:teuthology.orchestra.run.smithi138.stdout: "wait": { 2024-04-25T11:47:23.614 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.614 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.614 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.614 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:23.615 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:23.615 INFO:teuthology.orchestra.run.smithi138.stdout: "throttle-write_buf_throttle-0x55dde34ce750": { 2024-04-25T11:47:23.615 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 0, 2024-04-25T11:47:23.615 INFO:teuthology.orchestra.run.smithi138.stdout: "max": 3758096384, 2024-04-25T11:47:23.615 INFO:teuthology.orchestra.run.smithi138.stdout: "get_started": 0, 2024-04-25T11:47:23.615 INFO:teuthology.orchestra.run.smithi138.stdout: "get": 0, 2024-04-25T11:47:23.615 INFO:teuthology.orchestra.run.smithi138.stdout: "get_sum": 0, 2024-04-25T11:47:23.615 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:23.615 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:23.615 INFO:teuthology.orchestra.run.smithi138.stdout: "take": 0, 2024-04-25T11:47:23.615 INFO:teuthology.orchestra.run.smithi138.stdout: "take_sum": 0, 2024-04-25T11:47:23.615 INFO:teuthology.orchestra.run.smithi138.stdout: "put": 0, 2024-04-25T11:47:23.615 INFO:teuthology.orchestra.run.smithi138.stdout: "put_sum": 0, 2024-04-25T11:47:23.615 INFO:teuthology.orchestra.run.smithi138.stdout: "wait": { 2024-04-25T11:47:23.615 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:23.615 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:23.616 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:23.616 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:23.616 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:23.616 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-25T11:47:23.799 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:23 smithi138 ceph-mon[32328]: pgmap v383: 97 pgs: 16 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 53 active+clean; 13 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 269 B/s wr, 9 op/s 2024-04-25T11:47:23.835 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:23 smithi156 ceph-mon[33045]: pgmap v383: 97 pgs: 16 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 53 active+clean; 13 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 269 B/s wr, 9 op/s 2024-04-25T11:47:23.972 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:23 smithi090 ceph-mon[29575]: pgmap v383: 97 pgs: 16 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 53 active+clean; 13 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 269 B/s wr, 9 op/s 2024-04-25T11:47:24.284 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-04-25T11:47:24.285 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-04-25T11:47:24.285 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-04-25T11:47:24.285 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=0 2024-04-25T11:47:24.285 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-04-25T11:47:24.285 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-25T11:47:24.285 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T11:47:24.285 DEBUG:tasks.check_counter:Getting stats from k 2024-04-25T11:47:24.285 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-25T11:47:24.546 INFO:teuthology.orchestra.run.smithi138.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T11:47:25.957 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:25 smithi138 ceph-mon[32328]: pgmap v384: 97 pgs: 11 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 63 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T11:47:26.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:25 smithi090 ceph-mon[29575]: pgmap v384: 97 pgs: 11 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 63 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T11:47:26.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:25 smithi156 ceph-mon[33045]: pgmap v384: 97 pgs: 11 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 63 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T11:47:26.552 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-25T11:47:26.552 INFO:teuthology.orchestra.run.smithi138.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T11:47:26.552 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_messages": 82, 2024-04-25T11:47:26.552 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_messages": 235, 2024-04-25T11:47:26.552 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_bytes": 48218, 2024-04-25T11:47:26.553 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_bytes": 452521, 2024-04-25T11:47:26.553 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_created_connections": 16, 2024-04-25T11:47:26.553 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_active_connections": 4, 2024-04-25T11:47:26.553 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_total_time": 0.022953315, 2024-04-25T11:47:26.553 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_send_time": 0.009154574, 2024-04-25T11:47:26.553 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_recv_time": 0.007645678, 2024-04-25T11:47:26.553 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_fast_dispatch_time": 0.000321092, 2024-04-25T11:47:26.553 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:26.553 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 230, 2024-04-25T11:47:26.553 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.026874618, 2024-04-25T11:47:26.553 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000116846 2024-04-25T11:47:26.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.554 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:26.554 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 114, 2024-04-25T11:47:26.554 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000044316, 2024-04-25T11:47:26.554 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000388 2024-04-25T11:47:26.554 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.554 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-25T11:47:26.554 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_encrypted_bytes": 353488 2024-04-25T11:47:26.554 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.554 INFO:teuthology.orchestra.run.smithi138.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T11:47:26.554 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_messages": 1401, 2024-04-25T11:47:26.554 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_messages": 1066, 2024-04-25T11:47:26.554 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_bytes": 433865, 2024-04-25T11:47:26.555 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_bytes": 521755, 2024-04-25T11:47:26.555 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_created_connections": 12, 2024-04-25T11:47:26.555 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_active_connections": 4, 2024-04-25T11:47:26.555 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_total_time": 0.121202578, 2024-04-25T11:47:26.555 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_send_time": 0.054483984, 2024-04-25T11:47:26.555 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_recv_time": 0.078729758, 2024-04-25T11:47:26.555 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_fast_dispatch_time": 0.004493015, 2024-04-25T11:47:26.555 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:26.555 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 1066, 2024-04-25T11:47:26.555 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.072454561, 2024-04-25T11:47:26.555 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000067968 2024-04-25T11:47:26.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.556 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:26.556 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 2272, 2024-04-25T11:47:26.556 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000638024, 2024-04-25T11:47:26.556 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000280 2024-04-25T11:47:26.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.556 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_encrypted_bytes": 1888, 2024-04-25T11:47:26.556 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_encrypted_bytes": 133744 2024-04-25T11:47:26.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.556 INFO:teuthology.orchestra.run.smithi138.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T11:47:26.556 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_messages": 428, 2024-04-25T11:47:26.556 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_messages": 410, 2024-04-25T11:47:26.556 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_bytes": 1671321, 2024-04-25T11:47:26.557 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_bytes": 683861, 2024-04-25T11:47:26.557 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_created_connections": 47, 2024-04-25T11:47:26.557 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_active_connections": 6, 2024-04-25T11:47:26.557 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_total_time": 0.078178784, 2024-04-25T11:47:26.557 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_send_time": 0.018860603, 2024-04-25T11:47:26.557 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_recv_time": 0.063574871, 2024-04-25T11:47:26.557 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_running_fast_dispatch_time": 0.006780895, 2024-04-25T11:47:26.557 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:26.557 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 410, 2024-04-25T11:47:26.557 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.045429754, 2024-04-25T11:47:26.557 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000110804 2024-04-25T11:47:26.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.558 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:26.558 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 155, 2024-04-25T11:47:26.558 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000036012, 2024-04-25T11:47:26.558 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000232 2024-04-25T11:47:26.558 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.558 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_recv_encrypted_bytes": 1617424, 2024-04-25T11:47:26.558 INFO:teuthology.orchestra.run.smithi138.stdout: "msgr_send_encrypted_bytes": 155520 2024-04-25T11:47:26.558 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.558 INFO:teuthology.orchestra.run.smithi138.stdout: "cct": { 2024-04-25T11:47:26.558 INFO:teuthology.orchestra.run.smithi138.stdout: "total_workers": 1, 2024-04-25T11:47:26.558 INFO:teuthology.orchestra.run.smithi138.stdout: "unhealthy_workers": 0 2024-04-25T11:47:26.558 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.559 INFO:teuthology.orchestra.run.smithi138.stdout: "finisher-MDSRank": { 2024-04-25T11:47:26.559 INFO:teuthology.orchestra.run.smithi138.stdout: "queue_len": 0, 2024-04-25T11:47:26.559 INFO:teuthology.orchestra.run.smithi138.stdout: "complete_latency": { 2024-04-25T11:47:26.559 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 33, 2024-04-25T11:47:26.559 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.001309060, 2024-04-25T11:47:26.559 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000039668 2024-04-25T11:47:26.559 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:26.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.559 INFO:teuthology.orchestra.run.smithi138.stdout: "finisher-PurgeQueue": { 2024-04-25T11:47:26.559 INFO:teuthology.orchestra.run.smithi138.stdout: "queue_len": 0, 2024-04-25T11:47:26.559 INFO:teuthology.orchestra.run.smithi138.stdout: "complete_latency": { 2024-04-25T11:47:26.559 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 2, 2024-04-25T11:47:26.559 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000006587, 2024-04-25T11:47:26.560 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000003293 2024-04-25T11:47:26.560 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:26.560 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.560 INFO:teuthology.orchestra.run.smithi138.stdout: "mds": { 2024-04-25T11:47:26.560 INFO:teuthology.orchestra.run.smithi138.stdout: "request": 0, 2024-04-25T11:47:26.560 INFO:teuthology.orchestra.run.smithi138.stdout: "reply": 0, 2024-04-25T11:47:26.560 INFO:teuthology.orchestra.run.smithi138.stdout: "reply_latency": { 2024-04-25T11:47:26.560 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.560 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.560 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.560 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.560 INFO:teuthology.orchestra.run.smithi138.stdout: "slow_reply": 0, 2024-04-25T11:47:26.560 INFO:teuthology.orchestra.run.smithi138.stdout: "forward": 0, 2024-04-25T11:47:26.560 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_fetch_complete": 0, 2024-04-25T11:47:26.561 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_fetch_keys": 0, 2024-04-25T11:47:26.561 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_commit": 11, 2024-04-25T11:47:26.561 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_split": 0, 2024-04-25T11:47:26.561 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_merge": 0, 2024-04-25T11:47:26.561 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes": 15, 2024-04-25T11:47:26.561 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes_top": 0, 2024-04-25T11:47:26.561 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes_bottom": 15, 2024-04-25T11:47:26.561 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes_pin_tail": 0, 2024-04-25T11:47:26.561 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes_pinned": 13, 2024-04-25T11:47:26.561 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes_expired": 8, 2024-04-25T11:47:26.561 INFO:teuthology.orchestra.run.smithi138.stdout: "inodes_with_caps": 0, 2024-04-25T11:47:26.561 INFO:teuthology.orchestra.run.smithi138.stdout: "caps": 0, 2024-04-25T11:47:26.561 INFO:teuthology.orchestra.run.smithi138.stdout: "subtrees": 3, 2024-04-25T11:47:26.562 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse": 7, 2024-04-25T11:47:26.562 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse_hit": 4, 2024-04-25T11:47:26.562 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse_forward": 0, 2024-04-25T11:47:26.562 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse_discover": 3, 2024-04-25T11:47:26.562 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse_dir_fetch": 0, 2024-04-25T11:47:26.562 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse_remote_ino": 0, 2024-04-25T11:47:26.562 INFO:teuthology.orchestra.run.smithi138.stdout: "traverse_lock": 0, 2024-04-25T11:47:26.562 INFO:teuthology.orchestra.run.smithi138.stdout: "load_cent": 0, 2024-04-25T11:47:26.562 INFO:teuthology.orchestra.run.smithi138.stdout: "q": 0, 2024-04-25T11:47:26.562 INFO:teuthology.orchestra.run.smithi138.stdout: "exported": 0, 2024-04-25T11:47:26.562 INFO:teuthology.orchestra.run.smithi138.stdout: "exported_inodes": 0, 2024-04-25T11:47:26.563 INFO:teuthology.orchestra.run.smithi138.stdout: "imported": 0, 2024-04-25T11:47:26.563 INFO:teuthology.orchestra.run.smithi138.stdout: "imported_inodes": 0, 2024-04-25T11:47:26.563 INFO:teuthology.orchestra.run.smithi138.stdout: "openino_dir_fetch": 0, 2024-04-25T11:47:26.563 INFO:teuthology.orchestra.run.smithi138.stdout: "openino_backtrace_fetch": 0, 2024-04-25T11:47:26.563 INFO:teuthology.orchestra.run.smithi138.stdout: "openino_peer_discover": 0, 2024-04-25T11:47:26.563 INFO:teuthology.orchestra.run.smithi138.stdout: "root_rfiles": 22, 2024-04-25T11:47:26.563 INFO:teuthology.orchestra.run.smithi138.stdout: "root_rbytes": 10738142561, 2024-04-25T11:47:26.563 INFO:teuthology.orchestra.run.smithi138.stdout: "root_rsnaps": 6, 2024-04-25T11:47:26.563 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T11:47:26.563 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_set_tag": 0, 2024-04-25T11:47:26.563 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T11:47:26.563 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_inotable_repaired": 0, 2024-04-25T11:47:26.564 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_dir_inodes": 0, 2024-04-25T11:47:26.564 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T11:47:26.564 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T11:47:26.564 INFO:teuthology.orchestra.run.smithi138.stdout: "scrub_file_inodes": 0, 2024-04-25T11:47:26.564 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_inode_file_caps": 0, 2024-04-25T11:47:26.564 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T11:47:26.564 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_cap_op_grant": 0, 2024-04-25T11:47:26.564 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T11:47:26.564 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T11:47:26.564 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T11:47:26.564 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_client_caps": 0, 2024-04-25T11:47:26.564 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_client_caps_dirty": 0, 2024-04-25T11:47:26.564 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_client_cap_release": 0, 2024-04-25T11:47:26.565 INFO:teuthology.orchestra.run.smithi138.stdout: "process_request_cap_release": 0 2024-04-25T11:47:26.565 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.565 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_cache": { 2024-04-25T11:47:26.565 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_update": 0, 2024-04-25T11:47:26.565 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_update_receipt": 9, 2024-04-25T11:47:26.565 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_try_discover": 6, 2024-04-25T11:47:26.565 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_send_discover": 8, 2024-04-25T11:47:26.565 INFO:teuthology.orchestra.run.smithi138.stdout: "dir_handle_discover": 0, 2024-04-25T11:47:26.565 INFO:teuthology.orchestra.run.smithi138.stdout: "num_strays": 0, 2024-04-25T11:47:26.565 INFO:teuthology.orchestra.run.smithi138.stdout: "num_strays_delayed": 0, 2024-04-25T11:47:26.565 INFO:teuthology.orchestra.run.smithi138.stdout: "num_strays_enqueuing": 0, 2024-04-25T11:47:26.565 INFO:teuthology.orchestra.run.smithi138.stdout: "strays_created": 0, 2024-04-25T11:47:26.565 INFO:teuthology.orchestra.run.smithi138.stdout: "strays_enqueued": 0, 2024-04-25T11:47:26.566 INFO:teuthology.orchestra.run.smithi138.stdout: "strays_reintegrated": 0, 2024-04-25T11:47:26.566 INFO:teuthology.orchestra.run.smithi138.stdout: "strays_migrated": 0, 2024-04-25T11:47:26.566 INFO:teuthology.orchestra.run.smithi138.stdout: "num_recovering_processing": 0, 2024-04-25T11:47:26.566 INFO:teuthology.orchestra.run.smithi138.stdout: "num_recovering_enqueued": 0, 2024-04-25T11:47:26.566 INFO:teuthology.orchestra.run.smithi138.stdout: "num_recovering_prioritized": 0, 2024-04-25T11:47:26.566 INFO:teuthology.orchestra.run.smithi138.stdout: "recovery_started": 0, 2024-04-25T11:47:26.566 INFO:teuthology.orchestra.run.smithi138.stdout: "recovery_completed": 0, 2024-04-25T11:47:26.566 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_quiesce_path": 0, 2024-04-25T11:47:26.566 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_quiesce_inode": 0, 2024-04-25T11:47:26.566 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T11:47:26.566 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_exportdir": 0, 2024-04-25T11:47:26.567 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_flush": 0, 2024-04-25T11:47:26.567 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_fragmentdir": 0, 2024-04-25T11:47:26.567 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_fragstats": 0, 2024-04-25T11:47:26.567 INFO:teuthology.orchestra.run.smithi138.stdout: "ireq_inodestats": 0 2024-04-25T11:47:26.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.567 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_log": { 2024-04-25T11:47:26.567 INFO:teuthology.orchestra.run.smithi138.stdout: "evlrg": 0, 2024-04-25T11:47:26.567 INFO:teuthology.orchestra.run.smithi138.stdout: "evadd": 3, 2024-04-25T11:47:26.567 INFO:teuthology.orchestra.run.smithi138.stdout: "evex": 0, 2024-04-25T11:47:26.567 INFO:teuthology.orchestra.run.smithi138.stdout: "evtrm": 0, 2024-04-25T11:47:26.567 INFO:teuthology.orchestra.run.smithi138.stdout: "ev": 3, 2024-04-25T11:47:26.567 INFO:teuthology.orchestra.run.smithi138.stdout: "evexg": 0, 2024-04-25T11:47:26.568 INFO:teuthology.orchestra.run.smithi138.stdout: "evexd": 0, 2024-04-25T11:47:26.568 INFO:teuthology.orchestra.run.smithi138.stdout: "segadd": 2, 2024-04-25T11:47:26.568 INFO:teuthology.orchestra.run.smithi138.stdout: "segex": 0, 2024-04-25T11:47:26.568 INFO:teuthology.orchestra.run.smithi138.stdout: "segtrm": 0, 2024-04-25T11:47:26.568 INFO:teuthology.orchestra.run.smithi138.stdout: "seg": 2, 2024-04-25T11:47:26.568 INFO:teuthology.orchestra.run.smithi138.stdout: "segmjr": 2, 2024-04-25T11:47:26.568 INFO:teuthology.orchestra.run.smithi138.stdout: "segexg": 0, 2024-04-25T11:47:26.568 INFO:teuthology.orchestra.run.smithi138.stdout: "segexd": 0, 2024-04-25T11:47:26.568 INFO:teuthology.orchestra.run.smithi138.stdout: "expos": 4194304, 2024-04-25T11:47:26.568 INFO:teuthology.orchestra.run.smithi138.stdout: "wrpos": 4195283, 2024-04-25T11:47:26.568 INFO:teuthology.orchestra.run.smithi138.stdout: "rdpos": 0, 2024-04-25T11:47:26.568 INFO:teuthology.orchestra.run.smithi138.stdout: "jlat": { 2024-04-25T11:47:26.569 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 2, 2024-04-25T11:47:26.569 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.003139700, 2024-04-25T11:47:26.569 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.001569850 2024-04-25T11:47:26.569 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.569 INFO:teuthology.orchestra.run.smithi138.stdout: "replayed": 0 2024-04-25T11:47:26.569 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.569 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_mem": { 2024-04-25T11:47:26.569 INFO:teuthology.orchestra.run.smithi138.stdout: "ino": 19, 2024-04-25T11:47:26.569 INFO:teuthology.orchestra.run.smithi138.stdout: "ino+": 23, 2024-04-25T11:47:26.569 INFO:teuthology.orchestra.run.smithi138.stdout: "ino-": 4, 2024-04-25T11:47:26.569 INFO:teuthology.orchestra.run.smithi138.stdout: "dir": 18, 2024-04-25T11:47:26.569 INFO:teuthology.orchestra.run.smithi138.stdout: "dir+": 22, 2024-04-25T11:47:26.570 INFO:teuthology.orchestra.run.smithi138.stdout: "dir-": 4, 2024-04-25T11:47:26.570 INFO:teuthology.orchestra.run.smithi138.stdout: "dn": 15, 2024-04-25T11:47:26.570 INFO:teuthology.orchestra.run.smithi138.stdout: "dn+": 23, 2024-04-25T11:47:26.570 INFO:teuthology.orchestra.run.smithi138.stdout: "dn-": 8, 2024-04-25T11:47:26.570 INFO:teuthology.orchestra.run.smithi138.stdout: "cap": 0, 2024-04-25T11:47:26.570 INFO:teuthology.orchestra.run.smithi138.stdout: "cap+": 0, 2024-04-25T11:47:26.570 INFO:teuthology.orchestra.run.smithi138.stdout: "cap-": 0, 2024-04-25T11:47:26.570 INFO:teuthology.orchestra.run.smithi138.stdout: "rss": 121116, 2024-04-25T11:47:26.570 INFO:teuthology.orchestra.run.smithi138.stdout: "heap": 223632 2024-04-25T11:47:26.570 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.570 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_server": { 2024-04-25T11:47:26.570 INFO:teuthology.orchestra.run.smithi138.stdout: "dispatch_client_request": 0, 2024-04-25T11:47:26.571 INFO:teuthology.orchestra.run.smithi138.stdout: "dispatch_server_request": 1, 2024-04-25T11:47:26.571 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_client_request": 0, 2024-04-25T11:47:26.571 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_client_session": 9, 2024-04-25T11:47:26.571 INFO:teuthology.orchestra.run.smithi138.stdout: "handle_peer_request": 2, 2024-04-25T11:47:26.571 INFO:teuthology.orchestra.run.smithi138.stdout: "req_create_latency": { 2024-04-25T11:47:26.571 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.571 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.571 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.572 INFO:teuthology.orchestra.run.smithi138.stdout: "req_getattr_latency": { 2024-04-25T11:47:26.572 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.572 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.572 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.572 INFO:teuthology.orchestra.run.smithi138.stdout: "req_getfilelock_latency": { 2024-04-25T11:47:26.572 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.572 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.572 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.573 INFO:teuthology.orchestra.run.smithi138.stdout: "req_link_latency": { 2024-04-25T11:47:26.573 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.573 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.573 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.573 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lookup_latency": { 2024-04-25T11:47:26.573 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.573 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.573 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.573 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lookuphash_latency": { 2024-04-25T11:47:26.574 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.574 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.574 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.574 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.574 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lookupino_latency": { 2024-04-25T11:47:26.574 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.574 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.574 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.574 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.574 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lookupname_latency": { 2024-04-25T11:47:26.574 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.575 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.575 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.575 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lookupparent_latency": { 2024-04-25T11:47:26.575 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.575 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.575 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.575 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lookupsnap_latency": { 2024-04-25T11:47:26.575 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.575 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.575 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.576 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.576 INFO:teuthology.orchestra.run.smithi138.stdout: "req_lssnap_latency": { 2024-04-25T11:47:26.576 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.576 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.576 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.576 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.576 INFO:teuthology.orchestra.run.smithi138.stdout: "req_mkdir_latency": { 2024-04-25T11:47:26.576 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.576 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.576 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.576 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.577 INFO:teuthology.orchestra.run.smithi138.stdout: "req_mknod_latency": { 2024-04-25T11:47:26.577 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.577 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.577 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.577 INFO:teuthology.orchestra.run.smithi138.stdout: "req_mksnap_latency": { 2024-04-25T11:47:26.577 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.577 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.577 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.577 INFO:teuthology.orchestra.run.smithi138.stdout: "req_open_latency": { 2024-04-25T11:47:26.578 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.578 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.578 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.578 INFO:teuthology.orchestra.run.smithi138.stdout: "req_readdir_latency": { 2024-04-25T11:47:26.578 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.578 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.578 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.578 INFO:teuthology.orchestra.run.smithi138.stdout: "req_rename_latency": { 2024-04-25T11:47:26.578 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.579 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.579 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.579 INFO:teuthology.orchestra.run.smithi138.stdout: "req_renamesnap_latency": { 2024-04-25T11:47:26.579 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.579 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.579 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.579 INFO:teuthology.orchestra.run.smithi138.stdout: "req_snapdiff_latency": { 2024-04-25T11:47:26.579 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.579 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.580 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.580 INFO:teuthology.orchestra.run.smithi138.stdout: "req_rmdir_latency": { 2024-04-25T11:47:26.580 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.580 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.580 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.580 INFO:teuthology.orchestra.run.smithi138.stdout: "req_rmsnap_latency": { 2024-04-25T11:47:26.580 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.580 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.580 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.581 INFO:teuthology.orchestra.run.smithi138.stdout: "req_rmxattr_latency": { 2024-04-25T11:47:26.581 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.581 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.581 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.581 INFO:teuthology.orchestra.run.smithi138.stdout: "req_setattr_latency": { 2024-04-25T11:47:26.581 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.581 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.581 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.581 INFO:teuthology.orchestra.run.smithi138.stdout: "req_setdirlayout_latency": { 2024-04-25T11:47:26.582 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.582 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.582 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.582 INFO:teuthology.orchestra.run.smithi138.stdout: "req_setfilelock_latency": { 2024-04-25T11:47:26.582 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.582 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.582 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.582 INFO:teuthology.orchestra.run.smithi138.stdout: "req_setlayout_latency": { 2024-04-25T11:47:26.582 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.583 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.583 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.583 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.583 INFO:teuthology.orchestra.run.smithi138.stdout: "req_setxattr_latency": { 2024-04-25T11:47:26.583 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.583 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.583 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.583 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.583 INFO:teuthology.orchestra.run.smithi138.stdout: "req_symlink_latency": { 2024-04-25T11:47:26.583 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.583 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.583 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.584 INFO:teuthology.orchestra.run.smithi138.stdout: "req_unlink_latency": { 2024-04-25T11:47:26.584 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.584 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.584 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.584 INFO:teuthology.orchestra.run.smithi138.stdout: "cap_revoke_eviction": 0, 2024-04-25T11:47:26.584 INFO:teuthology.orchestra.run.smithi138.stdout: "cap_acquisition_throttle": 0, 2024-04-25T11:47:26.584 INFO:teuthology.orchestra.run.smithi138.stdout: "req_getvxattr_latency": { 2024-04-25T11:47:26.584 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.584 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.584 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.585 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:26.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.585 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_sessions": { 2024-04-25T11:47:26.585 INFO:teuthology.orchestra.run.smithi138.stdout: "session_count": 1, 2024-04-25T11:47:26.585 INFO:teuthology.orchestra.run.smithi138.stdout: "session_add": 1, 2024-04-25T11:47:26.585 INFO:teuthology.orchestra.run.smithi138.stdout: "session_remove": 0, 2024-04-25T11:47:26.585 INFO:teuthology.orchestra.run.smithi138.stdout: "sessions_open": 1, 2024-04-25T11:47:26.585 INFO:teuthology.orchestra.run.smithi138.stdout: "sessions_stale": 0, 2024-04-25T11:47:26.585 INFO:teuthology.orchestra.run.smithi138.stdout: "total_load": 0, 2024-04-25T11:47:26.585 INFO:teuthology.orchestra.run.smithi138.stdout: "average_load": 0, 2024-04-25T11:47:26.585 INFO:teuthology.orchestra.run.smithi138.stdout: "avg_session_uptime": 682, 2024-04-25T11:47:26.586 INFO:teuthology.orchestra.run.smithi138.stdout: "mdthresh_evicted": 0 2024-04-25T11:47:26.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.586 INFO:teuthology.orchestra.run.smithi138.stdout: "mempool": { 2024-04-25T11:47:26.586 INFO:teuthology.orchestra.run.smithi138.stdout: "bloom_filter_bytes": 0, 2024-04-25T11:47:26.586 INFO:teuthology.orchestra.run.smithi138.stdout: "bloom_filter_items": 0, 2024-04-25T11:47:26.586 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T11:47:26.586 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_alloc_items": 0, 2024-04-25T11:47:26.586 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T11:47:26.586 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_data_items": 0, 2024-04-25T11:47:26.586 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T11:47:26.586 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T11:47:26.587 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T11:47:26.587 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T11:47:26.587 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T11:47:26.587 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_other_items": 0, 2024-04-25T11:47:26.587 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T11:47:26.587 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T11:47:26.587 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_extent_bytes": 0, 2024-04-25T11:47:26.587 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_extent_items": 0, 2024-04-25T11:47:26.587 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_blob_bytes": 0, 2024-04-25T11:47:26.587 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_blob_items": 0, 2024-04-25T11:47:26.587 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T11:47:26.588 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T11:47:26.588 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T11:47:26.588 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T11:47:26.588 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T11:47:26.588 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_fsck_items": 0, 2024-04-25T11:47:26.588 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_txc_bytes": 0, 2024-04-25T11:47:26.588 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_txc_items": 0, 2024-04-25T11:47:26.588 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T11:47:26.588 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T11:47:26.588 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_writing_bytes": 0, 2024-04-25T11:47:26.588 INFO:teuthology.orchestra.run.smithi138.stdout: "bluestore_writing_items": 0, 2024-04-25T11:47:26.589 INFO:teuthology.orchestra.run.smithi138.stdout: "bluefs_bytes": 0, 2024-04-25T11:47:26.589 INFO:teuthology.orchestra.run.smithi138.stdout: "bluefs_items": 0, 2024-04-25T11:47:26.589 INFO:teuthology.orchestra.run.smithi138.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T11:47:26.589 INFO:teuthology.orchestra.run.smithi138.stdout: "bluefs_file_reader_items": 0, 2024-04-25T11:47:26.589 INFO:teuthology.orchestra.run.smithi138.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T11:47:26.589 INFO:teuthology.orchestra.run.smithi138.stdout: "bluefs_file_writer_items": 0, 2024-04-25T11:47:26.589 INFO:teuthology.orchestra.run.smithi138.stdout: "buffer_anon_bytes": 270487, 2024-04-25T11:47:26.589 INFO:teuthology.orchestra.run.smithi138.stdout: "buffer_anon_items": 91, 2024-04-25T11:47:26.589 INFO:teuthology.orchestra.run.smithi138.stdout: "buffer_meta_bytes": 0, 2024-04-25T11:47:26.589 INFO:teuthology.orchestra.run.smithi138.stdout: "buffer_meta_items": 0, 2024-04-25T11:47:26.589 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_bytes": 0, 2024-04-25T11:47:26.590 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_items": 0, 2024-04-25T11:47:26.590 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_mapbl_bytes": 0, 2024-04-25T11:47:26.590 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_mapbl_items": 0, 2024-04-25T11:47:26.590 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_pglog_bytes": 0, 2024-04-25T11:47:26.590 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_pglog_items": 0, 2024-04-25T11:47:26.590 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap_bytes": 8632, 2024-04-25T11:47:26.590 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap_items": 180, 2024-04-25T11:47:26.590 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T11:47:26.590 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap_mapping_items": 0, 2024-04-25T11:47:26.590 INFO:teuthology.orchestra.run.smithi138.stdout: "pgmap_bytes": 0, 2024-04-25T11:47:26.590 INFO:teuthology.orchestra.run.smithi138.stdout: "pgmap_items": 0, 2024-04-25T11:47:26.591 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_co_bytes": 78397, 2024-04-25T11:47:26.591 INFO:teuthology.orchestra.run.smithi138.stdout: "mds_co_items": 311, 2024-04-25T11:47:26.591 INFO:teuthology.orchestra.run.smithi138.stdout: "unittest_1_bytes": 0, 2024-04-25T11:47:26.591 INFO:teuthology.orchestra.run.smithi138.stdout: "unittest_1_items": 0, 2024-04-25T11:47:26.591 INFO:teuthology.orchestra.run.smithi138.stdout: "unittest_2_bytes": 0, 2024-04-25T11:47:26.591 INFO:teuthology.orchestra.run.smithi138.stdout: "unittest_2_items": 0 2024-04-25T11:47:26.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.591 INFO:teuthology.orchestra.run.smithi138.stdout: "objecter": { 2024-04-25T11:47:26.591 INFO:teuthology.orchestra.run.smithi138.stdout: "op_active": 0, 2024-04-25T11:47:26.591 INFO:teuthology.orchestra.run.smithi138.stdout: "op_laggy": 0, 2024-04-25T11:47:26.591 INFO:teuthology.orchestra.run.smithi138.stdout: "op_send": 37, 2024-04-25T11:47:26.591 INFO:teuthology.orchestra.run.smithi138.stdout: "op_send_bytes": 11191, 2024-04-25T11:47:26.592 INFO:teuthology.orchestra.run.smithi138.stdout: "op_resend": 33, 2024-04-25T11:47:26.592 INFO:teuthology.orchestra.run.smithi138.stdout: "op_reply": 37, 2024-04-25T11:47:26.592 INFO:teuthology.orchestra.run.smithi138.stdout: "op_latency": { 2024-04-25T11:47:26.592 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 37, 2024-04-25T11:47:26.592 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.218995134, 2024-04-25T11:47:26.592 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.005918787 2024-04-25T11:47:26.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.592 INFO:teuthology.orchestra.run.smithi138.stdout: "op_inflight": 0, 2024-04-25T11:47:26.592 INFO:teuthology.orchestra.run.smithi138.stdout: "oplen_avg": { 2024-04-25T11:47:26.592 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 37, 2024-04-25T11:47:26.592 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 59 2024-04-25T11:47:26.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.593 INFO:teuthology.orchestra.run.smithi138.stdout: "op": 37, 2024-04-25T11:47:26.593 INFO:teuthology.orchestra.run.smithi138.stdout: "op_r": 0, 2024-04-25T11:47:26.593 INFO:teuthology.orchestra.run.smithi138.stdout: "op_w": 37, 2024-04-25T11:47:26.593 INFO:teuthology.orchestra.run.smithi138.stdout: "op_rmw": 0, 2024-04-25T11:47:26.593 INFO:teuthology.orchestra.run.smithi138.stdout: "op_pg": 0, 2024-04-25T11:47:26.593 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_stat": 0, 2024-04-25T11:47:26.593 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_create": 10, 2024-04-25T11:47:26.593 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_read": 0, 2024-04-25T11:47:26.593 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_write": 2, 2024-04-25T11:47:26.593 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_writefull": 6, 2024-04-25T11:47:26.593 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_writesame": 0, 2024-04-25T11:47:26.593 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_append": 0, 2024-04-25T11:47:26.594 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_zero": 0, 2024-04-25T11:47:26.594 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_truncate": 0, 2024-04-25T11:47:26.594 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_delete": 6, 2024-04-25T11:47:26.594 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_mapext": 0, 2024-04-25T11:47:26.594 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_sparse_read": 0, 2024-04-25T11:47:26.594 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_clonerange": 0, 2024-04-25T11:47:26.594 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_getxattr": 0, 2024-04-25T11:47:26.594 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_setxattr": 20, 2024-04-25T11:47:26.594 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_cmpxattr": 0, 2024-04-25T11:47:26.594 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_rmxattr": 0, 2024-04-25T11:47:26.594 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_resetxattrs": 0, 2024-04-25T11:47:26.595 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_call": 0, 2024-04-25T11:47:26.595 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_watch": 0, 2024-04-25T11:47:26.595 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_notify": 0, 2024-04-25T11:47:26.595 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T11:47:26.595 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_pgls": 0, 2024-04-25T11:47:26.595 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_pgls_filter": 0, 2024-04-25T11:47:26.595 INFO:teuthology.orchestra.run.smithi138.stdout: "osdop_other": 0, 2024-04-25T11:47:26.595 INFO:teuthology.orchestra.run.smithi138.stdout: "linger_active": 0, 2024-04-25T11:47:26.595 INFO:teuthology.orchestra.run.smithi138.stdout: "linger_send": 0, 2024-04-25T11:47:26.595 INFO:teuthology.orchestra.run.smithi138.stdout: "linger_resend": 0, 2024-04-25T11:47:26.595 INFO:teuthology.orchestra.run.smithi138.stdout: "linger_ping": 0, 2024-04-25T11:47:26.595 INFO:teuthology.orchestra.run.smithi138.stdout: "poolop_active": 0, 2024-04-25T11:47:26.596 INFO:teuthology.orchestra.run.smithi138.stdout: "poolop_send": 0, 2024-04-25T11:47:26.596 INFO:teuthology.orchestra.run.smithi138.stdout: "poolop_resend": 0, 2024-04-25T11:47:26.596 INFO:teuthology.orchestra.run.smithi138.stdout: "poolstat_active": 0, 2024-04-25T11:47:26.596 INFO:teuthology.orchestra.run.smithi138.stdout: "poolstat_send": 0, 2024-04-25T11:47:26.596 INFO:teuthology.orchestra.run.smithi138.stdout: "poolstat_resend": 0, 2024-04-25T11:47:26.596 INFO:teuthology.orchestra.run.smithi138.stdout: "statfs_active": 0, 2024-04-25T11:47:26.596 INFO:teuthology.orchestra.run.smithi138.stdout: "statfs_send": 0, 2024-04-25T11:47:26.596 INFO:teuthology.orchestra.run.smithi138.stdout: "statfs_resend": 0, 2024-04-25T11:47:26.596 INFO:teuthology.orchestra.run.smithi138.stdout: "command_active": 0, 2024-04-25T11:47:26.596 INFO:teuthology.orchestra.run.smithi138.stdout: "command_send": 0, 2024-04-25T11:47:26.596 INFO:teuthology.orchestra.run.smithi138.stdout: "command_resend": 0, 2024-04-25T11:47:26.596 INFO:teuthology.orchestra.run.smithi138.stdout: "map_epoch": 93, 2024-04-25T11:47:26.596 INFO:teuthology.orchestra.run.smithi138.stdout: "map_full": 0, 2024-04-25T11:47:26.597 INFO:teuthology.orchestra.run.smithi138.stdout: "map_inc": 16, 2024-04-25T11:47:26.597 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_sessions": 9, 2024-04-25T11:47:26.597 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_session_open": 9, 2024-04-25T11:47:26.597 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_session_close": 0, 2024-04-25T11:47:26.597 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_laggy": 0, 2024-04-25T11:47:26.597 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_wr": 14, 2024-04-25T11:47:26.597 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_rd": 0, 2024-04-25T11:47:26.597 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_del": 1 2024-04-25T11:47:26.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.597 INFO:teuthology.orchestra.run.smithi138.stdout: "oft": { 2024-04-25T11:47:26.598 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_total_objs": 1, 2024-04-25T11:47:26.598 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_total_kv_pairs": 0, 2024-04-25T11:47:26.598 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_total_updates": 0, 2024-04-25T11:47:26.598 INFO:teuthology.orchestra.run.smithi138.stdout: "omap_total_removes": 0 2024-04-25T11:47:26.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.598 INFO:teuthology.orchestra.run.smithi138.stdout: "osd-slow-ops": { 2024-04-25T11:47:26.598 INFO:teuthology.orchestra.run.smithi138.stdout: "slow_ops_count": 0 2024-04-25T11:47:26.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.598 INFO:teuthology.orchestra.run.smithi138.stdout: "purge_queue": { 2024-04-25T11:47:26.598 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_executing_ops": 0, 2024-04-25T11:47:26.599 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T11:47:26.599 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_executing": 0, 2024-04-25T11:47:26.599 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_executing_high_water": 0, 2024-04-25T11:47:26.599 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_executed_ops": 0, 2024-04-25T11:47:26.599 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_executed": 0, 2024-04-25T11:47:26.599 INFO:teuthology.orchestra.run.smithi138.stdout: "pq_item_in_journal": 0 2024-04-25T11:47:26.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.599 INFO:teuthology.orchestra.run.smithi138.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T11:47:26.599 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 0, 2024-04-25T11:47:26.599 INFO:teuthology.orchestra.run.smithi138.stdout: "max": 104857600, 2024-04-25T11:47:26.599 INFO:teuthology.orchestra.run.smithi138.stdout: "get_started": 0, 2024-04-25T11:47:26.599 INFO:teuthology.orchestra.run.smithi138.stdout: "get": 1907, 2024-04-25T11:47:26.600 INFO:teuthology.orchestra.run.smithi138.stdout: "get_sum": 1962941, 2024-04-25T11:47:26.600 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:26.600 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_success": 1907, 2024-04-25T11:47:26.600 INFO:teuthology.orchestra.run.smithi138.stdout: "take": 0, 2024-04-25T11:47:26.600 INFO:teuthology.orchestra.run.smithi138.stdout: "take_sum": 0, 2024-04-25T11:47:26.600 INFO:teuthology.orchestra.run.smithi138.stdout: "put": 1907, 2024-04-25T11:47:26.600 INFO:teuthology.orchestra.run.smithi138.stdout: "put_sum": 1962941, 2024-04-25T11:47:26.600 INFO:teuthology.orchestra.run.smithi138.stdout: "wait": { 2024-04-25T11:47:26.600 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.600 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.600 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.600 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:26.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.601 INFO:teuthology.orchestra.run.smithi138.stdout: "throttle-objecter_bytes": { 2024-04-25T11:47:26.601 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 0, 2024-04-25T11:47:26.601 INFO:teuthology.orchestra.run.smithi138.stdout: "max": 104857600, 2024-04-25T11:47:26.601 INFO:teuthology.orchestra.run.smithi138.stdout: "get_started": 0, 2024-04-25T11:47:26.601 INFO:teuthology.orchestra.run.smithi138.stdout: "get": 0, 2024-04-25T11:47:26.601 INFO:teuthology.orchestra.run.smithi138.stdout: "get_sum": 0, 2024-04-25T11:47:26.601 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:26.601 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:26.601 INFO:teuthology.orchestra.run.smithi138.stdout: "take": 37, 2024-04-25T11:47:26.601 INFO:teuthology.orchestra.run.smithi138.stdout: "take_sum": 11191, 2024-04-25T11:47:26.601 INFO:teuthology.orchestra.run.smithi138.stdout: "put": 37, 2024-04-25T11:47:26.601 INFO:teuthology.orchestra.run.smithi138.stdout: "put_sum": 11191, 2024-04-25T11:47:26.602 INFO:teuthology.orchestra.run.smithi138.stdout: "wait": { 2024-04-25T11:47:26.602 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.602 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.602 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.602 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:26.602 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.602 INFO:teuthology.orchestra.run.smithi138.stdout: "throttle-objecter_ops": { 2024-04-25T11:47:26.602 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 0, 2024-04-25T11:47:26.602 INFO:teuthology.orchestra.run.smithi138.stdout: "max": 1024, 2024-04-25T11:47:26.602 INFO:teuthology.orchestra.run.smithi138.stdout: "get_started": 0, 2024-04-25T11:47:26.603 INFO:teuthology.orchestra.run.smithi138.stdout: "get": 0, 2024-04-25T11:47:26.603 INFO:teuthology.orchestra.run.smithi138.stdout: "get_sum": 0, 2024-04-25T11:47:26.603 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:26.603 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:26.603 INFO:teuthology.orchestra.run.smithi138.stdout: "take": 37, 2024-04-25T11:47:26.603 INFO:teuthology.orchestra.run.smithi138.stdout: "take_sum": 37, 2024-04-25T11:47:26.603 INFO:teuthology.orchestra.run.smithi138.stdout: "put": 37, 2024-04-25T11:47:26.603 INFO:teuthology.orchestra.run.smithi138.stdout: "put_sum": 37, 2024-04-25T11:47:26.603 INFO:teuthology.orchestra.run.smithi138.stdout: "wait": { 2024-04-25T11:47:26.603 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.603 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.603 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.604 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:26.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.604 INFO:teuthology.orchestra.run.smithi138.stdout: "throttle-write_buf_throttle": { 2024-04-25T11:47:26.604 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 0, 2024-04-25T11:47:26.604 INFO:teuthology.orchestra.run.smithi138.stdout: "max": 3758096384, 2024-04-25T11:47:26.604 INFO:teuthology.orchestra.run.smithi138.stdout: "get_started": 0, 2024-04-25T11:47:26.604 INFO:teuthology.orchestra.run.smithi138.stdout: "get": 0, 2024-04-25T11:47:26.604 INFO:teuthology.orchestra.run.smithi138.stdout: "get_sum": 0, 2024-04-25T11:47:26.604 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:26.604 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:26.604 INFO:teuthology.orchestra.run.smithi138.stdout: "take": 0, 2024-04-25T11:47:26.604 INFO:teuthology.orchestra.run.smithi138.stdout: "take_sum": 0, 2024-04-25T11:47:26.604 INFO:teuthology.orchestra.run.smithi138.stdout: "put": 0, 2024-04-25T11:47:26.604 INFO:teuthology.orchestra.run.smithi138.stdout: "put_sum": 0, 2024-04-25T11:47:26.605 INFO:teuthology.orchestra.run.smithi138.stdout: "wait": { 2024-04-25T11:47:26.605 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.605 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.605 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.605 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:26.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-25T11:47:26.605 INFO:teuthology.orchestra.run.smithi138.stdout: "throttle-write_buf_throttle-0x56500b2b85b0": { 2024-04-25T11:47:26.605 INFO:teuthology.orchestra.run.smithi138.stdout: "val": 0, 2024-04-25T11:47:26.605 INFO:teuthology.orchestra.run.smithi138.stdout: "max": 3758096384, 2024-04-25T11:47:26.605 INFO:teuthology.orchestra.run.smithi138.stdout: "get_started": 0, 2024-04-25T11:47:26.605 INFO:teuthology.orchestra.run.smithi138.stdout: "get": 3, 2024-04-25T11:47:26.605 INFO:teuthology.orchestra.run.smithi138.stdout: "get_sum": 979, 2024-04-25T11:47:26.605 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:26.605 INFO:teuthology.orchestra.run.smithi138.stdout: "get_or_fail_success": 3, 2024-04-25T11:47:26.606 INFO:teuthology.orchestra.run.smithi138.stdout: "take": 0, 2024-04-25T11:47:26.606 INFO:teuthology.orchestra.run.smithi138.stdout: "take_sum": 0, 2024-04-25T11:47:26.606 INFO:teuthology.orchestra.run.smithi138.stdout: "put": 2, 2024-04-25T11:47:26.606 INFO:teuthology.orchestra.run.smithi138.stdout: "put_sum": 979, 2024-04-25T11:47:26.606 INFO:teuthology.orchestra.run.smithi138.stdout: "wait": { 2024-04-25T11:47:26.606 INFO:teuthology.orchestra.run.smithi138.stdout: "avgcount": 0, 2024-04-25T11:47:26.606 INFO:teuthology.orchestra.run.smithi138.stdout: "sum": 0.000000000, 2024-04-25T11:47:26.606 INFO:teuthology.orchestra.run.smithi138.stdout: "avgtime": 0.000000000 2024-04-25T11:47:26.606 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:26.606 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-25T11:47:26.606 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-25T11:47:27.227 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-04-25T11:47:27.228 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-04-25T11:47:27.228 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update=0 2024-04-25T11:47:27.228 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update_receipt=9 2024-04-25T11:47:27.228 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=6 2024-04-25T11:47:27.228 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-04-25T11:47:27.228 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T11:47:27.228 DEBUG:tasks.check_counter:Getting stats from c 2024-04-25T11:47:27.228 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-25T11:47:27.477 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T11:47:28.180 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:27 smithi156 ceph-mon[33045]: pgmap v385: 97 pgs: 6 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 75 active+clean; 11 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T11:47:28.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:27 smithi090 ceph-mon[29575]: pgmap v385: 97 pgs: 6 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 75 active+clean; 11 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T11:47:28.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:27 smithi138 ceph-mon[32328]: pgmap v385: 97 pgs: 6 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 75 active+clean; 11 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T11:47:29.150 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:28 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:47:29.150 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:28 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:47:29.150 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:28 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:47:29.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:28 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:47:29.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:28 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:47:29.222 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:28 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:47:29.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:28 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T11:47:29.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:28 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:47:29.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:28 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:47:29.674 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-04-25T11:47:29.674 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T11:47:29.675 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 2447, 2024-04-25T11:47:29.675 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 2449, 2024-04-25T11:47:29.675 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 726266, 2024-04-25T11:47:29.675 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 778153, 2024-04-25T11:47:29.675 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 6, 2024-04-25T11:47:29.675 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 2, 2024-04-25T11:47:29.675 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.231044020, 2024-04-25T11:47:29.675 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.070041128, 2024-04-25T11:47:29.675 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.063215695, 2024-04-25T11:47:29.675 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.080527756, 2024-04-25T11:47:29.675 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:29.675 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2447, 2024-04-25T11:47:29.675 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.088801282, 2024-04-25T11:47:29.675 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000036289 2024-04-25T11:47:29.675 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.676 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:29.676 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.676 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.676 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.676 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.676 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T11:47:29.676 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T11:47:29.676 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.676 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T11:47:29.676 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 279, 2024-04-25T11:47:29.676 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 258, 2024-04-25T11:47:29.676 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 1628796, 2024-04-25T11:47:29.676 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 149050, 2024-04-25T11:47:29.676 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 3, 2024-04-25T11:47:29.677 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 2, 2024-04-25T11:47:29.677 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.055300656, 2024-04-25T11:47:29.677 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.015737810, 2024-04-25T11:47:29.677 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.044942216, 2024-04-25T11:47:29.677 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.006508539, 2024-04-25T11:47:29.677 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:29.677 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 257, 2024-04-25T11:47:29.677 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.012308103, 2024-04-25T11:47:29.677 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000047891 2024-04-25T11:47:29.677 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.677 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:29.677 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.677 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.677 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.678 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.678 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 1628240, 2024-04-25T11:47:29.678 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 148352 2024-04-25T11:47:29.678 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.678 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T11:47:29.678 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 7, 2024-04-25T11:47:29.678 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 183, 2024-04-25T11:47:29.678 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 3580, 2024-04-25T11:47:29.678 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 482830, 2024-04-25T11:47:29.678 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 6, 2024-04-25T11:47:29.678 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 4, 2024-04-25T11:47:29.678 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.011640345, 2024-04-25T11:47:29.678 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.006900510, 2024-04-25T11:47:29.678 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.002816899, 2024-04-25T11:47:29.679 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.000026063, 2024-04-25T11:47:29.679 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:29.679 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 181, 2024-04-25T11:47:29.679 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.023766232, 2024-04-25T11:47:29.679 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000131305 2024-04-25T11:47:29.679 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.679 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:29.679 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.679 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.679 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.679 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.679 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-25T11:47:29.679 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 482512 2024-04-25T11:47:29.680 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.680 INFO:teuthology.orchestra.run.smithi156.stdout: "cct": { 2024-04-25T11:47:29.680 INFO:teuthology.orchestra.run.smithi156.stdout: "total_workers": 1, 2024-04-25T11:47:29.680 INFO:teuthology.orchestra.run.smithi156.stdout: "unhealthy_workers": 0 2024-04-25T11:47:29.680 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.680 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-MDSRank": { 2024-04-25T11:47:29.680 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-04-25T11:47:29.680 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-04-25T11:47:29.680 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 4144, 2024-04-25T11:47:29.680 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.125549991, 2024-04-25T11:47:29.680 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000030296 2024-04-25T11:47:29.680 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:29.680 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.680 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-PurgeQueue": { 2024-04-25T11:47:29.680 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-04-25T11:47:29.681 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-04-25T11:47:29.681 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.681 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.681 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.681 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:29.681 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.681 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": { 2024-04-25T11:47:29.681 INFO:teuthology.orchestra.run.smithi156.stdout: "request": 0, 2024-04-25T11:47:29.681 INFO:teuthology.orchestra.run.smithi156.stdout: "reply": 0, 2024-04-25T11:47:29.681 INFO:teuthology.orchestra.run.smithi156.stdout: "reply_latency": { 2024-04-25T11:47:29.681 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.681 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.681 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.681 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.681 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_reply": 0, 2024-04-25T11:47:29.682 INFO:teuthology.orchestra.run.smithi156.stdout: "forward": 0, 2024-04-25T11:47:29.682 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_complete": 0, 2024-04-25T11:47:29.682 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_keys": 0, 2024-04-25T11:47:29.682 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_commit": 0, 2024-04-25T11:47:29.682 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_split": 0, 2024-04-25T11:47:29.682 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_merge": 0, 2024-04-25T11:47:29.682 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes": 10, 2024-04-25T11:47:29.682 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_top": 0, 2024-04-25T11:47:29.682 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_bottom": 10, 2024-04-25T11:47:29.682 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pin_tail": 0, 2024-04-25T11:47:29.682 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pinned": 10, 2024-04-25T11:47:29.682 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_expired": 0, 2024-04-25T11:47:29.683 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_with_caps": 0, 2024-04-25T11:47:29.683 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": 0, 2024-04-25T11:47:29.683 INFO:teuthology.orchestra.run.smithi156.stdout: "subtrees": 3, 2024-04-25T11:47:29.683 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse": 0, 2024-04-25T11:47:29.683 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_hit": 0, 2024-04-25T11:47:29.683 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_forward": 0, 2024-04-25T11:47:29.683 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_discover": 0, 2024-04-25T11:47:29.683 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_dir_fetch": 0, 2024-04-25T11:47:29.683 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_remote_ino": 0, 2024-04-25T11:47:29.683 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_lock": 0, 2024-04-25T11:47:29.683 INFO:teuthology.orchestra.run.smithi156.stdout: "load_cent": 0, 2024-04-25T11:47:29.683 INFO:teuthology.orchestra.run.smithi156.stdout: "q": 0, 2024-04-25T11:47:29.683 INFO:teuthology.orchestra.run.smithi156.stdout: "exported": 0, 2024-04-25T11:47:29.684 INFO:teuthology.orchestra.run.smithi156.stdout: "exported_inodes": 0, 2024-04-25T11:47:29.684 INFO:teuthology.orchestra.run.smithi156.stdout: "imported": 0, 2024-04-25T11:47:29.684 INFO:teuthology.orchestra.run.smithi156.stdout: "imported_inodes": 0, 2024-04-25T11:47:29.684 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_dir_fetch": 0, 2024-04-25T11:47:29.684 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_backtrace_fetch": 0, 2024-04-25T11:47:29.684 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_peer_discover": 0, 2024-04-25T11:47:29.684 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rfiles": 0, 2024-04-25T11:47:29.684 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rbytes": 0, 2024-04-25T11:47:29.684 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rsnaps": 0, 2024-04-25T11:47:29.684 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T11:47:29.684 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_set_tag": 0, 2024-04-25T11:47:29.684 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T11:47:29.684 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_inotable_repaired": 0, 2024-04-25T11:47:29.685 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_inodes": 0, 2024-04-25T11:47:29.685 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T11:47:29.685 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T11:47:29.685 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_file_inodes": 0, 2024-04-25T11:47:29.685 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_inode_file_caps": 0, 2024-04-25T11:47:29.685 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T11:47:29.685 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_grant": 0, 2024-04-25T11:47:29.685 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T11:47:29.685 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T11:47:29.685 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T11:47:29.685 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps": 0, 2024-04-25T11:47:29.685 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps_dirty": 0, 2024-04-25T11:47:29.685 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_cap_release": 0, 2024-04-25T11:47:29.685 INFO:teuthology.orchestra.run.smithi156.stdout: "process_request_cap_release": 0 2024-04-25T11:47:29.685 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.686 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_cache": { 2024-04-25T11:47:29.686 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update": 0, 2024-04-25T11:47:29.686 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update_receipt": 0, 2024-04-25T11:47:29.686 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_try_discover": 0, 2024-04-25T11:47:29.686 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_send_discover": 0, 2024-04-25T11:47:29.686 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_handle_discover": 0, 2024-04-25T11:47:29.686 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays": 0, 2024-04-25T11:47:29.686 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_delayed": 0, 2024-04-25T11:47:29.686 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_enqueuing": 0, 2024-04-25T11:47:29.686 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_created": 0, 2024-04-25T11:47:29.687 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_enqueued": 0, 2024-04-25T11:47:29.687 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_reintegrated": 0, 2024-04-25T11:47:29.687 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_migrated": 0, 2024-04-25T11:47:29.687 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_processing": 0, 2024-04-25T11:47:29.687 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_enqueued": 0, 2024-04-25T11:47:29.687 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_prioritized": 0, 2024-04-25T11:47:29.687 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_started": 0, 2024-04-25T11:47:29.687 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_completed": 0, 2024-04-25T11:47:29.687 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_path": 0, 2024-04-25T11:47:29.687 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_inode": 0, 2024-04-25T11:47:29.687 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T11:47:29.687 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_exportdir": 0, 2024-04-25T11:47:29.687 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_flush": 0, 2024-04-25T11:47:29.687 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragmentdir": 0, 2024-04-25T11:47:29.688 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragstats": 0, 2024-04-25T11:47:29.688 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_inodestats": 0 2024-04-25T11:47:29.688 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.688 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_log": { 2024-04-25T11:47:29.688 INFO:teuthology.orchestra.run.smithi156.stdout: "evlrg": 0, 2024-04-25T11:47:29.688 INFO:teuthology.orchestra.run.smithi156.stdout: "evadd": 0, 2024-04-25T11:47:29.688 INFO:teuthology.orchestra.run.smithi156.stdout: "evex": 0, 2024-04-25T11:47:29.688 INFO:teuthology.orchestra.run.smithi156.stdout: "evtrm": 0, 2024-04-25T11:47:29.688 INFO:teuthology.orchestra.run.smithi156.stdout: "ev": 22, 2024-04-25T11:47:29.688 INFO:teuthology.orchestra.run.smithi156.stdout: "evexg": 0, 2024-04-25T11:47:29.688 INFO:teuthology.orchestra.run.smithi156.stdout: "evexd": 0, 2024-04-25T11:47:29.688 INFO:teuthology.orchestra.run.smithi156.stdout: "segadd": 0, 2024-04-25T11:47:29.688 INFO:teuthology.orchestra.run.smithi156.stdout: "segex": 0, 2024-04-25T11:47:29.688 INFO:teuthology.orchestra.run.smithi156.stdout: "segtrm": 0, 2024-04-25T11:47:29.688 INFO:teuthology.orchestra.run.smithi156.stdout: "seg": 2, 2024-04-25T11:47:29.689 INFO:teuthology.orchestra.run.smithi156.stdout: "segmjr": 2, 2024-04-25T11:47:29.689 INFO:teuthology.orchestra.run.smithi156.stdout: "segexg": 0, 2024-04-25T11:47:29.689 INFO:teuthology.orchestra.run.smithi156.stdout: "segexd": 0, 2024-04-25T11:47:29.689 INFO:teuthology.orchestra.run.smithi156.stdout: "expos": 4194304, 2024-04-25T11:47:29.689 INFO:teuthology.orchestra.run.smithi156.stdout: "wrpos": 4228097, 2024-04-25T11:47:29.689 INFO:teuthology.orchestra.run.smithi156.stdout: "rdpos": 4227523, 2024-04-25T11:47:29.689 INFO:teuthology.orchestra.run.smithi156.stdout: "jlat": { 2024-04-25T11:47:29.689 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.689 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.689 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.689 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.689 INFO:teuthology.orchestra.run.smithi156.stdout: "replayed": 22 2024-04-25T11:47:29.689 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.689 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_mem": { 2024-04-25T11:47:29.689 INFO:teuthology.orchestra.run.smithi156.stdout: "ino": 11, 2024-04-25T11:47:29.689 INFO:teuthology.orchestra.run.smithi156.stdout: "ino+": 20, 2024-04-25T11:47:29.690 INFO:teuthology.orchestra.run.smithi156.stdout: "ino-": 9, 2024-04-25T11:47:29.690 INFO:teuthology.orchestra.run.smithi156.stdout: "dir": 5, 2024-04-25T11:47:29.690 INFO:teuthology.orchestra.run.smithi156.stdout: "dir+": 13, 2024-04-25T11:47:29.690 INFO:teuthology.orchestra.run.smithi156.stdout: "dir-": 8, 2024-04-25T11:47:29.690 INFO:teuthology.orchestra.run.smithi156.stdout: "dn": 11, 2024-04-25T11:47:29.690 INFO:teuthology.orchestra.run.smithi156.stdout: "dn+": 19, 2024-04-25T11:47:29.690 INFO:teuthology.orchestra.run.smithi156.stdout: "dn-": 8, 2024-04-25T11:47:29.690 INFO:teuthology.orchestra.run.smithi156.stdout: "cap": 0, 2024-04-25T11:47:29.690 INFO:teuthology.orchestra.run.smithi156.stdout: "cap+": 0, 2024-04-25T11:47:29.690 INFO:teuthology.orchestra.run.smithi156.stdout: "cap-": 0, 2024-04-25T11:47:29.690 INFO:teuthology.orchestra.run.smithi156.stdout: "rss": 119780, 2024-04-25T11:47:29.690 INFO:teuthology.orchestra.run.smithi156.stdout: "heap": 215440 2024-04-25T11:47:29.690 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.691 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_server": { 2024-04-25T11:47:29.691 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_client_request": 0, 2024-04-25T11:47:29.691 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_server_request": 0, 2024-04-25T11:47:29.691 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_request": 0, 2024-04-25T11:47:29.691 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_session": 0, 2024-04-25T11:47:29.691 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_peer_request": 0, 2024-04-25T11:47:29.691 INFO:teuthology.orchestra.run.smithi156.stdout: "req_create_latency": { 2024-04-25T11:47:29.691 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.691 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.691 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.691 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.691 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getattr_latency": { 2024-04-25T11:47:29.691 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.691 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.692 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.692 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.692 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getfilelock_latency": { 2024-04-25T11:47:29.692 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.692 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.692 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.692 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.692 INFO:teuthology.orchestra.run.smithi156.stdout: "req_link_latency": { 2024-04-25T11:47:29.692 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.692 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.692 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.692 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.692 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookup_latency": { 2024-04-25T11:47:29.692 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.693 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.693 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.693 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.693 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookuphash_latency": { 2024-04-25T11:47:29.693 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.693 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.693 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.693 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.693 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupino_latency": { 2024-04-25T11:47:29.693 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.693 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.693 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.693 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.693 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupname_latency": { 2024-04-25T11:47:29.693 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.693 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.694 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.694 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.694 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupparent_latency": { 2024-04-25T11:47:29.694 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.694 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.694 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.694 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.694 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupsnap_latency": { 2024-04-25T11:47:29.694 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.694 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.694 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.694 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.694 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lssnap_latency": { 2024-04-25T11:47:29.695 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.695 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.695 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.695 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.695 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mkdir_latency": { 2024-04-25T11:47:29.695 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.695 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.696 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.696 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.696 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mknod_latency": { 2024-04-25T11:47:29.696 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.696 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.696 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.696 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.696 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mksnap_latency": { 2024-04-25T11:47:29.696 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.696 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.696 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.696 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.696 INFO:teuthology.orchestra.run.smithi156.stdout: "req_open_latency": { 2024-04-25T11:47:29.696 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.697 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.697 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.697 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.697 INFO:teuthology.orchestra.run.smithi156.stdout: "req_readdir_latency": { 2024-04-25T11:47:29.697 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.697 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.697 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.697 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.697 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rename_latency": { 2024-04-25T11:47:29.697 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.697 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.697 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.697 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.697 INFO:teuthology.orchestra.run.smithi156.stdout: "req_renamesnap_latency": { 2024-04-25T11:47:29.697 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.698 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.698 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.698 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.698 INFO:teuthology.orchestra.run.smithi156.stdout: "req_snapdiff_latency": { 2024-04-25T11:47:29.698 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.698 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.698 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.698 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.698 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmdir_latency": { 2024-04-25T11:47:29.698 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.698 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.698 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.698 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.699 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmsnap_latency": { 2024-04-25T11:47:29.699 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.699 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.699 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.699 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.699 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmxattr_latency": { 2024-04-25T11:47:29.699 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.699 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.699 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.699 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.699 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setattr_latency": { 2024-04-25T11:47:29.699 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.699 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.699 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.699 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.700 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setdirlayout_latency": { 2024-04-25T11:47:29.700 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.700 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.700 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.700 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.700 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setfilelock_latency": { 2024-04-25T11:47:29.700 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.700 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.700 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.700 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.700 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setlayout_latency": { 2024-04-25T11:47:29.700 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.700 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.700 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.700 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.701 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setxattr_latency": { 2024-04-25T11:47:29.701 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.701 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.701 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.701 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.701 INFO:teuthology.orchestra.run.smithi156.stdout: "req_symlink_latency": { 2024-04-25T11:47:29.701 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.701 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.701 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.701 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.701 INFO:teuthology.orchestra.run.smithi156.stdout: "req_unlink_latency": { 2024-04-25T11:47:29.701 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.701 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.701 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.701 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.702 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_revoke_eviction": 0, 2024-04-25T11:47:29.702 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_acquisition_throttle": 0, 2024-04-25T11:47:29.702 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getvxattr_latency": { 2024-04-25T11:47:29.702 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.702 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.702 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.702 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:29.702 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.702 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_sessions": { 2024-04-25T11:47:29.702 INFO:teuthology.orchestra.run.smithi156.stdout: "session_count": 2, 2024-04-25T11:47:29.702 INFO:teuthology.orchestra.run.smithi156.stdout: "session_add": 4, 2024-04-25T11:47:29.702 INFO:teuthology.orchestra.run.smithi156.stdout: "session_remove": 2, 2024-04-25T11:47:29.702 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_open": 4, 2024-04-25T11:47:29.702 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_stale": 0, 2024-04-25T11:47:29.703 INFO:teuthology.orchestra.run.smithi156.stdout: "total_load": 0, 2024-04-25T11:47:29.703 INFO:teuthology.orchestra.run.smithi156.stdout: "average_load": 0, 2024-04-25T11:47:29.703 INFO:teuthology.orchestra.run.smithi156.stdout: "avg_session_uptime": 3137, 2024-04-25T11:47:29.703 INFO:teuthology.orchestra.run.smithi156.stdout: "mdthresh_evicted": 0 2024-04-25T11:47:29.703 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.703 INFO:teuthology.orchestra.run.smithi156.stdout: "mempool": { 2024-04-25T11:47:29.703 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_bytes": 0, 2024-04-25T11:47:29.703 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_items": 0, 2024-04-25T11:47:29.703 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T11:47:29.703 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_items": 0, 2024-04-25T11:47:29.703 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T11:47:29.703 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_items": 0, 2024-04-25T11:47:29.703 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T11:47:29.703 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T11:47:29.703 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T11:47:29.704 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T11:47:29.704 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T11:47:29.704 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_items": 0, 2024-04-25T11:47:29.704 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T11:47:29.704 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T11:47:29.704 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_bytes": 0, 2024-04-25T11:47:29.704 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_items": 0, 2024-04-25T11:47:29.704 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_bytes": 0, 2024-04-25T11:47:29.704 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_items": 0, 2024-04-25T11:47:29.704 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T11:47:29.704 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T11:47:29.704 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T11:47:29.704 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T11:47:29.704 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T11:47:29.704 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_items": 0, 2024-04-25T11:47:29.705 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_bytes": 0, 2024-04-25T11:47:29.705 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_items": 0, 2024-04-25T11:47:29.705 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T11:47:29.705 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T11:47:29.705 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_bytes": 0, 2024-04-25T11:47:29.705 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_items": 0, 2024-04-25T11:47:29.705 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_bytes": 0, 2024-04-25T11:47:29.705 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_items": 0, 2024-04-25T11:47:29.705 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T11:47:29.705 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_items": 0, 2024-04-25T11:47:29.705 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T11:47:29.705 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_items": 0, 2024-04-25T11:47:29.705 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_bytes": 181525, 2024-04-25T11:47:29.705 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_items": 52, 2024-04-25T11:47:29.705 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_bytes": 0, 2024-04-25T11:47:29.706 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_items": 0, 2024-04-25T11:47:29.706 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_bytes": 0, 2024-04-25T11:47:29.706 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_items": 0, 2024-04-25T11:47:29.706 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_bytes": 0, 2024-04-25T11:47:29.706 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_items": 0, 2024-04-25T11:47:29.706 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_bytes": 0, 2024-04-25T11:47:29.706 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_items": 0, 2024-04-25T11:47:29.706 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_bytes": 8632, 2024-04-25T11:47:29.706 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_items": 180, 2024-04-25T11:47:29.706 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T11:47:29.706 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_items": 0, 2024-04-25T11:47:29.706 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_bytes": 0, 2024-04-25T11:47:29.706 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_items": 0, 2024-04-25T11:47:29.707 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_bytes": 43142, 2024-04-25T11:47:29.707 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_items": 232, 2024-04-25T11:47:29.707 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_bytes": 0, 2024-04-25T11:47:29.707 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_items": 0, 2024-04-25T11:47:29.707 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_bytes": 0, 2024-04-25T11:47:29.707 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_items": 0 2024-04-25T11:47:29.707 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.707 INFO:teuthology.orchestra.run.smithi156.stdout: "objecter": { 2024-04-25T11:47:29.707 INFO:teuthology.orchestra.run.smithi156.stdout: "op_active": 0, 2024-04-25T11:47:29.707 INFO:teuthology.orchestra.run.smithi156.stdout: "op_laggy": 0, 2024-04-25T11:47:29.707 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send": 2450, 2024-04-25T11:47:29.707 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send_bytes": 21, 2024-04-25T11:47:29.707 INFO:teuthology.orchestra.run.smithi156.stdout: "op_resend": 3, 2024-04-25T11:47:29.707 INFO:teuthology.orchestra.run.smithi156.stdout: "op_reply": 2450, 2024-04-25T11:47:29.708 INFO:teuthology.orchestra.run.smithi156.stdout: "op_latency": { 2024-04-25T11:47:29.708 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2450, 2024-04-25T11:47:29.708 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 8.559863639, 2024-04-25T11:47:29.708 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.003493821 2024-04-25T11:47:29.708 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.708 INFO:teuthology.orchestra.run.smithi156.stdout: "op_inflight": 0, 2024-04-25T11:47:29.708 INFO:teuthology.orchestra.run.smithi156.stdout: "oplen_avg": { 2024-04-25T11:47:29.708 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2450, 2024-04-25T11:47:29.708 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 2451 2024-04-25T11:47:29.708 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.708 INFO:teuthology.orchestra.run.smithi156.stdout: "op": 2450, 2024-04-25T11:47:29.708 INFO:teuthology.orchestra.run.smithi156.stdout: "op_r": 2450, 2024-04-25T11:47:29.708 INFO:teuthology.orchestra.run.smithi156.stdout: "op_w": 0, 2024-04-25T11:47:29.708 INFO:teuthology.orchestra.run.smithi156.stdout: "op_rmw": 0, 2024-04-25T11:47:29.708 INFO:teuthology.orchestra.run.smithi156.stdout: "op_pg": 0, 2024-04-25T11:47:29.708 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_stat": 1584, 2024-04-25T11:47:29.709 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_create": 0, 2024-04-25T11:47:29.709 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_read": 864, 2024-04-25T11:47:29.709 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_write": 0, 2024-04-25T11:47:29.709 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writefull": 0, 2024-04-25T11:47:29.709 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writesame": 0, 2024-04-25T11:47:29.709 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_append": 0, 2024-04-25T11:47:29.709 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_zero": 0, 2024-04-25T11:47:29.709 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_truncate": 0, 2024-04-25T11:47:29.709 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_delete": 0, 2024-04-25T11:47:29.709 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_mapext": 0, 2024-04-25T11:47:29.709 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_sparse_read": 0, 2024-04-25T11:47:29.709 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_clonerange": 0, 2024-04-25T11:47:29.709 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_getxattr": 1, 2024-04-25T11:47:29.709 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_setxattr": 0, 2024-04-25T11:47:29.709 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_cmpxattr": 0, 2024-04-25T11:47:29.710 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_rmxattr": 0, 2024-04-25T11:47:29.710 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_resetxattrs": 0, 2024-04-25T11:47:29.710 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_call": 0, 2024-04-25T11:47:29.710 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_watch": 0, 2024-04-25T11:47:29.710 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_notify": 0, 2024-04-25T11:47:29.710 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T11:47:29.710 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls": 0, 2024-04-25T11:47:29.710 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls_filter": 0, 2024-04-25T11:47:29.710 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_other": 0, 2024-04-25T11:47:29.710 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_active": 0, 2024-04-25T11:47:29.710 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_send": 0, 2024-04-25T11:47:29.710 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_resend": 0, 2024-04-25T11:47:29.710 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_ping": 0, 2024-04-25T11:47:29.710 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_active": 0, 2024-04-25T11:47:29.710 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_send": 0, 2024-04-25T11:47:29.711 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_resend": 0, 2024-04-25T11:47:29.711 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_active": 0, 2024-04-25T11:47:29.711 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_send": 0, 2024-04-25T11:47:29.711 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_resend": 0, 2024-04-25T11:47:29.711 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_active": 0, 2024-04-25T11:47:29.711 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_send": 0, 2024-04-25T11:47:29.711 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_resend": 0, 2024-04-25T11:47:29.711 INFO:teuthology.orchestra.run.smithi156.stdout: "command_active": 0, 2024-04-25T11:47:29.711 INFO:teuthology.orchestra.run.smithi156.stdout: "command_send": 0, 2024-04-25T11:47:29.711 INFO:teuthology.orchestra.run.smithi156.stdout: "command_resend": 0, 2024-04-25T11:47:29.711 INFO:teuthology.orchestra.run.smithi156.stdout: "map_epoch": 93, 2024-04-25T11:47:29.711 INFO:teuthology.orchestra.run.smithi156.stdout: "map_full": 0, 2024-04-25T11:47:29.711 INFO:teuthology.orchestra.run.smithi156.stdout: "map_inc": 16, 2024-04-25T11:47:29.711 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_sessions": 4, 2024-04-25T11:47:29.711 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_open": 4, 2024-04-25T11:47:29.711 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_close": 0, 2024-04-25T11:47:29.712 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_laggy": 0, 2024-04-25T11:47:29.712 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_wr": 0, 2024-04-25T11:47:29.712 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_rd": 2, 2024-04-25T11:47:29.712 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_del": 0 2024-04-25T11:47:29.712 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.712 INFO:teuthology.orchestra.run.smithi156.stdout: "oft": { 2024-04-25T11:47:29.712 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_objs": 0, 2024-04-25T11:47:29.712 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_kv_pairs": 0, 2024-04-25T11:47:29.712 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_updates": 0, 2024-04-25T11:47:29.712 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_removes": 0 2024-04-25T11:47:29.712 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.712 INFO:teuthology.orchestra.run.smithi156.stdout: "osd-slow-ops": { 2024-04-25T11:47:29.712 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_ops_count": 0 2024-04-25T11:47:29.712 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.712 INFO:teuthology.orchestra.run.smithi156.stdout: "purge_queue": { 2024-04-25T11:47:29.713 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops": 0, 2024-04-25T11:47:29.713 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T11:47:29.713 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing": 0, 2024-04-25T11:47:29.713 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_high_water": 0, 2024-04-25T11:47:29.713 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed_ops": 0, 2024-04-25T11:47:29.713 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed": 0, 2024-04-25T11:47:29.713 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_item_in_journal": 0 2024-04-25T11:47:29.713 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.713 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T11:47:29.713 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T11:47:29.713 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-04-25T11:47:29.713 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T11:47:29.713 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 2729, 2024-04-25T11:47:29.713 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 2095880, 2024-04-25T11:47:29.713 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:29.713 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 2729, 2024-04-25T11:47:29.714 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-04-25T11:47:29.714 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-04-25T11:47:29.714 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 2729, 2024-04-25T11:47:29.714 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 2095880, 2024-04-25T11:47:29.714 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T11:47:29.714 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.714 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.714 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.714 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:29.714 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.714 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_bytes": { 2024-04-25T11:47:29.714 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T11:47:29.714 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-04-25T11:47:29.714 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T11:47:29.715 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-04-25T11:47:29.715 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-04-25T11:47:29.715 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:29.715 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:29.715 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 2450, 2024-04-25T11:47:29.715 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 33798, 2024-04-25T11:47:29.715 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 2450, 2024-04-25T11:47:29.715 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 33798, 2024-04-25T11:47:29.715 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T11:47:29.715 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.715 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.715 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.715 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:29.715 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.715 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_ops": { 2024-04-25T11:47:29.715 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T11:47:29.716 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 1024, 2024-04-25T11:47:29.716 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T11:47:29.716 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-04-25T11:47:29.716 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-04-25T11:47:29.716 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:29.716 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:29.716 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 2450, 2024-04-25T11:47:29.716 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 2450, 2024-04-25T11:47:29.716 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 2450, 2024-04-25T11:47:29.716 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 2450, 2024-04-25T11:47:29.716 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T11:47:29.716 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.716 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.716 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.716 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:29.717 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.717 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle": { 2024-04-25T11:47:29.717 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T11:47:29.717 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-04-25T11:47:29.717 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T11:47:29.717 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-04-25T11:47:29.717 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-04-25T11:47:29.717 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:29.717 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:29.717 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-04-25T11:47:29.717 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-04-25T11:47:29.717 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 0, 2024-04-25T11:47:29.717 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 0, 2024-04-25T11:47:29.717 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T11:47:29.717 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.717 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.718 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.718 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:29.718 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:29.718 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle-0x560077bd2c30": { 2024-04-25T11:47:29.718 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T11:47:29.718 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-04-25T11:47:29.718 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T11:47:29.718 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-04-25T11:47:29.718 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-04-25T11:47:29.718 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:29.718 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:29.718 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-04-25T11:47:29.718 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-04-25T11:47:29.718 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 0, 2024-04-25T11:47:29.719 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 0, 2024-04-25T11:47:29.719 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T11:47:29.719 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:29.719 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:29.719 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:29.719 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:29.719 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:29.719 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-04-25T11:47:30.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:30 smithi156 ceph-mon[33045]: pgmap v386: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T11:47:30.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:30 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:47:30.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:30 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:47:30.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:30 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:47:30.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:30 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:47:30.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:30 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:47:30.447 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-25T11:47:30.447 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-04-25T11:47:30.447 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-04-25T11:47:30.447 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-04-25T11:47:30.447 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=0 2024-04-25T11:47:30.447 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-25T11:47:30.447 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T11:47:30.447 DEBUG:tasks.check_counter:Getting stats from f 2024-04-25T11:47:30.447 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-25T11:47:30.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:30 smithi090 ceph-mon[29575]: pgmap v386: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T11:47:30.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:30 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:47:30.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:30 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:47:30.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:30 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:47:30.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:30 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:47:30.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:30 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:47:30.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:30 smithi138 ceph-mon[32328]: pgmap v386: 97 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 89 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T11:47:30.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:30 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:47:30.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:30 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:47:30.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:30 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T11:47:30.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:30 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T11:47:30.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:30 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' 2024-04-25T11:47:30.692 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T11:47:32.403 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:32 smithi156 ceph-mon[33045]: pgmap v387: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-25T11:47:32.404 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-04-25T11:47:32.404 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T11:47:32.404 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 97, 2024-04-25T11:47:32.404 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 246, 2024-04-25T11:47:32.404 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 52788, 2024-04-25T11:47:32.404 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 436760, 2024-04-25T11:47:32.404 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 13, 2024-04-25T11:47:32.404 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 2, 2024-04-25T11:47:32.404 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.027026043, 2024-04-25T11:47:32.404 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.009649800, 2024-04-25T11:47:32.404 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.011084178, 2024-04-25T11:47:32.404 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.001085847, 2024-04-25T11:47:32.404 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:32.404 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 246, 2024-04-25T11:47:32.405 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.053228564, 2024-04-25T11:47:32.405 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000216376 2024-04-25T11:47:32.405 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.405 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:32.405 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 132, 2024-04-25T11:47:32.405 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000044365, 2024-04-25T11:47:32.405 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000336 2024-04-25T11:47:32.405 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.405 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 1088, 2024-04-25T11:47:32.405 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 356016 2024-04-25T11:47:32.405 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.405 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T11:47:32.405 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 378, 2024-04-25T11:47:32.405 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 402, 2024-04-25T11:47:32.406 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 1642468, 2024-04-25T11:47:32.406 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 701292, 2024-04-25T11:47:32.406 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 53, 2024-04-25T11:47:32.406 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 8, 2024-04-25T11:47:32.406 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.088555995, 2024-04-25T11:47:32.406 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.020626834, 2024-04-25T11:47:32.406 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.073727640, 2024-04-25T11:47:32.406 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.007125436, 2024-04-25T11:47:32.406 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:32.406 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 393, 2024-04-25T11:47:32.406 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.026972954, 2024-04-25T11:47:32.406 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000068633 2024-04-25T11:47:32.406 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.406 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:32.406 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 99, 2024-04-25T11:47:32.407 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000026294, 2024-04-25T11:47:32.407 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000265 2024-04-25T11:47:32.407 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.407 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 1630432, 2024-04-25T11:47:32.407 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 154432 2024-04-25T11:47:32.407 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.407 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T11:47:32.407 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 1478, 2024-04-25T11:47:32.407 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 1120, 2024-04-25T11:47:32.407 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 495370, 2024-04-25T11:47:32.407 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 584657, 2024-04-25T11:47:32.407 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 10, 2024-04-25T11:47:32.407 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 3, 2024-04-25T11:47:32.407 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.115656615, 2024-04-25T11:47:32.407 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.043192155, 2024-04-25T11:47:32.408 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.078555599, 2024-04-25T11:47:32.408 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.005512180, 2024-04-25T11:47:32.408 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:32.408 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1124, 2024-04-25T11:47:32.408 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.052531209, 2024-04-25T11:47:32.408 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000046735 2024-04-25T11:47:32.408 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.408 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:32.408 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2408, 2024-04-25T11:47:32.408 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000633397, 2024-04-25T11:47:32.408 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000263 2024-04-25T11:47:32.408 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.408 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 4080, 2024-04-25T11:47:32.408 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 130000 2024-04-25T11:47:32.408 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.408 INFO:teuthology.orchestra.run.smithi156.stdout: "cct": { 2024-04-25T11:47:32.409 INFO:teuthology.orchestra.run.smithi156.stdout: "total_workers": 1, 2024-04-25T11:47:32.409 INFO:teuthology.orchestra.run.smithi156.stdout: "unhealthy_workers": 0 2024-04-25T11:47:32.409 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.409 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-MDSRank": { 2024-04-25T11:47:32.409 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-04-25T11:47:32.409 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-04-25T11:47:32.409 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 49, 2024-04-25T11:47:32.409 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.004116935, 2024-04-25T11:47:32.409 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000084019 2024-04-25T11:47:32.409 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:32.409 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.409 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-PurgeQueue": { 2024-04-25T11:47:32.409 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-04-25T11:47:32.409 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-04-25T11:47:32.409 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2, 2024-04-25T11:47:32.409 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000007567, 2024-04-25T11:47:32.410 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000003783 2024-04-25T11:47:32.410 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:32.410 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.410 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": { 2024-04-25T11:47:32.410 INFO:teuthology.orchestra.run.smithi156.stdout: "request": 1, 2024-04-25T11:47:32.410 INFO:teuthology.orchestra.run.smithi156.stdout: "reply": 1, 2024-04-25T11:47:32.410 INFO:teuthology.orchestra.run.smithi156.stdout: "reply_latency": { 2024-04-25T11:47:32.410 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1, 2024-04-25T11:47:32.410 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000126230, 2024-04-25T11:47:32.410 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000126230 2024-04-25T11:47:32.410 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.410 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_reply": 0, 2024-04-25T11:47:32.410 INFO:teuthology.orchestra.run.smithi156.stdout: "forward": 0, 2024-04-25T11:47:32.410 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_complete": 0, 2024-04-25T11:47:32.410 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_keys": 0, 2024-04-25T11:47:32.410 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_commit": 11, 2024-04-25T11:47:32.411 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_split": 0, 2024-04-25T11:47:32.411 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_merge": 0, 2024-04-25T11:47:32.411 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes": 16, 2024-04-25T11:47:32.411 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_top": 0, 2024-04-25T11:47:32.411 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_bottom": 16, 2024-04-25T11:47:32.411 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pin_tail": 0, 2024-04-25T11:47:32.411 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pinned": 13, 2024-04-25T11:47:32.411 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_expired": 8, 2024-04-25T11:47:32.411 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_with_caps": 1, 2024-04-25T11:47:32.411 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": 1, 2024-04-25T11:47:32.411 INFO:teuthology.orchestra.run.smithi156.stdout: "subtrees": 3, 2024-04-25T11:47:32.411 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse": 10, 2024-04-25T11:47:32.411 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_hit": 6, 2024-04-25T11:47:32.411 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_forward": 0, 2024-04-25T11:47:32.411 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_discover": 4, 2024-04-25T11:47:32.411 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_dir_fetch": 0, 2024-04-25T11:47:32.412 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_remote_ino": 0, 2024-04-25T11:47:32.412 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_lock": 0, 2024-04-25T11:47:32.412 INFO:teuthology.orchestra.run.smithi156.stdout: "load_cent": 0, 2024-04-25T11:47:32.412 INFO:teuthology.orchestra.run.smithi156.stdout: "q": 0, 2024-04-25T11:47:32.412 INFO:teuthology.orchestra.run.smithi156.stdout: "exported": 0, 2024-04-25T11:47:32.412 INFO:teuthology.orchestra.run.smithi156.stdout: "exported_inodes": 0, 2024-04-25T11:47:32.412 INFO:teuthology.orchestra.run.smithi156.stdout: "imported": 0, 2024-04-25T11:47:32.412 INFO:teuthology.orchestra.run.smithi156.stdout: "imported_inodes": 0, 2024-04-25T11:47:32.412 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_dir_fetch": 0, 2024-04-25T11:47:32.412 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_backtrace_fetch": 0, 2024-04-25T11:47:32.412 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_peer_discover": 0, 2024-04-25T11:47:32.412 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rfiles": 22, 2024-04-25T11:47:32.412 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rbytes": 10738142561, 2024-04-25T11:47:32.412 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rsnaps": 6, 2024-04-25T11:47:32.412 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T11:47:32.412 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_set_tag": 0, 2024-04-25T11:47:32.413 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T11:47:32.413 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_inotable_repaired": 0, 2024-04-25T11:47:32.413 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_inodes": 0, 2024-04-25T11:47:32.413 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T11:47:32.413 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T11:47:32.413 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_file_inodes": 0, 2024-04-25T11:47:32.413 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_inode_file_caps": 0, 2024-04-25T11:47:32.413 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T11:47:32.413 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_grant": 0, 2024-04-25T11:47:32.413 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T11:47:32.413 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T11:47:32.413 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T11:47:32.413 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps": 0, 2024-04-25T11:47:32.413 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps_dirty": 0, 2024-04-25T11:47:32.413 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_cap_release": 0, 2024-04-25T11:47:32.413 INFO:teuthology.orchestra.run.smithi156.stdout: "process_request_cap_release": 0 2024-04-25T11:47:32.414 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.414 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_cache": { 2024-04-25T11:47:32.414 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update": 0, 2024-04-25T11:47:32.414 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update_receipt": 14, 2024-04-25T11:47:32.414 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_try_discover": 9, 2024-04-25T11:47:32.414 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_send_discover": 11, 2024-04-25T11:47:32.414 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_handle_discover": 0, 2024-04-25T11:47:32.414 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays": 0, 2024-04-25T11:47:32.414 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_delayed": 0, 2024-04-25T11:47:32.414 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_enqueuing": 0, 2024-04-25T11:47:32.414 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_created": 0, 2024-04-25T11:47:32.414 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_enqueued": 0, 2024-04-25T11:47:32.414 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_reintegrated": 0, 2024-04-25T11:47:32.414 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_migrated": 0, 2024-04-25T11:47:32.414 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_processing": 0, 2024-04-25T11:47:32.414 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_enqueued": 0, 2024-04-25T11:47:32.415 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_prioritized": 0, 2024-04-25T11:47:32.415 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_started": 0, 2024-04-25T11:47:32.415 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_completed": 0, 2024-04-25T11:47:32.415 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_path": 0, 2024-04-25T11:47:32.415 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_inode": 0, 2024-04-25T11:47:32.415 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T11:47:32.415 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_exportdir": 0, 2024-04-25T11:47:32.415 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_flush": 0, 2024-04-25T11:47:32.415 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragmentdir": 0, 2024-04-25T11:47:32.415 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragstats": 0, 2024-04-25T11:47:32.415 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_inodestats": 0 2024-04-25T11:47:32.415 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.415 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_log": { 2024-04-25T11:47:32.415 INFO:teuthology.orchestra.run.smithi156.stdout: "evlrg": 0, 2024-04-25T11:47:32.415 INFO:teuthology.orchestra.run.smithi156.stdout: "evadd": 7, 2024-04-25T11:47:32.415 INFO:teuthology.orchestra.run.smithi156.stdout: "evex": 0, 2024-04-25T11:47:32.416 INFO:teuthology.orchestra.run.smithi156.stdout: "evtrm": 0, 2024-04-25T11:47:32.416 INFO:teuthology.orchestra.run.smithi156.stdout: "ev": 7, 2024-04-25T11:47:32.416 INFO:teuthology.orchestra.run.smithi156.stdout: "evexg": 0, 2024-04-25T11:47:32.416 INFO:teuthology.orchestra.run.smithi156.stdout: "evexd": 0, 2024-04-25T11:47:32.416 INFO:teuthology.orchestra.run.smithi156.stdout: "segadd": 2, 2024-04-25T11:47:32.416 INFO:teuthology.orchestra.run.smithi156.stdout: "segex": 0, 2024-04-25T11:47:32.416 INFO:teuthology.orchestra.run.smithi156.stdout: "segtrm": 0, 2024-04-25T11:47:32.416 INFO:teuthology.orchestra.run.smithi156.stdout: "seg": 2, 2024-04-25T11:47:32.416 INFO:teuthology.orchestra.run.smithi156.stdout: "segmjr": 2, 2024-04-25T11:47:32.416 INFO:teuthology.orchestra.run.smithi156.stdout: "segexg": 0, 2024-04-25T11:47:32.416 INFO:teuthology.orchestra.run.smithi156.stdout: "segexd": 0, 2024-04-25T11:47:32.416 INFO:teuthology.orchestra.run.smithi156.stdout: "expos": 4194304, 2024-04-25T11:47:32.416 INFO:teuthology.orchestra.run.smithi156.stdout: "wrpos": 4196264, 2024-04-25T11:47:32.416 INFO:teuthology.orchestra.run.smithi156.stdout: "rdpos": 0, 2024-04-25T11:47:32.416 INFO:teuthology.orchestra.run.smithi156.stdout: "jlat": { 2024-04-25T11:47:32.416 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 6, 2024-04-25T11:47:32.417 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.014832595, 2024-04-25T11:47:32.417 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.002472099 2024-04-25T11:47:32.417 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.417 INFO:teuthology.orchestra.run.smithi156.stdout: "replayed": 0 2024-04-25T11:47:32.417 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.417 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_mem": { 2024-04-25T11:47:32.417 INFO:teuthology.orchestra.run.smithi156.stdout: "ino": 20, 2024-04-25T11:47:32.417 INFO:teuthology.orchestra.run.smithi156.stdout: "ino+": 24, 2024-04-25T11:47:32.417 INFO:teuthology.orchestra.run.smithi156.stdout: "ino-": 4, 2024-04-25T11:47:32.417 INFO:teuthology.orchestra.run.smithi156.stdout: "dir": 19, 2024-04-25T11:47:32.417 INFO:teuthology.orchestra.run.smithi156.stdout: "dir+": 24, 2024-04-25T11:47:32.417 INFO:teuthology.orchestra.run.smithi156.stdout: "dir-": 5, 2024-04-25T11:47:32.417 INFO:teuthology.orchestra.run.smithi156.stdout: "dn": 16, 2024-04-25T11:47:32.417 INFO:teuthology.orchestra.run.smithi156.stdout: "dn+": 24, 2024-04-25T11:47:32.417 INFO:teuthology.orchestra.run.smithi156.stdout: "dn-": 8, 2024-04-25T11:47:32.417 INFO:teuthology.orchestra.run.smithi156.stdout: "cap": 1, 2024-04-25T11:47:32.418 INFO:teuthology.orchestra.run.smithi156.stdout: "cap+": 1, 2024-04-25T11:47:32.418 INFO:teuthology.orchestra.run.smithi156.stdout: "cap-": 0, 2024-04-25T11:47:32.418 INFO:teuthology.orchestra.run.smithi156.stdout: "rss": 122196, 2024-04-25T11:47:32.418 INFO:teuthology.orchestra.run.smithi156.stdout: "heap": 223632 2024-04-25T11:47:32.418 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.418 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_server": { 2024-04-25T11:47:32.418 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_client_request": 1, 2024-04-25T11:47:32.418 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_server_request": 1, 2024-04-25T11:47:32.418 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_request": 1, 2024-04-25T11:47:32.418 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_session": 18, 2024-04-25T11:47:32.418 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_peer_request": 2, 2024-04-25T11:47:32.418 INFO:teuthology.orchestra.run.smithi156.stdout: "req_create_latency": { 2024-04-25T11:47:32.418 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.418 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.418 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.418 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.419 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getattr_latency": { 2024-04-25T11:47:32.419 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.419 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.419 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.419 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.419 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getfilelock_latency": { 2024-04-25T11:47:32.419 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.419 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.419 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.419 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.419 INFO:teuthology.orchestra.run.smithi156.stdout: "req_link_latency": { 2024-04-25T11:47:32.419 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.420 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.420 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.420 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.420 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookup_latency": { 2024-04-25T11:47:32.420 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.420 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.420 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.420 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.420 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookuphash_latency": { 2024-04-25T11:47:32.420 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.420 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.420 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.420 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.420 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupino_latency": { 2024-04-25T11:47:32.420 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1, 2024-04-25T11:47:32.420 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000126230, 2024-04-25T11:47:32.421 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000126230 2024-04-25T11:47:32.421 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.421 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupname_latency": { 2024-04-25T11:47:32.421 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.421 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.421 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.421 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.421 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupparent_latency": { 2024-04-25T11:47:32.421 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.421 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.421 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.421 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.421 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupsnap_latency": { 2024-04-25T11:47:32.421 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.421 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.422 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.422 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.422 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lssnap_latency": { 2024-04-25T11:47:32.422 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.422 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.422 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.422 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.422 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mkdir_latency": { 2024-04-25T11:47:32.422 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.423 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.423 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.423 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.423 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mknod_latency": { 2024-04-25T11:47:32.423 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.423 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.423 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.423 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.423 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mksnap_latency": { 2024-04-25T11:47:32.423 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.423 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.423 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.423 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.423 INFO:teuthology.orchestra.run.smithi156.stdout: "req_open_latency": { 2024-04-25T11:47:32.424 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.424 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.424 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.424 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.424 INFO:teuthology.orchestra.run.smithi156.stdout: "req_readdir_latency": { 2024-04-25T11:47:32.424 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.424 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.424 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.424 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.424 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rename_latency": { 2024-04-25T11:47:32.424 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.424 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.424 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.424 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.424 INFO:teuthology.orchestra.run.smithi156.stdout: "req_renamesnap_latency": { 2024-04-25T11:47:32.424 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.425 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.425 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.425 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.425 INFO:teuthology.orchestra.run.smithi156.stdout: "req_snapdiff_latency": { 2024-04-25T11:47:32.425 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.425 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.425 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.425 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.425 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmdir_latency": { 2024-04-25T11:47:32.425 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.425 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.425 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.425 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.425 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmsnap_latency": { 2024-04-25T11:47:32.425 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.425 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.426 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.426 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.426 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmxattr_latency": { 2024-04-25T11:47:32.426 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.426 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.426 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.426 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.426 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setattr_latency": { 2024-04-25T11:47:32.426 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.426 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.426 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.426 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.426 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setdirlayout_latency": { 2024-04-25T11:47:32.426 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.426 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.427 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.427 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.427 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setfilelock_latency": { 2024-04-25T11:47:32.427 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.427 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.427 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.427 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.427 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setlayout_latency": { 2024-04-25T11:47:32.427 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.427 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.427 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.427 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.427 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setxattr_latency": { 2024-04-25T11:47:32.427 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.428 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.428 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.428 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.428 INFO:teuthology.orchestra.run.smithi156.stdout: "req_symlink_latency": { 2024-04-25T11:47:32.428 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.428 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.428 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.428 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.428 INFO:teuthology.orchestra.run.smithi156.stdout: "req_unlink_latency": { 2024-04-25T11:47:32.428 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.428 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.428 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.428 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.428 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_revoke_eviction": 0, 2024-04-25T11:47:32.428 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_acquisition_throttle": 0, 2024-04-25T11:47:32.428 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getvxattr_latency": { 2024-04-25T11:47:32.429 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.429 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.429 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.429 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:32.429 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.429 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_sessions": { 2024-04-25T11:47:32.429 INFO:teuthology.orchestra.run.smithi156.stdout: "session_count": 1, 2024-04-25T11:47:32.429 INFO:teuthology.orchestra.run.smithi156.stdout: "session_add": 3, 2024-04-25T11:47:32.429 INFO:teuthology.orchestra.run.smithi156.stdout: "session_remove": 2, 2024-04-25T11:47:32.429 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_open": 1, 2024-04-25T11:47:32.429 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_stale": 0, 2024-04-25T11:47:32.429 INFO:teuthology.orchestra.run.smithi156.stdout: "total_load": 1, 2024-04-25T11:47:32.429 INFO:teuthology.orchestra.run.smithi156.stdout: "average_load": 0, 2024-04-25T11:47:32.429 INFO:teuthology.orchestra.run.smithi156.stdout: "avg_session_uptime": 733, 2024-04-25T11:47:32.429 INFO:teuthology.orchestra.run.smithi156.stdout: "mdthresh_evicted": 0 2024-04-25T11:47:32.430 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.430 INFO:teuthology.orchestra.run.smithi156.stdout: "mempool": { 2024-04-25T11:47:32.430 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_bytes": 0, 2024-04-25T11:47:32.430 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_items": 0, 2024-04-25T11:47:32.430 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T11:47:32.430 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_items": 0, 2024-04-25T11:47:32.430 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T11:47:32.430 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_items": 0, 2024-04-25T11:47:32.430 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T11:47:32.430 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T11:47:32.430 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T11:47:32.430 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T11:47:32.430 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T11:47:32.430 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_items": 0, 2024-04-25T11:47:32.431 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T11:47:32.431 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T11:47:32.431 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_bytes": 0, 2024-04-25T11:47:32.431 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_items": 0, 2024-04-25T11:47:32.431 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_bytes": 0, 2024-04-25T11:47:32.431 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_items": 0, 2024-04-25T11:47:32.431 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T11:47:32.431 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T11:47:32.431 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T11:47:32.431 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T11:47:32.431 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T11:47:32.431 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_items": 0, 2024-04-25T11:47:32.431 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_bytes": 0, 2024-04-25T11:47:32.431 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_items": 0, 2024-04-25T11:47:32.431 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T11:47:32.431 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T11:47:32.431 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_bytes": 0, 2024-04-25T11:47:32.432 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_items": 0, 2024-04-25T11:47:32.432 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_bytes": 0, 2024-04-25T11:47:32.432 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_items": 0, 2024-04-25T11:47:32.432 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T11:47:32.432 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_items": 0, 2024-04-25T11:47:32.432 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T11:47:32.432 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_items": 0, 2024-04-25T11:47:32.432 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_bytes": 270007, 2024-04-25T11:47:32.432 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_items": 83, 2024-04-25T11:47:32.432 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_bytes": 0, 2024-04-25T11:47:32.432 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_items": 0, 2024-04-25T11:47:32.432 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_bytes": 0, 2024-04-25T11:47:32.432 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_items": 0, 2024-04-25T11:47:32.432 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_bytes": 0, 2024-04-25T11:47:32.432 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_items": 0, 2024-04-25T11:47:32.432 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_bytes": 0, 2024-04-25T11:47:32.433 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_items": 0, 2024-04-25T11:47:32.433 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_bytes": 8632, 2024-04-25T11:47:32.433 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_items": 180, 2024-04-25T11:47:32.433 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T11:47:32.433 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_items": 0, 2024-04-25T11:47:32.433 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_bytes": 0, 2024-04-25T11:47:32.433 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_items": 0, 2024-04-25T11:47:32.433 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_bytes": 82973, 2024-04-25T11:47:32.433 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_items": 328, 2024-04-25T11:47:32.433 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_bytes": 0, 2024-04-25T11:47:32.433 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_items": 0, 2024-04-25T11:47:32.433 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_bytes": 0, 2024-04-25T11:47:32.433 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_items": 0 2024-04-25T11:47:32.433 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.433 INFO:teuthology.orchestra.run.smithi156.stdout: "objecter": { 2024-04-25T11:47:32.434 INFO:teuthology.orchestra.run.smithi156.stdout: "op_active": 0, 2024-04-25T11:47:32.434 INFO:teuthology.orchestra.run.smithi156.stdout: "op_laggy": 0, 2024-04-25T11:47:32.434 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send": 44, 2024-04-25T11:47:32.434 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send_bytes": 12563, 2024-04-25T11:47:32.434 INFO:teuthology.orchestra.run.smithi156.stdout: "op_resend": 33, 2024-04-25T11:47:32.434 INFO:teuthology.orchestra.run.smithi156.stdout: "op_reply": 44, 2024-04-25T11:47:32.434 INFO:teuthology.orchestra.run.smithi156.stdout: "op_latency": { 2024-04-25T11:47:32.434 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 44, 2024-04-25T11:47:32.434 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.253995974, 2024-04-25T11:47:32.434 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.005772635 2024-04-25T11:47:32.434 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.434 INFO:teuthology.orchestra.run.smithi156.stdout: "op_inflight": 0, 2024-04-25T11:47:32.434 INFO:teuthology.orchestra.run.smithi156.stdout: "oplen_avg": { 2024-04-25T11:47:32.434 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 44, 2024-04-25T11:47:32.434 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 67 2024-04-25T11:47:32.435 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.435 INFO:teuthology.orchestra.run.smithi156.stdout: "op": 44, 2024-04-25T11:47:32.435 INFO:teuthology.orchestra.run.smithi156.stdout: "op_r": 0, 2024-04-25T11:47:32.435 INFO:teuthology.orchestra.run.smithi156.stdout: "op_w": 44, 2024-04-25T11:47:32.435 INFO:teuthology.orchestra.run.smithi156.stdout: "op_rmw": 0, 2024-04-25T11:47:32.435 INFO:teuthology.orchestra.run.smithi156.stdout: "op_pg": 0, 2024-04-25T11:47:32.435 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_stat": 0, 2024-04-25T11:47:32.435 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_create": 10, 2024-04-25T11:47:32.435 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_read": 0, 2024-04-25T11:47:32.435 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_write": 6, 2024-04-25T11:47:32.435 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writefull": 8, 2024-04-25T11:47:32.435 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writesame": 0, 2024-04-25T11:47:32.435 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_append": 0, 2024-04-25T11:47:32.435 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_zero": 0, 2024-04-25T11:47:32.435 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_truncate": 0, 2024-04-25T11:47:32.435 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_delete": 6, 2024-04-25T11:47:32.436 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_mapext": 0, 2024-04-25T11:47:32.436 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_sparse_read": 0, 2024-04-25T11:47:32.436 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_clonerange": 0, 2024-04-25T11:47:32.436 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_getxattr": 0, 2024-04-25T11:47:32.436 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_setxattr": 20, 2024-04-25T11:47:32.436 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_cmpxattr": 0, 2024-04-25T11:47:32.436 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_rmxattr": 0, 2024-04-25T11:47:32.436 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_resetxattrs": 0, 2024-04-25T11:47:32.436 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_call": 0, 2024-04-25T11:47:32.436 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_watch": 0, 2024-04-25T11:47:32.436 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_notify": 0, 2024-04-25T11:47:32.436 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T11:47:32.436 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls": 0, 2024-04-25T11:47:32.436 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls_filter": 0, 2024-04-25T11:47:32.436 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_other": 0, 2024-04-25T11:47:32.436 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_active": 0, 2024-04-25T11:47:32.437 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_send": 0, 2024-04-25T11:47:32.437 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_resend": 0, 2024-04-25T11:47:32.437 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_ping": 0, 2024-04-25T11:47:32.437 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_active": 0, 2024-04-25T11:47:32.437 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_send": 0, 2024-04-25T11:47:32.437 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_resend": 0, 2024-04-25T11:47:32.437 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_active": 0, 2024-04-25T11:47:32.437 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_send": 0, 2024-04-25T11:47:32.437 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_resend": 0, 2024-04-25T11:47:32.437 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_active": 0, 2024-04-25T11:47:32.437 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_send": 0, 2024-04-25T11:47:32.437 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_resend": 0, 2024-04-25T11:47:32.437 INFO:teuthology.orchestra.run.smithi156.stdout: "command_active": 0, 2024-04-25T11:47:32.437 INFO:teuthology.orchestra.run.smithi156.stdout: "command_send": 0, 2024-04-25T11:47:32.437 INFO:teuthology.orchestra.run.smithi156.stdout: "command_resend": 0, 2024-04-25T11:47:32.438 INFO:teuthology.orchestra.run.smithi156.stdout: "map_epoch": 93, 2024-04-25T11:47:32.438 INFO:teuthology.orchestra.run.smithi156.stdout: "map_full": 0, 2024-04-25T11:47:32.438 INFO:teuthology.orchestra.run.smithi156.stdout: "map_inc": 16, 2024-04-25T11:47:32.438 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_sessions": 8, 2024-04-25T11:47:32.438 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_open": 8, 2024-04-25T11:47:32.438 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_close": 0, 2024-04-25T11:47:32.438 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_laggy": 0, 2024-04-25T11:47:32.438 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_wr": 16, 2024-04-25T11:47:32.438 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_rd": 0, 2024-04-25T11:47:32.438 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_del": 1 2024-04-25T11:47:32.438 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.438 INFO:teuthology.orchestra.run.smithi156.stdout: "oft": { 2024-04-25T11:47:32.438 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_objs": 1, 2024-04-25T11:47:32.438 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_kv_pairs": 0, 2024-04-25T11:47:32.439 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_updates": 0, 2024-04-25T11:47:32.439 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_removes": 0 2024-04-25T11:47:32.439 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.439 INFO:teuthology.orchestra.run.smithi156.stdout: "osd-slow-ops": { 2024-04-25T11:47:32.439 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_ops_count": 0 2024-04-25T11:47:32.439 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.439 INFO:teuthology.orchestra.run.smithi156.stdout: "purge_queue": { 2024-04-25T11:47:32.439 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops": 0, 2024-04-25T11:47:32.439 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T11:47:32.439 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing": 0, 2024-04-25T11:47:32.439 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_high_water": 0, 2024-04-25T11:47:32.439 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed_ops": 0, 2024-04-25T11:47:32.439 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed": 0, 2024-04-25T11:47:32.439 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_item_in_journal": 0 2024-04-25T11:47:32.439 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.440 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T11:47:32.440 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T11:47:32.440 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-04-25T11:47:32.440 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T11:47:32.440 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 1949, 2024-04-25T11:47:32.440 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 1996283, 2024-04-25T11:47:32.440 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:32.440 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 1949, 2024-04-25T11:47:32.440 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-04-25T11:47:32.440 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-04-25T11:47:32.440 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 1949, 2024-04-25T11:47:32.440 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 1996283, 2024-04-25T11:47:32.440 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T11:47:32.440 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.440 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.440 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.441 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:32.441 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.441 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_bytes": { 2024-04-25T11:47:32.441 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T11:47:32.441 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-04-25T11:47:32.441 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T11:47:32.441 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-04-25T11:47:32.441 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-04-25T11:47:32.441 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:32.441 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:32.441 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 44, 2024-04-25T11:47:32.441 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 12563, 2024-04-25T11:47:32.441 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 44, 2024-04-25T11:47:32.441 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 12563, 2024-04-25T11:47:32.441 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T11:47:32.441 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.442 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.442 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.442 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:32.442 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.442 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_ops": { 2024-04-25T11:47:32.442 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T11:47:32.442 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 1024, 2024-04-25T11:47:32.442 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T11:47:32.442 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-04-25T11:47:32.442 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-04-25T11:47:32.442 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:32.442 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:32.442 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 44, 2024-04-25T11:47:32.442 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 44, 2024-04-25T11:47:32.443 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 44, 2024-04-25T11:47:32.443 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 44, 2024-04-25T11:47:32.443 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T11:47:32.443 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.443 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.443 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.443 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:32.443 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.443 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle": { 2024-04-25T11:47:32.443 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T11:47:32.443 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-04-25T11:47:32.443 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T11:47:32.443 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-04-25T11:47:32.443 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-04-25T11:47:32.443 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:32.444 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:32.444 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-04-25T11:47:32.444 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-04-25T11:47:32.444 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 0, 2024-04-25T11:47:32.444 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 0, 2024-04-25T11:47:32.444 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T11:47:32.444 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.444 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.444 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.444 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:32.444 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:32.444 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle-0x55ca6a6cc750": { 2024-04-25T11:47:32.444 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T11:47:32.444 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-04-25T11:47:32.444 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T11:47:32.445 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 7, 2024-04-25T11:47:32.445 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 1960, 2024-04-25T11:47:32.445 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:32.445 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 7, 2024-04-25T11:47:32.445 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-04-25T11:47:32.445 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-04-25T11:47:32.445 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 6, 2024-04-25T11:47:32.445 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 1960, 2024-04-25T11:47:32.445 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T11:47:32.445 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:32.445 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:32.445 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:32.445 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:32.445 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:32.445 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-04-25T11:47:32.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:32 smithi090 ceph-mon[29575]: pgmap v387: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-25T11:47:32.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:32 smithi138 ceph-mon[32328]: pgmap v387: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-25T11:47:33.199 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-25T11:47:33.199 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-25T11:47:33.199 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-04-25T11:47:33.199 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=14 2024-04-25T11:47:33.199 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-04-25T11:47:33.199 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-25T11:47:33.199 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T11:47:33.199 DEBUG:tasks.check_counter:Getting stats from i 2024-04-25T11:47:33.200 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-25T11:47:33.440 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T11:47:34.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:34 smithi156 ceph-mon[33045]: pgmap v388: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T11:47:34.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:34 smithi156 ceph-mon[33045]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T11:47:34.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:34 smithi090 ceph-mon[29575]: pgmap v388: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T11:47:34.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:34 smithi090 ceph-mon[29575]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T11:47:34.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:34 smithi138 ceph-mon[32328]: pgmap v388: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T11:47:34.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:34 smithi138 ceph-mon[32328]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T11:47:35.247 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-04-25T11:47:35.247 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T11:47:35.247 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 139, 2024-04-25T11:47:35.247 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 295, 2024-04-25T11:47:35.247 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 70063, 2024-04-25T11:47:35.248 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 509358, 2024-04-25T11:47:35.248 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 17, 2024-04-25T11:47:35.248 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 5, 2024-04-25T11:47:35.248 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.030689978, 2024-04-25T11:47:35.248 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.009774653, 2024-04-25T11:47:35.248 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.009802289, 2024-04-25T11:47:35.248 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.001811327, 2024-04-25T11:47:35.248 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:35.248 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 289, 2024-04-25T11:47:35.248 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.040869578, 2024-04-25T11:47:35.248 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000141417 2024-04-25T11:47:35.248 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.248 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:35.248 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 132, 2024-04-25T11:47:35.249 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000037494, 2024-04-25T11:47:35.249 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000284 2024-04-25T11:47:35.249 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.249 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 4080, 2024-04-25T11:47:35.249 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 359424 2024-04-25T11:47:35.249 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.249 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T11:47:35.249 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 1594, 2024-04-25T11:47:35.249 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 1314, 2024-04-25T11:47:35.249 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 495475, 2024-04-25T11:47:35.249 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 749196, 2024-04-25T11:47:35.249 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 13, 2024-04-25T11:47:35.249 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 5, 2024-04-25T11:47:35.250 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.138119104, 2024-04-25T11:47:35.250 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.053327568, 2024-04-25T11:47:35.250 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.096364556, 2024-04-25T11:47:35.250 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.006041858, 2024-04-25T11:47:35.250 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:35.250 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1316, 2024-04-25T11:47:35.250 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.059001988, 2024-04-25T11:47:35.250 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000044834 2024-04-25T11:47:35.250 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.250 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:35.250 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2714, 2024-04-25T11:47:35.250 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000705469, 2024-04-25T11:47:35.250 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000259 2024-04-25T11:47:35.251 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.251 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 3456, 2024-04-25T11:47:35.251 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 139920 2024-04-25T11:47:35.251 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.251 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T11:47:35.251 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 487, 2024-04-25T11:47:35.251 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 468, 2024-04-25T11:47:35.251 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 1707761, 2024-04-25T11:47:35.251 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 712017, 2024-04-25T11:47:35.251 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 48, 2024-04-25T11:47:35.251 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 4, 2024-04-25T11:47:35.251 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.085624521, 2024-04-25T11:47:35.251 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.022556354, 2024-04-25T11:47:35.251 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.066173827, 2024-04-25T11:47:35.252 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.007930020, 2024-04-25T11:47:35.252 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:35.252 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 467, 2024-04-25T11:47:35.252 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.035062230, 2024-04-25T11:47:35.252 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000075079 2024-04-25T11:47:35.252 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.252 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:35.252 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 180, 2024-04-25T11:47:35.252 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000039783, 2024-04-25T11:47:35.252 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000221 2024-04-25T11:47:35.252 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.252 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 1641984, 2024-04-25T11:47:35.252 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 169712 2024-04-25T11:47:35.252 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.253 INFO:teuthology.orchestra.run.smithi156.stdout: "cct": { 2024-04-25T11:47:35.253 INFO:teuthology.orchestra.run.smithi156.stdout: "total_workers": 1, 2024-04-25T11:47:35.253 INFO:teuthology.orchestra.run.smithi156.stdout: "unhealthy_workers": 0 2024-04-25T11:47:35.253 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.253 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-MDSRank": { 2024-04-25T11:47:35.253 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-04-25T11:47:35.253 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-04-25T11:47:35.253 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 108, 2024-04-25T11:47:35.253 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.007820874, 2024-04-25T11:47:35.253 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000072415 2024-04-25T11:47:35.253 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:35.253 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.253 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-PurgeQueue": { 2024-04-25T11:47:35.253 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-04-25T11:47:35.254 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-04-25T11:47:35.254 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 17, 2024-04-25T11:47:35.254 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000526823, 2024-04-25T11:47:35.254 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000030989 2024-04-25T11:47:35.254 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:35.254 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.254 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": { 2024-04-25T11:47:35.254 INFO:teuthology.orchestra.run.smithi156.stdout: "request": 13, 2024-04-25T11:47:35.254 INFO:teuthology.orchestra.run.smithi156.stdout: "reply": 9, 2024-04-25T11:47:35.254 INFO:teuthology.orchestra.run.smithi156.stdout: "reply_latency": { 2024-04-25T11:47:35.254 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 9, 2024-04-25T11:47:35.254 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.006235569, 2024-04-25T11:47:35.254 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000692841 2024-04-25T11:47:35.255 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.255 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_reply": 0, 2024-04-25T11:47:35.255 INFO:teuthology.orchestra.run.smithi156.stdout: "forward": 4, 2024-04-25T11:47:35.255 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_complete": 0, 2024-04-25T11:47:35.255 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_keys": 0, 2024-04-25T11:47:35.255 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_commit": 11, 2024-04-25T11:47:35.255 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_split": 0, 2024-04-25T11:47:35.255 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_merge": 0, 2024-04-25T11:47:35.255 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes": 22, 2024-04-25T11:47:35.255 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_top": 1, 2024-04-25T11:47:35.255 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_bottom": 21, 2024-04-25T11:47:35.255 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pin_tail": 0, 2024-04-25T11:47:35.255 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pinned": 20, 2024-04-25T11:47:35.256 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_expired": 7, 2024-04-25T11:47:35.256 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_with_caps": 1, 2024-04-25T11:47:35.256 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": 1, 2024-04-25T11:47:35.256 INFO:teuthology.orchestra.run.smithi156.stdout: "subtrees": 4, 2024-04-25T11:47:35.256 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse": 31, 2024-04-25T11:47:35.256 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_hit": 20, 2024-04-25T11:47:35.256 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_forward": 2, 2024-04-25T11:47:35.256 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_discover": 4, 2024-04-25T11:47:35.256 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_dir_fetch": 0, 2024-04-25T11:47:35.256 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_remote_ino": 0, 2024-04-25T11:47:35.256 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_lock": 0, 2024-04-25T11:47:35.256 INFO:teuthology.orchestra.run.smithi156.stdout: "load_cent": 0, 2024-04-25T11:47:35.256 INFO:teuthology.orchestra.run.smithi156.stdout: "q": 0, 2024-04-25T11:47:35.256 INFO:teuthology.orchestra.run.smithi156.stdout: "exported": 0, 2024-04-25T11:47:35.256 INFO:teuthology.orchestra.run.smithi156.stdout: "exported_inodes": 0, 2024-04-25T11:47:35.256 INFO:teuthology.orchestra.run.smithi156.stdout: "imported": 1, 2024-04-25T11:47:35.257 INFO:teuthology.orchestra.run.smithi156.stdout: "imported_inodes": 2, 2024-04-25T11:47:35.257 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_dir_fetch": 0, 2024-04-25T11:47:35.257 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_backtrace_fetch": 34, 2024-04-25T11:47:35.257 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_peer_discover": 0, 2024-04-25T11:47:35.257 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rfiles": 22, 2024-04-25T11:47:35.257 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rbytes": 10738142561, 2024-04-25T11:47:35.257 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rsnaps": 6, 2024-04-25T11:47:35.257 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_fetch": 34, 2024-04-25T11:47:35.257 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_set_tag": 0, 2024-04-25T11:47:35.257 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T11:47:35.257 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_inotable_repaired": 0, 2024-04-25T11:47:35.257 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_inodes": 0, 2024-04-25T11:47:35.257 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T11:47:35.257 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T11:47:35.258 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_file_inodes": 34, 2024-04-25T11:47:35.258 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_inode_file_caps": 0, 2024-04-25T11:47:35.258 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_revoke": 1, 2024-04-25T11:47:35.258 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_grant": 3, 2024-04-25T11:47:35.258 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T11:47:35.258 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T11:47:35.258 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T11:47:35.258 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps": 1, 2024-04-25T11:47:35.258 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps_dirty": 0, 2024-04-25T11:47:35.258 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_cap_release": 2, 2024-04-25T11:47:35.258 INFO:teuthology.orchestra.run.smithi156.stdout: "process_request_cap_release": 4 2024-04-25T11:47:35.258 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.258 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_cache": { 2024-04-25T11:47:35.258 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update": 3, 2024-04-25T11:47:35.259 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update_receipt": 11, 2024-04-25T11:47:35.259 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_try_discover": 7, 2024-04-25T11:47:35.259 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_send_discover": 9, 2024-04-25T11:47:35.259 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_handle_discover": 3, 2024-04-25T11:47:35.259 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays": 0, 2024-04-25T11:47:35.259 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_delayed": 0, 2024-04-25T11:47:35.259 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_enqueuing": 0, 2024-04-25T11:47:35.259 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_created": 2, 2024-04-25T11:47:35.259 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_enqueued": 2, 2024-04-25T11:47:35.259 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_reintegrated": 0, 2024-04-25T11:47:35.259 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_migrated": 0, 2024-04-25T11:47:35.259 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_processing": 0, 2024-04-25T11:47:35.259 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_enqueued": 0, 2024-04-25T11:47:35.259 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_prioritized": 0, 2024-04-25T11:47:35.260 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_started": 0, 2024-04-25T11:47:35.260 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_completed": 0, 2024-04-25T11:47:35.260 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_path": 0, 2024-04-25T11:47:35.260 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_inode": 0, 2024-04-25T11:47:35.260 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T11:47:35.260 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_exportdir": 0, 2024-04-25T11:47:35.260 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_flush": 0, 2024-04-25T11:47:35.260 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragmentdir": 0, 2024-04-25T11:47:35.260 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragstats": 0, 2024-04-25T11:47:35.260 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_inodestats": 0 2024-04-25T11:47:35.260 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.260 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_log": { 2024-04-25T11:47:35.260 INFO:teuthology.orchestra.run.smithi156.stdout: "evlrg": 0, 2024-04-25T11:47:35.260 INFO:teuthology.orchestra.run.smithi156.stdout: "evadd": 22, 2024-04-25T11:47:35.260 INFO:teuthology.orchestra.run.smithi156.stdout: "evex": 0, 2024-04-25T11:47:35.261 INFO:teuthology.orchestra.run.smithi156.stdout: "evtrm": 0, 2024-04-25T11:47:35.261 INFO:teuthology.orchestra.run.smithi156.stdout: "ev": 22, 2024-04-25T11:47:35.261 INFO:teuthology.orchestra.run.smithi156.stdout: "evexg": 0, 2024-04-25T11:47:35.261 INFO:teuthology.orchestra.run.smithi156.stdout: "evexd": 0, 2024-04-25T11:47:35.261 INFO:teuthology.orchestra.run.smithi156.stdout: "segadd": 2, 2024-04-25T11:47:35.261 INFO:teuthology.orchestra.run.smithi156.stdout: "segex": 0, 2024-04-25T11:47:35.261 INFO:teuthology.orchestra.run.smithi156.stdout: "segtrm": 0, 2024-04-25T11:47:35.261 INFO:teuthology.orchestra.run.smithi156.stdout: "seg": 2, 2024-04-25T11:47:35.261 INFO:teuthology.orchestra.run.smithi156.stdout: "segmjr": 2, 2024-04-25T11:47:35.261 INFO:teuthology.orchestra.run.smithi156.stdout: "segexg": 0, 2024-04-25T11:47:35.261 INFO:teuthology.orchestra.run.smithi156.stdout: "segexd": 0, 2024-04-25T11:47:35.261 INFO:teuthology.orchestra.run.smithi156.stdout: "expos": 4194304, 2024-04-25T11:47:35.261 INFO:teuthology.orchestra.run.smithi156.stdout: "wrpos": 4228097, 2024-04-25T11:47:35.261 INFO:teuthology.orchestra.run.smithi156.stdout: "rdpos": 0, 2024-04-25T11:47:35.261 INFO:teuthology.orchestra.run.smithi156.stdout: "jlat": { 2024-04-25T11:47:35.261 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 17, 2024-04-25T11:47:35.262 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 1.771931299, 2024-04-25T11:47:35.262 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.104231252 2024-04-25T11:47:35.262 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.262 INFO:teuthology.orchestra.run.smithi156.stdout: "replayed": 0 2024-04-25T11:47:35.262 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.262 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_mem": { 2024-04-25T11:47:35.262 INFO:teuthology.orchestra.run.smithi156.stdout: "ino": 25, 2024-04-25T11:47:35.262 INFO:teuthology.orchestra.run.smithi156.stdout: "ino+": 30, 2024-04-25T11:47:35.262 INFO:teuthology.orchestra.run.smithi156.stdout: "ino-": 5, 2024-04-25T11:47:35.262 INFO:teuthology.orchestra.run.smithi156.stdout: "dir": 20, 2024-04-25T11:47:35.262 INFO:teuthology.orchestra.run.smithi156.stdout: "dir+": 24, 2024-04-25T11:47:35.262 INFO:teuthology.orchestra.run.smithi156.stdout: "dir-": 4, 2024-04-25T11:47:35.262 INFO:teuthology.orchestra.run.smithi156.stdout: "dn": 23, 2024-04-25T11:47:35.263 INFO:teuthology.orchestra.run.smithi156.stdout: "dn+": 32, 2024-04-25T11:47:35.263 INFO:teuthology.orchestra.run.smithi156.stdout: "dn-": 9, 2024-04-25T11:47:35.263 INFO:teuthology.orchestra.run.smithi156.stdout: "cap": 1, 2024-04-25T11:47:35.263 INFO:teuthology.orchestra.run.smithi156.stdout: "cap+": 8, 2024-04-25T11:47:35.263 INFO:teuthology.orchestra.run.smithi156.stdout: "cap-": 7, 2024-04-25T11:47:35.263 INFO:teuthology.orchestra.run.smithi156.stdout: "rss": 123132, 2024-04-25T11:47:35.263 INFO:teuthology.orchestra.run.smithi156.stdout: "heap": 223632 2024-04-25T11:47:35.263 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.263 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_server": { 2024-04-25T11:47:35.263 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_client_request": 17, 2024-04-25T11:47:35.263 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_server_request": 1, 2024-04-25T11:47:35.263 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_request": 13, 2024-04-25T11:47:35.263 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_session": 27, 2024-04-25T11:47:35.263 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_peer_request": 48, 2024-04-25T11:47:35.263 INFO:teuthology.orchestra.run.smithi156.stdout: "req_create_latency": { 2024-04-25T11:47:35.264 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.264 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.264 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.264 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.264 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getattr_latency": { 2024-04-25T11:47:35.264 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 4, 2024-04-25T11:47:35.264 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.001022619, 2024-04-25T11:47:35.264 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000255654 2024-04-25T11:47:35.264 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.264 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getfilelock_latency": { 2024-04-25T11:47:35.264 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.264 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.264 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.264 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.264 INFO:teuthology.orchestra.run.smithi156.stdout: "req_link_latency": { 2024-04-25T11:47:35.265 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.265 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.265 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.265 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.265 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookup_latency": { 2024-04-25T11:47:35.265 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2, 2024-04-25T11:47:35.265 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000502521, 2024-04-25T11:47:35.265 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000251260 2024-04-25T11:47:35.265 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.265 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookuphash_latency": { 2024-04-25T11:47:35.265 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.265 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.265 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.265 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.266 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupino_latency": { 2024-04-25T11:47:35.266 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.266 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.266 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.266 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.266 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupname_latency": { 2024-04-25T11:47:35.266 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.266 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.266 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.266 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.266 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupparent_latency": { 2024-04-25T11:47:35.266 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.267 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.267 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.267 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.267 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupsnap_latency": { 2024-04-25T11:47:35.267 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.267 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.267 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.267 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.267 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lssnap_latency": { 2024-04-25T11:47:35.267 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.267 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.267 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.267 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.268 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mkdir_latency": { 2024-04-25T11:47:35.268 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.268 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.268 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.268 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.269 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mknod_latency": { 2024-04-25T11:47:35.269 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.269 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.269 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.269 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.269 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mksnap_latency": { 2024-04-25T11:47:35.269 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.269 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.269 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.269 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.269 INFO:teuthology.orchestra.run.smithi156.stdout: "req_open_latency": { 2024-04-25T11:47:35.269 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.269 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.270 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.270 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.270 INFO:teuthology.orchestra.run.smithi156.stdout: "req_readdir_latency": { 2024-04-25T11:47:35.270 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1, 2024-04-25T11:47:35.270 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000527506, 2024-04-25T11:47:35.270 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000527506 2024-04-25T11:47:35.270 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.270 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rename_latency": { 2024-04-25T11:47:35.270 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.270 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.270 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.270 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.270 INFO:teuthology.orchestra.run.smithi156.stdout: "req_renamesnap_latency": { 2024-04-25T11:47:35.270 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.271 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.271 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.271 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.271 INFO:teuthology.orchestra.run.smithi156.stdout: "req_snapdiff_latency": { 2024-04-25T11:47:35.271 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.271 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.271 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.271 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.271 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmdir_latency": { 2024-04-25T11:47:35.271 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.271 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.271 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.271 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.271 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmsnap_latency": { 2024-04-25T11:47:35.271 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.272 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.272 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.272 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.272 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmxattr_latency": { 2024-04-25T11:47:35.272 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.272 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.272 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.272 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.272 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setattr_latency": { 2024-04-25T11:47:35.272 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.272 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.272 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.272 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.272 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setdirlayout_latency": { 2024-04-25T11:47:35.272 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.273 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.273 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.273 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.273 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setfilelock_latency": { 2024-04-25T11:47:35.273 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.273 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.273 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.273 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.273 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setlayout_latency": { 2024-04-25T11:47:35.273 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.273 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.273 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.273 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.273 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setxattr_latency": { 2024-04-25T11:47:35.273 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.274 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.274 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.274 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.274 INFO:teuthology.orchestra.run.smithi156.stdout: "req_symlink_latency": { 2024-04-25T11:47:35.274 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.274 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.274 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.274 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.274 INFO:teuthology.orchestra.run.smithi156.stdout: "req_unlink_latency": { 2024-04-25T11:47:35.274 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2, 2024-04-25T11:47:35.274 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.004182923, 2024-04-25T11:47:35.274 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.002091461 2024-04-25T11:47:35.274 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.275 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_revoke_eviction": 0, 2024-04-25T11:47:35.275 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_acquisition_throttle": 0, 2024-04-25T11:47:35.275 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getvxattr_latency": { 2024-04-25T11:47:35.275 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.275 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.275 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.275 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:35.275 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.275 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_sessions": { 2024-04-25T11:47:35.275 INFO:teuthology.orchestra.run.smithi156.stdout: "session_count": 2, 2024-04-25T11:47:35.275 INFO:teuthology.orchestra.run.smithi156.stdout: "session_add": 4, 2024-04-25T11:47:35.275 INFO:teuthology.orchestra.run.smithi156.stdout: "session_remove": 2, 2024-04-25T11:47:35.276 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_open": 2, 2024-04-25T11:47:35.276 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_stale": 0, 2024-04-25T11:47:35.276 INFO:teuthology.orchestra.run.smithi156.stdout: "total_load": 4, 2024-04-25T11:47:35.276 INFO:teuthology.orchestra.run.smithi156.stdout: "average_load": 2, 2024-04-25T11:47:35.276 INFO:teuthology.orchestra.run.smithi156.stdout: "avg_session_uptime": 719, 2024-04-25T11:47:35.276 INFO:teuthology.orchestra.run.smithi156.stdout: "mdthresh_evicted": 0 2024-04-25T11:47:35.276 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.276 INFO:teuthology.orchestra.run.smithi156.stdout: "mempool": { 2024-04-25T11:47:35.276 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_bytes": 0, 2024-04-25T11:47:35.276 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_items": 0, 2024-04-25T11:47:35.276 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T11:47:35.276 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_items": 0, 2024-04-25T11:47:35.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T11:47:35.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_items": 0, 2024-04-25T11:47:35.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T11:47:35.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T11:47:35.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T11:47:35.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T11:47:35.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T11:47:35.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_items": 0, 2024-04-25T11:47:35.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T11:47:35.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T11:47:35.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_bytes": 0, 2024-04-25T11:47:35.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_items": 0, 2024-04-25T11:47:35.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_bytes": 0, 2024-04-25T11:47:35.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_items": 0, 2024-04-25T11:47:35.278 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T11:47:35.278 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T11:47:35.278 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T11:47:35.278 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T11:47:35.278 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T11:47:35.278 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_items": 0, 2024-04-25T11:47:35.278 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_bytes": 0, 2024-04-25T11:47:35.278 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_items": 0, 2024-04-25T11:47:35.278 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T11:47:35.278 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T11:47:35.279 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_bytes": 0, 2024-04-25T11:47:35.279 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_items": 0, 2024-04-25T11:47:35.279 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_bytes": 0, 2024-04-25T11:47:35.279 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_items": 0, 2024-04-25T11:47:35.279 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T11:47:35.279 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_items": 0, 2024-04-25T11:47:35.279 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T11:47:35.279 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_items": 0, 2024-04-25T11:47:35.279 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_bytes": 287152, 2024-04-25T11:47:35.279 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_items": 101, 2024-04-25T11:47:35.279 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_bytes": 0, 2024-04-25T11:47:35.279 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_items": 0, 2024-04-25T11:47:35.279 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_bytes": 0, 2024-04-25T11:47:35.280 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_items": 0, 2024-04-25T11:47:35.280 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_bytes": 0, 2024-04-25T11:47:35.280 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_items": 0, 2024-04-25T11:47:35.280 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_bytes": 0, 2024-04-25T11:47:35.280 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_items": 0, 2024-04-25T11:47:35.280 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_bytes": 8632, 2024-04-25T11:47:35.280 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_items": 180, 2024-04-25T11:47:35.280 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T11:47:35.280 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_items": 0, 2024-04-25T11:47:35.280 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_bytes": 0, 2024-04-25T11:47:35.280 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_items": 0, 2024-04-25T11:47:35.280 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_bytes": 103569, 2024-04-25T11:47:35.280 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_items": 519, 2024-04-25T11:47:35.280 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_bytes": 0, 2024-04-25T11:47:35.281 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_items": 0, 2024-04-25T11:47:35.281 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_bytes": 0, 2024-04-25T11:47:35.281 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_items": 0 2024-04-25T11:47:35.281 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.281 INFO:teuthology.orchestra.run.smithi156.stdout: "objecter": { 2024-04-25T11:47:35.281 INFO:teuthology.orchestra.run.smithi156.stdout: "op_active": 0, 2024-04-25T11:47:35.281 INFO:teuthology.orchestra.run.smithi156.stdout: "op_laggy": 0, 2024-04-25T11:47:35.281 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send": 109, 2024-04-25T11:47:35.281 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send_bytes": 46609, 2024-04-25T11:47:35.281 INFO:teuthology.orchestra.run.smithi156.stdout: "op_resend": 28, 2024-04-25T11:47:35.281 INFO:teuthology.orchestra.run.smithi156.stdout: "op_reply": 109, 2024-04-25T11:47:35.281 INFO:teuthology.orchestra.run.smithi156.stdout: "op_latency": { 2024-04-25T11:47:35.282 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 109, 2024-04-25T11:47:35.282 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 4.573927388, 2024-04-25T11:47:35.282 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.041962636 2024-04-25T11:47:35.282 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.282 INFO:teuthology.orchestra.run.smithi156.stdout: "op_inflight": 0, 2024-04-25T11:47:35.282 INFO:teuthology.orchestra.run.smithi156.stdout: "oplen_avg": { 2024-04-25T11:47:35.282 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 109, 2024-04-25T11:47:35.282 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 137 2024-04-25T11:47:35.282 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.282 INFO:teuthology.orchestra.run.smithi156.stdout: "op": 109, 2024-04-25T11:47:35.282 INFO:teuthology.orchestra.run.smithi156.stdout: "op_r": 35, 2024-04-25T11:47:35.282 INFO:teuthology.orchestra.run.smithi156.stdout: "op_w": 74, 2024-04-25T11:47:35.283 INFO:teuthology.orchestra.run.smithi156.stdout: "op_rmw": 0, 2024-04-25T11:47:35.283 INFO:teuthology.orchestra.run.smithi156.stdout: "op_pg": 0, 2024-04-25T11:47:35.283 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_stat": 0, 2024-04-25T11:47:35.283 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_create": 10, 2024-04-25T11:47:35.283 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_read": 1, 2024-04-25T11:47:35.283 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_write": 18, 2024-04-25T11:47:35.283 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writefull": 13, 2024-04-25T11:47:35.283 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writesame": 0, 2024-04-25T11:47:35.283 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_append": 0, 2024-04-25T11:47:35.283 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_zero": 0, 2024-04-25T11:47:35.283 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_truncate": 0, 2024-04-25T11:47:35.283 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_delete": 14, 2024-04-25T11:47:35.283 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_mapext": 0, 2024-04-25T11:47:35.283 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_sparse_read": 0, 2024-04-25T11:47:35.284 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_clonerange": 0, 2024-04-25T11:47:35.284 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_getxattr": 34, 2024-04-25T11:47:35.284 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_setxattr": 20, 2024-04-25T11:47:35.284 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_cmpxattr": 0, 2024-04-25T11:47:35.284 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_rmxattr": 0, 2024-04-25T11:47:35.284 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_resetxattrs": 0, 2024-04-25T11:47:35.284 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_call": 0, 2024-04-25T11:47:35.284 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_watch": 0, 2024-04-25T11:47:35.284 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_notify": 0, 2024-04-25T11:47:35.284 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T11:47:35.284 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls": 0, 2024-04-25T11:47:35.284 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls_filter": 0, 2024-04-25T11:47:35.284 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_other": 0, 2024-04-25T11:47:35.284 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_active": 0, 2024-04-25T11:47:35.284 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_send": 0, 2024-04-25T11:47:35.285 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_resend": 0, 2024-04-25T11:47:35.285 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_ping": 0, 2024-04-25T11:47:35.285 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_active": 0, 2024-04-25T11:47:35.285 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_send": 0, 2024-04-25T11:47:35.285 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_resend": 0, 2024-04-25T11:47:35.285 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_active": 0, 2024-04-25T11:47:35.285 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_send": 0, 2024-04-25T11:47:35.285 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_resend": 0, 2024-04-25T11:47:35.285 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_active": 0, 2024-04-25T11:47:35.285 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_send": 0, 2024-04-25T11:47:35.285 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_resend": 0, 2024-04-25T11:47:35.285 INFO:teuthology.orchestra.run.smithi156.stdout: "command_active": 0, 2024-04-25T11:47:35.285 INFO:teuthology.orchestra.run.smithi156.stdout: "command_send": 0, 2024-04-25T11:47:35.285 INFO:teuthology.orchestra.run.smithi156.stdout: "command_resend": 0, 2024-04-25T11:47:35.285 INFO:teuthology.orchestra.run.smithi156.stdout: "map_epoch": 94, 2024-04-25T11:47:35.285 INFO:teuthology.orchestra.run.smithi156.stdout: "map_full": 0, 2024-04-25T11:47:35.286 INFO:teuthology.orchestra.run.smithi156.stdout: "map_inc": 17, 2024-04-25T11:47:35.286 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_sessions": 9, 2024-04-25T11:47:35.286 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_open": 9, 2024-04-25T11:47:35.286 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_close": 0, 2024-04-25T11:47:35.286 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_laggy": 0, 2024-04-25T11:47:35.286 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_wr": 23, 2024-04-25T11:47:35.286 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_rd": 0, 2024-04-25T11:47:35.286 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_del": 4 2024-04-25T11:47:35.286 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.286 INFO:teuthology.orchestra.run.smithi156.stdout: "oft": { 2024-04-25T11:47:35.286 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_objs": 1, 2024-04-25T11:47:35.286 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_kv_pairs": 0, 2024-04-25T11:47:35.286 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_updates": 0, 2024-04-25T11:47:35.286 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_removes": 0 2024-04-25T11:47:35.287 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.287 INFO:teuthology.orchestra.run.smithi156.stdout: "osd-slow-ops": { 2024-04-25T11:47:35.287 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_ops_count": 0 2024-04-25T11:47:35.287 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.287 INFO:teuthology.orchestra.run.smithi156.stdout: "purge_queue": { 2024-04-25T11:47:35.287 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops": 0, 2024-04-25T11:47:35.287 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops_high_water": 2, 2024-04-25T11:47:35.287 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing": 0, 2024-04-25T11:47:35.287 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_high_water": 2, 2024-04-25T11:47:35.287 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed_ops": 2, 2024-04-25T11:47:35.287 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed": 2, 2024-04-25T11:47:35.287 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_item_in_journal": 0 2024-04-25T11:47:35.287 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.288 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T11:47:35.288 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T11:47:35.288 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-04-25T11:47:35.288 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T11:47:35.288 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 2216, 2024-04-25T11:47:35.288 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 2052940, 2024-04-25T11:47:35.288 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:35.288 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 2216, 2024-04-25T11:47:35.288 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-04-25T11:47:35.288 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-04-25T11:47:35.288 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 2216, 2024-04-25T11:47:35.288 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 2052940, 2024-04-25T11:47:35.288 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T11:47:35.288 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.288 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.289 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.289 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:35.289 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.289 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_bytes": { 2024-04-25T11:47:35.289 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T11:47:35.289 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-04-25T11:47:35.289 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T11:47:35.289 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-04-25T11:47:35.289 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-04-25T11:47:35.289 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:35.289 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:35.289 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 109, 2024-04-25T11:47:35.289 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 46907, 2024-04-25T11:47:35.289 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 109, 2024-04-25T11:47:35.289 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 46907, 2024-04-25T11:47:35.289 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T11:47:35.290 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.290 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.290 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.290 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:35.290 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.290 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_ops": { 2024-04-25T11:47:35.290 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T11:47:35.290 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 1024, 2024-04-25T11:47:35.290 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T11:47:35.290 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-04-25T11:47:35.290 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-04-25T11:47:35.290 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:35.290 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:35.291 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 109, 2024-04-25T11:47:35.291 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 109, 2024-04-25T11:47:35.291 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 109, 2024-04-25T11:47:35.291 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 109, 2024-04-25T11:47:35.291 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T11:47:35.291 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.291 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.291 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.291 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:35.291 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.291 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle": { 2024-04-25T11:47:35.291 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T11:47:35.291 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-04-25T11:47:35.291 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T11:47:35.292 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 2, 2024-04-25T11:47:35.292 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 298, 2024-04-25T11:47:35.292 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:35.292 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 2, 2024-04-25T11:47:35.292 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-04-25T11:47:35.292 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-04-25T11:47:35.292 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 1, 2024-04-25T11:47:35.292 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 298, 2024-04-25T11:47:35.292 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T11:47:35.292 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.292 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.292 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.292 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:35.292 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:35.292 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle-0x5565567e4750": { 2024-04-25T11:47:35.292 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T11:47:35.293 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-04-25T11:47:35.293 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T11:47:35.293 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 22, 2024-04-25T11:47:35.293 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 33793, 2024-04-25T11:47:35.293 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:35.293 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 22, 2024-04-25T11:47:35.293 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-04-25T11:47:35.293 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-04-25T11:47:35.293 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 17, 2024-04-25T11:47:35.293 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 33793, 2024-04-25T11:47:35.293 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T11:47:35.293 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:35.294 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:35.294 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:35.294 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:35.294 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:35.294 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-04-25T11:47:36.097 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-04-25T11:47:36.097 INFO:tasks.check_counter:Daemon mds.i mds.imported=1 2024-04-25T11:47:36.097 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=3 2024-04-25T11:47:36.097 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=11 2024-04-25T11:47:36.097 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-04-25T11:47:36.097 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-25T11:47:36.097 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T11:47:36.097 DEBUG:tasks.check_counter:Getting stats from l 2024-04-25T11:47:36.098 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-25T11:47:36.338 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:36 smithi156 ceph-mon[33045]: pgmap v390: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T11:47:36.338 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T11:47:36.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:36 smithi090 ceph-mon[29575]: pgmap v390: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T11:47:36.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:36 smithi138 ceph-mon[32328]: pgmap v390: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T11:47:38.001 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-04-25T11:47:38.001 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T11:47:38.001 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 864, 2024-04-25T11:47:38.001 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 864, 2024-04-25T11:47:38.001 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 230831, 2024-04-25T11:47:38.002 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 273992, 2024-04-25T11:47:38.002 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 6, 2024-04-25T11:47:38.002 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 2, 2024-04-25T11:47:38.002 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.082384867, 2024-04-25T11:47:38.002 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.025677728, 2024-04-25T11:47:38.002 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.022112799, 2024-04-25T11:47:38.002 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.025769885, 2024-04-25T11:47:38.002 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:38.002 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 863, 2024-04-25T11:47:38.002 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.035713143, 2024-04-25T11:47:38.002 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000041382 2024-04-25T11:47:38.002 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.002 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:38.003 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.003 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.003 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.003 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.003 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-25T11:47:38.003 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T11:47:38.003 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.003 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T11:47:38.003 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 1042, 2024-04-25T11:47:38.003 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 1028, 2024-04-25T11:47:38.003 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 1789806, 2024-04-25T11:47:38.003 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 395661, 2024-04-25T11:47:38.003 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 4, 2024-04-25T11:47:38.004 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 3, 2024-04-25T11:47:38.004 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.135590389, 2024-04-25T11:47:38.004 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.035545391, 2024-04-25T11:47:38.004 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.068656518, 2024-04-25T11:47:38.004 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.038908420, 2024-04-25T11:47:38.004 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:38.004 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1027, 2024-04-25T11:47:38.004 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.036982325, 2024-04-25T11:47:38.004 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000036010 2024-04-25T11:47:38.004 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.004 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:38.004 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.004 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.004 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.005 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.005 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 1601840, 2024-04-25T11:47:38.005 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 152704 2024-04-25T11:47:38.005 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.005 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T11:47:38.005 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 854, 2024-04-25T11:47:38.005 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 1037, 2024-04-25T11:47:38.005 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 286592, 2024-04-25T11:47:38.005 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 755496, 2024-04-25T11:47:38.005 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 6, 2024-04-25T11:47:38.005 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 4, 2024-04-25T11:47:38.005 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.116386435, 2024-04-25T11:47:38.005 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.039937305, 2024-04-25T11:47:38.005 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.029613528, 2024-04-25T11:47:38.006 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.034986053, 2024-04-25T11:47:38.006 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T11:47:38.006 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1034, 2024-04-25T11:47:38.006 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.069299181, 2024-04-25T11:47:38.006 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000067020 2024-04-25T11:47:38.006 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.006 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-04-25T11:47:38.006 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.006 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.006 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.006 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.006 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T11:47:38.006 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 484560 2024-04-25T11:47:38.006 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.007 INFO:teuthology.orchestra.run.smithi156.stdout: "cct": { 2024-04-25T11:47:38.007 INFO:teuthology.orchestra.run.smithi156.stdout: "total_workers": 1, 2024-04-25T11:47:38.007 INFO:teuthology.orchestra.run.smithi156.stdout: "unhealthy_workers": 0 2024-04-25T11:47:38.007 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.007 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-MDSRank": { 2024-04-25T11:47:38.007 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-04-25T11:47:38.007 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-04-25T11:47:38.007 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 4113, 2024-04-25T11:47:38.007 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.134714379, 2024-04-25T11:47:38.007 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000032753 2024-04-25T11:47:38.007 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:38.007 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.007 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-PurgeQueue": { 2024-04-25T11:47:38.007 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-04-25T11:47:38.007 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-04-25T11:47:38.008 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.008 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.008 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.008 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:38.008 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.008 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": { 2024-04-25T11:47:38.008 INFO:teuthology.orchestra.run.smithi156.stdout: "request": 0, 2024-04-25T11:47:38.008 INFO:teuthology.orchestra.run.smithi156.stdout: "reply": 0, 2024-04-25T11:47:38.008 INFO:teuthology.orchestra.run.smithi156.stdout: "reply_latency": { 2024-04-25T11:47:38.008 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.008 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.008 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.008 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.008 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_reply": 0, 2024-04-25T11:47:38.008 INFO:teuthology.orchestra.run.smithi156.stdout: "forward": 0, 2024-04-25T11:47:38.009 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_complete": 0, 2024-04-25T11:47:38.009 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_keys": 0, 2024-04-25T11:47:38.009 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_commit": 0, 2024-04-25T11:47:38.009 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_split": 0, 2024-04-25T11:47:38.009 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_merge": 0, 2024-04-25T11:47:38.009 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes": 0, 2024-04-25T11:47:38.009 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_top": 0, 2024-04-25T11:47:38.009 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_bottom": 0, 2024-04-25T11:47:38.009 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pin_tail": 0, 2024-04-25T11:47:38.009 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pinned": 0, 2024-04-25T11:47:38.009 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_expired": 0, 2024-04-25T11:47:38.009 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_with_caps": 0, 2024-04-25T11:47:38.009 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": 0, 2024-04-25T11:47:38.009 INFO:teuthology.orchestra.run.smithi156.stdout: "subtrees": 1, 2024-04-25T11:47:38.010 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse": 0, 2024-04-25T11:47:38.010 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_hit": 0, 2024-04-25T11:47:38.010 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_forward": 0, 2024-04-25T11:47:38.010 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_discover": 0, 2024-04-25T11:47:38.010 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_dir_fetch": 0, 2024-04-25T11:47:38.010 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_remote_ino": 0, 2024-04-25T11:47:38.010 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_lock": 0, 2024-04-25T11:47:38.010 INFO:teuthology.orchestra.run.smithi156.stdout: "load_cent": 0, 2024-04-25T11:47:38.010 INFO:teuthology.orchestra.run.smithi156.stdout: "q": 0, 2024-04-25T11:47:38.010 INFO:teuthology.orchestra.run.smithi156.stdout: "exported": 0, 2024-04-25T11:47:38.010 INFO:teuthology.orchestra.run.smithi156.stdout: "exported_inodes": 0, 2024-04-25T11:47:38.010 INFO:teuthology.orchestra.run.smithi156.stdout: "imported": 0, 2024-04-25T11:47:38.010 INFO:teuthology.orchestra.run.smithi156.stdout: "imported_inodes": 0, 2024-04-25T11:47:38.010 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_dir_fetch": 0, 2024-04-25T11:47:38.011 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_backtrace_fetch": 0, 2024-04-25T11:47:38.011 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_peer_discover": 0, 2024-04-25T11:47:38.011 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rfiles": 0, 2024-04-25T11:47:38.011 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rbytes": 0, 2024-04-25T11:47:38.011 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rsnaps": 0, 2024-04-25T11:47:38.011 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T11:47:38.011 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_set_tag": 0, 2024-04-25T11:47:38.011 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T11:47:38.011 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_inotable_repaired": 0, 2024-04-25T11:47:38.011 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_inodes": 0, 2024-04-25T11:47:38.011 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T11:47:38.011 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T11:47:38.011 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_file_inodes": 0, 2024-04-25T11:47:38.011 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_inode_file_caps": 0, 2024-04-25T11:47:38.011 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T11:47:38.012 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_grant": 0, 2024-04-25T11:47:38.012 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T11:47:38.012 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T11:47:38.012 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T11:47:38.012 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps": 0, 2024-04-25T11:47:38.012 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps_dirty": 0, 2024-04-25T11:47:38.012 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_cap_release": 0, 2024-04-25T11:47:38.012 INFO:teuthology.orchestra.run.smithi156.stdout: "process_request_cap_release": 0 2024-04-25T11:47:38.012 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.012 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_cache": { 2024-04-25T11:47:38.012 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update": 0, 2024-04-25T11:47:38.012 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update_receipt": 0, 2024-04-25T11:47:38.012 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_try_discover": 0, 2024-04-25T11:47:38.012 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_send_discover": 0, 2024-04-25T11:47:38.013 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_handle_discover": 0, 2024-04-25T11:47:38.013 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays": 0, 2024-04-25T11:47:38.013 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_delayed": 0, 2024-04-25T11:47:38.013 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_enqueuing": 0, 2024-04-25T11:47:38.013 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_created": 0, 2024-04-25T11:47:38.013 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_enqueued": 0, 2024-04-25T11:47:38.013 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_reintegrated": 0, 2024-04-25T11:47:38.013 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_migrated": 0, 2024-04-25T11:47:38.013 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_processing": 0, 2024-04-25T11:47:38.013 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_enqueued": 0, 2024-04-25T11:47:38.013 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_prioritized": 0, 2024-04-25T11:47:38.013 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_started": 0, 2024-04-25T11:47:38.013 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_completed": 0, 2024-04-25T11:47:38.013 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_path": 0, 2024-04-25T11:47:38.014 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_inode": 0, 2024-04-25T11:47:38.014 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T11:47:38.014 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_exportdir": 0, 2024-04-25T11:47:38.014 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_flush": 0, 2024-04-25T11:47:38.014 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragmentdir": 0, 2024-04-25T11:47:38.014 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragstats": 0, 2024-04-25T11:47:38.014 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_inodestats": 0 2024-04-25T11:47:38.014 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.014 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_log": { 2024-04-25T11:47:38.014 INFO:teuthology.orchestra.run.smithi156.stdout: "evlrg": 0, 2024-04-25T11:47:38.014 INFO:teuthology.orchestra.run.smithi156.stdout: "evadd": 0, 2024-04-25T11:47:38.014 INFO:teuthology.orchestra.run.smithi156.stdout: "evex": 0, 2024-04-25T11:47:38.014 INFO:teuthology.orchestra.run.smithi156.stdout: "evtrm": 0, 2024-04-25T11:47:38.014 INFO:teuthology.orchestra.run.smithi156.stdout: "ev": 7, 2024-04-25T11:47:38.015 INFO:teuthology.orchestra.run.smithi156.stdout: "evexg": 0, 2024-04-25T11:47:38.015 INFO:teuthology.orchestra.run.smithi156.stdout: "evexd": 0, 2024-04-25T11:47:38.015 INFO:teuthology.orchestra.run.smithi156.stdout: "segadd": 0, 2024-04-25T11:47:38.015 INFO:teuthology.orchestra.run.smithi156.stdout: "segex": 0, 2024-04-25T11:47:38.015 INFO:teuthology.orchestra.run.smithi156.stdout: "segtrm": 0, 2024-04-25T11:47:38.015 INFO:teuthology.orchestra.run.smithi156.stdout: "seg": 2, 2024-04-25T11:47:38.015 INFO:teuthology.orchestra.run.smithi156.stdout: "segmjr": 2, 2024-04-25T11:47:38.015 INFO:teuthology.orchestra.run.smithi156.stdout: "segexg": 0, 2024-04-25T11:47:38.015 INFO:teuthology.orchestra.run.smithi156.stdout: "segexd": 0, 2024-04-25T11:47:38.015 INFO:teuthology.orchestra.run.smithi156.stdout: "expos": 4194304, 2024-04-25T11:47:38.015 INFO:teuthology.orchestra.run.smithi156.stdout: "wrpos": 4196264, 2024-04-25T11:47:38.015 INFO:teuthology.orchestra.run.smithi156.stdout: "rdpos": 4196123, 2024-04-25T11:47:38.015 INFO:teuthology.orchestra.run.smithi156.stdout: "jlat": { 2024-04-25T11:47:38.015 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.016 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.016 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.016 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.016 INFO:teuthology.orchestra.run.smithi156.stdout: "replayed": 7 2024-04-25T11:47:38.016 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.016 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_mem": { 2024-04-25T11:47:38.016 INFO:teuthology.orchestra.run.smithi156.stdout: "ino": 3, 2024-04-25T11:47:38.016 INFO:teuthology.orchestra.run.smithi156.stdout: "ino+": 3, 2024-04-25T11:47:38.016 INFO:teuthology.orchestra.run.smithi156.stdout: "ino-": 0, 2024-04-25T11:47:38.016 INFO:teuthology.orchestra.run.smithi156.stdout: "dir": 1, 2024-04-25T11:47:38.016 INFO:teuthology.orchestra.run.smithi156.stdout: "dir+": 1, 2024-04-25T11:47:38.016 INFO:teuthology.orchestra.run.smithi156.stdout: "dir-": 0, 2024-04-25T11:47:38.016 INFO:teuthology.orchestra.run.smithi156.stdout: "dn": 0, 2024-04-25T11:47:38.016 INFO:teuthology.orchestra.run.smithi156.stdout: "dn+": 0, 2024-04-25T11:47:38.016 INFO:teuthology.orchestra.run.smithi156.stdout: "dn-": 0, 2024-04-25T11:47:38.017 INFO:teuthology.orchestra.run.smithi156.stdout: "cap": 0, 2024-04-25T11:47:38.017 INFO:teuthology.orchestra.run.smithi156.stdout: "cap+": 0, 2024-04-25T11:47:38.017 INFO:teuthology.orchestra.run.smithi156.stdout: "cap-": 0, 2024-04-25T11:47:38.017 INFO:teuthology.orchestra.run.smithi156.stdout: "rss": 120148, 2024-04-25T11:47:38.017 INFO:teuthology.orchestra.run.smithi156.stdout: "heap": 215440 2024-04-25T11:47:38.017 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.017 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_server": { 2024-04-25T11:47:38.017 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_client_request": 0, 2024-04-25T11:47:38.017 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_server_request": 0, 2024-04-25T11:47:38.017 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_request": 0, 2024-04-25T11:47:38.017 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_session": 0, 2024-04-25T11:47:38.017 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_peer_request": 0, 2024-04-25T11:47:38.017 INFO:teuthology.orchestra.run.smithi156.stdout: "req_create_latency": { 2024-04-25T11:47:38.017 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.018 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.018 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.018 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.018 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getattr_latency": { 2024-04-25T11:47:38.018 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.018 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.018 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.018 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.018 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getfilelock_latency": { 2024-04-25T11:47:38.018 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.018 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.018 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.018 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.018 INFO:teuthology.orchestra.run.smithi156.stdout: "req_link_latency": { 2024-04-25T11:47:38.019 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.019 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.019 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.019 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.019 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookup_latency": { 2024-04-25T11:47:38.019 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.019 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.019 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.019 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.019 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookuphash_latency": { 2024-04-25T11:47:38.019 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.019 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.019 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.019 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.020 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupino_latency": { 2024-04-25T11:47:38.020 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.020 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.020 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.020 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.020 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupname_latency": { 2024-04-25T11:47:38.020 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.020 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.020 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.020 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.020 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupparent_latency": { 2024-04-25T11:47:38.020 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.020 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.020 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.020 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.021 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupsnap_latency": { 2024-04-25T11:47:38.021 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.021 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.021 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.021 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.021 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lssnap_latency": { 2024-04-25T11:47:38.021 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.021 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.021 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.021 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.021 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mkdir_latency": { 2024-04-25T11:47:38.021 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.022 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.022 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.022 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.022 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mknod_latency": { 2024-04-25T11:47:38.022 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.022 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.022 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.022 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.022 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mksnap_latency": { 2024-04-25T11:47:38.023 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.023 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.023 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.023 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.023 INFO:teuthology.orchestra.run.smithi156.stdout: "req_open_latency": { 2024-04-25T11:47:38.023 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.023 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.023 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.023 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.023 INFO:teuthology.orchestra.run.smithi156.stdout: "req_readdir_latency": { 2024-04-25T11:47:38.023 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.023 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.023 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.023 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.023 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rename_latency": { 2024-04-25T11:47:38.024 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.024 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.024 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.024 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.024 INFO:teuthology.orchestra.run.smithi156.stdout: "req_renamesnap_latency": { 2024-04-25T11:47:38.024 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.024 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.024 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.024 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.024 INFO:teuthology.orchestra.run.smithi156.stdout: "req_snapdiff_latency": { 2024-04-25T11:47:38.024 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.024 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.024 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.024 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.025 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmdir_latency": { 2024-04-25T11:47:38.025 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.025 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.025 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.025 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.025 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmsnap_latency": { 2024-04-25T11:47:38.025 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.025 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.025 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.025 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.025 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmxattr_latency": { 2024-04-25T11:47:38.025 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.025 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.025 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.026 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.026 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setattr_latency": { 2024-04-25T11:47:38.026 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.026 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.026 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.026 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.026 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setdirlayout_latency": { 2024-04-25T11:47:38.026 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.026 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.026 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.026 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.026 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setfilelock_latency": { 2024-04-25T11:47:38.026 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.026 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.027 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.027 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.027 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setlayout_latency": { 2024-04-25T11:47:38.027 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.027 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.027 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.027 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.027 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setxattr_latency": { 2024-04-25T11:47:38.027 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.027 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.027 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.027 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.027 INFO:teuthology.orchestra.run.smithi156.stdout: "req_symlink_latency": { 2024-04-25T11:47:38.028 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.028 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.028 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.028 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.028 INFO:teuthology.orchestra.run.smithi156.stdout: "req_unlink_latency": { 2024-04-25T11:47:38.028 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.028 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.028 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.028 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.028 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_revoke_eviction": 0, 2024-04-25T11:47:38.028 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_acquisition_throttle": 0, 2024-04-25T11:47:38.028 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getvxattr_latency": { 2024-04-25T11:47:38.028 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.028 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.028 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.029 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:38.029 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.029 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_sessions": { 2024-04-25T11:47:38.029 INFO:teuthology.orchestra.run.smithi156.stdout: "session_count": 1, 2024-04-25T11:47:38.029 INFO:teuthology.orchestra.run.smithi156.stdout: "session_add": 3, 2024-04-25T11:47:38.029 INFO:teuthology.orchestra.run.smithi156.stdout: "session_remove": 2, 2024-04-25T11:47:38.029 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_open": 2, 2024-04-25T11:47:38.029 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_stale": 0, 2024-04-25T11:47:38.029 INFO:teuthology.orchestra.run.smithi156.stdout: "total_load": 0, 2024-04-25T11:47:38.029 INFO:teuthology.orchestra.run.smithi156.stdout: "average_load": 0, 2024-04-25T11:47:38.029 INFO:teuthology.orchestra.run.smithi156.stdout: "avg_session_uptime": 3146, 2024-04-25T11:47:38.029 INFO:teuthology.orchestra.run.smithi156.stdout: "mdthresh_evicted": 0 2024-04-25T11:47:38.029 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.029 INFO:teuthology.orchestra.run.smithi156.stdout: "mempool": { 2024-04-25T11:47:38.030 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_bytes": 0, 2024-04-25T11:47:38.030 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_items": 0, 2024-04-25T11:47:38.030 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T11:47:38.030 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_items": 0, 2024-04-25T11:47:38.030 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T11:47:38.030 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_items": 0, 2024-04-25T11:47:38.030 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T11:47:38.030 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T11:47:38.030 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T11:47:38.030 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T11:47:38.030 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T11:47:38.030 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_items": 0, 2024-04-25T11:47:38.030 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T11:47:38.031 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T11:47:38.031 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_bytes": 0, 2024-04-25T11:47:38.031 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_items": 0, 2024-04-25T11:47:38.031 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_bytes": 0, 2024-04-25T11:47:38.031 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_items": 0, 2024-04-25T11:47:38.031 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T11:47:38.031 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T11:47:38.031 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T11:47:38.031 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T11:47:38.031 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T11:47:38.031 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_items": 0, 2024-04-25T11:47:38.031 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_bytes": 0, 2024-04-25T11:47:38.031 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_items": 0, 2024-04-25T11:47:38.031 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T11:47:38.031 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T11:47:38.032 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_bytes": 0, 2024-04-25T11:47:38.032 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_items": 0, 2024-04-25T11:47:38.032 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_bytes": 0, 2024-04-25T11:47:38.032 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_items": 0, 2024-04-25T11:47:38.032 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T11:47:38.032 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_items": 0, 2024-04-25T11:47:38.032 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T11:47:38.032 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_items": 0, 2024-04-25T11:47:38.032 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_bytes": 189383, 2024-04-25T11:47:38.032 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_items": 50, 2024-04-25T11:47:38.032 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_bytes": 0, 2024-04-25T11:47:38.032 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_items": 0, 2024-04-25T11:47:38.032 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_bytes": 0, 2024-04-25T11:47:38.032 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_items": 0, 2024-04-25T11:47:38.033 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_bytes": 0, 2024-04-25T11:47:38.033 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_items": 0, 2024-04-25T11:47:38.033 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_bytes": 0, 2024-04-25T11:47:38.033 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_items": 0, 2024-04-25T11:47:38.033 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_bytes": 8512, 2024-04-25T11:47:38.033 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_items": 177, 2024-04-25T11:47:38.033 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T11:47:38.033 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_items": 0, 2024-04-25T11:47:38.033 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_bytes": 0, 2024-04-25T11:47:38.033 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_items": 0, 2024-04-25T11:47:38.033 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_bytes": 8512, 2024-04-25T11:47:38.033 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_items": 13, 2024-04-25T11:47:38.033 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_bytes": 0, 2024-04-25T11:47:38.033 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_items": 0, 2024-04-25T11:47:38.033 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_bytes": 0, 2024-04-25T11:47:38.034 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_items": 0 2024-04-25T11:47:38.034 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.034 INFO:teuthology.orchestra.run.smithi156.stdout: "objecter": { 2024-04-25T11:47:38.034 INFO:teuthology.orchestra.run.smithi156.stdout: "op_active": 0, 2024-04-25T11:47:38.034 INFO:teuthology.orchestra.run.smithi156.stdout: "op_laggy": 0, 2024-04-25T11:47:38.034 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send": 2476, 2024-04-25T11:47:38.034 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send_bytes": 21, 2024-04-25T11:47:38.034 INFO:teuthology.orchestra.run.smithi156.stdout: "op_resend": 3, 2024-04-25T11:47:38.034 INFO:teuthology.orchestra.run.smithi156.stdout: "op_reply": 2476, 2024-04-25T11:47:38.034 INFO:teuthology.orchestra.run.smithi156.stdout: "op_latency": { 2024-04-25T11:47:38.034 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2476, 2024-04-25T11:47:38.034 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 9.813843429, 2024-04-25T11:47:38.034 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.003963587 2024-04-25T11:47:38.034 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.035 INFO:teuthology.orchestra.run.smithi156.stdout: "op_inflight": 0, 2024-04-25T11:47:38.035 INFO:teuthology.orchestra.run.smithi156.stdout: "oplen_avg": { 2024-04-25T11:47:38.035 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2476, 2024-04-25T11:47:38.035 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 2477 2024-04-25T11:47:38.035 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.035 INFO:teuthology.orchestra.run.smithi156.stdout: "op": 2476, 2024-04-25T11:47:38.035 INFO:teuthology.orchestra.run.smithi156.stdout: "op_r": 2476, 2024-04-25T11:47:38.035 INFO:teuthology.orchestra.run.smithi156.stdout: "op_w": 0, 2024-04-25T11:47:38.035 INFO:teuthology.orchestra.run.smithi156.stdout: "op_rmw": 0, 2024-04-25T11:47:38.035 INFO:teuthology.orchestra.run.smithi156.stdout: "op_pg": 0, 2024-04-25T11:47:38.035 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_stat": 1614, 2024-04-25T11:47:38.035 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_create": 0, 2024-04-25T11:47:38.035 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_read": 860, 2024-04-25T11:47:38.035 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_write": 0, 2024-04-25T11:47:38.035 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writefull": 0, 2024-04-25T11:47:38.036 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writesame": 0, 2024-04-25T11:47:38.036 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_append": 0, 2024-04-25T11:47:38.036 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_zero": 0, 2024-04-25T11:47:38.036 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_truncate": 0, 2024-04-25T11:47:38.036 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_delete": 0, 2024-04-25T11:47:38.036 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_mapext": 0, 2024-04-25T11:47:38.036 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_sparse_read": 0, 2024-04-25T11:47:38.036 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_clonerange": 0, 2024-04-25T11:47:38.036 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_getxattr": 1, 2024-04-25T11:47:38.036 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_setxattr": 0, 2024-04-25T11:47:38.036 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_cmpxattr": 0, 2024-04-25T11:47:38.036 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_rmxattr": 0, 2024-04-25T11:47:38.036 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_resetxattrs": 0, 2024-04-25T11:47:38.036 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_call": 0, 2024-04-25T11:47:38.037 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_watch": 0, 2024-04-25T11:47:38.037 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_notify": 0, 2024-04-25T11:47:38.037 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T11:47:38.037 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls": 0, 2024-04-25T11:47:38.037 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls_filter": 0, 2024-04-25T11:47:38.037 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_other": 0, 2024-04-25T11:47:38.037 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_active": 0, 2024-04-25T11:47:38.037 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_send": 0, 2024-04-25T11:47:38.037 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_resend": 0, 2024-04-25T11:47:38.037 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_ping": 0, 2024-04-25T11:47:38.037 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_active": 0, 2024-04-25T11:47:38.037 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_send": 0, 2024-04-25T11:47:38.037 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_resend": 0, 2024-04-25T11:47:38.037 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_active": 0, 2024-04-25T11:47:38.037 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_send": 0, 2024-04-25T11:47:38.038 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_resend": 0, 2024-04-25T11:47:38.038 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_active": 0, 2024-04-25T11:47:38.038 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_send": 0, 2024-04-25T11:47:38.038 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_resend": 0, 2024-04-25T11:47:38.038 INFO:teuthology.orchestra.run.smithi156.stdout: "command_active": 0, 2024-04-25T11:47:38.038 INFO:teuthology.orchestra.run.smithi156.stdout: "command_send": 0, 2024-04-25T11:47:38.038 INFO:teuthology.orchestra.run.smithi156.stdout: "command_resend": 0, 2024-04-25T11:47:38.038 INFO:teuthology.orchestra.run.smithi156.stdout: "map_epoch": 94, 2024-04-25T11:47:38.038 INFO:teuthology.orchestra.run.smithi156.stdout: "map_full": 0, 2024-04-25T11:47:38.038 INFO:teuthology.orchestra.run.smithi156.stdout: "map_inc": 17, 2024-04-25T11:47:38.038 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_sessions": 5, 2024-04-25T11:47:38.038 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_open": 5, 2024-04-25T11:47:38.038 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_close": 0, 2024-04-25T11:47:38.038 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_laggy": 0, 2024-04-25T11:47:38.039 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_wr": 0, 2024-04-25T11:47:38.039 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_rd": 2, 2024-04-25T11:47:38.039 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_del": 0 2024-04-25T11:47:38.039 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.039 INFO:teuthology.orchestra.run.smithi156.stdout: "oft": { 2024-04-25T11:47:38.039 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_objs": 0, 2024-04-25T11:47:38.039 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_kv_pairs": 0, 2024-04-25T11:47:38.039 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_updates": 0, 2024-04-25T11:47:38.039 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_removes": 0 2024-04-25T11:47:38.039 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.039 INFO:teuthology.orchestra.run.smithi156.stdout: "osd-slow-ops": { 2024-04-25T11:47:38.039 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_ops_count": 0 2024-04-25T11:47:38.039 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.039 INFO:teuthology.orchestra.run.smithi156.stdout: "purge_queue": { 2024-04-25T11:47:38.039 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops": 0, 2024-04-25T11:47:38.039 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T11:47:38.040 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing": 0, 2024-04-25T11:47:38.040 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_high_water": 0, 2024-04-25T11:47:38.040 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed_ops": 0, 2024-04-25T11:47:38.040 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed": 0, 2024-04-25T11:47:38.040 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_item_in_journal": 0 2024-04-25T11:47:38.040 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.040 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T11:47:38.040 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T11:47:38.040 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-04-25T11:47:38.040 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T11:47:38.040 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 2756, 2024-04-25T11:47:38.040 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 2041969, 2024-04-25T11:47:38.040 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:38.040 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 2756, 2024-04-25T11:47:38.041 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-04-25T11:47:38.041 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-04-25T11:47:38.041 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 2756, 2024-04-25T11:47:38.041 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 2041969, 2024-04-25T11:47:38.041 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T11:47:38.041 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.041 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.041 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.041 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:38.041 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.041 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_bytes": { 2024-04-25T11:47:38.041 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T11:47:38.041 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-04-25T11:47:38.041 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T11:47:38.041 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-04-25T11:47:38.042 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-04-25T11:47:38.042 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:38.042 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:38.042 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 2476, 2024-04-25T11:47:38.042 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 1965, 2024-04-25T11:47:38.042 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 2476, 2024-04-25T11:47:38.042 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 1965, 2024-04-25T11:47:38.042 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T11:47:38.042 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.042 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.042 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.042 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:38.042 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.042 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_ops": { 2024-04-25T11:47:38.042 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T11:47:38.043 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 1024, 2024-04-25T11:47:38.043 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T11:47:38.043 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-04-25T11:47:38.043 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-04-25T11:47:38.043 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:38.043 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:38.043 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 2476, 2024-04-25T11:47:38.043 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 2476, 2024-04-25T11:47:38.043 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 2476, 2024-04-25T11:47:38.043 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 2476, 2024-04-25T11:47:38.043 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T11:47:38.043 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.043 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.043 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.043 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:38.044 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.044 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle": { 2024-04-25T11:47:38.044 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T11:47:38.044 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-04-25T11:47:38.044 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T11:47:38.044 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-04-25T11:47:38.044 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-04-25T11:47:38.044 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:38.044 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:38.044 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-04-25T11:47:38.044 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-04-25T11:47:38.044 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 0, 2024-04-25T11:47:38.044 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 0, 2024-04-25T11:47:38.044 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T11:47:38.044 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.044 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.045 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.045 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:38.045 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-25T11:47:38.045 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle-0x5625b6936750": { 2024-04-25T11:47:38.045 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-04-25T11:47:38.045 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-04-25T11:47:38.045 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-04-25T11:47:38.045 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-04-25T11:47:38.045 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-04-25T11:47:38.045 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-04-25T11:47:38.045 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-04-25T11:47:38.045 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-04-25T11:47:38.045 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-04-25T11:47:38.045 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 0, 2024-04-25T11:47:38.045 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 0, 2024-04-25T11:47:38.045 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-04-25T11:47:38.046 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-04-25T11:47:38.046 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-04-25T11:47:38.046 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-04-25T11:47:38.046 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:38.046 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-25T11:47:38.046 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-04-25T11:47:38.335 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:38 smithi156 ceph-mon[33045]: pgmap v391: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T11:47:38.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:38 smithi090 ceph-mon[29575]: pgmap v391: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T11:47:38.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:38 smithi138 ceph-mon[32328]: pgmap v391: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T11:47:38.805 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-04-25T11:47:38.805 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-04-25T11:47:38.805 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update=0 2024-04-25T11:47:38.805 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update_receipt=0 2024-04-25T11:47:38.805 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=0 2024-04-25T11:47:38.805 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-04-25T11:47:38.805 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T11:47:38.806 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-25T11:47:38.817 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-25T11:47:38.817 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T11:47:38.817 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T11:47:38.837 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-25T11:47:38.837 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T11:47:38.896 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:47:39.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.072+0000 7fb9db746640 1 -- 172.21.15.90:0/1960128526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9d41033a0 msgr2=0x7fb9d41037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:39.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.072+0000 7fb9db746640 1 --2- 172.21.15.90:0/1960128526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9d41033a0 0x7fb9d41037b0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc009900 tx=0x7fb9cc02a410 comp rx=0 tx=0).stop 2024-04-25T11:47:39.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.073+0000 7fb9db746640 1 -- 172.21.15.90:0/1960128526 shutdown_connections 2024-04-25T11:47:39.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.073+0000 7fb9db746640 1 --2- 172.21.15.90:0/1960128526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb9d4107ce0 0x7fb9d410f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.073+0000 7fb9db746640 1 --2- 172.21.15.90:0/1960128526 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb9d4107260 0x7fb9d4107670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.073+0000 7fb9db746640 1 --2- 172.21.15.90:0/1960128526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9d41033a0 0x7fb9d41037b0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.073+0000 7fb9db746640 1 -- 172.21.15.90:0/1960128526 >> 172.21.15.90:0/1960128526 conn(0x7fb9d4077890 msgr2=0x7fb9d4077cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:39.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.073+0000 7fb9db746640 1 -- 172.21.15.90:0/1960128526 shutdown_connections 2024-04-25T11:47:39.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.073+0000 7fb9db746640 1 -- 172.21.15.90:0/1960128526 wait complete. 2024-04-25T11:47:39.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.073+0000 7fb9db746640 1 Processor -- start 2024-04-25T11:47:39.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.073+0000 7fb9db746640 1 -- start start 2024-04-25T11:47:39.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.074+0000 7fb9db746640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb9d41033a0 0x7fb9d41a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:39.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.074+0000 7fb9db746640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9d4107260 0x7fb9d41a2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:39.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.074+0000 7fb9db746640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb9d4107ce0 0x7fb9d419c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:39.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.074+0000 7fb9db746640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d4112be0 con 0x7fb9d4107260 2024-04-25T11:47:39.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.074+0000 7fb9db746640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d4112a60 con 0x7fb9d41033a0 2024-04-25T11:47:39.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.074+0000 7fb9db746640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d4112d60 con 0x7fb9d4107ce0 2024-04-25T11:47:39.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.074+0000 7fb9d8ce1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9d4107260 0x7fb9d41a2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:39.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.074+0000 7fb9d8ce1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9d4107260 0x7fb9d41a2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54642/0 (socket says 172.21.15.90:54642) 2024-04-25T11:47:39.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.074+0000 7fb9d8ce1640 1 -- 172.21.15.90:0/210079164 learned_addr learned my addr 172.21.15.90:0/210079164 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:47:39.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.074+0000 7fb9d9ce3640 1 --2- 172.21.15.90:0/210079164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb9d4107ce0 0x7fb9d419c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.074+0000 7fb9d94e2640 1 --2- 172.21.15.90:0/210079164 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb9d41033a0 0x7fb9d41a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.074+0000 7fb9d8ce1640 1 -- 172.21.15.90:0/210079164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb9d4107ce0 msgr2=0x7fb9d419c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.074+0000 7fb9d8ce1640 1 --2- 172.21.15.90:0/210079164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb9d4107ce0 0x7fb9d419c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.074+0000 7fb9d8ce1640 1 -- 172.21.15.90:0/210079164 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb9d41033a0 msgr2=0x7fb9d41a2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.074+0000 7fb9d8ce1640 1 --2- 172.21.15.90:0/210079164 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb9d41033a0 0x7fb9d41a2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.074+0000 7fb9d8ce1640 1 -- 172.21.15.90:0/210079164 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9d419ce30 con 0x7fb9d4107260 2024-04-25T11:47:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.074+0000 7fb9d8ce1640 1 --2- 172.21.15.90:0/210079164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9d4107260 0x7fb9d41a2ac0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fb9c800ea50 tx=0x7fb9c800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.074+0000 7fb9d27fc640 1 -- 172.21.15.90:0/210079164 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c800cdc0 con 0x7fb9d4107260 2024-04-25T11:47:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.074+0000 7fb9db746640 1 -- 172.21.15.90:0/210079164 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9d419d0c0 con 0x7fb9d4107260 2024-04-25T11:47:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.074+0000 7fb9d27fc640 1 -- 172.21.15.90:0/210079164 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9c8011920 con 0x7fb9d4107260 2024-04-25T11:47:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.074+0000 7fb9d27fc640 1 -- 172.21.15.90:0/210079164 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c801aae0 con 0x7fb9d4107260 2024-04-25T11:47:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.074+0000 7fb9db746640 1 -- 172.21.15.90:0/210079164 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9d41a9320 con 0x7fb9d4107260 2024-04-25T11:47:39.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.076+0000 7fb9db746640 1 -- 172.21.15.90:0/210079164 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9d406a190 con 0x7fb9d4107260 2024-04-25T11:47:39.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.077+0000 7fb9d27fc640 1 -- 172.21.15.90:0/210079164 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fb9c801ac80 con 0x7fb9d4107260 2024-04-25T11:47:39.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.078+0000 7fb9d27fc640 1 --2- 172.21.15.90:0/210079164 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb9bc0afc60 0x7fb9bc0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:39.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.078+0000 7fb9d94e2640 1 --2- 172.21.15.90:0/210079164 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb9bc0afc60 0x7fb9bc0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:39.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.078+0000 7fb9d27fc640 1 -- 172.21.15.90:0/210079164 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8658+0+0 (secure 0 0 0) 0x7fb9c80df8d0 con 0x7fb9d4107260 2024-04-25T11:47:39.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.078+0000 7fb9d94e2640 1 --2- 172.21.15.90:0/210079164 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb9bc0afc60 0x7fb9bc0b2170 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc000c00 tx=0x7fb9cc031400 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:39.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.081+0000 7fb9d27fc640 1 -- 172.21.15.90:0/210079164 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb9c8055ff0 con 0x7fb9d4107260 2024-04-25T11:47:39.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.263+0000 7fb9db746640 1 -- 172.21.15.90:0/210079164 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb9d4105850 con 0x7fb9d4107260 2024-04-25T11:47:39.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.265+0000 7fb9d27fc640 1 -- 172.21.15.90:0/210079164 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+23366 (secure 0 0 0) 0x7fb9c8016030 con 0x7fb9d4107260 2024-04-25T11:47:39.250 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:47:39.251 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":94,"fsid":"58bd4e7a-02f6-11ef-bc93-c7b262605968","created":"2024-04-25T11:25:23.574550+0000","modified":"2024-04-25T11:47:33.924541+0000","last_up_change":"2024-04-25T11:32:23.852211+0000","last_in_change":"2024-04-25T11:32:04.188507+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:28:42.746649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:33:08.442925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:33:09.352886+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":93,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e54d888-e67b-4951-8ae7-e355fb6b10a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6803","nonce":1201629159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6805","nonce":1201629159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6809","nonce":1201629159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6807","nonce":1201629159}]},"public_addr":"172.21.15.90:6803/1201629159","cluster_addr":"172.21.15.90:6805/1201629159","heartbeat_back_addr":"172.21.15.90:6809/1201629159","heartbeat_front_addr":"172.21.15.90:6807/1201629159","state":["exists","up"]},{"osd":1,"uuid":"91a71c78-dae9-4bc7-9f6f-9e55cfefbe28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6801","nonce":3976851933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6803","nonce":3976851933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6807","nonce":3976851933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6805","nonce":3976851933}]},"public_addr":"172.21.15.138:6801/3976851933","cluster_addr":"172.21.15.138:6803/3976851933","heartbeat_back_addr":"172.21.15.138:6807/3976851933","heartbeat_front_addr":"172.21.15.138:6805/3976851933","state":["exists","up"]},{"osd":2,"uuid":"daff6c7f-80fa-4859-9339-ef7a5e840832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6801","nonce":1505807670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6803","nonce":1505807670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6807","nonce":1505807670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6805","nonce":1505807670}]},"public_addr":"172.21.15.156:6801/1505807670","cluster_addr":"172.21.15.156:6803/1505807670","heartbeat_back_addr":"172.21.15.156:6807/1505807670","heartbeat_front_addr":"172.21.15.156:6805/1505807670","state":["exists","up"]},{"osd":3,"uuid":"33fd94ca-7465-401a-bc1e-706b6b31b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6811","nonce":2520012383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6813","nonce":2520012383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6817","nonce":2520012383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6815","nonce":2520012383}]},"public_addr":"172.21.15.90:6811/2520012383","cluster_addr":"172.21.15.90:6813/2520012383","heartbeat_back_addr":"172.21.15.90:6817/2520012383","heartbeat_front_addr":"172.21.15.90:6815/2520012383","state":["exists","up"]},{"osd":4,"uuid":"c5326212-750c-49fd-96e3-d1e62bb4b7f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6809","nonce":3752678279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6811","nonce":3752678279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6815","nonce":3752678279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6813","nonce":3752678279}]},"public_addr":"172.21.15.138:6809/3752678279","cluster_addr":"172.21.15.138:6811/3752678279","heartbeat_back_addr":"172.21.15.138:6815/3752678279","heartbeat_front_addr":"172.21.15.138:6813/3752678279","state":["exists","up"]},{"osd":5,"uuid":"d04a1156-820b-4b43-be89-e16a8910c3aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6809","nonce":3591210191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6811","nonce":3591210191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6815","nonce":3591210191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6813","nonce":3591210191}]},"public_addr":"172.21.15.156:6809/3591210191","cluster_addr":"172.21.15.156:6811/3591210191","heartbeat_back_addr":"172.21.15.156:6815/3591210191","heartbeat_front_addr":"172.21.15.156:6813/3591210191","state":["exists","up"]},{"osd":6,"uuid":"75675aa3-e978-490c-8205-13d9868a3d6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6819","nonce":2364955456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6821","nonce":2364955456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6825","nonce":2364955456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6823","nonce":2364955456}]},"public_addr":"172.21.15.90:6819/2364955456","cluster_addr":"172.21.15.90:6821/2364955456","heartbeat_back_addr":"172.21.15.90:6825/2364955456","heartbeat_front_addr":"172.21.15.90:6823/2364955456","state":["exists","up"]},{"osd":7,"uuid":"1a95fdf5-f548-4271-bbcd-e330f7955eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6817","nonce":4012476387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6819","nonce":4012476387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6823","nonce":4012476387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6821","nonce":4012476387}]},"public_addr":"172.21.15.138:6817/4012476387","cluster_addr":"172.21.15.138:6819/4012476387","heartbeat_back_addr":"172.21.15.138:6823/4012476387","heartbeat_front_addr":"172.21.15.138:6821/4012476387","state":["exists","up"]},{"osd":8,"uuid":"605e10b5-5806-4c1f-8e28-a7994d34ddce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6817","nonce":101092224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6819","nonce":101092224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6823","nonce":101092224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6821","nonce":101092224}]},"public_addr":"172.21.15.156:6817/101092224","cluster_addr":"172.21.15.156:6819/101092224","heartbeat_back_addr":"172.21.15.156:6823/101092224","heartbeat_front_addr":"172.21.15.156:6821/101092224","state":["exists","up"]},{"osd":9,"uuid":"d70fdf5e-9d0f-483d-a7d8-505b79b8a418","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6827","nonce":1171265966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6829","nonce":1171265966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6833","nonce":1171265966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6831","nonce":1171265966}]},"public_addr":"172.21.15.90:6827/1171265966","cluster_addr":"172.21.15.90:6829/1171265966","heartbeat_back_addr":"172.21.15.90:6833/1171265966","heartbeat_front_addr":"172.21.15.90:6831/1171265966","state":["exists","up"]},{"osd":10,"uuid":"1ec86c7d-c3a4-432d-9664-8e40836e1e20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6825","nonce":4264666232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6827","nonce":4264666232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6831","nonce":4264666232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6829","nonce":4264666232}]},"public_addr":"172.21.15.138:6825/4264666232","cluster_addr":"172.21.15.138:6827/4264666232","heartbeat_back_addr":"172.21.15.138:6831/4264666232","heartbeat_front_addr":"172.21.15.138:6829/4264666232","state":["exists","up"]},{"osd":11,"uuid":"51cbe729-292e-4d7b-b8e9-e1e2821ffe76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6825","nonce":1224251744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6827","nonce":1224251744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6831","nonce":1224251744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6829","nonce":1224251744}]},"public_addr":"172.21.15.156:6825/1224251744","cluster_addr":"172.21.15.156:6827/1224251744","heartbeat_back_addr":"172.21.15.156:6831/1224251744","heartbeat_front_addr":"172.21.15.156:6829/1224251744","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:27:50.794226+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:15.030278+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:38.518163+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:03.107351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:27.500707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:51.463545+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:16.015076+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:41.182071+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:05.362309+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:31.048936+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:56.488509+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:32:22.129178+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":7}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/813599224":"2024-04-26T11:35:08.201598+0000","172.21.15.90:6800/219449516":"2024-04-26T11:35:08.201598+0000","172.21.15.90:0/478282854":"2024-04-26T11:35:08.201598+0000","172.21.15.90:0/2202319376":"2024-04-26T11:35:08.201598+0000","172.21.15.90:0/2381633953":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6800/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/2190846900":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/888963322":"2024-04-26T11:35:08.201598+0000","172.21.15.90:6801/219449516":"2024-04-26T11:35:08.201598+0000","172.21.15.90:0/1707792941":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6800/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6801/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/6061052":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1849633392":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6801/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/2277490702":"2024-04-26T11:26:16.655278+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:47:39.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.267+0000 7fb9db746640 1 -- 172.21.15.90:0/210079164 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb9bc0afc60 msgr2=0x7fb9bc0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:39.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.267+0000 7fb9db746640 1 --2- 172.21.15.90:0/210079164 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb9bc0afc60 0x7fb9bc0b2170 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc000c00 tx=0x7fb9cc031400 comp rx=0 tx=0).stop 2024-04-25T11:47:39.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.267+0000 7fb9db746640 1 -- 172.21.15.90:0/210079164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9d4107260 msgr2=0x7fb9d41a2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:39.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.267+0000 7fb9db746640 1 --2- 172.21.15.90:0/210079164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9d4107260 0x7fb9d41a2ac0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fb9c800ea50 tx=0x7fb9c800ef10 comp rx=0 tx=0).stop 2024-04-25T11:47:39.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.267+0000 7fb9db746640 1 -- 172.21.15.90:0/210079164 shutdown_connections 2024-04-25T11:47:39.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.267+0000 7fb9db746640 1 --2- 172.21.15.90:0/210079164 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb9bc0afc60 0x7fb9bc0b2170 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.267+0000 7fb9db746640 1 --2- 172.21.15.90:0/210079164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb9d4107ce0 0x7fb9d419c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.267+0000 7fb9db746640 1 --2- 172.21.15.90:0/210079164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9d4107260 0x7fb9d41a2ac0 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.267+0000 7fb9db746640 1 --2- 172.21.15.90:0/210079164 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb9d41033a0 0x7fb9d41a2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.267+0000 7fb9db746640 1 -- 172.21.15.90:0/210079164 >> 172.21.15.90:0/210079164 conn(0x7fb9d4077890 msgr2=0x7fb9d410e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:39.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.267+0000 7fb9db746640 1 -- 172.21.15.90:0/210079164 shutdown_connections 2024-04-25T11:47:39.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.267+0000 7fb9db746640 1 -- 172.21.15.90:0/210079164 wait complete. 2024-04-25T11:47:39.264 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T11:28:42.746649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T11:33:08.442925+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T11:33:09.352886+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 19, 'snap_epoch': 93, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T11:47:39.265 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T11:47:39.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.391+0000 7fb85cc67640 1 -- 172.21.15.90:0/199162939 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb858108a20 msgr2=0x7fb8581025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:39.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.391+0000 7fb85cc67640 1 --2- 172.21.15.90:0/199162939 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb858108a20 0x7fb8581025b0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fb84c009900 tx=0x7fb84c02a410 comp rx=0 tx=0).stop 2024-04-25T11:47:39.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.392+0000 7fb85cc67640 1 -- 172.21.15.90:0/199162939 shutdown_connections 2024-04-25T11:47:39.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.392+0000 7fb85cc67640 1 --2- 172.21.15.90:0/199162939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb85810d380 0x7fb85810f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.392+0000 7fb85cc67640 1 --2- 172.21.15.90:0/199162939 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb858102af0 0x7fb858102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.392+0000 7fb85cc67640 1 --2- 172.21.15.90:0/199162939 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb858108a20 0x7fb8581025b0 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.392+0000 7fb85cc67640 1 -- 172.21.15.90:0/199162939 >> 172.21.15.90:0/199162939 conn(0x7fb858078100 msgr2=0x7fb858078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:39.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.392+0000 7fb85cc67640 1 -- 172.21.15.90:0/199162939 shutdown_connections 2024-04-25T11:47:39.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.392+0000 7fb85cc67640 1 -- 172.21.15.90:0/199162939 wait complete. 2024-04-25T11:47:39.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.393+0000 7fb85cc67640 1 Processor -- start 2024-04-25T11:47:39.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.393+0000 7fb85cc67640 1 -- start start 2024-04-25T11:47:39.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.393+0000 7fb85cc67640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb858102af0 0x7fb8581a2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.393+0000 7fb85cc67640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb858108a20 0x7fb8581a2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.393+0000 7fb85cc67640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb85810d380 0x7fb85819c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.393+0000 7fb85cc67640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb858112be0 con 0x7fb85810d380 2024-04-25T11:47:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.393+0000 7fb85cc67640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb858112a60 con 0x7fb858102af0 2024-04-25T11:47:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.393+0000 7fb85cc67640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb858112d60 con 0x7fb858108a20 2024-04-25T11:47:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.393+0000 7fb856d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb85810d380 0x7fb85819c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.393+0000 7fb856d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb85810d380 0x7fb85819c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54666/0 (socket says 172.21.15.90:54666) 2024-04-25T11:47:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.393+0000 7fb856d9d640 1 -- 172.21.15.90:0/2392146972 learned_addr learned my addr 172.21.15.90:0/2392146972 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:47:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.393+0000 7fb855d9b640 1 --2- 172.21.15.90:0/2392146972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb858108a20 0x7fb8581a2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.393+0000 7fb85659c640 1 --2- 172.21.15.90:0/2392146972 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb858102af0 0x7fb8581a2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.393+0000 7fb856d9d640 1 -- 172.21.15.90:0/2392146972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb858108a20 msgr2=0x7fb8581a2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.393+0000 7fb856d9d640 1 --2- 172.21.15.90:0/2392146972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb858108a20 0x7fb8581a2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.393+0000 7fb856d9d640 1 -- 172.21.15.90:0/2392146972 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb858102af0 msgr2=0x7fb8581a2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.393+0000 7fb856d9d640 1 --2- 172.21.15.90:0/2392146972 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb858102af0 0x7fb8581a2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.393+0000 7fb856d9d640 1 -- 172.21.15.90:0/2392146972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb85819cec0 con 0x7fb85810d380 2024-04-25T11:47:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.393+0000 7fb856d9d640 1 --2- 172.21.15.90:0/2392146972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb85810d380 0x7fb85819c790 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fb85000eaa0 tx=0x7fb85000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:39.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.394+0000 7fb8477fe640 1 -- 172.21.15.90:0/2392146972 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb85000cdb0 con 0x7fb85810d380 2024-04-25T11:47:39.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.394+0000 7fb85cc67640 1 -- 172.21.15.90:0/2392146972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb85819d150 con 0x7fb85810d380 2024-04-25T11:47:39.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.394+0000 7fb8477fe640 1 -- 172.21.15.90:0/2392146972 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb850005020 con 0x7fb85810d380 2024-04-25T11:47:39.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.394+0000 7fb8477fe640 1 -- 172.21.15.90:0/2392146972 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8500116d0 con 0x7fb85810d380 2024-04-25T11:47:39.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.394+0000 7fb85cc67640 1 -- 172.21.15.90:0/2392146972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8581a93b0 con 0x7fb85810d380 2024-04-25T11:47:39.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.395+0000 7fb85cc67640 1 -- 172.21.15.90:0/2392146972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb858103cd0 con 0x7fb85810d380 2024-04-25T11:47:39.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.396+0000 7fb8477fe640 1 -- 172.21.15.90:0/2392146972 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fb8500051d0 con 0x7fb85810d380 2024-04-25T11:47:39.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.397+0000 7fb8477fe640 1 --2- 172.21.15.90:0/2392146972 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8400afce0 0x7fb8400b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:39.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.397+0000 7fb85659c640 1 --2- 172.21.15.90:0/2392146972 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8400afce0 0x7fb8400b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:39.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.397+0000 7fb8477fe640 1 -- 172.21.15.90:0/2392146972 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8658+0+0 (secure 0 0 0) 0x7fb8500df4e0 con 0x7fb85810d380 2024-04-25T11:47:39.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.398+0000 7fb85659c640 1 --2- 172.21.15.90:0/2392146972 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8400afce0 0x7fb8400b21f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb84c000c00 tx=0x7fb84c031400 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:39.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.401+0000 7fb8477fe640 1 -- 172.21.15.90:0/2392146972 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb850016030 con 0x7fb85810d380 2024-04-25T11:47:39.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.580+0000 7fb85cc67640 1 -- 172.21.15.90:0/2392146972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb8581a9580 con 0x7fb85810d380 2024-04-25T11:47:39.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.580+0000 7fb8477fe640 1 -- 172.21.15.90:0/2392146972 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7fb8500a7720 con 0x7fb85810d380 2024-04-25T11:47:39.565 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-25T11:47:39.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.582+0000 7fb85cc67640 1 -- 172.21.15.90:0/2392146972 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8400afce0 msgr2=0x7fb8400b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:39.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.582+0000 7fb85cc67640 1 --2- 172.21.15.90:0/2392146972 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8400afce0 0x7fb8400b21f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb84c000c00 tx=0x7fb84c031400 comp rx=0 tx=0).stop 2024-04-25T11:47:39.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.582+0000 7fb85cc67640 1 -- 172.21.15.90:0/2392146972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb85810d380 msgr2=0x7fb85819c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:39.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.582+0000 7fb85cc67640 1 --2- 172.21.15.90:0/2392146972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb85810d380 0x7fb85819c790 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fb85000eaa0 tx=0x7fb85000ef60 comp rx=0 tx=0).stop 2024-04-25T11:47:39.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.582+0000 7fb85cc67640 1 -- 172.21.15.90:0/2392146972 shutdown_connections 2024-04-25T11:47:39.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.582+0000 7fb85cc67640 1 --2- 172.21.15.90:0/2392146972 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fb8400afce0 0x7fb8400b21f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.582+0000 7fb85cc67640 1 --2- 172.21.15.90:0/2392146972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb85810d380 0x7fb85819c790 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.582+0000 7fb85cc67640 1 --2- 172.21.15.90:0/2392146972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb858108a20 0x7fb8581a2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.582+0000 7fb85cc67640 1 --2- 172.21.15.90:0/2392146972 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb858102af0 0x7fb8581a2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.582+0000 7fb85cc67640 1 -- 172.21.15.90:0/2392146972 >> 172.21.15.90:0/2392146972 conn(0x7fb858078100 msgr2=0x7fb858101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:39.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.583+0000 7fb85cc67640 1 -- 172.21.15.90:0/2392146972 shutdown_connections 2024-04-25T11:47:39.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.583+0000 7fb85cc67640 1 -- 172.21.15.90:0/2392146972 wait complete. 2024-04-25T11:47:39.579 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T11:47:39.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.704+0000 7f4a123c5640 1 -- 172.21.15.90:0/751457358 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a0c109fb0 msgr2=0x7f4a0c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:39.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.704+0000 7f4a123c5640 1 --2- 172.21.15.90:0/751457358 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a0c109fb0 0x7f4a0c102140 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f4a00009900 tx=0x7f4a0002a410 comp rx=0 tx=0).stop 2024-04-25T11:47:39.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.704+0000 7f4a123c5640 1 -- 172.21.15.90:0/751457358 shutdown_connections 2024-04-25T11:47:39.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.704+0000 7f4a123c5640 1 --2- 172.21.15.90:0/751457358 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a0c1027b0 0x7f4a0c110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.704+0000 7f4a123c5640 1 --2- 172.21.15.90:0/751457358 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a0c109fb0 0x7f4a0c102140 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.704+0000 7f4a123c5640 1 --2- 172.21.15.90:0/751457358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4a0c109b10 0x7f4a0c101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.704+0000 7f4a123c5640 1 -- 172.21.15.90:0/751457358 >> 172.21.15.90:0/751457358 conn(0x7f4a0c0fd850 msgr2=0x7f4a0c0ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:39.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.704+0000 7f4a123c5640 1 -- 172.21.15.90:0/751457358 shutdown_connections 2024-04-25T11:47:39.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.704+0000 7f4a123c5640 1 -- 172.21.15.90:0/751457358 wait complete. 2024-04-25T11:47:39.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.705+0000 7f4a123c5640 1 Processor -- start 2024-04-25T11:47:39.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.705+0000 7f4a123c5640 1 -- start start 2024-04-25T11:47:39.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.705+0000 7f4a123c5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a0c1027b0 0x7f4a0c19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:39.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.705+0000 7f4a123c5640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4a0c109b10 0x7f4a0c19ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:39.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.705+0000 7f4a123c5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a0c109fb0 0x7f4a0c19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:39.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.705+0000 7f4a123c5640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c113d00 con 0x7f4a0c1027b0 2024-04-25T11:47:39.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.705+0000 7f4a123c5640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c113b80 con 0x7f4a0c109b10 2024-04-25T11:47:39.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.706+0000 7f4a123c5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c113e80 con 0x7f4a0c109fb0 2024-04-25T11:47:39.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.706+0000 7f4a0bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a0c1027b0 0x7f4a0c19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:39.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.706+0000 7f4a0bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a0c1027b0 0x7f4a0c19a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54682/0 (socket says 172.21.15.90:54682) 2024-04-25T11:47:39.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.706+0000 7f4a0bfff640 1 -- 172.21.15.90:0/3283382070 learned_addr learned my addr 172.21.15.90:0/3283382070 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:47:39.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.706+0000 7f4a0b7fe640 1 --2- 172.21.15.90:0/3283382070 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4a0c109b10 0x7f4a0c19ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:39.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.706+0000 7f4a10962640 1 --2- 172.21.15.90:0/3283382070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a0c109fb0 0x7f4a0c19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:39.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.706+0000 7f4a0bfff640 1 -- 172.21.15.90:0/3283382070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a0c109fb0 msgr2=0x7f4a0c19efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:39.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.706+0000 7f4a0bfff640 1 --2- 172.21.15.90:0/3283382070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a0c109fb0 0x7f4a0c19efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.706+0000 7f4a0bfff640 1 -- 172.21.15.90:0/3283382070 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4a0c109b10 msgr2=0x7f4a0c19ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:39.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.706+0000 7f4a0bfff640 1 --2- 172.21.15.90:0/3283382070 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4a0c109b10 0x7f4a0c19ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.706+0000 7f4a0bfff640 1 -- 172.21.15.90:0/3283382070 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a0c19f5e0 con 0x7f4a0c1027b0 2024-04-25T11:47:39.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.706+0000 7f4a0bfff640 1 --2- 172.21.15.90:0/3283382070 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a0c1027b0 0x7f4a0c19a780 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00c890 tx=0x7f49fc00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:39.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.706+0000 7f4a097fa640 1 -- 172.21.15.90:0/3283382070 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc004220 con 0x7f4a0c1027b0 2024-04-25T11:47:39.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.706+0000 7f4a123c5640 1 -- 172.21.15.90:0/3283382070 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a0c19f8d0 con 0x7f4a0c1027b0 2024-04-25T11:47:39.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.706+0000 7f4a097fa640 1 -- 172.21.15.90:0/3283382070 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49fc0043c0 con 0x7f4a0c1027b0 2024-04-25T11:47:39.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.706+0000 7f4a097fa640 1 -- 172.21.15.90:0/3283382070 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc004a00 con 0x7f4a0c1027b0 2024-04-25T11:47:39.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.706+0000 7f4a123c5640 1 -- 172.21.15.90:0/3283382070 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a0c1a7250 con 0x7f4a0c1027b0 2024-04-25T11:47:39.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.707+0000 7f4a123c5640 1 -- 172.21.15.90:0/3283382070 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a0c06a160 con 0x7f4a0c1027b0 2024-04-25T11:47:39.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.709+0000 7f4a097fa640 1 -- 172.21.15.90:0/3283382070 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f49fc0079b0 con 0x7f4a0c1027b0 2024-04-25T11:47:39.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.709+0000 7f4a097fa640 1 --2- 172.21.15.90:0/3283382070 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f49f40afc60 0x7f49f40b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:39.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.709+0000 7f4a097fa640 1 -- 172.21.15.90:0/3283382070 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8658+0+0 (secure 0 0 0) 0x7f49fc0df110 con 0x7f4a0c1027b0 2024-04-25T11:47:39.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.709+0000 7f4a0b7fe640 1 --2- 172.21.15.90:0/3283382070 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f49f40afc60 0x7f49f40b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:39.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.710+0000 7f4a0b7fe640 1 --2- 172.21.15.90:0/3283382070 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f49f40afc60 0x7f49f40b2170 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4a00009230 tx=0x7f4a00031400 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:39.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.713+0000 7f4a097fa640 1 -- 172.21.15.90:0/3283382070 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f49fc014030 con 0x7f4a0c1027b0 2024-04-25T11:47:39.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.889+0000 7f4a123c5640 1 -- 172.21.15.90:0/3283382070 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4a0c1a7420 con 0x7f4a0c1027b0 2024-04-25T11:47:39.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.890+0000 7f4a097fa640 1 -- 172.21.15.90:0/3283382070 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f49fc0a72d0 con 0x7f4a0c1027b0 2024-04-25T11:47:39.875 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-25T11:47:39.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.892+0000 7f4a123c5640 1 -- 172.21.15.90:0/3283382070 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f49f40afc60 msgr2=0x7f49f40b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:39.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.892+0000 7f4a123c5640 1 --2- 172.21.15.90:0/3283382070 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f49f40afc60 0x7f49f40b2170 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4a00009230 tx=0x7f4a00031400 comp rx=0 tx=0).stop 2024-04-25T11:47:39.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.892+0000 7f4a123c5640 1 -- 172.21.15.90:0/3283382070 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a0c1027b0 msgr2=0x7f4a0c19a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:39.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.892+0000 7f4a123c5640 1 --2- 172.21.15.90:0/3283382070 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a0c1027b0 0x7f4a0c19a780 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00c890 tx=0x7f49fc00cd50 comp rx=0 tx=0).stop 2024-04-25T11:47:39.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.892+0000 7f4a123c5640 1 -- 172.21.15.90:0/3283382070 shutdown_connections 2024-04-25T11:47:39.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.892+0000 7f4a123c5640 1 --2- 172.21.15.90:0/3283382070 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f49f40afc60 0x7f49f40b2170 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.892+0000 7f4a123c5640 1 --2- 172.21.15.90:0/3283382070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a0c109fb0 0x7f4a0c19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.892+0000 7f4a123c5640 1 --2- 172.21.15.90:0/3283382070 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4a0c109b10 0x7f4a0c19ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.892+0000 7f4a123c5640 1 --2- 172.21.15.90:0/3283382070 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a0c1027b0 0x7f4a0c19a780 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:39.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.892+0000 7f4a123c5640 1 -- 172.21.15.90:0/3283382070 >> 172.21.15.90:0/3283382070 conn(0x7f4a0c0fd850 msgr2=0x7f4a0c0fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:39.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.892+0000 7f4a123c5640 1 -- 172.21.15.90:0/3283382070 shutdown_connections 2024-04-25T11:47:39.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:39.892+0000 7f4a123c5640 1 -- 172.21.15.90:0/3283382070 wait complete. 2024-04-25T11:47:39.888 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T11:47:40.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.013+0000 7fbdee696640 1 -- 172.21.15.90:0/2154642160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbde8108a20 msgr2=0x7fbde81025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:40.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.013+0000 7fbdee696640 1 --2- 172.21.15.90:0/2154642160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbde8108a20 0x7fbde81025b0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8009900 tx=0x7fbdd802a410 comp rx=0 tx=0).stop 2024-04-25T11:47:40.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.013+0000 7fbdee696640 1 -- 172.21.15.90:0/2154642160 shutdown_connections 2024-04-25T11:47:40.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.013+0000 7fbdee696640 1 --2- 172.21.15.90:0/2154642160 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbde810d380 0x7fbde810f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:40.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.013+0000 7fbdee696640 1 --2- 172.21.15.90:0/2154642160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbde8102af0 0x7fbde8102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:40.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.013+0000 7fbdee696640 1 --2- 172.21.15.90:0/2154642160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbde8108a20 0x7fbde81025b0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:40.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.013+0000 7fbdee696640 1 -- 172.21.15.90:0/2154642160 >> 172.21.15.90:0/2154642160 conn(0x7fbde8078100 msgr2=0x7fbde8078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:40.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.013+0000 7fbdee696640 1 -- 172.21.15.90:0/2154642160 shutdown_connections 2024-04-25T11:47:40.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.013+0000 7fbdee696640 1 -- 172.21.15.90:0/2154642160 wait complete. 2024-04-25T11:47:40.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.013+0000 7fbdee696640 1 Processor -- start 2024-04-25T11:47:40.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.013+0000 7fbdee696640 1 -- start start 2024-04-25T11:47:40.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.014+0000 7fbdee696640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbde8102af0 0x7fbde819a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:40.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.014+0000 7fbdee696640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbde8108a20 0x7fbde819ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:40.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.014+0000 7fbdee696640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbde810d380 0x7fbde819efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:40.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.014+0000 7fbdee696640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbde8112be0 con 0x7fbde8102af0 2024-04-25T11:47:40.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.014+0000 7fbdee696640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbde8112a60 con 0x7fbde810d380 2024-04-25T11:47:40.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.014+0000 7fbdee696640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbde8112d60 con 0x7fbde8108a20 2024-04-25T11:47:40.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.014+0000 7fbde7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbde8102af0 0x7fbde819a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:40.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.014+0000 7fbde7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbde8102af0 0x7fbde819a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54700/0 (socket says 172.21.15.90:54700) 2024-04-25T11:47:40.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.014+0000 7fbde7fff640 1 -- 172.21.15.90:0/451280366 learned_addr learned my addr 172.21.15.90:0/451280366 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:47:40.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.014+0000 7fbde77fe640 1 --2- 172.21.15.90:0/451280366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbde8108a20 0x7fbde819ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:40.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.014+0000 7fbde7fff640 1 -- 172.21.15.90:0/451280366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbde8108a20 msgr2=0x7fbde819ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:40.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.014+0000 7fbdecc33640 1 --2- 172.21.15.90:0/451280366 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbde810d380 0x7fbde819efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:40.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.014+0000 7fbde7fff640 1 --2- 172.21.15.90:0/451280366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbde8108a20 0x7fbde819ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:40.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.014+0000 7fbde7fff640 1 -- 172.21.15.90:0/451280366 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbde810d380 msgr2=0x7fbde819efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:40.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.014+0000 7fbde7fff640 1 --2- 172.21.15.90:0/451280366 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbde810d380 0x7fbde819efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:40.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.014+0000 7fbde7fff640 1 -- 172.21.15.90:0/451280366 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbde819f590 con 0x7fbde8102af0 2024-04-25T11:47:40.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.014+0000 7fbdecc33640 1 --2- 172.21.15.90:0/451280366 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbde810d380 0x7fbde819efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:47:40.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.014+0000 7fbde7fff640 1 --2- 172.21.15.90:0/451280366 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbde8102af0 0x7fbde819a7b0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8004270 tx=0x7fbdd80042a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:40.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.015+0000 7fbde57fa640 1 -- 172.21.15.90:0/451280366 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd803f030 con 0x7fbde8102af0 2024-04-25T11:47:40.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.015+0000 7fbde57fa640 1 -- 172.21.15.90:0/451280366 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdd8003cb0 con 0x7fbde8102af0 2024-04-25T11:47:40.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.015+0000 7fbdee696640 1 -- 172.21.15.90:0/451280366 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbde819f820 con 0x7fbde8102af0 2024-04-25T11:47:40.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.015+0000 7fbde57fa640 1 -- 172.21.15.90:0/451280366 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd8035070 con 0x7fbde8102af0 2024-04-25T11:47:40.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.015+0000 7fbdee696640 1 -- 172.21.15.90:0/451280366 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbde819fd10 con 0x7fbde8102af0 2024-04-25T11:47:40.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.016+0000 7fbdee696640 1 -- 172.21.15.90:0/451280366 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbde806afa0 con 0x7fbde8102af0 2024-04-25T11:47:40.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.017+0000 7fbde57fa640 1 -- 172.21.15.90:0/451280366 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7fbdd8042030 con 0x7fbde8102af0 2024-04-25T11:47:40.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.018+0000 7fbde57fa640 1 --2- 172.21.15.90:0/451280366 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbdd00afc60 0x7fbdd00b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:40.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.018+0000 7fbde77fe640 1 --2- 172.21.15.90:0/451280366 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbdd00afc60 0x7fbdd00b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:40.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.018+0000 7fbde57fa640 1 -- 172.21.15.90:0/451280366 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8658+0+0 (secure 0 0 0) 0x7fbdd80ffb90 con 0x7fbde8102af0 2024-04-25T11:47:40.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.019+0000 7fbde77fe640 1 --2- 172.21.15.90:0/451280366 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbdd00afc60 0x7fbdd00b2170 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbde8101e00 tx=0x7fbddc005c80 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:40.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.022+0000 7fbde57fa640 1 -- 172.21.15.90:0/451280366 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbdd80c8380 con 0x7fbde8102af0 2024-04-25T11:47:40.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.203+0000 7fbdee696640 1 -- 172.21.15.90:0/451280366 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbde81a7320 con 0x7fbde8102af0 2024-04-25T11:47:40.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.203+0000 7fbde57fa640 1 -- 172.21.15.90:0/451280366 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fbdd8003e60 con 0x7fbde8102af0 2024-04-25T11:47:40.188 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-25T11:47:40.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.205+0000 7fbdee696640 1 -- 172.21.15.90:0/451280366 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbdd00afc60 msgr2=0x7fbdd00b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:40.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.205+0000 7fbdee696640 1 --2- 172.21.15.90:0/451280366 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbdd00afc60 0x7fbdd00b2170 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbde8101e00 tx=0x7fbddc005c80 comp rx=0 tx=0).stop 2024-04-25T11:47:40.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.205+0000 7fbdee696640 1 -- 172.21.15.90:0/451280366 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbde8102af0 msgr2=0x7fbde819a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:40.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.205+0000 7fbdee696640 1 --2- 172.21.15.90:0/451280366 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbde8102af0 0x7fbde819a7b0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8004270 tx=0x7fbdd80042a0 comp rx=0 tx=0).stop 2024-04-25T11:47:40.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.205+0000 7fbdee696640 1 -- 172.21.15.90:0/451280366 shutdown_connections 2024-04-25T11:47:40.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.205+0000 7fbdee696640 1 --2- 172.21.15.90:0/451280366 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7fbdd00afc60 0x7fbdd00b2170 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:40.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.205+0000 7fbdee696640 1 --2- 172.21.15.90:0/451280366 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbde810d380 0x7fbde819efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:40.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.205+0000 7fbdee696640 1 --2- 172.21.15.90:0/451280366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbde8108a20 0x7fbde819ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:40.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.205+0000 7fbdee696640 1 --2- 172.21.15.90:0/451280366 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbde8102af0 0x7fbde819a7b0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:40.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.205+0000 7fbdee696640 1 -- 172.21.15.90:0/451280366 >> 172.21.15.90:0/451280366 conn(0x7fbde8078100 msgr2=0x7fbde8069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:40.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.205+0000 7fbdee696640 1 -- 172.21.15.90:0/451280366 shutdown_connections 2024-04-25T11:47:40.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.205+0000 7fbdee696640 1 -- 172.21.15.90:0/451280366 wait complete. 2024-04-25T11:47:40.200 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:40 smithi090 ceph-mon[29575]: pgmap v392: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T11:47:40.200 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:40 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/210079164' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:47:40.200 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:40 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2392146972' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:47:40.200 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:40 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3283382070' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:47:40.201 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T11:47:40.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.325+0000 7ffa0f355640 1 -- 172.21.15.90:0/2787106557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa08109b10 msgr2=0x7ffa081036b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:40.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.325+0000 7ffa0f355640 1 --2- 172.21.15.90:0/2787106557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa08109b10 0x7ffa081036b0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7ffa04009ca0 tx=0x7ffa0402a8c0 comp rx=0 tx=0).stop 2024-04-25T11:47:40.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.326+0000 7ffa0f355640 1 -- 172.21.15.90:0/2787106557 shutdown_connections 2024-04-25T11:47:40.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.326+0000 7ffa0f355640 1 --2- 172.21.15.90:0/2787106557 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ffa08104260 0x7ffa08110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:40.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.326+0000 7ffa0f355640 1 --2- 172.21.15.90:0/2787106557 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa08109fb0 0x7ffa08103bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:40.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.326+0000 7ffa0f355640 1 --2- 172.21.15.90:0/2787106557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa08109b10 0x7ffa081036b0 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:40.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.326+0000 7ffa0f355640 1 -- 172.21.15.90:0/2787106557 >> 172.21.15.90:0/2787106557 conn(0x7ffa080ff1d0 msgr2=0x7ffa08101620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:40.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.326+0000 7ffa0f355640 1 -- 172.21.15.90:0/2787106557 shutdown_connections 2024-04-25T11:47:40.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.326+0000 7ffa0f355640 1 -- 172.21.15.90:0/2787106557 wait complete. 2024-04-25T11:47:40.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.326+0000 7ffa0f355640 1 Processor -- start 2024-04-25T11:47:40.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.326+0000 7ffa0f355640 1 -- start start 2024-04-25T11:47:40.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.326+0000 7ffa0f355640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ffa08104260 0x7ffa0819a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:40.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.326+0000 7ffa0f355640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa08109b10 0x7ffa0819ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:40.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.326+0000 7ffa0f355640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa08109fb0 0x7ffa0819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:40.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.326+0000 7ffa0f355640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ffa08113d00 con 0x7ffa08109b10 2024-04-25T11:47:40.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.326+0000 7ffa0f355640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ffa08113b80 con 0x7ffa08104260 2024-04-25T11:47:40.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.326+0000 7ffa0f355640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ffa08113e80 con 0x7ffa08109fb0 2024-04-25T11:47:40.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.327+0000 7ffa0c8f0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa08109b10 0x7ffa0819ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:40.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.327+0000 7ffa0c8f0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa08109b10 0x7ffa0819ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54722/0 (socket says 172.21.15.90:54722) 2024-04-25T11:47:40.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.327+0000 7ffa0c8f0640 1 -- 172.21.15.90:0/1036513258 learned_addr learned my addr 172.21.15.90:0/1036513258 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:47:40.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.327+0000 7ffa0d0f1640 1 --2- 172.21.15.90:0/1036513258 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ffa08104260 0x7ffa0819a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:40.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.327+0000 7ffa0d8f2640 1 --2- 172.21.15.90:0/1036513258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa08109fb0 0x7ffa0819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:40.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.327+0000 7ffa0c8f0640 1 -- 172.21.15.90:0/1036513258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa08109fb0 msgr2=0x7ffa0819efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:40.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.327+0000 7ffa0c8f0640 1 --2- 172.21.15.90:0/1036513258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa08109fb0 0x7ffa0819efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:40.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.327+0000 7ffa0c8f0640 1 -- 172.21.15.90:0/1036513258 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ffa08104260 msgr2=0x7ffa0819a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:40.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.327+0000 7ffa0c8f0640 1 --2- 172.21.15.90:0/1036513258 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ffa08104260 0x7ffa0819a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:40.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.327+0000 7ffa0c8f0640 1 -- 172.21.15.90:0/1036513258 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa0819f600 con 0x7ffa08109b10 2024-04-25T11:47:40.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.327+0000 7ffa0d8f2640 1 --2- 172.21.15.90:0/1036513258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa08109fb0 0x7ffa0819efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:47:40.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.327+0000 7ffa0c8f0640 1 --2- 172.21.15.90:0/1036513258 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa08109b10 0x7ffa0819ad00 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8005d00 tx=0x7ff9f800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:40.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.327+0000 7ff9fe7fc640 1 -- 172.21.15.90:0/1036513258 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f8017070 con 0x7ffa08109b10 2024-04-25T11:47:40.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.327+0000 7ff9fe7fc640 1 -- 172.21.15.90:0/1036513258 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9f8009c30 con 0x7ffa08109b10 2024-04-25T11:47:40.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.327+0000 7ffa0f355640 1 -- 172.21.15.90:0/1036513258 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa0819f8f0 con 0x7ffa08109b10 2024-04-25T11:47:40.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.327+0000 7ff9fe7fc640 1 -- 172.21.15.90:0/1036513258 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f800f620 con 0x7ffa08109b10 2024-04-25T11:47:40.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.327+0000 7ffa0f355640 1 -- 172.21.15.90:0/1036513258 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa0819fdf0 con 0x7ffa08109b10 2024-04-25T11:47:40.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.328+0000 7ffa0f355640 1 -- 172.21.15.90:0/1036513258 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa0806a160 con 0x7ffa08109b10 2024-04-25T11:47:40.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.330+0000 7ff9fe7fc640 1 -- 172.21.15.90:0/1036513258 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7ff9f801a070 con 0x7ffa08109b10 2024-04-25T11:47:40.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.331+0000 7ff9fe7fc640 1 --2- 172.21.15.90:0/1036513258 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff9f40afc60 0x7ff9f40b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:40.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.331+0000 7ffa0d0f1640 1 --2- 172.21.15.90:0/1036513258 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff9f40afc60 0x7ff9f40b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:40.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.331+0000 7ff9fe7fc640 1 -- 172.21.15.90:0/1036513258 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8658+0+0 (secure 0 0 0) 0x7ff9f80dead0 con 0x7ffa08109b10 2024-04-25T11:47:40.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.331+0000 7ffa0d0f1640 1 --2- 172.21.15.90:0/1036513258 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff9f40afc60 0x7ff9f40b2170 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ffa04005e30 tx=0x7ffa04009360 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:40.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.335+0000 7ff9fe7fc640 1 -- 172.21.15.90:0/1036513258 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffa0819fdf0 con 0x7ffa08109b10 2024-04-25T11:47:40.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.527+0000 7ffa0f355640 1 -- 172.21.15.90:0/1036513258 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ffa0819baf0 con 0x7ffa08109b10 2024-04-25T11:47:40.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.528+0000 7ff9fe7fc640 1 -- 172.21.15.90:0/1036513258 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v94) ==== 81+0+934 (secure 0 0 0) 0x7ffa0819baf0 con 0x7ffa08109b10 2024-04-25T11:47:40.513 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/813599224 2024-04-26T11:35:08.201598+0000 2024-04-25T11:47:40.513 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6800/219449516 2024-04-26T11:35:08.201598+0000 2024-04-25T11:47:40.513 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/478282854 2024-04-26T11:35:08.201598+0000 2024-04-25T11:47:40.513 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/2202319376 2024-04-26T11:35:08.201598+0000 2024-04-25T11:47:40.513 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/2381633953 2024-04-26T11:26:16.655278+0000 2024-04-25T11:47:40.513 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6800/71478430 2024-04-26T11:26:16.655278+0000 2024-04-25T11:47:40.513 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/2190846900 2024-04-26T11:25:50.322011+0000 2024-04-25T11:47:40.513 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/888963322 2024-04-26T11:35:08.201598+0000 2024-04-25T11:47:40.513 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6801/219449516 2024-04-26T11:35:08.201598+0000 2024-04-25T11:47:40.514 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/1707792941 2024-04-26T11:25:50.322011+0000 2024-04-25T11:47:40.514 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6800/3122821311 2024-04-26T11:25:50.322011+0000 2024-04-25T11:47:40.514 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6801/71478430 2024-04-26T11:26:16.655278+0000 2024-04-25T11:47:40.514 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/6061052 2024-04-26T11:25:50.322011+0000 2024-04-25T11:47:40.514 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/1849633392 2024-04-26T11:26:16.655278+0000 2024-04-25T11:47:40.514 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6801/3122821311 2024-04-26T11:25:50.322011+0000 2024-04-25T11:47:40.514 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/2277490702 2024-04-26T11:26:16.655278+0000 2024-04-25T11:47:40.514 INFO:teuthology.orchestra.run.smithi090.stderr:listed 16 entries 2024-04-25T11:47:40.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.530+0000 7ffa0f355640 1 -- 172.21.15.90:0/1036513258 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff9f40afc60 msgr2=0x7ff9f40b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:40.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.530+0000 7ffa0f355640 1 --2- 172.21.15.90:0/1036513258 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff9f40afc60 0x7ff9f40b2170 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ffa04005e30 tx=0x7ffa04009360 comp rx=0 tx=0).stop 2024-04-25T11:47:40.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.530+0000 7ffa0f355640 1 -- 172.21.15.90:0/1036513258 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa08109b10 msgr2=0x7ffa0819ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:40.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.530+0000 7ffa0f355640 1 --2- 172.21.15.90:0/1036513258 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa08109b10 0x7ffa0819ad00 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8005d00 tx=0x7ff9f800dde0 comp rx=0 tx=0).stop 2024-04-25T11:47:40.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.530+0000 7ffa0f355640 1 -- 172.21.15.90:0/1036513258 shutdown_connections 2024-04-25T11:47:40.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.530+0000 7ffa0f355640 1 --2- 172.21.15.90:0/1036513258 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff9f40afc60 0x7ff9f40b2170 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:40.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.530+0000 7ffa0f355640 1 --2- 172.21.15.90:0/1036513258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa08109fb0 0x7ffa0819efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:40.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.530+0000 7ffa0f355640 1 --2- 172.21.15.90:0/1036513258 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa08109b10 0x7ffa0819ad00 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:40.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.530+0000 7ffa0f355640 1 --2- 172.21.15.90:0/1036513258 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ffa08104260 0x7ffa0819a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:40.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.530+0000 7ffa0f355640 1 -- 172.21.15.90:0/1036513258 >> 172.21.15.90:0/1036513258 conn(0x7ffa080ff1d0 msgr2=0x7ffa080ff5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:40.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.530+0000 7ffa0f355640 1 -- 172.21.15.90:0/1036513258 shutdown_connections 2024-04-25T11:47:40.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.530+0000 7ffa0f355640 1 -- 172.21.15.90:0/1036513258 wait complete. 2024-04-25T11:47:40.526 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T11:47:40.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:47:40.527 DEBUG:teuthology.orchestra.run.smithi090:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-25T11:47:40.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:40 smithi138 ceph-mon[32328]: pgmap v392: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T11:47:40.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:40 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/210079164' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:47:40.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:40 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2392146972' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:47:40.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:40 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3283382070' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:47:40.576 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T11:47:40.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:47:40.576 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T11:47:40.584 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:40 smithi156 ceph-mon[33045]: pgmap v392: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T11:47:40.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:40 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/210079164' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:47:40.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:40 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2392146972' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T11:47:40.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:40 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3283382070' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T11:47:40.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:47:40.624 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-04-25T11:47:40.652 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T11:47:40.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:47:40.653 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T11:47:40.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:47:40.718 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link delete ceph-brx 2024-04-25T11:47:40.824 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-25T11:47:40.837 INFO:tasks.cephadm:Teardown begin 2024-04-25T11:47:40.837 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T11:47:40.837 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:47:40.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.961+0000 7f8f274f9640 1 -- 172.21.15.90:0/902235797 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f20109fd0 msgr2=0x7f8f20103c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:40.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.961+0000 7f8f274f9640 1 --2- 172.21.15.90:0/902235797 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f20109fd0 0x7f8f20103c10 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f8f10009900 tx=0x7f8f1002a410 comp rx=0 tx=0).stop 2024-04-25T11:47:40.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.961+0000 7f8f274f9640 1 -- 172.21.15.90:0/902235797 shutdown_connections 2024-04-25T11:47:40.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.961+0000 7f8f274f9640 1 --2- 172.21.15.90:0/902235797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f20104280 0x7f8f20110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:40.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.961+0000 7f8f274f9640 1 --2- 172.21.15.90:0/902235797 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f20109fd0 0x7f8f20103c10 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:40.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.961+0000 7f8f274f9640 1 --2- 172.21.15.90:0/902235797 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8f20109b30 0x7f8f201036d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:40.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.961+0000 7f8f274f9640 1 -- 172.21.15.90:0/902235797 >> 172.21.15.90:0/902235797 conn(0x7f8f200ff210 msgr2=0x7f8f20101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:40.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.961+0000 7f8f274f9640 1 -- 172.21.15.90:0/902235797 shutdown_connections 2024-04-25T11:47:40.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.961+0000 7f8f274f9640 1 -- 172.21.15.90:0/902235797 wait complete. 2024-04-25T11:47:40.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.961+0000 7f8f274f9640 1 Processor -- start 2024-04-25T11:47:40.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.961+0000 7f8f274f9640 1 -- start start 2024-04-25T11:47:40.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.962+0000 7f8f274f9640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8f20104280 0x7f8f2019a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:40.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.962+0000 7f8f274f9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f20109b30 0x7f8f2019ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:40.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.962+0000 7f8f274f9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f20109fd0 0x7f8f2019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:40.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.962+0000 7f8f274f9640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8f20113d20 con 0x7f8f20109fd0 2024-04-25T11:47:40.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.962+0000 7f8f274f9640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8f20113ba0 con 0x7f8f20104280 2024-04-25T11:47:40.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.962+0000 7f8f274f9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8f20113ea0 con 0x7f8f20109b30 2024-04-25T11:47:40.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.962+0000 7f8f25a96640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f20109fd0 0x7f8f2019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:40.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.962+0000 7f8f25a96640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f20109fd0 0x7f8f2019efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54738/0 (socket says 172.21.15.90:54738) 2024-04-25T11:47:40.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.962+0000 7f8f25a96640 1 -- 172.21.15.90:0/2717014362 learned_addr learned my addr 172.21.15.90:0/2717014362 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:47:40.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.962+0000 7f8f24a94640 1 --2- 172.21.15.90:0/2717014362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f20109b30 0x7f8f2019ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:40.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.962+0000 7f8f25a96640 1 -- 172.21.15.90:0/2717014362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f20109b30 msgr2=0x7f8f2019ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:40.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.962+0000 7f8f25a96640 1 --2- 172.21.15.90:0/2717014362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f20109b30 0x7f8f2019ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:40.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.962+0000 7f8f25295640 1 --2- 172.21.15.90:0/2717014362 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8f20104280 0x7f8f2019a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:40.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.962+0000 7f8f25a96640 1 -- 172.21.15.90:0/2717014362 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8f20104280 msgr2=0x7f8f2019a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:40.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.962+0000 7f8f25a96640 1 --2- 172.21.15.90:0/2717014362 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8f20104280 0x7f8f2019a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:40.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.962+0000 7f8f25a96640 1 -- 172.21.15.90:0/2717014362 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f2019f600 con 0x7f8f20109fd0 2024-04-25T11:47:40.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.962+0000 7f8f25a96640 1 --2- 172.21.15.90:0/2717014362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f20109fd0 0x7f8f2019efc0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f8f1800eaa0 tx=0x7f8f1800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:40.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.962+0000 7f8f167fc640 1 -- 172.21.15.90:0/2717014362 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f1800cdb0 con 0x7f8f20109fd0 2024-04-25T11:47:40.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.962+0000 7f8f274f9640 1 -- 172.21.15.90:0/2717014362 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f2019f8f0 con 0x7f8f20109fd0 2024-04-25T11:47:40.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.962+0000 7f8f167fc640 1 -- 172.21.15.90:0/2717014362 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f18005020 con 0x7f8f20109fd0 2024-04-25T11:47:40.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.962+0000 7f8f167fc640 1 -- 172.21.15.90:0/2717014362 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f180116d0 con 0x7f8f20109fd0 2024-04-25T11:47:40.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.963+0000 7f8f274f9640 1 -- 172.21.15.90:0/2717014362 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f201a7270 con 0x7f8f20109fd0 2024-04-25T11:47:40.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.963+0000 7f8f274f9640 1 -- 172.21.15.90:0/2717014362 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f2006a190 con 0x7f8f20109fd0 2024-04-25T11:47:40.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.965+0000 7f8f167fc640 1 -- 172.21.15.90:0/2717014362 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f8f180054a0 con 0x7f8f20109fd0 2024-04-25T11:47:40.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.965+0000 7f8f167fc640 1 --2- 172.21.15.90:0/2717014362 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8f0c0afc60 0x7f8f0c0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:40.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.966+0000 7f8f25295640 1 --2- 172.21.15.90:0/2717014362 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8f0c0afc60 0x7f8f0c0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:40.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.966+0000 7f8f167fc640 1 -- 172.21.15.90:0/2717014362 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8658+0+0 (secure 0 0 0) 0x7f8f180df730 con 0x7f8f20109fd0 2024-04-25T11:47:40.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.966+0000 7f8f25295640 1 --2- 172.21.15.90:0/2717014362 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8f0c0afc60 0x7f8f0c0b2170 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8f20104810 tx=0x7f8f1c005ed0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:40.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:40.969+0000 7f8f167fc640 1 -- 172.21.15.90:0/2717014362 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8f18016030 con 0x7f8f20109fd0 2024-04-25T11:47:41.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.186+0000 7f8f274f9640 1 -- 172.21.15.90:0/2717014362 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8f2019baf0 con 0x7f8f20109fd0 2024-04-25T11:47:41.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.187+0000 7f8f167fc640 1 -- 172.21.15.90:0/2717014362 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11931 (secure 0 0 0) 0x7f8f180a7f40 con 0x7f8f20109fd0 2024-04-25T11:47:41.172 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:47:41.172 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:46:52.065160+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524,24475,24499,24458,24431]},"id":1}]} 2024-04-25T11:47:41.173 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 29 2024-04-25T11:47:41.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.189+0000 7f8f274f9640 1 -- 172.21.15.90:0/2717014362 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8f0c0afc60 msgr2=0x7f8f0c0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:41.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.189+0000 7f8f274f9640 1 --2- 172.21.15.90:0/2717014362 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8f0c0afc60 0x7f8f0c0b2170 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8f20104810 tx=0x7f8f1c005ed0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.189+0000 7f8f274f9640 1 -- 172.21.15.90:0/2717014362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f20109fd0 msgr2=0x7f8f2019efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:41.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.189+0000 7f8f274f9640 1 --2- 172.21.15.90:0/2717014362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f20109fd0 0x7f8f2019efc0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f8f1800eaa0 tx=0x7f8f1800ef60 comp rx=0 tx=0).stop 2024-04-25T11:47:41.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.189+0000 7f8f274f9640 1 -- 172.21.15.90:0/2717014362 shutdown_connections 2024-04-25T11:47:41.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.189+0000 7f8f274f9640 1 --2- 172.21.15.90:0/2717014362 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8f0c0afc60 0x7f8f0c0b2170 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.189+0000 7f8f274f9640 1 --2- 172.21.15.90:0/2717014362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f20109fd0 0x7f8f2019efc0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.189+0000 7f8f274f9640 1 --2- 172.21.15.90:0/2717014362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f20109b30 0x7f8f2019ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.189+0000 7f8f274f9640 1 --2- 172.21.15.90:0/2717014362 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8f20104280 0x7f8f2019a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.189+0000 7f8f274f9640 1 -- 172.21.15.90:0/2717014362 >> 172.21.15.90:0/2717014362 conn(0x7f8f200ff210 msgr2=0x7f8f2010f780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:41.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.189+0000 7f8f274f9640 1 -- 172.21.15.90:0/2717014362 shutdown_connections 2024-04-25T11:47:41.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.189+0000 7f8f274f9640 1 -- 172.21.15.90:0/2717014362 wait complete. 2024-04-25T11:47:41.186 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:47:41.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.308+0000 7ff08829f640 1 -- 172.21.15.90:0/2398442246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff080102410 msgr2=0x7ff080102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:41.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.308+0000 7ff08829f640 1 --2- 172.21.15.90:0/2398442246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff080102410 0x7ff080102820 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7ff070009ca0 tx=0x7ff07002a8c0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.308+0000 7ff08829f640 1 -- 172.21.15.90:0/2398442246 shutdown_connections 2024-04-25T11:47:41.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.308+0000 7ff08829f640 1 --2- 172.21.15.90:0/2398442246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff080107460 0x7ff08010ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.308+0000 7ff08829f640 1 --2- 172.21.15.90:0/2398442246 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff0801069a0 0x7ff080106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.308+0000 7ff08829f640 1 --2- 172.21.15.90:0/2398442246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff080102410 0x7ff080102820 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.308+0000 7ff08829f640 1 -- 172.21.15.90:0/2398442246 >> 172.21.15.90:0/2398442246 conn(0x7ff0800fd850 msgr2=0x7ff0800ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.308+0000 7ff08829f640 1 -- 172.21.15.90:0/2398442246 shutdown_connections 2024-04-25T11:47:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.308+0000 7ff08829f640 1 -- 172.21.15.90:0/2398442246 wait complete. 2024-04-25T11:47:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.309+0000 7ff08829f640 1 Processor -- start 2024-04-25T11:47:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.309+0000 7ff08829f640 1 -- start start 2024-04-25T11:47:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.309+0000 7ff08829f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff080102410 0x7ff080199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.309+0000 7ff08829f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0801069a0 0x7ff080199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.309+0000 7ff08829f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff080107460 0x7ff08019de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.309+0000 7ff08829f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff080112bf0 con 0x7ff0801069a0 2024-04-25T11:47:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.309+0000 7ff08829f640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff080112a70 con 0x7ff080102410 2024-04-25T11:47:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.309+0000 7ff08829f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff080112d70 con 0x7ff080107460 2024-04-25T11:47:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.309+0000 7ff08583a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0801069a0 0x7ff080199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.309+0000 7ff08583a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0801069a0 0x7ff080199b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54748/0 (socket says 172.21.15.90:54748) 2024-04-25T11:47:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.309+0000 7ff08583a640 1 -- 172.21.15.90:0/2217465538 learned_addr learned my addr 172.21.15.90:0/2217465538 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:47:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.309+0000 7ff08603b640 1 --2- 172.21.15.90:0/2217465538 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff080102410 0x7ff080199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.309+0000 7ff08683c640 1 --2- 172.21.15.90:0/2217465538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff080107460 0x7ff08019de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.309+0000 7ff08583a640 1 -- 172.21.15.90:0/2217465538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff080107460 msgr2=0x7ff08019de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.309+0000 7ff08583a640 1 --2- 172.21.15.90:0/2217465538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff080107460 0x7ff08019de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.309+0000 7ff08583a640 1 -- 172.21.15.90:0/2217465538 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff080102410 msgr2=0x7ff080199610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.309+0000 7ff08583a640 1 --2- 172.21.15.90:0/2217465538 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff080102410 0x7ff080199610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.309+0000 7ff08583a640 1 -- 172.21.15.90:0/2217465538 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff08019e3f0 con 0x7ff0801069a0 2024-04-25T11:47:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.309+0000 7ff08683c640 1 --2- 172.21.15.90:0/2217465538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff080107460 0x7ff08019de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:47:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.309+0000 7ff08583a640 1 --2- 172.21.15.90:0/2217465538 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0801069a0 0x7ff080199b70 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7ff07400c9a0 tx=0x7ff07400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.310+0000 7ff07f7fe640 1 -- 172.21.15.90:0/2217465538 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff074014070 con 0x7ff0801069a0 2024-04-25T11:47:41.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.310+0000 7ff07f7fe640 1 -- 172.21.15.90:0/2217465538 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff074004440 con 0x7ff0801069a0 2024-04-25T11:47:41.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.310+0000 7ff08829f640 1 -- 172.21.15.90:0/2217465538 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff08019e6e0 con 0x7ff0801069a0 2024-04-25T11:47:41.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.310+0000 7ff07f7fe640 1 -- 172.21.15.90:0/2217465538 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff07400f450 con 0x7ff0801069a0 2024-04-25T11:47:41.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.310+0000 7ff08829f640 1 -- 172.21.15.90:0/2217465538 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0801a60e0 con 0x7ff0801069a0 2024-04-25T11:47:41.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.311+0000 7ff08829f640 1 -- 172.21.15.90:0/2217465538 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff08006a190 con 0x7ff0801069a0 2024-04-25T11:47:41.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.312+0000 7ff07f7fe640 1 -- 172.21.15.90:0/2217465538 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7ff074017070 con 0x7ff0801069a0 2024-04-25T11:47:41.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.313+0000 7ff07f7fe640 1 --2- 172.21.15.90:0/2217465538 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff0680afce0 0x7ff0680b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:41.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.313+0000 7ff08603b640 1 --2- 172.21.15.90:0/2217465538 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff0680afce0 0x7ff0680b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:41.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.313+0000 7ff07f7fe640 1 -- 172.21.15.90:0/2217465538 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8658+0+0 (secure 0 0 0) 0x7ff0740df360 con 0x7ff0801069a0 2024-04-25T11:47:41.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.313+0000 7ff08603b640 1 --2- 172.21.15.90:0/2217465538 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff0680afce0 0x7ff0680b21f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff070009a60 tx=0x7ff0700093e0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:41.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.316+0000 7ff07f7fe640 1 -- 172.21.15.90:0/2217465538 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff07401e030 con 0x7ff0801069a0 2024-04-25T11:47:41.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:41 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/451280366' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:47:41.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:41 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/1036513258' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T11:47:41.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.536+0000 7ff08829f640 1 -- 172.21.15.90:0/2217465538 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff08019a960 con 0x7ff0801069a0 2024-04-25T11:47:41.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.537+0000 7ff07f7fe640 1 -- 172.21.15.90:0/2217465538 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11931 (secure 0 0 0) 0x7ff0740a7bf0 con 0x7ff0801069a0 2024-04-25T11:47:41.522 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:47:41.522 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061411806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":4061411806},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061411806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/3050267616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":3050267616},{"type":"v1","addr":"172.21.15.90:6837","nonce":3050267616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T11:33:10.373468+0000","modified":"2024-04-25T11:46:52.065160+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24499,"mds_2":24475,"mds_3":24458,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14637":{"gid":14637,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6839/3101160958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":3101160958},{"type":"v1","addr":"172.21.15.90:6839","nonce":3101160958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6841/2617461073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":2617461073},{"type":"v1","addr":"172.21.15.90:6841","nonce":2617461073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/31237375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":31237375},{"type":"v1","addr":"172.21.15.156:6833","nonce":31237375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.138:6833/51785544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6832","nonce":51785544},{"type":"v1","addr":"172.21.15.138:6833","nonce":51785544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"e","rank":3,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.138:6835/162690028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6834","nonce":162690028},{"type":"v1","addr":"172.21.15.138:6835","nonce":162690028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.156:6835/1641860251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1641860251},{"type":"v1","addr":"172.21.15.156:6835","nonce":1641860251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3466257977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3466257977},{"type":"v1","addr":"172.21.15.156:6837","nonce":3466257977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.138:6837/2598829089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6836","nonce":2598829089},{"type":"v1","addr":"172.21.15.138:6837","nonce":2598829089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":5,"addr":"172.21.15.156:6839/4179744597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":4179744597},{"type":"v1","addr":"172.21.15.156:6839","nonce":4179744597}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.138:6839/1849836338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6838","nonce":1849836338},{"type":"v1","addr":"172.21.15.138:6839","nonce":1849836338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24524,"qdb_cluster":[24524,24475,24499,24458,24431]},"id":1}]} 2024-04-25T11:47:41.522 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 29 2024-04-25T11:47:41.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.539+0000 7ff08829f640 1 -- 172.21.15.90:0/2217465538 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff0680afce0 msgr2=0x7ff0680b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:41.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.539+0000 7ff08829f640 1 --2- 172.21.15.90:0/2217465538 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff0680afce0 0x7ff0680b21f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff070009a60 tx=0x7ff0700093e0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.539+0000 7ff08829f640 1 -- 172.21.15.90:0/2217465538 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0801069a0 msgr2=0x7ff080199b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:41.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.539+0000 7ff08829f640 1 --2- 172.21.15.90:0/2217465538 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0801069a0 0x7ff080199b70 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7ff07400c9a0 tx=0x7ff07400ce60 comp rx=0 tx=0).stop 2024-04-25T11:47:41.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.539+0000 7ff08829f640 1 -- 172.21.15.90:0/2217465538 shutdown_connections 2024-04-25T11:47:41.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.539+0000 7ff08829f640 1 --2- 172.21.15.90:0/2217465538 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7ff0680afce0 0x7ff0680b21f0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.539+0000 7ff08829f640 1 --2- 172.21.15.90:0/2217465538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff080107460 0x7ff08019de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.539+0000 7ff08829f640 1 --2- 172.21.15.90:0/2217465538 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0801069a0 0x7ff080199b70 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.539+0000 7ff08829f640 1 --2- 172.21.15.90:0/2217465538 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff080102410 0x7ff080199610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.539+0000 7ff08829f640 1 -- 172.21.15.90:0/2217465538 >> 172.21.15.90:0/2217465538 conn(0x7ff0800fd850 msgr2=0x7ff08010db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:41.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.539+0000 7ff08829f640 1 -- 172.21.15.90:0/2217465538 shutdown_connections 2024-04-25T11:47:41.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.539+0000 7ff08829f640 1 -- 172.21.15.90:0/2217465538 wait complete. 2024-04-25T11:47:41.536 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:47:41.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:41 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/451280366' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:47:41.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:41 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/1036513258' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T11:47:41.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:41 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/451280366' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T11:47:41.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:41 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/1036513258' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T11:47:41.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.659+0000 7f059a37a640 1 -- 172.21.15.90:0/2824248353 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0594108a20 msgr2=0x7f05941025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:41.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.659+0000 7f059a37a640 1 --2- 172.21.15.90:0/2824248353 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0594108a20 0x7f05941025b0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f057c009900 tx=0x7f057c02a410 comp rx=0 tx=0).stop 2024-04-25T11:47:41.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.659+0000 7f059a37a640 1 -- 172.21.15.90:0/2824248353 shutdown_connections 2024-04-25T11:47:41.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.659+0000 7f059a37a640 1 --2- 172.21.15.90:0/2824248353 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f059410d380 0x7f059410f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.659+0000 7f059a37a640 1 --2- 172.21.15.90:0/2824248353 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0594102af0 0x7f0594102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.659+0000 7f059a37a640 1 --2- 172.21.15.90:0/2824248353 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0594108a20 0x7f05941025b0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.659+0000 7f059a37a640 1 -- 172.21.15.90:0/2824248353 >> 172.21.15.90:0/2824248353 conn(0x7f0594078110 msgr2=0x7f0594078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:41.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.659+0000 7f059a37a640 1 -- 172.21.15.90:0/2824248353 shutdown_connections 2024-04-25T11:47:41.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.659+0000 7f059a37a640 1 -- 172.21.15.90:0/2824248353 wait complete. 2024-04-25T11:47:41.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.659+0000 7f059a37a640 1 Processor -- start 2024-04-25T11:47:41.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.659+0000 7f059a37a640 1 -- start start 2024-04-25T11:47:41.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.660+0000 7f059a37a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0594102af0 0x7f059419b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:41.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.660+0000 7f059a37a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0594108a20 0x7f059419ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:41.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.660+0000 7f059a37a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f059410d380 0x7f059419bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:41.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.660+0000 7f059a37a640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0594112be0 con 0x7f059410d380 2024-04-25T11:47:41.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.660+0000 7f059a37a640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0594112a60 con 0x7f0594102af0 2024-04-25T11:47:41.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.660+0000 7f059a37a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0594112d60 con 0x7f0594108a20 2024-04-25T11:47:41.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.660+0000 7f0598917640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f059410d380 0x7f059419bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:41.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.660+0000 7f0598917640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f059410d380 0x7f059419bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48482/0 (socket says 172.21.15.90:48482) 2024-04-25T11:47:41.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.660+0000 7f0598917640 1 -- 172.21.15.90:0/367871940 learned_addr learned my addr 172.21.15.90:0/367871940 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:47:41.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.660+0000 7f0598917640 1 -- 172.21.15.90:0/367871940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0594108a20 msgr2=0x7f059419ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:41.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.660+0000 7f0598917640 1 --2- 172.21.15.90:0/367871940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0594108a20 0x7f059419ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.660+0000 7f0598917640 1 -- 172.21.15.90:0/367871940 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0594102af0 msgr2=0x7f059419b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:41.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.660+0000 7f0598917640 1 --2- 172.21.15.90:0/367871940 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0594102af0 0x7f059419b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.660+0000 7f0598917640 1 -- 172.21.15.90:0/367871940 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05941a6830 con 0x7f059410d380 2024-04-25T11:47:41.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.660+0000 7f0598917640 1 --2- 172.21.15.90:0/367871940 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f059410d380 0x7f059419bfa0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f058c009fd0 tx=0x7f058c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:41.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.661+0000 7f05917fa640 1 -- 172.21.15.90:0/367871940 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058c017070 con 0x7f059410d380 2024-04-25T11:47:41.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.661+0000 7f05917fa640 1 -- 172.21.15.90:0/367871940 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f058c003e20 con 0x7f059410d380 2024-04-25T11:47:41.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.661+0000 7f059a37a640 1 -- 172.21.15.90:0/367871940 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05941a6b20 con 0x7f059410d380 2024-04-25T11:47:41.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.661+0000 7f05917fa640 1 -- 172.21.15.90:0/367871940 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058c010460 con 0x7f059410d380 2024-04-25T11:47:41.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.661+0000 7f059a37a640 1 -- 172.21.15.90:0/367871940 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05941a7020 con 0x7f059410d380 2024-04-25T11:47:41.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.662+0000 7f059a37a640 1 -- 172.21.15.90:0/367871940 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f059406afa0 con 0x7f059410d380 2024-04-25T11:47:41.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.663+0000 7f05917fa640 1 -- 172.21.15.90:0/367871940 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f058c003970 con 0x7f059410d380 2024-04-25T11:47:41.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.664+0000 7f05917fa640 1 --2- 172.21.15.90:0/367871940 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f05800afc60 0x7f05800b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:41.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.664+0000 7f05917fa640 1 -- 172.21.15.90:0/367871940 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8658+0+0 (secure 0 0 0) 0x7f058c0dee60 con 0x7f059410d380 2024-04-25T11:47:41.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.664+0000 7f0593fff640 1 --2- 172.21.15.90:0/367871940 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f05800afc60 0x7f05800b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:41.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.664+0000 7f0593fff640 1 --2- 172.21.15.90:0/367871940 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f05800afc60 0x7f05800b2170 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f057c02a930 tx=0x7f057c031400 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:41.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.667+0000 7f05917fa640 1 -- 172.21.15.90:0/367871940 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f058c0a7670 con 0x7f059410d380 2024-04-25T11:47:41.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.846+0000 7f059a37a640 1 -- 172.21.15.90:0/367871940 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0594106950 con 0x7f059410d380 2024-04-25T11:47:41.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.848+0000 7f05917fa640 1 -- 172.21.15.90:0/367871940 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+23366 (secure 0 0 0) 0x7f058c0a6f70 con 0x7f059410d380 2024-04-25T11:47:41.833 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-25T11:47:41.834 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":94,"fsid":"58bd4e7a-02f6-11ef-bc93-c7b262605968","created":"2024-04-25T11:25:23.574550+0000","modified":"2024-04-25T11:47:33.924541+0000","last_up_change":"2024-04-25T11:32:23.852211+0000","last_in_change":"2024-04-25T11:32:04.188507+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:28:42.746649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:33:08.442925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:33:09.352886+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":93,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e54d888-e67b-4951-8ae7-e355fb6b10a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6803","nonce":1201629159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6805","nonce":1201629159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6809","nonce":1201629159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":1201629159},{"type":"v1","addr":"172.21.15.90:6807","nonce":1201629159}]},"public_addr":"172.21.15.90:6803/1201629159","cluster_addr":"172.21.15.90:6805/1201629159","heartbeat_back_addr":"172.21.15.90:6809/1201629159","heartbeat_front_addr":"172.21.15.90:6807/1201629159","state":["exists","up"]},{"osd":1,"uuid":"91a71c78-dae9-4bc7-9f6f-9e55cfefbe28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6801","nonce":3976851933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6803","nonce":3976851933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6807","nonce":3976851933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":3976851933},{"type":"v1","addr":"172.21.15.138:6805","nonce":3976851933}]},"public_addr":"172.21.15.138:6801/3976851933","cluster_addr":"172.21.15.138:6803/3976851933","heartbeat_back_addr":"172.21.15.138:6807/3976851933","heartbeat_front_addr":"172.21.15.138:6805/3976851933","state":["exists","up"]},{"osd":2,"uuid":"daff6c7f-80fa-4859-9339-ef7a5e840832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6801","nonce":1505807670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6803","nonce":1505807670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6807","nonce":1505807670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1505807670},{"type":"v1","addr":"172.21.15.156:6805","nonce":1505807670}]},"public_addr":"172.21.15.156:6801/1505807670","cluster_addr":"172.21.15.156:6803/1505807670","heartbeat_back_addr":"172.21.15.156:6807/1505807670","heartbeat_front_addr":"172.21.15.156:6805/1505807670","state":["exists","up"]},{"osd":3,"uuid":"33fd94ca-7465-401a-bc1e-706b6b31b9b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6811","nonce":2520012383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6813","nonce":2520012383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6817","nonce":2520012383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2520012383},{"type":"v1","addr":"172.21.15.90:6815","nonce":2520012383}]},"public_addr":"172.21.15.90:6811/2520012383","cluster_addr":"172.21.15.90:6813/2520012383","heartbeat_back_addr":"172.21.15.90:6817/2520012383","heartbeat_front_addr":"172.21.15.90:6815/2520012383","state":["exists","up"]},{"osd":4,"uuid":"c5326212-750c-49fd-96e3-d1e62bb4b7f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6809","nonce":3752678279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6811","nonce":3752678279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6815","nonce":3752678279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":3752678279},{"type":"v1","addr":"172.21.15.138:6813","nonce":3752678279}]},"public_addr":"172.21.15.138:6809/3752678279","cluster_addr":"172.21.15.138:6811/3752678279","heartbeat_back_addr":"172.21.15.138:6815/3752678279","heartbeat_front_addr":"172.21.15.138:6813/3752678279","state":["exists","up"]},{"osd":5,"uuid":"d04a1156-820b-4b43-be89-e16a8910c3aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6809","nonce":3591210191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6811","nonce":3591210191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6815","nonce":3591210191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":3591210191},{"type":"v1","addr":"172.21.15.156:6813","nonce":3591210191}]},"public_addr":"172.21.15.156:6809/3591210191","cluster_addr":"172.21.15.156:6811/3591210191","heartbeat_back_addr":"172.21.15.156:6815/3591210191","heartbeat_front_addr":"172.21.15.156:6813/3591210191","state":["exists","up"]},{"osd":6,"uuid":"75675aa3-e978-490c-8205-13d9868a3d6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6819","nonce":2364955456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6821","nonce":2364955456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6825","nonce":2364955456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2364955456},{"type":"v1","addr":"172.21.15.90:6823","nonce":2364955456}]},"public_addr":"172.21.15.90:6819/2364955456","cluster_addr":"172.21.15.90:6821/2364955456","heartbeat_back_addr":"172.21.15.90:6825/2364955456","heartbeat_front_addr":"172.21.15.90:6823/2364955456","state":["exists","up"]},{"osd":7,"uuid":"1a95fdf5-f548-4271-bbcd-e330f7955eaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6817","nonce":4012476387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6819","nonce":4012476387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6823","nonce":4012476387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":4012476387},{"type":"v1","addr":"172.21.15.138:6821","nonce":4012476387}]},"public_addr":"172.21.15.138:6817/4012476387","cluster_addr":"172.21.15.138:6819/4012476387","heartbeat_back_addr":"172.21.15.138:6823/4012476387","heartbeat_front_addr":"172.21.15.138:6821/4012476387","state":["exists","up"]},{"osd":8,"uuid":"605e10b5-5806-4c1f-8e28-a7994d34ddce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6817","nonce":101092224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6819","nonce":101092224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6823","nonce":101092224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":101092224},{"type":"v1","addr":"172.21.15.156:6821","nonce":101092224}]},"public_addr":"172.21.15.156:6817/101092224","cluster_addr":"172.21.15.156:6819/101092224","heartbeat_back_addr":"172.21.15.156:6823/101092224","heartbeat_front_addr":"172.21.15.156:6821/101092224","state":["exists","up"]},{"osd":9,"uuid":"d70fdf5e-9d0f-483d-a7d8-505b79b8a418","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6827","nonce":1171265966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6829","nonce":1171265966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6833","nonce":1171265966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1171265966},{"type":"v1","addr":"172.21.15.90:6831","nonce":1171265966}]},"public_addr":"172.21.15.90:6827/1171265966","cluster_addr":"172.21.15.90:6829/1171265966","heartbeat_back_addr":"172.21.15.90:6833/1171265966","heartbeat_front_addr":"172.21.15.90:6831/1171265966","state":["exists","up"]},{"osd":10,"uuid":"1ec86c7d-c3a4-432d-9664-8e40836e1e20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6825","nonce":4264666232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6826","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6827","nonce":4264666232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6830","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6831","nonce":4264666232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6828","nonce":4264666232},{"type":"v1","addr":"172.21.15.138:6829","nonce":4264666232}]},"public_addr":"172.21.15.138:6825/4264666232","cluster_addr":"172.21.15.138:6827/4264666232","heartbeat_back_addr":"172.21.15.138:6831/4264666232","heartbeat_front_addr":"172.21.15.138:6829/4264666232","state":["exists","up"]},{"osd":11,"uuid":"51cbe729-292e-4d7b-b8e9-e1e2821ffe76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6825","nonce":1224251744}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6827","nonce":1224251744}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6831","nonce":1224251744}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1224251744},{"type":"v1","addr":"172.21.15.156:6829","nonce":1224251744}]},"public_addr":"172.21.15.156:6825/1224251744","cluster_addr":"172.21.15.156:6827/1224251744","heartbeat_back_addr":"172.21.15.156:6831/1224251744","heartbeat_front_addr":"172.21.15.156:6829/1224251744","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:27:50.794226+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:15.030278+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:28:38.518163+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:03.107351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:27.500707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:29:51.463545+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:16.015076+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:30:41.182071+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:05.362309+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:31.048936+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:31:56.488509+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:32:22.129178+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":7}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.37","mappings":[{"from":1,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/813599224":"2024-04-26T11:35:08.201598+0000","172.21.15.90:6800/219449516":"2024-04-26T11:35:08.201598+0000","172.21.15.90:0/478282854":"2024-04-26T11:35:08.201598+0000","172.21.15.90:0/2202319376":"2024-04-26T11:35:08.201598+0000","172.21.15.90:0/2381633953":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6800/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/2190846900":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/888963322":"2024-04-26T11:35:08.201598+0000","172.21.15.90:6801/219449516":"2024-04-26T11:35:08.201598+0000","172.21.15.90:0/1707792941":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6800/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:6801/71478430":"2024-04-26T11:26:16.655278+0000","172.21.15.90:0/6061052":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/1849633392":"2024-04-26T11:26:16.655278+0000","172.21.15.90:6801/3122821311":"2024-04-26T11:25:50.322011+0000","172.21.15.90:0/2277490702":"2024-04-26T11:26:16.655278+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:47:41.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.850+0000 7f059a37a640 1 -- 172.21.15.90:0/367871940 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f05800afc60 msgr2=0x7f05800b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:41.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.850+0000 7f059a37a640 1 --2- 172.21.15.90:0/367871940 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f05800afc60 0x7f05800b2170 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f057c02a930 tx=0x7f057c031400 comp rx=0 tx=0).stop 2024-04-25T11:47:41.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.850+0000 7f059a37a640 1 -- 172.21.15.90:0/367871940 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f059410d380 msgr2=0x7f059419bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:41.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.850+0000 7f059a37a640 1 --2- 172.21.15.90:0/367871940 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f059410d380 0x7f059419bfa0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f058c009fd0 tx=0x7f058c00b260 comp rx=0 tx=0).stop 2024-04-25T11:47:41.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.850+0000 7f059a37a640 1 -- 172.21.15.90:0/367871940 shutdown_connections 2024-04-25T11:47:41.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.850+0000 7f059a37a640 1 --2- 172.21.15.90:0/367871940 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f05800afc60 0x7f05800b2170 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.850+0000 7f059a37a640 1 --2- 172.21.15.90:0/367871940 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f059410d380 0x7f059419bfa0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.850+0000 7f059a37a640 1 --2- 172.21.15.90:0/367871940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0594108a20 0x7f059419ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.850+0000 7f059a37a640 1 --2- 172.21.15.90:0/367871940 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0594102af0 0x7f059419b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.850+0000 7f059a37a640 1 -- 172.21.15.90:0/367871940 >> 172.21.15.90:0/367871940 conn(0x7f0594078110 msgr2=0x7f05941040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:41.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.850+0000 7f059a37a640 1 -- 172.21.15.90:0/367871940 shutdown_connections 2024-04-25T11:47:41.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.850+0000 7f059a37a640 1 -- 172.21.15.90:0/367871940 wait complete. 2024-04-25T11:47:41.847 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T11:47:41.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.972+0000 7fef10196640 1 -- 172.21.15.90:0/95592815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef08104260 msgr2=0x7fef08110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:41.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.972+0000 7fef10196640 1 --2- 172.21.15.90:0/95592815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef08104260 0x7fef08110630 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fef00009ca0 tx=0x7fef0002a8c0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.972+0000 7fef10196640 1 -- 172.21.15.90:0/95592815 shutdown_connections 2024-04-25T11:47:41.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.972+0000 7fef10196640 1 --2- 172.21.15.90:0/95592815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef08104260 0x7fef08110630 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.972+0000 7fef10196640 1 --2- 172.21.15.90:0/95592815 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fef08109fb0 0x7fef08103bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.972+0000 7fef10196640 1 --2- 172.21.15.90:0/95592815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef08109b10 0x7fef081036b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.972+0000 7fef10196640 1 -- 172.21.15.90:0/95592815 >> 172.21.15.90:0/95592815 conn(0x7fef080ff210 msgr2=0x7fef08101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:41.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.972+0000 7fef10196640 1 -- 172.21.15.90:0/95592815 shutdown_connections 2024-04-25T11:47:41.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.972+0000 7fef10196640 1 -- 172.21.15.90:0/95592815 wait complete. 2024-04-25T11:47:41.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.972+0000 7fef10196640 1 Processor -- start 2024-04-25T11:47:41.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.973+0000 7fef10196640 1 -- start start 2024-04-25T11:47:41.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.973+0000 7fef10196640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef08104260 0x7fef0819a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:41.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.973+0000 7fef10196640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef08109b10 0x7fef0819ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:41.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.973+0000 7fef10196640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fef08109fb0 0x7fef0819eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:41.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.973+0000 7fef10196640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fef08113d00 con 0x7fef08104260 2024-04-25T11:47:41.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.973+0000 7fef10196640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fef08113b80 con 0x7fef08109fb0 2024-04-25T11:47:41.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.973+0000 7fef10196640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fef08113e80 con 0x7fef08109b10 2024-04-25T11:47:41.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.973+0000 7fef0df32640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef08104260 0x7fef0819a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:41.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.973+0000 7fef0df32640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef08104260 0x7fef0819a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48492/0 (socket says 172.21.15.90:48492) 2024-04-25T11:47:41.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.973+0000 7fef0df32640 1 -- 172.21.15.90:0/2719561625 learned_addr learned my addr 172.21.15.90:0/2719561625 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:47:41.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.973+0000 7fef0d731640 1 --2- 172.21.15.90:0/2719561625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef08109b10 0x7fef0819ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:41.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.973+0000 7fef0df32640 1 -- 172.21.15.90:0/2719561625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef08109b10 msgr2=0x7fef0819ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:41.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.973+0000 7fef0df32640 1 --2- 172.21.15.90:0/2719561625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef08109b10 0x7fef0819ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.973+0000 7fef0df32640 1 -- 172.21.15.90:0/2719561625 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fef08109fb0 msgr2=0x7fef0819eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:41.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.973+0000 7fef0df32640 1 --2- 172.21.15.90:0/2719561625 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fef08109fb0 0x7fef0819eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:41.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.973+0000 7fef0df32640 1 -- 172.21.15.90:0/2719561625 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef0819f480 con 0x7fef08104260 2024-04-25T11:47:41.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.973+0000 7fef0df32640 1 --2- 172.21.15.90:0/2719561625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef08104260 0x7fef0819a6a0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7feef800c9a0 tx=0x7feef800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:41.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.974+0000 7feefeffd640 1 -- 172.21.15.90:0/2719561625 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef8014070 con 0x7fef08104260 2024-04-25T11:47:41.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.974+0000 7feefeffd640 1 -- 172.21.15.90:0/2719561625 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feef8004440 con 0x7fef08104260 2024-04-25T11:47:41.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.974+0000 7feefeffd640 1 -- 172.21.15.90:0/2719561625 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef800f450 con 0x7fef08104260 2024-04-25T11:47:41.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.974+0000 7fef10196640 1 -- 172.21.15.90:0/2719561625 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef0819f770 con 0x7fef08104260 2024-04-25T11:47:41.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.974+0000 7fef10196640 1 -- 172.21.15.90:0/2719561625 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef0819fcc0 con 0x7fef08104260 2024-04-25T11:47:41.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.975+0000 7fef10196640 1 -- 172.21.15.90:0/2719561625 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef0806a160 con 0x7fef08104260 2024-04-25T11:47:41.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.976+0000 7feefeffd640 1 -- 172.21.15.90:0/2719561625 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7feef8017070 con 0x7fef08104260 2024-04-25T11:47:41.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.977+0000 7feefeffd640 1 --2- 172.21.15.90:0/2719561625 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7feef40afc60 0x7feef40b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:41.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.977+0000 7fef0d731640 1 --2- 172.21.15.90:0/2719561625 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7feef40afc60 0x7feef40b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:41.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.977+0000 7feefeffd640 1 -- 172.21.15.90:0/2719561625 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8658+0+0 (secure 0 0 0) 0x7feef80df3a0 con 0x7fef08104260 2024-04-25T11:47:41.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.977+0000 7fef0d731640 1 --2- 172.21.15.90:0/2719561625 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7feef40afc60 0x7feef40b2170 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fef080ff7b0 tx=0x7fef0400a2d0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:41.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:41.980+0000 7feefeffd640 1 -- 172.21.15.90:0/2719561625 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feef801e030 con 0x7fef08104260 2024-04-25T11:47:42.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.199+0000 7fef10196640 1 -- 172.21.15.90:0/2719561625 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fef0819b9f0 con 0x7fef08104260 2024-04-25T11:47:42.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.202+0000 7feefeffd640 1 -- 172.21.15.90:0/2719561625 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v30) ==== 161+0+0 (secure 0 0 0) 0x7feef80a7bb0 con 0x7fef08104260 2024-04-25T11:47:42.187 INFO:teuthology.orchestra.run.smithi090.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T11:47:42.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.204+0000 7fef10196640 1 -- 172.21.15.90:0/2719561625 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7feef40afc60 msgr2=0x7feef40b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:42.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.204+0000 7fef10196640 1 --2- 172.21.15.90:0/2719561625 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7feef40afc60 0x7feef40b2170 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fef080ff7b0 tx=0x7fef0400a2d0 comp rx=0 tx=0).stop 2024-04-25T11:47:42.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.204+0000 7fef10196640 1 -- 172.21.15.90:0/2719561625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef08104260 msgr2=0x7fef0819a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:42.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.204+0000 7fef10196640 1 --2- 172.21.15.90:0/2719561625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef08104260 0x7fef0819a6a0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7feef800c9a0 tx=0x7feef800ce60 comp rx=0 tx=0).stop 2024-04-25T11:47:42.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.205+0000 7fef10196640 1 -- 172.21.15.90:0/2719561625 shutdown_connections 2024-04-25T11:47:42.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.205+0000 7fef10196640 1 --2- 172.21.15.90:0/2719561625 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7feef40afc60 0x7feef40b2170 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:42.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.205+0000 7fef10196640 1 --2- 172.21.15.90:0/2719561625 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fef08109fb0 0x7fef0819eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:42.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.205+0000 7fef10196640 1 --2- 172.21.15.90:0/2719561625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef08109b10 0x7fef0819ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:42.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.205+0000 7fef10196640 1 --2- 172.21.15.90:0/2719561625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef08104260 0x7fef0819a6a0 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:42.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.205+0000 7fef10196640 1 -- 172.21.15.90:0/2719561625 >> 172.21.15.90:0/2719561625 conn(0x7fef080ff210 msgr2=0x7fef08106520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:42.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.205+0000 7fef10196640 1 -- 172.21.15.90:0/2719561625 shutdown_connections 2024-04-25T11:47:42.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.205+0000 7fef10196640 1 -- 172.21.15.90:0/2719561625 wait complete. 2024-04-25T11:47:42.202 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T11:47:42.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.336+0000 7f32b4630640 1 -- 172.21.15.90:0/665912962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32ac102880 msgr2=0x7f32ac102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:42.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.336+0000 7f32b4630640 1 --2- 172.21.15.90:0/665912962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32ac102880 0x7f32ac102c90 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f329c009900 tx=0x7f329c02a410 comp rx=0 tx=0).stop 2024-04-25T11:47:42.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.336+0000 7f32b4630640 1 -- 172.21.15.90:0/665912962 shutdown_connections 2024-04-25T11:47:42.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.336+0000 7f32b4630640 1 --2- 172.21.15.90:0/665912962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32ac110420 0x7f32ac112840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:42.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.336+0000 7f32b4630640 1 --2- 172.21.15.90:0/665912962 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f32ac10dac0 0x7f32ac10fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:42.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.336+0000 7f32b4630640 1 --2- 172.21.15.90:0/665912962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32ac102880 0x7f32ac102c90 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:42.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.336+0000 7f32b4630640 1 -- 172.21.15.90:0/665912962 >> 172.21.15.90:0/665912962 conn(0x7f32ac078100 msgr2=0x7f32ac078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:42.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.336+0000 7f32b4630640 1 -- 172.21.15.90:0/665912962 shutdown_connections 2024-04-25T11:47:42.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.336+0000 7f32b4630640 1 -- 172.21.15.90:0/665912962 wait complete. 2024-04-25T11:47:42.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.337+0000 7f32b4630640 1 Processor -- start 2024-04-25T11:47:42.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.337+0000 7f32b4630640 1 -- start start 2024-04-25T11:47:42.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.337+0000 7f32b4630640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32ac102880 0x7f32ac1a25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:42.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.337+0000 7f32b4630640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32ac10dac0 0x7f32ac1a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:42.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.337+0000 7f32b4630640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f32ac110420 0x7f32ac19c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:42.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.337+0000 7f32b4630640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f32ac115ec0 con 0x7f32ac10dac0 2024-04-25T11:47:42.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.337+0000 7f32b4630640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f32ac115d40 con 0x7f32ac110420 2024-04-25T11:47:42.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.337+0000 7f32b4630640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f32ac116040 con 0x7f32ac102880 2024-04-25T11:47:42.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.337+0000 7f32b1bcb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32ac10dac0 0x7f32ac1a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:42.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.337+0000 7f32b1bcb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32ac10dac0 0x7f32ac1a2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48562/0 (socket says 172.21.15.90:48562) 2024-04-25T11:47:42.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.337+0000 7f32b1bcb640 1 -- 172.21.15.90:0/4067764342 learned_addr learned my addr 172.21.15.90:0/4067764342 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:47:42.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.337+0000 7f32b23cc640 1 --2- 172.21.15.90:0/4067764342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32ac102880 0x7f32ac1a25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:42.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.337+0000 7f32b2bcd640 1 --2- 172.21.15.90:0/4067764342 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f32ac110420 0x7f32ac19c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:42.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.337+0000 7f32b1bcb640 1 -- 172.21.15.90:0/4067764342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32ac102880 msgr2=0x7f32ac1a25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:42.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.337+0000 7f32b1bcb640 1 --2- 172.21.15.90:0/4067764342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32ac102880 0x7f32ac1a25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:42.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.337+0000 7f32b1bcb640 1 -- 172.21.15.90:0/4067764342 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f32ac110420 msgr2=0x7f32ac19c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:42.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.337+0000 7f32b1bcb640 1 --2- 172.21.15.90:0/4067764342 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f32ac110420 0x7f32ac19c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:42.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.337+0000 7f32b1bcb640 1 -- 172.21.15.90:0/4067764342 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32ac19ce70 con 0x7f32ac10dac0 2024-04-25T11:47:42.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.338+0000 7f32b1bcb640 1 --2- 172.21.15.90:0/4067764342 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32ac10dac0 0x7f32ac1a2b00 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f32a800d740 tx=0x7f32a800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:42.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.338+0000 7f32a37fe640 1 -- 172.21.15.90:0/4067764342 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a8017070 con 0x7f32ac10dac0 2024-04-25T11:47:42.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.338+0000 7f32a37fe640 1 -- 172.21.15.90:0/4067764342 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32a800de50 con 0x7f32ac10dac0 2024-04-25T11:47:42.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.338+0000 7f32a37fe640 1 -- 172.21.15.90:0/4067764342 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a8003c20 con 0x7f32ac10dac0 2024-04-25T11:47:42.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.338+0000 7f32b4630640 1 -- 172.21.15.90:0/4067764342 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32ac19d100 con 0x7f32ac10dac0 2024-04-25T11:47:42.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.338+0000 7f32b4630640 1 -- 172.21.15.90:0/4067764342 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32ac1a9360 con 0x7f32ac10dac0 2024-04-25T11:47:42.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.339+0000 7f32b4630640 1 -- 172.21.15.90:0/4067764342 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32ac06a6f0 con 0x7f32ac10dac0 2024-04-25T11:47:42.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.340+0000 7f32a37fe640 1 -- 172.21.15.90:0/4067764342 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f32a8003dc0 con 0x7f32ac10dac0 2024-04-25T11:47:42.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.341+0000 7f32a37fe640 1 --2- 172.21.15.90:0/4067764342 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f32980afce0 0x7f32980b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:42.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.341+0000 7f32b23cc640 1 --2- 172.21.15.90:0/4067764342 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f32980afce0 0x7f32980b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:42.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.341+0000 7f32a37fe640 1 -- 172.21.15.90:0/4067764342 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9044+0+0 (secure 0 0 0) 0x7f32a80df180 con 0x7f32ac10dac0 2024-04-25T11:47:42.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.342+0000 7f32b23cc640 1 --2- 172.21.15.90:0/4067764342 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f32980afce0 0x7f32980b21f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f329c031440 tx=0x7f329c005d50 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:42.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.349+0000 7f32a37fe640 1 -- 172.21.15.90:0/4067764342 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f32a80a7830 con 0x7f32ac10dac0 2024-04-25T11:47:42.472 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 25 11:47:42 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-a[71356]: ignoring --setuser ceph since I am not root 2024-04-25T11:47:42.472 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 25 11:47:42 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-a[71356]: ignoring --setgroup ceph since I am not root 2024-04-25T11:47:42.472 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 25 11:47:42 smithi090 ceph-mds[71362]: -- 172.21.15.90:0/4227204880 <== mon.2 v2:172.21.15.138:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a2e8ae7a40 con 0x55a2e8b2f600 2024-04-25T11:47:42.472 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 25 11:47:42 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-a[71356]: starting mds.a at 2024-04-25T11:47:42.473 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 25 11:47:42 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-g[70996]: ignoring --setuser ceph since I am not root 2024-04-25T11:47:42.473 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 25 11:47:42 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-g[70996]: ignoring --setgroup ceph since I am not root 2024-04-25T11:47:42.473 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 25 11:47:42 smithi090 ceph-mds[71009]: -- 172.21.15.90:0/892504782 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5623601b9a40 con 0x562360201600 2024-04-25T11:47:42.473 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 25 11:47:42 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-g[70996]: starting mds.g at 2024-04-25T11:47:42.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:42 smithi090 ceph-mon[29575]: pgmap v393: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T11:47:42.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:42 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2717014362' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:47:42.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:42 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2217465538' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:47:42.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:42 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/367871940' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:47:42.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:42 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-a[29571]: 2024-04-25T11:47:42.200+0000 7fb08c005700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T11:47:42.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:42 smithi138 ceph-mon[32328]: pgmap v393: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T11:47:42.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:42 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2717014362' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:47:42.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:42 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2217465538' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:47:42.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:42 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/367871940' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:47:42.549 INFO:journalctl@ceph.mds.h.smithi138.stdout:Apr 25 11:47:42 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-h[62247]: ignoring --setuser ceph since I am not root 2024-04-25T11:47:42.550 INFO:journalctl@ceph.mds.h.smithi138.stdout:Apr 25 11:47:42 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-h[62247]: ignoring --setgroup ceph since I am not root 2024-04-25T11:47:42.550 INFO:journalctl@ceph.mds.h.smithi138.stdout:Apr 25 11:47:42 smithi138 ceph-mds[62251]: -- 172.21.15.138:0/438445092 <== mon.2 v2:172.21.15.138:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56516ac47a40 con 0x56516ac8fa80 2024-04-25T11:47:42.550 INFO:journalctl@ceph.mds.h.smithi138.stdout:Apr 25 11:47:42 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-h[62247]: starting mds.h at 2024-04-25T11:47:42.550 INFO:journalctl@ceph.mds.e.smithi138.stdout:Apr 25 11:47:42 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-e[61937]: ignoring --setuser ceph since I am not root 2024-04-25T11:47:42.550 INFO:journalctl@ceph.mds.e.smithi138.stdout:Apr 25 11:47:42 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-e[61937]: ignoring --setgroup ceph since I am not root 2024-04-25T11:47:42.550 INFO:journalctl@ceph.mds.e.smithi138.stdout:Apr 25 11:47:42 smithi138 ceph-mds[61941]: -- 172.21.15.138:0/1483791965 <== mon.2 v2:172.21.15.138:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560517221a40 con 0x560517269180 2024-04-25T11:47:42.550 INFO:journalctl@ceph.mds.e.smithi138.stdout:Apr 25 11:47:42 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-e[61937]: starting mds.e at 2024-04-25T11:47:42.550 INFO:journalctl@ceph.mds.k.smithi138.stdout:Apr 25 11:47:42 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-k[61626]: ignoring --setuser ceph since I am not root 2024-04-25T11:47:42.551 INFO:journalctl@ceph.mds.k.smithi138.stdout:Apr 25 11:47:42 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-k[61626]: ignoring --setgroup ceph since I am not root 2024-04-25T11:47:42.551 INFO:journalctl@ceph.mds.k.smithi138.stdout:Apr 25 11:47:42 smithi138 ceph-mds[61630]: -- 172.21.15.138:0/1484692225 <== mon.2 v2:172.21.15.138:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b8e0e69a40 con 0x55b8e0eb1a80 2024-04-25T11:47:42.551 INFO:journalctl@ceph.mds.k.smithi138.stdout:Apr 25 11:47:42 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-k[61626]: starting mds.k at 2024-04-25T11:47:42.551 INFO:journalctl@ceph.mds.k.smithi138.stdout:Apr 25 11:47:42 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-k[61626]: 2024-04-25T11:47:42.278+0000 7f92da5de700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6833/0: (98) Address already in use 2024-04-25T11:47:42.551 INFO:journalctl@ceph.mds.b.smithi138.stdout:Apr 25 11:47:42 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-b[62562]: ignoring --setuser ceph since I am not root 2024-04-25T11:47:42.551 INFO:journalctl@ceph.mds.b.smithi138.stdout:Apr 25 11:47:42 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-b[62562]: ignoring --setgroup ceph since I am not root 2024-04-25T11:47:42.551 INFO:journalctl@ceph.mds.b.smithi138.stdout:Apr 25 11:47:42 smithi138 ceph-mds[62566]: -- 172.21.15.138:0/228883376 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55fb13d2e4e0 con 0x55fb13d79a80 2024-04-25T11:47:42.551 INFO:journalctl@ceph.mds.b.smithi138.stdout:Apr 25 11:47:42 smithi138 ceph-mds[62566]: -- 172.21.15.138:0/228883376 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fb13d31a40 con 0x55fb13d79a80 2024-04-25T11:47:42.551 INFO:journalctl@ceph.mds.b.smithi138.stdout:Apr 25 11:47:42 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-b[62562]: starting mds.b at 2024-04-25T11:47:42.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:42.570+0000 7f32b4630640 1 -- 172.21.15.90:0/4067764342 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f32ac1a9530 con 0x7f32ac10dac0 2024-04-25T11:47:42.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:42 smithi156 ceph-mon[33045]: pgmap v393: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T11:47:42.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:42 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2717014362' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:47:42.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:42 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2217465538' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T11:47:42.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:42 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/367871940' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T11:47:42.585 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 11:47:42 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-c[63779]: ignoring --setuser ceph since I am not root 2024-04-25T11:47:42.586 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 11:47:42 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-c[63779]: ignoring --setgroup ceph since I am not root 2024-04-25T11:47:42.586 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 11:47:42 smithi156 ceph-mds[63783]: -- 172.21.15.156:0/3260969630 <== mon.2 v2:172.21.15.138:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557787185a40 con 0x5577871cd600 2024-04-25T11:47:42.586 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 11:47:42 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-c[63779]: starting mds.c at 2024-04-25T11:47:42.586 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 11:47:42 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-f[63468]: ignoring --setuser ceph since I am not root 2024-04-25T11:47:42.586 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 11:47:42 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-f[63468]: ignoring --setgroup ceph since I am not root 2024-04-25T11:47:42.586 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 11:47:42 smithi156 ceph-mds[63472]: -- 172.21.15.156:0/3855624582 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5582ca383a40 con 0x5582ca3cad00 2024-04-25T11:47:42.586 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 11:47:42 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-f[63468]: starting mds.f at 2024-04-25T11:47:42.587 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 11:47:42 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-i[64087]: ignoring --setuser ceph since I am not root 2024-04-25T11:47:42.587 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 11:47:42 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-i[64087]: ignoring --setgroup ceph since I am not root 2024-04-25T11:47:42.587 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 11:47:42 smithi156 ceph-mds[64091]: -- 172.21.15.156:0/1714208672 <== mon.0 v2:172.21.15.90:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558e19085a40 con 0x558e190cd180 2024-04-25T11:47:42.587 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 11:47:42 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-i[64087]: starting mds.i at 2024-04-25T11:47:42.587 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 11:47:42 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-l[63156]: ignoring --setuser ceph since I am not root 2024-04-25T11:47:42.587 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 11:47:42 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-l[63156]: ignoring --setgroup ceph since I am not root 2024-04-25T11:47:42.587 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 11:47:42 smithi156 ceph-mds[63160]: -- 172.21.15.156:0/3753532932 <== mon.2 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5572a05804e0 con 0x5572a05cba80 2024-04-25T11:47:42.587 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 11:47:42 smithi156 ceph-mds[63160]: -- 172.21.15.156:0/3753532932 <== mon.2 v2:172.21.15.138:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5572a0583a40 con 0x5572a05cba80 2024-04-25T11:47:42.587 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 11:47:42 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-l[63156]: starting mds.l at 2024-04-25T11:47:43.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.211+0000 7f32a37fe640 1 -- 172.21.15.90:0/4067764342 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v31) ==== 104+0+0 (secure 0 0 0) 0x7f32a80a7130 con 0x7f32ac10dac0 2024-04-25T11:47:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.213+0000 7f32b4630640 1 -- 172.21.15.90:0/4067764342 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f32980afce0 msgr2=0x7f32980b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.213+0000 7f32b4630640 1 --2- 172.21.15.90:0/4067764342 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f32980afce0 0x7f32980b21f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f329c031440 tx=0x7f329c005d50 comp rx=0 tx=0).stop 2024-04-25T11:47:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.213+0000 7f32b4630640 1 -- 172.21.15.90:0/4067764342 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32ac10dac0 msgr2=0x7f32ac1a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.213+0000 7f32b4630640 1 --2- 172.21.15.90:0/4067764342 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32ac10dac0 0x7f32ac1a2b00 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f32a800d740 tx=0x7f32a800dc00 comp rx=0 tx=0).stop 2024-04-25T11:47:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.213+0000 7f32b4630640 1 -- 172.21.15.90:0/4067764342 shutdown_connections 2024-04-25T11:47:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.213+0000 7f32b4630640 1 --2- 172.21.15.90:0/4067764342 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f32980afce0 0x7f32980b21f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.213+0000 7f32b4630640 1 --2- 172.21.15.90:0/4067764342 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f32ac110420 0x7f32ac19c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.213+0000 7f32b4630640 1 --2- 172.21.15.90:0/4067764342 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32ac10dac0 0x7f32ac1a2b00 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.213+0000 7f32b4630640 1 --2- 172.21.15.90:0/4067764342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f32ac102880 0x7f32ac1a25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.213+0000 7f32b4630640 1 -- 172.21.15.90:0/4067764342 >> 172.21.15.90:0/4067764342 conn(0x7f32ac078100 msgr2=0x7f32ac101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.213+0000 7f32b4630640 1 -- 172.21.15.90:0/4067764342 shutdown_connections 2024-04-25T11:47:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.213+0000 7f32b4630640 1 -- 172.21.15.90:0/4067764342 wait complete. 2024-04-25T11:47:43.213 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T11:47:43.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.343+0000 7f3be35c5640 1 -- 172.21.15.90:0/262000343 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bdc109fc0 msgr2=0x7f3bdc102130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:43.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.343+0000 7f3be35c5640 1 --2- 172.21.15.90:0/262000343 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bdc109fc0 0x7f3bdc102130 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc009900 tx=0x7f3bcc02a410 comp rx=0 tx=0).stop 2024-04-25T11:47:43.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.343+0000 7f3be35c5640 1 -- 172.21.15.90:0/262000343 shutdown_connections 2024-04-25T11:47:43.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.343+0000 7f3be35c5640 1 --2- 172.21.15.90:0/262000343 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3bdc1027a0 0x7f3bdc110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:43.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.343+0000 7f3be35c5640 1 --2- 172.21.15.90:0/262000343 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bdc109fc0 0x7f3bdc102130 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:43.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.343+0000 7f3be35c5640 1 --2- 172.21.15.90:0/262000343 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bdc109b20 0x7f3bdc101bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:43.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.343+0000 7f3be35c5640 1 -- 172.21.15.90:0/262000343 >> 172.21.15.90:0/262000343 conn(0x7f3bdc077880 msgr2=0x7f3bdc077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:43.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.343+0000 7f3be35c5640 1 -- 172.21.15.90:0/262000343 shutdown_connections 2024-04-25T11:47:43.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.343+0000 7f3be35c5640 1 -- 172.21.15.90:0/262000343 wait complete. 2024-04-25T11:47:43.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.344+0000 7f3be35c5640 1 Processor -- start 2024-04-25T11:47:43.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.344+0000 7f3be35c5640 1 -- start start 2024-04-25T11:47:43.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.344+0000 7f3be35c5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bdc1027a0 0x7f3bdc19a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:43.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.344+0000 7f3be35c5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bdc109b20 0x7f3bdc19ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:43.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.344+0000 7f3be35c5640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3bdc109fc0 0x7f3bdc19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:43.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.344+0000 7f3be35c5640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3bdc113d10 con 0x7f3bdc1027a0 2024-04-25T11:47:43.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.344+0000 7f3be35c5640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3bdc113b90 con 0x7f3bdc109fc0 2024-04-25T11:47:43.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.344+0000 7f3be35c5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3bdc113e90 con 0x7f3bdc109b20 2024-04-25T11:47:43.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.344+0000 7f3be1361640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bdc1027a0 0x7f3bdc19a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:43.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.344+0000 7f3be1361640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bdc1027a0 0x7f3bdc19a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48582/0 (socket says 172.21.15.90:48582) 2024-04-25T11:47:43.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.344+0000 7f3be1361640 1 -- 172.21.15.90:0/116162311 learned_addr learned my addr 172.21.15.90:0/116162311 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:47:43.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.344+0000 7f3be0b60640 1 --2- 172.21.15.90:0/116162311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bdc109b20 0x7f3bdc19ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:43.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.344+0000 7f3be1b62640 1 --2- 172.21.15.90:0/116162311 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3bdc109fc0 0x7f3bdc19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:43.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.344+0000 7f3be1361640 1 -- 172.21.15.90:0/116162311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bdc109b20 msgr2=0x7f3bdc19ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:43.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.344+0000 7f3be1361640 1 --2- 172.21.15.90:0/116162311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bdc109b20 0x7f3bdc19ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:43.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.344+0000 7f3be1361640 1 -- 172.21.15.90:0/116162311 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3bdc109fc0 msgr2=0x7f3bdc19eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:43.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.344+0000 7f3be1361640 1 --2- 172.21.15.90:0/116162311 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3bdc109fc0 0x7f3bdc19eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:43.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.344+0000 7f3be1361640 1 -- 172.21.15.90:0/116162311 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bdc19f630 con 0x7f3bdc1027a0 2024-04-25T11:47:43.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.344+0000 7f3be1b62640 1 --2- 172.21.15.90:0/116162311 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3bdc109fc0 0x7f3bdc19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:47:43.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.344+0000 7f3be0b60640 1 --2- 172.21.15.90:0/116162311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bdc109b20 0x7f3bdc19ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:47:43.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.344+0000 7f3be1361640 1 --2- 172.21.15.90:0/116162311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bdc1027a0 0x7f3bdc19a7d0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f3bd800d740 tx=0x7f3bd800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:43.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.345+0000 7f3bd27fc640 1 -- 172.21.15.90:0/116162311 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd8017070 con 0x7f3bdc1027a0 2024-04-25T11:47:43.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.345+0000 7f3be35c5640 1 -- 172.21.15.90:0/116162311 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bdc19f920 con 0x7f3bdc1027a0 2024-04-25T11:47:43.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.345+0000 7f3bd27fc640 1 -- 172.21.15.90:0/116162311 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bd800de50 con 0x7f3bdc1027a0 2024-04-25T11:47:43.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.345+0000 7f3bd27fc640 1 -- 172.21.15.90:0/116162311 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd8004ce0 con 0x7f3bdc1027a0 2024-04-25T11:47:43.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.345+0000 7f3be35c5640 1 -- 172.21.15.90:0/116162311 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bdc1a72a0 con 0x7f3bdc1027a0 2024-04-25T11:47:43.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.346+0000 7f3be35c5640 1 -- 172.21.15.90:0/116162311 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bdc104f10 con 0x7f3bdc1027a0 2024-04-25T11:47:43.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.347+0000 7f3bd27fc640 1 -- 172.21.15.90:0/116162311 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f3bd8004e80 con 0x7f3bdc1027a0 2024-04-25T11:47:43.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.348+0000 7f3bd27fc640 1 --2- 172.21.15.90:0/116162311 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3bc80afc60 0x7f3bc80b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:43.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.348+0000 7f3be0b60640 1 --2- 172.21.15.90:0/116162311 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3bc80afc60 0x7f3bc80b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:43.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.348+0000 7f3bd27fc640 1 -- 172.21.15.90:0/116162311 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9044+0+0 (secure 0 0 0) 0x7f3bd80df0b0 con 0x7f3bdc1027a0 2024-04-25T11:47:43.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.348+0000 7f3be0b60640 1 --2- 172.21.15.90:0/116162311 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3bc80afc60 0x7f3bc80b2170 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc009230 tx=0x7f3bcc000bc0 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:43.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.352+0000 7f3bd27fc640 1 -- 172.21.15.90:0/116162311 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3bd80a76e0 con 0x7f3bdc1027a0 2024-04-25T11:47:43.429 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:43 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2719561625' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T11:47:43.429 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:43 smithi090 ceph-mon[29575]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T11:47:43.429 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:43 smithi090 ceph-mon[29575]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T11:47:43.429 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:43 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/2719561625' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T11:47:43.429 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:43 smithi090 ceph-mon[29575]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T11:47:43.429 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:43 smithi090 ceph-mon[29575]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-25T11:47:43.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:43 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/4067764342' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T11:47:43.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:43.537+0000 7f3be35c5640 1 -- 172.21.15.90:0/116162311 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3bdc0692e0 con 0x7f3bdc1027a0 2024-04-25T11:47:43.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:43 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2719561625' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T11:47:43.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:43 smithi138 ceph-mon[32328]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T11:47:43.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:43 smithi138 ceph-mon[32328]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T11:47:43.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:43 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/2719561625' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T11:47:43.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:43 smithi138 ceph-mon[32328]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T11:47:43.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:43 smithi138 ceph-mon[32328]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-25T11:47:43.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:43 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/4067764342' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T11:47:43.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:43 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2719561625' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T11:47:43.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:43 smithi156 ceph-mon[33045]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T11:47:43.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:43 smithi156 ceph-mon[33045]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T11:47:43.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:43 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/2719561625' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T11:47:43.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:43 smithi156 ceph-mon[33045]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T11:47:43.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:43 smithi156 ceph-mon[33045]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-25T11:47:43.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:43 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/4067764342' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T11:47:44.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.214+0000 7f3bd27fc640 1 -- 172.21.15.90:0/116162311 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v96) ==== 181+0+0 (secure 0 0 0) 0x7f3bd80a6fe0 con 0x7f3bdc1027a0 2024-04-25T11:47:44.200 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_metadata' removed 2024-04-25T11:47:44.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.217+0000 7f3be35c5640 1 -- 172.21.15.90:0/116162311 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3bc80afc60 msgr2=0x7f3bc80b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:44.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.217+0000 7f3be35c5640 1 --2- 172.21.15.90:0/116162311 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3bc80afc60 0x7f3bc80b2170 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc009230 tx=0x7f3bcc000bc0 comp rx=0 tx=0).stop 2024-04-25T11:47:44.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.217+0000 7f3be35c5640 1 -- 172.21.15.90:0/116162311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bdc1027a0 msgr2=0x7f3bdc19a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:44.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.217+0000 7f3be35c5640 1 --2- 172.21.15.90:0/116162311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bdc1027a0 0x7f3bdc19a7d0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f3bd800d740 tx=0x7f3bd800dc00 comp rx=0 tx=0).stop 2024-04-25T11:47:44.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.217+0000 7f3be35c5640 1 -- 172.21.15.90:0/116162311 shutdown_connections 2024-04-25T11:47:44.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.217+0000 7f3be35c5640 1 --2- 172.21.15.90:0/116162311 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f3bc80afc60 0x7f3bc80b2170 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:44.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.217+0000 7f3be35c5640 1 --2- 172.21.15.90:0/116162311 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3bdc109fc0 0x7f3bdc19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:44.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.217+0000 7f3be35c5640 1 --2- 172.21.15.90:0/116162311 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bdc109b20 0x7f3bdc19ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:44.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.217+0000 7f3be35c5640 1 --2- 172.21.15.90:0/116162311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bdc1027a0 0x7f3bdc19a7d0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:44.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.217+0000 7f3be35c5640 1 -- 172.21.15.90:0/116162311 >> 172.21.15.90:0/116162311 conn(0x7f3bdc077880 msgr2=0x7f3bdc0ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:44.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.217+0000 7f3be35c5640 1 -- 172.21.15.90:0/116162311 shutdown_connections 2024-04-25T11:47:44.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.217+0000 7f3be35c5640 1 -- 172.21.15.90:0/116162311 wait complete. 2024-04-25T11:47:44.216 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T11:47:44.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.344+0000 7f8ad83e3640 1 -- 172.21.15.90:0/1165901443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ad0103880 msgr2=0x7f8ad010f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:44.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.344+0000 7f8ad83e3640 1 --2- 172.21.15.90:0/1165901443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ad0103880 0x7f8ad010f690 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8009ca0 tx=0x7f8ac802a8c0 comp rx=0 tx=0).stop 2024-04-25T11:47:44.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.344+0000 7f8ad83e3640 1 -- 172.21.15.90:0/1165901443 shutdown_connections 2024-04-25T11:47:44.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.344+0000 7f8ad83e3640 1 --2- 172.21.15.90:0/1165901443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ad0103880 0x7f8ad010f690 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:44.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.344+0000 7f8ad83e3640 1 --2- 172.21.15.90:0/1165901443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ad01090a0 0x7f8ad0103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:44.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.344+0000 7f8ad83e3640 1 --2- 172.21.15.90:0/1165901443 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ad0108c00 0x7f8ad0102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:44.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.344+0000 7f8ad83e3640 1 -- 172.21.15.90:0/1165901443 >> 172.21.15.90:0/1165901443 conn(0x7f8ad00fd810 msgr2=0x7f8ad00ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:44.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.344+0000 7f8ad83e3640 1 -- 172.21.15.90:0/1165901443 shutdown_connections 2024-04-25T11:47:44.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.344+0000 7f8ad83e3640 1 -- 172.21.15.90:0/1165901443 wait complete. 2024-04-25T11:47:44.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.345+0000 7f8ad83e3640 1 Processor -- start 2024-04-25T11:47:44.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.345+0000 7f8ad83e3640 1 -- start start 2024-04-25T11:47:44.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.345+0000 7f8ad83e3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ad0103880 0x7f8ad019b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:44.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.345+0000 7f8ad83e3640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ad0108c00 0x7f8ad019bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:44.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.345+0000 7f8ad83e3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ad01090a0 0x7f8ad019c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:44.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.345+0000 7f8ad83e3640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad0112e20 con 0x7f8ad0103880 2024-04-25T11:47:44.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.345+0000 7f8ad83e3640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad0112ca0 con 0x7f8ad0108c00 2024-04-25T11:47:44.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.345+0000 7f8ad83e3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad0112fa0 con 0x7f8ad01090a0 2024-04-25T11:47:44.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.345+0000 7f8ad617f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ad0103880 0x7f8ad019b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:44.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.345+0000 7f8ad617f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ad0103880 0x7f8ad019b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48600/0 (socket says 172.21.15.90:48600) 2024-04-25T11:47:44.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.345+0000 7f8ad617f640 1 -- 172.21.15.90:0/3715283822 learned_addr learned my addr 172.21.15.90:0/3715283822 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-25T11:47:44.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.345+0000 7f8ad597e640 1 --2- 172.21.15.90:0/3715283822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ad0108c00 0x7f8ad019bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:44.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.346+0000 7f8ad6980640 1 --2- 172.21.15.90:0/3715283822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ad01090a0 0x7f8ad019c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:44.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.346+0000 7f8ad617f640 1 -- 172.21.15.90:0/3715283822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ad01090a0 msgr2=0x7f8ad019c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:44.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.346+0000 7f8ad617f640 1 --2- 172.21.15.90:0/3715283822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ad01090a0 0x7f8ad019c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:44.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.346+0000 7f8ad617f640 1 -- 172.21.15.90:0/3715283822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ad0108c00 msgr2=0x7f8ad019bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:44.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.346+0000 7f8ad617f640 1 --2- 172.21.15.90:0/3715283822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ad0108c00 0x7f8ad019bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:44.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.346+0000 7f8ad617f640 1 -- 172.21.15.90:0/3715283822 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad01a6a20 con 0x7f8ad0103880 2024-04-25T11:47:44.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.346+0000 7f8ad617f640 1 --2- 172.21.15.90:0/3715283822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ad0103880 0x7f8ad019b6e0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f8acc00d740 tx=0x7f8acc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:44.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.346+0000 7f8ac77fe640 1 -- 172.21.15.90:0/3715283822 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8acc017070 con 0x7f8ad0103880 2024-04-25T11:47:44.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.346+0000 7f8ac77fe640 1 -- 172.21.15.90:0/3715283822 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8acc00de50 con 0x7f8ad0103880 2024-04-25T11:47:44.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.346+0000 7f8ad83e3640 1 -- 172.21.15.90:0/3715283822 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ad01a6d10 con 0x7f8ad0103880 2024-04-25T11:47:44.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.346+0000 7f8ac77fe640 1 -- 172.21.15.90:0/3715283822 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8acc00f620 con 0x7f8ad0103880 2024-04-25T11:47:44.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.346+0000 7f8ad83e3640 1 -- 172.21.15.90:0/3715283822 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ad01a7200 con 0x7f8ad0103880 2024-04-25T11:47:44.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.347+0000 7f8ad83e3640 1 -- 172.21.15.90:0/3715283822 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ad0069310 con 0x7f8ad0103880 2024-04-25T11:47:44.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.349+0000 7f8ac77fe640 1 -- 172.21.15.90:0/3715283822 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 147651+0+0 (secure 0 0 0) 0x7f8acc01a070 con 0x7f8ad0103880 2024-04-25T11:47:44.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.349+0000 7f8ac77fe640 1 --2- 172.21.15.90:0/3715283822 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8abc0afc60 0x7f8abc0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:47:44.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.349+0000 7f8ad597e640 1 --2- 172.21.15.90:0/3715283822 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8abc0afc60 0x7f8abc0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:47:44.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.349+0000 7f8ac77fe640 1 -- 172.21.15.90:0/3715283822 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8490+0+0 (secure 0 0 0) 0x7f8acc0de9a0 con 0x7f8ad0103880 2024-04-25T11:47:44.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.350+0000 7f8ad597e640 1 --2- 172.21.15.90:0/3715283822 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8abc0afc60 0x7f8abc0b2170 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0107df0 tx=0x7f8ac000b410 comp rx=0 tx=0).ready entity=mgr.15366 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:47:44.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.353+0000 7f8ac77fe640 1 -- 172.21.15.90:0/3715283822 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8acc0a7260 con 0x7f8ad0103880 2024-04-25T11:47:44.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: pgmap v395: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T11:47:44.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T11:47:44.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T11:47:44.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: Cluster is now healthy 2024-04-25T11:47:44.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/4067764342' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T11:47:44.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.90:6838/3978229027,v1:172.21.15.90:6840/3978229027] up:boot 2024-04-25T11:47:44.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.90:6839/578860467,v1:172.21.15.90:6841/578860467] up:boot 2024-04-25T11:47:44.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.138:6833/2817416125,v1:172.21.15.138:6837/2817416125] up:boot 2024-04-25T11:47:44.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.156:6832/2697160122,v1:172.21.15.156:6835/2697160122] up:boot 2024-04-25T11:47:44.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.138:6832/1346169979,v1:172.21.15.138:6836/1346169979] up:boot 2024-04-25T11:47:44.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.156:6834/2548444140,v1:172.21.15.156:6837/2548444140] up:boot 2024-04-25T11:47:44.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.138:6834/3264980726,v1:172.21.15.138:6839/3264980726] up:boot 2024-04-25T11:47:44.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.138:6835/1014831896,v1:172.21.15.138:6838/1014831896] up:boot 2024-04-25T11:47:44.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.156:6836/3554366843,v1:172.21.15.156:6838/3554366843] up:boot 2024-04-25T11:47:44.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: mds.? [v2:172.21.15.156:6833/1394283979,v1:172.21.15.156:6839/1394283979] up:boot 2024-04-25T11:47:44.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: fsmap 12 up:standby 2024-04-25T11:47:44.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T11:47:44.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T11:47:44.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T11:47:44.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T11:47:44.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T11:47:44.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T11:47:44.474 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T11:47:44.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T11:47:44.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T11:47:44.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T11:47:44.475 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:44 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/116162311' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T11:47:44.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:44.532+0000 7f8ad83e3640 1 -- 172.21.15.90:0/3715283822 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8ad01021d0 con 0x7f8ad0103880 2024-04-25T11:47:44.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: pgmap v395: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T11:47:44.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T11:47:44.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T11:47:44.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: Cluster is now healthy 2024-04-25T11:47:44.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/4067764342' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T11:47:44.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.90:6838/3978229027,v1:172.21.15.90:6840/3978229027] up:boot 2024-04-25T11:47:44.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.90:6839/578860467,v1:172.21.15.90:6841/578860467] up:boot 2024-04-25T11:47:44.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.138:6833/2817416125,v1:172.21.15.138:6837/2817416125] up:boot 2024-04-25T11:47:44.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.156:6832/2697160122,v1:172.21.15.156:6835/2697160122] up:boot 2024-04-25T11:47:44.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.138:6832/1346169979,v1:172.21.15.138:6836/1346169979] up:boot 2024-04-25T11:47:44.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.156:6834/2548444140,v1:172.21.15.156:6837/2548444140] up:boot 2024-04-25T11:47:44.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.138:6834/3264980726,v1:172.21.15.138:6839/3264980726] up:boot 2024-04-25T11:47:44.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.138:6835/1014831896,v1:172.21.15.138:6838/1014831896] up:boot 2024-04-25T11:47:44.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.156:6836/3554366843,v1:172.21.15.156:6838/3554366843] up:boot 2024-04-25T11:47:44.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: mds.? [v2:172.21.15.156:6833/1394283979,v1:172.21.15.156:6839/1394283979] up:boot 2024-04-25T11:47:44.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: fsmap 12 up:standby 2024-04-25T11:47:44.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T11:47:44.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T11:47:44.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T11:47:44.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T11:47:44.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T11:47:44.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T11:47:44.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T11:47:44.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T11:47:44.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T11:47:44.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T11:47:44.551 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:44 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/116162311' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T11:47:44.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: pgmap v395: 97 pgs: 97 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T11:47:44.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T11:47:44.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T11:47:44.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: Cluster is now healthy 2024-04-25T11:47:44.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/4067764342' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T11:47:44.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.90:6838/3978229027,v1:172.21.15.90:6840/3978229027] up:boot 2024-04-25T11:47:44.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.90:6839/578860467,v1:172.21.15.90:6841/578860467] up:boot 2024-04-25T11:47:44.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.138:6833/2817416125,v1:172.21.15.138:6837/2817416125] up:boot 2024-04-25T11:47:44.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.156:6832/2697160122,v1:172.21.15.156:6835/2697160122] up:boot 2024-04-25T11:47:44.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.138:6832/1346169979,v1:172.21.15.138:6836/1346169979] up:boot 2024-04-25T11:47:44.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.156:6834/2548444140,v1:172.21.15.156:6837/2548444140] up:boot 2024-04-25T11:47:44.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.138:6834/3264980726,v1:172.21.15.138:6839/3264980726] up:boot 2024-04-25T11:47:44.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.138:6835/1014831896,v1:172.21.15.138:6838/1014831896] up:boot 2024-04-25T11:47:44.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.156:6836/3554366843,v1:172.21.15.156:6838/3554366843] up:boot 2024-04-25T11:47:44.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: mds.? [v2:172.21.15.156:6833/1394283979,v1:172.21.15.156:6839/1394283979] up:boot 2024-04-25T11:47:44.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: fsmap 12 up:standby 2024-04-25T11:47:44.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T11:47:44.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T11:47:44.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T11:47:44.586 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T11:47:44.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T11:47:44.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T11:47:44.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T11:47:44.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T11:47:44.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T11:47:44.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: from='mgr.15366 172.21.15.90:0/3118242456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T11:47:44.587 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:44 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/116162311' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T11:47:45.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:45.216+0000 7f8ac77fe640 1 -- 172.21.15.90:0/3715283822 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v97) ==== 169+0+0 (secure 0 0 0) 0x7f8ad01021d0 con 0x7f8ad0103880 2024-04-25T11:47:45.201 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data' removed 2024-04-25T11:47:45.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:45.218+0000 7f8ad83e3640 1 -- 172.21.15.90:0/3715283822 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8abc0afc60 msgr2=0x7f8abc0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:45.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:45.218+0000 7f8ad83e3640 1 --2- 172.21.15.90:0/3715283822 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8abc0afc60 0x7f8abc0b2170 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0107df0 tx=0x7f8ac000b410 comp rx=0 tx=0).stop 2024-04-25T11:47:45.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:45.218+0000 7f8ad83e3640 1 -- 172.21.15.90:0/3715283822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ad0103880 msgr2=0x7f8ad019b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:47:45.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:45.218+0000 7f8ad83e3640 1 --2- 172.21.15.90:0/3715283822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ad0103880 0x7f8ad019b6e0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f8acc00d740 tx=0x7f8acc00dc00 comp rx=0 tx=0).stop 2024-04-25T11:47:45.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:45.219+0000 7f8ad83e3640 1 -- 172.21.15.90:0/3715283822 shutdown_connections 2024-04-25T11:47:45.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:45.219+0000 7f8ad83e3640 1 --2- 172.21.15.90:0/3715283822 >> [v2:172.21.15.90:6800/4078377640,v1:172.21.15.90:6801/4078377640] conn(0x7f8abc0afc60 0x7f8abc0b2170 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:45.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:45.219+0000 7f8ad83e3640 1 --2- 172.21.15.90:0/3715283822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ad01090a0 0x7f8ad019c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:45.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:45.219+0000 7f8ad83e3640 1 --2- 172.21.15.90:0/3715283822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ad0108c00 0x7f8ad019bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:45.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:45.219+0000 7f8ad83e3640 1 --2- 172.21.15.90:0/3715283822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ad0103880 0x7f8ad019b6e0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:47:45.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:45.219+0000 7f8ad83e3640 1 -- 172.21.15.90:0/3715283822 >> 172.21.15.90:0/3715283822 conn(0x7f8ad00fd810 msgr2=0x7f8ad00ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:47:45.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:45.219+0000 7f8ad83e3640 1 -- 172.21.15.90:0/3715283822 shutdown_connections 2024-04-25T11:47:45.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-25T11:47:45.219+0000 7f8ad83e3640 1 -- 172.21.15.90:0/3715283822 wait complete. 2024-04-25T11:47:45.227 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T11:47:45.266 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T11:47:45.301 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T11:47:45.341 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-25T11:47:45.341 DEBUG:teuthology.orchestra.run.smithi090:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T11:47:45.359 DEBUG:teuthology.orchestra.run.smithi138:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T11:47:45.374 DEBUG:teuthology.orchestra.run.smithi156:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T11:47:45.395 INFO:tasks.cephadm:Stopping all daemons... 2024-04-25T11:47:45.395 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-25T11:47:45.396 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mon.a 2024-04-25T11:47:45.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:45 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/116162311' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T11:47:45.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:45 smithi090 ceph-mon[29575]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T11:47:45.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:45 smithi090 ceph-mon[29575]: pgmap v397: 65 pgs: 65 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:47:45.472 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:45 smithi090 ceph-mon[29575]: from='client.? 172.21.15.90:0/3715283822' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T11:47:45.473 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:45 smithi090 systemd[1]: Stopping Ceph mon.a for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:47:45.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:45 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/116162311' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T11:47:45.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:45 smithi138 ceph-mon[32328]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T11:47:45.549 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:45 smithi138 ceph-mon[32328]: pgmap v397: 65 pgs: 65 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:47:45.550 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:45 smithi138 ceph-mon[32328]: from='client.? 172.21.15.90:0/3715283822' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T11:47:45.584 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:45 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/116162311' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T11:47:45.584 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:45 smithi156 ceph-mon[33045]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T11:47:45.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:45 smithi156 ceph-mon[33045]: pgmap v397: 65 pgs: 65 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T11:47:45.585 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:45 smithi156 ceph-mon[33045]: from='client.? 172.21.15.90:0/3715283822' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T11:47:45.972 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:45 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-a[29571]: 2024-04-25T11:47:45.697+0000 7fb091810700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:47:45.972 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:45 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-a[29571]: 2024-04-25T11:47:45.697+0000 7fb091810700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-25T11:47:45.972 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:45 smithi090 podman[102672]: 2024-04-25 11:47:45.75882541 +0000 UTC m=+0.156619723 container died 285c6eedf8b9106e32566c376993ff10f0e71c2f4dde2fcfead525fad39050d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-25T11:47:46.401 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:46 smithi090 podman[102672]: 2024-04-25 11:47:46.114724139 +0000 UTC m=+0.512518457 container cleanup 285c6eedf8b9106e32566c376993ff10f0e71c2f4dde2fcfead525fad39050d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T11:47:46.401 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 25 11:47:46 smithi090 bash[102672]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-a 2024-04-25T11:47:46.551 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mon.a.service' 2024-04-25T11:47:46.589 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:47:46.589 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-25T11:47:46.590 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-25T11:47:46.590 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mon.b 2024-04-25T11:47:46.878 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:46 smithi138 systemd[1]: Stopping Ceph mon.b for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:47:47.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:46 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-b[32324]: 2024-04-25T11:47:46.892+0000 7f066b8b2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:47:47.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:46 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-b[32324]: 2024-04-25T11:47:46.892+0000 7f066b8b2700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-25T11:47:47.299 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:47 smithi138 podman[68617]: 2024-04-25 11:47:47.085923831 +0000 UTC m=+0.273708184 container died f9ca4b15f41e885c53666a54d044277bc52233a951949a8e2f81d79daa49850b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-b, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T11:47:47.595 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:47 smithi138 podman[68617]: 2024-04-25 11:47:47.417408505 +0000 UTC m=+0.605192839 container cleanup f9ca4b15f41e885c53666a54d044277bc52233a951949a8e2f81d79daa49850b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-b, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T11:47:47.596 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:47 smithi138 bash[68617]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-b 2024-04-25T11:47:47.835 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mon.b.service' 2024-04-25T11:47:47.855 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:47 smithi138 podman[68632]: 2024-04-25 11:47:47.61052445 +0000 UTC m=+0.522267133 container remove f9ca4b15f41e885c53666a54d044277bc52233a951949a8e2f81d79daa49850b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-b, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-25T11:47:47.855 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:47 smithi138 systemd[1]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mon.b.service: Deactivated successfully. 2024-04-25T11:47:47.855 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:47 smithi138 systemd[1]: Stopped Ceph mon.b for 58bd4e7a-02f6-11ef-bc93-c7b262605968. 2024-04-25T11:47:47.856 INFO:journalctl@ceph.mon.b.smithi138.stdout:Apr 25 11:47:47 smithi138 systemd[1]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mon.b.service: Consumed 14.663s CPU time. 2024-04-25T11:47:47.875 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:47:47.875 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-25T11:47:47.875 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-25T11:47:47.875 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mon.c 2024-04-25T11:47:48.163 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:47 smithi156 systemd[1]: Stopping Ceph mon.c for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:47:48.428 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:48 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-c[33041]: 2024-04-25T11:47:48.178+0000 7f2bfe0a4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:47:48.428 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:48 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-c[33041]: 2024-04-25T11:47:48.178+0000 7f2bfe0a4700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-25T11:47:48.715 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:48 smithi156 podman[69981]: 2024-04-25 11:47:48.443540622 +0000 UTC m=+0.376201180 container died ad026314adcac9d5e4d29022c460be023c2c49020219cd7388aa04337b6eb872 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-c, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-25T11:47:49.085 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:48 smithi156 podman[69981]: 2024-04-25 11:47:48.901380717 +0000 UTC m=+0.834041290 container cleanup ad026314adcac9d5e4d29022c460be023c2c49020219cd7388aa04337b6eb872 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-25T11:47:49.085 INFO:journalctl@ceph.mon.c.smithi156.stdout:Apr 25 11:47:48 smithi156 bash[69981]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mon-c 2024-04-25T11:47:49.318 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mon.c.service' 2024-04-25T11:47:49.404 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:47:49.404 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-25T11:47:49.404 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-25T11:47:49.404 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mgr.x 2024-04-25T11:47:49.697 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:47:49 smithi090 systemd[1]: Stopping Ceph mgr.x for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:47:49.972 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:47:49 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:47:49.711+0000 7f4402f17700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:47:49.972 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:47:49 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x[29806]: 2024-04-25T11:47:49.711+0000 7f4402f17700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-25T11:47:49.972 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:47:49 smithi090 podman[102795]: 2024-04-25 11:47:49.754042177 +0000 UTC m=+0.153791521 container died 89807c68fe179e40f7bbd2e18fdea4b8165d6ee97b0f33c86f24fc1edc5012f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T11:47:50.364 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:47:50 smithi090 podman[102795]: 2024-04-25 11:47:50.111359568 +0000 UTC m=+0.511108912 container cleanup 89807c68fe179e40f7bbd2e18fdea4b8165d6ee97b0f33c86f24fc1edc5012f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-25T11:47:50.364 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 25 11:47:50 smithi090 bash[102795]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-x 2024-04-25T11:47:50.521 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-25T11:47:50.555 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:47:50.556 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-25T11:47:50.556 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-25T11:47:50.556 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mgr.y 2024-04-25T11:47:50.903 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:47:50 smithi138 systemd[1]: Stopping Ceph mgr.y for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:47:51.225 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:47:50 smithi138 podman[68741]: 2024-04-25 11:47:50.918961045 +0000 UTC m=+0.156728171 container died a7e64dbe4f67ffc5e6073d568df015f511b24a2e32b8c39b55d05693f4704f7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-25T11:47:51.549 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:47:51 smithi138 podman[68741]: 2024-04-25 11:47:51.34715537 +0000 UTC m=+0.584922498 container cleanup a7e64dbe4f67ffc5e6073d568df015f511b24a2e32b8c39b55d05693f4704f7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-25T11:47:51.549 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:47:51 smithi138 bash[68741]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y 2024-04-25T11:47:51.812 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:47:51 smithi138 podman[68755]: 2024-04-25 11:47:51.573501505 +0000 UTC m=+0.658036649 container remove a7e64dbe4f67ffc5e6073d568df015f511b24a2e32b8c39b55d05693f4704f7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-y, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T11:47:51.812 INFO:journalctl@ceph.mgr.y.smithi138.stdout:Apr 25 11:47:51 smithi138 systemd[1]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-04-25T11:47:51.827 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-25T11:47:51.859 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:47:51.859 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-25T11:47:51.859 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-25T11:47:51.859 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mgr.z 2024-04-25T11:47:52.207 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:47:51 smithi156 systemd[1]: Stopping Ceph mgr.z for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:47:52.469 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:47:52 smithi156 podman[70104]: 2024-04-25 11:47:52.22286819 +0000 UTC m=+0.152960411 container died 6b0ef806886423b86d8c37c33e2ade481dbd2e522ba29a5148451ce5fc436c3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-25T11:47:52.735 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:47:52 smithi156 podman[70104]: 2024-04-25 11:47:52.558162267 +0000 UTC m=+0.488254496 container cleanup 6b0ef806886423b86d8c37c33e2ade481dbd2e522ba29a5148451ce5fc436c3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-25T11:47:52.735 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:47:52 smithi156 bash[70104]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z 2024-04-25T11:47:52.981 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-25T11:47:53.001 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:47:52 smithi156 podman[70118]: 2024-04-25 11:47:52.750779331 +0000 UTC m=+0.531052004 container remove 6b0ef806886423b86d8c37c33e2ade481dbd2e522ba29a5148451ce5fc436c3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mgr-z, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-25T11:47:53.001 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:47:52 smithi156 systemd[1]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-04-25T11:47:53.001 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:47:52 smithi156 systemd[1]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mgr.z.service: Failed with result 'exit-code'. 2024-04-25T11:47:53.002 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:47:52 smithi156 systemd[1]: Stopped Ceph mgr.z for 58bd4e7a-02f6-11ef-bc93-c7b262605968. 2024-04-25T11:47:53.002 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Apr 25 11:47:52 smithi156 systemd[1]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mgr.z.service: Consumed 21.097s CPU time. 2024-04-25T11:47:53.013 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:47:53.014 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-25T11:47:53.014 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-25T11:47:53.014 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.0 2024-04-25T11:47:53.222 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 25 11:47:53 smithi090 systemd[1]: Stopping Ceph osd.0 for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:47:53.722 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 25 11:47:53 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-0[44792]: 2024-04-25T11:47:53.341+0000 7f0e596a3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:47:53.723 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 25 11:47:53 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-0[44792]: 2024-04-25T11:47:53.341+0000 7f0e596a3700 -1 osd.0 97 *** Got signal Terminated *** 2024-04-25T11:47:53.723 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 25 11:47:53 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-0[44792]: 2024-04-25T11:47:53.341+0000 7f0e596a3700 -1 osd.0 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T11:47:59.223 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 25 11:47:58 smithi090 podman[102915]: 2024-04-25 11:47:58.978891794 +0000 UTC m=+5.769527584 container died 94273764a14436662b51ebcd01af10a19111f71b27d677d5f3b3eb7a1481f0bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T11:47:59.722 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 25 11:47:59 smithi090 podman[102915]: 2024-04-25 11:47:59.330272406 +0000 UTC m=+6.120908195 container cleanup 94273764a14436662b51ebcd01af10a19111f71b27d677d5f3b3eb7a1481f0bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-04-25T11:47:59.722 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 25 11:47:59 smithi090 bash[102915]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-0 2024-04-25T11:47:59.722 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 25 11:47:59 smithi090 podman[102931]: 2024-04-25 11:47:59.539997341 +0000 UTC m=+0.556083329 container remove 94273764a14436662b51ebcd01af10a19111f71b27d677d5f3b3eb7a1481f0bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T11:48:00.222 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 25 11:47:59 smithi090 podman[103022]: 2024-04-25 11:47:59.84298103 +0000 UTC m=+0.018266988 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T11:48:00.223 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 25 11:47:59 smithi090 podman[103022]: 2024-04-25 11:47:59.94083172 +0000 UTC m=+0.116117672 container create 1c33526e4e0df1965e5b2c07e43f534229d68ff8003c852d1fca1262483e97f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-25T11:48:00.223 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 25 11:48:00 smithi090 podman[103022]: 2024-04-25 11:48:00.186951955 +0000 UTC m=+0.362237914 container init 1c33526e4e0df1965e5b2c07e43f534229d68ff8003c852d1fca1262483e97f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T11:48:00.223 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 25 11:48:00 smithi090 podman[103022]: 2024-04-25 11:48:00.19045208 +0000 UTC m=+0.365738039 container start 1c33526e4e0df1965e5b2c07e43f534229d68ff8003c852d1fca1262483e97f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-25T11:48:00.722 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 25 11:48:00 smithi090 podman[103022]: 2024-04-25 11:48:00.280079277 +0000 UTC m=+0.455365243 container attach 1c33526e4e0df1965e5b2c07e43f534229d68ff8003c852d1fca1262483e97f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-25T11:48:00.722 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 25 11:48:00 smithi090 podman[103022]: 2024-04-25 11:48:00.479282641 +0000 UTC m=+0.654568593 container died 1c33526e4e0df1965e5b2c07e43f534229d68ff8003c852d1fca1262483e97f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-25T11:48:01.051 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.0.service' 2024-04-25T11:48:01.090 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:48:01.090 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-25T11:48:01.090 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-25T11:48:01.090 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.1 2024-04-25T11:48:01.386 INFO:journalctl@ceph.osd.1.smithi138.stdout:Apr 25 11:48:01 smithi138 systemd[1]: Stopping Ceph osd.1 for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:48:01.799 INFO:journalctl@ceph.osd.1.smithi138.stdout:Apr 25 11:48:01 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-1[36919]: 2024-04-25T11:48:01.400+0000 7fa1b6329700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:48:01.799 INFO:journalctl@ceph.osd.1.smithi138.stdout:Apr 25 11:48:01 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-1[36919]: 2024-04-25T11:48:01.400+0000 7fa1b6329700 -1 osd.1 97 *** Got signal Terminated *** 2024-04-25T11:48:01.799 INFO:journalctl@ceph.osd.1.smithi138.stdout:Apr 25 11:48:01 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-1[36919]: 2024-04-25T11:48:01.400+0000 7fa1b6329700 -1 osd.1 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T11:48:07.119 INFO:journalctl@ceph.osd.1.smithi138.stdout:Apr 25 11:48:06 smithi138 podman[68865]: 2024-04-25 11:48:06.848654696 +0000 UTC m=+5.555660257 container died 5bb785777d3e8f9a6d0f6d8c94ab289579d44fd3cfdf660ed1f3faef9f6e67b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-1, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T11:48:07.461 INFO:journalctl@ceph.osd.1.smithi138.stdout:Apr 25 11:48:07 smithi138 podman[68865]: 2024-04-25 11:48:07.234058266 +0000 UTC m=+5.941063826 container cleanup 5bb785777d3e8f9a6d0f6d8c94ab289579d44fd3cfdf660ed1f3faef9f6e67b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-1, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2) 2024-04-25T11:48:07.462 INFO:journalctl@ceph.osd.1.smithi138.stdout:Apr 25 11:48:07 smithi138 bash[68865]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-1 2024-04-25T11:48:07.779 INFO:journalctl@ceph.osd.1.smithi138.stdout:Apr 25 11:48:07 smithi138 podman[68883]: 2024-04-25 11:48:07.476989479 +0000 UTC m=+0.627753129 container remove 5bb785777d3e8f9a6d0f6d8c94ab289579d44fd3cfdf660ed1f3faef9f6e67b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-1, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T11:48:08.049 INFO:journalctl@ceph.osd.1.smithi138.stdout:Apr 25 11:48:07 smithi138 podman[68974]: 2024-04-25 11:48:07.794623795 +0000 UTC m=+0.103120554 container create a8bcc8a3f7ce0b7693d8c6fe8216aa32ecc4c3629b51bb5ff78f5562ab8b261c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-25T11:48:08.049 INFO:journalctl@ceph.osd.1.smithi138.stdout:Apr 25 11:48:07 smithi138 podman[68974]: 2024-04-25 11:48:07.70916204 +0000 UTC m=+0.017658768 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T11:48:08.049 INFO:journalctl@ceph.osd.1.smithi138.stdout:Apr 25 11:48:07 smithi138 podman[68974]: 2024-04-25 11:48:07.996756095 +0000 UTC m=+0.305252805 container init a8bcc8a3f7ce0b7693d8c6fe8216aa32ecc4c3629b51bb5ff78f5562ab8b261c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-25T11:48:08.049 INFO:journalctl@ceph.osd.1.smithi138.stdout:Apr 25 11:48:07 smithi138 podman[68974]: 2024-04-25 11:48:07.999773743 +0000 UTC m=+0.308270462 container start a8bcc8a3f7ce0b7693d8c6fe8216aa32ecc4c3629b51bb5ff78f5562ab8b261c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-1-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T11:48:08.549 INFO:journalctl@ceph.osd.1.smithi138.stdout:Apr 25 11:48:08 smithi138 podman[68974]: 2024-04-25 11:48:08.071632681 +0000 UTC m=+0.380129403 container attach a8bcc8a3f7ce0b7693d8c6fe8216aa32ecc4c3629b51bb5ff78f5562ab8b261c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-25T11:48:08.549 INFO:journalctl@ceph.osd.1.smithi138.stdout:Apr 25 11:48:08 smithi138 podman[68974]: 2024-04-25 11:48:08.294206723 +0000 UTC m=+0.602703438 container died a8bcc8a3f7ce0b7693d8c6fe8216aa32ecc4c3629b51bb5ff78f5562ab8b261c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T11:48:08.967 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.1.service' 2024-04-25T11:48:09.006 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:48:09.006 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-25T11:48:09.006 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-25T11:48:09.006 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.2 2024-04-25T11:48:09.335 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 11:48:09 smithi156 systemd[1]: Stopping Ceph osd.2 for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:48:09.602 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 11:48:09 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-2[38187]: 2024-04-25T11:48:09.352+0000 7f1d76671700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:48:09.603 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 11:48:09 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-2[38187]: 2024-04-25T11:48:09.352+0000 7f1d76671700 -1 osd.2 97 *** Got signal Terminated *** 2024-04-25T11:48:09.603 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 11:48:09 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-2[38187]: 2024-04-25T11:48:09.352+0000 7f1d76671700 -1 osd.2 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T11:48:15.121 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 11:48:14 smithi156 podman[70228]: 2024-04-25 11:48:14.860561955 +0000 UTC m=+5.645609818 container died 85c7fb478862f32377ca20d2d9b68d55daea50b86726c123cffa2532dd5bf8c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-25T11:48:15.455 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 11:48:15 smithi156 podman[70228]: 2024-04-25 11:48:15.22784003 +0000 UTC m=+6.012887891 container cleanup 85c7fb478862f32377ca20d2d9b68d55daea50b86726c123cffa2532dd5bf8c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-2, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-25T11:48:15.456 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 11:48:15 smithi156 bash[70228]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-2 2024-04-25T11:48:15.767 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 11:48:15 smithi156 podman[70257]: 2024-04-25 11:48:15.470683999 +0000 UTC m=+0.605394774 container remove 85c7fb478862f32377ca20d2d9b68d55daea50b86726c123cffa2532dd5bf8c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-2, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-04-25T11:48:16.059 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 11:48:15 smithi156 podman[70346]: 2024-04-25 11:48:15.686117436 +0000 UTC m=+0.018018727 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T11:48:16.059 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 11:48:15 smithi156 podman[70346]: 2024-04-25 11:48:15.821707671 +0000 UTC m=+0.153609007 container create 41ecf008b3e480ce256e1c6e1ad4b2d0160a3842e10eaacf1804962e98e8189f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2) 2024-04-25T11:48:16.059 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 11:48:16 smithi156 podman[70346]: 2024-04-25 11:48:16.074411627 +0000 UTC m=+0.406312907 container init 41ecf008b3e480ce256e1c6e1ad4b2d0160a3842e10eaacf1804962e98e8189f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T11:48:16.335 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 11:48:16 smithi156 podman[70346]: 2024-04-25 11:48:16.077625068 +0000 UTC m=+0.409526353 container start 41ecf008b3e480ce256e1c6e1ad4b2d0160a3842e10eaacf1804962e98e8189f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T11:48:16.335 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 11:48:16 smithi156 podman[70346]: 2024-04-25 11:48:16.149172535 +0000 UTC m=+0.481073829 container attach 41ecf008b3e480ce256e1c6e1ad4b2d0160a3842e10eaacf1804962e98e8189f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T11:48:16.703 INFO:journalctl@ceph.osd.2.smithi156.stdout:Apr 25 11:48:16 smithi156 podman[70346]: 2024-04-25 11:48:16.37488107 +0000 UTC m=+0.706782350 container died 41ecf008b3e480ce256e1c6e1ad4b2d0160a3842e10eaacf1804962e98e8189f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-25T11:48:16.989 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.2.service' 2024-04-25T11:48:17.029 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:48:17.029 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-25T11:48:17.030 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-25T11:48:17.030 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.3 2024-04-25T11:48:17.322 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 25 11:48:17 smithi090 systemd[1]: Stopping Ceph osd.3 for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:48:17.722 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 25 11:48:17 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-3[51333]: 2024-04-25T11:48:17.335+0000 7fe32e3b9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:48:17.723 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 25 11:48:17 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-3[51333]: 2024-04-25T11:48:17.336+0000 7fe32e3b9700 -1 osd.3 97 *** Got signal Terminated *** 2024-04-25T11:48:17.723 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 25 11:48:17 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-3[51333]: 2024-04-25T11:48:17.336+0000 7fe32e3b9700 -1 osd.3 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T11:48:19.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:19 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:19.509+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:20.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:19 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:19.851+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:20.472 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 25 11:48:20 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-3[51333]: 2024-04-25T11:48:20.160+0000 7fe32a96f700 -1 osd.3 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.691908+0000 front 2024-04-25T11:47:54.691885+0000 (oldest deadline 2024-04-25T11:48:19.391972+0000) 2024-04-25T11:48:20.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:20 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:20.549+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:21.084 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:20 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:20.822+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:21.160 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:20 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:20.861+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:21.472 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 25 11:48:21 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-3[51333]: 2024-04-25T11:48:21.174+0000 7fe32a96f700 -1 osd.3 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.691908+0000 front 2024-04-25T11:47:54.691885+0000 (oldest deadline 2024-04-25T11:48:19.391972+0000) 2024-04-25T11:48:21.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:21 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:21.562+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:22.084 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:21 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:21.831+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:22.123 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:21 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:21.880+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:22.472 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 25 11:48:22 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-3[51333]: 2024-04-25T11:48:22.137+0000 7fe32a96f700 -1 osd.3 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.691908+0000 front 2024-04-25T11:47:54.691885+0000 (oldest deadline 2024-04-25T11:48:19.391972+0000) 2024-04-25T11:48:22.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:22 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:22.273+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.406865+0000 front 2024-04-25T11:47:57.406761+0000 (oldest deadline 2024-04-25T11:48:21.506872+0000) 2024-04-25T11:48:22.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:22 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:22.551+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:23.084 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:22 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:22.847+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:23.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:22 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:22.880+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:23.222 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 25 11:48:22 smithi090 podman[103143]: 2024-04-25 11:48:22.973556246 +0000 UTC m=+5.730921426 container died c38a020443bfdf650a4531c52d6c8c0421d75e0e278646730673b5ccc363fd9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-3, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-25T11:48:23.583 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:23 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:23.244+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.406865+0000 front 2024-04-25T11:47:57.406761+0000 (oldest deadline 2024-04-25T11:48:21.506872+0000) 2024-04-25T11:48:23.584 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 25 11:48:23 smithi090 podman[103143]: 2024-04-25 11:48:23.347423346 +0000 UTC m=+6.104788556 container cleanup c38a020443bfdf650a4531c52d6c8c0421d75e0e278646730673b5ccc363fd9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-3, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-25T11:48:23.584 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 25 11:48:23 smithi090 bash[103143]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-3 2024-04-25T11:48:23.944 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 25 11:48:23 smithi090 podman[103157]: 2024-04-25 11:48:23.598999286 +0000 UTC m=+0.628235567 container remove c38a020443bfdf650a4531c52d6c8c0421d75e0e278646730673b5ccc363fd9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-3, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-25T11:48:23.944 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:23 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:23.835+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:24.049 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:23 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:23.599+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:24.154 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:23 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:23.857+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:24.222 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:24 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:24.198+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.406865+0000 front 2024-04-25T11:47:57.406761+0000 (oldest deadline 2024-04-25T11:48:21.506872+0000) 2024-04-25T11:48:24.223 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 25 11:48:23 smithi090 podman[103251]: 2024-04-25 11:48:23.862696957 +0000 UTC m=+0.017718787 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T11:48:24.223 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 25 11:48:23 smithi090 podman[103251]: 2024-04-25 11:48:23.966502421 +0000 UTC m=+0.121524238 container create 522d42a373dee7b7d341cc190861d1f9e9495e452b62b97cd6f16f4052a4269d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T11:48:24.223 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 25 11:48:24 smithi090 podman[103251]: 2024-04-25 11:48:24.194514216 +0000 UTC m=+0.349536036 container init 522d42a373dee7b7d341cc190861d1f9e9495e452b62b97cd6f16f4052a4269d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T11:48:24.223 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 25 11:48:24 smithi090 podman[103251]: 2024-04-25 11:48:24.197664837 +0000 UTC m=+0.352686655 container start 522d42a373dee7b7d341cc190861d1f9e9495e452b62b97cd6f16f4052a4269d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-25T11:48:24.585 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:24 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:24.169+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:58.231765+0000 front 2024-04-25T11:47:58.231620+0000 (oldest deadline 2024-04-25T11:48:24.131776+0000) 2024-04-25T11:48:24.722 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 25 11:48:24 smithi090 podman[103251]: 2024-04-25 11:48:24.304306123 +0000 UTC m=+0.459327954 container attach 522d42a373dee7b7d341cc190861d1f9e9495e452b62b97cd6f16f4052a4269d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5) 2024-04-25T11:48:24.722 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 25 11:48:24 smithi090 podman[103251]: 2024-04-25 11:48:24.488262651 +0000 UTC m=+0.643284468 container died 522d42a373dee7b7d341cc190861d1f9e9495e452b62b97cd6f16f4052a4269d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-25T11:48:25.034 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:24 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:24.786+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:25.049 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:24 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:24.608+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:25.053 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.3.service' 2024-04-25T11:48:25.093 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:48:25.093 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-25T11:48:25.094 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-25T11:48:25.094 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.4 2024-04-25T11:48:25.177 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:24 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:24.884+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:25.305 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:25 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:25.174+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.406865+0000 front 2024-04-25T11:47:57.406761+0000 (oldest deadline 2024-04-25T11:48:21.506872+0000) 2024-04-25T11:48:25.412 INFO:journalctl@ceph.osd.4.smithi138.stdout:Apr 25 11:48:25 smithi138 systemd[1]: Stopping Ceph osd.4 for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:48:25.584 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:25 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:25.192+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:58.231765+0000 front 2024-04-25T11:47:58.231620+0000 (oldest deadline 2024-04-25T11:48:24.131776+0000) 2024-04-25T11:48:25.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:25 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:25.587+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:25.799 INFO:journalctl@ceph.osd.4.smithi138.stdout:Apr 25 11:48:25 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-4[43600]: 2024-04-25T11:48:25.427+0000 7f6903157700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:48:25.799 INFO:journalctl@ceph.osd.4.smithi138.stdout:Apr 25 11:48:25 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-4[43600]: 2024-04-25T11:48:25.427+0000 7f6903157700 -1 osd.4 97 *** Got signal Terminated *** 2024-04-25T11:48:25.799 INFO:journalctl@ceph.osd.4.smithi138.stdout:Apr 25 11:48:25 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-4[43600]: 2024-04-25T11:48:25.427+0000 7f6903157700 -1 osd.4 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T11:48:26.202 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:25 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:25.786+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:26.202 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:25 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:25.862+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:26.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:26 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:26.216+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.406865+0000 front 2024-04-25T11:47:57.406761+0000 (oldest deadline 2024-04-25T11:48:21.506872+0000) 2024-04-25T11:48:26.584 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:26 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:26.217+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:58.231765+0000 front 2024-04-25T11:47:58.231620+0000 (oldest deadline 2024-04-25T11:48:24.131776+0000) 2024-04-25T11:48:26.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:26 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:26.545+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:27.190 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:26 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:26.793+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:27.239 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:26 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:26.890+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:27.475 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:27 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:27.205+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.406865+0000 front 2024-04-25T11:47:57.406761+0000 (oldest deadline 2024-04-25T11:48:21.506872+0000) 2024-04-25T11:48:27.588 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:27 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:27.253+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:58.231765+0000 front 2024-04-25T11:47:58.231620+0000 (oldest deadline 2024-04-25T11:48:24.131776+0000) 2024-04-25T11:48:27.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:27 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:27.503+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:28.205 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:27 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:27.805+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:28.288 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:27 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:27.925+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:28.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:28 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:28.219+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.406865+0000 front 2024-04-25T11:47:57.406761+0000 (oldest deadline 2024-04-25T11:48:21.506872+0000) 2024-04-25T11:48:28.584 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:28 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:28.302+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:58.231765+0000 front 2024-04-25T11:47:58.231620+0000 (oldest deadline 2024-04-25T11:48:24.131776+0000) 2024-04-25T11:48:28.585 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:28 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:28.302+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:04.632804+0000 front 2024-04-25T11:48:04.632689+0000 (oldest deadline 2024-04-25T11:48:28.132834+0000) 2024-04-25T11:48:28.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:28 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:28.551+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:28.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:28 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:28.551+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:28.799 INFO:journalctl@ceph.osd.4.smithi138.stdout:Apr 25 11:48:28 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-4[43600]: 2024-04-25T11:48:28.696+0000 7f68fb705700 -1 osd.4 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.545987+0000 front 2024-04-25T11:48:05.545897+0000 (oldest deadline 2024-04-25T11:48:27.845993+0000) 2024-04-25T11:48:29.199 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:28 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:28.812+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:29.314 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:28 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:28.969+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:29.314 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:28 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:28.969+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:29.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:29 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:29.214+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.406865+0000 front 2024-04-25T11:47:57.406761+0000 (oldest deadline 2024-04-25T11:48:21.506872+0000) 2024-04-25T11:48:29.584 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:29 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:29.328+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:58.231765+0000 front 2024-04-25T11:47:58.231620+0000 (oldest deadline 2024-04-25T11:48:24.131776+0000) 2024-04-25T11:48:29.585 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:29 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:29.328+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:04.632804+0000 front 2024-04-25T11:48:04.632689+0000 (oldest deadline 2024-04-25T11:48:28.132834+0000) 2024-04-25T11:48:30.049 INFO:journalctl@ceph.osd.4.smithi138.stdout:Apr 25 11:48:29 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-4[43600]: 2024-04-25T11:48:29.716+0000 7f68fb705700 -1 osd.4 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.545987+0000 front 2024-04-25T11:48:05.545897+0000 (oldest deadline 2024-04-25T11:48:27.845993+0000) 2024-04-25T11:48:30.049 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:29 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:29.573+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:30.049 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:29 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:29.573+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:30.182 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:29 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:29.839+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:30.283 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:29 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:29.969+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:30.283 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:29 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:29.969+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:30.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:30 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:30.197+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.406865+0000 front 2024-04-25T11:47:57.406761+0000 (oldest deadline 2024-04-25T11:48:21.506872+0000) 2024-04-25T11:48:30.585 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:30 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:30.298+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:58.231765+0000 front 2024-04-25T11:47:58.231620+0000 (oldest deadline 2024-04-25T11:48:24.131776+0000) 2024-04-25T11:48:30.585 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:30 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:30.298+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:04.632804+0000 front 2024-04-25T11:48:04.632689+0000 (oldest deadline 2024-04-25T11:48:28.132834+0000) 2024-04-25T11:48:30.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:30 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:30.550+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:30.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:30 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:30.550+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:31.179 INFO:journalctl@ceph.osd.4.smithi138.stdout:Apr 25 11:48:30 smithi138 podman[69091]: 2024-04-25 11:48:30.921495816 +0000 UTC m=+5.612190045 container died 50984d573bf4cd7e41a9c156fd7c20487309f2b596e7326695edeaeb2f684de5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-4, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-25T11:48:31.180 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:30 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:30.893+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:06.074147+0000 front 2024-04-25T11:48:06.074119+0000 (oldest deadline 2024-04-25T11:48:30.174234+0000) 2024-04-25T11:48:31.191 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:30 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:30.812+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:31.191 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:30 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:30.812+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:31.253 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:31 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:31.001+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:31.254 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:31 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:31.001+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:31.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:31 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:31.206+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.406865+0000 front 2024-04-25T11:47:57.406761+0000 (oldest deadline 2024-04-25T11:48:21.506872+0000) 2024-04-25T11:48:31.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:31 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:31.206+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.008166+0000 front 2024-04-25T11:48:05.008025+0000 (oldest deadline 2024-04-25T11:48:30.307962+0000) 2024-04-25T11:48:31.498 INFO:journalctl@ceph.osd.4.smithi138.stdout:Apr 25 11:48:31 smithi138 podman[69091]: 2024-04-25 11:48:31.276993436 +0000 UTC m=+5.967687659 container cleanup 50984d573bf4cd7e41a9c156fd7c20487309f2b596e7326695edeaeb2f684de5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-4, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T11:48:31.498 INFO:journalctl@ceph.osd.4.smithi138.stdout:Apr 25 11:48:31 smithi138 bash[69091]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-4 2024-04-25T11:48:31.584 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:31 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:31.268+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:58.231765+0000 front 2024-04-25T11:47:58.231620+0000 (oldest deadline 2024-04-25T11:48:24.131776+0000) 2024-04-25T11:48:31.585 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:31 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:31.268+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:04.632804+0000 front 2024-04-25T11:48:04.632689+0000 (oldest deadline 2024-04-25T11:48:28.132834+0000) 2024-04-25T11:48:31.799 INFO:journalctl@ceph.osd.4.smithi138.stdout:Apr 25 11:48:31 smithi138 podman[69110]: 2024-04-25 11:48:31.553365033 +0000 UTC m=+0.631668980 container remove 50984d573bf4cd7e41a9c156fd7c20487309f2b596e7326695edeaeb2f684de5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-4, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-25T11:48:31.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:31 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:31.512+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:31.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:31 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:31.512+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:32.149 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:31 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:31.888+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:06.074147+0000 front 2024-04-25T11:48:06.074119+0000 (oldest deadline 2024-04-25T11:48:30.174234+0000) 2024-04-25T11:48:32.150 INFO:journalctl@ceph.osd.4.smithi138.stdout:Apr 25 11:48:31 smithi138 podman[69200]: 2024-04-25 11:48:31.791827926 +0000 UTC m=+0.018025428 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T11:48:32.150 INFO:journalctl@ceph.osd.4.smithi138.stdout:Apr 25 11:48:31 smithi138 podman[69200]: 2024-04-25 11:48:31.904287057 +0000 UTC m=+0.130484585 container create 6bd67c0cb1a6af160c11af0c16e891dee83030cf255b3e5fa4028c07d578c44e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5) 2024-04-25T11:48:32.196 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:31 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:31.778+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:32.196 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:31 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:31.778+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:32.335 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:32 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:32.254+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:58.231765+0000 front 2024-04-25T11:47:58.231620+0000 (oldest deadline 2024-04-25T11:48:24.131776+0000) 2024-04-25T11:48:32.335 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:32 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:32.254+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:04.632804+0000 front 2024-04-25T11:48:04.632689+0000 (oldest deadline 2024-04-25T11:48:28.132834+0000) 2024-04-25T11:48:32.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:32 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:32.044+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:32.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:32 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:32.044+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:32.451 INFO:journalctl@ceph.osd.4.smithi138.stdout:Apr 25 11:48:32 smithi138 podman[69200]: 2024-04-25 11:48:32.164867464 +0000 UTC m=+0.391064950 container init 6bd67c0cb1a6af160c11af0c16e891dee83030cf255b3e5fa4028c07d578c44e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T11:48:32.451 INFO:journalctl@ceph.osd.4.smithi138.stdout:Apr 25 11:48:32 smithi138 podman[69200]: 2024-04-25 11:48:32.168089508 +0000 UTC m=+0.394286997 container start 6bd67c0cb1a6af160c11af0c16e891dee83030cf255b3e5fa4028c07d578c44e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-25T11:48:32.452 INFO:journalctl@ceph.osd.4.smithi138.stdout:Apr 25 11:48:32 smithi138 podman[69200]: 2024-04-25 11:48:32.239890764 +0000 UTC m=+0.466088263 container attach 6bd67c0cb1a6af160c11af0c16e891dee83030cf255b3e5fa4028c07d578c44e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-25T11:48:32.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:32 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:32.211+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.406865+0000 front 2024-04-25T11:47:57.406761+0000 (oldest deadline 2024-04-25T11:48:21.506872+0000) 2024-04-25T11:48:32.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:32 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:32.211+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.008166+0000 front 2024-04-25T11:48:05.008025+0000 (oldest deadline 2024-04-25T11:48:30.307962+0000) 2024-04-25T11:48:32.735 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:32 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:32.473+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:32.735 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:32 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:32.473+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:32.736 INFO:journalctl@ceph.osd.4.smithi138.stdout:Apr 25 11:48:32 smithi138 podman[69200]: 2024-04-25 11:48:32.466536706 +0000 UTC m=+0.692734198 container died 6bd67c0cb1a6af160c11af0c16e891dee83030cf255b3e5fa4028c07d578c44e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-25T11:48:32.992 INFO:journalctl@ceph.osd.4.smithi138.stdout:Apr 25 11:48:32 smithi138 podman[69226]: 2024-04-25 11:48:32.98461014 +0000 UTC m=+0.512747121 container remove 6bd67c0cb1a6af160c11af0c16e891dee83030cf255b3e5fa4028c07d578c44e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T11:48:32.993 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:32 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:32.908+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:06.074147+0000 front 2024-04-25T11:48:06.074119+0000 (oldest deadline 2024-04-25T11:48:30.174234+0000) 2024-04-25T11:48:32.998 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.4.service' 2024-04-25T11:48:33.036 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:48:33.037 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-25T11:48:33.037 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-25T11:48:33.037 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.5 2024-04-25T11:48:33.063 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:32 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:32.798+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:33.063 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:33 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:33.016+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:33.063 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:33 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:33.016+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:33.206 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:32 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:32.813+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:33.206 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:32 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:32.813+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:33.334 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:33 smithi156 systemd[1]: Stopping Ceph osd.5 for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:48:33.335 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:33 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:33.268+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:58.231765+0000 front 2024-04-25T11:47:58.231620+0000 (oldest deadline 2024-04-25T11:48:24.131776+0000) 2024-04-25T11:48:33.335 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:33 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:33.268+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:04.632804+0000 front 2024-04-25T11:48:04.632689+0000 (oldest deadline 2024-04-25T11:48:28.132834+0000) 2024-04-25T11:48:33.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:33 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:33.221+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.406865+0000 front 2024-04-25T11:47:57.406761+0000 (oldest deadline 2024-04-25T11:48:21.506872+0000) 2024-04-25T11:48:33.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:33 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:33.221+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.008166+0000 front 2024-04-25T11:48:05.008025+0000 (oldest deadline 2024-04-25T11:48:30.307962+0000) 2024-04-25T11:48:33.779 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:33 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:33.370+0000 7f5d1f2ce700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:48:33.779 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:33 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:33.370+0000 7f5d1f2ce700 -1 osd.5 97 *** Got signal Terminated *** 2024-04-25T11:48:33.779 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:33 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:33.370+0000 7f5d1f2ce700 -1 osd.5 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T11:48:33.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:33 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:33.517+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:33.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:33 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:33.517+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:34.084 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:33 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:33.792+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:34.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:34 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:34.025+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:34.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:34 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:34.025+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:34.185 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:33 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:33.789+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:34.185 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:33 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:33.789+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:34.299 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:33 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:33.902+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:06.074147+0000 front 2024-04-25T11:48:06.074119+0000 (oldest deadline 2024-04-25T11:48:30.174234+0000) 2024-04-25T11:48:34.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:34 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:34.199+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.406865+0000 front 2024-04-25T11:47:57.406761+0000 (oldest deadline 2024-04-25T11:48:21.506872+0000) 2024-04-25T11:48:34.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:34 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:34.199+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.008166+0000 front 2024-04-25T11:48:05.008025+0000 (oldest deadline 2024-04-25T11:48:30.307962+0000) 2024-04-25T11:48:34.584 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:34 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:34.299+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:58.231765+0000 front 2024-04-25T11:47:58.231620+0000 (oldest deadline 2024-04-25T11:48:24.131776+0000) 2024-04-25T11:48:34.585 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:34 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:34.299+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:04.632804+0000 front 2024-04-25T11:48:04.632689+0000 (oldest deadline 2024-04-25T11:48:28.132834+0000) 2024-04-25T11:48:34.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:34 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:34.517+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:34.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:34 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:34.517+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:34.997 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:34 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:34.756+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:35.169 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:34 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:34.758+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:35.169 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:34 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:34.758+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:35.292 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:35 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:35.012+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:35.292 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:35 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:35.012+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:35.292 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:35 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:35.012+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:48:35.299 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:34 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:34.878+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:06.074147+0000 front 2024-04-25T11:48:06.074119+0000 (oldest deadline 2024-04-25T11:48:30.174234+0000) 2024-04-25T11:48:35.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:35 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:35.184+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.406865+0000 front 2024-04-25T11:47:57.406761+0000 (oldest deadline 2024-04-25T11:48:21.506872+0000) 2024-04-25T11:48:35.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:35 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:35.184+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.008166+0000 front 2024-04-25T11:48:05.008025+0000 (oldest deadline 2024-04-25T11:48:30.307962+0000) 2024-04-25T11:48:35.584 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:35 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:35.306+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:58.231765+0000 front 2024-04-25T11:47:58.231620+0000 (oldest deadline 2024-04-25T11:48:24.131776+0000) 2024-04-25T11:48:35.585 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:35 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:35.306+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:04.632804+0000 front 2024-04-25T11:48:04.632689+0000 (oldest deadline 2024-04-25T11:48:28.132834+0000) 2024-04-25T11:48:35.585 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:35 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:35.306+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.334540+0000 front 2024-04-25T11:48:13.334712+0000 (oldest deadline 2024-04-25T11:48:35.034582+0000) 2024-04-25T11:48:35.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:35 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:35.489+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:35.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:35 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:35.489+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:36.018 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:35 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:35.722+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:36.209 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:35 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:35.788+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:36.209 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:35 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:35.788+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:36.299 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:35 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:35.852+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:06.074147+0000 front 2024-04-25T11:48:06.074119+0000 (oldest deadline 2024-04-25T11:48:30.174234+0000) 2024-04-25T11:48:36.334 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:36 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:36.031+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:36.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:36 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:36.031+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:36.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:36 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:36.031+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:48:36.335 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:36 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:36.256+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:58.231765+0000 front 2024-04-25T11:47:58.231620+0000 (oldest deadline 2024-04-25T11:48:24.131776+0000) 2024-04-25T11:48:36.335 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:36 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:36.256+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:04.632804+0000 front 2024-04-25T11:48:04.632689+0000 (oldest deadline 2024-04-25T11:48:28.132834+0000) 2024-04-25T11:48:36.335 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:36 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:36.256+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.334540+0000 front 2024-04-25T11:48:13.334712+0000 (oldest deadline 2024-04-25T11:48:35.034582+0000) 2024-04-25T11:48:36.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:36 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:36.224+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.406865+0000 front 2024-04-25T11:47:57.406761+0000 (oldest deadline 2024-04-25T11:48:21.506872+0000) 2024-04-25T11:48:36.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:36 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:36.224+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.008166+0000 front 2024-04-25T11:48:05.008025+0000 (oldest deadline 2024-04-25T11:48:30.307962+0000) 2024-04-25T11:48:36.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:36 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:36.466+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:36.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:36 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:36.466+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:37.084 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:36 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:36.763+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:37.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:37 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:37.005+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:37.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:37 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:37.005+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:37.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:37 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:37.005+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:48:37.213 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:36 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:36.805+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:37.213 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:36 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:36.805+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:37.213 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:36 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:36.805+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:48:37.299 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:36 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:36.822+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:06.074147+0000 front 2024-04-25T11:48:06.074119+0000 (oldest deadline 2024-04-25T11:48:30.174234+0000) 2024-04-25T11:48:37.299 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:36 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:36.822+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.275200+0000 front 2024-04-25T11:48:11.275155+0000 (oldest deadline 2024-04-25T11:48:35.975125+0000) 2024-04-25T11:48:37.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:37 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:37.228+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.406865+0000 front 2024-04-25T11:47:57.406761+0000 (oldest deadline 2024-04-25T11:48:21.506872+0000) 2024-04-25T11:48:37.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:37 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:37.228+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.008166+0000 front 2024-04-25T11:48:05.008025+0000 (oldest deadline 2024-04-25T11:48:30.307962+0000) 2024-04-25T11:48:37.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:37 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:37.228+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.509536+0000 front 2024-04-25T11:48:12.509626+0000 (oldest deadline 2024-04-25T11:48:36.609306+0000) 2024-04-25T11:48:37.572 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:37 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:37.439+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:37.572 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:37 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:37.439+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:37.584 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:37 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:37.232+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:58.231765+0000 front 2024-04-25T11:47:58.231620+0000 (oldest deadline 2024-04-25T11:48:24.131776+0000) 2024-04-25T11:48:37.584 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:37 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:37.232+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:04.632804+0000 front 2024-04-25T11:48:04.632689+0000 (oldest deadline 2024-04-25T11:48:28.132834+0000) 2024-04-25T11:48:37.584 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:37 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:37.232+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.334540+0000 front 2024-04-25T11:48:13.334712+0000 (oldest deadline 2024-04-25T11:48:35.034582+0000) 2024-04-25T11:48:38.002 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:37 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:37.751+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:38.049 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:37 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:37.834+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:06.074147+0000 front 2024-04-25T11:48:06.074119+0000 (oldest deadline 2024-04-25T11:48:30.174234+0000) 2024-04-25T11:48:38.049 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:37 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:37.834+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.275200+0000 front 2024-04-25T11:48:11.275155+0000 (oldest deadline 2024-04-25T11:48:35.975125+0000) 2024-04-25T11:48:38.199 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:37 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:37.776+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:38.200 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:37 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:37.776+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:38.200 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:37 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:37.776+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:48:38.335 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:38 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:38.214+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:58.231765+0000 front 2024-04-25T11:47:58.231620+0000 (oldest deadline 2024-04-25T11:48:24.131776+0000) 2024-04-25T11:48:38.335 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:38 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:38.214+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:04.632804+0000 front 2024-04-25T11:48:04.632689+0000 (oldest deadline 2024-04-25T11:48:28.132834+0000) 2024-04-25T11:48:38.335 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:38 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5[44693]: 2024-04-25T11:48:38.214+0000 7f5d1b884700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.334540+0000 front 2024-04-25T11:48:13.334712+0000 (oldest deadline 2024-04-25T11:48:35.034582+0000) 2024-04-25T11:48:38.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:38 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:38.017+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:38.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:38 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:38.017+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:38.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:38 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:38.017+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:48:38.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:38 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:38.213+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.406865+0000 front 2024-04-25T11:47:57.406761+0000 (oldest deadline 2024-04-25T11:48:21.506872+0000) 2024-04-25T11:48:38.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:38 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:38.213+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.008166+0000 front 2024-04-25T11:48:05.008025+0000 (oldest deadline 2024-04-25T11:48:30.307962+0000) 2024-04-25T11:48:38.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:38 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:38.213+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.509536+0000 front 2024-04-25T11:48:12.509626+0000 (oldest deadline 2024-04-25T11:48:36.609306+0000) 2024-04-25T11:48:38.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:38 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:38.479+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:38.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:38 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:38.479+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:38.963 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:38 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:38.714+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:39.172 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:38 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:38.785+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:39.173 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:38 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:38.785+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:39.173 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:38 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:38.785+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:48:39.299 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:38 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:38.814+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:06.074147+0000 front 2024-04-25T11:48:06.074119+0000 (oldest deadline 2024-04-25T11:48:30.174234+0000) 2024-04-25T11:48:39.299 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:38 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:38.814+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.275200+0000 front 2024-04-25T11:48:11.275155+0000 (oldest deadline 2024-04-25T11:48:35.975125+0000) 2024-04-25T11:48:39.306 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:39 smithi156 podman[70452]: 2024-04-25 11:48:39.060576802 +0000 UTC m=+5.811477721 container died 5ea6488d47943c1627a2450c36d75ce7e61afaca05ec54648b363dbfe912e44f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-25T11:48:39.306 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:38 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:38.978+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:39.307 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:38 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:38.978+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:39.307 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:38 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:38.978+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:48:39.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:39 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:39.186+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.406865+0000 front 2024-04-25T11:47:57.406761+0000 (oldest deadline 2024-04-25T11:48:21.506872+0000) 2024-04-25T11:48:39.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:39 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:39.186+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.008166+0000 front 2024-04-25T11:48:05.008025+0000 (oldest deadline 2024-04-25T11:48:30.307962+0000) 2024-04-25T11:48:39.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:39 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:39.186+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.509536+0000 front 2024-04-25T11:48:12.509626+0000 (oldest deadline 2024-04-25T11:48:36.609306+0000) 2024-04-25T11:48:39.584 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:39 smithi156 podman[70452]: 2024-04-25 11:48:39.412535528 +0000 UTC m=+6.163436434 container cleanup 5ea6488d47943c1627a2450c36d75ce7e61afaca05ec54648b363dbfe912e44f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-25T11:48:39.585 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:39 smithi156 bash[70452]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5 2024-04-25T11:48:39.791 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:39 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:39.518+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:39.792 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:39 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:39.518+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:39.792 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:39 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:39.518+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:48:40.000 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:39 smithi156 podman[70471]: 2024-04-25 11:48:39.714232404 +0000 UTC m=+0.651381251 container remove 5ea6488d47943c1627a2450c36d75ce7e61afaca05ec54648b363dbfe912e44f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T11:48:40.000 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:39 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:39.729+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:40.000 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:39 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:39.730+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:14.137664+0000 front 2024-04-25T11:48:14.137787+0000 (oldest deadline 2024-04-25T11:48:39.437882+0000) 2024-04-25T11:48:40.049 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:39 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:39.806+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:06.074147+0000 front 2024-04-25T11:48:06.074119+0000 (oldest deadline 2024-04-25T11:48:30.174234+0000) 2024-04-25T11:48:40.049 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:39 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:39.806+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.275200+0000 front 2024-04-25T11:48:11.275155+0000 (oldest deadline 2024-04-25T11:48:35.975125+0000) 2024-04-25T11:48:40.188 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:39 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:39.830+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:40.188 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:39 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:39.830+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:40.188 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:39 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:39.830+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:48:40.293 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:40 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:40.015+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:40.293 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:40 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:40.015+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:40.294 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:40 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:40.015+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:48:40.294 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:40 smithi156 podman[70562]: 2024-04-25 11:48:39.972924395 +0000 UTC m=+0.017698729 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T11:48:40.294 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:40 smithi156 podman[70562]: 2024-04-25 11:48:40.098370797 +0000 UTC m=+0.143145164 container create 72bbf139676154c94c2f2ca79131b45800568103e722123e2f62fd761defb63b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=) 2024-04-25T11:48:40.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:40 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:40.202+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.406865+0000 front 2024-04-25T11:47:57.406761+0000 (oldest deadline 2024-04-25T11:48:21.506872+0000) 2024-04-25T11:48:40.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:40 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:40.202+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.008166+0000 front 2024-04-25T11:48:05.008025+0000 (oldest deadline 2024-04-25T11:48:30.307962+0000) 2024-04-25T11:48:40.472 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:40 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:40.202+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.509536+0000 front 2024-04-25T11:48:12.509626+0000 (oldest deadline 2024-04-25T11:48:36.609306+0000) 2024-04-25T11:48:40.585 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:40 smithi156 podman[70562]: 2024-04-25 11:48:40.309013458 +0000 UTC m=+0.353787779 container init 72bbf139676154c94c2f2ca79131b45800568103e722123e2f62fd761defb63b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=) 2024-04-25T11:48:40.585 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:40 smithi156 podman[70562]: 2024-04-25 11:48:40.312275687 +0000 UTC m=+0.357050017 container start 72bbf139676154c94c2f2ca79131b45800568103e722123e2f62fd761defb63b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD) 2024-04-25T11:48:40.585 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:40 smithi156 podman[70562]: 2024-04-25 11:48:40.383700199 +0000 UTC m=+0.428474524 container attach 72bbf139676154c94c2f2ca79131b45800568103e722123e2f62fd761defb63b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-25T11:48:40.765 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:40 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:40.480+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:40.765 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:40 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:40.480+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:40.765 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:40 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:40.480+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:48:40.879 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 25 11:48:40 smithi156 podman[70562]: 2024-04-25 11:48:40.606993773 +0000 UTC m=+0.651768095 container died 72bbf139676154c94c2f2ca79131b45800568103e722123e2f62fd761defb63b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-25T11:48:40.879 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:40 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:40.751+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:40.879 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:40 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:40.751+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:14.137664+0000 front 2024-04-25T11:48:14.137787+0000 (oldest deadline 2024-04-25T11:48:39.437882+0000) 2024-04-25T11:48:41.049 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:40 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:40.780+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:06.074147+0000 front 2024-04-25T11:48:06.074119+0000 (oldest deadline 2024-04-25T11:48:30.174234+0000) 2024-04-25T11:48:41.049 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:40 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:40.780+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.275200+0000 front 2024-04-25T11:48:11.275155+0000 (oldest deadline 2024-04-25T11:48:35.975125+0000) 2024-04-25T11:48:41.154 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:40 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:40.971+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:41.155 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:40 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:40.971+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:41.155 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:40 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:40.971+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:48:41.179 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.5.service' 2024-04-25T11:48:41.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:40 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:40.854+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:41.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:40 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:40.854+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:41.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:40 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:40.854+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:48:41.256 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:48:41.257 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-25T11:48:41.257 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-25T11:48:41.257 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.6 2024-04-25T11:48:41.548 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:41 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:41.237+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.406865+0000 front 2024-04-25T11:47:57.406761+0000 (oldest deadline 2024-04-25T11:48:21.506872+0000) 2024-04-25T11:48:41.548 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:41 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:41.237+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.008166+0000 front 2024-04-25T11:48:05.008025+0000 (oldest deadline 2024-04-25T11:48:30.307962+0000) 2024-04-25T11:48:41.548 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:41 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:41.237+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.509536+0000 front 2024-04-25T11:48:12.509626+0000 (oldest deadline 2024-04-25T11:48:36.609306+0000) 2024-04-25T11:48:41.548 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:41 smithi090 systemd[1]: Stopping Ceph osd.6 for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:48:41.548 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:41 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:41.561+0000 7f4b751dc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:48:41.548 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:41 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:41.561+0000 7f4b751dc700 -1 osd.6 97 *** Got signal Terminated *** 2024-04-25T11:48:41.548 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:41 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:41.561+0000 7f4b751dc700 -1 osd.6 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T11:48:41.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:41 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:41.475+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:41.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:41 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:41.475+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:41.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:41 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:41.475+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:48:42.084 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:41 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:41.758+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:42.085 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:41 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:41.758+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:14.137664+0000 front 2024-04-25T11:48:14.137787+0000 (oldest deadline 2024-04-25T11:48:39.437882+0000) 2024-04-25T11:48:42.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:41 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:41.981+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:42.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:41 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:41.981+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:42.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:41 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:41.981+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:48:42.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:41 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:41.823+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:42.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:41 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:41.823+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:42.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:41 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:41.823+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:48:42.299 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:41 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:41.830+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:06.074147+0000 front 2024-04-25T11:48:06.074119+0000 (oldest deadline 2024-04-25T11:48:30.174234+0000) 2024-04-25T11:48:42.299 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:41 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:41.830+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.275200+0000 front 2024-04-25T11:48:11.275155+0000 (oldest deadline 2024-04-25T11:48:35.975125+0000) 2024-04-25T11:48:42.722 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:42 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:42.287+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.406865+0000 front 2024-04-25T11:47:57.406761+0000 (oldest deadline 2024-04-25T11:48:21.506872+0000) 2024-04-25T11:48:42.722 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:42 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:42.287+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.008166+0000 front 2024-04-25T11:48:05.008025+0000 (oldest deadline 2024-04-25T11:48:30.307962+0000) 2024-04-25T11:48:42.722 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:42 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:42.287+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.509536+0000 front 2024-04-25T11:48:12.509626+0000 (oldest deadline 2024-04-25T11:48:36.609306+0000) 2024-04-25T11:48:42.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:42 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:42.466+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:42.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:42 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:42.466+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:42.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:42 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:42.466+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:48:43.084 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:42 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:42.795+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:43.085 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:42 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:42.795+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:14.137664+0000 front 2024-04-25T11:48:14.137787+0000 (oldest deadline 2024-04-25T11:48:39.437882+0000) 2024-04-25T11:48:43.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:42 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:42.948+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:43.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:42 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:42.948+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:43.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:42 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:42.948+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:48:43.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:42 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:42.829+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:43.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:42 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:42.829+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:43.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:42 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:42.829+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:48:43.299 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:42 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:42.878+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:06.074147+0000 front 2024-04-25T11:48:06.074119+0000 (oldest deadline 2024-04-25T11:48:30.174234+0000) 2024-04-25T11:48:43.299 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:42 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:42.878+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.275200+0000 front 2024-04-25T11:48:11.275155+0000 (oldest deadline 2024-04-25T11:48:35.975125+0000) 2024-04-25T11:48:43.722 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:43 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:43.301+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.406865+0000 front 2024-04-25T11:47:57.406761+0000 (oldest deadline 2024-04-25T11:48:21.506872+0000) 2024-04-25T11:48:43.722 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:43 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:43.301+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.008166+0000 front 2024-04-25T11:48:05.008025+0000 (oldest deadline 2024-04-25T11:48:30.307962+0000) 2024-04-25T11:48:43.722 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:43 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:43.301+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.509536+0000 front 2024-04-25T11:48:12.509626+0000 (oldest deadline 2024-04-25T11:48:36.609306+0000) 2024-04-25T11:48:43.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:43 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:43.426+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:43.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:43 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:43.426+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:43.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:43 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:43.426+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:48:43.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:43 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:43.426+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:48:44.084 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:43 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:43.812+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:44.085 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:43 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:43.812+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:14.137664+0000 front 2024-04-25T11:48:14.137787+0000 (oldest deadline 2024-04-25T11:48:39.437882+0000) 2024-04-25T11:48:44.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:43 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:43.960+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:44.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:43 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:43.960+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:44.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:43 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:43.960+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:48:44.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:43 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:43.862+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:44.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:43 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:43.862+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:44.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:43 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:43.862+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:48:44.299 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:43 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:43.908+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:06.074147+0000 front 2024-04-25T11:48:06.074119+0000 (oldest deadline 2024-04-25T11:48:30.174234+0000) 2024-04-25T11:48:44.299 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:43 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:43.908+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.275200+0000 front 2024-04-25T11:48:11.275155+0000 (oldest deadline 2024-04-25T11:48:35.975125+0000) 2024-04-25T11:48:44.299 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:43 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:43.908+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:17.076331+0000 front 2024-04-25T11:48:17.076403+0000 (oldest deadline 2024-04-25T11:48:42.976405+0000) 2024-04-25T11:48:44.722 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:44 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:44.327+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.406865+0000 front 2024-04-25T11:47:57.406761+0000 (oldest deadline 2024-04-25T11:48:21.506872+0000) 2024-04-25T11:48:44.722 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:44 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:44.327+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.008166+0000 front 2024-04-25T11:48:05.008025+0000 (oldest deadline 2024-04-25T11:48:30.307962+0000) 2024-04-25T11:48:44.722 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:44 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:44.327+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.509536+0000 front 2024-04-25T11:48:12.509626+0000 (oldest deadline 2024-04-25T11:48:36.609306+0000) 2024-04-25T11:48:44.723 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:44 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:44.327+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.910369+0000 front 2024-04-25T11:48:18.910411+0000 (oldest deadline 2024-04-25T11:48:44.210394+0000) 2024-04-25T11:48:44.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:44 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:44.446+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:44.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:44 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:44.446+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:44.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:44 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:44.446+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:48:44.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:44 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:44.446+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:48:45.084 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:44 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:44.944+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:45.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:44 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:44.944+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:45.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:44 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:44.944+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:48:45.085 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:44 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:44.826+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:45.085 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:44 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:44.827+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:14.137664+0000 front 2024-04-25T11:48:14.137787+0000 (oldest deadline 2024-04-25T11:48:39.437882+0000) 2024-04-25T11:48:45.085 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:44 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:44.827+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:19.938994+0000 front 2024-04-25T11:48:19.938930+0000 (oldest deadline 2024-04-25T11:48:44.638995+0000) 2024-04-25T11:48:45.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:44 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:44.898+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:45.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:44 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:44.898+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:45.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:44 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:44.898+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:48:45.299 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:44 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:44.860+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:06.074147+0000 front 2024-04-25T11:48:06.074119+0000 (oldest deadline 2024-04-25T11:48:30.174234+0000) 2024-04-25T11:48:45.299 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:44 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:44.860+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.275200+0000 front 2024-04-25T11:48:11.275155+0000 (oldest deadline 2024-04-25T11:48:35.975125+0000) 2024-04-25T11:48:45.299 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:44 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:44.860+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:17.076331+0000 front 2024-04-25T11:48:17.076403+0000 (oldest deadline 2024-04-25T11:48:42.976405+0000) 2024-04-25T11:48:45.722 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:45 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:45.328+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.406865+0000 front 2024-04-25T11:47:57.406761+0000 (oldest deadline 2024-04-25T11:48:21.506872+0000) 2024-04-25T11:48:45.722 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:45 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:45.328+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.008166+0000 front 2024-04-25T11:48:05.008025+0000 (oldest deadline 2024-04-25T11:48:30.307962+0000) 2024-04-25T11:48:45.722 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:45 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:45.328+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.509536+0000 front 2024-04-25T11:48:12.509626+0000 (oldest deadline 2024-04-25T11:48:36.609306+0000) 2024-04-25T11:48:45.722 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:45 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:45.328+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.910369+0000 front 2024-04-25T11:48:18.910411+0000 (oldest deadline 2024-04-25T11:48:44.210394+0000) 2024-04-25T11:48:45.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:45 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:45.398+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:45.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:45 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:45.398+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:45.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:45 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:45.398+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:48:45.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:45 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:45.398+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:48:46.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:45 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:45.869+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:46.223 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:45 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:45.869+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:46.223 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:45 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:45.869+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:48:46.299 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:45 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:45.838+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:06.074147+0000 front 2024-04-25T11:48:06.074119+0000 (oldest deadline 2024-04-25T11:48:30.174234+0000) 2024-04-25T11:48:46.299 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:45 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:45.838+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.275200+0000 front 2024-04-25T11:48:11.275155+0000 (oldest deadline 2024-04-25T11:48:35.975125+0000) 2024-04-25T11:48:46.299 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:45 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:45.838+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:17.076331+0000 front 2024-04-25T11:48:17.076403+0000 (oldest deadline 2024-04-25T11:48:42.976405+0000) 2024-04-25T11:48:46.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:45 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:45.963+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:46.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:45 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:45.963+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:46.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:45 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:45.963+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:48:46.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:45 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:45.865+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:46.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:45 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:45.865+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:14.137664+0000 front 2024-04-25T11:48:14.137787+0000 (oldest deadline 2024-04-25T11:48:39.437882+0000) 2024-04-25T11:48:46.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:45 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:45.865+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:19.938994+0000 front 2024-04-25T11:48:19.938930+0000 (oldest deadline 2024-04-25T11:48:44.638995+0000) 2024-04-25T11:48:46.722 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:46 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:46.338+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.406865+0000 front 2024-04-25T11:47:57.406761+0000 (oldest deadline 2024-04-25T11:48:21.506872+0000) 2024-04-25T11:48:46.722 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:46 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:46.338+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.008166+0000 front 2024-04-25T11:48:05.008025+0000 (oldest deadline 2024-04-25T11:48:30.307962+0000) 2024-04-25T11:48:46.723 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:46 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:46.338+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.509536+0000 front 2024-04-25T11:48:12.509626+0000 (oldest deadline 2024-04-25T11:48:36.609306+0000) 2024-04-25T11:48:46.723 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:46 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6[58065]: 2024-04-25T11:48:46.338+0000 7f4b6d78a700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.910369+0000 front 2024-04-25T11:48:18.910411+0000 (oldest deadline 2024-04-25T11:48:44.210394+0000) 2024-04-25T11:48:46.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:46 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:46.364+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:46.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:46 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:46.364+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:46.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:46 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:46.364+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:48:46.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:46 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:46.364+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:48:47.185 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:46 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:46.873+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:47.185 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:46 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:46.873+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:47.185 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:46 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:46.873+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:48:47.299 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:46 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:46.867+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:06.074147+0000 front 2024-04-25T11:48:06.074119+0000 (oldest deadline 2024-04-25T11:48:30.174234+0000) 2024-04-25T11:48:47.299 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:46 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:46.867+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.275200+0000 front 2024-04-25T11:48:11.275155+0000 (oldest deadline 2024-04-25T11:48:35.975125+0000) 2024-04-25T11:48:47.299 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:46 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:46.867+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:17.076331+0000 front 2024-04-25T11:48:17.076403+0000 (oldest deadline 2024-04-25T11:48:42.976405+0000) 2024-04-25T11:48:47.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:46 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:46.956+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:47.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:46 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:46.956+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:47.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:46 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:46.956+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:48:47.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:46 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:46.884+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:47.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:46 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:46.884+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:14.137664+0000 front 2024-04-25T11:48:14.137787+0000 (oldest deadline 2024-04-25T11:48:39.437882+0000) 2024-04-25T11:48:47.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:46 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:46.884+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:19.938994+0000 front 2024-04-25T11:48:19.938930+0000 (oldest deadline 2024-04-25T11:48:44.638995+0000) 2024-04-25T11:48:47.450 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:47 smithi090 podman[103367]: 2024-04-25 11:48:47.200596234 +0000 UTC m=+5.738668622 container died 970baa2826bfc017d161f6ff23b9e967ca5c91004fed8d0b474d5ab797e7ff97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-25T11:48:47.722 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:47 smithi090 podman[103367]: 2024-04-25 11:48:47.556062017 +0000 UTC m=+6.094134393 container cleanup 970baa2826bfc017d161f6ff23b9e967ca5c91004fed8d0b474d5ab797e7ff97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T11:48:47.722 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:47 smithi090 bash[103367]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6 2024-04-25T11:48:47.800 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:47 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:47.405+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:47.800 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:47 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:47.405+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:47.800 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:47 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:47.405+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:48:47.800 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:47 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:47.405+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:48:48.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:47 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:47.917+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:48.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:47 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:47.917+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:48.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:47 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:47.917+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:48:48.085 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:47 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:47.838+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:48.085 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:47 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:47.838+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:14.137664+0000 front 2024-04-25T11:48:14.137787+0000 (oldest deadline 2024-04-25T11:48:39.437882+0000) 2024-04-25T11:48:48.085 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:47 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:47.838+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:19.938994+0000 front 2024-04-25T11:48:19.938930+0000 (oldest deadline 2024-04-25T11:48:44.638995+0000) 2024-04-25T11:48:48.116 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:47 smithi090 podman[103384]: 2024-04-25 11:48:47.824212621 +0000 UTC m=+0.618322440 container remove 970baa2826bfc017d161f6ff23b9e967ca5c91004fed8d0b474d5ab797e7ff97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-25T11:48:48.116 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:47 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:47.848+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:48.116 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:47 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:47.848+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:48.116 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:47 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:47.848+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:48:48.302 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:47 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:47.833+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:06.074147+0000 front 2024-04-25T11:48:06.074119+0000 (oldest deadline 2024-04-25T11:48:30.174234+0000) 2024-04-25T11:48:48.302 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:47 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:47.833+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.275200+0000 front 2024-04-25T11:48:11.275155+0000 (oldest deadline 2024-04-25T11:48:35.975125+0000) 2024-04-25T11:48:48.302 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:47 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:47.833+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:17.076331+0000 front 2024-04-25T11:48:17.076403+0000 (oldest deadline 2024-04-25T11:48:42.976405+0000) 2024-04-25T11:48:48.404 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:48 smithi090 podman[103471]: 2024-04-25 11:48:48.035004845 +0000 UTC m=+0.018118985 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T11:48:48.404 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:48 smithi090 podman[103471]: 2024-04-25 11:48:48.183637174 +0000 UTC m=+0.166751357 container create 350126208b833a54ea9d9bd3fa0ce8791f24ff86bca5da9d2d1f24423560a777 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True) 2024-04-25T11:48:48.404 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:48 smithi090 podman[103471]: 2024-04-25 11:48:48.419381579 +0000 UTC m=+0.402495702 container init 350126208b833a54ea9d9bd3fa0ce8791f24ff86bca5da9d2d1f24423560a777 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-04-25T11:48:48.702 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:48 smithi090 podman[103471]: 2024-04-25 11:48:48.422417359 +0000 UTC m=+0.405531484 container start 350126208b833a54ea9d9bd3fa0ce8791f24ff86bca5da9d2d1f24423560a777 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-25T11:48:48.703 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:48 smithi090 podman[103471]: 2024-04-25 11:48:48.511153309 +0000 UTC m=+0.494267449 container attach 350126208b833a54ea9d9bd3fa0ce8791f24ff86bca5da9d2d1f24423560a777 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T11:48:48.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:48 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:48.377+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:48.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:48 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:48.377+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:48.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:48 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:48.377+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:48:48.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:48 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:48.377+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:48:48.972 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 25 11:48:48 smithi090 podman[103471]: 2024-04-25 11:48:48.718284092 +0000 UTC m=+0.701398222 container died 350126208b833a54ea9d9bd3fa0ce8791f24ff86bca5da9d2d1f24423560a777 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-25T11:48:48.972 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:48 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:48.850+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:48.973 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:48 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:48.850+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:48.973 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:48 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:48.850+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:48:48.973 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:48 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:48.850+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:21.054749+0000 front 2024-04-25T11:48:21.054782+0000 (oldest deadline 2024-04-25T11:48:48.055431+0000) 2024-04-25T11:48:49.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.6.service' 2024-04-25T11:48:49.298 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:48 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:48.834+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:06.074147+0000 front 2024-04-25T11:48:06.074119+0000 (oldest deadline 2024-04-25T11:48:30.174234+0000) 2024-04-25T11:48:49.299 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:48 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:48.834+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.275200+0000 front 2024-04-25T11:48:11.275155+0000 (oldest deadline 2024-04-25T11:48:35.975125+0000) 2024-04-25T11:48:49.299 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:48 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:48.834+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:17.076331+0000 front 2024-04-25T11:48:17.076403+0000 (oldest deadline 2024-04-25T11:48:42.976405+0000) 2024-04-25T11:48:49.313 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:48:49.313 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-25T11:48:49.313 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-25T11:48:49.313 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.7 2024-04-25T11:48:49.334 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:48 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:48.911+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:49.334 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:48 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:48.911+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:49.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:48 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:48.911+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:48:49.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:48 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:48.872+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:49.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:48 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:48.872+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:14.137664+0000 front 2024-04-25T11:48:14.137787+0000 (oldest deadline 2024-04-25T11:48:39.437882+0000) 2024-04-25T11:48:49.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:48 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:48.872+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:19.938994+0000 front 2024-04-25T11:48:19.938930+0000 (oldest deadline 2024-04-25T11:48:44.638995+0000) 2024-04-25T11:48:49.614 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:49 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:49.352+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:49.614 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:49 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:49.352+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:49.615 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:49 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:49.352+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:48:49.615 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:49 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:49.352+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:48:49.615 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:49 smithi138 systemd[1]: Stopping Ceph osd.7 for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:48:49.615 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:49 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:49.628+0000 7f69a8189700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:48:49.615 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:49 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:49.628+0000 7f69a8189700 -1 osd.7 97 *** Got signal Terminated *** 2024-04-25T11:48:49.615 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:49 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:49.628+0000 7f69a8189700 -1 osd.7 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T11:48:50.049 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:49 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:49.794+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:06.074147+0000 front 2024-04-25T11:48:06.074119+0000 (oldest deadline 2024-04-25T11:48:30.174234+0000) 2024-04-25T11:48:50.049 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:49 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:49.794+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.275200+0000 front 2024-04-25T11:48:11.275155+0000 (oldest deadline 2024-04-25T11:48:35.975125+0000) 2024-04-25T11:48:50.049 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:49 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:49.794+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:17.076331+0000 front 2024-04-25T11:48:17.076403+0000 (oldest deadline 2024-04-25T11:48:42.976405+0000) 2024-04-25T11:48:50.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:49 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:49.899+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:50.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:49 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:49.899+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:50.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:49 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:49.899+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:48:50.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:49 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:49.899+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:21.054749+0000 front 2024-04-25T11:48:21.054782+0000 (oldest deadline 2024-04-25T11:48:48.055431+0000) 2024-04-25T11:48:50.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:49 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:49.901+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:50.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:49 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:49.901+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:50.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:49 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:49.901+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:48:50.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:49 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:49.851+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:50.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:49 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:49.851+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:14.137664+0000 front 2024-04-25T11:48:14.137787+0000 (oldest deadline 2024-04-25T11:48:39.437882+0000) 2024-04-25T11:48:50.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:49 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:49.851+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:19.938994+0000 front 2024-04-25T11:48:19.938930+0000 (oldest deadline 2024-04-25T11:48:44.638995+0000) 2024-04-25T11:48:50.765 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:50 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:50.324+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:50.766 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:50 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:50.324+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:50.766 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:50 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:50.324+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:48:50.766 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:50 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:50.324+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:48:51.049 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:50 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:50.780+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:06.074147+0000 front 2024-04-25T11:48:06.074119+0000 (oldest deadline 2024-04-25T11:48:30.174234+0000) 2024-04-25T11:48:51.049 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:50 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:50.780+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.275200+0000 front 2024-04-25T11:48:11.275155+0000 (oldest deadline 2024-04-25T11:48:35.975125+0000) 2024-04-25T11:48:51.049 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:50 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:50.780+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:17.076331+0000 front 2024-04-25T11:48:17.076403+0000 (oldest deadline 2024-04-25T11:48:42.976405+0000) 2024-04-25T11:48:51.084 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:50 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:50.949+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:51.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:50 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:50.949+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:51.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:50 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:50.949+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:48:51.085 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:50 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:50.808+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:51.085 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:50 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:50.808+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:14.137664+0000 front 2024-04-25T11:48:14.137787+0000 (oldest deadline 2024-04-25T11:48:39.437882+0000) 2024-04-25T11:48:51.085 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:50 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:50.808+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:19.938994+0000 front 2024-04-25T11:48:19.938930+0000 (oldest deadline 2024-04-25T11:48:44.638995+0000) 2024-04-25T11:48:51.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:50 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:50.928+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:51.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:50 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:50.928+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:51.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:50 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:50.928+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:48:51.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:50 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:50.928+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:21.054749+0000 front 2024-04-25T11:48:21.054782+0000 (oldest deadline 2024-04-25T11:48:48.055431+0000) 2024-04-25T11:48:51.762 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:51 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:51.337+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:51.762 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:51 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:51.337+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:51.762 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:51 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:51.337+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:48:51.763 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:51 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:51.337+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:48:52.049 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:51 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:51.777+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:06.074147+0000 front 2024-04-25T11:48:06.074119+0000 (oldest deadline 2024-04-25T11:48:30.174234+0000) 2024-04-25T11:48:52.049 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:51 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:51.777+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.275200+0000 front 2024-04-25T11:48:11.275155+0000 (oldest deadline 2024-04-25T11:48:35.975125+0000) 2024-04-25T11:48:52.049 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:51 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:51.777+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:17.076331+0000 front 2024-04-25T11:48:17.076403+0000 (oldest deadline 2024-04-25T11:48:42.976405+0000) 2024-04-25T11:48:52.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:51 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:51.942+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:52.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:51 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:51.942+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:52.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:51 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:51.942+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:48:52.085 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:51 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:51.814+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:52.086 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:51 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:51.814+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:14.137664+0000 front 2024-04-25T11:48:14.137787+0000 (oldest deadline 2024-04-25T11:48:39.437882+0000) 2024-04-25T11:48:52.086 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:51 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:51.814+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:19.938994+0000 front 2024-04-25T11:48:19.938930+0000 (oldest deadline 2024-04-25T11:48:44.638995+0000) 2024-04-25T11:48:52.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:51 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:51.927+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:52.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:51 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:51.927+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:52.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:51 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:51.927+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:48:52.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:51 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:51.927+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:21.054749+0000 front 2024-04-25T11:48:21.054782+0000 (oldest deadline 2024-04-25T11:48:48.055431+0000) 2024-04-25T11:48:52.749 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:52 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:52.358+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:52.749 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:52 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:52.358+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:52.749 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:52 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:52.358+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:48:52.749 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:52 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:52.358+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:48:52.749 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:52 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:52.358+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:48:53.054 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:52 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:52.763+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:06.074147+0000 front 2024-04-25T11:48:06.074119+0000 (oldest deadline 2024-04-25T11:48:30.174234+0000) 2024-04-25T11:48:53.054 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:52 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:52.763+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.275200+0000 front 2024-04-25T11:48:11.275155+0000 (oldest deadline 2024-04-25T11:48:35.975125+0000) 2024-04-25T11:48:53.054 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:52 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:52.763+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:17.076331+0000 front 2024-04-25T11:48:17.076403+0000 (oldest deadline 2024-04-25T11:48:42.976405+0000) 2024-04-25T11:48:53.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:52 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:52.945+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:53.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:52 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:52.945+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:53.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:52 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:52.945+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:48:53.085 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:52 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:52.804+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:53.085 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:52 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:52.804+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:14.137664+0000 front 2024-04-25T11:48:14.137787+0000 (oldest deadline 2024-04-25T11:48:39.437882+0000) 2024-04-25T11:48:53.085 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:52 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:52.804+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:19.938994+0000 front 2024-04-25T11:48:19.938930+0000 (oldest deadline 2024-04-25T11:48:44.638995+0000) 2024-04-25T11:48:53.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:52 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:52.912+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:53.223 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:52 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:52.912+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:53.223 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:52 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:52.912+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:48:53.223 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:52 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:52.912+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:21.054749+0000 front 2024-04-25T11:48:21.054782+0000 (oldest deadline 2024-04-25T11:48:48.055431+0000) 2024-04-25T11:48:53.729 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:53 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:53.334+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:53.729 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:53 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:53.334+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:53.729 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:53 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:53.334+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:48:53.729 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:53 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:53.334+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:48:53.729 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:53 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:53.334+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:48:54.049 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:53 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:53.744+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:06.074147+0000 front 2024-04-25T11:48:06.074119+0000 (oldest deadline 2024-04-25T11:48:30.174234+0000) 2024-04-25T11:48:54.049 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:53 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:53.744+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.275200+0000 front 2024-04-25T11:48:11.275155+0000 (oldest deadline 2024-04-25T11:48:35.975125+0000) 2024-04-25T11:48:54.049 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:53 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:53.744+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:17.076331+0000 front 2024-04-25T11:48:17.076403+0000 (oldest deadline 2024-04-25T11:48:42.976405+0000) 2024-04-25T11:48:54.049 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:53 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7[50188]: 2024-04-25T11:48:53.744+0000 7f69a0737700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:28.877668+0000 front 2024-04-25T11:48:28.877645+0000 (oldest deadline 2024-04-25T11:48:52.977815+0000) 2024-04-25T11:48:54.085 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:53 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:53.782+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:54.085 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:53 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:53.782+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:14.137664+0000 front 2024-04-25T11:48:14.137787+0000 (oldest deadline 2024-04-25T11:48:39.437882+0000) 2024-04-25T11:48:54.085 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:53 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:53.782+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:19.938994+0000 front 2024-04-25T11:48:19.938930+0000 (oldest deadline 2024-04-25T11:48:44.638995+0000) 2024-04-25T11:48:54.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:53 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:53.935+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:54.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:53 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:53.935+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:54.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:53 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:53.935+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:48:54.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:53 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:53.935+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:48:54.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:53 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:53.922+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:54.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:53 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:53.922+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:54.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:53 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:53.922+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:48:54.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:53 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:53.922+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:21.054749+0000 front 2024-04-25T11:48:21.054782+0000 (oldest deadline 2024-04-25T11:48:48.055431+0000) 2024-04-25T11:48:54.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:54 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:54.288+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:54.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:54 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:54.288+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:54.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:54 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:54.288+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:48:54.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:54 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:54.288+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:48:54.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:54 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:54.288+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:48:55.085 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:54 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:54.832+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:55.085 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:54 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:54.832+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:14.137664+0000 front 2024-04-25T11:48:14.137787+0000 (oldest deadline 2024-04-25T11:48:39.437882+0000) 2024-04-25T11:48:55.085 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:54 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:54.832+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:19.938994+0000 front 2024-04-25T11:48:19.938930+0000 (oldest deadline 2024-04-25T11:48:44.638995+0000) 2024-04-25T11:48:55.085 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:54 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:54.832+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:29.940384+0000 front 2024-04-25T11:48:29.940296+0000 (oldest deadline 2024-04-25T11:48:54.640194+0000) 2024-04-25T11:48:55.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:54 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:54.919+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:55.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:54 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:54.919+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:55.086 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:54 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:54.919+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:48:55.086 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:54 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:54.919+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:48:55.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:54 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:54.889+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:55.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:54 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:54.889+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:55.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:54 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:54.889+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:48:55.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:54 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:54.889+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:21.054749+0000 front 2024-04-25T11:48:21.054782+0000 (oldest deadline 2024-04-25T11:48:48.055431+0000) 2024-04-25T11:48:55.481 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:55 smithi138 podman[69318]: 2024-04-25 11:48:55.225460066 +0000 UTC m=+5.701337865 container died 1df0a9ab48adcbdffaf095d3991a71ece313b0412f24c4e0c438a8b74f209230 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T11:48:55.481 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:55 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:55.251+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:55.481 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:55 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:55.251+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:55.482 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:55 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:55.251+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:48:55.482 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:55 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:55.251+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:48:55.482 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:55 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:55.251+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:48:55.799 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:55 smithi138 podman[69318]: 2024-04-25 11:48:55.561538869 +0000 UTC m=+6.037416663 container cleanup 1df0a9ab48adcbdffaf095d3991a71ece313b0412f24c4e0c438a8b74f209230 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T11:48:55.799 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:55 smithi138 bash[69318]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7 2024-04-25T11:48:55.799 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:55 smithi138 podman[69335]: 2024-04-25 11:48:55.737502148 +0000 UTC m=+0.510406886 container remove 1df0a9ab48adcbdffaf095d3991a71ece313b0412f24c4e0c438a8b74f209230 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424) 2024-04-25T11:48:56.084 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:55 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:55.839+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:56.085 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:55 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:55.839+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:14.137664+0000 front 2024-04-25T11:48:14.137787+0000 (oldest deadline 2024-04-25T11:48:39.437882+0000) 2024-04-25T11:48:56.085 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:55 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:55.839+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:19.938994+0000 front 2024-04-25T11:48:19.938930+0000 (oldest deadline 2024-04-25T11:48:44.638995+0000) 2024-04-25T11:48:56.085 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:55 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:55.839+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:29.940384+0000 front 2024-04-25T11:48:29.940296+0000 (oldest deadline 2024-04-25T11:48:54.640194+0000) 2024-04-25T11:48:56.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:55 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:55.898+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:56.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:55 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:55.898+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:56.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:55 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:55.898+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:48:56.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:55 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:55.898+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:48:56.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:55 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:55.886+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:56.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:55 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:55.886+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:56.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:55 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:55.886+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:48:56.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:55 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:55.886+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:21.054749+0000 front 2024-04-25T11:48:21.054782+0000 (oldest deadline 2024-04-25T11:48:48.055431+0000) 2024-04-25T11:48:56.425 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:56 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:56.280+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:56.425 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:56 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:56.280+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:56.425 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:56 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:56.280+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:48:56.425 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:56 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:56.280+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:48:56.426 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:56 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:56.280+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:48:56.426 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:56 smithi138 podman[69428]: 2024-04-25 11:48:56.078367084 +0000 UTC m=+0.018106135 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T11:48:56.426 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:56 smithi138 podman[69428]: 2024-04-25 11:48:56.205147652 +0000 UTC m=+0.144886702 container create 76ebbe33d421795467d479078a95652841f197241b8d53be416225b685f5dc37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T11:48:56.426 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:56 smithi138 podman[69428]: 2024-04-25 11:48:56.44077215 +0000 UTC m=+0.380511201 container init 76ebbe33d421795467d479078a95652841f197241b8d53be416225b685f5dc37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7-deactivate, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-25T11:48:56.720 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:56 smithi138 podman[69428]: 2024-04-25 11:48:56.444032413 +0000 UTC m=+0.383771472 container start 76ebbe33d421795467d479078a95652841f197241b8d53be416225b685f5dc37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD) 2024-04-25T11:48:56.720 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:56 smithi138 podman[69428]: 2024-04-25 11:48:56.524050346 +0000 UTC m=+0.463789408 container attach 76ebbe33d421795467d479078a95652841f197241b8d53be416225b685f5dc37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5) 2024-04-25T11:48:57.049 INFO:journalctl@ceph.osd.7.smithi138.stdout:Apr 25 11:48:56 smithi138 podman[69428]: 2024-04-25 11:48:56.73579983 +0000 UTC m=+0.675538890 container died 76ebbe33d421795467d479078a95652841f197241b8d53be416225b685f5dc37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-25T11:48:57.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:56 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:56.892+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:57.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:56 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:56.892+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:57.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:56 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:56.892+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:48:57.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:56 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:56.892+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:21.054749+0000 front 2024-04-25T11:48:21.054782+0000 (oldest deadline 2024-04-25T11:48:48.055431+0000) 2024-04-25T11:48:57.249 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.7.service' 2024-04-25T11:48:57.286 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:48:57.287 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-25T11:48:57.287 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-25T11:48:57.287 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.8 2024-04-25T11:48:57.315 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:56 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:56.870+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:57.315 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:56 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:56.870+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:14.137664+0000 front 2024-04-25T11:48:14.137787+0000 (oldest deadline 2024-04-25T11:48:39.437882+0000) 2024-04-25T11:48:57.315 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:56 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:56.870+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:19.938994+0000 front 2024-04-25T11:48:19.938930+0000 (oldest deadline 2024-04-25T11:48:44.638995+0000) 2024-04-25T11:48:57.315 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:56 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:56.870+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:29.940384+0000 front 2024-04-25T11:48:29.940296+0000 (oldest deadline 2024-04-25T11:48:54.640194+0000) 2024-04-25T11:48:57.315 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:56 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:56.906+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:57.316 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:56 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:56.906+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:57.316 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:56 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:56.906+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:48:57.316 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:56 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:56.906+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:48:57.498 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:57 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:57.260+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:57.498 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:57 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:57.260+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:57.499 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:57 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:57.260+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:48:57.499 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:57 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:57.260+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:48:57.499 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:57 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:57.260+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:48:57.585 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:57 smithi156 systemd[1]: Stopping Ceph osd.8 for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:48:57.897 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:57 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:57.605+0000 7fe6c909f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:48:57.897 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:57 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:57.605+0000 7fe6c909f700 -1 osd.8 97 *** Got signal Terminated *** 2024-04-25T11:48:57.897 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:57 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:57.605+0000 7fe6c909f700 -1 osd.8 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T11:48:58.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:57 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:57.909+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:58.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:57 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:57.909+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:58.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:57 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:57.909+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:48:58.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:57 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:57.909+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:21.054749+0000 front 2024-04-25T11:48:21.054782+0000 (oldest deadline 2024-04-25T11:48:48.055431+0000) 2024-04-25T11:48:58.334 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:57 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:57.936+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:58.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:57 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:57.936+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:58.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:57 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:57.936+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:48:58.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:57 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:57.936+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:48:58.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:57 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:57.912+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:58.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:57 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:57.912+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:14.137664+0000 front 2024-04-25T11:48:14.137787+0000 (oldest deadline 2024-04-25T11:48:39.437882+0000) 2024-04-25T11:48:58.336 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:57 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:57.912+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:19.938994+0000 front 2024-04-25T11:48:19.938930+0000 (oldest deadline 2024-04-25T11:48:44.638995+0000) 2024-04-25T11:48:58.336 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:57 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:57.912+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:29.940384+0000 front 2024-04-25T11:48:29.940296+0000 (oldest deadline 2024-04-25T11:48:54.640194+0000) 2024-04-25T11:48:58.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:58 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:58.230+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:58.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:58 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:58.230+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:58.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:58 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:58.230+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:48:58.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:58 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:58.230+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:48:58.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:58 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:58.230+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:48:59.123 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:58 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:58.879+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:48:59.124 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:58 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:58.879+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:48:59.124 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:58 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:58.879+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:48:59.124 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:58 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:58.879+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:21.054749+0000 front 2024-04-25T11:48:21.054782+0000 (oldest deadline 2024-04-25T11:48:48.055431+0000) 2024-04-25T11:48:59.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:58 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:58.950+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:48:59.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:58 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:58.950+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:14.137664+0000 front 2024-04-25T11:48:14.137787+0000 (oldest deadline 2024-04-25T11:48:39.437882+0000) 2024-04-25T11:48:59.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:58 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:58.950+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:19.938994+0000 front 2024-04-25T11:48:19.938930+0000 (oldest deadline 2024-04-25T11:48:44.638995+0000) 2024-04-25T11:48:59.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:58 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:58.950+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:29.940384+0000 front 2024-04-25T11:48:29.940296+0000 (oldest deadline 2024-04-25T11:48:54.640194+0000) 2024-04-25T11:48:59.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:58 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:58.887+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:48:59.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:58 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:58.887+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:48:59.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:58 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:58.887+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:48:59.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:58 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:58.887+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:48:59.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:59 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:59.255+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:48:59.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:59 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:59.255+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:48:59.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:59 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:59.255+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:48:59.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:59 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:59.255+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:48:59.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:48:59 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:48:59.255+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:49:00.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:59 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:59.844+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:49:00.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:59 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:59.844+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:49:00.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:59 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:59.844+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:49:00.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:59 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:59.844+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:21.054749+0000 front 2024-04-25T11:48:21.054782+0000 (oldest deadline 2024-04-25T11:48:48.055431+0000) 2024-04-25T11:49:00.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:48:59 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:48:59.844+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:36.757710+0000 front 2024-04-25T11:48:36.757692+0000 (oldest deadline 2024-04-25T11:48:59.057641+0000) 2024-04-25T11:49:00.334 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:59 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:59.886+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:00.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:59 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:59.886+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:00.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:59 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:59.886+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:00.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:48:59 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:48:59.886+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:00.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:59 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:59.976+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:49:00.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:59 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:59.976+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:14.137664+0000 front 2024-04-25T11:48:14.137787+0000 (oldest deadline 2024-04-25T11:48:39.437882+0000) 2024-04-25T11:49:00.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:59 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:59.976+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:19.938994+0000 front 2024-04-25T11:48:19.938930+0000 (oldest deadline 2024-04-25T11:48:44.638995+0000) 2024-04-25T11:49:00.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:48:59 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:48:59.976+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:29.940384+0000 front 2024-04-25T11:48:29.940296+0000 (oldest deadline 2024-04-25T11:48:54.640194+0000) 2024-04-25T11:49:00.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:00 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:00.253+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:49:00.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:00 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:00.253+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:49:00.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:00 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:00.253+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:49:00.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:00 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:00.253+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:49:00.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:00 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:00.253+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:49:00.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:00 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:00.253+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:38.117491+0000 front 2024-04-25T11:48:38.117467+0000 (oldest deadline 2024-04-25T11:48:59.817342+0000) 2024-04-25T11:49:01.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:00 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:00.882+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:49:01.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:00 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:00.882+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:49:01.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:00 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:00.882+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:49:01.223 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:00 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:00.882+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:21.054749+0000 front 2024-04-25T11:48:21.054782+0000 (oldest deadline 2024-04-25T11:48:48.055431+0000) 2024-04-25T11:49:01.223 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:00 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:00.882+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:36.757710+0000 front 2024-04-25T11:48:36.757692+0000 (oldest deadline 2024-04-25T11:48:59.057641+0000) 2024-04-25T11:49:01.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:49:00 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:49:00.983+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:49:01.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:49:00 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:49:00.983+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:14.137664+0000 front 2024-04-25T11:48:14.137787+0000 (oldest deadline 2024-04-25T11:48:39.437882+0000) 2024-04-25T11:49:01.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:49:00 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:49:00.983+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:19.938994+0000 front 2024-04-25T11:48:19.938930+0000 (oldest deadline 2024-04-25T11:48:44.638995+0000) 2024-04-25T11:49:01.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:49:00 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:49:00.983+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:29.940384+0000 front 2024-04-25T11:48:29.940296+0000 (oldest deadline 2024-04-25T11:48:54.640194+0000) 2024-04-25T11:49:01.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:00 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:00.932+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:01.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:00 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:00.932+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:01.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:00 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:00.932+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:01.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:00 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:00.932+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:01.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:01 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:01.267+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:49:01.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:01 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:01.267+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:49:01.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:01 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:01.267+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:49:01.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:01 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:01.267+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:49:01.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:01 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:01.267+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:49:01.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:01 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:01.267+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:38.117491+0000 front 2024-04-25T11:48:38.117467+0000 (oldest deadline 2024-04-25T11:48:59.817342+0000) 2024-04-25T11:49:02.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:01 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:01.928+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:49:02.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:01 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:01.928+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:49:02.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:01 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:01.928+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:49:02.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:01 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:01.928+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:21.054749+0000 front 2024-04-25T11:48:21.054782+0000 (oldest deadline 2024-04-25T11:48:48.055431+0000) 2024-04-25T11:49:02.223 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:01 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:01.928+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:36.757710+0000 front 2024-04-25T11:48:36.757692+0000 (oldest deadline 2024-04-25T11:48:59.057641+0000) 2024-04-25T11:49:02.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:49:01 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:49:01.964+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.636070+0000 front 2024-04-25T11:48:03.636094+0000 (oldest deadline 2024-04-25T11:48:31.836579+0000) 2024-04-25T11:49:02.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:49:01 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:49:01.964+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:14.137664+0000 front 2024-04-25T11:48:14.137787+0000 (oldest deadline 2024-04-25T11:48:39.437882+0000) 2024-04-25T11:49:02.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:49:01 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:49:01.964+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:19.938994+0000 front 2024-04-25T11:48:19.938930+0000 (oldest deadline 2024-04-25T11:48:44.638995+0000) 2024-04-25T11:49:02.335 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:49:01 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8[51671]: 2024-04-25T11:49:01.964+0000 7fe6c5655700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:29.940384+0000 front 2024-04-25T11:48:29.940296+0000 (oldest deadline 2024-04-25T11:48:54.640194+0000) 2024-04-25T11:49:02.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:01 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:01.934+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:02.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:01 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:01.934+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:02.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:01 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:01.934+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:02.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:01 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:01.934+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:02.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:02 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:02.238+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:49:02.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:02 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:02.238+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:49:02.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:02 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:02.238+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:49:02.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:02 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:02.238+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:49:02.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:02 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:02.238+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:49:02.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:02 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:02.238+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:38.117491+0000 front 2024-04-25T11:48:38.117467+0000 (oldest deadline 2024-04-25T11:48:59.817342+0000) 2024-04-25T11:49:03.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:02 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:02.915+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:49:03.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:02 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:02.915+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:49:03.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:02 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:02.915+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:49:03.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:02 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:02.915+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:21.054749+0000 front 2024-04-25T11:48:21.054782+0000 (oldest deadline 2024-04-25T11:48:48.055431+0000) 2024-04-25T11:49:03.223 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:02 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:02.915+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:36.757710+0000 front 2024-04-25T11:48:36.757692+0000 (oldest deadline 2024-04-25T11:48:59.057641+0000) 2024-04-25T11:49:03.324 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:49:03 smithi156 podman[70679]: 2024-04-25 11:49:03.093729963 +0000 UTC m=+5.597864504 container died 0ffaee681c9d4d69105de236a0c26873d6873f55c485c0910774e650d4af05c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-25T11:49:03.324 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:02 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:02.952+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:03.325 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:02 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:02.952+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:03.325 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:02 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:02.952+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:03.325 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:02 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:02.952+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:03.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:03 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:03.283+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:49:03.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:03 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:03.283+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:49:03.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:03 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:03.283+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:49:03.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:03 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:03.283+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:49:03.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:03 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:03.283+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:49:03.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:03 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:03.283+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:38.117491+0000 front 2024-04-25T11:48:38.117467+0000 (oldest deadline 2024-04-25T11:48:59.817342+0000) 2024-04-25T11:49:03.584 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:49:03 smithi156 podman[70679]: 2024-04-25 11:49:03.413835301 +0000 UTC m=+5.917969839 container cleanup 0ffaee681c9d4d69105de236a0c26873d6873f55c485c0910774e650d4af05c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T11:49:03.584 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:49:03 smithi156 bash[70679]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8 2024-04-25T11:49:03.963 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:49:03 smithi156 podman[70697]: 2024-04-25 11:49:03.614708413 +0000 UTC m=+0.517486829 container remove 0ffaee681c9d4d69105de236a0c26873d6873f55c485c0910774e650d4af05c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-25T11:49:04.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:03 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:03.951+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:49:04.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:03 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:03.951+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:49:04.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:03 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:03.951+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:49:04.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:03 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:03.951+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:21.054749+0000 front 2024-04-25T11:48:21.054782+0000 (oldest deadline 2024-04-25T11:48:48.055431+0000) 2024-04-25T11:49:04.223 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:03 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:03.951+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:36.757710+0000 front 2024-04-25T11:48:36.757692+0000 (oldest deadline 2024-04-25T11:48:59.057641+0000) 2024-04-25T11:49:04.269 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:49:04 smithi156 podman[70791]: 2024-04-25 11:49:03.92924149 +0000 UTC m=+0.018873452 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T11:49:04.269 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:49:04 smithi156 podman[70791]: 2024-04-25 11:49:04.048978431 +0000 UTC m=+0.138610393 container create 2ca8dc126a5147e4bb7abd3e776506a4f00b287f27967660a48b7e0b1c23768d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T11:49:04.269 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:49:04 smithi156 podman[70791]: 2024-04-25 11:49:04.284804841 +0000 UTC m=+0.374436802 container init 2ca8dc126a5147e4bb7abd3e776506a4f00b287f27967660a48b7e0b1c23768d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-25T11:49:04.270 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:03 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:03.978+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:04.270 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:03 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:03.978+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:04.270 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:03 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:03.978+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:04.270 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:03 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:03.978+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:04.270 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:03 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:03.978+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:33.502984+0000 front 2024-04-25T11:48:33.502896+0000 (oldest deadline 2024-04-25T11:49:03.503535+0000) 2024-04-25T11:49:04.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:04 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:04.276+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:49:04.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:04 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:04.276+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:49:04.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:04 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:04.276+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:49:04.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:04 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:04.276+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:49:04.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:04 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:04.276+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:49:04.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:04 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:04.276+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:38.117491+0000 front 2024-04-25T11:48:38.117467+0000 (oldest deadline 2024-04-25T11:48:59.817342+0000) 2024-04-25T11:49:04.566 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:49:04 smithi156 podman[70791]: 2024-04-25 11:49:04.28819127 +0000 UTC m=+0.377823232 container start 2ca8dc126a5147e4bb7abd3e776506a4f00b287f27967660a48b7e0b1c23768d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-25T11:49:04.566 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:49:04 smithi156 podman[70791]: 2024-04-25 11:49:04.368784163 +0000 UTC m=+0.458416146 container attach 2ca8dc126a5147e4bb7abd3e776506a4f00b287f27967660a48b7e0b1c23768d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T11:49:04.834 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:49:04 smithi156 podman[70791]: 2024-04-25 11:49:04.581722605 +0000 UTC m=+0.671354571 container died 2ca8dc126a5147e4bb7abd3e776506a4f00b287f27967660a48b7e0b1c23768d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T11:49:05.150 INFO:journalctl@ceph.osd.8.smithi156.stdout:Apr 25 11:49:05 smithi156 podman[70817]: 2024-04-25 11:49:05.148362867 +0000 UTC m=+0.558023976 container remove 2ca8dc126a5147e4bb7abd3e776506a4f00b287f27967660a48b7e0b1c23768d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=) 2024-04-25T11:49:05.150 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:04 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:04.998+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:05.150 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:04 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:04.998+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:05.150 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:04 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:04.998+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:05.151 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:04 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:04.998+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:05.151 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:04 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:04.998+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:33.502984+0000 front 2024-04-25T11:48:33.502896+0000 (oldest deadline 2024-04-25T11:49:03.503535+0000) 2024-04-25T11:49:05.156 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.8.service' 2024-04-25T11:49:05.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:04 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:04.952+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:49:05.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:04 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:04.952+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:49:05.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:04 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:04.952+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:49:05.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:04 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:04.952+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:21.054749+0000 front 2024-04-25T11:48:21.054782+0000 (oldest deadline 2024-04-25T11:48:48.055431+0000) 2024-04-25T11:49:05.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:04 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:04.952+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:36.757710+0000 front 2024-04-25T11:48:36.757692+0000 (oldest deadline 2024-04-25T11:48:59.057641+0000) 2024-04-25T11:49:05.235 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:49:05.235 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-25T11:49:05.235 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-25T11:49:05.235 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.9 2024-04-25T11:49:05.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:05 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:05.268+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:49:05.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:05 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:05.268+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:49:05.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:05 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:05.268+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:49:05.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:05 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:05.268+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:49:05.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:05 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:05.268+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:49:05.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:05 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:05.268+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:38.117491+0000 front 2024-04-25T11:48:38.117467+0000 (oldest deadline 2024-04-25T11:48:59.817342+0000) 2024-04-25T11:49:05.597 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:05 smithi090 systemd[1]: Stopping Ceph osd.9 for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:49:05.597 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:05 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:05.612+0000 7fe6599ed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:49:05.597 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:05 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:05.612+0000 7fe6599ed700 -1 osd.9 97 *** Got signal Terminated *** 2024-04-25T11:49:05.597 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:05 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:05.612+0000 7fe6599ed700 -1 osd.9 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T11:49:06.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:05 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:05.982+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:06.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:05 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:05.982+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:06.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:05 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:05.982+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:06.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:05 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:05.982+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:06.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:05 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:05.982+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:33.502984+0000 front 2024-04-25T11:48:33.502896+0000 (oldest deadline 2024-04-25T11:49:03.503535+0000) 2024-04-25T11:49:06.472 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:06 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:05.999+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:49:06.472 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:06 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:05.999+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:49:06.472 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:06 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:05.999+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:49:06.473 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:06 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:05.999+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:21.054749+0000 front 2024-04-25T11:48:21.054782+0000 (oldest deadline 2024-04-25T11:48:48.055431+0000) 2024-04-25T11:49:06.473 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:06 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:05.999+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:36.757710+0000 front 2024-04-25T11:48:36.757692+0000 (oldest deadline 2024-04-25T11:48:59.057641+0000) 2024-04-25T11:49:06.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:06 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:06.280+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:49:06.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:06 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:06.280+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:49:06.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:06 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:06.280+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:49:06.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:06 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:06.280+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:49:06.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:06 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:06.280+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:49:06.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:06 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:06.280+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:38.117491+0000 front 2024-04-25T11:48:38.117467+0000 (oldest deadline 2024-04-25T11:48:59.817342+0000) 2024-04-25T11:49:07.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:07 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:07.001+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:07.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:07 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:07.001+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:07.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:07 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:07.001+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:07.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:07 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:07.001+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:07.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:07 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:07.001+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:33.502984+0000 front 2024-04-25T11:48:33.502896+0000 (oldest deadline 2024-04-25T11:49:03.503535+0000) 2024-04-25T11:49:07.472 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:07 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:07.037+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:49:07.472 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:07 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:07.037+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:49:07.472 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:07 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:07.037+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:49:07.473 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:07 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:07.037+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:21.054749+0000 front 2024-04-25T11:48:21.054782+0000 (oldest deadline 2024-04-25T11:48:48.055431+0000) 2024-04-25T11:49:07.473 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:07 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:07.037+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:36.757710+0000 front 2024-04-25T11:48:36.757692+0000 (oldest deadline 2024-04-25T11:48:59.057641+0000) 2024-04-25T11:49:07.550 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:07 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:07.250+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:49:07.550 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:07 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:07.250+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:49:07.550 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:07 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:07.250+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:49:07.550 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:07 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:07.250+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:49:07.550 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:07 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:07.250+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:49:07.551 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:07 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:07.250+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:38.117491+0000 front 2024-04-25T11:48:38.117467+0000 (oldest deadline 2024-04-25T11:48:59.817342+0000) 2024-04-25T11:49:07.551 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:07 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:07.250+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:46.119251+0000 front 2024-04-25T11:48:46.119183+0000 (oldest deadline 2024-04-25T11:49:07.219135+0000) 2024-04-25T11:49:08.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:07 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:07.963+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:08.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:07 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:07.963+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:08.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:07 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:07.963+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:08.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:07 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:07.963+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:08.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:07 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:07.963+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:33.502984+0000 front 2024-04-25T11:48:33.502896+0000 (oldest deadline 2024-04-25T11:49:03.503535+0000) 2024-04-25T11:49:08.472 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:08 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:08.042+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:49:08.472 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:08 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:08.042+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:49:08.472 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:08 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:08.043+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:49:08.472 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:08 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:08.043+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:21.054749+0000 front 2024-04-25T11:48:21.054782+0000 (oldest deadline 2024-04-25T11:48:48.055431+0000) 2024-04-25T11:49:08.472 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:08 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:08.043+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:36.757710+0000 front 2024-04-25T11:48:36.757692+0000 (oldest deadline 2024-04-25T11:48:59.057641+0000) 2024-04-25T11:49:08.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:08 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:08.217+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:49:08.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:08 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:08.217+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:49:08.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:08 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:08.217+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:49:08.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:08 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:08.217+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:49:08.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:08 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:08.217+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:49:08.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:08 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:08.217+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:38.117491+0000 front 2024-04-25T11:48:38.117467+0000 (oldest deadline 2024-04-25T11:48:59.817342+0000) 2024-04-25T11:49:08.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:08 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:08.217+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:46.119251+0000 front 2024-04-25T11:48:46.119183+0000 (oldest deadline 2024-04-25T11:49:07.219135+0000) 2024-04-25T11:49:09.236 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:08 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:08.967+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:09.236 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:08 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:08.967+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:09.236 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:08 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:08.967+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:09.237 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:08 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:08.967+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:09.237 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:08 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:08.967+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:33.502984+0000 front 2024-04-25T11:48:33.502896+0000 (oldest deadline 2024-04-25T11:49:03.503535+0000) 2024-04-25T11:49:09.237 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:08 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:08.967+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:43.504106+0000 front 2024-04-25T11:48:43.504027+0000 (oldest deadline 2024-04-25T11:49:08.204005+0000) 2024-04-25T11:49:09.472 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:09 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:09.078+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:49:09.472 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:09 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:09.078+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:49:09.472 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:09 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:09.078+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:49:09.472 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:09 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:09.078+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:21.054749+0000 front 2024-04-25T11:48:21.054782+0000 (oldest deadline 2024-04-25T11:48:48.055431+0000) 2024-04-25T11:49:09.472 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:09 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:09.078+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:36.757710+0000 front 2024-04-25T11:48:36.757692+0000 (oldest deadline 2024-04-25T11:48:59.057641+0000) 2024-04-25T11:49:09.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:09 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:09.236+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:49:09.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:09 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:09.236+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:49:09.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:09 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:09.236+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:49:09.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:09 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:09.236+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:49:09.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:09 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:09.236+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:49:09.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:09 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:09.236+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:38.117491+0000 front 2024-04-25T11:48:38.117467+0000 (oldest deadline 2024-04-25T11:48:59.817342+0000) 2024-04-25T11:49:09.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:09 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:09.236+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:46.119251+0000 front 2024-04-25T11:48:46.119183+0000 (oldest deadline 2024-04-25T11:49:07.219135+0000) 2024-04-25T11:49:10.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:10 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:10.091+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:57.150927+0000 front 2024-04-25T11:47:57.150957+0000 (oldest deadline 2024-04-25T11:48:20.651128+0000) 2024-04-25T11:49:10.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:10 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:10.091+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:05.252660+0000 front 2024-04-25T11:48:05.252864+0000 (oldest deadline 2024-04-25T11:48:29.952542+0000) 2024-04-25T11:49:10.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:10 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:10.091+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:11.654101+0000 front 2024-04-25T11:48:11.654090+0000 (oldest deadline 2024-04-25T11:48:36.353760+0000) 2024-04-25T11:49:10.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:10 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:10.091+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:21.054749+0000 front 2024-04-25T11:48:21.054782+0000 (oldest deadline 2024-04-25T11:48:48.055431+0000) 2024-04-25T11:49:10.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:10 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9[64699]: 2024-04-25T11:49:10.091+0000 7fe655fa3700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:36.757710+0000 front 2024-04-25T11:48:36.757692+0000 (oldest deadline 2024-04-25T11:48:59.057641+0000) 2024-04-25T11:49:10.247 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:09 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:09.965+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:10.247 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:09 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:09.965+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:10.248 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:09 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:09.965+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:10.248 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:09 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:09.965+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:10.248 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:09 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:09.965+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:33.502984+0000 front 2024-04-25T11:48:33.502896+0000 (oldest deadline 2024-04-25T11:49:03.503535+0000) 2024-04-25T11:49:10.248 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:09 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:09.965+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:43.504106+0000 front 2024-04-25T11:48:43.504027+0000 (oldest deadline 2024-04-25T11:49:08.204005+0000) 2024-04-25T11:49:10.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:10 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:10.283+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:49:10.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:10 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:10.283+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:49:10.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:10 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:10.283+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:49:10.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:10 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:10.283+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:49:10.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:10 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:10.283+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:49:10.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:10 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:10.283+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:38.117491+0000 front 2024-04-25T11:48:38.117467+0000 (oldest deadline 2024-04-25T11:48:59.817342+0000) 2024-04-25T11:49:10.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:10 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:10.283+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:46.119251+0000 front 2024-04-25T11:48:46.119183+0000 (oldest deadline 2024-04-25T11:49:07.219135+0000) 2024-04-25T11:49:11.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:10 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:10.980+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:11.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:10 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:10.980+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:11.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:10 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:10.980+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:11.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:10 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:10.980+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:11.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:10 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:10.980+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:33.502984+0000 front 2024-04-25T11:48:33.502896+0000 (oldest deadline 2024-04-25T11:49:03.503535+0000) 2024-04-25T11:49:11.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:10 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:10.980+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:43.504106+0000 front 2024-04-25T11:48:43.504027+0000 (oldest deadline 2024-04-25T11:49:08.204005+0000) 2024-04-25T11:49:11.430 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:11 smithi090 podman[103589]: 2024-04-25 11:49:11.140731944 +0000 UTC m=+5.686444439 container died 1954684f948a4c6ba4b8c78f7a7cf332dbb85e4b10392d5a30a2a41f9bcd41f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-25T11:49:11.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:11 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:11.245+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:49:11.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:11 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:11.245+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:49:11.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:11 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:11.245+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:49:11.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:11 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:11.245+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:49:11.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:11 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:11.245+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:49:11.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:11 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:11.245+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:38.117491+0000 front 2024-04-25T11:48:38.117467+0000 (oldest deadline 2024-04-25T11:48:59.817342+0000) 2024-04-25T11:49:11.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:11 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:11.245+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:46.119251+0000 front 2024-04-25T11:48:46.119183+0000 (oldest deadline 2024-04-25T11:49:07.219135+0000) 2024-04-25T11:49:11.722 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:11 smithi090 podman[103589]: 2024-04-25 11:49:11.446053977 +0000 UTC m=+5.991766484 container cleanup 1954684f948a4c6ba4b8c78f7a7cf332dbb85e4b10392d5a30a2a41f9bcd41f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-25T11:49:11.722 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:11 smithi090 bash[103589]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9 2024-04-25T11:49:11.722 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:11 smithi090 podman[103607]: 2024-04-25 11:49:11.665200434 +0000 UTC m=+0.520723803 container remove 1954684f948a4c6ba4b8c78f7a7cf332dbb85e4b10392d5a30a2a41f9bcd41f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-25T11:49:12.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:11 smithi090 podman[103698]: 2024-04-25 11:49:11.98281606 +0000 UTC m=+0.114115234 container create b3c43738ff2f3387b0f89a193bfcf9b0b0b5705d9d927d732ec17e00b1c09728 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-25T11:49:12.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:11 smithi090 podman[103698]: 2024-04-25 11:49:11.886517616 +0000 UTC m=+0.017816805 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T11:49:12.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:12 smithi090 podman[103698]: 2024-04-25 11:49:12.210282632 +0000 UTC m=+0.341581806 container init b3c43738ff2f3387b0f89a193bfcf9b0b0b5705d9d927d732ec17e00b1c09728 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True) 2024-04-25T11:49:12.222 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:12 smithi090 podman[103698]: 2024-04-25 11:49:12.213272996 +0000 UTC m=+0.344572176 container start b3c43738ff2f3387b0f89a193bfcf9b0b0b5705d9d927d732ec17e00b1c09728 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-25T11:49:12.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:11 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:11.971+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:12.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:11 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:11.971+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:12.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:11 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:11.971+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:12.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:11 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:11.971+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:12.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:11 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:11.971+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:33.502984+0000 front 2024-04-25T11:48:33.502896+0000 (oldest deadline 2024-04-25T11:49:03.503535+0000) 2024-04-25T11:49:12.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:11 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:11.971+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:43.504106+0000 front 2024-04-25T11:48:43.504027+0000 (oldest deadline 2024-04-25T11:49:08.204005+0000) 2024-04-25T11:49:12.486 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:12 smithi090 podman[103698]: 2024-04-25 11:49:12.302143088 +0000 UTC m=+0.433442264 container attach b3c43738ff2f3387b0f89a193bfcf9b0b0b5705d9d927d732ec17e00b1c09728 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True) 2024-04-25T11:49:12.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:12 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:12.250+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:49:12.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:12 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:12.250+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:49:12.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:12 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:12.250+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:49:12.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:12 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:12.250+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:49:12.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:12 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:12.250+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:49:12.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:12 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:12.250+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:38.117491+0000 front 2024-04-25T11:48:38.117467+0000 (oldest deadline 2024-04-25T11:48:59.817342+0000) 2024-04-25T11:49:12.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:12 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:12.250+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:46.119251+0000 front 2024-04-25T11:48:46.119183+0000 (oldest deadline 2024-04-25T11:49:07.219135+0000) 2024-04-25T11:49:12.781 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 25 11:49:12 smithi090 podman[103698]: 2024-04-25 11:49:12.501871859 +0000 UTC m=+0.633171027 container died b3c43738ff2f3387b0f89a193bfcf9b0b0b5705d9d927d732ec17e00b1c09728 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-25T11:49:13.052 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.9.service' 2024-04-25T11:49:13.091 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:49:13.091 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-25T11:49:13.091 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-25T11:49:13.091 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.10 2024-04-25T11:49:13.334 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:12 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:12.963+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:13.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:12 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:12.963+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:13.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:12 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:12.963+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:13.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:12 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:12.963+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:13.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:12 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:12.963+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:33.502984+0000 front 2024-04-25T11:48:33.502896+0000 (oldest deadline 2024-04-25T11:49:03.503535+0000) 2024-04-25T11:49:13.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:12 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:12.963+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:43.504106+0000 front 2024-04-25T11:48:43.504027+0000 (oldest deadline 2024-04-25T11:49:08.204005+0000) 2024-04-25T11:49:13.414 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:13 smithi138 systemd[1]: Stopping Ceph osd.10 for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:49:13.414 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:13 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:13.280+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:49:13.414 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:13 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:13.280+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:49:13.414 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:13 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:13.280+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:49:13.415 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:13 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:13.280+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:49:13.415 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:13 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:13.280+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:49:13.415 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:13 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:13.280+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:38.117491+0000 front 2024-04-25T11:48:38.117467+0000 (oldest deadline 2024-04-25T11:48:59.817342+0000) 2024-04-25T11:49:13.415 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:13 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:13.281+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:46.119251+0000 front 2024-04-25T11:48:46.119183+0000 (oldest deadline 2024-04-25T11:49:07.219135+0000) 2024-04-25T11:49:13.415 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:13 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:13.428+0000 7f6fdf15e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:49:13.415 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:13 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:13.428+0000 7f6fdf15e700 -1 osd.10 97 *** Got signal Terminated *** 2024-04-25T11:49:13.415 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:13 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:13.428+0000 7f6fdf15e700 -1 osd.10 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T11:49:14.334 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:13 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:13.920+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:14.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:13 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:13.920+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:14.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:13 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:13.920+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:14.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:13 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:13.920+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:14.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:13 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:13.920+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:33.502984+0000 front 2024-04-25T11:48:33.502896+0000 (oldest deadline 2024-04-25T11:49:03.503535+0000) 2024-04-25T11:49:14.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:13 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:13.920+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:43.504106+0000 front 2024-04-25T11:48:43.504027+0000 (oldest deadline 2024-04-25T11:49:08.204005+0000) 2024-04-25T11:49:14.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:14 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:14.316+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:49:14.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:14 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:14.316+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:49:14.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:14 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:14.316+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:49:14.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:14 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:14.316+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:49:14.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:14 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:14.316+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:49:14.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:14 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:14.316+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:38.117491+0000 front 2024-04-25T11:48:38.117467+0000 (oldest deadline 2024-04-25T11:48:59.817342+0000) 2024-04-25T11:49:14.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:14 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:14.316+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:46.119251+0000 front 2024-04-25T11:48:46.119183+0000 (oldest deadline 2024-04-25T11:49:07.219135+0000) 2024-04-25T11:49:15.334 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:14 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:14.880+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:15.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:14 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:14.880+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:15.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:14 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:14.880+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:15.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:14 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:14.880+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:15.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:14 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:14.880+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:33.502984+0000 front 2024-04-25T11:48:33.502896+0000 (oldest deadline 2024-04-25T11:49:03.503535+0000) 2024-04-25T11:49:15.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:14 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:14.880+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:43.504106+0000 front 2024-04-25T11:48:43.504027+0000 (oldest deadline 2024-04-25T11:49:08.204005+0000) 2024-04-25T11:49:15.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:15 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:15.277+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:49:15.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:15 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:15.277+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:49:15.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:15 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:15.277+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:49:15.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:15 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:15.277+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:49:15.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:15 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:15.277+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:49:15.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:15 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:15.277+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:38.117491+0000 front 2024-04-25T11:48:38.117467+0000 (oldest deadline 2024-04-25T11:48:59.817342+0000) 2024-04-25T11:49:15.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:15 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:15.277+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:46.119251+0000 front 2024-04-25T11:48:46.119183+0000 (oldest deadline 2024-04-25T11:49:07.219135+0000) 2024-04-25T11:49:16.338 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:15 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:15.923+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:16.338 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:15 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:15.923+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:16.338 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:15 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:15.923+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:16.338 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:15 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:15.923+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:16.339 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:15 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:15.923+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:33.502984+0000 front 2024-04-25T11:48:33.502896+0000 (oldest deadline 2024-04-25T11:49:03.503535+0000) 2024-04-25T11:49:16.339 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:15 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:15.923+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:43.504106+0000 front 2024-04-25T11:48:43.504027+0000 (oldest deadline 2024-04-25T11:49:08.204005+0000) 2024-04-25T11:49:16.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:16 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:16.268+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:49:16.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:16 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:16.268+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:49:16.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:16 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:16.268+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:49:16.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:16 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:16.268+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:49:16.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:16 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:16.268+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:49:16.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:16 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:16.268+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:38.117491+0000 front 2024-04-25T11:48:38.117467+0000 (oldest deadline 2024-04-25T11:48:59.817342+0000) 2024-04-25T11:49:16.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:16 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:16.268+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:46.119251+0000 front 2024-04-25T11:48:46.119183+0000 (oldest deadline 2024-04-25T11:49:07.219135+0000) 2024-04-25T11:49:17.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:16 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:16.914+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:17.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:16 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:16.914+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:17.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:16 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:16.914+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:17.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:16 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:16.914+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:17.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:16 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:16.914+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:33.502984+0000 front 2024-04-25T11:48:33.502896+0000 (oldest deadline 2024-04-25T11:49:03.503535+0000) 2024-04-25T11:49:17.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:16 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:16.914+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:43.504106+0000 front 2024-04-25T11:48:43.504027+0000 (oldest deadline 2024-04-25T11:49:08.204005+0000) 2024-04-25T11:49:17.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:17 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:17.308+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:49:17.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:17 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:17.308+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:49:17.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:17 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:17.308+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:49:17.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:17 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:17.308+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:49:17.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:17 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:17.308+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:49:17.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:17 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:17.308+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:38.117491+0000 front 2024-04-25T11:48:38.117467+0000 (oldest deadline 2024-04-25T11:48:59.817342+0000) 2024-04-25T11:49:17.549 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:17 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:17.308+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:46.119251+0000 front 2024-04-25T11:48:46.119183+0000 (oldest deadline 2024-04-25T11:49:07.219135+0000) 2024-04-25T11:49:18.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:17 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:17.873+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:18.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:17 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:17.873+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:18.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:17 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:17.873+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:18.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:17 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:17.873+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:18.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:17 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:17.873+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:33.502984+0000 front 2024-04-25T11:48:33.502896+0000 (oldest deadline 2024-04-25T11:49:03.503535+0000) 2024-04-25T11:49:18.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:17 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:17.873+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:43.504106+0000 front 2024-04-25T11:48:43.504027+0000 (oldest deadline 2024-04-25T11:49:08.204005+0000) 2024-04-25T11:49:18.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:18 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:18.331+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.311421+0000 front 2024-04-25T11:47:54.311453+0000 (oldest deadline 2024-04-25T11:48:19.011455+0000) 2024-04-25T11:49:18.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:18 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:18.331+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.612933+0000 front 2024-04-25T11:48:03.612965+0000 (oldest deadline 2024-04-25T11:48:27.713100+0000) 2024-04-25T11:49:18.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:18 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:18.331+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:12.914826+0000 front 2024-04-25T11:48:12.914811+0000 (oldest deadline 2024-04-25T11:48:38.715065+0000) 2024-04-25T11:49:18.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:18 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:18.331+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-25T11:48:18.715690+0000 front 2024-04-25T11:48:18.715706+0000 (oldest deadline 2024-04-25T11:48:43.415670+0000) 2024-04-25T11:49:18.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:18 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:18.331+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.516644+0000 front 2024-04-25T11:48:27.516682+0000 (oldest deadline 2024-04-25T11:48:52.216664+0000) 2024-04-25T11:49:18.800 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:18 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:18.331+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:38.117491+0000 front 2024-04-25T11:48:38.117467+0000 (oldest deadline 2024-04-25T11:48:59.817342+0000) 2024-04-25T11:49:18.800 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:18 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10[56890]: 2024-04-25T11:49:18.331+0000 7f6fd770c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:46.119251+0000 front 2024-04-25T11:48:46.119183+0000 (oldest deadline 2024-04-25T11:49:07.219135+0000) 2024-04-25T11:49:19.230 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:18 smithi138 podman[69545]: 2024-04-25 11:49:18.995475276 +0000 UTC m=+5.691319035 container died 5c8c90aefd2c57d121c9964a07ae52054cfefaaaae05a386b5059e4ea813137a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-25T11:49:19.334 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:18 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:18.857+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:19.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:18 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:18.857+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:19.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:18 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:18.857+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:19.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:18 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:18.857+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:19.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:18 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:18.857+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:33.502984+0000 front 2024-04-25T11:48:33.502896+0000 (oldest deadline 2024-04-25T11:49:03.503535+0000) 2024-04-25T11:49:19.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:18 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:18.857+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:43.504106+0000 front 2024-04-25T11:48:43.504027+0000 (oldest deadline 2024-04-25T11:49:08.204005+0000) 2024-04-25T11:49:19.335 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:18 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:18.857+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6820 osd.7 since back 2024-04-25T11:48:52.805600+0000 front 2024-04-25T11:48:52.805520+0000 (oldest deadline 2024-04-25T11:49:18.105382+0000) 2024-04-25T11:49:19.525 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:19 smithi138 podman[69545]: 2024-04-25 11:49:19.337078287 +0000 UTC m=+6.032922025 container cleanup 5c8c90aefd2c57d121c9964a07ae52054cfefaaaae05a386b5059e4ea813137a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-25T11:49:19.525 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:19 smithi138 bash[69545]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10 2024-04-25T11:49:19.799 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:19 smithi138 podman[69564]: 2024-04-25 11:49:19.540485003 +0000 UTC m=+0.539987221 container remove 5c8c90aefd2c57d121c9964a07ae52054cfefaaaae05a386b5059e4ea813137a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-04-25T11:49:20.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:19 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:19.819+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:20.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:19 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:19.819+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:20.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:19 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:19.819+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:20.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:19 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:19.819+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:20.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:19 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:19.819+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:33.502984+0000 front 2024-04-25T11:48:33.502896+0000 (oldest deadline 2024-04-25T11:49:03.503535+0000) 2024-04-25T11:49:20.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:19 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:19.819+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:43.504106+0000 front 2024-04-25T11:48:43.504027+0000 (oldest deadline 2024-04-25T11:49:08.204005+0000) 2024-04-25T11:49:20.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:19 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:19.819+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6820 osd.7 since back 2024-04-25T11:48:52.805600+0000 front 2024-04-25T11:48:52.805520+0000 (oldest deadline 2024-04-25T11:49:18.105382+0000) 2024-04-25T11:49:20.234 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:19 smithi138 podman[69655]: 2024-04-25 11:49:19.939085286 +0000 UTC m=+0.107595590 container create f8aaa9c263eb6ecd1355ab6f814188b48631c85203fce86744ba2e79dd7e7055 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-25T11:49:20.234 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:19 smithi138 podman[69655]: 2024-04-25 11:49:19.849493204 +0000 UTC m=+0.018003521 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T11:49:20.234 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:20 smithi138 podman[69655]: 2024-04-25 11:49:20.17484935 +0000 UTC m=+0.343359654 container init f8aaa9c263eb6ecd1355ab6f814188b48631c85203fce86744ba2e79dd7e7055 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-25T11:49:20.235 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:20 smithi138 podman[69655]: 2024-04-25 11:49:20.178427829 +0000 UTC m=+0.346938135 container start f8aaa9c263eb6ecd1355ab6f814188b48631c85203fce86744ba2e79dd7e7055 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10-deactivate, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-25T11:49:20.235 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:20 smithi138 podman[69655]: 2024-04-25 11:49:20.24968215 +0000 UTC m=+0.418192468 container attach f8aaa9c263eb6ecd1355ab6f814188b48631c85203fce86744ba2e79dd7e7055 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-25T11:49:20.548 INFO:journalctl@ceph.osd.10.smithi138.stdout:Apr 25 11:49:20 smithi138 podman[69655]: 2024-04-25 11:49:20.475984844 +0000 UTC m=+0.644495148 container died f8aaa9c263eb6ecd1355ab6f814188b48631c85203fce86744ba2e79dd7e7055 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-10-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-25T11:49:21.034 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.10.service' 2024-04-25T11:49:21.073 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:49:21.073 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-25T11:49:21.073 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-25T11:49:21.073 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.11 2024-04-25T11:49:21.084 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:20 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:20.809+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:21.084 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:20 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:20.809+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:21.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:20 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:20.809+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:21.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:20 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:20.809+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:21.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:20 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:20.809+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:33.502984+0000 front 2024-04-25T11:48:33.502896+0000 (oldest deadline 2024-04-25T11:49:03.503535+0000) 2024-04-25T11:49:21.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:20 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:20.809+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:43.504106+0000 front 2024-04-25T11:48:43.504027+0000 (oldest deadline 2024-04-25T11:49:08.204005+0000) 2024-04-25T11:49:21.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:20 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:20.809+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6820 osd.7 since back 2024-04-25T11:48:52.805600+0000 front 2024-04-25T11:48:52.805520+0000 (oldest deadline 2024-04-25T11:49:18.105382+0000) 2024-04-25T11:49:21.368 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:21 smithi156 systemd[1]: Stopping Ceph osd.11 for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:49:21.773 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:21 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:21.382+0000 7fdc7cbe6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:49:21.773 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:21 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:21.382+0000 7fdc7cbe6700 -1 osd.11 97 *** Got signal Terminated *** 2024-04-25T11:49:21.773 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:21 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:21.382+0000 7fdc7cbe6700 -1 osd.11 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T11:49:21.773 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:21 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:21.787+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:21.773 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:21 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:21.787+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:22.084 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:21 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:21.787+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:22.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:21 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:21.787+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:22.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:21 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:21.787+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:33.502984+0000 front 2024-04-25T11:48:33.502896+0000 (oldest deadline 2024-04-25T11:49:03.503535+0000) 2024-04-25T11:49:22.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:21 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:21.787+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:43.504106+0000 front 2024-04-25T11:48:43.504027+0000 (oldest deadline 2024-04-25T11:49:08.204005+0000) 2024-04-25T11:49:22.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:21 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:21.787+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6820 osd.7 since back 2024-04-25T11:48:52.805600+0000 front 2024-04-25T11:48:52.805520+0000 (oldest deadline 2024-04-25T11:49:18.105382+0000) 2024-04-25T11:49:23.084 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:22 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:22.807+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:23.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:22 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:22.807+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:23.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:22 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:22.807+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:23.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:22 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:22.807+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:23.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:22 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:22.807+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:33.502984+0000 front 2024-04-25T11:48:33.502896+0000 (oldest deadline 2024-04-25T11:49:03.503535+0000) 2024-04-25T11:49:23.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:22 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:22.807+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:43.504106+0000 front 2024-04-25T11:48:43.504027+0000 (oldest deadline 2024-04-25T11:49:08.204005+0000) 2024-04-25T11:49:23.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:22 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:22.807+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6820 osd.7 since back 2024-04-25T11:48:52.805600+0000 front 2024-04-25T11:48:52.805520+0000 (oldest deadline 2024-04-25T11:49:18.105382+0000) 2024-04-25T11:49:24.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:23 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:23.767+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:24.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:23 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:23.767+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:24.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:23 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:23.767+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:24.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:23 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:23.767+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:24.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:23 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:23.767+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:33.502984+0000 front 2024-04-25T11:48:33.502896+0000 (oldest deadline 2024-04-25T11:49:03.503535+0000) 2024-04-25T11:49:24.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:23 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:23.767+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:43.504106+0000 front 2024-04-25T11:48:43.504027+0000 (oldest deadline 2024-04-25T11:49:08.204005+0000) 2024-04-25T11:49:24.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:23 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:23.767+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6820 osd.7 since back 2024-04-25T11:48:52.805600+0000 front 2024-04-25T11:48:52.805520+0000 (oldest deadline 2024-04-25T11:49:18.105382+0000) 2024-04-25T11:49:24.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:23 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:23.767+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6820 osd.8 since back 2024-04-25T11:48:58.606225+0000 front 2024-04-25T11:48:58.606264+0000 (oldest deadline 2024-04-25T11:49:23.306312+0000) 2024-04-25T11:49:25.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:24 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:24.765+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:25.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:24 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:24.765+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:25.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:24 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:24.765+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:25.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:24 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:24.765+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:25.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:24 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:24.765+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:33.502984+0000 front 2024-04-25T11:48:33.502896+0000 (oldest deadline 2024-04-25T11:49:03.503535+0000) 2024-04-25T11:49:25.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:24 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:24.765+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:43.504106+0000 front 2024-04-25T11:48:43.504027+0000 (oldest deadline 2024-04-25T11:49:08.204005+0000) 2024-04-25T11:49:25.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:24 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:24.765+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6820 osd.7 since back 2024-04-25T11:48:52.805600+0000 front 2024-04-25T11:48:52.805520+0000 (oldest deadline 2024-04-25T11:49:18.105382+0000) 2024-04-25T11:49:25.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:24 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:24.765+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6820 osd.8 since back 2024-04-25T11:48:58.606225+0000 front 2024-04-25T11:48:58.606264+0000 (oldest deadline 2024-04-25T11:49:23.306312+0000) 2024-04-25T11:49:26.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:25 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:25.814+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-25T11:47:54.495726+0000 front 2024-04-25T11:47:54.495663+0000 (oldest deadline 2024-04-25T11:48:19.195631+0000) 2024-04-25T11:49:26.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:25 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:25.814+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6804 osd.1 since back 2024-04-25T11:48:03.896919+0000 front 2024-04-25T11:48:03.897024+0000 (oldest deadline 2024-04-25T11:48:28.596987+0000) 2024-04-25T11:49:26.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:25 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:25.814+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2024-04-25T11:48:13.798754+0000 front 2024-04-25T11:48:13.798719+0000 (oldest deadline 2024-04-25T11:48:34.899004+0000) 2024-04-25T11:49:26.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:25 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:25.814+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6812 osd.4 since back 2024-04-25T11:48:27.701781+0000 front 2024-04-25T11:48:27.701754+0000 (oldest deadline 2024-04-25T11:48:53.001903+0000) 2024-04-25T11:49:26.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:25 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:25.814+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2024-04-25T11:48:33.502984+0000 front 2024-04-25T11:48:33.502896+0000 (oldest deadline 2024-04-25T11:49:03.503535+0000) 2024-04-25T11:49:26.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:25 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:25.814+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-25T11:48:43.504106+0000 front 2024-04-25T11:48:43.504027+0000 (oldest deadline 2024-04-25T11:49:08.204005+0000) 2024-04-25T11:49:26.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:25 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:25.814+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.138:6820 osd.7 since back 2024-04-25T11:48:52.805600+0000 front 2024-04-25T11:48:52.805520+0000 (oldest deadline 2024-04-25T11:49:18.105382+0000) 2024-04-25T11:49:26.085 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:25 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11[58435]: 2024-04-25T11:49:25.814+0000 7fdc75194700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.156:6820 osd.8 since back 2024-04-25T11:48:58.606225+0000 front 2024-04-25T11:48:58.606264+0000 (oldest deadline 2024-04-25T11:49:23.306312+0000) 2024-04-25T11:49:27.260 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:26 smithi156 podman[70909]: 2024-04-25 11:49:26.99210055 +0000 UTC m=+5.704579762 container died 2cedbc52dd3f2e9c177d6f8e67108a7ffbd318732c9cf2e7ca9e6e25769c96a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True) 2024-04-25T11:49:27.584 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:27 smithi156 podman[70909]: 2024-04-25 11:49:27.349667346 +0000 UTC m=+6.062146565 container cleanup 2cedbc52dd3f2e9c177d6f8e67108a7ffbd318732c9cf2e7ca9e6e25769c96a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5) 2024-04-25T11:49:27.584 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:27 smithi156 bash[70909]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11 2024-04-25T11:49:27.886 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:27 smithi156 podman[70927]: 2024-04-25 11:49:27.600803708 +0000 UTC m=+0.613143899 container remove 2cedbc52dd3f2e9c177d6f8e67108a7ffbd318732c9cf2e7ca9e6e25769c96a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-25T11:49:28.138 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:27 smithi156 podman[71018]: 2024-04-25 11:49:27.901950694 +0000 UTC m=+0.102362896 container create cd8113dece805be2e67a64c5d2e797287aa7d613240a6c1570b044520532d0cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-25T11:49:28.139 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:27 smithi156 podman[71018]: 2024-04-25 11:49:27.817509114 +0000 UTC m=+0.017921329 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T11:49:28.139 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:28 smithi156 podman[71018]: 2024-04-25 11:49:28.079302468 +0000 UTC m=+0.279714666 container init cd8113dece805be2e67a64c5d2e797287aa7d613240a6c1570b044520532d0cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T11:49:28.139 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:28 smithi156 podman[71018]: 2024-04-25 11:49:28.082381326 +0000 UTC m=+0.282793525 container start cd8113dece805be2e67a64c5d2e797287aa7d613240a6c1570b044520532d0cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-04-25T11:49:28.139 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:28 smithi156 podman[71018]: 2024-04-25 11:49:28.154069408 +0000 UTC m=+0.354481607 container attach cd8113dece805be2e67a64c5d2e797287aa7d613240a6c1570b044520532d0cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-25T11:49:28.585 INFO:journalctl@ceph.osd.11.smithi156.stdout:Apr 25 11:49:28 smithi156 podman[71018]: 2024-04-25 11:49:28.378120934 +0000 UTC m=+0.578533133 container died cd8113dece805be2e67a64c5d2e797287aa7d613240a6c1570b044520532d0cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T11:49:28.957 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@osd.11.service' 2024-04-25T11:49:29.034 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:49:29.034 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-25T11:49:29.034 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-25T11:49:29.034 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.a 2024-04-25T11:49:29.329 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 25 11:49:29 smithi090 systemd[1]: Stopping Ceph mds.a for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:49:29.722 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 25 11:49:29 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-a[71356]: 2024-04-25T11:49:29.343+0000 7f1883c21700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:49:29.722 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 25 11:49:29 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-a[71356]: 2024-04-25T11:49:29.343+0000 7f1883c21700 -1 mds.a *** got signal Terminated *** 2024-04-25T11:49:33.604 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 25 11:49:33 smithi090 podman[103816]: 2024-04-25 11:49:33.359297254 +0000 UTC m=+4.114516729 container died eb3aab653da13720fe7d2caad8c460f4da60c58945c528353c9990db69e853b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True) 2024-04-25T11:49:33.879 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 25 11:49:33 smithi090 podman[103816]: 2024-04-25 11:49:33.694138219 +0000 UTC m=+4.449357703 container cleanup eb3aab653da13720fe7d2caad8c460f4da60c58945c528353c9990db69e853b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-25T11:49:33.880 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 25 11:49:33 smithi090 bash[103816]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-a 2024-04-25T11:49:34.104 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.a.service' 2024-04-25T11:49:34.139 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:49:34.140 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-25T11:49:34.140 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-25T11:49:34.140 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.d 2024-04-25T11:49:34.450 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 25 11:49:34 smithi090 systemd[1]: Stopping Ceph mds.d for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:49:34.722 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 25 11:49:34 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-d[70399]: 2024-04-25T11:49:34.463+0000 7f41e22c7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:49:34.722 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 25 11:49:34 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-d[70399]: 2024-04-25T11:49:34.463+0000 7f41e22c7700 -1 mds.d *** got signal Terminated *** 2024-04-25T11:49:35.722 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 25 11:49:35 smithi090 podman[103949]: 2024-04-25 11:49:35.447091071 +0000 UTC m=+1.076172205 container died 9b9aed298d55ebb3eb51efb57fef4f8d21fcb332ba8cc499da7b15443a2337dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T11:49:36.070 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 25 11:49:35 smithi090 podman[103949]: 2024-04-25 11:49:35.774596653 +0000 UTC m=+1.403677786 container cleanup 9b9aed298d55ebb3eb51efb57fef4f8d21fcb332ba8cc499da7b15443a2337dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-25T11:49:36.070 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 25 11:49:35 smithi090 bash[103949]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-d 2024-04-25T11:49:36.070 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 25 11:49:36 smithi090 podman[103963]: 2024-04-25 11:49:36.000911643 +0000 UTC m=+0.554800549 container remove 9b9aed298d55ebb3eb51efb57fef4f8d21fcb332ba8cc499da7b15443a2337dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-d, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-25T11:49:36.091 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.d.service' 2024-04-25T11:49:36.123 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:49:36.123 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-25T11:49:36.123 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-25T11:49:36.123 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.g 2024-04-25T11:49:36.447 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 25 11:49:36 smithi090 systemd[1]: Stopping Ceph mds.g for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:49:36.722 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 25 11:49:36 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-g[70996]: 2024-04-25T11:49:36.461+0000 7f22b0e7d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:49:36.722 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 25 11:49:36 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-g[70996]: 2024-04-25T11:49:36.461+0000 7f22b0e7d700 -1 mds.g *** got signal Terminated *** 2024-04-25T11:49:38.613 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 25 11:49:38 smithi090 podman[104070]: 2024-04-25 11:49:38.295395106 +0000 UTC m=+1.931301359 container died d9063c55f074c836cac7fc4c59d478c8049ff82b830f8f07d9d8bf71dc6e0ba0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-g, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True) 2024-04-25T11:49:38.972 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 25 11:49:38 smithi090 podman[104070]: 2024-04-25 11:49:38.628998727 +0000 UTC m=+2.264904980 container cleanup d9063c55f074c836cac7fc4c59d478c8049ff82b830f8f07d9d8bf71dc6e0ba0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-g, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-25T11:49:38.972 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 25 11:49:38 smithi090 bash[104070]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-g 2024-04-25T11:49:38.972 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 25 11:49:38 smithi090 podman[104083]: 2024-04-25 11:49:38.855432629 +0000 UTC m=+0.558511618 container remove d9063c55f074c836cac7fc4c59d478c8049ff82b830f8f07d9d8bf71dc6e0ba0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-g, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T11:49:39.076 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.g.service' 2024-04-25T11:49:39.108 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:49:39.108 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-25T11:49:39.108 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-25T11:49:39.108 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.j 2024-04-25T11:49:39.417 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 25 11:49:39 smithi090 systemd[1]: Stopping Ceph mds.j for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:49:39.722 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 25 11:49:39 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-j[70012]: 2024-04-25T11:49:39.431+0000 7fac71831700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:49:39.722 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 25 11:49:39 smithi090 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-j[70012]: 2024-04-25T11:49:39.432+0000 7fac71831700 -1 mds.j *** got signal Terminated *** 2024-04-25T11:49:40.969 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 25 11:49:40 smithi090 podman[104190]: 2024-04-25 11:49:40.721939695 +0000 UTC m=+1.381725328 container died 4de44129c29a78115e9f372c5e1057ac9a7d8197ca9e0f2ca8bbfb5e5dcd3f35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True) 2024-04-25T11:49:41.222 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 25 11:49:41 smithi090 podman[104190]: 2024-04-25 11:49:41.085132287 +0000 UTC m=+1.744917919 container cleanup 4de44129c29a78115e9f372c5e1057ac9a7d8197ca9e0f2ca8bbfb5e5dcd3f35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-j, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-25T11:49:41.222 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 25 11:49:41 smithi090 bash[104190]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-j 2024-04-25T11:49:41.490 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.j.service' 2024-04-25T11:49:41.521 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:49:41.521 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-25T11:49:41.521 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-25T11:49:41.522 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.b 2024-04-25T11:49:41.799 INFO:journalctl@ceph.mds.b.smithi138.stdout:Apr 25 11:49:41 smithi138 systemd[1]: Stopping Ceph mds.b for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:49:42.064 INFO:journalctl@ceph.mds.b.smithi138.stdout:Apr 25 11:49:41 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-b[62562]: 2024-04-25T11:49:41.911+0000 7f88eb9a9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:49:42.064 INFO:journalctl@ceph.mds.b.smithi138.stdout:Apr 25 11:49:41 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-b[62562]: 2024-04-25T11:49:41.911+0000 7f88eb9a9700 -1 mds.b *** got signal Terminated *** 2024-04-25T11:49:46.298 INFO:journalctl@ceph.mds.b.smithi138.stdout:Apr 25 11:49:45 smithi138 podman[69761]: 2024-04-25 11:49:45.928428295 +0000 UTC m=+4.206135820 container died 64820dcad7edccb2506992fc68dd796f0b0255a9e3709a3794ced5cd81689477 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=) 2024-04-25T11:49:46.666 INFO:journalctl@ceph.mds.b.smithi138.stdout:Apr 25 11:49:46 smithi138 podman[69761]: 2024-04-25 11:49:46.410772517 +0000 UTC m=+4.688480045 container cleanup 64820dcad7edccb2506992fc68dd796f0b0255a9e3709a3794ced5cd81689477 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-b, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux ) 2024-04-25T11:49:46.666 INFO:journalctl@ceph.mds.b.smithi138.stdout:Apr 25 11:49:46 smithi138 bash[69761]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-b 2024-04-25T11:49:46.823 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.b.service' 2024-04-25T11:49:46.858 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:49:46.858 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-25T11:49:46.859 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-25T11:49:46.859 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.e 2024-04-25T11:49:47.174 INFO:journalctl@ceph.mds.e.smithi138.stdout:Apr 25 11:49:46 smithi138 systemd[1]: Stopping Ceph mds.e for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:49:47.548 INFO:journalctl@ceph.mds.e.smithi138.stdout:Apr 25 11:49:47 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-e[61937]: 2024-04-25T11:49:47.189+0000 7f99d2890700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:49:47.549 INFO:journalctl@ceph.mds.e.smithi138.stdout:Apr 25 11:49:47 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-e[61937]: 2024-04-25T11:49:47.189+0000 7f99d2890700 -1 mds.e *** got signal Terminated *** 2024-04-25T11:49:50.549 INFO:journalctl@ceph.mds.e.smithi138.stdout:Apr 25 11:49:50 smithi138 podman[69894]: 2024-04-25 11:49:50.30483802 +0000 UTC m=+3.210068406 container died c48443ecc17aa084bf96d2c805e8d50304eec5dd1e6ec51b4728171b8a2e6888 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-e, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-25T11:49:51.049 INFO:journalctl@ceph.mds.e.smithi138.stdout:Apr 25 11:49:50 smithi138 podman[69894]: 2024-04-25 11:49:50.856070792 +0000 UTC m=+3.761301177 container cleanup c48443ecc17aa084bf96d2c805e8d50304eec5dd1e6ec51b4728171b8a2e6888 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-e, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-25T11:49:51.049 INFO:journalctl@ceph.mds.e.smithi138.stdout:Apr 25 11:49:50 smithi138 bash[69894]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-e 2024-04-25T11:49:51.263 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.e.service' 2024-04-25T11:49:51.295 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:49:51.295 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-25T11:49:51.295 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-25T11:49:51.295 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.h 2024-04-25T11:49:51.550 INFO:journalctl@ceph.mds.h.smithi138.stdout:Apr 25 11:49:51 smithi138 systemd[1]: Stopping Ceph mds.h for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:49:52.048 INFO:journalctl@ceph.mds.h.smithi138.stdout:Apr 25 11:49:51 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-h[62247]: 2024-04-25T11:49:51.642+0000 7fdf539b9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:49:52.049 INFO:journalctl@ceph.mds.h.smithi138.stdout:Apr 25 11:49:51 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-h[62247]: 2024-04-25T11:49:51.642+0000 7fdf539b9700 -1 mds.h *** got signal Terminated *** 2024-04-25T11:49:54.548 INFO:journalctl@ceph.mds.h.smithi138.stdout:Apr 25 11:49:54 smithi138 podman[70013]: 2024-04-25 11:49:54.302475718 +0000 UTC m=+2.768409763 container died e39188a45eb272d3118ebeea3732fe11c726730d713a33f660bc5288968e342d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-h, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-25T11:49:54.951 INFO:journalctl@ceph.mds.h.smithi138.stdout:Apr 25 11:49:54 smithi138 podman[70013]: 2024-04-25 11:49:54.609380896 +0000 UTC m=+3.075314938 container cleanup e39188a45eb272d3118ebeea3732fe11c726730d713a33f660bc5288968e342d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-h, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-25T11:49:54.951 INFO:journalctl@ceph.mds.h.smithi138.stdout:Apr 25 11:49:54 smithi138 bash[70013]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-h 2024-04-25T11:49:54.951 INFO:journalctl@ceph.mds.h.smithi138.stdout:Apr 25 11:49:54 smithi138 podman[70026]: 2024-04-25 11:49:54.768961525 +0000 UTC m=+0.462952799 container remove e39188a45eb272d3118ebeea3732fe11c726730d713a33f660bc5288968e342d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-h, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.build-date=20240424) 2024-04-25T11:49:54.968 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.h.service' 2024-04-25T11:49:54.998 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:49:54.999 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-25T11:49:54.999 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-25T11:49:54.999 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.k 2024-04-25T11:49:55.299 INFO:journalctl@ceph.mds.k.smithi138.stdout:Apr 25 11:49:55 smithi138 systemd[1]: Stopping Ceph mds.k for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:49:55.798 INFO:journalctl@ceph.mds.k.smithi138.stdout:Apr 25 11:49:55 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-k[61626]: 2024-04-25T11:49:55.337+0000 7f92d85da700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:49:55.799 INFO:journalctl@ceph.mds.k.smithi138.stdout:Apr 25 11:49:55 smithi138 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-k[61626]: 2024-04-25T11:49:55.337+0000 7f92d85da700 -1 mds.k *** got signal Terminated *** 2024-04-25T11:49:58.540 INFO:journalctl@ceph.mds.k.smithi138.stdout:Apr 25 11:49:58 smithi138 podman[70135]: 2024-04-25 11:49:58.299244183 +0000 UTC m=+3.063555641 container died 745d946e79eaa9572a961b40968828843f4cff28881b93ee28a19a53e1f5c871 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True) 2024-04-25T11:49:58.799 INFO:journalctl@ceph.mds.k.smithi138.stdout:Apr 25 11:49:58 smithi138 podman[70135]: 2024-04-25 11:49:58.637924922 +0000 UTC m=+3.402236381 container cleanup 745d946e79eaa9572a961b40968828843f4cff28881b93ee28a19a53e1f5c871 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-k, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-25T11:49:58.799 INFO:journalctl@ceph.mds.k.smithi138.stdout:Apr 25 11:49:58 smithi138 bash[70135]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-k 2024-04-25T11:49:59.041 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.k.service' 2024-04-25T11:49:59.061 INFO:journalctl@ceph.mds.k.smithi138.stdout:Apr 25 11:49:58 smithi138 podman[70149]: 2024-04-25 11:49:58.822012296 +0000 UTC m=+0.519533243 container remove 745d946e79eaa9572a961b40968828843f4cff28881b93ee28a19a53e1f5c871 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-k, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-25T11:49:59.061 INFO:journalctl@ceph.mds.k.smithi138.stdout:Apr 25 11:49:59 smithi138 systemd[1]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.k.service: Deactivated successfully. 2024-04-25T11:49:59.061 INFO:journalctl@ceph.mds.k.smithi138.stdout:Apr 25 11:49:59 smithi138 systemd[1]: Stopped Ceph mds.k for 58bd4e7a-02f6-11ef-bc93-c7b262605968. 2024-04-25T11:49:59.061 INFO:journalctl@ceph.mds.k.smithi138.stdout:Apr 25 11:49:59 smithi138 systemd[1]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.k.service: Consumed 2.214s CPU time. 2024-04-25T11:49:59.071 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:49:59.071 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-25T11:49:59.071 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-25T11:49:59.071 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.c 2024-04-25T11:49:59.442 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 11:49:59 smithi156 systemd[1]: Stopping Ceph mds.c for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:49:59.699 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 11:49:59 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-c[63779]: 2024-04-25T11:49:59.457+0000 7f49b18d3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:49:59.699 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 11:49:59 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-c[63779]: 2024-04-25T11:49:59.457+0000 7f49b18d3700 -1 mds.c *** got signal Terminated *** 2024-04-25T11:50:02.584 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 11:50:02 smithi156 podman[71124]: 2024-04-25 11:50:02.306767143 +0000 UTC m=+3.018964220 container died 23e823f56e5e6b40692550da8d7186d88f5d522144bc21f73aed549b16895d1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, ceph=True) 2024-04-25T11:50:02.926 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 11:50:02 smithi156 podman[71124]: 2024-04-25 11:50:02.70735067 +0000 UTC m=+3.419547743 container cleanup 23e823f56e5e6b40692550da8d7186d88f5d522144bc21f73aed549b16895d1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-c, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T11:50:02.926 INFO:journalctl@ceph.mds.c.smithi156.stdout:Apr 25 11:50:02 smithi156 bash[71124]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-c 2024-04-25T11:50:03.050 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.c.service' 2024-04-25T11:50:03.084 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:50:03.084 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-25T11:50:03.084 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-25T11:50:03.085 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.f 2024-04-25T11:50:03.334 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 11:50:03 smithi156 systemd[1]: Stopping Ceph mds.f for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:50:03.834 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 11:50:03 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-f[63468]: 2024-04-25T11:50:03.461+0000 7fda10c45700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:50:03.834 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 11:50:03 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-f[63468]: 2024-04-25T11:50:03.461+0000 7fda10c45700 -1 mds.f *** got signal Terminated *** 2024-04-25T11:50:06.584 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 11:50:06 smithi156 podman[71258]: 2024-04-25 11:50:06.297877833 +0000 UTC m=+2.971422662 container died d546e8c332cf47cc7de2d3c7fbf93f81c48de9c2cb2427c4f9640baf31c1a9de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-f, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-25T11:50:06.950 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 11:50:06 smithi156 podman[71258]: 2024-04-25 11:50:06.628033515 +0000 UTC m=+3.301578340 container cleanup d546e8c332cf47cc7de2d3c7fbf93f81c48de9c2cb2427c4f9640baf31c1a9de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-f, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-25T11:50:06.950 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 11:50:06 smithi156 bash[71258]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-f 2024-04-25T11:50:06.950 INFO:journalctl@ceph.mds.f.smithi156.stdout:Apr 25 11:50:06 smithi156 podman[71272]: 2024-04-25 11:50:06.837527265 +0000 UTC m=+0.536125422 container remove d546e8c332cf47cc7de2d3c7fbf93f81c48de9c2cb2427c4f9640baf31c1a9de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-f, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-04-25T11:50:06.968 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.f.service' 2024-04-25T11:50:07.000 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:50:07.000 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-25T11:50:07.000 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-25T11:50:07.000 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.i 2024-04-25T11:50:07.310 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 11:50:07 smithi156 systemd[1]: Stopping Ceph mds.i for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:50:07.584 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 11:50:07 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-i[64087]: 2024-04-25T11:50:07.323+0000 7f48abb41700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:50:07.584 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 11:50:07 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-i[64087]: 2024-04-25T11:50:07.323+0000 7f48abb41700 -1 mds.i *** got signal Terminated *** 2024-04-25T11:50:10.585 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 11:50:10 smithi156 podman[71378]: 2024-04-25 11:50:10.302515828 +0000 UTC m=+3.074115164 container died e24b5c5c6c38b0d12aec2103648a4c3385d794a2540e134784da2ce4ef43a33f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-25T11:50:11.084 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 11:50:10 smithi156 podman[71378]: 2024-04-25 11:50:10.856316934 +0000 UTC m=+3.627916268 container cleanup e24b5c5c6c38b0d12aec2103648a4c3385d794a2540e134784da2ce4ef43a33f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-i, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-25T11:50:11.084 INFO:journalctl@ceph.mds.i.smithi156.stdout:Apr 25 11:50:10 smithi156 bash[71378]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-i 2024-04-25T11:50:11.229 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.i.service' 2024-04-25T11:50:11.261 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:50:11.261 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-25T11:50:11.261 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-25T11:50:11.261 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.l 2024-04-25T11:50:11.578 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 11:50:11 smithi156 systemd[1]: Stopping Ceph mds.l for 58bd4e7a-02f6-11ef-bc93-c7b262605968... 2024-04-25T11:50:11.834 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 11:50:11 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-l[63156]: 2024-04-25T11:50:11.592+0000 7fbc3d14d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T11:50:11.834 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 11:50:11 smithi156 ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-l[63156]: 2024-04-25T11:50:11.592+0000 7fbc3d14d700 -1 mds.l *** got signal Terminated *** 2024-04-25T11:50:14.562 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 11:50:14 smithi156 podman[71498]: 2024-04-25 11:50:14.302230006 +0000 UTC m=+2.807736799 container died f5c838c4b13542b8d220ffd7c9efca1834b0adf01b81504d03fb23a136bb63d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-25T11:50:14.834 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 11:50:14 smithi156 podman[71498]: 2024-04-25 11:50:14.651914922 +0000 UTC m=+3.157421723 container cleanup f5c838c4b13542b8d220ffd7c9efca1834b0adf01b81504d03fb23a136bb63d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-l, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-25T11:50:14.834 INFO:journalctl@ceph.mds.l.smithi156.stdout:Apr 25 11:50:14 smithi156 bash[71498]: ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968-mds-l 2024-04-25T11:50:15.066 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-58bd4e7a-02f6-11ef-bc93-c7b262605968@mds.l.service' 2024-04-25T11:50:15.098 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T11:50:15.098 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-25T11:50:15.098 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T11:50:15.289 INFO:teuthology.orchestra.run.smithi090.stdout:Deleting cluster with fsid: 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:50:17.704 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T11:50:17.896 INFO:teuthology.orchestra.run.smithi138.stdout:Deleting cluster with fsid: 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:50:20.384 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T11:50:20.573 INFO:teuthology.orchestra.run.smithi156.stdout:Deleting cluster with fsid: 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:50:22.845 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T11:50:22.880 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T11:50:22.911 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T11:50:22.937 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-25T11:50:22.939 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673136/remote/smithi090/crash 2024-04-25T11:50:22.939 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/crash -- . 2024-04-25T11:50:22.966 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T11:50:22.966 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2024-04-25T11:50:22.968 DEBUG:teuthology.misc:Transferring archived files from smithi138:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673136/remote/smithi138/crash 2024-04-25T11:50:22.969 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/crash -- . 2024-04-25T11:50:23.100 INFO:teuthology.orchestra.run.smithi138.stderr:tar: /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T11:50:23.100 INFO:teuthology.orchestra.run.smithi138.stderr:tar: Error is not recoverable: exiting now 2024-04-25T11:50:23.102 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673136/remote/smithi156/crash 2024-04-25T11:50:23.103 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/crash -- . 2024-04-25T11:50:23.140 INFO:teuthology.orchestra.run.smithi156.stderr:tar: /var/lib/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T11:50:23.140 INFO:teuthology.orchestra.run.smithi156.stderr:tar: Error is not recoverable: exiting now 2024-04-25T11:50:23.141 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-25T11:50:23.142 DEBUG:teuthology.orchestra.run.smithi090:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-25T11:50:23.176 INFO:tasks.cephadm:Compressing logs... 2024-04-25T11:50:23.176 DEBUG:teuthology.orchestra.run.smithi090:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T11:50:23.219 DEBUG:teuthology.orchestra.run.smithi138:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T11:50:23.222 DEBUG:teuthology.orchestra.run.smithi156:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T11:50:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.log 2024-04-25T11:50:23.243 INFO:teuthology.orchestra.run.smithi090.stderr:: No such file or directory 2024-04-25T11:50:23.243 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T11:50:23.243 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-25T11:50:23.245 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-25T11:50:23.245 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-25T11:50:23.245 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-25T11:50:23.245 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.j.log: /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-25T11:50:23.245 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-25T11:50:23.245 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-25T11:50:23.246 INFO:teuthology.orchestra.run.smithi156.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose: No such file or directory -- 2024-04-25T11:50:23.246 INFO:teuthology.orchestra.run.smithi156.stderr: /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.log 2024-04-25T11:50:23.247 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-25T11:50:23.247 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-25T11:50:23.257 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mon.a.log: 90.3% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.audit.log.gz 86.6% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T11:50:23.258 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-25T11:50:23.258 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T11:50:23.258 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-25T11:50:23.258 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.cephadm.log: 84.2% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T11:50:23.258 INFO:teuthology.orchestra.run.smithi090.stderr: 94.6% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-25T11:50:23.258 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T11:50:23.258 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-25T11:50:23.261 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.log: gzip -5/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.2.log: --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T11:50:23.261 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-25T11:50:23.261 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-25T11:50:23.261 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.audit.log: /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-25T11:50:23.261 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-25T11:50:23.261 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-25T11:50:23.261 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mgr.z.log: 86.6% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T11:50:23.262 INFO:teuthology.orchestra.run.smithi156.stderr: 90.5% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T11:50:23.262 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T11:50:23.262 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-25T11:50:23.262 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.cephadm.log: 83.7% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T11:50:23.262 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-25T11:50:23.262 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.11.log: 92.1% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-25T11:50:23.263 INFO:teuthology.orchestra.run.smithi138.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.log 2024-04-25T11:50:23.263 INFO:teuthology.orchestra.run.smithi138.stderr:: No such file or directory 2024-04-25T11:50:23.263 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T11:50:23.263 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-25T11:50:23.263 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-25T11:50:23.263 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-25T11:50:23.264 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-25T11:50:23.264 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-25T11:50:23.264 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-25T11:50:23.264 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-25T11:50:23.264 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.h.log: 86.6% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T11:50:23.264 INFO:teuthology.orchestra.run.smithi138.stderr: 90.5% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T11:50:23.264 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T11:50:23.264 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-25T11:50:23.264 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.cephadm.log: 83.8% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T11:50:23.264 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T11:50:23.264 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.e.log: 92.0% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-25T11:50:23.264 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-25T11:50:23.265 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-25T11:50:23.265 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.d.log: 91.1% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T11:50:23.267 INFO:teuthology.orchestra.run.smithi090.stderr: 94.5% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-25T11:50:23.270 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T11:50:23.273 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-25T11:50:23.274 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T11:50:23.275 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T11:50:23.281 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T11:50:23.282 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-25T11:50:23.290 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T11:50:23.291 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.k.log: /var/log/ceph/cephadm.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T11:50:23.311 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.l.log: /var/log/ceph/cephadm.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T11:50:23.311 INFO:teuthology.orchestra.run.smithi156.stderr: 91.0% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T11:50:23.313 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.9.log: /var/log/ceph/cephadm.log: 94.5% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-25T11:50:23.313 INFO:teuthology.orchestra.run.smithi090.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T11:50:23.321 INFO:teuthology.orchestra.run.smithi156.stderr: 94.3% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-25T11:50:23.337 INFO:teuthology.orchestra.run.smithi090.stderr: 94.5% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-25T11:50:23.342 INFO:teuthology.orchestra.run.smithi156.stderr: 92.4% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-25T11:50:23.351 INFO:teuthology.orchestra.run.smithi138.stderr: 92.5% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-25T11:50:23.370 INFO:teuthology.orchestra.run.smithi138.stderr: 92.5% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-25T11:50:23.374 INFO:teuthology.orchestra.run.smithi156.stderr: 94.4% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-25T11:50:23.405 INFO:teuthology.orchestra.run.smithi156.stderr: 92.4% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-25T11:50:23.527 INFO:teuthology.orchestra.run.smithi090.stderr: 90.1% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-25T11:50:23.602 INFO:teuthology.orchestra.run.smithi156.stderr: 91.9% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-25T11:50:23.795 INFO:teuthology.orchestra.run.smithi138.stderr: 92.0% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-25T11:50:24.610 INFO:teuthology.orchestra.run.smithi090.stderr: 90.5% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-25T11:50:26.755 INFO:teuthology.orchestra.run.smithi156.stderr: 93.4% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-25T11:50:26.830 INFO:teuthology.orchestra.run.smithi090.stderr: 93.3% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-25T11:50:26.974 INFO:teuthology.orchestra.run.smithi156.stderr: 93.4% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-25T11:50:27.120 INFO:teuthology.orchestra.run.smithi156.stderr: 93.3% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-25T11:50:27.644 INFO:teuthology.orchestra.run.smithi156.stderr: 93.4% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-25T11:50:27.646 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-04-25T11:50:27.646 INFO:teuthology.orchestra.run.smithi156.stderr:real 0m4.409s 2024-04-25T11:50:27.646 INFO:teuthology.orchestra.run.smithi156.stderr:user 0m15.600s 2024-04-25T11:50:27.646 INFO:teuthology.orchestra.run.smithi156.stderr:sys 0m0.613s 2024-04-25T11:50:27.709 INFO:teuthology.orchestra.run.smithi138.stderr: 93.2% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-25T11:50:27.763 INFO:teuthology.orchestra.run.smithi090.stderr: 93.3% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-25T11:50:27.824 INFO:teuthology.orchestra.run.smithi090.stderr: 93.4% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-25T11:50:27.960 INFO:teuthology.orchestra.run.smithi090.stderr: 93.2% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-25T11:50:27.962 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-25T11:50:27.962 INFO:teuthology.orchestra.run.smithi090.stderr:real 0m4.729s 2024-04-25T11:50:27.962 INFO:teuthology.orchestra.run.smithi090.stderr:user 0m18.471s 2024-04-25T11:50:27.962 INFO:teuthology.orchestra.run.smithi090.stderr:sys 0m0.712s 2024-04-25T11:50:29.076 INFO:teuthology.orchestra.run.smithi138.stderr: 97.4% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-25T11:50:29.197 INFO:teuthology.orchestra.run.smithi138.stderr: 93.4% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-25T11:50:29.331 INFO:teuthology.orchestra.run.smithi138.stderr: 93.2% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-25T11:50:30.017 INFO:teuthology.orchestra.run.smithi138.stderr: 93.3% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-25T11:51:30.234 INFO:teuthology.orchestra.run.smithi138.stderr: 94.5% -- replaced with /var/log/ceph/58bd4e7a-02f6-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-25T11:51:30.236 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-04-25T11:51:30.236 INFO:teuthology.orchestra.run.smithi138.stderr:real 1m6.999s 2024-04-25T11:51:30.236 INFO:teuthology.orchestra.run.smithi138.stderr:user 1m33.192s 2024-04-25T11:51:30.236 INFO:teuthology.orchestra.run.smithi138.stderr:sys 0m3.383s 2024-04-25T11:51:30.236 INFO:tasks.cephadm:Archiving logs... 2024-04-25T11:51:30.237 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673136/remote/smithi090/log 2024-04-25T11:51:30.238 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T11:51:33.026 DEBUG:teuthology.misc:Transferring archived files from smithi138:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673136/remote/smithi138/log 2024-04-25T11:51:33.027 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T11:51:46.302 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673136/remote/smithi156/log 2024-04-25T11:51:46.303 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T11:51:48.724 INFO:tasks.cephadm:Removing cluster... 2024-04-25T11:51:48.724 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 --force 2024-04-25T11:51:48.914 INFO:teuthology.orchestra.run.smithi090.stdout:Deleting cluster with fsid: 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:51:49.191 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 --force 2024-04-25T11:51:49.382 INFO:teuthology.orchestra.run.smithi138.stdout:Deleting cluster with fsid: 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:51:49.746 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 58bd4e7a-02f6-11ef-bc93-c7b262605968 --force 2024-04-25T11:51:49.939 INFO:teuthology.orchestra.run.smithi156.stdout:Deleting cluster with fsid: 58bd4e7a-02f6-11ef-bc93-c7b262605968 2024-04-25T11:51:50.261 INFO:tasks.cephadm:Removing cephadm ... 2024-04-25T11:51:50.262 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T11:51:50.280 DEBUG:teuthology.orchestra.run.smithi138:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T11:51:50.297 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T11:51:50.313 INFO:tasks.cephadm:Teardown complete 2024-04-25T11:51:50.313 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T11:51:50.325 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T11:51:50.326 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T11:51:50.328 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T11:51:50.339 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T11:51:50.439 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T11:51:50.450 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T11:51:50.451 DEBUG:teuthology.orchestra.run.smithi090:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T11:51:50.453 DEBUG:teuthology.orchestra.run.smithi138:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T11:51:50.456 DEBUG:teuthology.orchestra.run.smithi156:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T11:51:50.466 INFO:teuthology.orchestra.run.smithi090.stderr:bash: line 1: ntpq: command not found 2024-04-25T11:51:50.468 INFO:teuthology.orchestra.run.smithi138.stderr:bash: line 1: ntpq: command not found 2024-04-25T11:51:50.469 INFO:teuthology.orchestra.run.smithi156.stderr:bash: line 1: ntpq: command not found 2024-04-25T11:51:50.471 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T11:51:50.471 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2024-04-25T11:51:50.471 INFO:teuthology.orchestra.run.smithi090.stdout:^* hv01.front.sepia.ceph.com 3 6 377 20 +711us[ +709us] +/- 50ms 2024-04-25T11:51:50.471 INFO:teuthology.orchestra.run.smithi090.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 24 -560us[ -562us] +/- 59ms 2024-04-25T11:51:50.472 INFO:teuthology.orchestra.run.smithi090.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 -268us[ -270us] +/- 66ms 2024-04-25T11:51:50.472 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T11:51:50.474 INFO:teuthology.orchestra.run.smithi138.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T11:51:50.474 INFO:teuthology.orchestra.run.smithi138.stdout:=============================================================================== 2024-04-25T11:51:50.474 INFO:teuthology.orchestra.run.smithi138.stdout:^* hv01.front.sepia.ceph.com 3 6 377 20 +730us[ +742us] +/- 50ms 2024-04-25T11:51:50.474 INFO:teuthology.orchestra.run.smithi138.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 20 -547us[ -547us] +/- 59ms 2024-04-25T11:51:50.474 INFO:teuthology.orchestra.run.smithi138.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 22 -258us[ -247us] +/- 66ms 2024-04-25T11:51:50.474 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T11:51:50.476 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T11:51:50.476 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2024-04-25T11:51:50.476 INFO:teuthology.orchestra.run.smithi156.stdout:^* hv01.front.sepia.ceph.com 3 6 377 18 +701us[ +704us] +/- 50ms 2024-04-25T11:51:50.476 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 18 -575us[ -575us] +/- 59ms 2024-04-25T11:51:50.476 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 -306us[ -303us] +/- 66ms 2024-04-25T11:51:50.476 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T11:51:50.477 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T11:51:50.488 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T11:51:50.488 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T11:51:50.499 DEBUG:teuthology.orchestra.run.smithi090:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T11:51:50.595 DEBUG:teuthology.orchestra.run.smithi138:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T11:51:50.659 DEBUG:teuthology.orchestra.run.smithi156:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T11:51:50.724 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T11:51:50.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T11:51:50.752 DEBUG:teuthology.orchestra.run.smithi138:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T11:51:50.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T11:51:50.781 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T11:51:50.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T11:51:50.807 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T11:51:50.819 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T11:51:50.829 INFO:teuthology.task.internal:Duration was 2075.438594 seconds 2024-04-25T11:51:50.830 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T11:51:50.844 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T11:51:50.844 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T11:51:50.847 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T11:51:50.849 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T11:51:50.882 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T11:51:50.883 INFO:teuthology.orchestra.run.smithi138.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T11:51:50.885 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T11:51:51.340 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T11:51:51.340 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi090.front.sepia.ceph.com 2024-04-25T11:51:51.340 DEBUG:teuthology.orchestra.run.smithi090:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T11:51:51.361 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi138.front.sepia.ceph.com 2024-04-25T11:51:51.361 DEBUG:teuthology.orchestra.run.smithi138:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T11:51:51.403 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi156.front.sepia.ceph.com 2024-04-25T11:51:51.404 DEBUG:teuthology.orchestra.run.smithi156:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T11:51:51.429 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T11:51:51.429 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T11:51:51.432 DEBUG:teuthology.orchestra.run.smithi138:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T11:51:51.446 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T11:51:51.494 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T11:51:51.495 DEBUG:teuthology.orchestra.run.smithi090:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T11:51:51.497 DEBUG:teuthology.orchestra.run.smithi138:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T11:51:51.511 DEBUG:teuthology.orchestra.run.smithi156:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T11:51:51.882 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T11:51:51.895 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T11:51:51.895 DEBUG:teuthology.orchestra.run.smithi090:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T11:51:51.918 DEBUG:teuthology.orchestra.run.smithi138:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T11:51:51.943 DEBUG:teuthology.orchestra.run.smithi156:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T11:51:51.968 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T11:51:51.982 DEBUG:teuthology.orchestra.run.smithi090:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T11:51:51.984 DEBUG:teuthology.orchestra.run.smithi138:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T11:51:51.988 DEBUG:teuthology.orchestra.run.smithi156:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T11:51:52.004 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = core 2024-04-25T11:51:52.006 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern = core 2024-04-25T11:51:52.030 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = core 2024-04-25T11:51:52.051 DEBUG:teuthology.orchestra.run.smithi090:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T11:51:52.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T11:51:52.072 DEBUG:teuthology.orchestra.run.smithi138:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T11:51:52.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T11:51:52.087 DEBUG:teuthology.orchestra.run.smithi156:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T11:51:52.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T11:51:52.106 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T11:51:52.119 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T11:51:52.119 DEBUG:teuthology.misc:Transferring archived files from smithi090:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673136/remote/smithi090 2024-04-25T11:51:52.120 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T11:51:52.164 DEBUG:teuthology.misc:Transferring archived files from smithi138:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673136/remote/smithi138 2024-04-25T11:51:52.164 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T11:51:52.203 DEBUG:teuthology.misc:Transferring archived files from smithi156:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673136/remote/smithi156 2024-04-25T11:51:52.203 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T11:51:52.239 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T11:51:52.239 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T11:51:52.242 DEBUG:teuthology.orchestra.run.smithi138:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T11:51:52.244 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T11:51:52.290 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T11:51:52.304 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T11:51:52.304 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T11:51:52.328 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T11:51:52.329 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T11:51:52.331 DEBUG:teuthology.orchestra.run.smithi138:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T11:51:52.333 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T11:51:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 11:51 /home/ubuntu/cephtest 2024-04-25T11:51:52.347 INFO:teuthology.orchestra.run.smithi138.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 11:51 /home/ubuntu/cephtest 2024-04-25T11:51:52.347 INFO:teuthology.orchestra.run.smithi156.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 11:51 /home/ubuntu/cephtest 2024-04-25T11:51:52.348 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T11:51:52.361 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T11:51:52.392 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} duration: 2075.438593864441 flavor: default owner: scheduled_pdonnell@teuthology success: true 2024-04-25T11:51:52.393 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T11:51:52.490 INFO:teuthology.run:pass